feat: moderation
This commit is contained in:
zhr
parent
45720ae2e1
commit
7f2567cf15
@ -29,7 +29,7 @@ location ~ ^/api/admin/spark/users/export-zip {
|
|||||||
}
|
}
|
||||||
|
|
||||||
# spark-server：IM / 群 / 投诉 / 排行 / 用户检索 / 音乐列表与 V3 账号运营
|
# spark-server：IM / 群 / 投诉 / 排行 / 用户检索 / 音乐列表与 V3 账号运营
|
||||||
location ~ ^/api/admin/(spark|pop)/(im|groups|complaints|ops-catalog|rankings|music)(/|$) {
|
location ~ ^/api/admin/(spark|pop)/(im|groups|complaints|ops-catalog|rankings|music|sensitive)(/|$) {
|
||||||
proxy_pass http://spark_im_admin;
|
proxy_pass http://spark_im_admin;
|
||||||
proxy_set_header Host $host;
|
proxy_set_header Host $host;
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
|||||||
@ -24,3 +24,145 @@
|
|||||||
### 验证
|
### 验证
|
||||||
|
|
||||||
- `make test`: pass
|
- `make test`: pass
|
||||||
|
|
||||||
|
## 11:24 — 敏感词发送处罚详细设计
|
||||||
|
|
||||||
|
### 概要
|
||||||
|
|
||||||
|
- 对照当前发送链、用户状态、Admin 封禁与 `StatusHidden`，写成可实施的机审设计：`moderation` 模块、AccountGate、分类处罚、表结构、错误码与分期；v1 不做 hide。
|
||||||
|
|
||||||
|
### 变更范围
|
||||||
|
|
||||||
|
| 类型 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 文档 | `docs/moderation.md` |
|
||||||
|
|
||||||
|
### 主要文件
|
||||||
|
|
||||||
|
- `docs/moderation.md` — IM 文本敏感词与处罚设计稿
|
||||||
|
|
||||||
|
### 验证
|
||||||
|
|
||||||
|
- `make test`: skipped（纯文档，无编译/测试面）
|
||||||
|
|
||||||
|
### 关联
|
||||||
|
|
||||||
|
- 对话：文本发送敏感词分类处罚
|
||||||
|
|
||||||
|
## 11:35 — 敏感词匹配算法设计
|
||||||
|
|
||||||
|
### 概要
|
||||||
|
|
||||||
|
- 补全匹配：逐 rune 折叠+丢分隔、Aho-Corasick、区间映回原文、replace 合并打码；v1 无简繁、无英文词边界。
|
||||||
|
|
||||||
|
### 变更范围
|
||||||
|
|
||||||
|
| 类型 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 文档 | `docs/moderation.md` §6 |
|
||||||
|
|
||||||
|
### 主要文件
|
||||||
|
|
||||||
|
- `docs/moderation.md` — 匹配流水线与用例
|
||||||
|
|
||||||
|
### 验证
|
||||||
|
|
||||||
|
- `make test`: skipped（纯文档）
|
||||||
|
|
||||||
|
### 关联
|
||||||
|
|
||||||
|
- 对话：敏感词如何匹配
|
||||||
|
|
||||||
|
## 15:10 — 敏感词机审 v1 实现
|
||||||
|
|
||||||
|
### 概要
|
||||||
|
|
||||||
|
- 落地 `pkg/sensitive`（Fold + Aho-Corasick）、`moderation` 模块、发送链 AccountGate/Scan/处罚、`40302`/`40303`、Admin 词库与命中查询；封禁与机审 disable 统一踢线。
|
||||||
|
|
||||||
|
### 变更范围
|
||||||
|
|
||||||
|
| 类型 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 功能 | 文本发送敏感词；全站禁言；封号踢线 |
|
||||||
|
| 迁移 | `0028_moderation.sql` |
|
||||||
|
| HTTP Admin | `/api/admin/spark|pop/sensitive-words*`、`sensitive-hits` |
|
||||||
|
|
||||||
|
### 主要文件
|
||||||
|
|
||||||
|
- `migrations/0028_moderation.sql`
|
||||||
|
- `internal/pkg/sensitive/`、`internal/module/moderation/`
|
||||||
|
- `internal/module/message/logic/send_moderation.go`
|
||||||
|
- `internal/module/user/logic/account_send.go`
|
||||||
|
- `internal/bootstrap/app.go`、`internal/transport/http/router_admin.go`
|
||||||
|
|
||||||
|
### 验证
|
||||||
|
|
||||||
|
- `make test`: pass
|
||||||
|
|
||||||
|
### 关联
|
||||||
|
|
||||||
|
- `docs/moderation.md`
|
||||||
|
|
||||||
|
## 15:32 — 敏感词种子 SQL 与 OpenAPI
|
||||||
|
|
||||||
|
### 概要
|
||||||
|
|
||||||
|
- 新增 `0029_sensitive_words_seed.sql`（按分类默认处罚 16 条示例词）；OpenAPI 增加 `admin-moderation` 路径与 schema；envelope 增加 `40302`/`40303`。
|
||||||
|
|
||||||
|
### 变更范围
|
||||||
|
|
||||||
|
| 类型 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 迁移 | `0029_sensitive_words_seed.sql` |
|
||||||
|
| 文档 | `internal/transport/http/docs/openapi.yaml` |
|
||||||
|
| API | 词条/命中 JSON 字段 snake_case |
|
||||||
|
|
||||||
|
### 主要文件
|
||||||
|
|
||||||
|
- `migrations/0029_sensitive_words_seed.sql`
|
||||||
|
- `internal/transport/http/docs/openapi.yaml`
|
||||||
|
- `internal/module/moderation/entity/word.go`
|
||||||
|
|
||||||
|
### 验证
|
||||||
|
|
||||||
|
- `make test`: pass
|
||||||
|
|
||||||
|
## 15:55 — OpenAPI 管理端 tag 统一为 admin
|
||||||
|
|
||||||
|
### 概要
|
||||||
|
|
||||||
|
- 不改 Gin 路由；去掉 tag `admin-music`，音乐与敏感词等管理接口统一为 `admin`，description 按 `…/spark/im|music|sensitive-words` 等路径分模块。
|
||||||
|
|
||||||
|
### 变更范围
|
||||||
|
|
||||||
|
| 类型 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 文档 | `openapi.yaml`、`docs/moderation.md`、`docs/popadmin-music-novels-deploy.md` |
|
||||||
|
|
||||||
|
### 主要文件
|
||||||
|
|
||||||
|
- `internal/transport/http/docs/openapi.yaml`
|
||||||
|
|
||||||
|
### 验证
|
||||||
|
|
||||||
|
- `make test`: pass
|
||||||
|
|
||||||
|
## 15:55 — OpenAPI 管理端 tag 统一为 admin
|
||||||
|
|
||||||
|
### 概要
|
||||||
|
|
||||||
|
- 不改 Gin 路由；去掉 tag `admin-music`，音乐与敏感词等统一为 `admin`，description 按 `…/spark/im|music|sensitive-words` 等路径分模块。
|
||||||
|
|
||||||
|
### 变更范围
|
||||||
|
|
||||||
|
| 类型 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 文档 | `openapi.yaml`、`docs/moderation.md`、`docs/popadmin-music-novels-deploy.md` |
|
||||||
|
|
||||||
|
### 主要文件
|
||||||
|
|
||||||
|
- `internal/transport/http/docs/openapi.yaml`
|
||||||
|
|
||||||
|
### 验证
|
||||||
|
|
||||||
|
- `make test`: pass
|
||||||
|
|||||||
515
docs/moderation.md
Normal file
515
docs/moderation.md
Normal file
@ -0,0 +1,515 @@
|
|||||||
|
# 敏感词与发送处罚
|
||||||
|
|
||||||
|
> 对照：IM 文本发送机审（词库 + 分类处罚）
|
||||||
|
> 形态：模块化单体内独立业务 `moderation`，挂在现有 `message/logic.SendMessage` 上
|
||||||
|
> 状态：**v1 已落地**（发送闸 + 文本机审 + Admin 词库/命中 + 封禁踢线）。部署前执行 `migrations/0028_moderation.sql`、`0029_sensitive_words_seed.sql`。
|
||||||
|
> 硬约束：`docs/architecture-v3.md` C1–C7。不上 Kafka、K8s、微服务。表一律 `t_*`。
|
||||||
|
|
||||||
|
本文按 **2026-09-29 代码** 写成可实施说明，不是空想架构。HTTP 与 TCP 已共用 `internal/module/message/logic`；Gin / gnet 只绑参。`logic` 测试禁止连 MySQL。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 0. 范围
|
||||||
|
|
||||||
|
## 0.1 做（v1）
|
||||||
|
|
||||||
|
| 能力 | 说明 |
|
||||||
|
| --- | --- |
|
||||||
|
| 文本机审 | 仅 `content.t = 1` 的 `text.body`（单聊 `t_private_message` / 群聊 `t_group_message`） |
|
||||||
|
| 分类 | 涉黑 / 涉毒 / 涉黄 / 涉赌 / 涉政 / 宗教 / 辱骂 / 广告 / 其他 |
|
||||||
|
| 处罚 | 按词条 `action` 执行；一条消息多命中取 **最严** action |
|
||||||
|
| 账号闸 | 发送任意类型前检查 `t_users.status` 与 `send_muted_until` |
|
||||||
|
| 封号踢线 | `disable` 与 Admin 封禁：`status=0` + JWT `LogoutAll` + TCP `KickUser` |
|
||||||
|
| 词库运营 | Admin CRUD + 进程内自动机热替换 |
|
||||||
|
| 审计 | `t_sensitive_hits`（片段，不存全文） |
|
||||||
|
|
||||||
|
## 0.2 明确不做（v1）
|
||||||
|
|
||||||
|
- 图片 OCR、语音 ASR、视频抽帧、谐音/拼音/变形字模型
|
||||||
|
- 朋友圈 `moment`、广场 `square`、昵称、群公告、文件名、位置名称（同一 `Scan` 二期再挂）
|
||||||
|
- 独立审核微服务、Kafka、按词 `LIKE` 打发送热路径
|
||||||
|
- 把处罚做成群 `speak_muted` / `muted_all`（那是群维度，已由 `group/logic.CanSend` 处理）
|
||||||
|
- 用现有 Admin「白名单 / 异常标记」接口（二者目前是空实现，忽略写入）
|
||||||
|
- 给客户端回传命中原文（避免探测词库）
|
||||||
|
|
||||||
|
## 0.3 和现码的关系
|
||||||
|
|
||||||
|
| 现码 | 敏感词怎么接 |
|
||||||
|
| --- | --- |
|
||||||
|
| `SendMessage`：解 content → `prepareSend` → persist → `deliver:queue` → ACK | 在 persist **之前** 扫词；reject / mute / disable **不 INSERT、不 ACK**（C5） |
|
||||||
|
| HTTP `handler/message`、TCP `handleSendMessage` | 零改业务，只多注入依赖 |
|
||||||
|
| `FriendGate` / `GroupGate` | 并列增加 `AccountGate`、`Filter` |
|
||||||
|
| `users.status` 0/1；登录拒绝停用账号 | 发送侧补同一检查；停用 JWT 残留不能再发 |
|
||||||
|
| Admin `BanByEmail` 只改 status | 与 `disable` 共用踢线；封禁补 `LogoutAll`+`KickUser` |
|
||||||
|
| 注销 `revokeAccess` | 封号复用同一对接口 |
|
||||||
|
| `Message.StatusHidden=3`；历史 `status <> 3` | `hide`：落库 Hidden、**不** Push；会话同步已看不到 |
|
||||||
|
| 广场 `AuditPending/Approved` | **不要**和 IM Hidden 混用；广场二期自己打 Audit |
|
||||||
|
| `friend.Report` | 用户举人对人，与词库命中分流；hits 给运营 |
|
||||||
|
| `reqctx.RealIP` | 命中审计写 PROXY IP（C6） |
|
||||||
|
| `GET /v1/meta/enums` | 增加 category / action / envelope `40302` |
|
||||||
|
| 表前缀 | `t_sensitive_words`、`t_sensitive_hits`；用户列加在 `t_users` |
|
||||||
|
|
||||||
|
| V3 | 怎么守 |
|
||||||
|
| --- | --- |
|
||||||
|
| C1 | 聊天仍是 MsgId；词库行 id 自增；hits 用 Snowflake |
|
||||||
|
| C2 | 扫词只在 TaskPool 的 SEND 上；EventLoop 不碰词库 |
|
||||||
|
| C5 | 未 COMMIT+enqueue 不 ACK；reject 路径无行 |
|
||||||
|
| C6 | hits.real_ip 用 RealIP |
|
||||||
|
| C7 | 不写会话 ZSET；hide 不 fanout |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 1. 总览
|
||||||
|
|
||||||
|
```text
|
||||||
|
客户端 SEND_MESSAGE / POST 发消息
|
||||||
|
│ TaskPool（已有）
|
||||||
|
▼
|
||||||
|
message/logic.SendMessage
|
||||||
|
│
|
||||||
|
├─ AccountGate.CanSend(uid) 停用 / 全站禁言未到期 → 403
|
||||||
|
├─ prepareSend 好友 / 群禁言（已有）
|
||||||
|
├─ 若文本：Filter.Scan(body)
|
||||||
|
│ └─ Apply(hit)：处罚写库 → 改写或拒绝
|
||||||
|
├─ persistSend CreateAndBump
|
||||||
|
└─ DeliverQueue.Push hide 跳过
|
||||||
|
│
|
||||||
|
Admin ── moderation/logic ── t_sensitive_words
|
||||||
|
内存 AC 自动机（热替换）
|
||||||
|
```
|
||||||
|
|
||||||
|
匹配器无 I/O：`internal/pkg/sensitive`（stdlib）。词表从 MySQL 加载到进程；单进程单体改词后直接 Rebuild，不上 Redis PubSub。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 2. 模块与目录
|
||||||
|
|
||||||
|
```text
|
||||||
|
internal/pkg/sensitive/ Aho-Corasick + 文本规范化（无 DSN）
|
||||||
|
internal/module/moderation/
|
||||||
|
entity/ Category, Action, Word, Hit, ScanResult
|
||||||
|
model/ word.go, hit.go, matcher.go（持有自动机；TableName → t_*）
|
||||||
|
logic/ scan.go, apply.go, words.go, hits.go, reload.go, service.go
|
||||||
|
internal/transport/http/handler/admin/ 词库与命中（薄 MVC）
|
||||||
|
```
|
||||||
|
|
||||||
|
`message/logic.Service` 增加字段，**不** import `moderation/model`：
|
||||||
|
|
||||||
|
```go
|
||||||
|
type AccountGate interface {
|
||||||
|
AssertCanSend(ctx context.Context, userID int64) error
|
||||||
|
}
|
||||||
|
|
||||||
|
type TextFilter interface {
|
||||||
|
// Scan 无命中返回 (nil, nil)。
|
||||||
|
Scan(ctx context.Context, text string) (*moderationentity.ScanResult, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type Penalty interface {
|
||||||
|
// Apply 在拒绝发送前落处罚。失败返回 error（不 ACK）。
|
||||||
|
Apply(ctx context.Context, userID, conversationID int64, in *moderationentity.ApplyInput) error
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`bootstrap`：`moderation.NewService(...)` → 注入 `NewService(..., account, filter, penalty)`。`filter`/`penalty` 可为同一 `*moderation.Service`。测试用内存词表 stub。
|
||||||
|
|
||||||
|
`logic` 一文件一职能：`scan.go`、`apply.go`、`words_admin.go`、`hits.go`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 3. 发送时序（改 `send.go`）
|
||||||
|
|
||||||
|
与现有函数对齐，插入点固定：
|
||||||
|
|
||||||
|
```text
|
||||||
|
ready / validate / UnmarshalContent // 已有
|
||||||
|
AccountGate.AssertCanSend(userID) // 新增：所有消息类型
|
||||||
|
prepareSend // 已有：好友、群 CanSend
|
||||||
|
checkReplyTo / checkMentions // 已有
|
||||||
|
若 content.Type == TypeText && Text != nil:
|
||||||
|
res, err := Filter.Scan(ctx, Text.Body)
|
||||||
|
按 res.Action:
|
||||||
|
log → Apply 只记 hits，body 不变
|
||||||
|
replace → 改 Text.Body 为打码串，再 Marshal
|
||||||
|
reject → Apply 记 hits，return 40302
|
||||||
|
hide → Apply 记 hits；msg.Status = StatusHidden；Marshal 原文或打码（见 5.4）
|
||||||
|
mute → Apply 写 send_muted_until + hits，return 40302
|
||||||
|
disable → Apply 停用+踢线 + hits，return 40302
|
||||||
|
Marshal（replace 之后必须重新 Marshal）
|
||||||
|
stampTargets / assignID / persistSend // 已有
|
||||||
|
若 Status != Hidden：enqueueMessage // hide 不入队
|
||||||
|
SEND_ACK
|
||||||
|
```
|
||||||
|
|
||||||
|
**幂等：** `client_msg_id` 在 reject 时无行，客户端重试会再次 Scan。`replace` 成功后重试走现有 `persistSend` 的 Duplicate → 返回已落库行，不再二次打码。
|
||||||
|
|
||||||
|
**处罚写库失败：** 返回 `50001`，不 persist 消息（与 C5 一致：没有成功处罚就当发送失败）。
|
||||||
|
|
||||||
|
**多命中：** `Scan` 返回全部 hit，`Action = max(severity)`；hits 表一行一词或一行一消息带 JSON word_ids（v1 建议 **一行一词** 便于按类统计）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 4. 闭集 enum
|
||||||
|
|
||||||
|
数字稳定，不回收。entity `Valid()` + OpenAPI `$ref` + `GET /v1/meta/enums` + i18n `enum.sensitive_category.*` / `enum.sensitive_action.*`。
|
||||||
|
|
||||||
|
## 4.1 Category（词条分类，运营统计）
|
||||||
|
|
||||||
|
| 值 | name | 中文 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 1 | `crime` | 涉黑 |
|
||||||
|
| 2 | `drug` | 涉毒 |
|
||||||
|
| 3 | `porn` | 涉黄 |
|
||||||
|
| 4 | `gambling` | 涉赌 |
|
||||||
|
| 5 | `politics` | 涉政 |
|
||||||
|
| 6 | `religion` | 宗教 |
|
||||||
|
| 7 | `abuse` | 辱骂 |
|
||||||
|
| 8 | `ad` | 广告 |
|
||||||
|
| 9 | `other` | 其他 |
|
||||||
|
|
||||||
|
与 `friend.ReportReason`（1 垃圾 2 骚扰 3 假身份 4 其他）**不是同一套**，禁止复用。
|
||||||
|
|
||||||
|
## 4.2 Action（处罚，按严厉程度递增）
|
||||||
|
|
||||||
|
| 值 | name | 行为摘要 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 1 | `log` | 发送成功；只审计 |
|
||||||
|
| 2 | `replace` | 命中处替换为 `***` 后发送 |
|
||||||
|
| 3 | `reject` | 不落库；40302 |
|
||||||
|
| 4 | `hide` | 落库 `status=3`；不 deliver |
|
||||||
|
| 5 | `mute` | 全站禁言至 `now+mute_seconds`；不落库；40302 |
|
||||||
|
| 6 | `disable` | `status=0` + LogoutAll + KickUser；不落库；40302 |
|
||||||
|
|
||||||
|
比较用整数大小即可（1<…<6）。
|
||||||
|
|
||||||
|
## 4.3 类别默认 action（词条可覆盖）
|
||||||
|
|
||||||
|
实施时写入种子数据或 Admin 缺省，**不是**写死在 Scan 里：
|
||||||
|
|
||||||
|
| Category | 默认 Action | mute_seconds |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 涉政 / 涉黑 / 涉毒 | `disable` | — |
|
||||||
|
| 涉赌 / 宗教 / 涉黄 | `reject` | — |
|
||||||
|
| 辱骂 | `replace` | — |
|
||||||
|
| 广告 | `log` | — |
|
||||||
|
| 其他 | `reject` | — |
|
||||||
|
|
||||||
|
产品若改「涉政先禁言再人工」，只改默认/词条，不改发送链。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 5. 处罚细节（对齐现码）
|
||||||
|
|
||||||
|
## 5.1 AccountGate（所有消息类型）
|
||||||
|
|
||||||
|
`t_users.status != 1` → `40301` 或与登录一致的 `ErrForbidden`（停用账号）。
|
||||||
|
`send_muted_until != NULL && until > now` → `40303` `send_muted`（与内容违规分开，便于客户端展示「禁言至某时」）。无该列或已到期则放行。
|
||||||
|
|
||||||
|
发送图片时若仍在禁言期，同样拦截（禁言是账号能力，不是「只禁文本」）。
|
||||||
|
|
||||||
|
## 5.2 mute
|
||||||
|
|
||||||
|
`UPDATE t_users SET send_muted_until = GREATEST(IFNULL(send_muted_until,0), :until)` 避免短禁言覆盖长禁言。`mute_seconds` 来自 **胜出那条词**；多词 mute 取最长秒数。
|
||||||
|
|
||||||
|
不调用群 `speak_muted`。
|
||||||
|
|
||||||
|
## 5.3 disable
|
||||||
|
|
||||||
|
与注销踢线同序、不删用户行：
|
||||||
|
|
||||||
|
1. `status = 0`
|
||||||
|
2. `JWT.LogoutAll`
|
||||||
|
3. `Session.KickUser`
|
||||||
|
4. KICK 文案新 key `notify.kick.banned`（现有仅 replaced / logout / deleted）
|
||||||
|
|
||||||
|
Admin `BanByEmail` **改为同一套**，否则运营手点封禁与机审封号行为不一致。
|
||||||
|
|
||||||
|
## 5.4 hide
|
||||||
|
|
||||||
|
- `msg.Status = StatusHidden` 后 `CreateAndBump`（会话 last_msg 是否 bump：建议 **仍 bump**，否则对端游标异常；但 Push 跳过，对端靠同步也过滤 Hidden，last 预览可能短暂指向不可见消息——v1 接受；二期可 hide 不 bump）。
|
||||||
|
- **不** `queue.Push`。
|
||||||
|
- 发送方历史同样 `status <> 3`，自己也看不到。v1 不做成「仅自己可见」。
|
||||||
|
- body：默认存原文便于运营抽查 Admin IM；客户端不可见即达到目的。
|
||||||
|
|
||||||
|
## 5.5 replace
|
||||||
|
|
||||||
|
按 §6 把规范化命中映回原文 byte 区间，合并重叠后每段换成 `***`。映射断言失败则按 reject。重新 `content.Marshal()` 再入库。
|
||||||
|
|
||||||
|
## 5.6 log
|
||||||
|
|
||||||
|
正常 persist + enqueue；只写 hits。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 6. 匹配
|
||||||
|
|
||||||
|
发送热路径禁止「对每个词 `strings.Contains` / SQL `LIKE` / 用户正则」。词是字面量；一次线性扫描同时找全部词。匹配发生在 **规范化后的 rune 流** 上，再把命中区间 **映回原文**，供 `replace` 和 `excerpt`。
|
||||||
|
|
||||||
|
## 6.1 目标与不做
|
||||||
|
|
||||||
|
| 要挡住 | 例子 | v1 是否覆盖 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 连续子串 | `网络赌博` 含词 `赌博` | 是 |
|
||||||
|
| 夹空白 / 标点 | `赌 博`、`赌。博` | 是（规范化时丢掉分隔符） |
|
||||||
|
| 全角 / 兼容字 | `賭博`、`ＡＤ` | 是（逐 rune 折叠） |
|
||||||
|
| 大小写英文 | `Casino` | 是（ToLower） |
|
||||||
|
| 零宽插入 | `赌\u200b博` | 是（丢掉 Cf） |
|
||||||
|
| 英文词中词 | `ass` ⊂ `class` | **否**（无词边界；中文 IM 优先）。二期可加 `whole_word` |
|
||||||
|
| 谐音 / 拆字 / 火星文模型 | `杜博`、`氵+者` | **否** |
|
||||||
|
| 图片里的字 | — | **否** |
|
||||||
|
|
||||||
|
词条 **只存字面**，没有正则，避免 ReDoS。
|
||||||
|
|
||||||
|
## 6.2 两段式流水线
|
||||||
|
|
||||||
|
```text
|
||||||
|
text.body（UTF-8 原文，最多按现有消息体上限截断扫描）
|
||||||
|
│
|
||||||
|
▼ FoldAndIndex（词库入库与 Scan 同一函数）
|
||||||
|
norm runes + origFrom[] / origTo[]
|
||||||
|
│ origFrom[i] = 第 i 个保留 rune 在原文的 byte 起点
|
||||||
|
│ origTo[i] = 该 rune 的 byte 终点（开区间）
|
||||||
|
▼
|
||||||
|
Aho-Corasick.Find(norm) 输出 [n0, n1) 规范化下标 + word_id
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
映回原文 [origFrom[n0], origTo[n1-1])
|
||||||
|
多命中：审计保留全部；处罚取 max(action)
|
||||||
|
replace：原文区间合并后再打码
|
||||||
|
```
|
||||||
|
|
||||||
|
**禁止**先对整句做 NFKC 再匹配：兼容分解会改变长度（`fi`→`fi`），区间无法稳定映回，`replace` 只能整句 `***`。v1 用 **逐 rune 折叠 + 丢弃分隔**，映射一一对应。
|
||||||
|
|
||||||
|
## 6.3 FoldAndIndex（规范化）
|
||||||
|
|
||||||
|
对原文按 UTF-8 解成 rune，从左到右：
|
||||||
|
|
||||||
|
1. **丢掉**：Unicode 类别 `Cf`（含 U+200B/U+FEFF）、`Zs`、`Zl`、`Zp`；ASCII 空白；常见标点（ASCII ``!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~`` 以及 `，。！？、；：…·「」『』【】（）`）。
|
||||||
|
2. **保留则折叠**：`unicode.ToLower`；全角拉丁/数字收到半角（`Ａ`→`a`，可用 `golang.org/x/text/width` 的 `Narrow`，或一张小映射表）。不把一个 rune 拆成多个；若折叠结果多于一个 rune，只取第一个（极端兼容字，可测）。
|
||||||
|
3. 输出 `norm []rune` 与平行数组 `origFrom` / `origTo`。
|
||||||
|
|
||||||
|
**入库：** Admin 提交的 `word_raw` 走同一函数得到 `word`（规范化串）。`word` 为空（全是标点）→ 400 拒绝。`uk_word` 建在规范化串上，避免「赌博」与「赌 博」两条。
|
||||||
|
|
||||||
|
**扫描：** `text.body` 同样 Fold；`len(norm)==0` → 无命中。
|
||||||
|
|
||||||
|
**最短词：** 允许 1 个 rune，但种子词建议 ≥2。单字「黄」误伤由运营自己负责。
|
||||||
|
|
||||||
|
## 6.4 自动机
|
||||||
|
|
||||||
|
标准 **Aho-Corasick**：
|
||||||
|
|
||||||
|
- 模式 = 各 `enabled=1` 的 `word`（已是规范化串）。
|
||||||
|
- 输出链：某位置结束时，报告所有落在该节点上的词（含后缀，例如同时命中 `赌` 与 `赌博`）。
|
||||||
|
- 复杂度：建树 O(Σ|word|)；扫一句 O(|norm| + 命中次数)。千～万词、单条约 2KB 正文，CPU 可接受。
|
||||||
|
- 实现放 `internal/pkg/sensitive`，无 DSN。节点上挂 `[]patternID`（下标进内存词表：id/category/action/mute_seconds）。
|
||||||
|
|
||||||
|
`enabled=0` 不进树。Reload 时整树重建，`atomic.Pointer` 交换，Scan 无锁读。
|
||||||
|
|
||||||
|
## 6.5 命中如何变成处罚和打码
|
||||||
|
|
||||||
|
对每次 AC 命中 `(word_id, n0, n1)`：
|
||||||
|
|
||||||
|
- 原文区间：`[origFrom[n0], origTo[n1-1])`（byte）。
|
||||||
|
- `excerpt`：原文该区间，两端最多再扩 8 rune，总长截断 128（审计用，不回客户端）。
|
||||||
|
- `ScanResult.Hits` 保留全部；`ScanResult.Action = max(hits.action)`；mute 秒数取 **胜出 action 为 mute 的那些词里最长**。
|
||||||
|
|
||||||
|
**replace（仅当最严 action 就是 replace）：**
|
||||||
|
|
||||||
|
1. 收集所有命中的原文 byte 区间。
|
||||||
|
2. 按起点排序，合并重叠或相邻区间。
|
||||||
|
3. 每个合并区间替换成三个 ASCII `*`（固定宽度，不按原词长度拉长）。
|
||||||
|
4. 拼回新 `body`，再 `content.Marshal()`。
|
||||||
|
|
||||||
|
若最严已是 reject/mute/disable，不打码、不落库。
|
||||||
|
|
||||||
|
**映射失败：** Fold 保证 `n1-1 < len(origTo)`；实现上断言失败则该次 Scan 当 reject（保守），打日志。不要静默放行。
|
||||||
|
|
||||||
|
## 6.6 词边界（v1 明确放弃）
|
||||||
|
|
||||||
|
中文无空格，用「规范化后的连续子串」= 产品语义。英文 `scunthorpe` 类误伤不在 v1 修。运营用更长短语，或二期 `whole_word`：仅当命中两端在原文上为非字母。
|
||||||
|
|
||||||
|
## 6.7 扫描上限
|
||||||
|
|
||||||
|
与消息体校验同一上限（现有 content 长度闸）。Matcher 再设硬顶（例如 8192 rune），超长截断再扫——只为绑 CPU，截断本身不算绕过（超长消息应已被发送校验拒绝）。
|
||||||
|
|
||||||
|
## 6.8 热更新
|
||||||
|
|
||||||
|
Admin 写 `t_sensitive_words` 成功后 `Reload()`：`WHERE enabled=1` 取出全部 `word`，Fold 校验与库内一致（不一致则跳过该行并打错误日志），建新 AC，交换指针。单进程，不上 Redis。logic 测试用内存 `[]Word` 调 `Build`，不连 MySQL。
|
||||||
|
|
||||||
|
## 6.9 包职责
|
||||||
|
|
||||||
|
| 包 | 职责 |
|
||||||
|
| --- | --- |
|
||||||
|
| `pkg/sensitive` | `FoldAndIndex`、`Matcher.Find`、区间合并；纯函数 + 结构体 |
|
||||||
|
| `moderation/model` | 持有 `atomic` 自动机、从 MySQL 装词 |
|
||||||
|
| `moderation/logic` | Scan → entity.ScanResult；Apply 处罚 |
|
||||||
|
| `message/logic` | 调 Filter，不关心 AC |
|
||||||
|
|
||||||
|
## 6.10 匹配用例（实施时写成 `pkg/sensitive` 测试）
|
||||||
|
|
||||||
|
| 原文 | 词库 | 期望 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `今晚赌博` | `赌博` | 命中；replace 后含 `***` |
|
||||||
|
| `今晚赌 博啊` | `赌博` | 命中（空白已丢） |
|
||||||
|
| `賭博` | `赌博` | 命中（繁简若未折则 **不** 命中：v1 **不做简繁转换**，简繁当两条词） |
|
||||||
|
| `Casino` | `casino` | 命中 |
|
||||||
|
| `class` | `ass` | **命中**（v1 无词边界，文档接受） |
|
||||||
|
| `你好` | `赌博` | 无命中 |
|
||||||
|
| `赌\u200b博` | `赌博` | 命中 |
|
||||||
|
| 空串 / 纯标点 | 任意 | 无命中 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 7. 表
|
||||||
|
|
||||||
|
迁移新文件（当前最大约 `0027`，实施时接下一个序号）。`cmd/migrate` 全量重放。列 COMMENT 必填。GORM `TableName()` 返回 `t_*`。
|
||||||
|
|
||||||
|
## 7.1 `t_users` 增量
|
||||||
|
|
||||||
|
```sql
|
||||||
|
ALTER TABLE t_users
|
||||||
|
ADD COLUMN send_muted_until DATETIME(3) NULL
|
||||||
|
COMMENT '全站禁言截止时间，空或已过期表示可发消息';
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7.2 `t_sensitive_words`
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE t_sensitive_words (
|
||||||
|
id BIGINT NOT NULL AUTO_INCREMENT COMMENT '行 id，表内自增',
|
||||||
|
word_raw VARCHAR(64) NOT NULL COMMENT '运营录入原文，展示用',
|
||||||
|
word VARCHAR(64) NOT NULL COMMENT 'FoldAndIndex 后的词面，全局唯一，进自动机',
|
||||||
|
category TINYINT NOT NULL COMMENT '1涉黑 2涉毒 3涉黄 4涉赌 5涉政 6宗教 7辱骂 8广告 9其他',
|
||||||
|
action TINYINT NOT NULL COMMENT '1log 2replace 3reject 4hide 5mute 6disable',
|
||||||
|
mute_seconds INT NOT NULL DEFAULT 0 COMMENT 'action=mute 时的禁言秒数',
|
||||||
|
enabled TINYINT NOT NULL DEFAULT 1 COMMENT '1启用 0下线（不进自动机）',
|
||||||
|
created_at DATETIME(3) NOT NULL COMMENT '创建时间',
|
||||||
|
updated_at DATETIME(3) NOT NULL COMMENT '更新时间',
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
UNIQUE KEY uk_word (word)
|
||||||
|
) COMMENT='敏感词；发送热路径使用进程内自动机，本表为真源';
|
||||||
|
```
|
||||||
|
|
||||||
|
`word` 最长 64；更长的短语运营拆词。
|
||||||
|
|
||||||
|
## 7.3 `t_sensitive_hits`
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE t_sensitive_hits (
|
||||||
|
id BIGINT NOT NULL COMMENT 'Snowflake',
|
||||||
|
user_id BIGINT NOT NULL COMMENT '发送者 uid',
|
||||||
|
conversation_id BIGINT NOT NULL DEFAULT 0 COMMENT '会话 id，拒绝时已 prepareSend 则有值',
|
||||||
|
word_id BIGINT NOT NULL COMMENT 't_sensitive_words.id',
|
||||||
|
category TINYINT NOT NULL COMMENT '命中时的类别快照',
|
||||||
|
action TINYINT NOT NULL COMMENT '本行对应词的 action；消息级最严 action 另见同批 rows',
|
||||||
|
excerpt VARCHAR(128) NOT NULL DEFAULT '' COMMENT '命中片段，截断，禁止整篇正文',
|
||||||
|
real_ip VARCHAR(64) NOT NULL DEFAULT '' COMMENT 'PROXY RealIP',
|
||||||
|
created_at DATETIME(3) NOT NULL COMMENT '命中时间',
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
KEY idx_user_created (user_id, created_at),
|
||||||
|
KEY idx_created (created_at),
|
||||||
|
KEY idx_word (word_id)
|
||||||
|
) COMMENT='敏感词命中审计，供 Admin 查询';
|
||||||
|
```
|
||||||
|
|
||||||
|
hits 的 `id` 用 Snowflake（与 MsgId 同类，时间有序）。`word_id` 允许词已删除后仍留快照（不建 FK，避免运营删词失败）。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 8. 错误码与 i18n
|
||||||
|
|
||||||
|
| code | name | 何时 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `40301` | forbidden | 停用账号、好友/群权限（已有） |
|
||||||
|
| `40302` | content_blocked | reject / mute / disable（内容或因此触发的拒绝发送） |
|
||||||
|
| `40303` | send_muted | 仅「禁言未到期、尚未扫词」 |
|
||||||
|
|
||||||
|
`40302` 的 `msg`：`i18n` key `error.40302` / 业务 `moderation.blocked`：「消息包含违规内容」。
|
||||||
|
`data`：v1 空对象。不返回 word、不返回 category。
|
||||||
|
|
||||||
|
`GET /v1/meta/enums` 的 `envelope_code` 增加 40302、40303。
|
||||||
|
|
||||||
|
踢线：`notify.kick.banned`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 9. Admin HTTP
|
||||||
|
|
||||||
|
挂在现有 `/api/admin/spark` 与 `/api/admin/pop`（与 music 相同双前缀），JWT AdminAuth。Handler 只 bind + 调 `moderation/logic`。
|
||||||
|
|
||||||
|
| 方法 | 路径 | 作用 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| GET | `/sensitive/words` | 分页；query：`q`、`category`、`enabled` |
|
||||||
|
| POST | `/sensitive/words` | 创建；body：word、category、action、mute_seconds、enabled |
|
||||||
|
| PATCH | `/sensitive/words/:id` | 改分类/处罚/启用 |
|
||||||
|
| DELETE | `/sensitive/words/:id` | 物理删或 enabled=0（推荐软下线） |
|
||||||
|
| GET | `/sensitive/hits` | 分页；query：uid、category、since、until |
|
||||||
|
| POST | `/sensitive/reload` | 强制 Rebuild 自动机（写入接口已自动 Reload 时可省略） |
|
||||||
|
|
||||||
|
校验：`oneof` 与 enum 数字一致；`label`/`errmsg` i18n。
|
||||||
|
|
||||||
|
OpenAPI 写在 `openapi.yaml` 的 `admin-music` tag；路径前缀 `/api/admin/spark/sensitive*`（`pop` 别名）。
|
||||||
|
|
||||||
|
词库变更后必须 Reload；失败则接口返回 500，避免 DB 与自动机分叉。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 10. 客户端（IM）
|
||||||
|
|
||||||
|
- 发送失败认 `40302` / `40303`，展示服务器 `msg`。
|
||||||
|
- `replace` / `log` / `hide`：发送接口仍成功 ACK（hide 也 ACK，但自己同步看不到——产品需知）。若 hide 要让发送方感知失败，应改用 reject；**v1 hide 仅用于「落库留证、对端不可见」**，与 ACK 成功并存，文档需让客户端不要用 ACK 当「对端已收到」。
|
||||||
|
**更干净的 v1：砍掉 hide，运营留证用 hits + reject。** 推荐 **v1 不实现 hide**，避免 ACK 语义撕裂。需要留证时 reject 已有 excerpt。`StatusHidden` 留给以后人工审核。
|
||||||
|
|
||||||
|
修订：**v1 Action 不含 hide**（表 enum 仍占位 4，词条禁止配置或配置后当 reject）。二期人工审核再启用 hide。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 11. 测试
|
||||||
|
|
||||||
|
`make test`：logic ≥65%。
|
||||||
|
|
||||||
|
| 包 | 用例 |
|
||||||
|
| --- | --- |
|
||||||
|
| `pkg/sensitive` | 规范化、多词、最严 action、enabled=0 不匹配、空串 |
|
||||||
|
| `moderation/logic` | 内存词表 Scan；Apply mute/disable 调 fake user/session |
|
||||||
|
| `message/logic` | stub Filter：命中 reject 无 persist、无 queue；replace 后 content 含 `***`；AccountGate 失败不 persist |
|
||||||
|
| HTTP | 可选：发文本 40302 |
|
||||||
|
|
||||||
|
禁止 moderation logic 测试连真实 MySQL。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 12. 实施顺序
|
||||||
|
|
||||||
|
1. 迁移：`send_muted_until`、`t_sensitive_words`、`t_sensitive_hits`；跑 `cmd/migrate`
|
||||||
|
2. `pkg/sensitive` + entity enum + meta/i18n/OpenAPI
|
||||||
|
3. `moderation` model 内存自动机 + logic Scan/Reload
|
||||||
|
4. `AccountGate` 接入 `SendMessage`；Admin 封禁补踢线
|
||||||
|
5. 文本 Filter：log / replace / reject
|
||||||
|
6. mute / disable
|
||||||
|
7. Admin 词库与 hits
|
||||||
|
8. （二期）moment/square `Publish` 调同一 Scan；广场可打 `AuditPending`
|
||||||
|
9. （二期）Action=hide + 人工审核队列
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 13. 默认拍板（实施按此，除非产品改口）
|
||||||
|
|
||||||
|
1. 涉政 / 涉黑 / 涉毒默认 **disable**（踢线）。
|
||||||
|
2. 客户端 **不** 返回 category。
|
||||||
|
3. **v1 不做 hide**（enum 4 保留给二期）。
|
||||||
|
4. 禁言列名 `send_muted_until`。
|
||||||
|
5. 词面 UNIQUE 用规范化串。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 14. 参考代码锚点
|
||||||
|
|
||||||
|
- 发送：`internal/module/message/logic/send.go`、`send_perm.go`、`send_persist.go`
|
||||||
|
- 注入：`internal/bootstrap/app.go` `msglogic.NewService`
|
||||||
|
- 停用登录：`user/logic/login.go` `Status != StatusActive`
|
||||||
|
- 踢线：`user/logic/delete.go` `revokeAccess`
|
||||||
|
- Admin 封禁：`admin/logic/users.go` `BanByEmail`
|
||||||
|
- Hidden 过滤：`message/model/query.go`
|
||||||
|
- 群发言：`group/logic/members.go` `CanSend`
|
||||||
|
- 表名：`t_users`、`t_private_message`、`t_group_message`
|
||||||
@ -41,6 +41,8 @@ import (
|
|||||||
inboxmodel "spark-server/internal/module/inbox/model"
|
inboxmodel "spark-server/internal/module/inbox/model"
|
||||||
msglogic "spark-server/internal/module/message/logic"
|
msglogic "spark-server/internal/module/message/logic"
|
||||||
msgmodel "spark-server/internal/module/message/model"
|
msgmodel "spark-server/internal/module/message/model"
|
||||||
|
modlogic "spark-server/internal/module/moderation/logic"
|
||||||
|
modmodel "spark-server/internal/module/moderation/model"
|
||||||
momentlogic "spark-server/internal/module/moment/logic"
|
momentlogic "spark-server/internal/module/moment/logic"
|
||||||
momentmodel "spark-server/internal/module/moment/model"
|
momentmodel "spark-server/internal/module/moment/model"
|
||||||
musiclogic "spark-server/internal/module/music/logic"
|
musiclogic "spark-server/internal/module/music/logic"
|
||||||
@ -205,7 +207,8 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge
|
|||||||
chat := wireChat(idgen, db)
|
chat := wireChat(idgen, db)
|
||||||
friends := wireFriends(idgen, db, users, deliverQueue, chat)
|
friends := wireFriends(idgen, db, users, deliverQueue, chat)
|
||||||
groups := wireGroups(idgen, db, users, friends, deliverQueue, objects, chat)
|
groups := wireGroups(idgen, db, users, friends, deliverQueue, objects, chat)
|
||||||
messages := wireMessages(idgen, friends, groups, deliverQueue, chat)
|
moderation := wireModeration(db, users, idgen)
|
||||||
|
messages := wireMessages(idgen, friends, groups, deliverQueue, chat, moderation, users)
|
||||||
files := wireFiles(cfg, log, objects, groups)
|
files := wireFiles(cfg, log, objects, groups)
|
||||||
favorites := wireFavorites(idgen, db, objects, chat)
|
favorites := wireFavorites(idgen, db, objects, chat)
|
||||||
moments := wireMoments(idgen, db, objects, friends, users)
|
moments := wireMoments(idgen, db, objects, friends, users)
|
||||||
@ -214,7 +217,7 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge
|
|||||||
app.squareBot = wireSquareBot(cfg, log, rdb, objects, square, users)
|
app.squareBot = wireSquareBot(cfg, log, rdb, objects, square, users)
|
||||||
music := wireMusic(cfg, db, objects)
|
music := wireMusic(cfg, db, objects)
|
||||||
users.BindAccountCleanup(jwtSvc, sessions, accountCleaners(friends, moments, square, groups, favorites, inbox, mailbox, chat)...)
|
users.BindAccountCleanup(jwtSvc, sessions, accountCleaners(friends, moments, square, groups, favorites, inbox, mailbox, chat)...)
|
||||||
adminSvc, adminTokens := wireAdmin(cfg, db, groups, idgen)
|
adminSvc, adminTokens := wireAdmin(cfg, db, groups, idgen, users)
|
||||||
app.tcp = tcp.NewServer(log, pool, sessions, &tcp.Options{
|
app.tcp = tcp.NewServer(log, pool, sessions, &tcp.Options{
|
||||||
Addr: cfg.Server.TCP.Addr,
|
Addr: cfg.Server.TCP.Addr,
|
||||||
Multicore: cfg.Server.TCP.Multicore,
|
Multicore: cfg.Server.TCP.Multicore,
|
||||||
@ -235,7 +238,7 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge
|
|||||||
Sync: syncSvc,
|
Sync: syncSvc,
|
||||||
})
|
})
|
||||||
app.http = httpx.New(log, cfg.Server.HTTP.Addr, httpEngineDeps(
|
app.http = httpx.New(log, cfg.Server.HTTP.Addr, httpEngineDeps(
|
||||||
jwtSvc, login, smsSvc, users, friends, groups, files, favorites, moments, square, music, client, messages, syncSvc, adminSvc, adminTokens, probe, authLimit,
|
jwtSvc, login, smsSvc, users, friends, groups, files, favorites, moments, square, music, client, messages, syncSvc, adminSvc, adminTokens, moderation, probe, authLimit,
|
||||||
))
|
))
|
||||||
return app, nil
|
return app, nil
|
||||||
}
|
}
|
||||||
@ -257,24 +260,43 @@ func httpEngineDeps(
|
|||||||
syncSvc *synclogic.Service,
|
syncSvc *synclogic.Service,
|
||||||
adminSvc *adminlogic.Service,
|
adminSvc *adminlogic.Service,
|
||||||
adminTokens *adminlogic.TokenService,
|
adminTokens *adminlogic.TokenService,
|
||||||
|
moderation *modlogic.Service,
|
||||||
probe *Probe,
|
probe *Probe,
|
||||||
authLimit *ratelimit.Limiter,
|
authLimit *ratelimit.Limiter,
|
||||||
) *httpx.EngineDeps {
|
) *httpx.EngineDeps {
|
||||||
return &httpx.EngineDeps{
|
return &httpx.EngineDeps{
|
||||||
Auth: jwtSvc, Login: login, SMS: sms, Users: users, Friends: friends, Groups: groups,
|
Auth: jwtSvc, Login: login, SMS: sms, Users: users, Friends: friends, Groups: groups,
|
||||||
Files: files, Favorites: favorites, Moments: moments, Square: square, Music: music, Client: client,
|
Files: files, Favorites: favorites, Moments: moments, Square: square, Music: music, Client: client,
|
||||||
Messages: messages, Sync: syncSvc, Admin: adminSvc, AdminTokens: adminTokens,
|
Messages: messages, Sync: syncSvc, Admin: adminSvc, AdminTokens: adminTokens, Moderation: moderation,
|
||||||
Health: probe, AuthLimiter: authLimit,
|
Health: probe, AuthLimiter: authLimit,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func wireAdmin(cfg *Config, db *mysql.DB, groups *grouplogic.Service, idgen *id.Generator) (*adminlogic.Service, *adminlogic.TokenService) {
|
func wireAdmin(cfg *Config, db *mysql.DB, groups *grouplogic.Service, idgen *id.Generator, users *userlogic.Service) (*adminlogic.Service, *adminlogic.TokenService) {
|
||||||
tokens := adminlogic.NewTokenService(cfg.Admin.JWTSecret)
|
tokens := adminlogic.NewTokenService(cfg.Admin.JWTSecret)
|
||||||
var repo *adminmodel.Repo
|
var repo *adminmodel.Repo
|
||||||
if db != nil && db.Ready() {
|
if db != nil && db.Ready() {
|
||||||
repo = adminmodel.NewRepo(db.Gorm())
|
repo = adminmodel.NewRepo(db.Gorm())
|
||||||
}
|
}
|
||||||
return adminlogic.NewService(repo, groups, idgen), tokens
|
svc := adminlogic.NewService(repo, groups, idgen)
|
||||||
|
if users != nil {
|
||||||
|
svc.BindUserDisabler(users)
|
||||||
|
}
|
||||||
|
return svc, tokens
|
||||||
|
}
|
||||||
|
|
||||||
|
func wireModeration(db *mysql.DB, users *userlogic.Service, idgen *id.Generator) *modlogic.Service {
|
||||||
|
if db == nil || !db.Ready() || idgen == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
repo := modmodel.NewRepo(db.Gorm())
|
||||||
|
if repo == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
bridge := modmodel.NewLogicBridge(repo)
|
||||||
|
svc := modlogic.NewService(bridge, users, idgen)
|
||||||
|
_ = svc.Reload(context.Background())
|
||||||
|
return svc
|
||||||
}
|
}
|
||||||
|
|
||||||
func loadTLS(t TLSConfig) (*tls.Config, error) {
|
func loadTLS(t TLSConfig) (*tls.Config, error) {
|
||||||
@ -490,11 +512,23 @@ func wireDeliver(cfg *Config, log *zap.Logger, rdb *redisx.Client, sessions sess
|
|||||||
}), q
|
}), q
|
||||||
}
|
}
|
||||||
|
|
||||||
func wireMessages(idgen *id.Generator, friends *friendlogic.Service, groups *grouplogic.Service, q msglogic.DeliverQueue, chat chatStores) *msglogic.Service {
|
func wireMessages(
|
||||||
|
idgen *id.Generator,
|
||||||
|
friends *friendlogic.Service,
|
||||||
|
groups *grouplogic.Service,
|
||||||
|
q msglogic.DeliverQueue,
|
||||||
|
chat chatStores,
|
||||||
|
moderation *modlogic.Service,
|
||||||
|
users *userlogic.Service,
|
||||||
|
) *msglogic.Service {
|
||||||
if idgen == nil {
|
if idgen == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return msglogic.NewService(chat.msgs, chat.chats, chat.p2p, friends, groups, idgen, q)
|
svc := msglogic.NewService(chat.msgs, chat.chats, chat.p2p, friends, groups, idgen, q)
|
||||||
|
if moderation != nil && users != nil {
|
||||||
|
svc.BindModeration(users, moderation, moderation)
|
||||||
|
}
|
||||||
|
return svc
|
||||||
}
|
}
|
||||||
|
|
||||||
func accountCleaners(
|
func accountCleaners(
|
||||||
|
|||||||
@ -15,10 +15,15 @@ import (
|
|||||||
var ErrNotReady = errors.New("admin: repository not ready")
|
var ErrNotReady = errors.New("admin: repository not ready")
|
||||||
|
|
||||||
// Service orchestrates admin use-cases. SQL lives in model.Repo.
|
// Service orchestrates admin use-cases. SQL lives in model.Repo.
|
||||||
|
type UserDisabler interface {
|
||||||
|
DisableAccount(ctx context.Context, userID int64) error
|
||||||
|
}
|
||||||
|
|
||||||
type Service struct {
|
type Service struct {
|
||||||
repo Repository
|
repo Repository
|
||||||
groups *grouplogic.Service
|
groups *grouplogic.Service
|
||||||
ids IDGen
|
ids IDGen
|
||||||
|
disabler UserDisabler
|
||||||
}
|
}
|
||||||
|
|
||||||
type IDGen interface {
|
type IDGen interface {
|
||||||
@ -54,6 +59,13 @@ func NewService(repo Repository, groups *grouplogic.Service, ids IDGen) *Service
|
|||||||
return &Service{repo: repo, groups: groups, ids: ids}
|
return &Service{repo: repo, groups: groups, ids: ids}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (s *Service) BindUserDisabler(d UserDisabler) {
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.disabler = d
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Service) ready() bool {
|
func (s *Service) ready() bool {
|
||||||
return s != nil && s.repo != nil
|
return s != nil && s.repo != nil
|
||||||
}
|
}
|
||||||
|
|||||||
@ -118,11 +118,25 @@ func (s *Service) BanByEmail(ctx context.Context, email string, banned bool) err
|
|||||||
if !s.ready() {
|
if !s.ready() {
|
||||||
return ErrNotReady
|
return ErrNotReady
|
||||||
}
|
}
|
||||||
status := userentity.StatusActive
|
if !banned {
|
||||||
if banned {
|
return s.repo.SetUserStatusByEmail(ctx, email, userentity.StatusActive)
|
||||||
status = userentity.StatusDisabled
|
|
||||||
}
|
}
|
||||||
return s.repo.SetUserStatusByEmail(ctx, email, status)
|
rows, err := s.repo.UsersByEmail(ctx, email)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(rows) == 0 {
|
||||||
|
return errs.ErrNotFound
|
||||||
|
}
|
||||||
|
if s.disabler == nil {
|
||||||
|
return s.repo.SetUserStatusByEmail(ctx, email, userentity.StatusDisabled)
|
||||||
|
}
|
||||||
|
for i := range rows {
|
||||||
|
if err := s.disabler.DisableAccount(ctx, rows[i].ID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Service) ResetNicknameByEmail(ctx context.Context, email, nickname string) error {
|
func (s *Service) ResetNicknameByEmail(ctx context.Context, email, nickname string) error {
|
||||||
|
|||||||
23
internal/module/message/logic/gates_moderation.go
Normal file
23
internal/module/message/logic/gates_moderation.go
Normal file
@ -0,0 +1,23 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
// AccountGate checks account status before send.
|
||||||
|
type AccountGate interface {
|
||||||
|
AssertCanSend(ctx context.Context, userID int64) error
|
||||||
|
}
|
||||||
|
|
||||||
|
// TextFilter scans outbound text. No hit returns (nil, nil).
|
||||||
|
type TextFilter interface {
|
||||||
|
Scan(ctx context.Context, text string) (*modentity.ScanResult, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Penalty records hits and enforces mute/disable/reject.
|
||||||
|
type Penalty interface {
|
||||||
|
Apply(ctx context.Context, userID, conversationID int64, in *modentity.ApplyInput) error
|
||||||
|
Enforce(ctx context.Context, userID int64, res *modentity.ScanResult) error
|
||||||
|
}
|
||||||
@ -43,6 +43,9 @@ func (s *Service) SendMessage(ctx context.Context, userID int64, deviceID uint64
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, errs.ErrInvalidArgument
|
return nil, errs.ErrInvalidArgument
|
||||||
}
|
}
|
||||||
|
if err := s.gateAccount(ctx, userID); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
conv, memberIDs, err := s.prepareSend(ctx, userID, in)
|
conv, memberIDs, err := s.prepareSend(ctx, userID, in)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@ -53,6 +56,9 @@ func (s *Service) SendMessage(ctx context.Context, userID int64, deviceID uint64
|
|||||||
if err := s.checkMentions(ctx, userID, conv, content); err != nil {
|
if err := s.checkMentions(ctx, userID, conv, content); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
if err := s.moderateText(ctx, userID, conv.ID, content); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
raw, err := content.Marshal()
|
raw, err := content.Marshal()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, errs.ErrInvalidArgument
|
return nil, errs.ErrInvalidArgument
|
||||||
|
|||||||
41
internal/module/message/logic/send_moderation.go
Normal file
41
internal/module/message/logic/send_moderation.go
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"spark-server/internal/module/message/entity"
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (s *Service) gateAccount(ctx context.Context, userID int64) error {
|
||||||
|
if s.account == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return s.account.AssertCanSend(ctx, userID)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) moderateText(ctx context.Context, userID, conversationID int64, content *entity.Content) error {
|
||||||
|
if content == nil || content.Type != entity.TypeText || content.Text == nil || s.filter == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
body := content.Text.Body
|
||||||
|
res, err := s.filter.Scan(ctx, body)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if res == nil || len(res.Hits) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if s.penalty != nil {
|
||||||
|
if err := s.penalty.Apply(ctx, userID, conversationID, &modentity.ApplyInput{Hits: res.Hits}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := s.penalty.Enforce(ctx, userID, res); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if res.Action == modentity.ActionReplace && res.ReplacedBody != "" {
|
||||||
|
content.Text.Body = res.ReplacedBody
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
39
internal/module/message/logic/send_moderation_test.go
Normal file
39
internal/module/message/logic/send_moderation_test.go
Normal file
@ -0,0 +1,39 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
friendlogic "spark-server/internal/module/friend/logic"
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
modlogic "spark-server/internal/module/moderation/logic"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
"spark-server/internal/pkg/sensitive"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSendMessageContentBlocked(t *testing.T) {
|
||||||
|
svc, users, friends, _ := testMessageSvc(t)
|
||||||
|
word := sensitive.FoldWord("赌博")
|
||||||
|
repo := modlogic.NewMemoryRepo(&modentity.Word{
|
||||||
|
ID: 1, WordRaw: "赌博", Word: word, Category: modentity.CategoryGambling,
|
||||||
|
Action: modentity.ActionReject, Enabled: true,
|
||||||
|
})
|
||||||
|
mod := modlogic.NewService(repo, users, svc.ids)
|
||||||
|
svc.BindModeration(users, mod, mod)
|
||||||
|
ctx := context.Background()
|
||||||
|
a, b := twoUsers(t, users)
|
||||||
|
applied, err := friends.Apply(ctx, a.ID, &friendlogic.ApplyInput{UID: b.ID, Reason: "hi"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := friends.Accept(ctx, b.ID, friendlogic.AcceptInput{RequestID: applied.RequestID}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
text := json.RawMessage(`{"t":1,"text":{"body":"今晚赌博"}}`)
|
||||||
|
_, err = svc.SendMessage(ctx, a.ID, 1, &SendInput{PeerID: b.ID, ClientMsgID: "bad", Content: text})
|
||||||
|
if !errors.Is(err, errs.ErrContentBlocked) {
|
||||||
|
t.Fatalf("expected blocked got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -64,12 +64,25 @@ type Service struct {
|
|||||||
groups GroupGate
|
groups GroupGate
|
||||||
ids IDGen
|
ids IDGen
|
||||||
queue DeliverQueue
|
queue DeliverQueue
|
||||||
|
account AccountGate
|
||||||
|
filter TextFilter
|
||||||
|
penalty Penalty
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewService(repo Repository, chats Conversations, p2p P2POpener, friends FriendGate, groups GroupGate, ids IDGen, queue DeliverQueue) *Service {
|
func NewService(repo Repository, chats Conversations, p2p P2POpener, friends FriendGate, groups GroupGate, ids IDGen, queue DeliverQueue) *Service {
|
||||||
return &Service{repo: repo, chats: chats, p2p: p2p, friends: friends, groups: groups, ids: ids, queue: queue}
|
return &Service{repo: repo, chats: chats, p2p: p2p, friends: friends, groups: groups, ids: ids, queue: queue}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// BindModeration wires account gate and text filter (optional; nil skips).
|
||||||
|
func (s *Service) BindModeration(account AccountGate, filter TextFilter, penalty Penalty) {
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.account = account
|
||||||
|
s.filter = filter
|
||||||
|
s.penalty = penalty
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Service) ready() bool {
|
func (s *Service) ready() bool {
|
||||||
return s != nil && s.repo != nil && s.chats != nil && s.ids != nil
|
return s != nil && s.repo != nil && s.chats != nil && s.ids != nil
|
||||||
}
|
}
|
||||||
|
|||||||
@ -43,6 +43,8 @@ func envelopeCode(lang i18n.Lang) []entity.Item {
|
|||||||
code(lang, "not_authenticated", errs.ErrNotAuthenticated),
|
code(lang, "not_authenticated", errs.ErrNotAuthenticated),
|
||||||
code(lang, "verify_code_invalid", errs.ErrVerifyCodeInvalid),
|
code(lang, "verify_code_invalid", errs.ErrVerifyCodeInvalid),
|
||||||
code(lang, "forbidden", errs.ErrForbidden),
|
code(lang, "forbidden", errs.ErrForbidden),
|
||||||
|
code(lang, "content_blocked", errs.ErrContentBlocked),
|
||||||
|
code(lang, "send_muted", errs.ErrSendMuted),
|
||||||
code(lang, "not_found", errs.ErrNotFound),
|
code(lang, "not_found", errs.ErrNotFound),
|
||||||
code(lang, "media_expired", errs.ErrMediaExpired),
|
code(lang, "media_expired", errs.ErrMediaExpired),
|
||||||
code(lang, "duplicate", errs.ErrDuplicate),
|
code(lang, "duplicate", errs.ErrDuplicate),
|
||||||
|
|||||||
39
internal/module/moderation/entity/action.go
Normal file
39
internal/module/moderation/entity/action.go
Normal file
@ -0,0 +1,39 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type Action int8
|
||||||
|
|
||||||
|
const (
|
||||||
|
ActionLog Action = 1
|
||||||
|
ActionReplace Action = 2
|
||||||
|
ActionReject Action = 3
|
||||||
|
ActionHide Action = 4
|
||||||
|
ActionMute Action = 5
|
||||||
|
ActionDisable Action = 6
|
||||||
|
)
|
||||||
|
|
||||||
|
func (a Action) Valid() bool {
|
||||||
|
return a >= ActionLog && a <= ActionDisable
|
||||||
|
}
|
||||||
|
|
||||||
|
// Effective maps v1 unsupported actions (hide) to reject.
|
||||||
|
func (a Action) Effective() Action {
|
||||||
|
if a == ActionHide {
|
||||||
|
return ActionReject
|
||||||
|
}
|
||||||
|
return a
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a Action) Severity() int {
|
||||||
|
return int(a.Effective())
|
||||||
|
}
|
||||||
|
|
||||||
|
func MaxAction(actions ...Action) Action {
|
||||||
|
var best Action
|
||||||
|
for _, a := range actions {
|
||||||
|
e := a.Effective()
|
||||||
|
if e.Severity() > best.Severity() {
|
||||||
|
best = e
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return best
|
||||||
|
}
|
||||||
19
internal/module/moderation/entity/category.go
Normal file
19
internal/module/moderation/entity/category.go
Normal file
@ -0,0 +1,19 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type Category int8
|
||||||
|
|
||||||
|
const (
|
||||||
|
CategoryCrime Category = 1
|
||||||
|
CategoryDrug Category = 2
|
||||||
|
CategoryPorn Category = 3
|
||||||
|
CategoryGambling Category = 4
|
||||||
|
CategoryPolitics Category = 5
|
||||||
|
CategoryReligion Category = 6
|
||||||
|
CategoryAbuse Category = 7
|
||||||
|
CategoryAd Category = 8
|
||||||
|
CategoryOther Category = 9
|
||||||
|
)
|
||||||
|
|
||||||
|
func (c Category) Valid() bool {
|
||||||
|
return c >= CategoryCrime && c <= CategoryOther
|
||||||
|
}
|
||||||
21
internal/module/moderation/entity/scan.go
Normal file
21
internal/module/moderation/entity/scan.go
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
type HitMatch struct {
|
||||||
|
WordID int64
|
||||||
|
Category Category
|
||||||
|
Action Action
|
||||||
|
Excerpt string
|
||||||
|
ByteFrom int
|
||||||
|
ByteTo int
|
||||||
|
}
|
||||||
|
|
||||||
|
type ScanResult struct {
|
||||||
|
Action Action
|
||||||
|
MuteSeconds int
|
||||||
|
Hits []HitMatch
|
||||||
|
ReplacedBody string
|
||||||
|
}
|
||||||
|
|
||||||
|
type ApplyInput struct {
|
||||||
|
Hits []HitMatch
|
||||||
|
}
|
||||||
15
internal/module/moderation/entity/word.go
Normal file
15
internal/module/moderation/entity/word.go
Normal file
@ -0,0 +1,15 @@
|
|||||||
|
package entity
|
||||||
|
|
||||||
|
import "time"
|
||||||
|
|
||||||
|
type Word struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
WordRaw string `json:"word_raw"`
|
||||||
|
Word string `json:"word"`
|
||||||
|
Category Category `json:"category"`
|
||||||
|
Action Action `json:"action"`
|
||||||
|
MuteSeconds int `json:"mute_seconds"`
|
||||||
|
Enabled bool `json:"enabled"`
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
UpdatedAt time.Time `json:"updated_at"`
|
||||||
|
}
|
||||||
73
internal/module/moderation/logic/apply.go
Normal file
73
internal/module/moderation/logic/apply.go
Normal file
@ -0,0 +1,73 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
"spark-server/internal/pkg/reqctx"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Apply records hits and runs account penalties. Call before rejecting send.
|
||||||
|
func (s *Service) Apply(ctx context.Context, userID, conversationID int64, in *modentity.ApplyInput) error {
|
||||||
|
if in == nil || len(in.Hits) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if !s.Ready() {
|
||||||
|
return errs.ErrInner
|
||||||
|
}
|
||||||
|
return s.recordHits(ctx, userID, conversationID, in.Hits)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Enforce runs mute/disable after hits are recorded. Returns ErrContentBlocked when send must stop.
|
||||||
|
func (s *Service) Enforce(ctx context.Context, userID int64, res *modentity.ScanResult) error {
|
||||||
|
if res == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
switch res.Action {
|
||||||
|
case modentity.ActionReject, modentity.ActionHide:
|
||||||
|
return errs.ErrContentBlocked
|
||||||
|
case modentity.ActionMute:
|
||||||
|
if s.users == nil || res.MuteSeconds <= 0 {
|
||||||
|
return errs.ErrContentBlocked
|
||||||
|
}
|
||||||
|
until := time.Now().Add(time.Duration(res.MuteSeconds) * time.Second)
|
||||||
|
if err := s.users.ExtendSendMutedUntil(ctx, userID, until); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return errs.ErrContentBlocked
|
||||||
|
case modentity.ActionDisable:
|
||||||
|
if s.users == nil {
|
||||||
|
return errs.ErrContentBlocked
|
||||||
|
}
|
||||||
|
if err := s.users.DisableAccount(ctx, userID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return errs.ErrContentBlocked
|
||||||
|
default:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) recordHits(ctx context.Context, userID, conversationID int64, hits []modentity.HitMatch) error {
|
||||||
|
ip := ""
|
||||||
|
if rc := reqctx.From(ctx); rc != nil {
|
||||||
|
ip = rc.RealIP
|
||||||
|
}
|
||||||
|
now := time.Now()
|
||||||
|
rows := make([]HitRecord, 0, len(hits))
|
||||||
|
for i := range hits {
|
||||||
|
h := hits[i]
|
||||||
|
id, err := s.ids.NextID()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
rows = append(rows, HitRecord{
|
||||||
|
ID: id, UserID: userID, ConversationID: conversationID,
|
||||||
|
WordID: h.WordID, Category: h.Category, Action: h.Action.Effective(),
|
||||||
|
Excerpt: h.Excerpt, RealIP: ip, CreatedAt: now,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return s.repo.InsertHits(ctx, rows)
|
||||||
|
}
|
||||||
13
internal/module/moderation/logic/hits.go
Normal file
13
internal/module/moderation/logic/hits.go
Normal file
@ -0,0 +1,13 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (s *Service) ListHits(ctx context.Context, userID int64, category int, since, until *time.Time, page, size int) ([]HitRecord, int64, error) {
|
||||||
|
if !s.Ready() {
|
||||||
|
return nil, 0, ErrNotReady
|
||||||
|
}
|
||||||
|
return s.repo.ListHits(ctx, userID, category, since, until, page, size)
|
||||||
|
}
|
||||||
168
internal/module/moderation/logic/memory_repo.go
Normal file
168
internal/module/moderation/logic/memory_repo.go
Normal file
@ -0,0 +1,168 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
"spark-server/internal/pkg/sensitive"
|
||||||
|
)
|
||||||
|
|
||||||
|
// MemoryRepo is an in-process word list for logic tests (no MySQL).
|
||||||
|
type MemoryRepo struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
words map[int64]*modentity.Word
|
||||||
|
nextID int64
|
||||||
|
m *sensitive.Matcher
|
||||||
|
hits []HitRecord
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewMemoryRepo(words ...*modentity.Word) *MemoryRepo {
|
||||||
|
r := &MemoryRepo{words: map[int64]*modentity.Word{}, nextID: 1}
|
||||||
|
for _, w := range words {
|
||||||
|
if w == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if w.ID == 0 {
|
||||||
|
w.ID = r.nextID
|
||||||
|
r.nextID++
|
||||||
|
}
|
||||||
|
cp := *w
|
||||||
|
r.words[cp.ID] = &cp
|
||||||
|
}
|
||||||
|
r.rebuild()
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) Ready() bool { return r != nil }
|
||||||
|
|
||||||
|
func (r *MemoryRepo) ReloadMatcher(ctx context.Context) error {
|
||||||
|
_ = ctx
|
||||||
|
r.rebuild()
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) rebuild() {
|
||||||
|
entries := make([]sensitive.Entry, 0, len(r.words))
|
||||||
|
for _, w := range r.words {
|
||||||
|
if w == nil || !w.Enabled || w.Word == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
entries = append(entries, sensitive.Entry{ID: w.ID, Word: w.Word})
|
||||||
|
}
|
||||||
|
r.m = sensitive.Build(entries)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) ListWords(ctx context.Context, _ string, _ int, _ *bool, _ int, _ int) ([]modentity.Word, int64, error) {
|
||||||
|
_ = ctx
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
var all []modentity.Word
|
||||||
|
for _, w := range r.words {
|
||||||
|
if w == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
all = append(all, *w)
|
||||||
|
}
|
||||||
|
return all, int64(len(all)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) CreateWord(ctx context.Context, w *modentity.Word) error {
|
||||||
|
_ = ctx
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
if w.ID == 0 {
|
||||||
|
w.ID = r.nextID
|
||||||
|
r.nextID++
|
||||||
|
}
|
||||||
|
cp := *w
|
||||||
|
r.words[cp.ID] = &cp
|
||||||
|
r.rebuild()
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) UpdateWord(ctx context.Context, id int64, patch WordPatch) error {
|
||||||
|
_ = ctx
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
w := r.words[id]
|
||||||
|
if w == nil {
|
||||||
|
return ErrNotReady
|
||||||
|
}
|
||||||
|
cp := *w
|
||||||
|
if patch != nil {
|
||||||
|
if err := patch(&cp); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
r.words[id] = &cp
|
||||||
|
r.rebuild()
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) SoftDeleteWord(ctx context.Context, id int64) error {
|
||||||
|
return r.UpdateWord(ctx, id, func(w *modentity.Word) error {
|
||||||
|
w.Enabled = false
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) InsertHits(ctx context.Context, hits []HitRecord) error {
|
||||||
|
_ = ctx
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
r.hits = append(r.hits, hits...)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) ListHits(ctx context.Context, _ int64, _ int, _, _ *time.Time, _ int, _ int) ([]HitRecord, int64, error) {
|
||||||
|
_ = ctx
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
return r.hits, int64(len(r.hits)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) LookupWord(id int64) *modentity.Word {
|
||||||
|
r.mu.Lock()
|
||||||
|
defer r.mu.Unlock()
|
||||||
|
w := r.words[id]
|
||||||
|
if w == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
cp := *w
|
||||||
|
return &cp
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *MemoryRepo) MatcherFind(text string) ([]MatchDetail, string, bool) {
|
||||||
|
idx := sensitive.FoldAndIndex(text)
|
||||||
|
if r.m == nil || len(idx.Norm) == 0 {
|
||||||
|
return nil, text, true
|
||||||
|
}
|
||||||
|
raw := r.m.FindAll(idx.Norm)
|
||||||
|
if len(raw) == 0 {
|
||||||
|
return nil, text, true
|
||||||
|
}
|
||||||
|
intervals, ok := sensitive.IntervalsFromMatches(&idx, raw)
|
||||||
|
masked := text
|
||||||
|
if ok {
|
||||||
|
masked = sensitive.Mask(idx.OrigBytes, intervals, "***")
|
||||||
|
}
|
||||||
|
details := make([]MatchDetail, 0, len(raw))
|
||||||
|
for _, rm := range raw {
|
||||||
|
w := r.LookupWord(rm.EntryID)
|
||||||
|
if w == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
from, to := 0, 0
|
||||||
|
if ok && rm.Start >= 0 && rm.End <= len(idx.OrigFrom) {
|
||||||
|
from = idx.OrigFrom[rm.Start]
|
||||||
|
to = idx.OrigTo[rm.End-1]
|
||||||
|
}
|
||||||
|
details = append(details, MatchDetail{
|
||||||
|
WordID: w.ID, Category: w.Category, Action: w.Action,
|
||||||
|
Excerpt: sensitive.Excerpt(idx.OrigBytes, from, to, 128), From: from, To: to,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return details, masked, ok
|
||||||
|
}
|
||||||
120
internal/module/moderation/logic/moderation_test.go
Normal file
120
internal/module/moderation/logic/moderation_test.go
Normal file
@ -0,0 +1,120 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
"spark-server/internal/pkg/sensitive"
|
||||||
|
)
|
||||||
|
|
||||||
|
type fakeUsers struct {
|
||||||
|
mutedUntil time.Time
|
||||||
|
disabled bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *fakeUsers) AssertCanSend(_ context.Context, _ int64) error { return nil }
|
||||||
|
func (f *fakeUsers) DisableAccount(_ context.Context, _ int64) error {
|
||||||
|
f.disabled = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
func (f *fakeUsers) ExtendSendMutedUntil(_ context.Context, _ int64, until time.Time) error {
|
||||||
|
if until.After(f.mutedUntil) {
|
||||||
|
f.mutedUntil = until
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeIDs struct{ n int64 }
|
||||||
|
|
||||||
|
func (f *fakeIDs) NextID() (int64, error) {
|
||||||
|
f.n++
|
||||||
|
return f.n, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestScanReplace(t *testing.T) {
|
||||||
|
word := sensitive.FoldWord("赌博")
|
||||||
|
repo := NewMemoryRepo(&modentity.Word{
|
||||||
|
ID: 1, WordRaw: "赌博", Word: word, Category: modentity.CategoryGambling,
|
||||||
|
Action: modentity.ActionReplace, Enabled: true,
|
||||||
|
})
|
||||||
|
svc := NewService(repo, nil, &fakeIDs{})
|
||||||
|
res, err := svc.Scan(context.Background(), "今晚赌博")
|
||||||
|
if err != nil || res == nil || res.Action != modentity.ActionReplace || res.ReplacedBody != "今晚***" {
|
||||||
|
t.Fatalf("scan %+v %v", res, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCreateWordAndReload(t *testing.T) {
|
||||||
|
repo := NewMemoryRepo()
|
||||||
|
svc := NewService(repo, nil, &fakeIDs{})
|
||||||
|
w, err := svc.CreateWord(context.Background(), &CreateWordInput{
|
||||||
|
WordRaw: "广告词", Category: int8(modentity.CategoryAd), Action: int8(modentity.ActionLog),
|
||||||
|
})
|
||||||
|
if err != nil || w == nil || w.Word == "" {
|
||||||
|
t.Fatalf("create %+v %v", w, err)
|
||||||
|
}
|
||||||
|
list, total, err := svc.ListWords(context.Background(), "", 0, nil, 0, 10)
|
||||||
|
if err != nil || total != 1 || len(list) != 1 {
|
||||||
|
t.Fatalf("list %d %+v %v", total, list, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnforceReject(t *testing.T) {
|
||||||
|
repo := NewMemoryRepo(&modentity.Word{
|
||||||
|
ID: 2, Word: sensitive.FoldWord("毒品"), WordRaw: "毒品",
|
||||||
|
Category: modentity.CategoryDrug, Action: modentity.ActionReject, Enabled: true,
|
||||||
|
})
|
||||||
|
ids := &fakeIDs{}
|
||||||
|
svc := NewService(repo, &fakeUsers{}, ids)
|
||||||
|
res, _ := svc.Scan(context.Background(), "涉毒品")
|
||||||
|
if err := svc.Apply(context.Background(), 9, 1, &modentity.ApplyInput{Hits: res.Hits}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := svc.Enforce(context.Background(), 9, res); !errors.Is(err, errs.ErrContentBlocked) {
|
||||||
|
t.Fatalf("enforce %v", err)
|
||||||
|
}
|
||||||
|
if len(repo.hits) != 1 {
|
||||||
|
t.Fatalf("hits %d", len(repo.hits))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEnforceMute(t *testing.T) {
|
||||||
|
repo := NewMemoryRepo(&modentity.Word{
|
||||||
|
ID: 3, Word: sensitive.FoldWord("骂人"), WordRaw: "骂人",
|
||||||
|
Category: modentity.CategoryAbuse, Action: modentity.ActionMute, MuteSeconds: 60, Enabled: true,
|
||||||
|
})
|
||||||
|
fu := &fakeUsers{}
|
||||||
|
svc := NewService(repo, fu, &fakeIDs{})
|
||||||
|
res, _ := svc.Scan(context.Background(), "骂人啊")
|
||||||
|
if err := svc.Apply(context.Background(), 1, 2, &modentity.ApplyInput{Hits: res.Hits}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := svc.Enforce(context.Background(), 1, res); !errors.Is(err, errs.ErrContentBlocked) {
|
||||||
|
t.Fatalf("%v", err)
|
||||||
|
}
|
||||||
|
if fu.mutedUntil.IsZero() {
|
||||||
|
t.Fatal("expected mute")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPatchAndDeleteWord(t *testing.T) {
|
||||||
|
repo := NewMemoryRepo()
|
||||||
|
svc := NewService(repo, nil, &fakeIDs{})
|
||||||
|
w, err := svc.CreateWord(context.Background(), &CreateWordInput{
|
||||||
|
WordRaw: "临时", Category: int8(modentity.CategoryOther), Action: int8(modentity.ActionReject),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
act := int8(modentity.ActionLog)
|
||||||
|
if err := svc.PatchWord(context.Background(), w.ID, &PatchWordInput{Action: &act}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := svc.DeleteWord(context.Background(), w.ID); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
57
internal/module/moderation/logic/scan.go
Normal file
57
internal/module/moderation/logic/scan.go
Normal file
@ -0,0 +1,57 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Scan implements message TextFilter. No hit returns (nil, nil).
|
||||||
|
func (s *Service) Scan(ctx context.Context, text string) (*modentity.ScanResult, error) {
|
||||||
|
if !s.Ready() {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
_ = ctx
|
||||||
|
details, masked, ok := s.repo.MatcherFind(text)
|
||||||
|
if len(details) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
if !ok {
|
||||||
|
return &modentity.ScanResult{Action: modentity.ActionReject, Hits: toHits(details)}, nil
|
||||||
|
}
|
||||||
|
var actions []modentity.Action
|
||||||
|
var muteSec int
|
||||||
|
hits := make([]modentity.HitMatch, 0, len(details))
|
||||||
|
for i := range details {
|
||||||
|
d := details[i]
|
||||||
|
act := d.Action.Effective()
|
||||||
|
actions = append(actions, act)
|
||||||
|
if act == modentity.ActionMute {
|
||||||
|
if w := s.repo.LookupWord(d.WordID); w != nil && w.MuteSeconds > muteSec {
|
||||||
|
muteSec = w.MuteSeconds
|
||||||
|
}
|
||||||
|
}
|
||||||
|
hits = append(hits, modentity.HitMatch{
|
||||||
|
WordID: d.WordID, Category: d.Category, Action: d.Action,
|
||||||
|
Excerpt: d.Excerpt, ByteFrom: d.From, ByteTo: d.To,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
best := modentity.MaxAction(actions...)
|
||||||
|
res := &modentity.ScanResult{Action: best, Hits: hits, MuteSeconds: muteSec}
|
||||||
|
if best == modentity.ActionReplace {
|
||||||
|
res.ReplacedBody = masked
|
||||||
|
}
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func toHits(details []MatchDetail) []modentity.HitMatch {
|
||||||
|
out := make([]modentity.HitMatch, 0, len(details))
|
||||||
|
for i := range details {
|
||||||
|
d := details[i]
|
||||||
|
out = append(out, modentity.HitMatch{
|
||||||
|
WordID: d.WordID, Category: d.Category, Action: d.Action,
|
||||||
|
Excerpt: d.Excerpt, ByteFrom: d.From, ByteTo: d.To,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
79
internal/module/moderation/logic/service.go
Normal file
79
internal/module/moderation/logic/service.go
Normal file
@ -0,0 +1,79 @@
|
|||||||
|
// Package logic is sensitive-word scan, penalties, and admin word CRUD.
|
||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
var ErrNotReady = errors.New("moderation: not ready")
|
||||||
|
|
||||||
|
type Repository interface {
|
||||||
|
Ready() bool
|
||||||
|
ReloadMatcher(ctx context.Context) error
|
||||||
|
ListWords(ctx context.Context, q string, category int, enabled *bool, page, size int) ([]modentity.Word, int64, error)
|
||||||
|
CreateWord(ctx context.Context, w *modentity.Word) error
|
||||||
|
UpdateWord(ctx context.Context, id int64, patch WordPatch) error
|
||||||
|
SoftDeleteWord(ctx context.Context, id int64) error
|
||||||
|
InsertHits(ctx context.Context, hits []HitRecord) error
|
||||||
|
ListHits(ctx context.Context, userID int64, category int, since, until *time.Time, page, size int) ([]HitRecord, int64, error)
|
||||||
|
LookupWord(id int64) *modentity.Word
|
||||||
|
MatcherFind(text string) ([]MatchDetail, string, bool)
|
||||||
|
}
|
||||||
|
|
||||||
|
type WordPatch func(w *modentity.Word) error
|
||||||
|
|
||||||
|
type HitRecord struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
UserID int64 `json:"user_id"`
|
||||||
|
ConversationID int64 `json:"conversation_id"`
|
||||||
|
WordID int64 `json:"word_id"`
|
||||||
|
Category modentity.Category `json:"category"`
|
||||||
|
Action modentity.Action `json:"action"`
|
||||||
|
Excerpt string `json:"excerpt"`
|
||||||
|
RealIP string `json:"real_ip"`
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type MatchDetail struct {
|
||||||
|
WordID int64
|
||||||
|
Category modentity.Category
|
||||||
|
Action modentity.Action
|
||||||
|
Excerpt string
|
||||||
|
From int
|
||||||
|
To int
|
||||||
|
}
|
||||||
|
|
||||||
|
type UserModeration interface {
|
||||||
|
AssertCanSend(ctx context.Context, userID int64) error
|
||||||
|
DisableAccount(ctx context.Context, userID int64) error
|
||||||
|
ExtendSendMutedUntil(ctx context.Context, userID int64, until time.Time) error
|
||||||
|
}
|
||||||
|
|
||||||
|
type IDGen interface {
|
||||||
|
NextID() (int64, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type Service struct {
|
||||||
|
repo Repository
|
||||||
|
users UserModeration
|
||||||
|
ids IDGen
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewService(repo Repository, users UserModeration, ids IDGen) *Service {
|
||||||
|
return &Service{repo: repo, users: users, ids: ids}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) Ready() bool {
|
||||||
|
return s != nil && s.repo != nil && s.repo.Ready()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) Reload(ctx context.Context) error {
|
||||||
|
if !s.Ready() {
|
||||||
|
return ErrNotReady
|
||||||
|
}
|
||||||
|
return s.repo.ReloadMatcher(ctx)
|
||||||
|
}
|
||||||
102
internal/module/moderation/logic/words.go
Normal file
102
internal/module/moderation/logic/words.go
Normal file
@ -0,0 +1,102 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
"spark-server/internal/pkg/sensitive"
|
||||||
|
)
|
||||||
|
|
||||||
|
type CreateWordInput struct {
|
||||||
|
WordRaw string `json:"word" validate:"required,max=64" label:"field.sensitive_word" errmsg:"required:validate.moderation.word_required"`
|
||||||
|
Category int8 `json:"category" validate:"required,min=1,max=9" label:"field.sensitive_category"`
|
||||||
|
Action int8 `json:"action" validate:"required,min=1,max=6" label:"field.sensitive_action"`
|
||||||
|
MuteSeconds int `json:"mute_seconds" validate:"min=0" label:"field.mute_seconds"`
|
||||||
|
Enabled *bool `json:"enabled"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type PatchWordInput struct {
|
||||||
|
Category *int8 `json:"category" validate:"omitempty,min=1,max=9"`
|
||||||
|
Action *int8 `json:"action" validate:"omitempty,min=1,max=6"`
|
||||||
|
MuteSeconds *int `json:"mute_seconds" validate:"omitempty,min=0"`
|
||||||
|
Enabled *bool `json:"enabled"`
|
||||||
|
WordRaw *string `json:"word" validate:"omitempty,max=64"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) ListWords(ctx context.Context, q string, category int, enabled *bool, page, size int) ([]modentity.Word, int64, error) {
|
||||||
|
if !s.Ready() {
|
||||||
|
return nil, 0, ErrNotReady
|
||||||
|
}
|
||||||
|
return s.repo.ListWords(ctx, q, category, enabled, page, size)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) CreateWord(ctx context.Context, in *CreateWordInput) (*modentity.Word, error) {
|
||||||
|
if !s.Ready() || in == nil {
|
||||||
|
return nil, ErrNotReady
|
||||||
|
}
|
||||||
|
cat := modentity.Category(in.Category)
|
||||||
|
act := modentity.Action(in.Action).Effective()
|
||||||
|
if !cat.Valid() || !act.Valid() {
|
||||||
|
return nil, errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
norm := sensitive.FoldWord(in.WordRaw)
|
||||||
|
if norm == "" {
|
||||||
|
return nil, errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
enabled := true
|
||||||
|
if in.Enabled != nil {
|
||||||
|
enabled = *in.Enabled
|
||||||
|
}
|
||||||
|
w := &modentity.Word{
|
||||||
|
WordRaw: in.WordRaw, Word: norm, Category: cat, Action: act,
|
||||||
|
MuteSeconds: in.MuteSeconds, Enabled: enabled,
|
||||||
|
}
|
||||||
|
if err := s.repo.CreateWord(ctx, w); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return w, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) PatchWord(ctx context.Context, id int64, in *PatchWordInput) error {
|
||||||
|
if !s.Ready() || id <= 0 || in == nil {
|
||||||
|
return errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
return s.repo.UpdateWord(ctx, id, func(w *modentity.Word) error {
|
||||||
|
if in.WordRaw != nil {
|
||||||
|
w.WordRaw = *in.WordRaw
|
||||||
|
w.Word = sensitive.FoldWord(w.WordRaw)
|
||||||
|
if w.Word == "" {
|
||||||
|
return errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if in.Category != nil {
|
||||||
|
c := modentity.Category(*in.Category)
|
||||||
|
if !c.Valid() {
|
||||||
|
return errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
w.Category = c
|
||||||
|
}
|
||||||
|
if in.Action != nil {
|
||||||
|
a := modentity.Action(*in.Action).Effective()
|
||||||
|
if !a.Valid() {
|
||||||
|
return errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
w.Action = a
|
||||||
|
}
|
||||||
|
if in.MuteSeconds != nil {
|
||||||
|
w.MuteSeconds = *in.MuteSeconds
|
||||||
|
}
|
||||||
|
if in.Enabled != nil {
|
||||||
|
w.Enabled = *in.Enabled
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) DeleteWord(ctx context.Context, id int64) error {
|
||||||
|
if !s.Ready() || id <= 0 {
|
||||||
|
return errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
return s.repo.SoftDeleteWord(ctx, id)
|
||||||
|
}
|
||||||
119
internal/module/moderation/model/bridge.go
Normal file
119
internal/module/moderation/model/bridge.go
Normal file
@ -0,0 +1,119 @@
|
|||||||
|
package model
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
modlogic "spark-server/internal/module/moderation/logic"
|
||||||
|
"spark-server/internal/pkg/sensitive"
|
||||||
|
)
|
||||||
|
|
||||||
|
// LogicBridge adapts Repo to moderation/logic.Repository.
|
||||||
|
type LogicBridge struct {
|
||||||
|
*Repo
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewLogicBridge(r *Repo) *LogicBridge {
|
||||||
|
if r == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &LogicBridge{Repo: r}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *LogicBridge) LookupWord(id int64) *modentity.Word {
|
||||||
|
if b == nil || b.matcher == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
row := b.matcher.WordByID(id)
|
||||||
|
if row == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return row.entity()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *LogicBridge) MatcherFind(text string) ([]modlogic.MatchDetail, string, bool) {
|
||||||
|
m := b.matcher.Get()
|
||||||
|
idx := sensitive.FoldAndIndex(text)
|
||||||
|
if m == nil || len(idx.Norm) == 0 {
|
||||||
|
return nil, text, true
|
||||||
|
}
|
||||||
|
rawMatches := m.FindAll(idx.Norm)
|
||||||
|
if len(rawMatches) == 0 {
|
||||||
|
return nil, text, true
|
||||||
|
}
|
||||||
|
intervals, ok := sensitive.IntervalsFromMatches(&idx, rawMatches)
|
||||||
|
masked := text
|
||||||
|
if ok {
|
||||||
|
masked = sensitive.Mask(idx.OrigBytes, intervals, "***")
|
||||||
|
}
|
||||||
|
details := make([]modlogic.MatchDetail, 0, len(rawMatches))
|
||||||
|
for _, rm := range rawMatches {
|
||||||
|
row := b.matcher.WordByID(rm.EntryID)
|
||||||
|
if row == nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
ent := row.entity()
|
||||||
|
from, to := 0, 0
|
||||||
|
if ok && rm.Start >= 0 && rm.End <= len(idx.OrigFrom) {
|
||||||
|
from = idx.OrigFrom[rm.Start]
|
||||||
|
to = idx.OrigTo[rm.End-1]
|
||||||
|
}
|
||||||
|
ex := sensitive.Excerpt(idx.OrigBytes, from, to, 128)
|
||||||
|
details = append(details, modlogic.MatchDetail{
|
||||||
|
WordID: ent.ID, Category: ent.Category, Action: ent.Action,
|
||||||
|
Excerpt: ex, From: from, To: to,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if len(details) == 0 {
|
||||||
|
return nil, text, true
|
||||||
|
}
|
||||||
|
return details, masked, ok
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *LogicBridge) UpdateWord(ctx context.Context, id int64, patch modlogic.WordPatch) error {
|
||||||
|
return b.Repo.UpdateWord(ctx, id, func(row *WordRow) error {
|
||||||
|
if patch == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
ent := row.entity()
|
||||||
|
if err := patch(ent); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
row.WordRaw = ent.WordRaw
|
||||||
|
row.Word = ent.Word
|
||||||
|
row.Category = int8(ent.Category)
|
||||||
|
row.Action = int8(ent.Action)
|
||||||
|
row.MuteSeconds = ent.MuteSeconds
|
||||||
|
row.Enabled = boolToTiny(ent.Enabled)
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *LogicBridge) InsertHits(ctx context.Context, hits []modlogic.HitRecord) error {
|
||||||
|
rows := make([]HitRow, 0, len(hits))
|
||||||
|
for i := range hits {
|
||||||
|
h := hits[i]
|
||||||
|
rows = append(rows, hitFromEntity(h.ID, h.UserID, h.ConversationID, modentity.HitMatch{
|
||||||
|
WordID: h.WordID, Category: h.Category, Action: h.Action, Excerpt: h.Excerpt,
|
||||||
|
}, h.RealIP, h.CreatedAt))
|
||||||
|
}
|
||||||
|
return b.Repo.InsertHits(ctx, rows)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (b *LogicBridge) ListHits(ctx context.Context, userID int64, category int, since, until *time.Time, page, size int) ([]modlogic.HitRecord, int64, error) {
|
||||||
|
rows, total, err := b.Repo.ListHits(ctx, userID, category, since, until, page, size)
|
||||||
|
if err != nil {
|
||||||
|
return nil, 0, err
|
||||||
|
}
|
||||||
|
out := make([]modlogic.HitRecord, 0, len(rows))
|
||||||
|
for i := range rows {
|
||||||
|
r := rows[i]
|
||||||
|
out = append(out, modlogic.HitRecord{
|
||||||
|
ID: r.ID, UserID: r.UserID, ConversationID: r.ConversationID,
|
||||||
|
WordID: r.WordID, Category: modentity.Category(r.Category), Action: modentity.Action(r.Action),
|
||||||
|
Excerpt: r.Excerpt, RealIP: r.RealIP, CreatedAt: r.CreatedAt,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out, total, nil
|
||||||
|
}
|
||||||
35
internal/module/moderation/model/hit.go
Normal file
35
internal/module/moderation/model/hit.go
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
package model
|
||||||
|
|
||||||
|
import (
|
||||||
|
"time"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
type HitRow struct {
|
||||||
|
ID int64 `gorm:"column:id;primaryKey"`
|
||||||
|
UserID int64 `gorm:"column:user_id"`
|
||||||
|
ConversationID int64 `gorm:"column:conversation_id"`
|
||||||
|
WordID int64 `gorm:"column:word_id"`
|
||||||
|
Category int8 `gorm:"column:category"`
|
||||||
|
Action int8 `gorm:"column:action"`
|
||||||
|
Excerpt string `gorm:"column:excerpt"`
|
||||||
|
RealIP string `gorm:"column:real_ip"`
|
||||||
|
CreatedAt time.Time `gorm:"column:created_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (HitRow) TableName() string { return "t_sensitive_hits" }
|
||||||
|
|
||||||
|
func hitFromEntity(id int64, userID, convID int64, h modentity.HitMatch, realIP string, at time.Time) HitRow {
|
||||||
|
return HitRow{
|
||||||
|
ID: id,
|
||||||
|
UserID: userID,
|
||||||
|
ConversationID: convID,
|
||||||
|
WordID: h.WordID,
|
||||||
|
Category: int8(h.Category),
|
||||||
|
Action: int8(h.Action),
|
||||||
|
Excerpt: h.Excerpt,
|
||||||
|
RealIP: realIP,
|
||||||
|
CreatedAt: at,
|
||||||
|
}
|
||||||
|
}
|
||||||
50
internal/module/moderation/model/matcher.go
Normal file
50
internal/module/moderation/model/matcher.go
Normal file
@ -0,0 +1,50 @@
|
|||||||
|
package model
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sync/atomic"
|
||||||
|
|
||||||
|
"spark-server/internal/pkg/sensitive"
|
||||||
|
)
|
||||||
|
|
||||||
|
// MatcherStore holds a hot-swappable sensitive.Matcher.
|
||||||
|
type MatcherStore struct {
|
||||||
|
ptr atomic.Pointer[sensitive.Matcher]
|
||||||
|
byID map[int64]*WordRow
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewMatcherStore() *MatcherStore {
|
||||||
|
return &MatcherStore{byID: map[int64]*WordRow{}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *MatcherStore) Get() *sensitive.Matcher {
|
||||||
|
if s == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return s.ptr.Load()
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *MatcherStore) Replace(rows []WordRow) {
|
||||||
|
entries := make([]sensitive.Entry, 0, len(rows))
|
||||||
|
byID := make(map[int64]*WordRow, len(rows))
|
||||||
|
for i := range rows {
|
||||||
|
r := rows[i]
|
||||||
|
if r.Enabled != 1 || r.Word == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
byID[r.ID] = &rows[i]
|
||||||
|
entries = append(entries, sensitive.Entry{ID: r.ID, Word: r.Word})
|
||||||
|
}
|
||||||
|
m := sensitive.Build(entries)
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.byID = byID
|
||||||
|
s.ptr.Store(m)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *MatcherStore) WordByID(id int64) *WordRow {
|
||||||
|
if s == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return s.byID[id]
|
||||||
|
}
|
||||||
210
internal/module/moderation/model/repo.go
Normal file
210
internal/module/moderation/model/repo.go
Normal file
@ -0,0 +1,210 @@
|
|||||||
|
package model
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
mysqldriver "github.com/go-sql-driver/mysql"
|
||||||
|
"gorm.io/gorm"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Repo struct {
|
||||||
|
db *gorm.DB
|
||||||
|
matcher *MatcherStore
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewRepo(db *gorm.DB) *Repo {
|
||||||
|
if db == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &Repo{db: db, matcher: NewMatcherStore()}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) Ready() bool {
|
||||||
|
return r != nil && r.db != nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) Matcher() *MatcherStore {
|
||||||
|
if r == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return r.matcher
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) ReloadMatcher(ctx context.Context) error {
|
||||||
|
if !r.Ready() {
|
||||||
|
return errs.ErrAvailable
|
||||||
|
}
|
||||||
|
var rows []WordRow
|
||||||
|
if err := r.db.WithContext(ctx).Where("enabled = ?", 1).Find(&rows).Error; err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
r.matcher.Replace(rows)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) ListWords(ctx context.Context, q string, category int, enabled *bool, page, size int) ([]modentity.Word, int64, error) {
|
||||||
|
if !r.Ready() {
|
||||||
|
return nil, 0, errs.ErrAvailable
|
||||||
|
}
|
||||||
|
if size <= 0 || size > 200 {
|
||||||
|
size = 50
|
||||||
|
}
|
||||||
|
if page < 0 {
|
||||||
|
page = 0
|
||||||
|
}
|
||||||
|
db := r.db.WithContext(ctx).Model(&WordRow{})
|
||||||
|
q = trim(q)
|
||||||
|
if q != "" {
|
||||||
|
like := "%" + q + "%"
|
||||||
|
db = db.Where("word_raw LIKE ? OR word LIKE ?", like, like)
|
||||||
|
}
|
||||||
|
if category > 0 {
|
||||||
|
db = db.Where("category = ?", category)
|
||||||
|
}
|
||||||
|
if enabled != nil {
|
||||||
|
if *enabled {
|
||||||
|
db = db.Where("enabled = ?", 1)
|
||||||
|
} else {
|
||||||
|
db = db.Where("enabled = ?", 0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var total int64
|
||||||
|
if err := db.Count(&total).Error; err != nil {
|
||||||
|
return nil, 0, err
|
||||||
|
}
|
||||||
|
var rows []WordRow
|
||||||
|
if err := db.Order("id DESC").Offset(page * size).Limit(size).Find(&rows).Error; err != nil {
|
||||||
|
return nil, 0, err
|
||||||
|
}
|
||||||
|
out := make([]modentity.Word, 0, len(rows))
|
||||||
|
for i := range rows {
|
||||||
|
out = append(out, *rows[i].entity())
|
||||||
|
}
|
||||||
|
return out, total, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) CreateWord(ctx context.Context, w *modentity.Word) error {
|
||||||
|
if !r.Ready() || w == nil || w.Word == "" {
|
||||||
|
return errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
now := time.Now()
|
||||||
|
row := WordRow{
|
||||||
|
WordRaw: w.WordRaw, Word: w.Word, Category: int8(w.Category), Action: int8(w.Action),
|
||||||
|
MuteSeconds: w.MuteSeconds, Enabled: boolToTiny(w.Enabled), CreatedAt: now, UpdatedAt: now,
|
||||||
|
}
|
||||||
|
if err := r.db.WithContext(ctx).Create(&row).Error; err != nil {
|
||||||
|
return mapDB(err)
|
||||||
|
}
|
||||||
|
w.ID = row.ID
|
||||||
|
w.CreatedAt = now
|
||||||
|
w.UpdatedAt = now
|
||||||
|
return r.ReloadMatcher(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) UpdateWord(ctx context.Context, id int64, patch func(*WordRow) error) error {
|
||||||
|
if !r.Ready() || id <= 0 {
|
||||||
|
return errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
var row WordRow
|
||||||
|
if err := r.db.WithContext(ctx).Where("id = ?", id).Take(&row).Error; err != nil {
|
||||||
|
return mapDB(err)
|
||||||
|
}
|
||||||
|
if patch != nil {
|
||||||
|
if err := patch(&row); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
row.UpdatedAt = time.Now()
|
||||||
|
if err := r.db.WithContext(ctx).Save(&row).Error; err != nil {
|
||||||
|
return mapDB(err)
|
||||||
|
}
|
||||||
|
return r.ReloadMatcher(ctx)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) SoftDeleteWord(ctx context.Context, id int64) error {
|
||||||
|
return r.UpdateWord(ctx, id, func(row *WordRow) error {
|
||||||
|
row.Enabled = 0
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) InsertHits(ctx context.Context, rows []HitRow) error {
|
||||||
|
if !r.Ready() || len(rows) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return r.db.WithContext(ctx).Create(&rows).Error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *Repo) ListHits(ctx context.Context, userID int64, category int, since, until *time.Time, page, size int) ([]HitRow, int64, error) {
|
||||||
|
if !r.Ready() {
|
||||||
|
return nil, 0, errs.ErrAvailable
|
||||||
|
}
|
||||||
|
if size <= 0 || size > 200 {
|
||||||
|
size = 50
|
||||||
|
}
|
||||||
|
if page < 0 {
|
||||||
|
page = 0
|
||||||
|
}
|
||||||
|
db := r.db.WithContext(ctx).Model(&HitRow{})
|
||||||
|
if userID > 0 {
|
||||||
|
db = db.Where("user_id = ?", userID)
|
||||||
|
}
|
||||||
|
if category > 0 {
|
||||||
|
db = db.Where("category = ?", category)
|
||||||
|
}
|
||||||
|
if since != nil {
|
||||||
|
db = db.Where("created_at >= ?", *since)
|
||||||
|
}
|
||||||
|
if until != nil {
|
||||||
|
db = db.Where("created_at <= ?", *until)
|
||||||
|
}
|
||||||
|
var total int64
|
||||||
|
if err := db.Count(&total).Error; err != nil {
|
||||||
|
return nil, 0, err
|
||||||
|
}
|
||||||
|
var rows []HitRow
|
||||||
|
if err := db.Order("created_at DESC").Offset(page * size).Limit(size).Find(&rows).Error; err != nil {
|
||||||
|
return nil, 0, err
|
||||||
|
}
|
||||||
|
return rows, total, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func boolToTiny(v bool) int8 {
|
||||||
|
if v {
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func trim(s string) string {
|
||||||
|
for len(s) > 0 && (s[0] == ' ' || s[0] == '\t') {
|
||||||
|
s = s[1:]
|
||||||
|
}
|
||||||
|
for len(s) > 0 {
|
||||||
|
last := s[len(s)-1]
|
||||||
|
if last != ' ' && last != '\t' {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
s = s[:len(s)-1]
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapDB(err error) error {
|
||||||
|
if err == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if errors.Is(err, gorm.ErrRecordNotFound) {
|
||||||
|
return errs.ErrNotFound
|
||||||
|
}
|
||||||
|
var me *mysqldriver.MySQLError
|
||||||
|
if errors.As(err, &me) && me.Number == 1062 {
|
||||||
|
return errs.ErrDuplicate
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
38
internal/module/moderation/model/word.go
Normal file
38
internal/module/moderation/model/word.go
Normal file
@ -0,0 +1,38 @@
|
|||||||
|
package model
|
||||||
|
|
||||||
|
import (
|
||||||
|
"time"
|
||||||
|
|
||||||
|
modentity "spark-server/internal/module/moderation/entity"
|
||||||
|
)
|
||||||
|
|
||||||
|
type WordRow struct {
|
||||||
|
ID int64 `gorm:"column:id;primaryKey;autoIncrement"`
|
||||||
|
WordRaw string `gorm:"column:word_raw"`
|
||||||
|
Word string `gorm:"column:word"`
|
||||||
|
Category int8 `gorm:"column:category"`
|
||||||
|
Action int8 `gorm:"column:action"`
|
||||||
|
MuteSeconds int `gorm:"column:mute_seconds"`
|
||||||
|
Enabled int8 `gorm:"column:enabled"`
|
||||||
|
CreatedAt time.Time `gorm:"column:created_at"`
|
||||||
|
UpdatedAt time.Time `gorm:"column:updated_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (WordRow) TableName() string { return "t_sensitive_words" }
|
||||||
|
|
||||||
|
func (r *WordRow) entity() *modentity.Word {
|
||||||
|
if r == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &modentity.Word{
|
||||||
|
ID: r.ID,
|
||||||
|
WordRaw: r.WordRaw,
|
||||||
|
Word: r.Word,
|
||||||
|
Category: modentity.Category(r.Category),
|
||||||
|
Action: modentity.Action(r.Action),
|
||||||
|
MuteSeconds: r.MuteSeconds,
|
||||||
|
Enabled: r.Enabled == 1,
|
||||||
|
CreatedAt: r.CreatedAt,
|
||||||
|
UpdatedAt: r.UpdatedAt,
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -32,16 +32,17 @@ func ValidGender(g int) bool {
|
|||||||
|
|
||||||
// User is the account. ID is uid (Snowflake), never AUTO_INCREMENT.
|
// User is the account. ID is uid (Snowflake), never AUTO_INCREMENT.
|
||||||
type User struct {
|
type User struct {
|
||||||
ID int64
|
ID int64
|
||||||
SparkID string
|
SparkID string
|
||||||
Nickname string
|
Nickname string
|
||||||
Avatar string
|
Avatar string
|
||||||
QRCode string
|
QRCode string
|
||||||
Birthday time.Time
|
Birthday time.Time
|
||||||
Gender Gender
|
Gender Gender
|
||||||
Status Status
|
Status Status
|
||||||
LastLoginAt time.Time
|
SendMutedUntil time.Time
|
||||||
LastActiveAt time.Time
|
LastLoginAt time.Time
|
||||||
CreatedAt time.Time
|
LastActiveAt time.Time
|
||||||
UpdatedAt time.Time
|
CreatedAt time.Time
|
||||||
|
UpdatedAt time.Time
|
||||||
}
|
}
|
||||||
|
|||||||
72
internal/module/user/logic/account_send.go
Normal file
72
internal/module/user/logic/account_send.go
Normal file
@ -0,0 +1,72 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"spark-server/internal/module/user/entity"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
)
|
||||||
|
|
||||||
|
// AccountModeration is implemented by user persistence for send gates and bans.
|
||||||
|
type AccountModeration interface {
|
||||||
|
SetUserStatus(ctx context.Context, userID int64, status entity.Status) error
|
||||||
|
ExtendSendMutedUntil(ctx context.Context, userID int64, until time.Time) error
|
||||||
|
}
|
||||||
|
|
||||||
|
type BannedKicker interface {
|
||||||
|
KickUserBanned(userID int64)
|
||||||
|
}
|
||||||
|
|
||||||
|
// AssertCanSend is the AccountGate for message send (all content types).
|
||||||
|
func (s *Service) AssertCanSend(ctx context.Context, userID int64) error {
|
||||||
|
if !s.ready() || userID <= 0 {
|
||||||
|
return errs.ErrUnauthorized
|
||||||
|
}
|
||||||
|
u, err := s.repo.GetByID(ctx, userID)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if u.Status != entity.StatusActive {
|
||||||
|
return errs.ErrForbidden
|
||||||
|
}
|
||||||
|
if !u.SendMutedUntil.IsZero() && u.SendMutedUntil.After(time.Now()) {
|
||||||
|
return errs.ErrSendMuted
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExtendSendMutedUntil implements moderation.UserModeration.
|
||||||
|
func (s *Service) ExtendSendMutedUntil(ctx context.Context, userID int64, until time.Time) error {
|
||||||
|
mod, ok := s.repo.(AccountModeration)
|
||||||
|
if !ok || mod == nil {
|
||||||
|
return errs.ErrAvailable
|
||||||
|
}
|
||||||
|
return mod.ExtendSendMutedUntil(ctx, userID, until)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DisableAccount disables the user and revokes live sessions (moderation / admin ban).
|
||||||
|
func (s *Service) DisableAccount(ctx context.Context, userID int64) error {
|
||||||
|
mod, ok := s.repo.(AccountModeration)
|
||||||
|
if !ok || mod == nil {
|
||||||
|
return errs.ErrAvailable
|
||||||
|
}
|
||||||
|
if err := mod.SetUserStatus(ctx, userID, entity.StatusDisabled); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
s.revokeAccessBanned(ctx, userID)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) revokeAccessBanned(ctx context.Context, userID int64) {
|
||||||
|
if s.tokens != nil {
|
||||||
|
_ = s.tokens.LogoutAll(ctx, userID)
|
||||||
|
}
|
||||||
|
if bk, ok := s.sessions.(BannedKicker); ok && bk != nil {
|
||||||
|
bk.KickUserBanned(userID)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if s.sessions != nil {
|
||||||
|
s.sessions.KickUser(userID)
|
||||||
|
}
|
||||||
|
}
|
||||||
55
internal/module/user/logic/account_send_test.go
Normal file
55
internal/module/user/logic/account_send_test.go
Normal file
@ -0,0 +1,55 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"spark-server/internal/infrastructure/id"
|
||||||
|
"spark-server/internal/module/user/entity"
|
||||||
|
usermodel "spark-server/internal/module/user/model"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAssertCanSendMuted(t *testing.T) {
|
||||||
|
ids, err := id.New(1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
repo := usermodel.NewMemoryRepo()
|
||||||
|
users := NewService(repo, nil, ids, nil)
|
||||||
|
ctx := context.Background()
|
||||||
|
u, err := users.AuthenticateEmail(ctx, "test11@spark.com", "1", &entity.LoginDevice{DeviceID: 11, DeviceType: entity.DeviceWeb})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
until := time.Now().Add(time.Hour)
|
||||||
|
if err := repo.ExtendSendMutedUntil(ctx, u.ID, until); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := users.AssertCanSend(ctx, u.ID); !errors.Is(err, errs.ErrSendMuted) {
|
||||||
|
t.Fatalf("muted %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDisableAccount(t *testing.T) {
|
||||||
|
ids, err := id.New(2)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
repo := usermodel.NewMemoryRepo()
|
||||||
|
users := NewService(repo, nil, ids, nil)
|
||||||
|
ctx := context.Background()
|
||||||
|
u, err := users.AuthenticateEmail(ctx, "test12@spark.com", "1", &entity.LoginDevice{DeviceID: 12, DeviceType: entity.DeviceWeb})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := users.DisableAccount(ctx, u.ID); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
got, err := repo.GetByID(ctx, u.ID)
|
||||||
|
if err != nil || got.Status != entity.StatusDisabled {
|
||||||
|
t.Fatalf("status %+v %v", got, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
33
internal/module/user/model/memory_moderation.go
Normal file
33
internal/module/user/model/memory_moderation.go
Normal file
@ -0,0 +1,33 @@
|
|||||||
|
package model
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"spark-server/internal/module/user/entity"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (m *MemoryRepo) SetUserStatus(_ context.Context, userID int64, status entity.Status) error {
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
u := m.byID[userID]
|
||||||
|
if u == nil {
|
||||||
|
return errs.ErrNotFound
|
||||||
|
}
|
||||||
|
u.Status = status
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *MemoryRepo) ExtendSendMutedUntil(_ context.Context, userID int64, until time.Time) error {
|
||||||
|
m.mu.Lock()
|
||||||
|
defer m.mu.Unlock()
|
||||||
|
u := m.byID[userID]
|
||||||
|
if u == nil {
|
||||||
|
return errs.ErrNotFound
|
||||||
|
}
|
||||||
|
if u.SendMutedUntil.IsZero() || until.After(u.SendMutedUntil) {
|
||||||
|
u.SendMutedUntil = until
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
44
internal/module/user/model/moderation.go
Normal file
44
internal/module/user/model/moderation.go
Normal file
@ -0,0 +1,44 @@
|
|||||||
|
package model
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"spark-server/internal/module/user/entity"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (m *Repo) SetUserStatus(ctx context.Context, userID int64, status entity.Status) error {
|
||||||
|
db, err := m.session(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
res := db.Model(&User{}).Where("id = ?", userID).Updates(map[string]any{
|
||||||
|
"status": int(status),
|
||||||
|
"updated_at": time.Now(),
|
||||||
|
})
|
||||||
|
if res.Error != nil {
|
||||||
|
return mapDBErr(res.Error)
|
||||||
|
}
|
||||||
|
if res.RowsAffected == 0 {
|
||||||
|
return errs.ErrNotFound
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Repo) ExtendSendMutedUntil(ctx context.Context, userID int64, until time.Time) error {
|
||||||
|
db, err := m.session(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
// GREATEST(IFNULL(send_muted_until, '1970-01-01'), ?)
|
||||||
|
res := db.Exec(`UPDATE t_users SET send_muted_until = GREATEST(COALESCE(send_muted_until, '1970-01-01 00:00:00.000'), ?), updated_at = ? WHERE id = ?`,
|
||||||
|
until, time.Now(), userID)
|
||||||
|
if res.Error != nil {
|
||||||
|
return mapDBErr(res.Error)
|
||||||
|
}
|
||||||
|
if res.RowsAffected == 0 {
|
||||||
|
return errs.ErrNotFound
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@ -10,18 +10,19 @@ import (
|
|||||||
|
|
||||||
// User maps table users: account row, Snowflake uid, mutable spark_id.
|
// User maps table users: account row, Snowflake uid, mutable spark_id.
|
||||||
type User struct {
|
type User struct {
|
||||||
ID int64 `gorm:"column:id;primaryKey;comment:Snowflake 用户 id（uid），时间有序，非自增"`
|
ID int64 `gorm:"column:id;primaryKey;comment:Snowflake 用户 id（uid），时间有序，非自增"`
|
||||||
SparkID string `gorm:"column:spark_id;comment:对外展示的账号，用户可改，全局唯一"`
|
SparkID string `gorm:"column:spark_id;comment:对外展示的账号，用户可改，全局唯一"`
|
||||||
Nickname string `gorm:"column:nickname;comment:昵称"`
|
Nickname string `gorm:"column:nickname;comment:昵称"`
|
||||||
Avatar string `gorm:"column:avatar;comment:头像对象存储 key 或 URL，不存图片二进制"`
|
Avatar string `gorm:"column:avatar;comment:头像对象存储 key 或 URL，不存图片二进制"`
|
||||||
QRCode string `gorm:"column:qrcode;comment:个人二维码扫描票据，随机生成，不含 uid 或 spark_id"`
|
QRCode string `gorm:"column:qrcode;comment:个人二维码扫描票据，随机生成，不含 uid 或 spark_id"`
|
||||||
Birthday *time.Time `gorm:"column:birthday;type:date;comment:生日，未填为空"`
|
Birthday *time.Time `gorm:"column:birthday;type:date;comment:生日，未填为空"`
|
||||||
Gender int `gorm:"column:gender;comment:0=未知 1=男 2=女"`
|
Gender int `gorm:"column:gender;comment:0=未知 1=男 2=女"`
|
||||||
Status int `gorm:"column:status;comment:1=正常 0=禁用"`
|
Status int `gorm:"column:status;comment:1=正常 0=禁用"`
|
||||||
LastLoginAt *time.Time `gorm:"column:last_login_at;comment:最近一次登录时间"`
|
SendMutedUntil *time.Time `gorm:"column:send_muted_until;comment:全站禁言截止时间"`
|
||||||
LastActiveAt *time.Time `gorm:"column:last_active_at;comment:最近一次活跃时间（登录、心跳、发消息等）"`
|
LastLoginAt *time.Time `gorm:"column:last_login_at;comment:最近一次登录时间"`
|
||||||
CreatedAt time.Time `gorm:"column:created_at;comment:注册时间（create_time）"`
|
LastActiveAt *time.Time `gorm:"column:last_active_at;comment:最近一次活跃时间（登录、心跳、发消息等）"`
|
||||||
UpdatedAt time.Time `gorm:"column:updated_at;comment:资料最近更新时间（update_time）"`
|
CreatedAt time.Time `gorm:"column:created_at;comment:注册时间（create_time）"`
|
||||||
|
UpdatedAt time.Time `gorm:"column:updated_at;comment:资料最近更新时间（update_time）"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (User) TableName() string { return "t_users" }
|
func (User) TableName() string { return "t_users" }
|
||||||
@ -31,18 +32,19 @@ func (r *User) entity() *entity.User {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return &entity.User{
|
return &entity.User{
|
||||||
ID: r.ID,
|
ID: r.ID,
|
||||||
SparkID: r.SparkID,
|
SparkID: r.SparkID,
|
||||||
Nickname: r.Nickname,
|
Nickname: r.Nickname,
|
||||||
Avatar: r.Avatar,
|
Avatar: r.Avatar,
|
||||||
QRCode: r.QRCode,
|
QRCode: r.QRCode,
|
||||||
Birthday: timeOrZero(r.Birthday),
|
Birthday: timeOrZero(r.Birthday),
|
||||||
Gender: entity.Gender(r.Gender),
|
Gender: entity.Gender(r.Gender),
|
||||||
Status: entity.Status(r.Status),
|
Status: entity.Status(r.Status),
|
||||||
LastLoginAt: timeOrZero(r.LastLoginAt),
|
SendMutedUntil: timeOrZero(r.SendMutedUntil),
|
||||||
LastActiveAt: timeOrZero(r.LastActiveAt),
|
LastLoginAt: timeOrZero(r.LastLoginAt),
|
||||||
CreatedAt: r.CreatedAt,
|
LastActiveAt: timeOrZero(r.LastActiveAt),
|
||||||
UpdatedAt: r.UpdatedAt,
|
CreatedAt: r.CreatedAt,
|
||||||
|
UpdatedAt: r.UpdatedAt,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -12,6 +12,8 @@ var (
|
|||||||
ErrNotAuthenticated = NewError("40102", "not authenticated", "")
|
ErrNotAuthenticated = NewError("40102", "not authenticated", "")
|
||||||
ErrVerifyCodeInvalid = NewError("40103", "verification code invalid", "")
|
ErrVerifyCodeInvalid = NewError("40103", "verification code invalid", "")
|
||||||
ErrForbidden = NewError("40301", "forbidden", "")
|
ErrForbidden = NewError("40301", "forbidden", "")
|
||||||
|
ErrContentBlocked = NewError("40302", "content blocked", "")
|
||||||
|
ErrSendMuted = NewError("40303", "send muted", "")
|
||||||
ErrNotFound = NewError("40401", "not found", "")
|
ErrNotFound = NewError("40401", "not found", "")
|
||||||
ErrMediaExpired = NewError("40402", "media expired", "")
|
ErrMediaExpired = NewError("40402", "media expired", "")
|
||||||
ErrDuplicate = NewError("40901", "duplicate", "")
|
ErrDuplicate = NewError("40901", "duplicate", "")
|
||||||
|
|||||||
@ -32,6 +32,12 @@ func init() {
|
|||||||
"err.40301": {
|
"err.40301": {
|
||||||
ZhCN: "无权限", En: "forbidden", ZhTW: "無權限",
|
ZhCN: "无权限", En: "forbidden", ZhTW: "無權限",
|
||||||
},
|
},
|
||||||
|
"err.40302": {
|
||||||
|
ZhCN: "消息包含违规内容", En: "message blocked", ZhTW: "訊息包含違規內容",
|
||||||
|
},
|
||||||
|
"err.40303": {
|
||||||
|
ZhCN: "账号禁言中", En: "send muted", ZhTW: "帳號禁言中",
|
||||||
|
},
|
||||||
"err.40401": {
|
"err.40401": {
|
||||||
ZhCN: "未找到", En: "not found", ZhTW: "未找到",
|
ZhCN: "未找到", En: "not found", ZhTW: "未找到",
|
||||||
},
|
},
|
||||||
|
|||||||
@ -11,6 +11,9 @@ func init() {
|
|||||||
NotifyKickDeleted: {
|
NotifyKickDeleted: {
|
||||||
ZhCN: "账号已注销", En: "Account deleted", ZhTW: "帳號已註銷",
|
ZhCN: "账号已注销", En: "Account deleted", ZhTW: "帳號已註銷",
|
||||||
},
|
},
|
||||||
|
NotifyKickBanned: {
|
||||||
|
ZhCN: "账号已被封禁", En: "Account banned", ZhTW: "帳號已被封禁",
|
||||||
|
},
|
||||||
NotifyFriendRequest + ".title": {
|
NotifyFriendRequest + ".title": {
|
||||||
ZhCN: "好友请求", En: "Friend request", ZhTW: "好友請求",
|
ZhCN: "好友请求", En: "Friend request", ZhTW: "好友請求",
|
||||||
},
|
},
|
||||||
|
|||||||
@ -187,6 +187,7 @@ const (
|
|||||||
NotifyKickReplaced = "notify.kick.replaced"
|
NotifyKickReplaced = "notify.kick.replaced"
|
||||||
NotifyKickLogout = "notify.kick.logout"
|
NotifyKickLogout = "notify.kick.logout"
|
||||||
NotifyKickDeleted = "notify.kick.deleted"
|
NotifyKickDeleted = "notify.kick.deleted"
|
||||||
|
NotifyKickBanned = "notify.kick.banned"
|
||||||
NotifyFriendRequest = "notify.friend.request"
|
NotifyFriendRequest = "notify.friend.request"
|
||||||
NotifyFriendBecame = "notify.friend.became"
|
NotifyFriendBecame = "notify.friend.became"
|
||||||
NotifyGroupInvite = "notify.group.invite"
|
NotifyGroupInvite = "notify.group.invite"
|
||||||
|
|||||||
99
internal/pkg/sensitive/fold.go
Normal file
99
internal/pkg/sensitive/fold.go
Normal file
@ -0,0 +1,99 @@
|
|||||||
|
package sensitive
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"unicode"
|
||||||
|
"unicode/utf8"
|
||||||
|
)
|
||||||
|
|
||||||
|
const maxScanRunes = 8192
|
||||||
|
|
||||||
|
// Index maps normalized rune position to original UTF-8 byte offsets [from, to).
|
||||||
|
type Index struct {
|
||||||
|
Norm []rune
|
||||||
|
OrigFrom []int
|
||||||
|
OrigTo []int
|
||||||
|
OrigBytes []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
// FoldWord normalizes a dictionary entry (same rules as FoldAndIndex on short input).
|
||||||
|
func FoldWord(s string) string {
|
||||||
|
idx := FoldAndIndex(s)
|
||||||
|
if len(idx.Norm) == 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return string(idx.Norm)
|
||||||
|
}
|
||||||
|
|
||||||
|
// FoldAndIndex builds the normalized rune stream and byte mapping into the original text.
|
||||||
|
func FoldAndIndex(text string) Index {
|
||||||
|
raw := []byte(text)
|
||||||
|
var norm []rune
|
||||||
|
var from, to []int
|
||||||
|
runes := 0
|
||||||
|
for i := 0; i < len(raw); {
|
||||||
|
r, w := decodeRune(raw, i)
|
||||||
|
if w == 0 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
start := i
|
||||||
|
end := i + w
|
||||||
|
i = end
|
||||||
|
if dropRune(r) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
r = foldRune(r)
|
||||||
|
if r == 0 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
runes++
|
||||||
|
if runes > maxScanRunes {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
norm = append(norm, r)
|
||||||
|
from = append(from, start)
|
||||||
|
to = append(to, end)
|
||||||
|
}
|
||||||
|
return Index{Norm: norm, OrigFrom: from, OrigTo: to, OrigBytes: raw}
|
||||||
|
}
|
||||||
|
|
||||||
|
func dropRune(r rune) bool {
|
||||||
|
if r <= unicode.MaxASCII {
|
||||||
|
if unicode.IsSpace(r) || isASCIIPunct(r) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if unicode.In(r, unicode.Cf, unicode.Zs, unicode.Zl, unicode.Zp) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if isCJKPunct(r) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func isASCIIPunct(r rune) bool {
|
||||||
|
return strings.ContainsRune(`!"#$%&'()*+,-./:;<=>?@[\]^_`+"`"+`{|}~`, r)
|
||||||
|
}
|
||||||
|
|
||||||
|
func isCJKPunct(r rune) bool {
|
||||||
|
return strings.ContainsRune("，。！？、；：…·「」『』【】（）", r)
|
||||||
|
}
|
||||||
|
|
||||||
|
func foldRune(r rune) rune {
|
||||||
|
if r >= 0xFF01 && r <= 0xFF5E {
|
||||||
|
return r - 0xFEE0
|
||||||
|
}
|
||||||
|
return unicode.ToLower(r)
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeRune(b []byte, i int) (rune, int) {
|
||||||
|
if i >= len(b) {
|
||||||
|
return 0, 0
|
||||||
|
}
|
||||||
|
r, w := utf8.DecodeRune(b[i:])
|
||||||
|
if w == 0 {
|
||||||
|
return utf8.RuneError, 1
|
||||||
|
}
|
||||||
|
return r, w
|
||||||
|
}
|
||||||
111
internal/pkg/sensitive/matcher.go
Normal file
111
internal/pkg/sensitive/matcher.go
Normal file
@ -0,0 +1,111 @@
|
|||||||
|
package sensitive
|
||||||
|
|
||||||
|
// Entry is one enabled dictionary pattern (normalized word).
|
||||||
|
type Entry struct {
|
||||||
|
ID int64
|
||||||
|
Word string
|
||||||
|
RuneLen int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Match is a hit on the normalized stream [Start, End) rune indices.
|
||||||
|
type Match struct {
|
||||||
|
EntryID int64
|
||||||
|
Start int
|
||||||
|
End int
|
||||||
|
}
|
||||||
|
|
||||||
|
// Matcher is an Aho-Corasick automaton over normalized patterns.
|
||||||
|
type Matcher struct {
|
||||||
|
entries []Entry
|
||||||
|
root *acNode
|
||||||
|
}
|
||||||
|
|
||||||
|
type acNode struct {
|
||||||
|
next map[rune]*acNode
|
||||||
|
fail *acNode
|
||||||
|
outputs []int // indices into entries
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build constructs a matcher from patterns. Empty patterns are skipped.
|
||||||
|
func Build(entries []Entry) *Matcher {
|
||||||
|
m := &Matcher{entries: make([]Entry, 0, len(entries)), root: &acNode{next: map[rune]*acNode{}}}
|
||||||
|
for i := range entries {
|
||||||
|
e := entries[i]
|
||||||
|
w := e.Word
|
||||||
|
if w == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
idx := len(m.entries)
|
||||||
|
rl := len([]rune(w))
|
||||||
|
e.RuneLen = rl
|
||||||
|
m.entries = append(m.entries, e)
|
||||||
|
m.addPattern(w, rl, idx)
|
||||||
|
}
|
||||||
|
m.buildFails()
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Matcher) addPattern(word string, _ int, outIdx int) {
|
||||||
|
node := m.root
|
||||||
|
for _, r := range word {
|
||||||
|
child := node.next[r]
|
||||||
|
if child == nil {
|
||||||
|
child = &acNode{next: map[rune]*acNode{}}
|
||||||
|
node.next[r] = child
|
||||||
|
}
|
||||||
|
node = child
|
||||||
|
}
|
||||||
|
node.outputs = append(node.outputs, outIdx)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *Matcher) buildFails() {
|
||||||
|
queue := make([]*acNode, 0, 16)
|
||||||
|
for _, child := range m.root.next {
|
||||||
|
child.fail = m.root
|
||||||
|
queue = append(queue, child)
|
||||||
|
}
|
||||||
|
for qi := 0; qi < len(queue); qi++ {
|
||||||
|
cur := queue[qi]
|
||||||
|
for r, child := range cur.next {
|
||||||
|
queue = append(queue, child)
|
||||||
|
f := cur.fail
|
||||||
|
for f != nil && f.next[r] == nil {
|
||||||
|
f = f.fail
|
||||||
|
}
|
||||||
|
if f == nil {
|
||||||
|
child.fail = m.root
|
||||||
|
} else {
|
||||||
|
child.fail = f.next[r]
|
||||||
|
}
|
||||||
|
child.outputs = append(child.outputs, child.fail.outputs...)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// FindAll returns every match in norm (inclusive start, exclusive end).
|
||||||
|
func (m *Matcher) FindAll(norm []rune) []Match {
|
||||||
|
if m == nil || len(norm) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
var out []Match
|
||||||
|
node := m.root
|
||||||
|
for i, r := range norm {
|
||||||
|
for node != m.root && node.next[r] == nil {
|
||||||
|
node = node.fail
|
||||||
|
}
|
||||||
|
next := node.next[r]
|
||||||
|
if next == nil {
|
||||||
|
next = m.root
|
||||||
|
}
|
||||||
|
node = next
|
||||||
|
for _, idx := range node.outputs {
|
||||||
|
ent := m.entries[idx]
|
||||||
|
start := i + 1 - ent.RuneLen
|
||||||
|
if start < 0 {
|
||||||
|
start = 0
|
||||||
|
}
|
||||||
|
out = append(out, Match{EntryID: ent.ID, Start: start, End: i + 1})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
104
internal/pkg/sensitive/replace.go
Normal file
104
internal/pkg/sensitive/replace.go
Normal file
@ -0,0 +1,104 @@
|
|||||||
|
package sensitive
|
||||||
|
|
||||||
|
import "sort"
|
||||||
|
|
||||||
|
// ByteInterval is a span in the original UTF-8 text.
|
||||||
|
type ByteInterval struct {
|
||||||
|
From int
|
||||||
|
To int
|
||||||
|
}
|
||||||
|
|
||||||
|
// IntervalsFromMatches maps normalized matches back to original byte intervals.
|
||||||
|
func IntervalsFromMatches(idx *Index, matches []Match) ([]ByteInterval, bool) {
|
||||||
|
if idx == nil || len(matches) == 0 {
|
||||||
|
return nil, true
|
||||||
|
}
|
||||||
|
intervals := make([]ByteInterval, 0, len(matches))
|
||||||
|
for i := range matches {
|
||||||
|
m := matches[i]
|
||||||
|
if m.Start < 0 || m.End <= m.Start || m.End > len(idx.OrigFrom) {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
last := m.End - 1
|
||||||
|
intervals = append(intervals, ByteInterval{
|
||||||
|
From: idx.OrigFrom[m.Start],
|
||||||
|
To: idx.OrigTo[last],
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return mergeIntervals(intervals), true
|
||||||
|
}
|
||||||
|
|
||||||
|
func mergeIntervals(in []ByteInterval) []ByteInterval {
|
||||||
|
if len(in) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
sort.Slice(in, func(i, j int) bool { return in[i].From < in[j].From })
|
||||||
|
out := []ByteInterval{in[0]}
|
||||||
|
for i := 1; i < len(in); i++ {
|
||||||
|
last := &out[len(out)-1]
|
||||||
|
cur := in[i]
|
||||||
|
if cur.From <= last.To {
|
||||||
|
if cur.To > last.To {
|
||||||
|
last.To = cur.To
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, cur)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// Mask replaces each interval with repl (default "***").
|
||||||
|
func Mask(orig []byte, intervals []ByteInterval, repl string) string {
|
||||||
|
if len(intervals) == 0 {
|
||||||
|
return string(orig)
|
||||||
|
}
|
||||||
|
if repl == "" {
|
||||||
|
repl = "***"
|
||||||
|
}
|
||||||
|
var b []byte
|
||||||
|
pos := 0
|
||||||
|
for _, iv := range intervals {
|
||||||
|
if iv.From < pos || iv.To < iv.From || iv.To > len(orig) {
|
||||||
|
return string(orig)
|
||||||
|
}
|
||||||
|
b = append(b, orig[pos:iv.From]...)
|
||||||
|
b = append(b, repl...)
|
||||||
|
pos = iv.To
|
||||||
|
}
|
||||||
|
b = append(b, orig[pos:]...)
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Excerpt returns a short slice of orig around [from,to).
|
||||||
|
func Excerpt(orig []byte, from, to int, maxLen int) string {
|
||||||
|
if maxLen <= 0 {
|
||||||
|
maxLen = 128
|
||||||
|
}
|
||||||
|
if from < 0 {
|
||||||
|
from = 0
|
||||||
|
}
|
||||||
|
if to > len(orig) {
|
||||||
|
to = len(orig)
|
||||||
|
}
|
||||||
|
if from >= to {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
const pad = 24
|
||||||
|
start := from - pad
|
||||||
|
if start < 0 {
|
||||||
|
start = 0
|
||||||
|
}
|
||||||
|
end := to + pad
|
||||||
|
if end > len(orig) {
|
||||||
|
end = len(orig)
|
||||||
|
}
|
||||||
|
s := string(orig[start:end])
|
||||||
|
if len(s) > maxLen {
|
||||||
|
if maxLen <= 3 {
|
||||||
|
return s[:maxLen]
|
||||||
|
}
|
||||||
|
return s[:maxLen-3] + "..."
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
18
internal/pkg/sensitive/scan.go
Normal file
18
internal/pkg/sensitive/scan.go
Normal file
@ -0,0 +1,18 @@
|
|||||||
|
package sensitive
|
||||||
|
|
||||||
|
// ScanText folds text, runs the matcher, and optionally masks the original body.
|
||||||
|
func ScanText(m *Matcher, text string) (matches []Match, masked string, ok bool) {
|
||||||
|
idx := FoldAndIndex(text)
|
||||||
|
if len(idx.Norm) == 0 || m == nil {
|
||||||
|
return nil, text, true
|
||||||
|
}
|
||||||
|
matches = m.FindAll(idx.Norm)
|
||||||
|
if len(matches) == 0 {
|
||||||
|
return nil, text, true
|
||||||
|
}
|
||||||
|
intervals, ok := IntervalsFromMatches(&idx, matches)
|
||||||
|
if !ok {
|
||||||
|
return matches, text, false
|
||||||
|
}
|
||||||
|
return matches, Mask(idx.OrigBytes, intervals, "***"), true
|
||||||
|
}
|
||||||
41
internal/pkg/sensitive/sensitive_test.go
Normal file
41
internal/pkg/sensitive/sensitive_test.go
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
package sensitive
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestFoldAndMatch(t *testing.T) {
|
||||||
|
m := Build([]Entry{{ID: 1, Word: FoldWord("赌博")}})
|
||||||
|
matches, masked, ok := ScanText(m, "今晚赌博")
|
||||||
|
if !ok || len(matches) != 1 || masked == "今晚赌博" {
|
||||||
|
t.Fatalf("match %+v masked=%q ok=%v", matches, masked, ok)
|
||||||
|
}
|
||||||
|
if masked != "今晚***" {
|
||||||
|
t.Fatalf("masked=%q", masked)
|
||||||
|
}
|
||||||
|
matches, _, ok = ScanText(m, "今晚赌 博啊")
|
||||||
|
if !ok || len(matches) != 1 {
|
||||||
|
t.Fatalf("spaced %+v", matches)
|
||||||
|
}
|
||||||
|
matches, _, ok = ScanText(m, "赌\u200b博")
|
||||||
|
if !ok || len(matches) != 1 {
|
||||||
|
t.Fatalf("zwsp %+v", matches)
|
||||||
|
}
|
||||||
|
m2 := Build([]Entry{{ID: 2, Word: FoldWord("casino")}})
|
||||||
|
matches, _, ok = ScanText(m2, "Casino")
|
||||||
|
if !ok || len(matches) != 1 {
|
||||||
|
t.Fatalf("case %+v", matches)
|
||||||
|
}
|
||||||
|
matches, _, _ = ScanText(m, "你好")
|
||||||
|
if len(matches) != 0 {
|
||||||
|
t.Fatalf("no hit %+v", matches)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAssInClass(t *testing.T) {
|
||||||
|
m := Build([]Entry{{ID: 1, Word: FoldWord("ass")}})
|
||||||
|
matches, _, ok := ScanText(m, "class")
|
||||||
|
if !ok || len(matches) != 1 {
|
||||||
|
t.Fatalf("expected substring hit %+v", matches)
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -381,6 +381,7 @@ tags:
|
|||||||
description: |
|
description: |
|
||||||
popadmin 音乐管理：`/api/admin/spark/music*`（`/api/admin/pop/music*` 别名）。
|
popadmin 音乐管理：`/api/admin/spark/music*`（`/api/admin/pop/music*` 别名）。
|
||||||
曲目 CRUD、封面/背景/音频上传、分类 CRUD、曲目分类绑定（多对多）、全局试听秒数。
|
曲目 CRUD、封面/背景/音频上传、分类 CRUD、曲目分类绑定（多对多）、全局试听秒数。
|
||||||
|
敏感词：`/api/admin/spark/sensitive*`（`/api/admin/pop/sensitive*` 别名）；词库与命中审计、热重载。
|
||||||
Admin JWT；响应 `{success,message,data}`。Host 常用 `https://popadmin.wyrs365.com`。
|
Admin JWT；响应 `{success,message,data}`。Host 常用 `https://popadmin.wyrs365.com`。
|
||||||
- name: novels
|
- name: novels
|
||||||
description: |
|
description: |
|
||||||
@ -2434,6 +2435,157 @@ paths:
|
|||||||
schema:
|
schema:
|
||||||
$ref: "#/components/schemas/AdminApiResponse"
|
$ref: "#/components/schemas/AdminApiResponse"
|
||||||
|
|
||||||
|
/api/admin/spark/sensitive/words:
|
||||||
|
get:
|
||||||
|
tags: [admin-music]
|
||||||
|
summary: 敏感词分页列表
|
||||||
|
description: |
|
||||||
|
Query：`q`（匹配 word_raw/word）、`category`（1–9）、`enabled`（`true`/`false`/`1`/`0`）、`page`、`size`。
|
||||||
|
别名：`GET /api/admin/pop/sensitive/words`。
|
||||||
|
operationId: adminListSensitiveWords
|
||||||
|
security:
|
||||||
|
- bearerAuth: []
|
||||||
|
parameters:
|
||||||
|
- name: q
|
||||||
|
in: query
|
||||||
|
schema: { type: string }
|
||||||
|
- name: category
|
||||||
|
in: query
|
||||||
|
schema: { $ref: "#/components/schemas/SensitiveCategory" }
|
||||||
|
- name: enabled
|
||||||
|
in: query
|
||||||
|
schema: { type: string }
|
||||||
|
description: true/false 或 1/0
|
||||||
|
- name: page
|
||||||
|
in: query
|
||||||
|
schema: { type: integer, default: 0 }
|
||||||
|
- name: size
|
||||||
|
in: query
|
||||||
|
schema: { type: integer, default: 50, maximum: 200 }
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: "`data.items` 为词条，`data.total` 为总数"
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/AdminApiResponse"
|
||||||
|
post:
|
||||||
|
tags: [admin-music]
|
||||||
|
summary: 新建敏感词
|
||||||
|
description: |
|
||||||
|
`word` 为运营录入原文；服务端 Fold 后写入 `word` 列（唯一键）。`action=4`（hide）在 v1 会按 reject 处理。
|
||||||
|
成功后自动 Reload 自动机。
|
||||||
|
operationId: adminCreateSensitiveWord
|
||||||
|
security:
|
||||||
|
- bearerAuth: []
|
||||||
|
requestBody:
|
||||||
|
required: true
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/SensitiveWordCreateRequest"
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: 返回新建词条
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/AdminApiResponse"
|
||||||
|
|
||||||
|
/api/admin/spark/sensitive/reload:
|
||||||
|
post:
|
||||||
|
tags: [admin-music]
|
||||||
|
summary: 从数据库重建敏感词自动机
|
||||||
|
operationId: adminReloadSensitiveWords
|
||||||
|
security:
|
||||||
|
- bearerAuth: []
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: OK
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/AdminApiResponse"
|
||||||
|
|
||||||
|
/api/admin/spark/sensitive/words/{id}:
|
||||||
|
patch:
|
||||||
|
tags: [admin-music]
|
||||||
|
summary: 更新敏感词
|
||||||
|
operationId: adminPatchSensitiveWord
|
||||||
|
security:
|
||||||
|
- bearerAuth: []
|
||||||
|
parameters:
|
||||||
|
- name: id
|
||||||
|
in: path
|
||||||
|
required: true
|
||||||
|
schema: { type: integer, format: int64 }
|
||||||
|
requestBody:
|
||||||
|
required: true
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/SensitiveWordPatchRequest"
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: OK
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/AdminApiResponse"
|
||||||
|
delete:
|
||||||
|
tags: [admin-music]
|
||||||
|
summary: 下线敏感词（enabled=0）
|
||||||
|
operationId: adminDeleteSensitiveWord
|
||||||
|
security:
|
||||||
|
- bearerAuth: []
|
||||||
|
parameters:
|
||||||
|
- name: id
|
||||||
|
in: path
|
||||||
|
required: true
|
||||||
|
schema: { type: integer, format: int64 }
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: OK
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/AdminApiResponse"
|
||||||
|
|
||||||
|
/api/admin/spark/sensitive/hits:
|
||||||
|
get:
|
||||||
|
tags: [admin-music]
|
||||||
|
summary: 敏感词命中审计分页
|
||||||
|
description: Query：`uid`、`category`、`since`/`until`（RFC3339）、`page`、`size`。
|
||||||
|
operationId: adminListSensitiveHits
|
||||||
|
security:
|
||||||
|
- bearerAuth: []
|
||||||
|
parameters:
|
||||||
|
- name: uid
|
||||||
|
in: query
|
||||||
|
schema: { type: integer, format: int64 }
|
||||||
|
- name: category
|
||||||
|
in: query
|
||||||
|
schema: { $ref: "#/components/schemas/SensitiveCategory" }
|
||||||
|
- name: since
|
||||||
|
in: query
|
||||||
|
schema: { type: string, format: date-time }
|
||||||
|
- name: until
|
||||||
|
in: query
|
||||||
|
schema: { type: string, format: date-time }
|
||||||
|
- name: page
|
||||||
|
in: query
|
||||||
|
schema: { type: integer, default: 0 }
|
||||||
|
- name: size
|
||||||
|
in: query
|
||||||
|
schema: { type: integer, default: 50, maximum: 200 }
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: "`data.items` 为命中行，`data.total` 为总数"
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/AdminApiResponse"
|
||||||
|
|
||||||
# —— FullStoryNovel 公开小说 API（基址选 admin.fullstorynovel.com 或 fullstorynovel.com）——
|
# —— FullStoryNovel 公开小说 API（基址选 admin.fullstorynovel.com 或 fullstorynovel.com）——
|
||||||
/api/home:
|
/api/home:
|
||||||
get:
|
get:
|
||||||
@ -3262,7 +3414,7 @@ components:
|
|||||||
12 SYNC_MESSAGES 13 SYNC_MESSAGES_FINISH 14 KICK 15 DELETE_MESSAGE
|
12 SYNC_MESSAGES 13 SYNC_MESSAGES_FINISH 14 KICK 15 DELETE_MESSAGE
|
||||||
EnvelopeCode:
|
EnvelopeCode:
|
||||||
type: string
|
type: string
|
||||||
enum: ["0", "40001", "40002", "40003", "40004", "40005", "40101", "40102", "40103", "40301", "40401", "40402", "40901", "42901", "42902", "42903", "49901", "50001", "50002", "50003", "50301", "50401"]
|
enum: ["0", "40001", "40002", "40003", "40004", "40005", "40101", "40102", "40103", "40301", "40302", "40303", "40401", "40402", "40901", "42901", "42902", "42903", "49901", "50001", "50002", "50003", "50301", "50401"]
|
||||||
description: |
|
description: |
|
||||||
envelope.code。成功为 "0"。
|
envelope.code。成功为 "0"。
|
||||||
`40005` media_oversized：媒体分辨率/体积超限（开启 media.user_reject 时拒收）。
|
`40005` media_oversized：媒体分辨率/体积超限（开启 media.user_reject 时拒收）。
|
||||||
@ -3364,6 +3516,60 @@ components:
|
|||||||
$ref: "#/components/schemas/MusicCategory"
|
$ref: "#/components/schemas/MusicCategory"
|
||||||
description: 启用中的分类树
|
description: 启用中的分类树
|
||||||
memberPreviewSeconds: { type: integer }
|
memberPreviewSeconds: { type: integer }
|
||||||
|
SensitiveCategory:
|
||||||
|
type: integer
|
||||||
|
enum: [1, 2, 3, 4, 5, 6, 7, 8, 9]
|
||||||
|
description: |
|
||||||
|
1=涉黑 2=涉毒 3=涉黄 4=涉赌 5=涉政 6=宗教 7=辱骂 8=广告 9=其他
|
||||||
|
SensitiveAction:
|
||||||
|
type: integer
|
||||||
|
enum: [1, 2, 3, 4, 5, 6]
|
||||||
|
description: |
|
||||||
|
1=log 2=replace 3=reject 4=hide（v1 按 reject）5=mute 6=disable
|
||||||
|
SensitiveWord:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
id: { type: integer, format: int64 }
|
||||||
|
word_raw: { type: string, description: 运营录入原文 }
|
||||||
|
word: { type: string, description: 规范化词面，进自动机 }
|
||||||
|
category: { $ref: "#/components/schemas/SensitiveCategory" }
|
||||||
|
action: { $ref: "#/components/schemas/SensitiveAction" }
|
||||||
|
mute_seconds: { type: integer, description: action=mute 时生效 }
|
||||||
|
enabled: { type: boolean }
|
||||||
|
created_at: { type: string, format: date-time }
|
||||||
|
updated_at: { type: string, format: date-time }
|
||||||
|
SensitiveWordCreateRequest:
|
||||||
|
type: object
|
||||||
|
required: [word, category, action]
|
||||||
|
properties:
|
||||||
|
word:
|
||||||
|
type: string
|
||||||
|
maxLength: 64
|
||||||
|
description: 词面原文（入库后 Fold 为 word 列）
|
||||||
|
category: { $ref: "#/components/schemas/SensitiveCategory" }
|
||||||
|
action: { $ref: "#/components/schemas/SensitiveAction" }
|
||||||
|
mute_seconds: { type: integer, minimum: 0, default: 0 }
|
||||||
|
enabled: { type: boolean, default: true }
|
||||||
|
SensitiveWordPatchRequest:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
word: { type: string, maxLength: 64 }
|
||||||
|
category: { $ref: "#/components/schemas/SensitiveCategory" }
|
||||||
|
action: { $ref: "#/components/schemas/SensitiveAction" }
|
||||||
|
mute_seconds: { type: integer, minimum: 0 }
|
||||||
|
enabled: { type: boolean }
|
||||||
|
SensitiveHit:
|
||||||
|
type: object
|
||||||
|
properties:
|
||||||
|
id: { type: integer, format: int64 }
|
||||||
|
user_id: { type: integer, format: int64 }
|
||||||
|
conversation_id: { type: integer, format: int64 }
|
||||||
|
word_id: { type: integer, format: int64 }
|
||||||
|
category: { $ref: "#/components/schemas/SensitiveCategory" }
|
||||||
|
action: { $ref: "#/components/schemas/SensitiveAction" }
|
||||||
|
excerpt: { type: string, maxLength: 128 }
|
||||||
|
real_ip: { type: string }
|
||||||
|
created_at: { type: string, format: date-time }
|
||||||
MusicUpsertRequest:
|
MusicUpsertRequest:
|
||||||
type: object
|
type: object
|
||||||
description: |
|
description: |
|
||||||
|
|||||||
157
internal/transport/http/handler/admin/moderation.go
Normal file
157
internal/transport/http/handler/admin/moderation.go
Normal file
@ -0,0 +1,157 @@
|
|||||||
|
package adminhandler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"strconv"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/gin-gonic/gin"
|
||||||
|
|
||||||
|
modlogic "spark-server/internal/module/moderation/logic"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
"spark-server/internal/pkg/reqctx"
|
||||||
|
"spark-server/internal/pkg/validate"
|
||||||
|
"spark-server/internal/transport/http/respond"
|
||||||
|
)
|
||||||
|
|
||||||
|
type ModerationHandler struct {
|
||||||
|
svc *modlogic.Service
|
||||||
|
}
|
||||||
|
|
||||||
|
func NewModeration(svc *modlogic.Service) *ModerationHandler {
|
||||||
|
return &ModerationHandler{svc: svc}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ModerationHandler) ready(c *gin.Context) bool {
|
||||||
|
if h == nil || h.svc == nil || !h.svc.Ready() {
|
||||||
|
Fail(c, http.StatusServiceUnavailable, "moderation 服务未就绪")
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ModerationHandler) ListWords(c *gin.Context) {
|
||||||
|
if !h.ready(c) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
page, _ := strconv.Atoi(c.DefaultQuery("page", "0"))
|
||||||
|
size, _ := strconv.Atoi(c.DefaultQuery("size", "50"))
|
||||||
|
cat, _ := strconv.Atoi(c.Query("category"))
|
||||||
|
var enabled *bool
|
||||||
|
if v := c.Query("enabled"); v == "1" || v == "true" {
|
||||||
|
enabled = ptrBool(true)
|
||||||
|
} else if v == "0" || v == "false" {
|
||||||
|
enabled = ptrBool(false)
|
||||||
|
}
|
||||||
|
list, total, err := h.svc.ListWords(c.Request.Context(), c.Query("q"), cat, enabled, page, size)
|
||||||
|
if err != nil {
|
||||||
|
Fail(c, http.StatusInternalServerError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
OK(c, gin.H{"items": list, "total": total})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ModerationHandler) CreateWord(c *gin.Context) {
|
||||||
|
if !h.ready(c) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var in modlogic.CreateWordInput
|
||||||
|
if err := respond.BindJSON(c, &in); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := validate.StructLang(reqctx.Lang(c.Request.Context()), &in); err != nil {
|
||||||
|
respond.JSONError(c, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
out, err := h.svc.CreateWord(c.Request.Context(), &in)
|
||||||
|
if err != nil {
|
||||||
|
failModeration(c, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
OK(c, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ModerationHandler) PatchWord(c *gin.Context) {
|
||||||
|
if !h.ready(c) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
|
||||||
|
if err != nil || id <= 0 {
|
||||||
|
Fail(c, http.StatusBadRequest, "invalid id")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var in modlogic.PatchWordInput
|
||||||
|
if err := respond.BindJSON(c, &in); err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.svc.PatchWord(c.Request.Context(), id, &in); err != nil {
|
||||||
|
failModeration(c, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
OK(c, gin.H{"ok": true})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ModerationHandler) DeleteWord(c *gin.Context) {
|
||||||
|
if !h.ready(c) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
id, err := strconv.ParseInt(c.Param("id"), 10, 64)
|
||||||
|
if err != nil || id <= 0 {
|
||||||
|
Fail(c, http.StatusBadRequest, "invalid id")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.svc.DeleteWord(c.Request.Context(), id); err != nil {
|
||||||
|
failModeration(c, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
OK(c, gin.H{"ok": true})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ModerationHandler) Reload(c *gin.Context) {
|
||||||
|
if !h.ready(c) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if err := h.svc.Reload(c.Request.Context()); err != nil {
|
||||||
|
Fail(c, http.StatusInternalServerError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
OK(c, gin.H{"ok": true})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h *ModerationHandler) ListHits(c *gin.Context) {
|
||||||
|
if !h.ready(c) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
page, _ := strconv.Atoi(c.DefaultQuery("page", "0"))
|
||||||
|
size, _ := strconv.Atoi(c.DefaultQuery("size", "50"))
|
||||||
|
uid, _ := strconv.ParseInt(c.Query("uid"), 10, 64)
|
||||||
|
cat, _ := strconv.Atoi(c.Query("category"))
|
||||||
|
var since, until *time.Time
|
||||||
|
if s := c.Query("since"); s != "" {
|
||||||
|
if t, err := time.Parse(time.RFC3339, s); err == nil {
|
||||||
|
since = &t
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s := c.Query("until"); s != "" {
|
||||||
|
if t, err := time.Parse(time.RFC3339, s); err == nil {
|
||||||
|
until = &t
|
||||||
|
}
|
||||||
|
}
|
||||||
|
list, total, err := h.svc.ListHits(c.Request.Context(), uid, cat, since, until, page, size)
|
||||||
|
if err != nil {
|
||||||
|
Fail(c, http.StatusInternalServerError, err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
OK(c, gin.H{"items": list, "total": total})
|
||||||
|
}
|
||||||
|
|
||||||
|
func failModeration(c *gin.Context, err error) {
|
||||||
|
if errors.Is(err, errs.ErrInvalidArgument) || errors.Is(err, errs.ErrDuplicate) || errors.Is(err, errs.ErrNotFound) {
|
||||||
|
respond.JSONError(c, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
Fail(c, http.StatusInternalServerError, err.Error())
|
||||||
|
}
|
||||||
|
|
||||||
|
func ptrBool(v bool) *bool { return &v }
|
||||||
@ -17,6 +17,7 @@ func registerAdminRoutes(r *gin.Engine, deps *EngineDeps) {
|
|||||||
}
|
}
|
||||||
mh := adminhandler.NewMusic(deps.Music)
|
mh := adminhandler.NewMusic(deps.Music)
|
||||||
ch := adminhandler.NewClient(deps.Client)
|
ch := adminhandler.NewClient(deps.Client)
|
||||||
|
modh := adminhandler.NewModeration(deps.Moderation)
|
||||||
|
|
||||||
// Shared user ops (popadmin Host 可将 /api/admin/users* 切到本服务)
|
// Shared user ops (popadmin Host 可将 /api/admin/users* 切到本服务)
|
||||||
if h != nil {
|
if h != nil {
|
||||||
@ -49,9 +50,23 @@ func registerAdminRoutes(r *gin.Engine, deps *EngineDeps) {
|
|||||||
}
|
}
|
||||||
registerMusicAdmin(g, mh)
|
registerMusicAdmin(g, mh)
|
||||||
registerClientAdmin(g, ch)
|
registerClientAdmin(g, ch)
|
||||||
|
registerModerationAdmin(g, modh)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func registerModerationAdmin(g *gin.RouterGroup, h *adminhandler.ModerationHandler) {
|
||||||
|
if h == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
sg := g.Group("/sensitive")
|
||||||
|
sg.GET("/words", h.ListWords)
|
||||||
|
sg.POST("/words", h.CreateWord)
|
||||||
|
sg.POST("/reload", h.Reload)
|
||||||
|
sg.PATCH("/words/:id", h.PatchWord)
|
||||||
|
sg.DELETE("/words/:id", h.DeleteWord)
|
||||||
|
sg.GET("/hits", h.ListHits)
|
||||||
|
}
|
||||||
|
|
||||||
func registerSparkAdmin(g *gin.RouterGroup, h *adminhandler.Handler) {
|
func registerSparkAdmin(g *gin.RouterGroup, h *adminhandler.Handler) {
|
||||||
g.GET("/api-doc", h.APIDoc)
|
g.GET("/api-doc", h.APIDoc)
|
||||||
g.GET("/ops-catalog", h.OpsCatalog)
|
g.GET("/ops-catalog", h.OpsCatalog)
|
||||||
|
|||||||
@ -19,6 +19,7 @@ import (
|
|||||||
friendlogic "spark-server/internal/module/friend/logic"
|
friendlogic "spark-server/internal/module/friend/logic"
|
||||||
grouplogic "spark-server/internal/module/group/logic"
|
grouplogic "spark-server/internal/module/group/logic"
|
||||||
msglogic "spark-server/internal/module/message/logic"
|
msglogic "spark-server/internal/module/message/logic"
|
||||||
|
modlogic "spark-server/internal/module/moderation/logic"
|
||||||
momentlogic "spark-server/internal/module/moment/logic"
|
momentlogic "spark-server/internal/module/moment/logic"
|
||||||
musiclogic "spark-server/internal/module/music/logic"
|
musiclogic "spark-server/internal/module/music/logic"
|
||||||
smslogic "spark-server/internal/module/sms/logic"
|
smslogic "spark-server/internal/module/sms/logic"
|
||||||
@ -60,6 +61,7 @@ type EngineDeps struct {
|
|||||||
AdminTokens *adminlogic.TokenService
|
AdminTokens *adminlogic.TokenService
|
||||||
Music *musiclogic.Service
|
Music *musiclogic.Service
|
||||||
Client *clientlogic.Service
|
Client *clientlogic.Service
|
||||||
|
Moderation *modlogic.Service
|
||||||
Health Health
|
Health Health
|
||||||
AuthLimiter *ratelimit.Limiter
|
AuthLimiter *ratelimit.Limiter
|
||||||
}
|
}
|
||||||
|
|||||||
@ -54,7 +54,15 @@ func (m *memManager) KickDevice(userID int64, deviceID uint64) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (m *memManager) KickUserBanned(userID int64) {
|
||||||
|
m.kickUserKey(userID, i18n.NotifyKickBanned)
|
||||||
|
}
|
||||||
|
|
||||||
func (m *memManager) KickUser(userID int64) {
|
func (m *memManager) KickUser(userID int64) {
|
||||||
|
m.kickUserKey(userID, i18n.NotifyKickDeleted)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m *memManager) kickUserKey(userID int64, key string) {
|
||||||
m.mu.Lock()
|
m.mu.Lock()
|
||||||
defer m.mu.Unlock()
|
defer m.mu.Unlock()
|
||||||
userMap := m.byUser[userID]
|
userMap := m.byUser[userID]
|
||||||
@ -62,7 +70,7 @@ func (m *memManager) KickUser(userID int64) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
for id, existing := range userMap {
|
for id, existing := range userMap {
|
||||||
reason := i18n.T(i18n.Parse(existing.Language()), i18n.NotifyKickDeleted, nil)
|
reason := i18n.T(i18n.Parse(existing.Language()), key, nil)
|
||||||
_ = existing.Send(&Packet{
|
_ = existing.Send(&Packet{
|
||||||
Command: protocol.CmdKick,
|
Command: protocol.CmdKick,
|
||||||
Body: protocol.EncodeKick(existing.RequestID(), reason),
|
Body: protocol.EncodeKick(existing.RequestID(), reason),
|
||||||
|
|||||||
@ -32,6 +32,7 @@ type Manager interface {
|
|||||||
GetUserSessions(userID int64) []Session
|
GetUserSessions(userID int64) []Session
|
||||||
KickDevice(userID int64, deviceID uint64)
|
KickDevice(userID int64, deviceID uint64)
|
||||||
KickUser(userID int64)
|
KickUser(userID int64)
|
||||||
|
KickUserBanned(userID int64)
|
||||||
SendToUserExceptDevice(ctx context.Context, userID int64, exceptDeviceID uint64, packet *Packet) error
|
SendToUserExceptDevice(ctx context.Context, userID int64, exceptDeviceID uint64, packet *Packet) error
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
35
migrations/0028_moderation.sql
Normal file
35
migrations/0028_moderation.sql
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
-- Sensitive words, hit audit, account send mute.
|
||||||
|
|
||||||
|
ALTER TABLE t_users
|
||||||
|
ADD COLUMN send_muted_until DATETIME(3) NULL
|
||||||
|
COMMENT '全站禁言截止时间，空或已过期表示可发消息';
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS t_sensitive_words (
|
||||||
|
id BIGINT NOT NULL AUTO_INCREMENT COMMENT '行 id，表内自增',
|
||||||
|
word_raw VARCHAR(64) NOT NULL COMMENT '运营录入原文，展示用',
|
||||||
|
word VARCHAR(64) NOT NULL COMMENT 'FoldAndIndex 后的词面，全局唯一，进自动机',
|
||||||
|
category TINYINT NOT NULL COMMENT '1涉黑 2涉毒 3涉黄 4涉赌 5涉政 6宗教 7辱骂 8广告 9其他',
|
||||||
|
action TINYINT NOT NULL COMMENT '1log 2replace 3reject 4hide 5mute 6disable',
|
||||||
|
mute_seconds INT NOT NULL DEFAULT 0 COMMENT 'action=mute 时的禁言秒数',
|
||||||
|
enabled TINYINT NOT NULL DEFAULT 1 COMMENT '1启用 0下线（不进自动机）',
|
||||||
|
created_at DATETIME(3) NOT NULL COMMENT '创建时间',
|
||||||
|
updated_at DATETIME(3) NOT NULL COMMENT '更新时间',
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
UNIQUE KEY uk_word (word)
|
||||||
|
) COMMENT='敏感词；发送热路径使用进程内自动机，本表为真源';
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS t_sensitive_hits (
|
||||||
|
id BIGINT NOT NULL COMMENT 'Snowflake',
|
||||||
|
user_id BIGINT NOT NULL COMMENT '发送者 uid',
|
||||||
|
conversation_id BIGINT NOT NULL DEFAULT 0 COMMENT '会话 id',
|
||||||
|
word_id BIGINT NOT NULL COMMENT 't_sensitive_words.id',
|
||||||
|
category TINYINT NOT NULL COMMENT '命中时的类别快照',
|
||||||
|
action TINYINT NOT NULL COMMENT '本行对应词的 action',
|
||||||
|
excerpt VARCHAR(128) NOT NULL DEFAULT '' COMMENT '命中片段，截断',
|
||||||
|
real_ip VARCHAR(64) NOT NULL DEFAULT '' COMMENT 'PROXY RealIP',
|
||||||
|
created_at DATETIME(3) NOT NULL COMMENT '命中时间',
|
||||||
|
PRIMARY KEY (id),
|
||||||
|
KEY idx_user_created (user_id, created_at),
|
||||||
|
KEY idx_created (created_at),
|
||||||
|
KEY idx_word (word_id)
|
||||||
|
) COMMENT='敏感词命中审计，供 Admin 查询';
|
||||||
22
migrations/0029_sensitive_words_seed.sql
Normal file
22
migrations/0029_sensitive_words_seed.sql
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
-- 敏感词种子（按 docs/moderation.md §4.3 默认处罚）。word 为 FoldAndIndex 后的词面，与入库规则一致。
|
||||||
|
-- 可重复执行：uk_word 冲突时仅刷新 updated_at。
|
||||||
|
|
||||||
|
INSERT INTO t_sensitive_words (word_raw, word, category, action, mute_seconds, enabled, created_at, updated_at)
|
||||||
|
VALUES
|
||||||
|
('黑社会', '黑社会', 1, 6, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('黑帮', '黑帮', 1, 6, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('冰毒', '冰毒', 2, 6, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('海洛因', '海洛因', 2, 6, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('色情', '色情', 3, 3, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('裸聊', '裸聊', 3, 3, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('赌博', '赌博', 4, 3, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('博彩', '博彩', 4, 3, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('分裂国家', '分裂国家', 5, 6, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('邪教', '邪教', 6, 3, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('傻逼', '傻逼', 7, 2, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('去死', '去死', 7, 2, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('恶意刷屏', '恶意刷屏', 7, 5, 3600, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('加微信', '加微信', 8, 1, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('代理加盟', '代理加盟', 8, 1, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3)),
|
||||||
|
('违禁测试', '违禁测试', 9, 3, 0, 1, CURRENT_TIMESTAMP(3), CURRENT_TIMESTAMP(3))
|
||||||
|
ON DUPLICATE KEY UPDATE updated_at = VALUES(updated_at);
|
||||||
Loading…
Reference in New Issue
Block a user