diff --git a/.cursor/skills/spark-im-v3/SKILL.md b/.cursor/skills/spark-im-v3/SKILL.md index 30fb772..193c051 100644 --- a/.cursor/skills/spark-im-v3/SKILL.md +++ b/.cursor/skills/spark-im-v3/SKILL.md @@ -95,7 +95,7 @@ Media: HTTP upload credential, message content is object key only. Frame max 64K Message types: text / image / voice / video / emoji / gif / file / location / system (`message/entity.Type` 1–9). `messages.content` stores serialized JSON bytes (BLOB), never a raw string. Closed type/status values: [enums.md](enums.md). TCP body is JSON envelope `{code,msg,request_id,data}`. AUTH request is `{"token","device_id","app_version","language"}` (not protobuf). -Public TLS is Nginx `:443` SNI: HTTP cert on loopback `:8443`, IM cert on loopback `:19001` then PROXY to gnet `:9090` (host `:9000` is rustfs). See `deploy/nginx/`. +Public TLS is Nginx `:443` SNI: HTTP cert on loopback `:8443` (`sparkapi` + `sparkr`/RustFS), IM cert on loopback `:19001` then PROXY to gnet `:9090` (host `:9000` is rustfs). See `deploy/nginx/`. AUTH/login rate-limit uses `reqctx.RealIP`. TaskPool workers recover panics. Deliver workers start with `RequeueStale` + `SETNX deliver:done:{msgId}`. ## When adding a feature diff --git a/README.md b/README.md index 409ee41..b4712b1 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,322 @@ # spark-server -Spark IM modular monolith. Architecture: [docs/architecture-v3.md](docs/architecture-v3.md). +Spark IM 服务端。Go 1.26 **模块化单体**(一台机器、一个进程)。公网只开 **443**,HTTP / IM / 对象存储用三个域名、Nginx SNI 分流。 -- Go 1.26 -- Gin HTTP on `127.0.0.1:8080`. JSON envelope `{"code","msg","request_id","data"}`. Docs: `GET /docs` (API), `/docs/enums`, `/docs/errors`. Spec: `GET /openapi.yaml`. -- Probes: `GET /healthz` liveness; `GET /readyz` readiness (HTTP 503 when MySQL/Redis/task pool/accepting are not ready). `env: prod` (or `SPARK_ENV=prod`) rejects empty MySQL DSN, empty Redis, and the default JWT secret. -- Logs: zap to console and/or rotating file (`log.console` / `log.file`). GORM SQL goes through the same zap. `request_id` is server-generated and returned in the response body (also as `X-Request-ID`). -- gnet TCP on loopback (local `127.0.0.1:8090`; ECS Docker `127.0.0.1:9090` because host `:9000` is rustfs). gnet v2.10 has no built-in TLS; Nginx terminates IM TLS and forwards PROXY+plaintext. Sample: `deploy/nginx/`. `make deploy` writes image/config/logs under `/data/spark-server/`. -- TCP Frame body is JSON `{code,msg,request_id,data}`. AUTH body: `{"token","device_id","app_version","language"}`. -- Public entry is Nginx `:443` only (SNI split) +仓库: +完整设计:[docs/architecture-v3.md](docs/architecture-v3.md) +在线协议:`GET /docs`(Swagger)、`/docs/enums`、`/docs/errors`,规范 `GET /openapi.yaml`。 -```bash -make start -make migrate -make lint +不上 Kafka、K8s、微服务。不要给 gnet 绑公网端口,不要在 EventLoop 里打 MySQL。 + +--- + +## 它做什么 + +即时通讯:登录、好友、群、发消息、增量同步、撤回删除、媒体直传。 + +| 层 | 职责 | +| --- | --- | +| 实时 | TCP 长连接(gnet)。在线推送 `MESSAGE`,心跳 30s,空闲 120s 断开 | +| 正文 | MySQL。Snowflake `MsgId` 是主键、排序、同步游标。没有 `ConversationSeq` | +| 会话列表 / 未读 | Redis ZSET。群热路径不写 MySQL inbox 行 | +| 扇出 | Redis `deliver:queue` + worker。`SEND_ACK` = MySQL COMMIT **且** 入队成功 | +| 媒体 | 客户端拿 Presigned URL 直传 RustFS。消息里只存 `object_key` | + +HTTP 和 TCP 调同一套 `internal/module/*/logic`。Gin handler 只做绑定和写信封。 + +--- + +## 生产入口 + +| 用途 | 域名 | 说明 | +| --- | --- | --- | +| HTTP API | `https://sparkapi.wyrs365.com` | REST `/v1/...`,文档 `/docs` | +| IM 长连接 | `sparkim.wyrs365.com:443` | TLS + 自定义 Frame,不是 WebSocket | +| 对象存储 | `https://sparkr.wyrs365.com` | RustFS S3 path-style,Presigned PUT/GET | + +本机开发:HTTP `http://127.0.0.1:8080`,TCP 明文 `127.0.0.1:8090`。 + +ECS 上进程只绑回环,由 Nginx 终结 TLS: + +```text +客户端 :443 (ssl_preread) + ├─ sparkapi.wyrs365.com → 127.0.0.1:8443 → Gin 127.0.0.1:18080 + ├─ sparkr.wyrs365.com → 127.0.0.1:8443 → RustFS 127.0.0.1:9000 + └─ sparkim.wyrs365.com → 127.0.0.1:19001 (IM TLS) + → PROXY → gnet 127.0.0.1:9090 ``` -`make start` runs `cmd/server` against `configs/config.yaml` (override with `make start CONFIG=path/to.yaml`). `make run` builds `bin/spark-server` first, then starts that binary. +主机 `:9000` 是 RustFS S3,所以 IM gnet 用 **9090**。安全组只放行 443。 -Agent conventions live in `.cursor/rules` and `.cursor/skills/spark-im-v3`. Lint conventions live in `.golangci.yaml`. +--- + +## 硬约束(不要改) + +| ID | 规则 | +| --- | --- | +| C1 | 消息正文在 MySQL。Snowflake `MsgId` 是 PK / 排序 / 同步游标。没有 ConversationSeq | +| C2 | gnet EventLoop 不调 MySQL / Redis / 对象存储 / 扇出。入站进 TaskPool;扇出走 `deliver:queue` | +| C3 | 保密靠 TLS。Frame 只有薄头 + 可选 gzip。没有应用层 ECDH/AES | +| C4 | 公网只有 443。gnet / Gin 绑回环 | +| C5 | `SEND_ACK` 必须在 COMMIT **且** deliver `Push` 成功之后。入队失败不 ACK | +| C6 | 限流 / 审计用 PROXY / `X-Real-IP` 的 `RealIP`,不用 gnet `RemoteAddr`(那是 127.0.0.1) | +| C7 | 会话列表 / 未读 / 预览在 Redis ZSET。群热路径不 upsert MySQL inbox | + +热路径禁止 `context.TODO()`。推送超时不踢连接。不要把媒体字节写进 `messages.content`。 + +--- + +## 统一响应 + +HTTP 与 TCP **应答 Body** 同一套信封。业务成败只看 `code` 字符串,不要用数字 `0` 比较。除 `/readyz` 未就绪是 HTTP 503 外,业务失败也是 HTTP 200。 + +```json +{"code":"0","msg":"ok","request_id":"3H8kPq2mN1R","data":{}} +``` + +| 字段 | 规则 | +| --- | --- | +| `code` | `"0"` 成功;其它失败。全表见 [/docs/errors](https://sparkapi.wyrs365.com/docs/errors) | +| `msg` | 展示文案,随语言变。成功一般为 `ok` | +| `request_id` | 服务端生成,**11 位 base62**(`0-9A-Za-z`)。客户端不用传。HTTP 同时写 Header `X-Request-ID` | +| `data` | 成功为业务对象;失败固定 `{}`。缺字段当空对象,不要当 `null` | + +`request_id` 和 TCP Frame 头里的 `RequestID`(uint64)不是一回事:后者是客户端填的配对号,服务端原样带回。 + +### HTTP 请求放哪 + +| 位置 | 内容 | +| --- | --- | +| Header | `Authorization: Bearer …`、`X-Device-ID`、`X-Language` 或 `Accept-Language`、`X-App-Version`、`Content-Type` | +| Body | POST/PATCH 业务字段。不要把 token / 语言 / device_id 塞进业务 JSON(登录 Body 带 `device_id`,刷新 Body 带 `refresh_token`) | +| Path | 资源 id:好友 uid、群号、MsgId | +| Query | GET 筛选 / 分页 | + +语言:`zh-CN`(缺省)/ `en` / `zh-TW`。网关写 `X-Real-IP`,客户端不要发。 + +--- + +## 登录 + +1. (可选)`POST /v1/auth/email-codes` Body `{"email"}`。 +2. `POST /v1/auth/login` 拿 `access_token` / `refresh_token`。`device_id` 由**客户端生成并持久化**,服务端不分配;写入 JWT `did`。 +3. 其它 HTTP:`Authorization: Bearer …`,可选 `X-Device-ID`(须等于登录时的 device_id)。 +4. 续期:`POST /v1/auth/token/refresh`,Body `{"refresh_token"}`,不要带 Bearer。 +5. 在线发消息、收推送、增量同步走 **TCP**。`POST /v1/messages` 只是 HTTP 调试通道,不要当 App 主路径。 + +测试号 `test01@spark.com`~`test10@spark.com` 可跳过发码,登录任意 `verify_code`。 + +`uid` 就是 Snowflake `users.id`。对外展示用可变的 `spark_id`,不要把 uid 写进对象存储路径。 + +--- + +## HTTP 资源 + +前缀 `/v1/{resource}`。 + +| 组 | 路径 | 说明 | +| --- | --- | --- | +| 探活 | `/healthz` `/readyz` | 存活;未就绪 HTTP 503 | +| 文档 | `/docs` `/docs/enums` `/docs/errors` `/openapi.yaml` | 无需登录 | +| 枚举 | `GET /v1/meta/enums` | 无需登录 | +| 认证 | `/v1/auth` | 发码、登录、刷新、登出 | +| 用户 | `/v1/users/me` | 资料 | +| 好友 | `/v1/friends` | 申请、同意、备注、置顶、拉黑、举报 | +| 群 | `/v1/groups` | 创建、资料、成员、邀请、踢人、转让、解散 | +| 文件 | `/v1/files` | 上传/下载凭证、批量、图片派生 | +| 收藏 | `/v1/favorites` | 收藏;媒体会 Copy 到 `perm/fav/` | +| 会话 | `/v1/conversations` | 列表、免打扰 | +| 消息 | `/v1/messages` | 历史 / 撤回 / 删除(主路径仍是 TCP) | + +--- + +## TCP 协议 + +不是 WebSocket、不是 HTTP、不是 Protobuf。对 `sparkim.wyrs365.com:443` 做 TLS(SNI 必须是该域名),之后写二进制 Frame。 + +### Frame(大端,固定 23 字节头) + +```text +offset 0 Magic uint16 0x5350(ASCII "SP") +offset 2 Version uint8 1 +offset 3 Command uint16 +offset 5 Flags uint16 bit0=1 表示 Body 是 gzip +offset 7 RequestID uint64 客户端配对,可 0;响应原样带回 +offset 15 SequenceID uint32 请求/响应配对;PUSH_ACK 必须带回推送帧的 SequenceID +offset 19 BodyLength uint32 Body 字节数(压缩后) +offset 23 Body bytes UTF-8 JSON +``` + +先读 23 字节,再按 `BodyLength` 读 Body。Magic 不对或总长超过 64KB 应断开。Body ≥ 512B 且 gzip 更小时可压缩。 + +**请求 Body 是业务 JSON,不要包信封。** 应答和 `MESSAGE` 推送才是 `{code,msg,request_id,data}`。失败时 Command 仍为请求 Command(发送失败不是 `SEND_ACK`)。 + +| 值 | 名 | 方向 | +| --- | --- | --- | +| 1 | AUTH | → 请求 `{"token","device_id","app_version","language"}` | +| 2 | HEARTBEAT | → 空 Body | +| 3 | GENERATE_MSG_ID | → 空;应答 `data.msg_id` | +| 4 | SEND_MESSAGE | → 与 HTTP 发消息同一 JSON | +| 5 | SEND_ACK | ← 仅 4 的成功应答 | +| 6 | MESSAGE | ← 推送 | +| 7 | PUSH_ACK | → 空;`SequenceID` = 推送帧的 SequenceID | +| 8 | READ_MESSAGE | → | +| 9 | RECALL_MESSAGE | → `{"msg_id"}` | +| 10 | SYNC_INBOX | → 会话列表增量 | +| 12 | SYNC_MESSAGES | → 某会话消息增量 | +| 14 | KICK | ← 同设备顶号 | +| 15 | DELETE_MESSAGE | → `{"msg_id"}` | + +除 AUTH / HEARTBEAT / PUSH_ACK 外须先 AUTH,否则 `40102`。 + +上线同步顺序:**先 `SYNC_INBOX`,再按会话 `SYNC_MESSAGES`**。游标是 Snowflake `MsgId` / inbox score,不是 ConversationSeq。 + +Java / Swift 示例:`docs/clients/`,OpenAPI 里也可点开。 + +--- + +## 发消息路径 + +必须按这个顺序(C5): + +1. TaskPool(禁止在 EventLoop 里做) +2. 权限(好友 / 拉黑 / 群禁言)——失败则不 INSERT +3. Snowflake MsgId(或客户端 MsgId,须在 ±5 分钟内) +4. MySQL 事务:消息表 + `conversations.last_msg_id` +5. `DeliverQueue.Push` +6. `SEND_ACK` + +第 5 步失败不 ACK。客户端用同一个 `client_msg_id` 重试。发送设备不回推 `MESSAGE`,其它设备必须推。同 `device_id` 顶旧连接,下发 `KICK`。 + +私聊与群消息分表存储。消息类型 1–9:文本 / 图片 / 语音 / 视频 / 表情 / gif / 文件 / 位置 / 系统。`content` 是 JSON 序列化后的 **bytes**,不是字符串。 + +--- + +## 媒体 `object_key` + +客户端 **不自己拼 key**。`POST /v1/files` 只传 `purpose` / `mime` / `size`(群头像再加 `group_id`),服务端签发 Presigned PUT。PUT 必须原样带返回的 `headers`(`Content-Type`、`Content-Length`、`If-None-Match: *`)。同一 key 第二次 PUT 为 412。 + +文件名是 **16 位 base62 随机串**(12 字节),不占用 Snowflake。路径不含 uid。 + +| purpose | object_key | 生命周期 | +| --- | --- | --- | +| `avatar` | `perm/avatar/{yyyyMMdd}/{name}.{ext}` | 永久 | +| `group_avatar` | `perm/group/{群号}/{name}.{ext}` | 永久 | +| `chat_image` | `chat/image/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 | +| `chat_voice` | `chat/voice/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 | +| `chat_video` | `chat/video/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 | +| `chat_gif` | `chat/gif/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 | +| `chat_file` | `chat/file/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 | + +日期 UTC `yyyyMMdd`。`ext` 由 mime 映射,未知为 `bin`。收藏复制到 `perm/fav/{yyyyMMdd}/{name}.{ext}`。 + +合法 key:以 `perm/` 或 `chat/` 开头;禁止 `..`、空白、`\`、前导 `/`;最长 512。 + +发图片:客户端只 PUT 原图,再 `POST /v1/files/variants` 让服务端出缩略图和预览。消息里只存 key。 + +进程访问 RustFS 用内网 `object_storage.endpoint`(Docker 里是 `rustfs:9000`),签给客户端的 URL 用 `public_endpoint`(`https://sparkr.wyrs365.com`)。 + +--- + +## 目录 + +```text +cmd/server 进程入口 +cmd/migrate 按序执行 migrations/*.sql +configs/config.yaml 本机开发配置 +deploy/docker/ 生产镜像、compose、deploy.sh +deploy/nginx/ 443 SNI(host/ 是这台 ECS 的 drop-in) +docs/architecture-v3.md 架构全文 +docs/clients/ TCP Java / Swift demo +internal/bootstrap 读配置、接线、生命周期 +internal/transport/http Gin;handler 只绑定 +internal/transport/tcp gnet Frame + TaskPool +internal/module/{biz}/entity 领域类型,无 I/O +internal/module/{biz}/model 一张表一个 Go 文件(GORM + COMMENT) +internal/module/{biz}/logic 一个功能一个文件,禁止 import model +internal/pkg errs / i18n / reply / reqctx / validate +internal/infrastructure MySQL / Redis / Snowflake / zap / RustFS +migrations/ SQL,改完必须在本机跑 migrate +``` + +`logic` 不 import `model`。接线在 `internal/bootstrap`。表和列必须有 COMMENT。 + +--- + +## 本机开发 + +需要:Go 1.26、MySQL 8、Redis。对象存储可空(上传接口会不可用)。 + +```bash +# 配置 configs/config.yaml(env: dev) +make migrate # 应用 migrations/*.sql +make start # go run ./cmd/server -config configs/config.yaml +# 或 +make run # 先编 bin/spark-server 再启动 +``` + +覆盖配置:`make start CONFIG=path/to.yaml`。 + +```bash +make test +make lint # funlen 100,lll 240 +``` + +探活: + +```bash +curl -sS http://127.0.0.1:8080/healthz +curl -sS http://127.0.0.1:8080/readyz +``` + +`env: prod`(或 `SPARK_ENV=prod`)会拒绝空 MySQL DSN、空 Redis、以及默认 JWT secret。 + +改了 `migrations/*.sql` 后自己跑 `make migrate`,不要留给部署。 + +--- + +## 部署(ECS) + +`make deploy` → `deploy/docker/deploy.sh`。默认 `root@47.236.66.9`,密钥 `~/.ssh/spark-ecs`。 + +主机目录: + +| 路径 | 内容 | +| --- | --- | +| `/data/spark-server/app` | 构建源码 | +| `/data/spark-server/configs/config.yaml` | 运行配置(来自 `deploy/docker/config.docker.yaml`) | +| `/data/spark-server/logs` | 进程日志 | +| `/data/spark-server/images/spark-server.tar.gz` | `docker save` | + +容器加入已有 `shell_default` 网络(mysql / redis / rustfs)。发布: + +- `127.0.0.1:18080` → 容器 HTTP 8080 +- `127.0.0.1:9090` → 容器 gnet 9090 + +Nginx drop-in:`deploy/nginx/host/`,安装脚本 `bash deploy/nginx/install-host.sh`。不要覆盖整份 `/etc/nginx/nginx.conf`。 + +Snowflake `snowflake_worker_id` 每进程唯一。时钟回拨会拒绝发号。 + +--- + +## 日志与 ID + +- zap 控制台和/或滚动文件(`log.console` / `log.file`)。GORM SQL 走同一套 zap。 +- HTTP `request_id`:11 位 base62,响应体 + `X-Request-ID`。 +- 对象文件名:16 位 base62,与 Snowflake 号段分开。 +- Snowflake 只给 uid、MsgId、会话 id、收藏行 id 等业务主键。 + +--- + +## 相关文档 + +| 文档 | 内容 | +| --- | --- | +| [docs/architecture-v3.md](docs/architecture-v3.md) | 架构全文 | +| `GET /docs` | OpenAPI / Swagger | +| `GET /docs/enums` | 可跳转枚举 | +| `GET /docs/errors` | 信封错误码 | +| [deploy/nginx/README.md](deploy/nginx/README.md) | 443 SNI | +| `docs/clients/` | TCP 客户端示例 | +| `.cursor/skills/spark-im-v3/` | 实现约定(给 Agent) | +| `.golangci.yaml` | 代码规范 | diff --git a/configs/config.yaml b/configs/config.yaml index 89c7364..4ec5ff4 100644 --- a/configs/config.yaml +++ b/configs/config.yaml @@ -84,6 +84,7 @@ message: object_storage: endpoint: "47.236.66.9:9000" + public_endpoint: "https://sparkr.wyrs365.com" access_key: "rustfs_spark" secret_key: 'kWvGtYT/~q+35q;3dSI<^P!MH+bgrpBg' bucket: "spark" diff --git a/deploy/docker/config.docker.yaml b/deploy/docker/config.docker.yaml index 7ded671..2952694 100644 --- a/deploy/docker/config.docker.yaml +++ b/deploy/docker/config.docker.yaml @@ -84,6 +84,7 @@ message: object_storage: endpoint: "rustfs:9000" + public_endpoint: "https://sparkr.wyrs365.com" access_key: "rustfs_spark" secret_key: 'kWvGtYT/~q+35q;3dSI<^P!MH+bgrpBg' bucket: "spark" diff --git a/deploy/docker/deploy.sh b/deploy/docker/deploy.sh index 04c6afc..111b198 100755 --- a/deploy/docker/deploy.sh +++ b/deploy/docker/deploy.sh @@ -33,6 +33,6 @@ rsync -az --delete \ "${SSH[@]}" "docker save spark-server:latest | gzip -c > $DATA/images/spark-server.tar.gz" "${SSH[@]}" "cd $REMOTE/deploy/docker && docker compose -f docker-compose.yaml up -d spark-server" "${SSH[@]}" "docker exec spark-server /usr/local/bin/spark-migrate -config /app/configs/config.yaml -dir /app/migrations" -"${SSH[@]}" "cp $REMOTE/deploy/nginx/host/spark-sni.conf /etc/nginx/stream.d/spark-sni.conf && nginx -t && systemctl reload nginx" +"${SSH[@]}" "cp $REMOTE/deploy/nginx/host/spark-sni.conf /etc/nginx/stream.d/spark-sni.conf && if [ -f /etc/letsencrypt/live/sparkr.wyrs365.com/fullchain.pem ]; then cp $REMOTE/deploy/nginx/host/sparkr.conf /etc/nginx/conf.d/sparkr.conf; fi && nginx -t && systemctl reload nginx" "${SSH[@]}" "docker ps --filter name=spark-server --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'" "${SSH[@]}" "curl -fsS http://127.0.0.1:18080/healthz && echo && curl -fsS http://127.0.0.1:18080/readyz && echo" diff --git a/deploy/nginx/README.md b/deploy/nginx/README.md index 05b242a..15bfed2 100644 --- a/deploy/nginx/README.md +++ b/deploy/nginx/README.md @@ -6,6 +6,8 @@ gnet v2 has no native TLS. IM certificates live here, not in spark-server `tcp.t Client --TLS--> :443 ssl_preread ├─ SNI sparkapi.wyrs365.com → 127.0.0.1:8443 (HTTP TLS) │ → proxy_pass http://127.0.0.1:18080 + X-Real-IP + ├─ SNI sparkr.wyrs365.com → 127.0.0.1:8443 (HTTP TLS) + │ → proxy_pass http://127.0.0.1:9000 RustFS S3 └─ SNI sparkim.wyrs365.com → 127.0.0.1:19001 (IM TLS) → PROXY protocol → 127.0.0.1:9090 gnet ``` @@ -13,6 +15,7 @@ Client --TLS--> :443 ssl_preread On this ECS, `:443` is shared with other wyrs HTTPS sites. Do **not** replace `/etc/nginx/nginx.conf`. Drop-ins: - `host/sparkapi.conf` → `/etc/nginx/conf.d/` +- `host/sparkr.conf` → `/etc/nginx/conf.d/` (RustFS S3; DNS A record `sparkr.wyrs365.com` → this host) - `host/spark-sni.conf` → `/etc/nginx/stream.d/` Existing `listen 443 ssl` vhosts move to `127.0.0.1:8443 ssl proxy_protocol` so SNI can take public 443. `00-spark-http.conf` sets `real_ip_header proxy_protocol` and `port_in_redirect off` so `$remote_addr` is the client and redirects stay on 443. @@ -20,9 +23,10 @@ Existing `listen 443 ssl` vhosts move to `127.0.0.1:8443 ssl proxy_protocol` so spark-server (Docker): - HTTP `127.0.0.1:18080` → container `:8080` -- TCP `127.0.0.1:9090` with `proxy_protocol: true` → container `:9090` (host `:9000` is rustfs S3) +- TCP `127.0.0.1:9090` with `proxy_protocol: true` → container `:9090` (host `:9000` is rustfs S3; bind `127.0.0.1:9000`, not public) - Image / config / logs: `/data/spark-server/{images,configs,logs}` - Public security group: 443 only +- Clients upload/download at `https://sparkr.wyrs365.com` (path-style). spark-server talks to `rustfs:9000` on the Docker network and signs URLs with `object_storage.public_endpoint`. ```bash bash deploy/nginx/install-host.sh diff --git a/deploy/nginx/host/spark-acme.conf b/deploy/nginx/host/spark-acme.conf index c33e359..c6e1e43 100644 --- a/deploy/nginx/host/spark-acme.conf +++ b/deploy/nginx/host/spark-acme.conf @@ -3,7 +3,7 @@ server { listen 80; listen [::]:80; - server_name sparkapi.wyrs365.com sparkim.wyrs365.com; + server_name sparkapi.wyrs365.com sparkim.wyrs365.com sparkr.wyrs365.com; location /.well-known/acme-challenge/ { root /var/www/certbot; diff --git a/deploy/nginx/host/spark-sni.conf b/deploy/nginx/host/spark-sni.conf index 38cc23a..a23bc92 100644 --- a/deploy/nginx/host/spark-sni.conf +++ b/deploy/nginx/host/spark-sni.conf @@ -1,6 +1,6 @@ # Spark IM + shared HTTPS on :443 (ssl_preread). # sparkim.wyrs365.com → IM TLS terminator → PROXY plaintext to gnet :9090 -# default SNI → existing HTTP sites on 127.0.0.1:8443 (proxy_protocol restores RealIP) +# default SNI → 127.0.0.1:8443 HTTP TLS (sparkapi / sparkr / other wyrs sites; PROXY restores RealIP) map $ssl_preread_server_name $spark_sni_backend { sparkim.wyrs365.com 127.0.0.1:19001; diff --git a/deploy/nginx/host/sparkr.conf b/deploy/nginx/host/sparkr.conf new file mode 100644 index 0000000..fe24bac --- /dev/null +++ b/deploy/nginx/host/sparkr.conf @@ -0,0 +1,58 @@ +# Spark RustFS S3 — sparkr.wyrs365.com +# Public :443 is stream ssl_preread (see stream.d/spark-sni.conf). +# TLS terminates on loopback :8443; proxy HTTP to Docker rustfs on :9000. +# Do not rewrite the URI: SigV4 signs host + path. + +upstream spark_rustfs { + server 127.0.0.1:9000; + keepalive 32; +} + +server { + listen 80; + listen [::]:80; + server_name sparkr.wyrs365.com; + + location /.well-known/acme-challenge/ { + root /var/www/certbot; + } + + location / { + return 301 https://$host$request_uri; + } +} + +server { + listen 127.0.0.1:8443 ssl proxy_protocol; + listen [::1]:8443 ssl proxy_protocol; + http2 on; + server_name sparkr.wyrs365.com; + + ssl_certificate /etc/letsencrypt/live/sparkr.wyrs365.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/sparkr.wyrs365.com/privkey.pem; + ssl_session_timeout 1d; + ssl_session_cache shared:SparkSSL:10m; + + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; + add_header X-Content-Type-Options nosniff always; + + ignore_invalid_headers off; + client_max_body_size 100m; + proxy_buffering off; + proxy_request_buffering off; + proxy_connect_timeout 300s; + proxy_send_timeout 300s; + proxy_read_timeout 300s; + proxy_http_version 1.1; + chunked_transfer_encoding off; + + location / { + proxy_pass http://spark_rustfs; + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto https; + proxy_set_header Connection ""; + proxy_cache_convert_head off; + } +} diff --git a/deploy/nginx/install-host.sh b/deploy/nginx/install-host.sh index 51a9371..3fa2732 100755 --- a/deploy/nginx/install-host.sh +++ b/deploy/nginx/install-host.sh @@ -5,33 +5,61 @@ set -euo pipefail ROOT="$(cd "$(dirname "$0")" && pwd)" HOST_DIR="$ROOT/host" BACKUP="/root/nginx-backup-$(date +%Y%m%d%H%M%S)" +DOMAINS=(sparkapi.wyrs365.com sparkim.wyrs365.com sparkr.wyrs365.com) -need_certs() { - [[ -f /etc/letsencrypt/live/sparkapi.wyrs365.com/fullchain.pem ]] \ - && [[ -f /etc/letsencrypt/live/sparkim.wyrs365.com/fullchain.pem ]] +cert_ok() { + [[ -f "/etc/letsencrypt/live/$1/fullchain.pem" ]] } -install_acme() { +all_certs_ok() { + local d + for d in "${DOMAINS[@]}"; do + cert_ok "$d" || return 1 + done +} + +install_acme_domain() { + local d=$1 mkdir -p /var/www/certbot - cp "$HOST_DIR/spark-acme.conf" /etc/nginx/conf.d/spark-acme.conf + cat > "/etc/nginx/conf.d/spark-acme-${d}.conf" < /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK' @@ -69,12 +99,11 @@ mkdir -p "$BACKUP" cp -a /etc/nginx "$BACKUP/nginx" echo "backup $BACKUP" -if ! need_certs; then - install_acme - issue_certs +if ! all_certs_ok; then + issue_missing_certs fi -if ! need_certs; then +if ! all_certs_ok; then echo "certificates missing" >&2 exit 1 fi diff --git a/internal/bootstrap/app.go b/internal/bootstrap/app.go index 4d7892b..b110fdf 100644 --- a/internal/bootstrap/app.go +++ b/internal/bootstrap/app.go @@ -138,6 +138,7 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge authLimit := ratelimit.New(cfg.RateLimit.AuthPerSecond) objects, err := objectstore.Open(&objectstore.Options{ Endpoint: cfg.ObjectStorage.Endpoint, + PublicEndpoint: cfg.ObjectStorage.PublicEndpoint, AccessKey: cfg.ObjectStorage.AccessKey, SecretKey: cfg.ObjectStorage.SecretKey, Bucket: cfg.ObjectStorage.Bucket, @@ -166,7 +167,7 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge friends := wireFriends(idgen, db, users, deliverQueue, chat) groups := wireGroups(idgen, db, users, friends, deliverQueue, objects, chat) messages := wireMessages(idgen, friends, groups, deliverQueue, chat) - files := wireFiles(cfg, objects, idgen, groups) + files := wireFiles(cfg, objects, groups) favorites := wireFavorites(idgen, db, objects, chat) app.tcp = tcp.NewServer(log, pool, sessions, &tcp.Options{ Addr: cfg.Server.TCP.Addr, @@ -276,11 +277,11 @@ func wireGroups(idgen *id.Generator, db *mysql.DB, users *userlogic.Service, fri return grouplogic.NewService(repo, users, friends, idgen, chat.msgs, q, objects) } -func wireFiles(cfg *Config, store *objectstore.Store, idgen *id.Generator, groups *grouplogic.Service) *uploadlogic.Service { - if store == nil || !store.Ready() || idgen == nil { +func wireFiles(cfg *Config, store *objectstore.Store, groups *grouplogic.Service) *uploadlogic.Service { + if store == nil || !store.Ready() { return nil } - return uploadlogic.NewService(store, idgen, uploadlogic.Options{ + return uploadlogic.NewService(store, uploadlogic.Options{ TTL: cfg.ObjectStorage.PresignTTL, Limit: ratelimit.New(cfg.RateLimit.UploadPerSecond), Groups: groups, diff --git a/internal/bootstrap/config.go b/internal/bootstrap/config.go index fe88b35..e2e4d1b 100644 --- a/internal/bootstrap/config.go +++ b/internal/bootstrap/config.go @@ -118,6 +118,7 @@ type RateLimitConfig struct { type ObjectStorageConfig struct { Endpoint string `yaml:"endpoint"` + PublicEndpoint string `yaml:"public_endpoint"` AccessKey string `yaml:"access_key"` SecretKey string `yaml:"secret_key"` Bucket string `yaml:"bucket"` @@ -229,6 +230,9 @@ func applyDefaults(cfg *Config) { if v := os.Getenv("SPARK_S3_ENDPOINT"); v != "" { cfg.ObjectStorage.Endpoint = v } + if v := os.Getenv("SPARK_S3_PUBLIC_ENDPOINT"); v != "" { + cfg.ObjectStorage.PublicEndpoint = v + } if v := os.Getenv("SPARK_S3_ACCESS_KEY"); v != "" { cfg.ObjectStorage.AccessKey = v } diff --git a/internal/infrastructure/objectstore/ops.go b/internal/infrastructure/objectstore/ops.go index fc93ae6..02ca246 100644 --- a/internal/infrastructure/objectstore/ops.go +++ b/internal/infrastructure/objectstore/ops.go @@ -18,13 +18,24 @@ import ( "spark-server/internal/pkg/errs" ) -const maxObjectBytes = 4 << 20 +const ( + maxGetBytes = 32 << 20 + maxPutBytes = 4 << 20 +) -func (s *Store) PresignPut(ctx context.Context, key, contentType string, ttl time.Duration) (string, error) { +func (s *Store) PresignPut(ctx context.Context, key, contentType string, size int64, ttl time.Duration) (string, error) { if !s.Ready() { return "", errs.ErrAvailable } - in := &s3.PutObjectInput{Bucket: aws.String(s.bucket), Key: aws.String(key)} + if size < 1 { + return "", errs.ErrInvalidArgument + } + in := &s3.PutObjectInput{ + Bucket: aws.String(s.bucket), + Key: aws.String(key), + ContentLength: aws.Int64(size), + IfNoneMatch: aws.String("*"), + } if contentType != "" { in.ContentType = aws.String(contentType) } @@ -61,11 +72,11 @@ func (s *Store) Get(ctx context.Context, key string) ([]byte, error) { return nil, mapObjectErr(err) } defer func() { _ = out.Body.Close() }() - data, err := io.ReadAll(io.LimitReader(out.Body, maxObjectBytes+1)) + data, err := io.ReadAll(io.LimitReader(out.Body, maxGetBytes+1)) if err != nil { return nil, mapObjectErr(err) } - if len(data) > maxObjectBytes { + if len(data) > maxGetBytes { return nil, errs.ErrInvalidArgument } return data, nil @@ -75,7 +86,7 @@ func (s *Store) Put(ctx context.Context, key string, body []byte, contentType st if !s.Ready() { return errs.ErrAvailable } - if len(body) == 0 || len(body) > maxObjectBytes { + if len(body) == 0 || len(body) > maxPutBytes { return errs.ErrInvalidArgument } in := &s3.PutObjectInput{ diff --git a/internal/infrastructure/objectstore/store.go b/internal/infrastructure/objectstore/store.go index 31310d7..63b9ce3 100644 --- a/internal/infrastructure/objectstore/store.go +++ b/internal/infrastructure/objectstore/store.go @@ -29,6 +29,7 @@ type Store struct { // Options configure Open. Empty Endpoint skips the connection. type Options struct { Endpoint string + PublicEndpoint string AccessKey string SecretKey string Bucket string @@ -42,7 +43,7 @@ func Open(o *Options) (*Store, error) { if o == nil || strings.TrimSpace(o.Endpoint) == "" { return &Store{}, nil } - host, secure, err := parseEndpoint(o.Endpoint, o.UseSSL) + client, presignBase, err := newClients(o) if err != nil { return nil, err } @@ -50,19 +51,14 @@ func Open(o *Options) (*Store, error) { if bucket == "" { bucket = "spark" } + s := &Store{client: client, presign: s3.NewPresignClient(presignBase), bucket: bucket, log: o.Log} + if s.log == nil { + s.log = zap.NewNop() + } region := strings.TrimSpace(o.Region) if region == "" { region = defaultRegion } - scheme := "http" - if secure { - scheme = "https" - } - client := newS3Client(region, scheme+"://"+host, o.AccessKey, o.SecretKey) - s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: bucket, log: o.Log} - if s.log == nil { - s.log = zap.NewNop() - } ctx, cancel := context.WithTimeout(context.Background(), defaultTimeout) defer cancel() if err := s.ensureBucket(ctx, region, o.ChatExpireDays); err != nil { @@ -83,6 +79,26 @@ func (s *Store) Ping(ctx context.Context) error { return err } +func newClients(o *Options) (*s3.Client, *s3.Client, error) { + region := strings.TrimSpace(o.Region) + if region == "" { + region = defaultRegion + } + host, secure, err := parseEndpoint(o.Endpoint, o.UseSSL) + if err != nil { + return nil, nil, err + } + client := newS3Client(region, endpointURL(host, secure), o.AccessKey, o.SecretKey) + if pub := strings.TrimSpace(o.PublicEndpoint); pub != "" { + phost, psecure, err := parseEndpoint(pub, true) + if err != nil { + return nil, nil, err + } + return client, newS3Client(region, endpointURL(phost, psecure), o.AccessKey, o.SecretKey), nil + } + return client, client, nil +} + func newS3Client(region, baseURL, access, secret string) *s3.Client { return s3.New(s3.Options{ Region: region, @@ -94,8 +110,15 @@ func newS3Client(region, baseURL, access, secret string) *s3.Client { }) } +func endpointURL(host string, secure bool) string { + if secure { + return "https://" + host + } + return "http://" + host +} + func parseEndpoint(raw string, useSSL bool) (host string, secure bool, err error) { - raw = strings.TrimSpace(raw) + raw = strings.TrimSpace(strings.TrimRight(raw, "/")) switch { case raw == "": return "", false, fmt.Errorf("objectstore: empty endpoint") diff --git a/internal/infrastructure/objectstore/store_test.go b/internal/infrastructure/objectstore/store_test.go index 0adbc78..fc77a0b 100644 --- a/internal/infrastructure/objectstore/store_test.go +++ b/internal/infrastructure/objectstore/store_test.go @@ -40,13 +40,17 @@ func TestOpenEmpty(t *testing.T) { func TestPresignLocal(t *testing.T) { client := newS3Client(defaultRegion, "http://127.0.0.1:9000", "ak", "sk") s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: "spark"} - u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", time.Hour) + u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 1024, time.Hour) if err != nil { t.Fatal(err) } if !strings.Contains(u, "127.0.0.1:9000/spark/chat/a.jpg") { t.Fatalf("%s", u) } + signed := strings.ToLower(u) + if !strings.Contains(signed, "content-length") || !strings.Contains(signed, "if-none-match") { + t.Fatalf("unsigned size/overwrite headers: %s", u) + } g, err := s.PresignGet(t.Context(), "perm/avatar/1.jpg", time.Minute) if err != nil { t.Fatal(err) @@ -56,6 +60,37 @@ func TestPresignLocal(t *testing.T) { } } +func TestPresignPutRejectsZeroSize(t *testing.T) { + client := newS3Client(defaultRegion, "http://127.0.0.1:9000", "ak", "sk") + s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: "spark"} + if _, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 0, time.Hour); !errors.Is(err, errs.ErrInvalidArgument) { + t.Fatalf("%v", err) + } +} + +func TestPresignPublicEndpoint(t *testing.T) { + internal, public, err := newClients(&Options{ + Endpoint: "http://rustfs:9000", + PublicEndpoint: "https://sparkr.wyrs365.com/", + AccessKey: "ak", + SecretKey: "sk", + }) + if err != nil { + t.Fatal(err) + } + s := &Store{client: internal, presign: s3.NewPresignClient(public), bucket: "spark"} + u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 1024, time.Hour) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(u, "https://sparkr.wyrs365.com/spark/chat/a.jpg") { + t.Fatalf("%s", u) + } + if strings.Contains(u, "rustfs:9000") { + t.Fatalf("internal host leaked: %s", u) + } +} + func TestGetPutNotReady(t *testing.T) { s, err := Open(&Options{}) if err != nil { diff --git a/internal/module/favorite/logic/copy.go b/internal/module/favorite/logic/copy.go index a72df2a..90fab90 100644 --- a/internal/module/favorite/logic/copy.go +++ b/internal/module/favorite/logic/copy.go @@ -7,9 +7,10 @@ import ( msgentity "spark-server/internal/module/message/entity" uploadentity "spark-server/internal/module/upload/entity" "spark-server/internal/pkg/errs" + "spark-server/internal/pkg/reqctx" ) -func (s *Service) copyMedia(ctx context.Context, userID int64, payload *msgentity.Content) ([]string, error) { +func (s *Service) copyMedia(ctx context.Context, payload *msgentity.Content) ([]string, error) { keys := payload.ObjectKeys() if len(keys) == 0 { return nil, nil @@ -24,12 +25,7 @@ func (s *Service) copyMedia(ctx context.Context, userID int64, payload *msgentit if _, ok := mapping[src]; ok { continue } - objectID, err := s.ids.NextID() - if err != nil { - s.deleteKeys(ctx, copied) - return nil, err - } - dst := uploadentity.BuildFavKey(userID, objectID, day, uploadentity.ExtOfKey(src)) + dst := uploadentity.BuildFavKey(reqctx.NewObjectName(), day, uploadentity.ExtOfKey(src)) if err := s.objects.Copy(ctx, src, dst); err != nil { s.deleteKeys(ctx, copied) return nil, err diff --git a/internal/module/favorite/logic/create.go b/internal/module/favorite/logic/create.go index 2c08c30..a0350ed 100644 --- a/internal/module/favorite/logic/create.go +++ b/internal/module/favorite/logic/create.go @@ -47,7 +47,7 @@ func (s *Service) Create(ctx context.Context, userID int64, in *CreateInput) (*I if err != nil { return nil, err } - copied, err := s.copyMedia(ctx, userID, payload) + copied, err := s.copyMedia(ctx, payload) if err != nil { return nil, err } diff --git a/internal/module/favorite/logic/favorite_test.go b/internal/module/favorite/logic/favorite_test.go index 970cfd5..a57df9f 100644 --- a/internal/module/favorite/logic/favorite_test.go +++ b/internal/module/favorite/logic/favorite_test.go @@ -56,8 +56,12 @@ func TestFavoriteCopiesMediaAndExpires(t *testing.T) { t.Fatal(err) } src := "chat/20260917/1/99.jpg" + thumb := "chat/20260917/1/t.jpg" + orig := "chat/20260917/1/o.jpg" objs.keys[src] = true - raw, err := msgentity.NewImage(&msgentity.Image{Key: src, Mime: "image/jpeg"}).Marshal() + objs.keys[thumb] = true + objs.keys[orig] = true + raw, err := msgentity.NewImage(&msgentity.Image{Key: src, ThumbKey: thumb, OrigKey: orig, Mime: "image/jpeg"}).Marshal() if err != nil { t.Fatal(err) } @@ -76,10 +80,10 @@ func TestFavoriteCopiesMediaAndExpires(t *testing.T) { if err != nil { t.Fatal(err) } - if item.MsgID != msgID || !strings.Contains(string(item.Content), "perm/fav/1/") { + if item.MsgID != msgID || !strings.Contains(string(item.Content), "perm/fav/") { t.Fatalf("%+v", item) } - if strings.Contains(string(item.Content), src) { + if strings.Contains(string(item.Content), src) || strings.Contains(string(item.Content), orig) { t.Fatal("still points at chat key") } again, err := svc.Create(ctx, 1, &CreateInput{MsgID: msgID}) diff --git a/internal/module/group/logic/avatar.go b/internal/module/group/logic/avatar.go index 105826e..57159cc 100644 --- a/internal/module/group/logic/avatar.go +++ b/internal/module/group/logic/avatar.go @@ -13,6 +13,7 @@ import ( "spark-server/internal/module/group/entity" uploadentity "spark-server/internal/module/upload/entity" userentity "spark-server/internal/module/user/entity" + "spark-server/internal/pkg/reqctx" ) const collageJPEG = "image/jpeg" @@ -63,10 +64,7 @@ func (s *Service) putAvatar(ctx context.Context, g *entity.Group, raw []byte, mi if g == nil || len(raw) == 0 { return false } - key := uploadentity.PrefixGroup + strconv.FormatInt(g.ID, 10) + "/default." + ext - if objectID, err := s.ids.NextID(); err == nil { - key = uploadentity.BuildObjectKey(uploadentity.PurposeGroupAvatar, g.ID, objectID, "", ext) - } + key := uploadentity.BuildObjectKey(uploadentity.PurposeGroupAvatar, g.ID, reqctx.NewObjectName(), "", ext) if err := s.objects.Put(ctx, key, raw, mime); err != nil { return false } diff --git a/internal/module/message/entity/content.go b/internal/module/message/entity/content.go index d3bf3be..6ff51be 100644 --- a/internal/module/message/entity/content.go +++ b/internal/module/message/entity/content.go @@ -121,7 +121,7 @@ func (c *Content) Validate() error { case TypeText: return validateText(c.Text) case TypeImage: - return validateMediaKey("image", c.Image.Key) + return validateImage(c.Image) case TypeVoice: return validateVoice(c.Voice) case TypeVideo: @@ -220,6 +220,16 @@ func validateMentions(c *Content) error { return nil } +func validateImage(img *Image) error { + if err := validateMediaKey("image", img.Key); err != nil { + return err + } + if img.Size < 0 || img.OrigSize < 0 || img.Width < 0 || img.Height < 0 || img.OrigW < 0 || img.OrigH < 0 { + return fmt.Errorf("content: image size") + } + return nil +} + func validateMediaKey(kind, key string) error { if strings.TrimSpace(key) == "" { return fmt.Errorf("content: %s key empty", kind) @@ -279,14 +289,21 @@ func (c *Content) ObjectKeys() []string { var keys []string add := func(k string) { k = strings.TrimSpace(k) - if k != "" { - keys = append(keys, k) + if k == "" { + return } + for _, existing := range keys { + if existing == k { + return + } + } + keys = append(keys, k) } switch { case c.Image != nil: add(c.Image.Key) add(c.Image.ThumbKey) + add(c.Image.OrigKey) case c.Voice != nil: add(c.Voice.Key) case c.Video != nil: @@ -316,6 +333,7 @@ func (c *Content) ReplaceObjectKeys(mapping map[string]string) { if c.Image != nil { repl(&c.Image.Key) repl(&c.Image.ThumbKey) + repl(&c.Image.OrigKey) } if c.Voice != nil { repl(&c.Voice.Key) diff --git a/internal/module/message/entity/content_test.go b/internal/module/message/entity/content_test.go index 283d5ba..ab43b37 100644 --- a/internal/module/message/entity/content_test.go +++ b/internal/module/message/entity/content_test.go @@ -80,6 +80,9 @@ func TestRejectsRawStringAndEmptyKey(t *testing.T) { if _, err := NewImage(&Image{}).Marshal(); err == nil || !strings.Contains(err.Error(), "key") { t.Fatalf("image key: %v", err) } + if _, err := NewImage(&Image{Key: "k", OrigSize: -1}).Marshal(); err == nil { + t.Fatal("negative orig_size") + } bad := &Content{Type: TypeText, Text: &Text{Body: "x"}, Image: &Image{Key: "k"}} if _, err := bad.Marshal(); err == nil { t.Fatal("two payloads") @@ -122,13 +125,21 @@ func TestMentionsRoundTrip(t *testing.T) { } func TestObjectKeysAndReplace(t *testing.T) { - img := NewImage(&Image{Key: "chat/a.jpg", ThumbKey: "chat/t.jpg"}) + img := NewImage(&Image{Key: "chat/a.jpg", ThumbKey: "chat/t.jpg", OrigKey: "chat/o.jpg", OrigSize: 4 << 20}) keys := img.ObjectKeys() - if len(keys) != 2 || keys[0] != "chat/a.jpg" { + if len(keys) != 3 || keys[0] != "chat/a.jpg" || keys[2] != "chat/o.jpg" { t.Fatalf("%v", keys) } - img.ReplaceObjectKeys(map[string]string{"chat/a.jpg": "perm/fav/1/x.jpg", "chat/t.jpg": "perm/fav/1/y.jpg"}) - if img.Image.Key != "perm/fav/1/x.jpg" || img.Image.ThumbKey != "perm/fav/1/y.jpg" { + img.ReplaceObjectKeys(map[string]string{ + "chat/a.jpg": "perm/fav/1/x.jpg", + "chat/t.jpg": "perm/fav/1/y.jpg", + "chat/o.jpg": "perm/fav/1/z.jpg", + }) + if img.Image.Key != "perm/fav/1/x.jpg" || img.Image.ThumbKey != "perm/fav/1/y.jpg" || img.Image.OrigKey != "perm/fav/1/z.jpg" { t.Fatalf("%+v", img.Image) } + same := NewImage(&Image{Key: "chat/a.jpg", OrigKey: "chat/a.jpg"}) + if got := same.ObjectKeys(); len(got) != 1 { + t.Fatalf("dedupe %v", got) + } } diff --git a/internal/module/message/entity/payload.go b/internal/module/message/entity/payload.go index d987e27..6e52782 100644 --- a/internal/module/message/entity/payload.go +++ b/internal/module/message/entity/payload.go @@ -5,7 +5,10 @@ type Text struct { Body string `json:"body"` } -// Image references an uploaded object. Bytes themselves are not stored in messages.content. +// Image references uploaded objects. Bytes are not stored in messages.content. +// WeChat-style: bubble uses thumb_key, tap-to-view uses key (compressed), +// 「查看原图」uses orig_key when the sender opted in. +// Clients PUT JPEG/PNG only (convert HEIC/WebP locally); the server does not transcode. type Image struct { Key string `json:"key"` Mime string `json:"mime,omitempty"` @@ -13,6 +16,10 @@ type Image struct { Height int `json:"h,omitempty"` Size int64 `json:"size,omitempty"` ThumbKey string `json:"thumb_key,omitempty"` + OrigKey string `json:"orig_key,omitempty"` + OrigSize int64 `json:"orig_size,omitempty"` + OrigW int `json:"orig_w,omitempty"` + OrigH int `json:"orig_h,omitempty"` } // Voice is an audio clip in object storage. diff --git a/internal/module/upload/entity/credential.go b/internal/module/upload/entity/credential.go index 90ed80f..3e9604b 100644 --- a/internal/module/upload/entity/credential.go +++ b/internal/module/upload/entity/credential.go @@ -9,6 +9,11 @@ type Credential struct { ExpiresIn int64 `json:"expires_in"` } +// CredentialSet is one HTTP round-trip for up to three PUTs (thumb / preview / orig). +type CredentialSet struct { + Items []Credential `json:"items"` +} + type Download struct { ObjectKey string `json:"object_key"` DownloadURL string `json:"download_url"` diff --git a/internal/module/upload/entity/purpose.go b/internal/module/upload/entity/purpose.go index c30a9ab..a86c511 100644 --- a/internal/module/upload/entity/purpose.go +++ b/internal/module/upload/entity/purpose.go @@ -16,11 +16,16 @@ const ( PurposeChatGIF = "chat_gif" PurposeChatFile = "chat_file" - PrefixPerm = "perm/" - PrefixChat = "chat/" - PrefixAvatar = "perm/avatar/" - PrefixGroup = "perm/group/" - PrefixFav = "perm/fav/" + PrefixPerm = "perm/" + PrefixChat = "chat/" + PrefixAvatar = "perm/avatar/" + PrefixGroup = "perm/group/" + PrefixFav = "perm/fav/" + PrefixChatImage = "chat/image/" + PrefixChatVoice = "chat/voice/" + PrefixChatVideo = "chat/video/" + PrefixChatGIF = "chat/gif/" + PrefixChatFile = "chat/file/" MethodPUT = "PUT" MethodGET = "GET" @@ -28,7 +33,8 @@ const ( const ( maxAvatar = 2 << 20 - maxImage = 10 << 20 + maxImage = 32 << 20 + maxGIF = 10 << 20 maxVoice = 16 << 20 maxVideo = 100 << 20 maxFile = 50 << 20 @@ -75,25 +81,46 @@ func ExtOfKey(key string) string { return ext } -func BuildObjectKey(purpose string, ownerID, objectID int64, day, ext string) string { +func BuildObjectKey(purpose string, ownerID int64, name, day, ext string) string { if ext == "" { ext = extDefault } + if name == "" { + name = "0" + } switch purpose { case PurposeAvatar: - return fmt.Sprintf("%s%d/%s/%d.%s", PrefixAvatar, ownerID, day, objectID, ext) + return fmt.Sprintf("%s%s/%s.%s", PrefixAvatar, day, name, ext) case PurposeGroupAvatar: - return fmt.Sprintf("%s%d/%d.%s", PrefixGroup, ownerID, objectID, ext) + return fmt.Sprintf("%s%d/%s.%s", PrefixGroup, ownerID, name, ext) default: - return fmt.Sprintf("%s%s/%d/%d.%s", PrefixChat, day, ownerID, objectID, ext) + return fmt.Sprintf("%s%s/%s/%s.%s", PrefixChat, chatKind(purpose), day, name, ext) } } -func BuildFavKey(userID, objectID int64, day, ext string) string { +func BuildFavKey(name, day, ext string) string { if ext == "" { ext = extDefault } - return fmt.Sprintf("%s%d/%s/%d.%s", PrefixFav, userID, day, objectID, ext) + if name == "" { + name = "0" + } + return fmt.Sprintf("%s%s/%s.%s", PrefixFav, day, name, ext) +} + +func chatKind(purpose string) string { + switch purpose { + case PurposeChatImage: + return "image" + case PurposeChatVoice: + return "voice" + case PurposeChatVideo: + return "video" + case PurposeChatGIF: + return "gif" + default: + return "file" + } } func ValidObjectKey(key string) bool { @@ -110,13 +137,13 @@ var specs = map[string]Spec{ PurposeAvatar: {Purpose: PurposeAvatar, MaxSize: maxAvatar, MIMEs: imageMIMEs}, PurposeGroupAvatar: {Purpose: PurposeGroupAvatar, MaxSize: maxAvatar, MIMEs: imageMIMEs}, PurposeChatImage: {Purpose: PurposeChatImage, MaxSize: maxImage, MIMEs: imageMIMEs}, - PurposeChatGIF: {Purpose: PurposeChatGIF, MaxSize: maxImage, MIMEs: gifMIMEs}, + PurposeChatGIF: {Purpose: PurposeChatGIF, MaxSize: maxGIF, MIMEs: gifMIMEs}, PurposeChatVoice: {Purpose: PurposeChatVoice, MaxSize: maxVoice, MIMEs: voiceMIMEs}, PurposeChatVideo: {Purpose: PurposeChatVideo, MaxSize: maxVideo, MIMEs: videoMIMEs}, PurposeChatFile: {Purpose: PurposeChatFile, MaxSize: maxFile, MIMEs: fileMIMEs}, } -var imageMIMEs = set("image/jpeg", "image/jpg", "image/png", "image/webp", "image/heic", "image/heif") +var imageMIMEs = set("image/jpeg", "image/jpg", "image/png") var gifMIMEs = set("image/gif", "image/webp") var voiceMIMEs = set("audio/mpeg", "audio/mp4", "audio/aac", "audio/amr", "audio/ogg", "audio/wav", "audio/webm", "audio/x-m4a") var videoMIMEs = set("video/mp4", "video/quicktime", "video/webm") @@ -129,7 +156,7 @@ var fileMIMEs = set( var mimeExt = map[string]string{ "image/jpeg": "jpg", "image/jpg": "jpg", "image/png": "png", "image/webp": "webp", - "image/heic": "heic", "image/heif": "heif", "image/gif": "gif", + "image/gif": "gif", "audio/mpeg": "mp3", "audio/mp4": "m4a", "audio/aac": "aac", "audio/amr": "amr", "audio/ogg": "ogg", "audio/wav": "wav", "audio/webm": "weba", "audio/x-m4a": "m4a", "video/mp4": "mp4", "video/quicktime": "mov", "video/webm": "webm", diff --git a/internal/module/upload/entity/purpose_test.go b/internal/module/upload/entity/purpose_test.go index 71830a4..935b635 100644 --- a/internal/module/upload/entity/purpose_test.go +++ b/internal/module/upload/entity/purpose_test.go @@ -6,20 +6,24 @@ import ( ) func TestBuildObjectKeyByPurpose(t *testing.T) { - avatar := BuildObjectKey(PurposeAvatar, 11, 99, "20260917", "jpg") - if avatar != "perm/avatar/11/20260917/99.jpg" { + avatar := BuildObjectKey(PurposeAvatar, 11, "99", "20260917", "jpg") + if avatar != "perm/avatar/20260917/99.jpg" { t.Fatalf("avatar %s", avatar) } - chat := BuildObjectKey(PurposeChatImage, 11, 99, "20260917", "jpg") - if chat != "chat/20260917/11/99.jpg" { + chat := BuildObjectKey(PurposeChatImage, 11, "99", "20260917", "jpg") + if chat != "chat/image/20260917/99.jpg" { t.Fatalf("chat %s", chat) } - group := BuildObjectKey(PurposeGroupAvatar, 7, 99, "20260917", "png") + voice := BuildObjectKey(PurposeChatVoice, 11, "aa", "20260917", "m4a") + if voice != "chat/voice/20260917/aa.m4a" { + t.Fatalf("voice %s", voice) + } + group := BuildObjectKey(PurposeGroupAvatar, 7, "99", "20260917", "png") if group != "perm/group/7/99.png" { t.Fatalf("group %s", group) } - fav := BuildFavKey(11, 7, "20260917", "m4a") - if fav != "perm/fav/11/20260917/7.m4a" { + fav := BuildFavKey("99", "20260917", "m4a") + if fav != "perm/fav/20260917/99.m4a" { t.Fatalf("fav %s", fav) } } @@ -49,7 +53,19 @@ func TestNormalizeMIMEAndExt(t *testing.T) { if _, ok := spec.MIMEs["audio/mpeg"]; !ok { t.Fatal("voice mime") } - if strings.Contains(BuildObjectKey(PurposeAvatar, 1, 1, "d", "jpg"), PrefixChat) { + img, ok := LookupSpec(PurposeChatImage) + if !ok { + t.Fatal("image spec") + } + if _, ok := img.MIMEs["image/jpeg"]; !ok { + t.Fatal("jpeg") + } + for _, mime := range []string{"image/heic", "image/heif", "image/webp"} { + if _, ok := img.MIMEs[mime]; ok { + t.Fatalf("interchange must not allow %s", mime) + } + } + if strings.Contains(BuildObjectKey(PurposeAvatar, 1, "1", "d", "jpg"), PrefixChat) { t.Fatal("avatar must be perm") } } diff --git a/internal/module/upload/logic/batch.go b/internal/module/upload/logic/batch.go new file mode 100644 index 0000000..a506280 --- /dev/null +++ b/internal/module/upload/logic/batch.go @@ -0,0 +1,49 @@ +package logic + +import ( + "context" + + "spark-server/internal/module/upload/entity" + "spark-server/internal/pkg/errs" + "spark-server/internal/pkg/reqctx" + "spark-server/internal/pkg/validate" +) + +const maxCredentialBatch = 3 + +// BatchCreateInput issues up to three Presigned PUTs in one call (thumb / preview / orig). +type BatchCreateInput struct { + Items []CreateInput `json:"items" validate:"required,min=1,max=3,dive" label:"field.upload_items" errmsg:"required:validate.upload.items_required;min:validate.upload.items_required;max:validate.upload.items_invalid"` +} + +func (s *Service) CreateCredentials(ctx context.Context, userID int64, in *BatchCreateInput) (*entity.CredentialSet, error) { + if !s.ready() { + return nil, errs.ErrAvailable + } + if userID <= 0 { + return nil, errs.ErrUnauthorized + } + if in == nil { + return nil, errs.ErrInvalidArgument + } + if err := validate.StructLang(reqctx.Lang(ctx), in); err != nil { + return nil, err + } + if n := len(in.Items); n < 1 || n > maxCredentialBatch { + return nil, errs.ErrInvalidArgument + } + for range in.Items { + if !s.allow(userID) { + return nil, errs.ErrRateLimited + } + } + items := make([]entity.Credential, 0, len(in.Items)) + for i := range in.Items { + cred, err := s.issuePut(ctx, userID, &in.Items[i]) + if err != nil { + return nil, err + } + items = append(items, *cred) + } + return &entity.CredentialSet{Items: items}, nil +} diff --git a/internal/module/upload/logic/credential.go b/internal/module/upload/logic/credential.go index 590667e..6ea7ed7 100644 --- a/internal/module/upload/logic/credential.go +++ b/internal/module/upload/logic/credential.go @@ -2,6 +2,7 @@ package logic import ( "context" + "strconv" "spark-server/internal/module/upload/entity" "spark-server/internal/pkg/errs" @@ -34,14 +35,15 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create if !s.allow(userID) { return nil, errs.ErrRateLimited } + return s.issuePut(ctx, userID, in) +} + +func (s *Service) issuePut(ctx context.Context, userID int64, in *CreateInput) (*entity.Credential, error) { _, mime, err := matchSpec(ctx, in) if err != nil { return nil, err } - objectID, err := s.ids.NextID() - if err != nil { - return nil, err - } + name := reqctx.NewObjectName() ownerID := userID if in.Purpose == entity.PurposeGroupAvatar { if err := s.readyGroupAvatar(ctx, userID, in.GroupID); err != nil { @@ -49,8 +51,8 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create } ownerID = in.GroupID } - key := entity.BuildObjectKey(in.Purpose, ownerID, objectID, dayOf(s.now()), entity.ExtForMIME(mime)) - url, err := s.store.PresignPut(ctx, key, mime, s.ttl) + key := entity.BuildObjectKey(in.Purpose, ownerID, name, dayOf(s.now()), entity.ExtForMIME(mime)) + url, err := s.store.PresignPut(ctx, key, mime, in.Size, s.ttl) if err != nil { return nil, err } @@ -58,7 +60,11 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create ObjectKey: key, UploadURL: url, Method: entity.MethodPUT, - Headers: map[string]string{"Content-Type": mime}, + Headers: map[string]string{ + "Content-Type": mime, + "Content-Length": strconv.FormatInt(in.Size, 10), + "If-None-Match": "*", + }, ExpiresIn: s.expiresIn(), }, nil } diff --git a/internal/module/upload/logic/credential_test.go b/internal/module/upload/logic/credential_test.go index e09c11c..62a24c7 100644 --- a/internal/module/upload/logic/credential_test.go +++ b/internal/module/upload/logic/credential_test.go @@ -7,7 +7,6 @@ import ( "testing" "time" - "spark-server/internal/infrastructure/id" "spark-server/internal/module/upload/entity" "spark-server/internal/pkg/errs" "spark-server/internal/pkg/reqctx" @@ -18,7 +17,7 @@ type fakeStore struct { putErr error } -func (f *fakeStore) PresignPut(_ context.Context, key, _ string, _ time.Duration) (string, error) { +func (f *fakeStore) PresignPut(_ context.Context, key, _ string, _ int64, _ time.Duration) (string, error) { if f.putErr != nil { return "", f.putErr } @@ -37,11 +36,7 @@ func (f *fakeStore) PresignGet(_ context.Context, key string, _ time.Duration) ( func testUpload(t *testing.T) *Service { t.Helper() - ids, err := id.New(1) - if err != nil { - t.Fatal(err) - } - return NewService(&fakeStore{}, ids, Options{ + return NewService(&fakeStore{}, Options{ TTL: 5 * time.Minute, Now: func() time.Time { return time.Date(2026, 9, 17, 8, 0, 0, 0, time.UTC) }, }) @@ -54,11 +49,13 @@ func TestCreateCredentialChatVsAvatar(t *testing.T) { if err != nil { t.Fatal(err) } - if !strings.HasPrefix(chat.ObjectKey, "chat/20260917/42/") || chat.Method != "PUT" || chat.Headers["Content-Type"] != "image/jpeg" { + if !strings.HasPrefix(chat.ObjectKey, "chat/image/20260917/") || chat.Method != "PUT" || + chat.Headers["Content-Type"] != "image/jpeg" || chat.Headers["Content-Length"] != "1024" || + chat.Headers["If-None-Match"] != "*" || strings.Contains(chat.ObjectKey, "/42/") { t.Fatalf("%+v", chat) } avatar, err := svc.CreateCredential(ctx, 42, &CreateInput{Purpose: entity.PurposeAvatar, Mime: "image/png", Size: 2048}) - if err != nil || !strings.HasPrefix(avatar.ObjectKey, "perm/avatar/42/20260917/") { + if err != nil || !strings.HasPrefix(avatar.ObjectKey, "perm/avatar/20260917/") { t.Fatalf("%+v %v", avatar, err) } group, err := svc.CreateCredential(ctx, 42, &CreateInput{Purpose: entity.PurposeGroupAvatar, Mime: "image/jpeg", Size: 1024, GroupID: 7}) @@ -79,6 +76,11 @@ func TestCreateCredentialRejectsBadMIMEAndSize(t *testing.T) { if _, err := svc.CreateCredential(ctx, 1, &CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: 50 << 20}); err == nil { t.Fatal("huge image") } + for _, mime := range []string{"image/heic", "image/heif", "image/webp"} { + if _, err := svc.CreateCredential(ctx, 1, &CreateInput{Purpose: entity.PurposeChatImage, Mime: mime, Size: 1024}); err == nil { + t.Fatalf("accepted %s", mime) + } + } if _, err := svc.CreateCredential(ctx, 0, &CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: 1}); !errors.Is(err, errs.ErrUnauthorized) { t.Fatalf("uid %v", err) } @@ -95,3 +97,24 @@ func TestCreateDownloadValidatesKey(t *testing.T) { t.Fatal("path escape") } } + +func TestCreateCredentialsBatch(t *testing.T) { + svc := testUpload(t) + ctx := reqctx.With(t.Context(), &reqctx.RequestContext{UserID: 42, Language: "zh-CN"}) + part := func(size int64) CreateInput { + return CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: size} + } + got, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{Items: []CreateInput{part(1024), part(2048), part(4096)}}) + if err != nil || got == nil || len(got.Items) != 3 { + t.Fatalf("%+v %v", got, err) + } + if got.Items[0].ObjectKey == got.Items[1].ObjectKey || got.Items[1].Headers["Content-Length"] != "2048" { + t.Fatalf("%+v", got.Items) + } + if _, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{}); err == nil { + t.Fatal("empty") + } + if _, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{Items: []CreateInput{part(1), part(1), part(1), part(1)}}); err == nil { + t.Fatal("too many") + } +} diff --git a/internal/module/upload/logic/service.go b/internal/module/upload/logic/service.go index 9a6983a..6c86b65 100644 --- a/internal/module/upload/logic/service.go +++ b/internal/module/upload/logic/service.go @@ -10,15 +10,10 @@ import ( // Presigner issues short-lived object-storage URLs. Infrastructure implements it. type Presigner interface { - PresignPut(ctx context.Context, key, contentType string, ttl time.Duration) (string, error) + PresignPut(ctx context.Context, key, contentType string, size int64, ttl time.Duration) (string, error) PresignGet(ctx context.Context, key string, ttl time.Duration) (string, error) } -type IDGen interface { - NextID() (int64, error) -} - -// AvatarACL gates group_avatar uploads. Nil skips the check (tests). type AvatarACL interface { CanSetAvatar(ctx context.Context, userID, groupID int64) error } @@ -33,14 +28,13 @@ type Options struct { // Service issues upload/download credentials. HTTP and TCP share it. type Service struct { store Presigner - ids IDGen ttl time.Duration limit *ratelimit.Limiter now func() time.Time groups AvatarACL } -func NewService(store Presigner, ids IDGen, opt Options) *Service { +func NewService(store Presigner, opt Options) *Service { ttl := opt.TTL if ttl <= 0 { ttl = 15 * time.Minute @@ -49,11 +43,11 @@ func NewService(store Presigner, ids IDGen, opt Options) *Service { if clock == nil { clock = func() time.Time { return time.Now().UTC() } } - return &Service{store: store, ids: ids, ttl: ttl, limit: opt.Limit, now: clock, groups: opt.Groups} + return &Service{store: store, ttl: ttl, limit: opt.Limit, now: clock, groups: opt.Groups} } func (s *Service) ready() bool { - return s != nil && s.store != nil && s.ids != nil + return s != nil && s.store != nil } func (s *Service) expiresIn() int64 { diff --git a/internal/module/user/logic/avatar.go b/internal/module/user/logic/avatar.go index ee02231..9a4abe9 100644 --- a/internal/module/user/logic/avatar.go +++ b/internal/module/user/logic/avatar.go @@ -7,6 +7,7 @@ import ( uploadentity "spark-server/internal/module/upload/entity" "spark-server/internal/module/user/entity" + "spark-server/internal/pkg/reqctx" ) const defaultAvatarMIME = "image/png" @@ -42,18 +43,14 @@ func (s *Service) stampDefaultAvatar(ctx context.Context, u *entity.User) { } func (s *Service) putLetterPNG(ctx context.Context, u *entity.User) string { - if s == nil || s.objects == nil || s.ids == nil || u == nil || u.ID <= 0 { + if s == nil || s.objects == nil || u == nil || u.ID <= 0 { return "" } raw, err := entity.LetterAvatarPNG(u.Nickname) if err != nil || len(raw) == 0 { return "" } - objectID, err := s.ids.NextID() - if err != nil { - return "" - } - key := uploadentity.BuildObjectKey(uploadentity.PurposeAvatar, u.ID, objectID, time.Now().Format("20060102"), "png") + key := uploadentity.BuildObjectKey(uploadentity.PurposeAvatar, u.ID, reqctx.NewObjectName(), time.Now().UTC().Format("20060102"), "png") if err := s.objects.Put(ctx, key, raw, defaultAvatarMIME); err != nil { return "" } diff --git a/internal/module/user/logic/login_test.go b/internal/module/user/logic/login_test.go index 49bedd1..c74428b 100644 --- a/internal/module/user/logic/login_test.go +++ b/internal/module/user/logic/login_test.go @@ -27,10 +27,15 @@ func TestAuthenticateTestEmailAnyCode(t *testing.T) { if u1.ID <= 0 || !entity.ValidSparkID(u1.SparkID) || u1.Nickname != "test01" || !entity.ValidQRCode(u1.QRCode) { t.Fatalf("%+v", u1) } - prefix := "perm/avatar/" + strconv.FormatInt(u1.ID, 10) + "/" - if !strings.HasPrefix(u1.Avatar, prefix) || !strings.HasSuffix(u1.Avatar, ".png") { + if !strings.HasPrefix(u1.Avatar, "perm/avatar/") || !strings.HasSuffix(u1.Avatar, ".png") { t.Fatalf("default avatar %s", u1.Avatar) } + uid := strconv.FormatInt(u1.ID, 10) + for _, part := range strings.Split(u1.Avatar, "/") { + if part == uid { + t.Fatalf("avatar leaked uid %s", u1.Avatar) + } + } store, ok := svc.objects.(*memObjects) if !ok { t.Fatal("mem objects") diff --git a/internal/pkg/i18n/catalog_validate_upload.go b/internal/pkg/i18n/catalog_validate_upload.go index b6e6ccd..2056036 100644 --- a/internal/pkg/i18n/catalog_validate_upload.go +++ b/internal/pkg/i18n/catalog_validate_upload.go @@ -11,6 +11,9 @@ func init() { KeyFieldSize: { ZhCN: "文件大小", En: "File size", ZhTW: "檔案大小", }, + KeyFieldUploadItems: { + ZhCN: "上传列表", En: "Upload list", ZhTW: "上傳列表", + }, KeyFieldObjectKey: { ZhCN: "对象键", En: "Object key", ZhTW: "物件鍵", }, @@ -50,6 +53,12 @@ func init() { KeyUploadGroupRequired: { ZhCN: "请选择群", En: "Please choose a group", ZhTW: "請選擇群", }, + KeyUploadItemsRequired: { + ZhCN: "请填写要上传的文件", En: "Please provide files to upload", ZhTW: "請填寫要上傳的檔案", + }, + KeyUploadItemsInvalid: { + ZhCN: "一次最多签发 3 个上传凭证", En: "At most 3 upload credentials per request", ZhTW: "一次最多簽發 3 個上傳憑證", + }, KeyFavoriteMsgRequired: { ZhCN: "请选择消息", En: "Please choose a message", ZhTW: "請選擇訊息", }, diff --git a/internal/pkg/i18n/keys.go b/internal/pkg/i18n/keys.go index adbcff0..2d7e935 100644 --- a/internal/pkg/i18n/keys.go +++ b/internal/pkg/i18n/keys.go @@ -42,6 +42,7 @@ const ( KeyFieldPurpose = "field.purpose" KeyFieldMime = "field.mime" KeyFieldSize = "field.size" + KeyFieldUploadItems = "field.upload_items" KeyFieldObjectKey = "field.object_key" KeyFieldMsgID = "field.msg_id" KeyFieldClientMsgID = "field.client_msg_id" @@ -90,6 +91,8 @@ const ( KeyUploadKeyRequired = "validate.upload.key_required" KeyUploadKeyInvalid = "validate.upload.key_invalid" KeyUploadGroupRequired = "validate.upload.group_required" + KeyUploadItemsRequired = "validate.upload.items_required" + KeyUploadItemsInvalid = "validate.upload.items_invalid" KeyFavoriteMsgRequired = "validate.favorite.msg_required" KeyFavoriteMsgInvalid = "validate.favorite.msg_invalid" KeyMessageClientMsgRequired = "validate.message.client_msg_required" diff --git a/internal/pkg/reqctx/reqctx.go b/internal/pkg/reqctx/reqctx.go index ab6223b..1a72c39 100644 --- a/internal/pkg/reqctx/reqctx.go +++ b/internal/pkg/reqctx/reqctx.go @@ -3,7 +3,8 @@ package reqctx import ( "context" "crypto/rand" - "encoding/hex" + "encoding/binary" + "math/big" "net" "strings" "time" @@ -91,11 +92,51 @@ func stripHost(v string) string { return v } -// NewID returns a 16-byte hex id for request_id. +const ( + base62Alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" + base62Len = 11 + objectNameLen = 16 + objectNameSrc = 12 +) + +// NewID returns an 8-byte base62 id (11 chars) for request_id and JWT jti. func NewID() string { - var b [16]byte + var b [8]byte if _, err := rand.Read(b[:]); err != nil { - return "00000000000000000000000000000000" + return "00000000000" } - return hex.EncodeToString(b[:]) + return encodeBase62(binary.BigEndian.Uint64(b[:])) +} + +// NewObjectName returns a 12-byte base62 id (16 chars) for object_key filenames. Not Snowflake. +func NewObjectName() string { + return randomBase62(objectNameSrc, objectNameLen) +} + +func encodeBase62(n uint64) string { + var out [base62Len]byte + for i := base62Len - 1; i >= 0; i-- { + out[i] = base62Alphabet[n%62] + n /= 62 + } + return string(out[:]) +} + +func randomBase62(byteN, width int) string { + if byteN < 1 || width < 1 { + return "" + } + b := make([]byte, byteN) + if _, err := rand.Read(b); err != nil { + return strings.Repeat("0", width) + } + n := new(big.Int).SetBytes(b) + out := make([]byte, width) + base := big.NewInt(62) + r := new(big.Int) + for i := width - 1; i >= 0; i-- { + n.DivMod(n, base, r) + out[i] = base62Alphabet[r.Int64()] + } + return string(out) } diff --git a/internal/pkg/reqctx/reqctx_test.go b/internal/pkg/reqctx/reqctx_test.go index ddd497e..b39e07c 100644 --- a/internal/pkg/reqctx/reqctx_test.go +++ b/internal/pkg/reqctx/reqctx_test.go @@ -1,6 +1,9 @@ package reqctx -import "testing" +import ( + "strings" + "testing" +) func TestParseHTTPRealIP(t *testing.T) { got := ParseHTTPRealIP(" 203.0.113.10 ", "198.51.100.1, 192.0.2.1", "192.0.2.1:1234") @@ -27,6 +30,48 @@ func TestRealIPFromContext(t *testing.T) { } } +func TestNewIDLength(t *testing.T) { + id := NewID() + if len(id) != 11 { + t.Fatalf("len %d %q", len(id), id) + } + for i := 0; i < len(id); i++ { + if !strings.ContainsRune(base62Alphabet, rune(id[i])) { + t.Fatalf("char %q in %q", id[i], id) + } + } + if id == NewID() { + t.Fatal("not unique") + } +} + +func TestNewObjectNameLength(t *testing.T) { + id := NewObjectName() + if len(id) != 16 { + t.Fatalf("len %d %q", len(id), id) + } + for i := 0; i < len(id); i++ { + if !strings.ContainsRune(base62Alphabet, rune(id[i])) { + t.Fatalf("char %q in %q", id[i], id) + } + } + if id == NewObjectName() { + t.Fatal("not unique") + } +} + +func TestEncodeBase62(t *testing.T) { + if got := encodeBase62(0); got != "00000000000" { + t.Fatalf("%q", got) + } + if got := encodeBase62(61); got != "0000000000z" { + t.Fatalf("%q", got) + } + if got := encodeBase62(62); got != "00000000010" { + t.Fatalf("%q", got) + } +} + func TestRequestIDFromContext(t *testing.T) { ctx := With(t.Context(), &RequestContext{RequestID: "abc"}) if RequestID(ctx) != "abc" { diff --git a/internal/transport/http/docs/openapi.yaml b/internal/transport/http/docs/openapi.yaml index f4c5853..e54cd4d 100644 --- a/internal/transport/http/docs/openapi.yaml +++ b/internal/transport/http/docs/openapi.yaml @@ -7,7 +7,7 @@ info: ## 环境与地址 - 公网只开放 `443`。Nginx 按 TLS SNI 分流:HTTP 证书打到 `sparkapi.wyrs365.com`,IM 证书打到 `sparkim.wyrs365.com`。进程本身只绑本机。 + 公网只开放 `443`。Nginx 按 TLS SNI 分流:HTTP 证书打到 `sparkapi.wyrs365.com`,IM 证书打到 `sparkim.wyrs365.com`,对象存储(RustFS S3)打到 `sparkr.wyrs365.com`。进程本身只绑本机。 | 环境 | 用途 | HTTP Base URL | IM 长连接 | | --- | --- | --- | --- | @@ -36,7 +36,7 @@ info: | --- | --- | --- | | `code` | **字符串** | `"0"` 成功;其它失败。不要用数字 `0` 比较。全表见 [错误码](/docs/errors)。 | | `msg` | 字符串 | 给人看的文案,随请求语言变化(`zh-CN` / `en` / `zh-TW`)。成功一般为 `ok`。 | - | `request_id` | 字符串 | 服务端生成的本次请求 id。客户端不用传;从响应体读取。与 Header `X-Request-ID` 相同。 | + | `request_id` | 字符串 | 服务端生成,11 位 base62(`0-9A-Za-z`)。客户端不用传;从响应体读取。与 Header `X-Request-ID` 相同。 | | `data` | object / array | 成功时为业务对象;失败时固定 `{}`。字段缺失当空对象,不要当 `null`。 | **判定成功只看 `code === "0"`。** 不要只看 HTTP 状态码。 @@ -655,7 +655,7 @@ paths: uid: 1940000000000000000 spark_id: alice_ok nickname: test01 - avatar: perm/avatar/1940000000000000000/20260917/1940000000000000001.png + avatar: perm/avatar/20260917/3H8kPq2mN1Rabcde.png qrcode: "-NDveu-9Q" birthday: "" gender: 0 @@ -1106,9 +1106,13 @@ paths: description: | 客户端只传 `purpose` / `mime` / `size`,服务端生成 object_key 并签发短时 Presigned PUT。 文件直传 RustFS,不经过 Gin。 - `avatar` 写入 `perm/avatar/{uid}/…`(不过期);`group_avatar` 写入 `perm/group/{群号}/…`(需 `group_id`,群主/管理员); - 聊天媒体写入 `chat/{yyyyMMdd}/{uid}/…`(桶生命周期过期)。 - 客户端 PUT 时必须带返回的 `headers`(至少 `Content-Type`)。消息与头像只存 `object_key`。 + `avatar` 写入 `perm/avatar/{yyyyMMdd}/{name}.{ext}`(不过期,路径不含 uid); + `group_avatar` 写入 `perm/group/{群号}/{name}.{ext}`(需 `group_id`,群主/管理员); + 聊天按类型:`chat/image|voice|video|gif|file/{yyyyMMdd}/{name}.{ext}`(桶生命周期按前缀过期)。 + `{name}` 为服务端 16 位 base62 随机串,不占用 Snowflake。 + 发图片:客户端压成 JPEG/PNG 后 `POST /v1/files/batch` 直传缩略图、压缩大图、原图(未勾选原图只传前两项)。 + 服务端不转码。iOS HEIC、Android WebP 须客户端先转 JPEG/PNG,两端才能互看。 + 客户端 PUT 必须原样带返回的 `headers`。消息只存 object_key。 operationId: createUploadCredential requestBody: required: true @@ -1131,12 +1135,61 @@ paths: code: "0" msg: ok data: - object_key: chat/20260917/1/2.jpg - upload_url: http://47.236.66.9:9000/spark/chat/20260917/1/2.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256 + object_key: chat/image/20260917/3H8kPq2mN1Rabcde.jpg + upload_url: https://sparkr.wyrs365.com/spark/chat/image/20260917/3H8kPq2mN1Rabcde.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256 method: PUT headers: Content-Type: image/jpeg + Content-Length: "245760" + If-None-Match: "*" expires_in: 900 + /v1/files/batch: + post: + tags: [files] + summary: 批量签发直传凭证 + description: | + 一次返回 1~3 个 Presigned PUT,顺序与 `items` 相同。聊天图片请按「缩略图、压缩大图、原图」排列;未勾选原图只传前两项。 + mime 仅 `image/jpeg` / `image/png`(HEIC、WebP 请客户端先转)。限流按条数计入 `upload_per_second`。文件仍直传 RustFS。 + operationId: createUploadCredentials + requestBody: + required: true + content: + application/json: + schema: + $ref: "#/components/schemas/CreateUploadBatchRequest" + example: + items: + - purpose: chat_image + mime: image/jpeg + size: 8192 + - purpose: chat_image + mime: image/jpeg + size: 245760 + - purpose: chat_image + mime: image/jpeg + size: 3500000 + responses: + "200": + description: 统一 envelope + content: + application/json: + schema: + $ref: "#/components/schemas/Envelope" + example: + code: "0" + msg: ok + data: + items: + - object_key: chat/image/20260917/3H8kPq2mN1Rabcde.jpg + method: PUT + headers: + Content-Type: image/jpeg + Content-Length: "8192" + If-None-Match: "*" + - object_key: chat/image/20260917/4kPq2mN1RabcdeXY.jpg + method: PUT + - object_key: chat/image/20260917/5mN1RabcdeXY12ab.jpg + method: PUT /v1/files/downloads: post: tags: [files] @@ -1181,7 +1234,7 @@ paths: tags: [favorites] summary: 收藏一条消息 description: | - 会话成员可将消息收入收藏。图片/语音/视频/文件会 CopyObject 到 `perm/fav/{uid}/…`。 + 会话成员可将消息收入收藏。图片/语音/视频/文件会 CopyObject 到 `perm/fav/{yyyyMMdd}/{name}.{ext}`。 源对象已从 `chat/` 过期删除则失败(`code=40402`),不提供本地补传。 同一 `msg_id` 重复收藏返回已有记录。撤回不影响已收藏快照。 operationId: createFavorite @@ -1477,8 +1530,8 @@ components: example: ok request_id: type: string - description: 服务端生成的本次请求 id。客户端不用传,从响应体读取。 - example: a1b2c3d4e5f6a7b8a1b2c3d4e5f6a7b8 + description: 服务端生成的本次请求 id(11 位 base62,`0-9A-Za-z`)。客户端不用传,从响应体读取。 + example: 3H8kPq2mN1R data: type: object additionalProperties: true @@ -1743,6 +1796,9 @@ components: mime: type: string example: image/jpeg + description: | + avatar / group_avatar / chat_image 仅 image/jpeg、image/jpg、image/png。 + iOS HEIC、Android WebP 须客户端转成 JPEG/PNG 再申请凭证。 size: type: integer format: int64 @@ -1752,6 +1808,16 @@ components: format: int64 minimum: 1 description: purpose=group_avatar 时必填,为自增群号 + CreateUploadBatchRequest: + type: object + required: [items] + properties: + items: + type: array + minItems: 1 + maxItems: 3 + items: + $ref: "#/components/schemas/CreateUploadRequest" CreateDownloadRequest: type: object required: [object_key] @@ -1792,6 +1858,39 @@ components: properties: body: type: string + image: + type: object + description: | + t=2。点开看 `key`(压缩预览);气泡用 `thumb_key`;`orig_key` 有则显示「查看原图」。 + 三档由客户端压成 JPEG/PNG 后 `POST /v1/files/batch` 直传。服务端不转码;不要传 HEIC/WebP。 + required: [key] + properties: + key: + type: string + description: 点开查看用的压缩图 object_key + thumb_key: + type: string + description: 气泡缩略图,可省略 + orig_key: + type: string + description: 原图 object_key;空则没有「查看原图」 + orig_size: + type: integer + format: int64 + description: 原图像素文件大小,供「查看原图 3.2MB」按钮 + orig_w: + type: integer + orig_h: + type: integer + mime: + type: string + w: + type: integer + h: + type: integer + size: + type: integer + format: int64 at_all: type: boolean description: "@所有人,仅群主/管理员" diff --git a/internal/transport/http/file_test.go b/internal/transport/http/file_test.go index ff79f88..f2491f0 100644 --- a/internal/transport/http/file_test.go +++ b/internal/transport/http/file_test.go @@ -27,11 +27,11 @@ import ( type stubPresign struct{} -func (stubPresign) PresignPut(_ context.Context, key, _ string, _ time.Duration) (string, error) { +func (s *stubPresign) PresignPut(_ context.Context, key, _ string, _ int64, _ time.Duration) (string, error) { return "http://rustfs.test/" + key + "?sig=1", nil } -func (stubPresign) PresignGet(_ context.Context, key string, _ time.Duration) (string, error) { +func (s *stubPresign) PresignGet(_ context.Context, key string, _ time.Duration) (string, error) { return "http://rustfs.test/" + key + "?get=1", nil } @@ -48,7 +48,7 @@ func testFileEngine(t *testing.T) (*ginEngineBundle, *id.Generator, *msgmodel.Me users := userlogic.NewService(usermodel.NewMemoryRepo(), usermodel.NewMemoryCodes(), ids, nil) msgs := msgmodel.NewMemoryRepo() convs := convmodel.NewMemoryP2P() - files := uploadlogic.NewService(stubPresign{}, ids, uploadlogic.Options{ + files := uploadlogic.NewService(&stubPresign{}, uploadlogic.Options{ Now: func() time.Time { return time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC) }, }) favs := favlogic.NewService(favmodel.NewMemoryRepo(), msgs, convs, nil, ids) @@ -71,13 +71,25 @@ func TestUploadCredentialHTTP(t *testing.T) { ObjectKey string `json:"object_key"` Method string `json:"method"` } - if err := json.Unmarshal(env.Data, &cred); err != nil || cred.Method != "PUT" || !strings.Contains(cred.ObjectKey, "chat/20260917/") { + if err := json.Unmarshal(env.Data, &cred); err != nil || cred.Method != "PUT" || !strings.Contains(cred.ObjectKey, "chat/image/20260917/") { t.Fatalf("%s %v", env.Data, err) } bad := authJSON(t, h, &tok, http.MethodPost, "/v1/files", `{"purpose":"avatar","mime":"application/pdf","size":10}`) if bad.Code != errs.ErrInvalidArgument.Code { t.Fatalf("mime %+v", bad) } + batch := authJSON(t, h, &tok, http.MethodPost, "/v1/files/batch", `{"items":[{"purpose":"chat_image","mime":"image/jpeg","size":1024},{"purpose":"chat_image","mime":"image/jpeg","size":2048}]}`) + if batch.Code != "0" { + t.Fatalf("batch %+v", batch) + } + var set struct { + Items []struct { + ObjectKey string `json:"object_key"` + } `json:"items"` + } + if err := json.Unmarshal(batch.Data, &set); err != nil || len(set.Items) != 2 || set.Items[0].ObjectKey == set.Items[1].ObjectKey { + t.Fatalf("%s %v", batch.Data, err) + } } func TestFavoriteTextHTTP(t *testing.T) { diff --git a/internal/transport/http/handler/file/handler.go b/internal/transport/http/handler/file/handler.go index 25dbaa0..59c1bda 100644 --- a/internal/transport/http/handler/file/handler.go +++ b/internal/transport/http/handler/file/handler.go @@ -36,6 +36,24 @@ func (h *Handler) Create(c *gin.Context) { respond.JSON(c, out) } +func (h *Handler) CreateBatch(c *gin.Context) { + uid, ok := currentUID(c) + if !ok { + return + } + var in uploadlogic.BatchCreateInput + if err := respond.BindJSON(c, &in); err != nil { + respond.JSONError(c, err) + return + } + out, err := h.files.CreateCredentials(c.Request.Context(), uid, &in) + if err != nil { + respond.JSONError(c, err) + return + } + respond.JSON(c, out) +} + func (h *Handler) Download(c *gin.Context) { uid, ok := currentUID(c) if !ok { diff --git a/internal/transport/http/router.go b/internal/transport/http/router.go index fc6362b..c96d3b7 100644 --- a/internal/transport/http/router.go +++ b/internal/transport/http/router.go @@ -132,6 +132,7 @@ func registerFileRoutes(v1 *gin.RouterGroup, deps *EngineDeps) { h := filehandler.New(deps.Files) g := v1.Group("/files") g.POST("", h.Create) + g.POST("/batch", h.CreateBatch) g.POST("/downloads", h.Download) } diff --git a/internal/transport/http/server_test.go b/internal/transport/http/server_test.go index a63cfbc..4e36458 100644 --- a/internal/transport/http/server_test.go +++ b/internal/transport/http/server_test.go @@ -34,7 +34,7 @@ func TestHealthzEnvelopeAndTraceHeaders(t *testing.T) { t.Fatalf("status %d", w.Code) } got := w.Header().Get("X-Request-ID") - if len(got) != 32 { + if len(got) != 11 { t.Fatalf("server request id %q headers %v", got, w.Header()) } if w.Header().Get("X-Trace-ID") != "" { @@ -47,7 +47,7 @@ func TestHealthzEnvelopeAndTraceHeaders(t *testing.T) { if env.Code != "0" || env.Msg != "ok" { t.Fatalf("%+v", env) } - if env.RequestID != got || len(env.RequestID) != 32 { + if env.RequestID != got || len(env.RequestID) != 11 { t.Fatalf("body request_id %q header %q", env.RequestID, got) } if !bytes.Contains(env.Data, []byte(`"status":"ok"`)) { @@ -82,7 +82,7 @@ func TestReadyzHTTPStatus(t *testing.T) { if env.Code != errs.ErrAvailable.Code || !bytes.Contains(env.Data, []byte(`"mysql":"down"`)) { t.Fatalf("%+v", env) } - if len(env.RequestID) != 32 { + if len(env.RequestID) != 11 { t.Fatalf("readyz request_id %q", env.RequestID) } }