feat: update object key

This commit is contained in:
zhr 2026-09-18 17:14:38 +08:00
parent 8aa8e0bad9
commit c11fbda141
40 changed files with 1014 additions and 162 deletions

View File

@ -95,7 +95,7 @@ Media: HTTP upload credential, message content is object key only. Frame max 64K
Message types: text / image / voice / video / emoji / gif / file / location / system (`message/entity.Type` 1–9). `messages.content` stores serialized JSON bytes (BLOB), never a raw string. Closed type/status values: [enums.md](enums.md).
TCP body is JSON envelope `{code,msg,request_id,data}`. AUTH request is `{"token","device_id","app_version","language"}` (not protobuf).
Public TLS is Nginx `:443` SNI: HTTP cert on loopback `:8443`, IM cert on loopback `:19001` then PROXY to gnet `:9090` (host `:9000` is rustfs). See `deploy/nginx/`.
Public TLS is Nginx `:443` SNI: HTTP cert on loopback `:8443` (`sparkapi` + `sparkr`/RustFS), IM cert on loopback `:19001` then PROXY to gnet `:9090` (host `:9000` is rustfs). See `deploy/nginx/`.
AUTH/login rate-limit uses `reqctx.RealIP`. TaskPool workers recover panics. Deliver workers start with `RequeueStale` + `SETNX deliver:done:{msgId}`.
## When adding a feature

329
README.md
View File

@ -1,21 +1,322 @@
# spark-server
Spark IM modular monolith. Architecture: [docs/architecture-v3.md](docs/architecture-v3.md).
Spark IM 服务端。Go 1.26 **模块化单体**(一台机器、一个进程)。公网只开 **443**,HTTP / IM / 对象存储用三个域名、Nginx SNI 分流。
- Go 1.26
- Gin HTTP on `127.0.0.1:8080`. JSON envelope `{"code","msg","request_id","data"}`. Docs: `GET /docs` (API), `/docs/enums`, `/docs/errors`. Spec: `GET /openapi.yaml`.
- Probes: `GET /healthz` liveness; `GET /readyz` readiness (HTTP 503 when MySQL/Redis/task pool/accepting are not ready). `env: prod` (or `SPARK_ENV=prod`) rejects empty MySQL DSN, empty Redis, and the default JWT secret.
- Logs: zap to console and/or rotating file (`log.console` / `log.file`). GORM SQL goes through the same zap. `request_id` is server-generated and returned in the response body (also as `X-Request-ID`).
- gnet TCP on loopback (local `127.0.0.1:8090`; ECS Docker `127.0.0.1:9090` because host `:9000` is rustfs). gnet v2.10 has no built-in TLS; Nginx terminates IM TLS and forwards PROXY+plaintext. Sample: `deploy/nginx/`. `make deploy` writes image/config/logs under `/data/spark-server/`.
- TCP Frame body is JSON `{code,msg,request_id,data}`. AUTH body: `{"token","device_id","app_version","language"}`.
- Public entry is Nginx `:443` only (SNI split)
仓库:<https://gitea.wyrs365.com/Spark/spark-server.git>
完整设计:[docs/architecture-v3.md](docs/architecture-v3.md)
在线协议:`GET /docs`(Swagger)、`/docs/enums`、`/docs/errors`,规范 `GET /openapi.yaml`。
```bash
make start
make migrate
make lint
不上 Kafka、K8s、微服务。不要给 gnet 绑公网端口,不要在 EventLoop 里打 MySQL。
---
## 它做什么
即时通讯:登录、好友、群、发消息、增量同步、撤回删除、媒体直传。
| 层 | 职责 |
| --- | --- |
| 实时 | TCP 长连接(gnet)。在线推送 `MESSAGE`,心跳 30s,空闲 120s 断开 |
| 正文 | MySQL。Snowflake `MsgId` 是主键、排序、同步游标。没有 `ConversationSeq` |
| 会话列表 / 未读 | Redis ZSET。群热路径不写 MySQL inbox 行 |
| 扇出 | Redis `deliver:queue` + worker。`SEND_ACK` = MySQL COMMIT **且** 入队成功 |
| 媒体 | 客户端拿 Presigned URL 直传 RustFS。消息里只存 `object_key` |
HTTP 和 TCP 调同一套 `internal/module/*/logic`。Gin handler 只做绑定和写信封。
---
## 生产入口
| 用途 | 域名 | 说明 |
| --- | --- | --- |
| HTTP API | `https://sparkapi.wyrs365.com` | REST `/v1/...`,文档 `/docs` |
| IM 长连接 | `sparkim.wyrs365.com:443` | TLS + 自定义 Frame,不是 WebSocket |
| 对象存储 | `https://sparkr.wyrs365.com` | RustFS S3 path-style,Presigned PUT/GET |
本机开发:HTTP `http://127.0.0.1:8080`,TCP 明文 `127.0.0.1:8090`。
ECS 上进程只绑回环,由 Nginx 终结 TLS:
```text
客户端 :443 (ssl_preread)
├─ sparkapi.wyrs365.com → 127.0.0.1:8443 → Gin 127.0.0.1:18080
├─ sparkr.wyrs365.com → 127.0.0.1:8443 → RustFS 127.0.0.1:9000
└─ sparkim.wyrs365.com → 127.0.0.1:19001 (IM TLS)
→ PROXY → gnet 127.0.0.1:9090
```
`make start` runs `cmd/server` against `configs/config.yaml` (override with `make start CONFIG=path/to.yaml`). `make run` builds `bin/spark-server` first, then starts that binary.
主机 `:9000` 是 RustFS S3,所以 IM gnet 用 **9090**。安全组只放行 443。
Agent conventions live in `.cursor/rules` and `.cursor/skills/spark-im-v3`. Lint conventions live in `.golangci.yaml`.
---
## 硬约束(不要改)
| ID | 规则 |
| --- | --- |
| C1 | 消息正文在 MySQL。Snowflake `MsgId` 是 PK / 排序 / 同步游标。没有 ConversationSeq |
| C2 | gnet EventLoop 不调 MySQL / Redis / 对象存储 / 扇出。入站进 TaskPool;扇出走 `deliver:queue` |
| C3 | 保密靠 TLS。Frame 只有薄头 + 可选 gzip。没有应用层 ECDH/AES |
| C4 | 公网只有 443。gnet / Gin 绑回环 |
| C5 | `SEND_ACK` 必须在 COMMIT **且** deliver `Push` 成功之后。入队失败不 ACK |
| C6 | 限流 / 审计用 PROXY / `X-Real-IP` 的 `RealIP`,不用 gnet `RemoteAddr`(那是 127.0.0.1) |
| C7 | 会话列表 / 未读 / 预览在 Redis ZSET。群热路径不 upsert MySQL inbox |
热路径禁止 `context.TODO()`。推送超时不踢连接。不要把媒体字节写进 `messages.content`。
---
## 统一响应
HTTP 与 TCP **应答 Body** 同一套信封。业务成败只看 `code` 字符串,不要用数字 `0` 比较。除 `/readyz` 未就绪是 HTTP 503 外,业务失败也是 HTTP 200。
```json
{"code":"0","msg":"ok","request_id":"3H8kPq2mN1R","data":{}}
```
| 字段 | 规则 |
| --- | --- |
| `code` | `"0"` 成功;其它失败。全表见 [/docs/errors](https://sparkapi.wyrs365.com/docs/errors) |
| `msg` | 展示文案,随语言变。成功一般为 `ok` |
| `request_id` | 服务端生成,**11 位 base62**(`0-9A-Za-z`)。客户端不用传。HTTP 同时写 Header `X-Request-ID` |
| `data` | 成功为业务对象;失败固定 `{}`。缺字段当空对象,不要当 `null` |
`request_id` 和 TCP Frame 头里的 `RequestID`(uint64)不是一回事:后者是客户端填的配对号,服务端原样带回。
### HTTP 请求放哪
| 位置 | 内容 |
| --- | --- |
| Header | `Authorization: Bearer …`、`X-Device-ID`、`X-Language` 或 `Accept-Language`、`X-App-Version`、`Content-Type` |
| Body | POST/PATCH 业务字段。不要把 token / 语言 / device_id 塞进业务 JSON(登录 Body 带 `device_id`,刷新 Body 带 `refresh_token`) |
| Path | 资源 id:好友 uid、群号、MsgId |
| Query | GET 筛选 / 分页 |
语言:`zh-CN`(缺省)/ `en` / `zh-TW`。网关写 `X-Real-IP`,客户端不要发。
---
## 登录
1. (可选)`POST /v1/auth/email-codes` Body `{"email"}`。
2. `POST /v1/auth/login` 拿 `access_token` / `refresh_token`。`device_id` 由**客户端生成并持久化**,服务端不分配;写入 JWT `did`。
3. 其它 HTTP:`Authorization: Bearer …`,可选 `X-Device-ID`(须等于登录时的 device_id)。
4. 续期:`POST /v1/auth/token/refresh`,Body `{"refresh_token"}`,不要带 Bearer。
5. 在线发消息、收推送、增量同步走 **TCP**。`POST /v1/messages` 只是 HTTP 调试通道,不要当 App 主路径。
测试号 `test01@spark.com`~`test10@spark.com` 可跳过发码,登录任意 `verify_code`。
`uid` 就是 Snowflake `users.id`。对外展示用可变的 `spark_id`,不要把 uid 写进对象存储路径。
---
## HTTP 资源
前缀 `/v1/{resource}`。
| 组 | 路径 | 说明 |
| --- | --- | --- |
| 探活 | `/healthz` `/readyz` | 存活;未就绪 HTTP 503 |
| 文档 | `/docs` `/docs/enums` `/docs/errors` `/openapi.yaml` | 无需登录 |
| 枚举 | `GET /v1/meta/enums` | 无需登录 |
| 认证 | `/v1/auth` | 发码、登录、刷新、登出 |
| 用户 | `/v1/users/me` | 资料 |
| 好友 | `/v1/friends` | 申请、同意、备注、置顶、拉黑、举报 |
| 群 | `/v1/groups` | 创建、资料、成员、邀请、踢人、转让、解散 |
| 文件 | `/v1/files` | 上传/下载凭证、批量、图片派生 |
| 收藏 | `/v1/favorites` | 收藏;媒体会 Copy 到 `perm/fav/` |
| 会话 | `/v1/conversations` | 列表、免打扰 |
| 消息 | `/v1/messages` | 历史 / 撤回 / 删除(主路径仍是 TCP) |
---
## TCP 协议
不是 WebSocket、不是 HTTP、不是 Protobuf。对 `sparkim.wyrs365.com:443` 做 TLS(SNI 必须是该域名),之后写二进制 Frame。
### Frame(大端,固定 23 字节头)
```text
offset 0 Magic uint16 0x5350(ASCII "SP")
offset 2 Version uint8 1
offset 3 Command uint16
offset 5 Flags uint16 bit0=1 表示 Body 是 gzip
offset 7 RequestID uint64 客户端配对,可 0;响应原样带回
offset 15 SequenceID uint32 请求/响应配对;PUSH_ACK 必须带回推送帧的 SequenceID
offset 19 BodyLength uint32 Body 字节数(压缩后)
offset 23 Body bytes UTF-8 JSON
```
先读 23 字节,再按 `BodyLength` 读 Body。Magic 不对或总长超过 64KB 应断开。Body ≥ 512B 且 gzip 更小时可压缩。
**请求 Body 是业务 JSON,不要包信封。** 应答和 `MESSAGE` 推送才是 `{code,msg,request_id,data}`。失败时 Command 仍为请求 Command(发送失败不是 `SEND_ACK`)。
| 值 | 名 | 方向 |
| --- | --- | --- |
| 1 | AUTH | → 请求 `{"token","device_id","app_version","language"}` |
| 2 | HEARTBEAT | → 空 Body |
| 3 | GENERATE_MSG_ID | → 空;应答 `data.msg_id` |
| 4 | SEND_MESSAGE | → 与 HTTP 发消息同一 JSON |
| 5 | SEND_ACK | ← 仅 4 的成功应答 |
| 6 | MESSAGE | ← 推送 |
| 7 | PUSH_ACK | → 空;`SequenceID` = 推送帧的 SequenceID |
| 8 | READ_MESSAGE | → |
| 9 | RECALL_MESSAGE | → `{"msg_id"}` |
| 10 | SYNC_INBOX | → 会话列表增量 |
| 12 | SYNC_MESSAGES | → 某会话消息增量 |
| 14 | KICK | ← 同设备顶号 |
| 15 | DELETE_MESSAGE | → `{"msg_id"}` |
除 AUTH / HEARTBEAT / PUSH_ACK 外须先 AUTH,否则 `40102`。
上线同步顺序:**先 `SYNC_INBOX`,再按会话 `SYNC_MESSAGES`**。游标是 Snowflake `MsgId` / inbox score,不是 ConversationSeq。
Java / Swift 示例:`docs/clients/`,OpenAPI 里也可点开。
---
## 发消息路径
必须按这个顺序(C5):
1. TaskPool(禁止在 EventLoop 里做)
2. 权限(好友 / 拉黑 / 群禁言)——失败则不 INSERT
3. Snowflake MsgId(或客户端 MsgId,须在 ±5 分钟内)
4. MySQL 事务:消息表 + `conversations.last_msg_id`
5. `DeliverQueue.Push`
6. `SEND_ACK`
第 5 步失败不 ACK。客户端用同一个 `client_msg_id` 重试。发送设备不回推 `MESSAGE`,其它设备必须推。同 `device_id` 顶旧连接,下发 `KICK`。
私聊与群消息分表存储。消息类型 1–9:文本 / 图片 / 语音 / 视频 / 表情 / gif / 文件 / 位置 / 系统。`content` 是 JSON 序列化后的 **bytes**,不是字符串。
---
## 媒体 `object_key`
客户端 **不自己拼 key**。`POST /v1/files` 只传 `purpose` / `mime` / `size`(群头像再加 `group_id`),服务端签发 Presigned PUT。PUT 必须原样带返回的 `headers`(`Content-Type`、`Content-Length`、`If-None-Match: *`)。同一 key 第二次 PUT 为 412。
文件名是 **16 位 base62 随机串**(12 字节),不占用 Snowflake。路径不含 uid。
| purpose | object_key | 生命周期 |
| --- | --- | --- |
| `avatar` | `perm/avatar/{yyyyMMdd}/{name}.{ext}` | 永久 |
| `group_avatar` | `perm/group/{群号}/{name}.{ext}` | 永久 |
| `chat_image` | `chat/image/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
| `chat_voice` | `chat/voice/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
| `chat_video` | `chat/video/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
| `chat_gif` | `chat/gif/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
| `chat_file` | `chat/file/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
日期 UTC `yyyyMMdd`。`ext` 由 mime 映射,未知为 `bin`。收藏复制到 `perm/fav/{yyyyMMdd}/{name}.{ext}`。
合法 key:以 `perm/` 或 `chat/` 开头;禁止 `..`、空白、`\`、前导 `/`;最长 512。
发图片:客户端只 PUT 原图,再 `POST /v1/files/variants` 让服务端出缩略图和预览。消息里只存 key。
进程访问 RustFS 用内网 `object_storage.endpoint`(Docker 里是 `rustfs:9000`),签给客户端的 URL 用 `public_endpoint`(`https://sparkr.wyrs365.com`)。
---
## 目录
```text
cmd/server 进程入口
cmd/migrate 按序执行 migrations/*.sql
configs/config.yaml 本机开发配置
deploy/docker/ 生产镜像、compose、deploy.sh
deploy/nginx/ 443 SNI(host/ 是这台 ECS 的 drop-in)
docs/architecture-v3.md 架构全文
docs/clients/ TCP Java / Swift demo
internal/bootstrap 读配置、接线、生命周期
internal/transport/http Gin;handler 只绑定
internal/transport/tcp gnet Frame + TaskPool
internal/module/{biz}/entity 领域类型,无 I/O
internal/module/{biz}/model 一张表一个 Go 文件(GORM + COMMENT)
internal/module/{biz}/logic 一个功能一个文件,禁止 import model
internal/pkg errs / i18n / reply / reqctx / validate
internal/infrastructure MySQL / Redis / Snowflake / zap / RustFS
migrations/ SQL,改完必须在本机跑 migrate
```
`logic` 不 import `model`。接线在 `internal/bootstrap`。表和列必须有 COMMENT。
---
## 本机开发
需要:Go 1.26、MySQL 8、Redis。对象存储可空(上传接口会不可用)。
```bash
# 配置 configs/config.yaml(env: dev)
make migrate # 应用 migrations/*.sql
make start # go run ./cmd/server -config configs/config.yaml
# 或
make run # 先编 bin/spark-server 再启动
```
覆盖配置:`make start CONFIG=path/to.yaml`。
```bash
make test
make lint # funlen 100,lll 240
```
探活:
```bash
curl -sS http://127.0.0.1:8080/healthz
curl -sS http://127.0.0.1:8080/readyz
```
`env: prod`(或 `SPARK_ENV=prod`)会拒绝空 MySQL DSN、空 Redis、以及默认 JWT secret。
改了 `migrations/*.sql` 后自己跑 `make migrate`,不要留给部署。
---
## 部署(ECS)
`make deploy` → `deploy/docker/deploy.sh`。默认 `root@47.236.66.9`,密钥 `~/.ssh/spark-ecs`。
主机目录:
| 路径 | 内容 |
| --- | --- |
| `/data/spark-server/app` | 构建源码 |
| `/data/spark-server/configs/config.yaml` | 运行配置(来自 `deploy/docker/config.docker.yaml`) |
| `/data/spark-server/logs` | 进程日志 |
| `/data/spark-server/images/spark-server.tar.gz` | `docker save` |
容器加入已有 `shell_default` 网络(mysql / redis / rustfs)。发布:
- `127.0.0.1:18080` → 容器 HTTP 8080
- `127.0.0.1:9090` → 容器 gnet 9090
Nginx drop-in:`deploy/nginx/host/`,安装脚本 `bash deploy/nginx/install-host.sh`。不要覆盖整份 `/etc/nginx/nginx.conf`。
Snowflake `snowflake_worker_id` 每进程唯一。时钟回拨会拒绝发号。
---
## 日志与 ID
- zap 控制台和/或滚动文件(`log.console` / `log.file`)。GORM SQL 走同一套 zap。
- HTTP `request_id`:11 位 base62,响应体 + `X-Request-ID`。
- 对象文件名:16 位 base62,与 Snowflake 号段分开。
- Snowflake 只给 uid、MsgId、会话 id、收藏行 id 等业务主键。
---
## 相关文档
| 文档 | 内容 |
| --- | --- |
| [docs/architecture-v3.md](docs/architecture-v3.md) | 架构全文 |
| `GET /docs` | OpenAPI / Swagger |
| `GET /docs/enums` | 可跳转枚举 |
| `GET /docs/errors` | 信封错误码 |
| [deploy/nginx/README.md](deploy/nginx/README.md) | 443 SNI |
| `docs/clients/` | TCP 客户端示例 |
| `.cursor/skills/spark-im-v3/` | 实现约定(给 Agent) |
| `.golangci.yaml` | 代码规范 |

View File

@ -84,6 +84,7 @@ message:
object_storage:
endpoint: "47.236.66.9:9000"
public_endpoint: "https://sparkr.wyrs365.com"
access_key: "rustfs_spark"
secret_key: 'kWvGtYT/~q+35q;3dSI<^P!MH+bgrpBg'
bucket: "spark"

View File

@ -84,6 +84,7 @@ message:
object_storage:
endpoint: "rustfs:9000"
public_endpoint: "https://sparkr.wyrs365.com"
access_key: "rustfs_spark"
secret_key: 'kWvGtYT/~q+35q;3dSI<^P!MH+bgrpBg'
bucket: "spark"

View File

@ -33,6 +33,6 @@ rsync -az --delete \
"${SSH[@]}" "docker save spark-server:latest | gzip -c > $DATA/images/spark-server.tar.gz"
"${SSH[@]}" "cd $REMOTE/deploy/docker && docker compose -f docker-compose.yaml up -d spark-server"
"${SSH[@]}" "docker exec spark-server /usr/local/bin/spark-migrate -config /app/configs/config.yaml -dir /app/migrations"
"${SSH[@]}" "cp $REMOTE/deploy/nginx/host/spark-sni.conf /etc/nginx/stream.d/spark-sni.conf && nginx -t && systemctl reload nginx"
"${SSH[@]}" "cp $REMOTE/deploy/nginx/host/spark-sni.conf /etc/nginx/stream.d/spark-sni.conf && if [ -f /etc/letsencrypt/live/sparkr.wyrs365.com/fullchain.pem ]; then cp $REMOTE/deploy/nginx/host/sparkr.conf /etc/nginx/conf.d/sparkr.conf; fi && nginx -t && systemctl reload nginx"
"${SSH[@]}" "docker ps --filter name=spark-server --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
"${SSH[@]}" "curl -fsS http://127.0.0.1:18080/healthz && echo && curl -fsS http://127.0.0.1:18080/readyz && echo"

View File

@ -6,6 +6,8 @@ gnet v2 has no native TLS. IM certificates live here, not in spark-server `tcp.t
Client --TLS--> :443 ssl_preread
├─ SNI sparkapi.wyrs365.com → 127.0.0.1:8443 (HTTP TLS)
│ → proxy_pass http://127.0.0.1:18080 + X-Real-IP
├─ SNI sparkr.wyrs365.com → 127.0.0.1:8443 (HTTP TLS)
│ → proxy_pass http://127.0.0.1:9000 RustFS S3
└─ SNI sparkim.wyrs365.com → 127.0.0.1:19001 (IM TLS)
→ PROXY protocol → 127.0.0.1:9090 gnet
```
@ -13,6 +15,7 @@ Client --TLS--> :443 ssl_preread
On this ECS, `:443` is shared with other wyrs HTTPS sites. Do **not** replace `/etc/nginx/nginx.conf`. Drop-ins:
- `host/sparkapi.conf` → `/etc/nginx/conf.d/`
- `host/sparkr.conf` → `/etc/nginx/conf.d/` (RustFS S3; DNS A record `sparkr.wyrs365.com` → this host)
- `host/spark-sni.conf` → `/etc/nginx/stream.d/`
Existing `listen 443 ssl` vhosts move to `127.0.0.1:8443 ssl proxy_protocol` so SNI can take public 443. `00-spark-http.conf` sets `real_ip_header proxy_protocol` and `port_in_redirect off` so `$remote_addr` is the client and redirects stay on 443.
@ -20,9 +23,10 @@ Existing `listen 443 ssl` vhosts move to `127.0.0.1:8443 ssl proxy_protocol` so
spark-server (Docker):
- HTTP `127.0.0.1:18080` → container `:8080`
- TCP `127.0.0.1:9090` with `proxy_protocol: true` → container `:9090` (host `:9000` is rustfs S3)
- TCP `127.0.0.1:9090` with `proxy_protocol: true` → container `:9090` (host `:9000` is rustfs S3; bind `127.0.0.1:9000`, not public)
- Image / config / logs: `/data/spark-server/{images,configs,logs}`
- Public security group: 443 only
- Clients upload/download at `https://sparkr.wyrs365.com` (path-style). spark-server talks to `rustfs:9000` on the Docker network and signs URLs with `object_storage.public_endpoint`.
```bash
bash deploy/nginx/install-host.sh

View File

@ -3,7 +3,7 @@
server {
listen 80;
listen [::]:80;
server_name sparkapi.wyrs365.com sparkim.wyrs365.com;
server_name sparkapi.wyrs365.com sparkim.wyrs365.com sparkr.wyrs365.com;
location /.well-known/acme-challenge/ {
root /var/www/certbot;

View File

@ -1,6 +1,6 @@
# Spark IM + shared HTTPS on :443 (ssl_preread).
# sparkim.wyrs365.com → IM TLS terminator → PROXY plaintext to gnet :9090
# default SNI → existing HTTP sites on 127.0.0.1:8443 (proxy_protocol restores RealIP)
# default SNI → 127.0.0.1:8443 HTTP TLS (sparkapi / sparkr / other wyrs sites; PROXY restores RealIP)
map $ssl_preread_server_name $spark_sni_backend {
sparkim.wyrs365.com 127.0.0.1:19001;

View File

@ -0,0 +1,58 @@
# Spark RustFS S3 — sparkr.wyrs365.com
# Public :443 is stream ssl_preread (see stream.d/spark-sni.conf).
# TLS terminates on loopback :8443; proxy HTTP to Docker rustfs on :9000.
# Do not rewrite the URI: SigV4 signs host + path.
upstream spark_rustfs {
server 127.0.0.1:9000;
keepalive 32;
}
server {
listen 80;
listen [::]:80;
server_name sparkr.wyrs365.com;
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 127.0.0.1:8443 ssl proxy_protocol;
listen [::1]:8443 ssl proxy_protocol;
http2 on;
server_name sparkr.wyrs365.com;
ssl_certificate /etc/letsencrypt/live/sparkr.wyrs365.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/sparkr.wyrs365.com/privkey.pem;
ssl_session_timeout 1d;
ssl_session_cache shared:SparkSSL:10m;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Content-Type-Options nosniff always;
ignore_invalid_headers off;
client_max_body_size 100m;
proxy_buffering off;
proxy_request_buffering off;
proxy_connect_timeout 300s;
proxy_send_timeout 300s;
proxy_read_timeout 300s;
proxy_http_version 1.1;
chunked_transfer_encoding off;
location / {
proxy_pass http://spark_rustfs;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Connection "";
proxy_cache_convert_head off;
}
}

View File

@ -5,33 +5,61 @@ set -euo pipefail
ROOT="$(cd "$(dirname "$0")" && pwd)"
HOST_DIR="$ROOT/host"
BACKUP="/root/nginx-backup-$(date +%Y%m%d%H%M%S)"
DOMAINS=(sparkapi.wyrs365.com sparkim.wyrs365.com sparkr.wyrs365.com)
need_certs() {
[[ -f /etc/letsencrypt/live/sparkapi.wyrs365.com/fullchain.pem ]] \
&& [[ -f /etc/letsencrypt/live/sparkim.wyrs365.com/fullchain.pem ]]
cert_ok() {
[[ -f "/etc/letsencrypt/live/$1/fullchain.pem" ]]
}
install_acme() {
all_certs_ok() {
local d
for d in "${DOMAINS[@]}"; do
cert_ok "$d" || return 1
done
}
install_acme_domain() {
local d=$1
mkdir -p /var/www/certbot
cp "$HOST_DIR/spark-acme.conf" /etc/nginx/conf.d/spark-acme.conf
cat > "/etc/nginx/conf.d/spark-acme-${d}.conf" <<EOF
server {
listen 80;
listen [::]:80;
server_name ${d};
location /.well-known/acme-challenge/ {
root /var/www/certbot;
}
location / {
return 404;
}
}
EOF
nginx -t
systemctl reload nginx
}
issue_certs() {
certbot certonly --webroot -w /var/www/certbot \
--non-interactive --agree-tos --keep-until-expiring \
-d sparkapi.wyrs365.com
certbot certonly --webroot -w /var/www/certbot \
--non-interactive --agree-tos --keep-until-expiring \
-d sparkim.wyrs365.com
issue_missing_certs() {
local d
for d in "${DOMAINS[@]}"; do
if cert_ok "$d"; then
continue
fi
install_acme_domain "$d"
certbot certonly --webroot -w /var/www/certbot \
--non-interactive --agree-tos --keep-until-expiring \
-d "$d"
rm -f "/etc/nginx/conf.d/spark-acme-${d}.conf"
done
}
rewrite_http_443() {
local f
for f in /etc/nginx/conf.d/*.conf; do
case "$(basename "$f")" in
sparkapi.conf|spark-acme.conf) continue ;;
sparkapi.conf|sparkr.conf|spark-acme.conf|00-spark-http.conf) continue ;;
spark-acme-*) continue ;;
esac
if grep -qE 'listen[[:space:]]+127\.0\.0\.1:8443' "$f"; then
continue
@ -48,8 +76,10 @@ rewrite_http_443() {
install_sni() {
rewrite_http_443
rm -f /etc/nginx/conf.d/spark-acme.conf
rm -f /etc/nginx/conf.d/spark-acme-*.conf
cp "$HOST_DIR/00-spark-http.conf" /etc/nginx/conf.d/00-spark-http.conf
cp "$HOST_DIR/sparkapi.conf" /etc/nginx/conf.d/sparkapi.conf
cp "$HOST_DIR/sparkr.conf" /etc/nginx/conf.d/sparkr.conf
cp "$HOST_DIR/spark-sni.conf" /etc/nginx/stream.d/spark-sni.conf
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK'
@ -69,12 +99,11 @@ mkdir -p "$BACKUP"
cp -a /etc/nginx "$BACKUP/nginx"
echo "backup $BACKUP"
if ! need_certs; then
install_acme
issue_certs
if ! all_certs_ok; then
issue_missing_certs
fi
if ! need_certs; then
if ! all_certs_ok; then
echo "certificates missing" >&2
exit 1
fi

View File

@ -138,6 +138,7 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge
authLimit := ratelimit.New(cfg.RateLimit.AuthPerSecond)
objects, err := objectstore.Open(&objectstore.Options{
Endpoint: cfg.ObjectStorage.Endpoint,
PublicEndpoint: cfg.ObjectStorage.PublicEndpoint,
AccessKey: cfg.ObjectStorage.AccessKey,
SecretKey: cfg.ObjectStorage.SecretKey,
Bucket: cfg.ObjectStorage.Bucket,
@ -166,7 +167,7 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge
friends := wireFriends(idgen, db, users, deliverQueue, chat)
groups := wireGroups(idgen, db, users, friends, deliverQueue, objects, chat)
messages := wireMessages(idgen, friends, groups, deliverQueue, chat)
files := wireFiles(cfg, objects, idgen, groups)
files := wireFiles(cfg, objects, groups)
favorites := wireFavorites(idgen, db, objects, chat)
app.tcp = tcp.NewServer(log, pool, sessions, &tcp.Options{
Addr: cfg.Server.TCP.Addr,
@ -276,11 +277,11 @@ func wireGroups(idgen *id.Generator, db *mysql.DB, users *userlogic.Service, fri
return grouplogic.NewService(repo, users, friends, idgen, chat.msgs, q, objects)
}
func wireFiles(cfg *Config, store *objectstore.Store, idgen *id.Generator, groups *grouplogic.Service) *uploadlogic.Service {
if store == nil || !store.Ready() || idgen == nil {
func wireFiles(cfg *Config, store *objectstore.Store, groups *grouplogic.Service) *uploadlogic.Service {
if store == nil || !store.Ready() {
return nil
}
return uploadlogic.NewService(store, idgen, uploadlogic.Options{
return uploadlogic.NewService(store, uploadlogic.Options{
TTL: cfg.ObjectStorage.PresignTTL,
Limit: ratelimit.New(cfg.RateLimit.UploadPerSecond),
Groups: groups,

View File

@ -118,6 +118,7 @@ type RateLimitConfig struct {
type ObjectStorageConfig struct {
Endpoint string `yaml:"endpoint"`
PublicEndpoint string `yaml:"public_endpoint"`
AccessKey string `yaml:"access_key"`
SecretKey string `yaml:"secret_key"`
Bucket string `yaml:"bucket"`
@ -229,6 +230,9 @@ func applyDefaults(cfg *Config) {
if v := os.Getenv("SPARK_S3_ENDPOINT"); v != "" {
cfg.ObjectStorage.Endpoint = v
}
if v := os.Getenv("SPARK_S3_PUBLIC_ENDPOINT"); v != "" {
cfg.ObjectStorage.PublicEndpoint = v
}
if v := os.Getenv("SPARK_S3_ACCESS_KEY"); v != "" {
cfg.ObjectStorage.AccessKey = v
}

View File

@ -18,13 +18,24 @@ import (
"spark-server/internal/pkg/errs"
)
const maxObjectBytes = 4 << 20
const (
maxGetBytes = 32 << 20
maxPutBytes = 4 << 20
)
func (s *Store) PresignPut(ctx context.Context, key, contentType string, ttl time.Duration) (string, error) {
func (s *Store) PresignPut(ctx context.Context, key, contentType string, size int64, ttl time.Duration) (string, error) {
if !s.Ready() {
return "", errs.ErrAvailable
}
in := &s3.PutObjectInput{Bucket: aws.String(s.bucket), Key: aws.String(key)}
if size < 1 {
return "", errs.ErrInvalidArgument
}
in := &s3.PutObjectInput{
Bucket: aws.String(s.bucket),
Key: aws.String(key),
ContentLength: aws.Int64(size),
IfNoneMatch: aws.String("*"),
}
if contentType != "" {
in.ContentType = aws.String(contentType)
}
@ -61,11 +72,11 @@ func (s *Store) Get(ctx context.Context, key string) ([]byte, error) {
return nil, mapObjectErr(err)
}
defer func() { _ = out.Body.Close() }()
data, err := io.ReadAll(io.LimitReader(out.Body, maxObjectBytes+1))
data, err := io.ReadAll(io.LimitReader(out.Body, maxGetBytes+1))
if err != nil {
return nil, mapObjectErr(err)
}
if len(data) > maxObjectBytes {
if len(data) > maxGetBytes {
return nil, errs.ErrInvalidArgument
}
return data, nil
@ -75,7 +86,7 @@ func (s *Store) Put(ctx context.Context, key string, body []byte, contentType st
if !s.Ready() {
return errs.ErrAvailable
}
if len(body) == 0 || len(body) > maxObjectBytes {
if len(body) == 0 || len(body) > maxPutBytes {
return errs.ErrInvalidArgument
}
in := &s3.PutObjectInput{

View File

@ -29,6 +29,7 @@ type Store struct {
// Options configure Open. Empty Endpoint skips the connection.
type Options struct {
Endpoint string
PublicEndpoint string
AccessKey string
SecretKey string
Bucket string
@ -42,7 +43,7 @@ func Open(o *Options) (*Store, error) {
if o == nil || strings.TrimSpace(o.Endpoint) == "" {
return &Store{}, nil
}
host, secure, err := parseEndpoint(o.Endpoint, o.UseSSL)
client, presignBase, err := newClients(o)
if err != nil {
return nil, err
}
@ -50,19 +51,14 @@ func Open(o *Options) (*Store, error) {
if bucket == "" {
bucket = "spark"
}
s := &Store{client: client, presign: s3.NewPresignClient(presignBase), bucket: bucket, log: o.Log}
if s.log == nil {
s.log = zap.NewNop()
}
region := strings.TrimSpace(o.Region)
if region == "" {
region = defaultRegion
}
scheme := "http"
if secure {
scheme = "https"
}
client := newS3Client(region, scheme+"://"+host, o.AccessKey, o.SecretKey)
s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: bucket, log: o.Log}
if s.log == nil {
s.log = zap.NewNop()
}
ctx, cancel := context.WithTimeout(context.Background(), defaultTimeout)
defer cancel()
if err := s.ensureBucket(ctx, region, o.ChatExpireDays); err != nil {
@ -83,6 +79,26 @@ func (s *Store) Ping(ctx context.Context) error {
return err
}
func newClients(o *Options) (*s3.Client, *s3.Client, error) {
region := strings.TrimSpace(o.Region)
if region == "" {
region = defaultRegion
}
host, secure, err := parseEndpoint(o.Endpoint, o.UseSSL)
if err != nil {
return nil, nil, err
}
client := newS3Client(region, endpointURL(host, secure), o.AccessKey, o.SecretKey)
if pub := strings.TrimSpace(o.PublicEndpoint); pub != "" {
phost, psecure, err := parseEndpoint(pub, true)
if err != nil {
return nil, nil, err
}
return client, newS3Client(region, endpointURL(phost, psecure), o.AccessKey, o.SecretKey), nil
}
return client, client, nil
}
func newS3Client(region, baseURL, access, secret string) *s3.Client {
return s3.New(s3.Options{
Region: region,
@ -94,8 +110,15 @@ func newS3Client(region, baseURL, access, secret string) *s3.Client {
})
}
func endpointURL(host string, secure bool) string {
if secure {
return "https://" + host
}
return "http://" + host
}
func parseEndpoint(raw string, useSSL bool) (host string, secure bool, err error) {
raw = strings.TrimSpace(raw)
raw = strings.TrimSpace(strings.TrimRight(raw, "/"))
switch {
case raw == "":
return "", false, fmt.Errorf("objectstore: empty endpoint")

View File

@ -40,13 +40,17 @@ func TestOpenEmpty(t *testing.T) {
func TestPresignLocal(t *testing.T) {
client := newS3Client(defaultRegion, "http://127.0.0.1:9000", "ak", "sk")
s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: "spark"}
u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", time.Hour)
u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 1024, time.Hour)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(u, "127.0.0.1:9000/spark/chat/a.jpg") {
t.Fatalf("%s", u)
}
signed := strings.ToLower(u)
if !strings.Contains(signed, "content-length") || !strings.Contains(signed, "if-none-match") {
t.Fatalf("unsigned size/overwrite headers: %s", u)
}
g, err := s.PresignGet(t.Context(), "perm/avatar/1.jpg", time.Minute)
if err != nil {
t.Fatal(err)
@ -56,6 +60,37 @@ func TestPresignLocal(t *testing.T) {
}
}
func TestPresignPutRejectsZeroSize(t *testing.T) {
client := newS3Client(defaultRegion, "http://127.0.0.1:9000", "ak", "sk")
s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: "spark"}
if _, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 0, time.Hour); !errors.Is(err, errs.ErrInvalidArgument) {
t.Fatalf("%v", err)
}
}
func TestPresignPublicEndpoint(t *testing.T) {
internal, public, err := newClients(&Options{
Endpoint: "http://rustfs:9000",
PublicEndpoint: "https://sparkr.wyrs365.com/",
AccessKey: "ak",
SecretKey: "sk",
})
if err != nil {
t.Fatal(err)
}
s := &Store{client: internal, presign: s3.NewPresignClient(public), bucket: "spark"}
u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 1024, time.Hour)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(u, "https://sparkr.wyrs365.com/spark/chat/a.jpg") {
t.Fatalf("%s", u)
}
if strings.Contains(u, "rustfs:9000") {
t.Fatalf("internal host leaked: %s", u)
}
}
func TestGetPutNotReady(t *testing.T) {
s, err := Open(&Options{})
if err != nil {

View File

@ -7,9 +7,10 @@ import (
msgentity "spark-server/internal/module/message/entity"
uploadentity "spark-server/internal/module/upload/entity"
"spark-server/internal/pkg/errs"
"spark-server/internal/pkg/reqctx"
)
func (s *Service) copyMedia(ctx context.Context, userID int64, payload *msgentity.Content) ([]string, error) {
func (s *Service) copyMedia(ctx context.Context, payload *msgentity.Content) ([]string, error) {
keys := payload.ObjectKeys()
if len(keys) == 0 {
return nil, nil
@ -24,12 +25,7 @@ func (s *Service) copyMedia(ctx context.Context, userID int64, payload *msgentit
if _, ok := mapping[src]; ok {
continue
}
objectID, err := s.ids.NextID()
if err != nil {
s.deleteKeys(ctx, copied)
return nil, err
}
dst := uploadentity.BuildFavKey(userID, objectID, day, uploadentity.ExtOfKey(src))
dst := uploadentity.BuildFavKey(reqctx.NewObjectName(), day, uploadentity.ExtOfKey(src))
if err := s.objects.Copy(ctx, src, dst); err != nil {
s.deleteKeys(ctx, copied)
return nil, err

View File

@ -47,7 +47,7 @@ func (s *Service) Create(ctx context.Context, userID int64, in *CreateInput) (*I
if err != nil {
return nil, err
}
copied, err := s.copyMedia(ctx, userID, payload)
copied, err := s.copyMedia(ctx, payload)
if err != nil {
return nil, err
}

View File

@ -56,8 +56,12 @@ func TestFavoriteCopiesMediaAndExpires(t *testing.T) {
t.Fatal(err)
}
src := "chat/20260917/1/99.jpg"
thumb := "chat/20260917/1/t.jpg"
orig := "chat/20260917/1/o.jpg"
objs.keys[src] = true
raw, err := msgentity.NewImage(&msgentity.Image{Key: src, Mime: "image/jpeg"}).Marshal()
objs.keys[thumb] = true
objs.keys[orig] = true
raw, err := msgentity.NewImage(&msgentity.Image{Key: src, ThumbKey: thumb, OrigKey: orig, Mime: "image/jpeg"}).Marshal()
if err != nil {
t.Fatal(err)
}
@ -76,10 +80,10 @@ func TestFavoriteCopiesMediaAndExpires(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if item.MsgID != msgID || !strings.Contains(string(item.Content), "perm/fav/1/") {
if item.MsgID != msgID || !strings.Contains(string(item.Content), "perm/fav/") {
t.Fatalf("%+v", item)
}
if strings.Contains(string(item.Content), src) {
if strings.Contains(string(item.Content), src) || strings.Contains(string(item.Content), orig) {
t.Fatal("still points at chat key")
}
again, err := svc.Create(ctx, 1, &CreateInput{MsgID: msgID})

View File

@ -13,6 +13,7 @@ import (
"spark-server/internal/module/group/entity"
uploadentity "spark-server/internal/module/upload/entity"
userentity "spark-server/internal/module/user/entity"
"spark-server/internal/pkg/reqctx"
)
const collageJPEG = "image/jpeg"
@ -63,10 +64,7 @@ func (s *Service) putAvatar(ctx context.Context, g *entity.Group, raw []byte, mi
if g == nil || len(raw) == 0 {
return false
}
key := uploadentity.PrefixGroup + strconv.FormatInt(g.ID, 10) + "/default." + ext
if objectID, err := s.ids.NextID(); err == nil {
key = uploadentity.BuildObjectKey(uploadentity.PurposeGroupAvatar, g.ID, objectID, "", ext)
}
key := uploadentity.BuildObjectKey(uploadentity.PurposeGroupAvatar, g.ID, reqctx.NewObjectName(), "", ext)
if err := s.objects.Put(ctx, key, raw, mime); err != nil {
return false
}

View File

@ -121,7 +121,7 @@ func (c *Content) Validate() error {
case TypeText:
return validateText(c.Text)
case TypeImage:
return validateMediaKey("image", c.Image.Key)
return validateImage(c.Image)
case TypeVoice:
return validateVoice(c.Voice)
case TypeVideo:
@ -220,6 +220,16 @@ func validateMentions(c *Content) error {
return nil
}
func validateImage(img *Image) error {
if err := validateMediaKey("image", img.Key); err != nil {
return err
}
if img.Size < 0 || img.OrigSize < 0 || img.Width < 0 || img.Height < 0 || img.OrigW < 0 || img.OrigH < 0 {
return fmt.Errorf("content: image size")
}
return nil
}
func validateMediaKey(kind, key string) error {
if strings.TrimSpace(key) == "" {
return fmt.Errorf("content: %s key empty", kind)
@ -279,14 +289,21 @@ func (c *Content) ObjectKeys() []string {
var keys []string
add := func(k string) {
k = strings.TrimSpace(k)
if k != "" {
keys = append(keys, k)
if k == "" {
return
}
for _, existing := range keys {
if existing == k {
return
}
}
keys = append(keys, k)
}
switch {
case c.Image != nil:
add(c.Image.Key)
add(c.Image.ThumbKey)
add(c.Image.OrigKey)
case c.Voice != nil:
add(c.Voice.Key)
case c.Video != nil:
@ -316,6 +333,7 @@ func (c *Content) ReplaceObjectKeys(mapping map[string]string) {
if c.Image != nil {
repl(&c.Image.Key)
repl(&c.Image.ThumbKey)
repl(&c.Image.OrigKey)
}
if c.Voice != nil {
repl(&c.Voice.Key)

View File

@ -80,6 +80,9 @@ func TestRejectsRawStringAndEmptyKey(t *testing.T) {
if _, err := NewImage(&Image{}).Marshal(); err == nil || !strings.Contains(err.Error(), "key") {
t.Fatalf("image key: %v", err)
}
if _, err := NewImage(&Image{Key: "k", OrigSize: -1}).Marshal(); err == nil {
t.Fatal("negative orig_size")
}
bad := &Content{Type: TypeText, Text: &Text{Body: "x"}, Image: &Image{Key: "k"}}
if _, err := bad.Marshal(); err == nil {
t.Fatal("two payloads")
@ -122,13 +125,21 @@ func TestMentionsRoundTrip(t *testing.T) {
}
func TestObjectKeysAndReplace(t *testing.T) {
img := NewImage(&Image{Key: "chat/a.jpg", ThumbKey: "chat/t.jpg"})
img := NewImage(&Image{Key: "chat/a.jpg", ThumbKey: "chat/t.jpg", OrigKey: "chat/o.jpg", OrigSize: 4 << 20})
keys := img.ObjectKeys()
if len(keys) != 2 || keys[0] != "chat/a.jpg" {
if len(keys) != 3 || keys[0] != "chat/a.jpg" || keys[2] != "chat/o.jpg" {
t.Fatalf("%v", keys)
}
img.ReplaceObjectKeys(map[string]string{"chat/a.jpg": "perm/fav/1/x.jpg", "chat/t.jpg": "perm/fav/1/y.jpg"})
if img.Image.Key != "perm/fav/1/x.jpg" || img.Image.ThumbKey != "perm/fav/1/y.jpg" {
img.ReplaceObjectKeys(map[string]string{
"chat/a.jpg": "perm/fav/1/x.jpg",
"chat/t.jpg": "perm/fav/1/y.jpg",
"chat/o.jpg": "perm/fav/1/z.jpg",
})
if img.Image.Key != "perm/fav/1/x.jpg" || img.Image.ThumbKey != "perm/fav/1/y.jpg" || img.Image.OrigKey != "perm/fav/1/z.jpg" {
t.Fatalf("%+v", img.Image)
}
same := NewImage(&Image{Key: "chat/a.jpg", OrigKey: "chat/a.jpg"})
if got := same.ObjectKeys(); len(got) != 1 {
t.Fatalf("dedupe %v", got)
}
}

View File

@ -5,7 +5,10 @@ type Text struct {
Body string `json:"body"`
}
// Image references an uploaded object. Bytes themselves are not stored in messages.content.
// Image references uploaded objects. Bytes are not stored in messages.content.
// WeChat-style: bubble uses thumb_key, tap-to-view uses key (compressed),
// 「查看原图」uses orig_key when the sender opted in.
// Clients PUT JPEG/PNG only (convert HEIC/WebP locally); the server does not transcode.
type Image struct {
Key string `json:"key"`
Mime string `json:"mime,omitempty"`
@ -13,6 +16,10 @@ type Image struct {
Height int `json:"h,omitempty"`
Size int64 `json:"size,omitempty"`
ThumbKey string `json:"thumb_key,omitempty"`
OrigKey string `json:"orig_key,omitempty"`
OrigSize int64 `json:"orig_size,omitempty"`
OrigW int `json:"orig_w,omitempty"`
OrigH int `json:"orig_h,omitempty"`
}
// Voice is an audio clip in object storage.

View File

@ -9,6 +9,11 @@ type Credential struct {
ExpiresIn int64 `json:"expires_in"`
}
// CredentialSet is one HTTP round-trip for up to three PUTs (thumb / preview / orig).
type CredentialSet struct {
Items []Credential `json:"items"`
}
type Download struct {
ObjectKey string `json:"object_key"`
DownloadURL string `json:"download_url"`

View File

@ -16,11 +16,16 @@ const (
PurposeChatGIF = "chat_gif"
PurposeChatFile = "chat_file"
PrefixPerm = "perm/"
PrefixChat = "chat/"
PrefixAvatar = "perm/avatar/"
PrefixGroup = "perm/group/"
PrefixFav = "perm/fav/"
PrefixPerm = "perm/"
PrefixChat = "chat/"
PrefixAvatar = "perm/avatar/"
PrefixGroup = "perm/group/"
PrefixFav = "perm/fav/"
PrefixChatImage = "chat/image/"
PrefixChatVoice = "chat/voice/"
PrefixChatVideo = "chat/video/"
PrefixChatGIF = "chat/gif/"
PrefixChatFile = "chat/file/"
MethodPUT = "PUT"
MethodGET = "GET"
@ -28,7 +33,8 @@ const (
const (
maxAvatar = 2 << 20
maxImage = 10 << 20
maxImage = 32 << 20
maxGIF = 10 << 20
maxVoice = 16 << 20
maxVideo = 100 << 20
maxFile = 50 << 20
@ -75,25 +81,46 @@ func ExtOfKey(key string) string {
return ext
}
func BuildObjectKey(purpose string, ownerID, objectID int64, day, ext string) string {
func BuildObjectKey(purpose string, ownerID int64, name, day, ext string) string {
if ext == "" {
ext = extDefault
}
if name == "" {
name = "0"
}
switch purpose {
case PurposeAvatar:
return fmt.Sprintf("%s%d/%s/%d.%s", PrefixAvatar, ownerID, day, objectID, ext)
return fmt.Sprintf("%s%s/%s.%s", PrefixAvatar, day, name, ext)
case PurposeGroupAvatar:
return fmt.Sprintf("%s%d/%d.%s", PrefixGroup, ownerID, objectID, ext)
return fmt.Sprintf("%s%d/%s.%s", PrefixGroup, ownerID, name, ext)
default:
return fmt.Sprintf("%s%s/%d/%d.%s", PrefixChat, day, ownerID, objectID, ext)
return fmt.Sprintf("%s%s/%s/%s.%s", PrefixChat, chatKind(purpose), day, name, ext)
}
}
func BuildFavKey(userID, objectID int64, day, ext string) string {
func BuildFavKey(name, day, ext string) string {
if ext == "" {
ext = extDefault
}
return fmt.Sprintf("%s%d/%s/%d.%s", PrefixFav, userID, day, objectID, ext)
if name == "" {
name = "0"
}
return fmt.Sprintf("%s%s/%s.%s", PrefixFav, day, name, ext)
}
func chatKind(purpose string) string {
switch purpose {
case PurposeChatImage:
return "image"
case PurposeChatVoice:
return "voice"
case PurposeChatVideo:
return "video"
case PurposeChatGIF:
return "gif"
default:
return "file"
}
}
func ValidObjectKey(key string) bool {
@ -110,13 +137,13 @@ var specs = map[string]Spec{
PurposeAvatar: {Purpose: PurposeAvatar, MaxSize: maxAvatar, MIMEs: imageMIMEs},
PurposeGroupAvatar: {Purpose: PurposeGroupAvatar, MaxSize: maxAvatar, MIMEs: imageMIMEs},
PurposeChatImage: {Purpose: PurposeChatImage, MaxSize: maxImage, MIMEs: imageMIMEs},
PurposeChatGIF: {Purpose: PurposeChatGIF, MaxSize: maxImage, MIMEs: gifMIMEs},
PurposeChatGIF: {Purpose: PurposeChatGIF, MaxSize: maxGIF, MIMEs: gifMIMEs},
PurposeChatVoice: {Purpose: PurposeChatVoice, MaxSize: maxVoice, MIMEs: voiceMIMEs},
PurposeChatVideo: {Purpose: PurposeChatVideo, MaxSize: maxVideo, MIMEs: videoMIMEs},
PurposeChatFile: {Purpose: PurposeChatFile, MaxSize: maxFile, MIMEs: fileMIMEs},
}
var imageMIMEs = set("image/jpeg", "image/jpg", "image/png", "image/webp", "image/heic", "image/heif")
var imageMIMEs = set("image/jpeg", "image/jpg", "image/png")
var gifMIMEs = set("image/gif", "image/webp")
var voiceMIMEs = set("audio/mpeg", "audio/mp4", "audio/aac", "audio/amr", "audio/ogg", "audio/wav", "audio/webm", "audio/x-m4a")
var videoMIMEs = set("video/mp4", "video/quicktime", "video/webm")
@ -129,7 +156,7 @@ var fileMIMEs = set(
var mimeExt = map[string]string{
"image/jpeg": "jpg", "image/jpg": "jpg", "image/png": "png", "image/webp": "webp",
"image/heic": "heic", "image/heif": "heif", "image/gif": "gif",
"image/gif": "gif",
"audio/mpeg": "mp3", "audio/mp4": "m4a", "audio/aac": "aac", "audio/amr": "amr",
"audio/ogg": "ogg", "audio/wav": "wav", "audio/webm": "weba", "audio/x-m4a": "m4a",
"video/mp4": "mp4", "video/quicktime": "mov", "video/webm": "webm",

View File

@ -6,20 +6,24 @@ import (
)
func TestBuildObjectKeyByPurpose(t *testing.T) {
avatar := BuildObjectKey(PurposeAvatar, 11, 99, "20260917", "jpg")
if avatar != "perm/avatar/11/20260917/99.jpg" {
avatar := BuildObjectKey(PurposeAvatar, 11, "99", "20260917", "jpg")
if avatar != "perm/avatar/20260917/99.jpg" {
t.Fatalf("avatar %s", avatar)
}
chat := BuildObjectKey(PurposeChatImage, 11, 99, "20260917", "jpg")
if chat != "chat/20260917/11/99.jpg" {
chat := BuildObjectKey(PurposeChatImage, 11, "99", "20260917", "jpg")
if chat != "chat/image/20260917/99.jpg" {
t.Fatalf("chat %s", chat)
}
group := BuildObjectKey(PurposeGroupAvatar, 7, 99, "20260917", "png")
voice := BuildObjectKey(PurposeChatVoice, 11, "aa", "20260917", "m4a")
if voice != "chat/voice/20260917/aa.m4a" {
t.Fatalf("voice %s", voice)
}
group := BuildObjectKey(PurposeGroupAvatar, 7, "99", "20260917", "png")
if group != "perm/group/7/99.png" {
t.Fatalf("group %s", group)
}
fav := BuildFavKey(11, 7, "20260917", "m4a")
if fav != "perm/fav/11/20260917/7.m4a" {
fav := BuildFavKey("99", "20260917", "m4a")
if fav != "perm/fav/20260917/99.m4a" {
t.Fatalf("fav %s", fav)
}
}
@ -49,7 +53,19 @@ func TestNormalizeMIMEAndExt(t *testing.T) {
if _, ok := spec.MIMEs["audio/mpeg"]; !ok {
t.Fatal("voice mime")
}
if strings.Contains(BuildObjectKey(PurposeAvatar, 1, 1, "d", "jpg"), PrefixChat) {
img, ok := LookupSpec(PurposeChatImage)
if !ok {
t.Fatal("image spec")
}
if _, ok := img.MIMEs["image/jpeg"]; !ok {
t.Fatal("jpeg")
}
for _, mime := range []string{"image/heic", "image/heif", "image/webp"} {
if _, ok := img.MIMEs[mime]; ok {
t.Fatalf("interchange must not allow %s", mime)
}
}
if strings.Contains(BuildObjectKey(PurposeAvatar, 1, "1", "d", "jpg"), PrefixChat) {
t.Fatal("avatar must be perm")
}
}

View File

@ -0,0 +1,49 @@
package logic
import (
"context"
"spark-server/internal/module/upload/entity"
"spark-server/internal/pkg/errs"
"spark-server/internal/pkg/reqctx"
"spark-server/internal/pkg/validate"
)
const maxCredentialBatch = 3
// BatchCreateInput issues up to three Presigned PUTs in one call (thumb / preview / orig).
type BatchCreateInput struct {
Items []CreateInput `json:"items" validate:"required,min=1,max=3,dive" label:"field.upload_items" errmsg:"required:validate.upload.items_required;min:validate.upload.items_required;max:validate.upload.items_invalid"`
}
func (s *Service) CreateCredentials(ctx context.Context, userID int64, in *BatchCreateInput) (*entity.CredentialSet, error) {
if !s.ready() {
return nil, errs.ErrAvailable
}
if userID <= 0 {
return nil, errs.ErrUnauthorized
}
if in == nil {
return nil, errs.ErrInvalidArgument
}
if err := validate.StructLang(reqctx.Lang(ctx), in); err != nil {
return nil, err
}
if n := len(in.Items); n < 1 || n > maxCredentialBatch {
return nil, errs.ErrInvalidArgument
}
for range in.Items {
if !s.allow(userID) {
return nil, errs.ErrRateLimited
}
}
items := make([]entity.Credential, 0, len(in.Items))
for i := range in.Items {
cred, err := s.issuePut(ctx, userID, &in.Items[i])
if err != nil {
return nil, err
}
items = append(items, *cred)
}
return &entity.CredentialSet{Items: items}, nil
}

View File

@ -2,6 +2,7 @@ package logic
import (
"context"
"strconv"
"spark-server/internal/module/upload/entity"
"spark-server/internal/pkg/errs"
@ -34,14 +35,15 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create
if !s.allow(userID) {
return nil, errs.ErrRateLimited
}
return s.issuePut(ctx, userID, in)
}
func (s *Service) issuePut(ctx context.Context, userID int64, in *CreateInput) (*entity.Credential, error) {
_, mime, err := matchSpec(ctx, in)
if err != nil {
return nil, err
}
objectID, err := s.ids.NextID()
if err != nil {
return nil, err
}
name := reqctx.NewObjectName()
ownerID := userID
if in.Purpose == entity.PurposeGroupAvatar {
if err := s.readyGroupAvatar(ctx, userID, in.GroupID); err != nil {
@ -49,8 +51,8 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create
}
ownerID = in.GroupID
}
key := entity.BuildObjectKey(in.Purpose, ownerID, objectID, dayOf(s.now()), entity.ExtForMIME(mime))
url, err := s.store.PresignPut(ctx, key, mime, s.ttl)
key := entity.BuildObjectKey(in.Purpose, ownerID, name, dayOf(s.now()), entity.ExtForMIME(mime))
url, err := s.store.PresignPut(ctx, key, mime, in.Size, s.ttl)
if err != nil {
return nil, err
}
@ -58,7 +60,11 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create
ObjectKey: key,
UploadURL: url,
Method: entity.MethodPUT,
Headers: map[string]string{"Content-Type": mime},
Headers: map[string]string{
"Content-Type": mime,
"Content-Length": strconv.FormatInt(in.Size, 10),
"If-None-Match": "*",
},
ExpiresIn: s.expiresIn(),
}, nil
}

View File

@ -7,7 +7,6 @@ import (
"testing"
"time"
"spark-server/internal/infrastructure/id"
"spark-server/internal/module/upload/entity"
"spark-server/internal/pkg/errs"
"spark-server/internal/pkg/reqctx"
@ -18,7 +17,7 @@ type fakeStore struct {
putErr error
}
func (f *fakeStore) PresignPut(_ context.Context, key, _ string, _ time.Duration) (string, error) {
func (f *fakeStore) PresignPut(_ context.Context, key, _ string, _ int64, _ time.Duration) (string, error) {
if f.putErr != nil {
return "", f.putErr
}
@ -37,11 +36,7 @@ func (f *fakeStore) PresignGet(_ context.Context, key string, _ time.Duration) (
func testUpload(t *testing.T) *Service {
t.Helper()
ids, err := id.New(1)
if err != nil {
t.Fatal(err)
}
return NewService(&fakeStore{}, ids, Options{
return NewService(&fakeStore{}, Options{
TTL: 5 * time.Minute,
Now: func() time.Time { return time.Date(2026, 9, 17, 8, 0, 0, 0, time.UTC) },
})
@ -54,11 +49,13 @@ func TestCreateCredentialChatVsAvatar(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if !strings.HasPrefix(chat.ObjectKey, "chat/20260917/42/") || chat.Method != "PUT" || chat.Headers["Content-Type"] != "image/jpeg" {
if !strings.HasPrefix(chat.ObjectKey, "chat/image/20260917/") || chat.Method != "PUT" ||
chat.Headers["Content-Type"] != "image/jpeg" || chat.Headers["Content-Length"] != "1024" ||
chat.Headers["If-None-Match"] != "*" || strings.Contains(chat.ObjectKey, "/42/") {
t.Fatalf("%+v", chat)
}
avatar, err := svc.CreateCredential(ctx, 42, &CreateInput{Purpose: entity.PurposeAvatar, Mime: "image/png", Size: 2048})
if err != nil || !strings.HasPrefix(avatar.ObjectKey, "perm/avatar/42/20260917/") {
if err != nil || !strings.HasPrefix(avatar.ObjectKey, "perm/avatar/20260917/") {
t.Fatalf("%+v %v", avatar, err)
}
group, err := svc.CreateCredential(ctx, 42, &CreateInput{Purpose: entity.PurposeGroupAvatar, Mime: "image/jpeg", Size: 1024, GroupID: 7})
@ -79,6 +76,11 @@ func TestCreateCredentialRejectsBadMIMEAndSize(t *testing.T) {
if _, err := svc.CreateCredential(ctx, 1, &CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: 50 << 20}); err == nil {
t.Fatal("huge image")
}
for _, mime := range []string{"image/heic", "image/heif", "image/webp"} {
if _, err := svc.CreateCredential(ctx, 1, &CreateInput{Purpose: entity.PurposeChatImage, Mime: mime, Size: 1024}); err == nil {
t.Fatalf("accepted %s", mime)
}
}
if _, err := svc.CreateCredential(ctx, 0, &CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: 1}); !errors.Is(err, errs.ErrUnauthorized) {
t.Fatalf("uid %v", err)
}
@ -95,3 +97,24 @@ func TestCreateDownloadValidatesKey(t *testing.T) {
t.Fatal("path escape")
}
}
func TestCreateCredentialsBatch(t *testing.T) {
svc := testUpload(t)
ctx := reqctx.With(t.Context(), &reqctx.RequestContext{UserID: 42, Language: "zh-CN"})
part := func(size int64) CreateInput {
return CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: size}
}
got, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{Items: []CreateInput{part(1024), part(2048), part(4096)}})
if err != nil || got == nil || len(got.Items) != 3 {
t.Fatalf("%+v %v", got, err)
}
if got.Items[0].ObjectKey == got.Items[1].ObjectKey || got.Items[1].Headers["Content-Length"] != "2048" {
t.Fatalf("%+v", got.Items)
}
if _, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{}); err == nil {
t.Fatal("empty")
}
if _, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{Items: []CreateInput{part(1), part(1), part(1), part(1)}}); err == nil {
t.Fatal("too many")
}
}

View File

@ -10,15 +10,10 @@ import (
// Presigner issues short-lived object-storage URLs. Infrastructure implements it.
type Presigner interface {
PresignPut(ctx context.Context, key, contentType string, ttl time.Duration) (string, error)
PresignPut(ctx context.Context, key, contentType string, size int64, ttl time.Duration) (string, error)
PresignGet(ctx context.Context, key string, ttl time.Duration) (string, error)
}
type IDGen interface {
NextID() (int64, error)
}
// AvatarACL gates group_avatar uploads. Nil skips the check (tests).
type AvatarACL interface {
CanSetAvatar(ctx context.Context, userID, groupID int64) error
}
@ -33,14 +28,13 @@ type Options struct {
// Service issues upload/download credentials. HTTP and TCP share it.
type Service struct {
store Presigner
ids IDGen
ttl time.Duration
limit *ratelimit.Limiter
now func() time.Time
groups AvatarACL
}
func NewService(store Presigner, ids IDGen, opt Options) *Service {
func NewService(store Presigner, opt Options) *Service {
ttl := opt.TTL
if ttl <= 0 {
ttl = 15 * time.Minute
@ -49,11 +43,11 @@ func NewService(store Presigner, ids IDGen, opt Options) *Service {
if clock == nil {
clock = func() time.Time { return time.Now().UTC() }
}
return &Service{store: store, ids: ids, ttl: ttl, limit: opt.Limit, now: clock, groups: opt.Groups}
return &Service{store: store, ttl: ttl, limit: opt.Limit, now: clock, groups: opt.Groups}
}
func (s *Service) ready() bool {
return s != nil && s.store != nil && s.ids != nil
return s != nil && s.store != nil
}
func (s *Service) expiresIn() int64 {

View File

@ -7,6 +7,7 @@ import (
uploadentity "spark-server/internal/module/upload/entity"
"spark-server/internal/module/user/entity"
"spark-server/internal/pkg/reqctx"
)
const defaultAvatarMIME = "image/png"
@ -42,18 +43,14 @@ func (s *Service) stampDefaultAvatar(ctx context.Context, u *entity.User) {
}
func (s *Service) putLetterPNG(ctx context.Context, u *entity.User) string {
if s == nil || s.objects == nil || s.ids == nil || u == nil || u.ID <= 0 {
if s == nil || s.objects == nil || u == nil || u.ID <= 0 {
return ""
}
raw, err := entity.LetterAvatarPNG(u.Nickname)
if err != nil || len(raw) == 0 {
return ""
}
objectID, err := s.ids.NextID()
if err != nil {
return ""
}
key := uploadentity.BuildObjectKey(uploadentity.PurposeAvatar, u.ID, objectID, time.Now().Format("20060102"), "png")
key := uploadentity.BuildObjectKey(uploadentity.PurposeAvatar, u.ID, reqctx.NewObjectName(), time.Now().UTC().Format("20060102"), "png")
if err := s.objects.Put(ctx, key, raw, defaultAvatarMIME); err != nil {
return ""
}

View File

@ -27,10 +27,15 @@ func TestAuthenticateTestEmailAnyCode(t *testing.T) {
if u1.ID <= 0 || !entity.ValidSparkID(u1.SparkID) || u1.Nickname != "test01" || !entity.ValidQRCode(u1.QRCode) {
t.Fatalf("%+v", u1)
}
prefix := "perm/avatar/" + strconv.FormatInt(u1.ID, 10) + "/"
if !strings.HasPrefix(u1.Avatar, prefix) || !strings.HasSuffix(u1.Avatar, ".png") {
if !strings.HasPrefix(u1.Avatar, "perm/avatar/") || !strings.HasSuffix(u1.Avatar, ".png") {
t.Fatalf("default avatar %s", u1.Avatar)
}
uid := strconv.FormatInt(u1.ID, 10)
for _, part := range strings.Split(u1.Avatar, "/") {
if part == uid {
t.Fatalf("avatar leaked uid %s", u1.Avatar)
}
}
store, ok := svc.objects.(*memObjects)
if !ok {
t.Fatal("mem objects")

View File

@ -11,6 +11,9 @@ func init() {
KeyFieldSize: {
ZhCN: "文件大小", En: "File size", ZhTW: "檔案大小",
},
KeyFieldUploadItems: {
ZhCN: "上传列表", En: "Upload list", ZhTW: "上傳列表",
},
KeyFieldObjectKey: {
ZhCN: "对象键", En: "Object key", ZhTW: "物件鍵",
},
@ -50,6 +53,12 @@ func init() {
KeyUploadGroupRequired: {
ZhCN: "请选择群", En: "Please choose a group", ZhTW: "請選擇群",
},
KeyUploadItemsRequired: {
ZhCN: "请填写要上传的文件", En: "Please provide files to upload", ZhTW: "請填寫要上傳的檔案",
},
KeyUploadItemsInvalid: {
ZhCN: "一次最多签发 3 个上传凭证", En: "At most 3 upload credentials per request", ZhTW: "一次最多簽發 3 個上傳憑證",
},
KeyFavoriteMsgRequired: {
ZhCN: "请选择消息", En: "Please choose a message", ZhTW: "請選擇訊息",
},

View File

@ -42,6 +42,7 @@ const (
KeyFieldPurpose = "field.purpose"
KeyFieldMime = "field.mime"
KeyFieldSize = "field.size"
KeyFieldUploadItems = "field.upload_items"
KeyFieldObjectKey = "field.object_key"
KeyFieldMsgID = "field.msg_id"
KeyFieldClientMsgID = "field.client_msg_id"
@ -90,6 +91,8 @@ const (
KeyUploadKeyRequired = "validate.upload.key_required"
KeyUploadKeyInvalid = "validate.upload.key_invalid"
KeyUploadGroupRequired = "validate.upload.group_required"
KeyUploadItemsRequired = "validate.upload.items_required"
KeyUploadItemsInvalid = "validate.upload.items_invalid"
KeyFavoriteMsgRequired = "validate.favorite.msg_required"
KeyFavoriteMsgInvalid = "validate.favorite.msg_invalid"
KeyMessageClientMsgRequired = "validate.message.client_msg_required"

View File

@ -3,7 +3,8 @@ package reqctx
import (
"context"
"crypto/rand"
"encoding/hex"
"encoding/binary"
"math/big"
"net"
"strings"
"time"
@ -91,11 +92,51 @@ func stripHost(v string) string {
return v
}
// NewID returns a 16-byte hex id for request_id.
const (
base62Alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
base62Len = 11
objectNameLen = 16
objectNameSrc = 12
)
// NewID returns an 8-byte base62 id (11 chars) for request_id and JWT jti.
func NewID() string {
var b [16]byte
var b [8]byte
if _, err := rand.Read(b[:]); err != nil {
return "00000000000000000000000000000000"
return "00000000000"
}
return hex.EncodeToString(b[:])
return encodeBase62(binary.BigEndian.Uint64(b[:]))
}
// NewObjectName returns a 12-byte base62 id (16 chars) for object_key filenames. Not Snowflake.
func NewObjectName() string {
return randomBase62(objectNameSrc, objectNameLen)
}
func encodeBase62(n uint64) string {
var out [base62Len]byte
for i := base62Len - 1; i >= 0; i-- {
out[i] = base62Alphabet[n%62]
n /= 62
}
return string(out[:])
}
func randomBase62(byteN, width int) string {
if byteN < 1 || width < 1 {
return ""
}
b := make([]byte, byteN)
if _, err := rand.Read(b); err != nil {
return strings.Repeat("0", width)
}
n := new(big.Int).SetBytes(b)
out := make([]byte, width)
base := big.NewInt(62)
r := new(big.Int)
for i := width - 1; i >= 0; i-- {
n.DivMod(n, base, r)
out[i] = base62Alphabet[r.Int64()]
}
return string(out)
}

View File

@ -1,6 +1,9 @@
package reqctx
import "testing"
import (
"strings"
"testing"
)
func TestParseHTTPRealIP(t *testing.T) {
got := ParseHTTPRealIP(" 203.0.113.10 ", "198.51.100.1, 192.0.2.1", "192.0.2.1:1234")
@ -27,6 +30,48 @@ func TestRealIPFromContext(t *testing.T) {
}
}
func TestNewIDLength(t *testing.T) {
id := NewID()
if len(id) != 11 {
t.Fatalf("len %d %q", len(id), id)
}
for i := 0; i < len(id); i++ {
if !strings.ContainsRune(base62Alphabet, rune(id[i])) {
t.Fatalf("char %q in %q", id[i], id)
}
}
if id == NewID() {
t.Fatal("not unique")
}
}
func TestNewObjectNameLength(t *testing.T) {
id := NewObjectName()
if len(id) != 16 {
t.Fatalf("len %d %q", len(id), id)
}
for i := 0; i < len(id); i++ {
if !strings.ContainsRune(base62Alphabet, rune(id[i])) {
t.Fatalf("char %q in %q", id[i], id)
}
}
if id == NewObjectName() {
t.Fatal("not unique")
}
}
func TestEncodeBase62(t *testing.T) {
if got := encodeBase62(0); got != "00000000000" {
t.Fatalf("%q", got)
}
if got := encodeBase62(61); got != "0000000000z" {
t.Fatalf("%q", got)
}
if got := encodeBase62(62); got != "00000000010" {
t.Fatalf("%q", got)
}
}
func TestRequestIDFromContext(t *testing.T) {
ctx := With(t.Context(), &RequestContext{RequestID: "abc"})
if RequestID(ctx) != "abc" {

View File

@ -7,7 +7,7 @@ info:
## 环境与地址
公网只开放 `443`。Nginx 按 TLS SNI 分流:HTTP 证书打到 `sparkapi.wyrs365.com`,IM 证书打到 `sparkim.wyrs365.com`。进程本身只绑本机。
公网只开放 `443`。Nginx 按 TLS SNI 分流:HTTP 证书打到 `sparkapi.wyrs365.com`,IM 证书打到 `sparkim.wyrs365.com`,对象存储(RustFS S3)打到 `sparkr.wyrs365.com`。进程本身只绑本机。
| 环境 | 用途 | HTTP Base URL | IM 长连接 |
| --- | --- | --- | --- |
@ -36,7 +36,7 @@ info:
| --- | --- | --- |
| `code` | **字符串** | `"0"` 成功;其它失败。不要用数字 `0` 比较。全表见 [错误码](/docs/errors)。 |
| `msg` | 字符串 | 给人看的文案,随请求语言变化(`zh-CN` / `en` / `zh-TW`)。成功一般为 `ok`。 |
| `request_id` | 字符串 | 服务端生成的本次请求 id。客户端不用传;从响应体读取。与 Header `X-Request-ID` 相同。 |
| `request_id` | 字符串 | 服务端生成,11 位 base62(`0-9A-Za-z`)。客户端不用传;从响应体读取。与 Header `X-Request-ID` 相同。 |
| `data` | object / array | 成功时为业务对象;失败时固定 `{}`。字段缺失当空对象,不要当 `null`。 |
**判定成功只看 `code === "0"`。** 不要只看 HTTP 状态码。
@ -655,7 +655,7 @@ paths:
uid: 1940000000000000000
spark_id: alice_ok
nickname: test01
avatar: perm/avatar/1940000000000000000/20260917/1940000000000000001.png
avatar: perm/avatar/20260917/3H8kPq2mN1Rabcde.png
qrcode: "-NDveu-9Q"
birthday: ""
gender: 0
@ -1106,9 +1106,13 @@ paths:
description: |
客户端只传 `purpose` / `mime` / `size`,服务端生成 object_key 并签发短时 Presigned PUT。
文件直传 RustFS,不经过 Gin。
`avatar` 写入 `perm/avatar/{uid}/…`(不过期);`group_avatar` 写入 `perm/group/{群号}/…`(需 `group_id`,群主/管理员);
聊天媒体写入 `chat/{yyyyMMdd}/{uid}/…`(桶生命周期过期)。
客户端 PUT 时必须带返回的 `headers`(至少 `Content-Type`)。消息与头像只存 `object_key`。
`avatar` 写入 `perm/avatar/{yyyyMMdd}/{name}.{ext}`(不过期,路径不含 uid);
`group_avatar` 写入 `perm/group/{群号}/{name}.{ext}`(需 `group_id`,群主/管理员);
聊天按类型:`chat/image|voice|video|gif|file/{yyyyMMdd}/{name}.{ext}`(桶生命周期按前缀过期)。
`{name}` 为服务端 16 位 base62 随机串,不占用 Snowflake。
发图片:客户端压成 JPEG/PNG 后 `POST /v1/files/batch` 直传缩略图、压缩大图、原图(未勾选原图只传前两项)。
服务端不转码。iOS HEIC、Android WebP 须客户端先转 JPEG/PNG,两端才能互看。
客户端 PUT 必须原样带返回的 `headers`。消息只存 object_key。
operationId: createUploadCredential
requestBody:
required: true
@ -1131,12 +1135,61 @@ paths:
code: "0"
msg: ok
data:
object_key: chat/20260917/1/2.jpg
upload_url: http://47.236.66.9:9000/spark/chat/20260917/1/2.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256
object_key: chat/image/20260917/3H8kPq2mN1Rabcde.jpg
upload_url: https://sparkr.wyrs365.com/spark/chat/image/20260917/3H8kPq2mN1Rabcde.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256
method: PUT
headers:
Content-Type: image/jpeg
Content-Length: "245760"
If-None-Match: "*"
expires_in: 900
/v1/files/batch:
post:
tags: [files]
summary: 批量签发直传凭证
description: |
一次返回 1~3 个 Presigned PUT,顺序与 `items` 相同。聊天图片请按「缩略图、压缩大图、原图」排列;未勾选原图只传前两项。
mime 仅 `image/jpeg` / `image/png`(HEIC、WebP 请客户端先转)。限流按条数计入 `upload_per_second`。文件仍直传 RustFS。
operationId: createUploadCredentials
requestBody:
required: true
content:
application/json:
schema:
$ref: "#/components/schemas/CreateUploadBatchRequest"
example:
items:
- purpose: chat_image
mime: image/jpeg
size: 8192
- purpose: chat_image
mime: image/jpeg
size: 245760
- purpose: chat_image
mime: image/jpeg
size: 3500000
responses:
"200":
description: 统一 envelope
content:
application/json:
schema:
$ref: "#/components/schemas/Envelope"
example:
code: "0"
msg: ok
data:
items:
- object_key: chat/image/20260917/3H8kPq2mN1Rabcde.jpg
method: PUT
headers:
Content-Type: image/jpeg
Content-Length: "8192"
If-None-Match: "*"
- object_key: chat/image/20260917/4kPq2mN1RabcdeXY.jpg
method: PUT
- object_key: chat/image/20260917/5mN1RabcdeXY12ab.jpg
method: PUT
/v1/files/downloads:
post:
tags: [files]
@ -1181,7 +1234,7 @@ paths:
tags: [favorites]
summary: 收藏一条消息
description: |
会话成员可将消息收入收藏。图片/语音/视频/文件会 CopyObject 到 `perm/fav/{uid}/…`。
会话成员可将消息收入收藏。图片/语音/视频/文件会 CopyObject 到 `perm/fav/{yyyyMMdd}/{name}.{ext}`。
源对象已从 `chat/` 过期删除则失败(`code=40402`),不提供本地补传。
同一 `msg_id` 重复收藏返回已有记录。撤回不影响已收藏快照。
operationId: createFavorite
@ -1477,8 +1530,8 @@ components:
example: ok
request_id:
type: string
description: 服务端生成的本次请求 id。客户端不用传,从响应体读取。
example: a1b2c3d4e5f6a7b8a1b2c3d4e5f6a7b8
description: 服务端生成的本次请求 id(11 位 base62,`0-9A-Za-z`)。客户端不用传,从响应体读取。
example: 3H8kPq2mN1R
data:
type: object
additionalProperties: true
@ -1743,6 +1796,9 @@ components:
mime:
type: string
example: image/jpeg
description: |
avatar / group_avatar / chat_image 仅 image/jpeg、image/jpg、image/png。
iOS HEIC、Android WebP 须客户端转成 JPEG/PNG 再申请凭证。
size:
type: integer
format: int64
@ -1752,6 +1808,16 @@ components:
format: int64
minimum: 1
description: purpose=group_avatar 时必填,为自增群号
CreateUploadBatchRequest:
type: object
required: [items]
properties:
items:
type: array
minItems: 1
maxItems: 3
items:
$ref: "#/components/schemas/CreateUploadRequest"
CreateDownloadRequest:
type: object
required: [object_key]
@ -1792,6 +1858,39 @@ components:
properties:
body:
type: string
image:
type: object
description: |
t=2。点开看 `key`(压缩预览);气泡用 `thumb_key`;`orig_key` 有则显示「查看原图」。
三档由客户端压成 JPEG/PNG 后 `POST /v1/files/batch` 直传。服务端不转码;不要传 HEIC/WebP。
required: [key]
properties:
key:
type: string
description: 点开查看用的压缩图 object_key
thumb_key:
type: string
description: 气泡缩略图,可省略
orig_key:
type: string
description: 原图 object_key;空则没有「查看原图」
orig_size:
type: integer
format: int64
description: 原图像素文件大小,供「查看原图 3.2MB」按钮
orig_w:
type: integer
orig_h:
type: integer
mime:
type: string
w:
type: integer
h:
type: integer
size:
type: integer
format: int64
at_all:
type: boolean
description: "@所有人,仅群主/管理员"

View File

@ -27,11 +27,11 @@ import (
type stubPresign struct{}
func (stubPresign) PresignPut(_ context.Context, key, _ string, _ time.Duration) (string, error) {
func (s *stubPresign) PresignPut(_ context.Context, key, _ string, _ int64, _ time.Duration) (string, error) {
return "http://rustfs.test/" + key + "?sig=1", nil
}
func (stubPresign) PresignGet(_ context.Context, key string, _ time.Duration) (string, error) {
func (s *stubPresign) PresignGet(_ context.Context, key string, _ time.Duration) (string, error) {
return "http://rustfs.test/" + key + "?get=1", nil
}
@ -48,7 +48,7 @@ func testFileEngine(t *testing.T) (*ginEngineBundle, *id.Generator, *msgmodel.Me
users := userlogic.NewService(usermodel.NewMemoryRepo(), usermodel.NewMemoryCodes(), ids, nil)
msgs := msgmodel.NewMemoryRepo()
convs := convmodel.NewMemoryP2P()
files := uploadlogic.NewService(stubPresign{}, ids, uploadlogic.Options{
files := uploadlogic.NewService(&stubPresign{}, uploadlogic.Options{
Now: func() time.Time { return time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC) },
})
favs := favlogic.NewService(favmodel.NewMemoryRepo(), msgs, convs, nil, ids)
@ -71,13 +71,25 @@ func TestUploadCredentialHTTP(t *testing.T) {
ObjectKey string `json:"object_key"`
Method string `json:"method"`
}
if err := json.Unmarshal(env.Data, &cred); err != nil || cred.Method != "PUT" || !strings.Contains(cred.ObjectKey, "chat/20260917/") {
if err := json.Unmarshal(env.Data, &cred); err != nil || cred.Method != "PUT" || !strings.Contains(cred.ObjectKey, "chat/image/20260917/") {
t.Fatalf("%s %v", env.Data, err)
}
bad := authJSON(t, h, &tok, http.MethodPost, "/v1/files", `{"purpose":"avatar","mime":"application/pdf","size":10}`)
if bad.Code != errs.ErrInvalidArgument.Code {
t.Fatalf("mime %+v", bad)
}
batch := authJSON(t, h, &tok, http.MethodPost, "/v1/files/batch", `{"items":[{"purpose":"chat_image","mime":"image/jpeg","size":1024},{"purpose":"chat_image","mime":"image/jpeg","size":2048}]}`)
if batch.Code != "0" {
t.Fatalf("batch %+v", batch)
}
var set struct {
Items []struct {
ObjectKey string `json:"object_key"`
} `json:"items"`
}
if err := json.Unmarshal(batch.Data, &set); err != nil || len(set.Items) != 2 || set.Items[0].ObjectKey == set.Items[1].ObjectKey {
t.Fatalf("%s %v", batch.Data, err)
}
}
func TestFavoriteTextHTTP(t *testing.T) {

View File

@ -36,6 +36,24 @@ func (h *Handler) Create(c *gin.Context) {
respond.JSON(c, out)
}
func (h *Handler) CreateBatch(c *gin.Context) {
uid, ok := currentUID(c)
if !ok {
return
}
var in uploadlogic.BatchCreateInput
if err := respond.BindJSON(c, &in); err != nil {
respond.JSONError(c, err)
return
}
out, err := h.files.CreateCredentials(c.Request.Context(), uid, &in)
if err != nil {
respond.JSONError(c, err)
return
}
respond.JSON(c, out)
}
func (h *Handler) Download(c *gin.Context) {
uid, ok := currentUID(c)
if !ok {

View File

@ -132,6 +132,7 @@ func registerFileRoutes(v1 *gin.RouterGroup, deps *EngineDeps) {
h := filehandler.New(deps.Files)
g := v1.Group("/files")
g.POST("", h.Create)
g.POST("/batch", h.CreateBatch)
g.POST("/downloads", h.Download)
}

View File

@ -34,7 +34,7 @@ func TestHealthzEnvelopeAndTraceHeaders(t *testing.T) {
t.Fatalf("status %d", w.Code)
}
got := w.Header().Get("X-Request-ID")
if len(got) != 32 {
if len(got) != 11 {
t.Fatalf("server request id %q headers %v", got, w.Header())
}
if w.Header().Get("X-Trace-ID") != "" {
@ -47,7 +47,7 @@ func TestHealthzEnvelopeAndTraceHeaders(t *testing.T) {
if env.Code != "0" || env.Msg != "ok" {
t.Fatalf("%+v", env)
}
if env.RequestID != got || len(env.RequestID) != 32 {
if env.RequestID != got || len(env.RequestID) != 11 {
t.Fatalf("body request_id %q header %q", env.RequestID, got)
}
if !bytes.Contains(env.Data, []byte(`"status":"ok"`)) {
@ -82,7 +82,7 @@ func TestReadyzHTTPStatus(t *testing.T) {
if env.Code != errs.ErrAvailable.Code || !bytes.Contains(env.Data, []byte(`"mysql":"down"`)) {
t.Fatalf("%+v", env)
}
if len(env.RequestID) != 32 {
if len(env.RequestID) != 11 {
t.Fatalf("readyz request_id %q", env.RequestID)
}
}