feat: update object key
This commit is contained in:
parent
8aa8e0bad9
commit
c11fbda141
@ -95,7 +95,7 @@ Media: HTTP upload credential, message content is object key only. Frame max 64K
|
|||||||
Message types: text / image / voice / video / emoji / gif / file / location / system (`message/entity.Type` 1–9). `messages.content` stores serialized JSON bytes (BLOB), never a raw string. Closed type/status values: [enums.md](enums.md).
|
Message types: text / image / voice / video / emoji / gif / file / location / system (`message/entity.Type` 1–9). `messages.content` stores serialized JSON bytes (BLOB), never a raw string. Closed type/status values: [enums.md](enums.md).
|
||||||
|
|
||||||
TCP body is JSON envelope `{code,msg,request_id,data}`. AUTH request is `{"token","device_id","app_version","language"}` (not protobuf).
|
TCP body is JSON envelope `{code,msg,request_id,data}`. AUTH request is `{"token","device_id","app_version","language"}` (not protobuf).
|
||||||
Public TLS is Nginx `:443` SNI: HTTP cert on loopback `:8443`, IM cert on loopback `:19001` then PROXY to gnet `:9090` (host `:9000` is rustfs). See `deploy/nginx/`.
|
Public TLS is Nginx `:443` SNI: HTTP cert on loopback `:8443` (`sparkapi` + `sparkr`/RustFS), IM cert on loopback `:19001` then PROXY to gnet `:9090` (host `:9000` is rustfs). See `deploy/nginx/`.
|
||||||
AUTH/login rate-limit uses `reqctx.RealIP`. TaskPool workers recover panics. Deliver workers start with `RequeueStale` + `SETNX deliver:done:{msgId}`.
|
AUTH/login rate-limit uses `reqctx.RealIP`. TaskPool workers recover panics. Deliver workers start with `RequeueStale` + `SETNX deliver:done:{msgId}`.
|
||||||
|
|
||||||
## When adding a feature
|
## When adding a feature
|
||||||
|
|||||||
329
README.md
329
README.md
@ -1,21 +1,322 @@
|
|||||||
# spark-server
|
# spark-server
|
||||||
|
|
||||||
Spark IM modular monolith. Architecture: [docs/architecture-v3.md](docs/architecture-v3.md).
|
Spark IM 服务端。Go 1.26 **模块化单体**(一台机器、一个进程)。公网只开 **443**,HTTP / IM / 对象存储用三个域名、Nginx SNI 分流。
|
||||||
|
|
||||||
- Go 1.26
|
仓库:<https://gitea.wyrs365.com/Spark/spark-server.git>
|
||||||
- Gin HTTP on `127.0.0.1:8080`. JSON envelope `{"code","msg","request_id","data"}`. Docs: `GET /docs` (API), `/docs/enums`, `/docs/errors`. Spec: `GET /openapi.yaml`.
|
完整设计:[docs/architecture-v3.md](docs/architecture-v3.md)
|
||||||
- Probes: `GET /healthz` liveness; `GET /readyz` readiness (HTTP 503 when MySQL/Redis/task pool/accepting are not ready). `env: prod` (or `SPARK_ENV=prod`) rejects empty MySQL DSN, empty Redis, and the default JWT secret.
|
在线协议:`GET /docs`(Swagger)、`/docs/enums`、`/docs/errors`,规范 `GET /openapi.yaml`。
|
||||||
- Logs: zap to console and/or rotating file (`log.console` / `log.file`). GORM SQL goes through the same zap. `request_id` is server-generated and returned in the response body (also as `X-Request-ID`).
|
|
||||||
- gnet TCP on loopback (local `127.0.0.1:8090`; ECS Docker `127.0.0.1:9090` because host `:9000` is rustfs). gnet v2.10 has no built-in TLS; Nginx terminates IM TLS and forwards PROXY+plaintext. Sample: `deploy/nginx/`. `make deploy` writes image/config/logs under `/data/spark-server/`.
|
|
||||||
- TCP Frame body is JSON `{code,msg,request_id,data}`. AUTH body: `{"token","device_id","app_version","language"}`.
|
|
||||||
- Public entry is Nginx `:443` only (SNI split)
|
|
||||||
|
|
||||||
```bash
|
不上 Kafka、K8s、微服务。不要给 gnet 绑公网端口,不要在 EventLoop 里打 MySQL。
|
||||||
make start
|
|
||||||
make migrate
|
---
|
||||||
make lint
|
|
||||||
|
## 它做什么
|
||||||
|
|
||||||
|
即时通讯:登录、好友、群、发消息、增量同步、撤回删除、媒体直传。
|
||||||
|
|
||||||
|
| 层 | 职责 |
|
||||||
|
| --- | --- |
|
||||||
|
| 实时 | TCP 长连接(gnet)。在线推送 `MESSAGE`,心跳 30s,空闲 120s 断开 |
|
||||||
|
| 正文 | MySQL。Snowflake `MsgId` 是主键、排序、同步游标。没有 `ConversationSeq` |
|
||||||
|
| 会话列表 / 未读 | Redis ZSET。群热路径不写 MySQL inbox 行 |
|
||||||
|
| 扇出 | Redis `deliver:queue` + worker。`SEND_ACK` = MySQL COMMIT **且** 入队成功 |
|
||||||
|
| 媒体 | 客户端拿 Presigned URL 直传 RustFS。消息里只存 `object_key` |
|
||||||
|
|
||||||
|
HTTP 和 TCP 调同一套 `internal/module/*/logic`。Gin handler 只做绑定和写信封。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 生产入口
|
||||||
|
|
||||||
|
| 用途 | 域名 | 说明 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| HTTP API | `https://sparkapi.wyrs365.com` | REST `/v1/...`,文档 `/docs` |
|
||||||
|
| IM 长连接 | `sparkim.wyrs365.com:443` | TLS + 自定义 Frame,不是 WebSocket |
|
||||||
|
| 对象存储 | `https://sparkr.wyrs365.com` | RustFS S3 path-style,Presigned PUT/GET |
|
||||||
|
|
||||||
|
本机开发:HTTP `http://127.0.0.1:8080`,TCP 明文 `127.0.0.1:8090`。
|
||||||
|
|
||||||
|
ECS 上进程只绑回环,由 Nginx 终结 TLS:
|
||||||
|
|
||||||
|
```text
|
||||||
|
客户端 :443 (ssl_preread)
|
||||||
|
├─ sparkapi.wyrs365.com → 127.0.0.1:8443 → Gin 127.0.0.1:18080
|
||||||
|
├─ sparkr.wyrs365.com → 127.0.0.1:8443 → RustFS 127.0.0.1:9000
|
||||||
|
└─ sparkim.wyrs365.com → 127.0.0.1:19001 (IM TLS)
|
||||||
|
→ PROXY → gnet 127.0.0.1:9090
|
||||||
```
|
```
|
||||||
|
|
||||||
`make start` runs `cmd/server` against `configs/config.yaml` (override with `make start CONFIG=path/to.yaml`). `make run` builds `bin/spark-server` first, then starts that binary.
|
主机 `:9000` 是 RustFS S3,所以 IM gnet 用 **9090**。安全组只放行 443。
|
||||||
|
|
||||||
Agent conventions live in `.cursor/rules` and `.cursor/skills/spark-im-v3`. Lint conventions live in `.golangci.yaml`.
|
---
|
||||||
|
|
||||||
|
## 硬约束(不要改)
|
||||||
|
|
||||||
|
| ID | 规则 |
|
||||||
|
| --- | --- |
|
||||||
|
| C1 | 消息正文在 MySQL。Snowflake `MsgId` 是 PK / 排序 / 同步游标。没有 ConversationSeq |
|
||||||
|
| C2 | gnet EventLoop 不调 MySQL / Redis / 对象存储 / 扇出。入站进 TaskPool;扇出走 `deliver:queue` |
|
||||||
|
| C3 | 保密靠 TLS。Frame 只有薄头 + 可选 gzip。没有应用层 ECDH/AES |
|
||||||
|
| C4 | 公网只有 443。gnet / Gin 绑回环 |
|
||||||
|
| C5 | `SEND_ACK` 必须在 COMMIT **且** deliver `Push` 成功之后。入队失败不 ACK |
|
||||||
|
| C6 | 限流 / 审计用 PROXY / `X-Real-IP` 的 `RealIP`,不用 gnet `RemoteAddr`(那是 127.0.0.1) |
|
||||||
|
| C7 | 会话列表 / 未读 / 预览在 Redis ZSET。群热路径不 upsert MySQL inbox |
|
||||||
|
|
||||||
|
热路径禁止 `context.TODO()`。推送超时不踢连接。不要把媒体字节写进 `messages.content`。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 统一响应
|
||||||
|
|
||||||
|
HTTP 与 TCP **应答 Body** 同一套信封。业务成败只看 `code` 字符串,不要用数字 `0` 比较。除 `/readyz` 未就绪是 HTTP 503 外,业务失败也是 HTTP 200。
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"code":"0","msg":"ok","request_id":"3H8kPq2mN1R","data":{}}
|
||||||
|
```
|
||||||
|
|
||||||
|
| 字段 | 规则 |
|
||||||
|
| --- | --- |
|
||||||
|
| `code` | `"0"` 成功;其它失败。全表见 [/docs/errors](https://sparkapi.wyrs365.com/docs/errors) |
|
||||||
|
| `msg` | 展示文案,随语言变。成功一般为 `ok` |
|
||||||
|
| `request_id` | 服务端生成,**11 位 base62**(`0-9A-Za-z`)。客户端不用传。HTTP 同时写 Header `X-Request-ID` |
|
||||||
|
| `data` | 成功为业务对象;失败固定 `{}`。缺字段当空对象,不要当 `null` |
|
||||||
|
|
||||||
|
`request_id` 和 TCP Frame 头里的 `RequestID`(uint64)不是一回事:后者是客户端填的配对号,服务端原样带回。
|
||||||
|
|
||||||
|
### HTTP 请求放哪
|
||||||
|
|
||||||
|
| 位置 | 内容 |
|
||||||
|
| --- | --- |
|
||||||
|
| Header | `Authorization: Bearer …`、`X-Device-ID`、`X-Language` 或 `Accept-Language`、`X-App-Version`、`Content-Type` |
|
||||||
|
| Body | POST/PATCH 业务字段。不要把 token / 语言 / device_id 塞进业务 JSON(登录 Body 带 `device_id`,刷新 Body 带 `refresh_token`) |
|
||||||
|
| Path | 资源 id:好友 uid、群号、MsgId |
|
||||||
|
| Query | GET 筛选 / 分页 |
|
||||||
|
|
||||||
|
语言:`zh-CN`(缺省)/ `en` / `zh-TW`。网关写 `X-Real-IP`,客户端不要发。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 登录
|
||||||
|
|
||||||
|
1. (可选)`POST /v1/auth/email-codes` Body `{"email"}`。
|
||||||
|
2. `POST /v1/auth/login` 拿 `access_token` / `refresh_token`。`device_id` 由**客户端生成并持久化**,服务端不分配;写入 JWT `did`。
|
||||||
|
3. 其它 HTTP:`Authorization: Bearer …`,可选 `X-Device-ID`(须等于登录时的 device_id)。
|
||||||
|
4. 续期:`POST /v1/auth/token/refresh`,Body `{"refresh_token"}`,不要带 Bearer。
|
||||||
|
5. 在线发消息、收推送、增量同步走 **TCP**。`POST /v1/messages` 只是 HTTP 调试通道,不要当 App 主路径。
|
||||||
|
|
||||||
|
测试号 `test01@spark.com`~`test10@spark.com` 可跳过发码,登录任意 `verify_code`。
|
||||||
|
|
||||||
|
`uid` 就是 Snowflake `users.id`。对外展示用可变的 `spark_id`,不要把 uid 写进对象存储路径。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## HTTP 资源
|
||||||
|
|
||||||
|
前缀 `/v1/{resource}`。
|
||||||
|
|
||||||
|
| 组 | 路径 | 说明 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 探活 | `/healthz` `/readyz` | 存活;未就绪 HTTP 503 |
|
||||||
|
| 文档 | `/docs` `/docs/enums` `/docs/errors` `/openapi.yaml` | 无需登录 |
|
||||||
|
| 枚举 | `GET /v1/meta/enums` | 无需登录 |
|
||||||
|
| 认证 | `/v1/auth` | 发码、登录、刷新、登出 |
|
||||||
|
| 用户 | `/v1/users/me` | 资料 |
|
||||||
|
| 好友 | `/v1/friends` | 申请、同意、备注、置顶、拉黑、举报 |
|
||||||
|
| 群 | `/v1/groups` | 创建、资料、成员、邀请、踢人、转让、解散 |
|
||||||
|
| 文件 | `/v1/files` | 上传/下载凭证、批量、图片派生 |
|
||||||
|
| 收藏 | `/v1/favorites` | 收藏;媒体会 Copy 到 `perm/fav/` |
|
||||||
|
| 会话 | `/v1/conversations` | 列表、免打扰 |
|
||||||
|
| 消息 | `/v1/messages` | 历史 / 撤回 / 删除(主路径仍是 TCP) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TCP 协议
|
||||||
|
|
||||||
|
不是 WebSocket、不是 HTTP、不是 Protobuf。对 `sparkim.wyrs365.com:443` 做 TLS(SNI 必须是该域名),之后写二进制 Frame。
|
||||||
|
|
||||||
|
### Frame(大端,固定 23 字节头)
|
||||||
|
|
||||||
|
```text
|
||||||
|
offset 0 Magic uint16 0x5350(ASCII "SP")
|
||||||
|
offset 2 Version uint8 1
|
||||||
|
offset 3 Command uint16
|
||||||
|
offset 5 Flags uint16 bit0=1 表示 Body 是 gzip
|
||||||
|
offset 7 RequestID uint64 客户端配对,可 0;响应原样带回
|
||||||
|
offset 15 SequenceID uint32 请求/响应配对;PUSH_ACK 必须带回推送帧的 SequenceID
|
||||||
|
offset 19 BodyLength uint32 Body 字节数(压缩后)
|
||||||
|
offset 23 Body bytes UTF-8 JSON
|
||||||
|
```
|
||||||
|
|
||||||
|
先读 23 字节,再按 `BodyLength` 读 Body。Magic 不对或总长超过 64KB 应断开。Body ≥ 512B 且 gzip 更小时可压缩。
|
||||||
|
|
||||||
|
**请求 Body 是业务 JSON,不要包信封。** 应答和 `MESSAGE` 推送才是 `{code,msg,request_id,data}`。失败时 Command 仍为请求 Command(发送失败不是 `SEND_ACK`)。
|
||||||
|
|
||||||
|
| 值 | 名 | 方向 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| 1 | AUTH | → 请求 `{"token","device_id","app_version","language"}` |
|
||||||
|
| 2 | HEARTBEAT | → 空 Body |
|
||||||
|
| 3 | GENERATE_MSG_ID | → 空;应答 `data.msg_id` |
|
||||||
|
| 4 | SEND_MESSAGE | → 与 HTTP 发消息同一 JSON |
|
||||||
|
| 5 | SEND_ACK | ← 仅 4 的成功应答 |
|
||||||
|
| 6 | MESSAGE | ← 推送 |
|
||||||
|
| 7 | PUSH_ACK | → 空;`SequenceID` = 推送帧的 SequenceID |
|
||||||
|
| 8 | READ_MESSAGE | → |
|
||||||
|
| 9 | RECALL_MESSAGE | → `{"msg_id"}` |
|
||||||
|
| 10 | SYNC_INBOX | → 会话列表增量 |
|
||||||
|
| 12 | SYNC_MESSAGES | → 某会话消息增量 |
|
||||||
|
| 14 | KICK | ← 同设备顶号 |
|
||||||
|
| 15 | DELETE_MESSAGE | → `{"msg_id"}` |
|
||||||
|
|
||||||
|
除 AUTH / HEARTBEAT / PUSH_ACK 外须先 AUTH,否则 `40102`。
|
||||||
|
|
||||||
|
上线同步顺序:**先 `SYNC_INBOX`,再按会话 `SYNC_MESSAGES`**。游标是 Snowflake `MsgId` / inbox score,不是 ConversationSeq。
|
||||||
|
|
||||||
|
Java / Swift 示例:`docs/clients/`,OpenAPI 里也可点开。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 发消息路径
|
||||||
|
|
||||||
|
必须按这个顺序(C5):
|
||||||
|
|
||||||
|
1. TaskPool(禁止在 EventLoop 里做)
|
||||||
|
2. 权限(好友 / 拉黑 / 群禁言)——失败则不 INSERT
|
||||||
|
3. Snowflake MsgId(或客户端 MsgId,须在 ±5 分钟内)
|
||||||
|
4. MySQL 事务:消息表 + `conversations.last_msg_id`
|
||||||
|
5. `DeliverQueue.Push`
|
||||||
|
6. `SEND_ACK`
|
||||||
|
|
||||||
|
第 5 步失败不 ACK。客户端用同一个 `client_msg_id` 重试。发送设备不回推 `MESSAGE`,其它设备必须推。同 `device_id` 顶旧连接,下发 `KICK`。
|
||||||
|
|
||||||
|
私聊与群消息分表存储。消息类型 1–9:文本 / 图片 / 语音 / 视频 / 表情 / gif / 文件 / 位置 / 系统。`content` 是 JSON 序列化后的 **bytes**,不是字符串。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 媒体 `object_key`
|
||||||
|
|
||||||
|
客户端 **不自己拼 key**。`POST /v1/files` 只传 `purpose` / `mime` / `size`(群头像再加 `group_id`),服务端签发 Presigned PUT。PUT 必须原样带返回的 `headers`(`Content-Type`、`Content-Length`、`If-None-Match: *`)。同一 key 第二次 PUT 为 412。
|
||||||
|
|
||||||
|
文件名是 **16 位 base62 随机串**(12 字节),不占用 Snowflake。路径不含 uid。
|
||||||
|
|
||||||
|
| purpose | object_key | 生命周期 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `avatar` | `perm/avatar/{yyyyMMdd}/{name}.{ext}` | 永久 |
|
||||||
|
| `group_avatar` | `perm/group/{群号}/{name}.{ext}` | 永久 |
|
||||||
|
| `chat_image` | `chat/image/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
|
||||||
|
| `chat_voice` | `chat/voice/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
|
||||||
|
| `chat_video` | `chat/video/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
|
||||||
|
| `chat_gif` | `chat/gif/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
|
||||||
|
| `chat_file` | `chat/file/{yyyyMMdd}/{name}.{ext}` | 桶生命周期 |
|
||||||
|
|
||||||
|
日期 UTC `yyyyMMdd`。`ext` 由 mime 映射,未知为 `bin`。收藏复制到 `perm/fav/{yyyyMMdd}/{name}.{ext}`。
|
||||||
|
|
||||||
|
合法 key:以 `perm/` 或 `chat/` 开头;禁止 `..`、空白、`\`、前导 `/`;最长 512。
|
||||||
|
|
||||||
|
发图片:客户端只 PUT 原图,再 `POST /v1/files/variants` 让服务端出缩略图和预览。消息里只存 key。
|
||||||
|
|
||||||
|
进程访问 RustFS 用内网 `object_storage.endpoint`(Docker 里是 `rustfs:9000`),签给客户端的 URL 用 `public_endpoint`(`https://sparkr.wyrs365.com`)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 目录
|
||||||
|
|
||||||
|
```text
|
||||||
|
cmd/server 进程入口
|
||||||
|
cmd/migrate 按序执行 migrations/*.sql
|
||||||
|
configs/config.yaml 本机开发配置
|
||||||
|
deploy/docker/ 生产镜像、compose、deploy.sh
|
||||||
|
deploy/nginx/ 443 SNI(host/ 是这台 ECS 的 drop-in)
|
||||||
|
docs/architecture-v3.md 架构全文
|
||||||
|
docs/clients/ TCP Java / Swift demo
|
||||||
|
internal/bootstrap 读配置、接线、生命周期
|
||||||
|
internal/transport/http Gin;handler 只绑定
|
||||||
|
internal/transport/tcp gnet Frame + TaskPool
|
||||||
|
internal/module/{biz}/entity 领域类型,无 I/O
|
||||||
|
internal/module/{biz}/model 一张表一个 Go 文件(GORM + COMMENT)
|
||||||
|
internal/module/{biz}/logic 一个功能一个文件,禁止 import model
|
||||||
|
internal/pkg errs / i18n / reply / reqctx / validate
|
||||||
|
internal/infrastructure MySQL / Redis / Snowflake / zap / RustFS
|
||||||
|
migrations/ SQL,改完必须在本机跑 migrate
|
||||||
|
```
|
||||||
|
|
||||||
|
`logic` 不 import `model`。接线在 `internal/bootstrap`。表和列必须有 COMMENT。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 本机开发
|
||||||
|
|
||||||
|
需要:Go 1.26、MySQL 8、Redis。对象存储可空(上传接口会不可用)。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 配置 configs/config.yaml(env: dev)
|
||||||
|
make migrate # 应用 migrations/*.sql
|
||||||
|
make start # go run ./cmd/server -config configs/config.yaml
|
||||||
|
# 或
|
||||||
|
make run # 先编 bin/spark-server 再启动
|
||||||
|
```
|
||||||
|
|
||||||
|
覆盖配置:`make start CONFIG=path/to.yaml`。
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make test
|
||||||
|
make lint # funlen 100,lll 240
|
||||||
|
```
|
||||||
|
|
||||||
|
探活:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -sS http://127.0.0.1:8080/healthz
|
||||||
|
curl -sS http://127.0.0.1:8080/readyz
|
||||||
|
```
|
||||||
|
|
||||||
|
`env: prod`(或 `SPARK_ENV=prod`)会拒绝空 MySQL DSN、空 Redis、以及默认 JWT secret。
|
||||||
|
|
||||||
|
改了 `migrations/*.sql` 后自己跑 `make migrate`,不要留给部署。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 部署(ECS)
|
||||||
|
|
||||||
|
`make deploy` → `deploy/docker/deploy.sh`。默认 `root@47.236.66.9`,密钥 `~/.ssh/spark-ecs`。
|
||||||
|
|
||||||
|
主机目录:
|
||||||
|
|
||||||
|
| 路径 | 内容 |
|
||||||
|
| --- | --- |
|
||||||
|
| `/data/spark-server/app` | 构建源码 |
|
||||||
|
| `/data/spark-server/configs/config.yaml` | 运行配置(来自 `deploy/docker/config.docker.yaml`) |
|
||||||
|
| `/data/spark-server/logs` | 进程日志 |
|
||||||
|
| `/data/spark-server/images/spark-server.tar.gz` | `docker save` |
|
||||||
|
|
||||||
|
容器加入已有 `shell_default` 网络(mysql / redis / rustfs)。发布:
|
||||||
|
|
||||||
|
- `127.0.0.1:18080` → 容器 HTTP 8080
|
||||||
|
- `127.0.0.1:9090` → 容器 gnet 9090
|
||||||
|
|
||||||
|
Nginx drop-in:`deploy/nginx/host/`,安装脚本 `bash deploy/nginx/install-host.sh`。不要覆盖整份 `/etc/nginx/nginx.conf`。
|
||||||
|
|
||||||
|
Snowflake `snowflake_worker_id` 每进程唯一。时钟回拨会拒绝发号。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 日志与 ID
|
||||||
|
|
||||||
|
- zap 控制台和/或滚动文件(`log.console` / `log.file`)。GORM SQL 走同一套 zap。
|
||||||
|
- HTTP `request_id`:11 位 base62,响应体 + `X-Request-ID`。
|
||||||
|
- 对象文件名:16 位 base62,与 Snowflake 号段分开。
|
||||||
|
- Snowflake 只给 uid、MsgId、会话 id、收藏行 id 等业务主键。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 相关文档
|
||||||
|
|
||||||
|
| 文档 | 内容 |
|
||||||
|
| --- | --- |
|
||||||
|
| [docs/architecture-v3.md](docs/architecture-v3.md) | 架构全文 |
|
||||||
|
| `GET /docs` | OpenAPI / Swagger |
|
||||||
|
| `GET /docs/enums` | 可跳转枚举 |
|
||||||
|
| `GET /docs/errors` | 信封错误码 |
|
||||||
|
| [deploy/nginx/README.md](deploy/nginx/README.md) | 443 SNI |
|
||||||
|
| `docs/clients/` | TCP 客户端示例 |
|
||||||
|
| `.cursor/skills/spark-im-v3/` | 实现约定(给 Agent) |
|
||||||
|
| `.golangci.yaml` | 代码规范 |
|
||||||
|
|||||||
@ -84,6 +84,7 @@ message:
|
|||||||
|
|
||||||
object_storage:
|
object_storage:
|
||||||
endpoint: "47.236.66.9:9000"
|
endpoint: "47.236.66.9:9000"
|
||||||
|
public_endpoint: "https://sparkr.wyrs365.com"
|
||||||
access_key: "rustfs_spark"
|
access_key: "rustfs_spark"
|
||||||
secret_key: 'kWvGtYT/~q+35q;3dSI<^P!MH+bgrpBg'
|
secret_key: 'kWvGtYT/~q+35q;3dSI<^P!MH+bgrpBg'
|
||||||
bucket: "spark"
|
bucket: "spark"
|
||||||
|
|||||||
@ -84,6 +84,7 @@ message:
|
|||||||
|
|
||||||
object_storage:
|
object_storage:
|
||||||
endpoint: "rustfs:9000"
|
endpoint: "rustfs:9000"
|
||||||
|
public_endpoint: "https://sparkr.wyrs365.com"
|
||||||
access_key: "rustfs_spark"
|
access_key: "rustfs_spark"
|
||||||
secret_key: 'kWvGtYT/~q+35q;3dSI<^P!MH+bgrpBg'
|
secret_key: 'kWvGtYT/~q+35q;3dSI<^P!MH+bgrpBg'
|
||||||
bucket: "spark"
|
bucket: "spark"
|
||||||
|
|||||||
@ -33,6 +33,6 @@ rsync -az --delete \
|
|||||||
"${SSH[@]}" "docker save spark-server:latest | gzip -c > $DATA/images/spark-server.tar.gz"
|
"${SSH[@]}" "docker save spark-server:latest | gzip -c > $DATA/images/spark-server.tar.gz"
|
||||||
"${SSH[@]}" "cd $REMOTE/deploy/docker && docker compose -f docker-compose.yaml up -d spark-server"
|
"${SSH[@]}" "cd $REMOTE/deploy/docker && docker compose -f docker-compose.yaml up -d spark-server"
|
||||||
"${SSH[@]}" "docker exec spark-server /usr/local/bin/spark-migrate -config /app/configs/config.yaml -dir /app/migrations"
|
"${SSH[@]}" "docker exec spark-server /usr/local/bin/spark-migrate -config /app/configs/config.yaml -dir /app/migrations"
|
||||||
"${SSH[@]}" "cp $REMOTE/deploy/nginx/host/spark-sni.conf /etc/nginx/stream.d/spark-sni.conf && nginx -t && systemctl reload nginx"
|
"${SSH[@]}" "cp $REMOTE/deploy/nginx/host/spark-sni.conf /etc/nginx/stream.d/spark-sni.conf && if [ -f /etc/letsencrypt/live/sparkr.wyrs365.com/fullchain.pem ]; then cp $REMOTE/deploy/nginx/host/sparkr.conf /etc/nginx/conf.d/sparkr.conf; fi && nginx -t && systemctl reload nginx"
|
||||||
"${SSH[@]}" "docker ps --filter name=spark-server --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
|
"${SSH[@]}" "docker ps --filter name=spark-server --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'"
|
||||||
"${SSH[@]}" "curl -fsS http://127.0.0.1:18080/healthz && echo && curl -fsS http://127.0.0.1:18080/readyz && echo"
|
"${SSH[@]}" "curl -fsS http://127.0.0.1:18080/healthz && echo && curl -fsS http://127.0.0.1:18080/readyz && echo"
|
||||||
|
|||||||
@ -6,6 +6,8 @@ gnet v2 has no native TLS. IM certificates live here, not in spark-server `tcp.t
|
|||||||
Client --TLS--> :443 ssl_preread
|
Client --TLS--> :443 ssl_preread
|
||||||
├─ SNI sparkapi.wyrs365.com → 127.0.0.1:8443 (HTTP TLS)
|
├─ SNI sparkapi.wyrs365.com → 127.0.0.1:8443 (HTTP TLS)
|
||||||
│ → proxy_pass http://127.0.0.1:18080 + X-Real-IP
|
│ → proxy_pass http://127.0.0.1:18080 + X-Real-IP
|
||||||
|
├─ SNI sparkr.wyrs365.com → 127.0.0.1:8443 (HTTP TLS)
|
||||||
|
│ → proxy_pass http://127.0.0.1:9000 RustFS S3
|
||||||
└─ SNI sparkim.wyrs365.com → 127.0.0.1:19001 (IM TLS)
|
└─ SNI sparkim.wyrs365.com → 127.0.0.1:19001 (IM TLS)
|
||||||
→ PROXY protocol → 127.0.0.1:9090 gnet
|
→ PROXY protocol → 127.0.0.1:9090 gnet
|
||||||
```
|
```
|
||||||
@ -13,6 +15,7 @@ Client --TLS--> :443 ssl_preread
|
|||||||
On this ECS, `:443` is shared with other wyrs HTTPS sites. Do **not** replace `/etc/nginx/nginx.conf`. Drop-ins:
|
On this ECS, `:443` is shared with other wyrs HTTPS sites. Do **not** replace `/etc/nginx/nginx.conf`. Drop-ins:
|
||||||
|
|
||||||
- `host/sparkapi.conf` → `/etc/nginx/conf.d/`
|
- `host/sparkapi.conf` → `/etc/nginx/conf.d/`
|
||||||
|
- `host/sparkr.conf` → `/etc/nginx/conf.d/` (RustFS S3; DNS A record `sparkr.wyrs365.com` → this host)
|
||||||
- `host/spark-sni.conf` → `/etc/nginx/stream.d/`
|
- `host/spark-sni.conf` → `/etc/nginx/stream.d/`
|
||||||
|
|
||||||
Existing `listen 443 ssl` vhosts move to `127.0.0.1:8443 ssl proxy_protocol` so SNI can take public 443. `00-spark-http.conf` sets `real_ip_header proxy_protocol` and `port_in_redirect off` so `$remote_addr` is the client and redirects stay on 443.
|
Existing `listen 443 ssl` vhosts move to `127.0.0.1:8443 ssl proxy_protocol` so SNI can take public 443. `00-spark-http.conf` sets `real_ip_header proxy_protocol` and `port_in_redirect off` so `$remote_addr` is the client and redirects stay on 443.
|
||||||
@ -20,9 +23,10 @@ Existing `listen 443 ssl` vhosts move to `127.0.0.1:8443 ssl proxy_protocol` so
|
|||||||
spark-server (Docker):
|
spark-server (Docker):
|
||||||
|
|
||||||
- HTTP `127.0.0.1:18080` → container `:8080`
|
- HTTP `127.0.0.1:18080` → container `:8080`
|
||||||
- TCP `127.0.0.1:9090` with `proxy_protocol: true` → container `:9090` (host `:9000` is rustfs S3)
|
- TCP `127.0.0.1:9090` with `proxy_protocol: true` → container `:9090` (host `:9000` is rustfs S3; bind `127.0.0.1:9000`, not public)
|
||||||
- Image / config / logs: `/data/spark-server/{images,configs,logs}`
|
- Image / config / logs: `/data/spark-server/{images,configs,logs}`
|
||||||
- Public security group: 443 only
|
- Public security group: 443 only
|
||||||
|
- Clients upload/download at `https://sparkr.wyrs365.com` (path-style). spark-server talks to `rustfs:9000` on the Docker network and signs URLs with `object_storage.public_endpoint`.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
bash deploy/nginx/install-host.sh
|
bash deploy/nginx/install-host.sh
|
||||||
|
|||||||
@ -3,7 +3,7 @@
|
|||||||
server {
|
server {
|
||||||
listen 80;
|
listen 80;
|
||||||
listen [::]:80;
|
listen [::]:80;
|
||||||
server_name sparkapi.wyrs365.com sparkim.wyrs365.com;
|
server_name sparkapi.wyrs365.com sparkim.wyrs365.com sparkr.wyrs365.com;
|
||||||
|
|
||||||
location /.well-known/acme-challenge/ {
|
location /.well-known/acme-challenge/ {
|
||||||
root /var/www/certbot;
|
root /var/www/certbot;
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
# Spark IM + shared HTTPS on :443 (ssl_preread).
|
# Spark IM + shared HTTPS on :443 (ssl_preread).
|
||||||
# sparkim.wyrs365.com → IM TLS terminator → PROXY plaintext to gnet :9090
|
# sparkim.wyrs365.com → IM TLS terminator → PROXY plaintext to gnet :9090
|
||||||
# default SNI → existing HTTP sites on 127.0.0.1:8443 (proxy_protocol restores RealIP)
|
# default SNI → 127.0.0.1:8443 HTTP TLS (sparkapi / sparkr / other wyrs sites; PROXY restores RealIP)
|
||||||
|
|
||||||
map $ssl_preread_server_name $spark_sni_backend {
|
map $ssl_preread_server_name $spark_sni_backend {
|
||||||
sparkim.wyrs365.com 127.0.0.1:19001;
|
sparkim.wyrs365.com 127.0.0.1:19001;
|
||||||
|
|||||||
58
deploy/nginx/host/sparkr.conf
Normal file
58
deploy/nginx/host/sparkr.conf
Normal file
@ -0,0 +1,58 @@
|
|||||||
|
# Spark RustFS S3 — sparkr.wyrs365.com
|
||||||
|
# Public :443 is stream ssl_preread (see stream.d/spark-sni.conf).
|
||||||
|
# TLS terminates on loopback :8443; proxy HTTP to Docker rustfs on :9000.
|
||||||
|
# Do not rewrite the URI: SigV4 signs host + path.
|
||||||
|
|
||||||
|
upstream spark_rustfs {
|
||||||
|
server 127.0.0.1:9000;
|
||||||
|
keepalive 32;
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
listen [::]:80;
|
||||||
|
server_name sparkr.wyrs365.com;
|
||||||
|
|
||||||
|
location /.well-known/acme-challenge/ {
|
||||||
|
root /var/www/certbot;
|
||||||
|
}
|
||||||
|
|
||||||
|
location / {
|
||||||
|
return 301 https://$host$request_uri;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
server {
|
||||||
|
listen 127.0.0.1:8443 ssl proxy_protocol;
|
||||||
|
listen [::1]:8443 ssl proxy_protocol;
|
||||||
|
http2 on;
|
||||||
|
server_name sparkr.wyrs365.com;
|
||||||
|
|
||||||
|
ssl_certificate /etc/letsencrypt/live/sparkr.wyrs365.com/fullchain.pem;
|
||||||
|
ssl_certificate_key /etc/letsencrypt/live/sparkr.wyrs365.com/privkey.pem;
|
||||||
|
ssl_session_timeout 1d;
|
||||||
|
ssl_session_cache shared:SparkSSL:10m;
|
||||||
|
|
||||||
|
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
||||||
|
add_header X-Content-Type-Options nosniff always;
|
||||||
|
|
||||||
|
ignore_invalid_headers off;
|
||||||
|
client_max_body_size 100m;
|
||||||
|
proxy_buffering off;
|
||||||
|
proxy_request_buffering off;
|
||||||
|
proxy_connect_timeout 300s;
|
||||||
|
proxy_send_timeout 300s;
|
||||||
|
proxy_read_timeout 300s;
|
||||||
|
proxy_http_version 1.1;
|
||||||
|
chunked_transfer_encoding off;
|
||||||
|
|
||||||
|
location / {
|
||||||
|
proxy_pass http://spark_rustfs;
|
||||||
|
proxy_set_header Host $http_host;
|
||||||
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
proxy_set_header X-Forwarded-Proto https;
|
||||||
|
proxy_set_header Connection "";
|
||||||
|
proxy_cache_convert_head off;
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -5,33 +5,61 @@ set -euo pipefail
|
|||||||
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
ROOT="$(cd "$(dirname "$0")" && pwd)"
|
||||||
HOST_DIR="$ROOT/host"
|
HOST_DIR="$ROOT/host"
|
||||||
BACKUP="/root/nginx-backup-$(date +%Y%m%d%H%M%S)"
|
BACKUP="/root/nginx-backup-$(date +%Y%m%d%H%M%S)"
|
||||||
|
DOMAINS=(sparkapi.wyrs365.com sparkim.wyrs365.com sparkr.wyrs365.com)
|
||||||
|
|
||||||
need_certs() {
|
cert_ok() {
|
||||||
[[ -f /etc/letsencrypt/live/sparkapi.wyrs365.com/fullchain.pem ]] \
|
[[ -f "/etc/letsencrypt/live/$1/fullchain.pem" ]]
|
||||||
&& [[ -f /etc/letsencrypt/live/sparkim.wyrs365.com/fullchain.pem ]]
|
|
||||||
}
|
}
|
||||||
|
|
||||||
install_acme() {
|
all_certs_ok() {
|
||||||
|
local d
|
||||||
|
for d in "${DOMAINS[@]}"; do
|
||||||
|
cert_ok "$d" || return 1
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
install_acme_domain() {
|
||||||
|
local d=$1
|
||||||
mkdir -p /var/www/certbot
|
mkdir -p /var/www/certbot
|
||||||
cp "$HOST_DIR/spark-acme.conf" /etc/nginx/conf.d/spark-acme.conf
|
cat > "/etc/nginx/conf.d/spark-acme-${d}.conf" <<EOF
|
||||||
|
server {
|
||||||
|
listen 80;
|
||||||
|
listen [::]:80;
|
||||||
|
server_name ${d};
|
||||||
|
|
||||||
|
location /.well-known/acme-challenge/ {
|
||||||
|
root /var/www/certbot;
|
||||||
|
}
|
||||||
|
|
||||||
|
location / {
|
||||||
|
return 404;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
EOF
|
||||||
nginx -t
|
nginx -t
|
||||||
systemctl reload nginx
|
systemctl reload nginx
|
||||||
}
|
}
|
||||||
|
|
||||||
issue_certs() {
|
issue_missing_certs() {
|
||||||
certbot certonly --webroot -w /var/www/certbot \
|
local d
|
||||||
--non-interactive --agree-tos --keep-until-expiring \
|
for d in "${DOMAINS[@]}"; do
|
||||||
-d sparkapi.wyrs365.com
|
if cert_ok "$d"; then
|
||||||
certbot certonly --webroot -w /var/www/certbot \
|
continue
|
||||||
--non-interactive --agree-tos --keep-until-expiring \
|
fi
|
||||||
-d sparkim.wyrs365.com
|
install_acme_domain "$d"
|
||||||
|
certbot certonly --webroot -w /var/www/certbot \
|
||||||
|
--non-interactive --agree-tos --keep-until-expiring \
|
||||||
|
-d "$d"
|
||||||
|
rm -f "/etc/nginx/conf.d/spark-acme-${d}.conf"
|
||||||
|
done
|
||||||
}
|
}
|
||||||
|
|
||||||
rewrite_http_443() {
|
rewrite_http_443() {
|
||||||
local f
|
local f
|
||||||
for f in /etc/nginx/conf.d/*.conf; do
|
for f in /etc/nginx/conf.d/*.conf; do
|
||||||
case "$(basename "$f")" in
|
case "$(basename "$f")" in
|
||||||
sparkapi.conf|spark-acme.conf) continue ;;
|
sparkapi.conf|sparkr.conf|spark-acme.conf|00-spark-http.conf) continue ;;
|
||||||
|
spark-acme-*) continue ;;
|
||||||
esac
|
esac
|
||||||
if grep -qE 'listen[[:space:]]+127\.0\.0\.1:8443' "$f"; then
|
if grep -qE 'listen[[:space:]]+127\.0\.0\.1:8443' "$f"; then
|
||||||
continue
|
continue
|
||||||
@ -48,8 +76,10 @@ rewrite_http_443() {
|
|||||||
install_sni() {
|
install_sni() {
|
||||||
rewrite_http_443
|
rewrite_http_443
|
||||||
rm -f /etc/nginx/conf.d/spark-acme.conf
|
rm -f /etc/nginx/conf.d/spark-acme.conf
|
||||||
|
rm -f /etc/nginx/conf.d/spark-acme-*.conf
|
||||||
cp "$HOST_DIR/00-spark-http.conf" /etc/nginx/conf.d/00-spark-http.conf
|
cp "$HOST_DIR/00-spark-http.conf" /etc/nginx/conf.d/00-spark-http.conf
|
||||||
cp "$HOST_DIR/sparkapi.conf" /etc/nginx/conf.d/sparkapi.conf
|
cp "$HOST_DIR/sparkapi.conf" /etc/nginx/conf.d/sparkapi.conf
|
||||||
|
cp "$HOST_DIR/sparkr.conf" /etc/nginx/conf.d/sparkr.conf
|
||||||
cp "$HOST_DIR/spark-sni.conf" /etc/nginx/stream.d/spark-sni.conf
|
cp "$HOST_DIR/spark-sni.conf" /etc/nginx/stream.d/spark-sni.conf
|
||||||
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
|
mkdir -p /etc/letsencrypt/renewal-hooks/deploy
|
||||||
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK'
|
cat > /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh <<'HOOK'
|
||||||
@ -69,12 +99,11 @@ mkdir -p "$BACKUP"
|
|||||||
cp -a /etc/nginx "$BACKUP/nginx"
|
cp -a /etc/nginx "$BACKUP/nginx"
|
||||||
echo "backup $BACKUP"
|
echo "backup $BACKUP"
|
||||||
|
|
||||||
if ! need_certs; then
|
if ! all_certs_ok; then
|
||||||
install_acme
|
issue_missing_certs
|
||||||
issue_certs
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if ! need_certs; then
|
if ! all_certs_ok; then
|
||||||
echo "certificates missing" >&2
|
echo "certificates missing" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|||||||
@ -138,6 +138,7 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge
|
|||||||
authLimit := ratelimit.New(cfg.RateLimit.AuthPerSecond)
|
authLimit := ratelimit.New(cfg.RateLimit.AuthPerSecond)
|
||||||
objects, err := objectstore.Open(&objectstore.Options{
|
objects, err := objectstore.Open(&objectstore.Options{
|
||||||
Endpoint: cfg.ObjectStorage.Endpoint,
|
Endpoint: cfg.ObjectStorage.Endpoint,
|
||||||
|
PublicEndpoint: cfg.ObjectStorage.PublicEndpoint,
|
||||||
AccessKey: cfg.ObjectStorage.AccessKey,
|
AccessKey: cfg.ObjectStorage.AccessKey,
|
||||||
SecretKey: cfg.ObjectStorage.SecretKey,
|
SecretKey: cfg.ObjectStorage.SecretKey,
|
||||||
Bucket: cfg.ObjectStorage.Bucket,
|
Bucket: cfg.ObjectStorage.Bucket,
|
||||||
@ -166,7 +167,7 @@ func finishApp(cfg *Config, log *zap.Logger, logClose func() error, idgen *id.Ge
|
|||||||
friends := wireFriends(idgen, db, users, deliverQueue, chat)
|
friends := wireFriends(idgen, db, users, deliverQueue, chat)
|
||||||
groups := wireGroups(idgen, db, users, friends, deliverQueue, objects, chat)
|
groups := wireGroups(idgen, db, users, friends, deliverQueue, objects, chat)
|
||||||
messages := wireMessages(idgen, friends, groups, deliverQueue, chat)
|
messages := wireMessages(idgen, friends, groups, deliverQueue, chat)
|
||||||
files := wireFiles(cfg, objects, idgen, groups)
|
files := wireFiles(cfg, objects, groups)
|
||||||
favorites := wireFavorites(idgen, db, objects, chat)
|
favorites := wireFavorites(idgen, db, objects, chat)
|
||||||
app.tcp = tcp.NewServer(log, pool, sessions, &tcp.Options{
|
app.tcp = tcp.NewServer(log, pool, sessions, &tcp.Options{
|
||||||
Addr: cfg.Server.TCP.Addr,
|
Addr: cfg.Server.TCP.Addr,
|
||||||
@ -276,11 +277,11 @@ func wireGroups(idgen *id.Generator, db *mysql.DB, users *userlogic.Service, fri
|
|||||||
return grouplogic.NewService(repo, users, friends, idgen, chat.msgs, q, objects)
|
return grouplogic.NewService(repo, users, friends, idgen, chat.msgs, q, objects)
|
||||||
}
|
}
|
||||||
|
|
||||||
func wireFiles(cfg *Config, store *objectstore.Store, idgen *id.Generator, groups *grouplogic.Service) *uploadlogic.Service {
|
func wireFiles(cfg *Config, store *objectstore.Store, groups *grouplogic.Service) *uploadlogic.Service {
|
||||||
if store == nil || !store.Ready() || idgen == nil {
|
if store == nil || !store.Ready() {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return uploadlogic.NewService(store, idgen, uploadlogic.Options{
|
return uploadlogic.NewService(store, uploadlogic.Options{
|
||||||
TTL: cfg.ObjectStorage.PresignTTL,
|
TTL: cfg.ObjectStorage.PresignTTL,
|
||||||
Limit: ratelimit.New(cfg.RateLimit.UploadPerSecond),
|
Limit: ratelimit.New(cfg.RateLimit.UploadPerSecond),
|
||||||
Groups: groups,
|
Groups: groups,
|
||||||
|
|||||||
@ -118,6 +118,7 @@ type RateLimitConfig struct {
|
|||||||
|
|
||||||
type ObjectStorageConfig struct {
|
type ObjectStorageConfig struct {
|
||||||
Endpoint string `yaml:"endpoint"`
|
Endpoint string `yaml:"endpoint"`
|
||||||
|
PublicEndpoint string `yaml:"public_endpoint"`
|
||||||
AccessKey string `yaml:"access_key"`
|
AccessKey string `yaml:"access_key"`
|
||||||
SecretKey string `yaml:"secret_key"`
|
SecretKey string `yaml:"secret_key"`
|
||||||
Bucket string `yaml:"bucket"`
|
Bucket string `yaml:"bucket"`
|
||||||
@ -229,6 +230,9 @@ func applyDefaults(cfg *Config) {
|
|||||||
if v := os.Getenv("SPARK_S3_ENDPOINT"); v != "" {
|
if v := os.Getenv("SPARK_S3_ENDPOINT"); v != "" {
|
||||||
cfg.ObjectStorage.Endpoint = v
|
cfg.ObjectStorage.Endpoint = v
|
||||||
}
|
}
|
||||||
|
if v := os.Getenv("SPARK_S3_PUBLIC_ENDPOINT"); v != "" {
|
||||||
|
cfg.ObjectStorage.PublicEndpoint = v
|
||||||
|
}
|
||||||
if v := os.Getenv("SPARK_S3_ACCESS_KEY"); v != "" {
|
if v := os.Getenv("SPARK_S3_ACCESS_KEY"); v != "" {
|
||||||
cfg.ObjectStorage.AccessKey = v
|
cfg.ObjectStorage.AccessKey = v
|
||||||
}
|
}
|
||||||
|
|||||||
@ -18,13 +18,24 @@ import (
|
|||||||
"spark-server/internal/pkg/errs"
|
"spark-server/internal/pkg/errs"
|
||||||
)
|
)
|
||||||
|
|
||||||
const maxObjectBytes = 4 << 20
|
const (
|
||||||
|
maxGetBytes = 32 << 20
|
||||||
|
maxPutBytes = 4 << 20
|
||||||
|
)
|
||||||
|
|
||||||
func (s *Store) PresignPut(ctx context.Context, key, contentType string, ttl time.Duration) (string, error) {
|
func (s *Store) PresignPut(ctx context.Context, key, contentType string, size int64, ttl time.Duration) (string, error) {
|
||||||
if !s.Ready() {
|
if !s.Ready() {
|
||||||
return "", errs.ErrAvailable
|
return "", errs.ErrAvailable
|
||||||
}
|
}
|
||||||
in := &s3.PutObjectInput{Bucket: aws.String(s.bucket), Key: aws.String(key)}
|
if size < 1 {
|
||||||
|
return "", errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
in := &s3.PutObjectInput{
|
||||||
|
Bucket: aws.String(s.bucket),
|
||||||
|
Key: aws.String(key),
|
||||||
|
ContentLength: aws.Int64(size),
|
||||||
|
IfNoneMatch: aws.String("*"),
|
||||||
|
}
|
||||||
if contentType != "" {
|
if contentType != "" {
|
||||||
in.ContentType = aws.String(contentType)
|
in.ContentType = aws.String(contentType)
|
||||||
}
|
}
|
||||||
@ -61,11 +72,11 @@ func (s *Store) Get(ctx context.Context, key string) ([]byte, error) {
|
|||||||
return nil, mapObjectErr(err)
|
return nil, mapObjectErr(err)
|
||||||
}
|
}
|
||||||
defer func() { _ = out.Body.Close() }()
|
defer func() { _ = out.Body.Close() }()
|
||||||
data, err := io.ReadAll(io.LimitReader(out.Body, maxObjectBytes+1))
|
data, err := io.ReadAll(io.LimitReader(out.Body, maxGetBytes+1))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, mapObjectErr(err)
|
return nil, mapObjectErr(err)
|
||||||
}
|
}
|
||||||
if len(data) > maxObjectBytes {
|
if len(data) > maxGetBytes {
|
||||||
return nil, errs.ErrInvalidArgument
|
return nil, errs.ErrInvalidArgument
|
||||||
}
|
}
|
||||||
return data, nil
|
return data, nil
|
||||||
@ -75,7 +86,7 @@ func (s *Store) Put(ctx context.Context, key string, body []byte, contentType st
|
|||||||
if !s.Ready() {
|
if !s.Ready() {
|
||||||
return errs.ErrAvailable
|
return errs.ErrAvailable
|
||||||
}
|
}
|
||||||
if len(body) == 0 || len(body) > maxObjectBytes {
|
if len(body) == 0 || len(body) > maxPutBytes {
|
||||||
return errs.ErrInvalidArgument
|
return errs.ErrInvalidArgument
|
||||||
}
|
}
|
||||||
in := &s3.PutObjectInput{
|
in := &s3.PutObjectInput{
|
||||||
|
|||||||
@ -29,6 +29,7 @@ type Store struct {
|
|||||||
// Options configure Open. Empty Endpoint skips the connection.
|
// Options configure Open. Empty Endpoint skips the connection.
|
||||||
type Options struct {
|
type Options struct {
|
||||||
Endpoint string
|
Endpoint string
|
||||||
|
PublicEndpoint string
|
||||||
AccessKey string
|
AccessKey string
|
||||||
SecretKey string
|
SecretKey string
|
||||||
Bucket string
|
Bucket string
|
||||||
@ -42,7 +43,7 @@ func Open(o *Options) (*Store, error) {
|
|||||||
if o == nil || strings.TrimSpace(o.Endpoint) == "" {
|
if o == nil || strings.TrimSpace(o.Endpoint) == "" {
|
||||||
return &Store{}, nil
|
return &Store{}, nil
|
||||||
}
|
}
|
||||||
host, secure, err := parseEndpoint(o.Endpoint, o.UseSSL)
|
client, presignBase, err := newClients(o)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@ -50,19 +51,14 @@ func Open(o *Options) (*Store, error) {
|
|||||||
if bucket == "" {
|
if bucket == "" {
|
||||||
bucket = "spark"
|
bucket = "spark"
|
||||||
}
|
}
|
||||||
|
s := &Store{client: client, presign: s3.NewPresignClient(presignBase), bucket: bucket, log: o.Log}
|
||||||
|
if s.log == nil {
|
||||||
|
s.log = zap.NewNop()
|
||||||
|
}
|
||||||
region := strings.TrimSpace(o.Region)
|
region := strings.TrimSpace(o.Region)
|
||||||
if region == "" {
|
if region == "" {
|
||||||
region = defaultRegion
|
region = defaultRegion
|
||||||
}
|
}
|
||||||
scheme := "http"
|
|
||||||
if secure {
|
|
||||||
scheme = "https"
|
|
||||||
}
|
|
||||||
client := newS3Client(region, scheme+"://"+host, o.AccessKey, o.SecretKey)
|
|
||||||
s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: bucket, log: o.Log}
|
|
||||||
if s.log == nil {
|
|
||||||
s.log = zap.NewNop()
|
|
||||||
}
|
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), defaultTimeout)
|
ctx, cancel := context.WithTimeout(context.Background(), defaultTimeout)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
if err := s.ensureBucket(ctx, region, o.ChatExpireDays); err != nil {
|
if err := s.ensureBucket(ctx, region, o.ChatExpireDays); err != nil {
|
||||||
@ -83,6 +79,26 @@ func (s *Store) Ping(ctx context.Context) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func newClients(o *Options) (*s3.Client, *s3.Client, error) {
|
||||||
|
region := strings.TrimSpace(o.Region)
|
||||||
|
if region == "" {
|
||||||
|
region = defaultRegion
|
||||||
|
}
|
||||||
|
host, secure, err := parseEndpoint(o.Endpoint, o.UseSSL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
client := newS3Client(region, endpointURL(host, secure), o.AccessKey, o.SecretKey)
|
||||||
|
if pub := strings.TrimSpace(o.PublicEndpoint); pub != "" {
|
||||||
|
phost, psecure, err := parseEndpoint(pub, true)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, err
|
||||||
|
}
|
||||||
|
return client, newS3Client(region, endpointURL(phost, psecure), o.AccessKey, o.SecretKey), nil
|
||||||
|
}
|
||||||
|
return client, client, nil
|
||||||
|
}
|
||||||
|
|
||||||
func newS3Client(region, baseURL, access, secret string) *s3.Client {
|
func newS3Client(region, baseURL, access, secret string) *s3.Client {
|
||||||
return s3.New(s3.Options{
|
return s3.New(s3.Options{
|
||||||
Region: region,
|
Region: region,
|
||||||
@ -94,8 +110,15 @@ func newS3Client(region, baseURL, access, secret string) *s3.Client {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func endpointURL(host string, secure bool) string {
|
||||||
|
if secure {
|
||||||
|
return "https://" + host
|
||||||
|
}
|
||||||
|
return "http://" + host
|
||||||
|
}
|
||||||
|
|
||||||
func parseEndpoint(raw string, useSSL bool) (host string, secure bool, err error) {
|
func parseEndpoint(raw string, useSSL bool) (host string, secure bool, err error) {
|
||||||
raw = strings.TrimSpace(raw)
|
raw = strings.TrimSpace(strings.TrimRight(raw, "/"))
|
||||||
switch {
|
switch {
|
||||||
case raw == "":
|
case raw == "":
|
||||||
return "", false, fmt.Errorf("objectstore: empty endpoint")
|
return "", false, fmt.Errorf("objectstore: empty endpoint")
|
||||||
|
|||||||
@ -40,13 +40,17 @@ func TestOpenEmpty(t *testing.T) {
|
|||||||
func TestPresignLocal(t *testing.T) {
|
func TestPresignLocal(t *testing.T) {
|
||||||
client := newS3Client(defaultRegion, "http://127.0.0.1:9000", "ak", "sk")
|
client := newS3Client(defaultRegion, "http://127.0.0.1:9000", "ak", "sk")
|
||||||
s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: "spark"}
|
s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: "spark"}
|
||||||
u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", time.Hour)
|
u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 1024, time.Hour)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if !strings.Contains(u, "127.0.0.1:9000/spark/chat/a.jpg") {
|
if !strings.Contains(u, "127.0.0.1:9000/spark/chat/a.jpg") {
|
||||||
t.Fatalf("%s", u)
|
t.Fatalf("%s", u)
|
||||||
}
|
}
|
||||||
|
signed := strings.ToLower(u)
|
||||||
|
if !strings.Contains(signed, "content-length") || !strings.Contains(signed, "if-none-match") {
|
||||||
|
t.Fatalf("unsigned size/overwrite headers: %s", u)
|
||||||
|
}
|
||||||
g, err := s.PresignGet(t.Context(), "perm/avatar/1.jpg", time.Minute)
|
g, err := s.PresignGet(t.Context(), "perm/avatar/1.jpg", time.Minute)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@ -56,6 +60,37 @@ func TestPresignLocal(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPresignPutRejectsZeroSize(t *testing.T) {
|
||||||
|
client := newS3Client(defaultRegion, "http://127.0.0.1:9000", "ak", "sk")
|
||||||
|
s := &Store{client: client, presign: s3.NewPresignClient(client), bucket: "spark"}
|
||||||
|
if _, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 0, time.Hour); !errors.Is(err, errs.ErrInvalidArgument) {
|
||||||
|
t.Fatalf("%v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPresignPublicEndpoint(t *testing.T) {
|
||||||
|
internal, public, err := newClients(&Options{
|
||||||
|
Endpoint: "http://rustfs:9000",
|
||||||
|
PublicEndpoint: "https://sparkr.wyrs365.com/",
|
||||||
|
AccessKey: "ak",
|
||||||
|
SecretKey: "sk",
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
s := &Store{client: internal, presign: s3.NewPresignClient(public), bucket: "spark"}
|
||||||
|
u, err := s.PresignPut(t.Context(), "chat/a.jpg", "image/jpeg", 1024, time.Hour)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(u, "https://sparkr.wyrs365.com/spark/chat/a.jpg") {
|
||||||
|
t.Fatalf("%s", u)
|
||||||
|
}
|
||||||
|
if strings.Contains(u, "rustfs:9000") {
|
||||||
|
t.Fatalf("internal host leaked: %s", u)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestGetPutNotReady(t *testing.T) {
|
func TestGetPutNotReady(t *testing.T) {
|
||||||
s, err := Open(&Options{})
|
s, err := Open(&Options{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@ -7,9 +7,10 @@ import (
|
|||||||
msgentity "spark-server/internal/module/message/entity"
|
msgentity "spark-server/internal/module/message/entity"
|
||||||
uploadentity "spark-server/internal/module/upload/entity"
|
uploadentity "spark-server/internal/module/upload/entity"
|
||||||
"spark-server/internal/pkg/errs"
|
"spark-server/internal/pkg/errs"
|
||||||
|
"spark-server/internal/pkg/reqctx"
|
||||||
)
|
)
|
||||||
|
|
||||||
func (s *Service) copyMedia(ctx context.Context, userID int64, payload *msgentity.Content) ([]string, error) {
|
func (s *Service) copyMedia(ctx context.Context, payload *msgentity.Content) ([]string, error) {
|
||||||
keys := payload.ObjectKeys()
|
keys := payload.ObjectKeys()
|
||||||
if len(keys) == 0 {
|
if len(keys) == 0 {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
@ -24,12 +25,7 @@ func (s *Service) copyMedia(ctx context.Context, userID int64, payload *msgentit
|
|||||||
if _, ok := mapping[src]; ok {
|
if _, ok := mapping[src]; ok {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
objectID, err := s.ids.NextID()
|
dst := uploadentity.BuildFavKey(reqctx.NewObjectName(), day, uploadentity.ExtOfKey(src))
|
||||||
if err != nil {
|
|
||||||
s.deleteKeys(ctx, copied)
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
dst := uploadentity.BuildFavKey(userID, objectID, day, uploadentity.ExtOfKey(src))
|
|
||||||
if err := s.objects.Copy(ctx, src, dst); err != nil {
|
if err := s.objects.Copy(ctx, src, dst); err != nil {
|
||||||
s.deleteKeys(ctx, copied)
|
s.deleteKeys(ctx, copied)
|
||||||
return nil, err
|
return nil, err
|
||||||
|
|||||||
@ -47,7 +47,7 @@ func (s *Service) Create(ctx context.Context, userID int64, in *CreateInput) (*I
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
copied, err := s.copyMedia(ctx, userID, payload)
|
copied, err := s.copyMedia(ctx, payload)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@ -56,8 +56,12 @@ func TestFavoriteCopiesMediaAndExpires(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
src := "chat/20260917/1/99.jpg"
|
src := "chat/20260917/1/99.jpg"
|
||||||
|
thumb := "chat/20260917/1/t.jpg"
|
||||||
|
orig := "chat/20260917/1/o.jpg"
|
||||||
objs.keys[src] = true
|
objs.keys[src] = true
|
||||||
raw, err := msgentity.NewImage(&msgentity.Image{Key: src, Mime: "image/jpeg"}).Marshal()
|
objs.keys[thumb] = true
|
||||||
|
objs.keys[orig] = true
|
||||||
|
raw, err := msgentity.NewImage(&msgentity.Image{Key: src, ThumbKey: thumb, OrigKey: orig, Mime: "image/jpeg"}).Marshal()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@ -76,10 +80,10 @@ func TestFavoriteCopiesMediaAndExpires(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if item.MsgID != msgID || !strings.Contains(string(item.Content), "perm/fav/1/") {
|
if item.MsgID != msgID || !strings.Contains(string(item.Content), "perm/fav/") {
|
||||||
t.Fatalf("%+v", item)
|
t.Fatalf("%+v", item)
|
||||||
}
|
}
|
||||||
if strings.Contains(string(item.Content), src) {
|
if strings.Contains(string(item.Content), src) || strings.Contains(string(item.Content), orig) {
|
||||||
t.Fatal("still points at chat key")
|
t.Fatal("still points at chat key")
|
||||||
}
|
}
|
||||||
again, err := svc.Create(ctx, 1, &CreateInput{MsgID: msgID})
|
again, err := svc.Create(ctx, 1, &CreateInput{MsgID: msgID})
|
||||||
|
|||||||
@ -13,6 +13,7 @@ import (
|
|||||||
"spark-server/internal/module/group/entity"
|
"spark-server/internal/module/group/entity"
|
||||||
uploadentity "spark-server/internal/module/upload/entity"
|
uploadentity "spark-server/internal/module/upload/entity"
|
||||||
userentity "spark-server/internal/module/user/entity"
|
userentity "spark-server/internal/module/user/entity"
|
||||||
|
"spark-server/internal/pkg/reqctx"
|
||||||
)
|
)
|
||||||
|
|
||||||
const collageJPEG = "image/jpeg"
|
const collageJPEG = "image/jpeg"
|
||||||
@ -63,10 +64,7 @@ func (s *Service) putAvatar(ctx context.Context, g *entity.Group, raw []byte, mi
|
|||||||
if g == nil || len(raw) == 0 {
|
if g == nil || len(raw) == 0 {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
key := uploadentity.PrefixGroup + strconv.FormatInt(g.ID, 10) + "/default." + ext
|
key := uploadentity.BuildObjectKey(uploadentity.PurposeGroupAvatar, g.ID, reqctx.NewObjectName(), "", ext)
|
||||||
if objectID, err := s.ids.NextID(); err == nil {
|
|
||||||
key = uploadentity.BuildObjectKey(uploadentity.PurposeGroupAvatar, g.ID, objectID, "", ext)
|
|
||||||
}
|
|
||||||
if err := s.objects.Put(ctx, key, raw, mime); err != nil {
|
if err := s.objects.Put(ctx, key, raw, mime); err != nil {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|||||||
@ -121,7 +121,7 @@ func (c *Content) Validate() error {
|
|||||||
case TypeText:
|
case TypeText:
|
||||||
return validateText(c.Text)
|
return validateText(c.Text)
|
||||||
case TypeImage:
|
case TypeImage:
|
||||||
return validateMediaKey("image", c.Image.Key)
|
return validateImage(c.Image)
|
||||||
case TypeVoice:
|
case TypeVoice:
|
||||||
return validateVoice(c.Voice)
|
return validateVoice(c.Voice)
|
||||||
case TypeVideo:
|
case TypeVideo:
|
||||||
@ -220,6 +220,16 @@ func validateMentions(c *Content) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateImage(img *Image) error {
|
||||||
|
if err := validateMediaKey("image", img.Key); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if img.Size < 0 || img.OrigSize < 0 || img.Width < 0 || img.Height < 0 || img.OrigW < 0 || img.OrigH < 0 {
|
||||||
|
return fmt.Errorf("content: image size")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func validateMediaKey(kind, key string) error {
|
func validateMediaKey(kind, key string) error {
|
||||||
if strings.TrimSpace(key) == "" {
|
if strings.TrimSpace(key) == "" {
|
||||||
return fmt.Errorf("content: %s key empty", kind)
|
return fmt.Errorf("content: %s key empty", kind)
|
||||||
@ -279,14 +289,21 @@ func (c *Content) ObjectKeys() []string {
|
|||||||
var keys []string
|
var keys []string
|
||||||
add := func(k string) {
|
add := func(k string) {
|
||||||
k = strings.TrimSpace(k)
|
k = strings.TrimSpace(k)
|
||||||
if k != "" {
|
if k == "" {
|
||||||
keys = append(keys, k)
|
return
|
||||||
}
|
}
|
||||||
|
for _, existing := range keys {
|
||||||
|
if existing == k {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
keys = append(keys, k)
|
||||||
}
|
}
|
||||||
switch {
|
switch {
|
||||||
case c.Image != nil:
|
case c.Image != nil:
|
||||||
add(c.Image.Key)
|
add(c.Image.Key)
|
||||||
add(c.Image.ThumbKey)
|
add(c.Image.ThumbKey)
|
||||||
|
add(c.Image.OrigKey)
|
||||||
case c.Voice != nil:
|
case c.Voice != nil:
|
||||||
add(c.Voice.Key)
|
add(c.Voice.Key)
|
||||||
case c.Video != nil:
|
case c.Video != nil:
|
||||||
@ -316,6 +333,7 @@ func (c *Content) ReplaceObjectKeys(mapping map[string]string) {
|
|||||||
if c.Image != nil {
|
if c.Image != nil {
|
||||||
repl(&c.Image.Key)
|
repl(&c.Image.Key)
|
||||||
repl(&c.Image.ThumbKey)
|
repl(&c.Image.ThumbKey)
|
||||||
|
repl(&c.Image.OrigKey)
|
||||||
}
|
}
|
||||||
if c.Voice != nil {
|
if c.Voice != nil {
|
||||||
repl(&c.Voice.Key)
|
repl(&c.Voice.Key)
|
||||||
|
|||||||
@ -80,6 +80,9 @@ func TestRejectsRawStringAndEmptyKey(t *testing.T) {
|
|||||||
if _, err := NewImage(&Image{}).Marshal(); err == nil || !strings.Contains(err.Error(), "key") {
|
if _, err := NewImage(&Image{}).Marshal(); err == nil || !strings.Contains(err.Error(), "key") {
|
||||||
t.Fatalf("image key: %v", err)
|
t.Fatalf("image key: %v", err)
|
||||||
}
|
}
|
||||||
|
if _, err := NewImage(&Image{Key: "k", OrigSize: -1}).Marshal(); err == nil {
|
||||||
|
t.Fatal("negative orig_size")
|
||||||
|
}
|
||||||
bad := &Content{Type: TypeText, Text: &Text{Body: "x"}, Image: &Image{Key: "k"}}
|
bad := &Content{Type: TypeText, Text: &Text{Body: "x"}, Image: &Image{Key: "k"}}
|
||||||
if _, err := bad.Marshal(); err == nil {
|
if _, err := bad.Marshal(); err == nil {
|
||||||
t.Fatal("two payloads")
|
t.Fatal("two payloads")
|
||||||
@ -122,13 +125,21 @@ func TestMentionsRoundTrip(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestObjectKeysAndReplace(t *testing.T) {
|
func TestObjectKeysAndReplace(t *testing.T) {
|
||||||
img := NewImage(&Image{Key: "chat/a.jpg", ThumbKey: "chat/t.jpg"})
|
img := NewImage(&Image{Key: "chat/a.jpg", ThumbKey: "chat/t.jpg", OrigKey: "chat/o.jpg", OrigSize: 4 << 20})
|
||||||
keys := img.ObjectKeys()
|
keys := img.ObjectKeys()
|
||||||
if len(keys) != 2 || keys[0] != "chat/a.jpg" {
|
if len(keys) != 3 || keys[0] != "chat/a.jpg" || keys[2] != "chat/o.jpg" {
|
||||||
t.Fatalf("%v", keys)
|
t.Fatalf("%v", keys)
|
||||||
}
|
}
|
||||||
img.ReplaceObjectKeys(map[string]string{"chat/a.jpg": "perm/fav/1/x.jpg", "chat/t.jpg": "perm/fav/1/y.jpg"})
|
img.ReplaceObjectKeys(map[string]string{
|
||||||
if img.Image.Key != "perm/fav/1/x.jpg" || img.Image.ThumbKey != "perm/fav/1/y.jpg" {
|
"chat/a.jpg": "perm/fav/1/x.jpg",
|
||||||
|
"chat/t.jpg": "perm/fav/1/y.jpg",
|
||||||
|
"chat/o.jpg": "perm/fav/1/z.jpg",
|
||||||
|
})
|
||||||
|
if img.Image.Key != "perm/fav/1/x.jpg" || img.Image.ThumbKey != "perm/fav/1/y.jpg" || img.Image.OrigKey != "perm/fav/1/z.jpg" {
|
||||||
t.Fatalf("%+v", img.Image)
|
t.Fatalf("%+v", img.Image)
|
||||||
}
|
}
|
||||||
|
same := NewImage(&Image{Key: "chat/a.jpg", OrigKey: "chat/a.jpg"})
|
||||||
|
if got := same.ObjectKeys(); len(got) != 1 {
|
||||||
|
t.Fatalf("dedupe %v", got)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@ -5,7 +5,10 @@ type Text struct {
|
|||||||
Body string `json:"body"`
|
Body string `json:"body"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Image references an uploaded object. Bytes themselves are not stored in messages.content.
|
// Image references uploaded objects. Bytes are not stored in messages.content.
|
||||||
|
// WeChat-style: bubble uses thumb_key, tap-to-view uses key (compressed),
|
||||||
|
// 「查看原图」uses orig_key when the sender opted in.
|
||||||
|
// Clients PUT JPEG/PNG only (convert HEIC/WebP locally); the server does not transcode.
|
||||||
type Image struct {
|
type Image struct {
|
||||||
Key string `json:"key"`
|
Key string `json:"key"`
|
||||||
Mime string `json:"mime,omitempty"`
|
Mime string `json:"mime,omitempty"`
|
||||||
@ -13,6 +16,10 @@ type Image struct {
|
|||||||
Height int `json:"h,omitempty"`
|
Height int `json:"h,omitempty"`
|
||||||
Size int64 `json:"size,omitempty"`
|
Size int64 `json:"size,omitempty"`
|
||||||
ThumbKey string `json:"thumb_key,omitempty"`
|
ThumbKey string `json:"thumb_key,omitempty"`
|
||||||
|
OrigKey string `json:"orig_key,omitempty"`
|
||||||
|
OrigSize int64 `json:"orig_size,omitempty"`
|
||||||
|
OrigW int `json:"orig_w,omitempty"`
|
||||||
|
OrigH int `json:"orig_h,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Voice is an audio clip in object storage.
|
// Voice is an audio clip in object storage.
|
||||||
|
|||||||
@ -9,6 +9,11 @@ type Credential struct {
|
|||||||
ExpiresIn int64 `json:"expires_in"`
|
ExpiresIn int64 `json:"expires_in"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// CredentialSet is one HTTP round-trip for up to three PUTs (thumb / preview / orig).
|
||||||
|
type CredentialSet struct {
|
||||||
|
Items []Credential `json:"items"`
|
||||||
|
}
|
||||||
|
|
||||||
type Download struct {
|
type Download struct {
|
||||||
ObjectKey string `json:"object_key"`
|
ObjectKey string `json:"object_key"`
|
||||||
DownloadURL string `json:"download_url"`
|
DownloadURL string `json:"download_url"`
|
||||||
|
|||||||
@ -16,11 +16,16 @@ const (
|
|||||||
PurposeChatGIF = "chat_gif"
|
PurposeChatGIF = "chat_gif"
|
||||||
PurposeChatFile = "chat_file"
|
PurposeChatFile = "chat_file"
|
||||||
|
|
||||||
PrefixPerm = "perm/"
|
PrefixPerm = "perm/"
|
||||||
PrefixChat = "chat/"
|
PrefixChat = "chat/"
|
||||||
PrefixAvatar = "perm/avatar/"
|
PrefixAvatar = "perm/avatar/"
|
||||||
PrefixGroup = "perm/group/"
|
PrefixGroup = "perm/group/"
|
||||||
PrefixFav = "perm/fav/"
|
PrefixFav = "perm/fav/"
|
||||||
|
PrefixChatImage = "chat/image/"
|
||||||
|
PrefixChatVoice = "chat/voice/"
|
||||||
|
PrefixChatVideo = "chat/video/"
|
||||||
|
PrefixChatGIF = "chat/gif/"
|
||||||
|
PrefixChatFile = "chat/file/"
|
||||||
|
|
||||||
MethodPUT = "PUT"
|
MethodPUT = "PUT"
|
||||||
MethodGET = "GET"
|
MethodGET = "GET"
|
||||||
@ -28,7 +33,8 @@ const (
|
|||||||
|
|
||||||
const (
|
const (
|
||||||
maxAvatar = 2 << 20
|
maxAvatar = 2 << 20
|
||||||
maxImage = 10 << 20
|
maxImage = 32 << 20
|
||||||
|
maxGIF = 10 << 20
|
||||||
maxVoice = 16 << 20
|
maxVoice = 16 << 20
|
||||||
maxVideo = 100 << 20
|
maxVideo = 100 << 20
|
||||||
maxFile = 50 << 20
|
maxFile = 50 << 20
|
||||||
@ -75,25 +81,46 @@ func ExtOfKey(key string) string {
|
|||||||
return ext
|
return ext
|
||||||
}
|
}
|
||||||
|
|
||||||
func BuildObjectKey(purpose string, ownerID, objectID int64, day, ext string) string {
|
func BuildObjectKey(purpose string, ownerID int64, name, day, ext string) string {
|
||||||
if ext == "" {
|
if ext == "" {
|
||||||
ext = extDefault
|
ext = extDefault
|
||||||
}
|
}
|
||||||
|
if name == "" {
|
||||||
|
name = "0"
|
||||||
|
}
|
||||||
switch purpose {
|
switch purpose {
|
||||||
case PurposeAvatar:
|
case PurposeAvatar:
|
||||||
return fmt.Sprintf("%s%d/%s/%d.%s", PrefixAvatar, ownerID, day, objectID, ext)
|
return fmt.Sprintf("%s%s/%s.%s", PrefixAvatar, day, name, ext)
|
||||||
case PurposeGroupAvatar:
|
case PurposeGroupAvatar:
|
||||||
return fmt.Sprintf("%s%d/%d.%s", PrefixGroup, ownerID, objectID, ext)
|
return fmt.Sprintf("%s%d/%s.%s", PrefixGroup, ownerID, name, ext)
|
||||||
default:
|
default:
|
||||||
return fmt.Sprintf("%s%s/%d/%d.%s", PrefixChat, day, ownerID, objectID, ext)
|
return fmt.Sprintf("%s%s/%s/%s.%s", PrefixChat, chatKind(purpose), day, name, ext)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func BuildFavKey(userID, objectID int64, day, ext string) string {
|
func BuildFavKey(name, day, ext string) string {
|
||||||
if ext == "" {
|
if ext == "" {
|
||||||
ext = extDefault
|
ext = extDefault
|
||||||
}
|
}
|
||||||
return fmt.Sprintf("%s%d/%s/%d.%s", PrefixFav, userID, day, objectID, ext)
|
if name == "" {
|
||||||
|
name = "0"
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%s%s/%s.%s", PrefixFav, day, name, ext)
|
||||||
|
}
|
||||||
|
|
||||||
|
func chatKind(purpose string) string {
|
||||||
|
switch purpose {
|
||||||
|
case PurposeChatImage:
|
||||||
|
return "image"
|
||||||
|
case PurposeChatVoice:
|
||||||
|
return "voice"
|
||||||
|
case PurposeChatVideo:
|
||||||
|
return "video"
|
||||||
|
case PurposeChatGIF:
|
||||||
|
return "gif"
|
||||||
|
default:
|
||||||
|
return "file"
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func ValidObjectKey(key string) bool {
|
func ValidObjectKey(key string) bool {
|
||||||
@ -110,13 +137,13 @@ var specs = map[string]Spec{
|
|||||||
PurposeAvatar: {Purpose: PurposeAvatar, MaxSize: maxAvatar, MIMEs: imageMIMEs},
|
PurposeAvatar: {Purpose: PurposeAvatar, MaxSize: maxAvatar, MIMEs: imageMIMEs},
|
||||||
PurposeGroupAvatar: {Purpose: PurposeGroupAvatar, MaxSize: maxAvatar, MIMEs: imageMIMEs},
|
PurposeGroupAvatar: {Purpose: PurposeGroupAvatar, MaxSize: maxAvatar, MIMEs: imageMIMEs},
|
||||||
PurposeChatImage: {Purpose: PurposeChatImage, MaxSize: maxImage, MIMEs: imageMIMEs},
|
PurposeChatImage: {Purpose: PurposeChatImage, MaxSize: maxImage, MIMEs: imageMIMEs},
|
||||||
PurposeChatGIF: {Purpose: PurposeChatGIF, MaxSize: maxImage, MIMEs: gifMIMEs},
|
PurposeChatGIF: {Purpose: PurposeChatGIF, MaxSize: maxGIF, MIMEs: gifMIMEs},
|
||||||
PurposeChatVoice: {Purpose: PurposeChatVoice, MaxSize: maxVoice, MIMEs: voiceMIMEs},
|
PurposeChatVoice: {Purpose: PurposeChatVoice, MaxSize: maxVoice, MIMEs: voiceMIMEs},
|
||||||
PurposeChatVideo: {Purpose: PurposeChatVideo, MaxSize: maxVideo, MIMEs: videoMIMEs},
|
PurposeChatVideo: {Purpose: PurposeChatVideo, MaxSize: maxVideo, MIMEs: videoMIMEs},
|
||||||
PurposeChatFile: {Purpose: PurposeChatFile, MaxSize: maxFile, MIMEs: fileMIMEs},
|
PurposeChatFile: {Purpose: PurposeChatFile, MaxSize: maxFile, MIMEs: fileMIMEs},
|
||||||
}
|
}
|
||||||
|
|
||||||
var imageMIMEs = set("image/jpeg", "image/jpg", "image/png", "image/webp", "image/heic", "image/heif")
|
var imageMIMEs = set("image/jpeg", "image/jpg", "image/png")
|
||||||
var gifMIMEs = set("image/gif", "image/webp")
|
var gifMIMEs = set("image/gif", "image/webp")
|
||||||
var voiceMIMEs = set("audio/mpeg", "audio/mp4", "audio/aac", "audio/amr", "audio/ogg", "audio/wav", "audio/webm", "audio/x-m4a")
|
var voiceMIMEs = set("audio/mpeg", "audio/mp4", "audio/aac", "audio/amr", "audio/ogg", "audio/wav", "audio/webm", "audio/x-m4a")
|
||||||
var videoMIMEs = set("video/mp4", "video/quicktime", "video/webm")
|
var videoMIMEs = set("video/mp4", "video/quicktime", "video/webm")
|
||||||
@ -129,7 +156,7 @@ var fileMIMEs = set(
|
|||||||
|
|
||||||
var mimeExt = map[string]string{
|
var mimeExt = map[string]string{
|
||||||
"image/jpeg": "jpg", "image/jpg": "jpg", "image/png": "png", "image/webp": "webp",
|
"image/jpeg": "jpg", "image/jpg": "jpg", "image/png": "png", "image/webp": "webp",
|
||||||
"image/heic": "heic", "image/heif": "heif", "image/gif": "gif",
|
"image/gif": "gif",
|
||||||
"audio/mpeg": "mp3", "audio/mp4": "m4a", "audio/aac": "aac", "audio/amr": "amr",
|
"audio/mpeg": "mp3", "audio/mp4": "m4a", "audio/aac": "aac", "audio/amr": "amr",
|
||||||
"audio/ogg": "ogg", "audio/wav": "wav", "audio/webm": "weba", "audio/x-m4a": "m4a",
|
"audio/ogg": "ogg", "audio/wav": "wav", "audio/webm": "weba", "audio/x-m4a": "m4a",
|
||||||
"video/mp4": "mp4", "video/quicktime": "mov", "video/webm": "webm",
|
"video/mp4": "mp4", "video/quicktime": "mov", "video/webm": "webm",
|
||||||
|
|||||||
@ -6,20 +6,24 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func TestBuildObjectKeyByPurpose(t *testing.T) {
|
func TestBuildObjectKeyByPurpose(t *testing.T) {
|
||||||
avatar := BuildObjectKey(PurposeAvatar, 11, 99, "20260917", "jpg")
|
avatar := BuildObjectKey(PurposeAvatar, 11, "99", "20260917", "jpg")
|
||||||
if avatar != "perm/avatar/11/20260917/99.jpg" {
|
if avatar != "perm/avatar/20260917/99.jpg" {
|
||||||
t.Fatalf("avatar %s", avatar)
|
t.Fatalf("avatar %s", avatar)
|
||||||
}
|
}
|
||||||
chat := BuildObjectKey(PurposeChatImage, 11, 99, "20260917", "jpg")
|
chat := BuildObjectKey(PurposeChatImage, 11, "99", "20260917", "jpg")
|
||||||
if chat != "chat/20260917/11/99.jpg" {
|
if chat != "chat/image/20260917/99.jpg" {
|
||||||
t.Fatalf("chat %s", chat)
|
t.Fatalf("chat %s", chat)
|
||||||
}
|
}
|
||||||
group := BuildObjectKey(PurposeGroupAvatar, 7, 99, "20260917", "png")
|
voice := BuildObjectKey(PurposeChatVoice, 11, "aa", "20260917", "m4a")
|
||||||
|
if voice != "chat/voice/20260917/aa.m4a" {
|
||||||
|
t.Fatalf("voice %s", voice)
|
||||||
|
}
|
||||||
|
group := BuildObjectKey(PurposeGroupAvatar, 7, "99", "20260917", "png")
|
||||||
if group != "perm/group/7/99.png" {
|
if group != "perm/group/7/99.png" {
|
||||||
t.Fatalf("group %s", group)
|
t.Fatalf("group %s", group)
|
||||||
}
|
}
|
||||||
fav := BuildFavKey(11, 7, "20260917", "m4a")
|
fav := BuildFavKey("99", "20260917", "m4a")
|
||||||
if fav != "perm/fav/11/20260917/7.m4a" {
|
if fav != "perm/fav/20260917/99.m4a" {
|
||||||
t.Fatalf("fav %s", fav)
|
t.Fatalf("fav %s", fav)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@ -49,7 +53,19 @@ func TestNormalizeMIMEAndExt(t *testing.T) {
|
|||||||
if _, ok := spec.MIMEs["audio/mpeg"]; !ok {
|
if _, ok := spec.MIMEs["audio/mpeg"]; !ok {
|
||||||
t.Fatal("voice mime")
|
t.Fatal("voice mime")
|
||||||
}
|
}
|
||||||
if strings.Contains(BuildObjectKey(PurposeAvatar, 1, 1, "d", "jpg"), PrefixChat) {
|
img, ok := LookupSpec(PurposeChatImage)
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("image spec")
|
||||||
|
}
|
||||||
|
if _, ok := img.MIMEs["image/jpeg"]; !ok {
|
||||||
|
t.Fatal("jpeg")
|
||||||
|
}
|
||||||
|
for _, mime := range []string{"image/heic", "image/heif", "image/webp"} {
|
||||||
|
if _, ok := img.MIMEs[mime]; ok {
|
||||||
|
t.Fatalf("interchange must not allow %s", mime)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.Contains(BuildObjectKey(PurposeAvatar, 1, "1", "d", "jpg"), PrefixChat) {
|
||||||
t.Fatal("avatar must be perm")
|
t.Fatal("avatar must be perm")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
49
internal/module/upload/logic/batch.go
Normal file
49
internal/module/upload/logic/batch.go
Normal file
@ -0,0 +1,49 @@
|
|||||||
|
package logic
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"spark-server/internal/module/upload/entity"
|
||||||
|
"spark-server/internal/pkg/errs"
|
||||||
|
"spark-server/internal/pkg/reqctx"
|
||||||
|
"spark-server/internal/pkg/validate"
|
||||||
|
)
|
||||||
|
|
||||||
|
const maxCredentialBatch = 3
|
||||||
|
|
||||||
|
// BatchCreateInput issues up to three Presigned PUTs in one call (thumb / preview / orig).
|
||||||
|
type BatchCreateInput struct {
|
||||||
|
Items []CreateInput `json:"items" validate:"required,min=1,max=3,dive" label:"field.upload_items" errmsg:"required:validate.upload.items_required;min:validate.upload.items_required;max:validate.upload.items_invalid"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) CreateCredentials(ctx context.Context, userID int64, in *BatchCreateInput) (*entity.CredentialSet, error) {
|
||||||
|
if !s.ready() {
|
||||||
|
return nil, errs.ErrAvailable
|
||||||
|
}
|
||||||
|
if userID <= 0 {
|
||||||
|
return nil, errs.ErrUnauthorized
|
||||||
|
}
|
||||||
|
if in == nil {
|
||||||
|
return nil, errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
if err := validate.StructLang(reqctx.Lang(ctx), in); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if n := len(in.Items); n < 1 || n > maxCredentialBatch {
|
||||||
|
return nil, errs.ErrInvalidArgument
|
||||||
|
}
|
||||||
|
for range in.Items {
|
||||||
|
if !s.allow(userID) {
|
||||||
|
return nil, errs.ErrRateLimited
|
||||||
|
}
|
||||||
|
}
|
||||||
|
items := make([]entity.Credential, 0, len(in.Items))
|
||||||
|
for i := range in.Items {
|
||||||
|
cred, err := s.issuePut(ctx, userID, &in.Items[i])
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
items = append(items, *cred)
|
||||||
|
}
|
||||||
|
return &entity.CredentialSet{Items: items}, nil
|
||||||
|
}
|
||||||
@ -2,6 +2,7 @@ package logic
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"strconv"
|
||||||
|
|
||||||
"spark-server/internal/module/upload/entity"
|
"spark-server/internal/module/upload/entity"
|
||||||
"spark-server/internal/pkg/errs"
|
"spark-server/internal/pkg/errs"
|
||||||
@ -34,14 +35,15 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create
|
|||||||
if !s.allow(userID) {
|
if !s.allow(userID) {
|
||||||
return nil, errs.ErrRateLimited
|
return nil, errs.ErrRateLimited
|
||||||
}
|
}
|
||||||
|
return s.issuePut(ctx, userID, in)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) issuePut(ctx context.Context, userID int64, in *CreateInput) (*entity.Credential, error) {
|
||||||
_, mime, err := matchSpec(ctx, in)
|
_, mime, err := matchSpec(ctx, in)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
objectID, err := s.ids.NextID()
|
name := reqctx.NewObjectName()
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
ownerID := userID
|
ownerID := userID
|
||||||
if in.Purpose == entity.PurposeGroupAvatar {
|
if in.Purpose == entity.PurposeGroupAvatar {
|
||||||
if err := s.readyGroupAvatar(ctx, userID, in.GroupID); err != nil {
|
if err := s.readyGroupAvatar(ctx, userID, in.GroupID); err != nil {
|
||||||
@ -49,8 +51,8 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create
|
|||||||
}
|
}
|
||||||
ownerID = in.GroupID
|
ownerID = in.GroupID
|
||||||
}
|
}
|
||||||
key := entity.BuildObjectKey(in.Purpose, ownerID, objectID, dayOf(s.now()), entity.ExtForMIME(mime))
|
key := entity.BuildObjectKey(in.Purpose, ownerID, name, dayOf(s.now()), entity.ExtForMIME(mime))
|
||||||
url, err := s.store.PresignPut(ctx, key, mime, s.ttl)
|
url, err := s.store.PresignPut(ctx, key, mime, in.Size, s.ttl)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@ -58,7 +60,11 @@ func (s *Service) CreateCredential(ctx context.Context, userID int64, in *Create
|
|||||||
ObjectKey: key,
|
ObjectKey: key,
|
||||||
UploadURL: url,
|
UploadURL: url,
|
||||||
Method: entity.MethodPUT,
|
Method: entity.MethodPUT,
|
||||||
Headers: map[string]string{"Content-Type": mime},
|
Headers: map[string]string{
|
||||||
|
"Content-Type": mime,
|
||||||
|
"Content-Length": strconv.FormatInt(in.Size, 10),
|
||||||
|
"If-None-Match": "*",
|
||||||
|
},
|
||||||
ExpiresIn: s.expiresIn(),
|
ExpiresIn: s.expiresIn(),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@ -7,7 +7,6 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"spark-server/internal/infrastructure/id"
|
|
||||||
"spark-server/internal/module/upload/entity"
|
"spark-server/internal/module/upload/entity"
|
||||||
"spark-server/internal/pkg/errs"
|
"spark-server/internal/pkg/errs"
|
||||||
"spark-server/internal/pkg/reqctx"
|
"spark-server/internal/pkg/reqctx"
|
||||||
@ -18,7 +17,7 @@ type fakeStore struct {
|
|||||||
putErr error
|
putErr error
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeStore) PresignPut(_ context.Context, key, _ string, _ time.Duration) (string, error) {
|
func (f *fakeStore) PresignPut(_ context.Context, key, _ string, _ int64, _ time.Duration) (string, error) {
|
||||||
if f.putErr != nil {
|
if f.putErr != nil {
|
||||||
return "", f.putErr
|
return "", f.putErr
|
||||||
}
|
}
|
||||||
@ -37,11 +36,7 @@ func (f *fakeStore) PresignGet(_ context.Context, key string, _ time.Duration) (
|
|||||||
|
|
||||||
func testUpload(t *testing.T) *Service {
|
func testUpload(t *testing.T) *Service {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
ids, err := id.New(1)
|
return NewService(&fakeStore{}, Options{
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
return NewService(&fakeStore{}, ids, Options{
|
|
||||||
TTL: 5 * time.Minute,
|
TTL: 5 * time.Minute,
|
||||||
Now: func() time.Time { return time.Date(2026, 9, 17, 8, 0, 0, 0, time.UTC) },
|
Now: func() time.Time { return time.Date(2026, 9, 17, 8, 0, 0, 0, time.UTC) },
|
||||||
})
|
})
|
||||||
@ -54,11 +49,13 @@ func TestCreateCredentialChatVsAvatar(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if !strings.HasPrefix(chat.ObjectKey, "chat/20260917/42/") || chat.Method != "PUT" || chat.Headers["Content-Type"] != "image/jpeg" {
|
if !strings.HasPrefix(chat.ObjectKey, "chat/image/20260917/") || chat.Method != "PUT" ||
|
||||||
|
chat.Headers["Content-Type"] != "image/jpeg" || chat.Headers["Content-Length"] != "1024" ||
|
||||||
|
chat.Headers["If-None-Match"] != "*" || strings.Contains(chat.ObjectKey, "/42/") {
|
||||||
t.Fatalf("%+v", chat)
|
t.Fatalf("%+v", chat)
|
||||||
}
|
}
|
||||||
avatar, err := svc.CreateCredential(ctx, 42, &CreateInput{Purpose: entity.PurposeAvatar, Mime: "image/png", Size: 2048})
|
avatar, err := svc.CreateCredential(ctx, 42, &CreateInput{Purpose: entity.PurposeAvatar, Mime: "image/png", Size: 2048})
|
||||||
if err != nil || !strings.HasPrefix(avatar.ObjectKey, "perm/avatar/42/20260917/") {
|
if err != nil || !strings.HasPrefix(avatar.ObjectKey, "perm/avatar/20260917/") {
|
||||||
t.Fatalf("%+v %v", avatar, err)
|
t.Fatalf("%+v %v", avatar, err)
|
||||||
}
|
}
|
||||||
group, err := svc.CreateCredential(ctx, 42, &CreateInput{Purpose: entity.PurposeGroupAvatar, Mime: "image/jpeg", Size: 1024, GroupID: 7})
|
group, err := svc.CreateCredential(ctx, 42, &CreateInput{Purpose: entity.PurposeGroupAvatar, Mime: "image/jpeg", Size: 1024, GroupID: 7})
|
||||||
@ -79,6 +76,11 @@ func TestCreateCredentialRejectsBadMIMEAndSize(t *testing.T) {
|
|||||||
if _, err := svc.CreateCredential(ctx, 1, &CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: 50 << 20}); err == nil {
|
if _, err := svc.CreateCredential(ctx, 1, &CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: 50 << 20}); err == nil {
|
||||||
t.Fatal("huge image")
|
t.Fatal("huge image")
|
||||||
}
|
}
|
||||||
|
for _, mime := range []string{"image/heic", "image/heif", "image/webp"} {
|
||||||
|
if _, err := svc.CreateCredential(ctx, 1, &CreateInput{Purpose: entity.PurposeChatImage, Mime: mime, Size: 1024}); err == nil {
|
||||||
|
t.Fatalf("accepted %s", mime)
|
||||||
|
}
|
||||||
|
}
|
||||||
if _, err := svc.CreateCredential(ctx, 0, &CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: 1}); !errors.Is(err, errs.ErrUnauthorized) {
|
if _, err := svc.CreateCredential(ctx, 0, &CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: 1}); !errors.Is(err, errs.ErrUnauthorized) {
|
||||||
t.Fatalf("uid %v", err)
|
t.Fatalf("uid %v", err)
|
||||||
}
|
}
|
||||||
@ -95,3 +97,24 @@ func TestCreateDownloadValidatesKey(t *testing.T) {
|
|||||||
t.Fatal("path escape")
|
t.Fatal("path escape")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCreateCredentialsBatch(t *testing.T) {
|
||||||
|
svc := testUpload(t)
|
||||||
|
ctx := reqctx.With(t.Context(), &reqctx.RequestContext{UserID: 42, Language: "zh-CN"})
|
||||||
|
part := func(size int64) CreateInput {
|
||||||
|
return CreateInput{Purpose: entity.PurposeChatImage, Mime: "image/jpeg", Size: size}
|
||||||
|
}
|
||||||
|
got, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{Items: []CreateInput{part(1024), part(2048), part(4096)}})
|
||||||
|
if err != nil || got == nil || len(got.Items) != 3 {
|
||||||
|
t.Fatalf("%+v %v", got, err)
|
||||||
|
}
|
||||||
|
if got.Items[0].ObjectKey == got.Items[1].ObjectKey || got.Items[1].Headers["Content-Length"] != "2048" {
|
||||||
|
t.Fatalf("%+v", got.Items)
|
||||||
|
}
|
||||||
|
if _, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{}); err == nil {
|
||||||
|
t.Fatal("empty")
|
||||||
|
}
|
||||||
|
if _, err := svc.CreateCredentials(ctx, 42, &BatchCreateInput{Items: []CreateInput{part(1), part(1), part(1), part(1)}}); err == nil {
|
||||||
|
t.Fatal("too many")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@ -10,15 +10,10 @@ import (
|
|||||||
|
|
||||||
// Presigner issues short-lived object-storage URLs. Infrastructure implements it.
|
// Presigner issues short-lived object-storage URLs. Infrastructure implements it.
|
||||||
type Presigner interface {
|
type Presigner interface {
|
||||||
PresignPut(ctx context.Context, key, contentType string, ttl time.Duration) (string, error)
|
PresignPut(ctx context.Context, key, contentType string, size int64, ttl time.Duration) (string, error)
|
||||||
PresignGet(ctx context.Context, key string, ttl time.Duration) (string, error)
|
PresignGet(ctx context.Context, key string, ttl time.Duration) (string, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
type IDGen interface {
|
|
||||||
NextID() (int64, error)
|
|
||||||
}
|
|
||||||
|
|
||||||
// AvatarACL gates group_avatar uploads. Nil skips the check (tests).
|
|
||||||
type AvatarACL interface {
|
type AvatarACL interface {
|
||||||
CanSetAvatar(ctx context.Context, userID, groupID int64) error
|
CanSetAvatar(ctx context.Context, userID, groupID int64) error
|
||||||
}
|
}
|
||||||
@ -33,14 +28,13 @@ type Options struct {
|
|||||||
// Service issues upload/download credentials. HTTP and TCP share it.
|
// Service issues upload/download credentials. HTTP and TCP share it.
|
||||||
type Service struct {
|
type Service struct {
|
||||||
store Presigner
|
store Presigner
|
||||||
ids IDGen
|
|
||||||
ttl time.Duration
|
ttl time.Duration
|
||||||
limit *ratelimit.Limiter
|
limit *ratelimit.Limiter
|
||||||
now func() time.Time
|
now func() time.Time
|
||||||
groups AvatarACL
|
groups AvatarACL
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewService(store Presigner, ids IDGen, opt Options) *Service {
|
func NewService(store Presigner, opt Options) *Service {
|
||||||
ttl := opt.TTL
|
ttl := opt.TTL
|
||||||
if ttl <= 0 {
|
if ttl <= 0 {
|
||||||
ttl = 15 * time.Minute
|
ttl = 15 * time.Minute
|
||||||
@ -49,11 +43,11 @@ func NewService(store Presigner, ids IDGen, opt Options) *Service {
|
|||||||
if clock == nil {
|
if clock == nil {
|
||||||
clock = func() time.Time { return time.Now().UTC() }
|
clock = func() time.Time { return time.Now().UTC() }
|
||||||
}
|
}
|
||||||
return &Service{store: store, ids: ids, ttl: ttl, limit: opt.Limit, now: clock, groups: opt.Groups}
|
return &Service{store: store, ttl: ttl, limit: opt.Limit, now: clock, groups: opt.Groups}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Service) ready() bool {
|
func (s *Service) ready() bool {
|
||||||
return s != nil && s.store != nil && s.ids != nil
|
return s != nil && s.store != nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (s *Service) expiresIn() int64 {
|
func (s *Service) expiresIn() int64 {
|
||||||
|
|||||||
@ -7,6 +7,7 @@ import (
|
|||||||
|
|
||||||
uploadentity "spark-server/internal/module/upload/entity"
|
uploadentity "spark-server/internal/module/upload/entity"
|
||||||
"spark-server/internal/module/user/entity"
|
"spark-server/internal/module/user/entity"
|
||||||
|
"spark-server/internal/pkg/reqctx"
|
||||||
)
|
)
|
||||||
|
|
||||||
const defaultAvatarMIME = "image/png"
|
const defaultAvatarMIME = "image/png"
|
||||||
@ -42,18 +43,14 @@ func (s *Service) stampDefaultAvatar(ctx context.Context, u *entity.User) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *Service) putLetterPNG(ctx context.Context, u *entity.User) string {
|
func (s *Service) putLetterPNG(ctx context.Context, u *entity.User) string {
|
||||||
if s == nil || s.objects == nil || s.ids == nil || u == nil || u.ID <= 0 {
|
if s == nil || s.objects == nil || u == nil || u.ID <= 0 {
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
raw, err := entity.LetterAvatarPNG(u.Nickname)
|
raw, err := entity.LetterAvatarPNG(u.Nickname)
|
||||||
if err != nil || len(raw) == 0 {
|
if err != nil || len(raw) == 0 {
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
objectID, err := s.ids.NextID()
|
key := uploadentity.BuildObjectKey(uploadentity.PurposeAvatar, u.ID, reqctx.NewObjectName(), time.Now().UTC().Format("20060102"), "png")
|
||||||
if err != nil {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
key := uploadentity.BuildObjectKey(uploadentity.PurposeAvatar, u.ID, objectID, time.Now().Format("20060102"), "png")
|
|
||||||
if err := s.objects.Put(ctx, key, raw, defaultAvatarMIME); err != nil {
|
if err := s.objects.Put(ctx, key, raw, defaultAvatarMIME); err != nil {
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|||||||
@ -27,10 +27,15 @@ func TestAuthenticateTestEmailAnyCode(t *testing.T) {
|
|||||||
if u1.ID <= 0 || !entity.ValidSparkID(u1.SparkID) || u1.Nickname != "test01" || !entity.ValidQRCode(u1.QRCode) {
|
if u1.ID <= 0 || !entity.ValidSparkID(u1.SparkID) || u1.Nickname != "test01" || !entity.ValidQRCode(u1.QRCode) {
|
||||||
t.Fatalf("%+v", u1)
|
t.Fatalf("%+v", u1)
|
||||||
}
|
}
|
||||||
prefix := "perm/avatar/" + strconv.FormatInt(u1.ID, 10) + "/"
|
if !strings.HasPrefix(u1.Avatar, "perm/avatar/") || !strings.HasSuffix(u1.Avatar, ".png") {
|
||||||
if !strings.HasPrefix(u1.Avatar, prefix) || !strings.HasSuffix(u1.Avatar, ".png") {
|
|
||||||
t.Fatalf("default avatar %s", u1.Avatar)
|
t.Fatalf("default avatar %s", u1.Avatar)
|
||||||
}
|
}
|
||||||
|
uid := strconv.FormatInt(u1.ID, 10)
|
||||||
|
for _, part := range strings.Split(u1.Avatar, "/") {
|
||||||
|
if part == uid {
|
||||||
|
t.Fatalf("avatar leaked uid %s", u1.Avatar)
|
||||||
|
}
|
||||||
|
}
|
||||||
store, ok := svc.objects.(*memObjects)
|
store, ok := svc.objects.(*memObjects)
|
||||||
if !ok {
|
if !ok {
|
||||||
t.Fatal("mem objects")
|
t.Fatal("mem objects")
|
||||||
|
|||||||
@ -11,6 +11,9 @@ func init() {
|
|||||||
KeyFieldSize: {
|
KeyFieldSize: {
|
||||||
ZhCN: "文件大小", En: "File size", ZhTW: "檔案大小",
|
ZhCN: "文件大小", En: "File size", ZhTW: "檔案大小",
|
||||||
},
|
},
|
||||||
|
KeyFieldUploadItems: {
|
||||||
|
ZhCN: "上传列表", En: "Upload list", ZhTW: "上傳列表",
|
||||||
|
},
|
||||||
KeyFieldObjectKey: {
|
KeyFieldObjectKey: {
|
||||||
ZhCN: "对象键", En: "Object key", ZhTW: "物件鍵",
|
ZhCN: "对象键", En: "Object key", ZhTW: "物件鍵",
|
||||||
},
|
},
|
||||||
@ -50,6 +53,12 @@ func init() {
|
|||||||
KeyUploadGroupRequired: {
|
KeyUploadGroupRequired: {
|
||||||
ZhCN: "请选择群", En: "Please choose a group", ZhTW: "請選擇群",
|
ZhCN: "请选择群", En: "Please choose a group", ZhTW: "請選擇群",
|
||||||
},
|
},
|
||||||
|
KeyUploadItemsRequired: {
|
||||||
|
ZhCN: "请填写要上传的文件", En: "Please provide files to upload", ZhTW: "請填寫要上傳的檔案",
|
||||||
|
},
|
||||||
|
KeyUploadItemsInvalid: {
|
||||||
|
ZhCN: "一次最多签发 3 个上传凭证", En: "At most 3 upload credentials per request", ZhTW: "一次最多簽發 3 個上傳憑證",
|
||||||
|
},
|
||||||
KeyFavoriteMsgRequired: {
|
KeyFavoriteMsgRequired: {
|
||||||
ZhCN: "请选择消息", En: "Please choose a message", ZhTW: "請選擇訊息",
|
ZhCN: "请选择消息", En: "Please choose a message", ZhTW: "請選擇訊息",
|
||||||
},
|
},
|
||||||
|
|||||||
@ -42,6 +42,7 @@ const (
|
|||||||
KeyFieldPurpose = "field.purpose"
|
KeyFieldPurpose = "field.purpose"
|
||||||
KeyFieldMime = "field.mime"
|
KeyFieldMime = "field.mime"
|
||||||
KeyFieldSize = "field.size"
|
KeyFieldSize = "field.size"
|
||||||
|
KeyFieldUploadItems = "field.upload_items"
|
||||||
KeyFieldObjectKey = "field.object_key"
|
KeyFieldObjectKey = "field.object_key"
|
||||||
KeyFieldMsgID = "field.msg_id"
|
KeyFieldMsgID = "field.msg_id"
|
||||||
KeyFieldClientMsgID = "field.client_msg_id"
|
KeyFieldClientMsgID = "field.client_msg_id"
|
||||||
@ -90,6 +91,8 @@ const (
|
|||||||
KeyUploadKeyRequired = "validate.upload.key_required"
|
KeyUploadKeyRequired = "validate.upload.key_required"
|
||||||
KeyUploadKeyInvalid = "validate.upload.key_invalid"
|
KeyUploadKeyInvalid = "validate.upload.key_invalid"
|
||||||
KeyUploadGroupRequired = "validate.upload.group_required"
|
KeyUploadGroupRequired = "validate.upload.group_required"
|
||||||
|
KeyUploadItemsRequired = "validate.upload.items_required"
|
||||||
|
KeyUploadItemsInvalid = "validate.upload.items_invalid"
|
||||||
KeyFavoriteMsgRequired = "validate.favorite.msg_required"
|
KeyFavoriteMsgRequired = "validate.favorite.msg_required"
|
||||||
KeyFavoriteMsgInvalid = "validate.favorite.msg_invalid"
|
KeyFavoriteMsgInvalid = "validate.favorite.msg_invalid"
|
||||||
KeyMessageClientMsgRequired = "validate.message.client_msg_required"
|
KeyMessageClientMsgRequired = "validate.message.client_msg_required"
|
||||||
|
|||||||
@ -3,7 +3,8 @@ package reqctx
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"crypto/rand"
|
"crypto/rand"
|
||||||
"encoding/hex"
|
"encoding/binary"
|
||||||
|
"math/big"
|
||||||
"net"
|
"net"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
@ -91,11 +92,51 @@ func stripHost(v string) string {
|
|||||||
return v
|
return v
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewID returns a 16-byte hex id for request_id.
|
const (
|
||||||
|
base62Alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
|
||||||
|
base62Len = 11
|
||||||
|
objectNameLen = 16
|
||||||
|
objectNameSrc = 12
|
||||||
|
)
|
||||||
|
|
||||||
|
// NewID returns an 8-byte base62 id (11 chars) for request_id and JWT jti.
|
||||||
func NewID() string {
|
func NewID() string {
|
||||||
var b [16]byte
|
var b [8]byte
|
||||||
if _, err := rand.Read(b[:]); err != nil {
|
if _, err := rand.Read(b[:]); err != nil {
|
||||||
return "00000000000000000000000000000000"
|
return "00000000000"
|
||||||
}
|
}
|
||||||
return hex.EncodeToString(b[:])
|
return encodeBase62(binary.BigEndian.Uint64(b[:]))
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewObjectName returns a 12-byte base62 id (16 chars) for object_key filenames. Not Snowflake.
|
||||||
|
func NewObjectName() string {
|
||||||
|
return randomBase62(objectNameSrc, objectNameLen)
|
||||||
|
}
|
||||||
|
|
||||||
|
func encodeBase62(n uint64) string {
|
||||||
|
var out [base62Len]byte
|
||||||
|
for i := base62Len - 1; i >= 0; i-- {
|
||||||
|
out[i] = base62Alphabet[n%62]
|
||||||
|
n /= 62
|
||||||
|
}
|
||||||
|
return string(out[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
func randomBase62(byteN, width int) string {
|
||||||
|
if byteN < 1 || width < 1 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
b := make([]byte, byteN)
|
||||||
|
if _, err := rand.Read(b); err != nil {
|
||||||
|
return strings.Repeat("0", width)
|
||||||
|
}
|
||||||
|
n := new(big.Int).SetBytes(b)
|
||||||
|
out := make([]byte, width)
|
||||||
|
base := big.NewInt(62)
|
||||||
|
r := new(big.Int)
|
||||||
|
for i := width - 1; i >= 0; i-- {
|
||||||
|
n.DivMod(n, base, r)
|
||||||
|
out[i] = base62Alphabet[r.Int64()]
|
||||||
|
}
|
||||||
|
return string(out)
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,6 +1,9 @@
|
|||||||
package reqctx
|
package reqctx
|
||||||
|
|
||||||
import "testing"
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
func TestParseHTTPRealIP(t *testing.T) {
|
func TestParseHTTPRealIP(t *testing.T) {
|
||||||
got := ParseHTTPRealIP(" 203.0.113.10 ", "198.51.100.1, 192.0.2.1", "192.0.2.1:1234")
|
got := ParseHTTPRealIP(" 203.0.113.10 ", "198.51.100.1, 192.0.2.1", "192.0.2.1:1234")
|
||||||
@ -27,6 +30,48 @@ func TestRealIPFromContext(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestNewIDLength(t *testing.T) {
|
||||||
|
id := NewID()
|
||||||
|
if len(id) != 11 {
|
||||||
|
t.Fatalf("len %d %q", len(id), id)
|
||||||
|
}
|
||||||
|
for i := 0; i < len(id); i++ {
|
||||||
|
if !strings.ContainsRune(base62Alphabet, rune(id[i])) {
|
||||||
|
t.Fatalf("char %q in %q", id[i], id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if id == NewID() {
|
||||||
|
t.Fatal("not unique")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNewObjectNameLength(t *testing.T) {
|
||||||
|
id := NewObjectName()
|
||||||
|
if len(id) != 16 {
|
||||||
|
t.Fatalf("len %d %q", len(id), id)
|
||||||
|
}
|
||||||
|
for i := 0; i < len(id); i++ {
|
||||||
|
if !strings.ContainsRune(base62Alphabet, rune(id[i])) {
|
||||||
|
t.Fatalf("char %q in %q", id[i], id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if id == NewObjectName() {
|
||||||
|
t.Fatal("not unique")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEncodeBase62(t *testing.T) {
|
||||||
|
if got := encodeBase62(0); got != "00000000000" {
|
||||||
|
t.Fatalf("%q", got)
|
||||||
|
}
|
||||||
|
if got := encodeBase62(61); got != "0000000000z" {
|
||||||
|
t.Fatalf("%q", got)
|
||||||
|
}
|
||||||
|
if got := encodeBase62(62); got != "00000000010" {
|
||||||
|
t.Fatalf("%q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRequestIDFromContext(t *testing.T) {
|
func TestRequestIDFromContext(t *testing.T) {
|
||||||
ctx := With(t.Context(), &RequestContext{RequestID: "abc"})
|
ctx := With(t.Context(), &RequestContext{RequestID: "abc"})
|
||||||
if RequestID(ctx) != "abc" {
|
if RequestID(ctx) != "abc" {
|
||||||
|
|||||||
@ -7,7 +7,7 @@ info:
|
|||||||
|
|
||||||
## 环境与地址
|
## 环境与地址
|
||||||
|
|
||||||
公网只开放 `443`。Nginx 按 TLS SNI 分流:HTTP 证书打到 `sparkapi.wyrs365.com`,IM 证书打到 `sparkim.wyrs365.com`。进程本身只绑本机。
|
公网只开放 `443`。Nginx 按 TLS SNI 分流:HTTP 证书打到 `sparkapi.wyrs365.com`,IM 证书打到 `sparkim.wyrs365.com`,对象存储(RustFS S3)打到 `sparkr.wyrs365.com`。进程本身只绑本机。
|
||||||
|
|
||||||
| 环境 | 用途 | HTTP Base URL | IM 长连接 |
|
| 环境 | 用途 | HTTP Base URL | IM 长连接 |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
@ -36,7 +36,7 @@ info:
|
|||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| `code` | **字符串** | `"0"` 成功;其它失败。不要用数字 `0` 比较。全表见 [错误码](/docs/errors)。 |
|
| `code` | **字符串** | `"0"` 成功;其它失败。不要用数字 `0` 比较。全表见 [错误码](/docs/errors)。 |
|
||||||
| `msg` | 字符串 | 给人看的文案,随请求语言变化(`zh-CN` / `en` / `zh-TW`)。成功一般为 `ok`。 |
|
| `msg` | 字符串 | 给人看的文案,随请求语言变化(`zh-CN` / `en` / `zh-TW`)。成功一般为 `ok`。 |
|
||||||
| `request_id` | 字符串 | 服务端生成的本次请求 id。客户端不用传;从响应体读取。与 Header `X-Request-ID` 相同。 |
|
| `request_id` | 字符串 | 服务端生成,11 位 base62(`0-9A-Za-z`)。客户端不用传;从响应体读取。与 Header `X-Request-ID` 相同。 |
|
||||||
| `data` | object / array | 成功时为业务对象;失败时固定 `{}`。字段缺失当空对象,不要当 `null`。 |
|
| `data` | object / array | 成功时为业务对象;失败时固定 `{}`。字段缺失当空对象,不要当 `null`。 |
|
||||||
|
|
||||||
**判定成功只看 `code === "0"`。** 不要只看 HTTP 状态码。
|
**判定成功只看 `code === "0"`。** 不要只看 HTTP 状态码。
|
||||||
@ -655,7 +655,7 @@ paths:
|
|||||||
uid: 1940000000000000000
|
uid: 1940000000000000000
|
||||||
spark_id: alice_ok
|
spark_id: alice_ok
|
||||||
nickname: test01
|
nickname: test01
|
||||||
avatar: perm/avatar/1940000000000000000/20260917/1940000000000000001.png
|
avatar: perm/avatar/20260917/3H8kPq2mN1Rabcde.png
|
||||||
qrcode: "-NDveu-9Q"
|
qrcode: "-NDveu-9Q"
|
||||||
birthday: ""
|
birthday: ""
|
||||||
gender: 0
|
gender: 0
|
||||||
@ -1106,9 +1106,13 @@ paths:
|
|||||||
description: |
|
description: |
|
||||||
客户端只传 `purpose` / `mime` / `size`,服务端生成 object_key 并签发短时 Presigned PUT。
|
客户端只传 `purpose` / `mime` / `size`,服务端生成 object_key 并签发短时 Presigned PUT。
|
||||||
文件直传 RustFS,不经过 Gin。
|
文件直传 RustFS,不经过 Gin。
|
||||||
`avatar` 写入 `perm/avatar/{uid}/…`(不过期);`group_avatar` 写入 `perm/group/{群号}/…`(需 `group_id`,群主/管理员);
|
`avatar` 写入 `perm/avatar/{yyyyMMdd}/{name}.{ext}`(不过期,路径不含 uid);
|
||||||
聊天媒体写入 `chat/{yyyyMMdd}/{uid}/…`(桶生命周期过期)。
|
`group_avatar` 写入 `perm/group/{群号}/{name}.{ext}`(需 `group_id`,群主/管理员);
|
||||||
客户端 PUT 时必须带返回的 `headers`(至少 `Content-Type`)。消息与头像只存 `object_key`。
|
聊天按类型:`chat/image|voice|video|gif|file/{yyyyMMdd}/{name}.{ext}`(桶生命周期按前缀过期)。
|
||||||
|
`{name}` 为服务端 16 位 base62 随机串,不占用 Snowflake。
|
||||||
|
发图片:客户端压成 JPEG/PNG 后 `POST /v1/files/batch` 直传缩略图、压缩大图、原图(未勾选原图只传前两项)。
|
||||||
|
服务端不转码。iOS HEIC、Android WebP 须客户端先转 JPEG/PNG,两端才能互看。
|
||||||
|
客户端 PUT 必须原样带返回的 `headers`。消息只存 object_key。
|
||||||
operationId: createUploadCredential
|
operationId: createUploadCredential
|
||||||
requestBody:
|
requestBody:
|
||||||
required: true
|
required: true
|
||||||
@ -1131,12 +1135,61 @@ paths:
|
|||||||
code: "0"
|
code: "0"
|
||||||
msg: ok
|
msg: ok
|
||||||
data:
|
data:
|
||||||
object_key: chat/20260917/1/2.jpg
|
object_key: chat/image/20260917/3H8kPq2mN1Rabcde.jpg
|
||||||
upload_url: http://47.236.66.9:9000/spark/chat/20260917/1/2.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256
|
upload_url: https://sparkr.wyrs365.com/spark/chat/image/20260917/3H8kPq2mN1Rabcde.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256
|
||||||
method: PUT
|
method: PUT
|
||||||
headers:
|
headers:
|
||||||
Content-Type: image/jpeg
|
Content-Type: image/jpeg
|
||||||
|
Content-Length: "245760"
|
||||||
|
If-None-Match: "*"
|
||||||
expires_in: 900
|
expires_in: 900
|
||||||
|
/v1/files/batch:
|
||||||
|
post:
|
||||||
|
tags: [files]
|
||||||
|
summary: 批量签发直传凭证
|
||||||
|
description: |
|
||||||
|
一次返回 1~3 个 Presigned PUT,顺序与 `items` 相同。聊天图片请按「缩略图、压缩大图、原图」排列;未勾选原图只传前两项。
|
||||||
|
mime 仅 `image/jpeg` / `image/png`(HEIC、WebP 请客户端先转)。限流按条数计入 `upload_per_second`。文件仍直传 RustFS。
|
||||||
|
operationId: createUploadCredentials
|
||||||
|
requestBody:
|
||||||
|
required: true
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/CreateUploadBatchRequest"
|
||||||
|
example:
|
||||||
|
items:
|
||||||
|
- purpose: chat_image
|
||||||
|
mime: image/jpeg
|
||||||
|
size: 8192
|
||||||
|
- purpose: chat_image
|
||||||
|
mime: image/jpeg
|
||||||
|
size: 245760
|
||||||
|
- purpose: chat_image
|
||||||
|
mime: image/jpeg
|
||||||
|
size: 3500000
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: 统一 envelope
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/Envelope"
|
||||||
|
example:
|
||||||
|
code: "0"
|
||||||
|
msg: ok
|
||||||
|
data:
|
||||||
|
items:
|
||||||
|
- object_key: chat/image/20260917/3H8kPq2mN1Rabcde.jpg
|
||||||
|
method: PUT
|
||||||
|
headers:
|
||||||
|
Content-Type: image/jpeg
|
||||||
|
Content-Length: "8192"
|
||||||
|
If-None-Match: "*"
|
||||||
|
- object_key: chat/image/20260917/4kPq2mN1RabcdeXY.jpg
|
||||||
|
method: PUT
|
||||||
|
- object_key: chat/image/20260917/5mN1RabcdeXY12ab.jpg
|
||||||
|
method: PUT
|
||||||
/v1/files/downloads:
|
/v1/files/downloads:
|
||||||
post:
|
post:
|
||||||
tags: [files]
|
tags: [files]
|
||||||
@ -1181,7 +1234,7 @@ paths:
|
|||||||
tags: [favorites]
|
tags: [favorites]
|
||||||
summary: 收藏一条消息
|
summary: 收藏一条消息
|
||||||
description: |
|
description: |
|
||||||
会话成员可将消息收入收藏。图片/语音/视频/文件会 CopyObject 到 `perm/fav/{uid}/…`。
|
会话成员可将消息收入收藏。图片/语音/视频/文件会 CopyObject 到 `perm/fav/{yyyyMMdd}/{name}.{ext}`。
|
||||||
源对象已从 `chat/` 过期删除则失败(`code=40402`),不提供本地补传。
|
源对象已从 `chat/` 过期删除则失败(`code=40402`),不提供本地补传。
|
||||||
同一 `msg_id` 重复收藏返回已有记录。撤回不影响已收藏快照。
|
同一 `msg_id` 重复收藏返回已有记录。撤回不影响已收藏快照。
|
||||||
operationId: createFavorite
|
operationId: createFavorite
|
||||||
@ -1477,8 +1530,8 @@ components:
|
|||||||
example: ok
|
example: ok
|
||||||
request_id:
|
request_id:
|
||||||
type: string
|
type: string
|
||||||
description: 服务端生成的本次请求 id。客户端不用传,从响应体读取。
|
description: 服务端生成的本次请求 id(11 位 base62,`0-9A-Za-z`)。客户端不用传,从响应体读取。
|
||||||
example: a1b2c3d4e5f6a7b8a1b2c3d4e5f6a7b8
|
example: 3H8kPq2mN1R
|
||||||
data:
|
data:
|
||||||
type: object
|
type: object
|
||||||
additionalProperties: true
|
additionalProperties: true
|
||||||
@ -1743,6 +1796,9 @@ components:
|
|||||||
mime:
|
mime:
|
||||||
type: string
|
type: string
|
||||||
example: image/jpeg
|
example: image/jpeg
|
||||||
|
description: |
|
||||||
|
avatar / group_avatar / chat_image 仅 image/jpeg、image/jpg、image/png。
|
||||||
|
iOS HEIC、Android WebP 须客户端转成 JPEG/PNG 再申请凭证。
|
||||||
size:
|
size:
|
||||||
type: integer
|
type: integer
|
||||||
format: int64
|
format: int64
|
||||||
@ -1752,6 +1808,16 @@ components:
|
|||||||
format: int64
|
format: int64
|
||||||
minimum: 1
|
minimum: 1
|
||||||
description: purpose=group_avatar 时必填,为自增群号
|
description: purpose=group_avatar 时必填,为自增群号
|
||||||
|
CreateUploadBatchRequest:
|
||||||
|
type: object
|
||||||
|
required: [items]
|
||||||
|
properties:
|
||||||
|
items:
|
||||||
|
type: array
|
||||||
|
minItems: 1
|
||||||
|
maxItems: 3
|
||||||
|
items:
|
||||||
|
$ref: "#/components/schemas/CreateUploadRequest"
|
||||||
CreateDownloadRequest:
|
CreateDownloadRequest:
|
||||||
type: object
|
type: object
|
||||||
required: [object_key]
|
required: [object_key]
|
||||||
@ -1792,6 +1858,39 @@ components:
|
|||||||
properties:
|
properties:
|
||||||
body:
|
body:
|
||||||
type: string
|
type: string
|
||||||
|
image:
|
||||||
|
type: object
|
||||||
|
description: |
|
||||||
|
t=2。点开看 `key`(压缩预览);气泡用 `thumb_key`;`orig_key` 有则显示「查看原图」。
|
||||||
|
三档由客户端压成 JPEG/PNG 后 `POST /v1/files/batch` 直传。服务端不转码;不要传 HEIC/WebP。
|
||||||
|
required: [key]
|
||||||
|
properties:
|
||||||
|
key:
|
||||||
|
type: string
|
||||||
|
description: 点开查看用的压缩图 object_key
|
||||||
|
thumb_key:
|
||||||
|
type: string
|
||||||
|
description: 气泡缩略图,可省略
|
||||||
|
orig_key:
|
||||||
|
type: string
|
||||||
|
description: 原图 object_key;空则没有「查看原图」
|
||||||
|
orig_size:
|
||||||
|
type: integer
|
||||||
|
format: int64
|
||||||
|
description: 原图像素文件大小,供「查看原图 3.2MB」按钮
|
||||||
|
orig_w:
|
||||||
|
type: integer
|
||||||
|
orig_h:
|
||||||
|
type: integer
|
||||||
|
mime:
|
||||||
|
type: string
|
||||||
|
w:
|
||||||
|
type: integer
|
||||||
|
h:
|
||||||
|
type: integer
|
||||||
|
size:
|
||||||
|
type: integer
|
||||||
|
format: int64
|
||||||
at_all:
|
at_all:
|
||||||
type: boolean
|
type: boolean
|
||||||
description: "@所有人,仅群主/管理员"
|
description: "@所有人,仅群主/管理员"
|
||||||
|
|||||||
@ -27,11 +27,11 @@ import (
|
|||||||
|
|
||||||
type stubPresign struct{}
|
type stubPresign struct{}
|
||||||
|
|
||||||
func (stubPresign) PresignPut(_ context.Context, key, _ string, _ time.Duration) (string, error) {
|
func (s *stubPresign) PresignPut(_ context.Context, key, _ string, _ int64, _ time.Duration) (string, error) {
|
||||||
return "http://rustfs.test/" + key + "?sig=1", nil
|
return "http://rustfs.test/" + key + "?sig=1", nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (stubPresign) PresignGet(_ context.Context, key string, _ time.Duration) (string, error) {
|
func (s *stubPresign) PresignGet(_ context.Context, key string, _ time.Duration) (string, error) {
|
||||||
return "http://rustfs.test/" + key + "?get=1", nil
|
return "http://rustfs.test/" + key + "?get=1", nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -48,7 +48,7 @@ func testFileEngine(t *testing.T) (*ginEngineBundle, *id.Generator, *msgmodel.Me
|
|||||||
users := userlogic.NewService(usermodel.NewMemoryRepo(), usermodel.NewMemoryCodes(), ids, nil)
|
users := userlogic.NewService(usermodel.NewMemoryRepo(), usermodel.NewMemoryCodes(), ids, nil)
|
||||||
msgs := msgmodel.NewMemoryRepo()
|
msgs := msgmodel.NewMemoryRepo()
|
||||||
convs := convmodel.NewMemoryP2P()
|
convs := convmodel.NewMemoryP2P()
|
||||||
files := uploadlogic.NewService(stubPresign{}, ids, uploadlogic.Options{
|
files := uploadlogic.NewService(&stubPresign{}, uploadlogic.Options{
|
||||||
Now: func() time.Time { return time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC) },
|
Now: func() time.Time { return time.Date(2026, 9, 17, 0, 0, 0, 0, time.UTC) },
|
||||||
})
|
})
|
||||||
favs := favlogic.NewService(favmodel.NewMemoryRepo(), msgs, convs, nil, ids)
|
favs := favlogic.NewService(favmodel.NewMemoryRepo(), msgs, convs, nil, ids)
|
||||||
@ -71,13 +71,25 @@ func TestUploadCredentialHTTP(t *testing.T) {
|
|||||||
ObjectKey string `json:"object_key"`
|
ObjectKey string `json:"object_key"`
|
||||||
Method string `json:"method"`
|
Method string `json:"method"`
|
||||||
}
|
}
|
||||||
if err := json.Unmarshal(env.Data, &cred); err != nil || cred.Method != "PUT" || !strings.Contains(cred.ObjectKey, "chat/20260917/") {
|
if err := json.Unmarshal(env.Data, &cred); err != nil || cred.Method != "PUT" || !strings.Contains(cred.ObjectKey, "chat/image/20260917/") {
|
||||||
t.Fatalf("%s %v", env.Data, err)
|
t.Fatalf("%s %v", env.Data, err)
|
||||||
}
|
}
|
||||||
bad := authJSON(t, h, &tok, http.MethodPost, "/v1/files", `{"purpose":"avatar","mime":"application/pdf","size":10}`)
|
bad := authJSON(t, h, &tok, http.MethodPost, "/v1/files", `{"purpose":"avatar","mime":"application/pdf","size":10}`)
|
||||||
if bad.Code != errs.ErrInvalidArgument.Code {
|
if bad.Code != errs.ErrInvalidArgument.Code {
|
||||||
t.Fatalf("mime %+v", bad)
|
t.Fatalf("mime %+v", bad)
|
||||||
}
|
}
|
||||||
|
batch := authJSON(t, h, &tok, http.MethodPost, "/v1/files/batch", `{"items":[{"purpose":"chat_image","mime":"image/jpeg","size":1024},{"purpose":"chat_image","mime":"image/jpeg","size":2048}]}`)
|
||||||
|
if batch.Code != "0" {
|
||||||
|
t.Fatalf("batch %+v", batch)
|
||||||
|
}
|
||||||
|
var set struct {
|
||||||
|
Items []struct {
|
||||||
|
ObjectKey string `json:"object_key"`
|
||||||
|
} `json:"items"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(batch.Data, &set); err != nil || len(set.Items) != 2 || set.Items[0].ObjectKey == set.Items[1].ObjectKey {
|
||||||
|
t.Fatalf("%s %v", batch.Data, err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestFavoriteTextHTTP(t *testing.T) {
|
func TestFavoriteTextHTTP(t *testing.T) {
|
||||||
|
|||||||
@ -36,6 +36,24 @@ func (h *Handler) Create(c *gin.Context) {
|
|||||||
respond.JSON(c, out)
|
respond.JSON(c, out)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (h *Handler) CreateBatch(c *gin.Context) {
|
||||||
|
uid, ok := currentUID(c)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var in uploadlogic.BatchCreateInput
|
||||||
|
if err := respond.BindJSON(c, &in); err != nil {
|
||||||
|
respond.JSONError(c, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
out, err := h.files.CreateCredentials(c.Request.Context(), uid, &in)
|
||||||
|
if err != nil {
|
||||||
|
respond.JSONError(c, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
respond.JSON(c, out)
|
||||||
|
}
|
||||||
|
|
||||||
func (h *Handler) Download(c *gin.Context) {
|
func (h *Handler) Download(c *gin.Context) {
|
||||||
uid, ok := currentUID(c)
|
uid, ok := currentUID(c)
|
||||||
if !ok {
|
if !ok {
|
||||||
|
|||||||
@ -132,6 +132,7 @@ func registerFileRoutes(v1 *gin.RouterGroup, deps *EngineDeps) {
|
|||||||
h := filehandler.New(deps.Files)
|
h := filehandler.New(deps.Files)
|
||||||
g := v1.Group("/files")
|
g := v1.Group("/files")
|
||||||
g.POST("", h.Create)
|
g.POST("", h.Create)
|
||||||
|
g.POST("/batch", h.CreateBatch)
|
||||||
g.POST("/downloads", h.Download)
|
g.POST("/downloads", h.Download)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -34,7 +34,7 @@ func TestHealthzEnvelopeAndTraceHeaders(t *testing.T) {
|
|||||||
t.Fatalf("status %d", w.Code)
|
t.Fatalf("status %d", w.Code)
|
||||||
}
|
}
|
||||||
got := w.Header().Get("X-Request-ID")
|
got := w.Header().Get("X-Request-ID")
|
||||||
if len(got) != 32 {
|
if len(got) != 11 {
|
||||||
t.Fatalf("server request id %q headers %v", got, w.Header())
|
t.Fatalf("server request id %q headers %v", got, w.Header())
|
||||||
}
|
}
|
||||||
if w.Header().Get("X-Trace-ID") != "" {
|
if w.Header().Get("X-Trace-ID") != "" {
|
||||||
@ -47,7 +47,7 @@ func TestHealthzEnvelopeAndTraceHeaders(t *testing.T) {
|
|||||||
if env.Code != "0" || env.Msg != "ok" {
|
if env.Code != "0" || env.Msg != "ok" {
|
||||||
t.Fatalf("%+v", env)
|
t.Fatalf("%+v", env)
|
||||||
}
|
}
|
||||||
if env.RequestID != got || len(env.RequestID) != 32 {
|
if env.RequestID != got || len(env.RequestID) != 11 {
|
||||||
t.Fatalf("body request_id %q header %q", env.RequestID, got)
|
t.Fatalf("body request_id %q header %q", env.RequestID, got)
|
||||||
}
|
}
|
||||||
if !bytes.Contains(env.Data, []byte(`"status":"ok"`)) {
|
if !bytes.Contains(env.Data, []byte(`"status":"ok"`)) {
|
||||||
@ -82,7 +82,7 @@ func TestReadyzHTTPStatus(t *testing.T) {
|
|||||||
if env.Code != errs.ErrAvailable.Code || !bytes.Contains(env.Data, []byte(`"mysql":"down"`)) {
|
if env.Code != errs.ErrAvailable.Code || !bytes.Contains(env.Data, []byte(`"mysql":"down"`)) {
|
||||||
t.Fatalf("%+v", env)
|
t.Fatalf("%+v", env)
|
||||||
}
|
}
|
||||||
if len(env.RequestID) != 32 {
|
if len(env.RequestID) != 11 {
|
||||||
t.Fatalf("readyz request_id %q", env.RequestID)
|
t.Fatalf("readyz request_id %q", env.RequestID)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user