feat: add ops center and node onboarding flow
This commit is contained in:
@@ -120,3 +120,10 @@ python deploy/linux/smoke_test.py --base-url http://127.0.0.1:8100
|
||||
- Linux 实机联调
|
||||
- 上线前复测
|
||||
- 真实服务器灰度验证
|
||||
|
||||
如果当前已经进入“海外单脑控制面 + 多机联调 + 准备发布”阶段,建议改为按下面顺序执行:
|
||||
|
||||
1. `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
2. `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export`
|
||||
3. `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review`
|
||||
4. `docs/26_domainCheck_发布前运行验证与交付模板.md`
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## 一、用途
|
||||
|
||||
用于在 Windows 本地把当前可交付内容整理成一份压缩包,便于:
|
||||
用于在 Windows 或 Linux 上把当前可交付内容整理成一份发布包,便于:
|
||||
|
||||
- 发给运维或部署同事
|
||||
- 存档版本快照
|
||||
@@ -16,31 +16,60 @@
|
||||
- `verify_domain_release.ps1`
|
||||
- `prepare_final_release.ps1`
|
||||
- `show_latest_release.ps1`
|
||||
- `package_domain_release.sh`
|
||||
- `verify_domain_release.sh`
|
||||
- `prepare_final_release.sh`
|
||||
- `show_latest_release.sh`
|
||||
|
||||
执行方式:
|
||||
Windows 执行方式:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\package_domain_release.ps1
|
||||
```
|
||||
|
||||
Linux / 海外主机执行方式:
|
||||
|
||||
```bash
|
||||
bash ./package_domain_release.sh
|
||||
```
|
||||
|
||||
如需一键完成“打包 + 验包 + 生成最终准备报告”:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\prepare_final_release.ps1
|
||||
```
|
||||
|
||||
```bash
|
||||
bash ./prepare_final_release.sh
|
||||
```
|
||||
|
||||
如需快速查看当前最新交付物:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
|
||||
```
|
||||
|
||||
```bash
|
||||
bash ./show_latest_release.sh
|
||||
```
|
||||
|
||||
注意:
|
||||
|
||||
- `show_latest_release.*` 现在不只是“显示最近一次 final report 是否存在”
|
||||
- 它会额外校验 `final_release_report.json` 是否真的对应当前 `latest_release.*`
|
||||
- 只有 `final_release_report_matches_latest=true` 时,`final_release_ok=true` 才有意义
|
||||
- `final_release_gate_decision / blocked_reasons / verify_ok / smoke_test_ok` 用于解释“为什么当前能签 / 不能签”
|
||||
- Ops Center 里的“直接创建 Release / 一键 Worker 灰度 / 一键 Control 发布”
|
||||
- 现在会硬性依赖 `final_release_ok=true`
|
||||
- 如果 final report 缺失、过期或门禁未 ready,前端会禁用按钮,后端接口也会拒绝执行
|
||||
|
||||
## 三、输出位置
|
||||
|
||||
打包后会生成:
|
||||
|
||||
- `release/domaincheck_release_时间戳/`
|
||||
- `release/domaincheck_release_时间戳.zip`
|
||||
- `release/domaincheck_release_时间戳.tar.gz`
|
||||
- `release/domaincheck_release_时间戳.sha256.txt`
|
||||
- `release/latest_release.txt`
|
||||
- `release/latest_release.json`
|
||||
@@ -50,6 +79,7 @@ powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
|
||||
|
||||
- `docs/`
|
||||
- `scripts/`
|
||||
- `scripts/` 中会同时带上 PowerShell 和 Shell 版发布脚本
|
||||
- `domain-api/`
|
||||
- `app`
|
||||
- `deploy`
|
||||
@@ -89,6 +119,10 @@ powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
|
||||
powershell -ExecutionPolicy Bypass -File .\verify_domain_release.ps1
|
||||
```
|
||||
|
||||
```bash
|
||||
bash ./verify_domain_release.sh
|
||||
```
|
||||
|
||||
5. Windows 联调先跑 `scripts/smoke_test_stack.ps1`
|
||||
6. Linux 部署前阅读:
|
||||
- `docs/05_domainCheck_Linux部署清单.md`
|
||||
@@ -106,6 +140,18 @@ powershell -ExecutionPolicy Bypass -File .\verify_domain_release.ps1
|
||||
|
||||
这样更适合交付、存档和迁移。
|
||||
|
||||
Windows 默认产物为 `.zip`,Linux 默认产物为 `.tar.gz`;两者都会生成同结构的 `release_manifest.json / .sha256.txt / latest_release.*`,便于海外控制面统一管理。
|
||||
|
||||
`latest_release.txt / latest_release.json` 用于快速定位“当前最新一份交付包”,避免人工翻目录。
|
||||
|
||||
若打包目录本身是 Git 仓库,脚本还会自动写入 `commit_sha / commit_ref`;海外控制面在创建 Release 时可以直接回填提交号,减少人工抄写出错。
|
||||
|
||||
`final_release_report.json` 用于记录最近一次“打包 + 验包”的最终状态。
|
||||
|
||||
`show_latest_release.*` 读取这个报告时,还会校验:
|
||||
|
||||
- `package_name`
|
||||
- `archive_path` / `zip_path`
|
||||
- `sha256`
|
||||
|
||||
也就是说,即使目录里留着一份旧的 `final_release_report.json`,只要它不是给当前最新包签出的,`final_release_ok` 也不会误报为 `true`。
|
||||
|
||||
@@ -65,9 +65,11 @@
|
||||
|
||||
1. 把最新交付包发送到目标 Linux 服务器
|
||||
2. 按 `docs/05`、`docs/13`、`docs/14` 的顺序执行部署、核对与收口
|
||||
3. 部署完成后再做一次正式服务态 `smoke test`
|
||||
4. 导出一份最终诊断包
|
||||
5. 进入灰度上线
|
||||
3. 如果当前已经进入海外单脑控制面阶段,再按 `docs/25` 执行统一收口
|
||||
4. 用 `docs/26` 完成发布前运行验证、证据导出与最终交付结论
|
||||
5. 部署完成后再做一次正式服务态 `smoke test`
|
||||
6. 导出一份最终诊断包
|
||||
7. 进入灰度上线
|
||||
|
||||
## 六、最终判断
|
||||
|
||||
|
||||
@@ -40,6 +40,51 @@
|
||||
|
||||
- `docs/20_domainCheck_临时海外控制面联调清单.md`
|
||||
|
||||
### 10. 海外主机集中运维与自动化方案
|
||||
|
||||
- `docs/21_domainCheck_海外主机集中运维与自动化方案.md`
|
||||
|
||||
### 11. 海外控制面统一入口
|
||||
|
||||
- `domain-api/deploy/multi-region/drive_ops_center.sh`
|
||||
- `domain-api/deploy/multi-region/export_go_live_bundle.sh`
|
||||
- `domain-api/deploy/multi-region/check_env_drift.sh`
|
||||
- `domain-api/deploy/multi-region/drive_release_hub.sh`
|
||||
- `domain-api/deploy/multi-region/drive_ops_action.sh`
|
||||
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
|
||||
- `domain-api/deploy/multi-region/check_ops_center_stack.sh`
|
||||
- `domain-api/deploy/multi-region/check_ops_contracts.sh`
|
||||
|
||||
### 12. 海外 Codex 驾驶员与 Ops Center 落地路线
|
||||
|
||||
- `docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md`
|
||||
|
||||
### 13. 终局运维架构设计:海外单脑控制面
|
||||
|
||||
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
|
||||
|
||||
### 14. Node Agent 协议与 Release Hub 设计
|
||||
|
||||
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
|
||||
|
||||
### 15. 运维 Contract Schema
|
||||
|
||||
- `docs/schemas/ops_agent_protocol.md`
|
||||
- `docs/schemas/release_hub_contract.md`
|
||||
- `docs/schemas/ops_driver_contract.md`
|
||||
- `docs/schemas/ops_job_contract.md`
|
||||
- `docs/schemas/ops_playbook_contract.md`
|
||||
- `docs/schemas/ops_observability_contract.md`
|
||||
- `docs/schemas/ops_stack_diagnosis_contract.md`
|
||||
|
||||
### 16. 海外单脑控制面上线收口总表
|
||||
|
||||
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
|
||||
### 17. 发布前运行验证与交付模板
|
||||
|
||||
- `docs/26_domainCheck_发布前运行验证与交付模板.md`
|
||||
|
||||
## 二、文档阅读顺序
|
||||
|
||||
### 1. 先看总体方案
|
||||
@@ -58,6 +103,19 @@
|
||||
- `docs/17_domainCheck_全流程部署实操手册.md`
|
||||
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
|
||||
- `docs/20_domainCheck_临时海外控制面联调清单.md`
|
||||
- `docs/21_domainCheck_海外主机集中运维与自动化方案.md`
|
||||
- `docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md`
|
||||
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
|
||||
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
|
||||
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
- `docs/26_domainCheck_发布前运行验证与交付模板.md`
|
||||
- `docs/schemas/ops_agent_protocol.md`
|
||||
- `docs/schemas/release_hub_contract.md`
|
||||
- `docs/schemas/ops_driver_contract.md`
|
||||
- `docs/schemas/ops_job_contract.md`
|
||||
- `docs/schemas/ops_playbook_contract.md`
|
||||
- `docs/schemas/ops_observability_contract.md`
|
||||
- `docs/schemas/ops_stack_diagnosis_contract.md`
|
||||
|
||||
### 3. 如果要回溯历史需求与问题
|
||||
|
||||
@@ -89,6 +147,11 @@
|
||||
- `domain-api/deploy/systemd/domain-worker.service`
|
||||
- `domain-api/deploy/linux/README.md`
|
||||
- `domain-api/deploy/multi-region/README.md`
|
||||
- `domain-api/deploy/multi-region/drive_ops_center.sh`
|
||||
- `domain-api/deploy/multi-region/drive_release_hub.sh`
|
||||
- `domain-api/deploy/multi-region/drive_ops_action.sh`
|
||||
- `domain-api/deploy/multi-region/check_ops_contracts.sh`
|
||||
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
|
||||
- `domain-api/deploy/multi-region/bootstrap_overseas.sh`
|
||||
- `domain-api/deploy/multi-region/bootstrap_mainland.sh`
|
||||
- `domain-api/deploy/linux/smoke_test.py`
|
||||
@@ -109,6 +172,8 @@
|
||||
- 配置迁移与备份
|
||||
- 打包与验包
|
||||
- 最终发布准备
|
||||
- 海外单脑控制面协议收口
|
||||
- Ops Center / Codex / CLI 驾驶 contract 收口
|
||||
|
||||
剩余工作只在真实 Linux 环境:
|
||||
|
||||
|
||||
@@ -31,6 +31,12 @@
|
||||
- 海外控制面 `runtime/sync-summary`
|
||||
- 能看到 `detect_result_batches`
|
||||
|
||||
如果当前已经切到“海外单脑控制面统一驾驶”模式,建议同时配套阅读:
|
||||
|
||||
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
|
||||
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
|
||||
|
||||
## 二、上线前必须确认的结论
|
||||
|
||||
上线前至少要确认下面这些结论同时成立:
|
||||
@@ -46,6 +52,34 @@
|
||||
|
||||
## 三、正式上线前执行顺序
|
||||
|
||||
### 0. 先跑一遍统一上线总检
|
||||
|
||||
如果当前已经进入“海外单脑控制面 + 多地域 / Node Agent”这条正式链路,建议先跑:
|
||||
|
||||
```bash
|
||||
cd /www/wwwroot/getDomain/domain-api
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://152.53.37.118:8100 summary
|
||||
```
|
||||
|
||||
期望:
|
||||
|
||||
- `go_live_status` 为 `ready` 或至少没有新的 `blocking_reasons`
|
||||
- `stack_status` 不为 `blocked`
|
||||
- `route_surface_complete=true`
|
||||
- `launchpad_status` 不为 `blocked`
|
||||
- 如果当前已经纳管执行节点,则 `remote_access_ready` 不应长期为 `0`
|
||||
- `next_step_action_code` / `operator_title` 应能直接说明当前第一优先动作
|
||||
- 如果 `log_sync_enabled=true`,则要额外看 `log_sync_state`、`log_sync_covered_nodes`
|
||||
|
||||
如果这里已经出现 `bootstrap_run / run_acceptance`,说明当前上线前仍有节点接管缺口,应先收口首个缺口节点,再继续下面的业务核验。
|
||||
|
||||
建议这样理解总检输出:
|
||||
|
||||
- `blocking_reasons` 只放真正会阻断上线的项,例如 `remote_agent_ready=0`
|
||||
- `warnings` 表示还能继续联调,但现在还不适合直接宣称“全部收口”
|
||||
- `recommended_commands.next_step` 就是当前最值得优先执行的一条命令
|
||||
- `recommended_commands.log_sync_logs` / `recommended_commands.log_sync_inspection` 用来补远端参与节点的日志观测
|
||||
|
||||
### 1. 核对服务状态
|
||||
|
||||
```bash
|
||||
|
||||
@@ -165,10 +165,20 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||
- `detect_worker_nodes`
|
||||
- 看谁在线
|
||||
- 看谁是 `worker_online=true`
|
||||
- 看谁 `detect_participating=true`
|
||||
- `detect_participating=true` 只能说明它曾被判断为“正在参与”
|
||||
- `detect_job_items`
|
||||
- 看任务到底被哪台节点 `claimed_by`
|
||||
- 这才是“当前真正干活的节点”
|
||||
- `runtime/status`
|
||||
- 看 `participation_summary.dispatch_active_nodes`
|
||||
- 这是“当前真正执行/领任务的节点”
|
||||
- 看 `participation_summary.non_participating_nodes`
|
||||
- 这是“在线但未参与的节点”
|
||||
- 看 `non_participating_nodes[].participation_state`
|
||||
- `standby` 表示在线待命
|
||||
- `load_syncing` 表示负载待确认,不要直接算成“正在干活”
|
||||
- 看 `log_sync`
|
||||
- 能直接确认远端日志回传是否开启、是关键还是全量、样本来自哪些节点
|
||||
|
||||
## 8. 常见误判
|
||||
|
||||
@@ -182,6 +192,7 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||
|
||||
- “有效执行节点”是可承担任务的在线节点数
|
||||
- “当前参与检测节点”是当前真的在领任务、跑任务的节点
|
||||
- “在线但未参与节点”是已经在线、可承接任务,但当前这轮还没分到任务的节点
|
||||
|
||||
如果 controller 没兼跑检测,通常只会看到独立 worker 在真正执行。
|
||||
|
||||
@@ -189,6 +200,26 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||
|
||||
这不一定是 bug,可能只是当前调度没有分到它。应以 `detect_job_items.claimed_by` 为准,不要只看服务在线。
|
||||
|
||||
### 8.4 页面显示“负载待确认”
|
||||
|
||||
这不是新的故障状态,而是为了避免误判:
|
||||
|
||||
- 节点已经上报 `busy` 或有 `current_load`
|
||||
- 但当前还没看到明确的 `items_claimed / items_running / processed_recent`
|
||||
|
||||
通常是心跳和任务快照还没完全对齐。先等下一轮刷新,再结合 `claimed_by` 和 `participation_summary` 判断,不要立刻当作“这台机器已经在跑检测”。
|
||||
|
||||
### 8.5 日志控制台看不到大陆节点过程
|
||||
|
||||
先看 `runtime/status` 或页面里的“远端日志回传”:
|
||||
|
||||
- 若 `enabled=false`
|
||||
- 说明本来就没开回传
|
||||
- 若 `enabled=true` 但 `line_count=0`
|
||||
- 说明开关已开,但当前还没有远端样本
|
||||
- 若 `source_nodes` 里没有目标大陆节点
|
||||
- 说明该节点这轮还没回传日志,先看它是否真的在参与检测
|
||||
|
||||
## 9. 联调完成后的回滚
|
||||
|
||||
如果你后续要切回正式海外机器,在大陆 controller 上把目标地址改回正式值即可:
|
||||
|
||||
708
docs/21_domainCheck_海外主机集中运维与自动化方案.md
Normal file
708
docs/21_domainCheck_海外主机集中运维与自动化方案.md
Normal file
@@ -0,0 +1,708 @@
|
||||
# 21 domainCheck 海外主机集中运维与自动化方案
|
||||
|
||||
如果当前你不是在做“为什么要这么设计”的方案评审,而是已经进入真实收口、准备上线阶段,建议先跳转:
|
||||
|
||||
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
|
||||
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
|
||||
|
||||
## 一、为什么要升级成“集中运维”
|
||||
|
||||
这段时间的多机联调已经证明一个事实:
|
||||
|
||||
- 现在这套“人在海外主机上看后台,再去大陆机器上手工跑命令、复制日志、判断状态”的方式,能联调,但效率很低
|
||||
- 一旦节点数从 `2` 台增加到 `3` 台、`5` 台、`10` 台,复杂度会快速失控
|
||||
- 国内机器上又不能稳定部署 Codex,因此不能指望每台机器都具备“本机智能调试能力”
|
||||
- 检测日志很大,靠人工复制日志片段,不适合做持续诊断
|
||||
|
||||
所以后续不应该继续堆更多“检查脚本”,而应该把整个体系升级成:
|
||||
|
||||
> 海外主机作为唯一运维控制面,统一接管大陆机器的安装、更新、重启、巡检、日志回流和故障诊断。
|
||||
|
||||
一句话目标:
|
||||
|
||||
> 只在海外主机操作,大陆机器尽量不需要人工登录。
|
||||
|
||||
---
|
||||
|
||||
## 二、最终目标形态
|
||||
|
||||
### 1. 海外主机承担什么
|
||||
|
||||
海外主机作为唯一控制面,承担:
|
||||
|
||||
- Web 后台
|
||||
- API 控制面
|
||||
- 运维任务中心
|
||||
- 发布包仓库
|
||||
- 节点注册与权限中心
|
||||
- 日志汇聚与诊断入口
|
||||
- 远程命令编排
|
||||
- 巡检结果展示
|
||||
|
||||
也就是说,后续所有动作都从海外主机发起:
|
||||
|
||||
- 新机器纳管
|
||||
- 初始化安装
|
||||
- 发布更新
|
||||
- 配置下发
|
||||
- 服务重启
|
||||
- 健康检查
|
||||
- 采集诊断包
|
||||
- 远端日志查看
|
||||
- 故障一键排查
|
||||
|
||||
### 2. 大陆机器承担什么
|
||||
|
||||
大陆机器不再承担复杂控制逻辑,只承担:
|
||||
|
||||
- `domaincheck worker/controller` 本体服务
|
||||
- 一个轻量 Node Agent
|
||||
- 本机 systemd / 日志 / 版本 / 健康信息暴露
|
||||
- 接收控制面任务并执行
|
||||
- 将执行结果、日志、诊断包回传到海外主机
|
||||
|
||||
这意味着大陆机器以后是“被管理对象”,而不是“人工登录操作对象”。
|
||||
|
||||
---
|
||||
|
||||
## 三、推荐方案
|
||||
|
||||
## 方案 A:海外控制面 + 大陆 Node Agent
|
||||
|
||||
这是我建议的主方案,也是长期最优方案。
|
||||
|
||||
### 核心思想
|
||||
|
||||
不要把“SSH 到每台机器执行命令”作为主链路,而是让每台大陆机器常驻一个 Agent:
|
||||
|
||||
- Agent 主动连海外控制面
|
||||
- Agent 拉取待执行任务
|
||||
- 本地执行 systemd / shell / 发布 / 巡检
|
||||
- Agent 把 stdout / stderr / 状态 / 日志游标回传
|
||||
|
||||
这样做的好处是:
|
||||
|
||||
- 不要求海外机能直接入站打通到大陆机
|
||||
- 不要求每次人工 SSH
|
||||
- 不怕 SSH 权限、跳板机、端口变化导致整套流程断掉
|
||||
- 天然适合 NAT、弱网络、多机扩容
|
||||
|
||||
### 交互方式
|
||||
|
||||
建议 Agent 使用下面其中一种方式主动连海外控制面:
|
||||
|
||||
#### 首选:HTTPS 长轮询
|
||||
|
||||
- `agent -> overseas-api`
|
||||
- 周期拉取任务
|
||||
- 周期上报心跳、版本、服务状态、日志摘要
|
||||
|
||||
优点:
|
||||
|
||||
- 实现最简单
|
||||
- 最容易兼容现有 Python / FastAPI 架构
|
||||
- 容易先落 MVP
|
||||
|
||||
#### 可升级:WebSocket 常连
|
||||
|
||||
- 建立长连接
|
||||
- 海外控制面可实时下发任务
|
||||
- Agent 实时回传执行日志
|
||||
|
||||
优点:
|
||||
|
||||
- 更实时
|
||||
- 日志流式体验更好
|
||||
|
||||
缺点:
|
||||
|
||||
- 第一版复杂度更高
|
||||
|
||||
### 为什么 Agent 比 SSH 更优
|
||||
|
||||
SSH 适合作为:
|
||||
|
||||
- 首次 bootstrap
|
||||
- 临时人工兜底
|
||||
- 非常规应急
|
||||
|
||||
但不适合作为日常主运维链路,因为:
|
||||
|
||||
- 节点一多,权限和连通性管理会变得脆弱
|
||||
- 脚本回显、超时、日志采集很难标准化
|
||||
- 人工 SSH 本质上还是“远程手工运维”
|
||||
|
||||
所以最佳做法是:
|
||||
|
||||
- SSH 只用于首次纳管
|
||||
- 日常统一走 Agent
|
||||
|
||||
## 三点五、当前已经落地的第一版操作闭环
|
||||
|
||||
这套方案现在已经不只是设计稿,仓库里已经有一批可以直接使用的统一入口:
|
||||
|
||||
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
|
||||
- `domain-api/deploy/multi-region/drive_ops_center.sh`
|
||||
- `domain-api/deploy/multi-region/drive_release_hub.sh`
|
||||
- `domain-api/deploy/multi-region/drive_ops_action.sh`
|
||||
- `domain-api/deploy/multi-region/build_node_agent_bootstrap_plan.sh`
|
||||
|
||||
推荐从海外控制面按这个顺序使用:
|
||||
|
||||
### 1. 初始化控制面配置
|
||||
|
||||
```bash
|
||||
cd /opt/domaincheck/domain-api
|
||||
bash domain-api/deploy/multi-region/init_ops_center_config.sh \
|
||||
/etc/default/domaincheck-ops-center \
|
||||
http://121.204.244.188:8100 \
|
||||
http://152.53.37.118:8100 \
|
||||
https://api.example.com
|
||||
```
|
||||
|
||||
### 2. 查看当前统一配置
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh config
|
||||
```
|
||||
|
||||
### 3. 在控制面本机生成发布包
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-package
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-preview
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-preview-smart worker
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-show
|
||||
```
|
||||
|
||||
### 4. 导出一份联调/交接报告
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export
|
||||
```
|
||||
|
||||
### 5. 为新节点导出接管计划
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-plan-export \
|
||||
/opt/domaincheck/domain-api/runtime/ops-center-reports/agent-plans \
|
||||
http://127.0.0.1:8100 \
|
||||
mainland-worker-02 \
|
||||
mainland \
|
||||
worker \
|
||||
https://api.example.com \
|
||||
/opt/domaincheck
|
||||
```
|
||||
|
||||
这五步对应的意义分别是:
|
||||
|
||||
- 先统一控制面默认地址和身份
|
||||
- 再确认驾驶舱当前在看哪个环境
|
||||
- 再把发布物、发布前预检和发版驾驶舱都收回统一入口,不再额外记根目录脚本
|
||||
- 再把现场收敛成可以回看、可以交接的报告
|
||||
- 最后为新增节点生成标准接管方案
|
||||
|
||||
这样后面无论是海外 Codex 自动驾驶、后台按钮,还是人工运维,都不再需要先去大陆机器手工拼命令。
|
||||
|
||||
## 三点六、现在发布驾驶舱已经进入统一数据口径
|
||||
|
||||
目前发布侧也已经不是零散接口拼出来的状态,而是收敛成了一套统一判断:
|
||||
|
||||
- `GET /api/v1/ops/releases/launchpad`
|
||||
- `ops overview -> release_hub.launchpad`
|
||||
- `check_release_hub.sh`
|
||||
- `drive_release_hub.sh launchpad`
|
||||
- 运维中枢页面里的“发布驾驶舱”
|
||||
|
||||
这几处现在复用的是同一套结论,都会统一告诉你:
|
||||
|
||||
- 当前是 `ready / attention / blocked`
|
||||
- 最新发布包是否可用
|
||||
- 最新 Release 是否已经建立
|
||||
- Worker 智能灰度是否可发
|
||||
- Control 发布是否可发
|
||||
- 下一步推荐动作是什么
|
||||
|
||||
同时,`GET /api/v1/ops/runbook` 里的标准作业路径 `release_progression` 也已经切到同一套发布驾驶舱判断。
|
||||
也就是说,海外控制面看到的“标准作业路径”和“发布驾驶舱”不会再出现一套说能发、一套说先补接管的分裂口径。
|
||||
|
||||
现在又往前推进了一步:
|
||||
|
||||
- `POST /api/v1/ops/runbook/sequences/{sequence_key}/execute`
|
||||
|
||||
后台上的标准作业路径按钮,后续都应该优先走这个入口,再由后端统一分发到 driver action / playbook / rollout 预案,而不是前端自己维护动作分支。
|
||||
|
||||
这样后续海外 Codex、后台按钮和命令行脚本拿到的就不再是三套不同口径,而是同一套“发布驾驶判断”。
|
||||
|
||||
## 三点七、现在回执队列也已经进入统一治理入口
|
||||
|
||||
除了发布驾驶舱,Node Agent 回执队列这一层现在也已经不再是“只能看、不能管”的状态。
|
||||
|
||||
当前已经统一收口成三类入口:
|
||||
|
||||
- 页面:
|
||||
- OpsCenter 托管节点表可以直接打开“回执队列治理”抽屉
|
||||
- 驾驶建议:
|
||||
- `dead_letter / retrying` 会优先落到标准动作模板
|
||||
- 海外单入口 CLI:
|
||||
- `drive_ops_center.sh queue-status`
|
||||
- `drive_ops_center.sh queue-records`
|
||||
- `drive_ops_center.sh queue-flush`
|
||||
- `drive_ops_center.sh queue-replay`
|
||||
- `drive_ops_center.sh queue-replay-record`
|
||||
- `drive_ops_center.sh queue-discard-record`
|
||||
|
||||
这意味着海外主机已经可以统一处理:
|
||||
|
||||
- 哪台节点存在积压 / 死信
|
||||
- 头部记录是什么
|
||||
- 什么时候应该立即冲刷
|
||||
- 什么时候应该重放
|
||||
- 什么时候应该说明原因后丢弃
|
||||
|
||||
当前阶段仍然明确限制为:
|
||||
|
||||
- `record_visibility=head_only`
|
||||
- 所有治理动作都落成正式 `ops job`
|
||||
- 不直接 SSH 到节点改队列文件
|
||||
|
||||
这样以后即使把远端全量死信记录正式投影到控制面,也只是“扩可见范围”,而不是重新推翻当前模型。
|
||||
|
||||
---
|
||||
|
||||
## 四、这套方案要解决的具体问题
|
||||
|
||||
## 1. 安装部署自动化
|
||||
|
||||
目标:
|
||||
|
||||
- 海外控制面点一次“新增节点”
|
||||
- 填一个节点模板
|
||||
- 大陆机器自动初始化
|
||||
|
||||
建议流程:
|
||||
|
||||
1. 海外后台新增节点
|
||||
2. 生成 `bootstrap token`
|
||||
3. 大陆机器只执行一次极短安装命令
|
||||
4. Node Agent 注册到海外控制面
|
||||
5. 控制面向该节点下发:
|
||||
- 拉取发布包
|
||||
- 解压
|
||||
- 生成 `/etc/default/...`
|
||||
- 安装 systemd
|
||||
- 启动服务
|
||||
6. 回传安装结果
|
||||
|
||||
这样后续加机器时,不需要再重复人工对照文档逐条敲。
|
||||
|
||||
## 2. 更新发布自动化
|
||||
|
||||
目标:
|
||||
|
||||
- 海外主机统一推版本
|
||||
- 大陆节点自动拉包、灰度更新、失败回滚
|
||||
|
||||
这里强烈建议:
|
||||
|
||||
> 后续不要让大陆机器自己跑 git 作为正式更新链路。
|
||||
|
||||
而是改成:
|
||||
|
||||
- 海外控制面构建发布包
|
||||
- 版本号固定
|
||||
- Agent 下载指定 release 包
|
||||
- 校验 checksum
|
||||
- 切换当前版本软链
|
||||
- 重启服务
|
||||
- 回传成功/失败
|
||||
|
||||
这样比远程 `git pull` 更稳,因为:
|
||||
|
||||
- 不依赖每台机器 git 权限
|
||||
- 不怕工作区脏文件
|
||||
- 不怕 root/www 用户混用
|
||||
- 可回滚
|
||||
|
||||
推荐目录形态:
|
||||
|
||||
```text
|
||||
/opt/domaincheck/releases/<version>/
|
||||
/opt/domaincheck/current -> /opt/domaincheck/releases/<version>/
|
||||
```
|
||||
|
||||
更新时:
|
||||
|
||||
- 下载新版本到 `releases`
|
||||
- 校验
|
||||
- 切换 `current`
|
||||
- `systemctl restart ...`
|
||||
|
||||
失败就回滚软链。
|
||||
|
||||
## 3. 远程命令执行自动化
|
||||
|
||||
目标:
|
||||
|
||||
- 海外控制面能发“结构化任务”
|
||||
- 不再让人手工复制命令
|
||||
|
||||
建议任务类型:
|
||||
|
||||
- `service.start`
|
||||
- `service.stop`
|
||||
- `service.restart`
|
||||
- `service.status`
|
||||
- `deploy.release`
|
||||
- `config.render`
|
||||
- `diagnostics.collect`
|
||||
- `logs.tail`
|
||||
- `script.run`
|
||||
- `health.check`
|
||||
|
||||
每个任务统一回传:
|
||||
|
||||
- 任务 ID
|
||||
- 节点编码
|
||||
- 开始时间 / 结束时间
|
||||
- exit code
|
||||
- stdout
|
||||
- stderr
|
||||
- 结构化结果 JSON
|
||||
|
||||
这样后面后台才能真正做“操作中心”。
|
||||
|
||||
## 4. 日志集中回流
|
||||
|
||||
目标:
|
||||
|
||||
- 海外后台就能看到大陆机器日志
|
||||
- 不再人工抄 `journalctl`
|
||||
|
||||
### 日志回流建议分三层
|
||||
|
||||
#### A. 关键事件流
|
||||
|
||||
只回传关键事件:
|
||||
|
||||
- 服务启动
|
||||
- 服务重启
|
||||
- 任务开始
|
||||
- 任务完成
|
||||
- 代理刷新失败
|
||||
- Redis/DB 连接异常
|
||||
- 第三方站点异常
|
||||
|
||||
适合默认长期开启。
|
||||
|
||||
#### B. 诊断模式日志流
|
||||
|
||||
像你现在提的“关闭 / 关键 / 全量回传”一样:
|
||||
|
||||
- `off`
|
||||
- `key`
|
||||
- `full`
|
||||
|
||||
这是正确方向,应该继续保留。
|
||||
|
||||
#### C. 诊断包
|
||||
|
||||
当出现疑难问题时,一键打包:
|
||||
|
||||
- 最近 N 分钟 `journalctl`
|
||||
- `runtime/status`
|
||||
- `runtime/cluster`
|
||||
- `detect_worker.log tail`
|
||||
- `/etc/default/*`
|
||||
- 当前版本号
|
||||
- 本机健康检查输出
|
||||
|
||||
然后上传海外主机。
|
||||
|
||||
这比全量实时传所有日志更经济,也更适合定位问题。
|
||||
|
||||
## 5. 健康巡检自动化
|
||||
|
||||
目标:
|
||||
|
||||
- 每台大陆机自动自检
|
||||
- 海外后台只看结果
|
||||
|
||||
建议 Agent 每 30 秒到 60 秒上报:
|
||||
|
||||
- 节点在线状态
|
||||
- 当前版本
|
||||
- API / Worker / Sync Agent 运行态
|
||||
- Redis / PostgreSQL / 磁盘 / 内存 / CPU
|
||||
- 最近告警
|
||||
- 当前任务负载
|
||||
- 最近日志时间
|
||||
|
||||
并把现有脚本升级为 Agent 内部检查项:
|
||||
|
||||
- `check_mainland_controller.sh`
|
||||
- `check_mainland_worker.sh`
|
||||
- `check_temp_topology.sh`
|
||||
- `check_worker_participation.sh`
|
||||
|
||||
后续不是人工执行这些脚本,而是 Agent 周期性执行并结构化上传结果。
|
||||
|
||||
---
|
||||
|
||||
## 五、建议的系统分层
|
||||
|
||||
## 1. 海外控制面
|
||||
|
||||
建议新增一个“运维控制模块”,逻辑上可放进 `domain-api`,后续再拆独立服务也可以。
|
||||
|
||||
### 需要的核心对象
|
||||
|
||||
#### `managed_nodes`
|
||||
|
||||
记录纳管节点:
|
||||
|
||||
- `node_code`
|
||||
- `region`
|
||||
- `role`
|
||||
- `hostname`
|
||||
- `ip`
|
||||
- `agent_version`
|
||||
- `current_release`
|
||||
- `status`
|
||||
- `ssh_enabled`
|
||||
- `agent_last_seen_at`
|
||||
- `tags`
|
||||
|
||||
#### `ops_jobs`
|
||||
|
||||
记录运维任务:
|
||||
|
||||
建议以后直接以 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 为正式母本。
|
||||
|
||||
- `job_id`
|
||||
- `job_type`
|
||||
- `target_nodes`
|
||||
- `payload`
|
||||
- `created_by`
|
||||
- `status`
|
||||
- `started_at`
|
||||
- `finished_at`
|
||||
|
||||
#### `ops_job_steps`
|
||||
|
||||
记录每个节点执行步骤:
|
||||
|
||||
- `node_code`
|
||||
- `step_name`
|
||||
- `status`
|
||||
- `stdout`
|
||||
- `stderr`
|
||||
- `result_json`
|
||||
|
||||
#### `node_log_streams`
|
||||
|
||||
记录日志游标和日志回流状态:
|
||||
|
||||
- `node_code`
|
||||
- `source`
|
||||
- `mode`
|
||||
- `cursor`
|
||||
- `last_received_at`
|
||||
|
||||
## 2. Node Agent
|
||||
|
||||
建议单独做一个轻量 Python 服务,比如:
|
||||
|
||||
```text
|
||||
domaincheck-node-agent
|
||||
```
|
||||
|
||||
职责:
|
||||
|
||||
- 周期心跳
|
||||
- 拉取任务
|
||||
- 本地执行
|
||||
- 采集日志
|
||||
- 上报结果
|
||||
- 生成诊断包
|
||||
|
||||
Agent 尽量独立于业务主程序,不要把它耦合进 `detect_worker.py`。
|
||||
|
||||
原因:
|
||||
|
||||
- 运维系统不能依赖业务进程是否正常
|
||||
- 即使 Worker 崩了,Agent 还应该活着,才能帮你排障
|
||||
|
||||
---
|
||||
|
||||
## 六、对你当前项目最合适的落地路线
|
||||
|
||||
## 第一阶段:先做“控制面集中化”
|
||||
|
||||
目标:
|
||||
|
||||
- 海外后台统一看到所有大陆机器
|
||||
- 一键拉诊断
|
||||
- 一键执行已有检查脚本
|
||||
- 一键切日志回传模式
|
||||
|
||||
这个阶段先不碰太多发布链路,优先把“看”和“查”统一。
|
||||
|
||||
### 交付物
|
||||
|
||||
- Node Agent MVP
|
||||
- 海外后台“节点管理”页
|
||||
- 海外后台“运维任务”页
|
||||
- 海外后台“远端日志”页
|
||||
- 海外后台“诊断包”页
|
||||
|
||||
## 第二阶段:再做“一键更新”
|
||||
|
||||
目标:
|
||||
|
||||
- 海外控制面选择版本
|
||||
- 指定节点灰度发布
|
||||
- 自动回滚
|
||||
|
||||
### 交付物
|
||||
|
||||
- 发布包生成器
|
||||
- `release manifest`
|
||||
- Agent 下载 / 校验 / 切换版本
|
||||
- 回滚机制
|
||||
|
||||
## 第三阶段:再做“一键安装新节点”
|
||||
|
||||
目标:
|
||||
|
||||
- 新机器只执行一次 bootstrap
|
||||
- 其余全部由海外控制面接管
|
||||
|
||||
### 交付物
|
||||
|
||||
- bootstrap token
|
||||
- 安装向导
|
||||
- 节点注册流程
|
||||
- 环境模板生成器
|
||||
|
||||
---
|
||||
|
||||
## 七、比“SSH 全接管”更优的地方
|
||||
|
||||
你提的方向是:
|
||||
|
||||
> 海外主机配置好 SSH,后续所有大陆机器都由代码内部接管
|
||||
|
||||
这个方向本身是对的,但如果只做“SSH 全接管”,还不够优秀。
|
||||
|
||||
更优解是:
|
||||
|
||||
> SSH 只作为纳管和兜底方式;日常统一走 Agent 主动连接。
|
||||
|
||||
这样好处是:
|
||||
|
||||
- 架构更稳
|
||||
- 不依赖每次远程入站打通
|
||||
- 不怕某些网络环境 SSH 不稳定
|
||||
- 更适合未来节点继续增加
|
||||
- 更容易做权限分级、审计、结果留痕
|
||||
|
||||
所以我建议最终定成:
|
||||
|
||||
### 运维主链路
|
||||
|
||||
- `海外控制面 -> Agent 任务编排 -> 大陆节点执行 -> 回传结果`
|
||||
|
||||
### 运维兜底链路
|
||||
|
||||
- `海外控制面 -> SSH -> 大陆节点`
|
||||
|
||||
---
|
||||
|
||||
## 八、和现有代码如何衔接
|
||||
|
||||
当前项目已经有一些很好的基础,不需要推翻:
|
||||
|
||||
- 多机节点模型
|
||||
- `runtime/status`
|
||||
- `runtime/cluster`
|
||||
- `sync-summary`
|
||||
- 一批巡检脚本
|
||||
- 日志回传开关雏形
|
||||
- 运行中心
|
||||
|
||||
这些都应该保留,并演进成:
|
||||
|
||||
### 当前脚本的未来定位
|
||||
|
||||
- `bootstrap_*.sh`
|
||||
- 保留为 bootstrap / 应急工具
|
||||
- `check_*.sh`
|
||||
- 演进成 Agent 内部的标准诊断动作
|
||||
- `runtime/status`
|
||||
- 继续作为统一运行态接口
|
||||
- `runtime/cluster`
|
||||
- 继续作为统一集群视图
|
||||
- `detect_result_projection / runtime_projection`
|
||||
- 继续作为跨地域观测基础
|
||||
|
||||
也就是说,现有代码不是废掉,而是从“人工脚本时代”升级成“控制面编排时代”。
|
||||
|
||||
---
|
||||
|
||||
## 九、我建议的最优落地决策
|
||||
|
||||
如果只选一个方向,我建议直接定成:
|
||||
|
||||
### 最优方案
|
||||
|
||||
- 海外主机为唯一控制面
|
||||
- 大陆节点统一部署 `domaincheck-node-agent`
|
||||
- Agent 主动访问海外控制面
|
||||
- 日常安装、更新、巡检、日志回流全部走 Agent
|
||||
- SSH 仅保留为 bootstrap 和应急手段
|
||||
- 正式更新统一改为 release 包分发,不再把 `git pull` 作为正式更新链路
|
||||
|
||||
这是当前最值得投入的方向,因为它能同时解决:
|
||||
|
||||
- 调试效率低
|
||||
- 多机管理复杂
|
||||
- 权限混乱
|
||||
- 日志分散
|
||||
- 更新不稳定
|
||||
- 新增节点成本高
|
||||
|
||||
---
|
||||
|
||||
## 十、建议的下一步执行顺序
|
||||
|
||||
不要再继续先补散点 bug,而是先把运维骨架建起来。
|
||||
|
||||
建议顺序:
|
||||
|
||||
1. 固化这份方案
|
||||
2. 新建 `node-agent` 设计草案
|
||||
3. 先做 Agent MVP
|
||||
4. 先接入:
|
||||
- 心跳
|
||||
- 远程任务
|
||||
- 诊断包
|
||||
- 日志 tail
|
||||
5. 海外后台新增:
|
||||
- 节点管理页
|
||||
- 运维任务页
|
||||
- 远端日志页
|
||||
6. 再把现有检查脚本封成 Agent 动作
|
||||
7. 最后再做发布链路自动化
|
||||
|
||||
---
|
||||
|
||||
## 十一、结论
|
||||
|
||||
当前最优策略不是“继续加脚本”,而是:
|
||||
|
||||
> 把 domainCheck 从“多机联调项目”升级成“海外控制面统一接管大陆节点的自动化运维系统”。
|
||||
|
||||
从长期看,这会比继续靠人工 SSH、人工复制日志、人工比对状态高效很多,也更符合你后续继续扩机器的目标。
|
||||
1052
docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md
Normal file
1052
docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md
Normal file
File diff suppressed because it is too large
Load Diff
1163
docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md
Normal file
1163
docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md
Normal file
File diff suppressed because it is too large
Load Diff
4421
docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md
Normal file
4421
docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md
Normal file
File diff suppressed because it is too large
Load Diff
430
docs/25_domainCheck_海外单脑控制面上线收口总表.md
Normal file
430
docs/25_domainCheck_海外单脑控制面上线收口总表.md
Normal file
@@ -0,0 +1,430 @@
|
||||
# 25 domainCheck 海外单脑控制面上线收口总表
|
||||
|
||||
## 一、这份文档解决什么问题
|
||||
|
||||
这份文档不是再讲设计,而是把当前项目进入正式上线前,真正需要收口的事项压成一张总表。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 海外主机已经作为单脑控制面
|
||||
- 大陆节点已经开始纳管
|
||||
- Ops Center / Codex Driver / CLI 都已经接入同一套 `ops` contract
|
||||
- 目标从“能联调”切换到“能稳定上线、能持续运维”
|
||||
|
||||
这份文档优先回答 4 个问题:
|
||||
|
||||
1. 现在能不能继续收口
|
||||
2. 现在能不能正式发布
|
||||
3. 还有哪些阻断项没清
|
||||
4. 下一步到底先跑哪条命令
|
||||
|
||||
---
|
||||
|
||||
## 二、上线前统一原则
|
||||
|
||||
上线前必须统一成下面这条工作方式:
|
||||
|
||||
- 海外主机作为唯一主驾驶席
|
||||
- 页面、CLI、Codex 共用同一套后端判断
|
||||
- 默认先看 `go-live-summary`
|
||||
- 真要解释原因时再下钻 `stack-diagnosis`
|
||||
- 真要执行动作时走 `driver-resolve / execute-resolved`
|
||||
- 真要发布或放量时走 `Release Hub / rollout`
|
||||
|
||||
不再推荐:
|
||||
|
||||
- 人工分别登录多台大陆机器拼状态
|
||||
- 每次上线都从 `journalctl + systemctl + curl` 临时组合判断
|
||||
- 前端、CLI、Codex 各自维护不同的默认下一步
|
||||
|
||||
---
|
||||
|
||||
## 三、正式上线门禁
|
||||
|
||||
### 1. 收口门禁
|
||||
|
||||
至少同时满足:
|
||||
|
||||
- `go_live_summary.go_live_status != blocked`
|
||||
- `stack_diagnosis.diagnosis.stack_status != blocked`
|
||||
- `route_surface_complete=true`
|
||||
- `driver-feed.automation_coverage.launch_status != blocked`
|
||||
- 首屏默认下一步已经明确,不再是模糊人工判断
|
||||
|
||||
### 2. 发布门禁
|
||||
|
||||
至少同时满足:
|
||||
|
||||
- `publish_ready=true`
|
||||
- `launchpad_status != blocked`
|
||||
- 不存在未处理的关键 `blocking_reasons`
|
||||
- 当前发布主车道已经清晰落在:
|
||||
- `review_smart_rollout_preview`
|
||||
- `review_control_rollout`
|
||||
- `publish_latest_worker`
|
||||
- `create_release_rollout_*`
|
||||
|
||||
### 3. 运维门禁
|
||||
|
||||
至少同时满足:
|
||||
|
||||
- 海外控制面 API 稳定在线
|
||||
- 大陆 controller / worker 心跳稳定
|
||||
- 节点接管缺口已经收口,或至少首个缺口节点已有明确恢复动作
|
||||
- 远端日志回传可按需开启、复核、关闭
|
||||
- 运行中心首屏能直接区分:
|
||||
- 在线但未参与
|
||||
- 正在执行 / 正在领任务
|
||||
|
||||
---
|
||||
|
||||
## 四、真正的起手顺序
|
||||
|
||||
每次准备上线、复核、值班接手时,都按这个顺序来:
|
||||
|
||||
### 第 1 步:看上线收口摘要
|
||||
|
||||
```bash
|
||||
cd /www/wwwroot/getDomain
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
|
||||
```
|
||||
|
||||
如果当前是临时海外控制面联调,也可以把第二个地址替换成海外目标 API。
|
||||
|
||||
第一眼重点只看:
|
||||
|
||||
- `go_live_status`
|
||||
- `publish_status`
|
||||
- `blocking_reasons`
|
||||
- `warnings`
|
||||
- `next_step_action_code`
|
||||
- `operator_title`
|
||||
|
||||
如果你怀疑当前是环境本身有漂移,而不是业务链路没收口,先执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh env-audit
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `status`
|
||||
- `headline`
|
||||
- `missing_items`
|
||||
- `tooling_items`
|
||||
- `runtime.preflight_ok`
|
||||
- `runtime.readiness_status`
|
||||
- `runtime.route_surface_complete`
|
||||
- `runtime.repo_capability_drift`
|
||||
- `runtime.runtime_may_need_restart`
|
||||
- `recommended_actions`
|
||||
|
||||
如果这里出现:
|
||||
|
||||
- `runtime.repo_capability_drift=true`
|
||||
- `runtime.runtime_may_need_restart=true`
|
||||
|
||||
则优先执行 `runtime-refresh-recover`,不要先把问题误判成“代码还没写完”。这通常表示:
|
||||
|
||||
- 仓库代码已经更新
|
||||
- 但运行中的 `domaincheck-api` 进程还没重启到这版代码
|
||||
|
||||
`runtime-refresh-recover` 会固定给出:
|
||||
|
||||
- 当前是否真的属于运行时版本漂移
|
||||
- 推荐先跑的 `runtime-refresh-recover` 统一恢复入口
|
||||
- 重启后应该按什么顺序继续:
|
||||
- `stack-diagnosis`
|
||||
- `node-bootstrap-plan`
|
||||
- `stack-next`
|
||||
- `doctor-decision`
|
||||
|
||||
如果你已经不想再手工一条条执行,而是想把“刷新运行时 + 再做总检复核”压成一次操作,直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
|
||||
```
|
||||
|
||||
它会固定串起:
|
||||
|
||||
1. `runtime-refresh-recover`
|
||||
2. `stack-diagnosis summary`
|
||||
3. `stack-next`
|
||||
4. `go-live-check summary`
|
||||
5. `doctor-decision`
|
||||
|
||||
如果要把这次上线前检查直接导出成一整包证据,而不是手工复制多段终端输出,直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
|
||||
```
|
||||
|
||||
它会统一导出:
|
||||
|
||||
- `env-audit`
|
||||
- `go-live-check` 摘要
|
||||
- `go-live-summary`
|
||||
- `stack-diagnosis`
|
||||
- `driver-feed`
|
||||
- `codex-brief`
|
||||
- `release-launchpad`
|
||||
- `doctor-decision`
|
||||
- `doctor-export`
|
||||
- `manifest.json`
|
||||
|
||||
其中 `manifest.json` 会直接标记:
|
||||
|
||||
- 环境审计当前是 `ready / attention / blocked`
|
||||
- 哪些 artifact 成功
|
||||
- 哪些 artifact 超时
|
||||
- 哪些 endpoint 缺失或返回异常
|
||||
- 当前推荐的阅读顺序
|
||||
|
||||
现在 Ops Center 首屏也会同步读取最新 bundle / manifest:
|
||||
|
||||
- 顶部 `总检决策`
|
||||
- 详情区 `总检主决策详情`
|
||||
- API `GET /api/v1/ops/doctor-decision`
|
||||
- 顶部 `交付证据`
|
||||
- 详情区 `交付证据详情`
|
||||
- API `GET /api/v1/ops/go-live-bundle`
|
||||
- 顶部 `正式复核`
|
||||
- 详情区 `正式复核详情`
|
||||
- API `GET /api/v1/ops/go-live-review`
|
||||
|
||||
如果你想先拿到一句“bundle manifest 正式复核是否通过”的统一结论,而不是直接跳到最终签收,也可以先执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review
|
||||
```
|
||||
|
||||
如果你想在 bundle 基础上直接得到一句“现在能不能签字上线”的统一结论,而不是人工再看多份 JSON,直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff
|
||||
```
|
||||
|
||||
或基于已有 bundle:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff /path/to/go-live-bundle
|
||||
```
|
||||
|
||||
这里的 `go-live-signoff` 已经不是另一套独立摘要,而是固定压缩:
|
||||
|
||||
- `go-live-review`
|
||||
- `doctor-decision`
|
||||
- `go-live-summary / 发布门禁 / launchpad 摘要`
|
||||
|
||||
所以页面首屏、CLI 与海外 Codex 看到的是同一份最终签字口径。
|
||||
|
||||
第一眼重点只看:
|
||||
|
||||
- `signoff_status`
|
||||
- `headline`
|
||||
- `release_gate`
|
||||
- `blocked_reasons`
|
||||
- `attention_reasons`
|
||||
- `decision`
|
||||
|
||||
### 第 2 步:看总检详情
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `stack_status`
|
||||
- `issues`
|
||||
- `next_step`
|
||||
- `quick_commands`
|
||||
|
||||
### 第 3 步:看驾驶主线
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `top_recommendation`
|
||||
- `automation_coverage`
|
||||
- `recommended_behavior`
|
||||
- `focus`
|
||||
|
||||
### 第 4 步:确认默认下一步
|
||||
|
||||
如果只是预览:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-next
|
||||
```
|
||||
|
||||
如果已经确认要执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-next run confirm
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、当前项目的主处理车道
|
||||
|
||||
上线前遇到问题时,不要发散排查,先归到下面 5 条主车道:
|
||||
|
||||
### 1. 节点接管车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- `remote_access_ready=0`
|
||||
- `bootstrap_run`
|
||||
- `run_acceptance`
|
||||
- `managed_nodes_agent_pending`
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-gap-check
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-gap-recover
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-onboarding http://127.0.0.1:8100 mainland-worker-01
|
||||
```
|
||||
|
||||
### 2. 远端日志车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- 首屏显示“现场日志覆盖不足”
|
||||
- `log_sync_enabled=false`
|
||||
- `line_count=0`
|
||||
- `source_nodes` 缺节点
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-check
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-recover
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 key
|
||||
```
|
||||
|
||||
### 3. 总检修复车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- `stack_status=blocked`
|
||||
- `surface_status=broken`
|
||||
- `contracts` / `launchpad` / `activity-stream` 缺口
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100
|
||||
bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100
|
||||
bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100
|
||||
```
|
||||
|
||||
### 4. 检测参与车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- 页面显示在线节点多,但真正跑检测的少
|
||||
- `claimed_by` 分布异常
|
||||
- 在线但未参与节点过多
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh activity-stream
|
||||
```
|
||||
|
||||
### 5. 发布 / 放量车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- `publish_ready=true`
|
||||
- `launchpad_status` 已可进入 Worker / Control rollout
|
||||
- 默认下一步已落在 Release Hub
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-focus-preview
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 六、首屏通过标准
|
||||
|
||||
Ops Center 首屏如果要算“可上线收工”,至少应满足:
|
||||
|
||||
- 首屏能同时显示:
|
||||
- 上线收口状态
|
||||
- 发布闸门状态
|
||||
- 自动化收口状态
|
||||
- 主处理车道
|
||||
- 默认下一步
|
||||
- 后端接管覆盖
|
||||
- 观察层完整度
|
||||
- 现场日志覆盖度
|
||||
- 首屏动作条可直接完成:
|
||||
- 默认下一步
|
||||
- 定位下一步
|
||||
- 看契约
|
||||
- 复制命令
|
||||
- 进入主处理面板
|
||||
- 首屏回执条能统一反馈:
|
||||
- 打开契约成功
|
||||
- 定位成功
|
||||
- 复制成功
|
||||
- 后端动作执行成功 / 警告 / 失败
|
||||
|
||||
如果这些已经稳定,就说明值班同事和海外 Codex 已经在同一个驾驶台上工作,而不是各自再拼逻辑。
|
||||
|
||||
---
|
||||
|
||||
## 七、通过标准后的最小上线动作
|
||||
|
||||
当上面门禁都通过后,建议最小上线动作顺序为:
|
||||
|
||||
1. 跑 `go-live-check`
|
||||
2. 看 `stack-diagnosis`
|
||||
3. 复核 `driver-feed.automation_coverage`
|
||||
4. 复核 `codex-brief.recommended_behavior`
|
||||
5. 若进入发布车道,先看 `release-launchpad`
|
||||
6. 对 `guarded_auto` 动作显式确认后再执行
|
||||
|
||||
上线前最后一轮建议保留证据:
|
||||
|
||||
- `go-live-summary` 输出
|
||||
- `stack-diagnosis` 输出
|
||||
- `driver-feed` 输出
|
||||
- `codex-brief` 输出
|
||||
- `release-launchpad` 输出
|
||||
- 前端构建成功记录
|
||||
|
||||
---
|
||||
|
||||
## 八、现在这套体系是否已经进入可上线状态
|
||||
|
||||
按当前仓库现状,可以认为已经具备:
|
||||
|
||||
- 单脑控制面的协议骨架
|
||||
- 首屏驾驶舱骨架
|
||||
- 总检与驾驶 contract
|
||||
- Release Hub / Driver / Node Agent 的联动骨架
|
||||
- 海外 CLI 单入口
|
||||
|
||||
但正式宣称“可以上线”,仍建议每次以这份总表复核,而不是只凭某一个页面截图或某一次联调成功就直接跳过。
|
||||
|
||||
这份文档的定位就是:
|
||||
|
||||
> 后续每次上线、值班接手、发布前复核,都先回到这里。
|
||||
|
||||
如果你已经准备进入“这次要不要真的发”的最终执行阶段,下一份应直接看:
|
||||
|
||||
- `docs/26_domainCheck_发布前运行验证与交付模板.md`
|
||||
635
docs/26_domainCheck_发布前运行验证与交付模板.md
Normal file
635
docs/26_domainCheck_发布前运行验证与交付模板.md
Normal file
@@ -0,0 +1,635 @@
|
||||
# 26 domainCheck 发布前运行验证与交付模板
|
||||
|
||||
## 一、这份模板怎么用
|
||||
|
||||
这份文档不是设计说明,而是正式发布前最后一轮执行模板。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 海外单脑控制面已经搭好
|
||||
- 大陆节点已经接入或正在收口
|
||||
- 代码已经更新到待发布版本
|
||||
- 现在要做最后一轮运行验证、证据留存、交付确认
|
||||
|
||||
这份模板分成 3 段:
|
||||
|
||||
1. 发布前运行验证
|
||||
2. 上线证据导出
|
||||
3. 最终交付结论模板
|
||||
|
||||
---
|
||||
|
||||
## 二、发布前运行验证
|
||||
|
||||
### 1. 先跑统一收口摘要
|
||||
|
||||
```bash
|
||||
cd /www/wwwroot/getDomain
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `go_live_status`
|
||||
- `publish_status`
|
||||
- `blocking_reasons`
|
||||
- `warnings`
|
||||
- `next_step_action_code`
|
||||
- `operator_title`
|
||||
- `launchpad_recommended_target_node_code`
|
||||
- `launchpad_recommended_recovery_label`
|
||||
- `launchpad_recommended_recovery_summary`
|
||||
- `launchpad_onboarding_bootstrap_pending_nodes`
|
||||
- `launchpad_onboarding_acceptance_ready_nodes`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `go_live_status != blocked`
|
||||
- `publish_status != blocked`
|
||||
|
||||
### 1.5 先看环境差异是否已收口
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh env-audit
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `status`
|
||||
- `headline`
|
||||
- `python.pytest_installed / fastapi_installed / uvicorn_installed`
|
||||
- `python.runtimes`
|
||||
- `services`
|
||||
- `runtime.preflight_ok`
|
||||
- `runtime.readiness_status`
|
||||
- `runtime.route_surface_complete`
|
||||
- `missing_items`
|
||||
- `tooling_items`
|
||||
|
||||
通过标准建议:
|
||||
|
||||
- `status != blocked`
|
||||
- `runtime.preflight_ok=true`
|
||||
- `route_surface_complete=true`
|
||||
- 如果 `missing_items` 里仍有关键基础项,先补环境,再继续业务复核
|
||||
- 如果只有 `tooling_items`,可以继续上线复核,但建议后补本机工具链
|
||||
- 如果 `runtime.runtime_may_need_restart=true`
|
||||
- 不要直接继续接管或发布链
|
||||
- 先按 `runtime-refresh-recover` 给出的固定顺序完成 API 重启与复检
|
||||
- 如果你希望把这一轮“重启 + 总检 + 默认下一步 + 上线摘要”压成一次复核
|
||||
- 直接执行 `go-live-recover`
|
||||
|
||||
### 1.6 先看托管节点接入面是否清楚
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `summary.agent_ready`
|
||||
- `summary.ssh_ready`
|
||||
- `summary.remote_access_ready`
|
||||
- `summary.remote_access_state_counts`
|
||||
- 每台节点的 `agent_state`
|
||||
- 每台节点的 `remote_access_state`
|
||||
- 每台节点的 `ssh_access_label`
|
||||
- 每台节点的 `ssh_entry`
|
||||
|
||||
通过标准建议:
|
||||
|
||||
- 如果某台节点还是 `agent_pending`
|
||||
- 先确认是否已经保存 SSH 入口
|
||||
- 如果节点尚未保存 SSH 入口,但你已经决定由海外控制面统一接管
|
||||
- 先补录:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bind-ssh http://127.0.0.1:8100 mainland-worker-01 121.204.244.248 root 22
|
||||
```
|
||||
|
||||
- 补录后立即复查:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-handover http://127.0.0.1:8100 mainland-worker-01
|
||||
```
|
||||
|
||||
- 如果已经 `agent_ready`
|
||||
- 说明该节点已进入标准远端执行器接管面
|
||||
- 如果还不是 `agent_ready`,但 `ssh_ready` 已经成立
|
||||
- 说明至少已经具备海外主控经 SSH 进入节点完成 bootstrap / 验收的前置条件
|
||||
|
||||
### 2. 再看总检详情
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `stack_status`
|
||||
- `issues`
|
||||
- `next_step`
|
||||
- `quick_commands`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `stack_status != blocked`
|
||||
- `issues` 里没有未处理的关键阻断项
|
||||
|
||||
### 3. 再看驾驶主线
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `top_recommendation`
|
||||
- `automation_coverage`
|
||||
- `recommended_behavior`
|
||||
- `focus`
|
||||
- `summary.launchpad_recommended_target_node_code`
|
||||
- `summary.launchpad_recommended_recovery_label`
|
||||
- `summary.launchpad_onboarding_bootstrap_pending_nodes`
|
||||
- `summary.launchpad_onboarding_acceptance_ready_nodes`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `automation_coverage.launch_status != blocked`
|
||||
- 默认下一步已经明确
|
||||
- 如果 `summary.launchpad_recommended_target_node_code` 非空
|
||||
- 值班同事应能直接知道当前卡在哪台节点
|
||||
- 不需要再回 launchpad 明细手工翻 gap rows
|
||||
|
||||
### 4. 若涉及发布,再看 Release Hub
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `launchpad_status`
|
||||
- `recommended_action_code`
|
||||
- `default_rollout_gate_status`
|
||||
- `recommended_target_node_code`
|
||||
- `recommended_recovery_label`
|
||||
- `recommended_recovery_summary`
|
||||
- `onboarding_bootstrap_pending_nodes`
|
||||
- `onboarding_acceptance_ready_nodes`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `launchpad_status != blocked`
|
||||
- `default_rollout_gate_status != blocked`
|
||||
|
||||
特殊判读:
|
||||
|
||||
- 如果 `recommended_action_code=api-restart`
|
||||
- 不要误判成“节点没接好”
|
||||
- 这更像是运行中的控制面 API 还没刷新到当前仓库的最新运维能力
|
||||
- 应先执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
|
||||
```
|
||||
|
||||
- 然后重新跑:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||||
```
|
||||
|
||||
补充说明:
|
||||
|
||||
- 从 `latest_release.json` 直接创建 Release,或执行“基于最新包的一键智能 Rollout”之前
|
||||
- 现在不仅要求 `smoke_test_ok=true`
|
||||
- 还要求当前 `final_release_report.json` 与最新包完全匹配,且 `final_release_ok=true`
|
||||
- 如果页面按钮被禁用,或 API 返回“最新发布包尚未完成最终签收”
|
||||
- 优先检查 `final_release_report_stale_reason`
|
||||
- 再检查 `final_release_gate_decision / final_release_gate_blocked_reasons`
|
||||
|
||||
### 5. 若涉及节点接管,补跑节点验收
|
||||
|
||||
如果这次发布包含:
|
||||
|
||||
- 新增大陆节点
|
||||
- 重新接管旧节点
|
||||
- 切换到 Node Agent 主接管模式
|
||||
|
||||
则不能只看总检摘要,还要对目标节点逐台完成验收。
|
||||
|
||||
先看验收计划:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-plan http://127.0.0.1:8100 mainland-worker-01
|
||||
```
|
||||
|
||||
确认无误后执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-worker-01 cli/acceptance
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `node_code`
|
||||
- `status`
|
||||
- `checks`
|
||||
- `blocking_issues`
|
||||
- `warnings`
|
||||
- `recommended_next_step`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `status != blocked`
|
||||
- 没有未处理的 `blocking_issues`
|
||||
- 目标节点已经不再停留在“bootstrap pending / acceptance pending”
|
||||
|
||||
### 6. 若涉及远端执行,补跑日志回传验证
|
||||
|
||||
如果当前版本准备进入:
|
||||
|
||||
- 多机值班
|
||||
- 海外单脑统一接管
|
||||
- 远端动作回执与现场日志复核
|
||||
|
||||
则需要确认日志回传链已经可用,而不是只看节点在线。
|
||||
|
||||
先检查:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-check
|
||||
```
|
||||
|
||||
如果存在缺口,再执行恢复:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-recover http://127.0.0.1:8100 key confirm cli/log-sync
|
||||
```
|
||||
|
||||
必要时抓一份节点现场日志:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 full
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `log_sync_enabled`
|
||||
- `line_count`
|
||||
- `source_nodes`
|
||||
- `missing_nodes`
|
||||
- `scene_log.status`
|
||||
|
||||
通过标准:
|
||||
|
||||
- 关键节点已出现在 `source_nodes`
|
||||
- `missing_nodes` 不包含当前发布主车道节点
|
||||
- 如需人工复核现场,`scene-node-log` 能取到有效日志
|
||||
|
||||
### 7. 若涉及 Node Agent 交付,补看队列健康
|
||||
|
||||
如果这轮发布已经进入“控制面发动作、节点拉任务、节点回执结果”的模式,就必须确认 delivery queue 没有假健康。
|
||||
|
||||
先看队列摘要:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-status http://127.0.0.1:8100 mainland-worker-01
|
||||
```
|
||||
|
||||
如果怀疑有堆积或死信,再看明细:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-records http://127.0.0.1:8100 mainland-worker-01 dead_letter
|
||||
```
|
||||
|
||||
如需重放:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-replay http://127.0.0.1:8100 mainland-worker-01 20 cli/queue-replay
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `delivery_queue.summary`
|
||||
- `dead_letter`
|
||||
- `retrying`
|
||||
- `pending`
|
||||
- `replayed_count`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `dead_letter=0`
|
||||
- 没有持续增长的 `retrying`
|
||||
- 队列不处于异常堆积状态
|
||||
|
||||
注意:
|
||||
|
||||
- `dead_letter > 0` 时,不应宣称“可正式发布”
|
||||
- 这种情况至少按 `publish_status = blocked` 处理
|
||||
|
||||
### 8. 最后跑一次 doctor 结论
|
||||
|
||||
前面的检查是分面复核,最后还要再收成一句话,让交班、值班、Codex 驾驶员看到同一结论。
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `status`
|
||||
- `decision`
|
||||
- `recommended_commands`
|
||||
- `recommended_reading_order`
|
||||
- `launchpad_alignment`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `status != blocked`
|
||||
- `decision` 与前面的 `go-live-check / stack-diagnosis / release-launchpad` 不冲突
|
||||
- `recommended_commands` 已经指向明确的下一跳,而不是泛化排查
|
||||
|
||||
### 9. 若希望直接生成“可否签字上线”的最终摘要
|
||||
|
||||
如果你不想手工再拼:
|
||||
|
||||
- bundle review
|
||||
- doctor 结论
|
||||
- 发布门禁
|
||||
|
||||
可以直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff
|
||||
```
|
||||
|
||||
如果已经有现成证据包,也可以直接基于 manifest 输出:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff /path/to/go-live-bundle
|
||||
```
|
||||
|
||||
这一步现在不是“另一套单独判断”,而是固定复用:
|
||||
|
||||
- `go-live-review`
|
||||
- `doctor-decision`
|
||||
- `go-live-summary / 发布门禁 / launchpad 摘要`
|
||||
|
||||
也就是说,页面首屏、CLI 和海外 Codex 看到的最终签收结论,已经开始共享同一份签字口径。
|
||||
|
||||
重点只看:
|
||||
|
||||
- `signoff_status`
|
||||
- `headline`
|
||||
- `release_gate`
|
||||
- `blocked_reasons`
|
||||
- `attention_reasons`
|
||||
- `decision`
|
||||
- `recommended_commands`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `signoff_status=ready`
|
||||
- 可进入最终人工签字或正式发布
|
||||
- `signoff_status=attention`
|
||||
- 说明现场已经接近完成,但仍应先清 attention 项
|
||||
- `signoff_status=blocked`
|
||||
- 本轮不应宣称收口完成
|
||||
|
||||
---
|
||||
|
||||
## 三、上线证据导出
|
||||
|
||||
### 1. 导出 bundle
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
|
||||
```
|
||||
|
||||
这会生成一整包上线前证据。
|
||||
|
||||
建议至少保留:
|
||||
|
||||
- `00_env_audit.txt`
|
||||
- `01_go_live_check_summary.txt`
|
||||
- `02_go_live_summary.json`
|
||||
- `03_stack_diagnosis.json`
|
||||
- `04_driver_feed.json`
|
||||
- `05_codex_brief.json`
|
||||
- `06_release_launchpad.json`
|
||||
- `manifest.json`
|
||||
|
||||
如果当前轮次涉及节点接管,建议额外保留:
|
||||
|
||||
- `agent_gap_export.json`
|
||||
- `node_bootstrap_plan.txt`
|
||||
- `node_acceptance_plan.json`
|
||||
|
||||
如果当前轮次涉及远端执行与日志回传,建议额外保留:
|
||||
|
||||
- `scene_log_*.txt`
|
||||
- `doctor_decision.json`
|
||||
|
||||
### 2. 直接复核 bundle
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review
|
||||
```
|
||||
|
||||
如果你已经切到 Ops Center 页面,也可以直接看:
|
||||
|
||||
- 首屏 `总检决策`
|
||||
- `总检主决策详情`
|
||||
- API `GET /api/v1/ops/doctor-decision`
|
||||
- 首屏 `交付证据`
|
||||
- `交付证据详情`
|
||||
- API `GET /api/v1/ops/go-live-bundle`
|
||||
- 首屏 `正式复核`
|
||||
- `正式复核详情`
|
||||
- API `GET /api/v1/ops/go-live-review`
|
||||
|
||||
记录结果:
|
||||
|
||||
- `status`
|
||||
- `headline`
|
||||
- `critical_failures`
|
||||
- `noncritical_failures`
|
||||
- `launchpad_alignment.consistent`
|
||||
- `env_audit.status`
|
||||
- `env_audit.missing_items`
|
||||
- `recommended_next_steps`
|
||||
|
||||
通过标准建议:
|
||||
|
||||
- `status=ready`
|
||||
- 可进入最终人工确认或正式发布
|
||||
- `status=attention`
|
||||
- 核心报告已齐,但仍需复核补充失败项或环境缺口
|
||||
- 也包括 `go_live_summary / stack_diagnosis / driver_feed / codex_brief` 的 launchpad 摘要出现不一致
|
||||
- `status=blocked`
|
||||
- 关键报告缺失,或环境审计已经判定阻断,不应直接宣称收口完成
|
||||
|
||||
如果 `env_audit.missing_items` 里出现:
|
||||
|
||||
- `missing_env:/etc/default/domaincheck-node-agent`
|
||||
|
||||
则这次 bundle review 已经不是泛泛地提示“环境缺口”,而是明确指向 Node Agent 接管还未收口。此时按下面顺序推进:
|
||||
|
||||
1. `agent-gap-export`
|
||||
2. `node-bootstrap-plan`
|
||||
3. 如果 summary 已提示 `runtime_may_need_restart=true`
|
||||
4. 完成恢复后重新导出 bundle,再做一次 `go-live-review`
|
||||
|
||||
### 3. 生成最终交付包
|
||||
|
||||
运行验证、bundle 复核都通过后,再生成最终交付包。
|
||||
|
||||
如果当前机器已经具备 smoke 运行条件,直接执行:
|
||||
|
||||
```bash
|
||||
bash ./prepare_final_release.sh
|
||||
```
|
||||
|
||||
如果当前机器只适合先做离线打包,可先跳过 smoke:
|
||||
|
||||
```bash
|
||||
DOMAINCHECK_SKIP_SMOKE_TEST=1 bash ./package_domain_release.sh
|
||||
bash ./verify_domain_release.sh
|
||||
```
|
||||
|
||||
后续再回到具备运行条件的机器上执行:
|
||||
|
||||
```bash
|
||||
bash ./prepare_final_release.sh
|
||||
```
|
||||
|
||||
最后统一查看最新交付结果:
|
||||
|
||||
```bash
|
||||
bash ./show_latest_release.sh
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `latest_release.package_name`
|
||||
- `latest_release.archive_path`
|
||||
- `latest_release.sha256`
|
||||
- `latest_release.smoke_test_ok`
|
||||
- `final_release_ok`
|
||||
- `final_release_report`
|
||||
- `final_release_report_ok`
|
||||
- `final_release_report_matches_latest`
|
||||
- `final_release_report_stale_reason`
|
||||
- `final_release_gate_decision`
|
||||
- `final_release_gate_blocked_reasons`
|
||||
- `final_release_gate_verify_ok`
|
||||
- `final_release_gate_smoke_test_ok`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `verify_domain_release.sh` 返回 `ok=true`
|
||||
- `latest_release.smoke_test_ok=true`
|
||||
- `final_release_ok=true`
|
||||
- `final_release_report_matches_latest=true`
|
||||
- `final_release_gate_decision=ready`
|
||||
- `final_release_gate_verify_ok=true`
|
||||
- `final_release_gate_smoke_test_ok=true`
|
||||
- `final_release_report` 已指向最新一轮生成的正式报告
|
||||
4. 先执行 `runtime-refresh-recover`
|
||||
5. 再重新导出 bootstrap plan
|
||||
|
||||
---
|
||||
|
||||
## 四、最终交付结论模板
|
||||
|
||||
下面这段可以直接作为正式发布前的交付结论模板使用。
|
||||
|
||||
### 模板正文
|
||||
|
||||
```text
|
||||
本轮发布前运行验证已完成。
|
||||
|
||||
一、统一收口结论
|
||||
- go_live_status: <填写>
|
||||
- publish_status: <填写>
|
||||
- stack_status: <填写>
|
||||
- automation_coverage.launch_status: <填写>
|
||||
- launchpad_status: <填写>
|
||||
|
||||
二、默认下一步
|
||||
- next_step_action_code: <填写>
|
||||
- operator_title: <填写>
|
||||
- launchpad_recommended_target_node_code: <填写>
|
||||
- launchpad_recommended_recovery_label: <填写>
|
||||
|
||||
三、上线证据包
|
||||
- bundle 路径: <填写>
|
||||
- manifest 复核状态: <填写 ready/attention/blocked>
|
||||
- go-live-review.headline: <填写>
|
||||
- manifest.review_status_hint: <填写>
|
||||
- critical_failures: <填写>
|
||||
- noncritical_failures: <填写>
|
||||
- manifest.launchpad_recommended_target_node_code: <填写>
|
||||
- manifest.launchpad_recommended_recovery_label: <填写>
|
||||
- manifest.launchpad_onboarding_bootstrap_pending_nodes: <填写>
|
||||
- manifest.launchpad_onboarding_acceptance_ready_nodes: <填写>
|
||||
- manifest.launchpad_alignment.consistent: <填写 true/false>
|
||||
|
||||
四、结论
|
||||
- <填写:可进入正式发布 / 仍需继续收口 / 暂不可发布>
|
||||
```
|
||||
|
||||
### 模板补充说明
|
||||
|
||||
填写“可进入正式发布”前,建议再做一次人工交叉确认:
|
||||
|
||||
- 发布门禁是否为 `ready` 或至少非 `blocked`
|
||||
- 当前主车道是否已经明确
|
||||
- 是否还存在 `dead_letter`
|
||||
- 是否还有节点停留在 `bootstrap pending / acceptance pending`
|
||||
- 是否已经保留可复盘的证据包与 manifest
|
||||
|
||||
---
|
||||
|
||||
## 五、交付给值班同事时最少要带什么
|
||||
|
||||
最少带下面 4 样:
|
||||
|
||||
1. `go-live-check` 摘要
|
||||
2. `stack-diagnosis` 结果
|
||||
3. `driver-feed / codex-brief` 结果
|
||||
4. `go-live-export` 生成的 `manifest.json`
|
||||
|
||||
如果值班同事需要直接接手节点问题,再多带 3 样:
|
||||
|
||||
1. `node-acceptance-run` 最近结果
|
||||
2. `log-sync-check` 最近结果
|
||||
3. `queue-status` 或 `queue-records dead_letter` 最近结果
|
||||
|
||||
这样交班的人不需要重新从零拼现场。
|
||||
|
||||
---
|
||||
|
||||
## 六、真正的收工标准
|
||||
|
||||
如果要算“这次版本已经可以收工”,建议至少同时满足:
|
||||
|
||||
- 前端构建成功
|
||||
- 海外单脑驾驶舱首屏状态正常
|
||||
- `go-live-review.status != blocked`
|
||||
- 发布门禁不为 `blocked`
|
||||
- Node Agent 目标节点已完成接管验收
|
||||
- 远端日志回传链可用或已确认本轮不依赖
|
||||
- delivery queue 不存在未处理 `dead_letter`
|
||||
- 当前默认下一步已经不是“继续补基础设施”,而是进入正式发布 / 观察 / 验收车道
|
||||
|
||||
到这一步,才适合说:
|
||||
|
||||
> 当前项目已经进入可上线、可交付、可持续运维状态。
|
||||
816
docs/schemas/ops_agent_protocol.md
Normal file
816
docs/schemas/ops_agent_protocol.md
Normal file
@@ -0,0 +1,816 @@
|
||||
# domainCheck Ops Agent Protocol
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 `Node Agent <-> Overseas Control Plane` 的正式 contract。
|
||||
|
||||
适用范围:
|
||||
|
||||
- `POST /api/v1/ops/agent/tokens`
|
||||
- `POST /api/v1/ops/agent/bootstrap-plan`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/handover`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/handover/bootstrap-plan`
|
||||
- `POST /api/v1/ops/agent/register`
|
||||
- `POST /api/v1/ops/agent/heartbeat`
|
||||
- `POST /api/v1/ops/agent/pull`
|
||||
- `POST /api/v1/ops/agent/jobs/{job_id}/start`
|
||||
- `POST /api/v1/ops/agent/jobs/{job_id}/complete`
|
||||
- `POST /api/v1/ops/agent/jobs/{job_id}/events`
|
||||
|
||||
设计原则:
|
||||
|
||||
- Agent 只执行声明过的结构化动作
|
||||
- 控制面负责调度、门禁、编排和审计
|
||||
- 所有回执都必须可重放、可去重、可死信化
|
||||
|
||||
---
|
||||
|
||||
## 2. 公共响应包裹
|
||||
|
||||
所有接口统一返回:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "ok",
|
||||
"detail_code": "",
|
||||
"data": {}
|
||||
}
|
||||
```
|
||||
|
||||
约束:
|
||||
|
||||
- `code=0` 表示成功
|
||||
- `code=1` 表示业务失败
|
||||
- `detail_code` 用于结构化错误语义
|
||||
- `message` 只承担可读说明,不承担机器判断主逻辑
|
||||
|
||||
常见 `detail_code`:
|
||||
|
||||
- `agent_token_invalid`
|
||||
- `agent_token_expired`
|
||||
- `agent_token_node_mismatch`
|
||||
- `ops_job_not_owned_by_agent`
|
||||
- `ops_job_invalid_status_transition`
|
||||
- `release_checksum_mismatch`
|
||||
- `release_health_check_failed`
|
||||
|
||||
---
|
||||
|
||||
## 3. 认证与请求头
|
||||
|
||||
Agent 相关接口统一使用:
|
||||
|
||||
- `Content-Type: application/json`
|
||||
- `X-Domaincheck-Agent-Token: <token>`
|
||||
|
||||
其中:
|
||||
|
||||
- `tokens`
|
||||
- `bootstrap-plan`
|
||||
|
||||
不要求 Agent Token。
|
||||
|
||||
其余接口都要求 Token。
|
||||
|
||||
---
|
||||
|
||||
## 4. 公共节点载荷
|
||||
|
||||
`register` 与 `heartbeat` 当前共享 `_base_payload()`。
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"title": "mainland-worker-02",
|
||||
"hostname": "host-a",
|
||||
"ip": "10.0.0.12",
|
||||
"agent_version": "0.1.0",
|
||||
"capabilities": [
|
||||
"service.start",
|
||||
"service.stop",
|
||||
"service.restart",
|
||||
"service.status",
|
||||
"health.snapshot",
|
||||
"logs.collect",
|
||||
"diagnostics.collect",
|
||||
"deploy.release"
|
||||
],
|
||||
"labels": {},
|
||||
"metadata": {
|
||||
"service_names": {
|
||||
"api": "domaincheck-api",
|
||||
"worker": "domaincheck-worker",
|
||||
"sync_agent": "domaincheck-sync-agent",
|
||||
"node_agent": "domaincheck-node-agent"
|
||||
},
|
||||
"delivery_queue": {
|
||||
"state": "healthy",
|
||||
"label": "正常",
|
||||
"reason": "当前没有待重试回执,也没有死信记录。",
|
||||
"pending_count": 0,
|
||||
"dead_letter_count": 0,
|
||||
"last_flush_at": "",
|
||||
"oldest_pending_at": "",
|
||||
"oldest_pending_request_id": "",
|
||||
"oldest_pending_kind": "",
|
||||
"oldest_dead_letter_at": "",
|
||||
"oldest_dead_letter_request_id": "",
|
||||
"oldest_dead_letter_kind": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
说明:
|
||||
|
||||
- `service_names` 是控制面识别节点服务拓扑的最小入口
|
||||
- `delivery_queue` 是控制面识别 Agent 回执现场的最小入口
|
||||
|
||||
---
|
||||
|
||||
## 5. Bootstrap Contract
|
||||
|
||||
### 5.1 Issue Token
|
||||
|
||||
`POST /api/v1/ops/agent/tokens`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"issued_by": "web-ui",
|
||||
"expires_in_hours": 72,
|
||||
"metadata": {
|
||||
"source": "ops-center"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
返回体关键字段:
|
||||
|
||||
- `token`
|
||||
- `token_preview`
|
||||
- `record_id`
|
||||
- `node_code`
|
||||
- `expires_at`
|
||||
- `created_at`
|
||||
|
||||
### 5.2 Bootstrap Plan
|
||||
|
||||
`POST /api/v1/ops/agent/bootstrap-plan`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"node_region": "mainland",
|
||||
"node_role": "worker",
|
||||
"issued_by": "web-ui",
|
||||
"expires_in_hours": 72,
|
||||
"control_plane_base_url": "https://ops.example.com",
|
||||
"root_dir": "/opt/domaincheck",
|
||||
"metadata": {
|
||||
"source": "ops-center"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
返回体关键字段:
|
||||
|
||||
- Token 相关字段
|
||||
- `control_plane_base_url`
|
||||
- `bootstrap_plan.node_code`
|
||||
- `bootstrap_plan.node_region`
|
||||
- `bootstrap_plan.node_role`
|
||||
- `bootstrap_plan.root_dir`
|
||||
- `bootstrap_plan.env_file`
|
||||
- `bootstrap_plan.service_name`
|
||||
- `bootstrap_plan.service_file`
|
||||
- `bootstrap_plan.install_script_path`
|
||||
- `bootstrap_plan.env_content`
|
||||
- `bootstrap_plan.command_lines`
|
||||
- `bootstrap_plan.command_block`
|
||||
- `bootstrap_plan.health_checks`
|
||||
- `bootstrap_plan.health_check_block`
|
||||
- `bootstrap_plan.bootstrap_script_name`
|
||||
- `bootstrap_plan.bootstrap_script_path`
|
||||
- `bootstrap_plan.bootstrap_script_content`
|
||||
- `bootstrap_plan.bootstrap_run_script_block`
|
||||
|
||||
约束:
|
||||
|
||||
- `bootstrap_plan` 是标准接入方案对象
|
||||
- 页面、Codex、CLI 必须复用这一个对象
|
||||
- 不允许各端再次手写 env 或 shell 逻辑
|
||||
|
||||
---
|
||||
|
||||
## 6. Runtime Contract
|
||||
|
||||
### 6.1 Register
|
||||
|
||||
`POST /api/v1/ops/agent/register`
|
||||
|
||||
语义:
|
||||
|
||||
- 校验 Token 与 `node_code`
|
||||
- Upsert Agent 运行态
|
||||
- 将节点推入托管目录候选态
|
||||
|
||||
最小成功返回字段:
|
||||
|
||||
- `node_code`
|
||||
- `expires_at`
|
||||
- `capabilities`
|
||||
|
||||
### 6.2 Heartbeat
|
||||
|
||||
`POST /api/v1/ops/agent/heartbeat`
|
||||
|
||||
语义:
|
||||
|
||||
- 刷新 `last_seen_at`
|
||||
- 刷新节点身份、服务名和队列快照
|
||||
- 维持控制面中的 Agent 在线态
|
||||
|
||||
最小成功返回字段:
|
||||
|
||||
- `node_code`
|
||||
- `server_time`
|
||||
- `expires_at`
|
||||
|
||||
### 6.3 Pull
|
||||
|
||||
`POST /api/v1/ops/agent/pull?limit=1`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02"
|
||||
}
|
||||
```
|
||||
|
||||
返回体关键字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"id": 123,
|
||||
"job_code": "ops-xxx",
|
||||
"action": "health.snapshot",
|
||||
"target_node_code": "mainland-worker-02",
|
||||
"status": "dispatching",
|
||||
"execution_mode": "remote-agent",
|
||||
"payload": {},
|
||||
"metadata": {},
|
||||
"policy": {},
|
||||
"steps": []
|
||||
}
|
||||
],
|
||||
"count": 1
|
||||
}
|
||||
```
|
||||
|
||||
约束:
|
||||
|
||||
- Agent 只能拿到属于自己的任务
|
||||
- Agent 不负责选择任务
|
||||
- 调度权始终在控制面
|
||||
|
||||
### 6.3.1 Agent Job Envelope
|
||||
|
||||
为了避免 Agent 再从自然语言动作名里“猜执行上下文”,`pull.data.jobs[]` 当前已经按统一任务包裹返回。
|
||||
|
||||
当前 `pull.data` 顶层也会额外带:
|
||||
|
||||
- `protocol_version`
|
||||
- `envelope_type`
|
||||
- `node_code`
|
||||
- `limit`
|
||||
- `count`
|
||||
|
||||
推荐最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"protocol_version": "ops-agent/v1",
|
||||
"envelope_type": "agent_job",
|
||||
"id": 123,
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-001",
|
||||
"job_type": "ops_action",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-02",
|
||||
"status": "dispatching",
|
||||
"execution_mode": "remote-agent",
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志",
|
||||
"job_ref": {
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-001",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-02"
|
||||
},
|
||||
"step_ref": {
|
||||
"step_id": 456,
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志"
|
||||
},
|
||||
"payload": {
|
||||
"service_name": "domaincheck-worker",
|
||||
"lines": 120,
|
||||
"include_agent_logs": false
|
||||
},
|
||||
"policy": {
|
||||
"timeout_seconds": 120,
|
||||
"stop_on_failure": true,
|
||||
"auto_approve": true
|
||||
},
|
||||
"metadata": {
|
||||
"requested_by": "playbook:inspection.standard",
|
||||
"source": "ops-center"
|
||||
},
|
||||
"release_context": {
|
||||
"release_id": 0,
|
||||
"release_version": "",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": ""
|
||||
},
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-001",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-02"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- Agent 必须优先消费:
|
||||
- `action`
|
||||
- `payload`
|
||||
- `policy`
|
||||
- Agent 不应通过:
|
||||
- `step_title`
|
||||
- `summary`
|
||||
- `notes`
|
||||
去反推真实执行参数
|
||||
- 与发布相关的任务应通过:
|
||||
- `release_context`
|
||||
传递 release / rollout 关联,而不是让 Agent 自己查询“当前版本”
|
||||
|
||||
这层的意义是:
|
||||
|
||||
- `ops job`
|
||||
- 仍是正式执行颗粒度
|
||||
- `Agent Job Envelope`
|
||||
- 是 Node Agent 拿到的稳定执行视图
|
||||
|
||||
这样同一套 Agent 才能同时承接:
|
||||
|
||||
- 标准巡检
|
||||
- 日志收集
|
||||
- 诊断采样
|
||||
- Release 部署
|
||||
- Rollout 回滚
|
||||
|
||||
### 6.4 Start
|
||||
|
||||
`POST /api/v1/ops/agent/jobs/{job_id}/start`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02"
|
||||
}
|
||||
```
|
||||
|
||||
语义:
|
||||
|
||||
- `dispatching` 表示已派发
|
||||
- `running` 表示节点已真实开工
|
||||
|
||||
### 6.5 Complete
|
||||
|
||||
`POST /api/v1/ops/agent/jobs/{job_id}/complete`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"client_request_id": "complete-ops-123-1",
|
||||
"status": "success",
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
"result": {
|
||||
"summary": "ok"
|
||||
},
|
||||
"error_message": ""
|
||||
}
|
||||
```
|
||||
|
||||
允许状态:
|
||||
|
||||
- `success`
|
||||
- `failed`
|
||||
- `partially_succeeded`
|
||||
|
||||
幂等约束:
|
||||
|
||||
- 使用 `client_request_id`
|
||||
- 控制面写入 `ops_jobs.last_agent_complete_request_id`
|
||||
- 同一完成回执可安全重放,不应再次制造副作用
|
||||
|
||||
推荐补充字段:
|
||||
|
||||
- `duration_ms`
|
||||
- `result.summary_text`
|
||||
- `result.focus_ref`
|
||||
|
||||
当前成功返回里也会额外带:
|
||||
|
||||
- `completion_summary.job_id`
|
||||
- `completion_summary.job_code`
|
||||
- `completion_summary.status`
|
||||
- `completion_summary.status_label`
|
||||
- `completion_summary.result_summary_text`
|
||||
- `completion_summary.focus_ref`
|
||||
- `completion_summary.deduplicated`
|
||||
|
||||
这样控制面在不展开 stdout/stderr 的情况下,也能直接把:
|
||||
|
||||
- 任务收口摘要
|
||||
- 后续跳转落点
|
||||
|
||||
并入 activity / inspection / rollout 观察面。
|
||||
|
||||
### 6.6 Events
|
||||
|
||||
`POST /api/v1/ops/agent/jobs/{job_id}/events`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"client_event_id": "event-ops-123-checksum-1",
|
||||
"step_id": 0,
|
||||
"event_type": "deploy_checksum_verified",
|
||||
"level": "info",
|
||||
"message": "checksum ok",
|
||||
"payload": {
|
||||
"checksum": "sha256:..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
去重约束:
|
||||
|
||||
- 使用 `client_event_id`
|
||||
- 控制面通过唯一索引去重
|
||||
- 同一事件可安全重放,不应重复落库
|
||||
|
||||
推荐补充字段:
|
||||
|
||||
- `occurred_at`
|
||||
- `summary_text`
|
||||
- `focus_ref`
|
||||
|
||||
当前成功返回里也会额外带:
|
||||
|
||||
- `event`
|
||||
- 即标准化后的事件对象
|
||||
- 内含 `event_key / summary_text / occurred_at / focus_ref`
|
||||
|
||||
这样事件流就不再只是“原始日志点”,而能直接成为:
|
||||
|
||||
- playbook run events
|
||||
- rollout events
|
||||
- activity-stream
|
||||
|
||||
共同消费的现场片段。
|
||||
|
||||
---
|
||||
|
||||
## 7. Agent 本地补发队列
|
||||
|
||||
当前主链路已经正式具备:
|
||||
|
||||
- `pending` 队列
|
||||
- `dead-letter` 队列
|
||||
- 定时 `_flush_delivery_queue()`
|
||||
- 永久性错误转死信
|
||||
|
||||
当前重点覆盖:
|
||||
|
||||
- `jobs/{id}/complete`
|
||||
- `jobs/{id}/events`
|
||||
|
||||
队列状态:
|
||||
|
||||
- `healthy`
|
||||
- `retrying`
|
||||
- `dead_letter`
|
||||
|
||||
语义:
|
||||
|
||||
- `healthy`:无积压,无死信
|
||||
- `retrying`:存在待补发回执,Agent 会继续自动重放
|
||||
- `dead_letter`:存在永久失败记录,需人工介入
|
||||
|
||||
控制面消费方式:
|
||||
|
||||
- 节点维度:
|
||||
- `delivery_queue_state`
|
||||
- `delivery_queue_label`
|
||||
- `delivery_queue_reason`
|
||||
- `delivery_queue_pending_count`
|
||||
- `delivery_queue_dead_letter_count`
|
||||
- 汇总维度:
|
||||
- `queue_retrying_nodes`
|
||||
- `queue_dead_letter_nodes`
|
||||
- `queue_pending_records`
|
||||
- `queue_dead_letter_records`
|
||||
|
||||
### 7.1 当前控制面已可见的最小现场
|
||||
|
||||
当前至少已经可以通过:
|
||||
|
||||
- `GET /api/v1/ops/nodes`
|
||||
|
||||
看到每台托管节点的队列现场:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"delivery_queue_state": "dead_letter",
|
||||
"delivery_queue_label": "死信 2",
|
||||
"delivery_queue_reason": "当前存在 2 条死信记录,建议优先查看节点日志或诊断编排。",
|
||||
"delivery_queue_pending_count": 0,
|
||||
"delivery_queue_dead_letter_count": 2,
|
||||
"delivery_queue_last_flush_at": "2026-04-18 06:10:00",
|
||||
"delivery_queue_oldest_pending_at": "",
|
||||
"delivery_queue_oldest_pending_request_id": "",
|
||||
"delivery_queue_oldest_pending_kind": "",
|
||||
"delivery_queue_oldest_dead_letter_at": "2026-04-18 05:59:00",
|
||||
"delivery_queue_oldest_dead_letter_request_id": "complete-ops-123-1",
|
||||
"delivery_queue_oldest_dead_letter_kind": "job_complete"
|
||||
}
|
||||
```
|
||||
|
||||
这已经足够让:
|
||||
|
||||
- 页面显示“节点存在死信”
|
||||
- `overview / driver-feed / codex-brief` 产出驾驶建议
|
||||
- CLI 快速判断现场是不是要先走日志或诊断
|
||||
|
||||
但这还不够支撑正式运维,因为它只能“看见死信”,还不能“处理死信”。
|
||||
|
||||
### 7.2 Delivery Queue Record 正式对象
|
||||
|
||||
后续控制面不应再把死信理解成一个纯计数器,而要把每条待补发 / 死信记录升级成正式对象。
|
||||
|
||||
推荐最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"record_id": "dq-mainland-worker-02-20260418-001",
|
||||
"node_code": "mainland-worker-02",
|
||||
"state": "dead_letter",
|
||||
"request_kind": "job_complete",
|
||||
"client_request_id": "complete-ops-123-1",
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-001",
|
||||
"target_api": "/api/v1/ops/agent/jobs/123/complete",
|
||||
"detail_code": "ops_job_invalid_status_transition",
|
||||
"error_message": "当前任务状态不允许 complete",
|
||||
"attempt_count": 6,
|
||||
"first_queued_at": "2026-04-18 05:58:00",
|
||||
"last_attempt_at": "2026-04-18 05:59:00",
|
||||
"next_retry_at": "",
|
||||
"payload_preview": {
|
||||
"status": "success"
|
||||
},
|
||||
"response_preview": {
|
||||
"code": 1,
|
||||
"detail_code": "ops_job_invalid_status_transition"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式约束:
|
||||
|
||||
- `record_id`
|
||||
- 是控制面侧唯一主键
|
||||
- `client_request_id`
|
||||
- 是 Agent 幂等键
|
||||
- `detail_code`
|
||||
- 是死信聚类主键,不能只靠 `message`
|
||||
- `payload_preview / response_preview`
|
||||
- 用于页面和 Codex 快速判断,不必默认展开完整原始报文
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `pending`
|
||||
- `retrying`
|
||||
- `dead_letter`
|
||||
- `replayed`
|
||||
- `discarded`
|
||||
|
||||
### 7.3 死信操作面正式入口
|
||||
|
||||
为了让“看见死信”之后不用回节点本地处理,推荐把操作面固定成下面这组接口。
|
||||
|
||||
当前已落地的第一阶段能力:
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||||
|
||||
当前控制面返回的记录可见性为:
|
||||
|
||||
- `record_visibility=head_only`
|
||||
|
||||
也就是:
|
||||
|
||||
- 当前已经能稳定查看节点级队列总览
|
||||
- 也能看到 `pending / dead_letter` 的头部记录摘要
|
||||
- 但还没有把节点本地全量 `pending/*.json / dead-letter/*.json` 正式同步到控制面
|
||||
|
||||
所以这两条 GET 入口现在属于“正式只读观察面”,而不是完整操作面。
|
||||
|
||||
#### a. 单节点队列总览
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"summary": {
|
||||
"state": "dead_letter",
|
||||
"pending_count": 0,
|
||||
"dead_letter_count": 2,
|
||||
"last_flush_at": "2026-04-18 06:10:00"
|
||||
},
|
||||
"oldest_pending_record": {},
|
||||
"oldest_dead_letter_record": {}
|
||||
}
|
||||
```
|
||||
|
||||
#### b. 队列记录列表
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||||
|
||||
推荐筛选参数:
|
||||
|
||||
- `state=pending|retrying|dead_letter|replayed|discarded`
|
||||
- `request_kind=job_complete|job_event`
|
||||
- `detail_code=...`
|
||||
- `group_by=detail_code|request_kind|target_api`
|
||||
- `limit=50`
|
||||
|
||||
当带 `group_by` 时,返回值应优先给出聚类摘要,而不是只返回平铺明细。
|
||||
|
||||
#### c. 单条死信重放
|
||||
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay`
|
||||
|
||||
当前已落地,但要明确当前阶段约束:
|
||||
|
||||
- 控制面仍然是 `record_visibility=head_only`
|
||||
- 所以单条动作目前只允许针对“当前可见头部记录”
|
||||
- 真正的执行不是控制面直接改远端文件,而是创建正式 `ops job`
|
||||
- 由目标节点 `Node Agent` 执行 `delivery.queue.replay`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"requested_by": "web-ui",
|
||||
"reason": "确认任务状态已修复,重放这条 complete 回执"
|
||||
}
|
||||
```
|
||||
|
||||
#### d. 批量重放
|
||||
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/replay`
|
||||
|
||||
当前已落地为正式操作面。
|
||||
|
||||
推荐请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"requested_by": "codex",
|
||||
"selector": {
|
||||
"state": "dead_letter",
|
||||
"detail_code": "ops_job_invalid_status_transition"
|
||||
},
|
||||
"limit": 20
|
||||
}
|
||||
```
|
||||
|
||||
#### e. 丢弃死信
|
||||
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard`
|
||||
|
||||
当前已落地,但与单条重放一样,仍然只允许针对当前可见头部死信记录发起单条动作。
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"discarded_by": "web-ui",
|
||||
"reason": "确认这条回执不再需要补发"
|
||||
}
|
||||
```
|
||||
|
||||
#### f. 主动冲刷队列
|
||||
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/flush`
|
||||
|
||||
当前已落地为正式操作面,执行方式同样是:
|
||||
|
||||
- 控制面创建 `ops job`
|
||||
- 目标节点 `Node Agent` 执行 `delivery.queue.flush`
|
||||
|
||||
语义:
|
||||
|
||||
- 不是“强行成功”
|
||||
- 而是立即触发一次 Agent 补发冲刷
|
||||
- 结果仍然回到 `pending / retrying / dead_letter / replayed`
|
||||
|
||||
### 7.4 为什么必须做成正式操作面
|
||||
|
||||
这层不是为了让页面多一个按钮,而是为了保证:
|
||||
|
||||
- 页面可以处理死信
|
||||
- CLI 可以处理死信
|
||||
- 海外 Codex 驾驶员可以处理死信
|
||||
|
||||
三者都不需要绕回:
|
||||
|
||||
- 手工 SSH 登录节点
|
||||
- 手工删本地文件
|
||||
- 手工重放某条 HTTP 请求
|
||||
|
||||
正式规则应该固定成:
|
||||
|
||||
- “死信查看” 走控制面对象
|
||||
- “死信重放 / 丢弃 / 冲刷” 走控制面对象
|
||||
- Agent 只负责执行,不负责决定如何处理死信
|
||||
|
||||
---
|
||||
|
||||
## 8. Job Status Enum
|
||||
|
||||
当前 Agent 主链路约束:
|
||||
|
||||
- `queued`
|
||||
- `dispatching`
|
||||
- `running`
|
||||
- `success`
|
||||
- `failed`
|
||||
- `partially_succeeded`
|
||||
|
||||
语义:
|
||||
|
||||
- `queued`:任务已创建,尚未派发
|
||||
- `dispatching`:控制面已派发,但节点尚未确认开工
|
||||
- `running`:节点已真实执行
|
||||
- `success`:成功完成
|
||||
- `failed`:失败完成
|
||||
- `partially_succeeded`:部分成功,需关注
|
||||
|
||||
---
|
||||
|
||||
## 9. 实现边界
|
||||
|
||||
Agent 只执行已声明动作:
|
||||
|
||||
- `service.start`
|
||||
- `service.stop`
|
||||
- `service.restart`
|
||||
- `service.status`
|
||||
- `health.snapshot`
|
||||
- `logs.collect`
|
||||
- `diagnostics.collect`
|
||||
- `deploy.release`
|
||||
|
||||
不允许控制面长期依赖任意 shell 下发。
|
||||
|
||||
`shell executor` 只能作为受限兜底能力存在,并必须挂审批与审计。
|
||||
56
docs/schemas/ops_doctor_decision_contract.md
Normal file
56
docs/schemas/ops_doctor_decision_contract.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# ops_doctor_decision_contract
|
||||
|
||||
## Purpose
|
||||
|
||||
冻结 `doctor-decision` 主决策 contract,让页面、CLI、Codex 都消费同一份“当前先看哪一面、先执行什么动作、为什么”的正式结论。
|
||||
|
||||
## Producer
|
||||
|
||||
- backend service `app.services.ops_service.get_ops_doctor_decision`
|
||||
- API `GET /api/v1/ops/doctor-decision`
|
||||
|
||||
## Primary Consumers
|
||||
|
||||
- `domain-web` Ops Center 顶部总检入口与主决策详情面板
|
||||
- `domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision`
|
||||
- 海外 Codex 驾驶员 / 自动驾驶策略层
|
||||
|
||||
## Required Payload
|
||||
|
||||
- `status`
|
||||
- `status_label`
|
||||
- `headline`
|
||||
- `detail`
|
||||
- `decision`
|
||||
- `summary`
|
||||
- `recommended_commands`
|
||||
- `contract_navigation`
|
||||
|
||||
## Decision Shape
|
||||
|
||||
- `decision.status`
|
||||
- `decision.reason_code`
|
||||
- `decision.headline`
|
||||
- `decision.detail`
|
||||
- `decision.preferred_surface`
|
||||
- `decision.next_action_code`
|
||||
- `decision.recommended_commands`
|
||||
|
||||
## Summary Shape
|
||||
|
||||
- `summary.required_failures`
|
||||
- `summary.optional_unavailable`
|
||||
- `summary.scene_log_reports_total`
|
||||
- `summary.scene_log_reports_ok`
|
||||
- `summary.scene_log_status_counts`
|
||||
- `summary.scene_log_problem_node_codes`
|
||||
- `summary.contract_surface_gaps`
|
||||
- `summary.launchpad_recommended_target_node_code`
|
||||
- `summary.launchpad_recommended_recovery_label`
|
||||
- `summary.launchpad_recommended_recovery_summary`
|
||||
|
||||
## Notes
|
||||
|
||||
- 优先读取最新 `go-live bundle` 中的 `doctor_decision` 产物。
|
||||
- 如果 bundle 尚未导出,后端必须回落到 live `stack-diagnosis + go-live-summary` 兜底,而不是直接返回空。
|
||||
- `preferred_surface` 必须稳定可消费,不能把面板跳转逻辑继续留给前端猜测。
|
||||
797
docs/schemas/ops_driver_contract.md
Normal file
797
docs/schemas/ops_driver_contract.md
Normal file
@@ -0,0 +1,797 @@
|
||||
# domainCheck Ops Driver Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 Overseas Ops Center / Codex Driver / CLI 共用的驾驶 contract。
|
||||
|
||||
核心原则:
|
||||
|
||||
- 后端负责推荐和解释
|
||||
- 前端负责展示和触发
|
||||
- Codex 负责选择入口和上下文
|
||||
- 是否允许自动执行由后端统一门禁
|
||||
- 页面与 Codex 都不能各自再推导一套优先级
|
||||
|
||||
---
|
||||
|
||||
## 2. 入口接口
|
||||
|
||||
当前驾驶 contract 主要来自:
|
||||
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `GET /api/v1/ops/driver-feed`
|
||||
- `GET /api/v1/ops/codex-brief`
|
||||
- `POST /api/v1/ops/driver-actions/preview`
|
||||
- `POST /api/v1/ops/driver-actions/resolve`
|
||||
- `POST /api/v1/ops/driver-actions/execute`
|
||||
- `POST /api/v1/ops/driver-actions/execute-resolved`
|
||||
- `POST /api/v1/ops/codex-actions/resolve`
|
||||
- `POST /api/v1/ops/codex-actions/execute`
|
||||
|
||||
补充配套:
|
||||
|
||||
- `GET /api/v1/ops/playbooks`
|
||||
- `GET /api/v1/ops/playbook-runs`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
|
||||
---
|
||||
|
||||
## 3. Driver Recommendation
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.driver_recommendations`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "node-agent-dead-letter",
|
||||
"priority": "先处理 Agent 死信",
|
||||
"title": "先处理 Agent 死信",
|
||||
"summary": "当前已有节点出现死信记录,继续堆动作会放大控制面与节点现场的不一致。",
|
||||
"reason": "Node Agent 回执队列已经出现死信。",
|
||||
"level_label": "最高优先",
|
||||
"tag_type": "danger",
|
||||
"primary_label": "查看 Worker 日志",
|
||||
"secondary_label": "打开标准巡检",
|
||||
"primary_action_code": "open_worker_logs",
|
||||
"secondary_action_code": "open_diagnostics",
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"primary_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"secondary_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"primary_action_payload": {},
|
||||
"secondary_action_payload": {},
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- 推荐不只给文案
|
||||
- 必须给动作码、节点和预填 payload
|
||||
- 主次动作允许节点和 payload 不同
|
||||
- 推荐对象允许直接携带 `focus_ref / primary_focus_ref / secondary_focus_ref`
|
||||
- 新客户端优先用 `focus_ref` 做统一定位,而不是再从动作码反推页面落点
|
||||
|
||||
---
|
||||
|
||||
## 4. Priority Recommendation
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.recommendation`
|
||||
|
||||
作用:
|
||||
|
||||
- 给首页顶部、CLI 摘要、Codex 首屏判断提供“当前第一优先动作”
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "driver_recommendations",
|
||||
"key": "node-agent-dead-letter",
|
||||
"priority": "先处理 Agent 死信",
|
||||
"title": "先处理 Agent 死信",
|
||||
"summary": "当前已有节点出现死信记录。",
|
||||
"reason": "Node Agent 回执队列已经出现死信。",
|
||||
"primary_action_code": "open_worker_logs",
|
||||
"secondary_action_code": "open_diagnostics"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Driver Feed
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/driver-feed`
|
||||
|
||||
作用:
|
||||
|
||||
- 提供比 `overview` 更直接给驾驶员消费的聚合结果
|
||||
|
||||
建议最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"top_recommendation": {},
|
||||
"driver_recommendations": [],
|
||||
"runbook_sequences": [],
|
||||
"activity_focus": [],
|
||||
"automation_coverage": {}
|
||||
}
|
||||
```
|
||||
|
||||
推荐补充约束:
|
||||
|
||||
- `top_recommendation`
|
||||
- 应该总是来自同一份 `entries` 列表,而不是额外拼一条“影子建议”
|
||||
- 这样页面、CLI、海外 Codex 在按 `key` 选中时才不会出现口径漂移
|
||||
- `driver_recommendations`
|
||||
- 普通驾驶建议,优先 `executor_kind=driver_action`
|
||||
- `runbook_sequences`
|
||||
- 标准作业路径,优先 `executor_kind=runbook_sequence`
|
||||
- `activity_focus`
|
||||
- 更偏观察与跟踪,不应伪装成“可立即执行”的驾驶动作
|
||||
- `activity_focus[*]`
|
||||
- 至少应保留:
|
||||
- `activity_key`
|
||||
- `activity_kind`
|
||||
- `summary`
|
||||
- `status`
|
||||
- `status_label`
|
||||
- `occurred_at`
|
||||
- `focus_ref`
|
||||
- `source_focus_ref`
|
||||
- `ui_intent`
|
||||
- 如果存在可落点观察入口
|
||||
- 允许补 `focus_action_code=focus_activity_item`
|
||||
- 但它的定位仍然是观察入口,而不是标准执行动作
|
||||
- `entries[*].focus_ref`
|
||||
- 是 driver-feed 的统一定位对象
|
||||
- 页面 / CLI / Codex 均应优先消费
|
||||
- `entries[*].primary_focus_ref` / `entries[*].secondary_focus_ref`
|
||||
- 是主次动作各自的稳定落点
|
||||
- 不允许调用方再根据 `primary_action_code` 自己猜“该跳 Release Hub、Launchpad 还是 Rollout Jobs”
|
||||
- `automation_coverage`
|
||||
- 是 Driver Feed 的自动化收口摘要
|
||||
- 用于统一回答:
|
||||
- 当前是否已经达到上线闸门
|
||||
- 当前还有多少动作停留在 preview-only
|
||||
- 当前后端是否已经足够接管首屏默认动作
|
||||
- 页面、CLI、Codex 不允许自行再统计一份“自动化覆盖率”
|
||||
|
||||
推荐最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"launch_ready": false,
|
||||
"launch_status": "attention",
|
||||
"summary": "仍有 3 个驾驶动作停留在 preview-only,暂未达到上线收口标准。",
|
||||
"total_actions": 12,
|
||||
"backend_handled_total": 9,
|
||||
"preview_only_total": 3,
|
||||
"ui_only_total": 0,
|
||||
"blocked_total": 0
|
||||
}
|
||||
```
|
||||
|
||||
其中 `runbook_sequences` 与 `activity_focus` 应继续区分:
|
||||
|
||||
- 这是建议动作
|
||||
- 还是固定标准作业路径
|
||||
- 还是最近值得优先处理的活动
|
||||
|
||||
推荐 CLI 消费顺序:
|
||||
|
||||
1. 显式 `entry_key`
|
||||
2. `top_recommendation`
|
||||
3. `entries[0]`
|
||||
|
||||
这样海外单入口就能稳定实现:
|
||||
|
||||
- `driver-focus-preview`
|
||||
- `driver-focus-run`
|
||||
|
||||
---
|
||||
|
||||
## 6. Codex Brief
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/codex-brief`
|
||||
|
||||
作用:
|
||||
|
||||
- 不让 Codex 自己从页面文案里猜“能不能自动执行”
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"mode": "guarded_auto",
|
||||
"recommended_behavior": "confirm_then_execute",
|
||||
"summary": "当前建议先处理 Agent 死信,再进入标准巡检。",
|
||||
"automation_coverage": {
|
||||
"launch_ready": false,
|
||||
"launch_status": "attention",
|
||||
"summary": "仍有 3 个驾驶动作停留在 preview-only。"
|
||||
},
|
||||
"target": {
|
||||
"kind": "driver_action",
|
||||
"action_code": "open_worker_logs",
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"action_payload": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐 `mode`:
|
||||
|
||||
- `safe_auto`
|
||||
- `guarded_auto`
|
||||
- `mixed`
|
||||
- `ui_only`
|
||||
- `blocked`
|
||||
|
||||
推荐 `recommended_behavior`:
|
||||
|
||||
- `auto_execute`
|
||||
- `confirm_then_execute`
|
||||
- `resolve_first`
|
||||
- `open_ui`
|
||||
- `blocked`
|
||||
|
||||
补充约束:
|
||||
|
||||
- `codex-brief.automation_coverage`
|
||||
- 应与 `driver-feed.automation_coverage` 保持同源
|
||||
- 供海外 Codex 驾驶员直接判断:
|
||||
- 是否允许默认走自动执行
|
||||
- 是否必须先预览或人工确认
|
||||
- 是否已经达到“可以上线”的自动化收口标准
|
||||
- `codex-brief.activity_focus`
|
||||
- 应直接透传 `driver-feed.activity_focus`
|
||||
- 供 Codex 在“当前没有可执行主线,但有观察焦点”时继续判断
|
||||
- `codex-brief.focus`
|
||||
- 优先使用 `entries[0]`
|
||||
- 当 `entries` 为空且 `activity_focus` 不为空时
|
||||
- 允许回退到首条带 `focus_action_code` 的 `activity_focus`
|
||||
- 这时推荐行为应偏向 `open_ui`
|
||||
- 不应伪装成自动执行动作
|
||||
|
||||
---
|
||||
|
||||
## 7. Driver Action Execute
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/driver-actions/execute`
|
||||
|
||||
作用:
|
||||
|
||||
- 把能标准化的驾驶动作收进统一后端执行入口
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"action_code": "run_standard_inspection",
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"action_payload": {}
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"handled": true,
|
||||
"mode": "playbook",
|
||||
"ui_intent": {
|
||||
"kind": "playbook_run_detail",
|
||||
"run_code": "run-20260418-001"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- 不只返回 handled / unhandled
|
||||
- 还要返回 `mode`
|
||||
- 还要返回 `ui_intent`
|
||||
|
||||
---
|
||||
|
||||
## 8. Driver Action Preview
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/driver-actions/preview`
|
||||
|
||||
作用:
|
||||
|
||||
- 给 OpsCenter / CLI / Codex Driver 统一生成驾驶动作契约预览
|
||||
- 冻结 `execution_chain / target_api / request_payload / 主次动作 section`
|
||||
- 不再让前端自己拼请求体和执行建议
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_label": "driver-feed",
|
||||
"title": "版本发布与放量",
|
||||
"summary": "当前可以进入正式发布路径。",
|
||||
"reason": "统一回到 Release / Rollout 闭环。",
|
||||
"executor_kind": "runbook_sequence",
|
||||
"sequence_key": "release_progression",
|
||||
"primary_label": "Worker 灰度",
|
||||
"secondary_label": "查看版本区",
|
||||
"secondary_action_code": "focus_release_hub",
|
||||
"requested_by": "web-ui/ops-center"
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"contract_key": "ops_driver_contract",
|
||||
"contract_version": "v1",
|
||||
"contract_schema_doc_path": "docs/schemas/ops_driver_contract.md",
|
||||
"execution_chain": "ops-driver -> runbook-sequence -> resolved driver action / ui-intent",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad"
|
||||
},
|
||||
"primary_focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
},
|
||||
"secondary_focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_detail"
|
||||
},
|
||||
"primary": {
|
||||
"label": "Worker 灰度",
|
||||
"action_code": "create_release_rollout_worker",
|
||||
"target_api": "/api/v1/ops/runbook/sequences/release_progression/execute",
|
||||
"target_method": "POST",
|
||||
"recommendation_label": "建议确认后执行",
|
||||
"automation_level_label": "guarded_auto",
|
||||
"risk_level_label": "high",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
},
|
||||
"request_payload": {
|
||||
"secondary": false,
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"node_codes": [
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"action_payload": {
|
||||
"release_id": 7,
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- preview 必须和 execute 走同一套 action 解析与 runbook resolve 逻辑
|
||||
- `primary` / `secondary` section 必须可直接给页面抽屉和 CLI 文本渲染复用
|
||||
- `request_payload` 必须是真正可发送到执行接口的 body,而不是页面内部临时对象
|
||||
- runbook sequence preview 必须经过后端 resolve,不能由页面自己猜 action_code
|
||||
- preview 返回体必须保留:
|
||||
- `focus_ref`
|
||||
- `primary_focus_ref`
|
||||
- `secondary_focus_ref`
|
||||
- `primary.focus_ref`
|
||||
- `secondary.focus_ref`
|
||||
|
||||
---
|
||||
|
||||
## 8.3. Driver Action Resolve / Execute Resolved
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/driver-actions/resolve`
|
||||
- `POST /api/v1/ops/driver-actions/execute-resolved`
|
||||
|
||||
作用:
|
||||
|
||||
- 把 `overview.driver_recommendations` / `driver-feed` / 页面按钮 / CLI 对普通驾驶动作的门禁判断统一下沉到后端
|
||||
- 页面不再根据 `preview` 自己推导 `直接执行 / 等确认 / 进入 UI / 阻断`
|
||||
- 后端统一输出:
|
||||
- `preview_request`
|
||||
- `preview`
|
||||
- `gate`
|
||||
- `selected_section`
|
||||
- `execution_result`
|
||||
|
||||
`resolve` 最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_label": "driver-feed",
|
||||
"title": "开启关键日志回传",
|
||||
"executor_kind": "driver_action",
|
||||
"primary_action_code": "enable_log_sync_key",
|
||||
"primary_action_payload": {},
|
||||
"requested_by": "web-ui/driver-feed",
|
||||
"secondary": false,
|
||||
"confirm": false
|
||||
}
|
||||
```
|
||||
|
||||
`resolve` 最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"preview_request": {},
|
||||
"preview": {},
|
||||
"selected_section": "primary",
|
||||
"focus_ref": {
|
||||
"kind": "execution_scene",
|
||||
"scene_key": "scene.logs"
|
||||
},
|
||||
"gate": {
|
||||
"decision": "execute_now",
|
||||
"decision_label": "直接执行",
|
||||
"decision_reason": "当前动作属于 safe_auto,可直接执行。",
|
||||
"will_execute": true,
|
||||
"recommendation": "auto_execute",
|
||||
"focus_ref": {
|
||||
"kind": "execution_scene",
|
||||
"scene_key": "scene.logs"
|
||||
},
|
||||
"action_code": "enable_log_sync_key",
|
||||
"target_api": "/api/v1/ops/driver-actions/execute"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`execute-resolved` 约束:
|
||||
|
||||
- 只有 `gate.will_execute=true` 才允许真正执行
|
||||
- `confirmation_required` 只有显式传入 `confirm=true` 才会放行
|
||||
- `open_ui / resolve_first / blocked` 必须返回非执行态,由页面或 CLI 继续展示契约
|
||||
- `driver-focus-preview / driver-focus-run`
|
||||
- 允许先从 `driver-feed.top_recommendation` 选出目标项
|
||||
- 也允许显式传入 `activity_focus.key`
|
||||
- 当目标来自 `activity_focus` 时
|
||||
- 应自动收口成 `focus_activity_item`
|
||||
- 只允许进入对应工作区,不应被当成标准执行动作
|
||||
- 再把它收口成同一份 `resolve / execute-resolved` 请求体
|
||||
- 不允许 CLI 再自己二次推导 gate
|
||||
- resolve 返回体必须保留:
|
||||
- `preview_request.focus_ref`
|
||||
- `preview_request.primary_focus_ref`
|
||||
- `preview_request.secondary_focus_ref`
|
||||
- `gate.focus_ref`
|
||||
- 顶层 `focus_ref`
|
||||
|
||||
---
|
||||
|
||||
## 8.5. Codex Action Resolve / Execute
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/codex-actions/resolve`
|
||||
- `POST /api/v1/ops/codex-actions/execute`
|
||||
|
||||
作用:
|
||||
|
||||
- 不再让 CLI / Codex 自己从 `codex-brief + preview` 拼一套门禁
|
||||
- 由后端统一输出:
|
||||
- 当前选中的 `selected_entry`
|
||||
- 对应的 `preview`
|
||||
- 统一门禁结论 `gate`
|
||||
- 最终是否允许执行
|
||||
|
||||
`resolve` 最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"entry_key": "runbook:release_progression",
|
||||
"requested_by": "cli/ops-center/codex-focus-preview",
|
||||
"source_label": "cli/codex-focus-preview",
|
||||
"include_secondary": true,
|
||||
"confirm": false
|
||||
}
|
||||
```
|
||||
|
||||
`resolve` 最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"entry_key": "runbook:release_progression",
|
||||
"selected_entry": {},
|
||||
"preview_request": {},
|
||||
"preview": {},
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
},
|
||||
"gate": {
|
||||
"decision": "confirmation_required",
|
||||
"decision_label": "等待确认",
|
||||
"recommendation": "confirm_then_execute",
|
||||
"recommendation_label": "建议确认后执行",
|
||||
"will_execute": false,
|
||||
"confirm_required": true,
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
},
|
||||
"target_api": "/api/v1/ops/runbook/sequences/release_progression/execute",
|
||||
"target_method": "POST",
|
||||
"request_payload": {
|
||||
"secondary": false,
|
||||
"requested_by": "cli/ops-center/codex-focus-preview",
|
||||
"action_payload": {
|
||||
"release_id": 7
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`execute` 要求:
|
||||
|
||||
- 必须复用 `resolve` 的同一套选中逻辑和门禁逻辑
|
||||
- `safe_auto`
|
||||
- 允许直接执行
|
||||
- `guarded_auto`
|
||||
- 只有 `confirm=true` 时允许执行
|
||||
- `resolve_first / open_ui / blocked`
|
||||
- 一律不得执行
|
||||
- 返回体必须保留:
|
||||
- `selected_entry`
|
||||
- `preview_request`
|
||||
- `preview`
|
||||
- `gate`
|
||||
- `focus_ref`
|
||||
- `execution_result`
|
||||
- `executed`
|
||||
|
||||
---
|
||||
|
||||
## 9. UI Intent
|
||||
|
||||
`ui_intent` 的职责不是让前端猜去哪,而是由后端直接声明:
|
||||
|
||||
- 应打开哪个详情
|
||||
- 应聚焦哪轮 run
|
||||
- 应落到哪个发布入口
|
||||
|
||||
当前建议固定支持:
|
||||
|
||||
- `playbook_run_detail`
|
||||
- `playbook_run_latest_events`
|
||||
- `job_events`
|
||||
- `rollout_jobs`
|
||||
- `managed_node_handover`
|
||||
- `managed_node_edit`
|
||||
- `open_release_dialog`
|
||||
- `open_rollout_dialog`
|
||||
- `focus_release_hub`
|
||||
- `focus_ref`
|
||||
|
||||
页面只做路由和抽屉承载,不做推荐解释。
|
||||
|
||||
---
|
||||
|
||||
## 10. Runbook Sequence
|
||||
|
||||
作用:
|
||||
|
||||
- 表达固定标准作业路径在“此刻”的推荐下一步
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "standard_inspection",
|
||||
"title": "标准巡检",
|
||||
"status": "warning",
|
||||
"status_label": "待补接管",
|
||||
"summary": "当前还没有满足标准巡检条件的节点。",
|
||||
"target_node_codes": [],
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad"
|
||||
},
|
||||
"primary_label": "直接执行标准巡检",
|
||||
"primary_action_code": "run_standard_inspection",
|
||||
"primary_action_payload": {},
|
||||
"primary_focus_ref": {},
|
||||
"secondary_label": "打开巡检编排",
|
||||
"secondary_action_code": "open_playbook_dialog",
|
||||
"secondary_action_payload": {
|
||||
"playbook_key": "inspection.standard"
|
||||
},
|
||||
"secondary_focus_ref": {}
|
||||
}
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- Runbook Sequence 是标准路径对象
|
||||
- 不是普通活动记录
|
||||
- 不是普通推荐卡
|
||||
- `focus_ref`
|
||||
- 表示这条标准路径整体对应的统一观察落点
|
||||
- `primary_focus_ref` / `secondary_focus_ref`
|
||||
- 表示主次动作各自应回看的稳定区域
|
||||
|
||||
---
|
||||
|
||||
## 11. Activity Stream
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
|
||||
作用:
|
||||
|
||||
- 把最近异常、编排、任务、Rollout 统一压成可钻取活动流
|
||||
|
||||
每条 activity 最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "playbook_run",
|
||||
"activity_key": "run-20260418-001",
|
||||
"status": "failed",
|
||||
"occurred_at": "2026-04-18 06:00:00",
|
||||
"summary": "标准巡检在 diagnostics.collect 步骤失败。",
|
||||
"ui_intent": {
|
||||
"kind": "playbook_run_detail",
|
||||
"run_code": "run-20260418-001"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐 `kind`:
|
||||
|
||||
- `playbook_run`
|
||||
- `ops_job`
|
||||
- `rollout`
|
||||
- `runbook_sequence`
|
||||
|
||||
---
|
||||
|
||||
## 12. 单脑驾驶舱首屏约束
|
||||
|
||||
Ops Center 首屏不是自由拼装页面,而是 Driver Contract 的正式消费者。
|
||||
|
||||
首屏必须优先消费同一组 contract:
|
||||
|
||||
- `overview.recommendation`
|
||||
- `driver-feed.top_recommendation`
|
||||
- `driver-feed.automation_coverage`
|
||||
- `codex-brief`
|
||||
|
||||
首屏建议固定为三层:
|
||||
|
||||
1. 驾驶结论
|
||||
2. 驾驶动作条
|
||||
3. 驾驶回执条
|
||||
|
||||
正式要求:
|
||||
|
||||
- 驾驶结论
|
||||
- 直接展示当前第一优先动作、自动化收口状态、是否达到上线闸门
|
||||
- 驾驶动作条
|
||||
- 只放“默认下一步 / 定位下一步 / 看契约 / 复制命令 / 进入主处理面板”这类高频动作
|
||||
- 这些按钮不能自己发明新规则,必须消费已有 `action_code / focus_ref / preview`
|
||||
- 驾驶回执条
|
||||
- 是首屏动作后的统一反馈区
|
||||
- 用于展示:
|
||||
- 已打开哪个契约
|
||||
- 已定位哪个焦点
|
||||
- 已复制哪个命令
|
||||
- 后端动作执行成功 / 警告 / 失败
|
||||
- 页面不应再把动作结果散落在多个局部 toast 里
|
||||
|
||||
也就是说:
|
||||
|
||||
- 首屏负责决策与触发
|
||||
- 下方详情区负责展开与下钻
|
||||
- 不允许详情区反向重写首屏推荐
|
||||
|
||||
---
|
||||
|
||||
## 13. 建议优先级顺序
|
||||
当前 contract 应优先遵守:
|
||||
|
||||
1. 先看异常编排
|
||||
2. 再看异常活动
|
||||
3. 再处理 Agent 死信 / 回执积压
|
||||
4. 再优先执行首个缺口节点的接入收口或接管验收
|
||||
5. 再处理现场日志回传和标准巡检
|
||||
6. 最后进入 Release / Rollout
|
||||
|
||||
这条顺序必须由后端维护,不应让前端或 Codex 再自行拼装。
|
||||
|
||||
补充约束:
|
||||
|
||||
- 当后端已经把首个缺口节点收敛成 `bootstrap_run / run_acceptance` 时:
|
||||
- `driver_feed.top_recommendation`
|
||||
- `codex_brief.focus`
|
||||
- `stack_diagnosis.next_step`
|
||||
都必须优先指向这一个节点和这一个动作
|
||||
- `fix_managed_nodes` 只能保留为兜底入口,不能再覆盖已经明确收敛的单节点恢复动作
|
||||
|
||||
---
|
||||
|
||||
## 14. 页面 / Codex / CLI 共用约束
|
||||
|
||||
同一条驾驶建议必须由后端一次性给出:
|
||||
|
||||
- 推荐标题
|
||||
- 推荐理由
|
||||
- 主动作
|
||||
- 次动作
|
||||
- 节点范围
|
||||
- 预填参数
|
||||
- UI 落点
|
||||
- 是否允许自动执行
|
||||
|
||||
这样三类入口才会真正共享一套驾驶 contract:
|
||||
|
||||
- OpsCenter 页面
|
||||
- 海外 Codex 驾驶员
|
||||
- CLI / 运维检查脚本
|
||||
|
||||
并且三类入口对 `recommended_behavior` 的执行语义也必须一致:
|
||||
|
||||
- `auto_execute`
|
||||
- 可直接执行
|
||||
- `confirm_then_execute`
|
||||
- 必须显式确认后才能执行
|
||||
- `resolve_first`
|
||||
- 只能先审阅和补上下文,不能直接执行
|
||||
- `open_ui`
|
||||
- 只能进入对应工作区,不能直接执行
|
||||
- `blocked`
|
||||
- 明确阻断,不能执行
|
||||
59
docs/schemas/ops_go_live_bundle_contract.md
Normal file
59
docs/schemas/ops_go_live_bundle_contract.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# ops_go_live_bundle_contract
|
||||
|
||||
## 目标
|
||||
|
||||
把发布前证据包与 `manifest.json` 复核结论压成一份统一 contract,供:
|
||||
|
||||
- Ops Center 交付面板
|
||||
- CLI `go-live-export / go-live-review`
|
||||
- 海外 Codex 驾驶员
|
||||
|
||||
共同消费。
|
||||
|
||||
它解决的问题不是“现在能不能上线”,而是:
|
||||
|
||||
> 现在有没有一份足够完整、足够可信、可交付可复盘的上线证据包
|
||||
|
||||
## 主接口
|
||||
|
||||
- `GET /api/v1/ops/go-live-bundle`
|
||||
|
||||
## 核心字段
|
||||
|
||||
- `status`
|
||||
- `missing`
|
||||
- `ready`
|
||||
- `attention`
|
||||
- `blocked`
|
||||
- `status_label`
|
||||
- `headline`
|
||||
- `bundle_available`
|
||||
- `report_dir`
|
||||
- `manifest_path`
|
||||
- `artifact_total`
|
||||
- `failure_total`
|
||||
- `critical_failures`
|
||||
- `noncritical_failures`
|
||||
- `env_audit`
|
||||
- `summary`
|
||||
- `recommended_commands`
|
||||
|
||||
## 判定原则
|
||||
|
||||
- 如果当前还没有 bundle / manifest
|
||||
- `status = missing`
|
||||
- 如果 manifest 明确标记 `review_status_hint=blocked`
|
||||
- `status = blocked`
|
||||
- 如果 manifest 已存在,但:
|
||||
- 仍有非关键失败
|
||||
- 环境审计提示 attention
|
||||
- launchpad 摘要存在漂移
|
||||
- 则 `status = attention`
|
||||
- 只有当 manifest 可读、review 为 ready、关键摘要一致
|
||||
- 才允许 `status = ready`
|
||||
|
||||
## 设计约束
|
||||
|
||||
- 页面不直接触发 shell 导出 bundle
|
||||
- 后端默认只读最新 bundle / manifest,不负责替代 CLI 执行导出
|
||||
- 当 bundle 缺失时,必须明确给出固定命令,而不是让操作者自己猜路径
|
||||
37
docs/schemas/ops_go_live_review_contract.md
Normal file
37
docs/schemas/ops_go_live_review_contract.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# ops_go_live_review_contract
|
||||
|
||||
## Purpose
|
||||
|
||||
冻结 `go-live-review` 复核结论 contract,让页面、CLI、Codex 都能消费同一份 bundle manifest 复核结果,而不是各自再解读一遍 artifact。
|
||||
|
||||
## Producer
|
||||
|
||||
- backend service `app.services.ops_service.get_ops_go_live_review`
|
||||
- API `GET /api/v1/ops/go-live-review`
|
||||
- CLI `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review`
|
||||
|
||||
## Primary Consumers
|
||||
|
||||
- `domain-web` Ops Center 证据复核面板
|
||||
- 海外 Codex 驾驶员
|
||||
- 发布前人工值班复核
|
||||
|
||||
## Required Payload
|
||||
|
||||
- `status`
|
||||
- `status_label`
|
||||
- `headline`
|
||||
- `artifact_total`
|
||||
- `failure_total`
|
||||
- `critical_failures`
|
||||
- `noncritical_failures`
|
||||
- `env_audit`
|
||||
- `launchpad_alignment`
|
||||
- `recommended_next_steps`
|
||||
- `recommended_commands`
|
||||
|
||||
## Notes
|
||||
|
||||
- 这层是 manifest 的正式“复核解释层”,不是单纯重复 bundle 基础信息。
|
||||
- 如果 `launchpad_alignment.consistent=false`,即使关键文件都存在,也必须至少回到 `attention`。
|
||||
- 页面和 Codex 应优先消费这层 `headline / recommended_next_steps`,而不是自行拼 review 口径。
|
||||
65
docs/schemas/ops_go_live_signoff_contract.md
Normal file
65
docs/schemas/ops_go_live_signoff_contract.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# ops_go_live_signoff_contract
|
||||
|
||||
## 目标
|
||||
|
||||
把发布前最终签收判断压成一份统一 contract,供:
|
||||
|
||||
- Ops Center 首屏
|
||||
- CLI `go-live-signoff`
|
||||
- 海外 Codex 驾驶员
|
||||
|
||||
共同消费。
|
||||
|
||||
它不是替代:
|
||||
|
||||
- `go-live-summary`
|
||||
- `go-live-review`
|
||||
- `doctor-decision`
|
||||
- `stack-diagnosis`
|
||||
- `driver-feed`
|
||||
- `codex-brief`
|
||||
|
||||
而是在这些 contract 之上给出最终一句:
|
||||
|
||||
> 现在能不能签字上线
|
||||
|
||||
## 主接口
|
||||
|
||||
- `GET /api/v1/ops/go-live-signoff`
|
||||
|
||||
## 核心字段
|
||||
|
||||
- `signoff_status`
|
||||
- `ready`
|
||||
- `attention`
|
||||
- `blocked`
|
||||
- `status_label`
|
||||
- `headline`
|
||||
- `release_gate`
|
||||
- `blocked_reasons`
|
||||
- `attention_reasons`
|
||||
- `decision`
|
||||
- `launchpad_alignment`
|
||||
- `recommended_commands`
|
||||
- `report_dir`
|
||||
- `manifest_path`
|
||||
|
||||
## 判定原则
|
||||
|
||||
- 只要 `go_live_summary / go_live_review / doctor_decision / publish_status / stack_status / automation / release_launchpad` 中任一明确阻断
|
||||
- `signoff_status = blocked`
|
||||
- 如果没有阻断,但存在:
|
||||
- `go-live-review=attention|missing`
|
||||
- `doctor-decision=attention|missing`
|
||||
- `attention`
|
||||
- launchpad 摘要漂移
|
||||
- 自动化仍未完全进入 ready
|
||||
- 则 `signoff_status = attention`
|
||||
- 只有当收口、正式复核、主决策、发布、自动化、launchpad 摘要一致
|
||||
- 才允许 `signoff_status = ready`
|
||||
|
||||
## 设计约束
|
||||
|
||||
- 页面不允许再自己拼一套“看起来能上”
|
||||
- CLI 不允许绕过这个 contract 单独下结论
|
||||
- Codex 驾驶员优先看这一份,再决定是否继续下钻
|
||||
740
docs/schemas/ops_job_contract.md
Normal file
740
docs/schemas/ops_job_contract.md
Normal file
@@ -0,0 +1,740 @@
|
||||
# domainCheck Ops Job Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 `ops job` 的正式 contract。
|
||||
|
||||
它是整套海外单脑控制面里最核心的执行对象。
|
||||
|
||||
正式定义:
|
||||
|
||||
> `ops job` 是一条可审计、可编排、可派发、可回执、可取消的标准运维任务记录。
|
||||
|
||||
它不是:
|
||||
|
||||
- 一条 shell 命令
|
||||
- 一个页面按钮点击结果
|
||||
- 一个仅供 agent 消费的内部对象
|
||||
|
||||
它应该同时服务于:
|
||||
|
||||
- 页面按钮
|
||||
- CLI
|
||||
- 海外 Codex 驾驶员
|
||||
- Node Agent
|
||||
- Release / Rollout
|
||||
- Playbook Run
|
||||
|
||||
也就是说:
|
||||
|
||||
- `playbook run`
|
||||
- 是编排聚合对象
|
||||
- `rollout`
|
||||
- 是发布推进对象
|
||||
- `ops job`
|
||||
- 是真正的执行颗粒度对象
|
||||
|
||||
---
|
||||
|
||||
## 2. 入口接口
|
||||
|
||||
当前已落地的主入口:
|
||||
|
||||
- `GET /api/v1/ops/jobs`
|
||||
- `GET /api/v1/ops/jobs/{job_id}`
|
||||
- `GET /api/v1/ops/jobs/{job_id}/events`
|
||||
- `POST /api/v1/ops/jobs`
|
||||
- `POST /api/v1/ops/jobs/batch`
|
||||
- `POST /api/v1/ops/policy/preview`
|
||||
- `POST /api/v1/ops/jobs/{job_id}/approve`
|
||||
- `POST /api/v1/ops/jobs/{job_id}/cancel`
|
||||
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
|
||||
|
||||
配套对象来源:
|
||||
|
||||
- `ops_jobs`
|
||||
- `ops_job_steps`
|
||||
- `ops_job_events`
|
||||
|
||||
正式约束:
|
||||
|
||||
- 页面、CLI、Codex 不能各自定义第 2 套任务模型
|
||||
- `playbook run` 只能聚合 `ops job`
|
||||
- `rollout` 只能批量生成 `ops job`
|
||||
- `Node Agent` 只能拉取和回执 `ops job`
|
||||
|
||||
---
|
||||
|
||||
## 3. 当前实现里的 Job Object
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/jobs`
|
||||
- `GET /api/v1/ops/jobs/{job_id}`
|
||||
- 创建 / 审批 / 取消 / 派发后的返回体中的 `job`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3",
|
||||
"action": "logs.collect",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"status": "queued",
|
||||
"status_label": "排队中",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"payload": {},
|
||||
"metadata": {},
|
||||
"result": {},
|
||||
"summary": "目标节点 mainland-worker-01 / 发起 web-ui/ops-center / 方式 Node Agent",
|
||||
"summary_text": "目标节点 mainland-worker-01 / 发起 web-ui/ops-center / 方式 Node Agent",
|
||||
"error_message": "",
|
||||
"created_at": "2026-04-18 06:45:00",
|
||||
"started_at": "",
|
||||
"finished_at": "",
|
||||
"updated_at": "2026-04-18 06:45:00",
|
||||
"risk_level": "medium",
|
||||
"approval_required": false,
|
||||
"approval_status": "approved",
|
||||
"approval_status_label": "已审批",
|
||||
"approved_by": "",
|
||||
"approved_at": "",
|
||||
"blocked_reason": "",
|
||||
"cancellation_reason": "",
|
||||
"dispatched_at": "",
|
||||
"target_selector": {},
|
||||
"policy": {},
|
||||
"rollout_id": 0,
|
||||
"steps_total": 0,
|
||||
"steps_running": 0,
|
||||
"steps_success": 0,
|
||||
"steps_failed": 0,
|
||||
"steps_terminal": 0,
|
||||
"steps_loaded": false,
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-01"
|
||||
},
|
||||
"steps": []
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- `job_code`
|
||||
- 是跨页面、CLI、事件流的稳定任务标识
|
||||
- `action`
|
||||
- 是执行语义主键,不应用 `title` 或自然语言代替
|
||||
- `target_type`
|
||||
- 定义任务作用范围,不应只靠是否传了 `target_node_code` 来猜
|
||||
- `execution_mode`
|
||||
- 是正式执行方式,不是附属备注
|
||||
- `policy`
|
||||
- 是创建时冻结的策略快照,不应由页面二次推导
|
||||
- `status_label / approval_status_label`
|
||||
- 是给页面、CLI、Codex 直接消费的人类可读标签
|
||||
- `summary_text`
|
||||
- 是任务摘要 contract,不要求调用方再用状态和节点自己拼一句中文
|
||||
- `steps_total / steps_running / steps_success / steps_failed / steps_terminal`
|
||||
- 是任务步骤聚合统计,避免前端和 CLI 再自行遍历 steps 统计
|
||||
- `focus_ref`
|
||||
- 是统一定位对象,后续页面、CLI、Codex 应优先消费它,而不是自己重组跳转参数
|
||||
|
||||
---
|
||||
|
||||
## 4. Job 状态机
|
||||
|
||||
### 4.1 当前实现中已使用的状态
|
||||
|
||||
当前后端已实际使用:
|
||||
|
||||
- `queued`
|
||||
- `awaiting_approval`
|
||||
- `blocked`
|
||||
- `running`
|
||||
- `success`
|
||||
- `failed`
|
||||
- `cancelled`
|
||||
|
||||
这些状态已经是正式 contract 的最小闭环。
|
||||
|
||||
### 4.2 推荐终局扩展状态
|
||||
|
||||
终局可继续扩展为:
|
||||
|
||||
- `draft`
|
||||
- `approved`
|
||||
- `queued`
|
||||
- `dispatching`
|
||||
- `running`
|
||||
- `verifying`
|
||||
- `partially_succeeded`
|
||||
- `rollback_running`
|
||||
- `rolled_back`
|
||||
- `success`
|
||||
- `failed`
|
||||
- `cancelled`
|
||||
- `timed_out`
|
||||
|
||||
但要明确区分:
|
||||
|
||||
- 当前实际 API 已稳定承诺的,是最小状态集
|
||||
- 终局扩展状态不能破坏当前状态语义
|
||||
|
||||
---
|
||||
|
||||
## 5. Execution Mode Contract
|
||||
|
||||
当前实现里正式出现的执行方式:
|
||||
|
||||
- `remote-agent`
|
||||
- `local-runtime`
|
||||
- `control-plane`
|
||||
- `ssh`
|
||||
|
||||
语义必须固定:
|
||||
|
||||
### 5.1 `remote-agent`
|
||||
|
||||
- 默认正式执行方式
|
||||
- 由目标节点 Node Agent 拉取并执行
|
||||
|
||||
### 5.2 `local-runtime`
|
||||
|
||||
- 仅用于当前节点本机即时执行
|
||||
- 主要用于本机 runtime 动作
|
||||
|
||||
### 5.3 `control-plane`
|
||||
|
||||
- 由海外控制面直接执行
|
||||
- 适用于控制面内生动作
|
||||
|
||||
### 5.4 `ssh`
|
||||
|
||||
- 只作为救援和过渡执行器
|
||||
- 不能成为大规模正式发布的默认方式
|
||||
|
||||
正式约束:
|
||||
|
||||
- 发布默认优先 `remote-agent`
|
||||
- `ssh` 只应是兜底链路
|
||||
- 页面、CLI、Codex 不应再自行创造第 5 种 execution mode
|
||||
|
||||
---
|
||||
|
||||
## 6. Target Type Contract
|
||||
|
||||
当前 contract 至少应允许:
|
||||
|
||||
- `node`
|
||||
- `batch`
|
||||
- `nodes`
|
||||
- `rollout`
|
||||
|
||||
语义建议固定为:
|
||||
|
||||
### 6.1 `node`
|
||||
|
||||
- 单节点任务
|
||||
|
||||
### 6.2 `batch` / `nodes`
|
||||
|
||||
- 多节点批量任务
|
||||
- 当前通常通过 `/ops/jobs/batch` 创建
|
||||
|
||||
### 6.3 `rollout`
|
||||
|
||||
- 该任务属于某轮 release rollout 推进的一部分
|
||||
|
||||
正式要求:
|
||||
|
||||
- 如果任务来自 rollout,必须能通过 `rollout_id` 回溯
|
||||
- 如果任务来自 playbook,应通过 `metadata` 标出 run 归属
|
||||
|
||||
---
|
||||
|
||||
## 7. Job Step Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/jobs/{job_id}`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 201,
|
||||
"step_key": "dispatch",
|
||||
"title": "调度动作",
|
||||
"node_code": "mainland-worker-01",
|
||||
"status": "queued",
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
"result": {},
|
||||
"created_at": "2026-04-18 06:45:00",
|
||||
"started_at": "",
|
||||
"finished_at": "",
|
||||
"updated_at": "2026-04-18 06:45:00"
|
||||
}
|
||||
```
|
||||
|
||||
当前实现说明:
|
||||
|
||||
- 现阶段每条 `ops job` 至少会先生成一个 `dispatch` step
|
||||
- 以后如果引入真正 DAG / 多步骤执行,也应继续沿用 `ops_job_steps`
|
||||
|
||||
正式约束:
|
||||
|
||||
- `step_key`
|
||||
- 是步骤主键,不应用标题代替
|
||||
- `status`
|
||||
- 必须与 job 状态联动,但不要求完全相同
|
||||
- `stdout / stderr`
|
||||
- 是步骤级输出,不应与整条 job 的 `result` 混成一层
|
||||
|
||||
---
|
||||
|
||||
## 8. Job Event Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/jobs/{job_id}/events`
|
||||
|
||||
最小字段建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 301,
|
||||
"event_key": "job-event:301",
|
||||
"job_id": 101,
|
||||
"step_id": 201,
|
||||
"node_code": "mainland-worker-01",
|
||||
"client_event_id": "evt-001",
|
||||
"event_type": "job_dispatch_requested",
|
||||
"level": "info",
|
||||
"level_label": "信息",
|
||||
"message": "任务等待 node-agent 拉取: 101",
|
||||
"summary": "任务等待 node-agent 拉取: 101",
|
||||
"payload": {},
|
||||
"has_payload": false,
|
||||
"occurred_at": "2026-04-18 06:45:03",
|
||||
"created_at": "2026-04-18 06:45:03"
|
||||
}
|
||||
```
|
||||
|
||||
当前代码里事件来源包括:
|
||||
|
||||
- job created
|
||||
- job blocked
|
||||
- job awaiting approval
|
||||
- job approved
|
||||
- job cancelled
|
||||
- job dispatch requested
|
||||
- job executed locally / over ssh / on control plane
|
||||
- agent start / complete / custom events
|
||||
|
||||
正式要求:
|
||||
|
||||
- `events` 是任务时间线,不是调试附属品
|
||||
- 页面、CLI、Codex 都应从这里读“任务做到哪一步了”
|
||||
- 不应重新从 stdout/stderr 推断关键状态
|
||||
|
||||
补充约束:
|
||||
|
||||
- `event_key`
|
||||
- 是事件行的稳定前端 / CLI 标识
|
||||
- `summary`
|
||||
- 是给页面、CLI、Codex 直接消费的短摘要,不要求调用方再拼接 `message`
|
||||
- `level_label`
|
||||
- 是面向人看的等级标签
|
||||
- `occurred_at`
|
||||
- 是统一观察面时间字段,允许与 `created_at` 同值
|
||||
|
||||
### 8.1 Event List Summary Contract
|
||||
|
||||
当前 `GET /api/v1/ops/jobs/{job_id}/events` 建议同时返回:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": {
|
||||
"job_id": 101,
|
||||
"returned_total": 12,
|
||||
"level_counts": {
|
||||
"info": 8,
|
||||
"warning": 3,
|
||||
"error": 1
|
||||
},
|
||||
"event_type_counts": {
|
||||
"job_created": 1,
|
||||
"job_dispatch_requested": 1,
|
||||
"executor_received": 1
|
||||
},
|
||||
"node_counts": {
|
||||
"mainland-worker-01": 12
|
||||
},
|
||||
"latest_at": "2026-04-18 06:45:08",
|
||||
"filters": {
|
||||
"limit": 50
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- 事件接口不只给一串数组
|
||||
- 至少要给:
|
||||
- 当前返回了多少条
|
||||
- 等级分布
|
||||
- 事件类型分布
|
||||
- 最近时间
|
||||
- 这样页面、CLI、Codex 才能先判断“有没有异常事件”再决定是否展开明细
|
||||
|
||||
---
|
||||
|
||||
## 9. List Contract 与 Compact 模式
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/jobs`
|
||||
|
||||
当前实现特点:
|
||||
|
||||
- 默认 `list_ops_jobs(..., compact=True)`
|
||||
- 返回 compact job
|
||||
- `steps` 被置空
|
||||
- `payload / metadata / result` 会被压缩
|
||||
|
||||
因此建议把列表 contract 明确成:
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3",
|
||||
"action": "logs.collect",
|
||||
"status": "queued",
|
||||
"execution_mode": "remote-agent",
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"payload": {
|
||||
"tail_lines": 120
|
||||
},
|
||||
"metadata": {},
|
||||
"result": {},
|
||||
"result_summary": {},
|
||||
"is_compact": true,
|
||||
"steps": []
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- 列表页默认只承诺 compact 视图
|
||||
- 详情页再承诺 full job + steps
|
||||
- 页面不应假设列表结果天然带完整 steps
|
||||
|
||||
---
|
||||
|
||||
## 10. Create Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "logs.collect",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"execution_mode": "remote-agent",
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"payload": {
|
||||
"tail_lines": 120
|
||||
},
|
||||
"metadata": {
|
||||
"source": "ops-center"
|
||||
},
|
||||
"auto_approve": false,
|
||||
"run_now": true
|
||||
}
|
||||
```
|
||||
|
||||
当前实现还支持:
|
||||
|
||||
- `target_selector`
|
||||
- `rollout_id`
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"job": {},
|
||||
"executed_immediately": false
|
||||
}
|
||||
```
|
||||
|
||||
关键语义:
|
||||
|
||||
- `executed_immediately=true`
|
||||
- 说明任务已被控制面即时执行,而不是停留在队列
|
||||
- `executed_immediately=false`
|
||||
- 说明任务进入了:
|
||||
- `queued`
|
||||
- `awaiting_approval`
|
||||
- `blocked`
|
||||
|
||||
正式要求:
|
||||
|
||||
- 创建接口不能只回答“建成功了”
|
||||
- 必须明确回答:
|
||||
- 当前状态是什么
|
||||
- 是否立即执行
|
||||
- 如果没执行,是在等审批、等 agent、还是被阻断
|
||||
|
||||
---
|
||||
|
||||
## 11. Batch Create Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs/batch`
|
||||
|
||||
最小请求体建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"template_key": "diagnostics.collect",
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-worker-02"
|
||||
],
|
||||
"execution_mode": "remote-agent",
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"payload": {},
|
||||
"metadata": {},
|
||||
"auto_approve": true
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs": [],
|
||||
"results": []
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- batch 只是创建方式,不是新的执行对象
|
||||
- batch 最终仍要展开成多条 `ops job`
|
||||
- 单条失败不能让结果完全丢失,必须保留逐项结果
|
||||
|
||||
---
|
||||
|
||||
## 12. Policy Preview Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/policy/preview`
|
||||
- `ops_jobs.policy`
|
||||
|
||||
这层是创建前的正式门禁对象。
|
||||
|
||||
当前创建任务时,后端已先做:
|
||||
|
||||
- 风险等级判断
|
||||
- 阻断判断
|
||||
- 审批要求判断
|
||||
|
||||
建议最小返回字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"blocked": false,
|
||||
"blocking_reasons": [],
|
||||
"approval_required": true,
|
||||
"approval_reasons": [
|
||||
"当前目标包含 control 节点"
|
||||
],
|
||||
"warnings": [],
|
||||
"recommendations": [],
|
||||
"risk_level": "high",
|
||||
"target_summary": {},
|
||||
"batch_plan": {},
|
||||
"cluster_guardrails": {},
|
||||
"operational_readiness": {}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- 页面、CLI、Codex 都应复用同一份 policy preview
|
||||
- 不能各自再写一套“能不能执行”的判断逻辑
|
||||
|
||||
---
|
||||
|
||||
## 13. Approval / Cancel / Dispatch Contract
|
||||
|
||||
### 13.1 Approve
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs/{job_id}/approve`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"approved_by": "admin"
|
||||
}
|
||||
```
|
||||
|
||||
语义:
|
||||
|
||||
- 仅 `awaiting_approval` 可审批
|
||||
- 审批后回到 `queued`
|
||||
|
||||
### 13.2 Cancel
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs/{job_id}/cancel`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"cancelled_by": "admin",
|
||||
"reason": "本轮灰度暂停"
|
||||
}
|
||||
```
|
||||
|
||||
语义:
|
||||
|
||||
- `success / failed / cancelled` 不可取消
|
||||
- 取消后 job 进入 `cancelled`
|
||||
- steps 同步进入 `cancelled` 或保持终态
|
||||
|
||||
### 13.3 Dispatch
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
|
||||
|
||||
语义:
|
||||
|
||||
- `queued` 才可派发
|
||||
- `control-plane`
|
||||
- 立即由控制面执行
|
||||
- `local-runtime`
|
||||
- 当前节点本机即时执行
|
||||
- `ssh`
|
||||
- 由 SSH 执行器执行
|
||||
- `remote-agent`
|
||||
- 保持 `queued`,等待对应 agent 拉取
|
||||
|
||||
这点必须明确:
|
||||
|
||||
> dispatch 不等于一定会立刻变成 running。
|
||||
|
||||
对于 `remote-agent`,它可能只是:
|
||||
|
||||
- 写入事件
|
||||
- 保持排队
|
||||
- 等节点稍后 pull
|
||||
|
||||
---
|
||||
|
||||
## 14. 与 Playbook / Rollout / Agent 的边界
|
||||
|
||||
### 14.1 与 Playbook
|
||||
|
||||
- `playbook run`
|
||||
- 负责展开、聚合、重跑、取消整轮编排
|
||||
- `ops job`
|
||||
- 负责单条执行记录
|
||||
|
||||
### 14.2 与 Rollout
|
||||
|
||||
- `rollout`
|
||||
- 负责发布批次推进
|
||||
- `ops job`
|
||||
- 负责每个节点每步的实际任务
|
||||
|
||||
### 14.3 与 Node Agent
|
||||
|
||||
- 控制面创建 `ops job`
|
||||
- agent 拉取 `ops job`
|
||||
- agent 回写 `ops_job_events`
|
||||
- agent 推进 job / step 状态
|
||||
|
||||
### 14.4 与 Observability
|
||||
|
||||
- `ops_observability_contract`
|
||||
- 看现场事实
|
||||
- `ops_job_contract`
|
||||
- 看执行动作本身
|
||||
|
||||
这两层不能混:
|
||||
|
||||
- 观察面告诉你“谁在跑、谁异常”
|
||||
- 任务面告诉你“这一条任务如何被创建、审批、派发、执行、取消”
|
||||
|
||||
---
|
||||
|
||||
## 15. 终局约束
|
||||
|
||||
后续不应再回退到下面几种模式:
|
||||
|
||||
### 1. 页面点按钮直接跑 shell
|
||||
|
||||
错误。
|
||||
|
||||
正确方式:
|
||||
|
||||
- 页面创建 `ops job`
|
||||
|
||||
### 2. Codex 直接 SSH 改现场
|
||||
|
||||
错误。
|
||||
|
||||
正确方式:
|
||||
|
||||
- Codex 创建 `ops job` / `playbook run`
|
||||
|
||||
### 3. 发布系统不经过任务层
|
||||
|
||||
错误。
|
||||
|
||||
正确方式:
|
||||
|
||||
- rollout 批量生成 `ops job`
|
||||
|
||||
### 4. 死信治理不留痕
|
||||
|
||||
错误。
|
||||
|
||||
正确方式:
|
||||
|
||||
- `flush / replay / discard` 都生成正式 `ops job`
|
||||
|
||||
所以可以把这份文档看成整套系统的“执行订单母本”。
|
||||
|
||||
后续只要这份 contract 稳住,页面、CLI、Codex、Agent、Release、Playbook 就都能围绕同一个中心对象协作。
|
||||
702
docs/schemas/ops_observability_contract.md
Normal file
702
docs/schemas/ops_observability_contract.md
Normal file
@@ -0,0 +1,702 @@
|
||||
# domainCheck Ops Observability Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结海外单脑控制面里“看现场”这一层的正式 contract。
|
||||
|
||||
它不负责:
|
||||
|
||||
- 发布决策
|
||||
- Driver 推荐
|
||||
- Playbook 编排
|
||||
|
||||
它负责回答 3 类问题:
|
||||
|
||||
1. 哪些节点现在真正处于执行现场
|
||||
2. 这些节点最近巡检收口结果是什么
|
||||
3. Node Agent 回执队列现在是否健康
|
||||
|
||||
也就是说,这份 contract 的职责是:
|
||||
|
||||
> 让页面、CLI、Codex 看到同一份现场事实,而不是各自从日志和状态字串里猜
|
||||
|
||||
---
|
||||
|
||||
## 2. 入口接口
|
||||
|
||||
建议把下面这些接口视为同一观察面的一组正式入口:
|
||||
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `GET /api/v1/ops/inspection-overview`
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/handover`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/flush`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/replay`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard`
|
||||
|
||||
配套关系:
|
||||
|
||||
- `overview`
|
||||
- 首页收口
|
||||
- `inspection-overview`
|
||||
- 节点巡检收口
|
||||
- `activity-stream`
|
||||
- 最近异常活动
|
||||
- `handover`
|
||||
- 单节点接管阻断与下一步建议
|
||||
- `delivery-queue`
|
||||
- Node Agent 回执健康
|
||||
|
||||
正式要求:
|
||||
|
||||
- 页面不能自己再定义第 5 套“观察口径”
|
||||
- CLI 不应重新从数据库拼现场
|
||||
- Codex 不应通过自然语言日志推断核心状态
|
||||
|
||||
---
|
||||
|
||||
## 3. Execution Scene Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `overview.execution_scene`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": "当前存在 2 台有效执行节点,其中 1 台正在领任务,1 台在线待命。",
|
||||
"dispatch_active_nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "busy",
|
||||
"participation_state": "dispatch_active",
|
||||
"participation_label": "正在执行",
|
||||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||||
"current_load": 9,
|
||||
"items_claimed": 9,
|
||||
"items_running": 5,
|
||||
"items_completed_recent": 14
|
||||
}
|
||||
],
|
||||
"recent_only_nodes": [],
|
||||
"standby_nodes": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"region": "mainland",
|
||||
"role": "control",
|
||||
"status": "online",
|
||||
"participation_state": "standby",
|
||||
"participation_label": "在线待命",
|
||||
"participation_reason": "节点在线、有效,但当前没有领任务、执行中任务或近窗吞吐",
|
||||
"current_load": 0
|
||||
}
|
||||
],
|
||||
"load_syncing_nodes": [],
|
||||
"counts": {
|
||||
"dispatch_active": 1,
|
||||
"recent_only": 0,
|
||||
"standby": 1,
|
||||
"load_syncing": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
分组语义必须固定:
|
||||
|
||||
- `dispatch_active_nodes`
|
||||
- 当前正在领任务、执行任务、或仍有实时工作负载
|
||||
- `recent_only_nodes`
|
||||
- 当前不再运行,但近窗刚有吞吐
|
||||
- `standby_nodes`
|
||||
- 在线、有效,但当前不参与
|
||||
- `load_syncing_nodes`
|
||||
- 节点与控制面状态尚未完全同步
|
||||
|
||||
正式约束:
|
||||
|
||||
- “在线”不等于“参与检测”
|
||||
- “有效执行节点”不等于“正在跑”
|
||||
- 页面展示、CLI 摘要、Codex 判断必须沿用这 4 类分组
|
||||
|
||||
节点参与态附加字段也应固定:
|
||||
|
||||
- `participation_state`
|
||||
- 机器判断主键,推荐值:
|
||||
- `dispatch_active`
|
||||
- `recent_only`
|
||||
- `standby`
|
||||
- `load_syncing`
|
||||
- `participation_label`
|
||||
- 直接给页面、CLI、Codex 展示的人类可读标签
|
||||
- `participation_reason`
|
||||
- 用一句稳定摘要解释为什么节点落在当前分组
|
||||
|
||||
正式要求:
|
||||
|
||||
- 页面必须直接区分:
|
||||
- 在线但未参与
|
||||
- 正在执行或领任务
|
||||
- 近窗刚参与但当前已收口
|
||||
- 新消费方不允许再从:
|
||||
- `status`
|
||||
- `current_load`
|
||||
- `items_running`
|
||||
手工推断“这个节点到底算不算正在参与”
|
||||
|
||||
---
|
||||
|
||||
## 4. Log Sync Runtime Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.execution_scene.log_sync`
|
||||
- 或 `overview.log_sync`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "partial_coverage",
|
||||
"status_label": "部分覆盖",
|
||||
"mode": "key",
|
||||
"mode_label": "关键回传",
|
||||
"enabled": true,
|
||||
"source_nodes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"source_node_summaries": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"line_count": 42,
|
||||
"key_line_count": 42,
|
||||
"full_line_count": 0,
|
||||
"last_at": "2026-04-18 06:40:00",
|
||||
"last_line": "域名检测完成: demo.com"
|
||||
}
|
||||
],
|
||||
"covered_participating_node_summaries": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"participation_state": "dispatch_active",
|
||||
"participation_label": "正在执行",
|
||||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||||
"line_count": 42,
|
||||
"key_line_count": 42,
|
||||
"full_line_count": 0,
|
||||
"last_at": "2026-04-18 06:40:00",
|
||||
"last_line": "域名检测完成: demo.com"
|
||||
}
|
||||
],
|
||||
"missing_participating_node_summaries": [
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"participation_state": "dispatch_active",
|
||||
"participation_label": "正在执行",
|
||||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||||
"missing_reason": "no_sample"
|
||||
}
|
||||
],
|
||||
"covered_participating_nodes": 1,
|
||||
"participating_nodes_total": 2,
|
||||
"missing_sample_node_codes": [
|
||||
"mainland-worker-02"
|
||||
],
|
||||
"last_line_at": "2026-04-18 06:40:00",
|
||||
"summary": "当前已覆盖 1/2 台参与节点,建议继续观察未回传样本节点。"
|
||||
}
|
||||
```
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `disabled`
|
||||
- `waiting_sample`
|
||||
- `partial_coverage`
|
||||
- `healthy`
|
||||
- `full_capture`
|
||||
|
||||
这层是“现场可见性”最关键的辅助状态。
|
||||
|
||||
正式要求:
|
||||
|
||||
- 如果存在参与节点但日志回传还没开,优先推荐开关键回传
|
||||
- 如果日志回传已开但没有样本,优先推荐抓 Worker 日志
|
||||
- 页面和 Codex 都不能再自己定义另一套日志覆盖结论
|
||||
|
||||
节点级日志覆盖字段也应固定:
|
||||
|
||||
- `source_nodes`
|
||||
- 为兼容旧页面保留的节点编码列表
|
||||
- `source_node_summaries`
|
||||
- 当前已有日志样本的节点级统计对象
|
||||
- `covered_participating_node_summaries`
|
||||
- 已参与且已拿到样本的节点对象
|
||||
- `missing_participating_node_summaries`
|
||||
- 已参与但尚未拿到样本的节点对象
|
||||
|
||||
`source_node_summaries[]` 至少应返回:
|
||||
|
||||
- `node_code`
|
||||
- `line_count`
|
||||
- `key_line_count`
|
||||
- `full_line_count`
|
||||
- `last_at`
|
||||
- `last_line`
|
||||
|
||||
`missing_participating_node_summaries[]` 至少应返回:
|
||||
|
||||
- `node_code`
|
||||
- `region`
|
||||
- `role`
|
||||
- `participation_state`
|
||||
- `participation_label`
|
||||
- `participation_reason`
|
||||
- `missing_reason`
|
||||
|
||||
推荐 `missing_reason`:
|
||||
|
||||
- `no_sample`
|
||||
- `delayed_flush`
|
||||
- `filtered_out`
|
||||
|
||||
兼容要求:
|
||||
|
||||
- 旧页面仍可继续消费:
|
||||
- `source_nodes`
|
||||
- `missing_sample_node_codes`
|
||||
- 新页面、CLI、Codex 应优先消费:
|
||||
- `source_node_summaries`
|
||||
- `covered_participating_node_summaries`
|
||||
- `missing_participating_node_summaries`
|
||||
|
||||
补充节点级现场入口:
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/scene-log`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"status": "healthy",
|
||||
"status_label": "关键覆盖",
|
||||
"mode": "key",
|
||||
"mode_label": "关键回传",
|
||||
"log_sync_enabled": true,
|
||||
"summary": "节点当前正在参与检测,已保留 42 条现场日志样本。",
|
||||
"node": {
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "busy",
|
||||
"current_load": 6,
|
||||
"last_heartbeat_at": "2026-04-18 06:40:00"
|
||||
},
|
||||
"participation": {
|
||||
"detect_participating": true,
|
||||
"participation_state": "claimed",
|
||||
"participation_label": "已领待跑",
|
||||
"participation_bucket": "dispatch_active",
|
||||
"participation_reason": "已领取任务,等待线程继续执行。"
|
||||
},
|
||||
"source_summary": {
|
||||
"line_count": 42,
|
||||
"key_line_count": 42,
|
||||
"full_line_count": 0,
|
||||
"last_at": "2026-04-18 06:40:00",
|
||||
"last_line": "[2026-04-18 06:40:00] [mainland-worker-01] 域名检测完成: demo.com"
|
||||
},
|
||||
"records_total": 42,
|
||||
"records_visible": 80,
|
||||
"records": [
|
||||
{
|
||||
"created_at": "2026-04-18 06:38:00",
|
||||
"node_code": "mainland-worker-01",
|
||||
"message": "开始检测域名: demo.com",
|
||||
"mode": "key",
|
||||
"line": "[2026-04-18 06:38:00] [mainland-worker-01] 开始检测域名: demo.com"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
节点级入口的正式语义:
|
||||
|
||||
- 这是“看某一台节点当前现场”的正式入口,不允许页面再自己从聚合样本中二次猜测
|
||||
- `status` 至少覆盖:
|
||||
- `healthy`
|
||||
- `full_capture`
|
||||
- `historical_sample`
|
||||
- `waiting_sample`
|
||||
- `missing_sample`
|
||||
- `disabled`
|
||||
- `standby`
|
||||
- `unknown`
|
||||
- 当节点当前在线但未参与时,必须明确返回 `standby`
|
||||
- 当节点正在参与但还没有样本时,必须明确返回 `waiting_sample`
|
||||
- 当节点已有历史样本但当前不在参与面中时,必须明确返回 `historical_sample`
|
||||
|
||||
---
|
||||
|
||||
## 5. Inspection Overview Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/inspection-overview`
|
||||
- `overview.inspection`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "attention",
|
||||
"status_label": "待处理",
|
||||
"summary": "存在 1 台节点最近缺少 Worker 日志收口。",
|
||||
"counts": {
|
||||
"healthy_nodes": 1,
|
||||
"attention_nodes": 1,
|
||||
"problem_nodes": 0
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"problem_kind": "missing_worker_logs",
|
||||
"problem_label": "缺少 Worker 日志收口",
|
||||
"problem_level": "warning",
|
||||
"recommended_action_code": "open_worker_logs",
|
||||
"ui_intent": {
|
||||
"kind": "node_inspection_detail",
|
||||
"node_code": "mainland-worker-01"
|
||||
},
|
||||
"latest_health_snapshot": {},
|
||||
"latest_worker_logs": {},
|
||||
"latest_diagnostics": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
设计要求:
|
||||
|
||||
- 巡检不是几条离散 job 记录
|
||||
- 必须按节点重新聚合
|
||||
- 必须直接给出:
|
||||
- `problem_kind`
|
||||
- `problem_label`
|
||||
- `problem_level`
|
||||
- `recommended_action_code`
|
||||
- `ui_intent`
|
||||
|
||||
这样页面、CLI、Codex 才能真正消费“节点巡检结论”,而不是再从几十条 job 记录里回推问题。
|
||||
|
||||
---
|
||||
|
||||
## 6. Inspection Row Contract
|
||||
|
||||
`rows[]` 中每个节点建议最少包含:
|
||||
|
||||
- `node_code`
|
||||
- `region`
|
||||
- `role`
|
||||
- `effective_worker`
|
||||
- `problem_kind`
|
||||
- `problem_label`
|
||||
- `problem_level`
|
||||
- `recommended_action_code`
|
||||
- `ui_intent`
|
||||
- `latest_health_snapshot`
|
||||
- `latest_worker_logs`
|
||||
- `latest_diagnostics`
|
||||
|
||||
其中 3 类最近结果建议统一结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"status_label": "成功",
|
||||
"occurred_at": "2026-04-18 06:35:00",
|
||||
"job_code": "ops-20260418-001",
|
||||
"summary": "最近一次 Worker 日志收集成功,共采样 120 行。"
|
||||
}
|
||||
```
|
||||
|
||||
正式边界:
|
||||
|
||||
- `logs.collect` 默认指 Worker 日志收口
|
||||
- Node Agent 辅助排障日志不应混成同一种巡检结果
|
||||
|
||||
---
|
||||
|
||||
## 7. Delivery Queue Summary Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node": {
|
||||
"node_code": "mainland-worker-01",
|
||||
"title": "mainland-worker-01"
|
||||
},
|
||||
"summary": {
|
||||
"state": "dead_letter",
|
||||
"label": "死信",
|
||||
"reason": "当前有 2 条记录进入死信。",
|
||||
"pending_count": 3,
|
||||
"dead_letter_count": 2,
|
||||
"oldest_pending_at": "2026-04-18 06:20:00",
|
||||
"oldest_dead_letter_at": "2026-04-18 06:18:00"
|
||||
},
|
||||
"record_visibility": "head_only",
|
||||
"capabilities": {
|
||||
"can_flush": true,
|
||||
"can_replay": true,
|
||||
"can_discard": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `healthy`
|
||||
- `retrying`
|
||||
- `dead_letter`
|
||||
- `stalled`
|
||||
|
||||
正式约束:
|
||||
|
||||
- 控制面要明确告诉操作者当前是 `head_only` 还是 `full_projection`
|
||||
- 不允许页面假装能列出远端完整目录
|
||||
- 不允许因为能看见死信就绕过 `ops job` 直接改文件
|
||||
|
||||
---
|
||||
|
||||
## 8. Delivery Queue Record Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||||
|
||||
当前阶段建议最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"records": [
|
||||
{
|
||||
"record_id": 101,
|
||||
"state": "dead_letter",
|
||||
"state_label": "死信",
|
||||
"request_id": "req-20260418-001",
|
||||
"kind": "job_complete",
|
||||
"created_at": "2026-04-18 06:18:00",
|
||||
"updated_at": "2026-04-18 06:19:00",
|
||||
"summary": "某次任务完成回执多次补发失败。",
|
||||
"visibility": "head_only"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"returned_records": 1,
|
||||
"record_visibility": "head_only"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
当前阶段要求:
|
||||
|
||||
- 单条 `replay / discard` 只针对控制面可见记录
|
||||
- 当前 `head_only` 不等于未来不能扩展
|
||||
- 以后若进入 `full_projection`,扩的是可见范围,不是重写操作模型
|
||||
|
||||
---
|
||||
|
||||
## 9. Delivery Queue Action Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `flush`
|
||||
- `replay`
|
||||
- `record replay`
|
||||
- `record discard`
|
||||
|
||||
正式要求:
|
||||
|
||||
- 所有动作都必须落成正式 `ops job`
|
||||
- 返回体要能告诉调用方:
|
||||
- 是否已受理
|
||||
- 生成了什么 job
|
||||
- 建议跳转到哪里观察
|
||||
|
||||
最小返回体建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"accepted": true,
|
||||
"job_code": "ops-20260418-queue-001",
|
||||
"ui_intent": {
|
||||
"kind": "job_events",
|
||||
"job_code": "ops-20260418-queue-001"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
这里的边界必须比“临时脚本”更严格:
|
||||
|
||||
- `discard` 必须带原因
|
||||
- `replay` 应支持限流 / limit
|
||||
- `flush` 应支持显式 `requested_by`
|
||||
|
||||
---
|
||||
|
||||
## 10. Activity Stream Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
|
||||
最小字段建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"kind": "ops_job",
|
||||
"activity_key": "ops-job:101",
|
||||
"title": "logs.collect",
|
||||
"subtitle": "ops-20260418064500-a1b2c3",
|
||||
"summary": "目标节点 mainland-worker-01 / 发起 web-ui / 方式 Node Agent",
|
||||
"summary_text": "目标节点 mainland-worker-01 / 发起 web-ui / 方式 Node Agent",
|
||||
"status": "running",
|
||||
"status_label": "执行中",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"occurred_at": "2026-04-18 06:45:08",
|
||||
"ui_intent_kind": "job_events",
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-01"
|
||||
}
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"status": "running",
|
||||
"status_label": "执行中",
|
||||
"summary_text": "最近活动里存在正在推进的现场、任务、编排或回传。",
|
||||
"filtered_total": 6,
|
||||
"unfiltered_total": 18,
|
||||
"latest_at": "2026-04-18 06:45:08"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- `activity-stream` 不是页面内部 table 数据
|
||||
- 它必须是:
|
||||
- 页面
|
||||
- CLI
|
||||
- Codex 驾驶员
|
||||
- 自动驾驶编排器
|
||||
共同消费的“最近活动摘要”
|
||||
|
||||
字段语义需要固定:
|
||||
|
||||
- `summary_text`
|
||||
- 是直接消费的活动短摘要
|
||||
- `status_label`
|
||||
- 是稳定的人类可读状态标签
|
||||
- `target_node_codes`
|
||||
- 是活动关联的节点口径,不允许调用方再从自然语言里猜
|
||||
- `ui_intent_kind`
|
||||
- 是当前活动默认落点类型
|
||||
- `focus_ref`
|
||||
- 是跨页面 / CLI / Codex 的统一定位对象
|
||||
|
||||
正式边界:
|
||||
|
||||
- 活动流应保留老字段以兼容现有页面
|
||||
- 新消费方优先用:
|
||||
- `summary_text`
|
||||
- `status_label`
|
||||
- `focus_ref`
|
||||
- 不允许新客户端再从 `title + subtitle + summary` 手工解析定位参数
|
||||
|
||||
## 11. 与 Driver / Playbook / Release 的边界关系
|
||||
|
||||
这份 contract 只负责观察面和现场治理,不负责替代其他 contract。
|
||||
|
||||
边界应明确为:
|
||||
|
||||
- `ops_driver_contract`
|
||||
- 决定“现在优先做什么”
|
||||
- `ops_playbook_contract`
|
||||
- 决定“标准多步动作怎么编排”
|
||||
- `release_hub_contract`
|
||||
- 决定“版本怎么发、怎么停、怎么回滚”
|
||||
- `ops_observability_contract`
|
||||
- 决定“现场事实是什么、当前缺口在哪里”
|
||||
|
||||
这样后续系统再扩展时,不会又回到“所有东西都写在 overview 一坨大 json 里”的状态。
|
||||
|
||||
---
|
||||
|
||||
## 12. 终局验收标准
|
||||
|
||||
如果这份 contract 真正落地,海外单脑控制面应该做到:
|
||||
|
||||
### 1. 不登录大陆机器也能回答现场问题
|
||||
|
||||
例如:
|
||||
|
||||
- 谁在跑
|
||||
- 谁在线但没跑
|
||||
- 谁近窗刚跑过
|
||||
- 谁日志没回来
|
||||
- 谁队列死信了
|
||||
|
||||
### 2. 不复制大日志也能先做判断
|
||||
|
||||
先看:
|
||||
|
||||
- execution scene
|
||||
- inspection overview
|
||||
- activity stream
|
||||
- delivery queue
|
||||
|
||||
再决定是否需要全量日志或 diagnostics bundle。
|
||||
|
||||
### 3. 页面、CLI、Codex 看到的是同一现场
|
||||
|
||||
不能出现:
|
||||
|
||||
- 页面说节点正在执行
|
||||
- CLI 说节点待命
|
||||
- Codex 说节点离线
|
||||
|
||||
### 4. 所有高风险动作都有正式观察落点
|
||||
|
||||
也就是:
|
||||
|
||||
- replay 后能看 job
|
||||
- discard 后能看审计
|
||||
- inspection 后能看节点收口
|
||||
- rollout 后能看 launchpad / gate / rollout jobs
|
||||
|
||||
这样这套系统才能真正从“调试台”升级成“驾驶舱”。
|
||||
563
docs/schemas/ops_playbook_contract.md
Normal file
563
docs/schemas/ops_playbook_contract.md
Normal file
@@ -0,0 +1,563 @@
|
||||
# domainCheck Ops Playbook Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 `playbook catalog / preview / run / events` 的正式 contract。
|
||||
|
||||
这层的定位不是“页面里的一个弹窗”,而是:
|
||||
|
||||
- 控制面的正式编排对象
|
||||
- 海外 Codex 驾驶员可消费的标准作业对象
|
||||
- Driver / Runbook / Release / Inspection 之间的统一中间层
|
||||
|
||||
核心原则:
|
||||
|
||||
- 一次标准编排 = 一次 `playbook run`
|
||||
- 页面、CLI、Codex 必须共用同一份 `playbook` / `playbook run` 结构
|
||||
- `playbook run` 只负责创建与聚合正式 `ops job`
|
||||
- 不允许前端再把“标准巡检 / 接管验收 / 现场日志”退回成零散按钮逻辑
|
||||
|
||||
建议与 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 配套阅读。
|
||||
|
||||
---
|
||||
|
||||
## 2. 当前接口面
|
||||
|
||||
当前已落地的主入口:
|
||||
|
||||
- `GET /api/v1/ops/playbooks`
|
||||
- `POST /api/v1/ops/playbooks/preview`
|
||||
- `POST /api/v1/ops/playbooks/execute`
|
||||
- `GET /api/v1/ops/playbook-runs`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
|
||||
- `POST /api/v1/ops/playbook-runs/{run_code}/rerun`
|
||||
- `POST /api/v1/ops/playbook-runs/{run_code}/cancel`
|
||||
|
||||
并且这层已经被以下入口复用:
|
||||
|
||||
- `driver action -> open_playbook_dialog`
|
||||
- `runbook sequence -> secondary_resolution`
|
||||
- `inspection`
|
||||
- `activity-stream`
|
||||
|
||||
所以后续不应再新造第四套“标准作业对象”。
|
||||
|
||||
---
|
||||
|
||||
## 3. Playbook Catalog Object
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/playbooks`
|
||||
- `preview / execute` 返回体中的 `playbook`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "inspection.standard",
|
||||
"group_key": "diagnostics",
|
||||
"group_title": "标准巡检",
|
||||
"title": "标准巡检",
|
||||
"description": "按 健康快照 -> Worker 日志 -> 诊断包 的顺序,对目标节点做一次标准联调巡检。",
|
||||
"target_scope": "node-set",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"default_execution_mode_label": "Node Agent",
|
||||
"execution_modes": [
|
||||
"remote-agent",
|
||||
"ssh"
|
||||
],
|
||||
"execution_mode_options": [
|
||||
{
|
||||
"value": "remote-agent",
|
||||
"label": "Node Agent",
|
||||
"is_default": true
|
||||
},
|
||||
{
|
||||
"value": "ssh",
|
||||
"label": "SSH",
|
||||
"is_default": false
|
||||
}
|
||||
],
|
||||
"default_auto_approve": true,
|
||||
"stop_on_failure": true,
|
||||
"steps": [
|
||||
{
|
||||
"step_key": "health",
|
||||
"title": "采集健康快照",
|
||||
"template_key": "health.snapshot",
|
||||
"payload": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
正式约束:
|
||||
|
||||
- `key`
|
||||
- 是 Playbook 的稳定主键
|
||||
- `group_key`
|
||||
- 用于页面、CLI、Codex 做标准分组,不应用 `title` 猜
|
||||
- `execution_modes`
|
||||
- 是能力白名单,不是建议文案
|
||||
- `stop_on_failure`
|
||||
- 是编排级行为,不是某一步骤的临时说明
|
||||
|
||||
---
|
||||
|
||||
## 4. Playbook Preview Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/playbooks/preview`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"playbook_key": "inspection.standard",
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"execution_mode": "remote-agent",
|
||||
"auto_approve": true,
|
||||
"requested_by": "web-ui"
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"playbook": {},
|
||||
"requested_by": "web-ui",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"auto_approve": true,
|
||||
"target_nodes_total": 1,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"step_count": 3,
|
||||
"expected_jobs_total": 3,
|
||||
"template_keys": [
|
||||
"health.snapshot",
|
||||
"logs.collect",
|
||||
"diagnostics.collect"
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"order": 1,
|
||||
"step_key": "health",
|
||||
"title": "采集健康快照",
|
||||
"template_key": "health.snapshot",
|
||||
"action": "health.snapshot",
|
||||
"payload": {},
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"auto_approve": true,
|
||||
"target_nodes_total": 1,
|
||||
"expected_jobs": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
预览阶段必须回答:
|
||||
|
||||
- 最终执行方式是什么
|
||||
- 总共会展开多少步
|
||||
- 每步会展开成什么模板
|
||||
- 总共会生成多少个 `ops job`
|
||||
|
||||
这样页面、CLI、Codex 才能在执行前看见“这次会做什么”。
|
||||
|
||||
---
|
||||
|
||||
## 5. Playbook Run Object
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/playbook-runs`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}`
|
||||
- `POST /api/v1/ops/playbooks/execute`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_key": "inspection.standard",
|
||||
"playbook_title": "标准巡检",
|
||||
"group_key": "diagnostics",
|
||||
"group_title": "标准巡检",
|
||||
"requested_by": "web-ui",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"stop_on_failure": true,
|
||||
"step_count": 3,
|
||||
"expected_jobs_total": 3,
|
||||
"target_nodes_total": 1,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"created_at": "2026-04-18 06:30:00",
|
||||
"updated_at": "2026-04-18 06:31:10",
|
||||
"status": "running",
|
||||
"status_label": "收口中",
|
||||
"status_counts": {
|
||||
"queued": 1,
|
||||
"running": 1,
|
||||
"success": 1
|
||||
},
|
||||
"jobs_total": 3,
|
||||
"success_jobs_total": 1,
|
||||
"terminal_jobs_total": 1,
|
||||
"completion_percent": 33.3,
|
||||
"steps_total": 3,
|
||||
"steps_success": 1,
|
||||
"steps_running": 1,
|
||||
"steps_problem": 0,
|
||||
"steps_terminal": 1,
|
||||
"focus_level": "warning",
|
||||
"focus_step_key": "worker_logs",
|
||||
"focus_step_title": "收集 Worker 日志",
|
||||
"focus_summary": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
|
||||
"summary": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
|
||||
"summary_text": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
|
||||
"focus_ref": {
|
||||
"kind": "playbook_run",
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_key": "inspection.standard",
|
||||
"group_key": "diagnostics",
|
||||
"focus_step_key": "worker_logs",
|
||||
"focus_step_title": "收集 Worker 日志",
|
||||
"event_key": "",
|
||||
"node_code": ""
|
||||
},
|
||||
"problem_steps": [],
|
||||
"active_steps": [],
|
||||
"latest_job": {},
|
||||
"steps": []
|
||||
}
|
||||
```
|
||||
|
||||
正式定义:
|
||||
|
||||
> `playbook run` 是一轮标准编排在控制面的聚合对象,不是某一条单独任务。
|
||||
|
||||
也就是说:
|
||||
|
||||
- `ops job` 是执行颗粒度
|
||||
- `playbook run` 是观察、重跑、取消、钻取的编排颗粒度
|
||||
|
||||
---
|
||||
|
||||
## 6. Playbook Step Object
|
||||
|
||||
`playbook run.steps[]` 的最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"step_key": "worker_logs",
|
||||
"title": "收集 Worker 日志",
|
||||
"template_key": "logs.collect",
|
||||
"order": 2,
|
||||
"jobs_total": 1,
|
||||
"nodes_total": 1,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"status_counts": {
|
||||
"running": 1
|
||||
},
|
||||
"terminal_jobs_total": 0,
|
||||
"success_jobs_total": 0,
|
||||
"status": "running",
|
||||
"status_label": "收口中",
|
||||
"completion_percent": 0.0,
|
||||
"summary": "收集 Worker 日志 当前处于执行中,可继续观察该步骤事件。",
|
||||
"summary_text": "收集 Worker 日志 当前处于执行中,可继续观察该步骤事件。",
|
||||
"focus_ref": {
|
||||
"kind": "playbook_run",
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_key": "inspection.standard",
|
||||
"group_key": "diagnostics",
|
||||
"focus_step_key": "worker_logs",
|
||||
"focus_step_title": "收集 Worker 日志",
|
||||
"event_key": "",
|
||||
"node_code": ""
|
||||
},
|
||||
"latest_job": {}
|
||||
}
|
||||
```
|
||||
|
||||
推荐 step 状态:
|
||||
|
||||
- `queued`
|
||||
- `running`
|
||||
- `success`
|
||||
- `attention`
|
||||
|
||||
规则:
|
||||
|
||||
- 只要该步骤出现失败、阻断、取消等终态问题,应收口为 `attention`
|
||||
- 页面与 Codex 都应优先盯 `problem_steps / active_steps`,而不是自己遍历所有子任务推断
|
||||
|
||||
---
|
||||
|
||||
## 7. Playbook Event Stream
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_run": {},
|
||||
"events": [
|
||||
{
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-123",
|
||||
"run_code": "playbook-20260418-001",
|
||||
"job_status": "running",
|
||||
"job_status_label": "收口中",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志",
|
||||
"event_type": "job_running",
|
||||
"level": "info",
|
||||
"level_label": "信息",
|
||||
"message": "开始收集 Worker 日志",
|
||||
"summary": "开始收集 Worker 日志",
|
||||
"summary_text": "开始收集 Worker 日志",
|
||||
"payload": {},
|
||||
"event_key": "job-event:1001",
|
||||
"occurred_at": "2026-04-18 06:31:00",
|
||||
"focus_ref": {
|
||||
"kind": "playbook_run",
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_key": "inspection.standard",
|
||||
"group_key": "diagnostics",
|
||||
"focus_step_key": "worker_logs",
|
||||
"focus_step_title": "收集 Worker 日志",
|
||||
"event_key": "job-event:1001",
|
||||
"node_code": "mainland-worker-01"
|
||||
},
|
||||
"created_at": "2026-04-18 06:31:00"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"returned_total": 1,
|
||||
"available_step_counts": {
|
||||
"worker_logs": 1
|
||||
},
|
||||
"available_node_counts": {
|
||||
"mainland-worker-01": 1
|
||||
},
|
||||
"level_counts": {
|
||||
"info": 1
|
||||
},
|
||||
"event_type_counts": {
|
||||
"job_running": 1
|
||||
},
|
||||
"job_status_counts": {
|
||||
"running": 1
|
||||
},
|
||||
"latest_at": "2026-04-18 06:31:00",
|
||||
"filters": {
|
||||
"step_key": "",
|
||||
"node_code": "",
|
||||
"limit": 80
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
这层必须支持:
|
||||
|
||||
- 按 `step_key` 过滤
|
||||
- 按 `node_code` 过滤
|
||||
- 快速看到最近活动时间
|
||||
|
||||
否则 `playbook run` 只能看总览,不能真正钻取。
|
||||
|
||||
---
|
||||
|
||||
## 8. 正式 Playbook Key 语义
|
||||
|
||||
当前最值得冻结成正式对象的 key:
|
||||
|
||||
### 8.1 `onboarding.bootstrap`
|
||||
|
||||
定位:
|
||||
|
||||
- 生成节点接入工单
|
||||
|
||||
执行方式:
|
||||
|
||||
- 仅 `control-plane`
|
||||
|
||||
正式语义:
|
||||
|
||||
- 不直接接管节点
|
||||
- 只创建标准接入方案对象
|
||||
|
||||
### 8.2 `onboarding.acceptance`
|
||||
|
||||
定位:
|
||||
|
||||
- 新节点接管后标准验收
|
||||
|
||||
当前标准步骤:
|
||||
|
||||
1. `health.snapshot`
|
||||
2. `service.status(node-agent)`
|
||||
3. `logs.collect(node-agent)`
|
||||
4. `service.status(worker)`
|
||||
5. `logs.collect(worker)`
|
||||
|
||||
### 8.3 `inspection.standard`
|
||||
|
||||
定位:
|
||||
|
||||
- 标准巡检
|
||||
|
||||
当前标准步骤:
|
||||
|
||||
1. `health.snapshot`
|
||||
2. `logs.collect(worker)`
|
||||
3. `diagnostics.collect`
|
||||
|
||||
正式要求:
|
||||
|
||||
- 这是 release / rollout 之前最核心的健康观察面
|
||||
|
||||
### 8.4 `scene.logs.key`
|
||||
|
||||
定位:
|
||||
|
||||
- 关键现场日志样本
|
||||
|
||||
正式要求:
|
||||
|
||||
- 默认比 `scene.logs.full` 更轻
|
||||
- 适合先看现场,再决定是否升级取证
|
||||
|
||||
### 8.5 `scene.logs.full`
|
||||
|
||||
定位:
|
||||
|
||||
- 全量现场取证
|
||||
|
||||
正式要求:
|
||||
|
||||
- 应明确比 `scene.logs.key` 更重
|
||||
- 默认应同时补诊断包
|
||||
|
||||
### 8.6 `scene.diagnostics`
|
||||
|
||||
定位:
|
||||
|
||||
- 轻量诊断包
|
||||
|
||||
正式要求:
|
||||
|
||||
- 适合作为参与节点的常规取证动作
|
||||
|
||||
---
|
||||
|
||||
## 9. 执行模式 Contract
|
||||
|
||||
推荐模式:
|
||||
|
||||
- `remote-agent`
|
||||
- `ssh`
|
||||
- `control-plane`
|
||||
|
||||
正式约束:
|
||||
|
||||
- `remote-agent`
|
||||
- 正式日常运维默认模式
|
||||
- `ssh`
|
||||
- 仅用于首发接管与应急救援
|
||||
- `control-plane`
|
||||
- 只适合控制面本地对象生成或极少量控制面动作
|
||||
|
||||
页面、CLI、Codex 不能自己发明第四种临时模式。
|
||||
|
||||
---
|
||||
|
||||
## 10. Playbook Run 状态机
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `queued`
|
||||
- `running`
|
||||
- `success`
|
||||
- `attention`
|
||||
|
||||
含义:
|
||||
|
||||
- `queued`
|
||||
- 子任务刚创建,尚未出现有效执行推进
|
||||
- `running`
|
||||
- 已有子任务进入 `dispatching / running / awaiting_approval`
|
||||
- `success`
|
||||
- 全部子任务收口成功
|
||||
- `attention`
|
||||
- 任一步骤出现失败、阻断、取消或部分成功,需要人工关注
|
||||
|
||||
正式规则:
|
||||
|
||||
- 页面上的“标准巡检状态”
|
||||
- `activity-stream` 中的 playbook run 摘要
|
||||
- Driver / Codex 的推荐动作
|
||||
|
||||
都必须共用这一套状态语义。
|
||||
|
||||
---
|
||||
|
||||
## 11. 与 Driver / Runbook / Release 的关系
|
||||
|
||||
正式收口规则:
|
||||
|
||||
- Driver 不直接“模拟巡检”
|
||||
- 应优先跳转或创建 `playbook run`
|
||||
- Runbook Sequence 不直接展开 shell
|
||||
- 应优先收口到 `driver action` 或 `playbook`
|
||||
- Release / Rollout 不自己维护另一套巡检摘要
|
||||
- 应优先消费 `inspection.standard` 的最近结果
|
||||
|
||||
也就是说:
|
||||
|
||||
- `scene` 是现场观察对象
|
||||
- `inspection` 是标准健康对象
|
||||
- `release / rollout` 是发布对象
|
||||
|
||||
三者不能再各自维护不同的“日志 / 诊断 / 巡检”定义。
|
||||
|
||||
---
|
||||
|
||||
## 12. 正式要求
|
||||
|
||||
后续继续演进时,必须保持:
|
||||
|
||||
1. `playbook catalog`
|
||||
- 是稳定 contract,而不是页面枚举
|
||||
2. `playbook preview`
|
||||
- 是执行前的唯一结构化预览面
|
||||
3. `playbook run`
|
||||
- 是页面、CLI、Codex 共同观察的一轮编排对象
|
||||
4. `playbook events`
|
||||
- 是整轮编排的钻取视角
|
||||
5. 接管验收、标准巡检、现场观察
|
||||
- 优先都收进 `playbook`
|
||||
|
||||
这样这套编排才能成为真正的正式平台能力,而不是一组“看起来像流程”的按钮。
|
||||
372
docs/schemas/ops_stack_diagnosis_contract.md
Normal file
372
docs/schemas/ops_stack_diagnosis_contract.md
Normal file
@@ -0,0 +1,372 @@
|
||||
# domainCheck Ops Stack Diagnosis Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结海外单脑控制面的“总检入口” contract。
|
||||
|
||||
它的职责不是替代:
|
||||
|
||||
- `overview`
|
||||
- `link-snapshot`
|
||||
- `inspection-overview`
|
||||
- `release launchpad`
|
||||
- `activity-stream`
|
||||
|
||||
而是把这些正式 contract 再收口成一份:
|
||||
|
||||
> 第一现场诊断结果
|
||||
|
||||
让下面这些消费者都先看同一份判断:
|
||||
|
||||
- 海外 Ops Center 页面
|
||||
- 海外 Codex 驾驶员
|
||||
- CLI 总检脚本
|
||||
- 后台按钮自动诊断
|
||||
|
||||
正式原则:
|
||||
|
||||
- 总检不能只回一段说明文字
|
||||
- 总检必须给出结构化问题清单
|
||||
- 总检必须给出默认下一步动作
|
||||
- 总检必须给出可以直接执行的推荐命令
|
||||
|
||||
---
|
||||
|
||||
## 2. 入口接口
|
||||
|
||||
主入口:
|
||||
|
||||
- `GET /api/v1/ops/go-live-summary`
|
||||
- `GET /api/v1/ops/stack-diagnosis`
|
||||
|
||||
配套来源:
|
||||
|
||||
- `GET /api/v1/ops/contracts`
|
||||
- `GET /api/v1/ops/link-snapshot`
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `GET /api/v1/ops/nodes`
|
||||
- `GET /api/v1/ops/releases/launchpad`
|
||||
- `GET /api/v1/ops/playbook-runs`
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
|
||||
推荐 query:
|
||||
|
||||
- `base_url`
|
||||
|
||||
用途:
|
||||
|
||||
- `go-live-summary`
|
||||
- 给 Ops Center 顶部收口卡、CLI 总检、Codex 驾驶员直接消费
|
||||
- 返回更偏“上线判断”的稳定摘要
|
||||
- 用于生成 `next_step.command`
|
||||
- 用于生成 `quick_commands`
|
||||
- 让 CLI / Codex / 页面复制出来的命令口径一致
|
||||
|
||||
---
|
||||
|
||||
## 3. 顶层结构
|
||||
|
||||
建议最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"generated_at": "2026-04-18 09:00:00",
|
||||
"diagnosis": {},
|
||||
"api": {},
|
||||
"surface_matrix": {},
|
||||
"contracts": {},
|
||||
"link_snapshot": {},
|
||||
"overview": {},
|
||||
"managed_nodes": {},
|
||||
"release_hub": {},
|
||||
"playbook_runs": {},
|
||||
"activity_stream": {}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- 顶层每一段都必须是稳定 key
|
||||
- 页面、CLI、Codex 不允许自己再拼第 2 套总检结构
|
||||
- `go-live-summary`
|
||||
- 是 `stack-diagnosis` 的稳定摘要层
|
||||
- 必须至少给出:
|
||||
- `go_live_status`
|
||||
- `blocking_reasons`
|
||||
- `warnings`
|
||||
- `launchpad_recommended_target_node_code`
|
||||
- `launchpad_recommended_recovery_label`
|
||||
- `launchpad_recommended_recovery_summary`
|
||||
- `launchpad_onboarding_bootstrap_pending_nodes`
|
||||
- `launchpad_onboarding_acceptance_ready_nodes`
|
||||
- `next_step_action_code`
|
||||
- `operator_title`
|
||||
- `recommended_commands`
|
||||
- `playbook_runs.problem_runs[*]`
|
||||
- 需要保留 `focus_ref`
|
||||
- 这样 CLI / Codex / 页面才能直接定位到问题编排的正式焦点
|
||||
- `activity_stream.top_items[*]`
|
||||
- 需要保留:
|
||||
- `occurred_at`
|
||||
- `focus_ref`
|
||||
- `source_focus_ref`
|
||||
- `ui_intent_kind`
|
||||
- 这样总检不会把“观察摘要”再次压扁成一段不可定位的纯文本
|
||||
- `go-live-summary` + `stack-diagnosis` + `driver-feed.automation_coverage`
|
||||
- 必须能共同驱动 Ops Center 首屏
|
||||
- 不允许页面自己从局部接口重新推导一份“收口状态”
|
||||
|
||||
---
|
||||
|
||||
## 4. Diagnosis Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `stack_diagnosis.diagnosis`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"contract_key": "ops_stack_diagnosis_contract",
|
||||
"contract_version": "v1",
|
||||
"registry_version": "2026-04-18",
|
||||
"base_url": "http://127.0.0.1:8100",
|
||||
"surface_status": "partial",
|
||||
"automation_status": "blocked",
|
||||
"stack_status": "blocked",
|
||||
"issue_total": 4,
|
||||
"blocking_issue_total": 1,
|
||||
"warning_issue_total": 3,
|
||||
"missing_surfaces": [
|
||||
"contracts"
|
||||
],
|
||||
"launchpad_recommended_target_node_code": "mainland-worker-01",
|
||||
"launchpad_recommended_recovery_label": "待执行接入",
|
||||
"launchpad_recommended_recovery_summary": "mainland-worker-01 还缺接入收口,先执行 bootstrap。",
|
||||
"launchpad_onboarding_bootstrap_pending_nodes": 1,
|
||||
"launchpad_onboarding_acceptance_ready_nodes": 0,
|
||||
"next_step": {
|
||||
"action_code": "bootstrap_run",
|
||||
"source": "issue:managed_nodes_agent_pending",
|
||||
"reason": "mainland-worker-01 还缺接入收口,先执行 bootstrap。",
|
||||
"command": "bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 bootstrap_run",
|
||||
"focus_ref": {
|
||||
"kind": "managed_node",
|
||||
"node_code": "mainland-worker-01"
|
||||
}
|
||||
},
|
||||
"recommended_actions": [],
|
||||
"issues": [],
|
||||
"operator_hints": [],
|
||||
"quick_commands": []
|
||||
}
|
||||
```
|
||||
|
||||
状态语义必须固定:
|
||||
|
||||
- `surface_status`
|
||||
- `healthy`
|
||||
- `partial`
|
||||
- `broken`
|
||||
- `automation_status`
|
||||
- `ready`
|
||||
- `attention`
|
||||
- `blocked`
|
||||
- `stack_status`
|
||||
- 总结论,面向人和程序
|
||||
- 推荐值为 `ready / attention / blocked`
|
||||
|
||||
正式要求:
|
||||
|
||||
- `next_step` 是默认下一步,而不是“可能动作之一”
|
||||
- 当首个缺口节点已经明确收敛为 `bootstrap_run / run_acceptance` 时,`next_step.action_code` 不应再退回成泛化的 `fix_managed_nodes`
|
||||
- 当存在 `runtime_build_schema_stale` 时,`next_step.action_code` 必须优先收敛为 `api-restart`
|
||||
- `issues` 必须按可处理性组织,而不是只堆原始异常文本
|
||||
- `quick_commands` 必须是可以直接复制执行的命令
|
||||
- `next_step.focus_ref`
|
||||
- 是首屏“定位下一步”的正式落点
|
||||
- 页面、CLI、Codex 不允许再根据 `action_code` 反推要跳去哪个区域
|
||||
- `diagnosis.launchpad_recommended_target_node_code`
|
||||
- 代表 stack-diagnosis 已经把 Release Hub 收敛出的首个接管缺口直接下沉到总检摘要
|
||||
- 页面、CLI、doctor-export、Codex 不应再分别回源二次推导
|
||||
- `diagnosis.launchpad_onboarding_bootstrap_pending_nodes / launchpad_onboarding_acceptance_ready_nodes`
|
||||
- 代表当前接管缺口的阶段性计数
|
||||
- 用来帮助总检直接区分“待 bootstrap”与“待 acceptance”
|
||||
- `go-live-summary.launchpad_recommended_target_node_code`
|
||||
- 是 launchpad 已经收敛出的首个接管缺口节点
|
||||
- 页面、CLI、Codex 应优先直接显示它,而不是重新扫描 gap rows
|
||||
- `go-live-summary.launchpad_onboarding_bootstrap_pending_nodes / launchpad_onboarding_acceptance_ready_nodes`
|
||||
- 用于区分当前是“待接入”还是“待验收”
|
||||
- 这两个计数应作为上线收口阶段的正式 warning 来源,而不是页面本地再统计
|
||||
|
||||
---
|
||||
|
||||
## 5. Issue Contract
|
||||
|
||||
`diagnosis.issues[]` 最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": "managed_nodes_agent_pending",
|
||||
"severity": "blocked",
|
||||
"layer": "managed_nodes",
|
||||
"summary": "托管节点虽然已经纳入 Ops Center,但 0 台 remote-agent 就绪,当前第一动作已收敛为首台节点接入收口。",
|
||||
"detail": "managed_enabled=3,remote_access_ready=0;首个缺口节点 mainland-worker-01:待执行接入。",
|
||||
"action_code": "bootstrap_run",
|
||||
"focus_ref": {
|
||||
"kind": "managed_node",
|
||||
"node_code": "mainland-worker-01"
|
||||
},
|
||||
"commands": [
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 bootstrap_run",
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-worker-01"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
严重级别建议固定为:
|
||||
|
||||
- `blocked`
|
||||
- `warning`
|
||||
- `info`
|
||||
|
||||
正式要求:
|
||||
|
||||
- `summary` 用于列表展示
|
||||
- `detail` 用于展开解释
|
||||
- `commands` 用于终端直接执行
|
||||
- 不允许只有中文文案,没有结构化定位信息
|
||||
|
||||
对于运行时版本漂移,还应允许出现这类 issue:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": "runtime_build_schema_stale",
|
||||
"severity": "warning",
|
||||
"layer": "runtime_build_info.schema",
|
||||
"summary": "仓库已经具备新的节点接管能力,但运行中的 build-info 仍未声明对应路由键,当前更像 API 还没重启到最新代码。",
|
||||
"detail": "supports_install_command_block=true;route_surface_declares_bootstrap_plan=false",
|
||||
"action_code": "api-restart",
|
||||
"focus_ref": {
|
||||
"kind": "runtime_build_info",
|
||||
"section": "schema",
|
||||
"expected_route_key": "ops_node_handover_bootstrap_plan"
|
||||
},
|
||||
"commands": [
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100",
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary",
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-worker-01"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Surface Matrix Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `stack_diagnosis.surface_matrix`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"status_counts": {
|
||||
"total": 8,
|
||||
"available": 7,
|
||||
"missing": 1
|
||||
},
|
||||
"items": [
|
||||
{
|
||||
"name": "contracts",
|
||||
"available": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Runtime Build Info Extension
|
||||
|
||||
`stack_diagnosis.runtime_build_info` 以及 `/api/v1/runtime/build-info` 建议至少包含:
|
||||
|
||||
```json
|
||||
{
|
||||
"repository_capabilities": {
|
||||
"supports_install_command_block": true,
|
||||
"supports_multi_layout_bootstrap": true
|
||||
},
|
||||
"route_surface_declares_bootstrap_plan": false,
|
||||
"runtime_schema_stale": true
|
||||
}
|
||||
```
|
||||
|
||||
语义要求:
|
||||
|
||||
- `repository_capabilities`
|
||||
- 表示当前仓库代码本身具备的能力
|
||||
- `route_surface_declares_bootstrap_plan`
|
||||
- 表示运行中的 API build-info 是否已经声明节点级 bootstrap-plan 路由键
|
||||
- `runtime_schema_stale`
|
||||
- 表示“仓库能力已到位,但运行中的 API 结构还停留在旧版”
|
||||
- 这是典型的“服务没重启到最新代码”信号,不应误判成“功能没开发完”
|
||||
|
||||
作用:
|
||||
|
||||
- 让页面和 CLI 直观看出“缺的是哪一层”
|
||||
- 避免只看到最终 blocked,却不知道缺的是 contract、launchpad 还是 activity-stream
|
||||
|
||||
---
|
||||
|
||||
## 7. 与其他 Contract 的关系
|
||||
|
||||
`ops_stack_diagnosis_contract` 的定位是:
|
||||
|
||||
- 不替代 `ops_driver_contract`
|
||||
- 不替代 `ops_observability_contract`
|
||||
- 不替代 `release_hub_contract`
|
||||
|
||||
它负责:
|
||||
|
||||
- 把它们统一收口成第一现场诊断
|
||||
|
||||
对于 Ops Center 页面还必须补充一条:
|
||||
|
||||
- 首屏负责“结论 + 默认下一步 + 复制命令 + 动作回执”
|
||||
- 下方详情区负责“issue / surface / launchpad / activity / runbook”的展开与下钻
|
||||
- 不允许详情区覆盖首屏的默认下一步结论
|
||||
|
||||
所以正确消费顺序应该是:
|
||||
|
||||
1. 先看 `stack-diagnosis`
|
||||
2. 再根据 `issues / next_step / focus_ref` 下钻到:
|
||||
- `overview`
|
||||
- `link-snapshot`
|
||||
- `inspection-overview`
|
||||
- `release launchpad`
|
||||
- `activity-stream`
|
||||
|
||||
---
|
||||
|
||||
## 8. CLI / Codex / 页面统一约束
|
||||
|
||||
CLI:
|
||||
|
||||
- `check_ops_center_stack.sh` 应优先调用 `GET /api/v1/ops/stack-diagnosis`
|
||||
|
||||
Codex:
|
||||
|
||||
- 默认先读取 `stack-diagnosis`
|
||||
- 再决定是继续 `driver-resolve`、`doctor` 还是细分检查
|
||||
|
||||
页面:
|
||||
|
||||
- 顶部总检卡片必须优先消费 `diagnosis`
|
||||
- 不能重新从多个 endpoint 拼一份影子状态
|
||||
|
||||
只有这样,海外单脑控制面才能真正进入:
|
||||
|
||||
> 单入口观察,分层下钻,统一执行
|
||||
609
docs/schemas/release_hub_contract.md
Normal file
609
docs/schemas/release_hub_contract.md
Normal file
@@ -0,0 +1,609 @@
|
||||
# domainCheck Release Hub Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 Release / Rollout / Launchpad / Gate 的正式 contract。
|
||||
|
||||
核心原则:
|
||||
|
||||
- 先有 Release,再有 Rollout
|
||||
- Release 是不可变版本对象
|
||||
- Rollout 是分批推进对象
|
||||
- Gate 是是否可发的统一门禁对象
|
||||
- Launchpad 是给页面与 Codex 消费的发布驾驶舱对象
|
||||
|
||||
建议与 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 配套阅读,因为 Rollout 的真正执行颗粒度最终仍然是 `ops job`。
|
||||
|
||||
---
|
||||
|
||||
## 2. Release Object
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/releases`
|
||||
- `GET /api/v1/ops/releases`
|
||||
- `GET /api/v1/ops/releases/latest`
|
||||
- `GET /api/v1/ops/releases/{id}`
|
||||
- `POST /api/v1/ops/releases/{id}/activate`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"commit_sha": "bd6bcb2",
|
||||
"status": "ready",
|
||||
"status_label": "已就绪",
|
||||
"artifact_url": "https://release.example.com/domaincheck-2026.04.18.tgz",
|
||||
"checksum": "sha256:...",
|
||||
"notes": "",
|
||||
"metadata": {},
|
||||
"created_by": "www",
|
||||
"created_at": "2026-04-18 05:30:00",
|
||||
"activated_at": "2026-04-18 05:35:00",
|
||||
"summary": "2026.04.18+bd6bcb2 已就绪,制品与校验信息齐备,可进入 Rollout。",
|
||||
"summary_text": "2026.04.18+bd6bcb2 已就绪,制品与校验信息齐备,可进入 Rollout。",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "release_detail"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
约束:
|
||||
|
||||
- 同一 `release_version` 只对应一份发布物
|
||||
- Release 创建后不应重新指向另一份 artifact
|
||||
- `artifact_url + checksum` 是发布可验证性的最小基线
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `draft`
|
||||
- `ready`
|
||||
- `active`
|
||||
- `archived`
|
||||
|
||||
---
|
||||
|
||||
## 3. Rollout Object
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/releases/{id}/rollouts`
|
||||
- `GET /api/v1/ops/rollouts/{id}`
|
||||
- `GET /api/v1/ops/rollouts/{id}/jobs`
|
||||
- `POST /api/v1/ops/rollouts/{id}/advance`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 11,
|
||||
"release_id": 1,
|
||||
"rollout_code": "rollout-20260418-001",
|
||||
"target_selector": {
|
||||
"region": "mainland",
|
||||
"role": "worker"
|
||||
},
|
||||
"target_nodes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"policy": {
|
||||
"batch_size": 1,
|
||||
"require_approval_between_batches": true,
|
||||
"rollback_on_failure": true
|
||||
},
|
||||
"status": "running",
|
||||
"status_label": "执行中",
|
||||
"batch_cursor": 1,
|
||||
"batches_total": 2,
|
||||
"jobs_total": 2,
|
||||
"jobs_created": 1,
|
||||
"result_summary": {},
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"summary": "当前已覆盖 1/2 台节点,仍有 1 条任务执行中。",
|
||||
"summary_text": "当前已覆盖 1/2 台节点,仍有 1 条任务执行中。",
|
||||
"focus_ref": {
|
||||
"kind": "release_rollout",
|
||||
"rollout_id": 11,
|
||||
"rollout_code": "rollout-20260418-001",
|
||||
"release_id": 1,
|
||||
"section": "rollout_jobs"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式定义:
|
||||
|
||||
> 某个 Release 在某批目标节点上的一次分批推进计划
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `planned`
|
||||
- `running`
|
||||
- `awaiting_approval`
|
||||
- `ready_for_next_batch`
|
||||
- `halted`
|
||||
- `completed`
|
||||
- `completed_with_issues`
|
||||
|
||||
---
|
||||
|
||||
## 4. Default Rollout Gate
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `overview.release_hub.default_rollout_gate`
|
||||
|
||||
这层是 Release Hub 最关键的后端 gate contract。
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "attention",
|
||||
"status_label": "待处理",
|
||||
"status_type": "warning",
|
||||
"summary": "当前已有 Release,但默认目标节点中仍有未接管或未通过巡检的节点。",
|
||||
"summary_text": "当前已有 Release,但默认目标节点中仍有未接管或未通过巡检的节点。",
|
||||
"release": {
|
||||
"id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"status": "ready",
|
||||
"status_label": "已就绪"
|
||||
},
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"default_target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"blocking_reasons": [],
|
||||
"warning_reasons": [
|
||||
"存在未通过标准巡检的节点"
|
||||
],
|
||||
"recommendations": [
|
||||
"先执行标准巡检,再进入 Rollout"
|
||||
],
|
||||
"operational_readiness": {
|
||||
"summary": "remote-agent 就绪 1/2,标准巡检通过 1/2",
|
||||
"rows": []
|
||||
},
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "default_rollout_gate"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `missing_release`
|
||||
- `release_not_ready`
|
||||
- `artifact_missing`
|
||||
- `no_targets`
|
||||
- `blocked`
|
||||
- `attention`
|
||||
- `ready`
|
||||
|
||||
要求:
|
||||
|
||||
- 首页驾驶建议
|
||||
- Release 页面门禁
|
||||
- Rollout 预检
|
||||
- Codex 自动驾驶
|
||||
|
||||
必须共用这同一份 Gate。
|
||||
|
||||
---
|
||||
|
||||
## 5. Default Rollout Execution
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.release_hub.default_rollout_execution`
|
||||
- `overview.release_hub.default_rollout_gate_options`
|
||||
|
||||
作用:
|
||||
|
||||
- 告诉页面 / Codex 这轮默认应使用什么执行方式
|
||||
- 给出其他可选 gate / mode
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"recommended_mode": "remote-agent",
|
||||
"recommended_mode_label": "Node Agent",
|
||||
"recommended_gate": {},
|
||||
"gates": {
|
||||
"remote-agent": {},
|
||||
"ssh": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐执行模式:
|
||||
|
||||
- `remote-agent`
|
||||
- `ssh`
|
||||
- `control-plane`
|
||||
|
||||
说明:
|
||||
|
||||
- 正式发布优先 `remote-agent`
|
||||
- `ssh` 只作为首发接管与紧急救援兜底
|
||||
- `control-plane` 不应用来承载大规模 Worker 发布
|
||||
|
||||
---
|
||||
|
||||
## 6. Release Launchpad
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.release_hub.launchpad`
|
||||
|
||||
作用:
|
||||
|
||||
- 给页面与 Codex 提供更偏“驾驶舱”的发布聚合视角
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"latest_package": {},
|
||||
"latest_release": {},
|
||||
"worker_rollout_preview": {},
|
||||
"control_rollout_preview": {},
|
||||
"launchpad_status": {
|
||||
"status": "attention",
|
||||
"status_label": "待处理",
|
||||
"summary": "建议先完成 Worker 点状灰度,再决定是否推进控制面。",
|
||||
"summary_text": "建议先完成 Worker 点状灰度,再决定是否推进控制面。",
|
||||
"recommended_action_code": "open_release_deploy_worker",
|
||||
"recommended_target_node_code": "mainland-worker-01",
|
||||
"recommended_recovery_label": "签发接入工单",
|
||||
"recommended_recovery_summary": "当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。",
|
||||
"onboarding_bootstrap_pending_nodes": 1,
|
||||
"onboarding_acceptance_ready_nodes": 0,
|
||||
"recommended_mode": "remote-agent",
|
||||
"recommended_mode_label": "Node Agent",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`launchpad_status` 至少应回答:
|
||||
|
||||
- 当前先补什么
|
||||
- 推荐动作码是什么
|
||||
- 当前首个缺口节点是谁
|
||||
- 当前恢复动作的人类标签是什么
|
||||
- 当前缺口更偏“待接入”还是“待验收”
|
||||
- 推荐执行方式是什么
|
||||
- 先看 Worker 预案还是先看 Control 预案
|
||||
|
||||
正式约束:
|
||||
|
||||
- `recommended_target_node_code`
|
||||
- 一旦 launchpad 已经收敛出首个缺口节点,就应直接返回
|
||||
- 页面、CLI、Codex 不应再分别从 gap rows 中自行挑第一台
|
||||
- `recommended_recovery_label`
|
||||
- 是给人看的动作标题,例如:
|
||||
- `签发接入工单`
|
||||
- `补执行面接管`
|
||||
- `执行接管验收`
|
||||
- `recommended_recovery_summary`
|
||||
- 是解释“为什么当前默认不是发版,而是先补接管”的正式摘要
|
||||
- `onboarding_bootstrap_pending_nodes / onboarding_acceptance_ready_nodes`
|
||||
- 是 launchpad 汇总层面的正式计数
|
||||
- 允许被 `go-live-summary`、CLI 摘要、Codex brief 直接复用
|
||||
|
||||
---
|
||||
|
||||
## 7. 发布动作 contract
|
||||
|
||||
当前发布相关动作不应再退回“通用发任务”语义,而应正式区分:
|
||||
|
||||
- `deploy.release.control`
|
||||
- `deploy.release.worker`
|
||||
- `deploy.release.custom`
|
||||
|
||||
这三个动作必须共用:
|
||||
|
||||
- 同一份 Gate 判断
|
||||
- 同一份 Release 版本对象
|
||||
- 同一份 Rollout 编排结果
|
||||
|
||||
---
|
||||
|
||||
## 8. Artifact 不可变要求
|
||||
|
||||
上线前建议把 Release 供应链固定为:
|
||||
|
||||
1. 构建产物
|
||||
2. 生成 `checksum`
|
||||
3. 生成 `manifest`
|
||||
4. 上传 artifact
|
||||
5. 创建 Release
|
||||
6. 激活 channel
|
||||
7. 创建 Rollout
|
||||
|
||||
Node Agent 侧部署链路要求:
|
||||
|
||||
1. 下载 artifact
|
||||
2. 校验 checksum
|
||||
3. 解压到 `releases/<version>`
|
||||
4. 切换 `current`
|
||||
5. 重启服务
|
||||
6. 健康检查
|
||||
7. 失败回滚
|
||||
|
||||
---
|
||||
|
||||
## 9. Rollout 与 Ops Job 的关系
|
||||
|
||||
约束:
|
||||
|
||||
- Rollout 是分批推进对象
|
||||
- Ops Job 是单次动作对象
|
||||
- Rollout 通过一批或多批 `deploy.release.*` Job 落地
|
||||
|
||||
控制面责任:
|
||||
|
||||
- 决定下一批是否生成 Job
|
||||
- 汇总 Job 结果
|
||||
- 反推 Rollout 状态
|
||||
- 决定是否暂停、继续或回滚
|
||||
|
||||
Node Agent 不负责:
|
||||
|
||||
- 推进下一批
|
||||
- 修改 Rollout 策略
|
||||
- 修改 Release 状态
|
||||
|
||||
---
|
||||
|
||||
## 10. 页面 / Codex 共同消费约束
|
||||
|
||||
以下对象必须作为稳定后端 contract 存在:
|
||||
|
||||
- `default_rollout_gate`
|
||||
- `default_rollout_execution`
|
||||
- `launchpad_status`
|
||||
- `worker_rollout_preview`
|
||||
- `control_rollout_preview`
|
||||
|
||||
页面只负责展示和少量交互兜底。
|
||||
|
||||
Codex 只负责消费后端给出的:
|
||||
|
||||
- 当前是否可发
|
||||
- 建议先发哪一侧
|
||||
- 应该使用什么 mode
|
||||
- 下一步推荐动作是什么
|
||||
|
||||
---
|
||||
|
||||
## 11. Artifact Manifest Contract
|
||||
|
||||
Release Hub 不能只停在:
|
||||
|
||||
- `artifact_url`
|
||||
- `checksum`
|
||||
|
||||
这只能证明“有个包”,还不能证明“这个包会把哪些服务改成什么样”。
|
||||
|
||||
建议把 `release.metadata.manifest` 固定为正式对象。
|
||||
|
||||
最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"artifact_kind": "tarball",
|
||||
"package_name": "domaincheck-2026.04.18+bd6bcb2.tgz",
|
||||
"package_size_bytes": 186542331,
|
||||
"checksum": "sha256:...",
|
||||
"build_id": "build-20260418-001",
|
||||
"commit_sha": "bd6bcb2",
|
||||
"built_at": "2026-04-18 05:20:00",
|
||||
"included_services": [
|
||||
"domaincheck-api",
|
||||
"domaincheck-worker",
|
||||
"domaincheck-sync-agent",
|
||||
"domaincheck-node-agent"
|
||||
],
|
||||
"required_env_keys": [
|
||||
"NODE_CODE",
|
||||
"NODE_REGION",
|
||||
"NODE_ROLE"
|
||||
],
|
||||
"health_checks": [
|
||||
{
|
||||
"name": "api_health",
|
||||
"type": "http",
|
||||
"url": "http://127.0.0.1:8100/health"
|
||||
}
|
||||
],
|
||||
"rollback_hint": "切回上一个 current 并重启对应服务"
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- Release 创建后,`manifest` 不应再被覆盖成另一份语义不同的发布物
|
||||
- Node Agent、页面、Codex 必须共用这同一份 manifest
|
||||
- 是否可发、怎么回滚、要验哪些服务,不应再散落在:
|
||||
- shell 脚本
|
||||
- 页面表单
|
||||
- 人工脑补
|
||||
|
||||
---
|
||||
|
||||
## 12. Operational Readiness Row Contract
|
||||
|
||||
`default_rollout_gate.operational_readiness.rows[]` 不应只是“几行提醒文案”,而应成为逐节点门禁对象。
|
||||
|
||||
最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"agent_managed": true,
|
||||
"execution_mode_ready": true,
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"inspection_status": "healthy",
|
||||
"inspection_status_label": "已通过",
|
||||
"current_release_version": "2026.04.11+1921318",
|
||||
"desired_release_version": "2026.04.18+bd6bcb2",
|
||||
"risk_level": "low",
|
||||
"blocking_reasons": [],
|
||||
"warning_reasons": [],
|
||||
"recommended_action_code": "",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "default_rollout_gate"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- `rows[]` 应能直接回答:
|
||||
- 这台节点能不能进下一轮 rollout
|
||||
- 不能的话卡在哪
|
||||
- 只是 warning 还是 hard block
|
||||
- 页面、CLI、Codex 不应再从:
|
||||
- inspection
|
||||
- runtime
|
||||
- nodes
|
||||
三处数据手工拼门禁结论
|
||||
|
||||
推荐风险分级:
|
||||
|
||||
- `low`
|
||||
- `medium`
|
||||
- `high`
|
||||
- `blocked`
|
||||
|
||||
---
|
||||
|
||||
## 13. Rollout Preview Contract
|
||||
|
||||
真正创建 Rollout 之前,还应保留一层正式 preview,而不是由页面自己预估批次和风险。
|
||||
|
||||
建议入口:
|
||||
|
||||
- `POST /api/v1/ops/releases/{id}/rollouts/preview`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"release_id": 1,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"policy": {
|
||||
"batch_size": 1,
|
||||
"require_approval_between_batches": true,
|
||||
"rollback_on_failure": true
|
||||
},
|
||||
"execution_mode": "remote-agent"
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"release": {},
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"target_nodes_total": 2,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"expected_batches_total": 2,
|
||||
"expected_jobs_total": 2,
|
||||
"gate": {},
|
||||
"batch_plan": [
|
||||
{
|
||||
"batch_no": 1,
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"risk_level": "low"
|
||||
},
|
||||
{
|
||||
"batch_no": 2,
|
||||
"node_codes": [
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"risk_level": "medium"
|
||||
}
|
||||
],
|
||||
"summary": "建议先灰度 1 台 Worker,再推进 Controller。",
|
||||
"summary_text": "建议先灰度 1 台 Worker,再推进 Controller。",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "rollout_preview"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- preview 是:
|
||||
- 页面发 rollout 前的统一预检
|
||||
- Codex 自动驾驶发 rollout 前的统一预检
|
||||
- CLI 批量发布前的统一预检
|
||||
- 不允许前端重新自己算:
|
||||
- 该分几批
|
||||
- 哪台应该先发
|
||||
- 哪台属于高风险
|
||||
|
||||
只有这样,Release Hub 才是真正的发布控制面,而不是“能看版本的页面”。
|
||||
Reference in New Issue
Block a user