feat: add ops center and node onboarding flow

This commit is contained in:
Your Name
2026-04-18 23:52:51 +08:00
parent 246838ae4c
commit b9c29481b5
142 changed files with 89727 additions and 186 deletions

98
HANDOFF_20260418_2321.md Normal file
View File

@@ -0,0 +1,98 @@
# HANDOFF 2026-04-18 23:21
## 当前最高优先级
大陆节点接管闭环最小路径
原因:
- 当前控制面能力已基本足够
- 当前最大阻塞不是新功能,而是大陆两台还停在:
- `pending_bootstrap`
- `agent_pending`
- `SSH 未配置`
- 只要这条链路不闭合,项目就仍然停留在“主控骨架可用,但统一接管未落地”的状态
## 推荐模型与推理等级
推荐:
- `GPT-5.4`
- `high`
原因:
- 当前阶段重心是优先级判断、闭环收口、执行顺序压缩
- 需要稳定判断力,不适合降到偏快但更容易发散的配置
- 当前阶段也暂时不需要切到 `xhigh`
## 下一轮如果继续,第一步先做什么
下一轮必须先做:
1. 录入大陆两台节点的 SSH 信息
2. 复查 `nodes`
3. 逐台跑 `node-handover`
4. 逐台跑 `node-bootstrap-plan`
建议顺序:
- `mainland-controller-01`
- `mainland-worker-01`
推荐第一组动作:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bind-ssh http://127.0.0.1:8100 mainland-controller-01 <ssh_host> <ssh_user> 22
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bind-ssh http://127.0.0.1:8100 mainland-worker-01 <ssh_host> <ssh_user> 22
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
bash domain-api/deploy/multi-region/drive_ops_center.sh node-handover http://127.0.0.1:8100 mainland-controller-01
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-controller-01
```
## 当前不要做的事情
现在不要做:
- 扩写新的架构设计文档
- 新增页面
- 新增模块
- 继续增强控制面专题能力
- 扩大 Release Hub 范围
- 扩大 runbook / playbook 范围
- 做与大陆节点接管闭环无直接关系的功能开发
## 当前阶段边界
本次已完成:
- 阶段校准
- 闭环优先级重排
- 任务板初始化
- 状态板初始化
- handoff 生成
本次未进入:
- 任何新实现
- 任何新页面或新模块
- 任何新的控制面增强项
## 下一轮完成主批次后的预期
如果大陆两台顺利完成:
- SSH 纳管
- bootstrap
- acceptance
- `remote_access_ready=3/3`
则项目总体进度预计进入:
- `60%~65%` 的稳定上线准备区间
那时下一阶段才应该切到:
- 发布前总检
- 正式收口
- 上线证据导出

155
HANDOFF_20260418_2344.md Normal file
View File

@@ -0,0 +1,155 @@
# HANDOFF 2026-04-18 23:44
## 当前最高优先级
仍然是:大陆节点接管闭环最小路径
但当前阶段已经从“补 SSH 入口”切到:
- 修正大陆节点环境内容
- 让 bootstrap 真正可以落地执行
## 今晚已完成的步骤
已完成:
1. 录入 `mainland-controller-01` SSH 信息
2. 录入 `mainland-worker-01` SSH 信息
3. 复查 `ops/nodes`
4. 逐台执行 `node-handover`
5. 逐台生成并复核 `node-bootstrap-plan`
6. 复核:
- `go-live-summary`
- `ops/nodes`
- `release-launchpad`
7. 手工验证 controller SSH 登录可达
## 当前结果
当前指标:
- `ssh_ready = 2`
- `remote_access_ready = 3`
- 是否达到 `remote_access_ready = 3/3`:是
但 A1 仍未闭环,因为:
- `mainland-controller-01` 仍是 `pending_bootstrap`
- `mainland-worker-01` 仍是 `pending_bootstrap`
- 两台都还没有完成:
- `bootstrap`
- `register / heartbeat`
- `acceptance`
## 两台节点推进到哪一步
### mainland-controller-01
已推进到:
- SSH 已录入
- `node-handover` 已完成
- `node-bootstrap-plan` 已生成
- 手工 SSH 登录成功
- 尝试执行 bootstrap 片段
准确卡点:
- 远端缺少 node-agent 所需文件
- bootstrap 执行时报错:
- `bootstrap_node_agent.sh not found under known layouts`
- `Unit domaincheck-node-agent.service could not be found`
阻塞分类:
- 主阻塞:环境问题
- 次阻塞:权限 / 外部条件问题
- 当前控制面自动 SSH 执行器默认走密钥模式
- 今晚验证可用的是密码 SSH
### mainland-worker-01
已推进到:
- SSH 已录入
- `node-handover` 已完成
- `node-bootstrap-plan` 已生成
准确卡点:
- 与 controller 同类
- 远端缺少:
- `domain-api/app/node_agent.py`
- `domain-api/app/api/routes/ops_agent.py`
- `domain-api/app/services/ops_agent_service.py`
- `domain-api/deploy/systemd/domain-node-agent.service`
- `domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example`
阻塞分类:
- 主阻塞:环境问题
## 收口结论
今晚属于“有明显进展,但 A1 未闭环”的收口结果。
明确进展:
-`remote_access_ready = 1/3` 推进到了 `3/3`
- 两台大陆节点都已进入 `ssh_ready`
- handover 与 bootstrap plan 链路都已打通
明确未完成:
- 两台大陆节点仍未完成 bootstrap
- 因此还没有 register / heartbeat
- 也没有进入 acceptance
## 下一轮如果继续,第一步先做什么
下一轮先做的不是新功能,而是修正大陆节点环境。
最小动作:
1. 确认大陆两台代码目录由 `www` 用户更新到包含 node-agent 文件的最新版本
2. 确认以下文件在两台大陆节点都存在:
- `domain-api/app/node_agent.py`
- `domain-api/app/api/routes/ops_agent.py`
- `domain-api/app/services/ops_agent_service.py`
- `domain-api/deploy/systemd/domain-node-agent.service`
- `domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example`
3. 然后重新执行:
- `node-bootstrap-plan`
- `bootstrap`
- `acceptance`
## 推荐模型与推理等级
继续推荐:
- `GPT-5.4`
- `high`
## 现在不要做的事情
不要做:
- 新页面
- 新模块
- 控制面增强项
- 新架构文档扩写
- Candidate B2
- 为了解决今晚问题而扩展新的运维功能
## 明天需要你补充或操作的内容
你明天需要安排的最小动作是:
- 让大陆两台节点的仓库内容更新到当前控制面所需版本
- 并且注意:
- git / 代码更新应继续使用 `www`
- 不要用 `root` 跑 git
## 当前一句话状态
今晚已经把大陆两台从“未纳管 SSH”推进到“SSH 可执行、bootstrap plan 可生成”,但两台都卡在相同的环境缺口上:远端仓库缺 node-agent 运行文件,因此 A1 还差最后的 bootstrap / register / acceptance 收口。

135
IMPLEMENTATION_STATUS.md Normal file
View File

@@ -0,0 +1,135 @@
# IMPLEMENTATION_STATUS
更新时间2026-04-18 23:44 CST
## 当前真实状态
阶段判断:
- 方案与控制面能力建设:约 `78%`
- 真正达到“可稳定上线、海外单脑统一接管大陆节点”:约 `58%`
当前已成立的事实:
- Ops Center 主控骨架已基本形成
- Release Hub 已具备主流程判断能力
- Node Agent 协议、日志回传、runbook / playbook、节点接管视图已具备基础可用性
- `drive_ops_center.sh` 已补齐:
- `nodes`
- `node-bind-ssh`
当前最新节点状态:
- `managed_enabled = 3`
- `agent_ready = 1`
- `ssh_ready = 2`
- `remote_access_ready = 3`
- `pending_bootstrap = 2`
节点明细:
- `overseas-control-01`
- `agent_ready`
- `remote_access_ready = true`
- `mainland-controller-01`
- `pending_bootstrap`
- `remote_access_state = ssh_ready`
- `SSH 已录入`
- `node-handover` 已完成
- `node-bootstrap-plan` 已生成
- 手工 SSH 登录已验证可达
- `mainland-worker-01`
- `pending_bootstrap`
- `remote_access_state = ssh_ready`
- `SSH 已录入`
- `node-handover` 已完成
- `node-bootstrap-plan` 已生成
## 代码闭环
已完成或基本具备:
- 统一查看节点接管状态
- 统一生成与查看 handover / bootstrap 入口
- 统一查看 go-live summary
- 统一查看日志回传覆盖情况
- 统一为托管节点录入 SSH 入口的 CLI 能力
- controller 与 worker 的 bootstrap plan 均可生成
当前代码侧已验证的事实:
- 当前主控链路可以把大陆节点从 `agent_pending` 推到 `ssh_ready`
- `node-handover``node-bootstrap-plan` 输出一致
- `go-live-summary` 已看到 `remote_access_ready = 3`
结论:
- 当前代码侧已经可以支撑“接管闭环”的执行顺序
- 当前主阻塞已不再是“还缺哪块控制面功能”
- 但当前 SSH 自动执行链仍默认走密钥模式,和今晚实际提供的密码登录方式不一致
## 外部条件闭环
当前仍未闭合的条件:
- 大陆两台虽然已录入 SSH但远端仓库未具备 node-agent 所需文件
- 尚未实际触发大陆两台的 bootstrap 执行
- 尚未建立两台大陆节点的 register / heartbeat 闭环
- 尚未完成两台大陆节点的 acceptance 闭环
结论:
- 当前最关键的剩余问题不是新代码,而是大陆节点环境与当前控制面能力不匹配
- 今晚已经不是“SSH 信息缺失”阶段,而是“大陆节点代码/部署内容缺失”阶段
## 明确依赖用户提供的信息
今晚已经提供并验证:
- `mainland-controller-01`
- `ssh_host=121.204.244.188`
- `ssh_user=root`
- 手工 SSH 登录成功
- `mainland-worker-01`
- `ssh_host=121.204.244.248`
- `ssh_user=root`
当前仍需要你明天确认或操作的,不再是 SSH 信息,而是以下环境修正路径二选一:
1. 让大陆两台节点仓库更新到包含 node-agent 文件的最新版本
2. 或把 node-agent 相关文件以正式发布包 / 正式部署方式补齐到大陆两台
在上述环境未修正前,以下链路不能真正闭合:
- `bootstrap execute`
- `register / heartbeat`
- `acceptance`
- `pending_bootstrap -> 已接管`
## 当前优先级判断
最高优先级:
- 大陆节点接管闭环最小路径
当前不应继续优先推进:
- 控制面横向增强
- 额外专题设计
- 与接管闭环无直接关系的发布能力细化
- 新的页面或模块
本轮新增阻塞分类结论:
- controller 阻塞:环境问题
- worker 阻塞:环境问题
- 自动 SSH 执行器限制:权限 / 外部条件问题
- 现链路仅按 `auth_mode=key` 工作
- 今晚实际验证可用的是密码 SSH
## 完成主批次后的预期
如果大陆两台环境补齐,并按最小闭环顺序完成:
- 项目整体可稳定上线进度预计提升到 `60%~65%`
- 一旦 `remote_access_ready=3/3` 成立,后续就从“架构推进期”切换到“发布收口期”

View File

@@ -6,13 +6,24 @@
- 新版后端服务 `domain-api/`
- 新版 Web 管理后台 `domain-web/`
- 全套交付与部署文档 `docs/`
- Windows 本地启动、打包、验包脚本
- Windows / Linux 启动、打包、验包脚本
如果你是第一次打开这个仓库,建议先看:
- [docs/12_domainCheck_导航索引.md](./docs/12_domainCheck_导航索引.md)
- [docs/04_domainCheck_WebLinux改造总体方案.md](./docs/04_domainCheck_WebLinux改造总体方案.md)
如果你当前已经进入“海外单脑控制面 / 多机联调 / 准备上线”阶段,建议直接从下面 4 份开始:
1. [docs/25_domainCheck_海外单脑控制面上线收口总表.md](./docs/25_domainCheck_海外单脑控制面上线收口总表.md)
2. [docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md](./docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md)
3. [docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md](./docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md)
4. [docs/schemas/ops_driver_contract.md](./docs/schemas/ops_driver_contract.md)
如果已经接近正式发布,建议再补一份:
- [docs/26_domainCheck_发布前运行验证与交付模板.md](./docs/26_domainCheck_发布前运行验证与交付模板.md)
## 目录结构
```text
@@ -25,10 +36,14 @@
├─ start_*.ps1 Windows 启动脚本
├─ stop_*.ps1 Windows 停止脚本
├─ smoke_test_stack.ps1 本地整栈自测
├─ package_domain_release.ps1 交付打包
├─ verify_domain_release.ps1 交付包校验
├─ prepare_final_release.ps1 最终发布准备
─ show_latest_release.ps1 查看最新交付包
├─ package_domain_release.ps1 Windows 交付打包
├─ verify_domain_release.ps1 Windows 交付包校验
├─ prepare_final_release.ps1 Windows 最终发布准备
─ show_latest_release.ps1 Windows 查看最新交付包
├─ package_domain_release.sh Linux / 海外主机交付打包
├─ verify_domain_release.sh Linux / 海外主机交付包校验
├─ prepare_final_release.sh Linux / 海外主机最终发布准备
└─ show_latest_release.sh Linux / 海外主机查看最新交付包
```
## 各目录用途
@@ -118,6 +133,15 @@
- [verify_domain_release.ps1](./verify_domain_release.ps1)
- [prepare_final_release.ps1](./prepare_final_release.ps1)
- [show_latest_release.ps1](./show_latest_release.ps1)
- [package_domain_release.sh](./package_domain_release.sh)
- [verify_domain_release.sh](./verify_domain_release.sh)
- [prepare_final_release.sh](./prepare_final_release.sh)
- [show_latest_release.sh](./show_latest_release.sh)
补充说明:
- `show_latest_release.ps1 / .sh` 现在会校验 `final_release_report.json` 是否真的对应当前 `latest_release.*`
- 不是“目录里有 final report 就算已经完成最终签收”
## 当前状态
@@ -142,6 +166,69 @@ Linux 阶段建议先看:
- [docs/11_domainCheck_Linux联调输入清单.md](./docs/11_domainCheck_Linux联调输入清单.md)
- [docs/12_domainCheck_导航索引.md](./docs/12_domainCheck_导航索引.md)
如果当前已经切到“海外单脑控制面 + 大陆托管节点 + 发布 / Rollout / 驾驶舱首屏”阶段,建议改为:
1. [docs/25_domainCheck_海外单脑控制面上线收口总表.md](./docs/25_domainCheck_海外单脑控制面上线收口总表.md)
2. [docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md](./docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md)
3. [docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md](./docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md)
4. [docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md](./docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md)
5. [docs/schemas/ops_stack_diagnosis_contract.md](./docs/schemas/ops_stack_diagnosis_contract.md)
6. [docs/schemas/ops_driver_contract.md](./docs/schemas/ops_driver_contract.md)
如果当前已经进入“海外主机集中运维 / 多机联调 / Release Hub”阶段最直接的统一入口是
- [domain-api/deploy/multi-region/drive_ops_center.sh](./domain-api/deploy/multi-region/drive_ops_center.sh)
- [domain-api/deploy/multi-region/drive_release_hub.sh](./domain-api/deploy/multi-region/drive_release_hub.sh)
- [domain-api/deploy/multi-region/drive_ops_action.sh](./domain-api/deploy/multi-region/drive_ops_action.sh)
- [domain-api/deploy/multi-region/init_ops_center_config.sh](./domain-api/deploy/multi-region/init_ops_center_config.sh)
其中:
- `drive_ops_center.sh`
用于海外控制面统一调度检查、Release Hub 和 Driver 动作
- `drive_release_hub.sh`
用于直接从命令行走 Release / Rollout 标准接口
- `drive_ops_action.sh`
用于直接从命令行走 `ops/driver-actions/execute`
- `init_ops_center_config.sh`
用于初始化 `/etc/default/domaincheck-ops-center`
现在 `drive_ops_center.sh` 还额外收进了本机发布链:
- `release-package`
- `release-verify`
- `release-show`
- `release-launchpad`
- `release-prepare`
- `release-preview`
- `release-preview-smart`
当前如果要用一条最短路径判断“现在能不能继续收口、能不能继续发布”,推荐先跑:
```bash
cd /www/wwwroot/getDomain
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
```
然后再按:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
```
继续下钻。
如果要把这轮检查直接导出成一整包上线证据,也可以直接执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
```
它会把 `go-live / stack-diagnosis / driver-feed / codex-brief / release-launchpad / doctor-*`
统一导出到 `domain-api/runtime/ops-center-reports/go-live-bundles/` 下,并生成一份 `manifest.json`
## Git 提交建议
建议提交:

148
TASK_BOARD.md Normal file
View File

@@ -0,0 +1,148 @@
# TASK_BOARD
更新时间2026-04-18 23:44 CST
## 当前主批次
唯一主批次:大陆节点接管闭环最小路径
目标:
- 把当前项目的最高优先级从“继续扩控制面能力”切到“完成大陆节点接管闭环”
- 只围绕这条最短路径推进:
- 录入大陆节点 SSH 入口
- 生成并复核 bootstrap plan
- 执行 bootstrap
- 建立 register / heartbeat
- 执行 acceptance
-`remote_access_ready``1/3` 提升到 `3/3`
主批次成功标准:
- `mainland-controller-01` 不再是 `pending_bootstrap`
- `mainland-worker-01` 不再是 `pending_bootstrap`
- `ssh_ready >= 2`
- `remote_access_ready = 3/3`
- 大陆两台都具备可由海外控制面统一发起标准动作的条件
当前完成度:
- 已完成 `SSH` 入口录入
- 已完成 `node-handover`
- 已完成 `node-bootstrap-plan`
- 已确认 `ssh_ready = 2`
- 已确认 `remote_access_ready = 3/3`
- 尚未完成:
- `bootstrap`
- `register / heartbeat`
- `acceptance`
- `pending_bootstrap -> 已接管`
## 主批次拆解
### Batch A1
名称:大陆节点接管闭环最小路径
边界:
- 只处理大陆两台:
- `mainland-controller-01`
- `mainland-worker-01`
- 只处理接管闭环,不扩展控制面功能
任务:
- 已完成:
- 录入 `mainland-controller-01` SSH 信息
- 录入 `mainland-worker-01` SSH 信息
- 复查 `ops/nodes`
- 逐台执行 `node-handover`
- 逐台生成并复核 `node-bootstrap-plan`
- 复核:
- `go-live-summary`
- `ops/nodes`
- `release-launchpad`
- 待完成:
- 执行 controller 的 `bootstrap -> register -> heartbeat`
- 执行 worker 的 `bootstrap -> register -> heartbeat`
- 对两台执行 `acceptance`
- 让两台都退出 `pending_bootstrap`
阻塞:
- controller 与 worker 两台大陆节点都缺少 node-agent 相关运行文件
- 远端实际缺失:
- `domain-api/app/node_agent.py`
- `domain-api/app/api/routes/ops_agent.py`
- `domain-api/app/services/ops_agent_service.py`
- `domain-api/deploy/systemd/domain-node-agent.service`
- `domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example`
- 因此 bootstrap 片段虽然已生成,但远端执行时无法完成安装与启动
阻塞分类:
- 主阻塞:环境问题
- 大陆节点上的仓库内容落后于当前控制面所需的 node-agent 文件集
- 次阻塞:权限 / 外部条件问题
- 当前 SSH 自动执行链默认走密钥模式
- 今晚实际可用的是密码 SSH
## 候选批次
### Candidate B1
名称:接管闭环完成后的上线收口
前提:
- 仅在 `remote_access_ready = 3/3` 后进入
范围:
- 重新跑 `go-live-check / stack-diagnosis / release-launchpad`
- 收口发布前证据
- 判断是否达到正式上线窗口
### Candidate B2
名称:接管闭环完成后的值班自动化细化
前提:
- 仅在大陆两台已进入稳定接管状态后进入
范围:
- 优化运维体验
- 优化日常值班命令
- 优化交接与证据导出
当前不进入:
- `Candidate B1`
- `Candidate B2`
原因:
- 当前虽然 `remote_access_ready = 3/3`
- 但大陆两台仍停在 `pending_bootstrap`
- A1 还没有真正闭环
## 暂停项
以下事项当前不继续推进:
- 新的控制面专题设计
- 新页面、新模块、新专题文档扩写
- 与大陆节点接管闭环无直接关系的 Release Hub 增强
- 与大陆节点接管闭环无直接关系的 runbook / playbook 扩展
- 更细粒度日志分析功能
- 额外的驾驶视图、美化项、信息聚合项
- 任何不能直接推动 `bootstrap -> acceptance -> remote_access_ready=3/3` 的功能开发
## 当前判断
当前项目不缺“继续横向扩能力”的理由,缺的是“把最后两台大陆节点真正纳入闭环”的收口动作。
因此下一轮只围绕主批次 A1 推进。

View File

@@ -120,3 +120,10 @@ python deploy/linux/smoke_test.py --base-url http://127.0.0.1:8100
- Linux 实机联调
- 上线前复测
- 真实服务器灰度验证
如果当前已经进入“海外单脑控制面 + 多机联调 + 准备发布”阶段,建议改为按下面顺序执行:
1. `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
2. `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export`
3. `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review`
4. `docs/26_domainCheck_发布前运行验证与交付模板.md`

View File

@@ -2,7 +2,7 @@
## 一、用途
用于在 Windows 本地把当前可交付内容整理成一份压缩包,便于:
用于在 Windows 或 Linux 上把当前可交付内容整理成一份发布包,便于:
- 发给运维或部署同事
- 存档版本快照
@@ -16,31 +16,60 @@
- `verify_domain_release.ps1`
- `prepare_final_release.ps1`
- `show_latest_release.ps1`
- `package_domain_release.sh`
- `verify_domain_release.sh`
- `prepare_final_release.sh`
- `show_latest_release.sh`
执行方式:
Windows 执行方式:
```powershell
powershell -ExecutionPolicy Bypass -File .\package_domain_release.ps1
```
Linux / 海外主机执行方式:
```bash
bash ./package_domain_release.sh
```
如需一键完成“打包 + 验包 + 生成最终准备报告”:
```powershell
powershell -ExecutionPolicy Bypass -File .\prepare_final_release.ps1
```
```bash
bash ./prepare_final_release.sh
```
如需快速查看当前最新交付物:
```powershell
powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
```
```bash
bash ./show_latest_release.sh
```
注意:
- `show_latest_release.*` 现在不只是“显示最近一次 final report 是否存在”
- 它会额外校验 `final_release_report.json` 是否真的对应当前 `latest_release.*`
- 只有 `final_release_report_matches_latest=true` 时,`final_release_ok=true` 才有意义
- `final_release_gate_decision / blocked_reasons / verify_ok / smoke_test_ok` 用于解释“为什么当前能签 / 不能签”
- Ops Center 里的“直接创建 Release / 一键 Worker 灰度 / 一键 Control 发布”
- 现在会硬性依赖 `final_release_ok=true`
- 如果 final report 缺失、过期或门禁未 ready前端会禁用按钮后端接口也会拒绝执行
## 三、输出位置
打包后会生成:
- `release/domaincheck_release_时间戳/`
- `release/domaincheck_release_时间戳.zip`
- `release/domaincheck_release_时间戳.tar.gz`
- `release/domaincheck_release_时间戳.sha256.txt`
- `release/latest_release.txt`
- `release/latest_release.json`
@@ -50,6 +79,7 @@ powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
- `docs/`
- `scripts/`
- `scripts/` 中会同时带上 PowerShell 和 Shell 版发布脚本
- `domain-api/`
- `app`
- `deploy`
@@ -89,6 +119,10 @@ powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
powershell -ExecutionPolicy Bypass -File .\verify_domain_release.ps1
```
```bash
bash ./verify_domain_release.sh
```
5. Windows 联调先跑 `scripts/smoke_test_stack.ps1`
6. Linux 部署前阅读:
- `docs/05_domainCheck_Linux部署清单.md`
@@ -106,6 +140,18 @@ powershell -ExecutionPolicy Bypass -File .\verify_domain_release.ps1
这样更适合交付、存档和迁移。
Windows 默认产物为 `.zip`Linux 默认产物为 `.tar.gz`;两者都会生成同结构的 `release_manifest.json / .sha256.txt / latest_release.*`,便于海外控制面统一管理。
`latest_release.txt / latest_release.json` 用于快速定位“当前最新一份交付包”,避免人工翻目录。
若打包目录本身是 Git 仓库,脚本还会自动写入 `commit_sha / commit_ref`;海外控制面在创建 Release 时可以直接回填提交号,减少人工抄写出错。
`final_release_report.json` 用于记录最近一次“打包 + 验包”的最终状态。
`show_latest_release.*` 读取这个报告时,还会校验:
- `package_name`
- `archive_path` / `zip_path`
- `sha256`
也就是说,即使目录里留着一份旧的 `final_release_report.json`,只要它不是给当前最新包签出的,`final_release_ok` 也不会误报为 `true`

View File

@@ -65,9 +65,11 @@
1. 把最新交付包发送到目标 Linux 服务器
2.`docs/05``docs/13``docs/14` 的顺序执行部署、核对与收口
3. 部署完成后再做一次正式服务态 `smoke test`
4. 导出一份最终诊断包
5. 进入灰度上线
3. 如果当前已经进入海外单脑控制面阶段,再按 `docs/25` 执行统一收口
4. `docs/26` 完成发布前运行验证、证据导出与最终交付结论
5. 部署完成后再做一次正式服务态 `smoke test`
6. 导出一份最终诊断包
7. 进入灰度上线
## 六、最终判断

View File

@@ -40,6 +40,51 @@
- `docs/20_domainCheck_临时海外控制面联调清单.md`
### 10. 海外主机集中运维与自动化方案
- `docs/21_domainCheck_海外主机集中运维与自动化方案.md`
### 11. 海外控制面统一入口
- `domain-api/deploy/multi-region/drive_ops_center.sh`
- `domain-api/deploy/multi-region/export_go_live_bundle.sh`
- `domain-api/deploy/multi-region/check_env_drift.sh`
- `domain-api/deploy/multi-region/drive_release_hub.sh`
- `domain-api/deploy/multi-region/drive_ops_action.sh`
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
- `domain-api/deploy/multi-region/check_ops_center_stack.sh`
- `domain-api/deploy/multi-region/check_ops_contracts.sh`
### 12. 海外 Codex 驾驶员与 Ops Center 落地路线
- `docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md`
### 13. 终局运维架构设计:海外单脑控制面
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
### 14. Node Agent 协议与 Release Hub 设计
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
### 15. 运维 Contract Schema
- `docs/schemas/ops_agent_protocol.md`
- `docs/schemas/release_hub_contract.md`
- `docs/schemas/ops_driver_contract.md`
- `docs/schemas/ops_job_contract.md`
- `docs/schemas/ops_playbook_contract.md`
- `docs/schemas/ops_observability_contract.md`
- `docs/schemas/ops_stack_diagnosis_contract.md`
### 16. 海外单脑控制面上线收口总表
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
### 17. 发布前运行验证与交付模板
- `docs/26_domainCheck_发布前运行验证与交付模板.md`
## 二、文档阅读顺序
### 1. 先看总体方案
@@ -58,6 +103,19 @@
- `docs/17_domainCheck_全流程部署实操手册.md`
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
- `docs/20_domainCheck_临时海外控制面联调清单.md`
- `docs/21_domainCheck_海外主机集中运维与自动化方案.md`
- `docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md`
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
- `docs/26_domainCheck_发布前运行验证与交付模板.md`
- `docs/schemas/ops_agent_protocol.md`
- `docs/schemas/release_hub_contract.md`
- `docs/schemas/ops_driver_contract.md`
- `docs/schemas/ops_job_contract.md`
- `docs/schemas/ops_playbook_contract.md`
- `docs/schemas/ops_observability_contract.md`
- `docs/schemas/ops_stack_diagnosis_contract.md`
### 3. 如果要回溯历史需求与问题
@@ -89,6 +147,11 @@
- `domain-api/deploy/systemd/domain-worker.service`
- `domain-api/deploy/linux/README.md`
- `domain-api/deploy/multi-region/README.md`
- `domain-api/deploy/multi-region/drive_ops_center.sh`
- `domain-api/deploy/multi-region/drive_release_hub.sh`
- `domain-api/deploy/multi-region/drive_ops_action.sh`
- `domain-api/deploy/multi-region/check_ops_contracts.sh`
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
- `domain-api/deploy/multi-region/bootstrap_overseas.sh`
- `domain-api/deploy/multi-region/bootstrap_mainland.sh`
- `domain-api/deploy/linux/smoke_test.py`
@@ -109,6 +172,8 @@
- 配置迁移与备份
- 打包与验包
- 最终发布准备
- 海外单脑控制面协议收口
- Ops Center / Codex / CLI 驾驶 contract 收口
剩余工作只在真实 Linux 环境:

View File

@@ -31,6 +31,12 @@
- 海外控制面 `runtime/sync-summary`
- 能看到 `detect_result_batches`
如果当前已经切到“海外单脑控制面统一驾驶”模式,建议同时配套阅读:
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
## 二、上线前必须确认的结论
上线前至少要确认下面这些结论同时成立:
@@ -46,6 +52,34 @@
## 三、正式上线前执行顺序
### 0. 先跑一遍统一上线总检
如果当前已经进入“海外单脑控制面 + 多地域 / Node Agent”这条正式链路建议先跑
```bash
cd /www/wwwroot/getDomain/domain-api
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://152.53.37.118:8100 summary
```
期望:
- `go_live_status``ready` 或至少没有新的 `blocking_reasons`
- `stack_status` 不为 `blocked`
- `route_surface_complete=true`
- `launchpad_status` 不为 `blocked`
- 如果当前已经纳管执行节点,则 `remote_access_ready` 不应长期为 `0`
- `next_step_action_code` / `operator_title` 应能直接说明当前第一优先动作
- 如果 `log_sync_enabled=true`,则要额外看 `log_sync_state``log_sync_covered_nodes`
如果这里已经出现 `bootstrap_run / run_acceptance`,说明当前上线前仍有节点接管缺口,应先收口首个缺口节点,再继续下面的业务核验。
建议这样理解总检输出:
- `blocking_reasons` 只放真正会阻断上线的项,例如 `remote_agent_ready=0`
- `warnings` 表示还能继续联调,但现在还不适合直接宣称“全部收口”
- `recommended_commands.next_step` 就是当前最值得优先执行的一条命令
- `recommended_commands.log_sync_logs` / `recommended_commands.log_sync_inspection` 用来补远端参与节点的日志观测
### 1. 核对服务状态
```bash

View File

@@ -165,10 +165,20 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
- `detect_worker_nodes`
- 看谁在线
- 看谁是 `worker_online=true`
- 看谁 `detect_participating=true`
- `detect_participating=true` 只能说明它曾被判断为“正在参与”
- `detect_job_items`
- 看任务到底被哪台节点 `claimed_by`
- 这才是“当前真正干活的节点”
- `runtime/status`
-`participation_summary.dispatch_active_nodes`
- 这是“当前真正执行/领任务的节点”
-`participation_summary.non_participating_nodes`
- 这是“在线但未参与的节点”
-`non_participating_nodes[].participation_state`
- `standby` 表示在线待命
- `load_syncing` 表示负载待确认,不要直接算成“正在干活”
-`log_sync`
- 能直接确认远端日志回传是否开启、是关键还是全量、样本来自哪些节点
## 8. 常见误判
@@ -182,6 +192,7 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
- “有效执行节点”是可承担任务的在线节点数
- “当前参与检测节点”是当前真的在领任务、跑任务的节点
- “在线但未参与节点”是已经在线、可承接任务,但当前这轮还没分到任务的节点
如果 controller 没兼跑检测,通常只会看到独立 worker 在真正执行。
@@ -189,6 +200,26 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
这不一定是 bug可能只是当前调度没有分到它。应以 `detect_job_items.claimed_by` 为准,不要只看服务在线。
### 8.4 页面显示“负载待确认”
这不是新的故障状态,而是为了避免误判:
- 节点已经上报 `busy` 或有 `current_load`
- 但当前还没看到明确的 `items_claimed / items_running / processed_recent`
通常是心跳和任务快照还没完全对齐。先等下一轮刷新,再结合 `claimed_by``participation_summary` 判断,不要立刻当作“这台机器已经在跑检测”。
### 8.5 日志控制台看不到大陆节点过程
先看 `runtime/status` 或页面里的“远端日志回传”:
-`enabled=false`
- 说明本来就没开回传
-`enabled=true``line_count=0`
- 说明开关已开,但当前还没有远端样本
-`source_nodes` 里没有目标大陆节点
- 说明该节点这轮还没回传日志,先看它是否真的在参与检测
## 9. 联调完成后的回滚
如果你后续要切回正式海外机器,在大陆 controller 上把目标地址改回正式值即可:

View File

@@ -0,0 +1,708 @@
# 21 domainCheck 海外主机集中运维与自动化方案
如果当前你不是在做“为什么要这么设计”的方案评审,而是已经进入真实收口、准备上线阶段,建议先跳转:
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
## 一、为什么要升级成“集中运维”
这段时间的多机联调已经证明一个事实:
- 现在这套“人在海外主机上看后台,再去大陆机器上手工跑命令、复制日志、判断状态”的方式,能联调,但效率很低
- 一旦节点数从 `2` 台增加到 `3` 台、`5` 台、`10` 台,复杂度会快速失控
- 国内机器上又不能稳定部署 Codex因此不能指望每台机器都具备“本机智能调试能力”
- 检测日志很大,靠人工复制日志片段,不适合做持续诊断
所以后续不应该继续堆更多“检查脚本”,而应该把整个体系升级成:
> 海外主机作为唯一运维控制面,统一接管大陆机器的安装、更新、重启、巡检、日志回流和故障诊断。
一句话目标:
> 只在海外主机操作,大陆机器尽量不需要人工登录。
---
## 二、最终目标形态
### 1. 海外主机承担什么
海外主机作为唯一控制面,承担:
- Web 后台
- API 控制面
- 运维任务中心
- 发布包仓库
- 节点注册与权限中心
- 日志汇聚与诊断入口
- 远程命令编排
- 巡检结果展示
也就是说,后续所有动作都从海外主机发起:
- 新机器纳管
- 初始化安装
- 发布更新
- 配置下发
- 服务重启
- 健康检查
- 采集诊断包
- 远端日志查看
- 故障一键排查
### 2. 大陆机器承担什么
大陆机器不再承担复杂控制逻辑,只承担:
- `domaincheck worker/controller` 本体服务
- 一个轻量 Node Agent
- 本机 systemd / 日志 / 版本 / 健康信息暴露
- 接收控制面任务并执行
- 将执行结果、日志、诊断包回传到海外主机
这意味着大陆机器以后是“被管理对象”,而不是“人工登录操作对象”。
---
## 三、推荐方案
## 方案 A海外控制面 + 大陆 Node Agent
这是我建议的主方案,也是长期最优方案。
### 核心思想
不要把“SSH 到每台机器执行命令”作为主链路,而是让每台大陆机器常驻一个 Agent
- Agent 主动连海外控制面
- Agent 拉取待执行任务
- 本地执行 systemd / shell / 发布 / 巡检
- Agent 把 stdout / stderr / 状态 / 日志游标回传
这样做的好处是:
- 不要求海外机能直接入站打通到大陆机
- 不要求每次人工 SSH
- 不怕 SSH 权限、跳板机、端口变化导致整套流程断掉
- 天然适合 NAT、弱网络、多机扩容
### 交互方式
建议 Agent 使用下面其中一种方式主动连海外控制面:
#### 首选HTTPS 长轮询
- `agent -> overseas-api`
- 周期拉取任务
- 周期上报心跳、版本、服务状态、日志摘要
优点:
- 实现最简单
- 最容易兼容现有 Python / FastAPI 架构
- 容易先落 MVP
#### 可升级WebSocket 常连
- 建立长连接
- 海外控制面可实时下发任务
- Agent 实时回传执行日志
优点:
- 更实时
- 日志流式体验更好
缺点:
- 第一版复杂度更高
### 为什么 Agent 比 SSH 更优
SSH 适合作为:
- 首次 bootstrap
- 临时人工兜底
- 非常规应急
但不适合作为日常主运维链路,因为:
- 节点一多,权限和连通性管理会变得脆弱
- 脚本回显、超时、日志采集很难标准化
- 人工 SSH 本质上还是“远程手工运维”
所以最佳做法是:
- SSH 只用于首次纳管
- 日常统一走 Agent
## 三点五、当前已经落地的第一版操作闭环
这套方案现在已经不只是设计稿,仓库里已经有一批可以直接使用的统一入口:
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
- `domain-api/deploy/multi-region/drive_ops_center.sh`
- `domain-api/deploy/multi-region/drive_release_hub.sh`
- `domain-api/deploy/multi-region/drive_ops_action.sh`
- `domain-api/deploy/multi-region/build_node_agent_bootstrap_plan.sh`
推荐从海外控制面按这个顺序使用:
### 1. 初始化控制面配置
```bash
cd /opt/domaincheck/domain-api
bash domain-api/deploy/multi-region/init_ops_center_config.sh \
/etc/default/domaincheck-ops-center \
http://121.204.244.188:8100 \
http://152.53.37.118:8100 \
https://api.example.com
```
### 2. 查看当前统一配置
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh config
```
### 3. 在控制面本机生成发布包
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh release-package
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
bash domain-api/deploy/multi-region/drive_ops_center.sh release-preview
bash domain-api/deploy/multi-region/drive_ops_center.sh release-preview-smart worker
bash domain-api/deploy/multi-region/drive_ops_center.sh release-show
```
### 4. 导出一份联调/交接报告
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export
```
### 5. 为新节点导出接管计划
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-plan-export \
/opt/domaincheck/domain-api/runtime/ops-center-reports/agent-plans \
http://127.0.0.1:8100 \
mainland-worker-02 \
mainland \
worker \
https://api.example.com \
/opt/domaincheck
```
这五步对应的意义分别是:
- 先统一控制面默认地址和身份
- 再确认驾驶舱当前在看哪个环境
- 再把发布物、发布前预检和发版驾驶舱都收回统一入口,不再额外记根目录脚本
- 再把现场收敛成可以回看、可以交接的报告
- 最后为新增节点生成标准接管方案
这样后面无论是海外 Codex 自动驾驶、后台按钮,还是人工运维,都不再需要先去大陆机器手工拼命令。
## 三点六、现在发布驾驶舱已经进入统一数据口径
目前发布侧也已经不是零散接口拼出来的状态,而是收敛成了一套统一判断:
- `GET /api/v1/ops/releases/launchpad`
- `ops overview -> release_hub.launchpad`
- `check_release_hub.sh`
- `drive_release_hub.sh launchpad`
- 运维中枢页面里的“发布驾驶舱”
这几处现在复用的是同一套结论,都会统一告诉你:
- 当前是 `ready / attention / blocked`
- 最新发布包是否可用
- 最新 Release 是否已经建立
- Worker 智能灰度是否可发
- Control 发布是否可发
- 下一步推荐动作是什么
同时,`GET /api/v1/ops/runbook` 里的标准作业路径 `release_progression` 也已经切到同一套发布驾驶舱判断。
也就是说,海外控制面看到的“标准作业路径”和“发布驾驶舱”不会再出现一套说能发、一套说先补接管的分裂口径。
现在又往前推进了一步:
- `POST /api/v1/ops/runbook/sequences/{sequence_key}/execute`
后台上的标准作业路径按钮,后续都应该优先走这个入口,再由后端统一分发到 driver action / playbook / rollout 预案,而不是前端自己维护动作分支。
这样后续海外 Codex、后台按钮和命令行脚本拿到的就不再是三套不同口径而是同一套“发布驾驶判断”。
## 三点七、现在回执队列也已经进入统一治理入口
除了发布驾驶舱Node Agent 回执队列这一层现在也已经不再是“只能看、不能管”的状态。
当前已经统一收口成三类入口:
- 页面:
- OpsCenter 托管节点表可以直接打开“回执队列治理”抽屉
- 驾驶建议:
- `dead_letter / retrying` 会优先落到标准动作模板
- 海外单入口 CLI
- `drive_ops_center.sh queue-status`
- `drive_ops_center.sh queue-records`
- `drive_ops_center.sh queue-flush`
- `drive_ops_center.sh queue-replay`
- `drive_ops_center.sh queue-replay-record`
- `drive_ops_center.sh queue-discard-record`
这意味着海外主机已经可以统一处理:
- 哪台节点存在积压 / 死信
- 头部记录是什么
- 什么时候应该立即冲刷
- 什么时候应该重放
- 什么时候应该说明原因后丢弃
当前阶段仍然明确限制为:
- `record_visibility=head_only`
- 所有治理动作都落成正式 `ops job`
- 不直接 SSH 到节点改队列文件
这样以后即使把远端全量死信记录正式投影到控制面,也只是“扩可见范围”,而不是重新推翻当前模型。
---
## 四、这套方案要解决的具体问题
## 1. 安装部署自动化
目标:
- 海外控制面点一次“新增节点”
- 填一个节点模板
- 大陆机器自动初始化
建议流程:
1. 海外后台新增节点
2. 生成 `bootstrap token`
3. 大陆机器只执行一次极短安装命令
4. Node Agent 注册到海外控制面
5. 控制面向该节点下发:
- 拉取发布包
- 解压
- 生成 `/etc/default/...`
- 安装 systemd
- 启动服务
6. 回传安装结果
这样后续加机器时,不需要再重复人工对照文档逐条敲。
## 2. 更新发布自动化
目标:
- 海外主机统一推版本
- 大陆节点自动拉包、灰度更新、失败回滚
这里强烈建议:
> 后续不要让大陆机器自己跑 git 作为正式更新链路。
而是改成:
- 海外控制面构建发布包
- 版本号固定
- Agent 下载指定 release 包
- 校验 checksum
- 切换当前版本软链
- 重启服务
- 回传成功/失败
这样比远程 `git pull` 更稳,因为:
- 不依赖每台机器 git 权限
- 不怕工作区脏文件
- 不怕 root/www 用户混用
- 可回滚
推荐目录形态:
```text
/opt/domaincheck/releases/<version>/
/opt/domaincheck/current -> /opt/domaincheck/releases/<version>/
```
更新时:
- 下载新版本到 `releases`
- 校验
- 切换 `current`
- `systemctl restart ...`
失败就回滚软链。
## 3. 远程命令执行自动化
目标:
- 海外控制面能发“结构化任务”
- 不再让人手工复制命令
建议任务类型:
- `service.start`
- `service.stop`
- `service.restart`
- `service.status`
- `deploy.release`
- `config.render`
- `diagnostics.collect`
- `logs.tail`
- `script.run`
- `health.check`
每个任务统一回传:
- 任务 ID
- 节点编码
- 开始时间 / 结束时间
- exit code
- stdout
- stderr
- 结构化结果 JSON
这样后面后台才能真正做“操作中心”。
## 4. 日志集中回流
目标:
- 海外后台就能看到大陆机器日志
- 不再人工抄 `journalctl`
### 日志回流建议分三层
#### A. 关键事件流
只回传关键事件:
- 服务启动
- 服务重启
- 任务开始
- 任务完成
- 代理刷新失败
- Redis/DB 连接异常
- 第三方站点异常
适合默认长期开启。
#### B. 诊断模式日志流
像你现在提的“关闭 / 关键 / 全量回传”一样:
- `off`
- `key`
- `full`
这是正确方向,应该继续保留。
#### C. 诊断包
当出现疑难问题时,一键打包:
- 最近 N 分钟 `journalctl`
- `runtime/status`
- `runtime/cluster`
- `detect_worker.log tail`
- `/etc/default/*`
- 当前版本号
- 本机健康检查输出
然后上传海外主机。
这比全量实时传所有日志更经济,也更适合定位问题。
## 5. 健康巡检自动化
目标:
- 每台大陆机自动自检
- 海外后台只看结果
建议 Agent 每 30 秒到 60 秒上报:
- 节点在线状态
- 当前版本
- API / Worker / Sync Agent 运行态
- Redis / PostgreSQL / 磁盘 / 内存 / CPU
- 最近告警
- 当前任务负载
- 最近日志时间
并把现有脚本升级为 Agent 内部检查项:
- `check_mainland_controller.sh`
- `check_mainland_worker.sh`
- `check_temp_topology.sh`
- `check_worker_participation.sh`
后续不是人工执行这些脚本,而是 Agent 周期性执行并结构化上传结果。
---
## 五、建议的系统分层
## 1. 海外控制面
建议新增一个“运维控制模块”,逻辑上可放进 `domain-api`,后续再拆独立服务也可以。
### 需要的核心对象
#### `managed_nodes`
记录纳管节点:
- `node_code`
- `region`
- `role`
- `hostname`
- `ip`
- `agent_version`
- `current_release`
- `status`
- `ssh_enabled`
- `agent_last_seen_at`
- `tags`
#### `ops_jobs`
记录运维任务:
建议以后直接以 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 为正式母本。
- `job_id`
- `job_type`
- `target_nodes`
- `payload`
- `created_by`
- `status`
- `started_at`
- `finished_at`
#### `ops_job_steps`
记录每个节点执行步骤:
- `node_code`
- `step_name`
- `status`
- `stdout`
- `stderr`
- `result_json`
#### `node_log_streams`
记录日志游标和日志回流状态:
- `node_code`
- `source`
- `mode`
- `cursor`
- `last_received_at`
## 2. Node Agent
建议单独做一个轻量 Python 服务,比如:
```text
domaincheck-node-agent
```
职责:
- 周期心跳
- 拉取任务
- 本地执行
- 采集日志
- 上报结果
- 生成诊断包
Agent 尽量独立于业务主程序,不要把它耦合进 `detect_worker.py`
原因:
- 运维系统不能依赖业务进程是否正常
- 即使 Worker 崩了Agent 还应该活着,才能帮你排障
---
## 六、对你当前项目最合适的落地路线
## 第一阶段:先做“控制面集中化”
目标:
- 海外后台统一看到所有大陆机器
- 一键拉诊断
- 一键执行已有检查脚本
- 一键切日志回传模式
这个阶段先不碰太多发布链路,优先把“看”和“查”统一。
### 交付物
- Node Agent MVP
- 海外后台“节点管理”页
- 海外后台“运维任务”页
- 海外后台“远端日志”页
- 海外后台“诊断包”页
## 第二阶段:再做“一键更新”
目标:
- 海外控制面选择版本
- 指定节点灰度发布
- 自动回滚
### 交付物
- 发布包生成器
- `release manifest`
- Agent 下载 / 校验 / 切换版本
- 回滚机制
## 第三阶段:再做“一键安装新节点”
目标:
- 新机器只执行一次 bootstrap
- 其余全部由海外控制面接管
### 交付物
- bootstrap token
- 安装向导
- 节点注册流程
- 环境模板生成器
---
## 七、比“SSH 全接管”更优的地方
你提的方向是:
> 海外主机配置好 SSH后续所有大陆机器都由代码内部接管
这个方向本身是对的但如果只做“SSH 全接管”,还不够优秀。
更优解是:
> SSH 只作为纳管和兜底方式;日常统一走 Agent 主动连接。
这样好处是:
- 架构更稳
- 不依赖每次远程入站打通
- 不怕某些网络环境 SSH 不稳定
- 更适合未来节点继续增加
- 更容易做权限分级、审计、结果留痕
所以我建议最终定成:
### 运维主链路
- `海外控制面 -> Agent 任务编排 -> 大陆节点执行 -> 回传结果`
### 运维兜底链路
- `海外控制面 -> SSH -> 大陆节点`
---
## 八、和现有代码如何衔接
当前项目已经有一些很好的基础,不需要推翻:
- 多机节点模型
- `runtime/status`
- `runtime/cluster`
- `sync-summary`
- 一批巡检脚本
- 日志回传开关雏形
- 运行中心
这些都应该保留,并演进成:
### 当前脚本的未来定位
- `bootstrap_*.sh`
- 保留为 bootstrap / 应急工具
- `check_*.sh`
- 演进成 Agent 内部的标准诊断动作
- `runtime/status`
- 继续作为统一运行态接口
- `runtime/cluster`
- 继续作为统一集群视图
- `detect_result_projection / runtime_projection`
- 继续作为跨地域观测基础
也就是说,现有代码不是废掉,而是从“人工脚本时代”升级成“控制面编排时代”。
---
## 九、我建议的最优落地决策
如果只选一个方向,我建议直接定成:
### 最优方案
- 海外主机为唯一控制面
- 大陆节点统一部署 `domaincheck-node-agent`
- Agent 主动访问海外控制面
- 日常安装、更新、巡检、日志回流全部走 Agent
- SSH 仅保留为 bootstrap 和应急手段
- 正式更新统一改为 release 包分发,不再把 `git pull` 作为正式更新链路
这是当前最值得投入的方向,因为它能同时解决:
- 调试效率低
- 多机管理复杂
- 权限混乱
- 日志分散
- 更新不稳定
- 新增节点成本高
---
## 十、建议的下一步执行顺序
不要再继续先补散点 bug而是先把运维骨架建起来。
建议顺序:
1. 固化这份方案
2. 新建 `node-agent` 设计草案
3. 先做 Agent MVP
4. 先接入:
- 心跳
- 远程任务
- 诊断包
- 日志 tail
5. 海外后台新增:
- 节点管理页
- 运维任务页
- 远端日志页
6. 再把现有检查脚本封成 Agent 动作
7. 最后再做发布链路自动化
---
## 十一、结论
当前最优策略不是“继续加脚本”,而是:
> 把 domainCheck 从“多机联调项目”升级成“海外控制面统一接管大陆节点的自动化运维系统”。
从长期看,这会比继续靠人工 SSH、人工复制日志、人工比对状态高效很多也更符合你后续继续扩机器的目标。

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,430 @@
# 25 domainCheck 海外单脑控制面上线收口总表
## 一、这份文档解决什么问题
这份文档不是再讲设计,而是把当前项目进入正式上线前,真正需要收口的事项压成一张总表。
适用场景:
- 海外主机已经作为单脑控制面
- 大陆节点已经开始纳管
- Ops Center / Codex Driver / CLI 都已经接入同一套 `ops` contract
- 目标从“能联调”切换到“能稳定上线、能持续运维”
这份文档优先回答 4 个问题:
1. 现在能不能继续收口
2. 现在能不能正式发布
3. 还有哪些阻断项没清
4. 下一步到底先跑哪条命令
---
## 二、上线前统一原则
上线前必须统一成下面这条工作方式:
- 海外主机作为唯一主驾驶席
- 页面、CLI、Codex 共用同一套后端判断
- 默认先看 `go-live-summary`
- 真要解释原因时再下钻 `stack-diagnosis`
- 真要执行动作时走 `driver-resolve / execute-resolved`
- 真要发布或放量时走 `Release Hub / rollout`
不再推荐:
- 人工分别登录多台大陆机器拼状态
- 每次上线都从 `journalctl + systemctl + curl` 临时组合判断
- 前端、CLI、Codex 各自维护不同的默认下一步
---
## 三、正式上线门禁
### 1. 收口门禁
至少同时满足:
- `go_live_summary.go_live_status != blocked`
- `stack_diagnosis.diagnosis.stack_status != blocked`
- `route_surface_complete=true`
- `driver-feed.automation_coverage.launch_status != blocked`
- 首屏默认下一步已经明确,不再是模糊人工判断
### 2. 发布门禁
至少同时满足:
- `publish_ready=true`
- `launchpad_status != blocked`
- 不存在未处理的关键 `blocking_reasons`
- 当前发布主车道已经清晰落在:
- `review_smart_rollout_preview`
- `review_control_rollout`
- `publish_latest_worker`
- `create_release_rollout_*`
### 3. 运维门禁
至少同时满足:
- 海外控制面 API 稳定在线
- 大陆 controller / worker 心跳稳定
- 节点接管缺口已经收口,或至少首个缺口节点已有明确恢复动作
- 远端日志回传可按需开启、复核、关闭
- 运行中心首屏能直接区分:
- 在线但未参与
- 正在执行 / 正在领任务
---
## 四、真正的起手顺序
每次准备上线、复核、值班接手时,都按这个顺序来:
### 第 1 步:看上线收口摘要
```bash
cd /www/wwwroot/getDomain
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
```
如果当前是临时海外控制面联调,也可以把第二个地址替换成海外目标 API。
第一眼重点只看:
- `go_live_status`
- `publish_status`
- `blocking_reasons`
- `warnings`
- `next_step_action_code`
- `operator_title`
如果你怀疑当前是环境本身有漂移,而不是业务链路没收口,先执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh env-audit
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
```
重点只看:
- `status`
- `headline`
- `missing_items`
- `tooling_items`
- `runtime.preflight_ok`
- `runtime.readiness_status`
- `runtime.route_surface_complete`
- `runtime.repo_capability_drift`
- `runtime.runtime_may_need_restart`
- `recommended_actions`
如果这里出现:
- `runtime.repo_capability_drift=true`
- `runtime.runtime_may_need_restart=true`
则优先执行 `runtime-refresh-recover`,不要先把问题误判成“代码还没写完”。这通常表示:
- 仓库代码已经更新
- 但运行中的 `domaincheck-api` 进程还没重启到这版代码
`runtime-refresh-recover` 会固定给出:
- 当前是否真的属于运行时版本漂移
- 推荐先跑的 `runtime-refresh-recover` 统一恢复入口
- 重启后应该按什么顺序继续:
- `stack-diagnosis`
- `node-bootstrap-plan`
- `stack-next`
- `doctor-decision`
如果你已经不想再手工一条条执行,而是想把“刷新运行时 + 再做总检复核”压成一次操作,直接执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
```
它会固定串起:
1. `runtime-refresh-recover`
2. `stack-diagnosis summary`
3. `stack-next`
4. `go-live-check summary`
5. `doctor-decision`
如果要把这次上线前检查直接导出成一整包证据,而不是手工复制多段终端输出,直接执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
```
它会统一导出:
- `env-audit`
- `go-live-check` 摘要
- `go-live-summary`
- `stack-diagnosis`
- `driver-feed`
- `codex-brief`
- `release-launchpad`
- `doctor-decision`
- `doctor-export`
- `manifest.json`
其中 `manifest.json` 会直接标记:
- 环境审计当前是 `ready / attention / blocked`
- 哪些 artifact 成功
- 哪些 artifact 超时
- 哪些 endpoint 缺失或返回异常
- 当前推荐的阅读顺序
现在 Ops Center 首屏也会同步读取最新 bundle / manifest
- 顶部 `总检决策`
- 详情区 `总检主决策详情`
- API `GET /api/v1/ops/doctor-decision`
- 顶部 `交付证据`
- 详情区 `交付证据详情`
- API `GET /api/v1/ops/go-live-bundle`
- 顶部 `正式复核`
- 详情区 `正式复核详情`
- API `GET /api/v1/ops/go-live-review`
如果你想先拿到一句“bundle manifest 正式复核是否通过”的统一结论,而不是直接跳到最终签收,也可以先执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review
```
如果你想在 bundle 基础上直接得到一句“现在能不能签字上线”的统一结论,而不是人工再看多份 JSON直接执行
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff
```
或基于已有 bundle
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff /path/to/go-live-bundle
```
这里的 `go-live-signoff` 已经不是另一套独立摘要,而是固定压缩:
- `go-live-review`
- `doctor-decision`
- `go-live-summary / 发布门禁 / launchpad 摘要`
所以页面首屏、CLI 与海外 Codex 看到的是同一份最终签字口径。
第一眼重点只看:
- `signoff_status`
- `headline`
- `release_gate`
- `blocked_reasons`
- `attention_reasons`
- `decision`
### 第 2 步:看总检详情
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
```
重点只看:
- `stack_status`
- `issues`
- `next_step`
- `quick_commands`
### 第 3 步:看驾驶主线
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
```
重点只看:
- `top_recommendation`
- `automation_coverage`
- `recommended_behavior`
- `focus`
### 第 4 步:确认默认下一步
如果只是预览:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-next
```
如果已经确认要执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-next run confirm
```
---
## 五、当前项目的主处理车道
上线前遇到问题时,不要发散排查,先归到下面 5 条主车道:
### 1. 节点接管车道
典型信号:
- `remote_access_ready=0`
- `bootstrap_run`
- `run_acceptance`
- `managed_nodes_agent_pending`
优先命令:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-gap-check
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-gap-recover
bash domain-api/deploy/multi-region/drive_ops_center.sh node-onboarding http://127.0.0.1:8100 mainland-worker-01
```
### 2. 远端日志车道
典型信号:
- 首屏显示“现场日志覆盖不足”
- `log_sync_enabled=false`
- `line_count=0`
- `source_nodes` 缺节点
优先命令:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-check
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-recover
bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 key
```
### 3. 总检修复车道
典型信号:
- `stack_status=blocked`
- `surface_status=broken`
- `contracts` / `launchpad` / `activity-stream` 缺口
优先命令:
```bash
bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100
bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100
bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100
```
### 4. 检测参与车道
典型信号:
- 页面显示在线节点多,但真正跑检测的少
- `claimed_by` 分布异常
- 在线但未参与节点过多
优先命令:
```bash
bash domain-api/deploy/multi-region/check_worker_participation.sh
bash domain-api/deploy/multi-region/drive_ops_center.sh activity-stream
```
### 5. 发布 / 放量车道
典型信号:
- `publish_ready=true`
- `launchpad_status` 已可进入 Worker / Control rollout
- 默认下一步已落在 Release Hub
优先命令:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-focus-preview
```
---
## 六、首屏通过标准
Ops Center 首屏如果要算“可上线收工”,至少应满足:
- 首屏能同时显示:
- 上线收口状态
- 发布闸门状态
- 自动化收口状态
- 主处理车道
- 默认下一步
- 后端接管覆盖
- 观察层完整度
- 现场日志覆盖度
- 首屏动作条可直接完成:
- 默认下一步
- 定位下一步
- 看契约
- 复制命令
- 进入主处理面板
- 首屏回执条能统一反馈:
- 打开契约成功
- 定位成功
- 复制成功
- 后端动作执行成功 / 警告 / 失败
如果这些已经稳定,就说明值班同事和海外 Codex 已经在同一个驾驶台上工作,而不是各自再拼逻辑。
---
## 七、通过标准后的最小上线动作
当上面门禁都通过后,建议最小上线动作顺序为:
1.`go-live-check`
2.`stack-diagnosis`
3. 复核 `driver-feed.automation_coverage`
4. 复核 `codex-brief.recommended_behavior`
5. 若进入发布车道,先看 `release-launchpad`
6.`guarded_auto` 动作显式确认后再执行
上线前最后一轮建议保留证据:
- `go-live-summary` 输出
- `stack-diagnosis` 输出
- `driver-feed` 输出
- `codex-brief` 输出
- `release-launchpad` 输出
- 前端构建成功记录
---
## 八、现在这套体系是否已经进入可上线状态
按当前仓库现状,可以认为已经具备:
- 单脑控制面的协议骨架
- 首屏驾驶舱骨架
- 总检与驾驶 contract
- Release Hub / Driver / Node Agent 的联动骨架
- 海外 CLI 单入口
但正式宣称“可以上线”,仍建议每次以这份总表复核,而不是只凭某一个页面截图或某一次联调成功就直接跳过。
这份文档的定位就是:
> 后续每次上线、值班接手、发布前复核,都先回到这里。
如果你已经准备进入“这次要不要真的发”的最终执行阶段,下一份应直接看:
- `docs/26_domainCheck_发布前运行验证与交付模板.md`

View File

@@ -0,0 +1,635 @@
# 26 domainCheck 发布前运行验证与交付模板
## 一、这份模板怎么用
这份文档不是设计说明,而是正式发布前最后一轮执行模板。
适用场景:
- 海外单脑控制面已经搭好
- 大陆节点已经接入或正在收口
- 代码已经更新到待发布版本
- 现在要做最后一轮运行验证、证据留存、交付确认
这份模板分成 3 段:
1. 发布前运行验证
2. 上线证据导出
3. 最终交付结论模板
---
## 二、发布前运行验证
### 1. 先跑统一收口摘要
```bash
cd /www/wwwroot/getDomain
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
```
记录结果:
- `go_live_status`
- `publish_status`
- `blocking_reasons`
- `warnings`
- `next_step_action_code`
- `operator_title`
- `launchpad_recommended_target_node_code`
- `launchpad_recommended_recovery_label`
- `launchpad_recommended_recovery_summary`
- `launchpad_onboarding_bootstrap_pending_nodes`
- `launchpad_onboarding_acceptance_ready_nodes`
通过标准:
- `go_live_status != blocked`
- `publish_status != blocked`
### 1.5 先看环境差异是否已收口
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh env-audit
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
```
重点只看:
- `status`
- `headline`
- `python.pytest_installed / fastapi_installed / uvicorn_installed`
- `python.runtimes`
- `services`
- `runtime.preflight_ok`
- `runtime.readiness_status`
- `runtime.route_surface_complete`
- `missing_items`
- `tooling_items`
通过标准建议:
- `status != blocked`
- `runtime.preflight_ok=true`
- `route_surface_complete=true`
- 如果 `missing_items` 里仍有关键基础项,先补环境,再继续业务复核
- 如果只有 `tooling_items`,可以继续上线复核,但建议后补本机工具链
- 如果 `runtime.runtime_may_need_restart=true`
- 不要直接继续接管或发布链
- 先按 `runtime-refresh-recover` 给出的固定顺序完成 API 重启与复检
- 如果你希望把这一轮“重启 + 总检 + 默认下一步 + 上线摘要”压成一次复核
- 直接执行 `go-live-recover`
### 1.6 先看托管节点接入面是否清楚
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
```
重点只看:
- `summary.agent_ready`
- `summary.ssh_ready`
- `summary.remote_access_ready`
- `summary.remote_access_state_counts`
- 每台节点的 `agent_state`
- 每台节点的 `remote_access_state`
- 每台节点的 `ssh_access_label`
- 每台节点的 `ssh_entry`
通过标准建议:
- 如果某台节点还是 `agent_pending`
- 先确认是否已经保存 SSH 入口
- 如果节点尚未保存 SSH 入口,但你已经决定由海外控制面统一接管
- 先补录:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bind-ssh http://127.0.0.1:8100 mainland-worker-01 121.204.244.248 root 22
```
- 补录后立即复查:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
bash domain-api/deploy/multi-region/drive_ops_center.sh node-handover http://127.0.0.1:8100 mainland-worker-01
```
- 如果已经 `agent_ready`
- 说明该节点已进入标准远端执行器接管面
- 如果还不是 `agent_ready`,但 `ssh_ready` 已经成立
- 说明至少已经具备海外主控经 SSH 进入节点完成 bootstrap / 验收的前置条件
### 2. 再看总检详情
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
```
记录结果:
- `stack_status`
- `issues`
- `next_step`
- `quick_commands`
通过标准:
- `stack_status != blocked`
- `issues` 里没有未处理的关键阻断项
### 3. 再看驾驶主线
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
```
记录结果:
- `top_recommendation`
- `automation_coverage`
- `recommended_behavior`
- `focus`
- `summary.launchpad_recommended_target_node_code`
- `summary.launchpad_recommended_recovery_label`
- `summary.launchpad_onboarding_bootstrap_pending_nodes`
- `summary.launchpad_onboarding_acceptance_ready_nodes`
通过标准:
- `automation_coverage.launch_status != blocked`
- 默认下一步已经明确
- 如果 `summary.launchpad_recommended_target_node_code` 非空
- 值班同事应能直接知道当前卡在哪台节点
- 不需要再回 launchpad 明细手工翻 gap rows
### 4. 若涉及发布,再看 Release Hub
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
```
记录结果:
- `launchpad_status`
- `recommended_action_code`
- `default_rollout_gate_status`
- `recommended_target_node_code`
- `recommended_recovery_label`
- `recommended_recovery_summary`
- `onboarding_bootstrap_pending_nodes`
- `onboarding_acceptance_ready_nodes`
通过标准:
- `launchpad_status != blocked`
- `default_rollout_gate_status != blocked`
特殊判读:
- 如果 `recommended_action_code=api-restart`
- 不要误判成“节点没接好”
- 这更像是运行中的控制面 API 还没刷新到当前仓库的最新运维能力
- 应先执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
```
- 然后重新跑:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
```
补充说明:
-`latest_release.json` 直接创建 Release或执行“基于最新包的一键智能 Rollout”之前
- 现在不仅要求 `smoke_test_ok=true`
- 还要求当前 `final_release_report.json` 与最新包完全匹配,且 `final_release_ok=true`
- 如果页面按钮被禁用,或 API 返回“最新发布包尚未完成最终签收”
- 优先检查 `final_release_report_stale_reason`
- 再检查 `final_release_gate_decision / final_release_gate_blocked_reasons`
### 5. 若涉及节点接管,补跑节点验收
如果这次发布包含:
- 新增大陆节点
- 重新接管旧节点
- 切换到 Node Agent 主接管模式
则不能只看总检摘要,还要对目标节点逐台完成验收。
先看验收计划:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-plan http://127.0.0.1:8100 mainland-worker-01
```
确认无误后执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-worker-01 cli/acceptance
```
记录结果:
- `node_code`
- `status`
- `checks`
- `blocking_issues`
- `warnings`
- `recommended_next_step`
通过标准:
- `status != blocked`
- 没有未处理的 `blocking_issues`
- 目标节点已经不再停留在“bootstrap pending / acceptance pending”
### 6. 若涉及远端执行,补跑日志回传验证
如果当前版本准备进入:
- 多机值班
- 海外单脑统一接管
- 远端动作回执与现场日志复核
则需要确认日志回传链已经可用,而不是只看节点在线。
先检查:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-check
```
如果存在缺口,再执行恢复:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-recover http://127.0.0.1:8100 key confirm cli/log-sync
```
必要时抓一份节点现场日志:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 full
```
记录结果:
- `log_sync_enabled`
- `line_count`
- `source_nodes`
- `missing_nodes`
- `scene_log.status`
通过标准:
- 关键节点已出现在 `source_nodes`
- `missing_nodes` 不包含当前发布主车道节点
- 如需人工复核现场,`scene-node-log` 能取到有效日志
### 7. 若涉及 Node Agent 交付,补看队列健康
如果这轮发布已经进入“控制面发动作、节点拉任务、节点回执结果”的模式,就必须确认 delivery queue 没有假健康。
先看队列摘要:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-status http://127.0.0.1:8100 mainland-worker-01
```
如果怀疑有堆积或死信,再看明细:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-records http://127.0.0.1:8100 mainland-worker-01 dead_letter
```
如需重放:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-replay http://127.0.0.1:8100 mainland-worker-01 20 cli/queue-replay
```
记录结果:
- `delivery_queue.summary`
- `dead_letter`
- `retrying`
- `pending`
- `replayed_count`
通过标准:
- `dead_letter=0`
- 没有持续增长的 `retrying`
- 队列不处于异常堆积状态
注意:
- `dead_letter > 0` 时,不应宣称“可正式发布”
- 这种情况至少按 `publish_status = blocked` 处理
### 8. 最后跑一次 doctor 结论
前面的检查是分面复核最后还要再收成一句话让交班、值班、Codex 驾驶员看到同一结论。
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision
```
记录结果:
- `status`
- `decision`
- `recommended_commands`
- `recommended_reading_order`
- `launchpad_alignment`
通过标准:
- `status != blocked`
- `decision` 与前面的 `go-live-check / stack-diagnosis / release-launchpad` 不冲突
- `recommended_commands` 已经指向明确的下一跳,而不是泛化排查
### 9. 若希望直接生成“可否签字上线”的最终摘要
如果你不想手工再拼:
- bundle review
- doctor 结论
- 发布门禁
可以直接执行:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff
```
如果已经有现成证据包,也可以直接基于 manifest 输出:
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff /path/to/go-live-bundle
```
这一步现在不是“另一套单独判断”,而是固定复用:
- `go-live-review`
- `doctor-decision`
- `go-live-summary / 发布门禁 / launchpad 摘要`
也就是说页面首屏、CLI 和海外 Codex 看到的最终签收结论,已经开始共享同一份签字口径。
重点只看:
- `signoff_status`
- `headline`
- `release_gate`
- `blocked_reasons`
- `attention_reasons`
- `decision`
- `recommended_commands`
通过标准:
- `signoff_status=ready`
- 可进入最终人工签字或正式发布
- `signoff_status=attention`
- 说明现场已经接近完成,但仍应先清 attention 项
- `signoff_status=blocked`
- 本轮不应宣称收口完成
---
## 三、上线证据导出
### 1. 导出 bundle
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
```
这会生成一整包上线前证据。
建议至少保留:
- `00_env_audit.txt`
- `01_go_live_check_summary.txt`
- `02_go_live_summary.json`
- `03_stack_diagnosis.json`
- `04_driver_feed.json`
- `05_codex_brief.json`
- `06_release_launchpad.json`
- `manifest.json`
如果当前轮次涉及节点接管,建议额外保留:
- `agent_gap_export.json`
- `node_bootstrap_plan.txt`
- `node_acceptance_plan.json`
如果当前轮次涉及远端执行与日志回传,建议额外保留:
- `scene_log_*.txt`
- `doctor_decision.json`
### 2. 直接复核 bundle
```bash
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review
```
如果你已经切到 Ops Center 页面,也可以直接看:
- 首屏 `总检决策`
- `总检主决策详情`
- API `GET /api/v1/ops/doctor-decision`
- 首屏 `交付证据`
- `交付证据详情`
- API `GET /api/v1/ops/go-live-bundle`
- 首屏 `正式复核`
- `正式复核详情`
- API `GET /api/v1/ops/go-live-review`
记录结果:
- `status`
- `headline`
- `critical_failures`
- `noncritical_failures`
- `launchpad_alignment.consistent`
- `env_audit.status`
- `env_audit.missing_items`
- `recommended_next_steps`
通过标准建议:
- `status=ready`
- 可进入最终人工确认或正式发布
- `status=attention`
- 核心报告已齐,但仍需复核补充失败项或环境缺口
- 也包括 `go_live_summary / stack_diagnosis / driver_feed / codex_brief` 的 launchpad 摘要出现不一致
- `status=blocked`
- 关键报告缺失,或环境审计已经判定阻断,不应直接宣称收口完成
如果 `env_audit.missing_items` 里出现:
- `missing_env:/etc/default/domaincheck-node-agent`
则这次 bundle review 已经不是泛泛地提示“环境缺口”,而是明确指向 Node Agent 接管还未收口。此时按下面顺序推进:
1. `agent-gap-export`
2. `node-bootstrap-plan`
3. 如果 summary 已提示 `runtime_may_need_restart=true`
4. 完成恢复后重新导出 bundle再做一次 `go-live-review`
### 3. 生成最终交付包
运行验证、bundle 复核都通过后,再生成最终交付包。
如果当前机器已经具备 smoke 运行条件,直接执行:
```bash
bash ./prepare_final_release.sh
```
如果当前机器只适合先做离线打包,可先跳过 smoke
```bash
DOMAINCHECK_SKIP_SMOKE_TEST=1 bash ./package_domain_release.sh
bash ./verify_domain_release.sh
```
后续再回到具备运行条件的机器上执行:
```bash
bash ./prepare_final_release.sh
```
最后统一查看最新交付结果:
```bash
bash ./show_latest_release.sh
```
重点只看:
- `latest_release.package_name`
- `latest_release.archive_path`
- `latest_release.sha256`
- `latest_release.smoke_test_ok`
- `final_release_ok`
- `final_release_report`
- `final_release_report_ok`
- `final_release_report_matches_latest`
- `final_release_report_stale_reason`
- `final_release_gate_decision`
- `final_release_gate_blocked_reasons`
- `final_release_gate_verify_ok`
- `final_release_gate_smoke_test_ok`
通过标准:
- `verify_domain_release.sh` 返回 `ok=true`
- `latest_release.smoke_test_ok=true`
- `final_release_ok=true`
- `final_release_report_matches_latest=true`
- `final_release_gate_decision=ready`
- `final_release_gate_verify_ok=true`
- `final_release_gate_smoke_test_ok=true`
- `final_release_report` 已指向最新一轮生成的正式报告
4. 先执行 `runtime-refresh-recover`
5. 再重新导出 bootstrap plan
---
## 四、最终交付结论模板
下面这段可以直接作为正式发布前的交付结论模板使用。
### 模板正文
```text
本轮发布前运行验证已完成。
一、统一收口结论
- go_live_status: <填写>
- publish_status: <填写>
- stack_status: <填写>
- automation_coverage.launch_status: <填写>
- launchpad_status: <填写>
二、默认下一步
- next_step_action_code: <填写>
- operator_title: <填写>
- launchpad_recommended_target_node_code: <填写>
- launchpad_recommended_recovery_label: <填写>
三、上线证据包
- bundle 路径: <填写>
- manifest 复核状态: <填写 ready/attention/blocked>
- go-live-review.headline: <填写>
- manifest.review_status_hint: <填写>
- critical_failures: <填写>
- noncritical_failures: <填写>
- manifest.launchpad_recommended_target_node_code: <填写>
- manifest.launchpad_recommended_recovery_label: <填写>
- manifest.launchpad_onboarding_bootstrap_pending_nodes: <填写>
- manifest.launchpad_onboarding_acceptance_ready_nodes: <填写>
- manifest.launchpad_alignment.consistent: <填写 true/false>
四、结论
- <填写:可进入正式发布 / 仍需继续收口 / 暂不可发布>
```
### 模板补充说明
填写“可进入正式发布”前,建议再做一次人工交叉确认:
- 发布门禁是否为 `ready` 或至少非 `blocked`
- 当前主车道是否已经明确
- 是否还存在 `dead_letter`
- 是否还有节点停留在 `bootstrap pending / acceptance pending`
- 是否已经保留可复盘的证据包与 manifest
---
## 五、交付给值班同事时最少要带什么
最少带下面 4 样:
1. `go-live-check` 摘要
2. `stack-diagnosis` 结果
3. `driver-feed / codex-brief` 结果
4. `go-live-export` 生成的 `manifest.json`
如果值班同事需要直接接手节点问题,再多带 3 样:
1. `node-acceptance-run` 最近结果
2. `log-sync-check` 最近结果
3. `queue-status``queue-records dead_letter` 最近结果
这样交班的人不需要重新从零拼现场。
---
## 六、真正的收工标准
如果要算“这次版本已经可以收工”,建议至少同时满足:
- 前端构建成功
- 海外单脑驾驶舱首屏状态正常
- `go-live-review.status != blocked`
- 发布门禁不为 `blocked`
- Node Agent 目标节点已完成接管验收
- 远端日志回传链可用或已确认本轮不依赖
- delivery queue 不存在未处理 `dead_letter`
- 当前默认下一步已经不是“继续补基础设施”,而是进入正式发布 / 观察 / 验收车道
到这一步,才适合说:
> 当前项目已经进入可上线、可交付、可持续运维状态。

View File

@@ -0,0 +1,816 @@
# domainCheck Ops Agent Protocol
## 1. 目标
这份文档用于冻结 `Node Agent <-> Overseas Control Plane` 的正式 contract。
适用范围:
- `POST /api/v1/ops/agent/tokens`
- `POST /api/v1/ops/agent/bootstrap-plan`
- `GET /api/v1/ops/nodes/{node_code}/handover`
- `POST /api/v1/ops/nodes/{node_code}/handover/bootstrap-plan`
- `POST /api/v1/ops/agent/register`
- `POST /api/v1/ops/agent/heartbeat`
- `POST /api/v1/ops/agent/pull`
- `POST /api/v1/ops/agent/jobs/{job_id}/start`
- `POST /api/v1/ops/agent/jobs/{job_id}/complete`
- `POST /api/v1/ops/agent/jobs/{job_id}/events`
设计原则:
- Agent 只执行声明过的结构化动作
- 控制面负责调度、门禁、编排和审计
- 所有回执都必须可重放、可去重、可死信化
---
## 2. 公共响应包裹
所有接口统一返回:
```json
{
"code": 0,
"message": "ok",
"detail_code": "",
"data": {}
}
```
约束:
- `code=0` 表示成功
- `code=1` 表示业务失败
- `detail_code` 用于结构化错误语义
- `message` 只承担可读说明,不承担机器判断主逻辑
常见 `detail_code`
- `agent_token_invalid`
- `agent_token_expired`
- `agent_token_node_mismatch`
- `ops_job_not_owned_by_agent`
- `ops_job_invalid_status_transition`
- `release_checksum_mismatch`
- `release_health_check_failed`
---
## 3. 认证与请求头
Agent 相关接口统一使用:
- `Content-Type: application/json`
- `X-Domaincheck-Agent-Token: <token>`
其中:
- `tokens`
- `bootstrap-plan`
不要求 Agent Token。
其余接口都要求 Token。
---
## 4. 公共节点载荷
`register``heartbeat` 当前共享 `_base_payload()`
最小字段:
```json
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"title": "mainland-worker-02",
"hostname": "host-a",
"ip": "10.0.0.12",
"agent_version": "0.1.0",
"capabilities": [
"service.start",
"service.stop",
"service.restart",
"service.status",
"health.snapshot",
"logs.collect",
"diagnostics.collect",
"deploy.release"
],
"labels": {},
"metadata": {
"service_names": {
"api": "domaincheck-api",
"worker": "domaincheck-worker",
"sync_agent": "domaincheck-sync-agent",
"node_agent": "domaincheck-node-agent"
},
"delivery_queue": {
"state": "healthy",
"label": "正常",
"reason": "当前没有待重试回执,也没有死信记录。",
"pending_count": 0,
"dead_letter_count": 0,
"last_flush_at": "",
"oldest_pending_at": "",
"oldest_pending_request_id": "",
"oldest_pending_kind": "",
"oldest_dead_letter_at": "",
"oldest_dead_letter_request_id": "",
"oldest_dead_letter_kind": ""
}
}
}
```
说明:
- `service_names` 是控制面识别节点服务拓扑的最小入口
- `delivery_queue` 是控制面识别 Agent 回执现场的最小入口
---
## 5. Bootstrap Contract
### 5.1 Issue Token
`POST /api/v1/ops/agent/tokens`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02",
"issued_by": "web-ui",
"expires_in_hours": 72,
"metadata": {
"source": "ops-center"
}
}
```
返回体关键字段:
- `token`
- `token_preview`
- `record_id`
- `node_code`
- `expires_at`
- `created_at`
### 5.2 Bootstrap Plan
`POST /api/v1/ops/agent/bootstrap-plan`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02",
"node_region": "mainland",
"node_role": "worker",
"issued_by": "web-ui",
"expires_in_hours": 72,
"control_plane_base_url": "https://ops.example.com",
"root_dir": "/opt/domaincheck",
"metadata": {
"source": "ops-center"
}
}
```
返回体关键字段:
- Token 相关字段
- `control_plane_base_url`
- `bootstrap_plan.node_code`
- `bootstrap_plan.node_region`
- `bootstrap_plan.node_role`
- `bootstrap_plan.root_dir`
- `bootstrap_plan.env_file`
- `bootstrap_plan.service_name`
- `bootstrap_plan.service_file`
- `bootstrap_plan.install_script_path`
- `bootstrap_plan.env_content`
- `bootstrap_plan.command_lines`
- `bootstrap_plan.command_block`
- `bootstrap_plan.health_checks`
- `bootstrap_plan.health_check_block`
- `bootstrap_plan.bootstrap_script_name`
- `bootstrap_plan.bootstrap_script_path`
- `bootstrap_plan.bootstrap_script_content`
- `bootstrap_plan.bootstrap_run_script_block`
约束:
- `bootstrap_plan` 是标准接入方案对象
- 页面、Codex、CLI 必须复用这一个对象
- 不允许各端再次手写 env 或 shell 逻辑
---
## 6. Runtime Contract
### 6.1 Register
`POST /api/v1/ops/agent/register`
语义:
- 校验 Token 与 `node_code`
- Upsert Agent 运行态
- 将节点推入托管目录候选态
最小成功返回字段:
- `node_code`
- `expires_at`
- `capabilities`
### 6.2 Heartbeat
`POST /api/v1/ops/agent/heartbeat`
语义:
- 刷新 `last_seen_at`
- 刷新节点身份、服务名和队列快照
- 维持控制面中的 Agent 在线态
最小成功返回字段:
- `node_code`
- `server_time`
- `expires_at`
### 6.3 Pull
`POST /api/v1/ops/agent/pull?limit=1`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02"
}
```
返回体关键字段:
```json
{
"jobs": [
{
"id": 123,
"job_code": "ops-xxx",
"action": "health.snapshot",
"target_node_code": "mainland-worker-02",
"status": "dispatching",
"execution_mode": "remote-agent",
"payload": {},
"metadata": {},
"policy": {},
"steps": []
}
],
"count": 1
}
```
约束:
- Agent 只能拿到属于自己的任务
- Agent 不负责选择任务
- 调度权始终在控制面
### 6.3.1 Agent Job Envelope
为了避免 Agent 再从自然语言动作名里“猜执行上下文”,`pull.data.jobs[]` 当前已经按统一任务包裹返回。
当前 `pull.data` 顶层也会额外带:
- `protocol_version`
- `envelope_type`
- `node_code`
- `limit`
- `count`
推荐最小结构:
```json
{
"protocol_version": "ops-agent/v1",
"envelope_type": "agent_job",
"id": 123,
"job_id": 123,
"job_code": "ops-20260418-001",
"job_type": "ops_action",
"action": "logs.collect",
"target_node_code": "mainland-worker-02",
"status": "dispatching",
"execution_mode": "remote-agent",
"step_key": "worker_logs",
"step_title": "收集 Worker 日志",
"job_ref": {
"job_id": 123,
"job_code": "ops-20260418-001",
"action": "logs.collect",
"target_node_code": "mainland-worker-02"
},
"step_ref": {
"step_id": 456,
"step_key": "worker_logs",
"step_title": "收集 Worker 日志"
},
"payload": {
"service_name": "domaincheck-worker",
"lines": 120,
"include_agent_logs": false
},
"policy": {
"timeout_seconds": 120,
"stop_on_failure": true,
"auto_approve": true
},
"metadata": {
"requested_by": "playbook:inspection.standard",
"source": "ops-center"
},
"release_context": {
"release_id": 0,
"release_version": "",
"rollout_id": 0,
"rollout_code": ""
},
"focus_ref": {
"kind": "ops_job",
"job_id": 123,
"job_code": "ops-20260418-001",
"action": "logs.collect",
"target_node_code": "mainland-worker-02"
}
}
```
正式要求:
- Agent 必须优先消费:
- `action`
- `payload`
- `policy`
- Agent 不应通过:
- `step_title`
- `summary`
- `notes`
去反推真实执行参数
- 与发布相关的任务应通过:
- `release_context`
传递 release / rollout 关联,而不是让 Agent 自己查询“当前版本”
这层的意义是:
- `ops job`
- 仍是正式执行颗粒度
- `Agent Job Envelope`
- 是 Node Agent 拿到的稳定执行视图
这样同一套 Agent 才能同时承接:
- 标准巡检
- 日志收集
- 诊断采样
- Release 部署
- Rollout 回滚
### 6.4 Start
`POST /api/v1/ops/agent/jobs/{job_id}/start`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02"
}
```
语义:
- `dispatching` 表示已派发
- `running` 表示节点已真实开工
### 6.5 Complete
`POST /api/v1/ops/agent/jobs/{job_id}/complete`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02",
"client_request_id": "complete-ops-123-1",
"status": "success",
"stdout": "",
"stderr": "",
"result": {
"summary": "ok"
},
"error_message": ""
}
```
允许状态:
- `success`
- `failed`
- `partially_succeeded`
幂等约束:
- 使用 `client_request_id`
- 控制面写入 `ops_jobs.last_agent_complete_request_id`
- 同一完成回执可安全重放,不应再次制造副作用
推荐补充字段:
- `duration_ms`
- `result.summary_text`
- `result.focus_ref`
当前成功返回里也会额外带:
- `completion_summary.job_id`
- `completion_summary.job_code`
- `completion_summary.status`
- `completion_summary.status_label`
- `completion_summary.result_summary_text`
- `completion_summary.focus_ref`
- `completion_summary.deduplicated`
这样控制面在不展开 stdout/stderr 的情况下,也能直接把:
- 任务收口摘要
- 后续跳转落点
并入 activity / inspection / rollout 观察面。
### 6.6 Events
`POST /api/v1/ops/agent/jobs/{job_id}/events`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02",
"client_event_id": "event-ops-123-checksum-1",
"step_id": 0,
"event_type": "deploy_checksum_verified",
"level": "info",
"message": "checksum ok",
"payload": {
"checksum": "sha256:..."
}
}
```
去重约束:
- 使用 `client_event_id`
- 控制面通过唯一索引去重
- 同一事件可安全重放,不应重复落库
推荐补充字段:
- `occurred_at`
- `summary_text`
- `focus_ref`
当前成功返回里也会额外带:
- `event`
- 即标准化后的事件对象
- 内含 `event_key / summary_text / occurred_at / focus_ref`
这样事件流就不再只是“原始日志点”,而能直接成为:
- playbook run events
- rollout events
- activity-stream
共同消费的现场片段。
---
## 7. Agent 本地补发队列
当前主链路已经正式具备:
- `pending` 队列
- `dead-letter` 队列
- 定时 `_flush_delivery_queue()`
- 永久性错误转死信
当前重点覆盖:
- `jobs/{id}/complete`
- `jobs/{id}/events`
队列状态:
- `healthy`
- `retrying`
- `dead_letter`
语义:
- `healthy`:无积压,无死信
- `retrying`存在待补发回执Agent 会继续自动重放
- `dead_letter`:存在永久失败记录,需人工介入
控制面消费方式:
- 节点维度:
- `delivery_queue_state`
- `delivery_queue_label`
- `delivery_queue_reason`
- `delivery_queue_pending_count`
- `delivery_queue_dead_letter_count`
- 汇总维度:
- `queue_retrying_nodes`
- `queue_dead_letter_nodes`
- `queue_pending_records`
- `queue_dead_letter_records`
### 7.1 当前控制面已可见的最小现场
当前至少已经可以通过:
- `GET /api/v1/ops/nodes`
看到每台托管节点的队列现场:
```json
{
"node_code": "mainland-worker-02",
"delivery_queue_state": "dead_letter",
"delivery_queue_label": "死信 2",
"delivery_queue_reason": "当前存在 2 条死信记录,建议优先查看节点日志或诊断编排。",
"delivery_queue_pending_count": 0,
"delivery_queue_dead_letter_count": 2,
"delivery_queue_last_flush_at": "2026-04-18 06:10:00",
"delivery_queue_oldest_pending_at": "",
"delivery_queue_oldest_pending_request_id": "",
"delivery_queue_oldest_pending_kind": "",
"delivery_queue_oldest_dead_letter_at": "2026-04-18 05:59:00",
"delivery_queue_oldest_dead_letter_request_id": "complete-ops-123-1",
"delivery_queue_oldest_dead_letter_kind": "job_complete"
}
```
这已经足够让:
- 页面显示“节点存在死信”
- `overview / driver-feed / codex-brief` 产出驾驶建议
- CLI 快速判断现场是不是要先走日志或诊断
但这还不够支撑正式运维,因为它只能“看见死信”,还不能“处理死信”。
### 7.2 Delivery Queue Record 正式对象
后续控制面不应再把死信理解成一个纯计数器,而要把每条待补发 / 死信记录升级成正式对象。
推荐最小结构:
```json
{
"record_id": "dq-mainland-worker-02-20260418-001",
"node_code": "mainland-worker-02",
"state": "dead_letter",
"request_kind": "job_complete",
"client_request_id": "complete-ops-123-1",
"job_id": 123,
"job_code": "ops-20260418-001",
"target_api": "/api/v1/ops/agent/jobs/123/complete",
"detail_code": "ops_job_invalid_status_transition",
"error_message": "当前任务状态不允许 complete",
"attempt_count": 6,
"first_queued_at": "2026-04-18 05:58:00",
"last_attempt_at": "2026-04-18 05:59:00",
"next_retry_at": "",
"payload_preview": {
"status": "success"
},
"response_preview": {
"code": 1,
"detail_code": "ops_job_invalid_status_transition"
}
}
```
正式约束:
- `record_id`
- 是控制面侧唯一主键
- `client_request_id`
- 是 Agent 幂等键
- `detail_code`
- 是死信聚类主键,不能只靠 `message`
- `payload_preview / response_preview`
- 用于页面和 Codex 快速判断,不必默认展开完整原始报文
推荐状态:
- `pending`
- `retrying`
- `dead_letter`
- `replayed`
- `discarded`
### 7.3 死信操作面正式入口
为了让“看见死信”之后不用回节点本地处理,推荐把操作面固定成下面这组接口。
当前已落地的第一阶段能力:
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
当前控制面返回的记录可见性为:
- `record_visibility=head_only`
也就是:
- 当前已经能稳定查看节点级队列总览
- 也能看到 `pending / dead_letter` 的头部记录摘要
- 但还没有把节点本地全量 `pending/*.json / dead-letter/*.json` 正式同步到控制面
所以这两条 GET 入口现在属于“正式只读观察面”,而不是完整操作面。
#### a. 单节点队列总览
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
最小返回体:
```json
{
"node_code": "mainland-worker-02",
"summary": {
"state": "dead_letter",
"pending_count": 0,
"dead_letter_count": 2,
"last_flush_at": "2026-04-18 06:10:00"
},
"oldest_pending_record": {},
"oldest_dead_letter_record": {}
}
```
#### b. 队列记录列表
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
推荐筛选参数:
- `state=pending|retrying|dead_letter|replayed|discarded`
- `request_kind=job_complete|job_event`
- `detail_code=...`
- `group_by=detail_code|request_kind|target_api`
- `limit=50`
当带 `group_by` 时,返回值应优先给出聚类摘要,而不是只返回平铺明细。
#### c. 单条死信重放
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay`
当前已落地,但要明确当前阶段约束:
- 控制面仍然是 `record_visibility=head_only`
- 所以单条动作目前只允许针对“当前可见头部记录”
- 真正的执行不是控制面直接改远端文件,而是创建正式 `ops job`
- 由目标节点 `Node Agent` 执行 `delivery.queue.replay`
最小请求体:
```json
{
"requested_by": "web-ui",
"reason": "确认任务状态已修复,重放这条 complete 回执"
}
```
#### d. 批量重放
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/replay`
当前已落地为正式操作面。
推荐请求体:
```json
{
"requested_by": "codex",
"selector": {
"state": "dead_letter",
"detail_code": "ops_job_invalid_status_transition"
},
"limit": 20
}
```
#### e. 丢弃死信
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard`
当前已落地,但与单条重放一样,仍然只允许针对当前可见头部死信记录发起单条动作。
最小请求体:
```json
{
"discarded_by": "web-ui",
"reason": "确认这条回执不再需要补发"
}
```
#### f. 主动冲刷队列
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/flush`
当前已落地为正式操作面,执行方式同样是:
- 控制面创建 `ops job`
- 目标节点 `Node Agent` 执行 `delivery.queue.flush`
语义:
- 不是“强行成功”
- 而是立即触发一次 Agent 补发冲刷
- 结果仍然回到 `pending / retrying / dead_letter / replayed`
### 7.4 为什么必须做成正式操作面
这层不是为了让页面多一个按钮,而是为了保证:
- 页面可以处理死信
- CLI 可以处理死信
- 海外 Codex 驾驶员可以处理死信
三者都不需要绕回:
- 手工 SSH 登录节点
- 手工删本地文件
- 手工重放某条 HTTP 请求
正式规则应该固定成:
- “死信查看” 走控制面对象
- “死信重放 / 丢弃 / 冲刷” 走控制面对象
- Agent 只负责执行,不负责决定如何处理死信
---
## 8. Job Status Enum
当前 Agent 主链路约束:
- `queued`
- `dispatching`
- `running`
- `success`
- `failed`
- `partially_succeeded`
语义:
- `queued`:任务已创建,尚未派发
- `dispatching`:控制面已派发,但节点尚未确认开工
- `running`:节点已真实执行
- `success`:成功完成
- `failed`:失败完成
- `partially_succeeded`:部分成功,需关注
---
## 9. 实现边界
Agent 只执行已声明动作:
- `service.start`
- `service.stop`
- `service.restart`
- `service.status`
- `health.snapshot`
- `logs.collect`
- `diagnostics.collect`
- `deploy.release`
不允许控制面长期依赖任意 shell 下发。
`shell executor` 只能作为受限兜底能力存在,并必须挂审批与审计。

View File

@@ -0,0 +1,56 @@
# ops_doctor_decision_contract
## Purpose
冻结 `doctor-decision` 主决策 contract让页面、CLI、Codex 都消费同一份“当前先看哪一面、先执行什么动作、为什么”的正式结论。
## Producer
- backend service `app.services.ops_service.get_ops_doctor_decision`
- API `GET /api/v1/ops/doctor-decision`
## Primary Consumers
- `domain-web` Ops Center 顶部总检入口与主决策详情面板
- `domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision`
- 海外 Codex 驾驶员 / 自动驾驶策略层
## Required Payload
- `status`
- `status_label`
- `headline`
- `detail`
- `decision`
- `summary`
- `recommended_commands`
- `contract_navigation`
## Decision Shape
- `decision.status`
- `decision.reason_code`
- `decision.headline`
- `decision.detail`
- `decision.preferred_surface`
- `decision.next_action_code`
- `decision.recommended_commands`
## Summary Shape
- `summary.required_failures`
- `summary.optional_unavailable`
- `summary.scene_log_reports_total`
- `summary.scene_log_reports_ok`
- `summary.scene_log_status_counts`
- `summary.scene_log_problem_node_codes`
- `summary.contract_surface_gaps`
- `summary.launchpad_recommended_target_node_code`
- `summary.launchpad_recommended_recovery_label`
- `summary.launchpad_recommended_recovery_summary`
## Notes
- 优先读取最新 `go-live bundle` 中的 `doctor_decision` 产物。
- 如果 bundle 尚未导出,后端必须回落到 live `stack-diagnosis + go-live-summary` 兜底,而不是直接返回空。
- `preferred_surface` 必须稳定可消费,不能把面板跳转逻辑继续留给前端猜测。

View File

@@ -0,0 +1,797 @@
# domainCheck Ops Driver Contract
## 1. 目标
这份文档用于冻结 Overseas Ops Center / Codex Driver / CLI 共用的驾驶 contract。
核心原则:
- 后端负责推荐和解释
- 前端负责展示和触发
- Codex 负责选择入口和上下文
- 是否允许自动执行由后端统一门禁
- 页面与 Codex 都不能各自再推导一套优先级
---
## 2. 入口接口
当前驾驶 contract 主要来自:
- `GET /api/v1/ops/overview`
- `GET /api/v1/ops/driver-feed`
- `GET /api/v1/ops/codex-brief`
- `POST /api/v1/ops/driver-actions/preview`
- `POST /api/v1/ops/driver-actions/resolve`
- `POST /api/v1/ops/driver-actions/execute`
- `POST /api/v1/ops/driver-actions/execute-resolved`
- `POST /api/v1/ops/codex-actions/resolve`
- `POST /api/v1/ops/codex-actions/execute`
补充配套:
- `GET /api/v1/ops/playbooks`
- `GET /api/v1/ops/playbook-runs`
- `GET /api/v1/ops/playbook-runs/{run_code}`
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
- `GET /api/v1/ops/activity-stream`
---
## 3. Driver Recommendation
来源:
- `overview.driver_recommendations`
最小字段:
```json
{
"key": "node-agent-dead-letter",
"priority": "先处理 Agent 死信",
"title": "先处理 Agent 死信",
"summary": "当前已有节点出现死信记录,继续堆动作会放大控制面与节点现场的不一致。",
"reason": "Node Agent 回执队列已经出现死信。",
"level_label": "最高优先",
"tag_type": "danger",
"primary_label": "查看 Worker 日志",
"secondary_label": "打开标准巡检",
"primary_action_code": "open_worker_logs",
"secondary_action_code": "open_diagnostics",
"node_codes": [
"mainland-worker-01"
],
"primary_node_codes": [
"mainland-worker-01"
],
"secondary_node_codes": [
"mainland-worker-01"
],
"primary_action_payload": {},
"secondary_action_payload": {},
"focus_ref": {
"kind": "ops_job",
"job_id": 101,
"job_code": "ops-20260418064500-a1b2c3"
}
}
```
要求:
- 推荐不只给文案
- 必须给动作码、节点和预填 payload
- 主次动作允许节点和 payload 不同
- 推荐对象允许直接携带 `focus_ref / primary_focus_ref / secondary_focus_ref`
- 新客户端优先用 `focus_ref` 做统一定位,而不是再从动作码反推页面落点
---
## 4. Priority Recommendation
来源:
- `overview.recommendation`
作用:
- 给首页顶部、CLI 摘要、Codex 首屏判断提供“当前第一优先动作”
最小字段:
```json
{
"source": "driver_recommendations",
"key": "node-agent-dead-letter",
"priority": "先处理 Agent 死信",
"title": "先处理 Agent 死信",
"summary": "当前已有节点出现死信记录。",
"reason": "Node Agent 回执队列已经出现死信。",
"primary_action_code": "open_worker_logs",
"secondary_action_code": "open_diagnostics"
}
```
---
## 5. Driver Feed
来源:
- `GET /api/v1/ops/driver-feed`
作用:
- 提供比 `overview` 更直接给驾驶员消费的聚合结果
建议最小结构:
```json
{
"top_recommendation": {},
"driver_recommendations": [],
"runbook_sequences": [],
"activity_focus": [],
"automation_coverage": {}
}
```
推荐补充约束:
- `top_recommendation`
- 应该总是来自同一份 `entries` 列表,而不是额外拼一条“影子建议”
- 这样页面、CLI、海外 Codex 在按 `key` 选中时才不会出现口径漂移
- `driver_recommendations`
- 普通驾驶建议,优先 `executor_kind=driver_action`
- `runbook_sequences`
- 标准作业路径,优先 `executor_kind=runbook_sequence`
- `activity_focus`
- 更偏观察与跟踪,不应伪装成“可立即执行”的驾驶动作
- `activity_focus[*]`
- 至少应保留:
- `activity_key`
- `activity_kind`
- `summary`
- `status`
- `status_label`
- `occurred_at`
- `focus_ref`
- `source_focus_ref`
- `ui_intent`
- 如果存在可落点观察入口
- 允许补 `focus_action_code=focus_activity_item`
- 但它的定位仍然是观察入口,而不是标准执行动作
- `entries[*].focus_ref`
- 是 driver-feed 的统一定位对象
- 页面 / CLI / Codex 均应优先消费
- `entries[*].primary_focus_ref` / `entries[*].secondary_focus_ref`
- 是主次动作各自的稳定落点
- 不允许调用方再根据 `primary_action_code` 自己猜“该跳 Release Hub、Launchpad 还是 Rollout Jobs”
- `automation_coverage`
- 是 Driver Feed 的自动化收口摘要
- 用于统一回答:
- 当前是否已经达到上线闸门
- 当前还有多少动作停留在 preview-only
- 当前后端是否已经足够接管首屏默认动作
- 页面、CLI、Codex 不允许自行再统计一份“自动化覆盖率”
推荐最小结构:
```json
{
"launch_ready": false,
"launch_status": "attention",
"summary": "仍有 3 个驾驶动作停留在 preview-only暂未达到上线收口标准。",
"total_actions": 12,
"backend_handled_total": 9,
"preview_only_total": 3,
"ui_only_total": 0,
"blocked_total": 0
}
```
其中 `runbook_sequences``activity_focus` 应继续区分:
- 这是建议动作
- 还是固定标准作业路径
- 还是最近值得优先处理的活动
推荐 CLI 消费顺序:
1. 显式 `entry_key`
2. `top_recommendation`
3. `entries[0]`
这样海外单入口就能稳定实现:
- `driver-focus-preview`
- `driver-focus-run`
---
## 6. Codex Brief
来源:
- `GET /api/v1/ops/codex-brief`
作用:
- 不让 Codex 自己从页面文案里猜“能不能自动执行”
最小字段:
```json
{
"mode": "guarded_auto",
"recommended_behavior": "confirm_then_execute",
"summary": "当前建议先处理 Agent 死信,再进入标准巡检。",
"automation_coverage": {
"launch_ready": false,
"launch_status": "attention",
"summary": "仍有 3 个驾驶动作停留在 preview-only。"
},
"target": {
"kind": "driver_action",
"action_code": "open_worker_logs",
"node_codes": [
"mainland-worker-01"
],
"action_payload": {}
}
}
```
推荐 `mode`
- `safe_auto`
- `guarded_auto`
- `mixed`
- `ui_only`
- `blocked`
推荐 `recommended_behavior`
- `auto_execute`
- `confirm_then_execute`
- `resolve_first`
- `open_ui`
- `blocked`
补充约束:
- `codex-brief.automation_coverage`
- 应与 `driver-feed.automation_coverage` 保持同源
- 供海外 Codex 驾驶员直接判断:
- 是否允许默认走自动执行
- 是否必须先预览或人工确认
- 是否已经达到“可以上线”的自动化收口标准
- `codex-brief.activity_focus`
- 应直接透传 `driver-feed.activity_focus`
- 供 Codex 在“当前没有可执行主线,但有观察焦点”时继续判断
- `codex-brief.focus`
- 优先使用 `entries[0]`
-`entries` 为空且 `activity_focus` 不为空时
- 允许回退到首条带 `focus_action_code``activity_focus`
- 这时推荐行为应偏向 `open_ui`
- 不应伪装成自动执行动作
---
## 7. Driver Action Execute
来源:
- `POST /api/v1/ops/driver-actions/execute`
作用:
- 把能标准化的驾驶动作收进统一后端执行入口
最小请求体:
```json
{
"action_code": "run_standard_inspection",
"node_codes": [
"mainland-worker-01"
],
"action_payload": {}
}
```
最小返回体:
```json
{
"handled": true,
"mode": "playbook",
"ui_intent": {
"kind": "playbook_run_detail",
"run_code": "run-20260418-001"
}
}
```
要求:
- 不只返回 handled / unhandled
- 还要返回 `mode`
- 还要返回 `ui_intent`
---
## 8. Driver Action Preview
来源:
- `POST /api/v1/ops/driver-actions/preview`
作用:
- 给 OpsCenter / CLI / Codex Driver 统一生成驾驶动作契约预览
- 冻结 `execution_chain / target_api / request_payload / 主次动作 section`
- 不再让前端自己拼请求体和执行建议
最小请求体:
```json
{
"source_label": "driver-feed",
"title": "版本发布与放量",
"summary": "当前可以进入正式发布路径。",
"reason": "统一回到 Release / Rollout 闭环。",
"executor_kind": "runbook_sequence",
"sequence_key": "release_progression",
"primary_label": "Worker 灰度",
"secondary_label": "查看版本区",
"secondary_action_code": "focus_release_hub",
"requested_by": "web-ui/ops-center"
}
```
最小返回体:
```json
{
"contract_key": "ops_driver_contract",
"contract_version": "v1",
"contract_schema_doc_path": "docs/schemas/ops_driver_contract.md",
"execution_chain": "ops-driver -> runbook-sequence -> resolved driver action / ui-intent",
"focus_ref": {
"kind": "release_hub",
"release_id": 7,
"section": "release_launchpad"
},
"primary_focus_ref": {
"kind": "release_hub",
"release_id": 7,
"section": "release_launchpad",
"mode": "worker"
},
"secondary_focus_ref": {
"kind": "release_hub",
"release_id": 7,
"section": "release_detail"
},
"primary": {
"label": "Worker 灰度",
"action_code": "create_release_rollout_worker",
"target_api": "/api/v1/ops/runbook/sequences/release_progression/execute",
"target_method": "POST",
"recommendation_label": "建议确认后执行",
"automation_level_label": "guarded_auto",
"risk_level_label": "high",
"focus_ref": {
"kind": "release_hub",
"release_id": 7,
"section": "release_launchpad",
"mode": "worker"
},
"request_payload": {
"secondary": false,
"requested_by": "web-ui/ops-center",
"node_codes": [
"mainland-controller-01"
],
"action_payload": {
"release_id": 7,
"focus_ref": {
"kind": "release_hub",
"release_id": 7,
"section": "release_launchpad",
"mode": "worker"
}
}
}
}
}
```
要求:
- preview 必须和 execute 走同一套 action 解析与 runbook resolve 逻辑
- `primary` / `secondary` section 必须可直接给页面抽屉和 CLI 文本渲染复用
- `request_payload` 必须是真正可发送到执行接口的 body而不是页面内部临时对象
- runbook sequence preview 必须经过后端 resolve不能由页面自己猜 action_code
- preview 返回体必须保留:
- `focus_ref`
- `primary_focus_ref`
- `secondary_focus_ref`
- `primary.focus_ref`
- `secondary.focus_ref`
---
## 8.3. Driver Action Resolve / Execute Resolved
来源:
- `POST /api/v1/ops/driver-actions/resolve`
- `POST /api/v1/ops/driver-actions/execute-resolved`
作用:
-`overview.driver_recommendations` / `driver-feed` / 页面按钮 / CLI 对普通驾驶动作的门禁判断统一下沉到后端
- 页面不再根据 `preview` 自己推导 `直接执行 / 等确认 / 进入 UI / 阻断`
- 后端统一输出:
- `preview_request`
- `preview`
- `gate`
- `selected_section`
- `execution_result`
`resolve` 最小请求体:
```json
{
"source_label": "driver-feed",
"title": "开启关键日志回传",
"executor_kind": "driver_action",
"primary_action_code": "enable_log_sync_key",
"primary_action_payload": {},
"requested_by": "web-ui/driver-feed",
"secondary": false,
"confirm": false
}
```
`resolve` 最小返回体:
```json
{
"preview_request": {},
"preview": {},
"selected_section": "primary",
"focus_ref": {
"kind": "execution_scene",
"scene_key": "scene.logs"
},
"gate": {
"decision": "execute_now",
"decision_label": "直接执行",
"decision_reason": "当前动作属于 safe_auto可直接执行。",
"will_execute": true,
"recommendation": "auto_execute",
"focus_ref": {
"kind": "execution_scene",
"scene_key": "scene.logs"
},
"action_code": "enable_log_sync_key",
"target_api": "/api/v1/ops/driver-actions/execute"
}
}
```
`execute-resolved` 约束:
- 只有 `gate.will_execute=true` 才允许真正执行
- `confirmation_required` 只有显式传入 `confirm=true` 才会放行
- `open_ui / resolve_first / blocked` 必须返回非执行态,由页面或 CLI 继续展示契约
- `driver-focus-preview / driver-focus-run`
- 允许先从 `driver-feed.top_recommendation` 选出目标项
- 也允许显式传入 `activity_focus.key`
- 当目标来自 `activity_focus`
- 应自动收口成 `focus_activity_item`
- 只允许进入对应工作区,不应被当成标准执行动作
- 再把它收口成同一份 `resolve / execute-resolved` 请求体
- 不允许 CLI 再自己二次推导 gate
- resolve 返回体必须保留:
- `preview_request.focus_ref`
- `preview_request.primary_focus_ref`
- `preview_request.secondary_focus_ref`
- `gate.focus_ref`
- 顶层 `focus_ref`
---
## 8.5. Codex Action Resolve / Execute
来源:
- `POST /api/v1/ops/codex-actions/resolve`
- `POST /api/v1/ops/codex-actions/execute`
作用:
- 不再让 CLI / Codex 自己从 `codex-brief + preview` 拼一套门禁
- 由后端统一输出:
- 当前选中的 `selected_entry`
- 对应的 `preview`
- 统一门禁结论 `gate`
- 最终是否允许执行
`resolve` 最小请求体:
```json
{
"entry_key": "runbook:release_progression",
"requested_by": "cli/ops-center/codex-focus-preview",
"source_label": "cli/codex-focus-preview",
"include_secondary": true,
"confirm": false
}
```
`resolve` 最小返回体:
```json
{
"entry_key": "runbook:release_progression",
"selected_entry": {},
"preview_request": {},
"preview": {},
"focus_ref": {
"kind": "release_hub",
"release_id": 7,
"section": "release_launchpad",
"mode": "worker"
},
"gate": {
"decision": "confirmation_required",
"decision_label": "等待确认",
"recommendation": "confirm_then_execute",
"recommendation_label": "建议确认后执行",
"will_execute": false,
"confirm_required": true,
"focus_ref": {
"kind": "release_hub",
"release_id": 7,
"section": "release_launchpad",
"mode": "worker"
},
"target_api": "/api/v1/ops/runbook/sequences/release_progression/execute",
"target_method": "POST",
"request_payload": {
"secondary": false,
"requested_by": "cli/ops-center/codex-focus-preview",
"action_payload": {
"release_id": 7
}
}
}
}
```
`execute` 要求:
- 必须复用 `resolve` 的同一套选中逻辑和门禁逻辑
- `safe_auto`
- 允许直接执行
- `guarded_auto`
- 只有 `confirm=true` 时允许执行
- `resolve_first / open_ui / blocked`
- 一律不得执行
- 返回体必须保留:
- `selected_entry`
- `preview_request`
- `preview`
- `gate`
- `focus_ref`
- `execution_result`
- `executed`
---
## 9. UI Intent
`ui_intent` 的职责不是让前端猜去哪,而是由后端直接声明:
- 应打开哪个详情
- 应聚焦哪轮 run
- 应落到哪个发布入口
当前建议固定支持:
- `playbook_run_detail`
- `playbook_run_latest_events`
- `job_events`
- `rollout_jobs`
- `managed_node_handover`
- `managed_node_edit`
- `open_release_dialog`
- `open_rollout_dialog`
- `focus_release_hub`
- `focus_ref`
页面只做路由和抽屉承载,不做推荐解释。
---
## 10. Runbook Sequence
作用:
- 表达固定标准作业路径在“此刻”的推荐下一步
最小字段:
```json
{
"key": "standard_inspection",
"title": "标准巡检",
"status": "warning",
"status_label": "待补接管",
"summary": "当前还没有满足标准巡检条件的节点。",
"target_node_codes": [],
"focus_ref": {
"kind": "release_hub",
"release_id": 7,
"section": "release_launchpad"
},
"primary_label": "直接执行标准巡检",
"primary_action_code": "run_standard_inspection",
"primary_action_payload": {},
"primary_focus_ref": {},
"secondary_label": "打开巡检编排",
"secondary_action_code": "open_playbook_dialog",
"secondary_action_payload": {
"playbook_key": "inspection.standard"
},
"secondary_focus_ref": {}
}
```
要求:
- Runbook Sequence 是标准路径对象
- 不是普通活动记录
- 不是普通推荐卡
- `focus_ref`
- 表示这条标准路径整体对应的统一观察落点
- `primary_focus_ref` / `secondary_focus_ref`
- 表示主次动作各自应回看的稳定区域
---
## 11. Activity Stream
来源:
- `GET /api/v1/ops/activity-stream`
作用:
- 把最近异常、编排、任务、Rollout 统一压成可钻取活动流
每条 activity 最小字段:
```json
{
"kind": "playbook_run",
"activity_key": "run-20260418-001",
"status": "failed",
"occurred_at": "2026-04-18 06:00:00",
"summary": "标准巡检在 diagnostics.collect 步骤失败。",
"ui_intent": {
"kind": "playbook_run_detail",
"run_code": "run-20260418-001"
}
}
```
推荐 `kind`
- `playbook_run`
- `ops_job`
- `rollout`
- `runbook_sequence`
---
## 12. 单脑驾驶舱首屏约束
Ops Center 首屏不是自由拼装页面,而是 Driver Contract 的正式消费者。
首屏必须优先消费同一组 contract
- `overview.recommendation`
- `driver-feed.top_recommendation`
- `driver-feed.automation_coverage`
- `codex-brief`
首屏建议固定为三层:
1. 驾驶结论
2. 驾驶动作条
3. 驾驶回执条
正式要求:
- 驾驶结论
- 直接展示当前第一优先动作、自动化收口状态、是否达到上线闸门
- 驾驶动作条
- 只放“默认下一步 / 定位下一步 / 看契约 / 复制命令 / 进入主处理面板”这类高频动作
- 这些按钮不能自己发明新规则,必须消费已有 `action_code / focus_ref / preview`
- 驾驶回执条
- 是首屏动作后的统一反馈区
- 用于展示:
- 已打开哪个契约
- 已定位哪个焦点
- 已复制哪个命令
- 后端动作执行成功 / 警告 / 失败
- 页面不应再把动作结果散落在多个局部 toast 里
也就是说:
- 首屏负责决策与触发
- 下方详情区负责展开与下钻
- 不允许详情区反向重写首屏推荐
---
## 13. 建议优先级顺序
当前 contract 应优先遵守:
1. 先看异常编排
2. 再看异常活动
3. 再处理 Agent 死信 / 回执积压
4. 再优先执行首个缺口节点的接入收口或接管验收
5. 再处理现场日志回传和标准巡检
6. 最后进入 Release / Rollout
这条顺序必须由后端维护,不应让前端或 Codex 再自行拼装。
补充约束:
- 当后端已经把首个缺口节点收敛成 `bootstrap_run / run_acceptance` 时:
- `driver_feed.top_recommendation`
- `codex_brief.focus`
- `stack_diagnosis.next_step`
都必须优先指向这一个节点和这一个动作
- `fix_managed_nodes` 只能保留为兜底入口,不能再覆盖已经明确收敛的单节点恢复动作
---
## 14. 页面 / Codex / CLI 共用约束
同一条驾驶建议必须由后端一次性给出:
- 推荐标题
- 推荐理由
- 主动作
- 次动作
- 节点范围
- 预填参数
- UI 落点
- 是否允许自动执行
这样三类入口才会真正共享一套驾驶 contract
- OpsCenter 页面
- 海外 Codex 驾驶员
- CLI / 运维检查脚本
并且三类入口对 `recommended_behavior` 的执行语义也必须一致:
- `auto_execute`
- 可直接执行
- `confirm_then_execute`
- 必须显式确认后才能执行
- `resolve_first`
- 只能先审阅和补上下文,不能直接执行
- `open_ui`
- 只能进入对应工作区,不能直接执行
- `blocked`
- 明确阻断,不能执行

View File

@@ -0,0 +1,59 @@
# ops_go_live_bundle_contract
## 目标
把发布前证据包与 `manifest.json` 复核结论压成一份统一 contract
- Ops Center 交付面板
- CLI `go-live-export / go-live-review`
- 海外 Codex 驾驶员
共同消费。
它解决的问题不是“现在能不能上线”,而是:
> 现在有没有一份足够完整、足够可信、可交付可复盘的上线证据包
## 主接口
- `GET /api/v1/ops/go-live-bundle`
## 核心字段
- `status`
- `missing`
- `ready`
- `attention`
- `blocked`
- `status_label`
- `headline`
- `bundle_available`
- `report_dir`
- `manifest_path`
- `artifact_total`
- `failure_total`
- `critical_failures`
- `noncritical_failures`
- `env_audit`
- `summary`
- `recommended_commands`
## 判定原则
- 如果当前还没有 bundle / manifest
- `status = missing`
- 如果 manifest 明确标记 `review_status_hint=blocked`
- `status = blocked`
- 如果 manifest 已存在,但:
- 仍有非关键失败
- 环境审计提示 attention
- launchpad 摘要存在漂移
-`status = attention`
- 只有当 manifest 可读、review 为 ready、关键摘要一致
- 才允许 `status = ready`
## 设计约束
- 页面不直接触发 shell 导出 bundle
- 后端默认只读最新 bundle / manifest不负责替代 CLI 执行导出
- 当 bundle 缺失时,必须明确给出固定命令,而不是让操作者自己猜路径

View File

@@ -0,0 +1,37 @@
# ops_go_live_review_contract
## Purpose
冻结 `go-live-review` 复核结论 contract让页面、CLI、Codex 都能消费同一份 bundle manifest 复核结果,而不是各自再解读一遍 artifact。
## Producer
- backend service `app.services.ops_service.get_ops_go_live_review`
- API `GET /api/v1/ops/go-live-review`
- CLI `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review`
## Primary Consumers
- `domain-web` Ops Center 证据复核面板
- 海外 Codex 驾驶员
- 发布前人工值班复核
## Required Payload
- `status`
- `status_label`
- `headline`
- `artifact_total`
- `failure_total`
- `critical_failures`
- `noncritical_failures`
- `env_audit`
- `launchpad_alignment`
- `recommended_next_steps`
- `recommended_commands`
## Notes
- 这层是 manifest 的正式“复核解释层”,不是单纯重复 bundle 基础信息。
- 如果 `launchpad_alignment.consistent=false`,即使关键文件都存在,也必须至少回到 `attention`
- 页面和 Codex 应优先消费这层 `headline / recommended_next_steps`,而不是自行拼 review 口径。

View File

@@ -0,0 +1,65 @@
# ops_go_live_signoff_contract
## 目标
把发布前最终签收判断压成一份统一 contract
- Ops Center 首屏
- CLI `go-live-signoff`
- 海外 Codex 驾驶员
共同消费。
它不是替代:
- `go-live-summary`
- `go-live-review`
- `doctor-decision`
- `stack-diagnosis`
- `driver-feed`
- `codex-brief`
而是在这些 contract 之上给出最终一句:
> 现在能不能签字上线
## 主接口
- `GET /api/v1/ops/go-live-signoff`
## 核心字段
- `signoff_status`
- `ready`
- `attention`
- `blocked`
- `status_label`
- `headline`
- `release_gate`
- `blocked_reasons`
- `attention_reasons`
- `decision`
- `launchpad_alignment`
- `recommended_commands`
- `report_dir`
- `manifest_path`
## 判定原则
- 只要 `go_live_summary / go_live_review / doctor_decision / publish_status / stack_status / automation / release_launchpad` 中任一明确阻断
- `signoff_status = blocked`
- 如果没有阻断,但存在:
- `go-live-review=attention|missing`
- `doctor-decision=attention|missing`
- `attention`
- launchpad 摘要漂移
- 自动化仍未完全进入 ready
-`signoff_status = attention`
- 只有当收口、正式复核、主决策、发布、自动化、launchpad 摘要一致
- 才允许 `signoff_status = ready`
## 设计约束
- 页面不允许再自己拼一套“看起来能上”
- CLI 不允许绕过这个 contract 单独下结论
- Codex 驾驶员优先看这一份,再决定是否继续下钻

View File

@@ -0,0 +1,740 @@
# domainCheck Ops Job Contract
## 1. 目标
这份文档用于冻结 `ops job` 的正式 contract。
它是整套海外单脑控制面里最核心的执行对象。
正式定义:
> `ops job` 是一条可审计、可编排、可派发、可回执、可取消的标准运维任务记录。
它不是:
- 一条 shell 命令
- 一个页面按钮点击结果
- 一个仅供 agent 消费的内部对象
它应该同时服务于:
- 页面按钮
- CLI
- 海外 Codex 驾驶员
- Node Agent
- Release / Rollout
- Playbook Run
也就是说:
- `playbook run`
- 是编排聚合对象
- `rollout`
- 是发布推进对象
- `ops job`
- 是真正的执行颗粒度对象
---
## 2. 入口接口
当前已落地的主入口:
- `GET /api/v1/ops/jobs`
- `GET /api/v1/ops/jobs/{job_id}`
- `GET /api/v1/ops/jobs/{job_id}/events`
- `POST /api/v1/ops/jobs`
- `POST /api/v1/ops/jobs/batch`
- `POST /api/v1/ops/policy/preview`
- `POST /api/v1/ops/jobs/{job_id}/approve`
- `POST /api/v1/ops/jobs/{job_id}/cancel`
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
配套对象来源:
- `ops_jobs`
- `ops_job_steps`
- `ops_job_events`
正式约束:
- 页面、CLI、Codex 不能各自定义第 2 套任务模型
- `playbook run` 只能聚合 `ops job`
- `rollout` 只能批量生成 `ops job`
- `Node Agent` 只能拉取和回执 `ops job`
---
## 3. 当前实现里的 Job Object
来源:
- `GET /api/v1/ops/jobs`
- `GET /api/v1/ops/jobs/{job_id}`
- 创建 / 审批 / 取消 / 派发后的返回体中的 `job`
最小字段:
```json
{
"id": 101,
"job_code": "ops-20260418064500-a1b2c3",
"action": "logs.collect",
"target_type": "node",
"target_node_code": "mainland-worker-01",
"target_node_codes": [
"mainland-worker-01"
],
"status": "queued",
"status_label": "排队中",
"execution_mode": "remote-agent",
"execution_mode_label": "Node Agent",
"requested_by": "web-ui/ops-center",
"payload": {},
"metadata": {},
"result": {},
"summary": "目标节点 mainland-worker-01 / 发起 web-ui/ops-center / 方式 Node Agent",
"summary_text": "目标节点 mainland-worker-01 / 发起 web-ui/ops-center / 方式 Node Agent",
"error_message": "",
"created_at": "2026-04-18 06:45:00",
"started_at": "",
"finished_at": "",
"updated_at": "2026-04-18 06:45:00",
"risk_level": "medium",
"approval_required": false,
"approval_status": "approved",
"approval_status_label": "已审批",
"approved_by": "",
"approved_at": "",
"blocked_reason": "",
"cancellation_reason": "",
"dispatched_at": "",
"target_selector": {},
"policy": {},
"rollout_id": 0,
"steps_total": 0,
"steps_running": 0,
"steps_success": 0,
"steps_failed": 0,
"steps_terminal": 0,
"steps_loaded": false,
"focus_ref": {
"kind": "ops_job",
"job_id": 101,
"job_code": "ops-20260418064500-a1b2c3",
"action": "logs.collect",
"target_node_code": "mainland-worker-01"
},
"steps": []
}
```
正式要求:
- `job_code`
- 是跨页面、CLI、事件流的稳定任务标识
- `action`
- 是执行语义主键,不应用 `title` 或自然语言代替
- `target_type`
- 定义任务作用范围,不应只靠是否传了 `target_node_code` 来猜
- `execution_mode`
- 是正式执行方式,不是附属备注
- `policy`
- 是创建时冻结的策略快照,不应由页面二次推导
- `status_label / approval_status_label`
- 是给页面、CLI、Codex 直接消费的人类可读标签
- `summary_text`
- 是任务摘要 contract不要求调用方再用状态和节点自己拼一句中文
- `steps_total / steps_running / steps_success / steps_failed / steps_terminal`
- 是任务步骤聚合统计,避免前端和 CLI 再自行遍历 steps 统计
- `focus_ref`
- 是统一定位对象后续页面、CLI、Codex 应优先消费它,而不是自己重组跳转参数
---
## 4. Job 状态机
### 4.1 当前实现中已使用的状态
当前后端已实际使用:
- `queued`
- `awaiting_approval`
- `blocked`
- `running`
- `success`
- `failed`
- `cancelled`
这些状态已经是正式 contract 的最小闭环。
### 4.2 推荐终局扩展状态
终局可继续扩展为:
- `draft`
- `approved`
- `queued`
- `dispatching`
- `running`
- `verifying`
- `partially_succeeded`
- `rollback_running`
- `rolled_back`
- `success`
- `failed`
- `cancelled`
- `timed_out`
但要明确区分:
- 当前实际 API 已稳定承诺的,是最小状态集
- 终局扩展状态不能破坏当前状态语义
---
## 5. Execution Mode Contract
当前实现里正式出现的执行方式:
- `remote-agent`
- `local-runtime`
- `control-plane`
- `ssh`
语义必须固定:
### 5.1 `remote-agent`
- 默认正式执行方式
- 由目标节点 Node Agent 拉取并执行
### 5.2 `local-runtime`
- 仅用于当前节点本机即时执行
- 主要用于本机 runtime 动作
### 5.3 `control-plane`
- 由海外控制面直接执行
- 适用于控制面内生动作
### 5.4 `ssh`
- 只作为救援和过渡执行器
- 不能成为大规模正式发布的默认方式
正式约束:
- 发布默认优先 `remote-agent`
- `ssh` 只应是兜底链路
- 页面、CLI、Codex 不应再自行创造第 5 种 execution mode
---
## 6. Target Type Contract
当前 contract 至少应允许:
- `node`
- `batch`
- `nodes`
- `rollout`
语义建议固定为:
### 6.1 `node`
- 单节点任务
### 6.2 `batch` / `nodes`
- 多节点批量任务
- 当前通常通过 `/ops/jobs/batch` 创建
### 6.3 `rollout`
- 该任务属于某轮 release rollout 推进的一部分
正式要求:
- 如果任务来自 rollout必须能通过 `rollout_id` 回溯
- 如果任务来自 playbook应通过 `metadata` 标出 run 归属
---
## 7. Job Step Contract
来源:
- `GET /api/v1/ops/jobs/{job_id}`
最小字段:
```json
{
"id": 201,
"step_key": "dispatch",
"title": "调度动作",
"node_code": "mainland-worker-01",
"status": "queued",
"stdout": "",
"stderr": "",
"result": {},
"created_at": "2026-04-18 06:45:00",
"started_at": "",
"finished_at": "",
"updated_at": "2026-04-18 06:45:00"
}
```
当前实现说明:
- 现阶段每条 `ops job` 至少会先生成一个 `dispatch` step
- 以后如果引入真正 DAG / 多步骤执行,也应继续沿用 `ops_job_steps`
正式约束:
- `step_key`
- 是步骤主键,不应用标题代替
- `status`
- 必须与 job 状态联动,但不要求完全相同
- `stdout / stderr`
- 是步骤级输出,不应与整条 job 的 `result` 混成一层
---
## 8. Job Event Contract
来源:
- `GET /api/v1/ops/jobs/{job_id}/events`
最小字段建议:
```json
{
"id": 301,
"event_key": "job-event:301",
"job_id": 101,
"step_id": 201,
"node_code": "mainland-worker-01",
"client_event_id": "evt-001",
"event_type": "job_dispatch_requested",
"level": "info",
"level_label": "信息",
"message": "任务等待 node-agent 拉取: 101",
"summary": "任务等待 node-agent 拉取: 101",
"payload": {},
"has_payload": false,
"occurred_at": "2026-04-18 06:45:03",
"created_at": "2026-04-18 06:45:03"
}
```
当前代码里事件来源包括:
- job created
- job blocked
- job awaiting approval
- job approved
- job cancelled
- job dispatch requested
- job executed locally / over ssh / on control plane
- agent start / complete / custom events
正式要求:
- `events` 是任务时间线,不是调试附属品
- 页面、CLI、Codex 都应从这里读“任务做到哪一步了”
- 不应重新从 stdout/stderr 推断关键状态
补充约束:
- `event_key`
- 是事件行的稳定前端 / CLI 标识
- `summary`
- 是给页面、CLI、Codex 直接消费的短摘要,不要求调用方再拼接 `message`
- `level_label`
- 是面向人看的等级标签
- `occurred_at`
- 是统一观察面时间字段,允许与 `created_at` 同值
### 8.1 Event List Summary Contract
当前 `GET /api/v1/ops/jobs/{job_id}/events` 建议同时返回:
```json
{
"summary": {
"job_id": 101,
"returned_total": 12,
"level_counts": {
"info": 8,
"warning": 3,
"error": 1
},
"event_type_counts": {
"job_created": 1,
"job_dispatch_requested": 1,
"executor_received": 1
},
"node_counts": {
"mainland-worker-01": 12
},
"latest_at": "2026-04-18 06:45:08",
"filters": {
"limit": 50
}
}
}
```
正式要求:
- 事件接口不只给一串数组
- 至少要给:
- 当前返回了多少条
- 等级分布
- 事件类型分布
- 最近时间
- 这样页面、CLI、Codex 才能先判断“有没有异常事件”再决定是否展开明细
---
## 9. List Contract 与 Compact 模式
来源:
- `GET /api/v1/ops/jobs`
当前实现特点:
- 默认 `list_ops_jobs(..., compact=True)`
- 返回 compact job
- `steps` 被置空
- `payload / metadata / result` 会被压缩
因此建议把列表 contract 明确成:
```json
{
"jobs": [
{
"id": 101,
"job_code": "ops-20260418064500-a1b2c3",
"action": "logs.collect",
"status": "queued",
"execution_mode": "remote-agent",
"requested_by": "web-ui/ops-center",
"payload": {
"tail_lines": 120
},
"metadata": {},
"result": {},
"result_summary": {},
"is_compact": true,
"steps": []
}
]
}
```
正式要求:
- 列表页默认只承诺 compact 视图
- 详情页再承诺 full job + steps
- 页面不应假设列表结果天然带完整 steps
---
## 10. Create Contract
来源:
- `POST /api/v1/ops/jobs`
最小请求体:
```json
{
"action": "logs.collect",
"target_type": "node",
"target_node_code": "mainland-worker-01",
"execution_mode": "remote-agent",
"requested_by": "web-ui/ops-center",
"payload": {
"tail_lines": 120
},
"metadata": {
"source": "ops-center"
},
"auto_approve": false,
"run_now": true
}
```
当前实现还支持:
- `target_selector`
- `rollout_id`
最小返回体:
```json
{
"job": {},
"executed_immediately": false
}
```
关键语义:
- `executed_immediately=true`
- 说明任务已被控制面即时执行,而不是停留在队列
- `executed_immediately=false`
- 说明任务进入了:
- `queued`
- `awaiting_approval`
- `blocked`
正式要求:
- 创建接口不能只回答“建成功了”
- 必须明确回答:
- 当前状态是什么
- 是否立即执行
- 如果没执行,是在等审批、等 agent、还是被阻断
---
## 11. Batch Create Contract
来源:
- `POST /api/v1/ops/jobs/batch`
最小请求体建议:
```json
{
"template_key": "diagnostics.collect",
"target_node_codes": [
"mainland-worker-01",
"mainland-worker-02"
],
"execution_mode": "remote-agent",
"requested_by": "web-ui/ops-center",
"payload": {},
"metadata": {},
"auto_approve": true
}
```
最小返回体建议:
```json
{
"jobs": [],
"results": []
}
```
正式要求:
- batch 只是创建方式,不是新的执行对象
- batch 最终仍要展开成多条 `ops job`
- 单条失败不能让结果完全丢失,必须保留逐项结果
---
## 12. Policy Preview Contract
来源:
- `POST /api/v1/ops/policy/preview`
- `ops_jobs.policy`
这层是创建前的正式门禁对象。
当前创建任务时,后端已先做:
- 风险等级判断
- 阻断判断
- 审批要求判断
建议最小返回字段:
```json
{
"blocked": false,
"blocking_reasons": [],
"approval_required": true,
"approval_reasons": [
"当前目标包含 control 节点"
],
"warnings": [],
"recommendations": [],
"risk_level": "high",
"target_summary": {},
"batch_plan": {},
"cluster_guardrails": {},
"operational_readiness": {}
}
```
正式要求:
- 页面、CLI、Codex 都应复用同一份 policy preview
- 不能各自再写一套“能不能执行”的判断逻辑
---
## 13. Approval / Cancel / Dispatch Contract
### 13.1 Approve
来源:
- `POST /api/v1/ops/jobs/{job_id}/approve`
最小请求体:
```json
{
"approved_by": "admin"
}
```
语义:
-`awaiting_approval` 可审批
- 审批后回到 `queued`
### 13.2 Cancel
来源:
- `POST /api/v1/ops/jobs/{job_id}/cancel`
最小请求体:
```json
{
"cancelled_by": "admin",
"reason": "本轮灰度暂停"
}
```
语义:
- `success / failed / cancelled` 不可取消
- 取消后 job 进入 `cancelled`
- steps 同步进入 `cancelled` 或保持终态
### 13.3 Dispatch
来源:
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
语义:
- `queued` 才可派发
- `control-plane`
- 立即由控制面执行
- `local-runtime`
- 当前节点本机即时执行
- `ssh`
- 由 SSH 执行器执行
- `remote-agent`
- 保持 `queued`,等待对应 agent 拉取
这点必须明确:
> dispatch 不等于一定会立刻变成 running。
对于 `remote-agent`,它可能只是:
- 写入事件
- 保持排队
- 等节点稍后 pull
---
## 14. 与 Playbook / Rollout / Agent 的边界
### 14.1 与 Playbook
- `playbook run`
- 负责展开、聚合、重跑、取消整轮编排
- `ops job`
- 负责单条执行记录
### 14.2 与 Rollout
- `rollout`
- 负责发布批次推进
- `ops job`
- 负责每个节点每步的实际任务
### 14.3 与 Node Agent
- 控制面创建 `ops job`
- agent 拉取 `ops job`
- agent 回写 `ops_job_events`
- agent 推进 job / step 状态
### 14.4 与 Observability
- `ops_observability_contract`
- 看现场事实
- `ops_job_contract`
- 看执行动作本身
这两层不能混:
- 观察面告诉你“谁在跑、谁异常”
- 任务面告诉你“这一条任务如何被创建、审批、派发、执行、取消”
---
## 15. 终局约束
后续不应再回退到下面几种模式:
### 1. 页面点按钮直接跑 shell
错误。
正确方式:
- 页面创建 `ops job`
### 2. Codex 直接 SSH 改现场
错误。
正确方式:
- Codex 创建 `ops job` / `playbook run`
### 3. 发布系统不经过任务层
错误。
正确方式:
- rollout 批量生成 `ops job`
### 4. 死信治理不留痕
错误。
正确方式:
- `flush / replay / discard` 都生成正式 `ops job`
所以可以把这份文档看成整套系统的“执行订单母本”。
后续只要这份 contract 稳住页面、CLI、Codex、Agent、Release、Playbook 就都能围绕同一个中心对象协作。

View File

@@ -0,0 +1,702 @@
# domainCheck Ops Observability Contract
## 1. 目标
这份文档用于冻结海外单脑控制面里“看现场”这一层的正式 contract。
它不负责:
- 发布决策
- Driver 推荐
- Playbook 编排
它负责回答 3 类问题:
1. 哪些节点现在真正处于执行现场
2. 这些节点最近巡检收口结果是什么
3. Node Agent 回执队列现在是否健康
也就是说,这份 contract 的职责是:
> 让页面、CLI、Codex 看到同一份现场事实,而不是各自从日志和状态字串里猜
---
## 2. 入口接口
建议把下面这些接口视为同一观察面的一组正式入口:
- `GET /api/v1/ops/overview`
- `GET /api/v1/ops/inspection-overview`
- `GET /api/v1/ops/activity-stream`
- `GET /api/v1/ops/nodes/{node_code}/handover`
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/flush`
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/replay`
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay`
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard`
配套关系:
- `overview`
- 首页收口
- `inspection-overview`
- 节点巡检收口
- `activity-stream`
- 最近异常活动
- `handover`
- 单节点接管阻断与下一步建议
- `delivery-queue`
- Node Agent 回执健康
正式要求:
- 页面不能自己再定义第 5 套“观察口径”
- CLI 不应重新从数据库拼现场
- Codex 不应通过自然语言日志推断核心状态
---
## 3. Execution Scene Contract
来源:
- `GET /api/v1/ops/overview`
- `overview.execution_scene`
最小字段:
```json
{
"summary": "当前存在 2 台有效执行节点,其中 1 台正在领任务1 台在线待命。",
"dispatch_active_nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "busy",
"participation_state": "dispatch_active",
"participation_label": "正在执行",
"participation_reason": "当前存在已领任务和执行中工作负载",
"current_load": 9,
"items_claimed": 9,
"items_running": 5,
"items_completed_recent": 14
}
],
"recent_only_nodes": [],
"standby_nodes": [
{
"node_code": "mainland-controller-01",
"region": "mainland",
"role": "control",
"status": "online",
"participation_state": "standby",
"participation_label": "在线待命",
"participation_reason": "节点在线、有效,但当前没有领任务、执行中任务或近窗吞吐",
"current_load": 0
}
],
"load_syncing_nodes": [],
"counts": {
"dispatch_active": 1,
"recent_only": 0,
"standby": 1,
"load_syncing": 0
}
}
```
分组语义必须固定:
- `dispatch_active_nodes`
- 当前正在领任务、执行任务、或仍有实时工作负载
- `recent_only_nodes`
- 当前不再运行,但近窗刚有吞吐
- `standby_nodes`
- 在线、有效,但当前不参与
- `load_syncing_nodes`
- 节点与控制面状态尚未完全同步
正式约束:
- “在线”不等于“参与检测”
- “有效执行节点”不等于“正在跑”
- 页面展示、CLI 摘要、Codex 判断必须沿用这 4 类分组
节点参与态附加字段也应固定:
- `participation_state`
- 机器判断主键,推荐值:
- `dispatch_active`
- `recent_only`
- `standby`
- `load_syncing`
- `participation_label`
- 直接给页面、CLI、Codex 展示的人类可读标签
- `participation_reason`
- 用一句稳定摘要解释为什么节点落在当前分组
正式要求:
- 页面必须直接区分:
- 在线但未参与
- 正在执行或领任务
- 近窗刚参与但当前已收口
- 新消费方不允许再从:
- `status`
- `current_load`
- `items_running`
手工推断“这个节点到底算不算正在参与”
---
## 4. Log Sync Runtime Contract
来源:
- `overview.execution_scene.log_sync`
-`overview.log_sync`
最小字段:
```json
{
"status": "partial_coverage",
"status_label": "部分覆盖",
"mode": "key",
"mode_label": "关键回传",
"enabled": true,
"source_nodes": [
"mainland-worker-01"
],
"source_node_summaries": [
{
"node_code": "mainland-worker-01",
"line_count": 42,
"key_line_count": 42,
"full_line_count": 0,
"last_at": "2026-04-18 06:40:00",
"last_line": "域名检测完成: demo.com"
}
],
"covered_participating_node_summaries": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"participation_state": "dispatch_active",
"participation_label": "正在执行",
"participation_reason": "当前存在已领任务和执行中工作负载",
"line_count": 42,
"key_line_count": 42,
"full_line_count": 0,
"last_at": "2026-04-18 06:40:00",
"last_line": "域名检测完成: demo.com"
}
],
"missing_participating_node_summaries": [
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"participation_state": "dispatch_active",
"participation_label": "正在执行",
"participation_reason": "当前存在已领任务和执行中工作负载",
"missing_reason": "no_sample"
}
],
"covered_participating_nodes": 1,
"participating_nodes_total": 2,
"missing_sample_node_codes": [
"mainland-worker-02"
],
"last_line_at": "2026-04-18 06:40:00",
"summary": "当前已覆盖 1/2 台参与节点,建议继续观察未回传样本节点。"
}
```
推荐状态:
- `disabled`
- `waiting_sample`
- `partial_coverage`
- `healthy`
- `full_capture`
这层是“现场可见性”最关键的辅助状态。
正式要求:
- 如果存在参与节点但日志回传还没开,优先推荐开关键回传
- 如果日志回传已开但没有样本,优先推荐抓 Worker 日志
- 页面和 Codex 都不能再自己定义另一套日志覆盖结论
节点级日志覆盖字段也应固定:
- `source_nodes`
- 为兼容旧页面保留的节点编码列表
- `source_node_summaries`
- 当前已有日志样本的节点级统计对象
- `covered_participating_node_summaries`
- 已参与且已拿到样本的节点对象
- `missing_participating_node_summaries`
- 已参与但尚未拿到样本的节点对象
`source_node_summaries[]` 至少应返回:
- `node_code`
- `line_count`
- `key_line_count`
- `full_line_count`
- `last_at`
- `last_line`
`missing_participating_node_summaries[]` 至少应返回:
- `node_code`
- `region`
- `role`
- `participation_state`
- `participation_label`
- `participation_reason`
- `missing_reason`
推荐 `missing_reason`
- `no_sample`
- `delayed_flush`
- `filtered_out`
兼容要求:
- 旧页面仍可继续消费:
- `source_nodes`
- `missing_sample_node_codes`
- 新页面、CLI、Codex 应优先消费:
- `source_node_summaries`
- `covered_participating_node_summaries`
- `missing_participating_node_summaries`
补充节点级现场入口:
- `GET /api/v1/ops/nodes/{node_code}/scene-log`
最小字段:
```json
{
"node_code": "mainland-worker-01",
"status": "healthy",
"status_label": "关键覆盖",
"mode": "key",
"mode_label": "关键回传",
"log_sync_enabled": true,
"summary": "节点当前正在参与检测,已保留 42 条现场日志样本。",
"node": {
"region": "mainland",
"role": "worker",
"status": "busy",
"current_load": 6,
"last_heartbeat_at": "2026-04-18 06:40:00"
},
"participation": {
"detect_participating": true,
"participation_state": "claimed",
"participation_label": "已领待跑",
"participation_bucket": "dispatch_active",
"participation_reason": "已领取任务,等待线程继续执行。"
},
"source_summary": {
"line_count": 42,
"key_line_count": 42,
"full_line_count": 0,
"last_at": "2026-04-18 06:40:00",
"last_line": "[2026-04-18 06:40:00] [mainland-worker-01] 域名检测完成: demo.com"
},
"records_total": 42,
"records_visible": 80,
"records": [
{
"created_at": "2026-04-18 06:38:00",
"node_code": "mainland-worker-01",
"message": "开始检测域名: demo.com",
"mode": "key",
"line": "[2026-04-18 06:38:00] [mainland-worker-01] 开始检测域名: demo.com"
}
]
}
```
节点级入口的正式语义:
- 这是“看某一台节点当前现场”的正式入口,不允许页面再自己从聚合样本中二次猜测
- `status` 至少覆盖:
- `healthy`
- `full_capture`
- `historical_sample`
- `waiting_sample`
- `missing_sample`
- `disabled`
- `standby`
- `unknown`
- 当节点当前在线但未参与时,必须明确返回 `standby`
- 当节点正在参与但还没有样本时,必须明确返回 `waiting_sample`
- 当节点已有历史样本但当前不在参与面中时,必须明确返回 `historical_sample`
---
## 5. Inspection Overview Contract
来源:
- `GET /api/v1/ops/inspection-overview`
- `overview.inspection`
最小字段:
```json
{
"status": "attention",
"status_label": "待处理",
"summary": "存在 1 台节点最近缺少 Worker 日志收口。",
"counts": {
"healthy_nodes": 1,
"attention_nodes": 1,
"problem_nodes": 0
},
"rows": [
{
"node_code": "mainland-worker-01",
"problem_kind": "missing_worker_logs",
"problem_label": "缺少 Worker 日志收口",
"problem_level": "warning",
"recommended_action_code": "open_worker_logs",
"ui_intent": {
"kind": "node_inspection_detail",
"node_code": "mainland-worker-01"
},
"latest_health_snapshot": {},
"latest_worker_logs": {},
"latest_diagnostics": {}
}
]
}
```
设计要求:
- 巡检不是几条离散 job 记录
- 必须按节点重新聚合
- 必须直接给出:
- `problem_kind`
- `problem_label`
- `problem_level`
- `recommended_action_code`
- `ui_intent`
这样页面、CLI、Codex 才能真正消费“节点巡检结论”,而不是再从几十条 job 记录里回推问题。
---
## 6. Inspection Row Contract
`rows[]` 中每个节点建议最少包含:
- `node_code`
- `region`
- `role`
- `effective_worker`
- `problem_kind`
- `problem_label`
- `problem_level`
- `recommended_action_code`
- `ui_intent`
- `latest_health_snapshot`
- `latest_worker_logs`
- `latest_diagnostics`
其中 3 类最近结果建议统一结构:
```json
{
"status": "success",
"status_label": "成功",
"occurred_at": "2026-04-18 06:35:00",
"job_code": "ops-20260418-001",
"summary": "最近一次 Worker 日志收集成功,共采样 120 行。"
}
```
正式边界:
- `logs.collect` 默认指 Worker 日志收口
- Node Agent 辅助排障日志不应混成同一种巡检结果
---
## 7. Delivery Queue Summary Contract
来源:
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
最小字段:
```json
{
"node": {
"node_code": "mainland-worker-01",
"title": "mainland-worker-01"
},
"summary": {
"state": "dead_letter",
"label": "死信",
"reason": "当前有 2 条记录进入死信。",
"pending_count": 3,
"dead_letter_count": 2,
"oldest_pending_at": "2026-04-18 06:20:00",
"oldest_dead_letter_at": "2026-04-18 06:18:00"
},
"record_visibility": "head_only",
"capabilities": {
"can_flush": true,
"can_replay": true,
"can_discard": true
}
}
```
推荐状态:
- `healthy`
- `retrying`
- `dead_letter`
- `stalled`
正式约束:
- 控制面要明确告诉操作者当前是 `head_only` 还是 `full_projection`
- 不允许页面假装能列出远端完整目录
- 不允许因为能看见死信就绕过 `ops job` 直接改文件
---
## 8. Delivery Queue Record Contract
来源:
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
当前阶段建议最小字段:
```json
{
"records": [
{
"record_id": 101,
"state": "dead_letter",
"state_label": "死信",
"request_id": "req-20260418-001",
"kind": "job_complete",
"created_at": "2026-04-18 06:18:00",
"updated_at": "2026-04-18 06:19:00",
"summary": "某次任务完成回执多次补发失败。",
"visibility": "head_only"
}
],
"summary": {
"returned_records": 1,
"record_visibility": "head_only"
}
}
```
当前阶段要求:
- 单条 `replay / discard` 只针对控制面可见记录
- 当前 `head_only` 不等于未来不能扩展
- 以后若进入 `full_projection`,扩的是可见范围,不是重写操作模型
---
## 9. Delivery Queue Action Contract
来源:
- `flush`
- `replay`
- `record replay`
- `record discard`
正式要求:
- 所有动作都必须落成正式 `ops job`
- 返回体要能告诉调用方:
- 是否已受理
- 生成了什么 job
- 建议跳转到哪里观察
最小返回体建议:
```json
{
"accepted": true,
"job_code": "ops-20260418-queue-001",
"ui_intent": {
"kind": "job_events",
"job_code": "ops-20260418-queue-001"
}
}
```
这里的边界必须比“临时脚本”更严格:
- `discard` 必须带原因
- `replay` 应支持限流 / limit
- `flush` 应支持显式 `requested_by`
---
## 10. Activity Stream Contract
来源:
- `GET /api/v1/ops/activity-stream`
最小字段建议:
```json
{
"items": [
{
"kind": "ops_job",
"activity_key": "ops-job:101",
"title": "logs.collect",
"subtitle": "ops-20260418064500-a1b2c3",
"summary": "目标节点 mainland-worker-01 / 发起 web-ui / 方式 Node Agent",
"summary_text": "目标节点 mainland-worker-01 / 发起 web-ui / 方式 Node Agent",
"status": "running",
"status_label": "执行中",
"execution_mode": "remote-agent",
"execution_mode_label": "Node Agent",
"target_node_codes": [
"mainland-worker-01"
],
"occurred_at": "2026-04-18 06:45:08",
"ui_intent_kind": "job_events",
"focus_ref": {
"kind": "ops_job",
"job_id": 101,
"job_code": "ops-20260418064500-a1b2c3",
"action": "logs.collect",
"target_node_code": "mainland-worker-01"
}
}
],
"summary": {
"status": "running",
"status_label": "执行中",
"summary_text": "最近活动里存在正在推进的现场、任务、编排或回传。",
"filtered_total": 6,
"unfiltered_total": 18,
"latest_at": "2026-04-18 06:45:08"
}
}
```
正式要求:
- `activity-stream` 不是页面内部 table 数据
- 它必须是:
- 页面
- CLI
- Codex 驾驶员
- 自动驾驶编排器
共同消费的“最近活动摘要”
字段语义需要固定:
- `summary_text`
- 是直接消费的活动短摘要
- `status_label`
- 是稳定的人类可读状态标签
- `target_node_codes`
- 是活动关联的节点口径,不允许调用方再从自然语言里猜
- `ui_intent_kind`
- 是当前活动默认落点类型
- `focus_ref`
- 是跨页面 / CLI / Codex 的统一定位对象
正式边界:
- 活动流应保留老字段以兼容现有页面
- 新消费方优先用:
- `summary_text`
- `status_label`
- `focus_ref`
- 不允许新客户端再从 `title + subtitle + summary` 手工解析定位参数
## 11. 与 Driver / Playbook / Release 的边界关系
这份 contract 只负责观察面和现场治理,不负责替代其他 contract。
边界应明确为:
- `ops_driver_contract`
- 决定“现在优先做什么”
- `ops_playbook_contract`
- 决定“标准多步动作怎么编排”
- `release_hub_contract`
- 决定“版本怎么发、怎么停、怎么回滚”
- `ops_observability_contract`
- 决定“现场事实是什么、当前缺口在哪里”
这样后续系统再扩展时,不会又回到“所有东西都写在 overview 一坨大 json 里”的状态。
---
## 12. 终局验收标准
如果这份 contract 真正落地,海外单脑控制面应该做到:
### 1. 不登录大陆机器也能回答现场问题
例如:
- 谁在跑
- 谁在线但没跑
- 谁近窗刚跑过
- 谁日志没回来
- 谁队列死信了
### 2. 不复制大日志也能先做判断
先看:
- execution scene
- inspection overview
- activity stream
- delivery queue
再决定是否需要全量日志或 diagnostics bundle。
### 3. 页面、CLI、Codex 看到的是同一现场
不能出现:
- 页面说节点正在执行
- CLI 说节点待命
- Codex 说节点离线
### 4. 所有高风险动作都有正式观察落点
也就是:
- replay 后能看 job
- discard 后能看审计
- inspection 后能看节点收口
- rollout 后能看 launchpad / gate / rollout jobs
这样这套系统才能真正从“调试台”升级成“驾驶舱”。

View File

@@ -0,0 +1,563 @@
# domainCheck Ops Playbook Contract
## 1. 目标
这份文档用于冻结 `playbook catalog / preview / run / events` 的正式 contract。
这层的定位不是“页面里的一个弹窗”,而是:
- 控制面的正式编排对象
- 海外 Codex 驾驶员可消费的标准作业对象
- Driver / Runbook / Release / Inspection 之间的统一中间层
核心原则:
- 一次标准编排 = 一次 `playbook run`
- 页面、CLI、Codex 必须共用同一份 `playbook` / `playbook run` 结构
- `playbook run` 只负责创建与聚合正式 `ops job`
- 不允许前端再把“标准巡检 / 接管验收 / 现场日志”退回成零散按钮逻辑
建议与 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 配套阅读。
---
## 2. 当前接口面
当前已落地的主入口:
- `GET /api/v1/ops/playbooks`
- `POST /api/v1/ops/playbooks/preview`
- `POST /api/v1/ops/playbooks/execute`
- `GET /api/v1/ops/playbook-runs`
- `GET /api/v1/ops/playbook-runs/{run_code}`
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
- `POST /api/v1/ops/playbook-runs/{run_code}/rerun`
- `POST /api/v1/ops/playbook-runs/{run_code}/cancel`
并且这层已经被以下入口复用:
- `driver action -> open_playbook_dialog`
- `runbook sequence -> secondary_resolution`
- `inspection`
- `activity-stream`
所以后续不应再新造第四套“标准作业对象”。
---
## 3. Playbook Catalog Object
来源:
- `GET /api/v1/ops/playbooks`
- `preview / execute` 返回体中的 `playbook`
最小字段:
```json
{
"key": "inspection.standard",
"group_key": "diagnostics",
"group_title": "标准巡检",
"title": "标准巡检",
"description": "按 健康快照 -> Worker 日志 -> 诊断包 的顺序,对目标节点做一次标准联调巡检。",
"target_scope": "node-set",
"default_execution_mode": "remote-agent",
"default_execution_mode_label": "Node Agent",
"execution_modes": [
"remote-agent",
"ssh"
],
"execution_mode_options": [
{
"value": "remote-agent",
"label": "Node Agent",
"is_default": true
},
{
"value": "ssh",
"label": "SSH",
"is_default": false
}
],
"default_auto_approve": true,
"stop_on_failure": true,
"steps": [
{
"step_key": "health",
"title": "采集健康快照",
"template_key": "health.snapshot",
"payload": {}
}
]
}
```
正式约束:
- `key`
- 是 Playbook 的稳定主键
- `group_key`
- 用于页面、CLI、Codex 做标准分组,不应用 `title`
- `execution_modes`
- 是能力白名单,不是建议文案
- `stop_on_failure`
- 是编排级行为,不是某一步骤的临时说明
---
## 4. Playbook Preview Contract
来源:
- `POST /api/v1/ops/playbooks/preview`
最小请求体:
```json
{
"playbook_key": "inspection.standard",
"node_codes": [
"mainland-worker-01"
],
"execution_mode": "remote-agent",
"auto_approve": true,
"requested_by": "web-ui"
}
```
最小返回体:
```json
{
"playbook": {},
"requested_by": "web-ui",
"execution_mode": "remote-agent",
"execution_mode_label": "Node Agent",
"auto_approve": true,
"target_nodes_total": 1,
"target_node_codes": [
"mainland-worker-01"
],
"step_count": 3,
"expected_jobs_total": 3,
"template_keys": [
"health.snapshot",
"logs.collect",
"diagnostics.collect"
],
"steps": [
{
"order": 1,
"step_key": "health",
"title": "采集健康快照",
"template_key": "health.snapshot",
"action": "health.snapshot",
"payload": {},
"execution_mode": "remote-agent",
"execution_mode_label": "Node Agent",
"auto_approve": true,
"target_nodes_total": 1,
"expected_jobs": 1
}
]
}
```
预览阶段必须回答:
- 最终执行方式是什么
- 总共会展开多少步
- 每步会展开成什么模板
- 总共会生成多少个 `ops job`
这样页面、CLI、Codex 才能在执行前看见“这次会做什么”。
---
## 5. Playbook Run Object
来源:
- `GET /api/v1/ops/playbook-runs`
- `GET /api/v1/ops/playbook-runs/{run_code}`
- `POST /api/v1/ops/playbooks/execute`
最小字段:
```json
{
"run_code": "playbook-20260418-001",
"playbook_key": "inspection.standard",
"playbook_title": "标准巡检",
"group_key": "diagnostics",
"group_title": "标准巡检",
"requested_by": "web-ui",
"execution_mode": "remote-agent",
"execution_mode_label": "Node Agent",
"stop_on_failure": true,
"step_count": 3,
"expected_jobs_total": 3,
"target_nodes_total": 1,
"target_node_codes": [
"mainland-worker-01"
],
"created_at": "2026-04-18 06:30:00",
"updated_at": "2026-04-18 06:31:10",
"status": "running",
"status_label": "收口中",
"status_counts": {
"queued": 1,
"running": 1,
"success": 1
},
"jobs_total": 3,
"success_jobs_total": 1,
"terminal_jobs_total": 1,
"completion_percent": 33.3,
"steps_total": 3,
"steps_success": 1,
"steps_running": 1,
"steps_problem": 0,
"steps_terminal": 1,
"focus_level": "warning",
"focus_step_key": "worker_logs",
"focus_step_title": "收集 Worker 日志",
"focus_summary": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
"summary": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
"summary_text": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
"focus_ref": {
"kind": "playbook_run",
"run_code": "playbook-20260418-001",
"playbook_key": "inspection.standard",
"group_key": "diagnostics",
"focus_step_key": "worker_logs",
"focus_step_title": "收集 Worker 日志",
"event_key": "",
"node_code": ""
},
"problem_steps": [],
"active_steps": [],
"latest_job": {},
"steps": []
}
```
正式定义:
> `playbook run` 是一轮标准编排在控制面的聚合对象,不是某一条单独任务。
也就是说:
- `ops job` 是执行颗粒度
- `playbook run` 是观察、重跑、取消、钻取的编排颗粒度
---
## 6. Playbook Step Object
`playbook run.steps[]` 的最小结构:
```json
{
"step_key": "worker_logs",
"title": "收集 Worker 日志",
"template_key": "logs.collect",
"order": 2,
"jobs_total": 1,
"nodes_total": 1,
"target_node_codes": [
"mainland-worker-01"
],
"status_counts": {
"running": 1
},
"terminal_jobs_total": 0,
"success_jobs_total": 0,
"status": "running",
"status_label": "收口中",
"completion_percent": 0.0,
"summary": "收集 Worker 日志 当前处于执行中,可继续观察该步骤事件。",
"summary_text": "收集 Worker 日志 当前处于执行中,可继续观察该步骤事件。",
"focus_ref": {
"kind": "playbook_run",
"run_code": "playbook-20260418-001",
"playbook_key": "inspection.standard",
"group_key": "diagnostics",
"focus_step_key": "worker_logs",
"focus_step_title": "收集 Worker 日志",
"event_key": "",
"node_code": ""
},
"latest_job": {}
}
```
推荐 step 状态:
- `queued`
- `running`
- `success`
- `attention`
规则:
- 只要该步骤出现失败、阻断、取消等终态问题,应收口为 `attention`
- 页面与 Codex 都应优先盯 `problem_steps / active_steps`,而不是自己遍历所有子任务推断
---
## 7. Playbook Event Stream
来源:
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
最小返回体:
```json
{
"run_code": "playbook-20260418-001",
"playbook_run": {},
"events": [
{
"job_id": 123,
"job_code": "ops-20260418-123",
"run_code": "playbook-20260418-001",
"job_status": "running",
"job_status_label": "收口中",
"action": "logs.collect",
"target_node_code": "mainland-worker-01",
"step_key": "worker_logs",
"step_title": "收集 Worker 日志",
"event_type": "job_running",
"level": "info",
"level_label": "信息",
"message": "开始收集 Worker 日志",
"summary": "开始收集 Worker 日志",
"summary_text": "开始收集 Worker 日志",
"payload": {},
"event_key": "job-event:1001",
"occurred_at": "2026-04-18 06:31:00",
"focus_ref": {
"kind": "playbook_run",
"run_code": "playbook-20260418-001",
"playbook_key": "inspection.standard",
"group_key": "diagnostics",
"focus_step_key": "worker_logs",
"focus_step_title": "收集 Worker 日志",
"event_key": "job-event:1001",
"node_code": "mainland-worker-01"
},
"created_at": "2026-04-18 06:31:00"
}
],
"summary": {
"total": 1,
"returned_total": 1,
"available_step_counts": {
"worker_logs": 1
},
"available_node_counts": {
"mainland-worker-01": 1
},
"level_counts": {
"info": 1
},
"event_type_counts": {
"job_running": 1
},
"job_status_counts": {
"running": 1
},
"latest_at": "2026-04-18 06:31:00",
"filters": {
"step_key": "",
"node_code": "",
"limit": 80
}
}
}
```
这层必须支持:
-`step_key` 过滤
-`node_code` 过滤
- 快速看到最近活动时间
否则 `playbook run` 只能看总览,不能真正钻取。
---
## 8. 正式 Playbook Key 语义
当前最值得冻结成正式对象的 key
### 8.1 `onboarding.bootstrap`
定位:
- 生成节点接入工单
执行方式:
-`control-plane`
正式语义:
- 不直接接管节点
- 只创建标准接入方案对象
### 8.2 `onboarding.acceptance`
定位:
- 新节点接管后标准验收
当前标准步骤:
1. `health.snapshot`
2. `service.status(node-agent)`
3. `logs.collect(node-agent)`
4. `service.status(worker)`
5. `logs.collect(worker)`
### 8.3 `inspection.standard`
定位:
- 标准巡检
当前标准步骤:
1. `health.snapshot`
2. `logs.collect(worker)`
3. `diagnostics.collect`
正式要求:
- 这是 release / rollout 之前最核心的健康观察面
### 8.4 `scene.logs.key`
定位:
- 关键现场日志样本
正式要求:
- 默认比 `scene.logs.full` 更轻
- 适合先看现场,再决定是否升级取证
### 8.5 `scene.logs.full`
定位:
- 全量现场取证
正式要求:
- 应明确比 `scene.logs.key` 更重
- 默认应同时补诊断包
### 8.6 `scene.diagnostics`
定位:
- 轻量诊断包
正式要求:
- 适合作为参与节点的常规取证动作
---
## 9. 执行模式 Contract
推荐模式:
- `remote-agent`
- `ssh`
- `control-plane`
正式约束:
- `remote-agent`
- 正式日常运维默认模式
- `ssh`
- 仅用于首发接管与应急救援
- `control-plane`
- 只适合控制面本地对象生成或极少量控制面动作
页面、CLI、Codex 不能自己发明第四种临时模式。
---
## 10. Playbook Run 状态机
推荐状态:
- `queued`
- `running`
- `success`
- `attention`
含义:
- `queued`
- 子任务刚创建,尚未出现有效执行推进
- `running`
- 已有子任务进入 `dispatching / running / awaiting_approval`
- `success`
- 全部子任务收口成功
- `attention`
- 任一步骤出现失败、阻断、取消或部分成功,需要人工关注
正式规则:
- 页面上的“标准巡检状态”
- `activity-stream` 中的 playbook run 摘要
- Driver / Codex 的推荐动作
都必须共用这一套状态语义。
---
## 11. 与 Driver / Runbook / Release 的关系
正式收口规则:
- Driver 不直接“模拟巡检”
- 应优先跳转或创建 `playbook run`
- Runbook Sequence 不直接展开 shell
- 应优先收口到 `driver action``playbook`
- Release / Rollout 不自己维护另一套巡检摘要
- 应优先消费 `inspection.standard` 的最近结果
也就是说:
- `scene` 是现场观察对象
- `inspection` 是标准健康对象
- `release / rollout` 是发布对象
三者不能再各自维护不同的“日志 / 诊断 / 巡检”定义。
---
## 12. 正式要求
后续继续演进时,必须保持:
1. `playbook catalog`
- 是稳定 contract而不是页面枚举
2. `playbook preview`
- 是执行前的唯一结构化预览面
3. `playbook run`
- 是页面、CLI、Codex 共同观察的一轮编排对象
4. `playbook events`
- 是整轮编排的钻取视角
5. 接管验收、标准巡检、现场观察
- 优先都收进 `playbook`
这样这套编排才能成为真正的正式平台能力,而不是一组“看起来像流程”的按钮。

View File

@@ -0,0 +1,372 @@
# domainCheck Ops Stack Diagnosis Contract
## 1. 目标
这份文档用于冻结海外单脑控制面的“总检入口” contract。
它的职责不是替代:
- `overview`
- `link-snapshot`
- `inspection-overview`
- `release launchpad`
- `activity-stream`
而是把这些正式 contract 再收口成一份:
> 第一现场诊断结果
让下面这些消费者都先看同一份判断:
- 海外 Ops Center 页面
- 海外 Codex 驾驶员
- CLI 总检脚本
- 后台按钮自动诊断
正式原则:
- 总检不能只回一段说明文字
- 总检必须给出结构化问题清单
- 总检必须给出默认下一步动作
- 总检必须给出可以直接执行的推荐命令
---
## 2. 入口接口
主入口:
- `GET /api/v1/ops/go-live-summary`
- `GET /api/v1/ops/stack-diagnosis`
配套来源:
- `GET /api/v1/ops/contracts`
- `GET /api/v1/ops/link-snapshot`
- `GET /api/v1/ops/overview`
- `GET /api/v1/ops/nodes`
- `GET /api/v1/ops/releases/launchpad`
- `GET /api/v1/ops/playbook-runs`
- `GET /api/v1/ops/activity-stream`
推荐 query
- `base_url`
用途:
- `go-live-summary`
- 给 Ops Center 顶部收口卡、CLI 总检、Codex 驾驶员直接消费
- 返回更偏“上线判断”的稳定摘要
- 用于生成 `next_step.command`
- 用于生成 `quick_commands`
- 让 CLI / Codex / 页面复制出来的命令口径一致
---
## 3. 顶层结构
建议最小结构:
```json
{
"generated_at": "2026-04-18 09:00:00",
"diagnosis": {},
"api": {},
"surface_matrix": {},
"contracts": {},
"link_snapshot": {},
"overview": {},
"managed_nodes": {},
"release_hub": {},
"playbook_runs": {},
"activity_stream": {}
}
```
正式要求:
- 顶层每一段都必须是稳定 key
- 页面、CLI、Codex 不允许自己再拼第 2 套总检结构
- `go-live-summary`
-`stack-diagnosis` 的稳定摘要层
- 必须至少给出:
- `go_live_status`
- `blocking_reasons`
- `warnings`
- `launchpad_recommended_target_node_code`
- `launchpad_recommended_recovery_label`
- `launchpad_recommended_recovery_summary`
- `launchpad_onboarding_bootstrap_pending_nodes`
- `launchpad_onboarding_acceptance_ready_nodes`
- `next_step_action_code`
- `operator_title`
- `recommended_commands`
- `playbook_runs.problem_runs[*]`
- 需要保留 `focus_ref`
- 这样 CLI / Codex / 页面才能直接定位到问题编排的正式焦点
- `activity_stream.top_items[*]`
- 需要保留:
- `occurred_at`
- `focus_ref`
- `source_focus_ref`
- `ui_intent_kind`
- 这样总检不会把“观察摘要”再次压扁成一段不可定位的纯文本
- `go-live-summary` + `stack-diagnosis` + `driver-feed.automation_coverage`
- 必须能共同驱动 Ops Center 首屏
- 不允许页面自己从局部接口重新推导一份“收口状态”
---
## 4. Diagnosis Contract
来源:
- `stack_diagnosis.diagnosis`
最小字段:
```json
{
"contract_key": "ops_stack_diagnosis_contract",
"contract_version": "v1",
"registry_version": "2026-04-18",
"base_url": "http://127.0.0.1:8100",
"surface_status": "partial",
"automation_status": "blocked",
"stack_status": "blocked",
"issue_total": 4,
"blocking_issue_total": 1,
"warning_issue_total": 3,
"missing_surfaces": [
"contracts"
],
"launchpad_recommended_target_node_code": "mainland-worker-01",
"launchpad_recommended_recovery_label": "待执行接入",
"launchpad_recommended_recovery_summary": "mainland-worker-01 还缺接入收口,先执行 bootstrap。",
"launchpad_onboarding_bootstrap_pending_nodes": 1,
"launchpad_onboarding_acceptance_ready_nodes": 0,
"next_step": {
"action_code": "bootstrap_run",
"source": "issue:managed_nodes_agent_pending",
"reason": "mainland-worker-01 还缺接入收口,先执行 bootstrap。",
"command": "bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 bootstrap_run",
"focus_ref": {
"kind": "managed_node",
"node_code": "mainland-worker-01"
}
},
"recommended_actions": [],
"issues": [],
"operator_hints": [],
"quick_commands": []
}
```
状态语义必须固定:
- `surface_status`
- `healthy`
- `partial`
- `broken`
- `automation_status`
- `ready`
- `attention`
- `blocked`
- `stack_status`
- 总结论,面向人和程序
- 推荐值为 `ready / attention / blocked`
正式要求:
- `next_step` 是默认下一步,而不是“可能动作之一”
- 当首个缺口节点已经明确收敛为 `bootstrap_run / run_acceptance` 时,`next_step.action_code` 不应再退回成泛化的 `fix_managed_nodes`
- 当存在 `runtime_build_schema_stale` 时,`next_step.action_code` 必须优先收敛为 `api-restart`
- `issues` 必须按可处理性组织,而不是只堆原始异常文本
- `quick_commands` 必须是可以直接复制执行的命令
- `next_step.focus_ref`
- 是首屏“定位下一步”的正式落点
- 页面、CLI、Codex 不允许再根据 `action_code` 反推要跳去哪个区域
- `diagnosis.launchpad_recommended_target_node_code`
- 代表 stack-diagnosis 已经把 Release Hub 收敛出的首个接管缺口直接下沉到总检摘要
- 页面、CLI、doctor-export、Codex 不应再分别回源二次推导
- `diagnosis.launchpad_onboarding_bootstrap_pending_nodes / launchpad_onboarding_acceptance_ready_nodes`
- 代表当前接管缺口的阶段性计数
- 用来帮助总检直接区分“待 bootstrap”与“待 acceptance”
- `go-live-summary.launchpad_recommended_target_node_code`
- 是 launchpad 已经收敛出的首个接管缺口节点
- 页面、CLI、Codex 应优先直接显示它,而不是重新扫描 gap rows
- `go-live-summary.launchpad_onboarding_bootstrap_pending_nodes / launchpad_onboarding_acceptance_ready_nodes`
- 用于区分当前是“待接入”还是“待验收”
- 这两个计数应作为上线收口阶段的正式 warning 来源,而不是页面本地再统计
---
## 5. Issue Contract
`diagnosis.issues[]` 最小字段:
```json
{
"code": "managed_nodes_agent_pending",
"severity": "blocked",
"layer": "managed_nodes",
"summary": "托管节点虽然已经纳入 Ops Center但 0 台 remote-agent 就绪,当前第一动作已收敛为首台节点接入收口。",
"detail": "managed_enabled=3remote_access_ready=0首个缺口节点 mainland-worker-01待执行接入。",
"action_code": "bootstrap_run",
"focus_ref": {
"kind": "managed_node",
"node_code": "mainland-worker-01"
},
"commands": [
"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 bootstrap_run",
"bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-worker-01"
]
}
```
严重级别建议固定为:
- `blocked`
- `warning`
- `info`
正式要求:
- `summary` 用于列表展示
- `detail` 用于展开解释
- `commands` 用于终端直接执行
- 不允许只有中文文案,没有结构化定位信息
对于运行时版本漂移,还应允许出现这类 issue
```json
{
"code": "runtime_build_schema_stale",
"severity": "warning",
"layer": "runtime_build_info.schema",
"summary": "仓库已经具备新的节点接管能力,但运行中的 build-info 仍未声明对应路由键,当前更像 API 还没重启到最新代码。",
"detail": "supports_install_command_block=trueroute_surface_declares_bootstrap_plan=false",
"action_code": "api-restart",
"focus_ref": {
"kind": "runtime_build_info",
"section": "schema",
"expected_route_key": "ops_node_handover_bootstrap_plan"
},
"commands": [
"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100",
"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary",
"bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-worker-01"
]
}
```
---
## 6. Surface Matrix Contract
来源:
- `stack_diagnosis.surface_matrix`
最小字段:
```json
{
"status_counts": {
"total": 8,
"available": 7,
"missing": 1
},
"items": [
{
"name": "contracts",
"available": false
}
]
}
```
---
## 7. Runtime Build Info Extension
`stack_diagnosis.runtime_build_info` 以及 `/api/v1/runtime/build-info` 建议至少包含:
```json
{
"repository_capabilities": {
"supports_install_command_block": true,
"supports_multi_layout_bootstrap": true
},
"route_surface_declares_bootstrap_plan": false,
"runtime_schema_stale": true
}
```
语义要求:
- `repository_capabilities`
- 表示当前仓库代码本身具备的能力
- `route_surface_declares_bootstrap_plan`
- 表示运行中的 API build-info 是否已经声明节点级 bootstrap-plan 路由键
- `runtime_schema_stale`
- 表示“仓库能力已到位,但运行中的 API 结构还停留在旧版”
- 这是典型的“服务没重启到最新代码”信号,不应误判成“功能没开发完”
作用:
- 让页面和 CLI 直观看出“缺的是哪一层”
- 避免只看到最终 blocked却不知道缺的是 contract、launchpad 还是 activity-stream
---
## 7. 与其他 Contract 的关系
`ops_stack_diagnosis_contract` 的定位是:
- 不替代 `ops_driver_contract`
- 不替代 `ops_observability_contract`
- 不替代 `release_hub_contract`
它负责:
- 把它们统一收口成第一现场诊断
对于 Ops Center 页面还必须补充一条:
- 首屏负责“结论 + 默认下一步 + 复制命令 + 动作回执”
- 下方详情区负责“issue / surface / launchpad / activity / runbook”的展开与下钻
- 不允许详情区覆盖首屏的默认下一步结论
所以正确消费顺序应该是:
1. 先看 `stack-diagnosis`
2. 再根据 `issues / next_step / focus_ref` 下钻到:
- `overview`
- `link-snapshot`
- `inspection-overview`
- `release launchpad`
- `activity-stream`
---
## 8. CLI / Codex / 页面统一约束
CLI
- `check_ops_center_stack.sh` 应优先调用 `GET /api/v1/ops/stack-diagnosis`
Codex
- 默认先读取 `stack-diagnosis`
- 再决定是继续 `driver-resolve``doctor` 还是细分检查
页面:
- 顶部总检卡片必须优先消费 `diagnosis`
- 不能重新从多个 endpoint 拼一份影子状态
只有这样,海外单脑控制面才能真正进入:
> 单入口观察,分层下钻,统一执行

View File

@@ -0,0 +1,609 @@
# domainCheck Release Hub Contract
## 1. 目标
这份文档用于冻结 Release / Rollout / Launchpad / Gate 的正式 contract。
核心原则:
- 先有 Release再有 Rollout
- Release 是不可变版本对象
- Rollout 是分批推进对象
- Gate 是是否可发的统一门禁对象
- Launchpad 是给页面与 Codex 消费的发布驾驶舱对象
建议与 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 配套阅读,因为 Rollout 的真正执行颗粒度最终仍然是 `ops job`
---
## 2. Release Object
来源:
- `POST /api/v1/ops/releases`
- `GET /api/v1/ops/releases`
- `GET /api/v1/ops/releases/latest`
- `GET /api/v1/ops/releases/{id}`
- `POST /api/v1/ops/releases/{id}/activate`
最小字段:
```json
{
"id": 1,
"release_version": "2026.04.18+bd6bcb2",
"channel": "stable",
"commit_sha": "bd6bcb2",
"status": "ready",
"status_label": "已就绪",
"artifact_url": "https://release.example.com/domaincheck-2026.04.18.tgz",
"checksum": "sha256:...",
"notes": "",
"metadata": {},
"created_by": "www",
"created_at": "2026-04-18 05:30:00",
"activated_at": "2026-04-18 05:35:00",
"summary": "2026.04.18+bd6bcb2 已就绪,制品与校验信息齐备,可进入 Rollout。",
"summary_text": "2026.04.18+bd6bcb2 已就绪,制品与校验信息齐备,可进入 Rollout。",
"focus_ref": {
"kind": "release_hub",
"release_id": 1,
"release_version": "2026.04.18+bd6bcb2",
"channel": "stable",
"rollout_id": 0,
"rollout_code": "",
"section": "release_detail"
}
}
```
约束:
- 同一 `release_version` 只对应一份发布物
- Release 创建后不应重新指向另一份 artifact
- `artifact_url + checksum` 是发布可验证性的最小基线
推荐状态:
- `draft`
- `ready`
- `active`
- `archived`
---
## 3. Rollout Object
来源:
- `POST /api/v1/ops/releases/{id}/rollouts`
- `GET /api/v1/ops/rollouts/{id}`
- `GET /api/v1/ops/rollouts/{id}/jobs`
- `POST /api/v1/ops/rollouts/{id}/advance`
最小字段:
```json
{
"id": 11,
"release_id": 1,
"rollout_code": "rollout-20260418-001",
"target_selector": {
"region": "mainland",
"role": "worker"
},
"target_nodes": [
"mainland-worker-01",
"mainland-controller-01"
],
"policy": {
"batch_size": 1,
"require_approval_between_batches": true,
"rollback_on_failure": true
},
"status": "running",
"status_label": "执行中",
"batch_cursor": 1,
"batches_total": 2,
"jobs_total": 2,
"jobs_created": 1,
"result_summary": {},
"target_node_codes": [
"mainland-worker-01",
"mainland-controller-01"
],
"summary": "当前已覆盖 1/2 台节点,仍有 1 条任务执行中。",
"summary_text": "当前已覆盖 1/2 台节点,仍有 1 条任务执行中。",
"focus_ref": {
"kind": "release_rollout",
"rollout_id": 11,
"rollout_code": "rollout-20260418-001",
"release_id": 1,
"section": "rollout_jobs"
}
}
```
正式定义:
> 某个 Release 在某批目标节点上的一次分批推进计划
推荐状态:
- `planned`
- `running`
- `awaiting_approval`
- `ready_for_next_batch`
- `halted`
- `completed`
- `completed_with_issues`
---
## 4. Default Rollout Gate
来源:
- `GET /api/v1/ops/overview`
- `overview.release_hub.default_rollout_gate`
这层是 Release Hub 最关键的后端 gate contract。
最小字段:
```json
{
"status": "attention",
"status_label": "待处理",
"status_type": "warning",
"summary": "当前已有 Release但默认目标节点中仍有未接管或未通过巡检的节点。",
"summary_text": "当前已有 Release但默认目标节点中仍有未接管或未通过巡检的节点。",
"release": {
"id": 1,
"release_version": "2026.04.18+bd6bcb2",
"status": "ready",
"status_label": "已就绪"
},
"execution_mode": "remote-agent",
"execution_mode_label": "Node Agent",
"target_node_codes": [
"mainland-worker-01"
],
"default_target_node_codes": [
"mainland-worker-01"
],
"blocking_reasons": [],
"warning_reasons": [
"存在未通过标准巡检的节点"
],
"recommendations": [
"先执行标准巡检,再进入 Rollout"
],
"operational_readiness": {
"summary": "remote-agent 就绪 1/2标准巡检通过 1/2",
"rows": []
},
"focus_ref": {
"kind": "release_hub",
"release_id": 1,
"release_version": "2026.04.18+bd6bcb2",
"channel": "stable",
"rollout_id": 0,
"rollout_code": "",
"section": "default_rollout_gate"
}
}
```
推荐状态:
- `missing_release`
- `release_not_ready`
- `artifact_missing`
- `no_targets`
- `blocked`
- `attention`
- `ready`
要求:
- 首页驾驶建议
- Release 页面门禁
- Rollout 预检
- Codex 自动驾驶
必须共用这同一份 Gate。
---
## 5. Default Rollout Execution
来源:
- `overview.release_hub.default_rollout_execution`
- `overview.release_hub.default_rollout_gate_options`
作用:
- 告诉页面 / Codex 这轮默认应使用什么执行方式
- 给出其他可选 gate / mode
最小字段:
```json
{
"recommended_mode": "remote-agent",
"recommended_mode_label": "Node Agent",
"recommended_gate": {},
"gates": {
"remote-agent": {},
"ssh": {}
}
}
```
推荐执行模式:
- `remote-agent`
- `ssh`
- `control-plane`
说明:
- 正式发布优先 `remote-agent`
- `ssh` 只作为首发接管与紧急救援兜底
- `control-plane` 不应用来承载大规模 Worker 发布
---
## 6. Release Launchpad
来源:
- `overview.release_hub.launchpad`
作用:
- 给页面与 Codex 提供更偏“驾驶舱”的发布聚合视角
最小字段:
```json
{
"latest_package": {},
"latest_release": {},
"worker_rollout_preview": {},
"control_rollout_preview": {},
"launchpad_status": {
"status": "attention",
"status_label": "待处理",
"summary": "建议先完成 Worker 点状灰度,再决定是否推进控制面。",
"summary_text": "建议先完成 Worker 点状灰度,再决定是否推进控制面。",
"recommended_action_code": "open_release_deploy_worker",
"recommended_target_node_code": "mainland-worker-01",
"recommended_recovery_label": "签发接入工单",
"recommended_recovery_summary": "当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。",
"onboarding_bootstrap_pending_nodes": 1,
"onboarding_acceptance_ready_nodes": 0,
"recommended_mode": "remote-agent",
"recommended_mode_label": "Node Agent",
"focus_ref": {
"kind": "release_hub",
"release_id": 1,
"release_version": "2026.04.18+bd6bcb2",
"channel": "stable",
"rollout_id": 0,
"rollout_code": "",
"section": "release_launchpad",
"mode": "worker"
}
}
}
```
`launchpad_status` 至少应回答:
- 当前先补什么
- 推荐动作码是什么
- 当前首个缺口节点是谁
- 当前恢复动作的人类标签是什么
- 当前缺口更偏“待接入”还是“待验收”
- 推荐执行方式是什么
- 先看 Worker 预案还是先看 Control 预案
正式约束:
- `recommended_target_node_code`
- 一旦 launchpad 已经收敛出首个缺口节点,就应直接返回
- 页面、CLI、Codex 不应再分别从 gap rows 中自行挑第一台
- `recommended_recovery_label`
- 是给人看的动作标题,例如:
- `签发接入工单`
- `补执行面接管`
- `执行接管验收`
- `recommended_recovery_summary`
- 是解释“为什么当前默认不是发版,而是先补接管”的正式摘要
- `onboarding_bootstrap_pending_nodes / onboarding_acceptance_ready_nodes`
- 是 launchpad 汇总层面的正式计数
- 允许被 `go-live-summary`、CLI 摘要、Codex brief 直接复用
---
## 7. 发布动作 contract
当前发布相关动作不应再退回“通用发任务”语义,而应正式区分:
- `deploy.release.control`
- `deploy.release.worker`
- `deploy.release.custom`
这三个动作必须共用:
- 同一份 Gate 判断
- 同一份 Release 版本对象
- 同一份 Rollout 编排结果
---
## 8. Artifact 不可变要求
上线前建议把 Release 供应链固定为:
1. 构建产物
2. 生成 `checksum`
3. 生成 `manifest`
4. 上传 artifact
5. 创建 Release
6. 激活 channel
7. 创建 Rollout
Node Agent 侧部署链路要求:
1. 下载 artifact
2. 校验 checksum
3. 解压到 `releases/<version>`
4. 切换 `current`
5. 重启服务
6. 健康检查
7. 失败回滚
---
## 9. Rollout 与 Ops Job 的关系
约束:
- Rollout 是分批推进对象
- Ops Job 是单次动作对象
- Rollout 通过一批或多批 `deploy.release.*` Job 落地
控制面责任:
- 决定下一批是否生成 Job
- 汇总 Job 结果
- 反推 Rollout 状态
- 决定是否暂停、继续或回滚
Node Agent 不负责:
- 推进下一批
- 修改 Rollout 策略
- 修改 Release 状态
---
## 10. 页面 / Codex 共同消费约束
以下对象必须作为稳定后端 contract 存在:
- `default_rollout_gate`
- `default_rollout_execution`
- `launchpad_status`
- `worker_rollout_preview`
- `control_rollout_preview`
页面只负责展示和少量交互兜底。
Codex 只负责消费后端给出的:
- 当前是否可发
- 建议先发哪一侧
- 应该使用什么 mode
- 下一步推荐动作是什么
---
## 11. Artifact Manifest Contract
Release Hub 不能只停在:
- `artifact_url`
- `checksum`
这只能证明“有个包”,还不能证明“这个包会把哪些服务改成什么样”。
建议把 `release.metadata.manifest` 固定为正式对象。
最小结构:
```json
{
"artifact_kind": "tarball",
"package_name": "domaincheck-2026.04.18+bd6bcb2.tgz",
"package_size_bytes": 186542331,
"checksum": "sha256:...",
"build_id": "build-20260418-001",
"commit_sha": "bd6bcb2",
"built_at": "2026-04-18 05:20:00",
"included_services": [
"domaincheck-api",
"domaincheck-worker",
"domaincheck-sync-agent",
"domaincheck-node-agent"
],
"required_env_keys": [
"NODE_CODE",
"NODE_REGION",
"NODE_ROLE"
],
"health_checks": [
{
"name": "api_health",
"type": "http",
"url": "http://127.0.0.1:8100/health"
}
],
"rollback_hint": "切回上一个 current 并重启对应服务"
}
```
正式要求:
- Release 创建后,`manifest` 不应再被覆盖成另一份语义不同的发布物
- Node Agent、页面、Codex 必须共用这同一份 manifest
- 是否可发、怎么回滚、要验哪些服务,不应再散落在:
- shell 脚本
- 页面表单
- 人工脑补
---
## 12. Operational Readiness Row Contract
`default_rollout_gate.operational_readiness.rows[]` 不应只是“几行提醒文案”,而应成为逐节点门禁对象。
最小结构:
```json
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"agent_managed": true,
"execution_mode_ready": true,
"execution_mode": "remote-agent",
"execution_mode_label": "Node Agent",
"inspection_status": "healthy",
"inspection_status_label": "已通过",
"current_release_version": "2026.04.11+1921318",
"desired_release_version": "2026.04.18+bd6bcb2",
"risk_level": "low",
"blocking_reasons": [],
"warning_reasons": [],
"recommended_action_code": "",
"focus_ref": {
"kind": "release_hub",
"release_id": 1,
"release_version": "2026.04.18+bd6bcb2",
"channel": "stable",
"rollout_id": 0,
"rollout_code": "",
"section": "default_rollout_gate"
}
}
```
正式要求:
- `rows[]` 应能直接回答:
- 这台节点能不能进下一轮 rollout
- 不能的话卡在哪
- 只是 warning 还是 hard block
- 页面、CLI、Codex 不应再从:
- inspection
- runtime
- nodes
三处数据手工拼门禁结论
推荐风险分级:
- `low`
- `medium`
- `high`
- `blocked`
---
## 13. Rollout Preview Contract
真正创建 Rollout 之前,还应保留一层正式 preview而不是由页面自己预估批次和风险。
建议入口:
- `POST /api/v1/ops/releases/{id}/rollouts/preview`
最小请求体:
```json
{
"release_id": 1,
"target_node_codes": [
"mainland-worker-01",
"mainland-controller-01"
],
"policy": {
"batch_size": 1,
"require_approval_between_batches": true,
"rollback_on_failure": true
},
"execution_mode": "remote-agent"
}
```
最小返回体:
```json
{
"release": {},
"execution_mode": "remote-agent",
"execution_mode_label": "Node Agent",
"target_nodes_total": 2,
"target_node_codes": [
"mainland-worker-01",
"mainland-controller-01"
],
"expected_batches_total": 2,
"expected_jobs_total": 2,
"gate": {},
"batch_plan": [
{
"batch_no": 1,
"node_codes": [
"mainland-worker-01"
],
"risk_level": "low"
},
{
"batch_no": 2,
"node_codes": [
"mainland-controller-01"
],
"risk_level": "medium"
}
],
"summary": "建议先灰度 1 台 Worker再推进 Controller。",
"summary_text": "建议先灰度 1 台 Worker再推进 Controller。",
"focus_ref": {
"kind": "release_hub",
"release_id": 1,
"release_version": "2026.04.18+bd6bcb2",
"channel": "stable",
"rollout_id": 0,
"rollout_code": "",
"section": "rollout_preview"
}
}
```
正式要求:
- preview 是:
- 页面发 rollout 前的统一预检
- Codex 自动驾驶发 rollout 前的统一预检
- CLI 批量发布前的统一预检
- 不允许前端重新自己算:
- 该分几批
- 哪台应该先发
- 哪台属于高风险
只有这样Release Hub 才是真正的发布控制面,而不是“能看版本的页面”。

View File

@@ -0,0 +1,633 @@
from __future__ import annotations
from fastapi import APIRouter
from app.schemas.common import ApiResponse
from app.services.ops_agent_service import (
build_managed_node_handover_bootstrap_plan,
execute_managed_node_onboarding_bootstrap,
execute_managed_node_onboarding_acceptance,
execute_managed_node_onboarding_recovery,
get_managed_node_delivery_queue,
get_managed_node_handover,
get_managed_node_onboarding,
list_managed_node_delivery_queue_records,
list_managed_nodes_with_agent_state,
list_ops_job_events,
preview_managed_node_onboarding_bootstrap,
preview_managed_node_onboarding_acceptance,
preview_managed_node_onboarding_recovery,
request_managed_node_delivery_queue_action,
summarize_ops_job_events,
)
from app.services.ops_job_service import (
approve_ops_job,
cancel_ops_job,
create_ops_job,
create_ops_job_batch,
dispatch_ops_job,
get_ops_job,
list_ops_jobs,
sync_managed_nodes_from_cluster,
upsert_managed_node,
)
from app.services.ops_playbook_service import (
cancel_ops_playbook_run,
execute_ops_playbook,
get_ops_playbook_run,
get_ops_playbooks,
get_recent_ops_playbook_runs,
list_ops_playbook_run_events,
preview_ops_playbook,
rerun_ops_playbook_run,
)
from app.services.ops_policy_service import preview_ops_job_policy
from app.services.ops_service import (
execute_codex_action,
execute_resolved_driver_action,
get_ops_activity_stream,
get_ops_codex_brief,
get_ops_contract_detail,
get_ops_contract_registry,
get_ops_doctor_decision,
get_ops_driver_feed,
get_ops_go_live_summary,
get_ops_go_live_signoff,
get_ops_go_live_bundle,
get_ops_go_live_review,
get_ops_link_snapshot,
get_ops_node_scene_log,
get_ops_stack_diagnosis,
execute_driver_action,
preview_driver_action,
resolve_codex_action,
resolve_driver_action_request,
resolve_ops_runbook_sequence,
execute_ops_runbook_sequence,
get_ops_blueprint,
get_ops_capabilities,
get_ops_inspection_overview,
get_ops_overview,
get_ops_runbook,
)
from app.services.ops_template_service import get_ops_action_templates
router = APIRouter(tags=["ops"])
@router.get("/ops/overview", response_model=ApiResponse)
def ops_overview() -> ApiResponse:
return ApiResponse(data=get_ops_overview())
@router.get("/ops/link-snapshot", response_model=ApiResponse)
def ops_link_snapshot() -> ApiResponse:
return ApiResponse(data=get_ops_link_snapshot())
@router.get("/ops/stack-diagnosis", response_model=ApiResponse)
def ops_stack_diagnosis(base_url: str = "") -> ApiResponse:
return ApiResponse(data=get_ops_stack_diagnosis(base_url=base_url))
@router.get("/ops/go-live-summary", response_model=ApiResponse)
def ops_go_live_summary(base_url: str = "") -> ApiResponse:
return ApiResponse(data=get_ops_go_live_summary(base_url=base_url))
@router.get("/ops/doctor-decision", response_model=ApiResponse)
def ops_doctor_decision(base_url: str = "", report_dir: str = "") -> ApiResponse:
return ApiResponse(data=get_ops_doctor_decision(base_url=base_url, report_dir=report_dir))
@router.get("/ops/go-live-signoff", response_model=ApiResponse)
def ops_go_live_signoff(base_url: str = "", report_dir: str = "") -> ApiResponse:
return ApiResponse(data=get_ops_go_live_signoff(base_url=base_url, report_dir=report_dir))
@router.get("/ops/go-live-bundle", response_model=ApiResponse)
def ops_go_live_bundle(base_url: str = "", report_dir: str = "") -> ApiResponse:
return ApiResponse(data=get_ops_go_live_bundle(base_url=base_url, report_dir=report_dir))
@router.get("/ops/go-live-review", response_model=ApiResponse)
def ops_go_live_review(base_url: str = "", report_dir: str = "") -> ApiResponse:
return ApiResponse(data=get_ops_go_live_review(base_url=base_url, report_dir=report_dir))
@router.get("/ops/inspection-overview", response_model=ApiResponse)
def ops_inspection_overview(
limit: int = 80,
status: str = "",
problem_kind: str = "",
query: str = "",
only_problem: bool = False,
only_participating: bool = False,
) -> ApiResponse:
return ApiResponse(
data=get_ops_inspection_overview(
fetch_limit=limit,
status=status,
problem_kind=problem_kind,
query=query,
only_problem=only_problem,
only_participating=only_participating,
)
)
@router.get("/ops/activity-stream", response_model=ApiResponse)
def ops_activity_stream(
limit: int = 18,
kind: str = "",
status: str = "",
execution_mode: str = "",
query: str = "",
) -> ApiResponse:
return ApiResponse(
data=get_ops_activity_stream(
limit=limit,
kind=kind,
status=status,
execution_mode=execution_mode,
query=query,
)
)
@router.get("/ops/driver-feed", response_model=ApiResponse)
def ops_driver_feed() -> ApiResponse:
return ApiResponse(data=get_ops_driver_feed())
@router.get("/ops/codex-brief", response_model=ApiResponse)
def ops_codex_brief() -> ApiResponse:
return ApiResponse(data=get_ops_codex_brief())
@router.get("/ops/capabilities", response_model=ApiResponse)
def ops_capabilities() -> ApiResponse:
return ApiResponse(data=get_ops_capabilities())
@router.get("/ops/contracts", response_model=ApiResponse)
def ops_contracts() -> ApiResponse:
return ApiResponse(data=get_ops_contract_registry())
@router.get("/ops/contracts/{contract_key}", response_model=ApiResponse)
def ops_contract_detail(contract_key: str) -> ApiResponse:
data = get_ops_contract_detail(contract_key)
if not data:
return ApiResponse(code=1, message="contract 不存在", data={})
return ApiResponse(data=data)
@router.get("/ops/action-templates", response_model=ApiResponse)
def ops_action_templates() -> ApiResponse:
return ApiResponse(data=get_ops_action_templates())
@router.get("/ops/playbooks", response_model=ApiResponse)
def ops_playbooks() -> ApiResponse:
return ApiResponse(data=get_ops_playbooks())
@router.get("/ops/playbook-runs", response_model=ApiResponse)
def ops_playbook_runs(
limit: int = 12,
status: str = "",
group_key: str = "",
query: str = "",
) -> ApiResponse:
return ApiResponse(
data=get_recent_ops_playbook_runs(
limit=limit,
status=status,
group_key=group_key,
query=query,
)
)
@router.get("/ops/playbook-runs/{run_code}", response_model=ApiResponse)
def ops_playbook_run_detail(run_code: str) -> ApiResponse:
data = get_ops_playbook_run(run_code)
if not data:
return ApiResponse(code=1, message="playbook run 不存在", data={})
return ApiResponse(data=data)
@router.get("/ops/playbook-runs/{run_code}/events", response_model=ApiResponse)
def ops_playbook_run_events(
run_code: str,
limit: int = 80,
step_key: str = "",
node_code: str = "",
) -> ApiResponse:
data = list_ops_playbook_run_events(
run_code,
limit=limit,
step_key=step_key,
node_code=node_code,
)
return ApiResponse(data=data)
@router.post("/ops/playbook-runs/{run_code}/rerun", response_model=ApiResponse)
def ops_playbook_run_rerun(run_code: str, payload: dict | None = None) -> ApiResponse:
payload = payload or {}
ok, message, data = rerun_ops_playbook_run(
run_code,
requested_by=str(payload.get("requested_by") or "api").strip() or "api",
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/playbook-runs/{run_code}/cancel", response_model=ApiResponse)
def ops_playbook_run_cancel(run_code: str, payload: dict | None = None) -> ApiResponse:
payload = payload or {}
ok, message, data = cancel_ops_playbook_run(
run_code,
cancelled_by=str(payload.get("cancelled_by") or "api").strip() or "api",
reason=str(payload.get("reason") or "").strip(),
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/playbooks/preview", response_model=ApiResponse)
def ops_playbook_preview(payload: dict) -> ApiResponse:
ok, message, data = preview_ops_playbook(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/playbooks/execute", response_model=ApiResponse)
def ops_playbook_execute(payload: dict) -> ApiResponse:
ok, message, data = execute_ops_playbook(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.get("/ops/blueprint", response_model=ApiResponse)
def ops_blueprint() -> ApiResponse:
return ApiResponse(data=get_ops_blueprint())
@router.get("/ops/runbook", response_model=ApiResponse)
def ops_runbook() -> ApiResponse:
return ApiResponse(data=get_ops_runbook())
@router.post("/ops/runbook/sequences/{sequence_key}/resolve", response_model=ApiResponse)
def ops_runbook_sequence_resolve(sequence_key: str, payload: dict | None = None) -> ApiResponse:
ok, message, data = resolve_ops_runbook_sequence(sequence_key, payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/runbook/sequences/{sequence_key}/execute", response_model=ApiResponse)
def ops_runbook_sequence_execute(sequence_key: str, payload: dict | None = None) -> ApiResponse:
ok, message, data = execute_ops_runbook_sequence(sequence_key, payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.get("/ops/nodes", response_model=ApiResponse)
def ops_nodes() -> ApiResponse:
return ApiResponse(data=list_managed_nodes_with_agent_state())
@router.get("/ops/nodes/{node_code}/handover", response_model=ApiResponse)
def ops_node_handover(
node_code: str,
control_plane_base_url: str = "",
root_dir: str = "",
) -> ApiResponse:
data = get_managed_node_handover(
node_code,
control_plane_base_url=control_plane_base_url,
root_dir=root_dir,
)
if not data:
return ApiResponse(code=1, message="节点不存在", data={})
return ApiResponse(data=data)
@router.get("/ops/nodes/{node_code}/onboarding", response_model=ApiResponse)
def ops_node_onboarding(
node_code: str,
control_plane_base_url: str = "",
root_dir: str = "",
) -> ApiResponse:
data = get_managed_node_onboarding(
node_code,
control_plane_base_url=control_plane_base_url,
root_dir=root_dir,
)
if not data:
return ApiResponse(code=1, message="节点不存在", data={})
return ApiResponse(data=data)
@router.post("/ops/nodes/{node_code}/onboarding/acceptance/preview", response_model=ApiResponse)
def ops_node_onboarding_acceptance_preview(node_code: str, payload: dict | None = None) -> ApiResponse:
normalized_payload = payload or {}
ok, message, data = preview_managed_node_onboarding_acceptance(
node_code=node_code,
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/{node_code}/onboarding/bootstrap/preview", response_model=ApiResponse)
def ops_node_onboarding_bootstrap_preview(node_code: str, payload: dict | None = None) -> ApiResponse:
normalized_payload = payload or {}
ok, message, data = preview_managed_node_onboarding_bootstrap(
node_code=node_code,
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/{node_code}/onboarding/bootstrap/execute", response_model=ApiResponse)
def ops_node_onboarding_bootstrap_execute(node_code: str, payload: dict | None = None) -> ApiResponse:
normalized_payload = payload or {}
ok, message, data = execute_managed_node_onboarding_bootstrap(
node_code=node_code,
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/{node_code}/onboarding/acceptance/execute", response_model=ApiResponse)
def ops_node_onboarding_acceptance_execute(node_code: str, payload: dict | None = None) -> ApiResponse:
normalized_payload = payload or {}
ok, message, data = execute_managed_node_onboarding_acceptance(
node_code=node_code,
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/{node_code}/onboarding/recovery/preview", response_model=ApiResponse)
def ops_node_onboarding_recovery_preview(node_code: str, payload: dict | None = None) -> ApiResponse:
normalized_payload = payload or {}
ok, message, data = preview_managed_node_onboarding_recovery(
node_code=node_code,
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/{node_code}/onboarding/recovery/execute", response_model=ApiResponse)
def ops_node_onboarding_recovery_execute(node_code: str, payload: dict | None = None) -> ApiResponse:
normalized_payload = payload or {}
ok, message, data = execute_managed_node_onboarding_recovery(
node_code=node_code,
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.get("/ops/nodes/{node_code}/scene-log", response_model=ApiResponse)
def ops_node_scene_log(
node_code: str,
limit: int = 80,
mode: str = "",
) -> ApiResponse:
data = get_ops_node_scene_log(node_code, limit=limit, mode=mode)
if not data:
return ApiResponse(code=1, message="节点不存在", data={})
return ApiResponse(data=data)
@router.post("/ops/nodes/{node_code}/handover/bootstrap-plan", response_model=ApiResponse)
def ops_node_handover_bootstrap_plan(node_code: str, payload: dict | None = None) -> ApiResponse:
normalized_payload = payload or {}
ok, message, data = build_managed_node_handover_bootstrap_plan(
node_code=node_code,
issued_by=str(normalized_payload.get("issued_by") or "api").strip() or "api",
expires_in_hours=int(normalized_payload.get("expires_in_hours") or 72),
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
metadata=normalized_payload.get("metadata") or {},
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.get("/ops/nodes/{node_code}/delivery-queue", response_model=ApiResponse)
def ops_node_delivery_queue(node_code: str) -> ApiResponse:
data = get_managed_node_delivery_queue(node_code)
if not data:
return ApiResponse(code=1, message="节点不存在", data={})
return ApiResponse(data=data)
@router.get("/ops/nodes/{node_code}/delivery-queue/records", response_model=ApiResponse)
def ops_node_delivery_queue_records(
node_code: str,
state: str = "",
limit: int = 50,
) -> ApiResponse:
data = list_managed_node_delivery_queue_records(
node_code,
state=state,
limit=limit,
)
if not data:
return ApiResponse(code=1, message="节点不存在", data={})
return ApiResponse(data=data)
@router.post("/ops/nodes/{node_code}/delivery-queue/flush", response_model=ApiResponse)
def ops_node_delivery_queue_flush(node_code: str, payload: dict | None = None) -> ApiResponse:
ok, message, data = request_managed_node_delivery_queue_action(
node_code,
"delivery.queue.flush",
payload=payload or {},
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/{node_code}/delivery-queue/replay", response_model=ApiResponse)
def ops_node_delivery_queue_replay(node_code: str, payload: dict | None = None) -> ApiResponse:
ok, message, data = request_managed_node_delivery_queue_action(
node_code,
"delivery.queue.replay",
payload=payload or {},
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay", response_model=ApiResponse)
def ops_node_delivery_queue_record_replay(
node_code: str,
record_id: str,
payload: dict | None = None,
) -> ApiResponse:
ok, message, data = request_managed_node_delivery_queue_action(
node_code,
"delivery.queue.replay",
payload=payload or {},
record_id=record_id,
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard", response_model=ApiResponse)
def ops_node_delivery_queue_record_discard(
node_code: str,
record_id: str,
payload: dict | None = None,
) -> ApiResponse:
ok, message, data = request_managed_node_delivery_queue_action(
node_code,
"delivery.queue.discard",
payload=payload or {},
record_id=record_id,
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes", response_model=ApiResponse)
def ops_nodes_upsert(payload: dict) -> ApiResponse:
ok, message, data = upsert_managed_node(payload)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/nodes/sync-from-cluster", response_model=ApiResponse)
def ops_nodes_sync_from_cluster(dry_run: bool = False) -> ApiResponse:
return ApiResponse(data=sync_managed_nodes_from_cluster(dry_run=dry_run))
@router.get("/ops/jobs", response_model=ApiResponse)
def ops_jobs(limit: int = 20, compact: bool = True, rollout_id: int = 0) -> ApiResponse:
normalized_rollout_id = int(rollout_id or 0)
jobs = list_ops_jobs(
limit=limit,
compact=compact,
rollout_id=normalized_rollout_id if normalized_rollout_id > 0 else None,
)
page_status_counts: dict[str, int] = {}
for item in jobs:
item_status = str(item.get("status") or "").strip() or "unknown"
page_status_counts[item_status] = int(page_status_counts.get(item_status, 0) or 0) + 1
return ApiResponse(
data={
"jobs": jobs,
"filters": {
"limit": int(limit or 20),
"compact": bool(compact),
"rollout_id": normalized_rollout_id,
},
"page_summary": {
"jobs_total": len(jobs),
"status_counts": page_status_counts,
},
}
)
@router.get("/ops/jobs/{job_id}", response_model=ApiResponse)
def ops_job_detail(job_id: int) -> ApiResponse:
data = get_ops_job(job_id)
if not data:
return ApiResponse(code=1, message="任务不存在", data={})
return ApiResponse(data=data)
@router.get("/ops/jobs/{job_id}/events", response_model=ApiResponse)
def ops_job_events(job_id: int, limit: int = 50) -> ApiResponse:
events = list_ops_job_events(job_id, limit=limit)
return ApiResponse(
data={
"events": events,
"summary": summarize_ops_job_events(events, job_id=job_id, limit=limit),
}
)
@router.post("/ops/driver-actions/execute", response_model=ApiResponse)
def ops_driver_action_execute(payload: dict) -> ApiResponse:
ok, message, data = execute_driver_action(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/driver-actions/preview", response_model=ApiResponse)
def ops_driver_action_preview(payload: dict) -> ApiResponse:
ok, message, data = preview_driver_action(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/driver-actions/resolve", response_model=ApiResponse)
def ops_driver_action_resolve(payload: dict | None = None) -> ApiResponse:
ok, message, data = resolve_driver_action_request(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/driver-actions/execute-resolved", response_model=ApiResponse)
def ops_driver_action_execute_resolved(payload: dict | None = None) -> ApiResponse:
ok, message, data = execute_resolved_driver_action(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/codex-actions/resolve", response_model=ApiResponse)
def ops_codex_action_resolve(payload: dict | None = None) -> ApiResponse:
ok, message, data = resolve_codex_action(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/codex-actions/execute", response_model=ApiResponse)
def ops_codex_action_execute(payload: dict | None = None) -> ApiResponse:
ok, message, data = execute_codex_action(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/jobs", response_model=ApiResponse)
def ops_jobs_create(payload: dict) -> ApiResponse:
ok, message, data = create_ops_job(payload)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/jobs/batch", response_model=ApiResponse)
def ops_jobs_create_batch(payload: dict) -> ApiResponse:
ok, message, data = create_ops_job_batch(payload)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/policy/preview", response_model=ApiResponse)
def ops_policy_preview(payload: dict) -> ApiResponse:
return ApiResponse(data=preview_ops_job_policy(payload))
@router.post("/ops/jobs/{job_id}/approve", response_model=ApiResponse)
def ops_job_approve(job_id: int, payload: dict | None = None) -> ApiResponse:
payload = payload or {}
ok, message, data = approve_ops_job(job_id, approved_by=str(payload.get("approved_by") or "api").strip() or "api")
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/jobs/{job_id}/cancel", response_model=ApiResponse)
def ops_job_cancel(job_id: int, payload: dict | None = None) -> ApiResponse:
payload = payload or {}
ok, message, data = cancel_ops_job(
job_id,
cancelled_by=str(payload.get("cancelled_by") or "api").strip() or "api",
reason=str(payload.get("reason") or "").strip(),
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/jobs/{job_id}/dispatch", response_model=ApiResponse)
def ops_job_dispatch(job_id: int) -> ApiResponse:
ok, message, data = dispatch_ops_job(job_id)
return ApiResponse(code=0 if ok else 1, message=message, data=data)

View File

@@ -0,0 +1,90 @@
from typing import Optional
from fastapi import APIRouter, Header
from app.schemas.common import ApiResponse
from app.services.ops_agent_service import (
agent_append_job_event,
agent_complete_job,
agent_heartbeat,
agent_mark_job_started,
agent_pull_jobs,
agent_register,
build_node_agent_bootstrap_plan,
issue_node_agent_token,
)
router = APIRouter(tags=["ops-agent"])
def _resolve_agent_token(x_domaincheck_agent_token: Optional[str]) -> str:
return str(x_domaincheck_agent_token or "").strip()
def _build_agent_response(ok: bool, message: str, data: object) -> ApiResponse:
detail_code = None
if isinstance(data, dict):
detail_code = str(data.get("detail_code") or "").strip() or None
return ApiResponse(code=0 if ok else 1, message=message, data=data, detail_code=detail_code)
@router.post("/ops/agent/tokens", response_model=ApiResponse)
def ops_agent_issue_token(payload: dict) -> ApiResponse:
ok, message, data = issue_node_agent_token(
node_code=str(payload.get("node_code") or "").strip(),
issued_by=str(payload.get("issued_by") or "api").strip() or "api",
expires_in_hours=int(payload.get("expires_in_hours") or 72),
metadata=payload.get("metadata") or {},
)
return _build_agent_response(ok, message, data)
@router.post("/ops/agent/bootstrap-plan", response_model=ApiResponse)
def ops_agent_bootstrap_plan(payload: dict) -> ApiResponse:
ok, message, data = build_node_agent_bootstrap_plan(
node_code=str(payload.get("node_code") or "").strip(),
node_region=str(payload.get("node_region") or "mainland").strip() or "mainland",
node_role=str(payload.get("node_role") or "worker").strip() or "worker",
issued_by=str(payload.get("issued_by") or "api").strip() or "api",
expires_in_hours=int(payload.get("expires_in_hours") or 72),
control_plane_base_url=str(payload.get("control_plane_base_url") or "").strip(),
root_dir=str(payload.get("root_dir") or "/opt/domaincheck").strip() or "/opt/domaincheck",
metadata=payload.get("metadata") or {},
)
return _build_agent_response(ok, message, data)
@router.post("/ops/agent/register", response_model=ApiResponse)
def ops_agent_register(payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
ok, message, data = agent_register(payload, token=_resolve_agent_token(x_domaincheck_agent_token))
return _build_agent_response(ok, message, data)
@router.post("/ops/agent/heartbeat", response_model=ApiResponse)
def ops_agent_heartbeat(payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
ok, message, data = agent_heartbeat(payload, token=_resolve_agent_token(x_domaincheck_agent_token))
return _build_agent_response(ok, message, data)
@router.post("/ops/agent/pull", response_model=ApiResponse)
def ops_agent_pull(payload: dict, limit: int = 1, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
ok, message, data = agent_pull_jobs(payload, token=_resolve_agent_token(x_domaincheck_agent_token), limit=limit)
return _build_agent_response(ok, message, data)
@router.post("/ops/agent/jobs/{job_id}/start", response_model=ApiResponse)
def ops_agent_job_start(job_id: int, payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
ok, message, data = agent_mark_job_started(job_id, payload, token=_resolve_agent_token(x_domaincheck_agent_token))
return _build_agent_response(ok, message, data)
@router.post("/ops/agent/jobs/{job_id}/complete", response_model=ApiResponse)
def ops_agent_job_complete(job_id: int, payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
ok, message, data = agent_complete_job(job_id, payload, token=_resolve_agent_token(x_domaincheck_agent_token))
return _build_agent_response(ok, message, data)
@router.post("/ops/agent/jobs/{job_id}/events", response_model=ApiResponse)
def ops_agent_job_event(job_id: int, payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
ok, message, data = agent_append_job_event(job_id, payload, token=_resolve_agent_token(x_domaincheck_agent_token))
return _build_agent_response(ok, message, data)

View File

@@ -0,0 +1,175 @@
from __future__ import annotations
from fastapi import APIRouter, Request
from fastapi.responses import FileResponse
from app.schemas.common import ApiResponse
from app.services.ops_release_service import (
advance_release_rollout,
activate_release,
create_release,
create_release_and_smart_rollout_from_latest_package,
create_release_from_latest_package,
create_release_rollout,
create_smart_release_rollout,
get_latest_release_package_metadata,
get_release_launchpad,
get_latest_release,
get_release_package_archive_path,
get_release_package_sha256_path,
get_release,
get_release_rollout,
list_release_rollout_jobs,
list_release_rollouts,
list_releases,
preview_release_from_latest_package,
preview_smart_release_rollout,
preview_smart_release_rollout_from_latest_package,
)
router = APIRouter(tags=["ops-release"])
@router.get("/ops/releases", response_model=ApiResponse)
def ops_releases(limit: int = 20, channel: str | None = None) -> ApiResponse:
return ApiResponse(data={"releases": list_releases(limit=limit, channel=channel)})
@router.get("/ops/releases/latest", response_model=ApiResponse)
def ops_releases_latest(channel: str = "stable") -> ApiResponse:
return ApiResponse(data=get_latest_release(channel=channel))
@router.get("/ops/releases/package-metadata/latest", response_model=ApiResponse)
def ops_releases_package_metadata_latest() -> ApiResponse:
return ApiResponse(data=get_latest_release_package_metadata())
@router.get("/ops/releases/launchpad", response_model=ApiResponse)
def ops_releases_launchpad(request: Request, channel: str = "stable") -> ApiResponse:
derived_base_url = str(request.base_url).rstrip("/")
return ApiResponse(data=get_release_launchpad(control_plane_base_url=derived_base_url, channel=channel))
@router.get("/ops/releases/packages/{package_name}/download")
def ops_release_package_download(package_name: str):
try:
archive_path = get_release_package_archive_path(package_name)
except ValueError:
return ApiResponse(code=1, message="发布包名称不合法", data={})
if not archive_path:
return ApiResponse(code=1, message="发布包不存在", data={})
media_type = "application/octet-stream"
archive_name = archive_path.name.lower()
if archive_name.endswith(".tar.gz"):
media_type = "application/gzip"
elif archive_name.endswith(".zip"):
media_type = "application/zip"
return FileResponse(path=archive_path, filename=archive_path.name, media_type=media_type)
@router.get("/ops/releases/packages/{package_name}/sha256")
def ops_release_package_sha256(package_name: str):
try:
sha256_path = get_release_package_sha256_path(package_name)
except ValueError:
return ApiResponse(code=1, message="发布包名称不合法", data={})
if not sha256_path:
return ApiResponse(code=1, message="发布包校验文件不存在", data={})
return FileResponse(path=sha256_path, filename=sha256_path.name, media_type="text/plain; charset=utf-8")
@router.get("/ops/releases/{release_id}", response_model=ApiResponse)
def ops_releases_detail(release_id: int) -> ApiResponse:
data = get_release(release_id)
if not data:
return ApiResponse(code=1, message="Release 不存在", data={})
return ApiResponse(data=data)
@router.post("/ops/releases", response_model=ApiResponse)
def ops_releases_create(payload: dict) -> ApiResponse:
ok, message, data = create_release(payload)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/releases/from-package/latest", response_model=ApiResponse)
def ops_releases_create_from_latest_package(request: Request, payload: dict | None = None) -> ApiResponse:
derived_base_url = str(request.base_url).rstrip("/")
ok, message, data = create_release_from_latest_package(payload or {}, control_plane_base_url=derived_base_url)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/releases/from-package/latest/preview", response_model=ApiResponse)
def ops_releases_preview_from_latest_package(request: Request, payload: dict | None = None) -> ApiResponse:
derived_base_url = str(request.base_url).rstrip("/")
ok, message, data = preview_release_from_latest_package(payload or {}, control_plane_base_url=derived_base_url)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/releases/from-package/latest/smart-rollout", response_model=ApiResponse)
def ops_releases_create_from_latest_package_and_smart_rollout(request: Request, payload: dict | None = None) -> ApiResponse:
derived_base_url = str(request.base_url).rstrip("/")
ok, message, data = create_release_and_smart_rollout_from_latest_package(
payload or {},
control_plane_base_url=derived_base_url,
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/releases/from-package/latest/smart-rollout-preview", response_model=ApiResponse)
def ops_releases_preview_from_latest_package_and_smart_rollout(request: Request, payload: dict | None = None) -> ApiResponse:
derived_base_url = str(request.base_url).rstrip("/")
ok, message, data = preview_smart_release_rollout_from_latest_package(
payload or {},
control_plane_base_url=derived_base_url,
)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/releases/{release_id}/activate", response_model=ApiResponse)
def ops_releases_activate(release_id: int) -> ApiResponse:
ok, message, data = activate_release(release_id)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/releases/{release_id}/smart-rollout-preview", response_model=ApiResponse)
def ops_releases_smart_rollout_preview(release_id: int, payload: dict | None = None) -> ApiResponse:
ok, message, data = preview_smart_release_rollout(release_id, payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/releases/{release_id}/smart-rollout", response_model=ApiResponse)
def ops_releases_smart_rollout(release_id: int, payload: dict | None = None) -> ApiResponse:
ok, message, data = create_smart_release_rollout(release_id, payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.get("/ops/releases/{release_id}/rollouts", response_model=ApiResponse)
def ops_release_rollouts(release_id: int, limit: int = 20) -> ApiResponse:
return ApiResponse(data={"rollouts": list_release_rollouts(release_id=release_id, limit=limit)})
@router.get("/ops/rollouts/{rollout_id}", response_model=ApiResponse)
def ops_rollout_detail(rollout_id: int) -> ApiResponse:
data = get_release_rollout(rollout_id)
if not data:
return ApiResponse(code=1, message="Rollout 不存在", data={})
return ApiResponse(data=data)
@router.get("/ops/rollouts/{rollout_id}/jobs", response_model=ApiResponse)
def ops_rollout_jobs(rollout_id: int, limit: int = 200) -> ApiResponse:
return ApiResponse(data={"jobs": list_release_rollout_jobs(rollout_id, limit=limit)})
@router.post("/ops/rollouts/{rollout_id}/advance", response_model=ApiResponse)
def ops_rollout_advance(rollout_id: int, payload: dict | None = None) -> ApiResponse:
ok, message, data = advance_release_rollout(rollout_id, payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/releases/{release_id}/rollouts", response_model=ApiResponse)
def ops_release_rollouts_create(release_id: int, payload: dict) -> ApiResponse:
ok, message, data = create_release_rollout(release_id, payload)
return ApiResponse(code=0 if ok else 1, message=message, data=data)

View File

@@ -3,6 +3,7 @@ from typing import Optional
from fastapi import APIRouter, Header
from app.schemas.common import ApiResponse
from app.services.build_info_service import get_runtime_build_info
from app.services.cluster_runtime_service import get_cluster_snapshot
from app.services.debug_event_service import get_debug_diagnosis, get_debug_event_overview, get_debug_handoff_report, ingest_debug_event, list_debug_events
from app.services.runtime_control_service import runtime_action
@@ -37,6 +38,11 @@ def runtime_cluster() -> ApiResponse:
return ApiResponse(data=get_cluster_snapshot())
@router.get("/runtime/build-info", response_model=ApiResponse)
def runtime_build_info() -> ApiResponse:
return ApiResponse(data=get_runtime_build_info())
@router.get("/runtime/sync-summary", response_model=ApiResponse)
def runtime_sync_summary() -> ApiResponse:
return ApiResponse(data=get_sync_summary())

View File

@@ -42,6 +42,13 @@ class Settings(BaseSettings):
sync_shared_token: str = ""
sync_batch_size: int = 200
sync_poll_interval_seconds: int = 30
build_manifest_path: str = ""
build_commit_sha: str = ""
build_commit_ref: str = ""
build_generated_at: str = ""
build_package_name: str = ""
build_checksum: str = ""
build_source_label: str = ""
@property
def cors_origins_list(self) -> list[str]:

View File

@@ -3,9 +3,13 @@ import threading
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from app.api.routes import auth, dashboard, settings as settings_routes, imports, detect, domains, exports, logs, runtime, juming, sensitive_words
from app.api.routes import auth, dashboard, settings as settings_routes, imports, detect, domains, exports, logs, runtime, juming, ops, ops_agent, ops_release, sensitive_words
from app.core.config import settings as app_settings
from app.services.build_info_service import get_runtime_build_info, remember_registered_route_paths
from app.services.cluster_runtime_service import ensure_runtime_schema, register_local_control_heartbeat
from app.services.ops_agent_service import ensure_ops_agent_schema
from app.services.ops_job_service import ensure_ops_schema
from app.services.ops_release_service import ensure_ops_release_schema
_heartbeat_stop_event = threading.Event()
@@ -36,6 +40,10 @@ app.add_middleware(
@app.on_event("startup")
def on_startup() -> None:
ensure_runtime_schema()
ensure_ops_schema()
ensure_ops_agent_schema()
ensure_ops_release_schema()
remember_registered_route_paths(route.path for route in app.routes)
register_local_control_heartbeat()
_heartbeat_stop_event.clear()
threading.Thread(target=_control_heartbeat_loop, name="control-heartbeat", daemon=True).start()
@@ -56,6 +64,7 @@ def health() -> dict:
"worker_mode": app_settings.worker_mode,
"api_host": app_settings.api_host,
"api_port": app_settings.api_port,
"build": get_runtime_build_info(route_paths=(route.path for route in app.routes)),
}
@@ -69,4 +78,8 @@ app.include_router(exports.router, prefix=app_settings.api_prefix)
app.include_router(logs.router, prefix=app_settings.api_prefix)
app.include_router(runtime.router, prefix=app_settings.api_prefix)
app.include_router(juming.router, prefix=app_settings.api_prefix)
app.include_router(ops.router, prefix=app_settings.api_prefix)
app.include_router(ops_agent.router, prefix=app_settings.api_prefix)
app.include_router(ops_release.router, prefix=app_settings.api_prefix)
app.include_router(sensitive_words.router, prefix=app_settings.api_prefix)
remember_registered_route_paths(route.path for route in app.routes)

View File

@@ -0,0 +1,991 @@
from __future__ import annotations
import json
import os
import socket
import subprocess
import time
import urllib.error
import urllib.request
from datetime import datetime
from uuid import uuid4
from app.services.ops_action_executor_core import (
build_service_name_map,
execute_structured_action,
supports_structured_action,
)
from app.services.ops_release_executor_core import (
execute_release_action,
normalize_release_health_check_services as _release_normalize_health_check_services,
normalize_text_list as _release_normalize_text_list,
)
CONTROL_PLANE_BASE_URL = str(os.getenv("OPS_CONTROL_PLANE_BASE_URL", "")).strip().rstrip("/")
AGENT_TOKEN = str(os.getenv("OPS_AGENT_TOKEN", "")).strip()
NODE_CODE = str(os.getenv("NODE_CODE", "")).strip()
NODE_REGION = str(os.getenv("NODE_REGION", "mainland")).strip() or "mainland"
NODE_ROLE = str(os.getenv("NODE_ROLE", "worker")).strip() or "worker"
AGENT_POLL_INTERVAL_SECONDS = max(2, int(os.getenv("OPS_AGENT_POLL_INTERVAL_SECONDS", "5") or 5))
WORKER_SERVICE_NAME = str(os.getenv("WORKER_SERVICE_NAME", os.getenv("WORKER_SERVICE", "domaincheck-worker"))).strip() or "domaincheck-worker"
API_SERVICE_NAME = str(os.getenv("API_SERVICE_NAME", "domaincheck-api")).strip() or "domaincheck-api"
SYNC_AGENT_SERVICE_NAME = str(os.getenv("SYNC_AGENT_SERVICE_NAME", "domaincheck-sync-agent")).strip() or "domaincheck-sync-agent"
NODE_AGENT_SERVICE_NAME = str(os.getenv("NODE_AGENT_SERVICE_NAME", "domaincheck-node-agent")).strip() or "domaincheck-node-agent"
AGENT_VERSION = "0.1.0"
_APP_DIR = os.path.dirname(os.path.abspath(__file__))
_PROJECT_DIR = os.path.dirname(_APP_DIR)
_DEFAULT_QUEUE_DIR = os.path.join(_PROJECT_DIR, "runtime", "node-agent-queue", NODE_CODE or "unbound")
AGENT_QUEUE_DIR = str(os.getenv("OPS_AGENT_QUEUE_DIR", _DEFAULT_QUEUE_DIR)).strip() or _DEFAULT_QUEUE_DIR
AGENT_QUEUE_FLUSH_LIMIT = max(1, int(os.getenv("OPS_AGENT_QUEUE_FLUSH_LIMIT", "20") or 20))
_PERMANENT_DELIVERY_DETAIL_CODES = {
"agent_node_code_required",
"ops_job_not_owned_by_agent",
"ops_job_invalid_status",
}
_LAST_QUEUE_FLUSH_SUMMARY = {
"scanned": 0,
"delivered": 0,
"deferred": 0,
"dead_letter": 0,
"last_flush_at": "",
}
def _normalize_text_list(raw_value: object) -> list[str]:
"""Keep node-agent payload normalization aligned with release executor helpers."""
return _release_normalize_text_list(raw_value)
def _normalize_release_health_check_services(
payload: dict | None,
restart_services: list[str] | None,
) -> list[str]:
"""Backward-compatible wrapper used by node-agent tests and payload shaping."""
return _release_normalize_health_check_services(
dict(payload or {}),
list(restart_services or []),
default_api_service_name=API_SERVICE_NAME,
)
def _log(message: str) -> None:
print(f"{datetime.now().isoformat(sep=' ', timespec='seconds')} [node-agent] {message}", flush=True)
def _json_env(name: str, fallback: object) -> object:
raw = str(os.getenv(name, "")).strip()
if not raw:
return fallback
try:
return json.loads(raw)
except Exception:
return fallback
AGENT_CAPABILITIES = _json_env(
"OPS_AGENT_CAPABILITIES",
[
"service.start",
"service.stop",
"service.restart",
"service.status",
"runtime.start_worker",
"runtime.stop_worker",
"runtime.restart_api",
"runtime.start_sync_agent",
"runtime.stop_sync_agent",
"health.snapshot",
"logs.collect",
"diagnostics.collect",
"delivery.queue.flush",
"delivery.queue.replay",
"delivery.queue.discard",
"deploy.release",
],
)
AGENT_LABELS = _json_env("OPS_AGENT_LABELS", {})
class AgentResponseError(RuntimeError):
def __init__(self, message: str, *, detail_code: str = "", payload: dict | None = None):
super().__init__(message)
self.detail_code = str(detail_code or "").strip()
self.payload = dict(payload or {})
def _now_text() -> str:
return datetime.now().isoformat(sep=" ", timespec="seconds")
def _queue_pending_dir() -> str:
return os.path.join(AGENT_QUEUE_DIR, "pending")
def _queue_dead_letter_dir() -> str:
return os.path.join(AGENT_QUEUE_DIR, "dead-letter")
def _queue_discarded_dir() -> str:
return os.path.join(AGENT_QUEUE_DIR, "discarded")
def _ensure_queue_dirs() -> None:
for path in (AGENT_QUEUE_DIR, _queue_pending_dir(), _queue_dead_letter_dir(), _queue_discarded_dir()):
os.makedirs(path, exist_ok=True)
def _write_json_file(path: str, payload: dict) -> None:
temp_path = f"{path}.tmp"
with open(temp_path, "w", encoding="utf-8") as handle:
json.dump(payload, handle, ensure_ascii=False, indent=2)
os.replace(temp_path, path)
def _read_json_file(path: str) -> dict:
with open(path, "r", encoding="utf-8") as handle:
return json.load(handle)
def _new_request_id(prefix: str, job_id: int) -> str:
normalized_prefix = str(prefix or "request").strip() or "request"
return f"{normalized_prefix}-{int(job_id or 0)}-{int(time.time() * 1000)}-{uuid4().hex[:10]}"
def _queue_file_names(directory: str) -> list[str]:
_ensure_queue_dirs()
try:
return sorted(file_name for file_name in os.listdir(directory) if file_name.endswith(".json"))
except FileNotFoundError:
return []
def _queue_head_record(directory: str) -> dict:
file_names = _queue_file_names(directory)
if not file_names:
return {}
file_path = os.path.join(directory, file_names[0])
try:
return _read_json_file(file_path)
except Exception:
return {}
def _queue_record_id(record: dict, *, file_name: str = "") -> str:
request_id = str((record or {}).get("request_id") or "").strip()
if request_id:
return request_id
normalized_file_name = str(file_name or "").strip()
if normalized_file_name.endswith(".json"):
return normalized_file_name[:-5]
return normalized_file_name
def _queue_entries(state: str) -> list[dict]:
normalized_state = str(state or "").strip()
if normalized_state == "pending":
directory = _queue_pending_dir()
elif normalized_state == "dead_letter":
directory = _queue_dead_letter_dir()
else:
return []
entries: list[dict] = []
for file_name in _queue_file_names(directory):
file_path = os.path.join(directory, file_name)
try:
record = _read_json_file(file_path)
except Exception:
continue
entries.append(
{
"state": normalized_state,
"file_name": file_name,
"file_path": file_path,
"record_id": _queue_record_id(record, file_name=file_name),
"record": dict(record or {}),
}
)
return entries
def _matches_delivery_selector(entry: dict, selector: dict, *, allowed_states: set[str] | None = None) -> bool:
normalized_selector = dict(selector or {})
record = dict(entry.get("record") or {})
state = str(entry.get("state") or "").strip()
if allowed_states and state not in allowed_states:
return False
selected_state = str(normalized_selector.get("state") or "").strip()
if selected_state and state != selected_state:
return False
selected_record_id = str(
normalized_selector.get("record_id")
or normalized_selector.get("request_id")
or ""
).strip()
if selected_record_id and str(entry.get("record_id") or "").strip() != selected_record_id:
return False
selected_request_kind = str(normalized_selector.get("request_kind") or "").strip()
if selected_request_kind and str(record.get("kind") or "").strip() != selected_request_kind:
return False
selected_detail_code = str(normalized_selector.get("detail_code") or "").strip()
if selected_detail_code and str(record.get("last_detail_code") or "").strip() != selected_detail_code:
return False
return True
def _trim_queue_preview(records: list[dict], *, limit: int = 5) -> list[dict]:
safe_limit = max(1, int(limit or 5))
preview: list[dict] = []
for entry in list(records or [])[:safe_limit]:
record = dict(entry.get("record") or {})
preview.append(
{
"record_id": str(entry.get("record_id") or "").strip(),
"state": str(entry.get("state") or "").strip(),
"request_kind": str(record.get("kind") or "").strip(),
"request_id": str(record.get("request_id") or "").strip(),
"detail_code": str(record.get("last_detail_code") or "").strip(),
"created_at": str(record.get("created_at") or "").strip(),
"updated_at": str(record.get("updated_at") or "").strip(),
}
)
return preview
def _normalize_queue_limit(raw_value: object, *, default: int = 20, minimum: int = 1, maximum: int = 200) -> int:
try:
value = int(raw_value or default)
except Exception:
value = int(default)
return max(minimum, min(maximum, value))
def _replay_dead_letter_records(payload: dict | None = None) -> tuple[bool, str, dict]:
normalized_payload = dict(payload or {})
selector = dict(normalized_payload.get("selector") or {})
selector["state"] = str(selector.get("state") or "dead_letter").strip() or "dead_letter"
limit = _normalize_queue_limit(normalized_payload.get("limit"), default=20)
flush_after_replay = bool(normalized_payload.get("flush_after_replay", True))
reason = str(normalized_payload.get("reason") or "").strip()
matched_entries = [
entry
for entry in _queue_entries("dead_letter")
if _matches_delivery_selector(entry, selector, allowed_states={"dead_letter"})
][:limit]
if not matched_entries:
return False, "未找到符合条件的死信记录", {
"selector": selector,
"limit": limit,
"queue": _delivery_queue_snapshot(),
}
replayed_total = 0
for entry in matched_entries:
record = dict(entry.get("record") or {})
record["updated_at"] = _now_text()
record["replay_count"] = int(record.get("replay_count") or 0) + 1
record["last_replay_at"] = _now_text()
if reason:
record["last_replay_reason"] = reason
record.pop("dead_letter_at", None)
record.pop("dead_letter_reason", None)
_store_pending_delivery(record)
try:
os.remove(str(entry.get("file_path") or ""))
except FileNotFoundError:
pass
replayed_total += 1
flush_summary = {}
if flush_after_replay and replayed_total > 0:
flush_summary = _flush_delivery_queue(limit=replayed_total)
result = {
"selector": selector,
"limit": limit,
"replayed_total": replayed_total,
"flush_after_replay": flush_after_replay,
"flush_summary": flush_summary,
"matched_records": _trim_queue_preview(matched_entries),
"queue": _delivery_queue_snapshot(),
}
return True, f"已重放 {replayed_total} 条死信记录", result
def _store_discarded_delivery(record: dict, *, discarded_by: str = "", reason: str = "") -> str:
_ensure_queue_dirs()
discarded_record = dict(record or {})
discarded_record["discarded_at"] = _now_text()
discarded_record["discarded_by"] = str(discarded_by or "").strip()
discarded_record["discard_reason"] = str(reason or "").strip()
file_name = f"{int(time.time() * 1000)}-{uuid4().hex}.json"
file_path = os.path.join(_queue_discarded_dir(), file_name)
_write_json_file(file_path, discarded_record)
return file_path
def _discard_dead_letter_records(payload: dict | None = None) -> tuple[bool, str, dict]:
normalized_payload = dict(payload or {})
selector = dict(normalized_payload.get("selector") or {})
selector["state"] = str(selector.get("state") or "dead_letter").strip() or "dead_letter"
limit = _normalize_queue_limit(normalized_payload.get("limit"), default=20)
discarded_by = str(normalized_payload.get("discarded_by") or "").strip()
reason = str(normalized_payload.get("reason") or "").strip()
matched_entries = [
entry
for entry in _queue_entries("dead_letter")
if _matches_delivery_selector(entry, selector, allowed_states={"dead_letter"})
][:limit]
if not matched_entries:
return False, "未找到符合条件的死信记录", {
"selector": selector,
"limit": limit,
"queue": _delivery_queue_snapshot(),
}
discarded_total = 0
for entry in matched_entries:
_store_discarded_delivery(
dict(entry.get("record") or {}),
discarded_by=discarded_by,
reason=reason,
)
try:
os.remove(str(entry.get("file_path") or ""))
except FileNotFoundError:
pass
discarded_total += 1
result = {
"selector": selector,
"limit": limit,
"discarded_total": discarded_total,
"discarded_by": discarded_by,
"reason": reason,
"matched_records": _trim_queue_preview(matched_entries),
"queue": _delivery_queue_snapshot(),
}
return True, f"已丢弃 {discarded_total} 条死信记录", result
def _delivery_queue_snapshot() -> dict:
pending_dir = _queue_pending_dir()
dead_letter_dir = _queue_dead_letter_dir()
pending_files = _queue_file_names(pending_dir)
dead_letter_files = _queue_file_names(dead_letter_dir)
pending_head = _queue_head_record(pending_dir)
dead_letter_head = _queue_head_record(dead_letter_dir)
pending_count = len(pending_files)
dead_letter_count = len(dead_letter_files)
state = "healthy"
label = "正常"
reason = "当前没有待重试回执,也没有死信记录。"
if dead_letter_count > 0:
state = "dead_letter"
label = f"死信 {dead_letter_count}"
reason = "存在语义失败的回执/事件,自动重试已停止,建议人工查看。"
elif pending_count > 0:
state = "retrying"
label = f"待重试 {pending_count}"
reason = "存在待重试的回执/事件Node Agent 会在后续 heartbeat/poll 周期继续回放。"
elif not pending_head and not dead_letter_head:
label = "正常"
return {
"state": state,
"label": label,
"reason": reason,
"pending_count": pending_count,
"dead_letter_count": dead_letter_count,
"oldest_pending_at": str(pending_head.get("created_at") or "").strip(),
"oldest_pending_request_id": str(pending_head.get("request_id") or "").strip(),
"oldest_pending_kind": str(pending_head.get("kind") or "").strip(),
"oldest_dead_letter_at": str(
dead_letter_head.get("dead_letter_at") or dead_letter_head.get("created_at") or ""
).strip(),
"oldest_dead_letter_request_id": str(dead_letter_head.get("request_id") or "").strip(),
"oldest_dead_letter_kind": str(dead_letter_head.get("kind") or "").strip(),
"last_flush_at": str(_LAST_QUEUE_FLUSH_SUMMARY.get("last_flush_at") or "").strip(),
"last_flush_delivered": int(_LAST_QUEUE_FLUSH_SUMMARY.get("delivered") or 0),
"last_flush_deferred": int(_LAST_QUEUE_FLUSH_SUMMARY.get("deferred") or 0),
"last_flush_dead_letter": int(_LAST_QUEUE_FLUSH_SUMMARY.get("dead_letter") or 0),
}
def _response_detail_code(response: dict) -> str:
if not isinstance(response, dict):
return ""
top_level = str(response.get("detail_code") or "").strip()
if top_level:
return top_level
data = response.get("data")
if isinstance(data, dict):
return str(data.get("detail_code") or "").strip()
return ""
def _ensure_ok_response(response: dict, fallback_message: str) -> dict:
raw_code = response.get("code", 1)
try:
normalized_code = int(raw_code if raw_code not in (None, "") else 1)
except Exception:
normalized_code = 1
if normalized_code == 0:
return response
raise AgentResponseError(
str(response.get("message") or fallback_message),
detail_code=_response_detail_code(response),
payload=(response.get("data") if isinstance(response.get("data"), dict) else {}),
)
def _build_delivery_record(kind: str, path: str, payload: dict, request_id: str) -> dict:
return {
"kind": str(kind or "").strip() or "delivery",
"path": str(path or "").strip(),
"payload": dict(payload or {}),
"request_id": str(request_id or "").strip(),
"attempt_count": 0,
"created_at": _now_text(),
"updated_at": _now_text(),
"last_error": "",
"last_detail_code": "",
}
def _store_pending_delivery(record: dict) -> str:
_ensure_queue_dirs()
file_name = f"{int(time.time() * 1000)}-{uuid4().hex}.json"
file_path = os.path.join(_queue_pending_dir(), file_name)
_write_json_file(file_path, record)
return file_path
def _store_dead_letter_delivery(record: dict, *, reason: str, detail_code: str = "") -> str:
_ensure_queue_dirs()
dead_record = dict(record or {})
dead_record["dead_letter_reason"] = str(reason or "").strip()
dead_record["dead_letter_at"] = _now_text()
dead_record["last_error"] = str(reason or "").strip()
dead_record["last_detail_code"] = str(detail_code or "").strip()
file_name = f"{int(time.time() * 1000)}-{uuid4().hex}.json"
file_path = os.path.join(_queue_dead_letter_dir(), file_name)
_write_json_file(file_path, dead_record)
return file_path
def _dispatch_delivery_record(record: dict) -> dict:
kind = str(record.get("kind") or "delivery").strip() or "delivery"
path = str(record.get("path") or "").strip()
payload = record.get("payload") or {}
timeout = 15 if kind == "job_event" else 30
response = _post(path, payload, timeout=timeout)
return _ensure_ok_response(response, f"{kind} failed")
def _flush_delivery_queue(limit: int | None = None) -> dict:
global _LAST_QUEUE_FLUSH_SUMMARY
_ensure_queue_dirs()
safe_limit = max(1, int(limit or AGENT_QUEUE_FLUSH_LIMIT or 1))
summary = {
"scanned": 0,
"delivered": 0,
"deferred": 0,
"dead_letter": 0,
}
file_names = sorted(
file_name
for file_name in os.listdir(_queue_pending_dir())
if file_name.endswith(".json")
)[:safe_limit]
for file_name in file_names:
summary["scanned"] += 1
file_path = os.path.join(_queue_pending_dir(), file_name)
try:
record = _read_json_file(file_path)
except Exception as exc:
_store_dead_letter_delivery({"file_name": file_name}, reason=f"invalid queue record: {exc}")
try:
os.remove(file_path)
except FileNotFoundError:
pass
summary["dead_letter"] += 1
continue
try:
_dispatch_delivery_record(record)
try:
os.remove(file_path)
except FileNotFoundError:
pass
summary["delivered"] += 1
except AgentResponseError as exc:
record["attempt_count"] = int(record.get("attempt_count") or 0) + 1
record["updated_at"] = _now_text()
record["last_error"] = str(exc)
record["last_detail_code"] = exc.detail_code
if exc.detail_code in _PERMANENT_DELIVERY_DETAIL_CODES:
_store_dead_letter_delivery(record, reason=str(exc), detail_code=exc.detail_code)
try:
os.remove(file_path)
except FileNotFoundError:
pass
summary["dead_letter"] += 1
_log(
f"delivery moved to dead-letter: kind={record.get('kind')} request_id={record.get('request_id')} "
f"detail_code={exc.detail_code or '-'} message={exc}"
)
else:
_write_json_file(file_path, record)
summary["deferred"] += 1
except Exception as exc:
record["attempt_count"] = int(record.get("attempt_count") or 0) + 1
record["updated_at"] = _now_text()
record["last_error"] = str(exc)
_write_json_file(file_path, record)
summary["deferred"] += 1
summary["last_flush_at"] = _now_text()
_LAST_QUEUE_FLUSH_SUMMARY = dict(summary)
return summary
def _deliver_or_queue(
*,
kind: str,
path: str,
payload: dict,
request_id: str,
timeout: int = 30,
) -> dict:
record = _build_delivery_record(kind, path, payload, request_id)
try:
response = _post(path, payload, timeout=timeout)
_ensure_ok_response(response, f"{kind} failed")
return {"state": "delivered", "request_id": request_id, "detail_code": "", "response": response}
except AgentResponseError as exc:
if exc.detail_code in _PERMANENT_DELIVERY_DETAIL_CODES:
file_path = _store_dead_letter_delivery(record, reason=str(exc), detail_code=exc.detail_code)
_log(
f"{kind} dead-lettered: request_id={request_id} detail_code={exc.detail_code or '-'} "
f"message={exc} file={file_path}"
)
return {"state": "dead_letter", "request_id": request_id, "detail_code": exc.detail_code, "file_path": file_path}
file_path = _store_pending_delivery({**record, "last_error": str(exc), "last_detail_code": exc.detail_code})
_log(
f"{kind} queued for retry: request_id={request_id} detail_code={exc.detail_code or '-'} "
f"message={exc} file={file_path}"
)
return {"state": "queued", "request_id": request_id, "detail_code": exc.detail_code, "file_path": file_path}
except Exception as exc:
file_path = _store_pending_delivery({**record, "last_error": str(exc)})
_log(f"{kind} queued for retry: request_id={request_id} message={exc} file={file_path}")
return {"state": "queued", "request_id": request_id, "detail_code": "", "file_path": file_path}
def _headers() -> dict[str, str]:
return {
"Content-Type": "application/json",
"X-Domaincheck-Agent-Token": AGENT_TOKEN,
}
def _request(method: str, path: str, payload: dict | None = None, timeout: int = 30) -> dict:
if not CONTROL_PLANE_BASE_URL:
raise RuntimeError("OPS_CONTROL_PLANE_BASE_URL 未配置")
if not AGENT_TOKEN:
raise RuntimeError("OPS_AGENT_TOKEN 未配置")
url = f"{CONTROL_PLANE_BASE_URL}{path}"
data = json.dumps(payload or {}, ensure_ascii=False).encode("utf-8")
request = urllib.request.Request(url=url, data=data, headers=_headers(), method=method.upper())
with urllib.request.urlopen(request, timeout=timeout) as response:
body = response.read().decode("utf-8", errors="ignore")
return json.loads(body or "{}")
def _post(path: str, payload: dict, timeout: int = 30) -> dict:
return _request("POST", path, payload, timeout=timeout)
def _hostname() -> str:
try:
return socket.gethostname()
except Exception:
return ""
def _ip() -> str:
try:
return socket.gethostbyname(socket.gethostname())
except Exception:
return ""
def _base_payload() -> dict:
return {
"node_code": NODE_CODE,
"region": NODE_REGION,
"role": NODE_ROLE,
"title": NODE_CODE,
"hostname": _hostname(),
"ip": _ip(),
"agent_version": AGENT_VERSION,
"capabilities": AGENT_CAPABILITIES,
"labels": AGENT_LABELS,
"metadata": {
"service_names": {
"api": API_SERVICE_NAME,
"worker": WORKER_SERVICE_NAME,
"sync_agent": SYNC_AGENT_SERVICE_NAME,
"node_agent": NODE_AGENT_SERVICE_NAME,
},
"delivery_queue": _delivery_queue_snapshot(),
},
}
def _run(command: list[str], timeout: int = 60) -> tuple[int, str, str]:
completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
return completed.returncode, completed.stdout.strip(), completed.stderr.strip()
def _execute_action(
action: str,
payload: dict,
*,
job_id: int | None = None,
job_context: dict | None = None,
) -> tuple[bool, str, dict]:
normalized_action = str(action or "").strip()
if normalized_action == "delivery.queue.flush":
limit = _normalize_queue_limit((payload or {}).get("limit"), default=20)
summary = _flush_delivery_queue(limit=limit)
return True, "Delivery Queue 已执行冲刷", {
"action": normalized_action,
"limit": limit,
"flush_summary": summary,
"queue": _delivery_queue_snapshot(),
}
if normalized_action == "delivery.queue.replay":
return _replay_dead_letter_records(payload)
if normalized_action == "delivery.queue.discard":
return _discard_dead_letter_records(payload)
if supports_structured_action(normalized_action):
return execute_structured_action(
normalized_action,
payload,
service_names=build_service_name_map(
api_service_name=API_SERVICE_NAME,
worker_service_name=WORKER_SERVICE_NAME,
sync_agent_service_name=SYNC_AGENT_SERVICE_NAME,
node_agent_service_name=NODE_AGENT_SERVICE_NAME,
),
runner=_run,
host_context={
"hostname": _hostname(),
"ip": _ip(),
},
)
if normalized_action == "deploy.release":
return execute_release_action(
dict(payload or {}),
run_command=_run,
default_api_service_name=API_SERVICE_NAME,
event_callback=(
(lambda event_type, message, level="info", payload=None: _job_event(
int(job_id),
event_type=event_type,
message=message,
level=level,
payload={
**dict(payload or {}),
"release_context": dict((job_context or {}).get("release_context") or {}),
"step_key": str((job_context or {}).get("step_key") or "").strip(),
},
summary_text=message,
focus_ref=dict((job_context or {}).get("focus_ref") or {}),
occurred_at=_now_text(),
))
if job_id
else None
),
urlopen_func=urllib.request.urlopen,
user_agent=f"domaincheck-node-agent/{AGENT_VERSION}",
)
return False, f"unsupported action: {normalized_action}", {"action": normalized_action}
def _register() -> None:
response = _post("/api/v1/ops/agent/register", _base_payload())
_ensure_ok_response(response, "agent register failed")
_log(f"registered: {response.get('message')}")
def _heartbeat() -> None:
response = _post("/api/v1/ops/agent/heartbeat", _base_payload())
_ensure_ok_response(response, "agent heartbeat failed")
def _pull_jobs() -> list[dict]:
response = _post(f"/api/v1/ops/agent/pull?limit=1", {"node_code": NODE_CODE})
_ensure_ok_response(response, "agent pull failed")
data = response.get("data") or {}
return list(data.get("jobs") or [])
def _normalize_agent_job(job: dict) -> dict:
normalized_job = dict(job or {})
focus_ref = normalized_job.get("focus_ref") if isinstance(normalized_job.get("focus_ref"), dict) else {}
release_context = (
normalized_job.get("release_context") if isinstance(normalized_job.get("release_context"), dict) else {}
)
step_ref = normalized_job.get("step_ref") if isinstance(normalized_job.get("step_ref"), dict) else {}
job_id = int(normalized_job.get("job_id") or normalized_job.get("id") or 0)
step_key = str(
normalized_job.get("step_key")
or step_ref.get("step_key")
or "dispatch"
).strip() or "dispatch"
step_title = str(
normalized_job.get("step_title")
or step_ref.get("step_title")
or normalized_job.get("summary")
or normalized_job.get("action")
or step_key
).strip() or step_key
if not focus_ref:
focus_ref = {
"kind": "ops_job",
"job_id": job_id,
"job_code": str(normalized_job.get("job_code") or "").strip(),
"action": str(normalized_job.get("action") or "").strip(),
"target_node_code": str(normalized_job.get("target_node_code") or "").strip(),
}
return {
**normalized_job,
"job_id": job_id,
"job_code": str(normalized_job.get("job_code") or "").strip(),
"job_type": str(normalized_job.get("job_type") or "ops_action").strip() or "ops_action",
"action": str(normalized_job.get("action") or "").strip(),
"payload": dict(normalized_job.get("payload") or {}),
"policy": dict(normalized_job.get("policy") or {}),
"focus_ref": focus_ref,
"release_context": dict(release_context or {}),
"step_key": step_key,
"step_title": step_title,
"step_ref": {
"step_id": int(step_ref.get("step_id") or 0),
"step_key": step_key,
"step_title": step_title,
},
}
def _job_start(job_id: int, *, job: dict | None = None) -> None:
normalized_job = _normalize_agent_job(job or {"job_id": job_id})
response = _post(
f"/api/v1/ops/agent/jobs/{job_id}/start",
{
"node_code": NODE_CODE,
"job_code": normalized_job.get("job_code") or "",
"action": normalized_job.get("action") or "",
"step_key": normalized_job.get("step_key") or "",
"focus_ref": normalized_job.get("focus_ref") or {},
},
)
_ensure_ok_response(response, "job start failed")
def _job_complete(
job_id: int,
*,
status: str,
stdout: str,
stderr: str,
result: dict,
error_message: str = "",
client_request_id: str | None = None,
duration_ms: int | None = None,
summary_text: str = "",
focus_ref: dict | None = None,
step_ref: dict | None = None,
release_context: dict | None = None,
) -> dict:
request_id = str(client_request_id or "").strip() or _new_request_id("complete", job_id)
payload = {
"node_code": NODE_CODE,
"status": status,
"stdout": stdout,
"stderr": stderr,
"result": result,
"error_message": error_message,
"client_request_id": request_id,
}
if duration_ms is not None:
payload["duration_ms"] = max(0, int(duration_ms or 0))
if str(summary_text or "").strip():
payload["summary_text"] = str(summary_text).strip()
if isinstance(focus_ref, dict) and focus_ref:
payload["focus_ref"] = dict(focus_ref)
if isinstance(step_ref, dict) and step_ref:
payload["step_ref"] = dict(step_ref)
if isinstance(release_context, dict) and release_context:
payload["release_context"] = dict(release_context)
return _deliver_or_queue(
kind="job_complete",
path=f"/api/v1/ops/agent/jobs/{job_id}/complete",
payload=payload,
request_id=request_id,
timeout=30,
)
def _job_event(
job_id: int,
*,
event_type: str,
message: str,
level: str = "info",
payload: dict | None = None,
client_event_id: str | None = None,
summary_text: str = "",
focus_ref: dict | None = None,
occurred_at: str = "",
) -> dict:
request_id = str(client_event_id or "").strip() or _new_request_id("event", job_id)
delivery_payload = {
"node_code": NODE_CODE,
"event_type": event_type,
"message": message,
"level": level,
"payload": payload or {},
"client_event_id": request_id,
}
if str(summary_text or "").strip():
delivery_payload["summary_text"] = str(summary_text).strip()
if isinstance(focus_ref, dict) and focus_ref:
delivery_payload["focus_ref"] = dict(focus_ref)
if str(occurred_at or "").strip():
delivery_payload["occurred_at"] = str(occurred_at).strip()
return _deliver_or_queue(
kind="job_event",
path=f"/api/v1/ops/agent/jobs/{job_id}/events",
payload=delivery_payload,
request_id=request_id,
timeout=15,
)
def _process_job(job: dict) -> None:
normalized_job = _normalize_agent_job(job)
job_id = int(normalized_job.get("job_id") or 0)
action = str(normalized_job.get("action") or "").strip()
payload = dict(normalized_job.get("payload") or {})
if job_id <= 0 or not action:
return
started_at = time.monotonic()
_log(
"job start: "
f"id={job_id} code={normalized_job.get('job_code') or '-'} "
f"type={normalized_job.get('job_type') or '-'} "
f"step={normalized_job.get('step_key') or '-'} action={action}"
)
start_delivery_state = "delivered"
start_delivery_error = ""
try:
_job_start(job_id, job=normalized_job)
except Exception as exc:
start_delivery_state = "failed_local"
start_delivery_error = str(exc)
_log(
"job start delivery failed, continue locally: "
f"id={job_id} code={normalized_job.get('job_code') or '-'} action={action} error={exc}"
)
_job_event(
job_id,
event_type="executor_received",
message=f"node agent accepted action {action}",
summary_text=f"已接单 {action}",
focus_ref=dict(normalized_job.get("focus_ref") or {}),
occurred_at=_now_text(),
payload={
"action": action,
"job_code": normalized_job.get("job_code") or "",
"job_type": normalized_job.get("job_type") or "",
"step_key": normalized_job.get("step_key") or "",
"step_title": normalized_job.get("step_title") or "",
"release_context": dict(normalized_job.get("release_context") or {}),
"start_delivery_state": start_delivery_state,
"start_delivery_error": start_delivery_error,
},
)
ok, message, result = _execute_action(action, payload, job_id=job_id, job_context=normalized_job)
stdout = str(result.get("stdout") or "")
stderr = str(result.get("stderr") or "")
duration_ms = max(0, int((time.monotonic() - started_at) * 1000))
summary_text = str(result.get("summary_text") or result.get("summary") or message).strip()
delivery = _job_complete(
job_id,
status="success" if ok else "failed",
stdout=stdout,
stderr=stderr,
result=result,
error_message="" if ok else message,
duration_ms=duration_ms,
summary_text=summary_text,
focus_ref=dict(normalized_job.get("focus_ref") or {}),
step_ref=dict(normalized_job.get("step_ref") or {}),
release_context=dict(normalized_job.get("release_context") or {}),
)
_log(
"job complete: "
f"id={job_id} code={normalized_job.get('job_code') or '-'} "
f"action={action} ok={ok} duration_ms={duration_ms} delivery={delivery.get('state')}"
)
def main() -> None:
if not NODE_CODE:
raise RuntimeError("NODE_CODE 未配置")
_log(f"starting node agent: node={NODE_CODE} role={NODE_ROLE} region={NODE_REGION}")
_ensure_queue_dirs()
_register()
last_heartbeat_at = 0.0
while True:
now = time.time()
try:
delivery_summary = _flush_delivery_queue(limit=AGENT_QUEUE_FLUSH_LIMIT)
if delivery_summary["delivered"] or delivery_summary["dead_letter"]:
_log(f"delivery queue flush: {delivery_summary}")
if now - last_heartbeat_at >= 15:
_heartbeat()
last_heartbeat_at = now
jobs = _pull_jobs()
if jobs:
for job in jobs:
_process_job(job)
else:
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
except urllib.error.HTTPError as exc:
_log(f"http error: {exc.code}")
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
except urllib.error.URLError as exc:
_log(f"url error: {exc}")
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
except Exception as exc:
_log(f"loop error: {exc}")
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
if __name__ == "__main__":
main()

View File

@@ -1,8 +1,10 @@
from typing import Any, Optional
from pydantic import BaseModel
from typing import Any
class ApiResponse(BaseModel):
code: int = 0
message: str = "ok"
data: Any = None
detail_code: Optional[str] = None

View File

@@ -0,0 +1,241 @@
from __future__ import annotations
import json
import subprocess
from pathlib import Path
from typing import Iterable
from app.core.config import settings
DOMAIN_API_ROOT = Path(__file__).resolve().parents[2]
WORKSPACE_ROOT = DOMAIN_API_ROOT.parent
_REGISTERED_ROUTE_PATHS: set[str] = set()
def remember_registered_route_paths(route_paths: Iterable[str] | None) -> None:
global _REGISTERED_ROUTE_PATHS
normalized = {
str(item).strip()
for item in list(route_paths or [])
if str(item).strip()
}
_REGISTERED_ROUTE_PATHS = normalized
def _read_json(path: Path) -> dict:
try:
return json.loads(path.read_text(encoding="utf-8"))
except Exception:
return {}
def _discover_manifest_candidate() -> tuple[Path | None, str]:
configured = str(settings.build_manifest_path or "").strip()
if configured:
path = Path(configured)
if path.exists():
return path, "configured_manifest"
root_manifest = WORKSPACE_ROOT / "release_manifest.json"
if root_manifest.exists():
return root_manifest, "release_manifest"
latest_release = WORKSPACE_ROOT / "release" / "latest_release.json"
if latest_release.exists():
return latest_release, "latest_release"
return None, ""
def _build_info_from_env() -> dict | None:
if not any(
[
str(settings.build_commit_sha or "").strip(),
str(settings.build_commit_ref or "").strip(),
str(settings.build_generated_at or "").strip(),
str(settings.build_package_name or "").strip(),
str(settings.build_checksum or "").strip(),
]
):
return None
return {
"source": str(settings.build_source_label or "").strip() or "env",
"package_name": str(settings.build_package_name or "").strip(),
"generated_at": str(settings.build_generated_at or "").strip(),
"commit_sha": str(settings.build_commit_sha or "").strip(),
"commit_ref": str(settings.build_commit_ref or "").strip(),
"checksum": str(settings.build_checksum or "").strip(),
"manifest_path": str(settings.build_manifest_path or "").strip(),
}
def _build_info_from_manifest() -> dict | None:
manifest_path, source = _discover_manifest_candidate()
if manifest_path is None:
return None
payload = _read_json(manifest_path)
if not payload:
return {
"source": source,
"package_name": "",
"generated_at": "",
"commit_sha": "",
"commit_ref": "",
"checksum": "",
"manifest_path": str(manifest_path),
}
checksum = str(payload.get("checksum") or payload.get("sha256") or "").strip()
return {
"source": source,
"package_name": str(payload.get("package_name") or "").strip(),
"generated_at": str(payload.get("generated_at") or "").strip(),
"commit_sha": str(payload.get("commit_sha") or "").strip(),
"commit_ref": str(payload.get("commit_ref") or "").strip(),
"checksum": checksum,
"manifest_path": str(manifest_path),
}
def _build_info_from_git() -> dict | None:
try:
inside_worktree = subprocess.run(
["git", "-C", str(WORKSPACE_ROOT), "rev-parse", "--is-inside-work-tree"],
check=False,
capture_output=True,
text=True,
timeout=2,
)
except Exception:
return None
if inside_worktree.returncode != 0 or str(inside_worktree.stdout or "").strip() != "true":
return None
def _read_git(*args: str) -> str:
try:
result = subprocess.run(
["git", "-C", str(WORKSPACE_ROOT), *args],
check=False,
capture_output=True,
text=True,
timeout=2,
)
except Exception:
return ""
if result.returncode != 0:
return ""
return str(result.stdout or "").strip()
return {
"source": "git",
"package_name": "",
"generated_at": "",
"commit_sha": _read_git("rev-parse", "--short=12", "HEAD"),
"commit_ref": _read_git("rev-parse", "--abbrev-ref", "HEAD"),
"checksum": "",
"manifest_path": "",
}
def _resolve_build_identity() -> dict:
return (
_build_info_from_env()
or _build_info_from_manifest()
or _build_info_from_git()
or {
"source": "unknown",
"package_name": "",
"generated_at": "",
"commit_sha": "",
"commit_ref": "",
"checksum": "",
"manifest_path": "",
}
)
def _expected_route_paths() -> dict[str, str]:
prefix = str(settings.api_prefix or "/api/v1").rstrip("/")
return {
"ops_contracts": f"{prefix}/ops/contracts",
"ops_contract_detail": f"{prefix}/ops/contracts/{{contract_key}}",
"ops_stack_diagnosis": f"{prefix}/ops/stack-diagnosis",
"ops_node_handover": f"{prefix}/ops/nodes/{{node_code}}/handover",
"ops_node_onboarding": f"{prefix}/ops/nodes/{{node_code}}/onboarding",
"ops_node_onboarding_bootstrap_preview": f"{prefix}/ops/nodes/{{node_code}}/onboarding/bootstrap/preview",
"ops_node_onboarding_bootstrap_execute": f"{prefix}/ops/nodes/{{node_code}}/onboarding/bootstrap/execute",
"ops_node_onboarding_acceptance_preview": f"{prefix}/ops/nodes/{{node_code}}/onboarding/acceptance/preview",
"ops_node_onboarding_acceptance_execute": f"{prefix}/ops/nodes/{{node_code}}/onboarding/acceptance/execute",
"ops_node_onboarding_recovery_preview": f"{prefix}/ops/nodes/{{node_code}}/onboarding/recovery/preview",
"ops_node_onboarding_recovery_execute": f"{prefix}/ops/nodes/{{node_code}}/onboarding/recovery/execute",
"ops_node_scene_log": f"{prefix}/ops/nodes/{{node_code}}/scene-log",
"ops_node_handover_bootstrap_plan": f"{prefix}/ops/nodes/{{node_code}}/handover/bootstrap-plan",
"ops_driver_feed": f"{prefix}/ops/driver-feed",
"ops_codex_brief": f"{prefix}/ops/codex-brief",
"ops_activity_stream": f"{prefix}/ops/activity-stream",
"ops_runbook_resolve": f"{prefix}/ops/runbook/sequences/{{sequence_key}}/resolve",
"ops_runbook_execute": f"{prefix}/ops/runbook/sequences/{{sequence_key}}/execute",
"runtime_build_info": f"{prefix}/runtime/build-info",
}
def _read_text(path: Path) -> str:
try:
return path.read_text(encoding="utf-8", errors="replace")
except Exception:
return ""
def _repository_capabilities() -> dict:
ops_agent_service_text = _read_text(DOMAIN_API_ROOT / "app" / "services" / "ops_agent_service.py")
bootstrap_node_agent_text = _read_text(DOMAIN_API_ROOT / "deploy" / "multi-region" / "bootstrap_node_agent.sh")
return {
"supports_install_command_block": (
"install_command_block" in ops_agent_service_text
and "_candidate_node_agent_install_paths" in ops_agent_service_text
),
"supports_multi_layout_bootstrap": "resolve_project_root" in bootstrap_node_agent_text,
}
def _build_route_surface(route_paths: Iterable[str] | None = None) -> dict:
expected = _expected_route_paths()
normalized_paths = {
str(item).strip()
for item in (list(route_paths) if route_paths is not None else list(_REGISTERED_ROUTE_PATHS))
if str(item).strip()
}
mode = "registered" if normalized_paths else "declared_contract"
flags = {key: (path in normalized_paths if normalized_paths else True) for key, path in expected.items()}
missing_keys = [key for key, available in flags.items() if not available]
return {
"mode": mode,
"registered_paths_total": len(normalized_paths),
"surface_flags": flags,
"expected_paths": expected,
"missing_keys": missing_keys,
"missing_paths": [expected[key] for key in missing_keys],
"surface_complete": len(missing_keys) == 0,
}
def get_runtime_build_info(route_paths: Iterable[str] | None = None) -> dict:
build = _resolve_build_identity()
route_surface = _build_route_surface(route_paths=route_paths)
return {
"source": str(build.get("source") or "").strip() or "unknown",
"package_name": str(build.get("package_name") or "").strip(),
"generated_at": str(build.get("generated_at") or "").strip(),
"commit_sha": str(build.get("commit_sha") or "").strip(),
"commit_ref": str(build.get("commit_ref") or "").strip(),
"checksum": str(build.get("checksum") or "").strip(),
"manifest_path": str(build.get("manifest_path") or "").strip(),
"workspace_root": str(WORKSPACE_ROOT),
"domain_api_root": str(DOMAIN_API_ROOT),
"repository_capabilities": _repository_capabilities(),
"route_surface": route_surface,
}

View File

@@ -147,15 +147,50 @@ def _build_remote_log_lines(
mode: str,
limit: int = 240,
) -> list[str]:
return _build_remote_log_snapshot(active_job, enabled=enabled, mode=mode, limit=limit)["lines"]
def _build_remote_log_snapshot(
active_job: dict | None,
*,
enabled: bool,
mode: str,
limit: int = 240,
) -> dict:
if not enabled:
return []
return {
"lines": [],
"line_count": 0,
"last_at": "",
"last_line": "",
"source_nodes": [],
"source_node_count": 0,
}
if not active_job:
return []
return {
"lines": [],
"line_count": 0,
"last_at": "",
"last_line": "",
"source_nodes": [],
"source_node_count": 0,
}
events = list(active_job.get("current_cycle_events") or active_job.get("recent_events") or [])
if not events:
return []
return {
"lines": [],
"line_count": 0,
"last_at": "",
"last_line": "",
"source_nodes": [],
"source_node_count": 0,
}
lines: list[str] = []
source_nodes: set[str] = set()
source_node_summaries: dict[str, dict] = {}
last_at = ""
last_line = ""
normalized_mode = str(mode or "key").strip().lower()
if normalized_mode not in {"key", "full"}:
normalized_mode = "key"
@@ -180,8 +215,47 @@ def _build_remote_log_lines(
continue
if len(message) > _REMOTE_LOG_MAX_CHARS:
message = f"{message[:_REMOTE_LOG_MAX_CHARS]}..."
lines.append(f"[{created_at}] [{node_code}] {message}")
return lines[-max(1, int(limit or 240)) :]
formatted_line = f"[{created_at}] [{node_code}] {message}"
lines.append(formatted_line)
source_nodes.add(node_code)
node_summary = source_node_summaries.setdefault(
node_code,
{
"node_code": node_code,
"line_count": 0,
"key_line_count": 0,
"full_line_count": 0,
"last_at": "",
"last_line": "",
},
)
node_summary["line_count"] += 1
if event_mode == "full":
node_summary["full_line_count"] += 1
else:
node_summary["key_line_count"] += 1
node_summary["last_at"] = created_at
node_summary["last_line"] = formatted_line
last_at = created_at
last_line = formatted_line
sliced_lines = lines[-max(1, int(limit or 240)) :]
sorted_source_node_summaries = sorted(
source_node_summaries.values(),
key=lambda item: (
str(item.get("last_at") or ""),
str(item.get("node_code") or ""),
),
reverse=True,
)
return {
"lines": sliced_lines,
"line_count": len(sliced_lines),
"last_at": last_at,
"last_line": last_line,
"source_nodes": sorted(source_nodes),
"source_node_count": len(source_nodes),
"source_node_summaries": sorted_source_node_summaries,
}
def _resolve_remote_log_lines(
@@ -195,6 +269,17 @@ def _resolve_remote_log_lines(
return _build_remote_log_lines(active_job, enabled=enabled, mode=mode, limit=limit)
def _resolve_remote_log_snapshot(
active_job: dict | None,
runs: list[dict],
*,
enabled: bool,
mode: str,
limit: int = 240,
) -> dict:
return _build_remote_log_snapshot(active_job, enabled=enabled, mode=mode, limit=limit)
def _load_runtime_state() -> dict:
try:
redis_client = get_redis()
@@ -404,13 +489,14 @@ def get_detect_status() -> dict:
runs = sync_detect_runs(runtime_snapshot, progress, settings_summary, active_job=active_job)
worker_log_sync_enabled = bool(runtime_settings.get("worker_log_sync_enabled", False))
worker_log_sync_mode = str(runtime_settings.get("worker_log_sync_mode", "key") or "key")
remote_log_lines = _resolve_remote_log_lines(
remote_log_snapshot = _resolve_remote_log_snapshot(
active_job,
runs,
enabled=worker_log_sync_enabled,
mode=worker_log_sync_mode,
limit=240,
)
remote_log_lines = list(remote_log_snapshot.get("lines") or [])
dependency_alerts = _extract_dependency_alerts(recent_lines)
append_detect_result_projection_if_changed(
detect={
@@ -465,6 +551,12 @@ def get_detect_status() -> dict:
"recent_warning": recent_proxy_warning,
"log_lines": recent_lines,
"remote_log_lines": remote_log_lines,
"remote_log_line_count": int(remote_log_snapshot.get("line_count", 0) or 0),
"remote_log_last_at": str(remote_log_snapshot.get("last_at") or ""),
"remote_log_last_line": str(remote_log_snapshot.get("last_line") or ""),
"remote_log_nodes": list(remote_log_snapshot.get("source_nodes") or []),
"remote_log_node_count": int(remote_log_snapshot.get("source_node_count", 0) or 0),
"remote_log_node_summaries": list(remote_log_snapshot.get("source_node_summaries") or []),
"runs": runs,
"active_job": active_job,
"worker_log_sync_enabled": worker_log_sync_enabled,

View File

@@ -0,0 +1,278 @@
from __future__ import annotations
from pathlib import Path
from typing import Protocol
from app.core.config import settings
STRUCTURED_ACTIONS = {
"health.snapshot",
"service.status",
"service.restart",
"service.start",
"service.stop",
"logs.collect",
"diagnostics.collect",
"runtime.start_worker",
"runtime.stop_worker",
"runtime.restart_api",
"runtime.start_sync_agent",
"runtime.stop_sync_agent",
}
class CommandRunner(Protocol):
def __call__(self, command: list[str], *, timeout: int = 60) -> tuple[int, str, str]:
...
def supports_structured_action(action: str) -> bool:
return str(action or "").strip() in STRUCTURED_ACTIONS
def trim_output(text: str, limit: int) -> str:
normalized = str(text or "").strip()
if len(normalized) <= int(limit):
return normalized
return normalized[-int(limit):]
def _read_tail_lines(path: Path, *, max_lines: int) -> list[str]:
if not path.exists() or not path.is_file():
return []
with path.open("r", encoding="utf-8", errors="replace") as handle:
return handle.read().splitlines()[-max_lines:]
def _runtime_logs_dir() -> Path:
return Path(__file__).resolve().parents[2] / "runtime" / "logs"
def _service_log_file_candidates(service_name: str, *, service_names: dict[str, str]) -> list[Path]:
normalized_service_name = str(service_name or "").strip()
if not normalized_service_name:
return []
domain_root = Path(settings.domain_root)
runtime_logs_dir = _runtime_logs_dir()
candidates: list[Path] = []
def append_candidate(path: Path) -> None:
if path not in candidates:
candidates.append(path)
if normalized_service_name == (service_names.get("worker") or "domaincheck-worker"):
append_candidate(domain_root / "detect_worker.log")
append_candidate(domain_root / "logs" / "detect_worker.log")
elif normalized_service_name == (service_names.get("api") or "domaincheck-api"):
append_candidate(runtime_logs_dir / "domain-api.stderr.log")
append_candidate(runtime_logs_dir / "domain-api.stdout.log")
append_candidate(domain_root / "logs" / "app.log")
return candidates
def _collect_log_file_fallback(
service_name: str,
*,
lines: int,
service_names: dict[str, str],
) -> tuple[str, list[str]]:
aggregated_lines: list[str] = []
used_paths: list[str] = []
for path in _service_log_file_candidates(service_name, service_names=service_names):
current_lines = _read_tail_lines(path, max_lines=lines)
if not current_lines:
continue
aggregated_lines.extend(current_lines)
used_paths.append(str(path))
if not aggregated_lines:
return "", []
return "\n".join(aggregated_lines[-lines:]), used_paths
def build_service_name_map(
*,
api_service_name: str,
worker_service_name: str,
sync_agent_service_name: str,
node_agent_service_name: str = "domaincheck-node-agent",
) -> dict[str, str]:
return {
"api": str(api_service_name or "").strip() or "domaincheck-api",
"worker": str(worker_service_name or "").strip() or "domaincheck-worker",
"sync_agent": str(sync_agent_service_name or "").strip() or "domaincheck-sync-agent",
"node_agent": str(node_agent_service_name or "").strip() or "domaincheck-node-agent",
}
def service_name_from_payload(payload: dict, *, default_worker_service_name: str) -> str:
service_name = str((payload or {}).get("service_name") or "").strip()
return service_name or str(default_worker_service_name or "").strip() or "domaincheck-worker"
def service_name_for_action(action: str, payload: dict, service_names: dict[str, str]) -> str:
normalized_action = str(action or "").strip()
if normalized_action in {"service.status", "service.restart", "service.start", "service.stop"}:
return service_name_from_payload(payload, default_worker_service_name=service_names.get("worker") or "")
runtime_action_map = {
"runtime.start_worker": service_names.get("worker") or "domaincheck-worker",
"runtime.stop_worker": service_names.get("worker") or "domaincheck-worker",
"runtime.restart_api": service_names.get("api") or "domaincheck-api",
"runtime.start_sync_agent": service_names.get("sync_agent") or "domaincheck-sync-agent",
"runtime.stop_sync_agent": service_names.get("sync_agent") or "domaincheck-sync-agent",
}
return str(runtime_action_map.get(normalized_action) or "").strip()
def systemctl_action_name(action: str) -> str:
normalized_action = str(action or "").strip()
if normalized_action in {"service.restart", "runtime.restart_api"}:
return "restart"
if normalized_action in {"service.start", "runtime.start_worker", "runtime.start_sync_agent"}:
return "start"
if normalized_action in {"service.stop", "runtime.stop_worker", "runtime.stop_sync_agent"}:
return "stop"
return ""
def execute_structured_action(
action: str,
payload: dict | None,
*,
service_names: dict[str, str],
runner: CommandRunner,
host_context: dict | None = None,
) -> tuple[bool, str, dict]:
normalized_action = str(action or "").strip()
normalized_payload = dict(payload or {})
normalized_service_names = {
str(key or "").strip(): str(value or "").strip()
for key, value in dict(service_names or {}).items()
if str(key or "").strip() and str(value or "").strip()
}
host_details = {
key: str((host_context or {}).get(key) or "").strip()
for key in ("hostname", "ip")
if str((host_context or {}).get(key) or "").strip()
}
if normalized_action == "health.snapshot":
checks: dict[str, dict] = {}
for key, service_name in normalized_service_names.items():
code, stdout, stderr = runner(["systemctl", "is-active", service_name], timeout=15)
checks[key] = {
"service_name": service_name,
"returncode": int(code or 0),
"state": stdout or stderr,
"ok": int(code or 0) == 0 and (stdout or stderr) == "active",
}
result = {"checks": checks}
result.update(host_details)
return True, "health snapshot collected", result
if normalized_action == "service.status":
service_name = service_name_from_payload(
normalized_payload,
default_worker_service_name=normalized_service_names.get("worker") or "",
)
code, stdout, stderr = runner(["systemctl", "status", service_name, "--no-pager", "-l"], timeout=45)
result = {
"service_name": service_name,
"stdout": trim_output(stdout, 12000),
"stderr": trim_output(stderr, 4000),
}
result.update(host_details)
return int(code or 0) == 0, stdout or stderr or f"{service_name} status collected", result
systemctl_action = systemctl_action_name(normalized_action)
if systemctl_action:
service_name = service_name_for_action(normalized_action, normalized_payload, normalized_service_names)
if not service_name:
return False, f"当前动作缺少 service_name: {normalized_action}", {"action": normalized_action}
code, stdout, stderr = runner(["systemctl", systemctl_action, service_name], timeout=45)
result = {
"service_name": service_name,
"systemctl_action": systemctl_action,
"stdout": trim_output(stdout, 12000),
"stderr": trim_output(stderr, 4000),
}
result.update(host_details)
return int(code or 0) == 0, stdout or stderr or f"{service_name} {systemctl_action} completed", result
if normalized_action == "logs.collect":
service_name = service_name_from_payload(
normalized_payload,
default_worker_service_name=normalized_service_names.get("worker") or "",
)
lines = max(20, min(int(normalized_payload.get("lines") or 120), 500))
code, stdout, stderr = runner(["journalctl", "-u", service_name, "-n", str(lines), "--no-pager"], timeout=45)
journal_output = stdout or stderr
fallback_output, fallback_paths = _collect_log_file_fallback(
service_name,
lines=lines,
service_names=normalized_service_names,
)
collection_source = "journal"
effective_output = journal_output
ok = int(code or 0) == 0
if (not ok or not stdout.strip()) and fallback_output:
collection_source = "file_fallback"
effective_output = fallback_output
ok = True
result = {
"service_name": service_name,
"lines": lines,
"collection_source": collection_source,
"fallback_used": bool(collection_source == "file_fallback"),
"fallback_reason": trim_output(journal_output, 4000) if collection_source == "file_fallback" else "",
"log_paths": fallback_paths,
"journal_returncode": int(code or 0),
"stdout": trim_output(effective_output, 20000),
"stderr": trim_output(stderr, 4000),
}
result.update(host_details)
return ok, effective_output or f"{service_name} logs collected", result
if normalized_action == "diagnostics.collect":
lines = max(20, min(int(normalized_payload.get("lines") or 200), 800))
diagnostics: dict[str, object] = {
"services": {},
"lines": lines,
}
diagnostics.update(host_details)
for key, service_name in normalized_service_names.items():
active_code, active_stdout, active_stderr = runner(["systemctl", "is-active", service_name], timeout=15)
status_code, status_stdout, status_stderr = runner(
["systemctl", "status", service_name, "--no-pager", "-l"],
timeout=45,
)
log_code, log_stdout, log_stderr = runner(
["journalctl", "-u", service_name, "-n", str(lines), "--no-pager"],
timeout=45,
)
log_output = log_stdout or log_stderr
fallback_output, fallback_paths = _collect_log_file_fallback(
service_name,
lines=lines,
service_names=normalized_service_names,
)
log_source = "journal"
if (int(log_code or 0) != 0 or not log_stdout.strip()) and fallback_output:
log_output = fallback_output
log_source = "file_fallback"
diagnostics["services"][key] = {
"service_name": service_name,
"active_returncode": int(active_code or 0),
"active_state": active_stdout or active_stderr,
"status_returncode": int(status_code or 0),
"status_output": trim_output(status_stdout or status_stderr, 12000),
"log_returncode": int(log_code or 0),
"log_source": log_source,
"log_paths": fallback_paths,
"log_output": trim_output(log_output, 20000),
}
return True, "diagnostics collected", diagnostics
return False, f"unsupported action: {normalized_action}", {"action": normalized_action}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,17 @@
from __future__ import annotations
OPS_MULTI_REGION_DIR = "domain-api/deploy/multi-region"
def ops_script_path(script_name: str) -> str:
normalized_script_name = str(script_name or "").strip().lstrip("/")
if not normalized_script_name:
return OPS_MULTI_REGION_DIR
return f"{OPS_MULTI_REGION_DIR}/{normalized_script_name}"
def build_bash_command(script_name: str, *args: object) -> str:
command_parts = ["bash", ops_script_path(script_name)]
command_parts.extend(str(arg or "").strip() for arg in args if str(arg or "").strip())
return " ".join(command_parts)

View File

@@ -0,0 +1,79 @@
from __future__ import annotations
from app.services.ops_action_executor_core import STRUCTURED_ACTIONS
_LOCAL_RUNTIME_ONLY_ACTIONS = {
"runtime.push_sync",
"runtime.pull_tasks",
}
_REMOTE_AGENT_ONLY_ACTIONS = {
"delivery.queue.flush",
"delivery.queue.replay",
"delivery.queue.discard",
}
_LOCAL_RUNTIME_ACTIONS = set(STRUCTURED_ACTIONS) | _LOCAL_RUNTIME_ONLY_ACTIONS | {"deploy.release"}
_SSH_ACTIONS = set(STRUCTURED_ACTIONS) | {"deploy.release"}
_REMOTE_AGENT_ACTIONS = set(STRUCTURED_ACTIONS) | _REMOTE_AGENT_ONLY_ACTIONS | {"deploy.release"}
_CONTROL_PLANE_ACTIONS = {
"node.bootstrap",
}
def supports_local_runtime_action(action: str) -> bool:
return str(action or "").strip() in _LOCAL_RUNTIME_ACTIONS
def supports_ssh_action(action: str) -> bool:
return str(action or "").strip() in _SSH_ACTIONS
def supports_remote_agent_action(action: str) -> bool:
return str(action or "").strip() in _REMOTE_AGENT_ACTIONS
def supports_control_plane_action(action: str) -> bool:
return str(action or "").strip() in _CONTROL_PLANE_ACTIONS
def validate_action_execution_mode(
action: str,
execution_mode: str,
*,
target_node_code: str = "",
current_node_code: str = "",
) -> tuple[bool, str]:
normalized_action = str(action or "").strip()
normalized_mode = str(execution_mode or "remote-agent").strip() or "remote-agent"
normalized_target_node_code = str(target_node_code or "").strip()
normalized_current_node_code = str(current_node_code or "").strip()
if normalized_mode == "local-runtime":
if not supports_local_runtime_action(normalized_action):
return False, f"{normalized_action} 当前不支持 local-runtime 执行方式。"
if (
normalized_target_node_code
and normalized_current_node_code
and normalized_target_node_code != normalized_current_node_code
):
return False, "local-runtime 仅支持当前控制面本机节点。"
return True, ""
if normalized_mode == "control-plane":
if not supports_control_plane_action(normalized_action):
return False, f"{normalized_action} 当前不支持 control-plane 执行方式。"
return True, ""
if normalized_mode == "ssh":
if not supports_ssh_action(normalized_action):
return False, f"{normalized_action} 当前不支持 ssh 执行方式。"
return True, ""
if normalized_mode == "remote-agent":
if not supports_remote_agent_action(normalized_action):
return False, f"{normalized_action} 当前不支持 remote-agent 执行方式。"
return True, ""
return False, f"未知执行方式: {normalized_mode}"

View File

@@ -0,0 +1,60 @@
from __future__ import annotations
def normalize_execution_modes(raw_modes: object, fallback_mode: str = "remote-agent") -> list[str]:
normalized_modes: list[str] = []
seen: set[str] = set()
fallback = str(fallback_mode or "remote-agent").strip() or "remote-agent"
for item in list(raw_modes or []):
mode = str(item or "").strip()
if not mode or mode in seen:
continue
seen.add(mode)
normalized_modes.append(mode)
if fallback and fallback not in seen:
normalized_modes.insert(0, fallback)
return normalized_modes or [fallback]
def execution_mode_label(execution_mode: str) -> str:
normalized_mode = str(execution_mode or "remote-agent").strip() or "remote-agent"
if normalized_mode == "ssh":
return "SSH"
if normalized_mode == "control-plane":
return "控制面"
if normalized_mode == "local-runtime":
return "本机运行时"
if normalized_mode == "remote-agent":
return "远端 Agent"
return normalized_mode
def build_execution_mode_options(raw_modes: object, fallback_mode: str = "remote-agent") -> list[dict]:
return [
{
"value": mode,
"label": execution_mode_label(mode),
}
for mode in normalize_execution_modes(raw_modes, fallback_mode)
]
def decorate_execution_mode_fields(
payload: dict | None = None,
*,
default_key: str = "default_execution_mode",
modes_key: str = "execution_modes",
) -> dict:
normalized_payload = dict(payload or {})
default_mode = str(normalized_payload.get(default_key) or "remote-agent").strip() or "remote-agent"
execution_mode_options = build_execution_mode_options(normalized_payload.get(modes_key) or [], default_mode)
normalized_payload[default_key] = default_mode
normalized_payload[modes_key] = [str(item.get("value") or "").strip() for item in execution_mode_options if str(item.get("value") or "").strip()]
normalized_payload["default_execution_mode_label"] = execution_mode_label(default_mode)
normalized_payload["execution_mode_options"] = execution_mode_options
normalized_payload["execution_mode_labels"] = [
str(item.get("label") or "").strip()
for item in execution_mode_options
if str(item.get("label") or "").strip()
]
return normalized_payload

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,494 @@
from __future__ import annotations
from app.core.config import settings
from app.services.cluster_runtime_service import get_cluster_snapshot
from app.services.ops_execution_capability_service import validate_action_execution_mode
from app.services.ops_release_service import (
_calculate_total_batches,
_normalize_rollout_policy,
_resolve_rollout_targets,
build_release_rollout_gate,
get_release,
)
_LOW_RISK_ACTIONS = {
"health.snapshot",
"runtime.push_sync",
"runtime.pull_tasks",
"logs.collect",
"diagnostics.collect",
"delivery.queue.flush",
}
_MEDIUM_RISK_ACTIONS = {
"runtime.start_worker",
"runtime.start_sync_agent",
"service.status",
"health.check",
"delivery.queue.replay",
}
_HIGH_RISK_ACTIONS = {
"runtime.stop_worker",
"runtime.stop_sync_agent",
"runtime.restart_api",
"service.restart",
"deploy.release",
"delivery.queue.discard",
}
_CRITICAL_RISK_ACTIONS = {
"deploy.rollback",
"node.bootstrap",
"cluster.reconfigure",
}
def _normalize_node_code_list(raw_value: object) -> list[str]:
if isinstance(raw_value, list):
return [str(item).strip() for item in raw_value if str(item).strip()]
text = str(raw_value or "").replace("\r", "\n").strip()
if not text:
return []
normalized = text.replace(",", "\n")
return [item.strip() for item in normalized.split("\n") if item.strip()]
def _dedupe_text_items(items: list[str]) -> list[str]:
return list(dict.fromkeys(str(item or "").strip() for item in items if str(item or "").strip()))
def _risk_level_for_action(action: str) -> str:
if action in _LOW_RISK_ACTIONS:
return "low"
if action in _HIGH_RISK_ACTIONS:
return "high"
if action in _CRITICAL_RISK_ACTIONS:
return "critical"
return "medium"
def _compact_target_node(item: dict) -> dict:
return {
"node_code": str(item.get("node_code") or ""),
"region": str(item.get("region") or ""),
"role": str(item.get("role") or ""),
"status": str(item.get("status") or ""),
"current_load": int(item.get("current_load", 0) or 0),
"is_effective_worker": bool(item.get("is_effective_worker", False)),
"detect_participating": bool(item.get("detect_participating", False)),
"is_enabled": bool(item.get("is_enabled", True)),
"deploy_channel": str(item.get("deploy_channel") or ""),
}
def _preview_action_payload_guardrails(
action: str,
payload: dict,
*,
execution_mode: str,
target_nodes_total: int,
) -> dict:
warnings: list[str] = []
blocking_reasons: list[str] = []
approval_reasons: list[str] = []
recommendations: list[str] = []
if action == "deploy.release":
release_version = str(payload.get("release_version") or "").strip()
artifact_url = str(payload.get("artifact_url") or "").strip()
if not release_version:
blocking_reasons.append("deploy.release 缺少 release_version当前不能创建正式发布任务。")
if not artifact_url:
blocking_reasons.append("deploy.release 缺少 artifact_url当前不能创建正式发布任务。")
if execution_mode not in {"remote-agent", "ssh"}:
blocking_reasons.append("deploy.release 仅支持 remote-agent 或 ssh 执行方式。")
if target_nodes_total > 1:
recommendations.append("建议先单节点 canary 验证发布任务,再逐步扩到更多节点。")
if action == "node.bootstrap":
if execution_mode != "control-plane":
blocking_reasons.append("node.bootstrap 仅支持 control-plane 执行方式。")
if target_nodes_total > 1:
recommendations.append("接管动作建议按单节点节奏推进,先确认首台节点接入成功后再继续放量。")
if action in _CRITICAL_RISK_ACTIONS:
approval_reasons.append("该动作属于 critical 风险动作,正式环境必须审批。")
elif action in _HIGH_RISK_ACTIONS:
approval_reasons.append("该动作属于 high 风险动作,建议审批后执行。")
return {
"warnings": _dedupe_text_items(warnings),
"blocking_reasons": _dedupe_text_items(blocking_reasons),
"approval_reasons": _dedupe_text_items(approval_reasons),
"recommendations": _dedupe_text_items(recommendations),
}
def _preview_single_node_policy(
action: str,
*,
target_node_code: str,
cluster: dict,
payload: dict | None = None,
execution_mode: str = "remote-agent",
target_type: str = "node",
include_payload_guardrails: bool = True,
) -> dict:
payload = dict(payload or {})
nodes = list(cluster.get("nodes") or [])
summary = cluster.get("summary") or {}
target_node = next((item for item in nodes if str(item.get("node_code") or "").strip() == target_node_code), {})
risk_level = _risk_level_for_action(action)
warnings: list[str] = []
blocking_reasons: list[str] = []
approval_reasons: list[str] = []
recommendations: list[str] = []
execution_supported, execution_reason = validate_action_execution_mode(
action,
execution_mode,
target_node_code=target_node_code,
current_node_code=settings.node_code,
)
if not execution_supported and execution_reason:
blocking_reasons.append(execution_reason)
if target_type == "node" and target_node_code and not target_node:
warnings.append("目标节点当前未出现在集群快照中,可能尚未注册或已离线。")
node_status = str(target_node.get("status") or "")
node_role = str(target_node.get("role") or "")
node_current_load = int(target_node.get("current_load", 0) or 0)
node_effective_worker = bool(target_node.get("is_effective_worker", False))
node_detect_participating = bool(target_node.get("detect_participating", False))
if target_node:
if node_status in {"busy"} and action in {"runtime.stop_worker", "runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
blocking_reasons.append("目标节点当前处于 busy 状态,不适合直接执行中断类动作。")
if node_detect_participating and action in {"runtime.stop_worker", "runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
blocking_reasons.append("目标节点正在参与检测,需先迁移负载或人工确认后再执行。")
if node_role == "control" and action in {"runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
approval_reasons.append("目标节点是 control 节点,建议强制走审批或维护窗口。")
if node_effective_worker and int(summary.get("online_worker_nodes", 0) or 0) <= 1 and action in {"runtime.stop_worker", "deploy.release", "deploy.rollback", "service.restart"}:
blocking_reasons.append("当前有效执行节点数不足 2执行该动作可能导致检测能力中断。")
if node_current_load > 0:
warnings.append(f"目标节点当前负载为 {node_current_load},建议优先观察或转移任务。")
if include_payload_guardrails:
payload_guardrails = _preview_action_payload_guardrails(
action,
payload,
execution_mode=execution_mode,
target_nodes_total=1,
)
warnings.extend(payload_guardrails.get("warnings") or [])
blocking_reasons.extend(payload_guardrails.get("blocking_reasons") or [])
approval_reasons.extend(payload_guardrails.get("approval_reasons") or [])
recommendations.extend(payload_guardrails.get("recommendations") or [])
warnings = _dedupe_text_items(warnings)
blocking_reasons = _dedupe_text_items(blocking_reasons)
approval_reasons = _dedupe_text_items(approval_reasons)
recommendations = _dedupe_text_items(recommendations)
approval_required = bool(approval_reasons)
blocked = bool(blocking_reasons)
return {
"action": action,
"target_type": target_type,
"target_node_code": target_node_code,
"execution_mode": execution_mode,
"risk_level": risk_level,
"approval_required": approval_required,
"blocked": blocked,
"blocking_reasons": blocking_reasons,
"approval_reasons": approval_reasons,
"warnings": warnings,
"recommendations": recommendations,
"target_node": {
"node_code": str(target_node.get("node_code") or ""),
"region": str(target_node.get("region") or ""),
"role": node_role,
"status": node_status,
"current_load": node_current_load,
"is_effective_worker": node_effective_worker,
"detect_participating": node_detect_participating,
} if target_node else {},
"cluster_guardrails": {
"online_worker_nodes": int(summary.get("online_worker_nodes", 0) or 0),
"dedicated_online_worker_nodes": int(summary.get("dedicated_online_worker_nodes", 0) or 0),
"online_control_nodes": int(summary.get("online_control_nodes", 0) or 0),
"busy_nodes": list(summary.get("busy_nodes") or []),
},
}
def _preview_node_batch_policy(action: str, payload: dict, cluster: dict) -> dict:
target_node_codes = _normalize_node_code_list(payload.get("target_node_codes") or payload.get("node_codes"))
execution_mode = str(payload.get("execution_mode") or "remote-agent").strip() or "remote-agent"
action_payload = dict(payload.get("payload") or {})
summary = cluster.get("summary") or {}
warnings: list[str] = []
blocking_reasons: list[str] = []
approval_reasons: list[str] = []
recommendations: list[str] = []
if not target_node_codes:
blocking_reasons.append("当前没有可预检的目标节点。")
shared_guardrails = _preview_action_payload_guardrails(
action,
action_payload,
execution_mode=execution_mode,
target_nodes_total=len(target_node_codes),
)
warnings.extend(shared_guardrails.get("warnings") or [])
blocking_reasons.extend(shared_guardrails.get("blocking_reasons") or [])
approval_reasons.extend(shared_guardrails.get("approval_reasons") or [])
recommendations.extend(shared_guardrails.get("recommendations") or [])
node_previews = [
_preview_single_node_policy(
action,
target_node_code=node_code,
cluster=cluster,
payload=action_payload,
execution_mode=execution_mode,
target_type="batch",
include_payload_guardrails=False,
)
for node_code in target_node_codes
]
blocked_nodes = [item for item in node_previews if bool(item.get("blocked", False))]
approval_nodes = [item for item in node_previews if bool(item.get("approval_required", False))]
warning_nodes = [item for item in node_previews if list(item.get("warnings") or [])]
missing_nodes = [item for item in node_previews if not (item.get("target_node") or {}).get("node_code")]
target_control_nodes = [item for item in node_previews if str((item.get("target_node") or {}).get("role") or "") == "control"]
if blocked_nodes:
blocking_reasons.append(f"共有 {len(blocked_nodes)} 个目标节点存在节点级阻断,需先处理后再批量执行。")
if approval_nodes:
approval_reasons.append(f"共有 {len(approval_nodes)} 个目标节点建议先审批。")
if warning_nodes:
warnings.append(f"共有 {len(warning_nodes)} 个目标节点存在需关注事项。")
if missing_nodes:
warnings.append(f"共有 {len(missing_nodes)} 个目标节点未出现在当前集群快照中。")
if target_control_nodes and len(target_node_codes) > 1:
recommendations.append("建议把 control 节点放到批次尾部,优先在 worker 或空闲节点验证。")
warnings = _dedupe_text_items(warnings)
blocking_reasons = _dedupe_text_items(blocking_reasons)
approval_reasons = _dedupe_text_items(approval_reasons)
recommendations = _dedupe_text_items(recommendations)
return {
"action": action,
"target_type": "batch",
"execution_mode": execution_mode,
"risk_level": _risk_level_for_action(action),
"approval_required": bool(approval_reasons),
"blocked": bool(blocking_reasons),
"blocking_reasons": blocking_reasons,
"approval_reasons": approval_reasons,
"warnings": warnings,
"recommendations": recommendations,
"target_summary": {
"nodes_total": len(target_node_codes),
"blocked_nodes": len(blocked_nodes),
"approval_nodes": len(approval_nodes),
"warning_nodes": len(warning_nodes),
"missing_nodes": len(missing_nodes),
},
"target_nodes": [
{
**(item.get("target_node") or {}),
"node_code": str((item.get("target_node") or {}).get("node_code") or item.get("target_node_code") or ""),
"risk_level": str(item.get("risk_level") or ""),
"approval_required": bool(item.get("approval_required", False)),
"blocked": bool(item.get("blocked", False)),
"blocking_reasons": list(item.get("blocking_reasons") or []),
"approval_reasons": list(item.get("approval_reasons") or []),
"warnings": list(item.get("warnings") or []),
}
for item in node_previews
],
"cluster_guardrails": {
"online_worker_nodes": int(summary.get("online_worker_nodes", 0) or 0),
"dedicated_online_worker_nodes": int(summary.get("dedicated_online_worker_nodes", 0) or 0),
"online_control_nodes": int(summary.get("online_control_nodes", 0) or 0),
"busy_nodes": list(summary.get("busy_nodes") or []),
},
}
def _preview_rollout_policy(action: str, payload: dict, cluster: dict) -> dict:
selector = payload.get("target_selector") or {}
input_policy = payload.get("policy") or {}
release_id = int(payload.get("release_id") or 0)
input_release = dict(payload.get("release") or {})
release = input_release or (get_release(release_id) if release_id > 0 else {})
target_nodes = _resolve_rollout_targets(selector)
normalized_policy = _normalize_rollout_policy(input_policy, total_targets=len(target_nodes))
execution_mode = str(normalized_policy.get("execution_mode") or "remote-agent").strip() or "remote-agent"
summary = cluster.get("summary") or {}
release_gate = build_release_rollout_gate(
release,
target_nodes,
execution_mode=execution_mode,
)
operational_readiness = dict(release_gate.get("operational_readiness") or {})
risk_level = _risk_level_for_action(action)
warnings: list[str] = []
blocking_reasons: list[str] = []
approval_reasons: list[str] = []
recommendations: list[str] = []
target_count = len(target_nodes)
first_batch_size = int(normalized_policy.get("first_batch_size") or 0)
batch_size = int(normalized_policy.get("batch_size") or 0)
batches_total = _calculate_total_batches(target_count, normalized_policy)
first_batch_targets = target_nodes[:first_batch_size] if first_batch_size > 0 else []
online_worker_nodes = int(summary.get("online_worker_nodes", 0) or 0)
dedicated_online_worker_nodes = int(summary.get("dedicated_online_worker_nodes", 0) or 0)
online_control_nodes = int(summary.get("online_control_nodes", 0) or 0)
target_control_nodes = [item for item in target_nodes if str(item.get("role") or "") == "control"]
target_worker_nodes = [item for item in target_nodes if str(item.get("role") or "") == "worker"]
target_effective_workers = [item for item in target_nodes if bool(item.get("is_effective_worker", False))]
first_batch_effective_workers = [item for item in first_batch_targets if bool(item.get("is_effective_worker", False))]
first_batch_control_nodes = [item for item in first_batch_targets if str(item.get("role") or "") == "control"]
busy_targets = [item for item in target_nodes if int(item.get("current_load", 0) or 0) > 0]
participating_targets = [item for item in target_nodes if bool(item.get("detect_participating", False))]
offline_targets = [item for item in target_nodes if str(item.get("status") or "") not in {"online", "busy"}]
if target_count <= 0:
blocking_reasons.append("当前筛选条件未命中任何可 rollout 的目标节点。")
if release_id > 0 and not input_release and not release:
blocking_reasons.append("指定的 Release 不存在,当前无法预检该 rollout。")
gate_status = str(release_gate.get("status") or "").strip()
if gate_status in {"release_not_ready", "artifact_missing"}:
warnings.append(str(release_gate.get("summary") or "").strip())
if offline_targets:
warnings.append(f"命中了 {len(offline_targets)} 个非在线节点,若保留 only_online=false发布时可能出现排队或失败。")
if busy_targets:
warnings.append(f"命中了 {len(busy_targets)} 个当前有负载的节点,建议优先确认任务是否可中断。")
if participating_targets:
warnings.append(f"命中了 {len(participating_targets)} 个正在参与检测的节点,建议优先滚动到空闲节点。")
if target_control_nodes:
approval_reasons.append("本次 rollout 覆盖 control 节点,建议强制审批,并把 control 放在最后一批。")
if first_batch_effective_workers and online_worker_nodes > 0 and len(first_batch_effective_workers) >= online_worker_nodes:
blocking_reasons.append("第一批会覆盖全部有效执行节点,检测能力可能瞬时归零。请缩小 first_batch_size。")
if first_batch_control_nodes and online_control_nodes > 0 and len(first_batch_control_nodes) >= online_control_nodes:
blocking_reasons.append("第一批会覆盖全部在线控制面,运维中枢本身可能不可用。请调整节点顺序或缩小 first_batch_size。")
if first_batch_size > 1 and target_control_nodes:
warnings.append("当前第一批大于 1且目标中包含 control 节点;默认建议先 worker、后 control。")
if action in _CRITICAL_RISK_ACTIONS:
approval_reasons.append("该动作属于 critical 风险动作,正式环境必须审批。")
elif action in _HIGH_RISK_ACTIONS:
approval_reasons.append("该动作属于 high 风险动作,建议审批后执行。")
blocking_reasons.extend(list(release_gate.get("blocking_reasons") or []))
warnings.extend(list(release_gate.get("warning_reasons") or []))
recommendations.extend(list(release_gate.get("recommendations") or []))
readiness_summary = dict(operational_readiness.get("summary") or {})
if target_control_nodes and int(readiness_summary.get("inspection_healthy_nodes", 0) or 0) < target_count:
approval_reasons.append("本次目标中存在巡检未完全通过的节点,建议先完成标准巡检并经人工确认后再发布。")
if target_count > 1 and first_batch_size > 1:
recommendations.append("正式环境建议 first_batch_size=1从单节点 canary 开始。")
if target_control_nodes:
recommendations.append("建议把 control 节点放在 rollout 末尾,并确保 worker 已先通过健康检查。")
if busy_targets or participating_targets:
recommendations.append("建议先等参与检测的节点降为空闲,再推进该批次。")
warnings = list(dict.fromkeys(item for item in warnings if str(item or "").strip()))
blocking_reasons = list(dict.fromkeys(item for item in blocking_reasons if str(item or "").strip()))
approval_reasons = list(dict.fromkeys(item for item in approval_reasons if str(item or "").strip()))
recommendations = list(dict.fromkeys(item for item in recommendations if str(item or "").strip()))
approval_required = bool(approval_reasons)
blocked = bool(blocking_reasons)
return {
"action": action,
"target_type": "rollout",
"risk_level": risk_level,
"approval_required": approval_required,
"blocked": blocked,
"blocking_reasons": blocking_reasons,
"approval_reasons": approval_reasons,
"warnings": warnings,
"recommendations": recommendations,
"target_summary": {
"nodes_total": target_count,
"worker_nodes": len(target_worker_nodes),
"control_nodes": len(target_control_nodes),
"effective_worker_nodes": len(target_effective_workers),
"busy_nodes": len(busy_targets),
"participating_nodes": len(participating_targets),
"offline_nodes": len(offline_targets),
},
"batch_plan": {
"first_batch_size": first_batch_size,
"batch_size": batch_size,
"batches_total": batches_total,
"first_batch_nodes": [_compact_target_node(item) for item in first_batch_targets],
"remaining_after_first_batch": max(target_count - len(first_batch_targets), 0),
},
"cluster_guardrails": {
"online_worker_nodes": online_worker_nodes,
"dedicated_online_worker_nodes": dedicated_online_worker_nodes,
"online_control_nodes": online_control_nodes,
"busy_nodes": list(summary.get("busy_nodes") or []),
},
"release": release,
"release_gate": release_gate,
"target_nodes": [_compact_target_node(item) for item in target_nodes],
"operational_readiness": operational_readiness,
}
def preview_ops_job_policy(payload: dict) -> dict:
action = str(payload.get("action") or "").strip()
target_type = str(payload.get("target_type") or "node").strip() or "node"
target_node_code = str(payload.get("target_node_code") or "").strip()
execution_mode = str(payload.get("execution_mode") or "remote-agent").strip() or "remote-agent"
action_payload = dict(payload.get("payload") or {})
cluster = get_cluster_snapshot()
if target_type == "rollout":
return _preview_rollout_policy(action, payload, cluster)
if target_type in {"batch", "nodes"} or list(payload.get("target_node_codes") or []):
return _preview_node_batch_policy(action, payload, cluster)
return _preview_single_node_policy(
action,
target_node_code=target_node_code,
cluster=cluster,
payload=action_payload,
execution_mode=execution_mode,
target_type=target_type,
include_payload_guardrails=True,
)

View File

@@ -0,0 +1,475 @@
from __future__ import annotations
import hashlib
import inspect
import json
import shutil
import tarfile
import textwrap
import time
import urllib.error
import urllib.request
from datetime import datetime
from pathlib import Path
def normalize_text_list(raw_value: object) -> list[str]:
if isinstance(raw_value, list):
return [str(item).strip() for item in raw_value if str(item).strip()]
text = str(raw_value or "").replace("\r", "\n").strip()
if not text:
return []
normalized = text.replace(",", "\n")
return [item.strip() for item in normalized.split("\n") if item.strip()]
def coerce_bool(raw_value: object, default: bool = False) -> bool:
if raw_value is None:
return bool(default)
if isinstance(raw_value, bool):
return raw_value
if isinstance(raw_value, (int, float)):
return bool(raw_value)
normalized = str(raw_value or "").strip().lower()
if normalized in {"1", "true", "yes", "y", "on", "enable", "enabled"}:
return True
if normalized in {"0", "false", "no", "n", "off", "disable", "disabled"}:
return False
return bool(default) if normalized == "" else bool(normalized)
def normalize_release_health_check_services(
payload: dict,
restart_services: list[str],
*,
default_api_service_name: str,
) -> list[str]:
raw_health_check_services = payload.get("health_check_services")
if raw_health_check_services is None:
health_check_service_source = restart_services or [default_api_service_name]
else:
health_check_service_source = raw_health_check_services
return normalize_text_list(health_check_service_source)
def collect_service_state(run_command, service_name: str) -> dict:
code, stdout, stderr = run_command(["systemctl", "is-active", service_name], timeout=15)
state = stdout or stderr
return {
"service_name": service_name,
"returncode": int(code or 0),
"state": state,
"ok": int(code or 0) == 0 and state == "active",
}
def check_health_url(url: str, timeout: int, *, user_agent: str, urlopen_func) -> dict:
request = urllib.request.Request(
url=str(url).strip(),
headers={"User-Agent": str(user_agent or "domaincheck-ops/0.1").strip()},
method="GET",
)
try:
with urlopen_func(request, timeout=timeout) as response:
body = response.read(4000).decode("utf-8", errors="ignore")
status_code = int(getattr(response, "status", 0) or response.getcode() or 0)
return {
"url": str(url).strip(),
"ok": 200 <= status_code < 400,
"http_status": status_code,
"body_preview": body[-1000:],
}
except urllib.error.HTTPError as exc:
body = exc.read().decode("utf-8", errors="ignore") if exc.fp else ""
return {
"url": str(url).strip(),
"ok": False,
"http_status": int(exc.code or 0),
"body_preview": body[-1000:],
"error": str(exc),
}
except Exception as exc:
return {
"url": str(url).strip(),
"ok": False,
"http_status": 0,
"body_preview": "",
"error": str(exc),
}
def run_release_health_checks(
*,
urls: list[str],
services: list[str],
timeout: int,
retries: int,
interval_seconds: int,
run_command,
user_agent: str,
urlopen_func,
) -> tuple[bool, dict]:
normalized_urls = [str(item).strip() for item in urls if str(item).strip()]
normalized_services = [str(item).strip() for item in services if str(item).strip()]
attempts: list[dict] = []
total_attempts = max(1, retries + 1)
for attempt_index in range(1, total_attempts + 1):
service_results = [collect_service_state(run_command, service_name) for service_name in normalized_services]
url_results = [
check_health_url(url, timeout=timeout, user_agent=user_agent, urlopen_func=urlopen_func)
for url in normalized_urls
]
services_ok = all(bool(item.get("ok", False)) for item in service_results) if service_results else True
urls_ok = all(bool(item.get("ok", False)) for item in url_results) if url_results else True
attempt_payload = {
"attempt": attempt_index,
"services": service_results,
"urls": url_results,
"ok": services_ok and urls_ok,
}
attempts.append(attempt_payload)
if attempt_payload["ok"]:
return True, {
"ok": True,
"attempts": attempts,
"services_checked": normalized_services,
"urls_checked": normalized_urls,
}
if attempt_index < total_attempts:
time.sleep(max(0, interval_seconds))
return False, {
"ok": False,
"attempts": attempts,
"services_checked": normalized_services,
"urls_checked": normalized_urls,
}
def safe_extract_tar(archive: tarfile.TarFile, target_dir: Path) -> None:
target_dir_resolved = target_dir.resolve()
members = archive.getmembers()
for member in members:
member_path = (target_dir / member.name).resolve()
if not str(member_path).startswith(str(target_dir_resolved)):
raise RuntimeError(f"unsafe archive member: {member.name}")
try:
archive.extractall(target_dir, members=members, filter="data")
except TypeError:
archive.extractall(target_dir, members=members)
def execute_release_action(
payload: dict,
*,
run_command,
default_api_service_name: str,
event_callback=None,
urlopen_func=None,
user_agent: str = "domaincheck-ops/0.1",
) -> tuple[bool, str, dict]:
normalized_payload = dict(payload or {})
event_callback = event_callback or (lambda event_type, message, level="info", payload=None: None)
urlopen_func = urlopen_func or urllib.request.urlopen
release_version = str(normalized_payload.get("release_version") or "").strip()
artifact_url = str(normalized_payload.get("artifact_url") or "").strip()
checksum = str(normalized_payload.get("checksum") or "").strip().lower()
install_root = Path(str(normalized_payload.get("install_root") or "/opt/domaincheck")).resolve()
switch_current = coerce_bool(normalized_payload.get("switch_current", True), default=True)
restart_services = normalize_text_list(normalized_payload.get("restart_services"))
health_check_urls = normalize_text_list(normalized_payload.get("health_check_urls"))
health_check_services = normalize_release_health_check_services(
normalized_payload,
restart_services,
default_api_service_name=default_api_service_name,
)
health_check_timeout_seconds = max(2, int(normalized_payload.get("health_check_timeout_seconds") or 10))
health_check_retries = max(0, int(normalized_payload.get("health_check_retries") or 2))
health_check_interval_seconds = max(0, int(normalized_payload.get("health_check_interval_seconds") or 2))
rollback_on_failure = coerce_bool(normalized_payload.get("rollback_on_failure", True), default=True)
if not release_version:
return False, "release_version missing", {}
if not artifact_url:
return False, "artifact_url missing", {}
downloads_dir = install_root / "downloads"
releases_dir = install_root / "releases"
target_dir = releases_dir / release_version
temp_dir = releases_dir / f"{release_version}.tmp"
artifact_path = downloads_dir / f"{release_version}.tar.gz"
current_link = install_root / "current"
previous_current_target = ""
downloads_dir.mkdir(parents=True, exist_ok=True)
releases_dir.mkdir(parents=True, exist_ok=True)
if current_link.exists():
try:
previous_current_target = str(current_link.resolve(strict=True))
except Exception:
try:
previous_current_target = str(current_link.resolve())
except Exception:
previous_current_target = ""
event_callback(
"deploy_download_started",
f"开始下载发布包: {release_version}",
payload={"artifact_url": artifact_url},
)
request = urllib.request.Request(
url=artifact_url,
headers={"User-Agent": str(user_agent or "domaincheck-ops/0.1").strip()},
method="GET",
)
with urlopen_func(request, timeout=120) as response:
artifact_bytes = response.read()
artifact_path.write_bytes(artifact_bytes)
event_callback(
"deploy_download_completed",
f"发布包下载完成: {release_version}",
payload={"artifact_path": str(artifact_path), "size_bytes": len(artifact_bytes)},
)
calculated_checksum = hashlib.sha256(artifact_bytes).hexdigest().lower()
if checksum and checksum != calculated_checksum:
return False, "release checksum mismatch", {
"expected_checksum": checksum,
"calculated_checksum": calculated_checksum,
}
event_callback(
"deploy_checksum_verified",
f"发布包校验通过: {release_version}",
payload={"checksum": calculated_checksum},
)
if temp_dir.exists():
shutil.rmtree(temp_dir, ignore_errors=True)
temp_dir.mkdir(parents=True, exist_ok=True)
with tarfile.open(artifact_path, "r:gz") as archive:
safe_extract_tar(archive, temp_dir)
if target_dir.exists():
extracted_target = target_dir
shutil.rmtree(temp_dir, ignore_errors=True)
else:
temp_dir.rename(target_dir)
extracted_target = target_dir
meta_path = extracted_target / ".release-meta.json"
meta_path.write_text(
json.dumps(
{
"release_version": release_version,
"artifact_url": artifact_url,
"checksum": calculated_checksum,
"deployed_at": datetime.now().isoformat(sep=" ", timespec="seconds"),
},
ensure_ascii=False,
indent=2,
),
encoding="utf-8",
)
if switch_current:
if current_link.is_symlink() or current_link.is_file():
current_link.unlink(missing_ok=True)
elif current_link.exists():
raise RuntimeError(f"current link path exists and is not a symlink/file: {current_link}")
current_link.symlink_to(extracted_target)
event_callback(
"deploy_current_switched",
f"current 已切换到 {release_version}",
payload={"current_link": str(current_link), "target": str(extracted_target)},
)
restarted: list[dict] = []
for service_name in restart_services:
normalized_service_name = str(service_name or "").strip()
if not normalized_service_name:
continue
event_callback(
"deploy_service_restart",
f"重启服务: {normalized_service_name}",
payload={"service_name": normalized_service_name},
)
code, stdout, stderr = run_command(["systemctl", "restart", normalized_service_name], timeout=30)
restarted.append(
{
"service_name": normalized_service_name,
"returncode": int(code or 0),
"stdout": stdout,
"stderr": stderr,
}
)
if int(code or 0) != 0:
return False, f"restart failed: {normalized_service_name}", {
"release_version": release_version,
"release_dir": str(extracted_target),
"artifact_path": str(artifact_path),
"checksum": calculated_checksum,
"previous_current_target": previous_current_target,
"restart_results": restarted,
}
health_ok, health_result = run_release_health_checks(
urls=health_check_urls,
services=health_check_services,
timeout=health_check_timeout_seconds,
retries=health_check_retries,
interval_seconds=health_check_interval_seconds,
run_command=run_command,
user_agent=user_agent,
urlopen_func=urlopen_func,
)
if not health_ok:
rollback_result = {
"attempted": rollback_on_failure,
"restored_to": previous_current_target,
"restart_results": [],
"health_result": health_result,
}
event_callback(
"deploy_health_failed",
f"发布健康检查失败: {release_version}",
level="error",
payload=health_result,
)
if rollback_on_failure and previous_current_target:
if current_link.is_symlink() or current_link.is_file():
current_link.unlink(missing_ok=True)
elif current_link.exists():
raise RuntimeError(f"current link path exists and is not a symlink/file: {current_link}")
current_link.symlink_to(Path(previous_current_target))
for service_name in restart_services:
normalized_service_name = str(service_name or "").strip()
if not normalized_service_name:
continue
code, stdout, stderr = run_command(["systemctl", "restart", normalized_service_name], timeout=30)
rollback_result["restart_results"].append(
{
"service_name": normalized_service_name,
"returncode": int(code or 0),
"stdout": stdout,
"stderr": stderr,
}
)
rollback_result["post_rollback_health"] = run_release_health_checks(
urls=health_check_urls,
services=health_check_services,
timeout=health_check_timeout_seconds,
retries=0,
interval_seconds=0,
run_command=run_command,
user_agent=user_agent,
urlopen_func=urlopen_func,
)[1]
event_callback(
"deploy_rollback_completed",
f"已回滚到旧版本: {previous_current_target}",
level="warning",
payload=rollback_result,
)
return False, "release health check failed", {
"release_version": release_version,
"release_dir": str(extracted_target),
"artifact_path": str(artifact_path),
"checksum": calculated_checksum,
"current_link": str(current_link),
"previous_current_target": previous_current_target,
"restart_results": restarted,
"health_check": health_result,
"rollback": rollback_result,
}
event_callback(
"deploy_health_passed",
f"发布健康检查通过: {release_version}",
payload=health_result,
)
return True, "release deployed", {
"release_version": release_version,
"release_dir": str(extracted_target),
"artifact_path": str(artifact_path),
"checksum": calculated_checksum,
"current_link": str(current_link),
"previous_current_target": previous_current_target,
"restart_results": restarted,
"health_check": health_result,
}
def build_remote_release_action_script(
payload: dict,
*,
default_api_service_name: str = "domaincheck-api",
user_agent: str = "domaincheck-ssh/0.1",
) -> str:
helper_functions = [
normalize_text_list,
coerce_bool,
normalize_release_health_check_services,
collect_service_state,
check_health_url,
run_release_health_checks,
safe_extract_tar,
execute_release_action,
]
helper_source = "\n\n".join(
textwrap.dedent(inspect.getsource(func)).strip("\n")
for func in helper_functions
)
return f"""from __future__ import annotations
import hashlib
import json
import shutil
import tarfile
import time
import urllib.error
import urllib.request
from datetime import datetime
from pathlib import Path
{helper_source}
PAYLOAD = {json.dumps(dict(payload or {{}}), ensure_ascii=False)}
DEFAULT_API_SERVICE_NAME = {json.dumps(str(default_api_service_name or 'domaincheck-api'), ensure_ascii=False)}
USER_AGENT = {json.dumps(str(user_agent or 'domaincheck-ssh/0.1'), ensure_ascii=False)}
def _run(command, timeout=60):
import subprocess
completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
return int(completed.returncode or 0), str(completed.stdout or "").strip(), str(completed.stderr or "").strip()
def _event_callback(event_type, message, level="info", payload=None):
return None
def main():
try:
ok, message, result = execute_release_action(
PAYLOAD,
run_command=_run,
default_api_service_name=DEFAULT_API_SERVICE_NAME,
event_callback=_event_callback,
urlopen_func=urllib.request.urlopen,
user_agent=USER_AGENT,
)
except Exception as exc:
ok, message, result = False, str(exc), {{"exception": str(exc), "action": "deploy.release"}}
print(json.dumps({{"ok": ok, "message": message, "result": result}}, ensure_ascii=False))
raise SystemExit(0 if ok else 1)
if __name__ == "__main__":
main()
"""

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,420 @@
from __future__ import annotations
import json
import subprocess
from app.core.config import settings
from app.services.ops_action_executor_core import (
STRUCTURED_ACTIONS,
build_service_name_map,
execute_structured_action,
trim_output,
)
from app.services.ops_release_executor_core import build_remote_release_action_script, execute_release_action
_SSH_SUPPORTED_ACTIONS = set(STRUCTURED_ACTIONS) | {"deploy.release"}
_SSH_CONNECT_TIMEOUT_SECONDS = 12
_SSH_REMOTE_TIMEOUT_SECONDS = {
"health.snapshot": 30,
"service.status": 45,
"service.restart": 45,
"service.start": 45,
"service.stop": 45,
"logs.collect": 45,
"diagnostics.collect": 90,
"runtime.start_worker": 45,
"runtime.stop_worker": 45,
"runtime.restart_api": 45,
"runtime.start_sync_agent": 45,
"runtime.stop_sync_agent": 45,
"deploy.release": 180,
}
def supports_ssh_execution(action: str) -> bool:
return str(action or "").strip() in _SSH_SUPPORTED_ACTIONS
def execute_local_support_action(action: str, payload: dict | None = None) -> tuple[bool, str, dict]:
normalized_action = str(action or "").strip()
if normalized_action not in _SSH_SUPPORTED_ACTIONS:
return False, f"当前未实现本机支持动作: {normalized_action}", {}
if normalized_action == "deploy.release":
return execute_release_action(
dict(payload or {}),
run_command=_run_local_command,
default_api_service_name=settings.api_service_name,
user_agent="domaincheck-local-runtime/0.1",
)
return execute_structured_action(
normalized_action,
dict(payload or {}),
service_names=_service_name_map(),
runner=_run_local_command,
)
def execute_ssh_action(node: dict, action: str, payload: dict | None = None) -> tuple[bool, str, dict]:
normalized_action = str(action or "").strip()
if normalized_action not in _SSH_SUPPORTED_ACTIONS:
return False, f"当前 SSH 执行器暂不支持动作: {normalized_action}", {}
ssh_host = str(node.get("ssh_host") or "").strip()
ssh_user = str(node.get("ssh_user") or "").strip()
ssh_port = max(1, int(node.get("ssh_port") or 22))
node_code = str(node.get("node_code") or "").strip()
if not ssh_host or not ssh_user:
return False, "目标节点缺少 SSH 主机或用户信息", {
"node_code": node_code,
"ssh_host": ssh_host,
"ssh_user": ssh_user,
"ssh_port": ssh_port,
}
normalized_payload = dict(payload or {})
service_names = _service_name_map()
if normalized_action == "deploy.release":
remote_script = build_remote_release_action_script(
normalized_payload,
default_api_service_name=service_names.get("api") or "domaincheck-api",
user_agent="domaincheck-ssh/0.1",
)
else:
remote_script = _build_remote_python_script(
action=normalized_action,
payload=normalized_payload,
service_names=service_names,
)
remote_command = "\n".join(
[
"set -euo pipefail",
"if command -v python3 >/dev/null 2>&1; then",
" PYTHON_BIN=python3",
"elif command -v python >/dev/null 2>&1; then",
" PYTHON_BIN=python",
"else",
' echo "{\\"ok\\": false, \\"message\\": \\"python interpreter missing on remote node\\", \\"result\\": {\\"executor\\": \\"ssh\\"}}"',
" exit 127",
"fi",
'"${PYTHON_BIN}" - <<\'PY\'',
remote_script.rstrip("\n"),
"PY",
]
)
ssh_command = [
"ssh",
"-o",
"BatchMode=yes",
"-o",
"PreferredAuthentications=publickey",
"-o",
"StrictHostKeyChecking=accept-new",
"-o",
f"ConnectTimeout={_SSH_CONNECT_TIMEOUT_SECONDS}",
"-p",
str(ssh_port),
f"{ssh_user}@{ssh_host}",
remote_command,
]
timeout_seconds = int(_SSH_REMOTE_TIMEOUT_SECONDS.get(normalized_action, 45) or 45) + _SSH_CONNECT_TIMEOUT_SECONDS
completed = subprocess.run(
ssh_command,
capture_output=True,
text=True,
timeout=timeout_seconds,
)
transport = {
"executor": "ssh",
"node_code": node_code,
"ssh_host": ssh_host,
"ssh_user": ssh_user,
"ssh_port": ssh_port,
"action": normalized_action,
"returncode": int(completed.returncode or 0),
}
parsed_ok, parsed_payload = _extract_json_from_output(completed.stdout or "")
if parsed_ok:
ok = bool(parsed_payload.get("ok", False))
message = str(parsed_payload.get("message") or "").strip() or ("SSH 动作执行成功" if ok else "SSH 动作执行失败")
result = dict(parsed_payload.get("result") or {})
result["transport"] = {
**transport,
"stdout_preview": trim_output(completed.stdout, 4000),
"stderr_preview": trim_output(completed.stderr, 2000),
}
if completed.returncode != 0 and ok:
ok = False
message = f"SSH 命令返回码异常: {completed.returncode}"
return ok, message, result
stderr_preview = trim_output(completed.stderr, 4000)
stdout_preview = trim_output(completed.stdout, 4000)
message = stderr_preview or stdout_preview or f"SSH 执行失败,返回码 {completed.returncode}"
return False, message, {
"executor": "ssh",
"action": normalized_action,
"transport": {
**transport,
"stdout_preview": stdout_preview,
"stderr_preview": stderr_preview,
},
}
def _service_name_map() -> dict[str, str]:
return build_service_name_map(
api_service_name=settings.api_service_name,
worker_service_name=settings.worker_service_name,
sync_agent_service_name=settings.sync_agent_service_name,
)
def _run_local_command(command: list[str], *, timeout: int = 60) -> tuple[int, str, str]:
completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
return int(completed.returncode or 0), str(completed.stdout or "").strip(), str(completed.stderr or "").strip()
def _build_remote_python_script(*, action: str, payload: dict, service_names: dict[str, str]) -> str:
return f"""import json
import subprocess
import sys
from pathlib import Path
SERVICE_NAMES = {json.dumps(service_names, ensure_ascii=False)}
ACTION = {json.dumps(str(action or '').strip(), ensure_ascii=False)}
PAYLOAD = {json.dumps(dict(payload or {}), ensure_ascii=False)}
def trim_output(text, limit):
value = str(text or "").strip()
if len(value) <= int(limit):
return value
return value[-int(limit):]
def run(cmd, timeout=60):
completed = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
return int(completed.returncode or 0), str(completed.stdout or "").strip(), str(completed.stderr or "").strip()
def read_tail_lines(path, max_lines):
file_path = Path(path)
if not file_path.exists() or not file_path.is_file():
return []
with file_path.open("r", encoding="utf-8", errors="replace") as handle:
return handle.read().splitlines()[-int(max_lines):]
def candidate_domain_roots():
roots = []
for candidate in (
Path.cwd() / "domainCheck",
Path("/opt/domaincheck/domainCheck"),
Path("/www/wwwroot/getDomain/domainCheck"),
):
if candidate not in roots:
roots.append(candidate)
return roots
def candidate_runtime_logs_dirs():
dirs = []
for candidate in (
Path.cwd() / "domain-api" / "runtime" / "logs",
Path("/opt/domaincheck/domain-api/runtime/logs"),
Path("/www/wwwroot/getDomain/domain-api/runtime/logs"),
):
if candidate not in dirs:
dirs.append(candidate)
return dirs
def service_log_file_candidates(service_name):
normalized_service_name = str(service_name or "").strip()
if not normalized_service_name:
return []
candidates = []
def append_candidate(path):
if path not in candidates:
candidates.append(path)
if normalized_service_name == (SERVICE_NAMES.get("worker") or "domaincheck-worker"):
for domain_root in candidate_domain_roots():
append_candidate(domain_root / "detect_worker.log")
append_candidate(domain_root / "logs" / "detect_worker.log")
elif normalized_service_name == (SERVICE_NAMES.get("api") or "domaincheck-api"):
for runtime_logs_dir in candidate_runtime_logs_dirs():
append_candidate(runtime_logs_dir / "domain-api.stderr.log")
append_candidate(runtime_logs_dir / "domain-api.stdout.log")
for domain_root in candidate_domain_roots():
append_candidate(domain_root / "logs" / "app.log")
return candidates
def collect_log_file_fallback(service_name, lines):
aggregated_lines = []
used_paths = []
for path in service_log_file_candidates(service_name):
current_lines = read_tail_lines(path, lines)
if not current_lines:
continue
aggregated_lines.extend(current_lines)
used_paths.append(str(path))
if not aggregated_lines:
return "", []
return "\\n".join(aggregated_lines[-int(lines):]), used_paths
def service_name_from_payload(payload):
value = str((payload or {{}}).get("service_name") or "").strip()
return value or SERVICE_NAMES.get("worker") or "domaincheck-worker"
def service_name_for_action(action, payload):
if action in ("service.status", "service.restart", "service.start", "service.stop"):
return service_name_from_payload(payload)
runtime_action_map = {{
"runtime.start_worker": SERVICE_NAMES.get("worker") or "domaincheck-worker",
"runtime.stop_worker": SERVICE_NAMES.get("worker") or "domaincheck-worker",
"runtime.restart_api": SERVICE_NAMES.get("api") or "domaincheck-api",
"runtime.start_sync_agent": SERVICE_NAMES.get("sync_agent") or "domaincheck-sync-agent",
"runtime.stop_sync_agent": SERVICE_NAMES.get("sync_agent") or "domaincheck-sync-agent",
}}
return str(runtime_action_map.get(action) or "").strip()
def systemctl_action_name(action):
if action in ("service.restart", "runtime.restart_api"):
return "restart"
if action in ("service.start", "runtime.start_worker", "runtime.start_sync_agent"):
return "start"
if action in ("service.stop", "runtime.stop_worker", "runtime.stop_sync_agent"):
return "stop"
return ""
def execute(action, payload):
if action == "health.snapshot":
checks = {{}}
for key, service_name in SERVICE_NAMES.items():
code, stdout, stderr = run(["systemctl", "is-active", service_name], timeout=15)
checks[key] = {{
"service_name": service_name,
"returncode": int(code or 0),
"state": stdout or stderr,
"ok": int(code or 0) == 0 and (stdout or stderr) == "active",
}}
return True, "health snapshot collected", {{"checks": checks}}
if action == "service.status":
service_name = service_name_from_payload(payload)
code, stdout, stderr = run(["systemctl", "status", service_name, "--no-pager", "-l"], timeout=45)
result = {{
"service_name": service_name,
"stdout": trim_output(stdout, 12000),
"stderr": trim_output(stderr, 4000),
}}
return int(code or 0) == 0, stdout or stderr or f"{{service_name}} status collected", result
systemctl_action = systemctl_action_name(action)
if systemctl_action:
service_name = service_name_for_action(action, payload)
if not service_name:
return False, f"missing service name for action: {{action}}", {{"action": action}}
code, stdout, stderr = run(["systemctl", systemctl_action, service_name], timeout=45)
result = {{
"service_name": service_name,
"systemctl_action": systemctl_action,
"stdout": trim_output(stdout, 12000),
"stderr": trim_output(stderr, 4000),
}}
return int(code or 0) == 0, stdout or stderr or f"{{service_name}} {{systemctl_action}} completed", result
if action == "logs.collect":
service_name = service_name_from_payload(payload)
lines = max(20, min(int((payload or {{}}).get("lines") or 120), 500))
code, stdout, stderr = run(["journalctl", "-u", service_name, "-n", str(lines), "--no-pager"], timeout=45)
journal_output = stdout or stderr
fallback_output, fallback_paths = collect_log_file_fallback(service_name, lines)
collection_source = "journal"
effective_output = journal_output
ok = int(code or 0) == 0
if (not ok or not stdout.strip()) and fallback_output:
collection_source = "file_fallback"
effective_output = fallback_output
ok = True
result = {{
"service_name": service_name,
"lines": lines,
"collection_source": collection_source,
"fallback_used": bool(collection_source == "file_fallback"),
"fallback_reason": trim_output(journal_output, 4000) if collection_source == "file_fallback" else "",
"log_paths": fallback_paths,
"journal_returncode": int(code or 0),
"stdout": trim_output(effective_output, 20000),
"stderr": trim_output(stderr, 4000),
}}
return ok, effective_output or f"{{service_name}} logs collected", result
if action == "diagnostics.collect":
lines = max(20, min(int((payload or {{}}).get("lines") or 200), 800))
diagnostics = {{
"services": {{}},
"lines": lines,
}}
for key, service_name in SERVICE_NAMES.items():
active_code, active_stdout, active_stderr = run(["systemctl", "is-active", service_name], timeout=15)
status_code, status_stdout, status_stderr = run(["systemctl", "status", service_name, "--no-pager", "-l"], timeout=45)
log_code, log_stdout, log_stderr = run(["journalctl", "-u", service_name, "-n", str(lines), "--no-pager"], timeout=45)
log_output = log_stdout or log_stderr
fallback_output, fallback_paths = collect_log_file_fallback(service_name, lines)
log_source = "journal"
if (int(log_code or 0) != 0 or not log_stdout.strip()) and fallback_output:
log_output = fallback_output
log_source = "file_fallback"
diagnostics["services"][key] = {{
"service_name": service_name,
"active_returncode": int(active_code or 0),
"active_state": active_stdout or active_stderr,
"status_returncode": int(status_code or 0),
"status_output": trim_output(status_stdout or status_stderr, 12000),
"log_returncode": int(log_code or 0),
"log_source": log_source,
"log_paths": fallback_paths,
"log_output": trim_output(log_output, 20000),
}}
return True, "diagnostics collected", diagnostics
return False, f"unsupported action: {{action}}", {{"action": action}}
def main():
try:
ok, message, result = execute(ACTION, PAYLOAD)
except Exception as exc:
ok, message, result = False, str(exc), {{"exception": str(exc), "action": ACTION}}
print(json.dumps({{"ok": ok, "message": message, "result": result}}, ensure_ascii=False))
raise SystemExit(0 if ok else 1)
if __name__ == "__main__":
main()
"""
def _extract_json_from_output(text: str) -> tuple[bool, dict]:
for raw_line in reversed(str(text or "").splitlines()):
line = str(raw_line or "").strip()
if not line:
continue
try:
data = json.loads(line)
except Exception:
continue
if isinstance(data, dict) and "ok" in data:
return True, data
return False, {}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,920 @@
from __future__ import annotations
from copy import deepcopy
from app.core.config import settings
from app.services.ops_execution_mode_service import decorate_execution_mode_fields, execution_mode_label
def _split_text_list(raw_value: object) -> list[str]:
if isinstance(raw_value, list):
return [str(item).strip() for item in raw_value if str(item).strip()]
text = str(raw_value or "").replace("\r", "\n")
if not text.strip():
return []
normalized = text.replace(",", "\n")
return [item.strip() for item in normalized.split("\n") if item.strip()]
def _coerce_bool(raw_value: object) -> bool:
if isinstance(raw_value, bool):
return raw_value
if isinstance(raw_value, (int, float)):
return bool(raw_value)
normalized = str(raw_value or "").strip().lower()
if normalized in {"1", "true", "yes", "y", "on", "enable", "enabled"}:
return True
if normalized in {"0", "false", "no", "n", "off", "disable", "disabled"}:
return False
return bool(normalized)
def _is_missing_value(field_type: str, value: object) -> bool:
if field_type == "text_list":
return not list(value or [])
if field_type == "number":
return value is None
if field_type == "boolean":
return value is None
return str(value or "").strip() == ""
def _service_options() -> list[dict]:
return [
{"label": f"API ({settings.api_service_name})", "value": settings.api_service_name},
{"label": f"Worker ({settings.worker_service_name})", "value": settings.worker_service_name},
{"label": f"Sync Agent ({settings.sync_agent_service_name})", "value": settings.sync_agent_service_name},
{"label": "Node Agent (domaincheck-node-agent)", "value": "domaincheck-node-agent"},
]
def _template_catalog() -> list[dict]:
service_options = _service_options()
return [
{
"group_key": "runtime",
"group_title": "运行时控制",
"group_description": "适合日常按钮化控制 Worker / API / Sync Agent。",
"items": [
{
"key": "runtime.start_worker",
"title": "启动 Worker",
"action": "runtime.start_worker",
"description": "启动目标节点的检测 Worker。",
"risk_level": "medium",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control", "worker"],
"default_auto_approve": False,
"fields": [],
},
{
"key": "runtime.stop_worker",
"title": "停止 Worker",
"action": "runtime.stop_worker",
"description": "停止目标节点的检测 Worker。",
"risk_level": "high",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control", "worker"],
"default_auto_approve": False,
"fields": [],
},
{
"key": "runtime.restart_api",
"title": "重启 API",
"action": "runtime.restart_api",
"description": "重启目标节点的 domaincheck-api 服务。",
"risk_level": "high",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control"],
"default_auto_approve": False,
"fields": [],
},
{
"key": "runtime.start_sync_agent",
"title": "启动 Sync Agent",
"action": "runtime.start_sync_agent",
"description": "启动目标节点的同步代理服务。",
"risk_level": "medium",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control"],
"default_auto_approve": False,
"fields": [],
},
{
"key": "runtime.stop_sync_agent",
"title": "停止 Sync Agent",
"action": "runtime.stop_sync_agent",
"description": "停止目标节点的同步代理服务。",
"risk_level": "high",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control"],
"default_auto_approve": False,
"fields": [],
},
{
"key": "service.restart",
"title": "重启任意服务",
"action": "service.restart",
"description": "对指定 systemd 服务执行 restart。",
"risk_level": "high",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control", "worker"],
"default_auto_approve": False,
"fields": [
{
"key": "service_name",
"label": "服务名",
"type": "select",
"required": True,
"default": settings.worker_service_name,
"options": service_options,
"help": "默认提供 API / Worker / Sync Agent / Node Agent。",
}
],
},
],
},
{
"group_key": "diagnostics",
"group_title": "巡检与取证",
"group_description": "适合做联调、排障、日志回收和节点健康诊断。",
"items": [
{
"key": "health.snapshot",
"title": "采集健康快照",
"action": "health.snapshot",
"description": "采集目标节点关键 systemd 服务状态。",
"risk_level": "low",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control", "worker"],
"default_auto_approve": True,
"fields": [],
},
{
"key": "logs.collect",
"title": "收集服务日志",
"action": "logs.collect",
"description": "抓取指定服务最近 journalctl 日志。",
"risk_level": "low",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control", "worker"],
"default_auto_approve": True,
"fields": [
{
"key": "service_name",
"label": "服务名",
"type": "select",
"required": True,
"default": settings.worker_service_name,
"options": service_options,
"help": "默认抓 Worker排 API 问题时可切到 domaincheck-api。",
},
{
"key": "lines",
"label": "日志行数",
"type": "number",
"required": True,
"default": 120,
"min": 20,
"max": 500,
"help": "journalctl -n 的行数,适合短时排障。",
},
],
},
{
"key": "diagnostics.collect",
"title": "收集诊断包",
"action": "diagnostics.collect",
"description": "打包 API / Worker / Sync Agent / Node Agent 的状态与近期日志。",
"risk_level": "low",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control", "worker"],
"default_auto_approve": True,
"fields": [
{
"key": "lines",
"label": "每个服务日志行数",
"type": "number",
"required": True,
"default": 200,
"min": 20,
"max": 800,
"help": "越大越适合深度排障,但任务返回会更重。",
}
],
},
{
"key": "service.status",
"title": "采集服务状态",
"action": "service.status",
"description": "执行 systemctl status --no-pager -l。",
"risk_level": "low",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
"target_roles": ["control", "worker"],
"default_auto_approve": True,
"fields": [
{
"key": "service_name",
"label": "服务名",
"type": "select",
"required": True,
"default": settings.worker_service_name,
"options": service_options,
"help": "用于查看特定服务当前状态和最近日志片段。",
}
],
},
],
},
{
"group_key": "delivery_queue",
"group_title": "回执队列治理",
"group_description": "适合统一处理 Node Agent 的 pending / dead-letter 回执队列,而不是手工登机删文件。",
"items": [
{
"key": "delivery.queue.flush",
"title": "立即冲刷回执队列",
"action": "delivery.queue.flush",
"description": "立即触发目标节点 Node Agent 冲刷 pending 回执队列。",
"risk_level": "low",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent"],
"target_roles": ["control", "worker"],
"default_auto_approve": True,
"fields": [
{
"key": "limit",
"label": "本次冲刷上限",
"type": "number",
"required": True,
"default": 20,
"min": 1,
"max": 200,
"help": "限制本次最多冲刷的回执数量,避免一次返回过大。",
}
],
},
{
"key": "delivery.queue.replay",
"title": "重放死信回执",
"action": "delivery.queue.replay",
"description": "把 dead-letter 记录重新放回 pending并可立即触发冲刷。",
"risk_level": "medium",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent"],
"target_roles": ["control", "worker"],
"default_auto_approve": False,
"fields": [
{
"key": "record_id",
"label": "单条记录 ID",
"type": "text",
"required": False,
"default": "",
"help": "留空表示按下面的 selector 条件批量匹配。",
},
{
"key": "request_kind",
"label": "记录类型",
"type": "text",
"required": False,
"default": "",
"help": "可选 job_complete / job_event。",
},
{
"key": "detail_code",
"label": "错误码",
"type": "text",
"required": False,
"default": "",
"help": "按 last_detail_code 过滤死信记录。",
},
{
"key": "limit",
"label": "重放上限",
"type": "number",
"required": True,
"default": 20,
"min": 1,
"max": 200,
"help": "批量重放时最多处理多少条死信。",
},
{
"key": "flush_after_replay",
"label": "重放后立即冲刷",
"type": "boolean",
"required": True,
"default": True,
"help": "开启后会把重放回 pending 的记录立即补发一次。",
},
{
"key": "reason",
"label": "重放原因",
"type": "text",
"required": False,
"default": "",
"wide": True,
"help": "可选,便于后续回溯为什么执行此次重放。",
},
],
},
{
"key": "delivery.queue.discard",
"title": "丢弃死信回执",
"action": "delivery.queue.discard",
"description": "把死信移出活动队列,保存到 discarded 归档目录。",
"risk_level": "high",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent"],
"target_roles": ["control", "worker"],
"default_auto_approve": False,
"fields": [
{
"key": "record_id",
"label": "单条记录 ID",
"type": "text",
"required": False,
"default": "",
"help": "留空表示按下面的 selector 条件批量匹配。",
},
{
"key": "request_kind",
"label": "记录类型",
"type": "text",
"required": False,
"default": "",
"help": "可选 job_complete / job_event。",
},
{
"key": "detail_code",
"label": "错误码",
"type": "text",
"required": False,
"default": "",
"help": "按 last_detail_code 过滤死信记录。",
},
{
"key": "limit",
"label": "丢弃上限",
"type": "number",
"required": True,
"default": 20,
"min": 1,
"max": 200,
"help": "批量丢弃时最多处理多少条死信。",
},
{
"key": "reason",
"label": "丢弃原因",
"type": "text",
"required": True,
"default": "",
"wide": True,
"help": "必须说明为何确认这些死信可以被放弃。",
},
],
},
],
},
{
"group_key": "onboarding",
"group_title": "接管与纳管",
"group_description": "适合在控制面生成节点接入工单、统一沉淀首轮接管动作。",
"items": [
{
"key": "node.bootstrap",
"title": "生成节点接入工单",
"action": "node.bootstrap",
"description": "在控制面签发 Node Agent Token并生成 bootstrap env / 脚本 / 一键落地命令。",
"risk_level": "critical",
"default_execution_mode": "control-plane",
"execution_modes": ["control-plane"],
"target_roles": ["control", "worker"],
"default_auto_approve": False,
"fields": [
{
"key": "control_plane_base_url",
"label": "控制面地址",
"type": "text",
"required": False,
"default": "",
"wide": True,
"help": "可留空。留空时由控制面自动回退到当前 API 地址。",
},
{
"key": "root_dir",
"label": "目标安装目录",
"type": "text",
"required": True,
"default": "/opt/domaincheck",
"help": "bootstrap_node_agent.sh 安装根目录。",
},
{
"key": "expires_in_hours",
"label": "Token 有效期(小时)",
"type": "number",
"required": True,
"default": 72,
"min": 1,
"max": 720,
"help": "控制新签发 token 的过期时间。",
},
{
"key": "node_region",
"label": "节点地域提示",
"type": "text",
"required": False,
"default": "",
"help": "可留空;留空时优先从托管节点或集群快照自动推断。",
},
{
"key": "node_role",
"label": "节点角色提示",
"type": "text",
"required": False,
"default": "",
"help": "可留空;留空时优先从托管节点或集群快照自动推断。",
},
],
},
],
},
{
"group_key": "release",
"group_title": "发布与变更",
"group_description": "适合把单节点灰度、点状修复和手工发布收编到标准 deploy.release 任务。",
"items": [
{
"key": "deploy.release.control",
"title": "发布控制面节点",
"action": "deploy.release",
"description": "面向 controller 节点的标准发布任务,默认会重启 API / Worker / Sync Agent 并执行 API 健康检查。",
"risk_level": "high",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "ssh"],
"target_roles": ["control"],
"default_auto_approve": False,
"fields": [
{
"key": "release_version",
"label": "版本号",
"type": "text",
"required": True,
"default": "",
"help": "例如 2026.04.18-rc1用于 releases/<version> 目录名。",
},
{
"key": "artifact_url",
"label": "发布包地址",
"type": "textarea",
"required": True,
"default": "",
"wide": True,
"rows": 2,
"help": "Node Agent 会在目标节点直接下载 tar.gz 发布包。",
},
{
"key": "checksum",
"label": "SHA256 校验",
"type": "text",
"required": False,
"default": "",
"wide": True,
"help": "可留空;填写后会强校验发布包完整性。",
},
{
"key": "install_root",
"label": "安装根目录",
"type": "text",
"required": True,
"default": "/opt/domaincheck",
"help": "目标节点上的 releases/current 根目录。",
},
{
"key": "restart_services",
"label": "重启服务",
"type": "text_list",
"required": False,
"default": [
settings.api_service_name,
settings.worker_service_name,
settings.sync_agent_service_name,
],
"wide": True,
"rows": 3,
"help": "每行一个服务。控制面默认重启 API / Worker / Sync Agent。",
},
{
"key": "health_check_urls",
"label": "健康检查 URL",
"type": "text_list",
"required": False,
"default": ["http://127.0.0.1:8100/health"],
"wide": True,
"rows": 2,
"help": "每行一个 URL。控制面默认探活本机 API /health。",
},
{
"key": "health_check_services",
"label": "健康检查服务",
"type": "text_list",
"required": False,
"default": [
settings.api_service_name,
settings.worker_service_name,
settings.sync_agent_service_name,
],
"wide": True,
"rows": 3,
"help": "每行一个 systemd 服务,发布后会检查其 active 状态。",
},
{
"key": "health_check_timeout_seconds",
"label": "URL 超时",
"type": "number",
"required": True,
"default": 10,
"min": 2,
"max": 120,
"help": "每次 URL 探活超时秒数。",
},
{
"key": "health_check_retries",
"label": "重试次数",
"type": "number",
"required": True,
"default": 2,
"min": 0,
"max": 10,
"help": "健康检查失败后的额外重试次数。",
},
{
"key": "health_check_interval_seconds",
"label": "重试间隔",
"type": "number",
"required": True,
"default": 2,
"min": 0,
"max": 60,
"help": "每次重试前的等待秒数。",
},
{
"key": "rollback_on_failure",
"label": "失败自动回滚",
"type": "boolean",
"required": True,
"default": True,
"help": "健康检查失败时自动切回旧 current 并重启服务。",
},
{
"key": "switch_current",
"label": "切换 current 软链",
"type": "boolean",
"required": True,
"default": True,
"help": "关闭后只解压版本目录,不切 current。",
},
],
},
{
"key": "deploy.release.worker",
"title": "发布 Worker 节点",
"action": "deploy.release",
"description": "面向独立 Worker 节点的标准发布任务,默认只重启 Worker 并校验 Worker 服务状态。",
"risk_level": "high",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "ssh"],
"target_roles": ["worker"],
"default_auto_approve": False,
"fields": [
{
"key": "release_version",
"label": "版本号",
"type": "text",
"required": True,
"default": "",
"help": "例如 2026.04.18-rc1用于 releases/<version> 目录名。",
},
{
"key": "artifact_url",
"label": "发布包地址",
"type": "textarea",
"required": True,
"default": "",
"wide": True,
"rows": 2,
"help": "Node Agent 会在目标节点直接下载 tar.gz 发布包。",
},
{
"key": "checksum",
"label": "SHA256 校验",
"type": "text",
"required": False,
"default": "",
"wide": True,
"help": "可留空;填写后会强校验发布包完整性。",
},
{
"key": "install_root",
"label": "安装根目录",
"type": "text",
"required": True,
"default": "/opt/domaincheck",
"help": "目标节点上的 releases/current 根目录。",
},
{
"key": "restart_services",
"label": "重启服务",
"type": "text_list",
"required": False,
"default": [settings.worker_service_name],
"wide": True,
"rows": 2,
"help": "每行一个服务。独立 Worker 默认只重启 Worker。",
},
{
"key": "health_check_urls",
"label": "健康检查 URL",
"type": "text_list",
"required": False,
"default": [],
"wide": True,
"rows": 2,
"help": "Worker 节点通常可留空;为空时不做 URL 探活。",
},
{
"key": "health_check_services",
"label": "健康检查服务",
"type": "text_list",
"required": False,
"default": [settings.worker_service_name],
"wide": True,
"rows": 2,
"help": "每行一个 systemd 服务,默认只校验 Worker active 状态。",
},
{
"key": "health_check_timeout_seconds",
"label": "URL 超时",
"type": "number",
"required": True,
"default": 10,
"min": 2,
"max": 120,
"help": "每次 URL 探活超时秒数。",
},
{
"key": "health_check_retries",
"label": "重试次数",
"type": "number",
"required": True,
"default": 2,
"min": 0,
"max": 10,
"help": "健康检查失败后的额外重试次数。",
},
{
"key": "health_check_interval_seconds",
"label": "重试间隔",
"type": "number",
"required": True,
"default": 2,
"min": 0,
"max": 60,
"help": "每次重试前的等待秒数。",
},
{
"key": "rollback_on_failure",
"label": "失败自动回滚",
"type": "boolean",
"required": True,
"default": True,
"help": "健康检查失败时自动切回旧 current 并重启服务。",
},
{
"key": "switch_current",
"label": "切换 current 软链",
"type": "boolean",
"required": True,
"default": True,
"help": "关闭后只解压版本目录,不切 current。",
},
],
},
{
"key": "deploy.release.custom",
"title": "发布自定义节点",
"action": "deploy.release",
"description": "完全自定义 deploy.release 参数,适合灰度验证、特殊节点或后续扩展场景。",
"risk_level": "high",
"default_execution_mode": "remote-agent",
"execution_modes": ["remote-agent", "ssh"],
"target_roles": ["control", "worker"],
"default_auto_approve": False,
"fields": [
{
"key": "release_version",
"label": "版本号",
"type": "text",
"required": True,
"default": "",
"help": "例如 2026.04.18-rc1用于 releases/<version> 目录名。",
},
{
"key": "artifact_url",
"label": "发布包地址",
"type": "textarea",
"required": True,
"default": "",
"wide": True,
"rows": 2,
"help": "Node Agent 会在目标节点直接下载 tar.gz 发布包。",
},
{
"key": "checksum",
"label": "SHA256 校验",
"type": "text",
"required": False,
"default": "",
"wide": True,
"help": "可留空;填写后会强校验发布包完整性。",
},
{
"key": "install_root",
"label": "安装根目录",
"type": "text",
"required": True,
"default": "/opt/domaincheck",
"help": "目标节点上的 releases/current 根目录。",
},
{
"key": "restart_services",
"label": "重启服务",
"type": "text_list",
"required": False,
"default": [],
"wide": True,
"rows": 3,
"help": "每行一个服务。留空表示只落盘版本,不主动重启任何服务。",
},
{
"key": "health_check_urls",
"label": "健康检查 URL",
"type": "text_list",
"required": False,
"default": [],
"wide": True,
"rows": 2,
"help": "每行一个 URL。留空表示不做 URL 探活。",
},
{
"key": "health_check_services",
"label": "健康检查服务",
"type": "text_list",
"required": False,
"default": [],
"wide": True,
"rows": 3,
"help": "每行一个 systemd 服务。显式留空时不会自动回落到默认 API 检查。",
},
{
"key": "health_check_timeout_seconds",
"label": "URL 超时",
"type": "number",
"required": True,
"default": 10,
"min": 2,
"max": 120,
"help": "每次 URL 探活超时秒数。",
},
{
"key": "health_check_retries",
"label": "重试次数",
"type": "number",
"required": True,
"default": 2,
"min": 0,
"max": 10,
"help": "健康检查失败后的额外重试次数。",
},
{
"key": "health_check_interval_seconds",
"label": "重试间隔",
"type": "number",
"required": True,
"default": 2,
"min": 0,
"max": 60,
"help": "每次重试前的等待秒数。",
},
{
"key": "rollback_on_failure",
"label": "失败自动回滚",
"type": "boolean",
"required": True,
"default": True,
"help": "健康检查失败时自动切回旧 current 并重启服务。",
},
{
"key": "switch_current",
"label": "切换 current 软链",
"type": "boolean",
"required": True,
"default": True,
"help": "关闭后只解压版本目录,不切 current。",
},
],
},
],
},
]
def get_ops_action_templates() -> dict:
groups = deepcopy(_template_catalog())
items: list[dict] = []
for group in groups:
group_key = str(group.get("group_key") or "").strip()
group_title = str(group.get("group_title") or "").strip()
group_description = str(group.get("group_description") or "").strip()
normalized_group_items: list[dict] = []
for item in list(group.get("items") or []):
normalized_item = decorate_execution_mode_fields(dict(item))
normalized_item["group_key"] = group_key
normalized_item["group_title"] = group_title
normalized_item["group_description"] = group_description
normalized_group_items.append(normalized_item)
items.append(normalized_item)
group["items"] = normalized_group_items
return {
"groups": groups,
"items": items,
"defaults": {
"requested_by": "web-ui",
"execution_mode": "remote-agent",
"execution_mode_label": execution_mode_label("remote-agent"),
},
}
def get_ops_action_template(template_key: str) -> dict:
normalized_key = str(template_key or "").strip()
if not normalized_key:
return {}
for group in _template_catalog():
group_key = str(group.get("group_key") or "").strip()
group_title = str(group.get("group_title") or "").strip()
group_description = str(group.get("group_description") or "").strip()
for item in list(group.get("items") or []):
if str(item.get("key") or "").strip() == normalized_key:
normalized_item = decorate_execution_mode_fields(deepcopy(item))
normalized_item["group_key"] = group_key
normalized_item["group_title"] = group_title
normalized_item["group_description"] = group_description
return normalized_item
return {}
def build_ops_template_payload(template_key: str, payload: dict | None = None) -> tuple[bool, str, dict]:
template = get_ops_action_template(template_key)
if not template:
return False, "动作模板不存在", {}
source_payload = dict(payload or {})
normalized_payload: dict = {}
for field in list(template.get("fields") or []):
field_key = str(field.get("key") or "").strip()
if not field_key:
continue
field_type = str(field.get("type") or "text").strip()
raw_value = source_payload.get(field_key, field.get("default"))
if field_type == "number":
try:
value = int(raw_value or 0)
except Exception:
return False, f"{field_key} 必须是数字", {}
min_value = field.get("min")
max_value = field.get("max")
if min_value is not None:
value = max(int(min_value), value)
if max_value is not None:
value = min(int(max_value), value)
elif field_type == "boolean":
value = _coerce_bool(raw_value)
elif field_type == "text_list":
value = _split_text_list(raw_value)
else:
value = str(raw_value or "").strip()
if bool(field.get("required", False)) and _is_missing_value(field_type, value):
return False, f"{field_key} 不能为空", {}
normalized_payload[field_key] = value
return True, "ok", normalized_payload

View File

@@ -7,6 +7,7 @@ from app.core.config import settings
from app.core.db import get_db
from app.core.files import read_json
from app.core.redis_client import get_redis
from app.services.build_info_service import get_runtime_build_info
from app.services.cluster_runtime_service import get_cluster_snapshot
from app.services.detect_service import get_detect_status
from app.services.detect_job_service import get_detect_capacity_plan, get_detect_queue_health
@@ -187,6 +188,89 @@ def _build_multi_region_readiness(
}
def _detect_participation_snapshot(*, row: dict) -> dict:
items_running = int(row.get("items_running", 0) or 0)
items_claimed = int(row.get("items_claimed", 0) or 0)
processed_recent = int(row.get("processed_recent", 0) or 0)
current_load = int(row.get("current_load", 0) or 0)
status = str(row.get("status") or "").strip().lower()
if items_running > 0:
return {
"participation_state": "running",
"participation_label": "执行中",
"participation_reason": f"当前正在执行 {items_running} 项检测任务。",
"is_current_participant": True,
"is_dispatch_active": True,
}
if items_claimed > 0:
return {
"participation_state": "claimed",
"participation_label": "已领待跑",
"participation_reason": f"已领取 {items_claimed} 项任务,等待线程继续执行。",
"is_current_participant": True,
"is_dispatch_active": True,
}
if processed_recent > 0:
return {
"participation_state": "recent_throughput",
"participation_label": "近窗有吞吐",
"participation_reason": f"近 15 分钟内已处理 {processed_recent} 项任务。",
"is_current_participant": True,
"is_dispatch_active": False,
}
if current_load > 0 or status == "busy":
load_value = max(current_load, 1)
return {
"participation_state": "load_syncing",
"participation_label": "负载待确认",
"participation_reason": f"节点当前负载为 {load_value},但还未观察到已领、执行中或近窗吞吐数据,先归入在线未参与观察。",
"is_current_participant": False,
"is_dispatch_active": False,
}
return {
"participation_state": "standby",
"participation_label": "在线待命",
"participation_reason": "当前未领任务、未执行任务,也没有近窗吞吐。",
"is_current_participant": False,
"is_dispatch_active": False,
}
def _build_detect_node_row(*, node_code: str, cluster_node: dict, job_node: dict, queue_node: dict) -> dict:
metadata = cluster_node.get("metadata") or {}
role = str(cluster_node.get("role") or job_node.get("role") or "worker")
region = str(cluster_node.get("region") or settings.node_region)
is_effective_worker = bool(cluster_node.get("is_effective_worker", False) or role == "worker")
items_total = int(job_node.get("items_total", metadata.get("job_items_total", 0)) or 0)
items_claimed = int(job_node.get("items_claimed", metadata.get("job_items_claimed", 0)) or 0)
items_running = int(job_node.get("items_running", metadata.get("job_items_running", 0)) or 0)
items_completed = int(job_node.get("items_completed", metadata.get("job_items_completed", 0)) or 0)
items_failed = int(job_node.get("items_failed", metadata.get("job_items_failed", 0)) or 0)
row = {
"node_code": node_code,
"role": role,
"region": region,
"status": str(cluster_node.get("status") or "unknown"),
"is_effective_worker": is_effective_worker,
"detect_participating": False,
"current_load": int(cluster_node.get("current_load", 0) or 0),
"items_total": items_total,
"items_pending": int(job_node.get("items_pending", max(items_total - items_claimed - items_completed - items_failed, 0)) or 0),
"items_claimed": items_claimed,
"items_running": items_running,
"items_completed": items_completed,
"items_failed": items_failed,
"processed_recent": int(queue_node.get("processed_recent", 0) or 0),
"processed_per_minute": float(queue_node.get("processed_per_minute", 0) or 0),
"last_heartbeat_at": str(cluster_node.get("last_heartbeat_at") or ""),
"metrics_source": "active_job" if bool(job_node) else ("cluster_metadata" if metadata else "derived"),
}
row.update(_detect_participation_snapshot(row=row))
row["detect_participating"] = bool(row.get("is_current_participant", False))
return row
def _build_participating_detect_nodes(*, cluster_snapshot: dict, detect_snapshot: dict, worker_runtime: dict) -> list[dict]:
cluster_nodes = list(cluster_snapshot.get("nodes") or [])
active_job = detect_snapshot.get("active_job") or {}
@@ -204,56 +288,25 @@ def _build_participating_detect_nodes(*, cluster_snapshot: dict, detect_snapshot
}
merged: list[dict] = []
seen: set[str] = set()
for item in list(active_job.get("node_stats") or []):
node_code = str(item.get("node_code") or "").strip()
if not node_code or node_code == "unassigned":
continue
seen.add(node_code)
job_map = {
str(item.get("node_code") or "").strip(): item
for item in list(active_job.get("node_stats") or [])
if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned"
}
candidate_codes = sorted(set(job_map) | set(queue_map) | set(cluster_map))
for node_code in candidate_codes:
cluster_node = cluster_map.get(node_code, {})
queue_node = queue_map.get(node_code, {})
merged.append(
{
"node_code": node_code,
"role": str(cluster_node.get("role") or "worker"),
"region": str(cluster_node.get("region") or settings.node_region),
"status": str(cluster_node.get("status") or "unknown"),
"is_effective_worker": bool(cluster_node.get("is_effective_worker", False) or str(cluster_node.get("role") or "") == "worker"),
"detect_participating": True,
"current_load": int(cluster_node.get("current_load", 0) or 0),
"items_total": int(item.get("items_total", 0) or 0),
"items_pending": int(item.get("items_pending", 0) or 0),
"items_claimed": int(item.get("items_claimed", 0) or 0),
"items_running": int(item.get("items_running", 0) or 0),
"items_completed": int(item.get("items_completed", 0) or 0),
"items_failed": int(item.get("items_failed", 0) or 0),
"processed_recent": int(queue_node.get("processed_recent", 0) or 0),
"processed_per_minute": float(queue_node.get("processed_per_minute", 0) or 0),
}
if cluster_node and not bool(cluster_node.get("is_effective_worker", False)):
continue
row = _build_detect_node_row(
node_code=node_code,
cluster_node=cluster_node,
job_node=job_map.get(node_code, {}),
queue_node=queue_map.get(node_code, {}),
)
if worker_runtime.get("running", False) and settings.node_code not in seen:
cluster_node = cluster_map.get(settings.node_code, {})
if cluster_node:
merged.append(
{
"node_code": settings.node_code,
"role": str(cluster_node.get("role") or settings.node_role),
"region": str(cluster_node.get("region") or settings.node_region),
"status": str(cluster_node.get("status") or "online"),
"is_effective_worker": True,
"detect_participating": True,
"current_load": int(cluster_node.get("current_load", 0) or 0),
"items_total": 0,
"items_pending": 0,
"items_claimed": 0,
"items_running": 0,
"items_completed": 0,
"items_failed": 0,
"processed_recent": 0,
"processed_per_minute": 0,
}
)
if not bool(row.get("is_current_participant", False)):
continue
merged.append(row)
return sorted(
merged,
@@ -261,11 +314,143 @@ def _build_participating_detect_nodes(*, cluster_snapshot: dict, detect_snapshot
-int(item.get("items_running", 0) or 0),
-int(item.get("items_claimed", 0) or 0),
-int(item.get("processed_recent", 0) or 0),
-int(item.get("current_load", 0) or 0),
str(item.get("node_code") or ""),
),
)
def _build_standby_detect_nodes(*, cluster_snapshot: dict, participating_nodes: list[dict]) -> list[dict]:
cluster_nodes = list(cluster_snapshot.get("nodes") or [])
participating_codes = {
str(item.get("node_code") or "").strip()
for item in list(participating_nodes or [])
if str(item.get("node_code") or "").strip()
}
standby_rows: list[dict] = []
for node in cluster_nodes:
node_code = str(node.get("node_code") or "").strip()
node_status = str(node.get("status") or "").strip()
node_current_load = int(node.get("current_load", 0) or 0)
metadata = node.get("metadata") or {}
if not node_code:
continue
if not bool(node.get("is_effective_worker", False)):
continue
if node_status not in {"online", "busy"}:
continue
if node_code in participating_codes:
continue
standby_state = "load_syncing" if node_current_load > 0 or node_status == "busy" else "standby"
standby_label = "负载待确认" if standby_state == "load_syncing" else "在线待命"
standby_reason = (
str(metadata.get("detail") or "").strip()
or str(metadata.get("phase_detail") or "").strip()
or (
f"当前阶段:{str(metadata.get('phase') or metadata.get('phase_label') or '').strip()}"
if str(metadata.get("phase") or metadata.get("phase_label") or "").strip()
else ""
)
or (
f"节点当前负载为 {node_current_load},但还未观察到已领、执行中或近窗吞吐数据。"
if standby_state == "load_syncing"
else "节点在线,当前未领任务、未执行任务,也没有近窗吞吐。"
)
)
standby_rows.append(
{
"node_code": node_code,
"role": str(node.get("role") or "worker"),
"region": str(node.get("region") or settings.node_region),
"status": node_status,
"is_effective_worker": True,
"detect_participating": False,
"participation_state": standby_state,
"participation_label": standby_label,
"participation_reason": standby_reason,
"current_load": node_current_load,
"last_heartbeat_at": str(node.get("last_heartbeat_at") or ""),
"phase": str(metadata.get("phase") or metadata.get("phase_label") or "").strip(),
"detail": str(metadata.get("detail") or metadata.get("phase_detail") or "").strip(),
"worker_online": bool(metadata.get("worker_online", False)),
"standby_reason": standby_reason,
}
)
return sorted(
standby_rows,
key=lambda item: (
str(item.get("status") or ""),
str(item.get("role") or ""),
str(item.get("node_code") or ""),
),
)
def _build_detect_participation_summary(
*,
participating_nodes: list[dict],
standby_nodes: list[dict],
cluster_snapshot: dict,
) -> dict:
cluster_nodes = list(cluster_snapshot.get("nodes") or [])
effective_online_nodes = [
node
for node in cluster_nodes
if bool(node.get("is_effective_worker", False)) and str(node.get("status") or "").strip() in {"online", "busy"}
]
dispatch_active_nodes = [
row for row in participating_nodes
if bool(row.get("is_dispatch_active", False))
]
recent_only_nodes = [
row for row in participating_nodes
if str(row.get("participation_state") or "").strip() == "recent_throughput"
]
load_syncing_nodes = [
row for row in standby_nodes
if str(row.get("participation_state") or "").strip() == "load_syncing"
]
pure_standby_nodes = [
row for row in standby_nodes
if str(row.get("participation_state") or "").strip() == "standby"
]
dedicated_worker_nodes = [
node for node in effective_online_nodes
if str(node.get("role") or "").strip() == "worker"
]
controller_worker_nodes = [
node for node in effective_online_nodes
if str(node.get("role") or "").strip() == "control"
]
summary_parts = [
f"有效执行节点 {len(effective_online_nodes)}",
f"正在执行/领任务 {len(dispatch_active_nodes)}",
f"近窗刚有吞吐 {len(recent_only_nodes)}",
f"在线但未参与 {len(standby_nodes)}",
]
if load_syncing_nodes:
summary_parts.append(f"其中负载待确认 {len(load_syncing_nodes)}")
return {
"effective_online_nodes": len(effective_online_nodes),
"participating_nodes": len(participating_nodes),
"dispatch_active_nodes": len(dispatch_active_nodes),
"recent_only_nodes": len(recent_only_nodes),
"non_participating_nodes": len(standby_nodes),
"standby_nodes": len(pure_standby_nodes),
"load_syncing_nodes": len(load_syncing_nodes),
"dedicated_worker_nodes": len(dedicated_worker_nodes),
"controller_worker_nodes": len(controller_worker_nodes),
"dispatch_active_node_codes": [str(item.get("node_code") or "") for item in dispatch_active_nodes],
"recent_only_node_codes": [str(item.get("node_code") or "") for item in recent_only_nodes],
"non_participating_node_codes": [str(item.get("node_code") or "") for item in standby_nodes],
"standby_node_codes": [str(item.get("node_code") or "") for item in pure_standby_nodes],
"load_syncing_node_codes": [str(item.get("node_code") or "") for item in load_syncing_nodes],
"summary": "".join(summary_parts),
}
def get_runtime_status() -> dict:
runtime_settings = get_runtime_settings()
worker_runtime = detect_worker_runtime()
@@ -314,6 +499,17 @@ def get_runtime_status() -> dict:
"worker_mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
"queue_health": queue_health,
"capacity_plan": capacity_plan,
"log_sync": {
"enabled": bool(runtime_settings.get("worker_log_sync_enabled", False)),
"mode": str(runtime_settings.get("worker_log_sync_mode", "key") or "key"),
"line_count": int(detect_snapshot.get("remote_log_line_count", 0) or 0),
"source_node_count": int(detect_snapshot.get("remote_log_node_count", 0) or 0),
"source_nodes": list(detect_snapshot.get("remote_log_nodes") or []),
"source_node_summaries": list(detect_snapshot.get("remote_log_node_summaries") or []),
"last_at": str(detect_snapshot.get("remote_log_last_at") or ""),
"last_line": str(detect_snapshot.get("remote_log_last_line") or ""),
"preview_lines": list(detect_snapshot.get("remote_log_lines") or [])[-20:],
},
}
if not worker_expected_on_this_node:
detect_payload.update(
@@ -349,6 +545,16 @@ def get_runtime_status() -> dict:
detect_snapshot=detect_payload,
worker_runtime=worker_runtime,
)
detect_payload["standby_nodes"] = _build_standby_detect_nodes(
cluster_snapshot=cluster_snapshot,
participating_nodes=detect_payload["participating_nodes"],
)
detect_payload["non_participating_nodes"] = list(detect_payload["standby_nodes"] or [])
detect_payload["participation_summary"] = _build_detect_participation_summary(
participating_nodes=detect_payload["participating_nodes"],
standby_nodes=detect_payload["non_participating_nodes"],
cluster_snapshot=cluster_snapshot,
)
append_runtime_projection_if_changed(detect=detect_payload, cluster=cluster_snapshot)
sync_summary = get_sync_summary(record_limit=5)
readiness = _build_multi_region_readiness(
@@ -357,6 +563,7 @@ def get_runtime_status() -> dict:
worker_runtime=worker_runtime,
sync_agent_runtime=sync_agent_runtime,
)
build_info = get_runtime_build_info()
return {
"api": {
@@ -371,6 +578,7 @@ def get_runtime_status() -> dict:
"health_url": f"http://127.0.0.1:{settings.api_port}/health",
"stdout_log": _runtime_log_path("domain-api.stdout.log"),
"stderr_log": _runtime_log_path("domain-api.stderr.log"),
"build": build_info,
},
"node": {
"code": settings.node_code,

View File

@@ -450,6 +450,17 @@ def append_runtime_projection_if_changed(
continue
local_participating = bool(node.get("detect_participating", False) or node.get("current_load", 0))
break
local_job_bucket = {}
for item in list(active_job.get("node_stats") or []):
if str(item.get("node_code") or "").strip() != settings.node_code:
continue
local_job_bucket = item
break
if not local_participating:
local_participating = bool(
int(local_job_bucket.get("items_running", 0) or 0) > 0
or int(local_job_bucket.get("items_claimed", 0) or 0) > 0
)
projection = {
"node": {
"node_code": settings.node_code,

File diff suppressed because it is too large Load Diff

View File

@@ -12,7 +12,7 @@ TEMPLATE_DIR="$API_DIR/deploy/multi-region/templates"
if [ "$ROLE" != "controller" ] && [ "$ROLE" != "worker" ]; then
echo "invalid role: $ROLE"
echo "usage: bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck [controller|worker]"
echo "usage: bash domain-api/deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck [controller|worker]"
exit 1
fi

View File

@@ -0,0 +1,61 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT_DIR="${1:-/opt/domaincheck}"
ENV_FILE="/etc/default/domaincheck-node-agent"
SERVICE_FILE="/etc/systemd/system/domaincheck-node-agent.service"
SERVICE_NAME="domaincheck-node-agent"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
resolve_project_root() {
local candidate
for candidate in \
"${ROOT_DIR}/domainCheck" \
"${ROOT_DIR}/domain-api" \
"${ROOT_DIR}" \
"$(cd "${SCRIPT_DIR}/../.." && pwd)"; do
if [[ -f "${candidate}/deploy/systemd/domain-node-agent.service" && -f "${candidate}/deploy/multi-region/templates/domaincheck-node-agent.env.example" ]]; then
printf '%s' "${candidate}"
return 0
fi
done
return 1
}
PROJECT_ROOT="$(resolve_project_root || true)"
if [[ -z "${PROJECT_ROOT}" ]]; then
echo "unable to resolve project root under ${ROOT_DIR} or ${SCRIPT_DIR}" >&2
exit 1
fi
TEMPLATE_FILE="${PROJECT_ROOT}/deploy/multi-region/templates/domaincheck-node-agent.env.example"
UNIT_TEMPLATE="${PROJECT_ROOT}/deploy/systemd/domain-node-agent.service"
echo "[1/5] install systemd unit"
install -m 0644 "${UNIT_TEMPLATE}" "${SERVICE_FILE}"
echo "[2/5] install env template if absent"
if [[ ! -f "${ENV_FILE}" ]]; then
install -m 0644 "${TEMPLATE_FILE}" "${ENV_FILE}"
echo "created ${ENV_FILE}"
else
echo "keep existing ${ENV_FILE}"
fi
echo "[3/5] reload systemd"
systemctl daemon-reload
echo "[4/5] enable service"
systemctl enable "${SERVICE_NAME}" >/dev/null
echo "[5/5] next steps"
echo "project_root=${PROJECT_ROOT}"
echo "edit ${ENV_FILE} and set:"
echo " OPS_CONTROL_PLANE_BASE_URL"
echo " OPS_AGENT_TOKEN"
echo " NODE_CODE / NODE_REGION / NODE_ROLE"
echo
echo "then run:"
echo " systemctl restart ${SERVICE_NAME}"
echo " systemctl status ${SERVICE_NAME} --no-pager -l"

View File

@@ -0,0 +1,91 @@
#!/usr/bin/env bash
set -euo pipefail
API_BASE_URL="${1:-http://127.0.0.1:8100}"
NODE_CODE="${2:-}"
NODE_REGION="${3:-mainland}"
NODE_ROLE="${4:-worker}"
CONTROL_PLANE_BASE_URL="${5:-${API_BASE_URL}}"
ROOT_DIR="${6:-/opt/domaincheck}"
ISSUED_BY="${ISSUED_BY:-cli-bootstrap}"
EXPIRES_IN_HOURS="${EXPIRES_IN_HOURS:-72}"
if [[ -z "${NODE_CODE}" ]]; then
echo "usage: $0 <api_base_url> <node_code> [node_region] [node_role] [control_plane_base_url] [root_dir]" >&2
exit 1
fi
NORMALIZED_API_BASE_URL="${API_BASE_URL%/}"
NORMALIZED_CONTROL_PLANE_BASE_URL="${CONTROL_PLANE_BASE_URL%/}"
if [[ "${NORMALIZED_API_BASE_URL}" != */api/v1 ]]; then
API_ENDPOINT="${NORMALIZED_API_BASE_URL}/api/v1/ops/agent/bootstrap-plan"
else
API_ENDPOINT="${NORMALIZED_API_BASE_URL}/ops/agent/bootstrap-plan"
fi
TMP_RESPONSE_FILE="$(mktemp)"
cleanup() {
rm -f "${TMP_RESPONSE_FILE}"
}
trap cleanup EXIT
python3 - "${NODE_CODE}" "${NODE_REGION}" "${NODE_ROLE}" "${ISSUED_BY}" "${EXPIRES_IN_HOURS}" "${NORMALIZED_CONTROL_PLANE_BASE_URL}" "${ROOT_DIR}" >"${TMP_RESPONSE_FILE}.payload" <<'PY'
import json
import sys
payload = {
"node_code": sys.argv[1],
"node_region": sys.argv[2],
"node_role": sys.argv[3],
"issued_by": sys.argv[4],
"expires_in_hours": int(sys.argv[5]),
"control_plane_base_url": sys.argv[6],
"root_dir": sys.argv[7],
"metadata": {
"issued_from": "build_node_agent_bootstrap_plan.sh",
},
}
print(json.dumps(payload, ensure_ascii=False))
PY
curl -fsS -X POST \
-H 'Content-Type: application/json' \
--data-binary @"${TMP_RESPONSE_FILE}.payload" \
"${API_ENDPOINT}" >"${TMP_RESPONSE_FILE}"
python3 - "${TMP_RESPONSE_FILE}" <<'PY'
import json
import sys
from pathlib import Path
response = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
response_code = response.get("code", 0)
if int(response_code if response_code is not None else 0) != 0:
raise SystemExit(response.get("message") or "bootstrap plan request failed")
data = response.get("data") or {}
plan = data.get("bootstrap_plan") or {}
print("[token]")
print(f"node_code={data.get('node_code', '')}")
print(f"token_preview={data.get('token_preview', '')}")
print(f"expires_at={data.get('expires_at', '')}")
print()
print("[env]")
print(plan.get("env_content") or "")
print()
print("[commands]")
print(plan.get("command_block") or "")
print()
print("[bootstrap-script]")
print(plan.get("bootstrap_script_content") or "")
print()
print("[bootstrap-run-block]")
print(plan.get("bootstrap_run_script_block") or "")
print()
print("[health-checks]")
for item in plan.get("health_checks") or []:
print(item)
PY
rm -f "${TMP_RESPONSE_FILE}.payload"

View File

@@ -0,0 +1,345 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
API_SERVICE="${API_SERVICE:-domaincheck-api}"
WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}"
SYNC_AGENT_SERVICE="${SYNC_AGENT_SERVICE:-domaincheck-sync-agent}"
NODE_AGENT_SERVICE="${NODE_AGENT_SERVICE:-domaincheck-node-agent}"
API_ENV_FILE="${API_ENV_FILE:-/etc/default/domaincheck-api}"
WORKER_ENV_FILE="${WORKER_ENV_FILE:-/etc/default/domaincheck-worker}"
NODE_AGENT_ENV_FILE="${NODE_AGENT_ENV_FILE:-/etc/default/domaincheck-node-agent}"
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
API_DIR="${PROJECT_ROOT}/domain-api"
WEB_DIR="${PROJECT_ROOT}/domain-web"
PYTHON_BIN="${PYTHON_BIN:-python3}"
FETCH_BODY=""
FETCH_STATUS="000"
fetch_json() {
local url="${1:-}"
local raw_text
raw_text="$(curl -sS \
--connect-timeout "${CURL_CONNECT_TIMEOUT:-3}" \
--max-time "${CURL_MAX_TIME:-10}" \
"${url}" \
-w $'\nHTTP_STATUS=%{http_code}' || true)"
FETCH_BODY="${raw_text%$'\n'HTTP_STATUS=*}"
FETCH_STATUS="${raw_text##*$'\n'HTTP_STATUS=}"
}
echo "[1/8] local command availability"
for cmd in bash curl systemctl "${PYTHON_BIN}" node npm; do
if command -v "${cmd}" >/dev/null 2>&1; then
printf '%s: ok -> %s\n' "${cmd}" "$(command -v "${cmd}")"
else
printf '%s: missing\n' "${cmd}"
fi
done
echo
echo "[2/8] python environment"
"${PYTHON_BIN}" - <<'PY' "${API_DIR}" "${WEB_DIR}"
import importlib.util
import json
import pathlib
import sys
api_dir = pathlib.Path(sys.argv[1])
web_dir = pathlib.Path(sys.argv[2])
def has_module(name: str) -> bool:
return importlib.util.find_spec(name) is not None
payload = {
"python_executable": sys.executable,
"python_version": sys.version.split()[0],
"pytest_installed": has_module("pytest"),
"uvicorn_installed": has_module("uvicorn"),
"fastapi_installed": has_module("fastapi"),
"api_requirements_exists": (api_dir / "requirements.txt").exists(),
"web_package_json_exists": (web_dir / "package.json").exists(),
}
print(json.dumps(payload, ensure_ascii=False, indent=2))
PY
echo
echo "[3/8] env files"
for env_file in "${API_ENV_FILE}" "${WORKER_ENV_FILE}" "${NODE_AGENT_ENV_FILE}"; do
if [[ -f "${env_file}" ]]; then
echo "exists: ${env_file}"
else
echo "missing: ${env_file}"
fi
done
echo
echo "[4/8] systemd service states"
for service in "${API_SERVICE}" "${WORKER_SERVICE}" "${SYNC_AGENT_SERVICE}" "${NODE_AGENT_SERVICE}"; do
if systemctl list-unit-files "${service}.service" --no-pager >/dev/null 2>&1; then
printf '%s: enabled=%s active=%s\n' \
"${service}" \
"$(systemctl is-enabled "${service}" 2>/dev/null || true)" \
"$(systemctl is-active "${service}" 2>/dev/null || true)"
else
printf '%s: not-installed\n' "${service}"
fi
done
echo
echo "[5/8] runtime preflight"
fetch_json "${BASE_URL}/api/v1/runtime/preflight"
printf 'status=%s\n%s\n\n' "${FETCH_STATUS}" "${FETCH_BODY}"
echo "[6/8] runtime readiness"
fetch_json "${BASE_URL}/api/v1/runtime/readiness"
printf 'status=%s\n%s\n\n' "${FETCH_STATUS}" "${FETCH_BODY}"
echo "[7/8] runtime build-info"
fetch_json "${BASE_URL}/api/v1/runtime/build-info"
printf 'status=%s\n%s\n\n' "${FETCH_STATUS}" "${FETCH_BODY}"
echo "[8/8] condensed env audit summary"
"${PYTHON_BIN}" - <<'PY' \
"${BASE_URL}" \
"${API_SERVICE}" \
"${WORKER_SERVICE}" \
"${SYNC_AGENT_SERVICE}" \
"${NODE_AGENT_SERVICE}" \
"${API_ENV_FILE}" \
"${WORKER_ENV_FILE}" \
"${NODE_AGENT_ENV_FILE}" \
"${PROJECT_ROOT}"
import importlib.util
import json
import pathlib
import subprocess
import sys
import urllib.request
from typing import Dict, List
base_url, api_service, worker_service, sync_agent_service, node_agent_service, api_env, worker_env, node_agent_env, project_root = sys.argv[1:10]
project_root_path = pathlib.Path(project_root)
def has_module(name: str) -> bool:
return importlib.util.find_spec(name) is not None
def inspect_python_runtime(path: str) -> Dict[str, object]:
runtime_path = pathlib.Path(path)
if not runtime_path.exists():
return {}
probe = r"""
import importlib.util
import json
import sys
mods = ["pytest", "fastapi", "uvicorn"]
print(json.dumps({
"python_executable": sys.executable,
"python_version": sys.version.split()[0],
"modules": {name: importlib.util.find_spec(name) is not None for name in mods},
}, ensure_ascii=False))
"""
try:
raw = subprocess.check_output([str(runtime_path), "-c", probe], text=True, stderr=subprocess.DEVNULL)
except Exception:
return {}
try:
payload = json.loads(raw)
except Exception:
return {}
payload["path"] = str(runtime_path)
return payload
def systemd_state(service: str) -> dict:
try:
enabled = subprocess.check_output(
["systemctl", "is-enabled", service],
text=True,
stderr=subprocess.DEVNULL,
).strip()
except Exception:
enabled = ""
try:
active = subprocess.check_output(
["systemctl", "is-active", service],
text=True,
stderr=subprocess.DEVNULL,
).strip()
except Exception:
active = ""
return {"enabled": enabled, "active": active}
def fetch_json(url: str) -> tuple[str, dict]:
try:
with urllib.request.urlopen(url, timeout=10) as resp:
status = str(resp.status)
raw = resp.read().decode("utf-8", errors="replace")
except Exception:
return "000", {}
try:
payload = json.loads(raw)
except Exception:
return status, {}
return status, payload.get("data") if isinstance(payload, dict) else {}
def read_git_value(*args: str) -> str:
try:
result = subprocess.check_output(
["git", "-C", str(project_root_path), *args],
text=True,
stderr=subprocess.DEVNULL,
).strip()
except Exception:
return ""
return result
def read_text(path: pathlib.Path) -> str:
try:
return path.read_text(encoding="utf-8", errors="replace")
except Exception:
return ""
local_commit_sha = read_git_value("rev-parse", "--short=12", "HEAD")
local_commit_ref = read_git_value("rev-parse", "--abbrev-ref", "HEAD")
ops_agent_service_text = read_text(project_root_path / "domain-api" / "app" / "services" / "ops_agent_service.py")
bootstrap_node_agent_text = read_text(project_root_path / "domain-api" / "deploy" / "multi-region" / "bootstrap_node_agent.sh")
repo_supports_install_command_block = (
"install_command_block" in ops_agent_service_text
and "_candidate_node_agent_install_paths" in ops_agent_service_text
)
repo_supports_multi_layout_bootstrap = "resolve_project_root" in bootstrap_node_agent_text
preflight_status, preflight = fetch_json(f"{base_url}/api/v1/runtime/preflight")
readiness_status, readiness = fetch_json(f"{base_url}/api/v1/runtime/readiness")
build_status, build = fetch_json(f"{base_url}/api/v1/runtime/build-info")
route_surface = build.get("route_surface") if isinstance(build, dict) else {}
missing = []
tooling_items: List[str] = []
for env_file in [api_env, worker_env, node_agent_env]:
if not pathlib.Path(env_file).exists():
missing.append(f"missing_env:{env_file}")
python_candidates: List[str] = []
seen_candidates = set()
for candidate in [
"/opt/domaincheck/domainCheck/.venv/bin/python",
str(pathlib.Path.cwd() / ".venv" / "bin" / "python"),
str(pathlib.Path.cwd() / "domain-api" / ".venv" / "bin" / "python"),
pathlib.Path(sys.executable).as_posix(),
]:
if candidate and candidate not in seen_candidates:
seen_candidates.add(candidate)
python_candidates.append(candidate)
python_runtimes = [item for item in (inspect_python_runtime(candidate) for candidate in python_candidates) if item]
def runtime_has_module(module_name: str) -> bool:
for runtime in python_runtimes:
modules = runtime.get("modules") or {}
if bool(modules.get(module_name)):
return True
return False
if not runtime_has_module("fastapi"):
missing.append("python_module:fastapi")
if not runtime_has_module("uvicorn"):
missing.append("python_module:uvicorn")
if not runtime_has_module("pytest"):
tooling_items.append("python_module:pytest")
runtime_commit_sha = str(build.get("commit_sha") or "").strip() if isinstance(build, dict) else ""
runtime_commit_ref = str(build.get("commit_ref") or "").strip() if isinstance(build, dict) else ""
runtime_route_missing_keys = [str(item).strip() for item in list((route_surface or {}).get("missing_keys") or []) if str(item).strip()]
runtime_expected_paths = {
str(key).strip(): str(value).strip()
for key, value in dict((route_surface or {}).get("expected_paths") or {}).items()
if str(key).strip()
}
runtime_bootstrap_plan_route_missing = "ops_node_handover_bootstrap_plan" in runtime_route_missing_keys
runtime_declares_bootstrap_plan_route = "ops_node_handover_bootstrap_plan" in runtime_expected_paths
runtime_repo_commit_drift = bool(local_commit_sha and runtime_commit_sha and local_commit_sha != runtime_commit_sha)
runtime_repo_capability_drift = bool(
repo_supports_install_command_block
and (runtime_bootstrap_plan_route_missing or not runtime_declares_bootstrap_plan_route)
)
recommended_actions: List[str] = []
if runtime_repo_capability_drift:
recommended_actions.append("systemctl restart domaincheck-api")
recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary")
recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-worker-01")
elif runtime_repo_commit_drift:
recommended_actions.append("systemctl restart domaincheck-api")
recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100")
payload = {
"status": "ready",
"headline": "当前环境没有发现新的基础阻断,可继续做上线复核。",
"repo": {
"project_root": str(project_root_path),
"commit_sha": local_commit_sha,
"commit_ref": local_commit_ref,
"supports_install_command_block": repo_supports_install_command_block,
"supports_multi_layout_bootstrap": repo_supports_multi_layout_bootstrap,
},
"python": {
"pytest_installed": runtime_has_module("pytest"),
"fastapi_installed": runtime_has_module("fastapi"),
"uvicorn_installed": runtime_has_module("uvicorn"),
"runtimes": python_runtimes,
},
"services": {
api_service: systemd_state(api_service),
worker_service: systemd_state(worker_service),
sync_agent_service: systemd_state(sync_agent_service),
node_agent_service: systemd_state(node_agent_service),
},
"runtime": {
"preflight_status_code": preflight_status,
"preflight_ok": bool((preflight or {}).get("ok", False)),
"readiness_status_code": readiness_status,
"readiness_status": str((readiness or {}).get("status") or ""),
"readiness_summary": str((readiness or {}).get("summary") or ""),
"build_status_code": build_status,
"build_commit_sha": runtime_commit_sha,
"build_commit_ref": runtime_commit_ref,
"route_surface_complete": bool((route_surface or {}).get("surface_complete", False)),
"route_surface_missing_keys": runtime_route_missing_keys,
"route_surface_declares_bootstrap_plan": runtime_declares_bootstrap_plan_route,
"repo_commit_drift": runtime_repo_commit_drift,
"repo_capability_drift": runtime_repo_capability_drift,
"runtime_may_need_restart": runtime_repo_commit_drift or runtime_repo_capability_drift,
},
"missing_items": missing,
"tooling_items": tooling_items,
"recommended_actions": recommended_actions,
}
if not bool((preflight or {}).get("ok", False)):
payload["status"] = "blocked"
payload["headline"] = "runtime/preflight 未通过,先修环境与依赖,再继续上线复核。"
elif str((readiness or {}).get("status") or "") == "blocking":
payload["status"] = "blocked"
payload["headline"] = "runtime/readiness 当前为 blocking现场仍不适合进入正式发布。"
elif runtime_repo_capability_drift:
payload["status"] = "attention"
payload["headline"] = "本地仓库已经具备新的 Node Agent 接管能力,但运行中的 API 路由面仍旧,当前更像是服务未重启到最新代码。"
elif runtime_repo_commit_drift:
payload["status"] = "attention"
payload["headline"] = "本地仓库 commit 与运行中 API build-info 不一致,建议先重启或重发当前版本,再继续上线复核。"
elif not bool((route_surface or {}).get("surface_complete", False)):
payload["status"] = "attention"
payload["headline"] = "运行中 API 路由面还不完整,建议先确认当前实例是否已经吃到最新版本。"
elif missing:
payload["status"] = "attention"
payload["headline"] = "基础环境存在缺口,虽然未必阻断运行,但建议先补齐后再交付。"
elif tooling_items:
payload["status"] = "ready"
payload["headline"] = "运行环境可继续上线复核,但本机工具链仍有缺口,建议后续补齐。"
print(json.dumps(payload, ensure_ascii=False, indent=2))
PY

View File

@@ -0,0 +1,423 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
OVERSEAS_BASE_URL="${2:-}"
VIEW_MODE="${3:-full}"
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
CURL_MAX_TIME="${CURL_MAX_TIME:-20}"
LOCAL_NODE_AGENT_SERVICE_NAME="${LOCAL_NODE_AGENT_SERVICE_NAME:-domaincheck-node-agent}"
LOCAL_NODE_AGENT_ENV_FILE="${LOCAL_NODE_AGENT_ENV_FILE:-/etc/default/domaincheck-node-agent}"
FETCH_BODY=""
FETCH_STATUS="000"
fetch_json() {
local url="${1:-}"
local raw_text
raw_text="$(curl -sS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
"${url}" \
-w $'\nHTTP_STATUS=%{http_code}' || true)"
FETCH_BODY="${raw_text%$'\n'HTTP_STATUS=*}"
FETCH_STATUS="${raw_text##*$'\n'HTTP_STATUS=}"
}
run_local_node_agent_probe() {
SERVICE_NAME="${LOCAL_NODE_AGENT_SERVICE_NAME}" \
ENV_FILE="${LOCAL_NODE_AGENT_ENV_FILE}" \
bash "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/check_node_agent.sh"
}
fetch_json "${BASE_URL}/api/v1/ops/stack-diagnosis?base_url=${BASE_URL}"
STACK_JSON="${FETCH_BODY}"
STACK_STATUS="${FETCH_STATUS}"
fetch_json "${BASE_URL}/api/v1/ops/go-live-summary?base_url=${BASE_URL}"
GO_LIVE_API_JSON="${FETCH_BODY}"
GO_LIVE_API_STATUS="${FETCH_STATUS}"
if [[ "${GO_LIVE_API_STATUS}" == 2* ]]; then
if [[ "${VIEW_MODE}" != "summary" ]]; then
echo "[1/1] go-live summary (api)"
fi
printf '%s\n' "${GO_LIVE_API_JSON}"
exit 0
fi
fetch_json "${BASE_URL}/api/v1/ops/contracts"
CONTRACTS_JSON="${FETCH_BODY}"
CONTRACTS_STATUS="${FETCH_STATUS}"
fetch_json "${BASE_URL}/api/v1/ops/releases/launchpad"
LAUNCHPAD_JSON="${FETCH_BODY}"
LAUNCHPAD_STATUS="${FETCH_STATUS}"
fetch_json "${BASE_URL}/api/v1/runtime/build-info"
BUILD_INFO_JSON="${FETCH_BODY}"
BUILD_INFO_STATUS="${FETCH_STATUS}"
fetch_json "${BASE_URL}/api/v1/ops/nodes"
NODES_JSON="${FETCH_BODY}"
NODES_STATUS="${FETCH_STATUS}"
fetch_json "${BASE_URL}/api/v1/ops/overview"
OVERVIEW_JSON="${FETCH_BODY}"
OVERVIEW_STATUS="${FETCH_STATUS}"
LOCAL_NODE_AGENT_RAW="$(run_local_node_agent_probe 2>&1 || true)"
if [[ "${VIEW_MODE}" != "summary" ]]; then
echo "[1/7] ops stack diagnosis"
printf 'status=%s\n%s\n\n' "${STACK_STATUS}" "${STACK_JSON}"
echo "[2/7] ops contracts"
printf 'status=%s\n%s\n\n' "${CONTRACTS_STATUS}" "${CONTRACTS_JSON}"
echo "[3/7] release launchpad"
printf 'status=%s\n%s\n\n' "${LAUNCHPAD_STATUS}" "${LAUNCHPAD_JSON}"
echo "[4/7] runtime build info"
printf 'status=%s\n%s\n\n' "${BUILD_INFO_STATUS}" "${BUILD_INFO_JSON}"
echo "[5/7] managed nodes"
printf 'status=%s\n%s\n\n' "${NODES_STATUS}" "${NODES_JSON}"
echo "[6/7] ops overview"
printf 'status=%s\n%s\n\n' "${OVERVIEW_STATUS}" "${OVERVIEW_JSON}"
echo "[7/7] local node agent probe"
printf '%s\n\n' "${LOCAL_NODE_AGENT_RAW}"
fi
echo "[8/8] go-live summary"
python3 - <<'PY' \
"${BASE_URL}" \
"${OVERSEAS_BASE_URL}" \
"${STACK_JSON}" \
"${STACK_STATUS}" \
"${CONTRACTS_JSON}" \
"${CONTRACTS_STATUS}" \
"${LAUNCHPAD_JSON}" \
"${LAUNCHPAD_STATUS}" \
"${BUILD_INFO_JSON}" \
"${BUILD_INFO_STATUS}" \
"${NODES_JSON}" \
"${NODES_STATUS}" \
"${OVERVIEW_JSON}" \
"${OVERVIEW_STATUS}" \
"${LOCAL_NODE_AGENT_RAW}" \
"${LOCAL_NODE_AGENT_SERVICE_NAME}"
import json
import sys
def load_payload(raw_text: str) -> dict:
try:
payload = json.loads(str(raw_text or ""))
except Exception:
return {}
return payload if isinstance(payload, dict) else {}
def load_data(raw_text: str) -> dict:
payload = load_payload(raw_text)
data = payload.get("data")
return data if isinstance(data, dict) else {}
def ensure_dict(value):
return value if isinstance(value, dict) else {}
def ensure_list(value):
return value if isinstance(value, list) else []
def ensure_str(value):
return value if isinstance(value, str) else ""
def truthy_http(status_code: str) -> bool:
return str(status_code or "").startswith("2")
base_url = str(sys.argv[1] or "http://127.0.0.1:8100").strip()
overseas_base_url = str(sys.argv[2] or "").strip()
stack_payload = load_data(sys.argv[3])
stack_status_code = str(sys.argv[4] or "000")
contracts_payload = load_data(sys.argv[5])
contracts_status_code = str(sys.argv[6] or "000")
launchpad_payload = load_data(sys.argv[7])
launchpad_status_code = str(sys.argv[8] or "000")
build_payload = load_data(sys.argv[9])
build_status_code = str(sys.argv[10] or "000")
nodes_payload = load_data(sys.argv[11])
nodes_status_code = str(sys.argv[12] or "000")
overview_payload = load_data(sys.argv[13])
overview_status_code = str(sys.argv[14] or "000")
local_node_agent_raw = str(sys.argv[15] or "")
local_node_agent_service_name = str(sys.argv[16] or "domaincheck-node-agent").strip()
diagnosis = ensure_dict(stack_payload.get("diagnosis"))
release_hub = ensure_dict(stack_payload.get("release_hub"))
managed_nodes = ensure_dict(stack_payload.get("managed_nodes"))
runtime_build_info = ensure_dict(stack_payload.get("runtime_build_info")) or build_payload
launchpad_status = ensure_dict(launchpad_payload.get("launchpad_status"))
launchpad_recommended_target_node_code = ensure_str(launchpad_status.get("recommended_target_node_code"))
launchpad_recommended_recovery_label = ensure_str(launchpad_status.get("recommended_recovery_label"))
launchpad_recommended_recovery_summary = ensure_str(launchpad_status.get("recommended_recovery_summary"))
launchpad_onboarding_bootstrap_pending_nodes = int(
launchpad_status.get("onboarding_bootstrap_pending_nodes", 0) or 0
)
launchpad_onboarding_acceptance_ready_nodes = int(
launchpad_status.get("onboarding_acceptance_ready_nodes", 0) or 0
)
contracts = ensure_list(contracts_payload.get("contracts"))
nodes_summary = ensure_dict(nodes_payload.get("summary")) or ensure_dict(managed_nodes.get("summary"))
route_surface = ensure_dict(runtime_build_info.get("route_surface"))
next_step = ensure_dict(diagnosis.get("next_step"))
operator_decision = ensure_dict(diagnosis.get("operator_decision"))
overview_recommendation = ensure_dict(overview_payload.get("recommendation"))
execution_scene = ensure_dict(overview_payload.get("execution_scene"))
log_sync = ensure_dict(execution_scene.get("log_sync"))
local_agent_enabled = ""
local_agent_active = ""
local_agent_installed = False
for line in [item.strip() for item in local_node_agent_raw.splitlines() if item.strip()]:
if line == "not-installed":
local_agent_installed = False
if line == "enabled":
local_agent_enabled = "enabled"
local_agent_installed = True
elif line == "disabled":
local_agent_enabled = "disabled"
local_agent_installed = True
elif line in {"active", "inactive", "failed"}:
local_agent_active = line
local_agent_installed = True
blocking_reasons = []
warnings = []
publish_blocking_reasons = []
publish_warnings = []
stack_status = ensure_str(diagnosis.get("stack_status")) or "unknown"
if stack_status == "blocked":
blocking_reasons.append("stack_diagnosis=blocked")
publish_blocking_reasons.append("stack_diagnosis=blocked")
elif stack_status == "attention":
warnings.append("stack_diagnosis=attention")
publish_warnings.append("stack_diagnosis=attention")
elif not truthy_http(stack_status_code):
blocking_reasons.append("stack_diagnosis_unavailable")
publish_blocking_reasons.append("stack_diagnosis_unavailable")
surface_complete = bool(route_surface.get("surface_complete", False))
if not surface_complete:
missing_keys = [ensure_str(item) for item in ensure_list(route_surface.get("missing_keys")) if ensure_str(item)]
blocking_reasons.append(
"route_surface_incomplete" + (f":{','.join(missing_keys)}" if missing_keys else "")
)
publish_blocking_reasons.append(
"route_surface_incomplete" + (f":{','.join(missing_keys)}" if missing_keys else "")
)
if not truthy_http(contracts_status_code) or len(contracts) <= 0:
blocking_reasons.append("contracts_unavailable")
publish_blocking_reasons.append("contracts_unavailable")
launchpad_status_code_value = ensure_str(launchpad_status.get("status")) or ensure_str(release_hub.get("launchpad_status")) or ""
if launchpad_status_code_value == "blocked":
blocking_reasons.append("release_launchpad=blocked")
publish_blocking_reasons.append("release_launchpad=blocked")
elif launchpad_status_code_value == "attention":
warnings.append("release_launchpad=attention")
publish_warnings.append("release_launchpad=attention")
elif not truthy_http(launchpad_status_code):
blocking_reasons.append("release_launchpad_unavailable")
publish_blocking_reasons.append("release_launchpad_unavailable")
if launchpad_onboarding_bootstrap_pending_nodes > 0:
warnings.append(f"launchpad_onboarding_bootstrap_pending={launchpad_onboarding_bootstrap_pending_nodes}")
publish_warnings.append(f"launchpad_onboarding_bootstrap_pending={launchpad_onboarding_bootstrap_pending_nodes}")
if launchpad_onboarding_acceptance_ready_nodes > 0:
warnings.append(f"launchpad_onboarding_acceptance_ready={launchpad_onboarding_acceptance_ready_nodes}")
publish_warnings.append(f"launchpad_onboarding_acceptance_ready={launchpad_onboarding_acceptance_ready_nodes}")
managed_enabled = int(nodes_summary.get("managed_enabled", 0) or 0)
remote_access_ready = int(nodes_summary.get("remote_access_ready", 0) or 0)
queue_dead_letter_nodes = int(nodes_summary.get("queue_dead_letter_nodes", 0) or 0)
if managed_enabled > 0 and remote_access_ready == 0:
blocking_reasons.append("remote_agent_ready=0")
publish_blocking_reasons.append("remote_agent_ready=0")
elif managed_enabled > 0 and remote_access_ready < managed_enabled:
warnings.append(f"remote_agent_ready={remote_access_ready}/{managed_enabled}")
publish_warnings.append(f"remote_agent_ready={remote_access_ready}/{managed_enabled}")
if queue_dead_letter_nodes > 0:
warnings.append(f"queue_dead_letter_nodes={queue_dead_letter_nodes}")
publish_warnings.append(f"queue_dead_letter_nodes={queue_dead_letter_nodes}")
if local_agent_installed and local_agent_active not in {"", "active"}:
warnings.append(f"{local_node_agent_service_name}={local_agent_active}")
log_sync_enabled = bool(log_sync.get("enabled"))
log_sync_state = ensure_str(log_sync.get("state"))
log_sync_mode = ensure_str(log_sync.get("mode"))
participating_nodes_total = int(log_sync.get("participating_node_count", 0) or 0)
covered_participating_nodes = int(log_sync.get("covered_participating_node_count", 0) or 0)
missing_participating_node_codes = [
ensure_str(item)
for item in ensure_list(log_sync.get("missing_participating_nodes"))
if ensure_str(item)
]
if log_sync_enabled and participating_nodes_total > 0 and covered_participating_nodes == 0:
warnings.append(f"log_sync_waiting_sample=0/{participating_nodes_total}")
publish_warnings.append(f"log_sync_waiting_sample=0/{participating_nodes_total}")
elif log_sync_enabled and participating_nodes_total > 0 and covered_participating_nodes < participating_nodes_total:
warnings.append(f"log_sync_partial={covered_participating_nodes}/{participating_nodes_total}")
publish_warnings.append(f"log_sync_partial={covered_participating_nodes}/{participating_nodes_total}")
operator_lane = ensure_str(operator_decision.get("lane"))
operator_title = ensure_str(operator_decision.get("title"))
operator_primary_command_key = ensure_str(operator_decision.get("primary_command_key"))
if not operator_lane:
if blocking_reasons:
operator_lane = "recovery"
elif warnings:
operator_lane = "verification"
else:
operator_lane = "go-live"
if not operator_title:
operator_title = (
ensure_str(overview_recommendation.get("title"))
or ensure_str(launchpad_status.get("status_label"))
or ("先处理阻塞项" if blocking_reasons else "可以进入上线复核")
)
next_step_action_code = (
ensure_str(next_step.get("action_code"))
or ensure_str(overview_recommendation.get("primary_action_code"))
or ensure_str(launchpad_status.get("recommended_action_code"))
)
next_step_reason = (
ensure_str(next_step.get("reason"))
or ensure_str(overview_recommendation.get("reason"))
or "来自 launchpad/overview fallback"
)
if not operator_primary_command_key:
operator_primary_command_key = next_step_action_code
go_live_status = "ready"
if blocking_reasons:
go_live_status = "blocked"
elif warnings:
go_live_status = "attention"
publish_status = "ready"
if publish_blocking_reasons:
publish_status = "blocked"
elif publish_warnings:
publish_status = "attention"
publish_ready = publish_status == "ready"
if publish_status == "ready":
publish_status_label = "可发布"
elif publish_status == "attention":
publish_status_label = "可发布但建议先复核"
else:
publish_status_label = "暂不可发布"
action_commands = {
"handover_first_gap": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} handover_first_gap",
"view_first_gap": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} view_first_gap",
"bootstrap_run": (
f"bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-run {base_url} {launchpad_recommended_target_node_code} cli"
if launchpad_recommended_target_node_code
else f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} bootstrap_run"
),
"run_acceptance": (
f"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run {base_url} {launchpad_recommended_target_node_code} cli"
if launchpad_recommended_target_node_code
else f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} run_acceptance"
),
"fix_managed_nodes": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} fix_managed_nodes",
"open_worker_logs_participating": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} open_worker_logs_participating",
"run_inspection_participating": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} run_inspection_participating",
}
recommended_commands = {
"stack_summary": f"bash domain-api/deploy/multi-region/check_ops_center_stack.sh {base_url} summary",
"contracts": f"bash domain-api/deploy/multi-region/check_ops_contracts.sh {base_url}",
"ops_plane": f"bash domain-api/deploy/multi-region/check_ops_plane.sh {base_url}",
"release_hub": f"bash domain-api/deploy/multi-region/check_release_hub.sh {base_url}",
"inspection": f"bash domain-api/deploy/multi-region/check_ops_inspection.sh {base_url}",
"node_agent": "bash domain-api/deploy/multi-region/check_node_agent.sh",
"overview": f"bash domain-api/deploy/multi-region/drive_ops_center.sh overview {base_url}",
"doctor_export": (
f"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live {base_url} {overseas_base_url}"
if overseas_base_url
else f"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live {base_url}"
),
}
if next_step_action_code and next_step_action_code in action_commands:
recommended_commands["next_step"] = action_commands[next_step_action_code]
if log_sync_enabled:
if "open_worker_logs_participating" in action_commands:
recommended_commands["log_sync_logs"] = action_commands["open_worker_logs_participating"]
if "run_inspection_participating" in action_commands:
recommended_commands["log_sync_inspection"] = action_commands["run_inspection_participating"]
summary = {
"base_url": base_url,
"overseas_base_url": overseas_base_url,
"go_live_status": go_live_status,
"publish_ready": publish_ready,
"publish_status": publish_status,
"publish_status_label": publish_status_label,
"stack_status": stack_status,
"contracts_ready": truthy_http(contracts_status_code) and len(contracts) > 0,
"contracts_total": len(contracts),
"launchpad_status": launchpad_status_code_value,
"launchpad_status_label": ensure_str(launchpad_status.get("status_label")),
"launchpad_recommended_action_code": ensure_str(launchpad_status.get("recommended_action_code")),
"launchpad_recommended_target_node_code": launchpad_recommended_target_node_code,
"launchpad_recommended_recovery_label": launchpad_recommended_recovery_label,
"launchpad_recommended_recovery_summary": launchpad_recommended_recovery_summary,
"launchpad_onboarding_bootstrap_pending_nodes": launchpad_onboarding_bootstrap_pending_nodes,
"launchpad_onboarding_acceptance_ready_nodes": launchpad_onboarding_acceptance_ready_nodes,
"route_surface_complete": surface_complete,
"route_surface_missing_keys": [
ensure_str(item) for item in ensure_list(route_surface.get("missing_keys")) if ensure_str(item)
],
"managed_enabled": managed_enabled,
"remote_access_ready": remote_access_ready,
"queue_dead_letter_nodes": queue_dead_letter_nodes,
"participating_nodes_total": participating_nodes_total,
"log_sync_enabled": log_sync_enabled,
"log_sync_state": log_sync_state,
"log_sync_mode": log_sync_mode,
"log_sync_covered_nodes": covered_participating_nodes,
"log_sync_missing_node_codes": missing_participating_node_codes,
"next_step_action_code": next_step_action_code,
"next_step_reason": next_step_reason,
"operator_lane": operator_lane,
"operator_title": operator_title,
"operator_primary_command_key": operator_primary_command_key,
"local_node_agent": {
"service_name": local_node_agent_service_name,
"installed": local_agent_installed,
"enabled": local_agent_enabled,
"active": local_agent_active,
},
"blocking_reasons": blocking_reasons,
"warnings": warnings,
"publish_blocking_reasons": publish_blocking_reasons,
"publish_warnings": publish_warnings,
"recommended_commands": recommended_commands,
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY

View File

@@ -0,0 +1,182 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${SCRIPT_DIR}/lib_ops_center.sh"
ops_center_load_config "${SCRIPT_DIR}"
TARGET_OR_BASE_URL="${1:-}"
SECOND_ARG="${2:-}"
REPORT_DIR=""
MANIFEST_PATH=""
GENERATED_BUNDLE="false"
MAINLAND_BASE_URL=""
OVERSEAS_BASE_URL=""
if [[ -n "${TARGET_OR_BASE_URL}" && ( -f "${TARGET_OR_BASE_URL}" || -d "${TARGET_OR_BASE_URL}" ) ]]; then
if [[ -d "${TARGET_OR_BASE_URL}" ]]; then
REPORT_DIR="${TARGET_OR_BASE_URL%/}"
MANIFEST_PATH="${REPORT_DIR}/manifest.json"
else
MANIFEST_PATH="${TARGET_OR_BASE_URL}"
REPORT_DIR="$(cd "$(dirname "${MANIFEST_PATH}")" && pwd)"
fi
else
MAINLAND_BASE_URL="${TARGET_OR_BASE_URL:-$(ops_center_resolve_mainland_api_base_url)}"
OVERSEAS_BASE_URL="$(normalize_optional_base_url_arg "${SECOND_ARG:-}" "$(ops_center_resolve_overseas_api_base_url)")"
REPORT_DIR="${OPS_CENTER_REPORT_ROOT}/go-live-bundles/go-live-signoff-$(timestamp_now)"
ops_center_ensure_report_dir "${REPORT_DIR}"
bash "${SCRIPT_DIR}/export_go_live_bundle.sh" "${REPORT_DIR}" "${MAINLAND_BASE_URL}" "${OVERSEAS_BASE_URL}" >/dev/null
MANIFEST_PATH="${REPORT_DIR}/manifest.json"
GENERATED_BUNDLE="true"
fi
if [[ ! -f "${MANIFEST_PATH}" ]]; then
echo "manifest not found: ${MANIFEST_PATH}" >&2
exit 1
fi
REVIEW_JSON="$(bash "${SCRIPT_DIR}/review_go_live_bundle.sh" "${MANIFEST_PATH}")"
DOCTOR_JSON="$(bash "${SCRIPT_DIR}/drive_ops_center.sh" doctor-decision "${MANIFEST_PATH}")"
python3 - <<'PY' \
"${REVIEW_JSON}" \
"${DOCTOR_JSON}" \
"${GENERATED_BUNDLE}" \
"${REPORT_DIR}" \
"${MANIFEST_PATH}" \
"${MAINLAND_BASE_URL}" \
"${OVERSEAS_BASE_URL}"
import json
import sys
review = json.loads(sys.argv[1])
doctor = json.loads(sys.argv[2])
generated_bundle = str(sys.argv[3]).strip().lower() == "true"
report_dir = str(sys.argv[4] or "").strip()
manifest_path = str(sys.argv[5] or "").strip()
mainland_base_url = str(sys.argv[6] or "").strip()
overseas_base_url = str(sys.argv[7] or "").strip()
go_live_summary = dict(review.get("go_live_summary") or {})
stack_diagnosis = dict(review.get("stack_diagnosis") or {})
driver_feed = dict(review.get("driver_feed") or {})
codex_brief = dict(review.get("codex_brief") or {})
launchpad_alignment = dict(review.get("launchpad_alignment") or {})
doctor_decision = dict(doctor.get("decision") or {})
doctor_summary = dict(doctor.get("summary") or {})
review_status = str(review.get("status") or "").strip()
doctor_status = str(doctor_decision.get("status") or "").strip()
go_live_status = str(go_live_summary.get("go_live_status") or "").strip()
publish_status = str(go_live_summary.get("publish_status") or "").strip()
stack_status = str(stack_diagnosis.get("stack_status") or "").strip()
driver_launch_status = str(driver_feed.get("launch_status") or "").strip()
codex_launch_status = str(codex_brief.get("launch_status") or "").strip()
def dedupe_commands(values):
result = []
seen = set()
for item in values:
text = str(item or "").strip()
if not text or text in seen:
continue
seen.add(text)
result.append(text)
return result
blocked_reasons = dedupe_commands(
[]
+ ([f"review:{review_status}"] if review_status == "blocked" else [])
+ ([f"doctor:{doctor_status}"] if doctor_status == "blocked" else [])
+ ([f"go_live:{go_live_status}"] if go_live_status == "blocked" else [])
+ ([f"publish:{publish_status}"] if publish_status == "blocked" else [])
+ ([f"stack:{stack_status}"] if stack_status == "blocked" else [])
+ ([f"driver_launch:{driver_launch_status}"] if driver_launch_status == "blocked" else [])
+ ([f"codex_launch:{codex_launch_status}"] if codex_launch_status == "blocked" else [])
)
attention_reasons = dedupe_commands(
[]
+ ([f"review:{review_status}"] if review_status == "attention" else [])
+ ([f"doctor:{doctor_status}"] if doctor_status == "attention" else [])
+ ([f"go_live:{go_live_status}"] if go_live_status == "attention" else [])
+ ([f"publish:{publish_status}"] if publish_status == "attention" else [])
+ ([f"stack:{stack_status}"] if stack_status == "attention" else [])
+ ([f"driver_launch:{driver_launch_status}"] if driver_launch_status == "attention" else [])
+ ([f"codex_launch:{codex_launch_status}"] if codex_launch_status == "attention" else [])
+ (["launchpad_alignment:inconsistent"] if not bool(launchpad_alignment.get("consistent", False)) else [])
)
if blocked_reasons:
signoff_status = "blocked"
headline = (
str(review.get("headline") or "").strip()
or str(doctor_decision.get("headline") or "").strip()
or "当前发布前验证仍存在阻断项,不应直接签字上线。"
)
elif attention_reasons:
signoff_status = "attention"
headline = (
str(doctor_decision.get("headline") or "").strip()
or str(review.get("headline") or "").strip()
or "当前现场已进入收口区间,但仍建议先完成 attention 项再正式签字。"
)
else:
signoff_status = "ready"
headline = "当前证据包、doctor 结论与发布门禁一致,可进入最终人工签字或正式发布。"
recommended_commands = dedupe_commands(
[]
+ list(doctor_decision.get("recommended_commands") or [])
+ [
f"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review {manifest_path}",
f"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision {manifest_path}",
]
)
payload = {
"signoff_status": signoff_status,
"headline": headline,
"generated_bundle": generated_bundle,
"report_dir": report_dir or str(doctor.get("report_dir") or review.get("report_dir") or ""),
"manifest_path": manifest_path or str(doctor.get("manifest_path") or ""),
"base_urls": {
"mainland": mainland_base_url,
"overseas": overseas_base_url,
},
"release_gate": {
"go_live_status": go_live_status,
"publish_status": publish_status,
"stack_status": stack_status,
"driver_launch_status": driver_launch_status,
"codex_launch_status": codex_launch_status,
"bundle_review_status": review_status,
"doctor_status": doctor_status,
},
"blocked_reasons": blocked_reasons,
"attention_reasons": attention_reasons,
"decision": {
"operator_title": str(go_live_summary.get("operator_title") or "").strip(),
"next_step_action_code": str(go_live_summary.get("next_step_action_code") or "").strip(),
"doctor_reason_code": str(doctor_decision.get("reason_code") or "").strip(),
"doctor_preferred_surface": str(doctor_decision.get("preferred_surface") or "").strip(),
"launchpad_recommended_target_node_code": str(go_live_summary.get("launchpad_recommended_target_node_code") or "").strip(),
"launchpad_recommended_recovery_label": str(go_live_summary.get("launchpad_recommended_recovery_label") or "").strip(),
"launchpad_recommended_recovery_summary": str(go_live_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
},
"launchpad_alignment": launchpad_alignment,
"recommended_commands": recommended_commands,
"bundle_review": review,
"doctor_decision": {
"summary": doctor_summary,
"decision": doctor_decision,
},
}
print(json.dumps(payload, ensure_ascii=False, indent=2))
PY

View File

@@ -0,0 +1,38 @@
#!/usr/bin/env bash
set -euo pipefail
SERVICE_NAME="${SERVICE_NAME:-domaincheck-node-agent}"
ENV_FILE="${ENV_FILE:-/etc/default/domaincheck-node-agent}"
echo "[1/4] environment file"
if [[ -f "${ENV_FILE}" ]]; then
echo "env file: ${ENV_FILE}"
else
echo "missing env file: ${ENV_FILE}"
fi
echo
echo "[2/4] key env summary"
if [[ -f "${ENV_FILE}" ]]; then
grep -E '^(OPS_CONTROL_PLANE_BASE_URL|NODE_CODE|NODE_REGION|NODE_ROLE|OPS_AGENT_POLL_INTERVAL_SECONDS|WORKER_SERVICE_NAME|API_SERVICE_NAME|SYNC_AGENT_SERVICE_NAME)=' "${ENV_FILE}" || true
else
echo "skip"
fi
echo
echo "[3/4] systemd status"
if systemctl list-unit-files "${SERVICE_NAME}.service" --no-pager >/dev/null 2>&1; then
systemctl is-enabled "${SERVICE_NAME}" || true
systemctl is-active "${SERVICE_NAME}" || true
else
echo "not-installed"
echo "inactive"
fi
echo
echo "[4/4] service detail"
if systemctl list-unit-files "${SERVICE_NAME}.service" --no-pager >/dev/null 2>&1; then
systemctl status "${SERVICE_NAME}" --no-pager -l | sed -n '1,25p' || true
else
echo "service not found: ${SERVICE_NAME}"
fi

View File

@@ -0,0 +1,65 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
NODE_CODE="${2:-}"
LIMIT="${3:-80}"
MODE="${4:-key}"
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
CURL_MAX_TIME="${CURL_MAX_TIME:-8}"
if [[ -z "${NODE_CODE}" ]]; then
echo "usage: bash domain-api/deploy/multi-region/check_node_scene_log.sh [base_url] <node_code> [limit] [key|full]" >&2
exit 1
fi
RAW_RESPONSE="$(curl -sS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
"${BASE_URL}/api/v1/ops/nodes/${NODE_CODE}/scene-log?limit=${LIMIT}&mode=${MODE}")"
echo "[1/2] raw response"
printf '%s\n\n' "${RAW_RESPONSE}"
echo "[2/2] condensed scene-log summary"
python3 - "${RAW_RESPONSE}" <<'PY'
import json
import sys
raw = str(sys.argv[1] or "")
payload = json.loads(raw)
data = dict(payload.get("data") or {})
participation = dict(data.get("participation") or {})
source_summary = dict(data.get("source_summary") or {})
records = list(data.get("records") or [])
latest = dict(data.get("latest_record") or {})
summary = {
"node_code": str(data.get("node_code") or ""),
"status": str(data.get("status") or ""),
"status_label": str(data.get("status_label") or ""),
"mode": str(data.get("mode") or ""),
"summary": str(data.get("summary") or ""),
"log_sync_enabled": bool(data.get("log_sync_enabled", False)),
"records_total": int(data.get("records_total", len(records)) or 0),
"missing_reason_code": str(data.get("missing_reason_code") or ""),
"participation": {
"detect_participating": bool(participation.get("detect_participating", False)),
"participation_state": str(participation.get("participation_state") or ""),
"participation_label": str(participation.get("participation_label") or ""),
"participation_reason": str(participation.get("participation_reason") or ""),
"current_load": int(participation.get("current_load", 0) or 0),
},
"source_summary": {
"line_count": int(source_summary.get("line_count", 0) or 0),
"key_line_count": int(source_summary.get("key_line_count", 0) or 0),
"full_line_count": int(source_summary.get("full_line_count", 0) or 0),
"last_at": str(source_summary.get("last_at") or ""),
"last_line": str(source_summary.get("last_line") or ""),
},
"latest_record": latest,
"sample_messages": [str(item.get("message") or "") for item in records[:8]],
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,55 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
contracts_json="$(curl -fsS "${BASE_URL}/api/v1/ops/contracts")"
echo "[1/3] ops contracts registry"
printf '%s\n\n' "${contracts_json}"
echo "[2/3] condensed summary"
python3 - <<'PY' "${contracts_json}"
import json
import sys
payload = json.loads(sys.argv[1])
data = payload.get("data") or {}
contracts = list(data.get("contracts") or [])
summary = {
"registry_version": str(data.get("registry_version") or ""),
"schema_version": str(data.get("schema_version") or ""),
"contracts_total": len(contracts),
"contract_keys": [str(item.get("key") or "") for item in contracts if str(item.get("key") or "")],
"docs_root": str(data.get("docs_root") or ""),
"discovery_entrypoints": list(data.get("discovery_entrypoints") or []),
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
echo
echo "[3/3] contract rows"
python3 - <<'PY' "${contracts_json}"
import json
import sys
payload = json.loads(sys.argv[1])
data = payload.get("data") or {}
contracts = list(data.get("contracts") or [])
rows = [
{
"key": str(item.get("key") or ""),
"version": str(item.get("version") or ""),
"status": str(item.get("status") or ""),
"primary_endpoint": str(item.get("primary_endpoint") or ""),
"schema_doc_path": str(item.get("schema_doc_path") or ""),
"consumers": list(item.get("consumers") or []),
}
for item in contracts
]
print(json.dumps(rows, ensure_ascii=False, indent=2))
PY

View File

@@ -0,0 +1,72 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
curl_json() {
curl -fsS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
"$1"
}
INSPECTION_JSON="$(curl_json "${BASE_URL}/api/v1/ops/inspection-overview?limit=120")"
echo "[1/5] inspection overview"
printf '%s\n\n' "${INSPECTION_JSON}"
echo "[2/5] condensed summary"
python3 - <<'PY' "${INSPECTION_JSON}"
import json
import sys
payload = json.loads(sys.argv[1]).get("data") or {}
print(json.dumps({
"rows_total": int(payload.get("rows_total") or 0),
"filtered_rows_total": int(payload.get("filtered_rows_total") or 0),
"visible_nodes_total": int(payload.get("visible_nodes_total") or 0),
"eligible_nodes_total": int(payload.get("eligible_nodes_total") or 0),
"handover_gap_nodes_total": int(payload.get("handover_gap_nodes_total") or 0),
"participating_nodes_total": int(payload.get("participating_nodes_total") or 0),
"summary": dict(payload.get("summary") or {}),
"available_problem_kind_counts": dict(payload.get("available_problem_kind_counts") or {}),
"filters": dict(payload.get("filters") or {}),
}, ensure_ascii=False, indent=2))
PY
echo
echo "[3/5] priority queue"
python3 - <<'PY' "${INSPECTION_JSON}"
import json
import sys
payload = json.loads(sys.argv[1]).get("data") or {}
rows = list(payload.get("priority_queue") or [])
print(json.dumps({
"total": len(rows),
"items": [
{
"node_code": str(item.get("node_code") or ""),
"overall_status": str(item.get("overall_status") or ""),
"problem_kind": str(item.get("problem_kind") or ""),
"problem_label": str(item.get("problem_label") or ""),
"recommended_action_code": str(item.get("recommended_action_code") or ""),
"last_updated_at": str(item.get("last_updated_at") or ""),
"issue_summary": str(item.get("issue_summary") or ""),
}
for item in rows[:12]
],
}, ensure_ascii=False, indent=2))
PY
echo
echo "[4/5] attention only"
curl_json "${BASE_URL}/api/v1/ops/inspection-overview?limit=80&status=attention&only_problem=true"
echo
echo
echo "[5/5] participating problems only"
curl_json "${BASE_URL}/api/v1/ops/inspection-overview?limit=80&only_problem=true&only_participating=true"
echo

View File

@@ -0,0 +1,35 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
PYTHON_BIN="${PYTHON_BIN:-python3}"
echo "[1/2] ops link snapshot"
SNAPSHOT_JSON="$(curl -fsS "${BASE_URL}/api/v1/ops/link-snapshot")"
echo "${SNAPSHOT_JSON}"
echo
echo
echo "[2/2] condensed summary"
"${PYTHON_BIN}" - <<'PY' "$SNAPSHOT_JSON"
import json
import sys
payload = json.loads(sys.argv[1]).get("data", {})
print(f"status: {payload.get('status', '-')}")
print(f"headline: {payload.get('headline', '-')}")
print()
print("summary:")
for line in payload.get("summary_lines", []) or []:
print(f"- {line}")
actions = payload.get("next_actions", []) or []
if actions:
print()
print("next_actions:")
for item in actions:
label = str(item.get("label", "") or "").strip() or "-"
action_code = str(item.get("action_code", "") or "").strip() or "-"
node_codes = "、".join(str(x).strip() for x in (item.get("node_codes", []) or []) if str(x).strip()) or "-"
print(f"- {label} / {action_code} / nodes: {node_codes}")
PY

View File

@@ -0,0 +1,609 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
curl_json() {
curl -fsS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
"$1"
}
curl_post_json() {
local url="$1"
local body="$2"
curl -fsS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
-X POST "${url}" \
-H 'Content-Type: application/json' \
-d "${body}"
}
OVERVIEW_JSON="$(curl_json "${BASE_URL}/api/v1/ops/overview")"
NODES_JSON="$(curl_json "${BASE_URL}/api/v1/ops/nodes")"
JOBS_JSON="$(curl_json "${BASE_URL}/api/v1/ops/jobs")"
ACTIVITY_JSON="$(curl_json "${BASE_URL}/api/v1/ops/activity-stream")"
LAUNCHPAD_JSON="$(curl_json "${BASE_URL}/api/v1/ops/releases/launchpad")"
LATEST_RELEASE_JSON="$(curl_json "${BASE_URL}/api/v1/ops/releases/latest")"
POLICY_PREVIEW_JSON="$(
curl_post_json "${BASE_URL}/api/v1/ops/policy/preview" \
'{"action":"runtime.restart_api","target_type":"node","target_node_code":"mainland-controller-01"}'
)"
echo "[1/12] ops overview"
printf '%s\n\n' "${OVERVIEW_JSON}"
echo "[2/12] top recommendation"
python3 - <<'PY' "${OVERVIEW_JSON}"
import json
import sys
payload = json.loads(sys.argv[1])
data = payload.get("data") or {}
recommendation = dict(data.get("recommendation") or {})
print(json.dumps({
"source": str(recommendation.get("source") or ""),
"key": str(recommendation.get("key") or ""),
"priority": str(recommendation.get("priority") or ""),
"summary": str(recommendation.get("summary") or ""),
"primary_action_code": str(recommendation.get("primary_action_code") or ""),
"secondary_action_code": str(recommendation.get("secondary_action_code") or ""),
"node_codes": list(recommendation.get("node_codes") or []),
}, ensure_ascii=False, indent=2))
PY
echo
echo "[3/12] driver recommendations (condensed)"
python3 - <<'PY' "${OVERVIEW_JSON}"
import json
import sys
payload = json.loads(sys.argv[1])
data = payload.get("data") or {}
recommendations = list(data.get("driver_recommendations") or [])
summary = {
"total": len(recommendations),
"items": [
{
"key": str(item.get("key") or ""),
"title": str(item.get("title") or ""),
"level_label": str(item.get("level_label") or ""),
"primary_action_code": str(item.get("primary_action_code") or ""),
"secondary_action_code": str(item.get("secondary_action_code") or ""),
"meta_text": str(item.get("meta_text") or ""),
}
for item in recommendations[:8]
],
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
echo
echo "[4/12] ops capabilities"
curl_json "${BASE_URL}/api/v1/ops/capabilities"
echo
echo
echo "[5/12] ops blueprint"
curl_json "${BASE_URL}/api/v1/ops/blueprint"
echo
echo
echo "[6/12] ops runbook"
curl_json "${BASE_URL}/api/v1/ops/runbook"
echo
echo
echo "[7/12] ops nodes"
printf '%s\n' "${NODES_JSON}"
echo
echo
echo "[8/12] ops jobs"
printf '%s\n' "${JOBS_JSON}"
echo
echo
echo "[9/12] ops activity stream"
printf '%s\n' "${ACTIVITY_JSON}"
echo
echo
echo "[10/12] release launchpad"
printf '%s\n' "${LAUNCHPAD_JSON}"
echo
echo
echo "[11/12] latest release"
printf '%s\n' "${LATEST_RELEASE_JSON}"
echo
echo
echo "[12/12] policy preview sample"
printf '%s\n' "${POLICY_PREVIEW_JSON}"
echo
echo
echo "[13/13] condensed summary"
python3 - <<'PY' \
"${BASE_URL}" \
"${OVERVIEW_JSON}" \
"${NODES_JSON}" \
"${JOBS_JSON}" \
"${ACTIVITY_JSON}" \
"${LAUNCHPAD_JSON}" \
"${LATEST_RELEASE_JSON}" \
"${POLICY_PREVIEW_JSON}"
import json
import sys
def as_dict(payload):
try:
raw = json.loads(payload)
except Exception:
return {}
return raw.get("data") or {}
def ensure_dict(value):
return value if isinstance(value, dict) else {}
def ensure_list(value):
return value if isinstance(value, list) else []
def ensure_str(value):
return value if isinstance(value, str) else ""
def command(name, *parts):
tokens = [
"bash",
"domain-api/deploy/multi-region/drive_ops_center.sh",
name,
base_url,
]
tokens.extend([str(part) for part in parts if str(part or "").strip()])
return " ".join(tokens)
def release_command(name, *parts):
tokens = [
"bash",
"domain-api/deploy/multi-region/drive_release_hub.sh",
base_url,
name,
]
tokens.extend([str(part) for part in parts if str(part or "").strip()])
return " ".join(tokens)
def build_operator_decision():
remote_access_state_counts = ensure_dict(node_summary.get("remote_access_state_counts"))
pending_bootstrap = int(node_summary.get("pending_bootstrap", 0) or 0)
runtime_only = int(node_summary.get("runtime_only", 0) or 0)
online_total = int(node_summary.get("online", 0) or 0)
participating_total = int(scene_log.get("participating_node_count", 0) or 0)
launchpad_blocked = bool(launchpad_workers.get("blocked")) or bool(launchpad_controls.get("blocked"))
running_total = int(jobs_summary.get("running_count", 0) or 0)
queued_total = int(jobs_summary.get("queued_count", 0) or 0)
if scene_log.get("status") == "disabled" and participating_total > 0:
return {
"lane": "observability",
"priority": "high",
"reason_code": "scene_log_disabled",
"title": "先恢复远端现场日志可见性",
"summary": ensure_str(scene_log.get("summary")),
"next_focus": ensure_dict(scene_log.get("focus_ref")),
"primary_command_key": "scene_log",
"secondary_command_key": "driver_focus_preview",
}
if pending_bootstrap > 0 or remote_access_state_counts:
return {
"lane": "node_handover",
"priority": "high",
"reason_code": "managed_nodes_not_ready",
"title": "先补齐节点接管与远程可达性",
"summary": (
f"当前 managed node 共 {int(node_summary.get('managed_total', 0) or 0)} 台,"
f"pending_bootstrap={pending_bootstrap}runtime_only={runtime_only}。"
),
"next_focus": {
"kind": "managed_node_handover",
"node_code": first_attention_node_code,
} if first_attention_node_code else {},
"primary_command_key": "node_handover",
"secondary_command_key": "node_bootstrap_plan",
}
if launchpad_blocked or str(launchpad_status.get("status") or "") == "attention":
return {
"lane": "release",
"priority": "medium",
"reason_code": "release_launchpad_attention",
"title": "检查发布驾驶舱与 rollout 阻断",
"summary": ensure_str(launchpad_status.get("summary")),
"next_focus": release_focus_ref,
"primary_command_key": "release_launchpad",
"secondary_command_key": "release_preview_latest",
}
if running_total > 0 or queued_total > 0:
return {
"lane": "ops_jobs",
"priority": "medium",
"reason_code": "jobs_active",
"title": "继续盯运行中任务与活动流",
"summary": (
f"当前 running={running_total}queued={queued_total}"
f"activity_status={ensure_str(activity_summary.get('status')) or 'unknown'}。"
),
"next_focus": {
"kind": "ops_jobs",
"section": "activity_stream",
},
"primary_command_key": "jobs",
"secondary_command_key": "activity_preview",
}
return {
"lane": "steady",
"priority": "low",
"reason_code": "steady_state",
"title": "当前进入稳定观察态",
"summary": (
f"online={online_total}attention_nodes={len(attention_nodes)}"
f"scene_log_status={ensure_str(scene_log.get('status')) or 'standby'}。"
),
"next_focus": recommendation_focus_ref or ensure_dict(scene_log.get("focus_ref")),
"primary_command_key": "driver_feed",
"secondary_command_key": "activity_preview",
}
def normalize_scene_log_observation(overview_data):
driver_feed = ensure_dict(overview_data.get("driver_feed"))
scene_log_data = ensure_dict(driver_feed.get("scene_log_observation"))
if scene_log_data:
normalized = dict(scene_log_data)
normalized["status"] = ensure_str(normalized.get("status")) or "standby"
normalized["status_label"] = ensure_str(normalized.get("status_label")) or "待命"
normalized["summary"] = ensure_str(normalized.get("summary")) or "当前现场日志观察暂无额外结论。"
normalized["focus_ref"] = ensure_dict(normalized.get("focus_ref"))
normalized["target_node_code"] = ensure_str(normalized.get("target_node_code"))
normalized["participating_node_count"] = int(normalized.get("participating_node_count", 0) or 0)
normalized["covered_participating_node_count"] = int(normalized.get("covered_participating_node_count", 0) or 0)
normalized["line_count"] = int(normalized.get("line_count", 0) or 0)
return normalized
execution_scene = ensure_dict(overview_data.get("execution_scene"))
log_sync = ensure_dict(execution_scene.get("log_sync"))
participation = ensure_dict(execution_scene.get("participation_summary"))
missing_nodes = ensure_list(log_sync.get("missing_participating_nodes"))
participating_nodes = ensure_list(execution_scene.get("participating_nodes"))
target_node_code = ""
if missing_nodes:
target_node_code = ensure_str(missing_nodes[0])
elif participating_nodes:
target_node_code = ensure_str(ensure_dict(participating_nodes[0]).get("node_code"))
participating_total = int(
log_sync.get("participating_node_count", participation.get("participating_nodes", len(participating_nodes)))
or participation.get("participating_nodes", len(participating_nodes))
or len(participating_nodes)
or 0
)
covered_total = int(log_sync.get("covered_participating_node_count", 0) or 0)
line_count = int(log_sync.get("line_count", 0) or 0)
enabled = bool(log_sync.get("enabled", False))
mode = ensure_str(log_sync.get("mode")) or "key"
status = "standby"
status_label = "待命"
summary = "当前没有参与检测节点,现场日志观察处于待命状态。"
if participating_total > 0 and not enabled:
status = "disabled"
status_label = "未开启"
summary = (
f"当前有 {participating_total} 台参与节点,但远端日志回传仍关闭,"
"海外控制面还看不到节点级现场日志。"
)
elif participating_total > 0 and (
line_count <= 0 or covered_total <= 0 or int(log_sync.get("missing_participating_node_count", len(missing_nodes)) or len(missing_nodes)) > 0
):
status = "waiting_sample"
status_label = "等待样本"
summary = (
f"远端日志回传已经开启,但参与节点只覆盖 {covered_total}/{participating_total}"
f"样本 {line_count} 条,仍需继续观察或下钻节点现场日志。"
)
elif participating_total > 0:
status = "ready"
status_label = "可下钻"
summary = (
f"远端日志回传已经形成样本,当前覆盖 {covered_total}/{participating_total} 台参与节点,"
"可以直接下钻节点现场日志。"
)
focus_ref = {}
if target_node_code:
focus_ref = {
"kind": "node_scene_log",
"node_code": target_node_code,
"mode": mode,
"limit": 120 if status == "waiting_sample" else 80,
}
return {
"status": status,
"status_label": status_label,
"summary": summary,
"focus_ref": focus_ref,
"target_node_code": target_node_code,
"participating_node_count": participating_total,
"covered_participating_node_count": covered_total,
"line_count": line_count,
}
base_url = ensure_str(sys.argv[1]) or "http://127.0.0.1:8100"
overview = as_dict(sys.argv[2])
nodes = as_dict(sys.argv[3])
jobs = as_dict(sys.argv[4])
activity = as_dict(sys.argv[5])
launchpad = as_dict(sys.argv[6])
latest_release = as_dict(sys.argv[7])
policy_preview = as_dict(sys.argv[8])
recommendation = ensure_dict(overview.get("recommendation"))
scene_log = normalize_scene_log_observation(overview)
managed_nodes = ensure_list(nodes.get("managed_nodes"))
node_summary = ensure_dict(nodes.get("summary"))
jobs_summary = ensure_dict(jobs.get("summary"))
activity_summary = ensure_dict(activity.get("summary"))
launchpad_release = ensure_dict(launchpad.get("release"))
launchpad_rec = ensure_dict(launchpad.get("top_recommendation"))
launchpad_workers = ensure_dict(launchpad.get("worker_rollout_preview"))
launchpad_controls = ensure_dict(launchpad.get("control_rollout_preview"))
launchpad_status = ensure_dict(launchpad.get("launchpad_status"))
latest_release_entry = latest_release if isinstance(latest_release, dict) else {}
latest_package_preview = ensure_dict(launchpad.get("latest_package_preview"))
online_nodes = [
node for node in managed_nodes
if str(node.get("status") or "").lower() in {"online", "busy"}
]
attention_nodes = [
{
"node_code": str(node.get("node_code") or ""),
"status": str(node.get("status") or ""),
"role": str(node.get("role") or ""),
"stack_status_label": str(node.get("stack_status_label") or ""),
"recommended_action_code": str(node.get("recommended_action_code") or ""),
"recommended_action_label": str(node.get("recommended_action_label") or ""),
"primary_issue": str(node.get("primary_issue") or ""),
}
for node in managed_nodes
if str(node.get("status") or "").lower() not in {"online", "busy"}
or str(node.get("stack_status") or "").lower() not in {"healthy", "ready", ""}
]
activity_items = ensure_list(activity.get("items"))
recent_activity = [
{
"kind": str(item.get("kind") or ""),
"title": str(item.get("title") or ""),
"level": str(item.get("level") or ""),
"created_at": str(item.get("created_at") or ""),
}
for item in activity_items[:5]
]
job_items = ensure_list(jobs.get("items"))
running_jobs = [
{
"job_key": str(item.get("job_key") or ""),
"status": str(item.get("status") or ""),
"title": str(item.get("title") or ""),
"progress_text": str(item.get("progress_text") or ""),
}
for item in job_items
if str(item.get("status") or "").lower() in {"running", "queued", "waiting"}
][:5]
recommendation_focus_ref = ensure_dict(recommendation.get("focus_ref"))
first_attention = attention_nodes[0] if attention_nodes else {}
first_attention_node_code = ensure_str(first_attention.get("node_code"))
scene_log_target_node_code = ensure_str(scene_log.get("target_node_code"))
latest_release_version = str(latest_release_entry.get("version") or latest_release_entry.get("release_version") or "")
release_focus_ref = ensure_dict(launchpad_rec.get("focus_ref"))
release_id = str(release_focus_ref.get("release_id") or "")
operator_decision = build_operator_decision()
primary_command_key = ensure_str(operator_decision.get("primary_command_key"))
secondary_command_key = ensure_str(operator_decision.get("secondary_command_key"))
result = {
"overview": {
"top_recommendation": {
"source": str(recommendation.get("source") or ""),
"key": str(recommendation.get("key") or ""),
"priority": str(recommendation.get("priority") or ""),
"summary": str(recommendation.get("summary") or ""),
"primary_action_code": str(recommendation.get("primary_action_code") or ""),
"secondary_action_code": str(recommendation.get("secondary_action_code") or ""),
"focus_ref": recommendation.get("focus_ref") or {},
"node_codes": list(recommendation.get("node_codes") or []),
},
"scene_log_observation": {
"status": str(scene_log.get("status") or ""),
"status_label": str(scene_log.get("status_label") or ""),
"summary": str(scene_log.get("summary") or ""),
"target_node_code": str(scene_log.get("target_node_code") or ""),
"participating_node_count": int(scene_log.get("participating_node_count") or 0),
"covered_participating_node_count": int(scene_log.get("covered_participating_node_count") or 0),
"line_count": int(scene_log.get("line_count") or 0),
"focus_ref": scene_log.get("focus_ref") or {},
},
},
"nodes": {
"managed_total": int(node_summary.get("managed_total") or len(managed_nodes)),
"online_or_busy_total": len(online_nodes),
"offline_total": int(node_summary.get("offline_count") or 0),
"attention_total": len(attention_nodes),
"attention_nodes": attention_nodes[:8],
},
"jobs": {
"running_total": int(jobs_summary.get("running_count") or 0),
"queued_total": int(jobs_summary.get("queued_count") or 0),
"failed_total": int(jobs_summary.get("failed_count") or 0),
"recent_running_jobs": running_jobs,
},
"activity_stream": {
"total": int(activity_summary.get("total") or len(activity_items)),
"error_count": int(activity_summary.get("error_count") or 0),
"warning_count": int(activity_summary.get("warning_count") or 0),
"running_job_count": int(activity_summary.get("running_job_count") or 0),
"recent_items": recent_activity,
},
"operator_decision": {
"lane": ensure_str(operator_decision.get("lane")),
"priority": ensure_str(operator_decision.get("priority")),
"reason_code": ensure_str(operator_decision.get("reason_code")),
"title": ensure_str(operator_decision.get("title")),
"summary": ensure_str(operator_decision.get("summary")),
"next_focus": ensure_dict(operator_decision.get("next_focus")),
"primary_command_key": primary_command_key,
"secondary_command_key": secondary_command_key,
},
"recommended_commands": {
"stack_diagnosis": "bash domain-api/deploy/multi-region/check_ops_center_stack.sh",
"driver_feed": command("driver-feed"),
"activity_preview": command("activity-preview"),
"driver_focus_preview": command("driver-focus-preview"),
"driver_focus_run": command("driver-focus-run"),
"top_recommendation_preview": (
command("driver-preview", str(recommendation.get("primary_action_code") or ""))
if str(recommendation.get("primary_action_code") or "").strip()
else ""
),
"top_recommendation_focus_preview": (
command("driver-focus-preview", f"entry:{str(recommendation.get('key') or '')}")
if str(recommendation.get("key") or "").strip()
else ""
),
"scene_log": (
command("scene-node-log", scene_log_target_node_code, "120", "key")
if scene_log_target_node_code
else ""
),
"scene_log_direct": (
f"bash domain-api/deploy/multi-region/check_node_scene_log.sh {base_url} {scene_log_target_node_code} 120 key"
if scene_log_target_node_code
else ""
),
"node_handover": (
command("node-handover", first_attention_node_code)
if first_attention_node_code
else command("stack-first-gap-handover")
),
"node_bootstrap_plan": (
command("node-bootstrap-plan", first_attention_node_code)
if first_attention_node_code
else ""
),
"delivery_queue": (
command("delivery-queue", first_attention_node_code)
if first_attention_node_code
else ""
),
"release_launchpad": release_command("launchpad"),
"release_preview_latest": release_command("preview-latest"),
"release_preview_latest_worker": release_command("preview-latest-smart", "worker"),
"release_preview_latest_control": release_command("preview-latest-smart", "control"),
"release_focus": (
command("driver-preview", "focus_release_hub")
if str(launchpad_rec.get("action_code") or "").strip()
else ""
),
"release_preview_selected": (
release_command("preview-release", release_id, "worker")
if release_id
else ""
),
"jobs": command("activity-stream", "kind=ops_job"),
},
"next_actions": {
"primary_command": "",
"secondary_command": "",
},
"release_launchpad": {
"status": str(launchpad_status.get("status") or ""),
"status_label": str(launchpad_status.get("status_label") or ""),
"summary": str(launchpad_status.get("summary") or ""),
"recommended_action_code": str(launchpad_rec.get("action_code") or ""),
"recommended_action_label": str(launchpad_rec.get("action_label") or ""),
"recommended_execution_mode": str(launchpad_rec.get("execution_mode") or ""),
"recommended_execution_mode_label": str(launchpad_rec.get("execution_mode_label") or ""),
"focus_ref": launchpad_rec.get("focus_ref") or {},
"release_version": str(launchpad_release.get("version") or launchpad_release.get("release_version") or ""),
"worker_rollout_preview": {
"available": bool(launchpad_workers.get("available")),
"blocked": bool(launchpad_workers.get("blocked")),
"execution_mode": str(launchpad_workers.get("execution_mode") or ""),
"execution_mode_label": str(launchpad_workers.get("execution_mode_label") or ""),
"target_node_codes": list(launchpad_workers.get("target_node_codes") or []),
},
"control_rollout_preview": {
"available": bool(launchpad_controls.get("available")),
"blocked": bool(launchpad_controls.get("blocked")),
"execution_mode": str(launchpad_controls.get("execution_mode") or ""),
"execution_mode_label": str(launchpad_controls.get("execution_mode_label") or ""),
"target_node_codes": list(launchpad_controls.get("target_node_codes") or []),
},
},
"latest_release": {
"version": str(latest_release_entry.get("version") or latest_release_entry.get("release_version") or ""),
"status": ensure_str(latest_release_entry.get("status")),
"status_label": ensure_str(latest_release_entry.get("status_label")),
"summary": ensure_str(latest_release_entry.get("summary") or latest_release_entry.get("summary_text")),
},
"policy_preview_sample": {
"execution_mode": str(policy_preview.get("execution_mode") or ""),
"risk_level": str(policy_preview.get("risk_level") or ""),
"approval_required": bool(policy_preview.get("approval_required")),
"blocked": bool(policy_preview.get("blocked")),
"blocking_reasons": ensure_list(policy_preview.get("blocking_reasons"))[:5],
"warnings": ensure_list(policy_preview.get("warnings"))[:5],
},
"focus_refs": {
"top_recommendation": recommendation_focus_ref,
"scene_log": ensure_dict(scene_log.get("focus_ref")),
"release_hub": release_focus_ref,
},
"rollout_targets": {
"worker": list(launchpad_workers.get("target_node_codes") or []),
"control": list(launchpad_controls.get("target_node_codes") or []),
},
"package_preview": {
"available": bool(latest_package_preview.get("release")),
"preview_source": ensure_str(latest_package_preview.get("preview_source")),
"message": ensure_str(latest_package_preview.get("message")),
},
}
result["next_actions"]["primary_command"] = ensure_str(
ensure_dict(result.get("recommended_commands")).get(primary_command_key)
)
result["next_actions"]["secondary_command"] = ensure_str(
ensure_dict(result.get("recommended_commands")).get(secondary_command_key)
)
print(json.dumps(result, ensure_ascii=False, indent=2))
PY
echo

View File

@@ -0,0 +1,373 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
INPUT_RELEASE_ID="${2:-}"
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
curl_json() {
curl -fsS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
"$1"
}
curl_post_json() {
local url="$1"
local body="$2"
curl -fsS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
-X POST "${url}" \
-H 'Content-Type: application/json' \
-d "${body}"
}
launchpad="$(curl_json "${BASE_URL}/api/v1/ops/releases/launchpad")"
latest_release="$(curl_json "${BASE_URL}/api/v1/ops/releases/latest")"
action_templates="$(curl_json "${BASE_URL}/api/v1/ops/action-templates")"
cluster_json="$(curl_json "${BASE_URL}/api/v1/runtime/cluster")"
release_id="${INPUT_RELEASE_ID}"
if [[ -z "${release_id}" ]]; then
release_id="$(python3 -c '
import json
import sys
payload = json.loads(sys.argv[1])
data = payload.get("data") or {}
value = data.get("id") or 0
print(value)
' "${latest_release}")"
fi
selected_release="${latest_release}"
if [[ -n "${release_id}" && "${release_id}" != "0" ]]; then
selected_release="$(curl_json "${BASE_URL}/api/v1/ops/releases/${release_id}")"
fi
build_release_preview_payload() {
local preview_mode="$1"
python3 - <<'PY' "${selected_release}" "${cluster_json}" "${preview_mode}"
import json
import sys
release_payload = json.loads(sys.argv[1]).get("data") or {}
cluster_payload = json.loads(sys.argv[2]).get("data") or {}
preview_mode = str(sys.argv[3] or "").strip().lower()
nodes = list(cluster_payload.get("nodes") or [])
def is_online(node):
return str(node.get("status") or "").strip() in {"online", "busy"}
control_nodes = [
str(node.get("node_code") or "").strip()
for node in nodes
if is_online(node) and str(node.get("role") or "").strip() == "control"
]
worker_nodes = [
str(node.get("node_code") or "").strip()
for node in nodes
if is_online(node)
and str(node.get("role") or "").strip() == "worker"
and bool(node.get("is_effective_worker", False))
]
online_nodes = [
str(node.get("node_code") or "").strip()
for node in nodes
if is_online(node)
]
payload = {
"release_version": str(release_payload.get("release_version") or "").strip(),
"artifact_url": str(release_payload.get("artifact_url") or "").strip(),
"checksum": str(release_payload.get("checksum") or "").strip(),
"install_root": "/opt/domaincheck",
"health_check_timeout_seconds": 10,
"health_check_retries": 2,
"health_check_interval_seconds": 2,
"rollback_on_failure": True,
"switch_current": True,
}
if preview_mode == "control":
template_key = "deploy.release.control"
target_node_codes = control_nodes
payload.update(
{
"restart_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"],
"health_check_urls": ["http://127.0.0.1:8100/health"],
"health_check_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"],
}
)
elif preview_mode == "worker":
template_key = "deploy.release.worker"
target_node_codes = worker_nodes
payload.update(
{
"restart_services": ["domaincheck-worker"],
"health_check_urls": [],
"health_check_services": ["domaincheck-worker"],
}
)
else:
template_key = "deploy.release.custom"
target_node_codes = (control_nodes[:1] + worker_nodes[:2]) or online_nodes[:3]
payload.update(
{
"restart_services": [],
"health_check_urls": [],
"health_check_services": [],
}
)
request_payload = {
"template_key": template_key,
"action": "deploy.release",
"target_type": "batch",
"execution_mode": "remote-agent",
"target_node_codes": [item for item in target_node_codes if item],
"payload": payload,
}
print(json.dumps(request_payload, ensure_ascii=False))
PY
}
print_preview_request_summary() {
local preview_payload="$1"
python3 - <<'PY' "${preview_payload}"
import json
import sys
payload = json.loads(sys.argv[1])
body = payload.get("payload") or {}
print(
json.dumps(
{
"template_key": payload.get("template_key") or "",
"action": payload.get("action") or "",
"execution_mode": payload.get("execution_mode") or "",
"target_node_codes": payload.get("target_node_codes") or [],
"release_version": body.get("release_version") or "",
"artifact_url_present": bool(str(body.get("artifact_url") or "").strip()),
"checksum_present": bool(str(body.get("checksum") or "").strip()),
"restart_services": body.get("restart_services") or [],
"health_check_urls": body.get("health_check_urls") or [],
"health_check_services": body.get("health_check_services") or [],
},
ensure_ascii=False,
indent=2,
)
)
PY
}
run_release_preview() {
local step_no="$1"
local preview_mode="$2"
local section_title="$3"
local preview_payload
preview_payload="$(build_release_preview_payload "${preview_mode}")"
echo "[${step_no}/10] ${section_title}"
print_preview_request_summary "${preview_payload}"
echo
curl_post_json "${BASE_URL}/api/v1/ops/policy/preview" "${preview_payload}"
echo
echo
}
echo "[1/11] release launchpad"
printf '%s\n\n' "${launchpad}"
echo "[2/11] latest release"
printf '%s\n\n' "${selected_release}"
echo "[3/11] recent releases"
curl_json "${BASE_URL}/api/v1/ops/releases?limit=8"
echo
echo
echo "[4/11] release deploy templates"
python3 - <<'PY' "${action_templates}"
import json
import sys
payload = json.loads(sys.argv[1])
groups = (payload.get("data") or {}).get("groups") or []
release_group = next((item for item in groups if str(item.get("group_key") or "") == "release"), {})
items = list(release_group.get("items") or [])
summary = {
"group_key": release_group.get("group_key") or "",
"group_title": release_group.get("group_title") or "",
"template_keys": [str(item.get("key") or "") for item in items],
"field_types": sorted(
{
str(field.get("type") or "")
for item in items
for field in list(item.get("fields") or [])
if str(field.get("type") or "")
}
),
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
echo
echo "[5/11] online release target summary"
python3 - <<'PY' "${cluster_json}"
import json
import sys
payload = json.loads(sys.argv[1]).get("data") or {}
nodes = list(payload.get("nodes") or [])
def is_online(node):
return str(node.get("status") or "").strip() in {"online", "busy"}
summary = {
"online_control_nodes": [
str(node.get("node_code") or "").strip()
for node in nodes
if is_online(node) and str(node.get("role") or "").strip() == "control"
],
"online_worker_nodes": [
str(node.get("node_code") or "").strip()
for node in nodes
if is_online(node) and str(node.get("role") or "").strip() == "worker" and bool(node.get("is_effective_worker", False))
],
"all_online_nodes": [
{
"node_code": str(node.get("node_code") or "").strip(),
"role": str(node.get("role") or "").strip(),
"status": str(node.get("status") or "").strip(),
"is_effective_worker": bool(node.get("is_effective_worker", False)),
}
for node in nodes
if is_online(node)
],
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
echo
echo
run_release_preview "6" "control" "deploy.release.control preview"
run_release_preview "7" "worker" "deploy.release.worker preview"
run_release_preview "8" "custom" "deploy.release.custom preview"
rollout_list='{"code":0,"message":"no release selected","data":{"rollouts":[]}}'
rollout_id="0"
if [[ -n "${release_id}" && "${release_id}" != "0" ]]; then
rollout_list="$(curl_json "${BASE_URL}/api/v1/ops/releases/${release_id}/rollouts?limit=10")"
rollout_id="$(python3 -c '
import json
import sys
payload = json.loads(sys.argv[1])
rollouts = (payload.get("data") or {}).get("rollouts") or []
print((rollouts[0] or {}).get("id") or 0 if rollouts else 0)
' "${rollout_list}")"
fi
echo "[9/11] rollout list for release ${release_id:-0}"
printf '%s\n\n' "${rollout_list}"
echo "[10/11] primary rollout detail"
if [[ "${rollout_id}" != "0" ]]; then
curl_json "${BASE_URL}/api/v1/ops/rollouts/${rollout_id}"
else
echo '{"code":0,"message":"no rollout","data":{}}'
fi
echo
echo
echo "[11/12] primary rollout jobs"
if [[ "${rollout_id}" != "0" ]]; then
curl_json "${BASE_URL}/api/v1/ops/rollouts/${rollout_id}/jobs?limit=50"
else
echo '{"code":0,"message":"no rollout","data":{"jobs":[]}}'
fi
echo
echo
echo "[12/12] condensed summary"
python3 - <<'PY' "${launchpad}" "${selected_release}" "${cluster_json}" "${rollout_list}" "${rollout_id}"
import json
import sys
launchpad_payload = json.loads(sys.argv[1] or "{}").get("data") or {}
selected_release = json.loads(sys.argv[2] or "{}").get("data") or {}
cluster_payload = json.loads(sys.argv[3] or "{}").get("data") or {}
rollout_list_payload = json.loads(sys.argv[4] or "{}").get("data") or {}
rollout_id = int(sys.argv[5] or 0)
launchpad_status = dict(launchpad_payload.get("launchpad_status") or {})
worker_preview = dict(launchpad_payload.get("worker_rollout_preview") or {})
worker_smart_rollout = dict(worker_preview.get("smart_rollout") or {})
control_preview = dict(launchpad_payload.get("control_rollout_preview") or {})
control_smart_rollout = dict(control_preview.get("smart_rollout") or {})
nodes = list(cluster_payload.get("nodes") or [])
rollouts = list(rollout_list_payload.get("rollouts") or [])
def is_online(node):
return str(node.get("status") or "").strip() in {"online", "busy"}
summary = {
"release": {
"id": int(selected_release.get("id") or 0),
"release_version": str(selected_release.get("release_version") or ""),
"channel": str(selected_release.get("channel") or ""),
"status": str(selected_release.get("status") or ""),
},
"launchpad": {
"status": str(launchpad_status.get("status") or ""),
"status_label": str(launchpad_status.get("status_label") or ""),
"summary": str(launchpad_status.get("summary") or ""),
"recommended_action_code": str(launchpad_status.get("recommended_action_code") or ""),
"recommended_target_node_code": str(launchpad_status.get("recommended_target_node_code") or ""),
"recommended_recovery_label": str(launchpad_status.get("recommended_recovery_label") or ""),
"recommended_recovery_summary": str(launchpad_status.get("recommended_recovery_summary") or ""),
"onboarding_bootstrap_pending_nodes": int(launchpad_status.get("onboarding_bootstrap_pending_nodes", 0) or 0),
"onboarding_acceptance_ready_nodes": int(launchpad_status.get("onboarding_acceptance_ready_nodes", 0) or 0),
"recommended_execution_mode": str(launchpad_status.get("recommended_execution_mode") or ""),
"recommended_execution_mode_label": str(launchpad_status.get("recommended_execution_mode_label") or ""),
"focus_ref": dict(launchpad_status.get("focus_ref") or {}),
},
"worker_rollout_preview": {
"available": bool(worker_smart_rollout.get("available", False)),
"blocked": bool(worker_preview.get("blocked", False)),
"requires_confirmation": bool(worker_preview.get("requires_confirmation", False)),
"execution_mode": str(worker_smart_rollout.get("execution_mode") or ""),
"target_node_codes": list(worker_smart_rollout.get("target_node_codes") or []),
},
"control_rollout_preview": {
"available": bool(control_smart_rollout.get("available", False)),
"blocked": bool(control_preview.get("blocked", False)),
"requires_confirmation": bool(control_preview.get("requires_confirmation", False)),
"execution_mode": str(control_smart_rollout.get("execution_mode") or ""),
"target_node_codes": list(control_smart_rollout.get("target_node_codes") or []),
},
"cluster": {
"online_control_nodes": [
str(node.get("node_code") or "").strip()
for node in nodes
if is_online(node) and str(node.get("role") or "").strip() == "control"
],
"online_worker_nodes": [
str(node.get("node_code") or "").strip()
for node in nodes
if is_online(node) and str(node.get("role") or "").strip() == "worker" and bool(node.get("is_effective_worker", False))
],
},
"rollouts": {
"count": len(rollouts),
"selected_rollout_id": rollout_id,
},
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
echo

View File

@@ -4,15 +4,24 @@ set -euo pipefail
MAINLAND_BASE_URL="${1:-http://127.0.0.1:8100}"
OVERSEAS_BASE_URL="${2:-http://152.53.37.118:8100}"
PYTHON_BIN="${PYTHON_BIN:-python3}"
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
curl_json() {
curl -fsS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
"$1"
}
echo "[1/5] mainland runtime cluster"
MAINLAND_CLUSTER="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/cluster")"
MAINLAND_CLUSTER="$(curl_json "${MAINLAND_BASE_URL}/api/v1/runtime/cluster")"
echo "${MAINLAND_CLUSTER}"
echo
echo
echo "[2/5] mainland runtime readiness"
MAINLAND_READINESS="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/readiness")"
MAINLAND_READINESS="$(curl_json "${MAINLAND_BASE_URL}/api/v1/runtime/readiness")"
echo "${MAINLAND_READINESS}"
echo
echo
@@ -79,7 +88,7 @@ echo
echo
echo "[4/5] overseas runtime cluster"
OVERSEAS_CLUSTER="$(curl -fsS "${OVERSEAS_BASE_URL}/api/v1/runtime/cluster")"
OVERSEAS_CLUSTER="$(curl_json "${OVERSEAS_BASE_URL}/api/v1/runtime/cluster")"
echo "${OVERSEAS_CLUSTER}"
echo
echo

View File

@@ -4,23 +4,44 @@ set -euo pipefail
DB_NAME="${DB_NAME:-domain}"
DB_USER="${DB_USER:-postgres}"
PSQL_BIN="${PSQL_BIN:-/www/server/pgsql/bin/psql}"
API_BASE_URL="${1:-${API_BASE_URL:-http://127.0.0.1:8100}}"
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
echo "[1/3] cluster nodes"
curl_json() {
curl -fsS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
"$1"
}
RUNTIME_STATUS_JSON="$(curl_json "${API_BASE_URL}/api/v1/runtime/status")"
echo "[1/4] cluster nodes"
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -Atc "
select
node_code || '|' ||
role || '|' ||
status || '|' ||
coalesce(current_load, 0) || '|' ||
coalesce(metadata_json->>'job_items_claimed', '0') || '|' ||
coalesce(metadata_json->>'job_items_running', '0') || '|' ||
coalesce(metadata_json->>'job_items_completed', '0') || '|' ||
to_char(last_heartbeat_at, 'YYYY-MM-DD HH24:MI:SS') || '|' ||
coalesce(metadata_json->>'worker_online', '') || '|' ||
coalesce(metadata_json->>'detect_participating', '')
(
case
when coalesce(current_load, 0) > 0 then 'true'
when coalesce(metadata_json->>'detect_participating', '') in ('true', 'false') then metadata_json->>'detect_participating'
else ''
end
)
from detect_worker_nodes
order by node_code;
"
echo
echo "[2/3] active job distribution"
echo "[2/4] active job distribution"
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -Atc "
select
coalesce(claimed_by, '') || '|' ||
@@ -39,59 +60,43 @@ order by claimed_by, status;
"
echo
echo "[3/3] condensed summary"
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -At <<'SQL'
with latest_job as (
select id, job_code, status
from detect_jobs
where status in ('pending','running')
order by id desc
limit 1
),
node_stats as (
select
node_code,
role,
status,
coalesce(metadata_json->>'worker_online', '') as worker_online,
coalesce(metadata_json->>'detect_participating', '') as detect_participating
from detect_worker_nodes
),
item_stats as (
select
coalesce(claimed_by, '') as claimed_by,
status,
count(*) as cnt
from detect_job_items
where job_id = (select id from latest_job)
group by claimed_by, status
)
select json_build_object(
'job', (
select json_build_object(
'job_code', coalesce(job_code, ''),
'status', coalesce(status, '')
)
from latest_job
),
'online_nodes', (
select coalesce(json_agg(json_build_object(
'node_code', node_code,
'role', role,
'status', status,
'worker_online', worker_online,
'detect_participating', detect_participating
) order by node_code), '[]'::json)
from node_stats
where status in ('online', 'busy')
),
'claimed_distribution', (
select coalesce(json_agg(json_build_object(
'claimed_by', claimed_by,
'status', status,
'count', cnt
) order by claimed_by, status), '[]'::json)
from item_stats
)
);
SQL
echo "[3/4] runtime participation snapshot"
printf "%s" "${RUNTIME_STATUS_JSON}" | python3 -c '
import json, sys
payload = json.load(sys.stdin).get("data", {})
detect = payload.get("detect", {})
print(json.dumps({
"node": payload.get("node", {}),
"participation_summary": detect.get("participation_summary", {}),
"participating_nodes": detect.get("participating_nodes", []),
"non_participating_nodes": detect.get("non_participating_nodes", detect.get("standby_nodes", [])),
"log_sync": detect.get("log_sync", {}),
}, ensure_ascii=False, indent=2))
'
echo
echo "[4/4] condensed summary"
printf "%s" "${RUNTIME_STATUS_JSON}" | python3 -c '
import json, sys
payload = json.load(sys.stdin).get("data", {})
detect = payload.get("detect", {})
summary = detect.get("participation_summary", {}) or {}
print(json.dumps({
"node": payload.get("node", {}),
"active_job": {
"job_code": (detect.get("active_job") or {}).get("job_code", ""),
"status": (detect.get("active_job") or {}).get("status", ""),
},
"participation_summary": summary,
"dispatch_active_nodes": summary.get("dispatch_active_node_codes", []),
"recent_only_nodes": summary.get("recent_only_node_codes", []),
"non_participating_nodes": summary.get("non_participating_node_codes", []),
"log_sync": {
"enabled": bool((detect.get("log_sync") or {}).get("enabled", False)),
"mode": (detect.get("log_sync") or {}).get("mode", ""),
"line_count": (detect.get("log_sync") or {}).get("line_count", 0),
"source_nodes": (detect.get("log_sync") or {}).get("source_nodes", []),
"last_at": (detect.get("log_sync") or {}).get("last_at", ""),
},
}, ensure_ascii=False, indent=2))
'

View File

@@ -0,0 +1,447 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${SCRIPT_DIR}/lib_ops_center.sh"
ops_center_load_config "${SCRIPT_DIR}"
usage() {
cat <<'EOF'
usage:
bash domain-api/deploy/multi-region/drive_ops_action.sh <base_url> <preview|execute> <action_code> [action_payload_json|@payload.json] [requested_by]
bash domain-api/deploy/multi-region/drive_ops_action.sh <base_url> <action_code> [action_payload_json|@payload.json] [requested_by]
examples:
bash domain-api/deploy/multi-region/drive_ops_action.sh \
http://127.0.0.1:8100 \
preview \
create_release_rollout_worker \
'{"release_id":12}'
bash domain-api/deploy/multi-region/drive_ops_action.sh \
http://127.0.0.1:8100 \
create_release_rollout_worker \
'{"release_id":12}'
NODE_CODES="mainland-worker-01,mainland-worker-02" \
bash domain-api/deploy/multi-region/drive_ops_action.sh \
http://127.0.0.1:8100 \
run_standard_inspection \
'{}'
notes:
- preview/execute 双模式共用同一份 action payload 和 node_codes
- 旧用法仍默认按 execute 处理
- NODE_CODES can be passed through env as comma/newline separated node_code list
- action_payload_json can also use @/path/to/file.json
- preview metadata can be passed through env:
SOURCE_LABEL / ACTION_TITLE / ACTION_SUMMARY / ACTION_REASON / ACTION_META_TEXT / PRIMARY_LABEL
EOF
}
BASE_URL="${1:-$(ops_center_resolve_mainland_api_base_url)}"
SECOND_ARG="${2:-}"
THIRD_ARG="${3:-}"
FOURTH_ARG="${4:-}"
FIFTH_ARG="${5:-}"
if [[ "${SECOND_ARG}" == "preview" || "${SECOND_ARG}" == "execute" ]]; then
OPERATION="${SECOND_ARG}"
ACTION_CODE="${THIRD_ARG:-}"
RAW_ACTION_PAYLOAD="${FOURTH_ARG:-}"
REQUESTED_BY="${FIFTH_ARG:-$(ops_center_resolve_requested_by)}"
else
OPERATION="execute"
ACTION_CODE="${SECOND_ARG:-}"
RAW_ACTION_PAYLOAD="${THIRD_ARG:-}"
REQUESTED_BY="${FOURTH_ARG:-$(ops_center_resolve_requested_by)}"
fi
if [[ -z "${RAW_ACTION_PAYLOAD}" ]]; then
RAW_ACTION_PAYLOAD='{}'
fi
NODE_CODES_INPUT="${NODE_CODES:-}"
SOURCE_LABEL="${SOURCE_LABEL:-cli/drive_ops_action}"
ACTION_TITLE="${ACTION_TITLE:-}"
ACTION_SUMMARY="${ACTION_SUMMARY:-}"
ACTION_REASON="${ACTION_REASON:-}"
ACTION_META_TEXT="${ACTION_META_TEXT:-}"
PRIMARY_LABEL="${PRIMARY_LABEL:-}"
print_json_or_raw() {
local raw_text="${1:-}"
python3 - "${raw_text}" <<'PY'
import json
import sys
raw_text = str(sys.argv[1] or "")
try:
payload = json.loads(raw_text)
except Exception:
print(raw_text)
else:
print(json.dumps(payload, ensure_ascii=False, indent=2))
PY
}
request_with_status() {
local method="$1"
local url="$2"
local body="${3:-}"
if [[ "${method}" == "POST" ]]; then
curl -sS -X POST "${url}" \
-H 'Content-Type: application/json' \
-d "${body}" \
-w $'\nHTTP_STATUS=%{http_code}'
else
curl -sS "${url}" -w $'\nHTTP_STATUS=%{http_code}'
fi
}
if [[ -z "${ACTION_CODE}" ]]; then
usage
exit 1
fi
if [[ "${OPERATION}" != "preview" && "${OPERATION}" != "execute" ]]; then
echo "operation must be preview or execute" >&2
usage
exit 1
fi
ACTION_PAYLOAD_JSON="${RAW_ACTION_PAYLOAD}"
if [[ "${ACTION_PAYLOAD_JSON}" == @* ]]; then
ACTION_PAYLOAD_PATH="${ACTION_PAYLOAD_JSON#@}"
if [[ ! -f "${ACTION_PAYLOAD_PATH}" ]]; then
echo "payload file not found: ${ACTION_PAYLOAD_PATH}" >&2
exit 1
fi
ACTION_PAYLOAD_JSON="$(cat "${ACTION_PAYLOAD_PATH}")"
fi
if [[ -z "${NODE_CODES_INPUT}" ]]; then
NODE_CODES_INPUT="$(python3 - <<'PY' "${BASE_URL}" "${ACTION_CODE}" "${ACTION_PAYLOAD_JSON}"
import json
import sys
import urllib.request
def normalize_codes(*groups):
result = []
for group in groups:
for item in list(group or []):
node_code = str(item or "").strip()
if node_code and node_code not in result:
result.append(node_code)
return result
base_url = str(sys.argv[1] or "").rstrip("/")
action_code = str(sys.argv[2] or "").strip()
payload_raw = str(sys.argv[3] or "").strip() or "{}"
try:
payload = json.loads(payload_raw)
except Exception:
payload = {}
if not isinstance(payload, dict):
payload = {}
payload_node_codes = normalize_codes(payload.get("node_codes") or [])
if payload_node_codes:
print(",".join(payload_node_codes))
raise SystemExit(0)
supported_actions = {
"open_worker_logs_participating",
"run_inspection_participating",
"open_worker_logs_standby",
"run_inspection_standby",
"open_worker_logs",
"run_standard_inspection",
}
if action_code not in supported_actions or not base_url:
print("")
raise SystemExit(0)
try:
with urllib.request.urlopen(f"{base_url}/api/v1/ops/overview", timeout=20) as response:
overview_payload = json.loads(response.read().decode("utf-8"))
overview = dict((overview_payload or {}).get("data") or {})
except Exception:
print("")
raise SystemExit(0)
execution_scene = dict(overview.get("execution_scene") or {})
participation_summary = dict(execution_scene.get("participation_summary") or {})
log_sync = dict(execution_scene.get("log_sync") or {})
dispatch_active = normalize_codes(participation_summary.get("dispatch_active_node_codes") or [])
recent_only = normalize_codes(participation_summary.get("recent_only_node_codes") or [])
standby = normalize_codes(
participation_summary.get("standby_node_codes") or [],
participation_summary.get("non_participating_node_codes") or [],
)
missing_logs = normalize_codes(log_sync.get("missing_participating_nodes") or [])
resolved_node_codes = []
if action_code in {"open_worker_logs_participating", "run_inspection_participating"}:
resolved_node_codes = normalize_codes(missing_logs, dispatch_active, recent_only)
elif action_code in {"open_worker_logs_standby", "run_inspection_standby"}:
resolved_node_codes = normalize_codes(standby)
elif action_code == "open_worker_logs":
resolved_node_codes = normalize_codes(missing_logs, dispatch_active, recent_only, standby)
elif action_code == "run_standard_inspection":
resolved_node_codes = normalize_codes(dispatch_active, recent_only, standby)
print(",".join(resolved_node_codes))
PY
)"
fi
REQUEST_JSON="$(python3 - <<'PY' "${ACTION_CODE}" "${ACTION_PAYLOAD_JSON}" "${REQUESTED_BY}" "${NODE_CODES_INPUT}"
import json
import sys
action_code = str(sys.argv[1] or "").strip()
payload_raw = str(sys.argv[2] or "").strip() or "{}"
requested_by = str(sys.argv[3] or "").strip() or "cli/drive_ops_action"
node_codes_raw = str(sys.argv[4] or "")
try:
action_payload = json.loads(payload_raw)
except Exception as exc: # pragma: no cover - cli validation
raise SystemExit(f"invalid action payload json: {exc}")
if not isinstance(action_payload, dict):
raise SystemExit("action payload must be a json object")
node_codes = []
seen = set()
for item in node_codes_raw.replace("\n", ",").split(","):
value = str(item or "").strip()
if not value or value in seen:
continue
seen.add(value)
node_codes.append(value)
request_payload = {
"action_code": action_code,
"node_codes": node_codes,
"action_payload": action_payload,
"requested_by": requested_by,
}
print(json.dumps(request_payload, ensure_ascii=False))
PY
)"
PREVIEW_REQUEST_JSON="$(python3 - <<'PY' "${ACTION_CODE}" "${ACTION_PAYLOAD_JSON}" "${REQUESTED_BY}" "${NODE_CODES_INPUT}" "${SOURCE_LABEL}" "${ACTION_TITLE}" "${ACTION_SUMMARY}" "${ACTION_REASON}" "${ACTION_META_TEXT}" "${PRIMARY_LABEL}"
import json
import sys
action_code = str(sys.argv[1] or "").strip()
payload_raw = str(sys.argv[2] or "").strip() or "{}"
requested_by = str(sys.argv[3] or "").strip() or "cli/drive_ops_action"
node_codes_raw = str(sys.argv[4] or "")
source_label = str(sys.argv[5] or "").strip() or "cli/drive_ops_action"
action_title = str(sys.argv[6] or "").strip()
action_summary = str(sys.argv[7] or "").strip()
action_reason = str(sys.argv[8] or "").strip()
action_meta_text = str(sys.argv[9] or "").strip()
primary_label = str(sys.argv[10] or "").strip()
try:
action_payload = json.loads(payload_raw)
except Exception as exc:
raise SystemExit(f"invalid action payload json: {exc}")
if not isinstance(action_payload, dict):
raise SystemExit("action payload must be a json object")
node_codes = []
seen = set()
for item in node_codes_raw.replace("\n", ",").split(","):
value = str(item or "").strip()
if not value or value in seen:
continue
seen.add(value)
node_codes.append(value)
request_payload = {
"source_label": source_label,
"title": action_title or primary_label or action_code,
"summary": action_summary,
"reason": action_reason,
"meta_text": action_meta_text,
"executor_kind": "driver_action",
"primary_label": primary_label or action_code,
"primary_action_code": action_code,
"primary_node_codes": node_codes,
"primary_action_payload": action_payload,
"requested_by": requested_by,
}
print(json.dumps(request_payload, ensure_ascii=False))
PY
)"
if [[ "${OPERATION}" == "preview" ]]; then
echo "[1/3] driver action preview request"
print_json_or_raw "${PREVIEW_REQUEST_JSON}"
echo
echo
RAW_HTTP_RESPONSE="$(request_with_status "POST" "${BASE_URL}/api/v1/ops/driver-actions/preview" "${PREVIEW_REQUEST_JSON}")"
HTTP_STATUS="${RAW_HTTP_RESPONSE##*$'\n'HTTP_STATUS=}"
RESPONSE_JSON="${RAW_HTTP_RESPONSE%$'\n'HTTP_STATUS=*}"
echo "[2/3] raw response"
printf '%s\n' "http_status=${HTTP_STATUS}"
print_json_or_raw "${RESPONSE_JSON}"
echo
echo
echo "[3/3] condensed summary"
python3 - "${RESPONSE_JSON}" "${HTTP_STATUS}" <<'PY'
import json
import sys
payload = json.loads(sys.argv[1])
http_status = int(sys.argv[2] or 0)
data = payload.get("data") or {}
primary = data.get("primary") or {}
secondary = data.get("secondary") or {}
raw_code = payload.get("code", 1)
if raw_code in (0, "0", False):
normalized_code = 0
else:
normalized_code = int(raw_code or 1)
summary = {
"http_status": http_status,
"code": normalized_code,
"message": str(payload.get("message") or ""),
"contract_key": str(data.get("contract_key") or ""),
"contract_version": str(data.get("contract_version") or ""),
"preview_endpoint": str(data.get("preview_endpoint") or ""),
"source_label": str(data.get("source_label") or ""),
"title": str(data.get("title") or ""),
"executor_kind": str(data.get("executor_kind") or ""),
"execution_chain": str(data.get("execution_chain") or ""),
"primary": {
"label": str(primary.get("label") or ""),
"action_code": str(primary.get("action_code") or ""),
"target_method": str(primary.get("target_method") or ""),
"target_api": str(primary.get("target_api") or ""),
"recommendation_label": str(primary.get("recommendation_label") or ""),
"automation_level_label": str(primary.get("automation_level_label") or ""),
"risk_level_label": str(primary.get("risk_level_label") or ""),
"node_codes": list(primary.get("node_codes") or []),
"request_payload": primary.get("request_payload") or {},
},
}
if secondary:
summary["secondary"] = {
"label": str(secondary.get("label") or ""),
"action_code": str(secondary.get("action_code") or ""),
"target_method": str(secondary.get("target_method") or ""),
"target_api": str(secondary.get("target_api") or ""),
"recommendation_label": str(secondary.get("recommendation_label") or ""),
"automation_level_label": str(secondary.get("automation_level_label") or ""),
"risk_level_label": str(secondary.get("risk_level_label") or ""),
"node_codes": list(secondary.get("node_codes") or []),
"request_payload": secondary.get("request_payload") or {},
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
else
echo "[1/3] driver action request"
print_json_or_raw "${REQUEST_JSON}"
echo
echo
RAW_HTTP_RESPONSE="$(request_with_status "POST" "${BASE_URL}/api/v1/ops/driver-actions/execute" "${REQUEST_JSON}")"
HTTP_STATUS="${RAW_HTTP_RESPONSE##*$'\n'HTTP_STATUS=}"
RESPONSE_JSON="${RAW_HTTP_RESPONSE%$'\n'HTTP_STATUS=*}"
echo "[2/3] raw response"
printf '%s\n' "http_status=${HTTP_STATUS}"
print_json_or_raw "${RESPONSE_JSON}"
echo
echo
echo "[3/3] condensed summary"
python3 - "${RESPONSE_JSON}" "${HTTP_STATUS}" <<'PY'
import json
import sys
payload = json.loads(sys.argv[1])
http_status = int(sys.argv[2] or 0)
data = payload.get("data") or {}
smart_release_result = data.get("smart_release_result") or {}
ui_intent = data.get("ui_intent") or {}
runtime_settings = data.get("runtime_settings") or {}
raw_code = payload.get("code", 1)
if raw_code in (0, "0", False):
normalized_code = 0
else:
normalized_code = int(raw_code or 1)
summary = {
"http_status": http_status,
"code": normalized_code,
"message": str(payload.get("message") or ""),
"handled": bool(data.get("handled", False)),
"action_code": str(data.get("action_code") or ""),
"mode": str(data.get("mode") or ""),
}
if smart_release_result:
release = smart_release_result.get("release") or {}
rollout = smart_release_result.get("rollout") or {}
smart_plan = smart_release_result.get("smart_rollout") or {}
preview = smart_release_result.get("preview") or {}
summary["smart_release_result"] = {
"release_id": int(release.get("id") or 0),
"release_version": str(release.get("release_version") or ""),
"release_created": bool(smart_release_result.get("release_created", False)),
"release_deduplicated": bool(smart_release_result.get("release_deduplicated", False)),
"rollout_created": bool(smart_release_result.get("rollout_created", False)),
"requires_confirmation": bool(smart_release_result.get("requires_confirmation", False)),
"blocked": bool(smart_release_result.get("blocked", False)),
"rollout_id": int(rollout.get("id") or 0),
"rollout_code": str(rollout.get("rollout_code") or ""),
"smart_mode": str(smart_plan.get("mode") or ""),
"smart_available": bool(smart_plan.get("available", False)),
"target_node_codes": list((smart_plan.get("target_selector") or {}).get("node_codes") or []),
"preview_warnings": list(preview.get("warnings") or []),
"preview_approval_required": bool(preview.get("approval_required", False)),
}
if ui_intent:
summary["ui_intent"] = {
"kind": str(ui_intent.get("kind") or ""),
"release_id": int(ui_intent.get("release_id") or 0),
"template_key": str(ui_intent.get("template_key") or ""),
"playbook_key": str(ui_intent.get("playbook_key") or ""),
"target_node_codes": list(ui_intent.get("target_node_codes") or []),
}
if runtime_settings:
summary["runtime_settings"] = {
"worker_log_sync_enabled": bool(runtime_settings.get("worker_log_sync_enabled", False)),
"worker_log_sync_mode": str(runtime_settings.get("worker_log_sync_mode") or ""),
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
fi
if [[ "${HTTP_STATUS}" -lt 200 || "${HTTP_STATUS}" -ge 300 ]]; then
exit 1
fi

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,476 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${SCRIPT_DIR}/lib_ops_center.sh"
ops_center_load_config "${SCRIPT_DIR}"
usage() {
cat <<'EOF'
usage:
bash domain-api/deploy/multi-region/drive_release_hub.sh <base_url> <command> [args...]
commands:
launchpad
latest-package
latest-release
preview-latest
preview-latest-smart <worker|control>
preview-release <release_id> <worker|control>
create-latest-release
list-releases [limit]
list-rollouts <release_id> [limit]
smart-latest <worker|control> [confirm]
smart-release <release_id> <worker|control> [confirm]
examples:
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 launchpad
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 latest-package
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 preview-latest
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 preview-latest-smart worker
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 create-latest-release
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 smart-latest worker
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 preview-release 18 control
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 smart-release 18 control confirm
EOF
}
BASE_URL="${1:-$(ops_center_resolve_mainland_api_base_url)}"
COMMAND="${2:-}"
ARG1="${3:-}"
ARG2="${4:-}"
ARG3="${5:-}"
REQUESTED_BY="${REQUESTED_BY:-$(ops_center_resolve_requested_by)}"
print_json_or_raw() {
local raw_text="${1:-}"
python3 - "${raw_text}" <<'PY'
import json
import sys
raw_text = str(sys.argv[1] or "")
try:
payload = json.loads(raw_text)
except Exception:
print(raw_text)
else:
print(json.dumps(payload, ensure_ascii=False, indent=2))
PY
}
request_with_status() {
local method="$1"
local url="$2"
local body="${3:-}"
if [[ "${method}" == "POST" ]]; then
curl -sS -X POST "${url}" \
-H 'Content-Type: application/json' \
-d "${body}" \
-w $'\nHTTP_STATUS=%{http_code}'
else
curl -sS "${url}" -w $'\nHTTP_STATUS=%{http_code}'
fi
}
if [[ -z "${COMMAND}" ]]; then
usage
exit 1
fi
METHOD="GET"
PATH_SUFFIX=""
REQUEST_JSON=""
confirm_to_bool() {
local raw_value="${1:-}"
case "$(printf '%s' "${raw_value}" | tr '[:upper:]' '[:lower:]')" in
1|true|yes|y|confirm|confirmed)
printf 'true'
;;
*)
printf 'false'
;;
esac
}
case "${COMMAND}" in
launchpad)
CHANNEL="${ARG1:-stable}"
PATH_SUFFIX="/api/v1/ops/releases/launchpad?channel=${CHANNEL}"
;;
latest-package)
PATH_SUFFIX="/api/v1/ops/releases/package-metadata/latest"
;;
latest-release)
PATH_SUFFIX="/api/v1/ops/releases/latest"
;;
preview-latest)
METHOD="POST"
PATH_SUFFIX="/api/v1/ops/releases/from-package/latest/preview"
REQUEST_JSON="$(python3 - <<'PY' "${REQUESTED_BY}"
import json
import sys
requested_by = str(sys.argv[1] or "").strip() or "cli/drive_release_hub"
payload = {
"created_by": f"{requested_by}/latest-package-preview",
}
print(json.dumps(payload, ensure_ascii=False))
PY
)"
;;
preview-latest-smart)
MODE="${ARG1:-}"
if [[ "${MODE}" != "worker" && "${MODE}" != "control" ]]; then
echo "mode must be worker or control" >&2
usage
exit 1
fi
METHOD="POST"
PATH_SUFFIX="/api/v1/ops/releases/from-package/latest/smart-rollout-preview"
REQUEST_JSON="$(python3 - <<'PY' "${MODE}" "${REQUESTED_BY}"
import json
import sys
mode = str(sys.argv[1] or "").strip()
requested_by = str(sys.argv[2] or "").strip() or "cli/drive_release_hub"
payload = {
"mode": mode,
"created_by": f"{requested_by}/latest-package-preview",
"rollout_created_by": f"{requested_by}/latest-package-preview/rollout",
}
print(json.dumps(payload, ensure_ascii=False))
PY
)"
;;
create-latest-release)
METHOD="POST"
PATH_SUFFIX="/api/v1/ops/releases/from-package/latest"
REQUEST_JSON="$(python3 - <<'PY' "${REQUESTED_BY}"
import json
import sys
requested_by = str(sys.argv[1] or "").strip() or "cli/drive_release_hub"
payload = {
"created_by": f"{requested_by}/latest-package",
}
print(json.dumps(payload, ensure_ascii=False))
PY
)"
;;
list-releases)
RELEASE_LIMIT="${ARG1:-10}"
PATH_SUFFIX="/api/v1/ops/releases?limit=${RELEASE_LIMIT}"
;;
list-rollouts)
RELEASE_ID="${ARG1:-}"
ROLLOUT_LIMIT="${ARG2:-10}"
if [[ -z "${RELEASE_ID}" ]]; then
echo "release_id is required for list-rollouts" >&2
usage
exit 1
fi
PATH_SUFFIX="/api/v1/ops/releases/${RELEASE_ID}/rollouts?limit=${ROLLOUT_LIMIT}"
;;
smart-latest)
MODE="${ARG1:-}"
if [[ "${MODE}" != "worker" && "${MODE}" != "control" ]]; then
echo "mode must be worker or control" >&2
usage
exit 1
fi
METHOD="POST"
PATH_SUFFIX="/api/v1/ops/releases/from-package/latest/smart-rollout"
CONFIRM_RISKY="$(confirm_to_bool "${ARG2:-}")"
REQUEST_JSON="$(python3 - <<'PY' "${MODE}" "${CONFIRM_RISKY}" "${REQUESTED_BY}"
import json
import sys
mode = str(sys.argv[1] or "").strip()
confirm_risky = str(sys.argv[2] or "").strip().lower() == "true"
requested_by = str(sys.argv[3] or "").strip() or "cli/drive_release_hub"
payload = {
"mode": mode,
"confirm_risky": confirm_risky,
"created_by": f"{requested_by}/latest-package",
"rollout_created_by": f"{requested_by}/latest-package/rollout",
}
print(json.dumps(payload, ensure_ascii=False))
PY
)"
;;
preview-release)
RELEASE_ID="${ARG1:-}"
MODE="${ARG2:-}"
if [[ -z "${RELEASE_ID}" ]]; then
echo "release_id is required for preview-release" >&2
usage
exit 1
fi
if [[ "${MODE}" != "worker" && "${MODE}" != "control" ]]; then
echo "mode must be worker or control" >&2
usage
exit 1
fi
METHOD="POST"
PATH_SUFFIX="/api/v1/ops/releases/${RELEASE_ID}/smart-rollout-preview"
REQUEST_JSON="$(python3 - <<'PY' "${MODE}" "${REQUESTED_BY}"
import json
import sys
mode = str(sys.argv[1] or "").strip()
requested_by = str(sys.argv[2] or "").strip() or "cli/drive_release_hub"
payload = {
"mode": mode,
"created_by": f"{requested_by}/existing-release-preview",
"rollout_created_by": f"{requested_by}/existing-release-preview/rollout",
}
print(json.dumps(payload, ensure_ascii=False))
PY
)"
;;
smart-release)
RELEASE_ID="${ARG1:-}"
MODE="${ARG2:-}"
if [[ -z "${RELEASE_ID}" ]]; then
echo "release_id is required for smart-release" >&2
usage
exit 1
fi
if [[ "${MODE}" != "worker" && "${MODE}" != "control" ]]; then
echo "mode must be worker or control" >&2
usage
exit 1
fi
METHOD="POST"
PATH_SUFFIX="/api/v1/ops/releases/${RELEASE_ID}/smart-rollout"
CONFIRM_RISKY="$(confirm_to_bool "${ARG3:-}")"
REQUEST_JSON="$(python3 - <<'PY' "${MODE}" "${CONFIRM_RISKY}" "${REQUESTED_BY}"
import json
import sys
mode = str(sys.argv[1] or "").strip()
confirm_risky = str(sys.argv[2] or "").strip().lower() == "true"
requested_by = str(sys.argv[3] or "").strip() or "cli/drive_release_hub"
payload = {
"mode": mode,
"confirm_risky": confirm_risky,
"created_by": f"{requested_by}/existing-release",
"rollout_created_by": f"{requested_by}/existing-release/rollout",
}
print(json.dumps(payload, ensure_ascii=False))
PY
)"
;;
*)
echo "unknown command: ${COMMAND}" >&2
usage
exit 1
;;
esac
echo "[1/3] request"
printf '%s\n' "method=${METHOD}"
printf '%s\n' "url=${BASE_URL}${PATH_SUFFIX}"
if [[ -n "${REQUEST_JSON}" ]]; then
print_json_or_raw "${REQUEST_JSON}"
else
echo "(no request body)"
fi
echo
echo
RAW_HTTP_RESPONSE="$(request_with_status "${METHOD}" "${BASE_URL}${PATH_SUFFIX}" "${REQUEST_JSON}")"
HTTP_STATUS="${RAW_HTTP_RESPONSE##*$'\n'HTTP_STATUS=}"
RESPONSE_JSON="${RAW_HTTP_RESPONSE%$'\n'HTTP_STATUS=*}"
echo "[2/3] raw response"
printf '%s\n' "http_status=${HTTP_STATUS}"
print_json_or_raw "${RESPONSE_JSON}"
echo
echo
echo "[3/3] condensed summary"
python3 - "${RESPONSE_JSON}" "${HTTP_STATUS}" <<'PY'
import json
import sys
raw_response = str(sys.argv[1] or "")
http_status = int(sys.argv[2] or 0)
try:
payload = json.loads(raw_response)
except Exception:
print(json.dumps({
"http_status": http_status,
"code": 1,
"message": "response is not valid json",
"raw_response": raw_response[:500],
}, ensure_ascii=False, indent=2))
sys.exit(0 if 200 <= http_status < 300 else 1)
data = payload.get("data") or {}
raw_code = payload.get("code", 1)
if raw_code in (0, "0", False):
normalized_code = 0
else:
normalized_code = int(raw_code or 1)
summary = {
"http_status": http_status,
"code": normalized_code,
"message": str(payload.get("message") or ""),
}
if isinstance(data, dict) and "package" in data:
package = dict(data.get("package") or {})
summary["package"] = {
"available": bool(package.get("available", False)),
"package_name": str(package.get("package_name") or ""),
"package_type": str(package.get("package_type") or ""),
"generated_at": str(package.get("generated_at") or ""),
"smoke_test_ok": bool(package.get("smoke_test_ok", False)),
"smoke_test_message": str(package.get("smoke_test_message") or ""),
}
if isinstance(data, dict) and "launchpad_status" in data:
launchpad_status = dict(data.get("launchpad_status") or {})
worker_preview = dict(data.get("worker_rollout_preview") or {})
worker_smart_rollout = dict(worker_preview.get("smart_rollout") or {})
control_preview = dict(data.get("control_rollout_preview") or {})
control_smart_rollout = dict(control_preview.get("smart_rollout") or {})
summary["launchpad"] = {
"channel": str(data.get("channel") or ""),
"status": str(launchpad_status.get("status") or ""),
"status_label": str(launchpad_status.get("status_label") or ""),
"summary": str(launchpad_status.get("summary") or ""),
"recommended_action_code": str(launchpad_status.get("recommended_action_code") or ""),
"recommended_target_node_code": str(launchpad_status.get("recommended_target_node_code") or ""),
"recommended_recovery_label": str(launchpad_status.get("recommended_recovery_label") or ""),
"recommended_recovery_summary": str(launchpad_status.get("recommended_recovery_summary") or ""),
"onboarding_bootstrap_pending_nodes": int(launchpad_status.get("onboarding_bootstrap_pending_nodes", 0) or 0),
"onboarding_acceptance_ready_nodes": int(launchpad_status.get("onboarding_acceptance_ready_nodes", 0) or 0),
"recommended_command": str(launchpad_status.get("recommended_command") or ""),
"recommended_execution_mode": str(launchpad_status.get("recommended_execution_mode") or ""),
"recommended_execution_mode_label": str(launchpad_status.get("recommended_execution_mode_label") or ""),
"focus_ref": dict(launchpad_status.get("focus_ref") or {}),
"latest_package": dict(data.get("latest_package") or {}),
"latest_release": {
"id": int(((data.get("latest_release") or {}).get("id") or 0)),
"release_version": str((data.get("latest_release") or {}).get("release_version") or ""),
"status": str((data.get("latest_release") or {}).get("status") or ""),
},
"worker_rollout_preview": {
"message": str(worker_preview.get("message") or ""),
"available": bool(worker_smart_rollout.get("available", False)),
"requires_confirmation": bool(worker_preview.get("requires_confirmation", False)),
"blocked": bool(worker_preview.get("blocked", False)),
"execution_mode": str(worker_smart_rollout.get("execution_mode") or ""),
"execution_mode_label": str(worker_smart_rollout.get("execution_mode_label") or ""),
"target_node_codes": list(worker_smart_rollout.get("target_node_codes") or []),
},
"control_rollout_preview": {
"message": str(control_preview.get("message") or ""),
"available": bool(control_smart_rollout.get("available", False)),
"requires_confirmation": bool(control_preview.get("requires_confirmation", False)),
"blocked": bool(control_preview.get("blocked", False)),
"execution_mode": str(control_smart_rollout.get("execution_mode") or ""),
"execution_mode_label": str(control_smart_rollout.get("execution_mode_label") or ""),
"target_node_codes": list(control_smart_rollout.get("target_node_codes") or []),
},
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
sys.exit(0)
if isinstance(data, dict) and "available" in data:
summary["latest_package"] = {
"available": bool(data.get("available", False)),
"package_name": str(data.get("package_name") or ""),
"package_type": str(data.get("package_type") or ""),
"generated_at": str(data.get("generated_at") or ""),
"commit_sha": str(data.get("commit_sha") or ""),
"smoke_test_ok": bool(data.get("smoke_test_ok", False)),
"smoke_test_message": str(data.get("smoke_test_message") or ""),
"reason": str(data.get("reason") or ""),
}
elif isinstance(data, dict) and "releases" in data:
releases = list(data.get("releases") or [])
summary["releases"] = {
"count": len(releases),
"items": [
{
"id": int(item.get("id") or 0),
"release_version": str(item.get("release_version") or ""),
"channel": str(item.get("channel") or ""),
"status": str(item.get("status") or ""),
}
for item in releases[:5]
],
}
elif isinstance(data, dict) and "rollouts" in data:
rollouts = list(data.get("rollouts") or [])
summary["rollouts"] = {
"count": len(rollouts),
"items": [
{
"id": int(item.get("id") or 0),
"rollout_code": str(item.get("rollout_code") or ""),
"status": str(item.get("status") or ""),
"jobs_total": int(item.get("jobs_total") or 0),
"jobs_created": int(item.get("jobs_created") or 0),
}
for item in rollouts[:5]
],
}
else:
release = data.get("release") or data
rollout = data.get("rollout") or {}
smart_rollout = data.get("smart_rollout") or {}
preview = data.get("preview") or {}
summary["preview_source"] = str(data.get("preview_source") or "")
summary["release"] = {
"id": int(release.get("id") or 0),
"release_version": str(release.get("release_version") or ""),
"channel": str(release.get("channel") or ""),
"status": str(release.get("status") or ""),
"is_preview": bool(release.get("is_preview", False)),
}
if rollout:
summary["rollout"] = {
"id": int(rollout.get("id") or 0),
"rollout_code": str(rollout.get("rollout_code") or ""),
"status": str(rollout.get("status") or ""),
}
if smart_rollout:
summary["smart_rollout"] = {
"available": bool(smart_rollout.get("available", False)),
"mode": str(smart_rollout.get("mode") or ""),
"mode_label": str(smart_rollout.get("mode_label") or ""),
"target_node_codes": list((smart_rollout.get("target_selector") or {}).get("node_codes") or []),
"summary": str(smart_rollout.get("summary") or smart_rollout.get("reason") or ""),
}
if preview:
summary["preview"] = {
"blocked": bool(preview.get("blocked", False)),
"approval_required": bool(preview.get("approval_required", False)),
"warnings": list(preview.get("warnings") or []),
}
summary["result_flags"] = {
"release_created": bool(data.get("release_created", False)),
"release_deduplicated": bool(data.get("release_deduplicated", False)),
"rollout_created": bool(data.get("rollout_created", False)),
"requires_confirmation": bool(data.get("requires_confirmation", False)),
"blocked": bool(data.get("blocked", False)),
}
print(json.dumps(summary, ensure_ascii=False, indent=2))
PY
if [[ "${HTTP_STATUS}" -lt 200 || "${HTTP_STATUS}" -ge 300 ]]; then
exit 1
fi

View File

@@ -0,0 +1,556 @@
#!/usr/bin/env bash
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "${SCRIPT_DIR}/lib_ops_center.sh"
ops_center_load_config "${SCRIPT_DIR}"
MAINLAND_BASE_URL_DEFAULT="$(ops_center_resolve_mainland_api_base_url)"
OVERSEAS_BASE_URL_DEFAULT="$(ops_center_resolve_overseas_api_base_url)"
REPORT_ROOT_DEFAULT="${OPS_CENTER_REPORT_ROOT}/go-live-bundles"
REPORT_DIR_INPUT="${1:-}"
MAINLAND_BASE_URL_INPUT="${2:-}"
OVERSEAS_BASE_URL_INPUT="${3:-}"
timestamp_now() {
date '+%Y%m%d_%H%M%S'
}
normalize_optional_base_url() {
local raw_value="${1:-}"
local fallback_value="${2:-}"
if [[ -z "${raw_value}" ]]; then
printf '%s' "${fallback_value}"
return
fi
case "${raw_value}" in
-|none|null|disabled)
printf ''
;;
*)
printf '%s' "${raw_value}"
;;
esac
}
if [[ -n "${REPORT_DIR_INPUT}" && "${REPORT_DIR_INPUT}" != http://* && "${REPORT_DIR_INPUT}" != https://* ]]; then
REPORT_DIR="${REPORT_DIR_INPUT}"
MAINLAND_BASE_URL="${MAINLAND_BASE_URL_INPUT:-${MAINLAND_BASE_URL_DEFAULT}}"
OVERSEAS_BASE_URL="$(normalize_optional_base_url "${OVERSEAS_BASE_URL_INPUT:-}" "${OVERSEAS_BASE_URL_DEFAULT}")"
else
REPORT_DIR="${REPORT_ROOT_DEFAULT}/go-live-bundle-$(timestamp_now)"
MAINLAND_BASE_URL="${REPORT_DIR_INPUT:-${MAINLAND_BASE_URL_DEFAULT}}"
OVERSEAS_BASE_URL="$(normalize_optional_base_url "${MAINLAND_BASE_URL_INPUT:-}" "${OVERSEAS_BASE_URL_DEFAULT}")"
fi
mkdir -p "${REPORT_DIR}"
CURL_CONNECT_TIMEOUT="${OPS_CENTER_CURL_CONNECT_TIMEOUT:-3}"
CURL_MAX_TIME="${OPS_CENTER_CURL_MAX_TIME:-15}"
STEP_COMMAND_TIMEOUT_SECONDS="${OPS_CENTER_REPORT_COMMAND_TIMEOUT_SECONDS:-45}"
DOCTOR_DECISION_TIMEOUT_SECONDS="${OPS_CENTER_REPORT_DOCTOR_DECISION_TIMEOUT_SECONDS:-120}"
DOCTOR_EXPORT_TIMEOUT_SECONDS="${OPS_CENTER_REPORT_DOCTOR_EXPORT_TIMEOUT_SECONDS:-180}"
DOCTOR_INNER_REPORT_TIMEOUT_SECONDS="${OPS_CENTER_DOCTOR_INNER_REPORT_TIMEOUT_SECONDS:-60}"
ENV_AUDIT_TXT_PATH="${REPORT_DIR}/00_env_audit.txt"
SUMMARY_TXT_PATH="${REPORT_DIR}/01_go_live_check_summary.txt"
GO_LIVE_JSON_PATH="${REPORT_DIR}/02_go_live_summary.json"
STACK_JSON_PATH="${REPORT_DIR}/03_stack_diagnosis.json"
DRIVER_FEED_JSON_PATH="${REPORT_DIR}/04_driver_feed.json"
CODEX_BRIEF_JSON_PATH="${REPORT_DIR}/05_codex_brief.json"
RELEASE_LAUNCHPAD_JSON_PATH="${REPORT_DIR}/06_release_launchpad.json"
DOCTOR_DECISION_JSON_PATH="${REPORT_DIR}/07_doctor_decision.json"
DOCTOR_EXPORT_STDOUT_PATH="${REPORT_DIR}/08_doctor_export_stdout.txt"
MANIFEST_PATH="${REPORT_DIR}/manifest.json"
declare -a STEP_RESULTS=()
pretty_print_json_file() {
local source_path="${1:-}"
python3 - "${source_path}" <<'PY'
import json
import pathlib
import sys
path = pathlib.Path(sys.argv[1])
raw_text = path.read_text(encoding="utf-8") if path.exists() else ""
try:
payload = json.loads(raw_text)
except Exception:
print(raw_text)
else:
print(json.dumps(payload, ensure_ascii=False, indent=2))
PY
}
record_step_result() {
local key="${1:-}"
local path="${2:-}"
local ok="${3:-false}"
local http_status="${4:-}"
local exit_code="${5:-0}"
STEP_RESULTS+=("${key}|${path}|${ok}|${http_status}|${exit_code}")
}
fetch_json_to_file() {
local key="${1:-}"
local url="${2:-}"
local output_path="${3:-}"
local tmp_path
tmp_path="$(mktemp)"
local raw_text http_status
raw_text="$(curl -sS \
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
--max-time "${CURL_MAX_TIME}" \
"${url}" \
-w $'\nHTTP_STATUS=%{http_code}' || true)"
http_status="${raw_text##*$'\n'HTTP_STATUS=}"
printf '%s' "${raw_text%$'\n'HTTP_STATUS=*}" > "${tmp_path}"
pretty_print_json_file "${tmp_path}" > "${output_path}"
rm -f "${tmp_path}"
if [[ "${http_status}" == 2* ]]; then
record_step_result "${key}" "${output_path}" "true" "${http_status}" "0"
return 0
fi
record_step_result "${key}" "${output_path}" "false" "${http_status}" "1"
return 1
}
capture_command_to_file() {
local key="${1:-}"
local output_path="${2:-}"
shift 2 || true
capture_command_to_file_with_timeout "${STEP_COMMAND_TIMEOUT_SECONDS}" "${key}" "${output_path}" "$@"
}
capture_command_to_file_with_timeout() {
local timeout_seconds="${1:-45}"
local key="${2:-}"
local output_path="${3:-}"
shift 3 || true
local -a wrapped_command=("$@")
local had_errexit="false"
case "$-" in
*e*) had_errexit="true" ;;
esac
if command -v timeout >/dev/null 2>&1; then
wrapped_command=(timeout "${timeout_seconds}" "$@")
fi
set +e
"${wrapped_command[@]}" > "${output_path}" 2>&1
local exit_code=$?
if [[ "${had_errexit}" == "true" ]]; then
set -e
else
set +e
fi
if [[ "${exit_code}" -eq 0 ]]; then
record_step_result "${key}" "${output_path}" "true" "" "${exit_code}"
return 0
fi
record_step_result "${key}" "${output_path}" "false" "" "${exit_code}"
return "${exit_code}"
}
materialize_go_live_summary_fallback() {
local summary_path="${1:-}"
local output_path="${2:-}"
python3 - "${summary_path}" "${output_path}" <<'PY'
import json
import pathlib
import sys
summary_path = pathlib.Path(sys.argv[1])
output_path = pathlib.Path(sys.argv[2])
raw_text = summary_path.read_text(encoding="utf-8") if summary_path.exists() else ""
start = raw_text.find("{")
end = raw_text.rfind("}")
if start < 0 or end < start:
sys.exit(1)
json_text = raw_text[start:end + 1]
payload = json.loads(json_text)
if not isinstance(payload, dict):
sys.exit(1)
output_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
PY
}
override_step_result() {
local key="${1:-}"
local path="${2:-}"
local ok="${3:-false}"
local http_status="${4:-}"
local exit_code="${5:-0}"
local -a updated_results=()
local raw_item existing_key
for raw_item in "${STEP_RESULTS[@]}"; do
existing_key="${raw_item%%|*}"
if [[ "${existing_key}" == "${key}" ]]; then
continue
fi
updated_results+=("${raw_item}")
done
STEP_RESULTS=("${updated_results[@]}")
record_step_result "${key}" "${path}" "${ok}" "${http_status}" "${exit_code}"
}
set +e
capture_command_to_file \
"env_audit" \
"${ENV_AUDIT_TXT_PATH}" \
bash "${SCRIPT_DIR}/drive_ops_center.sh" env-audit "${MAINLAND_BASE_URL}"
capture_command_to_file \
"go_live_check_summary" \
"${SUMMARY_TXT_PATH}" \
bash "${SCRIPT_DIR}/drive_ops_center.sh" go-live-check "${MAINLAND_BASE_URL}" "${OVERSEAS_BASE_URL}" summary
fetch_json_to_file \
"go_live_summary" \
"${MAINLAND_BASE_URL}/api/v1/ops/go-live-summary?base_url=${MAINLAND_BASE_URL}" \
"${GO_LIVE_JSON_PATH}"
if [[ ! -s "${GO_LIVE_JSON_PATH}" || "$(sed -n '1p' "${GO_LIVE_JSON_PATH}" 2>/dev/null)" != "{"* ]]; then
if materialize_go_live_summary_fallback "${SUMMARY_TXT_PATH}" "${GO_LIVE_JSON_PATH}" 2>/dev/null; then
override_step_result "go_live_summary" "${GO_LIVE_JSON_PATH}" "true" "fallback" "0"
fi
fi
if [[ "$(python3 - "${GO_LIVE_JSON_PATH}" <<'PY'
import json
import pathlib
import sys
path = pathlib.Path(sys.argv[1])
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except Exception:
print("invalid")
raise SystemExit(0)
if isinstance(payload, dict) and payload.get("detail") == "Not Found":
print("not_found")
else:
print("ok")
PY
)" == "not_found" ]]; then
if materialize_go_live_summary_fallback "${SUMMARY_TXT_PATH}" "${GO_LIVE_JSON_PATH}" 2>/dev/null; then
override_step_result "go_live_summary" "${GO_LIVE_JSON_PATH}" "true" "fallback" "0"
fi
fi
fetch_json_to_file \
"stack_diagnosis" \
"${MAINLAND_BASE_URL}/api/v1/ops/stack-diagnosis?base_url=${MAINLAND_BASE_URL}" \
"${STACK_JSON_PATH}"
fetch_json_to_file \
"driver_feed" \
"${MAINLAND_BASE_URL}/api/v1/ops/driver-feed" \
"${DRIVER_FEED_JSON_PATH}"
fetch_json_to_file \
"codex_brief" \
"${MAINLAND_BASE_URL}/api/v1/ops/codex-brief" \
"${CODEX_BRIEF_JSON_PATH}"
fetch_json_to_file \
"release_launchpad" \
"${MAINLAND_BASE_URL}/api/v1/ops/releases/launchpad" \
"${RELEASE_LAUNCHPAD_JSON_PATH}"
capture_command_to_file_with_timeout \
"${DOCTOR_EXPORT_TIMEOUT_SECONDS}" \
"doctor_export" \
"${DOCTOR_EXPORT_STDOUT_PATH}" \
env "OPS_CENTER_DOCTOR_REPORT_TIMEOUT_SECONDS=${DOCTOR_INNER_REPORT_TIMEOUT_SECONDS}" \
bash "${SCRIPT_DIR}/drive_ops_center.sh" doctor-export "${REPORT_DIR}/doctor-export" "${MAINLAND_BASE_URL}" "${OVERSEAS_BASE_URL}"
capture_command_to_file_with_timeout \
"${DOCTOR_DECISION_TIMEOUT_SECONDS}" \
"doctor_decision" \
"${DOCTOR_DECISION_JSON_PATH}" \
bash "${SCRIPT_DIR}/drive_ops_center.sh" doctor-decision "${REPORT_DIR}/doctor-export"
set -e
python3 - <<'PY' \
"${MANIFEST_PATH}" \
"${REPORT_DIR}" \
"${MAINLAND_BASE_URL}" \
"${OVERSEAS_BASE_URL}" \
"${ENV_AUDIT_TXT_PATH}" \
"${SUMMARY_TXT_PATH}" \
"${GO_LIVE_JSON_PATH}" \
"${STACK_JSON_PATH}" \
"${DRIVER_FEED_JSON_PATH}" \
"${CODEX_BRIEF_JSON_PATH}" \
"${RELEASE_LAUNCHPAD_JSON_PATH}" \
"${DOCTOR_DECISION_JSON_PATH}" \
"${DOCTOR_EXPORT_STDOUT_PATH}" \
"${STEP_RESULTS[@]}"
import json
import pathlib
import sys
manifest_path = pathlib.Path(sys.argv[1])
report_dir = pathlib.Path(sys.argv[2])
mainland_base_url = sys.argv[3]
overseas_base_url = sys.argv[4]
file_keys = [
"env_audit",
"go_live_check_summary",
"go_live_summary",
"stack_diagnosis",
"driver_feed",
"codex_brief",
"release_launchpad",
"doctor_decision",
"doctor_export",
]
file_paths = {
"env_audit": pathlib.Path(sys.argv[5]),
"go_live_check_summary": pathlib.Path(sys.argv[6]),
"go_live_summary": pathlib.Path(sys.argv[7]),
"stack_diagnosis": pathlib.Path(sys.argv[8]),
"driver_feed": pathlib.Path(sys.argv[9]),
"codex_brief": pathlib.Path(sys.argv[10]),
"release_launchpad": pathlib.Path(sys.argv[11]),
"doctor_decision": pathlib.Path(sys.argv[12]),
"doctor_export": pathlib.Path(sys.argv[13]),
}
step_results = {}
for raw_item in sys.argv[14:]:
parts = raw_item.split("|", 4)
if len(parts) != 5:
continue
key, path_value, ok_value, http_status, exit_code = parts
step_results[key] = {
"path": path_value,
"ok": ok_value == "true",
"http_status": http_status,
"exit_code": int(exit_code or 0),
}
def load_json_file(path: pathlib.Path) -> dict:
if not path.is_file():
return {}
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except Exception:
return {}
return payload if isinstance(payload, dict) else {}
def load_env_audit_summary(path: pathlib.Path) -> dict:
if not path.is_file():
return {}
raw_text = path.read_text(encoding="utf-8", errors="replace")
marker = "[8/8] condensed env audit summary"
candidate_text = raw_text.split(marker, 1)[1].strip() if marker in raw_text else raw_text.strip()
if not candidate_text:
return {}
try:
payload = json.loads(candidate_text)
except Exception:
return {}
return payload if isinstance(payload, dict) else {}
failures = []
artifacts = []
for key in file_keys:
artifact_path = file_paths[key]
result = dict(step_results.get(key) or {})
if not result:
result = {
"path": str(artifact_path),
"ok": artifact_path.exists(),
"http_status": "",
"exit_code": 0 if artifact_path.exists() else 1,
}
entry = {
"key": key,
"path": str(artifact_path),
"exists": artifact_path.exists(),
"ok": bool(result.get("ok", False)),
"http_status": str(result.get("http_status") or ""),
"exit_code": int(result.get("exit_code", 0) or 0),
}
artifacts.append(entry)
if not entry["ok"]:
failures.append(key)
go_live_summary_payload = load_json_file(file_paths["go_live_summary"])
stack_diagnosis_payload = load_json_file(file_paths["stack_diagnosis"])
driver_feed_payload = load_json_file(file_paths["driver_feed"])
codex_brief_payload = load_json_file(file_paths["codex_brief"])
env_audit_summary = load_env_audit_summary(file_paths["env_audit"])
stack_diagnosis = dict(stack_diagnosis_payload.get("diagnosis") or {})
driver_feed_summary = dict(driver_feed_payload.get("summary") or {})
codex_brief_summary = dict(codex_brief_payload.get("summary") or {})
def first_nonempty(*values: object) -> str:
for value in values:
normalized = str(value or "").strip()
if normalized:
return normalized
return ""
def nonempty_distinct(values: dict[str, str]) -> list[str]:
return sorted({str(value or "").strip() for value in values.values() if str(value or "").strip()})
def distinct_ints(values: dict[str, int], available_sources: set[str]) -> list[int]:
return sorted({int(values[source]) for source in values if source in available_sources})
launchpad_target_node_codes = {
"go_live_summary": str(go_live_summary_payload.get("launchpad_recommended_target_node_code") or "").strip(),
"stack_diagnosis": str(stack_diagnosis.get("launchpad_recommended_target_node_code") or "").strip(),
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_target_node_code") or "").strip(),
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_target_node_code") or "").strip(),
}
launchpad_recovery_labels = {
"go_live_summary": str(go_live_summary_payload.get("launchpad_recommended_recovery_label") or "").strip(),
"stack_diagnosis": str(stack_diagnosis.get("launchpad_recommended_recovery_label") or "").strip(),
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_recovery_label") or "").strip(),
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_recovery_label") or "").strip(),
}
launchpad_recovery_summaries = {
"go_live_summary": str(go_live_summary_payload.get("launchpad_recommended_recovery_summary") or "").strip(),
"stack_diagnosis": str(stack_diagnosis.get("launchpad_recommended_recovery_summary") or "").strip(),
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
}
launchpad_bootstrap_pending_nodes = {
"go_live_summary": int(go_live_summary_payload.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
"stack_diagnosis": int(stack_diagnosis.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
"driver_feed": int(driver_feed_summary.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
"codex_brief": int(codex_brief_summary.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
}
launchpad_acceptance_ready_nodes = {
"go_live_summary": int(go_live_summary_payload.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
"stack_diagnosis": int(stack_diagnosis.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
"driver_feed": int(driver_feed_summary.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
"codex_brief": int(codex_brief_summary.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
}
available_alignment_sources = {
source
for source, payload_value in {
"go_live_summary": go_live_summary_payload,
"stack_diagnosis": stack_diagnosis_payload,
"driver_feed": driver_feed_payload,
"codex_brief": codex_brief_payload,
}.items()
if isinstance(payload_value, dict) and payload_value
}
launchpad_alignment = {
"consistent": True,
"target_node_code_consistent": True,
"recovery_label_consistent": True,
"recovery_summary_consistent": True,
"bootstrap_pending_consistent": True,
"acceptance_ready_consistent": True,
"available_sources": sorted(available_alignment_sources),
"target_node_codes": launchpad_target_node_codes,
"recovery_labels": launchpad_recovery_labels,
"recovery_summaries": launchpad_recovery_summaries,
"bootstrap_pending_nodes": launchpad_bootstrap_pending_nodes,
"acceptance_ready_nodes": launchpad_acceptance_ready_nodes,
}
launchpad_alignment["target_node_code_consistent"] = len(nonempty_distinct({
source: value for source, value in launchpad_target_node_codes.items() if source in available_alignment_sources
})) <= 1
launchpad_alignment["recovery_label_consistent"] = len(nonempty_distinct({
source: value for source, value in launchpad_recovery_labels.items() if source in available_alignment_sources
})) <= 1
launchpad_alignment["recovery_summary_consistent"] = len(nonempty_distinct({
source: value for source, value in launchpad_recovery_summaries.items() if source in available_alignment_sources
})) <= 1
launchpad_alignment["bootstrap_pending_consistent"] = len(
distinct_ints(launchpad_bootstrap_pending_nodes, available_alignment_sources)
) <= 1
launchpad_alignment["acceptance_ready_consistent"] = len(
distinct_ints(launchpad_acceptance_ready_nodes, available_alignment_sources)
) <= 1
launchpad_alignment["consistent"] = all([
bool(launchpad_alignment["target_node_code_consistent"]),
bool(launchpad_alignment["recovery_label_consistent"]),
bool(launchpad_alignment["recovery_summary_consistent"]),
bool(launchpad_alignment["bootstrap_pending_consistent"]),
bool(launchpad_alignment["acceptance_ready_consistent"]),
])
env_audit_status = str(env_audit_summary.get("status") or "").strip()
env_audit_runtime_may_need_restart = bool(((env_audit_summary.get("runtime") or {}).get("runtime_may_need_restart", False)))
review_status_hint = "ready"
review_headline = "核心上线证据已经齐备,可进入最终人工复核或正式发布门禁。"
if failures or env_audit_status == "blocked":
review_status_hint = "blocked"
review_headline = "当前导出包仍有关键阻断项,不能直接作为正式上线终稿。"
elif env_audit_status == "attention" or not launchpad_alignment["consistent"]:
review_status_hint = "attention"
if not launchpad_alignment["consistent"]:
review_headline = "核心报告虽已导出,但 launchpad 摘要在不同 surface 之间存在漂移,建议先重新复核。"
else:
review_headline = "核心报告已齐,但环境审计仍提示存在待收口事项。"
payload = {
"report_dir": str(report_dir),
"mainland_base_url": mainland_base_url,
"overseas_base_url": overseas_base_url,
"artifacts": artifacts,
"failures": failures,
"summary": {
"ok": len(failures) == 0,
"artifact_total": len(artifacts),
"failure_total": len(failures),
"review_status_hint": review_status_hint,
"review_headline": review_headline,
"env_audit_status": env_audit_status,
"env_audit_runtime_may_need_restart": env_audit_runtime_may_need_restart,
"launchpad_recommended_target_node_code": first_nonempty(
go_live_summary_payload.get("launchpad_recommended_target_node_code"),
stack_diagnosis.get("launchpad_recommended_target_node_code"),
driver_feed_summary.get("launchpad_recommended_target_node_code"),
codex_brief_summary.get("launchpad_recommended_target_node_code"),
),
"launchpad_recommended_recovery_label": first_nonempty(
go_live_summary_payload.get("launchpad_recommended_recovery_label"),
stack_diagnosis.get("launchpad_recommended_recovery_label"),
driver_feed_summary.get("launchpad_recommended_recovery_label"),
codex_brief_summary.get("launchpad_recommended_recovery_label"),
),
"launchpad_recommended_recovery_summary": first_nonempty(
go_live_summary_payload.get("launchpad_recommended_recovery_summary"),
stack_diagnosis.get("launchpad_recommended_recovery_summary"),
driver_feed_summary.get("launchpad_recommended_recovery_summary"),
codex_brief_summary.get("launchpad_recommended_recovery_summary"),
),
"launchpad_onboarding_bootstrap_pending_nodes": max(launchpad_bootstrap_pending_nodes.values() or [0]),
"launchpad_onboarding_acceptance_ready_nodes": max(launchpad_acceptance_ready_nodes.values() or [0]),
"launchpad_alignment": launchpad_alignment,
"recommended_reading_order": [
"00_env_audit.txt",
"01_go_live_check_summary.txt",
"02_go_live_summary.json",
"03_stack_diagnosis.json",
"04_driver_feed.json",
"05_codex_brief.json",
"06_release_launchpad.json",
"07_doctor_decision.json",
"08_doctor_export_stdout.txt",
],
},
}
manifest_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
print(json.dumps(payload, ensure_ascii=False, indent=2))
PY

View File

@@ -0,0 +1,135 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
TEMPLATE_PATH="${SCRIPT_DIR}/templates/domaincheck-ops-center.env.example"
usage() {
cat <<'EOF'
usage:
bash domain-api/deploy/multi-region/init_ops_center_config.sh \
[config_path] [mainland_api_base_url] [overseas_api_base_url] [control_plane_base_url]
examples:
bash domain-api/deploy/multi-region/init_ops_center_config.sh
bash domain-api/deploy/multi-region/init_ops_center_config.sh /etc/default/domaincheck-ops-center \
http://121.204.244.188:8100 \
http://152.53.37.118:8100 \
https://api.example.com
notes:
- existing config files are preserved unless FORCE=true
- advanced fields can be overridden with env vars:
OPS_ACTIVE_PROFILE
OPS_PROFILE_NAMES
OPS_DEFAULT_REQUESTED_BY
OPS_DEFAULT_ISSUED_BY
OPS_DEFAULT_EXPIRES_IN_HOURS
OPS_DEFAULT_ROOT_DIR
OPS_REPORT_ROOT
EOF
}
if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
usage
exit 0
fi
TARGET_PATH="${1:-/etc/default/domaincheck-ops-center}"
MAINLAND_API_BASE_URL="${2:-${OPS_MAINLAND_API_BASE_URL:-http://127.0.0.1:8100}}"
OVERSEAS_API_BASE_URL="${3:-${OPS_OVERSEAS_API_BASE_URL:-http://127.0.0.1:8100}}"
CONTROL_PLANE_BASE_URL="${4:-${OPS_DEFAULT_CONTROL_PLANE_BASE_URL:-}}"
ACTIVE_PROFILE="${OPS_ACTIVE_PROFILE:-default}"
PROFILE_NAMES="${OPS_PROFILE_NAMES:-${ACTIVE_PROFILE}}"
DEFAULT_REQUESTED_BY="${OPS_DEFAULT_REQUESTED_BY:-cli/ops-center}"
DEFAULT_ISSUED_BY="${OPS_DEFAULT_ISSUED_BY:-ops-center}"
DEFAULT_EXPIRES_IN_HOURS="${OPS_DEFAULT_EXPIRES_IN_HOURS:-72}"
DEFAULT_ROOT_DIR="${OPS_DEFAULT_ROOT_DIR:-/opt/domaincheck}"
REPORT_ROOT="${OPS_REPORT_ROOT:-}"
NODE_AGENT_SERVICE_NAME="${OPS_NODE_AGENT_SERVICE_NAME:-domaincheck-node-agent}"
NODE_AGENT_ENV_FILE="${OPS_NODE_AGENT_ENV_FILE:-/etc/default/domaincheck-node-agent}"
FORCE_WRITE="${FORCE:-false}"
if [[ ! -f "${TEMPLATE_PATH}" ]]; then
echo "template not found: ${TEMPLATE_PATH}" >&2
exit 1
fi
mkdir -p "$(dirname "${TARGET_PATH}")"
if [[ ! -f "${TARGET_PATH}" || "${FORCE_WRITE}" == "true" ]]; then
cp "${TEMPLATE_PATH}" "${TARGET_PATH}"
fi
upsert_key() {
local key="$1"
local value="$2"
if grep -q "^${key}=" "${TARGET_PATH}"; then
sed -i "s#^${key}=.*#${key}=${value}#" "${TARGET_PATH}"
else
printf '%s=%s\n' "${key}" "${value}" >>"${TARGET_PATH}"
fi
}
upsert_commented_key() {
local key="$1"
local value="$2"
if [[ -z "${value}" ]]; then
return 0
fi
upsert_key "${key}" "${value}"
}
upsert_key "OPS_ACTIVE_PROFILE" "${ACTIVE_PROFILE}"
upsert_key "OPS_PROFILE_NAMES" "${PROFILE_NAMES}"
upsert_key "OPS_MAINLAND_API_BASE_URL" "${MAINLAND_API_BASE_URL}"
upsert_key "OPS_OVERSEAS_API_BASE_URL" "${OVERSEAS_API_BASE_URL}"
upsert_key "OPS_DEFAULT_CONTROL_PLANE_BASE_URL" "${CONTROL_PLANE_BASE_URL}"
upsert_key "OPS_DEFAULT_REQUESTED_BY" "${DEFAULT_REQUESTED_BY}"
upsert_key "OPS_DEFAULT_ISSUED_BY" "${DEFAULT_ISSUED_BY}"
upsert_key "OPS_DEFAULT_EXPIRES_IN_HOURS" "${DEFAULT_EXPIRES_IN_HOURS}"
upsert_key "OPS_DEFAULT_ROOT_DIR" "${DEFAULT_ROOT_DIR}"
upsert_key "OPS_NODE_AGENT_SERVICE_NAME" "${NODE_AGENT_SERVICE_NAME}"
upsert_key "OPS_NODE_AGENT_ENV_FILE" "${NODE_AGENT_ENV_FILE}"
upsert_commented_key "OPS_REPORT_ROOT" "${REPORT_ROOT}"
python3 - <<'PY' "${TARGET_PATH}"
import json
import sys
from pathlib import Path
config_path = Path(sys.argv[1])
keys = [
"OPS_ACTIVE_PROFILE",
"OPS_PROFILE_NAMES",
"OPS_MAINLAND_API_BASE_URL",
"OPS_OVERSEAS_API_BASE_URL",
"OPS_DEFAULT_CONTROL_PLANE_BASE_URL",
"OPS_DEFAULT_REQUESTED_BY",
"OPS_DEFAULT_ISSUED_BY",
"OPS_DEFAULT_EXPIRES_IN_HOURS",
"OPS_DEFAULT_ROOT_DIR",
"OPS_NODE_AGENT_SERVICE_NAME",
"OPS_NODE_AGENT_ENV_FILE",
"OPS_REPORT_ROOT",
]
values = {}
for line in config_path.read_text(encoding="utf-8").splitlines():
if not line or line.startswith("#") or "=" not in line:
continue
key, value = line.split("=", 1)
if key in keys:
values[key] = value
print(json.dumps({
"ok": True,
"config_path": str(config_path),
"summary": values,
"next_commands": [
"bash domain-api/deploy/multi-region/drive_ops_center.sh config",
"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export",
"bash domain-api/deploy/multi-region/drive_ops_center.sh agent-check",
],
}, ensure_ascii=False, indent=2))
PY

View File

@@ -100,7 +100,7 @@ REDIS_DB=0
EOF
cd "${DOMAIN_API_DIR}"
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
bash domain-api/deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
cat >/etc/default/domaincheck-worker <<EOF
WORKER_MODE=linux-systemd
@@ -138,4 +138,4 @@ chown -R www:www "${REPO_DIR}"
echo
echo "国内 controller 部署完成,建议继续执行:"
echo "cd ${DOMAIN_API_DIR} && bash deploy/multi-region/check_mainland_controller.sh"
echo "cd ${DOMAIN_API_DIR} && bash domain-api/deploy/multi-region/check_mainland_controller.sh"

View File

@@ -101,7 +101,7 @@ REDIS_DB=0
EOF
cd "${DOMAIN_API_DIR}"
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
bash domain-api/deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
cat >/etc/default/domaincheck-worker <<EOF
WORKER_MODE=linux-systemd

View File

@@ -185,6 +185,15 @@ sed -i 's#^NODE_ROLE=.*#NODE_ROLE=control#' /etc/default/domaincheck-api
sed -i 's#^SYNC_PUSH_ENABLED=.*#SYNC_PUSH_ENABLED=false#' /etc/default/domaincheck-api
sed -i "s#^CORS_ORIGINS=.*#CORS_ORIGINS='[\\\"https://${ADMIN_DOMAIN}\\\",\\\"http://127.0.0.1:3201\\\",\\\"http://localhost:3201\\\"]'#" /etc/default/domaincheck-api
OPS_DEFAULT_REQUESTED_BY="cli/${OVERSEAS_NODE_CODE}" \
OPS_DEFAULT_ISSUED_BY="${OVERSEAS_NODE_CODE}" \
OPS_DEFAULT_ROOT_DIR="${BASE_DIR}" \
bash "${DOMAIN_API_DIR}/deploy/multi-region/init_ops_center_config.sh" \
/etc/default/domaincheck-ops-center \
http://127.0.0.1:8100 \
http://127.0.0.1:8100 \
"https://${API_DOMAIN}" >/dev/null
cp "${DOMAIN_API_DIR}/deploy/systemd/domain-api.service" /etc/systemd/system/domaincheck-api.service
cp "${DOMAIN_API_DIR}/deploy/systemd/domain-worker.service" /etc/systemd/system/domaincheck-worker.service
@@ -206,3 +215,5 @@ echo "国外主控机部署完成,建议继续执行:"
echo "curl http://127.0.0.1:8100/health"
echo "curl http://127.0.0.1:8100/api/v1/runtime/preflight"
echo "curl http://127.0.0.1:8100/api/v1/runtime/readiness"
echo "bash ${DOMAIN_API_DIR}/deploy/multi-region/drive_ops_center.sh config"
echo "如后续接入大陆 controller请记得修改 /etc/default/domaincheck-ops-center 中的 OPS_MAINLAND_API_BASE_URL"

View File

@@ -0,0 +1,106 @@
#!/usr/bin/env bash
ops_center_normalize_profile_key() {
local raw_value="${1:-default}"
printf '%s' "${raw_value}" | tr '[:lower:]-./' '[:upper:]____' | tr -cd 'A-Z0-9_'
}
ops_center_load_config() {
local caller_dir="${1:-}"
if [[ -z "${caller_dir}" ]]; then
caller_dir="$(pwd)"
fi
local env_active_profile="${OPS_ACTIVE_PROFILE:-}"
local env_profile_names="${OPS_PROFILE_NAMES:-}"
local env_report_root="${OPS_REPORT_ROOT:-}"
local env_mainland_api_base_url="${OPS_MAINLAND_API_BASE_URL:-}"
local env_overseas_api_base_url="${OPS_OVERSEAS_API_BASE_URL:-}"
local env_control_plane_base_url="${OPS_DEFAULT_CONTROL_PLANE_BASE_URL:-}"
local env_requested_by="${OPS_DEFAULT_REQUESTED_BY:-}"
local env_issued_by="${OPS_DEFAULT_ISSUED_BY:-}"
local env_expires_in_hours="${OPS_DEFAULT_EXPIRES_IN_HOURS:-}"
local env_root_dir="${OPS_DEFAULT_ROOT_DIR:-}"
local env_node_agent_service_name="${OPS_NODE_AGENT_SERVICE_NAME:-}"
local env_node_agent_env_file="${OPS_NODE_AGENT_ENV_FILE:-}"
OPS_CENTER_SCRIPT_DIR="${caller_dir}"
OPS_CENTER_API_DIR="$(cd "${OPS_CENTER_SCRIPT_DIR}/../.." && pwd)"
OPS_CENTER_PROJECT_ROOT="$(cd "${OPS_CENTER_API_DIR}/.." && pwd)"
OPS_CENTER_RUNTIME_DIR="${OPS_CENTER_API_DIR}/runtime"
OPS_CENTER_REPORT_ROOT_DEFAULT="${OPS_CENTER_RUNTIME_DIR}/ops-center-reports"
OPS_CENTER_CONFIG_FILE="${DOMAINCHECK_OPS_CENTER_ENV:-${OPS_CENTER_ENV_FILE:-/etc/default/domaincheck-ops-center}}"
if [[ -f "${OPS_CENTER_CONFIG_FILE}" ]]; then
# shellcheck disable=SC1090
source "${OPS_CENTER_CONFIG_FILE}"
fi
OPS_CENTER_ACTIVE_PROFILE="${OPS_CENTER_PROFILE:-${DOMAINCHECK_OPS_CENTER_PROFILE:-${env_active_profile:-${OPS_ACTIVE_PROFILE:-default}}}}"
OPS_CENTER_ACTIVE_PROFILE_KEY="$(ops_center_normalize_profile_key "${OPS_CENTER_ACTIVE_PROFILE}")"
OPS_CENTER_PROFILE_NAMES="${env_profile_names:-${OPS_PROFILE_NAMES:-${OPS_CENTER_ACTIVE_PROFILE}}}"
OPS_CENTER_REPORT_ROOT="${env_report_root:-${OPS_REPORT_ROOT:-${OPS_CENTER_REPORT_ROOT_DEFAULT}}}"
OPS_CENTER_ENV_MAINLAND_API_BASE_URL="${env_mainland_api_base_url}"
OPS_CENTER_ENV_OVERSEAS_API_BASE_URL="${env_overseas_api_base_url}"
OPS_CENTER_ENV_CONTROL_PLANE_BASE_URL="${env_control_plane_base_url}"
OPS_CENTER_ENV_REQUESTED_BY="${env_requested_by}"
OPS_CENTER_ENV_ISSUED_BY="${env_issued_by}"
OPS_CENTER_ENV_EXPIRES_IN_HOURS="${env_expires_in_hours}"
OPS_CENTER_ENV_ROOT_DIR="${env_root_dir}"
OPS_CENTER_ENV_NODE_AGENT_SERVICE_NAME="${env_node_agent_service_name}"
OPS_CENTER_ENV_NODE_AGENT_ENV_FILE="${env_node_agent_env_file}"
}
ops_center_resolve_profile_value() {
local suffix="$1"
local fallback_value="${2:-}"
local profile_key="${OPS_CENTER_ACTIVE_PROFILE_KEY:-DEFAULT}"
local variable_name="OPS_PROFILE_${profile_key}_${suffix}"
local profile_value="${!variable_name:-}"
if [[ -n "${profile_value}" ]]; then
printf '%s' "${profile_value}"
else
printf '%s' "${fallback_value}"
fi
}
ops_center_resolve_mainland_api_base_url() {
ops_center_resolve_profile_value "MAINLAND_API_BASE_URL" "${OPS_CENTER_ENV_MAINLAND_API_BASE_URL:-${OPS_MAINLAND_API_BASE_URL:-http://127.0.0.1:8100}}"
}
ops_center_resolve_overseas_api_base_url() {
ops_center_resolve_profile_value "OVERSEAS_API_BASE_URL" "${OPS_CENTER_ENV_OVERSEAS_API_BASE_URL:-${OPS_OVERSEAS_API_BASE_URL:-http://127.0.0.1:8100}}"
}
ops_center_resolve_control_plane_base_url() {
ops_center_resolve_profile_value "DEFAULT_CONTROL_PLANE_BASE_URL" "${OPS_CENTER_ENV_CONTROL_PLANE_BASE_URL:-${OPS_DEFAULT_CONTROL_PLANE_BASE_URL:-}}"
}
ops_center_resolve_requested_by() {
ops_center_resolve_profile_value "DEFAULT_REQUESTED_BY" "${OPS_CENTER_ENV_REQUESTED_BY:-${OPS_DEFAULT_REQUESTED_BY:-cli/ops-center}}"
}
ops_center_resolve_issued_by() {
ops_center_resolve_profile_value "DEFAULT_ISSUED_BY" "${OPS_CENTER_ENV_ISSUED_BY:-${OPS_DEFAULT_ISSUED_BY:-ops-center}}"
}
ops_center_resolve_expires_in_hours() {
ops_center_resolve_profile_value "DEFAULT_EXPIRES_IN_HOURS" "${OPS_CENTER_ENV_EXPIRES_IN_HOURS:-${OPS_DEFAULT_EXPIRES_IN_HOURS:-72}}"
}
ops_center_resolve_root_dir() {
ops_center_resolve_profile_value "DEFAULT_ROOT_DIR" "${OPS_CENTER_ENV_ROOT_DIR:-${OPS_DEFAULT_ROOT_DIR:-/opt/domaincheck}}"
}
ops_center_resolve_node_agent_service_name() {
ops_center_resolve_profile_value "NODE_AGENT_SERVICE_NAME" "${OPS_CENTER_ENV_NODE_AGENT_SERVICE_NAME:-${OPS_NODE_AGENT_SERVICE_NAME:-domaincheck-node-agent}}"
}
ops_center_resolve_node_agent_env_file() {
ops_center_resolve_profile_value "NODE_AGENT_ENV_FILE" "${OPS_CENTER_ENV_NODE_AGENT_ENV_FILE:-${OPS_NODE_AGENT_ENV_FILE:-/etc/default/domaincheck-node-agent}}"
}
ops_center_ensure_report_dir() {
local report_dir="${1:-${OPS_CENTER_REPORT_ROOT_DEFAULT}}"
mkdir -p "${report_dir}"
}

View File

@@ -0,0 +1,317 @@
#!/usr/bin/env bash
set -euo pipefail
TARGET_PATH="${1:-}"
if [[ -z "${TARGET_PATH}" ]]; then
echo "usage: bash domain-api/deploy/multi-region/review_go_live_bundle.sh <manifest.json|report_dir>" >&2
exit 1
fi
if [[ -d "${TARGET_PATH}" ]]; then
MANIFEST_PATH="${TARGET_PATH%/}/manifest.json"
else
MANIFEST_PATH="${TARGET_PATH}"
fi
if [[ ! -f "${MANIFEST_PATH}" ]]; then
echo "manifest not found: ${MANIFEST_PATH}" >&2
exit 1
fi
python3 - "${MANIFEST_PATH}" <<'PY'
import json
import pathlib
import sys
manifest_path = pathlib.Path(sys.argv[1])
payload = json.loads(manifest_path.read_text(encoding="utf-8"))
artifacts = payload.get("artifacts") or []
failures = payload.get("failures") or []
summary = payload.get("summary") or {}
critical_keys = {"go_live_summary", "stack_diagnosis", "driver_feed", "codex_brief", "release_launchpad"}
artifact_by_key = {str(item.get("key") or ""): item for item in artifacts if isinstance(item, dict)}
critical_failures = [key for key in critical_keys if not bool((artifact_by_key.get(key) or {}).get("ok", False))]
noncritical_failures = [key for key in failures if key not in critical_keys]
env_audit_artifact = artifact_by_key.get("env_audit") or {}
env_audit_status = ""
env_audit_headline = ""
env_audit_missing_items = []
env_audit_recommended_actions = []
env_audit_blocking = False
env_audit_attention = False
env_audit_runtime_may_need_restart = False
go_live_summary_payload = {}
stack_diagnosis_payload = {}
driver_feed_payload = {}
codex_brief_payload = {}
def load_json_artifact(key: str) -> dict:
artifact = artifact_by_key.get(key) or {}
artifact_path = pathlib.Path(str(artifact.get("path") or ""))
if not artifact_path.is_file():
return {}
try:
loaded = json.loads(artifact_path.read_text(encoding="utf-8"))
except Exception:
return {}
return loaded if isinstance(loaded, dict) else {}
env_audit_path = pathlib.Path(str(env_audit_artifact.get("path") or ""))
if env_audit_path.is_file():
raw_text = env_audit_path.read_text(encoding="utf-8", errors="replace")
marker = "[8/8] condensed env audit summary"
candidate_text = raw_text.split(marker, 1)[1].strip() if marker in raw_text else raw_text.strip()
if candidate_text:
try:
env_payload = json.loads(candidate_text)
except Exception:
env_payload = {}
env_audit_status = str(env_payload.get("status") or "")
env_audit_headline = str(env_payload.get("headline") or "")
env_audit_missing_items = list(env_payload.get("missing_items") or [])
env_audit_recommended_actions = [str(item).strip() for item in list(env_payload.get("recommended_actions") or []) if str(item).strip()]
env_audit_runtime_may_need_restart = bool(((env_payload.get("runtime") or {}).get("runtime_may_need_restart", False)))
env_audit_blocking = env_audit_status == "blocked"
env_audit_attention = env_audit_status == "attention"
go_live_summary_payload = load_json_artifact("go_live_summary")
stack_diagnosis_payload = load_json_artifact("stack_diagnosis")
driver_feed_payload = load_json_artifact("driver_feed")
codex_brief_payload = load_json_artifact("codex_brief")
go_live_launchpad_target_node_code = str(go_live_summary_payload.get("launchpad_recommended_target_node_code") or "").strip()
go_live_launchpad_recovery_label = str(go_live_summary_payload.get("launchpad_recommended_recovery_label") or "").strip()
go_live_launchpad_recovery_summary = str(go_live_summary_payload.get("launchpad_recommended_recovery_summary") or "").strip()
go_live_launchpad_bootstrap_pending_nodes = int(go_live_summary_payload.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0)
go_live_launchpad_acceptance_ready_nodes = int(go_live_summary_payload.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0)
stack_diagnosis = stack_diagnosis_payload.get("diagnosis") or {}
stack_launchpad_target_node_code = str(stack_diagnosis.get("launchpad_recommended_target_node_code") or "").strip()
stack_launchpad_recovery_label = str(stack_diagnosis.get("launchpad_recommended_recovery_label") or "").strip()
stack_launchpad_recovery_summary = str(stack_diagnosis.get("launchpad_recommended_recovery_summary") or "").strip()
stack_launchpad_bootstrap_pending_nodes = int(stack_diagnosis.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0)
stack_launchpad_acceptance_ready_nodes = int(stack_diagnosis.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0)
driver_feed_summary = driver_feed_payload.get("summary") or {}
codex_brief_summary = codex_brief_payload.get("summary") or {}
def nonempty_distinct(values: dict[str, str]) -> list[str]:
return sorted({str(value or "").strip() for value in values.values() if str(value or "").strip()})
def distinct_ints(values: dict[str, int], available_sources: set[str]) -> list[int]:
return sorted({int(values[source]) for source in values if source in available_sources})
launchpad_target_node_codes = {
"go_live_summary": go_live_launchpad_target_node_code,
"stack_diagnosis": stack_launchpad_target_node_code,
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_target_node_code") or "").strip(),
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_target_node_code") or "").strip(),
}
launchpad_recovery_labels = {
"go_live_summary": go_live_launchpad_recovery_label,
"stack_diagnosis": stack_launchpad_recovery_label,
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_recovery_label") or "").strip(),
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_recovery_label") or "").strip(),
}
launchpad_recovery_summaries = {
"go_live_summary": go_live_launchpad_recovery_summary,
"stack_diagnosis": stack_launchpad_recovery_summary,
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
}
launchpad_bootstrap_pending_nodes = {
"go_live_summary": go_live_launchpad_bootstrap_pending_nodes,
"stack_diagnosis": stack_launchpad_bootstrap_pending_nodes,
"driver_feed": int(driver_feed_summary.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
"codex_brief": int(codex_brief_summary.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
}
launchpad_acceptance_ready_nodes = {
"go_live_summary": go_live_launchpad_acceptance_ready_nodes,
"stack_diagnosis": stack_launchpad_acceptance_ready_nodes,
"driver_feed": int(driver_feed_summary.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
"codex_brief": int(codex_brief_summary.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
}
available_alignment_sources = {
source
for source, payload_value in {
"go_live_summary": go_live_summary_payload,
"stack_diagnosis": stack_diagnosis_payload,
"driver_feed": driver_feed_payload,
"codex_brief": codex_brief_payload,
}.items()
if isinstance(payload_value, dict) and payload_value
}
launchpad_target_consistent = len(nonempty_distinct({
source: value for source, value in launchpad_target_node_codes.items() if source in available_alignment_sources
})) <= 1
launchpad_recovery_label_consistent = len(nonempty_distinct({
source: value for source, value in launchpad_recovery_labels.items() if source in available_alignment_sources
})) <= 1
launchpad_recovery_summary_consistent = len(nonempty_distinct({
source: value for source, value in launchpad_recovery_summaries.items() if source in available_alignment_sources
})) <= 1
launchpad_bootstrap_pending_consistent = len(distinct_ints(launchpad_bootstrap_pending_nodes, available_alignment_sources)) <= 1
launchpad_acceptance_ready_consistent = len(distinct_ints(launchpad_acceptance_ready_nodes, available_alignment_sources)) <= 1
launchpad_alignment_consistent = all([
launchpad_target_consistent,
launchpad_recovery_label_consistent,
launchpad_recovery_summary_consistent,
launchpad_bootstrap_pending_consistent,
launchpad_acceptance_ready_consistent,
])
node_agent_env_missing = any(
str(item).strip() == "missing_env:/etc/default/domaincheck-node-agent"
for item in env_audit_missing_items
)
if critical_failures or env_audit_blocking:
status = "blocked"
if critical_failures and env_audit_blocking:
headline = "关键报告存在缺失,且环境审计已给出阻断结论,当前不适合上线。"
elif critical_failures:
headline = "上线证据包仍缺关键报告,暂不建议直接宣称收口完成。"
else:
headline = "环境审计已给出阻断结论,需先修复现场环境后再进入发布门禁。"
elif noncritical_failures or env_audit_attention or not launchpad_alignment_consistent:
status = "attention"
if not launchpad_alignment_consistent:
headline = "核心报告已齐,但 launchpad 摘要在不同 surface 之间存在漂移,建议先复核再发版。"
elif noncritical_failures and env_audit_attention:
headline = "核心报告已齐,但补充报告和环境审计都提示仍有缺口,建议先复核再发版。"
elif noncritical_failures:
headline = "核心报告已齐,但仍有补充报告超时或失败,建议先复核再发版。"
else:
headline = "核心报告已齐,但环境审计仍提示基础缺口,建议补齐后再发版。"
else:
status = "ready"
headline = "核心上线证据已经齐备,可进入最终人工复核或正式发布门禁。"
result = {
"report_dir": str(payload.get("report_dir") or manifest_path.parent),
"status": status,
"headline": headline,
"artifact_total": int(summary.get("artifact_total", len(artifacts)) or len(artifacts)),
"failure_total": int(summary.get("failure_total", len(failures)) or len(failures)),
"critical_failures": critical_failures,
"noncritical_failures": noncritical_failures,
"env_audit": {
"ok": bool(env_audit_artifact.get("ok", False)),
"status": env_audit_status,
"headline": env_audit_headline,
"missing_items": env_audit_missing_items,
"runtime_may_need_restart": env_audit_runtime_may_need_restart,
"recommended_actions": env_audit_recommended_actions,
},
"go_live_summary": {
"go_live_status": str(go_live_summary_payload.get("go_live_status") or "").strip(),
"publish_status": str(go_live_summary_payload.get("publish_status") or "").strip(),
"operator_title": str(go_live_summary_payload.get("operator_title") or "").strip(),
"next_step_action_code": str(go_live_summary_payload.get("next_step_action_code") or "").strip(),
"launchpad_recommended_target_node_code": go_live_launchpad_target_node_code,
"launchpad_recommended_recovery_label": go_live_launchpad_recovery_label,
"launchpad_recommended_recovery_summary": go_live_launchpad_recovery_summary,
"launchpad_onboarding_bootstrap_pending_nodes": go_live_launchpad_bootstrap_pending_nodes,
"launchpad_onboarding_acceptance_ready_nodes": go_live_launchpad_acceptance_ready_nodes,
},
"stack_diagnosis": {
"stack_status": str((stack_diagnosis_payload.get("diagnosis") or {}).get("stack_status") or "").strip(),
"operator_title": str(((stack_diagnosis_payload.get("diagnosis") or {}).get("operator_decision") or {}).get("title") or "").strip(),
"next_step_action_code": str(((stack_diagnosis_payload.get("diagnosis") or {}).get("next_step") or {}).get("action_code") or "").strip(),
"launchpad_recommended_target_node_code": stack_launchpad_target_node_code,
"launchpad_recommended_recovery_label": stack_launchpad_recovery_label,
"launchpad_recommended_recovery_summary": stack_launchpad_recovery_summary,
"launchpad_onboarding_bootstrap_pending_nodes": stack_launchpad_bootstrap_pending_nodes,
"launchpad_onboarding_acceptance_ready_nodes": stack_launchpad_acceptance_ready_nodes,
},
"driver_feed": {
"headline": str(driver_feed_payload.get("headline") or "").strip(),
"launch_status": str(((driver_feed_payload.get("automation_coverage") or {}).get("launch_status") or "")).strip(),
"launch_ready": bool(((driver_feed_payload.get("automation_coverage") or {}).get("launch_ready", False))),
"launchpad_recommended_target_node_code": str(
((driver_feed_payload.get("summary") or {}).get("launchpad_recommended_target_node_code") or "")
).strip(),
"launchpad_recommended_recovery_label": str(
((driver_feed_payload.get("summary") or {}).get("launchpad_recommended_recovery_label") or "")
).strip(),
},
"codex_brief": {
"headline": str(codex_brief_payload.get("headline") or "").strip(),
"focus_entry_key": str(((codex_brief_payload.get("focus") or {}).get("entry_key") or "")).strip(),
"launch_status": str(((codex_brief_payload.get("automation_coverage") or {}).get("launch_status") or "")).strip(),
"launch_ready": bool(((codex_brief_payload.get("automation_coverage") or {}).get("launch_ready", False))),
"launchpad_recommended_target_node_code": str(
((codex_brief_payload.get("summary") or {}).get("launchpad_recommended_target_node_code") or "")
).strip(),
"launchpad_recommended_recovery_label": str(
((codex_brief_payload.get("summary") or {}).get("launchpad_recommended_recovery_label") or "")
).strip(),
},
"launchpad_alignment": {
"consistent": launchpad_alignment_consistent,
"target_node_code_consistent": launchpad_target_consistent,
"recovery_label_consistent": launchpad_recovery_label_consistent,
"recovery_summary_consistent": launchpad_recovery_summary_consistent,
"bootstrap_pending_consistent": launchpad_bootstrap_pending_consistent,
"acceptance_ready_consistent": launchpad_acceptance_ready_consistent,
"available_sources": sorted(available_alignment_sources),
"target_node_codes": launchpad_target_node_codes,
"recovery_labels": launchpad_recovery_labels,
"recovery_summaries": launchpad_recovery_summaries,
"bootstrap_pending_nodes": launchpad_bootstrap_pending_nodes,
"acceptance_ready_nodes": launchpad_acceptance_ready_nodes,
},
"recommended_reading_order": list(summary.get("recommended_reading_order") or []),
"recommended_next_steps": [
"先看 00_env_audit.txt确认当前机器环境是否存在阻断或注意项",
"先看 02_go_live_summary.json 和 03_stack_diagnosis.json",
"再看 04_driver_feed.json 和 05_codex_brief.json",
"若涉及发布门禁,再看 06_release_launchpad.json",
"若 manifest 里仍有失败项,再看对应 artifact 原文件",
]
+ (
[
"当前缺口指向 Node Agent 未接管:先执行 agent-gap-export拿到首个缺口节点的接管包",
"随后执行 node-bootstrap-plan为缺口节点生成可直接落地的接管脚本",
"如果仓库代码已更新但 bootstrap-plan 输出仍偏旧,先重启控制面 API 再重新导出接管计划",
]
if node_agent_env_missing
else []
)
+ (
[
"env-audit 已识别运行中 API 可能仍是旧代码:先执行 drive_ops_center.sh runtime-refresh-recover按固定链路完成 API 重启与复检",
"若仍需人工逐条确认,再执行 recommended_actions 里的 API 重启与复检命令"
]
if env_audit_runtime_may_need_restart
else []
)
+ (
[
f"当前 launchpad 缺口已经收敛到节点 {go_live_launchpad_target_node_code},建议优先执行:{go_live_launchpad_recovery_label or '节点恢复'}",
"先对照 02_go_live_summary.json、04_driver_feed.json、05_codex_brief.json 中的 launchpad 字段,确认三处摘要是否一致",
(go_live_launchpad_recovery_summary or "如果 recovery_summary 仍为空,再回到 06_release_launchpad.json 查看 gap rows 详情"),
]
if go_live_launchpad_target_node_code
else []
)
+ (
[
"当前 bundle 内的 launchpad 摘要在 go_live_summary / stack_diagnosis / driver_feed / codex_brief 之间存在不一致,先不要直接交付。",
"优先复核 02_go_live_summary.json 与 03_stack_diagnosis.json再检查 04_driver_feed.json 与 05_codex_brief.json 是否使用了同一版控制面数据。",
"若只有局部摘要偏旧,先执行 drive_ops_center.sh runtime-refresh-recover并重新导出 go-live bundle。",
]
if not launchpad_alignment_consistent
else []
),
}
print(json.dumps(result, ensure_ascii=False, indent=2))
PY

View File

@@ -0,0 +1,15 @@
OPS_CONTROL_PLANE_BASE_URL=http://127.0.0.1:8100
OPS_AGENT_TOKEN=
OPS_AGENT_POLL_INTERVAL_SECONDS=5
NODE_CODE=mainland-worker-01
NODE_REGION=mainland
NODE_ROLE=worker
WORKER_SERVICE_NAME=domaincheck-worker
API_SERVICE_NAME=domaincheck-api
SYNC_AGENT_SERVICE_NAME=domaincheck-sync-agent
NODE_AGENT_SERVICE_NAME=domaincheck-node-agent
OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","health.snapshot","logs.collect","diagnostics.collect","deploy.release"]
OPS_AGENT_LABELS={}

View File

@@ -0,0 +1,49 @@
# Overseas control-plane unified entrypoint config
#
# Recommended target path:
# /etc/default/domaincheck-ops-center
#
# Usage:
# export DOMAINCHECK_OPS_CENTER_ENV=/etc/default/domaincheck-ops-center
# bash domain-api/deploy/multi-region/drive_ops_center.sh doctor
# Active profile name
OPS_ACTIVE_PROFILE=default
# Optional profile list shown by `drive_ops_center.sh config`
# OPS_PROFILE_NAMES=default,prod,staging
# Mainland controller API base URL
OPS_MAINLAND_API_BASE_URL=http://127.0.0.1:8100
# Overseas control-plane API base URL
OPS_OVERSEAS_API_BASE_URL=http://127.0.0.1:8100
# Public control-plane base URL used in generated node-agent bootstrap plans
OPS_DEFAULT_CONTROL_PLANE_BASE_URL=https://api.example.com
# Default request identity used by CLI wrappers
OPS_DEFAULT_REQUESTED_BY=cli/ops-center
# Default metadata used when generating node-agent bootstrap plans
OPS_DEFAULT_ISSUED_BY=ops-center
OPS_DEFAULT_EXPIRES_IN_HOURS=72
OPS_DEFAULT_ROOT_DIR=/opt/domaincheck
# Optional report export root
# OPS_REPORT_ROOT=/opt/domaincheck/domain-api/runtime/ops-center-reports
# Local node-agent status check defaults
OPS_NODE_AGENT_SERVICE_NAME=domaincheck-node-agent
OPS_NODE_AGENT_ENV_FILE=/etc/default/domaincheck-node-agent
# Optional per-profile overrides:
# OPS_PROFILE_PROD_MAINLAND_API_BASE_URL=http://10.0.0.10:8100
# OPS_PROFILE_PROD_OVERSEAS_API_BASE_URL=https://api.example.com
# OPS_PROFILE_PROD_DEFAULT_CONTROL_PLANE_BASE_URL=https://api.example.com
# OPS_PROFILE_PROD_DEFAULT_REQUESTED_BY=cli/ops-center-prod
# OPS_PROFILE_PROD_DEFAULT_ISSUED_BY=ops-center-prod
# OPS_PROFILE_PROD_DEFAULT_EXPIRES_IN_HOURS=72
# OPS_PROFILE_PROD_DEFAULT_ROOT_DIR=/opt/domaincheck
# OPS_PROFILE_PROD_NODE_AGENT_SERVICE_NAME=domaincheck-node-agent
# OPS_PROFILE_PROD_NODE_AGENT_ENV_FILE=/etc/default/domaincheck-node-agent

View File

@@ -0,0 +1,17 @@
[Unit]
Description=domainCheck Node Agent
After=network.target
[Service]
Type=simple
WorkingDirectory=/opt/domaincheck/domain-api
EnvironmentFile=-/etc/default/domaincheck-node-agent
Environment="PATH=/home/www/.nvm/versions/node/v20.20.2/bin:/home/www/.local/bin:/home/www/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m app.node_agent
Restart=always
RestartSec=5
User=www
Group=www
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,98 @@
import json
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from app.core.config import settings
from app.services.build_info_service import (
_expected_route_paths,
get_runtime_build_info,
remember_registered_route_paths,
)
class BuildInfoServiceTests(unittest.TestCase):
def tearDown(self) -> None:
remember_registered_route_paths([])
def test_runtime_build_info_prefers_configured_manifest_and_reports_registered_surface(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
manifest_path = Path(temp_dir) / "release_manifest.json"
manifest_path.write_text(
json.dumps(
{
"package_name": "domaincheck_release_demo",
"generated_at": "2026-04-18T12:00:00",
"commit_sha": "bd6bcb2",
"commit_ref": "main",
"sha256": "demo-sha256",
},
ensure_ascii=False,
),
encoding="utf-8",
)
route_paths = list(_expected_route_paths().values())
remember_registered_route_paths(route_paths)
with patch.multiple(
settings,
build_manifest_path=str(manifest_path),
build_commit_sha="",
build_commit_ref="",
build_generated_at="",
build_package_name="",
build_checksum="",
build_source_label="",
):
payload = get_runtime_build_info()
self.assertEqual("configured_manifest", payload["source"])
self.assertEqual("domaincheck_release_demo", payload["package_name"])
self.assertEqual("bd6bcb2", payload["commit_sha"])
self.assertEqual("main", payload["commit_ref"])
self.assertEqual("demo-sha256", payload["checksum"])
self.assertEqual(str(manifest_path), payload["manifest_path"])
self.assertTrue(payload["route_surface"]["surface_complete"])
self.assertEqual([], payload["route_surface"]["missing_keys"])
self.assertEqual("registered", payload["route_surface"]["mode"])
self.assertIn("repository_capabilities", payload)
self.assertIn("supports_install_command_block", payload["repository_capabilities"])
self.assertIn("supports_multi_layout_bootstrap", payload["repository_capabilities"])
def test_runtime_build_info_env_override_wins_and_missing_routes_are_visible(self) -> None:
with patch.multiple(
settings,
build_manifest_path="",
build_commit_sha="246838ae4c07",
build_commit_ref="main",
build_generated_at="2026-04-18T12:10:00",
build_package_name="domaincheck_release_env",
build_checksum="env-sha256",
build_source_label="env-override",
):
payload = get_runtime_build_info(route_paths=[_expected_route_paths()["ops_contracts"]])
self.assertEqual("env-override", payload["source"])
self.assertEqual("domaincheck_release_env", payload["package_name"])
self.assertEqual("246838ae4c07", payload["commit_sha"])
self.assertEqual("main", payload["commit_ref"])
self.assertEqual("env-sha256", payload["checksum"])
self.assertFalse(payload["route_surface"]["surface_complete"])
self.assertIn("ops_stack_diagnosis", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_bootstrap_preview", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_bootstrap_execute", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_acceptance_preview", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_acceptance_execute", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_recovery_preview", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_recovery_execute", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_scene_log", payload["route_surface"]["missing_keys"])
self.assertIn("runtime_build_info", payload["route_surface"]["missing_keys"])
self.assertEqual("registered", payload["route_surface"]["mode"])
self.assertIn("repository_capabilities", payload)
self.assertIn("supports_install_command_block", payload["repository_capabilities"])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,111 @@
from __future__ import annotations
import os
import re
import subprocess
import unittest
from pathlib import Path
class DriveOpsCenterDocsTests(unittest.TestCase):
def test_authoritative_docs_only_reference_supported_commands(self) -> None:
repo_root = Path(__file__).resolve().parents[2]
script_path = repo_root / "domain-api" / "deploy" / "multi-region" / "drive_ops_center.sh"
doc_paths = [
repo_root / "domain-api" / "deploy" / "multi-region" / "README.md",
repo_root / "docs" / "25_domainCheck_海外单脑控制面上线收口总表.md",
repo_root / "docs" / "26_domainCheck_发布前运行验证与交付模板.md",
]
help_output = subprocess.run(
["bash", str(script_path), "help"],
cwd=repo_root,
check=True,
capture_output=True,
text=True,
).stdout
supported_commands = self._extract_supported_commands(help_output)
documented_commands = self._extract_documented_commands(doc_paths)
missing_commands = sorted(documented_commands - supported_commands)
self.assertEqual(
[],
missing_commands,
msg=(
"These drive_ops_center.sh commands are referenced by authoritative docs "
"but missing from the CLI help output: "
+ ", ".join(missing_commands)
),
)
self.assertIn("stack-diagnosis", documented_commands)
self.assertIn("stack-diagnosis", supported_commands)
def test_stack_diagnosis_summary_argument_normalization_does_not_treat_summary_as_host(self) -> None:
repo_root = Path(__file__).resolve().parents[2]
script_path = repo_root / "domain-api" / "deploy" / "multi-region" / "drive_ops_center.sh"
env = dict(os.environ)
env["DOMAINCHECK_OPS_CENTER_ENV"] = "/tmp/domaincheck-ops-center-does-not-exist"
env["OPS_MAINLAND_API_BASE_URL"] = "http://127.0.0.1:9"
result = subprocess.run(
["bash", str(script_path), "stack-diagnosis", "summary"],
cwd=repo_root,
check=True,
capture_output=True,
text=True,
env=env,
)
combined_output = f"{result.stdout}\n{result.stderr}"
self.assertNotIn("Could not resolve host: summary", combined_output)
self.assertNotIn("NameError", combined_output)
self.assertIn("[9/9] condensed stack summary", combined_output)
def test_env_drift_recommends_stack_diagnosis_instead_of_stack_check(self) -> None:
repo_root = Path(__file__).resolve().parents[2]
script_path = repo_root / "domain-api" / "deploy" / "multi-region" / "check_env_drift.sh"
script_text = script_path.read_text(encoding="utf-8")
self.assertIn(
'recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary")',
script_text,
)
self.assertNotIn(
'recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh stack-check http://127.0.0.1:8100 summary")',
script_text,
)
@staticmethod
def _extract_supported_commands(help_output: str) -> set[str]:
commands: set[str] = set()
in_commands_section = False
for raw_line in help_output.splitlines():
line = raw_line.rstrip()
if line.strip() == "commands:":
in_commands_section = True
continue
if line.strip() == "examples:":
break
if not in_commands_section:
continue
stripped = line.strip()
if not stripped:
continue
command = stripped.split()[0]
if re.fullmatch(r"[a-z0-9-]+", command):
commands.add(command)
return commands
@staticmethod
def _extract_documented_commands(doc_paths: list[Path]) -> set[str]:
pattern = re.compile(r"drive_ops_center\.sh\s+([a-z0-9-]+)")
commands: set[str] = set()
for doc_path in doc_paths:
text = doc_path.read_text(encoding="utf-8")
commands.update(match.group(1) for match in pattern.finditer(text))
return commands
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,284 @@
import json
import os
import tempfile
import unittest
import urllib.error
from unittest.mock import patch
from app import node_agent
class NodeAgentDeliveryQueueTests(unittest.TestCase):
def test_base_payload_includes_delivery_queue_snapshot(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
node_agent._ensure_queue_dirs()
payload = node_agent._base_payload()
delivery_queue = payload["metadata"]["delivery_queue"]
self.assertEqual("healthy", delivery_queue["state"])
self.assertEqual(0, delivery_queue["pending_count"])
self.assertEqual(0, delivery_queue["dead_letter_count"])
def test_job_event_network_failure_is_queued_for_retry(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir), patch.object(
node_agent,
"_post",
side_effect=urllib.error.URLError("offline"),
):
result = node_agent._job_event(
7,
event_type="executor_received",
message="accepted",
client_event_id="evt-001",
summary_text="已接单",
focus_ref={"kind": "ops_job", "job_id": 7},
)
self.assertEqual("queued", result["state"])
pending_dir = os.path.join(temp_dir, "pending")
pending_files = sorted(os.listdir(pending_dir))
self.assertEqual(1, len(pending_files))
with open(os.path.join(pending_dir, pending_files[0]), "r", encoding="utf-8") as handle:
record = json.load(handle)
self.assertEqual("evt-001", record["payload"]["client_event_id"])
self.assertEqual("已接单", record["payload"]["summary_text"])
self.assertEqual("ops_job", record["payload"]["focus_ref"]["kind"])
self.assertEqual("job_event", record["kind"])
def test_job_complete_transport_failure_preserves_client_request_id(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir), patch.object(
node_agent,
"_post",
side_effect=urllib.error.URLError("offline"),
):
result = node_agent._job_complete(
9,
status="success",
stdout="done",
stderr="",
result={"ok": True},
client_request_id="complete-001",
duration_ms=321,
summary_text="执行成功",
focus_ref={"kind": "ops_job", "job_id": 9},
)
self.assertEqual("queued", result["state"])
pending_dir = os.path.join(temp_dir, "pending")
pending_files = sorted(os.listdir(pending_dir))
self.assertEqual(1, len(pending_files))
with open(os.path.join(pending_dir, pending_files[0]), "r", encoding="utf-8") as handle:
record = json.load(handle)
self.assertEqual("complete-001", record["payload"]["client_request_id"])
self.assertEqual(321, record["payload"]["duration_ms"])
self.assertEqual("执行成功", record["payload"]["summary_text"])
self.assertEqual("ops_job", record["payload"]["focus_ref"]["kind"])
self.assertEqual("job_complete", record["kind"])
def test_flush_delivery_queue_delivers_pending_records(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
record = node_agent._build_delivery_record(
"job_event",
"/api/v1/ops/agent/jobs/7/events",
{
"node_code": "mainland-worker-01",
"event_type": "executor_received",
"message": "accepted",
"client_event_id": "evt-002",
},
"evt-002",
)
node_agent._store_pending_delivery(record)
with patch.object(node_agent, "_post", return_value={"code": 0, "message": "ok"}):
summary = node_agent._flush_delivery_queue(limit=10)
self.assertEqual(1, summary["delivered"])
self.assertEqual([], os.listdir(os.path.join(temp_dir, "pending")))
def test_flush_delivery_queue_moves_semantic_failure_to_dead_letter(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
record = node_agent._build_delivery_record(
"job_complete",
"/api/v1/ops/agent/jobs/9/complete",
{
"node_code": "mainland-worker-01",
"status": "success",
"client_request_id": "complete-002",
},
"complete-002",
)
node_agent._store_pending_delivery(record)
with patch.object(
node_agent,
"_post",
return_value={
"code": 1,
"message": "任务不存在或不属于当前节点",
"detail_code": "ops_job_not_owned_by_agent",
},
):
summary = node_agent._flush_delivery_queue(limit=10)
self.assertEqual(1, summary["dead_letter"])
self.assertEqual([], os.listdir(os.path.join(temp_dir, "pending")))
self.assertEqual(1, len(os.listdir(os.path.join(temp_dir, "dead-letter"))))
def test_replay_dead_letter_records_moves_record_back_to_pending_and_flushes(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
record = node_agent._build_delivery_record(
"job_event",
"/api/v1/ops/agent/jobs/7/events",
{
"node_code": "mainland-worker-01",
"event_type": "executor_received",
"message": "accepted",
"client_event_id": "evt-003",
},
"evt-003",
)
node_agent._store_dead_letter_delivery(
record,
reason="temporary failure",
detail_code="transient_error",
)
with patch.object(node_agent, "_post", return_value={"code": 0, "message": "ok"}):
ok, message, data = node_agent._execute_action(
"delivery.queue.replay",
{
"selector": {"record_id": "evt-003"},
"limit": 10,
"flush_after_replay": True,
"reason": "恢复后重放",
},
)
self.assertTrue(ok)
self.assertEqual("已重放 1 条死信记录", message)
self.assertEqual(1, data["replayed_total"])
self.assertEqual(1, data["flush_summary"]["delivered"])
self.assertEqual([], os.listdir(os.path.join(temp_dir, "dead-letter")))
self.assertEqual([], os.listdir(os.path.join(temp_dir, "pending")))
def test_discard_dead_letter_records_archives_record(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
record = node_agent._build_delivery_record(
"job_complete",
"/api/v1/ops/agent/jobs/8/complete",
{
"node_code": "mainland-worker-01",
"status": "failed",
"client_request_id": "complete-003",
},
"complete-003",
)
node_agent._store_dead_letter_delivery(
record,
reason="permanent failure",
detail_code="ops_job_invalid_status",
)
ok, message, data = node_agent._execute_action(
"delivery.queue.discard",
{
"selector": {"record_id": "complete-003"},
"limit": 10,
"discarded_by": "web-ui",
"reason": "确认不需要再补发",
},
)
self.assertTrue(ok)
self.assertEqual("已丢弃 1 条死信记录", message)
self.assertEqual(1, data["discarded_total"])
self.assertEqual([], os.listdir(os.path.join(temp_dir, "dead-letter")))
discarded_files = os.listdir(os.path.join(temp_dir, "discarded"))
self.assertEqual(1, len(discarded_files))
with open(os.path.join(temp_dir, "discarded", discarded_files[0]), "r", encoding="utf-8") as handle:
archived = json.load(handle)
self.assertEqual("web-ui", archived["discarded_by"])
self.assertEqual("确认不需要再补发", archived["discard_reason"])
def test_process_job_passes_envelope_context_to_completion(self) -> None:
job = {
"id": 12,
"job_code": "ops-20260418-000012",
"job_type": "release_deploy",
"action": "logs.collect",
"payload": {"service_name": "domaincheck-worker", "lines": 120},
"step_key": "worker_logs",
"step_title": "收集 Worker 日志",
"focus_ref": {"kind": "ops_job", "job_id": 12, "job_code": "ops-20260418-000012"},
"release_context": {"release_version": "v2026.04.18", "rollout_id": 3},
"step_ref": {"step_id": 22, "step_key": "worker_logs", "step_title": "收集 Worker 日志"},
}
with patch.object(node_agent, "_job_start") as mock_job_start, patch.object(
node_agent,
"_job_event",
) as mock_job_event, patch.object(
node_agent,
"_execute_action",
return_value=(True, "logs collected", {"summary": "收集完成", "stdout": "ok", "stderr": ""}),
) as mock_execute_action, patch.object(
node_agent,
"_job_complete",
return_value={"state": "sent"},
) as mock_job_complete:
node_agent._process_job(job)
self.assertEqual("worker_logs", mock_job_start.call_args.kwargs["job"]["step_key"])
self.assertEqual("ops_job", mock_job_event.call_args.kwargs["focus_ref"]["kind"])
self.assertEqual("已接单 logs.collect", mock_job_event.call_args.kwargs["summary_text"])
self.assertEqual("logs.collect", mock_execute_action.call_args.args[0])
self.assertEqual("v2026.04.18", mock_job_complete.call_args.kwargs["release_context"]["release_version"])
self.assertEqual("ops_job", mock_job_complete.call_args.kwargs["focus_ref"]["kind"])
self.assertEqual("收集完成", mock_job_complete.call_args.kwargs["summary_text"])
self.assertGreaterEqual(mock_job_complete.call_args.kwargs["duration_ms"], 0)
def test_process_job_continues_when_job_start_delivery_fails(self) -> None:
job = {
"id": 18,
"job_code": "ops-20260418-000018",
"job_type": "ops_action",
"action": "logs.collect",
"payload": {"service_name": "domaincheck-worker", "lines": 50},
"step_key": "worker_logs",
"step_title": "收集 Worker 日志",
"focus_ref": {"kind": "ops_job", "job_id": 18, "job_code": "ops-20260418-000018"},
"release_context": {},
"step_ref": {"step_id": 28, "step_key": "worker_logs", "step_title": "收集 Worker 日志"},
}
with patch.object(
node_agent,
"_job_start",
side_effect=urllib.error.URLError("temporary offline"),
) as mock_job_start, patch.object(
node_agent,
"_job_event",
return_value={"state": "queued"},
) as mock_job_event, patch.object(
node_agent,
"_execute_action",
return_value=(True, "logs collected", {"summary": "收集完成", "stdout": "ok", "stderr": ""}),
) as mock_execute_action, patch.object(
node_agent,
"_job_complete",
return_value={"state": "sent"},
) as mock_job_complete:
node_agent._process_job(job)
mock_job_start.assert_called_once()
mock_execute_action.assert_called_once()
mock_job_complete.assert_called_once()
event_payload = mock_job_event.call_args.kwargs["payload"]
self.assertEqual("failed_local", event_payload["start_delivery_state"])
self.assertIn("temporary offline", event_payload["start_delivery_error"])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,41 @@
import unittest
from app import node_agent
class NodeAgentReleaseNormalizationTests(unittest.TestCase):
def test_normalize_text_list_supports_strings_and_lists(self) -> None:
self.assertEqual(
["domaincheck-api", "domaincheck-worker"],
node_agent._normalize_text_list("domaincheck-api\ndomaincheck-worker"),
)
self.assertEqual(
["domaincheck-api", "domaincheck-worker"],
node_agent._normalize_text_list(["domaincheck-api", " domaincheck-worker "]),
)
self.assertEqual([], node_agent._normalize_text_list(""))
def test_health_check_services_fall_back_to_restart_services_when_missing(self) -> None:
result = node_agent._normalize_release_health_check_services(
{},
["domaincheck-worker"],
)
self.assertEqual(["domaincheck-worker"], result)
def test_health_check_services_can_be_explicitly_empty(self) -> None:
result = node_agent._normalize_release_health_check_services(
{"health_check_services": ""},
["domaincheck-worker"],
)
self.assertEqual([], result)
def test_health_check_services_accept_multiline_string(self) -> None:
result = node_agent._normalize_release_health_check_services(
{"health_check_services": "domaincheck-api\ndomaincheck-sync-agent"},
["domaincheck-worker"],
)
self.assertEqual(["domaincheck-api", "domaincheck-sync-agent"], result)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,115 @@
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from app.services.ops_action_executor_core import build_service_name_map, execute_structured_action
class _FakeRunner:
def __init__(self, responses: dict[tuple[str, ...], tuple[int, str, str]]) -> None:
self.responses = responses
self.calls: list[tuple[str, ...]] = []
def __call__(self, command: list[str], *, timeout: int = 60) -> tuple[int, str, str]:
key = tuple(command)
self.calls.append(key)
return self.responses.get(key, (0, "", ""))
class OpsActionExecutorCoreTests(unittest.TestCase):
def test_health_snapshot_includes_host_context_and_ok_flags(self) -> None:
runner = _FakeRunner(
{
("systemctl", "is-active", "domaincheck-api"): (0, "active", ""),
("systemctl", "is-active", "domaincheck-worker"): (3, "inactive", ""),
("systemctl", "is-active", "domaincheck-sync-agent"): (0, "active", ""),
("systemctl", "is-active", "domaincheck-node-agent"): (0, "active", ""),
}
)
ok, message, data = execute_structured_action(
"health.snapshot",
{},
service_names=build_service_name_map(
api_service_name="domaincheck-api",
worker_service_name="domaincheck-worker",
sync_agent_service_name="domaincheck-sync-agent",
),
runner=runner,
host_context={"hostname": "mainland-controller-01", "ip": "121.204.244.188"},
)
self.assertTrue(ok)
self.assertEqual("health snapshot collected", message)
self.assertEqual("mainland-controller-01", data["hostname"])
self.assertEqual("121.204.244.188", data["ip"])
self.assertTrue(data["checks"]["api"]["ok"])
self.assertFalse(data["checks"]["worker"]["ok"])
self.assertEqual(4, len(runner.calls))
def test_logs_collect_defaults_to_worker_service_and_clamps_lines(self) -> None:
runner = _FakeRunner(
{
("journalctl", "-u", "domaincheck-worker", "-n", "500", "--no-pager"): (
0,
"line-1\nline-2",
"",
)
}
)
ok, message, data = execute_structured_action(
"logs.collect",
{"lines": 9999},
service_names=build_service_name_map(
api_service_name="domaincheck-api",
worker_service_name="domaincheck-worker",
sync_agent_service_name="domaincheck-sync-agent",
),
runner=runner,
)
self.assertTrue(ok)
self.assertEqual("line-1\nline-2", message)
self.assertEqual("domaincheck-worker", data["service_name"])
self.assertEqual(500, data["lines"])
self.assertEqual(
("journalctl", "-u", "domaincheck-worker", "-n", "500", "--no-pager"),
runner.calls[0],
)
def test_logs_collect_falls_back_to_worker_log_file_when_journal_is_denied(self) -> None:
runner = _FakeRunner(
{
("journalctl", "-u", "domaincheck-worker", "-n", "120", "--no-pager"): (
1,
"",
"No journal files were opened due to insufficient permissions.",
)
}
)
with tempfile.TemporaryDirectory() as temp_dir:
log_path = Path(temp_dir) / "detect_worker.log"
log_path.write_text("worker-a\nworker-b\n", encoding="utf-8")
with patch("app.services.ops_action_executor_core.settings.domain_root", temp_dir):
ok, message, data = execute_structured_action(
"logs.collect",
{"lines": 120},
service_names=build_service_name_map(
api_service_name="domaincheck-api",
worker_service_name="domaincheck-worker",
sync_agent_service_name="domaincheck-sync-agent",
),
runner=runner,
)
self.assertTrue(ok)
self.assertEqual("worker-a\nworker-b", message)
self.assertEqual("file_fallback", data["collection_source"])
self.assertTrue(data["fallback_used"])
self.assertIn("detect_worker.log", data["log_paths"][0])
if __name__ == "__main__":
unittest.main()

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,96 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
from app.api.routes.ops import (
ops_doctor_decision,
ops_go_live_bundle,
ops_go_live_review,
ops_go_live_signoff,
ops_go_live_summary,
)
class OpsApiRoutesTestCase(unittest.TestCase):
@patch("app.api.routes.ops.get_ops_go_live_signoff")
def test_go_live_signoff_route_uses_service_payload(self, mock_get_ops_go_live_signoff) -> None:
mock_get_ops_go_live_signoff.return_value = {
"signoff_status": "ready",
"headline": "ok",
}
response = ops_go_live_signoff(base_url="http://127.0.0.1:8100", report_dir="/tmp/signoff")
self.assertEqual(0, response.code)
self.assertEqual("ready", response.data["signoff_status"])
self.assertEqual("ok", response.data["headline"])
mock_get_ops_go_live_signoff.assert_called_once_with(
base_url="http://127.0.0.1:8100",
report_dir="/tmp/signoff",
)
@patch("app.api.routes.ops.get_ops_go_live_summary")
def test_go_live_summary_route_passes_base_url(self, mock_get_ops_go_live_summary) -> None:
mock_get_ops_go_live_summary.return_value = {
"go_live_status": "attention",
}
response = ops_go_live_summary(base_url="http://example.test")
self.assertEqual(0, response.code)
self.assertEqual("attention", response.data["go_live_status"])
mock_get_ops_go_live_summary.assert_called_once_with(base_url="http://example.test")
@patch("app.api.routes.ops.get_ops_go_live_bundle")
def test_go_live_bundle_route_passes_base_url_and_report_dir(self, mock_get_ops_go_live_bundle) -> None:
mock_get_ops_go_live_bundle.return_value = {
"status": "missing",
}
response = ops_go_live_bundle(base_url="http://example.test", report_dir="/tmp/go-live")
self.assertEqual(0, response.code)
self.assertEqual("missing", response.data["status"])
mock_get_ops_go_live_bundle.assert_called_once_with(
base_url="http://example.test",
report_dir="/tmp/go-live",
)
@patch("app.api.routes.ops.get_ops_go_live_review")
def test_go_live_review_route_passes_base_url_and_report_dir(self, mock_get_ops_go_live_review) -> None:
mock_get_ops_go_live_review.return_value = {
"status": "attention",
"headline": "review",
}
response = ops_go_live_review(base_url="http://example.test", report_dir="/tmp/review")
self.assertEqual(0, response.code)
self.assertEqual("attention", response.data["status"])
self.assertEqual("review", response.data["headline"])
mock_get_ops_go_live_review.assert_called_once_with(
base_url="http://example.test",
report_dir="/tmp/review",
)
@patch("app.api.routes.ops.get_ops_doctor_decision")
def test_doctor_decision_route_passes_base_url_and_report_dir(self, mock_get_ops_doctor_decision) -> None:
mock_get_ops_doctor_decision.return_value = {
"status": "attention",
"headline": "doctor",
}
response = ops_doctor_decision(base_url="http://example.test", report_dir="/tmp/doctor")
self.assertEqual(0, response.code)
self.assertEqual("attention", response.data["status"])
self.assertEqual("doctor", response.data["headline"])
mock_get_ops_doctor_decision.assert_called_once_with(
base_url="http://example.test",
report_dir="/tmp/doctor",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,44 @@
import unittest
from app.services.ops_command_service import build_bash_command, ops_script_path
class OpsCommandServiceTests(unittest.TestCase):
def test_ops_script_path_prefixes_multi_region_directory(self) -> None:
self.assertEqual(
"domain-api/deploy/multi-region/drive_ops_center.sh",
ops_script_path("drive_ops_center.sh"),
)
def test_ops_script_path_normalizes_leading_slash(self) -> None:
self.assertEqual(
"domain-api/deploy/multi-region/check_ops_center_stack.sh",
ops_script_path("/check_ops_center_stack.sh"),
)
def test_build_bash_command_joins_non_empty_args(self) -> None:
self.assertEqual(
"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 handover_first_gap",
build_bash_command(
"drive_ops_center.sh",
"driver-resolve",
"http://127.0.0.1:8100",
"handover_first_gap",
),
)
def test_build_bash_command_skips_blank_args(self) -> None:
self.assertEqual(
"bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 launchpad",
build_bash_command(
"drive_release_hub.sh",
"http://127.0.0.1:8100",
"",
"launchpad",
" ",
),
)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,391 @@
import json
import unittest
from datetime import datetime
from unittest.mock import MagicMock, patch
from app.core.config import settings
from app.services.ops_job_service import (
_execute_control_plane_job,
_serialize_job_row,
create_ops_job,
dispatch_ops_job,
upsert_managed_node,
)
class OpsJobServiceTests(unittest.TestCase):
def test_serialize_job_row_adds_execution_mode_label(self) -> None:
row = (
9,
"ops-2026041801-abcd12",
"logs.collect",
"node",
"mainland-worker-01",
"queued",
"remote-agent",
"api",
json.dumps({"service_name": "domaincheck-worker", "lines": 120}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
"",
datetime(2026, 4, 18, 10, 0, 0),
None,
None,
datetime(2026, 4, 18, 10, 0, 0),
"low",
False,
"approved",
"",
None,
"",
"",
None,
json.dumps({}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
None,
)
job = _serialize_job_row(row)
self.assertEqual("remote-agent", job["execution_mode"])
self.assertEqual("远端 Agent", job["execution_mode_label"])
self.assertEqual("排队中", job["status_label"])
self.assertEqual("已审批", job["approval_status_label"])
self.assertEqual(["mainland-worker-01"], job["target_node_codes"])
self.assertEqual(0, job["steps_total"])
self.assertFalse(job["steps_loaded"])
self.assertFalse(job["is_compact"])
self.assertEqual("ops_job", job["focus_ref"]["kind"])
self.assertEqual(9, job["focus_ref"]["job_id"])
self.assertIn("目标节点 mainland-worker-01", job["summary_text"])
def test_serialize_job_row_counts_loaded_steps(self) -> None:
row = (
10,
"ops-2026041801-efgh34",
"diagnostics.collect",
"node",
"mainland-worker-02",
"running",
"ssh",
"web-ui",
json.dumps({}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
"",
datetime(2026, 4, 18, 10, 5, 0),
datetime(2026, 4, 18, 10, 5, 2),
None,
datetime(2026, 4, 18, 10, 5, 3),
"medium",
True,
"approved",
"admin",
datetime(2026, 4, 18, 10, 5, 1),
"",
"",
datetime(2026, 4, 18, 10, 5, 2),
json.dumps({}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
None,
)
job = _serialize_job_row(
row,
steps=[
{"id": 1, "status": "success"},
{"id": 2, "status": "running"},
{"id": 3, "status": "failed"},
],
)
self.assertEqual("执行中", job["status_label"])
self.assertEqual("已审批", job["approval_status_label"])
self.assertEqual(3, job["steps_total"])
self.assertEqual(1, job["steps_running"])
self.assertEqual(1, job["steps_success"])
self.assertEqual(1, job["steps_failed"])
self.assertEqual(2, job["steps_terminal"])
self.assertTrue(job["steps_loaded"])
self.assertIn("mainland-worker-02", job["summary_text"])
@patch("app.services.ops_job_service.list_managed_nodes")
@patch("app.services.ops_agent_service.build_node_agent_bootstrap_plan")
def test_execute_control_plane_bootstrap_uses_managed_node_defaults(
self,
mock_build_node_agent_bootstrap_plan,
mock_list_managed_nodes,
) -> None:
mock_list_managed_nodes.return_value = [
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
}
]
mock_build_node_agent_bootstrap_plan.return_value = (
True,
"节点 Agent 接入方案已生成",
{
"token_preview": "agent-...1234",
"bootstrap_plan": {
"bootstrap_script_name": "bootstrap-node-agent-mainland-worker-02.sh",
},
},
)
ok, message, data = _execute_control_plane_job(
"node.bootstrap",
target_node_code="mainland-worker-02",
payload={
"root_dir": "/opt/domaincheck",
"expires_in_hours": 12,
},
requested_by="unit-test",
metadata={"source": "test"},
)
self.assertTrue(ok)
self.assertEqual("节点 Agent 接入方案已生成", message)
self.assertEqual("control-plane", data["execution_mode"])
self.assertEqual("mainland", data["node_region"])
self.assertEqual("worker", data["node_role"])
mock_build_node_agent_bootstrap_plan.assert_called_once_with(
node_code="mainland-worker-02",
node_region="mainland",
node_role="worker",
issued_by="unit-test",
expires_in_hours=12,
control_plane_base_url="",
root_dir="/opt/domaincheck",
metadata={
"source": "test",
"issued_from": "ops-job/control-plane",
"target_node_code": "mainland-worker-02",
"node_region": "mainland",
"node_role": "worker",
},
)
@patch("app.services.ops_job_service.get_db")
def test_upsert_managed_node_preserves_remote_fields_when_sync_payload_omits_them(self, mock_get_db) -> None:
existing_metadata = {"operator_note": "keep-me"}
merged_metadata = {"operator_note": "keep-me", "cluster_status": "online"}
existing_row = (
"mainland-worker-01",
"mainland",
"worker",
"Mainland Worker 01",
"121.204.244.248",
22,
"root",
"key",
"stable",
False,
json.dumps(existing_metadata, ensure_ascii=False),
datetime(2026, 4, 18, 1, 0, 0),
None,
datetime(2026, 4, 18, 1, 5, 0),
)
returned_row = (
"mainland-worker-01",
"mainland",
"worker",
"S244-248",
"121.204.244.248",
22,
"root",
"key",
"stable",
False,
json.dumps(merged_metadata, ensure_ascii=False),
datetime(2026, 4, 18, 1, 0, 0),
None,
datetime(2026, 4, 18, 1, 6, 0),
)
conn = MagicMock()
cursor = MagicMock()
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
db_ctx.__exit__.return_value = False
cursor_ctx.__enter__.return_value = cursor
cursor_ctx.__exit__.return_value = False
conn.cursor.return_value = cursor_ctx
mock_get_db.return_value = db_ctx
cursor.fetchone.side_effect = [existing_row, returned_row]
ok, message, data = upsert_managed_node(
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"title": "S244-248",
"metadata": {
"cluster_status": "online",
},
}
)
self.assertTrue(ok)
self.assertEqual("托管节点已保存", message)
node = data["node"]
self.assertEqual("121.204.244.248", node["ssh_host"])
self.assertEqual("root", node["ssh_user"])
self.assertFalse(node["is_enabled"])
self.assertEqual("keep-me", node["metadata"]["operator_note"])
self.assertEqual("online", node["metadata"]["cluster_status"])
insert_call = cursor.execute.call_args_list[-1]
insert_params = insert_call.args[1]
self.assertEqual("121.204.244.248", insert_params[4])
self.assertEqual(22, insert_params[5])
self.assertEqual("root", insert_params[6])
self.assertFalse(insert_params[9])
self.assertEqual(merged_metadata, json.loads(insert_params[10]))
@patch("app.services.ops_agent_service.append_ops_job_event")
@patch("app.services.ops_job_service._execute_ssh_job_record")
@patch("app.services.ops_job_service.get_db")
def test_dispatch_ops_job_ssh_executes_via_ssh_record(
self,
mock_get_db,
mock_execute_ssh_job_record,
mock_append_ops_job_event,
) -> None:
conn = MagicMock()
cursor = MagicMock()
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
db_ctx.__exit__.return_value = False
cursor_ctx.__enter__.return_value = cursor
cursor_ctx.__exit__.return_value = False
conn.cursor.return_value = cursor_ctx
mock_get_db.return_value = db_ctx
cursor.fetchone.return_value = (
9,
"ops-2026041801-abcd12",
"logs.collect",
"node",
"mainland-worker-01",
"queued",
"ssh",
"api",
json.dumps({"service_name": "domaincheck-worker", "lines": 120}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
"",
datetime(2026, 4, 18, 10, 0, 0),
None,
None,
datetime(2026, 4, 18, 10, 0, 0),
"low",
False,
"approved",
"",
None,
"",
"",
None,
json.dumps({}, ensure_ascii=False),
json.dumps({}, ensure_ascii=False),
None,
)
mock_execute_ssh_job_record.return_value = (
True,
"worker logs collected",
{"job": {"id": 9, "status": "success"}},
)
ok, message, data = dispatch_ops_job(9)
self.assertTrue(ok)
self.assertEqual("已按 SSH 执行派发worker logs collected", message)
self.assertEqual("success", data["job"]["status"])
mock_execute_ssh_job_record.assert_called_once_with(9)
mock_append_ops_job_event.assert_called_once()
event_kwargs = mock_append_ops_job_event.call_args.kwargs
self.assertEqual("job_dispatch_requested", event_kwargs["event_type"])
self.assertEqual("mainland-worker-01", event_kwargs["node_code"])
self.assertEqual("ssh", event_kwargs["payload"]["execution_mode"])
@patch("app.services.ops_agent_service.append_ops_job_event")
@patch("app.services.ops_job_service.get_ops_job")
@patch("app.services.ops_job_service._execute_local_job")
@patch("app.services.ops_policy_service.preview_ops_job_policy")
@patch("app.services.ops_job_service.get_db")
def test_create_ops_job_auto_uses_local_runtime_for_local_structured_action(
self,
mock_get_db,
mock_preview_ops_job_policy,
mock_execute_local_job,
mock_get_ops_job,
mock_append_ops_job_event,
) -> None:
conn = MagicMock()
cursor = MagicMock()
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
db_ctx.__exit__.return_value = False
cursor_ctx.__enter__.return_value = cursor
cursor_ctx.__exit__.return_value = False
conn.cursor.return_value = cursor_ctx
mock_get_db.return_value = db_ctx
cursor.fetchone.return_value = (42,)
mock_preview_ops_job_policy.return_value = {
"risk_level": "low",
"approval_required": False,
"blocked": False,
"blocking_reasons": [],
}
mock_execute_local_job.return_value = (
True,
"worker logs collected",
{"executor": "local-runtime"},
)
mock_get_ops_job.return_value = {
"id": 42,
"status": "success",
"execution_mode": "local-runtime",
}
ok, message, data = create_ops_job(
{
"action": "logs.collect",
"target_node_code": settings.node_code,
"payload": {
"service_name": "domaincheck-worker",
"lines": 120,
},
"requested_by": "unit-test",
}
)
self.assertTrue(ok)
self.assertEqual("worker logs collected", message)
self.assertTrue(data["executed_immediately"])
self.assertEqual("local-runtime", data["job"]["execution_mode"])
mock_execute_local_job.assert_called_once_with(
"logs.collect",
{
"service_name": "domaincheck-worker",
"lines": 120,
},
)
insert_call = next(
call
for call in cursor.execute.call_args_list
if "INSERT INTO ops_jobs" in str(call.args[0])
)
insert_params = insert_call.args[1]
self.assertEqual("local-runtime", insert_params[5])
mock_append_ops_job_event.assert_called_once()
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,220 @@
import unittest
from unittest.mock import patch
from app.services.ops_playbook_service import (
get_recent_ops_playbook_runs,
list_ops_playbook_run_events,
preview_ops_playbook,
)
class OpsPlaybookServiceTests(unittest.TestCase):
def test_onboarding_bootstrap_preview_uses_control_plane(self) -> None:
ok, message, data = preview_ops_playbook(
{
"playbook_key": "onboarding.bootstrap",
"node_codes": ["mainland-worker-02"],
}
)
self.assertTrue(ok, message)
self.assertEqual("playbook 预览生成成功", message)
self.assertEqual("onboarding.bootstrap", data["playbook"]["key"])
self.assertEqual("control-plane", data["execution_mode"])
self.assertEqual("控制面", data["execution_mode_label"])
self.assertEqual(["control-plane"], data["playbook"]["execution_modes"])
self.assertEqual("控制面", data["playbook"]["default_execution_mode_label"])
self.assertEqual("控制面", data["playbook"]["execution_mode_options"][0]["label"])
self.assertTrue(data["auto_approve"])
self.assertEqual(1, data["step_count"])
step = data["steps"][0]
self.assertEqual("bootstrap", step["step_key"])
self.assertEqual("node.bootstrap", step["template_key"])
self.assertEqual("node.bootstrap", step["action"])
self.assertEqual("control-plane", step["execution_mode"])
self.assertEqual(1, step["expected_jobs"])
def test_scene_logs_key_preview_allows_ssh_execution_mode(self) -> None:
ok, message, data = preview_ops_playbook(
{
"playbook_key": "scene.logs.key",
"node_codes": ["mainland-worker-01"],
"execution_mode": "ssh",
}
)
self.assertTrue(ok, message)
self.assertEqual("playbook 预览生成成功", message)
self.assertEqual("ssh", data["execution_mode"])
self.assertEqual("SSH", data["execution_mode_label"])
self.assertEqual(["remote-agent", "ssh"], data["playbook"]["execution_modes"])
self.assertEqual("ssh", data["steps"][0]["execution_mode"])
self.assertEqual("SSH", data["steps"][0]["execution_mode_label"])
def test_onboarding_bootstrap_preview_rejects_unsupported_execution_mode(self) -> None:
ok, message, data = preview_ops_playbook(
{
"playbook_key": "onboarding.bootstrap",
"node_codes": ["mainland-worker-02"],
"execution_mode": "ssh",
}
)
self.assertFalse(ok)
self.assertEqual("该 playbook 仅支持 control-plane", message)
self.assertEqual(["control-plane"], data["playbook"]["execution_modes"])
@patch("app.services.ops_playbook_service.list_managed_nodes_with_agent_state")
@patch("app.services.ops_playbook_service.list_ops_jobs")
def test_recent_onboarding_bootstrap_run_focus_summary_reflects_agent_state(
self,
mock_list_ops_jobs,
mock_list_managed_nodes_with_agent_state,
) -> None:
mock_list_ops_jobs.return_value = [
{
"id": 31,
"job_code": "ops-bootstrap-31",
"action": "node.bootstrap",
"status": "success",
"target_node_code": "mainland-worker-02",
"created_at": "2026-04-17 10:00:00",
"updated_at": "2026-04-17 10:01:00",
"metadata": {
"playbook": {
"run_code": "pbr-bootstrap-1",
"key": "onboarding.bootstrap",
"title": "生成节点接入工单",
"group_key": "onboarding",
"group_title": "接管验收",
"requested_by": "tester",
"execution_mode": "control-plane",
"step_key": "bootstrap",
"step_title": "生成 Node Agent 接入工单",
"step_order": 1,
"step_count": 1,
"expected_jobs_total": 1,
"target_nodes_total": 1,
"template_key": "node.bootstrap",
}
},
"result": {
"bootstrap_plan": {
"bootstrap_script_name": "bootstrap-node-agent-mainland-worker-02.sh",
}
},
}
]
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-worker-02",
"agent_state": "pending_bootstrap",
"is_managed": True,
}
]
}
payload = get_recent_ops_playbook_runs(limit=10, scan_limit=20)
self.assertEqual(1, len(payload["runs"]))
run = payload["runs"][0]
self.assertEqual("success", run["status"])
self.assertEqual("成功", run["status_label"])
self.assertEqual("控制面", run["execution_mode_label"])
self.assertEqual("warning", run["focus_level"])
self.assertIn("待执行 bootstrap 脚本", run["focus_summary"])
self.assertEqual(run["focus_summary"], run["summary_text"])
self.assertEqual(1, run["steps_total"])
self.assertEqual("pbr-bootstrap-1", run["focus_ref"]["run_code"])
self.assertEqual("生成 Node Agent 接入工单", run["steps"][0]["title"])
self.assertEqual("成功", run["steps"][0]["status_label"])
self.assertTrue(run["steps"][0]["summary_text"])
@patch("app.services.ops_playbook_service.list_managed_nodes_with_agent_state")
@patch("app.services.ops_playbook_service.list_ops_job_events_for_jobs")
@patch("app.services.ops_playbook_service.list_ops_jobs")
def test_playbook_run_events_add_contract_aliases(
self,
mock_list_ops_jobs,
mock_list_ops_job_events_for_jobs,
mock_list_managed_nodes_with_agent_state,
) -> None:
mock_list_managed_nodes_with_agent_state.return_value = {"nodes": []}
mock_list_ops_jobs.return_value = [
{
"id": 41,
"job_code": "ops-inspection-41",
"action": "logs.collect",
"status": "running",
"target_node_code": "mainland-worker-01",
"created_at": "2026-04-18 06:30:00",
"updated_at": "2026-04-18 06:31:00",
"metadata": {
"playbook": {
"run_code": "pbr-inspection-1",
"key": "inspection.standard",
"title": "标准巡检",
"group_key": "diagnostics",
"group_title": "标准巡检",
"requested_by": "tester",
"execution_mode": "remote-agent",
"step_key": "worker_logs",
"step_title": "收集 Worker 日志",
"step_order": 2,
"step_count": 3,
"expected_jobs_total": 3,
"target_nodes_total": 1,
"template_key": "logs.collect",
}
},
}
]
mock_list_ops_job_events_for_jobs.return_value = [
{
"id": 501,
"event_key": "job-event:501",
"job_id": 41,
"node_code": "mainland-worker-01",
"event_type": "job_running",
"level": "info",
"level_label": "信息",
"message": "开始收集 Worker 日志",
"summary": "开始收集 Worker 日志",
"summary_text": "远端 Agent 已开始收集 Worker 日志",
"focus_ref": {
"kind": "ops_job",
"job_id": 41,
"job_code": "ops-inspection-41",
"action": "logs.collect",
"target_node_code": "mainland-worker-01",
"event_key": "job-event:501",
"step_key": "worker_logs",
},
"payload": {},
"created_at": "2026-04-18 06:31:00",
"occurred_at": "2026-04-18 06:31:00",
"ui_intent": {"kind": "job_detail", "job_id": 41, "focus_event_id": 501},
}
]
payload = list_ops_playbook_run_events("pbr-inspection-1", limit=10)
self.assertEqual("inspection.standard", payload["playbook_run"]["playbook_key"])
self.assertEqual(1, len(payload["events"]))
event = payload["events"][0]
self.assertEqual("收口中", event["job_status_label"])
self.assertEqual("远端 Agent 已开始收集 Worker 日志", event["summary_text"])
self.assertEqual("worker_logs", event["focus_ref"]["focus_step_key"])
self.assertEqual("job-event:501", event["focus_ref"]["event_key"])
self.assertEqual("ops_job", event["source_focus_ref"]["kind"])
self.assertEqual("job-event:501", event["source_focus_ref"]["event_key"])
summary = payload["summary"]
self.assertEqual(1, summary["returned_total"])
self.assertEqual({"job_running": 1}, summary["event_type_counts"])
self.assertEqual({"running": 1}, summary["job_status_counts"])
self.assertEqual(10, summary["filters"]["limit"])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,253 @@
import unittest
from unittest.mock import patch
from app.services.ops_policy_service import preview_ops_job_policy
class OpsPolicyServiceTests(unittest.TestCase):
@patch("app.services.ops_policy_service.get_cluster_snapshot")
def test_node_bootstrap_preview_requires_control_plane(self, mock_cluster_snapshot) -> None:
mock_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
"detect_participating": False,
}
],
"summary": {
"online_worker_nodes": 2,
"dedicated_online_worker_nodes": 1,
"online_control_nodes": 1,
"busy_nodes": [],
},
}
preview = preview_ops_job_policy(
{
"action": "node.bootstrap",
"target_type": "node",
"target_node_code": "mainland-worker-01",
"execution_mode": "remote-agent",
"payload": {},
}
)
self.assertEqual("critical", preview["risk_level"])
self.assertTrue(preview["blocked"])
self.assertTrue(preview["approval_required"])
self.assertIn("node.bootstrap 仅支持 control-plane", " ".join(preview["blocking_reasons"]))
@patch("app.services.ops_policy_service.get_cluster_snapshot")
def test_deploy_release_batch_preview_summarizes_node_risks(self, mock_cluster_snapshot) -> None:
mock_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-controller-01",
"region": "mainland",
"role": "control",
"status": "busy",
"current_load": 3,
"is_effective_worker": True,
"detect_participating": False,
},
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
"detect_participating": False,
},
],
"summary": {
"online_worker_nodes": 2,
"dedicated_online_worker_nodes": 1,
"online_control_nodes": 1,
"busy_nodes": ["mainland-controller-01"],
},
}
preview = preview_ops_job_policy(
{
"action": "deploy.release",
"target_type": "batch",
"target_node_codes": ["mainland-controller-01", "mainland-worker-01"],
"execution_mode": "remote-agent",
"payload": {
"release_version": "2026.04.18-rc1",
"artifact_url": "https://example.com/release.tar.gz",
},
}
)
self.assertEqual("batch", preview["target_type"])
self.assertTrue(preview["blocked"])
self.assertTrue(preview["approval_required"])
self.assertEqual(2, preview["target_summary"]["nodes_total"])
self.assertEqual(1, preview["target_summary"]["blocked_nodes"])
self.assertEqual(1, preview["target_summary"]["approval_nodes"])
self.assertEqual(1, preview["target_summary"]["warning_nodes"])
self.assertEqual(2, len(preview["target_nodes"]))
controller_row = next(item for item in preview["target_nodes"] if item["node_code"] == "mainland-controller-01")
self.assertTrue(controller_row["blocked"])
self.assertTrue(controller_row["approval_required"])
@patch("app.services.ops_policy_service.get_cluster_snapshot")
def test_deploy_release_preview_blocks_missing_artifact_and_wrong_execution_mode(self, mock_cluster_snapshot) -> None:
mock_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
"detect_participating": False,
}
],
"summary": {
"online_worker_nodes": 2,
"dedicated_online_worker_nodes": 1,
"online_control_nodes": 1,
"busy_nodes": [],
},
}
preview = preview_ops_job_policy(
{
"action": "deploy.release",
"target_type": "node",
"target_node_code": "mainland-worker-01",
"execution_mode": "local-runtime",
"payload": {
"release_version": "2026.04.18-rc1",
"artifact_url": "",
},
}
)
self.assertTrue(preview["blocked"])
self.assertIn("deploy.release 缺少 artifact_url", " ".join(preview["blocking_reasons"]))
self.assertIn("deploy.release 仅支持 remote-agent 或 ssh", " ".join(preview["blocking_reasons"]))
@patch("app.services.ops_policy_service.get_cluster_snapshot")
def test_deploy_release_preview_allows_ssh_execution_mode(self, mock_cluster_snapshot) -> None:
mock_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
"detect_participating": False,
}
],
"summary": {
"online_worker_nodes": 2,
"dedicated_online_worker_nodes": 1,
"online_control_nodes": 1,
"busy_nodes": [],
},
}
preview = preview_ops_job_policy(
{
"action": "deploy.release",
"target_type": "node",
"target_node_code": "mainland-worker-01",
"execution_mode": "ssh",
"payload": {
"release_version": "2026.04.18-rc1",
"artifact_url": "https://example.com/release.tar.gz",
},
}
)
self.assertFalse(any("仅支持 remote-agent 或 ssh" in item for item in preview["blocking_reasons"]))
self.assertTrue(preview["approval_required"])
@patch("app.services.ops_policy_service.get_cluster_snapshot")
def test_service_status_preview_blocks_control_plane_execution_mode(self, mock_cluster_snapshot) -> None:
mock_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
"detect_participating": False,
}
],
"summary": {
"online_worker_nodes": 2,
"dedicated_online_worker_nodes": 1,
"online_control_nodes": 1,
"busy_nodes": [],
},
}
preview = preview_ops_job_policy(
{
"action": "service.status",
"target_type": "node",
"target_node_code": "mainland-worker-01",
"execution_mode": "control-plane",
"payload": {"service_name": "domaincheck-worker"},
}
)
self.assertTrue(preview["blocked"])
self.assertIn("service.status 当前不支持 control-plane 执行方式", " ".join(preview["blocking_reasons"]))
@patch("app.services.ops_policy_service.get_cluster_snapshot")
def test_logs_collect_preview_blocks_local_runtime_on_remote_node(self, mock_cluster_snapshot) -> None:
mock_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
"detect_participating": False,
}
],
"summary": {
"online_worker_nodes": 2,
"dedicated_online_worker_nodes": 1,
"online_control_nodes": 1,
"busy_nodes": [],
},
}
preview = preview_ops_job_policy(
{
"action": "logs.collect",
"target_type": "node",
"target_node_code": "mainland-worker-01",
"execution_mode": "local-runtime",
"payload": {
"service_name": "domaincheck-worker",
"lines": 120,
},
}
)
self.assertTrue(preview["blocked"])
self.assertIn("local-runtime 仅支持当前控制面本机节点", " ".join(preview["blocking_reasons"]))
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,89 @@
import io
import tarfile
import tempfile
import unittest
from pathlib import Path
from app.services.ops_release_executor_core import build_remote_release_action_script, execute_release_action
class _BytesResponse:
def __init__(self, data: bytes) -> None:
self._data = data
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def read(self, *args, **kwargs) -> bytes:
return self._data
def _build_release_archive() -> bytes:
buffer = io.BytesIO()
with tarfile.open(fileobj=buffer, mode="w:gz") as archive:
content = b"hello-release"
info = tarfile.TarInfo(name="bundle/README.txt")
info.size = len(content)
archive.addfile(info, io.BytesIO(content))
return buffer.getvalue()
class OpsReleaseExecutorCoreTests(unittest.TestCase):
def test_execute_release_action_deploys_archive_and_switches_current(self) -> None:
archive_bytes = _build_release_archive()
commands: list[tuple[str, ...]] = []
def run_command(command: list[str], *, timeout: int = 60):
commands.append(tuple(command))
if command[:2] == ["systemctl", "restart"]:
return 0, "", ""
if command[:2] == ["systemctl", "is-active"]:
return 0, "active", ""
return 0, "", ""
with tempfile.TemporaryDirectory() as tmpdir:
ok, message, data = execute_release_action(
{
"release_version": "2026.04.18-rc1",
"artifact_url": "https://example.com/domaincheck.tar.gz",
"install_root": tmpdir,
"restart_services": ["domaincheck-worker"],
"health_check_services": ["domaincheck-worker"],
"health_check_urls": [],
"rollback_on_failure": "true",
"switch_current": "true",
},
run_command=run_command,
default_api_service_name="domaincheck-api",
urlopen_func=lambda request, timeout=120: _BytesResponse(archive_bytes),
user_agent="unit-test/0.1",
)
self.assertTrue(ok, message)
self.assertEqual("release deployed", message)
self.assertEqual("2026.04.18-rc1", data["release_version"])
current_link = Path(tmpdir) / "current"
self.assertTrue(current_link.is_symlink())
release_dir = Path(data["release_dir"])
self.assertTrue(release_dir.exists())
self.assertTrue((release_dir / ".release-meta.json").exists())
self.assertIn(("systemctl", "restart", "domaincheck-worker"), commands)
self.assertIn(("systemctl", "is-active", "domaincheck-worker"), commands)
def test_build_remote_release_action_script_is_valid_python(self) -> None:
script = build_remote_release_action_script(
{
"release_version": "2026.04.18-rc1",
"artifact_url": "https://example.com/domaincheck.tar.gz",
}
)
compile(script, "<remote-release-script>", "exec")
self.assertIn("execute_release_action(", script)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,93 @@
import json
import unittest
from datetime import datetime
from app.services.ops_release_service import _serialize_release_row, _serialize_rollout_row
class OpsReleaseServiceContractTests(unittest.TestCase):
def test_serialize_release_row_adds_contract_aliases(self) -> None:
row = (
9,
"2026.04.18+bd6bcb2",
"stable",
"bd6bcb2",
"ready",
"https://example.com/domaincheck.tar.gz",
"sha256:test",
"release notes",
json.dumps(
{
"source": "package",
"manifest": {
"artifact_kind": "tarball",
"package_name": "domaincheck-2026.04.18+bd6bcb2.tgz",
"included_services": ["domaincheck-api", "domaincheck-worker"],
"rollback_hint": "切回上一个 current",
},
}
),
"www",
datetime(2026, 4, 18, 9, 0, 0),
datetime(2026, 4, 18, 9, 5, 0),
datetime(2026, 4, 18, 9, 6, 0),
)
payload = _serialize_release_row(row)
self.assertEqual("已就绪", payload["status_label"])
self.assertIn("已就绪", payload["summary_text"])
self.assertEqual("release_hub", payload["focus_ref"]["kind"])
self.assertEqual(9, payload["focus_ref"]["release_id"])
self.assertEqual("release_detail", payload["focus_ref"]["section"])
self.assertEqual("tarball", payload["manifest"]["artifact_kind"])
self.assertEqual("domaincheck-2026.04.18+bd6bcb2.tgz", payload["manifest"]["package_name"])
self.assertEqual(["domaincheck-api", "domaincheck-worker"], payload["manifest"]["included_services"])
def test_serialize_rollout_row_adds_contract_aliases(self) -> None:
row = (
11,
9,
"rollout-20260418090000-a1b2",
json.dumps({"region": "mainland", "role": "worker"}),
json.dumps(
{
"items": [
{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker"},
{"node_code": "mainland-worker-02", "region": "mainland", "role": "worker"},
]
}
),
json.dumps({"execution_mode": "ssh", "batch_size": 1}),
"running",
1,
2,
2,
1,
json.dumps(
{
"status": "running",
"launched_targets": 1,
"remaining_targets": 1,
"active_jobs": 1,
"current_batch_index": 1,
"batches_total": 2,
}
),
"web-ui",
datetime(2026, 4, 18, 9, 10, 0),
datetime(2026, 4, 18, 9, 11, 0),
)
payload = _serialize_rollout_row(row)
self.assertEqual("执行中", payload["status_label"])
self.assertEqual(["mainland-worker-01", "mainland-worker-02"], payload["target_node_codes"])
self.assertIn("当前已覆盖 1/2 台节点", payload["summary_text"])
self.assertEqual("release_rollout", payload["focus_ref"]["kind"])
self.assertEqual(11, payload["focus_ref"]["rollout_id"])
self.assertEqual("rollout_jobs", payload["focus_ref"]["section"])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,432 @@
import unittest
from unittest.mock import patch
from app.services.ops_release_service import (
build_release_execution_mode_recommendation,
build_release_rollout_gate,
build_rollout_target_operational_readiness,
)
class _EmptyCursor:
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def execute(self, query, params=None) -> None:
self.query = query
self.params = params
def fetchall(self):
return []
def fetchone(self):
return None
class _EmptyConnection:
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def cursor(self):
return _EmptyCursor()
class OpsReleaseServiceExecutionModeTests(unittest.TestCase):
@patch("app.services.ops_release_service.get_db")
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
@patch("app.services.ops_job_service.list_managed_nodes")
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
def test_build_rollout_target_operational_readiness_accepts_ssh_ready_nodes(
self,
mock_get_cluster_snapshot,
mock_list_managed_nodes,
mock_get_managed_node_onboarding,
mock_ensure_ops_agent_schema,
mock_get_db,
) -> None:
mock_ensure_ops_agent_schema.return_value = None
mock_get_managed_node_onboarding.return_value = {
"onboarding_stage": {"code": "acceptance_ready", "label": "待接管验收"},
"summary": "Node Agent 已在线,可以直接跑 onboarding.acceptance。",
"recovery_decision": {
"action": "run_acceptance",
"label": "执行接管验收",
"summary": "Node Agent 已在线,可以直接跑 onboarding.acceptance。",
"command_hint": "bash drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-worker-01 cli",
"window": "acceptance",
},
}
mock_get_db.return_value = _EmptyConnection()
mock_get_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
}
]
}
mock_list_managed_nodes.return_value = [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"ssh_host": "121.204.244.248",
"ssh_user": "root",
"metadata": {},
"last_seen_at": "",
}
]
readiness = build_rollout_target_operational_readiness(
[{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker"}],
execution_mode="ssh",
)
self.assertEqual("ssh", readiness["execution_mode"])
self.assertEqual(1, readiness["summary"]["ssh_ready_nodes"])
self.assertEqual(1, readiness["summary"]["execution_ready_nodes"])
self.assertEqual([], readiness["blocking_reasons"])
self.assertTrue(readiness["rows"][0]["ssh_ready"])
self.assertTrue(readiness["rows"][0]["execution_ready"])
self.assertTrue(readiness["rows"][0]["agent_managed"])
self.assertEqual("ssh", readiness["rows"][0]["execution_mode"])
self.assertEqual("SSH", readiness["rows"][0]["execution_mode_label"])
self.assertEqual("medium", readiness["rows"][0]["risk_level"])
self.assertEqual("run_standard_inspection", readiness["rows"][0]["recommended_action_code"])
self.assertEqual("acceptance_ready", readiness["rows"][0]["onboarding_stage_code"])
self.assertEqual("run_acceptance", readiness["rows"][0]["recovery_action"])
self.assertEqual(1, readiness["summary"]["onboarding_acceptance_ready_nodes"])
@patch("app.services.ops_release_service.get_db")
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
@patch("app.services.ops_job_service.list_managed_nodes")
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
def test_build_rollout_target_operational_readiness_adds_release_context_and_focus_ref(
self,
mock_get_cluster_snapshot,
mock_list_managed_nodes,
mock_get_managed_node_onboarding,
mock_ensure_ops_agent_schema,
mock_get_db,
) -> None:
mock_ensure_ops_agent_schema.return_value = None
mock_get_managed_node_onboarding.return_value = {
"onboarding_stage": {"code": "ready", "label": "已接管"},
"summary": "当前节点无需额外恢复动作。",
"recovery_decision": {
"action": "noop",
"label": "当前无需额外恢复动作",
"summary": "当前节点暂无需要执行的接管恢复动作。",
"command_hint": "",
"window": "none",
},
}
mock_get_db.return_value = _EmptyConnection()
mock_get_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
"metadata": {"current_release_version": "2026.04.11+1921318"},
}
]
}
mock_list_managed_nodes.return_value = [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"ssh_host": "121.204.244.248",
"ssh_user": "root",
"metadata": {"current_release_version": "2026.04.11+1921318"},
"last_seen_at": "2026-04-18 08:00:00",
}
]
readiness = build_rollout_target_operational_readiness(
[{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker"}],
execution_mode="remote-agent",
desired_release={"id": 18, "release_version": "2026.04.18+bd6bcb2", "channel": "stable"},
)
row = readiness["rows"][0]
self.assertEqual("2026.04.11+1921318", row["current_release_version"])
self.assertEqual("2026.04.18+bd6bcb2", row["desired_release_version"])
self.assertEqual("release_hub", row["focus_ref"]["kind"])
self.assertEqual("default_rollout_gate", row["focus_ref"]["section"])
self.assertEqual("mainland-worker-01", row["focus_ref"]["node_code"])
self.assertEqual("noop", row["recovery_action"])
@patch("app.services.ops_release_service.get_db")
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
@patch("app.services.ops_job_service.list_managed_nodes")
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
def test_build_rollout_target_operational_readiness_prefers_recovery_decision_for_blocked_nodes(
self,
mock_get_cluster_snapshot,
mock_list_managed_nodes,
mock_get_managed_node_onboarding,
mock_ensure_ops_agent_schema,
mock_get_db,
) -> None:
mock_ensure_ops_agent_schema.return_value = None
mock_get_db.return_value = _EmptyConnection()
mock_get_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
}
]
}
mock_list_managed_nodes.return_value = [
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"ssh_host": "121.204.244.249",
"ssh_user": "root",
"metadata": {},
"last_seen_at": "",
}
]
mock_get_managed_node_onboarding.return_value = {
"onboarding_stage": {"code": "pending_bootstrap", "label": "待接入"},
"summary": "已签发有效 Agent Token但节点尚未完成 register/heartbeat。",
"recovery_decision": {
"action": "bootstrap_run",
"label": "签发接入工单",
"summary": "当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。",
"command_hint": "bash drive_ops_center.sh node-bootstrap-run http://127.0.0.1:8100 mainland-worker-02 cli",
"window": "bootstrap",
},
}
readiness = build_rollout_target_operational_readiness(
[{"node_code": "mainland-worker-02", "region": "mainland", "role": "worker"}],
execution_mode="remote-agent",
)
row = readiness["rows"][0]
self.assertEqual("bootstrap_run", row["recovery_action"])
self.assertEqual("bootstrap_run", row["recommended_action_code"])
self.assertEqual(1, readiness["summary"]["onboarding_bootstrap_pending_nodes"])
self.assertTrue(
any("onboarding.bootstrap" in item for item in readiness["recommendations"])
)
@patch("app.services.ops_release_service.get_db")
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
@patch("app.services.ops_job_service.list_managed_nodes")
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
def test_build_rollout_target_operational_readiness_blocks_nodes_with_dead_letter_queue(
self,
mock_get_cluster_snapshot,
mock_list_managed_nodes,
mock_get_managed_node_onboarding,
mock_ensure_ops_agent_schema,
mock_get_db,
) -> None:
mock_ensure_ops_agent_schema.return_value = None
mock_get_db.return_value = _EmptyConnection()
mock_get_managed_node_onboarding.return_value = {
"onboarding_stage": {"code": "ready", "label": "已接管"},
"summary": "当前节点无需额外恢复动作。",
"recovery_decision": {
"action": "noop",
"label": "当前无需额外恢复动作",
"summary": "当前节点暂无需要执行的接管恢复动作。",
"command_hint": "",
"window": "none",
},
}
mock_get_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-03",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
}
]
}
mock_list_managed_nodes.return_value = [
{
"node_code": "mainland-worker-03",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"ssh_host": "121.204.244.250",
"ssh_user": "root",
"metadata": {
"delivery_queue": {
"state": "dead_letter",
"label": "死信 1",
"reason": "当前存在 1 条死信记录",
"pending_count": 0,
"dead_letter_count": 1,
"oldest_dead_letter_at": "2026-04-18 09:01:00",
}
},
"last_seen_at": "2026-04-18 09:05:00",
}
]
readiness = build_rollout_target_operational_readiness(
[{"node_code": "mainland-worker-03", "region": "mainland", "role": "worker"}],
execution_mode="remote-agent",
)
row = readiness["rows"][0]
self.assertEqual("dead_letter", row["delivery_queue_state"])
self.assertEqual(1, row["delivery_queue_dead_letter_count"])
self.assertTrue(
any("死信" in item for item in row["blocking_reasons"])
)
self.assertEqual(1, readiness["summary"]["queue_dead_letter_nodes"])
self.assertTrue(
any("死信回执" in item for item in readiness["blocking_reasons"])
)
self.assertTrue(
any("死信队列" in item for item in readiness["recommendations"])
)
@patch("app.services.ops_release_service.build_rollout_target_operational_readiness")
def test_build_release_rollout_gate_uses_execution_ready_count_for_ssh(self, mock_build_readiness) -> None:
mock_build_readiness.return_value = {
"execution_mode": "ssh",
"summary": {
"nodes_total": 2,
"execution_ready_nodes": 1,
"inspection_healthy_nodes": 2,
},
"rows": [],
"blocking_reasons": ["有 1 台目标节点尚未配置完整 SSH 入口,当前不适合直接 SSH 发布。"],
"warning_reasons": [],
"recommendations": [],
}
gate = build_release_rollout_gate(
{
"id": 11,
"release_version": "2026.04.18-rc1",
"status": "ready",
"artifact_url": "https://example.com/domaincheck.tar.gz",
},
[{"node_code": "mainland-worker-01"}, {"node_code": "mainland-worker-02"}],
execution_mode="ssh",
)
self.assertEqual("ssh", gate["execution_mode"])
self.assertEqual("SSH", gate["execution_mode_label"])
self.assertEqual("blocked", gate["status"])
self.assertIn("1/2 台 SSH 就绪", gate["summary"])
@patch("app.services.ops_release_service.build_rollout_target_operational_readiness")
def test_build_release_rollout_gate_uses_human_label_for_remote_agent(self, mock_build_readiness) -> None:
mock_build_readiness.return_value = {
"execution_mode": "remote-agent",
"summary": {
"nodes_total": 2,
"execution_ready_nodes": 1,
"inspection_healthy_nodes": 2,
},
"rows": [],
"blocking_reasons": ["有 1 台目标节点 Agent 未在线,当前不适合直接 remote-agent 发布。"],
"warning_reasons": [],
"recommendations": [],
}
gate = build_release_rollout_gate(
{
"id": 12,
"release_version": "2026.04.18-rc1",
"status": "ready",
"artifact_url": "https://example.com/domaincheck.tar.gz",
},
[{"node_code": "mainland-worker-01"}, {"node_code": "mainland-worker-02"}],
execution_mode="remote-agent",
)
self.assertEqual("remote-agent", gate["execution_mode"])
self.assertEqual("远端 Agent", gate["execution_mode_label"])
self.assertIn("1/2 台 远端 Agent 就绪", gate["summary"])
@patch("app.services.ops_release_service.build_release_rollout_gate")
def test_build_release_execution_mode_recommendation_prefers_ssh_when_gate_is_better(self, mock_build_gate) -> None:
def _fake_gate(release, target_nodes, *, execution_mode="remote-agent"):
if execution_mode == "ssh":
return {
"status": "ready",
"status_label": "可发 Rollout",
"summary": "SSH 路径已收口",
"execution_mode": "ssh",
"execution_mode_label": "SSH",
"blocking_reasons": [],
"warning_reasons": [],
"operational_readiness": {
"summary": {"nodes_total": 2, "execution_ready_nodes": 2},
"rows": [],
},
}
return {
"status": "blocked",
"status_label": "存在阻断",
"summary": "remote-agent 尚未收口",
"execution_mode": "remote-agent",
"execution_mode_label": "远端 Agent",
"blocking_reasons": ["Agent 未就绪"],
"warning_reasons": [],
"operational_readiness": {
"summary": {"nodes_total": 2, "execution_ready_nodes": 1},
"rows": [],
},
}
mock_build_gate.side_effect = _fake_gate
recommendation = build_release_execution_mode_recommendation(
{
"id": 18,
"release_version": "2026.04.18-rc2",
"status": "ready",
"artifact_url": "https://example.com/domaincheck.tar.gz",
},
[{"node_code": "mainland-worker-01"}, {"node_code": "mainland-worker-02"}],
)
self.assertEqual("ssh", recommendation["recommended_mode"])
self.assertEqual("SSH", recommendation["recommended_mode_label"])
self.assertEqual("ready", recommendation["recommended_gate"]["status"])
self.assertIn("SSH", recommendation["reason"])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,212 @@
import unittest
from app.services.ops_release_service import (
_build_release_launchpad_status,
_compact_latest_smart_rollout_preview,
)
class OpsReleaseServiceLaunchpadTests(unittest.TestCase):
def test_launchpad_status_blocks_when_package_missing(self) -> None:
status = _build_release_launchpad_status(
package={"available": False},
latest_package_preview={},
worker_preview={},
control_preview={},
)
self.assertEqual("blocked", status["status"])
self.assertEqual("release_package", status["recommended_action_code"])
self.assertEqual(status["summary"], status["summary_text"])
self.assertEqual("release_hub", status["focus_ref"]["kind"])
self.assertEqual("release_launchpad", status["focus_ref"]["section"])
def test_launchpad_status_blocks_when_rollout_preview_has_blockers(self) -> None:
status = _build_release_launchpad_status(
package={"available": True},
latest_package_preview={},
worker_preview={"policy_preview": {"blocked": True}},
control_preview={"policy_preview": {"blocked": False}},
)
self.assertEqual("blocked", status["status"])
self.assertEqual("fix_rollout_blockers", status["recommended_action_code"])
self.assertEqual(status["summary"], status["summary_text"])
def test_launchpad_status_blocks_when_final_signoff_missing(self) -> None:
status = _build_release_launchpad_status(
package={
"available": True,
"final_release_ok": False,
"final_release_report_available": True,
"final_release_report_ok": True,
"final_release_report_matches_latest": False,
"final_release_report_stale_reason": "final_release_report does not match latest_release: package_name",
"final_release_gate_decision": "ready",
"final_release_gate_blocked_reasons": [],
},
latest_package_preview={},
worker_preview={"smart_rollout": {"available": True}, "policy_preview": {}},
control_preview={"smart_rollout": {"available": True}, "policy_preview": {}},
)
self.assertEqual("blocked", status["status"])
self.assertEqual("release_prepare", status["recommended_action_code"])
self.assertIn("final_release_report does not match latest_release: package_name", status["summary"])
self.assertIn("release-prepare", status["recommended_command"])
self.assertEqual(status["summary"], status["summary_text"])
def test_launchpad_status_prefers_runtime_refresh_when_runtime_schema_is_stale(self) -> None:
status = _build_release_launchpad_status(
package={"available": True, "final_release_ok": True},
latest_package_preview={},
worker_preview={"smart_rollout": {"available": True}, "policy_preview": {}},
control_preview={"smart_rollout": {"available": True}, "policy_preview": {}},
runtime_build_info={
"runtime_schema_stale": True,
"route_surface_complete": True,
"route_surface_missing_keys": ["ops_node_handover_bootstrap_plan"],
},
)
self.assertEqual("attention", status["status"])
self.assertEqual("api-restart", status["recommended_action_code"])
self.assertEqual("待刷新运行时", status["status_label"])
self.assertTrue(status["runtime_refresh_required"])
self.assertTrue(status["runtime_schema_stale"])
self.assertIn("runtime-refresh-recover", status["recommended_command"])
self.assertEqual(status["summary"], status["summary_text"])
def test_launchpad_status_marks_attention_when_confirmation_required(self) -> None:
status = _build_release_launchpad_status(
package={"available": True},
latest_package_preview={},
worker_preview={"requires_confirmation": True, "policy_preview": {}},
control_preview={"policy_preview": {}},
)
self.assertEqual("attention", status["status"])
self.assertEqual("review_smart_rollout_preview", status["recommended_action_code"])
self.assertEqual("release_launchpad", status["focus_ref"]["section"])
def test_launchpad_status_ready_when_worker_rollout_available(self) -> None:
status = _build_release_launchpad_status(
package={"available": True},
latest_package_preview={},
worker_preview={"smart_rollout": {"available": True, "execution_mode": "ssh", "execution_mode_label": "SSH"}, "policy_preview": {}},
control_preview={"smart_rollout": {"available": False}, "policy_preview": {}},
)
self.assertEqual("ready", status["status"])
self.assertEqual("publish_latest_worker", status["recommended_action_code"])
self.assertEqual("ssh", status["recommended_execution_mode"])
self.assertEqual("SSH", status["recommended_execution_mode_label"])
self.assertEqual(status["summary"], status["summary_text"])
def test_launchpad_status_requires_managed_nodes_when_no_targets_available(self) -> None:
status = _build_release_launchpad_status(
package={"available": True},
latest_package_preview={},
worker_preview={"smart_rollout": {"available": False}, "policy_preview": {}},
control_preview={"smart_rollout": {"available": False}, "policy_preview": {}},
)
self.assertEqual("attention", status["status"])
self.assertEqual("fix_managed_nodes", status["recommended_action_code"])
self.assertEqual("release_hub", status["focus_ref"]["kind"])
def test_launchpad_status_prefers_bootstrap_recovery_when_gap_row_has_recovery_action(self) -> None:
status = _build_release_launchpad_status(
package={"available": True},
latest_package_preview={},
worker_preview={
"smart_rollout": {"available": False},
"policy_preview": {
"blocked": True,
"release_gate": {
"operational_readiness": {
"rows": [
{
"node_code": "mainland-worker-02",
"recovery_action": "bootstrap_run",
"recovery_label": "签发接入工单",
"recovery_summary": "当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。",
}
]
}
},
},
},
control_preview={"smart_rollout": {"available": False}, "policy_preview": {}},
)
self.assertEqual("blocked", status["status"])
self.assertEqual("bootstrap_run", status["recommended_action_code"])
self.assertEqual("mainland-worker-02", status["recommended_target_node_code"])
self.assertEqual("签发接入工单", status["recommended_recovery_label"])
self.assertEqual("当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。", status["recommended_recovery_summary"])
self.assertEqual(1, status["onboarding_bootstrap_pending_nodes"])
self.assertEqual(0, status["onboarding_acceptance_ready_nodes"])
self.assertEqual(1, status["onboarding_gap_nodes_total"])
self.assertIn("node-bootstrap-run", status["recommended_command"])
def test_compact_latest_smart_rollout_preview_adds_rollout_preview_alias(self) -> None:
preview = _compact_latest_smart_rollout_preview(
"智能 Rollout 预检通过",
{
"release": {
"id": 18,
"release_version": "2026.04.18+bd6bcb2",
"channel": "stable",
"status": "ready",
"artifact_url": "https://example.com/domaincheck.tar.gz",
},
"package": {
"available": True,
"package_name": "domaincheck-2026.04.18+bd6bcb2",
"package_type": "tar.gz",
"sha256": "sha256:test",
"metadata": {
"manifest": {
"included_services": ["domaincheck-api", "domaincheck-worker"],
}
},
},
"smart_rollout": {
"available": True,
"execution_mode": "remote-agent",
"execution_mode_label": "远端 Agent",
"summary": "已按 Worker 预选 2 台节点。",
"target_selector": {"node_codes": ["mainland-worker-01", "mainland-worker-02"]},
"target_nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"delivery_queue_state": "healthy",
},
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"delivery_queue_state": "retrying",
},
],
},
"preview": {
"target_summary": {"nodes_total": 2},
"batch_plan": {"batches_total": 2, "first_batch_size": 1},
"release_gate": {"status": "ready", "summary": "当前可推进。"},
},
},
)
self.assertEqual("remote-agent", preview["rollout_preview"]["execution_mode"])
self.assertEqual(2, preview["rollout_preview"]["target_nodes_total"])
self.assertEqual(2, preview["rollout_preview"]["expected_batches_total"])
self.assertEqual(2, preview["rollout_preview"]["expected_jobs_total"])
self.assertEqual("release_hub", preview["rollout_preview"]["focus_ref"]["kind"])
self.assertEqual("healthy", preview["smart_rollout"]["target_nodes"][0]["delivery_queue_state"])
self.assertEqual("retrying", preview["smart_rollout"]["target_nodes"][1]["delivery_queue_state"])
self.assertEqual("rollout_preview", preview["rollout_preview"]["focus_ref"]["section"])
self.assertEqual(["domaincheck-api", "domaincheck-worker"], preview["package"]["manifest"]["included_services"])

View File

@@ -0,0 +1,915 @@
import json
import os
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from app.services.ops_release_service import (
build_smart_release_rollout_plan,
create_release_and_smart_rollout_from_latest_package,
create_release_from_latest_package,
create_smart_release_rollout,
get_latest_release_package_metadata,
get_release_package_archive_path,
get_release_package_sha256_path,
)
class OpsReleaseServicePackageMetadataTests(unittest.TestCase):
def test_get_latest_release_package_metadata_reads_latest_linux_package(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
release_root = Path(tmpdir)
package_name = "domaincheck_release_20260418_120000"
staging_path = release_root / package_name
archive_path = release_root / f"{package_name}.tar.gz"
sha256_path = release_root / f"{package_name}.sha256.txt"
staging_path.mkdir(parents=True, exist_ok=True)
archive_path.write_bytes(b"linux-package")
sha256_path.write_text("sha256=test-sha256", encoding="utf-8")
(staging_path / "release_manifest.json").write_text(
json.dumps(
{
"package_name": package_name,
"package_type": "tar.gz",
"included": {
"docs": ["08_domainCheck_交付说明.md"],
"scripts": ["package_domain_release.sh", "verify_domain_release.sh"],
},
},
ensure_ascii=False,
),
encoding="utf-8",
)
(release_root / "latest_release.json").write_text(
json.dumps(
{
"package_name": package_name,
"package_type": "tar.gz",
"generated_at": "2026-04-18T12:00:00",
"staging_path": str(staging_path),
"archive_path": str(archive_path),
"sha256_path": str(sha256_path),
"sha256": "test-sha256",
"smoke_test_ok": True,
},
ensure_ascii=False,
),
encoding="utf-8",
)
(release_root / "final_release_report.json").write_text(
json.dumps(
{
"ok": True,
"latest_release": {
"package_name": package_name,
"archive_path": str(archive_path),
"sha256": "test-sha256",
},
"release_gate": {
"decision": "ready",
"blocked_reasons": [],
"verify_ok": True,
"smoke_test_ok": True,
},
},
ensure_ascii=False,
),
encoding="utf-8",
)
with patch.dict(os.environ, {"DOMAINCHECK_RELEASE_ROOT": str(release_root)}):
data = get_latest_release_package_metadata()
self.assertTrue(data["available"])
self.assertEqual(package_name, data["package_name"])
self.assertEqual("tar.gz", data["package_type"])
self.assertTrue(data["archive_present"])
self.assertTrue(data["sha256_present"])
self.assertTrue(data["staging_present"])
self.assertEqual("test-sha256", data["sha256"])
self.assertEqual(f"/api/v1/ops/releases/packages/{package_name}/download", data["download_path"])
self.assertEqual(package_name, data["release_version_suggestion"])
self.assertIn("自检通过", data["notes_suggestion"])
self.assertTrue(data["final_release_ok"])
self.assertTrue(data["final_release_report_available"])
self.assertTrue(data["final_release_report_matches_latest"])
self.assertEqual("ready", data["final_release_gate_decision"])
self.assertEqual([], data["final_release_gate_blocked_reasons"])
self.assertEqual(package_name, data["metadata"]["manifest"]["package_name"])
def test_get_release_package_paths_support_metadata_fallback_for_zip(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
release_root = Path(tmpdir)
package_name = "domaincheck_release_windows_demo"
package_dir = release_root / "exports"
package_dir.mkdir(parents=True, exist_ok=True)
archive_path = package_dir / f"{package_name}.zip"
sha256_path = package_dir / f"{package_name}.sha256.txt"
archive_path.write_bytes(b"zip-package")
sha256_path.write_text("sha256=zip-sha256", encoding="utf-8")
(release_root / "latest_release.json").write_text(
json.dumps(
{
"package_name": package_name,
"package_type": "zip",
"generated_at": "2026-04-18T12:10:00",
"zip_path": str(archive_path),
"sha256_path": str(sha256_path),
"sha256": "zip-sha256",
"smoke_test_ok": False,
},
ensure_ascii=False,
),
encoding="utf-8",
)
with patch.dict(os.environ, {"DOMAINCHECK_RELEASE_ROOT": str(release_root)}):
resolved_archive = get_release_package_archive_path(package_name)
resolved_sha256 = get_release_package_sha256_path(package_name)
metadata = get_latest_release_package_metadata()
self.assertEqual(archive_path, resolved_archive)
self.assertEqual(sha256_path, resolved_sha256)
self.assertEqual("zip", metadata["package_type"])
self.assertEqual(archive_path.name, metadata["archive_filename"])
self.assertFalse(metadata["smoke_test_ok"])
def test_get_latest_release_package_metadata_returns_unavailable_when_missing(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
with patch.dict(os.environ, {"DOMAINCHECK_RELEASE_ROOT": tmpdir}):
data = get_latest_release_package_metadata()
self.assertFalse(data["available"])
self.assertIn("latest_release.json", data["reason"])
def test_get_latest_release_package_metadata_marks_stale_final_release_report(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
release_root = Path(tmpdir)
package_name = "domaincheck_release_20260418_130000"
archive_path = release_root / f"{package_name}.tar.gz"
sha256_path = release_root / f"{package_name}.sha256.txt"
archive_path.write_bytes(b"linux-package")
sha256_path.write_text("sha256=test-sha256", encoding="utf-8")
(release_root / "latest_release.json").write_text(
json.dumps(
{
"package_name": package_name,
"package_type": "tar.gz",
"archive_path": str(archive_path),
"sha256_path": str(sha256_path),
"sha256": "test-sha256",
"smoke_test_ok": True,
},
ensure_ascii=False,
),
encoding="utf-8",
)
(release_root / "final_release_report.json").write_text(
json.dumps(
{
"ok": True,
"latest_release": {
"package_name": "domaincheck_release_20260418_old",
"archive_path": str(release_root / "domaincheck_release_20260418_old.tar.gz"),
"sha256": "old-sha256",
},
"release_gate": {
"decision": "ready",
"blocked_reasons": [],
"verify_ok": True,
"smoke_test_ok": True,
},
},
ensure_ascii=False,
),
encoding="utf-8",
)
with patch.dict(os.environ, {"DOMAINCHECK_RELEASE_ROOT": str(release_root)}):
data = get_latest_release_package_metadata()
self.assertTrue(data["available"])
self.assertFalse(data["final_release_ok"])
self.assertTrue(data["final_release_report_available"])
self.assertTrue(data["final_release_report_ok"])
self.assertFalse(data["final_release_report_matches_latest"])
self.assertIn("package_name", data["final_release_report_stale_reason"])
@patch("app.services.ops_release_service.create_release")
@patch("app.services.ops_release_service._get_release_by_version")
@patch("app.services.ops_release_service.get_latest_release_package_metadata")
def test_create_release_from_latest_package_builds_payload(
self,
mock_get_latest_release_package_metadata,
mock_get_release_by_version,
mock_create_release,
) -> None:
mock_get_latest_release_package_metadata.return_value = {
"available": True,
"package_name": "domaincheck_release_20260418_120000",
"release_version_suggestion": "domaincheck_release_20260418_120000",
"download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_120000/download",
"sha256_download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_120000/sha256",
"sha256": "test-sha256",
"commit_sha": "bd6bcb2",
"notes_suggestion": "导入自本机发布包 domaincheck_release_20260418_120000.tar.gz自检通过",
"smoke_test_ok": True,
"final_release_ok": True,
"final_release_report_available": True,
"final_release_report_ok": True,
"final_release_report_matches_latest": True,
"final_release_gate_decision": "ready",
"final_release_gate_blocked_reasons": [],
"metadata": {
"package_name": "domaincheck_release_20260418_120000",
"package_type": "tar.gz",
},
}
mock_get_release_by_version.return_value = {}
mock_create_release.return_value = (
True,
"Release 已创建",
{
"release": {
"id": 7,
"release_version": "domaincheck_release_20260418_120000",
}
},
)
ok, message, data = create_release_from_latest_package(
{
"channel": "stable",
"status": "ready",
"created_by": "test-suite",
},
control_plane_base_url="https://ops.example.com/api/v1",
)
self.assertTrue(ok)
self.assertEqual("Release 已创建", message)
mock_create_release.assert_called_once()
create_payload = mock_create_release.call_args.args[0]
self.assertEqual("domaincheck_release_20260418_120000", create_payload["release_version"])
self.assertEqual("bd6bcb2", create_payload["commit_sha"])
self.assertEqual(
"https://ops.example.com/api/v1/ops/releases/packages/domaincheck_release_20260418_120000/download",
create_payload["artifact_url"],
)
self.assertEqual("test-sha256", create_payload["checksum"])
self.assertEqual("latest_release_package", create_payload["metadata"]["created_from"])
self.assertEqual(
"https://ops.example.com/api/v1/ops/releases/packages/domaincheck_release_20260418_120000/download",
create_payload["metadata"]["package_source"]["download_url"],
)
self.assertFalse(data["deduplicated"])
@patch("app.services.ops_release_service.create_release")
@patch("app.services.ops_release_service._get_release_by_version")
@patch("app.services.ops_release_service.get_latest_release_package_metadata")
def test_create_release_from_latest_package_is_idempotent(
self,
mock_get_latest_release_package_metadata,
mock_get_release_by_version,
mock_create_release,
) -> None:
mock_get_latest_release_package_metadata.return_value = {
"available": True,
"package_name": "domaincheck_release_20260418_120000",
"release_version_suggestion": "domaincheck_release_20260418_120000",
"smoke_test_ok": True,
"final_release_ok": True,
"final_release_report_available": True,
"final_release_report_ok": True,
"final_release_report_matches_latest": True,
"final_release_gate_decision": "ready",
"final_release_gate_blocked_reasons": [],
}
mock_get_release_by_version.return_value = {
"id": 9,
"release_version": "domaincheck_release_20260418_120000",
}
ok, message, data = create_release_from_latest_package({}, control_plane_base_url="https://ops.example.com")
self.assertTrue(ok)
self.assertEqual("Release 已存在,按幂等返回", message)
self.assertTrue(data["deduplicated"])
self.assertEqual(9, data["release"]["id"])
mock_create_release.assert_not_called()
@patch("app.services.ops_release_service.create_release")
@patch("app.services.ops_release_service._get_release_by_version")
@patch("app.services.ops_release_service.get_latest_release_package_metadata")
def test_create_release_from_latest_package_blocks_when_final_signoff_missing(
self,
mock_get_latest_release_package_metadata,
mock_get_release_by_version,
mock_create_release,
) -> None:
mock_get_latest_release_package_metadata.return_value = {
"available": True,
"package_name": "domaincheck_release_20260418_130000",
"release_version_suggestion": "domaincheck_release_20260418_130000",
"download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_130000/download",
"sha256_download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_130000/sha256",
"sha256": "test-sha256",
"commit_sha": "bd6bcb2",
"smoke_test_ok": False,
"final_release_ok": False,
"final_release_report_available": False,
"final_release_report_ok": None,
"final_release_report_matches_latest": None,
"final_release_gate_decision": "",
"final_release_gate_blocked_reasons": [],
"metadata": {
"package_name": "domaincheck_release_20260418_130000",
"package_type": "tar.gz",
},
}
mock_get_release_by_version.return_value = {}
ok, message, data = create_release_from_latest_package(
{
"channel": "stable",
"status": "ready",
"created_by": "test-suite",
},
control_plane_base_url="https://ops.example.com/api/v1",
)
self.assertFalse(ok)
self.assertIn("最终签收", message)
self.assertFalse(data["final_release_gate"]["ready"])
self.assertIn("smoke_test_failed_or_skipped", data["final_release_gate"]["blocked_reasons"])
self.assertIn("final_release_report_missing", data["final_release_gate"]["blocked_reasons"])
mock_create_release.assert_not_called()
@patch("app.services.ops_release_service.create_release")
@patch("app.services.ops_release_service._get_release_by_version")
@patch("app.services.ops_release_service.get_latest_release_package_metadata")
def test_create_release_from_latest_package_blocks_when_final_report_is_stale(
self,
mock_get_latest_release_package_metadata,
mock_get_release_by_version,
mock_create_release,
) -> None:
mock_get_latest_release_package_metadata.return_value = {
"available": True,
"package_name": "domaincheck_release_20260418_130001",
"release_version_suggestion": "domaincheck_release_20260418_130001",
"download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_130001/download",
"sha256_download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_130001/sha256",
"sha256": "test-sha256",
"commit_sha": "bd6bcb2",
"smoke_test_ok": True,
"final_release_ok": False,
"final_release_report_available": True,
"final_release_report_ok": True,
"final_release_report_matches_latest": False,
"final_release_report_stale_reason": "final_release_report does not match latest_release: package_name",
"final_release_gate_decision": "ready",
"final_release_gate_blocked_reasons": [],
"metadata": {
"package_name": "domaincheck_release_20260418_130001",
"package_type": "tar.gz",
},
}
mock_get_release_by_version.return_value = {
"id": 19,
"release_version": "domaincheck_release_20260418_130001",
}
ok, message, data = create_release_from_latest_package(
{"created_by": "test-suite"},
control_plane_base_url="https://ops.example.com/api/v1",
)
self.assertFalse(ok)
self.assertIn("最终签收", message)
self.assertFalse(data["final_release_gate"]["ready"])
self.assertEqual(
"final_release_report does not match latest_release: package_name",
data["final_release_gate"]["stale_reason"],
)
self.assertIn("final_release_report_not_matching_latest", data["final_release_gate"]["blocked_reasons"])
self.assertEqual(19, data["release"]["id"])
mock_create_release.assert_not_called()
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
def test_build_smart_release_rollout_plan_prefers_remote_ready_workers(
self,
mock_list_managed_nodes_with_agent_state,
) -> None:
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
},
{
"node_code": "mainland-controller-01",
"region": "mainland",
"role": "control",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"cluster_status": "online",
"cluster_current_load": 1,
"cluster_is_effective_worker": True,
"cluster_detect_participating": True,
},
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": False,
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
},
]
}
worker_plan = build_smart_release_rollout_plan("worker")
control_plan = build_smart_release_rollout_plan("control")
self.assertTrue(worker_plan["available"])
self.assertEqual("worker", worker_plan["mode"])
self.assertEqual(["mainland-worker-01"], worker_plan["target_selector"]["node_codes"])
self.assertEqual(["domaincheck-worker"], worker_plan["policy"]["deploy_payload"]["restart_services"])
self.assertTrue(control_plan["available"])
self.assertEqual("control", control_plan["mode"])
self.assertEqual(["mainland-controller-01"], control_plan["target_selector"]["node_codes"])
self.assertIn("domaincheck-api", control_plan["policy"]["deploy_payload"]["restart_services"])
self.assertEqual("remote-agent", worker_plan["policy"]["execution_mode"])
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
def test_build_smart_release_rollout_plan_falls_back_to_ssh_ready_workers(
self,
mock_list_managed_nodes_with_agent_state,
) -> None:
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": False,
"has_ssh_access": True,
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
},
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": False,
"has_ssh_access": True,
"cluster_status": "busy",
"cluster_current_load": 1,
"cluster_is_effective_worker": True,
"cluster_detect_participating": True,
},
]
}
worker_plan = build_smart_release_rollout_plan("worker")
self.assertTrue(worker_plan["available"])
self.assertEqual("worker", worker_plan["mode"])
self.assertEqual("ssh", worker_plan["execution_mode"])
self.assertEqual("SSH", worker_plan["execution_mode_label"])
self.assertEqual(
["mainland-worker-01", "mainland-worker-02"],
worker_plan["target_selector"]["node_codes"],
)
self.assertEqual("ssh", worker_plan["policy"]["execution_mode"])
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
def test_build_smart_release_rollout_plan_excludes_dead_letter_nodes_from_ready_candidates(
self,
mock_list_managed_nodes_with_agent_state,
) -> None:
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"has_ssh_access": True,
"delivery_queue_state": "dead_letter",
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
},
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"has_ssh_access": True,
"delivery_queue_state": "healthy",
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
},
]
}
worker_plan = build_smart_release_rollout_plan("worker")
self.assertTrue(worker_plan["available"])
self.assertEqual(["mainland-worker-02"], worker_plan["target_selector"]["node_codes"])
self.assertEqual("healthy", worker_plan["target_nodes"][0]["delivery_queue_state"])
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
def test_build_smart_release_rollout_plan_returns_unavailable_when_only_dead_letter_nodes_exist(
self,
mock_list_managed_nodes_with_agent_state,
) -> None:
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"has_ssh_access": True,
"delivery_queue_state": "dead_letter",
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
}
]
}
worker_plan = build_smart_release_rollout_plan("worker")
self.assertFalse(worker_plan["available"])
self.assertIn("死信回执", worker_plan["reason"])
self.assertEqual("dead_letter", worker_plan["target_nodes"][0]["delivery_queue_state"])
@patch("app.services.ops_release_service.create_release_rollout")
@patch("app.services.ops_policy_service.preview_ops_job_policy")
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
@patch("app.services.ops_release_service.create_release_from_latest_package")
def test_create_release_and_smart_rollout_stops_when_latest_package_gate_blocks(
self,
mock_create_release_from_latest_package,
mock_list_managed_nodes_with_agent_state,
mock_preview_ops_job_policy,
mock_create_release_rollout,
) -> None:
mock_create_release_from_latest_package.return_value = (
False,
"最新发布包尚未完成最终签收,暂不允许创建 Release 或发起基于最新包的智能 Rollout。",
{
"package": {
"available": True,
"package_name": "domaincheck_release_20260418_blocked",
"final_release_ok": False,
},
"final_release_gate": {
"ready": False,
"status": "blocked",
"blocked_reasons": ["final_release_report_missing"],
},
},
)
ok, message, data = create_release_and_smart_rollout_from_latest_package(
{"mode": "worker"},
control_plane_base_url="https://ops.example.com",
)
self.assertFalse(ok)
self.assertIn("最终签收", message)
self.assertFalse(data["final_release_gate"]["ready"])
mock_list_managed_nodes_with_agent_state.assert_not_called()
mock_preview_ops_job_policy.assert_not_called()
mock_create_release_rollout.assert_not_called()
@patch("app.services.ops_release_service.create_release_rollout")
@patch("app.services.ops_policy_service.preview_ops_job_policy")
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
@patch("app.services.ops_release_service.create_release_from_latest_package")
def test_create_release_and_smart_rollout_requires_confirmation_before_rollout(
self,
mock_create_release_from_latest_package,
mock_list_managed_nodes_with_agent_state,
mock_preview_ops_job_policy,
mock_create_release_rollout,
) -> None:
mock_create_release_from_latest_package.return_value = (
True,
"Release 已创建",
{
"release": {"id": 21, "release_version": "domaincheck_release_20260418_120000"},
"deduplicated": False,
},
)
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-controller-01",
"region": "mainland",
"role": "control",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
}
]
}
mock_preview_ops_job_policy.return_value = {
"blocked": False,
"approval_required": True,
"warnings": ["本次 rollout 覆盖 control 节点,建议审批。"],
"release_gate": {"status_label": "待补巡检", "summary": "建议先确认控制面健康。"},
}
ok, message, data = create_release_and_smart_rollout_from_latest_package(
{"mode": "control"},
control_plane_base_url="https://ops.example.com",
)
self.assertTrue(ok)
self.assertEqual("Release 已创建,智能 Rollout 需要确认", message)
self.assertTrue(data["requires_confirmation"])
self.assertFalse(data["rollout_created"])
self.assertEqual(["mainland-controller-01"], data["smart_rollout"]["target_selector"]["node_codes"])
mock_create_release_rollout.assert_not_called()
@patch("app.services.ops_release_service.create_release_rollout")
@patch("app.services.ops_policy_service.preview_ops_job_policy")
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
@patch("app.services.ops_release_service.create_release_from_latest_package")
def test_create_release_and_smart_rollout_creates_rollout_after_confirmation(
self,
mock_create_release_from_latest_package,
mock_list_managed_nodes_with_agent_state,
mock_preview_ops_job_policy,
mock_create_release_rollout,
) -> None:
mock_create_release_from_latest_package.return_value = (
True,
"Release 已创建",
{
"release": {"id": 22, "release_version": "domaincheck_release_20260418_120000"},
"deduplicated": True,
},
)
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
}
]
}
mock_preview_ops_job_policy.return_value = {
"blocked": False,
"approval_required": False,
"warnings": [],
"release_gate": {"status_label": "可发 Rollout", "summary": "当前可推进。"},
}
mock_create_release_rollout.return_value = (
True,
"Release rollout 已创建",
{
"rollout": {"id": 5, "release_id": 22, "rollout_code": "rollout-20260418-demo"},
"jobs": [],
},
)
ok, message, data = create_release_and_smart_rollout_from_latest_package(
{"mode": "worker", "confirm_risky": True},
control_plane_base_url="https://ops.example.com",
)
self.assertTrue(ok)
self.assertEqual("Release rollout 已创建", message)
self.assertTrue(data["rollout_created"])
self.assertEqual(5, data["rollout"]["id"])
mock_create_release_rollout.assert_called_once()
rollout_payload = mock_create_release_rollout.call_args.args[1]
self.assertEqual(["mainland-worker-01"], rollout_payload["target_selector"]["node_codes"])
self.assertEqual("remote-agent", rollout_payload["policy"]["execution_mode"])
@patch("app.services.ops_release_service.create_release_rollout")
@patch("app.services.ops_policy_service.preview_ops_job_policy")
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
@patch("app.services.ops_release_service.create_release_from_latest_package")
def test_create_release_and_smart_rollout_uses_ssh_when_only_ssh_path_is_ready(
self,
mock_create_release_from_latest_package,
mock_list_managed_nodes_with_agent_state,
mock_preview_ops_job_policy,
mock_create_release_rollout,
) -> None:
mock_create_release_from_latest_package.return_value = (
True,
"Release 已创建",
{
"release": {"id": 23, "release_version": "domaincheck_release_20260418_ssh"},
"deduplicated": True,
},
)
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": False,
"has_ssh_access": True,
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
}
]
}
mock_preview_ops_job_policy.return_value = {
"blocked": False,
"approval_required": False,
"warnings": [],
"release_gate": {"status_label": "可发 Rollout", "summary": "当前可推进。"},
}
mock_create_release_rollout.return_value = (
True,
"Release rollout 已创建",
{
"rollout": {"id": 6, "release_id": 23, "rollout_code": "rollout-20260418-ssh"},
"jobs": [],
},
)
ok, message, data = create_release_and_smart_rollout_from_latest_package(
{"mode": "worker", "confirm_risky": True},
control_plane_base_url="https://ops.example.com",
)
self.assertTrue(ok)
self.assertEqual("Release rollout 已创建", message)
self.assertTrue(data["rollout_created"])
rollout_payload = mock_create_release_rollout.call_args.args[1]
self.assertEqual("ssh", rollout_payload["policy"]["execution_mode"])
@patch("app.services.ops_release_service.get_release")
@patch("app.services.ops_release_service.create_release_rollout")
@patch("app.services.ops_policy_service.preview_ops_job_policy")
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
def test_create_smart_release_rollout_for_existing_release_requires_confirmation(
self,
mock_list_managed_nodes_with_agent_state,
mock_preview_ops_job_policy,
mock_create_release_rollout,
mock_get_release,
) -> None:
mock_get_release.return_value = {
"id": 41,
"release_version": "domaincheck_release_existing",
"channel": "stable",
}
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-controller-01",
"region": "mainland",
"role": "control",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
}
]
}
mock_preview_ops_job_policy.return_value = {
"blocked": False,
"approval_required": True,
"warnings": ["控制面节点建议审批。"],
"release_gate": {"status_label": "待审批", "summary": "需要审批。"},
}
ok, message, data = create_smart_release_rollout(41, {"mode": "control"})
self.assertTrue(ok)
self.assertEqual("智能 Rollout 需要确认", message)
self.assertFalse(data["release_created"])
self.assertTrue(data["requires_confirmation"])
self.assertFalse(data["rollout_created"])
self.assertEqual(["mainland-controller-01"], data["smart_rollout"]["target_selector"]["node_codes"])
mock_create_release_rollout.assert_not_called()
@patch("app.services.ops_release_service.get_release")
@patch("app.services.ops_release_service.create_release_rollout")
@patch("app.services.ops_policy_service.preview_ops_job_policy")
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
def test_create_smart_release_rollout_for_existing_release_creates_rollout(
self,
mock_list_managed_nodes_with_agent_state,
mock_preview_ops_job_policy,
mock_create_release_rollout,
mock_get_release,
) -> None:
mock_get_release.return_value = {
"id": 42,
"release_version": "domaincheck_release_existing",
"channel": "stable",
}
mock_list_managed_nodes_with_agent_state.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"is_managed": True,
"is_agent_online": True,
"cluster_status": "online",
"cluster_current_load": 0,
"cluster_is_effective_worker": True,
"cluster_detect_participating": False,
}
]
}
mock_preview_ops_job_policy.return_value = {
"blocked": False,
"approval_required": False,
"warnings": [],
"release_gate": {"status_label": "可发 Rollout", "summary": "当前可推进。"},
}
mock_create_release_rollout.return_value = (
True,
"Release rollout 已创建",
{
"rollout": {"id": 8, "release_id": 42, "rollout_code": "rollout-20260418-existing"},
"jobs": [],
},
)
ok, message, data = create_smart_release_rollout(42, {"mode": "worker", "confirm_risky": True})
self.assertTrue(ok)
self.assertEqual("Release rollout 已创建", message)
self.assertFalse(data["release_created"])
self.assertTrue(data["rollout_created"])
self.assertEqual(8, data["rollout"]["id"])
mock_create_release_rollout.assert_called_once()
rollout_payload = mock_create_release_rollout.call_args.args[1]
self.assertEqual(["mainland-worker-01"], rollout_payload["target_selector"]["node_codes"])
self.assertEqual("remote-agent", rollout_payload["policy"]["execution_mode"])
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,118 @@
import subprocess
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from app.services.ops_runtime_executor_service import execute_local_support_action, execute_ssh_action
class OpsRuntimeExecutorServiceTests(unittest.TestCase):
@patch("app.services.ops_runtime_executor_service.subprocess.run")
def test_execute_ssh_action_parses_structured_json_result(self, mock_run) -> None:
mock_run.return_value = subprocess.CompletedProcess(
args=["ssh"],
returncode=0,
stdout='login banner\n{"ok": true, "message": "health snapshot collected", "result": {"checks": {"api": {"state": "active"}}}}\n',
stderr="",
)
ok, message, data = execute_ssh_action(
{
"node_code": "mainland-worker-01",
"ssh_host": "121.204.244.248",
"ssh_port": 22,
"ssh_user": "root",
},
"health.snapshot",
{},
)
self.assertTrue(ok)
self.assertEqual("health snapshot collected", message)
self.assertEqual("active", data["checks"]["api"]["state"])
self.assertEqual("ssh", data["transport"]["executor"])
self.assertEqual("121.204.244.248", data["transport"]["ssh_host"])
issued_command = mock_run.call_args.args[0]
self.assertEqual("ssh", issued_command[0])
self.assertIn("root@121.204.244.248", issued_command)
@patch("app.services.ops_runtime_executor_service.subprocess.run")
def test_execute_local_support_action_collects_logs_with_payload(self, mock_run) -> None:
mock_run.return_value = subprocess.CompletedProcess(
args=["journalctl"],
returncode=0,
stdout="log line 1\nlog line 2",
stderr="",
)
ok, message, data = execute_local_support_action(
"logs.collect",
{
"service_name": "domaincheck-worker",
"lines": 180,
},
)
self.assertTrue(ok)
self.assertEqual("log line 1\nlog line 2", message)
self.assertEqual("domaincheck-worker", data["service_name"])
self.assertEqual(180, data["lines"])
issued_command = mock_run.call_args.args[0]
self.assertEqual(
["journalctl", "-u", "domaincheck-worker", "-n", "180", "--no-pager"],
issued_command,
)
@patch("app.services.ops_runtime_executor_service.subprocess.run")
def test_execute_local_support_action_falls_back_to_worker_log_file(self, mock_run) -> None:
mock_run.return_value = subprocess.CompletedProcess(
args=["journalctl"],
returncode=1,
stdout="",
stderr="No journal files were opened due to insufficient permissions.",
)
with tempfile.TemporaryDirectory() as temp_dir:
log_path = Path(temp_dir) / "detect_worker.log"
log_path.write_text("fallback-1\nfallback-2\n", encoding="utf-8")
with patch("app.services.ops_action_executor_core.settings.domain_root", temp_dir):
ok, message, data = execute_local_support_action(
"logs.collect",
{
"service_name": "domaincheck-worker",
"lines": 180,
},
)
self.assertTrue(ok)
self.assertEqual("fallback-1\nfallback-2", message)
self.assertEqual("file_fallback", data["collection_source"])
self.assertTrue(data["fallback_used"])
@patch("app.services.ops_runtime_executor_service.subprocess.run")
def test_execute_local_support_action_restarts_service(self, mock_run) -> None:
mock_run.return_value = subprocess.CompletedProcess(
args=["systemctl"],
returncode=0,
stdout="",
stderr="",
)
ok, message, data = execute_local_support_action(
"service.restart",
{
"service_name": "domaincheck-worker",
},
)
self.assertTrue(ok)
self.assertEqual("domaincheck-worker restart completed", message)
self.assertEqual("domaincheck-worker", data["service_name"])
self.assertEqual("restart", data["systemctl_action"])
issued_command = mock_run.call_args.args[0]
self.assertEqual(["systemctl", "restart", "domaincheck-worker"], issued_command)
if __name__ == "__main__":
unittest.main()

Some files were not shown because too many files have changed in this diff Show More