feat: add ops center and node onboarding flow
This commit is contained in:
98
HANDOFF_20260418_2321.md
Normal file
98
HANDOFF_20260418_2321.md
Normal file
@@ -0,0 +1,98 @@
|
||||
# HANDOFF 2026-04-18 23:21
|
||||
|
||||
## 当前最高优先级
|
||||
|
||||
大陆节点接管闭环最小路径
|
||||
|
||||
原因:
|
||||
|
||||
- 当前控制面能力已基本足够
|
||||
- 当前最大阻塞不是新功能,而是大陆两台还停在:
|
||||
- `pending_bootstrap`
|
||||
- `agent_pending`
|
||||
- `SSH 未配置`
|
||||
- 只要这条链路不闭合,项目就仍然停留在“主控骨架可用,但统一接管未落地”的状态
|
||||
|
||||
## 推荐模型与推理等级
|
||||
|
||||
推荐:
|
||||
|
||||
- `GPT-5.4`
|
||||
- `high`
|
||||
|
||||
原因:
|
||||
|
||||
- 当前阶段重心是优先级判断、闭环收口、执行顺序压缩
|
||||
- 需要稳定判断力,不适合降到偏快但更容易发散的配置
|
||||
- 当前阶段也暂时不需要切到 `xhigh`
|
||||
|
||||
## 下一轮如果继续,第一步先做什么
|
||||
|
||||
下一轮必须先做:
|
||||
|
||||
1. 录入大陆两台节点的 SSH 信息
|
||||
2. 复查 `nodes`
|
||||
3. 逐台跑 `node-handover`
|
||||
4. 逐台跑 `node-bootstrap-plan`
|
||||
|
||||
建议顺序:
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
|
||||
推荐第一组动作:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bind-ssh http://127.0.0.1:8100 mainland-controller-01 <ssh_host> <ssh_user> 22
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bind-ssh http://127.0.0.1:8100 mainland-worker-01 <ssh_host> <ssh_user> 22
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-handover http://127.0.0.1:8100 mainland-controller-01
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-controller-01
|
||||
```
|
||||
|
||||
## 当前不要做的事情
|
||||
|
||||
现在不要做:
|
||||
|
||||
- 扩写新的架构设计文档
|
||||
- 新增页面
|
||||
- 新增模块
|
||||
- 继续增强控制面专题能力
|
||||
- 扩大 Release Hub 范围
|
||||
- 扩大 runbook / playbook 范围
|
||||
- 做与大陆节点接管闭环无直接关系的功能开发
|
||||
|
||||
## 当前阶段边界
|
||||
|
||||
本次已完成:
|
||||
|
||||
- 阶段校准
|
||||
- 闭环优先级重排
|
||||
- 任务板初始化
|
||||
- 状态板初始化
|
||||
- handoff 生成
|
||||
|
||||
本次未进入:
|
||||
|
||||
- 任何新实现
|
||||
- 任何新页面或新模块
|
||||
- 任何新的控制面增强项
|
||||
|
||||
## 下一轮完成主批次后的预期
|
||||
|
||||
如果大陆两台顺利完成:
|
||||
|
||||
- SSH 纳管
|
||||
- bootstrap
|
||||
- acceptance
|
||||
- `remote_access_ready=3/3`
|
||||
|
||||
则项目总体进度预计进入:
|
||||
|
||||
- `60%~65%` 的稳定上线准备区间
|
||||
|
||||
那时下一阶段才应该切到:
|
||||
|
||||
- 发布前总检
|
||||
- 正式收口
|
||||
- 上线证据导出
|
||||
155
HANDOFF_20260418_2344.md
Normal file
155
HANDOFF_20260418_2344.md
Normal file
@@ -0,0 +1,155 @@
|
||||
# HANDOFF 2026-04-18 23:44
|
||||
|
||||
## 当前最高优先级
|
||||
|
||||
仍然是:大陆节点接管闭环最小路径
|
||||
|
||||
但当前阶段已经从“补 SSH 入口”切到:
|
||||
|
||||
- 修正大陆节点环境内容
|
||||
- 让 bootstrap 真正可以落地执行
|
||||
|
||||
## 今晚已完成的步骤
|
||||
|
||||
已完成:
|
||||
|
||||
1. 录入 `mainland-controller-01` SSH 信息
|
||||
2. 录入 `mainland-worker-01` SSH 信息
|
||||
3. 复查 `ops/nodes`
|
||||
4. 逐台执行 `node-handover`
|
||||
5. 逐台生成并复核 `node-bootstrap-plan`
|
||||
6. 复核:
|
||||
- `go-live-summary`
|
||||
- `ops/nodes`
|
||||
- `release-launchpad`
|
||||
7. 手工验证 controller SSH 登录可达
|
||||
|
||||
## 当前结果
|
||||
|
||||
当前指标:
|
||||
|
||||
- `ssh_ready = 2`
|
||||
- `remote_access_ready = 3`
|
||||
- 是否达到 `remote_access_ready = 3/3`:是
|
||||
|
||||
但 A1 仍未闭环,因为:
|
||||
|
||||
- `mainland-controller-01` 仍是 `pending_bootstrap`
|
||||
- `mainland-worker-01` 仍是 `pending_bootstrap`
|
||||
- 两台都还没有完成:
|
||||
- `bootstrap`
|
||||
- `register / heartbeat`
|
||||
- `acceptance`
|
||||
|
||||
## 两台节点推进到哪一步
|
||||
|
||||
### mainland-controller-01
|
||||
|
||||
已推进到:
|
||||
|
||||
- SSH 已录入
|
||||
- `node-handover` 已完成
|
||||
- `node-bootstrap-plan` 已生成
|
||||
- 手工 SSH 登录成功
|
||||
- 尝试执行 bootstrap 片段
|
||||
|
||||
准确卡点:
|
||||
|
||||
- 远端缺少 node-agent 所需文件
|
||||
- bootstrap 执行时报错:
|
||||
- `bootstrap_node_agent.sh not found under known layouts`
|
||||
- `Unit domaincheck-node-agent.service could not be found`
|
||||
|
||||
阻塞分类:
|
||||
|
||||
- 主阻塞:环境问题
|
||||
- 次阻塞:权限 / 外部条件问题
|
||||
- 当前控制面自动 SSH 执行器默认走密钥模式
|
||||
- 今晚验证可用的是密码 SSH
|
||||
|
||||
### mainland-worker-01
|
||||
|
||||
已推进到:
|
||||
|
||||
- SSH 已录入
|
||||
- `node-handover` 已完成
|
||||
- `node-bootstrap-plan` 已生成
|
||||
|
||||
准确卡点:
|
||||
|
||||
- 与 controller 同类
|
||||
- 远端缺少:
|
||||
- `domain-api/app/node_agent.py`
|
||||
- `domain-api/app/api/routes/ops_agent.py`
|
||||
- `domain-api/app/services/ops_agent_service.py`
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- `domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example`
|
||||
|
||||
阻塞分类:
|
||||
|
||||
- 主阻塞:环境问题
|
||||
|
||||
## 收口结论
|
||||
|
||||
今晚属于“有明显进展,但 A1 未闭环”的收口结果。
|
||||
|
||||
明确进展:
|
||||
|
||||
- 从 `remote_access_ready = 1/3` 推进到了 `3/3`
|
||||
- 两台大陆节点都已进入 `ssh_ready`
|
||||
- handover 与 bootstrap plan 链路都已打通
|
||||
|
||||
明确未完成:
|
||||
|
||||
- 两台大陆节点仍未完成 bootstrap
|
||||
- 因此还没有 register / heartbeat
|
||||
- 也没有进入 acceptance
|
||||
|
||||
## 下一轮如果继续,第一步先做什么
|
||||
|
||||
下一轮先做的不是新功能,而是修正大陆节点环境。
|
||||
|
||||
最小动作:
|
||||
|
||||
1. 确认大陆两台代码目录由 `www` 用户更新到包含 node-agent 文件的最新版本
|
||||
2. 确认以下文件在两台大陆节点都存在:
|
||||
- `domain-api/app/node_agent.py`
|
||||
- `domain-api/app/api/routes/ops_agent.py`
|
||||
- `domain-api/app/services/ops_agent_service.py`
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- `domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example`
|
||||
3. 然后重新执行:
|
||||
- `node-bootstrap-plan`
|
||||
- `bootstrap`
|
||||
- `acceptance`
|
||||
|
||||
## 推荐模型与推理等级
|
||||
|
||||
继续推荐:
|
||||
|
||||
- `GPT-5.4`
|
||||
- `high`
|
||||
|
||||
## 现在不要做的事情
|
||||
|
||||
不要做:
|
||||
|
||||
- 新页面
|
||||
- 新模块
|
||||
- 控制面增强项
|
||||
- 新架构文档扩写
|
||||
- Candidate B2
|
||||
- 为了解决今晚问题而扩展新的运维功能
|
||||
|
||||
## 明天需要你补充或操作的内容
|
||||
|
||||
你明天需要安排的最小动作是:
|
||||
|
||||
- 让大陆两台节点的仓库内容更新到当前控制面所需版本
|
||||
- 并且注意:
|
||||
- git / 代码更新应继续使用 `www`
|
||||
- 不要用 `root` 跑 git
|
||||
|
||||
## 当前一句话状态
|
||||
|
||||
今晚已经把大陆两台从“未纳管 SSH”推进到“SSH 可执行、bootstrap plan 可生成”,但两台都卡在相同的环境缺口上:远端仓库缺 node-agent 运行文件,因此 A1 还差最后的 bootstrap / register / acceptance 收口。
|
||||
135
IMPLEMENTATION_STATUS.md
Normal file
135
IMPLEMENTATION_STATUS.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# IMPLEMENTATION_STATUS
|
||||
|
||||
更新时间:2026-04-18 23:44 CST
|
||||
|
||||
## 当前真实状态
|
||||
|
||||
阶段判断:
|
||||
|
||||
- 方案与控制面能力建设:约 `78%`
|
||||
- 真正达到“可稳定上线、海外单脑统一接管大陆节点”:约 `58%`
|
||||
|
||||
当前已成立的事实:
|
||||
|
||||
- Ops Center 主控骨架已基本形成
|
||||
- Release Hub 已具备主流程判断能力
|
||||
- Node Agent 协议、日志回传、runbook / playbook、节点接管视图已具备基础可用性
|
||||
- `drive_ops_center.sh` 已补齐:
|
||||
- `nodes`
|
||||
- `node-bind-ssh`
|
||||
|
||||
当前最新节点状态:
|
||||
|
||||
- `managed_enabled = 3`
|
||||
- `agent_ready = 1`
|
||||
- `ssh_ready = 2`
|
||||
- `remote_access_ready = 3`
|
||||
- `pending_bootstrap = 2`
|
||||
|
||||
节点明细:
|
||||
|
||||
- `overseas-control-01`
|
||||
- `agent_ready`
|
||||
- `remote_access_ready = true`
|
||||
- `mainland-controller-01`
|
||||
- `pending_bootstrap`
|
||||
- `remote_access_state = ssh_ready`
|
||||
- `SSH 已录入`
|
||||
- `node-handover` 已完成
|
||||
- `node-bootstrap-plan` 已生成
|
||||
- 手工 SSH 登录已验证可达
|
||||
- `mainland-worker-01`
|
||||
- `pending_bootstrap`
|
||||
- `remote_access_state = ssh_ready`
|
||||
- `SSH 已录入`
|
||||
- `node-handover` 已完成
|
||||
- `node-bootstrap-plan` 已生成
|
||||
|
||||
## 代码闭环
|
||||
|
||||
已完成或基本具备:
|
||||
|
||||
- 统一查看节点接管状态
|
||||
- 统一生成与查看 handover / bootstrap 入口
|
||||
- 统一查看 go-live summary
|
||||
- 统一查看日志回传覆盖情况
|
||||
- 统一为托管节点录入 SSH 入口的 CLI 能力
|
||||
- controller 与 worker 的 bootstrap plan 均可生成
|
||||
|
||||
当前代码侧已验证的事实:
|
||||
|
||||
- 当前主控链路可以把大陆节点从 `agent_pending` 推到 `ssh_ready`
|
||||
- `node-handover` 与 `node-bootstrap-plan` 输出一致
|
||||
- `go-live-summary` 已看到 `remote_access_ready = 3`
|
||||
|
||||
结论:
|
||||
|
||||
- 当前代码侧已经可以支撑“接管闭环”的执行顺序
|
||||
- 当前主阻塞已不再是“还缺哪块控制面功能”
|
||||
- 但当前 SSH 自动执行链仍默认走密钥模式,和今晚实际提供的密码登录方式不一致
|
||||
|
||||
## 外部条件闭环
|
||||
|
||||
当前仍未闭合的条件:
|
||||
|
||||
- 大陆两台虽然已录入 SSH,但远端仓库未具备 node-agent 所需文件
|
||||
- 尚未实际触发大陆两台的 bootstrap 执行
|
||||
- 尚未建立两台大陆节点的 register / heartbeat 闭环
|
||||
- 尚未完成两台大陆节点的 acceptance 闭环
|
||||
|
||||
结论:
|
||||
|
||||
- 当前最关键的剩余问题不是新代码,而是大陆节点环境与当前控制面能力不匹配
|
||||
- 今晚已经不是“SSH 信息缺失”阶段,而是“大陆节点代码/部署内容缺失”阶段
|
||||
|
||||
## 明确依赖用户提供的信息
|
||||
|
||||
今晚已经提供并验证:
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `ssh_host=121.204.244.188`
|
||||
- `ssh_user=root`
|
||||
- 手工 SSH 登录成功
|
||||
- `mainland-worker-01`
|
||||
- `ssh_host=121.204.244.248`
|
||||
- `ssh_user=root`
|
||||
|
||||
当前仍需要你明天确认或操作的,不再是 SSH 信息,而是以下环境修正路径二选一:
|
||||
|
||||
1. 让大陆两台节点仓库更新到包含 node-agent 文件的最新版本
|
||||
2. 或把 node-agent 相关文件以正式发布包 / 正式部署方式补齐到大陆两台
|
||||
|
||||
在上述环境未修正前,以下链路不能真正闭合:
|
||||
|
||||
- `bootstrap execute`
|
||||
- `register / heartbeat`
|
||||
- `acceptance`
|
||||
- `pending_bootstrap -> 已接管`
|
||||
|
||||
## 当前优先级判断
|
||||
|
||||
最高优先级:
|
||||
|
||||
- 大陆节点接管闭环最小路径
|
||||
|
||||
当前不应继续优先推进:
|
||||
|
||||
- 控制面横向增强
|
||||
- 额外专题设计
|
||||
- 与接管闭环无直接关系的发布能力细化
|
||||
- 新的页面或模块
|
||||
|
||||
本轮新增阻塞分类结论:
|
||||
|
||||
- controller 阻塞:环境问题
|
||||
- worker 阻塞:环境问题
|
||||
- 自动 SSH 执行器限制:权限 / 外部条件问题
|
||||
- 现链路仅按 `auth_mode=key` 工作
|
||||
- 今晚实际验证可用的是密码 SSH
|
||||
|
||||
## 完成主批次后的预期
|
||||
|
||||
如果大陆两台环境补齐,并按最小闭环顺序完成:
|
||||
|
||||
- 项目整体可稳定上线进度预计提升到 `60%~65%`
|
||||
- 一旦 `remote_access_ready=3/3` 成立,后续就从“架构推进期”切换到“发布收口期”
|
||||
97
README.md
97
README.md
@@ -6,13 +6,24 @@
|
||||
- 新版后端服务 `domain-api/`
|
||||
- 新版 Web 管理后台 `domain-web/`
|
||||
- 全套交付与部署文档 `docs/`
|
||||
- Windows 本地启动、打包、验包脚本
|
||||
- Windows / Linux 启动、打包、验包脚本
|
||||
|
||||
如果你是第一次打开这个仓库,建议先看:
|
||||
|
||||
- [docs/12_domainCheck_导航索引.md](./docs/12_domainCheck_导航索引.md)
|
||||
- [docs/04_domainCheck_WebLinux改造总体方案.md](./docs/04_domainCheck_WebLinux改造总体方案.md)
|
||||
|
||||
如果你当前已经进入“海外单脑控制面 / 多机联调 / 准备上线”阶段,建议直接从下面 4 份开始:
|
||||
|
||||
1. [docs/25_domainCheck_海外单脑控制面上线收口总表.md](./docs/25_domainCheck_海外单脑控制面上线收口总表.md)
|
||||
2. [docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md](./docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md)
|
||||
3. [docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md](./docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md)
|
||||
4. [docs/schemas/ops_driver_contract.md](./docs/schemas/ops_driver_contract.md)
|
||||
|
||||
如果已经接近正式发布,建议再补一份:
|
||||
|
||||
- [docs/26_domainCheck_发布前运行验证与交付模板.md](./docs/26_domainCheck_发布前运行验证与交付模板.md)
|
||||
|
||||
## 目录结构
|
||||
|
||||
```text
|
||||
@@ -25,10 +36,14 @@
|
||||
├─ start_*.ps1 Windows 启动脚本
|
||||
├─ stop_*.ps1 Windows 停止脚本
|
||||
├─ smoke_test_stack.ps1 本地整栈自测
|
||||
├─ package_domain_release.ps1 交付打包
|
||||
├─ verify_domain_release.ps1 交付包校验
|
||||
├─ prepare_final_release.ps1 最终发布准备
|
||||
└─ show_latest_release.ps1 查看最新交付包
|
||||
├─ package_domain_release.ps1 Windows 交付打包
|
||||
├─ verify_domain_release.ps1 Windows 交付包校验
|
||||
├─ prepare_final_release.ps1 Windows 最终发布准备
|
||||
├─ show_latest_release.ps1 Windows 查看最新交付包
|
||||
├─ package_domain_release.sh Linux / 海外主机交付打包
|
||||
├─ verify_domain_release.sh Linux / 海外主机交付包校验
|
||||
├─ prepare_final_release.sh Linux / 海外主机最终发布准备
|
||||
└─ show_latest_release.sh Linux / 海外主机查看最新交付包
|
||||
```
|
||||
|
||||
## 各目录用途
|
||||
@@ -118,6 +133,15 @@
|
||||
- [verify_domain_release.ps1](./verify_domain_release.ps1)
|
||||
- [prepare_final_release.ps1](./prepare_final_release.ps1)
|
||||
- [show_latest_release.ps1](./show_latest_release.ps1)
|
||||
- [package_domain_release.sh](./package_domain_release.sh)
|
||||
- [verify_domain_release.sh](./verify_domain_release.sh)
|
||||
- [prepare_final_release.sh](./prepare_final_release.sh)
|
||||
- [show_latest_release.sh](./show_latest_release.sh)
|
||||
|
||||
补充说明:
|
||||
|
||||
- `show_latest_release.ps1 / .sh` 现在会校验 `final_release_report.json` 是否真的对应当前 `latest_release.*`
|
||||
- 不是“目录里有 final report 就算已经完成最终签收”
|
||||
|
||||
## 当前状态
|
||||
|
||||
@@ -142,6 +166,69 @@ Linux 阶段建议先看:
|
||||
- [docs/11_domainCheck_Linux联调输入清单.md](./docs/11_domainCheck_Linux联调输入清单.md)
|
||||
- [docs/12_domainCheck_导航索引.md](./docs/12_domainCheck_导航索引.md)
|
||||
|
||||
如果当前已经切到“海外单脑控制面 + 大陆托管节点 + 发布 / Rollout / 驾驶舱首屏”阶段,建议改为:
|
||||
|
||||
1. [docs/25_domainCheck_海外单脑控制面上线收口总表.md](./docs/25_domainCheck_海外单脑控制面上线收口总表.md)
|
||||
2. [docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md](./docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md)
|
||||
3. [docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md](./docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md)
|
||||
4. [docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md](./docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md)
|
||||
5. [docs/schemas/ops_stack_diagnosis_contract.md](./docs/schemas/ops_stack_diagnosis_contract.md)
|
||||
6. [docs/schemas/ops_driver_contract.md](./docs/schemas/ops_driver_contract.md)
|
||||
|
||||
如果当前已经进入“海外主机集中运维 / 多机联调 / Release Hub”阶段,最直接的统一入口是:
|
||||
|
||||
- [domain-api/deploy/multi-region/drive_ops_center.sh](./domain-api/deploy/multi-region/drive_ops_center.sh)
|
||||
- [domain-api/deploy/multi-region/drive_release_hub.sh](./domain-api/deploy/multi-region/drive_release_hub.sh)
|
||||
- [domain-api/deploy/multi-region/drive_ops_action.sh](./domain-api/deploy/multi-region/drive_ops_action.sh)
|
||||
- [domain-api/deploy/multi-region/init_ops_center_config.sh](./domain-api/deploy/multi-region/init_ops_center_config.sh)
|
||||
|
||||
其中:
|
||||
|
||||
- `drive_ops_center.sh`
|
||||
用于海外控制面统一调度检查、Release Hub 和 Driver 动作
|
||||
- `drive_release_hub.sh`
|
||||
用于直接从命令行走 Release / Rollout 标准接口
|
||||
- `drive_ops_action.sh`
|
||||
用于直接从命令行走 `ops/driver-actions/execute`
|
||||
- `init_ops_center_config.sh`
|
||||
用于初始化 `/etc/default/domaincheck-ops-center`
|
||||
|
||||
现在 `drive_ops_center.sh` 还额外收进了本机发布链:
|
||||
|
||||
- `release-package`
|
||||
- `release-verify`
|
||||
- `release-show`
|
||||
- `release-launchpad`
|
||||
- `release-prepare`
|
||||
- `release-preview`
|
||||
- `release-preview-smart`
|
||||
|
||||
当前如果要用一条最短路径判断“现在能不能继续收口、能不能继续发布”,推荐先跑:
|
||||
|
||||
```bash
|
||||
cd /www/wwwroot/getDomain
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
|
||||
```
|
||||
|
||||
然后再按:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
|
||||
```
|
||||
|
||||
继续下钻。
|
||||
|
||||
如果要把这轮检查直接导出成一整包上线证据,也可以直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
|
||||
```
|
||||
|
||||
它会把 `go-live / stack-diagnosis / driver-feed / codex-brief / release-launchpad / doctor-*`
|
||||
统一导出到 `domain-api/runtime/ops-center-reports/go-live-bundles/` 下,并生成一份 `manifest.json`。
|
||||
|
||||
## Git 提交建议
|
||||
|
||||
建议提交:
|
||||
|
||||
148
TASK_BOARD.md
Normal file
148
TASK_BOARD.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# TASK_BOARD
|
||||
|
||||
更新时间:2026-04-18 23:44 CST
|
||||
|
||||
## 当前主批次
|
||||
|
||||
唯一主批次:大陆节点接管闭环最小路径
|
||||
|
||||
目标:
|
||||
|
||||
- 把当前项目的最高优先级从“继续扩控制面能力”切到“完成大陆节点接管闭环”
|
||||
- 只围绕这条最短路径推进:
|
||||
- 录入大陆节点 SSH 入口
|
||||
- 生成并复核 bootstrap plan
|
||||
- 执行 bootstrap
|
||||
- 建立 register / heartbeat
|
||||
- 执行 acceptance
|
||||
- 将 `remote_access_ready` 从 `1/3` 提升到 `3/3`
|
||||
|
||||
主批次成功标准:
|
||||
|
||||
- `mainland-controller-01` 不再是 `pending_bootstrap`
|
||||
- `mainland-worker-01` 不再是 `pending_bootstrap`
|
||||
- `ssh_ready >= 2`
|
||||
- `remote_access_ready = 3/3`
|
||||
- 大陆两台都具备可由海外控制面统一发起标准动作的条件
|
||||
|
||||
当前完成度:
|
||||
|
||||
- 已完成 `SSH` 入口录入
|
||||
- 已完成 `node-handover`
|
||||
- 已完成 `node-bootstrap-plan`
|
||||
- 已确认 `ssh_ready = 2`
|
||||
- 已确认 `remote_access_ready = 3/3`
|
||||
- 尚未完成:
|
||||
- `bootstrap`
|
||||
- `register / heartbeat`
|
||||
- `acceptance`
|
||||
- `pending_bootstrap -> 已接管`
|
||||
|
||||
## 主批次拆解
|
||||
|
||||
### Batch A1
|
||||
|
||||
名称:大陆节点接管闭环最小路径
|
||||
|
||||
边界:
|
||||
|
||||
- 只处理大陆两台:
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
- 只处理接管闭环,不扩展控制面功能
|
||||
|
||||
任务:
|
||||
|
||||
- 已完成:
|
||||
- 录入 `mainland-controller-01` SSH 信息
|
||||
- 录入 `mainland-worker-01` SSH 信息
|
||||
- 复查 `ops/nodes`
|
||||
- 逐台执行 `node-handover`
|
||||
- 逐台生成并复核 `node-bootstrap-plan`
|
||||
- 复核:
|
||||
- `go-live-summary`
|
||||
- `ops/nodes`
|
||||
- `release-launchpad`
|
||||
- 待完成:
|
||||
- 执行 controller 的 `bootstrap -> register -> heartbeat`
|
||||
- 执行 worker 的 `bootstrap -> register -> heartbeat`
|
||||
- 对两台执行 `acceptance`
|
||||
- 让两台都退出 `pending_bootstrap`
|
||||
|
||||
阻塞:
|
||||
|
||||
- controller 与 worker 两台大陆节点都缺少 node-agent 相关运行文件
|
||||
- 远端实际缺失:
|
||||
- `domain-api/app/node_agent.py`
|
||||
- `domain-api/app/api/routes/ops_agent.py`
|
||||
- `domain-api/app/services/ops_agent_service.py`
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- `domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example`
|
||||
- 因此 bootstrap 片段虽然已生成,但远端执行时无法完成安装与启动
|
||||
|
||||
阻塞分类:
|
||||
|
||||
- 主阻塞:环境问题
|
||||
- 大陆节点上的仓库内容落后于当前控制面所需的 node-agent 文件集
|
||||
- 次阻塞:权限 / 外部条件问题
|
||||
- 当前 SSH 自动执行链默认走密钥模式
|
||||
- 今晚实际可用的是密码 SSH
|
||||
|
||||
## 候选批次
|
||||
|
||||
### Candidate B1
|
||||
|
||||
名称:接管闭环完成后的上线收口
|
||||
|
||||
前提:
|
||||
|
||||
- 仅在 `remote_access_ready = 3/3` 后进入
|
||||
|
||||
范围:
|
||||
|
||||
- 重新跑 `go-live-check / stack-diagnosis / release-launchpad`
|
||||
- 收口发布前证据
|
||||
- 判断是否达到正式上线窗口
|
||||
|
||||
### Candidate B2
|
||||
|
||||
名称:接管闭环完成后的值班自动化细化
|
||||
|
||||
前提:
|
||||
|
||||
- 仅在大陆两台已进入稳定接管状态后进入
|
||||
|
||||
范围:
|
||||
|
||||
- 优化运维体验
|
||||
- 优化日常值班命令
|
||||
- 优化交接与证据导出
|
||||
|
||||
当前不进入:
|
||||
|
||||
- `Candidate B1`
|
||||
- `Candidate B2`
|
||||
|
||||
原因:
|
||||
|
||||
- 当前虽然 `remote_access_ready = 3/3`
|
||||
- 但大陆两台仍停在 `pending_bootstrap`
|
||||
- A1 还没有真正闭环
|
||||
|
||||
## 暂停项
|
||||
|
||||
以下事项当前不继续推进:
|
||||
|
||||
- 新的控制面专题设计
|
||||
- 新页面、新模块、新专题文档扩写
|
||||
- 与大陆节点接管闭环无直接关系的 Release Hub 增强
|
||||
- 与大陆节点接管闭环无直接关系的 runbook / playbook 扩展
|
||||
- 更细粒度日志分析功能
|
||||
- 额外的驾驶视图、美化项、信息聚合项
|
||||
- 任何不能直接推动 `bootstrap -> acceptance -> remote_access_ready=3/3` 的功能开发
|
||||
|
||||
## 当前判断
|
||||
|
||||
当前项目不缺“继续横向扩能力”的理由,缺的是“把最后两台大陆节点真正纳入闭环”的收口动作。
|
||||
|
||||
因此下一轮只围绕主批次 A1 推进。
|
||||
@@ -120,3 +120,10 @@ python deploy/linux/smoke_test.py --base-url http://127.0.0.1:8100
|
||||
- Linux 实机联调
|
||||
- 上线前复测
|
||||
- 真实服务器灰度验证
|
||||
|
||||
如果当前已经进入“海外单脑控制面 + 多机联调 + 准备发布”阶段,建议改为按下面顺序执行:
|
||||
|
||||
1. `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
2. `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export`
|
||||
3. `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review`
|
||||
4. `docs/26_domainCheck_发布前运行验证与交付模板.md`
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## 一、用途
|
||||
|
||||
用于在 Windows 本地把当前可交付内容整理成一份压缩包,便于:
|
||||
用于在 Windows 或 Linux 上把当前可交付内容整理成一份发布包,便于:
|
||||
|
||||
- 发给运维或部署同事
|
||||
- 存档版本快照
|
||||
@@ -16,31 +16,60 @@
|
||||
- `verify_domain_release.ps1`
|
||||
- `prepare_final_release.ps1`
|
||||
- `show_latest_release.ps1`
|
||||
- `package_domain_release.sh`
|
||||
- `verify_domain_release.sh`
|
||||
- `prepare_final_release.sh`
|
||||
- `show_latest_release.sh`
|
||||
|
||||
执行方式:
|
||||
Windows 执行方式:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\package_domain_release.ps1
|
||||
```
|
||||
|
||||
Linux / 海外主机执行方式:
|
||||
|
||||
```bash
|
||||
bash ./package_domain_release.sh
|
||||
```
|
||||
|
||||
如需一键完成“打包 + 验包 + 生成最终准备报告”:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\prepare_final_release.ps1
|
||||
```
|
||||
|
||||
```bash
|
||||
bash ./prepare_final_release.sh
|
||||
```
|
||||
|
||||
如需快速查看当前最新交付物:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
|
||||
```
|
||||
|
||||
```bash
|
||||
bash ./show_latest_release.sh
|
||||
```
|
||||
|
||||
注意:
|
||||
|
||||
- `show_latest_release.*` 现在不只是“显示最近一次 final report 是否存在”
|
||||
- 它会额外校验 `final_release_report.json` 是否真的对应当前 `latest_release.*`
|
||||
- 只有 `final_release_report_matches_latest=true` 时,`final_release_ok=true` 才有意义
|
||||
- `final_release_gate_decision / blocked_reasons / verify_ok / smoke_test_ok` 用于解释“为什么当前能签 / 不能签”
|
||||
- Ops Center 里的“直接创建 Release / 一键 Worker 灰度 / 一键 Control 发布”
|
||||
- 现在会硬性依赖 `final_release_ok=true`
|
||||
- 如果 final report 缺失、过期或门禁未 ready,前端会禁用按钮,后端接口也会拒绝执行
|
||||
|
||||
## 三、输出位置
|
||||
|
||||
打包后会生成:
|
||||
|
||||
- `release/domaincheck_release_时间戳/`
|
||||
- `release/domaincheck_release_时间戳.zip`
|
||||
- `release/domaincheck_release_时间戳.tar.gz`
|
||||
- `release/domaincheck_release_时间戳.sha256.txt`
|
||||
- `release/latest_release.txt`
|
||||
- `release/latest_release.json`
|
||||
@@ -50,6 +79,7 @@ powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
|
||||
|
||||
- `docs/`
|
||||
- `scripts/`
|
||||
- `scripts/` 中会同时带上 PowerShell 和 Shell 版发布脚本
|
||||
- `domain-api/`
|
||||
- `app`
|
||||
- `deploy`
|
||||
@@ -89,6 +119,10 @@ powershell -ExecutionPolicy Bypass -File .\show_latest_release.ps1
|
||||
powershell -ExecutionPolicy Bypass -File .\verify_domain_release.ps1
|
||||
```
|
||||
|
||||
```bash
|
||||
bash ./verify_domain_release.sh
|
||||
```
|
||||
|
||||
5. Windows 联调先跑 `scripts/smoke_test_stack.ps1`
|
||||
6. Linux 部署前阅读:
|
||||
- `docs/05_domainCheck_Linux部署清单.md`
|
||||
@@ -106,6 +140,18 @@ powershell -ExecutionPolicy Bypass -File .\verify_domain_release.ps1
|
||||
|
||||
这样更适合交付、存档和迁移。
|
||||
|
||||
Windows 默认产物为 `.zip`,Linux 默认产物为 `.tar.gz`;两者都会生成同结构的 `release_manifest.json / .sha256.txt / latest_release.*`,便于海外控制面统一管理。
|
||||
|
||||
`latest_release.txt / latest_release.json` 用于快速定位“当前最新一份交付包”,避免人工翻目录。
|
||||
|
||||
若打包目录本身是 Git 仓库,脚本还会自动写入 `commit_sha / commit_ref`;海外控制面在创建 Release 时可以直接回填提交号,减少人工抄写出错。
|
||||
|
||||
`final_release_report.json` 用于记录最近一次“打包 + 验包”的最终状态。
|
||||
|
||||
`show_latest_release.*` 读取这个报告时,还会校验:
|
||||
|
||||
- `package_name`
|
||||
- `archive_path` / `zip_path`
|
||||
- `sha256`
|
||||
|
||||
也就是说,即使目录里留着一份旧的 `final_release_report.json`,只要它不是给当前最新包签出的,`final_release_ok` 也不会误报为 `true`。
|
||||
|
||||
@@ -65,9 +65,11 @@
|
||||
|
||||
1. 把最新交付包发送到目标 Linux 服务器
|
||||
2. 按 `docs/05`、`docs/13`、`docs/14` 的顺序执行部署、核对与收口
|
||||
3. 部署完成后再做一次正式服务态 `smoke test`
|
||||
4. 导出一份最终诊断包
|
||||
5. 进入灰度上线
|
||||
3. 如果当前已经进入海外单脑控制面阶段,再按 `docs/25` 执行统一收口
|
||||
4. 用 `docs/26` 完成发布前运行验证、证据导出与最终交付结论
|
||||
5. 部署完成后再做一次正式服务态 `smoke test`
|
||||
6. 导出一份最终诊断包
|
||||
7. 进入灰度上线
|
||||
|
||||
## 六、最终判断
|
||||
|
||||
|
||||
@@ -40,6 +40,51 @@
|
||||
|
||||
- `docs/20_domainCheck_临时海外控制面联调清单.md`
|
||||
|
||||
### 10. 海外主机集中运维与自动化方案
|
||||
|
||||
- `docs/21_domainCheck_海外主机集中运维与自动化方案.md`
|
||||
|
||||
### 11. 海外控制面统一入口
|
||||
|
||||
- `domain-api/deploy/multi-region/drive_ops_center.sh`
|
||||
- `domain-api/deploy/multi-region/export_go_live_bundle.sh`
|
||||
- `domain-api/deploy/multi-region/check_env_drift.sh`
|
||||
- `domain-api/deploy/multi-region/drive_release_hub.sh`
|
||||
- `domain-api/deploy/multi-region/drive_ops_action.sh`
|
||||
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
|
||||
- `domain-api/deploy/multi-region/check_ops_center_stack.sh`
|
||||
- `domain-api/deploy/multi-region/check_ops_contracts.sh`
|
||||
|
||||
### 12. 海外 Codex 驾驶员与 Ops Center 落地路线
|
||||
|
||||
- `docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md`
|
||||
|
||||
### 13. 终局运维架构设计:海外单脑控制面
|
||||
|
||||
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
|
||||
|
||||
### 14. Node Agent 协议与 Release Hub 设计
|
||||
|
||||
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
|
||||
|
||||
### 15. 运维 Contract Schema
|
||||
|
||||
- `docs/schemas/ops_agent_protocol.md`
|
||||
- `docs/schemas/release_hub_contract.md`
|
||||
- `docs/schemas/ops_driver_contract.md`
|
||||
- `docs/schemas/ops_job_contract.md`
|
||||
- `docs/schemas/ops_playbook_contract.md`
|
||||
- `docs/schemas/ops_observability_contract.md`
|
||||
- `docs/schemas/ops_stack_diagnosis_contract.md`
|
||||
|
||||
### 16. 海外单脑控制面上线收口总表
|
||||
|
||||
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
|
||||
### 17. 发布前运行验证与交付模板
|
||||
|
||||
- `docs/26_domainCheck_发布前运行验证与交付模板.md`
|
||||
|
||||
## 二、文档阅读顺序
|
||||
|
||||
### 1. 先看总体方案
|
||||
@@ -58,6 +103,19 @@
|
||||
- `docs/17_domainCheck_全流程部署实操手册.md`
|
||||
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
|
||||
- `docs/20_domainCheck_临时海外控制面联调清单.md`
|
||||
- `docs/21_domainCheck_海外主机集中运维与自动化方案.md`
|
||||
- `docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md`
|
||||
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
|
||||
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
|
||||
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
- `docs/26_domainCheck_发布前运行验证与交付模板.md`
|
||||
- `docs/schemas/ops_agent_protocol.md`
|
||||
- `docs/schemas/release_hub_contract.md`
|
||||
- `docs/schemas/ops_driver_contract.md`
|
||||
- `docs/schemas/ops_job_contract.md`
|
||||
- `docs/schemas/ops_playbook_contract.md`
|
||||
- `docs/schemas/ops_observability_contract.md`
|
||||
- `docs/schemas/ops_stack_diagnosis_contract.md`
|
||||
|
||||
### 3. 如果要回溯历史需求与问题
|
||||
|
||||
@@ -89,6 +147,11 @@
|
||||
- `domain-api/deploy/systemd/domain-worker.service`
|
||||
- `domain-api/deploy/linux/README.md`
|
||||
- `domain-api/deploy/multi-region/README.md`
|
||||
- `domain-api/deploy/multi-region/drive_ops_center.sh`
|
||||
- `domain-api/deploy/multi-region/drive_release_hub.sh`
|
||||
- `domain-api/deploy/multi-region/drive_ops_action.sh`
|
||||
- `domain-api/deploy/multi-region/check_ops_contracts.sh`
|
||||
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
|
||||
- `domain-api/deploy/multi-region/bootstrap_overseas.sh`
|
||||
- `domain-api/deploy/multi-region/bootstrap_mainland.sh`
|
||||
- `domain-api/deploy/linux/smoke_test.py`
|
||||
@@ -109,6 +172,8 @@
|
||||
- 配置迁移与备份
|
||||
- 打包与验包
|
||||
- 最终发布准备
|
||||
- 海外单脑控制面协议收口
|
||||
- Ops Center / Codex / CLI 驾驶 contract 收口
|
||||
|
||||
剩余工作只在真实 Linux 环境:
|
||||
|
||||
|
||||
@@ -31,6 +31,12 @@
|
||||
- 海外控制面 `runtime/sync-summary`
|
||||
- 能看到 `detect_result_batches`
|
||||
|
||||
如果当前已经切到“海外单脑控制面统一驾驶”模式,建议同时配套阅读:
|
||||
|
||||
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
|
||||
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
|
||||
|
||||
## 二、上线前必须确认的结论
|
||||
|
||||
上线前至少要确认下面这些结论同时成立:
|
||||
@@ -46,6 +52,34 @@
|
||||
|
||||
## 三、正式上线前执行顺序
|
||||
|
||||
### 0. 先跑一遍统一上线总检
|
||||
|
||||
如果当前已经进入“海外单脑控制面 + 多地域 / Node Agent”这条正式链路,建议先跑:
|
||||
|
||||
```bash
|
||||
cd /www/wwwroot/getDomain/domain-api
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://152.53.37.118:8100 summary
|
||||
```
|
||||
|
||||
期望:
|
||||
|
||||
- `go_live_status` 为 `ready` 或至少没有新的 `blocking_reasons`
|
||||
- `stack_status` 不为 `blocked`
|
||||
- `route_surface_complete=true`
|
||||
- `launchpad_status` 不为 `blocked`
|
||||
- 如果当前已经纳管执行节点,则 `remote_access_ready` 不应长期为 `0`
|
||||
- `next_step_action_code` / `operator_title` 应能直接说明当前第一优先动作
|
||||
- 如果 `log_sync_enabled=true`,则要额外看 `log_sync_state`、`log_sync_covered_nodes`
|
||||
|
||||
如果这里已经出现 `bootstrap_run / run_acceptance`,说明当前上线前仍有节点接管缺口,应先收口首个缺口节点,再继续下面的业务核验。
|
||||
|
||||
建议这样理解总检输出:
|
||||
|
||||
- `blocking_reasons` 只放真正会阻断上线的项,例如 `remote_agent_ready=0`
|
||||
- `warnings` 表示还能继续联调,但现在还不适合直接宣称“全部收口”
|
||||
- `recommended_commands.next_step` 就是当前最值得优先执行的一条命令
|
||||
- `recommended_commands.log_sync_logs` / `recommended_commands.log_sync_inspection` 用来补远端参与节点的日志观测
|
||||
|
||||
### 1. 核对服务状态
|
||||
|
||||
```bash
|
||||
|
||||
@@ -165,10 +165,20 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||
- `detect_worker_nodes`
|
||||
- 看谁在线
|
||||
- 看谁是 `worker_online=true`
|
||||
- 看谁 `detect_participating=true`
|
||||
- `detect_participating=true` 只能说明它曾被判断为“正在参与”
|
||||
- `detect_job_items`
|
||||
- 看任务到底被哪台节点 `claimed_by`
|
||||
- 这才是“当前真正干活的节点”
|
||||
- `runtime/status`
|
||||
- 看 `participation_summary.dispatch_active_nodes`
|
||||
- 这是“当前真正执行/领任务的节点”
|
||||
- 看 `participation_summary.non_participating_nodes`
|
||||
- 这是“在线但未参与的节点”
|
||||
- 看 `non_participating_nodes[].participation_state`
|
||||
- `standby` 表示在线待命
|
||||
- `load_syncing` 表示负载待确认,不要直接算成“正在干活”
|
||||
- 看 `log_sync`
|
||||
- 能直接确认远端日志回传是否开启、是关键还是全量、样本来自哪些节点
|
||||
|
||||
## 8. 常见误判
|
||||
|
||||
@@ -182,6 +192,7 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||
|
||||
- “有效执行节点”是可承担任务的在线节点数
|
||||
- “当前参与检测节点”是当前真的在领任务、跑任务的节点
|
||||
- “在线但未参与节点”是已经在线、可承接任务,但当前这轮还没分到任务的节点
|
||||
|
||||
如果 controller 没兼跑检测,通常只会看到独立 worker 在真正执行。
|
||||
|
||||
@@ -189,6 +200,26 @@ bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||
|
||||
这不一定是 bug,可能只是当前调度没有分到它。应以 `detect_job_items.claimed_by` 为准,不要只看服务在线。
|
||||
|
||||
### 8.4 页面显示“负载待确认”
|
||||
|
||||
这不是新的故障状态,而是为了避免误判:
|
||||
|
||||
- 节点已经上报 `busy` 或有 `current_load`
|
||||
- 但当前还没看到明确的 `items_claimed / items_running / processed_recent`
|
||||
|
||||
通常是心跳和任务快照还没完全对齐。先等下一轮刷新,再结合 `claimed_by` 和 `participation_summary` 判断,不要立刻当作“这台机器已经在跑检测”。
|
||||
|
||||
### 8.5 日志控制台看不到大陆节点过程
|
||||
|
||||
先看 `runtime/status` 或页面里的“远端日志回传”:
|
||||
|
||||
- 若 `enabled=false`
|
||||
- 说明本来就没开回传
|
||||
- 若 `enabled=true` 但 `line_count=0`
|
||||
- 说明开关已开,但当前还没有远端样本
|
||||
- 若 `source_nodes` 里没有目标大陆节点
|
||||
- 说明该节点这轮还没回传日志,先看它是否真的在参与检测
|
||||
|
||||
## 9. 联调完成后的回滚
|
||||
|
||||
如果你后续要切回正式海外机器,在大陆 controller 上把目标地址改回正式值即可:
|
||||
|
||||
708
docs/21_domainCheck_海外主机集中运维与自动化方案.md
Normal file
708
docs/21_domainCheck_海外主机集中运维与自动化方案.md
Normal file
@@ -0,0 +1,708 @@
|
||||
# 21 domainCheck 海外主机集中运维与自动化方案
|
||||
|
||||
如果当前你不是在做“为什么要这么设计”的方案评审,而是已经进入真实收口、准备上线阶段,建议先跳转:
|
||||
|
||||
- `docs/25_domainCheck_海外单脑控制面上线收口总表.md`
|
||||
- `docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md`
|
||||
- `docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md`
|
||||
|
||||
## 一、为什么要升级成“集中运维”
|
||||
|
||||
这段时间的多机联调已经证明一个事实:
|
||||
|
||||
- 现在这套“人在海外主机上看后台,再去大陆机器上手工跑命令、复制日志、判断状态”的方式,能联调,但效率很低
|
||||
- 一旦节点数从 `2` 台增加到 `3` 台、`5` 台、`10` 台,复杂度会快速失控
|
||||
- 国内机器上又不能稳定部署 Codex,因此不能指望每台机器都具备“本机智能调试能力”
|
||||
- 检测日志很大,靠人工复制日志片段,不适合做持续诊断
|
||||
|
||||
所以后续不应该继续堆更多“检查脚本”,而应该把整个体系升级成:
|
||||
|
||||
> 海外主机作为唯一运维控制面,统一接管大陆机器的安装、更新、重启、巡检、日志回流和故障诊断。
|
||||
|
||||
一句话目标:
|
||||
|
||||
> 只在海外主机操作,大陆机器尽量不需要人工登录。
|
||||
|
||||
---
|
||||
|
||||
## 二、最终目标形态
|
||||
|
||||
### 1. 海外主机承担什么
|
||||
|
||||
海外主机作为唯一控制面,承担:
|
||||
|
||||
- Web 后台
|
||||
- API 控制面
|
||||
- 运维任务中心
|
||||
- 发布包仓库
|
||||
- 节点注册与权限中心
|
||||
- 日志汇聚与诊断入口
|
||||
- 远程命令编排
|
||||
- 巡检结果展示
|
||||
|
||||
也就是说,后续所有动作都从海外主机发起:
|
||||
|
||||
- 新机器纳管
|
||||
- 初始化安装
|
||||
- 发布更新
|
||||
- 配置下发
|
||||
- 服务重启
|
||||
- 健康检查
|
||||
- 采集诊断包
|
||||
- 远端日志查看
|
||||
- 故障一键排查
|
||||
|
||||
### 2. 大陆机器承担什么
|
||||
|
||||
大陆机器不再承担复杂控制逻辑,只承担:
|
||||
|
||||
- `domaincheck worker/controller` 本体服务
|
||||
- 一个轻量 Node Agent
|
||||
- 本机 systemd / 日志 / 版本 / 健康信息暴露
|
||||
- 接收控制面任务并执行
|
||||
- 将执行结果、日志、诊断包回传到海外主机
|
||||
|
||||
这意味着大陆机器以后是“被管理对象”,而不是“人工登录操作对象”。
|
||||
|
||||
---
|
||||
|
||||
## 三、推荐方案
|
||||
|
||||
## 方案 A:海外控制面 + 大陆 Node Agent
|
||||
|
||||
这是我建议的主方案,也是长期最优方案。
|
||||
|
||||
### 核心思想
|
||||
|
||||
不要把“SSH 到每台机器执行命令”作为主链路,而是让每台大陆机器常驻一个 Agent:
|
||||
|
||||
- Agent 主动连海外控制面
|
||||
- Agent 拉取待执行任务
|
||||
- 本地执行 systemd / shell / 发布 / 巡检
|
||||
- Agent 把 stdout / stderr / 状态 / 日志游标回传
|
||||
|
||||
这样做的好处是:
|
||||
|
||||
- 不要求海外机能直接入站打通到大陆机
|
||||
- 不要求每次人工 SSH
|
||||
- 不怕 SSH 权限、跳板机、端口变化导致整套流程断掉
|
||||
- 天然适合 NAT、弱网络、多机扩容
|
||||
|
||||
### 交互方式
|
||||
|
||||
建议 Agent 使用下面其中一种方式主动连海外控制面:
|
||||
|
||||
#### 首选:HTTPS 长轮询
|
||||
|
||||
- `agent -> overseas-api`
|
||||
- 周期拉取任务
|
||||
- 周期上报心跳、版本、服务状态、日志摘要
|
||||
|
||||
优点:
|
||||
|
||||
- 实现最简单
|
||||
- 最容易兼容现有 Python / FastAPI 架构
|
||||
- 容易先落 MVP
|
||||
|
||||
#### 可升级:WebSocket 常连
|
||||
|
||||
- 建立长连接
|
||||
- 海外控制面可实时下发任务
|
||||
- Agent 实时回传执行日志
|
||||
|
||||
优点:
|
||||
|
||||
- 更实时
|
||||
- 日志流式体验更好
|
||||
|
||||
缺点:
|
||||
|
||||
- 第一版复杂度更高
|
||||
|
||||
### 为什么 Agent 比 SSH 更优
|
||||
|
||||
SSH 适合作为:
|
||||
|
||||
- 首次 bootstrap
|
||||
- 临时人工兜底
|
||||
- 非常规应急
|
||||
|
||||
但不适合作为日常主运维链路,因为:
|
||||
|
||||
- 节点一多,权限和连通性管理会变得脆弱
|
||||
- 脚本回显、超时、日志采集很难标准化
|
||||
- 人工 SSH 本质上还是“远程手工运维”
|
||||
|
||||
所以最佳做法是:
|
||||
|
||||
- SSH 只用于首次纳管
|
||||
- 日常统一走 Agent
|
||||
|
||||
## 三点五、当前已经落地的第一版操作闭环
|
||||
|
||||
这套方案现在已经不只是设计稿,仓库里已经有一批可以直接使用的统一入口:
|
||||
|
||||
- `domain-api/deploy/multi-region/init_ops_center_config.sh`
|
||||
- `domain-api/deploy/multi-region/drive_ops_center.sh`
|
||||
- `domain-api/deploy/multi-region/drive_release_hub.sh`
|
||||
- `domain-api/deploy/multi-region/drive_ops_action.sh`
|
||||
- `domain-api/deploy/multi-region/build_node_agent_bootstrap_plan.sh`
|
||||
|
||||
推荐从海外控制面按这个顺序使用:
|
||||
|
||||
### 1. 初始化控制面配置
|
||||
|
||||
```bash
|
||||
cd /opt/domaincheck/domain-api
|
||||
bash domain-api/deploy/multi-region/init_ops_center_config.sh \
|
||||
/etc/default/domaincheck-ops-center \
|
||||
http://121.204.244.188:8100 \
|
||||
http://152.53.37.118:8100 \
|
||||
https://api.example.com
|
||||
```
|
||||
|
||||
### 2. 查看当前统一配置
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh config
|
||||
```
|
||||
|
||||
### 3. 在控制面本机生成发布包
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-package
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-preview
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-preview-smart worker
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-show
|
||||
```
|
||||
|
||||
### 4. 导出一份联调/交接报告
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export
|
||||
```
|
||||
|
||||
### 5. 为新节点导出接管计划
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-plan-export \
|
||||
/opt/domaincheck/domain-api/runtime/ops-center-reports/agent-plans \
|
||||
http://127.0.0.1:8100 \
|
||||
mainland-worker-02 \
|
||||
mainland \
|
||||
worker \
|
||||
https://api.example.com \
|
||||
/opt/domaincheck
|
||||
```
|
||||
|
||||
这五步对应的意义分别是:
|
||||
|
||||
- 先统一控制面默认地址和身份
|
||||
- 再确认驾驶舱当前在看哪个环境
|
||||
- 再把发布物、发布前预检和发版驾驶舱都收回统一入口,不再额外记根目录脚本
|
||||
- 再把现场收敛成可以回看、可以交接的报告
|
||||
- 最后为新增节点生成标准接管方案
|
||||
|
||||
这样后面无论是海外 Codex 自动驾驶、后台按钮,还是人工运维,都不再需要先去大陆机器手工拼命令。
|
||||
|
||||
## 三点六、现在发布驾驶舱已经进入统一数据口径
|
||||
|
||||
目前发布侧也已经不是零散接口拼出来的状态,而是收敛成了一套统一判断:
|
||||
|
||||
- `GET /api/v1/ops/releases/launchpad`
|
||||
- `ops overview -> release_hub.launchpad`
|
||||
- `check_release_hub.sh`
|
||||
- `drive_release_hub.sh launchpad`
|
||||
- 运维中枢页面里的“发布驾驶舱”
|
||||
|
||||
这几处现在复用的是同一套结论,都会统一告诉你:
|
||||
|
||||
- 当前是 `ready / attention / blocked`
|
||||
- 最新发布包是否可用
|
||||
- 最新 Release 是否已经建立
|
||||
- Worker 智能灰度是否可发
|
||||
- Control 发布是否可发
|
||||
- 下一步推荐动作是什么
|
||||
|
||||
同时,`GET /api/v1/ops/runbook` 里的标准作业路径 `release_progression` 也已经切到同一套发布驾驶舱判断。
|
||||
也就是说,海外控制面看到的“标准作业路径”和“发布驾驶舱”不会再出现一套说能发、一套说先补接管的分裂口径。
|
||||
|
||||
现在又往前推进了一步:
|
||||
|
||||
- `POST /api/v1/ops/runbook/sequences/{sequence_key}/execute`
|
||||
|
||||
后台上的标准作业路径按钮,后续都应该优先走这个入口,再由后端统一分发到 driver action / playbook / rollout 预案,而不是前端自己维护动作分支。
|
||||
|
||||
这样后续海外 Codex、后台按钮和命令行脚本拿到的就不再是三套不同口径,而是同一套“发布驾驶判断”。
|
||||
|
||||
## 三点七、现在回执队列也已经进入统一治理入口
|
||||
|
||||
除了发布驾驶舱,Node Agent 回执队列这一层现在也已经不再是“只能看、不能管”的状态。
|
||||
|
||||
当前已经统一收口成三类入口:
|
||||
|
||||
- 页面:
|
||||
- OpsCenter 托管节点表可以直接打开“回执队列治理”抽屉
|
||||
- 驾驶建议:
|
||||
- `dead_letter / retrying` 会优先落到标准动作模板
|
||||
- 海外单入口 CLI:
|
||||
- `drive_ops_center.sh queue-status`
|
||||
- `drive_ops_center.sh queue-records`
|
||||
- `drive_ops_center.sh queue-flush`
|
||||
- `drive_ops_center.sh queue-replay`
|
||||
- `drive_ops_center.sh queue-replay-record`
|
||||
- `drive_ops_center.sh queue-discard-record`
|
||||
|
||||
这意味着海外主机已经可以统一处理:
|
||||
|
||||
- 哪台节点存在积压 / 死信
|
||||
- 头部记录是什么
|
||||
- 什么时候应该立即冲刷
|
||||
- 什么时候应该重放
|
||||
- 什么时候应该说明原因后丢弃
|
||||
|
||||
当前阶段仍然明确限制为:
|
||||
|
||||
- `record_visibility=head_only`
|
||||
- 所有治理动作都落成正式 `ops job`
|
||||
- 不直接 SSH 到节点改队列文件
|
||||
|
||||
这样以后即使把远端全量死信记录正式投影到控制面,也只是“扩可见范围”,而不是重新推翻当前模型。
|
||||
|
||||
---
|
||||
|
||||
## 四、这套方案要解决的具体问题
|
||||
|
||||
## 1. 安装部署自动化
|
||||
|
||||
目标:
|
||||
|
||||
- 海外控制面点一次“新增节点”
|
||||
- 填一个节点模板
|
||||
- 大陆机器自动初始化
|
||||
|
||||
建议流程:
|
||||
|
||||
1. 海外后台新增节点
|
||||
2. 生成 `bootstrap token`
|
||||
3. 大陆机器只执行一次极短安装命令
|
||||
4. Node Agent 注册到海外控制面
|
||||
5. 控制面向该节点下发:
|
||||
- 拉取发布包
|
||||
- 解压
|
||||
- 生成 `/etc/default/...`
|
||||
- 安装 systemd
|
||||
- 启动服务
|
||||
6. 回传安装结果
|
||||
|
||||
这样后续加机器时,不需要再重复人工对照文档逐条敲。
|
||||
|
||||
## 2. 更新发布自动化
|
||||
|
||||
目标:
|
||||
|
||||
- 海外主机统一推版本
|
||||
- 大陆节点自动拉包、灰度更新、失败回滚
|
||||
|
||||
这里强烈建议:
|
||||
|
||||
> 后续不要让大陆机器自己跑 git 作为正式更新链路。
|
||||
|
||||
而是改成:
|
||||
|
||||
- 海外控制面构建发布包
|
||||
- 版本号固定
|
||||
- Agent 下载指定 release 包
|
||||
- 校验 checksum
|
||||
- 切换当前版本软链
|
||||
- 重启服务
|
||||
- 回传成功/失败
|
||||
|
||||
这样比远程 `git pull` 更稳,因为:
|
||||
|
||||
- 不依赖每台机器 git 权限
|
||||
- 不怕工作区脏文件
|
||||
- 不怕 root/www 用户混用
|
||||
- 可回滚
|
||||
|
||||
推荐目录形态:
|
||||
|
||||
```text
|
||||
/opt/domaincheck/releases/<version>/
|
||||
/opt/domaincheck/current -> /opt/domaincheck/releases/<version>/
|
||||
```
|
||||
|
||||
更新时:
|
||||
|
||||
- 下载新版本到 `releases`
|
||||
- 校验
|
||||
- 切换 `current`
|
||||
- `systemctl restart ...`
|
||||
|
||||
失败就回滚软链。
|
||||
|
||||
## 3. 远程命令执行自动化
|
||||
|
||||
目标:
|
||||
|
||||
- 海外控制面能发“结构化任务”
|
||||
- 不再让人手工复制命令
|
||||
|
||||
建议任务类型:
|
||||
|
||||
- `service.start`
|
||||
- `service.stop`
|
||||
- `service.restart`
|
||||
- `service.status`
|
||||
- `deploy.release`
|
||||
- `config.render`
|
||||
- `diagnostics.collect`
|
||||
- `logs.tail`
|
||||
- `script.run`
|
||||
- `health.check`
|
||||
|
||||
每个任务统一回传:
|
||||
|
||||
- 任务 ID
|
||||
- 节点编码
|
||||
- 开始时间 / 结束时间
|
||||
- exit code
|
||||
- stdout
|
||||
- stderr
|
||||
- 结构化结果 JSON
|
||||
|
||||
这样后面后台才能真正做“操作中心”。
|
||||
|
||||
## 4. 日志集中回流
|
||||
|
||||
目标:
|
||||
|
||||
- 海外后台就能看到大陆机器日志
|
||||
- 不再人工抄 `journalctl`
|
||||
|
||||
### 日志回流建议分三层
|
||||
|
||||
#### A. 关键事件流
|
||||
|
||||
只回传关键事件:
|
||||
|
||||
- 服务启动
|
||||
- 服务重启
|
||||
- 任务开始
|
||||
- 任务完成
|
||||
- 代理刷新失败
|
||||
- Redis/DB 连接异常
|
||||
- 第三方站点异常
|
||||
|
||||
适合默认长期开启。
|
||||
|
||||
#### B. 诊断模式日志流
|
||||
|
||||
像你现在提的“关闭 / 关键 / 全量回传”一样:
|
||||
|
||||
- `off`
|
||||
- `key`
|
||||
- `full`
|
||||
|
||||
这是正确方向,应该继续保留。
|
||||
|
||||
#### C. 诊断包
|
||||
|
||||
当出现疑难问题时,一键打包:
|
||||
|
||||
- 最近 N 分钟 `journalctl`
|
||||
- `runtime/status`
|
||||
- `runtime/cluster`
|
||||
- `detect_worker.log tail`
|
||||
- `/etc/default/*`
|
||||
- 当前版本号
|
||||
- 本机健康检查输出
|
||||
|
||||
然后上传海外主机。
|
||||
|
||||
这比全量实时传所有日志更经济,也更适合定位问题。
|
||||
|
||||
## 5. 健康巡检自动化
|
||||
|
||||
目标:
|
||||
|
||||
- 每台大陆机自动自检
|
||||
- 海外后台只看结果
|
||||
|
||||
建议 Agent 每 30 秒到 60 秒上报:
|
||||
|
||||
- 节点在线状态
|
||||
- 当前版本
|
||||
- API / Worker / Sync Agent 运行态
|
||||
- Redis / PostgreSQL / 磁盘 / 内存 / CPU
|
||||
- 最近告警
|
||||
- 当前任务负载
|
||||
- 最近日志时间
|
||||
|
||||
并把现有脚本升级为 Agent 内部检查项:
|
||||
|
||||
- `check_mainland_controller.sh`
|
||||
- `check_mainland_worker.sh`
|
||||
- `check_temp_topology.sh`
|
||||
- `check_worker_participation.sh`
|
||||
|
||||
后续不是人工执行这些脚本,而是 Agent 周期性执行并结构化上传结果。
|
||||
|
||||
---
|
||||
|
||||
## 五、建议的系统分层
|
||||
|
||||
## 1. 海外控制面
|
||||
|
||||
建议新增一个“运维控制模块”,逻辑上可放进 `domain-api`,后续再拆独立服务也可以。
|
||||
|
||||
### 需要的核心对象
|
||||
|
||||
#### `managed_nodes`
|
||||
|
||||
记录纳管节点:
|
||||
|
||||
- `node_code`
|
||||
- `region`
|
||||
- `role`
|
||||
- `hostname`
|
||||
- `ip`
|
||||
- `agent_version`
|
||||
- `current_release`
|
||||
- `status`
|
||||
- `ssh_enabled`
|
||||
- `agent_last_seen_at`
|
||||
- `tags`
|
||||
|
||||
#### `ops_jobs`
|
||||
|
||||
记录运维任务:
|
||||
|
||||
建议以后直接以 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 为正式母本。
|
||||
|
||||
- `job_id`
|
||||
- `job_type`
|
||||
- `target_nodes`
|
||||
- `payload`
|
||||
- `created_by`
|
||||
- `status`
|
||||
- `started_at`
|
||||
- `finished_at`
|
||||
|
||||
#### `ops_job_steps`
|
||||
|
||||
记录每个节点执行步骤:
|
||||
|
||||
- `node_code`
|
||||
- `step_name`
|
||||
- `status`
|
||||
- `stdout`
|
||||
- `stderr`
|
||||
- `result_json`
|
||||
|
||||
#### `node_log_streams`
|
||||
|
||||
记录日志游标和日志回流状态:
|
||||
|
||||
- `node_code`
|
||||
- `source`
|
||||
- `mode`
|
||||
- `cursor`
|
||||
- `last_received_at`
|
||||
|
||||
## 2. Node Agent
|
||||
|
||||
建议单独做一个轻量 Python 服务,比如:
|
||||
|
||||
```text
|
||||
domaincheck-node-agent
|
||||
```
|
||||
|
||||
职责:
|
||||
|
||||
- 周期心跳
|
||||
- 拉取任务
|
||||
- 本地执行
|
||||
- 采集日志
|
||||
- 上报结果
|
||||
- 生成诊断包
|
||||
|
||||
Agent 尽量独立于业务主程序,不要把它耦合进 `detect_worker.py`。
|
||||
|
||||
原因:
|
||||
|
||||
- 运维系统不能依赖业务进程是否正常
|
||||
- 即使 Worker 崩了,Agent 还应该活着,才能帮你排障
|
||||
|
||||
---
|
||||
|
||||
## 六、对你当前项目最合适的落地路线
|
||||
|
||||
## 第一阶段:先做“控制面集中化”
|
||||
|
||||
目标:
|
||||
|
||||
- 海外后台统一看到所有大陆机器
|
||||
- 一键拉诊断
|
||||
- 一键执行已有检查脚本
|
||||
- 一键切日志回传模式
|
||||
|
||||
这个阶段先不碰太多发布链路,优先把“看”和“查”统一。
|
||||
|
||||
### 交付物
|
||||
|
||||
- Node Agent MVP
|
||||
- 海外后台“节点管理”页
|
||||
- 海外后台“运维任务”页
|
||||
- 海外后台“远端日志”页
|
||||
- 海外后台“诊断包”页
|
||||
|
||||
## 第二阶段:再做“一键更新”
|
||||
|
||||
目标:
|
||||
|
||||
- 海外控制面选择版本
|
||||
- 指定节点灰度发布
|
||||
- 自动回滚
|
||||
|
||||
### 交付物
|
||||
|
||||
- 发布包生成器
|
||||
- `release manifest`
|
||||
- Agent 下载 / 校验 / 切换版本
|
||||
- 回滚机制
|
||||
|
||||
## 第三阶段:再做“一键安装新节点”
|
||||
|
||||
目标:
|
||||
|
||||
- 新机器只执行一次 bootstrap
|
||||
- 其余全部由海外控制面接管
|
||||
|
||||
### 交付物
|
||||
|
||||
- bootstrap token
|
||||
- 安装向导
|
||||
- 节点注册流程
|
||||
- 环境模板生成器
|
||||
|
||||
---
|
||||
|
||||
## 七、比“SSH 全接管”更优的地方
|
||||
|
||||
你提的方向是:
|
||||
|
||||
> 海外主机配置好 SSH,后续所有大陆机器都由代码内部接管
|
||||
|
||||
这个方向本身是对的,但如果只做“SSH 全接管”,还不够优秀。
|
||||
|
||||
更优解是:
|
||||
|
||||
> SSH 只作为纳管和兜底方式;日常统一走 Agent 主动连接。
|
||||
|
||||
这样好处是:
|
||||
|
||||
- 架构更稳
|
||||
- 不依赖每次远程入站打通
|
||||
- 不怕某些网络环境 SSH 不稳定
|
||||
- 更适合未来节点继续增加
|
||||
- 更容易做权限分级、审计、结果留痕
|
||||
|
||||
所以我建议最终定成:
|
||||
|
||||
### 运维主链路
|
||||
|
||||
- `海外控制面 -> Agent 任务编排 -> 大陆节点执行 -> 回传结果`
|
||||
|
||||
### 运维兜底链路
|
||||
|
||||
- `海外控制面 -> SSH -> 大陆节点`
|
||||
|
||||
---
|
||||
|
||||
## 八、和现有代码如何衔接
|
||||
|
||||
当前项目已经有一些很好的基础,不需要推翻:
|
||||
|
||||
- 多机节点模型
|
||||
- `runtime/status`
|
||||
- `runtime/cluster`
|
||||
- `sync-summary`
|
||||
- 一批巡检脚本
|
||||
- 日志回传开关雏形
|
||||
- 运行中心
|
||||
|
||||
这些都应该保留,并演进成:
|
||||
|
||||
### 当前脚本的未来定位
|
||||
|
||||
- `bootstrap_*.sh`
|
||||
- 保留为 bootstrap / 应急工具
|
||||
- `check_*.sh`
|
||||
- 演进成 Agent 内部的标准诊断动作
|
||||
- `runtime/status`
|
||||
- 继续作为统一运行态接口
|
||||
- `runtime/cluster`
|
||||
- 继续作为统一集群视图
|
||||
- `detect_result_projection / runtime_projection`
|
||||
- 继续作为跨地域观测基础
|
||||
|
||||
也就是说,现有代码不是废掉,而是从“人工脚本时代”升级成“控制面编排时代”。
|
||||
|
||||
---
|
||||
|
||||
## 九、我建议的最优落地决策
|
||||
|
||||
如果只选一个方向,我建议直接定成:
|
||||
|
||||
### 最优方案
|
||||
|
||||
- 海外主机为唯一控制面
|
||||
- 大陆节点统一部署 `domaincheck-node-agent`
|
||||
- Agent 主动访问海外控制面
|
||||
- 日常安装、更新、巡检、日志回流全部走 Agent
|
||||
- SSH 仅保留为 bootstrap 和应急手段
|
||||
- 正式更新统一改为 release 包分发,不再把 `git pull` 作为正式更新链路
|
||||
|
||||
这是当前最值得投入的方向,因为它能同时解决:
|
||||
|
||||
- 调试效率低
|
||||
- 多机管理复杂
|
||||
- 权限混乱
|
||||
- 日志分散
|
||||
- 更新不稳定
|
||||
- 新增节点成本高
|
||||
|
||||
---
|
||||
|
||||
## 十、建议的下一步执行顺序
|
||||
|
||||
不要再继续先补散点 bug,而是先把运维骨架建起来。
|
||||
|
||||
建议顺序:
|
||||
|
||||
1. 固化这份方案
|
||||
2. 新建 `node-agent` 设计草案
|
||||
3. 先做 Agent MVP
|
||||
4. 先接入:
|
||||
- 心跳
|
||||
- 远程任务
|
||||
- 诊断包
|
||||
- 日志 tail
|
||||
5. 海外后台新增:
|
||||
- 节点管理页
|
||||
- 运维任务页
|
||||
- 远端日志页
|
||||
6. 再把现有检查脚本封成 Agent 动作
|
||||
7. 最后再做发布链路自动化
|
||||
|
||||
---
|
||||
|
||||
## 十一、结论
|
||||
|
||||
当前最优策略不是“继续加脚本”,而是:
|
||||
|
||||
> 把 domainCheck 从“多机联调项目”升级成“海外控制面统一接管大陆节点的自动化运维系统”。
|
||||
|
||||
从长期看,这会比继续靠人工 SSH、人工复制日志、人工比对状态高效很多,也更符合你后续继续扩机器的目标。
|
||||
1052
docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md
Normal file
1052
docs/22_domainCheck_海外Codex驾驶员与OpsCenter落地路线.md
Normal file
File diff suppressed because it is too large
Load Diff
1163
docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md
Normal file
1163
docs/23_domainCheck_终局运维架构设计_海外单脑控制面.md
Normal file
File diff suppressed because it is too large
Load Diff
4421
docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md
Normal file
4421
docs/24_domainCheck_NodeAgent协议与ReleaseHub设计.md
Normal file
File diff suppressed because it is too large
Load Diff
430
docs/25_domainCheck_海外单脑控制面上线收口总表.md
Normal file
430
docs/25_domainCheck_海外单脑控制面上线收口总表.md
Normal file
@@ -0,0 +1,430 @@
|
||||
# 25 domainCheck 海外单脑控制面上线收口总表
|
||||
|
||||
## 一、这份文档解决什么问题
|
||||
|
||||
这份文档不是再讲设计,而是把当前项目进入正式上线前,真正需要收口的事项压成一张总表。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 海外主机已经作为单脑控制面
|
||||
- 大陆节点已经开始纳管
|
||||
- Ops Center / Codex Driver / CLI 都已经接入同一套 `ops` contract
|
||||
- 目标从“能联调”切换到“能稳定上线、能持续运维”
|
||||
|
||||
这份文档优先回答 4 个问题:
|
||||
|
||||
1. 现在能不能继续收口
|
||||
2. 现在能不能正式发布
|
||||
3. 还有哪些阻断项没清
|
||||
4. 下一步到底先跑哪条命令
|
||||
|
||||
---
|
||||
|
||||
## 二、上线前统一原则
|
||||
|
||||
上线前必须统一成下面这条工作方式:
|
||||
|
||||
- 海外主机作为唯一主驾驶席
|
||||
- 页面、CLI、Codex 共用同一套后端判断
|
||||
- 默认先看 `go-live-summary`
|
||||
- 真要解释原因时再下钻 `stack-diagnosis`
|
||||
- 真要执行动作时走 `driver-resolve / execute-resolved`
|
||||
- 真要发布或放量时走 `Release Hub / rollout`
|
||||
|
||||
不再推荐:
|
||||
|
||||
- 人工分别登录多台大陆机器拼状态
|
||||
- 每次上线都从 `journalctl + systemctl + curl` 临时组合判断
|
||||
- 前端、CLI、Codex 各自维护不同的默认下一步
|
||||
|
||||
---
|
||||
|
||||
## 三、正式上线门禁
|
||||
|
||||
### 1. 收口门禁
|
||||
|
||||
至少同时满足:
|
||||
|
||||
- `go_live_summary.go_live_status != blocked`
|
||||
- `stack_diagnosis.diagnosis.stack_status != blocked`
|
||||
- `route_surface_complete=true`
|
||||
- `driver-feed.automation_coverage.launch_status != blocked`
|
||||
- 首屏默认下一步已经明确,不再是模糊人工判断
|
||||
|
||||
### 2. 发布门禁
|
||||
|
||||
至少同时满足:
|
||||
|
||||
- `publish_ready=true`
|
||||
- `launchpad_status != blocked`
|
||||
- 不存在未处理的关键 `blocking_reasons`
|
||||
- 当前发布主车道已经清晰落在:
|
||||
- `review_smart_rollout_preview`
|
||||
- `review_control_rollout`
|
||||
- `publish_latest_worker`
|
||||
- `create_release_rollout_*`
|
||||
|
||||
### 3. 运维门禁
|
||||
|
||||
至少同时满足:
|
||||
|
||||
- 海外控制面 API 稳定在线
|
||||
- 大陆 controller / worker 心跳稳定
|
||||
- 节点接管缺口已经收口,或至少首个缺口节点已有明确恢复动作
|
||||
- 远端日志回传可按需开启、复核、关闭
|
||||
- 运行中心首屏能直接区分:
|
||||
- 在线但未参与
|
||||
- 正在执行 / 正在领任务
|
||||
|
||||
---
|
||||
|
||||
## 四、真正的起手顺序
|
||||
|
||||
每次准备上线、复核、值班接手时,都按这个顺序来:
|
||||
|
||||
### 第 1 步:看上线收口摘要
|
||||
|
||||
```bash
|
||||
cd /www/wwwroot/getDomain
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
|
||||
```
|
||||
|
||||
如果当前是临时海外控制面联调,也可以把第二个地址替换成海外目标 API。
|
||||
|
||||
第一眼重点只看:
|
||||
|
||||
- `go_live_status`
|
||||
- `publish_status`
|
||||
- `blocking_reasons`
|
||||
- `warnings`
|
||||
- `next_step_action_code`
|
||||
- `operator_title`
|
||||
|
||||
如果你怀疑当前是环境本身有漂移,而不是业务链路没收口,先执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh env-audit
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `status`
|
||||
- `headline`
|
||||
- `missing_items`
|
||||
- `tooling_items`
|
||||
- `runtime.preflight_ok`
|
||||
- `runtime.readiness_status`
|
||||
- `runtime.route_surface_complete`
|
||||
- `runtime.repo_capability_drift`
|
||||
- `runtime.runtime_may_need_restart`
|
||||
- `recommended_actions`
|
||||
|
||||
如果这里出现:
|
||||
|
||||
- `runtime.repo_capability_drift=true`
|
||||
- `runtime.runtime_may_need_restart=true`
|
||||
|
||||
则优先执行 `runtime-refresh-recover`,不要先把问题误判成“代码还没写完”。这通常表示:
|
||||
|
||||
- 仓库代码已经更新
|
||||
- 但运行中的 `domaincheck-api` 进程还没重启到这版代码
|
||||
|
||||
`runtime-refresh-recover` 会固定给出:
|
||||
|
||||
- 当前是否真的属于运行时版本漂移
|
||||
- 推荐先跑的 `runtime-refresh-recover` 统一恢复入口
|
||||
- 重启后应该按什么顺序继续:
|
||||
- `stack-diagnosis`
|
||||
- `node-bootstrap-plan`
|
||||
- `stack-next`
|
||||
- `doctor-decision`
|
||||
|
||||
如果你已经不想再手工一条条执行,而是想把“刷新运行时 + 再做总检复核”压成一次操作,直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
|
||||
```
|
||||
|
||||
它会固定串起:
|
||||
|
||||
1. `runtime-refresh-recover`
|
||||
2. `stack-diagnosis summary`
|
||||
3. `stack-next`
|
||||
4. `go-live-check summary`
|
||||
5. `doctor-decision`
|
||||
|
||||
如果要把这次上线前检查直接导出成一整包证据,而不是手工复制多段终端输出,直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
|
||||
```
|
||||
|
||||
它会统一导出:
|
||||
|
||||
- `env-audit`
|
||||
- `go-live-check` 摘要
|
||||
- `go-live-summary`
|
||||
- `stack-diagnosis`
|
||||
- `driver-feed`
|
||||
- `codex-brief`
|
||||
- `release-launchpad`
|
||||
- `doctor-decision`
|
||||
- `doctor-export`
|
||||
- `manifest.json`
|
||||
|
||||
其中 `manifest.json` 会直接标记:
|
||||
|
||||
- 环境审计当前是 `ready / attention / blocked`
|
||||
- 哪些 artifact 成功
|
||||
- 哪些 artifact 超时
|
||||
- 哪些 endpoint 缺失或返回异常
|
||||
- 当前推荐的阅读顺序
|
||||
|
||||
现在 Ops Center 首屏也会同步读取最新 bundle / manifest:
|
||||
|
||||
- 顶部 `总检决策`
|
||||
- 详情区 `总检主决策详情`
|
||||
- API `GET /api/v1/ops/doctor-decision`
|
||||
- 顶部 `交付证据`
|
||||
- 详情区 `交付证据详情`
|
||||
- API `GET /api/v1/ops/go-live-bundle`
|
||||
- 顶部 `正式复核`
|
||||
- 详情区 `正式复核详情`
|
||||
- API `GET /api/v1/ops/go-live-review`
|
||||
|
||||
如果你想先拿到一句“bundle manifest 正式复核是否通过”的统一结论,而不是直接跳到最终签收,也可以先执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review
|
||||
```
|
||||
|
||||
如果你想在 bundle 基础上直接得到一句“现在能不能签字上线”的统一结论,而不是人工再看多份 JSON,直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff
|
||||
```
|
||||
|
||||
或基于已有 bundle:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff /path/to/go-live-bundle
|
||||
```
|
||||
|
||||
这里的 `go-live-signoff` 已经不是另一套独立摘要,而是固定压缩:
|
||||
|
||||
- `go-live-review`
|
||||
- `doctor-decision`
|
||||
- `go-live-summary / 发布门禁 / launchpad 摘要`
|
||||
|
||||
所以页面首屏、CLI 与海外 Codex 看到的是同一份最终签字口径。
|
||||
|
||||
第一眼重点只看:
|
||||
|
||||
- `signoff_status`
|
||||
- `headline`
|
||||
- `release_gate`
|
||||
- `blocked_reasons`
|
||||
- `attention_reasons`
|
||||
- `decision`
|
||||
|
||||
### 第 2 步:看总检详情
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `stack_status`
|
||||
- `issues`
|
||||
- `next_step`
|
||||
- `quick_commands`
|
||||
|
||||
### 第 3 步:看驾驶主线
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `top_recommendation`
|
||||
- `automation_coverage`
|
||||
- `recommended_behavior`
|
||||
- `focus`
|
||||
|
||||
### 第 4 步:确认默认下一步
|
||||
|
||||
如果只是预览:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-next
|
||||
```
|
||||
|
||||
如果已经确认要执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-next run confirm
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 五、当前项目的主处理车道
|
||||
|
||||
上线前遇到问题时,不要发散排查,先归到下面 5 条主车道:
|
||||
|
||||
### 1. 节点接管车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- `remote_access_ready=0`
|
||||
- `bootstrap_run`
|
||||
- `run_acceptance`
|
||||
- `managed_nodes_agent_pending`
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-gap-check
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh agent-gap-recover
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-onboarding http://127.0.0.1:8100 mainland-worker-01
|
||||
```
|
||||
|
||||
### 2. 远端日志车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- 首屏显示“现场日志覆盖不足”
|
||||
- `log_sync_enabled=false`
|
||||
- `line_count=0`
|
||||
- `source_nodes` 缺节点
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-check
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-recover
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 key
|
||||
```
|
||||
|
||||
### 3. 总检修复车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- `stack_status=blocked`
|
||||
- `surface_status=broken`
|
||||
- `contracts` / `launchpad` / `activity-stream` 缺口
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100
|
||||
bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100
|
||||
bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100
|
||||
```
|
||||
|
||||
### 4. 检测参与车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- 页面显示在线节点多,但真正跑检测的少
|
||||
- `claimed_by` 分布异常
|
||||
- 在线但未参与节点过多
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh activity-stream
|
||||
```
|
||||
|
||||
### 5. 发布 / 放量车道
|
||||
|
||||
典型信号:
|
||||
|
||||
- `publish_ready=true`
|
||||
- `launchpad_status` 已可进入 Worker / Control rollout
|
||||
- 默认下一步已落在 Release Hub
|
||||
|
||||
优先命令:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-focus-preview
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 六、首屏通过标准
|
||||
|
||||
Ops Center 首屏如果要算“可上线收工”,至少应满足:
|
||||
|
||||
- 首屏能同时显示:
|
||||
- 上线收口状态
|
||||
- 发布闸门状态
|
||||
- 自动化收口状态
|
||||
- 主处理车道
|
||||
- 默认下一步
|
||||
- 后端接管覆盖
|
||||
- 观察层完整度
|
||||
- 现场日志覆盖度
|
||||
- 首屏动作条可直接完成:
|
||||
- 默认下一步
|
||||
- 定位下一步
|
||||
- 看契约
|
||||
- 复制命令
|
||||
- 进入主处理面板
|
||||
- 首屏回执条能统一反馈:
|
||||
- 打开契约成功
|
||||
- 定位成功
|
||||
- 复制成功
|
||||
- 后端动作执行成功 / 警告 / 失败
|
||||
|
||||
如果这些已经稳定,就说明值班同事和海外 Codex 已经在同一个驾驶台上工作,而不是各自再拼逻辑。
|
||||
|
||||
---
|
||||
|
||||
## 七、通过标准后的最小上线动作
|
||||
|
||||
当上面门禁都通过后,建议最小上线动作顺序为:
|
||||
|
||||
1. 跑 `go-live-check`
|
||||
2. 看 `stack-diagnosis`
|
||||
3. 复核 `driver-feed.automation_coverage`
|
||||
4. 复核 `codex-brief.recommended_behavior`
|
||||
5. 若进入发布车道,先看 `release-launchpad`
|
||||
6. 对 `guarded_auto` 动作显式确认后再执行
|
||||
|
||||
上线前最后一轮建议保留证据:
|
||||
|
||||
- `go-live-summary` 输出
|
||||
- `stack-diagnosis` 输出
|
||||
- `driver-feed` 输出
|
||||
- `codex-brief` 输出
|
||||
- `release-launchpad` 输出
|
||||
- 前端构建成功记录
|
||||
|
||||
---
|
||||
|
||||
## 八、现在这套体系是否已经进入可上线状态
|
||||
|
||||
按当前仓库现状,可以认为已经具备:
|
||||
|
||||
- 单脑控制面的协议骨架
|
||||
- 首屏驾驶舱骨架
|
||||
- 总检与驾驶 contract
|
||||
- Release Hub / Driver / Node Agent 的联动骨架
|
||||
- 海外 CLI 单入口
|
||||
|
||||
但正式宣称“可以上线”,仍建议每次以这份总表复核,而不是只凭某一个页面截图或某一次联调成功就直接跳过。
|
||||
|
||||
这份文档的定位就是:
|
||||
|
||||
> 后续每次上线、值班接手、发布前复核,都先回到这里。
|
||||
|
||||
如果你已经准备进入“这次要不要真的发”的最终执行阶段,下一份应直接看:
|
||||
|
||||
- `docs/26_domainCheck_发布前运行验证与交付模板.md`
|
||||
635
docs/26_domainCheck_发布前运行验证与交付模板.md
Normal file
635
docs/26_domainCheck_发布前运行验证与交付模板.md
Normal file
@@ -0,0 +1,635 @@
|
||||
# 26 domainCheck 发布前运行验证与交付模板
|
||||
|
||||
## 一、这份模板怎么用
|
||||
|
||||
这份文档不是设计说明,而是正式发布前最后一轮执行模板。
|
||||
|
||||
适用场景:
|
||||
|
||||
- 海外单脑控制面已经搭好
|
||||
- 大陆节点已经接入或正在收口
|
||||
- 代码已经更新到待发布版本
|
||||
- 现在要做最后一轮运行验证、证据留存、交付确认
|
||||
|
||||
这份模板分成 3 段:
|
||||
|
||||
1. 发布前运行验证
|
||||
2. 上线证据导出
|
||||
3. 最终交付结论模板
|
||||
|
||||
---
|
||||
|
||||
## 二、发布前运行验证
|
||||
|
||||
### 1. 先跑统一收口摘要
|
||||
|
||||
```bash
|
||||
cd /www/wwwroot/getDomain
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `go_live_status`
|
||||
- `publish_status`
|
||||
- `blocking_reasons`
|
||||
- `warnings`
|
||||
- `next_step_action_code`
|
||||
- `operator_title`
|
||||
- `launchpad_recommended_target_node_code`
|
||||
- `launchpad_recommended_recovery_label`
|
||||
- `launchpad_recommended_recovery_summary`
|
||||
- `launchpad_onboarding_bootstrap_pending_nodes`
|
||||
- `launchpad_onboarding_acceptance_ready_nodes`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `go_live_status != blocked`
|
||||
- `publish_status != blocked`
|
||||
|
||||
### 1.5 先看环境差异是否已收口
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh env-audit
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `status`
|
||||
- `headline`
|
||||
- `python.pytest_installed / fastapi_installed / uvicorn_installed`
|
||||
- `python.runtimes`
|
||||
- `services`
|
||||
- `runtime.preflight_ok`
|
||||
- `runtime.readiness_status`
|
||||
- `runtime.route_surface_complete`
|
||||
- `missing_items`
|
||||
- `tooling_items`
|
||||
|
||||
通过标准建议:
|
||||
|
||||
- `status != blocked`
|
||||
- `runtime.preflight_ok=true`
|
||||
- `route_surface_complete=true`
|
||||
- 如果 `missing_items` 里仍有关键基础项,先补环境,再继续业务复核
|
||||
- 如果只有 `tooling_items`,可以继续上线复核,但建议后补本机工具链
|
||||
- 如果 `runtime.runtime_may_need_restart=true`
|
||||
- 不要直接继续接管或发布链
|
||||
- 先按 `runtime-refresh-recover` 给出的固定顺序完成 API 重启与复检
|
||||
- 如果你希望把这一轮“重启 + 总检 + 默认下一步 + 上线摘要”压成一次复核
|
||||
- 直接执行 `go-live-recover`
|
||||
|
||||
### 1.6 先看托管节点接入面是否清楚
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `summary.agent_ready`
|
||||
- `summary.ssh_ready`
|
||||
- `summary.remote_access_ready`
|
||||
- `summary.remote_access_state_counts`
|
||||
- 每台节点的 `agent_state`
|
||||
- 每台节点的 `remote_access_state`
|
||||
- 每台节点的 `ssh_access_label`
|
||||
- 每台节点的 `ssh_entry`
|
||||
|
||||
通过标准建议:
|
||||
|
||||
- 如果某台节点还是 `agent_pending`
|
||||
- 先确认是否已经保存 SSH 入口
|
||||
- 如果节点尚未保存 SSH 入口,但你已经决定由海外控制面统一接管
|
||||
- 先补录:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bind-ssh http://127.0.0.1:8100 mainland-worker-01 121.204.244.248 root 22
|
||||
```
|
||||
|
||||
- 补录后立即复查:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-handover http://127.0.0.1:8100 mainland-worker-01
|
||||
```
|
||||
|
||||
- 如果已经 `agent_ready`
|
||||
- 说明该节点已进入标准远端执行器接管面
|
||||
- 如果还不是 `agent_ready`,但 `ssh_ready` 已经成立
|
||||
- 说明至少已经具备海外主控经 SSH 进入节点完成 bootstrap / 验收的前置条件
|
||||
|
||||
### 2. 再看总检详情
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `stack_status`
|
||||
- `issues`
|
||||
- `next_step`
|
||||
- `quick_commands`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `stack_status != blocked`
|
||||
- `issues` 里没有未处理的关键阻断项
|
||||
|
||||
### 3. 再看驾驶主线
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `top_recommendation`
|
||||
- `automation_coverage`
|
||||
- `recommended_behavior`
|
||||
- `focus`
|
||||
- `summary.launchpad_recommended_target_node_code`
|
||||
- `summary.launchpad_recommended_recovery_label`
|
||||
- `summary.launchpad_onboarding_bootstrap_pending_nodes`
|
||||
- `summary.launchpad_onboarding_acceptance_ready_nodes`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `automation_coverage.launch_status != blocked`
|
||||
- 默认下一步已经明确
|
||||
- 如果 `summary.launchpad_recommended_target_node_code` 非空
|
||||
- 值班同事应能直接知道当前卡在哪台节点
|
||||
- 不需要再回 launchpad 明细手工翻 gap rows
|
||||
|
||||
### 4. 若涉及发布,再看 Release Hub
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `launchpad_status`
|
||||
- `recommended_action_code`
|
||||
- `default_rollout_gate_status`
|
||||
- `recommended_target_node_code`
|
||||
- `recommended_recovery_label`
|
||||
- `recommended_recovery_summary`
|
||||
- `onboarding_bootstrap_pending_nodes`
|
||||
- `onboarding_acceptance_ready_nodes`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `launchpad_status != blocked`
|
||||
- `default_rollout_gate_status != blocked`
|
||||
|
||||
特殊判读:
|
||||
|
||||
- 如果 `recommended_action_code=api-restart`
|
||||
- 不要误判成“节点没接好”
|
||||
- 这更像是运行中的控制面 API 还没刷新到当前仓库的最新运维能力
|
||||
- 应先执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
|
||||
```
|
||||
|
||||
- 然后重新跑:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||||
```
|
||||
|
||||
补充说明:
|
||||
|
||||
- 从 `latest_release.json` 直接创建 Release,或执行“基于最新包的一键智能 Rollout”之前
|
||||
- 现在不仅要求 `smoke_test_ok=true`
|
||||
- 还要求当前 `final_release_report.json` 与最新包完全匹配,且 `final_release_ok=true`
|
||||
- 如果页面按钮被禁用,或 API 返回“最新发布包尚未完成最终签收”
|
||||
- 优先检查 `final_release_report_stale_reason`
|
||||
- 再检查 `final_release_gate_decision / final_release_gate_blocked_reasons`
|
||||
|
||||
### 5. 若涉及节点接管,补跑节点验收
|
||||
|
||||
如果这次发布包含:
|
||||
|
||||
- 新增大陆节点
|
||||
- 重新接管旧节点
|
||||
- 切换到 Node Agent 主接管模式
|
||||
|
||||
则不能只看总检摘要,还要对目标节点逐台完成验收。
|
||||
|
||||
先看验收计划:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-plan http://127.0.0.1:8100 mainland-worker-01
|
||||
```
|
||||
|
||||
确认无误后执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-worker-01 cli/acceptance
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `node_code`
|
||||
- `status`
|
||||
- `checks`
|
||||
- `blocking_issues`
|
||||
- `warnings`
|
||||
- `recommended_next_step`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `status != blocked`
|
||||
- 没有未处理的 `blocking_issues`
|
||||
- 目标节点已经不再停留在“bootstrap pending / acceptance pending”
|
||||
|
||||
### 6. 若涉及远端执行,补跑日志回传验证
|
||||
|
||||
如果当前版本准备进入:
|
||||
|
||||
- 多机值班
|
||||
- 海外单脑统一接管
|
||||
- 远端动作回执与现场日志复核
|
||||
|
||||
则需要确认日志回传链已经可用,而不是只看节点在线。
|
||||
|
||||
先检查:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-check
|
||||
```
|
||||
|
||||
如果存在缺口,再执行恢复:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-recover http://127.0.0.1:8100 key confirm cli/log-sync
|
||||
```
|
||||
|
||||
必要时抓一份节点现场日志:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 full
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `log_sync_enabled`
|
||||
- `line_count`
|
||||
- `source_nodes`
|
||||
- `missing_nodes`
|
||||
- `scene_log.status`
|
||||
|
||||
通过标准:
|
||||
|
||||
- 关键节点已出现在 `source_nodes`
|
||||
- `missing_nodes` 不包含当前发布主车道节点
|
||||
- 如需人工复核现场,`scene-node-log` 能取到有效日志
|
||||
|
||||
### 7. 若涉及 Node Agent 交付,补看队列健康
|
||||
|
||||
如果这轮发布已经进入“控制面发动作、节点拉任务、节点回执结果”的模式,就必须确认 delivery queue 没有假健康。
|
||||
|
||||
先看队列摘要:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-status http://127.0.0.1:8100 mainland-worker-01
|
||||
```
|
||||
|
||||
如果怀疑有堆积或死信,再看明细:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-records http://127.0.0.1:8100 mainland-worker-01 dead_letter
|
||||
```
|
||||
|
||||
如需重放:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-replay http://127.0.0.1:8100 mainland-worker-01 20 cli/queue-replay
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `delivery_queue.summary`
|
||||
- `dead_letter`
|
||||
- `retrying`
|
||||
- `pending`
|
||||
- `replayed_count`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `dead_letter=0`
|
||||
- 没有持续增长的 `retrying`
|
||||
- 队列不处于异常堆积状态
|
||||
|
||||
注意:
|
||||
|
||||
- `dead_letter > 0` 时,不应宣称“可正式发布”
|
||||
- 这种情况至少按 `publish_status = blocked` 处理
|
||||
|
||||
### 8. 最后跑一次 doctor 结论
|
||||
|
||||
前面的检查是分面复核,最后还要再收成一句话,让交班、值班、Codex 驾驶员看到同一结论。
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision
|
||||
```
|
||||
|
||||
记录结果:
|
||||
|
||||
- `status`
|
||||
- `decision`
|
||||
- `recommended_commands`
|
||||
- `recommended_reading_order`
|
||||
- `launchpad_alignment`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `status != blocked`
|
||||
- `decision` 与前面的 `go-live-check / stack-diagnosis / release-launchpad` 不冲突
|
||||
- `recommended_commands` 已经指向明确的下一跳,而不是泛化排查
|
||||
|
||||
### 9. 若希望直接生成“可否签字上线”的最终摘要
|
||||
|
||||
如果你不想手工再拼:
|
||||
|
||||
- bundle review
|
||||
- doctor 结论
|
||||
- 发布门禁
|
||||
|
||||
可以直接执行:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff
|
||||
```
|
||||
|
||||
如果已经有现成证据包,也可以直接基于 manifest 输出:
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff /path/to/go-live-bundle
|
||||
```
|
||||
|
||||
这一步现在不是“另一套单独判断”,而是固定复用:
|
||||
|
||||
- `go-live-review`
|
||||
- `doctor-decision`
|
||||
- `go-live-summary / 发布门禁 / launchpad 摘要`
|
||||
|
||||
也就是说,页面首屏、CLI 和海外 Codex 看到的最终签收结论,已经开始共享同一份签字口径。
|
||||
|
||||
重点只看:
|
||||
|
||||
- `signoff_status`
|
||||
- `headline`
|
||||
- `release_gate`
|
||||
- `blocked_reasons`
|
||||
- `attention_reasons`
|
||||
- `decision`
|
||||
- `recommended_commands`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `signoff_status=ready`
|
||||
- 可进入最终人工签字或正式发布
|
||||
- `signoff_status=attention`
|
||||
- 说明现场已经接近完成,但仍应先清 attention 项
|
||||
- `signoff_status=blocked`
|
||||
- 本轮不应宣称收口完成
|
||||
|
||||
---
|
||||
|
||||
## 三、上线证据导出
|
||||
|
||||
### 1. 导出 bundle
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
|
||||
```
|
||||
|
||||
这会生成一整包上线前证据。
|
||||
|
||||
建议至少保留:
|
||||
|
||||
- `00_env_audit.txt`
|
||||
- `01_go_live_check_summary.txt`
|
||||
- `02_go_live_summary.json`
|
||||
- `03_stack_diagnosis.json`
|
||||
- `04_driver_feed.json`
|
||||
- `05_codex_brief.json`
|
||||
- `06_release_launchpad.json`
|
||||
- `manifest.json`
|
||||
|
||||
如果当前轮次涉及节点接管,建议额外保留:
|
||||
|
||||
- `agent_gap_export.json`
|
||||
- `node_bootstrap_plan.txt`
|
||||
- `node_acceptance_plan.json`
|
||||
|
||||
如果当前轮次涉及远端执行与日志回传,建议额外保留:
|
||||
|
||||
- `scene_log_*.txt`
|
||||
- `doctor_decision.json`
|
||||
|
||||
### 2. 直接复核 bundle
|
||||
|
||||
```bash
|
||||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review
|
||||
```
|
||||
|
||||
如果你已经切到 Ops Center 页面,也可以直接看:
|
||||
|
||||
- 首屏 `总检决策`
|
||||
- `总检主决策详情`
|
||||
- API `GET /api/v1/ops/doctor-decision`
|
||||
- 首屏 `交付证据`
|
||||
- `交付证据详情`
|
||||
- API `GET /api/v1/ops/go-live-bundle`
|
||||
- 首屏 `正式复核`
|
||||
- `正式复核详情`
|
||||
- API `GET /api/v1/ops/go-live-review`
|
||||
|
||||
记录结果:
|
||||
|
||||
- `status`
|
||||
- `headline`
|
||||
- `critical_failures`
|
||||
- `noncritical_failures`
|
||||
- `launchpad_alignment.consistent`
|
||||
- `env_audit.status`
|
||||
- `env_audit.missing_items`
|
||||
- `recommended_next_steps`
|
||||
|
||||
通过标准建议:
|
||||
|
||||
- `status=ready`
|
||||
- 可进入最终人工确认或正式发布
|
||||
- `status=attention`
|
||||
- 核心报告已齐,但仍需复核补充失败项或环境缺口
|
||||
- 也包括 `go_live_summary / stack_diagnosis / driver_feed / codex_brief` 的 launchpad 摘要出现不一致
|
||||
- `status=blocked`
|
||||
- 关键报告缺失,或环境审计已经判定阻断,不应直接宣称收口完成
|
||||
|
||||
如果 `env_audit.missing_items` 里出现:
|
||||
|
||||
- `missing_env:/etc/default/domaincheck-node-agent`
|
||||
|
||||
则这次 bundle review 已经不是泛泛地提示“环境缺口”,而是明确指向 Node Agent 接管还未收口。此时按下面顺序推进:
|
||||
|
||||
1. `agent-gap-export`
|
||||
2. `node-bootstrap-plan`
|
||||
3. 如果 summary 已提示 `runtime_may_need_restart=true`
|
||||
4. 完成恢复后重新导出 bundle,再做一次 `go-live-review`
|
||||
|
||||
### 3. 生成最终交付包
|
||||
|
||||
运行验证、bundle 复核都通过后,再生成最终交付包。
|
||||
|
||||
如果当前机器已经具备 smoke 运行条件,直接执行:
|
||||
|
||||
```bash
|
||||
bash ./prepare_final_release.sh
|
||||
```
|
||||
|
||||
如果当前机器只适合先做离线打包,可先跳过 smoke:
|
||||
|
||||
```bash
|
||||
DOMAINCHECK_SKIP_SMOKE_TEST=1 bash ./package_domain_release.sh
|
||||
bash ./verify_domain_release.sh
|
||||
```
|
||||
|
||||
后续再回到具备运行条件的机器上执行:
|
||||
|
||||
```bash
|
||||
bash ./prepare_final_release.sh
|
||||
```
|
||||
|
||||
最后统一查看最新交付结果:
|
||||
|
||||
```bash
|
||||
bash ./show_latest_release.sh
|
||||
```
|
||||
|
||||
重点只看:
|
||||
|
||||
- `latest_release.package_name`
|
||||
- `latest_release.archive_path`
|
||||
- `latest_release.sha256`
|
||||
- `latest_release.smoke_test_ok`
|
||||
- `final_release_ok`
|
||||
- `final_release_report`
|
||||
- `final_release_report_ok`
|
||||
- `final_release_report_matches_latest`
|
||||
- `final_release_report_stale_reason`
|
||||
- `final_release_gate_decision`
|
||||
- `final_release_gate_blocked_reasons`
|
||||
- `final_release_gate_verify_ok`
|
||||
- `final_release_gate_smoke_test_ok`
|
||||
|
||||
通过标准:
|
||||
|
||||
- `verify_domain_release.sh` 返回 `ok=true`
|
||||
- `latest_release.smoke_test_ok=true`
|
||||
- `final_release_ok=true`
|
||||
- `final_release_report_matches_latest=true`
|
||||
- `final_release_gate_decision=ready`
|
||||
- `final_release_gate_verify_ok=true`
|
||||
- `final_release_gate_smoke_test_ok=true`
|
||||
- `final_release_report` 已指向最新一轮生成的正式报告
|
||||
4. 先执行 `runtime-refresh-recover`
|
||||
5. 再重新导出 bootstrap plan
|
||||
|
||||
---
|
||||
|
||||
## 四、最终交付结论模板
|
||||
|
||||
下面这段可以直接作为正式发布前的交付结论模板使用。
|
||||
|
||||
### 模板正文
|
||||
|
||||
```text
|
||||
本轮发布前运行验证已完成。
|
||||
|
||||
一、统一收口结论
|
||||
- go_live_status: <填写>
|
||||
- publish_status: <填写>
|
||||
- stack_status: <填写>
|
||||
- automation_coverage.launch_status: <填写>
|
||||
- launchpad_status: <填写>
|
||||
|
||||
二、默认下一步
|
||||
- next_step_action_code: <填写>
|
||||
- operator_title: <填写>
|
||||
- launchpad_recommended_target_node_code: <填写>
|
||||
- launchpad_recommended_recovery_label: <填写>
|
||||
|
||||
三、上线证据包
|
||||
- bundle 路径: <填写>
|
||||
- manifest 复核状态: <填写 ready/attention/blocked>
|
||||
- go-live-review.headline: <填写>
|
||||
- manifest.review_status_hint: <填写>
|
||||
- critical_failures: <填写>
|
||||
- noncritical_failures: <填写>
|
||||
- manifest.launchpad_recommended_target_node_code: <填写>
|
||||
- manifest.launchpad_recommended_recovery_label: <填写>
|
||||
- manifest.launchpad_onboarding_bootstrap_pending_nodes: <填写>
|
||||
- manifest.launchpad_onboarding_acceptance_ready_nodes: <填写>
|
||||
- manifest.launchpad_alignment.consistent: <填写 true/false>
|
||||
|
||||
四、结论
|
||||
- <填写:可进入正式发布 / 仍需继续收口 / 暂不可发布>
|
||||
```
|
||||
|
||||
### 模板补充说明
|
||||
|
||||
填写“可进入正式发布”前,建议再做一次人工交叉确认:
|
||||
|
||||
- 发布门禁是否为 `ready` 或至少非 `blocked`
|
||||
- 当前主车道是否已经明确
|
||||
- 是否还存在 `dead_letter`
|
||||
- 是否还有节点停留在 `bootstrap pending / acceptance pending`
|
||||
- 是否已经保留可复盘的证据包与 manifest
|
||||
|
||||
---
|
||||
|
||||
## 五、交付给值班同事时最少要带什么
|
||||
|
||||
最少带下面 4 样:
|
||||
|
||||
1. `go-live-check` 摘要
|
||||
2. `stack-diagnosis` 结果
|
||||
3. `driver-feed / codex-brief` 结果
|
||||
4. `go-live-export` 生成的 `manifest.json`
|
||||
|
||||
如果值班同事需要直接接手节点问题,再多带 3 样:
|
||||
|
||||
1. `node-acceptance-run` 最近结果
|
||||
2. `log-sync-check` 最近结果
|
||||
3. `queue-status` 或 `queue-records dead_letter` 最近结果
|
||||
|
||||
这样交班的人不需要重新从零拼现场。
|
||||
|
||||
---
|
||||
|
||||
## 六、真正的收工标准
|
||||
|
||||
如果要算“这次版本已经可以收工”,建议至少同时满足:
|
||||
|
||||
- 前端构建成功
|
||||
- 海外单脑驾驶舱首屏状态正常
|
||||
- `go-live-review.status != blocked`
|
||||
- 发布门禁不为 `blocked`
|
||||
- Node Agent 目标节点已完成接管验收
|
||||
- 远端日志回传链可用或已确认本轮不依赖
|
||||
- delivery queue 不存在未处理 `dead_letter`
|
||||
- 当前默认下一步已经不是“继续补基础设施”,而是进入正式发布 / 观察 / 验收车道
|
||||
|
||||
到这一步,才适合说:
|
||||
|
||||
> 当前项目已经进入可上线、可交付、可持续运维状态。
|
||||
816
docs/schemas/ops_agent_protocol.md
Normal file
816
docs/schemas/ops_agent_protocol.md
Normal file
@@ -0,0 +1,816 @@
|
||||
# domainCheck Ops Agent Protocol
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 `Node Agent <-> Overseas Control Plane` 的正式 contract。
|
||||
|
||||
适用范围:
|
||||
|
||||
- `POST /api/v1/ops/agent/tokens`
|
||||
- `POST /api/v1/ops/agent/bootstrap-plan`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/handover`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/handover/bootstrap-plan`
|
||||
- `POST /api/v1/ops/agent/register`
|
||||
- `POST /api/v1/ops/agent/heartbeat`
|
||||
- `POST /api/v1/ops/agent/pull`
|
||||
- `POST /api/v1/ops/agent/jobs/{job_id}/start`
|
||||
- `POST /api/v1/ops/agent/jobs/{job_id}/complete`
|
||||
- `POST /api/v1/ops/agent/jobs/{job_id}/events`
|
||||
|
||||
设计原则:
|
||||
|
||||
- Agent 只执行声明过的结构化动作
|
||||
- 控制面负责调度、门禁、编排和审计
|
||||
- 所有回执都必须可重放、可去重、可死信化
|
||||
|
||||
---
|
||||
|
||||
## 2. 公共响应包裹
|
||||
|
||||
所有接口统一返回:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": 0,
|
||||
"message": "ok",
|
||||
"detail_code": "",
|
||||
"data": {}
|
||||
}
|
||||
```
|
||||
|
||||
约束:
|
||||
|
||||
- `code=0` 表示成功
|
||||
- `code=1` 表示业务失败
|
||||
- `detail_code` 用于结构化错误语义
|
||||
- `message` 只承担可读说明,不承担机器判断主逻辑
|
||||
|
||||
常见 `detail_code`:
|
||||
|
||||
- `agent_token_invalid`
|
||||
- `agent_token_expired`
|
||||
- `agent_token_node_mismatch`
|
||||
- `ops_job_not_owned_by_agent`
|
||||
- `ops_job_invalid_status_transition`
|
||||
- `release_checksum_mismatch`
|
||||
- `release_health_check_failed`
|
||||
|
||||
---
|
||||
|
||||
## 3. 认证与请求头
|
||||
|
||||
Agent 相关接口统一使用:
|
||||
|
||||
- `Content-Type: application/json`
|
||||
- `X-Domaincheck-Agent-Token: <token>`
|
||||
|
||||
其中:
|
||||
|
||||
- `tokens`
|
||||
- `bootstrap-plan`
|
||||
|
||||
不要求 Agent Token。
|
||||
|
||||
其余接口都要求 Token。
|
||||
|
||||
---
|
||||
|
||||
## 4. 公共节点载荷
|
||||
|
||||
`register` 与 `heartbeat` 当前共享 `_base_payload()`。
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"title": "mainland-worker-02",
|
||||
"hostname": "host-a",
|
||||
"ip": "10.0.0.12",
|
||||
"agent_version": "0.1.0",
|
||||
"capabilities": [
|
||||
"service.start",
|
||||
"service.stop",
|
||||
"service.restart",
|
||||
"service.status",
|
||||
"health.snapshot",
|
||||
"logs.collect",
|
||||
"diagnostics.collect",
|
||||
"deploy.release"
|
||||
],
|
||||
"labels": {},
|
||||
"metadata": {
|
||||
"service_names": {
|
||||
"api": "domaincheck-api",
|
||||
"worker": "domaincheck-worker",
|
||||
"sync_agent": "domaincheck-sync-agent",
|
||||
"node_agent": "domaincheck-node-agent"
|
||||
},
|
||||
"delivery_queue": {
|
||||
"state": "healthy",
|
||||
"label": "正常",
|
||||
"reason": "当前没有待重试回执,也没有死信记录。",
|
||||
"pending_count": 0,
|
||||
"dead_letter_count": 0,
|
||||
"last_flush_at": "",
|
||||
"oldest_pending_at": "",
|
||||
"oldest_pending_request_id": "",
|
||||
"oldest_pending_kind": "",
|
||||
"oldest_dead_letter_at": "",
|
||||
"oldest_dead_letter_request_id": "",
|
||||
"oldest_dead_letter_kind": ""
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
说明:
|
||||
|
||||
- `service_names` 是控制面识别节点服务拓扑的最小入口
|
||||
- `delivery_queue` 是控制面识别 Agent 回执现场的最小入口
|
||||
|
||||
---
|
||||
|
||||
## 5. Bootstrap Contract
|
||||
|
||||
### 5.1 Issue Token
|
||||
|
||||
`POST /api/v1/ops/agent/tokens`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"issued_by": "web-ui",
|
||||
"expires_in_hours": 72,
|
||||
"metadata": {
|
||||
"source": "ops-center"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
返回体关键字段:
|
||||
|
||||
- `token`
|
||||
- `token_preview`
|
||||
- `record_id`
|
||||
- `node_code`
|
||||
- `expires_at`
|
||||
- `created_at`
|
||||
|
||||
### 5.2 Bootstrap Plan
|
||||
|
||||
`POST /api/v1/ops/agent/bootstrap-plan`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"node_region": "mainland",
|
||||
"node_role": "worker",
|
||||
"issued_by": "web-ui",
|
||||
"expires_in_hours": 72,
|
||||
"control_plane_base_url": "https://ops.example.com",
|
||||
"root_dir": "/opt/domaincheck",
|
||||
"metadata": {
|
||||
"source": "ops-center"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
返回体关键字段:
|
||||
|
||||
- Token 相关字段
|
||||
- `control_plane_base_url`
|
||||
- `bootstrap_plan.node_code`
|
||||
- `bootstrap_plan.node_region`
|
||||
- `bootstrap_plan.node_role`
|
||||
- `bootstrap_plan.root_dir`
|
||||
- `bootstrap_plan.env_file`
|
||||
- `bootstrap_plan.service_name`
|
||||
- `bootstrap_plan.service_file`
|
||||
- `bootstrap_plan.install_script_path`
|
||||
- `bootstrap_plan.env_content`
|
||||
- `bootstrap_plan.command_lines`
|
||||
- `bootstrap_plan.command_block`
|
||||
- `bootstrap_plan.health_checks`
|
||||
- `bootstrap_plan.health_check_block`
|
||||
- `bootstrap_plan.bootstrap_script_name`
|
||||
- `bootstrap_plan.bootstrap_script_path`
|
||||
- `bootstrap_plan.bootstrap_script_content`
|
||||
- `bootstrap_plan.bootstrap_run_script_block`
|
||||
|
||||
约束:
|
||||
|
||||
- `bootstrap_plan` 是标准接入方案对象
|
||||
- 页面、Codex、CLI 必须复用这一个对象
|
||||
- 不允许各端再次手写 env 或 shell 逻辑
|
||||
|
||||
---
|
||||
|
||||
## 6. Runtime Contract
|
||||
|
||||
### 6.1 Register
|
||||
|
||||
`POST /api/v1/ops/agent/register`
|
||||
|
||||
语义:
|
||||
|
||||
- 校验 Token 与 `node_code`
|
||||
- Upsert Agent 运行态
|
||||
- 将节点推入托管目录候选态
|
||||
|
||||
最小成功返回字段:
|
||||
|
||||
- `node_code`
|
||||
- `expires_at`
|
||||
- `capabilities`
|
||||
|
||||
### 6.2 Heartbeat
|
||||
|
||||
`POST /api/v1/ops/agent/heartbeat`
|
||||
|
||||
语义:
|
||||
|
||||
- 刷新 `last_seen_at`
|
||||
- 刷新节点身份、服务名和队列快照
|
||||
- 维持控制面中的 Agent 在线态
|
||||
|
||||
最小成功返回字段:
|
||||
|
||||
- `node_code`
|
||||
- `server_time`
|
||||
- `expires_at`
|
||||
|
||||
### 6.3 Pull
|
||||
|
||||
`POST /api/v1/ops/agent/pull?limit=1`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02"
|
||||
}
|
||||
```
|
||||
|
||||
返回体关键字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"id": 123,
|
||||
"job_code": "ops-xxx",
|
||||
"action": "health.snapshot",
|
||||
"target_node_code": "mainland-worker-02",
|
||||
"status": "dispatching",
|
||||
"execution_mode": "remote-agent",
|
||||
"payload": {},
|
||||
"metadata": {},
|
||||
"policy": {},
|
||||
"steps": []
|
||||
}
|
||||
],
|
||||
"count": 1
|
||||
}
|
||||
```
|
||||
|
||||
约束:
|
||||
|
||||
- Agent 只能拿到属于自己的任务
|
||||
- Agent 不负责选择任务
|
||||
- 调度权始终在控制面
|
||||
|
||||
### 6.3.1 Agent Job Envelope
|
||||
|
||||
为了避免 Agent 再从自然语言动作名里“猜执行上下文”,`pull.data.jobs[]` 当前已经按统一任务包裹返回。
|
||||
|
||||
当前 `pull.data` 顶层也会额外带:
|
||||
|
||||
- `protocol_version`
|
||||
- `envelope_type`
|
||||
- `node_code`
|
||||
- `limit`
|
||||
- `count`
|
||||
|
||||
推荐最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"protocol_version": "ops-agent/v1",
|
||||
"envelope_type": "agent_job",
|
||||
"id": 123,
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-001",
|
||||
"job_type": "ops_action",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-02",
|
||||
"status": "dispatching",
|
||||
"execution_mode": "remote-agent",
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志",
|
||||
"job_ref": {
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-001",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-02"
|
||||
},
|
||||
"step_ref": {
|
||||
"step_id": 456,
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志"
|
||||
},
|
||||
"payload": {
|
||||
"service_name": "domaincheck-worker",
|
||||
"lines": 120,
|
||||
"include_agent_logs": false
|
||||
},
|
||||
"policy": {
|
||||
"timeout_seconds": 120,
|
||||
"stop_on_failure": true,
|
||||
"auto_approve": true
|
||||
},
|
||||
"metadata": {
|
||||
"requested_by": "playbook:inspection.standard",
|
||||
"source": "ops-center"
|
||||
},
|
||||
"release_context": {
|
||||
"release_id": 0,
|
||||
"release_version": "",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": ""
|
||||
},
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-001",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-02"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- Agent 必须优先消费:
|
||||
- `action`
|
||||
- `payload`
|
||||
- `policy`
|
||||
- Agent 不应通过:
|
||||
- `step_title`
|
||||
- `summary`
|
||||
- `notes`
|
||||
去反推真实执行参数
|
||||
- 与发布相关的任务应通过:
|
||||
- `release_context`
|
||||
传递 release / rollout 关联,而不是让 Agent 自己查询“当前版本”
|
||||
|
||||
这层的意义是:
|
||||
|
||||
- `ops job`
|
||||
- 仍是正式执行颗粒度
|
||||
- `Agent Job Envelope`
|
||||
- 是 Node Agent 拿到的稳定执行视图
|
||||
|
||||
这样同一套 Agent 才能同时承接:
|
||||
|
||||
- 标准巡检
|
||||
- 日志收集
|
||||
- 诊断采样
|
||||
- Release 部署
|
||||
- Rollout 回滚
|
||||
|
||||
### 6.4 Start
|
||||
|
||||
`POST /api/v1/ops/agent/jobs/{job_id}/start`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02"
|
||||
}
|
||||
```
|
||||
|
||||
语义:
|
||||
|
||||
- `dispatching` 表示已派发
|
||||
- `running` 表示节点已真实开工
|
||||
|
||||
### 6.5 Complete
|
||||
|
||||
`POST /api/v1/ops/agent/jobs/{job_id}/complete`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"client_request_id": "complete-ops-123-1",
|
||||
"status": "success",
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
"result": {
|
||||
"summary": "ok"
|
||||
},
|
||||
"error_message": ""
|
||||
}
|
||||
```
|
||||
|
||||
允许状态:
|
||||
|
||||
- `success`
|
||||
- `failed`
|
||||
- `partially_succeeded`
|
||||
|
||||
幂等约束:
|
||||
|
||||
- 使用 `client_request_id`
|
||||
- 控制面写入 `ops_jobs.last_agent_complete_request_id`
|
||||
- 同一完成回执可安全重放,不应再次制造副作用
|
||||
|
||||
推荐补充字段:
|
||||
|
||||
- `duration_ms`
|
||||
- `result.summary_text`
|
||||
- `result.focus_ref`
|
||||
|
||||
当前成功返回里也会额外带:
|
||||
|
||||
- `completion_summary.job_id`
|
||||
- `completion_summary.job_code`
|
||||
- `completion_summary.status`
|
||||
- `completion_summary.status_label`
|
||||
- `completion_summary.result_summary_text`
|
||||
- `completion_summary.focus_ref`
|
||||
- `completion_summary.deduplicated`
|
||||
|
||||
这样控制面在不展开 stdout/stderr 的情况下,也能直接把:
|
||||
|
||||
- 任务收口摘要
|
||||
- 后续跳转落点
|
||||
|
||||
并入 activity / inspection / rollout 观察面。
|
||||
|
||||
### 6.6 Events
|
||||
|
||||
`POST /api/v1/ops/agent/jobs/{job_id}/events`
|
||||
|
||||
请求体最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"client_event_id": "event-ops-123-checksum-1",
|
||||
"step_id": 0,
|
||||
"event_type": "deploy_checksum_verified",
|
||||
"level": "info",
|
||||
"message": "checksum ok",
|
||||
"payload": {
|
||||
"checksum": "sha256:..."
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
去重约束:
|
||||
|
||||
- 使用 `client_event_id`
|
||||
- 控制面通过唯一索引去重
|
||||
- 同一事件可安全重放,不应重复落库
|
||||
|
||||
推荐补充字段:
|
||||
|
||||
- `occurred_at`
|
||||
- `summary_text`
|
||||
- `focus_ref`
|
||||
|
||||
当前成功返回里也会额外带:
|
||||
|
||||
- `event`
|
||||
- 即标准化后的事件对象
|
||||
- 内含 `event_key / summary_text / occurred_at / focus_ref`
|
||||
|
||||
这样事件流就不再只是“原始日志点”,而能直接成为:
|
||||
|
||||
- playbook run events
|
||||
- rollout events
|
||||
- activity-stream
|
||||
|
||||
共同消费的现场片段。
|
||||
|
||||
---
|
||||
|
||||
## 7. Agent 本地补发队列
|
||||
|
||||
当前主链路已经正式具备:
|
||||
|
||||
- `pending` 队列
|
||||
- `dead-letter` 队列
|
||||
- 定时 `_flush_delivery_queue()`
|
||||
- 永久性错误转死信
|
||||
|
||||
当前重点覆盖:
|
||||
|
||||
- `jobs/{id}/complete`
|
||||
- `jobs/{id}/events`
|
||||
|
||||
队列状态:
|
||||
|
||||
- `healthy`
|
||||
- `retrying`
|
||||
- `dead_letter`
|
||||
|
||||
语义:
|
||||
|
||||
- `healthy`:无积压,无死信
|
||||
- `retrying`:存在待补发回执,Agent 会继续自动重放
|
||||
- `dead_letter`:存在永久失败记录,需人工介入
|
||||
|
||||
控制面消费方式:
|
||||
|
||||
- 节点维度:
|
||||
- `delivery_queue_state`
|
||||
- `delivery_queue_label`
|
||||
- `delivery_queue_reason`
|
||||
- `delivery_queue_pending_count`
|
||||
- `delivery_queue_dead_letter_count`
|
||||
- 汇总维度:
|
||||
- `queue_retrying_nodes`
|
||||
- `queue_dead_letter_nodes`
|
||||
- `queue_pending_records`
|
||||
- `queue_dead_letter_records`
|
||||
|
||||
### 7.1 当前控制面已可见的最小现场
|
||||
|
||||
当前至少已经可以通过:
|
||||
|
||||
- `GET /api/v1/ops/nodes`
|
||||
|
||||
看到每台托管节点的队列现场:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"delivery_queue_state": "dead_letter",
|
||||
"delivery_queue_label": "死信 2",
|
||||
"delivery_queue_reason": "当前存在 2 条死信记录,建议优先查看节点日志或诊断编排。",
|
||||
"delivery_queue_pending_count": 0,
|
||||
"delivery_queue_dead_letter_count": 2,
|
||||
"delivery_queue_last_flush_at": "2026-04-18 06:10:00",
|
||||
"delivery_queue_oldest_pending_at": "",
|
||||
"delivery_queue_oldest_pending_request_id": "",
|
||||
"delivery_queue_oldest_pending_kind": "",
|
||||
"delivery_queue_oldest_dead_letter_at": "2026-04-18 05:59:00",
|
||||
"delivery_queue_oldest_dead_letter_request_id": "complete-ops-123-1",
|
||||
"delivery_queue_oldest_dead_letter_kind": "job_complete"
|
||||
}
|
||||
```
|
||||
|
||||
这已经足够让:
|
||||
|
||||
- 页面显示“节点存在死信”
|
||||
- `overview / driver-feed / codex-brief` 产出驾驶建议
|
||||
- CLI 快速判断现场是不是要先走日志或诊断
|
||||
|
||||
但这还不够支撑正式运维,因为它只能“看见死信”,还不能“处理死信”。
|
||||
|
||||
### 7.2 Delivery Queue Record 正式对象
|
||||
|
||||
后续控制面不应再把死信理解成一个纯计数器,而要把每条待补发 / 死信记录升级成正式对象。
|
||||
|
||||
推荐最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"record_id": "dq-mainland-worker-02-20260418-001",
|
||||
"node_code": "mainland-worker-02",
|
||||
"state": "dead_letter",
|
||||
"request_kind": "job_complete",
|
||||
"client_request_id": "complete-ops-123-1",
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-001",
|
||||
"target_api": "/api/v1/ops/agent/jobs/123/complete",
|
||||
"detail_code": "ops_job_invalid_status_transition",
|
||||
"error_message": "当前任务状态不允许 complete",
|
||||
"attempt_count": 6,
|
||||
"first_queued_at": "2026-04-18 05:58:00",
|
||||
"last_attempt_at": "2026-04-18 05:59:00",
|
||||
"next_retry_at": "",
|
||||
"payload_preview": {
|
||||
"status": "success"
|
||||
},
|
||||
"response_preview": {
|
||||
"code": 1,
|
||||
"detail_code": "ops_job_invalid_status_transition"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式约束:
|
||||
|
||||
- `record_id`
|
||||
- 是控制面侧唯一主键
|
||||
- `client_request_id`
|
||||
- 是 Agent 幂等键
|
||||
- `detail_code`
|
||||
- 是死信聚类主键,不能只靠 `message`
|
||||
- `payload_preview / response_preview`
|
||||
- 用于页面和 Codex 快速判断,不必默认展开完整原始报文
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `pending`
|
||||
- `retrying`
|
||||
- `dead_letter`
|
||||
- `replayed`
|
||||
- `discarded`
|
||||
|
||||
### 7.3 死信操作面正式入口
|
||||
|
||||
为了让“看见死信”之后不用回节点本地处理,推荐把操作面固定成下面这组接口。
|
||||
|
||||
当前已落地的第一阶段能力:
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||||
|
||||
当前控制面返回的记录可见性为:
|
||||
|
||||
- `record_visibility=head_only`
|
||||
|
||||
也就是:
|
||||
|
||||
- 当前已经能稳定查看节点级队列总览
|
||||
- 也能看到 `pending / dead_letter` 的头部记录摘要
|
||||
- 但还没有把节点本地全量 `pending/*.json / dead-letter/*.json` 正式同步到控制面
|
||||
|
||||
所以这两条 GET 入口现在属于“正式只读观察面”,而不是完整操作面。
|
||||
|
||||
#### a. 单节点队列总览
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"summary": {
|
||||
"state": "dead_letter",
|
||||
"pending_count": 0,
|
||||
"dead_letter_count": 2,
|
||||
"last_flush_at": "2026-04-18 06:10:00"
|
||||
},
|
||||
"oldest_pending_record": {},
|
||||
"oldest_dead_letter_record": {}
|
||||
}
|
||||
```
|
||||
|
||||
#### b. 队列记录列表
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||||
|
||||
推荐筛选参数:
|
||||
|
||||
- `state=pending|retrying|dead_letter|replayed|discarded`
|
||||
- `request_kind=job_complete|job_event`
|
||||
- `detail_code=...`
|
||||
- `group_by=detail_code|request_kind|target_api`
|
||||
- `limit=50`
|
||||
|
||||
当带 `group_by` 时,返回值应优先给出聚类摘要,而不是只返回平铺明细。
|
||||
|
||||
#### c. 单条死信重放
|
||||
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay`
|
||||
|
||||
当前已落地,但要明确当前阶段约束:
|
||||
|
||||
- 控制面仍然是 `record_visibility=head_only`
|
||||
- 所以单条动作目前只允许针对“当前可见头部记录”
|
||||
- 真正的执行不是控制面直接改远端文件,而是创建正式 `ops job`
|
||||
- 由目标节点 `Node Agent` 执行 `delivery.queue.replay`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"requested_by": "web-ui",
|
||||
"reason": "确认任务状态已修复,重放这条 complete 回执"
|
||||
}
|
||||
```
|
||||
|
||||
#### d. 批量重放
|
||||
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/replay`
|
||||
|
||||
当前已落地为正式操作面。
|
||||
|
||||
推荐请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"requested_by": "codex",
|
||||
"selector": {
|
||||
"state": "dead_letter",
|
||||
"detail_code": "ops_job_invalid_status_transition"
|
||||
},
|
||||
"limit": 20
|
||||
}
|
||||
```
|
||||
|
||||
#### e. 丢弃死信
|
||||
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard`
|
||||
|
||||
当前已落地,但与单条重放一样,仍然只允许针对当前可见头部死信记录发起单条动作。
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"discarded_by": "web-ui",
|
||||
"reason": "确认这条回执不再需要补发"
|
||||
}
|
||||
```
|
||||
|
||||
#### f. 主动冲刷队列
|
||||
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/flush`
|
||||
|
||||
当前已落地为正式操作面,执行方式同样是:
|
||||
|
||||
- 控制面创建 `ops job`
|
||||
- 目标节点 `Node Agent` 执行 `delivery.queue.flush`
|
||||
|
||||
语义:
|
||||
|
||||
- 不是“强行成功”
|
||||
- 而是立即触发一次 Agent 补发冲刷
|
||||
- 结果仍然回到 `pending / retrying / dead_letter / replayed`
|
||||
|
||||
### 7.4 为什么必须做成正式操作面
|
||||
|
||||
这层不是为了让页面多一个按钮,而是为了保证:
|
||||
|
||||
- 页面可以处理死信
|
||||
- CLI 可以处理死信
|
||||
- 海外 Codex 驾驶员可以处理死信
|
||||
|
||||
三者都不需要绕回:
|
||||
|
||||
- 手工 SSH 登录节点
|
||||
- 手工删本地文件
|
||||
- 手工重放某条 HTTP 请求
|
||||
|
||||
正式规则应该固定成:
|
||||
|
||||
- “死信查看” 走控制面对象
|
||||
- “死信重放 / 丢弃 / 冲刷” 走控制面对象
|
||||
- Agent 只负责执行,不负责决定如何处理死信
|
||||
|
||||
---
|
||||
|
||||
## 8. Job Status Enum
|
||||
|
||||
当前 Agent 主链路约束:
|
||||
|
||||
- `queued`
|
||||
- `dispatching`
|
||||
- `running`
|
||||
- `success`
|
||||
- `failed`
|
||||
- `partially_succeeded`
|
||||
|
||||
语义:
|
||||
|
||||
- `queued`:任务已创建,尚未派发
|
||||
- `dispatching`:控制面已派发,但节点尚未确认开工
|
||||
- `running`:节点已真实执行
|
||||
- `success`:成功完成
|
||||
- `failed`:失败完成
|
||||
- `partially_succeeded`:部分成功,需关注
|
||||
|
||||
---
|
||||
|
||||
## 9. 实现边界
|
||||
|
||||
Agent 只执行已声明动作:
|
||||
|
||||
- `service.start`
|
||||
- `service.stop`
|
||||
- `service.restart`
|
||||
- `service.status`
|
||||
- `health.snapshot`
|
||||
- `logs.collect`
|
||||
- `diagnostics.collect`
|
||||
- `deploy.release`
|
||||
|
||||
不允许控制面长期依赖任意 shell 下发。
|
||||
|
||||
`shell executor` 只能作为受限兜底能力存在,并必须挂审批与审计。
|
||||
56
docs/schemas/ops_doctor_decision_contract.md
Normal file
56
docs/schemas/ops_doctor_decision_contract.md
Normal file
@@ -0,0 +1,56 @@
|
||||
# ops_doctor_decision_contract
|
||||
|
||||
## Purpose
|
||||
|
||||
冻结 `doctor-decision` 主决策 contract,让页面、CLI、Codex 都消费同一份“当前先看哪一面、先执行什么动作、为什么”的正式结论。
|
||||
|
||||
## Producer
|
||||
|
||||
- backend service `app.services.ops_service.get_ops_doctor_decision`
|
||||
- API `GET /api/v1/ops/doctor-decision`
|
||||
|
||||
## Primary Consumers
|
||||
|
||||
- `domain-web` Ops Center 顶部总检入口与主决策详情面板
|
||||
- `domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision`
|
||||
- 海外 Codex 驾驶员 / 自动驾驶策略层
|
||||
|
||||
## Required Payload
|
||||
|
||||
- `status`
|
||||
- `status_label`
|
||||
- `headline`
|
||||
- `detail`
|
||||
- `decision`
|
||||
- `summary`
|
||||
- `recommended_commands`
|
||||
- `contract_navigation`
|
||||
|
||||
## Decision Shape
|
||||
|
||||
- `decision.status`
|
||||
- `decision.reason_code`
|
||||
- `decision.headline`
|
||||
- `decision.detail`
|
||||
- `decision.preferred_surface`
|
||||
- `decision.next_action_code`
|
||||
- `decision.recommended_commands`
|
||||
|
||||
## Summary Shape
|
||||
|
||||
- `summary.required_failures`
|
||||
- `summary.optional_unavailable`
|
||||
- `summary.scene_log_reports_total`
|
||||
- `summary.scene_log_reports_ok`
|
||||
- `summary.scene_log_status_counts`
|
||||
- `summary.scene_log_problem_node_codes`
|
||||
- `summary.contract_surface_gaps`
|
||||
- `summary.launchpad_recommended_target_node_code`
|
||||
- `summary.launchpad_recommended_recovery_label`
|
||||
- `summary.launchpad_recommended_recovery_summary`
|
||||
|
||||
## Notes
|
||||
|
||||
- 优先读取最新 `go-live bundle` 中的 `doctor_decision` 产物。
|
||||
- 如果 bundle 尚未导出,后端必须回落到 live `stack-diagnosis + go-live-summary` 兜底,而不是直接返回空。
|
||||
- `preferred_surface` 必须稳定可消费,不能把面板跳转逻辑继续留给前端猜测。
|
||||
797
docs/schemas/ops_driver_contract.md
Normal file
797
docs/schemas/ops_driver_contract.md
Normal file
@@ -0,0 +1,797 @@
|
||||
# domainCheck Ops Driver Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 Overseas Ops Center / Codex Driver / CLI 共用的驾驶 contract。
|
||||
|
||||
核心原则:
|
||||
|
||||
- 后端负责推荐和解释
|
||||
- 前端负责展示和触发
|
||||
- Codex 负责选择入口和上下文
|
||||
- 是否允许自动执行由后端统一门禁
|
||||
- 页面与 Codex 都不能各自再推导一套优先级
|
||||
|
||||
---
|
||||
|
||||
## 2. 入口接口
|
||||
|
||||
当前驾驶 contract 主要来自:
|
||||
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `GET /api/v1/ops/driver-feed`
|
||||
- `GET /api/v1/ops/codex-brief`
|
||||
- `POST /api/v1/ops/driver-actions/preview`
|
||||
- `POST /api/v1/ops/driver-actions/resolve`
|
||||
- `POST /api/v1/ops/driver-actions/execute`
|
||||
- `POST /api/v1/ops/driver-actions/execute-resolved`
|
||||
- `POST /api/v1/ops/codex-actions/resolve`
|
||||
- `POST /api/v1/ops/codex-actions/execute`
|
||||
|
||||
补充配套:
|
||||
|
||||
- `GET /api/v1/ops/playbooks`
|
||||
- `GET /api/v1/ops/playbook-runs`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
|
||||
---
|
||||
|
||||
## 3. Driver Recommendation
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.driver_recommendations`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "node-agent-dead-letter",
|
||||
"priority": "先处理 Agent 死信",
|
||||
"title": "先处理 Agent 死信",
|
||||
"summary": "当前已有节点出现死信记录,继续堆动作会放大控制面与节点现场的不一致。",
|
||||
"reason": "Node Agent 回执队列已经出现死信。",
|
||||
"level_label": "最高优先",
|
||||
"tag_type": "danger",
|
||||
"primary_label": "查看 Worker 日志",
|
||||
"secondary_label": "打开标准巡检",
|
||||
"primary_action_code": "open_worker_logs",
|
||||
"secondary_action_code": "open_diagnostics",
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"primary_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"secondary_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"primary_action_payload": {},
|
||||
"secondary_action_payload": {},
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- 推荐不只给文案
|
||||
- 必须给动作码、节点和预填 payload
|
||||
- 主次动作允许节点和 payload 不同
|
||||
- 推荐对象允许直接携带 `focus_ref / primary_focus_ref / secondary_focus_ref`
|
||||
- 新客户端优先用 `focus_ref` 做统一定位,而不是再从动作码反推页面落点
|
||||
|
||||
---
|
||||
|
||||
## 4. Priority Recommendation
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.recommendation`
|
||||
|
||||
作用:
|
||||
|
||||
- 给首页顶部、CLI 摘要、Codex 首屏判断提供“当前第一优先动作”
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"source": "driver_recommendations",
|
||||
"key": "node-agent-dead-letter",
|
||||
"priority": "先处理 Agent 死信",
|
||||
"title": "先处理 Agent 死信",
|
||||
"summary": "当前已有节点出现死信记录。",
|
||||
"reason": "Node Agent 回执队列已经出现死信。",
|
||||
"primary_action_code": "open_worker_logs",
|
||||
"secondary_action_code": "open_diagnostics"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Driver Feed
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/driver-feed`
|
||||
|
||||
作用:
|
||||
|
||||
- 提供比 `overview` 更直接给驾驶员消费的聚合结果
|
||||
|
||||
建议最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"top_recommendation": {},
|
||||
"driver_recommendations": [],
|
||||
"runbook_sequences": [],
|
||||
"activity_focus": [],
|
||||
"automation_coverage": {}
|
||||
}
|
||||
```
|
||||
|
||||
推荐补充约束:
|
||||
|
||||
- `top_recommendation`
|
||||
- 应该总是来自同一份 `entries` 列表,而不是额外拼一条“影子建议”
|
||||
- 这样页面、CLI、海外 Codex 在按 `key` 选中时才不会出现口径漂移
|
||||
- `driver_recommendations`
|
||||
- 普通驾驶建议,优先 `executor_kind=driver_action`
|
||||
- `runbook_sequences`
|
||||
- 标准作业路径,优先 `executor_kind=runbook_sequence`
|
||||
- `activity_focus`
|
||||
- 更偏观察与跟踪,不应伪装成“可立即执行”的驾驶动作
|
||||
- `activity_focus[*]`
|
||||
- 至少应保留:
|
||||
- `activity_key`
|
||||
- `activity_kind`
|
||||
- `summary`
|
||||
- `status`
|
||||
- `status_label`
|
||||
- `occurred_at`
|
||||
- `focus_ref`
|
||||
- `source_focus_ref`
|
||||
- `ui_intent`
|
||||
- 如果存在可落点观察入口
|
||||
- 允许补 `focus_action_code=focus_activity_item`
|
||||
- 但它的定位仍然是观察入口,而不是标准执行动作
|
||||
- `entries[*].focus_ref`
|
||||
- 是 driver-feed 的统一定位对象
|
||||
- 页面 / CLI / Codex 均应优先消费
|
||||
- `entries[*].primary_focus_ref` / `entries[*].secondary_focus_ref`
|
||||
- 是主次动作各自的稳定落点
|
||||
- 不允许调用方再根据 `primary_action_code` 自己猜“该跳 Release Hub、Launchpad 还是 Rollout Jobs”
|
||||
- `automation_coverage`
|
||||
- 是 Driver Feed 的自动化收口摘要
|
||||
- 用于统一回答:
|
||||
- 当前是否已经达到上线闸门
|
||||
- 当前还有多少动作停留在 preview-only
|
||||
- 当前后端是否已经足够接管首屏默认动作
|
||||
- 页面、CLI、Codex 不允许自行再统计一份“自动化覆盖率”
|
||||
|
||||
推荐最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"launch_ready": false,
|
||||
"launch_status": "attention",
|
||||
"summary": "仍有 3 个驾驶动作停留在 preview-only,暂未达到上线收口标准。",
|
||||
"total_actions": 12,
|
||||
"backend_handled_total": 9,
|
||||
"preview_only_total": 3,
|
||||
"ui_only_total": 0,
|
||||
"blocked_total": 0
|
||||
}
|
||||
```
|
||||
|
||||
其中 `runbook_sequences` 与 `activity_focus` 应继续区分:
|
||||
|
||||
- 这是建议动作
|
||||
- 还是固定标准作业路径
|
||||
- 还是最近值得优先处理的活动
|
||||
|
||||
推荐 CLI 消费顺序:
|
||||
|
||||
1. 显式 `entry_key`
|
||||
2. `top_recommendation`
|
||||
3. `entries[0]`
|
||||
|
||||
这样海外单入口就能稳定实现:
|
||||
|
||||
- `driver-focus-preview`
|
||||
- `driver-focus-run`
|
||||
|
||||
---
|
||||
|
||||
## 6. Codex Brief
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/codex-brief`
|
||||
|
||||
作用:
|
||||
|
||||
- 不让 Codex 自己从页面文案里猜“能不能自动执行”
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"mode": "guarded_auto",
|
||||
"recommended_behavior": "confirm_then_execute",
|
||||
"summary": "当前建议先处理 Agent 死信,再进入标准巡检。",
|
||||
"automation_coverage": {
|
||||
"launch_ready": false,
|
||||
"launch_status": "attention",
|
||||
"summary": "仍有 3 个驾驶动作停留在 preview-only。"
|
||||
},
|
||||
"target": {
|
||||
"kind": "driver_action",
|
||||
"action_code": "open_worker_logs",
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"action_payload": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐 `mode`:
|
||||
|
||||
- `safe_auto`
|
||||
- `guarded_auto`
|
||||
- `mixed`
|
||||
- `ui_only`
|
||||
- `blocked`
|
||||
|
||||
推荐 `recommended_behavior`:
|
||||
|
||||
- `auto_execute`
|
||||
- `confirm_then_execute`
|
||||
- `resolve_first`
|
||||
- `open_ui`
|
||||
- `blocked`
|
||||
|
||||
补充约束:
|
||||
|
||||
- `codex-brief.automation_coverage`
|
||||
- 应与 `driver-feed.automation_coverage` 保持同源
|
||||
- 供海外 Codex 驾驶员直接判断:
|
||||
- 是否允许默认走自动执行
|
||||
- 是否必须先预览或人工确认
|
||||
- 是否已经达到“可以上线”的自动化收口标准
|
||||
- `codex-brief.activity_focus`
|
||||
- 应直接透传 `driver-feed.activity_focus`
|
||||
- 供 Codex 在“当前没有可执行主线,但有观察焦点”时继续判断
|
||||
- `codex-brief.focus`
|
||||
- 优先使用 `entries[0]`
|
||||
- 当 `entries` 为空且 `activity_focus` 不为空时
|
||||
- 允许回退到首条带 `focus_action_code` 的 `activity_focus`
|
||||
- 这时推荐行为应偏向 `open_ui`
|
||||
- 不应伪装成自动执行动作
|
||||
|
||||
---
|
||||
|
||||
## 7. Driver Action Execute
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/driver-actions/execute`
|
||||
|
||||
作用:
|
||||
|
||||
- 把能标准化的驾驶动作收进统一后端执行入口
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"action_code": "run_standard_inspection",
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"action_payload": {}
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"handled": true,
|
||||
"mode": "playbook",
|
||||
"ui_intent": {
|
||||
"kind": "playbook_run_detail",
|
||||
"run_code": "run-20260418-001"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- 不只返回 handled / unhandled
|
||||
- 还要返回 `mode`
|
||||
- 还要返回 `ui_intent`
|
||||
|
||||
---
|
||||
|
||||
## 8. Driver Action Preview
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/driver-actions/preview`
|
||||
|
||||
作用:
|
||||
|
||||
- 给 OpsCenter / CLI / Codex Driver 统一生成驾驶动作契约预览
|
||||
- 冻结 `execution_chain / target_api / request_payload / 主次动作 section`
|
||||
- 不再让前端自己拼请求体和执行建议
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_label": "driver-feed",
|
||||
"title": "版本发布与放量",
|
||||
"summary": "当前可以进入正式发布路径。",
|
||||
"reason": "统一回到 Release / Rollout 闭环。",
|
||||
"executor_kind": "runbook_sequence",
|
||||
"sequence_key": "release_progression",
|
||||
"primary_label": "Worker 灰度",
|
||||
"secondary_label": "查看版本区",
|
||||
"secondary_action_code": "focus_release_hub",
|
||||
"requested_by": "web-ui/ops-center"
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"contract_key": "ops_driver_contract",
|
||||
"contract_version": "v1",
|
||||
"contract_schema_doc_path": "docs/schemas/ops_driver_contract.md",
|
||||
"execution_chain": "ops-driver -> runbook-sequence -> resolved driver action / ui-intent",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad"
|
||||
},
|
||||
"primary_focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
},
|
||||
"secondary_focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_detail"
|
||||
},
|
||||
"primary": {
|
||||
"label": "Worker 灰度",
|
||||
"action_code": "create_release_rollout_worker",
|
||||
"target_api": "/api/v1/ops/runbook/sequences/release_progression/execute",
|
||||
"target_method": "POST",
|
||||
"recommendation_label": "建议确认后执行",
|
||||
"automation_level_label": "guarded_auto",
|
||||
"risk_level_label": "high",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
},
|
||||
"request_payload": {
|
||||
"secondary": false,
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"node_codes": [
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"action_payload": {
|
||||
"release_id": 7,
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- preview 必须和 execute 走同一套 action 解析与 runbook resolve 逻辑
|
||||
- `primary` / `secondary` section 必须可直接给页面抽屉和 CLI 文本渲染复用
|
||||
- `request_payload` 必须是真正可发送到执行接口的 body,而不是页面内部临时对象
|
||||
- runbook sequence preview 必须经过后端 resolve,不能由页面自己猜 action_code
|
||||
- preview 返回体必须保留:
|
||||
- `focus_ref`
|
||||
- `primary_focus_ref`
|
||||
- `secondary_focus_ref`
|
||||
- `primary.focus_ref`
|
||||
- `secondary.focus_ref`
|
||||
|
||||
---
|
||||
|
||||
## 8.3. Driver Action Resolve / Execute Resolved
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/driver-actions/resolve`
|
||||
- `POST /api/v1/ops/driver-actions/execute-resolved`
|
||||
|
||||
作用:
|
||||
|
||||
- 把 `overview.driver_recommendations` / `driver-feed` / 页面按钮 / CLI 对普通驾驶动作的门禁判断统一下沉到后端
|
||||
- 页面不再根据 `preview` 自己推导 `直接执行 / 等确认 / 进入 UI / 阻断`
|
||||
- 后端统一输出:
|
||||
- `preview_request`
|
||||
- `preview`
|
||||
- `gate`
|
||||
- `selected_section`
|
||||
- `execution_result`
|
||||
|
||||
`resolve` 最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"source_label": "driver-feed",
|
||||
"title": "开启关键日志回传",
|
||||
"executor_kind": "driver_action",
|
||||
"primary_action_code": "enable_log_sync_key",
|
||||
"primary_action_payload": {},
|
||||
"requested_by": "web-ui/driver-feed",
|
||||
"secondary": false,
|
||||
"confirm": false
|
||||
}
|
||||
```
|
||||
|
||||
`resolve` 最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"preview_request": {},
|
||||
"preview": {},
|
||||
"selected_section": "primary",
|
||||
"focus_ref": {
|
||||
"kind": "execution_scene",
|
||||
"scene_key": "scene.logs"
|
||||
},
|
||||
"gate": {
|
||||
"decision": "execute_now",
|
||||
"decision_label": "直接执行",
|
||||
"decision_reason": "当前动作属于 safe_auto,可直接执行。",
|
||||
"will_execute": true,
|
||||
"recommendation": "auto_execute",
|
||||
"focus_ref": {
|
||||
"kind": "execution_scene",
|
||||
"scene_key": "scene.logs"
|
||||
},
|
||||
"action_code": "enable_log_sync_key",
|
||||
"target_api": "/api/v1/ops/driver-actions/execute"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`execute-resolved` 约束:
|
||||
|
||||
- 只有 `gate.will_execute=true` 才允许真正执行
|
||||
- `confirmation_required` 只有显式传入 `confirm=true` 才会放行
|
||||
- `open_ui / resolve_first / blocked` 必须返回非执行态,由页面或 CLI 继续展示契约
|
||||
- `driver-focus-preview / driver-focus-run`
|
||||
- 允许先从 `driver-feed.top_recommendation` 选出目标项
|
||||
- 也允许显式传入 `activity_focus.key`
|
||||
- 当目标来自 `activity_focus` 时
|
||||
- 应自动收口成 `focus_activity_item`
|
||||
- 只允许进入对应工作区,不应被当成标准执行动作
|
||||
- 再把它收口成同一份 `resolve / execute-resolved` 请求体
|
||||
- 不允许 CLI 再自己二次推导 gate
|
||||
- resolve 返回体必须保留:
|
||||
- `preview_request.focus_ref`
|
||||
- `preview_request.primary_focus_ref`
|
||||
- `preview_request.secondary_focus_ref`
|
||||
- `gate.focus_ref`
|
||||
- 顶层 `focus_ref`
|
||||
|
||||
---
|
||||
|
||||
## 8.5. Codex Action Resolve / Execute
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/codex-actions/resolve`
|
||||
- `POST /api/v1/ops/codex-actions/execute`
|
||||
|
||||
作用:
|
||||
|
||||
- 不再让 CLI / Codex 自己从 `codex-brief + preview` 拼一套门禁
|
||||
- 由后端统一输出:
|
||||
- 当前选中的 `selected_entry`
|
||||
- 对应的 `preview`
|
||||
- 统一门禁结论 `gate`
|
||||
- 最终是否允许执行
|
||||
|
||||
`resolve` 最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"entry_key": "runbook:release_progression",
|
||||
"requested_by": "cli/ops-center/codex-focus-preview",
|
||||
"source_label": "cli/codex-focus-preview",
|
||||
"include_secondary": true,
|
||||
"confirm": false
|
||||
}
|
||||
```
|
||||
|
||||
`resolve` 最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"entry_key": "runbook:release_progression",
|
||||
"selected_entry": {},
|
||||
"preview_request": {},
|
||||
"preview": {},
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
},
|
||||
"gate": {
|
||||
"decision": "confirmation_required",
|
||||
"decision_label": "等待确认",
|
||||
"recommendation": "confirm_then_execute",
|
||||
"recommendation_label": "建议确认后执行",
|
||||
"will_execute": false,
|
||||
"confirm_required": true,
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
},
|
||||
"target_api": "/api/v1/ops/runbook/sequences/release_progression/execute",
|
||||
"target_method": "POST",
|
||||
"request_payload": {
|
||||
"secondary": false,
|
||||
"requested_by": "cli/ops-center/codex-focus-preview",
|
||||
"action_payload": {
|
||||
"release_id": 7
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`execute` 要求:
|
||||
|
||||
- 必须复用 `resolve` 的同一套选中逻辑和门禁逻辑
|
||||
- `safe_auto`
|
||||
- 允许直接执行
|
||||
- `guarded_auto`
|
||||
- 只有 `confirm=true` 时允许执行
|
||||
- `resolve_first / open_ui / blocked`
|
||||
- 一律不得执行
|
||||
- 返回体必须保留:
|
||||
- `selected_entry`
|
||||
- `preview_request`
|
||||
- `preview`
|
||||
- `gate`
|
||||
- `focus_ref`
|
||||
- `execution_result`
|
||||
- `executed`
|
||||
|
||||
---
|
||||
|
||||
## 9. UI Intent
|
||||
|
||||
`ui_intent` 的职责不是让前端猜去哪,而是由后端直接声明:
|
||||
|
||||
- 应打开哪个详情
|
||||
- 应聚焦哪轮 run
|
||||
- 应落到哪个发布入口
|
||||
|
||||
当前建议固定支持:
|
||||
|
||||
- `playbook_run_detail`
|
||||
- `playbook_run_latest_events`
|
||||
- `job_events`
|
||||
- `rollout_jobs`
|
||||
- `managed_node_handover`
|
||||
- `managed_node_edit`
|
||||
- `open_release_dialog`
|
||||
- `open_rollout_dialog`
|
||||
- `focus_release_hub`
|
||||
- `focus_ref`
|
||||
|
||||
页面只做路由和抽屉承载,不做推荐解释。
|
||||
|
||||
---
|
||||
|
||||
## 10. Runbook Sequence
|
||||
|
||||
作用:
|
||||
|
||||
- 表达固定标准作业路径在“此刻”的推荐下一步
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "standard_inspection",
|
||||
"title": "标准巡检",
|
||||
"status": "warning",
|
||||
"status_label": "待补接管",
|
||||
"summary": "当前还没有满足标准巡检条件的节点。",
|
||||
"target_node_codes": [],
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 7,
|
||||
"section": "release_launchpad"
|
||||
},
|
||||
"primary_label": "直接执行标准巡检",
|
||||
"primary_action_code": "run_standard_inspection",
|
||||
"primary_action_payload": {},
|
||||
"primary_focus_ref": {},
|
||||
"secondary_label": "打开巡检编排",
|
||||
"secondary_action_code": "open_playbook_dialog",
|
||||
"secondary_action_payload": {
|
||||
"playbook_key": "inspection.standard"
|
||||
},
|
||||
"secondary_focus_ref": {}
|
||||
}
|
||||
```
|
||||
|
||||
要求:
|
||||
|
||||
- Runbook Sequence 是标准路径对象
|
||||
- 不是普通活动记录
|
||||
- 不是普通推荐卡
|
||||
- `focus_ref`
|
||||
- 表示这条标准路径整体对应的统一观察落点
|
||||
- `primary_focus_ref` / `secondary_focus_ref`
|
||||
- 表示主次动作各自应回看的稳定区域
|
||||
|
||||
---
|
||||
|
||||
## 11. Activity Stream
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
|
||||
作用:
|
||||
|
||||
- 把最近异常、编排、任务、Rollout 统一压成可钻取活动流
|
||||
|
||||
每条 activity 最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"kind": "playbook_run",
|
||||
"activity_key": "run-20260418-001",
|
||||
"status": "failed",
|
||||
"occurred_at": "2026-04-18 06:00:00",
|
||||
"summary": "标准巡检在 diagnostics.collect 步骤失败。",
|
||||
"ui_intent": {
|
||||
"kind": "playbook_run_detail",
|
||||
"run_code": "run-20260418-001"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐 `kind`:
|
||||
|
||||
- `playbook_run`
|
||||
- `ops_job`
|
||||
- `rollout`
|
||||
- `runbook_sequence`
|
||||
|
||||
---
|
||||
|
||||
## 12. 单脑驾驶舱首屏约束
|
||||
|
||||
Ops Center 首屏不是自由拼装页面,而是 Driver Contract 的正式消费者。
|
||||
|
||||
首屏必须优先消费同一组 contract:
|
||||
|
||||
- `overview.recommendation`
|
||||
- `driver-feed.top_recommendation`
|
||||
- `driver-feed.automation_coverage`
|
||||
- `codex-brief`
|
||||
|
||||
首屏建议固定为三层:
|
||||
|
||||
1. 驾驶结论
|
||||
2. 驾驶动作条
|
||||
3. 驾驶回执条
|
||||
|
||||
正式要求:
|
||||
|
||||
- 驾驶结论
|
||||
- 直接展示当前第一优先动作、自动化收口状态、是否达到上线闸门
|
||||
- 驾驶动作条
|
||||
- 只放“默认下一步 / 定位下一步 / 看契约 / 复制命令 / 进入主处理面板”这类高频动作
|
||||
- 这些按钮不能自己发明新规则,必须消费已有 `action_code / focus_ref / preview`
|
||||
- 驾驶回执条
|
||||
- 是首屏动作后的统一反馈区
|
||||
- 用于展示:
|
||||
- 已打开哪个契约
|
||||
- 已定位哪个焦点
|
||||
- 已复制哪个命令
|
||||
- 后端动作执行成功 / 警告 / 失败
|
||||
- 页面不应再把动作结果散落在多个局部 toast 里
|
||||
|
||||
也就是说:
|
||||
|
||||
- 首屏负责决策与触发
|
||||
- 下方详情区负责展开与下钻
|
||||
- 不允许详情区反向重写首屏推荐
|
||||
|
||||
---
|
||||
|
||||
## 13. 建议优先级顺序
|
||||
当前 contract 应优先遵守:
|
||||
|
||||
1. 先看异常编排
|
||||
2. 再看异常活动
|
||||
3. 再处理 Agent 死信 / 回执积压
|
||||
4. 再优先执行首个缺口节点的接入收口或接管验收
|
||||
5. 再处理现场日志回传和标准巡检
|
||||
6. 最后进入 Release / Rollout
|
||||
|
||||
这条顺序必须由后端维护,不应让前端或 Codex 再自行拼装。
|
||||
|
||||
补充约束:
|
||||
|
||||
- 当后端已经把首个缺口节点收敛成 `bootstrap_run / run_acceptance` 时:
|
||||
- `driver_feed.top_recommendation`
|
||||
- `codex_brief.focus`
|
||||
- `stack_diagnosis.next_step`
|
||||
都必须优先指向这一个节点和这一个动作
|
||||
- `fix_managed_nodes` 只能保留为兜底入口,不能再覆盖已经明确收敛的单节点恢复动作
|
||||
|
||||
---
|
||||
|
||||
## 14. 页面 / Codex / CLI 共用约束
|
||||
|
||||
同一条驾驶建议必须由后端一次性给出:
|
||||
|
||||
- 推荐标题
|
||||
- 推荐理由
|
||||
- 主动作
|
||||
- 次动作
|
||||
- 节点范围
|
||||
- 预填参数
|
||||
- UI 落点
|
||||
- 是否允许自动执行
|
||||
|
||||
这样三类入口才会真正共享一套驾驶 contract:
|
||||
|
||||
- OpsCenter 页面
|
||||
- 海外 Codex 驾驶员
|
||||
- CLI / 运维检查脚本
|
||||
|
||||
并且三类入口对 `recommended_behavior` 的执行语义也必须一致:
|
||||
|
||||
- `auto_execute`
|
||||
- 可直接执行
|
||||
- `confirm_then_execute`
|
||||
- 必须显式确认后才能执行
|
||||
- `resolve_first`
|
||||
- 只能先审阅和补上下文,不能直接执行
|
||||
- `open_ui`
|
||||
- 只能进入对应工作区,不能直接执行
|
||||
- `blocked`
|
||||
- 明确阻断,不能执行
|
||||
59
docs/schemas/ops_go_live_bundle_contract.md
Normal file
59
docs/schemas/ops_go_live_bundle_contract.md
Normal file
@@ -0,0 +1,59 @@
|
||||
# ops_go_live_bundle_contract
|
||||
|
||||
## 目标
|
||||
|
||||
把发布前证据包与 `manifest.json` 复核结论压成一份统一 contract,供:
|
||||
|
||||
- Ops Center 交付面板
|
||||
- CLI `go-live-export / go-live-review`
|
||||
- 海外 Codex 驾驶员
|
||||
|
||||
共同消费。
|
||||
|
||||
它解决的问题不是“现在能不能上线”,而是:
|
||||
|
||||
> 现在有没有一份足够完整、足够可信、可交付可复盘的上线证据包
|
||||
|
||||
## 主接口
|
||||
|
||||
- `GET /api/v1/ops/go-live-bundle`
|
||||
|
||||
## 核心字段
|
||||
|
||||
- `status`
|
||||
- `missing`
|
||||
- `ready`
|
||||
- `attention`
|
||||
- `blocked`
|
||||
- `status_label`
|
||||
- `headline`
|
||||
- `bundle_available`
|
||||
- `report_dir`
|
||||
- `manifest_path`
|
||||
- `artifact_total`
|
||||
- `failure_total`
|
||||
- `critical_failures`
|
||||
- `noncritical_failures`
|
||||
- `env_audit`
|
||||
- `summary`
|
||||
- `recommended_commands`
|
||||
|
||||
## 判定原则
|
||||
|
||||
- 如果当前还没有 bundle / manifest
|
||||
- `status = missing`
|
||||
- 如果 manifest 明确标记 `review_status_hint=blocked`
|
||||
- `status = blocked`
|
||||
- 如果 manifest 已存在,但:
|
||||
- 仍有非关键失败
|
||||
- 环境审计提示 attention
|
||||
- launchpad 摘要存在漂移
|
||||
- 则 `status = attention`
|
||||
- 只有当 manifest 可读、review 为 ready、关键摘要一致
|
||||
- 才允许 `status = ready`
|
||||
|
||||
## 设计约束
|
||||
|
||||
- 页面不直接触发 shell 导出 bundle
|
||||
- 后端默认只读最新 bundle / manifest,不负责替代 CLI 执行导出
|
||||
- 当 bundle 缺失时,必须明确给出固定命令,而不是让操作者自己猜路径
|
||||
37
docs/schemas/ops_go_live_review_contract.md
Normal file
37
docs/schemas/ops_go_live_review_contract.md
Normal file
@@ -0,0 +1,37 @@
|
||||
# ops_go_live_review_contract
|
||||
|
||||
## Purpose
|
||||
|
||||
冻结 `go-live-review` 复核结论 contract,让页面、CLI、Codex 都能消费同一份 bundle manifest 复核结果,而不是各自再解读一遍 artifact。
|
||||
|
||||
## Producer
|
||||
|
||||
- backend service `app.services.ops_service.get_ops_go_live_review`
|
||||
- API `GET /api/v1/ops/go-live-review`
|
||||
- CLI `bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review`
|
||||
|
||||
## Primary Consumers
|
||||
|
||||
- `domain-web` Ops Center 证据复核面板
|
||||
- 海外 Codex 驾驶员
|
||||
- 发布前人工值班复核
|
||||
|
||||
## Required Payload
|
||||
|
||||
- `status`
|
||||
- `status_label`
|
||||
- `headline`
|
||||
- `artifact_total`
|
||||
- `failure_total`
|
||||
- `critical_failures`
|
||||
- `noncritical_failures`
|
||||
- `env_audit`
|
||||
- `launchpad_alignment`
|
||||
- `recommended_next_steps`
|
||||
- `recommended_commands`
|
||||
|
||||
## Notes
|
||||
|
||||
- 这层是 manifest 的正式“复核解释层”,不是单纯重复 bundle 基础信息。
|
||||
- 如果 `launchpad_alignment.consistent=false`,即使关键文件都存在,也必须至少回到 `attention`。
|
||||
- 页面和 Codex 应优先消费这层 `headline / recommended_next_steps`,而不是自行拼 review 口径。
|
||||
65
docs/schemas/ops_go_live_signoff_contract.md
Normal file
65
docs/schemas/ops_go_live_signoff_contract.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# ops_go_live_signoff_contract
|
||||
|
||||
## 目标
|
||||
|
||||
把发布前最终签收判断压成一份统一 contract,供:
|
||||
|
||||
- Ops Center 首屏
|
||||
- CLI `go-live-signoff`
|
||||
- 海外 Codex 驾驶员
|
||||
|
||||
共同消费。
|
||||
|
||||
它不是替代:
|
||||
|
||||
- `go-live-summary`
|
||||
- `go-live-review`
|
||||
- `doctor-decision`
|
||||
- `stack-diagnosis`
|
||||
- `driver-feed`
|
||||
- `codex-brief`
|
||||
|
||||
而是在这些 contract 之上给出最终一句:
|
||||
|
||||
> 现在能不能签字上线
|
||||
|
||||
## 主接口
|
||||
|
||||
- `GET /api/v1/ops/go-live-signoff`
|
||||
|
||||
## 核心字段
|
||||
|
||||
- `signoff_status`
|
||||
- `ready`
|
||||
- `attention`
|
||||
- `blocked`
|
||||
- `status_label`
|
||||
- `headline`
|
||||
- `release_gate`
|
||||
- `blocked_reasons`
|
||||
- `attention_reasons`
|
||||
- `decision`
|
||||
- `launchpad_alignment`
|
||||
- `recommended_commands`
|
||||
- `report_dir`
|
||||
- `manifest_path`
|
||||
|
||||
## 判定原则
|
||||
|
||||
- 只要 `go_live_summary / go_live_review / doctor_decision / publish_status / stack_status / automation / release_launchpad` 中任一明确阻断
|
||||
- `signoff_status = blocked`
|
||||
- 如果没有阻断,但存在:
|
||||
- `go-live-review=attention|missing`
|
||||
- `doctor-decision=attention|missing`
|
||||
- `attention`
|
||||
- launchpad 摘要漂移
|
||||
- 自动化仍未完全进入 ready
|
||||
- 则 `signoff_status = attention`
|
||||
- 只有当收口、正式复核、主决策、发布、自动化、launchpad 摘要一致
|
||||
- 才允许 `signoff_status = ready`
|
||||
|
||||
## 设计约束
|
||||
|
||||
- 页面不允许再自己拼一套“看起来能上”
|
||||
- CLI 不允许绕过这个 contract 单独下结论
|
||||
- Codex 驾驶员优先看这一份,再决定是否继续下钻
|
||||
740
docs/schemas/ops_job_contract.md
Normal file
740
docs/schemas/ops_job_contract.md
Normal file
@@ -0,0 +1,740 @@
|
||||
# domainCheck Ops Job Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 `ops job` 的正式 contract。
|
||||
|
||||
它是整套海外单脑控制面里最核心的执行对象。
|
||||
|
||||
正式定义:
|
||||
|
||||
> `ops job` 是一条可审计、可编排、可派发、可回执、可取消的标准运维任务记录。
|
||||
|
||||
它不是:
|
||||
|
||||
- 一条 shell 命令
|
||||
- 一个页面按钮点击结果
|
||||
- 一个仅供 agent 消费的内部对象
|
||||
|
||||
它应该同时服务于:
|
||||
|
||||
- 页面按钮
|
||||
- CLI
|
||||
- 海外 Codex 驾驶员
|
||||
- Node Agent
|
||||
- Release / Rollout
|
||||
- Playbook Run
|
||||
|
||||
也就是说:
|
||||
|
||||
- `playbook run`
|
||||
- 是编排聚合对象
|
||||
- `rollout`
|
||||
- 是发布推进对象
|
||||
- `ops job`
|
||||
- 是真正的执行颗粒度对象
|
||||
|
||||
---
|
||||
|
||||
## 2. 入口接口
|
||||
|
||||
当前已落地的主入口:
|
||||
|
||||
- `GET /api/v1/ops/jobs`
|
||||
- `GET /api/v1/ops/jobs/{job_id}`
|
||||
- `GET /api/v1/ops/jobs/{job_id}/events`
|
||||
- `POST /api/v1/ops/jobs`
|
||||
- `POST /api/v1/ops/jobs/batch`
|
||||
- `POST /api/v1/ops/policy/preview`
|
||||
- `POST /api/v1/ops/jobs/{job_id}/approve`
|
||||
- `POST /api/v1/ops/jobs/{job_id}/cancel`
|
||||
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
|
||||
|
||||
配套对象来源:
|
||||
|
||||
- `ops_jobs`
|
||||
- `ops_job_steps`
|
||||
- `ops_job_events`
|
||||
|
||||
正式约束:
|
||||
|
||||
- 页面、CLI、Codex 不能各自定义第 2 套任务模型
|
||||
- `playbook run` 只能聚合 `ops job`
|
||||
- `rollout` 只能批量生成 `ops job`
|
||||
- `Node Agent` 只能拉取和回执 `ops job`
|
||||
|
||||
---
|
||||
|
||||
## 3. 当前实现里的 Job Object
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/jobs`
|
||||
- `GET /api/v1/ops/jobs/{job_id}`
|
||||
- 创建 / 审批 / 取消 / 派发后的返回体中的 `job`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3",
|
||||
"action": "logs.collect",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"status": "queued",
|
||||
"status_label": "排队中",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"payload": {},
|
||||
"metadata": {},
|
||||
"result": {},
|
||||
"summary": "目标节点 mainland-worker-01 / 发起 web-ui/ops-center / 方式 Node Agent",
|
||||
"summary_text": "目标节点 mainland-worker-01 / 发起 web-ui/ops-center / 方式 Node Agent",
|
||||
"error_message": "",
|
||||
"created_at": "2026-04-18 06:45:00",
|
||||
"started_at": "",
|
||||
"finished_at": "",
|
||||
"updated_at": "2026-04-18 06:45:00",
|
||||
"risk_level": "medium",
|
||||
"approval_required": false,
|
||||
"approval_status": "approved",
|
||||
"approval_status_label": "已审批",
|
||||
"approved_by": "",
|
||||
"approved_at": "",
|
||||
"blocked_reason": "",
|
||||
"cancellation_reason": "",
|
||||
"dispatched_at": "",
|
||||
"target_selector": {},
|
||||
"policy": {},
|
||||
"rollout_id": 0,
|
||||
"steps_total": 0,
|
||||
"steps_running": 0,
|
||||
"steps_success": 0,
|
||||
"steps_failed": 0,
|
||||
"steps_terminal": 0,
|
||||
"steps_loaded": false,
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-01"
|
||||
},
|
||||
"steps": []
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- `job_code`
|
||||
- 是跨页面、CLI、事件流的稳定任务标识
|
||||
- `action`
|
||||
- 是执行语义主键,不应用 `title` 或自然语言代替
|
||||
- `target_type`
|
||||
- 定义任务作用范围,不应只靠是否传了 `target_node_code` 来猜
|
||||
- `execution_mode`
|
||||
- 是正式执行方式,不是附属备注
|
||||
- `policy`
|
||||
- 是创建时冻结的策略快照,不应由页面二次推导
|
||||
- `status_label / approval_status_label`
|
||||
- 是给页面、CLI、Codex 直接消费的人类可读标签
|
||||
- `summary_text`
|
||||
- 是任务摘要 contract,不要求调用方再用状态和节点自己拼一句中文
|
||||
- `steps_total / steps_running / steps_success / steps_failed / steps_terminal`
|
||||
- 是任务步骤聚合统计,避免前端和 CLI 再自行遍历 steps 统计
|
||||
- `focus_ref`
|
||||
- 是统一定位对象,后续页面、CLI、Codex 应优先消费它,而不是自己重组跳转参数
|
||||
|
||||
---
|
||||
|
||||
## 4. Job 状态机
|
||||
|
||||
### 4.1 当前实现中已使用的状态
|
||||
|
||||
当前后端已实际使用:
|
||||
|
||||
- `queued`
|
||||
- `awaiting_approval`
|
||||
- `blocked`
|
||||
- `running`
|
||||
- `success`
|
||||
- `failed`
|
||||
- `cancelled`
|
||||
|
||||
这些状态已经是正式 contract 的最小闭环。
|
||||
|
||||
### 4.2 推荐终局扩展状态
|
||||
|
||||
终局可继续扩展为:
|
||||
|
||||
- `draft`
|
||||
- `approved`
|
||||
- `queued`
|
||||
- `dispatching`
|
||||
- `running`
|
||||
- `verifying`
|
||||
- `partially_succeeded`
|
||||
- `rollback_running`
|
||||
- `rolled_back`
|
||||
- `success`
|
||||
- `failed`
|
||||
- `cancelled`
|
||||
- `timed_out`
|
||||
|
||||
但要明确区分:
|
||||
|
||||
- 当前实际 API 已稳定承诺的,是最小状态集
|
||||
- 终局扩展状态不能破坏当前状态语义
|
||||
|
||||
---
|
||||
|
||||
## 5. Execution Mode Contract
|
||||
|
||||
当前实现里正式出现的执行方式:
|
||||
|
||||
- `remote-agent`
|
||||
- `local-runtime`
|
||||
- `control-plane`
|
||||
- `ssh`
|
||||
|
||||
语义必须固定:
|
||||
|
||||
### 5.1 `remote-agent`
|
||||
|
||||
- 默认正式执行方式
|
||||
- 由目标节点 Node Agent 拉取并执行
|
||||
|
||||
### 5.2 `local-runtime`
|
||||
|
||||
- 仅用于当前节点本机即时执行
|
||||
- 主要用于本机 runtime 动作
|
||||
|
||||
### 5.3 `control-plane`
|
||||
|
||||
- 由海外控制面直接执行
|
||||
- 适用于控制面内生动作
|
||||
|
||||
### 5.4 `ssh`
|
||||
|
||||
- 只作为救援和过渡执行器
|
||||
- 不能成为大规模正式发布的默认方式
|
||||
|
||||
正式约束:
|
||||
|
||||
- 发布默认优先 `remote-agent`
|
||||
- `ssh` 只应是兜底链路
|
||||
- 页面、CLI、Codex 不应再自行创造第 5 种 execution mode
|
||||
|
||||
---
|
||||
|
||||
## 6. Target Type Contract
|
||||
|
||||
当前 contract 至少应允许:
|
||||
|
||||
- `node`
|
||||
- `batch`
|
||||
- `nodes`
|
||||
- `rollout`
|
||||
|
||||
语义建议固定为:
|
||||
|
||||
### 6.1 `node`
|
||||
|
||||
- 单节点任务
|
||||
|
||||
### 6.2 `batch` / `nodes`
|
||||
|
||||
- 多节点批量任务
|
||||
- 当前通常通过 `/ops/jobs/batch` 创建
|
||||
|
||||
### 6.3 `rollout`
|
||||
|
||||
- 该任务属于某轮 release rollout 推进的一部分
|
||||
|
||||
正式要求:
|
||||
|
||||
- 如果任务来自 rollout,必须能通过 `rollout_id` 回溯
|
||||
- 如果任务来自 playbook,应通过 `metadata` 标出 run 归属
|
||||
|
||||
---
|
||||
|
||||
## 7. Job Step Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/jobs/{job_id}`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 201,
|
||||
"step_key": "dispatch",
|
||||
"title": "调度动作",
|
||||
"node_code": "mainland-worker-01",
|
||||
"status": "queued",
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
"result": {},
|
||||
"created_at": "2026-04-18 06:45:00",
|
||||
"started_at": "",
|
||||
"finished_at": "",
|
||||
"updated_at": "2026-04-18 06:45:00"
|
||||
}
|
||||
```
|
||||
|
||||
当前实现说明:
|
||||
|
||||
- 现阶段每条 `ops job` 至少会先生成一个 `dispatch` step
|
||||
- 以后如果引入真正 DAG / 多步骤执行,也应继续沿用 `ops_job_steps`
|
||||
|
||||
正式约束:
|
||||
|
||||
- `step_key`
|
||||
- 是步骤主键,不应用标题代替
|
||||
- `status`
|
||||
- 必须与 job 状态联动,但不要求完全相同
|
||||
- `stdout / stderr`
|
||||
- 是步骤级输出,不应与整条 job 的 `result` 混成一层
|
||||
|
||||
---
|
||||
|
||||
## 8. Job Event Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/jobs/{job_id}/events`
|
||||
|
||||
最小字段建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 301,
|
||||
"event_key": "job-event:301",
|
||||
"job_id": 101,
|
||||
"step_id": 201,
|
||||
"node_code": "mainland-worker-01",
|
||||
"client_event_id": "evt-001",
|
||||
"event_type": "job_dispatch_requested",
|
||||
"level": "info",
|
||||
"level_label": "信息",
|
||||
"message": "任务等待 node-agent 拉取: 101",
|
||||
"summary": "任务等待 node-agent 拉取: 101",
|
||||
"payload": {},
|
||||
"has_payload": false,
|
||||
"occurred_at": "2026-04-18 06:45:03",
|
||||
"created_at": "2026-04-18 06:45:03"
|
||||
}
|
||||
```
|
||||
|
||||
当前代码里事件来源包括:
|
||||
|
||||
- job created
|
||||
- job blocked
|
||||
- job awaiting approval
|
||||
- job approved
|
||||
- job cancelled
|
||||
- job dispatch requested
|
||||
- job executed locally / over ssh / on control plane
|
||||
- agent start / complete / custom events
|
||||
|
||||
正式要求:
|
||||
|
||||
- `events` 是任务时间线,不是调试附属品
|
||||
- 页面、CLI、Codex 都应从这里读“任务做到哪一步了”
|
||||
- 不应重新从 stdout/stderr 推断关键状态
|
||||
|
||||
补充约束:
|
||||
|
||||
- `event_key`
|
||||
- 是事件行的稳定前端 / CLI 标识
|
||||
- `summary`
|
||||
- 是给页面、CLI、Codex 直接消费的短摘要,不要求调用方再拼接 `message`
|
||||
- `level_label`
|
||||
- 是面向人看的等级标签
|
||||
- `occurred_at`
|
||||
- 是统一观察面时间字段,允许与 `created_at` 同值
|
||||
|
||||
### 8.1 Event List Summary Contract
|
||||
|
||||
当前 `GET /api/v1/ops/jobs/{job_id}/events` 建议同时返回:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": {
|
||||
"job_id": 101,
|
||||
"returned_total": 12,
|
||||
"level_counts": {
|
||||
"info": 8,
|
||||
"warning": 3,
|
||||
"error": 1
|
||||
},
|
||||
"event_type_counts": {
|
||||
"job_created": 1,
|
||||
"job_dispatch_requested": 1,
|
||||
"executor_received": 1
|
||||
},
|
||||
"node_counts": {
|
||||
"mainland-worker-01": 12
|
||||
},
|
||||
"latest_at": "2026-04-18 06:45:08",
|
||||
"filters": {
|
||||
"limit": 50
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- 事件接口不只给一串数组
|
||||
- 至少要给:
|
||||
- 当前返回了多少条
|
||||
- 等级分布
|
||||
- 事件类型分布
|
||||
- 最近时间
|
||||
- 这样页面、CLI、Codex 才能先判断“有没有异常事件”再决定是否展开明细
|
||||
|
||||
---
|
||||
|
||||
## 9. List Contract 与 Compact 模式
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/jobs`
|
||||
|
||||
当前实现特点:
|
||||
|
||||
- 默认 `list_ops_jobs(..., compact=True)`
|
||||
- 返回 compact job
|
||||
- `steps` 被置空
|
||||
- `payload / metadata / result` 会被压缩
|
||||
|
||||
因此建议把列表 contract 明确成:
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3",
|
||||
"action": "logs.collect",
|
||||
"status": "queued",
|
||||
"execution_mode": "remote-agent",
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"payload": {
|
||||
"tail_lines": 120
|
||||
},
|
||||
"metadata": {},
|
||||
"result": {},
|
||||
"result_summary": {},
|
||||
"is_compact": true,
|
||||
"steps": []
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- 列表页默认只承诺 compact 视图
|
||||
- 详情页再承诺 full job + steps
|
||||
- 页面不应假设列表结果天然带完整 steps
|
||||
|
||||
---
|
||||
|
||||
## 10. Create Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"action": "logs.collect",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"execution_mode": "remote-agent",
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"payload": {
|
||||
"tail_lines": 120
|
||||
},
|
||||
"metadata": {
|
||||
"source": "ops-center"
|
||||
},
|
||||
"auto_approve": false,
|
||||
"run_now": true
|
||||
}
|
||||
```
|
||||
|
||||
当前实现还支持:
|
||||
|
||||
- `target_selector`
|
||||
- `rollout_id`
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"job": {},
|
||||
"executed_immediately": false
|
||||
}
|
||||
```
|
||||
|
||||
关键语义:
|
||||
|
||||
- `executed_immediately=true`
|
||||
- 说明任务已被控制面即时执行,而不是停留在队列
|
||||
- `executed_immediately=false`
|
||||
- 说明任务进入了:
|
||||
- `queued`
|
||||
- `awaiting_approval`
|
||||
- `blocked`
|
||||
|
||||
正式要求:
|
||||
|
||||
- 创建接口不能只回答“建成功了”
|
||||
- 必须明确回答:
|
||||
- 当前状态是什么
|
||||
- 是否立即执行
|
||||
- 如果没执行,是在等审批、等 agent、还是被阻断
|
||||
|
||||
---
|
||||
|
||||
## 11. Batch Create Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs/batch`
|
||||
|
||||
最小请求体建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"template_key": "diagnostics.collect",
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-worker-02"
|
||||
],
|
||||
"execution_mode": "remote-agent",
|
||||
"requested_by": "web-ui/ops-center",
|
||||
"payload": {},
|
||||
"metadata": {},
|
||||
"auto_approve": true
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs": [],
|
||||
"results": []
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- batch 只是创建方式,不是新的执行对象
|
||||
- batch 最终仍要展开成多条 `ops job`
|
||||
- 单条失败不能让结果完全丢失,必须保留逐项结果
|
||||
|
||||
---
|
||||
|
||||
## 12. Policy Preview Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/policy/preview`
|
||||
- `ops_jobs.policy`
|
||||
|
||||
这层是创建前的正式门禁对象。
|
||||
|
||||
当前创建任务时,后端已先做:
|
||||
|
||||
- 风险等级判断
|
||||
- 阻断判断
|
||||
- 审批要求判断
|
||||
|
||||
建议最小返回字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"blocked": false,
|
||||
"blocking_reasons": [],
|
||||
"approval_required": true,
|
||||
"approval_reasons": [
|
||||
"当前目标包含 control 节点"
|
||||
],
|
||||
"warnings": [],
|
||||
"recommendations": [],
|
||||
"risk_level": "high",
|
||||
"target_summary": {},
|
||||
"batch_plan": {},
|
||||
"cluster_guardrails": {},
|
||||
"operational_readiness": {}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- 页面、CLI、Codex 都应复用同一份 policy preview
|
||||
- 不能各自再写一套“能不能执行”的判断逻辑
|
||||
|
||||
---
|
||||
|
||||
## 13. Approval / Cancel / Dispatch Contract
|
||||
|
||||
### 13.1 Approve
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs/{job_id}/approve`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"approved_by": "admin"
|
||||
}
|
||||
```
|
||||
|
||||
语义:
|
||||
|
||||
- 仅 `awaiting_approval` 可审批
|
||||
- 审批后回到 `queued`
|
||||
|
||||
### 13.2 Cancel
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs/{job_id}/cancel`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"cancelled_by": "admin",
|
||||
"reason": "本轮灰度暂停"
|
||||
}
|
||||
```
|
||||
|
||||
语义:
|
||||
|
||||
- `success / failed / cancelled` 不可取消
|
||||
- 取消后 job 进入 `cancelled`
|
||||
- steps 同步进入 `cancelled` 或保持终态
|
||||
|
||||
### 13.3 Dispatch
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
|
||||
|
||||
语义:
|
||||
|
||||
- `queued` 才可派发
|
||||
- `control-plane`
|
||||
- 立即由控制面执行
|
||||
- `local-runtime`
|
||||
- 当前节点本机即时执行
|
||||
- `ssh`
|
||||
- 由 SSH 执行器执行
|
||||
- `remote-agent`
|
||||
- 保持 `queued`,等待对应 agent 拉取
|
||||
|
||||
这点必须明确:
|
||||
|
||||
> dispatch 不等于一定会立刻变成 running。
|
||||
|
||||
对于 `remote-agent`,它可能只是:
|
||||
|
||||
- 写入事件
|
||||
- 保持排队
|
||||
- 等节点稍后 pull
|
||||
|
||||
---
|
||||
|
||||
## 14. 与 Playbook / Rollout / Agent 的边界
|
||||
|
||||
### 14.1 与 Playbook
|
||||
|
||||
- `playbook run`
|
||||
- 负责展开、聚合、重跑、取消整轮编排
|
||||
- `ops job`
|
||||
- 负责单条执行记录
|
||||
|
||||
### 14.2 与 Rollout
|
||||
|
||||
- `rollout`
|
||||
- 负责发布批次推进
|
||||
- `ops job`
|
||||
- 负责每个节点每步的实际任务
|
||||
|
||||
### 14.3 与 Node Agent
|
||||
|
||||
- 控制面创建 `ops job`
|
||||
- agent 拉取 `ops job`
|
||||
- agent 回写 `ops_job_events`
|
||||
- agent 推进 job / step 状态
|
||||
|
||||
### 14.4 与 Observability
|
||||
|
||||
- `ops_observability_contract`
|
||||
- 看现场事实
|
||||
- `ops_job_contract`
|
||||
- 看执行动作本身
|
||||
|
||||
这两层不能混:
|
||||
|
||||
- 观察面告诉你“谁在跑、谁异常”
|
||||
- 任务面告诉你“这一条任务如何被创建、审批、派发、执行、取消”
|
||||
|
||||
---
|
||||
|
||||
## 15. 终局约束
|
||||
|
||||
后续不应再回退到下面几种模式:
|
||||
|
||||
### 1. 页面点按钮直接跑 shell
|
||||
|
||||
错误。
|
||||
|
||||
正确方式:
|
||||
|
||||
- 页面创建 `ops job`
|
||||
|
||||
### 2. Codex 直接 SSH 改现场
|
||||
|
||||
错误。
|
||||
|
||||
正确方式:
|
||||
|
||||
- Codex 创建 `ops job` / `playbook run`
|
||||
|
||||
### 3. 发布系统不经过任务层
|
||||
|
||||
错误。
|
||||
|
||||
正确方式:
|
||||
|
||||
- rollout 批量生成 `ops job`
|
||||
|
||||
### 4. 死信治理不留痕
|
||||
|
||||
错误。
|
||||
|
||||
正确方式:
|
||||
|
||||
- `flush / replay / discard` 都生成正式 `ops job`
|
||||
|
||||
所以可以把这份文档看成整套系统的“执行订单母本”。
|
||||
|
||||
后续只要这份 contract 稳住,页面、CLI、Codex、Agent、Release、Playbook 就都能围绕同一个中心对象协作。
|
||||
702
docs/schemas/ops_observability_contract.md
Normal file
702
docs/schemas/ops_observability_contract.md
Normal file
@@ -0,0 +1,702 @@
|
||||
# domainCheck Ops Observability Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结海外单脑控制面里“看现场”这一层的正式 contract。
|
||||
|
||||
它不负责:
|
||||
|
||||
- 发布决策
|
||||
- Driver 推荐
|
||||
- Playbook 编排
|
||||
|
||||
它负责回答 3 类问题:
|
||||
|
||||
1. 哪些节点现在真正处于执行现场
|
||||
2. 这些节点最近巡检收口结果是什么
|
||||
3. Node Agent 回执队列现在是否健康
|
||||
|
||||
也就是说,这份 contract 的职责是:
|
||||
|
||||
> 让页面、CLI、Codex 看到同一份现场事实,而不是各自从日志和状态字串里猜
|
||||
|
||||
---
|
||||
|
||||
## 2. 入口接口
|
||||
|
||||
建议把下面这些接口视为同一观察面的一组正式入口:
|
||||
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `GET /api/v1/ops/inspection-overview`
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/handover`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/flush`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/replay`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay`
|
||||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard`
|
||||
|
||||
配套关系:
|
||||
|
||||
- `overview`
|
||||
- 首页收口
|
||||
- `inspection-overview`
|
||||
- 节点巡检收口
|
||||
- `activity-stream`
|
||||
- 最近异常活动
|
||||
- `handover`
|
||||
- 单节点接管阻断与下一步建议
|
||||
- `delivery-queue`
|
||||
- Node Agent 回执健康
|
||||
|
||||
正式要求:
|
||||
|
||||
- 页面不能自己再定义第 5 套“观察口径”
|
||||
- CLI 不应重新从数据库拼现场
|
||||
- Codex 不应通过自然语言日志推断核心状态
|
||||
|
||||
---
|
||||
|
||||
## 3. Execution Scene Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `overview.execution_scene`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": "当前存在 2 台有效执行节点,其中 1 台正在领任务,1 台在线待命。",
|
||||
"dispatch_active_nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "busy",
|
||||
"participation_state": "dispatch_active",
|
||||
"participation_label": "正在执行",
|
||||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||||
"current_load": 9,
|
||||
"items_claimed": 9,
|
||||
"items_running": 5,
|
||||
"items_completed_recent": 14
|
||||
}
|
||||
],
|
||||
"recent_only_nodes": [],
|
||||
"standby_nodes": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"region": "mainland",
|
||||
"role": "control",
|
||||
"status": "online",
|
||||
"participation_state": "standby",
|
||||
"participation_label": "在线待命",
|
||||
"participation_reason": "节点在线、有效,但当前没有领任务、执行中任务或近窗吞吐",
|
||||
"current_load": 0
|
||||
}
|
||||
],
|
||||
"load_syncing_nodes": [],
|
||||
"counts": {
|
||||
"dispatch_active": 1,
|
||||
"recent_only": 0,
|
||||
"standby": 1,
|
||||
"load_syncing": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
分组语义必须固定:
|
||||
|
||||
- `dispatch_active_nodes`
|
||||
- 当前正在领任务、执行任务、或仍有实时工作负载
|
||||
- `recent_only_nodes`
|
||||
- 当前不再运行,但近窗刚有吞吐
|
||||
- `standby_nodes`
|
||||
- 在线、有效,但当前不参与
|
||||
- `load_syncing_nodes`
|
||||
- 节点与控制面状态尚未完全同步
|
||||
|
||||
正式约束:
|
||||
|
||||
- “在线”不等于“参与检测”
|
||||
- “有效执行节点”不等于“正在跑”
|
||||
- 页面展示、CLI 摘要、Codex 判断必须沿用这 4 类分组
|
||||
|
||||
节点参与态附加字段也应固定:
|
||||
|
||||
- `participation_state`
|
||||
- 机器判断主键,推荐值:
|
||||
- `dispatch_active`
|
||||
- `recent_only`
|
||||
- `standby`
|
||||
- `load_syncing`
|
||||
- `participation_label`
|
||||
- 直接给页面、CLI、Codex 展示的人类可读标签
|
||||
- `participation_reason`
|
||||
- 用一句稳定摘要解释为什么节点落在当前分组
|
||||
|
||||
正式要求:
|
||||
|
||||
- 页面必须直接区分:
|
||||
- 在线但未参与
|
||||
- 正在执行或领任务
|
||||
- 近窗刚参与但当前已收口
|
||||
- 新消费方不允许再从:
|
||||
- `status`
|
||||
- `current_load`
|
||||
- `items_running`
|
||||
手工推断“这个节点到底算不算正在参与”
|
||||
|
||||
---
|
||||
|
||||
## 4. Log Sync Runtime Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.execution_scene.log_sync`
|
||||
- 或 `overview.log_sync`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "partial_coverage",
|
||||
"status_label": "部分覆盖",
|
||||
"mode": "key",
|
||||
"mode_label": "关键回传",
|
||||
"enabled": true,
|
||||
"source_nodes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"source_node_summaries": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"line_count": 42,
|
||||
"key_line_count": 42,
|
||||
"full_line_count": 0,
|
||||
"last_at": "2026-04-18 06:40:00",
|
||||
"last_line": "域名检测完成: demo.com"
|
||||
}
|
||||
],
|
||||
"covered_participating_node_summaries": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"participation_state": "dispatch_active",
|
||||
"participation_label": "正在执行",
|
||||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||||
"line_count": 42,
|
||||
"key_line_count": 42,
|
||||
"full_line_count": 0,
|
||||
"last_at": "2026-04-18 06:40:00",
|
||||
"last_line": "域名检测完成: demo.com"
|
||||
}
|
||||
],
|
||||
"missing_participating_node_summaries": [
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"participation_state": "dispatch_active",
|
||||
"participation_label": "正在执行",
|
||||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||||
"missing_reason": "no_sample"
|
||||
}
|
||||
],
|
||||
"covered_participating_nodes": 1,
|
||||
"participating_nodes_total": 2,
|
||||
"missing_sample_node_codes": [
|
||||
"mainland-worker-02"
|
||||
],
|
||||
"last_line_at": "2026-04-18 06:40:00",
|
||||
"summary": "当前已覆盖 1/2 台参与节点,建议继续观察未回传样本节点。"
|
||||
}
|
||||
```
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `disabled`
|
||||
- `waiting_sample`
|
||||
- `partial_coverage`
|
||||
- `healthy`
|
||||
- `full_capture`
|
||||
|
||||
这层是“现场可见性”最关键的辅助状态。
|
||||
|
||||
正式要求:
|
||||
|
||||
- 如果存在参与节点但日志回传还没开,优先推荐开关键回传
|
||||
- 如果日志回传已开但没有样本,优先推荐抓 Worker 日志
|
||||
- 页面和 Codex 都不能再自己定义另一套日志覆盖结论
|
||||
|
||||
节点级日志覆盖字段也应固定:
|
||||
|
||||
- `source_nodes`
|
||||
- 为兼容旧页面保留的节点编码列表
|
||||
- `source_node_summaries`
|
||||
- 当前已有日志样本的节点级统计对象
|
||||
- `covered_participating_node_summaries`
|
||||
- 已参与且已拿到样本的节点对象
|
||||
- `missing_participating_node_summaries`
|
||||
- 已参与但尚未拿到样本的节点对象
|
||||
|
||||
`source_node_summaries[]` 至少应返回:
|
||||
|
||||
- `node_code`
|
||||
- `line_count`
|
||||
- `key_line_count`
|
||||
- `full_line_count`
|
||||
- `last_at`
|
||||
- `last_line`
|
||||
|
||||
`missing_participating_node_summaries[]` 至少应返回:
|
||||
|
||||
- `node_code`
|
||||
- `region`
|
||||
- `role`
|
||||
- `participation_state`
|
||||
- `participation_label`
|
||||
- `participation_reason`
|
||||
- `missing_reason`
|
||||
|
||||
推荐 `missing_reason`:
|
||||
|
||||
- `no_sample`
|
||||
- `delayed_flush`
|
||||
- `filtered_out`
|
||||
|
||||
兼容要求:
|
||||
|
||||
- 旧页面仍可继续消费:
|
||||
- `source_nodes`
|
||||
- `missing_sample_node_codes`
|
||||
- 新页面、CLI、Codex 应优先消费:
|
||||
- `source_node_summaries`
|
||||
- `covered_participating_node_summaries`
|
||||
- `missing_participating_node_summaries`
|
||||
|
||||
补充节点级现场入口:
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/scene-log`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"status": "healthy",
|
||||
"status_label": "关键覆盖",
|
||||
"mode": "key",
|
||||
"mode_label": "关键回传",
|
||||
"log_sync_enabled": true,
|
||||
"summary": "节点当前正在参与检测,已保留 42 条现场日志样本。",
|
||||
"node": {
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "busy",
|
||||
"current_load": 6,
|
||||
"last_heartbeat_at": "2026-04-18 06:40:00"
|
||||
},
|
||||
"participation": {
|
||||
"detect_participating": true,
|
||||
"participation_state": "claimed",
|
||||
"participation_label": "已领待跑",
|
||||
"participation_bucket": "dispatch_active",
|
||||
"participation_reason": "已领取任务,等待线程继续执行。"
|
||||
},
|
||||
"source_summary": {
|
||||
"line_count": 42,
|
||||
"key_line_count": 42,
|
||||
"full_line_count": 0,
|
||||
"last_at": "2026-04-18 06:40:00",
|
||||
"last_line": "[2026-04-18 06:40:00] [mainland-worker-01] 域名检测完成: demo.com"
|
||||
},
|
||||
"records_total": 42,
|
||||
"records_visible": 80,
|
||||
"records": [
|
||||
{
|
||||
"created_at": "2026-04-18 06:38:00",
|
||||
"node_code": "mainland-worker-01",
|
||||
"message": "开始检测域名: demo.com",
|
||||
"mode": "key",
|
||||
"line": "[2026-04-18 06:38:00] [mainland-worker-01] 开始检测域名: demo.com"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
节点级入口的正式语义:
|
||||
|
||||
- 这是“看某一台节点当前现场”的正式入口,不允许页面再自己从聚合样本中二次猜测
|
||||
- `status` 至少覆盖:
|
||||
- `healthy`
|
||||
- `full_capture`
|
||||
- `historical_sample`
|
||||
- `waiting_sample`
|
||||
- `missing_sample`
|
||||
- `disabled`
|
||||
- `standby`
|
||||
- `unknown`
|
||||
- 当节点当前在线但未参与时,必须明确返回 `standby`
|
||||
- 当节点正在参与但还没有样本时,必须明确返回 `waiting_sample`
|
||||
- 当节点已有历史样本但当前不在参与面中时,必须明确返回 `historical_sample`
|
||||
|
||||
---
|
||||
|
||||
## 5. Inspection Overview Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/inspection-overview`
|
||||
- `overview.inspection`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "attention",
|
||||
"status_label": "待处理",
|
||||
"summary": "存在 1 台节点最近缺少 Worker 日志收口。",
|
||||
"counts": {
|
||||
"healthy_nodes": 1,
|
||||
"attention_nodes": 1,
|
||||
"problem_nodes": 0
|
||||
},
|
||||
"rows": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"problem_kind": "missing_worker_logs",
|
||||
"problem_label": "缺少 Worker 日志收口",
|
||||
"problem_level": "warning",
|
||||
"recommended_action_code": "open_worker_logs",
|
||||
"ui_intent": {
|
||||
"kind": "node_inspection_detail",
|
||||
"node_code": "mainland-worker-01"
|
||||
},
|
||||
"latest_health_snapshot": {},
|
||||
"latest_worker_logs": {},
|
||||
"latest_diagnostics": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
设计要求:
|
||||
|
||||
- 巡检不是几条离散 job 记录
|
||||
- 必须按节点重新聚合
|
||||
- 必须直接给出:
|
||||
- `problem_kind`
|
||||
- `problem_label`
|
||||
- `problem_level`
|
||||
- `recommended_action_code`
|
||||
- `ui_intent`
|
||||
|
||||
这样页面、CLI、Codex 才能真正消费“节点巡检结论”,而不是再从几十条 job 记录里回推问题。
|
||||
|
||||
---
|
||||
|
||||
## 6. Inspection Row Contract
|
||||
|
||||
`rows[]` 中每个节点建议最少包含:
|
||||
|
||||
- `node_code`
|
||||
- `region`
|
||||
- `role`
|
||||
- `effective_worker`
|
||||
- `problem_kind`
|
||||
- `problem_label`
|
||||
- `problem_level`
|
||||
- `recommended_action_code`
|
||||
- `ui_intent`
|
||||
- `latest_health_snapshot`
|
||||
- `latest_worker_logs`
|
||||
- `latest_diagnostics`
|
||||
|
||||
其中 3 类最近结果建议统一结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "success",
|
||||
"status_label": "成功",
|
||||
"occurred_at": "2026-04-18 06:35:00",
|
||||
"job_code": "ops-20260418-001",
|
||||
"summary": "最近一次 Worker 日志收集成功,共采样 120 行。"
|
||||
}
|
||||
```
|
||||
|
||||
正式边界:
|
||||
|
||||
- `logs.collect` 默认指 Worker 日志收口
|
||||
- Node Agent 辅助排障日志不应混成同一种巡检结果
|
||||
|
||||
---
|
||||
|
||||
## 7. Delivery Queue Summary Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"node": {
|
||||
"node_code": "mainland-worker-01",
|
||||
"title": "mainland-worker-01"
|
||||
},
|
||||
"summary": {
|
||||
"state": "dead_letter",
|
||||
"label": "死信",
|
||||
"reason": "当前有 2 条记录进入死信。",
|
||||
"pending_count": 3,
|
||||
"dead_letter_count": 2,
|
||||
"oldest_pending_at": "2026-04-18 06:20:00",
|
||||
"oldest_dead_letter_at": "2026-04-18 06:18:00"
|
||||
},
|
||||
"record_visibility": "head_only",
|
||||
"capabilities": {
|
||||
"can_flush": true,
|
||||
"can_replay": true,
|
||||
"can_discard": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `healthy`
|
||||
- `retrying`
|
||||
- `dead_letter`
|
||||
- `stalled`
|
||||
|
||||
正式约束:
|
||||
|
||||
- 控制面要明确告诉操作者当前是 `head_only` 还是 `full_projection`
|
||||
- 不允许页面假装能列出远端完整目录
|
||||
- 不允许因为能看见死信就绕过 `ops job` 直接改文件
|
||||
|
||||
---
|
||||
|
||||
## 8. Delivery Queue Record Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||||
|
||||
当前阶段建议最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"records": [
|
||||
{
|
||||
"record_id": 101,
|
||||
"state": "dead_letter",
|
||||
"state_label": "死信",
|
||||
"request_id": "req-20260418-001",
|
||||
"kind": "job_complete",
|
||||
"created_at": "2026-04-18 06:18:00",
|
||||
"updated_at": "2026-04-18 06:19:00",
|
||||
"summary": "某次任务完成回执多次补发失败。",
|
||||
"visibility": "head_only"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"returned_records": 1,
|
||||
"record_visibility": "head_only"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
当前阶段要求:
|
||||
|
||||
- 单条 `replay / discard` 只针对控制面可见记录
|
||||
- 当前 `head_only` 不等于未来不能扩展
|
||||
- 以后若进入 `full_projection`,扩的是可见范围,不是重写操作模型
|
||||
|
||||
---
|
||||
|
||||
## 9. Delivery Queue Action Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `flush`
|
||||
- `replay`
|
||||
- `record replay`
|
||||
- `record discard`
|
||||
|
||||
正式要求:
|
||||
|
||||
- 所有动作都必须落成正式 `ops job`
|
||||
- 返回体要能告诉调用方:
|
||||
- 是否已受理
|
||||
- 生成了什么 job
|
||||
- 建议跳转到哪里观察
|
||||
|
||||
最小返回体建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"accepted": true,
|
||||
"job_code": "ops-20260418-queue-001",
|
||||
"ui_intent": {
|
||||
"kind": "job_events",
|
||||
"job_code": "ops-20260418-queue-001"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
这里的边界必须比“临时脚本”更严格:
|
||||
|
||||
- `discard` 必须带原因
|
||||
- `replay` 应支持限流 / limit
|
||||
- `flush` 应支持显式 `requested_by`
|
||||
|
||||
---
|
||||
|
||||
## 10. Activity Stream Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
|
||||
最小字段建议:
|
||||
|
||||
```json
|
||||
{
|
||||
"items": [
|
||||
{
|
||||
"kind": "ops_job",
|
||||
"activity_key": "ops-job:101",
|
||||
"title": "logs.collect",
|
||||
"subtitle": "ops-20260418064500-a1b2c3",
|
||||
"summary": "目标节点 mainland-worker-01 / 发起 web-ui / 方式 Node Agent",
|
||||
"summary_text": "目标节点 mainland-worker-01 / 发起 web-ui / 方式 Node Agent",
|
||||
"status": "running",
|
||||
"status_label": "执行中",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"occurred_at": "2026-04-18 06:45:08",
|
||||
"ui_intent_kind": "job_events",
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 101,
|
||||
"job_code": "ops-20260418064500-a1b2c3",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-01"
|
||||
}
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"status": "running",
|
||||
"status_label": "执行中",
|
||||
"summary_text": "最近活动里存在正在推进的现场、任务、编排或回传。",
|
||||
"filtered_total": 6,
|
||||
"unfiltered_total": 18,
|
||||
"latest_at": "2026-04-18 06:45:08"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- `activity-stream` 不是页面内部 table 数据
|
||||
- 它必须是:
|
||||
- 页面
|
||||
- CLI
|
||||
- Codex 驾驶员
|
||||
- 自动驾驶编排器
|
||||
共同消费的“最近活动摘要”
|
||||
|
||||
字段语义需要固定:
|
||||
|
||||
- `summary_text`
|
||||
- 是直接消费的活动短摘要
|
||||
- `status_label`
|
||||
- 是稳定的人类可读状态标签
|
||||
- `target_node_codes`
|
||||
- 是活动关联的节点口径,不允许调用方再从自然语言里猜
|
||||
- `ui_intent_kind`
|
||||
- 是当前活动默认落点类型
|
||||
- `focus_ref`
|
||||
- 是跨页面 / CLI / Codex 的统一定位对象
|
||||
|
||||
正式边界:
|
||||
|
||||
- 活动流应保留老字段以兼容现有页面
|
||||
- 新消费方优先用:
|
||||
- `summary_text`
|
||||
- `status_label`
|
||||
- `focus_ref`
|
||||
- 不允许新客户端再从 `title + subtitle + summary` 手工解析定位参数
|
||||
|
||||
## 11. 与 Driver / Playbook / Release 的边界关系
|
||||
|
||||
这份 contract 只负责观察面和现场治理,不负责替代其他 contract。
|
||||
|
||||
边界应明确为:
|
||||
|
||||
- `ops_driver_contract`
|
||||
- 决定“现在优先做什么”
|
||||
- `ops_playbook_contract`
|
||||
- 决定“标准多步动作怎么编排”
|
||||
- `release_hub_contract`
|
||||
- 决定“版本怎么发、怎么停、怎么回滚”
|
||||
- `ops_observability_contract`
|
||||
- 决定“现场事实是什么、当前缺口在哪里”
|
||||
|
||||
这样后续系统再扩展时,不会又回到“所有东西都写在 overview 一坨大 json 里”的状态。
|
||||
|
||||
---
|
||||
|
||||
## 12. 终局验收标准
|
||||
|
||||
如果这份 contract 真正落地,海外单脑控制面应该做到:
|
||||
|
||||
### 1. 不登录大陆机器也能回答现场问题
|
||||
|
||||
例如:
|
||||
|
||||
- 谁在跑
|
||||
- 谁在线但没跑
|
||||
- 谁近窗刚跑过
|
||||
- 谁日志没回来
|
||||
- 谁队列死信了
|
||||
|
||||
### 2. 不复制大日志也能先做判断
|
||||
|
||||
先看:
|
||||
|
||||
- execution scene
|
||||
- inspection overview
|
||||
- activity stream
|
||||
- delivery queue
|
||||
|
||||
再决定是否需要全量日志或 diagnostics bundle。
|
||||
|
||||
### 3. 页面、CLI、Codex 看到的是同一现场
|
||||
|
||||
不能出现:
|
||||
|
||||
- 页面说节点正在执行
|
||||
- CLI 说节点待命
|
||||
- Codex 说节点离线
|
||||
|
||||
### 4. 所有高风险动作都有正式观察落点
|
||||
|
||||
也就是:
|
||||
|
||||
- replay 后能看 job
|
||||
- discard 后能看审计
|
||||
- inspection 后能看节点收口
|
||||
- rollout 后能看 launchpad / gate / rollout jobs
|
||||
|
||||
这样这套系统才能真正从“调试台”升级成“驾驶舱”。
|
||||
563
docs/schemas/ops_playbook_contract.md
Normal file
563
docs/schemas/ops_playbook_contract.md
Normal file
@@ -0,0 +1,563 @@
|
||||
# domainCheck Ops Playbook Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 `playbook catalog / preview / run / events` 的正式 contract。
|
||||
|
||||
这层的定位不是“页面里的一个弹窗”,而是:
|
||||
|
||||
- 控制面的正式编排对象
|
||||
- 海外 Codex 驾驶员可消费的标准作业对象
|
||||
- Driver / Runbook / Release / Inspection 之间的统一中间层
|
||||
|
||||
核心原则:
|
||||
|
||||
- 一次标准编排 = 一次 `playbook run`
|
||||
- 页面、CLI、Codex 必须共用同一份 `playbook` / `playbook run` 结构
|
||||
- `playbook run` 只负责创建与聚合正式 `ops job`
|
||||
- 不允许前端再把“标准巡检 / 接管验收 / 现场日志”退回成零散按钮逻辑
|
||||
|
||||
建议与 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 配套阅读。
|
||||
|
||||
---
|
||||
|
||||
## 2. 当前接口面
|
||||
|
||||
当前已落地的主入口:
|
||||
|
||||
- `GET /api/v1/ops/playbooks`
|
||||
- `POST /api/v1/ops/playbooks/preview`
|
||||
- `POST /api/v1/ops/playbooks/execute`
|
||||
- `GET /api/v1/ops/playbook-runs`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
|
||||
- `POST /api/v1/ops/playbook-runs/{run_code}/rerun`
|
||||
- `POST /api/v1/ops/playbook-runs/{run_code}/cancel`
|
||||
|
||||
并且这层已经被以下入口复用:
|
||||
|
||||
- `driver action -> open_playbook_dialog`
|
||||
- `runbook sequence -> secondary_resolution`
|
||||
- `inspection`
|
||||
- `activity-stream`
|
||||
|
||||
所以后续不应再新造第四套“标准作业对象”。
|
||||
|
||||
---
|
||||
|
||||
## 3. Playbook Catalog Object
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/playbooks`
|
||||
- `preview / execute` 返回体中的 `playbook`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"key": "inspection.standard",
|
||||
"group_key": "diagnostics",
|
||||
"group_title": "标准巡检",
|
||||
"title": "标准巡检",
|
||||
"description": "按 健康快照 -> Worker 日志 -> 诊断包 的顺序,对目标节点做一次标准联调巡检。",
|
||||
"target_scope": "node-set",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"default_execution_mode_label": "Node Agent",
|
||||
"execution_modes": [
|
||||
"remote-agent",
|
||||
"ssh"
|
||||
],
|
||||
"execution_mode_options": [
|
||||
{
|
||||
"value": "remote-agent",
|
||||
"label": "Node Agent",
|
||||
"is_default": true
|
||||
},
|
||||
{
|
||||
"value": "ssh",
|
||||
"label": "SSH",
|
||||
"is_default": false
|
||||
}
|
||||
],
|
||||
"default_auto_approve": true,
|
||||
"stop_on_failure": true,
|
||||
"steps": [
|
||||
{
|
||||
"step_key": "health",
|
||||
"title": "采集健康快照",
|
||||
"template_key": "health.snapshot",
|
||||
"payload": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
正式约束:
|
||||
|
||||
- `key`
|
||||
- 是 Playbook 的稳定主键
|
||||
- `group_key`
|
||||
- 用于页面、CLI、Codex 做标准分组,不应用 `title` 猜
|
||||
- `execution_modes`
|
||||
- 是能力白名单,不是建议文案
|
||||
- `stop_on_failure`
|
||||
- 是编排级行为,不是某一步骤的临时说明
|
||||
|
||||
---
|
||||
|
||||
## 4. Playbook Preview Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/playbooks/preview`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"playbook_key": "inspection.standard",
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"execution_mode": "remote-agent",
|
||||
"auto_approve": true,
|
||||
"requested_by": "web-ui"
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"playbook": {},
|
||||
"requested_by": "web-ui",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"auto_approve": true,
|
||||
"target_nodes_total": 1,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"step_count": 3,
|
||||
"expected_jobs_total": 3,
|
||||
"template_keys": [
|
||||
"health.snapshot",
|
||||
"logs.collect",
|
||||
"diagnostics.collect"
|
||||
],
|
||||
"steps": [
|
||||
{
|
||||
"order": 1,
|
||||
"step_key": "health",
|
||||
"title": "采集健康快照",
|
||||
"template_key": "health.snapshot",
|
||||
"action": "health.snapshot",
|
||||
"payload": {},
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"auto_approve": true,
|
||||
"target_nodes_total": 1,
|
||||
"expected_jobs": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
预览阶段必须回答:
|
||||
|
||||
- 最终执行方式是什么
|
||||
- 总共会展开多少步
|
||||
- 每步会展开成什么模板
|
||||
- 总共会生成多少个 `ops job`
|
||||
|
||||
这样页面、CLI、Codex 才能在执行前看见“这次会做什么”。
|
||||
|
||||
---
|
||||
|
||||
## 5. Playbook Run Object
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/playbook-runs`
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}`
|
||||
- `POST /api/v1/ops/playbooks/execute`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_key": "inspection.standard",
|
||||
"playbook_title": "标准巡检",
|
||||
"group_key": "diagnostics",
|
||||
"group_title": "标准巡检",
|
||||
"requested_by": "web-ui",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"stop_on_failure": true,
|
||||
"step_count": 3,
|
||||
"expected_jobs_total": 3,
|
||||
"target_nodes_total": 1,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"created_at": "2026-04-18 06:30:00",
|
||||
"updated_at": "2026-04-18 06:31:10",
|
||||
"status": "running",
|
||||
"status_label": "收口中",
|
||||
"status_counts": {
|
||||
"queued": 1,
|
||||
"running": 1,
|
||||
"success": 1
|
||||
},
|
||||
"jobs_total": 3,
|
||||
"success_jobs_total": 1,
|
||||
"terminal_jobs_total": 1,
|
||||
"completion_percent": 33.3,
|
||||
"steps_total": 3,
|
||||
"steps_success": 1,
|
||||
"steps_running": 1,
|
||||
"steps_problem": 0,
|
||||
"steps_terminal": 1,
|
||||
"focus_level": "warning",
|
||||
"focus_step_key": "worker_logs",
|
||||
"focus_step_title": "收集 Worker 日志",
|
||||
"focus_summary": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
|
||||
"summary": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
|
||||
"summary_text": "收集 Worker 日志 当前仍在执行,先等待回执或检查对应节点事件流。",
|
||||
"focus_ref": {
|
||||
"kind": "playbook_run",
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_key": "inspection.standard",
|
||||
"group_key": "diagnostics",
|
||||
"focus_step_key": "worker_logs",
|
||||
"focus_step_title": "收集 Worker 日志",
|
||||
"event_key": "",
|
||||
"node_code": ""
|
||||
},
|
||||
"problem_steps": [],
|
||||
"active_steps": [],
|
||||
"latest_job": {},
|
||||
"steps": []
|
||||
}
|
||||
```
|
||||
|
||||
正式定义:
|
||||
|
||||
> `playbook run` 是一轮标准编排在控制面的聚合对象,不是某一条单独任务。
|
||||
|
||||
也就是说:
|
||||
|
||||
- `ops job` 是执行颗粒度
|
||||
- `playbook run` 是观察、重跑、取消、钻取的编排颗粒度
|
||||
|
||||
---
|
||||
|
||||
## 6. Playbook Step Object
|
||||
|
||||
`playbook run.steps[]` 的最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"step_key": "worker_logs",
|
||||
"title": "收集 Worker 日志",
|
||||
"template_key": "logs.collect",
|
||||
"order": 2,
|
||||
"jobs_total": 1,
|
||||
"nodes_total": 1,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"status_counts": {
|
||||
"running": 1
|
||||
},
|
||||
"terminal_jobs_total": 0,
|
||||
"success_jobs_total": 0,
|
||||
"status": "running",
|
||||
"status_label": "收口中",
|
||||
"completion_percent": 0.0,
|
||||
"summary": "收集 Worker 日志 当前处于执行中,可继续观察该步骤事件。",
|
||||
"summary_text": "收集 Worker 日志 当前处于执行中,可继续观察该步骤事件。",
|
||||
"focus_ref": {
|
||||
"kind": "playbook_run",
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_key": "inspection.standard",
|
||||
"group_key": "diagnostics",
|
||||
"focus_step_key": "worker_logs",
|
||||
"focus_step_title": "收集 Worker 日志",
|
||||
"event_key": "",
|
||||
"node_code": ""
|
||||
},
|
||||
"latest_job": {}
|
||||
}
|
||||
```
|
||||
|
||||
推荐 step 状态:
|
||||
|
||||
- `queued`
|
||||
- `running`
|
||||
- `success`
|
||||
- `attention`
|
||||
|
||||
规则:
|
||||
|
||||
- 只要该步骤出现失败、阻断、取消等终态问题,应收口为 `attention`
|
||||
- 页面与 Codex 都应优先盯 `problem_steps / active_steps`,而不是自己遍历所有子任务推断
|
||||
|
||||
---
|
||||
|
||||
## 7. Playbook Event Stream
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/playbook-runs/{run_code}/events`
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_run": {},
|
||||
"events": [
|
||||
{
|
||||
"job_id": 123,
|
||||
"job_code": "ops-20260418-123",
|
||||
"run_code": "playbook-20260418-001",
|
||||
"job_status": "running",
|
||||
"job_status_label": "收口中",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志",
|
||||
"event_type": "job_running",
|
||||
"level": "info",
|
||||
"level_label": "信息",
|
||||
"message": "开始收集 Worker 日志",
|
||||
"summary": "开始收集 Worker 日志",
|
||||
"summary_text": "开始收集 Worker 日志",
|
||||
"payload": {},
|
||||
"event_key": "job-event:1001",
|
||||
"occurred_at": "2026-04-18 06:31:00",
|
||||
"focus_ref": {
|
||||
"kind": "playbook_run",
|
||||
"run_code": "playbook-20260418-001",
|
||||
"playbook_key": "inspection.standard",
|
||||
"group_key": "diagnostics",
|
||||
"focus_step_key": "worker_logs",
|
||||
"focus_step_title": "收集 Worker 日志",
|
||||
"event_key": "job-event:1001",
|
||||
"node_code": "mainland-worker-01"
|
||||
},
|
||||
"created_at": "2026-04-18 06:31:00"
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"total": 1,
|
||||
"returned_total": 1,
|
||||
"available_step_counts": {
|
||||
"worker_logs": 1
|
||||
},
|
||||
"available_node_counts": {
|
||||
"mainland-worker-01": 1
|
||||
},
|
||||
"level_counts": {
|
||||
"info": 1
|
||||
},
|
||||
"event_type_counts": {
|
||||
"job_running": 1
|
||||
},
|
||||
"job_status_counts": {
|
||||
"running": 1
|
||||
},
|
||||
"latest_at": "2026-04-18 06:31:00",
|
||||
"filters": {
|
||||
"step_key": "",
|
||||
"node_code": "",
|
||||
"limit": 80
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
这层必须支持:
|
||||
|
||||
- 按 `step_key` 过滤
|
||||
- 按 `node_code` 过滤
|
||||
- 快速看到最近活动时间
|
||||
|
||||
否则 `playbook run` 只能看总览,不能真正钻取。
|
||||
|
||||
---
|
||||
|
||||
## 8. 正式 Playbook Key 语义
|
||||
|
||||
当前最值得冻结成正式对象的 key:
|
||||
|
||||
### 8.1 `onboarding.bootstrap`
|
||||
|
||||
定位:
|
||||
|
||||
- 生成节点接入工单
|
||||
|
||||
执行方式:
|
||||
|
||||
- 仅 `control-plane`
|
||||
|
||||
正式语义:
|
||||
|
||||
- 不直接接管节点
|
||||
- 只创建标准接入方案对象
|
||||
|
||||
### 8.2 `onboarding.acceptance`
|
||||
|
||||
定位:
|
||||
|
||||
- 新节点接管后标准验收
|
||||
|
||||
当前标准步骤:
|
||||
|
||||
1. `health.snapshot`
|
||||
2. `service.status(node-agent)`
|
||||
3. `logs.collect(node-agent)`
|
||||
4. `service.status(worker)`
|
||||
5. `logs.collect(worker)`
|
||||
|
||||
### 8.3 `inspection.standard`
|
||||
|
||||
定位:
|
||||
|
||||
- 标准巡检
|
||||
|
||||
当前标准步骤:
|
||||
|
||||
1. `health.snapshot`
|
||||
2. `logs.collect(worker)`
|
||||
3. `diagnostics.collect`
|
||||
|
||||
正式要求:
|
||||
|
||||
- 这是 release / rollout 之前最核心的健康观察面
|
||||
|
||||
### 8.4 `scene.logs.key`
|
||||
|
||||
定位:
|
||||
|
||||
- 关键现场日志样本
|
||||
|
||||
正式要求:
|
||||
|
||||
- 默认比 `scene.logs.full` 更轻
|
||||
- 适合先看现场,再决定是否升级取证
|
||||
|
||||
### 8.5 `scene.logs.full`
|
||||
|
||||
定位:
|
||||
|
||||
- 全量现场取证
|
||||
|
||||
正式要求:
|
||||
|
||||
- 应明确比 `scene.logs.key` 更重
|
||||
- 默认应同时补诊断包
|
||||
|
||||
### 8.6 `scene.diagnostics`
|
||||
|
||||
定位:
|
||||
|
||||
- 轻量诊断包
|
||||
|
||||
正式要求:
|
||||
|
||||
- 适合作为参与节点的常规取证动作
|
||||
|
||||
---
|
||||
|
||||
## 9. 执行模式 Contract
|
||||
|
||||
推荐模式:
|
||||
|
||||
- `remote-agent`
|
||||
- `ssh`
|
||||
- `control-plane`
|
||||
|
||||
正式约束:
|
||||
|
||||
- `remote-agent`
|
||||
- 正式日常运维默认模式
|
||||
- `ssh`
|
||||
- 仅用于首发接管与应急救援
|
||||
- `control-plane`
|
||||
- 只适合控制面本地对象生成或极少量控制面动作
|
||||
|
||||
页面、CLI、Codex 不能自己发明第四种临时模式。
|
||||
|
||||
---
|
||||
|
||||
## 10. Playbook Run 状态机
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `queued`
|
||||
- `running`
|
||||
- `success`
|
||||
- `attention`
|
||||
|
||||
含义:
|
||||
|
||||
- `queued`
|
||||
- 子任务刚创建,尚未出现有效执行推进
|
||||
- `running`
|
||||
- 已有子任务进入 `dispatching / running / awaiting_approval`
|
||||
- `success`
|
||||
- 全部子任务收口成功
|
||||
- `attention`
|
||||
- 任一步骤出现失败、阻断、取消或部分成功,需要人工关注
|
||||
|
||||
正式规则:
|
||||
|
||||
- 页面上的“标准巡检状态”
|
||||
- `activity-stream` 中的 playbook run 摘要
|
||||
- Driver / Codex 的推荐动作
|
||||
|
||||
都必须共用这一套状态语义。
|
||||
|
||||
---
|
||||
|
||||
## 11. 与 Driver / Runbook / Release 的关系
|
||||
|
||||
正式收口规则:
|
||||
|
||||
- Driver 不直接“模拟巡检”
|
||||
- 应优先跳转或创建 `playbook run`
|
||||
- Runbook Sequence 不直接展开 shell
|
||||
- 应优先收口到 `driver action` 或 `playbook`
|
||||
- Release / Rollout 不自己维护另一套巡检摘要
|
||||
- 应优先消费 `inspection.standard` 的最近结果
|
||||
|
||||
也就是说:
|
||||
|
||||
- `scene` 是现场观察对象
|
||||
- `inspection` 是标准健康对象
|
||||
- `release / rollout` 是发布对象
|
||||
|
||||
三者不能再各自维护不同的“日志 / 诊断 / 巡检”定义。
|
||||
|
||||
---
|
||||
|
||||
## 12. 正式要求
|
||||
|
||||
后续继续演进时,必须保持:
|
||||
|
||||
1. `playbook catalog`
|
||||
- 是稳定 contract,而不是页面枚举
|
||||
2. `playbook preview`
|
||||
- 是执行前的唯一结构化预览面
|
||||
3. `playbook run`
|
||||
- 是页面、CLI、Codex 共同观察的一轮编排对象
|
||||
4. `playbook events`
|
||||
- 是整轮编排的钻取视角
|
||||
5. 接管验收、标准巡检、现场观察
|
||||
- 优先都收进 `playbook`
|
||||
|
||||
这样这套编排才能成为真正的正式平台能力,而不是一组“看起来像流程”的按钮。
|
||||
372
docs/schemas/ops_stack_diagnosis_contract.md
Normal file
372
docs/schemas/ops_stack_diagnosis_contract.md
Normal file
@@ -0,0 +1,372 @@
|
||||
# domainCheck Ops Stack Diagnosis Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结海外单脑控制面的“总检入口” contract。
|
||||
|
||||
它的职责不是替代:
|
||||
|
||||
- `overview`
|
||||
- `link-snapshot`
|
||||
- `inspection-overview`
|
||||
- `release launchpad`
|
||||
- `activity-stream`
|
||||
|
||||
而是把这些正式 contract 再收口成一份:
|
||||
|
||||
> 第一现场诊断结果
|
||||
|
||||
让下面这些消费者都先看同一份判断:
|
||||
|
||||
- 海外 Ops Center 页面
|
||||
- 海外 Codex 驾驶员
|
||||
- CLI 总检脚本
|
||||
- 后台按钮自动诊断
|
||||
|
||||
正式原则:
|
||||
|
||||
- 总检不能只回一段说明文字
|
||||
- 总检必须给出结构化问题清单
|
||||
- 总检必须给出默认下一步动作
|
||||
- 总检必须给出可以直接执行的推荐命令
|
||||
|
||||
---
|
||||
|
||||
## 2. 入口接口
|
||||
|
||||
主入口:
|
||||
|
||||
- `GET /api/v1/ops/go-live-summary`
|
||||
- `GET /api/v1/ops/stack-diagnosis`
|
||||
|
||||
配套来源:
|
||||
|
||||
- `GET /api/v1/ops/contracts`
|
||||
- `GET /api/v1/ops/link-snapshot`
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `GET /api/v1/ops/nodes`
|
||||
- `GET /api/v1/ops/releases/launchpad`
|
||||
- `GET /api/v1/ops/playbook-runs`
|
||||
- `GET /api/v1/ops/activity-stream`
|
||||
|
||||
推荐 query:
|
||||
|
||||
- `base_url`
|
||||
|
||||
用途:
|
||||
|
||||
- `go-live-summary`
|
||||
- 给 Ops Center 顶部收口卡、CLI 总检、Codex 驾驶员直接消费
|
||||
- 返回更偏“上线判断”的稳定摘要
|
||||
- 用于生成 `next_step.command`
|
||||
- 用于生成 `quick_commands`
|
||||
- 让 CLI / Codex / 页面复制出来的命令口径一致
|
||||
|
||||
---
|
||||
|
||||
## 3. 顶层结构
|
||||
|
||||
建议最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"generated_at": "2026-04-18 09:00:00",
|
||||
"diagnosis": {},
|
||||
"api": {},
|
||||
"surface_matrix": {},
|
||||
"contracts": {},
|
||||
"link_snapshot": {},
|
||||
"overview": {},
|
||||
"managed_nodes": {},
|
||||
"release_hub": {},
|
||||
"playbook_runs": {},
|
||||
"activity_stream": {}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- 顶层每一段都必须是稳定 key
|
||||
- 页面、CLI、Codex 不允许自己再拼第 2 套总检结构
|
||||
- `go-live-summary`
|
||||
- 是 `stack-diagnosis` 的稳定摘要层
|
||||
- 必须至少给出:
|
||||
- `go_live_status`
|
||||
- `blocking_reasons`
|
||||
- `warnings`
|
||||
- `launchpad_recommended_target_node_code`
|
||||
- `launchpad_recommended_recovery_label`
|
||||
- `launchpad_recommended_recovery_summary`
|
||||
- `launchpad_onboarding_bootstrap_pending_nodes`
|
||||
- `launchpad_onboarding_acceptance_ready_nodes`
|
||||
- `next_step_action_code`
|
||||
- `operator_title`
|
||||
- `recommended_commands`
|
||||
- `playbook_runs.problem_runs[*]`
|
||||
- 需要保留 `focus_ref`
|
||||
- 这样 CLI / Codex / 页面才能直接定位到问题编排的正式焦点
|
||||
- `activity_stream.top_items[*]`
|
||||
- 需要保留:
|
||||
- `occurred_at`
|
||||
- `focus_ref`
|
||||
- `source_focus_ref`
|
||||
- `ui_intent_kind`
|
||||
- 这样总检不会把“观察摘要”再次压扁成一段不可定位的纯文本
|
||||
- `go-live-summary` + `stack-diagnosis` + `driver-feed.automation_coverage`
|
||||
- 必须能共同驱动 Ops Center 首屏
|
||||
- 不允许页面自己从局部接口重新推导一份“收口状态”
|
||||
|
||||
---
|
||||
|
||||
## 4. Diagnosis Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `stack_diagnosis.diagnosis`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"contract_key": "ops_stack_diagnosis_contract",
|
||||
"contract_version": "v1",
|
||||
"registry_version": "2026-04-18",
|
||||
"base_url": "http://127.0.0.1:8100",
|
||||
"surface_status": "partial",
|
||||
"automation_status": "blocked",
|
||||
"stack_status": "blocked",
|
||||
"issue_total": 4,
|
||||
"blocking_issue_total": 1,
|
||||
"warning_issue_total": 3,
|
||||
"missing_surfaces": [
|
||||
"contracts"
|
||||
],
|
||||
"launchpad_recommended_target_node_code": "mainland-worker-01",
|
||||
"launchpad_recommended_recovery_label": "待执行接入",
|
||||
"launchpad_recommended_recovery_summary": "mainland-worker-01 还缺接入收口,先执行 bootstrap。",
|
||||
"launchpad_onboarding_bootstrap_pending_nodes": 1,
|
||||
"launchpad_onboarding_acceptance_ready_nodes": 0,
|
||||
"next_step": {
|
||||
"action_code": "bootstrap_run",
|
||||
"source": "issue:managed_nodes_agent_pending",
|
||||
"reason": "mainland-worker-01 还缺接入收口,先执行 bootstrap。",
|
||||
"command": "bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 bootstrap_run",
|
||||
"focus_ref": {
|
||||
"kind": "managed_node",
|
||||
"node_code": "mainland-worker-01"
|
||||
}
|
||||
},
|
||||
"recommended_actions": [],
|
||||
"issues": [],
|
||||
"operator_hints": [],
|
||||
"quick_commands": []
|
||||
}
|
||||
```
|
||||
|
||||
状态语义必须固定:
|
||||
|
||||
- `surface_status`
|
||||
- `healthy`
|
||||
- `partial`
|
||||
- `broken`
|
||||
- `automation_status`
|
||||
- `ready`
|
||||
- `attention`
|
||||
- `blocked`
|
||||
- `stack_status`
|
||||
- 总结论,面向人和程序
|
||||
- 推荐值为 `ready / attention / blocked`
|
||||
|
||||
正式要求:
|
||||
|
||||
- `next_step` 是默认下一步,而不是“可能动作之一”
|
||||
- 当首个缺口节点已经明确收敛为 `bootstrap_run / run_acceptance` 时,`next_step.action_code` 不应再退回成泛化的 `fix_managed_nodes`
|
||||
- 当存在 `runtime_build_schema_stale` 时,`next_step.action_code` 必须优先收敛为 `api-restart`
|
||||
- `issues` 必须按可处理性组织,而不是只堆原始异常文本
|
||||
- `quick_commands` 必须是可以直接复制执行的命令
|
||||
- `next_step.focus_ref`
|
||||
- 是首屏“定位下一步”的正式落点
|
||||
- 页面、CLI、Codex 不允许再根据 `action_code` 反推要跳去哪个区域
|
||||
- `diagnosis.launchpad_recommended_target_node_code`
|
||||
- 代表 stack-diagnosis 已经把 Release Hub 收敛出的首个接管缺口直接下沉到总检摘要
|
||||
- 页面、CLI、doctor-export、Codex 不应再分别回源二次推导
|
||||
- `diagnosis.launchpad_onboarding_bootstrap_pending_nodes / launchpad_onboarding_acceptance_ready_nodes`
|
||||
- 代表当前接管缺口的阶段性计数
|
||||
- 用来帮助总检直接区分“待 bootstrap”与“待 acceptance”
|
||||
- `go-live-summary.launchpad_recommended_target_node_code`
|
||||
- 是 launchpad 已经收敛出的首个接管缺口节点
|
||||
- 页面、CLI、Codex 应优先直接显示它,而不是重新扫描 gap rows
|
||||
- `go-live-summary.launchpad_onboarding_bootstrap_pending_nodes / launchpad_onboarding_acceptance_ready_nodes`
|
||||
- 用于区分当前是“待接入”还是“待验收”
|
||||
- 这两个计数应作为上线收口阶段的正式 warning 来源,而不是页面本地再统计
|
||||
|
||||
---
|
||||
|
||||
## 5. Issue Contract
|
||||
|
||||
`diagnosis.issues[]` 最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": "managed_nodes_agent_pending",
|
||||
"severity": "blocked",
|
||||
"layer": "managed_nodes",
|
||||
"summary": "托管节点虽然已经纳入 Ops Center,但 0 台 remote-agent 就绪,当前第一动作已收敛为首台节点接入收口。",
|
||||
"detail": "managed_enabled=3,remote_access_ready=0;首个缺口节点 mainland-worker-01:待执行接入。",
|
||||
"action_code": "bootstrap_run",
|
||||
"focus_ref": {
|
||||
"kind": "managed_node",
|
||||
"node_code": "mainland-worker-01"
|
||||
},
|
||||
"commands": [
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 bootstrap_run",
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-worker-01"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
严重级别建议固定为:
|
||||
|
||||
- `blocked`
|
||||
- `warning`
|
||||
- `info`
|
||||
|
||||
正式要求:
|
||||
|
||||
- `summary` 用于列表展示
|
||||
- `detail` 用于展开解释
|
||||
- `commands` 用于终端直接执行
|
||||
- 不允许只有中文文案,没有结构化定位信息
|
||||
|
||||
对于运行时版本漂移,还应允许出现这类 issue:
|
||||
|
||||
```json
|
||||
{
|
||||
"code": "runtime_build_schema_stale",
|
||||
"severity": "warning",
|
||||
"layer": "runtime_build_info.schema",
|
||||
"summary": "仓库已经具备新的节点接管能力,但运行中的 build-info 仍未声明对应路由键,当前更像 API 还没重启到最新代码。",
|
||||
"detail": "supports_install_command_block=true;route_surface_declares_bootstrap_plan=false",
|
||||
"action_code": "api-restart",
|
||||
"focus_ref": {
|
||||
"kind": "runtime_build_info",
|
||||
"section": "schema",
|
||||
"expected_route_key": "ops_node_handover_bootstrap_plan"
|
||||
},
|
||||
"commands": [
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100",
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary",
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-worker-01"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Surface Matrix Contract
|
||||
|
||||
来源:
|
||||
|
||||
- `stack_diagnosis.surface_matrix`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"status_counts": {
|
||||
"total": 8,
|
||||
"available": 7,
|
||||
"missing": 1
|
||||
},
|
||||
"items": [
|
||||
{
|
||||
"name": "contracts",
|
||||
"available": false
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. Runtime Build Info Extension
|
||||
|
||||
`stack_diagnosis.runtime_build_info` 以及 `/api/v1/runtime/build-info` 建议至少包含:
|
||||
|
||||
```json
|
||||
{
|
||||
"repository_capabilities": {
|
||||
"supports_install_command_block": true,
|
||||
"supports_multi_layout_bootstrap": true
|
||||
},
|
||||
"route_surface_declares_bootstrap_plan": false,
|
||||
"runtime_schema_stale": true
|
||||
}
|
||||
```
|
||||
|
||||
语义要求:
|
||||
|
||||
- `repository_capabilities`
|
||||
- 表示当前仓库代码本身具备的能力
|
||||
- `route_surface_declares_bootstrap_plan`
|
||||
- 表示运行中的 API build-info 是否已经声明节点级 bootstrap-plan 路由键
|
||||
- `runtime_schema_stale`
|
||||
- 表示“仓库能力已到位,但运行中的 API 结构还停留在旧版”
|
||||
- 这是典型的“服务没重启到最新代码”信号,不应误判成“功能没开发完”
|
||||
|
||||
作用:
|
||||
|
||||
- 让页面和 CLI 直观看出“缺的是哪一层”
|
||||
- 避免只看到最终 blocked,却不知道缺的是 contract、launchpad 还是 activity-stream
|
||||
|
||||
---
|
||||
|
||||
## 7. 与其他 Contract 的关系
|
||||
|
||||
`ops_stack_diagnosis_contract` 的定位是:
|
||||
|
||||
- 不替代 `ops_driver_contract`
|
||||
- 不替代 `ops_observability_contract`
|
||||
- 不替代 `release_hub_contract`
|
||||
|
||||
它负责:
|
||||
|
||||
- 把它们统一收口成第一现场诊断
|
||||
|
||||
对于 Ops Center 页面还必须补充一条:
|
||||
|
||||
- 首屏负责“结论 + 默认下一步 + 复制命令 + 动作回执”
|
||||
- 下方详情区负责“issue / surface / launchpad / activity / runbook”的展开与下钻
|
||||
- 不允许详情区覆盖首屏的默认下一步结论
|
||||
|
||||
所以正确消费顺序应该是:
|
||||
|
||||
1. 先看 `stack-diagnosis`
|
||||
2. 再根据 `issues / next_step / focus_ref` 下钻到:
|
||||
- `overview`
|
||||
- `link-snapshot`
|
||||
- `inspection-overview`
|
||||
- `release launchpad`
|
||||
- `activity-stream`
|
||||
|
||||
---
|
||||
|
||||
## 8. CLI / Codex / 页面统一约束
|
||||
|
||||
CLI:
|
||||
|
||||
- `check_ops_center_stack.sh` 应优先调用 `GET /api/v1/ops/stack-diagnosis`
|
||||
|
||||
Codex:
|
||||
|
||||
- 默认先读取 `stack-diagnosis`
|
||||
- 再决定是继续 `driver-resolve`、`doctor` 还是细分检查
|
||||
|
||||
页面:
|
||||
|
||||
- 顶部总检卡片必须优先消费 `diagnosis`
|
||||
- 不能重新从多个 endpoint 拼一份影子状态
|
||||
|
||||
只有这样,海外单脑控制面才能真正进入:
|
||||
|
||||
> 单入口观察,分层下钻,统一执行
|
||||
609
docs/schemas/release_hub_contract.md
Normal file
609
docs/schemas/release_hub_contract.md
Normal file
@@ -0,0 +1,609 @@
|
||||
# domainCheck Release Hub Contract
|
||||
|
||||
## 1. 目标
|
||||
|
||||
这份文档用于冻结 Release / Rollout / Launchpad / Gate 的正式 contract。
|
||||
|
||||
核心原则:
|
||||
|
||||
- 先有 Release,再有 Rollout
|
||||
- Release 是不可变版本对象
|
||||
- Rollout 是分批推进对象
|
||||
- Gate 是是否可发的统一门禁对象
|
||||
- Launchpad 是给页面与 Codex 消费的发布驾驶舱对象
|
||||
|
||||
建议与 [ops_job_contract.md](/www/wwwroot/getDomain/docs/schemas/ops_job_contract.md:1) 配套阅读,因为 Rollout 的真正执行颗粒度最终仍然是 `ops job`。
|
||||
|
||||
---
|
||||
|
||||
## 2. Release Object
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/releases`
|
||||
- `GET /api/v1/ops/releases`
|
||||
- `GET /api/v1/ops/releases/latest`
|
||||
- `GET /api/v1/ops/releases/{id}`
|
||||
- `POST /api/v1/ops/releases/{id}/activate`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"commit_sha": "bd6bcb2",
|
||||
"status": "ready",
|
||||
"status_label": "已就绪",
|
||||
"artifact_url": "https://release.example.com/domaincheck-2026.04.18.tgz",
|
||||
"checksum": "sha256:...",
|
||||
"notes": "",
|
||||
"metadata": {},
|
||||
"created_by": "www",
|
||||
"created_at": "2026-04-18 05:30:00",
|
||||
"activated_at": "2026-04-18 05:35:00",
|
||||
"summary": "2026.04.18+bd6bcb2 已就绪,制品与校验信息齐备,可进入 Rollout。",
|
||||
"summary_text": "2026.04.18+bd6bcb2 已就绪,制品与校验信息齐备,可进入 Rollout。",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "release_detail"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
约束:
|
||||
|
||||
- 同一 `release_version` 只对应一份发布物
|
||||
- Release 创建后不应重新指向另一份 artifact
|
||||
- `artifact_url + checksum` 是发布可验证性的最小基线
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `draft`
|
||||
- `ready`
|
||||
- `active`
|
||||
- `archived`
|
||||
|
||||
---
|
||||
|
||||
## 3. Rollout Object
|
||||
|
||||
来源:
|
||||
|
||||
- `POST /api/v1/ops/releases/{id}/rollouts`
|
||||
- `GET /api/v1/ops/rollouts/{id}`
|
||||
- `GET /api/v1/ops/rollouts/{id}/jobs`
|
||||
- `POST /api/v1/ops/rollouts/{id}/advance`
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": 11,
|
||||
"release_id": 1,
|
||||
"rollout_code": "rollout-20260418-001",
|
||||
"target_selector": {
|
||||
"region": "mainland",
|
||||
"role": "worker"
|
||||
},
|
||||
"target_nodes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"policy": {
|
||||
"batch_size": 1,
|
||||
"require_approval_between_batches": true,
|
||||
"rollback_on_failure": true
|
||||
},
|
||||
"status": "running",
|
||||
"status_label": "执行中",
|
||||
"batch_cursor": 1,
|
||||
"batches_total": 2,
|
||||
"jobs_total": 2,
|
||||
"jobs_created": 1,
|
||||
"result_summary": {},
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"summary": "当前已覆盖 1/2 台节点,仍有 1 条任务执行中。",
|
||||
"summary_text": "当前已覆盖 1/2 台节点,仍有 1 条任务执行中。",
|
||||
"focus_ref": {
|
||||
"kind": "release_rollout",
|
||||
"rollout_id": 11,
|
||||
"rollout_code": "rollout-20260418-001",
|
||||
"release_id": 1,
|
||||
"section": "rollout_jobs"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式定义:
|
||||
|
||||
> 某个 Release 在某批目标节点上的一次分批推进计划
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `planned`
|
||||
- `running`
|
||||
- `awaiting_approval`
|
||||
- `ready_for_next_batch`
|
||||
- `halted`
|
||||
- `completed`
|
||||
- `completed_with_issues`
|
||||
|
||||
---
|
||||
|
||||
## 4. Default Rollout Gate
|
||||
|
||||
来源:
|
||||
|
||||
- `GET /api/v1/ops/overview`
|
||||
- `overview.release_hub.default_rollout_gate`
|
||||
|
||||
这层是 Release Hub 最关键的后端 gate contract。
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "attention",
|
||||
"status_label": "待处理",
|
||||
"status_type": "warning",
|
||||
"summary": "当前已有 Release,但默认目标节点中仍有未接管或未通过巡检的节点。",
|
||||
"summary_text": "当前已有 Release,但默认目标节点中仍有未接管或未通过巡检的节点。",
|
||||
"release": {
|
||||
"id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"status": "ready",
|
||||
"status_label": "已就绪"
|
||||
},
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"default_target_node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"blocking_reasons": [],
|
||||
"warning_reasons": [
|
||||
"存在未通过标准巡检的节点"
|
||||
],
|
||||
"recommendations": [
|
||||
"先执行标准巡检,再进入 Rollout"
|
||||
],
|
||||
"operational_readiness": {
|
||||
"summary": "remote-agent 就绪 1/2,标准巡检通过 1/2",
|
||||
"rows": []
|
||||
},
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "default_rollout_gate"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐状态:
|
||||
|
||||
- `missing_release`
|
||||
- `release_not_ready`
|
||||
- `artifact_missing`
|
||||
- `no_targets`
|
||||
- `blocked`
|
||||
- `attention`
|
||||
- `ready`
|
||||
|
||||
要求:
|
||||
|
||||
- 首页驾驶建议
|
||||
- Release 页面门禁
|
||||
- Rollout 预检
|
||||
- Codex 自动驾驶
|
||||
|
||||
必须共用这同一份 Gate。
|
||||
|
||||
---
|
||||
|
||||
## 5. Default Rollout Execution
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.release_hub.default_rollout_execution`
|
||||
- `overview.release_hub.default_rollout_gate_options`
|
||||
|
||||
作用:
|
||||
|
||||
- 告诉页面 / Codex 这轮默认应使用什么执行方式
|
||||
- 给出其他可选 gate / mode
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"recommended_mode": "remote-agent",
|
||||
"recommended_mode_label": "Node Agent",
|
||||
"recommended_gate": {},
|
||||
"gates": {
|
||||
"remote-agent": {},
|
||||
"ssh": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
推荐执行模式:
|
||||
|
||||
- `remote-agent`
|
||||
- `ssh`
|
||||
- `control-plane`
|
||||
|
||||
说明:
|
||||
|
||||
- 正式发布优先 `remote-agent`
|
||||
- `ssh` 只作为首发接管与紧急救援兜底
|
||||
- `control-plane` 不应用来承载大规模 Worker 发布
|
||||
|
||||
---
|
||||
|
||||
## 6. Release Launchpad
|
||||
|
||||
来源:
|
||||
|
||||
- `overview.release_hub.launchpad`
|
||||
|
||||
作用:
|
||||
|
||||
- 给页面与 Codex 提供更偏“驾驶舱”的发布聚合视角
|
||||
|
||||
最小字段:
|
||||
|
||||
```json
|
||||
{
|
||||
"latest_package": {},
|
||||
"latest_release": {},
|
||||
"worker_rollout_preview": {},
|
||||
"control_rollout_preview": {},
|
||||
"launchpad_status": {
|
||||
"status": "attention",
|
||||
"status_label": "待处理",
|
||||
"summary": "建议先完成 Worker 点状灰度,再决定是否推进控制面。",
|
||||
"summary_text": "建议先完成 Worker 点状灰度,再决定是否推进控制面。",
|
||||
"recommended_action_code": "open_release_deploy_worker",
|
||||
"recommended_target_node_code": "mainland-worker-01",
|
||||
"recommended_recovery_label": "签发接入工单",
|
||||
"recommended_recovery_summary": "当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。",
|
||||
"onboarding_bootstrap_pending_nodes": 1,
|
||||
"onboarding_acceptance_ready_nodes": 0,
|
||||
"recommended_mode": "remote-agent",
|
||||
"recommended_mode_label": "Node Agent",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "release_launchpad",
|
||||
"mode": "worker"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`launchpad_status` 至少应回答:
|
||||
|
||||
- 当前先补什么
|
||||
- 推荐动作码是什么
|
||||
- 当前首个缺口节点是谁
|
||||
- 当前恢复动作的人类标签是什么
|
||||
- 当前缺口更偏“待接入”还是“待验收”
|
||||
- 推荐执行方式是什么
|
||||
- 先看 Worker 预案还是先看 Control 预案
|
||||
|
||||
正式约束:
|
||||
|
||||
- `recommended_target_node_code`
|
||||
- 一旦 launchpad 已经收敛出首个缺口节点,就应直接返回
|
||||
- 页面、CLI、Codex 不应再分别从 gap rows 中自行挑第一台
|
||||
- `recommended_recovery_label`
|
||||
- 是给人看的动作标题,例如:
|
||||
- `签发接入工单`
|
||||
- `补执行面接管`
|
||||
- `执行接管验收`
|
||||
- `recommended_recovery_summary`
|
||||
- 是解释“为什么当前默认不是发版,而是先补接管”的正式摘要
|
||||
- `onboarding_bootstrap_pending_nodes / onboarding_acceptance_ready_nodes`
|
||||
- 是 launchpad 汇总层面的正式计数
|
||||
- 允许被 `go-live-summary`、CLI 摘要、Codex brief 直接复用
|
||||
|
||||
---
|
||||
|
||||
## 7. 发布动作 contract
|
||||
|
||||
当前发布相关动作不应再退回“通用发任务”语义,而应正式区分:
|
||||
|
||||
- `deploy.release.control`
|
||||
- `deploy.release.worker`
|
||||
- `deploy.release.custom`
|
||||
|
||||
这三个动作必须共用:
|
||||
|
||||
- 同一份 Gate 判断
|
||||
- 同一份 Release 版本对象
|
||||
- 同一份 Rollout 编排结果
|
||||
|
||||
---
|
||||
|
||||
## 8. Artifact 不可变要求
|
||||
|
||||
上线前建议把 Release 供应链固定为:
|
||||
|
||||
1. 构建产物
|
||||
2. 生成 `checksum`
|
||||
3. 生成 `manifest`
|
||||
4. 上传 artifact
|
||||
5. 创建 Release
|
||||
6. 激活 channel
|
||||
7. 创建 Rollout
|
||||
|
||||
Node Agent 侧部署链路要求:
|
||||
|
||||
1. 下载 artifact
|
||||
2. 校验 checksum
|
||||
3. 解压到 `releases/<version>`
|
||||
4. 切换 `current`
|
||||
5. 重启服务
|
||||
6. 健康检查
|
||||
7. 失败回滚
|
||||
|
||||
---
|
||||
|
||||
## 9. Rollout 与 Ops Job 的关系
|
||||
|
||||
约束:
|
||||
|
||||
- Rollout 是分批推进对象
|
||||
- Ops Job 是单次动作对象
|
||||
- Rollout 通过一批或多批 `deploy.release.*` Job 落地
|
||||
|
||||
控制面责任:
|
||||
|
||||
- 决定下一批是否生成 Job
|
||||
- 汇总 Job 结果
|
||||
- 反推 Rollout 状态
|
||||
- 决定是否暂停、继续或回滚
|
||||
|
||||
Node Agent 不负责:
|
||||
|
||||
- 推进下一批
|
||||
- 修改 Rollout 策略
|
||||
- 修改 Release 状态
|
||||
|
||||
---
|
||||
|
||||
## 10. 页面 / Codex 共同消费约束
|
||||
|
||||
以下对象必须作为稳定后端 contract 存在:
|
||||
|
||||
- `default_rollout_gate`
|
||||
- `default_rollout_execution`
|
||||
- `launchpad_status`
|
||||
- `worker_rollout_preview`
|
||||
- `control_rollout_preview`
|
||||
|
||||
页面只负责展示和少量交互兜底。
|
||||
|
||||
Codex 只负责消费后端给出的:
|
||||
|
||||
- 当前是否可发
|
||||
- 建议先发哪一侧
|
||||
- 应该使用什么 mode
|
||||
- 下一步推荐动作是什么
|
||||
|
||||
---
|
||||
|
||||
## 11. Artifact Manifest Contract
|
||||
|
||||
Release Hub 不能只停在:
|
||||
|
||||
- `artifact_url`
|
||||
- `checksum`
|
||||
|
||||
这只能证明“有个包”,还不能证明“这个包会把哪些服务改成什么样”。
|
||||
|
||||
建议把 `release.metadata.manifest` 固定为正式对象。
|
||||
|
||||
最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"artifact_kind": "tarball",
|
||||
"package_name": "domaincheck-2026.04.18+bd6bcb2.tgz",
|
||||
"package_size_bytes": 186542331,
|
||||
"checksum": "sha256:...",
|
||||
"build_id": "build-20260418-001",
|
||||
"commit_sha": "bd6bcb2",
|
||||
"built_at": "2026-04-18 05:20:00",
|
||||
"included_services": [
|
||||
"domaincheck-api",
|
||||
"domaincheck-worker",
|
||||
"domaincheck-sync-agent",
|
||||
"domaincheck-node-agent"
|
||||
],
|
||||
"required_env_keys": [
|
||||
"NODE_CODE",
|
||||
"NODE_REGION",
|
||||
"NODE_ROLE"
|
||||
],
|
||||
"health_checks": [
|
||||
{
|
||||
"name": "api_health",
|
||||
"type": "http",
|
||||
"url": "http://127.0.0.1:8100/health"
|
||||
}
|
||||
],
|
||||
"rollback_hint": "切回上一个 current 并重启对应服务"
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- Release 创建后,`manifest` 不应再被覆盖成另一份语义不同的发布物
|
||||
- Node Agent、页面、Codex 必须共用这同一份 manifest
|
||||
- 是否可发、怎么回滚、要验哪些服务,不应再散落在:
|
||||
- shell 脚本
|
||||
- 页面表单
|
||||
- 人工脑补
|
||||
|
||||
---
|
||||
|
||||
## 12. Operational Readiness Row Contract
|
||||
|
||||
`default_rollout_gate.operational_readiness.rows[]` 不应只是“几行提醒文案”,而应成为逐节点门禁对象。
|
||||
|
||||
最小结构:
|
||||
|
||||
```json
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"agent_managed": true,
|
||||
"execution_mode_ready": true,
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"inspection_status": "healthy",
|
||||
"inspection_status_label": "已通过",
|
||||
"current_release_version": "2026.04.11+1921318",
|
||||
"desired_release_version": "2026.04.18+bd6bcb2",
|
||||
"risk_level": "low",
|
||||
"blocking_reasons": [],
|
||||
"warning_reasons": [],
|
||||
"recommended_action_code": "",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "default_rollout_gate"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- `rows[]` 应能直接回答:
|
||||
- 这台节点能不能进下一轮 rollout
|
||||
- 不能的话卡在哪
|
||||
- 只是 warning 还是 hard block
|
||||
- 页面、CLI、Codex 不应再从:
|
||||
- inspection
|
||||
- runtime
|
||||
- nodes
|
||||
三处数据手工拼门禁结论
|
||||
|
||||
推荐风险分级:
|
||||
|
||||
- `low`
|
||||
- `medium`
|
||||
- `high`
|
||||
- `blocked`
|
||||
|
||||
---
|
||||
|
||||
## 13. Rollout Preview Contract
|
||||
|
||||
真正创建 Rollout 之前,还应保留一层正式 preview,而不是由页面自己预估批次和风险。
|
||||
|
||||
建议入口:
|
||||
|
||||
- `POST /api/v1/ops/releases/{id}/rollouts/preview`
|
||||
|
||||
最小请求体:
|
||||
|
||||
```json
|
||||
{
|
||||
"release_id": 1,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"policy": {
|
||||
"batch_size": 1,
|
||||
"require_approval_between_batches": true,
|
||||
"rollback_on_failure": true
|
||||
},
|
||||
"execution_mode": "remote-agent"
|
||||
}
|
||||
```
|
||||
|
||||
最小返回体:
|
||||
|
||||
```json
|
||||
{
|
||||
"release": {},
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "Node Agent",
|
||||
"target_nodes_total": 2,
|
||||
"target_node_codes": [
|
||||
"mainland-worker-01",
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"expected_batches_total": 2,
|
||||
"expected_jobs_total": 2,
|
||||
"gate": {},
|
||||
"batch_plan": [
|
||||
{
|
||||
"batch_no": 1,
|
||||
"node_codes": [
|
||||
"mainland-worker-01"
|
||||
],
|
||||
"risk_level": "low"
|
||||
},
|
||||
{
|
||||
"batch_no": 2,
|
||||
"node_codes": [
|
||||
"mainland-controller-01"
|
||||
],
|
||||
"risk_level": "medium"
|
||||
}
|
||||
],
|
||||
"summary": "建议先灰度 1 台 Worker,再推进 Controller。",
|
||||
"summary_text": "建议先灰度 1 台 Worker,再推进 Controller。",
|
||||
"focus_ref": {
|
||||
"kind": "release_hub",
|
||||
"release_id": 1,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"rollout_id": 0,
|
||||
"rollout_code": "",
|
||||
"section": "rollout_preview"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
正式要求:
|
||||
|
||||
- preview 是:
|
||||
- 页面发 rollout 前的统一预检
|
||||
- Codex 自动驾驶发 rollout 前的统一预检
|
||||
- CLI 批量发布前的统一预检
|
||||
- 不允许前端重新自己算:
|
||||
- 该分几批
|
||||
- 哪台应该先发
|
||||
- 哪台属于高风险
|
||||
|
||||
只有这样,Release Hub 才是真正的发布控制面,而不是“能看版本的页面”。
|
||||
633
domain-api/app/api/routes/ops.py
Normal file
633
domain-api/app/api/routes/ops.py
Normal file
@@ -0,0 +1,633 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.schemas.common import ApiResponse
|
||||
from app.services.ops_agent_service import (
|
||||
build_managed_node_handover_bootstrap_plan,
|
||||
execute_managed_node_onboarding_bootstrap,
|
||||
execute_managed_node_onboarding_acceptance,
|
||||
execute_managed_node_onboarding_recovery,
|
||||
get_managed_node_delivery_queue,
|
||||
get_managed_node_handover,
|
||||
get_managed_node_onboarding,
|
||||
list_managed_node_delivery_queue_records,
|
||||
list_managed_nodes_with_agent_state,
|
||||
list_ops_job_events,
|
||||
preview_managed_node_onboarding_bootstrap,
|
||||
preview_managed_node_onboarding_acceptance,
|
||||
preview_managed_node_onboarding_recovery,
|
||||
request_managed_node_delivery_queue_action,
|
||||
summarize_ops_job_events,
|
||||
)
|
||||
from app.services.ops_job_service import (
|
||||
approve_ops_job,
|
||||
cancel_ops_job,
|
||||
create_ops_job,
|
||||
create_ops_job_batch,
|
||||
dispatch_ops_job,
|
||||
get_ops_job,
|
||||
list_ops_jobs,
|
||||
sync_managed_nodes_from_cluster,
|
||||
upsert_managed_node,
|
||||
)
|
||||
from app.services.ops_playbook_service import (
|
||||
cancel_ops_playbook_run,
|
||||
execute_ops_playbook,
|
||||
get_ops_playbook_run,
|
||||
get_ops_playbooks,
|
||||
get_recent_ops_playbook_runs,
|
||||
list_ops_playbook_run_events,
|
||||
preview_ops_playbook,
|
||||
rerun_ops_playbook_run,
|
||||
)
|
||||
from app.services.ops_policy_service import preview_ops_job_policy
|
||||
from app.services.ops_service import (
|
||||
execute_codex_action,
|
||||
execute_resolved_driver_action,
|
||||
get_ops_activity_stream,
|
||||
get_ops_codex_brief,
|
||||
get_ops_contract_detail,
|
||||
get_ops_contract_registry,
|
||||
get_ops_doctor_decision,
|
||||
get_ops_driver_feed,
|
||||
get_ops_go_live_summary,
|
||||
get_ops_go_live_signoff,
|
||||
get_ops_go_live_bundle,
|
||||
get_ops_go_live_review,
|
||||
get_ops_link_snapshot,
|
||||
get_ops_node_scene_log,
|
||||
get_ops_stack_diagnosis,
|
||||
execute_driver_action,
|
||||
preview_driver_action,
|
||||
resolve_codex_action,
|
||||
resolve_driver_action_request,
|
||||
resolve_ops_runbook_sequence,
|
||||
execute_ops_runbook_sequence,
|
||||
get_ops_blueprint,
|
||||
get_ops_capabilities,
|
||||
get_ops_inspection_overview,
|
||||
get_ops_overview,
|
||||
get_ops_runbook,
|
||||
)
|
||||
from app.services.ops_template_service import get_ops_action_templates
|
||||
|
||||
router = APIRouter(tags=["ops"])
|
||||
|
||||
|
||||
@router.get("/ops/overview", response_model=ApiResponse)
|
||||
def ops_overview() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_overview())
|
||||
|
||||
|
||||
@router.get("/ops/link-snapshot", response_model=ApiResponse)
|
||||
def ops_link_snapshot() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_link_snapshot())
|
||||
|
||||
|
||||
@router.get("/ops/stack-diagnosis", response_model=ApiResponse)
|
||||
def ops_stack_diagnosis(base_url: str = "") -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_stack_diagnosis(base_url=base_url))
|
||||
|
||||
|
||||
@router.get("/ops/go-live-summary", response_model=ApiResponse)
|
||||
def ops_go_live_summary(base_url: str = "") -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_go_live_summary(base_url=base_url))
|
||||
|
||||
|
||||
@router.get("/ops/doctor-decision", response_model=ApiResponse)
|
||||
def ops_doctor_decision(base_url: str = "", report_dir: str = "") -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_doctor_decision(base_url=base_url, report_dir=report_dir))
|
||||
|
||||
|
||||
@router.get("/ops/go-live-signoff", response_model=ApiResponse)
|
||||
def ops_go_live_signoff(base_url: str = "", report_dir: str = "") -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_go_live_signoff(base_url=base_url, report_dir=report_dir))
|
||||
|
||||
|
||||
@router.get("/ops/go-live-bundle", response_model=ApiResponse)
|
||||
def ops_go_live_bundle(base_url: str = "", report_dir: str = "") -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_go_live_bundle(base_url=base_url, report_dir=report_dir))
|
||||
|
||||
|
||||
@router.get("/ops/go-live-review", response_model=ApiResponse)
|
||||
def ops_go_live_review(base_url: str = "", report_dir: str = "") -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_go_live_review(base_url=base_url, report_dir=report_dir))
|
||||
|
||||
|
||||
@router.get("/ops/inspection-overview", response_model=ApiResponse)
|
||||
def ops_inspection_overview(
|
||||
limit: int = 80,
|
||||
status: str = "",
|
||||
problem_kind: str = "",
|
||||
query: str = "",
|
||||
only_problem: bool = False,
|
||||
only_participating: bool = False,
|
||||
) -> ApiResponse:
|
||||
return ApiResponse(
|
||||
data=get_ops_inspection_overview(
|
||||
fetch_limit=limit,
|
||||
status=status,
|
||||
problem_kind=problem_kind,
|
||||
query=query,
|
||||
only_problem=only_problem,
|
||||
only_participating=only_participating,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ops/activity-stream", response_model=ApiResponse)
|
||||
def ops_activity_stream(
|
||||
limit: int = 18,
|
||||
kind: str = "",
|
||||
status: str = "",
|
||||
execution_mode: str = "",
|
||||
query: str = "",
|
||||
) -> ApiResponse:
|
||||
return ApiResponse(
|
||||
data=get_ops_activity_stream(
|
||||
limit=limit,
|
||||
kind=kind,
|
||||
status=status,
|
||||
execution_mode=execution_mode,
|
||||
query=query,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ops/driver-feed", response_model=ApiResponse)
|
||||
def ops_driver_feed() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_driver_feed())
|
||||
|
||||
|
||||
@router.get("/ops/codex-brief", response_model=ApiResponse)
|
||||
def ops_codex_brief() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_codex_brief())
|
||||
|
||||
|
||||
@router.get("/ops/capabilities", response_model=ApiResponse)
|
||||
def ops_capabilities() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_capabilities())
|
||||
|
||||
|
||||
@router.get("/ops/contracts", response_model=ApiResponse)
|
||||
def ops_contracts() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_contract_registry())
|
||||
|
||||
|
||||
@router.get("/ops/contracts/{contract_key}", response_model=ApiResponse)
|
||||
def ops_contract_detail(contract_key: str) -> ApiResponse:
|
||||
data = get_ops_contract_detail(contract_key)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="contract 不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.get("/ops/action-templates", response_model=ApiResponse)
|
||||
def ops_action_templates() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_action_templates())
|
||||
|
||||
|
||||
@router.get("/ops/playbooks", response_model=ApiResponse)
|
||||
def ops_playbooks() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_playbooks())
|
||||
|
||||
|
||||
@router.get("/ops/playbook-runs", response_model=ApiResponse)
|
||||
def ops_playbook_runs(
|
||||
limit: int = 12,
|
||||
status: str = "",
|
||||
group_key: str = "",
|
||||
query: str = "",
|
||||
) -> ApiResponse:
|
||||
return ApiResponse(
|
||||
data=get_recent_ops_playbook_runs(
|
||||
limit=limit,
|
||||
status=status,
|
||||
group_key=group_key,
|
||||
query=query,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ops/playbook-runs/{run_code}", response_model=ApiResponse)
|
||||
def ops_playbook_run_detail(run_code: str) -> ApiResponse:
|
||||
data = get_ops_playbook_run(run_code)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="playbook run 不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.get("/ops/playbook-runs/{run_code}/events", response_model=ApiResponse)
|
||||
def ops_playbook_run_events(
|
||||
run_code: str,
|
||||
limit: int = 80,
|
||||
step_key: str = "",
|
||||
node_code: str = "",
|
||||
) -> ApiResponse:
|
||||
data = list_ops_playbook_run_events(
|
||||
run_code,
|
||||
limit=limit,
|
||||
step_key=step_key,
|
||||
node_code=node_code,
|
||||
)
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.post("/ops/playbook-runs/{run_code}/rerun", response_model=ApiResponse)
|
||||
def ops_playbook_run_rerun(run_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
payload = payload or {}
|
||||
ok, message, data = rerun_ops_playbook_run(
|
||||
run_code,
|
||||
requested_by=str(payload.get("requested_by") or "api").strip() or "api",
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/playbook-runs/{run_code}/cancel", response_model=ApiResponse)
|
||||
def ops_playbook_run_cancel(run_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
payload = payload or {}
|
||||
ok, message, data = cancel_ops_playbook_run(
|
||||
run_code,
|
||||
cancelled_by=str(payload.get("cancelled_by") or "api").strip() or "api",
|
||||
reason=str(payload.get("reason") or "").strip(),
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/playbooks/preview", response_model=ApiResponse)
|
||||
def ops_playbook_preview(payload: dict) -> ApiResponse:
|
||||
ok, message, data = preview_ops_playbook(payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/playbooks/execute", response_model=ApiResponse)
|
||||
def ops_playbook_execute(payload: dict) -> ApiResponse:
|
||||
ok, message, data = execute_ops_playbook(payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.get("/ops/blueprint", response_model=ApiResponse)
|
||||
def ops_blueprint() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_blueprint())
|
||||
|
||||
|
||||
@router.get("/ops/runbook", response_model=ApiResponse)
|
||||
def ops_runbook() -> ApiResponse:
|
||||
return ApiResponse(data=get_ops_runbook())
|
||||
|
||||
|
||||
@router.post("/ops/runbook/sequences/{sequence_key}/resolve", response_model=ApiResponse)
|
||||
def ops_runbook_sequence_resolve(sequence_key: str, payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = resolve_ops_runbook_sequence(sequence_key, payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/runbook/sequences/{sequence_key}/execute", response_model=ApiResponse)
|
||||
def ops_runbook_sequence_execute(sequence_key: str, payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = execute_ops_runbook_sequence(sequence_key, payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.get("/ops/nodes", response_model=ApiResponse)
|
||||
def ops_nodes() -> ApiResponse:
|
||||
return ApiResponse(data=list_managed_nodes_with_agent_state())
|
||||
|
||||
|
||||
@router.get("/ops/nodes/{node_code}/handover", response_model=ApiResponse)
|
||||
def ops_node_handover(
|
||||
node_code: str,
|
||||
control_plane_base_url: str = "",
|
||||
root_dir: str = "",
|
||||
) -> ApiResponse:
|
||||
data = get_managed_node_handover(
|
||||
node_code,
|
||||
control_plane_base_url=control_plane_base_url,
|
||||
root_dir=root_dir,
|
||||
)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="节点不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.get("/ops/nodes/{node_code}/onboarding", response_model=ApiResponse)
|
||||
def ops_node_onboarding(
|
||||
node_code: str,
|
||||
control_plane_base_url: str = "",
|
||||
root_dir: str = "",
|
||||
) -> ApiResponse:
|
||||
data = get_managed_node_onboarding(
|
||||
node_code,
|
||||
control_plane_base_url=control_plane_base_url,
|
||||
root_dir=root_dir,
|
||||
)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="节点不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/onboarding/acceptance/preview", response_model=ApiResponse)
|
||||
def ops_node_onboarding_acceptance_preview(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
normalized_payload = payload or {}
|
||||
ok, message, data = preview_managed_node_onboarding_acceptance(
|
||||
node_code=node_code,
|
||||
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
|
||||
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
|
||||
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/onboarding/bootstrap/preview", response_model=ApiResponse)
|
||||
def ops_node_onboarding_bootstrap_preview(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
normalized_payload = payload or {}
|
||||
ok, message, data = preview_managed_node_onboarding_bootstrap(
|
||||
node_code=node_code,
|
||||
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
|
||||
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
|
||||
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/onboarding/bootstrap/execute", response_model=ApiResponse)
|
||||
def ops_node_onboarding_bootstrap_execute(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
normalized_payload = payload or {}
|
||||
ok, message, data = execute_managed_node_onboarding_bootstrap(
|
||||
node_code=node_code,
|
||||
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
|
||||
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
|
||||
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/onboarding/acceptance/execute", response_model=ApiResponse)
|
||||
def ops_node_onboarding_acceptance_execute(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
normalized_payload = payload or {}
|
||||
ok, message, data = execute_managed_node_onboarding_acceptance(
|
||||
node_code=node_code,
|
||||
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
|
||||
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
|
||||
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/onboarding/recovery/preview", response_model=ApiResponse)
|
||||
def ops_node_onboarding_recovery_preview(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
normalized_payload = payload or {}
|
||||
ok, message, data = preview_managed_node_onboarding_recovery(
|
||||
node_code=node_code,
|
||||
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
|
||||
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
|
||||
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/onboarding/recovery/execute", response_model=ApiResponse)
|
||||
def ops_node_onboarding_recovery_execute(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
normalized_payload = payload or {}
|
||||
ok, message, data = execute_managed_node_onboarding_recovery(
|
||||
node_code=node_code,
|
||||
requested_by=str(normalized_payload.get("requested_by") or "api").strip() or "api",
|
||||
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
|
||||
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.get("/ops/nodes/{node_code}/scene-log", response_model=ApiResponse)
|
||||
def ops_node_scene_log(
|
||||
node_code: str,
|
||||
limit: int = 80,
|
||||
mode: str = "",
|
||||
) -> ApiResponse:
|
||||
data = get_ops_node_scene_log(node_code, limit=limit, mode=mode)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="节点不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/handover/bootstrap-plan", response_model=ApiResponse)
|
||||
def ops_node_handover_bootstrap_plan(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
normalized_payload = payload or {}
|
||||
ok, message, data = build_managed_node_handover_bootstrap_plan(
|
||||
node_code=node_code,
|
||||
issued_by=str(normalized_payload.get("issued_by") or "api").strip() or "api",
|
||||
expires_in_hours=int(normalized_payload.get("expires_in_hours") or 72),
|
||||
control_plane_base_url=str(normalized_payload.get("control_plane_base_url") or "").strip(),
|
||||
root_dir=str(normalized_payload.get("root_dir") or "").strip(),
|
||||
metadata=normalized_payload.get("metadata") or {},
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.get("/ops/nodes/{node_code}/delivery-queue", response_model=ApiResponse)
|
||||
def ops_node_delivery_queue(node_code: str) -> ApiResponse:
|
||||
data = get_managed_node_delivery_queue(node_code)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="节点不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.get("/ops/nodes/{node_code}/delivery-queue/records", response_model=ApiResponse)
|
||||
def ops_node_delivery_queue_records(
|
||||
node_code: str,
|
||||
state: str = "",
|
||||
limit: int = 50,
|
||||
) -> ApiResponse:
|
||||
data = list_managed_node_delivery_queue_records(
|
||||
node_code,
|
||||
state=state,
|
||||
limit=limit,
|
||||
)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="节点不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/delivery-queue/flush", response_model=ApiResponse)
|
||||
def ops_node_delivery_queue_flush(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = request_managed_node_delivery_queue_action(
|
||||
node_code,
|
||||
"delivery.queue.flush",
|
||||
payload=payload or {},
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/delivery-queue/replay", response_model=ApiResponse)
|
||||
def ops_node_delivery_queue_replay(node_code: str, payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = request_managed_node_delivery_queue_action(
|
||||
node_code,
|
||||
"delivery.queue.replay",
|
||||
payload=payload or {},
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay", response_model=ApiResponse)
|
||||
def ops_node_delivery_queue_record_replay(
|
||||
node_code: str,
|
||||
record_id: str,
|
||||
payload: dict | None = None,
|
||||
) -> ApiResponse:
|
||||
ok, message, data = request_managed_node_delivery_queue_action(
|
||||
node_code,
|
||||
"delivery.queue.replay",
|
||||
payload=payload or {},
|
||||
record_id=record_id,
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard", response_model=ApiResponse)
|
||||
def ops_node_delivery_queue_record_discard(
|
||||
node_code: str,
|
||||
record_id: str,
|
||||
payload: dict | None = None,
|
||||
) -> ApiResponse:
|
||||
ok, message, data = request_managed_node_delivery_queue_action(
|
||||
node_code,
|
||||
"delivery.queue.discard",
|
||||
payload=payload or {},
|
||||
record_id=record_id,
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes", response_model=ApiResponse)
|
||||
def ops_nodes_upsert(payload: dict) -> ApiResponse:
|
||||
ok, message, data = upsert_managed_node(payload)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/nodes/sync-from-cluster", response_model=ApiResponse)
|
||||
def ops_nodes_sync_from_cluster(dry_run: bool = False) -> ApiResponse:
|
||||
return ApiResponse(data=sync_managed_nodes_from_cluster(dry_run=dry_run))
|
||||
|
||||
|
||||
@router.get("/ops/jobs", response_model=ApiResponse)
|
||||
def ops_jobs(limit: int = 20, compact: bool = True, rollout_id: int = 0) -> ApiResponse:
|
||||
normalized_rollout_id = int(rollout_id or 0)
|
||||
jobs = list_ops_jobs(
|
||||
limit=limit,
|
||||
compact=compact,
|
||||
rollout_id=normalized_rollout_id if normalized_rollout_id > 0 else None,
|
||||
)
|
||||
page_status_counts: dict[str, int] = {}
|
||||
for item in jobs:
|
||||
item_status = str(item.get("status") or "").strip() or "unknown"
|
||||
page_status_counts[item_status] = int(page_status_counts.get(item_status, 0) or 0) + 1
|
||||
return ApiResponse(
|
||||
data={
|
||||
"jobs": jobs,
|
||||
"filters": {
|
||||
"limit": int(limit or 20),
|
||||
"compact": bool(compact),
|
||||
"rollout_id": normalized_rollout_id,
|
||||
},
|
||||
"page_summary": {
|
||||
"jobs_total": len(jobs),
|
||||
"status_counts": page_status_counts,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ops/jobs/{job_id}", response_model=ApiResponse)
|
||||
def ops_job_detail(job_id: int) -> ApiResponse:
|
||||
data = get_ops_job(job_id)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="任务不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.get("/ops/jobs/{job_id}/events", response_model=ApiResponse)
|
||||
def ops_job_events(job_id: int, limit: int = 50) -> ApiResponse:
|
||||
events = list_ops_job_events(job_id, limit=limit)
|
||||
return ApiResponse(
|
||||
data={
|
||||
"events": events,
|
||||
"summary": summarize_ops_job_events(events, job_id=job_id, limit=limit),
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.post("/ops/driver-actions/execute", response_model=ApiResponse)
|
||||
def ops_driver_action_execute(payload: dict) -> ApiResponse:
|
||||
ok, message, data = execute_driver_action(payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/driver-actions/preview", response_model=ApiResponse)
|
||||
def ops_driver_action_preview(payload: dict) -> ApiResponse:
|
||||
ok, message, data = preview_driver_action(payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/driver-actions/resolve", response_model=ApiResponse)
|
||||
def ops_driver_action_resolve(payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = resolve_driver_action_request(payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/driver-actions/execute-resolved", response_model=ApiResponse)
|
||||
def ops_driver_action_execute_resolved(payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = execute_resolved_driver_action(payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/codex-actions/resolve", response_model=ApiResponse)
|
||||
def ops_codex_action_resolve(payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = resolve_codex_action(payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/codex-actions/execute", response_model=ApiResponse)
|
||||
def ops_codex_action_execute(payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = execute_codex_action(payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/jobs", response_model=ApiResponse)
|
||||
def ops_jobs_create(payload: dict) -> ApiResponse:
|
||||
ok, message, data = create_ops_job(payload)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/jobs/batch", response_model=ApiResponse)
|
||||
def ops_jobs_create_batch(payload: dict) -> ApiResponse:
|
||||
ok, message, data = create_ops_job_batch(payload)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/policy/preview", response_model=ApiResponse)
|
||||
def ops_policy_preview(payload: dict) -> ApiResponse:
|
||||
return ApiResponse(data=preview_ops_job_policy(payload))
|
||||
|
||||
|
||||
@router.post("/ops/jobs/{job_id}/approve", response_model=ApiResponse)
|
||||
def ops_job_approve(job_id: int, payload: dict | None = None) -> ApiResponse:
|
||||
payload = payload or {}
|
||||
ok, message, data = approve_ops_job(job_id, approved_by=str(payload.get("approved_by") or "api").strip() or "api")
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/jobs/{job_id}/cancel", response_model=ApiResponse)
|
||||
def ops_job_cancel(job_id: int, payload: dict | None = None) -> ApiResponse:
|
||||
payload = payload or {}
|
||||
ok, message, data = cancel_ops_job(
|
||||
job_id,
|
||||
cancelled_by=str(payload.get("cancelled_by") or "api").strip() or "api",
|
||||
reason=str(payload.get("reason") or "").strip(),
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/jobs/{job_id}/dispatch", response_model=ApiResponse)
|
||||
def ops_job_dispatch(job_id: int) -> ApiResponse:
|
||||
ok, message, data = dispatch_ops_job(job_id)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
90
domain-api/app/api/routes/ops_agent.py
Normal file
90
domain-api/app/api/routes/ops_agent.py
Normal file
@@ -0,0 +1,90 @@
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Header
|
||||
|
||||
from app.schemas.common import ApiResponse
|
||||
from app.services.ops_agent_service import (
|
||||
agent_append_job_event,
|
||||
agent_complete_job,
|
||||
agent_heartbeat,
|
||||
agent_mark_job_started,
|
||||
agent_pull_jobs,
|
||||
agent_register,
|
||||
build_node_agent_bootstrap_plan,
|
||||
issue_node_agent_token,
|
||||
)
|
||||
|
||||
router = APIRouter(tags=["ops-agent"])
|
||||
|
||||
|
||||
def _resolve_agent_token(x_domaincheck_agent_token: Optional[str]) -> str:
|
||||
return str(x_domaincheck_agent_token or "").strip()
|
||||
|
||||
|
||||
def _build_agent_response(ok: bool, message: str, data: object) -> ApiResponse:
|
||||
detail_code = None
|
||||
if isinstance(data, dict):
|
||||
detail_code = str(data.get("detail_code") or "").strip() or None
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data, detail_code=detail_code)
|
||||
|
||||
|
||||
@router.post("/ops/agent/tokens", response_model=ApiResponse)
|
||||
def ops_agent_issue_token(payload: dict) -> ApiResponse:
|
||||
ok, message, data = issue_node_agent_token(
|
||||
node_code=str(payload.get("node_code") or "").strip(),
|
||||
issued_by=str(payload.get("issued_by") or "api").strip() or "api",
|
||||
expires_in_hours=int(payload.get("expires_in_hours") or 72),
|
||||
metadata=payload.get("metadata") or {},
|
||||
)
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/bootstrap-plan", response_model=ApiResponse)
|
||||
def ops_agent_bootstrap_plan(payload: dict) -> ApiResponse:
|
||||
ok, message, data = build_node_agent_bootstrap_plan(
|
||||
node_code=str(payload.get("node_code") or "").strip(),
|
||||
node_region=str(payload.get("node_region") or "mainland").strip() or "mainland",
|
||||
node_role=str(payload.get("node_role") or "worker").strip() or "worker",
|
||||
issued_by=str(payload.get("issued_by") or "api").strip() or "api",
|
||||
expires_in_hours=int(payload.get("expires_in_hours") or 72),
|
||||
control_plane_base_url=str(payload.get("control_plane_base_url") or "").strip(),
|
||||
root_dir=str(payload.get("root_dir") or "/opt/domaincheck").strip() or "/opt/domaincheck",
|
||||
metadata=payload.get("metadata") or {},
|
||||
)
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/register", response_model=ApiResponse)
|
||||
def ops_agent_register(payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = agent_register(payload, token=_resolve_agent_token(x_domaincheck_agent_token))
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/heartbeat", response_model=ApiResponse)
|
||||
def ops_agent_heartbeat(payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = agent_heartbeat(payload, token=_resolve_agent_token(x_domaincheck_agent_token))
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/pull", response_model=ApiResponse)
|
||||
def ops_agent_pull(payload: dict, limit: int = 1, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = agent_pull_jobs(payload, token=_resolve_agent_token(x_domaincheck_agent_token), limit=limit)
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/jobs/{job_id}/start", response_model=ApiResponse)
|
||||
def ops_agent_job_start(job_id: int, payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = agent_mark_job_started(job_id, payload, token=_resolve_agent_token(x_domaincheck_agent_token))
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/jobs/{job_id}/complete", response_model=ApiResponse)
|
||||
def ops_agent_job_complete(job_id: int, payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = agent_complete_job(job_id, payload, token=_resolve_agent_token(x_domaincheck_agent_token))
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/jobs/{job_id}/events", response_model=ApiResponse)
|
||||
def ops_agent_job_event(job_id: int, payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = agent_append_job_event(job_id, payload, token=_resolve_agent_token(x_domaincheck_agent_token))
|
||||
return _build_agent_response(ok, message, data)
|
||||
175
domain-api/app/api/routes/ops_release.py
Normal file
175
domain-api/app/api/routes/ops_release.py
Normal file
@@ -0,0 +1,175 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from fastapi import APIRouter, Request
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from app.schemas.common import ApiResponse
|
||||
from app.services.ops_release_service import (
|
||||
advance_release_rollout,
|
||||
activate_release,
|
||||
create_release,
|
||||
create_release_and_smart_rollout_from_latest_package,
|
||||
create_release_from_latest_package,
|
||||
create_release_rollout,
|
||||
create_smart_release_rollout,
|
||||
get_latest_release_package_metadata,
|
||||
get_release_launchpad,
|
||||
get_latest_release,
|
||||
get_release_package_archive_path,
|
||||
get_release_package_sha256_path,
|
||||
get_release,
|
||||
get_release_rollout,
|
||||
list_release_rollout_jobs,
|
||||
list_release_rollouts,
|
||||
list_releases,
|
||||
preview_release_from_latest_package,
|
||||
preview_smart_release_rollout,
|
||||
preview_smart_release_rollout_from_latest_package,
|
||||
)
|
||||
|
||||
router = APIRouter(tags=["ops-release"])
|
||||
|
||||
|
||||
@router.get("/ops/releases", response_model=ApiResponse)
|
||||
def ops_releases(limit: int = 20, channel: str | None = None) -> ApiResponse:
|
||||
return ApiResponse(data={"releases": list_releases(limit=limit, channel=channel)})
|
||||
|
||||
|
||||
@router.get("/ops/releases/latest", response_model=ApiResponse)
|
||||
def ops_releases_latest(channel: str = "stable") -> ApiResponse:
|
||||
return ApiResponse(data=get_latest_release(channel=channel))
|
||||
|
||||
|
||||
@router.get("/ops/releases/package-metadata/latest", response_model=ApiResponse)
|
||||
def ops_releases_package_metadata_latest() -> ApiResponse:
|
||||
return ApiResponse(data=get_latest_release_package_metadata())
|
||||
|
||||
|
||||
@router.get("/ops/releases/launchpad", response_model=ApiResponse)
|
||||
def ops_releases_launchpad(request: Request, channel: str = "stable") -> ApiResponse:
|
||||
derived_base_url = str(request.base_url).rstrip("/")
|
||||
return ApiResponse(data=get_release_launchpad(control_plane_base_url=derived_base_url, channel=channel))
|
||||
|
||||
|
||||
@router.get("/ops/releases/packages/{package_name}/download")
|
||||
def ops_release_package_download(package_name: str):
|
||||
try:
|
||||
archive_path = get_release_package_archive_path(package_name)
|
||||
except ValueError:
|
||||
return ApiResponse(code=1, message="发布包名称不合法", data={})
|
||||
if not archive_path:
|
||||
return ApiResponse(code=1, message="发布包不存在", data={})
|
||||
media_type = "application/octet-stream"
|
||||
archive_name = archive_path.name.lower()
|
||||
if archive_name.endswith(".tar.gz"):
|
||||
media_type = "application/gzip"
|
||||
elif archive_name.endswith(".zip"):
|
||||
media_type = "application/zip"
|
||||
return FileResponse(path=archive_path, filename=archive_path.name, media_type=media_type)
|
||||
|
||||
|
||||
@router.get("/ops/releases/packages/{package_name}/sha256")
|
||||
def ops_release_package_sha256(package_name: str):
|
||||
try:
|
||||
sha256_path = get_release_package_sha256_path(package_name)
|
||||
except ValueError:
|
||||
return ApiResponse(code=1, message="发布包名称不合法", data={})
|
||||
if not sha256_path:
|
||||
return ApiResponse(code=1, message="发布包校验文件不存在", data={})
|
||||
return FileResponse(path=sha256_path, filename=sha256_path.name, media_type="text/plain; charset=utf-8")
|
||||
|
||||
|
||||
@router.get("/ops/releases/{release_id}", response_model=ApiResponse)
|
||||
def ops_releases_detail(release_id: int) -> ApiResponse:
|
||||
data = get_release(release_id)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="Release 不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases", response_model=ApiResponse)
|
||||
def ops_releases_create(payload: dict) -> ApiResponse:
|
||||
ok, message, data = create_release(payload)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases/from-package/latest", response_model=ApiResponse)
|
||||
def ops_releases_create_from_latest_package(request: Request, payload: dict | None = None) -> ApiResponse:
|
||||
derived_base_url = str(request.base_url).rstrip("/")
|
||||
ok, message, data = create_release_from_latest_package(payload or {}, control_plane_base_url=derived_base_url)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases/from-package/latest/preview", response_model=ApiResponse)
|
||||
def ops_releases_preview_from_latest_package(request: Request, payload: dict | None = None) -> ApiResponse:
|
||||
derived_base_url = str(request.base_url).rstrip("/")
|
||||
ok, message, data = preview_release_from_latest_package(payload or {}, control_plane_base_url=derived_base_url)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases/from-package/latest/smart-rollout", response_model=ApiResponse)
|
||||
def ops_releases_create_from_latest_package_and_smart_rollout(request: Request, payload: dict | None = None) -> ApiResponse:
|
||||
derived_base_url = str(request.base_url).rstrip("/")
|
||||
ok, message, data = create_release_and_smart_rollout_from_latest_package(
|
||||
payload or {},
|
||||
control_plane_base_url=derived_base_url,
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases/from-package/latest/smart-rollout-preview", response_model=ApiResponse)
|
||||
def ops_releases_preview_from_latest_package_and_smart_rollout(request: Request, payload: dict | None = None) -> ApiResponse:
|
||||
derived_base_url = str(request.base_url).rstrip("/")
|
||||
ok, message, data = preview_smart_release_rollout_from_latest_package(
|
||||
payload or {},
|
||||
control_plane_base_url=derived_base_url,
|
||||
)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases/{release_id}/activate", response_model=ApiResponse)
|
||||
def ops_releases_activate(release_id: int) -> ApiResponse:
|
||||
ok, message, data = activate_release(release_id)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases/{release_id}/smart-rollout-preview", response_model=ApiResponse)
|
||||
def ops_releases_smart_rollout_preview(release_id: int, payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = preview_smart_release_rollout(release_id, payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases/{release_id}/smart-rollout", response_model=ApiResponse)
|
||||
def ops_releases_smart_rollout(release_id: int, payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = create_smart_release_rollout(release_id, payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.get("/ops/releases/{release_id}/rollouts", response_model=ApiResponse)
|
||||
def ops_release_rollouts(release_id: int, limit: int = 20) -> ApiResponse:
|
||||
return ApiResponse(data={"rollouts": list_release_rollouts(release_id=release_id, limit=limit)})
|
||||
|
||||
|
||||
@router.get("/ops/rollouts/{rollout_id}", response_model=ApiResponse)
|
||||
def ops_rollout_detail(rollout_id: int) -> ApiResponse:
|
||||
data = get_release_rollout(rollout_id)
|
||||
if not data:
|
||||
return ApiResponse(code=1, message="Rollout 不存在", data={})
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.get("/ops/rollouts/{rollout_id}/jobs", response_model=ApiResponse)
|
||||
def ops_rollout_jobs(rollout_id: int, limit: int = 200) -> ApiResponse:
|
||||
return ApiResponse(data={"jobs": list_release_rollout_jobs(rollout_id, limit=limit)})
|
||||
|
||||
|
||||
@router.post("/ops/rollouts/{rollout_id}/advance", response_model=ApiResponse)
|
||||
def ops_rollout_advance(rollout_id: int, payload: dict | None = None) -> ApiResponse:
|
||||
ok, message, data = advance_release_rollout(rollout_id, payload or {})
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
|
||||
@router.post("/ops/releases/{release_id}/rollouts", response_model=ApiResponse)
|
||||
def ops_release_rollouts_create(release_id: int, payload: dict) -> ApiResponse:
|
||||
ok, message, data = create_release_rollout(release_id, payload)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
@@ -3,6 +3,7 @@ from typing import Optional
|
||||
from fastapi import APIRouter, Header
|
||||
|
||||
from app.schemas.common import ApiResponse
|
||||
from app.services.build_info_service import get_runtime_build_info
|
||||
from app.services.cluster_runtime_service import get_cluster_snapshot
|
||||
from app.services.debug_event_service import get_debug_diagnosis, get_debug_event_overview, get_debug_handoff_report, ingest_debug_event, list_debug_events
|
||||
from app.services.runtime_control_service import runtime_action
|
||||
@@ -37,6 +38,11 @@ def runtime_cluster() -> ApiResponse:
|
||||
return ApiResponse(data=get_cluster_snapshot())
|
||||
|
||||
|
||||
@router.get("/runtime/build-info", response_model=ApiResponse)
|
||||
def runtime_build_info() -> ApiResponse:
|
||||
return ApiResponse(data=get_runtime_build_info())
|
||||
|
||||
|
||||
@router.get("/runtime/sync-summary", response_model=ApiResponse)
|
||||
def runtime_sync_summary() -> ApiResponse:
|
||||
return ApiResponse(data=get_sync_summary())
|
||||
|
||||
@@ -42,6 +42,13 @@ class Settings(BaseSettings):
|
||||
sync_shared_token: str = ""
|
||||
sync_batch_size: int = 200
|
||||
sync_poll_interval_seconds: int = 30
|
||||
build_manifest_path: str = ""
|
||||
build_commit_sha: str = ""
|
||||
build_commit_ref: str = ""
|
||||
build_generated_at: str = ""
|
||||
build_package_name: str = ""
|
||||
build_checksum: str = ""
|
||||
build_source_label: str = ""
|
||||
|
||||
@property
|
||||
def cors_origins_list(self) -> list[str]:
|
||||
|
||||
@@ -3,9 +3,13 @@ import threading
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
from app.api.routes import auth, dashboard, settings as settings_routes, imports, detect, domains, exports, logs, runtime, juming, sensitive_words
|
||||
from app.api.routes import auth, dashboard, settings as settings_routes, imports, detect, domains, exports, logs, runtime, juming, ops, ops_agent, ops_release, sensitive_words
|
||||
from app.core.config import settings as app_settings
|
||||
from app.services.build_info_service import get_runtime_build_info, remember_registered_route_paths
|
||||
from app.services.cluster_runtime_service import ensure_runtime_schema, register_local_control_heartbeat
|
||||
from app.services.ops_agent_service import ensure_ops_agent_schema
|
||||
from app.services.ops_job_service import ensure_ops_schema
|
||||
from app.services.ops_release_service import ensure_ops_release_schema
|
||||
_heartbeat_stop_event = threading.Event()
|
||||
|
||||
|
||||
@@ -36,6 +40,10 @@ app.add_middleware(
|
||||
@app.on_event("startup")
|
||||
def on_startup() -> None:
|
||||
ensure_runtime_schema()
|
||||
ensure_ops_schema()
|
||||
ensure_ops_agent_schema()
|
||||
ensure_ops_release_schema()
|
||||
remember_registered_route_paths(route.path for route in app.routes)
|
||||
register_local_control_heartbeat()
|
||||
_heartbeat_stop_event.clear()
|
||||
threading.Thread(target=_control_heartbeat_loop, name="control-heartbeat", daemon=True).start()
|
||||
@@ -56,6 +64,7 @@ def health() -> dict:
|
||||
"worker_mode": app_settings.worker_mode,
|
||||
"api_host": app_settings.api_host,
|
||||
"api_port": app_settings.api_port,
|
||||
"build": get_runtime_build_info(route_paths=(route.path for route in app.routes)),
|
||||
}
|
||||
|
||||
|
||||
@@ -69,4 +78,8 @@ app.include_router(exports.router, prefix=app_settings.api_prefix)
|
||||
app.include_router(logs.router, prefix=app_settings.api_prefix)
|
||||
app.include_router(runtime.router, prefix=app_settings.api_prefix)
|
||||
app.include_router(juming.router, prefix=app_settings.api_prefix)
|
||||
app.include_router(ops.router, prefix=app_settings.api_prefix)
|
||||
app.include_router(ops_agent.router, prefix=app_settings.api_prefix)
|
||||
app.include_router(ops_release.router, prefix=app_settings.api_prefix)
|
||||
app.include_router(sensitive_words.router, prefix=app_settings.api_prefix)
|
||||
remember_registered_route_paths(route.path for route in app.routes)
|
||||
|
||||
991
domain-api/app/node_agent.py
Normal file
991
domain-api/app/node_agent.py
Normal file
@@ -0,0 +1,991 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime
|
||||
from uuid import uuid4
|
||||
|
||||
from app.services.ops_action_executor_core import (
|
||||
build_service_name_map,
|
||||
execute_structured_action,
|
||||
supports_structured_action,
|
||||
)
|
||||
from app.services.ops_release_executor_core import (
|
||||
execute_release_action,
|
||||
normalize_release_health_check_services as _release_normalize_health_check_services,
|
||||
normalize_text_list as _release_normalize_text_list,
|
||||
)
|
||||
|
||||
|
||||
CONTROL_PLANE_BASE_URL = str(os.getenv("OPS_CONTROL_PLANE_BASE_URL", "")).strip().rstrip("/")
|
||||
AGENT_TOKEN = str(os.getenv("OPS_AGENT_TOKEN", "")).strip()
|
||||
NODE_CODE = str(os.getenv("NODE_CODE", "")).strip()
|
||||
NODE_REGION = str(os.getenv("NODE_REGION", "mainland")).strip() or "mainland"
|
||||
NODE_ROLE = str(os.getenv("NODE_ROLE", "worker")).strip() or "worker"
|
||||
AGENT_POLL_INTERVAL_SECONDS = max(2, int(os.getenv("OPS_AGENT_POLL_INTERVAL_SECONDS", "5") or 5))
|
||||
WORKER_SERVICE_NAME = str(os.getenv("WORKER_SERVICE_NAME", os.getenv("WORKER_SERVICE", "domaincheck-worker"))).strip() or "domaincheck-worker"
|
||||
API_SERVICE_NAME = str(os.getenv("API_SERVICE_NAME", "domaincheck-api")).strip() or "domaincheck-api"
|
||||
SYNC_AGENT_SERVICE_NAME = str(os.getenv("SYNC_AGENT_SERVICE_NAME", "domaincheck-sync-agent")).strip() or "domaincheck-sync-agent"
|
||||
NODE_AGENT_SERVICE_NAME = str(os.getenv("NODE_AGENT_SERVICE_NAME", "domaincheck-node-agent")).strip() or "domaincheck-node-agent"
|
||||
AGENT_VERSION = "0.1.0"
|
||||
_APP_DIR = os.path.dirname(os.path.abspath(__file__))
|
||||
_PROJECT_DIR = os.path.dirname(_APP_DIR)
|
||||
_DEFAULT_QUEUE_DIR = os.path.join(_PROJECT_DIR, "runtime", "node-agent-queue", NODE_CODE or "unbound")
|
||||
AGENT_QUEUE_DIR = str(os.getenv("OPS_AGENT_QUEUE_DIR", _DEFAULT_QUEUE_DIR)).strip() or _DEFAULT_QUEUE_DIR
|
||||
AGENT_QUEUE_FLUSH_LIMIT = max(1, int(os.getenv("OPS_AGENT_QUEUE_FLUSH_LIMIT", "20") or 20))
|
||||
_PERMANENT_DELIVERY_DETAIL_CODES = {
|
||||
"agent_node_code_required",
|
||||
"ops_job_not_owned_by_agent",
|
||||
"ops_job_invalid_status",
|
||||
}
|
||||
_LAST_QUEUE_FLUSH_SUMMARY = {
|
||||
"scanned": 0,
|
||||
"delivered": 0,
|
||||
"deferred": 0,
|
||||
"dead_letter": 0,
|
||||
"last_flush_at": "",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_text_list(raw_value: object) -> list[str]:
|
||||
"""Keep node-agent payload normalization aligned with release executor helpers."""
|
||||
return _release_normalize_text_list(raw_value)
|
||||
|
||||
|
||||
def _normalize_release_health_check_services(
|
||||
payload: dict | None,
|
||||
restart_services: list[str] | None,
|
||||
) -> list[str]:
|
||||
"""Backward-compatible wrapper used by node-agent tests and payload shaping."""
|
||||
return _release_normalize_health_check_services(
|
||||
dict(payload or {}),
|
||||
list(restart_services or []),
|
||||
default_api_service_name=API_SERVICE_NAME,
|
||||
)
|
||||
|
||||
|
||||
def _log(message: str) -> None:
|
||||
print(f"{datetime.now().isoformat(sep=' ', timespec='seconds')} [node-agent] {message}", flush=True)
|
||||
|
||||
|
||||
def _json_env(name: str, fallback: object) -> object:
|
||||
raw = str(os.getenv(name, "")).strip()
|
||||
if not raw:
|
||||
return fallback
|
||||
try:
|
||||
return json.loads(raw)
|
||||
except Exception:
|
||||
return fallback
|
||||
|
||||
|
||||
AGENT_CAPABILITIES = _json_env(
|
||||
"OPS_AGENT_CAPABILITIES",
|
||||
[
|
||||
"service.start",
|
||||
"service.stop",
|
||||
"service.restart",
|
||||
"service.status",
|
||||
"runtime.start_worker",
|
||||
"runtime.stop_worker",
|
||||
"runtime.restart_api",
|
||||
"runtime.start_sync_agent",
|
||||
"runtime.stop_sync_agent",
|
||||
"health.snapshot",
|
||||
"logs.collect",
|
||||
"diagnostics.collect",
|
||||
"delivery.queue.flush",
|
||||
"delivery.queue.replay",
|
||||
"delivery.queue.discard",
|
||||
"deploy.release",
|
||||
],
|
||||
)
|
||||
AGENT_LABELS = _json_env("OPS_AGENT_LABELS", {})
|
||||
|
||||
|
||||
class AgentResponseError(RuntimeError):
|
||||
def __init__(self, message: str, *, detail_code: str = "", payload: dict | None = None):
|
||||
super().__init__(message)
|
||||
self.detail_code = str(detail_code or "").strip()
|
||||
self.payload = dict(payload or {})
|
||||
|
||||
|
||||
def _now_text() -> str:
|
||||
return datetime.now().isoformat(sep=" ", timespec="seconds")
|
||||
|
||||
|
||||
def _queue_pending_dir() -> str:
|
||||
return os.path.join(AGENT_QUEUE_DIR, "pending")
|
||||
|
||||
|
||||
def _queue_dead_letter_dir() -> str:
|
||||
return os.path.join(AGENT_QUEUE_DIR, "dead-letter")
|
||||
|
||||
|
||||
def _queue_discarded_dir() -> str:
|
||||
return os.path.join(AGENT_QUEUE_DIR, "discarded")
|
||||
|
||||
|
||||
def _ensure_queue_dirs() -> None:
|
||||
for path in (AGENT_QUEUE_DIR, _queue_pending_dir(), _queue_dead_letter_dir(), _queue_discarded_dir()):
|
||||
os.makedirs(path, exist_ok=True)
|
||||
|
||||
|
||||
def _write_json_file(path: str, payload: dict) -> None:
|
||||
temp_path = f"{path}.tmp"
|
||||
with open(temp_path, "w", encoding="utf-8") as handle:
|
||||
json.dump(payload, handle, ensure_ascii=False, indent=2)
|
||||
os.replace(temp_path, path)
|
||||
|
||||
|
||||
def _read_json_file(path: str) -> dict:
|
||||
with open(path, "r", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
|
||||
|
||||
def _new_request_id(prefix: str, job_id: int) -> str:
|
||||
normalized_prefix = str(prefix or "request").strip() or "request"
|
||||
return f"{normalized_prefix}-{int(job_id or 0)}-{int(time.time() * 1000)}-{uuid4().hex[:10]}"
|
||||
|
||||
|
||||
def _queue_file_names(directory: str) -> list[str]:
|
||||
_ensure_queue_dirs()
|
||||
try:
|
||||
return sorted(file_name for file_name in os.listdir(directory) if file_name.endswith(".json"))
|
||||
except FileNotFoundError:
|
||||
return []
|
||||
|
||||
|
||||
def _queue_head_record(directory: str) -> dict:
|
||||
file_names = _queue_file_names(directory)
|
||||
if not file_names:
|
||||
return {}
|
||||
file_path = os.path.join(directory, file_names[0])
|
||||
try:
|
||||
return _read_json_file(file_path)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _queue_record_id(record: dict, *, file_name: str = "") -> str:
|
||||
request_id = str((record or {}).get("request_id") or "").strip()
|
||||
if request_id:
|
||||
return request_id
|
||||
normalized_file_name = str(file_name or "").strip()
|
||||
if normalized_file_name.endswith(".json"):
|
||||
return normalized_file_name[:-5]
|
||||
return normalized_file_name
|
||||
|
||||
|
||||
def _queue_entries(state: str) -> list[dict]:
|
||||
normalized_state = str(state or "").strip()
|
||||
if normalized_state == "pending":
|
||||
directory = _queue_pending_dir()
|
||||
elif normalized_state == "dead_letter":
|
||||
directory = _queue_dead_letter_dir()
|
||||
else:
|
||||
return []
|
||||
|
||||
entries: list[dict] = []
|
||||
for file_name in _queue_file_names(directory):
|
||||
file_path = os.path.join(directory, file_name)
|
||||
try:
|
||||
record = _read_json_file(file_path)
|
||||
except Exception:
|
||||
continue
|
||||
entries.append(
|
||||
{
|
||||
"state": normalized_state,
|
||||
"file_name": file_name,
|
||||
"file_path": file_path,
|
||||
"record_id": _queue_record_id(record, file_name=file_name),
|
||||
"record": dict(record or {}),
|
||||
}
|
||||
)
|
||||
return entries
|
||||
|
||||
|
||||
def _matches_delivery_selector(entry: dict, selector: dict, *, allowed_states: set[str] | None = None) -> bool:
|
||||
normalized_selector = dict(selector or {})
|
||||
record = dict(entry.get("record") or {})
|
||||
state = str(entry.get("state") or "").strip()
|
||||
if allowed_states and state not in allowed_states:
|
||||
return False
|
||||
|
||||
selected_state = str(normalized_selector.get("state") or "").strip()
|
||||
if selected_state and state != selected_state:
|
||||
return False
|
||||
|
||||
selected_record_id = str(
|
||||
normalized_selector.get("record_id")
|
||||
or normalized_selector.get("request_id")
|
||||
or ""
|
||||
).strip()
|
||||
if selected_record_id and str(entry.get("record_id") or "").strip() != selected_record_id:
|
||||
return False
|
||||
|
||||
selected_request_kind = str(normalized_selector.get("request_kind") or "").strip()
|
||||
if selected_request_kind and str(record.get("kind") or "").strip() != selected_request_kind:
|
||||
return False
|
||||
|
||||
selected_detail_code = str(normalized_selector.get("detail_code") or "").strip()
|
||||
if selected_detail_code and str(record.get("last_detail_code") or "").strip() != selected_detail_code:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def _trim_queue_preview(records: list[dict], *, limit: int = 5) -> list[dict]:
|
||||
safe_limit = max(1, int(limit or 5))
|
||||
preview: list[dict] = []
|
||||
for entry in list(records or [])[:safe_limit]:
|
||||
record = dict(entry.get("record") or {})
|
||||
preview.append(
|
||||
{
|
||||
"record_id": str(entry.get("record_id") or "").strip(),
|
||||
"state": str(entry.get("state") or "").strip(),
|
||||
"request_kind": str(record.get("kind") or "").strip(),
|
||||
"request_id": str(record.get("request_id") or "").strip(),
|
||||
"detail_code": str(record.get("last_detail_code") or "").strip(),
|
||||
"created_at": str(record.get("created_at") or "").strip(),
|
||||
"updated_at": str(record.get("updated_at") or "").strip(),
|
||||
}
|
||||
)
|
||||
return preview
|
||||
|
||||
|
||||
def _normalize_queue_limit(raw_value: object, *, default: int = 20, minimum: int = 1, maximum: int = 200) -> int:
|
||||
try:
|
||||
value = int(raw_value or default)
|
||||
except Exception:
|
||||
value = int(default)
|
||||
return max(minimum, min(maximum, value))
|
||||
|
||||
|
||||
def _replay_dead_letter_records(payload: dict | None = None) -> tuple[bool, str, dict]:
|
||||
normalized_payload = dict(payload or {})
|
||||
selector = dict(normalized_payload.get("selector") or {})
|
||||
selector["state"] = str(selector.get("state") or "dead_letter").strip() or "dead_letter"
|
||||
limit = _normalize_queue_limit(normalized_payload.get("limit"), default=20)
|
||||
flush_after_replay = bool(normalized_payload.get("flush_after_replay", True))
|
||||
reason = str(normalized_payload.get("reason") or "").strip()
|
||||
|
||||
matched_entries = [
|
||||
entry
|
||||
for entry in _queue_entries("dead_letter")
|
||||
if _matches_delivery_selector(entry, selector, allowed_states={"dead_letter"})
|
||||
][:limit]
|
||||
if not matched_entries:
|
||||
return False, "未找到符合条件的死信记录", {
|
||||
"selector": selector,
|
||||
"limit": limit,
|
||||
"queue": _delivery_queue_snapshot(),
|
||||
}
|
||||
|
||||
replayed_total = 0
|
||||
for entry in matched_entries:
|
||||
record = dict(entry.get("record") or {})
|
||||
record["updated_at"] = _now_text()
|
||||
record["replay_count"] = int(record.get("replay_count") or 0) + 1
|
||||
record["last_replay_at"] = _now_text()
|
||||
if reason:
|
||||
record["last_replay_reason"] = reason
|
||||
record.pop("dead_letter_at", None)
|
||||
record.pop("dead_letter_reason", None)
|
||||
_store_pending_delivery(record)
|
||||
try:
|
||||
os.remove(str(entry.get("file_path") or ""))
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
replayed_total += 1
|
||||
|
||||
flush_summary = {}
|
||||
if flush_after_replay and replayed_total > 0:
|
||||
flush_summary = _flush_delivery_queue(limit=replayed_total)
|
||||
|
||||
result = {
|
||||
"selector": selector,
|
||||
"limit": limit,
|
||||
"replayed_total": replayed_total,
|
||||
"flush_after_replay": flush_after_replay,
|
||||
"flush_summary": flush_summary,
|
||||
"matched_records": _trim_queue_preview(matched_entries),
|
||||
"queue": _delivery_queue_snapshot(),
|
||||
}
|
||||
return True, f"已重放 {replayed_total} 条死信记录", result
|
||||
|
||||
|
||||
def _store_discarded_delivery(record: dict, *, discarded_by: str = "", reason: str = "") -> str:
|
||||
_ensure_queue_dirs()
|
||||
discarded_record = dict(record or {})
|
||||
discarded_record["discarded_at"] = _now_text()
|
||||
discarded_record["discarded_by"] = str(discarded_by or "").strip()
|
||||
discarded_record["discard_reason"] = str(reason or "").strip()
|
||||
file_name = f"{int(time.time() * 1000)}-{uuid4().hex}.json"
|
||||
file_path = os.path.join(_queue_discarded_dir(), file_name)
|
||||
_write_json_file(file_path, discarded_record)
|
||||
return file_path
|
||||
|
||||
|
||||
def _discard_dead_letter_records(payload: dict | None = None) -> tuple[bool, str, dict]:
|
||||
normalized_payload = dict(payload or {})
|
||||
selector = dict(normalized_payload.get("selector") or {})
|
||||
selector["state"] = str(selector.get("state") or "dead_letter").strip() or "dead_letter"
|
||||
limit = _normalize_queue_limit(normalized_payload.get("limit"), default=20)
|
||||
discarded_by = str(normalized_payload.get("discarded_by") or "").strip()
|
||||
reason = str(normalized_payload.get("reason") or "").strip()
|
||||
|
||||
matched_entries = [
|
||||
entry
|
||||
for entry in _queue_entries("dead_letter")
|
||||
if _matches_delivery_selector(entry, selector, allowed_states={"dead_letter"})
|
||||
][:limit]
|
||||
if not matched_entries:
|
||||
return False, "未找到符合条件的死信记录", {
|
||||
"selector": selector,
|
||||
"limit": limit,
|
||||
"queue": _delivery_queue_snapshot(),
|
||||
}
|
||||
|
||||
discarded_total = 0
|
||||
for entry in matched_entries:
|
||||
_store_discarded_delivery(
|
||||
dict(entry.get("record") or {}),
|
||||
discarded_by=discarded_by,
|
||||
reason=reason,
|
||||
)
|
||||
try:
|
||||
os.remove(str(entry.get("file_path") or ""))
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
discarded_total += 1
|
||||
|
||||
result = {
|
||||
"selector": selector,
|
||||
"limit": limit,
|
||||
"discarded_total": discarded_total,
|
||||
"discarded_by": discarded_by,
|
||||
"reason": reason,
|
||||
"matched_records": _trim_queue_preview(matched_entries),
|
||||
"queue": _delivery_queue_snapshot(),
|
||||
}
|
||||
return True, f"已丢弃 {discarded_total} 条死信记录", result
|
||||
|
||||
|
||||
def _delivery_queue_snapshot() -> dict:
|
||||
pending_dir = _queue_pending_dir()
|
||||
dead_letter_dir = _queue_dead_letter_dir()
|
||||
pending_files = _queue_file_names(pending_dir)
|
||||
dead_letter_files = _queue_file_names(dead_letter_dir)
|
||||
pending_head = _queue_head_record(pending_dir)
|
||||
dead_letter_head = _queue_head_record(dead_letter_dir)
|
||||
pending_count = len(pending_files)
|
||||
dead_letter_count = len(dead_letter_files)
|
||||
|
||||
state = "healthy"
|
||||
label = "正常"
|
||||
reason = "当前没有待重试回执,也没有死信记录。"
|
||||
if dead_letter_count > 0:
|
||||
state = "dead_letter"
|
||||
label = f"死信 {dead_letter_count}"
|
||||
reason = "存在语义失败的回执/事件,自动重试已停止,建议人工查看。"
|
||||
elif pending_count > 0:
|
||||
state = "retrying"
|
||||
label = f"待重试 {pending_count}"
|
||||
reason = "存在待重试的回执/事件,Node Agent 会在后续 heartbeat/poll 周期继续回放。"
|
||||
elif not pending_head and not dead_letter_head:
|
||||
label = "正常"
|
||||
|
||||
return {
|
||||
"state": state,
|
||||
"label": label,
|
||||
"reason": reason,
|
||||
"pending_count": pending_count,
|
||||
"dead_letter_count": dead_letter_count,
|
||||
"oldest_pending_at": str(pending_head.get("created_at") or "").strip(),
|
||||
"oldest_pending_request_id": str(pending_head.get("request_id") or "").strip(),
|
||||
"oldest_pending_kind": str(pending_head.get("kind") or "").strip(),
|
||||
"oldest_dead_letter_at": str(
|
||||
dead_letter_head.get("dead_letter_at") or dead_letter_head.get("created_at") or ""
|
||||
).strip(),
|
||||
"oldest_dead_letter_request_id": str(dead_letter_head.get("request_id") or "").strip(),
|
||||
"oldest_dead_letter_kind": str(dead_letter_head.get("kind") or "").strip(),
|
||||
"last_flush_at": str(_LAST_QUEUE_FLUSH_SUMMARY.get("last_flush_at") or "").strip(),
|
||||
"last_flush_delivered": int(_LAST_QUEUE_FLUSH_SUMMARY.get("delivered") or 0),
|
||||
"last_flush_deferred": int(_LAST_QUEUE_FLUSH_SUMMARY.get("deferred") or 0),
|
||||
"last_flush_dead_letter": int(_LAST_QUEUE_FLUSH_SUMMARY.get("dead_letter") or 0),
|
||||
}
|
||||
|
||||
|
||||
def _response_detail_code(response: dict) -> str:
|
||||
if not isinstance(response, dict):
|
||||
return ""
|
||||
top_level = str(response.get("detail_code") or "").strip()
|
||||
if top_level:
|
||||
return top_level
|
||||
data = response.get("data")
|
||||
if isinstance(data, dict):
|
||||
return str(data.get("detail_code") or "").strip()
|
||||
return ""
|
||||
|
||||
|
||||
def _ensure_ok_response(response: dict, fallback_message: str) -> dict:
|
||||
raw_code = response.get("code", 1)
|
||||
try:
|
||||
normalized_code = int(raw_code if raw_code not in (None, "") else 1)
|
||||
except Exception:
|
||||
normalized_code = 1
|
||||
if normalized_code == 0:
|
||||
return response
|
||||
raise AgentResponseError(
|
||||
str(response.get("message") or fallback_message),
|
||||
detail_code=_response_detail_code(response),
|
||||
payload=(response.get("data") if isinstance(response.get("data"), dict) else {}),
|
||||
)
|
||||
|
||||
|
||||
def _build_delivery_record(kind: str, path: str, payload: dict, request_id: str) -> dict:
|
||||
return {
|
||||
"kind": str(kind or "").strip() or "delivery",
|
||||
"path": str(path or "").strip(),
|
||||
"payload": dict(payload or {}),
|
||||
"request_id": str(request_id or "").strip(),
|
||||
"attempt_count": 0,
|
||||
"created_at": _now_text(),
|
||||
"updated_at": _now_text(),
|
||||
"last_error": "",
|
||||
"last_detail_code": "",
|
||||
}
|
||||
|
||||
|
||||
def _store_pending_delivery(record: dict) -> str:
|
||||
_ensure_queue_dirs()
|
||||
file_name = f"{int(time.time() * 1000)}-{uuid4().hex}.json"
|
||||
file_path = os.path.join(_queue_pending_dir(), file_name)
|
||||
_write_json_file(file_path, record)
|
||||
return file_path
|
||||
|
||||
|
||||
def _store_dead_letter_delivery(record: dict, *, reason: str, detail_code: str = "") -> str:
|
||||
_ensure_queue_dirs()
|
||||
dead_record = dict(record or {})
|
||||
dead_record["dead_letter_reason"] = str(reason or "").strip()
|
||||
dead_record["dead_letter_at"] = _now_text()
|
||||
dead_record["last_error"] = str(reason or "").strip()
|
||||
dead_record["last_detail_code"] = str(detail_code or "").strip()
|
||||
file_name = f"{int(time.time() * 1000)}-{uuid4().hex}.json"
|
||||
file_path = os.path.join(_queue_dead_letter_dir(), file_name)
|
||||
_write_json_file(file_path, dead_record)
|
||||
return file_path
|
||||
|
||||
|
||||
def _dispatch_delivery_record(record: dict) -> dict:
|
||||
kind = str(record.get("kind") or "delivery").strip() or "delivery"
|
||||
path = str(record.get("path") or "").strip()
|
||||
payload = record.get("payload") or {}
|
||||
timeout = 15 if kind == "job_event" else 30
|
||||
response = _post(path, payload, timeout=timeout)
|
||||
return _ensure_ok_response(response, f"{kind} failed")
|
||||
|
||||
|
||||
def _flush_delivery_queue(limit: int | None = None) -> dict:
|
||||
global _LAST_QUEUE_FLUSH_SUMMARY
|
||||
_ensure_queue_dirs()
|
||||
safe_limit = max(1, int(limit or AGENT_QUEUE_FLUSH_LIMIT or 1))
|
||||
summary = {
|
||||
"scanned": 0,
|
||||
"delivered": 0,
|
||||
"deferred": 0,
|
||||
"dead_letter": 0,
|
||||
}
|
||||
file_names = sorted(
|
||||
file_name
|
||||
for file_name in os.listdir(_queue_pending_dir())
|
||||
if file_name.endswith(".json")
|
||||
)[:safe_limit]
|
||||
for file_name in file_names:
|
||||
summary["scanned"] += 1
|
||||
file_path = os.path.join(_queue_pending_dir(), file_name)
|
||||
try:
|
||||
record = _read_json_file(file_path)
|
||||
except Exception as exc:
|
||||
_store_dead_letter_delivery({"file_name": file_name}, reason=f"invalid queue record: {exc}")
|
||||
try:
|
||||
os.remove(file_path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
summary["dead_letter"] += 1
|
||||
continue
|
||||
|
||||
try:
|
||||
_dispatch_delivery_record(record)
|
||||
try:
|
||||
os.remove(file_path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
summary["delivered"] += 1
|
||||
except AgentResponseError as exc:
|
||||
record["attempt_count"] = int(record.get("attempt_count") or 0) + 1
|
||||
record["updated_at"] = _now_text()
|
||||
record["last_error"] = str(exc)
|
||||
record["last_detail_code"] = exc.detail_code
|
||||
if exc.detail_code in _PERMANENT_DELIVERY_DETAIL_CODES:
|
||||
_store_dead_letter_delivery(record, reason=str(exc), detail_code=exc.detail_code)
|
||||
try:
|
||||
os.remove(file_path)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
summary["dead_letter"] += 1
|
||||
_log(
|
||||
f"delivery moved to dead-letter: kind={record.get('kind')} request_id={record.get('request_id')} "
|
||||
f"detail_code={exc.detail_code or '-'} message={exc}"
|
||||
)
|
||||
else:
|
||||
_write_json_file(file_path, record)
|
||||
summary["deferred"] += 1
|
||||
except Exception as exc:
|
||||
record["attempt_count"] = int(record.get("attempt_count") or 0) + 1
|
||||
record["updated_at"] = _now_text()
|
||||
record["last_error"] = str(exc)
|
||||
_write_json_file(file_path, record)
|
||||
summary["deferred"] += 1
|
||||
summary["last_flush_at"] = _now_text()
|
||||
_LAST_QUEUE_FLUSH_SUMMARY = dict(summary)
|
||||
return summary
|
||||
|
||||
|
||||
def _deliver_or_queue(
|
||||
*,
|
||||
kind: str,
|
||||
path: str,
|
||||
payload: dict,
|
||||
request_id: str,
|
||||
timeout: int = 30,
|
||||
) -> dict:
|
||||
record = _build_delivery_record(kind, path, payload, request_id)
|
||||
try:
|
||||
response = _post(path, payload, timeout=timeout)
|
||||
_ensure_ok_response(response, f"{kind} failed")
|
||||
return {"state": "delivered", "request_id": request_id, "detail_code": "", "response": response}
|
||||
except AgentResponseError as exc:
|
||||
if exc.detail_code in _PERMANENT_DELIVERY_DETAIL_CODES:
|
||||
file_path = _store_dead_letter_delivery(record, reason=str(exc), detail_code=exc.detail_code)
|
||||
_log(
|
||||
f"{kind} dead-lettered: request_id={request_id} detail_code={exc.detail_code or '-'} "
|
||||
f"message={exc} file={file_path}"
|
||||
)
|
||||
return {"state": "dead_letter", "request_id": request_id, "detail_code": exc.detail_code, "file_path": file_path}
|
||||
file_path = _store_pending_delivery({**record, "last_error": str(exc), "last_detail_code": exc.detail_code})
|
||||
_log(
|
||||
f"{kind} queued for retry: request_id={request_id} detail_code={exc.detail_code or '-'} "
|
||||
f"message={exc} file={file_path}"
|
||||
)
|
||||
return {"state": "queued", "request_id": request_id, "detail_code": exc.detail_code, "file_path": file_path}
|
||||
except Exception as exc:
|
||||
file_path = _store_pending_delivery({**record, "last_error": str(exc)})
|
||||
_log(f"{kind} queued for retry: request_id={request_id} message={exc} file={file_path}")
|
||||
return {"state": "queued", "request_id": request_id, "detail_code": "", "file_path": file_path}
|
||||
|
||||
|
||||
def _headers() -> dict[str, str]:
|
||||
return {
|
||||
"Content-Type": "application/json",
|
||||
"X-Domaincheck-Agent-Token": AGENT_TOKEN,
|
||||
}
|
||||
|
||||
|
||||
def _request(method: str, path: str, payload: dict | None = None, timeout: int = 30) -> dict:
|
||||
if not CONTROL_PLANE_BASE_URL:
|
||||
raise RuntimeError("OPS_CONTROL_PLANE_BASE_URL 未配置")
|
||||
if not AGENT_TOKEN:
|
||||
raise RuntimeError("OPS_AGENT_TOKEN 未配置")
|
||||
url = f"{CONTROL_PLANE_BASE_URL}{path}"
|
||||
data = json.dumps(payload or {}, ensure_ascii=False).encode("utf-8")
|
||||
request = urllib.request.Request(url=url, data=data, headers=_headers(), method=method.upper())
|
||||
with urllib.request.urlopen(request, timeout=timeout) as response:
|
||||
body = response.read().decode("utf-8", errors="ignore")
|
||||
return json.loads(body or "{}")
|
||||
|
||||
|
||||
def _post(path: str, payload: dict, timeout: int = 30) -> dict:
|
||||
return _request("POST", path, payload, timeout=timeout)
|
||||
|
||||
|
||||
def _hostname() -> str:
|
||||
try:
|
||||
return socket.gethostname()
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def _ip() -> str:
|
||||
try:
|
||||
return socket.gethostbyname(socket.gethostname())
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def _base_payload() -> dict:
|
||||
return {
|
||||
"node_code": NODE_CODE,
|
||||
"region": NODE_REGION,
|
||||
"role": NODE_ROLE,
|
||||
"title": NODE_CODE,
|
||||
"hostname": _hostname(),
|
||||
"ip": _ip(),
|
||||
"agent_version": AGENT_VERSION,
|
||||
"capabilities": AGENT_CAPABILITIES,
|
||||
"labels": AGENT_LABELS,
|
||||
"metadata": {
|
||||
"service_names": {
|
||||
"api": API_SERVICE_NAME,
|
||||
"worker": WORKER_SERVICE_NAME,
|
||||
"sync_agent": SYNC_AGENT_SERVICE_NAME,
|
||||
"node_agent": NODE_AGENT_SERVICE_NAME,
|
||||
},
|
||||
"delivery_queue": _delivery_queue_snapshot(),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _run(command: list[str], timeout: int = 60) -> tuple[int, str, str]:
|
||||
completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
|
||||
return completed.returncode, completed.stdout.strip(), completed.stderr.strip()
|
||||
|
||||
|
||||
def _execute_action(
|
||||
action: str,
|
||||
payload: dict,
|
||||
*,
|
||||
job_id: int | None = None,
|
||||
job_context: dict | None = None,
|
||||
) -> tuple[bool, str, dict]:
|
||||
normalized_action = str(action or "").strip()
|
||||
if normalized_action == "delivery.queue.flush":
|
||||
limit = _normalize_queue_limit((payload or {}).get("limit"), default=20)
|
||||
summary = _flush_delivery_queue(limit=limit)
|
||||
return True, "Delivery Queue 已执行冲刷", {
|
||||
"action": normalized_action,
|
||||
"limit": limit,
|
||||
"flush_summary": summary,
|
||||
"queue": _delivery_queue_snapshot(),
|
||||
}
|
||||
if normalized_action == "delivery.queue.replay":
|
||||
return _replay_dead_letter_records(payload)
|
||||
if normalized_action == "delivery.queue.discard":
|
||||
return _discard_dead_letter_records(payload)
|
||||
if supports_structured_action(normalized_action):
|
||||
return execute_structured_action(
|
||||
normalized_action,
|
||||
payload,
|
||||
service_names=build_service_name_map(
|
||||
api_service_name=API_SERVICE_NAME,
|
||||
worker_service_name=WORKER_SERVICE_NAME,
|
||||
sync_agent_service_name=SYNC_AGENT_SERVICE_NAME,
|
||||
node_agent_service_name=NODE_AGENT_SERVICE_NAME,
|
||||
),
|
||||
runner=_run,
|
||||
host_context={
|
||||
"hostname": _hostname(),
|
||||
"ip": _ip(),
|
||||
},
|
||||
)
|
||||
if normalized_action == "deploy.release":
|
||||
return execute_release_action(
|
||||
dict(payload or {}),
|
||||
run_command=_run,
|
||||
default_api_service_name=API_SERVICE_NAME,
|
||||
event_callback=(
|
||||
(lambda event_type, message, level="info", payload=None: _job_event(
|
||||
int(job_id),
|
||||
event_type=event_type,
|
||||
message=message,
|
||||
level=level,
|
||||
payload={
|
||||
**dict(payload or {}),
|
||||
"release_context": dict((job_context or {}).get("release_context") or {}),
|
||||
"step_key": str((job_context or {}).get("step_key") or "").strip(),
|
||||
},
|
||||
summary_text=message,
|
||||
focus_ref=dict((job_context or {}).get("focus_ref") or {}),
|
||||
occurred_at=_now_text(),
|
||||
))
|
||||
if job_id
|
||||
else None
|
||||
),
|
||||
urlopen_func=urllib.request.urlopen,
|
||||
user_agent=f"domaincheck-node-agent/{AGENT_VERSION}",
|
||||
)
|
||||
return False, f"unsupported action: {normalized_action}", {"action": normalized_action}
|
||||
|
||||
|
||||
def _register() -> None:
|
||||
response = _post("/api/v1/ops/agent/register", _base_payload())
|
||||
_ensure_ok_response(response, "agent register failed")
|
||||
_log(f"registered: {response.get('message')}")
|
||||
|
||||
|
||||
def _heartbeat() -> None:
|
||||
response = _post("/api/v1/ops/agent/heartbeat", _base_payload())
|
||||
_ensure_ok_response(response, "agent heartbeat failed")
|
||||
|
||||
|
||||
def _pull_jobs() -> list[dict]:
|
||||
response = _post(f"/api/v1/ops/agent/pull?limit=1", {"node_code": NODE_CODE})
|
||||
_ensure_ok_response(response, "agent pull failed")
|
||||
data = response.get("data") or {}
|
||||
return list(data.get("jobs") or [])
|
||||
|
||||
|
||||
def _normalize_agent_job(job: dict) -> dict:
|
||||
normalized_job = dict(job or {})
|
||||
focus_ref = normalized_job.get("focus_ref") if isinstance(normalized_job.get("focus_ref"), dict) else {}
|
||||
release_context = (
|
||||
normalized_job.get("release_context") if isinstance(normalized_job.get("release_context"), dict) else {}
|
||||
)
|
||||
step_ref = normalized_job.get("step_ref") if isinstance(normalized_job.get("step_ref"), dict) else {}
|
||||
job_id = int(normalized_job.get("job_id") or normalized_job.get("id") or 0)
|
||||
step_key = str(
|
||||
normalized_job.get("step_key")
|
||||
or step_ref.get("step_key")
|
||||
or "dispatch"
|
||||
).strip() or "dispatch"
|
||||
step_title = str(
|
||||
normalized_job.get("step_title")
|
||||
or step_ref.get("step_title")
|
||||
or normalized_job.get("summary")
|
||||
or normalized_job.get("action")
|
||||
or step_key
|
||||
).strip() or step_key
|
||||
if not focus_ref:
|
||||
focus_ref = {
|
||||
"kind": "ops_job",
|
||||
"job_id": job_id,
|
||||
"job_code": str(normalized_job.get("job_code") or "").strip(),
|
||||
"action": str(normalized_job.get("action") or "").strip(),
|
||||
"target_node_code": str(normalized_job.get("target_node_code") or "").strip(),
|
||||
}
|
||||
return {
|
||||
**normalized_job,
|
||||
"job_id": job_id,
|
||||
"job_code": str(normalized_job.get("job_code") or "").strip(),
|
||||
"job_type": str(normalized_job.get("job_type") or "ops_action").strip() or "ops_action",
|
||||
"action": str(normalized_job.get("action") or "").strip(),
|
||||
"payload": dict(normalized_job.get("payload") or {}),
|
||||
"policy": dict(normalized_job.get("policy") or {}),
|
||||
"focus_ref": focus_ref,
|
||||
"release_context": dict(release_context or {}),
|
||||
"step_key": step_key,
|
||||
"step_title": step_title,
|
||||
"step_ref": {
|
||||
"step_id": int(step_ref.get("step_id") or 0),
|
||||
"step_key": step_key,
|
||||
"step_title": step_title,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _job_start(job_id: int, *, job: dict | None = None) -> None:
|
||||
normalized_job = _normalize_agent_job(job or {"job_id": job_id})
|
||||
response = _post(
|
||||
f"/api/v1/ops/agent/jobs/{job_id}/start",
|
||||
{
|
||||
"node_code": NODE_CODE,
|
||||
"job_code": normalized_job.get("job_code") or "",
|
||||
"action": normalized_job.get("action") or "",
|
||||
"step_key": normalized_job.get("step_key") or "",
|
||||
"focus_ref": normalized_job.get("focus_ref") or {},
|
||||
},
|
||||
)
|
||||
_ensure_ok_response(response, "job start failed")
|
||||
|
||||
|
||||
def _job_complete(
|
||||
job_id: int,
|
||||
*,
|
||||
status: str,
|
||||
stdout: str,
|
||||
stderr: str,
|
||||
result: dict,
|
||||
error_message: str = "",
|
||||
client_request_id: str | None = None,
|
||||
duration_ms: int | None = None,
|
||||
summary_text: str = "",
|
||||
focus_ref: dict | None = None,
|
||||
step_ref: dict | None = None,
|
||||
release_context: dict | None = None,
|
||||
) -> dict:
|
||||
request_id = str(client_request_id or "").strip() or _new_request_id("complete", job_id)
|
||||
payload = {
|
||||
"node_code": NODE_CODE,
|
||||
"status": status,
|
||||
"stdout": stdout,
|
||||
"stderr": stderr,
|
||||
"result": result,
|
||||
"error_message": error_message,
|
||||
"client_request_id": request_id,
|
||||
}
|
||||
if duration_ms is not None:
|
||||
payload["duration_ms"] = max(0, int(duration_ms or 0))
|
||||
if str(summary_text or "").strip():
|
||||
payload["summary_text"] = str(summary_text).strip()
|
||||
if isinstance(focus_ref, dict) and focus_ref:
|
||||
payload["focus_ref"] = dict(focus_ref)
|
||||
if isinstance(step_ref, dict) and step_ref:
|
||||
payload["step_ref"] = dict(step_ref)
|
||||
if isinstance(release_context, dict) and release_context:
|
||||
payload["release_context"] = dict(release_context)
|
||||
return _deliver_or_queue(
|
||||
kind="job_complete",
|
||||
path=f"/api/v1/ops/agent/jobs/{job_id}/complete",
|
||||
payload=payload,
|
||||
request_id=request_id,
|
||||
timeout=30,
|
||||
)
|
||||
|
||||
|
||||
def _job_event(
|
||||
job_id: int,
|
||||
*,
|
||||
event_type: str,
|
||||
message: str,
|
||||
level: str = "info",
|
||||
payload: dict | None = None,
|
||||
client_event_id: str | None = None,
|
||||
summary_text: str = "",
|
||||
focus_ref: dict | None = None,
|
||||
occurred_at: str = "",
|
||||
) -> dict:
|
||||
request_id = str(client_event_id or "").strip() or _new_request_id("event", job_id)
|
||||
delivery_payload = {
|
||||
"node_code": NODE_CODE,
|
||||
"event_type": event_type,
|
||||
"message": message,
|
||||
"level": level,
|
||||
"payload": payload or {},
|
||||
"client_event_id": request_id,
|
||||
}
|
||||
if str(summary_text or "").strip():
|
||||
delivery_payload["summary_text"] = str(summary_text).strip()
|
||||
if isinstance(focus_ref, dict) and focus_ref:
|
||||
delivery_payload["focus_ref"] = dict(focus_ref)
|
||||
if str(occurred_at or "").strip():
|
||||
delivery_payload["occurred_at"] = str(occurred_at).strip()
|
||||
return _deliver_or_queue(
|
||||
kind="job_event",
|
||||
path=f"/api/v1/ops/agent/jobs/{job_id}/events",
|
||||
payload=delivery_payload,
|
||||
request_id=request_id,
|
||||
timeout=15,
|
||||
)
|
||||
|
||||
|
||||
def _process_job(job: dict) -> None:
|
||||
normalized_job = _normalize_agent_job(job)
|
||||
job_id = int(normalized_job.get("job_id") or 0)
|
||||
action = str(normalized_job.get("action") or "").strip()
|
||||
payload = dict(normalized_job.get("payload") or {})
|
||||
if job_id <= 0 or not action:
|
||||
return
|
||||
|
||||
started_at = time.monotonic()
|
||||
_log(
|
||||
"job start: "
|
||||
f"id={job_id} code={normalized_job.get('job_code') or '-'} "
|
||||
f"type={normalized_job.get('job_type') or '-'} "
|
||||
f"step={normalized_job.get('step_key') or '-'} action={action}"
|
||||
)
|
||||
start_delivery_state = "delivered"
|
||||
start_delivery_error = ""
|
||||
try:
|
||||
_job_start(job_id, job=normalized_job)
|
||||
except Exception as exc:
|
||||
start_delivery_state = "failed_local"
|
||||
start_delivery_error = str(exc)
|
||||
_log(
|
||||
"job start delivery failed, continue locally: "
|
||||
f"id={job_id} code={normalized_job.get('job_code') or '-'} action={action} error={exc}"
|
||||
)
|
||||
_job_event(
|
||||
job_id,
|
||||
event_type="executor_received",
|
||||
message=f"node agent accepted action {action}",
|
||||
summary_text=f"已接单 {action}",
|
||||
focus_ref=dict(normalized_job.get("focus_ref") or {}),
|
||||
occurred_at=_now_text(),
|
||||
payload={
|
||||
"action": action,
|
||||
"job_code": normalized_job.get("job_code") or "",
|
||||
"job_type": normalized_job.get("job_type") or "",
|
||||
"step_key": normalized_job.get("step_key") or "",
|
||||
"step_title": normalized_job.get("step_title") or "",
|
||||
"release_context": dict(normalized_job.get("release_context") or {}),
|
||||
"start_delivery_state": start_delivery_state,
|
||||
"start_delivery_error": start_delivery_error,
|
||||
},
|
||||
)
|
||||
ok, message, result = _execute_action(action, payload, job_id=job_id, job_context=normalized_job)
|
||||
stdout = str(result.get("stdout") or "")
|
||||
stderr = str(result.get("stderr") or "")
|
||||
duration_ms = max(0, int((time.monotonic() - started_at) * 1000))
|
||||
summary_text = str(result.get("summary_text") or result.get("summary") or message).strip()
|
||||
delivery = _job_complete(
|
||||
job_id,
|
||||
status="success" if ok else "failed",
|
||||
stdout=stdout,
|
||||
stderr=stderr,
|
||||
result=result,
|
||||
error_message="" if ok else message,
|
||||
duration_ms=duration_ms,
|
||||
summary_text=summary_text,
|
||||
focus_ref=dict(normalized_job.get("focus_ref") or {}),
|
||||
step_ref=dict(normalized_job.get("step_ref") or {}),
|
||||
release_context=dict(normalized_job.get("release_context") or {}),
|
||||
)
|
||||
_log(
|
||||
"job complete: "
|
||||
f"id={job_id} code={normalized_job.get('job_code') or '-'} "
|
||||
f"action={action} ok={ok} duration_ms={duration_ms} delivery={delivery.get('state')}"
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if not NODE_CODE:
|
||||
raise RuntimeError("NODE_CODE 未配置")
|
||||
_log(f"starting node agent: node={NODE_CODE} role={NODE_ROLE} region={NODE_REGION}")
|
||||
_ensure_queue_dirs()
|
||||
_register()
|
||||
last_heartbeat_at = 0.0
|
||||
|
||||
while True:
|
||||
now = time.time()
|
||||
try:
|
||||
delivery_summary = _flush_delivery_queue(limit=AGENT_QUEUE_FLUSH_LIMIT)
|
||||
if delivery_summary["delivered"] or delivery_summary["dead_letter"]:
|
||||
_log(f"delivery queue flush: {delivery_summary}")
|
||||
if now - last_heartbeat_at >= 15:
|
||||
_heartbeat()
|
||||
last_heartbeat_at = now
|
||||
jobs = _pull_jobs()
|
||||
if jobs:
|
||||
for job in jobs:
|
||||
_process_job(job)
|
||||
else:
|
||||
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
|
||||
except urllib.error.HTTPError as exc:
|
||||
_log(f"http error: {exc.code}")
|
||||
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
|
||||
except urllib.error.URLError as exc:
|
||||
_log(f"url error: {exc}")
|
||||
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
|
||||
except Exception as exc:
|
||||
_log(f"loop error: {exc}")
|
||||
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,8 +1,10 @@
|
||||
from typing import Any, Optional
|
||||
|
||||
from pydantic import BaseModel
|
||||
from typing import Any
|
||||
|
||||
|
||||
class ApiResponse(BaseModel):
|
||||
code: int = 0
|
||||
message: str = "ok"
|
||||
data: Any = None
|
||||
detail_code: Optional[str] = None
|
||||
|
||||
241
domain-api/app/services/build_info_service.py
Normal file
241
domain-api/app/services/build_info_service.py
Normal file
@@ -0,0 +1,241 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
|
||||
DOMAIN_API_ROOT = Path(__file__).resolve().parents[2]
|
||||
WORKSPACE_ROOT = DOMAIN_API_ROOT.parent
|
||||
_REGISTERED_ROUTE_PATHS: set[str] = set()
|
||||
|
||||
|
||||
def remember_registered_route_paths(route_paths: Iterable[str] | None) -> None:
|
||||
global _REGISTERED_ROUTE_PATHS
|
||||
normalized = {
|
||||
str(item).strip()
|
||||
for item in list(route_paths or [])
|
||||
if str(item).strip()
|
||||
}
|
||||
_REGISTERED_ROUTE_PATHS = normalized
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict:
|
||||
try:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _discover_manifest_candidate() -> tuple[Path | None, str]:
|
||||
configured = str(settings.build_manifest_path or "").strip()
|
||||
if configured:
|
||||
path = Path(configured)
|
||||
if path.exists():
|
||||
return path, "configured_manifest"
|
||||
|
||||
root_manifest = WORKSPACE_ROOT / "release_manifest.json"
|
||||
if root_manifest.exists():
|
||||
return root_manifest, "release_manifest"
|
||||
|
||||
latest_release = WORKSPACE_ROOT / "release" / "latest_release.json"
|
||||
if latest_release.exists():
|
||||
return latest_release, "latest_release"
|
||||
|
||||
return None, ""
|
||||
|
||||
|
||||
def _build_info_from_env() -> dict | None:
|
||||
if not any(
|
||||
[
|
||||
str(settings.build_commit_sha or "").strip(),
|
||||
str(settings.build_commit_ref or "").strip(),
|
||||
str(settings.build_generated_at or "").strip(),
|
||||
str(settings.build_package_name or "").strip(),
|
||||
str(settings.build_checksum or "").strip(),
|
||||
]
|
||||
):
|
||||
return None
|
||||
|
||||
return {
|
||||
"source": str(settings.build_source_label or "").strip() or "env",
|
||||
"package_name": str(settings.build_package_name or "").strip(),
|
||||
"generated_at": str(settings.build_generated_at or "").strip(),
|
||||
"commit_sha": str(settings.build_commit_sha or "").strip(),
|
||||
"commit_ref": str(settings.build_commit_ref or "").strip(),
|
||||
"checksum": str(settings.build_checksum or "").strip(),
|
||||
"manifest_path": str(settings.build_manifest_path or "").strip(),
|
||||
}
|
||||
|
||||
|
||||
def _build_info_from_manifest() -> dict | None:
|
||||
manifest_path, source = _discover_manifest_candidate()
|
||||
if manifest_path is None:
|
||||
return None
|
||||
|
||||
payload = _read_json(manifest_path)
|
||||
if not payload:
|
||||
return {
|
||||
"source": source,
|
||||
"package_name": "",
|
||||
"generated_at": "",
|
||||
"commit_sha": "",
|
||||
"commit_ref": "",
|
||||
"checksum": "",
|
||||
"manifest_path": str(manifest_path),
|
||||
}
|
||||
|
||||
checksum = str(payload.get("checksum") or payload.get("sha256") or "").strip()
|
||||
return {
|
||||
"source": source,
|
||||
"package_name": str(payload.get("package_name") or "").strip(),
|
||||
"generated_at": str(payload.get("generated_at") or "").strip(),
|
||||
"commit_sha": str(payload.get("commit_sha") or "").strip(),
|
||||
"commit_ref": str(payload.get("commit_ref") or "").strip(),
|
||||
"checksum": checksum,
|
||||
"manifest_path": str(manifest_path),
|
||||
}
|
||||
|
||||
|
||||
def _build_info_from_git() -> dict | None:
|
||||
try:
|
||||
inside_worktree = subprocess.run(
|
||||
["git", "-C", str(WORKSPACE_ROOT), "rev-parse", "--is-inside-work-tree"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=2,
|
||||
)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
if inside_worktree.returncode != 0 or str(inside_worktree.stdout or "").strip() != "true":
|
||||
return None
|
||||
|
||||
def _read_git(*args: str) -> str:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "-C", str(WORKSPACE_ROOT), *args],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=2,
|
||||
)
|
||||
except Exception:
|
||||
return ""
|
||||
if result.returncode != 0:
|
||||
return ""
|
||||
return str(result.stdout or "").strip()
|
||||
|
||||
return {
|
||||
"source": "git",
|
||||
"package_name": "",
|
||||
"generated_at": "",
|
||||
"commit_sha": _read_git("rev-parse", "--short=12", "HEAD"),
|
||||
"commit_ref": _read_git("rev-parse", "--abbrev-ref", "HEAD"),
|
||||
"checksum": "",
|
||||
"manifest_path": "",
|
||||
}
|
||||
|
||||
|
||||
def _resolve_build_identity() -> dict:
|
||||
return (
|
||||
_build_info_from_env()
|
||||
or _build_info_from_manifest()
|
||||
or _build_info_from_git()
|
||||
or {
|
||||
"source": "unknown",
|
||||
"package_name": "",
|
||||
"generated_at": "",
|
||||
"commit_sha": "",
|
||||
"commit_ref": "",
|
||||
"checksum": "",
|
||||
"manifest_path": "",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _expected_route_paths() -> dict[str, str]:
|
||||
prefix = str(settings.api_prefix or "/api/v1").rstrip("/")
|
||||
return {
|
||||
"ops_contracts": f"{prefix}/ops/contracts",
|
||||
"ops_contract_detail": f"{prefix}/ops/contracts/{{contract_key}}",
|
||||
"ops_stack_diagnosis": f"{prefix}/ops/stack-diagnosis",
|
||||
"ops_node_handover": f"{prefix}/ops/nodes/{{node_code}}/handover",
|
||||
"ops_node_onboarding": f"{prefix}/ops/nodes/{{node_code}}/onboarding",
|
||||
"ops_node_onboarding_bootstrap_preview": f"{prefix}/ops/nodes/{{node_code}}/onboarding/bootstrap/preview",
|
||||
"ops_node_onboarding_bootstrap_execute": f"{prefix}/ops/nodes/{{node_code}}/onboarding/bootstrap/execute",
|
||||
"ops_node_onboarding_acceptance_preview": f"{prefix}/ops/nodes/{{node_code}}/onboarding/acceptance/preview",
|
||||
"ops_node_onboarding_acceptance_execute": f"{prefix}/ops/nodes/{{node_code}}/onboarding/acceptance/execute",
|
||||
"ops_node_onboarding_recovery_preview": f"{prefix}/ops/nodes/{{node_code}}/onboarding/recovery/preview",
|
||||
"ops_node_onboarding_recovery_execute": f"{prefix}/ops/nodes/{{node_code}}/onboarding/recovery/execute",
|
||||
"ops_node_scene_log": f"{prefix}/ops/nodes/{{node_code}}/scene-log",
|
||||
"ops_node_handover_bootstrap_plan": f"{prefix}/ops/nodes/{{node_code}}/handover/bootstrap-plan",
|
||||
"ops_driver_feed": f"{prefix}/ops/driver-feed",
|
||||
"ops_codex_brief": f"{prefix}/ops/codex-brief",
|
||||
"ops_activity_stream": f"{prefix}/ops/activity-stream",
|
||||
"ops_runbook_resolve": f"{prefix}/ops/runbook/sequences/{{sequence_key}}/resolve",
|
||||
"ops_runbook_execute": f"{prefix}/ops/runbook/sequences/{{sequence_key}}/execute",
|
||||
"runtime_build_info": f"{prefix}/runtime/build-info",
|
||||
}
|
||||
|
||||
|
||||
def _read_text(path: Path) -> str:
|
||||
try:
|
||||
return path.read_text(encoding="utf-8", errors="replace")
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
|
||||
def _repository_capabilities() -> dict:
|
||||
ops_agent_service_text = _read_text(DOMAIN_API_ROOT / "app" / "services" / "ops_agent_service.py")
|
||||
bootstrap_node_agent_text = _read_text(DOMAIN_API_ROOT / "deploy" / "multi-region" / "bootstrap_node_agent.sh")
|
||||
return {
|
||||
"supports_install_command_block": (
|
||||
"install_command_block" in ops_agent_service_text
|
||||
and "_candidate_node_agent_install_paths" in ops_agent_service_text
|
||||
),
|
||||
"supports_multi_layout_bootstrap": "resolve_project_root" in bootstrap_node_agent_text,
|
||||
}
|
||||
|
||||
|
||||
def _build_route_surface(route_paths: Iterable[str] | None = None) -> dict:
|
||||
expected = _expected_route_paths()
|
||||
normalized_paths = {
|
||||
str(item).strip()
|
||||
for item in (list(route_paths) if route_paths is not None else list(_REGISTERED_ROUTE_PATHS))
|
||||
if str(item).strip()
|
||||
}
|
||||
mode = "registered" if normalized_paths else "declared_contract"
|
||||
flags = {key: (path in normalized_paths if normalized_paths else True) for key, path in expected.items()}
|
||||
missing_keys = [key for key, available in flags.items() if not available]
|
||||
return {
|
||||
"mode": mode,
|
||||
"registered_paths_total": len(normalized_paths),
|
||||
"surface_flags": flags,
|
||||
"expected_paths": expected,
|
||||
"missing_keys": missing_keys,
|
||||
"missing_paths": [expected[key] for key in missing_keys],
|
||||
"surface_complete": len(missing_keys) == 0,
|
||||
}
|
||||
|
||||
|
||||
def get_runtime_build_info(route_paths: Iterable[str] | None = None) -> dict:
|
||||
build = _resolve_build_identity()
|
||||
route_surface = _build_route_surface(route_paths=route_paths)
|
||||
return {
|
||||
"source": str(build.get("source") or "").strip() or "unknown",
|
||||
"package_name": str(build.get("package_name") or "").strip(),
|
||||
"generated_at": str(build.get("generated_at") or "").strip(),
|
||||
"commit_sha": str(build.get("commit_sha") or "").strip(),
|
||||
"commit_ref": str(build.get("commit_ref") or "").strip(),
|
||||
"checksum": str(build.get("checksum") or "").strip(),
|
||||
"manifest_path": str(build.get("manifest_path") or "").strip(),
|
||||
"workspace_root": str(WORKSPACE_ROOT),
|
||||
"domain_api_root": str(DOMAIN_API_ROOT),
|
||||
"repository_capabilities": _repository_capabilities(),
|
||||
"route_surface": route_surface,
|
||||
}
|
||||
@@ -147,15 +147,50 @@ def _build_remote_log_lines(
|
||||
mode: str,
|
||||
limit: int = 240,
|
||||
) -> list[str]:
|
||||
return _build_remote_log_snapshot(active_job, enabled=enabled, mode=mode, limit=limit)["lines"]
|
||||
|
||||
|
||||
def _build_remote_log_snapshot(
|
||||
active_job: dict | None,
|
||||
*,
|
||||
enabled: bool,
|
||||
mode: str,
|
||||
limit: int = 240,
|
||||
) -> dict:
|
||||
if not enabled:
|
||||
return []
|
||||
return {
|
||||
"lines": [],
|
||||
"line_count": 0,
|
||||
"last_at": "",
|
||||
"last_line": "",
|
||||
"source_nodes": [],
|
||||
"source_node_count": 0,
|
||||
}
|
||||
if not active_job:
|
||||
return []
|
||||
return {
|
||||
"lines": [],
|
||||
"line_count": 0,
|
||||
"last_at": "",
|
||||
"last_line": "",
|
||||
"source_nodes": [],
|
||||
"source_node_count": 0,
|
||||
}
|
||||
events = list(active_job.get("current_cycle_events") or active_job.get("recent_events") or [])
|
||||
if not events:
|
||||
return []
|
||||
return {
|
||||
"lines": [],
|
||||
"line_count": 0,
|
||||
"last_at": "",
|
||||
"last_line": "",
|
||||
"source_nodes": [],
|
||||
"source_node_count": 0,
|
||||
}
|
||||
|
||||
lines: list[str] = []
|
||||
source_nodes: set[str] = set()
|
||||
source_node_summaries: dict[str, dict] = {}
|
||||
last_at = ""
|
||||
last_line = ""
|
||||
normalized_mode = str(mode or "key").strip().lower()
|
||||
if normalized_mode not in {"key", "full"}:
|
||||
normalized_mode = "key"
|
||||
@@ -180,8 +215,47 @@ def _build_remote_log_lines(
|
||||
continue
|
||||
if len(message) > _REMOTE_LOG_MAX_CHARS:
|
||||
message = f"{message[:_REMOTE_LOG_MAX_CHARS]}..."
|
||||
lines.append(f"[{created_at}] [{node_code}] {message}")
|
||||
return lines[-max(1, int(limit or 240)) :]
|
||||
formatted_line = f"[{created_at}] [{node_code}] {message}"
|
||||
lines.append(formatted_line)
|
||||
source_nodes.add(node_code)
|
||||
node_summary = source_node_summaries.setdefault(
|
||||
node_code,
|
||||
{
|
||||
"node_code": node_code,
|
||||
"line_count": 0,
|
||||
"key_line_count": 0,
|
||||
"full_line_count": 0,
|
||||
"last_at": "",
|
||||
"last_line": "",
|
||||
},
|
||||
)
|
||||
node_summary["line_count"] += 1
|
||||
if event_mode == "full":
|
||||
node_summary["full_line_count"] += 1
|
||||
else:
|
||||
node_summary["key_line_count"] += 1
|
||||
node_summary["last_at"] = created_at
|
||||
node_summary["last_line"] = formatted_line
|
||||
last_at = created_at
|
||||
last_line = formatted_line
|
||||
sliced_lines = lines[-max(1, int(limit or 240)) :]
|
||||
sorted_source_node_summaries = sorted(
|
||||
source_node_summaries.values(),
|
||||
key=lambda item: (
|
||||
str(item.get("last_at") or ""),
|
||||
str(item.get("node_code") or ""),
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
return {
|
||||
"lines": sliced_lines,
|
||||
"line_count": len(sliced_lines),
|
||||
"last_at": last_at,
|
||||
"last_line": last_line,
|
||||
"source_nodes": sorted(source_nodes),
|
||||
"source_node_count": len(source_nodes),
|
||||
"source_node_summaries": sorted_source_node_summaries,
|
||||
}
|
||||
|
||||
|
||||
def _resolve_remote_log_lines(
|
||||
@@ -195,6 +269,17 @@ def _resolve_remote_log_lines(
|
||||
return _build_remote_log_lines(active_job, enabled=enabled, mode=mode, limit=limit)
|
||||
|
||||
|
||||
def _resolve_remote_log_snapshot(
|
||||
active_job: dict | None,
|
||||
runs: list[dict],
|
||||
*,
|
||||
enabled: bool,
|
||||
mode: str,
|
||||
limit: int = 240,
|
||||
) -> dict:
|
||||
return _build_remote_log_snapshot(active_job, enabled=enabled, mode=mode, limit=limit)
|
||||
|
||||
|
||||
def _load_runtime_state() -> dict:
|
||||
try:
|
||||
redis_client = get_redis()
|
||||
@@ -404,13 +489,14 @@ def get_detect_status() -> dict:
|
||||
runs = sync_detect_runs(runtime_snapshot, progress, settings_summary, active_job=active_job)
|
||||
worker_log_sync_enabled = bool(runtime_settings.get("worker_log_sync_enabled", False))
|
||||
worker_log_sync_mode = str(runtime_settings.get("worker_log_sync_mode", "key") or "key")
|
||||
remote_log_lines = _resolve_remote_log_lines(
|
||||
remote_log_snapshot = _resolve_remote_log_snapshot(
|
||||
active_job,
|
||||
runs,
|
||||
enabled=worker_log_sync_enabled,
|
||||
mode=worker_log_sync_mode,
|
||||
limit=240,
|
||||
)
|
||||
remote_log_lines = list(remote_log_snapshot.get("lines") or [])
|
||||
dependency_alerts = _extract_dependency_alerts(recent_lines)
|
||||
append_detect_result_projection_if_changed(
|
||||
detect={
|
||||
@@ -465,6 +551,12 @@ def get_detect_status() -> dict:
|
||||
"recent_warning": recent_proxy_warning,
|
||||
"log_lines": recent_lines,
|
||||
"remote_log_lines": remote_log_lines,
|
||||
"remote_log_line_count": int(remote_log_snapshot.get("line_count", 0) or 0),
|
||||
"remote_log_last_at": str(remote_log_snapshot.get("last_at") or ""),
|
||||
"remote_log_last_line": str(remote_log_snapshot.get("last_line") or ""),
|
||||
"remote_log_nodes": list(remote_log_snapshot.get("source_nodes") or []),
|
||||
"remote_log_node_count": int(remote_log_snapshot.get("source_node_count", 0) or 0),
|
||||
"remote_log_node_summaries": list(remote_log_snapshot.get("source_node_summaries") or []),
|
||||
"runs": runs,
|
||||
"active_job": active_job,
|
||||
"worker_log_sync_enabled": worker_log_sync_enabled,
|
||||
|
||||
278
domain-api/app/services/ops_action_executor_core.py
Normal file
278
domain-api/app/services/ops_action_executor_core.py
Normal file
@@ -0,0 +1,278 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
|
||||
STRUCTURED_ACTIONS = {
|
||||
"health.snapshot",
|
||||
"service.status",
|
||||
"service.restart",
|
||||
"service.start",
|
||||
"service.stop",
|
||||
"logs.collect",
|
||||
"diagnostics.collect",
|
||||
"runtime.start_worker",
|
||||
"runtime.stop_worker",
|
||||
"runtime.restart_api",
|
||||
"runtime.start_sync_agent",
|
||||
"runtime.stop_sync_agent",
|
||||
}
|
||||
|
||||
|
||||
class CommandRunner(Protocol):
|
||||
def __call__(self, command: list[str], *, timeout: int = 60) -> tuple[int, str, str]:
|
||||
...
|
||||
|
||||
|
||||
def supports_structured_action(action: str) -> bool:
|
||||
return str(action or "").strip() in STRUCTURED_ACTIONS
|
||||
|
||||
|
||||
def trim_output(text: str, limit: int) -> str:
|
||||
normalized = str(text or "").strip()
|
||||
if len(normalized) <= int(limit):
|
||||
return normalized
|
||||
return normalized[-int(limit):]
|
||||
|
||||
|
||||
def _read_tail_lines(path: Path, *, max_lines: int) -> list[str]:
|
||||
if not path.exists() or not path.is_file():
|
||||
return []
|
||||
with path.open("r", encoding="utf-8", errors="replace") as handle:
|
||||
return handle.read().splitlines()[-max_lines:]
|
||||
|
||||
|
||||
def _runtime_logs_dir() -> Path:
|
||||
return Path(__file__).resolve().parents[2] / "runtime" / "logs"
|
||||
|
||||
|
||||
def _service_log_file_candidates(service_name: str, *, service_names: dict[str, str]) -> list[Path]:
|
||||
normalized_service_name = str(service_name or "").strip()
|
||||
if not normalized_service_name:
|
||||
return []
|
||||
|
||||
domain_root = Path(settings.domain_root)
|
||||
runtime_logs_dir = _runtime_logs_dir()
|
||||
candidates: list[Path] = []
|
||||
|
||||
def append_candidate(path: Path) -> None:
|
||||
if path not in candidates:
|
||||
candidates.append(path)
|
||||
|
||||
if normalized_service_name == (service_names.get("worker") or "domaincheck-worker"):
|
||||
append_candidate(domain_root / "detect_worker.log")
|
||||
append_candidate(domain_root / "logs" / "detect_worker.log")
|
||||
elif normalized_service_name == (service_names.get("api") or "domaincheck-api"):
|
||||
append_candidate(runtime_logs_dir / "domain-api.stderr.log")
|
||||
append_candidate(runtime_logs_dir / "domain-api.stdout.log")
|
||||
append_candidate(domain_root / "logs" / "app.log")
|
||||
|
||||
return candidates
|
||||
|
||||
|
||||
def _collect_log_file_fallback(
|
||||
service_name: str,
|
||||
*,
|
||||
lines: int,
|
||||
service_names: dict[str, str],
|
||||
) -> tuple[str, list[str]]:
|
||||
aggregated_lines: list[str] = []
|
||||
used_paths: list[str] = []
|
||||
for path in _service_log_file_candidates(service_name, service_names=service_names):
|
||||
current_lines = _read_tail_lines(path, max_lines=lines)
|
||||
if not current_lines:
|
||||
continue
|
||||
aggregated_lines.extend(current_lines)
|
||||
used_paths.append(str(path))
|
||||
if not aggregated_lines:
|
||||
return "", []
|
||||
return "\n".join(aggregated_lines[-lines:]), used_paths
|
||||
|
||||
|
||||
def build_service_name_map(
|
||||
*,
|
||||
api_service_name: str,
|
||||
worker_service_name: str,
|
||||
sync_agent_service_name: str,
|
||||
node_agent_service_name: str = "domaincheck-node-agent",
|
||||
) -> dict[str, str]:
|
||||
return {
|
||||
"api": str(api_service_name or "").strip() or "domaincheck-api",
|
||||
"worker": str(worker_service_name or "").strip() or "domaincheck-worker",
|
||||
"sync_agent": str(sync_agent_service_name or "").strip() or "domaincheck-sync-agent",
|
||||
"node_agent": str(node_agent_service_name or "").strip() or "domaincheck-node-agent",
|
||||
}
|
||||
|
||||
|
||||
def service_name_from_payload(payload: dict, *, default_worker_service_name: str) -> str:
|
||||
service_name = str((payload or {}).get("service_name") or "").strip()
|
||||
return service_name or str(default_worker_service_name or "").strip() or "domaincheck-worker"
|
||||
|
||||
|
||||
def service_name_for_action(action: str, payload: dict, service_names: dict[str, str]) -> str:
|
||||
normalized_action = str(action or "").strip()
|
||||
if normalized_action in {"service.status", "service.restart", "service.start", "service.stop"}:
|
||||
return service_name_from_payload(payload, default_worker_service_name=service_names.get("worker") or "")
|
||||
runtime_action_map = {
|
||||
"runtime.start_worker": service_names.get("worker") or "domaincheck-worker",
|
||||
"runtime.stop_worker": service_names.get("worker") or "domaincheck-worker",
|
||||
"runtime.restart_api": service_names.get("api") or "domaincheck-api",
|
||||
"runtime.start_sync_agent": service_names.get("sync_agent") or "domaincheck-sync-agent",
|
||||
"runtime.stop_sync_agent": service_names.get("sync_agent") or "domaincheck-sync-agent",
|
||||
}
|
||||
return str(runtime_action_map.get(normalized_action) or "").strip()
|
||||
|
||||
|
||||
def systemctl_action_name(action: str) -> str:
|
||||
normalized_action = str(action or "").strip()
|
||||
if normalized_action in {"service.restart", "runtime.restart_api"}:
|
||||
return "restart"
|
||||
if normalized_action in {"service.start", "runtime.start_worker", "runtime.start_sync_agent"}:
|
||||
return "start"
|
||||
if normalized_action in {"service.stop", "runtime.stop_worker", "runtime.stop_sync_agent"}:
|
||||
return "stop"
|
||||
return ""
|
||||
|
||||
|
||||
def execute_structured_action(
|
||||
action: str,
|
||||
payload: dict | None,
|
||||
*,
|
||||
service_names: dict[str, str],
|
||||
runner: CommandRunner,
|
||||
host_context: dict | None = None,
|
||||
) -> tuple[bool, str, dict]:
|
||||
normalized_action = str(action or "").strip()
|
||||
normalized_payload = dict(payload or {})
|
||||
normalized_service_names = {
|
||||
str(key or "").strip(): str(value or "").strip()
|
||||
for key, value in dict(service_names or {}).items()
|
||||
if str(key or "").strip() and str(value or "").strip()
|
||||
}
|
||||
host_details = {
|
||||
key: str((host_context or {}).get(key) or "").strip()
|
||||
for key in ("hostname", "ip")
|
||||
if str((host_context or {}).get(key) or "").strip()
|
||||
}
|
||||
|
||||
if normalized_action == "health.snapshot":
|
||||
checks: dict[str, dict] = {}
|
||||
for key, service_name in normalized_service_names.items():
|
||||
code, stdout, stderr = runner(["systemctl", "is-active", service_name], timeout=15)
|
||||
checks[key] = {
|
||||
"service_name": service_name,
|
||||
"returncode": int(code or 0),
|
||||
"state": stdout or stderr,
|
||||
"ok": int(code or 0) == 0 and (stdout or stderr) == "active",
|
||||
}
|
||||
result = {"checks": checks}
|
||||
result.update(host_details)
|
||||
return True, "health snapshot collected", result
|
||||
|
||||
if normalized_action == "service.status":
|
||||
service_name = service_name_from_payload(
|
||||
normalized_payload,
|
||||
default_worker_service_name=normalized_service_names.get("worker") or "",
|
||||
)
|
||||
code, stdout, stderr = runner(["systemctl", "status", service_name, "--no-pager", "-l"], timeout=45)
|
||||
result = {
|
||||
"service_name": service_name,
|
||||
"stdout": trim_output(stdout, 12000),
|
||||
"stderr": trim_output(stderr, 4000),
|
||||
}
|
||||
result.update(host_details)
|
||||
return int(code or 0) == 0, stdout or stderr or f"{service_name} status collected", result
|
||||
|
||||
systemctl_action = systemctl_action_name(normalized_action)
|
||||
if systemctl_action:
|
||||
service_name = service_name_for_action(normalized_action, normalized_payload, normalized_service_names)
|
||||
if not service_name:
|
||||
return False, f"当前动作缺少 service_name: {normalized_action}", {"action": normalized_action}
|
||||
code, stdout, stderr = runner(["systemctl", systemctl_action, service_name], timeout=45)
|
||||
result = {
|
||||
"service_name": service_name,
|
||||
"systemctl_action": systemctl_action,
|
||||
"stdout": trim_output(stdout, 12000),
|
||||
"stderr": trim_output(stderr, 4000),
|
||||
}
|
||||
result.update(host_details)
|
||||
return int(code or 0) == 0, stdout or stderr or f"{service_name} {systemctl_action} completed", result
|
||||
|
||||
if normalized_action == "logs.collect":
|
||||
service_name = service_name_from_payload(
|
||||
normalized_payload,
|
||||
default_worker_service_name=normalized_service_names.get("worker") or "",
|
||||
)
|
||||
lines = max(20, min(int(normalized_payload.get("lines") or 120), 500))
|
||||
code, stdout, stderr = runner(["journalctl", "-u", service_name, "-n", str(lines), "--no-pager"], timeout=45)
|
||||
journal_output = stdout or stderr
|
||||
fallback_output, fallback_paths = _collect_log_file_fallback(
|
||||
service_name,
|
||||
lines=lines,
|
||||
service_names=normalized_service_names,
|
||||
)
|
||||
collection_source = "journal"
|
||||
effective_output = journal_output
|
||||
ok = int(code or 0) == 0
|
||||
if (not ok or not stdout.strip()) and fallback_output:
|
||||
collection_source = "file_fallback"
|
||||
effective_output = fallback_output
|
||||
ok = True
|
||||
result = {
|
||||
"service_name": service_name,
|
||||
"lines": lines,
|
||||
"collection_source": collection_source,
|
||||
"fallback_used": bool(collection_source == "file_fallback"),
|
||||
"fallback_reason": trim_output(journal_output, 4000) if collection_source == "file_fallback" else "",
|
||||
"log_paths": fallback_paths,
|
||||
"journal_returncode": int(code or 0),
|
||||
"stdout": trim_output(effective_output, 20000),
|
||||
"stderr": trim_output(stderr, 4000),
|
||||
}
|
||||
result.update(host_details)
|
||||
return ok, effective_output or f"{service_name} logs collected", result
|
||||
|
||||
if normalized_action == "diagnostics.collect":
|
||||
lines = max(20, min(int(normalized_payload.get("lines") or 200), 800))
|
||||
diagnostics: dict[str, object] = {
|
||||
"services": {},
|
||||
"lines": lines,
|
||||
}
|
||||
diagnostics.update(host_details)
|
||||
for key, service_name in normalized_service_names.items():
|
||||
active_code, active_stdout, active_stderr = runner(["systemctl", "is-active", service_name], timeout=15)
|
||||
status_code, status_stdout, status_stderr = runner(
|
||||
["systemctl", "status", service_name, "--no-pager", "-l"],
|
||||
timeout=45,
|
||||
)
|
||||
log_code, log_stdout, log_stderr = runner(
|
||||
["journalctl", "-u", service_name, "-n", str(lines), "--no-pager"],
|
||||
timeout=45,
|
||||
)
|
||||
log_output = log_stdout or log_stderr
|
||||
fallback_output, fallback_paths = _collect_log_file_fallback(
|
||||
service_name,
|
||||
lines=lines,
|
||||
service_names=normalized_service_names,
|
||||
)
|
||||
log_source = "journal"
|
||||
if (int(log_code or 0) != 0 or not log_stdout.strip()) and fallback_output:
|
||||
log_output = fallback_output
|
||||
log_source = "file_fallback"
|
||||
diagnostics["services"][key] = {
|
||||
"service_name": service_name,
|
||||
"active_returncode": int(active_code or 0),
|
||||
"active_state": active_stdout or active_stderr,
|
||||
"status_returncode": int(status_code or 0),
|
||||
"status_output": trim_output(status_stdout or status_stderr, 12000),
|
||||
"log_returncode": int(log_code or 0),
|
||||
"log_source": log_source,
|
||||
"log_paths": fallback_paths,
|
||||
"log_output": trim_output(log_output, 20000),
|
||||
}
|
||||
return True, "diagnostics collected", diagnostics
|
||||
|
||||
return False, f"unsupported action: {normalized_action}", {"action": normalized_action}
|
||||
2852
domain-api/app/services/ops_agent_service.py
Normal file
2852
domain-api/app/services/ops_agent_service.py
Normal file
File diff suppressed because it is too large
Load Diff
17
domain-api/app/services/ops_command_service.py
Normal file
17
domain-api/app/services/ops_command_service.py
Normal file
@@ -0,0 +1,17 @@
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
OPS_MULTI_REGION_DIR = "domain-api/deploy/multi-region"
|
||||
|
||||
|
||||
def ops_script_path(script_name: str) -> str:
|
||||
normalized_script_name = str(script_name or "").strip().lstrip("/")
|
||||
if not normalized_script_name:
|
||||
return OPS_MULTI_REGION_DIR
|
||||
return f"{OPS_MULTI_REGION_DIR}/{normalized_script_name}"
|
||||
|
||||
|
||||
def build_bash_command(script_name: str, *args: object) -> str:
|
||||
command_parts = ["bash", ops_script_path(script_name)]
|
||||
command_parts.extend(str(arg or "").strip() for arg in args if str(arg or "").strip())
|
||||
return " ".join(command_parts)
|
||||
79
domain-api/app/services/ops_execution_capability_service.py
Normal file
79
domain-api/app/services/ops_execution_capability_service.py
Normal file
@@ -0,0 +1,79 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from app.services.ops_action_executor_core import STRUCTURED_ACTIONS
|
||||
|
||||
|
||||
_LOCAL_RUNTIME_ONLY_ACTIONS = {
|
||||
"runtime.push_sync",
|
||||
"runtime.pull_tasks",
|
||||
}
|
||||
|
||||
_REMOTE_AGENT_ONLY_ACTIONS = {
|
||||
"delivery.queue.flush",
|
||||
"delivery.queue.replay",
|
||||
"delivery.queue.discard",
|
||||
}
|
||||
|
||||
_LOCAL_RUNTIME_ACTIONS = set(STRUCTURED_ACTIONS) | _LOCAL_RUNTIME_ONLY_ACTIONS | {"deploy.release"}
|
||||
_SSH_ACTIONS = set(STRUCTURED_ACTIONS) | {"deploy.release"}
|
||||
_REMOTE_AGENT_ACTIONS = set(STRUCTURED_ACTIONS) | _REMOTE_AGENT_ONLY_ACTIONS | {"deploy.release"}
|
||||
_CONTROL_PLANE_ACTIONS = {
|
||||
"node.bootstrap",
|
||||
}
|
||||
|
||||
|
||||
def supports_local_runtime_action(action: str) -> bool:
|
||||
return str(action or "").strip() in _LOCAL_RUNTIME_ACTIONS
|
||||
|
||||
|
||||
def supports_ssh_action(action: str) -> bool:
|
||||
return str(action or "").strip() in _SSH_ACTIONS
|
||||
|
||||
|
||||
def supports_remote_agent_action(action: str) -> bool:
|
||||
return str(action or "").strip() in _REMOTE_AGENT_ACTIONS
|
||||
|
||||
|
||||
def supports_control_plane_action(action: str) -> bool:
|
||||
return str(action or "").strip() in _CONTROL_PLANE_ACTIONS
|
||||
|
||||
|
||||
def validate_action_execution_mode(
|
||||
action: str,
|
||||
execution_mode: str,
|
||||
*,
|
||||
target_node_code: str = "",
|
||||
current_node_code: str = "",
|
||||
) -> tuple[bool, str]:
|
||||
normalized_action = str(action or "").strip()
|
||||
normalized_mode = str(execution_mode or "remote-agent").strip() or "remote-agent"
|
||||
normalized_target_node_code = str(target_node_code or "").strip()
|
||||
normalized_current_node_code = str(current_node_code or "").strip()
|
||||
|
||||
if normalized_mode == "local-runtime":
|
||||
if not supports_local_runtime_action(normalized_action):
|
||||
return False, f"{normalized_action} 当前不支持 local-runtime 执行方式。"
|
||||
if (
|
||||
normalized_target_node_code
|
||||
and normalized_current_node_code
|
||||
and normalized_target_node_code != normalized_current_node_code
|
||||
):
|
||||
return False, "local-runtime 仅支持当前控制面本机节点。"
|
||||
return True, ""
|
||||
|
||||
if normalized_mode == "control-plane":
|
||||
if not supports_control_plane_action(normalized_action):
|
||||
return False, f"{normalized_action} 当前不支持 control-plane 执行方式。"
|
||||
return True, ""
|
||||
|
||||
if normalized_mode == "ssh":
|
||||
if not supports_ssh_action(normalized_action):
|
||||
return False, f"{normalized_action} 当前不支持 ssh 执行方式。"
|
||||
return True, ""
|
||||
|
||||
if normalized_mode == "remote-agent":
|
||||
if not supports_remote_agent_action(normalized_action):
|
||||
return False, f"{normalized_action} 当前不支持 remote-agent 执行方式。"
|
||||
return True, ""
|
||||
|
||||
return False, f"未知执行方式: {normalized_mode}"
|
||||
60
domain-api/app/services/ops_execution_mode_service.py
Normal file
60
domain-api/app/services/ops_execution_mode_service.py
Normal file
@@ -0,0 +1,60 @@
|
||||
from __future__ import annotations
|
||||
|
||||
|
||||
def normalize_execution_modes(raw_modes: object, fallback_mode: str = "remote-agent") -> list[str]:
|
||||
normalized_modes: list[str] = []
|
||||
seen: set[str] = set()
|
||||
fallback = str(fallback_mode or "remote-agent").strip() or "remote-agent"
|
||||
for item in list(raw_modes or []):
|
||||
mode = str(item or "").strip()
|
||||
if not mode or mode in seen:
|
||||
continue
|
||||
seen.add(mode)
|
||||
normalized_modes.append(mode)
|
||||
if fallback and fallback not in seen:
|
||||
normalized_modes.insert(0, fallback)
|
||||
return normalized_modes or [fallback]
|
||||
|
||||
|
||||
def execution_mode_label(execution_mode: str) -> str:
|
||||
normalized_mode = str(execution_mode or "remote-agent").strip() or "remote-agent"
|
||||
if normalized_mode == "ssh":
|
||||
return "SSH"
|
||||
if normalized_mode == "control-plane":
|
||||
return "控制面"
|
||||
if normalized_mode == "local-runtime":
|
||||
return "本机运行时"
|
||||
if normalized_mode == "remote-agent":
|
||||
return "远端 Agent"
|
||||
return normalized_mode
|
||||
|
||||
|
||||
def build_execution_mode_options(raw_modes: object, fallback_mode: str = "remote-agent") -> list[dict]:
|
||||
return [
|
||||
{
|
||||
"value": mode,
|
||||
"label": execution_mode_label(mode),
|
||||
}
|
||||
for mode in normalize_execution_modes(raw_modes, fallback_mode)
|
||||
]
|
||||
|
||||
|
||||
def decorate_execution_mode_fields(
|
||||
payload: dict | None = None,
|
||||
*,
|
||||
default_key: str = "default_execution_mode",
|
||||
modes_key: str = "execution_modes",
|
||||
) -> dict:
|
||||
normalized_payload = dict(payload or {})
|
||||
default_mode = str(normalized_payload.get(default_key) or "remote-agent").strip() or "remote-agent"
|
||||
execution_mode_options = build_execution_mode_options(normalized_payload.get(modes_key) or [], default_mode)
|
||||
normalized_payload[default_key] = default_mode
|
||||
normalized_payload[modes_key] = [str(item.get("value") or "").strip() for item in execution_mode_options if str(item.get("value") or "").strip()]
|
||||
normalized_payload["default_execution_mode_label"] = execution_mode_label(default_mode)
|
||||
normalized_payload["execution_mode_options"] = execution_mode_options
|
||||
normalized_payload["execution_mode_labels"] = [
|
||||
str(item.get("label") or "").strip()
|
||||
for item in execution_mode_options
|
||||
if str(item.get("label") or "").strip()
|
||||
]
|
||||
return normalized_payload
|
||||
1682
domain-api/app/services/ops_job_service.py
Normal file
1682
domain-api/app/services/ops_job_service.py
Normal file
File diff suppressed because it is too large
Load Diff
1263
domain-api/app/services/ops_playbook_service.py
Normal file
1263
domain-api/app/services/ops_playbook_service.py
Normal file
File diff suppressed because it is too large
Load Diff
494
domain-api/app/services/ops_policy_service.py
Normal file
494
domain-api/app/services/ops_policy_service.py
Normal file
@@ -0,0 +1,494 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.cluster_runtime_service import get_cluster_snapshot
|
||||
from app.services.ops_execution_capability_service import validate_action_execution_mode
|
||||
from app.services.ops_release_service import (
|
||||
_calculate_total_batches,
|
||||
_normalize_rollout_policy,
|
||||
_resolve_rollout_targets,
|
||||
build_release_rollout_gate,
|
||||
get_release,
|
||||
)
|
||||
|
||||
|
||||
_LOW_RISK_ACTIONS = {
|
||||
"health.snapshot",
|
||||
"runtime.push_sync",
|
||||
"runtime.pull_tasks",
|
||||
"logs.collect",
|
||||
"diagnostics.collect",
|
||||
"delivery.queue.flush",
|
||||
}
|
||||
|
||||
_MEDIUM_RISK_ACTIONS = {
|
||||
"runtime.start_worker",
|
||||
"runtime.start_sync_agent",
|
||||
"service.status",
|
||||
"health.check",
|
||||
"delivery.queue.replay",
|
||||
}
|
||||
|
||||
_HIGH_RISK_ACTIONS = {
|
||||
"runtime.stop_worker",
|
||||
"runtime.stop_sync_agent",
|
||||
"runtime.restart_api",
|
||||
"service.restart",
|
||||
"deploy.release",
|
||||
"delivery.queue.discard",
|
||||
}
|
||||
|
||||
_CRITICAL_RISK_ACTIONS = {
|
||||
"deploy.rollback",
|
||||
"node.bootstrap",
|
||||
"cluster.reconfigure",
|
||||
}
|
||||
|
||||
|
||||
def _normalize_node_code_list(raw_value: object) -> list[str]:
|
||||
if isinstance(raw_value, list):
|
||||
return [str(item).strip() for item in raw_value if str(item).strip()]
|
||||
text = str(raw_value or "").replace("\r", "\n").strip()
|
||||
if not text:
|
||||
return []
|
||||
normalized = text.replace(",", "\n")
|
||||
return [item.strip() for item in normalized.split("\n") if item.strip()]
|
||||
|
||||
|
||||
def _dedupe_text_items(items: list[str]) -> list[str]:
|
||||
return list(dict.fromkeys(str(item or "").strip() for item in items if str(item or "").strip()))
|
||||
|
||||
|
||||
def _risk_level_for_action(action: str) -> str:
|
||||
if action in _LOW_RISK_ACTIONS:
|
||||
return "low"
|
||||
if action in _HIGH_RISK_ACTIONS:
|
||||
return "high"
|
||||
if action in _CRITICAL_RISK_ACTIONS:
|
||||
return "critical"
|
||||
return "medium"
|
||||
|
||||
|
||||
def _compact_target_node(item: dict) -> dict:
|
||||
return {
|
||||
"node_code": str(item.get("node_code") or ""),
|
||||
"region": str(item.get("region") or ""),
|
||||
"role": str(item.get("role") or ""),
|
||||
"status": str(item.get("status") or ""),
|
||||
"current_load": int(item.get("current_load", 0) or 0),
|
||||
"is_effective_worker": bool(item.get("is_effective_worker", False)),
|
||||
"detect_participating": bool(item.get("detect_participating", False)),
|
||||
"is_enabled": bool(item.get("is_enabled", True)),
|
||||
"deploy_channel": str(item.get("deploy_channel") or ""),
|
||||
}
|
||||
|
||||
|
||||
def _preview_action_payload_guardrails(
|
||||
action: str,
|
||||
payload: dict,
|
||||
*,
|
||||
execution_mode: str,
|
||||
target_nodes_total: int,
|
||||
) -> dict:
|
||||
warnings: list[str] = []
|
||||
blocking_reasons: list[str] = []
|
||||
approval_reasons: list[str] = []
|
||||
recommendations: list[str] = []
|
||||
|
||||
if action == "deploy.release":
|
||||
release_version = str(payload.get("release_version") or "").strip()
|
||||
artifact_url = str(payload.get("artifact_url") or "").strip()
|
||||
if not release_version:
|
||||
blocking_reasons.append("deploy.release 缺少 release_version,当前不能创建正式发布任务。")
|
||||
if not artifact_url:
|
||||
blocking_reasons.append("deploy.release 缺少 artifact_url,当前不能创建正式发布任务。")
|
||||
if execution_mode not in {"remote-agent", "ssh"}:
|
||||
blocking_reasons.append("deploy.release 仅支持 remote-agent 或 ssh 执行方式。")
|
||||
if target_nodes_total > 1:
|
||||
recommendations.append("建议先单节点 canary 验证发布任务,再逐步扩到更多节点。")
|
||||
|
||||
if action == "node.bootstrap":
|
||||
if execution_mode != "control-plane":
|
||||
blocking_reasons.append("node.bootstrap 仅支持 control-plane 执行方式。")
|
||||
if target_nodes_total > 1:
|
||||
recommendations.append("接管动作建议按单节点节奏推进,先确认首台节点接入成功后再继续放量。")
|
||||
|
||||
if action in _CRITICAL_RISK_ACTIONS:
|
||||
approval_reasons.append("该动作属于 critical 风险动作,正式环境必须审批。")
|
||||
elif action in _HIGH_RISK_ACTIONS:
|
||||
approval_reasons.append("该动作属于 high 风险动作,建议审批后执行。")
|
||||
|
||||
return {
|
||||
"warnings": _dedupe_text_items(warnings),
|
||||
"blocking_reasons": _dedupe_text_items(blocking_reasons),
|
||||
"approval_reasons": _dedupe_text_items(approval_reasons),
|
||||
"recommendations": _dedupe_text_items(recommendations),
|
||||
}
|
||||
|
||||
|
||||
def _preview_single_node_policy(
|
||||
action: str,
|
||||
*,
|
||||
target_node_code: str,
|
||||
cluster: dict,
|
||||
payload: dict | None = None,
|
||||
execution_mode: str = "remote-agent",
|
||||
target_type: str = "node",
|
||||
include_payload_guardrails: bool = True,
|
||||
) -> dict:
|
||||
payload = dict(payload or {})
|
||||
nodes = list(cluster.get("nodes") or [])
|
||||
summary = cluster.get("summary") or {}
|
||||
target_node = next((item for item in nodes if str(item.get("node_code") or "").strip() == target_node_code), {})
|
||||
|
||||
risk_level = _risk_level_for_action(action)
|
||||
warnings: list[str] = []
|
||||
blocking_reasons: list[str] = []
|
||||
approval_reasons: list[str] = []
|
||||
recommendations: list[str] = []
|
||||
|
||||
execution_supported, execution_reason = validate_action_execution_mode(
|
||||
action,
|
||||
execution_mode,
|
||||
target_node_code=target_node_code,
|
||||
current_node_code=settings.node_code,
|
||||
)
|
||||
if not execution_supported and execution_reason:
|
||||
blocking_reasons.append(execution_reason)
|
||||
|
||||
if target_type == "node" and target_node_code and not target_node:
|
||||
warnings.append("目标节点当前未出现在集群快照中,可能尚未注册或已离线。")
|
||||
|
||||
node_status = str(target_node.get("status") or "")
|
||||
node_role = str(target_node.get("role") or "")
|
||||
node_current_load = int(target_node.get("current_load", 0) or 0)
|
||||
node_effective_worker = bool(target_node.get("is_effective_worker", False))
|
||||
node_detect_participating = bool(target_node.get("detect_participating", False))
|
||||
|
||||
if target_node:
|
||||
if node_status in {"busy"} and action in {"runtime.stop_worker", "runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
|
||||
blocking_reasons.append("目标节点当前处于 busy 状态,不适合直接执行中断类动作。")
|
||||
|
||||
if node_detect_participating and action in {"runtime.stop_worker", "runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
|
||||
blocking_reasons.append("目标节点正在参与检测,需先迁移负载或人工确认后再执行。")
|
||||
|
||||
if node_role == "control" and action in {"runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
|
||||
approval_reasons.append("目标节点是 control 节点,建议强制走审批或维护窗口。")
|
||||
|
||||
if node_effective_worker and int(summary.get("online_worker_nodes", 0) or 0) <= 1 and action in {"runtime.stop_worker", "deploy.release", "deploy.rollback", "service.restart"}:
|
||||
blocking_reasons.append("当前有效执行节点数不足 2,执行该动作可能导致检测能力中断。")
|
||||
|
||||
if node_current_load > 0:
|
||||
warnings.append(f"目标节点当前负载为 {node_current_load},建议优先观察或转移任务。")
|
||||
|
||||
if include_payload_guardrails:
|
||||
payload_guardrails = _preview_action_payload_guardrails(
|
||||
action,
|
||||
payload,
|
||||
execution_mode=execution_mode,
|
||||
target_nodes_total=1,
|
||||
)
|
||||
warnings.extend(payload_guardrails.get("warnings") or [])
|
||||
blocking_reasons.extend(payload_guardrails.get("blocking_reasons") or [])
|
||||
approval_reasons.extend(payload_guardrails.get("approval_reasons") or [])
|
||||
recommendations.extend(payload_guardrails.get("recommendations") or [])
|
||||
|
||||
warnings = _dedupe_text_items(warnings)
|
||||
blocking_reasons = _dedupe_text_items(blocking_reasons)
|
||||
approval_reasons = _dedupe_text_items(approval_reasons)
|
||||
recommendations = _dedupe_text_items(recommendations)
|
||||
|
||||
approval_required = bool(approval_reasons)
|
||||
blocked = bool(blocking_reasons)
|
||||
|
||||
return {
|
||||
"action": action,
|
||||
"target_type": target_type,
|
||||
"target_node_code": target_node_code,
|
||||
"execution_mode": execution_mode,
|
||||
"risk_level": risk_level,
|
||||
"approval_required": approval_required,
|
||||
"blocked": blocked,
|
||||
"blocking_reasons": blocking_reasons,
|
||||
"approval_reasons": approval_reasons,
|
||||
"warnings": warnings,
|
||||
"recommendations": recommendations,
|
||||
"target_node": {
|
||||
"node_code": str(target_node.get("node_code") or ""),
|
||||
"region": str(target_node.get("region") or ""),
|
||||
"role": node_role,
|
||||
"status": node_status,
|
||||
"current_load": node_current_load,
|
||||
"is_effective_worker": node_effective_worker,
|
||||
"detect_participating": node_detect_participating,
|
||||
} if target_node else {},
|
||||
"cluster_guardrails": {
|
||||
"online_worker_nodes": int(summary.get("online_worker_nodes", 0) or 0),
|
||||
"dedicated_online_worker_nodes": int(summary.get("dedicated_online_worker_nodes", 0) or 0),
|
||||
"online_control_nodes": int(summary.get("online_control_nodes", 0) or 0),
|
||||
"busy_nodes": list(summary.get("busy_nodes") or []),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _preview_node_batch_policy(action: str, payload: dict, cluster: dict) -> dict:
|
||||
target_node_codes = _normalize_node_code_list(payload.get("target_node_codes") or payload.get("node_codes"))
|
||||
execution_mode = str(payload.get("execution_mode") or "remote-agent").strip() or "remote-agent"
|
||||
action_payload = dict(payload.get("payload") or {})
|
||||
summary = cluster.get("summary") or {}
|
||||
|
||||
warnings: list[str] = []
|
||||
blocking_reasons: list[str] = []
|
||||
approval_reasons: list[str] = []
|
||||
recommendations: list[str] = []
|
||||
|
||||
if not target_node_codes:
|
||||
blocking_reasons.append("当前没有可预检的目标节点。")
|
||||
|
||||
shared_guardrails = _preview_action_payload_guardrails(
|
||||
action,
|
||||
action_payload,
|
||||
execution_mode=execution_mode,
|
||||
target_nodes_total=len(target_node_codes),
|
||||
)
|
||||
warnings.extend(shared_guardrails.get("warnings") or [])
|
||||
blocking_reasons.extend(shared_guardrails.get("blocking_reasons") or [])
|
||||
approval_reasons.extend(shared_guardrails.get("approval_reasons") or [])
|
||||
recommendations.extend(shared_guardrails.get("recommendations") or [])
|
||||
|
||||
node_previews = [
|
||||
_preview_single_node_policy(
|
||||
action,
|
||||
target_node_code=node_code,
|
||||
cluster=cluster,
|
||||
payload=action_payload,
|
||||
execution_mode=execution_mode,
|
||||
target_type="batch",
|
||||
include_payload_guardrails=False,
|
||||
)
|
||||
for node_code in target_node_codes
|
||||
]
|
||||
|
||||
blocked_nodes = [item for item in node_previews if bool(item.get("blocked", False))]
|
||||
approval_nodes = [item for item in node_previews if bool(item.get("approval_required", False))]
|
||||
warning_nodes = [item for item in node_previews if list(item.get("warnings") or [])]
|
||||
missing_nodes = [item for item in node_previews if not (item.get("target_node") or {}).get("node_code")]
|
||||
target_control_nodes = [item for item in node_previews if str((item.get("target_node") or {}).get("role") or "") == "control"]
|
||||
|
||||
if blocked_nodes:
|
||||
blocking_reasons.append(f"共有 {len(blocked_nodes)} 个目标节点存在节点级阻断,需先处理后再批量执行。")
|
||||
if approval_nodes:
|
||||
approval_reasons.append(f"共有 {len(approval_nodes)} 个目标节点建议先审批。")
|
||||
if warning_nodes:
|
||||
warnings.append(f"共有 {len(warning_nodes)} 个目标节点存在需关注事项。")
|
||||
if missing_nodes:
|
||||
warnings.append(f"共有 {len(missing_nodes)} 个目标节点未出现在当前集群快照中。")
|
||||
if target_control_nodes and len(target_node_codes) > 1:
|
||||
recommendations.append("建议把 control 节点放到批次尾部,优先在 worker 或空闲节点验证。")
|
||||
|
||||
warnings = _dedupe_text_items(warnings)
|
||||
blocking_reasons = _dedupe_text_items(blocking_reasons)
|
||||
approval_reasons = _dedupe_text_items(approval_reasons)
|
||||
recommendations = _dedupe_text_items(recommendations)
|
||||
|
||||
return {
|
||||
"action": action,
|
||||
"target_type": "batch",
|
||||
"execution_mode": execution_mode,
|
||||
"risk_level": _risk_level_for_action(action),
|
||||
"approval_required": bool(approval_reasons),
|
||||
"blocked": bool(blocking_reasons),
|
||||
"blocking_reasons": blocking_reasons,
|
||||
"approval_reasons": approval_reasons,
|
||||
"warnings": warnings,
|
||||
"recommendations": recommendations,
|
||||
"target_summary": {
|
||||
"nodes_total": len(target_node_codes),
|
||||
"blocked_nodes": len(blocked_nodes),
|
||||
"approval_nodes": len(approval_nodes),
|
||||
"warning_nodes": len(warning_nodes),
|
||||
"missing_nodes": len(missing_nodes),
|
||||
},
|
||||
"target_nodes": [
|
||||
{
|
||||
**(item.get("target_node") or {}),
|
||||
"node_code": str((item.get("target_node") or {}).get("node_code") or item.get("target_node_code") or ""),
|
||||
"risk_level": str(item.get("risk_level") or ""),
|
||||
"approval_required": bool(item.get("approval_required", False)),
|
||||
"blocked": bool(item.get("blocked", False)),
|
||||
"blocking_reasons": list(item.get("blocking_reasons") or []),
|
||||
"approval_reasons": list(item.get("approval_reasons") or []),
|
||||
"warnings": list(item.get("warnings") or []),
|
||||
}
|
||||
for item in node_previews
|
||||
],
|
||||
"cluster_guardrails": {
|
||||
"online_worker_nodes": int(summary.get("online_worker_nodes", 0) or 0),
|
||||
"dedicated_online_worker_nodes": int(summary.get("dedicated_online_worker_nodes", 0) or 0),
|
||||
"online_control_nodes": int(summary.get("online_control_nodes", 0) or 0),
|
||||
"busy_nodes": list(summary.get("busy_nodes") or []),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _preview_rollout_policy(action: str, payload: dict, cluster: dict) -> dict:
|
||||
selector = payload.get("target_selector") or {}
|
||||
input_policy = payload.get("policy") or {}
|
||||
release_id = int(payload.get("release_id") or 0)
|
||||
input_release = dict(payload.get("release") or {})
|
||||
release = input_release or (get_release(release_id) if release_id > 0 else {})
|
||||
target_nodes = _resolve_rollout_targets(selector)
|
||||
normalized_policy = _normalize_rollout_policy(input_policy, total_targets=len(target_nodes))
|
||||
execution_mode = str(normalized_policy.get("execution_mode") or "remote-agent").strip() or "remote-agent"
|
||||
summary = cluster.get("summary") or {}
|
||||
release_gate = build_release_rollout_gate(
|
||||
release,
|
||||
target_nodes,
|
||||
execution_mode=execution_mode,
|
||||
)
|
||||
operational_readiness = dict(release_gate.get("operational_readiness") or {})
|
||||
|
||||
risk_level = _risk_level_for_action(action)
|
||||
warnings: list[str] = []
|
||||
blocking_reasons: list[str] = []
|
||||
approval_reasons: list[str] = []
|
||||
recommendations: list[str] = []
|
||||
|
||||
target_count = len(target_nodes)
|
||||
first_batch_size = int(normalized_policy.get("first_batch_size") or 0)
|
||||
batch_size = int(normalized_policy.get("batch_size") or 0)
|
||||
batches_total = _calculate_total_batches(target_count, normalized_policy)
|
||||
first_batch_targets = target_nodes[:first_batch_size] if first_batch_size > 0 else []
|
||||
|
||||
online_worker_nodes = int(summary.get("online_worker_nodes", 0) or 0)
|
||||
dedicated_online_worker_nodes = int(summary.get("dedicated_online_worker_nodes", 0) or 0)
|
||||
online_control_nodes = int(summary.get("online_control_nodes", 0) or 0)
|
||||
|
||||
target_control_nodes = [item for item in target_nodes if str(item.get("role") or "") == "control"]
|
||||
target_worker_nodes = [item for item in target_nodes if str(item.get("role") or "") == "worker"]
|
||||
target_effective_workers = [item for item in target_nodes if bool(item.get("is_effective_worker", False))]
|
||||
first_batch_effective_workers = [item for item in first_batch_targets if bool(item.get("is_effective_worker", False))]
|
||||
first_batch_control_nodes = [item for item in first_batch_targets if str(item.get("role") or "") == "control"]
|
||||
|
||||
busy_targets = [item for item in target_nodes if int(item.get("current_load", 0) or 0) > 0]
|
||||
participating_targets = [item for item in target_nodes if bool(item.get("detect_participating", False))]
|
||||
offline_targets = [item for item in target_nodes if str(item.get("status") or "") not in {"online", "busy"}]
|
||||
|
||||
if target_count <= 0:
|
||||
blocking_reasons.append("当前筛选条件未命中任何可 rollout 的目标节点。")
|
||||
if release_id > 0 and not input_release and not release:
|
||||
blocking_reasons.append("指定的 Release 不存在,当前无法预检该 rollout。")
|
||||
gate_status = str(release_gate.get("status") or "").strip()
|
||||
if gate_status in {"release_not_ready", "artifact_missing"}:
|
||||
warnings.append(str(release_gate.get("summary") or "").strip())
|
||||
|
||||
if offline_targets:
|
||||
warnings.append(f"命中了 {len(offline_targets)} 个非在线节点,若保留 only_online=false,发布时可能出现排队或失败。")
|
||||
|
||||
if busy_targets:
|
||||
warnings.append(f"命中了 {len(busy_targets)} 个当前有负载的节点,建议优先确认任务是否可中断。")
|
||||
|
||||
if participating_targets:
|
||||
warnings.append(f"命中了 {len(participating_targets)} 个正在参与检测的节点,建议优先滚动到空闲节点。")
|
||||
|
||||
if target_control_nodes:
|
||||
approval_reasons.append("本次 rollout 覆盖 control 节点,建议强制审批,并把 control 放在最后一批。")
|
||||
|
||||
if first_batch_effective_workers and online_worker_nodes > 0 and len(first_batch_effective_workers) >= online_worker_nodes:
|
||||
blocking_reasons.append("第一批会覆盖全部有效执行节点,检测能力可能瞬时归零。请缩小 first_batch_size。")
|
||||
|
||||
if first_batch_control_nodes and online_control_nodes > 0 and len(first_batch_control_nodes) >= online_control_nodes:
|
||||
blocking_reasons.append("第一批会覆盖全部在线控制面,运维中枢本身可能不可用。请调整节点顺序或缩小 first_batch_size。")
|
||||
|
||||
if first_batch_size > 1 and target_control_nodes:
|
||||
warnings.append("当前第一批大于 1,且目标中包含 control 节点;默认建议先 worker、后 control。")
|
||||
|
||||
if action in _CRITICAL_RISK_ACTIONS:
|
||||
approval_reasons.append("该动作属于 critical 风险动作,正式环境必须审批。")
|
||||
elif action in _HIGH_RISK_ACTIONS:
|
||||
approval_reasons.append("该动作属于 high 风险动作,建议审批后执行。")
|
||||
|
||||
blocking_reasons.extend(list(release_gate.get("blocking_reasons") or []))
|
||||
warnings.extend(list(release_gate.get("warning_reasons") or []))
|
||||
recommendations.extend(list(release_gate.get("recommendations") or []))
|
||||
|
||||
readiness_summary = dict(operational_readiness.get("summary") or {})
|
||||
if target_control_nodes and int(readiness_summary.get("inspection_healthy_nodes", 0) or 0) < target_count:
|
||||
approval_reasons.append("本次目标中存在巡检未完全通过的节点,建议先完成标准巡检并经人工确认后再发布。")
|
||||
|
||||
if target_count > 1 and first_batch_size > 1:
|
||||
recommendations.append("正式环境建议 first_batch_size=1,从单节点 canary 开始。")
|
||||
if target_control_nodes:
|
||||
recommendations.append("建议把 control 节点放在 rollout 末尾,并确保 worker 已先通过健康检查。")
|
||||
if busy_targets or participating_targets:
|
||||
recommendations.append("建议先等参与检测的节点降为空闲,再推进该批次。")
|
||||
|
||||
warnings = list(dict.fromkeys(item for item in warnings if str(item or "").strip()))
|
||||
blocking_reasons = list(dict.fromkeys(item for item in blocking_reasons if str(item or "").strip()))
|
||||
approval_reasons = list(dict.fromkeys(item for item in approval_reasons if str(item or "").strip()))
|
||||
recommendations = list(dict.fromkeys(item for item in recommendations if str(item or "").strip()))
|
||||
|
||||
approval_required = bool(approval_reasons)
|
||||
blocked = bool(blocking_reasons)
|
||||
|
||||
return {
|
||||
"action": action,
|
||||
"target_type": "rollout",
|
||||
"risk_level": risk_level,
|
||||
"approval_required": approval_required,
|
||||
"blocked": blocked,
|
||||
"blocking_reasons": blocking_reasons,
|
||||
"approval_reasons": approval_reasons,
|
||||
"warnings": warnings,
|
||||
"recommendations": recommendations,
|
||||
"target_summary": {
|
||||
"nodes_total": target_count,
|
||||
"worker_nodes": len(target_worker_nodes),
|
||||
"control_nodes": len(target_control_nodes),
|
||||
"effective_worker_nodes": len(target_effective_workers),
|
||||
"busy_nodes": len(busy_targets),
|
||||
"participating_nodes": len(participating_targets),
|
||||
"offline_nodes": len(offline_targets),
|
||||
},
|
||||
"batch_plan": {
|
||||
"first_batch_size": first_batch_size,
|
||||
"batch_size": batch_size,
|
||||
"batches_total": batches_total,
|
||||
"first_batch_nodes": [_compact_target_node(item) for item in first_batch_targets],
|
||||
"remaining_after_first_batch": max(target_count - len(first_batch_targets), 0),
|
||||
},
|
||||
"cluster_guardrails": {
|
||||
"online_worker_nodes": online_worker_nodes,
|
||||
"dedicated_online_worker_nodes": dedicated_online_worker_nodes,
|
||||
"online_control_nodes": online_control_nodes,
|
||||
"busy_nodes": list(summary.get("busy_nodes") or []),
|
||||
},
|
||||
"release": release,
|
||||
"release_gate": release_gate,
|
||||
"target_nodes": [_compact_target_node(item) for item in target_nodes],
|
||||
"operational_readiness": operational_readiness,
|
||||
}
|
||||
|
||||
|
||||
def preview_ops_job_policy(payload: dict) -> dict:
|
||||
action = str(payload.get("action") or "").strip()
|
||||
target_type = str(payload.get("target_type") or "node").strip() or "node"
|
||||
target_node_code = str(payload.get("target_node_code") or "").strip()
|
||||
execution_mode = str(payload.get("execution_mode") or "remote-agent").strip() or "remote-agent"
|
||||
action_payload = dict(payload.get("payload") or {})
|
||||
|
||||
cluster = get_cluster_snapshot()
|
||||
if target_type == "rollout":
|
||||
return _preview_rollout_policy(action, payload, cluster)
|
||||
if target_type in {"batch", "nodes"} or list(payload.get("target_node_codes") or []):
|
||||
return _preview_node_batch_policy(action, payload, cluster)
|
||||
|
||||
return _preview_single_node_policy(
|
||||
action,
|
||||
target_node_code=target_node_code,
|
||||
cluster=cluster,
|
||||
payload=action_payload,
|
||||
execution_mode=execution_mode,
|
||||
target_type=target_type,
|
||||
include_payload_guardrails=True,
|
||||
)
|
||||
475
domain-api/app/services/ops_release_executor_core.py
Normal file
475
domain-api/app/services/ops_release_executor_core.py
Normal file
@@ -0,0 +1,475 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import inspect
|
||||
import json
|
||||
import shutil
|
||||
import tarfile
|
||||
import textwrap
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def normalize_text_list(raw_value: object) -> list[str]:
|
||||
if isinstance(raw_value, list):
|
||||
return [str(item).strip() for item in raw_value if str(item).strip()]
|
||||
text = str(raw_value or "").replace("\r", "\n").strip()
|
||||
if not text:
|
||||
return []
|
||||
normalized = text.replace(",", "\n")
|
||||
return [item.strip() for item in normalized.split("\n") if item.strip()]
|
||||
|
||||
|
||||
def coerce_bool(raw_value: object, default: bool = False) -> bool:
|
||||
if raw_value is None:
|
||||
return bool(default)
|
||||
if isinstance(raw_value, bool):
|
||||
return raw_value
|
||||
if isinstance(raw_value, (int, float)):
|
||||
return bool(raw_value)
|
||||
normalized = str(raw_value or "").strip().lower()
|
||||
if normalized in {"1", "true", "yes", "y", "on", "enable", "enabled"}:
|
||||
return True
|
||||
if normalized in {"0", "false", "no", "n", "off", "disable", "disabled"}:
|
||||
return False
|
||||
return bool(default) if normalized == "" else bool(normalized)
|
||||
|
||||
|
||||
def normalize_release_health_check_services(
|
||||
payload: dict,
|
||||
restart_services: list[str],
|
||||
*,
|
||||
default_api_service_name: str,
|
||||
) -> list[str]:
|
||||
raw_health_check_services = payload.get("health_check_services")
|
||||
if raw_health_check_services is None:
|
||||
health_check_service_source = restart_services or [default_api_service_name]
|
||||
else:
|
||||
health_check_service_source = raw_health_check_services
|
||||
return normalize_text_list(health_check_service_source)
|
||||
|
||||
|
||||
def collect_service_state(run_command, service_name: str) -> dict:
|
||||
code, stdout, stderr = run_command(["systemctl", "is-active", service_name], timeout=15)
|
||||
state = stdout or stderr
|
||||
return {
|
||||
"service_name": service_name,
|
||||
"returncode": int(code or 0),
|
||||
"state": state,
|
||||
"ok": int(code or 0) == 0 and state == "active",
|
||||
}
|
||||
|
||||
|
||||
def check_health_url(url: str, timeout: int, *, user_agent: str, urlopen_func) -> dict:
|
||||
request = urllib.request.Request(
|
||||
url=str(url).strip(),
|
||||
headers={"User-Agent": str(user_agent or "domaincheck-ops/0.1").strip()},
|
||||
method="GET",
|
||||
)
|
||||
try:
|
||||
with urlopen_func(request, timeout=timeout) as response:
|
||||
body = response.read(4000).decode("utf-8", errors="ignore")
|
||||
status_code = int(getattr(response, "status", 0) or response.getcode() or 0)
|
||||
return {
|
||||
"url": str(url).strip(),
|
||||
"ok": 200 <= status_code < 400,
|
||||
"http_status": status_code,
|
||||
"body_preview": body[-1000:],
|
||||
}
|
||||
except urllib.error.HTTPError as exc:
|
||||
body = exc.read().decode("utf-8", errors="ignore") if exc.fp else ""
|
||||
return {
|
||||
"url": str(url).strip(),
|
||||
"ok": False,
|
||||
"http_status": int(exc.code or 0),
|
||||
"body_preview": body[-1000:],
|
||||
"error": str(exc),
|
||||
}
|
||||
except Exception as exc:
|
||||
return {
|
||||
"url": str(url).strip(),
|
||||
"ok": False,
|
||||
"http_status": 0,
|
||||
"body_preview": "",
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
|
||||
def run_release_health_checks(
|
||||
*,
|
||||
urls: list[str],
|
||||
services: list[str],
|
||||
timeout: int,
|
||||
retries: int,
|
||||
interval_seconds: int,
|
||||
run_command,
|
||||
user_agent: str,
|
||||
urlopen_func,
|
||||
) -> tuple[bool, dict]:
|
||||
normalized_urls = [str(item).strip() for item in urls if str(item).strip()]
|
||||
normalized_services = [str(item).strip() for item in services if str(item).strip()]
|
||||
attempts: list[dict] = []
|
||||
total_attempts = max(1, retries + 1)
|
||||
|
||||
for attempt_index in range(1, total_attempts + 1):
|
||||
service_results = [collect_service_state(run_command, service_name) for service_name in normalized_services]
|
||||
url_results = [
|
||||
check_health_url(url, timeout=timeout, user_agent=user_agent, urlopen_func=urlopen_func)
|
||||
for url in normalized_urls
|
||||
]
|
||||
services_ok = all(bool(item.get("ok", False)) for item in service_results) if service_results else True
|
||||
urls_ok = all(bool(item.get("ok", False)) for item in url_results) if url_results else True
|
||||
attempt_payload = {
|
||||
"attempt": attempt_index,
|
||||
"services": service_results,
|
||||
"urls": url_results,
|
||||
"ok": services_ok and urls_ok,
|
||||
}
|
||||
attempts.append(attempt_payload)
|
||||
if attempt_payload["ok"]:
|
||||
return True, {
|
||||
"ok": True,
|
||||
"attempts": attempts,
|
||||
"services_checked": normalized_services,
|
||||
"urls_checked": normalized_urls,
|
||||
}
|
||||
if attempt_index < total_attempts:
|
||||
time.sleep(max(0, interval_seconds))
|
||||
|
||||
return False, {
|
||||
"ok": False,
|
||||
"attempts": attempts,
|
||||
"services_checked": normalized_services,
|
||||
"urls_checked": normalized_urls,
|
||||
}
|
||||
|
||||
|
||||
def safe_extract_tar(archive: tarfile.TarFile, target_dir: Path) -> None:
|
||||
target_dir_resolved = target_dir.resolve()
|
||||
members = archive.getmembers()
|
||||
for member in members:
|
||||
member_path = (target_dir / member.name).resolve()
|
||||
if not str(member_path).startswith(str(target_dir_resolved)):
|
||||
raise RuntimeError(f"unsafe archive member: {member.name}")
|
||||
try:
|
||||
archive.extractall(target_dir, members=members, filter="data")
|
||||
except TypeError:
|
||||
archive.extractall(target_dir, members=members)
|
||||
|
||||
|
||||
def execute_release_action(
|
||||
payload: dict,
|
||||
*,
|
||||
run_command,
|
||||
default_api_service_name: str,
|
||||
event_callback=None,
|
||||
urlopen_func=None,
|
||||
user_agent: str = "domaincheck-ops/0.1",
|
||||
) -> tuple[bool, str, dict]:
|
||||
normalized_payload = dict(payload or {})
|
||||
event_callback = event_callback or (lambda event_type, message, level="info", payload=None: None)
|
||||
urlopen_func = urlopen_func or urllib.request.urlopen
|
||||
|
||||
release_version = str(normalized_payload.get("release_version") or "").strip()
|
||||
artifact_url = str(normalized_payload.get("artifact_url") or "").strip()
|
||||
checksum = str(normalized_payload.get("checksum") or "").strip().lower()
|
||||
install_root = Path(str(normalized_payload.get("install_root") or "/opt/domaincheck")).resolve()
|
||||
switch_current = coerce_bool(normalized_payload.get("switch_current", True), default=True)
|
||||
restart_services = normalize_text_list(normalized_payload.get("restart_services"))
|
||||
health_check_urls = normalize_text_list(normalized_payload.get("health_check_urls"))
|
||||
health_check_services = normalize_release_health_check_services(
|
||||
normalized_payload,
|
||||
restart_services,
|
||||
default_api_service_name=default_api_service_name,
|
||||
)
|
||||
health_check_timeout_seconds = max(2, int(normalized_payload.get("health_check_timeout_seconds") or 10))
|
||||
health_check_retries = max(0, int(normalized_payload.get("health_check_retries") or 2))
|
||||
health_check_interval_seconds = max(0, int(normalized_payload.get("health_check_interval_seconds") or 2))
|
||||
rollback_on_failure = coerce_bool(normalized_payload.get("rollback_on_failure", True), default=True)
|
||||
|
||||
if not release_version:
|
||||
return False, "release_version missing", {}
|
||||
if not artifact_url:
|
||||
return False, "artifact_url missing", {}
|
||||
|
||||
downloads_dir = install_root / "downloads"
|
||||
releases_dir = install_root / "releases"
|
||||
target_dir = releases_dir / release_version
|
||||
temp_dir = releases_dir / f"{release_version}.tmp"
|
||||
artifact_path = downloads_dir / f"{release_version}.tar.gz"
|
||||
current_link = install_root / "current"
|
||||
previous_current_target = ""
|
||||
downloads_dir.mkdir(parents=True, exist_ok=True)
|
||||
releases_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if current_link.exists():
|
||||
try:
|
||||
previous_current_target = str(current_link.resolve(strict=True))
|
||||
except Exception:
|
||||
try:
|
||||
previous_current_target = str(current_link.resolve())
|
||||
except Exception:
|
||||
previous_current_target = ""
|
||||
|
||||
event_callback(
|
||||
"deploy_download_started",
|
||||
f"开始下载发布包: {release_version}",
|
||||
payload={"artifact_url": artifact_url},
|
||||
)
|
||||
request = urllib.request.Request(
|
||||
url=artifact_url,
|
||||
headers={"User-Agent": str(user_agent or "domaincheck-ops/0.1").strip()},
|
||||
method="GET",
|
||||
)
|
||||
with urlopen_func(request, timeout=120) as response:
|
||||
artifact_bytes = response.read()
|
||||
artifact_path.write_bytes(artifact_bytes)
|
||||
event_callback(
|
||||
"deploy_download_completed",
|
||||
f"发布包下载完成: {release_version}",
|
||||
payload={"artifact_path": str(artifact_path), "size_bytes": len(artifact_bytes)},
|
||||
)
|
||||
|
||||
calculated_checksum = hashlib.sha256(artifact_bytes).hexdigest().lower()
|
||||
if checksum and checksum != calculated_checksum:
|
||||
return False, "release checksum mismatch", {
|
||||
"expected_checksum": checksum,
|
||||
"calculated_checksum": calculated_checksum,
|
||||
}
|
||||
event_callback(
|
||||
"deploy_checksum_verified",
|
||||
f"发布包校验通过: {release_version}",
|
||||
payload={"checksum": calculated_checksum},
|
||||
)
|
||||
|
||||
if temp_dir.exists():
|
||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||
|
||||
temp_dir.mkdir(parents=True, exist_ok=True)
|
||||
with tarfile.open(artifact_path, "r:gz") as archive:
|
||||
safe_extract_tar(archive, temp_dir)
|
||||
|
||||
if target_dir.exists():
|
||||
extracted_target = target_dir
|
||||
shutil.rmtree(temp_dir, ignore_errors=True)
|
||||
else:
|
||||
temp_dir.rename(target_dir)
|
||||
extracted_target = target_dir
|
||||
|
||||
meta_path = extracted_target / ".release-meta.json"
|
||||
meta_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"release_version": release_version,
|
||||
"artifact_url": artifact_url,
|
||||
"checksum": calculated_checksum,
|
||||
"deployed_at": datetime.now().isoformat(sep=" ", timespec="seconds"),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
if switch_current:
|
||||
if current_link.is_symlink() or current_link.is_file():
|
||||
current_link.unlink(missing_ok=True)
|
||||
elif current_link.exists():
|
||||
raise RuntimeError(f"current link path exists and is not a symlink/file: {current_link}")
|
||||
current_link.symlink_to(extracted_target)
|
||||
event_callback(
|
||||
"deploy_current_switched",
|
||||
f"current 已切换到 {release_version}",
|
||||
payload={"current_link": str(current_link), "target": str(extracted_target)},
|
||||
)
|
||||
|
||||
restarted: list[dict] = []
|
||||
for service_name in restart_services:
|
||||
normalized_service_name = str(service_name or "").strip()
|
||||
if not normalized_service_name:
|
||||
continue
|
||||
event_callback(
|
||||
"deploy_service_restart",
|
||||
f"重启服务: {normalized_service_name}",
|
||||
payload={"service_name": normalized_service_name},
|
||||
)
|
||||
code, stdout, stderr = run_command(["systemctl", "restart", normalized_service_name], timeout=30)
|
||||
restarted.append(
|
||||
{
|
||||
"service_name": normalized_service_name,
|
||||
"returncode": int(code or 0),
|
||||
"stdout": stdout,
|
||||
"stderr": stderr,
|
||||
}
|
||||
)
|
||||
if int(code or 0) != 0:
|
||||
return False, f"restart failed: {normalized_service_name}", {
|
||||
"release_version": release_version,
|
||||
"release_dir": str(extracted_target),
|
||||
"artifact_path": str(artifact_path),
|
||||
"checksum": calculated_checksum,
|
||||
"previous_current_target": previous_current_target,
|
||||
"restart_results": restarted,
|
||||
}
|
||||
|
||||
health_ok, health_result = run_release_health_checks(
|
||||
urls=health_check_urls,
|
||||
services=health_check_services,
|
||||
timeout=health_check_timeout_seconds,
|
||||
retries=health_check_retries,
|
||||
interval_seconds=health_check_interval_seconds,
|
||||
run_command=run_command,
|
||||
user_agent=user_agent,
|
||||
urlopen_func=urlopen_func,
|
||||
)
|
||||
if not health_ok:
|
||||
rollback_result = {
|
||||
"attempted": rollback_on_failure,
|
||||
"restored_to": previous_current_target,
|
||||
"restart_results": [],
|
||||
"health_result": health_result,
|
||||
}
|
||||
event_callback(
|
||||
"deploy_health_failed",
|
||||
f"发布健康检查失败: {release_version}",
|
||||
level="error",
|
||||
payload=health_result,
|
||||
)
|
||||
if rollback_on_failure and previous_current_target:
|
||||
if current_link.is_symlink() or current_link.is_file():
|
||||
current_link.unlink(missing_ok=True)
|
||||
elif current_link.exists():
|
||||
raise RuntimeError(f"current link path exists and is not a symlink/file: {current_link}")
|
||||
current_link.symlink_to(Path(previous_current_target))
|
||||
for service_name in restart_services:
|
||||
normalized_service_name = str(service_name or "").strip()
|
||||
if not normalized_service_name:
|
||||
continue
|
||||
code, stdout, stderr = run_command(["systemctl", "restart", normalized_service_name], timeout=30)
|
||||
rollback_result["restart_results"].append(
|
||||
{
|
||||
"service_name": normalized_service_name,
|
||||
"returncode": int(code or 0),
|
||||
"stdout": stdout,
|
||||
"stderr": stderr,
|
||||
}
|
||||
)
|
||||
rollback_result["post_rollback_health"] = run_release_health_checks(
|
||||
urls=health_check_urls,
|
||||
services=health_check_services,
|
||||
timeout=health_check_timeout_seconds,
|
||||
retries=0,
|
||||
interval_seconds=0,
|
||||
run_command=run_command,
|
||||
user_agent=user_agent,
|
||||
urlopen_func=urlopen_func,
|
||||
)[1]
|
||||
event_callback(
|
||||
"deploy_rollback_completed",
|
||||
f"已回滚到旧版本: {previous_current_target}",
|
||||
level="warning",
|
||||
payload=rollback_result,
|
||||
)
|
||||
return False, "release health check failed", {
|
||||
"release_version": release_version,
|
||||
"release_dir": str(extracted_target),
|
||||
"artifact_path": str(artifact_path),
|
||||
"checksum": calculated_checksum,
|
||||
"current_link": str(current_link),
|
||||
"previous_current_target": previous_current_target,
|
||||
"restart_results": restarted,
|
||||
"health_check": health_result,
|
||||
"rollback": rollback_result,
|
||||
}
|
||||
|
||||
event_callback(
|
||||
"deploy_health_passed",
|
||||
f"发布健康检查通过: {release_version}",
|
||||
payload=health_result,
|
||||
)
|
||||
return True, "release deployed", {
|
||||
"release_version": release_version,
|
||||
"release_dir": str(extracted_target),
|
||||
"artifact_path": str(artifact_path),
|
||||
"checksum": calculated_checksum,
|
||||
"current_link": str(current_link),
|
||||
"previous_current_target": previous_current_target,
|
||||
"restart_results": restarted,
|
||||
"health_check": health_result,
|
||||
}
|
||||
|
||||
|
||||
def build_remote_release_action_script(
|
||||
payload: dict,
|
||||
*,
|
||||
default_api_service_name: str = "domaincheck-api",
|
||||
user_agent: str = "domaincheck-ssh/0.1",
|
||||
) -> str:
|
||||
helper_functions = [
|
||||
normalize_text_list,
|
||||
coerce_bool,
|
||||
normalize_release_health_check_services,
|
||||
collect_service_state,
|
||||
check_health_url,
|
||||
run_release_health_checks,
|
||||
safe_extract_tar,
|
||||
execute_release_action,
|
||||
]
|
||||
helper_source = "\n\n".join(
|
||||
textwrap.dedent(inspect.getsource(func)).strip("\n")
|
||||
for func in helper_functions
|
||||
)
|
||||
return f"""from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import tarfile
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
{helper_source}
|
||||
|
||||
|
||||
PAYLOAD = {json.dumps(dict(payload or {{}}), ensure_ascii=False)}
|
||||
DEFAULT_API_SERVICE_NAME = {json.dumps(str(default_api_service_name or 'domaincheck-api'), ensure_ascii=False)}
|
||||
USER_AGENT = {json.dumps(str(user_agent or 'domaincheck-ssh/0.1'), ensure_ascii=False)}
|
||||
|
||||
|
||||
def _run(command, timeout=60):
|
||||
import subprocess
|
||||
|
||||
completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
|
||||
return int(completed.returncode or 0), str(completed.stdout or "").strip(), str(completed.stderr or "").strip()
|
||||
|
||||
|
||||
def _event_callback(event_type, message, level="info", payload=None):
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
ok, message, result = execute_release_action(
|
||||
PAYLOAD,
|
||||
run_command=_run,
|
||||
default_api_service_name=DEFAULT_API_SERVICE_NAME,
|
||||
event_callback=_event_callback,
|
||||
urlopen_func=urllib.request.urlopen,
|
||||
user_agent=USER_AGENT,
|
||||
)
|
||||
except Exception as exc:
|
||||
ok, message, result = False, str(exc), {{"exception": str(exc), "action": "deploy.release"}}
|
||||
print(json.dumps({{"ok": ok, "message": message, "result": result}}, ensure_ascii=False))
|
||||
raise SystemExit(0 if ok else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
"""
|
||||
3448
domain-api/app/services/ops_release_service.py
Normal file
3448
domain-api/app/services/ops_release_service.py
Normal file
File diff suppressed because it is too large
Load Diff
420
domain-api/app/services/ops_runtime_executor_service.py
Normal file
420
domain-api/app/services/ops_runtime_executor_service.py
Normal file
@@ -0,0 +1,420 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.ops_action_executor_core import (
|
||||
STRUCTURED_ACTIONS,
|
||||
build_service_name_map,
|
||||
execute_structured_action,
|
||||
trim_output,
|
||||
)
|
||||
from app.services.ops_release_executor_core import build_remote_release_action_script, execute_release_action
|
||||
|
||||
|
||||
_SSH_SUPPORTED_ACTIONS = set(STRUCTURED_ACTIONS) | {"deploy.release"}
|
||||
_SSH_CONNECT_TIMEOUT_SECONDS = 12
|
||||
_SSH_REMOTE_TIMEOUT_SECONDS = {
|
||||
"health.snapshot": 30,
|
||||
"service.status": 45,
|
||||
"service.restart": 45,
|
||||
"service.start": 45,
|
||||
"service.stop": 45,
|
||||
"logs.collect": 45,
|
||||
"diagnostics.collect": 90,
|
||||
"runtime.start_worker": 45,
|
||||
"runtime.stop_worker": 45,
|
||||
"runtime.restart_api": 45,
|
||||
"runtime.start_sync_agent": 45,
|
||||
"runtime.stop_sync_agent": 45,
|
||||
"deploy.release": 180,
|
||||
}
|
||||
|
||||
|
||||
def supports_ssh_execution(action: str) -> bool:
|
||||
return str(action or "").strip() in _SSH_SUPPORTED_ACTIONS
|
||||
|
||||
|
||||
def execute_local_support_action(action: str, payload: dict | None = None) -> tuple[bool, str, dict]:
|
||||
normalized_action = str(action or "").strip()
|
||||
if normalized_action not in _SSH_SUPPORTED_ACTIONS:
|
||||
return False, f"当前未实现本机支持动作: {normalized_action}", {}
|
||||
if normalized_action == "deploy.release":
|
||||
return execute_release_action(
|
||||
dict(payload or {}),
|
||||
run_command=_run_local_command,
|
||||
default_api_service_name=settings.api_service_name,
|
||||
user_agent="domaincheck-local-runtime/0.1",
|
||||
)
|
||||
return execute_structured_action(
|
||||
normalized_action,
|
||||
dict(payload or {}),
|
||||
service_names=_service_name_map(),
|
||||
runner=_run_local_command,
|
||||
)
|
||||
|
||||
|
||||
def execute_ssh_action(node: dict, action: str, payload: dict | None = None) -> tuple[bool, str, dict]:
|
||||
normalized_action = str(action or "").strip()
|
||||
if normalized_action not in _SSH_SUPPORTED_ACTIONS:
|
||||
return False, f"当前 SSH 执行器暂不支持动作: {normalized_action}", {}
|
||||
|
||||
ssh_host = str(node.get("ssh_host") or "").strip()
|
||||
ssh_user = str(node.get("ssh_user") or "").strip()
|
||||
ssh_port = max(1, int(node.get("ssh_port") or 22))
|
||||
node_code = str(node.get("node_code") or "").strip()
|
||||
if not ssh_host or not ssh_user:
|
||||
return False, "目标节点缺少 SSH 主机或用户信息", {
|
||||
"node_code": node_code,
|
||||
"ssh_host": ssh_host,
|
||||
"ssh_user": ssh_user,
|
||||
"ssh_port": ssh_port,
|
||||
}
|
||||
|
||||
normalized_payload = dict(payload or {})
|
||||
service_names = _service_name_map()
|
||||
if normalized_action == "deploy.release":
|
||||
remote_script = build_remote_release_action_script(
|
||||
normalized_payload,
|
||||
default_api_service_name=service_names.get("api") or "domaincheck-api",
|
||||
user_agent="domaincheck-ssh/0.1",
|
||||
)
|
||||
else:
|
||||
remote_script = _build_remote_python_script(
|
||||
action=normalized_action,
|
||||
payload=normalized_payload,
|
||||
service_names=service_names,
|
||||
)
|
||||
remote_command = "\n".join(
|
||||
[
|
||||
"set -euo pipefail",
|
||||
"if command -v python3 >/dev/null 2>&1; then",
|
||||
" PYTHON_BIN=python3",
|
||||
"elif command -v python >/dev/null 2>&1; then",
|
||||
" PYTHON_BIN=python",
|
||||
"else",
|
||||
' echo "{\\"ok\\": false, \\"message\\": \\"python interpreter missing on remote node\\", \\"result\\": {\\"executor\\": \\"ssh\\"}}"',
|
||||
" exit 127",
|
||||
"fi",
|
||||
'"${PYTHON_BIN}" - <<\'PY\'',
|
||||
remote_script.rstrip("\n"),
|
||||
"PY",
|
||||
]
|
||||
)
|
||||
ssh_command = [
|
||||
"ssh",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"PreferredAuthentications=publickey",
|
||||
"-o",
|
||||
"StrictHostKeyChecking=accept-new",
|
||||
"-o",
|
||||
f"ConnectTimeout={_SSH_CONNECT_TIMEOUT_SECONDS}",
|
||||
"-p",
|
||||
str(ssh_port),
|
||||
f"{ssh_user}@{ssh_host}",
|
||||
remote_command,
|
||||
]
|
||||
timeout_seconds = int(_SSH_REMOTE_TIMEOUT_SECONDS.get(normalized_action, 45) or 45) + _SSH_CONNECT_TIMEOUT_SECONDS
|
||||
completed = subprocess.run(
|
||||
ssh_command,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout_seconds,
|
||||
)
|
||||
|
||||
transport = {
|
||||
"executor": "ssh",
|
||||
"node_code": node_code,
|
||||
"ssh_host": ssh_host,
|
||||
"ssh_user": ssh_user,
|
||||
"ssh_port": ssh_port,
|
||||
"action": normalized_action,
|
||||
"returncode": int(completed.returncode or 0),
|
||||
}
|
||||
parsed_ok, parsed_payload = _extract_json_from_output(completed.stdout or "")
|
||||
if parsed_ok:
|
||||
ok = bool(parsed_payload.get("ok", False))
|
||||
message = str(parsed_payload.get("message") or "").strip() or ("SSH 动作执行成功" if ok else "SSH 动作执行失败")
|
||||
result = dict(parsed_payload.get("result") or {})
|
||||
result["transport"] = {
|
||||
**transport,
|
||||
"stdout_preview": trim_output(completed.stdout, 4000),
|
||||
"stderr_preview": trim_output(completed.stderr, 2000),
|
||||
}
|
||||
if completed.returncode != 0 and ok:
|
||||
ok = False
|
||||
message = f"SSH 命令返回码异常: {completed.returncode}"
|
||||
return ok, message, result
|
||||
|
||||
stderr_preview = trim_output(completed.stderr, 4000)
|
||||
stdout_preview = trim_output(completed.stdout, 4000)
|
||||
message = stderr_preview or stdout_preview or f"SSH 执行失败,返回码 {completed.returncode}"
|
||||
return False, message, {
|
||||
"executor": "ssh",
|
||||
"action": normalized_action,
|
||||
"transport": {
|
||||
**transport,
|
||||
"stdout_preview": stdout_preview,
|
||||
"stderr_preview": stderr_preview,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _service_name_map() -> dict[str, str]:
|
||||
return build_service_name_map(
|
||||
api_service_name=settings.api_service_name,
|
||||
worker_service_name=settings.worker_service_name,
|
||||
sync_agent_service_name=settings.sync_agent_service_name,
|
||||
)
|
||||
|
||||
|
||||
def _run_local_command(command: list[str], *, timeout: int = 60) -> tuple[int, str, str]:
|
||||
completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
|
||||
return int(completed.returncode or 0), str(completed.stdout or "").strip(), str(completed.stderr or "").strip()
|
||||
|
||||
|
||||
def _build_remote_python_script(*, action: str, payload: dict, service_names: dict[str, str]) -> str:
|
||||
return f"""import json
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
SERVICE_NAMES = {json.dumps(service_names, ensure_ascii=False)}
|
||||
ACTION = {json.dumps(str(action or '').strip(), ensure_ascii=False)}
|
||||
PAYLOAD = {json.dumps(dict(payload or {}), ensure_ascii=False)}
|
||||
|
||||
|
||||
def trim_output(text, limit):
|
||||
value = str(text or "").strip()
|
||||
if len(value) <= int(limit):
|
||||
return value
|
||||
return value[-int(limit):]
|
||||
|
||||
|
||||
def run(cmd, timeout=60):
|
||||
completed = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
|
||||
return int(completed.returncode or 0), str(completed.stdout or "").strip(), str(completed.stderr or "").strip()
|
||||
|
||||
|
||||
def read_tail_lines(path, max_lines):
|
||||
file_path = Path(path)
|
||||
if not file_path.exists() or not file_path.is_file():
|
||||
return []
|
||||
with file_path.open("r", encoding="utf-8", errors="replace") as handle:
|
||||
return handle.read().splitlines()[-int(max_lines):]
|
||||
|
||||
|
||||
def candidate_domain_roots():
|
||||
roots = []
|
||||
for candidate in (
|
||||
Path.cwd() / "domainCheck",
|
||||
Path("/opt/domaincheck/domainCheck"),
|
||||
Path("/www/wwwroot/getDomain/domainCheck"),
|
||||
):
|
||||
if candidate not in roots:
|
||||
roots.append(candidate)
|
||||
return roots
|
||||
|
||||
|
||||
def candidate_runtime_logs_dirs():
|
||||
dirs = []
|
||||
for candidate in (
|
||||
Path.cwd() / "domain-api" / "runtime" / "logs",
|
||||
Path("/opt/domaincheck/domain-api/runtime/logs"),
|
||||
Path("/www/wwwroot/getDomain/domain-api/runtime/logs"),
|
||||
):
|
||||
if candidate not in dirs:
|
||||
dirs.append(candidate)
|
||||
return dirs
|
||||
|
||||
|
||||
def service_log_file_candidates(service_name):
|
||||
normalized_service_name = str(service_name or "").strip()
|
||||
if not normalized_service_name:
|
||||
return []
|
||||
candidates = []
|
||||
|
||||
def append_candidate(path):
|
||||
if path not in candidates:
|
||||
candidates.append(path)
|
||||
|
||||
if normalized_service_name == (SERVICE_NAMES.get("worker") or "domaincheck-worker"):
|
||||
for domain_root in candidate_domain_roots():
|
||||
append_candidate(domain_root / "detect_worker.log")
|
||||
append_candidate(domain_root / "logs" / "detect_worker.log")
|
||||
elif normalized_service_name == (SERVICE_NAMES.get("api") or "domaincheck-api"):
|
||||
for runtime_logs_dir in candidate_runtime_logs_dirs():
|
||||
append_candidate(runtime_logs_dir / "domain-api.stderr.log")
|
||||
append_candidate(runtime_logs_dir / "domain-api.stdout.log")
|
||||
for domain_root in candidate_domain_roots():
|
||||
append_candidate(domain_root / "logs" / "app.log")
|
||||
|
||||
return candidates
|
||||
|
||||
|
||||
def collect_log_file_fallback(service_name, lines):
|
||||
aggregated_lines = []
|
||||
used_paths = []
|
||||
for path in service_log_file_candidates(service_name):
|
||||
current_lines = read_tail_lines(path, lines)
|
||||
if not current_lines:
|
||||
continue
|
||||
aggregated_lines.extend(current_lines)
|
||||
used_paths.append(str(path))
|
||||
if not aggregated_lines:
|
||||
return "", []
|
||||
return "\\n".join(aggregated_lines[-int(lines):]), used_paths
|
||||
|
||||
|
||||
def service_name_from_payload(payload):
|
||||
value = str((payload or {{}}).get("service_name") or "").strip()
|
||||
return value or SERVICE_NAMES.get("worker") or "domaincheck-worker"
|
||||
|
||||
|
||||
def service_name_for_action(action, payload):
|
||||
if action in ("service.status", "service.restart", "service.start", "service.stop"):
|
||||
return service_name_from_payload(payload)
|
||||
runtime_action_map = {{
|
||||
"runtime.start_worker": SERVICE_NAMES.get("worker") or "domaincheck-worker",
|
||||
"runtime.stop_worker": SERVICE_NAMES.get("worker") or "domaincheck-worker",
|
||||
"runtime.restart_api": SERVICE_NAMES.get("api") or "domaincheck-api",
|
||||
"runtime.start_sync_agent": SERVICE_NAMES.get("sync_agent") or "domaincheck-sync-agent",
|
||||
"runtime.stop_sync_agent": SERVICE_NAMES.get("sync_agent") or "domaincheck-sync-agent",
|
||||
}}
|
||||
return str(runtime_action_map.get(action) or "").strip()
|
||||
|
||||
|
||||
def systemctl_action_name(action):
|
||||
if action in ("service.restart", "runtime.restart_api"):
|
||||
return "restart"
|
||||
if action in ("service.start", "runtime.start_worker", "runtime.start_sync_agent"):
|
||||
return "start"
|
||||
if action in ("service.stop", "runtime.stop_worker", "runtime.stop_sync_agent"):
|
||||
return "stop"
|
||||
return ""
|
||||
|
||||
|
||||
def execute(action, payload):
|
||||
if action == "health.snapshot":
|
||||
checks = {{}}
|
||||
for key, service_name in SERVICE_NAMES.items():
|
||||
code, stdout, stderr = run(["systemctl", "is-active", service_name], timeout=15)
|
||||
checks[key] = {{
|
||||
"service_name": service_name,
|
||||
"returncode": int(code or 0),
|
||||
"state": stdout or stderr,
|
||||
"ok": int(code or 0) == 0 and (stdout or stderr) == "active",
|
||||
}}
|
||||
return True, "health snapshot collected", {{"checks": checks}}
|
||||
|
||||
if action == "service.status":
|
||||
service_name = service_name_from_payload(payload)
|
||||
code, stdout, stderr = run(["systemctl", "status", service_name, "--no-pager", "-l"], timeout=45)
|
||||
result = {{
|
||||
"service_name": service_name,
|
||||
"stdout": trim_output(stdout, 12000),
|
||||
"stderr": trim_output(stderr, 4000),
|
||||
}}
|
||||
return int(code or 0) == 0, stdout or stderr or f"{{service_name}} status collected", result
|
||||
|
||||
systemctl_action = systemctl_action_name(action)
|
||||
if systemctl_action:
|
||||
service_name = service_name_for_action(action, payload)
|
||||
if not service_name:
|
||||
return False, f"missing service name for action: {{action}}", {{"action": action}}
|
||||
code, stdout, stderr = run(["systemctl", systemctl_action, service_name], timeout=45)
|
||||
result = {{
|
||||
"service_name": service_name,
|
||||
"systemctl_action": systemctl_action,
|
||||
"stdout": trim_output(stdout, 12000),
|
||||
"stderr": trim_output(stderr, 4000),
|
||||
}}
|
||||
return int(code or 0) == 0, stdout or stderr or f"{{service_name}} {{systemctl_action}} completed", result
|
||||
|
||||
if action == "logs.collect":
|
||||
service_name = service_name_from_payload(payload)
|
||||
lines = max(20, min(int((payload or {{}}).get("lines") or 120), 500))
|
||||
code, stdout, stderr = run(["journalctl", "-u", service_name, "-n", str(lines), "--no-pager"], timeout=45)
|
||||
journal_output = stdout or stderr
|
||||
fallback_output, fallback_paths = collect_log_file_fallback(service_name, lines)
|
||||
collection_source = "journal"
|
||||
effective_output = journal_output
|
||||
ok = int(code or 0) == 0
|
||||
if (not ok or not stdout.strip()) and fallback_output:
|
||||
collection_source = "file_fallback"
|
||||
effective_output = fallback_output
|
||||
ok = True
|
||||
result = {{
|
||||
"service_name": service_name,
|
||||
"lines": lines,
|
||||
"collection_source": collection_source,
|
||||
"fallback_used": bool(collection_source == "file_fallback"),
|
||||
"fallback_reason": trim_output(journal_output, 4000) if collection_source == "file_fallback" else "",
|
||||
"log_paths": fallback_paths,
|
||||
"journal_returncode": int(code or 0),
|
||||
"stdout": trim_output(effective_output, 20000),
|
||||
"stderr": trim_output(stderr, 4000),
|
||||
}}
|
||||
return ok, effective_output or f"{{service_name}} logs collected", result
|
||||
|
||||
if action == "diagnostics.collect":
|
||||
lines = max(20, min(int((payload or {{}}).get("lines") or 200), 800))
|
||||
diagnostics = {{
|
||||
"services": {{}},
|
||||
"lines": lines,
|
||||
}}
|
||||
for key, service_name in SERVICE_NAMES.items():
|
||||
active_code, active_stdout, active_stderr = run(["systemctl", "is-active", service_name], timeout=15)
|
||||
status_code, status_stdout, status_stderr = run(["systemctl", "status", service_name, "--no-pager", "-l"], timeout=45)
|
||||
log_code, log_stdout, log_stderr = run(["journalctl", "-u", service_name, "-n", str(lines), "--no-pager"], timeout=45)
|
||||
log_output = log_stdout or log_stderr
|
||||
fallback_output, fallback_paths = collect_log_file_fallback(service_name, lines)
|
||||
log_source = "journal"
|
||||
if (int(log_code or 0) != 0 or not log_stdout.strip()) and fallback_output:
|
||||
log_output = fallback_output
|
||||
log_source = "file_fallback"
|
||||
diagnostics["services"][key] = {{
|
||||
"service_name": service_name,
|
||||
"active_returncode": int(active_code or 0),
|
||||
"active_state": active_stdout or active_stderr,
|
||||
"status_returncode": int(status_code or 0),
|
||||
"status_output": trim_output(status_stdout or status_stderr, 12000),
|
||||
"log_returncode": int(log_code or 0),
|
||||
"log_source": log_source,
|
||||
"log_paths": fallback_paths,
|
||||
"log_output": trim_output(log_output, 20000),
|
||||
}}
|
||||
return True, "diagnostics collected", diagnostics
|
||||
|
||||
return False, f"unsupported action: {{action}}", {{"action": action}}
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
ok, message, result = execute(ACTION, PAYLOAD)
|
||||
except Exception as exc:
|
||||
ok, message, result = False, str(exc), {{"exception": str(exc), "action": ACTION}}
|
||||
print(json.dumps({{"ok": ok, "message": message, "result": result}}, ensure_ascii=False))
|
||||
raise SystemExit(0 if ok else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
"""
|
||||
|
||||
|
||||
def _extract_json_from_output(text: str) -> tuple[bool, dict]:
|
||||
for raw_line in reversed(str(text or "").splitlines()):
|
||||
line = str(raw_line or "").strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
data = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if isinstance(data, dict) and "ok" in data:
|
||||
return True, data
|
||||
return False, {}
|
||||
11611
domain-api/app/services/ops_service.py
Normal file
11611
domain-api/app/services/ops_service.py
Normal file
File diff suppressed because it is too large
Load Diff
920
domain-api/app/services/ops_template_service.py
Normal file
920
domain-api/app/services/ops_template_service.py
Normal file
@@ -0,0 +1,920 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from copy import deepcopy
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.ops_execution_mode_service import decorate_execution_mode_fields, execution_mode_label
|
||||
|
||||
|
||||
def _split_text_list(raw_value: object) -> list[str]:
|
||||
if isinstance(raw_value, list):
|
||||
return [str(item).strip() for item in raw_value if str(item).strip()]
|
||||
text = str(raw_value or "").replace("\r", "\n")
|
||||
if not text.strip():
|
||||
return []
|
||||
normalized = text.replace(",", "\n")
|
||||
return [item.strip() for item in normalized.split("\n") if item.strip()]
|
||||
|
||||
|
||||
def _coerce_bool(raw_value: object) -> bool:
|
||||
if isinstance(raw_value, bool):
|
||||
return raw_value
|
||||
if isinstance(raw_value, (int, float)):
|
||||
return bool(raw_value)
|
||||
normalized = str(raw_value or "").strip().lower()
|
||||
if normalized in {"1", "true", "yes", "y", "on", "enable", "enabled"}:
|
||||
return True
|
||||
if normalized in {"0", "false", "no", "n", "off", "disable", "disabled"}:
|
||||
return False
|
||||
return bool(normalized)
|
||||
|
||||
|
||||
def _is_missing_value(field_type: str, value: object) -> bool:
|
||||
if field_type == "text_list":
|
||||
return not list(value or [])
|
||||
if field_type == "number":
|
||||
return value is None
|
||||
if field_type == "boolean":
|
||||
return value is None
|
||||
return str(value or "").strip() == ""
|
||||
|
||||
|
||||
def _service_options() -> list[dict]:
|
||||
return [
|
||||
{"label": f"API ({settings.api_service_name})", "value": settings.api_service_name},
|
||||
{"label": f"Worker ({settings.worker_service_name})", "value": settings.worker_service_name},
|
||||
{"label": f"Sync Agent ({settings.sync_agent_service_name})", "value": settings.sync_agent_service_name},
|
||||
{"label": "Node Agent (domaincheck-node-agent)", "value": "domaincheck-node-agent"},
|
||||
]
|
||||
|
||||
|
||||
def _template_catalog() -> list[dict]:
|
||||
service_options = _service_options()
|
||||
return [
|
||||
{
|
||||
"group_key": "runtime",
|
||||
"group_title": "运行时控制",
|
||||
"group_description": "适合日常按钮化控制 Worker / API / Sync Agent。",
|
||||
"items": [
|
||||
{
|
||||
"key": "runtime.start_worker",
|
||||
"title": "启动 Worker",
|
||||
"action": "runtime.start_worker",
|
||||
"description": "启动目标节点的检测 Worker。",
|
||||
"risk_level": "medium",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [],
|
||||
},
|
||||
{
|
||||
"key": "runtime.stop_worker",
|
||||
"title": "停止 Worker",
|
||||
"action": "runtime.stop_worker",
|
||||
"description": "停止目标节点的检测 Worker。",
|
||||
"risk_level": "high",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [],
|
||||
},
|
||||
{
|
||||
"key": "runtime.restart_api",
|
||||
"title": "重启 API",
|
||||
"action": "runtime.restart_api",
|
||||
"description": "重启目标节点的 domaincheck-api 服务。",
|
||||
"risk_level": "high",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [],
|
||||
},
|
||||
{
|
||||
"key": "runtime.start_sync_agent",
|
||||
"title": "启动 Sync Agent",
|
||||
"action": "runtime.start_sync_agent",
|
||||
"description": "启动目标节点的同步代理服务。",
|
||||
"risk_level": "medium",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [],
|
||||
},
|
||||
{
|
||||
"key": "runtime.stop_sync_agent",
|
||||
"title": "停止 Sync Agent",
|
||||
"action": "runtime.stop_sync_agent",
|
||||
"description": "停止目标节点的同步代理服务。",
|
||||
"risk_level": "high",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [],
|
||||
},
|
||||
{
|
||||
"key": "service.restart",
|
||||
"title": "重启任意服务",
|
||||
"action": "service.restart",
|
||||
"description": "对指定 systemd 服务执行 restart。",
|
||||
"risk_level": "high",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [
|
||||
{
|
||||
"key": "service_name",
|
||||
"label": "服务名",
|
||||
"type": "select",
|
||||
"required": True,
|
||||
"default": settings.worker_service_name,
|
||||
"options": service_options,
|
||||
"help": "默认提供 API / Worker / Sync Agent / Node Agent。",
|
||||
}
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"group_key": "diagnostics",
|
||||
"group_title": "巡检与取证",
|
||||
"group_description": "适合做联调、排障、日志回收和节点健康诊断。",
|
||||
"items": [
|
||||
{
|
||||
"key": "health.snapshot",
|
||||
"title": "采集健康快照",
|
||||
"action": "health.snapshot",
|
||||
"description": "采集目标节点关键 systemd 服务状态。",
|
||||
"risk_level": "low",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": True,
|
||||
"fields": [],
|
||||
},
|
||||
{
|
||||
"key": "logs.collect",
|
||||
"title": "收集服务日志",
|
||||
"action": "logs.collect",
|
||||
"description": "抓取指定服务最近 journalctl 日志。",
|
||||
"risk_level": "low",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": True,
|
||||
"fields": [
|
||||
{
|
||||
"key": "service_name",
|
||||
"label": "服务名",
|
||||
"type": "select",
|
||||
"required": True,
|
||||
"default": settings.worker_service_name,
|
||||
"options": service_options,
|
||||
"help": "默认抓 Worker;排 API 问题时可切到 domaincheck-api。",
|
||||
},
|
||||
{
|
||||
"key": "lines",
|
||||
"label": "日志行数",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 120,
|
||||
"min": 20,
|
||||
"max": 500,
|
||||
"help": "journalctl -n 的行数,适合短时排障。",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"key": "diagnostics.collect",
|
||||
"title": "收集诊断包",
|
||||
"action": "diagnostics.collect",
|
||||
"description": "打包 API / Worker / Sync Agent / Node Agent 的状态与近期日志。",
|
||||
"risk_level": "low",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": True,
|
||||
"fields": [
|
||||
{
|
||||
"key": "lines",
|
||||
"label": "每个服务日志行数",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 200,
|
||||
"min": 20,
|
||||
"max": 800,
|
||||
"help": "越大越适合深度排障,但任务返回会更重。",
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"key": "service.status",
|
||||
"title": "采集服务状态",
|
||||
"action": "service.status",
|
||||
"description": "执行 systemctl status --no-pager -l。",
|
||||
"risk_level": "low",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "local-runtime", "ssh"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": True,
|
||||
"fields": [
|
||||
{
|
||||
"key": "service_name",
|
||||
"label": "服务名",
|
||||
"type": "select",
|
||||
"required": True,
|
||||
"default": settings.worker_service_name,
|
||||
"options": service_options,
|
||||
"help": "用于查看特定服务当前状态和最近日志片段。",
|
||||
}
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"group_key": "delivery_queue",
|
||||
"group_title": "回执队列治理",
|
||||
"group_description": "适合统一处理 Node Agent 的 pending / dead-letter 回执队列,而不是手工登机删文件。",
|
||||
"items": [
|
||||
{
|
||||
"key": "delivery.queue.flush",
|
||||
"title": "立即冲刷回执队列",
|
||||
"action": "delivery.queue.flush",
|
||||
"description": "立即触发目标节点 Node Agent 冲刷 pending 回执队列。",
|
||||
"risk_level": "low",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": True,
|
||||
"fields": [
|
||||
{
|
||||
"key": "limit",
|
||||
"label": "本次冲刷上限",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 20,
|
||||
"min": 1,
|
||||
"max": 200,
|
||||
"help": "限制本次最多冲刷的回执数量,避免一次返回过大。",
|
||||
}
|
||||
],
|
||||
},
|
||||
{
|
||||
"key": "delivery.queue.replay",
|
||||
"title": "重放死信回执",
|
||||
"action": "delivery.queue.replay",
|
||||
"description": "把 dead-letter 记录重新放回 pending,并可立即触发冲刷。",
|
||||
"risk_level": "medium",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [
|
||||
{
|
||||
"key": "record_id",
|
||||
"label": "单条记录 ID",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"help": "留空表示按下面的 selector 条件批量匹配。",
|
||||
},
|
||||
{
|
||||
"key": "request_kind",
|
||||
"label": "记录类型",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"help": "可选 job_complete / job_event。",
|
||||
},
|
||||
{
|
||||
"key": "detail_code",
|
||||
"label": "错误码",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"help": "按 last_detail_code 过滤死信记录。",
|
||||
},
|
||||
{
|
||||
"key": "limit",
|
||||
"label": "重放上限",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 20,
|
||||
"min": 1,
|
||||
"max": 200,
|
||||
"help": "批量重放时最多处理多少条死信。",
|
||||
},
|
||||
{
|
||||
"key": "flush_after_replay",
|
||||
"label": "重放后立即冲刷",
|
||||
"type": "boolean",
|
||||
"required": True,
|
||||
"default": True,
|
||||
"help": "开启后会把重放回 pending 的记录立即补发一次。",
|
||||
},
|
||||
{
|
||||
"key": "reason",
|
||||
"label": "重放原因",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"help": "可选,便于后续回溯为什么执行此次重放。",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"key": "delivery.queue.discard",
|
||||
"title": "丢弃死信回执",
|
||||
"action": "delivery.queue.discard",
|
||||
"description": "把死信移出活动队列,保存到 discarded 归档目录。",
|
||||
"risk_level": "high",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [
|
||||
{
|
||||
"key": "record_id",
|
||||
"label": "单条记录 ID",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"help": "留空表示按下面的 selector 条件批量匹配。",
|
||||
},
|
||||
{
|
||||
"key": "request_kind",
|
||||
"label": "记录类型",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"help": "可选 job_complete / job_event。",
|
||||
},
|
||||
{
|
||||
"key": "detail_code",
|
||||
"label": "错误码",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"help": "按 last_detail_code 过滤死信记录。",
|
||||
},
|
||||
{
|
||||
"key": "limit",
|
||||
"label": "丢弃上限",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 20,
|
||||
"min": 1,
|
||||
"max": 200,
|
||||
"help": "批量丢弃时最多处理多少条死信。",
|
||||
},
|
||||
{
|
||||
"key": "reason",
|
||||
"label": "丢弃原因",
|
||||
"type": "text",
|
||||
"required": True,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"help": "必须说明为何确认这些死信可以被放弃。",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"group_key": "onboarding",
|
||||
"group_title": "接管与纳管",
|
||||
"group_description": "适合在控制面生成节点接入工单、统一沉淀首轮接管动作。",
|
||||
"items": [
|
||||
{
|
||||
"key": "node.bootstrap",
|
||||
"title": "生成节点接入工单",
|
||||
"action": "node.bootstrap",
|
||||
"description": "在控制面签发 Node Agent Token,并生成 bootstrap env / 脚本 / 一键落地命令。",
|
||||
"risk_level": "critical",
|
||||
"default_execution_mode": "control-plane",
|
||||
"execution_modes": ["control-plane"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [
|
||||
{
|
||||
"key": "control_plane_base_url",
|
||||
"label": "控制面地址",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"help": "可留空。留空时由控制面自动回退到当前 API 地址。",
|
||||
},
|
||||
{
|
||||
"key": "root_dir",
|
||||
"label": "目标安装目录",
|
||||
"type": "text",
|
||||
"required": True,
|
||||
"default": "/opt/domaincheck",
|
||||
"help": "bootstrap_node_agent.sh 安装根目录。",
|
||||
},
|
||||
{
|
||||
"key": "expires_in_hours",
|
||||
"label": "Token 有效期(小时)",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 72,
|
||||
"min": 1,
|
||||
"max": 720,
|
||||
"help": "控制新签发 token 的过期时间。",
|
||||
},
|
||||
{
|
||||
"key": "node_region",
|
||||
"label": "节点地域提示",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"help": "可留空;留空时优先从托管节点或集群快照自动推断。",
|
||||
},
|
||||
{
|
||||
"key": "node_role",
|
||||
"label": "节点角色提示",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"help": "可留空;留空时优先从托管节点或集群快照自动推断。",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"group_key": "release",
|
||||
"group_title": "发布与变更",
|
||||
"group_description": "适合把单节点灰度、点状修复和手工发布收编到标准 deploy.release 任务。",
|
||||
"items": [
|
||||
{
|
||||
"key": "deploy.release.control",
|
||||
"title": "发布控制面节点",
|
||||
"action": "deploy.release",
|
||||
"description": "面向 controller 节点的标准发布任务,默认会重启 API / Worker / Sync Agent 并执行 API 健康检查。",
|
||||
"risk_level": "high",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "ssh"],
|
||||
"target_roles": ["control"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [
|
||||
{
|
||||
"key": "release_version",
|
||||
"label": "版本号",
|
||||
"type": "text",
|
||||
"required": True,
|
||||
"default": "",
|
||||
"help": "例如 2026.04.18-rc1,用于 releases/<version> 目录名。",
|
||||
},
|
||||
{
|
||||
"key": "artifact_url",
|
||||
"label": "发布包地址",
|
||||
"type": "textarea",
|
||||
"required": True,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"rows": 2,
|
||||
"help": "Node Agent 会在目标节点直接下载 tar.gz 发布包。",
|
||||
},
|
||||
{
|
||||
"key": "checksum",
|
||||
"label": "SHA256 校验",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"help": "可留空;填写后会强校验发布包完整性。",
|
||||
},
|
||||
{
|
||||
"key": "install_root",
|
||||
"label": "安装根目录",
|
||||
"type": "text",
|
||||
"required": True,
|
||||
"default": "/opt/domaincheck",
|
||||
"help": "目标节点上的 releases/current 根目录。",
|
||||
},
|
||||
{
|
||||
"key": "restart_services",
|
||||
"label": "重启服务",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": [
|
||||
settings.api_service_name,
|
||||
settings.worker_service_name,
|
||||
settings.sync_agent_service_name,
|
||||
],
|
||||
"wide": True,
|
||||
"rows": 3,
|
||||
"help": "每行一个服务。控制面默认重启 API / Worker / Sync Agent。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_urls",
|
||||
"label": "健康检查 URL",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": ["http://127.0.0.1:8100/health"],
|
||||
"wide": True,
|
||||
"rows": 2,
|
||||
"help": "每行一个 URL。控制面默认探活本机 API /health。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_services",
|
||||
"label": "健康检查服务",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": [
|
||||
settings.api_service_name,
|
||||
settings.worker_service_name,
|
||||
settings.sync_agent_service_name,
|
||||
],
|
||||
"wide": True,
|
||||
"rows": 3,
|
||||
"help": "每行一个 systemd 服务,发布后会检查其 active 状态。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_timeout_seconds",
|
||||
"label": "URL 超时",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 10,
|
||||
"min": 2,
|
||||
"max": 120,
|
||||
"help": "每次 URL 探活超时秒数。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_retries",
|
||||
"label": "重试次数",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 2,
|
||||
"min": 0,
|
||||
"max": 10,
|
||||
"help": "健康检查失败后的额外重试次数。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_interval_seconds",
|
||||
"label": "重试间隔",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 2,
|
||||
"min": 0,
|
||||
"max": 60,
|
||||
"help": "每次重试前的等待秒数。",
|
||||
},
|
||||
{
|
||||
"key": "rollback_on_failure",
|
||||
"label": "失败自动回滚",
|
||||
"type": "boolean",
|
||||
"required": True,
|
||||
"default": True,
|
||||
"help": "健康检查失败时自动切回旧 current 并重启服务。",
|
||||
},
|
||||
{
|
||||
"key": "switch_current",
|
||||
"label": "切换 current 软链",
|
||||
"type": "boolean",
|
||||
"required": True,
|
||||
"default": True,
|
||||
"help": "关闭后只解压版本目录,不切 current。",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"key": "deploy.release.worker",
|
||||
"title": "发布 Worker 节点",
|
||||
"action": "deploy.release",
|
||||
"description": "面向独立 Worker 节点的标准发布任务,默认只重启 Worker 并校验 Worker 服务状态。",
|
||||
"risk_level": "high",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "ssh"],
|
||||
"target_roles": ["worker"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [
|
||||
{
|
||||
"key": "release_version",
|
||||
"label": "版本号",
|
||||
"type": "text",
|
||||
"required": True,
|
||||
"default": "",
|
||||
"help": "例如 2026.04.18-rc1,用于 releases/<version> 目录名。",
|
||||
},
|
||||
{
|
||||
"key": "artifact_url",
|
||||
"label": "发布包地址",
|
||||
"type": "textarea",
|
||||
"required": True,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"rows": 2,
|
||||
"help": "Node Agent 会在目标节点直接下载 tar.gz 发布包。",
|
||||
},
|
||||
{
|
||||
"key": "checksum",
|
||||
"label": "SHA256 校验",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"help": "可留空;填写后会强校验发布包完整性。",
|
||||
},
|
||||
{
|
||||
"key": "install_root",
|
||||
"label": "安装根目录",
|
||||
"type": "text",
|
||||
"required": True,
|
||||
"default": "/opt/domaincheck",
|
||||
"help": "目标节点上的 releases/current 根目录。",
|
||||
},
|
||||
{
|
||||
"key": "restart_services",
|
||||
"label": "重启服务",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": [settings.worker_service_name],
|
||||
"wide": True,
|
||||
"rows": 2,
|
||||
"help": "每行一个服务。独立 Worker 默认只重启 Worker。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_urls",
|
||||
"label": "健康检查 URL",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": [],
|
||||
"wide": True,
|
||||
"rows": 2,
|
||||
"help": "Worker 节点通常可留空;为空时不做 URL 探活。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_services",
|
||||
"label": "健康检查服务",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": [settings.worker_service_name],
|
||||
"wide": True,
|
||||
"rows": 2,
|
||||
"help": "每行一个 systemd 服务,默认只校验 Worker active 状态。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_timeout_seconds",
|
||||
"label": "URL 超时",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 10,
|
||||
"min": 2,
|
||||
"max": 120,
|
||||
"help": "每次 URL 探活超时秒数。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_retries",
|
||||
"label": "重试次数",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 2,
|
||||
"min": 0,
|
||||
"max": 10,
|
||||
"help": "健康检查失败后的额外重试次数。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_interval_seconds",
|
||||
"label": "重试间隔",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 2,
|
||||
"min": 0,
|
||||
"max": 60,
|
||||
"help": "每次重试前的等待秒数。",
|
||||
},
|
||||
{
|
||||
"key": "rollback_on_failure",
|
||||
"label": "失败自动回滚",
|
||||
"type": "boolean",
|
||||
"required": True,
|
||||
"default": True,
|
||||
"help": "健康检查失败时自动切回旧 current 并重启服务。",
|
||||
},
|
||||
{
|
||||
"key": "switch_current",
|
||||
"label": "切换 current 软链",
|
||||
"type": "boolean",
|
||||
"required": True,
|
||||
"default": True,
|
||||
"help": "关闭后只解压版本目录,不切 current。",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"key": "deploy.release.custom",
|
||||
"title": "发布自定义节点",
|
||||
"action": "deploy.release",
|
||||
"description": "完全自定义 deploy.release 参数,适合灰度验证、特殊节点或后续扩展场景。",
|
||||
"risk_level": "high",
|
||||
"default_execution_mode": "remote-agent",
|
||||
"execution_modes": ["remote-agent", "ssh"],
|
||||
"target_roles": ["control", "worker"],
|
||||
"default_auto_approve": False,
|
||||
"fields": [
|
||||
{
|
||||
"key": "release_version",
|
||||
"label": "版本号",
|
||||
"type": "text",
|
||||
"required": True,
|
||||
"default": "",
|
||||
"help": "例如 2026.04.18-rc1,用于 releases/<version> 目录名。",
|
||||
},
|
||||
{
|
||||
"key": "artifact_url",
|
||||
"label": "发布包地址",
|
||||
"type": "textarea",
|
||||
"required": True,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"rows": 2,
|
||||
"help": "Node Agent 会在目标节点直接下载 tar.gz 发布包。",
|
||||
},
|
||||
{
|
||||
"key": "checksum",
|
||||
"label": "SHA256 校验",
|
||||
"type": "text",
|
||||
"required": False,
|
||||
"default": "",
|
||||
"wide": True,
|
||||
"help": "可留空;填写后会强校验发布包完整性。",
|
||||
},
|
||||
{
|
||||
"key": "install_root",
|
||||
"label": "安装根目录",
|
||||
"type": "text",
|
||||
"required": True,
|
||||
"default": "/opt/domaincheck",
|
||||
"help": "目标节点上的 releases/current 根目录。",
|
||||
},
|
||||
{
|
||||
"key": "restart_services",
|
||||
"label": "重启服务",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": [],
|
||||
"wide": True,
|
||||
"rows": 3,
|
||||
"help": "每行一个服务。留空表示只落盘版本,不主动重启任何服务。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_urls",
|
||||
"label": "健康检查 URL",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": [],
|
||||
"wide": True,
|
||||
"rows": 2,
|
||||
"help": "每行一个 URL。留空表示不做 URL 探活。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_services",
|
||||
"label": "健康检查服务",
|
||||
"type": "text_list",
|
||||
"required": False,
|
||||
"default": [],
|
||||
"wide": True,
|
||||
"rows": 3,
|
||||
"help": "每行一个 systemd 服务。显式留空时不会自动回落到默认 API 检查。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_timeout_seconds",
|
||||
"label": "URL 超时",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 10,
|
||||
"min": 2,
|
||||
"max": 120,
|
||||
"help": "每次 URL 探活超时秒数。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_retries",
|
||||
"label": "重试次数",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 2,
|
||||
"min": 0,
|
||||
"max": 10,
|
||||
"help": "健康检查失败后的额外重试次数。",
|
||||
},
|
||||
{
|
||||
"key": "health_check_interval_seconds",
|
||||
"label": "重试间隔",
|
||||
"type": "number",
|
||||
"required": True,
|
||||
"default": 2,
|
||||
"min": 0,
|
||||
"max": 60,
|
||||
"help": "每次重试前的等待秒数。",
|
||||
},
|
||||
{
|
||||
"key": "rollback_on_failure",
|
||||
"label": "失败自动回滚",
|
||||
"type": "boolean",
|
||||
"required": True,
|
||||
"default": True,
|
||||
"help": "健康检查失败时自动切回旧 current 并重启服务。",
|
||||
},
|
||||
{
|
||||
"key": "switch_current",
|
||||
"label": "切换 current 软链",
|
||||
"type": "boolean",
|
||||
"required": True,
|
||||
"default": True,
|
||||
"help": "关闭后只解压版本目录,不切 current。",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def get_ops_action_templates() -> dict:
|
||||
groups = deepcopy(_template_catalog())
|
||||
items: list[dict] = []
|
||||
for group in groups:
|
||||
group_key = str(group.get("group_key") or "").strip()
|
||||
group_title = str(group.get("group_title") or "").strip()
|
||||
group_description = str(group.get("group_description") or "").strip()
|
||||
normalized_group_items: list[dict] = []
|
||||
for item in list(group.get("items") or []):
|
||||
normalized_item = decorate_execution_mode_fields(dict(item))
|
||||
normalized_item["group_key"] = group_key
|
||||
normalized_item["group_title"] = group_title
|
||||
normalized_item["group_description"] = group_description
|
||||
normalized_group_items.append(normalized_item)
|
||||
items.append(normalized_item)
|
||||
group["items"] = normalized_group_items
|
||||
return {
|
||||
"groups": groups,
|
||||
"items": items,
|
||||
"defaults": {
|
||||
"requested_by": "web-ui",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": execution_mode_label("remote-agent"),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def get_ops_action_template(template_key: str) -> dict:
|
||||
normalized_key = str(template_key or "").strip()
|
||||
if not normalized_key:
|
||||
return {}
|
||||
for group in _template_catalog():
|
||||
group_key = str(group.get("group_key") or "").strip()
|
||||
group_title = str(group.get("group_title") or "").strip()
|
||||
group_description = str(group.get("group_description") or "").strip()
|
||||
for item in list(group.get("items") or []):
|
||||
if str(item.get("key") or "").strip() == normalized_key:
|
||||
normalized_item = decorate_execution_mode_fields(deepcopy(item))
|
||||
normalized_item["group_key"] = group_key
|
||||
normalized_item["group_title"] = group_title
|
||||
normalized_item["group_description"] = group_description
|
||||
return normalized_item
|
||||
return {}
|
||||
|
||||
|
||||
def build_ops_template_payload(template_key: str, payload: dict | None = None) -> tuple[bool, str, dict]:
|
||||
template = get_ops_action_template(template_key)
|
||||
if not template:
|
||||
return False, "动作模板不存在", {}
|
||||
|
||||
source_payload = dict(payload or {})
|
||||
normalized_payload: dict = {}
|
||||
for field in list(template.get("fields") or []):
|
||||
field_key = str(field.get("key") or "").strip()
|
||||
if not field_key:
|
||||
continue
|
||||
field_type = str(field.get("type") or "text").strip()
|
||||
raw_value = source_payload.get(field_key, field.get("default"))
|
||||
if field_type == "number":
|
||||
try:
|
||||
value = int(raw_value or 0)
|
||||
except Exception:
|
||||
return False, f"{field_key} 必须是数字", {}
|
||||
min_value = field.get("min")
|
||||
max_value = field.get("max")
|
||||
if min_value is not None:
|
||||
value = max(int(min_value), value)
|
||||
if max_value is not None:
|
||||
value = min(int(max_value), value)
|
||||
elif field_type == "boolean":
|
||||
value = _coerce_bool(raw_value)
|
||||
elif field_type == "text_list":
|
||||
value = _split_text_list(raw_value)
|
||||
else:
|
||||
value = str(raw_value or "").strip()
|
||||
if bool(field.get("required", False)) and _is_missing_value(field_type, value):
|
||||
return False, f"{field_key} 不能为空", {}
|
||||
normalized_payload[field_key] = value
|
||||
return True, "ok", normalized_payload
|
||||
@@ -7,6 +7,7 @@ from app.core.config import settings
|
||||
from app.core.db import get_db
|
||||
from app.core.files import read_json
|
||||
from app.core.redis_client import get_redis
|
||||
from app.services.build_info_service import get_runtime_build_info
|
||||
from app.services.cluster_runtime_service import get_cluster_snapshot
|
||||
from app.services.detect_service import get_detect_status
|
||||
from app.services.detect_job_service import get_detect_capacity_plan, get_detect_queue_health
|
||||
@@ -187,6 +188,89 @@ def _build_multi_region_readiness(
|
||||
}
|
||||
|
||||
|
||||
def _detect_participation_snapshot(*, row: dict) -> dict:
|
||||
items_running = int(row.get("items_running", 0) or 0)
|
||||
items_claimed = int(row.get("items_claimed", 0) or 0)
|
||||
processed_recent = int(row.get("processed_recent", 0) or 0)
|
||||
current_load = int(row.get("current_load", 0) or 0)
|
||||
status = str(row.get("status") or "").strip().lower()
|
||||
|
||||
if items_running > 0:
|
||||
return {
|
||||
"participation_state": "running",
|
||||
"participation_label": "执行中",
|
||||
"participation_reason": f"当前正在执行 {items_running} 项检测任务。",
|
||||
"is_current_participant": True,
|
||||
"is_dispatch_active": True,
|
||||
}
|
||||
if items_claimed > 0:
|
||||
return {
|
||||
"participation_state": "claimed",
|
||||
"participation_label": "已领待跑",
|
||||
"participation_reason": f"已领取 {items_claimed} 项任务,等待线程继续执行。",
|
||||
"is_current_participant": True,
|
||||
"is_dispatch_active": True,
|
||||
}
|
||||
if processed_recent > 0:
|
||||
return {
|
||||
"participation_state": "recent_throughput",
|
||||
"participation_label": "近窗有吞吐",
|
||||
"participation_reason": f"近 15 分钟内已处理 {processed_recent} 项任务。",
|
||||
"is_current_participant": True,
|
||||
"is_dispatch_active": False,
|
||||
}
|
||||
if current_load > 0 or status == "busy":
|
||||
load_value = max(current_load, 1)
|
||||
return {
|
||||
"participation_state": "load_syncing",
|
||||
"participation_label": "负载待确认",
|
||||
"participation_reason": f"节点当前负载为 {load_value},但还未观察到已领、执行中或近窗吞吐数据,先归入在线未参与观察。",
|
||||
"is_current_participant": False,
|
||||
"is_dispatch_active": False,
|
||||
}
|
||||
return {
|
||||
"participation_state": "standby",
|
||||
"participation_label": "在线待命",
|
||||
"participation_reason": "当前未领任务、未执行任务,也没有近窗吞吐。",
|
||||
"is_current_participant": False,
|
||||
"is_dispatch_active": False,
|
||||
}
|
||||
|
||||
|
||||
def _build_detect_node_row(*, node_code: str, cluster_node: dict, job_node: dict, queue_node: dict) -> dict:
|
||||
metadata = cluster_node.get("metadata") or {}
|
||||
role = str(cluster_node.get("role") or job_node.get("role") or "worker")
|
||||
region = str(cluster_node.get("region") or settings.node_region)
|
||||
is_effective_worker = bool(cluster_node.get("is_effective_worker", False) or role == "worker")
|
||||
items_total = int(job_node.get("items_total", metadata.get("job_items_total", 0)) or 0)
|
||||
items_claimed = int(job_node.get("items_claimed", metadata.get("job_items_claimed", 0)) or 0)
|
||||
items_running = int(job_node.get("items_running", metadata.get("job_items_running", 0)) or 0)
|
||||
items_completed = int(job_node.get("items_completed", metadata.get("job_items_completed", 0)) or 0)
|
||||
items_failed = int(job_node.get("items_failed", metadata.get("job_items_failed", 0)) or 0)
|
||||
row = {
|
||||
"node_code": node_code,
|
||||
"role": role,
|
||||
"region": region,
|
||||
"status": str(cluster_node.get("status") or "unknown"),
|
||||
"is_effective_worker": is_effective_worker,
|
||||
"detect_participating": False,
|
||||
"current_load": int(cluster_node.get("current_load", 0) or 0),
|
||||
"items_total": items_total,
|
||||
"items_pending": int(job_node.get("items_pending", max(items_total - items_claimed - items_completed - items_failed, 0)) or 0),
|
||||
"items_claimed": items_claimed,
|
||||
"items_running": items_running,
|
||||
"items_completed": items_completed,
|
||||
"items_failed": items_failed,
|
||||
"processed_recent": int(queue_node.get("processed_recent", 0) or 0),
|
||||
"processed_per_minute": float(queue_node.get("processed_per_minute", 0) or 0),
|
||||
"last_heartbeat_at": str(cluster_node.get("last_heartbeat_at") or ""),
|
||||
"metrics_source": "active_job" if bool(job_node) else ("cluster_metadata" if metadata else "derived"),
|
||||
}
|
||||
row.update(_detect_participation_snapshot(row=row))
|
||||
row["detect_participating"] = bool(row.get("is_current_participant", False))
|
||||
return row
|
||||
|
||||
|
||||
def _build_participating_detect_nodes(*, cluster_snapshot: dict, detect_snapshot: dict, worker_runtime: dict) -> list[dict]:
|
||||
cluster_nodes = list(cluster_snapshot.get("nodes") or [])
|
||||
active_job = detect_snapshot.get("active_job") or {}
|
||||
@@ -204,56 +288,25 @@ def _build_participating_detect_nodes(*, cluster_snapshot: dict, detect_snapshot
|
||||
}
|
||||
|
||||
merged: list[dict] = []
|
||||
seen: set[str] = set()
|
||||
for item in list(active_job.get("node_stats") or []):
|
||||
node_code = str(item.get("node_code") or "").strip()
|
||||
if not node_code or node_code == "unassigned":
|
||||
continue
|
||||
seen.add(node_code)
|
||||
job_map = {
|
||||
str(item.get("node_code") or "").strip(): item
|
||||
for item in list(active_job.get("node_stats") or [])
|
||||
if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned"
|
||||
}
|
||||
candidate_codes = sorted(set(job_map) | set(queue_map) | set(cluster_map))
|
||||
for node_code in candidate_codes:
|
||||
cluster_node = cluster_map.get(node_code, {})
|
||||
queue_node = queue_map.get(node_code, {})
|
||||
merged.append(
|
||||
{
|
||||
"node_code": node_code,
|
||||
"role": str(cluster_node.get("role") or "worker"),
|
||||
"region": str(cluster_node.get("region") or settings.node_region),
|
||||
"status": str(cluster_node.get("status") or "unknown"),
|
||||
"is_effective_worker": bool(cluster_node.get("is_effective_worker", False) or str(cluster_node.get("role") or "") == "worker"),
|
||||
"detect_participating": True,
|
||||
"current_load": int(cluster_node.get("current_load", 0) or 0),
|
||||
"items_total": int(item.get("items_total", 0) or 0),
|
||||
"items_pending": int(item.get("items_pending", 0) or 0),
|
||||
"items_claimed": int(item.get("items_claimed", 0) or 0),
|
||||
"items_running": int(item.get("items_running", 0) or 0),
|
||||
"items_completed": int(item.get("items_completed", 0) or 0),
|
||||
"items_failed": int(item.get("items_failed", 0) or 0),
|
||||
"processed_recent": int(queue_node.get("processed_recent", 0) or 0),
|
||||
"processed_per_minute": float(queue_node.get("processed_per_minute", 0) or 0),
|
||||
}
|
||||
if cluster_node and not bool(cluster_node.get("is_effective_worker", False)):
|
||||
continue
|
||||
row = _build_detect_node_row(
|
||||
node_code=node_code,
|
||||
cluster_node=cluster_node,
|
||||
job_node=job_map.get(node_code, {}),
|
||||
queue_node=queue_map.get(node_code, {}),
|
||||
)
|
||||
|
||||
if worker_runtime.get("running", False) and settings.node_code not in seen:
|
||||
cluster_node = cluster_map.get(settings.node_code, {})
|
||||
if cluster_node:
|
||||
merged.append(
|
||||
{
|
||||
"node_code": settings.node_code,
|
||||
"role": str(cluster_node.get("role") or settings.node_role),
|
||||
"region": str(cluster_node.get("region") or settings.node_region),
|
||||
"status": str(cluster_node.get("status") or "online"),
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": True,
|
||||
"current_load": int(cluster_node.get("current_load", 0) or 0),
|
||||
"items_total": 0,
|
||||
"items_pending": 0,
|
||||
"items_claimed": 0,
|
||||
"items_running": 0,
|
||||
"items_completed": 0,
|
||||
"items_failed": 0,
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0,
|
||||
}
|
||||
)
|
||||
if not bool(row.get("is_current_participant", False)):
|
||||
continue
|
||||
merged.append(row)
|
||||
|
||||
return sorted(
|
||||
merged,
|
||||
@@ -261,11 +314,143 @@ def _build_participating_detect_nodes(*, cluster_snapshot: dict, detect_snapshot
|
||||
-int(item.get("items_running", 0) or 0),
|
||||
-int(item.get("items_claimed", 0) or 0),
|
||||
-int(item.get("processed_recent", 0) or 0),
|
||||
-int(item.get("current_load", 0) or 0),
|
||||
str(item.get("node_code") or ""),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _build_standby_detect_nodes(*, cluster_snapshot: dict, participating_nodes: list[dict]) -> list[dict]:
|
||||
cluster_nodes = list(cluster_snapshot.get("nodes") or [])
|
||||
participating_codes = {
|
||||
str(item.get("node_code") or "").strip()
|
||||
for item in list(participating_nodes or [])
|
||||
if str(item.get("node_code") or "").strip()
|
||||
}
|
||||
standby_rows: list[dict] = []
|
||||
for node in cluster_nodes:
|
||||
node_code = str(node.get("node_code") or "").strip()
|
||||
node_status = str(node.get("status") or "").strip()
|
||||
node_current_load = int(node.get("current_load", 0) or 0)
|
||||
metadata = node.get("metadata") or {}
|
||||
if not node_code:
|
||||
continue
|
||||
if not bool(node.get("is_effective_worker", False)):
|
||||
continue
|
||||
if node_status not in {"online", "busy"}:
|
||||
continue
|
||||
if node_code in participating_codes:
|
||||
continue
|
||||
standby_state = "load_syncing" if node_current_load > 0 or node_status == "busy" else "standby"
|
||||
standby_label = "负载待确认" if standby_state == "load_syncing" else "在线待命"
|
||||
standby_reason = (
|
||||
str(metadata.get("detail") or "").strip()
|
||||
or str(metadata.get("phase_detail") or "").strip()
|
||||
or (
|
||||
f"当前阶段:{str(metadata.get('phase') or metadata.get('phase_label') or '').strip()}"
|
||||
if str(metadata.get("phase") or metadata.get("phase_label") or "").strip()
|
||||
else ""
|
||||
)
|
||||
or (
|
||||
f"节点当前负载为 {node_current_load},但还未观察到已领、执行中或近窗吞吐数据。"
|
||||
if standby_state == "load_syncing"
|
||||
else "节点在线,当前未领任务、未执行任务,也没有近窗吞吐。"
|
||||
)
|
||||
)
|
||||
standby_rows.append(
|
||||
{
|
||||
"node_code": node_code,
|
||||
"role": str(node.get("role") or "worker"),
|
||||
"region": str(node.get("region") or settings.node_region),
|
||||
"status": node_status,
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": False,
|
||||
"participation_state": standby_state,
|
||||
"participation_label": standby_label,
|
||||
"participation_reason": standby_reason,
|
||||
"current_load": node_current_load,
|
||||
"last_heartbeat_at": str(node.get("last_heartbeat_at") or ""),
|
||||
"phase": str(metadata.get("phase") or metadata.get("phase_label") or "").strip(),
|
||||
"detail": str(metadata.get("detail") or metadata.get("phase_detail") or "").strip(),
|
||||
"worker_online": bool(metadata.get("worker_online", False)),
|
||||
"standby_reason": standby_reason,
|
||||
}
|
||||
)
|
||||
return sorted(
|
||||
standby_rows,
|
||||
key=lambda item: (
|
||||
str(item.get("status") or ""),
|
||||
str(item.get("role") or ""),
|
||||
str(item.get("node_code") or ""),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _build_detect_participation_summary(
|
||||
*,
|
||||
participating_nodes: list[dict],
|
||||
standby_nodes: list[dict],
|
||||
cluster_snapshot: dict,
|
||||
) -> dict:
|
||||
cluster_nodes = list(cluster_snapshot.get("nodes") or [])
|
||||
effective_online_nodes = [
|
||||
node
|
||||
for node in cluster_nodes
|
||||
if bool(node.get("is_effective_worker", False)) and str(node.get("status") or "").strip() in {"online", "busy"}
|
||||
]
|
||||
dispatch_active_nodes = [
|
||||
row for row in participating_nodes
|
||||
if bool(row.get("is_dispatch_active", False))
|
||||
]
|
||||
recent_only_nodes = [
|
||||
row for row in participating_nodes
|
||||
if str(row.get("participation_state") or "").strip() == "recent_throughput"
|
||||
]
|
||||
load_syncing_nodes = [
|
||||
row for row in standby_nodes
|
||||
if str(row.get("participation_state") or "").strip() == "load_syncing"
|
||||
]
|
||||
pure_standby_nodes = [
|
||||
row for row in standby_nodes
|
||||
if str(row.get("participation_state") or "").strip() == "standby"
|
||||
]
|
||||
dedicated_worker_nodes = [
|
||||
node for node in effective_online_nodes
|
||||
if str(node.get("role") or "").strip() == "worker"
|
||||
]
|
||||
controller_worker_nodes = [
|
||||
node for node in effective_online_nodes
|
||||
if str(node.get("role") or "").strip() == "control"
|
||||
]
|
||||
|
||||
summary_parts = [
|
||||
f"有效执行节点 {len(effective_online_nodes)} 台",
|
||||
f"正在执行/领任务 {len(dispatch_active_nodes)} 台",
|
||||
f"近窗刚有吞吐 {len(recent_only_nodes)} 台",
|
||||
f"在线但未参与 {len(standby_nodes)} 台",
|
||||
]
|
||||
if load_syncing_nodes:
|
||||
summary_parts.append(f"其中负载待确认 {len(load_syncing_nodes)} 台")
|
||||
|
||||
return {
|
||||
"effective_online_nodes": len(effective_online_nodes),
|
||||
"participating_nodes": len(participating_nodes),
|
||||
"dispatch_active_nodes": len(dispatch_active_nodes),
|
||||
"recent_only_nodes": len(recent_only_nodes),
|
||||
"non_participating_nodes": len(standby_nodes),
|
||||
"standby_nodes": len(pure_standby_nodes),
|
||||
"load_syncing_nodes": len(load_syncing_nodes),
|
||||
"dedicated_worker_nodes": len(dedicated_worker_nodes),
|
||||
"controller_worker_nodes": len(controller_worker_nodes),
|
||||
"dispatch_active_node_codes": [str(item.get("node_code") or "") for item in dispatch_active_nodes],
|
||||
"recent_only_node_codes": [str(item.get("node_code") or "") for item in recent_only_nodes],
|
||||
"non_participating_node_codes": [str(item.get("node_code") or "") for item in standby_nodes],
|
||||
"standby_node_codes": [str(item.get("node_code") or "") for item in pure_standby_nodes],
|
||||
"load_syncing_node_codes": [str(item.get("node_code") or "") for item in load_syncing_nodes],
|
||||
"summary": ";".join(summary_parts),
|
||||
}
|
||||
|
||||
|
||||
def get_runtime_status() -> dict:
|
||||
runtime_settings = get_runtime_settings()
|
||||
worker_runtime = detect_worker_runtime()
|
||||
@@ -314,6 +499,17 @@ def get_runtime_status() -> dict:
|
||||
"worker_mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
|
||||
"queue_health": queue_health,
|
||||
"capacity_plan": capacity_plan,
|
||||
"log_sync": {
|
||||
"enabled": bool(runtime_settings.get("worker_log_sync_enabled", False)),
|
||||
"mode": str(runtime_settings.get("worker_log_sync_mode", "key") or "key"),
|
||||
"line_count": int(detect_snapshot.get("remote_log_line_count", 0) or 0),
|
||||
"source_node_count": int(detect_snapshot.get("remote_log_node_count", 0) or 0),
|
||||
"source_nodes": list(detect_snapshot.get("remote_log_nodes") or []),
|
||||
"source_node_summaries": list(detect_snapshot.get("remote_log_node_summaries") or []),
|
||||
"last_at": str(detect_snapshot.get("remote_log_last_at") or ""),
|
||||
"last_line": str(detect_snapshot.get("remote_log_last_line") or ""),
|
||||
"preview_lines": list(detect_snapshot.get("remote_log_lines") or [])[-20:],
|
||||
},
|
||||
}
|
||||
if not worker_expected_on_this_node:
|
||||
detect_payload.update(
|
||||
@@ -349,6 +545,16 @@ def get_runtime_status() -> dict:
|
||||
detect_snapshot=detect_payload,
|
||||
worker_runtime=worker_runtime,
|
||||
)
|
||||
detect_payload["standby_nodes"] = _build_standby_detect_nodes(
|
||||
cluster_snapshot=cluster_snapshot,
|
||||
participating_nodes=detect_payload["participating_nodes"],
|
||||
)
|
||||
detect_payload["non_participating_nodes"] = list(detect_payload["standby_nodes"] or [])
|
||||
detect_payload["participation_summary"] = _build_detect_participation_summary(
|
||||
participating_nodes=detect_payload["participating_nodes"],
|
||||
standby_nodes=detect_payload["non_participating_nodes"],
|
||||
cluster_snapshot=cluster_snapshot,
|
||||
)
|
||||
append_runtime_projection_if_changed(detect=detect_payload, cluster=cluster_snapshot)
|
||||
sync_summary = get_sync_summary(record_limit=5)
|
||||
readiness = _build_multi_region_readiness(
|
||||
@@ -357,6 +563,7 @@ def get_runtime_status() -> dict:
|
||||
worker_runtime=worker_runtime,
|
||||
sync_agent_runtime=sync_agent_runtime,
|
||||
)
|
||||
build_info = get_runtime_build_info()
|
||||
|
||||
return {
|
||||
"api": {
|
||||
@@ -371,6 +578,7 @@ def get_runtime_status() -> dict:
|
||||
"health_url": f"http://127.0.0.1:{settings.api_port}/health",
|
||||
"stdout_log": _runtime_log_path("domain-api.stdout.log"),
|
||||
"stderr_log": _runtime_log_path("domain-api.stderr.log"),
|
||||
"build": build_info,
|
||||
},
|
||||
"node": {
|
||||
"code": settings.node_code,
|
||||
|
||||
@@ -450,6 +450,17 @@ def append_runtime_projection_if_changed(
|
||||
continue
|
||||
local_participating = bool(node.get("detect_participating", False) or node.get("current_load", 0))
|
||||
break
|
||||
local_job_bucket = {}
|
||||
for item in list(active_job.get("node_stats") or []):
|
||||
if str(item.get("node_code") or "").strip() != settings.node_code:
|
||||
continue
|
||||
local_job_bucket = item
|
||||
break
|
||||
if not local_participating:
|
||||
local_participating = bool(
|
||||
int(local_job_bucket.get("items_running", 0) or 0) > 0
|
||||
or int(local_job_bucket.get("items_claimed", 0) or 0) > 0
|
||||
)
|
||||
projection = {
|
||||
"node": {
|
||||
"node_code": settings.node_code,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -12,7 +12,7 @@ TEMPLATE_DIR="$API_DIR/deploy/multi-region/templates"
|
||||
|
||||
if [ "$ROLE" != "controller" ] && [ "$ROLE" != "worker" ]; then
|
||||
echo "invalid role: $ROLE"
|
||||
echo "usage: bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck [controller|worker]"
|
||||
echo "usage: bash domain-api/deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck [controller|worker]"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
61
domain-api/deploy/multi-region/bootstrap_node_agent.sh
Executable file
61
domain-api/deploy/multi-region/bootstrap_node_agent.sh
Executable file
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT_DIR="${1:-/opt/domaincheck}"
|
||||
ENV_FILE="/etc/default/domaincheck-node-agent"
|
||||
SERVICE_FILE="/etc/systemd/system/domaincheck-node-agent.service"
|
||||
SERVICE_NAME="domaincheck-node-agent"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
resolve_project_root() {
|
||||
local candidate
|
||||
for candidate in \
|
||||
"${ROOT_DIR}/domainCheck" \
|
||||
"${ROOT_DIR}/domain-api" \
|
||||
"${ROOT_DIR}" \
|
||||
"$(cd "${SCRIPT_DIR}/../.." && pwd)"; do
|
||||
if [[ -f "${candidate}/deploy/systemd/domain-node-agent.service" && -f "${candidate}/deploy/multi-region/templates/domaincheck-node-agent.env.example" ]]; then
|
||||
printf '%s' "${candidate}"
|
||||
return 0
|
||||
fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
PROJECT_ROOT="$(resolve_project_root || true)"
|
||||
if [[ -z "${PROJECT_ROOT}" ]]; then
|
||||
echo "unable to resolve project root under ${ROOT_DIR} or ${SCRIPT_DIR}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
TEMPLATE_FILE="${PROJECT_ROOT}/deploy/multi-region/templates/domaincheck-node-agent.env.example"
|
||||
UNIT_TEMPLATE="${PROJECT_ROOT}/deploy/systemd/domain-node-agent.service"
|
||||
|
||||
echo "[1/5] install systemd unit"
|
||||
install -m 0644 "${UNIT_TEMPLATE}" "${SERVICE_FILE}"
|
||||
|
||||
echo "[2/5] install env template if absent"
|
||||
if [[ ! -f "${ENV_FILE}" ]]; then
|
||||
install -m 0644 "${TEMPLATE_FILE}" "${ENV_FILE}"
|
||||
echo "created ${ENV_FILE}"
|
||||
else
|
||||
echo "keep existing ${ENV_FILE}"
|
||||
fi
|
||||
|
||||
echo "[3/5] reload systemd"
|
||||
systemctl daemon-reload
|
||||
|
||||
echo "[4/5] enable service"
|
||||
systemctl enable "${SERVICE_NAME}" >/dev/null
|
||||
|
||||
echo "[5/5] next steps"
|
||||
echo "project_root=${PROJECT_ROOT}"
|
||||
echo "edit ${ENV_FILE} and set:"
|
||||
echo " OPS_CONTROL_PLANE_BASE_URL"
|
||||
echo " OPS_AGENT_TOKEN"
|
||||
echo " NODE_CODE / NODE_REGION / NODE_ROLE"
|
||||
echo
|
||||
echo "then run:"
|
||||
echo " systemctl restart ${SERVICE_NAME}"
|
||||
echo " systemctl status ${SERVICE_NAME} --no-pager -l"
|
||||
91
domain-api/deploy/multi-region/build_node_agent_bootstrap_plan.sh
Executable file
91
domain-api/deploy/multi-region/build_node_agent_bootstrap_plan.sh
Executable file
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
API_BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
NODE_CODE="${2:-}"
|
||||
NODE_REGION="${3:-mainland}"
|
||||
NODE_ROLE="${4:-worker}"
|
||||
CONTROL_PLANE_BASE_URL="${5:-${API_BASE_URL}}"
|
||||
ROOT_DIR="${6:-/opt/domaincheck}"
|
||||
ISSUED_BY="${ISSUED_BY:-cli-bootstrap}"
|
||||
EXPIRES_IN_HOURS="${EXPIRES_IN_HOURS:-72}"
|
||||
|
||||
if [[ -z "${NODE_CODE}" ]]; then
|
||||
echo "usage: $0 <api_base_url> <node_code> [node_region] [node_role] [control_plane_base_url] [root_dir]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
NORMALIZED_API_BASE_URL="${API_BASE_URL%/}"
|
||||
NORMALIZED_CONTROL_PLANE_BASE_URL="${CONTROL_PLANE_BASE_URL%/}"
|
||||
if [[ "${NORMALIZED_API_BASE_URL}" != */api/v1 ]]; then
|
||||
API_ENDPOINT="${NORMALIZED_API_BASE_URL}/api/v1/ops/agent/bootstrap-plan"
|
||||
else
|
||||
API_ENDPOINT="${NORMALIZED_API_BASE_URL}/ops/agent/bootstrap-plan"
|
||||
fi
|
||||
|
||||
TMP_RESPONSE_FILE="$(mktemp)"
|
||||
cleanup() {
|
||||
rm -f "${TMP_RESPONSE_FILE}"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
python3 - "${NODE_CODE}" "${NODE_REGION}" "${NODE_ROLE}" "${ISSUED_BY}" "${EXPIRES_IN_HOURS}" "${NORMALIZED_CONTROL_PLANE_BASE_URL}" "${ROOT_DIR}" >"${TMP_RESPONSE_FILE}.payload" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = {
|
||||
"node_code": sys.argv[1],
|
||||
"node_region": sys.argv[2],
|
||||
"node_role": sys.argv[3],
|
||||
"issued_by": sys.argv[4],
|
||||
"expires_in_hours": int(sys.argv[5]),
|
||||
"control_plane_base_url": sys.argv[6],
|
||||
"root_dir": sys.argv[7],
|
||||
"metadata": {
|
||||
"issued_from": "build_node_agent_bootstrap_plan.sh",
|
||||
},
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False))
|
||||
PY
|
||||
|
||||
curl -fsS -X POST \
|
||||
-H 'Content-Type: application/json' \
|
||||
--data-binary @"${TMP_RESPONSE_FILE}.payload" \
|
||||
"${API_ENDPOINT}" >"${TMP_RESPONSE_FILE}"
|
||||
|
||||
python3 - "${TMP_RESPONSE_FILE}" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
response = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
|
||||
response_code = response.get("code", 0)
|
||||
if int(response_code if response_code is not None else 0) != 0:
|
||||
raise SystemExit(response.get("message") or "bootstrap plan request failed")
|
||||
|
||||
data = response.get("data") or {}
|
||||
plan = data.get("bootstrap_plan") or {}
|
||||
|
||||
print("[token]")
|
||||
print(f"node_code={data.get('node_code', '')}")
|
||||
print(f"token_preview={data.get('token_preview', '')}")
|
||||
print(f"expires_at={data.get('expires_at', '')}")
|
||||
print()
|
||||
print("[env]")
|
||||
print(plan.get("env_content") or "")
|
||||
print()
|
||||
print("[commands]")
|
||||
print(plan.get("command_block") or "")
|
||||
print()
|
||||
print("[bootstrap-script]")
|
||||
print(plan.get("bootstrap_script_content") or "")
|
||||
print()
|
||||
print("[bootstrap-run-block]")
|
||||
print(plan.get("bootstrap_run_script_block") or "")
|
||||
print()
|
||||
print("[health-checks]")
|
||||
for item in plan.get("health_checks") or []:
|
||||
print(item)
|
||||
PY
|
||||
|
||||
rm -f "${TMP_RESPONSE_FILE}.payload"
|
||||
345
domain-api/deploy/multi-region/check_env_drift.sh
Normal file
345
domain-api/deploy/multi-region/check_env_drift.sh
Normal file
@@ -0,0 +1,345 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
API_SERVICE="${API_SERVICE:-domaincheck-api}"
|
||||
WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}"
|
||||
SYNC_AGENT_SERVICE="${SYNC_AGENT_SERVICE:-domaincheck-sync-agent}"
|
||||
NODE_AGENT_SERVICE="${NODE_AGENT_SERVICE:-domaincheck-node-agent}"
|
||||
API_ENV_FILE="${API_ENV_FILE:-/etc/default/domaincheck-api}"
|
||||
WORKER_ENV_FILE="${WORKER_ENV_FILE:-/etc/default/domaincheck-worker}"
|
||||
NODE_AGENT_ENV_FILE="${NODE_AGENT_ENV_FILE:-/etc/default/domaincheck-node-agent}"
|
||||
PROJECT_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)"
|
||||
API_DIR="${PROJECT_ROOT}/domain-api"
|
||||
WEB_DIR="${PROJECT_ROOT}/domain-web"
|
||||
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||
|
||||
FETCH_BODY=""
|
||||
FETCH_STATUS="000"
|
||||
|
||||
fetch_json() {
|
||||
local url="${1:-}"
|
||||
local raw_text
|
||||
raw_text="$(curl -sS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT:-3}" \
|
||||
--max-time "${CURL_MAX_TIME:-10}" \
|
||||
"${url}" \
|
||||
-w $'\nHTTP_STATUS=%{http_code}' || true)"
|
||||
FETCH_BODY="${raw_text%$'\n'HTTP_STATUS=*}"
|
||||
FETCH_STATUS="${raw_text##*$'\n'HTTP_STATUS=}"
|
||||
}
|
||||
|
||||
echo "[1/8] local command availability"
|
||||
for cmd in bash curl systemctl "${PYTHON_BIN}" node npm; do
|
||||
if command -v "${cmd}" >/dev/null 2>&1; then
|
||||
printf '%s: ok -> %s\n' "${cmd}" "$(command -v "${cmd}")"
|
||||
else
|
||||
printf '%s: missing\n' "${cmd}"
|
||||
fi
|
||||
done
|
||||
echo
|
||||
|
||||
echo "[2/8] python environment"
|
||||
"${PYTHON_BIN}" - <<'PY' "${API_DIR}" "${WEB_DIR}"
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
api_dir = pathlib.Path(sys.argv[1])
|
||||
web_dir = pathlib.Path(sys.argv[2])
|
||||
|
||||
def has_module(name: str) -> bool:
|
||||
return importlib.util.find_spec(name) is not None
|
||||
|
||||
payload = {
|
||||
"python_executable": sys.executable,
|
||||
"python_version": sys.version.split()[0],
|
||||
"pytest_installed": has_module("pytest"),
|
||||
"uvicorn_installed": has_module("uvicorn"),
|
||||
"fastapi_installed": has_module("fastapi"),
|
||||
"api_requirements_exists": (api_dir / "requirements.txt").exists(),
|
||||
"web_package_json_exists": (web_dir / "package.json").exists(),
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
|
||||
echo "[3/8] env files"
|
||||
for env_file in "${API_ENV_FILE}" "${WORKER_ENV_FILE}" "${NODE_AGENT_ENV_FILE}"; do
|
||||
if [[ -f "${env_file}" ]]; then
|
||||
echo "exists: ${env_file}"
|
||||
else
|
||||
echo "missing: ${env_file}"
|
||||
fi
|
||||
done
|
||||
echo
|
||||
|
||||
echo "[4/8] systemd service states"
|
||||
for service in "${API_SERVICE}" "${WORKER_SERVICE}" "${SYNC_AGENT_SERVICE}" "${NODE_AGENT_SERVICE}"; do
|
||||
if systemctl list-unit-files "${service}.service" --no-pager >/dev/null 2>&1; then
|
||||
printf '%s: enabled=%s active=%s\n' \
|
||||
"${service}" \
|
||||
"$(systemctl is-enabled "${service}" 2>/dev/null || true)" \
|
||||
"$(systemctl is-active "${service}" 2>/dev/null || true)"
|
||||
else
|
||||
printf '%s: not-installed\n' "${service}"
|
||||
fi
|
||||
done
|
||||
echo
|
||||
|
||||
echo "[5/8] runtime preflight"
|
||||
fetch_json "${BASE_URL}/api/v1/runtime/preflight"
|
||||
printf 'status=%s\n%s\n\n' "${FETCH_STATUS}" "${FETCH_BODY}"
|
||||
|
||||
echo "[6/8] runtime readiness"
|
||||
fetch_json "${BASE_URL}/api/v1/runtime/readiness"
|
||||
printf 'status=%s\n%s\n\n' "${FETCH_STATUS}" "${FETCH_BODY}"
|
||||
|
||||
echo "[7/8] runtime build-info"
|
||||
fetch_json "${BASE_URL}/api/v1/runtime/build-info"
|
||||
printf 'status=%s\n%s\n\n' "${FETCH_STATUS}" "${FETCH_BODY}"
|
||||
|
||||
echo "[8/8] condensed env audit summary"
|
||||
"${PYTHON_BIN}" - <<'PY' \
|
||||
"${BASE_URL}" \
|
||||
"${API_SERVICE}" \
|
||||
"${WORKER_SERVICE}" \
|
||||
"${SYNC_AGENT_SERVICE}" \
|
||||
"${NODE_AGENT_SERVICE}" \
|
||||
"${API_ENV_FILE}" \
|
||||
"${WORKER_ENV_FILE}" \
|
||||
"${NODE_AGENT_ENV_FILE}" \
|
||||
"${PROJECT_ROOT}"
|
||||
import importlib.util
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.request
|
||||
from typing import Dict, List
|
||||
|
||||
base_url, api_service, worker_service, sync_agent_service, node_agent_service, api_env, worker_env, node_agent_env, project_root = sys.argv[1:10]
|
||||
project_root_path = pathlib.Path(project_root)
|
||||
|
||||
def has_module(name: str) -> bool:
|
||||
return importlib.util.find_spec(name) is not None
|
||||
|
||||
def inspect_python_runtime(path: str) -> Dict[str, object]:
|
||||
runtime_path = pathlib.Path(path)
|
||||
if not runtime_path.exists():
|
||||
return {}
|
||||
probe = r"""
|
||||
import importlib.util
|
||||
import json
|
||||
import sys
|
||||
|
||||
mods = ["pytest", "fastapi", "uvicorn"]
|
||||
print(json.dumps({
|
||||
"python_executable": sys.executable,
|
||||
"python_version": sys.version.split()[0],
|
||||
"modules": {name: importlib.util.find_spec(name) is not None for name in mods},
|
||||
}, ensure_ascii=False))
|
||||
"""
|
||||
try:
|
||||
raw = subprocess.check_output([str(runtime_path), "-c", probe], text=True, stderr=subprocess.DEVNULL)
|
||||
except Exception:
|
||||
return {}
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
except Exception:
|
||||
return {}
|
||||
payload["path"] = str(runtime_path)
|
||||
return payload
|
||||
|
||||
def systemd_state(service: str) -> dict:
|
||||
try:
|
||||
enabled = subprocess.check_output(
|
||||
["systemctl", "is-enabled", service],
|
||||
text=True,
|
||||
stderr=subprocess.DEVNULL,
|
||||
).strip()
|
||||
except Exception:
|
||||
enabled = ""
|
||||
try:
|
||||
active = subprocess.check_output(
|
||||
["systemctl", "is-active", service],
|
||||
text=True,
|
||||
stderr=subprocess.DEVNULL,
|
||||
).strip()
|
||||
except Exception:
|
||||
active = ""
|
||||
return {"enabled": enabled, "active": active}
|
||||
|
||||
def fetch_json(url: str) -> tuple[str, dict]:
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=10) as resp:
|
||||
status = str(resp.status)
|
||||
raw = resp.read().decode("utf-8", errors="replace")
|
||||
except Exception:
|
||||
return "000", {}
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
except Exception:
|
||||
return status, {}
|
||||
return status, payload.get("data") if isinstance(payload, dict) else {}
|
||||
|
||||
def read_git_value(*args: str) -> str:
|
||||
try:
|
||||
result = subprocess.check_output(
|
||||
["git", "-C", str(project_root_path), *args],
|
||||
text=True,
|
||||
stderr=subprocess.DEVNULL,
|
||||
).strip()
|
||||
except Exception:
|
||||
return ""
|
||||
return result
|
||||
|
||||
def read_text(path: pathlib.Path) -> str:
|
||||
try:
|
||||
return path.read_text(encoding="utf-8", errors="replace")
|
||||
except Exception:
|
||||
return ""
|
||||
|
||||
local_commit_sha = read_git_value("rev-parse", "--short=12", "HEAD")
|
||||
local_commit_ref = read_git_value("rev-parse", "--abbrev-ref", "HEAD")
|
||||
ops_agent_service_text = read_text(project_root_path / "domain-api" / "app" / "services" / "ops_agent_service.py")
|
||||
bootstrap_node_agent_text = read_text(project_root_path / "domain-api" / "deploy" / "multi-region" / "bootstrap_node_agent.sh")
|
||||
|
||||
repo_supports_install_command_block = (
|
||||
"install_command_block" in ops_agent_service_text
|
||||
and "_candidate_node_agent_install_paths" in ops_agent_service_text
|
||||
)
|
||||
repo_supports_multi_layout_bootstrap = "resolve_project_root" in bootstrap_node_agent_text
|
||||
|
||||
preflight_status, preflight = fetch_json(f"{base_url}/api/v1/runtime/preflight")
|
||||
readiness_status, readiness = fetch_json(f"{base_url}/api/v1/runtime/readiness")
|
||||
build_status, build = fetch_json(f"{base_url}/api/v1/runtime/build-info")
|
||||
route_surface = build.get("route_surface") if isinstance(build, dict) else {}
|
||||
|
||||
missing = []
|
||||
tooling_items: List[str] = []
|
||||
for env_file in [api_env, worker_env, node_agent_env]:
|
||||
if not pathlib.Path(env_file).exists():
|
||||
missing.append(f"missing_env:{env_file}")
|
||||
|
||||
python_candidates: List[str] = []
|
||||
seen_candidates = set()
|
||||
for candidate in [
|
||||
"/opt/domaincheck/domainCheck/.venv/bin/python",
|
||||
str(pathlib.Path.cwd() / ".venv" / "bin" / "python"),
|
||||
str(pathlib.Path.cwd() / "domain-api" / ".venv" / "bin" / "python"),
|
||||
pathlib.Path(sys.executable).as_posix(),
|
||||
]:
|
||||
if candidate and candidate not in seen_candidates:
|
||||
seen_candidates.add(candidate)
|
||||
python_candidates.append(candidate)
|
||||
|
||||
python_runtimes = [item for item in (inspect_python_runtime(candidate) for candidate in python_candidates) if item]
|
||||
|
||||
def runtime_has_module(module_name: str) -> bool:
|
||||
for runtime in python_runtimes:
|
||||
modules = runtime.get("modules") or {}
|
||||
if bool(modules.get(module_name)):
|
||||
return True
|
||||
return False
|
||||
|
||||
if not runtime_has_module("fastapi"):
|
||||
missing.append("python_module:fastapi")
|
||||
if not runtime_has_module("uvicorn"):
|
||||
missing.append("python_module:uvicorn")
|
||||
if not runtime_has_module("pytest"):
|
||||
tooling_items.append("python_module:pytest")
|
||||
|
||||
runtime_commit_sha = str(build.get("commit_sha") or "").strip() if isinstance(build, dict) else ""
|
||||
runtime_commit_ref = str(build.get("commit_ref") or "").strip() if isinstance(build, dict) else ""
|
||||
runtime_route_missing_keys = [str(item).strip() for item in list((route_surface or {}).get("missing_keys") or []) if str(item).strip()]
|
||||
runtime_expected_paths = {
|
||||
str(key).strip(): str(value).strip()
|
||||
for key, value in dict((route_surface or {}).get("expected_paths") or {}).items()
|
||||
if str(key).strip()
|
||||
}
|
||||
runtime_bootstrap_plan_route_missing = "ops_node_handover_bootstrap_plan" in runtime_route_missing_keys
|
||||
runtime_declares_bootstrap_plan_route = "ops_node_handover_bootstrap_plan" in runtime_expected_paths
|
||||
runtime_repo_commit_drift = bool(local_commit_sha and runtime_commit_sha and local_commit_sha != runtime_commit_sha)
|
||||
runtime_repo_capability_drift = bool(
|
||||
repo_supports_install_command_block
|
||||
and (runtime_bootstrap_plan_route_missing or not runtime_declares_bootstrap_plan_route)
|
||||
)
|
||||
|
||||
recommended_actions: List[str] = []
|
||||
if runtime_repo_capability_drift:
|
||||
recommended_actions.append("systemctl restart domaincheck-api")
|
||||
recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary")
|
||||
recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-plan http://127.0.0.1:8100 mainland-worker-01")
|
||||
elif runtime_repo_commit_drift:
|
||||
recommended_actions.append("systemctl restart domaincheck-api")
|
||||
recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100")
|
||||
|
||||
payload = {
|
||||
"status": "ready",
|
||||
"headline": "当前环境没有发现新的基础阻断,可继续做上线复核。",
|
||||
"repo": {
|
||||
"project_root": str(project_root_path),
|
||||
"commit_sha": local_commit_sha,
|
||||
"commit_ref": local_commit_ref,
|
||||
"supports_install_command_block": repo_supports_install_command_block,
|
||||
"supports_multi_layout_bootstrap": repo_supports_multi_layout_bootstrap,
|
||||
},
|
||||
"python": {
|
||||
"pytest_installed": runtime_has_module("pytest"),
|
||||
"fastapi_installed": runtime_has_module("fastapi"),
|
||||
"uvicorn_installed": runtime_has_module("uvicorn"),
|
||||
"runtimes": python_runtimes,
|
||||
},
|
||||
"services": {
|
||||
api_service: systemd_state(api_service),
|
||||
worker_service: systemd_state(worker_service),
|
||||
sync_agent_service: systemd_state(sync_agent_service),
|
||||
node_agent_service: systemd_state(node_agent_service),
|
||||
},
|
||||
"runtime": {
|
||||
"preflight_status_code": preflight_status,
|
||||
"preflight_ok": bool((preflight or {}).get("ok", False)),
|
||||
"readiness_status_code": readiness_status,
|
||||
"readiness_status": str((readiness or {}).get("status") or ""),
|
||||
"readiness_summary": str((readiness or {}).get("summary") or ""),
|
||||
"build_status_code": build_status,
|
||||
"build_commit_sha": runtime_commit_sha,
|
||||
"build_commit_ref": runtime_commit_ref,
|
||||
"route_surface_complete": bool((route_surface or {}).get("surface_complete", False)),
|
||||
"route_surface_missing_keys": runtime_route_missing_keys,
|
||||
"route_surface_declares_bootstrap_plan": runtime_declares_bootstrap_plan_route,
|
||||
"repo_commit_drift": runtime_repo_commit_drift,
|
||||
"repo_capability_drift": runtime_repo_capability_drift,
|
||||
"runtime_may_need_restart": runtime_repo_commit_drift or runtime_repo_capability_drift,
|
||||
},
|
||||
"missing_items": missing,
|
||||
"tooling_items": tooling_items,
|
||||
"recommended_actions": recommended_actions,
|
||||
}
|
||||
|
||||
if not bool((preflight or {}).get("ok", False)):
|
||||
payload["status"] = "blocked"
|
||||
payload["headline"] = "runtime/preflight 未通过,先修环境与依赖,再继续上线复核。"
|
||||
elif str((readiness or {}).get("status") or "") == "blocking":
|
||||
payload["status"] = "blocked"
|
||||
payload["headline"] = "runtime/readiness 当前为 blocking,现场仍不适合进入正式发布。"
|
||||
elif runtime_repo_capability_drift:
|
||||
payload["status"] = "attention"
|
||||
payload["headline"] = "本地仓库已经具备新的 Node Agent 接管能力,但运行中的 API 路由面仍旧,当前更像是服务未重启到最新代码。"
|
||||
elif runtime_repo_commit_drift:
|
||||
payload["status"] = "attention"
|
||||
payload["headline"] = "本地仓库 commit 与运行中 API build-info 不一致,建议先重启或重发当前版本,再继续上线复核。"
|
||||
elif not bool((route_surface or {}).get("surface_complete", False)):
|
||||
payload["status"] = "attention"
|
||||
payload["headline"] = "运行中 API 路由面还不完整,建议先确认当前实例是否已经吃到最新版本。"
|
||||
elif missing:
|
||||
payload["status"] = "attention"
|
||||
payload["headline"] = "基础环境存在缺口,虽然未必阻断运行,但建议先补齐后再交付。"
|
||||
elif tooling_items:
|
||||
payload["status"] = "ready"
|
||||
payload["headline"] = "运行环境可继续上线复核,但本机工具链仍有缺口,建议后续补齐。"
|
||||
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
423
domain-api/deploy/multi-region/check_go_live.sh
Normal file
423
domain-api/deploy/multi-region/check_go_live.sh
Normal file
@@ -0,0 +1,423 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
OVERSEAS_BASE_URL="${2:-}"
|
||||
VIEW_MODE="${3:-full}"
|
||||
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
|
||||
CURL_MAX_TIME="${CURL_MAX_TIME:-20}"
|
||||
LOCAL_NODE_AGENT_SERVICE_NAME="${LOCAL_NODE_AGENT_SERVICE_NAME:-domaincheck-node-agent}"
|
||||
LOCAL_NODE_AGENT_ENV_FILE="${LOCAL_NODE_AGENT_ENV_FILE:-/etc/default/domaincheck-node-agent}"
|
||||
|
||||
FETCH_BODY=""
|
||||
FETCH_STATUS="000"
|
||||
|
||||
fetch_json() {
|
||||
local url="${1:-}"
|
||||
local raw_text
|
||||
raw_text="$(curl -sS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
"${url}" \
|
||||
-w $'\nHTTP_STATUS=%{http_code}' || true)"
|
||||
FETCH_BODY="${raw_text%$'\n'HTTP_STATUS=*}"
|
||||
FETCH_STATUS="${raw_text##*$'\n'HTTP_STATUS=}"
|
||||
}
|
||||
|
||||
run_local_node_agent_probe() {
|
||||
SERVICE_NAME="${LOCAL_NODE_AGENT_SERVICE_NAME}" \
|
||||
ENV_FILE="${LOCAL_NODE_AGENT_ENV_FILE}" \
|
||||
bash "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/check_node_agent.sh"
|
||||
}
|
||||
|
||||
fetch_json "${BASE_URL}/api/v1/ops/stack-diagnosis?base_url=${BASE_URL}"
|
||||
STACK_JSON="${FETCH_BODY}"
|
||||
STACK_STATUS="${FETCH_STATUS}"
|
||||
|
||||
fetch_json "${BASE_URL}/api/v1/ops/go-live-summary?base_url=${BASE_URL}"
|
||||
GO_LIVE_API_JSON="${FETCH_BODY}"
|
||||
GO_LIVE_API_STATUS="${FETCH_STATUS}"
|
||||
|
||||
if [[ "${GO_LIVE_API_STATUS}" == 2* ]]; then
|
||||
if [[ "${VIEW_MODE}" != "summary" ]]; then
|
||||
echo "[1/1] go-live summary (api)"
|
||||
fi
|
||||
printf '%s\n' "${GO_LIVE_API_JSON}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
fetch_json "${BASE_URL}/api/v1/ops/contracts"
|
||||
CONTRACTS_JSON="${FETCH_BODY}"
|
||||
CONTRACTS_STATUS="${FETCH_STATUS}"
|
||||
|
||||
fetch_json "${BASE_URL}/api/v1/ops/releases/launchpad"
|
||||
LAUNCHPAD_JSON="${FETCH_BODY}"
|
||||
LAUNCHPAD_STATUS="${FETCH_STATUS}"
|
||||
|
||||
fetch_json "${BASE_URL}/api/v1/runtime/build-info"
|
||||
BUILD_INFO_JSON="${FETCH_BODY}"
|
||||
BUILD_INFO_STATUS="${FETCH_STATUS}"
|
||||
|
||||
fetch_json "${BASE_URL}/api/v1/ops/nodes"
|
||||
NODES_JSON="${FETCH_BODY}"
|
||||
NODES_STATUS="${FETCH_STATUS}"
|
||||
|
||||
fetch_json "${BASE_URL}/api/v1/ops/overview"
|
||||
OVERVIEW_JSON="${FETCH_BODY}"
|
||||
OVERVIEW_STATUS="${FETCH_STATUS}"
|
||||
|
||||
LOCAL_NODE_AGENT_RAW="$(run_local_node_agent_probe 2>&1 || true)"
|
||||
|
||||
if [[ "${VIEW_MODE}" != "summary" ]]; then
|
||||
echo "[1/7] ops stack diagnosis"
|
||||
printf 'status=%s\n%s\n\n' "${STACK_STATUS}" "${STACK_JSON}"
|
||||
|
||||
echo "[2/7] ops contracts"
|
||||
printf 'status=%s\n%s\n\n' "${CONTRACTS_STATUS}" "${CONTRACTS_JSON}"
|
||||
|
||||
echo "[3/7] release launchpad"
|
||||
printf 'status=%s\n%s\n\n' "${LAUNCHPAD_STATUS}" "${LAUNCHPAD_JSON}"
|
||||
|
||||
echo "[4/7] runtime build info"
|
||||
printf 'status=%s\n%s\n\n' "${BUILD_INFO_STATUS}" "${BUILD_INFO_JSON}"
|
||||
|
||||
echo "[5/7] managed nodes"
|
||||
printf 'status=%s\n%s\n\n' "${NODES_STATUS}" "${NODES_JSON}"
|
||||
|
||||
echo "[6/7] ops overview"
|
||||
printf 'status=%s\n%s\n\n' "${OVERVIEW_STATUS}" "${OVERVIEW_JSON}"
|
||||
|
||||
echo "[7/7] local node agent probe"
|
||||
printf '%s\n\n' "${LOCAL_NODE_AGENT_RAW}"
|
||||
fi
|
||||
|
||||
echo "[8/8] go-live summary"
|
||||
python3 - <<'PY' \
|
||||
"${BASE_URL}" \
|
||||
"${OVERSEAS_BASE_URL}" \
|
||||
"${STACK_JSON}" \
|
||||
"${STACK_STATUS}" \
|
||||
"${CONTRACTS_JSON}" \
|
||||
"${CONTRACTS_STATUS}" \
|
||||
"${LAUNCHPAD_JSON}" \
|
||||
"${LAUNCHPAD_STATUS}" \
|
||||
"${BUILD_INFO_JSON}" \
|
||||
"${BUILD_INFO_STATUS}" \
|
||||
"${NODES_JSON}" \
|
||||
"${NODES_STATUS}" \
|
||||
"${OVERVIEW_JSON}" \
|
||||
"${OVERVIEW_STATUS}" \
|
||||
"${LOCAL_NODE_AGENT_RAW}" \
|
||||
"${LOCAL_NODE_AGENT_SERVICE_NAME}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
|
||||
def load_payload(raw_text: str) -> dict:
|
||||
try:
|
||||
payload = json.loads(str(raw_text or ""))
|
||||
except Exception:
|
||||
return {}
|
||||
return payload if isinstance(payload, dict) else {}
|
||||
|
||||
|
||||
def load_data(raw_text: str) -> dict:
|
||||
payload = load_payload(raw_text)
|
||||
data = payload.get("data")
|
||||
return data if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def ensure_dict(value):
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def ensure_list(value):
|
||||
return value if isinstance(value, list) else []
|
||||
|
||||
|
||||
def ensure_str(value):
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def truthy_http(status_code: str) -> bool:
|
||||
return str(status_code or "").startswith("2")
|
||||
|
||||
|
||||
base_url = str(sys.argv[1] or "http://127.0.0.1:8100").strip()
|
||||
overseas_base_url = str(sys.argv[2] or "").strip()
|
||||
stack_payload = load_data(sys.argv[3])
|
||||
stack_status_code = str(sys.argv[4] or "000")
|
||||
contracts_payload = load_data(sys.argv[5])
|
||||
contracts_status_code = str(sys.argv[6] or "000")
|
||||
launchpad_payload = load_data(sys.argv[7])
|
||||
launchpad_status_code = str(sys.argv[8] or "000")
|
||||
build_payload = load_data(sys.argv[9])
|
||||
build_status_code = str(sys.argv[10] or "000")
|
||||
nodes_payload = load_data(sys.argv[11])
|
||||
nodes_status_code = str(sys.argv[12] or "000")
|
||||
overview_payload = load_data(sys.argv[13])
|
||||
overview_status_code = str(sys.argv[14] or "000")
|
||||
local_node_agent_raw = str(sys.argv[15] or "")
|
||||
local_node_agent_service_name = str(sys.argv[16] or "domaincheck-node-agent").strip()
|
||||
|
||||
diagnosis = ensure_dict(stack_payload.get("diagnosis"))
|
||||
release_hub = ensure_dict(stack_payload.get("release_hub"))
|
||||
managed_nodes = ensure_dict(stack_payload.get("managed_nodes"))
|
||||
runtime_build_info = ensure_dict(stack_payload.get("runtime_build_info")) or build_payload
|
||||
launchpad_status = ensure_dict(launchpad_payload.get("launchpad_status"))
|
||||
launchpad_recommended_target_node_code = ensure_str(launchpad_status.get("recommended_target_node_code"))
|
||||
launchpad_recommended_recovery_label = ensure_str(launchpad_status.get("recommended_recovery_label"))
|
||||
launchpad_recommended_recovery_summary = ensure_str(launchpad_status.get("recommended_recovery_summary"))
|
||||
launchpad_onboarding_bootstrap_pending_nodes = int(
|
||||
launchpad_status.get("onboarding_bootstrap_pending_nodes", 0) or 0
|
||||
)
|
||||
launchpad_onboarding_acceptance_ready_nodes = int(
|
||||
launchpad_status.get("onboarding_acceptance_ready_nodes", 0) or 0
|
||||
)
|
||||
contracts = ensure_list(contracts_payload.get("contracts"))
|
||||
nodes_summary = ensure_dict(nodes_payload.get("summary")) or ensure_dict(managed_nodes.get("summary"))
|
||||
route_surface = ensure_dict(runtime_build_info.get("route_surface"))
|
||||
next_step = ensure_dict(diagnosis.get("next_step"))
|
||||
operator_decision = ensure_dict(diagnosis.get("operator_decision"))
|
||||
overview_recommendation = ensure_dict(overview_payload.get("recommendation"))
|
||||
execution_scene = ensure_dict(overview_payload.get("execution_scene"))
|
||||
log_sync = ensure_dict(execution_scene.get("log_sync"))
|
||||
|
||||
local_agent_enabled = ""
|
||||
local_agent_active = ""
|
||||
local_agent_installed = False
|
||||
for line in [item.strip() for item in local_node_agent_raw.splitlines() if item.strip()]:
|
||||
if line == "not-installed":
|
||||
local_agent_installed = False
|
||||
if line == "enabled":
|
||||
local_agent_enabled = "enabled"
|
||||
local_agent_installed = True
|
||||
elif line == "disabled":
|
||||
local_agent_enabled = "disabled"
|
||||
local_agent_installed = True
|
||||
elif line in {"active", "inactive", "failed"}:
|
||||
local_agent_active = line
|
||||
local_agent_installed = True
|
||||
|
||||
blocking_reasons = []
|
||||
warnings = []
|
||||
publish_blocking_reasons = []
|
||||
publish_warnings = []
|
||||
|
||||
stack_status = ensure_str(diagnosis.get("stack_status")) or "unknown"
|
||||
if stack_status == "blocked":
|
||||
blocking_reasons.append("stack_diagnosis=blocked")
|
||||
publish_blocking_reasons.append("stack_diagnosis=blocked")
|
||||
elif stack_status == "attention":
|
||||
warnings.append("stack_diagnosis=attention")
|
||||
publish_warnings.append("stack_diagnosis=attention")
|
||||
elif not truthy_http(stack_status_code):
|
||||
blocking_reasons.append("stack_diagnosis_unavailable")
|
||||
publish_blocking_reasons.append("stack_diagnosis_unavailable")
|
||||
|
||||
surface_complete = bool(route_surface.get("surface_complete", False))
|
||||
if not surface_complete:
|
||||
missing_keys = [ensure_str(item) for item in ensure_list(route_surface.get("missing_keys")) if ensure_str(item)]
|
||||
blocking_reasons.append(
|
||||
"route_surface_incomplete" + (f":{','.join(missing_keys)}" if missing_keys else "")
|
||||
)
|
||||
publish_blocking_reasons.append(
|
||||
"route_surface_incomplete" + (f":{','.join(missing_keys)}" if missing_keys else "")
|
||||
)
|
||||
|
||||
if not truthy_http(contracts_status_code) or len(contracts) <= 0:
|
||||
blocking_reasons.append("contracts_unavailable")
|
||||
publish_blocking_reasons.append("contracts_unavailable")
|
||||
|
||||
launchpad_status_code_value = ensure_str(launchpad_status.get("status")) or ensure_str(release_hub.get("launchpad_status")) or ""
|
||||
if launchpad_status_code_value == "blocked":
|
||||
blocking_reasons.append("release_launchpad=blocked")
|
||||
publish_blocking_reasons.append("release_launchpad=blocked")
|
||||
elif launchpad_status_code_value == "attention":
|
||||
warnings.append("release_launchpad=attention")
|
||||
publish_warnings.append("release_launchpad=attention")
|
||||
elif not truthy_http(launchpad_status_code):
|
||||
blocking_reasons.append("release_launchpad_unavailable")
|
||||
publish_blocking_reasons.append("release_launchpad_unavailable")
|
||||
|
||||
if launchpad_onboarding_bootstrap_pending_nodes > 0:
|
||||
warnings.append(f"launchpad_onboarding_bootstrap_pending={launchpad_onboarding_bootstrap_pending_nodes}")
|
||||
publish_warnings.append(f"launchpad_onboarding_bootstrap_pending={launchpad_onboarding_bootstrap_pending_nodes}")
|
||||
if launchpad_onboarding_acceptance_ready_nodes > 0:
|
||||
warnings.append(f"launchpad_onboarding_acceptance_ready={launchpad_onboarding_acceptance_ready_nodes}")
|
||||
publish_warnings.append(f"launchpad_onboarding_acceptance_ready={launchpad_onboarding_acceptance_ready_nodes}")
|
||||
|
||||
managed_enabled = int(nodes_summary.get("managed_enabled", 0) or 0)
|
||||
remote_access_ready = int(nodes_summary.get("remote_access_ready", 0) or 0)
|
||||
queue_dead_letter_nodes = int(nodes_summary.get("queue_dead_letter_nodes", 0) or 0)
|
||||
if managed_enabled > 0 and remote_access_ready == 0:
|
||||
blocking_reasons.append("remote_agent_ready=0")
|
||||
publish_blocking_reasons.append("remote_agent_ready=0")
|
||||
elif managed_enabled > 0 and remote_access_ready < managed_enabled:
|
||||
warnings.append(f"remote_agent_ready={remote_access_ready}/{managed_enabled}")
|
||||
publish_warnings.append(f"remote_agent_ready={remote_access_ready}/{managed_enabled}")
|
||||
|
||||
if queue_dead_letter_nodes > 0:
|
||||
warnings.append(f"queue_dead_letter_nodes={queue_dead_letter_nodes}")
|
||||
publish_warnings.append(f"queue_dead_letter_nodes={queue_dead_letter_nodes}")
|
||||
|
||||
if local_agent_installed and local_agent_active not in {"", "active"}:
|
||||
warnings.append(f"{local_node_agent_service_name}={local_agent_active}")
|
||||
|
||||
log_sync_enabled = bool(log_sync.get("enabled"))
|
||||
log_sync_state = ensure_str(log_sync.get("state"))
|
||||
log_sync_mode = ensure_str(log_sync.get("mode"))
|
||||
participating_nodes_total = int(log_sync.get("participating_node_count", 0) or 0)
|
||||
covered_participating_nodes = int(log_sync.get("covered_participating_node_count", 0) or 0)
|
||||
missing_participating_node_codes = [
|
||||
ensure_str(item)
|
||||
for item in ensure_list(log_sync.get("missing_participating_nodes"))
|
||||
if ensure_str(item)
|
||||
]
|
||||
if log_sync_enabled and participating_nodes_total > 0 and covered_participating_nodes == 0:
|
||||
warnings.append(f"log_sync_waiting_sample=0/{participating_nodes_total}")
|
||||
publish_warnings.append(f"log_sync_waiting_sample=0/{participating_nodes_total}")
|
||||
elif log_sync_enabled and participating_nodes_total > 0 and covered_participating_nodes < participating_nodes_total:
|
||||
warnings.append(f"log_sync_partial={covered_participating_nodes}/{participating_nodes_total}")
|
||||
publish_warnings.append(f"log_sync_partial={covered_participating_nodes}/{participating_nodes_total}")
|
||||
|
||||
operator_lane = ensure_str(operator_decision.get("lane"))
|
||||
operator_title = ensure_str(operator_decision.get("title"))
|
||||
operator_primary_command_key = ensure_str(operator_decision.get("primary_command_key"))
|
||||
if not operator_lane:
|
||||
if blocking_reasons:
|
||||
operator_lane = "recovery"
|
||||
elif warnings:
|
||||
operator_lane = "verification"
|
||||
else:
|
||||
operator_lane = "go-live"
|
||||
if not operator_title:
|
||||
operator_title = (
|
||||
ensure_str(overview_recommendation.get("title"))
|
||||
or ensure_str(launchpad_status.get("status_label"))
|
||||
or ("先处理阻塞项" if blocking_reasons else "可以进入上线复核")
|
||||
)
|
||||
|
||||
next_step_action_code = (
|
||||
ensure_str(next_step.get("action_code"))
|
||||
or ensure_str(overview_recommendation.get("primary_action_code"))
|
||||
or ensure_str(launchpad_status.get("recommended_action_code"))
|
||||
)
|
||||
next_step_reason = (
|
||||
ensure_str(next_step.get("reason"))
|
||||
or ensure_str(overview_recommendation.get("reason"))
|
||||
or "来自 launchpad/overview fallback"
|
||||
)
|
||||
if not operator_primary_command_key:
|
||||
operator_primary_command_key = next_step_action_code
|
||||
|
||||
go_live_status = "ready"
|
||||
if blocking_reasons:
|
||||
go_live_status = "blocked"
|
||||
elif warnings:
|
||||
go_live_status = "attention"
|
||||
|
||||
publish_status = "ready"
|
||||
if publish_blocking_reasons:
|
||||
publish_status = "blocked"
|
||||
elif publish_warnings:
|
||||
publish_status = "attention"
|
||||
publish_ready = publish_status == "ready"
|
||||
if publish_status == "ready":
|
||||
publish_status_label = "可发布"
|
||||
elif publish_status == "attention":
|
||||
publish_status_label = "可发布但建议先复核"
|
||||
else:
|
||||
publish_status_label = "暂不可发布"
|
||||
|
||||
action_commands = {
|
||||
"handover_first_gap": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} handover_first_gap",
|
||||
"view_first_gap": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} view_first_gap",
|
||||
"bootstrap_run": (
|
||||
f"bash domain-api/deploy/multi-region/drive_ops_center.sh node-bootstrap-run {base_url} {launchpad_recommended_target_node_code} cli"
|
||||
if launchpad_recommended_target_node_code
|
||||
else f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} bootstrap_run"
|
||||
),
|
||||
"run_acceptance": (
|
||||
f"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run {base_url} {launchpad_recommended_target_node_code} cli"
|
||||
if launchpad_recommended_target_node_code
|
||||
else f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} run_acceptance"
|
||||
),
|
||||
"fix_managed_nodes": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} fix_managed_nodes",
|
||||
"open_worker_logs_participating": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} open_worker_logs_participating",
|
||||
"run_inspection_participating": f"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve {base_url} run_inspection_participating",
|
||||
}
|
||||
|
||||
recommended_commands = {
|
||||
"stack_summary": f"bash domain-api/deploy/multi-region/check_ops_center_stack.sh {base_url} summary",
|
||||
"contracts": f"bash domain-api/deploy/multi-region/check_ops_contracts.sh {base_url}",
|
||||
"ops_plane": f"bash domain-api/deploy/multi-region/check_ops_plane.sh {base_url}",
|
||||
"release_hub": f"bash domain-api/deploy/multi-region/check_release_hub.sh {base_url}",
|
||||
"inspection": f"bash domain-api/deploy/multi-region/check_ops_inspection.sh {base_url}",
|
||||
"node_agent": "bash domain-api/deploy/multi-region/check_node_agent.sh",
|
||||
"overview": f"bash domain-api/deploy/multi-region/drive_ops_center.sh overview {base_url}",
|
||||
"doctor_export": (
|
||||
f"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live {base_url} {overseas_base_url}"
|
||||
if overseas_base_url
|
||||
else f"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live {base_url}"
|
||||
),
|
||||
}
|
||||
if next_step_action_code and next_step_action_code in action_commands:
|
||||
recommended_commands["next_step"] = action_commands[next_step_action_code]
|
||||
if log_sync_enabled:
|
||||
if "open_worker_logs_participating" in action_commands:
|
||||
recommended_commands["log_sync_logs"] = action_commands["open_worker_logs_participating"]
|
||||
if "run_inspection_participating" in action_commands:
|
||||
recommended_commands["log_sync_inspection"] = action_commands["run_inspection_participating"]
|
||||
|
||||
summary = {
|
||||
"base_url": base_url,
|
||||
"overseas_base_url": overseas_base_url,
|
||||
"go_live_status": go_live_status,
|
||||
"publish_ready": publish_ready,
|
||||
"publish_status": publish_status,
|
||||
"publish_status_label": publish_status_label,
|
||||
"stack_status": stack_status,
|
||||
"contracts_ready": truthy_http(contracts_status_code) and len(contracts) > 0,
|
||||
"contracts_total": len(contracts),
|
||||
"launchpad_status": launchpad_status_code_value,
|
||||
"launchpad_status_label": ensure_str(launchpad_status.get("status_label")),
|
||||
"launchpad_recommended_action_code": ensure_str(launchpad_status.get("recommended_action_code")),
|
||||
"launchpad_recommended_target_node_code": launchpad_recommended_target_node_code,
|
||||
"launchpad_recommended_recovery_label": launchpad_recommended_recovery_label,
|
||||
"launchpad_recommended_recovery_summary": launchpad_recommended_recovery_summary,
|
||||
"launchpad_onboarding_bootstrap_pending_nodes": launchpad_onboarding_bootstrap_pending_nodes,
|
||||
"launchpad_onboarding_acceptance_ready_nodes": launchpad_onboarding_acceptance_ready_nodes,
|
||||
"route_surface_complete": surface_complete,
|
||||
"route_surface_missing_keys": [
|
||||
ensure_str(item) for item in ensure_list(route_surface.get("missing_keys")) if ensure_str(item)
|
||||
],
|
||||
"managed_enabled": managed_enabled,
|
||||
"remote_access_ready": remote_access_ready,
|
||||
"queue_dead_letter_nodes": queue_dead_letter_nodes,
|
||||
"participating_nodes_total": participating_nodes_total,
|
||||
"log_sync_enabled": log_sync_enabled,
|
||||
"log_sync_state": log_sync_state,
|
||||
"log_sync_mode": log_sync_mode,
|
||||
"log_sync_covered_nodes": covered_participating_nodes,
|
||||
"log_sync_missing_node_codes": missing_participating_node_codes,
|
||||
"next_step_action_code": next_step_action_code,
|
||||
"next_step_reason": next_step_reason,
|
||||
"operator_lane": operator_lane,
|
||||
"operator_title": operator_title,
|
||||
"operator_primary_command_key": operator_primary_command_key,
|
||||
"local_node_agent": {
|
||||
"service_name": local_node_agent_service_name,
|
||||
"installed": local_agent_installed,
|
||||
"enabled": local_agent_enabled,
|
||||
"active": local_agent_active,
|
||||
},
|
||||
"blocking_reasons": blocking_reasons,
|
||||
"warnings": warnings,
|
||||
"publish_blocking_reasons": publish_blocking_reasons,
|
||||
"publish_warnings": publish_warnings,
|
||||
"recommended_commands": recommended_commands,
|
||||
}
|
||||
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
182
domain-api/deploy/multi-region/check_go_live_signoff.sh
Normal file
182
domain-api/deploy/multi-region/check_go_live_signoff.sh
Normal file
@@ -0,0 +1,182 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${SCRIPT_DIR}/lib_ops_center.sh"
|
||||
ops_center_load_config "${SCRIPT_DIR}"
|
||||
|
||||
TARGET_OR_BASE_URL="${1:-}"
|
||||
SECOND_ARG="${2:-}"
|
||||
|
||||
REPORT_DIR=""
|
||||
MANIFEST_PATH=""
|
||||
GENERATED_BUNDLE="false"
|
||||
MAINLAND_BASE_URL=""
|
||||
OVERSEAS_BASE_URL=""
|
||||
|
||||
if [[ -n "${TARGET_OR_BASE_URL}" && ( -f "${TARGET_OR_BASE_URL}" || -d "${TARGET_OR_BASE_URL}" ) ]]; then
|
||||
if [[ -d "${TARGET_OR_BASE_URL}" ]]; then
|
||||
REPORT_DIR="${TARGET_OR_BASE_URL%/}"
|
||||
MANIFEST_PATH="${REPORT_DIR}/manifest.json"
|
||||
else
|
||||
MANIFEST_PATH="${TARGET_OR_BASE_URL}"
|
||||
REPORT_DIR="$(cd "$(dirname "${MANIFEST_PATH}")" && pwd)"
|
||||
fi
|
||||
else
|
||||
MAINLAND_BASE_URL="${TARGET_OR_BASE_URL:-$(ops_center_resolve_mainland_api_base_url)}"
|
||||
OVERSEAS_BASE_URL="$(normalize_optional_base_url_arg "${SECOND_ARG:-}" "$(ops_center_resolve_overseas_api_base_url)")"
|
||||
REPORT_DIR="${OPS_CENTER_REPORT_ROOT}/go-live-bundles/go-live-signoff-$(timestamp_now)"
|
||||
ops_center_ensure_report_dir "${REPORT_DIR}"
|
||||
bash "${SCRIPT_DIR}/export_go_live_bundle.sh" "${REPORT_DIR}" "${MAINLAND_BASE_URL}" "${OVERSEAS_BASE_URL}" >/dev/null
|
||||
MANIFEST_PATH="${REPORT_DIR}/manifest.json"
|
||||
GENERATED_BUNDLE="true"
|
||||
fi
|
||||
|
||||
if [[ ! -f "${MANIFEST_PATH}" ]]; then
|
||||
echo "manifest not found: ${MANIFEST_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
REVIEW_JSON="$(bash "${SCRIPT_DIR}/review_go_live_bundle.sh" "${MANIFEST_PATH}")"
|
||||
DOCTOR_JSON="$(bash "${SCRIPT_DIR}/drive_ops_center.sh" doctor-decision "${MANIFEST_PATH}")"
|
||||
|
||||
python3 - <<'PY' \
|
||||
"${REVIEW_JSON}" \
|
||||
"${DOCTOR_JSON}" \
|
||||
"${GENERATED_BUNDLE}" \
|
||||
"${REPORT_DIR}" \
|
||||
"${MANIFEST_PATH}" \
|
||||
"${MAINLAND_BASE_URL}" \
|
||||
"${OVERSEAS_BASE_URL}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
review = json.loads(sys.argv[1])
|
||||
doctor = json.loads(sys.argv[2])
|
||||
generated_bundle = str(sys.argv[3]).strip().lower() == "true"
|
||||
report_dir = str(sys.argv[4] or "").strip()
|
||||
manifest_path = str(sys.argv[5] or "").strip()
|
||||
mainland_base_url = str(sys.argv[6] or "").strip()
|
||||
overseas_base_url = str(sys.argv[7] or "").strip()
|
||||
|
||||
go_live_summary = dict(review.get("go_live_summary") or {})
|
||||
stack_diagnosis = dict(review.get("stack_diagnosis") or {})
|
||||
driver_feed = dict(review.get("driver_feed") or {})
|
||||
codex_brief = dict(review.get("codex_brief") or {})
|
||||
launchpad_alignment = dict(review.get("launchpad_alignment") or {})
|
||||
doctor_decision = dict(doctor.get("decision") or {})
|
||||
doctor_summary = dict(doctor.get("summary") or {})
|
||||
|
||||
review_status = str(review.get("status") or "").strip()
|
||||
doctor_status = str(doctor_decision.get("status") or "").strip()
|
||||
go_live_status = str(go_live_summary.get("go_live_status") or "").strip()
|
||||
publish_status = str(go_live_summary.get("publish_status") or "").strip()
|
||||
stack_status = str(stack_diagnosis.get("stack_status") or "").strip()
|
||||
driver_launch_status = str(driver_feed.get("launch_status") or "").strip()
|
||||
codex_launch_status = str(codex_brief.get("launch_status") or "").strip()
|
||||
|
||||
|
||||
def dedupe_commands(values):
|
||||
result = []
|
||||
seen = set()
|
||||
for item in values:
|
||||
text = str(item or "").strip()
|
||||
if not text or text in seen:
|
||||
continue
|
||||
seen.add(text)
|
||||
result.append(text)
|
||||
return result
|
||||
|
||||
|
||||
blocked_reasons = dedupe_commands(
|
||||
[]
|
||||
+ ([f"review:{review_status}"] if review_status == "blocked" else [])
|
||||
+ ([f"doctor:{doctor_status}"] if doctor_status == "blocked" else [])
|
||||
+ ([f"go_live:{go_live_status}"] if go_live_status == "blocked" else [])
|
||||
+ ([f"publish:{publish_status}"] if publish_status == "blocked" else [])
|
||||
+ ([f"stack:{stack_status}"] if stack_status == "blocked" else [])
|
||||
+ ([f"driver_launch:{driver_launch_status}"] if driver_launch_status == "blocked" else [])
|
||||
+ ([f"codex_launch:{codex_launch_status}"] if codex_launch_status == "blocked" else [])
|
||||
)
|
||||
|
||||
attention_reasons = dedupe_commands(
|
||||
[]
|
||||
+ ([f"review:{review_status}"] if review_status == "attention" else [])
|
||||
+ ([f"doctor:{doctor_status}"] if doctor_status == "attention" else [])
|
||||
+ ([f"go_live:{go_live_status}"] if go_live_status == "attention" else [])
|
||||
+ ([f"publish:{publish_status}"] if publish_status == "attention" else [])
|
||||
+ ([f"stack:{stack_status}"] if stack_status == "attention" else [])
|
||||
+ ([f"driver_launch:{driver_launch_status}"] if driver_launch_status == "attention" else [])
|
||||
+ ([f"codex_launch:{codex_launch_status}"] if codex_launch_status == "attention" else [])
|
||||
+ (["launchpad_alignment:inconsistent"] if not bool(launchpad_alignment.get("consistent", False)) else [])
|
||||
)
|
||||
|
||||
if blocked_reasons:
|
||||
signoff_status = "blocked"
|
||||
headline = (
|
||||
str(review.get("headline") or "").strip()
|
||||
or str(doctor_decision.get("headline") or "").strip()
|
||||
or "当前发布前验证仍存在阻断项,不应直接签字上线。"
|
||||
)
|
||||
elif attention_reasons:
|
||||
signoff_status = "attention"
|
||||
headline = (
|
||||
str(doctor_decision.get("headline") or "").strip()
|
||||
or str(review.get("headline") or "").strip()
|
||||
or "当前现场已进入收口区间,但仍建议先完成 attention 项再正式签字。"
|
||||
)
|
||||
else:
|
||||
signoff_status = "ready"
|
||||
headline = "当前证据包、doctor 结论与发布门禁一致,可进入最终人工签字或正式发布。"
|
||||
|
||||
recommended_commands = dedupe_commands(
|
||||
[]
|
||||
+ list(doctor_decision.get("recommended_commands") or [])
|
||||
+ [
|
||||
f"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review {manifest_path}",
|
||||
f"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision {manifest_path}",
|
||||
]
|
||||
)
|
||||
|
||||
payload = {
|
||||
"signoff_status": signoff_status,
|
||||
"headline": headline,
|
||||
"generated_bundle": generated_bundle,
|
||||
"report_dir": report_dir or str(doctor.get("report_dir") or review.get("report_dir") or ""),
|
||||
"manifest_path": manifest_path or str(doctor.get("manifest_path") or ""),
|
||||
"base_urls": {
|
||||
"mainland": mainland_base_url,
|
||||
"overseas": overseas_base_url,
|
||||
},
|
||||
"release_gate": {
|
||||
"go_live_status": go_live_status,
|
||||
"publish_status": publish_status,
|
||||
"stack_status": stack_status,
|
||||
"driver_launch_status": driver_launch_status,
|
||||
"codex_launch_status": codex_launch_status,
|
||||
"bundle_review_status": review_status,
|
||||
"doctor_status": doctor_status,
|
||||
},
|
||||
"blocked_reasons": blocked_reasons,
|
||||
"attention_reasons": attention_reasons,
|
||||
"decision": {
|
||||
"operator_title": str(go_live_summary.get("operator_title") or "").strip(),
|
||||
"next_step_action_code": str(go_live_summary.get("next_step_action_code") or "").strip(),
|
||||
"doctor_reason_code": str(doctor_decision.get("reason_code") or "").strip(),
|
||||
"doctor_preferred_surface": str(doctor_decision.get("preferred_surface") or "").strip(),
|
||||
"launchpad_recommended_target_node_code": str(go_live_summary.get("launchpad_recommended_target_node_code") or "").strip(),
|
||||
"launchpad_recommended_recovery_label": str(go_live_summary.get("launchpad_recommended_recovery_label") or "").strip(),
|
||||
"launchpad_recommended_recovery_summary": str(go_live_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
|
||||
},
|
||||
"launchpad_alignment": launchpad_alignment,
|
||||
"recommended_commands": recommended_commands,
|
||||
"bundle_review": review,
|
||||
"doctor_decision": {
|
||||
"summary": doctor_summary,
|
||||
"decision": doctor_decision,
|
||||
},
|
||||
}
|
||||
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
38
domain-api/deploy/multi-region/check_node_agent.sh
Executable file
38
domain-api/deploy/multi-region/check_node_agent.sh
Executable file
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SERVICE_NAME="${SERVICE_NAME:-domaincheck-node-agent}"
|
||||
ENV_FILE="${ENV_FILE:-/etc/default/domaincheck-node-agent}"
|
||||
|
||||
echo "[1/4] environment file"
|
||||
if [[ -f "${ENV_FILE}" ]]; then
|
||||
echo "env file: ${ENV_FILE}"
|
||||
else
|
||||
echo "missing env file: ${ENV_FILE}"
|
||||
fi
|
||||
echo
|
||||
|
||||
echo "[2/4] key env summary"
|
||||
if [[ -f "${ENV_FILE}" ]]; then
|
||||
grep -E '^(OPS_CONTROL_PLANE_BASE_URL|NODE_CODE|NODE_REGION|NODE_ROLE|OPS_AGENT_POLL_INTERVAL_SECONDS|WORKER_SERVICE_NAME|API_SERVICE_NAME|SYNC_AGENT_SERVICE_NAME)=' "${ENV_FILE}" || true
|
||||
else
|
||||
echo "skip"
|
||||
fi
|
||||
echo
|
||||
|
||||
echo "[3/4] systemd status"
|
||||
if systemctl list-unit-files "${SERVICE_NAME}.service" --no-pager >/dev/null 2>&1; then
|
||||
systemctl is-enabled "${SERVICE_NAME}" || true
|
||||
systemctl is-active "${SERVICE_NAME}" || true
|
||||
else
|
||||
echo "not-installed"
|
||||
echo "inactive"
|
||||
fi
|
||||
echo
|
||||
|
||||
echo "[4/4] service detail"
|
||||
if systemctl list-unit-files "${SERVICE_NAME}.service" --no-pager >/dev/null 2>&1; then
|
||||
systemctl status "${SERVICE_NAME}" --no-pager -l | sed -n '1,25p' || true
|
||||
else
|
||||
echo "service not found: ${SERVICE_NAME}"
|
||||
fi
|
||||
65
domain-api/deploy/multi-region/check_node_scene_log.sh
Normal file
65
domain-api/deploy/multi-region/check_node_scene_log.sh
Normal file
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
NODE_CODE="${2:-}"
|
||||
LIMIT="${3:-80}"
|
||||
MODE="${4:-key}"
|
||||
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
|
||||
CURL_MAX_TIME="${CURL_MAX_TIME:-8}"
|
||||
|
||||
if [[ -z "${NODE_CODE}" ]]; then
|
||||
echo "usage: bash domain-api/deploy/multi-region/check_node_scene_log.sh [base_url] <node_code> [limit] [key|full]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RAW_RESPONSE="$(curl -sS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
"${BASE_URL}/api/v1/ops/nodes/${NODE_CODE}/scene-log?limit=${LIMIT}&mode=${MODE}")"
|
||||
|
||||
echo "[1/2] raw response"
|
||||
printf '%s\n\n' "${RAW_RESPONSE}"
|
||||
|
||||
echo "[2/2] condensed scene-log summary"
|
||||
python3 - "${RAW_RESPONSE}" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
raw = str(sys.argv[1] or "")
|
||||
payload = json.loads(raw)
|
||||
data = dict(payload.get("data") or {})
|
||||
participation = dict(data.get("participation") or {})
|
||||
source_summary = dict(data.get("source_summary") or {})
|
||||
records = list(data.get("records") or [])
|
||||
latest = dict(data.get("latest_record") or {})
|
||||
|
||||
summary = {
|
||||
"node_code": str(data.get("node_code") or ""),
|
||||
"status": str(data.get("status") or ""),
|
||||
"status_label": str(data.get("status_label") or ""),
|
||||
"mode": str(data.get("mode") or ""),
|
||||
"summary": str(data.get("summary") or ""),
|
||||
"log_sync_enabled": bool(data.get("log_sync_enabled", False)),
|
||||
"records_total": int(data.get("records_total", len(records)) or 0),
|
||||
"missing_reason_code": str(data.get("missing_reason_code") or ""),
|
||||
"participation": {
|
||||
"detect_participating": bool(participation.get("detect_participating", False)),
|
||||
"participation_state": str(participation.get("participation_state") or ""),
|
||||
"participation_label": str(participation.get("participation_label") or ""),
|
||||
"participation_reason": str(participation.get("participation_reason") or ""),
|
||||
"current_load": int(participation.get("current_load", 0) or 0),
|
||||
},
|
||||
"source_summary": {
|
||||
"line_count": int(source_summary.get("line_count", 0) or 0),
|
||||
"key_line_count": int(source_summary.get("key_line_count", 0) or 0),
|
||||
"full_line_count": int(source_summary.get("full_line_count", 0) or 0),
|
||||
"last_at": str(source_summary.get("last_at") or ""),
|
||||
"last_line": str(source_summary.get("last_line") or ""),
|
||||
},
|
||||
"latest_record": latest,
|
||||
"sample_messages": [str(item.get("message") or "") for item in records[:8]],
|
||||
}
|
||||
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
1272
domain-api/deploy/multi-region/check_ops_center_stack.sh
Normal file
1272
domain-api/deploy/multi-region/check_ops_center_stack.sh
Normal file
File diff suppressed because it is too large
Load Diff
55
domain-api/deploy/multi-region/check_ops_contracts.sh
Executable file
55
domain-api/deploy/multi-region/check_ops_contracts.sh
Executable file
@@ -0,0 +1,55 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
|
||||
contracts_json="$(curl -fsS "${BASE_URL}/api/v1/ops/contracts")"
|
||||
|
||||
echo "[1/3] ops contracts registry"
|
||||
printf '%s\n\n' "${contracts_json}"
|
||||
|
||||
echo "[2/3] condensed summary"
|
||||
python3 - <<'PY' "${contracts_json}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
data = payload.get("data") or {}
|
||||
contracts = list(data.get("contracts") or [])
|
||||
|
||||
summary = {
|
||||
"registry_version": str(data.get("registry_version") or ""),
|
||||
"schema_version": str(data.get("schema_version") or ""),
|
||||
"contracts_total": len(contracts),
|
||||
"contract_keys": [str(item.get("key") or "") for item in contracts if str(item.get("key") or "")],
|
||||
"docs_root": str(data.get("docs_root") or ""),
|
||||
"discovery_entrypoints": list(data.get("discovery_entrypoints") or []),
|
||||
}
|
||||
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
|
||||
echo "[3/3] contract rows"
|
||||
python3 - <<'PY' "${contracts_json}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
data = payload.get("data") or {}
|
||||
contracts = list(data.get("contracts") or [])
|
||||
|
||||
rows = [
|
||||
{
|
||||
"key": str(item.get("key") or ""),
|
||||
"version": str(item.get("version") or ""),
|
||||
"status": str(item.get("status") or ""),
|
||||
"primary_endpoint": str(item.get("primary_endpoint") or ""),
|
||||
"schema_doc_path": str(item.get("schema_doc_path") or ""),
|
||||
"consumers": list(item.get("consumers") or []),
|
||||
}
|
||||
for item in contracts
|
||||
]
|
||||
|
||||
print(json.dumps(rows, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
72
domain-api/deploy/multi-region/check_ops_inspection.sh
Executable file
72
domain-api/deploy/multi-region/check_ops_inspection.sh
Executable file
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
|
||||
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
|
||||
|
||||
curl_json() {
|
||||
curl -fsS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
"$1"
|
||||
}
|
||||
|
||||
INSPECTION_JSON="$(curl_json "${BASE_URL}/api/v1/ops/inspection-overview?limit=120")"
|
||||
|
||||
echo "[1/5] inspection overview"
|
||||
printf '%s\n\n' "${INSPECTION_JSON}"
|
||||
|
||||
echo "[2/5] condensed summary"
|
||||
python3 - <<'PY' "${INSPECTION_JSON}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1]).get("data") or {}
|
||||
print(json.dumps({
|
||||
"rows_total": int(payload.get("rows_total") or 0),
|
||||
"filtered_rows_total": int(payload.get("filtered_rows_total") or 0),
|
||||
"visible_nodes_total": int(payload.get("visible_nodes_total") or 0),
|
||||
"eligible_nodes_total": int(payload.get("eligible_nodes_total") or 0),
|
||||
"handover_gap_nodes_total": int(payload.get("handover_gap_nodes_total") or 0),
|
||||
"participating_nodes_total": int(payload.get("participating_nodes_total") or 0),
|
||||
"summary": dict(payload.get("summary") or {}),
|
||||
"available_problem_kind_counts": dict(payload.get("available_problem_kind_counts") or {}),
|
||||
"filters": dict(payload.get("filters") or {}),
|
||||
}, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
|
||||
echo "[3/5] priority queue"
|
||||
python3 - <<'PY' "${INSPECTION_JSON}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1]).get("data") or {}
|
||||
rows = list(payload.get("priority_queue") or [])
|
||||
print(json.dumps({
|
||||
"total": len(rows),
|
||||
"items": [
|
||||
{
|
||||
"node_code": str(item.get("node_code") or ""),
|
||||
"overall_status": str(item.get("overall_status") or ""),
|
||||
"problem_kind": str(item.get("problem_kind") or ""),
|
||||
"problem_label": str(item.get("problem_label") or ""),
|
||||
"recommended_action_code": str(item.get("recommended_action_code") or ""),
|
||||
"last_updated_at": str(item.get("last_updated_at") or ""),
|
||||
"issue_summary": str(item.get("issue_summary") or ""),
|
||||
}
|
||||
for item in rows[:12]
|
||||
],
|
||||
}, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
|
||||
echo "[4/5] attention only"
|
||||
curl_json "${BASE_URL}/api/v1/ops/inspection-overview?limit=80&status=attention&only_problem=true"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[5/5] participating problems only"
|
||||
curl_json "${BASE_URL}/api/v1/ops/inspection-overview?limit=80&only_problem=true&only_participating=true"
|
||||
echo
|
||||
35
domain-api/deploy/multi-region/check_ops_link_snapshot.sh
Normal file
35
domain-api/deploy/multi-region/check_ops_link_snapshot.sh
Normal file
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||
|
||||
echo "[1/2] ops link snapshot"
|
||||
SNAPSHOT_JSON="$(curl -fsS "${BASE_URL}/api/v1/ops/link-snapshot")"
|
||||
echo "${SNAPSHOT_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[2/2] condensed summary"
|
||||
"${PYTHON_BIN}" - <<'PY' "$SNAPSHOT_JSON"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1]).get("data", {})
|
||||
print(f"status: {payload.get('status', '-')}")
|
||||
print(f"headline: {payload.get('headline', '-')}")
|
||||
print()
|
||||
print("summary:")
|
||||
for line in payload.get("summary_lines", []) or []:
|
||||
print(f"- {line}")
|
||||
|
||||
actions = payload.get("next_actions", []) or []
|
||||
if actions:
|
||||
print()
|
||||
print("next_actions:")
|
||||
for item in actions:
|
||||
label = str(item.get("label", "") or "").strip() or "-"
|
||||
action_code = str(item.get("action_code", "") or "").strip() or "-"
|
||||
node_codes = "、".join(str(x).strip() for x in (item.get("node_codes", []) or []) if str(x).strip()) or "-"
|
||||
print(f"- {label} / {action_code} / nodes: {node_codes}")
|
||||
PY
|
||||
609
domain-api/deploy/multi-region/check_ops_plane.sh
Normal file
609
domain-api/deploy/multi-region/check_ops_plane.sh
Normal file
@@ -0,0 +1,609 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
|
||||
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
|
||||
|
||||
curl_json() {
|
||||
curl -fsS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
"$1"
|
||||
}
|
||||
|
||||
curl_post_json() {
|
||||
local url="$1"
|
||||
local body="$2"
|
||||
curl -fsS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
-X POST "${url}" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "${body}"
|
||||
}
|
||||
|
||||
OVERVIEW_JSON="$(curl_json "${BASE_URL}/api/v1/ops/overview")"
|
||||
NODES_JSON="$(curl_json "${BASE_URL}/api/v1/ops/nodes")"
|
||||
JOBS_JSON="$(curl_json "${BASE_URL}/api/v1/ops/jobs")"
|
||||
ACTIVITY_JSON="$(curl_json "${BASE_URL}/api/v1/ops/activity-stream")"
|
||||
LAUNCHPAD_JSON="$(curl_json "${BASE_URL}/api/v1/ops/releases/launchpad")"
|
||||
LATEST_RELEASE_JSON="$(curl_json "${BASE_URL}/api/v1/ops/releases/latest")"
|
||||
POLICY_PREVIEW_JSON="$(
|
||||
curl_post_json "${BASE_URL}/api/v1/ops/policy/preview" \
|
||||
'{"action":"runtime.restart_api","target_type":"node","target_node_code":"mainland-controller-01"}'
|
||||
)"
|
||||
|
||||
echo "[1/12] ops overview"
|
||||
printf '%s\n\n' "${OVERVIEW_JSON}"
|
||||
|
||||
echo "[2/12] top recommendation"
|
||||
python3 - <<'PY' "${OVERVIEW_JSON}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
data = payload.get("data") or {}
|
||||
recommendation = dict(data.get("recommendation") or {})
|
||||
print(json.dumps({
|
||||
"source": str(recommendation.get("source") or ""),
|
||||
"key": str(recommendation.get("key") or ""),
|
||||
"priority": str(recommendation.get("priority") or ""),
|
||||
"summary": str(recommendation.get("summary") or ""),
|
||||
"primary_action_code": str(recommendation.get("primary_action_code") or ""),
|
||||
"secondary_action_code": str(recommendation.get("secondary_action_code") or ""),
|
||||
"node_codes": list(recommendation.get("node_codes") or []),
|
||||
}, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
|
||||
echo "[3/12] driver recommendations (condensed)"
|
||||
python3 - <<'PY' "${OVERVIEW_JSON}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
data = payload.get("data") or {}
|
||||
recommendations = list(data.get("driver_recommendations") or [])
|
||||
summary = {
|
||||
"total": len(recommendations),
|
||||
"items": [
|
||||
{
|
||||
"key": str(item.get("key") or ""),
|
||||
"title": str(item.get("title") or ""),
|
||||
"level_label": str(item.get("level_label") or ""),
|
||||
"primary_action_code": str(item.get("primary_action_code") or ""),
|
||||
"secondary_action_code": str(item.get("secondary_action_code") or ""),
|
||||
"meta_text": str(item.get("meta_text") or ""),
|
||||
}
|
||||
for item in recommendations[:8]
|
||||
],
|
||||
}
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
|
||||
echo "[4/12] ops capabilities"
|
||||
curl_json "${BASE_URL}/api/v1/ops/capabilities"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[5/12] ops blueprint"
|
||||
curl_json "${BASE_URL}/api/v1/ops/blueprint"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[6/12] ops runbook"
|
||||
curl_json "${BASE_URL}/api/v1/ops/runbook"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[7/12] ops nodes"
|
||||
printf '%s\n' "${NODES_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[8/12] ops jobs"
|
||||
printf '%s\n' "${JOBS_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[9/12] ops activity stream"
|
||||
printf '%s\n' "${ACTIVITY_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[10/12] release launchpad"
|
||||
printf '%s\n' "${LAUNCHPAD_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[11/12] latest release"
|
||||
printf '%s\n' "${LATEST_RELEASE_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[12/12] policy preview sample"
|
||||
printf '%s\n' "${POLICY_PREVIEW_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[13/13] condensed summary"
|
||||
python3 - <<'PY' \
|
||||
"${BASE_URL}" \
|
||||
"${OVERVIEW_JSON}" \
|
||||
"${NODES_JSON}" \
|
||||
"${JOBS_JSON}" \
|
||||
"${ACTIVITY_JSON}" \
|
||||
"${LAUNCHPAD_JSON}" \
|
||||
"${LATEST_RELEASE_JSON}" \
|
||||
"${POLICY_PREVIEW_JSON}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
|
||||
def as_dict(payload):
|
||||
try:
|
||||
raw = json.loads(payload)
|
||||
except Exception:
|
||||
return {}
|
||||
return raw.get("data") or {}
|
||||
|
||||
|
||||
def ensure_dict(value):
|
||||
return value if isinstance(value, dict) else {}
|
||||
|
||||
|
||||
def ensure_list(value):
|
||||
return value if isinstance(value, list) else []
|
||||
|
||||
|
||||
def ensure_str(value):
|
||||
return value if isinstance(value, str) else ""
|
||||
|
||||
|
||||
def command(name, *parts):
|
||||
tokens = [
|
||||
"bash",
|
||||
"domain-api/deploy/multi-region/drive_ops_center.sh",
|
||||
name,
|
||||
base_url,
|
||||
]
|
||||
tokens.extend([str(part) for part in parts if str(part or "").strip()])
|
||||
return " ".join(tokens)
|
||||
|
||||
|
||||
def release_command(name, *parts):
|
||||
tokens = [
|
||||
"bash",
|
||||
"domain-api/deploy/multi-region/drive_release_hub.sh",
|
||||
base_url,
|
||||
name,
|
||||
]
|
||||
tokens.extend([str(part) for part in parts if str(part or "").strip()])
|
||||
return " ".join(tokens)
|
||||
|
||||
|
||||
def build_operator_decision():
|
||||
remote_access_state_counts = ensure_dict(node_summary.get("remote_access_state_counts"))
|
||||
pending_bootstrap = int(node_summary.get("pending_bootstrap", 0) or 0)
|
||||
runtime_only = int(node_summary.get("runtime_only", 0) or 0)
|
||||
online_total = int(node_summary.get("online", 0) or 0)
|
||||
participating_total = int(scene_log.get("participating_node_count", 0) or 0)
|
||||
launchpad_blocked = bool(launchpad_workers.get("blocked")) or bool(launchpad_controls.get("blocked"))
|
||||
running_total = int(jobs_summary.get("running_count", 0) or 0)
|
||||
queued_total = int(jobs_summary.get("queued_count", 0) or 0)
|
||||
|
||||
if scene_log.get("status") == "disabled" and participating_total > 0:
|
||||
return {
|
||||
"lane": "observability",
|
||||
"priority": "high",
|
||||
"reason_code": "scene_log_disabled",
|
||||
"title": "先恢复远端现场日志可见性",
|
||||
"summary": ensure_str(scene_log.get("summary")),
|
||||
"next_focus": ensure_dict(scene_log.get("focus_ref")),
|
||||
"primary_command_key": "scene_log",
|
||||
"secondary_command_key": "driver_focus_preview",
|
||||
}
|
||||
|
||||
if pending_bootstrap > 0 or remote_access_state_counts:
|
||||
return {
|
||||
"lane": "node_handover",
|
||||
"priority": "high",
|
||||
"reason_code": "managed_nodes_not_ready",
|
||||
"title": "先补齐节点接管与远程可达性",
|
||||
"summary": (
|
||||
f"当前 managed node 共 {int(node_summary.get('managed_total', 0) or 0)} 台,"
|
||||
f"pending_bootstrap={pending_bootstrap},runtime_only={runtime_only}。"
|
||||
),
|
||||
"next_focus": {
|
||||
"kind": "managed_node_handover",
|
||||
"node_code": first_attention_node_code,
|
||||
} if first_attention_node_code else {},
|
||||
"primary_command_key": "node_handover",
|
||||
"secondary_command_key": "node_bootstrap_plan",
|
||||
}
|
||||
|
||||
if launchpad_blocked or str(launchpad_status.get("status") or "") == "attention":
|
||||
return {
|
||||
"lane": "release",
|
||||
"priority": "medium",
|
||||
"reason_code": "release_launchpad_attention",
|
||||
"title": "检查发布驾驶舱与 rollout 阻断",
|
||||
"summary": ensure_str(launchpad_status.get("summary")),
|
||||
"next_focus": release_focus_ref,
|
||||
"primary_command_key": "release_launchpad",
|
||||
"secondary_command_key": "release_preview_latest",
|
||||
}
|
||||
|
||||
if running_total > 0 or queued_total > 0:
|
||||
return {
|
||||
"lane": "ops_jobs",
|
||||
"priority": "medium",
|
||||
"reason_code": "jobs_active",
|
||||
"title": "继续盯运行中任务与活动流",
|
||||
"summary": (
|
||||
f"当前 running={running_total},queued={queued_total},"
|
||||
f"activity_status={ensure_str(activity_summary.get('status')) or 'unknown'}。"
|
||||
),
|
||||
"next_focus": {
|
||||
"kind": "ops_jobs",
|
||||
"section": "activity_stream",
|
||||
},
|
||||
"primary_command_key": "jobs",
|
||||
"secondary_command_key": "activity_preview",
|
||||
}
|
||||
|
||||
return {
|
||||
"lane": "steady",
|
||||
"priority": "low",
|
||||
"reason_code": "steady_state",
|
||||
"title": "当前进入稳定观察态",
|
||||
"summary": (
|
||||
f"online={online_total},attention_nodes={len(attention_nodes)},"
|
||||
f"scene_log_status={ensure_str(scene_log.get('status')) or 'standby'}。"
|
||||
),
|
||||
"next_focus": recommendation_focus_ref or ensure_dict(scene_log.get("focus_ref")),
|
||||
"primary_command_key": "driver_feed",
|
||||
"secondary_command_key": "activity_preview",
|
||||
}
|
||||
|
||||
|
||||
def normalize_scene_log_observation(overview_data):
|
||||
driver_feed = ensure_dict(overview_data.get("driver_feed"))
|
||||
scene_log_data = ensure_dict(driver_feed.get("scene_log_observation"))
|
||||
if scene_log_data:
|
||||
normalized = dict(scene_log_data)
|
||||
normalized["status"] = ensure_str(normalized.get("status")) or "standby"
|
||||
normalized["status_label"] = ensure_str(normalized.get("status_label")) or "待命"
|
||||
normalized["summary"] = ensure_str(normalized.get("summary")) or "当前现场日志观察暂无额外结论。"
|
||||
normalized["focus_ref"] = ensure_dict(normalized.get("focus_ref"))
|
||||
normalized["target_node_code"] = ensure_str(normalized.get("target_node_code"))
|
||||
normalized["participating_node_count"] = int(normalized.get("participating_node_count", 0) or 0)
|
||||
normalized["covered_participating_node_count"] = int(normalized.get("covered_participating_node_count", 0) or 0)
|
||||
normalized["line_count"] = int(normalized.get("line_count", 0) or 0)
|
||||
return normalized
|
||||
|
||||
execution_scene = ensure_dict(overview_data.get("execution_scene"))
|
||||
log_sync = ensure_dict(execution_scene.get("log_sync"))
|
||||
participation = ensure_dict(execution_scene.get("participation_summary"))
|
||||
missing_nodes = ensure_list(log_sync.get("missing_participating_nodes"))
|
||||
participating_nodes = ensure_list(execution_scene.get("participating_nodes"))
|
||||
target_node_code = ""
|
||||
if missing_nodes:
|
||||
target_node_code = ensure_str(missing_nodes[0])
|
||||
elif participating_nodes:
|
||||
target_node_code = ensure_str(ensure_dict(participating_nodes[0]).get("node_code"))
|
||||
|
||||
participating_total = int(
|
||||
log_sync.get("participating_node_count", participation.get("participating_nodes", len(participating_nodes)))
|
||||
or participation.get("participating_nodes", len(participating_nodes))
|
||||
or len(participating_nodes)
|
||||
or 0
|
||||
)
|
||||
covered_total = int(log_sync.get("covered_participating_node_count", 0) or 0)
|
||||
line_count = int(log_sync.get("line_count", 0) or 0)
|
||||
enabled = bool(log_sync.get("enabled", False))
|
||||
mode = ensure_str(log_sync.get("mode")) or "key"
|
||||
status = "standby"
|
||||
status_label = "待命"
|
||||
summary = "当前没有参与检测节点,现场日志观察处于待命状态。"
|
||||
if participating_total > 0 and not enabled:
|
||||
status = "disabled"
|
||||
status_label = "未开启"
|
||||
summary = (
|
||||
f"当前有 {participating_total} 台参与节点,但远端日志回传仍关闭,"
|
||||
"海外控制面还看不到节点级现场日志。"
|
||||
)
|
||||
elif participating_total > 0 and (
|
||||
line_count <= 0 or covered_total <= 0 or int(log_sync.get("missing_participating_node_count", len(missing_nodes)) or len(missing_nodes)) > 0
|
||||
):
|
||||
status = "waiting_sample"
|
||||
status_label = "等待样本"
|
||||
summary = (
|
||||
f"远端日志回传已经开启,但参与节点只覆盖 {covered_total}/{participating_total},"
|
||||
f"样本 {line_count} 条,仍需继续观察或下钻节点现场日志。"
|
||||
)
|
||||
elif participating_total > 0:
|
||||
status = "ready"
|
||||
status_label = "可下钻"
|
||||
summary = (
|
||||
f"远端日志回传已经形成样本,当前覆盖 {covered_total}/{participating_total} 台参与节点,"
|
||||
"可以直接下钻节点现场日志。"
|
||||
)
|
||||
|
||||
focus_ref = {}
|
||||
if target_node_code:
|
||||
focus_ref = {
|
||||
"kind": "node_scene_log",
|
||||
"node_code": target_node_code,
|
||||
"mode": mode,
|
||||
"limit": 120 if status == "waiting_sample" else 80,
|
||||
}
|
||||
return {
|
||||
"status": status,
|
||||
"status_label": status_label,
|
||||
"summary": summary,
|
||||
"focus_ref": focus_ref,
|
||||
"target_node_code": target_node_code,
|
||||
"participating_node_count": participating_total,
|
||||
"covered_participating_node_count": covered_total,
|
||||
"line_count": line_count,
|
||||
}
|
||||
|
||||
|
||||
base_url = ensure_str(sys.argv[1]) or "http://127.0.0.1:8100"
|
||||
overview = as_dict(sys.argv[2])
|
||||
nodes = as_dict(sys.argv[3])
|
||||
jobs = as_dict(sys.argv[4])
|
||||
activity = as_dict(sys.argv[5])
|
||||
launchpad = as_dict(sys.argv[6])
|
||||
latest_release = as_dict(sys.argv[7])
|
||||
policy_preview = as_dict(sys.argv[8])
|
||||
|
||||
recommendation = ensure_dict(overview.get("recommendation"))
|
||||
scene_log = normalize_scene_log_observation(overview)
|
||||
managed_nodes = ensure_list(nodes.get("managed_nodes"))
|
||||
node_summary = ensure_dict(nodes.get("summary"))
|
||||
jobs_summary = ensure_dict(jobs.get("summary"))
|
||||
activity_summary = ensure_dict(activity.get("summary"))
|
||||
launchpad_release = ensure_dict(launchpad.get("release"))
|
||||
launchpad_rec = ensure_dict(launchpad.get("top_recommendation"))
|
||||
launchpad_workers = ensure_dict(launchpad.get("worker_rollout_preview"))
|
||||
launchpad_controls = ensure_dict(launchpad.get("control_rollout_preview"))
|
||||
launchpad_status = ensure_dict(launchpad.get("launchpad_status"))
|
||||
latest_release_entry = latest_release if isinstance(latest_release, dict) else {}
|
||||
latest_package_preview = ensure_dict(launchpad.get("latest_package_preview"))
|
||||
|
||||
online_nodes = [
|
||||
node for node in managed_nodes
|
||||
if str(node.get("status") or "").lower() in {"online", "busy"}
|
||||
]
|
||||
attention_nodes = [
|
||||
{
|
||||
"node_code": str(node.get("node_code") or ""),
|
||||
"status": str(node.get("status") or ""),
|
||||
"role": str(node.get("role") or ""),
|
||||
"stack_status_label": str(node.get("stack_status_label") or ""),
|
||||
"recommended_action_code": str(node.get("recommended_action_code") or ""),
|
||||
"recommended_action_label": str(node.get("recommended_action_label") or ""),
|
||||
"primary_issue": str(node.get("primary_issue") or ""),
|
||||
}
|
||||
for node in managed_nodes
|
||||
if str(node.get("status") or "").lower() not in {"online", "busy"}
|
||||
or str(node.get("stack_status") or "").lower() not in {"healthy", "ready", ""}
|
||||
]
|
||||
|
||||
activity_items = ensure_list(activity.get("items"))
|
||||
recent_activity = [
|
||||
{
|
||||
"kind": str(item.get("kind") or ""),
|
||||
"title": str(item.get("title") or ""),
|
||||
"level": str(item.get("level") or ""),
|
||||
"created_at": str(item.get("created_at") or ""),
|
||||
}
|
||||
for item in activity_items[:5]
|
||||
]
|
||||
|
||||
job_items = ensure_list(jobs.get("items"))
|
||||
running_jobs = [
|
||||
{
|
||||
"job_key": str(item.get("job_key") or ""),
|
||||
"status": str(item.get("status") or ""),
|
||||
"title": str(item.get("title") or ""),
|
||||
"progress_text": str(item.get("progress_text") or ""),
|
||||
}
|
||||
for item in job_items
|
||||
if str(item.get("status") or "").lower() in {"running", "queued", "waiting"}
|
||||
][:5]
|
||||
|
||||
recommendation_focus_ref = ensure_dict(recommendation.get("focus_ref"))
|
||||
first_attention = attention_nodes[0] if attention_nodes else {}
|
||||
first_attention_node_code = ensure_str(first_attention.get("node_code"))
|
||||
scene_log_target_node_code = ensure_str(scene_log.get("target_node_code"))
|
||||
latest_release_version = str(latest_release_entry.get("version") or latest_release_entry.get("release_version") or "")
|
||||
release_focus_ref = ensure_dict(launchpad_rec.get("focus_ref"))
|
||||
release_id = str(release_focus_ref.get("release_id") or "")
|
||||
operator_decision = build_operator_decision()
|
||||
primary_command_key = ensure_str(operator_decision.get("primary_command_key"))
|
||||
secondary_command_key = ensure_str(operator_decision.get("secondary_command_key"))
|
||||
|
||||
result = {
|
||||
"overview": {
|
||||
"top_recommendation": {
|
||||
"source": str(recommendation.get("source") or ""),
|
||||
"key": str(recommendation.get("key") or ""),
|
||||
"priority": str(recommendation.get("priority") or ""),
|
||||
"summary": str(recommendation.get("summary") or ""),
|
||||
"primary_action_code": str(recommendation.get("primary_action_code") or ""),
|
||||
"secondary_action_code": str(recommendation.get("secondary_action_code") or ""),
|
||||
"focus_ref": recommendation.get("focus_ref") or {},
|
||||
"node_codes": list(recommendation.get("node_codes") or []),
|
||||
},
|
||||
"scene_log_observation": {
|
||||
"status": str(scene_log.get("status") or ""),
|
||||
"status_label": str(scene_log.get("status_label") or ""),
|
||||
"summary": str(scene_log.get("summary") or ""),
|
||||
"target_node_code": str(scene_log.get("target_node_code") or ""),
|
||||
"participating_node_count": int(scene_log.get("participating_node_count") or 0),
|
||||
"covered_participating_node_count": int(scene_log.get("covered_participating_node_count") or 0),
|
||||
"line_count": int(scene_log.get("line_count") or 0),
|
||||
"focus_ref": scene_log.get("focus_ref") or {},
|
||||
},
|
||||
},
|
||||
"nodes": {
|
||||
"managed_total": int(node_summary.get("managed_total") or len(managed_nodes)),
|
||||
"online_or_busy_total": len(online_nodes),
|
||||
"offline_total": int(node_summary.get("offline_count") or 0),
|
||||
"attention_total": len(attention_nodes),
|
||||
"attention_nodes": attention_nodes[:8],
|
||||
},
|
||||
"jobs": {
|
||||
"running_total": int(jobs_summary.get("running_count") or 0),
|
||||
"queued_total": int(jobs_summary.get("queued_count") or 0),
|
||||
"failed_total": int(jobs_summary.get("failed_count") or 0),
|
||||
"recent_running_jobs": running_jobs,
|
||||
},
|
||||
"activity_stream": {
|
||||
"total": int(activity_summary.get("total") or len(activity_items)),
|
||||
"error_count": int(activity_summary.get("error_count") or 0),
|
||||
"warning_count": int(activity_summary.get("warning_count") or 0),
|
||||
"running_job_count": int(activity_summary.get("running_job_count") or 0),
|
||||
"recent_items": recent_activity,
|
||||
},
|
||||
"operator_decision": {
|
||||
"lane": ensure_str(operator_decision.get("lane")),
|
||||
"priority": ensure_str(operator_decision.get("priority")),
|
||||
"reason_code": ensure_str(operator_decision.get("reason_code")),
|
||||
"title": ensure_str(operator_decision.get("title")),
|
||||
"summary": ensure_str(operator_decision.get("summary")),
|
||||
"next_focus": ensure_dict(operator_decision.get("next_focus")),
|
||||
"primary_command_key": primary_command_key,
|
||||
"secondary_command_key": secondary_command_key,
|
||||
},
|
||||
"recommended_commands": {
|
||||
"stack_diagnosis": "bash domain-api/deploy/multi-region/check_ops_center_stack.sh",
|
||||
"driver_feed": command("driver-feed"),
|
||||
"activity_preview": command("activity-preview"),
|
||||
"driver_focus_preview": command("driver-focus-preview"),
|
||||
"driver_focus_run": command("driver-focus-run"),
|
||||
"top_recommendation_preview": (
|
||||
command("driver-preview", str(recommendation.get("primary_action_code") or ""))
|
||||
if str(recommendation.get("primary_action_code") or "").strip()
|
||||
else ""
|
||||
),
|
||||
"top_recommendation_focus_preview": (
|
||||
command("driver-focus-preview", f"entry:{str(recommendation.get('key') or '')}")
|
||||
if str(recommendation.get("key") or "").strip()
|
||||
else ""
|
||||
),
|
||||
"scene_log": (
|
||||
command("scene-node-log", scene_log_target_node_code, "120", "key")
|
||||
if scene_log_target_node_code
|
||||
else ""
|
||||
),
|
||||
"scene_log_direct": (
|
||||
f"bash domain-api/deploy/multi-region/check_node_scene_log.sh {base_url} {scene_log_target_node_code} 120 key"
|
||||
if scene_log_target_node_code
|
||||
else ""
|
||||
),
|
||||
"node_handover": (
|
||||
command("node-handover", first_attention_node_code)
|
||||
if first_attention_node_code
|
||||
else command("stack-first-gap-handover")
|
||||
),
|
||||
"node_bootstrap_plan": (
|
||||
command("node-bootstrap-plan", first_attention_node_code)
|
||||
if first_attention_node_code
|
||||
else ""
|
||||
),
|
||||
"delivery_queue": (
|
||||
command("delivery-queue", first_attention_node_code)
|
||||
if first_attention_node_code
|
||||
else ""
|
||||
),
|
||||
"release_launchpad": release_command("launchpad"),
|
||||
"release_preview_latest": release_command("preview-latest"),
|
||||
"release_preview_latest_worker": release_command("preview-latest-smart", "worker"),
|
||||
"release_preview_latest_control": release_command("preview-latest-smart", "control"),
|
||||
"release_focus": (
|
||||
command("driver-preview", "focus_release_hub")
|
||||
if str(launchpad_rec.get("action_code") or "").strip()
|
||||
else ""
|
||||
),
|
||||
"release_preview_selected": (
|
||||
release_command("preview-release", release_id, "worker")
|
||||
if release_id
|
||||
else ""
|
||||
),
|
||||
"jobs": command("activity-stream", "kind=ops_job"),
|
||||
},
|
||||
"next_actions": {
|
||||
"primary_command": "",
|
||||
"secondary_command": "",
|
||||
},
|
||||
"release_launchpad": {
|
||||
"status": str(launchpad_status.get("status") or ""),
|
||||
"status_label": str(launchpad_status.get("status_label") or ""),
|
||||
"summary": str(launchpad_status.get("summary") or ""),
|
||||
"recommended_action_code": str(launchpad_rec.get("action_code") or ""),
|
||||
"recommended_action_label": str(launchpad_rec.get("action_label") or ""),
|
||||
"recommended_execution_mode": str(launchpad_rec.get("execution_mode") or ""),
|
||||
"recommended_execution_mode_label": str(launchpad_rec.get("execution_mode_label") or ""),
|
||||
"focus_ref": launchpad_rec.get("focus_ref") or {},
|
||||
"release_version": str(launchpad_release.get("version") or launchpad_release.get("release_version") or ""),
|
||||
"worker_rollout_preview": {
|
||||
"available": bool(launchpad_workers.get("available")),
|
||||
"blocked": bool(launchpad_workers.get("blocked")),
|
||||
"execution_mode": str(launchpad_workers.get("execution_mode") or ""),
|
||||
"execution_mode_label": str(launchpad_workers.get("execution_mode_label") or ""),
|
||||
"target_node_codes": list(launchpad_workers.get("target_node_codes") or []),
|
||||
},
|
||||
"control_rollout_preview": {
|
||||
"available": bool(launchpad_controls.get("available")),
|
||||
"blocked": bool(launchpad_controls.get("blocked")),
|
||||
"execution_mode": str(launchpad_controls.get("execution_mode") or ""),
|
||||
"execution_mode_label": str(launchpad_controls.get("execution_mode_label") or ""),
|
||||
"target_node_codes": list(launchpad_controls.get("target_node_codes") or []),
|
||||
},
|
||||
},
|
||||
"latest_release": {
|
||||
"version": str(latest_release_entry.get("version") or latest_release_entry.get("release_version") or ""),
|
||||
"status": ensure_str(latest_release_entry.get("status")),
|
||||
"status_label": ensure_str(latest_release_entry.get("status_label")),
|
||||
"summary": ensure_str(latest_release_entry.get("summary") or latest_release_entry.get("summary_text")),
|
||||
},
|
||||
"policy_preview_sample": {
|
||||
"execution_mode": str(policy_preview.get("execution_mode") or ""),
|
||||
"risk_level": str(policy_preview.get("risk_level") or ""),
|
||||
"approval_required": bool(policy_preview.get("approval_required")),
|
||||
"blocked": bool(policy_preview.get("blocked")),
|
||||
"blocking_reasons": ensure_list(policy_preview.get("blocking_reasons"))[:5],
|
||||
"warnings": ensure_list(policy_preview.get("warnings"))[:5],
|
||||
},
|
||||
"focus_refs": {
|
||||
"top_recommendation": recommendation_focus_ref,
|
||||
"scene_log": ensure_dict(scene_log.get("focus_ref")),
|
||||
"release_hub": release_focus_ref,
|
||||
},
|
||||
"rollout_targets": {
|
||||
"worker": list(launchpad_workers.get("target_node_codes") or []),
|
||||
"control": list(launchpad_controls.get("target_node_codes") or []),
|
||||
},
|
||||
"package_preview": {
|
||||
"available": bool(latest_package_preview.get("release")),
|
||||
"preview_source": ensure_str(latest_package_preview.get("preview_source")),
|
||||
"message": ensure_str(latest_package_preview.get("message")),
|
||||
},
|
||||
}
|
||||
|
||||
result["next_actions"]["primary_command"] = ensure_str(
|
||||
ensure_dict(result.get("recommended_commands")).get(primary_command_key)
|
||||
)
|
||||
result["next_actions"]["secondary_command"] = ensure_str(
|
||||
ensure_dict(result.get("recommended_commands")).get(secondary_command_key)
|
||||
)
|
||||
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
373
domain-api/deploy/multi-region/check_release_hub.sh
Normal file
373
domain-api/deploy/multi-region/check_release_hub.sh
Normal file
@@ -0,0 +1,373 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
INPUT_RELEASE_ID="${2:-}"
|
||||
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
|
||||
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
|
||||
|
||||
curl_json() {
|
||||
curl -fsS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
"$1"
|
||||
}
|
||||
|
||||
curl_post_json() {
|
||||
local url="$1"
|
||||
local body="$2"
|
||||
curl -fsS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
-X POST "${url}" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "${body}"
|
||||
}
|
||||
|
||||
launchpad="$(curl_json "${BASE_URL}/api/v1/ops/releases/launchpad")"
|
||||
latest_release="$(curl_json "${BASE_URL}/api/v1/ops/releases/latest")"
|
||||
action_templates="$(curl_json "${BASE_URL}/api/v1/ops/action-templates")"
|
||||
cluster_json="$(curl_json "${BASE_URL}/api/v1/runtime/cluster")"
|
||||
release_id="${INPUT_RELEASE_ID}"
|
||||
|
||||
if [[ -z "${release_id}" ]]; then
|
||||
release_id="$(python3 -c '
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
data = payload.get("data") or {}
|
||||
value = data.get("id") or 0
|
||||
print(value)
|
||||
' "${latest_release}")"
|
||||
fi
|
||||
|
||||
selected_release="${latest_release}"
|
||||
if [[ -n "${release_id}" && "${release_id}" != "0" ]]; then
|
||||
selected_release="$(curl_json "${BASE_URL}/api/v1/ops/releases/${release_id}")"
|
||||
fi
|
||||
|
||||
build_release_preview_payload() {
|
||||
local preview_mode="$1"
|
||||
python3 - <<'PY' "${selected_release}" "${cluster_json}" "${preview_mode}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
release_payload = json.loads(sys.argv[1]).get("data") or {}
|
||||
cluster_payload = json.loads(sys.argv[2]).get("data") or {}
|
||||
preview_mode = str(sys.argv[3] or "").strip().lower()
|
||||
|
||||
nodes = list(cluster_payload.get("nodes") or [])
|
||||
|
||||
def is_online(node):
|
||||
return str(node.get("status") or "").strip() in {"online", "busy"}
|
||||
|
||||
control_nodes = [
|
||||
str(node.get("node_code") or "").strip()
|
||||
for node in nodes
|
||||
if is_online(node) and str(node.get("role") or "").strip() == "control"
|
||||
]
|
||||
worker_nodes = [
|
||||
str(node.get("node_code") or "").strip()
|
||||
for node in nodes
|
||||
if is_online(node)
|
||||
and str(node.get("role") or "").strip() == "worker"
|
||||
and bool(node.get("is_effective_worker", False))
|
||||
]
|
||||
online_nodes = [
|
||||
str(node.get("node_code") or "").strip()
|
||||
for node in nodes
|
||||
if is_online(node)
|
||||
]
|
||||
|
||||
payload = {
|
||||
"release_version": str(release_payload.get("release_version") or "").strip(),
|
||||
"artifact_url": str(release_payload.get("artifact_url") or "").strip(),
|
||||
"checksum": str(release_payload.get("checksum") or "").strip(),
|
||||
"install_root": "/opt/domaincheck",
|
||||
"health_check_timeout_seconds": 10,
|
||||
"health_check_retries": 2,
|
||||
"health_check_interval_seconds": 2,
|
||||
"rollback_on_failure": True,
|
||||
"switch_current": True,
|
||||
}
|
||||
|
||||
if preview_mode == "control":
|
||||
template_key = "deploy.release.control"
|
||||
target_node_codes = control_nodes
|
||||
payload.update(
|
||||
{
|
||||
"restart_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"],
|
||||
"health_check_urls": ["http://127.0.0.1:8100/health"],
|
||||
"health_check_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"],
|
||||
}
|
||||
)
|
||||
elif preview_mode == "worker":
|
||||
template_key = "deploy.release.worker"
|
||||
target_node_codes = worker_nodes
|
||||
payload.update(
|
||||
{
|
||||
"restart_services": ["domaincheck-worker"],
|
||||
"health_check_urls": [],
|
||||
"health_check_services": ["domaincheck-worker"],
|
||||
}
|
||||
)
|
||||
else:
|
||||
template_key = "deploy.release.custom"
|
||||
target_node_codes = (control_nodes[:1] + worker_nodes[:2]) or online_nodes[:3]
|
||||
payload.update(
|
||||
{
|
||||
"restart_services": [],
|
||||
"health_check_urls": [],
|
||||
"health_check_services": [],
|
||||
}
|
||||
)
|
||||
|
||||
request_payload = {
|
||||
"template_key": template_key,
|
||||
"action": "deploy.release",
|
||||
"target_type": "batch",
|
||||
"execution_mode": "remote-agent",
|
||||
"target_node_codes": [item for item in target_node_codes if item],
|
||||
"payload": payload,
|
||||
}
|
||||
|
||||
print(json.dumps(request_payload, ensure_ascii=False))
|
||||
PY
|
||||
}
|
||||
|
||||
print_preview_request_summary() {
|
||||
local preview_payload="$1"
|
||||
python3 - <<'PY' "${preview_payload}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
body = payload.get("payload") or {}
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"template_key": payload.get("template_key") or "",
|
||||
"action": payload.get("action") or "",
|
||||
"execution_mode": payload.get("execution_mode") or "",
|
||||
"target_node_codes": payload.get("target_node_codes") or [],
|
||||
"release_version": body.get("release_version") or "",
|
||||
"artifact_url_present": bool(str(body.get("artifact_url") or "").strip()),
|
||||
"checksum_present": bool(str(body.get("checksum") or "").strip()),
|
||||
"restart_services": body.get("restart_services") or [],
|
||||
"health_check_urls": body.get("health_check_urls") or [],
|
||||
"health_check_services": body.get("health_check_services") or [],
|
||||
},
|
||||
ensure_ascii=False,
|
||||
indent=2,
|
||||
)
|
||||
)
|
||||
PY
|
||||
}
|
||||
|
||||
run_release_preview() {
|
||||
local step_no="$1"
|
||||
local preview_mode="$2"
|
||||
local section_title="$3"
|
||||
local preview_payload
|
||||
preview_payload="$(build_release_preview_payload "${preview_mode}")"
|
||||
|
||||
echo "[${step_no}/10] ${section_title}"
|
||||
print_preview_request_summary "${preview_payload}"
|
||||
echo
|
||||
curl_post_json "${BASE_URL}/api/v1/ops/policy/preview" "${preview_payload}"
|
||||
echo
|
||||
echo
|
||||
}
|
||||
|
||||
echo "[1/11] release launchpad"
|
||||
printf '%s\n\n' "${launchpad}"
|
||||
|
||||
echo "[2/11] latest release"
|
||||
printf '%s\n\n' "${selected_release}"
|
||||
|
||||
echo "[3/11] recent releases"
|
||||
curl_json "${BASE_URL}/api/v1/ops/releases?limit=8"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[4/11] release deploy templates"
|
||||
python3 - <<'PY' "${action_templates}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
groups = (payload.get("data") or {}).get("groups") or []
|
||||
release_group = next((item for item in groups if str(item.get("group_key") or "") == "release"), {})
|
||||
items = list(release_group.get("items") or [])
|
||||
summary = {
|
||||
"group_key": release_group.get("group_key") or "",
|
||||
"group_title": release_group.get("group_title") or "",
|
||||
"template_keys": [str(item.get("key") or "") for item in items],
|
||||
"field_types": sorted(
|
||||
{
|
||||
str(field.get("type") or "")
|
||||
for item in items
|
||||
for field in list(item.get("fields") or [])
|
||||
if str(field.get("type") or "")
|
||||
}
|
||||
),
|
||||
}
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
|
||||
echo "[5/11] online release target summary"
|
||||
python3 - <<'PY' "${cluster_json}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1]).get("data") or {}
|
||||
nodes = list(payload.get("nodes") or [])
|
||||
|
||||
def is_online(node):
|
||||
return str(node.get("status") or "").strip() in {"online", "busy"}
|
||||
|
||||
summary = {
|
||||
"online_control_nodes": [
|
||||
str(node.get("node_code") or "").strip()
|
||||
for node in nodes
|
||||
if is_online(node) and str(node.get("role") or "").strip() == "control"
|
||||
],
|
||||
"online_worker_nodes": [
|
||||
str(node.get("node_code") or "").strip()
|
||||
for node in nodes
|
||||
if is_online(node) and str(node.get("role") or "").strip() == "worker" and bool(node.get("is_effective_worker", False))
|
||||
],
|
||||
"all_online_nodes": [
|
||||
{
|
||||
"node_code": str(node.get("node_code") or "").strip(),
|
||||
"role": str(node.get("role") or "").strip(),
|
||||
"status": str(node.get("status") or "").strip(),
|
||||
"is_effective_worker": bool(node.get("is_effective_worker", False)),
|
||||
}
|
||||
for node in nodes
|
||||
if is_online(node)
|
||||
],
|
||||
}
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
echo
|
||||
|
||||
run_release_preview "6" "control" "deploy.release.control preview"
|
||||
run_release_preview "7" "worker" "deploy.release.worker preview"
|
||||
run_release_preview "8" "custom" "deploy.release.custom preview"
|
||||
|
||||
rollout_list='{"code":0,"message":"no release selected","data":{"rollouts":[]}}'
|
||||
rollout_id="0"
|
||||
if [[ -n "${release_id}" && "${release_id}" != "0" ]]; then
|
||||
rollout_list="$(curl_json "${BASE_URL}/api/v1/ops/releases/${release_id}/rollouts?limit=10")"
|
||||
rollout_id="$(python3 -c '
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
rollouts = (payload.get("data") or {}).get("rollouts") or []
|
||||
print((rollouts[0] or {}).get("id") or 0 if rollouts else 0)
|
||||
' "${rollout_list}")"
|
||||
fi
|
||||
|
||||
echo "[9/11] rollout list for release ${release_id:-0}"
|
||||
printf '%s\n\n' "${rollout_list}"
|
||||
|
||||
echo "[10/11] primary rollout detail"
|
||||
if [[ "${rollout_id}" != "0" ]]; then
|
||||
curl_json "${BASE_URL}/api/v1/ops/rollouts/${rollout_id}"
|
||||
else
|
||||
echo '{"code":0,"message":"no rollout","data":{}}'
|
||||
fi
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[11/12] primary rollout jobs"
|
||||
if [[ "${rollout_id}" != "0" ]]; then
|
||||
curl_json "${BASE_URL}/api/v1/ops/rollouts/${rollout_id}/jobs?limit=50"
|
||||
else
|
||||
echo '{"code":0,"message":"no rollout","data":{"jobs":[]}}'
|
||||
fi
|
||||
echo
|
||||
|
||||
echo
|
||||
echo "[12/12] condensed summary"
|
||||
python3 - <<'PY' "${launchpad}" "${selected_release}" "${cluster_json}" "${rollout_list}" "${rollout_id}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
launchpad_payload = json.loads(sys.argv[1] or "{}").get("data") or {}
|
||||
selected_release = json.loads(sys.argv[2] or "{}").get("data") or {}
|
||||
cluster_payload = json.loads(sys.argv[3] or "{}").get("data") or {}
|
||||
rollout_list_payload = json.loads(sys.argv[4] or "{}").get("data") or {}
|
||||
rollout_id = int(sys.argv[5] or 0)
|
||||
|
||||
launchpad_status = dict(launchpad_payload.get("launchpad_status") or {})
|
||||
worker_preview = dict(launchpad_payload.get("worker_rollout_preview") or {})
|
||||
worker_smart_rollout = dict(worker_preview.get("smart_rollout") or {})
|
||||
control_preview = dict(launchpad_payload.get("control_rollout_preview") or {})
|
||||
control_smart_rollout = dict(control_preview.get("smart_rollout") or {})
|
||||
nodes = list(cluster_payload.get("nodes") or [])
|
||||
rollouts = list(rollout_list_payload.get("rollouts") or [])
|
||||
|
||||
def is_online(node):
|
||||
return str(node.get("status") or "").strip() in {"online", "busy"}
|
||||
|
||||
summary = {
|
||||
"release": {
|
||||
"id": int(selected_release.get("id") or 0),
|
||||
"release_version": str(selected_release.get("release_version") or ""),
|
||||
"channel": str(selected_release.get("channel") or ""),
|
||||
"status": str(selected_release.get("status") or ""),
|
||||
},
|
||||
"launchpad": {
|
||||
"status": str(launchpad_status.get("status") or ""),
|
||||
"status_label": str(launchpad_status.get("status_label") or ""),
|
||||
"summary": str(launchpad_status.get("summary") or ""),
|
||||
"recommended_action_code": str(launchpad_status.get("recommended_action_code") or ""),
|
||||
"recommended_target_node_code": str(launchpad_status.get("recommended_target_node_code") or ""),
|
||||
"recommended_recovery_label": str(launchpad_status.get("recommended_recovery_label") or ""),
|
||||
"recommended_recovery_summary": str(launchpad_status.get("recommended_recovery_summary") or ""),
|
||||
"onboarding_bootstrap_pending_nodes": int(launchpad_status.get("onboarding_bootstrap_pending_nodes", 0) or 0),
|
||||
"onboarding_acceptance_ready_nodes": int(launchpad_status.get("onboarding_acceptance_ready_nodes", 0) or 0),
|
||||
"recommended_execution_mode": str(launchpad_status.get("recommended_execution_mode") or ""),
|
||||
"recommended_execution_mode_label": str(launchpad_status.get("recommended_execution_mode_label") or ""),
|
||||
"focus_ref": dict(launchpad_status.get("focus_ref") or {}),
|
||||
},
|
||||
"worker_rollout_preview": {
|
||||
"available": bool(worker_smart_rollout.get("available", False)),
|
||||
"blocked": bool(worker_preview.get("blocked", False)),
|
||||
"requires_confirmation": bool(worker_preview.get("requires_confirmation", False)),
|
||||
"execution_mode": str(worker_smart_rollout.get("execution_mode") or ""),
|
||||
"target_node_codes": list(worker_smart_rollout.get("target_node_codes") or []),
|
||||
},
|
||||
"control_rollout_preview": {
|
||||
"available": bool(control_smart_rollout.get("available", False)),
|
||||
"blocked": bool(control_preview.get("blocked", False)),
|
||||
"requires_confirmation": bool(control_preview.get("requires_confirmation", False)),
|
||||
"execution_mode": str(control_smart_rollout.get("execution_mode") or ""),
|
||||
"target_node_codes": list(control_smart_rollout.get("target_node_codes") or []),
|
||||
},
|
||||
"cluster": {
|
||||
"online_control_nodes": [
|
||||
str(node.get("node_code") or "").strip()
|
||||
for node in nodes
|
||||
if is_online(node) and str(node.get("role") or "").strip() == "control"
|
||||
],
|
||||
"online_worker_nodes": [
|
||||
str(node.get("node_code") or "").strip()
|
||||
for node in nodes
|
||||
if is_online(node) and str(node.get("role") or "").strip() == "worker" and bool(node.get("is_effective_worker", False))
|
||||
],
|
||||
},
|
||||
"rollouts": {
|
||||
"count": len(rollouts),
|
||||
"selected_rollout_id": rollout_id,
|
||||
},
|
||||
}
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
echo
|
||||
@@ -4,15 +4,24 @@ set -euo pipefail
|
||||
MAINLAND_BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
OVERSEAS_BASE_URL="${2:-http://152.53.37.118:8100}"
|
||||
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
|
||||
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
|
||||
|
||||
curl_json() {
|
||||
curl -fsS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
"$1"
|
||||
}
|
||||
|
||||
echo "[1/5] mainland runtime cluster"
|
||||
MAINLAND_CLUSTER="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/cluster")"
|
||||
MAINLAND_CLUSTER="$(curl_json "${MAINLAND_BASE_URL}/api/v1/runtime/cluster")"
|
||||
echo "${MAINLAND_CLUSTER}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[2/5] mainland runtime readiness"
|
||||
MAINLAND_READINESS="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/readiness")"
|
||||
MAINLAND_READINESS="$(curl_json "${MAINLAND_BASE_URL}/api/v1/runtime/readiness")"
|
||||
echo "${MAINLAND_READINESS}"
|
||||
echo
|
||||
echo
|
||||
@@ -79,7 +88,7 @@ echo
|
||||
echo
|
||||
|
||||
echo "[4/5] overseas runtime cluster"
|
||||
OVERSEAS_CLUSTER="$(curl -fsS "${OVERSEAS_BASE_URL}/api/v1/runtime/cluster")"
|
||||
OVERSEAS_CLUSTER="$(curl_json "${OVERSEAS_BASE_URL}/api/v1/runtime/cluster")"
|
||||
echo "${OVERSEAS_CLUSTER}"
|
||||
echo
|
||||
echo
|
||||
|
||||
@@ -4,23 +4,44 @@ set -euo pipefail
|
||||
DB_NAME="${DB_NAME:-domain}"
|
||||
DB_USER="${DB_USER:-postgres}"
|
||||
PSQL_BIN="${PSQL_BIN:-/www/server/pgsql/bin/psql}"
|
||||
API_BASE_URL="${1:-${API_BASE_URL:-http://127.0.0.1:8100}}"
|
||||
CURL_CONNECT_TIMEOUT="${CURL_CONNECT_TIMEOUT:-3}"
|
||||
CURL_MAX_TIME="${CURL_MAX_TIME:-10}"
|
||||
|
||||
echo "[1/3] cluster nodes"
|
||||
curl_json() {
|
||||
curl -fsS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
"$1"
|
||||
}
|
||||
|
||||
RUNTIME_STATUS_JSON="$(curl_json "${API_BASE_URL}/api/v1/runtime/status")"
|
||||
|
||||
echo "[1/4] cluster nodes"
|
||||
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -Atc "
|
||||
select
|
||||
node_code || '|' ||
|
||||
role || '|' ||
|
||||
status || '|' ||
|
||||
coalesce(current_load, 0) || '|' ||
|
||||
coalesce(metadata_json->>'job_items_claimed', '0') || '|' ||
|
||||
coalesce(metadata_json->>'job_items_running', '0') || '|' ||
|
||||
coalesce(metadata_json->>'job_items_completed', '0') || '|' ||
|
||||
to_char(last_heartbeat_at, 'YYYY-MM-DD HH24:MI:SS') || '|' ||
|
||||
coalesce(metadata_json->>'worker_online', '') || '|' ||
|
||||
coalesce(metadata_json->>'detect_participating', '')
|
||||
(
|
||||
case
|
||||
when coalesce(current_load, 0) > 0 then 'true'
|
||||
when coalesce(metadata_json->>'detect_participating', '') in ('true', 'false') then metadata_json->>'detect_participating'
|
||||
else ''
|
||||
end
|
||||
)
|
||||
from detect_worker_nodes
|
||||
order by node_code;
|
||||
"
|
||||
echo
|
||||
|
||||
echo "[2/3] active job distribution"
|
||||
echo "[2/4] active job distribution"
|
||||
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -Atc "
|
||||
select
|
||||
coalesce(claimed_by, '') || '|' ||
|
||||
@@ -39,59 +60,43 @@ order by claimed_by, status;
|
||||
"
|
||||
echo
|
||||
|
||||
echo "[3/3] condensed summary"
|
||||
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -At <<'SQL'
|
||||
with latest_job as (
|
||||
select id, job_code, status
|
||||
from detect_jobs
|
||||
where status in ('pending','running')
|
||||
order by id desc
|
||||
limit 1
|
||||
),
|
||||
node_stats as (
|
||||
select
|
||||
node_code,
|
||||
role,
|
||||
status,
|
||||
coalesce(metadata_json->>'worker_online', '') as worker_online,
|
||||
coalesce(metadata_json->>'detect_participating', '') as detect_participating
|
||||
from detect_worker_nodes
|
||||
),
|
||||
item_stats as (
|
||||
select
|
||||
coalesce(claimed_by, '') as claimed_by,
|
||||
status,
|
||||
count(*) as cnt
|
||||
from detect_job_items
|
||||
where job_id = (select id from latest_job)
|
||||
group by claimed_by, status
|
||||
)
|
||||
select json_build_object(
|
||||
'job', (
|
||||
select json_build_object(
|
||||
'job_code', coalesce(job_code, ''),
|
||||
'status', coalesce(status, '')
|
||||
)
|
||||
from latest_job
|
||||
),
|
||||
'online_nodes', (
|
||||
select coalesce(json_agg(json_build_object(
|
||||
'node_code', node_code,
|
||||
'role', role,
|
||||
'status', status,
|
||||
'worker_online', worker_online,
|
||||
'detect_participating', detect_participating
|
||||
) order by node_code), '[]'::json)
|
||||
from node_stats
|
||||
where status in ('online', 'busy')
|
||||
),
|
||||
'claimed_distribution', (
|
||||
select coalesce(json_agg(json_build_object(
|
||||
'claimed_by', claimed_by,
|
||||
'status', status,
|
||||
'count', cnt
|
||||
) order by claimed_by, status), '[]'::json)
|
||||
from item_stats
|
||||
)
|
||||
);
|
||||
SQL
|
||||
echo "[3/4] runtime participation snapshot"
|
||||
printf "%s" "${RUNTIME_STATUS_JSON}" | python3 -c '
|
||||
import json, sys
|
||||
payload = json.load(sys.stdin).get("data", {})
|
||||
detect = payload.get("detect", {})
|
||||
print(json.dumps({
|
||||
"node": payload.get("node", {}),
|
||||
"participation_summary": detect.get("participation_summary", {}),
|
||||
"participating_nodes": detect.get("participating_nodes", []),
|
||||
"non_participating_nodes": detect.get("non_participating_nodes", detect.get("standby_nodes", [])),
|
||||
"log_sync": detect.get("log_sync", {}),
|
||||
}, ensure_ascii=False, indent=2))
|
||||
'
|
||||
echo
|
||||
|
||||
echo "[4/4] condensed summary"
|
||||
printf "%s" "${RUNTIME_STATUS_JSON}" | python3 -c '
|
||||
import json, sys
|
||||
payload = json.load(sys.stdin).get("data", {})
|
||||
detect = payload.get("detect", {})
|
||||
summary = detect.get("participation_summary", {}) or {}
|
||||
print(json.dumps({
|
||||
"node": payload.get("node", {}),
|
||||
"active_job": {
|
||||
"job_code": (detect.get("active_job") or {}).get("job_code", ""),
|
||||
"status": (detect.get("active_job") or {}).get("status", ""),
|
||||
},
|
||||
"participation_summary": summary,
|
||||
"dispatch_active_nodes": summary.get("dispatch_active_node_codes", []),
|
||||
"recent_only_nodes": summary.get("recent_only_node_codes", []),
|
||||
"non_participating_nodes": summary.get("non_participating_node_codes", []),
|
||||
"log_sync": {
|
||||
"enabled": bool((detect.get("log_sync") or {}).get("enabled", False)),
|
||||
"mode": (detect.get("log_sync") or {}).get("mode", ""),
|
||||
"line_count": (detect.get("log_sync") or {}).get("line_count", 0),
|
||||
"source_nodes": (detect.get("log_sync") or {}).get("source_nodes", []),
|
||||
"last_at": (detect.get("log_sync") or {}).get("last_at", ""),
|
||||
},
|
||||
}, ensure_ascii=False, indent=2))
|
||||
'
|
||||
|
||||
447
domain-api/deploy/multi-region/drive_ops_action.sh
Executable file
447
domain-api/deploy/multi-region/drive_ops_action.sh
Executable file
@@ -0,0 +1,447 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${SCRIPT_DIR}/lib_ops_center.sh"
|
||||
ops_center_load_config "${SCRIPT_DIR}"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
usage:
|
||||
bash domain-api/deploy/multi-region/drive_ops_action.sh <base_url> <preview|execute> <action_code> [action_payload_json|@payload.json] [requested_by]
|
||||
bash domain-api/deploy/multi-region/drive_ops_action.sh <base_url> <action_code> [action_payload_json|@payload.json] [requested_by]
|
||||
|
||||
examples:
|
||||
bash domain-api/deploy/multi-region/drive_ops_action.sh \
|
||||
http://127.0.0.1:8100 \
|
||||
preview \
|
||||
create_release_rollout_worker \
|
||||
'{"release_id":12}'
|
||||
|
||||
bash domain-api/deploy/multi-region/drive_ops_action.sh \
|
||||
http://127.0.0.1:8100 \
|
||||
create_release_rollout_worker \
|
||||
'{"release_id":12}'
|
||||
|
||||
NODE_CODES="mainland-worker-01,mainland-worker-02" \
|
||||
bash domain-api/deploy/multi-region/drive_ops_action.sh \
|
||||
http://127.0.0.1:8100 \
|
||||
run_standard_inspection \
|
||||
'{}'
|
||||
|
||||
notes:
|
||||
- preview/execute 双模式共用同一份 action payload 和 node_codes
|
||||
- 旧用法仍默认按 execute 处理
|
||||
- NODE_CODES can be passed through env as comma/newline separated node_code list
|
||||
- action_payload_json can also use @/path/to/file.json
|
||||
- preview metadata can be passed through env:
|
||||
SOURCE_LABEL / ACTION_TITLE / ACTION_SUMMARY / ACTION_REASON / ACTION_META_TEXT / PRIMARY_LABEL
|
||||
EOF
|
||||
}
|
||||
|
||||
BASE_URL="${1:-$(ops_center_resolve_mainland_api_base_url)}"
|
||||
SECOND_ARG="${2:-}"
|
||||
THIRD_ARG="${3:-}"
|
||||
FOURTH_ARG="${4:-}"
|
||||
FIFTH_ARG="${5:-}"
|
||||
|
||||
if [[ "${SECOND_ARG}" == "preview" || "${SECOND_ARG}" == "execute" ]]; then
|
||||
OPERATION="${SECOND_ARG}"
|
||||
ACTION_CODE="${THIRD_ARG:-}"
|
||||
RAW_ACTION_PAYLOAD="${FOURTH_ARG:-}"
|
||||
REQUESTED_BY="${FIFTH_ARG:-$(ops_center_resolve_requested_by)}"
|
||||
else
|
||||
OPERATION="execute"
|
||||
ACTION_CODE="${SECOND_ARG:-}"
|
||||
RAW_ACTION_PAYLOAD="${THIRD_ARG:-}"
|
||||
REQUESTED_BY="${FOURTH_ARG:-$(ops_center_resolve_requested_by)}"
|
||||
fi
|
||||
|
||||
if [[ -z "${RAW_ACTION_PAYLOAD}" ]]; then
|
||||
RAW_ACTION_PAYLOAD='{}'
|
||||
fi
|
||||
|
||||
NODE_CODES_INPUT="${NODE_CODES:-}"
|
||||
SOURCE_LABEL="${SOURCE_LABEL:-cli/drive_ops_action}"
|
||||
ACTION_TITLE="${ACTION_TITLE:-}"
|
||||
ACTION_SUMMARY="${ACTION_SUMMARY:-}"
|
||||
ACTION_REASON="${ACTION_REASON:-}"
|
||||
ACTION_META_TEXT="${ACTION_META_TEXT:-}"
|
||||
PRIMARY_LABEL="${PRIMARY_LABEL:-}"
|
||||
|
||||
print_json_or_raw() {
|
||||
local raw_text="${1:-}"
|
||||
python3 - "${raw_text}" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
raw_text = str(sys.argv[1] or "")
|
||||
try:
|
||||
payload = json.loads(raw_text)
|
||||
except Exception:
|
||||
print(raw_text)
|
||||
else:
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
}
|
||||
|
||||
request_with_status() {
|
||||
local method="$1"
|
||||
local url="$2"
|
||||
local body="${3:-}"
|
||||
if [[ "${method}" == "POST" ]]; then
|
||||
curl -sS -X POST "${url}" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "${body}" \
|
||||
-w $'\nHTTP_STATUS=%{http_code}'
|
||||
else
|
||||
curl -sS "${url}" -w $'\nHTTP_STATUS=%{http_code}'
|
||||
fi
|
||||
}
|
||||
|
||||
if [[ -z "${ACTION_CODE}" ]]; then
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${OPERATION}" != "preview" && "${OPERATION}" != "execute" ]]; then
|
||||
echo "operation must be preview or execute" >&2
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ACTION_PAYLOAD_JSON="${RAW_ACTION_PAYLOAD}"
|
||||
if [[ "${ACTION_PAYLOAD_JSON}" == @* ]]; then
|
||||
ACTION_PAYLOAD_PATH="${ACTION_PAYLOAD_JSON#@}"
|
||||
if [[ ! -f "${ACTION_PAYLOAD_PATH}" ]]; then
|
||||
echo "payload file not found: ${ACTION_PAYLOAD_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
ACTION_PAYLOAD_JSON="$(cat "${ACTION_PAYLOAD_PATH}")"
|
||||
fi
|
||||
|
||||
if [[ -z "${NODE_CODES_INPUT}" ]]; then
|
||||
NODE_CODES_INPUT="$(python3 - <<'PY' "${BASE_URL}" "${ACTION_CODE}" "${ACTION_PAYLOAD_JSON}"
|
||||
import json
|
||||
import sys
|
||||
import urllib.request
|
||||
|
||||
|
||||
def normalize_codes(*groups):
|
||||
result = []
|
||||
for group in groups:
|
||||
for item in list(group or []):
|
||||
node_code = str(item or "").strip()
|
||||
if node_code and node_code not in result:
|
||||
result.append(node_code)
|
||||
return result
|
||||
|
||||
|
||||
base_url = str(sys.argv[1] or "").rstrip("/")
|
||||
action_code = str(sys.argv[2] or "").strip()
|
||||
payload_raw = str(sys.argv[3] or "").strip() or "{}"
|
||||
|
||||
try:
|
||||
payload = json.loads(payload_raw)
|
||||
except Exception:
|
||||
payload = {}
|
||||
if not isinstance(payload, dict):
|
||||
payload = {}
|
||||
|
||||
payload_node_codes = normalize_codes(payload.get("node_codes") or [])
|
||||
if payload_node_codes:
|
||||
print(",".join(payload_node_codes))
|
||||
raise SystemExit(0)
|
||||
|
||||
supported_actions = {
|
||||
"open_worker_logs_participating",
|
||||
"run_inspection_participating",
|
||||
"open_worker_logs_standby",
|
||||
"run_inspection_standby",
|
||||
"open_worker_logs",
|
||||
"run_standard_inspection",
|
||||
}
|
||||
if action_code not in supported_actions or not base_url:
|
||||
print("")
|
||||
raise SystemExit(0)
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(f"{base_url}/api/v1/ops/overview", timeout=20) as response:
|
||||
overview_payload = json.loads(response.read().decode("utf-8"))
|
||||
overview = dict((overview_payload or {}).get("data") or {})
|
||||
except Exception:
|
||||
print("")
|
||||
raise SystemExit(0)
|
||||
|
||||
execution_scene = dict(overview.get("execution_scene") or {})
|
||||
participation_summary = dict(execution_scene.get("participation_summary") or {})
|
||||
log_sync = dict(execution_scene.get("log_sync") or {})
|
||||
|
||||
dispatch_active = normalize_codes(participation_summary.get("dispatch_active_node_codes") or [])
|
||||
recent_only = normalize_codes(participation_summary.get("recent_only_node_codes") or [])
|
||||
standby = normalize_codes(
|
||||
participation_summary.get("standby_node_codes") or [],
|
||||
participation_summary.get("non_participating_node_codes") or [],
|
||||
)
|
||||
missing_logs = normalize_codes(log_sync.get("missing_participating_nodes") or [])
|
||||
|
||||
resolved_node_codes = []
|
||||
if action_code in {"open_worker_logs_participating", "run_inspection_participating"}:
|
||||
resolved_node_codes = normalize_codes(missing_logs, dispatch_active, recent_only)
|
||||
elif action_code in {"open_worker_logs_standby", "run_inspection_standby"}:
|
||||
resolved_node_codes = normalize_codes(standby)
|
||||
elif action_code == "open_worker_logs":
|
||||
resolved_node_codes = normalize_codes(missing_logs, dispatch_active, recent_only, standby)
|
||||
elif action_code == "run_standard_inspection":
|
||||
resolved_node_codes = normalize_codes(dispatch_active, recent_only, standby)
|
||||
|
||||
print(",".join(resolved_node_codes))
|
||||
PY
|
||||
)"
|
||||
fi
|
||||
|
||||
REQUEST_JSON="$(python3 - <<'PY' "${ACTION_CODE}" "${ACTION_PAYLOAD_JSON}" "${REQUESTED_BY}" "${NODE_CODES_INPUT}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
action_code = str(sys.argv[1] or "").strip()
|
||||
payload_raw = str(sys.argv[2] or "").strip() or "{}"
|
||||
requested_by = str(sys.argv[3] or "").strip() or "cli/drive_ops_action"
|
||||
node_codes_raw = str(sys.argv[4] or "")
|
||||
|
||||
try:
|
||||
action_payload = json.loads(payload_raw)
|
||||
except Exception as exc: # pragma: no cover - cli validation
|
||||
raise SystemExit(f"invalid action payload json: {exc}")
|
||||
|
||||
if not isinstance(action_payload, dict):
|
||||
raise SystemExit("action payload must be a json object")
|
||||
|
||||
node_codes = []
|
||||
seen = set()
|
||||
for item in node_codes_raw.replace("\n", ",").split(","):
|
||||
value = str(item or "").strip()
|
||||
if not value or value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
node_codes.append(value)
|
||||
|
||||
request_payload = {
|
||||
"action_code": action_code,
|
||||
"node_codes": node_codes,
|
||||
"action_payload": action_payload,
|
||||
"requested_by": requested_by,
|
||||
}
|
||||
print(json.dumps(request_payload, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
|
||||
PREVIEW_REQUEST_JSON="$(python3 - <<'PY' "${ACTION_CODE}" "${ACTION_PAYLOAD_JSON}" "${REQUESTED_BY}" "${NODE_CODES_INPUT}" "${SOURCE_LABEL}" "${ACTION_TITLE}" "${ACTION_SUMMARY}" "${ACTION_REASON}" "${ACTION_META_TEXT}" "${PRIMARY_LABEL}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
action_code = str(sys.argv[1] or "").strip()
|
||||
payload_raw = str(sys.argv[2] or "").strip() or "{}"
|
||||
requested_by = str(sys.argv[3] or "").strip() or "cli/drive_ops_action"
|
||||
node_codes_raw = str(sys.argv[4] or "")
|
||||
source_label = str(sys.argv[5] or "").strip() or "cli/drive_ops_action"
|
||||
action_title = str(sys.argv[6] or "").strip()
|
||||
action_summary = str(sys.argv[7] or "").strip()
|
||||
action_reason = str(sys.argv[8] or "").strip()
|
||||
action_meta_text = str(sys.argv[9] or "").strip()
|
||||
primary_label = str(sys.argv[10] or "").strip()
|
||||
|
||||
try:
|
||||
action_payload = json.loads(payload_raw)
|
||||
except Exception as exc:
|
||||
raise SystemExit(f"invalid action payload json: {exc}")
|
||||
|
||||
if not isinstance(action_payload, dict):
|
||||
raise SystemExit("action payload must be a json object")
|
||||
|
||||
node_codes = []
|
||||
seen = set()
|
||||
for item in node_codes_raw.replace("\n", ",").split(","):
|
||||
value = str(item or "").strip()
|
||||
if not value or value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
node_codes.append(value)
|
||||
|
||||
request_payload = {
|
||||
"source_label": source_label,
|
||||
"title": action_title or primary_label or action_code,
|
||||
"summary": action_summary,
|
||||
"reason": action_reason,
|
||||
"meta_text": action_meta_text,
|
||||
"executor_kind": "driver_action",
|
||||
"primary_label": primary_label or action_code,
|
||||
"primary_action_code": action_code,
|
||||
"primary_node_codes": node_codes,
|
||||
"primary_action_payload": action_payload,
|
||||
"requested_by": requested_by,
|
||||
}
|
||||
print(json.dumps(request_payload, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
|
||||
if [[ "${OPERATION}" == "preview" ]]; then
|
||||
echo "[1/3] driver action preview request"
|
||||
print_json_or_raw "${PREVIEW_REQUEST_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
RAW_HTTP_RESPONSE="$(request_with_status "POST" "${BASE_URL}/api/v1/ops/driver-actions/preview" "${PREVIEW_REQUEST_JSON}")"
|
||||
HTTP_STATUS="${RAW_HTTP_RESPONSE##*$'\n'HTTP_STATUS=}"
|
||||
RESPONSE_JSON="${RAW_HTTP_RESPONSE%$'\n'HTTP_STATUS=*}"
|
||||
|
||||
echo "[2/3] raw response"
|
||||
printf '%s\n' "http_status=${HTTP_STATUS}"
|
||||
print_json_or_raw "${RESPONSE_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[3/3] condensed summary"
|
||||
python3 - "${RESPONSE_JSON}" "${HTTP_STATUS}" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
http_status = int(sys.argv[2] or 0)
|
||||
data = payload.get("data") or {}
|
||||
primary = data.get("primary") or {}
|
||||
secondary = data.get("secondary") or {}
|
||||
|
||||
raw_code = payload.get("code", 1)
|
||||
if raw_code in (0, "0", False):
|
||||
normalized_code = 0
|
||||
else:
|
||||
normalized_code = int(raw_code or 1)
|
||||
|
||||
summary = {
|
||||
"http_status": http_status,
|
||||
"code": normalized_code,
|
||||
"message": str(payload.get("message") or ""),
|
||||
"contract_key": str(data.get("contract_key") or ""),
|
||||
"contract_version": str(data.get("contract_version") or ""),
|
||||
"preview_endpoint": str(data.get("preview_endpoint") or ""),
|
||||
"source_label": str(data.get("source_label") or ""),
|
||||
"title": str(data.get("title") or ""),
|
||||
"executor_kind": str(data.get("executor_kind") or ""),
|
||||
"execution_chain": str(data.get("execution_chain") or ""),
|
||||
"primary": {
|
||||
"label": str(primary.get("label") or ""),
|
||||
"action_code": str(primary.get("action_code") or ""),
|
||||
"target_method": str(primary.get("target_method") or ""),
|
||||
"target_api": str(primary.get("target_api") or ""),
|
||||
"recommendation_label": str(primary.get("recommendation_label") or ""),
|
||||
"automation_level_label": str(primary.get("automation_level_label") or ""),
|
||||
"risk_level_label": str(primary.get("risk_level_label") or ""),
|
||||
"node_codes": list(primary.get("node_codes") or []),
|
||||
"request_payload": primary.get("request_payload") or {},
|
||||
},
|
||||
}
|
||||
|
||||
if secondary:
|
||||
summary["secondary"] = {
|
||||
"label": str(secondary.get("label") or ""),
|
||||
"action_code": str(secondary.get("action_code") or ""),
|
||||
"target_method": str(secondary.get("target_method") or ""),
|
||||
"target_api": str(secondary.get("target_api") or ""),
|
||||
"recommendation_label": str(secondary.get("recommendation_label") or ""),
|
||||
"automation_level_label": str(secondary.get("automation_level_label") or ""),
|
||||
"risk_level_label": str(secondary.get("risk_level_label") or ""),
|
||||
"node_codes": list(secondary.get("node_codes") or []),
|
||||
"request_payload": secondary.get("request_payload") or {},
|
||||
}
|
||||
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
else
|
||||
echo "[1/3] driver action request"
|
||||
print_json_or_raw "${REQUEST_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
RAW_HTTP_RESPONSE="$(request_with_status "POST" "${BASE_URL}/api/v1/ops/driver-actions/execute" "${REQUEST_JSON}")"
|
||||
HTTP_STATUS="${RAW_HTTP_RESPONSE##*$'\n'HTTP_STATUS=}"
|
||||
RESPONSE_JSON="${RAW_HTTP_RESPONSE%$'\n'HTTP_STATUS=*}"
|
||||
|
||||
echo "[2/3] raw response"
|
||||
printf '%s\n' "http_status=${HTTP_STATUS}"
|
||||
print_json_or_raw "${RESPONSE_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[3/3] condensed summary"
|
||||
python3 - "${RESPONSE_JSON}" "${HTTP_STATUS}" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
payload = json.loads(sys.argv[1])
|
||||
http_status = int(sys.argv[2] or 0)
|
||||
data = payload.get("data") or {}
|
||||
smart_release_result = data.get("smart_release_result") or {}
|
||||
ui_intent = data.get("ui_intent") or {}
|
||||
runtime_settings = data.get("runtime_settings") or {}
|
||||
|
||||
raw_code = payload.get("code", 1)
|
||||
if raw_code in (0, "0", False):
|
||||
normalized_code = 0
|
||||
else:
|
||||
normalized_code = int(raw_code or 1)
|
||||
|
||||
summary = {
|
||||
"http_status": http_status,
|
||||
"code": normalized_code,
|
||||
"message": str(payload.get("message") or ""),
|
||||
"handled": bool(data.get("handled", False)),
|
||||
"action_code": str(data.get("action_code") or ""),
|
||||
"mode": str(data.get("mode") or ""),
|
||||
}
|
||||
|
||||
if smart_release_result:
|
||||
release = smart_release_result.get("release") or {}
|
||||
rollout = smart_release_result.get("rollout") or {}
|
||||
smart_plan = smart_release_result.get("smart_rollout") or {}
|
||||
preview = smart_release_result.get("preview") or {}
|
||||
summary["smart_release_result"] = {
|
||||
"release_id": int(release.get("id") or 0),
|
||||
"release_version": str(release.get("release_version") or ""),
|
||||
"release_created": bool(smart_release_result.get("release_created", False)),
|
||||
"release_deduplicated": bool(smart_release_result.get("release_deduplicated", False)),
|
||||
"rollout_created": bool(smart_release_result.get("rollout_created", False)),
|
||||
"requires_confirmation": bool(smart_release_result.get("requires_confirmation", False)),
|
||||
"blocked": bool(smart_release_result.get("blocked", False)),
|
||||
"rollout_id": int(rollout.get("id") or 0),
|
||||
"rollout_code": str(rollout.get("rollout_code") or ""),
|
||||
"smart_mode": str(smart_plan.get("mode") or ""),
|
||||
"smart_available": bool(smart_plan.get("available", False)),
|
||||
"target_node_codes": list((smart_plan.get("target_selector") or {}).get("node_codes") or []),
|
||||
"preview_warnings": list(preview.get("warnings") or []),
|
||||
"preview_approval_required": bool(preview.get("approval_required", False)),
|
||||
}
|
||||
|
||||
if ui_intent:
|
||||
summary["ui_intent"] = {
|
||||
"kind": str(ui_intent.get("kind") or ""),
|
||||
"release_id": int(ui_intent.get("release_id") or 0),
|
||||
"template_key": str(ui_intent.get("template_key") or ""),
|
||||
"playbook_key": str(ui_intent.get("playbook_key") or ""),
|
||||
"target_node_codes": list(ui_intent.get("target_node_codes") or []),
|
||||
}
|
||||
|
||||
if runtime_settings:
|
||||
summary["runtime_settings"] = {
|
||||
"worker_log_sync_enabled": bool(runtime_settings.get("worker_log_sync_enabled", False)),
|
||||
"worker_log_sync_mode": str(runtime_settings.get("worker_log_sync_mode") or ""),
|
||||
}
|
||||
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
fi
|
||||
|
||||
if [[ "${HTTP_STATUS}" -lt 200 || "${HTTP_STATUS}" -ge 300 ]]; then
|
||||
exit 1
|
||||
fi
|
||||
7776
domain-api/deploy/multi-region/drive_ops_center.sh
Executable file
7776
domain-api/deploy/multi-region/drive_ops_center.sh
Executable file
File diff suppressed because it is too large
Load Diff
476
domain-api/deploy/multi-region/drive_release_hub.sh
Executable file
476
domain-api/deploy/multi-region/drive_release_hub.sh
Executable file
@@ -0,0 +1,476 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${SCRIPT_DIR}/lib_ops_center.sh"
|
||||
ops_center_load_config "${SCRIPT_DIR}"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
usage:
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh <base_url> <command> [args...]
|
||||
|
||||
commands:
|
||||
launchpad
|
||||
latest-package
|
||||
latest-release
|
||||
preview-latest
|
||||
preview-latest-smart <worker|control>
|
||||
preview-release <release_id> <worker|control>
|
||||
create-latest-release
|
||||
list-releases [limit]
|
||||
list-rollouts <release_id> [limit]
|
||||
smart-latest <worker|control> [confirm]
|
||||
smart-release <release_id> <worker|control> [confirm]
|
||||
|
||||
examples:
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 launchpad
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 latest-package
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 preview-latest
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 preview-latest-smart worker
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 create-latest-release
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 smart-latest worker
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 preview-release 18 control
|
||||
bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 smart-release 18 control confirm
|
||||
EOF
|
||||
}
|
||||
|
||||
BASE_URL="${1:-$(ops_center_resolve_mainland_api_base_url)}"
|
||||
COMMAND="${2:-}"
|
||||
ARG1="${3:-}"
|
||||
ARG2="${4:-}"
|
||||
ARG3="${5:-}"
|
||||
REQUESTED_BY="${REQUESTED_BY:-$(ops_center_resolve_requested_by)}"
|
||||
|
||||
print_json_or_raw() {
|
||||
local raw_text="${1:-}"
|
||||
python3 - "${raw_text}" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
raw_text = str(sys.argv[1] or "")
|
||||
try:
|
||||
payload = json.loads(raw_text)
|
||||
except Exception:
|
||||
print(raw_text)
|
||||
else:
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
}
|
||||
|
||||
request_with_status() {
|
||||
local method="$1"
|
||||
local url="$2"
|
||||
local body="${3:-}"
|
||||
if [[ "${method}" == "POST" ]]; then
|
||||
curl -sS -X POST "${url}" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "${body}" \
|
||||
-w $'\nHTTP_STATUS=%{http_code}'
|
||||
else
|
||||
curl -sS "${url}" -w $'\nHTTP_STATUS=%{http_code}'
|
||||
fi
|
||||
}
|
||||
|
||||
if [[ -z "${COMMAND}" ]]; then
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
|
||||
METHOD="GET"
|
||||
PATH_SUFFIX=""
|
||||
REQUEST_JSON=""
|
||||
|
||||
confirm_to_bool() {
|
||||
local raw_value="${1:-}"
|
||||
case "$(printf '%s' "${raw_value}" | tr '[:upper:]' '[:lower:]')" in
|
||||
1|true|yes|y|confirm|confirmed)
|
||||
printf 'true'
|
||||
;;
|
||||
*)
|
||||
printf 'false'
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
case "${COMMAND}" in
|
||||
launchpad)
|
||||
CHANNEL="${ARG1:-stable}"
|
||||
PATH_SUFFIX="/api/v1/ops/releases/launchpad?channel=${CHANNEL}"
|
||||
;;
|
||||
latest-package)
|
||||
PATH_SUFFIX="/api/v1/ops/releases/package-metadata/latest"
|
||||
;;
|
||||
latest-release)
|
||||
PATH_SUFFIX="/api/v1/ops/releases/latest"
|
||||
;;
|
||||
preview-latest)
|
||||
METHOD="POST"
|
||||
PATH_SUFFIX="/api/v1/ops/releases/from-package/latest/preview"
|
||||
REQUEST_JSON="$(python3 - <<'PY' "${REQUESTED_BY}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
requested_by = str(sys.argv[1] or "").strip() or "cli/drive_release_hub"
|
||||
payload = {
|
||||
"created_by": f"{requested_by}/latest-package-preview",
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
;;
|
||||
preview-latest-smart)
|
||||
MODE="${ARG1:-}"
|
||||
if [[ "${MODE}" != "worker" && "${MODE}" != "control" ]]; then
|
||||
echo "mode must be worker or control" >&2
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
METHOD="POST"
|
||||
PATH_SUFFIX="/api/v1/ops/releases/from-package/latest/smart-rollout-preview"
|
||||
REQUEST_JSON="$(python3 - <<'PY' "${MODE}" "${REQUESTED_BY}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
mode = str(sys.argv[1] or "").strip()
|
||||
requested_by = str(sys.argv[2] or "").strip() or "cli/drive_release_hub"
|
||||
|
||||
payload = {
|
||||
"mode": mode,
|
||||
"created_by": f"{requested_by}/latest-package-preview",
|
||||
"rollout_created_by": f"{requested_by}/latest-package-preview/rollout",
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
;;
|
||||
create-latest-release)
|
||||
METHOD="POST"
|
||||
PATH_SUFFIX="/api/v1/ops/releases/from-package/latest"
|
||||
REQUEST_JSON="$(python3 - <<'PY' "${REQUESTED_BY}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
requested_by = str(sys.argv[1] or "").strip() or "cli/drive_release_hub"
|
||||
payload = {
|
||||
"created_by": f"{requested_by}/latest-package",
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
;;
|
||||
list-releases)
|
||||
RELEASE_LIMIT="${ARG1:-10}"
|
||||
PATH_SUFFIX="/api/v1/ops/releases?limit=${RELEASE_LIMIT}"
|
||||
;;
|
||||
list-rollouts)
|
||||
RELEASE_ID="${ARG1:-}"
|
||||
ROLLOUT_LIMIT="${ARG2:-10}"
|
||||
if [[ -z "${RELEASE_ID}" ]]; then
|
||||
echo "release_id is required for list-rollouts" >&2
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
PATH_SUFFIX="/api/v1/ops/releases/${RELEASE_ID}/rollouts?limit=${ROLLOUT_LIMIT}"
|
||||
;;
|
||||
smart-latest)
|
||||
MODE="${ARG1:-}"
|
||||
if [[ "${MODE}" != "worker" && "${MODE}" != "control" ]]; then
|
||||
echo "mode must be worker or control" >&2
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
METHOD="POST"
|
||||
PATH_SUFFIX="/api/v1/ops/releases/from-package/latest/smart-rollout"
|
||||
CONFIRM_RISKY="$(confirm_to_bool "${ARG2:-}")"
|
||||
REQUEST_JSON="$(python3 - <<'PY' "${MODE}" "${CONFIRM_RISKY}" "${REQUESTED_BY}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
mode = str(sys.argv[1] or "").strip()
|
||||
confirm_risky = str(sys.argv[2] or "").strip().lower() == "true"
|
||||
requested_by = str(sys.argv[3] or "").strip() or "cli/drive_release_hub"
|
||||
|
||||
payload = {
|
||||
"mode": mode,
|
||||
"confirm_risky": confirm_risky,
|
||||
"created_by": f"{requested_by}/latest-package",
|
||||
"rollout_created_by": f"{requested_by}/latest-package/rollout",
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
;;
|
||||
preview-release)
|
||||
RELEASE_ID="${ARG1:-}"
|
||||
MODE="${ARG2:-}"
|
||||
if [[ -z "${RELEASE_ID}" ]]; then
|
||||
echo "release_id is required for preview-release" >&2
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
if [[ "${MODE}" != "worker" && "${MODE}" != "control" ]]; then
|
||||
echo "mode must be worker or control" >&2
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
METHOD="POST"
|
||||
PATH_SUFFIX="/api/v1/ops/releases/${RELEASE_ID}/smart-rollout-preview"
|
||||
REQUEST_JSON="$(python3 - <<'PY' "${MODE}" "${REQUESTED_BY}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
mode = str(sys.argv[1] or "").strip()
|
||||
requested_by = str(sys.argv[2] or "").strip() or "cli/drive_release_hub"
|
||||
|
||||
payload = {
|
||||
"mode": mode,
|
||||
"created_by": f"{requested_by}/existing-release-preview",
|
||||
"rollout_created_by": f"{requested_by}/existing-release-preview/rollout",
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
;;
|
||||
smart-release)
|
||||
RELEASE_ID="${ARG1:-}"
|
||||
MODE="${ARG2:-}"
|
||||
if [[ -z "${RELEASE_ID}" ]]; then
|
||||
echo "release_id is required for smart-release" >&2
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
if [[ "${MODE}" != "worker" && "${MODE}" != "control" ]]; then
|
||||
echo "mode must be worker or control" >&2
|
||||
usage
|
||||
exit 1
|
||||
fi
|
||||
METHOD="POST"
|
||||
PATH_SUFFIX="/api/v1/ops/releases/${RELEASE_ID}/smart-rollout"
|
||||
CONFIRM_RISKY="$(confirm_to_bool "${ARG3:-}")"
|
||||
REQUEST_JSON="$(python3 - <<'PY' "${MODE}" "${CONFIRM_RISKY}" "${REQUESTED_BY}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
mode = str(sys.argv[1] or "").strip()
|
||||
confirm_risky = str(sys.argv[2] or "").strip().lower() == "true"
|
||||
requested_by = str(sys.argv[3] or "").strip() or "cli/drive_release_hub"
|
||||
|
||||
payload = {
|
||||
"mode": mode,
|
||||
"confirm_risky": confirm_risky,
|
||||
"created_by": f"{requested_by}/existing-release",
|
||||
"rollout_created_by": f"{requested_by}/existing-release/rollout",
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False))
|
||||
PY
|
||||
)"
|
||||
;;
|
||||
*)
|
||||
echo "unknown command: ${COMMAND}" >&2
|
||||
usage
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
echo "[1/3] request"
|
||||
printf '%s\n' "method=${METHOD}"
|
||||
printf '%s\n' "url=${BASE_URL}${PATH_SUFFIX}"
|
||||
if [[ -n "${REQUEST_JSON}" ]]; then
|
||||
print_json_or_raw "${REQUEST_JSON}"
|
||||
else
|
||||
echo "(no request body)"
|
||||
fi
|
||||
echo
|
||||
echo
|
||||
|
||||
RAW_HTTP_RESPONSE="$(request_with_status "${METHOD}" "${BASE_URL}${PATH_SUFFIX}" "${REQUEST_JSON}")"
|
||||
HTTP_STATUS="${RAW_HTTP_RESPONSE##*$'\n'HTTP_STATUS=}"
|
||||
RESPONSE_JSON="${RAW_HTTP_RESPONSE%$'\n'HTTP_STATUS=*}"
|
||||
|
||||
echo "[2/3] raw response"
|
||||
printf '%s\n' "http_status=${HTTP_STATUS}"
|
||||
print_json_or_raw "${RESPONSE_JSON}"
|
||||
echo
|
||||
echo
|
||||
|
||||
echo "[3/3] condensed summary"
|
||||
python3 - "${RESPONSE_JSON}" "${HTTP_STATUS}" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
raw_response = str(sys.argv[1] or "")
|
||||
http_status = int(sys.argv[2] or 0)
|
||||
try:
|
||||
payload = json.loads(raw_response)
|
||||
except Exception:
|
||||
print(json.dumps({
|
||||
"http_status": http_status,
|
||||
"code": 1,
|
||||
"message": "response is not valid json",
|
||||
"raw_response": raw_response[:500],
|
||||
}, ensure_ascii=False, indent=2))
|
||||
sys.exit(0 if 200 <= http_status < 300 else 1)
|
||||
|
||||
data = payload.get("data") or {}
|
||||
|
||||
raw_code = payload.get("code", 1)
|
||||
if raw_code in (0, "0", False):
|
||||
normalized_code = 0
|
||||
else:
|
||||
normalized_code = int(raw_code or 1)
|
||||
|
||||
summary = {
|
||||
"http_status": http_status,
|
||||
"code": normalized_code,
|
||||
"message": str(payload.get("message") or ""),
|
||||
}
|
||||
|
||||
if isinstance(data, dict) and "package" in data:
|
||||
package = dict(data.get("package") or {})
|
||||
summary["package"] = {
|
||||
"available": bool(package.get("available", False)),
|
||||
"package_name": str(package.get("package_name") or ""),
|
||||
"package_type": str(package.get("package_type") or ""),
|
||||
"generated_at": str(package.get("generated_at") or ""),
|
||||
"smoke_test_ok": bool(package.get("smoke_test_ok", False)),
|
||||
"smoke_test_message": str(package.get("smoke_test_message") or ""),
|
||||
}
|
||||
|
||||
if isinstance(data, dict) and "launchpad_status" in data:
|
||||
launchpad_status = dict(data.get("launchpad_status") or {})
|
||||
worker_preview = dict(data.get("worker_rollout_preview") or {})
|
||||
worker_smart_rollout = dict(worker_preview.get("smart_rollout") or {})
|
||||
control_preview = dict(data.get("control_rollout_preview") or {})
|
||||
control_smart_rollout = dict(control_preview.get("smart_rollout") or {})
|
||||
summary["launchpad"] = {
|
||||
"channel": str(data.get("channel") or ""),
|
||||
"status": str(launchpad_status.get("status") or ""),
|
||||
"status_label": str(launchpad_status.get("status_label") or ""),
|
||||
"summary": str(launchpad_status.get("summary") or ""),
|
||||
"recommended_action_code": str(launchpad_status.get("recommended_action_code") or ""),
|
||||
"recommended_target_node_code": str(launchpad_status.get("recommended_target_node_code") or ""),
|
||||
"recommended_recovery_label": str(launchpad_status.get("recommended_recovery_label") or ""),
|
||||
"recommended_recovery_summary": str(launchpad_status.get("recommended_recovery_summary") or ""),
|
||||
"onboarding_bootstrap_pending_nodes": int(launchpad_status.get("onboarding_bootstrap_pending_nodes", 0) or 0),
|
||||
"onboarding_acceptance_ready_nodes": int(launchpad_status.get("onboarding_acceptance_ready_nodes", 0) or 0),
|
||||
"recommended_command": str(launchpad_status.get("recommended_command") or ""),
|
||||
"recommended_execution_mode": str(launchpad_status.get("recommended_execution_mode") or ""),
|
||||
"recommended_execution_mode_label": str(launchpad_status.get("recommended_execution_mode_label") or ""),
|
||||
"focus_ref": dict(launchpad_status.get("focus_ref") or {}),
|
||||
"latest_package": dict(data.get("latest_package") or {}),
|
||||
"latest_release": {
|
||||
"id": int(((data.get("latest_release") or {}).get("id") or 0)),
|
||||
"release_version": str((data.get("latest_release") or {}).get("release_version") or ""),
|
||||
"status": str((data.get("latest_release") or {}).get("status") or ""),
|
||||
},
|
||||
"worker_rollout_preview": {
|
||||
"message": str(worker_preview.get("message") or ""),
|
||||
"available": bool(worker_smart_rollout.get("available", False)),
|
||||
"requires_confirmation": bool(worker_preview.get("requires_confirmation", False)),
|
||||
"blocked": bool(worker_preview.get("blocked", False)),
|
||||
"execution_mode": str(worker_smart_rollout.get("execution_mode") or ""),
|
||||
"execution_mode_label": str(worker_smart_rollout.get("execution_mode_label") or ""),
|
||||
"target_node_codes": list(worker_smart_rollout.get("target_node_codes") or []),
|
||||
},
|
||||
"control_rollout_preview": {
|
||||
"message": str(control_preview.get("message") or ""),
|
||||
"available": bool(control_smart_rollout.get("available", False)),
|
||||
"requires_confirmation": bool(control_preview.get("requires_confirmation", False)),
|
||||
"blocked": bool(control_preview.get("blocked", False)),
|
||||
"execution_mode": str(control_smart_rollout.get("execution_mode") or ""),
|
||||
"execution_mode_label": str(control_smart_rollout.get("execution_mode_label") or ""),
|
||||
"target_node_codes": list(control_smart_rollout.get("target_node_codes") or []),
|
||||
},
|
||||
}
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
sys.exit(0)
|
||||
|
||||
if isinstance(data, dict) and "available" in data:
|
||||
summary["latest_package"] = {
|
||||
"available": bool(data.get("available", False)),
|
||||
"package_name": str(data.get("package_name") or ""),
|
||||
"package_type": str(data.get("package_type") or ""),
|
||||
"generated_at": str(data.get("generated_at") or ""),
|
||||
"commit_sha": str(data.get("commit_sha") or ""),
|
||||
"smoke_test_ok": bool(data.get("smoke_test_ok", False)),
|
||||
"smoke_test_message": str(data.get("smoke_test_message") or ""),
|
||||
"reason": str(data.get("reason") or ""),
|
||||
}
|
||||
elif isinstance(data, dict) and "releases" in data:
|
||||
releases = list(data.get("releases") or [])
|
||||
summary["releases"] = {
|
||||
"count": len(releases),
|
||||
"items": [
|
||||
{
|
||||
"id": int(item.get("id") or 0),
|
||||
"release_version": str(item.get("release_version") or ""),
|
||||
"channel": str(item.get("channel") or ""),
|
||||
"status": str(item.get("status") or ""),
|
||||
}
|
||||
for item in releases[:5]
|
||||
],
|
||||
}
|
||||
elif isinstance(data, dict) and "rollouts" in data:
|
||||
rollouts = list(data.get("rollouts") or [])
|
||||
summary["rollouts"] = {
|
||||
"count": len(rollouts),
|
||||
"items": [
|
||||
{
|
||||
"id": int(item.get("id") or 0),
|
||||
"rollout_code": str(item.get("rollout_code") or ""),
|
||||
"status": str(item.get("status") or ""),
|
||||
"jobs_total": int(item.get("jobs_total") or 0),
|
||||
"jobs_created": int(item.get("jobs_created") or 0),
|
||||
}
|
||||
for item in rollouts[:5]
|
||||
],
|
||||
}
|
||||
else:
|
||||
release = data.get("release") or data
|
||||
rollout = data.get("rollout") or {}
|
||||
smart_rollout = data.get("smart_rollout") or {}
|
||||
preview = data.get("preview") or {}
|
||||
summary["preview_source"] = str(data.get("preview_source") or "")
|
||||
summary["release"] = {
|
||||
"id": int(release.get("id") or 0),
|
||||
"release_version": str(release.get("release_version") or ""),
|
||||
"channel": str(release.get("channel") or ""),
|
||||
"status": str(release.get("status") or ""),
|
||||
"is_preview": bool(release.get("is_preview", False)),
|
||||
}
|
||||
if rollout:
|
||||
summary["rollout"] = {
|
||||
"id": int(rollout.get("id") or 0),
|
||||
"rollout_code": str(rollout.get("rollout_code") or ""),
|
||||
"status": str(rollout.get("status") or ""),
|
||||
}
|
||||
if smart_rollout:
|
||||
summary["smart_rollout"] = {
|
||||
"available": bool(smart_rollout.get("available", False)),
|
||||
"mode": str(smart_rollout.get("mode") or ""),
|
||||
"mode_label": str(smart_rollout.get("mode_label") or ""),
|
||||
"target_node_codes": list((smart_rollout.get("target_selector") or {}).get("node_codes") or []),
|
||||
"summary": str(smart_rollout.get("summary") or smart_rollout.get("reason") or ""),
|
||||
}
|
||||
if preview:
|
||||
summary["preview"] = {
|
||||
"blocked": bool(preview.get("blocked", False)),
|
||||
"approval_required": bool(preview.get("approval_required", False)),
|
||||
"warnings": list(preview.get("warnings") or []),
|
||||
}
|
||||
summary["result_flags"] = {
|
||||
"release_created": bool(data.get("release_created", False)),
|
||||
"release_deduplicated": bool(data.get("release_deduplicated", False)),
|
||||
"rollout_created": bool(data.get("rollout_created", False)),
|
||||
"requires_confirmation": bool(data.get("requires_confirmation", False)),
|
||||
"blocked": bool(data.get("blocked", False)),
|
||||
}
|
||||
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
|
||||
if [[ "${HTTP_STATUS}" -lt 200 || "${HTTP_STATUS}" -ge 300 ]]; then
|
||||
exit 1
|
||||
fi
|
||||
556
domain-api/deploy/multi-region/export_go_live_bundle.sh
Executable file
556
domain-api/deploy/multi-region/export_go_live_bundle.sh
Executable file
@@ -0,0 +1,556 @@
|
||||
#!/usr/bin/env bash
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "${SCRIPT_DIR}/lib_ops_center.sh"
|
||||
ops_center_load_config "${SCRIPT_DIR}"
|
||||
|
||||
MAINLAND_BASE_URL_DEFAULT="$(ops_center_resolve_mainland_api_base_url)"
|
||||
OVERSEAS_BASE_URL_DEFAULT="$(ops_center_resolve_overseas_api_base_url)"
|
||||
REPORT_ROOT_DEFAULT="${OPS_CENTER_REPORT_ROOT}/go-live-bundles"
|
||||
|
||||
REPORT_DIR_INPUT="${1:-}"
|
||||
MAINLAND_BASE_URL_INPUT="${2:-}"
|
||||
OVERSEAS_BASE_URL_INPUT="${3:-}"
|
||||
|
||||
timestamp_now() {
|
||||
date '+%Y%m%d_%H%M%S'
|
||||
}
|
||||
|
||||
normalize_optional_base_url() {
|
||||
local raw_value="${1:-}"
|
||||
local fallback_value="${2:-}"
|
||||
if [[ -z "${raw_value}" ]]; then
|
||||
printf '%s' "${fallback_value}"
|
||||
return
|
||||
fi
|
||||
case "${raw_value}" in
|
||||
-|none|null|disabled)
|
||||
printf ''
|
||||
;;
|
||||
*)
|
||||
printf '%s' "${raw_value}"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
if [[ -n "${REPORT_DIR_INPUT}" && "${REPORT_DIR_INPUT}" != http://* && "${REPORT_DIR_INPUT}" != https://* ]]; then
|
||||
REPORT_DIR="${REPORT_DIR_INPUT}"
|
||||
MAINLAND_BASE_URL="${MAINLAND_BASE_URL_INPUT:-${MAINLAND_BASE_URL_DEFAULT}}"
|
||||
OVERSEAS_BASE_URL="$(normalize_optional_base_url "${OVERSEAS_BASE_URL_INPUT:-}" "${OVERSEAS_BASE_URL_DEFAULT}")"
|
||||
else
|
||||
REPORT_DIR="${REPORT_ROOT_DEFAULT}/go-live-bundle-$(timestamp_now)"
|
||||
MAINLAND_BASE_URL="${REPORT_DIR_INPUT:-${MAINLAND_BASE_URL_DEFAULT}}"
|
||||
OVERSEAS_BASE_URL="$(normalize_optional_base_url "${MAINLAND_BASE_URL_INPUT:-}" "${OVERSEAS_BASE_URL_DEFAULT}")"
|
||||
fi
|
||||
|
||||
mkdir -p "${REPORT_DIR}"
|
||||
|
||||
CURL_CONNECT_TIMEOUT="${OPS_CENTER_CURL_CONNECT_TIMEOUT:-3}"
|
||||
CURL_MAX_TIME="${OPS_CENTER_CURL_MAX_TIME:-15}"
|
||||
STEP_COMMAND_TIMEOUT_SECONDS="${OPS_CENTER_REPORT_COMMAND_TIMEOUT_SECONDS:-45}"
|
||||
DOCTOR_DECISION_TIMEOUT_SECONDS="${OPS_CENTER_REPORT_DOCTOR_DECISION_TIMEOUT_SECONDS:-120}"
|
||||
DOCTOR_EXPORT_TIMEOUT_SECONDS="${OPS_CENTER_REPORT_DOCTOR_EXPORT_TIMEOUT_SECONDS:-180}"
|
||||
DOCTOR_INNER_REPORT_TIMEOUT_SECONDS="${OPS_CENTER_DOCTOR_INNER_REPORT_TIMEOUT_SECONDS:-60}"
|
||||
|
||||
ENV_AUDIT_TXT_PATH="${REPORT_DIR}/00_env_audit.txt"
|
||||
SUMMARY_TXT_PATH="${REPORT_DIR}/01_go_live_check_summary.txt"
|
||||
GO_LIVE_JSON_PATH="${REPORT_DIR}/02_go_live_summary.json"
|
||||
STACK_JSON_PATH="${REPORT_DIR}/03_stack_diagnosis.json"
|
||||
DRIVER_FEED_JSON_PATH="${REPORT_DIR}/04_driver_feed.json"
|
||||
CODEX_BRIEF_JSON_PATH="${REPORT_DIR}/05_codex_brief.json"
|
||||
RELEASE_LAUNCHPAD_JSON_PATH="${REPORT_DIR}/06_release_launchpad.json"
|
||||
DOCTOR_DECISION_JSON_PATH="${REPORT_DIR}/07_doctor_decision.json"
|
||||
DOCTOR_EXPORT_STDOUT_PATH="${REPORT_DIR}/08_doctor_export_stdout.txt"
|
||||
MANIFEST_PATH="${REPORT_DIR}/manifest.json"
|
||||
|
||||
declare -a STEP_RESULTS=()
|
||||
|
||||
pretty_print_json_file() {
|
||||
local source_path="${1:-}"
|
||||
python3 - "${source_path}" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
path = pathlib.Path(sys.argv[1])
|
||||
raw_text = path.read_text(encoding="utf-8") if path.exists() else ""
|
||||
try:
|
||||
payload = json.loads(raw_text)
|
||||
except Exception:
|
||||
print(raw_text)
|
||||
else:
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
}
|
||||
|
||||
record_step_result() {
|
||||
local key="${1:-}"
|
||||
local path="${2:-}"
|
||||
local ok="${3:-false}"
|
||||
local http_status="${4:-}"
|
||||
local exit_code="${5:-0}"
|
||||
STEP_RESULTS+=("${key}|${path}|${ok}|${http_status}|${exit_code}")
|
||||
}
|
||||
|
||||
fetch_json_to_file() {
|
||||
local key="${1:-}"
|
||||
local url="${2:-}"
|
||||
local output_path="${3:-}"
|
||||
local tmp_path
|
||||
tmp_path="$(mktemp)"
|
||||
local raw_text http_status
|
||||
raw_text="$(curl -sS \
|
||||
--connect-timeout "${CURL_CONNECT_TIMEOUT}" \
|
||||
--max-time "${CURL_MAX_TIME}" \
|
||||
"${url}" \
|
||||
-w $'\nHTTP_STATUS=%{http_code}' || true)"
|
||||
http_status="${raw_text##*$'\n'HTTP_STATUS=}"
|
||||
printf '%s' "${raw_text%$'\n'HTTP_STATUS=*}" > "${tmp_path}"
|
||||
pretty_print_json_file "${tmp_path}" > "${output_path}"
|
||||
rm -f "${tmp_path}"
|
||||
if [[ "${http_status}" == 2* ]]; then
|
||||
record_step_result "${key}" "${output_path}" "true" "${http_status}" "0"
|
||||
return 0
|
||||
fi
|
||||
record_step_result "${key}" "${output_path}" "false" "${http_status}" "1"
|
||||
return 1
|
||||
}
|
||||
|
||||
capture_command_to_file() {
|
||||
local key="${1:-}"
|
||||
local output_path="${2:-}"
|
||||
shift 2 || true
|
||||
capture_command_to_file_with_timeout "${STEP_COMMAND_TIMEOUT_SECONDS}" "${key}" "${output_path}" "$@"
|
||||
}
|
||||
|
||||
capture_command_to_file_with_timeout() {
|
||||
local timeout_seconds="${1:-45}"
|
||||
local key="${2:-}"
|
||||
local output_path="${3:-}"
|
||||
shift 3 || true
|
||||
local -a wrapped_command=("$@")
|
||||
local had_errexit="false"
|
||||
case "$-" in
|
||||
*e*) had_errexit="true" ;;
|
||||
esac
|
||||
if command -v timeout >/dev/null 2>&1; then
|
||||
wrapped_command=(timeout "${timeout_seconds}" "$@")
|
||||
fi
|
||||
set +e
|
||||
"${wrapped_command[@]}" > "${output_path}" 2>&1
|
||||
local exit_code=$?
|
||||
if [[ "${had_errexit}" == "true" ]]; then
|
||||
set -e
|
||||
else
|
||||
set +e
|
||||
fi
|
||||
if [[ "${exit_code}" -eq 0 ]]; then
|
||||
record_step_result "${key}" "${output_path}" "true" "" "${exit_code}"
|
||||
return 0
|
||||
fi
|
||||
record_step_result "${key}" "${output_path}" "false" "" "${exit_code}"
|
||||
return "${exit_code}"
|
||||
}
|
||||
|
||||
materialize_go_live_summary_fallback() {
|
||||
local summary_path="${1:-}"
|
||||
local output_path="${2:-}"
|
||||
python3 - "${summary_path}" "${output_path}" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
summary_path = pathlib.Path(sys.argv[1])
|
||||
output_path = pathlib.Path(sys.argv[2])
|
||||
raw_text = summary_path.read_text(encoding="utf-8") if summary_path.exists() else ""
|
||||
start = raw_text.find("{")
|
||||
end = raw_text.rfind("}")
|
||||
if start < 0 or end < start:
|
||||
sys.exit(1)
|
||||
json_text = raw_text[start:end + 1]
|
||||
payload = json.loads(json_text)
|
||||
if not isinstance(payload, dict):
|
||||
sys.exit(1)
|
||||
output_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
PY
|
||||
}
|
||||
|
||||
override_step_result() {
|
||||
local key="${1:-}"
|
||||
local path="${2:-}"
|
||||
local ok="${3:-false}"
|
||||
local http_status="${4:-}"
|
||||
local exit_code="${5:-0}"
|
||||
local -a updated_results=()
|
||||
local raw_item existing_key
|
||||
for raw_item in "${STEP_RESULTS[@]}"; do
|
||||
existing_key="${raw_item%%|*}"
|
||||
if [[ "${existing_key}" == "${key}" ]]; then
|
||||
continue
|
||||
fi
|
||||
updated_results+=("${raw_item}")
|
||||
done
|
||||
STEP_RESULTS=("${updated_results[@]}")
|
||||
record_step_result "${key}" "${path}" "${ok}" "${http_status}" "${exit_code}"
|
||||
}
|
||||
|
||||
set +e
|
||||
capture_command_to_file \
|
||||
"env_audit" \
|
||||
"${ENV_AUDIT_TXT_PATH}" \
|
||||
bash "${SCRIPT_DIR}/drive_ops_center.sh" env-audit "${MAINLAND_BASE_URL}"
|
||||
|
||||
capture_command_to_file \
|
||||
"go_live_check_summary" \
|
||||
"${SUMMARY_TXT_PATH}" \
|
||||
bash "${SCRIPT_DIR}/drive_ops_center.sh" go-live-check "${MAINLAND_BASE_URL}" "${OVERSEAS_BASE_URL}" summary
|
||||
|
||||
fetch_json_to_file \
|
||||
"go_live_summary" \
|
||||
"${MAINLAND_BASE_URL}/api/v1/ops/go-live-summary?base_url=${MAINLAND_BASE_URL}" \
|
||||
"${GO_LIVE_JSON_PATH}"
|
||||
|
||||
if [[ ! -s "${GO_LIVE_JSON_PATH}" || "$(sed -n '1p' "${GO_LIVE_JSON_PATH}" 2>/dev/null)" != "{"* ]]; then
|
||||
if materialize_go_live_summary_fallback "${SUMMARY_TXT_PATH}" "${GO_LIVE_JSON_PATH}" 2>/dev/null; then
|
||||
override_step_result "go_live_summary" "${GO_LIVE_JSON_PATH}" "true" "fallback" "0"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$(python3 - "${GO_LIVE_JSON_PATH}" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
path = pathlib.Path(sys.argv[1])
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
print("invalid")
|
||||
raise SystemExit(0)
|
||||
|
||||
if isinstance(payload, dict) and payload.get("detail") == "Not Found":
|
||||
print("not_found")
|
||||
else:
|
||||
print("ok")
|
||||
PY
|
||||
)" == "not_found" ]]; then
|
||||
if materialize_go_live_summary_fallback "${SUMMARY_TXT_PATH}" "${GO_LIVE_JSON_PATH}" 2>/dev/null; then
|
||||
override_step_result "go_live_summary" "${GO_LIVE_JSON_PATH}" "true" "fallback" "0"
|
||||
fi
|
||||
fi
|
||||
|
||||
fetch_json_to_file \
|
||||
"stack_diagnosis" \
|
||||
"${MAINLAND_BASE_URL}/api/v1/ops/stack-diagnosis?base_url=${MAINLAND_BASE_URL}" \
|
||||
"${STACK_JSON_PATH}"
|
||||
|
||||
fetch_json_to_file \
|
||||
"driver_feed" \
|
||||
"${MAINLAND_BASE_URL}/api/v1/ops/driver-feed" \
|
||||
"${DRIVER_FEED_JSON_PATH}"
|
||||
|
||||
fetch_json_to_file \
|
||||
"codex_brief" \
|
||||
"${MAINLAND_BASE_URL}/api/v1/ops/codex-brief" \
|
||||
"${CODEX_BRIEF_JSON_PATH}"
|
||||
|
||||
fetch_json_to_file \
|
||||
"release_launchpad" \
|
||||
"${MAINLAND_BASE_URL}/api/v1/ops/releases/launchpad" \
|
||||
"${RELEASE_LAUNCHPAD_JSON_PATH}"
|
||||
|
||||
capture_command_to_file_with_timeout \
|
||||
"${DOCTOR_EXPORT_TIMEOUT_SECONDS}" \
|
||||
"doctor_export" \
|
||||
"${DOCTOR_EXPORT_STDOUT_PATH}" \
|
||||
env "OPS_CENTER_DOCTOR_REPORT_TIMEOUT_SECONDS=${DOCTOR_INNER_REPORT_TIMEOUT_SECONDS}" \
|
||||
bash "${SCRIPT_DIR}/drive_ops_center.sh" doctor-export "${REPORT_DIR}/doctor-export" "${MAINLAND_BASE_URL}" "${OVERSEAS_BASE_URL}"
|
||||
|
||||
capture_command_to_file_with_timeout \
|
||||
"${DOCTOR_DECISION_TIMEOUT_SECONDS}" \
|
||||
"doctor_decision" \
|
||||
"${DOCTOR_DECISION_JSON_PATH}" \
|
||||
bash "${SCRIPT_DIR}/drive_ops_center.sh" doctor-decision "${REPORT_DIR}/doctor-export"
|
||||
set -e
|
||||
|
||||
python3 - <<'PY' \
|
||||
"${MANIFEST_PATH}" \
|
||||
"${REPORT_DIR}" \
|
||||
"${MAINLAND_BASE_URL}" \
|
||||
"${OVERSEAS_BASE_URL}" \
|
||||
"${ENV_AUDIT_TXT_PATH}" \
|
||||
"${SUMMARY_TXT_PATH}" \
|
||||
"${GO_LIVE_JSON_PATH}" \
|
||||
"${STACK_JSON_PATH}" \
|
||||
"${DRIVER_FEED_JSON_PATH}" \
|
||||
"${CODEX_BRIEF_JSON_PATH}" \
|
||||
"${RELEASE_LAUNCHPAD_JSON_PATH}" \
|
||||
"${DOCTOR_DECISION_JSON_PATH}" \
|
||||
"${DOCTOR_EXPORT_STDOUT_PATH}" \
|
||||
"${STEP_RESULTS[@]}"
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
manifest_path = pathlib.Path(sys.argv[1])
|
||||
report_dir = pathlib.Path(sys.argv[2])
|
||||
mainland_base_url = sys.argv[3]
|
||||
overseas_base_url = sys.argv[4]
|
||||
|
||||
file_keys = [
|
||||
"env_audit",
|
||||
"go_live_check_summary",
|
||||
"go_live_summary",
|
||||
"stack_diagnosis",
|
||||
"driver_feed",
|
||||
"codex_brief",
|
||||
"release_launchpad",
|
||||
"doctor_decision",
|
||||
"doctor_export",
|
||||
]
|
||||
file_paths = {
|
||||
"env_audit": pathlib.Path(sys.argv[5]),
|
||||
"go_live_check_summary": pathlib.Path(sys.argv[6]),
|
||||
"go_live_summary": pathlib.Path(sys.argv[7]),
|
||||
"stack_diagnosis": pathlib.Path(sys.argv[8]),
|
||||
"driver_feed": pathlib.Path(sys.argv[9]),
|
||||
"codex_brief": pathlib.Path(sys.argv[10]),
|
||||
"release_launchpad": pathlib.Path(sys.argv[11]),
|
||||
"doctor_decision": pathlib.Path(sys.argv[12]),
|
||||
"doctor_export": pathlib.Path(sys.argv[13]),
|
||||
}
|
||||
|
||||
step_results = {}
|
||||
for raw_item in sys.argv[14:]:
|
||||
parts = raw_item.split("|", 4)
|
||||
if len(parts) != 5:
|
||||
continue
|
||||
key, path_value, ok_value, http_status, exit_code = parts
|
||||
step_results[key] = {
|
||||
"path": path_value,
|
||||
"ok": ok_value == "true",
|
||||
"http_status": http_status,
|
||||
"exit_code": int(exit_code or 0),
|
||||
}
|
||||
|
||||
|
||||
def load_json_file(path: pathlib.Path) -> dict:
|
||||
if not path.is_file():
|
||||
return {}
|
||||
try:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return {}
|
||||
return payload if isinstance(payload, dict) else {}
|
||||
|
||||
|
||||
def load_env_audit_summary(path: pathlib.Path) -> dict:
|
||||
if not path.is_file():
|
||||
return {}
|
||||
raw_text = path.read_text(encoding="utf-8", errors="replace")
|
||||
marker = "[8/8] condensed env audit summary"
|
||||
candidate_text = raw_text.split(marker, 1)[1].strip() if marker in raw_text else raw_text.strip()
|
||||
if not candidate_text:
|
||||
return {}
|
||||
try:
|
||||
payload = json.loads(candidate_text)
|
||||
except Exception:
|
||||
return {}
|
||||
return payload if isinstance(payload, dict) else {}
|
||||
|
||||
failures = []
|
||||
artifacts = []
|
||||
for key in file_keys:
|
||||
artifact_path = file_paths[key]
|
||||
result = dict(step_results.get(key) or {})
|
||||
if not result:
|
||||
result = {
|
||||
"path": str(artifact_path),
|
||||
"ok": artifact_path.exists(),
|
||||
"http_status": "",
|
||||
"exit_code": 0 if artifact_path.exists() else 1,
|
||||
}
|
||||
entry = {
|
||||
"key": key,
|
||||
"path": str(artifact_path),
|
||||
"exists": artifact_path.exists(),
|
||||
"ok": bool(result.get("ok", False)),
|
||||
"http_status": str(result.get("http_status") or ""),
|
||||
"exit_code": int(result.get("exit_code", 0) or 0),
|
||||
}
|
||||
artifacts.append(entry)
|
||||
if not entry["ok"]:
|
||||
failures.append(key)
|
||||
|
||||
go_live_summary_payload = load_json_file(file_paths["go_live_summary"])
|
||||
stack_diagnosis_payload = load_json_file(file_paths["stack_diagnosis"])
|
||||
driver_feed_payload = load_json_file(file_paths["driver_feed"])
|
||||
codex_brief_payload = load_json_file(file_paths["codex_brief"])
|
||||
env_audit_summary = load_env_audit_summary(file_paths["env_audit"])
|
||||
|
||||
stack_diagnosis = dict(stack_diagnosis_payload.get("diagnosis") or {})
|
||||
driver_feed_summary = dict(driver_feed_payload.get("summary") or {})
|
||||
codex_brief_summary = dict(codex_brief_payload.get("summary") or {})
|
||||
|
||||
|
||||
def first_nonempty(*values: object) -> str:
|
||||
for value in values:
|
||||
normalized = str(value or "").strip()
|
||||
if normalized:
|
||||
return normalized
|
||||
return ""
|
||||
|
||||
|
||||
def nonempty_distinct(values: dict[str, str]) -> list[str]:
|
||||
return sorted({str(value or "").strip() for value in values.values() if str(value or "").strip()})
|
||||
|
||||
|
||||
def distinct_ints(values: dict[str, int], available_sources: set[str]) -> list[int]:
|
||||
return sorted({int(values[source]) for source in values if source in available_sources})
|
||||
|
||||
|
||||
launchpad_target_node_codes = {
|
||||
"go_live_summary": str(go_live_summary_payload.get("launchpad_recommended_target_node_code") or "").strip(),
|
||||
"stack_diagnosis": str(stack_diagnosis.get("launchpad_recommended_target_node_code") or "").strip(),
|
||||
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_target_node_code") or "").strip(),
|
||||
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_target_node_code") or "").strip(),
|
||||
}
|
||||
launchpad_recovery_labels = {
|
||||
"go_live_summary": str(go_live_summary_payload.get("launchpad_recommended_recovery_label") or "").strip(),
|
||||
"stack_diagnosis": str(stack_diagnosis.get("launchpad_recommended_recovery_label") or "").strip(),
|
||||
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_recovery_label") or "").strip(),
|
||||
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_recovery_label") or "").strip(),
|
||||
}
|
||||
launchpad_recovery_summaries = {
|
||||
"go_live_summary": str(go_live_summary_payload.get("launchpad_recommended_recovery_summary") or "").strip(),
|
||||
"stack_diagnosis": str(stack_diagnosis.get("launchpad_recommended_recovery_summary") or "").strip(),
|
||||
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
|
||||
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
|
||||
}
|
||||
launchpad_bootstrap_pending_nodes = {
|
||||
"go_live_summary": int(go_live_summary_payload.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
|
||||
"stack_diagnosis": int(stack_diagnosis.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
|
||||
"driver_feed": int(driver_feed_summary.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
|
||||
"codex_brief": int(codex_brief_summary.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
|
||||
}
|
||||
launchpad_acceptance_ready_nodes = {
|
||||
"go_live_summary": int(go_live_summary_payload.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
|
||||
"stack_diagnosis": int(stack_diagnosis.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
|
||||
"driver_feed": int(driver_feed_summary.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
|
||||
"codex_brief": int(codex_brief_summary.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
|
||||
}
|
||||
available_alignment_sources = {
|
||||
source
|
||||
for source, payload_value in {
|
||||
"go_live_summary": go_live_summary_payload,
|
||||
"stack_diagnosis": stack_diagnosis_payload,
|
||||
"driver_feed": driver_feed_payload,
|
||||
"codex_brief": codex_brief_payload,
|
||||
}.items()
|
||||
if isinstance(payload_value, dict) and payload_value
|
||||
}
|
||||
launchpad_alignment = {
|
||||
"consistent": True,
|
||||
"target_node_code_consistent": True,
|
||||
"recovery_label_consistent": True,
|
||||
"recovery_summary_consistent": True,
|
||||
"bootstrap_pending_consistent": True,
|
||||
"acceptance_ready_consistent": True,
|
||||
"available_sources": sorted(available_alignment_sources),
|
||||
"target_node_codes": launchpad_target_node_codes,
|
||||
"recovery_labels": launchpad_recovery_labels,
|
||||
"recovery_summaries": launchpad_recovery_summaries,
|
||||
"bootstrap_pending_nodes": launchpad_bootstrap_pending_nodes,
|
||||
"acceptance_ready_nodes": launchpad_acceptance_ready_nodes,
|
||||
}
|
||||
launchpad_alignment["target_node_code_consistent"] = len(nonempty_distinct({
|
||||
source: value for source, value in launchpad_target_node_codes.items() if source in available_alignment_sources
|
||||
})) <= 1
|
||||
launchpad_alignment["recovery_label_consistent"] = len(nonempty_distinct({
|
||||
source: value for source, value in launchpad_recovery_labels.items() if source in available_alignment_sources
|
||||
})) <= 1
|
||||
launchpad_alignment["recovery_summary_consistent"] = len(nonempty_distinct({
|
||||
source: value for source, value in launchpad_recovery_summaries.items() if source in available_alignment_sources
|
||||
})) <= 1
|
||||
launchpad_alignment["bootstrap_pending_consistent"] = len(
|
||||
distinct_ints(launchpad_bootstrap_pending_nodes, available_alignment_sources)
|
||||
) <= 1
|
||||
launchpad_alignment["acceptance_ready_consistent"] = len(
|
||||
distinct_ints(launchpad_acceptance_ready_nodes, available_alignment_sources)
|
||||
) <= 1
|
||||
launchpad_alignment["consistent"] = all([
|
||||
bool(launchpad_alignment["target_node_code_consistent"]),
|
||||
bool(launchpad_alignment["recovery_label_consistent"]),
|
||||
bool(launchpad_alignment["recovery_summary_consistent"]),
|
||||
bool(launchpad_alignment["bootstrap_pending_consistent"]),
|
||||
bool(launchpad_alignment["acceptance_ready_consistent"]),
|
||||
])
|
||||
|
||||
env_audit_status = str(env_audit_summary.get("status") or "").strip()
|
||||
env_audit_runtime_may_need_restart = bool(((env_audit_summary.get("runtime") or {}).get("runtime_may_need_restart", False)))
|
||||
review_status_hint = "ready"
|
||||
review_headline = "核心上线证据已经齐备,可进入最终人工复核或正式发布门禁。"
|
||||
if failures or env_audit_status == "blocked":
|
||||
review_status_hint = "blocked"
|
||||
review_headline = "当前导出包仍有关键阻断项,不能直接作为正式上线终稿。"
|
||||
elif env_audit_status == "attention" or not launchpad_alignment["consistent"]:
|
||||
review_status_hint = "attention"
|
||||
if not launchpad_alignment["consistent"]:
|
||||
review_headline = "核心报告虽已导出,但 launchpad 摘要在不同 surface 之间存在漂移,建议先重新复核。"
|
||||
else:
|
||||
review_headline = "核心报告已齐,但环境审计仍提示存在待收口事项。"
|
||||
|
||||
payload = {
|
||||
"report_dir": str(report_dir),
|
||||
"mainland_base_url": mainland_base_url,
|
||||
"overseas_base_url": overseas_base_url,
|
||||
"artifacts": artifacts,
|
||||
"failures": failures,
|
||||
"summary": {
|
||||
"ok": len(failures) == 0,
|
||||
"artifact_total": len(artifacts),
|
||||
"failure_total": len(failures),
|
||||
"review_status_hint": review_status_hint,
|
||||
"review_headline": review_headline,
|
||||
"env_audit_status": env_audit_status,
|
||||
"env_audit_runtime_may_need_restart": env_audit_runtime_may_need_restart,
|
||||
"launchpad_recommended_target_node_code": first_nonempty(
|
||||
go_live_summary_payload.get("launchpad_recommended_target_node_code"),
|
||||
stack_diagnosis.get("launchpad_recommended_target_node_code"),
|
||||
driver_feed_summary.get("launchpad_recommended_target_node_code"),
|
||||
codex_brief_summary.get("launchpad_recommended_target_node_code"),
|
||||
),
|
||||
"launchpad_recommended_recovery_label": first_nonempty(
|
||||
go_live_summary_payload.get("launchpad_recommended_recovery_label"),
|
||||
stack_diagnosis.get("launchpad_recommended_recovery_label"),
|
||||
driver_feed_summary.get("launchpad_recommended_recovery_label"),
|
||||
codex_brief_summary.get("launchpad_recommended_recovery_label"),
|
||||
),
|
||||
"launchpad_recommended_recovery_summary": first_nonempty(
|
||||
go_live_summary_payload.get("launchpad_recommended_recovery_summary"),
|
||||
stack_diagnosis.get("launchpad_recommended_recovery_summary"),
|
||||
driver_feed_summary.get("launchpad_recommended_recovery_summary"),
|
||||
codex_brief_summary.get("launchpad_recommended_recovery_summary"),
|
||||
),
|
||||
"launchpad_onboarding_bootstrap_pending_nodes": max(launchpad_bootstrap_pending_nodes.values() or [0]),
|
||||
"launchpad_onboarding_acceptance_ready_nodes": max(launchpad_acceptance_ready_nodes.values() or [0]),
|
||||
"launchpad_alignment": launchpad_alignment,
|
||||
"recommended_reading_order": [
|
||||
"00_env_audit.txt",
|
||||
"01_go_live_check_summary.txt",
|
||||
"02_go_live_summary.json",
|
||||
"03_stack_diagnosis.json",
|
||||
"04_driver_feed.json",
|
||||
"05_codex_brief.json",
|
||||
"06_release_launchpad.json",
|
||||
"07_doctor_decision.json",
|
||||
"08_doctor_export_stdout.txt",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
manifest_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
135
domain-api/deploy/multi-region/init_ops_center_config.sh
Executable file
135
domain-api/deploy/multi-region/init_ops_center_config.sh
Executable file
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
TEMPLATE_PATH="${SCRIPT_DIR}/templates/domaincheck-ops-center.env.example"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
usage:
|
||||
bash domain-api/deploy/multi-region/init_ops_center_config.sh \
|
||||
[config_path] [mainland_api_base_url] [overseas_api_base_url] [control_plane_base_url]
|
||||
|
||||
examples:
|
||||
bash domain-api/deploy/multi-region/init_ops_center_config.sh
|
||||
bash domain-api/deploy/multi-region/init_ops_center_config.sh /etc/default/domaincheck-ops-center \
|
||||
http://121.204.244.188:8100 \
|
||||
http://152.53.37.118:8100 \
|
||||
https://api.example.com
|
||||
|
||||
notes:
|
||||
- existing config files are preserved unless FORCE=true
|
||||
- advanced fields can be overridden with env vars:
|
||||
OPS_ACTIVE_PROFILE
|
||||
OPS_PROFILE_NAMES
|
||||
OPS_DEFAULT_REQUESTED_BY
|
||||
OPS_DEFAULT_ISSUED_BY
|
||||
OPS_DEFAULT_EXPIRES_IN_HOURS
|
||||
OPS_DEFAULT_ROOT_DIR
|
||||
OPS_REPORT_ROOT
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "-h" || "${1:-}" == "--help" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TARGET_PATH="${1:-/etc/default/domaincheck-ops-center}"
|
||||
MAINLAND_API_BASE_URL="${2:-${OPS_MAINLAND_API_BASE_URL:-http://127.0.0.1:8100}}"
|
||||
OVERSEAS_API_BASE_URL="${3:-${OPS_OVERSEAS_API_BASE_URL:-http://127.0.0.1:8100}}"
|
||||
CONTROL_PLANE_BASE_URL="${4:-${OPS_DEFAULT_CONTROL_PLANE_BASE_URL:-}}"
|
||||
|
||||
ACTIVE_PROFILE="${OPS_ACTIVE_PROFILE:-default}"
|
||||
PROFILE_NAMES="${OPS_PROFILE_NAMES:-${ACTIVE_PROFILE}}"
|
||||
DEFAULT_REQUESTED_BY="${OPS_DEFAULT_REQUESTED_BY:-cli/ops-center}"
|
||||
DEFAULT_ISSUED_BY="${OPS_DEFAULT_ISSUED_BY:-ops-center}"
|
||||
DEFAULT_EXPIRES_IN_HOURS="${OPS_DEFAULT_EXPIRES_IN_HOURS:-72}"
|
||||
DEFAULT_ROOT_DIR="${OPS_DEFAULT_ROOT_DIR:-/opt/domaincheck}"
|
||||
REPORT_ROOT="${OPS_REPORT_ROOT:-}"
|
||||
NODE_AGENT_SERVICE_NAME="${OPS_NODE_AGENT_SERVICE_NAME:-domaincheck-node-agent}"
|
||||
NODE_AGENT_ENV_FILE="${OPS_NODE_AGENT_ENV_FILE:-/etc/default/domaincheck-node-agent}"
|
||||
FORCE_WRITE="${FORCE:-false}"
|
||||
|
||||
if [[ ! -f "${TEMPLATE_PATH}" ]]; then
|
||||
echo "template not found: ${TEMPLATE_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$(dirname "${TARGET_PATH}")"
|
||||
|
||||
if [[ ! -f "${TARGET_PATH}" || "${FORCE_WRITE}" == "true" ]]; then
|
||||
cp "${TEMPLATE_PATH}" "${TARGET_PATH}"
|
||||
fi
|
||||
|
||||
upsert_key() {
|
||||
local key="$1"
|
||||
local value="$2"
|
||||
if grep -q "^${key}=" "${TARGET_PATH}"; then
|
||||
sed -i "s#^${key}=.*#${key}=${value}#" "${TARGET_PATH}"
|
||||
else
|
||||
printf '%s=%s\n' "${key}" "${value}" >>"${TARGET_PATH}"
|
||||
fi
|
||||
}
|
||||
|
||||
upsert_commented_key() {
|
||||
local key="$1"
|
||||
local value="$2"
|
||||
if [[ -z "${value}" ]]; then
|
||||
return 0
|
||||
fi
|
||||
upsert_key "${key}" "${value}"
|
||||
}
|
||||
|
||||
upsert_key "OPS_ACTIVE_PROFILE" "${ACTIVE_PROFILE}"
|
||||
upsert_key "OPS_PROFILE_NAMES" "${PROFILE_NAMES}"
|
||||
upsert_key "OPS_MAINLAND_API_BASE_URL" "${MAINLAND_API_BASE_URL}"
|
||||
upsert_key "OPS_OVERSEAS_API_BASE_URL" "${OVERSEAS_API_BASE_URL}"
|
||||
upsert_key "OPS_DEFAULT_CONTROL_PLANE_BASE_URL" "${CONTROL_PLANE_BASE_URL}"
|
||||
upsert_key "OPS_DEFAULT_REQUESTED_BY" "${DEFAULT_REQUESTED_BY}"
|
||||
upsert_key "OPS_DEFAULT_ISSUED_BY" "${DEFAULT_ISSUED_BY}"
|
||||
upsert_key "OPS_DEFAULT_EXPIRES_IN_HOURS" "${DEFAULT_EXPIRES_IN_HOURS}"
|
||||
upsert_key "OPS_DEFAULT_ROOT_DIR" "${DEFAULT_ROOT_DIR}"
|
||||
upsert_key "OPS_NODE_AGENT_SERVICE_NAME" "${NODE_AGENT_SERVICE_NAME}"
|
||||
upsert_key "OPS_NODE_AGENT_ENV_FILE" "${NODE_AGENT_ENV_FILE}"
|
||||
upsert_commented_key "OPS_REPORT_ROOT" "${REPORT_ROOT}"
|
||||
|
||||
python3 - <<'PY' "${TARGET_PATH}"
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
config_path = Path(sys.argv[1])
|
||||
keys = [
|
||||
"OPS_ACTIVE_PROFILE",
|
||||
"OPS_PROFILE_NAMES",
|
||||
"OPS_MAINLAND_API_BASE_URL",
|
||||
"OPS_OVERSEAS_API_BASE_URL",
|
||||
"OPS_DEFAULT_CONTROL_PLANE_BASE_URL",
|
||||
"OPS_DEFAULT_REQUESTED_BY",
|
||||
"OPS_DEFAULT_ISSUED_BY",
|
||||
"OPS_DEFAULT_EXPIRES_IN_HOURS",
|
||||
"OPS_DEFAULT_ROOT_DIR",
|
||||
"OPS_NODE_AGENT_SERVICE_NAME",
|
||||
"OPS_NODE_AGENT_ENV_FILE",
|
||||
"OPS_REPORT_ROOT",
|
||||
]
|
||||
values = {}
|
||||
for line in config_path.read_text(encoding="utf-8").splitlines():
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, value = line.split("=", 1)
|
||||
if key in keys:
|
||||
values[key] = value
|
||||
|
||||
print(json.dumps({
|
||||
"ok": True,
|
||||
"config_path": str(config_path),
|
||||
"summary": values,
|
||||
"next_commands": [
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh config",
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export",
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh agent-check",
|
||||
],
|
||||
}, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
@@ -100,7 +100,7 @@ REDIS_DB=0
|
||||
EOF
|
||||
|
||||
cd "${DOMAIN_API_DIR}"
|
||||
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
|
||||
bash domain-api/deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
|
||||
|
||||
cat >/etc/default/domaincheck-worker <<EOF
|
||||
WORKER_MODE=linux-systemd
|
||||
@@ -138,4 +138,4 @@ chown -R www:www "${REPO_DIR}"
|
||||
|
||||
echo
|
||||
echo "国内 controller 部署完成,建议继续执行:"
|
||||
echo "cd ${DOMAIN_API_DIR} && bash deploy/multi-region/check_mainland_controller.sh"
|
||||
echo "cd ${DOMAIN_API_DIR} && bash domain-api/deploy/multi-region/check_mainland_controller.sh"
|
||||
|
||||
@@ -101,7 +101,7 @@ REDIS_DB=0
|
||||
EOF
|
||||
|
||||
cd "${DOMAIN_API_DIR}"
|
||||
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
|
||||
bash domain-api/deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
|
||||
|
||||
cat >/etc/default/domaincheck-worker <<EOF
|
||||
WORKER_MODE=linux-systemd
|
||||
|
||||
@@ -185,6 +185,15 @@ sed -i 's#^NODE_ROLE=.*#NODE_ROLE=control#' /etc/default/domaincheck-api
|
||||
sed -i 's#^SYNC_PUSH_ENABLED=.*#SYNC_PUSH_ENABLED=false#' /etc/default/domaincheck-api
|
||||
sed -i "s#^CORS_ORIGINS=.*#CORS_ORIGINS='[\\\"https://${ADMIN_DOMAIN}\\\",\\\"http://127.0.0.1:3201\\\",\\\"http://localhost:3201\\\"]'#" /etc/default/domaincheck-api
|
||||
|
||||
OPS_DEFAULT_REQUESTED_BY="cli/${OVERSEAS_NODE_CODE}" \
|
||||
OPS_DEFAULT_ISSUED_BY="${OVERSEAS_NODE_CODE}" \
|
||||
OPS_DEFAULT_ROOT_DIR="${BASE_DIR}" \
|
||||
bash "${DOMAIN_API_DIR}/deploy/multi-region/init_ops_center_config.sh" \
|
||||
/etc/default/domaincheck-ops-center \
|
||||
http://127.0.0.1:8100 \
|
||||
http://127.0.0.1:8100 \
|
||||
"https://${API_DOMAIN}" >/dev/null
|
||||
|
||||
cp "${DOMAIN_API_DIR}/deploy/systemd/domain-api.service" /etc/systemd/system/domaincheck-api.service
|
||||
cp "${DOMAIN_API_DIR}/deploy/systemd/domain-worker.service" /etc/systemd/system/domaincheck-worker.service
|
||||
|
||||
@@ -206,3 +215,5 @@ echo "国外主控机部署完成,建议继续执行:"
|
||||
echo "curl http://127.0.0.1:8100/health"
|
||||
echo "curl http://127.0.0.1:8100/api/v1/runtime/preflight"
|
||||
echo "curl http://127.0.0.1:8100/api/v1/runtime/readiness"
|
||||
echo "bash ${DOMAIN_API_DIR}/deploy/multi-region/drive_ops_center.sh config"
|
||||
echo "如后续接入大陆 controller,请记得修改 /etc/default/domaincheck-ops-center 中的 OPS_MAINLAND_API_BASE_URL"
|
||||
|
||||
106
domain-api/deploy/multi-region/lib_ops_center.sh
Normal file
106
domain-api/deploy/multi-region/lib_ops_center.sh
Normal file
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
ops_center_normalize_profile_key() {
|
||||
local raw_value="${1:-default}"
|
||||
printf '%s' "${raw_value}" | tr '[:lower:]-./' '[:upper:]____' | tr -cd 'A-Z0-9_'
|
||||
}
|
||||
|
||||
ops_center_load_config() {
|
||||
local caller_dir="${1:-}"
|
||||
if [[ -z "${caller_dir}" ]]; then
|
||||
caller_dir="$(pwd)"
|
||||
fi
|
||||
|
||||
local env_active_profile="${OPS_ACTIVE_PROFILE:-}"
|
||||
local env_profile_names="${OPS_PROFILE_NAMES:-}"
|
||||
local env_report_root="${OPS_REPORT_ROOT:-}"
|
||||
local env_mainland_api_base_url="${OPS_MAINLAND_API_BASE_URL:-}"
|
||||
local env_overseas_api_base_url="${OPS_OVERSEAS_API_BASE_URL:-}"
|
||||
local env_control_plane_base_url="${OPS_DEFAULT_CONTROL_PLANE_BASE_URL:-}"
|
||||
local env_requested_by="${OPS_DEFAULT_REQUESTED_BY:-}"
|
||||
local env_issued_by="${OPS_DEFAULT_ISSUED_BY:-}"
|
||||
local env_expires_in_hours="${OPS_DEFAULT_EXPIRES_IN_HOURS:-}"
|
||||
local env_root_dir="${OPS_DEFAULT_ROOT_DIR:-}"
|
||||
local env_node_agent_service_name="${OPS_NODE_AGENT_SERVICE_NAME:-}"
|
||||
local env_node_agent_env_file="${OPS_NODE_AGENT_ENV_FILE:-}"
|
||||
|
||||
OPS_CENTER_SCRIPT_DIR="${caller_dir}"
|
||||
OPS_CENTER_API_DIR="$(cd "${OPS_CENTER_SCRIPT_DIR}/../.." && pwd)"
|
||||
OPS_CENTER_PROJECT_ROOT="$(cd "${OPS_CENTER_API_DIR}/.." && pwd)"
|
||||
OPS_CENTER_RUNTIME_DIR="${OPS_CENTER_API_DIR}/runtime"
|
||||
OPS_CENTER_REPORT_ROOT_DEFAULT="${OPS_CENTER_RUNTIME_DIR}/ops-center-reports"
|
||||
OPS_CENTER_CONFIG_FILE="${DOMAINCHECK_OPS_CENTER_ENV:-${OPS_CENTER_ENV_FILE:-/etc/default/domaincheck-ops-center}}"
|
||||
|
||||
if [[ -f "${OPS_CENTER_CONFIG_FILE}" ]]; then
|
||||
# shellcheck disable=SC1090
|
||||
source "${OPS_CENTER_CONFIG_FILE}"
|
||||
fi
|
||||
|
||||
OPS_CENTER_ACTIVE_PROFILE="${OPS_CENTER_PROFILE:-${DOMAINCHECK_OPS_CENTER_PROFILE:-${env_active_profile:-${OPS_ACTIVE_PROFILE:-default}}}}"
|
||||
OPS_CENTER_ACTIVE_PROFILE_KEY="$(ops_center_normalize_profile_key "${OPS_CENTER_ACTIVE_PROFILE}")"
|
||||
OPS_CENTER_PROFILE_NAMES="${env_profile_names:-${OPS_PROFILE_NAMES:-${OPS_CENTER_ACTIVE_PROFILE}}}"
|
||||
OPS_CENTER_REPORT_ROOT="${env_report_root:-${OPS_REPORT_ROOT:-${OPS_CENTER_REPORT_ROOT_DEFAULT}}}"
|
||||
OPS_CENTER_ENV_MAINLAND_API_BASE_URL="${env_mainland_api_base_url}"
|
||||
OPS_CENTER_ENV_OVERSEAS_API_BASE_URL="${env_overseas_api_base_url}"
|
||||
OPS_CENTER_ENV_CONTROL_PLANE_BASE_URL="${env_control_plane_base_url}"
|
||||
OPS_CENTER_ENV_REQUESTED_BY="${env_requested_by}"
|
||||
OPS_CENTER_ENV_ISSUED_BY="${env_issued_by}"
|
||||
OPS_CENTER_ENV_EXPIRES_IN_HOURS="${env_expires_in_hours}"
|
||||
OPS_CENTER_ENV_ROOT_DIR="${env_root_dir}"
|
||||
OPS_CENTER_ENV_NODE_AGENT_SERVICE_NAME="${env_node_agent_service_name}"
|
||||
OPS_CENTER_ENV_NODE_AGENT_ENV_FILE="${env_node_agent_env_file}"
|
||||
}
|
||||
|
||||
ops_center_resolve_profile_value() {
|
||||
local suffix="$1"
|
||||
local fallback_value="${2:-}"
|
||||
local profile_key="${OPS_CENTER_ACTIVE_PROFILE_KEY:-DEFAULT}"
|
||||
local variable_name="OPS_PROFILE_${profile_key}_${suffix}"
|
||||
local profile_value="${!variable_name:-}"
|
||||
if [[ -n "${profile_value}" ]]; then
|
||||
printf '%s' "${profile_value}"
|
||||
else
|
||||
printf '%s' "${fallback_value}"
|
||||
fi
|
||||
}
|
||||
|
||||
ops_center_resolve_mainland_api_base_url() {
|
||||
ops_center_resolve_profile_value "MAINLAND_API_BASE_URL" "${OPS_CENTER_ENV_MAINLAND_API_BASE_URL:-${OPS_MAINLAND_API_BASE_URL:-http://127.0.0.1:8100}}"
|
||||
}
|
||||
|
||||
ops_center_resolve_overseas_api_base_url() {
|
||||
ops_center_resolve_profile_value "OVERSEAS_API_BASE_URL" "${OPS_CENTER_ENV_OVERSEAS_API_BASE_URL:-${OPS_OVERSEAS_API_BASE_URL:-http://127.0.0.1:8100}}"
|
||||
}
|
||||
|
||||
ops_center_resolve_control_plane_base_url() {
|
||||
ops_center_resolve_profile_value "DEFAULT_CONTROL_PLANE_BASE_URL" "${OPS_CENTER_ENV_CONTROL_PLANE_BASE_URL:-${OPS_DEFAULT_CONTROL_PLANE_BASE_URL:-}}"
|
||||
}
|
||||
|
||||
ops_center_resolve_requested_by() {
|
||||
ops_center_resolve_profile_value "DEFAULT_REQUESTED_BY" "${OPS_CENTER_ENV_REQUESTED_BY:-${OPS_DEFAULT_REQUESTED_BY:-cli/ops-center}}"
|
||||
}
|
||||
|
||||
ops_center_resolve_issued_by() {
|
||||
ops_center_resolve_profile_value "DEFAULT_ISSUED_BY" "${OPS_CENTER_ENV_ISSUED_BY:-${OPS_DEFAULT_ISSUED_BY:-ops-center}}"
|
||||
}
|
||||
|
||||
ops_center_resolve_expires_in_hours() {
|
||||
ops_center_resolve_profile_value "DEFAULT_EXPIRES_IN_HOURS" "${OPS_CENTER_ENV_EXPIRES_IN_HOURS:-${OPS_DEFAULT_EXPIRES_IN_HOURS:-72}}"
|
||||
}
|
||||
|
||||
ops_center_resolve_root_dir() {
|
||||
ops_center_resolve_profile_value "DEFAULT_ROOT_DIR" "${OPS_CENTER_ENV_ROOT_DIR:-${OPS_DEFAULT_ROOT_DIR:-/opt/domaincheck}}"
|
||||
}
|
||||
|
||||
ops_center_resolve_node_agent_service_name() {
|
||||
ops_center_resolve_profile_value "NODE_AGENT_SERVICE_NAME" "${OPS_CENTER_ENV_NODE_AGENT_SERVICE_NAME:-${OPS_NODE_AGENT_SERVICE_NAME:-domaincheck-node-agent}}"
|
||||
}
|
||||
|
||||
ops_center_resolve_node_agent_env_file() {
|
||||
ops_center_resolve_profile_value "NODE_AGENT_ENV_FILE" "${OPS_CENTER_ENV_NODE_AGENT_ENV_FILE:-${OPS_NODE_AGENT_ENV_FILE:-/etc/default/domaincheck-node-agent}}"
|
||||
}
|
||||
|
||||
ops_center_ensure_report_dir() {
|
||||
local report_dir="${1:-${OPS_CENTER_REPORT_ROOT_DEFAULT}}"
|
||||
mkdir -p "${report_dir}"
|
||||
}
|
||||
317
domain-api/deploy/multi-region/review_go_live_bundle.sh
Executable file
317
domain-api/deploy/multi-region/review_go_live_bundle.sh
Executable file
@@ -0,0 +1,317 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
TARGET_PATH="${1:-}"
|
||||
|
||||
if [[ -z "${TARGET_PATH}" ]]; then
|
||||
echo "usage: bash domain-api/deploy/multi-region/review_go_live_bundle.sh <manifest.json|report_dir>" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ -d "${TARGET_PATH}" ]]; then
|
||||
MANIFEST_PATH="${TARGET_PATH%/}/manifest.json"
|
||||
else
|
||||
MANIFEST_PATH="${TARGET_PATH}"
|
||||
fi
|
||||
|
||||
if [[ ! -f "${MANIFEST_PATH}" ]]; then
|
||||
echo "manifest not found: ${MANIFEST_PATH}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python3 - "${MANIFEST_PATH}" <<'PY'
|
||||
import json
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
manifest_path = pathlib.Path(sys.argv[1])
|
||||
payload = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
|
||||
artifacts = payload.get("artifacts") or []
|
||||
failures = payload.get("failures") or []
|
||||
summary = payload.get("summary") or {}
|
||||
|
||||
critical_keys = {"go_live_summary", "stack_diagnosis", "driver_feed", "codex_brief", "release_launchpad"}
|
||||
artifact_by_key = {str(item.get("key") or ""): item for item in artifacts if isinstance(item, dict)}
|
||||
critical_failures = [key for key in critical_keys if not bool((artifact_by_key.get(key) or {}).get("ok", False))]
|
||||
noncritical_failures = [key for key in failures if key not in critical_keys]
|
||||
env_audit_artifact = artifact_by_key.get("env_audit") or {}
|
||||
|
||||
env_audit_status = ""
|
||||
env_audit_headline = ""
|
||||
env_audit_missing_items = []
|
||||
env_audit_recommended_actions = []
|
||||
env_audit_blocking = False
|
||||
env_audit_attention = False
|
||||
env_audit_runtime_may_need_restart = False
|
||||
go_live_summary_payload = {}
|
||||
stack_diagnosis_payload = {}
|
||||
driver_feed_payload = {}
|
||||
codex_brief_payload = {}
|
||||
|
||||
def load_json_artifact(key: str) -> dict:
|
||||
artifact = artifact_by_key.get(key) or {}
|
||||
artifact_path = pathlib.Path(str(artifact.get("path") or ""))
|
||||
if not artifact_path.is_file():
|
||||
return {}
|
||||
try:
|
||||
loaded = json.loads(artifact_path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return {}
|
||||
return loaded if isinstance(loaded, dict) else {}
|
||||
|
||||
env_audit_path = pathlib.Path(str(env_audit_artifact.get("path") or ""))
|
||||
if env_audit_path.is_file():
|
||||
raw_text = env_audit_path.read_text(encoding="utf-8", errors="replace")
|
||||
marker = "[8/8] condensed env audit summary"
|
||||
candidate_text = raw_text.split(marker, 1)[1].strip() if marker in raw_text else raw_text.strip()
|
||||
if candidate_text:
|
||||
try:
|
||||
env_payload = json.loads(candidate_text)
|
||||
except Exception:
|
||||
env_payload = {}
|
||||
env_audit_status = str(env_payload.get("status") or "")
|
||||
env_audit_headline = str(env_payload.get("headline") or "")
|
||||
env_audit_missing_items = list(env_payload.get("missing_items") or [])
|
||||
env_audit_recommended_actions = [str(item).strip() for item in list(env_payload.get("recommended_actions") or []) if str(item).strip()]
|
||||
env_audit_runtime_may_need_restart = bool(((env_payload.get("runtime") or {}).get("runtime_may_need_restart", False)))
|
||||
env_audit_blocking = env_audit_status == "blocked"
|
||||
env_audit_attention = env_audit_status == "attention"
|
||||
|
||||
go_live_summary_payload = load_json_artifact("go_live_summary")
|
||||
stack_diagnosis_payload = load_json_artifact("stack_diagnosis")
|
||||
driver_feed_payload = load_json_artifact("driver_feed")
|
||||
codex_brief_payload = load_json_artifact("codex_brief")
|
||||
|
||||
go_live_launchpad_target_node_code = str(go_live_summary_payload.get("launchpad_recommended_target_node_code") or "").strip()
|
||||
go_live_launchpad_recovery_label = str(go_live_summary_payload.get("launchpad_recommended_recovery_label") or "").strip()
|
||||
go_live_launchpad_recovery_summary = str(go_live_summary_payload.get("launchpad_recommended_recovery_summary") or "").strip()
|
||||
go_live_launchpad_bootstrap_pending_nodes = int(go_live_summary_payload.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0)
|
||||
go_live_launchpad_acceptance_ready_nodes = int(go_live_summary_payload.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0)
|
||||
stack_diagnosis = stack_diagnosis_payload.get("diagnosis") or {}
|
||||
stack_launchpad_target_node_code = str(stack_diagnosis.get("launchpad_recommended_target_node_code") or "").strip()
|
||||
stack_launchpad_recovery_label = str(stack_diagnosis.get("launchpad_recommended_recovery_label") or "").strip()
|
||||
stack_launchpad_recovery_summary = str(stack_diagnosis.get("launchpad_recommended_recovery_summary") or "").strip()
|
||||
stack_launchpad_bootstrap_pending_nodes = int(stack_diagnosis.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0)
|
||||
stack_launchpad_acceptance_ready_nodes = int(stack_diagnosis.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0)
|
||||
driver_feed_summary = driver_feed_payload.get("summary") or {}
|
||||
codex_brief_summary = codex_brief_payload.get("summary") or {}
|
||||
|
||||
|
||||
def nonempty_distinct(values: dict[str, str]) -> list[str]:
|
||||
return sorted({str(value or "").strip() for value in values.values() if str(value or "").strip()})
|
||||
|
||||
|
||||
def distinct_ints(values: dict[str, int], available_sources: set[str]) -> list[int]:
|
||||
return sorted({int(values[source]) for source in values if source in available_sources})
|
||||
|
||||
|
||||
launchpad_target_node_codes = {
|
||||
"go_live_summary": go_live_launchpad_target_node_code,
|
||||
"stack_diagnosis": stack_launchpad_target_node_code,
|
||||
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_target_node_code") or "").strip(),
|
||||
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_target_node_code") or "").strip(),
|
||||
}
|
||||
launchpad_recovery_labels = {
|
||||
"go_live_summary": go_live_launchpad_recovery_label,
|
||||
"stack_diagnosis": stack_launchpad_recovery_label,
|
||||
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_recovery_label") or "").strip(),
|
||||
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_recovery_label") or "").strip(),
|
||||
}
|
||||
launchpad_recovery_summaries = {
|
||||
"go_live_summary": go_live_launchpad_recovery_summary,
|
||||
"stack_diagnosis": stack_launchpad_recovery_summary,
|
||||
"driver_feed": str(driver_feed_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
|
||||
"codex_brief": str(codex_brief_summary.get("launchpad_recommended_recovery_summary") or "").strip(),
|
||||
}
|
||||
launchpad_bootstrap_pending_nodes = {
|
||||
"go_live_summary": go_live_launchpad_bootstrap_pending_nodes,
|
||||
"stack_diagnosis": stack_launchpad_bootstrap_pending_nodes,
|
||||
"driver_feed": int(driver_feed_summary.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
|
||||
"codex_brief": int(codex_brief_summary.get("launchpad_onboarding_bootstrap_pending_nodes", 0) or 0),
|
||||
}
|
||||
launchpad_acceptance_ready_nodes = {
|
||||
"go_live_summary": go_live_launchpad_acceptance_ready_nodes,
|
||||
"stack_diagnosis": stack_launchpad_acceptance_ready_nodes,
|
||||
"driver_feed": int(driver_feed_summary.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
|
||||
"codex_brief": int(codex_brief_summary.get("launchpad_onboarding_acceptance_ready_nodes", 0) or 0),
|
||||
}
|
||||
|
||||
available_alignment_sources = {
|
||||
source
|
||||
for source, payload_value in {
|
||||
"go_live_summary": go_live_summary_payload,
|
||||
"stack_diagnosis": stack_diagnosis_payload,
|
||||
"driver_feed": driver_feed_payload,
|
||||
"codex_brief": codex_brief_payload,
|
||||
}.items()
|
||||
if isinstance(payload_value, dict) and payload_value
|
||||
}
|
||||
|
||||
launchpad_target_consistent = len(nonempty_distinct({
|
||||
source: value for source, value in launchpad_target_node_codes.items() if source in available_alignment_sources
|
||||
})) <= 1
|
||||
launchpad_recovery_label_consistent = len(nonempty_distinct({
|
||||
source: value for source, value in launchpad_recovery_labels.items() if source in available_alignment_sources
|
||||
})) <= 1
|
||||
launchpad_recovery_summary_consistent = len(nonempty_distinct({
|
||||
source: value for source, value in launchpad_recovery_summaries.items() if source in available_alignment_sources
|
||||
})) <= 1
|
||||
launchpad_bootstrap_pending_consistent = len(distinct_ints(launchpad_bootstrap_pending_nodes, available_alignment_sources)) <= 1
|
||||
launchpad_acceptance_ready_consistent = len(distinct_ints(launchpad_acceptance_ready_nodes, available_alignment_sources)) <= 1
|
||||
launchpad_alignment_consistent = all([
|
||||
launchpad_target_consistent,
|
||||
launchpad_recovery_label_consistent,
|
||||
launchpad_recovery_summary_consistent,
|
||||
launchpad_bootstrap_pending_consistent,
|
||||
launchpad_acceptance_ready_consistent,
|
||||
])
|
||||
|
||||
node_agent_env_missing = any(
|
||||
str(item).strip() == "missing_env:/etc/default/domaincheck-node-agent"
|
||||
for item in env_audit_missing_items
|
||||
)
|
||||
|
||||
if critical_failures or env_audit_blocking:
|
||||
status = "blocked"
|
||||
if critical_failures and env_audit_blocking:
|
||||
headline = "关键报告存在缺失,且环境审计已给出阻断结论,当前不适合上线。"
|
||||
elif critical_failures:
|
||||
headline = "上线证据包仍缺关键报告,暂不建议直接宣称收口完成。"
|
||||
else:
|
||||
headline = "环境审计已给出阻断结论,需先修复现场环境后再进入发布门禁。"
|
||||
elif noncritical_failures or env_audit_attention or not launchpad_alignment_consistent:
|
||||
status = "attention"
|
||||
if not launchpad_alignment_consistent:
|
||||
headline = "核心报告已齐,但 launchpad 摘要在不同 surface 之间存在漂移,建议先复核再发版。"
|
||||
elif noncritical_failures and env_audit_attention:
|
||||
headline = "核心报告已齐,但补充报告和环境审计都提示仍有缺口,建议先复核再发版。"
|
||||
elif noncritical_failures:
|
||||
headline = "核心报告已齐,但仍有补充报告超时或失败,建议先复核再发版。"
|
||||
else:
|
||||
headline = "核心报告已齐,但环境审计仍提示基础缺口,建议补齐后再发版。"
|
||||
else:
|
||||
status = "ready"
|
||||
headline = "核心上线证据已经齐备,可进入最终人工复核或正式发布门禁。"
|
||||
|
||||
result = {
|
||||
"report_dir": str(payload.get("report_dir") or manifest_path.parent),
|
||||
"status": status,
|
||||
"headline": headline,
|
||||
"artifact_total": int(summary.get("artifact_total", len(artifacts)) or len(artifacts)),
|
||||
"failure_total": int(summary.get("failure_total", len(failures)) or len(failures)),
|
||||
"critical_failures": critical_failures,
|
||||
"noncritical_failures": noncritical_failures,
|
||||
"env_audit": {
|
||||
"ok": bool(env_audit_artifact.get("ok", False)),
|
||||
"status": env_audit_status,
|
||||
"headline": env_audit_headline,
|
||||
"missing_items": env_audit_missing_items,
|
||||
"runtime_may_need_restart": env_audit_runtime_may_need_restart,
|
||||
"recommended_actions": env_audit_recommended_actions,
|
||||
},
|
||||
"go_live_summary": {
|
||||
"go_live_status": str(go_live_summary_payload.get("go_live_status") or "").strip(),
|
||||
"publish_status": str(go_live_summary_payload.get("publish_status") or "").strip(),
|
||||
"operator_title": str(go_live_summary_payload.get("operator_title") or "").strip(),
|
||||
"next_step_action_code": str(go_live_summary_payload.get("next_step_action_code") or "").strip(),
|
||||
"launchpad_recommended_target_node_code": go_live_launchpad_target_node_code,
|
||||
"launchpad_recommended_recovery_label": go_live_launchpad_recovery_label,
|
||||
"launchpad_recommended_recovery_summary": go_live_launchpad_recovery_summary,
|
||||
"launchpad_onboarding_bootstrap_pending_nodes": go_live_launchpad_bootstrap_pending_nodes,
|
||||
"launchpad_onboarding_acceptance_ready_nodes": go_live_launchpad_acceptance_ready_nodes,
|
||||
},
|
||||
"stack_diagnosis": {
|
||||
"stack_status": str((stack_diagnosis_payload.get("diagnosis") or {}).get("stack_status") or "").strip(),
|
||||
"operator_title": str(((stack_diagnosis_payload.get("diagnosis") or {}).get("operator_decision") or {}).get("title") or "").strip(),
|
||||
"next_step_action_code": str(((stack_diagnosis_payload.get("diagnosis") or {}).get("next_step") or {}).get("action_code") or "").strip(),
|
||||
"launchpad_recommended_target_node_code": stack_launchpad_target_node_code,
|
||||
"launchpad_recommended_recovery_label": stack_launchpad_recovery_label,
|
||||
"launchpad_recommended_recovery_summary": stack_launchpad_recovery_summary,
|
||||
"launchpad_onboarding_bootstrap_pending_nodes": stack_launchpad_bootstrap_pending_nodes,
|
||||
"launchpad_onboarding_acceptance_ready_nodes": stack_launchpad_acceptance_ready_nodes,
|
||||
},
|
||||
"driver_feed": {
|
||||
"headline": str(driver_feed_payload.get("headline") or "").strip(),
|
||||
"launch_status": str(((driver_feed_payload.get("automation_coverage") or {}).get("launch_status") or "")).strip(),
|
||||
"launch_ready": bool(((driver_feed_payload.get("automation_coverage") or {}).get("launch_ready", False))),
|
||||
"launchpad_recommended_target_node_code": str(
|
||||
((driver_feed_payload.get("summary") or {}).get("launchpad_recommended_target_node_code") or "")
|
||||
).strip(),
|
||||
"launchpad_recommended_recovery_label": str(
|
||||
((driver_feed_payload.get("summary") or {}).get("launchpad_recommended_recovery_label") or "")
|
||||
).strip(),
|
||||
},
|
||||
"codex_brief": {
|
||||
"headline": str(codex_brief_payload.get("headline") or "").strip(),
|
||||
"focus_entry_key": str(((codex_brief_payload.get("focus") or {}).get("entry_key") or "")).strip(),
|
||||
"launch_status": str(((codex_brief_payload.get("automation_coverage") or {}).get("launch_status") or "")).strip(),
|
||||
"launch_ready": bool(((codex_brief_payload.get("automation_coverage") or {}).get("launch_ready", False))),
|
||||
"launchpad_recommended_target_node_code": str(
|
||||
((codex_brief_payload.get("summary") or {}).get("launchpad_recommended_target_node_code") or "")
|
||||
).strip(),
|
||||
"launchpad_recommended_recovery_label": str(
|
||||
((codex_brief_payload.get("summary") or {}).get("launchpad_recommended_recovery_label") or "")
|
||||
).strip(),
|
||||
},
|
||||
"launchpad_alignment": {
|
||||
"consistent": launchpad_alignment_consistent,
|
||||
"target_node_code_consistent": launchpad_target_consistent,
|
||||
"recovery_label_consistent": launchpad_recovery_label_consistent,
|
||||
"recovery_summary_consistent": launchpad_recovery_summary_consistent,
|
||||
"bootstrap_pending_consistent": launchpad_bootstrap_pending_consistent,
|
||||
"acceptance_ready_consistent": launchpad_acceptance_ready_consistent,
|
||||
"available_sources": sorted(available_alignment_sources),
|
||||
"target_node_codes": launchpad_target_node_codes,
|
||||
"recovery_labels": launchpad_recovery_labels,
|
||||
"recovery_summaries": launchpad_recovery_summaries,
|
||||
"bootstrap_pending_nodes": launchpad_bootstrap_pending_nodes,
|
||||
"acceptance_ready_nodes": launchpad_acceptance_ready_nodes,
|
||||
},
|
||||
"recommended_reading_order": list(summary.get("recommended_reading_order") or []),
|
||||
"recommended_next_steps": [
|
||||
"先看 00_env_audit.txt,确认当前机器环境是否存在阻断或注意项",
|
||||
"先看 02_go_live_summary.json 和 03_stack_diagnosis.json",
|
||||
"再看 04_driver_feed.json 和 05_codex_brief.json",
|
||||
"若涉及发布门禁,再看 06_release_launchpad.json",
|
||||
"若 manifest 里仍有失败项,再看对应 artifact 原文件",
|
||||
]
|
||||
+ (
|
||||
[
|
||||
"当前缺口指向 Node Agent 未接管:先执行 agent-gap-export,拿到首个缺口节点的接管包",
|
||||
"随后执行 node-bootstrap-plan,为缺口节点生成可直接落地的接管脚本",
|
||||
"如果仓库代码已更新但 bootstrap-plan 输出仍偏旧,先重启控制面 API 再重新导出接管计划",
|
||||
]
|
||||
if node_agent_env_missing
|
||||
else []
|
||||
)
|
||||
+ (
|
||||
[
|
||||
"env-audit 已识别运行中 API 可能仍是旧代码:先执行 drive_ops_center.sh runtime-refresh-recover,按固定链路完成 API 重启与复检",
|
||||
"若仍需人工逐条确认,再执行 recommended_actions 里的 API 重启与复检命令"
|
||||
]
|
||||
if env_audit_runtime_may_need_restart
|
||||
else []
|
||||
)
|
||||
+ (
|
||||
[
|
||||
f"当前 launchpad 缺口已经收敛到节点 {go_live_launchpad_target_node_code},建议优先执行:{go_live_launchpad_recovery_label or '节点恢复'}",
|
||||
"先对照 02_go_live_summary.json、04_driver_feed.json、05_codex_brief.json 中的 launchpad 字段,确认三处摘要是否一致",
|
||||
(go_live_launchpad_recovery_summary or "如果 recovery_summary 仍为空,再回到 06_release_launchpad.json 查看 gap rows 详情"),
|
||||
]
|
||||
if go_live_launchpad_target_node_code
|
||||
else []
|
||||
)
|
||||
+ (
|
||||
[
|
||||
"当前 bundle 内的 launchpad 摘要在 go_live_summary / stack_diagnosis / driver_feed / codex_brief 之间存在不一致,先不要直接交付。",
|
||||
"优先复核 02_go_live_summary.json 与 03_stack_diagnosis.json,再检查 04_driver_feed.json 与 05_codex_brief.json 是否使用了同一版控制面数据。",
|
||||
"若只有局部摘要偏旧,先执行 drive_ops_center.sh runtime-refresh-recover,并重新导出 go-live bundle。",
|
||||
]
|
||||
if not launchpad_alignment_consistent
|
||||
else []
|
||||
),
|
||||
}
|
||||
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
PY
|
||||
@@ -0,0 +1,15 @@
|
||||
OPS_CONTROL_PLANE_BASE_URL=http://127.0.0.1:8100
|
||||
OPS_AGENT_TOKEN=
|
||||
OPS_AGENT_POLL_INTERVAL_SECONDS=5
|
||||
|
||||
NODE_CODE=mainland-worker-01
|
||||
NODE_REGION=mainland
|
||||
NODE_ROLE=worker
|
||||
|
||||
WORKER_SERVICE_NAME=domaincheck-worker
|
||||
API_SERVICE_NAME=domaincheck-api
|
||||
SYNC_AGENT_SERVICE_NAME=domaincheck-sync-agent
|
||||
NODE_AGENT_SERVICE_NAME=domaincheck-node-agent
|
||||
|
||||
OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","health.snapshot","logs.collect","diagnostics.collect","deploy.release"]
|
||||
OPS_AGENT_LABELS={}
|
||||
@@ -0,0 +1,49 @@
|
||||
# Overseas control-plane unified entrypoint config
|
||||
#
|
||||
# Recommended target path:
|
||||
# /etc/default/domaincheck-ops-center
|
||||
#
|
||||
# Usage:
|
||||
# export DOMAINCHECK_OPS_CENTER_ENV=/etc/default/domaincheck-ops-center
|
||||
# bash domain-api/deploy/multi-region/drive_ops_center.sh doctor
|
||||
|
||||
# Active profile name
|
||||
OPS_ACTIVE_PROFILE=default
|
||||
|
||||
# Optional profile list shown by `drive_ops_center.sh config`
|
||||
# OPS_PROFILE_NAMES=default,prod,staging
|
||||
|
||||
# Mainland controller API base URL
|
||||
OPS_MAINLAND_API_BASE_URL=http://127.0.0.1:8100
|
||||
|
||||
# Overseas control-plane API base URL
|
||||
OPS_OVERSEAS_API_BASE_URL=http://127.0.0.1:8100
|
||||
|
||||
# Public control-plane base URL used in generated node-agent bootstrap plans
|
||||
OPS_DEFAULT_CONTROL_PLANE_BASE_URL=https://api.example.com
|
||||
|
||||
# Default request identity used by CLI wrappers
|
||||
OPS_DEFAULT_REQUESTED_BY=cli/ops-center
|
||||
|
||||
# Default metadata used when generating node-agent bootstrap plans
|
||||
OPS_DEFAULT_ISSUED_BY=ops-center
|
||||
OPS_DEFAULT_EXPIRES_IN_HOURS=72
|
||||
OPS_DEFAULT_ROOT_DIR=/opt/domaincheck
|
||||
|
||||
# Optional report export root
|
||||
# OPS_REPORT_ROOT=/opt/domaincheck/domain-api/runtime/ops-center-reports
|
||||
|
||||
# Local node-agent status check defaults
|
||||
OPS_NODE_AGENT_SERVICE_NAME=domaincheck-node-agent
|
||||
OPS_NODE_AGENT_ENV_FILE=/etc/default/domaincheck-node-agent
|
||||
|
||||
# Optional per-profile overrides:
|
||||
# OPS_PROFILE_PROD_MAINLAND_API_BASE_URL=http://10.0.0.10:8100
|
||||
# OPS_PROFILE_PROD_OVERSEAS_API_BASE_URL=https://api.example.com
|
||||
# OPS_PROFILE_PROD_DEFAULT_CONTROL_PLANE_BASE_URL=https://api.example.com
|
||||
# OPS_PROFILE_PROD_DEFAULT_REQUESTED_BY=cli/ops-center-prod
|
||||
# OPS_PROFILE_PROD_DEFAULT_ISSUED_BY=ops-center-prod
|
||||
# OPS_PROFILE_PROD_DEFAULT_EXPIRES_IN_HOURS=72
|
||||
# OPS_PROFILE_PROD_DEFAULT_ROOT_DIR=/opt/domaincheck
|
||||
# OPS_PROFILE_PROD_NODE_AGENT_SERVICE_NAME=domaincheck-node-agent
|
||||
# OPS_PROFILE_PROD_NODE_AGENT_ENV_FILE=/etc/default/domaincheck-node-agent
|
||||
17
domain-api/deploy/systemd/domain-node-agent.service
Normal file
17
domain-api/deploy/systemd/domain-node-agent.service
Normal file
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=domainCheck Node Agent
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
WorkingDirectory=/opt/domaincheck/domain-api
|
||||
EnvironmentFile=-/etc/default/domaincheck-node-agent
|
||||
Environment="PATH=/home/www/.nvm/versions/node/v20.20.2/bin:/home/www/.local/bin:/home/www/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
|
||||
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m app.node_agent
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
User=www
|
||||
Group=www
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
98
domain-api/tests/test_build_info_service.py
Normal file
98
domain-api/tests/test_build_info_service.py
Normal file
@@ -0,0 +1,98 @@
|
||||
import json
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.build_info_service import (
|
||||
_expected_route_paths,
|
||||
get_runtime_build_info,
|
||||
remember_registered_route_paths,
|
||||
)
|
||||
|
||||
|
||||
class BuildInfoServiceTests(unittest.TestCase):
|
||||
def tearDown(self) -> None:
|
||||
remember_registered_route_paths([])
|
||||
|
||||
def test_runtime_build_info_prefers_configured_manifest_and_reports_registered_surface(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
manifest_path = Path(temp_dir) / "release_manifest.json"
|
||||
manifest_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"package_name": "domaincheck_release_demo",
|
||||
"generated_at": "2026-04-18T12:00:00",
|
||||
"commit_sha": "bd6bcb2",
|
||||
"commit_ref": "main",
|
||||
"sha256": "demo-sha256",
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
route_paths = list(_expected_route_paths().values())
|
||||
remember_registered_route_paths(route_paths)
|
||||
|
||||
with patch.multiple(
|
||||
settings,
|
||||
build_manifest_path=str(manifest_path),
|
||||
build_commit_sha="",
|
||||
build_commit_ref="",
|
||||
build_generated_at="",
|
||||
build_package_name="",
|
||||
build_checksum="",
|
||||
build_source_label="",
|
||||
):
|
||||
payload = get_runtime_build_info()
|
||||
|
||||
self.assertEqual("configured_manifest", payload["source"])
|
||||
self.assertEqual("domaincheck_release_demo", payload["package_name"])
|
||||
self.assertEqual("bd6bcb2", payload["commit_sha"])
|
||||
self.assertEqual("main", payload["commit_ref"])
|
||||
self.assertEqual("demo-sha256", payload["checksum"])
|
||||
self.assertEqual(str(manifest_path), payload["manifest_path"])
|
||||
self.assertTrue(payload["route_surface"]["surface_complete"])
|
||||
self.assertEqual([], payload["route_surface"]["missing_keys"])
|
||||
self.assertEqual("registered", payload["route_surface"]["mode"])
|
||||
self.assertIn("repository_capabilities", payload)
|
||||
self.assertIn("supports_install_command_block", payload["repository_capabilities"])
|
||||
self.assertIn("supports_multi_layout_bootstrap", payload["repository_capabilities"])
|
||||
|
||||
def test_runtime_build_info_env_override_wins_and_missing_routes_are_visible(self) -> None:
|
||||
with patch.multiple(
|
||||
settings,
|
||||
build_manifest_path="",
|
||||
build_commit_sha="246838ae4c07",
|
||||
build_commit_ref="main",
|
||||
build_generated_at="2026-04-18T12:10:00",
|
||||
build_package_name="domaincheck_release_env",
|
||||
build_checksum="env-sha256",
|
||||
build_source_label="env-override",
|
||||
):
|
||||
payload = get_runtime_build_info(route_paths=[_expected_route_paths()["ops_contracts"]])
|
||||
|
||||
self.assertEqual("env-override", payload["source"])
|
||||
self.assertEqual("domaincheck_release_env", payload["package_name"])
|
||||
self.assertEqual("246838ae4c07", payload["commit_sha"])
|
||||
self.assertEqual("main", payload["commit_ref"])
|
||||
self.assertEqual("env-sha256", payload["checksum"])
|
||||
self.assertFalse(payload["route_surface"]["surface_complete"])
|
||||
self.assertIn("ops_stack_diagnosis", payload["route_surface"]["missing_keys"])
|
||||
self.assertIn("ops_node_onboarding_bootstrap_preview", payload["route_surface"]["missing_keys"])
|
||||
self.assertIn("ops_node_onboarding_bootstrap_execute", payload["route_surface"]["missing_keys"])
|
||||
self.assertIn("ops_node_onboarding_acceptance_preview", payload["route_surface"]["missing_keys"])
|
||||
self.assertIn("ops_node_onboarding_acceptance_execute", payload["route_surface"]["missing_keys"])
|
||||
self.assertIn("ops_node_onboarding_recovery_preview", payload["route_surface"]["missing_keys"])
|
||||
self.assertIn("ops_node_onboarding_recovery_execute", payload["route_surface"]["missing_keys"])
|
||||
self.assertIn("ops_node_scene_log", payload["route_surface"]["missing_keys"])
|
||||
self.assertIn("runtime_build_info", payload["route_surface"]["missing_keys"])
|
||||
self.assertEqual("registered", payload["route_surface"]["mode"])
|
||||
self.assertIn("repository_capabilities", payload)
|
||||
self.assertIn("supports_install_command_block", payload["repository_capabilities"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
111
domain-api/tests/test_drive_ops_center_docs.py
Normal file
111
domain-api/tests/test_drive_ops_center_docs.py
Normal file
@@ -0,0 +1,111 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class DriveOpsCenterDocsTests(unittest.TestCase):
|
||||
def test_authoritative_docs_only_reference_supported_commands(self) -> None:
|
||||
repo_root = Path(__file__).resolve().parents[2]
|
||||
script_path = repo_root / "domain-api" / "deploy" / "multi-region" / "drive_ops_center.sh"
|
||||
doc_paths = [
|
||||
repo_root / "domain-api" / "deploy" / "multi-region" / "README.md",
|
||||
repo_root / "docs" / "25_domainCheck_海外单脑控制面上线收口总表.md",
|
||||
repo_root / "docs" / "26_domainCheck_发布前运行验证与交付模板.md",
|
||||
]
|
||||
|
||||
help_output = subprocess.run(
|
||||
["bash", str(script_path), "help"],
|
||||
cwd=repo_root,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout
|
||||
|
||||
supported_commands = self._extract_supported_commands(help_output)
|
||||
documented_commands = self._extract_documented_commands(doc_paths)
|
||||
|
||||
missing_commands = sorted(documented_commands - supported_commands)
|
||||
self.assertEqual(
|
||||
[],
|
||||
missing_commands,
|
||||
msg=(
|
||||
"These drive_ops_center.sh commands are referenced by authoritative docs "
|
||||
"but missing from the CLI help output: "
|
||||
+ ", ".join(missing_commands)
|
||||
),
|
||||
)
|
||||
self.assertIn("stack-diagnosis", documented_commands)
|
||||
self.assertIn("stack-diagnosis", supported_commands)
|
||||
|
||||
def test_stack_diagnosis_summary_argument_normalization_does_not_treat_summary_as_host(self) -> None:
|
||||
repo_root = Path(__file__).resolve().parents[2]
|
||||
script_path = repo_root / "domain-api" / "deploy" / "multi-region" / "drive_ops_center.sh"
|
||||
env = dict(os.environ)
|
||||
env["DOMAINCHECK_OPS_CENTER_ENV"] = "/tmp/domaincheck-ops-center-does-not-exist"
|
||||
env["OPS_MAINLAND_API_BASE_URL"] = "http://127.0.0.1:9"
|
||||
|
||||
result = subprocess.run(
|
||||
["bash", str(script_path), "stack-diagnosis", "summary"],
|
||||
cwd=repo_root,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
)
|
||||
|
||||
combined_output = f"{result.stdout}\n{result.stderr}"
|
||||
self.assertNotIn("Could not resolve host: summary", combined_output)
|
||||
self.assertNotIn("NameError", combined_output)
|
||||
self.assertIn("[9/9] condensed stack summary", combined_output)
|
||||
|
||||
def test_env_drift_recommends_stack_diagnosis_instead_of_stack_check(self) -> None:
|
||||
repo_root = Path(__file__).resolve().parents[2]
|
||||
script_path = repo_root / "domain-api" / "deploy" / "multi-region" / "check_env_drift.sh"
|
||||
script_text = script_path.read_text(encoding="utf-8")
|
||||
|
||||
self.assertIn(
|
||||
'recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary")',
|
||||
script_text,
|
||||
)
|
||||
self.assertNotIn(
|
||||
'recommended_actions.append("bash domain-api/deploy/multi-region/drive_ops_center.sh stack-check http://127.0.0.1:8100 summary")',
|
||||
script_text,
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _extract_supported_commands(help_output: str) -> set[str]:
|
||||
commands: set[str] = set()
|
||||
in_commands_section = False
|
||||
for raw_line in help_output.splitlines():
|
||||
line = raw_line.rstrip()
|
||||
if line.strip() == "commands:":
|
||||
in_commands_section = True
|
||||
continue
|
||||
if line.strip() == "examples:":
|
||||
break
|
||||
if not in_commands_section:
|
||||
continue
|
||||
stripped = line.strip()
|
||||
if not stripped:
|
||||
continue
|
||||
command = stripped.split()[0]
|
||||
if re.fullmatch(r"[a-z0-9-]+", command):
|
||||
commands.add(command)
|
||||
return commands
|
||||
|
||||
@staticmethod
|
||||
def _extract_documented_commands(doc_paths: list[Path]) -> set[str]:
|
||||
pattern = re.compile(r"drive_ops_center\.sh\s+([a-z0-9-]+)")
|
||||
commands: set[str] = set()
|
||||
for doc_path in doc_paths:
|
||||
text = doc_path.read_text(encoding="utf-8")
|
||||
commands.update(match.group(1) for match in pattern.finditer(text))
|
||||
return commands
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
284
domain-api/tests/test_node_agent_delivery_queue.py
Normal file
284
domain-api/tests/test_node_agent_delivery_queue.py
Normal file
@@ -0,0 +1,284 @@
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
from unittest.mock import patch
|
||||
|
||||
from app import node_agent
|
||||
|
||||
|
||||
class NodeAgentDeliveryQueueTests(unittest.TestCase):
|
||||
def test_base_payload_includes_delivery_queue_snapshot(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
|
||||
node_agent._ensure_queue_dirs()
|
||||
payload = node_agent._base_payload()
|
||||
|
||||
delivery_queue = payload["metadata"]["delivery_queue"]
|
||||
self.assertEqual("healthy", delivery_queue["state"])
|
||||
self.assertEqual(0, delivery_queue["pending_count"])
|
||||
self.assertEqual(0, delivery_queue["dead_letter_count"])
|
||||
|
||||
def test_job_event_network_failure_is_queued_for_retry(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir), patch.object(
|
||||
node_agent,
|
||||
"_post",
|
||||
side_effect=urllib.error.URLError("offline"),
|
||||
):
|
||||
result = node_agent._job_event(
|
||||
7,
|
||||
event_type="executor_received",
|
||||
message="accepted",
|
||||
client_event_id="evt-001",
|
||||
summary_text="已接单",
|
||||
focus_ref={"kind": "ops_job", "job_id": 7},
|
||||
)
|
||||
|
||||
self.assertEqual("queued", result["state"])
|
||||
pending_dir = os.path.join(temp_dir, "pending")
|
||||
pending_files = sorted(os.listdir(pending_dir))
|
||||
self.assertEqual(1, len(pending_files))
|
||||
with open(os.path.join(pending_dir, pending_files[0]), "r", encoding="utf-8") as handle:
|
||||
record = json.load(handle)
|
||||
self.assertEqual("evt-001", record["payload"]["client_event_id"])
|
||||
self.assertEqual("已接单", record["payload"]["summary_text"])
|
||||
self.assertEqual("ops_job", record["payload"]["focus_ref"]["kind"])
|
||||
self.assertEqual("job_event", record["kind"])
|
||||
|
||||
def test_job_complete_transport_failure_preserves_client_request_id(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir), patch.object(
|
||||
node_agent,
|
||||
"_post",
|
||||
side_effect=urllib.error.URLError("offline"),
|
||||
):
|
||||
result = node_agent._job_complete(
|
||||
9,
|
||||
status="success",
|
||||
stdout="done",
|
||||
stderr="",
|
||||
result={"ok": True},
|
||||
client_request_id="complete-001",
|
||||
duration_ms=321,
|
||||
summary_text="执行成功",
|
||||
focus_ref={"kind": "ops_job", "job_id": 9},
|
||||
)
|
||||
|
||||
self.assertEqual("queued", result["state"])
|
||||
pending_dir = os.path.join(temp_dir, "pending")
|
||||
pending_files = sorted(os.listdir(pending_dir))
|
||||
self.assertEqual(1, len(pending_files))
|
||||
with open(os.path.join(pending_dir, pending_files[0]), "r", encoding="utf-8") as handle:
|
||||
record = json.load(handle)
|
||||
self.assertEqual("complete-001", record["payload"]["client_request_id"])
|
||||
self.assertEqual(321, record["payload"]["duration_ms"])
|
||||
self.assertEqual("执行成功", record["payload"]["summary_text"])
|
||||
self.assertEqual("ops_job", record["payload"]["focus_ref"]["kind"])
|
||||
self.assertEqual("job_complete", record["kind"])
|
||||
|
||||
def test_flush_delivery_queue_delivers_pending_records(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
|
||||
record = node_agent._build_delivery_record(
|
||||
"job_event",
|
||||
"/api/v1/ops/agent/jobs/7/events",
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"event_type": "executor_received",
|
||||
"message": "accepted",
|
||||
"client_event_id": "evt-002",
|
||||
},
|
||||
"evt-002",
|
||||
)
|
||||
node_agent._store_pending_delivery(record)
|
||||
with patch.object(node_agent, "_post", return_value={"code": 0, "message": "ok"}):
|
||||
summary = node_agent._flush_delivery_queue(limit=10)
|
||||
|
||||
self.assertEqual(1, summary["delivered"])
|
||||
self.assertEqual([], os.listdir(os.path.join(temp_dir, "pending")))
|
||||
|
||||
def test_flush_delivery_queue_moves_semantic_failure_to_dead_letter(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
|
||||
record = node_agent._build_delivery_record(
|
||||
"job_complete",
|
||||
"/api/v1/ops/agent/jobs/9/complete",
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"status": "success",
|
||||
"client_request_id": "complete-002",
|
||||
},
|
||||
"complete-002",
|
||||
)
|
||||
node_agent._store_pending_delivery(record)
|
||||
with patch.object(
|
||||
node_agent,
|
||||
"_post",
|
||||
return_value={
|
||||
"code": 1,
|
||||
"message": "任务不存在或不属于当前节点",
|
||||
"detail_code": "ops_job_not_owned_by_agent",
|
||||
},
|
||||
):
|
||||
summary = node_agent._flush_delivery_queue(limit=10)
|
||||
|
||||
self.assertEqual(1, summary["dead_letter"])
|
||||
self.assertEqual([], os.listdir(os.path.join(temp_dir, "pending")))
|
||||
self.assertEqual(1, len(os.listdir(os.path.join(temp_dir, "dead-letter"))))
|
||||
|
||||
def test_replay_dead_letter_records_moves_record_back_to_pending_and_flushes(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
|
||||
record = node_agent._build_delivery_record(
|
||||
"job_event",
|
||||
"/api/v1/ops/agent/jobs/7/events",
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"event_type": "executor_received",
|
||||
"message": "accepted",
|
||||
"client_event_id": "evt-003",
|
||||
},
|
||||
"evt-003",
|
||||
)
|
||||
node_agent._store_dead_letter_delivery(
|
||||
record,
|
||||
reason="temporary failure",
|
||||
detail_code="transient_error",
|
||||
)
|
||||
with patch.object(node_agent, "_post", return_value={"code": 0, "message": "ok"}):
|
||||
ok, message, data = node_agent._execute_action(
|
||||
"delivery.queue.replay",
|
||||
{
|
||||
"selector": {"record_id": "evt-003"},
|
||||
"limit": 10,
|
||||
"flush_after_replay": True,
|
||||
"reason": "恢复后重放",
|
||||
},
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("已重放 1 条死信记录", message)
|
||||
self.assertEqual(1, data["replayed_total"])
|
||||
self.assertEqual(1, data["flush_summary"]["delivered"])
|
||||
self.assertEqual([], os.listdir(os.path.join(temp_dir, "dead-letter")))
|
||||
self.assertEqual([], os.listdir(os.path.join(temp_dir, "pending")))
|
||||
|
||||
def test_discard_dead_letter_records_archives_record(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir):
|
||||
record = node_agent._build_delivery_record(
|
||||
"job_complete",
|
||||
"/api/v1/ops/agent/jobs/8/complete",
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"status": "failed",
|
||||
"client_request_id": "complete-003",
|
||||
},
|
||||
"complete-003",
|
||||
)
|
||||
node_agent._store_dead_letter_delivery(
|
||||
record,
|
||||
reason="permanent failure",
|
||||
detail_code="ops_job_invalid_status",
|
||||
)
|
||||
ok, message, data = node_agent._execute_action(
|
||||
"delivery.queue.discard",
|
||||
{
|
||||
"selector": {"record_id": "complete-003"},
|
||||
"limit": 10,
|
||||
"discarded_by": "web-ui",
|
||||
"reason": "确认不需要再补发",
|
||||
},
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("已丢弃 1 条死信记录", message)
|
||||
self.assertEqual(1, data["discarded_total"])
|
||||
self.assertEqual([], os.listdir(os.path.join(temp_dir, "dead-letter")))
|
||||
discarded_files = os.listdir(os.path.join(temp_dir, "discarded"))
|
||||
self.assertEqual(1, len(discarded_files))
|
||||
with open(os.path.join(temp_dir, "discarded", discarded_files[0]), "r", encoding="utf-8") as handle:
|
||||
archived = json.load(handle)
|
||||
self.assertEqual("web-ui", archived["discarded_by"])
|
||||
self.assertEqual("确认不需要再补发", archived["discard_reason"])
|
||||
|
||||
def test_process_job_passes_envelope_context_to_completion(self) -> None:
|
||||
job = {
|
||||
"id": 12,
|
||||
"job_code": "ops-20260418-000012",
|
||||
"job_type": "release_deploy",
|
||||
"action": "logs.collect",
|
||||
"payload": {"service_name": "domaincheck-worker", "lines": 120},
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志",
|
||||
"focus_ref": {"kind": "ops_job", "job_id": 12, "job_code": "ops-20260418-000012"},
|
||||
"release_context": {"release_version": "v2026.04.18", "rollout_id": 3},
|
||||
"step_ref": {"step_id": 22, "step_key": "worker_logs", "step_title": "收集 Worker 日志"},
|
||||
}
|
||||
with patch.object(node_agent, "_job_start") as mock_job_start, patch.object(
|
||||
node_agent,
|
||||
"_job_event",
|
||||
) as mock_job_event, patch.object(
|
||||
node_agent,
|
||||
"_execute_action",
|
||||
return_value=(True, "logs collected", {"summary": "收集完成", "stdout": "ok", "stderr": ""}),
|
||||
) as mock_execute_action, patch.object(
|
||||
node_agent,
|
||||
"_job_complete",
|
||||
return_value={"state": "sent"},
|
||||
) as mock_job_complete:
|
||||
node_agent._process_job(job)
|
||||
|
||||
self.assertEqual("worker_logs", mock_job_start.call_args.kwargs["job"]["step_key"])
|
||||
self.assertEqual("ops_job", mock_job_event.call_args.kwargs["focus_ref"]["kind"])
|
||||
self.assertEqual("已接单 logs.collect", mock_job_event.call_args.kwargs["summary_text"])
|
||||
self.assertEqual("logs.collect", mock_execute_action.call_args.args[0])
|
||||
self.assertEqual("v2026.04.18", mock_job_complete.call_args.kwargs["release_context"]["release_version"])
|
||||
self.assertEqual("ops_job", mock_job_complete.call_args.kwargs["focus_ref"]["kind"])
|
||||
self.assertEqual("收集完成", mock_job_complete.call_args.kwargs["summary_text"])
|
||||
self.assertGreaterEqual(mock_job_complete.call_args.kwargs["duration_ms"], 0)
|
||||
|
||||
def test_process_job_continues_when_job_start_delivery_fails(self) -> None:
|
||||
job = {
|
||||
"id": 18,
|
||||
"job_code": "ops-20260418-000018",
|
||||
"job_type": "ops_action",
|
||||
"action": "logs.collect",
|
||||
"payload": {"service_name": "domaincheck-worker", "lines": 50},
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志",
|
||||
"focus_ref": {"kind": "ops_job", "job_id": 18, "job_code": "ops-20260418-000018"},
|
||||
"release_context": {},
|
||||
"step_ref": {"step_id": 28, "step_key": "worker_logs", "step_title": "收集 Worker 日志"},
|
||||
}
|
||||
with patch.object(
|
||||
node_agent,
|
||||
"_job_start",
|
||||
side_effect=urllib.error.URLError("temporary offline"),
|
||||
) as mock_job_start, patch.object(
|
||||
node_agent,
|
||||
"_job_event",
|
||||
return_value={"state": "queued"},
|
||||
) as mock_job_event, patch.object(
|
||||
node_agent,
|
||||
"_execute_action",
|
||||
return_value=(True, "logs collected", {"summary": "收集完成", "stdout": "ok", "stderr": ""}),
|
||||
) as mock_execute_action, patch.object(
|
||||
node_agent,
|
||||
"_job_complete",
|
||||
return_value={"state": "sent"},
|
||||
) as mock_job_complete:
|
||||
node_agent._process_job(job)
|
||||
|
||||
mock_job_start.assert_called_once()
|
||||
mock_execute_action.assert_called_once()
|
||||
mock_job_complete.assert_called_once()
|
||||
event_payload = mock_job_event.call_args.kwargs["payload"]
|
||||
self.assertEqual("failed_local", event_payload["start_delivery_state"])
|
||||
self.assertIn("temporary offline", event_payload["start_delivery_error"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
41
domain-api/tests/test_node_agent_release_normalization.py
Normal file
41
domain-api/tests/test_node_agent_release_normalization.py
Normal file
@@ -0,0 +1,41 @@
|
||||
import unittest
|
||||
|
||||
from app import node_agent
|
||||
|
||||
|
||||
class NodeAgentReleaseNormalizationTests(unittest.TestCase):
|
||||
def test_normalize_text_list_supports_strings_and_lists(self) -> None:
|
||||
self.assertEqual(
|
||||
["domaincheck-api", "domaincheck-worker"],
|
||||
node_agent._normalize_text_list("domaincheck-api\ndomaincheck-worker"),
|
||||
)
|
||||
self.assertEqual(
|
||||
["domaincheck-api", "domaincheck-worker"],
|
||||
node_agent._normalize_text_list(["domaincheck-api", " domaincheck-worker "]),
|
||||
)
|
||||
self.assertEqual([], node_agent._normalize_text_list(""))
|
||||
|
||||
def test_health_check_services_fall_back_to_restart_services_when_missing(self) -> None:
|
||||
result = node_agent._normalize_release_health_check_services(
|
||||
{},
|
||||
["domaincheck-worker"],
|
||||
)
|
||||
self.assertEqual(["domaincheck-worker"], result)
|
||||
|
||||
def test_health_check_services_can_be_explicitly_empty(self) -> None:
|
||||
result = node_agent._normalize_release_health_check_services(
|
||||
{"health_check_services": ""},
|
||||
["domaincheck-worker"],
|
||||
)
|
||||
self.assertEqual([], result)
|
||||
|
||||
def test_health_check_services_accept_multiline_string(self) -> None:
|
||||
result = node_agent._normalize_release_health_check_services(
|
||||
{"health_check_services": "domaincheck-api\ndomaincheck-sync-agent"},
|
||||
["domaincheck-worker"],
|
||||
)
|
||||
self.assertEqual(["domaincheck-api", "domaincheck-sync-agent"], result)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
115
domain-api/tests/test_ops_action_executor_core.py
Normal file
115
domain-api/tests/test_ops_action_executor_core.py
Normal file
@@ -0,0 +1,115 @@
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.services.ops_action_executor_core import build_service_name_map, execute_structured_action
|
||||
|
||||
|
||||
class _FakeRunner:
|
||||
def __init__(self, responses: dict[tuple[str, ...], tuple[int, str, str]]) -> None:
|
||||
self.responses = responses
|
||||
self.calls: list[tuple[str, ...]] = []
|
||||
|
||||
def __call__(self, command: list[str], *, timeout: int = 60) -> tuple[int, str, str]:
|
||||
key = tuple(command)
|
||||
self.calls.append(key)
|
||||
return self.responses.get(key, (0, "", ""))
|
||||
|
||||
|
||||
class OpsActionExecutorCoreTests(unittest.TestCase):
|
||||
def test_health_snapshot_includes_host_context_and_ok_flags(self) -> None:
|
||||
runner = _FakeRunner(
|
||||
{
|
||||
("systemctl", "is-active", "domaincheck-api"): (0, "active", ""),
|
||||
("systemctl", "is-active", "domaincheck-worker"): (3, "inactive", ""),
|
||||
("systemctl", "is-active", "domaincheck-sync-agent"): (0, "active", ""),
|
||||
("systemctl", "is-active", "domaincheck-node-agent"): (0, "active", ""),
|
||||
}
|
||||
)
|
||||
|
||||
ok, message, data = execute_structured_action(
|
||||
"health.snapshot",
|
||||
{},
|
||||
service_names=build_service_name_map(
|
||||
api_service_name="domaincheck-api",
|
||||
worker_service_name="domaincheck-worker",
|
||||
sync_agent_service_name="domaincheck-sync-agent",
|
||||
),
|
||||
runner=runner,
|
||||
host_context={"hostname": "mainland-controller-01", "ip": "121.204.244.188"},
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("health snapshot collected", message)
|
||||
self.assertEqual("mainland-controller-01", data["hostname"])
|
||||
self.assertEqual("121.204.244.188", data["ip"])
|
||||
self.assertTrue(data["checks"]["api"]["ok"])
|
||||
self.assertFalse(data["checks"]["worker"]["ok"])
|
||||
self.assertEqual(4, len(runner.calls))
|
||||
|
||||
def test_logs_collect_defaults_to_worker_service_and_clamps_lines(self) -> None:
|
||||
runner = _FakeRunner(
|
||||
{
|
||||
("journalctl", "-u", "domaincheck-worker", "-n", "500", "--no-pager"): (
|
||||
0,
|
||||
"line-1\nline-2",
|
||||
"",
|
||||
)
|
||||
}
|
||||
)
|
||||
|
||||
ok, message, data = execute_structured_action(
|
||||
"logs.collect",
|
||||
{"lines": 9999},
|
||||
service_names=build_service_name_map(
|
||||
api_service_name="domaincheck-api",
|
||||
worker_service_name="domaincheck-worker",
|
||||
sync_agent_service_name="domaincheck-sync-agent",
|
||||
),
|
||||
runner=runner,
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("line-1\nline-2", message)
|
||||
self.assertEqual("domaincheck-worker", data["service_name"])
|
||||
self.assertEqual(500, data["lines"])
|
||||
self.assertEqual(
|
||||
("journalctl", "-u", "domaincheck-worker", "-n", "500", "--no-pager"),
|
||||
runner.calls[0],
|
||||
)
|
||||
|
||||
def test_logs_collect_falls_back_to_worker_log_file_when_journal_is_denied(self) -> None:
|
||||
runner = _FakeRunner(
|
||||
{
|
||||
("journalctl", "-u", "domaincheck-worker", "-n", "120", "--no-pager"): (
|
||||
1,
|
||||
"",
|
||||
"No journal files were opened due to insufficient permissions.",
|
||||
)
|
||||
}
|
||||
)
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
log_path = Path(temp_dir) / "detect_worker.log"
|
||||
log_path.write_text("worker-a\nworker-b\n", encoding="utf-8")
|
||||
with patch("app.services.ops_action_executor_core.settings.domain_root", temp_dir):
|
||||
ok, message, data = execute_structured_action(
|
||||
"logs.collect",
|
||||
{"lines": 120},
|
||||
service_names=build_service_name_map(
|
||||
api_service_name="domaincheck-api",
|
||||
worker_service_name="domaincheck-worker",
|
||||
sync_agent_service_name="domaincheck-sync-agent",
|
||||
),
|
||||
runner=runner,
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("worker-a\nworker-b", message)
|
||||
self.assertEqual("file_fallback", data["collection_source"])
|
||||
self.assertTrue(data["fallback_used"])
|
||||
self.assertIn("detect_worker.log", data["log_paths"][0])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
1181
domain-api/tests/test_ops_agent_service.py
Normal file
1181
domain-api/tests/test_ops_agent_service.py
Normal file
File diff suppressed because it is too large
Load Diff
96
domain-api/tests/test_ops_api_routes.py
Normal file
96
domain-api/tests/test_ops_api_routes.py
Normal file
@@ -0,0 +1,96 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.api.routes.ops import (
|
||||
ops_doctor_decision,
|
||||
ops_go_live_bundle,
|
||||
ops_go_live_review,
|
||||
ops_go_live_signoff,
|
||||
ops_go_live_summary,
|
||||
)
|
||||
|
||||
|
||||
class OpsApiRoutesTestCase(unittest.TestCase):
|
||||
@patch("app.api.routes.ops.get_ops_go_live_signoff")
|
||||
def test_go_live_signoff_route_uses_service_payload(self, mock_get_ops_go_live_signoff) -> None:
|
||||
mock_get_ops_go_live_signoff.return_value = {
|
||||
"signoff_status": "ready",
|
||||
"headline": "ok",
|
||||
}
|
||||
|
||||
response = ops_go_live_signoff(base_url="http://127.0.0.1:8100", report_dir="/tmp/signoff")
|
||||
|
||||
self.assertEqual(0, response.code)
|
||||
self.assertEqual("ready", response.data["signoff_status"])
|
||||
self.assertEqual("ok", response.data["headline"])
|
||||
mock_get_ops_go_live_signoff.assert_called_once_with(
|
||||
base_url="http://127.0.0.1:8100",
|
||||
report_dir="/tmp/signoff",
|
||||
)
|
||||
|
||||
@patch("app.api.routes.ops.get_ops_go_live_summary")
|
||||
def test_go_live_summary_route_passes_base_url(self, mock_get_ops_go_live_summary) -> None:
|
||||
mock_get_ops_go_live_summary.return_value = {
|
||||
"go_live_status": "attention",
|
||||
}
|
||||
|
||||
response = ops_go_live_summary(base_url="http://example.test")
|
||||
|
||||
self.assertEqual(0, response.code)
|
||||
self.assertEqual("attention", response.data["go_live_status"])
|
||||
mock_get_ops_go_live_summary.assert_called_once_with(base_url="http://example.test")
|
||||
|
||||
@patch("app.api.routes.ops.get_ops_go_live_bundle")
|
||||
def test_go_live_bundle_route_passes_base_url_and_report_dir(self, mock_get_ops_go_live_bundle) -> None:
|
||||
mock_get_ops_go_live_bundle.return_value = {
|
||||
"status": "missing",
|
||||
}
|
||||
|
||||
response = ops_go_live_bundle(base_url="http://example.test", report_dir="/tmp/go-live")
|
||||
|
||||
self.assertEqual(0, response.code)
|
||||
self.assertEqual("missing", response.data["status"])
|
||||
mock_get_ops_go_live_bundle.assert_called_once_with(
|
||||
base_url="http://example.test",
|
||||
report_dir="/tmp/go-live",
|
||||
)
|
||||
|
||||
@patch("app.api.routes.ops.get_ops_go_live_review")
|
||||
def test_go_live_review_route_passes_base_url_and_report_dir(self, mock_get_ops_go_live_review) -> None:
|
||||
mock_get_ops_go_live_review.return_value = {
|
||||
"status": "attention",
|
||||
"headline": "review",
|
||||
}
|
||||
|
||||
response = ops_go_live_review(base_url="http://example.test", report_dir="/tmp/review")
|
||||
|
||||
self.assertEqual(0, response.code)
|
||||
self.assertEqual("attention", response.data["status"])
|
||||
self.assertEqual("review", response.data["headline"])
|
||||
mock_get_ops_go_live_review.assert_called_once_with(
|
||||
base_url="http://example.test",
|
||||
report_dir="/tmp/review",
|
||||
)
|
||||
|
||||
@patch("app.api.routes.ops.get_ops_doctor_decision")
|
||||
def test_doctor_decision_route_passes_base_url_and_report_dir(self, mock_get_ops_doctor_decision) -> None:
|
||||
mock_get_ops_doctor_decision.return_value = {
|
||||
"status": "attention",
|
||||
"headline": "doctor",
|
||||
}
|
||||
|
||||
response = ops_doctor_decision(base_url="http://example.test", report_dir="/tmp/doctor")
|
||||
|
||||
self.assertEqual(0, response.code)
|
||||
self.assertEqual("attention", response.data["status"])
|
||||
self.assertEqual("doctor", response.data["headline"])
|
||||
mock_get_ops_doctor_decision.assert_called_once_with(
|
||||
base_url="http://example.test",
|
||||
report_dir="/tmp/doctor",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
44
domain-api/tests/test_ops_command_service.py
Normal file
44
domain-api/tests/test_ops_command_service.py
Normal file
@@ -0,0 +1,44 @@
|
||||
import unittest
|
||||
|
||||
from app.services.ops_command_service import build_bash_command, ops_script_path
|
||||
|
||||
|
||||
class OpsCommandServiceTests(unittest.TestCase):
|
||||
def test_ops_script_path_prefixes_multi_region_directory(self) -> None:
|
||||
self.assertEqual(
|
||||
"domain-api/deploy/multi-region/drive_ops_center.sh",
|
||||
ops_script_path("drive_ops_center.sh"),
|
||||
)
|
||||
|
||||
def test_ops_script_path_normalizes_leading_slash(self) -> None:
|
||||
self.assertEqual(
|
||||
"domain-api/deploy/multi-region/check_ops_center_stack.sh",
|
||||
ops_script_path("/check_ops_center_stack.sh"),
|
||||
)
|
||||
|
||||
def test_build_bash_command_joins_non_empty_args(self) -> None:
|
||||
self.assertEqual(
|
||||
"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 handover_first_gap",
|
||||
build_bash_command(
|
||||
"drive_ops_center.sh",
|
||||
"driver-resolve",
|
||||
"http://127.0.0.1:8100",
|
||||
"handover_first_gap",
|
||||
),
|
||||
)
|
||||
|
||||
def test_build_bash_command_skips_blank_args(self) -> None:
|
||||
self.assertEqual(
|
||||
"bash domain-api/deploy/multi-region/drive_release_hub.sh http://127.0.0.1:8100 launchpad",
|
||||
build_bash_command(
|
||||
"drive_release_hub.sh",
|
||||
"http://127.0.0.1:8100",
|
||||
"",
|
||||
"launchpad",
|
||||
" ",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
391
domain-api/tests/test_ops_job_service.py
Normal file
391
domain-api/tests/test_ops_job_service.py
Normal file
@@ -0,0 +1,391 @@
|
||||
import json
|
||||
import unittest
|
||||
from datetime import datetime
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.ops_job_service import (
|
||||
_execute_control_plane_job,
|
||||
_serialize_job_row,
|
||||
create_ops_job,
|
||||
dispatch_ops_job,
|
||||
upsert_managed_node,
|
||||
)
|
||||
|
||||
|
||||
class OpsJobServiceTests(unittest.TestCase):
|
||||
def test_serialize_job_row_adds_execution_mode_label(self) -> None:
|
||||
row = (
|
||||
9,
|
||||
"ops-2026041801-abcd12",
|
||||
"logs.collect",
|
||||
"node",
|
||||
"mainland-worker-01",
|
||||
"queued",
|
||||
"remote-agent",
|
||||
"api",
|
||||
json.dumps({"service_name": "domaincheck-worker", "lines": 120}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
"",
|
||||
datetime(2026, 4, 18, 10, 0, 0),
|
||||
None,
|
||||
None,
|
||||
datetime(2026, 4, 18, 10, 0, 0),
|
||||
"low",
|
||||
False,
|
||||
"approved",
|
||||
"",
|
||||
None,
|
||||
"",
|
||||
"",
|
||||
None,
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
None,
|
||||
)
|
||||
|
||||
job = _serialize_job_row(row)
|
||||
|
||||
self.assertEqual("remote-agent", job["execution_mode"])
|
||||
self.assertEqual("远端 Agent", job["execution_mode_label"])
|
||||
self.assertEqual("排队中", job["status_label"])
|
||||
self.assertEqual("已审批", job["approval_status_label"])
|
||||
self.assertEqual(["mainland-worker-01"], job["target_node_codes"])
|
||||
self.assertEqual(0, job["steps_total"])
|
||||
self.assertFalse(job["steps_loaded"])
|
||||
self.assertFalse(job["is_compact"])
|
||||
self.assertEqual("ops_job", job["focus_ref"]["kind"])
|
||||
self.assertEqual(9, job["focus_ref"]["job_id"])
|
||||
self.assertIn("目标节点 mainland-worker-01", job["summary_text"])
|
||||
|
||||
def test_serialize_job_row_counts_loaded_steps(self) -> None:
|
||||
row = (
|
||||
10,
|
||||
"ops-2026041801-efgh34",
|
||||
"diagnostics.collect",
|
||||
"node",
|
||||
"mainland-worker-02",
|
||||
"running",
|
||||
"ssh",
|
||||
"web-ui",
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
"",
|
||||
datetime(2026, 4, 18, 10, 5, 0),
|
||||
datetime(2026, 4, 18, 10, 5, 2),
|
||||
None,
|
||||
datetime(2026, 4, 18, 10, 5, 3),
|
||||
"medium",
|
||||
True,
|
||||
"approved",
|
||||
"admin",
|
||||
datetime(2026, 4, 18, 10, 5, 1),
|
||||
"",
|
||||
"",
|
||||
datetime(2026, 4, 18, 10, 5, 2),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
None,
|
||||
)
|
||||
|
||||
job = _serialize_job_row(
|
||||
row,
|
||||
steps=[
|
||||
{"id": 1, "status": "success"},
|
||||
{"id": 2, "status": "running"},
|
||||
{"id": 3, "status": "failed"},
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual("执行中", job["status_label"])
|
||||
self.assertEqual("已审批", job["approval_status_label"])
|
||||
self.assertEqual(3, job["steps_total"])
|
||||
self.assertEqual(1, job["steps_running"])
|
||||
self.assertEqual(1, job["steps_success"])
|
||||
self.assertEqual(1, job["steps_failed"])
|
||||
self.assertEqual(2, job["steps_terminal"])
|
||||
self.assertTrue(job["steps_loaded"])
|
||||
self.assertIn("mainland-worker-02", job["summary_text"])
|
||||
|
||||
@patch("app.services.ops_job_service.list_managed_nodes")
|
||||
@patch("app.services.ops_agent_service.build_node_agent_bootstrap_plan")
|
||||
def test_execute_control_plane_bootstrap_uses_managed_node_defaults(
|
||||
self,
|
||||
mock_build_node_agent_bootstrap_plan,
|
||||
mock_list_managed_nodes,
|
||||
) -> None:
|
||||
mock_list_managed_nodes.return_value = [
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
}
|
||||
]
|
||||
mock_build_node_agent_bootstrap_plan.return_value = (
|
||||
True,
|
||||
"节点 Agent 接入方案已生成",
|
||||
{
|
||||
"token_preview": "agent-...1234",
|
||||
"bootstrap_plan": {
|
||||
"bootstrap_script_name": "bootstrap-node-agent-mainland-worker-02.sh",
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
ok, message, data = _execute_control_plane_job(
|
||||
"node.bootstrap",
|
||||
target_node_code="mainland-worker-02",
|
||||
payload={
|
||||
"root_dir": "/opt/domaincheck",
|
||||
"expires_in_hours": 12,
|
||||
},
|
||||
requested_by="unit-test",
|
||||
metadata={"source": "test"},
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("节点 Agent 接入方案已生成", message)
|
||||
self.assertEqual("control-plane", data["execution_mode"])
|
||||
self.assertEqual("mainland", data["node_region"])
|
||||
self.assertEqual("worker", data["node_role"])
|
||||
mock_build_node_agent_bootstrap_plan.assert_called_once_with(
|
||||
node_code="mainland-worker-02",
|
||||
node_region="mainland",
|
||||
node_role="worker",
|
||||
issued_by="unit-test",
|
||||
expires_in_hours=12,
|
||||
control_plane_base_url="",
|
||||
root_dir="/opt/domaincheck",
|
||||
metadata={
|
||||
"source": "test",
|
||||
"issued_from": "ops-job/control-plane",
|
||||
"target_node_code": "mainland-worker-02",
|
||||
"node_region": "mainland",
|
||||
"node_role": "worker",
|
||||
},
|
||||
)
|
||||
|
||||
@patch("app.services.ops_job_service.get_db")
|
||||
def test_upsert_managed_node_preserves_remote_fields_when_sync_payload_omits_them(self, mock_get_db) -> None:
|
||||
existing_metadata = {"operator_note": "keep-me"}
|
||||
merged_metadata = {"operator_note": "keep-me", "cluster_status": "online"}
|
||||
existing_row = (
|
||||
"mainland-worker-01",
|
||||
"mainland",
|
||||
"worker",
|
||||
"Mainland Worker 01",
|
||||
"121.204.244.248",
|
||||
22,
|
||||
"root",
|
||||
"key",
|
||||
"stable",
|
||||
False,
|
||||
json.dumps(existing_metadata, ensure_ascii=False),
|
||||
datetime(2026, 4, 18, 1, 0, 0),
|
||||
None,
|
||||
datetime(2026, 4, 18, 1, 5, 0),
|
||||
)
|
||||
returned_row = (
|
||||
"mainland-worker-01",
|
||||
"mainland",
|
||||
"worker",
|
||||
"S244-248",
|
||||
"121.204.244.248",
|
||||
22,
|
||||
"root",
|
||||
"key",
|
||||
"stable",
|
||||
False,
|
||||
json.dumps(merged_metadata, ensure_ascii=False),
|
||||
datetime(2026, 4, 18, 1, 0, 0),
|
||||
None,
|
||||
datetime(2026, 4, 18, 1, 6, 0),
|
||||
)
|
||||
|
||||
conn = MagicMock()
|
||||
cursor = MagicMock()
|
||||
db_ctx = MagicMock()
|
||||
cursor_ctx = MagicMock()
|
||||
db_ctx.__enter__.return_value = conn
|
||||
db_ctx.__exit__.return_value = False
|
||||
cursor_ctx.__enter__.return_value = cursor
|
||||
cursor_ctx.__exit__.return_value = False
|
||||
conn.cursor.return_value = cursor_ctx
|
||||
mock_get_db.return_value = db_ctx
|
||||
cursor.fetchone.side_effect = [existing_row, returned_row]
|
||||
|
||||
ok, message, data = upsert_managed_node(
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"title": "S244-248",
|
||||
"metadata": {
|
||||
"cluster_status": "online",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("托管节点已保存", message)
|
||||
node = data["node"]
|
||||
self.assertEqual("121.204.244.248", node["ssh_host"])
|
||||
self.assertEqual("root", node["ssh_user"])
|
||||
self.assertFalse(node["is_enabled"])
|
||||
self.assertEqual("keep-me", node["metadata"]["operator_note"])
|
||||
self.assertEqual("online", node["metadata"]["cluster_status"])
|
||||
|
||||
insert_call = cursor.execute.call_args_list[-1]
|
||||
insert_params = insert_call.args[1]
|
||||
self.assertEqual("121.204.244.248", insert_params[4])
|
||||
self.assertEqual(22, insert_params[5])
|
||||
self.assertEqual("root", insert_params[6])
|
||||
self.assertFalse(insert_params[9])
|
||||
self.assertEqual(merged_metadata, json.loads(insert_params[10]))
|
||||
|
||||
@patch("app.services.ops_agent_service.append_ops_job_event")
|
||||
@patch("app.services.ops_job_service._execute_ssh_job_record")
|
||||
@patch("app.services.ops_job_service.get_db")
|
||||
def test_dispatch_ops_job_ssh_executes_via_ssh_record(
|
||||
self,
|
||||
mock_get_db,
|
||||
mock_execute_ssh_job_record,
|
||||
mock_append_ops_job_event,
|
||||
) -> None:
|
||||
conn = MagicMock()
|
||||
cursor = MagicMock()
|
||||
db_ctx = MagicMock()
|
||||
cursor_ctx = MagicMock()
|
||||
db_ctx.__enter__.return_value = conn
|
||||
db_ctx.__exit__.return_value = False
|
||||
cursor_ctx.__enter__.return_value = cursor
|
||||
cursor_ctx.__exit__.return_value = False
|
||||
conn.cursor.return_value = cursor_ctx
|
||||
mock_get_db.return_value = db_ctx
|
||||
cursor.fetchone.return_value = (
|
||||
9,
|
||||
"ops-2026041801-abcd12",
|
||||
"logs.collect",
|
||||
"node",
|
||||
"mainland-worker-01",
|
||||
"queued",
|
||||
"ssh",
|
||||
"api",
|
||||
json.dumps({"service_name": "domaincheck-worker", "lines": 120}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
"",
|
||||
datetime(2026, 4, 18, 10, 0, 0),
|
||||
None,
|
||||
None,
|
||||
datetime(2026, 4, 18, 10, 0, 0),
|
||||
"low",
|
||||
False,
|
||||
"approved",
|
||||
"",
|
||||
None,
|
||||
"",
|
||||
"",
|
||||
None,
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
json.dumps({}, ensure_ascii=False),
|
||||
None,
|
||||
)
|
||||
mock_execute_ssh_job_record.return_value = (
|
||||
True,
|
||||
"worker logs collected",
|
||||
{"job": {"id": 9, "status": "success"}},
|
||||
)
|
||||
|
||||
ok, message, data = dispatch_ops_job(9)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("已按 SSH 执行派发:worker logs collected", message)
|
||||
self.assertEqual("success", data["job"]["status"])
|
||||
mock_execute_ssh_job_record.assert_called_once_with(9)
|
||||
mock_append_ops_job_event.assert_called_once()
|
||||
event_kwargs = mock_append_ops_job_event.call_args.kwargs
|
||||
self.assertEqual("job_dispatch_requested", event_kwargs["event_type"])
|
||||
self.assertEqual("mainland-worker-01", event_kwargs["node_code"])
|
||||
self.assertEqual("ssh", event_kwargs["payload"]["execution_mode"])
|
||||
|
||||
@patch("app.services.ops_agent_service.append_ops_job_event")
|
||||
@patch("app.services.ops_job_service.get_ops_job")
|
||||
@patch("app.services.ops_job_service._execute_local_job")
|
||||
@patch("app.services.ops_policy_service.preview_ops_job_policy")
|
||||
@patch("app.services.ops_job_service.get_db")
|
||||
def test_create_ops_job_auto_uses_local_runtime_for_local_structured_action(
|
||||
self,
|
||||
mock_get_db,
|
||||
mock_preview_ops_job_policy,
|
||||
mock_execute_local_job,
|
||||
mock_get_ops_job,
|
||||
mock_append_ops_job_event,
|
||||
) -> None:
|
||||
conn = MagicMock()
|
||||
cursor = MagicMock()
|
||||
db_ctx = MagicMock()
|
||||
cursor_ctx = MagicMock()
|
||||
db_ctx.__enter__.return_value = conn
|
||||
db_ctx.__exit__.return_value = False
|
||||
cursor_ctx.__enter__.return_value = cursor
|
||||
cursor_ctx.__exit__.return_value = False
|
||||
conn.cursor.return_value = cursor_ctx
|
||||
mock_get_db.return_value = db_ctx
|
||||
cursor.fetchone.return_value = (42,)
|
||||
|
||||
mock_preview_ops_job_policy.return_value = {
|
||||
"risk_level": "low",
|
||||
"approval_required": False,
|
||||
"blocked": False,
|
||||
"blocking_reasons": [],
|
||||
}
|
||||
mock_execute_local_job.return_value = (
|
||||
True,
|
||||
"worker logs collected",
|
||||
{"executor": "local-runtime"},
|
||||
)
|
||||
mock_get_ops_job.return_value = {
|
||||
"id": 42,
|
||||
"status": "success",
|
||||
"execution_mode": "local-runtime",
|
||||
}
|
||||
|
||||
ok, message, data = create_ops_job(
|
||||
{
|
||||
"action": "logs.collect",
|
||||
"target_node_code": settings.node_code,
|
||||
"payload": {
|
||||
"service_name": "domaincheck-worker",
|
||||
"lines": 120,
|
||||
},
|
||||
"requested_by": "unit-test",
|
||||
}
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("worker logs collected", message)
|
||||
self.assertTrue(data["executed_immediately"])
|
||||
self.assertEqual("local-runtime", data["job"]["execution_mode"])
|
||||
mock_execute_local_job.assert_called_once_with(
|
||||
"logs.collect",
|
||||
{
|
||||
"service_name": "domaincheck-worker",
|
||||
"lines": 120,
|
||||
},
|
||||
)
|
||||
|
||||
insert_call = next(
|
||||
call
|
||||
for call in cursor.execute.call_args_list
|
||||
if "INSERT INTO ops_jobs" in str(call.args[0])
|
||||
)
|
||||
insert_params = insert_call.args[1]
|
||||
self.assertEqual("local-runtime", insert_params[5])
|
||||
mock_append_ops_job_event.assert_called_once()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
220
domain-api/tests/test_ops_playbook_service.py
Normal file
220
domain-api/tests/test_ops_playbook_service.py
Normal file
@@ -0,0 +1,220 @@
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.services.ops_playbook_service import (
|
||||
get_recent_ops_playbook_runs,
|
||||
list_ops_playbook_run_events,
|
||||
preview_ops_playbook,
|
||||
)
|
||||
|
||||
|
||||
class OpsPlaybookServiceTests(unittest.TestCase):
|
||||
def test_onboarding_bootstrap_preview_uses_control_plane(self) -> None:
|
||||
ok, message, data = preview_ops_playbook(
|
||||
{
|
||||
"playbook_key": "onboarding.bootstrap",
|
||||
"node_codes": ["mainland-worker-02"],
|
||||
}
|
||||
)
|
||||
|
||||
self.assertTrue(ok, message)
|
||||
self.assertEqual("playbook 预览生成成功", message)
|
||||
self.assertEqual("onboarding.bootstrap", data["playbook"]["key"])
|
||||
self.assertEqual("control-plane", data["execution_mode"])
|
||||
self.assertEqual("控制面", data["execution_mode_label"])
|
||||
self.assertEqual(["control-plane"], data["playbook"]["execution_modes"])
|
||||
self.assertEqual("控制面", data["playbook"]["default_execution_mode_label"])
|
||||
self.assertEqual("控制面", data["playbook"]["execution_mode_options"][0]["label"])
|
||||
self.assertTrue(data["auto_approve"])
|
||||
self.assertEqual(1, data["step_count"])
|
||||
step = data["steps"][0]
|
||||
self.assertEqual("bootstrap", step["step_key"])
|
||||
self.assertEqual("node.bootstrap", step["template_key"])
|
||||
self.assertEqual("node.bootstrap", step["action"])
|
||||
self.assertEqual("control-plane", step["execution_mode"])
|
||||
self.assertEqual(1, step["expected_jobs"])
|
||||
|
||||
def test_scene_logs_key_preview_allows_ssh_execution_mode(self) -> None:
|
||||
ok, message, data = preview_ops_playbook(
|
||||
{
|
||||
"playbook_key": "scene.logs.key",
|
||||
"node_codes": ["mainland-worker-01"],
|
||||
"execution_mode": "ssh",
|
||||
}
|
||||
)
|
||||
|
||||
self.assertTrue(ok, message)
|
||||
self.assertEqual("playbook 预览生成成功", message)
|
||||
self.assertEqual("ssh", data["execution_mode"])
|
||||
self.assertEqual("SSH", data["execution_mode_label"])
|
||||
self.assertEqual(["remote-agent", "ssh"], data["playbook"]["execution_modes"])
|
||||
self.assertEqual("ssh", data["steps"][0]["execution_mode"])
|
||||
self.assertEqual("SSH", data["steps"][0]["execution_mode_label"])
|
||||
|
||||
def test_onboarding_bootstrap_preview_rejects_unsupported_execution_mode(self) -> None:
|
||||
ok, message, data = preview_ops_playbook(
|
||||
{
|
||||
"playbook_key": "onboarding.bootstrap",
|
||||
"node_codes": ["mainland-worker-02"],
|
||||
"execution_mode": "ssh",
|
||||
}
|
||||
)
|
||||
|
||||
self.assertFalse(ok)
|
||||
self.assertEqual("该 playbook 仅支持 control-plane", message)
|
||||
self.assertEqual(["control-plane"], data["playbook"]["execution_modes"])
|
||||
|
||||
@patch("app.services.ops_playbook_service.list_managed_nodes_with_agent_state")
|
||||
@patch("app.services.ops_playbook_service.list_ops_jobs")
|
||||
def test_recent_onboarding_bootstrap_run_focus_summary_reflects_agent_state(
|
||||
self,
|
||||
mock_list_ops_jobs,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
) -> None:
|
||||
mock_list_ops_jobs.return_value = [
|
||||
{
|
||||
"id": 31,
|
||||
"job_code": "ops-bootstrap-31",
|
||||
"action": "node.bootstrap",
|
||||
"status": "success",
|
||||
"target_node_code": "mainland-worker-02",
|
||||
"created_at": "2026-04-17 10:00:00",
|
||||
"updated_at": "2026-04-17 10:01:00",
|
||||
"metadata": {
|
||||
"playbook": {
|
||||
"run_code": "pbr-bootstrap-1",
|
||||
"key": "onboarding.bootstrap",
|
||||
"title": "生成节点接入工单",
|
||||
"group_key": "onboarding",
|
||||
"group_title": "接管验收",
|
||||
"requested_by": "tester",
|
||||
"execution_mode": "control-plane",
|
||||
"step_key": "bootstrap",
|
||||
"step_title": "生成 Node Agent 接入工单",
|
||||
"step_order": 1,
|
||||
"step_count": 1,
|
||||
"expected_jobs_total": 1,
|
||||
"target_nodes_total": 1,
|
||||
"template_key": "node.bootstrap",
|
||||
}
|
||||
},
|
||||
"result": {
|
||||
"bootstrap_plan": {
|
||||
"bootstrap_script_name": "bootstrap-node-agent-mainland-worker-02.sh",
|
||||
}
|
||||
},
|
||||
}
|
||||
]
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"agent_state": "pending_bootstrap",
|
||||
"is_managed": True,
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
payload = get_recent_ops_playbook_runs(limit=10, scan_limit=20)
|
||||
|
||||
self.assertEqual(1, len(payload["runs"]))
|
||||
run = payload["runs"][0]
|
||||
self.assertEqual("success", run["status"])
|
||||
self.assertEqual("成功", run["status_label"])
|
||||
self.assertEqual("控制面", run["execution_mode_label"])
|
||||
self.assertEqual("warning", run["focus_level"])
|
||||
self.assertIn("待执行 bootstrap 脚本", run["focus_summary"])
|
||||
self.assertEqual(run["focus_summary"], run["summary_text"])
|
||||
self.assertEqual(1, run["steps_total"])
|
||||
self.assertEqual("pbr-bootstrap-1", run["focus_ref"]["run_code"])
|
||||
self.assertEqual("生成 Node Agent 接入工单", run["steps"][0]["title"])
|
||||
self.assertEqual("成功", run["steps"][0]["status_label"])
|
||||
self.assertTrue(run["steps"][0]["summary_text"])
|
||||
|
||||
@patch("app.services.ops_playbook_service.list_managed_nodes_with_agent_state")
|
||||
@patch("app.services.ops_playbook_service.list_ops_job_events_for_jobs")
|
||||
@patch("app.services.ops_playbook_service.list_ops_jobs")
|
||||
def test_playbook_run_events_add_contract_aliases(
|
||||
self,
|
||||
mock_list_ops_jobs,
|
||||
mock_list_ops_job_events_for_jobs,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
) -> None:
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {"nodes": []}
|
||||
mock_list_ops_jobs.return_value = [
|
||||
{
|
||||
"id": 41,
|
||||
"job_code": "ops-inspection-41",
|
||||
"action": "logs.collect",
|
||||
"status": "running",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"created_at": "2026-04-18 06:30:00",
|
||||
"updated_at": "2026-04-18 06:31:00",
|
||||
"metadata": {
|
||||
"playbook": {
|
||||
"run_code": "pbr-inspection-1",
|
||||
"key": "inspection.standard",
|
||||
"title": "标准巡检",
|
||||
"group_key": "diagnostics",
|
||||
"group_title": "标准巡检",
|
||||
"requested_by": "tester",
|
||||
"execution_mode": "remote-agent",
|
||||
"step_key": "worker_logs",
|
||||
"step_title": "收集 Worker 日志",
|
||||
"step_order": 2,
|
||||
"step_count": 3,
|
||||
"expected_jobs_total": 3,
|
||||
"target_nodes_total": 1,
|
||||
"template_key": "logs.collect",
|
||||
}
|
||||
},
|
||||
}
|
||||
]
|
||||
mock_list_ops_job_events_for_jobs.return_value = [
|
||||
{
|
||||
"id": 501,
|
||||
"event_key": "job-event:501",
|
||||
"job_id": 41,
|
||||
"node_code": "mainland-worker-01",
|
||||
"event_type": "job_running",
|
||||
"level": "info",
|
||||
"level_label": "信息",
|
||||
"message": "开始收集 Worker 日志",
|
||||
"summary": "开始收集 Worker 日志",
|
||||
"summary_text": "远端 Agent 已开始收集 Worker 日志",
|
||||
"focus_ref": {
|
||||
"kind": "ops_job",
|
||||
"job_id": 41,
|
||||
"job_code": "ops-inspection-41",
|
||||
"action": "logs.collect",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"event_key": "job-event:501",
|
||||
"step_key": "worker_logs",
|
||||
},
|
||||
"payload": {},
|
||||
"created_at": "2026-04-18 06:31:00",
|
||||
"occurred_at": "2026-04-18 06:31:00",
|
||||
"ui_intent": {"kind": "job_detail", "job_id": 41, "focus_event_id": 501},
|
||||
}
|
||||
]
|
||||
|
||||
payload = list_ops_playbook_run_events("pbr-inspection-1", limit=10)
|
||||
|
||||
self.assertEqual("inspection.standard", payload["playbook_run"]["playbook_key"])
|
||||
self.assertEqual(1, len(payload["events"]))
|
||||
event = payload["events"][0]
|
||||
self.assertEqual("收口中", event["job_status_label"])
|
||||
self.assertEqual("远端 Agent 已开始收集 Worker 日志", event["summary_text"])
|
||||
self.assertEqual("worker_logs", event["focus_ref"]["focus_step_key"])
|
||||
self.assertEqual("job-event:501", event["focus_ref"]["event_key"])
|
||||
self.assertEqual("ops_job", event["source_focus_ref"]["kind"])
|
||||
self.assertEqual("job-event:501", event["source_focus_ref"]["event_key"])
|
||||
summary = payload["summary"]
|
||||
self.assertEqual(1, summary["returned_total"])
|
||||
self.assertEqual({"job_running": 1}, summary["event_type_counts"])
|
||||
self.assertEqual({"running": 1}, summary["job_status_counts"])
|
||||
self.assertEqual(10, summary["filters"]["limit"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
253
domain-api/tests/test_ops_policy_service.py
Normal file
253
domain-api/tests/test_ops_policy_service.py
Normal file
@@ -0,0 +1,253 @@
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.services.ops_policy_service import preview_ops_job_policy
|
||||
|
||||
|
||||
class OpsPolicyServiceTests(unittest.TestCase):
|
||||
@patch("app.services.ops_policy_service.get_cluster_snapshot")
|
||||
def test_node_bootstrap_preview_requires_control_plane(self, mock_cluster_snapshot) -> None:
|
||||
mock_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": False,
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"online_worker_nodes": 2,
|
||||
"dedicated_online_worker_nodes": 1,
|
||||
"online_control_nodes": 1,
|
||||
"busy_nodes": [],
|
||||
},
|
||||
}
|
||||
|
||||
preview = preview_ops_job_policy(
|
||||
{
|
||||
"action": "node.bootstrap",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"execution_mode": "remote-agent",
|
||||
"payload": {},
|
||||
}
|
||||
)
|
||||
|
||||
self.assertEqual("critical", preview["risk_level"])
|
||||
self.assertTrue(preview["blocked"])
|
||||
self.assertTrue(preview["approval_required"])
|
||||
self.assertIn("node.bootstrap 仅支持 control-plane", " ".join(preview["blocking_reasons"]))
|
||||
|
||||
@patch("app.services.ops_policy_service.get_cluster_snapshot")
|
||||
def test_deploy_release_batch_preview_summarizes_node_risks(self, mock_cluster_snapshot) -> None:
|
||||
mock_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"region": "mainland",
|
||||
"role": "control",
|
||||
"status": "busy",
|
||||
"current_load": 3,
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": False,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": False,
|
||||
},
|
||||
],
|
||||
"summary": {
|
||||
"online_worker_nodes": 2,
|
||||
"dedicated_online_worker_nodes": 1,
|
||||
"online_control_nodes": 1,
|
||||
"busy_nodes": ["mainland-controller-01"],
|
||||
},
|
||||
}
|
||||
|
||||
preview = preview_ops_job_policy(
|
||||
{
|
||||
"action": "deploy.release",
|
||||
"target_type": "batch",
|
||||
"target_node_codes": ["mainland-controller-01", "mainland-worker-01"],
|
||||
"execution_mode": "remote-agent",
|
||||
"payload": {
|
||||
"release_version": "2026.04.18-rc1",
|
||||
"artifact_url": "https://example.com/release.tar.gz",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
self.assertEqual("batch", preview["target_type"])
|
||||
self.assertTrue(preview["blocked"])
|
||||
self.assertTrue(preview["approval_required"])
|
||||
self.assertEqual(2, preview["target_summary"]["nodes_total"])
|
||||
self.assertEqual(1, preview["target_summary"]["blocked_nodes"])
|
||||
self.assertEqual(1, preview["target_summary"]["approval_nodes"])
|
||||
self.assertEqual(1, preview["target_summary"]["warning_nodes"])
|
||||
self.assertEqual(2, len(preview["target_nodes"]))
|
||||
controller_row = next(item for item in preview["target_nodes"] if item["node_code"] == "mainland-controller-01")
|
||||
self.assertTrue(controller_row["blocked"])
|
||||
self.assertTrue(controller_row["approval_required"])
|
||||
|
||||
@patch("app.services.ops_policy_service.get_cluster_snapshot")
|
||||
def test_deploy_release_preview_blocks_missing_artifact_and_wrong_execution_mode(self, mock_cluster_snapshot) -> None:
|
||||
mock_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": False,
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"online_worker_nodes": 2,
|
||||
"dedicated_online_worker_nodes": 1,
|
||||
"online_control_nodes": 1,
|
||||
"busy_nodes": [],
|
||||
},
|
||||
}
|
||||
|
||||
preview = preview_ops_job_policy(
|
||||
{
|
||||
"action": "deploy.release",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"execution_mode": "local-runtime",
|
||||
"payload": {
|
||||
"release_version": "2026.04.18-rc1",
|
||||
"artifact_url": "",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
self.assertTrue(preview["blocked"])
|
||||
self.assertIn("deploy.release 缺少 artifact_url", " ".join(preview["blocking_reasons"]))
|
||||
self.assertIn("deploy.release 仅支持 remote-agent 或 ssh", " ".join(preview["blocking_reasons"]))
|
||||
|
||||
@patch("app.services.ops_policy_service.get_cluster_snapshot")
|
||||
def test_deploy_release_preview_allows_ssh_execution_mode(self, mock_cluster_snapshot) -> None:
|
||||
mock_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": False,
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"online_worker_nodes": 2,
|
||||
"dedicated_online_worker_nodes": 1,
|
||||
"online_control_nodes": 1,
|
||||
"busy_nodes": [],
|
||||
},
|
||||
}
|
||||
|
||||
preview = preview_ops_job_policy(
|
||||
{
|
||||
"action": "deploy.release",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"execution_mode": "ssh",
|
||||
"payload": {
|
||||
"release_version": "2026.04.18-rc1",
|
||||
"artifact_url": "https://example.com/release.tar.gz",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
self.assertFalse(any("仅支持 remote-agent 或 ssh" in item for item in preview["blocking_reasons"]))
|
||||
self.assertTrue(preview["approval_required"])
|
||||
|
||||
@patch("app.services.ops_policy_service.get_cluster_snapshot")
|
||||
def test_service_status_preview_blocks_control_plane_execution_mode(self, mock_cluster_snapshot) -> None:
|
||||
mock_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": False,
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"online_worker_nodes": 2,
|
||||
"dedicated_online_worker_nodes": 1,
|
||||
"online_control_nodes": 1,
|
||||
"busy_nodes": [],
|
||||
},
|
||||
}
|
||||
|
||||
preview = preview_ops_job_policy(
|
||||
{
|
||||
"action": "service.status",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"execution_mode": "control-plane",
|
||||
"payload": {"service_name": "domaincheck-worker"},
|
||||
}
|
||||
)
|
||||
|
||||
self.assertTrue(preview["blocked"])
|
||||
self.assertIn("service.status 当前不支持 control-plane 执行方式", " ".join(preview["blocking_reasons"]))
|
||||
|
||||
@patch("app.services.ops_policy_service.get_cluster_snapshot")
|
||||
def test_logs_collect_preview_blocks_local_runtime_on_remote_node(self, mock_cluster_snapshot) -> None:
|
||||
mock_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
"detect_participating": False,
|
||||
}
|
||||
],
|
||||
"summary": {
|
||||
"online_worker_nodes": 2,
|
||||
"dedicated_online_worker_nodes": 1,
|
||||
"online_control_nodes": 1,
|
||||
"busy_nodes": [],
|
||||
},
|
||||
}
|
||||
|
||||
preview = preview_ops_job_policy(
|
||||
{
|
||||
"action": "logs.collect",
|
||||
"target_type": "node",
|
||||
"target_node_code": "mainland-worker-01",
|
||||
"execution_mode": "local-runtime",
|
||||
"payload": {
|
||||
"service_name": "domaincheck-worker",
|
||||
"lines": 120,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
self.assertTrue(preview["blocked"])
|
||||
self.assertIn("local-runtime 仅支持当前控制面本机节点", " ".join(preview["blocking_reasons"]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
89
domain-api/tests/test_ops_release_executor_core.py
Normal file
89
domain-api/tests/test_ops_release_executor_core.py
Normal file
@@ -0,0 +1,89 @@
|
||||
import io
|
||||
import tarfile
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from app.services.ops_release_executor_core import build_remote_release_action_script, execute_release_action
|
||||
|
||||
|
||||
class _BytesResponse:
|
||||
def __init__(self, data: bytes) -> None:
|
||||
self._data = data
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def read(self, *args, **kwargs) -> bytes:
|
||||
return self._data
|
||||
|
||||
|
||||
def _build_release_archive() -> bytes:
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w:gz") as archive:
|
||||
content = b"hello-release"
|
||||
info = tarfile.TarInfo(name="bundle/README.txt")
|
||||
info.size = len(content)
|
||||
archive.addfile(info, io.BytesIO(content))
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
class OpsReleaseExecutorCoreTests(unittest.TestCase):
|
||||
def test_execute_release_action_deploys_archive_and_switches_current(self) -> None:
|
||||
archive_bytes = _build_release_archive()
|
||||
commands: list[tuple[str, ...]] = []
|
||||
|
||||
def run_command(command: list[str], *, timeout: int = 60):
|
||||
commands.append(tuple(command))
|
||||
if command[:2] == ["systemctl", "restart"]:
|
||||
return 0, "", ""
|
||||
if command[:2] == ["systemctl", "is-active"]:
|
||||
return 0, "active", ""
|
||||
return 0, "", ""
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
ok, message, data = execute_release_action(
|
||||
{
|
||||
"release_version": "2026.04.18-rc1",
|
||||
"artifact_url": "https://example.com/domaincheck.tar.gz",
|
||||
"install_root": tmpdir,
|
||||
"restart_services": ["domaincheck-worker"],
|
||||
"health_check_services": ["domaincheck-worker"],
|
||||
"health_check_urls": [],
|
||||
"rollback_on_failure": "true",
|
||||
"switch_current": "true",
|
||||
},
|
||||
run_command=run_command,
|
||||
default_api_service_name="domaincheck-api",
|
||||
urlopen_func=lambda request, timeout=120: _BytesResponse(archive_bytes),
|
||||
user_agent="unit-test/0.1",
|
||||
)
|
||||
|
||||
self.assertTrue(ok, message)
|
||||
self.assertEqual("release deployed", message)
|
||||
self.assertEqual("2026.04.18-rc1", data["release_version"])
|
||||
current_link = Path(tmpdir) / "current"
|
||||
self.assertTrue(current_link.is_symlink())
|
||||
release_dir = Path(data["release_dir"])
|
||||
self.assertTrue(release_dir.exists())
|
||||
self.assertTrue((release_dir / ".release-meta.json").exists())
|
||||
self.assertIn(("systemctl", "restart", "domaincheck-worker"), commands)
|
||||
self.assertIn(("systemctl", "is-active", "domaincheck-worker"), commands)
|
||||
|
||||
def test_build_remote_release_action_script_is_valid_python(self) -> None:
|
||||
script = build_remote_release_action_script(
|
||||
{
|
||||
"release_version": "2026.04.18-rc1",
|
||||
"artifact_url": "https://example.com/domaincheck.tar.gz",
|
||||
}
|
||||
)
|
||||
|
||||
compile(script, "<remote-release-script>", "exec")
|
||||
self.assertIn("execute_release_action(", script)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
93
domain-api/tests/test_ops_release_service_contracts.py
Normal file
93
domain-api/tests/test_ops_release_service_contracts.py
Normal file
@@ -0,0 +1,93 @@
|
||||
import json
|
||||
import unittest
|
||||
from datetime import datetime
|
||||
|
||||
from app.services.ops_release_service import _serialize_release_row, _serialize_rollout_row
|
||||
|
||||
|
||||
class OpsReleaseServiceContractTests(unittest.TestCase):
|
||||
def test_serialize_release_row_adds_contract_aliases(self) -> None:
|
||||
row = (
|
||||
9,
|
||||
"2026.04.18+bd6bcb2",
|
||||
"stable",
|
||||
"bd6bcb2",
|
||||
"ready",
|
||||
"https://example.com/domaincheck.tar.gz",
|
||||
"sha256:test",
|
||||
"release notes",
|
||||
json.dumps(
|
||||
{
|
||||
"source": "package",
|
||||
"manifest": {
|
||||
"artifact_kind": "tarball",
|
||||
"package_name": "domaincheck-2026.04.18+bd6bcb2.tgz",
|
||||
"included_services": ["domaincheck-api", "domaincheck-worker"],
|
||||
"rollback_hint": "切回上一个 current",
|
||||
},
|
||||
}
|
||||
),
|
||||
"www",
|
||||
datetime(2026, 4, 18, 9, 0, 0),
|
||||
datetime(2026, 4, 18, 9, 5, 0),
|
||||
datetime(2026, 4, 18, 9, 6, 0),
|
||||
)
|
||||
|
||||
payload = _serialize_release_row(row)
|
||||
|
||||
self.assertEqual("已就绪", payload["status_label"])
|
||||
self.assertIn("已就绪", payload["summary_text"])
|
||||
self.assertEqual("release_hub", payload["focus_ref"]["kind"])
|
||||
self.assertEqual(9, payload["focus_ref"]["release_id"])
|
||||
self.assertEqual("release_detail", payload["focus_ref"]["section"])
|
||||
self.assertEqual("tarball", payload["manifest"]["artifact_kind"])
|
||||
self.assertEqual("domaincheck-2026.04.18+bd6bcb2.tgz", payload["manifest"]["package_name"])
|
||||
self.assertEqual(["domaincheck-api", "domaincheck-worker"], payload["manifest"]["included_services"])
|
||||
|
||||
def test_serialize_rollout_row_adds_contract_aliases(self) -> None:
|
||||
row = (
|
||||
11,
|
||||
9,
|
||||
"rollout-20260418090000-a1b2",
|
||||
json.dumps({"region": "mainland", "role": "worker"}),
|
||||
json.dumps(
|
||||
{
|
||||
"items": [
|
||||
{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker"},
|
||||
{"node_code": "mainland-worker-02", "region": "mainland", "role": "worker"},
|
||||
]
|
||||
}
|
||||
),
|
||||
json.dumps({"execution_mode": "ssh", "batch_size": 1}),
|
||||
"running",
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
1,
|
||||
json.dumps(
|
||||
{
|
||||
"status": "running",
|
||||
"launched_targets": 1,
|
||||
"remaining_targets": 1,
|
||||
"active_jobs": 1,
|
||||
"current_batch_index": 1,
|
||||
"batches_total": 2,
|
||||
}
|
||||
),
|
||||
"web-ui",
|
||||
datetime(2026, 4, 18, 9, 10, 0),
|
||||
datetime(2026, 4, 18, 9, 11, 0),
|
||||
)
|
||||
|
||||
payload = _serialize_rollout_row(row)
|
||||
|
||||
self.assertEqual("执行中", payload["status_label"])
|
||||
self.assertEqual(["mainland-worker-01", "mainland-worker-02"], payload["target_node_codes"])
|
||||
self.assertIn("当前已覆盖 1/2 台节点", payload["summary_text"])
|
||||
self.assertEqual("release_rollout", payload["focus_ref"]["kind"])
|
||||
self.assertEqual(11, payload["focus_ref"]["rollout_id"])
|
||||
self.assertEqual("rollout_jobs", payload["focus_ref"]["section"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
432
domain-api/tests/test_ops_release_service_execution_modes.py
Normal file
432
domain-api/tests/test_ops_release_service_execution_modes.py
Normal file
@@ -0,0 +1,432 @@
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.services.ops_release_service import (
|
||||
build_release_execution_mode_recommendation,
|
||||
build_release_rollout_gate,
|
||||
build_rollout_target_operational_readiness,
|
||||
)
|
||||
|
||||
|
||||
class _EmptyCursor:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def execute(self, query, params=None) -> None:
|
||||
self.query = query
|
||||
self.params = params
|
||||
|
||||
def fetchall(self):
|
||||
return []
|
||||
|
||||
def fetchone(self):
|
||||
return None
|
||||
|
||||
|
||||
class _EmptyConnection:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def cursor(self):
|
||||
return _EmptyCursor()
|
||||
|
||||
|
||||
class OpsReleaseServiceExecutionModeTests(unittest.TestCase):
|
||||
@patch("app.services.ops_release_service.get_db")
|
||||
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
|
||||
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
|
||||
@patch("app.services.ops_job_service.list_managed_nodes")
|
||||
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
|
||||
def test_build_rollout_target_operational_readiness_accepts_ssh_ready_nodes(
|
||||
self,
|
||||
mock_get_cluster_snapshot,
|
||||
mock_list_managed_nodes,
|
||||
mock_get_managed_node_onboarding,
|
||||
mock_ensure_ops_agent_schema,
|
||||
mock_get_db,
|
||||
) -> None:
|
||||
mock_ensure_ops_agent_schema.return_value = None
|
||||
mock_get_managed_node_onboarding.return_value = {
|
||||
"onboarding_stage": {"code": "acceptance_ready", "label": "待接管验收"},
|
||||
"summary": "Node Agent 已在线,可以直接跑 onboarding.acceptance。",
|
||||
"recovery_decision": {
|
||||
"action": "run_acceptance",
|
||||
"label": "执行接管验收",
|
||||
"summary": "Node Agent 已在线,可以直接跑 onboarding.acceptance。",
|
||||
"command_hint": "bash drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-worker-01 cli",
|
||||
"window": "acceptance",
|
||||
},
|
||||
}
|
||||
mock_get_db.return_value = _EmptyConnection()
|
||||
mock_get_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_list_managed_nodes.return_value = [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"ssh_host": "121.204.244.248",
|
||||
"ssh_user": "root",
|
||||
"metadata": {},
|
||||
"last_seen_at": "",
|
||||
}
|
||||
]
|
||||
|
||||
readiness = build_rollout_target_operational_readiness(
|
||||
[{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker"}],
|
||||
execution_mode="ssh",
|
||||
)
|
||||
|
||||
self.assertEqual("ssh", readiness["execution_mode"])
|
||||
self.assertEqual(1, readiness["summary"]["ssh_ready_nodes"])
|
||||
self.assertEqual(1, readiness["summary"]["execution_ready_nodes"])
|
||||
self.assertEqual([], readiness["blocking_reasons"])
|
||||
self.assertTrue(readiness["rows"][0]["ssh_ready"])
|
||||
self.assertTrue(readiness["rows"][0]["execution_ready"])
|
||||
self.assertTrue(readiness["rows"][0]["agent_managed"])
|
||||
self.assertEqual("ssh", readiness["rows"][0]["execution_mode"])
|
||||
self.assertEqual("SSH", readiness["rows"][0]["execution_mode_label"])
|
||||
self.assertEqual("medium", readiness["rows"][0]["risk_level"])
|
||||
self.assertEqual("run_standard_inspection", readiness["rows"][0]["recommended_action_code"])
|
||||
self.assertEqual("acceptance_ready", readiness["rows"][0]["onboarding_stage_code"])
|
||||
self.assertEqual("run_acceptance", readiness["rows"][0]["recovery_action"])
|
||||
self.assertEqual(1, readiness["summary"]["onboarding_acceptance_ready_nodes"])
|
||||
|
||||
@patch("app.services.ops_release_service.get_db")
|
||||
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
|
||||
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
|
||||
@patch("app.services.ops_job_service.list_managed_nodes")
|
||||
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
|
||||
def test_build_rollout_target_operational_readiness_adds_release_context_and_focus_ref(
|
||||
self,
|
||||
mock_get_cluster_snapshot,
|
||||
mock_list_managed_nodes,
|
||||
mock_get_managed_node_onboarding,
|
||||
mock_ensure_ops_agent_schema,
|
||||
mock_get_db,
|
||||
) -> None:
|
||||
mock_ensure_ops_agent_schema.return_value = None
|
||||
mock_get_managed_node_onboarding.return_value = {
|
||||
"onboarding_stage": {"code": "ready", "label": "已接管"},
|
||||
"summary": "当前节点无需额外恢复动作。",
|
||||
"recovery_decision": {
|
||||
"action": "noop",
|
||||
"label": "当前无需额外恢复动作",
|
||||
"summary": "当前节点暂无需要执行的接管恢复动作。",
|
||||
"command_hint": "",
|
||||
"window": "none",
|
||||
},
|
||||
}
|
||||
mock_get_db.return_value = _EmptyConnection()
|
||||
mock_get_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
"metadata": {"current_release_version": "2026.04.11+1921318"},
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_list_managed_nodes.return_value = [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"ssh_host": "121.204.244.248",
|
||||
"ssh_user": "root",
|
||||
"metadata": {"current_release_version": "2026.04.11+1921318"},
|
||||
"last_seen_at": "2026-04-18 08:00:00",
|
||||
}
|
||||
]
|
||||
|
||||
readiness = build_rollout_target_operational_readiness(
|
||||
[{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker"}],
|
||||
execution_mode="remote-agent",
|
||||
desired_release={"id": 18, "release_version": "2026.04.18+bd6bcb2", "channel": "stable"},
|
||||
)
|
||||
|
||||
row = readiness["rows"][0]
|
||||
self.assertEqual("2026.04.11+1921318", row["current_release_version"])
|
||||
self.assertEqual("2026.04.18+bd6bcb2", row["desired_release_version"])
|
||||
self.assertEqual("release_hub", row["focus_ref"]["kind"])
|
||||
self.assertEqual("default_rollout_gate", row["focus_ref"]["section"])
|
||||
self.assertEqual("mainland-worker-01", row["focus_ref"]["node_code"])
|
||||
self.assertEqual("noop", row["recovery_action"])
|
||||
|
||||
@patch("app.services.ops_release_service.get_db")
|
||||
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
|
||||
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
|
||||
@patch("app.services.ops_job_service.list_managed_nodes")
|
||||
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
|
||||
def test_build_rollout_target_operational_readiness_prefers_recovery_decision_for_blocked_nodes(
|
||||
self,
|
||||
mock_get_cluster_snapshot,
|
||||
mock_list_managed_nodes,
|
||||
mock_get_managed_node_onboarding,
|
||||
mock_ensure_ops_agent_schema,
|
||||
mock_get_db,
|
||||
) -> None:
|
||||
mock_ensure_ops_agent_schema.return_value = None
|
||||
mock_get_db.return_value = _EmptyConnection()
|
||||
mock_get_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_list_managed_nodes.return_value = [
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"ssh_host": "121.204.244.249",
|
||||
"ssh_user": "root",
|
||||
"metadata": {},
|
||||
"last_seen_at": "",
|
||||
}
|
||||
]
|
||||
mock_get_managed_node_onboarding.return_value = {
|
||||
"onboarding_stage": {"code": "pending_bootstrap", "label": "待接入"},
|
||||
"summary": "已签发有效 Agent Token,但节点尚未完成 register/heartbeat。",
|
||||
"recovery_decision": {
|
||||
"action": "bootstrap_run",
|
||||
"label": "签发接入工单",
|
||||
"summary": "当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。",
|
||||
"command_hint": "bash drive_ops_center.sh node-bootstrap-run http://127.0.0.1:8100 mainland-worker-02 cli",
|
||||
"window": "bootstrap",
|
||||
},
|
||||
}
|
||||
|
||||
readiness = build_rollout_target_operational_readiness(
|
||||
[{"node_code": "mainland-worker-02", "region": "mainland", "role": "worker"}],
|
||||
execution_mode="remote-agent",
|
||||
)
|
||||
|
||||
row = readiness["rows"][0]
|
||||
self.assertEqual("bootstrap_run", row["recovery_action"])
|
||||
self.assertEqual("bootstrap_run", row["recommended_action_code"])
|
||||
self.assertEqual(1, readiness["summary"]["onboarding_bootstrap_pending_nodes"])
|
||||
self.assertTrue(
|
||||
any("onboarding.bootstrap" in item for item in readiness["recommendations"])
|
||||
)
|
||||
|
||||
@patch("app.services.ops_release_service.get_db")
|
||||
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
|
||||
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
|
||||
@patch("app.services.ops_job_service.list_managed_nodes")
|
||||
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
|
||||
def test_build_rollout_target_operational_readiness_blocks_nodes_with_dead_letter_queue(
|
||||
self,
|
||||
mock_get_cluster_snapshot,
|
||||
mock_list_managed_nodes,
|
||||
mock_get_managed_node_onboarding,
|
||||
mock_ensure_ops_agent_schema,
|
||||
mock_get_db,
|
||||
) -> None:
|
||||
mock_ensure_ops_agent_schema.return_value = None
|
||||
mock_get_db.return_value = _EmptyConnection()
|
||||
mock_get_managed_node_onboarding.return_value = {
|
||||
"onboarding_stage": {"code": "ready", "label": "已接管"},
|
||||
"summary": "当前节点无需额外恢复动作。",
|
||||
"recovery_decision": {
|
||||
"action": "noop",
|
||||
"label": "当前无需额外恢复动作",
|
||||
"summary": "当前节点暂无需要执行的接管恢复动作。",
|
||||
"command_hint": "",
|
||||
"window": "none",
|
||||
},
|
||||
}
|
||||
mock_get_cluster_snapshot.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-03",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"status": "online",
|
||||
"current_load": 0,
|
||||
"is_effective_worker": True,
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_list_managed_nodes.return_value = [
|
||||
{
|
||||
"node_code": "mainland-worker-03",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"ssh_host": "121.204.244.250",
|
||||
"ssh_user": "root",
|
||||
"metadata": {
|
||||
"delivery_queue": {
|
||||
"state": "dead_letter",
|
||||
"label": "死信 1",
|
||||
"reason": "当前存在 1 条死信记录",
|
||||
"pending_count": 0,
|
||||
"dead_letter_count": 1,
|
||||
"oldest_dead_letter_at": "2026-04-18 09:01:00",
|
||||
}
|
||||
},
|
||||
"last_seen_at": "2026-04-18 09:05:00",
|
||||
}
|
||||
]
|
||||
|
||||
readiness = build_rollout_target_operational_readiness(
|
||||
[{"node_code": "mainland-worker-03", "region": "mainland", "role": "worker"}],
|
||||
execution_mode="remote-agent",
|
||||
)
|
||||
|
||||
row = readiness["rows"][0]
|
||||
self.assertEqual("dead_letter", row["delivery_queue_state"])
|
||||
self.assertEqual(1, row["delivery_queue_dead_letter_count"])
|
||||
self.assertTrue(
|
||||
any("死信" in item for item in row["blocking_reasons"])
|
||||
)
|
||||
self.assertEqual(1, readiness["summary"]["queue_dead_letter_nodes"])
|
||||
self.assertTrue(
|
||||
any("死信回执" in item for item in readiness["blocking_reasons"])
|
||||
)
|
||||
self.assertTrue(
|
||||
any("死信队列" in item for item in readiness["recommendations"])
|
||||
)
|
||||
|
||||
@patch("app.services.ops_release_service.build_rollout_target_operational_readiness")
|
||||
def test_build_release_rollout_gate_uses_execution_ready_count_for_ssh(self, mock_build_readiness) -> None:
|
||||
mock_build_readiness.return_value = {
|
||||
"execution_mode": "ssh",
|
||||
"summary": {
|
||||
"nodes_total": 2,
|
||||
"execution_ready_nodes": 1,
|
||||
"inspection_healthy_nodes": 2,
|
||||
},
|
||||
"rows": [],
|
||||
"blocking_reasons": ["有 1 台目标节点尚未配置完整 SSH 入口,当前不适合直接 SSH 发布。"],
|
||||
"warning_reasons": [],
|
||||
"recommendations": [],
|
||||
}
|
||||
|
||||
gate = build_release_rollout_gate(
|
||||
{
|
||||
"id": 11,
|
||||
"release_version": "2026.04.18-rc1",
|
||||
"status": "ready",
|
||||
"artifact_url": "https://example.com/domaincheck.tar.gz",
|
||||
},
|
||||
[{"node_code": "mainland-worker-01"}, {"node_code": "mainland-worker-02"}],
|
||||
execution_mode="ssh",
|
||||
)
|
||||
|
||||
self.assertEqual("ssh", gate["execution_mode"])
|
||||
self.assertEqual("SSH", gate["execution_mode_label"])
|
||||
self.assertEqual("blocked", gate["status"])
|
||||
self.assertIn("1/2 台 SSH 就绪", gate["summary"])
|
||||
|
||||
@patch("app.services.ops_release_service.build_rollout_target_operational_readiness")
|
||||
def test_build_release_rollout_gate_uses_human_label_for_remote_agent(self, mock_build_readiness) -> None:
|
||||
mock_build_readiness.return_value = {
|
||||
"execution_mode": "remote-agent",
|
||||
"summary": {
|
||||
"nodes_total": 2,
|
||||
"execution_ready_nodes": 1,
|
||||
"inspection_healthy_nodes": 2,
|
||||
},
|
||||
"rows": [],
|
||||
"blocking_reasons": ["有 1 台目标节点 Agent 未在线,当前不适合直接 remote-agent 发布。"],
|
||||
"warning_reasons": [],
|
||||
"recommendations": [],
|
||||
}
|
||||
|
||||
gate = build_release_rollout_gate(
|
||||
{
|
||||
"id": 12,
|
||||
"release_version": "2026.04.18-rc1",
|
||||
"status": "ready",
|
||||
"artifact_url": "https://example.com/domaincheck.tar.gz",
|
||||
},
|
||||
[{"node_code": "mainland-worker-01"}, {"node_code": "mainland-worker-02"}],
|
||||
execution_mode="remote-agent",
|
||||
)
|
||||
|
||||
self.assertEqual("remote-agent", gate["execution_mode"])
|
||||
self.assertEqual("远端 Agent", gate["execution_mode_label"])
|
||||
self.assertIn("1/2 台 远端 Agent 就绪", gate["summary"])
|
||||
|
||||
@patch("app.services.ops_release_service.build_release_rollout_gate")
|
||||
def test_build_release_execution_mode_recommendation_prefers_ssh_when_gate_is_better(self, mock_build_gate) -> None:
|
||||
def _fake_gate(release, target_nodes, *, execution_mode="remote-agent"):
|
||||
if execution_mode == "ssh":
|
||||
return {
|
||||
"status": "ready",
|
||||
"status_label": "可发 Rollout",
|
||||
"summary": "SSH 路径已收口",
|
||||
"execution_mode": "ssh",
|
||||
"execution_mode_label": "SSH",
|
||||
"blocking_reasons": [],
|
||||
"warning_reasons": [],
|
||||
"operational_readiness": {
|
||||
"summary": {"nodes_total": 2, "execution_ready_nodes": 2},
|
||||
"rows": [],
|
||||
},
|
||||
}
|
||||
return {
|
||||
"status": "blocked",
|
||||
"status_label": "存在阻断",
|
||||
"summary": "remote-agent 尚未收口",
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "远端 Agent",
|
||||
"blocking_reasons": ["Agent 未就绪"],
|
||||
"warning_reasons": [],
|
||||
"operational_readiness": {
|
||||
"summary": {"nodes_total": 2, "execution_ready_nodes": 1},
|
||||
"rows": [],
|
||||
},
|
||||
}
|
||||
|
||||
mock_build_gate.side_effect = _fake_gate
|
||||
|
||||
recommendation = build_release_execution_mode_recommendation(
|
||||
{
|
||||
"id": 18,
|
||||
"release_version": "2026.04.18-rc2",
|
||||
"status": "ready",
|
||||
"artifact_url": "https://example.com/domaincheck.tar.gz",
|
||||
},
|
||||
[{"node_code": "mainland-worker-01"}, {"node_code": "mainland-worker-02"}],
|
||||
)
|
||||
|
||||
self.assertEqual("ssh", recommendation["recommended_mode"])
|
||||
self.assertEqual("SSH", recommendation["recommended_mode_label"])
|
||||
self.assertEqual("ready", recommendation["recommended_gate"]["status"])
|
||||
self.assertIn("SSH", recommendation["reason"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
212
domain-api/tests/test_ops_release_service_launchpad.py
Normal file
212
domain-api/tests/test_ops_release_service_launchpad.py
Normal file
@@ -0,0 +1,212 @@
|
||||
import unittest
|
||||
|
||||
from app.services.ops_release_service import (
|
||||
_build_release_launchpad_status,
|
||||
_compact_latest_smart_rollout_preview,
|
||||
)
|
||||
|
||||
|
||||
class OpsReleaseServiceLaunchpadTests(unittest.TestCase):
|
||||
def test_launchpad_status_blocks_when_package_missing(self) -> None:
|
||||
status = _build_release_launchpad_status(
|
||||
package={"available": False},
|
||||
latest_package_preview={},
|
||||
worker_preview={},
|
||||
control_preview={},
|
||||
)
|
||||
|
||||
self.assertEqual("blocked", status["status"])
|
||||
self.assertEqual("release_package", status["recommended_action_code"])
|
||||
self.assertEqual(status["summary"], status["summary_text"])
|
||||
self.assertEqual("release_hub", status["focus_ref"]["kind"])
|
||||
self.assertEqual("release_launchpad", status["focus_ref"]["section"])
|
||||
|
||||
def test_launchpad_status_blocks_when_rollout_preview_has_blockers(self) -> None:
|
||||
status = _build_release_launchpad_status(
|
||||
package={"available": True},
|
||||
latest_package_preview={},
|
||||
worker_preview={"policy_preview": {"blocked": True}},
|
||||
control_preview={"policy_preview": {"blocked": False}},
|
||||
)
|
||||
|
||||
self.assertEqual("blocked", status["status"])
|
||||
self.assertEqual("fix_rollout_blockers", status["recommended_action_code"])
|
||||
self.assertEqual(status["summary"], status["summary_text"])
|
||||
|
||||
def test_launchpad_status_blocks_when_final_signoff_missing(self) -> None:
|
||||
status = _build_release_launchpad_status(
|
||||
package={
|
||||
"available": True,
|
||||
"final_release_ok": False,
|
||||
"final_release_report_available": True,
|
||||
"final_release_report_ok": True,
|
||||
"final_release_report_matches_latest": False,
|
||||
"final_release_report_stale_reason": "final_release_report does not match latest_release: package_name",
|
||||
"final_release_gate_decision": "ready",
|
||||
"final_release_gate_blocked_reasons": [],
|
||||
},
|
||||
latest_package_preview={},
|
||||
worker_preview={"smart_rollout": {"available": True}, "policy_preview": {}},
|
||||
control_preview={"smart_rollout": {"available": True}, "policy_preview": {}},
|
||||
)
|
||||
|
||||
self.assertEqual("blocked", status["status"])
|
||||
self.assertEqual("release_prepare", status["recommended_action_code"])
|
||||
self.assertIn("final_release_report does not match latest_release: package_name", status["summary"])
|
||||
self.assertIn("release-prepare", status["recommended_command"])
|
||||
self.assertEqual(status["summary"], status["summary_text"])
|
||||
|
||||
def test_launchpad_status_prefers_runtime_refresh_when_runtime_schema_is_stale(self) -> None:
|
||||
status = _build_release_launchpad_status(
|
||||
package={"available": True, "final_release_ok": True},
|
||||
latest_package_preview={},
|
||||
worker_preview={"smart_rollout": {"available": True}, "policy_preview": {}},
|
||||
control_preview={"smart_rollout": {"available": True}, "policy_preview": {}},
|
||||
runtime_build_info={
|
||||
"runtime_schema_stale": True,
|
||||
"route_surface_complete": True,
|
||||
"route_surface_missing_keys": ["ops_node_handover_bootstrap_plan"],
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual("attention", status["status"])
|
||||
self.assertEqual("api-restart", status["recommended_action_code"])
|
||||
self.assertEqual("待刷新运行时", status["status_label"])
|
||||
self.assertTrue(status["runtime_refresh_required"])
|
||||
self.assertTrue(status["runtime_schema_stale"])
|
||||
self.assertIn("runtime-refresh-recover", status["recommended_command"])
|
||||
self.assertEqual(status["summary"], status["summary_text"])
|
||||
|
||||
def test_launchpad_status_marks_attention_when_confirmation_required(self) -> None:
|
||||
status = _build_release_launchpad_status(
|
||||
package={"available": True},
|
||||
latest_package_preview={},
|
||||
worker_preview={"requires_confirmation": True, "policy_preview": {}},
|
||||
control_preview={"policy_preview": {}},
|
||||
)
|
||||
|
||||
self.assertEqual("attention", status["status"])
|
||||
self.assertEqual("review_smart_rollout_preview", status["recommended_action_code"])
|
||||
self.assertEqual("release_launchpad", status["focus_ref"]["section"])
|
||||
|
||||
def test_launchpad_status_ready_when_worker_rollout_available(self) -> None:
|
||||
status = _build_release_launchpad_status(
|
||||
package={"available": True},
|
||||
latest_package_preview={},
|
||||
worker_preview={"smart_rollout": {"available": True, "execution_mode": "ssh", "execution_mode_label": "SSH"}, "policy_preview": {}},
|
||||
control_preview={"smart_rollout": {"available": False}, "policy_preview": {}},
|
||||
)
|
||||
|
||||
self.assertEqual("ready", status["status"])
|
||||
self.assertEqual("publish_latest_worker", status["recommended_action_code"])
|
||||
self.assertEqual("ssh", status["recommended_execution_mode"])
|
||||
self.assertEqual("SSH", status["recommended_execution_mode_label"])
|
||||
self.assertEqual(status["summary"], status["summary_text"])
|
||||
|
||||
def test_launchpad_status_requires_managed_nodes_when_no_targets_available(self) -> None:
|
||||
status = _build_release_launchpad_status(
|
||||
package={"available": True},
|
||||
latest_package_preview={},
|
||||
worker_preview={"smart_rollout": {"available": False}, "policy_preview": {}},
|
||||
control_preview={"smart_rollout": {"available": False}, "policy_preview": {}},
|
||||
)
|
||||
|
||||
self.assertEqual("attention", status["status"])
|
||||
self.assertEqual("fix_managed_nodes", status["recommended_action_code"])
|
||||
self.assertEqual("release_hub", status["focus_ref"]["kind"])
|
||||
|
||||
def test_launchpad_status_prefers_bootstrap_recovery_when_gap_row_has_recovery_action(self) -> None:
|
||||
status = _build_release_launchpad_status(
|
||||
package={"available": True},
|
||||
latest_package_preview={},
|
||||
worker_preview={
|
||||
"smart_rollout": {"available": False},
|
||||
"policy_preview": {
|
||||
"blocked": True,
|
||||
"release_gate": {
|
||||
"operational_readiness": {
|
||||
"rows": [
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"recovery_action": "bootstrap_run",
|
||||
"recovery_label": "签发接入工单",
|
||||
"recovery_summary": "当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。",
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
},
|
||||
},
|
||||
control_preview={"smart_rollout": {"available": False}, "policy_preview": {}},
|
||||
)
|
||||
|
||||
self.assertEqual("blocked", status["status"])
|
||||
self.assertEqual("bootstrap_run", status["recommended_action_code"])
|
||||
self.assertEqual("mainland-worker-02", status["recommended_target_node_code"])
|
||||
self.assertEqual("签发接入工单", status["recommended_recovery_label"])
|
||||
self.assertEqual("当前节点仍处于接入阶段,建议先签发 onboarding.bootstrap。", status["recommended_recovery_summary"])
|
||||
self.assertEqual(1, status["onboarding_bootstrap_pending_nodes"])
|
||||
self.assertEqual(0, status["onboarding_acceptance_ready_nodes"])
|
||||
self.assertEqual(1, status["onboarding_gap_nodes_total"])
|
||||
self.assertIn("node-bootstrap-run", status["recommended_command"])
|
||||
|
||||
def test_compact_latest_smart_rollout_preview_adds_rollout_preview_alias(self) -> None:
|
||||
preview = _compact_latest_smart_rollout_preview(
|
||||
"智能 Rollout 预检通过",
|
||||
{
|
||||
"release": {
|
||||
"id": 18,
|
||||
"release_version": "2026.04.18+bd6bcb2",
|
||||
"channel": "stable",
|
||||
"status": "ready",
|
||||
"artifact_url": "https://example.com/domaincheck.tar.gz",
|
||||
},
|
||||
"package": {
|
||||
"available": True,
|
||||
"package_name": "domaincheck-2026.04.18+bd6bcb2",
|
||||
"package_type": "tar.gz",
|
||||
"sha256": "sha256:test",
|
||||
"metadata": {
|
||||
"manifest": {
|
||||
"included_services": ["domaincheck-api", "domaincheck-worker"],
|
||||
}
|
||||
},
|
||||
},
|
||||
"smart_rollout": {
|
||||
"available": True,
|
||||
"execution_mode": "remote-agent",
|
||||
"execution_mode_label": "远端 Agent",
|
||||
"summary": "已按 Worker 预选 2 台节点。",
|
||||
"target_selector": {"node_codes": ["mainland-worker-01", "mainland-worker-02"]},
|
||||
"target_nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"delivery_queue_state": "healthy",
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"delivery_queue_state": "retrying",
|
||||
},
|
||||
],
|
||||
},
|
||||
"preview": {
|
||||
"target_summary": {"nodes_total": 2},
|
||||
"batch_plan": {"batches_total": 2, "first_batch_size": 1},
|
||||
"release_gate": {"status": "ready", "summary": "当前可推进。"},
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual("remote-agent", preview["rollout_preview"]["execution_mode"])
|
||||
self.assertEqual(2, preview["rollout_preview"]["target_nodes_total"])
|
||||
self.assertEqual(2, preview["rollout_preview"]["expected_batches_total"])
|
||||
self.assertEqual(2, preview["rollout_preview"]["expected_jobs_total"])
|
||||
self.assertEqual("release_hub", preview["rollout_preview"]["focus_ref"]["kind"])
|
||||
self.assertEqual("healthy", preview["smart_rollout"]["target_nodes"][0]["delivery_queue_state"])
|
||||
self.assertEqual("retrying", preview["smart_rollout"]["target_nodes"][1]["delivery_queue_state"])
|
||||
self.assertEqual("rollout_preview", preview["rollout_preview"]["focus_ref"]["section"])
|
||||
self.assertEqual(["domaincheck-api", "domaincheck-worker"], preview["package"]["manifest"]["included_services"])
|
||||
915
domain-api/tests/test_ops_release_service_package_metadata.py
Normal file
915
domain-api/tests/test_ops_release_service_package_metadata.py
Normal file
@@ -0,0 +1,915 @@
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.services.ops_release_service import (
|
||||
build_smart_release_rollout_plan,
|
||||
create_release_and_smart_rollout_from_latest_package,
|
||||
create_release_from_latest_package,
|
||||
create_smart_release_rollout,
|
||||
get_latest_release_package_metadata,
|
||||
get_release_package_archive_path,
|
||||
get_release_package_sha256_path,
|
||||
)
|
||||
|
||||
|
||||
class OpsReleaseServicePackageMetadataTests(unittest.TestCase):
|
||||
def test_get_latest_release_package_metadata_reads_latest_linux_package(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
release_root = Path(tmpdir)
|
||||
package_name = "domaincheck_release_20260418_120000"
|
||||
staging_path = release_root / package_name
|
||||
archive_path = release_root / f"{package_name}.tar.gz"
|
||||
sha256_path = release_root / f"{package_name}.sha256.txt"
|
||||
|
||||
staging_path.mkdir(parents=True, exist_ok=True)
|
||||
archive_path.write_bytes(b"linux-package")
|
||||
sha256_path.write_text("sha256=test-sha256", encoding="utf-8")
|
||||
(staging_path / "release_manifest.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"package_name": package_name,
|
||||
"package_type": "tar.gz",
|
||||
"included": {
|
||||
"docs": ["08_domainCheck_交付说明.md"],
|
||||
"scripts": ["package_domain_release.sh", "verify_domain_release.sh"],
|
||||
},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(release_root / "latest_release.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"package_name": package_name,
|
||||
"package_type": "tar.gz",
|
||||
"generated_at": "2026-04-18T12:00:00",
|
||||
"staging_path": str(staging_path),
|
||||
"archive_path": str(archive_path),
|
||||
"sha256_path": str(sha256_path),
|
||||
"sha256": "test-sha256",
|
||||
"smoke_test_ok": True,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(release_root / "final_release_report.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"ok": True,
|
||||
"latest_release": {
|
||||
"package_name": package_name,
|
||||
"archive_path": str(archive_path),
|
||||
"sha256": "test-sha256",
|
||||
},
|
||||
"release_gate": {
|
||||
"decision": "ready",
|
||||
"blocked_reasons": [],
|
||||
"verify_ok": True,
|
||||
"smoke_test_ok": True,
|
||||
},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
with patch.dict(os.environ, {"DOMAINCHECK_RELEASE_ROOT": str(release_root)}):
|
||||
data = get_latest_release_package_metadata()
|
||||
|
||||
self.assertTrue(data["available"])
|
||||
self.assertEqual(package_name, data["package_name"])
|
||||
self.assertEqual("tar.gz", data["package_type"])
|
||||
self.assertTrue(data["archive_present"])
|
||||
self.assertTrue(data["sha256_present"])
|
||||
self.assertTrue(data["staging_present"])
|
||||
self.assertEqual("test-sha256", data["sha256"])
|
||||
self.assertEqual(f"/api/v1/ops/releases/packages/{package_name}/download", data["download_path"])
|
||||
self.assertEqual(package_name, data["release_version_suggestion"])
|
||||
self.assertIn("自检通过", data["notes_suggestion"])
|
||||
self.assertTrue(data["final_release_ok"])
|
||||
self.assertTrue(data["final_release_report_available"])
|
||||
self.assertTrue(data["final_release_report_matches_latest"])
|
||||
self.assertEqual("ready", data["final_release_gate_decision"])
|
||||
self.assertEqual([], data["final_release_gate_blocked_reasons"])
|
||||
self.assertEqual(package_name, data["metadata"]["manifest"]["package_name"])
|
||||
|
||||
def test_get_release_package_paths_support_metadata_fallback_for_zip(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
release_root = Path(tmpdir)
|
||||
package_name = "domaincheck_release_windows_demo"
|
||||
package_dir = release_root / "exports"
|
||||
package_dir.mkdir(parents=True, exist_ok=True)
|
||||
archive_path = package_dir / f"{package_name}.zip"
|
||||
sha256_path = package_dir / f"{package_name}.sha256.txt"
|
||||
|
||||
archive_path.write_bytes(b"zip-package")
|
||||
sha256_path.write_text("sha256=zip-sha256", encoding="utf-8")
|
||||
(release_root / "latest_release.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"package_name": package_name,
|
||||
"package_type": "zip",
|
||||
"generated_at": "2026-04-18T12:10:00",
|
||||
"zip_path": str(archive_path),
|
||||
"sha256_path": str(sha256_path),
|
||||
"sha256": "zip-sha256",
|
||||
"smoke_test_ok": False,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
with patch.dict(os.environ, {"DOMAINCHECK_RELEASE_ROOT": str(release_root)}):
|
||||
resolved_archive = get_release_package_archive_path(package_name)
|
||||
resolved_sha256 = get_release_package_sha256_path(package_name)
|
||||
metadata = get_latest_release_package_metadata()
|
||||
|
||||
self.assertEqual(archive_path, resolved_archive)
|
||||
self.assertEqual(sha256_path, resolved_sha256)
|
||||
self.assertEqual("zip", metadata["package_type"])
|
||||
self.assertEqual(archive_path.name, metadata["archive_filename"])
|
||||
self.assertFalse(metadata["smoke_test_ok"])
|
||||
|
||||
def test_get_latest_release_package_metadata_returns_unavailable_when_missing(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
with patch.dict(os.environ, {"DOMAINCHECK_RELEASE_ROOT": tmpdir}):
|
||||
data = get_latest_release_package_metadata()
|
||||
|
||||
self.assertFalse(data["available"])
|
||||
self.assertIn("latest_release.json", data["reason"])
|
||||
|
||||
def test_get_latest_release_package_metadata_marks_stale_final_release_report(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tmpdir:
|
||||
release_root = Path(tmpdir)
|
||||
package_name = "domaincheck_release_20260418_130000"
|
||||
archive_path = release_root / f"{package_name}.tar.gz"
|
||||
sha256_path = release_root / f"{package_name}.sha256.txt"
|
||||
|
||||
archive_path.write_bytes(b"linux-package")
|
||||
sha256_path.write_text("sha256=test-sha256", encoding="utf-8")
|
||||
(release_root / "latest_release.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"package_name": package_name,
|
||||
"package_type": "tar.gz",
|
||||
"archive_path": str(archive_path),
|
||||
"sha256_path": str(sha256_path),
|
||||
"sha256": "test-sha256",
|
||||
"smoke_test_ok": True,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(release_root / "final_release_report.json").write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"ok": True,
|
||||
"latest_release": {
|
||||
"package_name": "domaincheck_release_20260418_old",
|
||||
"archive_path": str(release_root / "domaincheck_release_20260418_old.tar.gz"),
|
||||
"sha256": "old-sha256",
|
||||
},
|
||||
"release_gate": {
|
||||
"decision": "ready",
|
||||
"blocked_reasons": [],
|
||||
"verify_ok": True,
|
||||
"smoke_test_ok": True,
|
||||
},
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
with patch.dict(os.environ, {"DOMAINCHECK_RELEASE_ROOT": str(release_root)}):
|
||||
data = get_latest_release_package_metadata()
|
||||
|
||||
self.assertTrue(data["available"])
|
||||
self.assertFalse(data["final_release_ok"])
|
||||
self.assertTrue(data["final_release_report_available"])
|
||||
self.assertTrue(data["final_release_report_ok"])
|
||||
self.assertFalse(data["final_release_report_matches_latest"])
|
||||
self.assertIn("package_name", data["final_release_report_stale_reason"])
|
||||
|
||||
@patch("app.services.ops_release_service.create_release")
|
||||
@patch("app.services.ops_release_service._get_release_by_version")
|
||||
@patch("app.services.ops_release_service.get_latest_release_package_metadata")
|
||||
def test_create_release_from_latest_package_builds_payload(
|
||||
self,
|
||||
mock_get_latest_release_package_metadata,
|
||||
mock_get_release_by_version,
|
||||
mock_create_release,
|
||||
) -> None:
|
||||
mock_get_latest_release_package_metadata.return_value = {
|
||||
"available": True,
|
||||
"package_name": "domaincheck_release_20260418_120000",
|
||||
"release_version_suggestion": "domaincheck_release_20260418_120000",
|
||||
"download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_120000/download",
|
||||
"sha256_download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_120000/sha256",
|
||||
"sha256": "test-sha256",
|
||||
"commit_sha": "bd6bcb2",
|
||||
"notes_suggestion": "导入自本机发布包 domaincheck_release_20260418_120000.tar.gz;自检通过",
|
||||
"smoke_test_ok": True,
|
||||
"final_release_ok": True,
|
||||
"final_release_report_available": True,
|
||||
"final_release_report_ok": True,
|
||||
"final_release_report_matches_latest": True,
|
||||
"final_release_gate_decision": "ready",
|
||||
"final_release_gate_blocked_reasons": [],
|
||||
"metadata": {
|
||||
"package_name": "domaincheck_release_20260418_120000",
|
||||
"package_type": "tar.gz",
|
||||
},
|
||||
}
|
||||
mock_get_release_by_version.return_value = {}
|
||||
mock_create_release.return_value = (
|
||||
True,
|
||||
"Release 已创建",
|
||||
{
|
||||
"release": {
|
||||
"id": 7,
|
||||
"release_version": "domaincheck_release_20260418_120000",
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
ok, message, data = create_release_from_latest_package(
|
||||
{
|
||||
"channel": "stable",
|
||||
"status": "ready",
|
||||
"created_by": "test-suite",
|
||||
},
|
||||
control_plane_base_url="https://ops.example.com/api/v1",
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("Release 已创建", message)
|
||||
mock_create_release.assert_called_once()
|
||||
create_payload = mock_create_release.call_args.args[0]
|
||||
self.assertEqual("domaincheck_release_20260418_120000", create_payload["release_version"])
|
||||
self.assertEqual("bd6bcb2", create_payload["commit_sha"])
|
||||
self.assertEqual(
|
||||
"https://ops.example.com/api/v1/ops/releases/packages/domaincheck_release_20260418_120000/download",
|
||||
create_payload["artifact_url"],
|
||||
)
|
||||
self.assertEqual("test-sha256", create_payload["checksum"])
|
||||
self.assertEqual("latest_release_package", create_payload["metadata"]["created_from"])
|
||||
self.assertEqual(
|
||||
"https://ops.example.com/api/v1/ops/releases/packages/domaincheck_release_20260418_120000/download",
|
||||
create_payload["metadata"]["package_source"]["download_url"],
|
||||
)
|
||||
self.assertFalse(data["deduplicated"])
|
||||
|
||||
@patch("app.services.ops_release_service.create_release")
|
||||
@patch("app.services.ops_release_service._get_release_by_version")
|
||||
@patch("app.services.ops_release_service.get_latest_release_package_metadata")
|
||||
def test_create_release_from_latest_package_is_idempotent(
|
||||
self,
|
||||
mock_get_latest_release_package_metadata,
|
||||
mock_get_release_by_version,
|
||||
mock_create_release,
|
||||
) -> None:
|
||||
mock_get_latest_release_package_metadata.return_value = {
|
||||
"available": True,
|
||||
"package_name": "domaincheck_release_20260418_120000",
|
||||
"release_version_suggestion": "domaincheck_release_20260418_120000",
|
||||
"smoke_test_ok": True,
|
||||
"final_release_ok": True,
|
||||
"final_release_report_available": True,
|
||||
"final_release_report_ok": True,
|
||||
"final_release_report_matches_latest": True,
|
||||
"final_release_gate_decision": "ready",
|
||||
"final_release_gate_blocked_reasons": [],
|
||||
}
|
||||
mock_get_release_by_version.return_value = {
|
||||
"id": 9,
|
||||
"release_version": "domaincheck_release_20260418_120000",
|
||||
}
|
||||
|
||||
ok, message, data = create_release_from_latest_package({}, control_plane_base_url="https://ops.example.com")
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("Release 已存在,按幂等返回", message)
|
||||
self.assertTrue(data["deduplicated"])
|
||||
self.assertEqual(9, data["release"]["id"])
|
||||
mock_create_release.assert_not_called()
|
||||
|
||||
@patch("app.services.ops_release_service.create_release")
|
||||
@patch("app.services.ops_release_service._get_release_by_version")
|
||||
@patch("app.services.ops_release_service.get_latest_release_package_metadata")
|
||||
def test_create_release_from_latest_package_blocks_when_final_signoff_missing(
|
||||
self,
|
||||
mock_get_latest_release_package_metadata,
|
||||
mock_get_release_by_version,
|
||||
mock_create_release,
|
||||
) -> None:
|
||||
mock_get_latest_release_package_metadata.return_value = {
|
||||
"available": True,
|
||||
"package_name": "domaincheck_release_20260418_130000",
|
||||
"release_version_suggestion": "domaincheck_release_20260418_130000",
|
||||
"download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_130000/download",
|
||||
"sha256_download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_130000/sha256",
|
||||
"sha256": "test-sha256",
|
||||
"commit_sha": "bd6bcb2",
|
||||
"smoke_test_ok": False,
|
||||
"final_release_ok": False,
|
||||
"final_release_report_available": False,
|
||||
"final_release_report_ok": None,
|
||||
"final_release_report_matches_latest": None,
|
||||
"final_release_gate_decision": "",
|
||||
"final_release_gate_blocked_reasons": [],
|
||||
"metadata": {
|
||||
"package_name": "domaincheck_release_20260418_130000",
|
||||
"package_type": "tar.gz",
|
||||
},
|
||||
}
|
||||
mock_get_release_by_version.return_value = {}
|
||||
|
||||
ok, message, data = create_release_from_latest_package(
|
||||
{
|
||||
"channel": "stable",
|
||||
"status": "ready",
|
||||
"created_by": "test-suite",
|
||||
},
|
||||
control_plane_base_url="https://ops.example.com/api/v1",
|
||||
)
|
||||
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("最终签收", message)
|
||||
self.assertFalse(data["final_release_gate"]["ready"])
|
||||
self.assertIn("smoke_test_failed_or_skipped", data["final_release_gate"]["blocked_reasons"])
|
||||
self.assertIn("final_release_report_missing", data["final_release_gate"]["blocked_reasons"])
|
||||
mock_create_release.assert_not_called()
|
||||
|
||||
@patch("app.services.ops_release_service.create_release")
|
||||
@patch("app.services.ops_release_service._get_release_by_version")
|
||||
@patch("app.services.ops_release_service.get_latest_release_package_metadata")
|
||||
def test_create_release_from_latest_package_blocks_when_final_report_is_stale(
|
||||
self,
|
||||
mock_get_latest_release_package_metadata,
|
||||
mock_get_release_by_version,
|
||||
mock_create_release,
|
||||
) -> None:
|
||||
mock_get_latest_release_package_metadata.return_value = {
|
||||
"available": True,
|
||||
"package_name": "domaincheck_release_20260418_130001",
|
||||
"release_version_suggestion": "domaincheck_release_20260418_130001",
|
||||
"download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_130001/download",
|
||||
"sha256_download_path": "/api/v1/ops/releases/packages/domaincheck_release_20260418_130001/sha256",
|
||||
"sha256": "test-sha256",
|
||||
"commit_sha": "bd6bcb2",
|
||||
"smoke_test_ok": True,
|
||||
"final_release_ok": False,
|
||||
"final_release_report_available": True,
|
||||
"final_release_report_ok": True,
|
||||
"final_release_report_matches_latest": False,
|
||||
"final_release_report_stale_reason": "final_release_report does not match latest_release: package_name",
|
||||
"final_release_gate_decision": "ready",
|
||||
"final_release_gate_blocked_reasons": [],
|
||||
"metadata": {
|
||||
"package_name": "domaincheck_release_20260418_130001",
|
||||
"package_type": "tar.gz",
|
||||
},
|
||||
}
|
||||
mock_get_release_by_version.return_value = {
|
||||
"id": 19,
|
||||
"release_version": "domaincheck_release_20260418_130001",
|
||||
}
|
||||
|
||||
ok, message, data = create_release_from_latest_package(
|
||||
{"created_by": "test-suite"},
|
||||
control_plane_base_url="https://ops.example.com/api/v1",
|
||||
)
|
||||
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("最终签收", message)
|
||||
self.assertFalse(data["final_release_gate"]["ready"])
|
||||
self.assertEqual(
|
||||
"final_release_report does not match latest_release: package_name",
|
||||
data["final_release_gate"]["stale_reason"],
|
||||
)
|
||||
self.assertIn("final_release_report_not_matching_latest", data["final_release_gate"]["blocked_reasons"])
|
||||
self.assertEqual(19, data["release"]["id"])
|
||||
mock_create_release.assert_not_called()
|
||||
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
def test_build_smart_release_rollout_plan_prefers_remote_ready_workers(
|
||||
self,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
) -> None:
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"region": "mainland",
|
||||
"role": "control",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 1,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": True,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": False,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
worker_plan = build_smart_release_rollout_plan("worker")
|
||||
control_plan = build_smart_release_rollout_plan("control")
|
||||
|
||||
self.assertTrue(worker_plan["available"])
|
||||
self.assertEqual("worker", worker_plan["mode"])
|
||||
self.assertEqual(["mainland-worker-01"], worker_plan["target_selector"]["node_codes"])
|
||||
self.assertEqual(["domaincheck-worker"], worker_plan["policy"]["deploy_payload"]["restart_services"])
|
||||
|
||||
self.assertTrue(control_plan["available"])
|
||||
self.assertEqual("control", control_plan["mode"])
|
||||
self.assertEqual(["mainland-controller-01"], control_plan["target_selector"]["node_codes"])
|
||||
self.assertIn("domaincheck-api", control_plan["policy"]["deploy_payload"]["restart_services"])
|
||||
self.assertEqual("remote-agent", worker_plan["policy"]["execution_mode"])
|
||||
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
def test_build_smart_release_rollout_plan_falls_back_to_ssh_ready_workers(
|
||||
self,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
) -> None:
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": False,
|
||||
"has_ssh_access": True,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": False,
|
||||
"has_ssh_access": True,
|
||||
"cluster_status": "busy",
|
||||
"cluster_current_load": 1,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": True,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
worker_plan = build_smart_release_rollout_plan("worker")
|
||||
|
||||
self.assertTrue(worker_plan["available"])
|
||||
self.assertEqual("worker", worker_plan["mode"])
|
||||
self.assertEqual("ssh", worker_plan["execution_mode"])
|
||||
self.assertEqual("SSH", worker_plan["execution_mode_label"])
|
||||
self.assertEqual(
|
||||
["mainland-worker-01", "mainland-worker-02"],
|
||||
worker_plan["target_selector"]["node_codes"],
|
||||
)
|
||||
self.assertEqual("ssh", worker_plan["policy"]["execution_mode"])
|
||||
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
def test_build_smart_release_rollout_plan_excludes_dead_letter_nodes_from_ready_candidates(
|
||||
self,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
) -> None:
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"has_ssh_access": True,
|
||||
"delivery_queue_state": "dead_letter",
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-02",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"has_ssh_access": True,
|
||||
"delivery_queue_state": "healthy",
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
worker_plan = build_smart_release_rollout_plan("worker")
|
||||
|
||||
self.assertTrue(worker_plan["available"])
|
||||
self.assertEqual(["mainland-worker-02"], worker_plan["target_selector"]["node_codes"])
|
||||
self.assertEqual("healthy", worker_plan["target_nodes"][0]["delivery_queue_state"])
|
||||
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
def test_build_smart_release_rollout_plan_returns_unavailable_when_only_dead_letter_nodes_exist(
|
||||
self,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
) -> None:
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"has_ssh_access": True,
|
||||
"delivery_queue_state": "dead_letter",
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
worker_plan = build_smart_release_rollout_plan("worker")
|
||||
|
||||
self.assertFalse(worker_plan["available"])
|
||||
self.assertIn("死信回执", worker_plan["reason"])
|
||||
self.assertEqual("dead_letter", worker_plan["target_nodes"][0]["delivery_queue_state"])
|
||||
|
||||
@patch("app.services.ops_release_service.create_release_rollout")
|
||||
@patch("app.services.ops_policy_service.preview_ops_job_policy")
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
@patch("app.services.ops_release_service.create_release_from_latest_package")
|
||||
def test_create_release_and_smart_rollout_stops_when_latest_package_gate_blocks(
|
||||
self,
|
||||
mock_create_release_from_latest_package,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
mock_preview_ops_job_policy,
|
||||
mock_create_release_rollout,
|
||||
) -> None:
|
||||
mock_create_release_from_latest_package.return_value = (
|
||||
False,
|
||||
"最新发布包尚未完成最终签收,暂不允许创建 Release 或发起基于最新包的智能 Rollout。",
|
||||
{
|
||||
"package": {
|
||||
"available": True,
|
||||
"package_name": "domaincheck_release_20260418_blocked",
|
||||
"final_release_ok": False,
|
||||
},
|
||||
"final_release_gate": {
|
||||
"ready": False,
|
||||
"status": "blocked",
|
||||
"blocked_reasons": ["final_release_report_missing"],
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
ok, message, data = create_release_and_smart_rollout_from_latest_package(
|
||||
{"mode": "worker"},
|
||||
control_plane_base_url="https://ops.example.com",
|
||||
)
|
||||
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("最终签收", message)
|
||||
self.assertFalse(data["final_release_gate"]["ready"])
|
||||
mock_list_managed_nodes_with_agent_state.assert_not_called()
|
||||
mock_preview_ops_job_policy.assert_not_called()
|
||||
mock_create_release_rollout.assert_not_called()
|
||||
|
||||
@patch("app.services.ops_release_service.create_release_rollout")
|
||||
@patch("app.services.ops_policy_service.preview_ops_job_policy")
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
@patch("app.services.ops_release_service.create_release_from_latest_package")
|
||||
def test_create_release_and_smart_rollout_requires_confirmation_before_rollout(
|
||||
self,
|
||||
mock_create_release_from_latest_package,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
mock_preview_ops_job_policy,
|
||||
mock_create_release_rollout,
|
||||
) -> None:
|
||||
mock_create_release_from_latest_package.return_value = (
|
||||
True,
|
||||
"Release 已创建",
|
||||
{
|
||||
"release": {"id": 21, "release_version": "domaincheck_release_20260418_120000"},
|
||||
"deduplicated": False,
|
||||
},
|
||||
)
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"region": "mainland",
|
||||
"role": "control",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_preview_ops_job_policy.return_value = {
|
||||
"blocked": False,
|
||||
"approval_required": True,
|
||||
"warnings": ["本次 rollout 覆盖 control 节点,建议审批。"],
|
||||
"release_gate": {"status_label": "待补巡检", "summary": "建议先确认控制面健康。"},
|
||||
}
|
||||
|
||||
ok, message, data = create_release_and_smart_rollout_from_latest_package(
|
||||
{"mode": "control"},
|
||||
control_plane_base_url="https://ops.example.com",
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("Release 已创建,智能 Rollout 需要确认", message)
|
||||
self.assertTrue(data["requires_confirmation"])
|
||||
self.assertFalse(data["rollout_created"])
|
||||
self.assertEqual(["mainland-controller-01"], data["smart_rollout"]["target_selector"]["node_codes"])
|
||||
mock_create_release_rollout.assert_not_called()
|
||||
|
||||
@patch("app.services.ops_release_service.create_release_rollout")
|
||||
@patch("app.services.ops_policy_service.preview_ops_job_policy")
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
@patch("app.services.ops_release_service.create_release_from_latest_package")
|
||||
def test_create_release_and_smart_rollout_creates_rollout_after_confirmation(
|
||||
self,
|
||||
mock_create_release_from_latest_package,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
mock_preview_ops_job_policy,
|
||||
mock_create_release_rollout,
|
||||
) -> None:
|
||||
mock_create_release_from_latest_package.return_value = (
|
||||
True,
|
||||
"Release 已创建",
|
||||
{
|
||||
"release": {"id": 22, "release_version": "domaincheck_release_20260418_120000"},
|
||||
"deduplicated": True,
|
||||
},
|
||||
)
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_preview_ops_job_policy.return_value = {
|
||||
"blocked": False,
|
||||
"approval_required": False,
|
||||
"warnings": [],
|
||||
"release_gate": {"status_label": "可发 Rollout", "summary": "当前可推进。"},
|
||||
}
|
||||
mock_create_release_rollout.return_value = (
|
||||
True,
|
||||
"Release rollout 已创建",
|
||||
{
|
||||
"rollout": {"id": 5, "release_id": 22, "rollout_code": "rollout-20260418-demo"},
|
||||
"jobs": [],
|
||||
},
|
||||
)
|
||||
|
||||
ok, message, data = create_release_and_smart_rollout_from_latest_package(
|
||||
{"mode": "worker", "confirm_risky": True},
|
||||
control_plane_base_url="https://ops.example.com",
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("Release rollout 已创建", message)
|
||||
self.assertTrue(data["rollout_created"])
|
||||
self.assertEqual(5, data["rollout"]["id"])
|
||||
mock_create_release_rollout.assert_called_once()
|
||||
rollout_payload = mock_create_release_rollout.call_args.args[1]
|
||||
self.assertEqual(["mainland-worker-01"], rollout_payload["target_selector"]["node_codes"])
|
||||
self.assertEqual("remote-agent", rollout_payload["policy"]["execution_mode"])
|
||||
|
||||
@patch("app.services.ops_release_service.create_release_rollout")
|
||||
@patch("app.services.ops_policy_service.preview_ops_job_policy")
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
@patch("app.services.ops_release_service.create_release_from_latest_package")
|
||||
def test_create_release_and_smart_rollout_uses_ssh_when_only_ssh_path_is_ready(
|
||||
self,
|
||||
mock_create_release_from_latest_package,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
mock_preview_ops_job_policy,
|
||||
mock_create_release_rollout,
|
||||
) -> None:
|
||||
mock_create_release_from_latest_package.return_value = (
|
||||
True,
|
||||
"Release 已创建",
|
||||
{
|
||||
"release": {"id": 23, "release_version": "domaincheck_release_20260418_ssh"},
|
||||
"deduplicated": True,
|
||||
},
|
||||
)
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": False,
|
||||
"has_ssh_access": True,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_preview_ops_job_policy.return_value = {
|
||||
"blocked": False,
|
||||
"approval_required": False,
|
||||
"warnings": [],
|
||||
"release_gate": {"status_label": "可发 Rollout", "summary": "当前可推进。"},
|
||||
}
|
||||
mock_create_release_rollout.return_value = (
|
||||
True,
|
||||
"Release rollout 已创建",
|
||||
{
|
||||
"rollout": {"id": 6, "release_id": 23, "rollout_code": "rollout-20260418-ssh"},
|
||||
"jobs": [],
|
||||
},
|
||||
)
|
||||
|
||||
ok, message, data = create_release_and_smart_rollout_from_latest_package(
|
||||
{"mode": "worker", "confirm_risky": True},
|
||||
control_plane_base_url="https://ops.example.com",
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("Release rollout 已创建", message)
|
||||
self.assertTrue(data["rollout_created"])
|
||||
rollout_payload = mock_create_release_rollout.call_args.args[1]
|
||||
self.assertEqual("ssh", rollout_payload["policy"]["execution_mode"])
|
||||
|
||||
@patch("app.services.ops_release_service.get_release")
|
||||
@patch("app.services.ops_release_service.create_release_rollout")
|
||||
@patch("app.services.ops_policy_service.preview_ops_job_policy")
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
def test_create_smart_release_rollout_for_existing_release_requires_confirmation(
|
||||
self,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
mock_preview_ops_job_policy,
|
||||
mock_create_release_rollout,
|
||||
mock_get_release,
|
||||
) -> None:
|
||||
mock_get_release.return_value = {
|
||||
"id": 41,
|
||||
"release_version": "domaincheck_release_existing",
|
||||
"channel": "stable",
|
||||
}
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"region": "mainland",
|
||||
"role": "control",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_preview_ops_job_policy.return_value = {
|
||||
"blocked": False,
|
||||
"approval_required": True,
|
||||
"warnings": ["控制面节点建议审批。"],
|
||||
"release_gate": {"status_label": "待审批", "summary": "需要审批。"},
|
||||
}
|
||||
|
||||
ok, message, data = create_smart_release_rollout(41, {"mode": "control"})
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("智能 Rollout 需要确认", message)
|
||||
self.assertFalse(data["release_created"])
|
||||
self.assertTrue(data["requires_confirmation"])
|
||||
self.assertFalse(data["rollout_created"])
|
||||
self.assertEqual(["mainland-controller-01"], data["smart_rollout"]["target_selector"]["node_codes"])
|
||||
mock_create_release_rollout.assert_not_called()
|
||||
|
||||
@patch("app.services.ops_release_service.get_release")
|
||||
@patch("app.services.ops_release_service.create_release_rollout")
|
||||
@patch("app.services.ops_policy_service.preview_ops_job_policy")
|
||||
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
|
||||
def test_create_smart_release_rollout_for_existing_release_creates_rollout(
|
||||
self,
|
||||
mock_list_managed_nodes_with_agent_state,
|
||||
mock_preview_ops_job_policy,
|
||||
mock_create_release_rollout,
|
||||
mock_get_release,
|
||||
) -> None:
|
||||
mock_get_release.return_value = {
|
||||
"id": 42,
|
||||
"release_version": "domaincheck_release_existing",
|
||||
"channel": "stable",
|
||||
}
|
||||
mock_list_managed_nodes_with_agent_state.return_value = {
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"region": "mainland",
|
||||
"role": "worker",
|
||||
"is_enabled": True,
|
||||
"is_managed": True,
|
||||
"is_agent_online": True,
|
||||
"cluster_status": "online",
|
||||
"cluster_current_load": 0,
|
||||
"cluster_is_effective_worker": True,
|
||||
"cluster_detect_participating": False,
|
||||
}
|
||||
]
|
||||
}
|
||||
mock_preview_ops_job_policy.return_value = {
|
||||
"blocked": False,
|
||||
"approval_required": False,
|
||||
"warnings": [],
|
||||
"release_gate": {"status_label": "可发 Rollout", "summary": "当前可推进。"},
|
||||
}
|
||||
mock_create_release_rollout.return_value = (
|
||||
True,
|
||||
"Release rollout 已创建",
|
||||
{
|
||||
"rollout": {"id": 8, "release_id": 42, "rollout_code": "rollout-20260418-existing"},
|
||||
"jobs": [],
|
||||
},
|
||||
)
|
||||
|
||||
ok, message, data = create_smart_release_rollout(42, {"mode": "worker", "confirm_risky": True})
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("Release rollout 已创建", message)
|
||||
self.assertFalse(data["release_created"])
|
||||
self.assertTrue(data["rollout_created"])
|
||||
self.assertEqual(8, data["rollout"]["id"])
|
||||
mock_create_release_rollout.assert_called_once()
|
||||
rollout_payload = mock_create_release_rollout.call_args.args[1]
|
||||
self.assertEqual(["mainland-worker-01"], rollout_payload["target_selector"]["node_codes"])
|
||||
self.assertEqual("remote-agent", rollout_payload["policy"]["execution_mode"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
118
domain-api/tests/test_ops_runtime_executor_service.py
Normal file
118
domain-api/tests/test_ops_runtime_executor_service.py
Normal file
@@ -0,0 +1,118 @@
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.services.ops_runtime_executor_service import execute_local_support_action, execute_ssh_action
|
||||
|
||||
|
||||
class OpsRuntimeExecutorServiceTests(unittest.TestCase):
|
||||
@patch("app.services.ops_runtime_executor_service.subprocess.run")
|
||||
def test_execute_ssh_action_parses_structured_json_result(self, mock_run) -> None:
|
||||
mock_run.return_value = subprocess.CompletedProcess(
|
||||
args=["ssh"],
|
||||
returncode=0,
|
||||
stdout='login banner\n{"ok": true, "message": "health snapshot collected", "result": {"checks": {"api": {"state": "active"}}}}\n',
|
||||
stderr="",
|
||||
)
|
||||
|
||||
ok, message, data = execute_ssh_action(
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"ssh_host": "121.204.244.248",
|
||||
"ssh_port": 22,
|
||||
"ssh_user": "root",
|
||||
},
|
||||
"health.snapshot",
|
||||
{},
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("health snapshot collected", message)
|
||||
self.assertEqual("active", data["checks"]["api"]["state"])
|
||||
self.assertEqual("ssh", data["transport"]["executor"])
|
||||
self.assertEqual("121.204.244.248", data["transport"]["ssh_host"])
|
||||
issued_command = mock_run.call_args.args[0]
|
||||
self.assertEqual("ssh", issued_command[0])
|
||||
self.assertIn("root@121.204.244.248", issued_command)
|
||||
|
||||
@patch("app.services.ops_runtime_executor_service.subprocess.run")
|
||||
def test_execute_local_support_action_collects_logs_with_payload(self, mock_run) -> None:
|
||||
mock_run.return_value = subprocess.CompletedProcess(
|
||||
args=["journalctl"],
|
||||
returncode=0,
|
||||
stdout="log line 1\nlog line 2",
|
||||
stderr="",
|
||||
)
|
||||
|
||||
ok, message, data = execute_local_support_action(
|
||||
"logs.collect",
|
||||
{
|
||||
"service_name": "domaincheck-worker",
|
||||
"lines": 180,
|
||||
},
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("log line 1\nlog line 2", message)
|
||||
self.assertEqual("domaincheck-worker", data["service_name"])
|
||||
self.assertEqual(180, data["lines"])
|
||||
issued_command = mock_run.call_args.args[0]
|
||||
self.assertEqual(
|
||||
["journalctl", "-u", "domaincheck-worker", "-n", "180", "--no-pager"],
|
||||
issued_command,
|
||||
)
|
||||
|
||||
@patch("app.services.ops_runtime_executor_service.subprocess.run")
|
||||
def test_execute_local_support_action_falls_back_to_worker_log_file(self, mock_run) -> None:
|
||||
mock_run.return_value = subprocess.CompletedProcess(
|
||||
args=["journalctl"],
|
||||
returncode=1,
|
||||
stdout="",
|
||||
stderr="No journal files were opened due to insufficient permissions.",
|
||||
)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
log_path = Path(temp_dir) / "detect_worker.log"
|
||||
log_path.write_text("fallback-1\nfallback-2\n", encoding="utf-8")
|
||||
with patch("app.services.ops_action_executor_core.settings.domain_root", temp_dir):
|
||||
ok, message, data = execute_local_support_action(
|
||||
"logs.collect",
|
||||
{
|
||||
"service_name": "domaincheck-worker",
|
||||
"lines": 180,
|
||||
},
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("fallback-1\nfallback-2", message)
|
||||
self.assertEqual("file_fallback", data["collection_source"])
|
||||
self.assertTrue(data["fallback_used"])
|
||||
|
||||
@patch("app.services.ops_runtime_executor_service.subprocess.run")
|
||||
def test_execute_local_support_action_restarts_service(self, mock_run) -> None:
|
||||
mock_run.return_value = subprocess.CompletedProcess(
|
||||
args=["systemctl"],
|
||||
returncode=0,
|
||||
stdout="",
|
||||
stderr="",
|
||||
)
|
||||
|
||||
ok, message, data = execute_local_support_action(
|
||||
"service.restart",
|
||||
{
|
||||
"service_name": "domaincheck-worker",
|
||||
},
|
||||
)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertEqual("domaincheck-worker restart completed", message)
|
||||
self.assertEqual("domaincheck-worker", data["service_name"])
|
||||
self.assertEqual("restart", data["systemctl_action"])
|
||||
issued_command = mock_run.call_args.args[0]
|
||||
self.assertEqual(["systemctl", "restart", "domaincheck-worker"], issued_command)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user