636 lines
16 KiB
Markdown
636 lines
16 KiB
Markdown
# 26 domainCheck 发布前运行验证与交付模板
|
||
|
||
## 一、这份模板怎么用
|
||
|
||
这份文档不是设计说明，而是正式发布前最后一轮执行模板。
|
||
|
||
适用场景：
|
||
|
||
- 海外单脑控制面已经搭好
|
||
- 大陆节点已经接入或正在收口
|
||
- 代码已经更新到待发布版本
|
||
- 现在要做最后一轮运行验证、证据留存、交付确认
|
||
|
||
这份模板分成 3 段：
|
||
|
||
1. 发布前运行验证
|
||
2. 上线证据导出
|
||
3. 最终交付结论模板
|
||
|
||
---
|
||
|
||
## 二、发布前运行验证
|
||
|
||
### 1. 先跑统一收口摘要
|
||
|
||
```bash
|
||
cd /www/wwwroot/getDomain
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-check http://127.0.0.1:8100 http://127.0.0.1:8100 summary
|
||
```
|
||
|
||
记录结果：
|
||
|
||
- `go_live_status`
|
||
- `publish_status`
|
||
- `blocking_reasons`
|
||
- `warnings`
|
||
- `next_step_action_code`
|
||
- `operator_title`
|
||
- `launchpad_recommended_target_node_code`
|
||
- `launchpad_recommended_recovery_label`
|
||
- `launchpad_recommended_recovery_summary`
|
||
- `launchpad_onboarding_bootstrap_pending_nodes`
|
||
- `launchpad_onboarding_acceptance_ready_nodes`
|
||
|
||
通过标准：
|
||
|
||
- `go_live_status != blocked`
|
||
- `publish_status != blocked`
|
||
|
||
### 1.5 先看环境差异是否已收口
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh env-audit
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
|
||
```
|
||
|
||
重点只看：
|
||
|
||
- `status`
|
||
- `headline`
|
||
- `python.pytest_installed / fastapi_installed / uvicorn_installed`
|
||
- `python.runtimes`
|
||
- `services`
|
||
- `runtime.preflight_ok`
|
||
- `runtime.readiness_status`
|
||
- `runtime.route_surface_complete`
|
||
- `missing_items`
|
||
- `tooling_items`
|
||
|
||
通过标准建议：
|
||
|
||
- `status != blocked`
|
||
- `runtime.preflight_ok=true`
|
||
- `route_surface_complete=true`
|
||
- 如果 `missing_items` 里仍有关键基础项，先补环境，再继续业务复核
|
||
- 如果只有 `tooling_items`，可以继续上线复核，但建议后补本机工具链
|
||
- 如果 `runtime.runtime_may_need_restart=true`
|
||
- 不要直接继续接管或发布链
|
||
- 先按 `runtime-refresh-recover` 给出的固定顺序完成 API 重启与复检
|
||
- 如果你希望把这一轮“重启 + 总检 + 默认下一步 + 上线摘要”压成一次复核
|
||
- 直接执行 `go-live-recover`
|
||
|
||
### 1.6 先看托管节点接入面是否清楚
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
|
||
```
|
||
|
||
重点只看：
|
||
|
||
- `summary.agent_ready`
|
||
- `summary.ssh_ready`
|
||
- `summary.remote_access_ready`
|
||
- `summary.remote_access_state_counts`
|
||
- 每台节点的 `agent_state`
|
||
- 每台节点的 `remote_access_state`
|
||
- 每台节点的 `ssh_access_label`
|
||
- 每台节点的 `ssh_entry`
|
||
|
||
通过标准建议：
|
||
|
||
- 如果某台节点还是 `agent_pending`
|
||
- 先确认是否已经保存 SSH 入口
|
||
- 如果节点尚未保存 SSH 入口，但你已经决定由海外控制面统一接管
|
||
- 先补录：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-bind-ssh http://127.0.0.1:8100 mainland-worker-01 121.204.244.248 root 22
|
||
```
|
||
|
||
- 补录后立即复查：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh nodes http://127.0.0.1:8100
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-handover http://127.0.0.1:8100 mainland-worker-01
|
||
```
|
||
|
||
- 如果已经 `agent_ready`
|
||
- 说明该节点已进入标准远端执行器接管面
|
||
- 如果还不是 `agent_ready`，但 `ssh_ready` 已经成立
|
||
- 说明至少已经具备海外主控经 SSH 进入节点完成 bootstrap / 验收的前置条件
|
||
|
||
### 2. 再看总检详情
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||
```
|
||
|
||
记录结果：
|
||
|
||
- `stack_status`
|
||
- `issues`
|
||
- `next_step`
|
||
- `quick_commands`
|
||
|
||
通过标准：
|
||
|
||
- `stack_status != blocked`
|
||
- `issues` 里没有未处理的关键阻断项
|
||
|
||
### 3. 再看驾驶主线
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh driver-feed
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh codex-brief
|
||
```
|
||
|
||
记录结果：
|
||
|
||
- `top_recommendation`
|
||
- `automation_coverage`
|
||
- `recommended_behavior`
|
||
- `focus`
|
||
- `summary.launchpad_recommended_target_node_code`
|
||
- `summary.launchpad_recommended_recovery_label`
|
||
- `summary.launchpad_onboarding_bootstrap_pending_nodes`
|
||
- `summary.launchpad_onboarding_acceptance_ready_nodes`
|
||
|
||
通过标准：
|
||
|
||
- `automation_coverage.launch_status != blocked`
|
||
- 默认下一步已经明确
|
||
- 如果 `summary.launchpad_recommended_target_node_code` 非空
|
||
- 值班同事应能直接知道当前卡在哪台节点
|
||
- 不需要再回 launchpad 明细手工翻 gap rows
|
||
|
||
### 4. 若涉及发布，再看 Release Hub
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||
```
|
||
|
||
记录结果：
|
||
|
||
- `launchpad_status`
|
||
- `recommended_action_code`
|
||
- `default_rollout_gate_status`
|
||
- `recommended_target_node_code`
|
||
- `recommended_recovery_label`
|
||
- `recommended_recovery_summary`
|
||
- `onboarding_bootstrap_pending_nodes`
|
||
- `onboarding_acceptance_ready_nodes`
|
||
|
||
通过标准：
|
||
|
||
- `launchpad_status != blocked`
|
||
- `default_rollout_gate_status != blocked`
|
||
|
||
特殊判读：
|
||
|
||
- 如果 `recommended_action_code=api-restart`
|
||
- 不要误判成“节点没接好”
|
||
- 这更像是运行中的控制面 API 还没刷新到当前仓库的最新运维能力
|
||
- 应先执行：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover
|
||
```
|
||
|
||
- 然后重新跑：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad
|
||
```
|
||
|
||
补充说明：
|
||
|
||
- 从 `latest_release.json` 直接创建 Release，或执行“基于最新包的一键智能 Rollout”之前
|
||
- 现在不仅要求 `smoke_test_ok=true`
|
||
- 还要求当前 `final_release_report.json` 与最新包完全匹配，且 `final_release_ok=true`
|
||
- 如果页面按钮被禁用，或 API 返回“最新发布包尚未完成最终签收”
|
||
- 优先检查 `final_release_report_stale_reason`
|
||
- 再检查 `final_release_gate_decision / final_release_gate_blocked_reasons`
|
||
|
||
### 5. 若涉及节点接管，补跑节点验收
|
||
|
||
如果这次发布包含：
|
||
|
||
- 新增大陆节点
|
||
- 重新接管旧节点
|
||
- 切换到 Node Agent 主接管模式
|
||
|
||
则不能只看总检摘要，还要对目标节点逐台完成验收。
|
||
|
||
先看验收计划：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-plan http://127.0.0.1:8100 mainland-worker-01
|
||
```
|
||
|
||
确认无误后执行：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-worker-01 cli/acceptance
|
||
```
|
||
|
||
记录结果：
|
||
|
||
- `node_code`
|
||
- `status`
|
||
- `checks`
|
||
- `blocking_issues`
|
||
- `warnings`
|
||
- `recommended_next_step`
|
||
|
||
通过标准：
|
||
|
||
- `status != blocked`
|
||
- 没有未处理的 `blocking_issues`
|
||
- 目标节点已经不再停留在“bootstrap pending / acceptance pending”
|
||
|
||
### 6. 若涉及远端执行，补跑日志回传验证
|
||
|
||
如果当前版本准备进入：
|
||
|
||
- 多机值班
|
||
- 海外单脑统一接管
|
||
- 远端动作回执与现场日志复核
|
||
|
||
则需要确认日志回传链已经可用，而不是只看节点在线。
|
||
|
||
先检查：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-check
|
||
```
|
||
|
||
如果存在缺口，再执行恢复：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh log-sync-recover http://127.0.0.1:8100 key confirm cli/log-sync
|
||
```
|
||
|
||
必要时抓一份节点现场日志：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 full
|
||
```
|
||
|
||
记录结果：
|
||
|
||
- `log_sync_enabled`
|
||
- `line_count`
|
||
- `source_nodes`
|
||
- `missing_nodes`
|
||
- `scene_log.status`
|
||
|
||
通过标准：
|
||
|
||
- 关键节点已出现在 `source_nodes`
|
||
- `missing_nodes` 不包含当前发布主车道节点
|
||
- 如需人工复核现场，`scene-node-log` 能取到有效日志
|
||
|
||
### 7. 若涉及 Node Agent 交付，补看队列健康
|
||
|
||
如果这轮发布已经进入“控制面发动作、节点拉任务、节点回执结果”的模式，就必须确认 delivery queue 没有假健康。
|
||
|
||
先看队列摘要：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-status http://127.0.0.1:8100 mainland-worker-01
|
||
```
|
||
|
||
如果怀疑有堆积或死信，再看明细：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-records http://127.0.0.1:8100 mainland-worker-01 dead_letter
|
||
```
|
||
|
||
如需重放：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh queue-replay http://127.0.0.1:8100 mainland-worker-01 20 cli/queue-replay
|
||
```
|
||
|
||
记录结果：
|
||
|
||
- `delivery_queue.summary`
|
||
- `dead_letter`
|
||
- `retrying`
|
||
- `pending`
|
||
- `replayed_count`
|
||
|
||
通过标准：
|
||
|
||
- `dead_letter=0`
|
||
- 没有持续增长的 `retrying`
|
||
- 队列不处于异常堆积状态
|
||
|
||
注意：
|
||
|
||
- `dead_letter > 0` 时，不应宣称“可正式发布”
|
||
- 这种情况至少按 `publish_status = blocked` 处理
|
||
|
||
### 8. 最后跑一次 doctor 结论
|
||
|
||
前面的检查是分面复核，最后还要再收成一句话，让交班、值班、Codex 驾驶员看到同一结论。
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-decision
|
||
```
|
||
|
||
记录结果：
|
||
|
||
- `status`
|
||
- `decision`
|
||
- `recommended_commands`
|
||
- `recommended_reading_order`
|
||
- `launchpad_alignment`
|
||
|
||
通过标准：
|
||
|
||
- `status != blocked`
|
||
- `decision` 与前面的 `go-live-check / stack-diagnosis / release-launchpad` 不冲突
|
||
- `recommended_commands` 已经指向明确的下一跳，而不是泛化排查
|
||
|
||
### 9. 若希望直接生成“可否签字上线”的最终摘要
|
||
|
||
如果你不想手工再拼：
|
||
|
||
- bundle review
|
||
- doctor 结论
|
||
- 发布门禁
|
||
|
||
可以直接执行：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff
|
||
```
|
||
|
||
如果已经有现成证据包，也可以直接基于 manifest 输出：
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-signoff /path/to/go-live-bundle
|
||
```
|
||
|
||
这一步现在不是“另一套单独判断”，而是固定复用：
|
||
|
||
- `go-live-review`
|
||
- `doctor-decision`
|
||
- `go-live-summary / 发布门禁 / launchpad 摘要`
|
||
|
||
也就是说，页面首屏、CLI 和海外 Codex 看到的最终签收结论，已经开始共享同一份签字口径。
|
||
|
||
重点只看：
|
||
|
||
- `signoff_status`
|
||
- `headline`
|
||
- `release_gate`
|
||
- `blocked_reasons`
|
||
- `attention_reasons`
|
||
- `decision`
|
||
- `recommended_commands`
|
||
|
||
通过标准：
|
||
|
||
- `signoff_status=ready`
|
||
- 可进入最终人工签字或正式发布
|
||
- `signoff_status=attention`
|
||
- 说明现场已经接近完成，但仍应先清 attention 项
|
||
- `signoff_status=blocked`
|
||
- 本轮不应宣称收口完成
|
||
|
||
---
|
||
|
||
## 三、上线证据导出
|
||
|
||
### 1. 导出 bundle
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-export
|
||
```
|
||
|
||
这会生成一整包上线前证据。
|
||
|
||
建议至少保留：
|
||
|
||
- `00_env_audit.txt`
|
||
- `01_go_live_check_summary.txt`
|
||
- `02_go_live_summary.json`
|
||
- `03_stack_diagnosis.json`
|
||
- `04_driver_feed.json`
|
||
- `05_codex_brief.json`
|
||
- `06_release_launchpad.json`
|
||
- `manifest.json`
|
||
|
||
如果当前轮次涉及节点接管，建议额外保留：
|
||
|
||
- `agent_gap_export.json`
|
||
- `node_bootstrap_plan.txt`
|
||
- `node_acceptance_plan.json`
|
||
|
||
如果当前轮次涉及远端执行与日志回传，建议额外保留：
|
||
|
||
- `scene_log_*.txt`
|
||
- `doctor_decision.json`
|
||
|
||
### 2. 直接复核 bundle
|
||
|
||
```bash
|
||
bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-review
|
||
```
|
||
|
||
如果你已经切到 Ops Center 页面，也可以直接看：
|
||
|
||
- 首屏 `总检决策`
|
||
- `总检主决策详情`
|
||
- API `GET /api/v1/ops/doctor-decision`
|
||
- 首屏 `交付证据`
|
||
- `交付证据详情`
|
||
- API `GET /api/v1/ops/go-live-bundle`
|
||
- 首屏 `正式复核`
|
||
- `正式复核详情`
|
||
- API `GET /api/v1/ops/go-live-review`
|
||
|
||
记录结果：
|
||
|
||
- `status`
|
||
- `headline`
|
||
- `critical_failures`
|
||
- `noncritical_failures`
|
||
- `launchpad_alignment.consistent`
|
||
- `env_audit.status`
|
||
- `env_audit.missing_items`
|
||
- `recommended_next_steps`
|
||
|
||
通过标准建议：
|
||
|
||
- `status=ready`
|
||
- 可进入最终人工确认或正式发布
|
||
- `status=attention`
|
||
- 核心报告已齐，但仍需复核补充失败项或环境缺口
|
||
- 也包括 `go_live_summary / stack_diagnosis / driver_feed / codex_brief` 的 launchpad 摘要出现不一致
|
||
- `status=blocked`
|
||
- 关键报告缺失，或环境审计已经判定阻断，不应直接宣称收口完成
|
||
|
||
如果 `env_audit.missing_items` 里出现：
|
||
|
||
- `missing_env:/etc/default/domaincheck-node-agent`
|
||
|
||
则这次 bundle review 已经不是泛泛地提示“环境缺口”，而是明确指向 Node Agent 接管还未收口。此时按下面顺序推进：
|
||
|
||
1. `agent-gap-export`
|
||
2. `node-bootstrap-plan`
|
||
3. 如果 summary 已提示 `runtime_may_need_restart=true`
|
||
4. 完成恢复后重新导出 bundle，再做一次 `go-live-review`
|
||
|
||
### 3. 生成最终交付包
|
||
|
||
运行验证、bundle 复核都通过后，再生成最终交付包。
|
||
|
||
如果当前机器已经具备 smoke 运行条件，直接执行：
|
||
|
||
```bash
|
||
bash ./prepare_final_release.sh
|
||
```
|
||
|
||
如果当前机器只适合先做离线打包，可先跳过 smoke：
|
||
|
||
```bash
|
||
DOMAINCHECK_SKIP_SMOKE_TEST=1 bash ./package_domain_release.sh
|
||
bash ./verify_domain_release.sh
|
||
```
|
||
|
||
后续再回到具备运行条件的机器上执行：
|
||
|
||
```bash
|
||
bash ./prepare_final_release.sh
|
||
```
|
||
|
||
最后统一查看最新交付结果：
|
||
|
||
```bash
|
||
bash ./show_latest_release.sh
|
||
```
|
||
|
||
重点只看：
|
||
|
||
- `latest_release.package_name`
|
||
- `latest_release.archive_path`
|
||
- `latest_release.sha256`
|
||
- `latest_release.smoke_test_ok`
|
||
- `final_release_ok`
|
||
- `final_release_report`
|
||
- `final_release_report_ok`
|
||
- `final_release_report_matches_latest`
|
||
- `final_release_report_stale_reason`
|
||
- `final_release_gate_decision`
|
||
- `final_release_gate_blocked_reasons`
|
||
- `final_release_gate_verify_ok`
|
||
- `final_release_gate_smoke_test_ok`
|
||
|
||
通过标准：
|
||
|
||
- `verify_domain_release.sh` 返回 `ok=true`
|
||
- `latest_release.smoke_test_ok=true`
|
||
- `final_release_ok=true`
|
||
- `final_release_report_matches_latest=true`
|
||
- `final_release_gate_decision=ready`
|
||
- `final_release_gate_verify_ok=true`
|
||
- `final_release_gate_smoke_test_ok=true`
|
||
- `final_release_report` 已指向最新一轮生成的正式报告
|
||
4. 先执行 `runtime-refresh-recover`
|
||
5. 再重新导出 bootstrap plan
|
||
|
||
---
|
||
|
||
## 四、最终交付结论模板
|
||
|
||
下面这段可以直接作为正式发布前的交付结论模板使用。
|
||
|
||
### 模板正文
|
||
|
||
```text
|
||
本轮发布前运行验证已完成。
|
||
|
||
一、统一收口结论
|
||
- go_live_status: <填写>
|
||
- publish_status: <填写>
|
||
- stack_status: <填写>
|
||
- automation_coverage.launch_status: <填写>
|
||
- launchpad_status: <填写>
|
||
|
||
二、默认下一步
|
||
- next_step_action_code: <填写>
|
||
- operator_title: <填写>
|
||
- launchpad_recommended_target_node_code: <填写>
|
||
- launchpad_recommended_recovery_label: <填写>
|
||
|
||
三、上线证据包
|
||
- bundle 路径: <填写>
|
||
- manifest 复核状态: <填写 ready/attention/blocked>
|
||
- go-live-review.headline: <填写>
|
||
- manifest.review_status_hint: <填写>
|
||
- critical_failures: <填写>
|
||
- noncritical_failures: <填写>
|
||
- manifest.launchpad_recommended_target_node_code: <填写>
|
||
- manifest.launchpad_recommended_recovery_label: <填写>
|
||
- manifest.launchpad_onboarding_bootstrap_pending_nodes: <填写>
|
||
- manifest.launchpad_onboarding_acceptance_ready_nodes: <填写>
|
||
- manifest.launchpad_alignment.consistent: <填写 true/false>
|
||
|
||
四、结论
|
||
- <填写：可进入正式发布 / 仍需继续收口 / 暂不可发布>
|
||
```
|
||
|
||
### 模板补充说明
|
||
|
||
填写“可进入正式发布”前，建议再做一次人工交叉确认：
|
||
|
||
- 发布门禁是否为 `ready` 或至少非 `blocked`
|
||
- 当前主车道是否已经明确
|
||
- 是否还存在 `dead_letter`
|
||
- 是否还有节点停留在 `bootstrap pending / acceptance pending`
|
||
- 是否已经保留可复盘的证据包与 manifest
|
||
|
||
---
|
||
|
||
## 五、交付给值班同事时最少要带什么
|
||
|
||
最少带下面 4 样：
|
||
|
||
1. `go-live-check` 摘要
|
||
2. `stack-diagnosis` 结果
|
||
3. `driver-feed / codex-brief` 结果
|
||
4. `go-live-export` 生成的 `manifest.json`
|
||
|
||
如果值班同事需要直接接手节点问题，再多带 3 样：
|
||
|
||
1. `node-acceptance-run` 最近结果
|
||
2. `log-sync-check` 最近结果
|
||
3. `queue-status` 或 `queue-records dead_letter` 最近结果
|
||
|
||
这样交班的人不需要重新从零拼现场。
|
||
|
||
---
|
||
|
||
## 六、真正的收工标准
|
||
|
||
如果要算“这次版本已经可以收工”，建议至少同时满足：
|
||
|
||
- 前端构建成功
|
||
- 海外单脑驾驶舱首屏状态正常
|
||
- `go-live-review.status != blocked`
|
||
- 发布门禁不为 `blocked`
|
||
- Node Agent 目标节点已完成接管验收
|
||
- 远端日志回传链可用或已确认本轮不依赖
|
||
- delivery queue 不存在未处理 `dead_letter`
|
||
- 当前默认下一步已经不是“继续补基础设施”，而是进入正式发布 / 观察 / 验收车道
|
||
|
||
到这一步，才适合说：
|
||
|
||
> 当前项目已经进入可上线、可交付、可持续运维状态。
|