docs: move ops runtime tracking under docs
This commit is contained in:
135
docs/ops_center_runtime/HANDOFF_20260419_0050.md
Normal file
135
docs/ops_center_runtime/HANDOFF_20260419_0050.md
Normal file
@@ -0,0 +1,135 @@
|
||||
# HANDOFF 2026-04-19 00:50 CST
|
||||
|
||||
## 本轮目标
|
||||
|
||||
只处理 acceptance 队列不消费问题,不扩功能,不新增页面或专题文档。
|
||||
|
||||
## 本轮完成
|
||||
|
||||
### 1. 定位并修复 acceptance 队列卡死
|
||||
|
||||
定位结果:
|
||||
|
||||
- 两台大陆节点都在持续请求 `/api/v1/ops/agent/pull?limit=1`
|
||||
- 控制面收到请求,但 acceptance job 长时间停留在 `queued`
|
||||
- 数据库中存在大量 `idle in transaction`
|
||||
- 事务卡在:
|
||||
- `UPDATE ops_job_steps ...`
|
||||
- 随后又在新连接里 `INSERT INTO ops_job_events ...`
|
||||
|
||||
代码修复:
|
||||
|
||||
- 文件:
|
||||
- [domain-api/app/services/ops_agent_service.py](/www/wwwroot/getDomain/domain-api/app/services/ops_agent_service.py)
|
||||
- 修改点:
|
||||
- `agent_pull_jobs`
|
||||
- `agent_mark_job_started`
|
||||
- `agent_complete_job`
|
||||
- 修复方式:
|
||||
- 把 `append_ops_job_event(...)` 从事务内部移到 `conn.commit()` 之后执行
|
||||
|
||||
### 2. 补充最小回归测试
|
||||
|
||||
- 文件:
|
||||
- [domain-api/tests/test_ops_agent_service.py](/www/wwwroot/getDomain/domain-api/tests/test_ops_agent_service.py)
|
||||
- 新增测试:
|
||||
- `test_agent_pull_jobs_commits_before_appending_event`
|
||||
- `test_agent_mark_job_started_commits_before_appending_event`
|
||||
- `test_agent_complete_job_commits_before_appending_event`
|
||||
- 验证结果:
|
||||
- `cd /www/wwwroot/getDomain/domain-api && /opt/domaincheck/domainCheck/.venv/bin/python -m unittest tests.test_ops_agent_service -q`
|
||||
- `Ran 27 tests ... OK`
|
||||
|
||||
### 3. 运行态验证
|
||||
|
||||
- `domaincheck-api` 已完成强制切换到新版本进程
|
||||
- 新进程:
|
||||
- `MainPID = 614403`
|
||||
- 运行态日志已确认:
|
||||
- `job 36 / 41` 开始执行
|
||||
- 已出现 `start / events / complete`
|
||||
|
||||
## 当前 acceptance 结果
|
||||
|
||||
### `mainland-worker-01`
|
||||
|
||||
- run:
|
||||
- `pbr-137e8b258b`
|
||||
- 当前状态:
|
||||
- `attention`
|
||||
- 结果:
|
||||
- `4 success + 1 failed`
|
||||
- 唯一失败步骤:
|
||||
- `node_agent_logs`
|
||||
- 失败原因:
|
||||
- `No journal files were opened due to insufficient permissions`
|
||||
|
||||
### `mainland-controller-01`
|
||||
|
||||
- run:
|
||||
- `pbr-bfed43ea46`
|
||||
- 当前状态:
|
||||
- `attention`
|
||||
- 结果:
|
||||
- `4 success + 1 failed`
|
||||
- 唯一失败步骤:
|
||||
- `node_agent_logs`
|
||||
- 失败原因:
|
||||
- `No journal files were opened due to insufficient permissions`
|
||||
|
||||
## 当前总检状态
|
||||
|
||||
`go-live-summary`:
|
||||
|
||||
- `go_live_status = attention`
|
||||
- `publish_status = attention`
|
||||
- `stack_status = attention`
|
||||
- `launchpad_status = attention`
|
||||
- `log_sync_state = full_capture`
|
||||
|
||||
`stack-diagnosis`:
|
||||
|
||||
- `surface_status = healthy`
|
||||
- `automation_status = attention`
|
||||
- `stack_status = attention`
|
||||
- `issue_total = 1`
|
||||
- `blocking_issue_total = 0`
|
||||
|
||||
## 当前最高优先级
|
||||
|
||||
唯一最高优先级:
|
||||
|
||||
- 修复远端 Node Agent 读取 `journalctl` 的权限
|
||||
|
||||
## 当前是否可上线签收
|
||||
|
||||
结论:
|
||||
|
||||
- 还不可上线签收
|
||||
|
||||
唯一剩余阻塞:
|
||||
|
||||
- 两条 acceptance run 都因为 `node_agent_logs` 权限不足而未全绿
|
||||
|
||||
## 下一轮若继续,必须先做什么
|
||||
|
||||
只做这一件事:
|
||||
|
||||
- 处理 `domaincheck-node-agent` 所在服务账号的 journal 读取权限
|
||||
|
||||
推荐最小方向:
|
||||
|
||||
- 让运行 Node Agent 的账号具备读取 systemd journal 的权限
|
||||
- 然后重跑 acceptance,或只补采失败日志步骤
|
||||
|
||||
## 现在不要做的事情
|
||||
|
||||
- 不要回退这次热修复
|
||||
- 不要重新 bootstrap
|
||||
- 不要重装大陆机器
|
||||
- 不要扩控制面功能
|
||||
- 不要进入正式发布
|
||||
|
||||
## 本轮一句话结论
|
||||
|
||||
接管主链已经打通,acceptance 队列事务 bug 已修复;当前只剩远端 `journalctl` 权限问题,解决后即可继续冲刺上线签收。
|
||||
Reference in New Issue
Block a user