d
This commit is contained in:
Your Name
287
docs/base.md
Normal file
287
docs/base.md
Normal file
@@ -0,0 +1,287 @@
|
||||
# 主线任务验收总表
|
||||
|
||||
更新时间：2026-04-21
|
||||
|
||||
## 当前执行原则
|
||||
|
||||
当前先只做一件事：
|
||||
|
||||
- 跑通主任务流程
|
||||
- 把 7 个大任务按“可测试、可验收”的标准推进
|
||||
- 并发、代理、展示层、CPU 打满这类细节优化先封存，不再抢主线
|
||||
|
||||
当前不再把“性能抠点”当主目标。
|
||||
后续额度恢复后，再继续做细节优化专项。
|
||||
|
||||
## 当前主线优先级
|
||||
|
||||
按下面顺序推进，不跳步：
|
||||
|
||||
1. 大任务 1 真实验收闭环
|
||||
2. 大任务 2 controller 编排闭环
|
||||
3. 大任务 3 统一结果状态机闭环
|
||||
4. 大任务 4 本地控制状态 + syncer/finalizer 闭环
|
||||
5. 大任务 5 固定 worker pool + 持续补位替换旧模型
|
||||
6. 大任务 6 时光机一期接入标准步骤
|
||||
7. 大任务 7 运营视角指标面板落地
|
||||
|
||||
## 7 个大任务当前状态
|
||||
|
||||
### 大任务 1
|
||||
单步骤任务底座落地
|
||||
|
||||
目标：
|
||||
先打通 `controller -> queue -> claim -> worker -> result -> controller` 的完整闭环。
|
||||
|
||||
当前状态：
|
||||
- 已有较完整代码底座
|
||||
- `single_step` / `step_code` 模型已基本成形
|
||||
- 现在缺的是“真实运行验收”而不是继续堆骨架
|
||||
|
||||
本任务验收只认下面 5 件事：
|
||||
|
||||
1. controller 能生成 `baidu_check` 任务
|
||||
2. worker 能拉到 `baidu_check` 任务并执行
|
||||
3. worker 能回传标准化结果
|
||||
4. controller 能更新本地步骤状态
|
||||
5. worker 超时未回传时，任务会回收重投
|
||||
|
||||
当前结论：
|
||||
- 代码方向已基本对齐
|
||||
- 2026-04-20 已完成一轮 live 验收打钩：
|
||||
- `controller` 已生成 `single_step / detect_baidu_site` 任务（例：`job_id=6`, `job_code=step-20260420153527-9fc7b5`）
|
||||
- `worker` 已精准领取当前 `job_id` 并执行，不再被旧 `pipeline / sync / legacy fallback` 污染
|
||||
- `worker` 已回传标准化结果到 `detect_job_items.result_payload_json`
|
||||
- `controller` 已可按 `job_id` 定向执行 `process_pipeline`，并把结果落到本地 `domain_detections.baidu_site`
|
||||
- “超时/遗留未回传”链路已验证会被节点启动释放并重新领取执行（`job_id=5` 在遗留 `running` 后被重新释放、重新领取并完成）
|
||||
- 当前主线结论：大任务 1 已达到“可验收通过”状态，可以继续把主精力切到大任务 2 / 3
|
||||
|
||||
### 大任务 2
|
||||
pipeline 编排器落地
|
||||
|
||||
目标：
|
||||
把“下一步跑什么”彻底收回 controller。
|
||||
|
||||
当前状态：
|
||||
- 基础方向已落到 controller 驱动
|
||||
- 还需要继续做“按后台勾选顺序推进 / 按域名属性跳过”的真实验收
|
||||
|
||||
本任务必须确认：
|
||||
- 同一个域名不会在 worker 里串完整条链
|
||||
- controller 是唯一推进者
|
||||
- 勾选顺序变化能影响下一步投递
|
||||
- 跳过规则生效
|
||||
|
||||
当前结论：
|
||||
- 已进入可验收阶段
|
||||
- 2026-04-20 已补上关键收口：
|
||||
- `single_step` 会话下，worker 只按当前 `job_id` 领取任务
|
||||
- `single_step` 会话下，旧的 `pipeline 推进 / sync pull / legacy fallback` 已被显式跳过
|
||||
- `process_pipeline` 已支持按 `job_id` 定向处理，方便 controller 精准推进当前步骤
|
||||
- 2026-04-20 又补完一轮 live 验收：
|
||||
- `pass`：`detect_baidu_site` 完成后，controller 已为同一 `job` 创建下一步 `detect_360_site`
|
||||
- `skip`：一口价域名在 `detect_register + detect_baidu_site` 配置下，会直接解析到 `detect_baidu_site`
|
||||
- 同时修掉了一个真实阻塞点：`process_pipeline()` 事务内再开第二连接写 `detect_run_events` 会把 controller 自己锁住；现已改成同事务同 cursor 写事件
|
||||
- 当前主线结论：大任务 2 的 controller 编排主干已可验收，剩下更多是扩步骤和补更细跳过规则
|
||||
|
||||
### 大任务 3
|
||||
步骤结果判定与重试策略落地
|
||||
|
||||
目标：
|
||||
统一 `pass / retry / black_hit / reject`。
|
||||
|
||||
当前状态：
|
||||
- 结果结构、重试入口、黑名单终止方向已基本进入主链
|
||||
- 还需要补 live 验收，重点是 TTL 回收、重投、终止规则
|
||||
|
||||
本任务必须确认：
|
||||
- 外部失败会重投当前步骤
|
||||
- 黑名单命中会终止后续步骤
|
||||
- 非黑名单业务不通过按规则终止
|
||||
- 不会出现同一步无限重试
|
||||
|
||||
当前结论：
|
||||
- 代码已明显推进
|
||||
- 2026-04-20 已完成 controller 侧 live 验收：
|
||||
- `retry`：`state=degraded` 会重投当前步骤，不推进下一步
|
||||
- `black_hit`：`state=blacklisted` 会终止后续步骤并结束当前 job
|
||||
- `reject`：`state=rejected` 会终止当前流程，不再重试
|
||||
- worker 侧也已补上业务失败 -> `rejected` 的结果态，不再把业务不通过和技术失败都混成 `failed`
|
||||
- 当前主线结论：大任务 3 的统一结果状态机已基本闭环，可继续往大任务 4 / 5 推进
|
||||
|
||||
### 大任务 4
|
||||
本地控制状态与海外主库同步落地
|
||||
|
||||
目标：
|
||||
controller 本地维护高频状态，syncer 批量同步海外主库，finalizer 标记流程完成。
|
||||
|
||||
当前状态：
|
||||
- 本地状态和部分同步链路已经在跑
|
||||
- 已补上“最近完成任务快照也继续产出结果投影”
|
||||
- 已补上结果导入后按 job_code 优先定位本地 job，并刷新本地 job 收尾状态
|
||||
- syncer/finalizer 还缺少一轮 live 闭环验收
|
||||
|
||||
本任务必须确认：
|
||||
- 高频链路不依赖 worker 频繁直写海外主库
|
||||
- controller 本地状态完整
|
||||
- 批量同步成功
|
||||
- 流程完成状态准确
|
||||
|
||||
当前结论：
|
||||
- 代码主链已进一步收口
|
||||
- 2026-04-21 已确认 `detect_result_projection / runtime_projection` 在 `detect_sync_records` 中持续产出且状态为 `projected`
|
||||
- 当前说明 syncer 主链已恢复，但“本地 job 收尾状态 + 主库最终账本完全一致”的终验还需要继续盯现场
|
||||
|
||||
### 大任务 5
|
||||
worker 池化与持续补位调度落地
|
||||
|
||||
目标：
|
||||
真正替掉“批量认领 + 批量等待”的旧模型。
|
||||
|
||||
当前状态：
|
||||
- 已经做了多轮并发热路径优化
|
||||
- 已进一步压缩 executor 内部 backlog，补位更接近固定槽位模型
|
||||
- 但还需要一轮 live 运行观察 claimed/running 曲线，确认旧的批量认领惯性已被压住
|
||||
|
||||
本任务必须确认：
|
||||
- 活跃槽位稳定贴近配置上限
|
||||
- 并发不再大起大落
|
||||
- 吞吐明显提升
|
||||
- 调度器自耗下降
|
||||
|
||||
当前结论：
|
||||
- 这是主线里仍然偏重的未完项
|
||||
- 2026-04-21 已继续做现场修正：
|
||||
- `claim_detect_job_items()` 已显式排除空 `step_code` 的 legacy 项，避免 worker 从标准队列入口继续误吞旧 whole-domain 项
|
||||
- `mainland-controller-01 / 121.204.244.188` 与 `mainland-worker-01 / 121.204.244.248` 均已确认在真大陆节点参与执行
|
||||
- 海外控制面 `queue_health / dashboard` 已补上“读取大陆 runtime/debug 近窗执行流”的兜底口径
|
||||
- 当前首页已能看到真实近窗吞吐，例如 `10-15 项/分钟` 量级、并能拆到 controller/worker 两个节点
|
||||
- 当前主线判断：
|
||||
- “高 claimed 假活跃”问题已继续缓解
|
||||
- “活吞吐不可见”问题已明显改善
|
||||
- 但“总 completed 账本持续增长”和“running 贴近线程上限”仍未彻底验收，所以大任务 5 仍未签字通过
|
||||
|
||||
### 大任务 6
|
||||
时光机一期流程落地
|
||||
|
||||
目标：
|
||||
时光机按“最近 5 年”方案接成标准步骤任务。
|
||||
|
||||
当前状态：
|
||||
- 现有项目里已有时光机相关检测逻辑
|
||||
- 但还没有完全按标准步骤任务方式接进新 pipeline 验收
|
||||
|
||||
本任务必须确认：
|
||||
- wayback 作为标准步骤任务接入
|
||||
- 最近 5 年快照策略稳定
|
||||
- 返回标准化结果
|
||||
- controller 能把它当普通步骤推进/终止
|
||||
|
||||
当前结论：
|
||||
- 2026-04-21 已落代码并通过测试：
|
||||
- `detect_wayback` 已作为标准 `single_step` 步骤接入 controller / worker 主链
|
||||
- 时光机一期已按“最近 5 年 + 命中即停”策略落地到 payload 和 detector
|
||||
- 焦点测试已通过
|
||||
- 但还缺 live 现场验收：
|
||||
- 真实任务投递
|
||||
- worker 执行
|
||||
- 标准化结果回传
|
||||
- controller 按普通步骤推进/终止
|
||||
- 当前主线结论：大任务 6 已进入“代码落地完成、待现场验收”状态
|
||||
|
||||
### 大任务 7
|
||||
运营视角指标与验收面板落地
|
||||
|
||||
目标：
|
||||
让后台能判断“有没有跑起来、卡在哪一步、多久跑完”。
|
||||
|
||||
当前状态：
|
||||
- 后端已有部分 runtime / detect status / debug event 统计基础
|
||||
- 但完整的运营视角指标面板还没有完全落地验收
|
||||
|
||||
本任务必须确认：
|
||||
- 每步骤队列数
|
||||
- 每步骤吞吐
|
||||
- 每分钟完成量
|
||||
- 当前 pipeline 分布
|
||||
- 重试数 / 黑名单数 / 失败数
|
||||
- 预估剩余时间
|
||||
|
||||
当前结论：
|
||||
- 2026-04-21 已落地首页最小运营面板，并补了第二层口径修正：
|
||||
- `completed / pending / running` 已切到“全活跃任务累计口径”
|
||||
- `步骤队列 / 节点吞吐 / 重试压力 / 有效执行节点` 已可直接展示
|
||||
- `ETA` 在无真实近窗完成量时会显示“待计算”，避免误报 0 小时
|
||||
- 2026-04-21 又补上“大陆 runtime/debug 近窗执行流”兜底：
|
||||
- `active_job` 已可显示 `runtime_job_code`
|
||||
- `processed_per_minute / completed_recent / node_throughput / step_queue` 已能贴近大陆真实执行
|
||||
- 首页已能直接区分 `db_job_code` 与 `runtime_job_code`
|
||||
- 当前结论：大任务 7 已进入“可运营分析并能指导现场排障”阶段，但仍需继续把“总 completed 账本”和“最终验收面板”完全统一
|
||||
|
||||
## 现在只做什么
|
||||
|
||||
当前只盯主线，不跑偏：
|
||||
|
||||
1. 把大任务 1 做成可真实验收
|
||||
2. 验收过后立刻推进大任务 2
|
||||
3. 再按顺序推进 3、4、5、6、7
|
||||
|
||||
如果某个问题只是：
|
||||
|
||||
- CPU 没吃满
|
||||
- 某个 timeout 还能再抠
|
||||
- 代理池还能更激进
|
||||
- 页面还能再改得更好看
|
||||
|
||||
都先不打断主线。
|
||||
|
||||
## 已封存的细节优化 backlog
|
||||
|
||||
下面这些不是不做，而是先封存：
|
||||
|
||||
### A. 并发/调度优化
|
||||
|
||||
- 固定 worker pool 彻底替换旧批量认领模型
|
||||
- claimed 回弹继续压缩
|
||||
- executor 内部排队继续收紧
|
||||
- 活跃 running 继续往上抬
|
||||
- controller / worker 双节点吞吐平衡
|
||||
|
||||
### B. DB 往返优化
|
||||
|
||||
- 继续合并 `domains` 表高频更新
|
||||
- 继续减少 `mark_running / finalize` 这类必要写库点开销
|
||||
- 能走 Redis 或本地状态的尽量不走高频 DB
|
||||
|
||||
### C. 外部请求链路优化
|
||||
|
||||
- 代理失败后的重试链继续压缩
|
||||
- 直连失败后回代理的等待窗口再收紧
|
||||
- 无代理窗口等待策略再优化
|
||||
- 外部请求 timeout 再按真实成功率调优
|
||||
|
||||
### D. 代理池策略优化
|
||||
|
||||
- 代理池刷新频率与补位策略继续增强
|
||||
- 失败代理淘汰与新代理拉取节奏继续优化
|
||||
- 控制“不过度预验证”和“不过度浪费线程”之间的平衡
|
||||
|
||||
### E. 运营展示层优化
|
||||
|
||||
- 更细的 runtime 面板
|
||||
- ETA / 每分钟吞吐更精细展示
|
||||
- 日志窗口布局继续优化
|
||||
- 非主线的页面交互增强
|
||||
|
||||
## 接下来执行口径
|
||||
|
||||
接下来统一按这个口径推进：
|
||||
|
||||
- 先验收主流程
|
||||
- 再补主流程缺口
|
||||
- 性能细节先记账，不抢主线
|
||||
- 每做完一个大任务，就给出“是否验收通过”的明确结论
|
||||
|
||||
一句话定调：
|
||||
|
||||
当前阶段不是“继续无限抠并发”，而是“先把 7 个大任务按主线流程逐个打通并验收”。
|
||||
49
docs/codex接手.md
Normal file
49
docs/codex接手.md
Normal file
@@ -0,0 +1,49 @@
|
||||
把下面这段直接发给新的 Codex 就行：
|
||||
|
||||
```text
|
||||
接手这个项目，请先不要发散，也不要先动展示层。你先完整阅读并基于现状继续推进主线。
|
||||
|
||||
项目路径：
|
||||
`/www/wwwroot/getDomain`
|
||||
|
||||
必须先看这几份文档：
|
||||
1. `docs/ops_center_runtime/HANDOFF_20260420_1920.md`
|
||||
2. `docs/base.md`
|
||||
3. `docs/test.md`
|
||||
|
||||
这次接手的硬性要求：
|
||||
1. 先以 `docs/ops_center_runtime/HANDOFF_20260420_1920.md` 为准建立上下文。
|
||||
2. 不要把当前这台机器当成真正的 mainland controller。
|
||||
3. 当前工作机是海外 12 核测试控制面，真正的 `mainland-controller-01` 是 `121.204.244.188`。
|
||||
4. 不要再优先动页面、展示层、面板文案。
|
||||
5. 不要用破坏性 git 命令，不要回滚现有脏工作区改动。
|
||||
6. 新增或修改代码前，先确认你改的是主线瓶颈，而不是辅助功能。
|
||||
|
||||
当前已经确认的事实：
|
||||
1. 真正的 `mainland-controller-01` 已重新对正。
|
||||
2. controller 的 node-agent 身份上报已经修好，现在控制面里应显示：
|
||||
- `agent_hostname = mainland-controller-01`
|
||||
- `agent_ip = 121.204.244.188`
|
||||
3. controller 远端发布链已经成功跑通过一次。
|
||||
4. 当前真正吃任务的是 `mainland-controller-01`。
|
||||
5. `mainland-worker-01` 目前是 agent 在线，但检测没有真正参与，现象是本地检测态里出现 `127.0.0.1:5432 connection refused`。
|
||||
6. 当前主线已经不是“节点身份问题”，而是“真 controller 吞吐”和“worker 恢复”。
|
||||
|
||||
你接手后只做两条主线：
|
||||
1. 恢复 `mainland-worker-01`，让它重新进入可参与检测状态。
|
||||
2. 继续压 `mainland-controller-01` 的真实吞吐，只盯 `claimed -> running -> completed` 的推进，不回展示层。
|
||||
|
||||
你要避免的坑：
|
||||
1. 不要再把本机当成 `mainland-controller-01`。
|
||||
2. 不要优先用本机 Python service 入口直接造 deploy job，优先通过运行中的 API HTTP 接口。
|
||||
3. 不要把 `job 332` 这种“node-agent 重启自己导致状态未优雅回写”的现象直接误判为真正失败。
|
||||
4. 不要跑偏到 UI、导出、日志窗口样式这些支线。
|
||||
|
||||
你开始后先做这三件事，再继续动手：
|
||||
1. 复述你理解的当前真实环境拓扑。
|
||||
2. 复述当前两条唯一主线任务。
|
||||
3. 给出你准备先验证的 3 个现场指标，再开始执行。
|
||||
|
||||
目标只有一个：
|
||||
先把主流程和真实并发跑起来，让 controller 真机和 worker 真正参与检测，再谈细节优化。
|
||||
```
|
||||
96
docs/ops_center_runtime/HANDOFF_20260419_0308.md
Normal file
96
docs/ops_center_runtime/HANDOFF_20260419_0308.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# HANDOFF 2026-04-19 03:08
|
||||
|
||||
## 本轮做了什么
|
||||
|
||||
本轮没有进入新实现，只做了最小链路核查：
|
||||
|
||||
1. 复查中央 `detect/job/active`
|
||||
2. 复查中央 `ops/nodes`
|
||||
3. 复查 `go-live-summary` / `stack-diagnosis` / `release-launchpad`
|
||||
4. 复查最新 `onboarding.acceptance`
|
||||
5. 通过远端 Agent 对 `mainland-controller-01` 执行：
|
||||
- `service.status`
|
||||
- `logs.collect`
|
||||
目标服务：`domaincheck-sync-agent`
|
||||
|
||||
## 本轮结论
|
||||
|
||||
当前状态已经进一步推进：
|
||||
|
||||
- 检测任务在跑
|
||||
- 节点接管在跑
|
||||
- 最新 acceptance 已成功
|
||||
- `mainland-controller-01` 的 `domaincheck-sync-agent` 已重启成功
|
||||
- 重启后首轮已把 mainland 逐条结果推回中央
|
||||
|
||||
## 关键证据
|
||||
|
||||
### 1. 检测任务确实在跑
|
||||
|
||||
- 活跃任务：`detect-20260417170546-96023e`
|
||||
- 参与节点：
|
||||
- `mainland-worker-01`
|
||||
- `mainland-controller-01`
|
||||
- `overseas-control-01`
|
||||
|
||||
### 2. 最新 acceptance 已成功
|
||||
|
||||
- `pbr-9ce5c85f17`：成功
|
||||
- `pbr-389abd618c`：成功
|
||||
|
||||
说明：
|
||||
|
||||
- 接管验收不再是当前唯一主阻塞
|
||||
|
||||
### 3. sync-agent 服务已重启到新进程
|
||||
|
||||
远端只读结果：
|
||||
|
||||
- 服务：`domaincheck-sync-agent`
|
||||
- 状态：`active (running)`
|
||||
- 启动时间：`2026-04-19 16:09:52 CST`
|
||||
- 运行命令：
|
||||
- `/opt/domaincheck/domainCheck/.venv/bin/python -m app.sync_agent`
|
||||
|
||||
### 4. 重启后首轮结果投影已推送成功
|
||||
|
||||
最新日志显示：
|
||||
|
||||
- `detect_result_projection`
|
||||
- `batch_count = 1`
|
||||
- `success_count = 1`
|
||||
- `event_import.imported_count = 10`
|
||||
|
||||
### 5. 中央 mainland 逐条结果已出现
|
||||
|
||||
- 新增 `detect_result_ingest`
|
||||
- `id = 5382`
|
||||
- `created_at = 2026-04-19 03:10:14`
|
||||
- 近期 mainland `domain_*`
|
||||
- `count = 10`
|
||||
|
||||
## 下一轮唯一剩余动作
|
||||
|
||||
下一轮不要发散实现，只做持续性复查：
|
||||
|
||||
1. `/api/v1/runtime/sync-summary`
|
||||
2. `/api/v1/detect/job/active`
|
||||
3. mainland `domain_started/domain_completed/domain_failed/domain_blacklisted`
|
||||
4. `/api/v1/ops/go-live-summary`
|
||||
5. `/api/v1/ops/stack-diagnosis`
|
||||
|
||||
重点判断：
|
||||
|
||||
- mainland `domain_*` 是否持续增长
|
||||
- 总检 attention 是否已主要退化为历史残留
|
||||
|
||||
## 继续不要做什么
|
||||
|
||||
- 不做新页面
|
||||
- 不做控制面增强
|
||||
- 不做发布动作
|
||||
- 不做 item 级最终回写
|
||||
|
||||
## 一句话结论
|
||||
|
||||
`mainland-controller-01` 的 `domaincheck-sync-agent` 重启后，逐条结果同步已经打通；当前工作重点从“修链路”切换到“验证持续性与上线签收口径”。
|
||||
128
docs/ops_center_runtime/HANDOFF_20260419_0318.md
Normal file
128
docs/ops_center_runtime/HANDOFF_20260419_0318.md
Normal file
@@ -0,0 +1,128 @@
|
||||
# HANDOFF 2026-04-19 03:18
|
||||
|
||||
## 本轮做了什么
|
||||
|
||||
本轮没有进入新实现，只做了运行态只读复查：
|
||||
|
||||
1. 对比一个完整 40 秒观察窗口前后的：
|
||||
- `/api/v1/detect/job/active`
|
||||
- `/api/v1/runtime/sync-summary`
|
||||
2. 复查中央 `detect_run_events`
|
||||
3. 复查：
|
||||
- `/api/v1/ops/go-live-summary`
|
||||
- `/api/v1/ops/stack-diagnosis`
|
||||
- `/api/v1/ops/activity-stream`
|
||||
4. 复查节点现场日志：
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
|
||||
## 本轮结论
|
||||
|
||||
这轮结论要纠偏：
|
||||
|
||||
- 接管链路基本完成
|
||||
- 同步链路已经打通过一次
|
||||
- 但检测执行面当前没有继续出新结果
|
||||
|
||||
因此当前主阻塞不再是“接管/同步未通”，而是：
|
||||
|
||||
- 检测执行停滞
|
||||
- 外部站点依赖或代理池可用性异常
|
||||
|
||||
## 关键证据
|
||||
|
||||
### 1. 三台节点都在参与，但近窗没有吞吐
|
||||
|
||||
活跃任务仍是：
|
||||
|
||||
- `detect-20260417170546-96023e`
|
||||
|
||||
参与节点：
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
- `overseas-control-01`
|
||||
|
||||
但 40 秒观察窗口前后完全一致：
|
||||
|
||||
- `progress_percent = 2.1`
|
||||
- `completed = 21`
|
||||
- `running = 14`
|
||||
- `claimed = 34`
|
||||
- `pending = 931`
|
||||
|
||||
### 2. mainland 逐条结果没有继续增长
|
||||
|
||||
中央查询结果：
|
||||
|
||||
- mainland `domain_*` 事件数仍为 `10`
|
||||
- 最新 mainland `detect_result_ingest` 仍是：
|
||||
- `id = 5382`
|
||||
- `created_at = 2026-04-19 03:10:14`
|
||||
|
||||
说明：
|
||||
|
||||
- 首批同步成功过
|
||||
- 但后续没有继续流入新结果
|
||||
|
||||
### 3. go-live 与 stack 口径已经比之前更完整
|
||||
|
||||
当前：
|
||||
|
||||
- `remote_access_ready = 3/3`
|
||||
- `log_sync_state = full_capture`
|
||||
- `go_live_status = attention`
|
||||
|
||||
这说明：
|
||||
|
||||
- 基础运维骨架已经起来
|
||||
- attention 现在不能只归因于接管未完成
|
||||
|
||||
### 4. controller 现场日志已直接指向代理/外部依赖问题
|
||||
|
||||
`mainland-controller-01` 现场日志显示：
|
||||
|
||||
- `当前可用代理数: 0`
|
||||
- `最近结果: 刷新成功，可用 0 个`
|
||||
- 检测链包含：
|
||||
- 注册
|
||||
- 百度 site
|
||||
- 360 site
|
||||
- 站长之家
|
||||
- 爱站
|
||||
- 时光机
|
||||
|
||||
### 5. 页面上的“外部站点异常”与现场证据一致
|
||||
|
||||
从当前现场判断：
|
||||
|
||||
- 这不是页面误报
|
||||
- 而是执行面确实卡在外部依赖/代理可用性上
|
||||
|
||||
## 下一轮唯一应该做什么
|
||||
|
||||
下一轮不要发散实现，只做 `J2-检测执行停滞收口批`：
|
||||
|
||||
1. 继续只读确认：
|
||||
- `/api/v1/detect/job/active`
|
||||
- `/api/v1/runtime/sync-summary`
|
||||
- `/api/v1/ops/activity-stream`
|
||||
- `/api/v1/ops/nodes/{node_code}/scene-log`
|
||||
2. 必要时补一轮：
|
||||
- `domaincheck-worker` 运行日志取证
|
||||
3. 只判断三件事：
|
||||
- 是否仍然 `近窗吞吐 0`
|
||||
- 是否仍然 `可用代理数 0`
|
||||
- 是否仍然没有新 `domain_*` 事件
|
||||
|
||||
## 现在不要做什么
|
||||
|
||||
- 不做新页面
|
||||
- 不做控制面增强
|
||||
- 不做发布动作
|
||||
- 不做 item 级最终回写
|
||||
- 不把当前状态误判成“已稳定可签收”
|
||||
|
||||
## 一句话结论
|
||||
|
||||
当前项目已经不是“接不管、看不见、不同步”，而是“接管和同步都基本打通了，但检测执行卡在外部站点/代理可用性问题上，导致任务挂起且没有继续产出”。
|
||||
117
docs/ops_center_runtime/HANDOFF_20260419_0324.md
Normal file
117
docs/ops_center_runtime/HANDOFF_20260419_0324.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# HANDOFF 2026-04-19 03:24
|
||||
|
||||
## 本轮做了什么
|
||||
|
||||
本轮没有进入新实现，只做了执行面根因收紧：
|
||||
|
||||
1. 继续观察活跃检测任务是否前进
|
||||
2. 继续观察 mainland `domain_*` 是否增长
|
||||
3. 通过正式 `ops job` 远端采集：
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
的 `domaincheck-worker` 日志
|
||||
4. 再盯一个 35 秒窗口，看 full capture 打开后，源日志时间是否继续前进
|
||||
|
||||
## 本轮结论
|
||||
|
||||
这轮已经能把问题说得更准：
|
||||
|
||||
- 不是“日志没回来”
|
||||
- 不是“全量日志没开”
|
||||
- 而是日志链路已经通了，但执行进程没有继续产生日志
|
||||
|
||||
当前第一主阻塞：
|
||||
|
||||
- `mainland-controller-01` 代理池可用数为 0
|
||||
|
||||
次级问题：
|
||||
|
||||
- `mainland-worker-01` 的时光机依赖异常会降级继续执行
|
||||
|
||||
## 关键证据
|
||||
|
||||
### 1. full capture 是开的，但源日志没继续前进
|
||||
|
||||
当前：
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `capture_at = 2026-04-19 03:21:59`
|
||||
- 源日志时间仍停在 `Apr 19 01:00:23`
|
||||
- `mainland-worker-01`
|
||||
- `capture_at = 2026-04-19 03:22:01`
|
||||
- 源日志时间仍停在 `Apr 19 02:19:56`
|
||||
|
||||
35 秒后再次复查：
|
||||
|
||||
- `capture_at` 没变化
|
||||
- `source_msg` 也没变化
|
||||
|
||||
说明：
|
||||
|
||||
- 日志回传本身不是主问题
|
||||
- 执行进程这段时间没有继续产生日志
|
||||
|
||||
### 2. controller 代理源能拉到数据，但所有代理都验不过
|
||||
|
||||
`mainland-controller-01` 的 `domaincheck-worker` 日志显示：
|
||||
|
||||
- 6 个代理源都能拉到原始代理
|
||||
- 抽样验证 24 个代理后：
|
||||
- `代理池刷新完成，共 0 个可用代理`
|
||||
- 失败集中在：
|
||||
- `ProxyError@https://m.baidu.com`
|
||||
- `Unable to connect to proxy`
|
||||
- `ConnectTimeoutError`
|
||||
|
||||
说明：
|
||||
|
||||
- 不是代理接口挂了
|
||||
- 是代理名单本身不可用
|
||||
|
||||
### 3. worker 还能跑，但时光机依赖异常会降级
|
||||
|
||||
`mainland-worker-01` 的 `domaincheck-worker` 日志显示：
|
||||
|
||||
- `时光机检测 外部依赖异常，步骤降级继续执行`
|
||||
- 同时仍可见：
|
||||
- `域名检测完成`
|
||||
- 收到新的控制消息时：
|
||||
- `收到启动检测指令，但检测任务已在运行，忽略重复启动`
|
||||
|
||||
说明：
|
||||
|
||||
- worker 不是完全不可用
|
||||
- 时光机异常存在，但不是最核心阻塞
|
||||
|
||||
### 4. 活跃任务仍然没有前进
|
||||
|
||||
- `progress_percent = 2.1`
|
||||
- `completed = 21`
|
||||
- `running = 14`
|
||||
- `claimed = 34`
|
||||
- `pending = 931`
|
||||
|
||||
并且 mainland `domain_*` 仍固定在 `10`
|
||||
|
||||
## 下一轮唯一应该做什么
|
||||
|
||||
继续只做 `J2-检测执行停滞收口批`：
|
||||
|
||||
1. 不扩功能
|
||||
2. 不改页面
|
||||
3. 只围绕 controller 代理池问题取证和恢复验证
|
||||
|
||||
唯一要确认的是：
|
||||
|
||||
- controller 代理池是否仍然 `可用 0`
|
||||
- 一旦代理恢复，`domain_*` 是否会继续增长
|
||||
|
||||
## 现在不要做什么
|
||||
|
||||
- 不把问题继续泛化成“外部站点都异常”
|
||||
- 不把问题误判为“日志回传没开”
|
||||
- 不做新页面、新模块、发布动作
|
||||
|
||||
## 一句话结论
|
||||
|
||||
当前项目不是“接管失败”，也不是“日志没回来”，而是“controller 侧拉到了代理名单，但代理全部校验失败，导致检测执行没有继续产生新结果”。
|
||||
66
docs/ops_center_runtime/HANDOFF_20260419_0426.md
Normal file
66
docs/ops_center_runtime/HANDOFF_20260419_0426.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# HANDOFF 2026-04-19 04:26 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已修复 `runtime/status` 兼容层，恢复旧字段：
|
||||
- `api_online`
|
||||
- `worker_online`
|
||||
- `cluster_summary`
|
||||
- `thread_count`
|
||||
- 已重启中央 `domaincheck-api`
|
||||
- 已重新构建 `domain-web` 前端静态包
|
||||
- 已确认大陆双节点日志回传同时进入中央：
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
- 已确认大陆两台线程配置真实生效：
|
||||
- `mainland-controller-01 -> 100`
|
||||
- `mainland-worker-01 -> 50`
|
||||
|
||||
## 当前真实状态
|
||||
|
||||
- `/api/v1/runtime/status`
|
||||
- `api_online = true`
|
||||
- `worker_online = true`
|
||||
- `cluster_summary.online_worker_nodes = 3`
|
||||
- `/api/v1/detect/status`
|
||||
- `progress.pending = 163815`
|
||||
- `progress.completed = 51`
|
||||
- `progress.running = 10`
|
||||
- `remote_log_line_count = 240`
|
||||
- `remote_log_node_count = 2`
|
||||
- 近 5 分钟中央 `detect_debug_events`
|
||||
- `mainland-controller-01 -> worker_log`
|
||||
- `mainland-worker-01 -> worker_log`
|
||||
|
||||
## 当前结论
|
||||
|
||||
- “API 离线 / 本机 Worker 离线 / 有效执行节点 0” 已不是后端真实状态
|
||||
- “全量日志没回来” 已不是后端真实状态
|
||||
- “100/50 并发没有下发” 也不是后端真实状态
|
||||
- 当前剩余主问题已经收紧为：
|
||||
- 检测实际吞吐仍偏低
|
||||
- 代理可用性与任务分发节奏仍在限制体感并发
|
||||
|
||||
## 下一轮唯一任务
|
||||
|
||||
只做：`J2-检测执行吞吐收口`
|
||||
|
||||
顺序：
|
||||
|
||||
1. 强刷浏览器，确认新前端包已经生效
|
||||
2. 复查 Detect 页是否恢复：
|
||||
- API 在线
|
||||
- 本机 Worker 在线
|
||||
- 有效执行节点 > 0
|
||||
- 日志窗口出现双节点日志
|
||||
3. 若显示已恢复，再继续只排查：
|
||||
- 为什么实际执行吞吐仍低于 `100/50`
|
||||
- 重点看代理可用性、任务领取节奏、线程实际活跃数
|
||||
|
||||
## 现在不要做
|
||||
|
||||
- 不扩新页面
|
||||
- 不扩控制面功能
|
||||
- 不新增专题文档
|
||||
- 不进入发布动作
|
||||
- 不切新大方向
|
||||
104
docs/ops_center_runtime/HANDOFF_20260419_1334.md
Normal file
104
docs/ops_center_runtime/HANDOFF_20260419_1334.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# HANDOFF 2026-04-19 13:34 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已修复 controller `sync-agent pull_tasks` 只导 `domains`、不导本地执行队列的问题
|
||||
- 已让 controller 持续创建本地镜像任务：
|
||||
- `sync-overseas-7380`
|
||||
- `sync-overseas-7383`
|
||||
- `sync-overseas-7392`
|
||||
- 后续批次持续增加
|
||||
- 已重启两台大陆 `domaincheck-worker`
|
||||
- 已确认两台大陆节点都进入真实任务队列：
|
||||
- `mainland-controller-01 -> 从任务队列获取到 800 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- 已修复 controller `runtime/detect_runs.json` 属主错误：
|
||||
- `root:root -> www:www`
|
||||
- 已确认 `runtime_projection` 最新同步恢复成功：
|
||||
- `sync_state = success`
|
||||
- `runtime_projection -> 投影推送成功`
|
||||
|
||||
## 当前真实状态
|
||||
|
||||
当前 `/api/v1/ops/nodes` 已显示：
|
||||
|
||||
- `remote_access_ready = 3`
|
||||
- `participating = 3`
|
||||
- `dispatch_active = 1`
|
||||
|
||||
大陆两台当前状态：
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `agent_state = online_busy`
|
||||
- `detect_runtime.max_threads = 100`
|
||||
- `processed_recent = 115`
|
||||
- `processed_per_minute = 7.67`
|
||||
- `is_current_participant = true`
|
||||
- `mainland-worker-01`
|
||||
- `agent_state = online_busy`
|
||||
- `detect_runtime.max_threads = 50`
|
||||
- `processed_recent = 632`
|
||||
- `processed_per_minute = 42.13`
|
||||
- `is_current_participant = true`
|
||||
|
||||
## 当前结论
|
||||
|
||||
- 大陆两台不是“在线但没干活”
|
||||
- 而是已经进入真实镜像队列执行
|
||||
- `100/50` 并发也不是停留在配置层，而是已经开始真实消费任务
|
||||
- 后台运行态同步链已经恢复
|
||||
|
||||
现在剩余问题已经收敛到：
|
||||
|
||||
- Detect 页面主计数 / 日志窗口是否完全跟上新的运行态
|
||||
- 恢复后的吞吐是否能持续稳定
|
||||
|
||||
## 一个关键口径说明
|
||||
|
||||
当前大陆执行是“镜像队列”模式：
|
||||
|
||||
- 海外待检测批次先被 controller 拉回大陆本地
|
||||
- 在大陆本地形成 `sync-overseas-*` 的 `detect_jobs/detect_job_items`
|
||||
- 再由大陆 worker 真正执行
|
||||
- 运行态和结果再同步回海外
|
||||
|
||||
所以现在不要再只盯中央原生 `detect_job_items.claimed/running` 判断大陆有没有参与。
|
||||
|
||||
应该优先看：
|
||||
|
||||
- `/api/v1/ops/nodes`
|
||||
- `processed_recent`
|
||||
- `processed_per_minute`
|
||||
- `detect_runtime.active_threads/max_threads`
|
||||
|
||||
## 下一轮唯一任务
|
||||
|
||||
只做：`J2-页面口径与吞吐稳定性收口`
|
||||
|
||||
顺序：
|
||||
|
||||
1. 继续观察 Detect 页面
|
||||
2. 确认日志窗口是否已经持续刷新
|
||||
3. 确认主计数是否开始跟随最新运行态
|
||||
4. 继续观察 1 到 2 个同步周期内：
|
||||
- `mainland-controller-01 processed_recent`
|
||||
- `mainland-worker-01 processed_recent`
|
||||
- `processed_per_minute`
|
||||
5. 若页面仍不跟，优先修页面读取口径，不扩功能
|
||||
|
||||
## 现在不要做
|
||||
|
||||
- 不扩新页面
|
||||
- 不扩控制面功能
|
||||
- 不新增模块
|
||||
- 不进入发布动作
|
||||
- 不切新方向
|
||||
|
||||
## 推荐模型
|
||||
|
||||
- 当前最合适：`GPT-5.4 + high`
|
||||
|
||||
原因：
|
||||
|
||||
- 现在主要是收口、验证、局部修复
|
||||
- 需要稳定推理，但不需要切到超高
|
||||
109
docs/ops_center_runtime/HANDOFF_20260419_1348.md
Normal file
109
docs/ops_center_runtime/HANDOFF_20260419_1348.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# HANDOFF 2026-04-19 13:48 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已定位并修复 `domainCheck/detect_worker.py` 的上下文绑定缺口：
|
||||
- `sync-pull` 控制消息只有
|
||||
- `target_job_id`
|
||||
- `target_job_code`
|
||||
- 旧逻辑只读取
|
||||
- `job_id`
|
||||
- `job_code`
|
||||
- 结果是 `mainland-worker-01` 明明已经在跑，但 `current_job_id` 为空，`worker_log` 被整段短路
|
||||
- 已把补丁同步到：
|
||||
- `mainland-controller-01:/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- `mainland-worker-01:/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- 已在两台大陆机完成：
|
||||
- `python -m py_compile /opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- `systemctl restart domaincheck-worker`
|
||||
|
||||
## 当前硬证据
|
||||
|
||||
### 1. mainland worker 日志回传已恢复
|
||||
|
||||
当前 `/api/v1/runtime/debug-events` 已出现：
|
||||
|
||||
- `mainland-worker-01 -> 开始执行检测任务，来源: redis-control`
|
||||
- `mainland-worker-01 -> 开始执行域名检测任务，正在加载配置`
|
||||
- `mainland-worker-01 -> 开始检测，正在刷新代理池`
|
||||
- `mainland-worker-01 -> 代理池刷新完成，共 7 个可用代理，来源链接 6 个，原始 265 个，验证 150 个`
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 1/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 2/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 3/50`
|
||||
|
||||
### 2. Detect 页面远端日志已经恢复双节点
|
||||
|
||||
当前 `/api/v1/detect/status` 已显示：
|
||||
|
||||
- `remote_log_node_count = 2`
|
||||
- `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]`
|
||||
|
||||
最近日志样本：
|
||||
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 1/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 2/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 3/50`
|
||||
- `mainland-controller-01 -> 当前实际线程数量: 1/100 ... 3/100`
|
||||
|
||||
### 3. 三台节点仍保持参与态
|
||||
|
||||
当前 `/api/v1/ops/nodes` 摘要：
|
||||
|
||||
- `online = 3`
|
||||
- `agent_ready = 3`
|
||||
- `remote_access_ready = 3`
|
||||
- `participating = 3`
|
||||
- `dispatch_active = 1`
|
||||
|
||||
## 当前判断
|
||||
|
||||
已经闭合：
|
||||
|
||||
- 大陆节点真实执行
|
||||
- mainland worker 日志回传
|
||||
- Detect 页面远端日志窗口只显示 controller 的问题
|
||||
|
||||
仍待收口：
|
||||
|
||||
- Detect 页面主计数：
|
||||
- `pending = 163678`
|
||||
- `completed = 195`
|
||||
- `running = 3`
|
||||
- 当前这些主计数还没有和这轮大陆执行立即同步推进
|
||||
|
||||
因此当前剩余唯一主问题已经缩成：
|
||||
|
||||
- `detect_result_projection / 结果统计回推` 是否还存在延迟或聚合口径缺口
|
||||
|
||||
## 下一轮唯一任务
|
||||
|
||||
只做：`J2-结果计数收口批`
|
||||
|
||||
顺序：
|
||||
|
||||
1. 复查 `detect_result_projection` 最新推送是否持续成功
|
||||
2. 复查 mainland 共享库里 `sync-overseas-*` 任务的完成/失败数量是否增长
|
||||
3. 复查 overseas `detect/status.progress` 是否跟着推进
|
||||
4. 若日志继续推进但主计数仍不动，只修结果统计回推口径，不扩功能
|
||||
|
||||
## 现在不要做
|
||||
|
||||
- 不扩新页面
|
||||
- 不扩控制面功能
|
||||
- 不新增模块
|
||||
- 不切新方向
|
||||
- 不直接进入发布动作
|
||||
|
||||
## 推荐模型
|
||||
|
||||
- 当前继续用：`GPT-5.4 + high`
|
||||
|
||||
原因：
|
||||
|
||||
- 现在是收口型问题
|
||||
- 需要稳定排查与小范围补丁
|
||||
- 不需要切超高推理
|
||||
65
docs/ops_center_runtime/HANDOFF_20260419_1416.md
Normal file
65
docs/ops_center_runtime/HANDOFF_20260419_1416.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# HANDOFF 2026-04-19 14:16 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已修复 `detect_worker.py` 中“代理不足反向限制并发”的热路径：
|
||||
- 代理池不足时改为后台补货
|
||||
- 不再因为 `current_pool_size < thread_count/2` 而同步刷新、拖慢线程拉起
|
||||
- 已对线程创建阶段的运行态 / worker_log 做节流：
|
||||
- 不再每起一个线程就同步打一轮重日志
|
||||
- 避免远端日志回传本身把并发拉低
|
||||
- 已把上述修复部署到：
|
||||
- overseas 当前 live 运行目录 `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- repo 目录 `/www/wwwroot/getDomain/domainCheck/detect_worker.py`
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
- 已补上“真实活跃检测线程计数”逻辑到代码文件中
|
||||
|
||||
## 当前结果
|
||||
|
||||
- `mainland-controller-01`
|
||||
- 海外后台当前已能看到：
|
||||
- `active_threads ~= 99~100`
|
||||
- `max_threads = 100`
|
||||
- 结论：
|
||||
- controller 的 `100` 并发已经真实跑起来
|
||||
- `mainland-worker-01`
|
||||
- 已重新接到新批次：
|
||||
- `source_record_id = 7679`
|
||||
- `target_job_code = sync-overseas-7679`
|
||||
- 最新日志已出现：
|
||||
- `开始执行域名检测任务`
|
||||
- `开始检测，刷新代理池`
|
||||
- `代理池刷新完成，共 23 个可用代理`
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- 本机进程线程量：
|
||||
- `152`
|
||||
- 结论：
|
||||
- worker 已重新进入真实执行
|
||||
- 但 overseas 后台对 `mainland-worker-01.active_threads/current_load` 的显示仍偏低
|
||||
|
||||
## 当前剩余问题
|
||||
|
||||
- 不是 controller 并发限制问题
|
||||
- 当前唯一剩余的运行面收口点是：
|
||||
- `mainland-worker-01` 的运行态上报 / 后台显示口径仍未完全对齐
|
||||
- 同时仍需继续观察：
|
||||
- Detect 页面主计数 `pending/completed/running`
|
||||
- 结果统计回推是否继续推进
|
||||
|
||||
## 下一轮建议
|
||||
|
||||
只继续做这两件事：
|
||||
|
||||
- 追 `mainland-worker-01` 的运行态采集链：
|
||||
- 为什么本机已进入新批次执行，但海外后台仍显示 `active_threads = 2`
|
||||
- 继续核对 Detect 主计数与结果回推：
|
||||
- 判断是统计延迟、投影延迟，还是任务结果尚未进入中央口径
|
||||
|
||||
不要做：
|
||||
|
||||
- 新页面
|
||||
- 新模块
|
||||
- 控制面增强
|
||||
- 发布动作
|
||||
- 新专题文档
|
||||
52
docs/ops_center_runtime/HANDOFF_20260419_1421.md
Normal file
52
docs/ops_center_runtime/HANDOFF_20260419_1421.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# HANDOFF_20260419_1421
|
||||
|
||||
更新时间：2026-04-19 14:21 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已完成检测页前端布局收口并上线验证：
|
||||
- `任务日志控制台` 已上移到事件列表前
|
||||
- `检测事件流` 已改成独立滚动区域
|
||||
- 事件表已限制最大高度，避免页面被长列表持续撑长
|
||||
- 已在真实线上静态目录重新构建：
|
||||
- `/www/wwwroot/getDomain/domain-web/dist`
|
||||
- 已验证线上 Nginx 实际指向该目录：
|
||||
- `/www/server/panel/vhost/nginx/domaincheck_3201.conf`
|
||||
- `/www/server/panel/vhost/nginx/domaincheck_152.53.37.118.conf`
|
||||
- 已验证本机线上端口可返回新页面：
|
||||
- `curl -I http://127.0.0.1:3201/ -> 200`
|
||||
- `index.html` 最后修改时间已更新到本轮
|
||||
- 新资源：
|
||||
- `/assets/DetectView-B9S1WMRT.js`
|
||||
- `/assets/DetectView-BFxMNRNR.css`
|
||||
|
||||
## 当前判断
|
||||
|
||||
- 这轮前端布局任务已经闭环
|
||||
- 如果浏览器仍显示旧布局，优先判断为缓存未刷新
|
||||
- 当前更值得继续追的不是页面结构，而是：
|
||||
- `mainland-worker-01` 运行态上报口径
|
||||
- Detect 主计数推进
|
||||
- 并发真实吞吐继续拉升
|
||||
|
||||
## 下一步建议
|
||||
|
||||
下一轮继续只围绕检测执行收口：
|
||||
|
||||
- 复核 `mainland-worker-01` 为什么本机线程已拉起，但后台 `active_threads/current_load` 仍偏低
|
||||
- 继续比对：
|
||||
- `/api/v1/detect/status`
|
||||
- `/api/v1/runtime/debug-events`
|
||||
- `/api/v1/ops/nodes`
|
||||
- 继续验证 Detect 页面主计数是否跟随真实执行推进
|
||||
|
||||
## 本轮关键命令证据
|
||||
|
||||
```bash
|
||||
cd /opt/domaincheck/domain-web && npm run build
|
||||
curl -I http://127.0.0.1:3201/
|
||||
curl http://127.0.0.1:3201/
|
||||
rg -n "event-stream-panel|任务日志控制台|检测事件流" \
|
||||
/www/wwwroot/getDomain/domain-web/dist/assets/DetectView-BFxMNRNR.css \
|
||||
/www/wwwroot/getDomain/domain-web/dist/assets/DetectView-B9S1WMRT.js
|
||||
```
|
||||
54
docs/ops_center_runtime/HANDOFF_20260419_1429.md
Normal file
54
docs/ops_center_runtime/HANDOFF_20260419_1429.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# HANDOFF_20260419_1429
|
||||
|
||||
更新时间：2026-04-19 14:29 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已修复代理链“配置已下发但长期停留未刷新”的问题
|
||||
- `domaincheck-worker` 现在会在以下时机主动刷新代理池：
|
||||
- worker 启动后
|
||||
- `proxy_config`
|
||||
- `thread_count`
|
||||
- `node_thread_counts`
|
||||
- `runtime_settings`
|
||||
- 以上配置更新后
|
||||
- 已重启：
|
||||
- `domaincheck-worker`
|
||||
- `domaincheck-api`
|
||||
- 已完成一次真实代理池刷新并拿到明确结果：
|
||||
- 原始代理数：`202`
|
||||
- 抽样验证数：`24`
|
||||
- 可用代理数：`0`
|
||||
|
||||
## 当前状态
|
||||
|
||||
- 当前后台不再显示误导性的“未刷新”
|
||||
- 当前真实状态为：
|
||||
- `proxy_runtime_label = 降级直连`
|
||||
- `proxy_runtime_reason = proxy_validation_zero`
|
||||
- `proxy_runtime_detail = 代理源最近返回了 202 个代理，已验证 24 个，但当前 0 个可用；系统已自动降级为直连继续执行；最近状态：刷新成功，可用 0 个`
|
||||
|
||||
## 关键判断
|
||||
|
||||
- 这批代理当前的主要问题不是“系统没去用”
|
||||
- 而是：
|
||||
- 代理源会返回很多 IP
|
||||
- 但 worker 实测校验时全部失败
|
||||
- 失败以 `ProxyError / ConnectTimeout` 为主
|
||||
|
||||
## 下一步建议
|
||||
|
||||
- 优先不要再纠结“为什么页面写未刷新”
|
||||
- 这一层已经修好
|
||||
- 下一轮如果继续优化代理，应只做以下两种之一：
|
||||
- 更换/补充代理供应组，重新验证可用率
|
||||
- 调整代理校验策略，但要接受更高的脏代理混入风险
|
||||
|
||||
## 现场证据
|
||||
|
||||
```text
|
||||
主动触发代理池刷新: worker_startup
|
||||
代理池链接拉取成功 ... 原始代理数: 50 / 50 / 0 / 50 / 2 / 50
|
||||
代理池验证已启用抽样模式，本次抽样 24 个代理进行可用性验证
|
||||
代理池刷新完成，共 0 个可用代理，来源链接 6 个
|
||||
```
|
||||
42
docs/ops_center_runtime/HANDOFF_20260419_1450.md
Normal file
42
docs/ops_center_runtime/HANDOFF_20260419_1450.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# HANDOFF_20260419_1450
|
||||
|
||||
更新时间：2026-04-19 14:50 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已继续优化代理链，不只停留在“看见 0 可用”
|
||||
- 已新增两项优化：
|
||||
- 扩展严格验证目标：
|
||||
- `baidu`
|
||||
- `m.baidu`
|
||||
- `qq`
|
||||
- `360`
|
||||
- 当严格验证 `0` 命中时，启用：
|
||||
- `宽松入池 + 快速隔离`
|
||||
|
||||
## 当前结果
|
||||
|
||||
- 当前最新代理刷新结果：
|
||||
- `proxy_last_refresh_total_items = 270`
|
||||
- `proxy_last_validated_count = 24`
|
||||
- `proxy_last_available_count = 2`
|
||||
- `proxy_last_refresh_status = 宽松入池 2 个（严格校验 0 命中）`
|
||||
- 后台当前已显示：
|
||||
- `available_proxy_count = 2`
|
||||
- `proxy_runtime_detail = 代理池当前可用 2 个代理，配置来源 6 个；最近状态：宽松入池 2 个（严格校验 0 命中）`
|
||||
|
||||
## 关键判断
|
||||
|
||||
- 严格校验下，这批代理整体质量依然偏差
|
||||
- 但现在已经不再是完全 `0` 可用
|
||||
- 系统已成功放出少量试跑代理，可以继续观察：
|
||||
- 是否带动检测吞吐
|
||||
- 是否很快被失败隔离重新打回 `0`
|
||||
|
||||
## 下一步
|
||||
|
||||
- 优先观察这 `2` 个试跑代理是否真的参与检测执行
|
||||
- 如果能参与并带来吞吐，再继续逐步放宽
|
||||
- 如果很快再次掉回 `0`，下一轮就应转到：
|
||||
- 补代理供应组
|
||||
- 或按地区/分组拆分代理质量统计
|
||||
64
docs/ops_center_runtime/HANDOFF_20260419_1454.md
Normal file
64
docs/ops_center_runtime/HANDOFF_20260419_1454.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# HANDOFF 2026-04-19 14:54 CST
|
||||
|
||||
## 本轮目标
|
||||
|
||||
把代理链从“预验证优先”切到用户指定的最小高效路径：
|
||||
|
||||
- 只去掉过期 / 非法代理
|
||||
- 不再单独做可用性预验证
|
||||
- 直接投入真实检测
|
||||
- 失败即淘汰并触发补刷
|
||||
|
||||
## 本轮已完成
|
||||
|
||||
- 已修改 [detect_worker.py](/www/wwwroot/getDomain/domainCheck/detect_worker.py)
|
||||
- `refresh_proxy_pool()`
|
||||
- 移除预验证入池逻辑
|
||||
- 仅做过期 / 非法 / 去重过滤
|
||||
- 直接把代理源返回结果入池
|
||||
- `remove_proxy()`
|
||||
- 代理失败后不再留在当前池尾部
|
||||
- 直接从当前池移除
|
||||
- 保留失败标记并在低水位时触发补刷
|
||||
- 已重启 `domaincheck-worker`
|
||||
- 已确认 worker 新日志进入运行态
|
||||
|
||||
## 最新运行证据
|
||||
|
||||
- worker 最新日志：
|
||||
- `代理池刷新完成，共 270 个入池代理，来源链接 6 个，原始 270 个，去过期 0 个，非法 0 个`
|
||||
- API 最新状态：
|
||||
- `available_proxy_count = 270`
|
||||
- `proxy_last_refresh_status = 直入池 270 个（跳过预验证）`
|
||||
- `proxy_last_validated_count = 0`
|
||||
- `proxy_runtime_reason = healthy`
|
||||
|
||||
## 当前判断
|
||||
|
||||
- 这轮目标已经完成
|
||||
- 代理链现在已经符合“直接跑任务，失败就丢弃代理并换新”的要求
|
||||
- 下一步不该再回头做单独代理预验证
|
||||
|
||||
## 下一步主任务
|
||||
|
||||
唯一主任务：
|
||||
|
||||
- 继续观察真实检测吞吐是否随直入池策略抬升
|
||||
- 重点看：
|
||||
- `/api/v1/detect/status`
|
||||
- `/api/v1/ops/nodes`
|
||||
- 检测页主计数 `pending/completed/running`
|
||||
- 失败代理淘汰后是否能持续自动补池
|
||||
|
||||
## 若下一轮继续
|
||||
|
||||
优先处理：
|
||||
|
||||
- “并发已下发但主计数不明显推进”的统计 / 调度口径问题
|
||||
|
||||
不要处理：
|
||||
|
||||
- 不要重新加回重预验证逻辑
|
||||
- 不要扩展新页面
|
||||
- 不要扩展控制面新模块
|
||||
- 不要发散到发布链或新专题文档
|
||||
107
docs/ops_center_runtime/HANDOFF_20260419_2053.md
Normal file
107
docs/ops_center_runtime/HANDOFF_20260419_2053.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# HANDOFF_20260419_2053
|
||||
|
||||
更新时间：2026-04-19 20:53 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 修复发布打包遗漏：
|
||||
- 最新发布包现在已包含 `domainCheck/`
|
||||
- 最新正式签收包：
|
||||
- `domaincheck_release_20260419_205437`
|
||||
- 修复发布任务观测性：
|
||||
- `domain-api/app/services/ops_release_executor_core.py`
|
||||
- 发布目录不可写时显式失败
|
||||
- 增加 `ExecStart` 与 `current` 软链对齐观测
|
||||
- `domain-api/app/node_agent.py`
|
||||
- 增加 job 级兜底异常回写，避免任务假性卡死
|
||||
- 将卡死的 worker 发布任务纠正为真实失败：
|
||||
- `release_id = 6`
|
||||
- `rollout_id = 5`
|
||||
- `job_id = 204`
|
||||
- 当前已从假 `running` 回正为 `failed`
|
||||
- 本次失败对应的 Release 版本：
|
||||
- `domaincheck_release_20260419_203933`
|
||||
|
||||
## 今晚新增关键结论
|
||||
|
||||
### 1. worker 发布失败的真实原因已明确
|
||||
|
||||
- `mainland-worker-01`
|
||||
- node-agent 日志：
|
||||
- `2026-04-19 20:40:15 [node-agent] loop error: [Errno 13] Permission denied: '/opt/domaincheck/downloads'`
|
||||
- 对应发布任务：
|
||||
- `job 204 = failed`
|
||||
- `rollout 5 = failed`
|
||||
- 当前失败原因已经正式回写：
|
||||
- `install_root not writable: /opt/domaincheck/downloads`
|
||||
|
||||
### 2. mainland 两台节点都不符合当前 ReleaseHub 发布模型
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `domaincheck-worker` 当前启动路径：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- `mainland-controller-01`
|
||||
- `domaincheck-worker` 当前启动路径：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
|
||||
这意味着：
|
||||
|
||||
- 当前线上服务不是跑在
|
||||
- `/opt/domaincheck/current/domainCheck/...`
|
||||
- 所以即使发布成功切了 `current` 软链
|
||||
- 也不会自动让现有 systemd 服务切到新版本
|
||||
|
||||
### 3. worker 当前不是新版本高吞吐样本
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `Active since = 2026-04-19 18:18:50 CST`
|
||||
- `Main PID = 635924`
|
||||
- `CPU = 18.994s`
|
||||
- 最近日志仍停留在旧代码行号：
|
||||
- `_run_detect_register:2465`
|
||||
- `detect_domain:2947`
|
||||
- `start_redis_subscription:3590`
|
||||
|
||||
说明：
|
||||
|
||||
- worker 仍是旧进程
|
||||
- 当前不能把它当成“已切新包并真实参与高吞吐”的有效证据
|
||||
|
||||
## 结论
|
||||
|
||||
当前唯一主阻塞不是前端、不是统计口径、也不是单纯并发参数。
|
||||
|
||||
当前唯一主阻塞是：
|
||||
|
||||
- mainland 节点 `deploy.release` 目录权限不满足
|
||||
- mainland 节点 `domaincheck-worker` 的 systemd `ExecStart` 与 ReleaseHub 的 `current` 发布模型不一致
|
||||
|
||||
在这两个问题修正前：
|
||||
|
||||
- 不建议继续对 mainland 节点做正式 rollout
|
||||
- 不建议继续把 worker 统计口径偏差当成纯展示层问题
|
||||
|
||||
## 下一步唯一建议
|
||||
|
||||
1. 先修 mainland 节点发布基座
|
||||
- 让 node-agent 对 `/opt/domaincheck/downloads`、`/opt/domaincheck/releases` 有写权限
|
||||
- 或明确改成一个 node-agent 真正可写的发布根目录
|
||||
2. 再统一 `domaincheck-worker.service`
|
||||
- 让 `ExecStart` 指向 `/opt/domaincheck/current/domainCheck/detect_worker.py`
|
||||
- 不再直指 `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
3. 完成后重新发起：
|
||||
- worker rollout
|
||||
- 验证 PID 切换
|
||||
- 验证代码行号切换到当前版本
|
||||
- 再看吞吐和 CPU 利用率
|
||||
|
||||
## 本轮改动文件
|
||||
|
||||
- `package_domain_release.sh`
|
||||
- `verify_domain_release.sh`
|
||||
- `package_domain_release.ps1`
|
||||
- `verify_domain_release.ps1`
|
||||
- `domain-api/app/services/ops_release_executor_core.py`
|
||||
- `domain-api/app/node_agent.py`
|
||||
- `docs/ops_center_runtime/TASK_BOARD.md`
|
||||
- `docs/ops_center_runtime/IMPLEMENTATION_STATUS.md`
|
||||
102
docs/ops_center_runtime/HANDOFF_20260419_2117.md
Normal file
102
docs/ops_center_runtime/HANDOFF_20260419_2117.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# HANDOFF_20260419_2117
|
||||
|
||||
更新时间：2026-04-19 21:17 CST
|
||||
|
||||
## 本轮结论
|
||||
|
||||
`mainland-worker-01` 的最新正式 rollout 已完成根因收口：
|
||||
|
||||
- `release_id = 7`
|
||||
- `rollout_id = 6`
|
||||
- `job_id = 211`
|
||||
|
||||
执行结果：
|
||||
|
||||
- 发布包下载成功
|
||||
- `/opt/domaincheck/downloads/domaincheck_release_20260419_205437.tar.gz`
|
||||
- release 解压成功
|
||||
- `/opt/domaincheck/releases/domaincheck_release_20260419_205437`
|
||||
- `current` 软链切换成功
|
||||
- `/opt/domaincheck/current -> /opt/domaincheck/releases/domaincheck_release_20260419_205437`
|
||||
- 最终失败在：
|
||||
- `systemctl restart domaincheck-worker`
|
||||
|
||||
控制面失败结果原文：
|
||||
|
||||
- `restart failed: domaincheck-worker`
|
||||
- `Failed to restart domaincheck-worker.service: Interactive authentication required.`
|
||||
|
||||
## 真实根因
|
||||
|
||||
不是发布包问题，也不是目录权限问题了。
|
||||
|
||||
当前根因是：
|
||||
|
||||
- `domaincheck-node-agent` 运行身份仍是 `www`
|
||||
- 它具备下载、解压、切换 `current` 的权限
|
||||
- 但不具备执行 systemd 服务重启的权限
|
||||
|
||||
所以当前所有以下动作在 mainland 机器上都会有同类风险：
|
||||
|
||||
- `deploy.release`
|
||||
- `service.restart`
|
||||
- 任何依赖 node-agent 直接操作 systemd 的动作
|
||||
|
||||
## 本轮已做代码修正
|
||||
|
||||
已修改：
|
||||
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
- `domain-api/deploy/multi-region/fix_mainland_release_base.sh`
|
||||
- 安装 node-agent drop-in 时补齐：
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
|
||||
已验证：
|
||||
|
||||
- `bash -n domain-api/deploy/multi-region/fix_mainland_release_base.sh`
|
||||
|
||||
## 现场下一步
|
||||
|
||||
下一步不要再重复发起新的 rollout，先把 `mainland-worker-01` 的 node-agent 切到 root。
|
||||
|
||||
建议现场执行：
|
||||
|
||||
```bash
|
||||
mkdir -p /etc/systemd/system/domaincheck-node-agent.service.d
|
||||
cat >/etc/systemd/system/domaincheck-node-agent.service.d/runtime-user.conf <<'EOF'
|
||||
[Service]
|
||||
User=root
|
||||
Group=root
|
||||
EOF
|
||||
systemctl daemon-reload
|
||||
systemctl restart domaincheck-node-agent
|
||||
systemctl status domaincheck-node-agent --no-pager -l
|
||||
```
|
||||
|
||||
完成后应立即复查：
|
||||
|
||||
```bash
|
||||
systemctl show domaincheck-node-agent -p User -p Group
|
||||
journalctl -u domaincheck-node-agent -n 80 --no-pager -l
|
||||
```
|
||||
|
||||
预期结果：
|
||||
|
||||
- `domaincheck-node-agent` 以 `root` 身份运行
|
||||
- 心跳恢复，`mainland-worker-01` 不再是 `stale`
|
||||
- 后续再发起 `deploy.release` 时，能够闭环到 `systemctl restart domaincheck-worker`
|
||||
|
||||
## 当前判断
|
||||
|
||||
当前主阻塞只剩一个：
|
||||
|
||||
- mainland node-agent 权限模型未切到 root
|
||||
|
||||
这个问题修完后，再重新发起 worker rollout，才有资格继续看：
|
||||
|
||||
- `domaincheck-worker` 是否切到新 PID
|
||||
- health check 是否通过
|
||||
- rollout 6 之后的 acceptance 是否可继续
|
||||
95
docs/ops_center_runtime/HANDOFF_20260419_2125.md
Normal file
95
docs/ops_center_runtime/HANDOFF_20260419_2125.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# HANDOFF_20260419_2125
|
||||
|
||||
更新时间：2026-04-19 21:25 CST
|
||||
|
||||
## 本轮结果
|
||||
|
||||
`mainland-worker-01` 的 release rollout 已正式成功收口。
|
||||
|
||||
关键信息：
|
||||
|
||||
- `release_id = 7`
|
||||
- `rollout_id = 7`
|
||||
- `job_id = 214`
|
||||
|
||||
最终状态：
|
||||
|
||||
- `job 214 = success`
|
||||
- `rollout 7 = completed`
|
||||
- `mainland-worker-01.agent_state = online_busy`
|
||||
|
||||
## 成功证据
|
||||
|
||||
控制面 `job events` 已记录：
|
||||
|
||||
- `event_type = agent_completed`
|
||||
- `summary_text = release deployed`
|
||||
|
||||
结果载荷中已确认：
|
||||
|
||||
- `release_version = domaincheck_release_20260419_205437`
|
||||
- `restart_results`
|
||||
- `domaincheck-worker.returncode = 0`
|
||||
- `health_check.ok = true`
|
||||
- `services_checked = ["domaincheck-worker"]`
|
||||
- `execstart_alignment.mismatched_services = []`
|
||||
|
||||
说明这次已经完整走通：
|
||||
|
||||
1. 下载发布包
|
||||
2. 解压 release
|
||||
3. 切换 `/opt/domaincheck/current`
|
||||
4. 重启 `domaincheck-worker`
|
||||
5. 健康检查通过
|
||||
|
||||
## 为什么这次成功
|
||||
|
||||
不是发布逻辑变化了，而是现网权限问题被修掉了。
|
||||
|
||||
上一轮失败根因：
|
||||
|
||||
- `domaincheck-node-agent` 以 `www` 身份运行
|
||||
- 无法执行 `systemctl restart domaincheck-worker`
|
||||
- systemd 返回：
|
||||
- `Interactive authentication required`
|
||||
|
||||
本轮修复：
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `domaincheck-node-agent` 已切为：
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
|
||||
## 当前剩余项
|
||||
|
||||
唯一建议立即同步的预防动作：
|
||||
|
||||
- 在 `mainland-controller-01` 上，也把 `domaincheck-node-agent` 切成 `root`
|
||||
|
||||
原因：
|
||||
|
||||
- worker 已经验证这就是 release 闭环最后一层权限门槛
|
||||
- controller 如果后续参与自身发布 / restart，同样会遇到同类 systemd 权限问题
|
||||
|
||||
建议执行：
|
||||
|
||||
```bash
|
||||
mkdir -p /etc/systemd/system/domaincheck-node-agent.service.d
|
||||
cat >/etc/systemd/system/domaincheck-node-agent.service.d/runtime-user.conf <<'EOF'
|
||||
[Service]
|
||||
User=root
|
||||
Group=root
|
||||
EOF
|
||||
systemctl daemon-reload
|
||||
systemctl restart domaincheck-node-agent
|
||||
systemctl show domaincheck-node-agent -p User -p Group
|
||||
```
|
||||
|
||||
## 当前判断
|
||||
|
||||
关于 mainland worker 的发布链，已经不再是阻塞项。
|
||||
|
||||
现在可以进入下一步：
|
||||
|
||||
- 再决定是否对 `mainland-controller-01` 做同样 root 切换
|
||||
- 然后继续 controller acceptance / rollout 收口
|
||||
563
docs/ops_center_runtime/HANDOFF_20260420_1920.md
Normal file
563
docs/ops_center_runtime/HANDOFF_20260420_1920.md
Normal file
@@ -0,0 +1,563 @@
|
||||
# HANDOFF_20260420_1920
|
||||
|
||||
更新时间：2026-04-20 19:20 CST
|
||||
|
||||
## 本轮结论
|
||||
|
||||
这轮最重要的收口已经完成：
|
||||
|
||||
- 已确认当前工作机 `/www/wwwroot/getDomain` 所在主机是海外测试控制面，不是国内 `mainland-controller-01`
|
||||
- 已把“假冒 mainland-controller-01”的本机配置纠正为 `overseas-control-01`
|
||||
- 已把真正的 `mainland-controller-01` 重新接回控制面，并完成一次成功的远端发布
|
||||
- 已修正 `mainland-controller-01` 的 node-agent 身份上报问题
|
||||
- 现在控制面里看到的 `mainland-controller-01` 已经是真实主机：
|
||||
- `hostname = mainland-controller-01`
|
||||
- `ip = 121.204.244.188`
|
||||
|
||||
一句话说当前状态：
|
||||
|
||||
“主线已经从‘节点身份混乱’推进到‘真 controller 已经对正并重新进场’，下一步应只盯 controller 真机吞吐和 worker 恢复，不要再回展示层。”
|
||||
|
||||
## 当前环境认知
|
||||
|
||||
### 1. 当前 Codex 所在机器不是大陆 controller
|
||||
|
||||
本地执行结果：
|
||||
|
||||
- `hostname = v2202604268673447256`
|
||||
- `nproc = 12`
|
||||
|
||||
这台机器是海外控制面测试机，不是用户说的 112 核 controller。
|
||||
|
||||
真正的国内 controller 是：
|
||||
|
||||
- `node_code = mainland-controller-01`
|
||||
- `ssh_host = 121.204.244.188`
|
||||
|
||||
真正的国内 worker 是：
|
||||
|
||||
- `node_code = mainland-worker-01`
|
||||
- `ssh_host = 121.204.244.248`
|
||||
|
||||
### 2. 本机服务身份已经纠正
|
||||
|
||||
为避免海外测试机继续伪装成 mainland controller，已经改过这些环境文件：
|
||||
|
||||
- `/etc/default/domaincheck-worker`
|
||||
- `/etc/default/domaincheck-node-agent`
|
||||
- `/etc/default/domaincheck-api`
|
||||
|
||||
修正后的核心值：
|
||||
|
||||
- `NODE_CODE=overseas-control-01`
|
||||
- `NODE_REGION=overseas`
|
||||
- `SYNC_PUSH_ENABLED=false`
|
||||
|
||||
并且已经执行过：
|
||||
|
||||
- 停止本机 `domaincheck-worker`
|
||||
- 停止本机 `domaincheck-node-agent`
|
||||
- 重启本机 `domaincheck-api`
|
||||
|
||||
当前海外控制面 API 进程环境已确认是：
|
||||
|
||||
- `NODE_CODE=overseas-control-01`
|
||||
- `NODE_REGION=overseas`
|
||||
|
||||
## 本轮已完成的关键修复
|
||||
|
||||
## A. 修复 controller node-agent 上报 localhost / 127.0.0.1
|
||||
|
||||
修改文件：
|
||||
|
||||
- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py)
|
||||
|
||||
修复内容：
|
||||
|
||||
- `_hostname()` 不再优先相信 `localhost`
|
||||
- `_ip()` 不再使用容易得到 `127.0.0.1` 的旧逻辑
|
||||
- 优先通过控制面地址推导本机出口 IP
|
||||
- 回退时也会过滤 loopback
|
||||
|
||||
本轮新增/补过的测试：
|
||||
|
||||
- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py)
|
||||
|
||||
注意：
|
||||
|
||||
- 当前仓库里这份修复已经在真 controller 上生效
|
||||
- handover 已显示：
|
||||
- `agent_hostname = mainland-controller-01`
|
||||
- `agent_ip = 121.204.244.188`
|
||||
- `cluster_hostname = mainland-controller-01`
|
||||
- `cluster_ip = 121.204.244.188`
|
||||
|
||||
## B. 修复 control 节点发布健康检查窗口过短
|
||||
|
||||
修改文件：
|
||||
|
||||
- [domain-api/app/services/ops_release_service.py](/www/wwwroot/getDomain/domain-api/app/services/ops_release_service.py)
|
||||
|
||||
修改目标：
|
||||
|
||||
- control 节点发布时，`domaincheck-api` 启动偏慢，旧健康检查窗口太短，会误判失败并回滚
|
||||
|
||||
代码里已改成：
|
||||
|
||||
- `health_check_timeout_seconds = 20`
|
||||
- `health_check_retries = 6`
|
||||
- `health_check_interval_seconds = 3`
|
||||
|
||||
但要注意一个坑：
|
||||
|
||||
- 这份代码虽然已改进源码
|
||||
- 海外控制面的运行中 API 进程还没有通过这份新源码重新部署
|
||||
- 所以 `smart-rollout-preview` 里看到的默认值一度仍然是旧的 `10 / 2 / 2`
|
||||
|
||||
因此本轮实际是通过“手工 remote-agent deploy job 显式带长窗口 payload”打通了 controller 发布。
|
||||
|
||||
## C. 真 controller 发布已经成功一次
|
||||
|
||||
成功任务：
|
||||
|
||||
- `job_id = 331`
|
||||
- `job_code = ops-20260420183116-c33723`
|
||||
- `action = deploy.release`
|
||||
- `target_node_code = mainland-controller-01`
|
||||
- `execution_mode = remote-agent`
|
||||
- `status = success`
|
||||
|
||||
本次使用的 release：
|
||||
|
||||
- `release_id = 27`
|
||||
- `release_version = domaincheck_release_20260420_182916`
|
||||
|
||||
发布结果要点：
|
||||
|
||||
- 发布包下载成功
|
||||
- checksum 校验成功
|
||||
- `/opt/domaincheck/current` 切换成功
|
||||
- `domaincheck-api / domaincheck-worker / domaincheck-sync-agent` 重启成功
|
||||
- 健康检查最终通过
|
||||
|
||||
尤其要记住：
|
||||
|
||||
- 这次成功不是靠 smart rollout 默认值
|
||||
- 是通过显式下发以下健康窗口打通的：
|
||||
- `health_check_timeout_seconds = 20`
|
||||
- `health_check_retries = 6`
|
||||
- `health_check_interval_seconds = 3`
|
||||
|
||||
## D. controller node-agent 已经重新连回
|
||||
|
||||
后续又触发了：
|
||||
|
||||
- `job_id = 332`
|
||||
- `action = service.restart`
|
||||
- `payload.service_name = domaincheck-node-agent`
|
||||
|
||||
这个任务本身还停留在 `running`，原因很正常：
|
||||
|
||||
- node-agent 在“执行重启自己”的过程中会打断自身回执链
|
||||
- 所以作业状态可能不会自然收尾
|
||||
|
||||
但实际效果已经发生：
|
||||
|
||||
- API 日志已看到 `121.204.244.188` 在 `2026-04-20 18:35:16` 重新开始：
|
||||
- `POST /api/v1/ops/agent/heartbeat`
|
||||
- `POST /api/v1/ops/agent/pull?limit=1`
|
||||
|
||||
因此这条 job 332 可以视为“结果已生效，但状态未优雅回写”的典型自重启任务。
|
||||
|
||||
## 当前实机状态
|
||||
|
||||
按最新控制面查询：
|
||||
|
||||
### mainland-controller-01
|
||||
|
||||
- `agent_online = true`
|
||||
- `cluster_status = busy`
|
||||
- `agent_hostname = mainland-controller-01`
|
||||
- `agent_ip = 121.204.244.188`
|
||||
- `current_load` 在本轮观察中约 `210 - 233`
|
||||
- `detect_runtime.active_threads` 在本轮观察中约 `210 - 233 / 2000`
|
||||
- 已经有持续日志回传
|
||||
|
||||
### mainland-worker-01
|
||||
|
||||
- node-agent 在线
|
||||
- 但当前没有真正参与检测
|
||||
- 控制面返回的 `detect_runtime` 错误为：
|
||||
- `connection to server at "127.0.0.1", port 5432 failed: Connection refused`
|
||||
|
||||
这意味着：
|
||||
|
||||
- worker 机现在不是主要算力来源
|
||||
- 目前真正吃任务的是 `mainland-controller-01`
|
||||
|
||||
## 当前主线瓶颈
|
||||
|
||||
目前主线已经不是“谁是 controller”了，当前瓶颈明确是下面两个：
|
||||
|
||||
1. `mainland-worker-01` 未恢复到可参与检测状态
|
||||
2. `mainland-controller-01` 虽然真实线程已抬到 200+，但控制面队列视角仍存在：
|
||||
- `claimed` 偏高
|
||||
- `running/completed` 推进不够理想
|
||||
- 吞吐没有完全吃透机器
|
||||
|
||||
也就是说：
|
||||
|
||||
- “节点身份问题”已基本打穿
|
||||
- “发布链路问题”已基本打穿
|
||||
- 下一步该只盯“真 controller 吞吐”和“worker 恢复”
|
||||
|
||||
## 本轮新增发现
|
||||
|
||||
### 1. mainland-worker-01 的 `127.0.0.1:5432` 更像是 node-agent 侧遥测链路问题
|
||||
|
||||
本轮继续排查后，发现这个问题至少有两层：
|
||||
|
||||
1. `domainCheck` 默认配置本身就是：
|
||||
- `DB_HOST = localhost`
|
||||
2. worker 机上的 `domaincheck-node-agent` systemd unit 当前只加载：
|
||||
- `/etc/default/domaincheck-api`
|
||||
- `/etc/default/domaincheck-node-agent`
|
||||
|
||||
但大陆 worker 快速安装脚本真正写入数据库与 Redis 指向的是：
|
||||
|
||||
- `/etc/default/domaincheck-worker`
|
||||
|
||||
也就是说，worker 机上很可能出现这种情况：
|
||||
|
||||
- `domaincheck-worker` 进程拿到的是正确的 `DB_HOST=${MAINLAND_CONTROLLER_IP}`
|
||||
- 但 `domaincheck-node-agent` 没有继承 `/etc/default/domaincheck-worker`
|
||||
- node-agent 内部去跑 `get_detect_status()` 时，就会落回 `domain-api` 默认配置：
|
||||
- `db_host = 127.0.0.1`
|
||||
|
||||
于是控制面上看到的现象就变成：
|
||||
|
||||
- `mainland-worker-01 agent 在线`
|
||||
- 但 `detect_runtime` 里报：
|
||||
- `connection to server at "127.0.0.1", port 5432 failed: Connection refused`
|
||||
|
||||
### 2. node-agent 当前把“DB 查询失败”和“worker 进程离线”混成了一种失败
|
||||
|
||||
`domain-api/app/node_agent.py` 里的 `_detect_runtime_snapshot()` 原来是：
|
||||
|
||||
- `get_detect_status()` 和 `detect_worker_runtime()` 放在同一个总 `try` 里
|
||||
|
||||
这会导致：
|
||||
|
||||
- 只要前面的 DB 查询失败
|
||||
- 后面的 worker systemd 运行态也一起被吞掉
|
||||
- 最终上报成：
|
||||
- `worker_online = false`
|
||||
- `service_running = false`
|
||||
|
||||
即使真实情况其实可能只是：
|
||||
|
||||
- worker 服务还活着
|
||||
- 只是 node-agent 在采集 detect status 时查错 DB 了
|
||||
|
||||
### 3. 本轮已在源码里补的最小修复
|
||||
|
||||
已修改：
|
||||
|
||||
- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py)
|
||||
- [domain-api/deploy/systemd/domain-node-agent.service](/www/wwwroot/getDomain/domain-api/deploy/systemd/domain-node-agent.service)
|
||||
- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py)
|
||||
|
||||
修复内容：
|
||||
|
||||
1. `_detect_runtime_snapshot()` 改成分层降级：
|
||||
- 先拿 `detect_worker_runtime()`
|
||||
- 再单独尝试 `get_detect_status()`
|
||||
- 即使 detect status 因 DB 异常失败，也保留真实的 worker service 运行态
|
||||
2. `domaincheck-node-agent.service` 追加：
|
||||
- `EnvironmentFile=-/etc/default/domaincheck-worker`
|
||||
|
||||
这样 worker 机上的 node-agent 也能直接继承 worker 真实使用的 `DB_HOST / REDIS_HOST`。
|
||||
|
||||
### 4. 这次修复的生效边界
|
||||
|
||||
要特别注意：
|
||||
|
||||
- `node_agent.py` 代码修复可以通过常规 release 进入真实运行目录并生效
|
||||
- 但 `domaincheck-node-agent.service` 属于 `/etc/systemd/system/` 下的系统级 unit 文件
|
||||
- 常规 `deploy.release` 只会切换 `/opt/domaincheck/current`，不会自动重装 systemd unit
|
||||
|
||||
所以这两项应分开看：
|
||||
|
||||
1. `node_agent.py`：
|
||||
- 可以走最小 release 先上真机
|
||||
2. `domain-node-agent.service`：
|
||||
- 需要后续补 systemd unit 落地动作
|
||||
- 至少要包含：
|
||||
- 覆盖 unit 文件或 drop-in
|
||||
- `systemctl daemon-reload`
|
||||
- `systemctl restart domaincheck-node-agent`
|
||||
|
||||
### 5. 本轮测试结果
|
||||
|
||||
已通过的焦点测试：
|
||||
|
||||
- `PYTHONPATH=/www/wwwroot/getDomain/domain-api /opt/domaincheck/domainCheck/.venv/bin/python -m unittest tests.test_node_agent_delivery_queue -v`
|
||||
|
||||
结果：
|
||||
|
||||
- `11 tests`
|
||||
- `OK`
|
||||
|
||||
新增验证点：
|
||||
|
||||
- `test_detect_runtime_snapshot_degrades_to_worker_runtime_when_detect_status_fails`
|
||||
|
||||
它验证了：
|
||||
|
||||
- 即使 `get_detect_status()` 抛出
|
||||
- `connection to server at "127.0.0.1", port 5432 failed`
|
||||
- node-agent 依然会保留：
|
||||
- `worker_online = true`
|
||||
- `service_running = true`
|
||||
- `phase_detail = active/running`
|
||||
|
||||
## 本轮最小发布动作
|
||||
|
||||
### 1. 已生成新发布包
|
||||
|
||||
本轮最小修复已重新打包：
|
||||
|
||||
- `release_version = domaincheck_release_20260420_215734`
|
||||
- `release_id = 28`
|
||||
|
||||
生成结果：
|
||||
|
||||
- `archive = /www/wwwroot/getDomain/release/domaincheck_release_20260420_215734.tar.gz`
|
||||
- `sha256 = ad2c41625503dab1efaeadbf0641c8f6875a3cad0581a3b9c7835ccc37cdc718`
|
||||
|
||||
### 2. 已向真节点发起最小 release 下发
|
||||
|
||||
本轮只为把 `node_agent.py` 先推进真实运行目录，已创建：
|
||||
|
||||
- `job_id = 335`
|
||||
- `target = mainland-controller-01`
|
||||
- `action = deploy.release`
|
||||
- `job_id = 336`
|
||||
- `target = mainland-worker-01`
|
||||
- `action = deploy.release`
|
||||
|
||||
这两条 job 都已经越过“等待拉取”，进入了 agent 执行阶段，并至少记录到：
|
||||
|
||||
- `executor_received`
|
||||
- `deploy_download_started`
|
||||
|
||||
说明：
|
||||
|
||||
- 真 controller 和真 worker 都已经接到了这版最小 release
|
||||
- 当前至少已经在真实节点上进入下载/发布链
|
||||
|
||||
### 3. 这次发布的真实目标
|
||||
|
||||
这次发布不是为了一次性解决所有 worker 恢复问题，而是为了先把下面这条代码修复推到真节点：
|
||||
|
||||
- `domain-api/app/node_agent.py`
|
||||
|
||||
目的：
|
||||
|
||||
- 即使 worker 节点的 `detect_status` 查询因 DB 指向错误失败
|
||||
- node-agent 也不再把真实的 worker service 运行态一并吞掉
|
||||
- 控制面能先看到更接近真实的 `worker_online / service_running`
|
||||
|
||||
### 4. 这次发布的限制
|
||||
|
||||
这次最小 release 暂时还不能自动完成下面这件事：
|
||||
|
||||
- 把 `/etc/systemd/system/domaincheck-node-agent.service` 更新为新版本
|
||||
|
||||
原因：
|
||||
|
||||
- 常规 `deploy.release` 切的是 `/opt/domaincheck/current`
|
||||
- 不会自动重装 systemd unit
|
||||
|
||||
因此当前判断是：
|
||||
|
||||
1. `node_agent.py` 代码修复：
|
||||
- 已经进入真实节点发布链
|
||||
2. `domaincheck-node-agent.service` 的 `EnvironmentFile=-/etc/default/domaincheck-worker`：
|
||||
- 仍需要后续单独补系统级落地动作
|
||||
|
||||
### 5. 当前发布观察结论
|
||||
|
||||
截至本次交接整理时：
|
||||
|
||||
- `job 335 / 336` 已启动执行
|
||||
- 但尚未在本轮记录中拿到最终 `success / failed` 收尾结论
|
||||
- 因此后续接手时，先做的第一件事之一，就是复查这两条 job 的最终状态与事件流
|
||||
|
||||
## 当前不要再踩的坑
|
||||
|
||||
### 1. 不要再把本机当成 mainland-controller-01
|
||||
|
||||
当前工作机是海外测试控制面，只负责：
|
||||
|
||||
- API
|
||||
- 控制面
|
||||
- 同步接收
|
||||
|
||||
不是 112 核大陆 controller。
|
||||
|
||||
### 2. 不要直接用本机 Python service 函数创建 deploy job
|
||||
|
||||
坑点：
|
||||
|
||||
- 直接在 shell 里跑本地 Python 服务函数时，读取到的 `settings.node_code` 可能不是运行中 API 进程的真实环境
|
||||
- 之前就出现过把 job 错判为 `local-runtime` 的情况
|
||||
|
||||
正确做法：
|
||||
|
||||
- 优先通过“正在运行的 API HTTP 接口”创建 job
|
||||
- 不要优先走本机 Python service 入口
|
||||
|
||||
推荐路径：
|
||||
|
||||
- `POST /api/v1/ops/releases/from-package/latest`
|
||||
- `POST /api/v1/ops/jobs`
|
||||
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
|
||||
|
||||
### 3. 不要把 job 332 当成硬故障
|
||||
|
||||
`job 332` 是 `service.restart domaincheck-node-agent`。
|
||||
|
||||
它卡在 `running` 不代表没生效，反而更像：
|
||||
|
||||
- node-agent 重启了自己
|
||||
- 回执链没能把 job 收尾
|
||||
|
||||
判断是否真的生效，应看：
|
||||
|
||||
- API 日志里有没有来自 `121.204.244.188` 的 heartbeat/pull
|
||||
- handover 里 `agent_hostname / agent_ip` 是否已刷新
|
||||
|
||||
### 4. 不要再回展示层
|
||||
|
||||
当前最值钱的推进方向仍然是：
|
||||
|
||||
- controller 真机吞吐
|
||||
- claim/running/completed 推进
|
||||
- worker 恢复
|
||||
|
||||
不要把额度再耗在页面、文案、展示层结构上。
|
||||
|
||||
## 这轮动过的关键文件
|
||||
|
||||
最关键的源码文件：
|
||||
|
||||
- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py)
|
||||
- [domain-api/app/services/ops_release_service.py](/www/wwwroot/getDomain/domain-api/app/services/ops_release_service.py)
|
||||
- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py)
|
||||
|
||||
此外，工作区还存在大量其他未提交改动，不要随意回滚：
|
||||
|
||||
- `domain-api/`
|
||||
- `domain-web/`
|
||||
- `domainCheck/`
|
||||
- `docs/ops_center_runtime/`
|
||||
|
||||
这是一个脏工作区，接手时必须小心，不要用破坏性 git 命令。
|
||||
|
||||
## 建议下一个 Codex 只做的两件事
|
||||
|
||||
### 任务 1：恢复 mainland-worker-01
|
||||
|
||||
目标：
|
||||
|
||||
- 让 `mainland-worker-01` 从“agent 在线但不参与检测”恢复到可执行检测
|
||||
|
||||
先查方向：
|
||||
|
||||
- 为什么它在本地检测态里访问 `127.0.0.1:5432` 失败
|
||||
- 是本机 PostgreSQL 没启
|
||||
- 还是 worker 的运行配置仍然错误指向本地 DB
|
||||
- 还是它本来就不该走本地 PostgreSQL，而应走远端/统一控制链
|
||||
|
||||
验收标准：
|
||||
|
||||
- `mainland-worker-01.detect_runtime.worker_online = true`
|
||||
- `mainland-worker-01.detect_runtime.active_threads > 0`
|
||||
- 能稳定进入参与节点
|
||||
|
||||
### 任务 2：继续抬 mainland-controller-01 真吞吐
|
||||
|
||||
目标：
|
||||
|
||||
- 只盯真 controller 的任务推进链，不碰页面
|
||||
|
||||
重点看：
|
||||
|
||||
- `claimed -> running -> completed` 是否持续推进
|
||||
- `items_claimed` 是否能下降
|
||||
- `processed_recent / processed_per_minute` 是否能抬起来
|
||||
- `active_threads` 与 `completed` 是否成正相关
|
||||
|
||||
验收标准：
|
||||
|
||||
- controller 持续有真实 completed 增长
|
||||
- running/claimed 更贴近“持续补位”而不是堆积
|
||||
- 机器性能使用能继续往上抬
|
||||
|
||||
## 可直接复用的验证点
|
||||
|
||||
### 1. 看节点 handover
|
||||
|
||||
接口：
|
||||
|
||||
- `GET /api/v1/ops/nodes/mainland-controller-01/handover`
|
||||
|
||||
这一项已经能确认：
|
||||
|
||||
- 当前是不是对到了真 controller
|
||||
- agent/ip/hostname 是否正确
|
||||
- 当前 load / detect_runtime 是否在动
|
||||
|
||||
### 2. 看节点列表
|
||||
|
||||
接口：
|
||||
|
||||
- `GET /api/v1/ops/nodes`
|
||||
|
||||
重点字段：
|
||||
|
||||
- `agent_hostname`
|
||||
- `agent_ip`
|
||||
- `cluster_hostname`
|
||||
- `cluster_ip`
|
||||
- `detect_runtime.active_threads`
|
||||
- `current_load`
|
||||
|
||||
### 3. 看海外控制面 API 日志
|
||||
|
||||
已验证有效：
|
||||
|
||||
- `journalctl -u domaincheck-api -n 200 --no-pager`
|
||||
|
||||
尤其可以筛：
|
||||
|
||||
- `121.204.244.188`
|
||||
- `/api/v1/ops/agent/heartbeat`
|
||||
- `/api/v1/ops/agent/pull`
|
||||
|
||||
### 4. 发布 controller 的正确方式
|
||||
|
||||
推荐继续使用运行中的 API HTTP 接口，而不是本地 Python service 调用。
|
||||
|
||||
## 交接结论
|
||||
|
||||
这一轮真正解决掉的，不是“性能问题”本身，而是它前面最大的认知阻塞：
|
||||
|
||||
- 之前一直有一部分判断建立在“本机就是 controller”这个错误前提上
|
||||
- 现在这个前提已经纠正
|
||||
- 真 controller 已经重新纳入控制面，而且身份上报正确、远端发布成功、线程也已经真实抬起来
|
||||
|
||||
接手人从这里继续时，应该把主线收缩成一句话：
|
||||
|
||||
“只盯 `mainland-controller-01` 的真实吞吐推进，并恢复 `mainland-worker-01`，不要再回展示层，也不要再把海外 12 核测试机当成主算力机。”
|
||||
@@ -1,23 +1,189 @@
|
||||
# IMPLEMENTATION_STATUS
|
||||
|
||||
更新时间：2026-04-19 03:41 CST
|
||||
更新时间：2026-04-19 21:25 CST
|
||||
|
||||
## 当前真实状态
|
||||
|
||||
阶段判断：
|
||||
|
||||
- 海外单脑接管能力：约 `95%`
|
||||
- 分布式检测真实执行能力：约 `88%~90%`
|
||||
- 距离“可稳定上线并放心用后台发起检测”：约 `86%~89%`
|
||||
- 海外单脑接管能力：约 `97%`
|
||||
- 发布闭环真实可用度：约 `65%~70%`
|
||||
- 分布式检测真实执行能力：约 `80%~85%`
|
||||
- 距离“可稳定上线并放心用后台发起检测”：约 `78%~82%`
|
||||
|
||||
## 21:17 最新校正
|
||||
|
||||
刚完成的 `mainland-worker-01` 正式 rollout 已给出最终根因：
|
||||
|
||||
- `job 211 / rollout 6 / release 7`
|
||||
- 发布包下载成功
|
||||
- release 解压成功
|
||||
- `current` 软链切换成功
|
||||
- 最终失败在服务重启：
|
||||
- `restart failed: domaincheck-worker`
|
||||
- `Failed to restart domaincheck-worker.service: Interactive authentication required.`
|
||||
|
||||
这说明：
|
||||
|
||||
- 当前 ReleaseHub 主链路本身已可推进到“切换 current”
|
||||
- 真正未闭环的是 mainland node-agent 的 systemd 权限
|
||||
- 只要 node-agent 仍以 `www` 运行，后续 `deploy.release / service.restart` 都会在 systemd 这一跳失败
|
||||
|
||||
已完成的代码侧修正：
|
||||
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- 改为 `User=root`
|
||||
- 改为 `Group=root`
|
||||
- `domain-api/deploy/multi-region/fix_mainland_release_base.sh`
|
||||
- 改为给 node-agent drop-in 写入 `User=root` / `Group=root`
|
||||
|
||||
因此当前真实上线完成度需要再补一句校正：
|
||||
|
||||
- 发布闭环真实可用度：
|
||||
- 不是“发布包模型还没修”
|
||||
- 而是“node-agent 权限模型还差最后一次现网切换”
|
||||
|
||||
## 21:25 最新进展
|
||||
|
||||
worker 侧的现网切换已经完成验证成功：
|
||||
|
||||
- `mainland-worker-01`
|
||||
- 已把 `domaincheck-node-agent` 切为 `root`
|
||||
- 随后重新发起：
|
||||
- `rollout_id = 7`
|
||||
- `job_id = 214`
|
||||
- 最终结果：
|
||||
- `job 214 = success`
|
||||
- `rollout 7 = completed`
|
||||
|
||||
正式成功证据：
|
||||
|
||||
- `agent_completed`
|
||||
- `summary_text = release deployed`
|
||||
- `restart_results[domaincheck-worker].returncode = 0`
|
||||
- `health_check.ok = true`
|
||||
- `systemd ExecStart` 已对齐：
|
||||
- `/opt/domaincheck/current/domainCheck/detect_worker.py`
|
||||
|
||||
因此当前关于发布闭环的真实判断应更新为：
|
||||
|
||||
- worker 发布闭环：
|
||||
- 已从“卡在 systemd restart 权限”推进到“真实成功”
|
||||
- 当前剩余风险不再在 worker
|
||||
- 当前剩余预防项在 controller：
|
||||
- `domaincheck-node-agent` 也应同步切为 `root`
|
||||
|
||||
## 顶部校正
|
||||
|
||||
今天晚上的最新结论，需要覆盖前面一部分偏乐观判断：
|
||||
|
||||
- 最新发布包问题已经定位并修复：
|
||||
- 之前发布包漏掉了 `domainCheck/`
|
||||
- 现在最新签收包 `domaincheck_release_20260419_205437` 已经包含 `domainCheck/`
|
||||
- 但对 `mainland-worker-01` 的正式发布验证证明：
|
||||
- 线上 mainland 节点目前还不满足现有 ReleaseHub 发布模型
|
||||
- 已确认的真实阻塞有两个：
|
||||
- `deploy.release` 在 `mainland-worker-01` 上会因为
|
||||
- `Permission denied: /opt/domaincheck/downloads`
|
||||
- 而直接失败
|
||||
- mainland 两台节点当前 `domaincheck-worker` 的 `ExecStart` 都直接指向：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- 而不是 `/opt/domaincheck/current/domainCheck/detect_worker.py`
|
||||
|
||||
这两个事实组合起来说明：
|
||||
|
||||
- 当前发布链虽然在控制面上已经成型
|
||||
- 但 mainland 线上节点的安装形态还是旧模式
|
||||
- 所以当前不能再把“已能稳定 rollout 新版本”算进上线完成度
|
||||
|
||||
## 今晚新增硬证据
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `job 204 / rollout 5` 已正式失败回写
|
||||
- 失败原因已收口为：
|
||||
- `install_root not writable: /opt/domaincheck/downloads`
|
||||
- 只读核验得到的实际服务状态仍是旧进程：
|
||||
- `Active since Sun 2026-04-19 18:18:50 CST`
|
||||
- `Main PID = 635924`
|
||||
- `CPU = 18.994s`
|
||||
- 说明 worker 当前并没有切到新包，也没有形成可信的新吞吐样本
|
||||
- `mainland-controller-01`
|
||||
- 当前 `domaincheck-worker` 虽然在高负载运行
|
||||
- 但服务启动路径同样是：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- 说明 controller 也没有对齐 `current` 软链发布模型
|
||||
|
||||
## 本轮代码侧新增收口
|
||||
|
||||
- 已补齐发布包构建：
|
||||
- `package_domain_release.sh`
|
||||
- `verify_domain_release.sh`
|
||||
- `package_domain_release.ps1`
|
||||
- `verify_domain_release.ps1`
|
||||
- 现在发布包会包含 `domainCheck/`
|
||||
- 已补齐发布执行器的失败可观测性：
|
||||
- `domain-api/app/services/ops_release_executor_core.py`
|
||||
- 现在会在发布目录不可写时显式返回失败结果
|
||||
- 同时补充 `ExecStart` 与 `current` 软链的对齐观测
|
||||
- 已补齐 node-agent 的兜底失败回写：
|
||||
- `domain-api/app/node_agent.py`
|
||||
- 避免以后再出现任务已经炸掉但控制面一直卡在 `running`
|
||||
|
||||
## 本轮最新结论
|
||||
|
||||
这轮结论需要更新为四段：
|
||||
新增结论：
|
||||
|
||||
- 接管与同步能力已经明显趋于完成
|
||||
- worker 控制消息补偿链已经完成线上验证
|
||||
- controller 运行环境漂移已经被现场修正
|
||||
- 当前唯一剩余主阻塞已经收紧到 controller 代理池无可用代理
|
||||
- 检测页观察面已完成一轮线上收口：
|
||||
- `任务日志控制台` 已移到事件表上方
|
||||
- `检测事件流` 已改成独立滚动区
|
||||
- 已在真实线上静态目录重建生产包并通过本机 `3201` 端口验证
|
||||
- 当前如果用户仍看到旧布局，优先判断为浏览器缓存而不是部署未生效
|
||||
- 代理链已完成策略切换：
|
||||
- worker 启动后会主动刷新代理池
|
||||
- 配置更新后也会主动刷新代理池
|
||||
- 后台已能区分：
|
||||
- `等待首刷`
|
||||
- `proxy_validation_zero`
|
||||
- `直入池`
|
||||
- 当前本机最新真实结果是：
|
||||
- 原始代理 `270`
|
||||
- 预验证 `0`
|
||||
- 直入池 `270`
|
||||
- 说明当前问题已经从“代理校验过严导致池为空”切换为“真实任务内动态淘汰失效代理”
|
||||
|
||||
- `mainland-controller-01`
|
||||
- 已确认解除“代理不足即同步刷新、反向压死并发”的旧限制
|
||||
- 海外后台当前已能看到：
|
||||
- `active_threads = 99~100`
|
||||
- `max_threads = 100`
|
||||
- 说明 controller 的 `100` 并发已经真实生效
|
||||
- `mainland-worker-01`
|
||||
- 已重新部署最新版 `detect_worker.py`
|
||||
- 已重新接收新的 `sync-pull` 批次并重进检测：
|
||||
- `开始执行域名检测任务`
|
||||
- `开始检测，刷新代理池`
|
||||
- `代理池刷新完成，共 23 个可用代理`
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- 当前 worker 进程本机线程量已到：
|
||||
- `152` 个 OS 线程
|
||||
- 说明它并不是没启动，而是已经进入新批次执行
|
||||
- 当前剩余偏差：
|
||||
- overseas 后台对 `mainland-worker-01.active_threads/current_load` 的显示仍偏低
|
||||
- 这已经从“真实执行问题”收敛为“运行态上报 / 显示口径问题”
|
||||
|
||||
这一轮不是只修了显示问题，而是把“大陆节点真实执行 + worker 日志回传”一起补齐了。
|
||||
|
||||
当前最新结论：
|
||||
|
||||
- controller 的 `pull_tasks -> 本地队列` 断点已经修通
|
||||
- 两台大陆 worker 都已经进入真实镜像队列执行
|
||||
- `mainland-worker-01` 的 `worker_log` 已开始直接回灌 overseas `detect_debug_events`
|
||||
- `runtime_projection` 权限问题已经修复，后台运行态同步恢复
|
||||
- `/api/v1/ops/nodes` 已能证明 controller 的高并发真实生效
|
||||
- 当前剩余问题已经收敛为：
|
||||
- `mainland-worker-01` 运行态显示口径仍需继续对齐
|
||||
- 检测页主计数口径是否完全跟上
|
||||
- 结果统计回推是否持续稳定
|
||||
|
||||
## 当前证据拆分
|
||||
|
||||
@@ -25,210 +191,273 @@
|
||||
|
||||
已经完成：
|
||||
|
||||
- `detect_result_projection` 支持 `recent_domain_events`
|
||||
- 中央 ingest 会把逐条事件写入 `detect_run_events`
|
||||
- `sync_agent` 会自动产出 `detect_result_projection`
|
||||
- worker 控制消息新增 `request_id`
|
||||
- worker 运行态心跳会补偿消费 pending 控制消息
|
||||
- worker 收到并处理控制消息后会按 `request_id` 清理 pending 指令
|
||||
- `sync_push_service.ingest_detect_task_projection`
|
||||
- 不再只导入 `domains`
|
||||
- 已改为同时创建本地 `detect_jobs`
|
||||
- 已改为同时创建本地 `detect_job_items`
|
||||
- 已把 `target_job_id / target_job_code / queued_count / worker_start_ok` 回写到同步结果
|
||||
- `sync_agent`
|
||||
- controller 现在会在每轮 `pull_tasks` 后自动唤起本地 worker
|
||||
- `detect_worker._set_active_cycle_context`
|
||||
- 已兼容 `sync-pull` 控制消息
|
||||
- 不再只读取 `job_id / job_code`
|
||||
- 现在会回退读取 `target_job_id / target_job_code`
|
||||
- 运行态同步链
|
||||
- controller `runtime/detect_runs.json` 权限已修正为 `www:www`
|
||||
- `runtime_projection` 已恢复成功推送
|
||||
|
||||
本地代码验证已通过：
|
||||
|
||||
- `unittest domain-api/tests/test_worker_control_service.py`
|
||||
- `python -m py_compile domainCheck/detect_worker.py domain-api/app/services/worker_control_service.py`
|
||||
- `python -m py_compile domain-api/app/services/sync_push_service.py`
|
||||
- `python -m py_compile domain-api/app/services/runtime_control_service.py`
|
||||
- `python -m py_compile domain-api/app/sync_agent.py`
|
||||
- `python -m py_compile domainCheck/detect_worker.py`
|
||||
|
||||
线上运行验证也已经出现正向证据：
|
||||
### 2. 接管 / 外部条件闭环
|
||||
|
||||
- `mainland-worker-01`
|
||||
- 启动后发现待执行控制指令
|
||||
- 接受 `start_detection`
|
||||
- 开始执行远程检测任务
|
||||
- `mainland-controller-01`
|
||||
- 修正 Redis 环境后
|
||||
- 重新接受 `start_detection`
|
||||
- 开始执行远程检测任务
|
||||
|
||||
### 2. 接管/同步闭环
|
||||
|
||||
当前已完成：
|
||||
已经完成：
|
||||
|
||||
- `remote_access_ready = 3/3`
|
||||
- `log_sync_state = full_capture`
|
||||
- `mainland-controller-01` 的 `domaincheck-sync-agent` 已重启到新进程
|
||||
- 首轮 `detect_result_projection` 推送成功过一次
|
||||
- 中央已收到 mainland 首批 `domain_*` 事件
|
||||
- `ssh_ready = 2`
|
||||
- 大陆 controller / worker 都可远程运维
|
||||
- controller `domaincheck-sync-agent` 在线
|
||||
- mainland 两台 `domaincheck-node-agent` 在线
|
||||
|
||||
这说明：
|
||||
当前仍依赖你额外输入的部分：
|
||||
|
||||
- mainland 到中央的基础同步链是活的
|
||||
- 结果投影链至少成功打通过一次
|
||||
- 暂无新的 SSH / 接管前置输入缺口
|
||||
- 如果页面仍显示旧状态，最多只需要你浏览器强刷确认，不需要再补 SSH 信息
|
||||
|
||||
### 2.1 前端部署闭环
|
||||
|
||||
已经完成：
|
||||
|
||||
- 在线静态根目录确认：
|
||||
- `/www/wwwroot/getDomain/domain-web/dist`
|
||||
- 在线 Nginx 配置确认：
|
||||
- `/www/server/panel/vhost/nginx/domaincheck_3201.conf`
|
||||
- `/www/server/panel/vhost/nginx/domaincheck_152.53.37.118.conf`
|
||||
- 生产包重建确认：
|
||||
- `vite build` 成功
|
||||
- 线上资源确认：
|
||||
- `DetectView-B9S1WMRT.js`
|
||||
- `DetectView-BFxMNRNR.css`
|
||||
|
||||
说明：
|
||||
|
||||
- 检测页布局调整不是只停留在源码
|
||||
- 已经进入线上可访问静态产物
|
||||
|
||||
### 2.2 代理运行态闭环
|
||||
|
||||
已经完成：
|
||||
|
||||
- worker 启动即触发代理池首刷
|
||||
- 配置变更后自动补刷
|
||||
- API 状态页已能准确显示：
|
||||
- `proxy_not_refreshed_yet`
|
||||
- `proxy_validation_zero`
|
||||
- `宽松入池`
|
||||
|
||||
当前最新证据：
|
||||
|
||||
- `proxy_last_refresh_status = 直入池 270 个（跳过预验证）`
|
||||
- `proxy_last_refresh_total_items = 270`
|
||||
- `proxy_last_validated_count = 0`
|
||||
- `proxy_last_available_count = 270`
|
||||
|
||||
说明：
|
||||
|
||||
- 代理配置本身已成功下发
|
||||
- 代理源接口本身也能返回大量原始代理
|
||||
- 当前不再把“预验证是否通过”作为入池门槛
|
||||
- 现在改为让真实检测来完成失效代理淘汰
|
||||
|
||||
### 3. 检测执行闭环
|
||||
|
||||
当前未完成：
|
||||
这一块现在已经从“怀疑恢复”进入“确认恢复”：
|
||||
|
||||
- 短观察窗口内：
|
||||
- `progress_percent` 仍是 `2.1`
|
||||
- `items_completed` 仍是 `21`
|
||||
- `items_claimed` 仍是 `34`
|
||||
- `items_pending` 仍是 `931`
|
||||
- 中央 recent events 已刷新到更晚时间
|
||||
- `runtime/sync-summary` 最新记录已继续增长
|
||||
- 但 completed 尚未继续上涨
|
||||
- `mainland-controller-01`
|
||||
- 已出现：
|
||||
- `从任务队列获取到 800 个需要检测的域名`
|
||||
- `最大线程数: 100`
|
||||
- `当前实际线程数量: 1/100 ... 6/100`
|
||||
- `mainland-worker-01`
|
||||
- 已出现：
|
||||
- `从任务队列获取到 400 个需要检测的域名`
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `当前实际线程数量: 1/50 ... 3/50`
|
||||
- `开始检测域名: 0-demagogo.com`
|
||||
- `开始检测域名: 00123321.com`
|
||||
- `开始检测域名: 001dm.com`
|
||||
|
||||
这说明：
|
||||
|
||||
- 当前不是单纯“页面没刷新”
|
||||
- 而是执行现场这段时间没有继续出结果
|
||||
- `100/50` 已不是“配置已写入但没生效”
|
||||
- 它已经进入真实任务消费
|
||||
- worker 的远端日志也已经跟着真实执行一起回传
|
||||
|
||||
### 4. 海外后台可视证据
|
||||
|
||||
`/api/v1/ops/nodes` 当前已显示：
|
||||
|
||||
- `summary.remote_access_ready = 3`
|
||||
- `summary.participating = 3`
|
||||
- `summary.dispatch_active = 1`
|
||||
|
||||
并且大陆两台都已经被判定为真实参与者：
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `agent_state = online_busy`
|
||||
- `participation_state = recent_throughput`
|
||||
- `processed_recent = 115`
|
||||
- `processed_per_minute = 7.67`
|
||||
- `detect_runtime.max_threads = 100`
|
||||
- `mainland-worker-01`
|
||||
- `agent_state = online_busy`
|
||||
- `participation_state = recent_throughput`
|
||||
- `processed_recent = 632`
|
||||
- `processed_per_minute = 42.13`
|
||||
- `detect_runtime.max_threads = 50`
|
||||
|
||||
这说明：
|
||||
|
||||
- 海外后台已经不只是看到“在线”
|
||||
- 现在已经能看到大陆节点的实际吞吐
|
||||
|
||||
## 本轮新增硬证据
|
||||
|
||||
通过中央观测面、节点现场日志和远端 `domaincheck-worker` 日志，已确认：
|
||||
### 证据 1：controller 的镜像队列已经真正落库
|
||||
|
||||
- `mainland-controller-01` 现场日志显示：
|
||||
- `当前可用代理数: 0`
|
||||
- `最近结果: 刷新成功，可用 0 个`
|
||||
- controller 新增远端日志显示：
|
||||
- 代理源拉取成功
|
||||
- 抽样校验后 `共 0 个可用代理`
|
||||
- 失败集中在：
|
||||
- `ProxyError@https://m.baidu.com`
|
||||
- `Unable to connect to proxy`
|
||||
- `ConnectTimeoutError`
|
||||
- worker 新增远端日志显示：
|
||||
- `发现待执行 Worker 控制指令`
|
||||
- `已接受检测启动指令`
|
||||
- `开始执行远程检测任务`
|
||||
- 说明线上补偿消费链已真正工作
|
||||
- controller 新增远端日志显示：
|
||||
- 初次重启后：
|
||||
- `Authentication required`
|
||||
- `maximum recursion depth exceeded`
|
||||
- 进一步排查确认：
|
||||
- `/etc/default/domaincheck-worker` 的 `REDIS_PASSWORD` 为空
|
||||
- 修正后再次重启：
|
||||
- `Redis 连接成功: 127.0.0.1:6379`
|
||||
- `已接受检测启动指令`
|
||||
- `开始执行远程检测任务`
|
||||
- 后续日志继续收紧到：
|
||||
- `代理已启用，但当前无可用代理`
|
||||
- 两台大陆节点 full capture 已开启，但源日志时间没有继续前进
|
||||
`sync-agent` 现场日志已出现：
|
||||
|
||||
这说明：
|
||||
- `target_job_code = sync-overseas-7380`
|
||||
- `queued_count = 200`
|
||||
- `worker_start_ok = True`
|
||||
|
||||
- worker 控制消息链不再是主阻塞
|
||||
- controller Redis 环境漂移也不再是主阻塞
|
||||
- 当前第一主阻塞已经进一步收紧到 controller 代理池不可用
|
||||
- worker 的时光机异常是客观存在的次级问题
|
||||
- 不是控制面未接管
|
||||
- 不是同步链未打通
|
||||
并持续产生：
|
||||
|
||||
## 本轮新增代码修复
|
||||
- `sync-overseas-7383`
|
||||
- `sync-overseas-7386`
|
||||
- `sync-overseas-7392`
|
||||
- `sync-overseas-7417`
|
||||
|
||||
本轮不是只停留在诊断，还补了一处运行态最小修复：
|
||||
说明：
|
||||
|
||||
### 修复点 1：控制消息唯一标识
|
||||
- 大陆 controller 现在不是只拉数据
|
||||
- 而是在持续形成可执行批次
|
||||
|
||||
- 文件：
|
||||
- `domain-api/app/services/worker_control_service.py`
|
||||
- 变更：
|
||||
- 每次 `send_worker_command(...)` 都附带 `request_id`
|
||||
- Redis `publish` 与 pending fallback 使用同一份消息体
|
||||
### 证据 2：两台大陆 worker 都已转入真实队列
|
||||
|
||||
### 修复点 2：worker 运行态补偿消费 pending 指令
|
||||
现场日志已明确出现：
|
||||
|
||||
- 文件：
|
||||
- `domainCheck/detect_worker.py`
|
||||
- 变更：
|
||||
- 心跳线程每轮会补偿尝试消费 pending 控制消息
|
||||
- 解决“worker 在线但 pubsub 消息漏收，导致 pending 指令长期不被消费”的风险
|
||||
- controller：
|
||||
- `从任务队列获取到 800 个需要检测的域名`
|
||||
- worker：
|
||||
- `从任务队列获取到 400 个需要检测的域名`
|
||||
|
||||
### 修复点 3：worker 收到指令后确认清理 pending
|
||||
说明：
|
||||
|
||||
- 文件：
|
||||
- `domainCheck/detect_worker.py`
|
||||
- 变更：
|
||||
- worker 实际收到控制消息后，会按 `request_id` 清理对应 pending 指令
|
||||
- 避免修复后又产生重复回放
|
||||
- 当前不是兼容旧链路假运行
|
||||
- 是镜像队列真运行
|
||||
|
||||
### 当前判断
|
||||
### 证据 3：runtime_projection 已恢复成功
|
||||
|
||||
这组修复解决的是：
|
||||
修复前：
|
||||
|
||||
- “检测启动已经发布，但 worker 可能静默漏收”的代码风险
|
||||
- `refresh runtime_projection failed: [Errno 13] Permission denied`
|
||||
|
||||
所以当前最准确的状态是：
|
||||
修复后：
|
||||
|
||||
- 代码级控制链缺口已补上
|
||||
- 线上部署验证已通过
|
||||
- 但 `controller` 代理池校验后 `0 available` 的现场阻塞仍然存在
|
||||
- 最新 `sync-agent` 日志已出现：
|
||||
- `sync_state = success`
|
||||
- `runtime_projection -> 投影推送成功`
|
||||
|
||||
说明：
|
||||
|
||||
- 后台运行态刷新和日志窗口不再被 controller 本地权限卡死
|
||||
|
||||
### 证据 4：`mainland-worker-01` 的 `worker_log` 已进入 overseas
|
||||
|
||||
`/api/v1/runtime/debug-events` 当前已能看到：
|
||||
|
||||
- `mainland-worker-01 -> 开始执行检测任务，来源: redis-control`
|
||||
- `mainland-worker-01 -> 开始执行域名检测任务，正在加载配置`
|
||||
- `mainland-worker-01 -> 开始检测，正在刷新代理池`
|
||||
- `mainland-worker-01 -> 代理池刷新完成，共 7 个可用代理，来源链接 6 个，原始 265 个，验证 150 个`
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 1/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 2/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 3/50`
|
||||
|
||||
说明：
|
||||
|
||||
- worker 的真实执行链已经进入 overseas 的日志视图
|
||||
- “后台看不到大陆 worker 在干活”这一层已经闭合
|
||||
|
||||
## 当前口径说明
|
||||
|
||||
这里有一个非常关键的判断口径已经变化：
|
||||
|
||||
- 大陆节点当前执行的是“海外任务在大陆本地落镜像队列，再把运行态和结果同步回海外”
|
||||
- 因此不能再只盯中央原生 `detect_job_items.claimed/running`
|
||||
- 现在应该同时看：
|
||||
- `/api/v1/ops/nodes`
|
||||
- `processed_recent`
|
||||
- `processed_per_minute`
|
||||
- `detect_runtime.active_threads/max_threads`
|
||||
- 大陆 controller 本地 `sync-overseas-*` 队列
|
||||
|
||||
如果只看中央原生队列，会误判“大陆没有参与”。
|
||||
|
||||
## 当前已闭合的问题
|
||||
|
||||
已闭合：
|
||||
|
||||
- 大陆 controller 无法 `pull_tasks`
|
||||
- Detect 页面误判“大陆没跑”
|
||||
- Detect 主统计口径不一致
|
||||
- Runtime / Queue / Detect 主摘要不统一
|
||||
- 中央逐条事件接收缺口
|
||||
- `sync_agent` 自动产出结果投影缺口
|
||||
- controller `pull_tasks` 只导入 domain、不导入本地队列
|
||||
- 大陆 worker 长时间卡在旧检测会话导致忽略新启动指令
|
||||
- controller `runtime_projection` 因文件属主错误无法推送
|
||||
- 大陆两台“在线但不确定是否真实执行”的状态模糊
|
||||
|
||||
## 当前唯一剩余问题
|
||||
## 当前剩余问题
|
||||
|
||||
当前唯一主问题仍然是：
|
||||
当前剩余问题已经缩成两个最小项：
|
||||
|
||||
- 检测执行面没有恢复到持续产出
|
||||
|
||||
但现在已经不需要再拆成“代码待验证”和“现场阻塞”两层。
|
||||
|
||||
当前唯一剩余现场阻塞就是：
|
||||
|
||||
- controller 代理池可用性为 0
|
||||
- 因此没有持续产生新的 domain 级结果
|
||||
- Detect 页面主计数 `pending/completed/running` 是否继续推进
|
||||
- 结果统计回推是否稳定，而不只是日志链路先恢复
|
||||
|
||||
换句话说：
|
||||
|
||||
- 现在不是逻辑未实现
|
||||
- 不是中央映射失败
|
||||
- 不是节点未接管
|
||||
- 而是执行现场没有继续产出，且 controller 侧卡在代理校验失败
|
||||
|
||||
## acceptance 当前状态
|
||||
|
||||
最新接管验收结果仍然成立：
|
||||
|
||||
- `pbr-9ce5c85f17`：`onboarding.acceptance` 成功
|
||||
- `pbr-389abd618c`：`onboarding.acceptance` 成功
|
||||
|
||||
旧的 `attention` run 依然是历史残留，但它们已经不是当前最真实的生产阻塞。
|
||||
- 现在不是接管问题
|
||||
- 不是链路不通问题
|
||||
- 不是日志完全回不来问题
|
||||
- 而是最后一层“主计数推进 + 结果统计收口”问题
|
||||
|
||||
## 当前是否可以继续跑检测测试
|
||||
|
||||
当前结论：
|
||||
|
||||
- 可以继续做最小运行态排查
|
||||
- 但不需要再优先验证 worker 控制消息链
|
||||
- 当前还不能把状态视作“后台检测已经稳定恢复”
|
||||
- 可以继续跑检测测试
|
||||
- 而且现在已经具备“后台可观测 + 大陆真实参与”的条件
|
||||
|
||||
## 当前是否建议直接上线
|
||||
|
||||
当前结论：
|
||||
|
||||
- 不建议现在按“可稳定上线”判断
|
||||
- 已接近可上线状态
|
||||
- 但我仍建议把它视为“准上线收口态”，不是最终完全签收态
|
||||
|
||||
原因不是接管面，而是执行面：
|
||||
原因：
|
||||
|
||||
- 三台节点都已接入
|
||||
- worker / controller 都已重新接上控制链
|
||||
- 但当前任务没有持续吞吐
|
||||
- controller 代理池全部验不过会直接影响检测产出
|
||||
- 主执行链已经恢复
|
||||
- 但还需要再确认 1 到 2 个同步周期内页面口径与吞吐稳定性
|
||||
|
||||
## 当前优先级判断
|
||||
|
||||
最高优先级：
|
||||
|
||||
- `J2-检测执行停滞收口批`
|
||||
- `J2-检测执行收口批`
|
||||
|
||||
当前不应继续推进：
|
||||
|
||||
@@ -236,18 +465,19 @@
|
||||
- 新模块
|
||||
- 新页面
|
||||
- 发布动作
|
||||
- 与检测执行停滞无关的工作
|
||||
- 与检测执行收口无关的工作
|
||||
|
||||
## 完成下一轮后的预期
|
||||
|
||||
如果下一轮确认：
|
||||
|
||||
- controller 代理池恢复可用
|
||||
- `domain_*` 开始继续增长
|
||||
- `items_completed` 和近窗吞吐重新前进
|
||||
- Detect 页面主计数已跟上
|
||||
- 日志窗口持续刷新
|
||||
- `detect_result_projection` 持续推进
|
||||
- `processed_recent / processed_per_minute` 持续增长
|
||||
|
||||
则整体可上线程度预计可回升到：
|
||||
则整体可上线程度预计可提升到：
|
||||
|
||||
- `92%~94%`
|
||||
- `96%~98%`
|
||||
|
||||
在那之前，当前口径应保持保守。
|
||||
|
||||
@@ -1,177 +1,456 @@
|
||||
# TASK_BOARD
|
||||
|
||||
更新时间：2026-04-19 03:41 CST
|
||||
更新时间：2026-04-19 21:25 CST
|
||||
|
||||
## 2026-04-20 主线切换说明
|
||||
|
||||
从这一刻开始，当前工作重心正式切回：
|
||||
|
||||
- 先跑通主任务流程
|
||||
- 先推进 7 个大任务的测试与验收
|
||||
- 并发、代理、DB 往返、展示层等细节优化先封存，不再抢主线
|
||||
|
||||
细节优化不删除，统一视为 backlog：
|
||||
|
||||
- worker pool 持续补位
|
||||
- claimed 回弹压缩
|
||||
- 高频 DB 更新继续合并
|
||||
- 代理失败链和等待窗口继续压缩
|
||||
- 运营视角页面继续细化
|
||||
|
||||
这些后续继续做，但不再打断主任务验收顺序。
|
||||
|
||||
当前主口径：
|
||||
|
||||
1. 大任务 1 真实验收闭环
|
||||
2. 大任务 2 controller 编排闭环
|
||||
3. 大任务 3 统一结果状态机闭环
|
||||
4. 大任务 4 本地控制状态 + syncer/finalizer 闭环
|
||||
5. 大任务 5 固定 worker pool 落地
|
||||
6. 大任务 6 时光机一期接入
|
||||
7. 大任务 7 运营视角面板闭环
|
||||
|
||||
## 21:17 最新补充
|
||||
|
||||
刚刚这轮真实 rollout 已经把 mainland worker 的最后一层阻塞拿实：
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `release_id = 7`
|
||||
- `rollout_id = 6`
|
||||
- `job_id = 211`
|
||||
- 发布链路并不是卡在下载或解压：
|
||||
- 发布包 `domaincheck_release_20260419_205437` 已下载到 `/opt/domaincheck/downloads`
|
||||
- release 已解压到 `/opt/domaincheck/releases/domaincheck_release_20260419_205437`
|
||||
- `/opt/domaincheck/current` 已切到新 release
|
||||
- 最终失败点已经明确：
|
||||
- `restart failed: domaincheck-worker`
|
||||
- systemd 原因：
|
||||
- `Interactive authentication required`
|
||||
- 这说明当前 mainland 节点的 `domaincheck-node-agent` 虽然能写发布目录，但因为以 `www` 身份运行，无法执行：
|
||||
- `systemctl restart domaincheck-worker`
|
||||
|
||||
当前新的唯一主阻塞已经进一步收敛为：
|
||||
|
||||
- node-agent systemd 运行身份不对
|
||||
- 需要把 `domaincheck-node-agent` 改成 `root` 运行，发布链最后一跳才能闭环
|
||||
|
||||
本轮已在仓库内同步修正：
|
||||
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
- `domain-api/deploy/multi-region/fix_mainland_release_base.sh`
|
||||
- node-agent drop-in 现在也会显式写入：
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
|
||||
## 21:25 worker rollout 收口
|
||||
|
||||
`mainland-worker-01` 已完成新一轮正式 rollout 收口成功：
|
||||
|
||||
- `release_id = 7`
|
||||
- `rollout_id = 7`
|
||||
- `job_id = 214`
|
||||
|
||||
结果确认：
|
||||
|
||||
- `job 214 = success`
|
||||
- `rollout 7 = completed`
|
||||
- `mainland-worker-01` 已恢复为：
|
||||
- `agent_state = online_busy`
|
||||
- 最近心跳恢复正常
|
||||
|
||||
控制面完成证据：
|
||||
|
||||
- `agent_completed`
|
||||
- `summary_text = release deployed`
|
||||
- `restart_results`
|
||||
- `domaincheck-worker.returncode = 0`
|
||||
- `health_check.ok = true`
|
||||
- `execstart_alignment.mismatched_services = []`
|
||||
|
||||
这说明 worker 当前已经真正完成：
|
||||
|
||||
- 新包下载
|
||||
- release 解压
|
||||
- `current` 切换
|
||||
- `domaincheck-worker` 重启
|
||||
- 健康检查通过
|
||||
|
||||
当前关于 mainland rollout 的唯一剩余预防项变成：
|
||||
|
||||
- `mainland-controller-01` 也应该同步把 `domaincheck-node-agent` 切到 `root`
|
||||
- 否则后续 controller 自身执行 `deploy.release / service.restart` 时还会踩到同类 systemd 权限问题
|
||||
|
||||
## 顶部校正
|
||||
|
||||
今天晚上的最新排查已经把当前主阻塞重新定性，之前“大陆两台已经完全进入统一新镜像执行”的判断需要收紧。
|
||||
|
||||
当前新增硬结论：
|
||||
|
||||
- 最新正式发布包已经重打并签收通过：
|
||||
- `domaincheck_release_20260419_205437`
|
||||
- 发布包现在已经确实包含 `domainCheck/`
|
||||
- 对 `mainland-worker-01` 发起正式 smart rollout 后，真实失败原因已经拿到：
|
||||
- `job 204 / rollout 5`
|
||||
- 本次失败对应的正式 Release 版本仍是：
|
||||
- `domaincheck_release_20260419_203933`
|
||||
- 失败原因：
|
||||
- `Permission denied: /opt/domaincheck/downloads`
|
||||
- 该失败已通过正式 agent complete 回写，后台不再是假 `running`
|
||||
- `mainland-worker-01` 当前运行中的 `domaincheck-worker` 仍是旧进程：
|
||||
- `Main PID = 635924`
|
||||
- `Active since = 2026-04-19 18:18:50 CST`
|
||||
- `CPU = 18.994s`
|
||||
- 说明它不是高吞吐新进程，而是老进程长期挂着
|
||||
- `mainland-controller-01` 当前运行中的 `domaincheck-worker` 虽然活跃，但服务启动路径同样是：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- 两台 mainland 节点当前 `domaincheck-worker` 的 systemd 启动路径都不是：
|
||||
- `/opt/domaincheck/current/domainCheck/detect_worker.py`
|
||||
|
||||
这意味着：
|
||||
|
||||
- 当前 ReleaseHub 的 `current -> release_version` 切换模型，与线上 mainland 节点的实际服务启动路径不一致
|
||||
- 即使发布包和签收链已经修好，线上节点也还没有具备“按当前发布模型热切版本”的条件
|
||||
- 当前唯一主阻塞已经从“并发参数是否下发”切换成：
|
||||
- mainland 节点发布权限/目录模型不匹配
|
||||
- mainland 节点服务启动路径与发布模型不匹配
|
||||
|
||||
## 当前主批次
|
||||
|
||||
唯一主批次：`J2-检测执行停滞收口批`
|
||||
唯一主批次：`J2-检测执行收口批`
|
||||
|
||||
目标：
|
||||
|
||||
- 不进入新页面
|
||||
- 不扩展控制面
|
||||
- 不新增发布动作
|
||||
- 只收口当前唯一真实阻塞：
|
||||
- 节点已接管
|
||||
- 同步已打通
|
||||
- 但检测执行没有继续产出新结果
|
||||
- 只收口三件已经缩小到运行面的事情：
|
||||
- 大陆节点必须真正进入统一镜像队列，而不是停留在兼容旧链路
|
||||
- 后台必须能看到大陆节点的实时运行态与日志
|
||||
- `100/50` 并发配置要从“已下发”推进到“真实有参与吞吐”
|
||||
|
||||
## 本轮最新状态
|
||||
|
||||
本轮已经完成“代码修复 -> 两台大陆节点部署 -> 运行态复查”的完整一轮验证。
|
||||
### 20:53 最新阻塞结论
|
||||
|
||||
### 已完成的最小修复
|
||||
- `mainland-worker-01` smart rollout 已真实失败，不再继续误判为执行中
|
||||
- 失败证据：
|
||||
- node-agent 日志：
|
||||
- `2026-04-19 20:40:15 [node-agent] loop error: [Errno 13] Permission denied: '/opt/domaincheck/downloads'`
|
||||
- 发布任务：
|
||||
- `job 204 = failed`
|
||||
- 发布批次：
|
||||
- `rollout 5 = failed`
|
||||
- 当前不应该继续把精力放在前端展示或并发口径微调上
|
||||
- 当前必须先收口：
|
||||
- mainland 节点 `deploy.release` 所需目录权限
|
||||
- mainland 节点 systemd `ExecStart` 与 `/opt/domaincheck/current` 的一致性
|
||||
|
||||
- `worker_control_service.send_worker_command(...)`
|
||||
- 为每条 worker 控制消息补上 `request_id`
|
||||
- 保证 Redis 发布与 pending fallback 使用同一份负载
|
||||
- `detect_worker`
|
||||
- 在运行态心跳里周期性补偿消费 pending 控制消息
|
||||
- 在真正收到控制消息后，按 `request_id` 清理 pending 指令
|
||||
这轮已经完成从“控制链打通”到“真实执行恢复 + worker 日志回传恢复”的关键跨越。
|
||||
|
||||
本地验证已通过：
|
||||
### 本轮前端已上线校验
|
||||
|
||||
- `unittest domain-api/tests/test_worker_control_service.py`
|
||||
- `python -m py_compile domainCheck/detect_worker.py domain-api/app/services/worker_control_service.py`
|
||||
- 检测页 `任务日志控制台` 已调整到事件列表上方
|
||||
- `检测事件流` 已改成独立滚动区：
|
||||
- 表格 `max-height = 320`
|
||||
- 避免事件越积越多把整个页面继续向下撑长
|
||||
- 已在线上实际静态目录重新构建：
|
||||
- `/www/wwwroot/getDomain/domain-web/dist`
|
||||
- 已确认线上 Nginx 指向该目录：
|
||||
- `domaincheck_3201.conf`
|
||||
- `domaincheck_152.53.37.118.conf`
|
||||
- 已通过 `http://127.0.0.1:3201/` 返回 `200` 且加载新资源：
|
||||
- `DetectView-B9S1WMRT.js`
|
||||
- `DetectView-BFxMNRNR.css`
|
||||
|
||||
### 已完成的线上验证
|
||||
### 本轮代理链已收口
|
||||
|
||||
- `mainland-worker-01`
|
||||
- 已拉到 `main` 最新提交 `c33f4f1`
|
||||
- 已重启 `domaincheck-worker`
|
||||
- 启动后明确出现：
|
||||
- `发现待执行 Worker 控制指令`
|
||||
- `已接受检测启动指令`
|
||||
- `开始执行远程检测任务`
|
||||
- 已修复 worker 在“代理配置已下发但尚未开始检测”时长期停留 `未刷新` 的问题
|
||||
- `domaincheck-worker` 现在会在以下时机主动触发代理池刷新：
|
||||
- worker 启动后
|
||||
- `proxy_config / thread_count / node_thread_counts / runtime_settings` 更新后
|
||||
- 当前后台状态已不再误报“未刷新”
|
||||
- 代理策略已切到：
|
||||
- 只去掉过期 / 非法代理
|
||||
- 跳过预验证
|
||||
- 直接入池执行
|
||||
- 真实失败后立即淘汰并补刷
|
||||
- 最新实测结果已经明确：
|
||||
- 原始代理 `270` 个
|
||||
- 预验证 `0` 个
|
||||
- 当前可用代理 `270` 个
|
||||
- 最近状态：
|
||||
- `proxy_last_refresh_status = 直入池 270 个（跳过预验证）`
|
||||
- `available_proxy_count = 270`
|
||||
|
||||
### 本轮已完成
|
||||
|
||||
- `sync_push_service`
|
||||
- 已修复 `detect_task_ingest` 只落 `domains`、不落本地 `detect_jobs/detect_job_items` 的缺口
|
||||
- controller 现在会为拉回来的海外批次持续创建本地镜像任务：
|
||||
- `sync-overseas-7380`
|
||||
- `sync-overseas-7383`
|
||||
- `sync-overseas-7392`
|
||||
- 持续增长中
|
||||
- `mainland-controller-01`
|
||||
- 已拉到 `main` 最新提交 `c33f4f1`
|
||||
- 已重启 `domaincheck-worker`
|
||||
- 发现运行环境漂移：
|
||||
- `/etc/default/domaincheck-worker` 中 `REDIS_PASSWORD` 为空
|
||||
- 已最小修正该节点运行环境后再次重启
|
||||
- 修正后明确出现：
|
||||
- `Redis 连接成功: 127.0.0.1:6379`
|
||||
- `已接受检测启动指令`
|
||||
- `开始执行远程检测任务`
|
||||
- 已明确进入真实队列：
|
||||
- `从任务队列获取到 800 个需要检测的域名`
|
||||
- `最大线程数: 100`
|
||||
- `当前实际线程数量: 1/100 ... 6/100`
|
||||
- `mainland-worker-01`
|
||||
- 已修复 `sync-pull` 控制消息上下文绑定缺口：
|
||||
- `detect_worker._set_active_cycle_context` 现在会读取
|
||||
- `target_job_id`
|
||||
- `target_job_code`
|
||||
- 已重启 `domaincheck-worker`
|
||||
- 已明确进入真实队列：
|
||||
- `从任务队列获取到 400 个需要检测的域名`
|
||||
- `最大线程数: 50`
|
||||
- `当前实际线程数量: 1/50 ... 3/50`
|
||||
- 已明确把运行日志回传到 overseas：
|
||||
- `开始执行检测任务，来源: redis-control`
|
||||
- `开始执行域名检测任务，正在加载配置`
|
||||
- `开始检测，正在刷新代理池`
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `当前实际线程数量: 1/50`
|
||||
- `当前实际线程数量: 2/50`
|
||||
- `当前实际线程数量: 3/50`
|
||||
- `runtime_projection`
|
||||
- 已定位 controller `domain-api/runtime/detect_runs.json` 权限错误
|
||||
- 已把 `runtime` 目录和 `detect_runs.json` 改回 `www:www`
|
||||
- `domaincheck-sync-agent` 最新日志已从
|
||||
- `partial_success`
|
||||
- 变成：
|
||||
- `sync_state = success`
|
||||
- `runtime_projection -> 投影推送成功`
|
||||
- 海外控制面 `/api/v1/ops/nodes`
|
||||
- 现已明确看到三台节点都在参与
|
||||
- `remote_access_ready = 3`
|
||||
- `participating = 3`
|
||||
- `mainland-controller-01.is_current_participant = true`
|
||||
- `mainland-worker-01.is_current_participant = true`
|
||||
|
||||
### 本轮仍未完成的事情
|
||||
### 当前最关键的新证据
|
||||
|
||||
- 中央 `items_completed` 仍未在短观察窗口内继续增长
|
||||
- `mainland-controller-01` 仍然持续报：
|
||||
- `代理已启用，但当前无可用代理`
|
||||
- 因此当前剩余阻塞已经进一步收紧到：
|
||||
- controller 现场代理池没有可用代理
|
||||
- `mainland-controller-01`
|
||||
- `processed_recent = 115`
|
||||
- `processed_per_minute = 7.67`
|
||||
- `detect_runtime.max_threads = 100`
|
||||
- `mainland-worker-01`
|
||||
- `processed_recent = 632`
|
||||
- `processed_per_minute = 42.13`
|
||||
- `detect_runtime.max_threads = 50`
|
||||
- `overseas-control-01`
|
||||
- 仍在执行中央原生队列
|
||||
- 当前判断：
|
||||
- 大陆节点已经不是“纸面在线”
|
||||
- 已经是真实参与检测执行
|
||||
|
||||
## 本轮最新复查结果
|
||||
## 当前主判断
|
||||
|
||||
本轮在完成修复部署后，中央与节点现场出现了新的运行证据：
|
||||
### 本轮新增结论
|
||||
|
||||
- 活跃任务仍是 `detect-20260417170546-96023e`
|
||||
- `mainland-worker-01` 最新事件时间已经从旧窗口推进到：
|
||||
- `2026-04-19 16:37:00`
|
||||
- `runtime/sync-summary` 最新记录已继续增长到：
|
||||
- `id = 5491`
|
||||
- `created_at = 2026-04-19 03:40:45`
|
||||
- 说明中央已经重新收到新的运行侧同步流量
|
||||
- 但短窗口内主进度仍未松动：
|
||||
- `completed = 21`
|
||||
- `running = 14`
|
||||
- `claimed = 34`
|
||||
- `pending = 931`
|
||||
- controller 现场最新日志已收紧为：
|
||||
- `代理已启用，但当前无可用代理`
|
||||
- `mainland-controller-01`
|
||||
- 已修复“代理数量不足反向限制并发”的热路径问题
|
||||
- 当前海外后台已稳定看到：
|
||||
- `active_threads ~= 99~100`
|
||||
- `max_threads = 100`
|
||||
- 说明 `100` 并发不再只是配置已下发，而是已进入真实运行态
|
||||
- `mainland-worker-01`
|
||||
- 已重新部署同版 `detect_worker.py`
|
||||
- 已重新接收到新的 `sync-pull` 批次：
|
||||
- `source_record_id = 7679`
|
||||
- `target_job_code = sync-overseas-7679`
|
||||
- 已完成代理抽样校验并进入：
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- 当前剩余问题已经缩成：
|
||||
- worker 本机已启动新批次并创建线程
|
||||
- 但海外后台对 `mainland-worker-01.active_threads` 的显示仍偏低，和本机进程线程量不完全一致
|
||||
|
||||
### 当前唯一剩余收口点
|
||||
|
||||
- 不再是 controller 并发限制问题
|
||||
- 当前唯一剩余收口点变成：
|
||||
- `mainland-worker-01` 的运行态上报口径仍需继续和真实执行量对齐
|
||||
- 以及检测主计数 `pending/completed/running` 继续推进
|
||||
|
||||
### 已经收口的部分
|
||||
|
||||
- 大陆节点不再停留在 `pending_bootstrap`
|
||||
- Agent + SSH 接管已经完成
|
||||
- `sync-agent pull_tasks` 已经真正生成本地镜像队列
|
||||
- 两台大陆 worker 已经真正吃到 `detect_job_items`
|
||||
- 后台运行态同步链已经恢复
|
||||
|
||||
### 当前剩余问题
|
||||
|
||||
- Detect 页面日志窗口已经不再是单节点
|
||||
- `/api/v1/detect/status` 已出现：
|
||||
- `remote_log_node_count = 2`
|
||||
- `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]`
|
||||
- 当前剩余问题缩成一个最小点：
|
||||
- 检测页主计数 `pending/completed/running` 还没有跟着这轮大陆执行立即推进
|
||||
- 前端布局问题已收口，后续不再停留在“页面结构挡住观察”
|
||||
- 代理链当前也已收口到真实口径，不再停留在“配置有了但没刷新”
|
||||
- 下一步应继续观察这 `270` 个直入池代理在真实检测里的淘汰速度与吞吐提升
|
||||
- 现在更应该同时看：
|
||||
- `/api/v1/detect/status`
|
||||
- `/api/v1/runtime/debug-events`
|
||||
- `/api/v1/ops/nodes`
|
||||
- worker `current actual threads` 日志
|
||||
|
||||
## 下一步唯一主批次
|
||||
|
||||
唯一主批次保持为：`J2-检测执行收口批`
|
||||
|
||||
下一步只做：
|
||||
|
||||
- 继续观察 Detect 页面主计数是否跟上最新运行态
|
||||
- 继续确认 `detect_result_projection` / 结果统计回推是否稳定推进
|
||||
- 继续收口 `mainland-worker-01` 的运行态上报，使后台显示与本机真实线程量一致
|
||||
- 若仍有显示偏差，只修统计/上报口径，不新扩功能
|
||||
|
||||
## 候选批次
|
||||
|
||||
### Candidate J3
|
||||
|
||||
名称：结果计数收口批
|
||||
|
||||
进入条件：
|
||||
|
||||
- 节点已经真实执行
|
||||
- 日志窗口已经恢复
|
||||
- 但 Detect 页面主计数仍不前进
|
||||
|
||||
只做：
|
||||
|
||||
- 复核 `detect_result_projection`
|
||||
- 复核结果统计回推
|
||||
- 不改控制面结构
|
||||
|
||||
### Candidate J4
|
||||
|
||||
名称：吞吐稳定性观察批
|
||||
|
||||
进入条件：
|
||||
|
||||
- 页面口径已恢复
|
||||
- 继续确认大陆两节点吞吐是否稳定，不再回落
|
||||
|
||||
只做：
|
||||
|
||||
- 继续观察 `processed_recent`
|
||||
- 继续观察代理池质量
|
||||
- 继续观察 `active_threads/max_threads`
|
||||
|
||||
## 暂停项
|
||||
|
||||
以下任务现在不应继续推进：
|
||||
|
||||
- 新页面
|
||||
- 新模块
|
||||
- 新发布动作
|
||||
- 新专题文档
|
||||
- 与检测执行收口无关的控制面增强
|
||||
|
||||
## 关键证据
|
||||
|
||||
### 证据 1：接管与同步已通
|
||||
### 证据 1：镜像队列已经落地
|
||||
|
||||
当前中央状态：
|
||||
controller `sync-agent` 最新 `task pull tick` 已出现：
|
||||
|
||||
- `remote_access_ready = 3/3`
|
||||
- `log_sync_state = full_capture`
|
||||
- mainland controller 的 `domaincheck-sync-agent` 已在新进程上运行
|
||||
- `target_job_id`
|
||||
- `target_job_code`
|
||||
- `queued_count = 200`
|
||||
- `worker_start_ok = True`
|
||||
|
||||
说明：
|
||||
|
||||
- 当前不是接管问题
|
||||
- 也不是日志回传问题
|
||||
- 更不是同步链完全断开
|
||||
- controller 现在不是只同步域名
|
||||
- 而是在本地持续创建可执行队列
|
||||
|
||||
### 证据 2：检测任务当前没有继续出新结果
|
||||
### 证据 2：大陆 worker 已进入真实队列
|
||||
|
||||
连续 40 秒前后对比结果完全一致：
|
||||
现场日志已经明确出现：
|
||||
|
||||
- `JOB_PROGRESS = 2.1`
|
||||
- `items_completed = 21`
|
||||
- `items_running = 14`
|
||||
- `items_claimed = 34`
|
||||
- `items_pending = 931`
|
||||
- `mainland-controller-01 -> 从任务队列获取到 800 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
|
||||
说明：
|
||||
|
||||
- 当前不是“页面慢一拍”
|
||||
- 而是执行面这段时间确实没有继续产出
|
||||
- “并发没起来”的主要根因已经修正
|
||||
- 它们不是在跑旧兼容链路
|
||||
|
||||
### 证据 3：中央 mainland 逐条结果没有继续增长
|
||||
### 证据 3：运行态同步已恢复
|
||||
|
||||
当前中央查询结果：
|
||||
controller `sync-agent` 最新日志已经出现：
|
||||
|
||||
- mainland `domain_started/domain_completed/domain_failed/domain_blacklisted`
|
||||
- 仍为 `10`
|
||||
- 最新 mainland `detect_result_ingest`
|
||||
- 仍为 `5382`
|
||||
- `sync_state = success`
|
||||
- `runtime_projection -> 投影推送成功`
|
||||
|
||||
说明：
|
||||
|
||||
- 首批同步成功过
|
||||
- 但后续并没有继续流入新逐条结果
|
||||
- 后台运行态/日志窗口链路已经不再被权限错误卡死
|
||||
|
||||
### 证据 4：controller 现场日志已指向代理池可用性为 0
|
||||
### 证据 4：海外 `/ops/nodes` 已把大陆节点判定为真实参与者
|
||||
|
||||
`mainland-controller-01` 现场日志显示：
|
||||
当前海外控制面已经显示：
|
||||
|
||||
- `当前可用代理数: 0`
|
||||
- `最近结果: 刷新成功，可用 0 个`
|
||||
- 免费检测链包含：
|
||||
- 注册查询
|
||||
- 百度 site
|
||||
- 360 site
|
||||
- 站长之家
|
||||
- 爱站
|
||||
- 时光机
|
||||
|
||||
进一步的远端 `logs.collect(domaincheck-worker)` 结果显示：
|
||||
|
||||
- controller 能从 6 个代理源成功拉到原始代理
|
||||
- 但在抽样验证后：
|
||||
- `代理池刷新完成，共 0 个可用代理`
|
||||
- 失败原因集中在：
|
||||
- `ProxyError@https://m.baidu.com`
|
||||
- `Unable to connect to proxy`
|
||||
- `ConnectTimeoutError`
|
||||
- `mainland-controller-01`
|
||||
- `online_busy`
|
||||
- `is_current_participant = true`
|
||||
- `processed_recent = 115`
|
||||
- `mainland-worker-01`
|
||||
- `online_busy`
|
||||
- `is_current_participant = true`
|
||||
- `processed_recent = 632`
|
||||
|
||||
说明：
|
||||
|
||||
- 当前不是代理源接口没返回
|
||||
- 而是“拿到的代理全部验不过”
|
||||
- 主阻塞已经可以精确收紧到 controller 代理池不可用
|
||||
- 现在大陆两台都已进入真实参与态
|
||||
- 下一步不再是接管问题，而是页面口径与吞吐稳定性问题
|
||||
|
||||
### 证据 5：worker 的时光机异常存在，但不是第一主因
|
||||
### 证据 5：Detect 页面远端日志已恢复双节点
|
||||
|
||||
`mainland-worker-01` 的远端 `domaincheck-worker` 日志显示：
|
||||
当前 `/api/v1/detect/status` 已显示：
|
||||
|
||||
- 存在：
|
||||
- `时光机检测 外部依赖异常，步骤降级继续执行`
|
||||
- 同时仍可见：
|
||||
- `域名检测完成`
|
||||
- 最近收到控制消息后：
|
||||
- `收到启动检测指令，但检测任务已在运行，忽略重复启动`
|
||||
- `remote_log_node_count = 2`
|
||||
- `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]`
|
||||
|
||||
最近日志样本已出现：
|
||||
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 1/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 2/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 3/50`
|
||||
|
||||
说明：
|
||||
|
||||
- worker 并不是完全不能执行
|
||||
- 时光机异常是客观存在的次级问题
|
||||
- 但它不像 controller 代理池为 0 那样直接卡住整体吞吐
|
||||
- worker 不再是“只在节点本地运行、页面看不到”
|
||||
- 检测页日志链已经真正接上 mainland worker
|
||||
|
||||
### 证据 6：full_capture 已开启，但源日志时间没有继续前进
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
683962
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 02:29:55","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":3,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"full_capture","status_label":"全量观察","status_type":"success","summary":"节点当前正在参与检测，已保留 1 条现场日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":1,"key_line_count":0,"full_line_count":1,"last_at":"2026-04-19 01:20:35","last_line":"[2026-04-19 01:20:35] [overseas-control-01] 2026-04-19 01:00:18.866 | INFO | __main__:start_detection:2575 - 开始执行域名检测任务"},"missing_reason_code":"","missing_reason":"","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 02:30:02"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract，供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,16 @@
|
||||
CYCLES="2"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="full_capture"
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS="attention"
|
||||
ISSUE_TOTAL="1"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES="playbook_runs_need_attention"
|
||||
LAUNCHPAD_STATUS="blocked"
|
||||
LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance"
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS="full_capture"
|
||||
SCENE_LINE_COUNT="1"
|
||||
GENERATED_AT="2026-04-19 02:29:55"
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 03:30:12","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"partial_coverage","log_sync_mode":"full","log_sync_covered_nodes":2,"log_sync_missing_node_codes":["overseas-control-01"],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"missing_sample","status_label":"缺少样本","status_type":"warning","summary":"当前还没有收到该参与节点的远端日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":0,"key_line_count":0,"full_line_count":0,"last_at":"","last_line":""},"missing_reason_code":"no_sample","missing_reason":"当前还没有收到该参与节点的远端日志样本。","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 03:30:19"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract，供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,16 @@
|
||||
CYCLES="3"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="partial_coverage"
|
||||
LOG_SYNC_MISSING_NODE_CODES="overseas-control-01"
|
||||
STACK_STATUS="attention"
|
||||
ISSUE_TOTAL="2"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES="remote_log_sync_waiting_sample,playbook_runs_need_attention"
|
||||
LAUNCHPAD_STATUS="blocked"
|
||||
LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance"
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS="missing_sample"
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT="2026-04-19 03:30:12"
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 03:31:26","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":3,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"full_capture","status_label":"全量观察","status_type":"success","summary":"节点当前正在参与检测，已保留 3 条现场日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":3,"key_line_count":0,"full_line_count":3,"last_at":"2026-04-19 03:30:30","last_line":"[2026-04-19 03:30:30] [overseas-control-01] 2026-04-19 01:59:40.686 | WARNING | __main__:start_detection_async:1073 - 收到启动检测指令，但检测任务已在运行，忽略重复启动"},"missing_reason_code":"","missing_reason":"","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 03:31:33"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract，供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,16 @@
|
||||
CYCLES="4"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="full_capture"
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS="attention"
|
||||
ISSUE_TOTAL="1"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES="playbook_runs_need_attention"
|
||||
LAUNCHPAD_STATUS="blocked"
|
||||
LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance"
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS="full_capture"
|
||||
SCENE_LINE_COUNT="3"
|
||||
GENERATED_AT="2026-04-19 03:31:26"
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 04:31:45","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"partial_coverage","log_sync_mode":"full","log_sync_covered_nodes":2,"log_sync_missing_node_codes":["overseas-control-01"],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"missing_sample","status_label":"缺少样本","status_type":"warning","summary":"当前还没有收到该参与节点的远端日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":0,"key_line_count":0,"full_line_count":0,"last_at":"","last_line":""},"missing_reason_code":"no_sample","missing_reason":"当前还没有收到该参与节点的远端日志样本。","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 04:31:52"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract，供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,16 @@
|
||||
CYCLES="5"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="partial_coverage"
|
||||
LOG_SYNC_MISSING_NODE_CODES="overseas-control-01"
|
||||
STACK_STATUS="attention"
|
||||
ISSUE_TOTAL="2"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES="remote_log_sync_waiting_sample,playbook_runs_need_attention"
|
||||
LAUNCHPAD_STATUS="blocked"
|
||||
LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance"
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS="missing_sample"
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT="2026-04-19 04:31:45"
|
||||
@@ -0,0 +1,16 @@
|
||||
CYCLES="6"
|
||||
GO_LIVE_STATUS=""
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE=""
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS=""
|
||||
ISSUE_TOTAL="0"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES=""
|
||||
LAUNCHPAD_STATUS=""
|
||||
LAUNCHPAD_RECOMMENDED_ACTION=""
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS=""
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT=""
|
||||
@@ -0,0 +1,16 @@
|
||||
CYCLES="7"
|
||||
GO_LIVE_STATUS=""
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE=""
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS=""
|
||||
ISSUE_TOTAL="0"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES=""
|
||||
LAUNCHPAD_STATUS=""
|
||||
LAUNCHPAD_RECOMMENDED_ACTION=""
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS=""
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT=""
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 04:34:28","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":1,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":1,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
@@ -0,0 +1,16 @@
|
||||
CYCLES="8"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="full_capture"
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS=""
|
||||
ISSUE_TOTAL="0"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES=""
|
||||
LAUNCHPAD_STATUS=""
|
||||
LAUNCHPAD_RECOMMENDED_ACTION=""
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS=""
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT="2026-04-19 04:34:28"
|
||||
@@ -0,0 +1,59 @@
|
||||
# night_run_20260419_012929 初步分析
|
||||
|
||||
## 结论
|
||||
|
||||
- 夜跑不是完全空跑，`cycle_1` 到 `cycle_8` 期间持续执行了巡检、收口和日志回传恢复动作。
|
||||
- 真正的中断点出现在 `2026-04-19 04:32:49` 到 `04:33:35`，本机 API `127.0.0.1:8100` 短时不可达，导致两轮自动动作直接失败。
|
||||
- 最终停止原因是 `signoff_ready_candidate`，这是“候选可签收”型收口，不等同于“整轮全绿、无异常结束”。
|
||||
|
||||
## 关键时间点
|
||||
|
||||
- `2026-04-19 01:29:29 +0800`
|
||||
- 夜跑启动。
|
||||
- `cycle_1` 到 `cycle_5`
|
||||
- 持续产出 `go_live.json`、`launchpad.json`、`playbook_runs.json`、`stack.json`、`scene_overseas_control_01.json`、`summary.env`。
|
||||
- `2026-04-19 04:32:04 +0800`
|
||||
- `run_inspection_participating` 成功创建 playbook，回执 `pbr-6d77b32f40`。
|
||||
- `2026-04-19 04:32:49 +0800`
|
||||
- `cycle=6`
|
||||
- `log-sync-recover` 调用失败。
|
||||
- `driver-run run_inspection_participating` 调用失败。
|
||||
- 错误为 `curl: (7) Failed to connect to 127.0.0.1 port 8100: Connection refused`。
|
||||
- `2026-04-19 04:33:34 +0800`
|
||||
- `cycle=7`
|
||||
- 同类动作再次失败，错误相同。
|
||||
- `2026-04-19 04:34:28 +0800`
|
||||
- `cycle=8`
|
||||
- `go_live=attention`
|
||||
- `log_sync=full_capture`
|
||||
- 夜跑停止，`reason=signoff_ready_candidate`。
|
||||
|
||||
## 已确认的问题
|
||||
|
||||
- 夜跑期间存在控制面 API 短时离线或重启窗口。
|
||||
- 自动恢复逻辑在 API 不可达时会直接失败，但日志里没有看到进一步的退避、跳过本轮、等待 API 恢复后的再确认闭环。
|
||||
- 当时的“可签收候选”判断，掺杂了 API 短时不可达窗口，所以不能把这次夜跑结论直接当成正式签收证据。
|
||||
|
||||
## 这轮修复后的关联状态
|
||||
|
||||
- 当前中央控制面已经恢复正常。
|
||||
- `runtime/cluster` 已恢复为 3 台有效执行节点。
|
||||
- `detect/status` 已恢复远端日志回传。
|
||||
- `mainland-controller-01` 当前已恢复 `100/100`。
|
||||
- `mainland-worker-01` 当前已进入活跃参与，中央已看到 `2/50`。
|
||||
|
||||
## 明天继续看时，优先检查
|
||||
|
||||
- `docs/ops_center_runtime/night_runs/night_run_20260419_012929.log`
|
||||
- 重点看 `04:32:49` 到 `04:34:28` 这段 API 拒绝连接窗口。
|
||||
- `docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/go_live.json`
|
||||
- 确认 `go_live=attention` 的具体触发项。
|
||||
- `docs/ops_center_runtime/night_runs/night_run_20260419_012929_report.md`
|
||||
- 对照夜跑最终报告和原始日志，确认是否把“候选可签收”误当成“正式通过”。
|
||||
|
||||
## 下一步建议
|
||||
|
||||
- 补一条夜跑期间的 API 可用性守护：
|
||||
- 发现 `127.0.0.1:8100` 不可达时，不立刻继续推进收口动作，先等待 API 恢复后重试。
|
||||
- 把“候选可签收”和“正式可签收”拆开：
|
||||
- 避免在 API 短时重启窗口里出现假阳性收口。
|
||||
@@ -0,0 +1,44 @@
|
||||
# NIGHT RUN REPORT night_run_20260419_012929
|
||||
|
||||
- Base URL: `http://127.0.0.1:8100`
|
||||
- Deadline: `2026-04-20 12:00:00 +0800`
|
||||
- Stop Reason: `signoff_ready_candidate`
|
||||
- Cycles: `8`
|
||||
- Log Sync Recover Runs: `4`
|
||||
- Inspection Runs: `4`
|
||||
- Inspection Churn Runs: `0`
|
||||
- Quick Rechecks: `4`
|
||||
|
||||
## Final Snapshot
|
||||
|
||||
- `go_live_status = attention`
|
||||
- `publish_ready = false`
|
||||
- `log_sync_state = full_capture`
|
||||
- `issue_total = 0`
|
||||
- `problem_runs_total = 0`
|
||||
- `launchpad_status = `
|
||||
- `launchpad_recommended_action = `
|
||||
- `problem_run_code = `
|
||||
|
||||
## Summary JSON
|
||||
|
||||
```json
|
||||
{
|
||||
"cycles": 8,
|
||||
"go_live_status": "attention",
|
||||
"publish_ready": false,
|
||||
"log_sync_state": "full_capture",
|
||||
"log_sync_missing_node_codes": [],
|
||||
"stack_status": "",
|
||||
"issue_total": 0,
|
||||
"blocking_issue_total": 0,
|
||||
"issue_codes": [],
|
||||
"launchpad_status": "",
|
||||
"launchpad_recommended_action": "",
|
||||
"problem_runs_total": 0,
|
||||
"problem_run_code": "",
|
||||
"scene_status": "",
|
||||
"scene_line_count": 0,
|
||||
"generated_at": "2026-04-19 04:34:28"
|
||||
}
|
||||
```
|
||||
@@ -1,20 +1,18 @@
|
||||
{
|
||||
"cycles": 1,
|
||||
"cycles": 8,
|
||||
"go_live_status": "attention",
|
||||
"publish_ready": false,
|
||||
"log_sync_state": "full_capture",
|
||||
"log_sync_missing_node_codes": [],
|
||||
"stack_status": "attention",
|
||||
"issue_total": 1,
|
||||
"stack_status": "",
|
||||
"issue_total": 0,
|
||||
"blocking_issue_total": 0,
|
||||
"issue_codes": [
|
||||
"playbook_runs_need_attention"
|
||||
],
|
||||
"launchpad_status": "blocked",
|
||||
"launchpad_recommended_action": "run_acceptance",
|
||||
"issue_codes": [],
|
||||
"launchpad_status": "",
|
||||
"launchpad_recommended_action": "",
|
||||
"problem_runs_total": 0,
|
||||
"problem_run_code": "",
|
||||
"scene_status": "full_capture",
|
||||
"scene_line_count": 1,
|
||||
"generated_at": "2026-04-19 01:29:38"
|
||||
"scene_status": "",
|
||||
"scene_line_count": 0,
|
||||
"generated_at": "2026-04-19 04:34:28"
|
||||
}
|
||||
114
docs/test.md
Normal file
114
docs/test.md
Normal file
@@ -0,0 +1,114 @@
|
||||
开个定时任务不停扫数据库，未检测的 快到期删除的，全部扫出来推送给注册任务队列；
|
||||
|
||||
注册检测：筛选出可以注册的，这个完成后，不管成功失败，返会标准化结果给 controller，告诉他，这个流程我走完了，如果是外部原因导致未能出结果就失败的，controller 会重新投入注册检测，更新数据库，如果是成功的，把对应的状态更新数据库，跟新注册状态；
|
||||
|
||||
controller 收到返回注册完成后， 以标准化格式下一步任务队列里面 ，等worker 来啦取，
|
||||
worker 只负责去controller 啦取的任务，处理任务，根据任务标准标识，安排对于的函数处理，所以woker只复制啦和跑，我没任务了，我很有空，我就去controller 获取任务，只要你给，我就跑，跑完返回结果给你；
|
||||
controller 在收到worker的啦取请求后，按后台勾选的配置任务队列，按顺序返回给woker，不是随便返回
|
||||
controller 返回任务时，要同时完成认领
|
||||
也就是：
|
||||
标记该 task 已分配给某个 worker
|
||||
进入 running 状态
|
||||
设置超时 TTL
|
||||
超时未回传则回收重投
|
||||
否则会出现：
|
||||
worker 拿了任务挂了
|
||||
controller 以为还在跑
|
||||
任务永远丢了
|
||||
|
||||
|
||||
xx检测：当前 xx 步骤执行并返回判定结果，这个完成后，不管成功失败，返会标准化结果给controller，告诉他，这个xx流程我走完了，如果是外部原因导致失败的，还没有出结果的，非业务判定不通过被跑到为黑名单的，controller 会重新投入当前xx任务队列，如果名中黑名单，直接跟新黑名单状态，不在分发到后面所有步骤，更新数据库，如果是成功的，跟新当前xx检测状态，
|
||||
|
||||
controller 收到某步骤返回结果后，先根据当前 后台 配置和该域名已完成步骤状态，解析该域名在本次流程中的下一勾选步骤；若存在下一步骤，则以标准化任务格式投入对应任务队列；若不存在，则标记本次流程完成
|
||||
|
||||
只按顺序处理后台勾选的任务
|
||||
注册检测
|
||||
百度检测
|
||||
站长检测
|
||||
爱站检测
|
||||
时光机检测
|
||||
聚查检测
|
||||
桔子检测
|
||||
|
||||
备注：
|
||||
时光机具体还要细分方案 ，目前项目内已经又对于的方案，加一个直晒前最近5年的快照，先跑通，后面可以继续细优化
|
||||
步骤都是按后台勾选，把勾选的跑完就算流程跑完，第一次没勾选的，下次勾选，可以直接跑够选的步骤，其他跳过；
|
||||
|
||||
|
||||
|
||||
海外主库(域名源/最终账本)
|
||||
↓
|
||||
controller dispatcher 拉取待处理域名
|
||||
↓
|
||||
按 pipeline 配置生成首个待执行步骤任务
|
||||
↓
|
||||
推入 Redis 对应步骤队列
|
||||
↓
|
||||
worker 向 controller 拉取任务
|
||||
↓
|
||||
controller 认领并分配任务给 worker
|
||||
↓
|
||||
worker 执行检测并回传标准化结果
|
||||
↓
|
||||
controller stage-processor 更新本地控制状态
|
||||
↓
|
||||
根据结果判断：
|
||||
- retry -> 重投当前步骤
|
||||
- black_hit -> 拉黑并终止
|
||||
- reject -> 终止/复核
|
||||
- pass -> 解析下一勾选步骤并投递
|
||||
↓
|
||||
controller syncer 批量同步海外主库
|
||||
↓
|
||||
controller finalizer 标记本次流程完成
|
||||
|
||||
|
||||
|
||||
|
||||
域名检测流程设计
|
||||
1. 总体原则
|
||||
海外主库负责域名源数据与最终账本持久化，不参与高频实时调度
|
||||
controller 负责任务编排、任务分发、结果处理、状态推进与批量同步
|
||||
Redis 负责各步骤待执行队列、运行中任务、重试任务与去重控制
|
||||
worker 仅负责向 controller 拉取任务、执行对应检测函数、回传标准化结果
|
||||
2. 注册任务投递
|
||||
定时任务持续扫描海外主库中的待检测域名，包括未检测域名、快到期删除域名等
|
||||
controller dispatcher 将符合条件的域名按标准化任务格式推入注册任务队列
|
||||
3. 步骤执行规则
|
||||
|
||||
每个检测步骤执行完成后，无论结果如何，worker 均需返回标准化结果给 controller。
|
||||
controller 根据结果类型做统一处理：
|
||||
|
||||
若为外部原因导致未得到有效结果，则根据重试策略重新投入当前步骤队列
|
||||
若为业务判定不通过，则更新当前步骤状态，并按策略终止本次流程
|
||||
若命中黑名单，则更新黑名单状态并终止后续所有步骤
|
||||
若业务判定通过，则更新当前步骤状态，并根据本次 pipeline 配置解析下一勾选步骤，投入对应任务队列
|
||||
4. pipeline 推进规则
|
||||
所有步骤按后台勾选生成本次 pipeline
|
||||
controller 仅按勾选顺序为单个域名推进下一步骤
|
||||
若某步骤在本次 pipeline 中未勾选，则直接跳过
|
||||
若前次未勾选、后次新增勾选，则可直接从已完成状态之后继续补跑，无需重跑已完成步骤
|
||||
5. worker 拉取规则
|
||||
worker 空闲时向 controller 发起拉取任务请求
|
||||
controller 根据任务队列状态、任务顺序与后台配置返回当前可执行任务
|
||||
worker 只负责执行任务，不负责流程判断、不负责决定下一步骤、不直接高频写海外主库
|
||||
6. 状态更新规则
|
||||
controller stage-processor 实时更新本地控制状态
|
||||
controller syncer 以批量方式将步骤状态、黑名单状态、流程状态同步至海外主库
|
||||
controller finalizer 在本次 pipeline 所有勾选步骤完成或流程被终止后，标记流程完成
|
||||
7. 当前步骤顺序
|
||||
|
||||
按后台勾选顺序处理以下任务：
|
||||
|
||||
注册检测
|
||||
百度检测
|
||||
站长检测
|
||||
爱站检测
|
||||
时光机检测
|
||||
聚查检测
|
||||
桔子检测
|
||||
8. 时光机一期方案
|
||||
当前先按最近 5 年快照执行简化方案，先跑通主流程
|
||||
后续再继续细化为更完整的时光机子流程
|
||||
|
||||
controller 机器如果性能足够剩余也可以部署worker 跑，
|
||||
Reference in New Issue
Block a user