diff --git a/docs/base.md b/docs/base.md new file mode 100644 index 0000000..afe3db2 --- /dev/null +++ b/docs/base.md @@ -0,0 +1,287 @@ +# 主线任务验收总表 + +更新时间:2026-04-21 + +## 当前执行原则 + +当前先只做一件事: + +- 跑通主任务流程 +- 把 7 个大任务按“可测试、可验收”的标准推进 +- 并发、代理、展示层、CPU 打满这类细节优化先封存,不再抢主线 + +当前不再把“性能抠点”当主目标。 +后续额度恢复后,再继续做细节优化专项。 + +## 当前主线优先级 + +按下面顺序推进,不跳步: + +1. 大任务 1 真实验收闭环 +2. 大任务 2 controller 编排闭环 +3. 大任务 3 统一结果状态机闭环 +4. 大任务 4 本地控制状态 + syncer/finalizer 闭环 +5. 大任务 5 固定 worker pool + 持续补位替换旧模型 +6. 大任务 6 时光机一期接入标准步骤 +7. 大任务 7 运营视角指标面板落地 + +## 7 个大任务当前状态 + +### 大任务 1 +单步骤任务底座落地 + +目标: +先打通 `controller -> queue -> claim -> worker -> result -> controller` 的完整闭环。 + +当前状态: +- 已有较完整代码底座 +- `single_step` / `step_code` 模型已基本成形 +- 现在缺的是“真实运行验收”而不是继续堆骨架 + +本任务验收只认下面 5 件事: + +1. controller 能生成 `baidu_check` 任务 +2. worker 能拉到 `baidu_check` 任务并执行 +3. worker 能回传标准化结果 +4. controller 能更新本地步骤状态 +5. worker 超时未回传时,任务会回收重投 + +当前结论: +- 代码方向已基本对齐 +- 2026-04-20 已完成一轮 live 验收打钩: + - `controller` 已生成 `single_step / detect_baidu_site` 任务(例:`job_id=6`, `job_code=step-20260420153527-9fc7b5`) + - `worker` 已精准领取当前 `job_id` 并执行,不再被旧 `pipeline / sync / legacy fallback` 污染 + - `worker` 已回传标准化结果到 `detect_job_items.result_payload_json` + - `controller` 已可按 `job_id` 定向执行 `process_pipeline`,并把结果落到本地 `domain_detections.baidu_site` + - “超时/遗留未回传”链路已验证会被节点启动释放并重新领取执行(`job_id=5` 在遗留 `running` 后被重新释放、重新领取并完成) +- 当前主线结论:大任务 1 已达到“可验收通过”状态,可以继续把主精力切到大任务 2 / 3 + +### 大任务 2 +pipeline 编排器落地 + +目标: +把“下一步跑什么”彻底收回 controller。 + +当前状态: +- 基础方向已落到 controller 驱动 +- 还需要继续做“按后台勾选顺序推进 / 按域名属性跳过”的真实验收 + +本任务必须确认: +- 同一个域名不会在 worker 里串完整条链 +- controller 是唯一推进者 +- 勾选顺序变化能影响下一步投递 +- 跳过规则生效 + +当前结论: +- 已进入可验收阶段 +- 2026-04-20 已补上关键收口: + - `single_step` 会话下,worker 只按当前 `job_id` 领取任务 + - `single_step` 会话下,旧的 `pipeline 推进 / sync pull / legacy fallback` 已被显式跳过 + - `process_pipeline` 已支持按 `job_id` 定向处理,方便 controller 精准推进当前步骤 +- 2026-04-20 又补完一轮 live 验收: + - `pass`:`detect_baidu_site` 完成后,controller 已为同一 `job` 创建下一步 `detect_360_site` + - `skip`:一口价域名在 `detect_register + detect_baidu_site` 配置下,会直接解析到 `detect_baidu_site` + - 同时修掉了一个真实阻塞点:`process_pipeline()` 事务内再开第二连接写 `detect_run_events` 会把 controller 自己锁住;现已改成同事务同 cursor 写事件 +- 当前主线结论:大任务 2 的 controller 编排主干已可验收,剩下更多是扩步骤和补更细跳过规则 + +### 大任务 3 +步骤结果判定与重试策略落地 + +目标: +统一 `pass / retry / black_hit / reject`。 + +当前状态: +- 结果结构、重试入口、黑名单终止方向已基本进入主链 +- 还需要补 live 验收,重点是 TTL 回收、重投、终止规则 + +本任务必须确认: +- 外部失败会重投当前步骤 +- 黑名单命中会终止后续步骤 +- 非黑名单业务不通过按规则终止 +- 不会出现同一步无限重试 + +当前结论: +- 代码已明显推进 +- 2026-04-20 已完成 controller 侧 live 验收: + - `retry`:`state=degraded` 会重投当前步骤,不推进下一步 + - `black_hit`:`state=blacklisted` 会终止后续步骤并结束当前 job + - `reject`:`state=rejected` 会终止当前流程,不再重试 + - worker 侧也已补上业务失败 -> `rejected` 的结果态,不再把业务不通过和技术失败都混成 `failed` +- 当前主线结论:大任务 3 的统一结果状态机已基本闭环,可继续往大任务 4 / 5 推进 + +### 大任务 4 +本地控制状态与海外主库同步落地 + +目标: +controller 本地维护高频状态,syncer 批量同步海外主库,finalizer 标记流程完成。 + +当前状态: +- 本地状态和部分同步链路已经在跑 +- 已补上“最近完成任务快照也继续产出结果投影” +- 已补上结果导入后按 job_code 优先定位本地 job,并刷新本地 job 收尾状态 +- syncer/finalizer 还缺少一轮 live 闭环验收 + +本任务必须确认: +- 高频链路不依赖 worker 频繁直写海外主库 +- controller 本地状态完整 +- 批量同步成功 +- 流程完成状态准确 + +当前结论: +- 代码主链已进一步收口 +- 2026-04-21 已确认 `detect_result_projection / runtime_projection` 在 `detect_sync_records` 中持续产出且状态为 `projected` +- 当前说明 syncer 主链已恢复,但“本地 job 收尾状态 + 主库最终账本完全一致”的终验还需要继续盯现场 + +### 大任务 5 +worker 池化与持续补位调度落地 + +目标: +真正替掉“批量认领 + 批量等待”的旧模型。 + +当前状态: +- 已经做了多轮并发热路径优化 +- 已进一步压缩 executor 内部 backlog,补位更接近固定槽位模型 +- 但还需要一轮 live 运行观察 claimed/running 曲线,确认旧的批量认领惯性已被压住 + +本任务必须确认: +- 活跃槽位稳定贴近配置上限 +- 并发不再大起大落 +- 吞吐明显提升 +- 调度器自耗下降 + +当前结论: +- 这是主线里仍然偏重的未完项 +- 2026-04-21 已继续做现场修正: + - `claim_detect_job_items()` 已显式排除空 `step_code` 的 legacy 项,避免 worker 从标准队列入口继续误吞旧 whole-domain 项 + - `mainland-controller-01 / 121.204.244.188` 与 `mainland-worker-01 / 121.204.244.248` 均已确认在真大陆节点参与执行 + - 海外控制面 `queue_health / dashboard` 已补上“读取大陆 runtime/debug 近窗执行流”的兜底口径 + - 当前首页已能看到真实近窗吞吐,例如 `10-15 项/分钟` 量级、并能拆到 controller/worker 两个节点 +- 当前主线判断: + - “高 claimed 假活跃”问题已继续缓解 + - “活吞吐不可见”问题已明显改善 + - 但“总 completed 账本持续增长”和“running 贴近线程上限”仍未彻底验收,所以大任务 5 仍未签字通过 + +### 大任务 6 +时光机一期流程落地 + +目标: +时光机按“最近 5 年”方案接成标准步骤任务。 + +当前状态: +- 现有项目里已有时光机相关检测逻辑 +- 但还没有完全按标准步骤任务方式接进新 pipeline 验收 + +本任务必须确认: +- wayback 作为标准步骤任务接入 +- 最近 5 年快照策略稳定 +- 返回标准化结果 +- controller 能把它当普通步骤推进/终止 + +当前结论: +- 2026-04-21 已落代码并通过测试: + - `detect_wayback` 已作为标准 `single_step` 步骤接入 controller / worker 主链 + - 时光机一期已按“最近 5 年 + 命中即停”策略落地到 payload 和 detector + - 焦点测试已通过 +- 但还缺 live 现场验收: + - 真实任务投递 + - worker 执行 + - 标准化结果回传 + - controller 按普通步骤推进/终止 +- 当前主线结论:大任务 6 已进入“代码落地完成、待现场验收”状态 + +### 大任务 7 +运营视角指标与验收面板落地 + +目标: +让后台能判断“有没有跑起来、卡在哪一步、多久跑完”。 + +当前状态: +- 后端已有部分 runtime / detect status / debug event 统计基础 +- 但完整的运营视角指标面板还没有完全落地验收 + +本任务必须确认: +- 每步骤队列数 +- 每步骤吞吐 +- 每分钟完成量 +- 当前 pipeline 分布 +- 重试数 / 黑名单数 / 失败数 +- 预估剩余时间 + +当前结论: +- 2026-04-21 已落地首页最小运营面板,并补了第二层口径修正: + - `completed / pending / running` 已切到“全活跃任务累计口径” + - `步骤队列 / 节点吞吐 / 重试压力 / 有效执行节点` 已可直接展示 + - `ETA` 在无真实近窗完成量时会显示“待计算”,避免误报 0 小时 + - 2026-04-21 又补上“大陆 runtime/debug 近窗执行流”兜底: + - `active_job` 已可显示 `runtime_job_code` + - `processed_per_minute / completed_recent / node_throughput / step_queue` 已能贴近大陆真实执行 + - 首页已能直接区分 `db_job_code` 与 `runtime_job_code` +- 当前结论:大任务 7 已进入“可运营分析并能指导现场排障”阶段,但仍需继续把“总 completed 账本”和“最终验收面板”完全统一 + +## 现在只做什么 + +当前只盯主线,不跑偏: + +1. 把大任务 1 做成可真实验收 +2. 验收过后立刻推进大任务 2 +3. 再按顺序推进 3、4、5、6、7 + +如果某个问题只是: + +- CPU 没吃满 +- 某个 timeout 还能再抠 +- 代理池还能更激进 +- 页面还能再改得更好看 + +都先不打断主线。 + +## 已封存的细节优化 backlog + +下面这些不是不做,而是先封存: + +### A. 并发/调度优化 + +- 固定 worker pool 彻底替换旧批量认领模型 +- claimed 回弹继续压缩 +- executor 内部排队继续收紧 +- 活跃 running 继续往上抬 +- controller / worker 双节点吞吐平衡 + +### B. DB 往返优化 + +- 继续合并 `domains` 表高频更新 +- 继续减少 `mark_running / finalize` 这类必要写库点开销 +- 能走 Redis 或本地状态的尽量不走高频 DB + +### C. 外部请求链路优化 + +- 代理失败后的重试链继续压缩 +- 直连失败后回代理的等待窗口再收紧 +- 无代理窗口等待策略再优化 +- 外部请求 timeout 再按真实成功率调优 + +### D. 代理池策略优化 + +- 代理池刷新频率与补位策略继续增强 +- 失败代理淘汰与新代理拉取节奏继续优化 +- 控制“不过度预验证”和“不过度浪费线程”之间的平衡 + +### E. 运营展示层优化 + +- 更细的 runtime 面板 +- ETA / 每分钟吞吐更精细展示 +- 日志窗口布局继续优化 +- 非主线的页面交互增强 + +## 接下来执行口径 + +接下来统一按这个口径推进: + +- 先验收主流程 +- 再补主流程缺口 +- 性能细节先记账,不抢主线 +- 每做完一个大任务,就给出“是否验收通过”的明确结论 + +一句话定调: + +当前阶段不是“继续无限抠并发”,而是“先把 7 个大任务按主线流程逐个打通并验收”。 diff --git a/docs/codex接手.md b/docs/codex接手.md new file mode 100644 index 0000000..08f385b --- /dev/null +++ b/docs/codex接手.md @@ -0,0 +1,49 @@ +把下面这段直接发给新的 Codex 就行: + +```text +接手这个项目,请先不要发散,也不要先动展示层。你先完整阅读并基于现状继续推进主线。 + +项目路径: +`/www/wwwroot/getDomain` + +必须先看这几份文档: +1. `docs/ops_center_runtime/HANDOFF_20260420_1920.md` +2. `docs/base.md` +3. `docs/test.md` + +这次接手的硬性要求: +1. 先以 `docs/ops_center_runtime/HANDOFF_20260420_1920.md` 为准建立上下文。 +2. 不要把当前这台机器当成真正的 mainland controller。 +3. 当前工作机是海外 12 核测试控制面,真正的 `mainland-controller-01` 是 `121.204.244.188`。 +4. 不要再优先动页面、展示层、面板文案。 +5. 不要用破坏性 git 命令,不要回滚现有脏工作区改动。 +6. 新增或修改代码前,先确认你改的是主线瓶颈,而不是辅助功能。 + +当前已经确认的事实: +1. 真正的 `mainland-controller-01` 已重新对正。 +2. controller 的 node-agent 身份上报已经修好,现在控制面里应显示: + - `agent_hostname = mainland-controller-01` + - `agent_ip = 121.204.244.188` +3. controller 远端发布链已经成功跑通过一次。 +4. 当前真正吃任务的是 `mainland-controller-01`。 +5. `mainland-worker-01` 目前是 agent 在线,但检测没有真正参与,现象是本地检测态里出现 `127.0.0.1:5432 connection refused`。 +6. 当前主线已经不是“节点身份问题”,而是“真 controller 吞吐”和“worker 恢复”。 + +你接手后只做两条主线: +1. 恢复 `mainland-worker-01`,让它重新进入可参与检测状态。 +2. 继续压 `mainland-controller-01` 的真实吞吐,只盯 `claimed -> running -> completed` 的推进,不回展示层。 + +你要避免的坑: +1. 不要再把本机当成 `mainland-controller-01`。 +2. 不要优先用本机 Python service 入口直接造 deploy job,优先通过运行中的 API HTTP 接口。 +3. 不要把 `job 332` 这种“node-agent 重启自己导致状态未优雅回写”的现象直接误判为真正失败。 +4. 不要跑偏到 UI、导出、日志窗口样式这些支线。 + +你开始后先做这三件事,再继续动手: +1. 复述你理解的当前真实环境拓扑。 +2. 复述当前两条唯一主线任务。 +3. 给出你准备先验证的 3 个现场指标,再开始执行。 + +目标只有一个: +先把主流程和真实并发跑起来,让 controller 真机和 worker 真正参与检测,再谈细节优化。 +``` diff --git a/docs/ops_center_runtime/HANDOFF_20260419_0308.md b/docs/ops_center_runtime/HANDOFF_20260419_0308.md new file mode 100644 index 0000000..48022e6 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_0308.md @@ -0,0 +1,96 @@ +# HANDOFF 2026-04-19 03:08 + +## 本轮做了什么 + +本轮没有进入新实现,只做了最小链路核查: + +1. 复查中央 `detect/job/active` +2. 复查中央 `ops/nodes` +3. 复查 `go-live-summary` / `stack-diagnosis` / `release-launchpad` +4. 复查最新 `onboarding.acceptance` +5. 通过远端 Agent 对 `mainland-controller-01` 执行: + - `service.status` + - `logs.collect` + 目标服务:`domaincheck-sync-agent` + +## 本轮结论 + +当前状态已经进一步推进: + +- 检测任务在跑 +- 节点接管在跑 +- 最新 acceptance 已成功 +- `mainland-controller-01` 的 `domaincheck-sync-agent` 已重启成功 +- 重启后首轮已把 mainland 逐条结果推回中央 + +## 关键证据 + +### 1. 检测任务确实在跑 + +- 活跃任务:`detect-20260417170546-96023e` +- 参与节点: + - `mainland-worker-01` + - `mainland-controller-01` + - `overseas-control-01` + +### 2. 最新 acceptance 已成功 + +- `pbr-9ce5c85f17`:成功 +- `pbr-389abd618c`:成功 + +说明: + +- 接管验收不再是当前唯一主阻塞 + +### 3. sync-agent 服务已重启到新进程 + +远端只读结果: + +- 服务:`domaincheck-sync-agent` +- 状态:`active (running)` +- 启动时间:`2026-04-19 16:09:52 CST` +- 运行命令: + - `/opt/domaincheck/domainCheck/.venv/bin/python -m app.sync_agent` + +### 4. 重启后首轮结果投影已推送成功 + +最新日志显示: + +- `detect_result_projection` + - `batch_count = 1` + - `success_count = 1` + - `event_import.imported_count = 10` + +### 5. 中央 mainland 逐条结果已出现 + +- 新增 `detect_result_ingest` + - `id = 5382` + - `created_at = 2026-04-19 03:10:14` +- 近期 mainland `domain_*` + - `count = 10` + +## 下一轮唯一剩余动作 + +下一轮不要发散实现,只做持续性复查: + +1. `/api/v1/runtime/sync-summary` +2. `/api/v1/detect/job/active` +3. mainland `domain_started/domain_completed/domain_failed/domain_blacklisted` +4. `/api/v1/ops/go-live-summary` +5. `/api/v1/ops/stack-diagnosis` + +重点判断: + +- mainland `domain_*` 是否持续增长 +- 总检 attention 是否已主要退化为历史残留 + +## 继续不要做什么 + +- 不做新页面 +- 不做控制面增强 +- 不做发布动作 +- 不做 item 级最终回写 + +## 一句话结论 + +`mainland-controller-01` 的 `domaincheck-sync-agent` 重启后,逐条结果同步已经打通;当前工作重点从“修链路”切换到“验证持续性与上线签收口径”。 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_0318.md b/docs/ops_center_runtime/HANDOFF_20260419_0318.md new file mode 100644 index 0000000..0da0979 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_0318.md @@ -0,0 +1,128 @@ +# HANDOFF 2026-04-19 03:18 + +## 本轮做了什么 + +本轮没有进入新实现,只做了运行态只读复查: + +1. 对比一个完整 40 秒观察窗口前后的: + - `/api/v1/detect/job/active` + - `/api/v1/runtime/sync-summary` +2. 复查中央 `detect_run_events` +3. 复查: + - `/api/v1/ops/go-live-summary` + - `/api/v1/ops/stack-diagnosis` + - `/api/v1/ops/activity-stream` +4. 复查节点现场日志: + - `mainland-controller-01` + - `mainland-worker-01` + +## 本轮结论 + +这轮结论要纠偏: + +- 接管链路基本完成 +- 同步链路已经打通过一次 +- 但检测执行面当前没有继续出新结果 + +因此当前主阻塞不再是“接管/同步未通”,而是: + +- 检测执行停滞 +- 外部站点依赖或代理池可用性异常 + +## 关键证据 + +### 1. 三台节点都在参与,但近窗没有吞吐 + +活跃任务仍是: + +- `detect-20260417170546-96023e` + +参与节点: + +- `mainland-controller-01` +- `mainland-worker-01` +- `overseas-control-01` + +但 40 秒观察窗口前后完全一致: + +- `progress_percent = 2.1` +- `completed = 21` +- `running = 14` +- `claimed = 34` +- `pending = 931` + +### 2. mainland 逐条结果没有继续增长 + +中央查询结果: + +- mainland `domain_*` 事件数仍为 `10` +- 最新 mainland `detect_result_ingest` 仍是: + - `id = 5382` + - `created_at = 2026-04-19 03:10:14` + +说明: + +- 首批同步成功过 +- 但后续没有继续流入新结果 + +### 3. go-live 与 stack 口径已经比之前更完整 + +当前: + +- `remote_access_ready = 3/3` +- `log_sync_state = full_capture` +- `go_live_status = attention` + +这说明: + +- 基础运维骨架已经起来 +- attention 现在不能只归因于接管未完成 + +### 4. controller 现场日志已直接指向代理/外部依赖问题 + +`mainland-controller-01` 现场日志显示: + +- `当前可用代理数: 0` +- `最近结果: 刷新成功,可用 0 个` +- 检测链包含: + - 注册 + - 百度 site + - 360 site + - 站长之家 + - 爱站 + - 时光机 + +### 5. 页面上的“外部站点异常”与现场证据一致 + +从当前现场判断: + +- 这不是页面误报 +- 而是执行面确实卡在外部依赖/代理可用性上 + +## 下一轮唯一应该做什么 + +下一轮不要发散实现,只做 `J2-检测执行停滞收口批`: + +1. 继续只读确认: + - `/api/v1/detect/job/active` + - `/api/v1/runtime/sync-summary` + - `/api/v1/ops/activity-stream` + - `/api/v1/ops/nodes/{node_code}/scene-log` +2. 必要时补一轮: + - `domaincheck-worker` 运行日志取证 +3. 只判断三件事: + - 是否仍然 `近窗吞吐 0` + - 是否仍然 `可用代理数 0` + - 是否仍然没有新 `domain_*` 事件 + +## 现在不要做什么 + +- 不做新页面 +- 不做控制面增强 +- 不做发布动作 +- 不做 item 级最终回写 +- 不把当前状态误判成“已稳定可签收” + +## 一句话结论 + +当前项目已经不是“接不管、看不见、不同步”,而是“接管和同步都基本打通了,但检测执行卡在外部站点/代理可用性问题上,导致任务挂起且没有继续产出”。 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_0324.md b/docs/ops_center_runtime/HANDOFF_20260419_0324.md new file mode 100644 index 0000000..05d1058 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_0324.md @@ -0,0 +1,117 @@ +# HANDOFF 2026-04-19 03:24 + +## 本轮做了什么 + +本轮没有进入新实现,只做了执行面根因收紧: + +1. 继续观察活跃检测任务是否前进 +2. 继续观察 mainland `domain_*` 是否增长 +3. 通过正式 `ops job` 远端采集: + - `mainland-controller-01` + - `mainland-worker-01` + 的 `domaincheck-worker` 日志 +4. 再盯一个 35 秒窗口,看 full capture 打开后,源日志时间是否继续前进 + +## 本轮结论 + +这轮已经能把问题说得更准: + +- 不是“日志没回来” +- 不是“全量日志没开” +- 而是日志链路已经通了,但执行进程没有继续产生日志 + +当前第一主阻塞: + +- `mainland-controller-01` 代理池可用数为 0 + +次级问题: + +- `mainland-worker-01` 的时光机依赖异常会降级继续执行 + +## 关键证据 + +### 1. full capture 是开的,但源日志没继续前进 + +当前: + +- `mainland-controller-01` + - `capture_at = 2026-04-19 03:21:59` + - 源日志时间仍停在 `Apr 19 01:00:23` +- `mainland-worker-01` + - `capture_at = 2026-04-19 03:22:01` + - 源日志时间仍停在 `Apr 19 02:19:56` + +35 秒后再次复查: + +- `capture_at` 没变化 +- `source_msg` 也没变化 + +说明: + +- 日志回传本身不是主问题 +- 执行进程这段时间没有继续产生日志 + +### 2. controller 代理源能拉到数据,但所有代理都验不过 + +`mainland-controller-01` 的 `domaincheck-worker` 日志显示: + +- 6 个代理源都能拉到原始代理 +- 抽样验证 24 个代理后: + - `代理池刷新完成,共 0 个可用代理` +- 失败集中在: + - `ProxyError@https://m.baidu.com` + - `Unable to connect to proxy` + - `ConnectTimeoutError` + +说明: + +- 不是代理接口挂了 +- 是代理名单本身不可用 + +### 3. worker 还能跑,但时光机依赖异常会降级 + +`mainland-worker-01` 的 `domaincheck-worker` 日志显示: + +- `时光机检测 外部依赖异常,步骤降级继续执行` +- 同时仍可见: + - `域名检测完成` +- 收到新的控制消息时: + - `收到启动检测指令,但检测任务已在运行,忽略重复启动` + +说明: + +- worker 不是完全不可用 +- 时光机异常存在,但不是最核心阻塞 + +### 4. 活跃任务仍然没有前进 + +- `progress_percent = 2.1` +- `completed = 21` +- `running = 14` +- `claimed = 34` +- `pending = 931` + +并且 mainland `domain_*` 仍固定在 `10` + +## 下一轮唯一应该做什么 + +继续只做 `J2-检测执行停滞收口批`: + +1. 不扩功能 +2. 不改页面 +3. 只围绕 controller 代理池问题取证和恢复验证 + +唯一要确认的是: + +- controller 代理池是否仍然 `可用 0` +- 一旦代理恢复,`domain_*` 是否会继续增长 + +## 现在不要做什么 + +- 不把问题继续泛化成“外部站点都异常” +- 不把问题误判为“日志回传没开” +- 不做新页面、新模块、发布动作 + +## 一句话结论 + +当前项目不是“接管失败”,也不是“日志没回来”,而是“controller 侧拉到了代理名单,但代理全部校验失败,导致检测执行没有继续产生新结果”。 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_0426.md b/docs/ops_center_runtime/HANDOFF_20260419_0426.md new file mode 100644 index 0000000..21b7641 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_0426.md @@ -0,0 +1,66 @@ +# HANDOFF 2026-04-19 04:26 CST + +## 本轮完成 + +- 已修复 `runtime/status` 兼容层,恢复旧字段: + - `api_online` + - `worker_online` + - `cluster_summary` + - `thread_count` +- 已重启中央 `domaincheck-api` +- 已重新构建 `domain-web` 前端静态包 +- 已确认大陆双节点日志回传同时进入中央: + - `mainland-controller-01` + - `mainland-worker-01` +- 已确认大陆两台线程配置真实生效: + - `mainland-controller-01 -> 100` + - `mainland-worker-01 -> 50` + +## 当前真实状态 + +- `/api/v1/runtime/status` + - `api_online = true` + - `worker_online = true` + - `cluster_summary.online_worker_nodes = 3` +- `/api/v1/detect/status` + - `progress.pending = 163815` + - `progress.completed = 51` + - `progress.running = 10` + - `remote_log_line_count = 240` + - `remote_log_node_count = 2` +- 近 5 分钟中央 `detect_debug_events` + - `mainland-controller-01 -> worker_log` + - `mainland-worker-01 -> worker_log` + +## 当前结论 + +- “API 离线 / 本机 Worker 离线 / 有效执行节点 0” 已不是后端真实状态 +- “全量日志没回来” 已不是后端真实状态 +- “100/50 并发没有下发” 也不是后端真实状态 +- 当前剩余主问题已经收紧为: + - 检测实际吞吐仍偏低 + - 代理可用性与任务分发节奏仍在限制体感并发 + +## 下一轮唯一任务 + +只做:`J2-检测执行吞吐收口` + +顺序: + +1. 强刷浏览器,确认新前端包已经生效 +2. 复查 Detect 页是否恢复: + - API 在线 + - 本机 Worker 在线 + - 有效执行节点 > 0 + - 日志窗口出现双节点日志 +3. 若显示已恢复,再继续只排查: + - 为什么实际执行吞吐仍低于 `100/50` + - 重点看代理可用性、任务领取节奏、线程实际活跃数 + +## 现在不要做 + +- 不扩新页面 +- 不扩控制面功能 +- 不新增专题文档 +- 不进入发布动作 +- 不切新大方向 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_1334.md b/docs/ops_center_runtime/HANDOFF_20260419_1334.md new file mode 100644 index 0000000..b262251 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_1334.md @@ -0,0 +1,104 @@ +# HANDOFF 2026-04-19 13:34 CST + +## 本轮完成 + +- 已修复 controller `sync-agent pull_tasks` 只导 `domains`、不导本地执行队列的问题 +- 已让 controller 持续创建本地镜像任务: + - `sync-overseas-7380` + - `sync-overseas-7383` + - `sync-overseas-7392` + - 后续批次持续增加 +- 已重启两台大陆 `domaincheck-worker` +- 已确认两台大陆节点都进入真实任务队列: + - `mainland-controller-01 -> 从任务队列获取到 800 个需要检测的域名` + - `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名` +- 已修复 controller `runtime/detect_runs.json` 属主错误: + - `root:root -> www:www` +- 已确认 `runtime_projection` 最新同步恢复成功: + - `sync_state = success` + - `runtime_projection -> 投影推送成功` + +## 当前真实状态 + +当前 `/api/v1/ops/nodes` 已显示: + +- `remote_access_ready = 3` +- `participating = 3` +- `dispatch_active = 1` + +大陆两台当前状态: + +- `mainland-controller-01` + - `agent_state = online_busy` + - `detect_runtime.max_threads = 100` + - `processed_recent = 115` + - `processed_per_minute = 7.67` + - `is_current_participant = true` +- `mainland-worker-01` + - `agent_state = online_busy` + - `detect_runtime.max_threads = 50` + - `processed_recent = 632` + - `processed_per_minute = 42.13` + - `is_current_participant = true` + +## 当前结论 + +- 大陆两台不是“在线但没干活” +- 而是已经进入真实镜像队列执行 +- `100/50` 并发也不是停留在配置层,而是已经开始真实消费任务 +- 后台运行态同步链已经恢复 + +现在剩余问题已经收敛到: + +- Detect 页面主计数 / 日志窗口是否完全跟上新的运行态 +- 恢复后的吞吐是否能持续稳定 + +## 一个关键口径说明 + +当前大陆执行是“镜像队列”模式: + +- 海外待检测批次先被 controller 拉回大陆本地 +- 在大陆本地形成 `sync-overseas-*` 的 `detect_jobs/detect_job_items` +- 再由大陆 worker 真正执行 +- 运行态和结果再同步回海外 + +所以现在不要再只盯中央原生 `detect_job_items.claimed/running` 判断大陆有没有参与。 + +应该优先看: + +- `/api/v1/ops/nodes` +- `processed_recent` +- `processed_per_minute` +- `detect_runtime.active_threads/max_threads` + +## 下一轮唯一任务 + +只做:`J2-页面口径与吞吐稳定性收口` + +顺序: + +1. 继续观察 Detect 页面 +2. 确认日志窗口是否已经持续刷新 +3. 确认主计数是否开始跟随最新运行态 +4. 继续观察 1 到 2 个同步周期内: + - `mainland-controller-01 processed_recent` + - `mainland-worker-01 processed_recent` + - `processed_per_minute` +5. 若页面仍不跟,优先修页面读取口径,不扩功能 + +## 现在不要做 + +- 不扩新页面 +- 不扩控制面功能 +- 不新增模块 +- 不进入发布动作 +- 不切新方向 + +## 推荐模型 + +- 当前最合适:`GPT-5.4 + high` + +原因: + +- 现在主要是收口、验证、局部修复 +- 需要稳定推理,但不需要切到超高 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_1348.md b/docs/ops_center_runtime/HANDOFF_20260419_1348.md new file mode 100644 index 0000000..012b1e3 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_1348.md @@ -0,0 +1,109 @@ +# HANDOFF 2026-04-19 13:48 CST + +## 本轮完成 + +- 已定位并修复 `domainCheck/detect_worker.py` 的上下文绑定缺口: + - `sync-pull` 控制消息只有 + - `target_job_id` + - `target_job_code` + - 旧逻辑只读取 + - `job_id` + - `job_code` + - 结果是 `mainland-worker-01` 明明已经在跑,但 `current_job_id` 为空,`worker_log` 被整段短路 +- 已把补丁同步到: + - `mainland-controller-01:/opt/domaincheck/domainCheck/detect_worker.py` + - `mainland-worker-01:/opt/domaincheck/domainCheck/detect_worker.py` +- 已在两台大陆机完成: + - `python -m py_compile /opt/domaincheck/domainCheck/detect_worker.py` + - `systemctl restart domaincheck-worker` + +## 当前硬证据 + +### 1. mainland worker 日志回传已恢复 + +当前 `/api/v1/runtime/debug-events` 已出现: + +- `mainland-worker-01 -> 开始执行检测任务,来源: redis-control` +- `mainland-worker-01 -> 开始执行域名检测任务,正在加载配置` +- `mainland-worker-01 -> 开始检测,正在刷新代理池` +- `mainland-worker-01 -> 代理池刷新完成,共 7 个可用代理,来源链接 6 个,原始 265 个,验证 150 个` +- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名` +- `mainland-worker-01 -> 开始创建线程,当前批次域名数: 400,最大线程数: 50` +- `mainland-worker-01 -> 当前实际线程数量: 1/50` +- `mainland-worker-01 -> 当前实际线程数量: 2/50` +- `mainland-worker-01 -> 当前实际线程数量: 3/50` + +### 2. Detect 页面远端日志已经恢复双节点 + +当前 `/api/v1/detect/status` 已显示: + +- `remote_log_node_count = 2` +- `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]` + +最近日志样本: + +- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名` +- `mainland-worker-01 -> 开始创建线程,当前批次域名数: 400,最大线程数: 50` +- `mainland-worker-01 -> 当前实际线程数量: 1/50` +- `mainland-worker-01 -> 当前实际线程数量: 2/50` +- `mainland-worker-01 -> 当前实际线程数量: 3/50` +- `mainland-controller-01 -> 当前实际线程数量: 1/100 ... 3/100` + +### 3. 三台节点仍保持参与态 + +当前 `/api/v1/ops/nodes` 摘要: + +- `online = 3` +- `agent_ready = 3` +- `remote_access_ready = 3` +- `participating = 3` +- `dispatch_active = 1` + +## 当前判断 + +已经闭合: + +- 大陆节点真实执行 +- mainland worker 日志回传 +- Detect 页面远端日志窗口只显示 controller 的问题 + +仍待收口: + +- Detect 页面主计数: + - `pending = 163678` + - `completed = 195` + - `running = 3` +- 当前这些主计数还没有和这轮大陆执行立即同步推进 + +因此当前剩余唯一主问题已经缩成: + +- `detect_result_projection / 结果统计回推` 是否还存在延迟或聚合口径缺口 + +## 下一轮唯一任务 + +只做:`J2-结果计数收口批` + +顺序: + +1. 复查 `detect_result_projection` 最新推送是否持续成功 +2. 复查 mainland 共享库里 `sync-overseas-*` 任务的完成/失败数量是否增长 +3. 复查 overseas `detect/status.progress` 是否跟着推进 +4. 若日志继续推进但主计数仍不动,只修结果统计回推口径,不扩功能 + +## 现在不要做 + +- 不扩新页面 +- 不扩控制面功能 +- 不新增模块 +- 不切新方向 +- 不直接进入发布动作 + +## 推荐模型 + +- 当前继续用:`GPT-5.4 + high` + +原因: + +- 现在是收口型问题 +- 需要稳定排查与小范围补丁 +- 不需要切超高推理 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_1416.md b/docs/ops_center_runtime/HANDOFF_20260419_1416.md new file mode 100644 index 0000000..ed346dc --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_1416.md @@ -0,0 +1,65 @@ +# HANDOFF 2026-04-19 14:16 CST + +## 本轮完成 + +- 已修复 `detect_worker.py` 中“代理不足反向限制并发”的热路径: + - 代理池不足时改为后台补货 + - 不再因为 `current_pool_size < thread_count/2` 而同步刷新、拖慢线程拉起 +- 已对线程创建阶段的运行态 / worker_log 做节流: + - 不再每起一个线程就同步打一轮重日志 + - 避免远端日志回传本身把并发拉低 +- 已把上述修复部署到: + - overseas 当前 live 运行目录 `/opt/domaincheck/domainCheck/detect_worker.py` + - repo 目录 `/www/wwwroot/getDomain/domainCheck/detect_worker.py` + - `mainland-controller-01` + - `mainland-worker-01` +- 已补上“真实活跃检测线程计数”逻辑到代码文件中 + +## 当前结果 + +- `mainland-controller-01` + - 海外后台当前已能看到: + - `active_threads ~= 99~100` + - `max_threads = 100` + - 结论: + - controller 的 `100` 并发已经真实跑起来 +- `mainland-worker-01` + - 已重新接到新批次: + - `source_record_id = 7679` + - `target_job_code = sync-overseas-7679` + - 最新日志已出现: + - `开始执行域名检测任务` + - `开始检测,刷新代理池` + - `代理池刷新完成,共 23 个可用代理` + - `开始创建线程,当前批次域名数: 400,最大线程数: 50` + - 本机进程线程量: + - `152` + - 结论: + - worker 已重新进入真实执行 + - 但 overseas 后台对 `mainland-worker-01.active_threads/current_load` 的显示仍偏低 + +## 当前剩余问题 + +- 不是 controller 并发限制问题 +- 当前唯一剩余的运行面收口点是: + - `mainland-worker-01` 的运行态上报 / 后台显示口径仍未完全对齐 +- 同时仍需继续观察: + - Detect 页面主计数 `pending/completed/running` + - 结果统计回推是否继续推进 + +## 下一轮建议 + +只继续做这两件事: + +- 追 `mainland-worker-01` 的运行态采集链: + - 为什么本机已进入新批次执行,但海外后台仍显示 `active_threads = 2` +- 继续核对 Detect 主计数与结果回推: + - 判断是统计延迟、投影延迟,还是任务结果尚未进入中央口径 + +不要做: + +- 新页面 +- 新模块 +- 控制面增强 +- 发布动作 +- 新专题文档 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_1421.md b/docs/ops_center_runtime/HANDOFF_20260419_1421.md new file mode 100644 index 0000000..0e36c20 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_1421.md @@ -0,0 +1,52 @@ +# HANDOFF_20260419_1421 + +更新时间:2026-04-19 14:21 CST + +## 本轮完成 + +- 已完成检测页前端布局收口并上线验证: + - `任务日志控制台` 已上移到事件列表前 + - `检测事件流` 已改成独立滚动区域 + - 事件表已限制最大高度,避免页面被长列表持续撑长 +- 已在真实线上静态目录重新构建: + - `/www/wwwroot/getDomain/domain-web/dist` +- 已验证线上 Nginx 实际指向该目录: + - `/www/server/panel/vhost/nginx/domaincheck_3201.conf` + - `/www/server/panel/vhost/nginx/domaincheck_152.53.37.118.conf` +- 已验证本机线上端口可返回新页面: + - `curl -I http://127.0.0.1:3201/ -> 200` + - `index.html` 最后修改时间已更新到本轮 + - 新资源: + - `/assets/DetectView-B9S1WMRT.js` + - `/assets/DetectView-BFxMNRNR.css` + +## 当前判断 + +- 这轮前端布局任务已经闭环 +- 如果浏览器仍显示旧布局,优先判断为缓存未刷新 +- 当前更值得继续追的不是页面结构,而是: + - `mainland-worker-01` 运行态上报口径 + - Detect 主计数推进 + - 并发真实吞吐继续拉升 + +## 下一步建议 + +下一轮继续只围绕检测执行收口: + +- 复核 `mainland-worker-01` 为什么本机线程已拉起,但后台 `active_threads/current_load` 仍偏低 +- 继续比对: + - `/api/v1/detect/status` + - `/api/v1/runtime/debug-events` + - `/api/v1/ops/nodes` +- 继续验证 Detect 页面主计数是否跟随真实执行推进 + +## 本轮关键命令证据 + +```bash +cd /opt/domaincheck/domain-web && npm run build +curl -I http://127.0.0.1:3201/ +curl http://127.0.0.1:3201/ +rg -n "event-stream-panel|任务日志控制台|检测事件流" \ + /www/wwwroot/getDomain/domain-web/dist/assets/DetectView-BFxMNRNR.css \ + /www/wwwroot/getDomain/domain-web/dist/assets/DetectView-B9S1WMRT.js +``` diff --git a/docs/ops_center_runtime/HANDOFF_20260419_1429.md b/docs/ops_center_runtime/HANDOFF_20260419_1429.md new file mode 100644 index 0000000..88e03ce --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_1429.md @@ -0,0 +1,54 @@ +# HANDOFF_20260419_1429 + +更新时间:2026-04-19 14:29 CST + +## 本轮完成 + +- 已修复代理链“配置已下发但长期停留未刷新”的问题 +- `domaincheck-worker` 现在会在以下时机主动刷新代理池: + - worker 启动后 + - `proxy_config` + - `thread_count` + - `node_thread_counts` + - `runtime_settings` + - 以上配置更新后 +- 已重启: + - `domaincheck-worker` + - `domaincheck-api` +- 已完成一次真实代理池刷新并拿到明确结果: + - 原始代理数:`202` + - 抽样验证数:`24` + - 可用代理数:`0` + +## 当前状态 + +- 当前后台不再显示误导性的“未刷新” +- 当前真实状态为: + - `proxy_runtime_label = 降级直连` + - `proxy_runtime_reason = proxy_validation_zero` + - `proxy_runtime_detail = 代理源最近返回了 202 个代理,已验证 24 个,但当前 0 个可用;系统已自动降级为直连继续执行;最近状态:刷新成功,可用 0 个` + +## 关键判断 + +- 这批代理当前的主要问题不是“系统没去用” +- 而是: + - 代理源会返回很多 IP + - 但 worker 实测校验时全部失败 + - 失败以 `ProxyError / ConnectTimeout` 为主 + +## 下一步建议 + +- 优先不要再纠结“为什么页面写未刷新” + - 这一层已经修好 +- 下一轮如果继续优化代理,应只做以下两种之一: + - 更换/补充代理供应组,重新验证可用率 + - 调整代理校验策略,但要接受更高的脏代理混入风险 + +## 现场证据 + +```text +主动触发代理池刷新: worker_startup +代理池链接拉取成功 ... 原始代理数: 50 / 50 / 0 / 50 / 2 / 50 +代理池验证已启用抽样模式,本次抽样 24 个代理进行可用性验证 +代理池刷新完成,共 0 个可用代理,来源链接 6 个 +``` diff --git a/docs/ops_center_runtime/HANDOFF_20260419_1450.md b/docs/ops_center_runtime/HANDOFF_20260419_1450.md new file mode 100644 index 0000000..867ee85 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_1450.md @@ -0,0 +1,42 @@ +# HANDOFF_20260419_1450 + +更新时间:2026-04-19 14:50 CST + +## 本轮完成 + +- 已继续优化代理链,不只停留在“看见 0 可用” +- 已新增两项优化: + - 扩展严格验证目标: + - `baidu` + - `m.baidu` + - `qq` + - `360` + - 当严格验证 `0` 命中时,启用: + - `宽松入池 + 快速隔离` + +## 当前结果 + +- 当前最新代理刷新结果: + - `proxy_last_refresh_total_items = 270` + - `proxy_last_validated_count = 24` + - `proxy_last_available_count = 2` + - `proxy_last_refresh_status = 宽松入池 2 个(严格校验 0 命中)` +- 后台当前已显示: + - `available_proxy_count = 2` + - `proxy_runtime_detail = 代理池当前可用 2 个代理,配置来源 6 个;最近状态:宽松入池 2 个(严格校验 0 命中)` + +## 关键判断 + +- 严格校验下,这批代理整体质量依然偏差 +- 但现在已经不再是完全 `0` 可用 +- 系统已成功放出少量试跑代理,可以继续观察: + - 是否带动检测吞吐 + - 是否很快被失败隔离重新打回 `0` + +## 下一步 + +- 优先观察这 `2` 个试跑代理是否真的参与检测执行 +- 如果能参与并带来吞吐,再继续逐步放宽 +- 如果很快再次掉回 `0`,下一轮就应转到: + - 补代理供应组 + - 或按地区/分组拆分代理质量统计 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_1454.md b/docs/ops_center_runtime/HANDOFF_20260419_1454.md new file mode 100644 index 0000000..d7fd6ce --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_1454.md @@ -0,0 +1,64 @@ +# HANDOFF 2026-04-19 14:54 CST + +## 本轮目标 + +把代理链从“预验证优先”切到用户指定的最小高效路径: + +- 只去掉过期 / 非法代理 +- 不再单独做可用性预验证 +- 直接投入真实检测 +- 失败即淘汰并触发补刷 + +## 本轮已完成 + +- 已修改 [detect_worker.py](/www/wwwroot/getDomain/domainCheck/detect_worker.py) + - `refresh_proxy_pool()` + - 移除预验证入池逻辑 + - 仅做过期 / 非法 / 去重过滤 + - 直接把代理源返回结果入池 + - `remove_proxy()` + - 代理失败后不再留在当前池尾部 + - 直接从当前池移除 + - 保留失败标记并在低水位时触发补刷 +- 已重启 `domaincheck-worker` +- 已确认 worker 新日志进入运行态 + +## 最新运行证据 + +- worker 最新日志: + - `代理池刷新完成,共 270 个入池代理,来源链接 6 个,原始 270 个,去过期 0 个,非法 0 个` +- API 最新状态: + - `available_proxy_count = 270` + - `proxy_last_refresh_status = 直入池 270 个(跳过预验证)` + - `proxy_last_validated_count = 0` + - `proxy_runtime_reason = healthy` + +## 当前判断 + +- 这轮目标已经完成 +- 代理链现在已经符合“直接跑任务,失败就丢弃代理并换新”的要求 +- 下一步不该再回头做单独代理预验证 + +## 下一步主任务 + +唯一主任务: + +- 继续观察真实检测吞吐是否随直入池策略抬升 +- 重点看: + - `/api/v1/detect/status` + - `/api/v1/ops/nodes` + - 检测页主计数 `pending/completed/running` + - 失败代理淘汰后是否能持续自动补池 + +## 若下一轮继续 + +优先处理: + +- “并发已下发但主计数不明显推进”的统计 / 调度口径问题 + +不要处理: + +- 不要重新加回重预验证逻辑 +- 不要扩展新页面 +- 不要扩展控制面新模块 +- 不要发散到发布链或新专题文档 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_2053.md b/docs/ops_center_runtime/HANDOFF_20260419_2053.md new file mode 100644 index 0000000..65bf3d3 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_2053.md @@ -0,0 +1,107 @@ +# HANDOFF_20260419_2053 + +更新时间:2026-04-19 20:53 CST + +## 本轮完成 + +- 修复发布打包遗漏: + - 最新发布包现在已包含 `domainCheck/` + - 最新正式签收包: + - `domaincheck_release_20260419_205437` +- 修复发布任务观测性: + - `domain-api/app/services/ops_release_executor_core.py` + - 发布目录不可写时显式失败 + - 增加 `ExecStart` 与 `current` 软链对齐观测 + - `domain-api/app/node_agent.py` + - 增加 job 级兜底异常回写,避免任务假性卡死 +- 将卡死的 worker 发布任务纠正为真实失败: + - `release_id = 6` + - `rollout_id = 5` + - `job_id = 204` + - 当前已从假 `running` 回正为 `failed` + - 本次失败对应的 Release 版本: + - `domaincheck_release_20260419_203933` + +## 今晚新增关键结论 + +### 1. worker 发布失败的真实原因已明确 + +- `mainland-worker-01` + - node-agent 日志: + - `2026-04-19 20:40:15 [node-agent] loop error: [Errno 13] Permission denied: '/opt/domaincheck/downloads'` +- 对应发布任务: + - `job 204 = failed` + - `rollout 5 = failed` +- 当前失败原因已经正式回写: + - `install_root not writable: /opt/domaincheck/downloads` + +### 2. mainland 两台节点都不符合当前 ReleaseHub 发布模型 + +- `mainland-worker-01` + - `domaincheck-worker` 当前启动路径: + - `/opt/domaincheck/domainCheck/detect_worker.py` +- `mainland-controller-01` + - `domaincheck-worker` 当前启动路径: + - `/opt/domaincheck/domainCheck/detect_worker.py` + +这意味着: + +- 当前线上服务不是跑在 + - `/opt/domaincheck/current/domainCheck/...` +- 所以即使发布成功切了 `current` 软链 +- 也不会自动让现有 systemd 服务切到新版本 + +### 3. worker 当前不是新版本高吞吐样本 + +- `mainland-worker-01` + - `Active since = 2026-04-19 18:18:50 CST` + - `Main PID = 635924` + - `CPU = 18.994s` + - 最近日志仍停留在旧代码行号: + - `_run_detect_register:2465` + - `detect_domain:2947` + - `start_redis_subscription:3590` + +说明: + +- worker 仍是旧进程 +- 当前不能把它当成“已切新包并真实参与高吞吐”的有效证据 + +## 结论 + +当前唯一主阻塞不是前端、不是统计口径、也不是单纯并发参数。 + +当前唯一主阻塞是: + +- mainland 节点 `deploy.release` 目录权限不满足 +- mainland 节点 `domaincheck-worker` 的 systemd `ExecStart` 与 ReleaseHub 的 `current` 发布模型不一致 + +在这两个问题修正前: + +- 不建议继续对 mainland 节点做正式 rollout +- 不建议继续把 worker 统计口径偏差当成纯展示层问题 + +## 下一步唯一建议 + +1. 先修 mainland 节点发布基座 + - 让 node-agent 对 `/opt/domaincheck/downloads`、`/opt/domaincheck/releases` 有写权限 + - 或明确改成一个 node-agent 真正可写的发布根目录 +2. 再统一 `domaincheck-worker.service` + - 让 `ExecStart` 指向 `/opt/domaincheck/current/domainCheck/detect_worker.py` + - 不再直指 `/opt/domaincheck/domainCheck/detect_worker.py` +3. 完成后重新发起: + - worker rollout + - 验证 PID 切换 + - 验证代码行号切换到当前版本 + - 再看吞吐和 CPU 利用率 + +## 本轮改动文件 + +- `package_domain_release.sh` +- `verify_domain_release.sh` +- `package_domain_release.ps1` +- `verify_domain_release.ps1` +- `domain-api/app/services/ops_release_executor_core.py` +- `domain-api/app/node_agent.py` +- `docs/ops_center_runtime/TASK_BOARD.md` +- `docs/ops_center_runtime/IMPLEMENTATION_STATUS.md` diff --git a/docs/ops_center_runtime/HANDOFF_20260419_2117.md b/docs/ops_center_runtime/HANDOFF_20260419_2117.md new file mode 100644 index 0000000..952a1b0 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_2117.md @@ -0,0 +1,102 @@ +# HANDOFF_20260419_2117 + +更新时间:2026-04-19 21:17 CST + +## 本轮结论 + +`mainland-worker-01` 的最新正式 rollout 已完成根因收口: + +- `release_id = 7` +- `rollout_id = 6` +- `job_id = 211` + +执行结果: + +- 发布包下载成功 + - `/opt/domaincheck/downloads/domaincheck_release_20260419_205437.tar.gz` +- release 解压成功 + - `/opt/domaincheck/releases/domaincheck_release_20260419_205437` +- `current` 软链切换成功 + - `/opt/domaincheck/current -> /opt/domaincheck/releases/domaincheck_release_20260419_205437` +- 最终失败在: + - `systemctl restart domaincheck-worker` + +控制面失败结果原文: + +- `restart failed: domaincheck-worker` +- `Failed to restart domaincheck-worker.service: Interactive authentication required.` + +## 真实根因 + +不是发布包问题,也不是目录权限问题了。 + +当前根因是: + +- `domaincheck-node-agent` 运行身份仍是 `www` +- 它具备下载、解压、切换 `current` 的权限 +- 但不具备执行 systemd 服务重启的权限 + +所以当前所有以下动作在 mainland 机器上都会有同类风险: + +- `deploy.release` +- `service.restart` +- 任何依赖 node-agent 直接操作 systemd 的动作 + +## 本轮已做代码修正 + +已修改: + +- `domain-api/deploy/systemd/domain-node-agent.service` + - `User=root` + - `Group=root` +- `domain-api/deploy/multi-region/fix_mainland_release_base.sh` + - 安装 node-agent drop-in 时补齐: + - `User=root` + - `Group=root` + +已验证: + +- `bash -n domain-api/deploy/multi-region/fix_mainland_release_base.sh` + +## 现场下一步 + +下一步不要再重复发起新的 rollout,先把 `mainland-worker-01` 的 node-agent 切到 root。 + +建议现场执行: + +```bash +mkdir -p /etc/systemd/system/domaincheck-node-agent.service.d +cat >/etc/systemd/system/domaincheck-node-agent.service.d/runtime-user.conf <<'EOF' +[Service] +User=root +Group=root +EOF +systemctl daemon-reload +systemctl restart domaincheck-node-agent +systemctl status domaincheck-node-agent --no-pager -l +``` + +完成后应立即复查: + +```bash +systemctl show domaincheck-node-agent -p User -p Group +journalctl -u domaincheck-node-agent -n 80 --no-pager -l +``` + +预期结果: + +- `domaincheck-node-agent` 以 `root` 身份运行 +- 心跳恢复,`mainland-worker-01` 不再是 `stale` +- 后续再发起 `deploy.release` 时,能够闭环到 `systemctl restart domaincheck-worker` + +## 当前判断 + +当前主阻塞只剩一个: + +- mainland node-agent 权限模型未切到 root + +这个问题修完后,再重新发起 worker rollout,才有资格继续看: + +- `domaincheck-worker` 是否切到新 PID +- health check 是否通过 +- rollout 6 之后的 acceptance 是否可继续 diff --git a/docs/ops_center_runtime/HANDOFF_20260419_2125.md b/docs/ops_center_runtime/HANDOFF_20260419_2125.md new file mode 100644 index 0000000..fe78616 --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260419_2125.md @@ -0,0 +1,95 @@ +# HANDOFF_20260419_2125 + +更新时间:2026-04-19 21:25 CST + +## 本轮结果 + +`mainland-worker-01` 的 release rollout 已正式成功收口。 + +关键信息: + +- `release_id = 7` +- `rollout_id = 7` +- `job_id = 214` + +最终状态: + +- `job 214 = success` +- `rollout 7 = completed` +- `mainland-worker-01.agent_state = online_busy` + +## 成功证据 + +控制面 `job events` 已记录: + +- `event_type = agent_completed` +- `summary_text = release deployed` + +结果载荷中已确认: + +- `release_version = domaincheck_release_20260419_205437` +- `restart_results` + - `domaincheck-worker.returncode = 0` +- `health_check.ok = true` +- `services_checked = ["domaincheck-worker"]` +- `execstart_alignment.mismatched_services = []` + +说明这次已经完整走通: + +1. 下载发布包 +2. 解压 release +3. 切换 `/opt/domaincheck/current` +4. 重启 `domaincheck-worker` +5. 健康检查通过 + +## 为什么这次成功 + +不是发布逻辑变化了,而是现网权限问题被修掉了。 + +上一轮失败根因: + +- `domaincheck-node-agent` 以 `www` 身份运行 +- 无法执行 `systemctl restart domaincheck-worker` +- systemd 返回: + - `Interactive authentication required` + +本轮修复: + +- `mainland-worker-01` + - `domaincheck-node-agent` 已切为: + - `User=root` + - `Group=root` + +## 当前剩余项 + +唯一建议立即同步的预防动作: + +- 在 `mainland-controller-01` 上,也把 `domaincheck-node-agent` 切成 `root` + +原因: + +- worker 已经验证这就是 release 闭环最后一层权限门槛 +- controller 如果后续参与自身发布 / restart,同样会遇到同类 systemd 权限问题 + +建议执行: + +```bash +mkdir -p /etc/systemd/system/domaincheck-node-agent.service.d +cat >/etc/systemd/system/domaincheck-node-agent.service.d/runtime-user.conf <<'EOF' +[Service] +User=root +Group=root +EOF +systemctl daemon-reload +systemctl restart domaincheck-node-agent +systemctl show domaincheck-node-agent -p User -p Group +``` + +## 当前判断 + +关于 mainland worker 的发布链,已经不再是阻塞项。 + +现在可以进入下一步: + +- 再决定是否对 `mainland-controller-01` 做同样 root 切换 +- 然后继续 controller acceptance / rollout 收口 diff --git a/docs/ops_center_runtime/HANDOFF_20260420_1920.md b/docs/ops_center_runtime/HANDOFF_20260420_1920.md new file mode 100644 index 0000000..9fdc2cb --- /dev/null +++ b/docs/ops_center_runtime/HANDOFF_20260420_1920.md @@ -0,0 +1,563 @@ +# HANDOFF_20260420_1920 + +更新时间:2026-04-20 19:20 CST + +## 本轮结论 + +这轮最重要的收口已经完成: + +- 已确认当前工作机 `/www/wwwroot/getDomain` 所在主机是海外测试控制面,不是国内 `mainland-controller-01` +- 已把“假冒 mainland-controller-01”的本机配置纠正为 `overseas-control-01` +- 已把真正的 `mainland-controller-01` 重新接回控制面,并完成一次成功的远端发布 +- 已修正 `mainland-controller-01` 的 node-agent 身份上报问题 +- 现在控制面里看到的 `mainland-controller-01` 已经是真实主机: + - `hostname = mainland-controller-01` + - `ip = 121.204.244.188` + +一句话说当前状态: + +“主线已经从‘节点身份混乱’推进到‘真 controller 已经对正并重新进场’,下一步应只盯 controller 真机吞吐和 worker 恢复,不要再回展示层。” + +## 当前环境认知 + +### 1. 当前 Codex 所在机器不是大陆 controller + +本地执行结果: + +- `hostname = v2202604268673447256` +- `nproc = 12` + +这台机器是海外控制面测试机,不是用户说的 112 核 controller。 + +真正的国内 controller 是: + +- `node_code = mainland-controller-01` +- `ssh_host = 121.204.244.188` + +真正的国内 worker 是: + +- `node_code = mainland-worker-01` +- `ssh_host = 121.204.244.248` + +### 2. 本机服务身份已经纠正 + +为避免海外测试机继续伪装成 mainland controller,已经改过这些环境文件: + +- `/etc/default/domaincheck-worker` +- `/etc/default/domaincheck-node-agent` +- `/etc/default/domaincheck-api` + +修正后的核心值: + +- `NODE_CODE=overseas-control-01` +- `NODE_REGION=overseas` +- `SYNC_PUSH_ENABLED=false` + +并且已经执行过: + +- 停止本机 `domaincheck-worker` +- 停止本机 `domaincheck-node-agent` +- 重启本机 `domaincheck-api` + +当前海外控制面 API 进程环境已确认是: + +- `NODE_CODE=overseas-control-01` +- `NODE_REGION=overseas` + +## 本轮已完成的关键修复 + +## A. 修复 controller node-agent 上报 localhost / 127.0.0.1 + +修改文件: + +- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py) + +修复内容: + +- `_hostname()` 不再优先相信 `localhost` +- `_ip()` 不再使用容易得到 `127.0.0.1` 的旧逻辑 +- 优先通过控制面地址推导本机出口 IP +- 回退时也会过滤 loopback + +本轮新增/补过的测试: + +- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py) + +注意: + +- 当前仓库里这份修复已经在真 controller 上生效 +- handover 已显示: + - `agent_hostname = mainland-controller-01` + - `agent_ip = 121.204.244.188` + - `cluster_hostname = mainland-controller-01` + - `cluster_ip = 121.204.244.188` + +## B. 修复 control 节点发布健康检查窗口过短 + +修改文件: + +- [domain-api/app/services/ops_release_service.py](/www/wwwroot/getDomain/domain-api/app/services/ops_release_service.py) + +修改目标: + +- control 节点发布时,`domaincheck-api` 启动偏慢,旧健康检查窗口太短,会误判失败并回滚 + +代码里已改成: + +- `health_check_timeout_seconds = 20` +- `health_check_retries = 6` +- `health_check_interval_seconds = 3` + +但要注意一个坑: + +- 这份代码虽然已改进源码 +- 海外控制面的运行中 API 进程还没有通过这份新源码重新部署 +- 所以 `smart-rollout-preview` 里看到的默认值一度仍然是旧的 `10 / 2 / 2` + +因此本轮实际是通过“手工 remote-agent deploy job 显式带长窗口 payload”打通了 controller 发布。 + +## C. 真 controller 发布已经成功一次 + +成功任务: + +- `job_id = 331` +- `job_code = ops-20260420183116-c33723` +- `action = deploy.release` +- `target_node_code = mainland-controller-01` +- `execution_mode = remote-agent` +- `status = success` + +本次使用的 release: + +- `release_id = 27` +- `release_version = domaincheck_release_20260420_182916` + +发布结果要点: + +- 发布包下载成功 +- checksum 校验成功 +- `/opt/domaincheck/current` 切换成功 +- `domaincheck-api / domaincheck-worker / domaincheck-sync-agent` 重启成功 +- 健康检查最终通过 + +尤其要记住: + +- 这次成功不是靠 smart rollout 默认值 +- 是通过显式下发以下健康窗口打通的: + - `health_check_timeout_seconds = 20` + - `health_check_retries = 6` + - `health_check_interval_seconds = 3` + +## D. controller node-agent 已经重新连回 + +后续又触发了: + +- `job_id = 332` +- `action = service.restart` +- `payload.service_name = domaincheck-node-agent` + +这个任务本身还停留在 `running`,原因很正常: + +- node-agent 在“执行重启自己”的过程中会打断自身回执链 +- 所以作业状态可能不会自然收尾 + +但实际效果已经发生: + +- API 日志已看到 `121.204.244.188` 在 `2026-04-20 18:35:16` 重新开始: + - `POST /api/v1/ops/agent/heartbeat` + - `POST /api/v1/ops/agent/pull?limit=1` + +因此这条 job 332 可以视为“结果已生效,但状态未优雅回写”的典型自重启任务。 + +## 当前实机状态 + +按最新控制面查询: + +### mainland-controller-01 + +- `agent_online = true` +- `cluster_status = busy` +- `agent_hostname = mainland-controller-01` +- `agent_ip = 121.204.244.188` +- `current_load` 在本轮观察中约 `210 - 233` +- `detect_runtime.active_threads` 在本轮观察中约 `210 - 233 / 2000` +- 已经有持续日志回传 + +### mainland-worker-01 + +- node-agent 在线 +- 但当前没有真正参与检测 +- 控制面返回的 `detect_runtime` 错误为: + - `connection to server at "127.0.0.1", port 5432 failed: Connection refused` + +这意味着: + +- worker 机现在不是主要算力来源 +- 目前真正吃任务的是 `mainland-controller-01` + +## 当前主线瓶颈 + +目前主线已经不是“谁是 controller”了,当前瓶颈明确是下面两个: + +1. `mainland-worker-01` 未恢复到可参与检测状态 +2. `mainland-controller-01` 虽然真实线程已抬到 200+,但控制面队列视角仍存在: + - `claimed` 偏高 + - `running/completed` 推进不够理想 + - 吞吐没有完全吃透机器 + +也就是说: + +- “节点身份问题”已基本打穿 +- “发布链路问题”已基本打穿 +- 下一步该只盯“真 controller 吞吐”和“worker 恢复” + +## 本轮新增发现 + +### 1. mainland-worker-01 的 `127.0.0.1:5432` 更像是 node-agent 侧遥测链路问题 + +本轮继续排查后,发现这个问题至少有两层: + +1. `domainCheck` 默认配置本身就是: + - `DB_HOST = localhost` +2. worker 机上的 `domaincheck-node-agent` systemd unit 当前只加载: + - `/etc/default/domaincheck-api` + - `/etc/default/domaincheck-node-agent` + +但大陆 worker 快速安装脚本真正写入数据库与 Redis 指向的是: + +- `/etc/default/domaincheck-worker` + +也就是说,worker 机上很可能出现这种情况: + +- `domaincheck-worker` 进程拿到的是正确的 `DB_HOST=${MAINLAND_CONTROLLER_IP}` +- 但 `domaincheck-node-agent` 没有继承 `/etc/default/domaincheck-worker` +- node-agent 内部去跑 `get_detect_status()` 时,就会落回 `domain-api` 默认配置: + - `db_host = 127.0.0.1` + +于是控制面上看到的现象就变成: + +- `mainland-worker-01 agent 在线` +- 但 `detect_runtime` 里报: + - `connection to server at "127.0.0.1", port 5432 failed: Connection refused` + +### 2. node-agent 当前把“DB 查询失败”和“worker 进程离线”混成了一种失败 + +`domain-api/app/node_agent.py` 里的 `_detect_runtime_snapshot()` 原来是: + +- `get_detect_status()` 和 `detect_worker_runtime()` 放在同一个总 `try` 里 + +这会导致: + +- 只要前面的 DB 查询失败 +- 后面的 worker systemd 运行态也一起被吞掉 +- 最终上报成: + - `worker_online = false` + - `service_running = false` + +即使真实情况其实可能只是: + +- worker 服务还活着 +- 只是 node-agent 在采集 detect status 时查错 DB 了 + +### 3. 本轮已在源码里补的最小修复 + +已修改: + +- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py) +- [domain-api/deploy/systemd/domain-node-agent.service](/www/wwwroot/getDomain/domain-api/deploy/systemd/domain-node-agent.service) +- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py) + +修复内容: + +1. `_detect_runtime_snapshot()` 改成分层降级: + - 先拿 `detect_worker_runtime()` + - 再单独尝试 `get_detect_status()` + - 即使 detect status 因 DB 异常失败,也保留真实的 worker service 运行态 +2. `domaincheck-node-agent.service` 追加: + - `EnvironmentFile=-/etc/default/domaincheck-worker` + +这样 worker 机上的 node-agent 也能直接继承 worker 真实使用的 `DB_HOST / REDIS_HOST`。 + +### 4. 这次修复的生效边界 + +要特别注意: + +- `node_agent.py` 代码修复可以通过常规 release 进入真实运行目录并生效 +- 但 `domaincheck-node-agent.service` 属于 `/etc/systemd/system/` 下的系统级 unit 文件 +- 常规 `deploy.release` 只会切换 `/opt/domaincheck/current`,不会自动重装 systemd unit + +所以这两项应分开看: + +1. `node_agent.py`: + - 可以走最小 release 先上真机 +2. `domain-node-agent.service`: + - 需要后续补 systemd unit 落地动作 + - 至少要包含: + - 覆盖 unit 文件或 drop-in + - `systemctl daemon-reload` + - `systemctl restart domaincheck-node-agent` + +### 5. 本轮测试结果 + +已通过的焦点测试: + +- `PYTHONPATH=/www/wwwroot/getDomain/domain-api /opt/domaincheck/domainCheck/.venv/bin/python -m unittest tests.test_node_agent_delivery_queue -v` + +结果: + +- `11 tests` +- `OK` + +新增验证点: + +- `test_detect_runtime_snapshot_degrades_to_worker_runtime_when_detect_status_fails` + +它验证了: + +- 即使 `get_detect_status()` 抛出 + - `connection to server at "127.0.0.1", port 5432 failed` +- node-agent 依然会保留: + - `worker_online = true` + - `service_running = true` + - `phase_detail = active/running` + +## 本轮最小发布动作 + +### 1. 已生成新发布包 + +本轮最小修复已重新打包: + +- `release_version = domaincheck_release_20260420_215734` +- `release_id = 28` + +生成结果: + +- `archive = /www/wwwroot/getDomain/release/domaincheck_release_20260420_215734.tar.gz` +- `sha256 = ad2c41625503dab1efaeadbf0641c8f6875a3cad0581a3b9c7835ccc37cdc718` + +### 2. 已向真节点发起最小 release 下发 + +本轮只为把 `node_agent.py` 先推进真实运行目录,已创建: + +- `job_id = 335` + - `target = mainland-controller-01` + - `action = deploy.release` +- `job_id = 336` + - `target = mainland-worker-01` + - `action = deploy.release` + +这两条 job 都已经越过“等待拉取”,进入了 agent 执行阶段,并至少记录到: + +- `executor_received` +- `deploy_download_started` + +说明: + +- 真 controller 和真 worker 都已经接到了这版最小 release +- 当前至少已经在真实节点上进入下载/发布链 + +### 3. 这次发布的真实目标 + +这次发布不是为了一次性解决所有 worker 恢复问题,而是为了先把下面这条代码修复推到真节点: + +- `domain-api/app/node_agent.py` + +目的: + +- 即使 worker 节点的 `detect_status` 查询因 DB 指向错误失败 +- node-agent 也不再把真实的 worker service 运行态一并吞掉 +- 控制面能先看到更接近真实的 `worker_online / service_running` + +### 4. 这次发布的限制 + +这次最小 release 暂时还不能自动完成下面这件事: + +- 把 `/etc/systemd/system/domaincheck-node-agent.service` 更新为新版本 + +原因: + +- 常规 `deploy.release` 切的是 `/opt/domaincheck/current` +- 不会自动重装 systemd unit + +因此当前判断是: + +1. `node_agent.py` 代码修复: + - 已经进入真实节点发布链 +2. `domaincheck-node-agent.service` 的 `EnvironmentFile=-/etc/default/domaincheck-worker`: + - 仍需要后续单独补系统级落地动作 + +### 5. 当前发布观察结论 + +截至本次交接整理时: + +- `job 335 / 336` 已启动执行 +- 但尚未在本轮记录中拿到最终 `success / failed` 收尾结论 +- 因此后续接手时,先做的第一件事之一,就是复查这两条 job 的最终状态与事件流 + +## 当前不要再踩的坑 + +### 1. 不要再把本机当成 mainland-controller-01 + +当前工作机是海外测试控制面,只负责: + +- API +- 控制面 +- 同步接收 + +不是 112 核大陆 controller。 + +### 2. 不要直接用本机 Python service 函数创建 deploy job + +坑点: + +- 直接在 shell 里跑本地 Python 服务函数时,读取到的 `settings.node_code` 可能不是运行中 API 进程的真实环境 +- 之前就出现过把 job 错判为 `local-runtime` 的情况 + +正确做法: + +- 优先通过“正在运行的 API HTTP 接口”创建 job +- 不要优先走本机 Python service 入口 + +推荐路径: + +- `POST /api/v1/ops/releases/from-package/latest` +- `POST /api/v1/ops/jobs` +- `POST /api/v1/ops/jobs/{job_id}/dispatch` + +### 3. 不要把 job 332 当成硬故障 + +`job 332` 是 `service.restart domaincheck-node-agent`。 + +它卡在 `running` 不代表没生效,反而更像: + +- node-agent 重启了自己 +- 回执链没能把 job 收尾 + +判断是否真的生效,应看: + +- API 日志里有没有来自 `121.204.244.188` 的 heartbeat/pull +- handover 里 `agent_hostname / agent_ip` 是否已刷新 + +### 4. 不要再回展示层 + +当前最值钱的推进方向仍然是: + +- controller 真机吞吐 +- claim/running/completed 推进 +- worker 恢复 + +不要把额度再耗在页面、文案、展示层结构上。 + +## 这轮动过的关键文件 + +最关键的源码文件: + +- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py) +- [domain-api/app/services/ops_release_service.py](/www/wwwroot/getDomain/domain-api/app/services/ops_release_service.py) +- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py) + +此外,工作区还存在大量其他未提交改动,不要随意回滚: + +- `domain-api/` +- `domain-web/` +- `domainCheck/` +- `docs/ops_center_runtime/` + +这是一个脏工作区,接手时必须小心,不要用破坏性 git 命令。 + +## 建议下一个 Codex 只做的两件事 + +### 任务 1:恢复 mainland-worker-01 + +目标: + +- 让 `mainland-worker-01` 从“agent 在线但不参与检测”恢复到可执行检测 + +先查方向: + +- 为什么它在本地检测态里访问 `127.0.0.1:5432` 失败 +- 是本机 PostgreSQL 没启 +- 还是 worker 的运行配置仍然错误指向本地 DB +- 还是它本来就不该走本地 PostgreSQL,而应走远端/统一控制链 + +验收标准: + +- `mainland-worker-01.detect_runtime.worker_online = true` +- `mainland-worker-01.detect_runtime.active_threads > 0` +- 能稳定进入参与节点 + +### 任务 2:继续抬 mainland-controller-01 真吞吐 + +目标: + +- 只盯真 controller 的任务推进链,不碰页面 + +重点看: + +- `claimed -> running -> completed` 是否持续推进 +- `items_claimed` 是否能下降 +- `processed_recent / processed_per_minute` 是否能抬起来 +- `active_threads` 与 `completed` 是否成正相关 + +验收标准: + +- controller 持续有真实 completed 增长 +- running/claimed 更贴近“持续补位”而不是堆积 +- 机器性能使用能继续往上抬 + +## 可直接复用的验证点 + +### 1. 看节点 handover + +接口: + +- `GET /api/v1/ops/nodes/mainland-controller-01/handover` + +这一项已经能确认: + +- 当前是不是对到了真 controller +- agent/ip/hostname 是否正确 +- 当前 load / detect_runtime 是否在动 + +### 2. 看节点列表 + +接口: + +- `GET /api/v1/ops/nodes` + +重点字段: + +- `agent_hostname` +- `agent_ip` +- `cluster_hostname` +- `cluster_ip` +- `detect_runtime.active_threads` +- `current_load` + +### 3. 看海外控制面 API 日志 + +已验证有效: + +- `journalctl -u domaincheck-api -n 200 --no-pager` + +尤其可以筛: + +- `121.204.244.188` +- `/api/v1/ops/agent/heartbeat` +- `/api/v1/ops/agent/pull` + +### 4. 发布 controller 的正确方式 + +推荐继续使用运行中的 API HTTP 接口,而不是本地 Python service 调用。 + +## 交接结论 + +这一轮真正解决掉的,不是“性能问题”本身,而是它前面最大的认知阻塞: + +- 之前一直有一部分判断建立在“本机就是 controller”这个错误前提上 +- 现在这个前提已经纠正 +- 真 controller 已经重新纳入控制面,而且身份上报正确、远端发布成功、线程也已经真实抬起来 + +接手人从这里继续时,应该把主线收缩成一句话: + +“只盯 `mainland-controller-01` 的真实吞吐推进,并恢复 `mainland-worker-01`,不要再回展示层,也不要再把海外 12 核测试机当成主算力机。” diff --git a/docs/ops_center_runtime/IMPLEMENTATION_STATUS.md b/docs/ops_center_runtime/IMPLEMENTATION_STATUS.md index 84a1774..b8fd978 100644 --- a/docs/ops_center_runtime/IMPLEMENTATION_STATUS.md +++ b/docs/ops_center_runtime/IMPLEMENTATION_STATUS.md @@ -1,23 +1,189 @@ # IMPLEMENTATION_STATUS -更新时间:2026-04-19 03:41 CST +更新时间:2026-04-19 21:25 CST ## 当前真实状态 阶段判断: -- 海外单脑接管能力:约 `95%` -- 分布式检测真实执行能力:约 `88%~90%` -- 距离“可稳定上线并放心用后台发起检测”:约 `86%~89%` +- 海外单脑接管能力:约 `97%` +- 发布闭环真实可用度:约 `65%~70%` +- 分布式检测真实执行能力:约 `80%~85%` +- 距离“可稳定上线并放心用后台发起检测”:约 `78%~82%` + +## 21:17 最新校正 + +刚完成的 `mainland-worker-01` 正式 rollout 已给出最终根因: + +- `job 211 / rollout 6 / release 7` + - 发布包下载成功 + - release 解压成功 + - `current` 软链切换成功 + - 最终失败在服务重启: + - `restart failed: domaincheck-worker` + - `Failed to restart domaincheck-worker.service: Interactive authentication required.` + +这说明: + +- 当前 ReleaseHub 主链路本身已可推进到“切换 current” +- 真正未闭环的是 mainland node-agent 的 systemd 权限 +- 只要 node-agent 仍以 `www` 运行,后续 `deploy.release / service.restart` 都会在 systemd 这一跳失败 + +已完成的代码侧修正: + +- `domain-api/deploy/systemd/domain-node-agent.service` + - 改为 `User=root` + - 改为 `Group=root` +- `domain-api/deploy/multi-region/fix_mainland_release_base.sh` + - 改为给 node-agent drop-in 写入 `User=root` / `Group=root` + +因此当前真实上线完成度需要再补一句校正: + +- 发布闭环真实可用度: + - 不是“发布包模型还没修” + - 而是“node-agent 权限模型还差最后一次现网切换” + +## 21:25 最新进展 + +worker 侧的现网切换已经完成验证成功: + +- `mainland-worker-01` + - 已把 `domaincheck-node-agent` 切为 `root` + - 随后重新发起: + - `rollout_id = 7` + - `job_id = 214` + - 最终结果: + - `job 214 = success` + - `rollout 7 = completed` + +正式成功证据: + +- `agent_completed` + - `summary_text = release deployed` +- `restart_results[domaincheck-worker].returncode = 0` +- `health_check.ok = true` +- `systemd ExecStart` 已对齐: + - `/opt/domaincheck/current/domainCheck/detect_worker.py` + +因此当前关于发布闭环的真实判断应更新为: + +- worker 发布闭环: + - 已从“卡在 systemd restart 权限”推进到“真实成功” +- 当前剩余风险不再在 worker +- 当前剩余预防项在 controller: + - `domaincheck-node-agent` 也应同步切为 `root` + +## 顶部校正 + +今天晚上的最新结论,需要覆盖前面一部分偏乐观判断: + +- 最新发布包问题已经定位并修复: + - 之前发布包漏掉了 `domainCheck/` + - 现在最新签收包 `domaincheck_release_20260419_205437` 已经包含 `domainCheck/` +- 但对 `mainland-worker-01` 的正式发布验证证明: + - 线上 mainland 节点目前还不满足现有 ReleaseHub 发布模型 +- 已确认的真实阻塞有两个: + - `deploy.release` 在 `mainland-worker-01` 上会因为 + - `Permission denied: /opt/domaincheck/downloads` + - 而直接失败 + - mainland 两台节点当前 `domaincheck-worker` 的 `ExecStart` 都直接指向: + - `/opt/domaincheck/domainCheck/detect_worker.py` + - 而不是 `/opt/domaincheck/current/domainCheck/detect_worker.py` + +这两个事实组合起来说明: + +- 当前发布链虽然在控制面上已经成型 +- 但 mainland 线上节点的安装形态还是旧模式 +- 所以当前不能再把“已能稳定 rollout 新版本”算进上线完成度 + +## 今晚新增硬证据 + +- `mainland-worker-01` + - `job 204 / rollout 5` 已正式失败回写 + - 失败原因已收口为: + - `install_root not writable: /opt/domaincheck/downloads` + - 只读核验得到的实际服务状态仍是旧进程: + - `Active since Sun 2026-04-19 18:18:50 CST` + - `Main PID = 635924` + - `CPU = 18.994s` + - 说明 worker 当前并没有切到新包,也没有形成可信的新吞吐样本 +- `mainland-controller-01` + - 当前 `domaincheck-worker` 虽然在高负载运行 + - 但服务启动路径同样是: + - `/opt/domaincheck/domainCheck/detect_worker.py` + - 说明 controller 也没有对齐 `current` 软链发布模型 + +## 本轮代码侧新增收口 + +- 已补齐发布包构建: + - `package_domain_release.sh` + - `verify_domain_release.sh` + - `package_domain_release.ps1` + - `verify_domain_release.ps1` + - 现在发布包会包含 `domainCheck/` +- 已补齐发布执行器的失败可观测性: + - `domain-api/app/services/ops_release_executor_core.py` + - 现在会在发布目录不可写时显式返回失败结果 + - 同时补充 `ExecStart` 与 `current` 软链的对齐观测 +- 已补齐 node-agent 的兜底失败回写: + - `domain-api/app/node_agent.py` + - 避免以后再出现任务已经炸掉但控制面一直卡在 `running` ## 本轮最新结论 -这轮结论需要更新为四段: +新增结论: -- 接管与同步能力已经明显趋于完成 -- worker 控制消息补偿链已经完成线上验证 -- controller 运行环境漂移已经被现场修正 -- 当前唯一剩余主阻塞已经收紧到 controller 代理池无可用代理 +- 检测页观察面已完成一轮线上收口: + - `任务日志控制台` 已移到事件表上方 + - `检测事件流` 已改成独立滚动区 + - 已在真实线上静态目录重建生产包并通过本机 `3201` 端口验证 + - 当前如果用户仍看到旧布局,优先判断为浏览器缓存而不是部署未生效 +- 代理链已完成策略切换: + - worker 启动后会主动刷新代理池 + - 配置更新后也会主动刷新代理池 + - 后台已能区分: + - `等待首刷` + - `proxy_validation_zero` + - `直入池` + - 当前本机最新真实结果是: + - 原始代理 `270` + - 预验证 `0` + - 直入池 `270` + - 说明当前问题已经从“代理校验过严导致池为空”切换为“真实任务内动态淘汰失效代理” + +- `mainland-controller-01` + - 已确认解除“代理不足即同步刷新、反向压死并发”的旧限制 + - 海外后台当前已能看到: + - `active_threads = 99~100` + - `max_threads = 100` + - 说明 controller 的 `100` 并发已经真实生效 +- `mainland-worker-01` + - 已重新部署最新版 `detect_worker.py` + - 已重新接收新的 `sync-pull` 批次并重进检测: + - `开始执行域名检测任务` + - `开始检测,刷新代理池` + - `代理池刷新完成,共 23 个可用代理` + - `开始创建线程,当前批次域名数: 400,最大线程数: 50` + - 当前 worker 进程本机线程量已到: + - `152` 个 OS 线程 + - 说明它并不是没启动,而是已经进入新批次执行 +- 当前剩余偏差: + - overseas 后台对 `mainland-worker-01.active_threads/current_load` 的显示仍偏低 + - 这已经从“真实执行问题”收敛为“运行态上报 / 显示口径问题” + +这一轮不是只修了显示问题,而是把“大陆节点真实执行 + worker 日志回传”一起补齐了。 + +当前最新结论: + +- controller 的 `pull_tasks -> 本地队列` 断点已经修通 +- 两台大陆 worker 都已经进入真实镜像队列执行 +- `mainland-worker-01` 的 `worker_log` 已开始直接回灌 overseas `detect_debug_events` +- `runtime_projection` 权限问题已经修复,后台运行态同步恢复 +- `/api/v1/ops/nodes` 已能证明 controller 的高并发真实生效 +- 当前剩余问题已经收敛为: + - `mainland-worker-01` 运行态显示口径仍需继续对齐 + - 检测页主计数口径是否完全跟上 + - 结果统计回推是否持续稳定 ## 当前证据拆分 @@ -25,210 +191,273 @@ 已经完成: -- `detect_result_projection` 支持 `recent_domain_events` -- 中央 ingest 会把逐条事件写入 `detect_run_events` -- `sync_agent` 会自动产出 `detect_result_projection` -- worker 控制消息新增 `request_id` -- worker 运行态心跳会补偿消费 pending 控制消息 -- worker 收到并处理控制消息后会按 `request_id` 清理 pending 指令 +- `sync_push_service.ingest_detect_task_projection` + - 不再只导入 `domains` + - 已改为同时创建本地 `detect_jobs` + - 已改为同时创建本地 `detect_job_items` + - 已把 `target_job_id / target_job_code / queued_count / worker_start_ok` 回写到同步结果 +- `sync_agent` + - controller 现在会在每轮 `pull_tasks` 后自动唤起本地 worker +- `detect_worker._set_active_cycle_context` + - 已兼容 `sync-pull` 控制消息 + - 不再只读取 `job_id / job_code` + - 现在会回退读取 `target_job_id / target_job_code` +- 运行态同步链 + - controller `runtime/detect_runs.json` 权限已修正为 `www:www` + - `runtime_projection` 已恢复成功推送 本地代码验证已通过: -- `unittest domain-api/tests/test_worker_control_service.py` -- `python -m py_compile domainCheck/detect_worker.py domain-api/app/services/worker_control_service.py` +- `python -m py_compile domain-api/app/services/sync_push_service.py` +- `python -m py_compile domain-api/app/services/runtime_control_service.py` +- `python -m py_compile domain-api/app/sync_agent.py` +- `python -m py_compile domainCheck/detect_worker.py` -线上运行验证也已经出现正向证据: +### 2. 接管 / 外部条件闭环 -- `mainland-worker-01` - - 启动后发现待执行控制指令 - - 接受 `start_detection` - - 开始执行远程检测任务 -- `mainland-controller-01` - - 修正 Redis 环境后 - - 重新接受 `start_detection` - - 开始执行远程检测任务 - -### 2. 接管/同步闭环 - -当前已完成: +已经完成: - `remote_access_ready = 3/3` -- `log_sync_state = full_capture` -- `mainland-controller-01` 的 `domaincheck-sync-agent` 已重启到新进程 -- 首轮 `detect_result_projection` 推送成功过一次 -- 中央已收到 mainland 首批 `domain_*` 事件 +- `ssh_ready = 2` +- 大陆 controller / worker 都可远程运维 +- controller `domaincheck-sync-agent` 在线 +- mainland 两台 `domaincheck-node-agent` 在线 -这说明: +当前仍依赖你额外输入的部分: -- mainland 到中央的基础同步链是活的 -- 结果投影链至少成功打通过一次 +- 暂无新的 SSH / 接管前置输入缺口 +- 如果页面仍显示旧状态,最多只需要你浏览器强刷确认,不需要再补 SSH 信息 + +### 2.1 前端部署闭环 + +已经完成: + +- 在线静态根目录确认: + - `/www/wwwroot/getDomain/domain-web/dist` +- 在线 Nginx 配置确认: + - `/www/server/panel/vhost/nginx/domaincheck_3201.conf` + - `/www/server/panel/vhost/nginx/domaincheck_152.53.37.118.conf` +- 生产包重建确认: + - `vite build` 成功 +- 线上资源确认: + - `DetectView-B9S1WMRT.js` + - `DetectView-BFxMNRNR.css` + +说明: + +- 检测页布局调整不是只停留在源码 +- 已经进入线上可访问静态产物 + +### 2.2 代理运行态闭环 + +已经完成: + +- worker 启动即触发代理池首刷 +- 配置变更后自动补刷 +- API 状态页已能准确显示: + - `proxy_not_refreshed_yet` + - `proxy_validation_zero` + - `宽松入池` + +当前最新证据: + +- `proxy_last_refresh_status = 直入池 270 个(跳过预验证)` +- `proxy_last_refresh_total_items = 270` +- `proxy_last_validated_count = 0` +- `proxy_last_available_count = 270` + +说明: + +- 代理配置本身已成功下发 +- 代理源接口本身也能返回大量原始代理 +- 当前不再把“预验证是否通过”作为入池门槛 +- 现在改为让真实检测来完成失效代理淘汰 ### 3. 检测执行闭环 -当前未完成: +这一块现在已经从“怀疑恢复”进入“确认恢复”: -- 短观察窗口内: - - `progress_percent` 仍是 `2.1` - - `items_completed` 仍是 `21` - - `items_claimed` 仍是 `34` - - `items_pending` 仍是 `931` -- 中央 recent events 已刷新到更晚时间 -- `runtime/sync-summary` 最新记录已继续增长 -- 但 completed 尚未继续上涨 +- `mainland-controller-01` + - 已出现: + - `从任务队列获取到 800 个需要检测的域名` + - `最大线程数: 100` + - `当前实际线程数量: 1/100 ... 6/100` +- `mainland-worker-01` + - 已出现: + - `从任务队列获取到 400 个需要检测的域名` + - `开始创建线程,当前批次域名数: 400,最大线程数: 50` + - `当前实际线程数量: 1/50 ... 3/50` + - `开始检测域名: 0-demagogo.com` + - `开始检测域名: 00123321.com` + - `开始检测域名: 001dm.com` 这说明: -- 当前不是单纯“页面没刷新” -- 而是执行现场这段时间没有继续出结果 +- `100/50` 已不是“配置已写入但没生效” +- 它已经进入真实任务消费 +- worker 的远端日志也已经跟着真实执行一起回传 + +### 4. 海外后台可视证据 + +`/api/v1/ops/nodes` 当前已显示: + +- `summary.remote_access_ready = 3` +- `summary.participating = 3` +- `summary.dispatch_active = 1` + +并且大陆两台都已经被判定为真实参与者: + +- `mainland-controller-01` + - `agent_state = online_busy` + - `participation_state = recent_throughput` + - `processed_recent = 115` + - `processed_per_minute = 7.67` + - `detect_runtime.max_threads = 100` +- `mainland-worker-01` + - `agent_state = online_busy` + - `participation_state = recent_throughput` + - `processed_recent = 632` + - `processed_per_minute = 42.13` + - `detect_runtime.max_threads = 50` + +这说明: + +- 海外后台已经不只是看到“在线” +- 现在已经能看到大陆节点的实际吞吐 ## 本轮新增硬证据 -通过中央观测面、节点现场日志和远端 `domaincheck-worker` 日志,已确认: +### 证据 1:controller 的镜像队列已经真正落库 -- `mainland-controller-01` 现场日志显示: - - `当前可用代理数: 0` - - `最近结果: 刷新成功,可用 0 个` -- controller 新增远端日志显示: - - 代理源拉取成功 - - 抽样校验后 `共 0 个可用代理` - - 失败集中在: - - `ProxyError@https://m.baidu.com` - - `Unable to connect to proxy` - - `ConnectTimeoutError` -- worker 新增远端日志显示: - - `发现待执行 Worker 控制指令` - - `已接受检测启动指令` - - `开始执行远程检测任务` - - 说明线上补偿消费链已真正工作 -- controller 新增远端日志显示: - - 初次重启后: - - `Authentication required` - - `maximum recursion depth exceeded` - - 进一步排查确认: - - `/etc/default/domaincheck-worker` 的 `REDIS_PASSWORD` 为空 - - 修正后再次重启: - - `Redis 连接成功: 127.0.0.1:6379` - - `已接受检测启动指令` - - `开始执行远程检测任务` - - 后续日志继续收紧到: - - `代理已启用,但当前无可用代理` -- 两台大陆节点 full capture 已开启,但源日志时间没有继续前进 +`sync-agent` 现场日志已出现: -这说明: +- `target_job_code = sync-overseas-7380` +- `queued_count = 200` +- `worker_start_ok = True` -- worker 控制消息链不再是主阻塞 -- controller Redis 环境漂移也不再是主阻塞 -- 当前第一主阻塞已经进一步收紧到 controller 代理池不可用 -- worker 的时光机异常是客观存在的次级问题 -- 不是控制面未接管 -- 不是同步链未打通 +并持续产生: -## 本轮新增代码修复 +- `sync-overseas-7383` +- `sync-overseas-7386` +- `sync-overseas-7392` +- `sync-overseas-7417` -本轮不是只停留在诊断,还补了一处运行态最小修复: +说明: -### 修复点 1:控制消息唯一标识 +- 大陆 controller 现在不是只拉数据 +- 而是在持续形成可执行批次 -- 文件: - - `domain-api/app/services/worker_control_service.py` -- 变更: - - 每次 `send_worker_command(...)` 都附带 `request_id` - - Redis `publish` 与 pending fallback 使用同一份消息体 +### 证据 2:两台大陆 worker 都已转入真实队列 -### 修复点 2:worker 运行态补偿消费 pending 指令 +现场日志已明确出现: -- 文件: - - `domainCheck/detect_worker.py` -- 变更: - - 心跳线程每轮会补偿尝试消费 pending 控制消息 - - 解决“worker 在线但 pubsub 消息漏收,导致 pending 指令长期不被消费”的风险 +- controller: + - `从任务队列获取到 800 个需要检测的域名` +- worker: + - `从任务队列获取到 400 个需要检测的域名` -### 修复点 3:worker 收到指令后确认清理 pending +说明: -- 文件: - - `domainCheck/detect_worker.py` -- 变更: - - worker 实际收到控制消息后,会按 `request_id` 清理对应 pending 指令 - - 避免修复后又产生重复回放 +- 当前不是兼容旧链路假运行 +- 是镜像队列真运行 -### 当前判断 +### 证据 3:runtime_projection 已恢复成功 -这组修复解决的是: +修复前: -- “检测启动已经发布,但 worker 可能静默漏收”的代码风险 +- `refresh runtime_projection failed: [Errno 13] Permission denied` -所以当前最准确的状态是: +修复后: -- 代码级控制链缺口已补上 -- 线上部署验证已通过 -- 但 `controller` 代理池校验后 `0 available` 的现场阻塞仍然存在 +- 最新 `sync-agent` 日志已出现: + - `sync_state = success` + - `runtime_projection -> 投影推送成功` + +说明: + +- 后台运行态刷新和日志窗口不再被 controller 本地权限卡死 + +### 证据 4:`mainland-worker-01` 的 `worker_log` 已进入 overseas + +`/api/v1/runtime/debug-events` 当前已能看到: + +- `mainland-worker-01 -> 开始执行检测任务,来源: redis-control` +- `mainland-worker-01 -> 开始执行域名检测任务,正在加载配置` +- `mainland-worker-01 -> 开始检测,正在刷新代理池` +- `mainland-worker-01 -> 代理池刷新完成,共 7 个可用代理,来源链接 6 个,原始 265 个,验证 150 个` +- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名` +- `mainland-worker-01 -> 开始创建线程,当前批次域名数: 400,最大线程数: 50` +- `mainland-worker-01 -> 当前实际线程数量: 1/50` +- `mainland-worker-01 -> 当前实际线程数量: 2/50` +- `mainland-worker-01 -> 当前实际线程数量: 3/50` + +说明: + +- worker 的真实执行链已经进入 overseas 的日志视图 +- “后台看不到大陆 worker 在干活”这一层已经闭合 + +## 当前口径说明 + +这里有一个非常关键的判断口径已经变化: + +- 大陆节点当前执行的是“海外任务在大陆本地落镜像队列,再把运行态和结果同步回海外” +- 因此不能再只盯中央原生 `detect_job_items.claimed/running` +- 现在应该同时看: + - `/api/v1/ops/nodes` + - `processed_recent` + - `processed_per_minute` + - `detect_runtime.active_threads/max_threads` + - 大陆 controller 本地 `sync-overseas-*` 队列 + +如果只看中央原生队列,会误判“大陆没有参与”。 ## 当前已闭合的问题 已闭合: -- 大陆 controller 无法 `pull_tasks` -- Detect 页面误判“大陆没跑” -- Detect 主统计口径不一致 -- Runtime / Queue / Detect 主摘要不统一 -- 中央逐条事件接收缺口 -- `sync_agent` 自动产出结果投影缺口 +- controller `pull_tasks` 只导入 domain、不导入本地队列 +- 大陆 worker 长时间卡在旧检测会话导致忽略新启动指令 +- controller `runtime_projection` 因文件属主错误无法推送 +- 大陆两台“在线但不确定是否真实执行”的状态模糊 -## 当前唯一剩余问题 +## 当前剩余问题 -当前唯一主问题仍然是: +当前剩余问题已经缩成两个最小项: -- 检测执行面没有恢复到持续产出 - -但现在已经不需要再拆成“代码待验证”和“现场阻塞”两层。 - -当前唯一剩余现场阻塞就是: - -- controller 代理池可用性为 0 -- 因此没有持续产生新的 domain 级结果 +- Detect 页面主计数 `pending/completed/running` 是否继续推进 +- 结果统计回推是否稳定,而不只是日志链路先恢复 换句话说: -- 现在不是逻辑未实现 -- 不是中央映射失败 -- 不是节点未接管 -- 而是执行现场没有继续产出,且 controller 侧卡在代理校验失败 - -## acceptance 当前状态 - -最新接管验收结果仍然成立: - -- `pbr-9ce5c85f17`:`onboarding.acceptance` 成功 -- `pbr-389abd618c`:`onboarding.acceptance` 成功 - -旧的 `attention` run 依然是历史残留,但它们已经不是当前最真实的生产阻塞。 +- 现在不是接管问题 +- 不是链路不通问题 +- 不是日志完全回不来问题 +- 而是最后一层“主计数推进 + 结果统计收口”问题 ## 当前是否可以继续跑检测测试 当前结论: -- 可以继续做最小运行态排查 -- 但不需要再优先验证 worker 控制消息链 -- 当前还不能把状态视作“后台检测已经稳定恢复” +- 可以继续跑检测测试 +- 而且现在已经具备“后台可观测 + 大陆真实参与”的条件 ## 当前是否建议直接上线 当前结论: -- 不建议现在按“可稳定上线”判断 +- 已接近可上线状态 +- 但我仍建议把它视为“准上线收口态”,不是最终完全签收态 -原因不是接管面,而是执行面: +原因: -- 三台节点都已接入 -- worker / controller 都已重新接上控制链 -- 但当前任务没有持续吞吐 -- controller 代理池全部验不过会直接影响检测产出 +- 主执行链已经恢复 +- 但还需要再确认 1 到 2 个同步周期内页面口径与吞吐稳定性 ## 当前优先级判断 最高优先级: -- `J2-检测执行停滞收口批` +- `J2-检测执行收口批` 当前不应继续推进: @@ -236,18 +465,19 @@ - 新模块 - 新页面 - 发布动作 -- 与检测执行停滞无关的工作 +- 与检测执行收口无关的工作 ## 完成下一轮后的预期 如果下一轮确认: -- controller 代理池恢复可用 -- `domain_*` 开始继续增长 -- `items_completed` 和近窗吞吐重新前进 +- Detect 页面主计数已跟上 +- 日志窗口持续刷新 +- `detect_result_projection` 持续推进 +- `processed_recent / processed_per_minute` 持续增长 -则整体可上线程度预计可回升到: +则整体可上线程度预计可提升到: -- `92%~94%` +- `96%~98%` 在那之前,当前口径应保持保守。 diff --git a/docs/ops_center_runtime/TASK_BOARD.md b/docs/ops_center_runtime/TASK_BOARD.md index fd32b83..30301e9 100644 --- a/docs/ops_center_runtime/TASK_BOARD.md +++ b/docs/ops_center_runtime/TASK_BOARD.md @@ -1,177 +1,456 @@ # TASK_BOARD -更新时间:2026-04-19 03:41 CST +更新时间:2026-04-19 21:25 CST + +## 2026-04-20 主线切换说明 + +从这一刻开始,当前工作重心正式切回: + +- 先跑通主任务流程 +- 先推进 7 个大任务的测试与验收 +- 并发、代理、DB 往返、展示层等细节优化先封存,不再抢主线 + +细节优化不删除,统一视为 backlog: + +- worker pool 持续补位 +- claimed 回弹压缩 +- 高频 DB 更新继续合并 +- 代理失败链和等待窗口继续压缩 +- 运营视角页面继续细化 + +这些后续继续做,但不再打断主任务验收顺序。 + +当前主口径: + +1. 大任务 1 真实验收闭环 +2. 大任务 2 controller 编排闭环 +3. 大任务 3 统一结果状态机闭环 +4. 大任务 4 本地控制状态 + syncer/finalizer 闭环 +5. 大任务 5 固定 worker pool 落地 +6. 大任务 6 时光机一期接入 +7. 大任务 7 运营视角面板闭环 + +## 21:17 最新补充 + +刚刚这轮真实 rollout 已经把 mainland worker 的最后一层阻塞拿实: + +- `mainland-worker-01` + - `release_id = 7` + - `rollout_id = 6` + - `job_id = 211` +- 发布链路并不是卡在下载或解压: + - 发布包 `domaincheck_release_20260419_205437` 已下载到 `/opt/domaincheck/downloads` + - release 已解压到 `/opt/domaincheck/releases/domaincheck_release_20260419_205437` + - `/opt/domaincheck/current` 已切到新 release +- 最终失败点已经明确: + - `restart failed: domaincheck-worker` + - systemd 原因: + - `Interactive authentication required` +- 这说明当前 mainland 节点的 `domaincheck-node-agent` 虽然能写发布目录,但因为以 `www` 身份运行,无法执行: + - `systemctl restart domaincheck-worker` + +当前新的唯一主阻塞已经进一步收敛为: + +- node-agent systemd 运行身份不对 +- 需要把 `domaincheck-node-agent` 改成 `root` 运行,发布链最后一跳才能闭环 + +本轮已在仓库内同步修正: + +- `domain-api/deploy/systemd/domain-node-agent.service` + - `User=root` + - `Group=root` +- `domain-api/deploy/multi-region/fix_mainland_release_base.sh` + - node-agent drop-in 现在也会显式写入: + - `User=root` + - `Group=root` + +## 21:25 worker rollout 收口 + +`mainland-worker-01` 已完成新一轮正式 rollout 收口成功: + +- `release_id = 7` +- `rollout_id = 7` +- `job_id = 214` + +结果确认: + +- `job 214 = success` +- `rollout 7 = completed` +- `mainland-worker-01` 已恢复为: + - `agent_state = online_busy` + - 最近心跳恢复正常 + +控制面完成证据: + +- `agent_completed` + - `summary_text = release deployed` +- `restart_results` + - `domaincheck-worker.returncode = 0` +- `health_check.ok = true` +- `execstart_alignment.mismatched_services = []` + +这说明 worker 当前已经真正完成: + +- 新包下载 +- release 解压 +- `current` 切换 +- `domaincheck-worker` 重启 +- 健康检查通过 + +当前关于 mainland rollout 的唯一剩余预防项变成: + +- `mainland-controller-01` 也应该同步把 `domaincheck-node-agent` 切到 `root` +- 否则后续 controller 自身执行 `deploy.release / service.restart` 时还会踩到同类 systemd 权限问题 + +## 顶部校正 + +今天晚上的最新排查已经把当前主阻塞重新定性,之前“大陆两台已经完全进入统一新镜像执行”的判断需要收紧。 + +当前新增硬结论: + +- 最新正式发布包已经重打并签收通过: + - `domaincheck_release_20260419_205437` + - 发布包现在已经确实包含 `domainCheck/` +- 对 `mainland-worker-01` 发起正式 smart rollout 后,真实失败原因已经拿到: + - `job 204 / rollout 5` + - 本次失败对应的正式 Release 版本仍是: + - `domaincheck_release_20260419_203933` + - 失败原因: + - `Permission denied: /opt/domaincheck/downloads` + - 该失败已通过正式 agent complete 回写,后台不再是假 `running` +- `mainland-worker-01` 当前运行中的 `domaincheck-worker` 仍是旧进程: + - `Main PID = 635924` + - `Active since = 2026-04-19 18:18:50 CST` + - `CPU = 18.994s` + - 说明它不是高吞吐新进程,而是老进程长期挂着 +- `mainland-controller-01` 当前运行中的 `domaincheck-worker` 虽然活跃,但服务启动路径同样是: + - `/opt/domaincheck/domainCheck/detect_worker.py` +- 两台 mainland 节点当前 `domaincheck-worker` 的 systemd 启动路径都不是: + - `/opt/domaincheck/current/domainCheck/detect_worker.py` + +这意味着: + +- 当前 ReleaseHub 的 `current -> release_version` 切换模型,与线上 mainland 节点的实际服务启动路径不一致 +- 即使发布包和签收链已经修好,线上节点也还没有具备“按当前发布模型热切版本”的条件 +- 当前唯一主阻塞已经从“并发参数是否下发”切换成: + - mainland 节点发布权限/目录模型不匹配 + - mainland 节点服务启动路径与发布模型不匹配 ## 当前主批次 -唯一主批次:`J2-检测执行停滞收口批` +唯一主批次:`J2-检测执行收口批` 目标: - 不进入新页面 - 不扩展控制面 - 不新增发布动作 -- 只收口当前唯一真实阻塞: - - 节点已接管 - - 同步已打通 - - 但检测执行没有继续产出新结果 +- 只收口三件已经缩小到运行面的事情: + - 大陆节点必须真正进入统一镜像队列,而不是停留在兼容旧链路 + - 后台必须能看到大陆节点的实时运行态与日志 + - `100/50` 并发配置要从“已下发”推进到“真实有参与吞吐” ## 本轮最新状态 -本轮已经完成“代码修复 -> 两台大陆节点部署 -> 运行态复查”的完整一轮验证。 +### 20:53 最新阻塞结论 -### 已完成的最小修复 +- `mainland-worker-01` smart rollout 已真实失败,不再继续误判为执行中 +- 失败证据: + - node-agent 日志: + - `2026-04-19 20:40:15 [node-agent] loop error: [Errno 13] Permission denied: '/opt/domaincheck/downloads'` + - 发布任务: + - `job 204 = failed` + - 发布批次: + - `rollout 5 = failed` +- 当前不应该继续把精力放在前端展示或并发口径微调上 +- 当前必须先收口: + - mainland 节点 `deploy.release` 所需目录权限 + - mainland 节点 systemd `ExecStart` 与 `/opt/domaincheck/current` 的一致性 -- `worker_control_service.send_worker_command(...)` - - 为每条 worker 控制消息补上 `request_id` - - 保证 Redis 发布与 pending fallback 使用同一份负载 -- `detect_worker` - - 在运行态心跳里周期性补偿消费 pending 控制消息 - - 在真正收到控制消息后,按 `request_id` 清理 pending 指令 +这轮已经完成从“控制链打通”到“真实执行恢复 + worker 日志回传恢复”的关键跨越。 -本地验证已通过: +### 本轮前端已上线校验 -- `unittest domain-api/tests/test_worker_control_service.py` -- `python -m py_compile domainCheck/detect_worker.py domain-api/app/services/worker_control_service.py` +- 检测页 `任务日志控制台` 已调整到事件列表上方 +- `检测事件流` 已改成独立滚动区: + - 表格 `max-height = 320` + - 避免事件越积越多把整个页面继续向下撑长 +- 已在线上实际静态目录重新构建: + - `/www/wwwroot/getDomain/domain-web/dist` +- 已确认线上 Nginx 指向该目录: + - `domaincheck_3201.conf` + - `domaincheck_152.53.37.118.conf` +- 已通过 `http://127.0.0.1:3201/` 返回 `200` 且加载新资源: + - `DetectView-B9S1WMRT.js` + - `DetectView-BFxMNRNR.css` -### 已完成的线上验证 +### 本轮代理链已收口 -- `mainland-worker-01` - - 已拉到 `main` 最新提交 `c33f4f1` - - 已重启 `domaincheck-worker` - - 启动后明确出现: - - `发现待执行 Worker 控制指令` - - `已接受检测启动指令` - - `开始执行远程检测任务` +- 已修复 worker 在“代理配置已下发但尚未开始检测”时长期停留 `未刷新` 的问题 +- `domaincheck-worker` 现在会在以下时机主动触发代理池刷新: + - worker 启动后 + - `proxy_config / thread_count / node_thread_counts / runtime_settings` 更新后 +- 当前后台状态已不再误报“未刷新” +- 代理策略已切到: + - 只去掉过期 / 非法代理 + - 跳过预验证 + - 直接入池执行 + - 真实失败后立即淘汰并补刷 +- 最新实测结果已经明确: + - 原始代理 `270` 个 + - 预验证 `0` 个 + - 当前可用代理 `270` 个 + - 最近状态: + - `proxy_last_refresh_status = 直入池 270 个(跳过预验证)` + - `available_proxy_count = 270` + +### 本轮已完成 + +- `sync_push_service` + - 已修复 `detect_task_ingest` 只落 `domains`、不落本地 `detect_jobs/detect_job_items` 的缺口 + - controller 现在会为拉回来的海外批次持续创建本地镜像任务: + - `sync-overseas-7380` + - `sync-overseas-7383` + - `sync-overseas-7392` + - 持续增长中 - `mainland-controller-01` - - 已拉到 `main` 最新提交 `c33f4f1` - 已重启 `domaincheck-worker` - - 发现运行环境漂移: - - `/etc/default/domaincheck-worker` 中 `REDIS_PASSWORD` 为空 - - 已最小修正该节点运行环境后再次重启 - - 修正后明确出现: - - `Redis 连接成功: 127.0.0.1:6379` - - `已接受检测启动指令` - - `开始执行远程检测任务` + - 已明确进入真实队列: + - `从任务队列获取到 800 个需要检测的域名` + - `最大线程数: 100` + - `当前实际线程数量: 1/100 ... 6/100` +- `mainland-worker-01` + - 已修复 `sync-pull` 控制消息上下文绑定缺口: + - `detect_worker._set_active_cycle_context` 现在会读取 + - `target_job_id` + - `target_job_code` + - 已重启 `domaincheck-worker` + - 已明确进入真实队列: + - `从任务队列获取到 400 个需要检测的域名` + - `最大线程数: 50` + - `当前实际线程数量: 1/50 ... 3/50` + - 已明确把运行日志回传到 overseas: + - `开始执行检测任务,来源: redis-control` + - `开始执行域名检测任务,正在加载配置` + - `开始检测,正在刷新代理池` + - `开始创建线程,当前批次域名数: 400,最大线程数: 50` + - `当前实际线程数量: 1/50` + - `当前实际线程数量: 2/50` + - `当前实际线程数量: 3/50` +- `runtime_projection` + - 已定位 controller `domain-api/runtime/detect_runs.json` 权限错误 + - 已把 `runtime` 目录和 `detect_runs.json` 改回 `www:www` + - `domaincheck-sync-agent` 最新日志已从 + - `partial_success` + - 变成: + - `sync_state = success` + - `runtime_projection -> 投影推送成功` +- 海外控制面 `/api/v1/ops/nodes` + - 现已明确看到三台节点都在参与 + - `remote_access_ready = 3` + - `participating = 3` + - `mainland-controller-01.is_current_participant = true` + - `mainland-worker-01.is_current_participant = true` -### 本轮仍未完成的事情 +### 当前最关键的新证据 -- 中央 `items_completed` 仍未在短观察窗口内继续增长 -- `mainland-controller-01` 仍然持续报: - - `代理已启用,但当前无可用代理` -- 因此当前剩余阻塞已经进一步收紧到: - - controller 现场代理池没有可用代理 +- `mainland-controller-01` + - `processed_recent = 115` + - `processed_per_minute = 7.67` + - `detect_runtime.max_threads = 100` +- `mainland-worker-01` + - `processed_recent = 632` + - `processed_per_minute = 42.13` + - `detect_runtime.max_threads = 50` +- `overseas-control-01` + - 仍在执行中央原生队列 +- 当前判断: + - 大陆节点已经不是“纸面在线” + - 已经是真实参与检测执行 -## 本轮最新复查结果 +## 当前主判断 -本轮在完成修复部署后,中央与节点现场出现了新的运行证据: +### 本轮新增结论 -- 活跃任务仍是 `detect-20260417170546-96023e` -- `mainland-worker-01` 最新事件时间已经从旧窗口推进到: - - `2026-04-19 16:37:00` -- `runtime/sync-summary` 最新记录已继续增长到: - - `id = 5491` - - `created_at = 2026-04-19 03:40:45` -- 说明中央已经重新收到新的运行侧同步流量 -- 但短窗口内主进度仍未松动: - - `completed = 21` - - `running = 14` - - `claimed = 34` - - `pending = 931` -- controller 现场最新日志已收紧为: - - `代理已启用,但当前无可用代理` +- `mainland-controller-01` + - 已修复“代理数量不足反向限制并发”的热路径问题 + - 当前海外后台已稳定看到: + - `active_threads ~= 99~100` + - `max_threads = 100` + - 说明 `100` 并发不再只是配置已下发,而是已进入真实运行态 +- `mainland-worker-01` + - 已重新部署同版 `detect_worker.py` + - 已重新接收到新的 `sync-pull` 批次: + - `source_record_id = 7679` + - `target_job_code = sync-overseas-7679` + - 已完成代理抽样校验并进入: + - `开始创建线程,当前批次域名数: 400,最大线程数: 50` + - 当前剩余问题已经缩成: + - worker 本机已启动新批次并创建线程 + - 但海外后台对 `mainland-worker-01.active_threads` 的显示仍偏低,和本机进程线程量不完全一致 + +### 当前唯一剩余收口点 + +- 不再是 controller 并发限制问题 +- 当前唯一剩余收口点变成: + - `mainland-worker-01` 的运行态上报口径仍需继续和真实执行量对齐 + - 以及检测主计数 `pending/completed/running` 继续推进 + +### 已经收口的部分 + +- 大陆节点不再停留在 `pending_bootstrap` +- Agent + SSH 接管已经完成 +- `sync-agent pull_tasks` 已经真正生成本地镜像队列 +- 两台大陆 worker 已经真正吃到 `detect_job_items` +- 后台运行态同步链已经恢复 + +### 当前剩余问题 + +- Detect 页面日志窗口已经不再是单节点 +- `/api/v1/detect/status` 已出现: + - `remote_log_node_count = 2` + - `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]` +- 当前剩余问题缩成一个最小点: + - 检测页主计数 `pending/completed/running` 还没有跟着这轮大陆执行立即推进 +- 前端布局问题已收口,后续不再停留在“页面结构挡住观察” +- 代理链当前也已收口到真实口径,不再停留在“配置有了但没刷新” +- 下一步应继续观察这 `270` 个直入池代理在真实检测里的淘汰速度与吞吐提升 +- 现在更应该同时看: + - `/api/v1/detect/status` + - `/api/v1/runtime/debug-events` + - `/api/v1/ops/nodes` + - worker `current actual threads` 日志 + +## 下一步唯一主批次 + +唯一主批次保持为:`J2-检测执行收口批` + +下一步只做: + +- 继续观察 Detect 页面主计数是否跟上最新运行态 +- 继续确认 `detect_result_projection` / 结果统计回推是否稳定推进 +- 继续收口 `mainland-worker-01` 的运行态上报,使后台显示与本机真实线程量一致 +- 若仍有显示偏差,只修统计/上报口径,不新扩功能 + +## 候选批次 + +### Candidate J3 + +名称:结果计数收口批 + +进入条件: + +- 节点已经真实执行 +- 日志窗口已经恢复 +- 但 Detect 页面主计数仍不前进 + +只做: + +- 复核 `detect_result_projection` +- 复核结果统计回推 +- 不改控制面结构 + +### Candidate J4 + +名称:吞吐稳定性观察批 + +进入条件: + +- 页面口径已恢复 +- 继续确认大陆两节点吞吐是否稳定,不再回落 + +只做: + +- 继续观察 `processed_recent` +- 继续观察代理池质量 +- 继续观察 `active_threads/max_threads` + +## 暂停项 + +以下任务现在不应继续推进: + +- 新页面 +- 新模块 +- 新发布动作 +- 新专题文档 +- 与检测执行收口无关的控制面增强 ## 关键证据 -### 证据 1:接管与同步已通 +### 证据 1:镜像队列已经落地 -当前中央状态: +controller `sync-agent` 最新 `task pull tick` 已出现: -- `remote_access_ready = 3/3` -- `log_sync_state = full_capture` -- mainland controller 的 `domaincheck-sync-agent` 已在新进程上运行 +- `target_job_id` +- `target_job_code` +- `queued_count = 200` +- `worker_start_ok = True` 说明: -- 当前不是接管问题 -- 也不是日志回传问题 -- 更不是同步链完全断开 +- controller 现在不是只同步域名 +- 而是在本地持续创建可执行队列 -### 证据 2:检测任务当前没有继续出新结果 +### 证据 2:大陆 worker 已进入真实队列 -连续 40 秒前后对比结果完全一致: +现场日志已经明确出现: -- `JOB_PROGRESS = 2.1` -- `items_completed = 21` -- `items_running = 14` -- `items_claimed = 34` -- `items_pending = 931` +- `mainland-controller-01 -> 从任务队列获取到 800 个需要检测的域名` +- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名` 说明: -- 当前不是“页面慢一拍” -- 而是执行面这段时间确实没有继续产出 +- “并发没起来”的主要根因已经修正 +- 它们不是在跑旧兼容链路 -### 证据 3:中央 mainland 逐条结果没有继续增长 +### 证据 3:运行态同步已恢复 -当前中央查询结果: +controller `sync-agent` 最新日志已经出现: -- mainland `domain_started/domain_completed/domain_failed/domain_blacklisted` - - 仍为 `10` -- 最新 mainland `detect_result_ingest` - - 仍为 `5382` +- `sync_state = success` +- `runtime_projection -> 投影推送成功` 说明: -- 首批同步成功过 -- 但后续并没有继续流入新逐条结果 +- 后台运行态/日志窗口链路已经不再被权限错误卡死 -### 证据 4:controller 现场日志已指向代理池可用性为 0 +### 证据 4:海外 `/ops/nodes` 已把大陆节点判定为真实参与者 -`mainland-controller-01` 现场日志显示: +当前海外控制面已经显示: -- `当前可用代理数: 0` -- `最近结果: 刷新成功,可用 0 个` -- 免费检测链包含: - - 注册查询 - - 百度 site - - 360 site - - 站长之家 - - 爱站 - - 时光机 - -进一步的远端 `logs.collect(domaincheck-worker)` 结果显示: - -- controller 能从 6 个代理源成功拉到原始代理 -- 但在抽样验证后: - - `代理池刷新完成,共 0 个可用代理` -- 失败原因集中在: - - `ProxyError@https://m.baidu.com` - - `Unable to connect to proxy` - - `ConnectTimeoutError` +- `mainland-controller-01` + - `online_busy` + - `is_current_participant = true` + - `processed_recent = 115` +- `mainland-worker-01` + - `online_busy` + - `is_current_participant = true` + - `processed_recent = 632` 说明: -- 当前不是代理源接口没返回 -- 而是“拿到的代理全部验不过” -- 主阻塞已经可以精确收紧到 controller 代理池不可用 +- 现在大陆两台都已进入真实参与态 +- 下一步不再是接管问题,而是页面口径与吞吐稳定性问题 -### 证据 5:worker 的时光机异常存在,但不是第一主因 +### 证据 5:Detect 页面远端日志已恢复双节点 -`mainland-worker-01` 的远端 `domaincheck-worker` 日志显示: +当前 `/api/v1/detect/status` 已显示: -- 存在: - - `时光机检测 外部依赖异常,步骤降级继续执行` -- 同时仍可见: - - `域名检测完成` -- 最近收到控制消息后: - - `收到启动检测指令,但检测任务已在运行,忽略重复启动` +- `remote_log_node_count = 2` +- `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]` + +最近日志样本已出现: + +- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名` +- `mainland-worker-01 -> 开始创建线程,当前批次域名数: 400,最大线程数: 50` +- `mainland-worker-01 -> 当前实际线程数量: 1/50` +- `mainland-worker-01 -> 当前实际线程数量: 2/50` +- `mainland-worker-01 -> 当前实际线程数量: 3/50` 说明: -- worker 并不是完全不能执行 -- 时光机异常是客观存在的次级问题 -- 但它不像 controller 代理池为 0 那样直接卡住整体吞吐 +- worker 不再是“只在节点本地运行、页面看不到” +- 检测页日志链已经真正接上 mainland worker ### 证据 6:full_capture 已开启,但源日志时间没有继续前进 diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929.pid b/docs/ops_center_runtime/night_runs/night_run_20260419_012929.pid deleted file mode 100644 index 303961c..0000000 --- a/docs/ops_center_runtime/night_runs/night_run_20260419_012929.pid +++ /dev/null @@ -1 +0,0 @@ -683962 diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/go_live.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/go_live.json new file mode 100644 index 0000000..1d898a3 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/go_live.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 02:29:55","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断,但仍有上线前关注项,建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":3,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/launchpad.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/launchpad.json new file mode 100644 index 0000000..904f058 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/launchpad.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"channel":"stable","control_plane_base_url":"http://127.0.0.1:8100","latest_package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"latest_release":{"id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_013833/download","artifact_url_present":true,"checksum":"9da7b89bf9796098f234f388944bab959114fbe667680e5582306b32e77b489d","created_by":"cli/ops-center/latest-package","created_at":"2026-04-18 01:38:35","updated_at":"2026-04-18 01:38:35","is_preview":false,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_013833","package_size_bytes":0,"checksum":"9da7b89bf9796098f234f388944bab959114fbe667680e5582306b32e77b489d","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18 01:38:35","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_013833 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_013833 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"release_summary":{"status_counts":{"ready":2},"total":2,"active_by_channel":{},"rollout_status_counts":{},"rollouts_total":0},"runtime_build_info":{"source":"latest_release","package_name":"domaincheck_release_20260418_215930","commit_sha":"246838ae4c07","commit_ref":"main","generated_at":"2026-04-18T21:59:32","route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"repository_supports_install_command_block":true,"runtime_schema_stale":false},"latest_package_preview":{"message":"最新发布包预览已生成","preview_source":"latest_package_draft","deduplicated":false,"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"resolved_artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","resolved_sha256_download_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256"},"worker_rollout_preview":{"message":"智能 Rollout 预检未通过","preview_source":"latest_package_draft","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"smart_rollout":{"available":true,"mode":"worker","mode_label":"Worker 灰度","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","reason":"","summary":"已按 Worker 灰度 预选 1 台节点,执行路径采用 远端 Agent;优先锁定 mainland;首批 1 台,后续每批 1 台。","dominant_region":"mainland","target_node_codes":["mainland-worker-01"],"target_nodes":[{"node_code":"mainland-worker-01","region":"mainland","role":"worker","status":"busy","current_load":9,"remote_agent_ready":true,"ssh_ready":true,"execution_ready":true,"detect_participating":true,"delivery_queue_state":"healthy"}],"policy":{"auto_start":true,"auto_dispatch":false,"auto_advance":true,"auto_approve":false,"continue_on_failure":false,"continue_on_partial":false,"execution_mode":"remote-agent","first_batch_size":1,"batch_size":1,"deploy_payload":{"restart_services":["domaincheck-worker"],"health_check_urls":[],"health_check_services":["domaincheck-worker"],"health_check_timeout_seconds":10,"health_check_retries":2,"health_check_interval_seconds":2,"rollback_on_failure":true,"switch_current":true}}},"policy_preview":{"risk_level":"high","approval_required":true,"blocked":true,"blocking_reasons":["当前筛选条件未命中任何可 rollout 的目标节点。"],"approval_reasons":["该动作属于 high 风险动作,建议审批后执行。"],"warnings":[],"recommendations":[],"target_summary":{"nodes_total":0,"worker_nodes":0,"control_nodes":0,"effective_worker_nodes":0,"busy_nodes":0,"participating_nodes":0,"offline_nodes":0},"batch_plan":{"first_batch_size":0,"batch_size":0,"batches_total":0,"first_batch_nodes":[],"remaining_after_first_batch":0},"cluster_guardrails":{"online_worker_nodes":3,"dedicated_online_worker_nodes":1,"online_control_nodes":2,"busy_nodes":["mainland-worker-01","overseas-control-01"]},"release_gate":{"status":"no_targets","status_label":"无默认目标","status_type":"warning","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[],"target_node_codes":[],"blocking_reasons":[],"warning_reasons":[],"recommendations":[],"operational_readiness":{"summary":{"nodes_total":0,"managed_nodes":0,"managed_enabled_nodes":0,"agent_online_nodes":0,"ssh_ready_nodes":0,"remote_agent_ready_nodes":0,"execution_ready_nodes":0,"inspection_healthy_nodes":0,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0},"rows":[]},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}}},"rollout_preview":{"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","target_nodes_total":0,"target_node_codes":["mainland-worker-01"],"expected_batches_total":0,"expected_jobs_total":0,"gate":{"status":"no_targets","status_label":"无默认目标","status_type":"warning","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[],"target_node_codes":[],"blocking_reasons":[],"warning_reasons":[],"recommendations":[],"operational_readiness":{"summary":{"nodes_total":0,"managed_nodes":0,"managed_enabled_nodes":0,"agent_online_nodes":0,"ssh_ready_nodes":0,"remote_agent_ready_nodes":0,"execution_ready_nodes":0,"inspection_healthy_nodes":0,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0},"rows":[]},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}},"batch_plan":{"first_batch_size":0,"batch_size":0,"batches_total":0,"first_batch_nodes":[],"remaining_after_first_batch":0},"summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"rollout_preview"},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。"},"requires_confirmation":false,"blocked":true,"rollout_created":false,"release_deduplicated":false},"control_rollout_preview":{"message":"智能 Rollout 需要确认","preview_source":"latest_package_draft","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"smart_rollout":{"available":true,"mode":"control","mode_label":"Control 发布","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","reason":"","summary":"已按 Control 发布 预选 1 台节点,执行路径采用 远端 Agent;优先锁定 mainland;首批 1 台,后续每批 1 台。","dominant_region":"mainland","target_node_codes":["mainland-controller-01"],"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"remote_agent_ready":true,"ssh_ready":true,"execution_ready":true,"detect_participating":true,"delivery_queue_state":"healthy"}],"policy":{"auto_start":true,"auto_dispatch":false,"auto_advance":true,"auto_approve":false,"continue_on_failure":false,"continue_on_partial":false,"execution_mode":"remote-agent","first_batch_size":1,"batch_size":1,"deploy_payload":{"restart_services":["domaincheck-api","domaincheck-worker","domaincheck-sync-agent"],"health_check_urls":["http://127.0.0.1:8100/health"],"health_check_services":["domaincheck-api","domaincheck-worker","domaincheck-sync-agent"],"health_check_timeout_seconds":10,"health_check_retries":2,"health_check_interval_seconds":2,"rollback_on_failure":true,"switch_current":true}}},"policy_preview":{"risk_level":"high","approval_required":true,"blocked":false,"blocking_reasons":[],"approval_reasons":["本次 rollout 覆盖 control 节点,建议强制审批,并把 control 放在最后一批。","该动作属于 high 风险动作,建议审批后执行。"],"warnings":["命中了 1 个当前有负载的节点,建议优先确认任务是否可中断。","命中了 1 个正在参与检测的节点,建议优先滚动到空闲节点。"],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。","建议把 control 节点放在 rollout 末尾,并确保 worker 已先通过健康检查。","建议先等参与检测的节点降为空闲,再推进该批次。"],"target_summary":{"nodes_total":1,"worker_nodes":0,"control_nodes":1,"effective_worker_nodes":1,"busy_nodes":1,"participating_nodes":1,"offline_nodes":0},"batch_plan":{"first_batch_size":1,"batch_size":1,"batches_total":1,"first_batch_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"deploy_channel":"stable"}],"remaining_after_first_batch":0},"cluster_guardrails":{"online_worker_nodes":3,"dedicated_online_worker_nodes":1,"online_control_nodes":2,"busy_nodes":["mainland-worker-01","overseas-control-01"]},"release_gate":{"status":"preview_ready","status_label":"预览可发 Rollout","status_type":"success","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"ssh_host":"121.204.244.188","ssh_user":"root","ssh_port":22,"auth_mode":"key","deploy_channel":"stable","title":"mainland-controller-01"}],"target_node_codes":["mainland-controller-01"],"blocking_reasons":[],"warning_reasons":[],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。"],"operational_readiness":{"summary":{"nodes_total":1,"managed_nodes":1,"managed_enabled_nodes":1,"agent_online_nodes":1,"ssh_ready_nodes":1,"remote_agent_ready_nodes":1,"execution_ready_nodes":1,"inspection_healthy_nodes":1,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_noop_nodes":0},"rows":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"agent_managed":true,"is_managed":true,"is_enabled":true,"agent_state":"online_busy","agent_state_label":"已接管","agent_reason":"Agent 最近心跳 9 秒前,节点当前负载 5。","is_agent_online":true,"has_ssh_access":true,"ssh_ready":true,"remote_agent_ready":true,"execution_ready":true,"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","inspection_status":"healthy","inspection_status_label":"巡检通过","inspection_reason":"最近一轮健康快照、Worker 日志和诊断包均已成功。","inspection_success_count":3,"inspection_ready":true,"inspection_status_map":{"health.snapshot":"success","logs.collect":"success","diagnostics.collect":"success"},"delivery_queue_state":"healthy","delivery_queue_label":"正常","delivery_queue_reason":"当前没有待重试回执,也没有死信记录。 最近一次队列冲刷:2026-04-19 15:29:32。","delivery_queue_pending_count":0,"delivery_queue_dead_letter_count":0,"current_release_version":"","desired_release_version":"domaincheck_release_20260418_215930","onboarding_stage_code":"acceptance_ready","onboarding_stage_label":"待接管验收","onboarding_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_action":"run_acceptance","recovery_label":"执行接管验收","recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_command_hint":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recovery_window":"acceptance","risk_level":"low","blocking_reasons":[],"warning_reasons":[],"recommended_action_code":"","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate","node_code":"mainland-controller-01"}}]},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}}},"rollout_preview":{"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"expected_batches_total":1,"expected_jobs_total":1,"gate":{"status":"preview_ready","status_label":"预览可发 Rollout","status_type":"success","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"ssh_host":"121.204.244.188","ssh_user":"root","ssh_port":22,"auth_mode":"key","deploy_channel":"stable","title":"mainland-controller-01"}],"target_node_codes":["mainland-controller-01"],"blocking_reasons":[],"warning_reasons":[],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。"],"operational_readiness":{"summary":{"nodes_total":1,"managed_nodes":1,"managed_enabled_nodes":1,"agent_online_nodes":1,"ssh_ready_nodes":1,"remote_agent_ready_nodes":1,"execution_ready_nodes":1,"inspection_healthy_nodes":1,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_noop_nodes":0},"rows":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"agent_managed":true,"is_managed":true,"is_enabled":true,"agent_state":"online_busy","agent_state_label":"已接管","agent_reason":"Agent 最近心跳 9 秒前,节点当前负载 5。","is_agent_online":true,"has_ssh_access":true,"ssh_ready":true,"remote_agent_ready":true,"execution_ready":true,"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","inspection_status":"healthy","inspection_status_label":"巡检通过","inspection_reason":"最近一轮健康快照、Worker 日志和诊断包均已成功。","inspection_success_count":3,"inspection_ready":true,"inspection_status_map":{"health.snapshot":"success","logs.collect":"success","diagnostics.collect":"success"},"delivery_queue_state":"healthy","delivery_queue_label":"正常","delivery_queue_reason":"当前没有待重试回执,也没有死信记录。 最近一次队列冲刷:2026-04-19 15:29:32。","delivery_queue_pending_count":0,"delivery_queue_dead_letter_count":0,"current_release_version":"","desired_release_version":"domaincheck_release_20260418_215930","onboarding_stage_code":"acceptance_ready","onboarding_stage_label":"待接管验收","onboarding_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_action":"run_acceptance","recovery_label":"执行接管验收","recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_command_hint":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recovery_window":"acceptance","risk_level":"low","blocking_reasons":[],"warning_reasons":[],"recommended_action_code":"","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate","node_code":"mainland-controller-01"}}]},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}},"batch_plan":{"first_batch_size":1,"batch_size":1,"batches_total":1,"first_batch_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"deploy_channel":"stable"}],"remaining_after_first_batch":0},"summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"rollout_preview"},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。"},"requires_confirmation":true,"blocked":false,"rollout_created":false,"release_deduplicated":false},"launchpad_status":{"status":"blocked","status_label":"发布门禁阻断","summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recommended_action_code":"run_acceptance","recommended_command":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recommended_execution_mode":"remote-agent","recommended_execution_mode_label":"远端 Agent","recommended_target_node_code":"mainland-controller-01","recommended_recovery_label":"执行接管验收","summary_text":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_gap_nodes_total":1,"recommended_recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recommended_onboarding_stage_code":"acceptance_ready","recommended_onboarding_stage_label":"待接管验收"}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/playbook_runs.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/playbook_runs.json new file mode 100644 index 0000000..bcb3eab --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/playbook_runs.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"runs":[{"run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/codex/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"created_at":"2026-04-19 01:20:34","updated_at":"2026-04-19 01:20:47","status_counts":{"success":9},"jobs_total":9,"success_jobs_total":9,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":68,"job_code":"ops-20260419012035-dfa01d","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:20:35","started_at":"2026-04-19 01:20:38","finished_at":"2026-04-19 01:20:40","updated_at":"2026-04-19 01:20:40","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":71,"job_code":"ops-20260419012035-634387","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:20:35","started_at":"2026-04-19 01:20:40","finished_at":"2026-04-19 01:20:42","updated_at":"2026-04-19 01:20:42","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":73,"job_code":"ops-20260419012036-4babe3","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:20:36","started_at":"2026-04-19 01:20:43","finished_at":"2026-04-19 01:20:47","updated_at":"2026-04-19 01:20:47","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":73,"job_code":"ops-20260419012036-4babe3","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:20:36","started_at":"2026-04-19 01:20:43","finished_at":"2026-04-19 01:20:47","updated_at":"2026-04-19 01:20:47","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":3,"steps_success":3,"steps_running":0,"steps_problem":0,"steps_terminal":3,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 01:12:57","updated_at":"2026-04-19 01:13:27","status_counts":{"success":5},"jobs_total":5,"success_jobs_total":5,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":56,"job_code":"ops-20260419011257-998c9e","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:57","started_at":"2026-04-19 01:13:03","finished_at":"2026-04-19 01:13:06","updated_at":"2026-04-19 01:13:06","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":58,"job_code":"ops-20260419011258-daf92f","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:07","finished_at":"2026-04-19 01:13:09","updated_at":"2026-04-19 01:13:09","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":60,"job_code":"ops-20260419011258-48e2cf","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:09","finished_at":"2026-04-19 01:13:12","updated_at":"2026-04-19 01:13:12","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 已完成收口。","summary_text":"收集 Node Agent 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":62,"job_code":"ops-20260419011258-591b52","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:19","updated_at":"2026-04-19 01:13:19","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":64,"job_code":"ops-20260419011258-a55cec","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:22","finished_at":"2026-04-19 01:13:27","updated_at":"2026-04-19 01:13:27","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":64,"job_code":"ops-20260419011258-a55cec","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:22","finished_at":"2026-04-19 01:13:27","updated_at":"2026-04-19 01:13:27","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":5,"steps_success":5,"steps_running":0,"steps_problem":0,"steps_terminal":5,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 01:12:57","updated_at":"2026-04-19 01:13:17","status_counts":{"success":5},"jobs_total":5,"success_jobs_total":5,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":57,"job_code":"ops-20260419011257-0818f7","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:57","started_at":"2026-04-19 01:12:59","finished_at":"2026-04-19 01:13:02","updated_at":"2026-04-19 01:13:02","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":59,"job_code":"ops-20260419011258-165f37","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:02","finished_at":"2026-04-19 01:13:04","updated_at":"2026-04-19 01:13:04","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":61,"job_code":"ops-20260419011258-b326cf","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:04","finished_at":"2026-04-19 01:13:07","updated_at":"2026-04-19 01:13:07","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 已完成收口。","summary_text":"收集 Node Agent 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":63,"job_code":"ops-20260419011258-154d4e","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:11","finished_at":"2026-04-19 01:13:14","updated_at":"2026-04-19 01:13:14","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":65,"job_code":"ops-20260419011258-9eef54","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:17","updated_at":"2026-04-19 01:13:17","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":65,"job_code":"ops-20260419011258-9eef54","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:17","updated_at":"2026-04-19 01:13:17","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":5,"steps_success":5,"steps_running":0,"steps_problem":0,"steps_terminal":5,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 01:10:09","updated_at":"2026-04-19 01:10:29","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":47,"job_code":"ops-20260419011009-cbd7d6","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:13","finished_at":"2026-04-19 01:10:15","updated_at":"2026-04-19 01:10:15","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":49,"job_code":"ops-20260419011009-98c6d6","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:16","finished_at":"2026-04-19 01:10:18","updated_at":"2026-04-19 01:10:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":51,"job_code":"ops-20260419011009-f0a298","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:18","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":53,"job_code":"ops-20260419011010-52f4b0","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:22","finished_at":"2026-04-19 01:10:26","updated_at":"2026-04-19 01:10:26","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":55,"job_code":"ops-20260419011010-9bea3c","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:27","finished_at":"2026-04-19 01:10:29","updated_at":"2026-04-19 01:10:29","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":55,"job_code":"ops-20260419011010-9bea3c","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:27","finished_at":"2026-04-19 01:10:29","updated_at":"2026-04-19 01:10:29","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":51,"job_code":"ops-20260419011009-f0a298","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:18","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 01:10:09","updated_at":"2026-04-19 01:10:27","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":46,"job_code":"ops-20260419011009-e075fe","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:09","finished_at":"2026-04-19 01:10:13","updated_at":"2026-04-19 01:10:13","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":48,"job_code":"ops-20260419011009-0a0c70","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:14","finished_at":"2026-04-19 01:10:18","updated_at":"2026-04-19 01:10:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":50,"job_code":"ops-20260419011009-c091a0","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:19","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":52,"job_code":"ops-20260419011010-1b63ee","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:21","finished_at":"2026-04-19 01:10:23","updated_at":"2026-04-19 01:10:23","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":54,"job_code":"ops-20260419011010-fb40be","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:24","finished_at":"2026-04-19 01:10:27","updated_at":"2026-04-19 01:10:27","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":54,"job_code":"ops-20260419011010-fb40be","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:24","finished_at":"2026-04-19 01:10:27","updated_at":"2026-04-19 01:10:27","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":50,"job_code":"ops-20260419011009-c091a0","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:19","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"nightly-b1","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 00:33:57","updated_at":"2026-04-19 00:49:41","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":41,"job_code":"ops-20260419003357-2be293","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:20","finished_at":"2026-04-19 00:49:22","updated_at":"2026-04-19 00:49:22","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":42,"job_code":"ops-20260419003357-1e65a2","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:23","finished_at":"2026-04-19 00:49:26","updated_at":"2026-04-19 00:49:26","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":43,"job_code":"ops-20260419003357-34e18d","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:30","updated_at":"2026-04-19 00:49:30","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":44,"job_code":"ops-20260419003357-6ad871","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:31","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":45,"job_code":"ops-20260419003357-e6ef6a","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:38","finished_at":"2026-04-19 00:49:41","updated_at":"2026-04-19 00:49:41","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":45,"job_code":"ops-20260419003357-e6ef6a","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:38","finished_at":"2026-04-19 00:49:41","updated_at":"2026-04-19 00:49:41","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":43,"job_code":"ops-20260419003357-34e18d","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:30","updated_at":"2026-04-19 00:49:30","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"nightly-a1","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 00:06:25","updated_at":"2026-04-19 00:49:34","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":36,"job_code":"ops-20260419000625-6f7db9","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:17","finished_at":"2026-04-19 00:49:20","updated_at":"2026-04-19 00:49:20","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":37,"job_code":"ops-20260419000625-0ecb78","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:20","finished_at":"2026-04-19 00:49:22","updated_at":"2026-04-19 00:49:22","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":38,"job_code":"ops-20260419000625-4de21f","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:25","finished_at":"2026-04-19 00:49:27","updated_at":"2026-04-19 00:49:27","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":39,"job_code":"ops-20260419000625-093ea8","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:31","updated_at":"2026-04-19 00:49:31","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":40,"job_code":"ops-20260419000625-28c085","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:33","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":40,"job_code":"ops-20260419000625-28c085","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:33","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":38,"job_code":"ops-20260419000625-4de21f","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:25","finished_at":"2026-04-19 00:49:27","updated_at":"2026-04-19 00:49:27","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","playbook_title":"关键现场日志","group_key":"scene","group_title":"执行现场","requested_by":"cli/ops-center/open_worker_logs_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":1,"expected_jobs_total":3,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:50:32","updated_at":"2026-04-19 00:49:18","status_counts":{"success":3},"jobs_total":3,"success_jobs_total":3,"terminal_jobs_total":3,"steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","template_key":"logs.collect","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":34,"job_code":"ops-20260418225032-8688b9","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:50:32","started_at":"2026-04-19 00:49:15","finished_at":"2026-04-19 00:49:18","updated_at":"2026-04-19 00:49:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 关键日志 已完成收口。","summary_text":"收集 Worker 关键日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}}],"latest_job":{"id":34,"job_code":"ops-20260418225032-8688b9","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:50:32","started_at":"2026-04-19 00:49:15","finished_at":"2026-04-19 00:49:18","updated_at":"2026-04-19 00:49:18","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/ops-center/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:45:53","updated_at":"2026-04-19 00:49:14","status_counts":{"success":8,"failed":1},"jobs_total":9,"success_jobs_total":8,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":22,"job_code":"ops-20260418224553-beaa1b","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:00","finished_at":"2026-04-19 00:49:05","updated_at":"2026-04-19 00:49:05","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"failed":1,"success":2},"terminal_jobs_total":3,"success_jobs_total":2,"latest_job":{"id":27,"job_code":"ops-20260418224553-600be5","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:06","finished_at":"2026-04-19 00:49:07","updated_at":"2026-04-19 00:49:07","error_message":""},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Worker 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":31,"job_code":"ops-20260418224553-897976","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:10","finished_at":"2026-04-19 00:49:14","updated_at":"2026-04-19 00:49:14","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":31,"job_code":"ops-20260418224553-897976","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:10","finished_at":"2026-04-19 00:49:14","updated_at":"2026-04-19 00:49:14","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"worker_logs","title":"收集 Worker 日志","status":"attention","reason_label":"失败","latest_job":{"id":27,"job_code":"ops-20260418224553-600be5","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:06","finished_at":"2026-04-19 00:49:07","updated_at":"2026-04-19 00:49:07","error_message":""}}],"active_steps":[],"focus_level":"danger","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","focus_summary":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":3,"steps_success":2,"steps_running":0,"steps_problem":1,"steps_terminal":3,"summary":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","playbook_title":"关键现场日志","group_key":"scene","group_title":"执行现场","requested_by":"cli/ops-center/open_worker_logs_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":1,"expected_jobs_total":3,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:45:53","updated_at":"2026-04-19 00:49:00","status_counts":{"failed":1,"success":2},"jobs_total":3,"success_jobs_total":2,"terminal_jobs_total":3,"steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","template_key":"logs.collect","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"failed":1,"success":2},"terminal_jobs_total":3,"success_jobs_total":2,"latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 关键日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Worker 关键日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}}],"latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","status":"attention","reason_label":"失败","latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""}}],"active_steps":[],"focus_level":"danger","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","focus_summary":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":1,"steps_success":0,"steps_running":0,"steps_problem":1,"steps_terminal":1,"summary":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}},{"run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","playbook_title":"生成节点接入工单","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"control-plane","execution_mode_label":"控制面","stop_on_failure":true,"step_count":1,"expected_jobs_total":1,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","status_counts":{"success":1},"jobs_total":1,"success_jobs_total":1,"terminal_jobs_total":1,"steps":[{"step_key":"bootstrap","title":"生成 Node Agent 接入工单","template_key":"node.bootstrap","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":20,"job_code":"ops-20260418194421-bc277a","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:21","started_at":"2026-04-18 19:44:21","finished_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"生成 Node Agent 接入工单 已完成收口。","summary_text":"生成 Node Agent 接入工单 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"bootstrap","focus_step_title":"生成 Node Agent 接入工单","event_key":"","node_code":""}}],"latest_job":{"id":20,"job_code":"ops-20260418194421-bc277a","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:21","started_at":"2026-04-18 19:44:21","finished_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","bootstrap_state_counts":{"total":1,"online":1,"pending_bootstrap":0,"runtime_only":0,"stale":0,"unmanaged":0,"unknown":0},"status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","summary_text":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","focus_ref":{"kind":"playbook_run","run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-d7a22054b2","playbook_key":"onboarding.bootstrap","playbook_title":"生成节点接入工单","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"control-plane","execution_mode_label":"控制面","stop_on_failure":true,"step_count":1,"expected_jobs_total":1,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-18 19:44:00","updated_at":"2026-04-18 19:44:00","status_counts":{"success":1},"jobs_total":1,"success_jobs_total":1,"terminal_jobs_total":1,"steps":[{"step_key":"bootstrap","title":"生成 Node Agent 接入工单","template_key":"node.bootstrap","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":19,"job_code":"ops-20260418194400-41b4dd","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:00","started_at":"2026-04-18 19:44:00","finished_at":"2026-04-18 19:44:00","updated_at":"2026-04-18 19:44:00","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"生成 Node Agent 接入工单 已完成收口。","summary_text":"生成 Node Agent 接入工单 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-d7a22054b2","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"bootstrap","focus_step_title":"生成 Node Agent 接入工单","event_key":"","node_code":""}}],"latest_job":{"id":19,"job_code":"ops-20260418194400-41b4dd","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:00","started_at":"2026-04-18 19:44:00","finished_at":"2026-04-18 19:44:00","updated_at":"2026-04-18 19:44:00","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","bootstrap_state_counts":{"total":1,"online":1,"pending_bootstrap":0,"runtime_only":0,"stale":0,"unmanaged":0,"unknown":0},"status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","summary_text":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","focus_ref":{"kind":"playbook_run","run_code":"pbr-d7a22054b2","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}}],"summary":{"total":12,"filtered_total":17,"unfiltered_total":17,"success":6,"running":0,"attention":6,"queued":0,"scanned_jobs":77,"available_status_counts":{"success":11,"attention":6},"available_group_counts":{"diagnostics":2,"onboarding":13,"scene":2},"filters":{"status":"","group_key":"","query":""}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/scene_overseas_control_01.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/scene_overseas_control_01.json new file mode 100644 index 0000000..652db72 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/scene_overseas_control_01.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"full_capture","status_label":"全量观察","status_type":"success","summary":"节点当前正在参与检测,已保留 1 条现场日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":1,"key_line_count":0,"full_line_count":1,"last_at":"2026-04-19 01:20:35","last_line":"[2026-04-19 01:20:35] [overseas-control-01] 2026-04-19 01:00:18.866 | INFO | __main__:start_detection:2575 - 开始执行域名检测任务"},"missing_reason_code":"","missing_reason":"","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 02:30:02"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/stack.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/stack.json new file mode 100644 index 0000000..b3066b3 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/stack.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"generated_at":"2026-04-19 02:30:01","diagnosis":{"contract_key":"ops_stack_diagnosis_contract","contract_version":"v1","registry_version":"2026-04-18","base_url":"http://127.0.0.1:8100","contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_driver_contract","ops_playbook_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_driver_contract","title":"Ops Driver Contract","status":"active","version":"v1","summary":"冻结 overview / driver-feed / codex-brief / driver-actions / runbook sequence 的统一驾驶 contract。","primary_endpoint":"/api/v1/ops/driver-feed","schema_doc_path":"docs/schemas/ops_driver_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_driver_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/driver-feed","/api/v1/ops/codex-brief","/api/v1/ops/driver-actions/resolve","/api/v1/ops/driver-actions/execute-resolved","/api/v1/ops/codex-actions/resolve","/api/v1/ops/codex-actions/execute","/api/v1/ops/activity-stream","/api/v1/ops/runbook","/api/v1/ops/runbook/sequences/{sequence_key}/resolve","/api/v1/ops/runbook/sequences/{sequence_key}/execute","/api/v1/ops/driver-actions/preview","/api/v1/ops/driver-actions/execute"],"related_contract_keys":["release_hub_contract","ops_playbook_contract","ops_observability_contract","ops_stack_diagnosis_contract"]},{"key":"ops_playbook_contract","title":"Ops Playbook Contract","status":"active","version":"v1","summary":"冻结 playbook catalog / preview / run / events 的正式编排 contract。","primary_endpoint":"/api/v1/ops/playbooks","schema_doc_path":"docs/schemas/ops_playbook_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_playbook_contract","discovery_endpoints":["/api/v1/ops/playbooks","/api/v1/ops/playbooks/preview","/api/v1/ops/playbooks/execute","/api/v1/ops/playbook-runs","/api/v1/ops/playbook-runs/{run_code}","/api/v1/ops/playbook-runs/{run_code}/events","/api/v1/ops/playbook-runs/{run_code}/rerun","/api/v1/ops/playbook-runs/{run_code}/cancel"],"related_contract_keys":["ops_job_contract","release_hub_contract","ops_driver_contract","ops_observability_contract","ops_stack_diagnosis_contract"]}]},"surface_status":"healthy","automation_status":"attention","stack_status":"attention","issue_total":1,"blocking_issue_total":0,"warning_issue_total":1,"info_issue_total":0,"missing_surfaces":[],"launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"next_step":{"action_code":"focus_playbook_run","source":"overview.primary","reason":"来自 overview.recommendation.primary_action_code","command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","focus_ref":{},"contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"recommended_actions":[{"action_code":"focus_playbook_run","source":"overview.primary","title":"先处理异常编排","reason":"来自 overview.recommendation.primary_action_code","focus_ref":{},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},{"action_code":"fix_managed_nodes","source":"release_hub.launchpad","title":"待补执行面","reason":"来自 release_launchpad.launchpad_status.recommended_action_code","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","contract_keys":["ops_stack_diagnosis_contract","ops_agent_protocol","release_hub_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_agent_protocol","release_hub_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_agent_protocol","title":"Node Agent Protocol","status":"active","version":"v1","summary":"冻结海外控制面与大陆 Node Agent 之间的认证、注册、心跳、拉任务、完成回执与事件回放 contract。","primary_endpoint":"/api/v1/ops/agent/bootstrap-plan","schema_doc_path":"docs/schemas/ops_agent_protocol.md","detail_endpoint":"/api/v1/ops/contracts/ops_agent_protocol","discovery_endpoints":["/api/v1/ops/agent/tokens","/api/v1/ops/agent/bootstrap-plan","/api/v1/ops/agent/register","/api/v1/ops/agent/heartbeat","/api/v1/ops/agent/pull","/api/v1/ops/agent/jobs/{job_id}/start","/api/v1/ops/agent/jobs/{job_id}/complete","/api/v1/ops/agent/jobs/{job_id}/events","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_observability_contract","ops_stack_diagnosis_contract"]},{"key":"release_hub_contract","title":"Release Hub Contract","status":"active","version":"v1","summary":"冻结 Release / Rollout / Launchpad / Default Gate 的对象模型与门禁口径。","primary_endpoint":"/api/v1/ops/releases/launchpad","schema_doc_path":"docs/schemas/release_hub_contract.md","detail_endpoint":"/api/v1/ops/contracts/release_hub_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/releases","/api/v1/ops/releases/latest","/api/v1/ops/releases/launchpad","/api/v1/ops/releases/{release_id}","/api/v1/ops/releases/{release_id}/rollouts","/api/v1/ops/rollouts/{rollout_id}","/api/v1/ops/rollouts/{rollout_id}/jobs","/api/v1/ops/rollouts/{rollout_id}/advance"],"related_contract_keys":["ops_job_contract","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]}]}},{"action_code":"open_playbook_run_latest_events","source":"overview.secondary","title":"先处理异常编排","reason":"来自 overview.recommendation.secondary_action_code","focus_ref":{},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}}],"operator_decision":{"lane":"ops_jobs","priority":"medium","reason_code":"playbook_runs_need_attention","title":"按总检默认下一步继续处理","summary":"来自 overview.recommendation.primary_action_code","next_focus":{},"primary_command_key":"focus_playbook_run","secondary_command_key":""},"next_actions":{"primary_command_key":"focus_playbook_run","primary_command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","secondary_command_key":"","secondary_command":""},"recommended_commands":{"stack-diagnosis":"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","stack-check":"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","doctor":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100","contracts":"bash domain-api/deploy/multi-region/drive_ops_center.sh contracts http://127.0.0.1:8100","runtime-refresh-recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100","api-restart":"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100","focus_playbook_run":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","fix_managed_nodes":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","open_playbook_run_latest_events":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","focus-release-hub":"bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad http://127.0.0.1:8100","release-launchpad-worker":"bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad http://127.0.0.1:8100 worker"},"issues":[{"code":"playbook_runs_need_attention","severity":"warning","layer":"playbook_runs","summary":"最近存在未完全收口的 playbook run,建议先确认执行序列是否停在中间步骤。","detail":"problem_runs_total=3","action_code":"","focus_ref":{},"commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100"],"contract_keys":["ops_stack_diagnosis_contract","ops_playbook_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_playbook_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_playbook_contract","title":"Ops Playbook Contract","status":"active","version":"v1","summary":"冻结 playbook catalog / preview / run / events 的正式编排 contract。","primary_endpoint":"/api/v1/ops/playbooks","schema_doc_path":"docs/schemas/ops_playbook_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_playbook_contract","discovery_endpoints":["/api/v1/ops/playbooks","/api/v1/ops/playbooks/preview","/api/v1/ops/playbooks/execute","/api/v1/ops/playbook-runs","/api/v1/ops/playbook-runs/{run_code}","/api/v1/ops/playbook-runs/{run_code}/events","/api/v1/ops/playbook-runs/{run_code}/rerun","/api/v1/ops/playbook-runs/{run_code}/cancel"],"related_contract_keys":["ops_job_contract","release_hub_contract","ops_driver_contract","ops_observability_contract","ops_stack_diagnosis_contract"]}]}}],"operator_hints":["当前总检可用,但仍有缺口;建议先按 next_step 收口,再进入更细粒度脚本。"],"quick_commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events"]},"api":{"http_status":"200","available":true,"service":"domain-api","version":"0.1.0","api_prefix":"/api/v1","worker_mode":"linux-systemd"},"surface_matrix":{"status_counts":{"total":9,"available":9,"missing":0},"items":[{"name":"api","available":true,"elapsed_ms":0},{"name":"contracts","available":true,"elapsed_ms":0},{"name":"link_snapshot","available":true,"elapsed_ms":0},{"name":"overview","available":true,"elapsed_ms":4513},{"name":"managed_nodes","available":true,"elapsed_ms":256},{"name":"release_hub","available":true,"elapsed_ms":0},{"name":"runtime_build_info","available":true,"elapsed_ms":0},{"name":"playbook_runs","available":true,"elapsed_ms":282},{"name":"activity_stream","available":true,"elapsed_ms":0}]},"contracts":{"http_status":"200","available":true,"elapsed_ms":0,"registry_version":"2026-04-18","schema_version":"v1","contracts_total":11,"contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract","ops_stack_diagnosis_contract","ops_doctor_decision_contract","ops_go_live_signoff_contract","ops_go_live_bundle_contract","ops_go_live_review_contract"],"error":""},"link_snapshot":{"http_status":"200","available":true,"elapsed_ms":0,"status":"attention","headline":"存在 1 个结果批次仍待推送。","next_action_codes":["focus_playbook_run","open_playbook_run_latest_events"],"error":""},"overview":{"http_status":"200","available":true,"elapsed_ms":4513,"recommendation_key":"playbook-run-attention-pbr-9cf64aa152","recommendation_title":"先处理异常编排","primary_action_code":"focus_playbook_run","secondary_action_code":"open_playbook_run_latest_events","focus_ref":{},"execution_scene_summary":"有效执行节点 3 台;正在执行/领任务 3 台;近窗刚有吞吐 0 台;在线但未参与 0 台","execution_scene_counts":{"dispatch_active":3,"recent_only":0,"standby":0,"load_syncing":0},"log_sync":{"status":"full_capture","status_label":"全量观察","mode":"full","mode_label":"全量回传","enabled":true,"covered_participating_nodes":3,"participating_nodes_total":3,"missing_sample_node_codes":[]},"error":""},"managed_nodes":{"http_status":"200","available":true,"elapsed_ms":256,"total":3,"managed_total":3,"managed_enabled":3,"online":3,"agent_ready":3,"ssh_ready":2,"remote_access_ready":3,"participating":3,"dispatch_active":3,"standby":0,"load_syncing":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"queue_pending_records":0,"queue_dead_letter_records":0,"agent_state_counts":{"online_busy":3},"remote_access_state_counts":{"hybrid_ready":2,"agent_ready":1},"delivery_queue_state_counts":{"healthy":3},"problem_nodes":[],"first_handover_gap_node_code":"","first_handover_gap":{},"error":""},"release_hub":{"http_status":"200","available":true,"elapsed_ms":0,"latest_release_version":"domaincheck_release_20260418_013833","latest_release_status":"ready","default_rollout_gate_status":"ready","default_rollout_gate_label":"可发 Rollout","default_rollout_gate_summary":"默认 Release 与默认 Rollout 目标均已通过门禁,当前可直接进入定向或灰度 Rollout。","default_rollout_gate_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"},"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_summary":"最新发布包已经可用,但当前没有可用于智能 Rollout 的在线启用执行节点,需要先补齐托管或接入状态。","recommended_action_code":"fix_managed_nodes","recommended_execution_mode":"","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"error":""},"runtime_build_info":{"http_status":"200","available":true,"elapsed_ms":0,"source":"latest_release","package_name":"domaincheck_release_20260418_215930","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","manifest_path":"/www/wwwroot/getDomain/release/latest_release.json","repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"route_surface":{"mode":"registered","registered_paths_total":150,"surface_flags":{"ops_contracts":true,"ops_contract_detail":true,"ops_stack_diagnosis":true,"ops_node_handover":true,"ops_node_onboarding":true,"ops_node_onboarding_bootstrap_preview":true,"ops_node_onboarding_bootstrap_execute":true,"ops_node_onboarding_acceptance_preview":true,"ops_node_onboarding_acceptance_execute":true,"ops_node_onboarding_recovery_preview":true,"ops_node_onboarding_recovery_execute":true,"ops_node_scene_log":true,"ops_node_handover_bootstrap_plan":true,"ops_driver_feed":true,"ops_codex_brief":true,"ops_activity_stream":true,"ops_runbook_resolve":true,"ops_runbook_execute":true,"runtime_build_info":true},"expected_paths":{"ops_contracts":"/api/v1/ops/contracts","ops_contract_detail":"/api/v1/ops/contracts/{contract_key}","ops_stack_diagnosis":"/api/v1/ops/stack-diagnosis","ops_node_handover":"/api/v1/ops/nodes/{node_code}/handover","ops_node_onboarding":"/api/v1/ops/nodes/{node_code}/onboarding","ops_node_onboarding_bootstrap_preview":"/api/v1/ops/nodes/{node_code}/onboarding/bootstrap/preview","ops_node_onboarding_bootstrap_execute":"/api/v1/ops/nodes/{node_code}/onboarding/bootstrap/execute","ops_node_onboarding_acceptance_preview":"/api/v1/ops/nodes/{node_code}/onboarding/acceptance/preview","ops_node_onboarding_acceptance_execute":"/api/v1/ops/nodes/{node_code}/onboarding/acceptance/execute","ops_node_onboarding_recovery_preview":"/api/v1/ops/nodes/{node_code}/onboarding/recovery/preview","ops_node_onboarding_recovery_execute":"/api/v1/ops/nodes/{node_code}/onboarding/recovery/execute","ops_node_scene_log":"/api/v1/ops/nodes/{node_code}/scene-log","ops_node_handover_bootstrap_plan":"/api/v1/ops/nodes/{node_code}/handover/bootstrap-plan","ops_driver_feed":"/api/v1/ops/driver-feed","ops_codex_brief":"/api/v1/ops/codex-brief","ops_activity_stream":"/api/v1/ops/activity-stream","ops_runbook_resolve":"/api/v1/ops/runbook/sequences/{sequence_key}/resolve","ops_runbook_execute":"/api/v1/ops/runbook/sequences/{sequence_key}/execute","runtime_build_info":"/api/v1/runtime/build-info"},"missing_keys":[],"missing_paths":[],"surface_complete":true},"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"error":""},"playbook_runs":{"http_status":"200","available":true,"elapsed_ms":282,"recent_total":6,"problem_runs_total":3,"problem_runs":[{"run_code":"pbr-9cf64aa152","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-2be333d3ce","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-bfed43ea46","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}}],"error":""},"activity_stream":{"http_status":"200","available":true,"elapsed_ms":0,"recent_total":8,"status_counts":{"running":1,"success":3,"failed":4},"top_items":[{"kind":"execution_scene","status":"running","summary":"有效执行节点 3 台;正在执行/领任务 3 台;近窗刚有吞吐 0 台;在线但未参与 0 台","occurred_at":"2026-04-19 02:29:56","focus_ref":{"kind":"execution_scene","scene_key":"execution-scene:live"},"source_focus_ref":{},"ui_intent_kind":"focus_execution_scene"},{"kind":"ops_job","status":"success","summary":"待检测任务批次拉取并入库成功","occurred_at":"2026-04-19 02:24:53","focus_ref":{"kind":"ops_job","job_id":87,"job_code":"ops-20260419022444-d717dc","action":"runtime.pull_tasks","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"success","summary":"已发送 Worker 控制指令: start_detection","occurred_at":"2026-04-19 02:20:25","focus_ref":{"kind":"ops_job","job_id":84,"job_code":"ops-20260419022013-516abe","action":"runtime.start_detection","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"failed","summary":"Failed to start domaincheck-worker.service: Interactive authentication required.\nSee system logs and 'systemctl status domaincheck-worker.service' for details.","occurred_at":"2026-04-19 02:20:23","focus_ref":{"kind":"ops_job","job_id":83,"job_code":"ops-20260419022013-f12b94","action":"runtime.start_worker","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"success","summary":"已发送 Worker 控制指令: start_detection","occurred_at":"2026-04-19 02:20:22","focus_ref":{"kind":"ops_job","job_id":86,"job_code":"ops-20260419022013-b0b762","action":"runtime.start_detection","target_node_code":"mainland-worker-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"}],"error":""}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/summary.env b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/summary.env new file mode 100644 index 0000000..0119bf6 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_2/summary.env @@ -0,0 +1,16 @@ +CYCLES="2" +GO_LIVE_STATUS="attention" +PUBLISH_READY="false" +LOG_SYNC_STATE="full_capture" +LOG_SYNC_MISSING_NODE_CODES="" +STACK_STATUS="attention" +ISSUE_TOTAL="1" +BLOCKING_ISSUE_TOTAL="0" +ISSUE_CODES="playbook_runs_need_attention" +LAUNCHPAD_STATUS="blocked" +LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance" +PROBLEM_RUNS_TOTAL="0" +PROBLEM_RUN_CODE="" +SCENE_STATUS="full_capture" +SCENE_LINE_COUNT="1" +GENERATED_AT="2026-04-19 02:29:55" diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/go_live.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/go_live.json new file mode 100644 index 0000000..d523565 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/go_live.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 03:30:12","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断,但仍有上线前关注项,建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"partial_coverage","log_sync_mode":"full","log_sync_covered_nodes":2,"log_sync_missing_node_codes":["overseas-control-01"],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/launchpad.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/launchpad.json new file mode 100644 index 0000000..cd47e46 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/launchpad.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"channel":"stable","control_plane_base_url":"http://127.0.0.1:8100","latest_package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"latest_release":{"id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_013833/download","artifact_url_present":true,"checksum":"9da7b89bf9796098f234f388944bab959114fbe667680e5582306b32e77b489d","created_by":"cli/ops-center/latest-package","created_at":"2026-04-18 01:38:35","updated_at":"2026-04-18 01:38:35","is_preview":false,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_013833","package_size_bytes":0,"checksum":"9da7b89bf9796098f234f388944bab959114fbe667680e5582306b32e77b489d","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18 01:38:35","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_013833 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_013833 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"release_summary":{"status_counts":{"ready":2},"total":2,"active_by_channel":{},"rollout_status_counts":{},"rollouts_total":0},"runtime_build_info":{"source":"latest_release","package_name":"domaincheck_release_20260418_215930","commit_sha":"246838ae4c07","commit_ref":"main","generated_at":"2026-04-18T21:59:32","route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"repository_supports_install_command_block":true,"runtime_schema_stale":false},"latest_package_preview":{"message":"最新发布包预览已生成","preview_source":"latest_package_draft","deduplicated":false,"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"resolved_artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","resolved_sha256_download_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256"},"worker_rollout_preview":{"message":"智能 Rollout 预检未通过","preview_source":"latest_package_draft","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"smart_rollout":{"available":true,"mode":"worker","mode_label":"Worker 灰度","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","reason":"","summary":"已按 Worker 灰度 预选 1 台节点,执行路径采用 远端 Agent;优先锁定 mainland;首批 1 台,后续每批 1 台。","dominant_region":"mainland","target_node_codes":["mainland-worker-01"],"target_nodes":[{"node_code":"mainland-worker-01","region":"mainland","role":"worker","status":"busy","current_load":9,"remote_agent_ready":true,"ssh_ready":true,"execution_ready":true,"detect_participating":true,"delivery_queue_state":"healthy"}],"policy":{"auto_start":true,"auto_dispatch":false,"auto_advance":true,"auto_approve":false,"continue_on_failure":false,"continue_on_partial":false,"execution_mode":"remote-agent","first_batch_size":1,"batch_size":1,"deploy_payload":{"restart_services":["domaincheck-worker"],"health_check_urls":[],"health_check_services":["domaincheck-worker"],"health_check_timeout_seconds":10,"health_check_retries":2,"health_check_interval_seconds":2,"rollback_on_failure":true,"switch_current":true}}},"policy_preview":{"risk_level":"high","approval_required":true,"blocked":true,"blocking_reasons":["当前筛选条件未命中任何可 rollout 的目标节点。"],"approval_reasons":["该动作属于 high 风险动作,建议审批后执行。"],"warnings":[],"recommendations":[],"target_summary":{"nodes_total":0,"worker_nodes":0,"control_nodes":0,"effective_worker_nodes":0,"busy_nodes":0,"participating_nodes":0,"offline_nodes":0},"batch_plan":{"first_batch_size":0,"batch_size":0,"batches_total":0,"first_batch_nodes":[],"remaining_after_first_batch":0},"cluster_guardrails":{"online_worker_nodes":3,"dedicated_online_worker_nodes":1,"online_control_nodes":2,"busy_nodes":["mainland-worker-01","overseas-control-01"]},"release_gate":{"status":"no_targets","status_label":"无默认目标","status_type":"warning","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[],"target_node_codes":[],"blocking_reasons":[],"warning_reasons":[],"recommendations":[],"operational_readiness":{"summary":{"nodes_total":0,"managed_nodes":0,"managed_enabled_nodes":0,"agent_online_nodes":0,"ssh_ready_nodes":0,"remote_agent_ready_nodes":0,"execution_ready_nodes":0,"inspection_healthy_nodes":0,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0},"rows":[]},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}}},"rollout_preview":{"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","target_nodes_total":0,"target_node_codes":["mainland-worker-01"],"expected_batches_total":0,"expected_jobs_total":0,"gate":{"status":"no_targets","status_label":"无默认目标","status_type":"warning","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[],"target_node_codes":[],"blocking_reasons":[],"warning_reasons":[],"recommendations":[],"operational_readiness":{"summary":{"nodes_total":0,"managed_nodes":0,"managed_enabled_nodes":0,"agent_online_nodes":0,"ssh_ready_nodes":0,"remote_agent_ready_nodes":0,"execution_ready_nodes":0,"inspection_healthy_nodes":0,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0},"rows":[]},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}},"batch_plan":{"first_batch_size":0,"batch_size":0,"batches_total":0,"first_batch_nodes":[],"remaining_after_first_batch":0},"summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"rollout_preview"},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。"},"requires_confirmation":false,"blocked":true,"rollout_created":false,"release_deduplicated":false},"control_rollout_preview":{"message":"智能 Rollout 需要确认","preview_source":"latest_package_draft","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"smart_rollout":{"available":true,"mode":"control","mode_label":"Control 发布","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","reason":"","summary":"已按 Control 发布 预选 1 台节点,执行路径采用 远端 Agent;优先锁定 mainland;首批 1 台,后续每批 1 台。","dominant_region":"mainland","target_node_codes":["mainland-controller-01"],"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"remote_agent_ready":true,"ssh_ready":true,"execution_ready":true,"detect_participating":true,"delivery_queue_state":"healthy"}],"policy":{"auto_start":true,"auto_dispatch":false,"auto_advance":true,"auto_approve":false,"continue_on_failure":false,"continue_on_partial":false,"execution_mode":"remote-agent","first_batch_size":1,"batch_size":1,"deploy_payload":{"restart_services":["domaincheck-api","domaincheck-worker","domaincheck-sync-agent"],"health_check_urls":["http://127.0.0.1:8100/health"],"health_check_services":["domaincheck-api","domaincheck-worker","domaincheck-sync-agent"],"health_check_timeout_seconds":10,"health_check_retries":2,"health_check_interval_seconds":2,"rollback_on_failure":true,"switch_current":true}}},"policy_preview":{"risk_level":"high","approval_required":true,"blocked":false,"blocking_reasons":[],"approval_reasons":["本次 rollout 覆盖 control 节点,建议强制审批,并把 control 放在最后一批。","该动作属于 high 风险动作,建议审批后执行。"],"warnings":["命中了 1 个当前有负载的节点,建议优先确认任务是否可中断。","命中了 1 个正在参与检测的节点,建议优先滚动到空闲节点。"],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。","建议把 control 节点放在 rollout 末尾,并确保 worker 已先通过健康检查。","建议先等参与检测的节点降为空闲,再推进该批次。"],"target_summary":{"nodes_total":1,"worker_nodes":0,"control_nodes":1,"effective_worker_nodes":1,"busy_nodes":1,"participating_nodes":1,"offline_nodes":0},"batch_plan":{"first_batch_size":1,"batch_size":1,"batches_total":1,"first_batch_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"deploy_channel":"stable"}],"remaining_after_first_batch":0},"cluster_guardrails":{"online_worker_nodes":3,"dedicated_online_worker_nodes":1,"online_control_nodes":2,"busy_nodes":["mainland-worker-01","overseas-control-01"]},"release_gate":{"status":"preview_ready","status_label":"预览可发 Rollout","status_type":"success","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"ssh_host":"121.204.244.188","ssh_user":"root","ssh_port":22,"auth_mode":"key","deploy_channel":"stable","title":"mainland-controller-01"}],"target_node_codes":["mainland-controller-01"],"blocking_reasons":[],"warning_reasons":[],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。"],"operational_readiness":{"summary":{"nodes_total":1,"managed_nodes":1,"managed_enabled_nodes":1,"agent_online_nodes":1,"ssh_ready_nodes":1,"remote_agent_ready_nodes":1,"execution_ready_nodes":1,"inspection_healthy_nodes":1,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_noop_nodes":0},"rows":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"agent_managed":true,"is_managed":true,"is_enabled":true,"agent_state":"online_busy","agent_state_label":"已接管","agent_reason":"Agent 最近心跳 10 秒前,节点当前负载 5。","is_agent_online":true,"has_ssh_access":true,"ssh_ready":true,"remote_agent_ready":true,"execution_ready":true,"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","inspection_status":"healthy","inspection_status_label":"巡检通过","inspection_reason":"最近一轮健康快照、Worker 日志和诊断包均已成功。","inspection_success_count":3,"inspection_ready":true,"inspection_status_map":{"health.snapshot":"success","logs.collect":"success","diagnostics.collect":"success"},"delivery_queue_state":"healthy","delivery_queue_label":"正常","delivery_queue_reason":"当前没有待重试回执,也没有死信记录。 最近一次队列冲刷:2026-04-19 16:29:47。","delivery_queue_pending_count":0,"delivery_queue_dead_letter_count":0,"current_release_version":"","desired_release_version":"domaincheck_release_20260418_215930","onboarding_stage_code":"acceptance_ready","onboarding_stage_label":"待接管验收","onboarding_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_action":"run_acceptance","recovery_label":"执行接管验收","recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_command_hint":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recovery_window":"acceptance","risk_level":"low","blocking_reasons":[],"warning_reasons":[],"recommended_action_code":"","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate","node_code":"mainland-controller-01"}}]},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}}},"rollout_preview":{"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"expected_batches_total":1,"expected_jobs_total":1,"gate":{"status":"preview_ready","status_label":"预览可发 Rollout","status_type":"success","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"ssh_host":"121.204.244.188","ssh_user":"root","ssh_port":22,"auth_mode":"key","deploy_channel":"stable","title":"mainland-controller-01"}],"target_node_codes":["mainland-controller-01"],"blocking_reasons":[],"warning_reasons":[],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。"],"operational_readiness":{"summary":{"nodes_total":1,"managed_nodes":1,"managed_enabled_nodes":1,"agent_online_nodes":1,"ssh_ready_nodes":1,"remote_agent_ready_nodes":1,"execution_ready_nodes":1,"inspection_healthy_nodes":1,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_noop_nodes":0},"rows":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"agent_managed":true,"is_managed":true,"is_enabled":true,"agent_state":"online_busy","agent_state_label":"已接管","agent_reason":"Agent 最近心跳 10 秒前,节点当前负载 5。","is_agent_online":true,"has_ssh_access":true,"ssh_ready":true,"remote_agent_ready":true,"execution_ready":true,"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","inspection_status":"healthy","inspection_status_label":"巡检通过","inspection_reason":"最近一轮健康快照、Worker 日志和诊断包均已成功。","inspection_success_count":3,"inspection_ready":true,"inspection_status_map":{"health.snapshot":"success","logs.collect":"success","diagnostics.collect":"success"},"delivery_queue_state":"healthy","delivery_queue_label":"正常","delivery_queue_reason":"当前没有待重试回执,也没有死信记录。 最近一次队列冲刷:2026-04-19 16:29:47。","delivery_queue_pending_count":0,"delivery_queue_dead_letter_count":0,"current_release_version":"","desired_release_version":"domaincheck_release_20260418_215930","onboarding_stage_code":"acceptance_ready","onboarding_stage_label":"待接管验收","onboarding_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_action":"run_acceptance","recovery_label":"执行接管验收","recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_command_hint":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recovery_window":"acceptance","risk_level":"low","blocking_reasons":[],"warning_reasons":[],"recommended_action_code":"","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate","node_code":"mainland-controller-01"}}]},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}},"batch_plan":{"first_batch_size":1,"batch_size":1,"batches_total":1,"first_batch_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"deploy_channel":"stable"}],"remaining_after_first_batch":0},"summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"rollout_preview"},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。"},"requires_confirmation":true,"blocked":false,"rollout_created":false,"release_deduplicated":false},"launchpad_status":{"status":"blocked","status_label":"发布门禁阻断","summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recommended_action_code":"run_acceptance","recommended_command":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recommended_execution_mode":"remote-agent","recommended_execution_mode_label":"远端 Agent","recommended_target_node_code":"mainland-controller-01","recommended_recovery_label":"执行接管验收","summary_text":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_gap_nodes_total":1,"recommended_recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recommended_onboarding_stage_code":"acceptance_ready","recommended_onboarding_stage_label":"待接管验收"}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/playbook_runs.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/playbook_runs.json new file mode 100644 index 0000000..e126a35 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/playbook_runs.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"runs":[{"run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/codex/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"created_at":"2026-04-19 01:20:34","updated_at":"2026-04-19 01:20:47","status_counts":{"success":9},"jobs_total":9,"success_jobs_total":9,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":68,"job_code":"ops-20260419012035-dfa01d","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:20:35","started_at":"2026-04-19 01:20:38","finished_at":"2026-04-19 01:20:40","updated_at":"2026-04-19 01:20:40","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":71,"job_code":"ops-20260419012035-634387","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:20:35","started_at":"2026-04-19 01:20:40","finished_at":"2026-04-19 01:20:42","updated_at":"2026-04-19 01:20:42","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":73,"job_code":"ops-20260419012036-4babe3","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:20:36","started_at":"2026-04-19 01:20:43","finished_at":"2026-04-19 01:20:47","updated_at":"2026-04-19 01:20:47","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":73,"job_code":"ops-20260419012036-4babe3","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:20:36","started_at":"2026-04-19 01:20:43","finished_at":"2026-04-19 01:20:47","updated_at":"2026-04-19 01:20:47","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":3,"steps_success":3,"steps_running":0,"steps_problem":0,"steps_terminal":3,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 01:12:57","updated_at":"2026-04-19 01:13:27","status_counts":{"success":5},"jobs_total":5,"success_jobs_total":5,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":56,"job_code":"ops-20260419011257-998c9e","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:57","started_at":"2026-04-19 01:13:03","finished_at":"2026-04-19 01:13:06","updated_at":"2026-04-19 01:13:06","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":58,"job_code":"ops-20260419011258-daf92f","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:07","finished_at":"2026-04-19 01:13:09","updated_at":"2026-04-19 01:13:09","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":60,"job_code":"ops-20260419011258-48e2cf","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:09","finished_at":"2026-04-19 01:13:12","updated_at":"2026-04-19 01:13:12","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 已完成收口。","summary_text":"收集 Node Agent 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":62,"job_code":"ops-20260419011258-591b52","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:19","updated_at":"2026-04-19 01:13:19","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":64,"job_code":"ops-20260419011258-a55cec","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:22","finished_at":"2026-04-19 01:13:27","updated_at":"2026-04-19 01:13:27","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":64,"job_code":"ops-20260419011258-a55cec","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:22","finished_at":"2026-04-19 01:13:27","updated_at":"2026-04-19 01:13:27","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":5,"steps_success":5,"steps_running":0,"steps_problem":0,"steps_terminal":5,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 01:12:57","updated_at":"2026-04-19 01:13:17","status_counts":{"success":5},"jobs_total":5,"success_jobs_total":5,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":57,"job_code":"ops-20260419011257-0818f7","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:57","started_at":"2026-04-19 01:12:59","finished_at":"2026-04-19 01:13:02","updated_at":"2026-04-19 01:13:02","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":59,"job_code":"ops-20260419011258-165f37","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:02","finished_at":"2026-04-19 01:13:04","updated_at":"2026-04-19 01:13:04","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":61,"job_code":"ops-20260419011258-b326cf","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:04","finished_at":"2026-04-19 01:13:07","updated_at":"2026-04-19 01:13:07","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 已完成收口。","summary_text":"收集 Node Agent 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":63,"job_code":"ops-20260419011258-154d4e","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:11","finished_at":"2026-04-19 01:13:14","updated_at":"2026-04-19 01:13:14","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":65,"job_code":"ops-20260419011258-9eef54","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:17","updated_at":"2026-04-19 01:13:17","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":65,"job_code":"ops-20260419011258-9eef54","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:17","updated_at":"2026-04-19 01:13:17","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":5,"steps_success":5,"steps_running":0,"steps_problem":0,"steps_terminal":5,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 01:10:09","updated_at":"2026-04-19 01:10:29","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":47,"job_code":"ops-20260419011009-cbd7d6","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:13","finished_at":"2026-04-19 01:10:15","updated_at":"2026-04-19 01:10:15","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":49,"job_code":"ops-20260419011009-98c6d6","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:16","finished_at":"2026-04-19 01:10:18","updated_at":"2026-04-19 01:10:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":51,"job_code":"ops-20260419011009-f0a298","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:18","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":53,"job_code":"ops-20260419011010-52f4b0","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:22","finished_at":"2026-04-19 01:10:26","updated_at":"2026-04-19 01:10:26","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":55,"job_code":"ops-20260419011010-9bea3c","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:27","finished_at":"2026-04-19 01:10:29","updated_at":"2026-04-19 01:10:29","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":55,"job_code":"ops-20260419011010-9bea3c","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:27","finished_at":"2026-04-19 01:10:29","updated_at":"2026-04-19 01:10:29","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":51,"job_code":"ops-20260419011009-f0a298","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:18","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 01:10:09","updated_at":"2026-04-19 01:10:27","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":46,"job_code":"ops-20260419011009-e075fe","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:09","finished_at":"2026-04-19 01:10:13","updated_at":"2026-04-19 01:10:13","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":48,"job_code":"ops-20260419011009-0a0c70","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:14","finished_at":"2026-04-19 01:10:18","updated_at":"2026-04-19 01:10:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":50,"job_code":"ops-20260419011009-c091a0","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:19","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":52,"job_code":"ops-20260419011010-1b63ee","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:21","finished_at":"2026-04-19 01:10:23","updated_at":"2026-04-19 01:10:23","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":54,"job_code":"ops-20260419011010-fb40be","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:24","finished_at":"2026-04-19 01:10:27","updated_at":"2026-04-19 01:10:27","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":54,"job_code":"ops-20260419011010-fb40be","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:24","finished_at":"2026-04-19 01:10:27","updated_at":"2026-04-19 01:10:27","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":50,"job_code":"ops-20260419011009-c091a0","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:19","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"nightly-b1","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 00:33:57","updated_at":"2026-04-19 00:49:41","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":41,"job_code":"ops-20260419003357-2be293","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:20","finished_at":"2026-04-19 00:49:22","updated_at":"2026-04-19 00:49:22","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":42,"job_code":"ops-20260419003357-1e65a2","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:23","finished_at":"2026-04-19 00:49:26","updated_at":"2026-04-19 00:49:26","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":43,"job_code":"ops-20260419003357-34e18d","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:30","updated_at":"2026-04-19 00:49:30","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":44,"job_code":"ops-20260419003357-6ad871","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:31","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":45,"job_code":"ops-20260419003357-e6ef6a","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:38","finished_at":"2026-04-19 00:49:41","updated_at":"2026-04-19 00:49:41","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":45,"job_code":"ops-20260419003357-e6ef6a","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:38","finished_at":"2026-04-19 00:49:41","updated_at":"2026-04-19 00:49:41","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":43,"job_code":"ops-20260419003357-34e18d","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:30","updated_at":"2026-04-19 00:49:30","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"nightly-a1","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 00:06:25","updated_at":"2026-04-19 00:49:34","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":36,"job_code":"ops-20260419000625-6f7db9","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:17","finished_at":"2026-04-19 00:49:20","updated_at":"2026-04-19 00:49:20","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":37,"job_code":"ops-20260419000625-0ecb78","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:20","finished_at":"2026-04-19 00:49:22","updated_at":"2026-04-19 00:49:22","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":38,"job_code":"ops-20260419000625-4de21f","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:25","finished_at":"2026-04-19 00:49:27","updated_at":"2026-04-19 00:49:27","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":39,"job_code":"ops-20260419000625-093ea8","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:31","updated_at":"2026-04-19 00:49:31","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":40,"job_code":"ops-20260419000625-28c085","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:33","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":40,"job_code":"ops-20260419000625-28c085","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:33","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":38,"job_code":"ops-20260419000625-4de21f","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:25","finished_at":"2026-04-19 00:49:27","updated_at":"2026-04-19 00:49:27","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","playbook_title":"关键现场日志","group_key":"scene","group_title":"执行现场","requested_by":"cli/ops-center/open_worker_logs_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":1,"expected_jobs_total":3,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:50:32","updated_at":"2026-04-19 00:49:18","status_counts":{"success":3},"jobs_total":3,"success_jobs_total":3,"terminal_jobs_total":3,"steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","template_key":"logs.collect","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":34,"job_code":"ops-20260418225032-8688b9","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:50:32","started_at":"2026-04-19 00:49:15","finished_at":"2026-04-19 00:49:18","updated_at":"2026-04-19 00:49:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 关键日志 已完成收口。","summary_text":"收集 Worker 关键日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}}],"latest_job":{"id":34,"job_code":"ops-20260418225032-8688b9","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:50:32","started_at":"2026-04-19 00:49:15","finished_at":"2026-04-19 00:49:18","updated_at":"2026-04-19 00:49:18","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/ops-center/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:45:53","updated_at":"2026-04-19 00:49:14","status_counts":{"success":8,"failed":1},"jobs_total":9,"success_jobs_total":8,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":22,"job_code":"ops-20260418224553-beaa1b","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:00","finished_at":"2026-04-19 00:49:05","updated_at":"2026-04-19 00:49:05","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"failed":1,"success":2},"terminal_jobs_total":3,"success_jobs_total":2,"latest_job":{"id":27,"job_code":"ops-20260418224553-600be5","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:06","finished_at":"2026-04-19 00:49:07","updated_at":"2026-04-19 00:49:07","error_message":""},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Worker 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":31,"job_code":"ops-20260418224553-897976","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:10","finished_at":"2026-04-19 00:49:14","updated_at":"2026-04-19 00:49:14","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":31,"job_code":"ops-20260418224553-897976","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:10","finished_at":"2026-04-19 00:49:14","updated_at":"2026-04-19 00:49:14","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"worker_logs","title":"收集 Worker 日志","status":"attention","reason_label":"失败","latest_job":{"id":27,"job_code":"ops-20260418224553-600be5","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:06","finished_at":"2026-04-19 00:49:07","updated_at":"2026-04-19 00:49:07","error_message":""}}],"active_steps":[],"focus_level":"danger","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","focus_summary":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":3,"steps_success":2,"steps_running":0,"steps_problem":1,"steps_terminal":3,"summary":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","playbook_title":"关键现场日志","group_key":"scene","group_title":"执行现场","requested_by":"cli/ops-center/open_worker_logs_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":1,"expected_jobs_total":3,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:45:53","updated_at":"2026-04-19 00:49:00","status_counts":{"failed":1,"success":2},"jobs_total":3,"success_jobs_total":2,"terminal_jobs_total":3,"steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","template_key":"logs.collect","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"failed":1,"success":2},"terminal_jobs_total":3,"success_jobs_total":2,"latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 关键日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Worker 关键日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}}],"latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","status":"attention","reason_label":"失败","latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""}}],"active_steps":[],"focus_level":"danger","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","focus_summary":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":1,"steps_success":0,"steps_running":0,"steps_problem":1,"steps_terminal":1,"summary":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}},{"run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","playbook_title":"生成节点接入工单","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"control-plane","execution_mode_label":"控制面","stop_on_failure":true,"step_count":1,"expected_jobs_total":1,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","status_counts":{"success":1},"jobs_total":1,"success_jobs_total":1,"terminal_jobs_total":1,"steps":[{"step_key":"bootstrap","title":"生成 Node Agent 接入工单","template_key":"node.bootstrap","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":20,"job_code":"ops-20260418194421-bc277a","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:21","started_at":"2026-04-18 19:44:21","finished_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"生成 Node Agent 接入工单 已完成收口。","summary_text":"生成 Node Agent 接入工单 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"bootstrap","focus_step_title":"生成 Node Agent 接入工单","event_key":"","node_code":""}}],"latest_job":{"id":20,"job_code":"ops-20260418194421-bc277a","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:21","started_at":"2026-04-18 19:44:21","finished_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","bootstrap_state_counts":{"total":1,"online":1,"pending_bootstrap":0,"runtime_only":0,"stale":0,"unmanaged":0,"unknown":0},"status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","summary_text":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","focus_ref":{"kind":"playbook_run","run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-d7a22054b2","playbook_key":"onboarding.bootstrap","playbook_title":"生成节点接入工单","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"control-plane","execution_mode_label":"控制面","stop_on_failure":true,"step_count":1,"expected_jobs_total":1,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-18 19:44:00","updated_at":"2026-04-18 19:44:00","status_counts":{"success":1},"jobs_total":1,"success_jobs_total":1,"terminal_jobs_total":1,"steps":[{"step_key":"bootstrap","title":"生成 Node Agent 接入工单","template_key":"node.bootstrap","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":19,"job_code":"ops-20260418194400-41b4dd","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:00","started_at":"2026-04-18 19:44:00","finished_at":"2026-04-18 19:44:00","updated_at":"2026-04-18 19:44:00","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"生成 Node Agent 接入工单 已完成收口。","summary_text":"生成 Node Agent 接入工单 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-d7a22054b2","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"bootstrap","focus_step_title":"生成 Node Agent 接入工单","event_key":"","node_code":""}}],"latest_job":{"id":19,"job_code":"ops-20260418194400-41b4dd","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:00","started_at":"2026-04-18 19:44:00","finished_at":"2026-04-18 19:44:00","updated_at":"2026-04-18 19:44:00","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","bootstrap_state_counts":{"total":1,"online":1,"pending_bootstrap":0,"runtime_only":0,"stale":0,"unmanaged":0,"unknown":0},"status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","summary_text":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","focus_ref":{"kind":"playbook_run","run_code":"pbr-d7a22054b2","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}}],"summary":{"total":12,"filtered_total":17,"unfiltered_total":17,"success":6,"running":0,"attention":6,"queued":0,"scanned_jobs":90,"available_status_counts":{"success":11,"attention":6},"available_group_counts":{"diagnostics":2,"onboarding":13,"scene":2},"filters":{"status":"","group_key":"","query":""}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/scene_overseas_control_01.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/scene_overseas_control_01.json new file mode 100644 index 0000000..e60aff0 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/scene_overseas_control_01.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"missing_sample","status_label":"缺少样本","status_type":"warning","summary":"当前还没有收到该参与节点的远端日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":0,"key_line_count":0,"full_line_count":0,"last_at":"","last_line":""},"missing_reason_code":"no_sample","missing_reason":"当前还没有收到该参与节点的远端日志样本。","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 03:30:19"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/stack.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/stack.json new file mode 100644 index 0000000..e80cbbe --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/stack.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"generated_at":"2026-04-19 03:30:18","diagnosis":{"contract_key":"ops_stack_diagnosis_contract","contract_version":"v1","registry_version":"2026-04-18","base_url":"http://127.0.0.1:8100","contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_driver_contract","ops_observability_contract","ops_playbook_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_driver_contract","title":"Ops Driver Contract","status":"active","version":"v1","summary":"冻结 overview / driver-feed / codex-brief / driver-actions / runbook sequence 的统一驾驶 contract。","primary_endpoint":"/api/v1/ops/driver-feed","schema_doc_path":"docs/schemas/ops_driver_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_driver_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/driver-feed","/api/v1/ops/codex-brief","/api/v1/ops/driver-actions/resolve","/api/v1/ops/driver-actions/execute-resolved","/api/v1/ops/codex-actions/resolve","/api/v1/ops/codex-actions/execute","/api/v1/ops/activity-stream","/api/v1/ops/runbook","/api/v1/ops/runbook/sequences/{sequence_key}/resolve","/api/v1/ops/runbook/sequences/{sequence_key}/execute","/api/v1/ops/driver-actions/preview","/api/v1/ops/driver-actions/execute"],"related_contract_keys":["release_hub_contract","ops_playbook_contract","ops_observability_contract","ops_stack_diagnosis_contract"]},{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_playbook_contract","title":"Ops Playbook Contract","status":"active","version":"v1","summary":"冻结 playbook catalog / preview / run / events 的正式编排 contract。","primary_endpoint":"/api/v1/ops/playbooks","schema_doc_path":"docs/schemas/ops_playbook_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_playbook_contract","discovery_endpoints":["/api/v1/ops/playbooks","/api/v1/ops/playbooks/preview","/api/v1/ops/playbooks/execute","/api/v1/ops/playbook-runs","/api/v1/ops/playbook-runs/{run_code}","/api/v1/ops/playbook-runs/{run_code}/events","/api/v1/ops/playbook-runs/{run_code}/rerun","/api/v1/ops/playbook-runs/{run_code}/cancel"],"related_contract_keys":["ops_job_contract","release_hub_contract","ops_driver_contract","ops_observability_contract","ops_stack_diagnosis_contract"]}]},"surface_status":"healthy","automation_status":"attention","stack_status":"attention","issue_total":2,"blocking_issue_total":0,"warning_issue_total":2,"info_issue_total":0,"missing_surfaces":[],"launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"next_step":{"action_code":"focus_playbook_run","source":"overview.primary","reason":"来自 overview.recommendation.primary_action_code","command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","focus_ref":{},"contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"recommended_actions":[{"action_code":"focus_playbook_run","source":"overview.primary","title":"先处理异常编排","reason":"来自 overview.recommendation.primary_action_code","focus_ref":{},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},{"action_code":"fix_managed_nodes","source":"release_hub.launchpad","title":"待补执行面","reason":"来自 release_launchpad.launchpad_status.recommended_action_code","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","contract_keys":["ops_stack_diagnosis_contract","ops_agent_protocol","release_hub_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_agent_protocol","release_hub_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_agent_protocol","title":"Node Agent Protocol","status":"active","version":"v1","summary":"冻结海外控制面与大陆 Node Agent 之间的认证、注册、心跳、拉任务、完成回执与事件回放 contract。","primary_endpoint":"/api/v1/ops/agent/bootstrap-plan","schema_doc_path":"docs/schemas/ops_agent_protocol.md","detail_endpoint":"/api/v1/ops/contracts/ops_agent_protocol","discovery_endpoints":["/api/v1/ops/agent/tokens","/api/v1/ops/agent/bootstrap-plan","/api/v1/ops/agent/register","/api/v1/ops/agent/heartbeat","/api/v1/ops/agent/pull","/api/v1/ops/agent/jobs/{job_id}/start","/api/v1/ops/agent/jobs/{job_id}/complete","/api/v1/ops/agent/jobs/{job_id}/events","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_observability_contract","ops_stack_diagnosis_contract"]},{"key":"release_hub_contract","title":"Release Hub Contract","status":"active","version":"v1","summary":"冻结 Release / Rollout / Launchpad / Default Gate 的对象模型与门禁口径。","primary_endpoint":"/api/v1/ops/releases/launchpad","schema_doc_path":"docs/schemas/release_hub_contract.md","detail_endpoint":"/api/v1/ops/contracts/release_hub_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/releases","/api/v1/ops/releases/latest","/api/v1/ops/releases/launchpad","/api/v1/ops/releases/{release_id}","/api/v1/ops/releases/{release_id}/rollouts","/api/v1/ops/rollouts/{rollout_id}","/api/v1/ops/rollouts/{rollout_id}/jobs","/api/v1/ops/rollouts/{rollout_id}/advance"],"related_contract_keys":["ops_job_contract","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]}]}},{"action_code":"open_playbook_run_latest_events","source":"overview.secondary","title":"先处理异常编排","reason":"来自 overview.recommendation.secondary_action_code","focus_ref":{},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},{"action_code":"open_worker_logs_participating","source":"issue:remote_log_sync_waiting_sample","title":"远端日志回传虽然已经开启,但参与检测节点的现场样本仍未完全形成。","reason":"参与检测节点 3 台,已覆盖 2 台,缺口节点 overseas-control-01。","focus_ref":{"kind":"node_scene_log","node_code":"overseas-control-01","mode":"full","limit":120,"source":"remote_log_sync_waiting_sample"},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","contract_keys":["ops_stack_diagnosis_contract","ops_observability_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_observability_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]}]}}],"operator_decision":{"lane":"ops_jobs","priority":"medium","reason_code":"playbook_runs_need_attention","title":"按总检默认下一步继续处理","summary":"来自 overview.recommendation.primary_action_code","next_focus":{},"primary_command_key":"focus_playbook_run","secondary_command_key":""},"next_actions":{"primary_command_key":"focus_playbook_run","primary_command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","secondary_command_key":"","secondary_command":""},"recommended_commands":{"stack-diagnosis":"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","stack-check":"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","doctor":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100","contracts":"bash domain-api/deploy/multi-region/drive_ops_center.sh contracts http://127.0.0.1:8100","runtime-refresh-recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100","api-restart":"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100","focus_playbook_run":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","fix_managed_nodes":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","open_playbook_run_latest_events":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","open_worker_logs_participating":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","focus-release-hub":"bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad http://127.0.0.1:8100","release-launchpad-worker":"bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad http://127.0.0.1:8100 worker"},"issues":[{"code":"remote_log_sync_waiting_sample","severity":"warning","layer":"overview.log_sync","summary":"远端日志回传虽然已经开启,但参与检测节点的现场样本仍未完全形成。","detail":"参与检测节点 3 台,已覆盖 2 台,缺口节点 overseas-control-01。","action_code":"open_worker_logs_participating","focus_ref":{"kind":"node_scene_log","node_code":"overseas-control-01","mode":"full","limit":120,"source":"remote_log_sync_waiting_sample"},"commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 overseas-control-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-controller-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"],"contract_keys":["ops_stack_diagnosis_contract","ops_observability_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_observability_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]}]}},{"code":"playbook_runs_need_attention","severity":"warning","layer":"playbook_runs","summary":"最近存在未完全收口的 playbook run,建议先确认执行序列是否停在中间步骤。","detail":"problem_runs_total=3","action_code":"","focus_ref":{},"commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100"],"contract_keys":["ops_stack_diagnosis_contract","ops_playbook_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_playbook_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_playbook_contract","title":"Ops Playbook Contract","status":"active","version":"v1","summary":"冻结 playbook catalog / preview / run / events 的正式编排 contract。","primary_endpoint":"/api/v1/ops/playbooks","schema_doc_path":"docs/schemas/ops_playbook_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_playbook_contract","discovery_endpoints":["/api/v1/ops/playbooks","/api/v1/ops/playbooks/preview","/api/v1/ops/playbooks/execute","/api/v1/ops/playbook-runs","/api/v1/ops/playbook-runs/{run_code}","/api/v1/ops/playbook-runs/{run_code}/events","/api/v1/ops/playbook-runs/{run_code}/rerun","/api/v1/ops/playbook-runs/{run_code}/cancel"],"related_contract_keys":["ops_job_contract","release_hub_contract","ops_driver_contract","ops_observability_contract","ops_stack_diagnosis_contract"]}]}}],"operator_hints":["当前总检可用,但仍有缺口;建议先按 next_step 收口,再进入更细粒度脚本。"],"quick_commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 overseas-control-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-controller-01 120 full"]},"api":{"http_status":"200","available":true,"service":"domain-api","version":"0.1.0","api_prefix":"/api/v1","worker_mode":"linux-systemd"},"surface_matrix":{"status_counts":{"total":9,"available":9,"missing":0},"items":[{"name":"api","available":true,"elapsed_ms":0},{"name":"contracts","available":true,"elapsed_ms":0},{"name":"link_snapshot","available":true,"elapsed_ms":0},{"name":"overview","available":true,"elapsed_ms":4579},{"name":"managed_nodes","available":true,"elapsed_ms":258},{"name":"release_hub","available":true,"elapsed_ms":0},{"name":"runtime_build_info","available":true,"elapsed_ms":0},{"name":"playbook_runs","available":true,"elapsed_ms":298},{"name":"activity_stream","available":true,"elapsed_ms":0}]},"contracts":{"http_status":"200","available":true,"elapsed_ms":0,"registry_version":"2026-04-18","schema_version":"v1","contracts_total":11,"contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract","ops_stack_diagnosis_contract","ops_doctor_decision_contract","ops_go_live_signoff_contract","ops_go_live_bundle_contract","ops_go_live_review_contract"],"error":""},"link_snapshot":{"http_status":"200","available":true,"elapsed_ms":0,"status":"attention","headline":"当前已有执行节点,但远端日志回传仍未覆盖全部参与节点,建议先补齐现场观测。","next_action_codes":["open_worker_logs_participating","run_inspection_participating"],"error":""},"overview":{"http_status":"200","available":true,"elapsed_ms":4579,"recommendation_key":"playbook-run-attention-pbr-9cf64aa152","recommendation_title":"先处理异常编排","primary_action_code":"focus_playbook_run","secondary_action_code":"open_playbook_run_latest_events","focus_ref":{},"execution_scene_summary":"有效执行节点 3 台;正在执行/领任务 3 台;近窗刚有吞吐 0 台;在线但未参与 0 台","execution_scene_counts":{"dispatch_active":3,"recent_only":0,"standby":0,"load_syncing":0},"log_sync":{"status":"partial_coverage","status_label":"部分覆盖","mode":"full","mode_label":"全量回传","enabled":true,"covered_participating_nodes":2,"participating_nodes_total":3,"missing_sample_node_codes":["overseas-control-01"]},"error":""},"managed_nodes":{"http_status":"200","available":true,"elapsed_ms":258,"total":3,"managed_total":3,"managed_enabled":3,"online":3,"agent_ready":3,"ssh_ready":2,"remote_access_ready":3,"participating":3,"dispatch_active":3,"standby":0,"load_syncing":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"queue_pending_records":0,"queue_dead_letter_records":0,"agent_state_counts":{"online_busy":3},"remote_access_state_counts":{"hybrid_ready":2,"agent_ready":1},"delivery_queue_state_counts":{"healthy":3},"problem_nodes":[],"first_handover_gap_node_code":"","first_handover_gap":{},"error":""},"release_hub":{"http_status":"200","available":true,"elapsed_ms":0,"latest_release_version":"domaincheck_release_20260418_013833","latest_release_status":"ready","default_rollout_gate_status":"ready","default_rollout_gate_label":"可发 Rollout","default_rollout_gate_summary":"默认 Release 与默认 Rollout 目标均已通过门禁,当前可直接进入定向或灰度 Rollout。","default_rollout_gate_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"},"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_summary":"最新发布包已经可用,但当前没有可用于智能 Rollout 的在线启用执行节点,需要先补齐托管或接入状态。","recommended_action_code":"fix_managed_nodes","recommended_execution_mode":"","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"error":""},"runtime_build_info":{"http_status":"200","available":true,"elapsed_ms":0,"source":"latest_release","package_name":"domaincheck_release_20260418_215930","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","manifest_path":"/www/wwwroot/getDomain/release/latest_release.json","repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"route_surface":{"mode":"registered","registered_paths_total":150,"surface_flags":{"ops_contracts":true,"ops_contract_detail":true,"ops_stack_diagnosis":true,"ops_node_handover":true,"ops_node_onboarding":true,"ops_node_onboarding_bootstrap_preview":true,"ops_node_onboarding_bootstrap_execute":true,"ops_node_onboarding_acceptance_preview":true,"ops_node_onboarding_acceptance_execute":true,"ops_node_onboarding_recovery_preview":true,"ops_node_onboarding_recovery_execute":true,"ops_node_scene_log":true,"ops_node_handover_bootstrap_plan":true,"ops_driver_feed":true,"ops_codex_brief":true,"ops_activity_stream":true,"ops_runbook_resolve":true,"ops_runbook_execute":true,"runtime_build_info":true},"expected_paths":{"ops_contracts":"/api/v1/ops/contracts","ops_contract_detail":"/api/v1/ops/contracts/{contract_key}","ops_stack_diagnosis":"/api/v1/ops/stack-diagnosis","ops_node_handover":"/api/v1/ops/nodes/{node_code}/handover","ops_node_onboarding":"/api/v1/ops/nodes/{node_code}/onboarding","ops_node_onboarding_bootstrap_preview":"/api/v1/ops/nodes/{node_code}/onboarding/bootstrap/preview","ops_node_onboarding_bootstrap_execute":"/api/v1/ops/nodes/{node_code}/onboarding/bootstrap/execute","ops_node_onboarding_acceptance_preview":"/api/v1/ops/nodes/{node_code}/onboarding/acceptance/preview","ops_node_onboarding_acceptance_execute":"/api/v1/ops/nodes/{node_code}/onboarding/acceptance/execute","ops_node_onboarding_recovery_preview":"/api/v1/ops/nodes/{node_code}/onboarding/recovery/preview","ops_node_onboarding_recovery_execute":"/api/v1/ops/nodes/{node_code}/onboarding/recovery/execute","ops_node_scene_log":"/api/v1/ops/nodes/{node_code}/scene-log","ops_node_handover_bootstrap_plan":"/api/v1/ops/nodes/{node_code}/handover/bootstrap-plan","ops_driver_feed":"/api/v1/ops/driver-feed","ops_codex_brief":"/api/v1/ops/codex-brief","ops_activity_stream":"/api/v1/ops/activity-stream","ops_runbook_resolve":"/api/v1/ops/runbook/sequences/{sequence_key}/resolve","ops_runbook_execute":"/api/v1/ops/runbook/sequences/{sequence_key}/execute","runtime_build_info":"/api/v1/runtime/build-info"},"missing_keys":[],"missing_paths":[],"surface_complete":true},"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"error":""},"playbook_runs":{"http_status":"200","available":true,"elapsed_ms":298,"recent_total":6,"problem_runs_total":3,"problem_runs":[{"run_code":"pbr-9cf64aa152","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-2be333d3ce","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-bfed43ea46","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}}],"error":""},"activity_stream":{"http_status":"200","available":true,"elapsed_ms":0,"recent_total":8,"status_counts":{"running":1,"success":4,"failed":3},"top_items":[{"kind":"execution_scene","status":"running","summary":"有效执行节点 3 台;正在执行/领任务 3 台;近窗刚有吞吐 0 台;在线但未参与 0 台","occurred_at":"2026-04-19 03:30:13","focus_ref":{"kind":"execution_scene","scene_key":"execution-scene:live"},"source_focus_ref":{},"ui_intent_kind":"focus_execution_scene"},{"kind":"ops_job","status":"success","summary":"已发送 Worker 控制指令: start_detection","occurred_at":"2026-04-19 03:26:51","focus_ref":{"kind":"ops_job","job_id":98,"job_code":"ops-20260419032632-aa3805","action":"runtime.start_detection","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"failed","summary":"Failed to start domaincheck-worker.service: Interactive authentication required.\nSee system logs and 'systemctl status domaincheck-worker.service' for details.","occurred_at":"2026-04-19 03:26:48","focus_ref":{"kind":"ops_job","job_id":97,"job_code":"ops-20260419032632-03e61f","action":"runtime.start_worker","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"success","summary":"待检测任务批次拉取并入库成功","occurred_at":"2026-04-19 03:26:45","focus_ref":{"kind":"ops_job","job_id":96,"job_code":"ops-20260419032632-40e23a","action":"runtime.pull_tasks","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"success","summary":"已发送 Worker 控制指令: start_detection","occurred_at":"2026-04-19 03:26:40","focus_ref":{"kind":"ops_job","job_id":100,"job_code":"ops-20260419032632-db54e0","action":"runtime.start_detection","target_node_code":"mainland-worker-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"}],"error":""}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/summary.env b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/summary.env new file mode 100644 index 0000000..7ebe18f --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_3/summary.env @@ -0,0 +1,16 @@ +CYCLES="3" +GO_LIVE_STATUS="attention" +PUBLISH_READY="false" +LOG_SYNC_STATE="partial_coverage" +LOG_SYNC_MISSING_NODE_CODES="overseas-control-01" +STACK_STATUS="attention" +ISSUE_TOTAL="2" +BLOCKING_ISSUE_TOTAL="0" +ISSUE_CODES="remote_log_sync_waiting_sample,playbook_runs_need_attention" +LAUNCHPAD_STATUS="blocked" +LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance" +PROBLEM_RUNS_TOTAL="0" +PROBLEM_RUN_CODE="" +SCENE_STATUS="missing_sample" +SCENE_LINE_COUNT="0" +GENERATED_AT="2026-04-19 03:30:12" diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/go_live.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/go_live.json new file mode 100644 index 0000000..d8f2971 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/go_live.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 03:31:26","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断,但仍有上线前关注项,建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":3,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/launchpad.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/launchpad.json new file mode 100644 index 0000000..dee1011 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/launchpad.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"channel":"stable","control_plane_base_url":"http://127.0.0.1:8100","latest_package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"latest_release":{"id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_013833/download","artifact_url_present":true,"checksum":"9da7b89bf9796098f234f388944bab959114fbe667680e5582306b32e77b489d","created_by":"cli/ops-center/latest-package","created_at":"2026-04-18 01:38:35","updated_at":"2026-04-18 01:38:35","is_preview":false,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_013833","package_size_bytes":0,"checksum":"9da7b89bf9796098f234f388944bab959114fbe667680e5582306b32e77b489d","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18 01:38:35","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_013833 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_013833 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"release_summary":{"status_counts":{"ready":2},"total":2,"active_by_channel":{},"rollout_status_counts":{},"rollouts_total":0},"runtime_build_info":{"source":"latest_release","package_name":"domaincheck_release_20260418_215930","commit_sha":"246838ae4c07","commit_ref":"main","generated_at":"2026-04-18T21:59:32","route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"repository_supports_install_command_block":true,"runtime_schema_stale":false},"latest_package_preview":{"message":"最新发布包预览已生成","preview_source":"latest_package_draft","deduplicated":false,"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"resolved_artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","resolved_sha256_download_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256"},"worker_rollout_preview":{"message":"智能 Rollout 预检未通过","preview_source":"latest_package_draft","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"smart_rollout":{"available":true,"mode":"worker","mode_label":"Worker 灰度","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","reason":"","summary":"已按 Worker 灰度 预选 1 台节点,执行路径采用 远端 Agent;优先锁定 mainland;首批 1 台,后续每批 1 台。","dominant_region":"mainland","target_node_codes":["mainland-worker-01"],"target_nodes":[{"node_code":"mainland-worker-01","region":"mainland","role":"worker","status":"busy","current_load":9,"remote_agent_ready":true,"ssh_ready":true,"execution_ready":true,"detect_participating":true,"delivery_queue_state":"healthy"}],"policy":{"auto_start":true,"auto_dispatch":false,"auto_advance":true,"auto_approve":false,"continue_on_failure":false,"continue_on_partial":false,"execution_mode":"remote-agent","first_batch_size":1,"batch_size":1,"deploy_payload":{"restart_services":["domaincheck-worker"],"health_check_urls":[],"health_check_services":["domaincheck-worker"],"health_check_timeout_seconds":10,"health_check_retries":2,"health_check_interval_seconds":2,"rollback_on_failure":true,"switch_current":true}}},"policy_preview":{"risk_level":"high","approval_required":true,"blocked":true,"blocking_reasons":["当前筛选条件未命中任何可 rollout 的目标节点。"],"approval_reasons":["该动作属于 high 风险动作,建议审批后执行。"],"warnings":[],"recommendations":[],"target_summary":{"nodes_total":0,"worker_nodes":0,"control_nodes":0,"effective_worker_nodes":0,"busy_nodes":0,"participating_nodes":0,"offline_nodes":0},"batch_plan":{"first_batch_size":0,"batch_size":0,"batches_total":0,"first_batch_nodes":[],"remaining_after_first_batch":0},"cluster_guardrails":{"online_worker_nodes":3,"dedicated_online_worker_nodes":1,"online_control_nodes":2,"busy_nodes":["mainland-worker-01","overseas-control-01"]},"release_gate":{"status":"no_targets","status_label":"无默认目标","status_type":"warning","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[],"target_node_codes":[],"blocking_reasons":[],"warning_reasons":[],"recommendations":[],"operational_readiness":{"summary":{"nodes_total":0,"managed_nodes":0,"managed_enabled_nodes":0,"agent_online_nodes":0,"ssh_ready_nodes":0,"remote_agent_ready_nodes":0,"execution_ready_nodes":0,"inspection_healthy_nodes":0,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0},"rows":[]},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}}},"rollout_preview":{"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","target_nodes_total":0,"target_node_codes":["mainland-worker-01"],"expected_batches_total":0,"expected_jobs_total":0,"gate":{"status":"no_targets","status_label":"无默认目标","status_type":"warning","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[],"target_node_codes":[],"blocking_reasons":[],"warning_reasons":[],"recommendations":[],"operational_readiness":{"summary":{"nodes_total":0,"managed_nodes":0,"managed_enabled_nodes":0,"agent_online_nodes":0,"ssh_ready_nodes":0,"remote_agent_ready_nodes":0,"execution_ready_nodes":0,"inspection_healthy_nodes":0,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0},"rows":[]},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}},"batch_plan":{"first_batch_size":0,"batch_size":0,"batches_total":0,"first_batch_nodes":[],"remaining_after_first_batch":0},"summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"rollout_preview"},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。"},"requires_confirmation":false,"blocked":true,"rollout_created":false,"release_deduplicated":false},"control_rollout_preview":{"message":"智能 Rollout 需要确认","preview_source":"latest_package_draft","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"smart_rollout":{"available":true,"mode":"control","mode_label":"Control 发布","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","reason":"","summary":"已按 Control 发布 预选 1 台节点,执行路径采用 远端 Agent;优先锁定 mainland;首批 1 台,后续每批 1 台。","dominant_region":"mainland","target_node_codes":["mainland-controller-01"],"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"remote_agent_ready":true,"ssh_ready":true,"execution_ready":true,"detect_participating":true,"delivery_queue_state":"healthy"}],"policy":{"auto_start":true,"auto_dispatch":false,"auto_advance":true,"auto_approve":false,"continue_on_failure":false,"continue_on_partial":false,"execution_mode":"remote-agent","first_batch_size":1,"batch_size":1,"deploy_payload":{"restart_services":["domaincheck-api","domaincheck-worker","domaincheck-sync-agent"],"health_check_urls":["http://127.0.0.1:8100/health"],"health_check_services":["domaincheck-api","domaincheck-worker","domaincheck-sync-agent"],"health_check_timeout_seconds":10,"health_check_retries":2,"health_check_interval_seconds":2,"rollback_on_failure":true,"switch_current":true}}},"policy_preview":{"risk_level":"high","approval_required":true,"blocked":false,"blocking_reasons":[],"approval_reasons":["本次 rollout 覆盖 control 节点,建议强制审批,并把 control 放在最后一批。","该动作属于 high 风险动作,建议审批后执行。"],"warnings":["命中了 1 个当前有负载的节点,建议优先确认任务是否可中断。","命中了 1 个正在参与检测的节点,建议优先滚动到空闲节点。"],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。","建议把 control 节点放在 rollout 末尾,并确保 worker 已先通过健康检查。","建议先等参与检测的节点降为空闲,再推进该批次。"],"target_summary":{"nodes_total":1,"worker_nodes":0,"control_nodes":1,"effective_worker_nodes":1,"busy_nodes":1,"participating_nodes":1,"offline_nodes":0},"batch_plan":{"first_batch_size":1,"batch_size":1,"batches_total":1,"first_batch_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"deploy_channel":"stable"}],"remaining_after_first_batch":0},"cluster_guardrails":{"online_worker_nodes":3,"dedicated_online_worker_nodes":1,"online_control_nodes":2,"busy_nodes":["mainland-worker-01","overseas-control-01"]},"release_gate":{"status":"preview_ready","status_label":"预览可发 Rollout","status_type":"success","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"ssh_host":"121.204.244.188","ssh_user":"root","ssh_port":22,"auth_mode":"key","deploy_channel":"stable","title":"mainland-controller-01"}],"target_node_codes":["mainland-controller-01"],"blocking_reasons":[],"warning_reasons":[],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。"],"operational_readiness":{"summary":{"nodes_total":1,"managed_nodes":1,"managed_enabled_nodes":1,"agent_online_nodes":1,"ssh_ready_nodes":1,"remote_agent_ready_nodes":1,"execution_ready_nodes":1,"inspection_healthy_nodes":1,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_noop_nodes":0},"rows":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"agent_managed":true,"is_managed":true,"is_enabled":true,"agent_state":"online_busy","agent_state_label":"已接管","agent_reason":"Agent 最近心跳 13 秒前,节点当前负载 5。","is_agent_online":true,"has_ssh_access":true,"ssh_ready":true,"remote_agent_ready":true,"execution_ready":true,"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","inspection_status":"healthy","inspection_status_label":"巡检通过","inspection_reason":"最近一轮健康快照、Worker 日志和诊断包均已成功。","inspection_success_count":3,"inspection_ready":true,"inspection_status_map":{"health.snapshot":"success","logs.collect":"success","diagnostics.collect":"success"},"delivery_queue_state":"healthy","delivery_queue_label":"正常","delivery_queue_reason":"当前没有待重试回执,也没有死信记录。 最近一次队列冲刷:2026-04-19 16:30:58。","delivery_queue_pending_count":0,"delivery_queue_dead_letter_count":0,"current_release_version":"","desired_release_version":"domaincheck_release_20260418_215930","onboarding_stage_code":"acceptance_ready","onboarding_stage_label":"待接管验收","onboarding_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_action":"run_acceptance","recovery_label":"执行接管验收","recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_command_hint":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recovery_window":"acceptance","risk_level":"low","blocking_reasons":[],"warning_reasons":[],"recommended_action_code":"","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate","node_code":"mainland-controller-01"}}]},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}}},"rollout_preview":{"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"expected_batches_total":1,"expected_jobs_total":1,"gate":{"status":"preview_ready","status_label":"预览可发 Rollout","status_type":"success","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"ssh_host":"121.204.244.188","ssh_user":"root","ssh_port":22,"auth_mode":"key","deploy_channel":"stable","title":"mainland-controller-01"}],"target_node_codes":["mainland-controller-01"],"blocking_reasons":[],"warning_reasons":[],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。"],"operational_readiness":{"summary":{"nodes_total":1,"managed_nodes":1,"managed_enabled_nodes":1,"agent_online_nodes":1,"ssh_ready_nodes":1,"remote_agent_ready_nodes":1,"execution_ready_nodes":1,"inspection_healthy_nodes":1,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_noop_nodes":0},"rows":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"agent_managed":true,"is_managed":true,"is_enabled":true,"agent_state":"online_busy","agent_state_label":"已接管","agent_reason":"Agent 最近心跳 13 秒前,节点当前负载 5。","is_agent_online":true,"has_ssh_access":true,"ssh_ready":true,"remote_agent_ready":true,"execution_ready":true,"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","inspection_status":"healthy","inspection_status_label":"巡检通过","inspection_reason":"最近一轮健康快照、Worker 日志和诊断包均已成功。","inspection_success_count":3,"inspection_ready":true,"inspection_status_map":{"health.snapshot":"success","logs.collect":"success","diagnostics.collect":"success"},"delivery_queue_state":"healthy","delivery_queue_label":"正常","delivery_queue_reason":"当前没有待重试回执,也没有死信记录。 最近一次队列冲刷:2026-04-19 16:30:58。","delivery_queue_pending_count":0,"delivery_queue_dead_letter_count":0,"current_release_version":"","desired_release_version":"domaincheck_release_20260418_215930","onboarding_stage_code":"acceptance_ready","onboarding_stage_label":"待接管验收","onboarding_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_action":"run_acceptance","recovery_label":"执行接管验收","recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_command_hint":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recovery_window":"acceptance","risk_level":"low","blocking_reasons":[],"warning_reasons":[],"recommended_action_code":"","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate","node_code":"mainland-controller-01"}}]},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}},"batch_plan":{"first_batch_size":1,"batch_size":1,"batches_total":1,"first_batch_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"deploy_channel":"stable"}],"remaining_after_first_batch":0},"summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"rollout_preview"},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。"},"requires_confirmation":true,"blocked":false,"rollout_created":false,"release_deduplicated":false},"launchpad_status":{"status":"blocked","status_label":"发布门禁阻断","summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recommended_action_code":"run_acceptance","recommended_command":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recommended_execution_mode":"remote-agent","recommended_execution_mode_label":"远端 Agent","recommended_target_node_code":"mainland-controller-01","recommended_recovery_label":"执行接管验收","summary_text":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_gap_nodes_total":1,"recommended_recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recommended_onboarding_stage_code":"acceptance_ready","recommended_onboarding_stage_label":"待接管验收"}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/playbook_runs.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/playbook_runs.json new file mode 100644 index 0000000..17b614c --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/playbook_runs.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"runs":[{"run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/night-pack/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"created_at":"2026-04-19 03:30:29","updated_at":"2026-04-19 03:30:39","status_counts":{"success":9},"jobs_total":9,"success_jobs_total":9,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":102,"job_code":"ops-20260419033030-e2eeb9","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 03:30:30","started_at":"2026-04-19 03:30:31","finished_at":"2026-04-19 03:30:32","updated_at":"2026-04-19 03:30:32","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":106,"job_code":"ops-20260419033030-2137ac","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 03:30:30","started_at":"2026-04-19 03:30:33","finished_at":"2026-04-19 03:30:35","updated_at":"2026-04-19 03:30:35","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":109,"job_code":"ops-20260419033030-545ae9","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 03:30:30","started_at":"2026-04-19 03:30:36","finished_at":"2026-04-19 03:30:39","updated_at":"2026-04-19 03:30:39","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":109,"job_code":"ops-20260419033030-545ae9","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 03:30:30","started_at":"2026-04-19 03:30:36","finished_at":"2026-04-19 03:30:39","updated_at":"2026-04-19 03:30:39","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":3,"steps_success":3,"steps_running":0,"steps_problem":0,"steps_terminal":3,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/codex/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"created_at":"2026-04-19 01:20:34","updated_at":"2026-04-19 01:20:47","status_counts":{"success":9},"jobs_total":9,"success_jobs_total":9,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":68,"job_code":"ops-20260419012035-dfa01d","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:20:35","started_at":"2026-04-19 01:20:38","finished_at":"2026-04-19 01:20:40","updated_at":"2026-04-19 01:20:40","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":71,"job_code":"ops-20260419012035-634387","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:20:35","started_at":"2026-04-19 01:20:40","finished_at":"2026-04-19 01:20:42","updated_at":"2026-04-19 01:20:42","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":73,"job_code":"ops-20260419012036-4babe3","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:20:36","started_at":"2026-04-19 01:20:43","finished_at":"2026-04-19 01:20:47","updated_at":"2026-04-19 01:20:47","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":73,"job_code":"ops-20260419012036-4babe3","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:20:36","started_at":"2026-04-19 01:20:43","finished_at":"2026-04-19 01:20:47","updated_at":"2026-04-19 01:20:47","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":3,"steps_success":3,"steps_running":0,"steps_problem":0,"steps_terminal":3,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 01:12:57","updated_at":"2026-04-19 01:13:27","status_counts":{"success":5},"jobs_total":5,"success_jobs_total":5,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":56,"job_code":"ops-20260419011257-998c9e","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:57","started_at":"2026-04-19 01:13:03","finished_at":"2026-04-19 01:13:06","updated_at":"2026-04-19 01:13:06","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":58,"job_code":"ops-20260419011258-daf92f","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:07","finished_at":"2026-04-19 01:13:09","updated_at":"2026-04-19 01:13:09","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":60,"job_code":"ops-20260419011258-48e2cf","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:09","finished_at":"2026-04-19 01:13:12","updated_at":"2026-04-19 01:13:12","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 已完成收口。","summary_text":"收集 Node Agent 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":62,"job_code":"ops-20260419011258-591b52","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:19","updated_at":"2026-04-19 01:13:19","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":64,"job_code":"ops-20260419011258-a55cec","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:22","finished_at":"2026-04-19 01:13:27","updated_at":"2026-04-19 01:13:27","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":64,"job_code":"ops-20260419011258-a55cec","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:22","finished_at":"2026-04-19 01:13:27","updated_at":"2026-04-19 01:13:27","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":5,"steps_success":5,"steps_running":0,"steps_problem":0,"steps_terminal":5,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 01:12:57","updated_at":"2026-04-19 01:13:17","status_counts":{"success":5},"jobs_total":5,"success_jobs_total":5,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":57,"job_code":"ops-20260419011257-0818f7","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:57","started_at":"2026-04-19 01:12:59","finished_at":"2026-04-19 01:13:02","updated_at":"2026-04-19 01:13:02","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":59,"job_code":"ops-20260419011258-165f37","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:02","finished_at":"2026-04-19 01:13:04","updated_at":"2026-04-19 01:13:04","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":61,"job_code":"ops-20260419011258-b326cf","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:04","finished_at":"2026-04-19 01:13:07","updated_at":"2026-04-19 01:13:07","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 已完成收口。","summary_text":"收集 Node Agent 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":63,"job_code":"ops-20260419011258-154d4e","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:11","finished_at":"2026-04-19 01:13:14","updated_at":"2026-04-19 01:13:14","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":65,"job_code":"ops-20260419011258-9eef54","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:17","updated_at":"2026-04-19 01:13:17","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":65,"job_code":"ops-20260419011258-9eef54","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:17","updated_at":"2026-04-19 01:13:17","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":5,"steps_success":5,"steps_running":0,"steps_problem":0,"steps_terminal":5,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 01:10:09","updated_at":"2026-04-19 01:10:29","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":47,"job_code":"ops-20260419011009-cbd7d6","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:13","finished_at":"2026-04-19 01:10:15","updated_at":"2026-04-19 01:10:15","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":49,"job_code":"ops-20260419011009-98c6d6","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:16","finished_at":"2026-04-19 01:10:18","updated_at":"2026-04-19 01:10:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":51,"job_code":"ops-20260419011009-f0a298","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:18","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":53,"job_code":"ops-20260419011010-52f4b0","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:22","finished_at":"2026-04-19 01:10:26","updated_at":"2026-04-19 01:10:26","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":55,"job_code":"ops-20260419011010-9bea3c","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:27","finished_at":"2026-04-19 01:10:29","updated_at":"2026-04-19 01:10:29","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":55,"job_code":"ops-20260419011010-9bea3c","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:27","finished_at":"2026-04-19 01:10:29","updated_at":"2026-04-19 01:10:29","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":51,"job_code":"ops-20260419011009-f0a298","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:18","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 01:10:09","updated_at":"2026-04-19 01:10:27","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":46,"job_code":"ops-20260419011009-e075fe","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:09","finished_at":"2026-04-19 01:10:13","updated_at":"2026-04-19 01:10:13","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":48,"job_code":"ops-20260419011009-0a0c70","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:14","finished_at":"2026-04-19 01:10:18","updated_at":"2026-04-19 01:10:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":50,"job_code":"ops-20260419011009-c091a0","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:19","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":52,"job_code":"ops-20260419011010-1b63ee","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:21","finished_at":"2026-04-19 01:10:23","updated_at":"2026-04-19 01:10:23","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":54,"job_code":"ops-20260419011010-fb40be","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:24","finished_at":"2026-04-19 01:10:27","updated_at":"2026-04-19 01:10:27","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":54,"job_code":"ops-20260419011010-fb40be","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:24","finished_at":"2026-04-19 01:10:27","updated_at":"2026-04-19 01:10:27","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":50,"job_code":"ops-20260419011009-c091a0","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:19","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"nightly-b1","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 00:33:57","updated_at":"2026-04-19 00:49:41","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":41,"job_code":"ops-20260419003357-2be293","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:20","finished_at":"2026-04-19 00:49:22","updated_at":"2026-04-19 00:49:22","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":42,"job_code":"ops-20260419003357-1e65a2","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:23","finished_at":"2026-04-19 00:49:26","updated_at":"2026-04-19 00:49:26","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":43,"job_code":"ops-20260419003357-34e18d","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:30","updated_at":"2026-04-19 00:49:30","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":44,"job_code":"ops-20260419003357-6ad871","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:31","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":45,"job_code":"ops-20260419003357-e6ef6a","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:38","finished_at":"2026-04-19 00:49:41","updated_at":"2026-04-19 00:49:41","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":45,"job_code":"ops-20260419003357-e6ef6a","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:38","finished_at":"2026-04-19 00:49:41","updated_at":"2026-04-19 00:49:41","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":43,"job_code":"ops-20260419003357-34e18d","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:30","updated_at":"2026-04-19 00:49:30","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"nightly-a1","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 00:06:25","updated_at":"2026-04-19 00:49:34","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":36,"job_code":"ops-20260419000625-6f7db9","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:17","finished_at":"2026-04-19 00:49:20","updated_at":"2026-04-19 00:49:20","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":37,"job_code":"ops-20260419000625-0ecb78","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:20","finished_at":"2026-04-19 00:49:22","updated_at":"2026-04-19 00:49:22","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":38,"job_code":"ops-20260419000625-4de21f","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:25","finished_at":"2026-04-19 00:49:27","updated_at":"2026-04-19 00:49:27","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":39,"job_code":"ops-20260419000625-093ea8","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:31","updated_at":"2026-04-19 00:49:31","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":40,"job_code":"ops-20260419000625-28c085","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:33","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":40,"job_code":"ops-20260419000625-28c085","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:33","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":38,"job_code":"ops-20260419000625-4de21f","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:25","finished_at":"2026-04-19 00:49:27","updated_at":"2026-04-19 00:49:27","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","playbook_title":"关键现场日志","group_key":"scene","group_title":"执行现场","requested_by":"cli/ops-center/open_worker_logs_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":1,"expected_jobs_total":3,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:50:32","updated_at":"2026-04-19 00:49:18","status_counts":{"success":3},"jobs_total":3,"success_jobs_total":3,"terminal_jobs_total":3,"steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","template_key":"logs.collect","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":34,"job_code":"ops-20260418225032-8688b9","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:50:32","started_at":"2026-04-19 00:49:15","finished_at":"2026-04-19 00:49:18","updated_at":"2026-04-19 00:49:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 关键日志 已完成收口。","summary_text":"收集 Worker 关键日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}}],"latest_job":{"id":34,"job_code":"ops-20260418225032-8688b9","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:50:32","started_at":"2026-04-19 00:49:15","finished_at":"2026-04-19 00:49:18","updated_at":"2026-04-19 00:49:18","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/ops-center/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:45:53","updated_at":"2026-04-19 00:49:14","status_counts":{"success":8,"failed":1},"jobs_total":9,"success_jobs_total":8,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":22,"job_code":"ops-20260418224553-beaa1b","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:00","finished_at":"2026-04-19 00:49:05","updated_at":"2026-04-19 00:49:05","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"failed":1,"success":2},"terminal_jobs_total":3,"success_jobs_total":2,"latest_job":{"id":27,"job_code":"ops-20260418224553-600be5","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:06","finished_at":"2026-04-19 00:49:07","updated_at":"2026-04-19 00:49:07","error_message":""},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Worker 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":31,"job_code":"ops-20260418224553-897976","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:10","finished_at":"2026-04-19 00:49:14","updated_at":"2026-04-19 00:49:14","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":31,"job_code":"ops-20260418224553-897976","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:10","finished_at":"2026-04-19 00:49:14","updated_at":"2026-04-19 00:49:14","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"worker_logs","title":"收集 Worker 日志","status":"attention","reason_label":"失败","latest_job":{"id":27,"job_code":"ops-20260418224553-600be5","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:06","finished_at":"2026-04-19 00:49:07","updated_at":"2026-04-19 00:49:07","error_message":""}}],"active_steps":[],"focus_level":"danger","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","focus_summary":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":3,"steps_success":2,"steps_running":0,"steps_problem":1,"steps_terminal":3,"summary":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","playbook_title":"关键现场日志","group_key":"scene","group_title":"执行现场","requested_by":"cli/ops-center/open_worker_logs_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":1,"expected_jobs_total":3,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:45:53","updated_at":"2026-04-19 00:49:00","status_counts":{"failed":1,"success":2},"jobs_total":3,"success_jobs_total":2,"terminal_jobs_total":3,"steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","template_key":"logs.collect","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"failed":1,"success":2},"terminal_jobs_total":3,"success_jobs_total":2,"latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 关键日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Worker 关键日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}}],"latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","status":"attention","reason_label":"失败","latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""}}],"active_steps":[],"focus_level":"danger","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","focus_summary":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":1,"steps_success":0,"steps_running":0,"steps_problem":1,"steps_terminal":1,"summary":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}},{"run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","playbook_title":"生成节点接入工单","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"control-plane","execution_mode_label":"控制面","stop_on_failure":true,"step_count":1,"expected_jobs_total":1,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","status_counts":{"success":1},"jobs_total":1,"success_jobs_total":1,"terminal_jobs_total":1,"steps":[{"step_key":"bootstrap","title":"生成 Node Agent 接入工单","template_key":"node.bootstrap","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":20,"job_code":"ops-20260418194421-bc277a","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:21","started_at":"2026-04-18 19:44:21","finished_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"生成 Node Agent 接入工单 已完成收口。","summary_text":"生成 Node Agent 接入工单 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"bootstrap","focus_step_title":"生成 Node Agent 接入工单","event_key":"","node_code":""}}],"latest_job":{"id":20,"job_code":"ops-20260418194421-bc277a","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:21","started_at":"2026-04-18 19:44:21","finished_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","bootstrap_state_counts":{"total":1,"online":1,"pending_bootstrap":0,"runtime_only":0,"stale":0,"unmanaged":0,"unknown":0},"status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","summary_text":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","focus_ref":{"kind":"playbook_run","run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}}],"summary":{"total":12,"filtered_total":18,"unfiltered_total":18,"success":6,"running":0,"attention":6,"queued":0,"scanned_jobs":99,"available_status_counts":{"success":12,"attention":6},"available_group_counts":{"diagnostics":3,"onboarding":13,"scene":2},"filters":{"status":"","group_key":"","query":""}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/scene_overseas_control_01.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/scene_overseas_control_01.json new file mode 100644 index 0000000..b6f8cfb --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/scene_overseas_control_01.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"full_capture","status_label":"全量观察","status_type":"success","summary":"节点当前正在参与检测,已保留 3 条现场日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":3,"key_line_count":0,"full_line_count":3,"last_at":"2026-04-19 03:30:30","last_line":"[2026-04-19 03:30:30] [overseas-control-01] 2026-04-19 01:59:40.686 | WARNING | __main__:start_detection_async:1073 - 收到启动检测指令,但检测任务已在运行,忽略重复启动"},"missing_reason_code":"","missing_reason":"","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 03:31:33"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/stack.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/stack.json new file mode 100644 index 0000000..9100897 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/stack.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"generated_at":"2026-04-19 03:31:31","diagnosis":{"contract_key":"ops_stack_diagnosis_contract","contract_version":"v1","registry_version":"2026-04-18","base_url":"http://127.0.0.1:8100","contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_driver_contract","ops_playbook_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_driver_contract","title":"Ops Driver Contract","status":"active","version":"v1","summary":"冻结 overview / driver-feed / codex-brief / driver-actions / runbook sequence 的统一驾驶 contract。","primary_endpoint":"/api/v1/ops/driver-feed","schema_doc_path":"docs/schemas/ops_driver_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_driver_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/driver-feed","/api/v1/ops/codex-brief","/api/v1/ops/driver-actions/resolve","/api/v1/ops/driver-actions/execute-resolved","/api/v1/ops/codex-actions/resolve","/api/v1/ops/codex-actions/execute","/api/v1/ops/activity-stream","/api/v1/ops/runbook","/api/v1/ops/runbook/sequences/{sequence_key}/resolve","/api/v1/ops/runbook/sequences/{sequence_key}/execute","/api/v1/ops/driver-actions/preview","/api/v1/ops/driver-actions/execute"],"related_contract_keys":["release_hub_contract","ops_playbook_contract","ops_observability_contract","ops_stack_diagnosis_contract"]},{"key":"ops_playbook_contract","title":"Ops Playbook Contract","status":"active","version":"v1","summary":"冻结 playbook catalog / preview / run / events 的正式编排 contract。","primary_endpoint":"/api/v1/ops/playbooks","schema_doc_path":"docs/schemas/ops_playbook_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_playbook_contract","discovery_endpoints":["/api/v1/ops/playbooks","/api/v1/ops/playbooks/preview","/api/v1/ops/playbooks/execute","/api/v1/ops/playbook-runs","/api/v1/ops/playbook-runs/{run_code}","/api/v1/ops/playbook-runs/{run_code}/events","/api/v1/ops/playbook-runs/{run_code}/rerun","/api/v1/ops/playbook-runs/{run_code}/cancel"],"related_contract_keys":["ops_job_contract","release_hub_contract","ops_driver_contract","ops_observability_contract","ops_stack_diagnosis_contract"]}]},"surface_status":"healthy","automation_status":"attention","stack_status":"attention","issue_total":1,"blocking_issue_total":0,"warning_issue_total":1,"info_issue_total":0,"missing_surfaces":[],"launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"next_step":{"action_code":"focus_playbook_run","source":"overview.primary","reason":"来自 overview.recommendation.primary_action_code","command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","focus_ref":{},"contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"recommended_actions":[{"action_code":"focus_playbook_run","source":"overview.primary","title":"先处理异常编排","reason":"来自 overview.recommendation.primary_action_code","focus_ref":{},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},{"action_code":"fix_managed_nodes","source":"release_hub.launchpad","title":"待补执行面","reason":"来自 release_launchpad.launchpad_status.recommended_action_code","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","contract_keys":["ops_stack_diagnosis_contract","ops_agent_protocol","release_hub_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_agent_protocol","release_hub_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_agent_protocol","title":"Node Agent Protocol","status":"active","version":"v1","summary":"冻结海外控制面与大陆 Node Agent 之间的认证、注册、心跳、拉任务、完成回执与事件回放 contract。","primary_endpoint":"/api/v1/ops/agent/bootstrap-plan","schema_doc_path":"docs/schemas/ops_agent_protocol.md","detail_endpoint":"/api/v1/ops/contracts/ops_agent_protocol","discovery_endpoints":["/api/v1/ops/agent/tokens","/api/v1/ops/agent/bootstrap-plan","/api/v1/ops/agent/register","/api/v1/ops/agent/heartbeat","/api/v1/ops/agent/pull","/api/v1/ops/agent/jobs/{job_id}/start","/api/v1/ops/agent/jobs/{job_id}/complete","/api/v1/ops/agent/jobs/{job_id}/events","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_observability_contract","ops_stack_diagnosis_contract"]},{"key":"release_hub_contract","title":"Release Hub Contract","status":"active","version":"v1","summary":"冻结 Release / Rollout / Launchpad / Default Gate 的对象模型与门禁口径。","primary_endpoint":"/api/v1/ops/releases/launchpad","schema_doc_path":"docs/schemas/release_hub_contract.md","detail_endpoint":"/api/v1/ops/contracts/release_hub_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/releases","/api/v1/ops/releases/latest","/api/v1/ops/releases/launchpad","/api/v1/ops/releases/{release_id}","/api/v1/ops/releases/{release_id}/rollouts","/api/v1/ops/rollouts/{rollout_id}","/api/v1/ops/rollouts/{rollout_id}/jobs","/api/v1/ops/rollouts/{rollout_id}/advance"],"related_contract_keys":["ops_job_contract","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]}]}},{"action_code":"open_playbook_run_latest_events","source":"overview.secondary","title":"先处理异常编排","reason":"来自 overview.recommendation.secondary_action_code","focus_ref":{},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}}],"operator_decision":{"lane":"ops_jobs","priority":"medium","reason_code":"playbook_runs_need_attention","title":"按总检默认下一步继续处理","summary":"来自 overview.recommendation.primary_action_code","next_focus":{},"primary_command_key":"focus_playbook_run","secondary_command_key":""},"next_actions":{"primary_command_key":"focus_playbook_run","primary_command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","secondary_command_key":"","secondary_command":""},"recommended_commands":{"stack-diagnosis":"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","stack-check":"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","doctor":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100","contracts":"bash domain-api/deploy/multi-region/drive_ops_center.sh contracts http://127.0.0.1:8100","runtime-refresh-recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100","api-restart":"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100","focus_playbook_run":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","fix_managed_nodes":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","open_playbook_run_latest_events":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","focus-release-hub":"bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad http://127.0.0.1:8100","release-launchpad-worker":"bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad http://127.0.0.1:8100 worker"},"issues":[{"code":"playbook_runs_need_attention","severity":"warning","layer":"playbook_runs","summary":"最近存在未完全收口的 playbook run,建议先确认执行序列是否停在中间步骤。","detail":"problem_runs_total=2","action_code":"","focus_ref":{},"commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100"],"contract_keys":["ops_stack_diagnosis_contract","ops_playbook_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_playbook_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_playbook_contract","title":"Ops Playbook Contract","status":"active","version":"v1","summary":"冻结 playbook catalog / preview / run / events 的正式编排 contract。","primary_endpoint":"/api/v1/ops/playbooks","schema_doc_path":"docs/schemas/ops_playbook_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_playbook_contract","discovery_endpoints":["/api/v1/ops/playbooks","/api/v1/ops/playbooks/preview","/api/v1/ops/playbooks/execute","/api/v1/ops/playbook-runs","/api/v1/ops/playbook-runs/{run_code}","/api/v1/ops/playbook-runs/{run_code}/events","/api/v1/ops/playbook-runs/{run_code}/rerun","/api/v1/ops/playbook-runs/{run_code}/cancel"],"related_contract_keys":["ops_job_contract","release_hub_contract","ops_driver_contract","ops_observability_contract","ops_stack_diagnosis_contract"]}]}}],"operator_hints":["当前总检可用,但仍有缺口;建议先按 next_step 收口,再进入更细粒度脚本。"],"quick_commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events"]},"api":{"http_status":"200","available":true,"service":"domain-api","version":"0.1.0","api_prefix":"/api/v1","worker_mode":"linux-systemd"},"surface_matrix":{"status_counts":{"total":9,"available":9,"missing":0},"items":[{"name":"api","available":true,"elapsed_ms":0},{"name":"contracts","available":true,"elapsed_ms":0},{"name":"link_snapshot","available":true,"elapsed_ms":0},{"name":"overview","available":true,"elapsed_ms":4643},{"name":"managed_nodes","available":true,"elapsed_ms":261},{"name":"release_hub","available":true,"elapsed_ms":0},{"name":"runtime_build_info","available":true,"elapsed_ms":0},{"name":"playbook_runs","available":true,"elapsed_ms":290},{"name":"activity_stream","available":true,"elapsed_ms":0}]},"contracts":{"http_status":"200","available":true,"elapsed_ms":0,"registry_version":"2026-04-18","schema_version":"v1","contracts_total":11,"contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract","ops_stack_diagnosis_contract","ops_doctor_decision_contract","ops_go_live_signoff_contract","ops_go_live_bundle_contract","ops_go_live_review_contract"],"error":""},"link_snapshot":{"http_status":"200","available":true,"elapsed_ms":0,"status":"attention","headline":"存在 1 个结果批次仍待推送。","next_action_codes":["focus_playbook_run","open_playbook_run_latest_events"],"error":""},"overview":{"http_status":"200","available":true,"elapsed_ms":4643,"recommendation_key":"playbook-run-attention-pbr-9cf64aa152","recommendation_title":"先处理异常编排","primary_action_code":"focus_playbook_run","secondary_action_code":"open_playbook_run_latest_events","focus_ref":{},"execution_scene_summary":"有效执行节点 3 台;正在执行/领任务 3 台;近窗刚有吞吐 0 台;在线但未参与 0 台","execution_scene_counts":{"dispatch_active":3,"recent_only":0,"standby":0,"load_syncing":0},"log_sync":{"status":"full_capture","status_label":"全量观察","mode":"full","mode_label":"全量回传","enabled":true,"covered_participating_nodes":3,"participating_nodes_total":3,"missing_sample_node_codes":[]},"error":""},"managed_nodes":{"http_status":"200","available":true,"elapsed_ms":261,"total":3,"managed_total":3,"managed_enabled":3,"online":3,"agent_ready":3,"ssh_ready":2,"remote_access_ready":3,"participating":3,"dispatch_active":3,"standby":0,"load_syncing":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"queue_pending_records":0,"queue_dead_letter_records":0,"agent_state_counts":{"online_busy":3},"remote_access_state_counts":{"hybrid_ready":2,"agent_ready":1},"delivery_queue_state_counts":{"healthy":3},"problem_nodes":[],"first_handover_gap_node_code":"","first_handover_gap":{},"error":""},"release_hub":{"http_status":"200","available":true,"elapsed_ms":0,"latest_release_version":"domaincheck_release_20260418_013833","latest_release_status":"ready","default_rollout_gate_status":"ready","default_rollout_gate_label":"可发 Rollout","default_rollout_gate_summary":"默认 Release 与默认 Rollout 目标均已通过门禁,当前可直接进入定向或灰度 Rollout。","default_rollout_gate_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"},"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_summary":"最新发布包已经可用,但当前没有可用于智能 Rollout 的在线启用执行节点,需要先补齐托管或接入状态。","recommended_action_code":"fix_managed_nodes","recommended_execution_mode":"","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"error":""},"runtime_build_info":{"http_status":"200","available":true,"elapsed_ms":0,"source":"latest_release","package_name":"domaincheck_release_20260418_215930","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","manifest_path":"/www/wwwroot/getDomain/release/latest_release.json","repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"route_surface":{"mode":"registered","registered_paths_total":150,"surface_flags":{"ops_contracts":true,"ops_contract_detail":true,"ops_stack_diagnosis":true,"ops_node_handover":true,"ops_node_onboarding":true,"ops_node_onboarding_bootstrap_preview":true,"ops_node_onboarding_bootstrap_execute":true,"ops_node_onboarding_acceptance_preview":true,"ops_node_onboarding_acceptance_execute":true,"ops_node_onboarding_recovery_preview":true,"ops_node_onboarding_recovery_execute":true,"ops_node_scene_log":true,"ops_node_handover_bootstrap_plan":true,"ops_driver_feed":true,"ops_codex_brief":true,"ops_activity_stream":true,"ops_runbook_resolve":true,"ops_runbook_execute":true,"runtime_build_info":true},"expected_paths":{"ops_contracts":"/api/v1/ops/contracts","ops_contract_detail":"/api/v1/ops/contracts/{contract_key}","ops_stack_diagnosis":"/api/v1/ops/stack-diagnosis","ops_node_handover":"/api/v1/ops/nodes/{node_code}/handover","ops_node_onboarding":"/api/v1/ops/nodes/{node_code}/onboarding","ops_node_onboarding_bootstrap_preview":"/api/v1/ops/nodes/{node_code}/onboarding/bootstrap/preview","ops_node_onboarding_bootstrap_execute":"/api/v1/ops/nodes/{node_code}/onboarding/bootstrap/execute","ops_node_onboarding_acceptance_preview":"/api/v1/ops/nodes/{node_code}/onboarding/acceptance/preview","ops_node_onboarding_acceptance_execute":"/api/v1/ops/nodes/{node_code}/onboarding/acceptance/execute","ops_node_onboarding_recovery_preview":"/api/v1/ops/nodes/{node_code}/onboarding/recovery/preview","ops_node_onboarding_recovery_execute":"/api/v1/ops/nodes/{node_code}/onboarding/recovery/execute","ops_node_scene_log":"/api/v1/ops/nodes/{node_code}/scene-log","ops_node_handover_bootstrap_plan":"/api/v1/ops/nodes/{node_code}/handover/bootstrap-plan","ops_driver_feed":"/api/v1/ops/driver-feed","ops_codex_brief":"/api/v1/ops/codex-brief","ops_activity_stream":"/api/v1/ops/activity-stream","ops_runbook_resolve":"/api/v1/ops/runbook/sequences/{sequence_key}/resolve","ops_runbook_execute":"/api/v1/ops/runbook/sequences/{sequence_key}/execute","runtime_build_info":"/api/v1/runtime/build-info"},"missing_keys":[],"missing_paths":[],"surface_complete":true},"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"error":""},"playbook_runs":{"http_status":"200","available":true,"elapsed_ms":290,"recent_total":6,"problem_runs_total":2,"problem_runs":[{"run_code":"pbr-9cf64aa152","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-2be333d3ce","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}}],"error":""},"activity_stream":{"http_status":"200","available":true,"elapsed_ms":0,"recent_total":8,"status_counts":{"running":2,"success":4,"failed":2},"top_items":[{"kind":"execution_scene","status":"running","summary":"有效执行节点 3 台;正在执行/领任务 3 台;近窗刚有吞吐 0 台;在线但未参与 0 台","occurred_at":"2026-04-19 03:31:26","focus_ref":{"kind":"execution_scene","scene_key":"execution-scene:live"},"source_focus_ref":{},"ui_intent_kind":"focus_execution_scene"},{"kind":"playbook_run","status":"success","summary":"当前整轮编排已全部收口。","occurred_at":"2026-04-19 03:30:39","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","focus_step_key":"","focus_step_title":""},"source_focus_ref":{},"ui_intent_kind":"playbook_run_detail"},{"kind":"log_sync","status":"running","summary":"参与节点已经全部被覆盖,当前处于全量日志观察模式,适合短时深度排障。","occurred_at":"2026-04-19 03:30:35","focus_ref":{"kind":"execution_scene","scene_key":"execution-scene:log-sync"},"source_focus_ref":{},"ui_intent_kind":"focus_execution_scene"},{"kind":"ops_job","status":"success","summary":"已发送 Worker 控制指令: start_detection","occurred_at":"2026-04-19 03:26:51","focus_ref":{"kind":"ops_job","job_id":98,"job_code":"ops-20260419032632-aa3805","action":"runtime.start_detection","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"failed","summary":"Failed to start domaincheck-worker.service: Interactive authentication required.\nSee system logs and 'systemctl status domaincheck-worker.service' for details.","occurred_at":"2026-04-19 03:26:48","focus_ref":{"kind":"ops_job","job_id":97,"job_code":"ops-20260419032632-03e61f","action":"runtime.start_worker","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"}],"error":""}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/summary.env b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/summary.env new file mode 100644 index 0000000..ba8b674 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_4/summary.env @@ -0,0 +1,16 @@ +CYCLES="4" +GO_LIVE_STATUS="attention" +PUBLISH_READY="false" +LOG_SYNC_STATE="full_capture" +LOG_SYNC_MISSING_NODE_CODES="" +STACK_STATUS="attention" +ISSUE_TOTAL="1" +BLOCKING_ISSUE_TOTAL="0" +ISSUE_CODES="playbook_runs_need_attention" +LAUNCHPAD_STATUS="blocked" +LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance" +PROBLEM_RUNS_TOTAL="0" +PROBLEM_RUN_CODE="" +SCENE_STATUS="full_capture" +SCENE_LINE_COUNT="3" +GENERATED_AT="2026-04-19 03:31:26" diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/go_live.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/go_live.json new file mode 100644 index 0000000..b307326 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/go_live.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 04:31:45","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断,但仍有上线前关注项,建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"partial_coverage","log_sync_mode":"full","log_sync_covered_nodes":2,"log_sync_missing_node_codes":["overseas-control-01"],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/launchpad.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/launchpad.json new file mode 100644 index 0000000..1816895 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/launchpad.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"channel":"stable","control_plane_base_url":"http://127.0.0.1:8100","latest_package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"latest_release":{"id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_013833/download","artifact_url_present":true,"checksum":"9da7b89bf9796098f234f388944bab959114fbe667680e5582306b32e77b489d","created_by":"cli/ops-center/latest-package","created_at":"2026-04-18 01:38:35","updated_at":"2026-04-18 01:38:35","is_preview":false,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_013833","package_size_bytes":0,"checksum":"9da7b89bf9796098f234f388944bab959114fbe667680e5582306b32e77b489d","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18 01:38:35","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_013833 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_013833 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"release_summary":{"status_counts":{"ready":2},"total":2,"active_by_channel":{},"rollout_status_counts":{},"rollouts_total":0},"runtime_build_info":{"source":"latest_release","package_name":"domaincheck_release_20260418_215930","commit_sha":"246838ae4c07","commit_ref":"main","generated_at":"2026-04-18T21:59:32","route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"repository_supports_install_command_block":true,"runtime_schema_stale":false},"latest_package_preview":{"message":"最新发布包预览已生成","preview_source":"latest_package_draft","deduplicated":false,"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"resolved_artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","resolved_sha256_download_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256"},"worker_rollout_preview":{"message":"智能 Rollout 预检未通过","preview_source":"latest_package_draft","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"smart_rollout":{"available":true,"mode":"worker","mode_label":"Worker 灰度","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","reason":"","summary":"已按 Worker 灰度 预选 1 台节点,执行路径采用 远端 Agent;优先锁定 mainland;首批 1 台,后续每批 1 台。","dominant_region":"mainland","target_node_codes":["mainland-worker-01"],"target_nodes":[{"node_code":"mainland-worker-01","region":"mainland","role":"worker","status":"busy","current_load":9,"remote_agent_ready":true,"ssh_ready":true,"execution_ready":true,"detect_participating":true,"delivery_queue_state":"healthy"}],"policy":{"auto_start":true,"auto_dispatch":false,"auto_advance":true,"auto_approve":false,"continue_on_failure":false,"continue_on_partial":false,"execution_mode":"remote-agent","first_batch_size":1,"batch_size":1,"deploy_payload":{"restart_services":["domaincheck-worker"],"health_check_urls":[],"health_check_services":["domaincheck-worker"],"health_check_timeout_seconds":10,"health_check_retries":2,"health_check_interval_seconds":2,"rollback_on_failure":true,"switch_current":true}}},"policy_preview":{"risk_level":"high","approval_required":true,"blocked":true,"blocking_reasons":["当前筛选条件未命中任何可 rollout 的目标节点。"],"approval_reasons":["该动作属于 high 风险动作,建议审批后执行。"],"warnings":[],"recommendations":[],"target_summary":{"nodes_total":0,"worker_nodes":0,"control_nodes":0,"effective_worker_nodes":0,"busy_nodes":0,"participating_nodes":0,"offline_nodes":0},"batch_plan":{"first_batch_size":0,"batch_size":0,"batches_total":0,"first_batch_nodes":[],"remaining_after_first_batch":0},"cluster_guardrails":{"online_worker_nodes":3,"dedicated_online_worker_nodes":1,"online_control_nodes":2,"busy_nodes":["mainland-worker-01","overseas-control-01"]},"release_gate":{"status":"no_targets","status_label":"无默认目标","status_type":"warning","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[],"target_node_codes":[],"blocking_reasons":[],"warning_reasons":[],"recommendations":[],"operational_readiness":{"summary":{"nodes_total":0,"managed_nodes":0,"managed_enabled_nodes":0,"agent_online_nodes":0,"ssh_ready_nodes":0,"remote_agent_ready_nodes":0,"execution_ready_nodes":0,"inspection_healthy_nodes":0,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0},"rows":[]},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}}},"rollout_preview":{"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","target_nodes_total":0,"target_node_codes":["mainland-worker-01"],"expected_batches_total":0,"expected_jobs_total":0,"gate":{"status":"no_targets","status_label":"无默认目标","status_type":"warning","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[],"target_node_codes":[],"blocking_reasons":[],"warning_reasons":[],"recommendations":[],"operational_readiness":{"summary":{"nodes_total":0,"managed_nodes":0,"managed_enabled_nodes":0,"agent_online_nodes":0,"ssh_ready_nodes":0,"remote_agent_ready_nodes":0,"execution_ready_nodes":0,"inspection_healthy_nodes":0,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0},"rows":[]},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}},"batch_plan":{"first_batch_size":0,"batch_size":0,"batches_total":0,"first_batch_nodes":[],"remaining_after_first_batch":0},"summary":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"rollout_preview"},"summary_text":"当前没有在线且可视为有效执行面的预览 Rollout 目标节点。"},"requires_confirmation":false,"blocked":true,"rollout_created":false,"release_deduplicated":false},"control_rollout_preview":{"message":"智能 Rollout 需要确认","preview_source":"latest_package_draft","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"package":{"available":true,"package_name":"domaincheck_release_20260418_215930","package_type":"tar.gz","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","smoke_test_ok":true,"smoke_test_message":"all checks passed (10/10)","final_release_ok":true,"final_release_report_available":true,"final_release_report_ok":true,"final_release_report_matches_latest":true,"final_release_report_package_name":"domaincheck_release_20260418_215930","final_release_report_stale_reason":"","final_release_gate_decision":"ready","final_release_gate_blocked_reasons":[],"final_release_gate_verify_ok":true,"final_release_gate_smoke_test_ok":true,"download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","sha256_download_path":"/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/sha256","reason":"","manifest":{"artifact_kind":"tar.gz","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"2026-04-18T21:59:32","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""}},"smart_rollout":{"available":true,"mode":"control","mode_label":"Control 发布","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","reason":"","summary":"已按 Control 发布 预选 1 台节点,执行路径采用 远端 Agent;优先锁定 mainland;首批 1 台,后续每批 1 台。","dominant_region":"mainland","target_node_codes":["mainland-controller-01"],"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"remote_agent_ready":true,"ssh_ready":true,"execution_ready":true,"detect_participating":true,"delivery_queue_state":"healthy"}],"policy":{"auto_start":true,"auto_dispatch":false,"auto_advance":true,"auto_approve":false,"continue_on_failure":false,"continue_on_partial":false,"execution_mode":"remote-agent","first_batch_size":1,"batch_size":1,"deploy_payload":{"restart_services":["domaincheck-api","domaincheck-worker","domaincheck-sync-agent"],"health_check_urls":["http://127.0.0.1:8100/health"],"health_check_services":["domaincheck-api","domaincheck-worker","domaincheck-sync-agent"],"health_check_timeout_seconds":10,"health_check_retries":2,"health_check_interval_seconds":2,"rollback_on_failure":true,"switch_current":true}}},"policy_preview":{"risk_level":"high","approval_required":true,"blocked":false,"blocking_reasons":[],"approval_reasons":["本次 rollout 覆盖 control 节点,建议强制审批,并把 control 放在最后一批。","该动作属于 high 风险动作,建议审批后执行。"],"warnings":["命中了 1 个当前有负载的节点,建议优先确认任务是否可中断。","命中了 1 个正在参与检测的节点,建议优先滚动到空闲节点。"],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。","建议把 control 节点放在 rollout 末尾,并确保 worker 已先通过健康检查。","建议先等参与检测的节点降为空闲,再推进该批次。"],"target_summary":{"nodes_total":1,"worker_nodes":0,"control_nodes":1,"effective_worker_nodes":1,"busy_nodes":1,"participating_nodes":1,"offline_nodes":0},"batch_plan":{"first_batch_size":1,"batch_size":1,"batches_total":1,"first_batch_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"deploy_channel":"stable"}],"remaining_after_first_batch":0},"cluster_guardrails":{"online_worker_nodes":3,"dedicated_online_worker_nodes":1,"online_control_nodes":2,"busy_nodes":["mainland-worker-01","overseas-control-01"]},"release_gate":{"status":"preview_ready","status_label":"预览可发 Rollout","status_type":"success","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"ssh_host":"121.204.244.188","ssh_user":"root","ssh_port":22,"auth_mode":"key","deploy_channel":"stable","title":"mainland-controller-01"}],"target_node_codes":["mainland-controller-01"],"blocking_reasons":[],"warning_reasons":[],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。"],"operational_readiness":{"summary":{"nodes_total":1,"managed_nodes":1,"managed_enabled_nodes":1,"agent_online_nodes":1,"ssh_ready_nodes":1,"remote_agent_ready_nodes":1,"execution_ready_nodes":1,"inspection_healthy_nodes":1,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_noop_nodes":0},"rows":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"agent_managed":true,"is_managed":true,"is_enabled":true,"agent_state":"online_busy","agent_state_label":"已接管","agent_reason":"Agent 最近心跳 17 秒前,节点当前负载 5。","is_agent_online":true,"has_ssh_access":true,"ssh_ready":true,"remote_agent_ready":true,"execution_ready":true,"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","inspection_status":"healthy","inspection_status_label":"巡检通过","inspection_reason":"最近一轮健康快照、Worker 日志和诊断包均已成功。","inspection_success_count":3,"inspection_ready":true,"inspection_status_map":{"health.snapshot":"success","logs.collect":"success","diagnostics.collect":"success"},"delivery_queue_state":"healthy","delivery_queue_label":"正常","delivery_queue_reason":"当前没有待重试回执,也没有死信记录。 最近一次队列冲刷:2026-04-19 17:31:14。","delivery_queue_pending_count":0,"delivery_queue_dead_letter_count":0,"current_release_version":"","desired_release_version":"domaincheck_release_20260418_215930","onboarding_stage_code":"acceptance_ready","onboarding_stage_label":"待接管验收","onboarding_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_action":"run_acceptance","recovery_label":"执行接管验收","recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_command_hint":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recovery_window":"acceptance","risk_level":"low","blocking_reasons":[],"warning_reasons":[],"recommended_action_code":"","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate","node_code":"mainland-controller-01"}}]},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}}},"rollout_preview":{"release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","commit_sha":"246838ae4c07","status":"ready","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","artifact_url_present":true,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","created_by":"api/release-launchpad","created_at":"","updated_at":"","is_preview":true,"manifest":{"artifact_kind":"","package_name":"domaincheck_release_20260418_215930","package_size_bytes":0,"checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","build_id":"","commit_sha":"246838ae4c07","built_at":"","included_services":[],"required_env_keys":[],"health_checks":[],"rollback_hint":""},"status_label":"已就绪","summary":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","summary_text":"domaincheck_release_20260418_215930 已就绪,制品与校验信息齐备,可进入 Rollout。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_detail"}},"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"expected_batches_total":1,"expected_jobs_total":1,"gate":{"status":"preview_ready","status_label":"预览可发 Rollout","status_type":"success","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","release":{"id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","status":"ready","status_label":"已就绪","artifact_url":"http://127.0.0.1:8100/api/v1/ops/releases/packages/domaincheck_release_20260418_215930/download","is_preview":true},"is_preview_release":true,"target_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"ssh_host":"121.204.244.188","ssh_user":"root","ssh_port":22,"auth_mode":"key","deploy_channel":"stable","title":"mainland-controller-01"}],"target_node_codes":["mainland-controller-01"],"blocking_reasons":[],"warning_reasons":[],"recommendations":["有 1 台目标节点已进入验收窗口,建议先执行 onboarding.acceptance。"],"operational_readiness":{"summary":{"nodes_total":1,"managed_nodes":1,"managed_enabled_nodes":1,"agent_online_nodes":1,"ssh_ready_nodes":1,"remote_agent_ready_nodes":1,"execution_ready_nodes":1,"inspection_healthy_nodes":1,"inspection_running_nodes":0,"inspection_attention_nodes":0,"inspection_missing_nodes":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_noop_nodes":0},"rows":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"agent_managed":true,"is_managed":true,"is_enabled":true,"agent_state":"online_busy","agent_state_label":"已接管","agent_reason":"Agent 最近心跳 17 秒前,节点当前负载 5。","is_agent_online":true,"has_ssh_access":true,"ssh_ready":true,"remote_agent_ready":true,"execution_ready":true,"execution_mode":"remote-agent","execution_mode_label":"远端 Agent","inspection_status":"healthy","inspection_status_label":"巡检通过","inspection_reason":"最近一轮健康快照、Worker 日志和诊断包均已成功。","inspection_success_count":3,"inspection_ready":true,"inspection_status_map":{"health.snapshot":"success","logs.collect":"success","diagnostics.collect":"success"},"delivery_queue_state":"healthy","delivery_queue_label":"正常","delivery_queue_reason":"当前没有待重试回执,也没有死信记录。 最近一次队列冲刷:2026-04-19 17:31:14。","delivery_queue_pending_count":0,"delivery_queue_dead_letter_count":0,"current_release_version":"","desired_release_version":"domaincheck_release_20260418_215930","onboarding_stage_code":"acceptance_ready","onboarding_stage_label":"待接管验收","onboarding_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_action":"run_acceptance","recovery_label":"执行接管验收","recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recovery_command_hint":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recovery_window":"acceptance","risk_level":"low","blocking_reasons":[],"warning_reasons":[],"recommended_action_code":"","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate","node_code":"mainland-controller-01"}}]},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"}},"batch_plan":{"first_batch_size":1,"batch_size":1,"batches_total":1,"first_batch_nodes":[{"node_code":"mainland-controller-01","region":"mainland","role":"control","status":"online","current_load":5,"is_effective_worker":true,"detect_participating":true,"is_enabled":true,"deploy_channel":"stable"}],"remaining_after_first_batch":0},"summary":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。","focus_ref":{"kind":"release_hub","release_id":0,"release_version":"domaincheck_release_20260418_215930","channel":"stable","rollout_id":0,"rollout_code":"","section":"rollout_preview"},"summary_text":"当前基于最新发布包推导出的 Release 预览与目标节点均已通过门禁,已具备进入定向或灰度 Rollout 的条件。"},"requires_confirmation":true,"blocked":false,"rollout_created":false,"release_deduplicated":false},"launchpad_status":{"status":"blocked","status_label":"发布门禁阻断","summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recommended_action_code":"run_acceptance","recommended_command":"bash domain-api/deploy/multi-region/drive_ops_center.sh node-acceptance-run http://127.0.0.1:8100 mainland-controller-01 cli","recommended_execution_mode":"remote-agent","recommended_execution_mode_label":"远端 Agent","recommended_target_node_code":"mainland-controller-01","recommended_recovery_label":"执行接管验收","summary_text":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"onboarding_bootstrap_pending_nodes":0,"onboarding_acceptance_ready_nodes":1,"onboarding_gap_nodes_total":1,"recommended_recovery_summary":"Node Agent 已在线,可以直接跑 onboarding.acceptance 做标准接管验收。","recommended_onboarding_stage_code":"acceptance_ready","recommended_onboarding_stage_label":"待接管验收"}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/playbook_runs.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/playbook_runs.json new file mode 100644 index 0000000..d790cd6 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/playbook_runs.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"runs":[{"run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/night-pack/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"created_at":"2026-04-19 03:30:29","updated_at":"2026-04-19 03:30:39","status_counts":{"success":9},"jobs_total":9,"success_jobs_total":9,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":102,"job_code":"ops-20260419033030-e2eeb9","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 03:30:30","started_at":"2026-04-19 03:30:31","finished_at":"2026-04-19 03:30:32","updated_at":"2026-04-19 03:30:32","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":106,"job_code":"ops-20260419033030-2137ac","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 03:30:30","started_at":"2026-04-19 03:30:33","finished_at":"2026-04-19 03:30:35","updated_at":"2026-04-19 03:30:35","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":109,"job_code":"ops-20260419033030-545ae9","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 03:30:30","started_at":"2026-04-19 03:30:36","finished_at":"2026-04-19 03:30:39","updated_at":"2026-04-19 03:30:39","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":109,"job_code":"ops-20260419033030-545ae9","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 03:30:30","started_at":"2026-04-19 03:30:36","finished_at":"2026-04-19 03:30:39","updated_at":"2026-04-19 03:30:39","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":3,"steps_success":3,"steps_running":0,"steps_problem":0,"steps_terminal":3,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-1a288c1c2d","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/codex/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"created_at":"2026-04-19 01:20:34","updated_at":"2026-04-19 01:20:47","status_counts":{"success":9},"jobs_total":9,"success_jobs_total":9,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":68,"job_code":"ops-20260419012035-dfa01d","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:20:35","started_at":"2026-04-19 01:20:38","finished_at":"2026-04-19 01:20:40","updated_at":"2026-04-19 01:20:40","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":71,"job_code":"ops-20260419012035-634387","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:20:35","started_at":"2026-04-19 01:20:40","finished_at":"2026-04-19 01:20:42","updated_at":"2026-04-19 01:20:42","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":73,"job_code":"ops-20260419012036-4babe3","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:20:36","started_at":"2026-04-19 01:20:43","finished_at":"2026-04-19 01:20:47","updated_at":"2026-04-19 01:20:47","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":73,"job_code":"ops-20260419012036-4babe3","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:20:36","started_at":"2026-04-19 01:20:43","finished_at":"2026-04-19 01:20:47","updated_at":"2026-04-19 01:20:47","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":3,"steps_success":3,"steps_running":0,"steps_problem":0,"steps_terminal":3,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-df533c6f4a","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 01:12:57","updated_at":"2026-04-19 01:13:27","status_counts":{"success":5},"jobs_total":5,"success_jobs_total":5,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":56,"job_code":"ops-20260419011257-998c9e","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:57","started_at":"2026-04-19 01:13:03","finished_at":"2026-04-19 01:13:06","updated_at":"2026-04-19 01:13:06","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":58,"job_code":"ops-20260419011258-daf92f","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:07","finished_at":"2026-04-19 01:13:09","updated_at":"2026-04-19 01:13:09","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":60,"job_code":"ops-20260419011258-48e2cf","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:09","finished_at":"2026-04-19 01:13:12","updated_at":"2026-04-19 01:13:12","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 已完成收口。","summary_text":"收集 Node Agent 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":62,"job_code":"ops-20260419011258-591b52","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:19","updated_at":"2026-04-19 01:13:19","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":64,"job_code":"ops-20260419011258-a55cec","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:22","finished_at":"2026-04-19 01:13:27","updated_at":"2026-04-19 01:13:27","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":64,"job_code":"ops-20260419011258-a55cec","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:22","finished_at":"2026-04-19 01:13:27","updated_at":"2026-04-19 01:13:27","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":5,"steps_success":5,"steps_running":0,"steps_problem":0,"steps_terminal":5,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9ce5c85f17","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 01:12:57","updated_at":"2026-04-19 01:13:17","status_counts":{"success":5},"jobs_total":5,"success_jobs_total":5,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":57,"job_code":"ops-20260419011257-0818f7","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:57","started_at":"2026-04-19 01:12:59","finished_at":"2026-04-19 01:13:02","updated_at":"2026-04-19 01:13:02","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":59,"job_code":"ops-20260419011258-165f37","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:02","finished_at":"2026-04-19 01:13:04","updated_at":"2026-04-19 01:13:04","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":61,"job_code":"ops-20260419011258-b326cf","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:04","finished_at":"2026-04-19 01:13:07","updated_at":"2026-04-19 01:13:07","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 已完成收口。","summary_text":"收集 Node Agent 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":63,"job_code":"ops-20260419011258-154d4e","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:11","finished_at":"2026-04-19 01:13:14","updated_at":"2026-04-19 01:13:14","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":65,"job_code":"ops-20260419011258-9eef54","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:17","updated_at":"2026-04-19 01:13:17","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":65,"job_code":"ops-20260419011258-9eef54","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:12:58","started_at":"2026-04-19 01:13:14","finished_at":"2026-04-19 01:13:17","updated_at":"2026-04-19 01:13:17","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":5,"steps_success":5,"steps_running":0,"steps_problem":0,"steps_terminal":5,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-389abd618c","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 01:10:09","updated_at":"2026-04-19 01:10:29","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":47,"job_code":"ops-20260419011009-cbd7d6","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:13","finished_at":"2026-04-19 01:10:15","updated_at":"2026-04-19 01:10:15","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":49,"job_code":"ops-20260419011009-98c6d6","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:16","finished_at":"2026-04-19 01:10:18","updated_at":"2026-04-19 01:10:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":51,"job_code":"ops-20260419011009-f0a298","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:18","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":53,"job_code":"ops-20260419011010-52f4b0","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:22","finished_at":"2026-04-19 01:10:26","updated_at":"2026-04-19 01:10:26","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":55,"job_code":"ops-20260419011010-9bea3c","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:27","finished_at":"2026-04-19 01:10:29","updated_at":"2026-04-19 01:10:29","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":55,"job_code":"ops-20260419011010-9bea3c","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:27","finished_at":"2026-04-19 01:10:29","updated_at":"2026-04-19 01:10:29","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":51,"job_code":"ops-20260419011009-f0a298","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:18","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 01:10:09","updated_at":"2026-04-19 01:10:27","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":46,"job_code":"ops-20260419011009-e075fe","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:09","finished_at":"2026-04-19 01:10:13","updated_at":"2026-04-19 01:10:13","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":48,"job_code":"ops-20260419011009-0a0c70","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:14","finished_at":"2026-04-19 01:10:18","updated_at":"2026-04-19 01:10:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":50,"job_code":"ops-20260419011009-c091a0","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:19","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":52,"job_code":"ops-20260419011010-1b63ee","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:21","finished_at":"2026-04-19 01:10:23","updated_at":"2026-04-19 01:10:23","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":54,"job_code":"ops-20260419011010-fb40be","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:24","finished_at":"2026-04-19 01:10:27","updated_at":"2026-04-19 01:10:27","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":54,"job_code":"ops-20260419011010-fb40be","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:10","started_at":"2026-04-19 01:10:24","finished_at":"2026-04-19 01:10:27","updated_at":"2026-04-19 01:10:27","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":50,"job_code":"ops-20260419011009-c091a0","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 01:10:09","started_at":"2026-04-19 01:10:19","finished_at":"2026-04-19 01:10:21","updated_at":"2026-04-19 01:10:21","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"nightly-b1","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-19 00:33:57","updated_at":"2026-04-19 00:49:41","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":41,"job_code":"ops-20260419003357-2be293","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:20","finished_at":"2026-04-19 00:49:22","updated_at":"2026-04-19 00:49:22","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":42,"job_code":"ops-20260419003357-1e65a2","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:23","finished_at":"2026-04-19 00:49:26","updated_at":"2026-04-19 00:49:26","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":43,"job_code":"ops-20260419003357-34e18d","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:30","updated_at":"2026-04-19 00:49:30","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":44,"job_code":"ops-20260419003357-6ad871","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:31","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":45,"job_code":"ops-20260419003357-e6ef6a","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:38","finished_at":"2026-04-19 00:49:41","updated_at":"2026-04-19 00:49:41","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":45,"job_code":"ops-20260419003357-e6ef6a","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:38","finished_at":"2026-04-19 00:49:41","updated_at":"2026-04-19 00:49:41","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":43,"job_code":"ops-20260419003357-34e18d","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-controller-01","created_at":"2026-04-19 00:33:57","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:30","updated_at":"2026-04-19 00:49:30","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-bfed43ea46","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","playbook_title":"接管后验收","group_key":"onboarding","group_title":"接管验收","requested_by":"nightly-a1","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":5,"expected_jobs_total":5,"target_nodes_total":1,"target_node_codes":["mainland-worker-01"],"created_at":"2026-04-19 00:06:25","updated_at":"2026-04-19 00:49:34","status_counts":{"success":4,"failed":1},"jobs_total":5,"success_jobs_total":4,"terminal_jobs_total":5,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":36,"job_code":"ops-20260419000625-6f7db9","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:17","finished_at":"2026-04-19 00:49:20","updated_at":"2026-04-19 00:49:20","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"node_agent_status","title":"采集 Node Agent 状态","template_key":"service.status","order":2,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":37,"job_code":"ops-20260419000625-0ecb78","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:20","finished_at":"2026-04-19 00:49:22","updated_at":"2026-04-19 00:49:22","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Node Agent 状态 已完成收口。","summary_text":"采集 Node Agent 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_status","focus_step_title":"采集 Node Agent 状态","event_key":"","node_code":""}},{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","template_key":"logs.collect","order":3,"jobs_total":1,"nodes_total":1,"status_counts":{"failed":1},"terminal_jobs_total":1,"success_jobs_total":0,"latest_job":{"id":38,"job_code":"ops-20260419000625-4de21f","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:25","finished_at":"2026-04-19 00:49:27","updated_at":"2026-04-19 00:49:27","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"step_key":"worker_status","title":"采集 Worker 状态","template_key":"service.status","order":4,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":39,"job_code":"ops-20260419000625-093ea8","action":"service.status","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:28","finished_at":"2026-04-19 00:49:31","updated_at":"2026-04-19 00:49:31","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"采集 Worker 状态 已完成收口。","summary_text":"采集 Worker 状态 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_status","focus_step_title":"采集 Worker 状态","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":5,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":40,"job_code":"ops-20260419000625-28c085","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:33","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-worker-01"],"summary":"收集 Worker 日志 已完成收口。","summary_text":"收集 Worker 日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}}],"latest_job":{"id":40,"job_code":"ops-20260419000625-28c085","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:33","finished_at":"2026-04-19 00:49:34","updated_at":"2026-04-19 00:49:34","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"node_agent_logs","title":"收集 Node Agent 日志","status":"attention","reason_label":"失败","latest_job":{"id":38,"job_code":"ops-20260419000625-4de21f","action":"logs.collect","status":"failed","status_label":"failed","target_node_code":"mainland-worker-01","created_at":"2026-04-19 00:06:25","started_at":"2026-04-19 00:49:25","finished_at":"2026-04-19 00:49:27","updated_at":"2026-04-19 00:49:27","error_message":"Hint: You are currently not seeing messages from other users and the system.\n Users in groups 'adm', 'systemd-journal', 'wheel' can see all messages.\n Pass -q to turn off this notice.\nNo journal files were opened due to insufficient permissions."}}],"active_steps":[],"focus_level":"danger","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":5,"steps_success":4,"steps_running":0,"steps_problem":1,"steps_terminal":5,"summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-137e8b258b","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","playbook_title":"关键现场日志","group_key":"scene","group_title":"执行现场","requested_by":"cli/ops-center/open_worker_logs_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":1,"expected_jobs_total":3,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:50:32","updated_at":"2026-04-19 00:49:18","status_counts":{"success":3},"jobs_total":3,"success_jobs_total":3,"terminal_jobs_total":3,"steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","template_key":"logs.collect","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":34,"job_code":"ops-20260418225032-8688b9","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:50:32","started_at":"2026-04-19 00:49:15","finished_at":"2026-04-19 00:49:18","updated_at":"2026-04-19 00:49:18","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 关键日志 已完成收口。","summary_text":"收集 Worker 关键日志 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}}],"latest_job":{"id":34,"job_code":"ops-20260418225032-8688b9","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:50:32","started_at":"2026-04-19 00:49:15","finished_at":"2026-04-19 00:49:18","updated_at":"2026-04-19 00:49:18","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"当前整轮编排已全部收口。","status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"当前整轮编排已全部收口。","summary_text":"当前整轮编排已全部收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-4df9934383","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}},{"run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","playbook_title":"标准巡检","group_key":"diagnostics","group_title":"标准巡检","requested_by":"cli/ops-center/run_inspection_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":3,"expected_jobs_total":9,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:45:53","updated_at":"2026-04-19 00:49:14","status_counts":{"success":8,"failed":1},"jobs_total":9,"success_jobs_total":8,"terminal_jobs_total":9,"steps":[{"step_key":"health","title":"采集健康快照","template_key":"health.snapshot","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":22,"job_code":"ops-20260418224553-beaa1b","action":"health.snapshot","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:00","finished_at":"2026-04-19 00:49:05","updated_at":"2026-04-19 00:49:05","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"采集健康快照 已完成收口。","summary_text":"采集健康快照 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"health","focus_step_title":"采集健康快照","event_key":"","node_code":""}},{"step_key":"worker_logs","title":"收集 Worker 日志","template_key":"logs.collect","order":2,"jobs_total":3,"nodes_total":3,"status_counts":{"failed":1,"success":2},"terminal_jobs_total":3,"success_jobs_total":2,"latest_job":{"id":27,"job_code":"ops-20260418224553-600be5","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:06","finished_at":"2026-04-19 00:49:07","updated_at":"2026-04-19 00:49:07","error_message":""},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Worker 日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"step_key":"diagnostics","title":"收集诊断包","template_key":"diagnostics.collect","order":3,"jobs_total":3,"nodes_total":3,"status_counts":{"success":3},"terminal_jobs_total":3,"success_jobs_total":3,"latest_job":{"id":31,"job_code":"ops-20260418224553-897976","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:10","finished_at":"2026-04-19 00:49:14","updated_at":"2026-04-19 00:49:14","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集诊断包 已完成收口。","summary_text":"收集诊断包 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"diagnostics","focus_step_title":"收集诊断包","event_key":"","node_code":""}}],"latest_job":{"id":31,"job_code":"ops-20260418224553-897976","action":"diagnostics.collect","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:10","finished_at":"2026-04-19 00:49:14","updated_at":"2026-04-19 00:49:14","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"worker_logs","title":"收集 Worker 日志","status":"attention","reason_label":"失败","latest_job":{"id":27,"job_code":"ops-20260418224553-600be5","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:49:06","finished_at":"2026-04-19 00:49:07","updated_at":"2026-04-19 00:49:07","error_message":""}}],"active_steps":[],"focus_level":"danger","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","focus_summary":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":3,"steps_success":2,"steps_running":0,"steps_problem":1,"steps_terminal":3,"summary":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Worker 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-de71dbbda4","playbook_key":"inspection.standard","group_key":"diagnostics","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 日志","event_key":"","node_code":""}},{"run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","playbook_title":"关键现场日志","group_key":"scene","group_title":"执行现场","requested_by":"cli/ops-center/open_worker_logs_participating","execution_mode":"remote-agent","execution_mode_label":"远端 Agent","stop_on_failure":true,"step_count":1,"expected_jobs_total":3,"target_nodes_total":3,"target_node_codes":["overseas-control-01","mainland-controller-01","mainland-worker-01"],"created_at":"2026-04-18 22:45:53","updated_at":"2026-04-19 00:49:00","status_counts":{"failed":1,"success":2},"jobs_total":3,"success_jobs_total":2,"terminal_jobs_total":3,"steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","template_key":"logs.collect","order":1,"jobs_total":3,"nodes_total":3,"status_counts":{"failed":1,"success":2},"terminal_jobs_total":3,"success_jobs_total":2,"latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""},"status":"attention","status_label":"待关注","completion_percent":100.0,"target_node_codes":["mainland-controller-01","mainland-worker-01","overseas-control-01"],"summary":"收集 Worker 关键日志 当前出现失败,建议先查看该步骤事件流。","summary_text":"收集 Worker 关键日志 当前出现失败,建议先查看该步骤事件流。","focus_ref":{"kind":"playbook_run","run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}}],"latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""},"status":"attention","completion_percent":100.0,"problem_steps":[{"step_key":"worker_logs","title":"收集 Worker 关键日志","status":"attention","reason_label":"失败","latest_job":{"id":21,"job_code":"ops-20260418224553-32c368","action":"logs.collect","status":"success","status_label":"成功","target_node_code":"mainland-worker-01","created_at":"2026-04-18 22:45:53","started_at":"2026-04-19 00:48:56","finished_at":"2026-04-19 00:49:00","updated_at":"2026-04-19 00:49:00","error_message":""}}],"active_steps":[],"focus_level":"danger","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","focus_summary":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","status_label":"待关注","steps_total":1,"steps_success":0,"steps_running":0,"steps_problem":1,"steps_terminal":1,"summary":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","summary_text":"收集 Worker 关键日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-8c7cf4e19e","playbook_key":"scene.logs.key","group_key":"scene","focus_step_key":"worker_logs","focus_step_title":"收集 Worker 关键日志","event_key":"","node_code":""}},{"run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","playbook_title":"生成节点接入工单","group_key":"onboarding","group_title":"接管验收","requested_by":"cli","execution_mode":"control-plane","execution_mode_label":"控制面","stop_on_failure":true,"step_count":1,"expected_jobs_total":1,"target_nodes_total":1,"target_node_codes":["mainland-controller-01"],"created_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","status_counts":{"success":1},"jobs_total":1,"success_jobs_total":1,"terminal_jobs_total":1,"steps":[{"step_key":"bootstrap","title":"生成 Node Agent 接入工单","template_key":"node.bootstrap","order":1,"jobs_total":1,"nodes_total":1,"status_counts":{"success":1},"terminal_jobs_total":1,"success_jobs_total":1,"latest_job":{"id":20,"job_code":"ops-20260418194421-bc277a","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:21","started_at":"2026-04-18 19:44:21","finished_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","error_message":""},"status":"success","status_label":"成功","completion_percent":100.0,"target_node_codes":["mainland-controller-01"],"summary":"生成 Node Agent 接入工单 已完成收口。","summary_text":"生成 Node Agent 接入工单 已完成收口。","focus_ref":{"kind":"playbook_run","run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"bootstrap","focus_step_title":"生成 Node Agent 接入工单","event_key":"","node_code":""}}],"latest_job":{"id":20,"job_code":"ops-20260418194421-bc277a","action":"node.bootstrap","status":"success","status_label":"成功","target_node_code":"mainland-controller-01","created_at":"2026-04-18 19:44:21","started_at":"2026-04-18 19:44:21","finished_at":"2026-04-18 19:44:21","updated_at":"2026-04-18 19:44:21","error_message":""},"status":"success","completion_percent":100.0,"problem_steps":[],"active_steps":[],"focus_level":"success","focus_step_key":"","focus_step_title":"","focus_summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","bootstrap_state_counts":{"total":1,"online":1,"pending_bootstrap":0,"runtime_only":0,"stale":0,"unmanaged":0,"unknown":0},"status_label":"成功","steps_total":1,"steps_success":1,"steps_running":0,"steps_problem":0,"steps_terminal":1,"summary":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","summary_text":"接入工单已生成,目标节点已完成 Agent 接入,可继续执行接管后验收。","focus_ref":{"kind":"playbook_run","run_code":"pbr-772e6e9e7b","playbook_key":"onboarding.bootstrap","group_key":"onboarding","focus_step_key":"","focus_step_title":"","event_key":"","node_code":""}}],"summary":{"total":12,"filtered_total":18,"unfiltered_total":18,"success":6,"running":0,"attention":6,"queued":0,"scanned_jobs":127,"available_status_counts":{"success":12,"attention":6},"available_group_counts":{"diagnostics":3,"onboarding":13,"scene":2},"filters":{"status":"","group_key":"","query":""}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/scene_overseas_control_01.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/scene_overseas_control_01.json new file mode 100644 index 0000000..1c18fe8 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/scene_overseas_control_01.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"missing_sample","status_label":"缺少样本","status_type":"warning","summary":"当前还没有收到该参与节点的远端日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":0,"key_line_count":0,"full_line_count":0,"last_at":"","last_line":""},"missing_reason_code":"no_sample","missing_reason":"当前还没有收到该参与节点的远端日志样本。","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 04:31:52"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/stack.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/stack.json new file mode 100644 index 0000000..c9662f4 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/stack.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"generated_at":"2026-04-19 04:31:51","diagnosis":{"contract_key":"ops_stack_diagnosis_contract","contract_version":"v1","registry_version":"2026-04-18","base_url":"http://127.0.0.1:8100","contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_driver_contract","ops_observability_contract","ops_playbook_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_driver_contract","title":"Ops Driver Contract","status":"active","version":"v1","summary":"冻结 overview / driver-feed / codex-brief / driver-actions / runbook sequence 的统一驾驶 contract。","primary_endpoint":"/api/v1/ops/driver-feed","schema_doc_path":"docs/schemas/ops_driver_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_driver_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/driver-feed","/api/v1/ops/codex-brief","/api/v1/ops/driver-actions/resolve","/api/v1/ops/driver-actions/execute-resolved","/api/v1/ops/codex-actions/resolve","/api/v1/ops/codex-actions/execute","/api/v1/ops/activity-stream","/api/v1/ops/runbook","/api/v1/ops/runbook/sequences/{sequence_key}/resolve","/api/v1/ops/runbook/sequences/{sequence_key}/execute","/api/v1/ops/driver-actions/preview","/api/v1/ops/driver-actions/execute"],"related_contract_keys":["release_hub_contract","ops_playbook_contract","ops_observability_contract","ops_stack_diagnosis_contract"]},{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_playbook_contract","title":"Ops Playbook Contract","status":"active","version":"v1","summary":"冻结 playbook catalog / preview / run / events 的正式编排 contract。","primary_endpoint":"/api/v1/ops/playbooks","schema_doc_path":"docs/schemas/ops_playbook_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_playbook_contract","discovery_endpoints":["/api/v1/ops/playbooks","/api/v1/ops/playbooks/preview","/api/v1/ops/playbooks/execute","/api/v1/ops/playbook-runs","/api/v1/ops/playbook-runs/{run_code}","/api/v1/ops/playbook-runs/{run_code}/events","/api/v1/ops/playbook-runs/{run_code}/rerun","/api/v1/ops/playbook-runs/{run_code}/cancel"],"related_contract_keys":["ops_job_contract","release_hub_contract","ops_driver_contract","ops_observability_contract","ops_stack_diagnosis_contract"]}]},"surface_status":"healthy","automation_status":"attention","stack_status":"attention","issue_total":2,"blocking_issue_total":0,"warning_issue_total":2,"info_issue_total":0,"missing_surfaces":[],"launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"next_step":{"action_code":"focus_playbook_run","source":"overview.primary","reason":"来自 overview.recommendation.primary_action_code","command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","focus_ref":{},"contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"recommended_actions":[{"action_code":"focus_playbook_run","source":"overview.primary","title":"先处理异常编排","reason":"来自 overview.recommendation.primary_action_code","focus_ref":{},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},{"action_code":"fix_managed_nodes","source":"release_hub.launchpad","title":"待补执行面","reason":"来自 release_launchpad.launchpad_status.recommended_action_code","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","contract_keys":["ops_stack_diagnosis_contract","ops_agent_protocol","release_hub_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_agent_protocol","release_hub_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_agent_protocol","title":"Node Agent Protocol","status":"active","version":"v1","summary":"冻结海外控制面与大陆 Node Agent 之间的认证、注册、心跳、拉任务、完成回执与事件回放 contract。","primary_endpoint":"/api/v1/ops/agent/bootstrap-plan","schema_doc_path":"docs/schemas/ops_agent_protocol.md","detail_endpoint":"/api/v1/ops/contracts/ops_agent_protocol","discovery_endpoints":["/api/v1/ops/agent/tokens","/api/v1/ops/agent/bootstrap-plan","/api/v1/ops/agent/register","/api/v1/ops/agent/heartbeat","/api/v1/ops/agent/pull","/api/v1/ops/agent/jobs/{job_id}/start","/api/v1/ops/agent/jobs/{job_id}/complete","/api/v1/ops/agent/jobs/{job_id}/events","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_observability_contract","ops_stack_diagnosis_contract"]},{"key":"release_hub_contract","title":"Release Hub Contract","status":"active","version":"v1","summary":"冻结 Release / Rollout / Launchpad / Default Gate 的对象模型与门禁口径。","primary_endpoint":"/api/v1/ops/releases/launchpad","schema_doc_path":"docs/schemas/release_hub_contract.md","detail_endpoint":"/api/v1/ops/contracts/release_hub_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/releases","/api/v1/ops/releases/latest","/api/v1/ops/releases/launchpad","/api/v1/ops/releases/{release_id}","/api/v1/ops/releases/{release_id}/rollouts","/api/v1/ops/rollouts/{rollout_id}","/api/v1/ops/rollouts/{rollout_id}/jobs","/api/v1/ops/rollouts/{rollout_id}/advance"],"related_contract_keys":["ops_job_contract","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]}]}},{"action_code":"open_playbook_run_latest_events","source":"overview.secondary","title":"先处理异常编排","reason":"来自 overview.recommendation.secondary_action_code","focus_ref":{},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","contract_keys":["ops_stack_diagnosis_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},{"action_code":"open_worker_logs_participating","source":"issue:remote_log_sync_waiting_sample","title":"远端日志回传虽然已经开启,但参与检测节点的现场样本仍未完全形成。","reason":"参与检测节点 3 台,已覆盖 2 台,缺口节点 overseas-control-01。","focus_ref":{"kind":"node_scene_log","node_code":"overseas-control-01","mode":"full","limit":120,"source":"remote_log_sync_waiting_sample"},"command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","contract_keys":["ops_stack_diagnosis_contract","ops_observability_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_observability_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]}]}}],"operator_decision":{"lane":"ops_jobs","priority":"medium","reason_code":"playbook_runs_need_attention","title":"按总检默认下一步继续处理","summary":"来自 overview.recommendation.primary_action_code","next_focus":{},"primary_command_key":"focus_playbook_run","secondary_command_key":""},"next_actions":{"primary_command_key":"focus_playbook_run","primary_command":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","secondary_command_key":"","secondary_command":""},"recommended_commands":{"stack-diagnosis":"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","stack-check":"bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","doctor":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100","contracts":"bash domain-api/deploy/multi-region/drive_ops_center.sh contracts http://127.0.0.1:8100","runtime-refresh-recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100","api-restart":"bash domain-api/deploy/multi-region/drive_ops_center.sh runtime-refresh-recover http://127.0.0.1:8100","focus_playbook_run":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","fix_managed_nodes":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","open_playbook_run_latest_events":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","open_worker_logs_participating":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","focus-release-hub":"bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad http://127.0.0.1:8100","release-launchpad-worker":"bash domain-api/deploy/multi-region/drive_ops_center.sh release-launchpad http://127.0.0.1:8100 worker"},"issues":[{"code":"remote_log_sync_waiting_sample","severity":"warning","layer":"overview.log_sync","summary":"远端日志回传虽然已经开启,但参与检测节点的现场样本仍未完全形成。","detail":"参与检测节点 3 台,已覆盖 2 台,缺口节点 overseas-control-01。","action_code":"open_worker_logs_participating","focus_ref":{"kind":"node_scene_log","node_code":"overseas-control-01","mode":"full","limit":120,"source":"remote_log_sync_waiting_sample"},"commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 overseas-control-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-controller-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"],"contract_keys":["ops_stack_diagnosis_contract","ops_observability_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_observability_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]}]}},{"code":"playbook_runs_need_attention","severity":"warning","layer":"playbook_runs","summary":"最近存在未完全收口的 playbook run,建议先确认执行序列是否停在中间步骤。","detail":"problem_runs_total=2","action_code":"","focus_ref":{},"commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100"],"contract_keys":["ops_stack_diagnosis_contract","ops_playbook_contract"],"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_stack_diagnosis_contract","contract_keys":["ops_stack_diagnosis_contract","ops_playbook_contract"],"contracts":[{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract,供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]},{"key":"ops_playbook_contract","title":"Ops Playbook Contract","status":"active","version":"v1","summary":"冻结 playbook catalog / preview / run / events 的正式编排 contract。","primary_endpoint":"/api/v1/ops/playbooks","schema_doc_path":"docs/schemas/ops_playbook_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_playbook_contract","discovery_endpoints":["/api/v1/ops/playbooks","/api/v1/ops/playbooks/preview","/api/v1/ops/playbooks/execute","/api/v1/ops/playbook-runs","/api/v1/ops/playbook-runs/{run_code}","/api/v1/ops/playbook-runs/{run_code}/events","/api/v1/ops/playbook-runs/{run_code}/rerun","/api/v1/ops/playbook-runs/{run_code}/cancel"],"related_contract_keys":["ops_job_contract","release_hub_contract","ops_driver_contract","ops_observability_contract","ops_stack_diagnosis_contract"]}]}}],"operator_hints":["当前总检可用,但仍有缺口;建议先按 next_step 收口,再进入更细粒度脚本。"],"quick_commands":["bash domain-api/deploy/multi-region/drive_ops_center.sh stack-diagnosis http://127.0.0.1:8100 summary","bash domain-api/deploy/multi-region/drive_ops_center.sh doctor http://127.0.0.1:8100","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 fix_managed_nodes","bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_playbook_run_latest_events","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 overseas-control-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-worker-01 120 full","bash domain-api/deploy/multi-region/drive_ops_center.sh scene-node-log http://127.0.0.1:8100 mainland-controller-01 120 full"]},"api":{"http_status":"200","available":true,"service":"domain-api","version":"0.1.0","api_prefix":"/api/v1","worker_mode":"linux-systemd"},"surface_matrix":{"status_counts":{"total":9,"available":9,"missing":0},"items":[{"name":"api","available":true,"elapsed_ms":0},{"name":"contracts","available":true,"elapsed_ms":0},{"name":"link_snapshot","available":true,"elapsed_ms":0},{"name":"overview","available":true,"elapsed_ms":5140},{"name":"managed_nodes","available":true,"elapsed_ms":278},{"name":"release_hub","available":true,"elapsed_ms":0},{"name":"runtime_build_info","available":true,"elapsed_ms":0},{"name":"playbook_runs","available":true,"elapsed_ms":332},{"name":"activity_stream","available":true,"elapsed_ms":0}]},"contracts":{"http_status":"200","available":true,"elapsed_ms":0,"registry_version":"2026-04-18","schema_version":"v1","contracts_total":11,"contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract","ops_stack_diagnosis_contract","ops_doctor_decision_contract","ops_go_live_signoff_contract","ops_go_live_bundle_contract","ops_go_live_review_contract"],"error":""},"link_snapshot":{"http_status":"200","available":true,"elapsed_ms":0,"status":"attention","headline":"当前已有执行节点,但远端日志回传仍未覆盖全部参与节点,建议先补齐现场观测。","next_action_codes":["open_worker_logs_participating","run_inspection_participating"],"error":""},"overview":{"http_status":"200","available":true,"elapsed_ms":5140,"recommendation_key":"playbook-run-attention-pbr-9cf64aa152","recommendation_title":"先处理异常编排","primary_action_code":"focus_playbook_run","secondary_action_code":"open_playbook_run_latest_events","focus_ref":{},"execution_scene_summary":"有效执行节点 3 台;正在执行/领任务 3 台;近窗刚有吞吐 0 台;在线但未参与 0 台","execution_scene_counts":{"dispatch_active":3,"recent_only":0,"standby":0,"load_syncing":0},"log_sync":{"status":"partial_coverage","status_label":"部分覆盖","mode":"full","mode_label":"全量回传","enabled":true,"covered_participating_nodes":2,"participating_nodes_total":3,"missing_sample_node_codes":["overseas-control-01"]},"error":""},"managed_nodes":{"http_status":"200","available":true,"elapsed_ms":278,"total":3,"managed_total":3,"managed_enabled":3,"online":3,"agent_ready":3,"ssh_ready":2,"remote_access_ready":3,"participating":3,"dispatch_active":3,"standby":0,"load_syncing":0,"queue_retrying_nodes":0,"queue_dead_letter_nodes":0,"queue_pending_records":0,"queue_dead_letter_records":0,"agent_state_counts":{"online_busy":3},"remote_access_state_counts":{"hybrid_ready":2,"agent_ready":1},"delivery_queue_state_counts":{"healthy":3},"problem_nodes":[],"first_handover_gap_node_code":"","first_handover_gap":{},"error":""},"release_hub":{"http_status":"200","available":true,"elapsed_ms":0,"latest_release_version":"domaincheck_release_20260418_013833","latest_release_status":"ready","default_rollout_gate_status":"ready","default_rollout_gate_label":"可发 Rollout","default_rollout_gate_summary":"默认 Release 与默认 Rollout 目标均已通过门禁,当前可直接进入定向或灰度 Rollout。","default_rollout_gate_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"default_rollout_gate"},"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_summary":"最新发布包已经可用,但当前没有可用于智能 Rollout 的在线启用执行节点,需要先补齐托管或接入状态。","recommended_action_code":"fix_managed_nodes","recommended_execution_mode":"","focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"},"error":""},"runtime_build_info":{"http_status":"200","available":true,"elapsed_ms":0,"source":"latest_release","package_name":"domaincheck_release_20260418_215930","generated_at":"2026-04-18T21:59:32","commit_sha":"246838ae4c07","commit_ref":"main","checksum":"69fe1074c472c4a7601f1d7f0a78866fc1a2011949e46d5d26ef603fc1aa5f7a","manifest_path":"/www/wwwroot/getDomain/release/latest_release.json","repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"route_surface":{"mode":"registered","registered_paths_total":150,"surface_flags":{"ops_contracts":true,"ops_contract_detail":true,"ops_stack_diagnosis":true,"ops_node_handover":true,"ops_node_onboarding":true,"ops_node_onboarding_bootstrap_preview":true,"ops_node_onboarding_bootstrap_execute":true,"ops_node_onboarding_acceptance_preview":true,"ops_node_onboarding_acceptance_execute":true,"ops_node_onboarding_recovery_preview":true,"ops_node_onboarding_recovery_execute":true,"ops_node_scene_log":true,"ops_node_handover_bootstrap_plan":true,"ops_driver_feed":true,"ops_codex_brief":true,"ops_activity_stream":true,"ops_runbook_resolve":true,"ops_runbook_execute":true,"runtime_build_info":true},"expected_paths":{"ops_contracts":"/api/v1/ops/contracts","ops_contract_detail":"/api/v1/ops/contracts/{contract_key}","ops_stack_diagnosis":"/api/v1/ops/stack-diagnosis","ops_node_handover":"/api/v1/ops/nodes/{node_code}/handover","ops_node_onboarding":"/api/v1/ops/nodes/{node_code}/onboarding","ops_node_onboarding_bootstrap_preview":"/api/v1/ops/nodes/{node_code}/onboarding/bootstrap/preview","ops_node_onboarding_bootstrap_execute":"/api/v1/ops/nodes/{node_code}/onboarding/bootstrap/execute","ops_node_onboarding_acceptance_preview":"/api/v1/ops/nodes/{node_code}/onboarding/acceptance/preview","ops_node_onboarding_acceptance_execute":"/api/v1/ops/nodes/{node_code}/onboarding/acceptance/execute","ops_node_onboarding_recovery_preview":"/api/v1/ops/nodes/{node_code}/onboarding/recovery/preview","ops_node_onboarding_recovery_execute":"/api/v1/ops/nodes/{node_code}/onboarding/recovery/execute","ops_node_scene_log":"/api/v1/ops/nodes/{node_code}/scene-log","ops_node_handover_bootstrap_plan":"/api/v1/ops/nodes/{node_code}/handover/bootstrap-plan","ops_driver_feed":"/api/v1/ops/driver-feed","ops_codex_brief":"/api/v1/ops/codex-brief","ops_activity_stream":"/api/v1/ops/activity-stream","ops_runbook_resolve":"/api/v1/ops/runbook/sequences/{sequence_key}/resolve","ops_runbook_execute":"/api/v1/ops/runbook/sequences/{sequence_key}/execute","runtime_build_info":"/api/v1/runtime/build-info"},"missing_keys":[],"missing_paths":[],"surface_complete":true},"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"error":""},"playbook_runs":{"http_status":"200","available":true,"elapsed_ms":332,"recent_total":6,"problem_runs_total":2,"problem_runs":[{"run_code":"pbr-9cf64aa152","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-9cf64aa152","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}},{"run_code":"pbr-2be333d3ce","status":"attention","status_label":"待关注","focus_step_key":"node_agent_logs","focus_summary":"收集 Node Agent 日志 当前出现失败,建议先看事件,再决定是否整轮重跑。","focus_ref":{"kind":"playbook_run","run_code":"pbr-2be333d3ce","playbook_key":"onboarding.acceptance","group_key":"onboarding","focus_step_key":"node_agent_logs","focus_step_title":"收集 Node Agent 日志","event_key":"","node_code":""}}],"error":""},"activity_stream":{"http_status":"200","available":true,"elapsed_ms":0,"recent_total":8,"status_counts":{"running":1,"attention":1,"success":3,"failed":3},"top_items":[{"kind":"execution_scene","status":"running","summary":"有效执行节点 3 台;正在执行/领任务 3 台;近窗刚有吞吐 0 台;在线但未参与 0 台","occurred_at":"2026-04-19 04:31:45","focus_ref":{"kind":"execution_scene","scene_key":"execution-scene:live"},"source_focus_ref":{},"ui_intent_kind":"focus_execution_scene"},{"kind":"log_sync","status":"attention","summary":"当前已有 2/3 台参与节点回传日志,仍有节点未被覆盖。 当前仍有未覆盖节点:overseas-control-01","occurred_at":"2026-04-19 04:31:42","focus_ref":{"kind":"execution_scene","scene_key":"execution-scene:log-sync"},"source_focus_ref":{},"ui_intent_kind":"focus_execution_scene"},{"kind":"ops_job","status":"success","summary":"已发送 Worker 控制指令: start_detection","occurred_at":"2026-04-19 04:21:32","focus_ref":{"kind":"ops_job","job_id":135,"job_code":"ops-20260419042114-4da9d9","action":"runtime.start_detection","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"failed","summary":"Failed to start domaincheck-worker.service: Interactive authentication required.\nSee system logs and 'systemctl status domaincheck-worker.service' for details.","occurred_at":"2026-04-19 04:21:29","focus_ref":{"kind":"ops_job","job_id":134,"job_code":"ops-20260419042114-cf8ccb","action":"runtime.start_worker","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"},{"kind":"ops_job","status":"success","summary":"待检测任务批次拉取并入库成功","occurred_at":"2026-04-19 04:21:27","focus_ref":{"kind":"ops_job","job_id":133,"job_code":"ops-20260419042114-5eaf56","action":"runtime.pull_tasks","target_node_code":"mainland-controller-01"},"source_focus_ref":{},"ui_intent_kind":"job_events"}],"error":""}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/summary.env b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/summary.env new file mode 100644 index 0000000..1c06cc1 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_5/summary.env @@ -0,0 +1,16 @@ +CYCLES="5" +GO_LIVE_STATUS="attention" +PUBLISH_READY="false" +LOG_SYNC_STATE="partial_coverage" +LOG_SYNC_MISSING_NODE_CODES="overseas-control-01" +STACK_STATUS="attention" +ISSUE_TOTAL="2" +BLOCKING_ISSUE_TOTAL="0" +ISSUE_CODES="remote_log_sync_waiting_sample,playbook_runs_need_attention" +LAUNCHPAD_STATUS="blocked" +LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance" +PROBLEM_RUNS_TOTAL="0" +PROBLEM_RUN_CODE="" +SCENE_STATUS="missing_sample" +SCENE_LINE_COUNT="0" +GENERATED_AT="2026-04-19 04:31:45" diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_6/summary.env b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_6/summary.env new file mode 100644 index 0000000..e097e3c --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_6/summary.env @@ -0,0 +1,16 @@ +CYCLES="6" +GO_LIVE_STATUS="" +PUBLISH_READY="false" +LOG_SYNC_STATE="" +LOG_SYNC_MISSING_NODE_CODES="" +STACK_STATUS="" +ISSUE_TOTAL="0" +BLOCKING_ISSUE_TOTAL="0" +ISSUE_CODES="" +LAUNCHPAD_STATUS="" +LAUNCHPAD_RECOMMENDED_ACTION="" +PROBLEM_RUNS_TOTAL="0" +PROBLEM_RUN_CODE="" +SCENE_STATUS="" +SCENE_LINE_COUNT="0" +GENERATED_AT="" diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_7/summary.env b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_7/summary.env new file mode 100644 index 0000000..c116181 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_7/summary.env @@ -0,0 +1,16 @@ +CYCLES="7" +GO_LIVE_STATUS="" +PUBLISH_READY="false" +LOG_SYNC_STATE="" +LOG_SYNC_MISSING_NODE_CODES="" +STACK_STATUS="" +ISSUE_TOTAL="0" +BLOCKING_ISSUE_TOTAL="0" +ISSUE_CODES="" +LAUNCHPAD_STATUS="" +LAUNCHPAD_RECOMMENDED_ACTION="" +PROBLEM_RUNS_TOTAL="0" +PROBLEM_RUN_CODE="" +SCENE_STATUS="" +SCENE_LINE_COUNT="0" +GENERATED_AT="" diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/go_live.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/go_live.json new file mode 100644 index 0000000..1aabde9 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/go_live.json @@ -0,0 +1 @@ +{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 04:34:28","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断,但仍有上线前关注项,建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":1,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":1,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null} \ No newline at end of file diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/summary.env b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/summary.env new file mode 100644 index 0000000..bfc008e --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/summary.env @@ -0,0 +1,16 @@ +CYCLES="8" +GO_LIVE_STATUS="attention" +PUBLISH_READY="false" +LOG_SYNC_STATE="full_capture" +LOG_SYNC_MISSING_NODE_CODES="" +STACK_STATUS="" +ISSUE_TOTAL="0" +BLOCKING_ISSUE_TOTAL="0" +ISSUE_CODES="" +LAUNCHPAD_STATUS="" +LAUNCHPAD_RECOMMENDED_ACTION="" +PROBLEM_RUNS_TOTAL="0" +PROBLEM_RUN_CODE="" +SCENE_STATUS="" +SCENE_LINE_COUNT="0" +GENERATED_AT="2026-04-19 04:34:28" diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929_analysis.md b/docs/ops_center_runtime/night_runs/night_run_20260419_012929_analysis.md new file mode 100644 index 0000000..23464cd --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929_analysis.md @@ -0,0 +1,59 @@ +# night_run_20260419_012929 初步分析 + +## 结论 + +- 夜跑不是完全空跑,`cycle_1` 到 `cycle_8` 期间持续执行了巡检、收口和日志回传恢复动作。 +- 真正的中断点出现在 `2026-04-19 04:32:49` 到 `04:33:35`,本机 API `127.0.0.1:8100` 短时不可达,导致两轮自动动作直接失败。 +- 最终停止原因是 `signoff_ready_candidate`,这是“候选可签收”型收口,不等同于“整轮全绿、无异常结束”。 + +## 关键时间点 + +- `2026-04-19 01:29:29 +0800` + - 夜跑启动。 +- `cycle_1` 到 `cycle_5` + - 持续产出 `go_live.json`、`launchpad.json`、`playbook_runs.json`、`stack.json`、`scene_overseas_control_01.json`、`summary.env`。 +- `2026-04-19 04:32:04 +0800` + - `run_inspection_participating` 成功创建 playbook,回执 `pbr-6d77b32f40`。 +- `2026-04-19 04:32:49 +0800` + - `cycle=6` + - `log-sync-recover` 调用失败。 + - `driver-run run_inspection_participating` 调用失败。 + - 错误为 `curl: (7) Failed to connect to 127.0.0.1 port 8100: Connection refused`。 +- `2026-04-19 04:33:34 +0800` + - `cycle=7` + - 同类动作再次失败,错误相同。 +- `2026-04-19 04:34:28 +0800` + - `cycle=8` + - `go_live=attention` + - `log_sync=full_capture` + - 夜跑停止,`reason=signoff_ready_candidate`。 + +## 已确认的问题 + +- 夜跑期间存在控制面 API 短时离线或重启窗口。 +- 自动恢复逻辑在 API 不可达时会直接失败,但日志里没有看到进一步的退避、跳过本轮、等待 API 恢复后的再确认闭环。 +- 当时的“可签收候选”判断,掺杂了 API 短时不可达窗口,所以不能把这次夜跑结论直接当成正式签收证据。 + +## 这轮修复后的关联状态 + +- 当前中央控制面已经恢复正常。 +- `runtime/cluster` 已恢复为 3 台有效执行节点。 +- `detect/status` 已恢复远端日志回传。 +- `mainland-controller-01` 当前已恢复 `100/100`。 +- `mainland-worker-01` 当前已进入活跃参与,中央已看到 `2/50`。 + +## 明天继续看时,优先检查 + +- `docs/ops_center_runtime/night_runs/night_run_20260419_012929.log` + - 重点看 `04:32:49` 到 `04:34:28` 这段 API 拒绝连接窗口。 +- `docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/go_live.json` + - 确认 `go_live=attention` 的具体触发项。 +- `docs/ops_center_runtime/night_runs/night_run_20260419_012929_report.md` + - 对照夜跑最终报告和原始日志,确认是否把“候选可签收”误当成“正式通过”。 + +## 下一步建议 + +- 补一条夜跑期间的 API 可用性守护: + - 发现 `127.0.0.1:8100` 不可达时,不立刻继续推进收口动作,先等待 API 恢复后重试。 +- 把“候选可签收”和“正式可签收”拆开: + - 避免在 API 短时重启窗口里出现假阳性收口。 diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929_report.md b/docs/ops_center_runtime/night_runs/night_run_20260419_012929_report.md new file mode 100644 index 0000000..dccfcb9 --- /dev/null +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929_report.md @@ -0,0 +1,44 @@ +# NIGHT RUN REPORT night_run_20260419_012929 + +- Base URL: `http://127.0.0.1:8100` +- Deadline: `2026-04-20 12:00:00 +0800` +- Stop Reason: `signoff_ready_candidate` +- Cycles: `8` +- Log Sync Recover Runs: `4` +- Inspection Runs: `4` +- Inspection Churn Runs: `0` +- Quick Rechecks: `4` + +## Final Snapshot + +- `go_live_status = attention` +- `publish_ready = false` +- `log_sync_state = full_capture` +- `issue_total = 0` +- `problem_runs_total = 0` +- `launchpad_status = ` +- `launchpad_recommended_action = ` +- `problem_run_code = ` + +## Summary JSON + +```json +{ + "cycles": 8, + "go_live_status": "attention", + "publish_ready": false, + "log_sync_state": "full_capture", + "log_sync_missing_node_codes": [], + "stack_status": "", + "issue_total": 0, + "blocking_issue_total": 0, + "issue_codes": [], + "launchpad_status": "", + "launchpad_recommended_action": "", + "problem_runs_total": 0, + "problem_run_code": "", + "scene_status": "", + "scene_line_count": 0, + "generated_at": "2026-04-19 04:34:28" +} +``` diff --git a/docs/ops_center_runtime/night_runs/night_run_20260419_012929_summary.json b/docs/ops_center_runtime/night_runs/night_run_20260419_012929_summary.json index b590cd9..8734a94 100644 --- a/docs/ops_center_runtime/night_runs/night_run_20260419_012929_summary.json +++ b/docs/ops_center_runtime/night_runs/night_run_20260419_012929_summary.json @@ -1,20 +1,18 @@ { - "cycles": 1, + "cycles": 8, "go_live_status": "attention", "publish_ready": false, "log_sync_state": "full_capture", "log_sync_missing_node_codes": [], - "stack_status": "attention", - "issue_total": 1, + "stack_status": "", + "issue_total": 0, "blocking_issue_total": 0, - "issue_codes": [ - "playbook_runs_need_attention" - ], - "launchpad_status": "blocked", - "launchpad_recommended_action": "run_acceptance", + "issue_codes": [], + "launchpad_status": "", + "launchpad_recommended_action": "", "problem_runs_total": 0, "problem_run_code": "", - "scene_status": "full_capture", - "scene_line_count": 1, - "generated_at": "2026-04-19 01:29:38" + "scene_status": "", + "scene_line_count": 0, + "generated_at": "2026-04-19 04:34:28" } \ No newline at end of file diff --git a/docs/test.md b/docs/test.md new file mode 100644 index 0000000..f99224f --- /dev/null +++ b/docs/test.md @@ -0,0 +1,114 @@ +开个定时任务不停扫数据库,未检测的 快到期删除的,全部扫出来推送给注册任务队列; + +注册检测:筛选出可以注册的,这个完成后,不管成功失败,返会标准化结果给 controller,告诉他,这个流程我走完了,如果是外部原因导致未能出结果就失败的,controller 会重新投入注册检测,更新数据库,如果是成功的,把对应的状态更新数据库,跟新注册状态; + +controller 收到返回注册完成后, 以标准化格式下一步任务队列里面 ,等worker 来啦取, +worker 只负责去controller 啦取的任务,处理任务,根据任务标准标识,安排对于的函数处理,所以woker只复制啦和跑,我没任务了,我很有空,我就去controller 获取任务,只要你给,我就跑,跑完返回结果给你; +controller 在收到worker的啦取请求后,按后台勾选的配置任务队列,按顺序返回给woker,不是随便返回 +controller 返回任务时,要同时完成认领 +也就是: +标记该 task 已分配给某个 worker +进入 running 状态 +设置超时 TTL +超时未回传则回收重投 +否则会出现: +worker 拿了任务挂了 +controller 以为还在跑 +任务永远丢了 + + +xx检测:当前 xx 步骤执行并返回判定结果,这个完成后,不管成功失败,返会标准化结果给controller,告诉他,这个xx流程我走完了,如果是外部原因导致失败的,还没有出结果的,非业务判定不通过被跑到为黑名单的,controller 会重新投入当前xx任务队列,如果名中黑名单,直接跟新黑名单状态,不在分发到后面所有步骤,更新数据库,如果是成功的,跟新当前xx检测状态, + +controller 收到某步骤返回结果后,先根据当前 后台 配置和该域名已完成步骤状态,解析该域名在本次流程中的下一勾选步骤;若存在下一步骤,则以标准化任务格式投入对应任务队列;若不存在,则标记本次流程完成 + +只按顺序处理后台勾选的任务 +注册检测 + 百度检测 + 站长检测 + 爱站检测 + 时光机检测 + 聚查检测 + 桔子检测 + + 备注: + 时光机具体还要细分方案 ,目前项目内已经又对于的方案,加一个直晒前最近5年的快照,先跑通,后面可以继续细优化 + 步骤都是按后台勾选,把勾选的跑完就算流程跑完,第一次没勾选的,下次勾选,可以直接跑够选的步骤,其他跳过; + + + +海外主库(域名源/最终账本) + ↓ +controller dispatcher 拉取待处理域名 + ↓ +按 pipeline 配置生成首个待执行步骤任务 + ↓ +推入 Redis 对应步骤队列 + ↓ +worker 向 controller 拉取任务 + ↓ +controller 认领并分配任务给 worker + ↓ +worker 执行检测并回传标准化结果 + ↓ +controller stage-processor 更新本地控制状态 + ↓ +根据结果判断: + - retry -> 重投当前步骤 + - black_hit -> 拉黑并终止 + - reject -> 终止/复核 + - pass -> 解析下一勾选步骤并投递 + ↓ +controller syncer 批量同步海外主库 + ↓ +controller finalizer 标记本次流程完成 + + + + +域名检测流程设计 +1. 总体原则 +海外主库负责域名源数据与最终账本持久化,不参与高频实时调度 +controller 负责任务编排、任务分发、结果处理、状态推进与批量同步 +Redis 负责各步骤待执行队列、运行中任务、重试任务与去重控制 +worker 仅负责向 controller 拉取任务、执行对应检测函数、回传标准化结果 +2. 注册任务投递 +定时任务持续扫描海外主库中的待检测域名,包括未检测域名、快到期删除域名等 +controller dispatcher 将符合条件的域名按标准化任务格式推入注册任务队列 +3. 步骤执行规则 + +每个检测步骤执行完成后,无论结果如何,worker 均需返回标准化结果给 controller。 +controller 根据结果类型做统一处理: + +若为外部原因导致未得到有效结果,则根据重试策略重新投入当前步骤队列 +若为业务判定不通过,则更新当前步骤状态,并按策略终止本次流程 +若命中黑名单,则更新黑名单状态并终止后续所有步骤 +若业务判定通过,则更新当前步骤状态,并根据本次 pipeline 配置解析下一勾选步骤,投入对应任务队列 +4. pipeline 推进规则 +所有步骤按后台勾选生成本次 pipeline +controller 仅按勾选顺序为单个域名推进下一步骤 +若某步骤在本次 pipeline 中未勾选,则直接跳过 +若前次未勾选、后次新增勾选,则可直接从已完成状态之后继续补跑,无需重跑已完成步骤 +5. worker 拉取规则 +worker 空闲时向 controller 发起拉取任务请求 +controller 根据任务队列状态、任务顺序与后台配置返回当前可执行任务 +worker 只负责执行任务,不负责流程判断、不负责决定下一步骤、不直接高频写海外主库 +6. 状态更新规则 +controller stage-processor 实时更新本地控制状态 +controller syncer 以批量方式将步骤状态、黑名单状态、流程状态同步至海外主库 +controller finalizer 在本次 pipeline 所有勾选步骤完成或流程被终止后,标记流程完成 +7. 当前步骤顺序 + +按后台勾选顺序处理以下任务: + +注册检测 +百度检测 +站长检测 +爱站检测 +时光机检测 +聚查检测 +桔子检测 +8. 时光机一期方案 +当前先按最近 5 年快照执行简化方案,先跑通主流程 +后续再继续细化为更完整的时光机子流程 + +controller 机器如果性能足够剩余也可以部署worker 跑, \ No newline at end of file diff --git a/domain-api/app/api/routes/detect.py b/domain-api/app/api/routes/detect.py index e137450..31d2ad8 100644 --- a/domain-api/app/api/routes/detect.py +++ b/domain-api/app/api/routes/detect.py @@ -4,6 +4,7 @@ from uuid import uuid4 from fastapi import APIRouter +from app.core.config import settings from app.schemas.common import ApiResponse from app.services.detect_job_service import ( append_detect_job_event, @@ -12,6 +13,7 @@ from app.services.detect_job_service import ( get_detect_job_summary, get_detect_queue_health, list_detect_jobs, + normalize_detect_step_code, ) from app.services.detect_service import get_detect_status from app.services.detect_run_service import create_detect_run_snapshot, finalize_detect_run, mark_detect_run_stopping @@ -22,6 +24,13 @@ from app.services.worker_control_service import send_worker_command, start_worke router = APIRouter(tags=["detect"]) +def _local_worker_expected_on_this_node() -> bool: + return not ( + str(settings.node_region or "").strip() == "overseas" + and str(settings.node_role or "").strip() == "control" + ) + + def _build_detect_action_result( *, action: str, @@ -255,6 +264,11 @@ def detect_queue_summary(window_minutes: int = 15) -> ApiResponse: return ApiResponse(data=get_detect_queue_health(window_minutes=window_minutes)) +@router.get("/detect/queue-health", response_model=ApiResponse) +def detect_queue_health_alias(window_minutes: int = 15) -> ApiResponse: + return ApiResponse(data=get_detect_queue_health(window_minutes=window_minutes)) + + @router.get("/detect/jobs/{job_id}", response_model=ApiResponse) def detect_job_detail(job_id: int) -> ApiResponse: data = get_detect_job_summary(job_id, event_limit=100) @@ -264,8 +278,17 @@ def detect_job_detail(job_id: int) -> ApiResponse: @router.post("/detect/start", response_model=ApiResponse) -def start_detect() -> ApiResponse: - job_summary = create_detect_job_if_needed(limit=1000, created_by="api") +def start_detect(step_code: str | None = None) -> ApiResponse: + normalized_step_code = normalize_detect_step_code(step_code) + if step_code and not normalized_step_code: + result = _build_detect_action_result( + action="start", + ok=False, + message=f"暂不支持步骤任务: {step_code}", + data={"job": None, "step_code": str(step_code or "").strip()}, + ) + return ApiResponse(code=1, message=f"暂不支持步骤任务: {step_code}", data=result) + job_summary = create_detect_job_if_needed(limit=10000, created_by="api", step_code=step_code) if not job_summary: result = _build_detect_action_result( action="start", @@ -292,42 +315,58 @@ def start_detect() -> ApiResponse: }, ) - ok, message = start_worker() - if not ok: - result = _build_detect_action_result( - action="start", - ok=False, - message=message, - data={"job": job_summary}, + local_worker_expected = _local_worker_expected_on_this_node() + if local_worker_expected: + ok, message = start_worker() + if not ok: + result = _build_detect_action_result( + action="start", + ok=False, + message=message, + data={"job": job_summary}, + ) + append_detect_job_event( + job_summary["job_id"], + event_type="job_dispatch_failed", + level="error", + message=f"启动 Worker 失败: {message}", + payload={"cycle_token": cycle_token}, + ) + return ApiResponse( + code=1, + message=message, + data=result, + ) + + command_ok, command_message = send_worker_command( + "start_detection", + payload={ + "cycle_token": cycle_token, + "job_id": job_summary["job_id"], + "job_code": job_summary["job_code"], + "task_mode": job_summary.get("task_mode", ""), + "step_code": job_summary.get("step_code", ""), + }, ) append_detect_job_event( job_summary["job_id"], - event_type="job_dispatch_failed", - level="error", - message=f"启动 Worker 失败: {message}", + event_type="job_dispatch_sent" if command_ok else "job_dispatch_rejected", + level="info" if command_ok else "error", + message=command_message, payload={"cycle_token": cycle_token}, ) - return ApiResponse( - code=1, - message=message, - data=result, + else: + ok = True + message = "当前节点为海外控制面,仅负责派单,不启动本机 Worker" + command_ok = True + command_message = "已跳过本机 Worker 启动,转为仅向大陆执行节点派发检测动作" + append_detect_job_event( + job_summary["job_id"], + event_type="job_dispatch_skipped_local", + level="info", + message=command_message, + payload={"cycle_token": cycle_token}, ) - - command_ok, command_message = send_worker_command( - "start_detection", - payload={ - "cycle_token": cycle_token, - "job_id": job_summary["job_id"], - "job_code": job_summary["job_code"], - }, - ) - append_detect_job_event( - job_summary["job_id"], - event_type="job_dispatch_sent" if command_ok else "job_dispatch_rejected", - level="info" if command_ok else "error", - message=command_message, - payload={"cycle_token": cycle_token}, - ) snapshot = get_detect_status() settings_payload = get_settings_payload() settings_summary = _build_settings_summary(settings_payload) diff --git a/domain-api/app/api/routes/ops_agent.py b/domain-api/app/api/routes/ops_agent.py index 7d7607e..1e1537c 100644 --- a/domain-api/app/api/routes/ops_agent.py +++ b/domain-api/app/api/routes/ops_agent.py @@ -9,6 +9,7 @@ from app.services.ops_agent_service import ( agent_heartbeat, agent_mark_job_started, agent_pull_jobs, + agent_pull_runtime_config, agent_register, build_node_agent_bootstrap_plan, issue_node_agent_token, @@ -72,6 +73,12 @@ def ops_agent_pull(payload: dict, limit: int = 1, x_domaincheck_agent_token: Opt return _build_agent_response(ok, message, data) +@router.post("/ops/agent/runtime-config", response_model=ApiResponse) +def ops_agent_runtime_config(payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse: + ok, message, data = agent_pull_runtime_config(payload, token=_resolve_agent_token(x_domaincheck_agent_token)) + return _build_agent_response(ok, message, data) + + @router.post("/ops/agent/jobs/{job_id}/start", response_model=ApiResponse) def ops_agent_job_start(job_id: int, payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse: ok, message, data = agent_mark_job_started(job_id, payload, token=_resolve_agent_token(x_domaincheck_agent_token)) diff --git a/domain-api/app/api/routes/runtime.py b/domain-api/app/api/routes/runtime.py index f8bccac..ac400a5 100644 --- a/domain-api/app/api/routes/runtime.py +++ b/domain-api/app/api/routes/runtime.py @@ -1,11 +1,12 @@ from typing import Optional -from fastapi import APIRouter, Header +from fastapi import APIRouter, Body, Header from app.schemas.common import ApiResponse from app.services.build_info_service import get_runtime_build_info from app.services.cluster_runtime_service import get_cluster_snapshot from app.services.debug_event_service import get_debug_diagnosis, get_debug_event_overview, get_debug_handoff_report, ingest_debug_event, list_debug_events +from app.services.detect_job_service import get_detect_queue_health from app.services.runtime_control_service import runtime_action from app.services.runtime_status_service import get_runtime_preflight, get_runtime_status from app.services.sync_push_service import ( @@ -103,6 +104,36 @@ def runtime_debug_handoff(window_minutes: int = 10, source_region: Optional[str] ) +@router.get("/runtime/queue-health", response_model=ApiResponse) +def runtime_queue_health(window_minutes: int = 15) -> ApiResponse: + return ApiResponse(data=get_detect_queue_health(window_minutes=window_minutes)) + + +@router.get("/runtime/health-handover", response_model=ApiResponse) +def runtime_health_handover( + node_code: Optional[str] = None, + window_minutes: int = 10, + source_region: Optional[str] = None, +) -> ApiResponse: + sync_summary = get_sync_summary() + runtime_status_payload = get_runtime_status() + data = get_debug_handoff_report( + window_minutes=window_minutes, + source_region=source_region, + sync_summary=sync_summary, + readiness=runtime_status_payload.get("readiness") or {}, + ) + normalized_node_code = str(node_code or "").strip() + if normalized_node_code: + data = dict(data) + data["nodes"] = [ + item + for item in list(data.get("nodes") or []) + if str(item.get("node_code") or "").strip() == normalized_node_code + ] + return ApiResponse(data=data) + + @router.post("/runtime/sync-ingest", response_model=ApiResponse) def runtime_sync_ingest(payload: dict, x_domaincheck_sync_token: Optional[str] = Header(default=None)) -> ApiResponse: ok, message, data = ingest_runtime_projection(payload, shared_token=x_domaincheck_sync_token) @@ -110,7 +141,7 @@ def runtime_sync_ingest(payload: dict, x_domaincheck_sync_token: Optional[str] = @router.get("/runtime/task-export", response_model=ApiResponse) -def runtime_task_export(limit: int = 200, x_domaincheck_sync_token: Optional[str] = Header(default=None)) -> ApiResponse: +def runtime_task_export(limit: int = 1000, x_domaincheck_sync_token: Optional[str] = Header(default=None)) -> ApiResponse: ok, message, data = export_detect_task_projection(limit=limit, shared_token=x_domaincheck_sync_token) return ApiResponse(code=0 if ok else 1, message=message, data=data) @@ -128,6 +159,6 @@ def runtime_debug_ingest(payload: dict, x_domaincheck_sync_token: Optional[str] @router.post("/runtime/actions/{action}", response_model=ApiResponse) -def runtime_action_trigger(action: str) -> ApiResponse: - ok, message, data = runtime_action(action) +def runtime_action_trigger(action: str, payload: Optional[dict] = Body(default=None)) -> ApiResponse: + ok, message, data = runtime_action(action, payload=payload) return ApiResponse(code=0 if ok else 1, message=message, data=data) diff --git a/domain-api/app/core/config.py b/domain-api/app/core/config.py index a7f5083..f68f502 100644 --- a/domain-api/app/core/config.py +++ b/domain-api/app/core/config.py @@ -40,8 +40,12 @@ class Settings(BaseSettings): sync_target_region: str = "overseas" sync_target_api_base_url: str = "" sync_shared_token: str = "" - sync_batch_size: int = 200 - sync_poll_interval_seconds: int = 30 + sync_batch_size: int = 5000 + sync_poll_interval_seconds: int = 2 + sync_pipeline_process_limit: int = 5000 + sync_pull_max_pending_items: int = 0 + sync_pull_max_register_pending_items: int = 0 + sync_pull_max_downstream_pending_items: int = 0 build_manifest_path: str = "" build_commit_sha: str = "" build_commit_ref: str = "" diff --git a/domain-api/app/core/db.py b/domain-api/app/core/db.py index bce16b5..231d8ed 100644 --- a/domain-api/app/core/db.py +++ b/domain-api/app/core/db.py @@ -1,6 +1,9 @@ from contextlib import contextmanager +from functools import wraps +import time import psycopg2 +from psycopg2 import errors from app.core.config import settings @@ -18,3 +21,39 @@ def get_db(): yield conn finally: conn.close() + + +_RETRYABLE_READ_ERRORS = ( + errors.DeadlockDetected, + errors.SerializationFailure, + errors.LockNotAvailable, +) + + +def is_retryable_read_error(exc: Exception) -> bool: + return isinstance(exc, _RETRYABLE_READ_ERRORS) + + +def is_retryable_db_error(exc: Exception) -> bool: + return isinstance(exc, _RETRYABLE_READ_ERRORS) + + +def db_read_retry(*, attempts: int = 3, initial_delay_seconds: float = 0.05, backoff: float = 2.0): + def decorator(func): + @wraps(func) + def wrapper(*args, **kwargs): + delay = max(0.0, float(initial_delay_seconds or 0.0)) + for attempt in range(1, max(1, int(attempts or 1)) + 1): + try: + return func(*args, **kwargs) + except Exception as exc: + if not is_retryable_read_error(exc) or attempt >= max(1, int(attempts or 1)): + raise + if delay > 0: + time.sleep(delay) + delay *= max(1.0, float(backoff or 1.0)) + return func(*args, **kwargs) + + return wrapper + + return decorator diff --git a/domain-api/app/core/files.py b/domain-api/app/core/files.py index b180be7..4cd894c 100644 --- a/domain-api/app/core/files.py +++ b/domain-api/app/core/files.py @@ -1,6 +1,7 @@ from __future__ import annotations import json +import os from pathlib import Path from datetime import datetime @@ -63,9 +64,35 @@ def tail_lines(relative_path: str, max_lines: int = 120) -> list[str]: def runtime_root() -> Path: - path = Path(__file__).resolve().parents[2] / "runtime" - path.mkdir(parents=True, exist_ok=True) - return path + env_override = str(os.getenv("DOMAIN_API_RUNTIME_ROOT", "") or "").strip() + candidates: list[Path] = [] + if env_override: + candidates.append(Path(env_override)) + + base_dir = Path(__file__).resolve().parents[2] + for parent in base_dir.parents: + if parent.name != "releases": + continue + # Released builds live under /opt/domaincheck/releases//domain-api. + # Runtime state must not be written back into the immutable release tree, + # otherwise sync-agent / detect runtime snapshots fail with permission errors. + candidates.append(parent.parent / "runtime" / "domain-api") + break + + candidates.append(base_dir / "runtime") + + last_error: OSError | None = None + for candidate in candidates: + try: + candidate.mkdir(parents=True, exist_ok=True) + return candidate + except OSError as exc: + last_error = exc + continue + + if last_error is not None: + raise last_error + raise RuntimeError("failed to resolve runtime root") def read_runtime_json(filename: str, default: dict | list | None = None): @@ -164,7 +191,16 @@ def load_detect_records() -> list[dict]: try: with path.open("r", encoding="utf-8") as handle: return json.load(handle) - except (json.JSONDecodeError, OSError): + except (json.JSONDecodeError, UnicodeDecodeError, OSError): + try: + raw = path.read_bytes() + text = raw.decode("utf-8", errors="replace") + decoder = json.JSONDecoder() + payload, _ = decoder.raw_decode(text) + if isinstance(payload, list): + return payload + except Exception: + pass return [] diff --git a/domain-api/app/node_agent.py b/domain-api/app/node_agent.py index cd84ce6..c975344 100644 --- a/domain-api/app/node_agent.py +++ b/domain-api/app/node_agent.py @@ -1,13 +1,16 @@ from __future__ import annotations +import hashlib import json import os import socket import subprocess +import traceback import time import urllib.error import urllib.request from datetime import datetime +from urllib.parse import urlparse from uuid import uuid4 from app.services.ops_action_executor_core import ( @@ -28,6 +31,44 @@ NODE_CODE = str(os.getenv("NODE_CODE", "")).strip() NODE_REGION = str(os.getenv("NODE_REGION", "mainland")).strip() or "mainland" NODE_ROLE = str(os.getenv("NODE_ROLE", "worker")).strip() or "worker" AGENT_POLL_INTERVAL_SECONDS = max(2, int(os.getenv("OPS_AGENT_POLL_INTERVAL_SECONDS", "5") or 5)) +AGENT_RUNTIME_CONFIG_SYNC_INTERVAL_SECONDS = max( + 10, + int(os.getenv("OPS_AGENT_RUNTIME_CONFIG_SYNC_INTERVAL_SECONDS", "30") or 30), +) +AGENT_HTTP_TIMEOUT_SECONDS = max(10, int(os.getenv("OPS_AGENT_HTTP_TIMEOUT_SECONDS", "60") or 60)) +AGENT_REGISTER_TIMEOUT_SECONDS = max( + 10, + int(os.getenv("OPS_AGENT_REGISTER_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS)) or AGENT_HTTP_TIMEOUT_SECONDS), +) +AGENT_HEARTBEAT_TIMEOUT_SECONDS = max( + 10, + int(os.getenv("OPS_AGENT_HEARTBEAT_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS)) or AGENT_HTTP_TIMEOUT_SECONDS), +) +AGENT_PULL_TIMEOUT_SECONDS = max( + 10, + int(os.getenv("OPS_AGENT_PULL_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS)) or AGENT_HTTP_TIMEOUT_SECONDS), +) +AGENT_RUNTIME_CONFIG_TIMEOUT_SECONDS = max( + 10, + int( + os.getenv("OPS_AGENT_RUNTIME_CONFIG_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS)) + or AGENT_HTTP_TIMEOUT_SECONDS + ), +) +AGENT_JOB_COMPLETE_TIMEOUT_SECONDS = max( + 10, + int( + os.getenv("OPS_AGENT_JOB_COMPLETE_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS)) + or AGENT_HTTP_TIMEOUT_SECONDS + ), +) +AGENT_JOB_EVENT_TIMEOUT_SECONDS = max( + 10, + int( + os.getenv("OPS_AGENT_JOB_EVENT_TIMEOUT_SECONDS", str(max(15, AGENT_HTTP_TIMEOUT_SECONDS // 2))) + or max(15, AGENT_HTTP_TIMEOUT_SECONDS // 2) + ), +) WORKER_SERVICE_NAME = str(os.getenv("WORKER_SERVICE_NAME", os.getenv("WORKER_SERVICE", "domaincheck-worker"))).strip() or "domaincheck-worker" API_SERVICE_NAME = str(os.getenv("API_SERVICE_NAME", "domaincheck-api")).strip() or "domaincheck-api" SYNC_AGENT_SERVICE_NAME = str(os.getenv("SYNC_AGENT_SERVICE_NAME", "domaincheck-sync-agent")).strip() or "domaincheck-sync-agent" @@ -50,6 +91,7 @@ _LAST_QUEUE_FLUSH_SUMMARY = { "dead_letter": 0, "last_flush_at": "", } +_LAST_RUNTIME_CONFIG_HASH = "" def _normalize_text_list(raw_value: object) -> list[str]: @@ -98,6 +140,7 @@ AGENT_CAPABILITIES = _json_env( "runtime.restart_api", "runtime.start_sync_agent", "runtime.stop_sync_agent", + "runtime.reset_lab_state", "health.snapshot", "logs.collect", "diagnostics.collect", @@ -601,7 +644,7 @@ def _headers() -> dict[str, str]: } -def _request(method: str, path: str, payload: dict | None = None, timeout: int = 30) -> dict: +def _request(method: str, path: str, payload: dict | None = None, timeout: int = AGENT_HTTP_TIMEOUT_SECONDS) -> dict: if not CONTROL_PLANE_BASE_URL: raise RuntimeError("OPS_CONTROL_PLANE_BASE_URL 未配置") if not AGENT_TOKEN: @@ -614,22 +657,286 @@ def _request(method: str, path: str, payload: dict | None = None, timeout: int = return json.loads(body or "{}") -def _post(path: str, payload: dict, timeout: int = 30) -> dict: +def _post(path: str, payload: dict, timeout: int = AGENT_HTTP_TIMEOUT_SECONDS) -> dict: return _request("POST", path, payload, timeout=timeout) -def _hostname() -> str: +def _pull_runtime_config(timeout: int = AGENT_RUNTIME_CONFIG_TIMEOUT_SECONDS) -> dict: + response = _post("/api/v1/ops/agent/runtime-config", {"node_code": NODE_CODE}, timeout=timeout) + _ensure_ok_response(response, "agent runtime config pull failed") + data = response.get("data") or {} + return dict(data.get("bundle") or {}) + + +def _publish_local_config_update(config_type: str) -> None: try: - return socket.gethostname() + from app.core.redis_client import get_redis + + redis_client = get_redis() + redis_client.publish("domain_tool:config_update", str(config_type or "").strip() or "config") except Exception: - return "" + return + + +def _apply_runtime_config(bundle: dict) -> bool: + global _LAST_RUNTIME_CONFIG_HASH + + normalized_bundle = dict(bundle or {}) + bundle_hash = str(normalized_bundle.get("config_hash") or "").strip() + if not bundle_hash: + bundle_hash = hashlib.sha256( + json.dumps(normalized_bundle, ensure_ascii=False, sort_keys=True).encode("utf-8") + ).hexdigest() + if bundle_hash and bundle_hash == _LAST_RUNTIME_CONFIG_HASH: + return False + + from app.core.files import write_json + from app.services.runtime_settings_service import update_runtime_settings + from app.services.sensitive_words_service import save_sensitive_words_payload + + detect_options = dict(normalized_bundle.get("detect_options") or {}) + proxy_config = dict(normalized_bundle.get("proxy_config") or {}) + thread_count = int(normalized_bundle.get("thread_count", 2) or 2) + node_thread_counts = dict(normalized_bundle.get("node_thread_counts") or {}) + runtime_settings = dict(normalized_bundle.get("runtime_settings") or {}) + sensitive_words = dict(normalized_bundle.get("sensitive_words") or {}) + sensitive_words_text = str(sensitive_words.get("text") or "") + sensitive_word_items = list(sensitive_words.get("items") or []) + + write_json("detect_options.json", detect_options) + write_json("proxy_config.json", proxy_config) + write_json("thread_count.json", {"thread_count": str(thread_count)}) + write_json("node_thread_counts.json", node_thread_counts) + write_json("runtime_settings.json", runtime_settings) + write_json("runtime/runtime_settings.json", runtime_settings) + write_json( + "runtime/sensitive_words.json", + { + "items": sensitive_word_items, + "text": sensitive_words_text, + "total": int(sensitive_words.get("total", 0) or 0), + }, + ) + + try: + update_runtime_settings(runtime_settings) + except Exception as exc: + _log(f"runtime settings local api sync skipped: {exc}") + + try: + save_sensitive_words_payload({"text": sensitive_words_text}) + except Exception as exc: + _log(f"sensitive words db sync skipped: {exc}") + + try: + from app.core.redis_client import get_redis + + redis_client = get_redis() + redis_client.set("domain_tool:detect_options", json.dumps(detect_options, ensure_ascii=False)) + redis_client.set("domain_tool:proxy_config", json.dumps(proxy_config, ensure_ascii=False)) + redis_client.set("domain_tool:thread_count", thread_count) + redis_client.set("domain_tool:node_thread_counts", json.dumps(node_thread_counts, ensure_ascii=False)) + redis_client.set("domain_tool:runtime_settings", json.dumps(runtime_settings, ensure_ascii=False)) + redis_client.set("domain_tool:sensitive_words", json.dumps(sensitive_word_items, ensure_ascii=False)) + redis_client.publish("domain_tool:config_update", "detect_options") + redis_client.publish("domain_tool:config_update", "proxy_config") + redis_client.publish("domain_tool:config_update", "thread_count") + redis_client.publish("domain_tool:config_update", "node_thread_counts") + redis_client.publish("domain_tool:config_update", "runtime_settings") + redis_client.publish("domain_tool:config_update", "sensitive_words") + except Exception: + _publish_local_config_update("detect_options") + _publish_local_config_update("proxy_config") + _publish_local_config_update("thread_count") + _publish_local_config_update("node_thread_counts") + _publish_local_config_update("runtime_settings") + _publish_local_config_update("sensitive_words") + + _LAST_RUNTIME_CONFIG_HASH = bundle_hash + _log( + "runtime config applied: " + f"thread_count={thread_count} " + f"node_override={node_thread_counts.get(NODE_CODE)} " + f"sensitive_words={int(sensitive_words.get('total', 0) or 0)}" + ) + return True + + +def _hostname() -> str: + generic_values = {"localhost", "localhost.localdomain", "ip6-localhost", "localhost6"} + candidates: list[str] = [] + try: + candidates.append(socket.gethostname()) + except Exception: + pass + try: + candidates.append(socket.getfqdn()) + except Exception: + pass + try: + candidates.append(os.uname().nodename) + except Exception: + pass + candidates.append(str(os.getenv("HOSTNAME", "")).strip()) + + for value in candidates: + normalized = str(value or "").strip() + if not normalized: + continue + if normalized.lower() in generic_values: + continue + return normalized + return NODE_CODE or "" + + +def _control_plane_host_port() -> tuple[str, int]: + parsed = urlparse(CONTROL_PLANE_BASE_URL if "://" in CONTROL_PLANE_BASE_URL else f"http://{CONTROL_PLANE_BASE_URL}") + host = str(parsed.hostname or "").strip() + if not host: + return "", 0 + port = int(parsed.port or (443 if parsed.scheme == "https" else 80)) + return host, port + + +def _first_non_loopback_ip(values: list[str]) -> str: + for value in values: + normalized = str(value or "").strip() + if not normalized or normalized.startswith("127.") or normalized == "::1": + continue + return normalized + return "" def _ip() -> str: + explicit_ip = str(os.getenv("OPS_AGENT_IP", "")).strip() + if explicit_ip: + return explicit_ip + + host, port = _control_plane_host_port() + if host and port: + for family in (socket.AF_INET, socket.AF_INET6): + sock = None + try: + sock = socket.socket(family, socket.SOCK_DGRAM) + sock.connect((host, port)) + local_ip = str(sock.getsockname()[0] or "").strip() + if local_ip and not local_ip.startswith("127.") and local_ip != "::1": + return local_ip + except Exception: + pass + finally: + if sock is not None: + try: + sock.close() + except Exception: + pass + + hostname = _hostname() + if hostname: + try: + addrinfo = socket.getaddrinfo(hostname, None) + resolved_ips = [str(item[4][0] or "").strip() for item in addrinfo if item and len(item) >= 5] + best_ip = _first_non_loopback_ip(resolved_ips) + if best_ip: + return best_ip + except Exception: + pass + try: - return socket.gethostbyname(socket.gethostname()) + fallback_ip = str(socket.gethostbyname(socket.gethostname()) or "").strip() + if fallback_ip and not fallback_ip.startswith("127."): + return fallback_ip except Exception: - return "" + pass + return "" + + +def _detect_runtime_snapshot() -> dict: + worker_runtime = {} + try: + from app.services.worker_control_service import detect_worker_runtime + + worker_runtime = detect_worker_runtime() or {} + except Exception as exc: + worker_runtime = { + "running": False, + "process_count": 0, + "latest_start_time": "", + "message": "", + "error": str(exc), + } + + try: + from app.services.detect_service import get_detect_status + + detect_status = get_detect_status() or {} + runtime_state = detect_status.get("runtime_state") if isinstance(detect_status.get("runtime_state"), dict) else {} + active_job = detect_status.get("active_job") if isinstance(detect_status.get("active_job"), dict) else {} + + active_threads = max(0, int(detect_status.get("active_thread_count", 0) or 0)) + max_threads = max(0, int(detect_status.get("max_thread_count", 0) or 0)) + current_load = max( + active_threads, + int(active_job.get("items_running", 0) or 0), + ) + phase_label = str(detect_status.get("phase_label") or "").strip() + phase_detail = str( + detect_status.get("phase_detail") + or detect_status.get("recent_event") + or detect_status.get("worker_runtime_message") + or "" + ).strip() + + inferred_worker_online = bool( + worker_runtime.get("running", False) + or detect_status.get("worker_online", False) + or runtime_state.get("service_running", False) + or current_load > 0 + or active_threads > 0 + ) + return { + "worker_online": inferred_worker_online, + "service_running": bool( + runtime_state.get("service_running", False) + or worker_runtime.get("running", False) + or inferred_worker_online + ), + "detecting": bool( + detect_status.get("detecting", False) + or runtime_state.get("detecting", False) + or current_load > 0 + or active_threads > 0 + ), + "active_threads": active_threads, + "max_threads": max_threads, + "current_load": current_load, + "phase_label": phase_label, + "phase_detail": phase_detail, + "recent_warning": str(detect_status.get("recent_warning") or "").strip(), + "updated_at": str(runtime_state.get("updated_at") or "").strip(), + "detect_participating": bool( + detect_status.get("detect_participating", False) + or current_load > 0 + or active_threads > 0 + ), + } + except Exception as exc: + worker_online = bool(worker_runtime.get("running", False)) + worker_message = str(worker_runtime.get("message") or "").strip() + return { + "worker_online": worker_online, + "service_running": worker_online, + "detecting": False, + "active_threads": 0, + "max_threads": 0, + "current_load": 0, + "phase_label": "", + "phase_detail": worker_message, + "recent_warning": "", + "updated_at": str(worker_runtime.get("latest_start_time") or "").strip(), + "detect_participating": False, + "error": str(exc), + } def _base_payload() -> dict: @@ -651,12 +958,37 @@ def _base_payload() -> dict: "node_agent": NODE_AGENT_SERVICE_NAME, }, "delivery_queue": _delivery_queue_snapshot(), + "detect_runtime": _detect_runtime_snapshot(), }, } def _run(command: list[str], timeout: int = 60) -> tuple[int, str, str]: completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout) + normalized_command = [str(part or "").strip() for part in command] + combined_output = f"{completed.stdout or ''}\n{completed.stderr or ''}".lower() + needs_sudo_retry = ( + normalized_command + and normalized_command[0] == "systemctl" + and completed.returncode != 0 + and "sudo" not in normalized_command + and any( + marker in combined_output + for marker in ( + "interactive authentication required", + "authentication is required", + "authorization not available", + "polkit", + ) + ) + ) + if needs_sudo_retry: + completed = subprocess.run( + ["sudo", "-n", *normalized_command], + capture_output=True, + text=True, + timeout=timeout, + ) return completed.returncode, completed.stdout.strip(), completed.stderr.strip() @@ -727,18 +1059,22 @@ def _execute_action( def _register() -> None: - response = _post("/api/v1/ops/agent/register", _base_payload()) + response = _post("/api/v1/ops/agent/register", _base_payload(), timeout=AGENT_REGISTER_TIMEOUT_SECONDS) _ensure_ok_response(response, "agent register failed") _log(f"registered: {response.get('message')}") def _heartbeat() -> None: - response = _post("/api/v1/ops/agent/heartbeat", _base_payload()) + response = _post("/api/v1/ops/agent/heartbeat", _base_payload(), timeout=AGENT_HEARTBEAT_TIMEOUT_SECONDS) _ensure_ok_response(response, "agent heartbeat failed") def _pull_jobs() -> list[dict]: - response = _post(f"/api/v1/ops/agent/pull?limit=1", {"node_code": NODE_CODE}) + response = _post( + f"/api/v1/ops/agent/pull?limit=1", + {"node_code": NODE_CODE}, + timeout=AGENT_PULL_TIMEOUT_SECONDS, + ) _ensure_ok_response(response, "agent pull failed") data = response.get("data") or {} return list(data.get("jobs") or []) @@ -847,7 +1183,7 @@ def _job_complete( path=f"/api/v1/ops/agent/jobs/{job_id}/complete", payload=payload, request_id=request_id, - timeout=30, + timeout=AGENT_JOB_COMPLETE_TIMEOUT_SECONDS, ) @@ -883,7 +1219,7 @@ def _job_event( path=f"/api/v1/ops/agent/jobs/{job_id}/events", payload=delivery_payload, request_id=request_id, - timeout=15, + timeout=AGENT_JOB_EVENT_TIMEOUT_SECONDS, ) @@ -931,7 +1267,22 @@ def _process_job(job: dict) -> None: "start_delivery_error": start_delivery_error, }, ) - ok, message, result = _execute_action(action, payload, job_id=job_id, job_context=normalized_job) + try: + ok, message, result = _execute_action(action, payload, job_id=job_id, job_context=normalized_job) + except Exception as exc: + ok = False + message = f"executor exception: {exc}" + result = { + "stdout": "", + "stderr": traceback.format_exc(), + "summary_text": message, + "exception_type": exc.__class__.__name__, + } + _log( + "job execute exception: " + f"id={job_id} code={normalized_job.get('job_code') or '-'} " + f"action={action} error={exc}" + ) stdout = str(result.get("stdout") or "") stderr = str(result.get("stderr") or "") duration_ms = max(0, int((time.monotonic() - started_at) * 1000)) @@ -956,6 +1307,57 @@ def _process_job(job: dict) -> None: ) +def _process_job_with_guard(job: dict) -> None: + normalized_job = _normalize_agent_job(job) + job_id = int(normalized_job.get("job_id") or 0) + action = str(normalized_job.get("action") or "").strip() + started_at = time.monotonic() + try: + _process_job(normalized_job) + except Exception as exc: + duration_ms = max(0, int((time.monotonic() - started_at) * 1000)) + traceback_text = traceback.format_exc() + summary_text = f"node-agent fatal exception: {exc}" + _log( + "job fatal exception: " + f"id={job_id} code={normalized_job.get('job_code') or '-'} " + f"action={action or '-'} error={exc}" + ) + if job_id <= 0: + raise + try: + delivery = _job_complete( + job_id, + status="failed", + stdout="", + stderr=traceback_text, + result={ + "stdout": "", + "stderr": traceback_text, + "summary_text": summary_text, + "exception_type": exc.__class__.__name__, + "fatal_loop_exception": True, + }, + error_message=summary_text, + duration_ms=duration_ms, + summary_text=summary_text, + focus_ref=dict(normalized_job.get("focus_ref") or {}), + step_ref=dict(normalized_job.get("step_ref") or {}), + release_context=dict(normalized_job.get("release_context") or {}), + ) + _log( + "job fatal exception completion: " + f"id={job_id} code={normalized_job.get('job_code') or '-'} " + f"action={action or '-'} delivery={delivery.get('state')}" + ) + except Exception as completion_exc: + _log( + "job fatal exception completion failed: " + f"id={job_id} code={normalized_job.get('job_code') or '-'} " + f"action={action or '-'} error={completion_exc}" + ) + + def main() -> None: if not NODE_CODE: raise RuntimeError("NODE_CODE 未配置") @@ -963,6 +1365,7 @@ def main() -> None: _ensure_queue_dirs() _register() last_heartbeat_at = 0.0 + last_runtime_config_sync_at = 0.0 while True: now = time.time() @@ -970,13 +1373,17 @@ def main() -> None: delivery_summary = _flush_delivery_queue(limit=AGENT_QUEUE_FLUSH_LIMIT) if delivery_summary["delivered"] or delivery_summary["dead_letter"]: _log(f"delivery queue flush: {delivery_summary}") + if now - last_runtime_config_sync_at >= AGENT_RUNTIME_CONFIG_SYNC_INTERVAL_SECONDS: + bundle = _pull_runtime_config() + _apply_runtime_config(bundle) + last_runtime_config_sync_at = now if now - last_heartbeat_at >= 15: _heartbeat() last_heartbeat_at = now jobs = _pull_jobs() if jobs: for job in jobs: - _process_job(job) + _process_job_with_guard(job) else: time.sleep(AGENT_POLL_INTERVAL_SECONDS) except urllib.error.HTTPError as exc: diff --git a/domain-api/app/services/cluster_runtime_service.py b/domain-api/app/services/cluster_runtime_service.py index 6bf6d71..72700d3 100644 --- a/domain-api/app/services/cluster_runtime_service.py +++ b/domain-api/app/services/cluster_runtime_service.py @@ -2,10 +2,11 @@ from __future__ import annotations import json import socket +import threading from datetime import datetime, timedelta from app.core.config import settings -from app.core.db import get_db +from app.core.db import db_read_retry, get_db _RUNTIME_SCHEMA_SQL = """ @@ -29,6 +30,8 @@ CREATE TABLE IF NOT EXISTS detect_jobs ( job_code VARCHAR(64) NOT NULL UNIQUE, source VARCHAR(64) NOT NULL DEFAULT 'manual', plan_hash VARCHAR(128) NOT NULL DEFAULT '', + task_mode VARCHAR(32) NOT NULL DEFAULT 'domain_pipeline', + step_code VARCHAR(64) NOT NULL DEFAULT '', status VARCHAR(32) NOT NULL DEFAULT 'pending', remark TEXT NOT NULL DEFAULT '', created_by VARCHAR(64) NOT NULL DEFAULT '', @@ -41,6 +44,7 @@ CREATE TABLE IF NOT EXISTS detect_job_items ( id BIGSERIAL PRIMARY KEY, job_id BIGINT NOT NULL REFERENCES detect_jobs(id) ON DELETE CASCADE, domain_id BIGINT NOT NULL, + step_code VARCHAR(64) NOT NULL DEFAULT '', status VARCHAR(32) NOT NULL DEFAULT 'pending', claimed_by VARCHAR(64) NOT NULL DEFAULT '', claim_token VARCHAR(64) NOT NULL DEFAULT '', @@ -48,16 +52,32 @@ CREATE TABLE IF NOT EXISTS detect_job_items ( attempt_count INTEGER NOT NULL DEFAULT 0, last_error TEXT NOT NULL DEFAULT '', result_version VARCHAR(64) NOT NULL DEFAULT '', + step_payload_json JSONB, + result_payload_json JSONB, started_at TIMESTAMP, finished_at TIMESTAMP, updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT uq_detect_job_items_job_domain UNIQUE (job_id, domain_id) + create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ); CREATE INDEX IF NOT EXISTS idx_detect_job_items_status_lease ON detect_job_items(status, lease_expires_at); +ALTER TABLE detect_jobs + ADD COLUMN IF NOT EXISTS task_mode VARCHAR(32) NOT NULL DEFAULT 'domain_pipeline', + ADD COLUMN IF NOT EXISTS step_code VARCHAR(64) NOT NULL DEFAULT ''; + +ALTER TABLE detect_job_items + ADD COLUMN IF NOT EXISTS step_code VARCHAR(64) NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS step_payload_json JSONB, + ADD COLUMN IF NOT EXISTS result_payload_json JSONB; + +ALTER TABLE detect_job_items + DROP CONSTRAINT IF EXISTS uq_detect_job_items_job_domain; + +CREATE UNIQUE INDEX IF NOT EXISTS idx_detect_job_items_job_domain_step +ON detect_job_items(job_id, domain_id, step_code); + CREATE TABLE IF NOT EXISTS detect_run_events ( id BIGSERIAL PRIMARY KEY, job_id BIGINT REFERENCES detect_jobs(id) ON DELETE SET NULL, @@ -90,6 +110,9 @@ _STALE_AFTER_SECONDS = 90 _OFFLINE_AFTER_MINUTES = 5 _PRUNE_IMPORTED_AFTER_MINUTES = 30 _PRUNE_GENERAL_AFTER_HOURS = 6 +_RUNTIME_SCHEMA_READY = False +_RUNTIME_SCHEMA_LOCK = threading.Lock() +_RUNTIME_SCHEMA_ADVISORY_LOCK_ID = 62021001 def _resolve_local_ip() -> str: @@ -110,12 +133,85 @@ def _decode_json(value: object) -> dict: return {} +def _control_node_supports_worker(*, region: object, metadata: dict | None) -> bool: + normalized_region = str(region or "").strip() + runtime_metadata = dict(metadata or {}) + active_threads = int(runtime_metadata.get("active_threads", 0) or 0) + max_threads = int(runtime_metadata.get("max_threads", 0) or 0) + if normalized_region != "mainland": + return False + return bool( + runtime_metadata.get("worker_online", False) + or runtime_metadata.get("detect_participating", False) + or active_threads > 0 + or max_threads > 0 + ) + + +def _metadata_idle_without_runtime_work(metadata: dict | None) -> bool: + runtime_metadata = dict(metadata or {}) + phase = str( + runtime_metadata.get("phase") + or runtime_metadata.get("phase_label") + or "" + ).strip().lower() + if phase not in {"idle", "completed", "stopped"}: + return False + + active_job_code = str(runtime_metadata.get("active_job_code") or "").strip() + counters = ( + int(runtime_metadata.get("job_items_total", 0) or 0), + int(runtime_metadata.get("job_items_claimed", 0) or 0), + int(runtime_metadata.get("job_items_running", 0) or 0), + int(runtime_metadata.get("job_items_completed", 0) or 0), + int(runtime_metadata.get("job_items_failed", 0) or 0), + ) + if active_job_code: + return False + return not any(value > 0 for value in counters) + + +def _load_managed_node_overlays() -> dict[str, dict]: + try: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT node_code, metadata_json, last_seen_at + FROM ops_managed_nodes + WHERE is_enabled = TRUE + """ + ) + rows = cur.fetchall() + except Exception: + return {} + + overlays: dict[str, dict] = {} + for row in rows: + node_code = str(row[0] or "").strip() + if not node_code: + continue + overlays[node_code] = { + "metadata": _decode_json(row[1]), + "last_seen_at": row[2], + } + return overlays + + def ensure_runtime_schema() -> None: - with get_db() as conn: - conn.autocommit = False - with conn.cursor() as cur: - cur.execute(_RUNTIME_SCHEMA_SQL) - conn.commit() + global _RUNTIME_SCHEMA_READY + if _RUNTIME_SCHEMA_READY: + return + with _RUNTIME_SCHEMA_LOCK: + if _RUNTIME_SCHEMA_READY: + return + with get_db() as conn: + conn.autocommit = False + with conn.cursor() as cur: + cur.execute("SELECT pg_advisory_xact_lock(%s)", (_RUNTIME_SCHEMA_ADVISORY_LOCK_ID,)) + cur.execute(_RUNTIME_SCHEMA_SQL) + conn.commit() + _RUNTIME_SCHEMA_READY = True def register_node_heartbeat( @@ -195,6 +291,43 @@ def cleanup_imported_runtime_nodes(*, region: str, role: str, keep_node_code: st conn.commit() +def cleanup_imported_runtime_nodes_many(*, region: str, role: str, keep_node_codes: list[str] | tuple[str, ...] | set[str]) -> None: + normalized_region = str(region or "").strip() or "unknown" + normalized_role = str(role or "").strip() or "unknown" + preserved_node_codes = sorted( + { + str(node_code or "").strip() + for node_code in (keep_node_codes or []) + if str(node_code or "").strip() + } + ) + if not preserved_node_codes: + return + + placeholders = ", ".join(["%s"] * len(preserved_node_codes)) + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + f""" + DELETE FROM detect_worker_nodes + WHERE region = %s + AND role = %s + AND ( + node_code = %s + OR (metadata_json->>'service') = 'runtime-ingest' + ) + AND node_code NOT IN ({placeholders}) + """, + ( + normalized_region, + normalized_role, + f"{normalized_region}-{normalized_role}-imported", + *preserved_node_codes, + ), + ) + conn.commit() + + def prune_expired_runtime_nodes() -> None: imported_cutoff = datetime.now() - timedelta(minutes=_PRUNE_IMPORTED_AFTER_MINUTES) general_cutoff = datetime.now() - timedelta(hours=_PRUNE_GENERAL_AFTER_HOURS) @@ -219,10 +352,12 @@ def prune_expired_runtime_nodes() -> None: def register_local_control_heartbeat() -> None: from app.services.detect_job_service import get_active_detect_job_summary + from app.services.detect_service import get_detect_status from app.services.worker_control_service import detect_worker_runtime worker_runtime = detect_worker_runtime() worker_online = bool(worker_runtime.get("running", False)) + detect_status = get_detect_status() active_job = get_active_detect_job_summary(event_limit=5) or {} node_stats = list(active_job.get("node_stats") or []) local_bucket = next( @@ -233,7 +368,9 @@ def register_local_control_heartbeat() -> None: items_claimed = int(local_bucket.get("items_claimed", 0) or 0) items_running = int(local_bucket.get("items_running", 0) or 0) items_completed = int(local_bucket.get("items_completed", 0) or 0) - current_load = max(items_running, items_claimed, 0) + active_threads = int(detect_status.get("active_thread_count", 0) or 0) + max_threads = int(detect_status.get("max_thread_count", 0) or 0) + current_load = max(items_running, active_threads, 0) detect_participating = bool(worker_online and (items_total > 0 or current_load > 0)) node_status = "busy" if current_load > 0 else "online" register_node_heartbeat( @@ -256,6 +393,10 @@ def register_local_control_heartbeat() -> None: "job_items_claimed": items_claimed, "job_items_running": items_running, "job_items_completed": items_completed, + "active_threads": active_threads, + "max_threads": max_threads, + "phase_label": str(detect_status.get("phase_label") or ""), + "phase_detail": str(detect_status.get("phase_detail") or ""), "updated_at": datetime.now().isoformat(timespec="seconds"), }, ) @@ -274,9 +415,11 @@ def _normalize_node_status(raw_status: str, last_heartbeat_at: datetime | None) return status +@db_read_retry() def get_cluster_snapshot() -> dict: prune_expired_runtime_nodes() register_local_control_heartbeat() + managed_overlays = _load_managed_node_overlays() with get_db() as conn: with conn.cursor() as cur: cur.execute( @@ -297,21 +440,50 @@ def get_cluster_snapshot() -> dict: cur.execute("SELECT count(*) FROM detect_job_items WHERE status IN ('pending', 'claimed', 'running')") active_items = cur.fetchone()[0] - nodes = [ - { - "node_code": row[0], - "region": row[1], - "role": row[2], - "hostname": row[3], - "ip": row[4], - "status": _normalize_node_status(row[5], row[9]), - "worker_version": row[6], - "current_load": row[7], - "metadata": _decode_json(row[8]), - "last_heartbeat_at": row[9].isoformat(sep=" ", timespec="seconds") if row[9] else "", - } - for row in rows - ] + nodes = [] + for row in rows: + node_code = str(row[0] or "").strip() + metadata = _decode_json(row[8]) + current_load = int(row[7] or 0) + sanitized_idle_runtime = _metadata_idle_without_runtime_work(metadata) + if sanitized_idle_runtime: + current_load = 0 + metadata["active_threads"] = 0 + metadata["detect_participating"] = False + metadata["sanitized_runtime_state"] = "idle_phase_zeroed" + runtime_last_heartbeat = row[9] + managed_overlay = managed_overlays.get(node_code) or {} + managed_last_seen = managed_overlay.get("last_seen_at") + overlay_is_newer = bool( + managed_last_seen + and (not runtime_last_heartbeat or managed_last_seen > runtime_last_heartbeat) + ) + effective_last_heartbeat = managed_last_seen if overlay_is_newer else runtime_last_heartbeat + normalized_status = _normalize_node_status(row[5], effective_last_heartbeat) + if sanitized_idle_runtime and normalized_status == "busy": + normalized_status = "online" + if overlay_is_newer and normalized_status in {"offline", "stale"}: + normalized_status = "busy" if current_load > 0 else "online" + if managed_last_seen: + metadata["agent_last_seen_at"] = managed_last_seen.isoformat(sep=" ", timespec="seconds") + if overlay_is_newer: + metadata["cluster_status_source"] = "managed-agent-overlay" + nodes.append( + { + "node_code": node_code, + "region": row[1], + "role": row[2], + "hostname": row[3], + "ip": row[4], + "status": normalized_status, + "worker_version": row[6], + "current_load": current_load, + "metadata": metadata, + "last_heartbeat_at": effective_last_heartbeat.isoformat(sep=" ", timespec="seconds") + if effective_last_heartbeat + else "", + } + ) status_counts: dict[str, int] = {} role_counts: dict[str, int] = {} region_counts: dict[str, int] = {} @@ -330,6 +502,12 @@ def get_cluster_snapshot() -> dict: metadata = node.get("metadata") or {} node_current_load = int(node.get("current_load", 0) or 0) effective_worker = False + if node_role == "control" and not _control_node_supports_worker(region=node_region, metadata=metadata): + node_current_load = 0 + node["current_load"] = 0 + if node_status == "busy": + node_status = "online" + node["status"] = "online" status_counts[node_status] = status_counts.get(node_status, 0) + 1 role_counts[node_role] = role_counts.get(node_role, 0) + 1 @@ -345,14 +523,16 @@ def get_cluster_snapshot() -> dict: dedicated_online_worker_nodes += 1 effective_worker = True elif node_role == "control" and node_status in {"online", "busy"}: - if bool(metadata.get("worker_online", False)) or bool(metadata.get("detect_participating", False)) or node_current_load > 0: + if _control_node_supports_worker(region=node_region, metadata=metadata): effective_worker = True if effective_worker: online_worker_nodes += 1 if node_role == "control" and node_status in {"online", "busy"}: online_control_nodes += 1 node["is_effective_worker"] = effective_worker - node["detect_participating"] = bool(metadata.get("detect_participating", False) or node_current_load > 0) + node["detect_participating"] = bool( + effective_worker and (metadata.get("detect_participating", False) or node_current_load > 0) + ) return { "nodes": nodes, diff --git a/domain-api/app/services/dashboard.py b/domain-api/app/services/dashboard.py index 9acf6c2..b3074ef 100644 --- a/domain-api/app/services/dashboard.py +++ b/domain-api/app/services/dashboard.py @@ -1,10 +1,390 @@ from __future__ import annotations from app.core.db import get_db +from app.services.detect_job_service import ( + _build_step_bucket, + get_active_detect_job_summary, + get_detect_capacity_plan, + get_detect_queue_health, +) from app.services.runtime_status_service import get_runtime_status +def _empty_active_jobs_aggregate(window_minutes: int) -> dict: + return { + "window_minutes": int(window_minutes or 15), + "active_jobs_total": 0, + "queue": { + "items_total": 0, + "pending": 0, + "claimed": 0, + "running": 0, + "completed": 0, + "blacklisted": 0, + "failed": 0, + "terminal": 0, + }, + "throughput": { + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + }, + "steps": [], + "nodes": [], + "retry_total": 0, + } + + +def _merge_step_queues_with_runtime_activity( + base_steps: list[dict] | None, + *, + runtime_activity: dict | None = None, + window_minutes: int = 15, + limit: int = 8, +) -> list[dict]: + safe_window_minutes = max(1, int(window_minutes or 15)) + normalized_limit = max(1, int(limit or 8)) + step_map: dict[str, dict] = {} + + for item in list(base_steps or []): + step_code = str(item.get("step_code") or "").strip() + if not step_code: + continue + bucket = _build_step_bucket(step_code) + bucket.update( + { + "items_total": int(item.get("items_total", 0) or 0), + "items_pending": int(item.get("items_pending", 0) or 0), + "items_claimed": int(item.get("items_claimed", 0) or 0), + "items_running": int(item.get("items_running", 0) or 0), + "items_completed": int(item.get("items_completed", 0) or 0), + "items_blacklisted": int(item.get("items_blacklisted", 0) or 0), + "items_failed": int(item.get("items_failed", 0) or 0), + "started_recent": int(item.get("started_recent", 0) or 0), + "processed_recent": int(item.get("processed_recent", 0) or 0), + "processed_per_minute": float(item.get("processed_per_minute", 0) or 0), + "completed_recent": int(item.get("completed_recent", 0) or 0), + "blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0), + "failed_recent": int(item.get("failed_recent", 0) or 0), + } + ) + step_map[step_code] = bucket + + runtime_step_stats = dict((runtime_activity or {}).get("step_stats") or {}) + for step_code, stats in runtime_step_stats.items(): + normalized_step_code = str(step_code or "").strip() + if not normalized_step_code: + continue + bucket = step_map.setdefault(normalized_step_code, _build_step_bucket(normalized_step_code)) + started_recent = int((stats or {}).get("started_recent", 0) or 0) + processed_recent = int((stats or {}).get("processed_recent", 0) or 0) + completed_recent = int((stats or {}).get("completed_recent", 0) or 0) + blacklisted_recent = int((stats or {}).get("blacklisted_recent", 0) or 0) + failed_recent = int((stats or {}).get("failed_recent", 0) or 0) + bucket["started_recent"] = max(int(bucket.get("started_recent", 0) or 0), started_recent) + bucket["processed_recent"] = max(int(bucket.get("processed_recent", 0) or 0), processed_recent) + bucket["completed_recent"] = max(int(bucket.get("completed_recent", 0) or 0), completed_recent) + bucket["blacklisted_recent"] = max(int(bucket.get("blacklisted_recent", 0) or 0), blacklisted_recent) + bucket["failed_recent"] = max(int(bucket.get("failed_recent", 0) or 0), failed_recent) + bucket["processed_per_minute"] = max( + float(bucket.get("processed_per_minute", 0) or 0), + round(processed_recent / safe_window_minutes, 2), + ) + + return sorted( + step_map.values(), + key=lambda item: ( + -int(item.get("items_pending", 0) or 0), + -int(item.get("items_running", 0) or 0), + -int(item.get("started_recent", 0) or 0), + -int(item.get("processed_recent", 0) or 0), + str(item.get("step_code") or ""), + ), + )[:normalized_limit] + + +def _align_active_jobs_aggregate_with_runtime( + aggregate: dict, + *, + runtime: dict, + queue_health: dict, +) -> dict: + normalized = dict(aggregate or {}) + node_payload = dict((runtime or {}).get("node") or {}) + if str(node_payload.get("region") or "").strip() != "overseas" or str(node_payload.get("role") or "").strip() != "control": + return normalized + + backlog = dict(((runtime or {}).get("detect") or {}).get("backlog") or {}) + snapshot_backlog = dict(queue_health.get("runtime_snapshot_backlog") or {}) + + def _backlog_value(key: str) -> int: + return max(int(backlog.get(key, 0) or 0), int(snapshot_backlog.get(key, 0) or 0)) + + pending_total = _backlog_value("pending_total") + claimed_total = _backlog_value("claimed_total") + running_total = _backlog_value("running_total") + completed_total = _backlog_value("completed_total") + blacklisted_total = _backlog_value("blacklisted_total") + failed_total = _backlog_value("failed_total") + + queue = dict(queue_health.get("queue") or {}) + throughput = dict(queue_health.get("throughput") or {}) + terminal_total = max( + completed_total + blacklisted_total + failed_total, + int(queue.get("completed", 0) or 0) + int(queue.get("blacklisted", 0) or 0) + int(queue.get("failed", 0) or 0), + ) + items_total = pending_total + claimed_total + running_total + terminal_total + has_runtime_work = items_total > 0 or bool(queue_health.get("has_active_job")) + + normalized["active_jobs_total"] = max( + int(normalized.get("active_jobs_total", 0) or 0), + 1 if has_runtime_work else 0, + ) + normalized["queue"] = { + "items_total": items_total, + "pending": pending_total, + "claimed": claimed_total, + "running": running_total, + "completed": max(completed_total, int(queue.get("completed", 0) or 0)), + "blacklisted": max(blacklisted_total, int(queue.get("blacklisted", 0) or 0)), + "failed": max(failed_total, int(queue.get("failed", 0) or 0)), + "terminal": terminal_total, + } + normalized["throughput"] = { + "processed_recent": int(throughput.get("processed_recent", 0) or 0), + "processed_per_minute": float(throughput.get("processed_per_minute", 0) or 0), + "completed_recent": int(throughput.get("completed_recent", 0) or 0), + "blacklisted_recent": int(throughput.get("blacklisted_recent", 0) or 0), + "failed_recent": int(throughput.get("failed_recent", 0) or 0), + } + normalized["steps"] = _merge_step_queues_with_runtime_activity( + list(queue_health.get("steps") or []), + runtime_activity=dict(queue_health.get("runtime_activity") or {}), + window_minutes=int(queue_health.get("window_minutes", normalized.get("window_minutes", 15)) or 15), + limit=8, + ) + normalized["nodes"] = list(queue_health.get("nodes") or []) + return normalized + + +def _fetch_active_jobs_aggregate(window_minutes: int = 15) -> dict: + safe_window_minutes = max(5, min(int(window_minutes or 15), 120)) + payload = _empty_active_jobs_aggregate(safe_window_minutes) + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id + FROM detect_jobs + WHERE status IN ('pending', 'running', 'partial_failed') + ORDER BY + CASE + WHEN status = 'running' THEN 0 + WHEN status = 'pending' THEN 1 + WHEN status = 'partial_failed' THEN 2 + ELSE 3 + END ASC, + COALESCE(started_at, created_at) DESC, + id DESC + """ + ) + job_ids = [int(row[0]) for row in cur.fetchall() if row and row[0] is not None] + if not job_ids: + return payload + + payload["active_jobs_total"] = len(job_ids) + + cur.execute( + """ + SELECT + COUNT(*) AS items_total, + COUNT(*) FILTER (WHERE status = 'pending') AS items_pending, + COUNT(*) FILTER (WHERE status = 'claimed') AS items_claimed, + COUNT(*) FILTER (WHERE status = 'running') AS items_running, + COUNT(*) FILTER (WHERE status = 'completed') AS items_completed, + COUNT(*) FILTER (WHERE status = 'blacklisted') AS items_blacklisted, + COUNT(*) FILTER (WHERE status = 'failed') AS items_failed + FROM detect_job_items + WHERE job_id = ANY(%s) + """, + (job_ids,), + ) + queue_row = cur.fetchone() or (0, 0, 0, 0, 0, 0, 0) + payload["queue"] = { + "items_total": int(queue_row[0] or 0), + "pending": int(queue_row[1] or 0), + "claimed": int(queue_row[2] or 0), + "running": int(queue_row[3] or 0), + "completed": int(queue_row[4] or 0), + "blacklisted": int(queue_row[5] or 0), + "failed": int(queue_row[6] or 0), + "terminal": int(queue_row[4] or 0) + int(queue_row[5] or 0) + int(queue_row[6] or 0), + } + + cur.execute( + """ + SELECT + COUNT(*) AS processed_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_completed') AS completed_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_blacklisted') AS blacklisted_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_failed') AS failed_recent + FROM detect_run_events + WHERE job_id = ANY(%s) + AND event_type IN ('domain_completed', 'domain_blacklisted', 'domain_failed') + AND created_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + """, + (job_ids, safe_window_minutes), + ) + throughput_row = cur.fetchone() or (0, 0, 0, 0) + processed_recent = int(throughput_row[0] or 0) + payload["throughput"] = { + "processed_recent": processed_recent, + "processed_per_minute": round(processed_recent / safe_window_minutes, 2), + "completed_recent": int(throughput_row[1] or 0), + "blacklisted_recent": int(throughput_row[2] or 0), + "failed_recent": int(throughput_row[3] or 0), + } + + cur.execute( + """ + SELECT COUNT(*) + FROM detect_job_items + WHERE job_id = ANY(%s) + AND attempt_count > 1 + """, + (job_ids,), + ) + payload["retry_total"] = int((cur.fetchone() or [0])[0] or 0) + + cur.execute( + """ + SELECT + COALESCE(NULLIF(step_code, ''), 'domain_pipeline') AS step_code, + COUNT(*) AS items_total, + COUNT(*) FILTER (WHERE status = 'pending') AS items_pending, + COUNT(*) FILTER (WHERE status = 'claimed') AS items_claimed, + COUNT(*) FILTER (WHERE status = 'running') AS items_running, + COUNT(*) FILTER (WHERE status = 'completed') AS items_completed, + COUNT(*) FILTER (WHERE status = 'blacklisted') AS items_blacklisted, + COUNT(*) FILTER (WHERE status = 'failed') AS items_failed, + COUNT(*) FILTER ( + WHERE status IN ('completed', 'blacklisted', 'failed') + AND finished_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + ) AS processed_recent + FROM detect_job_items + WHERE job_id = ANY(%s) + GROUP BY COALESCE(NULLIF(step_code, ''), 'domain_pipeline') + ORDER BY + COUNT(*) FILTER (WHERE status = 'pending') DESC, + COUNT(*) FILTER (WHERE status = 'running') DESC, + COUNT(*) DESC, + COALESCE(NULLIF(step_code, ''), 'domain_pipeline') ASC + LIMIT 8 + """, + (safe_window_minutes, job_ids), + ) + steps: list[dict] = [] + for row in cur.fetchall(): + step_code = str(row[0] or "domain_pipeline") + bucket = _build_step_bucket(step_code) + bucket.update( + { + "items_total": int(row[1] or 0), + "items_pending": int(row[2] or 0), + "items_claimed": int(row[3] or 0), + "items_running": int(row[4] or 0), + "items_completed": int(row[5] or 0), + "items_blacklisted": int(row[6] or 0), + "items_failed": int(row[7] or 0), + "processed_recent": int(row[8] or 0), + "processed_per_minute": round(int(row[8] or 0) / safe_window_minutes, 2), + } + ) + steps.append(bucket) + payload["steps"] = steps + + cur.execute( + """ + SELECT + COALESCE(NULLIF(claimed_by, ''), 'unassigned') AS node_code, + COUNT(*) FILTER (WHERE status = 'claimed') AS items_claimed, + COUNT(*) FILTER (WHERE status = 'running') AS items_running, + COUNT(*) FILTER (WHERE status = 'completed') AS items_completed_total + FROM detect_job_items + WHERE job_id = ANY(%s) + GROUP BY COALESCE(NULLIF(claimed_by, ''), 'unassigned') + """, + (job_ids,), + ) + node_map = { + str(row[0] or "unassigned"): { + "node_code": str(row[0] or "unassigned"), + "items_running": int(row[2] or 0), + "items_claimed": int(row[1] or 0), + "items_completed": int(row[3] or 0), + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "failed_recent": 0, + "blacklisted_recent": 0, + } + for row in cur.fetchall() + } + cur.execute( + """ + SELECT + COALESCE(NULLIF(node_code, ''), 'unassigned') AS node_code, + COUNT(*) AS processed_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_completed') AS completed_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_blacklisted') AS blacklisted_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_failed') AS failed_recent + FROM detect_run_events + WHERE job_id = ANY(%s) + AND event_type IN ('domain_completed', 'domain_blacklisted', 'domain_failed') + AND created_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + GROUP BY COALESCE(NULLIF(node_code, ''), 'unassigned') + """, + (job_ids, safe_window_minutes), + ) + for row in cur.fetchall(): + node_code = str(row[0] or "unassigned") + bucket = node_map.setdefault( + node_code, + { + "node_code": node_code, + "items_running": 0, + "items_claimed": 0, + "items_completed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "failed_recent": 0, + "blacklisted_recent": 0, + }, + ) + processed_recent = int(row[1] or 0) + bucket["processed_recent"] = processed_recent + bucket["processed_per_minute"] = round(processed_recent / safe_window_minutes, 2) + bucket["completed_recent"] = int(row[2] or 0) + bucket["blacklisted_recent"] = int(row[3] or 0) + bucket["failed_recent"] = int(row[4] or 0) + + payload["nodes"] = sorted( + node_map.values(), + key=lambda item: ( + -int(item.get("items_running", 0) or 0), + -int(item.get("processed_recent", 0) or 0), + -int(item.get("items_claimed", 0) or 0), + str(item.get("node_code") or ""), + ), + )[:8] + return payload + + def fetch_overview() -> dict: + window_minutes = 15 queries = { "domains_total": "select count(*) from domains", "pending_total": "select count(*) from domains where detect_status = 0", @@ -12,6 +392,8 @@ def fetch_overview() -> dict: "running_total": "select count(*) from domains where detect_status = 2", "blacklist_total": "select count(*) from domains where detect_status = 3", "failed_total": "select count(*) from domains where detect_status = 4", + "registerable_total": "select count(*) from domains where detect_status = 1 and register_status = 2", + "purchasable_total": "select count(*) from domains where detect_status = 1 and register_status = 2 and coalesce(use_status, 0) = 0", "sensitive_words_total": "select count(*) from sensitive_words", } result: dict[str, int | str] = {} @@ -23,6 +405,11 @@ def fetch_overview() -> dict: result[key] = cur.fetchone()[0] except Exception: result[key] = 0 + + active_jobs_aggregate = _fetch_active_jobs_aggregate(window_minutes=window_minutes) + active_job = get_active_detect_job_summary(event_limit=20) or {} + aggregate_queue = active_jobs_aggregate.get("queue") or {} + runtime = get_runtime_status() cluster_summary = ((runtime.get("cluster") or {}).get("summary") or {}) online_worker_nodes = int(cluster_summary.get("online_worker_nodes", 0) or 0) @@ -37,4 +424,274 @@ def fetch_overview() -> dict: result["worker_mode"] = runtime["worker"]["mode"] result["node_region"] = runtime["node"]["region"] result["node_role"] = runtime["node"]["role"] + + queue_health = get_detect_queue_health(window_minutes=window_minutes) + active_jobs_aggregate = _align_active_jobs_aggregate_with_runtime( + active_jobs_aggregate, + runtime=runtime, + queue_health=queue_health, + ) + runtime_snapshot_backlog = dict(queue_health.get("runtime_snapshot_backlog") or {}) + aggregate_queue = active_jobs_aggregate.get("queue") or {} + aggregate_queue_health = { + "has_active_job": bool(int(active_jobs_aggregate.get("active_jobs_total", 0) or 0) > 0), + "queue": aggregate_queue, + "throughput": active_jobs_aggregate.get("throughput") or {}, + } + selected_queue_health = aggregate_queue_health if aggregate_queue_health["has_active_job"] else queue_health + if float((queue_health.get("throughput") or {}).get("processed_per_minute", 0) or 0) > float( + (selected_queue_health.get("throughput") or {}).get("processed_per_minute", 0) or 0 + ): + selected_queue_health = queue_health + capacity_plan = get_detect_capacity_plan( + queue_health=selected_queue_health, + online_worker_nodes=online_worker_nodes, + ) + + retry_total = int(active_jobs_aggregate.get("retry_total", 0) or 0) + step_queue: list[dict] = [] + node_throughput: list[dict] = [] + bottleneck_step: dict | None = None + active_job_summary: dict | None = None + + if queue_health.get("has_active_job"): + job_payload = queue_health.get("job") or {} + queue_payload = queue_health.get("queue") or {} + throughput_payload = queue_health.get("throughput") or {} + runtime_job_code = str(job_payload.get("runtime_job_code") or "").strip() + display_job_code = runtime_job_code or str(job_payload.get("job_code") or "") + active_job_summary = { + "job_id": int(job_payload.get("job_id", 0) or 0), + "job_code": display_job_code, + "db_job_code": str(job_payload.get("job_code") or ""), + "runtime_job_code": runtime_job_code, + "status": str(job_payload.get("status") or ""), + "progress_percent": float(job_payload.get("progress_percent", 0) or 0), + "items_total": int(queue_payload.get("items_total", 0) or 0), + "items_pending": int(queue_payload.get("pending", 0) or 0), + "items_claimed": int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0), + "items_running": int(queue_payload.get("running", 0) or 0), + "items_display_running": int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0), + "items_completed": int(queue_payload.get("completed", 0) or 0), + "items_blacklisted": int(queue_payload.get("blacklisted", 0) or 0), + "items_failed": int(queue_payload.get("failed", 0) or 0), + "processed_per_minute": float(throughput_payload.get("processed_per_minute", 0) or 0), + "processed_recent": int(throughput_payload.get("processed_recent", 0) or 0), + "completed_recent": int(throughput_payload.get("completed_recent", 0) or 0), + "failed_recent": int(throughput_payload.get("failed_recent", 0) or 0), + "blacklisted_recent": int(throughput_payload.get("blacklisted_recent", 0) or 0), + "active_jobs_total": int(active_jobs_aggregate.get("active_jobs_total", 0) or 0), + } + + queue_pending_total = 0 + queue_claimed_total = 0 + queue_running_total = 0 + queue_display_running_total = 0 + queue_completed_total = 0 + queue_blacklist_total = 0 + queue_failed_total = 0 + backlog_payload = ((runtime.get("detect") or {}).get("backlog") or {}) + backlog_pending_total = max( + int(backlog_payload.get("pending_total", 0) or 0), + int(runtime_snapshot_backlog.get("pending_total", 0) or 0), + ) + backlog_claimed_total = max( + int(backlog_payload.get("claimed_total", 0) or 0), + int(runtime_snapshot_backlog.get("claimed_total", 0) or 0), + ) + backlog_running_total = max( + int(backlog_payload.get("running_total", 0) or 0), + int(runtime_snapshot_backlog.get("running_total", 0) or 0), + ) + backlog_register_pending_total = max( + int(backlog_payload.get("register_pending", 0) or 0), + int(runtime_snapshot_backlog.get("register_pending", 0) or 0), + ) + backlog_downstream_pending_total = max( + int(backlog_payload.get("downstream_pending", 0) or 0), + int(runtime_snapshot_backlog.get("downstream_pending", 0) or 0), + ) + if queue_health.get("has_active_job"): + queue_payload = queue_health.get("queue") or {} + queue_pending_total = int(queue_payload.get("pending", 0) or 0) + queue_claimed_total = int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0) + queue_running_total = int(queue_payload.get("running", 0) or 0) + queue_display_running_total = int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0) + queue_completed_total = int(queue_payload.get("completed", 0) or 0) + queue_blacklist_total = int(queue_payload.get("blacklisted", 0) or 0) + queue_failed_total = int(queue_payload.get("failed", 0) or 0) + elif int(active_jobs_aggregate.get("active_jobs_total", 0) or 0) > 0: + queue_pending_total = int(aggregate_queue.get("pending", 0) or 0) + queue_claimed_total = int(aggregate_queue.get("claimed", 0) or 0) + queue_running_total = int(aggregate_queue.get("running", 0) or 0) + queue_display_running_total = queue_running_total + queue_completed_total = int(aggregate_queue.get("completed", 0) or 0) + queue_blacklist_total = int(aggregate_queue.get("blacklisted", 0) or 0) + queue_failed_total = int(aggregate_queue.get("failed", 0) or 0) + elif active_job: + queue_pending_total = int(active_job.get("items_pending", 0) or 0) + queue_claimed_total = int(active_job.get("items_claimed", 0) or 0) + queue_running_total = int(active_job.get("items_running", 0) or 0) + queue_display_running_total = int( + active_job.get("display_items_running", active_job.get("items_running", 0)) or 0 + ) + queue_completed_total = int(active_job.get("items_completed", 0) or 0) + queue_blacklist_total = int(active_job.get("items_blacklisted", 0) or 0) + queue_failed_total = int(active_job.get("items_failed", 0) or 0) + + aggregate_step_queue = [ + { + "step_code": str(item.get("step_code") or ""), + "step_name": str(item.get("step_name") or ""), + "items_pending": int(item.get("items_pending", 0) or 0), + "items_running": int(item.get("items_running", 0) or 0), + "items_claimed": int(item.get("items_claimed", 0) or 0), + "items_completed": int(item.get("items_completed", 0) or 0), + "items_blacklisted": int(item.get("items_blacklisted", 0) or 0), + "items_failed": int(item.get("items_failed", 0) or 0), + "started_recent": int(item.get("started_recent", 0) or 0), + "processed_per_minute": float(item.get("processed_per_minute", 0) or 0), + "processed_recent": int(item.get("processed_recent", 0) or 0), + "completed_recent": int(item.get("completed_recent", 0) or 0), + "blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0), + "failed_recent": int(item.get("failed_recent", 0) or 0), + } + for item in list(active_jobs_aggregate.get("steps") or [])[:8] + ] + aggregate_node_throughput = [ + { + "node_code": str(item.get("node_code") or ""), + "items_pending": int(item.get("items_pending", 0) or 0), + "items_running": int(item.get("items_running", 0) or 0), + "display_running": int(item.get("display_running", item.get("items_running", 0)) or 0), + "items_claimed": int(item.get("items_claimed", 0) or 0), + "current_load": int(item.get("current_load", item.get("display_running", 0)) or 0), + "active_threads": int(item.get("active_threads", 0) or 0), + "max_threads": int(item.get("max_threads", 0) or 0), + "processed_recent": int(item.get("processed_recent", 0) or 0), + "processed_per_minute": float(item.get("processed_per_minute", 0) or 0), + "completed_recent": int(item.get("completed_recent", 0) or 0), + "failed_recent": int(item.get("failed_recent", 0) or 0), + "blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0), + } + for item in list(active_jobs_aggregate.get("nodes") or [])[:8] + ] + queue_step_queue = [ + { + "step_code": str(item.get("step_code") or ""), + "step_name": str(item.get("step_name") or ""), + "items_pending": int(item.get("items_pending", 0) or 0), + "items_running": int(item.get("items_running", 0) or 0), + "items_claimed": int(item.get("items_claimed", 0) or 0), + "items_completed": int(item.get("items_completed", 0) or 0), + "items_blacklisted": int(item.get("items_blacklisted", 0) or 0), + "items_failed": int(item.get("items_failed", 0) or 0), + "started_recent": int(item.get("started_recent", 0) or 0), + "processed_per_minute": float(item.get("processed_per_minute", 0) or 0), + "processed_recent": int(item.get("processed_recent", 0) or 0), + "completed_recent": int(item.get("completed_recent", 0) or 0), + "blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0), + "failed_recent": int(item.get("failed_recent", 0) or 0), + } + for item in _merge_step_queues_with_runtime_activity( + list(queue_health.get("steps") or []), + runtime_activity=dict(queue_health.get("runtime_activity") or {}), + window_minutes=window_minutes, + limit=8, + ) + ] + queue_node_throughput = [ + { + "node_code": str(item.get("node_code") or ""), + "items_pending": int(item.get("items_pending", 0) or 0), + "items_running": int(item.get("items_running", 0) or 0), + "display_running": int(item.get("display_running", item.get("items_running", 0)) or 0), + "items_claimed": int(item.get("items_claimed", 0) or 0), + "current_load": int(item.get("current_load", item.get("display_running", 0)) or 0), + "active_threads": int(item.get("active_threads", 0) or 0), + "max_threads": int(item.get("max_threads", 0) or 0), + "processed_recent": int(item.get("processed_recent", 0) or 0), + "processed_per_minute": float(item.get("processed_per_minute", 0) or 0), + "completed_recent": int(item.get("completed_recent", 0) or 0), + "failed_recent": int(item.get("failed_recent", 0) or 0), + "blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0), + } + for item in list(queue_health.get("nodes") or [])[:8] + ] + step_queue = aggregate_step_queue + node_throughput = aggregate_node_throughput + aggregate_ppm = float((active_jobs_aggregate.get("throughput") or {}).get("processed_per_minute", 0) or 0) + queue_ppm = float((queue_health.get("throughput") or {}).get("processed_per_minute", 0) or 0) + if queue_health.get("has_active_job") or queue_ppm > aggregate_ppm: + step_queue = queue_step_queue + node_throughput = queue_node_throughput + if step_queue: + bottleneck_step = max( + step_queue, + key=lambda item: ( + int(item.get("items_pending", 0) or 0), + int(item.get("items_running", 0) or 0), + -float(item.get("processed_per_minute", 0) or 0), + ), + ) + + result["active_job"] = active_job_summary or {} + result["queue_health"] = queue_health + result["active_jobs_aggregate"] = active_jobs_aggregate + result["capacity_plan"] = capacity_plan + result["step_queue"] = step_queue + result["node_throughput"] = node_throughput + result["retry_total"] = retry_total + result["bottleneck_step"] = bottleneck_step or {} + aggregate_throughput = active_jobs_aggregate.get("throughput") or {} + queue_throughput = queue_health.get("throughput") or {} + ops_processed_per_minute = float(aggregate_throughput.get("processed_per_minute", 0) or 0) + ops_processed_recent = int(aggregate_throughput.get("processed_recent", 0) or 0) + ops_completed_recent = int(aggregate_throughput.get("completed_recent", 0) or 0) + ops_failed_recent = int(aggregate_throughput.get("failed_recent", 0) or 0) + ops_blacklisted_recent = int(aggregate_throughput.get("blacklisted_recent", 0) or 0) + if queue_health.get("has_active_job") or float(queue_throughput.get("processed_per_minute", 0) or 0) > ops_processed_per_minute: + ops_processed_per_minute = float(queue_throughput.get("processed_per_minute", 0) or 0) + ops_processed_recent = int(queue_throughput.get("processed_recent", 0) or 0) + ops_completed_recent = int(queue_throughput.get("completed_recent", 0) or 0) + ops_failed_recent = int(queue_throughput.get("failed_recent", 0) or 0) + ops_blacklisted_recent = int(queue_throughput.get("blacklisted_recent", 0) or 0) + result["ops_summary"] = { + "active_jobs_total": int(active_jobs_aggregate.get("active_jobs_total", 0) or 0), + "processed_per_minute": ops_processed_per_minute, + "processed_recent": ops_processed_recent, + "completed_recent": ops_completed_recent, + "failed_recent": ops_failed_recent, + "blacklisted_recent": ops_blacklisted_recent, + "estimated_hours_remaining": float(capacity_plan.get("estimated_hours_remaining", 0) or 0), + "remaining_items": int(capacity_plan.get("remaining_items", 0) or 0), + "recommended_additional_workers": int(capacity_plan.get("recommended_additional_workers", 0) or 0), + "online_worker_nodes": online_worker_nodes, + "dedicated_online_worker_nodes": dedicated_online_worker_nodes, + "active_execution_nodes": sum( + 1 + for item in node_throughput + if int(item.get("items_running", 0) or 0) > 0 + or int(item.get("items_claimed", 0) or 0) > 0 + or int(item.get("processed_recent", 0) or 0) > 0 + ), + } + result["processed_per_minute"] = ops_processed_per_minute + result["processed_recent"] = ops_processed_recent + result["completed_recent"] = ops_completed_recent + result["failed_recent"] = ops_failed_recent + result["blacklisted_recent"] = ops_blacklisted_recent + result["active_execution_nodes"] = int(result["ops_summary"]["active_execution_nodes"] or 0) + result["queue_pending_total"] = queue_pending_total + result["queue_claimed_total"] = queue_claimed_total + result["queue_running_total"] = queue_running_total + result["queue_display_running_total"] = max(queue_display_running_total, queue_running_total) + result["queue_completed_total"] = queue_completed_total + result["queue_blacklist_total"] = queue_blacklist_total + result["queue_failed_total"] = queue_failed_total + result["backlog_pending_total"] = max(backlog_pending_total, queue_pending_total) + result["backlog_claimed_total"] = max(backlog_claimed_total, queue_claimed_total) + result["backlog_running_total"] = max(backlog_running_total, queue_running_total) + result["backlog_register_pending_total"] = backlog_register_pending_total + result["backlog_downstream_pending_total"] = backlog_downstream_pending_total return result diff --git a/domain-api/app/services/debug_event_service.py b/domain-api/app/services/debug_event_service.py index 68913bc..713cfc7 100644 --- a/domain-api/app/services/debug_event_service.py +++ b/domain-api/app/services/debug_event_service.py @@ -2,12 +2,13 @@ from __future__ import annotations import json import socket +import threading import urllib.error import urllib.request from datetime import datetime, timedelta from app.core.config import settings -from app.core.db import get_db +from app.core.db import db_read_retry, get_db _DEBUG_SCHEMA_SQL = """ @@ -27,6 +28,9 @@ CREATE INDEX IF NOT EXISTS idx_detect_debug_events_created ON detect_debug_events(created_at DESC); """ +_DEBUG_SCHEMA_READY = False +_DEBUG_SCHEMA_LOCK = threading.Lock() + def _format_time(value: datetime | None) -> str: return value.isoformat(sep=" ", timespec="seconds") if value else "" @@ -82,11 +86,18 @@ def _debug_ingest_url(base_url: str) -> str: def ensure_debug_event_schema() -> None: - with get_db() as conn: - conn.autocommit = False - with conn.cursor() as cur: - cur.execute(_DEBUG_SCHEMA_SQL) - conn.commit() + global _DEBUG_SCHEMA_READY + if _DEBUG_SCHEMA_READY: + return + with _DEBUG_SCHEMA_LOCK: + if _DEBUG_SCHEMA_READY: + return + with get_db() as conn: + conn.autocommit = False + with conn.cursor() as cur: + cur.execute(_DEBUG_SCHEMA_SQL) + conn.commit() + _DEBUG_SCHEMA_READY = True def append_debug_event( @@ -159,12 +170,185 @@ def append_debug_event( return record_id +def _load_debug_event_record(record_id: int) -> dict | None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, source_region, node_code, service, event_type, level, message, payload_json, created_at + FROM detect_debug_events + WHERE id = %s + LIMIT 1 + """, + (int(record_id),), + ) + row = cur.fetchone() + if not row: + return None + return { + "id": int(row[0]), + "source_region": str(row[1] or ""), + "node_code": str(row[2] or ""), + "service": str(row[3] or ""), + "event_type": str(row[4] or ""), + "level": str(row[5] or "info"), + "message": str(row[6] or ""), + "payload": row[7] if isinstance(row[7], dict) else {}, + "created_at": _format_time(row[8]), + } + + +def _normalize_worker_log_event(debug_event: dict) -> dict | None: + payload = dict(debug_event.get("payload") or {}) + message = _normalize_message(debug_event.get("message"), fallback="worker_log") + domain = str(payload.get("domain") or "").strip().lower() + status = str(payload.get("status") or "").strip().lower() + if not domain and ":" in message: + domain = message.rsplit(":", 1)[-1].strip().lower() + + event_type = "" + if status == "completed": + event_type = "domain_completed" + elif status == "failed": + event_type = "domain_failed" + elif status == "blacklisted": + event_type = "domain_blacklisted" + elif "开始检测域名" in message: + event_type = "domain_started" + elif "域名检测完成" in message: + event_type = "domain_completed" + payload.setdefault("status", "completed") + elif "域名已命中黑名单" in message or "命中黑名单" in message: + event_type = "domain_blacklisted" + payload.setdefault("status", "blacklisted") + elif "域名检测失败" in message: + event_type = "domain_failed" + payload.setdefault("status", "failed") + + if not event_type or not domain: + return None + + payload.setdefault("domain", domain) + payload["imported_from_debug_event"] = True + payload["debug_event_record_id"] = int(debug_event.get("id") or 0) + payload["debug_event_source_region"] = str(debug_event.get("source_region") or "") + return { + "node_code": str(debug_event.get("node_code") or "").strip(), + "event_type": event_type, + "level": str(debug_event.get("level") or "info"), + "message": message, + "payload": payload, + "created_at": str(debug_event.get("created_at") or "").strip(), + } + + +def _ingest_worker_log_into_active_job(debug_event: dict) -> dict: + if str(debug_event.get("event_type") or "").strip() != "worker_log": + return {"imported": False, "reason": "not_worker_log"} + + normalized_event = _normalize_worker_log_event(debug_event) + if not normalized_event: + return {"imported": False, "reason": "not_domain_progress_event"} + + from app.services.detect_job_service import get_active_detect_job_summary + from app.services.sync_push_service import ( + _apply_detect_result_event_to_domain, + _apply_detect_result_event_to_job_item, + ) + + active_job = get_active_detect_job_summary(event_limit=1) or {} + target_job_id = int(active_job.get("job_id") or 0) + if target_job_id <= 0: + return {"imported": False, "reason": "no_active_job"} + + debug_event_record_id = int(debug_event.get("id") or 0) + updated_job_items = 0 + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id + FROM detect_run_events + WHERE job_id = %s + AND (payload_json->>'debug_event_record_id') = %s + ORDER BY id DESC + LIMIT 1 + """, + (target_job_id, str(debug_event_record_id)), + ) + existing = cur.fetchone() + if existing: + return { + "imported": False, + "reason": "deduplicated", + "target_job_id": target_job_id, + "detect_run_event_id": int(existing[0]), + } + + created_at = _parse_time(normalized_event.get("created_at")) + payload_json = _safe_json_dumps(normalized_event.get("payload") or {}) + if created_at: + cur.execute( + """ + INSERT INTO detect_run_events ( + job_id, node_code, event_type, level, message, payload_json, created_at + ) VALUES (%s, %s, %s, %s, %s, %s::jsonb, %s) + RETURNING id + """, + ( + target_job_id, + normalized_event["node_code"], + normalized_event["event_type"], + normalized_event["level"], + normalized_event["message"], + payload_json, + created_at, + ), + ) + else: + cur.execute( + """ + INSERT INTO detect_run_events ( + job_id, node_code, event_type, level, message, payload_json + ) VALUES (%s, %s, %s, %s, %s, %s::jsonb) + RETURNING id + """, + ( + target_job_id, + normalized_event["node_code"], + normalized_event["event_type"], + normalized_event["level"], + normalized_event["message"], + payload_json, + ), + ) + detect_run_event_id = int((cur.fetchone() or [0])[0] or 0) + _apply_detect_result_event_to_domain(cur, normalized_event) + updated_job_items = _apply_detect_result_event_to_job_item( + cur, + target_job_id=target_job_id, + event=normalized_event, + ) + conn.commit() + return { + "imported": True, + "reason": "imported", + "target_job_id": target_job_id, + "detect_run_event_id": detect_run_event_id, + "updated_job_items": updated_job_items, + "event_type": normalized_event["event_type"], + "domain": str((normalized_event.get("payload") or {}).get("domain") or ""), + } + + +@db_read_retry() def list_debug_events( limit: int = 50, *, service: str | None = None, event_type: str | None = None, source_region: str | None = None, + node_code: str | None = None, level: str | None = None, before_id: int | None = None, after_id: int | None = None, @@ -183,6 +367,9 @@ def list_debug_events( if str(source_region or "").strip(): conditions.append("source_region = %s") params.append(str(source_region).strip()) + if str(node_code or "").strip(): + conditions.append("node_code = %s") + params.append(str(node_code).strip()) if str(level or "").strip(): conditions.append("level = %s") params.append(str(level).strip()) @@ -237,6 +424,7 @@ def list_debug_events( } +@db_read_retry() def get_debug_event_overview(*, window_minutes: int = 10, source_region: str | None = None) -> dict: ensure_debug_event_schema() safe_window = max(1, min(int(window_minutes or 10), 180)) @@ -671,7 +859,24 @@ def ingest_debug_event(payload: dict, *, shared_token: str | None = None) -> tup message=_normalize_message(payload.get("message"), fallback="remote debug event"), payload=payload.get("payload") or {}, ) - return True, "调试事件接收成功", {"record_id": record_id} + debug_event = _load_debug_event_record(record_id) or { + "id": int(record_id), + "source_region": str(payload.get("source_region") or settings.node_region), + "node_code": str(payload.get("node_code") or ""), + "service": str(payload.get("service") or "unknown"), + "event_type": str(payload.get("event_type") or "event"), + "level": str(payload.get("level") or "info"), + "message": _normalize_message(payload.get("message"), fallback="remote debug event"), + "payload": payload.get("payload") or {}, + "created_at": "", + } + job_import = {} + if str(debug_event.get("service") or "").strip() == "worker-event": + try: + job_import = _ingest_worker_log_into_active_job(debug_event) + except Exception as exc: + job_import = {"imported": False, "reason": f"job_import_failed: {exc}"} + return True, "调试事件接收成功", {"record_id": record_id, "job_import": job_import} def push_debug_event( diff --git a/domain-api/app/services/detect_job_service.py b/domain-api/app/services/detect_job_service.py index 594d81e..3e02113 100644 --- a/domain-api/app/services/detect_job_service.py +++ b/domain-api/app/services/detect_job_service.py @@ -1,17 +1,67 @@ from __future__ import annotations import json +import time from datetime import datetime import math from uuid import uuid4 from app.core.config import settings -from app.core.db import get_db +from app.core.db import db_read_retry, get_db, is_retryable_db_error from app.services.debug_event_service import push_debug_event +from app.services.settings_service import get_settings_payload ACTIVE_JOB_STATUSES = ("pending", "running") _RUNTIME_NODE_STALE_MINUTES = 10 +_DEFAULT_TASK_MODE = "domain_pipeline" +_PIPELINE_STEP_ORDER = [ + "detect_register", + "detect_baidu_site", + "detect_360_site", + "detect_chinaz", + "detect_aizhan", + "detect_wayback", + "detect_jucha", + "detect_juziseo", +] +_PIPELINE_STEP_FIELDS = { + "detect_baidu_site": "baidu_site", + "detect_360_site": "qihu360_site", + "detect_chinaz": "chinaz_info", + "detect_aizhan": "aizhan_info", + "detect_wayback": "wayback_info", + "detect_jucha": "jucha_info", + "detect_juziseo": "juziseo_info", +} +_PIPELINE_STEP_LABELS = { + "detect_register": "注册状态检测", + "detect_baidu_site": "百度site检测", + "detect_360_site": "360 site检测", + "detect_chinaz": "站长之家检测", + "detect_aizhan": "爱站检测", + "detect_wayback": "时光机检测", + "detect_jucha": "聚查检测", + "detect_juziseo": "桔子检测", +} +_PIPELINE_STEP_RETRY_LIMITS = { + "default": 3, + "detect_wayback": 2, +} +_SINGLE_STEP_JOB_SPECS = { + "detect_baidu_site": { + "step_code": "detect_baidu_site", + "task_mode": "single_step", + "label": "百度site检测", + "source": "api-step-start", + }, + "detect_wayback": { + "step_code": "detect_wayback", + "task_mode": "single_step", + "label": "时光机检测", + "source": "api-step-start", + }, +} def _selection_sql() -> str: @@ -30,6 +80,237 @@ def _format_time(value: datetime | None) -> str: return value.isoformat(sep=" ", timespec="seconds") if value else "" +def _normalize_sync_region(value: str | None, fallback: str) -> str: + text = str(value or "").strip() + if not text or text == "unknown": + return str(fallback or "unknown").strip() or "unknown" + return text + + +def normalize_detect_step_code(step_code: object) -> str: + normalized = str(step_code or "").strip() + if not normalized: + return "" + return normalized if normalized in _SINGLE_STEP_JOB_SPECS else "" + + +def resolve_detect_job_definition(step_code: object = None) -> dict: + normalized_step_code = normalize_detect_step_code(step_code) + if normalized_step_code: + spec = dict(_SINGLE_STEP_JOB_SPECS[normalized_step_code]) + spec["is_single_step"] = True + return spec + return { + "step_code": "", + "task_mode": _DEFAULT_TASK_MODE, + "label": "全流程检测", + "source": "api-start", + "is_single_step": False, + } + + +def _step_label(step_code: str) -> str: + return _PIPELINE_STEP_LABELS.get(str(step_code or "").strip(), str(step_code or "").strip() or "unknown") + + +def _json_state(value: object) -> str: + if isinstance(value, dict): + return str(value.get("state") or "").strip().lower() + return "" + + +def _pipeline_retry_limit(step_code: str) -> int: + return int(_PIPELINE_STEP_RETRY_LIMITS.get(str(step_code or "").strip(), _PIPELINE_STEP_RETRY_LIMITS["default"]) or 1) + + +def _classify_pipeline_item_outcome( + *, + item_status: str, + result_payload: dict | None, + step_code: str, + attempt_count: int, +) -> dict: + normalized_item_status = str(item_status or "").strip().lower() + payload = dict(result_payload or {}) + result_state = str(payload.get("state") or "").strip().lower() + retry_limit = _pipeline_retry_limit(step_code) + retry_recommended = bool(payload.get("retry_recommended")) + legacy_retry_candidate = normalized_item_status == "failed" and not result_state + + if normalized_item_status == "blacklisted" or result_state == "blacklisted": + return { + "action": "black_hit", + "result_state": result_state or "blacklisted", + "retry_limit": retry_limit, + "should_retry": False, + "reason_code": "blacklisted", + } + + should_retry = ( + result_state in {"degraded", "error"} + or retry_recommended + or legacy_retry_candidate + ) + if should_retry: + return { + "action": "retry" if int(attempt_count or 0) < retry_limit else "reject", + "result_state": result_state or ("degraded" if retry_recommended else "failed"), + "retry_limit": retry_limit, + "should_retry": int(attempt_count or 0) < retry_limit, + "reason_code": "external_retry" if int(attempt_count or 0) < retry_limit else "retry_exhausted", + } + + if normalized_item_status == "failed" or result_state in {"failed", "rejected", "reject"}: + return { + "action": "reject", + "result_state": result_state or "failed", + "retry_limit": retry_limit, + "should_retry": False, + "reason_code": "business_reject", + } + + return { + "action": "pass", + "result_state": result_state or "passed", + "retry_limit": retry_limit, + "should_retry": False, + "reason_code": "passed", + } + + +def _load_pipeline_order(settings_payload: dict | None = None) -> list[str]: + payload = settings_payload or get_settings_payload() + detect_options = dict(payload.get("detect_options") or {}) + configured_order = list(detect_options.get("detect_order") or []) + normalized_order = [item for item in configured_order if item in _PIPELINE_STEP_ORDER] + for item in _PIPELINE_STEP_ORDER: + if item not in normalized_order: + normalized_order.append(item) + return [item for item in normalized_order if bool(detect_options.get(item, False))] + + +def _load_domain_pipeline_snapshot(cur, domain_id: int) -> dict | None: + cur.execute( + """ + SELECT + d.id, + d.domain, + d.source_type, + d.register_status, + d.detect_status, + d.use_status, + d.expire_date, + dd.baidu_site, + dd.qihu360_site, + dd.wayback_info, + dd.chinaz_info, + dd.aizhan_info, + dd.jucha_info, + dd.juziseo_info + FROM domains AS d + LEFT JOIN domain_detections AS dd + ON dd.domain_id = d.id + WHERE d.id = %s + LIMIT 1 + """, + (int(domain_id),), + ) + row = cur.fetchone() + if not row: + return None + return { + "id": int(row[0]), + "domain": row[1] or "", + "source_type": int(row[2] or 0), + "register_status": int(row[3] or 0), + "detect_status": int(row[4] or 0), + "use_status": int(row[5] or 0), + "expire_date": _format_time(row[6]) if row[6] else "", + "baidu_site": _decode_payload(row[7]), + "qihu360_site": _decode_payload(row[8]), + "wayback_info": _decode_payload(row[9]), + "chinaz_info": _decode_payload(row[10]), + "aizhan_info": _decode_payload(row[11]), + "jucha_info": _decode_payload(row[12]), + "juziseo_info": _decode_payload(row[13]), + } + + +def _is_pipeline_step_completed(domain_snapshot: dict, step_code: str) -> bool: + normalized_step_code = str(step_code or "").strip() + if normalized_step_code == "detect_register": + return int(domain_snapshot.get("register_status", 0) or 0) not in {0, 10} + field_name = _PIPELINE_STEP_FIELDS.get(normalized_step_code) + if not field_name: + return False + state = _json_state(domain_snapshot.get(field_name)) + return state in {"passed", "blacklisted"} + + +def _should_skip_pipeline_step(domain_snapshot: dict, step_code: str) -> bool: + normalized_step_code = str(step_code or "").strip() + if normalized_step_code == "detect_register" and int(domain_snapshot.get("source_type", 0) or 0) == 1: + return True + return False + + +def resolve_domain_pipeline_step(domain_snapshot: dict, *, settings_payload: dict | None = None, after_step_code: str | None = None) -> str: + pipeline_order = _load_pipeline_order(settings_payload=settings_payload) + if not pipeline_order: + return "" + normalized_after = str(after_step_code or "").strip() + seen_after = not normalized_after + for step_code in pipeline_order: + if not seen_after: + if step_code == normalized_after: + seen_after = True + continue + if _should_skip_pipeline_step(domain_snapshot, step_code): + continue + if _is_pipeline_step_completed(domain_snapshot, step_code): + continue + return step_code + return "" + + +def resolve_initial_domain_pipeline_item( + domain_snapshot: dict, + *, + settings_payload: dict | None = None, + retry_count: int = 0, +) -> tuple[str, dict | None]: + step_code = resolve_domain_pipeline_step(domain_snapshot, settings_payload=settings_payload) + if not step_code: + return "", None + return step_code, _build_step_payload( + step_code=step_code, + domain_snapshot=domain_snapshot, + settings_payload=settings_payload, + retry_count=retry_count, + ) + + +def _build_step_payload(*, step_code: str, domain_snapshot: dict, settings_payload: dict | None = None, retry_count: int = 0) -> dict: + pipeline_order = _load_pipeline_order(settings_payload=settings_payload) + payload = { + "step_code": str(step_code or "").strip(), + "step_name": _step_label(step_code), + "pipeline_order": pipeline_order, + "retry_count": int(retry_count or 0), + "domain": str(domain_snapshot.get("domain") or "").strip(), + "source_type": int(domain_snapshot.get("source_type", 0) or 0), + } + if str(step_code or "").strip() == "detect_wayback": + payload.update( + { + "wayback_strategy": "recent_years", + "wayback_recent_years": 5, + "wayback_stop_on_first_hit": True, + } + ) + return payload + + def _decode_payload(value: object) -> dict: if isinstance(value, dict): return value @@ -41,6 +322,35 @@ def _decode_payload(value: object) -> dict: return {} +def _upsert_domain_detection_field(cur, domain_id: int, field_name: str, payload: dict) -> bool: + normalized_field_name = str(field_name or "").strip() + if not normalized_field_name or normalized_field_name not in _PIPELINE_STEP_FIELDS.values(): + return False + normalized_payload = dict(payload or {}) + cur.execute("SELECT id FROM domain_detections WHERE domain_id = %s LIMIT 1", (int(domain_id),)) + existing = cur.fetchone() + payload_json = json.dumps(normalized_payload, ensure_ascii=False) + if existing: + cur.execute( + f""" + UPDATE domain_detections + SET {normalized_field_name} = %s::jsonb, + update_time = NOW() + WHERE domain_id = %s + """, + (payload_json, int(domain_id)), + ) + return True + cur.execute( + f""" + INSERT INTO domain_detections (domain_id, {normalized_field_name}, create_time, update_time) + VALUES (%s, %s::jsonb, NOW(), NOW()) + """, + (int(domain_id), payload_json), + ) + return True + + def _int_value(value: object) -> int: try: return int(value or 0) @@ -48,19 +358,64 @@ def _int_value(value: object) -> int: return 0 +def _effective_runtime_load(*, items_running: object, active_threads: object) -> int: + return max(_int_value(items_running), _int_value(active_threads), 0) + + +def _control_node_supports_worker(*, region: object, metadata: dict | None) -> bool: + normalized_region = str(region or "").strip() + runtime_metadata = dict(metadata or {}) + active_threads = int(runtime_metadata.get("active_threads", 0) or 0) + max_threads = int(runtime_metadata.get("max_threads", 0) or 0) + if normalized_region != "mainland": + return False + return bool( + runtime_metadata.get("worker_online", False) + or runtime_metadata.get("detect_participating", False) + or active_threads > 0 + or max_threads > 0 + ) + + def _build_runtime_display_bucket(row: tuple) -> dict | None: node_code = str(row[0] or "").strip() if not node_code: return None + region = str(row[1] or "").strip() + role = str(row[2] or "").strip() metadata = _decode_payload(row[5]) + if role == "control" and not _control_node_supports_worker(region=region, metadata=metadata): + return None items_total = _int_value(metadata.get("job_items_total")) items_claimed = _int_value(metadata.get("job_items_claimed")) items_running = _int_value(metadata.get("job_items_running")) items_completed = _int_value(metadata.get("job_items_completed")) items_failed = _int_value(metadata.get("job_items_failed")) - current_load = _int_value(row[4]) + raw_current_load = _int_value(row[4]) + active_threads = _int_value(metadata.get("active_threads")) + max_threads = _int_value(metadata.get("max_threads")) + current_load = _effective_runtime_load(items_running=items_running, active_threads=active_threads) detect_participating = bool(metadata.get("detect_participating", False)) - if items_total <= 0 and items_claimed <= 0 and items_running <= 0 and items_completed <= 0 and items_failed <= 0 and current_load <= 0 and not detect_participating: + if ( + settings.node_region == "overseas" + and settings.node_role == "control" + and node_code == str(settings.node_code or "").strip() + and not detect_participating + ): + # Overseas control nodes may report their own control-plane load into + # detect_worker_nodes; that load should never be counted as worker execution. + return None + if ( + items_total <= 0 + and items_claimed <= 0 + and items_running <= 0 + and items_completed <= 0 + and items_failed <= 0 + and current_load <= 0 + and raw_current_load <= 0 + and active_threads <= 0 + and not detect_participating + ): return None items_pending = max(items_total - items_claimed - items_running - items_completed - items_failed, 0) return { @@ -69,14 +424,17 @@ def _build_runtime_display_bucket(row: tuple) -> dict | None: "items_pending": items_pending, "items_claimed": items_claimed, "items_running": items_running, + "display_running": current_load, "items_completed": items_completed, "items_blacklisted": 0, "items_failed": items_failed, "metrics_source": str(metadata.get("service") or "runtime").strip() or "runtime", - "region": str(row[1] or "").strip(), - "role": str(row[2] or "").strip(), + "region": region, + "role": role, "status": str(row[3] or "").strip(), "current_load": current_load, + "active_threads": active_threads, + "max_threads": max_threads, "last_heartbeat_at": _format_time(row[6]) if row[6] else "", } @@ -93,6 +451,7 @@ def _merge_display_node_stats(*, local_node_stats: list[dict], runtime_node_rows "items_pending": 0, "items_claimed": 0, "items_running": 0, + "display_running": 0, "items_completed": 0, "items_blacklisted": 0, "items_failed": 0, @@ -101,6 +460,8 @@ def _merge_display_node_stats(*, local_node_stats: list[dict], runtime_node_rows "role": "", "status": "", "current_load": 0, + "active_threads": 0, + "max_threads": 0, "last_heartbeat_at": "", }, ) @@ -115,6 +476,7 @@ def _merge_display_node_stats(*, local_node_stats: list[dict], runtime_node_rows "items_pending", "items_claimed", "items_running", + "display_running", "items_completed", "items_blacklisted", "items_failed", @@ -134,10 +496,13 @@ def _merge_display_node_stats(*, local_node_stats: list[dict], runtime_node_rows "items_pending", "items_claimed", "items_running", + "display_running", "items_completed", "items_blacklisted", "items_failed", "current_load", + "active_threads", + "max_threads", ): bucket[key] = max(_int_value(bucket.get(key)), _int_value(runtime_bucket.get(key))) for key in ("metrics_source", "region", "role", "status", "last_heartbeat_at"): @@ -148,6 +513,7 @@ def _merge_display_node_stats(*, local_node_stats: list[dict], runtime_node_rows merged.values(), key=lambda item: ( str(item.get("node_code") or "") == "unassigned", + -_int_value(item.get("active_threads")), -_int_value(item.get("items_running")), -_int_value(item.get("items_claimed")), -_int_value(item.get("items_total")), @@ -172,29 +538,59 @@ def _build_display_summary(node_stats: list[dict]) -> dict: effective_nodes = [ item for item in list(node_stats or []) - if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned" + if str(item.get("node_code") or "").strip() + and str(item.get("node_code") or "").strip() != "unassigned" + and not ( + settings.node_region == "overseas" + and settings.node_role == "control" + and str(item.get("node_code") or "").strip() == str(settings.node_code or "").strip() + ) ] return { "items_claimed": sum(_int_value(item.get("items_claimed")) for item in effective_nodes), "items_running": sum(_int_value(item.get("items_running")) for item in effective_nodes), + "display_running": sum( + _effective_runtime_load( + items_running=item.get("items_running"), + active_threads=item.get("active_threads"), + ) + for item in effective_nodes + ), + "current_load": sum( + _effective_runtime_load( + items_running=item.get("items_running"), + active_threads=item.get("active_threads"), + ) + for item in effective_nodes + ), + "active_threads": sum(_int_value(item.get("active_threads")) for item in effective_nodes), + "max_threads": sum(_int_value(item.get("max_threads")) for item in effective_nodes), "items_completed": sum(_int_value(item.get("items_completed")) for item in effective_nodes), "items_failed": sum(_int_value(item.get("items_failed")) for item in effective_nodes), "active_nodes": [ str(item.get("node_code") or "").strip() for item in effective_nodes - if _int_value(item.get("items_claimed")) > 0 or _int_value(item.get("items_running")) > 0 + if _effective_runtime_load( + items_running=item.get("items_running"), + active_threads=item.get("active_threads"), + ) > 0 ], } def _normalize_node_bucket(item: dict) -> dict: node_code = str(item.get("node_code") or "").strip() + items_running = _int_value(item.get("items_running")) + active_threads = _int_value(item.get("active_threads")) + max_threads = _int_value(item.get("max_threads")) + current_load = _effective_runtime_load(items_running=items_running, active_threads=active_threads) return { "node_code": node_code, "items_total": _int_value(item.get("items_total")), "items_pending": _int_value(item.get("items_pending")), "items_claimed": _int_value(item.get("items_claimed")), - "items_running": _int_value(item.get("items_running")), + "items_running": items_running, + "display_running": current_load, "items_completed": _int_value(item.get("items_completed")), "items_blacklisted": _int_value(item.get("items_blacklisted")), "items_failed": _int_value(item.get("items_failed")), @@ -202,7 +598,9 @@ def _normalize_node_bucket(item: dict) -> dict: "region": str(item.get("region") or "").strip(), "role": str(item.get("role") or "").strip(), "status": str(item.get("status") or "").strip(), - "current_load": _int_value(item.get("current_load")), + "current_load": current_load, + "active_threads": active_threads, + "max_threads": max_threads, "last_heartbeat_at": str(item.get("last_heartbeat_at") or "").strip(), } @@ -215,39 +613,65 @@ def _build_effective_node_stats( normalized_total = max(0, _int_value(raw_items_total)) assigned_buckets: list[dict] = [] assigned_total = 0 + unassigned_bucket: dict | None = None for item in list(distributed_node_stats or []): bucket = _normalize_node_bucket(item) - if not bucket["node_code"] or bucket["node_code"] == "unassigned": + if not bucket["node_code"]: + continue + if bucket["node_code"] == "unassigned": + unassigned_bucket = bucket + continue + if ( + settings.node_region == "overseas" + and settings.node_role == "control" + and bucket["node_code"] == str(settings.node_code or "").strip() + ): continue assigned_buckets.append(bucket) assigned_total += bucket["items_total"] + if unassigned_bucket and unassigned_bucket["items_total"] > 0: + assigned_buckets.append(unassigned_bucket) + assigned_total += unassigned_bucket["items_total"] + effective_total = max(normalized_total, assigned_total) remainder_total = max(effective_total - assigned_total, 0) if remainder_total > 0: - assigned_buckets.append( - { - "node_code": "unassigned", - "items_total": remainder_total, - "items_pending": remainder_total, - "items_claimed": 0, - "items_running": 0, - "items_completed": 0, - "items_blacklisted": 0, - "items_failed": 0, - "metrics_source": "central_queue", - "region": "", - "role": "", - "status": "", - "current_load": 0, - "last_heartbeat_at": "", - } - ) + if unassigned_bucket: + unassigned_bucket["items_total"] += remainder_total + unassigned_bucket["items_pending"] += remainder_total + assigned_buckets = [ + unassigned_bucket if str(item.get("node_code") or "").strip() == "unassigned" else item + for item in assigned_buckets + ] + else: + assigned_buckets.append( + { + "node_code": "unassigned", + "items_total": remainder_total, + "items_pending": remainder_total, + "items_claimed": 0, + "items_running": 0, + "display_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + "metrics_source": "central_queue", + "region": "", + "role": "", + "status": "", + "current_load": 0, + "active_threads": 0, + "max_threads": 0, + "last_heartbeat_at": "", + } + ) return sorted( assigned_buckets, key=lambda item: ( str(item.get("node_code") or "") == "unassigned", + -_int_value(item.get("active_threads")), -_int_value(item.get("items_running")), -_int_value(item.get("items_claimed")), -_int_value(item.get("items_total")), @@ -256,27 +680,58 @@ def _build_effective_node_stats( ) -def _build_effective_summary(*, node_stats: list[dict], raw_items_total: int, raw_items_blacklisted: int) -> dict: +def _build_effective_summary( + *, + node_stats: list[dict], + raw_items_total: int, + raw_items_pending: int, + raw_items_claimed: int, + raw_items_running: int, + raw_items_completed: int, + raw_items_blacklisted: int, + raw_items_failed: int, +) -> dict: effective_total = max(0, _int_value(raw_items_total)) + effective_pending = max(0, _int_value(raw_items_pending)) + effective_claimed = max(0, _int_value(raw_items_claimed)) + effective_running = max(0, _int_value(raw_items_running)) + effective_completed = max(0, _int_value(raw_items_completed)) effective_blacklisted = max(0, _int_value(raw_items_blacklisted)) - effective_pending = sum(_int_value(item.get("items_pending")) for item in list(node_stats or [])) - effective_claimed = sum(_int_value(item.get("items_claimed")) for item in list(node_stats or [])) - effective_running = sum(_int_value(item.get("items_running")) for item in list(node_stats or [])) - effective_completed = sum(_int_value(item.get("items_completed")) for item in list(node_stats or [])) - effective_failed = sum(_int_value(item.get("items_failed")) for item in list(node_stats or [])) + effective_failed = max(0, _int_value(raw_items_failed)) effective_terminal = effective_completed + effective_blacklisted + effective_failed return { "items_total": effective_total, - "items_pending": max(effective_pending, 0), - "items_claimed": max(effective_claimed, 0), - "items_running": max(effective_running, 0), - "items_completed": max(effective_completed, 0), + "items_pending": effective_pending, + "items_claimed": effective_claimed, + "items_running": effective_running, + "items_completed": effective_completed, "items_blacklisted": effective_blacklisted, - "items_failed": max(effective_failed, 0), + "items_failed": effective_failed, "items_terminal": max(effective_terminal, 0), } +def _build_step_bucket(step_code: str) -> dict: + normalized_step_code = str(step_code or "").strip() + return { + "step_code": normalized_step_code, + "step_name": _step_label(normalized_step_code), + "items_total": 0, + "items_pending": 0, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "started_recent": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + } + + def _extract_current_cycle_events(events: list[dict]) -> tuple[str, list[dict]]: if not events: return "", [] @@ -313,6 +768,342 @@ def _extract_current_cycle_events(events: list[dict]) -> tuple[str, list[dict]]: return cycle_token, current_cycle +def _classify_runtime_debug_event(*, event_type: str, message: str, payload: dict) -> dict: + normalized_event_type = str(event_type or "").strip() + normalized_message = str(message or "").strip() + normalized_payload = payload if isinstance(payload, dict) else {} + normalized_job_code = str(normalized_payload.get("job_code") or "").strip() + normalized_node_code = str(normalized_payload.get("node_code") or "").strip() + result = { + "job_code": normalized_job_code, + "node_code": normalized_node_code, + "started": False, + "terminal": False, + "terminal_status": "", + "step_code": "", + } + + if normalized_event_type in {"domain_started", "domain_completed", "domain_failed", "domain_blacklisted"}: + result["started"] = normalized_event_type == "domain_started" + result["terminal"] = normalized_event_type in {"domain_completed", "domain_failed", "domain_blacklisted"} + terminal_map = { + "domain_completed": "completed", + "domain_failed": "failed", + "domain_blacklisted": "blacklisted", + } + result["terminal_status"] = terminal_map.get(normalized_event_type, "") + result["step_code"] = normalize_detect_step_code(normalized_payload.get("detect_key")) or "detect_register" + return result + + if normalized_event_type != "worker_log": + return result + + step_markers = ( + ("detect_register", ("| step=注册状态检测 |", "detect_register")), + ("detect_baidu_site", ("| step=百度site检测 |", "detect_baidu_site", "百度阶段", "detect.baidu")), + ("detect_360_site", ("| step=360收录检测 |", "detect_360_site", "360阶段", "detect.c360")), + ("detect_chinaz", ("| step=站长之家检测 |", "detect_chinaz", "站长之家", "detect.chinaz")), + ("detect_aizhan", ("| step=爱站检测 |", "detect_aizhan", "爱站", "detect.aizhan")), + ("detect_wayback", ("| step=时光机检测 |", "detect_wayback", "时光机检测")), + ("detect_jucha", ("| step=聚查检测 |", "detect_jucha", "聚查", "detect.jucha")), + ("detect_juziseo", ("| step=桔子检测 |", "detect_juziseo", "桔子", "juziseo")), + ) + for step_code, markers in step_markers: + if any(marker in normalized_message for marker in markers): + result["step_code"] = step_code + break + + if "| stage=started |" in normalized_message or "开始检测域名:" in normalized_message: + result["started"] = True + return result + + if "| stage=single_step_finalized |" in normalized_message: + result["terminal"] = True + if "final_status=completed" in normalized_message: + result["terminal_status"] = "completed" + elif "final_status=failed" in normalized_message: + result["terminal_status"] = "failed" + elif "final_status=blacklisted" in normalized_message: + result["terminal_status"] = "blacklisted" + return result + + return result + + +def _load_runtime_activity_snapshot(window_minutes: int) -> dict: + safe_window_minutes = max(5, min(int(window_minutes or 15), 120)) + snapshot = { + "processed_recent": 0, + "completed_recent": 0, + "failed_recent": 0, + "blacklisted_recent": 0, + "focus_job_code": "", + "job_codes": [], + "step_code": "detect_register", + "step_stats": {}, + "nodes": {}, + } + if not (settings.node_region == "overseas" and settings.node_role == "control"): + return snapshot + + if settings.node_region == "overseas" and settings.node_role == "control": + source_region = "mainland" + else: + source_region = _normalize_sync_region(settings.sync_source_region, settings.node_region) + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT + COALESCE(NULLIF(node_code, ''), 'unknown') AS node_code, + event_type, + message, + payload_json, + created_at + FROM detect_debug_events + WHERE source_region = %s + AND created_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + AND event_type IN ('worker_log', 'domain_started', 'domain_completed', 'domain_failed', 'domain_blacklisted') + ORDER BY created_at DESC, id DESC + LIMIT 2000 + """, + (source_region, safe_window_minutes), + ) + rows = cur.fetchall() + + job_activity: dict[str, dict] = {} + node_stats: dict[str, dict] = {} + step_activity: dict[str, dict] = {} + for row in rows: + node_code = str(row[0] or "unknown").strip() or "unknown" + event_type = str(row[1] or "").strip() + message = str(row[2] or "").strip() + payload = _decode_payload(row[3]) + created_at = _format_time(row[4]) + classified = _classify_runtime_debug_event(event_type=event_type, message=message, payload=payload) + job_code = str(classified.get("job_code") or "").strip() + step_code = str(classified.get("step_code") or "").strip() or "detect_register" + + if job_code: + bucket = job_activity.setdefault( + job_code, + { + "job_code": job_code, + "activity_count": 0, + "started_count": 0, + "terminal_count": 0, + "latest_at": "", + }, + ) + bucket["activity_count"] += 1 + if classified.get("started"): + bucket["started_count"] += 1 + if classified.get("terminal"): + bucket["terminal_count"] += 1 + if created_at > str(bucket.get("latest_at") or ""): + bucket["latest_at"] = created_at + + if classified.get("terminal"): + snapshot["processed_recent"] += 1 + step_bucket = step_activity.setdefault( + step_code, + { + "step_code": step_code, + "started_recent": 0, + "processed_recent": 0, + "completed_recent": 0, + "failed_recent": 0, + "blacklisted_recent": 0, + }, + ) + step_bucket["processed_recent"] += 1 + node_bucket = node_stats.setdefault( + node_code, + { + "node_code": node_code, + "processed_recent": 0, + "completed_recent": 0, + "failed_recent": 0, + "blacklisted_recent": 0, + }, + ) + node_bucket["processed_recent"] += 1 + terminal_status = str(classified.get("terminal_status") or "").strip() + if terminal_status == "completed": + snapshot["completed_recent"] += 1 + node_bucket["completed_recent"] += 1 + step_bucket["completed_recent"] += 1 + elif terminal_status == "failed": + snapshot["failed_recent"] += 1 + node_bucket["failed_recent"] += 1 + step_bucket["failed_recent"] += 1 + elif terminal_status == "blacklisted": + snapshot["blacklisted_recent"] += 1 + node_bucket["blacklisted_recent"] += 1 + step_bucket["blacklisted_recent"] += 1 + + if classified.get("started"): + step_bucket = step_activity.setdefault( + step_code, + { + "step_code": step_code, + "started_recent": 0, + "processed_recent": 0, + "completed_recent": 0, + "failed_recent": 0, + "blacklisted_recent": 0, + }, + ) + step_bucket["started_recent"] += 1 + + if step_activity: + ordered_steps = sorted( + step_activity.values(), + key=lambda item: ( + -int(item.get("processed_recent", 0) or 0), + -int(item.get("started_recent", 0) or 0), + str(item.get("step_code") or ""), + ), + ) + snapshot["step_code"] = str((ordered_steps[0] or {}).get("step_code") or "detect_register") + snapshot["step_stats"] = { + str(item.get("step_code") or ""): { + "step_code": str(item.get("step_code") or ""), + "started_recent": int(item.get("started_recent", 0) or 0), + "processed_recent": int(item.get("processed_recent", 0) or 0), + "completed_recent": int(item.get("completed_recent", 0) or 0), + "failed_recent": int(item.get("failed_recent", 0) or 0), + "blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0), + } + for item in ordered_steps + if str(item.get("step_code") or "").strip() + } + if job_activity: + ordered_jobs = sorted( + job_activity.values(), + key=lambda item: ( + -int(item.get("terminal_count", 0) or 0), + -int(item.get("activity_count", 0) or 0), + str(item.get("latest_at") or ""), + str(item.get("job_code") or ""), + ), + ) + snapshot["focus_job_code"] = str(ordered_jobs[0].get("job_code") or "") + snapshot["job_codes"] = [str(item.get("job_code") or "") for item in ordered_jobs[:5] if str(item.get("job_code") or "")] + snapshot["nodes"] = node_stats + return snapshot + + +@db_read_retry() +def _load_latest_runtime_active_job_snapshot(window_minutes: int) -> dict: + safe_window_minutes = max(5, min(int(window_minutes or 15), 120)) + if not (settings.node_region == "overseas" and settings.node_role == "control"): + return {} + + source_region = "mainland" + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT payload_json, created_at + FROM detect_debug_events + WHERE source_region = %s + AND service = 'detect-runtime' + AND event_type = 'active_job_snapshot' + AND created_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + ORDER BY created_at DESC, id DESC + LIMIT 1 + """, + (source_region, safe_window_minutes), + ) + row = cur.fetchone() + + if not row: + return {} + + payload = _decode_payload(row[0]) + if not isinstance(payload, dict): + return {} + payload["_created_at"] = _format_time(row[1]) + return payload + + +def _select_runtime_snapshot_events( + runtime_snapshot: dict, + *, + job_code: str = "", + job_id: int = 0, + event_limit: int = 20, +) -> list[dict]: + safe_limit = max(1, min(int(event_limit or 20), 100)) + target_job_code = str(job_code or "").strip() + target_job_id = int(job_id or 0) + selected: list[dict] = [] + for raw_event in list(runtime_snapshot.get("recent_events") or []): + if not isinstance(raw_event, dict): + continue + payload = raw_event.get("payload") + normalized_payload = payload if isinstance(payload, dict) else _decode_payload(payload) + event_job_code = str(normalized_payload.get("job_code") or "").strip() + event_job_id = _int_value(raw_event.get("job_id")) + if target_job_code: + if event_job_code == target_job_code or (target_job_id > 0 and event_job_id == target_job_id): + selected.append( + { + "node_code": str(raw_event.get("node_code") or "").strip(), + "event_type": str(raw_event.get("event_type") or "").strip(), + "level": str(raw_event.get("level") or "info").strip() or "info", + "message": str(raw_event.get("message") or "").strip(), + "payload": normalized_payload, + "created_at": str(raw_event.get("created_at") or "").strip(), + } + ) + else: + selected.append( + { + "node_code": str(raw_event.get("node_code") or "").strip(), + "event_type": str(raw_event.get("event_type") or "").strip(), + "level": str(raw_event.get("level") or "info").strip() or "info", + "message": str(raw_event.get("message") or "").strip(), + "payload": normalized_payload, + "created_at": str(raw_event.get("created_at") or "").strip(), + } + ) + if len(selected) >= safe_limit: + break + return selected + + +def _filter_events_for_job(events: list[dict], *, job_code: str = "", job_id: int = 0, limit: int = 20) -> list[dict]: + target_job_code = str(job_code or "").strip() + target_job_id = int(job_id or 0) + safe_limit = max(1, min(int(limit or 20), 100)) + filtered: list[dict] = [] + for raw_event in list(events or []): + if not isinstance(raw_event, dict): + continue + payload = raw_event.get("payload") + normalized_payload = payload if isinstance(payload, dict) else _decode_payload(payload) + event_job_code = str(normalized_payload.get("job_code") or "").strip() + event_job_id = _int_value(raw_event.get("job_id")) + if target_job_code: + if event_job_code != target_job_code and (target_job_id <= 0 or event_job_id != target_job_id): + continue + filtered.append( + { + "node_code": str(raw_event.get("node_code") or "").strip(), + "event_type": str(raw_event.get("event_type") or "").strip(), + "level": str(raw_event.get("level") or "info").strip() or "info", + "message": str(raw_event.get("message") or "").strip(), + "payload": normalized_payload, + "created_at": str(raw_event.get("created_at") or "").strip(), + } + ) + if len(filtered) >= safe_limit: + break + return filtered + + def _fetch_job_summary(cur, job_row, event_limit: int = 20) -> dict: job_id = job_row[0] cur.execute( @@ -357,6 +1148,24 @@ def _fetch_job_summary(cur, job_row, event_limit: int = 20) -> dict: bucket[field_name] += int(count) bucket["items_total"] += int(count) + cur.execute( + """ + SELECT COALESCE(NULLIF(step_code, ''), 'domain_pipeline') AS step_code, status, count(*) + FROM detect_job_items + WHERE job_id = %s + GROUP BY COALESCE(NULLIF(step_code, ''), 'domain_pipeline'), status + ORDER BY step_code ASC, status ASC + """, + (job_id,), + ) + step_buckets: dict[str, dict] = {} + for step_code, status, count in cur.fetchall(): + bucket = step_buckets.setdefault(str(step_code or ""), _build_step_bucket(str(step_code or ""))) + field_name = f"items_{status}" + if field_name in bucket: + bucket[field_name] += int(count) + bucket["items_total"] += int(count) + cur.execute( """ SELECT node_code, event_type, level, message, payload_json, created_at @@ -398,18 +1207,25 @@ def _fetch_job_summary(cur, job_row, event_limit: int = 20) -> dict: effective_summary = _build_effective_summary( node_stats=effective_node_stats, raw_items_total=raw_total, + raw_items_pending=raw_pending, + raw_items_claimed=raw_claimed, + raw_items_running=raw_running, + raw_items_completed=raw_completed, raw_items_blacklisted=raw_blacklisted, + raw_items_failed=raw_failed, ) display_summary = _build_display_summary(effective_node_stats) return { "job_id": job_id, "job_code": job_row[1], "source": job_row[2], - "status": job_row[3], - "created_by": job_row[4], - "created_at": _format_time(job_row[5]), - "started_at": _format_time(job_row[6]), - "finished_at": _format_time(job_row[7]), + "task_mode": job_row[3] or _DEFAULT_TASK_MODE, + "step_code": job_row[4] or "", + "status": job_row[5], + "created_by": job_row[6], + "created_at": _format_time(job_row[7]), + "started_at": _format_time(job_row[8]), + "finished_at": _format_time(job_row[9]), "items_total": int(effective_summary.get("items_total", 0) or 0), "items_pending": int(effective_summary.get("items_pending", 0) or 0), "items_claimed": int(effective_summary.get("items_claimed", 0) or 0), @@ -431,10 +1247,22 @@ def _fetch_job_summary(cur, job_row, event_limit: int = 20) -> dict: "raw_items_failed": raw_failed, "raw_items_terminal": raw_terminal, "raw_node_stats": list(node_buckets.values()), + "raw_step_stats": list(step_buckets.values()), "node_stats": effective_node_stats, "distributed_node_stats": effective_node_stats, + "step_stats": sorted( + step_buckets.values(), + key=lambda item: ( + -_int_value(item.get("items_running")), + -_int_value(item.get("items_pending")), + str(item.get("step_code") or ""), + ), + ), "display_items_claimed": int(display_summary.get("items_claimed", 0) or 0), - "display_items_running": int(display_summary.get("items_running", 0) or 0), + "display_items_running": int(display_summary.get("display_running", display_summary.get("items_running", 0)) or 0), + "display_current_load": int(display_summary.get("current_load", 0) or 0), + "display_active_threads": int(display_summary.get("active_threads", 0) or 0), + "display_max_threads": int(display_summary.get("max_threads", 0) or 0), "display_items_completed": int(display_summary.get("items_completed", 0) or 0), "display_items_failed": int(display_summary.get("items_failed", 0) or 0), "display_active_node_codes": list(display_summary.get("active_nodes") or []), @@ -446,12 +1274,164 @@ def _fetch_job_summary(cur, job_row, event_limit: int = 20) -> dict: } +def _enrich_active_job_summary_with_runtime( + summary: dict | None, + *, + event_limit: int = 20, + window_minutes: int = 15, + runtime_activity: dict | None = None, + runtime_snapshot: dict | None = None, + recent_domain_events: list[dict] | None = None, +) -> dict | None: + if not summary: + return summary + enriched = dict(summary) + runtime_activity = dict(runtime_activity or _load_runtime_activity_snapshot(window_minutes)) + runtime_snapshot = dict(runtime_snapshot or _load_latest_runtime_active_job_snapshot(window_minutes)) + runtime_job_code = str(runtime_activity.get("focus_job_code") or "").strip() + runtime_job_codes = [ + str(item or "").strip() + for item in list(runtime_activity.get("job_codes") or []) + if str(item or "").strip() + ] + if runtime_job_code: + enriched["runtime_job_code"] = runtime_job_code + if runtime_job_codes: + enriched["runtime_job_codes"] = runtime_job_codes + + enriched["processed_recent"] = max( + int(enriched.get("processed_recent", 0) or 0), + int(runtime_activity.get("processed_recent", 0) or 0), + ) + enriched["processed_per_minute"] = max( + float(enriched.get("processed_per_minute", 0) or 0), + round(int(runtime_activity.get("processed_recent", 0) or 0) / max(1, int(window_minutes or 15)), 2), + ) + enriched["completed_recent"] = max( + int(enriched.get("completed_recent", 0) or 0), + int(runtime_activity.get("completed_recent", 0) or 0), + ) + enriched["failed_recent"] = max( + int(enriched.get("failed_recent", 0) or 0), + int(runtime_activity.get("failed_recent", 0) or 0), + ) + enriched["blacklisted_recent"] = max( + int(enriched.get("blacklisted_recent", 0) or 0), + int(runtime_activity.get("blacklisted_recent", 0) or 0), + ) + + if recent_domain_events is not None: + enriched["recent_domain_events"] = list(recent_domain_events) + elif not enriched.get("recent_domain_events"): + try: + from app.services.sync_record_service import _collect_recent_domain_events + + enriched["recent_domain_events"] = _collect_recent_domain_events( + enriched, + limit=max(10, min(int(event_limit or 20), 60)), + ) + except Exception: + enriched["recent_domain_events"] = [] + if runtime_snapshot: + runtime_job = dict(runtime_snapshot.get("job") or {}) + runtime_queue = dict((runtime_snapshot.get("queue_health") or {}).get("queue") or {}) + runtime_nodes = list((runtime_snapshot.get("queue_health") or {}).get("nodes") or []) + runtime_steps = list((runtime_snapshot.get("queue_health") or {}).get("steps") or []) + runtime_job_nodes = list(runtime_job.get("node_stats") or []) + if runtime_job: + runtime_snapshot_job_id = int(runtime_job.get("job_id", 0) or 0) + enriched["runtime_snapshot_job_id"] = runtime_snapshot_job_id + runtime_snapshot_job_code = str(runtime_job.get("job_code") or "").strip() + enriched["runtime_snapshot_job_code"] = runtime_snapshot_job_code + if runtime_snapshot_job_code: + enriched["job_code"] = runtime_snapshot_job_code + enriched["runtime_job_code"] = runtime_snapshot_job_code + merged_runtime_codes = [runtime_snapshot_job_code] + [ + item for item in list(enriched.get("runtime_job_codes") or []) if str(item or "").strip() != runtime_snapshot_job_code + ] + enriched["runtime_job_codes"] = merged_runtime_codes[:5] + if runtime_snapshot_job_id > 0: + enriched["job_id"] = runtime_snapshot_job_id + runtime_status = str(runtime_job.get("status") or "").strip() + if runtime_status: + enriched["status"] = runtime_status + if float(runtime_job.get("progress_percent", 0) or 0) > float(enriched.get("progress_percent", 0) or 0): + enriched["progress_percent"] = float(runtime_job.get("progress_percent", 0) or 0) + if runtime_queue: + enriched["runtime_snapshot_queue"] = runtime_queue + runtime_items_total = int(runtime_queue.get("items_total", 0) or 0) + if runtime_items_total > 0: + enriched["items_total"] = runtime_items_total + enriched["items_pending"] = int(runtime_queue.get("pending", 0) or 0) + enriched["items_claimed"] = int(runtime_queue.get("claimed", 0) or 0) + enriched["items_running"] = int(runtime_queue.get("running", 0) or 0) + enriched["items_completed"] = int(runtime_queue.get("completed", 0) or 0) + enriched["items_blacklisted"] = int(runtime_queue.get("blacklisted", 0) or 0) + enriched["items_failed"] = int(runtime_queue.get("failed", 0) or 0) + enriched["items_terminal"] = int(runtime_queue.get("terminal", 0) or 0) + enriched["display_items_claimed"] = int(runtime_queue.get("display_claimed", enriched.get("items_claimed", 0)) or 0) + enriched["display_items_running"] = int(runtime_queue.get("display_running", enriched.get("items_running", 0)) or 0) + if runtime_job_nodes: + enriched["node_stats"] = list(runtime_job_nodes) + enriched["distributed_node_stats"] = list(runtime_job_nodes) + runtime_display_summary = _build_display_summary(list(runtime_job_nodes)) + enriched["display_items_claimed"] = int(runtime_display_summary.get("items_claimed", enriched.get("display_items_claimed", 0)) or 0) + enriched["display_items_running"] = int(runtime_display_summary.get("display_running", enriched.get("display_items_running", 0)) or 0) + enriched["display_current_load"] = int(runtime_display_summary.get("current_load", enriched.get("display_current_load", 0)) or 0) + enriched["display_active_threads"] = int(runtime_display_summary.get("active_threads", enriched.get("display_active_threads", 0)) or 0) + enriched["display_max_threads"] = int(runtime_display_summary.get("max_threads", enriched.get("display_max_threads", 0)) or 0) + enriched["display_items_completed"] = int(runtime_display_summary.get("items_completed", enriched.get("display_items_completed", 0)) or 0) + enriched["display_items_failed"] = int(runtime_display_summary.get("items_failed", enriched.get("display_items_failed", 0)) or 0) + enriched["display_active_node_codes"] = list(runtime_display_summary.get("active_nodes") or []) + elif runtime_nodes: + enriched["node_stats"] = list(runtime_nodes) + enriched["distributed_node_stats"] = list(runtime_nodes) + if runtime_steps: + enriched["step_stats"] = list(runtime_steps) + runtime_recent_events = _select_runtime_snapshot_events( + runtime_snapshot, + job_code=str(runtime_job.get("job_code") or "").strip(), + job_id=int(runtime_job.get("job_id", 0) or 0), + event_limit=event_limit, + ) + if runtime_recent_events: + enriched["recent_events"] = runtime_recent_events + current_cycle_token, current_cycle_events = _extract_current_cycle_events(runtime_recent_events) + enriched["current_cycle_token"] = current_cycle_token + enriched["current_cycle_events"] = current_cycle_events + enriched["latest_event"] = runtime_recent_events[0] + enriched["latest_cycle_event"] = current_cycle_events[0] if current_cycle_events else runtime_recent_events[0] + aligned_job_code = str(enriched.get("job_code") or enriched.get("runtime_job_code") or "").strip() + aligned_job_id = int(enriched.get("job_id", 0) or 0) + aligned_recent_events = _filter_events_for_job( + list(enriched.get("recent_events") or []), + job_code=aligned_job_code, + job_id=aligned_job_id, + limit=event_limit, + ) + if list(enriched.get("recent_events") or []) and not aligned_recent_events: + enriched["recent_events"] = [] + enriched["current_cycle_token"] = "" + enriched["current_cycle_events"] = [] + enriched["latest_event"] = None + enriched["latest_cycle_event"] = None + elif aligned_recent_events: + enriched["recent_events"] = aligned_recent_events + current_cycle_token, current_cycle_events = _extract_current_cycle_events(aligned_recent_events) + enriched["current_cycle_token"] = current_cycle_token + enriched["current_cycle_events"] = current_cycle_events + enriched["latest_event"] = aligned_recent_events[0] + enriched["latest_cycle_event"] = current_cycle_events[0] if current_cycle_events else aligned_recent_events[0] + return enriched + + +@db_read_retry() def get_detect_job_summary(job_id: int, event_limit: int = 20) -> dict | None: with get_db() as conn: with conn.cursor() as cur: cur.execute( """ - SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at + SELECT id, job_code, source, task_mode, step_code, status, created_by, created_at, started_at, finished_at FROM detect_jobs WHERE id = %s LIMIT 1 @@ -461,25 +1441,154 @@ def get_detect_job_summary(job_id: int, event_limit: int = 20) -> dict | None: row = cur.fetchone() if not row: return None - return _fetch_job_summary(cur, row, event_limit=event_limit) + return _enrich_active_job_summary_with_runtime(_fetch_job_summary(cur, row, event_limit=event_limit), event_limit=event_limit) +@db_read_retry() def get_active_detect_job_summary(event_limit: int = 20) -> dict | None: with get_db() as conn: with conn.cursor() as cur: cur.execute( """ - SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at + SELECT id, job_code, source, task_mode, step_code, status, created_by, created_at, started_at, finished_at FROM detect_jobs WHERE status IN ('pending', 'running') - ORDER BY created_at DESC, id DESC + ORDER BY + CASE WHEN status = 'running' THEN 0 ELSE 1 END ASC, + COALESCE(started_at, created_at) DESC, + id DESC LIMIT 1 """ ) row = cur.fetchone() if not row: return None - return _fetch_job_summary(cur, row, event_limit=event_limit) + return _enrich_active_job_summary_with_runtime(_fetch_job_summary(cur, row, event_limit=event_limit), event_limit=event_limit) + + +@db_read_retry() +def get_latest_detect_job_summary( + *, + event_limit: int = 20, + statuses: tuple[str, ...] | list[str] | None = None, + recent_minutes: int | None = None, +) -> dict | None: + normalized_statuses = [ + str(item or "").strip() + for item in list(statuses or []) + if str(item or "").strip() + ] + if not normalized_statuses: + normalized_statuses = ["pending", "running", "completed", "partial_failed", "failed"] + + where_clauses = ["status = ANY(%s)"] + params: list[object] = [normalized_statuses] + if recent_minutes is not None: + safe_recent_minutes = max(1, min(int(recent_minutes or 1), 24 * 60)) + where_clauses.append( + """ + COALESCE(finished_at, started_at, created_at) >= + CURRENT_TIMESTAMP - (%s || ' minutes')::interval + """ + ) + params.append(safe_recent_minutes) + + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + f""" + SELECT id, job_code, source, task_mode, step_code, status, created_by, created_at, started_at, finished_at + FROM detect_jobs + WHERE {' AND '.join(where_clauses)} + ORDER BY + CASE + WHEN status = 'running' THEN 0 + WHEN status = 'pending' THEN 1 + WHEN status = 'completed' THEN 2 + WHEN status = 'partial_failed' THEN 3 + WHEN status = 'failed' THEN 4 + ELSE 5 + END ASC, + COALESCE(finished_at, started_at, created_at) DESC, + id DESC + LIMIT 1 + """, + tuple(params), + ) + row = cur.fetchone() + if not row: + return None + return _enrich_active_job_summary_with_runtime(_fetch_job_summary(cur, row, event_limit=event_limit), event_limit=event_limit) + + +@db_read_retry() +def get_latest_unprojected_detect_job_summary( + *, + event_limit: int = 20, + statuses: tuple[str, ...] | list[str] | None = None, + recent_minutes: int | None = None, + source_region: str | None = None, + target_region: str | None = None, +) -> dict | None: + normalized_statuses = [ + str(item or "").strip() + for item in list(statuses or []) + if str(item or "").strip() + ] + if not normalized_statuses: + normalized_statuses = ["completed", "partial_failed", "failed"] + + normalized_source_region = _normalize_sync_region( + source_region, + _normalize_sync_region(settings.sync_source_region, settings.node_region), + ) + normalized_target_region = _normalize_sync_region( + target_region, + _normalize_sync_region(settings.sync_target_region, "overseas"), + ) + + where_clauses = [ + "job.status = ANY(%s)", + """ + NOT EXISTS ( + SELECT 1 + FROM detect_sync_records AS sync_record + WHERE sync_record.sync_type = 'detect_result_projection' + AND sync_record.source_region = %s + AND sync_record.target_region = %s + AND (sync_record.payload_json->'projection'->'job'->>'job_id') = job.id::text + ) + """, + ] + params: list[object] = [normalized_statuses, normalized_source_region, normalized_target_region] + if recent_minutes is not None: + safe_recent_minutes = max(1, min(int(recent_minutes or 1), 24 * 60)) + where_clauses.append( + """ + COALESCE(job.finished_at, job.started_at, job.created_at) >= + CURRENT_TIMESTAMP - (%s || ' minutes')::interval + """ + ) + params.append(safe_recent_minutes) + + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + f""" + SELECT job.id, job.job_code, job.source, job.task_mode, job.step_code, job.status, job.created_by, job.created_at, job.started_at, job.finished_at + FROM detect_jobs AS job + WHERE {' AND '.join(where_clauses)} + ORDER BY + COALESCE(job.finished_at, job.started_at, job.created_at) DESC, + job.id DESC + LIMIT 1 + """, + tuple(params), + ) + row = cur.fetchone() + if not row: + return None + return _enrich_active_job_summary_with_runtime(_fetch_job_summary(cur, row, event_limit=event_limit), event_limit=event_limit) def list_detect_jobs(limit: int = 20) -> list[dict]: @@ -487,7 +1596,7 @@ def list_detect_jobs(limit: int = 20) -> list[dict]: with conn.cursor() as cur: cur.execute( """ - SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at + SELECT id, job_code, source, task_mode, step_code, status, created_by, created_at, started_at, finished_at FROM detect_jobs ORDER BY created_at DESC, id DESC LIMIT %s @@ -526,9 +1635,12 @@ def list_recent_detect_run_events(limit: int = 20) -> list[dict]: ] +@db_read_retry() def get_detect_queue_health(window_minutes: int = 15) -> dict: window_minutes = max(5, min(int(window_minutes or 15), 120)) active_job = get_active_detect_job_summary(event_limit=10) + runtime_activity = _load_runtime_activity_snapshot(window_minutes) + runtime_snapshot = _load_latest_runtime_active_job_snapshot(window_minutes) if not active_job: return { "window_minutes": window_minutes, @@ -551,13 +1663,15 @@ def get_detect_queue_health(window_minutes: int = 15) -> dict: "expiring_soon_leases": 0, }, "throughput": { - "processed_recent": 0, - "processed_per_minute": 0, - "completed_recent": 0, - "blacklisted_recent": 0, - "failed_recent": 0, + "processed_recent": int(runtime_activity.get("processed_recent", 0) or 0), + "processed_per_minute": round(int(runtime_activity.get("processed_recent", 0) or 0) / window_minutes, 2), + "completed_recent": int(runtime_activity.get("completed_recent", 0) or 0), + "blacklisted_recent": int(runtime_activity.get("blacklisted_recent", 0) or 0), + "failed_recent": int(runtime_activity.get("failed_recent", 0) or 0), }, "nodes": [], + "steps": [], + "runtime_activity": runtime_activity, } job_id = int(active_job["job_id"]) @@ -601,6 +1715,36 @@ def get_detect_queue_health(window_minutes: int = 15) -> dict: ) throughput_rows = cur.fetchall() + cur.execute( + """ + SELECT + COALESCE(NULLIF(step_code, ''), 'domain_pipeline') AS step_code, + COUNT(*) FILTER ( + WHERE status IN ('completed', 'blacklisted', 'failed') + AND finished_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + ) AS processed_recent, + COUNT(*) FILTER ( + WHERE status = 'completed' + AND finished_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + ) AS completed_recent, + COUNT(*) FILTER ( + WHERE status = 'blacklisted' + AND finished_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + ) AS blacklisted_recent, + COUNT(*) FILTER ( + WHERE status = 'failed' + AND finished_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + ) AS failed_recent + FROM detect_job_items + WHERE job_id = %s + GROUP BY COALESCE(NULLIF(step_code, ''), 'domain_pipeline') + ORDER BY step_code ASC + """, + (window_minutes, window_minutes, window_minutes, window_minutes, job_id), + ) + step_throughput_rows = cur.fetchall() + runtime_display_rows = _load_runtime_display_rows(cur) + oldest_pending_at = _format_time(lease_row[0]) if lease_row and lease_row[0] else "" nearest_lease_expiry_at = _format_time(lease_row[1]) if lease_row and lease_row[1] else "" oldest_pending_age_minutes = 0 @@ -650,6 +1794,19 @@ def get_detect_queue_health(window_minutes: int = 15) -> dict: total_completed_recent = 0 total_blacklisted_recent = 0 total_failed_recent = 0 + step_map = { + str(item.get("step_code") or "domain_pipeline"): { + **_build_step_bucket(str(item.get("step_code") or "domain_pipeline")), + "items_total": _int_value(item.get("items_total")), + "items_pending": _int_value(item.get("items_pending")), + "items_claimed": _int_value(item.get("items_claimed")), + "items_running": _int_value(item.get("items_running")), + "items_completed": _int_value(item.get("items_completed")), + "items_blacklisted": _int_value(item.get("items_blacklisted")), + "items_failed": _int_value(item.get("items_failed")), + } + for item in active_job.get("step_stats") or [] + } for row in throughput_rows: node_code = str(row[0] or "unknown") bucket = node_map.setdefault( @@ -684,35 +1841,403 @@ def get_detect_queue_health(window_minutes: int = 15) -> dict: total_blacklisted_recent += blacklisted_recent total_failed_recent += failed_recent + for row in list(runtime_display_rows or []): + runtime_bucket = _build_runtime_display_bucket(row) + if not runtime_bucket: + continue + node_code = str(runtime_bucket.get("node_code") or "unknown") + bucket = node_map.setdefault( + node_code, + { + "node_code": node_code, + "items_total": 0, + "items_pending": 0, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + "metrics_source": "runtime", + "display_running": 0, + "current_load": 0, + "active_threads": 0, + "max_threads": 0, + "region": "", + "role": "", + "status": "", + "last_heartbeat_at": "", + }, + ) + runtime_display_running = _effective_runtime_load( + items_running=runtime_bucket.get("items_running"), + active_threads=runtime_bucket.get("active_threads"), + ) + for key in ( + "items_total", + "items_pending", + "items_claimed", + "items_completed", + "items_blacklisted", + "items_failed", + "current_load", + "active_threads", + "max_threads", + ): + bucket[key] = max(_int_value(bucket.get(key)), _int_value(runtime_bucket.get(key))) + bucket["display_running"] = max( + _int_value(bucket.get("display_running")), + runtime_display_running, + ) + bucket["items_running"] = max( + _int_value(bucket.get("items_running")), + _int_value(runtime_bucket.get("items_running")), + ) + for key in ("metrics_source", "region", "role", "status", "last_heartbeat_at"): + if str(runtime_bucket.get(key) or "").strip(): + bucket[key] = runtime_bucket.get(key) + + for node_code, runtime_bucket in dict(runtime_activity.get("nodes") or {}).items(): + bucket = node_map.setdefault( + node_code, + { + "node_code": node_code, + "items_total": 0, + "items_pending": 0, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + "metrics_source": "runtime", + "display_running": 0, + "current_load": 0, + "active_threads": 0, + "max_threads": 0, + "region": "", + "role": "", + "status": "", + "last_heartbeat_at": "", + }, + ) + runtime_processed_recent = int(runtime_bucket.get("processed_recent", 0) or 0) + if runtime_processed_recent > int(bucket.get("processed_recent", 0) or 0): + bucket["processed_recent"] = runtime_processed_recent + bucket["processed_per_minute"] = round(runtime_processed_recent / window_minutes, 2) + bucket["completed_recent"] = int(runtime_bucket.get("completed_recent", 0) or 0) + bucket["blacklisted_recent"] = int(runtime_bucket.get("blacklisted_recent", 0) or 0) + bucket["failed_recent"] = int(runtime_bucket.get("failed_recent", 0) or 0) + + for row in step_throughput_rows: + step_code = str(row[0] or "domain_pipeline") + bucket = step_map.setdefault(step_code, _build_step_bucket(step_code)) + processed_recent = int(row[1] or 0) + completed_recent = int(row[2] or 0) + blacklisted_recent = int(row[3] or 0) + failed_recent = int(row[4] or 0) + bucket["processed_recent"] = processed_recent + bucket["processed_per_minute"] = round(processed_recent / window_minutes, 2) + bucket["completed_recent"] = completed_recent + bucket["blacklisted_recent"] = blacklisted_recent + bucket["failed_recent"] = failed_recent + + runtime_step_stats = dict(runtime_activity.get("step_stats") or {}) + for runtime_step_code, runtime_step_bucket in runtime_step_stats.items(): + normalized_step_code = str(runtime_step_code or "").strip() + if not normalized_step_code: + continue + bucket = step_map.setdefault(normalized_step_code, _build_step_bucket(normalized_step_code)) + runtime_processed_recent = int(runtime_step_bucket.get("processed_recent", 0) or 0) + runtime_started_recent = int(runtime_step_bucket.get("started_recent", 0) or 0) + bucket["started_recent"] = max(int(bucket.get("started_recent", 0) or 0), runtime_started_recent) + if runtime_processed_recent > int(bucket.get("processed_recent", 0) or 0): + bucket["processed_recent"] = runtime_processed_recent + bucket["processed_per_minute"] = round(runtime_processed_recent / window_minutes, 2) + bucket["completed_recent"] = int(runtime_step_bucket.get("completed_recent", 0) or 0) + bucket["blacklisted_recent"] = int(runtime_step_bucket.get("blacklisted_recent", 0) or 0) + bucket["failed_recent"] = int(runtime_step_bucket.get("failed_recent", 0) or 0) + elif int(bucket.get("processed_recent", 0) or 0) <= 0 and runtime_started_recent > 0: + bucket["processed_recent"] = runtime_started_recent + bucket["processed_per_minute"] = round(runtime_started_recent / window_minutes, 2) + + runtime_step_code = str(runtime_activity.get("step_code") or "").strip() + runtime_processed_recent = int(runtime_activity.get("processed_recent", 0) or 0) + if runtime_step_code and runtime_processed_recent > 0 and runtime_step_code not in runtime_step_stats: + runtime_bucket = step_map.setdefault(runtime_step_code, _build_step_bucket(runtime_step_code)) + if runtime_processed_recent > int(runtime_bucket.get("processed_recent", 0) or 0): + runtime_bucket["processed_recent"] = runtime_processed_recent + runtime_bucket["processed_per_minute"] = round(runtime_processed_recent / window_minutes, 2) + runtime_bucket["completed_recent"] = int(runtime_activity.get("completed_recent", 0) or 0) + runtime_bucket["blacklisted_recent"] = int(runtime_activity.get("blacklisted_recent", 0) or 0) + runtime_bucket["failed_recent"] = int(runtime_activity.get("failed_recent", 0) or 0) + + total_processed_recent = max(total_processed_recent, int(runtime_activity.get("processed_recent", 0) or 0)) + total_completed_recent = max(total_completed_recent, int(runtime_activity.get("completed_recent", 0) or 0)) + total_blacklisted_recent = max(total_blacklisted_recent, int(runtime_activity.get("blacklisted_recent", 0) or 0)) + total_failed_recent = max(total_failed_recent, int(runtime_activity.get("failed_recent", 0) or 0)) + + runtime_snapshot_job = dict(runtime_snapshot.get("job") or {}) + runtime_snapshot_queue = dict((runtime_snapshot.get("queue_health") or {}).get("queue") or {}) + runtime_snapshot_nodes = list((runtime_snapshot.get("queue_health") or {}).get("nodes") or []) + runtime_snapshot_steps = list((runtime_snapshot.get("queue_health") or {}).get("steps") or []) + runtime_snapshot_job_code = str(runtime_snapshot_job.get("job_code") or "").strip() + + items_total = int(active_job.get("items_total", 0) or 0) + pending_items = int(active_job.get("items_pending", 0) or 0) + claimed_items = int(active_job.get("items_claimed", 0) or 0) + running_items = int(active_job.get("items_running", 0) or 0) + completed_items = int(active_job.get("items_completed", 0) or 0) + blacklisted_items = int(active_job.get("items_blacklisted", 0) or 0) + failed_items = int(active_job.get("items_failed", 0) or 0) + display_claimed = int(active_job.get("display_items_claimed", active_job.get("items_claimed", 0)) or 0) + display_running = int( + active_job.get("display_active_threads", active_job.get("display_items_running", active_job.get("items_running", 0))) + or 0 + ) + progress_percent = float(active_job.get("progress_percent", 0) or 0) + + if runtime_snapshot_queue: + runtime_items_total = int(runtime_snapshot_queue.get("items_total", 0) or 0) + if runtime_items_total > 0: + items_total = runtime_items_total + pending_items = int(runtime_snapshot_queue.get("pending", 0) or 0) + claimed_items = int(runtime_snapshot_queue.get("claimed", 0) or 0) + running_items = int(runtime_snapshot_queue.get("running", 0) or 0) + completed_items = int(runtime_snapshot_queue.get("completed", 0) or 0) + blacklisted_items = int(runtime_snapshot_queue.get("blacklisted", 0) or 0) + failed_items = int(runtime_snapshot_queue.get("failed", 0) or 0) + display_claimed = int(runtime_snapshot_queue.get("display_claimed", claimed_items) or 0) + display_running = int(runtime_snapshot_queue.get("display_running", running_items) or 0) + progress_percent = float(runtime_snapshot_job.get("progress_percent", progress_percent) or 0) + + if runtime_snapshot_nodes: + node_map = { + str(item.get("node_code") or "unknown"): { + "node_code": str(item.get("node_code") or "unknown"), + "items_total": _int_value(item.get("items_total")), + "items_pending": _int_value(item.get("items_pending")), + "items_claimed": _int_value(item.get("items_claimed")), + "items_running": _int_value(item.get("items_running")), + "items_completed": _int_value(item.get("items_completed")), + "items_blacklisted": _int_value(item.get("items_blacklisted")), + "items_failed": _int_value(item.get("items_failed")), + "processed_recent": _int_value(item.get("processed_recent")), + "processed_per_minute": float(item.get("processed_per_minute", 0) or 0), + "completed_recent": _int_value(item.get("completed_recent")), + "blacklisted_recent": _int_value(item.get("blacklisted_recent")), + "failed_recent": _int_value(item.get("failed_recent")), + "metrics_source": str(item.get("metrics_source") or "").strip(), + } + for item in runtime_snapshot_nodes + } + if runtime_snapshot_steps: + step_map = { + str(item.get("step_code") or "domain_pipeline"): { + **_build_step_bucket(str(item.get("step_code") or "domain_pipeline")), + "items_total": _int_value(item.get("items_total")), + "items_pending": _int_value(item.get("items_pending")), + "items_claimed": _int_value(item.get("items_claimed")), + "items_running": _int_value(item.get("items_running")), + "items_completed": _int_value(item.get("items_completed")), + "items_blacklisted": _int_value(item.get("items_blacklisted")), + "items_failed": _int_value(item.get("items_failed")), + "processed_recent": _int_value(item.get("processed_recent")), + "processed_per_minute": float(item.get("processed_per_minute", 0) or 0), + "completed_recent": _int_value(item.get("completed_recent")), + "blacklisted_recent": _int_value(item.get("blacklisted_recent")), + "failed_recent": _int_value(item.get("failed_recent")), + } + for item in runtime_snapshot_steps + } + + for row in list(runtime_display_rows or []): + runtime_bucket = _build_runtime_display_bucket(row) + if not runtime_bucket: + continue + node_code = str(runtime_bucket.get("node_code") or "unknown") + bucket = node_map.setdefault( + node_code, + { + "node_code": node_code, + "items_total": 0, + "items_pending": 0, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + "metrics_source": "runtime", + "display_running": 0, + "current_load": 0, + "active_threads": 0, + "max_threads": 0, + "region": "", + "role": "", + "status": "", + "last_heartbeat_at": "", + }, + ) + runtime_display_running = _effective_runtime_load( + items_running=runtime_bucket.get("items_running"), + active_threads=runtime_bucket.get("active_threads"), + ) + for key in ( + "items_total", + "items_pending", + "items_claimed", + "items_completed", + "items_blacklisted", + "items_failed", + "current_load", + "active_threads", + "max_threads", + ): + bucket[key] = max(_int_value(bucket.get(key)), _int_value(runtime_bucket.get(key))) + bucket["display_running"] = max( + _int_value(bucket.get("display_running")), + runtime_display_running, + ) + bucket["items_running"] = max( + _int_value(bucket.get("items_running")), + _int_value(runtime_bucket.get("items_running")), + ) + for key in ("metrics_source", "region", "role", "status", "last_heartbeat_at"): + if str(runtime_bucket.get(key) or "").strip(): + bucket[key] = runtime_bucket.get(key) + + for node_code, runtime_bucket in dict(runtime_activity.get("nodes") or {}).items(): + bucket = node_map.setdefault( + node_code, + { + "node_code": node_code, + "items_total": 0, + "items_pending": 0, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + "metrics_source": "runtime", + }, + ) + runtime_processed_recent = int(runtime_bucket.get("processed_recent", 0) or 0) + if runtime_processed_recent > int(bucket.get("processed_recent", 0) or 0): + bucket["processed_recent"] = runtime_processed_recent + bucket["processed_per_minute"] = round(runtime_processed_recent / window_minutes, 2) + bucket["completed_recent"] = int(runtime_bucket.get("completed_recent", 0) or 0) + bucket["blacklisted_recent"] = int(runtime_bucket.get("blacklisted_recent", 0) or 0) + bucket["failed_recent"] = int(runtime_bucket.get("failed_recent", 0) or 0) + if not str(bucket.get("metrics_source") or "").strip(): + bucket["metrics_source"] = "runtime" + + for bucket in node_map.values(): + active_threads = int(bucket.get("active_threads", 0) or 0) + current_load = int(bucket.get("current_load", 0) or 0) + display_running_value = max( + int(bucket.get("display_running", 0) or 0), + active_threads, + current_load, + ) + bucket["display_running"] = display_running_value + if active_threads > 0: + bucket["items_running"] = min( + max(int(bucket.get("items_running", 0) or 0), active_threads), + active_threads, + ) + bucket["current_load"] = max(current_load, active_threads) + + for bucket in step_map.values(): + runtime_bucket = dict(runtime_activity.get("step_stats") or {}).get(str(bucket.get("step_code") or "").strip()) or {} + if runtime_bucket: + bucket["started_recent"] = max( + int(bucket.get("started_recent", 0) or 0), + int(runtime_bucket.get("started_recent", 0) or 0), + ) + runtime_processed_recent = int(runtime_bucket.get("processed_recent", 0) or 0) + if runtime_processed_recent > int(bucket.get("processed_recent", 0) or 0): + bucket["processed_recent"] = runtime_processed_recent + bucket["processed_per_minute"] = round(runtime_processed_recent / window_minutes, 2) + bucket["completed_recent"] = int(runtime_bucket.get("completed_recent", 0) or 0) + bucket["blacklisted_recent"] = int(runtime_bucket.get("blacklisted_recent", 0) or 0) + bucket["failed_recent"] = int(runtime_bucket.get("failed_recent", 0) or 0) + + normalized_running_items = sum( + int(item.get("items_running", 0) or 0) + for item in node_map.values() + if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned" + ) + normalized_display_running = sum( + int(item.get("display_running", 0) or 0) + for item in node_map.values() + if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned" + ) + # Prefer normalized per-node sums once runtime heartbeats are present. + # Stale snapshot queue rows can temporarily over-report thread load from an + # older cycle, which is exactly what misleads the ops dashboard. + if normalized_running_items > 0: + running_items = normalized_running_items + if normalized_display_running > 0: + display_running = normalized_display_running + nodes = sorted( node_map.values(), key=lambda item: ( -int(item.get("processed_recent", 0) or 0), + -int(item.get("display_running", item.get("items_running", 0)) or 0), -int(item.get("items_running", 0) or 0), str(item.get("node_code") or ""), ), ) - items_total = int(active_job.get("items_total", 0) or 0) - terminal = int(active_job.get("items_terminal", 0) or 0) + terminal = completed_items + blacklisted_items + failed_items + steps = sorted( + step_map.values(), + key=lambda item: ( + -int(item.get("processed_recent", 0) or 0), + -int(item.get("items_running", 0) or 0), + -int(item.get("items_pending", 0) or 0), + str(item.get("step_code") or ""), + ), + ) return { "window_minutes": window_minutes, "has_active_job": True, "job": { "job_id": job_id, "job_code": active_job.get("job_code", ""), + "runtime_job_code": runtime_snapshot_job_code or str(runtime_activity.get("focus_job_code") or "").strip(), + "runtime_job_codes": list(runtime_activity.get("job_codes") or []), "status": active_job.get("status", ""), - "progress_percent": active_job.get("progress_percent", 0), + "progress_percent": progress_percent, }, "queue": { "items_total": items_total, - "pending": int(active_job.get("items_pending", 0) or 0), - "claimed": int(active_job.get("items_claimed", 0) or 0), - "running": int(active_job.get("items_running", 0) or 0), - "display_claimed": int(active_job.get("display_items_claimed", active_job.get("items_claimed", 0)) or 0), - "display_running": int(active_job.get("display_items_running", active_job.get("items_running", 0)) or 0), - "completed": int(active_job.get("items_completed", 0) or 0), - "blacklisted": int(active_job.get("items_blacklisted", 0) or 0), - "failed": int(active_job.get("items_failed", 0) or 0), + "pending": pending_items, + "claimed": claimed_items, + "running": running_items, + "display_claimed": display_claimed, + "display_running": display_running, + "completed": completed_items, + "blacklisted": blacklisted_items, + "failed": failed_items, "terminal": terminal, "terminal_percent": round((terminal / items_total) * 100, 2) if items_total else 0, "oldest_pending_at": oldest_pending_at, @@ -729,6 +2254,9 @@ def get_detect_queue_health(window_minutes: int = 15) -> dict: "failed_recent": total_failed_recent, }, "nodes": nodes, + "steps": steps, + "runtime_activity": runtime_activity, + "runtime_snapshot_backlog": dict(runtime_snapshot.get("backlog") or {}), } @@ -799,6 +2327,480 @@ def get_detect_capacity_plan(*, queue_health: dict | None = None, online_worker_ } +def process_detect_pipeline(limit: int = 5000, job_id: int | None = None) -> dict: + safe_limit = max(1, min(int(limit or 5000), 5000)) + normalized_job_id = int(job_id or 0) if job_id not in (None, "", 0, "0") else 0 + summary = { + "processed_items": 0, + "advanced_items": 0, + "retried_items": 0, + "terminal_items": 0, + "skipped_items": 0, + "created_next_items": 0, + "retry_reset_items": 0, + "legacy_items_upgraded": 0, + "legacy_items_completed": 0, + "job_ids": [], + } + + with get_db() as conn: + conn.autocommit = False + with conn.cursor() as cur: + settings_payload = get_settings_payload() + touched_job_ids: set[int] = set() + cur.execute( + """ + SELECT item.id, item.job_id, item.domain_id + FROM detect_job_items AS item + JOIN detect_jobs AS job + ON job.id = item.job_id + WHERE job.task_mode = 'domain_pipeline' + AND (%s = 0 OR item.job_id = %s) + AND job.status IN ('pending', 'running', 'completed', 'failed', 'partial_failed') + AND COALESCE(item.step_code, '') = '' + AND item.status = 'pending' + ORDER BY item.id ASC + LIMIT %s + FOR UPDATE SKIP LOCKED + """, + (normalized_job_id, normalized_job_id, safe_limit), + ) + legacy_rows = cur.fetchall() + for row in legacy_rows: + item_id = int(row[0]) + legacy_job_id = int(row[1]) + domain_id = int(row[2]) + domain_snapshot = _load_domain_pipeline_snapshot(cur, domain_id) + if not domain_snapshot: + continue + step_code, step_payload = resolve_initial_domain_pipeline_item( + domain_snapshot, + settings_payload=settings_payload, + ) + if step_code and step_payload: + cur.execute( + """ + UPDATE detect_job_items + SET step_code = %s, + step_payload_json = %s::jsonb, + updated_at = CURRENT_TIMESTAMP + WHERE id = %s + """, + (step_code, json.dumps(step_payload, ensure_ascii=False), item_id), + ) + touched_job_ids.add(legacy_job_id) + summary["processed_items"] += 1 + summary["legacy_items_upgraded"] += 1 + continue + + processed_payload = { + "controller_processed": True, + "controller_action": "pipeline_already_complete", + "controller_processed_at": datetime.now().isoformat(timespec="seconds"), + } + cur.execute( + """ + UPDATE detect_job_items + SET status = 'completed', + finished_at = COALESCE(finished_at, CURRENT_TIMESTAMP), + result_payload_json = %s::jsonb, + updated_at = CURRENT_TIMESTAMP + WHERE id = %s + """, + (json.dumps(processed_payload, ensure_ascii=False), item_id), + ) + touched_job_ids.add(legacy_job_id) + summary["processed_items"] += 1 + summary["legacy_items_completed"] += 1 + + cur.execute( + """ + SELECT item.id, item.job_id, item.domain_id, item.step_code, item.status, item.attempt_count, item.result_payload_json, job.task_mode + FROM detect_job_items AS item + JOIN detect_jobs AS job + ON job.id = item.job_id + WHERE job.task_mode IN ('domain_pipeline', 'single_step') + AND (%s = 0 OR item.job_id = %s) + AND job.status IN ('pending', 'running', 'completed', 'failed', 'partial_failed') + AND COALESCE(item.step_code, '') <> '' + AND item.status IN ('completed', 'blacklisted', 'failed') + AND COALESCE(item.result_payload_json->>'controller_processed', 'false') <> 'true' + ORDER BY COALESCE(item.finished_at, item.updated_at) ASC, item.id ASC + LIMIT %s + FOR UPDATE SKIP LOCKED + """, + (normalized_job_id, normalized_job_id, safe_limit), + ) + rows = cur.fetchall() + + for row in rows: + item_id = int(row[0]) + job_id = int(row[1]) + domain_id = int(row[2]) + current_step_code = str(row[3] or "").strip() + item_status = str(row[4] or "").strip() + attempt_count = int(row[5] or 0) + result_payload = _decode_payload(row[6]) + task_mode = str(row[7] or "").strip() + domain_snapshot = _load_domain_pipeline_snapshot(cur, domain_id) + if not domain_snapshot: + cur.execute( + """ + UPDATE detect_job_items + SET result_payload_json = jsonb_set(COALESCE(result_payload_json, '{}'::jsonb), '{controller_processed}', 'true'::jsonb, true), + updated_at = CURRENT_TIMESTAMP + WHERE id = %s + """, + (item_id,), + ) + summary["processed_items"] += 1 + summary["skipped_items"] += 1 + continue + + if task_mode == "single_step": + result_field_name = str( + result_payload.get("field_name") or _PIPELINE_STEP_FIELDS.get(current_step_code) or "" + ).strip() + persisted_payload = { + key: value + for key, value in dict(result_payload or {}).items() + if key not in {"controller_processed", "controller_action", "controller_processed_at"} + } + action = "single_step_ignored" + if result_field_name: + persisted_payload.setdefault("step_code", current_step_code) + persisted_payload.setdefault("step_name", _step_label(current_step_code)) + persisted_payload.setdefault("field_name", result_field_name) + persisted_payload.setdefault("task_mode", "single_step") + persisted_payload.setdefault("domain_id", domain_id) + _upsert_domain_detection_field(cur, domain_id, result_field_name, persisted_payload) + action = "single_step_persisted" + processed_payload = { + **dict(result_payload or {}), + "controller_processed": True, + "controller_action": action, + "controller_processed_at": datetime.now().isoformat(timespec="seconds"), + } + cur.execute( + """ + UPDATE detect_job_items + SET result_payload_json = %s::jsonb, + updated_at = CURRENT_TIMESTAMP + WHERE id = %s + """, + (json.dumps(processed_payload, ensure_ascii=False), item_id), + ) + touched_job_ids.add(job_id) + summary["processed_items"] += 1 + if action == "single_step_persisted": + summary["advanced_items"] += 1 + else: + summary["skipped_items"] += 1 + continue + + result_message = str(result_payload.get("message") or "").strip() + outcome = _classify_pipeline_item_outcome( + item_status=item_status, + result_payload=result_payload, + step_code=current_step_code, + attempt_count=attempt_count, + ) + action = str(outcome.get("action") or "pass").strip() + result_state = str(outcome.get("result_state") or "").strip().lower() + retry_limit = int(outcome.get("retry_limit", 0) or 0) + + if action == "retry": + retry_payload = { + **result_payload, + "controller_action": "retry", + "controller_processed": False, + "controller_retry_count": int(attempt_count), + "controller_reason_code": str(outcome.get("reason_code") or "external_retry"), + } + cur.execute( + """ + UPDATE detect_job_items + SET status = 'pending', + claimed_by = '', + claim_token = '', + lease_expires_at = NULL, + finished_at = NULL, + last_error = %s, + result_payload_json = %s::jsonb, + updated_at = CURRENT_TIMESTAMP + WHERE id = %s + """, + ( + result_message or f"{_step_label(current_step_code)} 执行失败,等待重试", + json.dumps(retry_payload, ensure_ascii=False), + item_id, + ), + ) + _append_detect_job_event_with_cursor( + cur, + job_id, + event_type="step_retry_scheduled", + message=f"{domain_snapshot['domain']} 的 {_step_label(current_step_code)} 已重投", + payload={ + "domain_id": domain_id, + "domain": domain_snapshot["domain"], + "step_code": current_step_code, + "attempt_count": attempt_count, + "retry_limit": retry_limit, + "result_state": result_state, + }, + ) + touched_job_ids.add(job_id) + summary["processed_items"] += 1 + summary["retried_items"] += 1 + summary["retry_reset_items"] += 1 + continue + + if action == "pass": + next_step_code = resolve_domain_pipeline_step( + domain_snapshot, + settings_payload=settings_payload, + after_step_code=current_step_code, + ) + if next_step_code: + next_payload = _build_step_payload( + step_code=next_step_code, + domain_snapshot=domain_snapshot, + settings_payload=settings_payload, + ) + cur.execute( + """ + INSERT INTO detect_job_items (job_id, domain_id, step_code, status, step_payload_json) + VALUES (%s, %s, %s, 'pending', %s::jsonb) + ON CONFLICT (job_id, domain_id, step_code) DO NOTHING + RETURNING id + """, + ( + job_id, + domain_id, + next_step_code, + json.dumps(next_payload, ensure_ascii=False), + ), + ) + created_next = bool(cur.fetchone()) + action = "pass" + summary["advanced_items"] += 1 + if created_next: + summary["created_next_items"] += 1 + _append_detect_job_event_with_cursor( + cur, + job_id, + event_type="pipeline_step_advanced", + message=f"{domain_snapshot['domain']} 进入下一步骤: {_step_label(next_step_code)}", + payload={ + "domain_id": domain_id, + "domain": domain_snapshot["domain"], + "current_step_code": current_step_code, + "next_step_code": next_step_code, + }, + ) + else: + action = "pipeline_completed" + _append_detect_job_event_with_cursor( + cur, + job_id, + event_type="pipeline_completed", + message=f"{domain_snapshot['domain']} 本次 pipeline 已完成", + payload={ + "domain_id": domain_id, + "domain": domain_snapshot["domain"], + "current_step_code": current_step_code, + }, + ) + + elif action == "black_hit": + _append_detect_job_event_with_cursor( + cur, + job_id, + event_type="pipeline_black_hit", + message=f"{domain_snapshot['domain']} 在 {_step_label(current_step_code)} 命中黑名单并终止后续步骤", + level="warning", + payload={ + "domain_id": domain_id, + "domain": domain_snapshot["domain"], + "current_step_code": current_step_code, + "result_state": result_state or "blacklisted", + }, + ) + elif action == "reject": + _append_detect_job_event_with_cursor( + cur, + job_id, + event_type="pipeline_rejected", + message=f"{domain_snapshot['domain']} 在 {_step_label(current_step_code)} 被判定终止", + level="warning", + payload={ + "domain_id": domain_id, + "domain": domain_snapshot["domain"], + "current_step_code": current_step_code, + "result_state": result_state or "failed", + "attempt_count": attempt_count, + "retry_limit": retry_limit, + "reason_code": str(outcome.get("reason_code") or ""), + }, + ) + + processed_payload = { + **result_payload, + "controller_processed": True, + "controller_action": action, + "controller_reason_code": str(outcome.get("reason_code") or ""), + "controller_processed_at": datetime.now().isoformat(timespec="seconds"), + } + if action == "pass": + processed_payload["controller_result_state"] = result_state or "passed" + cur.execute( + """ + UPDATE detect_job_items + SET result_payload_json = %s::jsonb, + updated_at = CURRENT_TIMESTAMP + WHERE id = %s + """, + (json.dumps(processed_payload, ensure_ascii=False), item_id), + ) + touched_job_ids.add(job_id) + summary["processed_items"] += 1 + if action in {"black_hit", "reject", "pipeline_completed"}: + summary["terminal_items"] += 1 + + for job_id in touched_job_ids: + cur.execute( + """ + SELECT + job.task_mode, + count(*) FILTER (WHERE item.status = 'pending') AS pending_count, + count(*) FILTER (WHERE item.status IN ('claimed', 'running')) AS dispatch_active_count, + count(*) FILTER (WHERE item.status = 'failed') AS failed_count, + count(*) FILTER (WHERE item.status IN ('completed', 'blacklisted')) AS done_count, + count(*) FILTER ( + WHERE item.status IN ('completed', 'blacklisted', 'failed') + AND COALESCE(item.step_code, '') <> '' + AND COALESCE(item.result_payload_json->>'controller_processed', 'false') <> 'true' + ) AS unprocessed_terminal_count + FROM detect_job_items AS item + JOIN detect_jobs AS job ON job.id = item.job_id + WHERE item.job_id = %s + GROUP BY job.task_mode + """, + (job_id,), + ) + refresh_row = cur.fetchone() + task_mode = str((refresh_row or [""])[0] or "").strip() + pending_count = int((refresh_row or ["", 0])[1] or 0) + dispatch_active_count = int((refresh_row or ["", 0, 0])[2] or 0) + failed_count = int((refresh_row or ["", 0, 0, 0])[3] or 0) + done_count = int((refresh_row or ["", 0, 0, 0, 0])[4] or 0) + unprocessed_terminal_count = int((refresh_row or ["", 0, 0, 0, 0, 0])[5] or 0) + if dispatch_active_count > 0 or (task_mode == "domain_pipeline" and unprocessed_terminal_count > 0): + cur.execute( + """ + UPDATE detect_jobs + SET status = 'running', + started_at = COALESCE(started_at, CURRENT_TIMESTAMP), + finished_at = NULL + WHERE id = %s + """, + (job_id,), + ) + elif pending_count > 0: + cur.execute( + """ + UPDATE detect_jobs + SET status = 'pending', + finished_at = NULL + WHERE id = %s + """, + (job_id,), + ) + else: + final_status = "completed" + if failed_count > 0 and done_count > 0: + final_status = "partial_failed" + elif failed_count > 0: + final_status = "failed" + cur.execute( + """ + UPDATE detect_jobs + SET status = %s, + finished_at = CURRENT_TIMESTAMP, + started_at = COALESCE(started_at, CURRENT_TIMESTAMP) + WHERE id = %s + """, + (final_status, job_id), + ) + summary["job_ids"].append(job_id) + + conn.commit() + + summary["job_ids"] = sorted(set(int(item) for item in summary["job_ids"])) + return summary + + +def process_detect_pipeline_now(limit: int = 2000, job_id: int | None = None) -> tuple[bool, str, dict]: + attempts = 3 + delay_seconds = 0.05 + for attempt in range(1, attempts + 1): + try: + data = process_detect_pipeline(limit=limit, job_id=job_id) + processed = int(data.get("processed_items", 0) or 0) + advanced = int(data.get("advanced_items", 0) or 0) + retried = int(data.get("retried_items", 0) or 0) + message = f"pipeline 处理完成:processed={processed}, advanced={advanced}, retried={retried}" + if attempt > 1: + message = f"{message}(deadlock 自动重试 {attempt - 1} 次后成功)" + return True, message, {"action": "process_pipeline", "retry_attempts": attempt - 1, **data} + except Exception as exc: + if not is_retryable_db_error(exc) or attempt >= attempts: + raise + time.sleep(delay_seconds) + delay_seconds *= 2 + + data = process_detect_pipeline(limit=limit, job_id=job_id) + processed = int(data.get("processed_items", 0) or 0) + advanced = int(data.get("advanced_items", 0) or 0) + retried = int(data.get("retried_items", 0) or 0) + message = f"pipeline 处理完成:processed={processed}, advanced={advanced}, retried={retried}" + return True, message, {"action": "process_pipeline", **data} + + +def _append_detect_job_event_with_cursor( + cur, + job_id: int, + *, + event_type: str, + message: str, + level: str = "info", + payload: dict | None = None, + node_code: str | None = None, +) -> bool: + cur.execute( + "SELECT 1 FROM detect_jobs WHERE id = %s LIMIT 1", + (int(job_id),), + ) + if cur.fetchone() is None: + return False + cur.execute( + """ + INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json) + VALUES (%s, %s, %s, %s, %s, %s::jsonb) + """, + ( + int(job_id), + node_code or settings.node_code, + str(event_type or "").strip() or "info", + str(level or "info").strip() or "info", + str(message or "").strip(), + json.dumps(payload or {}, ensure_ascii=False), + ), + ) + return True + + def append_detect_job_event( job_id: int, *, @@ -808,23 +2810,20 @@ def append_detect_job_event( payload: dict | None = None, node_code: str | None = None, ) -> None: + inserted = False with get_db() as conn: with conn.cursor() as cur: - cur.execute( - """ - INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json) - VALUES (%s, %s, %s, %s, %s, %s::jsonb) - """, - ( - int(job_id), - node_code or settings.node_code, - str(event_type or "").strip() or "info", - str(level or "info").strip() or "info", - str(message or "").strip(), - json.dumps(payload or {}, ensure_ascii=False), - ), + inserted = _append_detect_job_event_with_cursor( + cur, + job_id, + event_type=event_type, + message=message, + level=level, + payload=payload, + node_code=node_code, ) - conn.commit() + if inserted: + conn.commit() try: push_debug_event( service="detect-job", @@ -841,10 +2840,18 @@ def append_detect_job_event( pass -def create_detect_job_if_needed(limit: int = 1000, created_by: str = "system") -> dict | None: +def create_detect_job_if_needed(limit: int = 1000, created_by: str = "system", step_code: str | None = None) -> dict | None: + job_definition = resolve_detect_job_definition(step_code) existing = get_active_detect_job_summary() if existing: - return existing + existing_task_mode = str(existing.get("task_mode") or "").strip() + existing_step_code = normalize_detect_step_code(existing.get("step_code")) + if ( + existing_task_mode == str(job_definition["task_mode"]).strip() + and existing_step_code == normalize_detect_step_code(job_definition["step_code"]) + ): + return existing + settings_payload = get_settings_payload() with get_db() as conn: conn.autocommit = False @@ -874,32 +2881,69 @@ def create_detect_job_if_needed(limit: int = 1000, created_by: str = "system") - with get_db() as conn: conn.autocommit = False with conn.cursor() as cur: - job_code = f"detect-{datetime.now().strftime('%Y%m%d%H%M%S')}-{uuid4().hex[:6]}" + domain_snapshots: list[dict] = [] + for domain_id in domain_ids: + snapshot = _load_domain_pipeline_snapshot(cur, int(domain_id)) + if snapshot: + domain_snapshots.append(snapshot) + if not domain_snapshots: + conn.rollback() + return None + + job_prefix = "step" if job_definition["is_single_step"] else "detect" + job_code = f"{job_prefix}-{datetime.now().strftime('%Y%m%d%H%M%S')}-{uuid4().hex[:6]}" plan_hash = uuid4().hex cur.execute( """ - INSERT INTO detect_jobs (job_code, source, plan_hash, status, remark, created_by) - VALUES (%s, %s, %s, 'pending', %s, %s) + INSERT INTO detect_jobs (job_code, source, plan_hash, task_mode, step_code, status, remark, created_by) + VALUES (%s, %s, %s, %s, %s, 'pending', %s, %s) RETURNING id """, ( job_code, - "api-start", + job_definition["source"], plan_hash, - f"API 创建检测任务,待检测域名 {len(domain_ids)} 个", + job_definition["task_mode"], + job_definition["step_code"], + f"API 创建{job_definition['label']}任务,待检测域名 {len(domain_ids)} 个", created_by, ), ) job_id = cur.fetchone()[0] - item_rows = [(job_id, domain_id) for domain_id in domain_ids] - cur.executemany( - """ - INSERT INTO detect_job_items (job_id, domain_id, status) - VALUES (%s, %s, 'pending') - ON CONFLICT (job_id, domain_id) DO NOTHING - """, - item_rows, - ) + queued_count = 0 + skipped_count = 0 + for snapshot in domain_snapshots: + item_step_code = ( + job_definition["step_code"] + if job_definition["is_single_step"] + else resolve_initial_domain_pipeline_item(snapshot, settings_payload=settings_payload)[0] + ) + if not item_step_code: + skipped_count += 1 + continue + _, step_payload = resolve_initial_domain_pipeline_item(snapshot, settings_payload=settings_payload) + cur.execute( + """ + INSERT INTO detect_job_items (job_id, domain_id, step_code, status, step_payload_json) + VALUES (%s, %s, %s, 'pending', %s::jsonb) + ON CONFLICT (job_id, domain_id, step_code) DO NOTHING + RETURNING id + """, + ( + job_id, + int(snapshot["id"]), + item_step_code, + json.dumps(step_payload, ensure_ascii=False), + ), + ) + if cur.fetchone(): + queued_count += 1 + else: + skipped_count += 1 + + if queued_count <= 0: + conn.rollback() + return None cur.execute( """ INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json) @@ -910,9 +2954,18 @@ def create_detect_job_if_needed(limit: int = 1000, created_by: str = "system") - settings.node_code, "job_created", "info", - f"创建检测任务 {job_code},共 {len(domain_ids)} 个域名", - '{"count": %s}' % len(domain_ids), + f"创建检测任务 {job_code},已入队 {queued_count} 个任务项", + json.dumps( + { + "count": len(domain_ids), + "queued_count": queued_count, + "skipped_count": skipped_count, + "task_mode": job_definition["task_mode"], + "step_code": job_definition["step_code"], + }, + ensure_ascii=False, + ), ), ) conn.commit() - return get_active_detect_job_summary() + return get_detect_job_summary(job_id) diff --git a/domain-api/app/services/detect_service.py b/domain-api/app/services/detect_service.py index fb58f5e..84ebafd 100644 --- a/domain-api/app/services/detect_service.py +++ b/domain-api/app/services/detect_service.py @@ -2,10 +2,14 @@ from __future__ import annotations import json import re -from datetime import datetime, timezone +import subprocess +from datetime import datetime, timedelta, timezone +from app.core.config import settings from app.core.db import get_db from app.core.files import resolve_domain_path, tail_lines from app.core.redis_client import get_redis +from app.services.debug_event_service import list_debug_events +from app.services.cluster_runtime_service import ensure_runtime_schema from app.services.runtime_settings_service import get_runtime_settings from app.services.detect_run_service import sync_detect_runs from app.services.detect_job_service import get_active_detect_job_summary @@ -16,9 +20,79 @@ from app.services.worker_control_service import detect_worker_runtime _PROXY_COUNT_RE = re.compile(r"当前可用代理数[::]\s*(\d+)") _THREAD_COUNT_RE = re.compile(r"当前实际线程数量[::]\s*(\d+)\s*/\s*(\d+)") +_STEP_TRACE_DOMAIN_RE = re.compile(r"domain=([^\s|]+)") +_REGISTER_DOMAIN_RE = re.compile(r"检测注册状态[::]\s*([^\s]+)") _RUNTIME_STATE_KEY = "domain_tool:detect_runtime_state" _TIMESTAMP_FORMATS = ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S") +_SYSLOG_TIMESTAMP_FORMAT = "%b %d %H:%M:%S" _REMOTE_LOG_MAX_CHARS = 500 +_REMOTE_DEBUG_EVENT_TYPES = { + "worker_log", + "active_job_snapshot", + "domain_started", + "domain_completed", + "domain_failed", + "domain_blacklisted", + "task_pull_success", + "task_pull_partial", + "task_pull_failed", + "queue_overdue_leases", +} + + +def _extract_remote_log_node_code(line: str) -> str: + text = str(line or "").strip() + if not text.startswith("["): + return "" + first_close = text.find("]") + if first_close < 0: + return "" + second_open = text.find("[", first_close + 1) + second_close = text.find("]", second_open + 1) if second_open >= 0 else -1 + if second_open < 0 or second_close < 0: + return "" + return text[second_open + 1:second_close].strip() + + +def _slice_remote_log_lines_fairly(lines: list[str], *, limit: int = 240, min_per_node: int = 12) -> list[str]: + safe_limit = max(1, int(limit or 240)) + if len(lines) <= safe_limit: + return list(lines or []) + + normalized_lines = [str(line or "").strip() for line in list(lines or []) if str(line or "").strip()] + if len(normalized_lines) <= safe_limit: + return normalized_lines + + if min_per_node <= 0: + return normalized_lines[-safe_limit:] + + kept_indexes: set[int] = set() + per_node_counts: dict[str, int] = {} + for index in range(len(normalized_lines) - 1, -1, -1): + node_code = _extract_remote_log_node_code(normalized_lines[index]) + if not node_code: + continue + current_count = int(per_node_counts.get(node_code, 0) or 0) + if current_count >= min_per_node: + continue + kept_indexes.add(index) + per_node_counts[node_code] = current_count + 1 + if len(kept_indexes) >= safe_limit: + break + + for index in range(len(normalized_lines) - 1, -1, -1): + if len(kept_indexes) >= safe_limit: + break + kept_indexes.add(index) + + return [normalized_lines[index] for index in sorted(kept_indexes)] + + +def _runtime_state_key(node_code: str | None = None) -> str: + normalized_node_code = str(node_code or settings.node_code or "").strip() + if not normalized_node_code: + return _RUNTIME_STATE_KEY + return f"{_RUNTIME_STATE_KEY}:{normalized_node_code}" def _extract_dependency_alerts(lines: list[str]) -> list[dict]: @@ -83,6 +157,26 @@ def _extract_active_thread_snapshot(lines: list[str]) -> dict: return {"active": 0, "max": 0} +def _estimate_active_threads_from_recent_lines(lines: list[str], *, limit: int) -> int: + if not lines: + return 0 + + active_domains: list[str] = [] + seen_domains: set[str] = set() + for line in reversed(lines[-80:]): + match = _STEP_TRACE_DOMAIN_RE.search(line) or _REGISTER_DOMAIN_RE.search(line) + if not match: + continue + domain = str(match.group(1) or "").strip() + if not domain or domain in seen_domains: + continue + seen_domains.add(domain) + active_domains.append(domain) + if len(active_domains) >= int(limit): + break + return len(active_domains) + + def _parse_time(raw: str | None) -> datetime | None: if not raw: return None @@ -101,10 +195,12 @@ def _parse_time(raw: str | None) -> datetime | None: return None -def _extract_log_time(line: str) -> datetime | None: +def _extract_log_time(line: str, *, reference_year: int | None = None) -> datetime | None: if len(line) < 19: - return None - candidates = [line[:26], line[:19]] + text = str(line or "").strip() + else: + text = str(line or "") + candidates = [text[:26], text[:19]] for candidate in candidates: for fmt in _TIMESTAMP_FORMATS: if len(candidate) != len(datetime.now().strftime(fmt)): @@ -113,14 +209,79 @@ def _extract_log_time(line: str) -> datetime | None: return datetime.strptime(candidate, fmt) except ValueError: continue + syslog_candidate = str(text[:15] or "").strip() + if syslog_candidate: + try: + parsed = datetime.strptime(syslog_candidate, _SYSLOG_TIMESTAMP_FORMAT) + return parsed.replace(year=int(reference_year or datetime.now().year)) + except ValueError: + pass return None +def _read_worker_journal_lines(service_name: str, *, max_lines: int) -> tuple[list[str], str | None]: + normalized_service_name = str(service_name or "").strip() + if not normalized_service_name: + return [], None + + try: + completed = subprocess.run( + ["journalctl", "-u", normalized_service_name, "-n", str(max_lines), "--no-pager"], + capture_output=True, + text=True, + timeout=12, + ) + except Exception: + return [], None + + output = str(completed.stdout or "").strip() + if completed.returncode != 0 or not output: + return [], None + + lines = [str(line or "").rstrip() for line in output.splitlines() if str(line or "").strip()] + if not lines: + return [], None + return lines[-max_lines:], datetime.now(timezone.utc).isoformat() + + +def _load_recent_worker_lines(runtime_settings: dict, *, max_lines: int = 160) -> tuple[bool, str | None, list[str]]: + worker_log = resolve_domain_path("detect_worker.log", "logs/detect_worker.log") + + worker_online = False + last_log_time: str | None = None + recent_lines = tail_lines("detect_worker.log", max_lines=max_lines) + if worker_log and worker_log.exists(): + modified = datetime.fromtimestamp(worker_log.stat().st_mtime, tz=timezone.utc) + last_log_time = modified.isoformat() + worker_online = (datetime.now(timezone.utc) - modified).total_seconds() < 180 + + if str(runtime_settings.get("worker_mode") or "").strip() == "linux-systemd": + service_name = str(runtime_settings.get("worker_service_name") or "").strip() or "domaincheck-worker" + journal_lines, journal_last_time = _read_worker_journal_lines(service_name, max_lines=max_lines) + if journal_lines: + recent_lines = journal_lines + worker_online = True + if journal_last_time: + last_log_time = journal_last_time + + return worker_online, last_log_time, recent_lines + + def _filter_lines_since(lines: list[str], started_at: str | None) -> list[str]: started_time = _parse_time(started_at) if not started_time: return lines - filtered = [line for line in lines if (_extract_log_time(line) or started_time) >= started_time] + filtered: list[str] = [] + parsed_any = False + for line in lines: + line_time = _extract_log_time(line, reference_year=started_time.year) + if line_time is None: + continue + parsed_any = True + if line_time >= started_time: + filtered.append(line) + if not parsed_any: + return lines return filtered or lines @@ -258,6 +419,204 @@ def _build_remote_log_snapshot( } +def _build_remote_log_snapshot_from_debug_events( + active_job: dict | None, + *, + enabled: bool, + mode: str, + limit: int = 240, +) -> dict: + if not enabled: + return { + "lines": [], + "line_count": 0, + "last_at": "", + "last_line": "", + "source_nodes": [], + "source_node_count": 0, + "source_node_summaries": [], + } + + normalized_mode = str(mode or "key").strip().lower() + if normalized_mode not in {"key", "full"}: + normalized_mode = "key" + + participating_node_codes = { + str(item.get("node_code") or "").strip() + for item in list((active_job or {}).get("node_stats") or []) + if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned" + } + created_after = (datetime.now() - timedelta(hours=6)).strftime("%Y-%m-%d %H:%M:%S") + safe_limit = max(1, int(limit or 240)) + node_limit = max(40, min(200, safe_limit)) + records: list[dict] = [] + if participating_node_codes: + for node_code in sorted(participating_node_codes): + payload = list_debug_events( + limit=node_limit, + created_after=created_after, + node_code=node_code, + ) + records.extend(list(payload.get("records") or [])) + records.sort( + key=lambda item: ( + str(item.get("created_at") or ""), + int(item.get("id") or 0), + ), + reverse=True, + ) + else: + payload = list_debug_events(limit=max(safe_limit * 4, 240), created_after=created_after) + records = list(payload.get("records") or []) + if not records: + return { + "lines": [], + "line_count": 0, + "last_at": "", + "last_line": "", + "source_nodes": [], + "source_node_count": 0, + "source_node_summaries": [], + } + + lines: list[str] = [] + source_nodes: set[str] = set() + source_node_summaries: dict[str, dict] = {} + last_at = "" + last_line = "" + + for record in reversed(records): + event_type = str(record.get("event_type") or "").strip() + if event_type not in _REMOTE_DEBUG_EVENT_TYPES: + continue + node_code = str(record.get("node_code") or "").strip() or "unknown" + if participating_node_codes and node_code not in participating_node_codes: + continue + message = str(record.get("message") or "").strip() + if not message: + continue + created_at = str(record.get("created_at") or "").strip() + payload = record.get("payload") if isinstance(record.get("payload"), dict) else {} + event_mode = str(payload.get("log_mode") or "key").strip().lower() + if event_mode not in {"key", "full"}: + event_mode = "key" + if normalized_mode != "full" and event_mode == "full": + continue + if len(message) > _REMOTE_LOG_MAX_CHARS: + message = f"{message[:_REMOTE_LOG_MAX_CHARS]}..." + formatted_line = f"[{created_at}] [{node_code}] {message}" + lines.append(formatted_line) + source_nodes.add(node_code) + node_summary = source_node_summaries.setdefault( + node_code, + { + "node_code": node_code, + "line_count": 0, + "key_line_count": 0, + "full_line_count": 0, + "last_at": "", + "last_line": "", + }, + ) + node_summary["line_count"] += 1 + if event_mode == "full": + node_summary["full_line_count"] += 1 + else: + node_summary["key_line_count"] += 1 + node_summary["last_at"] = created_at + node_summary["last_line"] = formatted_line + last_at = created_at + last_line = formatted_line + + sliced_lines = _slice_remote_log_lines_fairly(lines, limit=safe_limit) + sorted_source_node_summaries = sorted( + source_node_summaries.values(), + key=lambda item: ( + str(item.get("last_at") or ""), + str(item.get("node_code") or ""), + ), + reverse=True, + ) + return { + "lines": sliced_lines, + "line_count": len(sliced_lines), + "last_at": last_at, + "last_line": last_line, + "source_nodes": sorted(source_nodes), + "source_node_count": len(source_nodes), + "source_node_summaries": sorted_source_node_summaries, + } + + +def _merge_remote_log_snapshots(primary: dict, secondary: dict, *, limit: int = 240) -> dict: + merged_lines: list[str] = [] + seen_lines: set[str] = set() + for raw_line in list(primary.get("lines") or []) + list(secondary.get("lines") or []): + line = str(raw_line or "").strip() + if not line or line in seen_lines: + continue + seen_lines.add(line) + merged_lines.append(line) + if limit > 0: + merged_lines = _slice_remote_log_lines_fairly(merged_lines, limit=limit) + + summaries: dict[str, dict] = {} + for snapshot in (primary, secondary): + for raw_summary in list(snapshot.get("source_node_summaries") or []): + if not isinstance(raw_summary, dict): + continue + node_code = str(raw_summary.get("node_code") or "").strip() + if not node_code: + continue + summary = summaries.setdefault( + node_code, + { + "node_code": node_code, + "line_count": 0, + "key_line_count": 0, + "full_line_count": 0, + "last_at": "", + "last_line": "", + }, + ) + summary["line_count"] = max(int(summary.get("line_count", 0) or 0), int(raw_summary.get("line_count", 0) or 0)) + summary["key_line_count"] = max(int(summary.get("key_line_count", 0) or 0), int(raw_summary.get("key_line_count", 0) or 0)) + summary["full_line_count"] = max(int(summary.get("full_line_count", 0) or 0), int(raw_summary.get("full_line_count", 0) or 0)) + raw_last_at = str(raw_summary.get("last_at") or "") + if raw_last_at >= str(summary.get("last_at") or ""): + summary["last_at"] = raw_last_at + summary["last_line"] = str(raw_summary.get("last_line") or "") + + source_nodes = sorted( + { + str(node_code or "").strip() + for node_code in list(primary.get("source_nodes") or []) + list(secondary.get("source_nodes") or []) + if str(node_code or "").strip() + } + ) + last_at = max(str(primary.get("last_at") or ""), str(secondary.get("last_at") or "")) + last_line = str(primary.get("last_line") or "") + if str(secondary.get("last_at") or "") >= str(primary.get("last_at") or ""): + last_line = str(secondary.get("last_line") or last_line) + + return { + "lines": merged_lines, + "line_count": len(merged_lines), + "last_at": last_at, + "last_line": last_line, + "source_nodes": source_nodes, + "source_node_count": len(source_nodes), + "source_node_summaries": sorted( + summaries.values(), + key=lambda item: ( + str(item.get("last_at") or ""), + str(item.get("node_code") or ""), + ), + reverse=True, + ), + } + + def _resolve_remote_log_lines( active_job: dict | None, runs: list[dict], @@ -277,17 +636,72 @@ def _resolve_remote_log_snapshot( mode: str, limit: int = 240, ) -> dict: - return _build_remote_log_snapshot(active_job, enabled=enabled, mode=mode, limit=limit) + primary_snapshot = _build_remote_log_snapshot(active_job, enabled=enabled, mode=mode, limit=limit) + debug_snapshot = _build_remote_log_snapshot_from_debug_events(active_job, enabled=enabled, mode=mode, limit=limit) + if int(primary_snapshot.get("line_count", 0) or 0) <= 0: + return debug_snapshot + if int(debug_snapshot.get("line_count", 0) or 0) <= 0: + return primary_snapshot + return _merge_remote_log_snapshots(primary_snapshot, debug_snapshot, limit=limit) def _load_runtime_state() -> dict: try: redis_client = get_redis() - raw = redis_client.get(_RUNTIME_STATE_KEY) - if not raw: + for key in (_runtime_state_key(), _RUNTIME_STATE_KEY): + raw = redis_client.get(key) + if not raw: + continue + data = json.loads(raw) + if not isinstance(data, dict): + continue + if key == _RUNTIME_STATE_KEY: + payload_node_code = str(data.get("node_code") or "").strip() + if payload_node_code and payload_node_code != str(settings.node_code or "").strip(): + continue + return data + return {} + except Exception: + return {} + + +def _load_runtime_state_from_cluster_node() -> dict: + try: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT current_load, metadata_json, last_heartbeat_at + FROM detect_worker_nodes + WHERE node_code = %s + LIMIT 1 + """, + (settings.node_code,), + ) + row = cur.fetchone() + if not row: return {} - data = json.loads(raw) - return data if isinstance(data, dict) else {} + current_load, metadata_json, last_heartbeat_at = row + metadata = metadata_json if isinstance(metadata_json, dict) else {} + if not metadata: + return {} + runtime_state = { + "node_code": settings.node_code, + "phase": str(metadata.get("phase") or metadata.get("phase_label") or "").strip(), + "detail": str(metadata.get("detail") or metadata.get("phase_detail") or "").strip(), + "service_running": True, + "detecting": bool(metadata.get("detecting", False) or int(current_load or 0) > 0), + "stop_requested": False, + "available_proxy_count": int(metadata.get("available_proxy_count", 0) or 0), + "active_threads": int(metadata.get("active_threads", 0) or 0), + "max_threads": int(metadata.get("max_threads", 0) or 0), + "job_id": metadata.get("job_id"), + "job_code": str(metadata.get("job_code") or metadata.get("active_job_code") or "").strip(), + "updated_at": str(metadata.get("updated_at") or (_format_time(last_heartbeat_at) if last_heartbeat_at else "")).strip(), + } + if runtime_state["detail"] or runtime_state["active_threads"] > 0 or runtime_state["max_threads"] > 0: + return runtime_state + return {} except Exception: return {} @@ -361,12 +775,51 @@ def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, a "supplier_empty": False, } + if refresh_status in {"", "未刷新"} and source_count > 0: + if allow_direct: + return { + "state": "degraded_direct", + "label": "等待首刷", + "detail": f"代理配置已下发,但代理池尚未完成首轮刷新;当前先按直连继续执行;最近状态:{refresh_status or '未刷新'}", + "direct_fallback_active": True, + "reason": "proxy_not_refreshed_yet", + "last_refresh_status": refresh_status or "未刷新", + "last_refresh_time": refresh_time, + "source_count": source_count, + "raw_items": raw_items, + "validated_count": validated, + "available_count": available, + "source_stats": source_stats, + "supplier_empty": False, + } + return { + "state": "warming_up", + "label": "等待首刷", + "detail": "代理配置已下发,但代理池尚未完成首轮刷新;由于未允许直连,检测链路会等待代理刷新完成", + "direct_fallback_active": False, + "reason": "proxy_not_refreshed_yet", + "last_refresh_status": refresh_status or "未刷新", + "last_refresh_time": refresh_time, + "source_count": source_count, + "raw_items": raw_items, + "validated_count": validated, + "available_count": available, + "source_stats": source_stats, + "supplier_empty": False, + } + if allow_direct: detail = "代理池当前无可用代理,已自动降级为直连继续执行" reason = "no_available_proxy" if supplier_empty: reason = "supplier_empty_pool" detail = "代理源最近都返回正常响应,但原始代理数为 0,当前判断为供应池为空;系统已自动降级为直连继续执行" + elif raw_items > 0 and validated > 0: + reason = "proxy_validation_zero" + detail = ( + f"代理源最近返回了 {raw_items} 个代理,已验证 {validated} 个,但当前 0 个可用;" + "系统已自动降级为直连继续执行" + ) if refresh_status: detail = f"{detail};最近状态:{refresh_status}" return { @@ -390,6 +843,12 @@ def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, a if supplier_empty: reason = "supplier_empty_pool" detail = "代理源最近都返回正常响应,但原始代理数为 0,当前判断为供应池为空;由于未允许直连,检测链路会等待代理恢复" + elif raw_items > 0 and validated > 0: + reason = "proxy_validation_zero" + detail = ( + f"代理源最近返回了 {raw_items} 个代理,已验证 {validated} 个,但当前 0 个可用;" + "由于未允许直连,检测链路会等待代理恢复" + ) if refresh_status: detail = f"{detail};最近状态:{refresh_status}" return { @@ -410,36 +869,45 @@ def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, a def get_detect_status() -> dict: + try: + ensure_runtime_schema() + except Exception: + # Node agent heartbeats should degrade gracefully even if runtime schema + # initialization is temporarily unavailable. + pass + queries = { "pending": "select count(*) from domains where detect_status = 0", "completed": "select count(*) from domains where detect_status = 1", "running": "select count(*) from domains where detect_status = 2", "blacklisted": "select count(*) from domains where detect_status = 3", "failed": "select count(*) from domains where detect_status = 4", + "registerable": "select count(*) from domains where detect_status = 1 and register_status = 2", } - progress: dict[str, int] = {} - with get_db() as conn: - with conn.cursor() as cur: - for key, query in queries.items(): - try: - cur.execute(query) - progress[key] = cur.fetchone()[0] - except Exception: - progress[key] = 0 + progress: dict[str, int] = {key: 0 for key in queries} + try: + with get_db() as conn: + with conn.cursor() as cur: + for key, query in queries.items(): + try: + cur.execute(query) + progress[key] = cur.fetchone()[0] + except Exception: + progress[key] = 0 + except Exception: + # Worker runtime/status pages should still render using runtime-state and + # cluster fallbacks even when the local DB endpoint is temporarily wrong + # or unreachable (for example remote worker nodes without direct DB access). + pass settings_payload = get_settings_payload() - worker_log = resolve_domain_path("detect_worker.log", "logs/detect_worker.log") - - worker_online = False - last_log_time = None - if worker_log and worker_log.exists(): - modified = datetime.fromtimestamp(worker_log.stat().st_mtime, tz=timezone.utc) - last_log_time = modified.isoformat() - worker_online = (datetime.now(timezone.utc) - modified).total_seconds() < 180 - - recent_lines = tail_lines("detect_worker.log", max_lines=160) + runtime_settings = get_runtime_settings() + worker_online, last_log_time, recent_lines = _load_recent_worker_lines(runtime_settings, max_lines=160) + runtime = detect_worker_runtime() runtime_state = _load_runtime_state() - runtime_started_at = runtime.get("latest_start_time", "") if 'runtime' in locals() else "" + if not runtime_state: + runtime_state = _load_runtime_state_from_cluster_node() + runtime_started_at = runtime.get("latest_start_time", "") recent_lines = _filter_lines_since(recent_lines, runtime_started_at) available_proxy_count = _extract_available_proxy_count(recent_lines) active_thread_snapshot = _extract_active_thread_snapshot(recent_lines) @@ -449,21 +917,35 @@ def get_detect_status() -> dict: "active": int(runtime_state.get("active_threads", active_thread_snapshot["active"]) or active_thread_snapshot["active"]), "max": int(runtime_state.get("max_threads", active_thread_snapshot["max"]) or active_thread_snapshot["max"]), } - progress_total = sum(progress.values()) + registerable_count = int(progress.get("registerable", 0) or 0) + progress_total = sum( + int(progress.get(key, 0) or 0) + for key in ("pending", "completed", "running", "blacklisted", "failed") + ) progress_done = progress.get("completed", 0) + progress.get("blacklisted", 0) + progress.get("failed", 0) progress_percent = round((progress_done / progress_total) * 100, 2) if progress_total > 0 else 0 - runtime = detect_worker_runtime() runtime_started_at = runtime.get("latest_start_time", "") recent_lines = _filter_lines_since(recent_lines, runtime_started_at) recent_proxy_warning = _normalize_recent_warning(runtime_state, recent_lines, available_proxy_count) proxy_runtime = _build_proxy_runtime_snapshot(settings_payload, runtime_state, available_proxy_count) thread_count_resolution = resolve_thread_count(settings_payload=settings_payload) effective_thread_count = int(thread_count_resolution["effective_thread_count"]) - runtime_settings = get_runtime_settings() worker_online = worker_online or runtime.get("running", False) if runtime_state.get("service_running") is True: worker_online = True - if not runtime_state.get("detecting", False) and not progress.get("running", 0): + if active_thread_snapshot["active"] <= 0 and (runtime_state.get("detecting", False) or runtime.get("running", False)): + estimated_active_threads = _estimate_active_threads_from_recent_lines( + recent_lines, + limit=max(1, effective_thread_count), + ) + if estimated_active_threads > 0: + active_thread_snapshot["active"] = estimated_active_threads + inferred_detecting = bool( + runtime_state.get("detecting", False) + or int(progress.get("running", 0) or 0) > 0 + or int(active_thread_snapshot.get("active", 0) or 0) > 0 + ) + if not inferred_detecting: active_thread_snapshot = {"active": 0, "max": active_thread_snapshot["max"] or effective_thread_count} settings_summary = { "thread_count": effective_thread_count, @@ -475,10 +957,59 @@ def get_detect_status() -> dict: "allow_direct": settings_payload["proxy_config"].get("allow_direct", False), "proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])), } - active_job = get_active_detect_job_summary(event_limit=240) + try: + active_job = get_active_detect_job_summary(event_limit=240) + except Exception: + active_job = None + if settings.node_region == "overseas" and settings.node_role == "control" and active_job: + progress = { + "pending": int(active_job.get("items_pending", 0) or 0), + "running": int( + active_job.get("display_active_threads", active_job.get("display_items_running", active_job.get("items_running", 0))) + or 0 + ), + "completed": int(active_job.get("items_completed", 0) or 0), + "failed": int(active_job.get("items_failed", 0) or 0), + "blacklisted": int(active_job.get("items_blacklisted", 0) or 0), + "registerable": registerable_count, + } + progress_percent = float(active_job.get("progress_percent", 0) or 0) + local_node_bucket = {} + for item in list((active_job or {}).get("node_stats") or []): + if str(item.get("node_code") or "").strip() == str(settings.node_code or "").strip(): + local_node_bucket = item + break + local_runtime_load = int( + local_node_bucket.get("active_threads") + or local_node_bucket.get("items_running") + or 0 + ) + local_runtime_max_threads = int(local_node_bucket.get("max_threads", 0) or 0) + if active_thread_snapshot["active"] <= 0 and local_runtime_load > 0: + active_thread_snapshot["active"] = local_runtime_load + if active_thread_snapshot["max"] <= 0: + active_thread_snapshot["max"] = local_runtime_max_threads or effective_thread_count + if settings.node_region == "overseas" and settings.node_role == "control" and active_job: + distributed_node_stats = list(active_job.get("distributed_node_stats") or active_job.get("node_stats") or []) + aggregated_active_threads = 0 + aggregated_max_threads = 0 + for item in distributed_node_stats: + node_code = str(item.get("node_code") or "").strip() + if not node_code or node_code == "unassigned": + continue + aggregated_active_threads += int( + item.get("active_threads") + or item.get("items_running") + or 0 + ) + aggregated_max_threads += int(item.get("max_threads", 0) or 0) + if aggregated_active_threads > 0: + active_thread_snapshot["active"] = aggregated_active_threads + if aggregated_max_threads > 0: + active_thread_snapshot["max"] = aggregated_max_threads runtime_snapshot = { **runtime, - "detecting": runtime_state.get("detecting", False), + "detecting": inferred_detecting, "proxy_runtime_state": proxy_runtime["state"], "proxy_runtime_label": proxy_runtime["label"], "proxy_runtime_detail": proxy_runtime["detail"], @@ -518,7 +1049,7 @@ def get_detect_status() -> dict: "runtime_state": runtime_state, "phase_label": runtime_state.get("phase", ""), "phase_detail": runtime_state.get("detail", ""), - "detecting": runtime_state.get("detecting", False), + "detecting": inferred_detecting, "thread_count": effective_thread_count, "thread_count_default": int(thread_count_resolution["default_thread_count"]), "thread_count_source": str(thread_count_resolution["source"]), @@ -549,6 +1080,7 @@ def get_detect_status() -> dict: "progress_percent": progress_percent, "recent_event": runtime_state.get("detail") or _recent_event(recent_lines), "recent_warning": recent_proxy_warning, + "aggregate_detect_view": bool(settings.node_region == "overseas" and settings.node_role == "control" and active_job), "log_lines": recent_lines, "remote_log_lines": remote_log_lines, "remote_log_line_count": int(remote_log_snapshot.get("line_count", 0) or 0), diff --git a/domain-api/app/services/export_service.py b/domain-api/app/services/export_service.py index 3756e60..115194a 100644 --- a/domain-api/app/services/export_service.py +++ b/domain-api/app/services/export_service.py @@ -145,7 +145,8 @@ def _write_txt(path, rows: list[dict]) -> None: def _write_csv(path, rows: list[dict]) -> None: - with path.open("w", encoding="utf-8", newline="") as handle: + # Add BOM so Excel on Chinese Windows opens CSV without mojibake. + with path.open("w", encoding="utf-8-sig", newline="") as handle: writer = csv.writer(handle) writer.writerow([label for _, label in EXPORT_HEADERS]) for row in rows: diff --git a/domain-api/app/services/juming_service.py b/domain-api/app/services/juming_service.py index be8c9ed..47179af 100644 --- a/domain-api/app/services/juming_service.py +++ b/domain-api/app/services/juming_service.py @@ -1,6 +1,8 @@ from __future__ import annotations import ast +import hashlib +from io import StringIO import pickle import re import sys @@ -35,6 +37,9 @@ LEGACY_JUMING_COOKIE_FILES = [ DELETE_LIST_SOURCE_TYPE = 2 FIXED_PRICE_SOURCE_TYPE = 1 JUMING_PREFERENCES_FILE = "juming_preferences.json" +JUMING_DELETE_IMPORT_STATE_FILE = "juming_delete_import_state.json" +IMPORT_BATCH_SIZE = 50000 +IMPORT_PROGRESS_EVERY = 100000 class TaskStoppedError(RuntimeError): @@ -125,6 +130,118 @@ def _persist_juming_cookie(cookie_jar: RequestsCookieJar) -> None: pass +def _empty_import_stats() -> dict[str, int]: + return { + "total": 0, + "valid": 0, + "added": 0, + "exists": 0, + "invalid": 0, + "failed": 0, + } + + +def _merge_import_stats(base: dict[str, int], delta: dict[str, int]) -> dict[str, int]: + merged = dict(base or _empty_import_stats()) + for key in ("total", "valid", "added", "exists", "invalid", "failed"): + merged[key] = int(merged.get(key, 0) or 0) + int((delta or {}).get(key, 0) or 0) + return merged + + +def _load_delete_import_state() -> dict[str, dict]: + payload = read_runtime_json(JUMING_DELETE_IMPORT_STATE_FILE, default={}) + return payload if isinstance(payload, dict) else {} + + +def _save_delete_import_state(payload: dict[str, dict]) -> None: + write_runtime_json(JUMING_DELETE_IMPORT_STATE_FILE, payload) + + +def _compute_domains_signature(domains: list[str]) -> str: + digest = hashlib.sha1() + for domain in domains: + digest.update(str(domain).strip().encode("utf-8", errors="ignore")) + digest.update(b"\n") + return digest.hexdigest() + + +def _looks_like_login_redirect(location: str) -> bool: + normalized = str(location or "").strip().lower() + if not normalized: + return False + return any( + marker in normalized + for marker in ( + "/login", + "user_zh", + "p_login", + "passport", + "sign", + ) + ) + + +def _looks_like_login_body(body: str) -> bool: + normalized = str(body or "").strip().lower() + if not normalized: + return False + return any( + marker in normalized + for marker in ( + "账号登录", + "请先登录", + "登录后查看", + "登录聚名", + "user_zh", + "p_login", + ) + ) + + +def _validate_juming_cookie( + cookie_jar: RequestsCookieJar | None, + *, + probe_date: str | None = None, +) -> tuple[bool, str]: + if cookie_jar is None or not _cookie_jar_to_dict(cookie_jar): + return False, "未检测到有效 Cookie" + + jm = JM() + jm.cookie = cookie_jar + probe_date = str(probe_date or date.today().isoformat()) + url = f"{jm.base_url}/newcha/del_down?scsj={probe_date}" + + try: + response = jm.session.get( + url, + headers=jm.headers, + cookies=jm.cookie, + allow_redirects=False, + timeout=10, + ) + except Exception as exc: + return False, f"登录态校验失败: {exc}" + + location = str(response.headers.get("Location") or "").strip() + if response.status_code in {301, 302, 303, 307, 308}: + if _looks_like_login_redirect(location): + return False, "聚名登录态已失效,请重新登录" + if location: + return True, f"删除列表下载链路校验通过: {probe_date}" + + try: + body = response.text + except Exception: + body = "" + + if _looks_like_login_body(body): + return False, "聚名登录态已失效,请重新登录" + + if response.ok: + return True, f"聚名 Cookie 已通过远端校验: {probe_date}" + return False, f"聚名登录态校验失败,HTTP {response.status_code}" + + def _jucha_cookie_status() -> dict: if JUCHA_COOKIE_FILE.exists(): return { @@ -186,11 +303,16 @@ def _load_juming_cookie() -> tuple[RequestsCookieJar | None, str]: def get_juming_status() -> dict: cookie_jar, storage = _load_juming_cookie() + cookie_valid, cookie_message = _validate_juming_cookie(cookie_jar) + cookie_count = len(_cookie_jar_to_dict(cookie_jar)) if cookie_jar is not None else 0 status = { - "cookie_ready": cookie_jar is not None, + "cookie_ready": bool(cookie_jar is not None and cookie_valid), + "cookie_present": cookie_jar is not None, + "cookie_valid": cookie_valid, + "cookie_message": cookie_message, "cookie_storage": storage, "cookie_file": str(JUMING_COOKIE_FILE), - "cookie_count": len(_cookie_jar_to_dict(cookie_jar)) if cookie_jar is not None else 0, + "cookie_count": cookie_count, "jucha": _jucha_cookie_status(), "supported_modes": [ {"label": "聚名一口价", "value": "fixed_price", "source_type": FIXED_PRICE_SOURCE_TYPE}, @@ -308,80 +430,146 @@ def _insert_domains( source_type: int, log: Callable[[str], None] | None = None, should_stop: Callable[[], bool] | None = None, + *, + announce_total: bool = True, + progress_label: str = "", ) -> dict: total = len(domains) - normalized_rows: list[tuple[str, str]] = [] invalid = 0 - - _emit_log(log, f"开始入库处理,共收到 {total} 个原始域名") - for value in domains: - _check_stop(should_stop) - normalized = normalize_domain(value) - if not normalized: - invalid += 1 - continue - tld = normalized.rsplit(".", 1)[-1] - normalized_rows.append((normalized, tld)) - - existing_set: set[str] = set() + progress_prefix = f"{progress_label} " if str(progress_label or "").strip() else "" + if announce_total: + _emit_log(log, f"{progress_prefix}开始入库处理,共收到 {total} 个原始域名") inserted = 0 + existing = 0 + processed = 0 + valid = 0 + last_progress_at = 0 + pending_batch: list[tuple[str, str]] = [] + pending_seen: set[str] = set() + stage_ready = False + + def emit_progress(force: bool = False) -> None: + nonlocal last_progress_at + if not force and processed - last_progress_at < IMPORT_PROGRESS_EVERY: + return + last_progress_at = processed + _emit_log( + log, + ( + f"{progress_prefix}入库进度:已处理 {processed}/{total}," + f"有效 {valid},新增 {inserted},已存在 {existing},无效 {invalid}" + ), + ) + + def ensure_stage_table(cur) -> None: + nonlocal stage_ready + if stage_ready: + return + cur.execute( + """ + create temporary table if not exists juming_import_stage ( + domain text primary key, + tld text not null + ) on commit preserve rows + """ + ) + stage_ready = True + + def stage_rows(cur, rows: list[tuple[str, str]]) -> None: + buffer = StringIO() + for domain, tld in rows: + buffer.write(f"{domain}\t{tld}\n") + buffer.seek(0) + cur.copy_from(buffer, "juming_import_stage", columns=("domain", "tld")) + + def flush_batch(cur, conn) -> None: + nonlocal inserted, existing + if not pending_batch: + return + ensure_stage_table(cur) + cur.execute("set local synchronous_commit = off") + cur.execute("truncate table juming_import_stage") + stage_rows(cur, pending_batch) + cur.execute( + """ + with existing_rows as ( + select count(*) + from juming_import_stage stage + join domains existing on existing.domain = stage.domain + ), + inserted as ( + insert into domains ( + domain, tld, source_type, use_status, detect_status, register_status, + has_beian, company_type, website_url, beian_year, snapshot_years, + expire_date, create_time, update_time, review_status, detect_time, + backlink_count, jucha_status, juziseo_status + ) + select + stage.domain, + stage.tld, + %s, + 0, 0, 0, + 1, null, null, null, null, + null, now(), now(), 0, null, + 0, 0, 0 + from juming_import_stage stage + left join domains existing on existing.domain = stage.domain + where existing.id is null + returning id + ), + task_insert as ( + insert into detect_tasks ( + domain_id, task_type, status, priority, retry_count, create_time, update_time + ) + select id, 1, 1, 5, 0, now(), now() + from inserted + returning 1 + ) + select + (select count(*) from inserted), + (select count(*) from task_insert), + (select count(*) from existing_rows) + """, + (source_type,), + ) + inserted_count, _task_count, existing_count = cur.fetchone() + inserted += int(inserted_count or 0) + existing += int(existing_count or 0) + conn.commit() + pending_batch.clear() + pending_seen.clear() with get_db() as conn: with conn.cursor() as cur: - normalized_domains = [row[0] for row in normalized_rows] - if normalized_domains: - cur.execute("select domain from domains where domain = any(%s)", (normalized_domains,)) - existing_set = {row[0] for row in cur.fetchall()} - if existing_set: - _emit_log(log, f"检测到 {len(existing_set)} 个已存在域名,将自动跳过") - - inserted_since_commit = 0 - for domain, tld in normalized_rows: + for value in domains: _check_stop(should_stop) - if domain in existing_set: + processed += 1 + normalized = normalize_domain(value) + if not normalized: + invalid += 1 + emit_progress() continue - cur.execute( - """ - insert into domains ( - domain, tld, source_type, use_status, detect_status, register_status, - has_beian, company_type, website_url, beian_year, snapshot_years, - expire_date, create_time, update_time, review_status, detect_time, - backlink_count, jucha_status, juziseo_status - ) values ( - %s, %s, %s, 0, 0, 0, - 1, null, null, null, null, - null, now(), now(), 0, null, - 0, 0, 0 - ) - returning id - """, - (domain, tld, source_type), - ) - domain_id = cur.fetchone()[0] - cur.execute( - """ - insert into detect_tasks (domain_id, task_type, status, priority, retry_count, create_time, update_time) - values (%s, 1, 1, 5, 0, now(), now()) - """, - (domain_id,), - ) - inserted += 1 - inserted_since_commit += 1 - if inserted_since_commit >= 500: - conn.commit() - inserted_since_commit = 0 - conn.commit() + valid += 1 + if normalized in pending_seen: + existing += 1 + emit_progress() + continue + pending_seen.add(normalized) + pending_batch.append((normalized, normalized.rsplit(".", 1)[-1])) + if len(pending_batch) >= IMPORT_BATCH_SIZE: + flush_batch(cur, conn) + emit_progress(force=True) + flush_batch(cur, conn) + emit_progress(force=True) - valid = len(normalized_rows) - exists = len(existing_set) - _emit_log(log, f"入库完成:有效 {valid},新增 {inserted},已存在 {exists},无效 {invalid}") + _emit_log(log, f"{progress_prefix}入库完成:有效 {valid},新增 {inserted},已存在 {existing},无效 {invalid}") return { "total": total, "valid": valid, "added": inserted, - "exists": exists, + "exists": existing, "invalid": invalid, - "failed": max(valid - exists - inserted, 0), + "failed": max(valid - existing - inserted, 0), } @@ -450,6 +638,98 @@ def _crawl_delete_list( return domains, dates +def _crawl_delete_list_and_import( + crawl_date: str, + auto_date: bool, + log: Callable[[str], None] | None = None, + should_stop: Callable[[], bool] | None = None, +) -> dict: + cookie_jar, _ = _load_juming_cookie() + jm = JM() + jm.cookie = cookie_jar or RequestsCookieJar() + + start_date = datetime.strptime(crawl_date, "%Y-%m-%d").date() + end_date = date.today() + timedelta(days=4) + current_date = start_date + dates: list[dict[str, int]] = [] + domains_found = 0 + sample_domains: list[str] = [] + stats = _empty_import_stats() + import_state = _load_delete_import_state() + _emit_log(log, f"开始采集删除列表:起始日期 {crawl_date},自动追加日期 {'开启' if auto_date else '关闭'}") + + while current_date <= end_date: + _check_stop(should_stop) + current_date_text = current_date.isoformat() + _emit_log(log, f"正在抓取 {current_date_text} 的删除列表") + domains_for_date = [item.strip() for item in jm.new_cha_del(current_date_text) if item.strip()] + domains_found += len(domains_for_date) + dates.append({"date": current_date_text, "count": len(domains_for_date)}) + _emit_log(log, f"{current_date_text} 抓取到 {len(domains_for_date)} 个域名,累计 {domains_found} 个") + if domains_for_date: + if len(sample_domains) < 20: + sample_domains.extend(domains_for_date[: max(0, 20 - len(sample_domains))]) + signature = _compute_domains_signature(domains_for_date) + cached = import_state.get(current_date_text) or {} + if ( + cached.get("signature") == signature + and int(cached.get("total", 0) or 0) == len(domains_for_date) + ): + cached_valid = int(cached.get("valid", 0) or 0) + cached_invalid = int(cached.get("invalid", 0) or 0) + date_stats = { + "total": len(domains_for_date), + "valid": cached_valid, + "added": 0, + "exists": cached_valid, + "invalid": cached_invalid, + "failed": 0, + } + _emit_log( + log, + ( + f"{current_date_text} 删除列表内容未变化,跳过重复入库:" + f"有效 {cached_valid},视为已存在 {cached_valid},无效 {cached_invalid}" + ), + ) + else: + date_stats = _insert_domains( + domains_for_date, + DELETE_LIST_SOURCE_TYPE, + log=log, + should_stop=should_stop, + announce_total=False, + progress_label=current_date_text, + ) + import_state[current_date_text] = { + "signature": signature, + "total": int(date_stats.get("total", 0) or 0), + "valid": int(date_stats.get("valid", 0) or 0), + "invalid": int(date_stats.get("invalid", 0) or 0), + "updated_at": datetime.now().isoformat(sep=" ", timespec="seconds"), + } + _save_delete_import_state(import_state) + stats = _merge_import_stats(stats, date_stats) + if not auto_date: + break + current_date += timedelta(days=1) + + _emit_log( + log, + ( + f"删除列表采集+入库完成:抓取 {domains_found} 个域名," + f"新增 {stats['added']},已存在 {stats['exists']},无效 {stats['invalid']}" + ), + ) + return { + "mode": "delete_list", + "dates": dates, + "domains_found": domains_found, + "stats": stats, + "sample_domains": sample_domains[:20], + } + + def crawl_juming( payload: dict, log: Callable[[str], None] | None = None, @@ -462,7 +742,10 @@ def crawl_juming( cookie_jar, storage = _load_juming_cookie() if cookie_jar is None: raise ValueError("未找到聚名 Cookie,请先在桌面版系统设置完成聚名登录,或将 Cookie 同步到服务器") - _emit_log(log, f"检测到聚名登录态,来源:{storage}") + cookie_valid, cookie_message = _validate_juming_cookie(cookie_jar) + if not cookie_valid: + raise ValueError(cookie_message) + _emit_log(log, f"检测到聚名登录态,来源:{storage},远端校验通过") _check_stop(should_stop) if mode == "fixed_price": @@ -482,13 +765,6 @@ def crawl_juming( crawl_date = str(payload.get("crawl_date") or date.today().isoformat()) auto_date = bool(payload.get("auto_date", True)) - domains, dates = _crawl_delete_list(crawl_date, auto_date, log=log, should_stop=should_stop) - stats = _insert_domains(domains, DELETE_LIST_SOURCE_TYPE, log=log, should_stop=should_stop) - return { - "mode": mode, - "cookie_storage": storage, - "dates": dates, - "domains_found": len(domains), - "stats": stats, - "sample_domains": domains[:20], - } + result = _crawl_delete_list_and_import(crawl_date, auto_date, log=log, should_stop=should_stop) + result["cookie_storage"] = storage + return result diff --git a/domain-api/app/services/juming_task_service.py b/domain-api/app/services/juming_task_service.py index 4ba7e05..4c091f8 100644 --- a/domain-api/app/services/juming_task_service.py +++ b/domain-api/app/services/juming_task_service.py @@ -10,6 +10,7 @@ from app.services.juming_service import TaskStoppedError, crawl_juming _JUMING_TASK_LOCK = threading.Lock() _MAX_LOG_LINES = 400 +_ACTIVE_TASK_IDS: set[str] = set() def _now() -> str: @@ -17,6 +18,7 @@ def _now() -> str: def list_juming_tasks() -> list[dict]: + _cleanup_orphaned_tasks() return load_juming_records() @@ -55,6 +57,43 @@ def _is_stop_requested(task_id: str) -> bool: return bool(target and target.get("cancel_requested")) +def _cleanup_orphaned_tasks() -> None: + with _JUMING_TASK_LOCK: + tasks = load_juming_records() + changed = False + for item in tasks: + status = str(item.get("status") or "").strip() + task_id = str(item.get("task_id") or "").strip() + if status == "running" and task_id and task_id not in _ACTIVE_TASK_IDS: + item["status"] = "failed" + item["phase"] = "failed" + item["phase_label"] = "失败" + item["cancel_requested"] = False + item["message"] = "任务因服务重启或进程中断而终止,请重新执行" + item["updated_at"] = _now() + logs = list(item.get("logs") or []) + logs.append(f"[{_now()}] 任务因服务重启或进程中断而终止,请重新执行") + item["logs"] = logs[-_MAX_LOG_LINES:] + changed = True + if changed: + _save_tasks(tasks) + + +def _ensure_no_active_task() -> None: + _cleanup_orphaned_tasks() + tasks = load_juming_records() + active = next( + ( + item + for item in tasks + if str(item.get("status") or "").strip() == "running" + ), + None, + ) + if active: + raise ValueError(f"已有聚名采集任务正在运行:{active.get('task_id')}") + + def _set_phase(task_id: str, phase: str, message: str | None = None) -> None: phase_labels = { "queued": "排队中", @@ -78,12 +117,18 @@ def _set_phase(task_id: str, phase: str, message: str | None = None) -> None: def _log_and_track_phase(task_id: str, message: str) -> None: if "开始采集" in message or "正在抓取" in message: _set_phase(task_id, "fetching", message) - elif "开始入库处理" in message or "入库完成" in message or "已存在域名" in message: + elif ( + "开始入库处理" in message + or "入库进度" in message + or "入库完成" in message + or "已存在域名" in message + ): _set_phase(task_id, "importing", message) _append_log(task_id, message) def _run_juming_task(task_id: str, payload: dict) -> None: + _ACTIVE_TASK_IDS.add(task_id) _update_task(task_id, status="running", started_at=_now(), message="聚名采集任务开始执行", cancel_requested=False) _set_phase(task_id, "starting", "聚名采集任务开始执行") _append_log(task_id, "任务已启动,正在准备读取聚名登录态") @@ -127,9 +172,12 @@ def _run_juming_task(task_id: str, payload: dict) -> None: ) _set_phase(task_id, "failed") _append_log(task_id, f"任务执行失败:{exc}") + finally: + _ACTIVE_TASK_IDS.discard(task_id) def create_juming_task(payload: dict) -> dict: + _ensure_no_active_task() task_id = uuid4().hex mode = str(payload.get("mode") or "delete_list").strip() or "delete_list" record = { @@ -154,12 +202,14 @@ def create_juming_task(payload: dict) -> dict: tasks.insert(0, record) _save_tasks(tasks) + _ACTIVE_TASK_IDS.add(task_id) worker = threading.Thread(target=_run_juming_task, args=(task_id, dict(payload or {})), daemon=True) worker.start() return record def retry_juming_task(task_id: str) -> dict: + _ensure_no_active_task() with _JUMING_TASK_LOCK: tasks = load_juming_records() target = next((item for item in tasks if item["task_id"] == task_id), None) diff --git a/domain-api/app/services/ops_action_executor_core.py b/domain-api/app/services/ops_action_executor_core.py index c615d88..13316e9 100644 --- a/domain-api/app/services/ops_action_executor_core.py +++ b/domain-api/app/services/ops_action_executor_core.py @@ -1,9 +1,15 @@ from __future__ import annotations +import os +import shutil from pathlib import Path from typing import Protocol +import psycopg2 +import redis + from app.core.config import settings +from app.core.files import runtime_root as api_runtime_root STRUCTURED_ACTIONS = { @@ -22,6 +28,7 @@ STRUCTURED_ACTIONS = { "runtime.restart_api", "runtime.start_sync_agent", "runtime.stop_sync_agent", + "runtime.reset_lab_state", } @@ -140,6 +147,233 @@ def systemctl_action_name(action: str) -> str: return "" +def _safe_bool(value: object, default: bool) -> bool: + if value is None: + return bool(default) + if isinstance(value, bool): + return value + text = str(value).strip().lower() + if not text: + return bool(default) + if text in {"1", "true", "yes", "on"}: + return True + if text in {"0", "false", "no", "off"}: + return False + return bool(default) + + +def _node_agent_queue_dir() -> Path: + explicit = str(os.getenv("OPS_AGENT_QUEUE_DIR", "") or "").strip() + if explicit: + return Path(explicit) + project_dir = Path(__file__).resolve().parents[2] + return project_dir / "runtime" / "node-agent-queue" / (settings.node_code or "unbound") + + +def _detect_install_root(base_path: Path) -> Path | None: + normalized = str(base_path.resolve()) + for marker in (f"{os.sep}releases{os.sep}", f"{os.sep}current{os.sep}"): + if marker in normalized: + return Path(normalized.split(marker, 1)[0]) + return None + + +def _domaincheck_runtime_root() -> Path: + explicit = str(os.getenv("DOMAINCHECK_RUNTIME_ROOT", "") or "").strip() + if explicit: + return Path(explicit) + domain_root = Path(settings.domain_root).resolve() + install_root = _detect_install_root(domain_root) + if install_root is not None: + return install_root / "runtime" / "domainCheck" + return domain_root + + +def _clear_path_contents(path: Path, *, preserve_names: set[str] | None = None) -> list[str]: + if not path.exists(): + return [] + cleared: list[str] = [] + preserved = preserve_names or set() + for child in path.iterdir(): + if child.name in preserved: + continue + if child.is_dir() and not child.is_symlink(): + shutil.rmtree(child, ignore_errors=False) + else: + child.unlink(missing_ok=True) + cleared.append(str(child)) + return cleared + + +def _reset_runtime_files( + *, + include_api_runtime: bool, + include_worker_runtime: bool, + include_node_agent_queue: bool, +) -> dict: + cleared_targets: dict[str, list[str]] = {} + if include_api_runtime: + api_root = api_runtime_root() + cleared_targets["api_runtime"] = _clear_path_contents(api_root, preserve_names={".env"}) + + if include_worker_runtime: + worker_root = _domaincheck_runtime_root() + worker_targets: dict[str, list[str]] = {} + for name in ("data", "logs"): + target = worker_root / name + worker_targets[name] = _clear_path_contents(target, preserve_names={".env"}) if target.exists() else [] + detect_worker_log = Path(settings.domain_root) / "detect_worker.log" + if detect_worker_log.exists(): + detect_worker_log.unlink(missing_ok=True) + worker_targets["files"] = [str(detect_worker_log)] + cleared_targets["worker_runtime"] = [ + item for values in worker_targets.values() for item in values + ] + + if include_node_agent_queue: + queue_dir = _node_agent_queue_dir() + cleared_targets["node_agent_queue"] = _clear_path_contents(queue_dir) if queue_dir.exists() else [] + + return cleared_targets + + +def _truncate_detect_runtime_tables(*, include_domains: bool) -> dict: + table_names = [ + "detect_debug_events", + "detect_run_events", + "detect_job_items", + "detect_jobs", + "detect_sync_records", + "detect_tasks", + "detect_worker_nodes", + "domain_detections", + "domain_blacklist", + ] + if include_domains: + table_names.append("domains") + + conn = psycopg2.connect( + host=settings.db_host, + port=settings.db_port, + dbname=settings.db_database, + user=settings.db_user, + password=settings.db_password, + ) + try: + conn.autocommit = False + with conn.cursor() as cur: + cur.execute( + f"TRUNCATE TABLE {', '.join(table_names)} RESTART IDENTITY CASCADE" + ) + conn.commit() + finally: + conn.close() + return {"tables": table_names, "include_domains": bool(include_domains)} + + +def _flush_runtime_redis() -> dict: + client = redis.Redis( + host=settings.redis_host, + port=settings.redis_port, + password=settings.redis_password or None, + db=settings.redis_db, + decode_responses=True, + socket_connect_timeout=5, + socket_timeout=5, + ) + try: + size_before = int(client.dbsize() or 0) + client.flushdb() + size_after = int(client.dbsize() or 0) + finally: + try: + client.close() + except Exception: + pass + return {"db": int(settings.redis_db), "size_before": size_before, "size_after": size_after} + + +def _run_service_command( + runner: CommandRunner, + service_name: str, + action: str, +) -> dict: + code, stdout, stderr = runner(["systemctl", action, service_name], timeout=60) + return { + "service_name": service_name, + "action": action, + "returncode": int(code or 0), + "stdout": trim_output(stdout, 4000), + "stderr": trim_output(stderr, 4000), + "ok": int(code or 0) == 0, + } + + +def _reset_lab_state( + payload: dict, + *, + runner: CommandRunner, +) -> tuple[bool, str, dict]: + normalized_payload = dict(payload or {}) + stop_services = [ + str(item).strip() + for item in list(normalized_payload.get("stop_services") or []) + if str(item).strip() + ] + start_services = [ + str(item).strip() + for item in list(normalized_payload.get("start_services") or []) + if str(item).strip() + ] + + clear_database = _safe_bool(normalized_payload.get("clear_database"), False) + clear_domains = _safe_bool(normalized_payload.get("clear_domains"), False) + clear_redis = _safe_bool(normalized_payload.get("clear_redis"), False) + clear_api_runtime = _safe_bool(normalized_payload.get("clear_api_runtime"), False) + clear_worker_runtime = _safe_bool(normalized_payload.get("clear_worker_runtime"), True) + clear_node_agent_queue = _safe_bool(normalized_payload.get("clear_node_agent_queue"), True) + + result: dict[str, object] = { + "stop_services": [], + "start_services": [], + "database": {}, + "redis": {}, + "runtime": {}, + } + errors: list[str] = [] + + for service_name in stop_services: + service_result = _run_service_command(runner, service_name, "stop") + result["stop_services"].append(service_result) + if not service_result["ok"]: + errors.append(f"stop {service_name} failed") + + try: + if clear_database: + result["database"] = _truncate_detect_runtime_tables(include_domains=clear_domains) + if clear_redis: + result["redis"] = _flush_runtime_redis() + result["runtime"] = _reset_runtime_files( + include_api_runtime=clear_api_runtime, + include_worker_runtime=clear_worker_runtime, + include_node_agent_queue=clear_node_agent_queue, + ) + except Exception as exc: + errors.append(str(exc)) + + for service_name in start_services: + service_result = _run_service_command(runner, service_name, "start") + result["start_services"].append(service_result) + if not service_result["ok"]: + errors.append(f"start {service_name} failed") + + ok = not errors + message = "lab runtime reset completed" if ok else "lab runtime reset finished with errors" + if errors: + result["errors"] = errors + return ok, message, result + + def execute_structured_action( action: str, payload: dict | None, @@ -291,4 +525,12 @@ def execute_structured_action( } return True, "diagnostics collected", diagnostics + if normalized_action == "runtime.reset_lab_state": + ok, message, result = _reset_lab_state( + normalized_payload, + runner=runner, + ) + result.update(host_details) + return ok, message, result + return False, f"unsupported action: {normalized_action}", {"action": normalized_action} diff --git a/domain-api/app/services/ops_agent_service.py b/domain-api/app/services/ops_agent_service.py index 79f75b1..6a827b5 100644 --- a/domain-api/app/services/ops_agent_service.py +++ b/domain-api/app/services/ops_agent_service.py @@ -12,6 +12,9 @@ from app.core.db import get_db from app.services.ops_command_service import build_bash_command from app.services.ops_job_service import create_ops_job, ensure_ops_schema, get_ops_job from app.services.ops_template_service import build_ops_template_payload, get_ops_action_template +from app.services.runtime_settings_service import get_runtime_settings +from app.services.sensitive_words_service import get_sensitive_words_payload +from app.services.settings_service import get_settings_payload _AGENT_SCHEMA_SQL = """ @@ -117,6 +120,48 @@ def _format_time(value: object) -> str: return "" +def _prefer_non_loopback_identity(primary: object, fallback: object) -> str: + primary_value = str(primary or "").strip() + fallback_value = str(fallback or "").strip() + invalid_values = {"", "localhost", "localhost.localdomain", "127.0.0.1", "::1"} + if primary_value and primary_value.lower() not in invalid_values: + return primary_value + return fallback_value + + +def _merge_detect_runtime_snapshot(cluster_metadata: dict, metadata: dict, current_load: int) -> dict: + active_threads = int(cluster_metadata.get("active_threads", metadata.get("active_threads", 0)) or 0) + max_threads = int(cluster_metadata.get("max_threads", metadata.get("max_threads", 0)) or 0) + inferred_worker_online = bool( + cluster_metadata.get("worker_online", metadata.get("worker_online", False)) + or cluster_metadata.get("service_running", metadata.get("service_running", False)) + or current_load > 0 + or active_threads > 0 + ) + inferred_detect_participating = bool( + cluster_metadata.get("detect_participating", metadata.get("detect_participating", False)) + or current_load > 0 + or active_threads > 0 + ) + return { + "worker_online": inferred_worker_online, + "detect_participating": inferred_detect_participating, + "active_threads": active_threads, + "max_threads": max_threads, + "current_load": current_load, + "phase_label": str(cluster_metadata.get("phase_label", metadata.get("phase_label", "")) or "").strip(), + "phase_detail": str(cluster_metadata.get("phase_detail", metadata.get("phase_detail", "")) or "").strip(), + "recent_warning": str(cluster_metadata.get("recent_warning", metadata.get("recent_warning", "")) or "").strip(), + "proxy_runtime_label": str( + cluster_metadata.get("proxy_runtime_label", metadata.get("proxy_runtime_label", "")) or "" + ).strip(), + "proxy_runtime_reason": str( + cluster_metadata.get("proxy_runtime_reason", metadata.get("proxy_runtime_reason", "")) or "" + ).strip(), + "updated_at": str(cluster_metadata.get("updated_at", metadata.get("updated_at", "")) or "").strip(), + } + + def _ops_job_event_level_label(level: object) -> str: normalized_level = str(level or "").strip().lower() mapping = { @@ -588,6 +633,7 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None = seen_node_codes.add(node_code) metadata = dict(node.get("metadata") or {}) cluster_node = cluster_map.get(node_code, {}) + cluster_metadata = dict(cluster_node.get("metadata") or {}) latest_token = latest_tokens.get(node_code, {}) latest_job = latest_jobs.get(node_code, {}) participation_row = dict(detect_participation_map.get(node_code) or {}) @@ -607,6 +653,15 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None = current_load=current_load, ) capabilities = list(metadata.get("capabilities") or []) + detect_runtime = _merge_detect_runtime_snapshot(cluster_metadata, metadata, current_load) + active_job_snapshot = { + "job_code": str(cluster_metadata.get("active_job_code", metadata.get("active_job_code", "")) or "").strip(), + "status": str(cluster_metadata.get("active_job_status", metadata.get("active_job_status", "")) or "").strip(), + "items_total": int(cluster_metadata.get("job_items_total", metadata.get("job_items_total", 0)) or 0), + "items_claimed": int(cluster_metadata.get("job_items_claimed", metadata.get("job_items_claimed", 0)) or 0), + "items_running": int(cluster_metadata.get("job_items_running", metadata.get("job_items_running", 0)) or 0), + "items_completed": int(cluster_metadata.get("job_items_completed", metadata.get("job_items_completed", 0)) or 0), + } merged_node = { **node, "is_managed": True, @@ -620,10 +675,14 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None = "agent_version": str(metadata.get("agent_version") or "").strip(), "agent_hostname": str(metadata.get("hostname") or "").strip(), "agent_ip": str(metadata.get("ip") or "").strip(), - "cluster_hostname": str(cluster_node.get("hostname") or "").strip(), - "cluster_ip": str(cluster_node.get("ip") or "").strip(), + "cluster_hostname": _prefer_non_loopback_identity(cluster_node.get("hostname"), metadata.get("hostname")), + "cluster_ip": _prefer_non_loopback_identity(cluster_node.get("ip"), metadata.get("ip")), "cluster_status": cluster_status, "cluster_current_load": current_load, + "current_load": current_load, + "detect_runtime": detect_runtime, + "runtime_state": detect_runtime, + "active_job": active_job_snapshot, "cluster_last_heartbeat_at": str(cluster_node.get("last_heartbeat_at") or metadata.get("last_heartbeat_at") or "").strip(), "cluster_is_effective_worker": bool(cluster_node.get("is_effective_worker", metadata.get("is_effective_worker", False))), "cluster_detect_participating": bool( @@ -688,6 +747,7 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None = if not node_code or node_code in seen_node_codes: continue metadata = dict(cluster_node.get("metadata") or {}) + cluster_metadata = dict(cluster_node.get("metadata") or {}) latest_token = latest_tokens.get(node_code, {}) latest_job = latest_jobs.get(node_code, {}) participation_row = dict(detect_participation_map.get(node_code) or {}) @@ -706,6 +766,15 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None = current_load=current_load, ) capabilities = list(metadata.get("capabilities") or []) + detect_runtime = _merge_detect_runtime_snapshot(cluster_metadata, metadata, current_load) + active_job_snapshot = { + "job_code": str(cluster_metadata.get("active_job_code", metadata.get("active_job_code", "")) or "").strip(), + "status": str(cluster_metadata.get("active_job_status", metadata.get("active_job_status", "")) or "").strip(), + "items_total": int(cluster_metadata.get("job_items_total", metadata.get("job_items_total", 0)) or 0), + "items_claimed": int(cluster_metadata.get("job_items_claimed", metadata.get("job_items_claimed", 0)) or 0), + "items_running": int(cluster_metadata.get("job_items_running", metadata.get("job_items_running", 0)) or 0), + "items_completed": int(cluster_metadata.get("job_items_completed", metadata.get("job_items_completed", 0)) or 0), + } fallback_node = { "node_code": node_code, "region": str(cluster_node.get("region") or "").strip(), @@ -732,12 +801,16 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None = "capabilities": capabilities, "capabilities_count": len(capabilities), "agent_version": str(metadata.get("agent_version") or "").strip(), - "agent_hostname": str(cluster_node.get("hostname") or metadata.get("hostname") or "").strip(), - "agent_ip": str(cluster_node.get("ip") or metadata.get("ip") or "").strip(), - "cluster_hostname": str(cluster_node.get("hostname") or "").strip(), - "cluster_ip": str(cluster_node.get("ip") or "").strip(), + "agent_hostname": _prefer_non_loopback_identity(cluster_node.get("hostname"), metadata.get("hostname")), + "agent_ip": _prefer_non_loopback_identity(cluster_node.get("ip"), metadata.get("ip")), + "cluster_hostname": _prefer_non_loopback_identity(cluster_node.get("hostname"), metadata.get("hostname")), + "cluster_ip": _prefer_non_loopback_identity(cluster_node.get("ip"), metadata.get("ip")), "cluster_status": cluster_status, "cluster_current_load": current_load, + "current_load": current_load, + "detect_runtime": detect_runtime, + "runtime_state": detect_runtime, + "active_job": active_job_snapshot, "cluster_last_heartbeat_at": str(cluster_node.get("last_heartbeat_at") or "").strip(), "cluster_is_effective_worker": bool(cluster_node.get("is_effective_worker", False)), "cluster_detect_participating": bool( @@ -2359,7 +2432,7 @@ def build_node_agent_bootstrap_plan( f"SYNC_AGENT_SERVICE_NAME={settings.sync_agent_service_name}", "NODE_AGENT_SERVICE_NAME=domaincheck-node-agent", "", - 'OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","health.snapshot","logs.collect","diagnostics.collect","deploy.release"]', + 'OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.start_detection","runtime.stop_detection","runtime.pull_tasks","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","runtime.reset_lab_state","health.snapshot","logs.collect","diagnostics.collect","deploy.release"]', "OPS_AGENT_LABELS={}", ] env_content = "\n".join(env_lines) @@ -2529,6 +2602,135 @@ def _upsert_agent_runtime(node_code: str, payload: dict) -> None: conn.commit() +def _load_existing_detect_node_runtime(node_code: str) -> dict: + normalized_node_code = str(node_code or "").strip() + if not normalized_node_code: + return {} + + try: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT status, current_load, metadata_json, last_heartbeat_at + FROM detect_worker_nodes + WHERE node_code = %s + LIMIT 1 + """, + (normalized_node_code,), + ) + row = cur.fetchone() + except Exception: + return {} + + if not row: + return {} + metadata = row[2] if isinstance(row[2], dict) else {} + return { + "status": str(row[0] or "").strip(), + "current_load": int(row[1] or 0), + "metadata": metadata, + "last_heartbeat_at": row[3], + } + + +def _upsert_agent_detect_runtime(node_code: str, payload: dict) -> None: + normalized_node_code = str(node_code or "").strip() + if not normalized_node_code: + return + + metadata = payload.get("metadata") if isinstance(payload.get("metadata"), dict) else {} + detect_runtime = metadata.get("detect_runtime") if isinstance(metadata.get("detect_runtime"), dict) else {} + if not detect_runtime: + return + + try: + from app.services.cluster_runtime_service import register_node_heartbeat + + region = str(payload.get("region") or "unknown").strip() or "unknown" + role = str(payload.get("role") or "worker").strip() or "worker" + active_threads = max(0, int(detect_runtime.get("active_threads") or 0)) + max_threads = max(0, int(detect_runtime.get("max_threads") or 0)) + current_load = max( + 0, + int(detect_runtime.get("current_load") or 0), + active_threads, + ) + worker_online = bool(detect_runtime.get("worker_online", False) or detect_runtime.get("service_running", False)) + detect_participating = bool(detect_runtime.get("detect_participating", False) or current_load > 0) + phase_label = str(detect_runtime.get("phase_label") or "").strip() + phase_detail = str(detect_runtime.get("phase_detail") or "").strip() + recent_warning = str(detect_runtime.get("recent_warning") or "").strip() + has_runtime_signal = any( + [ + worker_online, + detect_participating, + current_load > 0, + active_threads > 0, + max_threads > 0, + bool(phase_label), + bool(phase_detail), + bool(recent_warning), + ] + ) + if not has_runtime_signal: + return + existing_runtime = _load_existing_detect_node_runtime(normalized_node_code) + existing_metadata = existing_runtime.get("metadata") if isinstance(existing_runtime.get("metadata"), dict) else {} + existing_active_threads = max(0, int(existing_metadata.get("active_threads", 0) or 0)) + existing_current_load = max(0, int(existing_runtime.get("current_load", 0) or 0), existing_active_threads) + existing_status = str(existing_runtime.get("status") or "").strip() + existing_last_heartbeat_at = existing_runtime.get("last_heartbeat_at") + existing_is_recent = False + if isinstance(existing_last_heartbeat_at, datetime): + current_time = ( + datetime.now(existing_last_heartbeat_at.tzinfo) + if existing_last_heartbeat_at.tzinfo + else datetime.now() + ) + existing_is_recent = (current_time - existing_last_heartbeat_at) <= timedelta(seconds=120) + normalized_phase_detail = phase_detail.lower() + generic_phase_detail = bool(normalized_phase_detail) and "/" in normalized_phase_detail and " " not in normalized_phase_detail + weak_agent_snapshot = bool( + worker_online + and current_load <= 0 + and active_threads <= 0 + and max_threads > 0 + and not phase_label + and (not phase_detail or generic_phase_detail) + and not recent_warning + ) + if weak_agent_snapshot and existing_is_recent and ( + existing_current_load > 0 or existing_status == "busy" + ): + return + status = "busy" if current_load > 0 else ("online" if worker_online else "unknown") + heartbeat_metadata = { + "service": "agent-heartbeat", + "worker_online": worker_online, + "detect_participating": detect_participating, + "active_threads": active_threads, + "max_threads": max_threads, + "phase_label": phase_label, + "phase_detail": phase_detail, + "recent_warning": recent_warning, + "updated_at": str(detect_runtime.get("updated_at") or "").strip(), + "agent_heartbeat_at": datetime.now().isoformat(timespec="seconds"), + } + register_node_heartbeat( + node_code=normalized_node_code, + region=region, + role=role, + status=status, + current_load=current_load, + metadata=heartbeat_metadata, + hostname_override=str(payload.get("hostname") or "").strip(), + ip_override=str(payload.get("ip") or "").strip(), + ) + except Exception: + return + + def agent_register(payload: dict, *, token: str) -> tuple[bool, str, dict]: node_code = str(payload.get("node_code") or "").strip() if not node_code: @@ -2537,6 +2739,7 @@ def agent_register(payload: dict, *, token: str) -> tuple[bool, str, dict]: if not ok: return False, message, auth _upsert_agent_runtime(node_code, payload) + _upsert_agent_detect_runtime(node_code, payload) return True, "Agent 注册成功", { "node_code": node_code, "expires_at": auth.get("expires_at", ""), @@ -2552,6 +2755,7 @@ def agent_heartbeat(payload: dict, *, token: str) -> tuple[bool, str, dict]: if not ok: return False, message, auth _upsert_agent_runtime(node_code, payload) + _upsert_agent_detect_runtime(node_code, payload) return True, "heartbeat ok", { "node_code": node_code, "server_time": _format_time(datetime.now()), @@ -2559,6 +2763,57 @@ def agent_heartbeat(payload: dict, *, token: str) -> tuple[bool, str, dict]: } +def _runtime_config_bundle_hash_payload(bundle: dict | None) -> dict: + normalized = dict(bundle or {}) + normalized.pop("config_hash", None) + normalized.pop("generated_at", None) + return normalized + + +def _build_agent_runtime_config_bundle(node_code: str) -> dict: + settings_payload = get_settings_payload() + runtime_settings = get_runtime_settings() + sensitive_words_payload = get_sensitive_words_payload() + bundle = { + "node_code": str(node_code or "").strip(), + "detect_options": dict(settings_payload.get("detect_options") or {}), + "proxy_config": dict(settings_payload.get("proxy_config") or {}), + "thread_count": int(settings_payload.get("thread_count", 2) or 2), + "node_thread_counts": dict(settings_payload.get("node_thread_counts") or {}), + "runtime_settings": dict(runtime_settings or {}), + "sensitive_words": { + "text": str(sensitive_words_payload.get("text") or ""), + "total": int(sensitive_words_payload.get("total", 0) or 0), + "items": list(sensitive_words_payload.get("items") or []), + }, + "generated_at": _format_time(datetime.now()), + } + bundle["config_hash"] = hashlib.sha256( + json.dumps( + _runtime_config_bundle_hash_payload(bundle), + ensure_ascii=False, + sort_keys=True, + ).encode("utf-8") + ).hexdigest() + return bundle + + +def agent_pull_runtime_config(payload: dict, *, token: str) -> tuple[bool, str, dict]: + node_code = str(payload.get("node_code") or "").strip() + if not node_code: + return _agent_error("node_code 不能为空", "agent_node_code_required") + ok, message, auth = _authenticate_agent_token(token, expected_node_code=node_code) + if not ok: + return False, message, auth + bundle = _build_agent_runtime_config_bundle(node_code) + return True, "runtime config ok", { + "node_code": node_code, + "server_time": _format_time(datetime.now()), + "expires_at": auth.get("expires_at", ""), + "bundle": bundle, + } + + def agent_pull_jobs(payload: dict, *, token: str, limit: int = 1) -> tuple[bool, str, dict]: node_code = str(payload.get("node_code") or "").strip() if not node_code: @@ -2568,6 +2823,7 @@ def agent_pull_jobs(payload: dict, *, token: str, limit: int = 1) -> tuple[bool, return False, message, _auth safe_limit = min(max(int(limit or 1), 1), 10) + dispatched_events: list[dict] = [] with get_db() as conn: conn.autocommit = False with conn.cursor() as cur: @@ -2604,15 +2860,19 @@ def agent_pull_jobs(payload: dict, *, token: str, limit: int = 1) -> tuple[bool, """, (job_id,), ) - append_ops_job_event( - job_id=job_id, - node_code=node_code, - event_type="agent_dispatched", - message=f"任务已派发给节点 {node_code}", - payload={"node_code": node_code}, + dispatched_events.append( + { + "job_id": job_id, + "node_code": node_code, + "event_type": "agent_dispatched", + "message": f"任务已派发给节点 {node_code}", + "payload": {"node_code": node_code}, + } ) jobs.append(_agent_job_envelope(get_ops_job(job_id))) conn.commit() + for event in dispatched_events: + append_ops_job_event(**event) return True, "ok", { "jobs": jobs, "count": len(jobs), @@ -2631,6 +2891,7 @@ def agent_mark_job_started(job_id: int, payload: dict, *, token: str) -> tuple[b if not ok: return False, message, _auth + started_event: dict | None = None with get_db() as conn: conn.autocommit = False with conn.cursor() as cur: @@ -2662,14 +2923,16 @@ def agent_mark_job_started(job_id: int, payload: dict, *, token: str) -> tuple[b ) step_rows = cur.fetchall() step_ids = [int(item[0]) for item in step_rows] - append_ops_job_event( - job_id=int(job_id), - node_code=node_code, - event_type="agent_started", - message=f"节点 {node_code} 已开始执行任务", - payload={"step_ids": step_ids}, - ) + started_event = { + "job_id": int(job_id), + "node_code": node_code, + "event_type": "agent_started", + "message": f"节点 {node_code} 已开始执行任务", + "payload": {"step_ids": step_ids}, + } conn.commit() + if started_event: + append_ops_job_event(**started_event) job = get_ops_job(int(job_id)) return True, "任务已标记为运行中", { "job": job, @@ -2710,6 +2973,7 @@ def agent_complete_job(job_id: int, payload: dict, *, token: str) -> tuple[bool, result["focus_ref"] = focus_ref event_level = "info" if job_status == "success" else ("warning" if job_status == "partially_succeeded" else "error") + completed_event: dict | None = None with get_db() as conn: conn.autocommit = False with conn.cursor() as cur: @@ -2789,22 +3053,24 @@ def agent_complete_job(job_id: int, payload: dict, *, token: str) -> tuple[bool, ) step_rows = cur.fetchall() step_ids = [int(item[0]) for item in step_rows] - append_ops_job_event( - job_id=int(job_id), - node_code=node_code, - client_event_id=(f"complete:{client_request_id}" if client_request_id else ""), - event_type="agent_completed", - message=f"节点 {node_code} 已完成任务,状态: {job_status}", - level=event_level, - payload={ + completed_event = { + "job_id": int(job_id), + "node_code": node_code, + "client_event_id": (f"complete:{client_request_id}" if client_request_id else ""), + "event_type": "agent_completed", + "message": f"节点 {node_code} 已完成任务,状态: {job_status}", + "level": event_level, + "payload": { "step_ids": step_ids, "result": result, "duration_ms": duration_ms, "summary_text": str(result.get("summary_text") or result.get("summary") or summary_text).strip(), "focus_ref": focus_ref, }, - ) + } conn.commit() + if completed_event: + append_ops_job_event(**completed_event) from app.services.ops_release_service import refresh_release_rollout_for_job refresh_release_rollout_for_job(int(job_id)) diff --git a/domain-api/app/services/ops_job_service.py b/domain-api/app/services/ops_job_service.py index d2e9295..9017f4b 100644 --- a/domain-api/app/services/ops_job_service.py +++ b/domain-api/app/services/ops_job_service.py @@ -39,6 +39,13 @@ CREATE TABLE IF NOT EXISTS ops_managed_nodes ( updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ); +CREATE TABLE IF NOT EXISTS ops_managed_node_secrets ( + node_code VARCHAR(64) PRIMARY KEY REFERENCES ops_managed_nodes(node_code) ON DELETE CASCADE, + ssh_password TEXT NOT NULL DEFAULT '', + ssh_private_key TEXT NOT NULL DEFAULT '', + updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + CREATE TABLE IF NOT EXISTS ops_jobs ( id BIGSERIAL PRIMARY KEY, job_code VARCHAR(64) NOT NULL UNIQUE, @@ -321,6 +328,107 @@ def _serialize_node_row(row: tuple) -> dict: } +def _load_node_secret_flags(node_codes: list[str]) -> dict[str, dict]: + normalized_codes = [str(item or "").strip() for item in node_codes if str(item or "").strip()] + if not normalized_codes: + return {} + result: dict[str, dict] = {} + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT node_code, ssh_password, ssh_private_key + FROM ops_managed_node_secrets + WHERE node_code = ANY(%s) + """, + (normalized_codes,), + ) + rows = cur.fetchall() + for row in rows: + node_code = str(row[0] or "").strip() + result[node_code] = { + "ssh_password_configured": bool(str(row[1] or "").strip()), + "ssh_private_key_configured": bool(str(row[2] or "").strip()), + } + return result + + +def _parse_ssh_entry(raw_value: object) -> dict: + raw = str(raw_value or "").strip() + if not raw: + return {} + parts = raw.split(maxsplit=2) + if len(parts) < 2: + return {} + host_port = str(parts[0] or "").strip() + ssh_user = str(parts[1] or "").strip() + secret = str(parts[2] or "").strip() if len(parts) >= 3 else "" + ssh_host = host_port + ssh_port = 22 + if ":" in host_port: + host_candidate, port_candidate = host_port.rsplit(":", 1) + if host_candidate and port_candidate.isdigit(): + ssh_host = host_candidate + ssh_port = max(int(port_candidate), 1) + if secret.startswith("<") and secret.endswith(">") and len(secret) >= 2: + secret = secret[1:-1].strip() + payload = { + "ssh_host": ssh_host, + "ssh_port": ssh_port, + "ssh_user": ssh_user, + } + if secret: + payload["auth_mode"] = "password" + payload["ssh_password"] = secret + return payload + + +def _upsert_managed_node_secret( + *, + node_code: str, + ssh_password: str = "", + ssh_private_key: str = "", + clear_ssh_password: bool = False, + clear_ssh_private_key: bool = False, +) -> None: + normalized_node_code = str(node_code or "").strip() + if not normalized_node_code: + return + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + INSERT INTO ops_managed_node_secrets ( + node_code, ssh_password, ssh_private_key, updated_at + ) VALUES (%s, %s, %s, CURRENT_TIMESTAMP) + ON CONFLICT (node_code) DO UPDATE SET + ssh_password = CASE + WHEN %s THEN '' + WHEN %s <> '' THEN %s + ELSE ops_managed_node_secrets.ssh_password + END, + ssh_private_key = CASE + WHEN %s THEN '' + WHEN %s <> '' THEN %s + ELSE ops_managed_node_secrets.ssh_private_key + END, + updated_at = CURRENT_TIMESTAMP + """, + ( + normalized_node_code, + "" if clear_ssh_password else ssh_password, + "" if clear_ssh_private_key else ssh_private_key, + clear_ssh_password, + ssh_password, + ssh_password, + clear_ssh_private_key, + ssh_private_key, + ssh_private_key, + ), + ) + conn.commit() + + def _pick_text_value(payload: dict, key: str, fallback: str = "", *, default: str = "") -> str: if key in payload: normalized = str(payload.get(key) or "").strip() @@ -397,6 +505,15 @@ def upsert_managed_node(payload: dict) -> tuple[bool, str, dict]: node_code = str(payload.get("node_code") or "").strip() if not node_code: return False, "node_code 不能为空", {} + parsed_ssh_entry = _parse_ssh_entry(payload.get("ssh_entry")) + merged_payload = { + **dict(payload or {}), + **{key: value for key, value in parsed_ssh_entry.items() if value not in ("", None)}, + } + ssh_password = str(merged_payload.get("ssh_password") or "").strip() + ssh_private_key = str(merged_payload.get("ssh_private_key") or "") + clear_ssh_password = bool(merged_payload.get("clear_ssh_password", False)) + clear_ssh_private_key = bool(merged_payload.get("clear_ssh_private_key", False)) with get_db() as conn: with conn.cursor() as cur: @@ -413,22 +530,22 @@ def upsert_managed_node(payload: dict) -> tuple[bool, str, dict]: existing_row = cur.fetchone() existing_node = _serialize_node_row(existing_row) if existing_row else {} existing_metadata = dict(existing_node.get("metadata") or {}) - incoming_metadata = dict(payload.get("metadata") or {}) + incoming_metadata = dict(merged_payload.get("metadata") or {}) - region = _pick_text_value(payload, "region", str(existing_node.get("region") or ""), default="unknown") or "unknown" - role = _pick_text_value(payload, "role", str(existing_node.get("role") or ""), default="worker") or "worker" - title = _pick_text_value(payload, "title", str(existing_node.get("title") or ""), default=node_code) or node_code - ssh_host = _pick_text_value(payload, "ssh_host", str(existing_node.get("ssh_host") or "")) - ssh_port = _pick_int_value(payload, "ssh_port", int(existing_node.get("ssh_port") or 22), default=22, minimum=1) - ssh_user = _pick_text_value(payload, "ssh_user", str(existing_node.get("ssh_user") or "")) - auth_mode = _pick_text_value(payload, "auth_mode", str(existing_node.get("auth_mode") or ""), default="key") or "key" + region = _pick_text_value(merged_payload, "region", str(existing_node.get("region") or ""), default="unknown") or "unknown" + role = _pick_text_value(merged_payload, "role", str(existing_node.get("role") or ""), default="worker") or "worker" + title = _pick_text_value(merged_payload, "title", str(existing_node.get("title") or ""), default=node_code) or node_code + ssh_host = _pick_text_value(merged_payload, "ssh_host", str(existing_node.get("ssh_host") or "")) + ssh_port = _pick_int_value(merged_payload, "ssh_port", int(existing_node.get("ssh_port") or 22), default=22, minimum=1) + ssh_user = _pick_text_value(merged_payload, "ssh_user", str(existing_node.get("ssh_user") or "")) + auth_mode = _pick_text_value(merged_payload, "auth_mode", str(existing_node.get("auth_mode") or ""), default="key") or "key" deploy_channel = _pick_text_value( - payload, + merged_payload, "deploy_channel", str(existing_node.get("deploy_channel") or ""), default="stable", ) or "stable" - is_enabled = bool(payload["is_enabled"]) if "is_enabled" in payload else bool(existing_node.get("is_enabled", True)) + is_enabled = bool(merged_payload["is_enabled"]) if "is_enabled" in merged_payload else bool(existing_node.get("is_enabled", True)) metadata = { **existing_metadata, **incoming_metadata, @@ -469,7 +586,16 @@ def upsert_managed_node(payload: dict) -> tuple[bool, str, dict]: ) row = cur.fetchone() conn.commit() - return True, "托管节点已保存", {"node": _serialize_node_row(row)} + _upsert_managed_node_secret( + node_code=node_code, + ssh_password=ssh_password, + ssh_private_key=ssh_private_key, + clear_ssh_password=clear_ssh_password, + clear_ssh_private_key=clear_ssh_private_key, + ) + node = _serialize_node_row(row) + node.update(_load_node_secret_flags([node_code]).get(node_code, {})) + return True, "托管节点已保存", {"node": node} def list_managed_nodes() -> list[dict]: @@ -485,7 +611,11 @@ def list_managed_nodes() -> list[dict]: """ ) rows = cur.fetchall() - return [_serialize_node_row(row) for row in rows] + items = [_serialize_node_row(row) for row in rows] + secret_flags = _load_node_secret_flags([str(item.get("node_code") or "") for item in items]) + for item in items: + item.update(secret_flags.get(str(item.get("node_code") or "").strip(), {})) + return items def sync_managed_nodes_from_cluster(*, dry_run: bool = False) -> dict: diff --git a/domain-api/app/services/ops_policy_service.py b/domain-api/app/services/ops_policy_service.py index eddc79f..07e657a 100644 --- a/domain-api/app/services/ops_policy_service.py +++ b/domain-api/app/services/ops_policy_service.py @@ -42,6 +42,7 @@ _CRITICAL_RISK_ACTIONS = { "deploy.rollback", "node.bootstrap", "cluster.reconfigure", + "runtime.reset_lab_state", } @@ -166,11 +167,23 @@ def _preview_single_node_policy( node_detect_participating = bool(target_node.get("detect_participating", False)) if target_node: - if node_status in {"busy"} and action in {"runtime.stop_worker", "runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}: - blocking_reasons.append("目标节点当前处于 busy 状态,不适合直接执行中断类动作。") + interrupt_actions = {"runtime.stop_worker", "runtime.restart_api", "service.restart"} + restart_like_actions = {"service.restart"} + rolling_deploy_actions = {"deploy.release", "deploy.rollback"} - if node_detect_participating and action in {"runtime.stop_worker", "runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}: + if node_status in {"busy"} and action in interrupt_actions - restart_like_actions: + blocking_reasons.append("目标节点当前处于 busy 状态,不适合直接执行中断类动作。") + elif node_status in {"busy"} and action in restart_like_actions: + warnings.append("目标节点当前处于 busy 状态,重启会带来瞬时抖动,请确认当前窗口可接受。") + elif node_status in {"busy"} and action in rolling_deploy_actions: + warnings.append("目标节点当前处于 busy 状态,滚动发布会触发服务重启,请确认当前窗口可接受短暂抖动。") + + if node_detect_participating and action in interrupt_actions - restart_like_actions: blocking_reasons.append("目标节点正在参与检测,需先迁移负载或人工确认后再执行。") + elif node_detect_participating and action in restart_like_actions: + warnings.append("目标节点正在参与检测,重启会中断当前任务,请确认剩余节点仍可承接负载。") + elif node_detect_participating and action in rolling_deploy_actions: + warnings.append("目标节点正在参与检测,建议优先采用单节点滚动发布,并确认其余节点仍可承接负载。") if node_role == "control" and action in {"runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}: approval_reasons.append("目标节点是 control 节点,建议强制走审批或维护窗口。") diff --git a/domain-api/app/services/ops_release_executor_core.py b/domain-api/app/services/ops_release_executor_core.py index 77fbeac..56c1730 100644 --- a/domain-api/app/services/ops_release_executor_core.py +++ b/domain-api/app/services/ops_release_executor_core.py @@ -1,8 +1,11 @@ from __future__ import annotations +import grp import hashlib import inspect import json +import os +import pwd import shutil import tarfile import textwrap @@ -13,6 +16,22 @@ from datetime import datetime from pathlib import Path +_SYSTEMD_TEMPLATE_SPECS = { + "domaincheck-api": { + "template": Path("domain-api/deploy/systemd/domain-api.service"), + }, + "domaincheck-worker": { + "template": Path("domain-api/deploy/systemd/domain-worker.service"), + }, + "domaincheck-sync-agent": { + "template": Path("domain-api/deploy/systemd/domain-sync-agent.service"), + }, + "domaincheck-node-agent": { + "template": Path("domain-api/deploy/systemd/domain-node-agent.service"), + }, +} + + def normalize_text_list(raw_value: object) -> list[str]: if isinstance(raw_value, list): return [str(item).strip() for item in raw_value if str(item).strip()] @@ -147,6 +166,219 @@ def run_release_health_checks( } +def ensure_directory_ready(path: Path) -> tuple[bool, dict]: + normalized_path = Path(path).resolve() + try: + normalized_path.mkdir(parents=True, exist_ok=True) + except Exception as exc: + return False, { + "path": str(normalized_path), + "error": str(exc), + "exception_type": exc.__class__.__name__, + } + return True, {"path": str(normalized_path)} + + +def _resolve_path_owner_group(path: Path) -> tuple[str, str]: + normalized_path = Path(path).resolve() + stat_info = normalized_path.stat() + owner_user = "" + owner_group = "" + try: + owner_user = pwd.getpwuid(stat_info.st_uid).pw_name + except Exception: + owner_user = "" + try: + owner_group = grp.getgrgid(stat_info.st_gid).gr_name + except Exception: + owner_group = "" + return owner_user, owner_group + + +def collect_service_identity(run_command, service_name: str) -> dict: + code, stdout, stderr = run_command( + ["systemctl", "show", service_name, "-p", "User", "-p", "Group", "--value"], + timeout=15, + ) + lines = [line.strip() for line in (stdout or stderr or "").splitlines()] + user = lines[0] if len(lines) >= 1 else "" + group = lines[1] if len(lines) >= 2 else "" + return { + "service_name": service_name, + "returncode": int(code or 0), + "user": user, + "group": group, + "ok": int(code or 0) == 0, + } + + +def _pick_release_owner_group( + releases_dir: Path, + service_identities: list[dict], +) -> tuple[str, str]: + for item in service_identities: + user = str(item.get("user") or "").strip() + group = str(item.get("group") or "").strip() + if user or group: + return user, group + owner_user, owner_group = _resolve_path_owner_group(releases_dir) + if owner_user or owner_group: + return owner_user, owner_group + return "", "" + + +def apply_release_permissions( + run_command, + *, + release_dir: Path, + owner_user: str, + owner_group: str, +) -> dict: + normalized_release_dir = Path(release_dir).resolve() + normalized_user = str(owner_user or "").strip() + normalized_group = str(owner_group or "").strip() + if not normalized_user and not normalized_group: + return { + "attempted": False, + "release_dir": str(normalized_release_dir), + "owner_user": normalized_user, + "owner_group": normalized_group, + "returncode": 0, + "stdout": "", + "stderr": "", + "ok": True, + } + owner_spec = f"{normalized_user}:{normalized_group}" if normalized_group else normalized_user + code, stdout, stderr = run_command( + ["chown", "-R", owner_spec, str(normalized_release_dir)], + timeout=180, + ) + return { + "attempted": True, + "release_dir": str(normalized_release_dir), + "owner_user": normalized_user, + "owner_group": normalized_group, + "owner_spec": owner_spec, + "returncode": int(code or 0), + "stdout": stdout, + "stderr": stderr, + "ok": int(code or 0) == 0, + } + + +def collect_service_execstart(run_command, service_name: str) -> dict: + code, stdout, stderr = run_command( + ["systemctl", "show", service_name, "-p", "ExecStart", "--value"], + timeout=15, + ) + execstart_value = stdout or stderr + return { + "service_name": service_name, + "returncode": int(code or 0), + "execstart": execstart_value, + "ok": int(code or 0) == 0, + } + + +def _write_text_file(path: Path, content: str) -> None: + normalized_path = Path(path).resolve() + normalized_path.parent.mkdir(parents=True, exist_ok=True) + normalized_path.write_text(str(content or "").rstrip() + "\n", encoding="utf-8") + + +def _systemd_dropin_content(service_name: str, install_root: str) -> str: + normalized_service_name = str(service_name or "").strip() + normalized_install_root = str(install_root or "").rstrip("/") + if normalized_service_name == "domaincheck-api": + return "\n".join( + [ + "[Service]", + f"WorkingDirectory={normalized_install_root}/current/domain-api", + "ExecStart=", + f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100", + ] + ) + if normalized_service_name == "domaincheck-worker": + return "\n".join( + [ + "[Service]", + f"WorkingDirectory={normalized_install_root}/current/domainCheck", + "ExecStart=", + f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python {normalized_install_root}/current/domainCheck/detect_worker.py", + ] + ) + if normalized_service_name == "domaincheck-sync-agent": + return "\n".join( + [ + "[Service]", + f"WorkingDirectory={normalized_install_root}/current/domain-api", + "ExecStart=", + f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python -m app.sync_agent", + ] + ) + if normalized_service_name == "domaincheck-node-agent": + return "\n".join( + [ + "[Service]", + "User=root", + "Group=root", + f"WorkingDirectory={normalized_install_root}/current/domain-api", + "ExecStart=", + f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python -m app.node_agent", + ] + ) + return "" + + +def _sync_release_systemd_units( + *, + release_dir: Path, + install_root: Path, + systemd_unit_root: Path, + switch_current: bool, +) -> dict: + normalized_release_dir = Path(release_dir).resolve() + normalized_systemd_root = Path(systemd_unit_root).resolve() + results: list[dict] = [] + synced_units: list[str] = [] + + for service_name, spec in _SYSTEMD_TEMPLATE_SPECS.items(): + template_path = normalized_release_dir / Path(spec["template"]) + if not template_path.exists(): + continue + + target_unit_path = normalized_systemd_root / f"{service_name}.service" + target_unit_path.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(template_path, target_unit_path) + os.chmod(target_unit_path, 0o644) + + dropin_path = normalized_systemd_root / f"{service_name}.service.d" / "current-path.conf" + dropin_written = False + if switch_current: + dropin_content = _systemd_dropin_content(service_name, str(install_root)) + if dropin_content: + _write_text_file(dropin_path, dropin_content) + dropin_written = True + + results.append( + { + "service_name": service_name, + "template_path": str(template_path), + "target_unit_path": str(target_unit_path), + "dropin_path": str(dropin_path) if dropin_written else "", + "dropin_written": dropin_written, + } + ) + synced_units.append(service_name) + + return { + "systemd_unit_root": str(normalized_systemd_root), + "synced_units": synced_units, + "results": results, + "daemon_reload_required": bool(synced_units), + } + + def safe_extract_tar(archive: tarfile.TarFile, target_dir: Path) -> None: target_dir_resolved = target_dir.resolve() members = archive.getmembers() @@ -189,6 +421,7 @@ def execute_release_action( health_check_retries = max(0, int(normalized_payload.get("health_check_retries") or 2)) health_check_interval_seconds = max(0, int(normalized_payload.get("health_check_interval_seconds") or 2)) rollback_on_failure = coerce_bool(normalized_payload.get("rollback_on_failure", True), default=True) + systemd_unit_root = Path(str(normalized_payload.get("systemd_unit_root") or "/etc/systemd/system")).resolve() if not release_version: return False, "release_version missing", {} @@ -202,8 +435,57 @@ def execute_release_action( artifact_path = downloads_dir / f"{release_version}.tar.gz" current_link = install_root / "current" previous_current_target = "" - downloads_dir.mkdir(parents=True, exist_ok=True) - releases_dir.mkdir(parents=True, exist_ok=True) + runtime_dirs = [downloads_dir, releases_dir] + prepared_dirs: list[dict] = [] + for directory in runtime_dirs: + ok, preparation = ensure_directory_ready(directory) + prepared_dirs.append(preparation) + if not ok: + event_callback( + "deploy_preflight_failed", + f"发布目录不可写: {preparation.get('path') or directory}", + level="error", + payload={ + "release_version": release_version, + "install_root": str(install_root), + "prepared_dirs": prepared_dirs, + }, + ) + return False, f"install_root not writable: {preparation.get('path') or directory}", { + "release_version": release_version, + "install_root": str(install_root), + "prepared_dirs": prepared_dirs, + } + + service_execstarts = [ + collect_service_execstart(run_command, service_name) + for service_name in restart_services + if str(service_name or "").strip() + ] + service_identities = [ + collect_service_identity(run_command, service_name) + for service_name in restart_services + if str(service_name or "").strip() + ] + current_link_text = str(current_link) + execstart_alignment = { + "switch_current": switch_current, + "current_link": current_link_text, + "services": service_execstarts, + "mismatched_services": [ + item.get("service_name") + for item in service_execstarts + if str(item.get("execstart") or "").strip() + and current_link_text not in str(item.get("execstart") or "") + ], + } + if switch_current and execstart_alignment["mismatched_services"]: + event_callback( + "deploy_execstart_mismatch", + "检测到目标服务 ExecStart 未引用 current 软链,发布后可能不会切到新版本", + level="warning", + payload=execstart_alignment, + ) if current_link.exists(): try: @@ -259,6 +541,38 @@ def execute_release_action( temp_dir.rename(target_dir) extracted_target = target_dir + release_owner_user, release_owner_group = _pick_release_owner_group(releases_dir, service_identities) + permission_result = apply_release_permissions( + run_command, + release_dir=extracted_target, + owner_user=release_owner_user, + owner_group=release_owner_group, + ) + if not permission_result.get("ok", False): + event_callback( + "deploy_permission_fix_failed", + f"发布目录权限修正失败: {release_version}", + level="error", + payload=permission_result, + ) + return False, "release permission fix failed", { + "release_version": release_version, + "release_dir": str(extracted_target), + "artifact_path": str(artifact_path), + "checksum": calculated_checksum, + "previous_current_target": previous_current_target, + "prepared_dirs": prepared_dirs, + "execstart_alignment": execstart_alignment, + "service_identities": service_identities, + "permission_result": permission_result, + } + if permission_result.get("attempted"): + event_callback( + "deploy_permissions_aligned", + f"发布目录权限已对齐: {release_version}", + payload=permission_result, + ) + meta_path = extracted_target / ".release-meta.json" meta_path.write_text( json.dumps( @@ -274,6 +588,50 @@ def execute_release_action( encoding="utf-8", ) + systemd_sync_result = _sync_release_systemd_units( + release_dir=extracted_target, + install_root=install_root, + systemd_unit_root=systemd_unit_root, + switch_current=switch_current, + ) + if systemd_sync_result.get("synced_units"): + event_callback( + "deploy_systemd_units_synced", + f"systemd 单元已同步: {', '.join(systemd_sync_result.get('synced_units') or [])}", + payload=systemd_sync_result, + ) + daemon_reload_result = { + "returncode": 0, + "stdout": "", + "stderr": "", + } + if systemd_sync_result.get("daemon_reload_required"): + code, stdout, stderr = run_command(["systemctl", "daemon-reload"], timeout=45) + daemon_reload_result = { + "returncode": int(code or 0), + "stdout": stdout, + "stderr": stderr, + } + if int(code or 0) != 0: + return False, "systemd daemon-reload failed", { + "release_version": release_version, + "release_dir": str(extracted_target), + "artifact_path": str(artifact_path), + "checksum": calculated_checksum, + "previous_current_target": previous_current_target, + "prepared_dirs": prepared_dirs, + "execstart_alignment": execstart_alignment, + "service_identities": service_identities, + "permission_result": permission_result, + "systemd_sync": systemd_sync_result, + "daemon_reload": daemon_reload_result, + } + event_callback( + "deploy_systemd_reloaded", + "systemd daemon-reload 完成", + payload=daemon_reload_result, + ) + if switch_current: if current_link.is_symlink() or current_link.is_file(): current_link.unlink(missing_ok=True) @@ -312,6 +670,12 @@ def execute_release_action( "artifact_path": str(artifact_path), "checksum": calculated_checksum, "previous_current_target": previous_current_target, + "prepared_dirs": prepared_dirs, + "execstart_alignment": execstart_alignment, + "service_identities": service_identities, + "permission_result": permission_result, + "systemd_sync": systemd_sync_result, + "daemon_reload": daemon_reload_result, "restart_results": restarted, } @@ -380,6 +744,10 @@ def execute_release_action( "checksum": calculated_checksum, "current_link": str(current_link), "previous_current_target": previous_current_target, + "prepared_dirs": prepared_dirs, + "execstart_alignment": execstart_alignment, + "systemd_sync": systemd_sync_result, + "daemon_reload": daemon_reload_result, "restart_results": restarted, "health_check": health_result, "rollback": rollback_result, @@ -397,6 +765,10 @@ def execute_release_action( "checksum": calculated_checksum, "current_link": str(current_link), "previous_current_target": previous_current_target, + "prepared_dirs": prepared_dirs, + "execstart_alignment": execstart_alignment, + "systemd_sync": systemd_sync_result, + "daemon_reload": daemon_reload_result, "restart_results": restarted, "health_check": health_result, } @@ -415,6 +787,15 @@ def build_remote_release_action_script( collect_service_state, check_health_url, run_release_health_checks, + ensure_directory_ready, + _resolve_path_owner_group, + collect_service_identity, + _pick_release_owner_group, + apply_release_permissions, + collect_service_execstart, + _write_text_file, + _systemd_dropin_content, + _sync_release_systemd_units, safe_extract_tar, execute_release_action, ] @@ -422,6 +803,17 @@ def build_remote_release_action_script( textwrap.dedent(inspect.getsource(func)).strip("\n") for func in helper_functions ) + systemd_template_specs_source = ( + "_SYSTEMD_TEMPLATE_SPECS = " + + repr( + { + service_name: { + "template": str(spec["template"]), + } + for service_name, spec in _SYSTEMD_TEMPLATE_SPECS.items() + } + ) + ) return f"""from __future__ import annotations import hashlib @@ -435,12 +827,15 @@ from datetime import datetime from pathlib import Path +{systemd_template_specs_source} + + {helper_source} -PAYLOAD = {json.dumps(dict(payload or {{}}), ensure_ascii=False)} -DEFAULT_API_SERVICE_NAME = {json.dumps(str(default_api_service_name or 'domaincheck-api'), ensure_ascii=False)} -USER_AGENT = {json.dumps(str(user_agent or 'domaincheck-ssh/0.1'), ensure_ascii=False)} +PAYLOAD = {repr(dict(payload or {}))} +DEFAULT_API_SERVICE_NAME = {repr(str(default_api_service_name or 'domaincheck-api'))} +USER_AGENT = {repr(str(user_agent or 'domaincheck-ssh/0.1'))} def _run(command, timeout=60): diff --git a/domain-api/app/services/ops_release_service.py b/domain-api/app/services/ops_release_service.py index 10871c5..49b5eb1 100644 --- a/domain-api/app/services/ops_release_service.py +++ b/domain-api/app/services/ops_release_service.py @@ -3,11 +3,13 @@ from __future__ import annotations import json import os import re +import socket import subprocess from datetime import datetime from math import ceil from pathlib import Path from threading import Lock +from urllib.parse import urlsplit, urlunsplit from uuid import uuid4 from app.core.db import get_db @@ -490,7 +492,119 @@ def _normalize_public_base_url(raw_value: str) -> str: normalized = str(raw_value or "").strip().rstrip("/") if normalized.endswith("/api/v1"): normalized = normalized[: -len("/api/v1")] - return normalized + return _rewrite_loopback_control_plane_url(normalized) + + +def _is_loopback_hostname(hostname: str) -> bool: + normalized = str(hostname or "").strip().lower().strip("[]") + return normalized in {"127.0.0.1", "localhost", "0.0.0.0", "::1"} + + +def _build_url_with_host(raw_url: str, *, host: str, scheme: str = "", port: int | None = None) -> str: + normalized_url = str(raw_url or "").strip() + if not normalized_url: + return "" + parsed = urlsplit(normalized_url) + if not parsed.scheme or not parsed.netloc: + return normalized_url + normalized_host = str(host or "").strip().strip("[]") + if not normalized_host: + return normalized_url + final_scheme = str(scheme or parsed.scheme or "http").strip() or "http" + final_port = parsed.port if port is None else int(port) + netloc = f"{normalized_host}:{final_port}" if final_port else normalized_host + return urlunsplit((final_scheme, netloc, parsed.path, parsed.query, parsed.fragment)) + + +def _resolve_public_control_plane_origin(loopback_url: str) -> str: + normalized_loopback_url = str(loopback_url or "").strip() + if not normalized_loopback_url: + return "" + parsed_loopback = urlsplit(normalized_loopback_url) + default_scheme = str(parsed_loopback.scheme or "http").strip() or "http" + default_port = parsed_loopback.port + + env_candidates = [ + os.getenv("OPS_CONTROL_PLANE_PUBLIC_BASE_URL", ""), + os.getenv("CONTROL_PLANE_PUBLIC_BASE_URL", ""), + os.getenv("OPS_CONTROL_PLANE_BASE_URL", ""), + ] + for candidate in env_candidates: + normalized_candidate = str(candidate or "").strip().rstrip("/") + if not normalized_candidate: + continue + parsed_candidate = urlsplit( + normalized_candidate if "://" in normalized_candidate else f"{default_scheme}://{normalized_candidate}" + ) + candidate_host = str(parsed_candidate.hostname or "").strip() + if candidate_host and not _is_loopback_hostname(candidate_host): + return _build_url_with_host( + normalized_loopback_url, + host=candidate_host, + scheme=str(parsed_candidate.scheme or default_scheme), + port=parsed_candidate.port if parsed_candidate.port is not None else default_port, + ) + + try: + from app.services.cluster_runtime_service import get_cluster_snapshot + + snapshot = get_cluster_snapshot() + local_hostnames = { + str(socket.gethostname() or "").strip().lower(), + str(socket.getfqdn() or "").strip().lower(), + } + fallback_control_hosts: list[str] = [] + for item in list(snapshot.get("nodes") or []): + if str(item.get("role") or "").strip() != "control": + continue + control_host = str(item.get("hostname") or "").strip().lower() + control_ip = str(item.get("ip") or "").strip() + if not control_ip or _is_loopback_hostname(control_ip): + continue + if control_host and control_host in local_hostnames: + return _build_url_with_host( + normalized_loopback_url, + host=control_ip, + scheme=default_scheme, + port=default_port, + ) + fallback_control_hosts.append(control_ip) + for control_ip in fallback_control_hosts: + if control_ip and not _is_loopback_hostname(control_ip): + return _build_url_with_host( + normalized_loopback_url, + host=control_ip, + scheme=default_scheme, + port=default_port, + ) + except Exception: + pass + + try: + resolved_host = str(socket.gethostbyname(socket.gethostname()) or "").strip() + if resolved_host and not _is_loopback_hostname(resolved_host): + return _build_url_with_host( + normalized_loopback_url, + host=resolved_host, + scheme=default_scheme, + port=default_port, + ) + except Exception: + pass + return "" + + +def _rewrite_loopback_control_plane_url(raw_url: str) -> str: + normalized = str(raw_url or "").strip() + if not normalized: + return "" + parsed = urlsplit(normalized) + if not parsed.scheme or not parsed.netloc: + return normalized + if not _is_loopback_hostname(str(parsed.hostname or "").strip()): + return normalized + resolved = _resolve_public_control_plane_origin(normalized) + return resolved or normalized def _build_absolute_release_package_url(base_url: str, raw_path: str) -> str: @@ -1684,7 +1798,7 @@ def _resolve_rollout_targets(selector: dict) -> list[dict]: continue if only_effective_workers and not bool(item.get("is_effective_worker", False)): continue - if only_online and str(item.get("status") or "") != "online": + if only_online and str(item.get("status") or "") not in {"online", "busy"}: continue targets.append(item) @@ -2037,9 +2151,12 @@ def _build_smart_rollout_role_policy(mode: str, *, execution_mode: str = "remote "restart_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"], "health_check_urls": ["http://127.0.0.1:8100/health"], "health_check_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"], - "health_check_timeout_seconds": 10, - "health_check_retries": 2, - "health_check_interval_seconds": 2, + # Control 节点启动期间会先经历较长的 import / startup hook, + # systemd 已经 active 但 /health 仍可能在 15-20 秒内拒绝连接。 + # 这里把健康检查窗口放宽到约 40 秒,避免被误回滚。 + "health_check_timeout_seconds": 20, + "health_check_retries": 9, + "health_check_interval_seconds": 4, "rollback_on_failure": True, "switch_current": True, } @@ -3254,18 +3371,56 @@ def refresh_release_rollout_for_job(job_id: int) -> dict: return refresh_release_rollout(rollout_id) -def _build_release_job_payload(release: dict, rollout: dict) -> dict: +def _default_release_deploy_payload_for_target(target: dict) -> dict: + role = str((target or {}).get("role") or "").strip().lower() + if role == "control": + return { + "restart_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"], + "health_check_urls": ["http://127.0.0.1:8100/health"], + "health_check_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"], + "health_check_timeout_seconds": 20, + "health_check_retries": 9, + "health_check_interval_seconds": 4, + } + return { + "restart_services": ["domaincheck-worker"], + "health_check_urls": [], + "health_check_services": ["domaincheck-worker"], + "health_check_timeout_seconds": 10, + "health_check_retries": 2, + "health_check_interval_seconds": 2, + } + + +def _build_release_job_payload(release: dict, rollout: dict, *, target: dict | None = None) -> dict: policy = dict(rollout.get("policy") or {}) deploy_payload = dict(policy.get("deploy_payload") or {}) + target_defaults = _default_release_deploy_payload_for_target(target or {}) + + if not [str(item).strip() for item in list(deploy_payload.get("restart_services") or []) if str(item).strip()]: + deploy_payload["restart_services"] = list(target_defaults.get("restart_services") or []) + if not [str(item).strip() for item in list(deploy_payload.get("health_check_services") or []) if str(item).strip()]: + deploy_payload["health_check_services"] = list(target_defaults.get("health_check_services") or []) + if not [str(item).strip() for item in list(deploy_payload.get("health_check_urls") or []) if str(item).strip()]: + deploy_payload["health_check_urls"] = list(target_defaults.get("health_check_urls") or []) + if deploy_payload.get("health_check_timeout_seconds") in (None, "", 0, "0"): + deploy_payload["health_check_timeout_seconds"] = int(target_defaults.get("health_check_timeout_seconds") or 10) + if deploy_payload.get("health_check_retries") in (None, "", 0, "0"): + deploy_payload["health_check_retries"] = int(target_defaults.get("health_check_retries") or 2) + if deploy_payload.get("health_check_interval_seconds") in (None, "", 0, "0"): + deploy_payload["health_check_interval_seconds"] = int(target_defaults.get("health_check_interval_seconds") or 2) + + artifact_url = _rewrite_loopback_control_plane_url(str(release.get("artifact_url") or "").strip()) return { "release_id": int(release.get("id") or 0), "rollout_id": int(rollout.get("id") or 0), "release_version": str(release.get("release_version") or ""), - "artifact_url": str(release.get("artifact_url") or ""), + "artifact_url": artifact_url, "checksum": str(release.get("checksum") or ""), "channel": str(release.get("channel") or ""), "commit_sha": str(release.get("commit_sha") or ""), "notes": str(release.get("notes") or ""), + "target_node_role": str((target or {}).get("role") or "").strip(), **deploy_payload, } @@ -3308,12 +3463,11 @@ def _enqueue_rollout_batch(rollout_id: int, *, created_by: str, reason: str = "m auto_dispatch = bool(policy.get("auto_dispatch", False)) auto_approve = bool(policy.get("auto_approve", False)) execution_mode = str(policy.get("execution_mode") or "remote-agent").strip() or "remote-agent" - job_payload = _build_release_job_payload(release, rollout) - for target in batch_targets: target_node_code = str(target.get("node_code") or "").strip() if not target_node_code: continue + job_payload = _build_release_job_payload(release, rollout, target=target) job_ok, _job_message, job_data = create_ops_job( { "action": "deploy.release", diff --git a/domain-api/app/services/ops_runtime_executor_service.py b/domain-api/app/services/ops_runtime_executor_service.py index a8ccea4..3f48f6a 100644 --- a/domain-api/app/services/ops_runtime_executor_service.py +++ b/domain-api/app/services/ops_runtime_executor_service.py @@ -1,8 +1,16 @@ from __future__ import annotations import json +import shlex import subprocess +from io import StringIO +try: + import paramiko +except ImportError: # pragma: no cover - exercised via graceful fallback tests + paramiko = None + +from app.core.db import get_db from app.core.config import settings from app.services.ops_action_executor_core import ( STRUCTURED_ACTIONS, @@ -76,6 +84,7 @@ def execute_ssh_action(node: dict, action: str, payload: dict | None = None) -> } normalized_payload = dict(payload or {}) + node_secret = _load_ssh_secret(node_code) service_names = _service_name_map() if normalized_action == "deploy.release": remote_script = build_remote_release_action_script( @@ -105,28 +114,69 @@ def execute_ssh_action(node: dict, action: str, payload: dict | None = None) -> "PY", ] ) - ssh_command = [ - "ssh", - "-o", - "BatchMode=yes", - "-o", - "PreferredAuthentications=publickey", - "-o", - "StrictHostKeyChecking=accept-new", - "-o", - f"ConnectTimeout={_SSH_CONNECT_TIMEOUT_SECONDS}", - "-p", - str(ssh_port), - f"{ssh_user}@{ssh_host}", - remote_command, - ] timeout_seconds = int(_SSH_REMOTE_TIMEOUT_SECONDS.get(normalized_action, 45) or 45) + _SSH_CONNECT_TIMEOUT_SECONDS - completed = subprocess.run( - ssh_command, - capture_output=True, - text=True, - timeout=timeout_seconds, - ) + auth_mode = str(node.get("auth_mode") or "").strip() or ("password" if node_secret.get("ssh_password") else "key") + ssh_password = str(node_secret.get("ssh_password") or "").strip() + ssh_private_key = str(node_secret.get("ssh_private_key") or "").strip() + if auth_mode == "key" and not ssh_private_key and ssh_password: + auth_mode = "password" + elif auth_mode == "password" and not ssh_password and ssh_private_key: + auth_mode = "key" + if (auth_mode == "password" and ssh_password) or ssh_private_key: + if paramiko is None: + return False, "当前环境未安装 paramiko,无法使用密码或私钥 SSH 执行", { + "executor": "ssh", + "action": normalized_action, + "transport": { + "executor": "ssh", + "node_code": node_code, + "ssh_host": ssh_host, + "ssh_user": ssh_user, + "ssh_port": ssh_port, + "auth_mode": auth_mode, + "action": normalized_action, + }, + } + if auth_mode == "password" and ssh_password: + completed = _run_paramiko_command( + ssh_host=ssh_host, + ssh_port=ssh_port, + ssh_user=ssh_user, + remote_command=remote_command, + timeout_seconds=timeout_seconds, + ssh_password=ssh_password, + ) + elif ssh_private_key: + completed = _run_paramiko_command( + ssh_host=ssh_host, + ssh_port=ssh_port, + ssh_user=ssh_user, + remote_command=remote_command, + timeout_seconds=timeout_seconds, + ssh_private_key=ssh_private_key, + ) + else: + ssh_command = [ + "ssh", + "-o", + "BatchMode=yes", + "-o", + "PreferredAuthentications=publickey", + "-o", + "StrictHostKeyChecking=accept-new", + "-o", + f"ConnectTimeout={_SSH_CONNECT_TIMEOUT_SECONDS}", + "-p", + str(ssh_port), + f"{ssh_user}@{ssh_host}", + remote_command, + ] + completed = subprocess.run( + ssh_command, + capture_output=True, + text=True, + timeout=timeout_seconds, + ) transport = { "executor": "ssh", @@ -134,6 +184,7 @@ def execute_ssh_action(node: dict, action: str, payload: dict | None = None) -> "ssh_host": ssh_host, "ssh_user": ssh_user, "ssh_port": ssh_port, + "auth_mode": auth_mode, "action": normalized_action, "returncode": int(completed.returncode or 0), } @@ -166,6 +217,93 @@ def execute_ssh_action(node: dict, action: str, payload: dict | None = None) -> } +def _load_ssh_secret(node_code: str) -> dict: + normalized_node_code = str(node_code or "").strip() + if not normalized_node_code: + return {} + try: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT ssh_password, ssh_private_key + FROM ops_managed_node_secrets + WHERE node_code = %s + LIMIT 1 + """, + (normalized_node_code,), + ) + row = cur.fetchone() + except Exception: + return {} + if not row: + return {} + return { + "ssh_password": str(row[0] or ""), + "ssh_private_key": str(row[1] or ""), + } + + +def _load_private_key(private_key_text: str) -> paramiko.PKey: + if paramiko is None: + raise RuntimeError("paramiko is not installed") + key_text = str(private_key_text or "") + for key_cls in (paramiko.Ed25519Key, paramiko.RSAKey, paramiko.ECDSAKey, paramiko.DSSKey): + try: + return key_cls.from_private_key(StringIO(key_text)) + except Exception: + continue + raise ValueError("无法识别 SSH 私钥格式") + + +def _run_paramiko_command( + *, + ssh_host: str, + ssh_port: int, + ssh_user: str, + remote_command: str, + timeout_seconds: int, + ssh_password: str = "", + ssh_private_key: str = "", +) -> subprocess.CompletedProcess: + if paramiko is None: + raise RuntimeError("paramiko is not installed") + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + connect_kwargs = { + "hostname": ssh_host, + "port": int(ssh_port), + "username": ssh_user, + "timeout": _SSH_CONNECT_TIMEOUT_SECONDS, + "banner_timeout": _SSH_CONNECT_TIMEOUT_SECONDS, + "auth_timeout": _SSH_CONNECT_TIMEOUT_SECONDS, + "look_for_keys": False, + "allow_agent": False, + } + if ssh_password: + connect_kwargs["password"] = ssh_password + elif ssh_private_key: + connect_kwargs["pkey"] = _load_private_key(ssh_private_key) + else: + connect_kwargs["look_for_keys"] = True + connect_kwargs["allow_agent"] = True + try: + client.connect(**connect_kwargs) + wrapped_command = f"bash -lc {shlex.quote(remote_command)}" + _, stdout, stderr = client.exec_command(wrapped_command, timeout=timeout_seconds) + returncode = int(stdout.channel.recv_exit_status()) + stdout_text = stdout.read().decode("utf-8", errors="replace") + stderr_text = stderr.read().decode("utf-8", errors="replace") + return subprocess.CompletedProcess( + args=["paramiko", f"{ssh_user}@{ssh_host}"], + returncode=returncode, + stdout=stdout_text, + stderr=stderr_text, + ) + finally: + client.close() + + def _service_name_map() -> dict[str, str]: return build_service_name_map( api_service_name=settings.api_service_name, @@ -199,6 +337,25 @@ def trim_output(text, limit): def run(cmd, timeout=60): completed = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + normalized_cmd = [str(part or "").strip() for part in cmd] + combined_output = f"{{completed.stdout or ''}}\\n{{completed.stderr or ''}}".lower() + needs_sudo_retry = ( + normalized_cmd + and normalized_cmd[0] == "systemctl" + and completed.returncode != 0 + and "sudo" not in normalized_cmd + and any( + marker in combined_output + for marker in ( + "interactive authentication required", + "authentication is required", + "authorization not available", + "polkit", + ) + ) + ) + if needs_sudo_retry: + completed = subprocess.run(["sudo", "-n", *normalized_cmd], capture_output=True, text=True, timeout=timeout) return int(completed.returncode or 0), str(completed.stdout or "").strip(), str(completed.stderr or "").strip() diff --git a/domain-api/app/services/runtime_control_service.py b/domain-api/app/services/runtime_control_service.py index 5f7d7ef..1fc8de7 100644 --- a/domain-api/app/services/runtime_control_service.py +++ b/domain-api/app/services/runtime_control_service.py @@ -6,6 +6,7 @@ from pathlib import Path from app.core.config import settings from app.services.debug_event_service import push_debug_event +from app.services.detect_job_service import process_detect_pipeline_now from app.services.sync_push_service import pull_detect_task_batch_now, push_runtime_projection_now from app.services.runtime_settings_service import get_runtime_settings from app.services.worker_control_service import _run_systemctl, normalize_systemctl_error, send_worker_command, start_worker, stop_worker @@ -180,6 +181,16 @@ def runtime_action(action: str, payload: dict | None = None) -> tuple[bool, str, result = _build_runtime_action_result(action=normalized_action, poll_after_seconds=2, refresh_runtime=True, ok=ok, message=message, data=data) _emit_runtime_action_event(normalized_action, stage="finished", ok=ok, message=message, data=result) return ok, message, result + if normalized_action == "process_pipeline": + process_limit = normalized_payload.get("limit") + process_job_id = normalized_payload.get("job_id") + ok, message, data = process_detect_pipeline_now( + limit=int(process_limit or 0) or None, + job_id=int(process_job_id or 0) or None, + ) + result = _build_runtime_action_result(action=normalized_action, poll_after_seconds=1, refresh_runtime=True, ok=ok, message=message, data=data) + _emit_runtime_action_event(normalized_action, stage="finished", ok=ok, message=message, data=result) + return ok, message, result if normalized_action == "start_detection": command_ok, command_message = send_worker_command( "start_detection", diff --git a/domain-api/app/services/runtime_status_service.py b/domain-api/app/services/runtime_status_service.py index d12231b..3a3699a 100644 --- a/domain-api/app/services/runtime_status_service.py +++ b/domain-api/app/services/runtime_status_service.py @@ -10,12 +10,151 @@ from app.core.redis_client import get_redis from app.services.build_info_service import get_runtime_build_info from app.services.cluster_runtime_service import get_cluster_snapshot from app.services.detect_service import get_detect_status -from app.services.detect_job_service import get_detect_capacity_plan, get_detect_queue_health +from app.services.detect_job_service import ( + _load_latest_runtime_active_job_snapshot, + get_detect_capacity_plan, + get_detect_queue_health, +) from app.services.sync_record_service import append_runtime_projection_if_changed, get_sync_summary from app.services.runtime_settings_service import get_runtime_settings from app.services.worker_control_service import detect_sync_agent_runtime, detect_worker_runtime +def _align_queue_health_with_backlog(queue_health: dict | None, backlog_snapshot: dict | None) -> dict: + normalized = dict(queue_health or {}) + queue = dict(normalized.get("queue") or {}) + backlog = dict(backlog_snapshot or {}) + + pending_total = max(int(queue.get("pending", 0) or 0), int(backlog.get("pending_total", 0) or 0)) + claimed_total = max(int(queue.get("claimed", 0) or 0), int(backlog.get("claimed_total", 0) or 0)) + running_total = max(int(queue.get("running", 0) or 0), int(backlog.get("running_total", 0) or 0)) + completed_total = max(int(queue.get("completed", 0) or 0), int(backlog.get("completed_total", 0) or 0)) + blacklisted_total = max(int(queue.get("blacklisted", 0) or 0), int(backlog.get("blacklisted_total", 0) or 0)) + failed_total = max(int(queue.get("failed", 0) or 0), int(backlog.get("failed_total", 0) or 0)) + terminal_total = max( + int(queue.get("terminal", 0) or 0), + completed_total + blacklisted_total + failed_total, + ) + + normalized["has_active_job"] = bool( + normalized.get("has_active_job") + or pending_total > 0 + or claimed_total > 0 + or running_total > 0 + or terminal_total > 0 + ) + normalized["queue"] = { + **queue, + "items_total": pending_total + claimed_total + running_total + terminal_total, + "pending": pending_total, + "claimed": claimed_total, + "running": running_total, + "completed": completed_total, + "blacklisted": blacklisted_total, + "failed": failed_total, + "terminal": terminal_total, + } + return normalized + + +def _decode_projection_payload(value: object) -> dict: + if isinstance(value, dict): + return dict(value) + if value in (None, ""): + return {} + try: + import json + + return dict(json.loads(value)) + except Exception: + return {} + + +def _load_detect_backlog_snapshot() -> dict: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT + COUNT(*) FILTER (WHERE item.status = 'pending') AS pending_total, + COUNT(*) FILTER (WHERE item.status = 'claimed') AS claimed_total, + COUNT(*) FILTER (WHERE item.status = 'running') AS running_total, + COUNT(*) FILTER (WHERE item.status = 'completed') AS completed_total, + COUNT(*) FILTER (WHERE item.status = 'blacklisted') AS blacklisted_total, + COUNT(*) FILTER (WHERE item.status = 'failed') AS failed_total, + COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code = 'detect_register') AS register_pending, + COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code <> 'detect_register') AS downstream_pending + FROM detect_job_items item + JOIN detect_jobs job ON job.id = item.job_id + WHERE job.status IN ('pending', 'running') + """ + ) + row = cur.fetchone() or (0, 0, 0, 0, 0, 0, 0, 0) + return { + "pending_total": int(row[0] or 0), + "claimed_total": int(row[1] or 0), + "running_total": int(row[2] or 0), + "completed_total": int(row[3] or 0), + "blacklisted_total": int(row[4] or 0), + "failed_total": int(row[5] or 0), + "register_pending": int(row[6] or 0), + "downstream_pending": int(row[7] or 0), + } + + +def _load_latest_remote_runtime_projection_backlog() -> dict: + if not (settings.node_region == "overseas" and settings.node_role == "control"): + return {} + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT payload_json + FROM detect_sync_records + WHERE sync_type = 'runtime_projection' + AND source_region = 'mainland' + AND target_region = 'overseas' + AND status IN ('projected', 'pushing', 'synced') + ORDER BY updated_at DESC, id DESC + LIMIT 1 + """ + ) + row = cur.fetchone() + if not row: + return {} + payload = _decode_projection_payload(row[0]) + projection = payload.get("projection") if isinstance(payload, dict) else {} + backlog = projection.get("backlog") if isinstance(projection, dict) else {} + if not isinstance(backlog, dict): + return {} + return { + "pending_total": int(backlog.get("pending_total", 0) or 0), + "claimed_total": int(backlog.get("claimed_total", 0) or 0), + "running_total": int(backlog.get("running_total", 0) or 0), + "completed_total": int(backlog.get("completed_total", 0) or 0), + "blacklisted_total": int(backlog.get("blacklisted_total", 0) or 0), + "failed_total": int(backlog.get("failed_total", 0) or 0), + "register_pending": int(backlog.get("register_pending", 0) or 0), + "downstream_pending": int(backlog.get("downstream_pending", 0) or 0), + } + + +def _merge_backlog_snapshots(primary: dict, secondary: dict) -> dict: + merged = dict(primary or {}) + for key in ( + "pending_total", + "claimed_total", + "running_total", + "completed_total", + "blacklisted_total", + "failed_total", + "register_pending", + "downstream_pending", + ): + merged[key] = max(int(merged.get(key, 0) or 0), int((secondary or {}).get(key, 0) or 0)) + return merged + + def _runtime_log_path(filename: str) -> str: path = Path(__file__).resolve().parents[2] / "runtime" / "logs" / filename return str(path) @@ -191,6 +330,8 @@ def _build_multi_region_readiness( def _detect_participation_snapshot(*, row: dict) -> dict: items_running = int(row.get("items_running", 0) or 0) items_claimed = int(row.get("items_claimed", 0) or 0) + active_threads = int(row.get("active_threads", 0) or 0) + max_threads = int(row.get("max_threads", 0) or 0) processed_recent = int(row.get("processed_recent", 0) or 0) current_load = int(row.get("current_load", 0) or 0) status = str(row.get("status") or "").strip().lower() @@ -211,6 +352,17 @@ def _detect_participation_snapshot(*, row: dict) -> dict: "is_current_participant": True, "is_dispatch_active": True, } + if active_threads > 0: + detail = f"当前活跃线程 {active_threads}" + if max_threads > 0: + detail = f"{detail}/{max_threads}" + return { + "participation_state": "runtime_active", + "participation_label": "执行中", + "participation_reason": detail, + "is_current_participant": True, + "is_dispatch_active": True, + } if processed_recent > 0: return { "participation_state": "recent_throughput", @@ -242,11 +394,52 @@ def _build_detect_node_row(*, node_code: str, cluster_node: dict, job_node: dict role = str(cluster_node.get("role") or job_node.get("role") or "worker") region = str(cluster_node.get("region") or settings.node_region) is_effective_worker = bool(cluster_node.get("is_effective_worker", False) or role == "worker") - items_total = int(job_node.get("items_total", metadata.get("job_items_total", 0)) or 0) - items_claimed = int(job_node.get("items_claimed", metadata.get("job_items_claimed", 0)) or 0) - items_running = int(job_node.get("items_running", metadata.get("job_items_running", 0)) or 0) - items_completed = int(job_node.get("items_completed", metadata.get("job_items_completed", 0)) or 0) - items_failed = int(job_node.get("items_failed", metadata.get("job_items_failed", 0)) or 0) + items_total = max( + int(job_node.get("items_total", 0) or 0), + int(queue_node.get("items_total", 0) or 0), + int(metadata.get("job_items_total", 0) or 0), + ) + items_claimed = max( + int(job_node.get("items_claimed", 0) or 0), + int(queue_node.get("items_claimed", 0) or 0), + int(metadata.get("job_items_claimed", 0) or 0), + ) + items_running = max( + int(job_node.get("items_running", 0) or 0), + int(queue_node.get("items_running", 0) or 0), + int(metadata.get("job_items_running", 0) or 0), + ) + items_completed = max( + int(job_node.get("items_completed", 0) or 0), + int(queue_node.get("items_completed", 0) or 0), + int(metadata.get("job_items_completed", 0) or 0), + ) + items_failed = max( + int(job_node.get("items_failed", 0) or 0), + int(queue_node.get("items_failed", 0) or 0), + int(metadata.get("job_items_failed", 0) or 0), + ) + items_blacklisted = max( + int(job_node.get("items_blacklisted", 0) or 0), + int(queue_node.get("items_blacklisted", 0) or 0), + ) + active_threads = max( + int((cluster_node.get("metadata") or {}).get("active_threads", 0) or 0), + int(job_node.get("active_threads", 0) or 0), + int(queue_node.get("active_threads", 0) or 0), + ) + max_threads = max( + int((cluster_node.get("metadata") or {}).get("max_threads", 0) or 0), + int(job_node.get("max_threads", 0) or 0), + int(queue_node.get("max_threads", 0) or 0), + ) + current_load = max(items_running, active_threads, 0) + derived_pending = max(items_total - items_claimed - items_running - items_completed - items_failed - items_blacklisted, 0) + items_pending = max( + int(job_node.get("items_pending", 0) or 0), + int(queue_node.get("items_pending", 0) or 0), + derived_pending, + ) row = { "node_code": node_code, "role": role, @@ -254,13 +447,16 @@ def _build_detect_node_row(*, node_code: str, cluster_node: dict, job_node: dict "status": str(cluster_node.get("status") or "unknown"), "is_effective_worker": is_effective_worker, "detect_participating": False, - "current_load": int(cluster_node.get("current_load", 0) or 0), + "current_load": current_load, "items_total": items_total, - "items_pending": int(job_node.get("items_pending", max(items_total - items_claimed - items_completed - items_failed, 0)) or 0), + "items_pending": items_pending, "items_claimed": items_claimed, "items_running": items_running, "items_completed": items_completed, + "items_blacklisted": items_blacklisted, "items_failed": items_failed, + "active_threads": active_threads, + "max_threads": max_threads, "processed_recent": int(queue_node.get("processed_recent", 0) or 0), "processed_per_minute": float(queue_node.get("processed_per_minute", 0) or 0), "last_heartbeat_at": str(cluster_node.get("last_heartbeat_at") or ""), @@ -465,8 +661,13 @@ def get_runtime_status() -> dict: effective_online_worker_nodes = int((cluster_snapshot.get("summary") or {}).get("online_worker_nodes", 0) or 0) if effective_online_worker_nodes <= 0 and worker_runtime.get("running", False): effective_online_worker_nodes = max(1, worker_runtime.get("process_count", 1) or 1) + backlog_snapshot = _load_detect_backlog_snapshot() + remote_backlog_snapshot = _load_latest_remote_runtime_projection_backlog() + runtime_snapshot_backlog = dict(_load_latest_runtime_active_job_snapshot(15).get("backlog") or {}) + backlog_snapshot = _merge_backlog_snapshots(backlog_snapshot, remote_backlog_snapshot) + backlog_snapshot = _merge_backlog_snapshots(backlog_snapshot, runtime_snapshot_backlog) capacity_plan = get_detect_capacity_plan( - queue_health=queue_health, + queue_health=_align_queue_health_with_backlog(queue_health, backlog_snapshot), online_worker_nodes=effective_online_worker_nodes, target_finish_hours=6, ) @@ -498,6 +699,7 @@ def get_runtime_status() -> dict: "worker_online": worker_runtime.get("running", False), "worker_mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")), "queue_health": queue_health, + "backlog": backlog_snapshot, "capacity_plan": capacity_plan, "log_sync": { "enabled": bool(runtime_settings.get("worker_log_sync_enabled", False)), @@ -565,8 +767,7 @@ def get_runtime_status() -> dict: ) build_info = get_runtime_build_info() - return { - "api": { + api_payload = { "service": "domain-api", "version": "0.1.0", "api_prefix": settings.api_prefix, @@ -579,13 +780,13 @@ def get_runtime_status() -> dict: "stdout_log": _runtime_log_path("domain-api.stdout.log"), "stderr_log": _runtime_log_path("domain-api.stderr.log"), "build": build_info, - }, - "node": { + } + node_payload = { "code": settings.node_code, "region": settings.node_region, "role": settings.node_role, - }, - "worker": { + } + worker_payload = { "mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")), "service_name": runtime_settings.get("worker_service_name", settings.worker_service_name), "running": worker_runtime.get("running", False), @@ -594,8 +795,8 @@ def get_runtime_status() -> dict: "latest_start_time": worker_runtime.get("latest_start_time", ""), "message": worker_runtime.get("message", ""), "log_path": str(Path(settings.domain_root) / "detect_worker.log"), - }, - "sync_agent": { + } + sync_agent_payload = { "mode": sync_agent_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")), "service_name": runtime_settings.get("sync_agent_service_name", settings.sync_agent_service_name), "running": sync_agent_runtime.get("running", False), @@ -603,7 +804,44 @@ def get_runtime_status() -> dict: "latest_start_time": sync_agent_runtime.get("latest_start_time", ""), "message": sync_agent_runtime.get("message", ""), "expected_on_this_node": settings.node_region == "mainland" and settings.node_role == "control", - }, + } + compatibility_payload = { + # Backward-compatible flat fields for older pages / stale built assets. + "api_online": bool(api_payload.get("pid")), + "api_service_name": api_payload.get("service_name", ""), + "worker_online": bool(worker_payload.get("running", False)), + "worker_mode": worker_payload.get("mode", ""), + "worker_service_name": worker_payload.get("service_name", ""), + "worker_process_count": worker_payload.get("process_count", 0), + "worker_latest_start_time": worker_payload.get("latest_start_time", ""), + "worker_runtime_message": worker_payload.get("message", ""), + "thread_count": detect_snapshot.get("thread_count", 0), + "thread_count_default": detect_snapshot.get("thread_count_default", 0), + "thread_count_source": detect_snapshot.get("thread_count_source", ""), + "thread_count_override": detect_snapshot.get("thread_count_override"), + "active_thread_count": detect_payload.get("active_thread_count", 0), + "max_thread_count": detect_payload.get("max_thread_count", 0), + "progress": detect_payload.get("progress", {}), + "backlog": detect_payload.get("backlog", {}), + "progress_percent": detect_payload.get("progress_percent", 0), + "available_proxy_count": detect_payload.get("available_proxy_count", 0), + "proxy_pool_count": detect_payload.get("proxy_pool_count", 0), + "proxy_runtime_label": detect_payload.get("proxy_runtime_label", ""), + "proxy_runtime_detail": detect_payload.get("proxy_runtime_detail", ""), + "proxy_runtime_reason": detect_payload.get("proxy_runtime_reason", ""), + "proxy_last_refresh_time": detect_payload.get("proxy_last_refresh_time", ""), + "recent_event": detect_payload.get("recent_event", ""), + "recent_warning": detect_payload.get("recent_warning", ""), + "runtime_state": worker_runtime.get("runtime_state") or {}, + "cluster_summary": cluster_snapshot.get("summary") or {}, + } + + return { + **compatibility_payload, + "api": api_payload, + "node": node_payload, + "worker": worker_payload, + "sync_agent": sync_agent_payload, "detect": detect_payload, "cluster": cluster_snapshot, "sync": sync_summary, diff --git a/domain-api/app/services/sensitive_words_service.py b/domain-api/app/services/sensitive_words_service.py index 401644b..bcf4c7c 100644 --- a/domain-api/app/services/sensitive_words_service.py +++ b/domain-api/app/services/sensitive_words_service.py @@ -1,6 +1,9 @@ from __future__ import annotations +import json + from app.core.db import get_db +from app.core.redis_client import get_redis def get_sensitive_words_payload() -> dict: @@ -54,6 +57,13 @@ def save_sensitive_words_payload(payload: dict) -> dict: ) conn.commit() + try: + redis_client = get_redis() + redis_client.set("domain_tool:sensitive_words", json.dumps(words, ensure_ascii=False)) + redis_client.publish("domain_tool:config_update", "sensitive_words") + except Exception: + pass + return { "total": len(words), "text": "\n".join(words), diff --git a/domain-api/app/services/settings_service.py b/domain-api/app/services/settings_service.py index 4e1a43a..edd4ba3 100644 --- a/domain-api/app/services/settings_service.py +++ b/domain-api/app/services/settings_service.py @@ -35,8 +35,8 @@ def _normalize_thread_count(value: object, *, field_name: str = "thread_count") thread_count = int(value) except Exception as exc: raise ValueError(f"{field_name} must be an integer") from exc - if thread_count < 1 or thread_count > 256: - raise ValueError(f"{field_name} out of range") + if thread_count < 1: + raise ValueError(f"{field_name} must be >= 1") return thread_count diff --git a/domain-api/app/services/sync_push_service.py b/domain-api/app/services/sync_push_service.py index d04e19c..6df4820 100644 --- a/domain-api/app/services/sync_push_service.py +++ b/domain-api/app/services/sync_push_service.py @@ -11,7 +11,16 @@ from uuid import uuid4 from app.core.config import settings from app.core.db import get_db -from app.services.cluster_runtime_service import cleanup_imported_runtime_nodes, register_node_heartbeat +from app.services.cluster_runtime_service import ( + cleanup_imported_runtime_nodes, + cleanup_imported_runtime_nodes_many, + register_node_heartbeat, +) +from app.services.detect_job_service import ( + _load_domain_pipeline_snapshot, + resolve_initial_domain_pipeline_item, +) +from app.services.settings_service import get_settings_payload from app.services.sync_record_service import _decode_json, _normalize_region @@ -80,10 +89,16 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv if not node_code: node_code = f"{region}-{role}-imported" + controller_current_load = max( + int(projection.get("active_thread_count", 0) or 0), + int(((projection.get("active_job") or {}).get("items_running", 0) or 0)), + ) metadata = { "service": "runtime-ingest", "projection_source_region": source_region, "worker_mode": projection.get("worker_mode", ""), + "active_threads": int(projection.get("active_thread_count", 0) or 0), + "max_threads": int(projection.get("max_thread_count", 0) or 0), "phase_label": projection.get("phase_label", ""), "phase_detail": projection.get("phase_detail", ""), "proxy_runtime_label": projection.get("proxy_runtime_label", ""), @@ -102,8 +117,8 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv node_code=node_code, region=region, role=role, - status="online", - current_load=int(((projection.get("progress") or {}).get("running", 0) or 0)), + status="busy" if controller_current_load > 0 else "online", + current_load=controller_current_load, metadata=metadata, hostname_override=hostname, ip_override=ip, @@ -111,15 +126,21 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv cleanup_imported_runtime_nodes(region=region, role=role, keep_node_code=node_code) active_job = projection.get("active_job") or {} + worker_node_codes: list[str] = [] for node_stat in list(active_job.get("node_stats") or []): worker_node_code = str(node_stat.get("node_code") or "").strip() if not worker_node_code or worker_node_code == "unassigned": continue + if worker_node_code == node_code: + continue items_running = int(node_stat.get("items_running", 0) or 0) items_claimed = int(node_stat.get("items_claimed", 0) or 0) items_total = int(node_stat.get("items_total", 0) or 0) - worker_status = "busy" if (items_running > 0 or items_claimed > 0) else "online" - worker_load = max(items_running, items_claimed, 0) + worker_runtime_load = int(node_stat.get("current_load", 0) or 0) + worker_active_threads = int(node_stat.get("active_threads", worker_runtime_load) or 0) + worker_max_threads = int(node_stat.get("max_threads", 0) or 0) + worker_load = max(worker_active_threads, items_running, 0) + worker_status = "busy" if worker_load > 0 else "online" worker_metadata = { "service": "runtime-ingest", "projection_source_region": source_region, @@ -128,10 +149,18 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv "phase_detail": projection.get("phase_detail", ""), "proxy_runtime_label": projection.get("proxy_runtime_label", ""), "proxy_runtime_reason": projection.get("proxy_runtime_reason", ""), + "active_threads": worker_active_threads, + "max_threads": worker_max_threads, "updated_at": _format_time(received_at or datetime.now()), "job_items_total": items_total, "job_items_running": items_running, "job_items_claimed": items_claimed, + "job_items_completed": int(node_stat.get("items_completed", 0) or 0), + "job_items_failed": int(node_stat.get("items_failed", 0) or 0), + "job_items_blacklisted": int(node_stat.get("items_blacklisted", 0) or 0), + "metrics_source": str(node_stat.get("metrics_source") or "runtime").strip() or "runtime", + "source_status": str(node_stat.get("status") or "").strip(), + "source_role": str(node_stat.get("role") or "worker").strip() or "worker", "derived_from": node_code, } register_node_heartbeat( @@ -144,7 +173,9 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv hostname_override=hostname, ip_override=ip, ) - cleanup_imported_runtime_nodes(region=region, role="worker", keep_node_code=worker_node_code) + worker_node_codes.append(worker_node_code) + if worker_node_codes: + cleanup_imported_runtime_nodes_many(region=region, role="worker", keep_node_codes=worker_node_codes) def _load_latest_projection(sync_type: str) -> dict | None: @@ -226,6 +257,88 @@ def _load_pushable_projections(sync_type: str, limit: int) -> list[dict]: return selected +def _estimate_total_worker_threads(settings_payload: dict | None = None) -> int: + payload = settings_payload if isinstance(settings_payload, dict) else get_settings_payload() + default_threads = max(1, int(payload.get("thread_count", 100) or 100)) + node_thread_counts = payload.get("node_thread_counts") if isinstance(payload.get("node_thread_counts"), dict) else {} + total_threads = 0 + for raw_value in node_thread_counts.values(): + try: + total_threads += max(0, int(raw_value or 0)) + except (TypeError, ValueError): + continue + return max(total_threads, default_threads) + + +def _load_local_detect_backlog_snapshot() -> dict: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT + COUNT(*) FILTER (WHERE item.status = 'pending') AS pending_total, + COUNT(*) FILTER (WHERE item.status = 'claimed') AS claimed_total, + COUNT(*) FILTER (WHERE item.status = 'running') AS running_total, + COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code = 'detect_register') AS register_pending, + COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code <> 'detect_register') AS downstream_pending + FROM detect_job_items item + JOIN detect_jobs job ON job.id = item.job_id + WHERE job.status IN ('pending', 'running') + """ + ) + row = cur.fetchone() or (0, 0, 0, 0, 0) + return { + "pending_total": int(row[0] or 0), + "claimed_total": int(row[1] or 0), + "running_total": int(row[2] or 0), + "register_pending": int(row[3] or 0), + "downstream_pending": int(row[4] or 0), + } + + +def _build_task_pull_backlog_limits(configured_limit: int, settings_payload: dict | None = None) -> dict: + estimated_total_threads = _estimate_total_worker_threads(settings_payload) + max_pending_total = int(settings.sync_pull_max_pending_items or 0) + if max_pending_total <= 0: + max_pending_total = max(int(configured_limit or 0), estimated_total_threads * 2) + + max_register_pending = int(settings.sync_pull_max_register_pending_items or 0) + if max_register_pending <= 0: + max_register_pending = max(max(500, int(configured_limit or 0) // 2), estimated_total_threads) + + max_downstream_pending = int(settings.sync_pull_max_downstream_pending_items or 0) + if max_downstream_pending <= 0: + max_downstream_pending = max(250, estimated_total_threads // 4) + + return { + "estimated_total_threads": estimated_total_threads, + "max_pending_total": max_pending_total, + "max_register_pending": max_register_pending, + "max_downstream_pending": max_downstream_pending, + } + + +def _should_throttle_task_pull(backlog_snapshot: dict, backlog_limits: dict) -> tuple[bool, str]: + pending_total = int(backlog_snapshot.get("pending_total", 0) or 0) + register_pending = int(backlog_snapshot.get("register_pending", 0) or 0) + downstream_pending = int(backlog_snapshot.get("downstream_pending", 0) or 0) + max_pending_total = int(backlog_limits.get("max_pending_total", 0) or 0) + max_register_pending = int(backlog_limits.get("max_register_pending", 0) or 0) + max_downstream_pending = int(backlog_limits.get("max_downstream_pending", 0) or 0) + + if max_pending_total > 0 and pending_total >= max_pending_total: + return True, "pending_total" + if ( + downstream_pending > 0 + and max_register_pending > 0 + and register_pending >= max_register_pending + ): + return True, "register_pending" + if max_downstream_pending > 0 and downstream_pending >= max_downstream_pending: + return True, "downstream_pending" + return False, "" + + def _latest_push_attempt(source_record_id: int, target_region: str, sync_type: str) -> dict | None: with get_db() as conn: with conn.cursor() as cur: @@ -308,10 +421,51 @@ def _task_selection_sql() -> str: """ +def _task_projection_limit(limit: int | None) -> int: + requested = max(1, int(limit or 5000)) + configured = max(5000, int(settings.sync_batch_size or 200)) + cap = max(10000, configured, 5000) + return max(1, min(requested, cap)) + + +def _task_projection_items_total(projection: dict) -> int: + payload = projection.get("payload") or {} + projection_payload = payload.get("projection") or {} + try: + return int(projection_payload.get("items_total", 0) or 0) + except Exception: + return 0 + + +def _task_projection_selection_limit(projection: dict) -> int: + payload = projection.get("payload") or {} + projection_payload = payload.get("projection") or {} + try: + return int(projection_payload.get("selection_limit", 0) or 0) + except Exception: + return 0 + + +def _mark_task_projection_superseded(record_id: int, *, reason: str) -> None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + UPDATE detect_sync_records + SET status = 'superseded', + error_message = %s, + updated_at = CURRENT_TIMESTAMP + WHERE id = %s + """, + (str(reason or "").strip()[:500], int(record_id)), + ) + conn.commit() + + def _load_pending_task_projection(limit: int) -> dict | None: source_region = _normalize_region(settings.sync_source_region, settings.node_region) target_region = _normalize_region(settings.sync_target_region, "overseas") - safe_limit = max(1, min(int(limit or 1000), max(1, int(settings.sync_batch_size or 200)))) + safe_limit = _task_projection_limit(limit) with get_db() as conn: with conn.cursor() as cur: cur.execute( @@ -340,6 +494,22 @@ def _load_pending_task_projection(limit: int) -> dict | None: latest_ingest = _latest_ingest_attempt(projection["id"], projection["target_region"], "detect_task_projection") if latest_ingest and latest_ingest["status"] == "received": continue + items_total = _task_projection_items_total(projection) + selection_limit = _task_projection_selection_limit(projection) + created_at = projection.get("created_at") + if ( + safe_limit >= 1000 + and max(items_total, selection_limit) > 0 + and max(items_total, selection_limit) < safe_limit + and isinstance(created_at, datetime) + ): + now = datetime.now(created_at.tzinfo) if created_at.tzinfo else datetime.now() + if now - created_at >= timedelta(minutes=10): + _mark_task_projection_superseded( + projection["id"], + reason=f"stale small task projection skipped: items_total={items_total}, selection_limit={selection_limit}, requested_limit={safe_limit}", + ) + continue return projection return None @@ -361,7 +531,7 @@ def export_detect_task_projection(limit: int = 1000, *, shared_token: str | None source_region = _normalize_region(settings.sync_source_region, settings.node_region) target_region = _normalize_region(settings.sync_target_region, "overseas") - safe_limit = max(1, min(int(limit or 1000), max(1, int(settings.sync_batch_size or 200)))) + safe_limit = _task_projection_limit(limit) with get_db() as conn: with conn.cursor() as cur: cur.execute(_task_selection_sql(), (safe_limit,)) @@ -528,6 +698,7 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N inserted_count = 0 updated_count = 0 + domain_ids: list[int] = [] for item in items: domain = str(item.get("domain") or "").strip().lower() if not domain: @@ -562,7 +733,7 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N ELSE EXCLUDED.detect_status END, update_time = CURRENT_TIMESTAMP - RETURNING (xmax = 0) AS inserted + RETURNING id, (xmax = 0) AS inserted """, ( domain, @@ -576,12 +747,120 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N int(item.get("juziseo_status") or 0), ), ) - inserted = bool((cur.fetchone() or [False])[0]) + row = cur.fetchone() or [0, False] + domain_id = int(row[0] or 0) + inserted = bool(row[1]) + if domain_id > 0: + domain_ids.append(domain_id) if inserted: inserted_count += 1 else: updated_count += 1 + target_job_code = f"sync-{source_region}-{source_record_id}" + target_job_remark = ( + f"同步拉取待检测批次 {str(projection.get('batch_code') or '').strip() or source_record_id}," + f"共 {len(domain_ids)} 个域名" + ) + cur.execute( + """ + INSERT INTO detect_jobs (job_code, source, plan_hash, task_mode, step_code, status, remark, created_by) + VALUES (%s, %s, %s, 'domain_pipeline', '', 'pending', %s, %s) + ON CONFLICT (job_code) DO UPDATE SET + source = EXCLUDED.source, + plan_hash = EXCLUDED.plan_hash, + task_mode = EXCLUDED.task_mode, + step_code = EXCLUDED.step_code, + remark = EXCLUDED.remark, + created_by = EXCLUDED.created_by, + status = CASE + WHEN detect_jobs.status IN ('completed', 'failed', 'cancelled') THEN 'pending' + ELSE detect_jobs.status + END, + started_at = CASE + WHEN detect_jobs.status IN ('completed', 'failed', 'cancelled') THEN NULL + ELSE detect_jobs.started_at + END, + finished_at = CASE + WHEN detect_jobs.status IN ('completed', 'failed', 'cancelled') THEN NULL + ELSE detect_jobs.finished_at + END + RETURNING id + """, + ( + target_job_code, + "sync-pull", + projection_hash, + target_job_remark, + "sync-agent", + ), + ) + target_job_id = int((cur.fetchone() or [0])[0] or 0) + + settings_payload = get_settings_payload() + queued_count = 0 + deduplicated_job_items = 0 + skipped_job_items = 0 + if target_job_id > 0: + for domain_id in domain_ids: + domain_snapshot = _load_domain_pipeline_snapshot(cur, int(domain_id)) + if not domain_snapshot: + skipped_job_items += 1 + continue + item_step_code, step_payload = resolve_initial_domain_pipeline_item( + domain_snapshot, + settings_payload=settings_payload, + ) + if not item_step_code or not step_payload: + skipped_job_items += 1 + continue + cur.execute( + """ + INSERT INTO detect_job_items (job_id, domain_id, step_code, status, step_payload_json) + VALUES (%s, %s, %s, 'pending', %s::jsonb) + ON CONFLICT (job_id, domain_id, step_code) DO NOTHING + RETURNING id + """, + ( + target_job_id, + domain_id, + item_step_code, + json.dumps(step_payload, ensure_ascii=False), + ), + ) + inserted_job_item = cur.fetchone() + if inserted_job_item: + queued_count += 1 + else: + deduplicated_job_items += 1 + + if queued_count > 0: + cur.execute( + """ + INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json) + VALUES (%s, %s, %s, %s, %s, %s::jsonb) + """, + ( + target_job_id, + settings.node_code, + "job_created", + "info", + f"同步拉取待检测批次 {target_job_code},共 {queued_count} 个任务项", + json.dumps( + { + "source_region": source_region, + "source_record_id": source_record_id, + "projection_hash": projection_hash, + "batch_code": str(projection.get("batch_code") or "").strip(), + "queued_count": queued_count, + "deduplicated_job_items": deduplicated_job_items, + "skipped_job_items": skipped_job_items, + }, + ensure_ascii=False, + ), + ), + ) + cur.execute( """ INSERT INTO detect_sync_records ( @@ -602,6 +881,11 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N "items_total": len(items), "inserted_count": inserted_count, "updated_count": updated_count, + "target_job_id": target_job_id, + "target_job_code": target_job_code, + "queued_count": queued_count, + "deduplicated_job_items": deduplicated_job_items, + "skipped_job_items": skipped_job_items, "received_at": _format_time(received_at), }, ensure_ascii=False, @@ -617,6 +901,11 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N "items_total": len(items), "inserted_count": inserted_count, "updated_count": updated_count, + "target_job_id": target_job_id, + "target_job_code": target_job_code, + "queued_count": queued_count, + "deduplicated_job_items": deduplicated_job_items, + "skipped_job_items": skipped_job_items, "deduplicated": False, } @@ -742,7 +1031,27 @@ def _extract_detect_result_projection_events( return events -def _resolve_detect_result_target_job_id() -> int: +def _resolve_detect_result_target_job_id(*, projection: dict) -> int: + source_job = projection.get("job") or {} + source_job_code = str(source_job.get("job_code") or "").strip() + + with get_db() as conn: + with conn.cursor() as cur: + if source_job_code: + cur.execute( + """ + SELECT id + FROM detect_jobs + WHERE job_code = %s + ORDER BY id DESC + LIMIT 1 + """, + (source_job_code,), + ) + row = cur.fetchone() + if row: + return int(row[0] or 0) + from app.services.detect_job_service import get_active_detect_job_summary active_job = get_active_detect_job_summary(event_limit=1) or {} @@ -759,6 +1068,163 @@ def _parse_event_created_at(value: str) -> datetime | None: return None +def _extract_event_domain(payload: dict, message: str) -> str: + domain = str(payload.get("domain") or "").strip().lower() + if domain: + return domain + text = str(message or "").strip() + if ":" in text: + candidate = text.rsplit(":", 1)[-1].strip().lower() + if candidate: + return candidate + return "" + + +def _apply_detect_result_event_to_domain(cur, event: dict) -> None: + payload = _decode_json(event.get("payload")) + domain = _extract_event_domain(payload, str(event.get("message") or "")) + if not domain: + return + + event_type = str(event.get("event_type") or "").strip() + created_at = _parse_event_created_at(str(event.get("created_at") or "")) + effective_time = created_at or datetime.now() + + if event_type == "domain_started": + cur.execute( + """ + UPDATE domains + SET detect_status = CASE + WHEN detect_status IN (1, 3) THEN detect_status + ELSE 2 + END, + update_time = CURRENT_TIMESTAMP + WHERE domain = %s + """, + (domain,), + ) + return + + if event_type == "domain_completed": + cur.execute( + """ + UPDATE domains + SET detect_status = 1, + detect_time = COALESCE(detect_time, %s), + update_time = CURRENT_TIMESTAMP + WHERE domain = %s + """, + (effective_time, domain), + ) + return + + if event_type == "domain_blacklisted": + cur.execute( + """ + UPDATE domains + SET detect_status = 3, + update_time = CURRENT_TIMESTAMP + WHERE domain = %s + """, + (domain,), + ) + return + + if event_type == "domain_failed": + cur.execute( + """ + UPDATE domains + SET detect_status = CASE + WHEN detect_status IN (1, 3) THEN detect_status + ELSE 4 + END, + update_time = CURRENT_TIMESTAMP + WHERE domain = %s + """, + (domain,), + ) + return + + +def _apply_detect_result_event_to_job_item(cur, *, target_job_id: int, event: dict) -> int: + if int(target_job_id or 0) <= 0: + return 0 + + payload = _decode_json(event.get("payload")) + domain = _extract_event_domain(payload, str(event.get("message") or "")) + if not domain: + return 0 + + event_type = str(event.get("event_type") or "").strip() + node_code = str(event.get("node_code") or "").strip() + message = str(event.get("message") or "").strip() + + if event_type == "domain_started": + cur.execute( + """ + UPDATE detect_job_items AS item + SET status = CASE + WHEN item.status IN ('completed', 'blacklisted', 'failed') THEN item.status + ELSE 'running' + END, + claimed_by = CASE + WHEN %s <> '' THEN %s + ELSE item.claimed_by + END, + started_at = COALESCE(item.started_at, CURRENT_TIMESTAMP), + updated_at = CURRENT_TIMESTAMP + FROM domains AS d + WHERE item.job_id = %s + AND item.domain_id = d.id + AND d.domain = %s + AND item.status IN ('pending', 'claimed', 'running') + """, + (node_code, node_code, int(target_job_id), domain), + ) + return int(cur.rowcount or 0) + + if event_type not in {"domain_completed", "domain_blacklisted", "domain_failed"}: + return 0 + + final_status = { + "domain_completed": "completed", + "domain_blacklisted": "blacklisted", + "domain_failed": "failed", + }[event_type] + cur.execute( + """ + UPDATE detect_job_items AS item + SET status = %s, + claimed_by = CASE + WHEN %s <> '' THEN %s + ELSE item.claimed_by + END, + finished_at = COALESCE(item.finished_at, CURRENT_TIMESTAMP), + updated_at = CURRENT_TIMESTAMP, + lease_expires_at = NULL, + last_error = CASE + WHEN %s = 'failed' THEN LEFT(%s, 1000) + ELSE item.last_error + END + FROM domains AS d + WHERE item.job_id = %s + AND item.domain_id = d.id + AND d.domain = %s + AND item.status IN ('pending', 'claimed', 'running') + """, + ( + final_status, + node_code, + node_code, + final_status, + message, + int(target_job_id), + domain, + ), + ) + return int(cur.rowcount or 0) + + def _import_detect_result_projection_events( *, source_region: str, @@ -773,12 +1239,11 @@ def _import_detect_result_projection_events( if not events: return {"imported_count": 0, "deduplicated_count": 0, "target_job_id": 0} - target_job_id = _resolve_detect_result_target_job_id() - if target_job_id <= 0: - return {"imported_count": 0, "deduplicated_count": 0, "target_job_id": 0} + target_job_id = _resolve_detect_result_target_job_id(projection=projection) imported_count = 0 deduplicated_count = 0 + updated_job_items = 0 with get_db() as conn: with conn.cursor() as cur: for event in events: @@ -831,12 +1296,27 @@ def _import_detect_result_projection_events( json.dumps(event["payload"], ensure_ascii=False), ), ) + _apply_detect_result_event_to_domain(cur, event) + if target_job_id > 0: + updated_job_items += _apply_detect_result_event_to_job_item( + cur, + target_job_id=target_job_id, + event=event, + ) imported_count += 1 conn.commit() + if target_job_id > 0: + from app.services.detect_job_service import refresh_detect_job_status + + try: + refresh_detect_job_status(target_job_id) + except Exception: + pass return { "imported_count": imported_count, "deduplicated_count": deduplicated_count, "target_job_id": target_job_id, + "updated_job_items": updated_job_items, } @@ -941,6 +1421,16 @@ def ingest_runtime_projection(payload: dict, *, shared_token: str | None = None) def _push_projection_now(sync_type: str, ingest_url: str) -> tuple[bool, str, dict]: + if sync_type == "runtime_projection": + # Regenerate the runtime snapshot before every push so the sync agent + # does not keep replaying a stale projection record while the worker + # thread count / phase is still changing. + from app.services.runtime_status_service import get_runtime_status + + try: + get_runtime_status() + except Exception as exc: + return False, f"刷新 runtime_projection 失败: {exc}", {"action": "push_sync", "sync_type": sync_type} source_record = _load_latest_projection(sync_type) if not source_record: return False, f"当前没有可推送的{sync_type}", {"action": "push_sync", "sync_type": sync_type} @@ -1177,15 +1667,33 @@ def pull_detect_task_batch_now(limit: int | None = None) -> tuple[bool, str, dic if not export_url or not ack_url: return False, "未配置任务拉取目标地址", {"action": "pull_tasks", "pull_state": "misconfigured", "ui_level": "warning", "poll_schedule_seconds": []} - safe_limit = max(1, min(int(limit or settings.sync_batch_size or 200), max(1, int(settings.sync_batch_size or 200)))) + configured_limit = max(5000, int(settings.sync_batch_size or 200)) + requested_limit = int(limit or configured_limit) + safe_limit = max(1, min(requested_limit, max(10000, configured_limit))) + settings_payload = get_settings_payload() + backlog_snapshot = _load_local_detect_backlog_snapshot() + backlog_limits = _build_task_pull_backlog_limits(configured_limit, settings_payload=settings_payload) + should_throttle, throttle_reason = _should_throttle_task_pull(backlog_snapshot, backlog_limits) + if should_throttle: + return True, "本地待处理积压较高,暂停拉取新批次", { + "action": "pull_tasks", + "pull_state": "throttled", + "ui_level": "info", + "poll_schedule_seconds": [1, 3], + "reason": throttle_reason, + **backlog_snapshot, + **backlog_limits, + } + request_url = f"{export_url}?limit={safe_limit}" + export_timeout = max(20, min(90, 15 + safe_limit // 40)) request = urllib.request.Request( request_url, headers={**({"X-Domaincheck-Sync-Token": settings.sync_shared_token} if settings.sync_shared_token else {})}, method="GET", ) try: - with urllib.request.urlopen(request, timeout=20) as response: + with urllib.request.urlopen(request, timeout=export_timeout) as response: raw = response.read().decode("utf-8") data = json.loads(raw) if raw else {} except json.JSONDecodeError as exc: @@ -1280,7 +1788,7 @@ def pull_detect_task_batch_now(limit: int | None = None) -> tuple[bool, str, dic **(ingest_data or {}), } - return True, "待检测任务批次拉取并入库成功", { + result = { "action": "pull_tasks", "pull_state": "success", "ui_level": "success", @@ -1291,3 +1799,27 @@ def pull_detect_task_batch_now(limit: int | None = None) -> tuple[bool, str, dic **(ingest_data or {}), "ack": ack_data, } + queued_count = int(result.get("queued_count", 0) or 0) + if queued_count > 0: + try: + from app.services.worker_control_service import send_worker_command + + start_ok, start_message = send_worker_command( + "start_detection", + payload={ + "source": "sync-pull", + "source_record_id": source_record_id, + "target_job_id": int(result.get("target_job_id", 0) or 0), + "target_job_code": str(result.get("target_job_code") or "").strip(), + }, + ) + result["worker_start_ok"] = bool(start_ok) + result["worker_start_message"] = str(start_message or "").strip() + except Exception as exc: + result["worker_start_ok"] = False + result["worker_start_message"] = f"同步入库后自动唤起 Worker 失败: {exc}" + result["ui_level"] = "warning" + result["pull_state"] = "worker_start_warning" + return True, "待检测任务批次拉取并入库成功;但自动唤起 Worker 失败", result + + return True, "待检测任务批次拉取并入库成功", result diff --git a/domain-api/app/services/sync_record_service.py b/domain-api/app/services/sync_record_service.py index 836ca7d..7b7910b 100644 --- a/domain-api/app/services/sync_record_service.py +++ b/domain-api/app/services/sync_record_service.py @@ -6,7 +6,7 @@ import socket from datetime import datetime, timedelta from app.core.config import settings -from app.core.db import get_db +from app.core.db import db_read_retry, get_db def _format_time(value: datetime | None) -> str: @@ -45,28 +45,94 @@ _DETECT_RESULT_EVENT_TYPES = { "domain_blacklisted", } +_TERMINAL_DETECT_RESULT_EVENT_TYPES = { + "domain_completed", + "domain_failed", + "domain_blacklisted", +} + def _collect_recent_domain_events(active_job: dict, limit: int = 30) -> list[dict]: - events = list(active_job.get("current_cycle_events") or active_job.get("recent_events") or []) + safe_limit = max(1, int(limit or 30)) + seen: set[tuple[str, str, str, str]] = set() normalized: list[dict] = [] - for event in reversed(events): - event_type = str(event.get("event_type") or "").strip() + + def _append_event(raw_event: dict) -> None: + event_type = str(raw_event.get("event_type") or "").strip() if event_type not in _DETECT_RESULT_EVENT_TYPES: - continue - payload = _decode_json(event.get("payload")) - normalized.append( - { - "node_code": str(event.get("node_code") or "").strip(), - "event_type": event_type, - "level": str(event.get("level") or "info").strip() or "info", - "message": str(event.get("message") or "").strip(), - "created_at": str(event.get("created_at") or "").strip(), - "payload": payload, - } + return + normalized_event = { + "node_code": str(raw_event.get("node_code") or "").strip(), + "event_type": event_type, + "level": str(raw_event.get("level") or "info").strip() or "info", + "message": str(raw_event.get("message") or "").strip(), + "created_at": str(raw_event.get("created_at") or "").strip(), + "payload": _decode_json(raw_event.get("payload")), + } + event_key = ( + normalized_event["node_code"], + normalized_event["event_type"], + normalized_event["message"], + normalized_event["created_at"], ) - if limit <= 0: - return normalized - return normalized[-int(limit):] + if event_key in seen: + return + seen.add(event_key) + normalized.append(normalized_event) + + # Keep a small slice of the current-cycle `domain_started` events so the + # remote log / live activity view still reflects the node's latest work. + for event in reversed(list(active_job.get("current_cycle_events") or active_job.get("recent_events") or [])): + if str(event.get("event_type") or "").strip() != "domain_started": + continue + _append_event(event) + if len(normalized) >= min(10, max(1, safe_limit // 3)): + break + + # Always pull the most recent terminal result events from the full job + # history. Otherwise a flood of newer `domain_started` events can hide + # terminal completions, and overseas will never advance completed counts. + job_id = int(active_job.get("job_id") or 0) + if job_id > 0: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT node_code, event_type, level, message, payload_json, created_at + FROM detect_run_events + WHERE job_id = %s + AND event_type IN ('domain_completed', 'domain_failed', 'domain_blacklisted') + ORDER BY created_at DESC, id DESC + LIMIT %s + """, + (job_id, max(safe_limit * 4, 60)), + ) + rows = cur.fetchall() + for row in reversed(rows): + _append_event( + { + "node_code": row[0] or "", + "event_type": row[1] or "", + "level": row[2] or "info", + "message": row[3] or "", + "payload": _decode_json(row[4]), + "created_at": _format_time(row[5]), + } + ) + else: + for event in reversed(list(active_job.get("current_cycle_events") or active_job.get("recent_events") or [])): + if str(event.get("event_type") or "").strip() in _TERMINAL_DETECT_RESULT_EVENT_TYPES: + _append_event(event) + + normalized.sort( + key=lambda item: ( + str(item.get("created_at") or ""), + str(item.get("node_code") or ""), + str(item.get("event_type") or ""), + str(item.get("message") or ""), + ) + ) + return normalized[-safe_limit:] def _build_detect_result_batch_digest(batch: dict | None) -> dict: @@ -154,6 +220,7 @@ def _should_append_runtime_projection(previous_payload: dict, current_projection return now - previous_created_at >= timedelta(seconds=45) +@db_read_retry() def list_sync_records(limit: int = 20) -> list[dict]: safe_limit = max(1, min(int(limit or 20), 200)) with get_db() as conn: @@ -221,6 +288,7 @@ def _latest_sync_record_by_source( } +@db_read_retry() def get_detect_result_sync_batches(limit: int = 5) -> dict: safe_limit = max(1, min(int(limit or 5), 20)) source_region = _normalize_region(settings.sync_source_region, settings.node_region) @@ -364,6 +432,7 @@ def get_detect_result_sync_batches(limit: int = 5) -> dict: } +@db_read_retry() def get_sync_summary(record_limit: int = 10) -> dict: source_region = _normalize_region(settings.sync_source_region, settings.node_region) target_region = _normalize_region(settings.sync_target_region, "overseas") @@ -465,56 +534,43 @@ def append_sync_record( return record_id -def append_runtime_projection_if_changed( +def _local_node_expected_to_execute_worker() -> bool: + node_role = str(settings.node_role or "").strip() + node_region = str(settings.node_region or "").strip() + return node_role == "worker" or (node_region == "mainland" and node_role == "control") + + +def _build_runtime_projection_payload( *, detect: dict, cluster: dict, - source_region: str | None = None, - target_region: str | None = None, -) -> int | None: - normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region)) - normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas")) + source_region: str, + target_region: str, +) -> dict: active_job = detect.get("active_job") or {} + local_worker_expected = _local_node_expected_to_execute_worker() local_participating = False for node in list(cluster.get("nodes") or []): if str(node.get("node_code") or "").strip() != settings.node_code: continue - local_participating = bool(node.get("detect_participating", False) or node.get("current_load", 0)) + if local_worker_expected: + local_participating = bool(node.get("detect_participating", False) or node.get("current_load", 0)) break local_job_bucket = {} - for item in list(active_job.get("node_stats") or []): - if str(item.get("node_code") or "").strip() != settings.node_code: - continue - local_job_bucket = item - break - if not local_participating: + if local_worker_expected: + for item in list(active_job.get("node_stats") or []): + if str(item.get("node_code") or "").strip() != settings.node_code: + continue + local_job_bucket = item + break + if local_worker_expected and not local_participating: local_participating = bool( int(local_job_bucket.get("items_running", 0) or 0) > 0 or int(local_job_bucket.get("items_claimed", 0) or 0) > 0 ) - projection = { - "node": { - "node_code": settings.node_code, - "region": settings.node_region, - "role": settings.node_role, - "hostname": socket.gethostname(), - "ip": _resolve_local_ip(), - }, - "worker_online": bool(detect.get("worker_online", False)), - "detect_participating": local_participating, - "worker_mode": detect.get("worker_mode", ""), - "phase_label": detect.get("phase_label", ""), - "phase_detail": detect.get("phase_detail", ""), - "proxy_runtime_label": detect.get("proxy_runtime_label", ""), - "proxy_runtime_reason": detect.get("proxy_runtime_reason", ""), - "progress": { - "pending": int((detect.get("progress") or {}).get("pending", 0) or 0), - "running": int((detect.get("progress") or {}).get("running", 0) or 0), - "completed": int((detect.get("progress") or {}).get("completed", 0) or 0), - "blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0), - "failed": int((detect.get("progress") or {}).get("failed", 0) or 0), - }, - "active_job": { + + projection_active_job = ( + { "job_id": active_job.get("job_id"), "job_code": active_job.get("job_code", ""), "status": active_job.get("status", ""), @@ -525,7 +581,58 @@ def append_runtime_projection_if_changed( "items_running": active_job.get("items_running", 0), "items_failed": active_job.get("items_failed", 0), "node_stats": list(active_job.get("node_stats") or []), + } + if local_worker_expected + else { + "job_id": None, + "job_code": "", + "status": "", + "progress_percent": 0, + "items_total": 0, + "items_terminal": 0, + "items_pending": 0, + "items_running": 0, + "items_failed": 0, + "node_stats": [], + } + ) + progress_payload = ( + { + "pending": int((detect.get("progress") or {}).get("pending", 0) or 0), + "running": int((detect.get("progress") or {}).get("running", 0) or 0), + "completed": int((detect.get("progress") or {}).get("completed", 0) or 0), + "blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0), + "failed": int((detect.get("progress") or {}).get("failed", 0) or 0), + } + if local_worker_expected + else { + "pending": 0, + "running": 0, + "completed": 0, + "blacklisted": 0, + "failed": 0, + } + ) + projection = { + "node": { + "node_code": settings.node_code, + "region": settings.node_region, + "role": settings.node_role, + "hostname": socket.gethostname(), + "ip": _resolve_local_ip(), }, + "worker_online": bool(detect.get("worker_online", False)) if local_worker_expected else False, + "detect_participating": local_participating if local_worker_expected else False, + "worker_mode": detect.get("worker_mode", ""), + "active_thread_count": int(detect.get("active_thread_count", 0) or 0) if local_worker_expected else 0, + "max_thread_count": int(detect.get("max_thread_count", 0) or 0) if local_worker_expected else 0, + "phase_label": detect.get("phase_label", ""), + "phase_detail": detect.get("phase_detail", ""), + "proxy_runtime_label": detect.get("proxy_runtime_label", ""), + "proxy_runtime_reason": detect.get("proxy_runtime_reason", ""), + "progress": progress_payload, + "backlog": dict(detect.get("backlog") or {}) if local_worker_expected else {}, + "active_job": projection_active_job, "cluster_summary": { "nodes_total": int(cluster.get("nodes_total", 0) or 0), "online_worker_nodes": int((cluster.get("summary") or {}).get("online_worker_nodes", 0) or 0), @@ -544,13 +651,32 @@ def append_runtime_projection_if_changed( for item in (detect.get("dependency_alerts") or [])[:3] ], } - payload = { + return { "projection": projection, "projection_hash": hashlib.sha1( json.dumps(projection, ensure_ascii=False, sort_keys=True).encode("utf-8") ).hexdigest(), + "source_region": source_region, + "target_region": target_region, } + +def append_runtime_projection_if_changed( + *, + detect: dict, + cluster: dict, + source_region: str | None = None, + target_region: str | None = None, +) -> int | None: + normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region)) + normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas")) + payload = _build_runtime_projection_payload( + detect=detect, + cluster=cluster, + source_region=normalized_source_region, + target_region=normalized_target_region, + ) + with get_db() as conn: with conn.cursor() as cur: cur.execute( @@ -570,7 +696,7 @@ def append_runtime_projection_if_changed( latest_created_at = latest[1] if latest else None if latest_payload.get("projection_hash") == payload["projection_hash"]: return None - if not _should_append_runtime_projection(latest_payload, projection, latest_created_at): + if not _should_append_runtime_projection(latest_payload, payload["projection"], latest_created_at): return None cur.execute( """ @@ -650,6 +776,7 @@ def append_detect_result_projection_if_changed( with get_db() as conn: with conn.cursor() as cur: + current_job_id = int((projection.get("job") or {}).get("job_id") or 0) cur.execute( """ SELECT payload_json, created_at @@ -657,10 +784,19 @@ def append_detect_result_projection_if_changed( WHERE sync_type = 'detect_result_projection' AND source_region = %s AND target_region = %s + AND ( + %s <= 0 + OR (payload_json->'projection'->'job'->>'job_id') = %s + ) ORDER BY created_at DESC, id DESC LIMIT 1 """, - (normalized_source_region, normalized_target_region), + ( + normalized_source_region, + normalized_target_region, + current_job_id, + str(current_job_id), + ), ) latest = cur.fetchone() latest_payload = _decode_json(latest[0]) if latest else {} diff --git a/domain-api/app/services/worker_control_service.py b/domain-api/app/services/worker_control_service.py index d4e78de..fe388b7 100644 --- a/domain-api/app/services/worker_control_service.py +++ b/domain-api/app/services/worker_control_service.py @@ -60,7 +60,12 @@ def normalize_systemctl_error(raw_message: str, *, service_name: str = "") -> st return f"{target} 控制失败,未返回可用错误信息" lowered = message.lower() - if "sudo: a password is required" in lowered or "authentication is required" in lowered: + if ( + "sudo: a password is required" in lowered + or "authentication is required" in lowered + or "interactive authentication required" in lowered + or "authorization not available" in lowered + ): target = normalized_service_name or "systemd 服务" return f"{target} 控制失败:当前运行用户没有免密 systemctl 权限,请为 API 进程授予对应 sudo/systemd 权限" if "unit " in lowered and " could not be found" in lowered: diff --git a/domain-api/app/sync_agent.py b/domain-api/app/sync_agent.py index add2a66..95e6396 100644 --- a/domain-api/app/sync_agent.py +++ b/domain-api/app/sync_agent.py @@ -5,13 +5,34 @@ import time from app.core.config import settings from app.services.debug_event_service import push_debug_event -from app.services.detect_job_service import get_active_detect_job_summary, get_detect_queue_health, list_recent_detect_run_events +from app.services.detect_job_service import ( + get_active_detect_job_summary, + get_detect_queue_health, + get_latest_detect_job_summary, + get_latest_unprojected_detect_job_summary, + list_recent_detect_run_events, + process_detect_pipeline_now, +) from app.services.sync_record_service import append_detect_result_projection_if_changed -from app.services.sync_push_service import pull_detect_task_batch_now, push_runtime_projection_now +from app.services.sync_push_service import ( + _load_local_detect_backlog_snapshot, + pull_detect_task_batch_now, + push_runtime_projection_now, +) logger = logging.getLogger("domaincheck.sync_agent") +_IDLE_SYNC_KEYWORDS = ( + "当前没有可推送", + "当前没有需要立即推送", + "已全部同步完成", + "无需重复发送", + "进行中", + "等待下个重试窗口", + "暂停拉取", +) + def _append_detect_result_projection_snapshot(active_job: dict) -> None: if not active_job: @@ -32,6 +53,159 @@ def _append_detect_result_projection_snapshot(active_job: dict) -> None: ) +def _is_idle_sync_message(message: str) -> bool: + normalized = str(message or "").strip() + return any(keyword in normalized for keyword in _IDLE_SYNC_KEYWORDS) + + +def _filter_runtime_events_for_job(events: list[dict], *, job_code: str = "", job_id: int = 0, limit: int = 8) -> list[dict]: + target_job_code = str(job_code or "").strip() + target_job_id = int(job_id or 0) + safe_limit = max(1, min(int(limit or 8), 50)) + filtered: list[dict] = [] + for raw_event in list(events or []): + if not isinstance(raw_event, dict): + continue + payload = raw_event.get("payload") if isinstance(raw_event.get("payload"), dict) else {} + event_job_code = str(payload.get("job_code") or "").strip() + event_job_id = int(raw_event.get("job_id") or 0) + if target_job_code and event_job_code != target_job_code and (target_job_id <= 0 or event_job_id != target_job_id): + continue + filtered.append(raw_event) + if len(filtered) >= safe_limit: + break + return filtered + + +def _build_aligned_queue_health_snapshot(active_job: dict, queue_health: dict | None) -> dict: + snapshot = dict(queue_health or {}) + if not active_job: + return snapshot + + active_job_code = str(active_job.get("runtime_job_code") or active_job.get("job_code") or "").strip() + queue_job = dict(snapshot.get("job") or {}) + queue_job_code = str(queue_job.get("runtime_job_code") or queue_job.get("job_code") or "").strip() + + if active_job_code and queue_job_code and active_job_code == queue_job_code: + return snapshot + + active_job_items_total = int(active_job.get("items_total", 0) or 0) + active_job_pending = int(active_job.get("items_pending", 0) or 0) + active_job_claimed = int(active_job.get("items_claimed", 0) or 0) + active_job_running = int(active_job.get("items_running", 0) or 0) + active_job_completed = int(active_job.get("items_completed", 0) or 0) + active_job_blacklisted = int(active_job.get("items_blacklisted", 0) or 0) + active_job_failed = int(active_job.get("items_failed", 0) or 0) + active_job_terminal = int( + active_job.get("items_terminal", active_job_completed + active_job_blacklisted + active_job_failed) or 0 + ) + display_claimed = int(active_job.get("display_items_claimed", active_job_claimed) or active_job_claimed) + display_running = int(active_job.get("display_items_running", active_job_running) or active_job_running) + + node_entries: list[dict] = [] + for node in list(active_job.get("node_stats") or []): + node_entries.append( + { + "node_code": str(node.get("node_code") or "").strip(), + "items_total": int(node.get("items_total", 0) or 0), + "items_pending": int(node.get("items_pending", 0) or 0), + "items_claimed": int(node.get("items_claimed", 0) or 0), + "items_running": int(node.get("items_running", 0) or 0), + "items_completed": int(node.get("items_completed", 0) or 0), + "items_blacklisted": int(node.get("items_blacklisted", 0) or 0), + "items_failed": int(node.get("items_failed", 0) or 0), + "processed_recent": int(node.get("processed_recent", 0) or 0), + "processed_per_minute": float(node.get("processed_per_minute", 0) or 0), + "completed_recent": int(node.get("completed_recent", 0) or 0), + "blacklisted_recent": int(node.get("blacklisted_recent", 0) or 0), + "failed_recent": int(node.get("failed_recent", 0) or 0), + "metrics_source": str(node.get("metrics_source") or "runtime"), + } + ) + + assigned_total = sum(int(item.get("items_total", 0) or 0) for item in node_entries) + unassigned_total = max(0, active_job_items_total - assigned_total) + if unassigned_total > 0: + node_entries.append( + { + "node_code": "unassigned", + "items_total": unassigned_total, + "items_pending": active_job_pending, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0.0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + "metrics_source": "central_queue", + } + ) + + snapshot["job"] = { + "job_id": active_job.get("job_id"), + "job_code": str(active_job.get("job_code") or "").strip(), + "runtime_job_code": active_job_code, + "status": str(active_job.get("status") or "").strip(), + "progress_percent": float(active_job.get("progress_percent", 0) or 0), + } + snapshot["queue"] = { + **dict(snapshot.get("queue") or {}), + "items_total": active_job_items_total, + "pending": active_job_pending, + "claimed": active_job_claimed, + "running": active_job_running, + "display_claimed": display_claimed, + "display_running": display_running, + "completed": active_job_completed, + "blacklisted": active_job_blacklisted, + "failed": active_job_failed, + "terminal": active_job_terminal, + "terminal_percent": round((active_job_terminal / active_job_items_total) * 100, 2) if active_job_items_total else 0.0, + } + snapshot["nodes"] = node_entries + return snapshot + + +def _select_projection_job_snapshot() -> dict | None: + active_job = get_active_detect_job_summary(event_limit=10) + if active_job: + return active_job + return get_latest_detect_job_summary( + event_limit=10, + statuses=("completed", "partial_failed", "failed"), + recent_minutes=20, + ) + + +def _select_projection_job_snapshots() -> list[dict]: + snapshots: list[dict] = [] + seen_job_ids: set[int] = set() + + active_job = get_active_detect_job_summary(event_limit=10) + if active_job: + active_job_id = int(active_job.get("job_id") or 0) + if active_job_id > 0 and active_job_id not in seen_job_ids: + snapshots.append(active_job) + seen_job_ids.add(active_job_id) + + latest_finished_job = get_latest_unprojected_detect_job_summary( + event_limit=10, + statuses=("completed", "partial_failed", "failed"), + recent_minutes=180, + ) + if latest_finished_job: + latest_finished_job_id = int(latest_finished_job.get("job_id") or 0) + if latest_finished_job_id > 0 and latest_finished_job_id not in seen_job_ids: + snapshots.append(latest_finished_job) + seen_job_ids.add(latest_finished_job_id) + + return snapshots + + def _emit_structured_tick( *, base_event_type: str, @@ -42,7 +216,13 @@ def _emit_structured_tick( payload = {"ok": ok, "data": data or {}} event_type = f"{base_event_type}_failed" level = "warning" - if ok: + if isinstance(data, dict) and str(data.get("pull_state") or "").strip() == "throttled": + event_type = f"{base_event_type}_idle" + level = "info" + elif not ok and _is_idle_sync_message(message): + event_type = f"{base_event_type}_idle" + level = "info" + elif ok: event_type = f"{base_event_type}_success" level = "info" if "但远端确认失败" in str(message or ""): @@ -70,7 +250,10 @@ def _emit_sync_result_breakdown(data: dict | None) -> None: result_data = item.get("data") or {} event_type = f"{sync_type}_sync_failed" level = "warning" - if ok: + if not ok and _is_idle_sync_message(message): + event_type = f"{sync_type}_sync_idle" + level = "info" + elif ok: event_type = f"{sync_type}_sync_success" level = "info" if isinstance(result_data, dict) and result_data.get("success_count") is not None: @@ -95,12 +278,31 @@ def _emit_sync_result_breakdown(data: dict | None) -> None: ) +def _run_pipeline_stage_processor() -> tuple[bool, str, dict]: + process_limit = max(500, min(int(settings.sync_pipeline_process_limit or 5000), 5000)) + ok, message, data = process_detect_pipeline_now(limit=process_limit) + push_debug_event( + service="sync-agent", + event_type="pipeline_tick_success" if ok else "pipeline_tick_failed", + level="info" if ok else "warning", + message=message, + payload={ + "ok": ok, + "limit": process_limit, + "data": data or {}, + }, + ) + return ok, message, data + + def main() -> None: logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", ) - interval = max(10, int(settings.sync_poll_interval_seconds or 30)) + # Old env files still ship SYNC_POLL_INTERVAL_SECONDS=30. Cap the interval + # so controller pull/pipeline ticks cannot be throttled into starvation. + interval = max(2, min(int(settings.sync_poll_interval_seconds or 2), 5)) logger.info( "sync agent started: node=%s source=%s target=%s interval=%ss enabled=%s", settings.node_code, @@ -111,9 +313,16 @@ def main() -> None: ) while True: try: + pipeline_ok, pipeline_message, pipeline_data = _run_pipeline_stage_processor() + logger.info( + "pipeline tick: ok=%s message=%s data=%s", + pipeline_ok, + pipeline_message, + pipeline_data, + ) active_job = get_active_detect_job_summary(event_limit=10) - if active_job: - _append_detect_result_projection_snapshot(active_job) + for projection_job in _select_projection_job_snapshots(): + _append_detect_result_projection_snapshot(projection_job) ok, message, data = push_runtime_projection_now() logger.info("sync tick: ok=%s message=%s data=%s", ok, message, data) push_debug_event( @@ -136,8 +345,16 @@ def main() -> None: ) _emit_structured_tick(base_event_type="task_pull", ok=pull_ok, message=pull_message, data=pull_data) if active_job: - queue_health = get_detect_queue_health(window_minutes=15) - recent_events = list_recent_detect_run_events(limit=8) + queue_health = _build_aligned_queue_health_snapshot( + active_job, + get_detect_queue_health(window_minutes=15), + ) + recent_events = _filter_runtime_events_for_job( + list_recent_detect_run_events(limit=24), + job_code=str(active_job.get("runtime_job_code") or active_job.get("job_code") or "").strip(), + job_id=int(active_job.get("job_id", 0) or 0), + limit=8, + ) push_debug_event( service="detect-runtime", event_type="active_job_snapshot", @@ -158,6 +375,7 @@ def main() -> None: "node_stats": list(active_job.get("node_stats") or []), }, "queue_health": queue_health, + "backlog": _load_local_detect_backlog_snapshot(), "recent_events": recent_events, }, ) diff --git a/domain-api/deploy/multi-region/fix_mainland_release_base.sh b/domain-api/deploy/multi-region/fix_mainland_release_base.sh new file mode 100755 index 0000000..dbd8235 --- /dev/null +++ b/domain-api/deploy/multi-region/fix_mainland_release_base.sh @@ -0,0 +1,112 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROLE="${1:-}" +BASE_DIR="${2:-/opt/domaincheck}" +WWW_USER="${WWW_USER:-www}" +WWW_GROUP="${WWW_GROUP:-www}" + +usage() { + cat <<'EOF' +usage: + bash domain-api/deploy/multi-region/fix_mainland_release_base.sh [base_dir] + +examples: + bash domain-api/deploy/multi-region/fix_mainland_release_base.sh worker + bash domain-api/deploy/multi-region/fix_mainland_release_base.sh control +EOF +} + +if [[ "${ROLE}" != "worker" && "${ROLE}" != "control" ]]; then + usage >&2 + exit 1 +fi + +if [[ "$(id -u)" != "0" ]]; then + echo "please run as root" >&2 + exit 1 +fi + +require_path() { + local path="$1" + if [[ ! -e "${path}" ]]; then + echo "required path missing: ${path}" >&2 + exit 1 + fi +} + +install_dropin() { + local service_name="$1" + local content="$2" + local dropin_dir="/etc/systemd/system/${service_name}.d" + local dropin_file="${dropin_dir}/current-path.conf" + mkdir -p "${dropin_dir}" + printf '%s\n' "${content}" > "${dropin_file}" + echo "installed ${dropin_file}" +} + +echo "[1/6] validate existing runtime" +require_path "${BASE_DIR}" +require_path "${BASE_DIR}/domainCheck" +require_path "${BASE_DIR}/domainCheck/.venv/bin/python" +require_path "${BASE_DIR}/domainCheck/detect_worker.py" +require_path "${BASE_DIR}/domain-api" + +echo "[2/6] make release root writable for ${WWW_USER}:${WWW_GROUP}" +chgrp "${WWW_GROUP}" "${BASE_DIR}" +chmod 2775 "${BASE_DIR}" +mkdir -p "${BASE_DIR}/downloads" "${BASE_DIR}/releases" +chown -R "${WWW_USER}:${WWW_GROUP}" "${BASE_DIR}/downloads" "${BASE_DIR}/releases" + +echo "[3/6] ensure current link exists" +if [[ -L "${BASE_DIR}/current" ]]; then + echo "keep existing symlink: ${BASE_DIR}/current -> $(readlink -f "${BASE_DIR}/current" || true)" +elif [[ -e "${BASE_DIR}/current" ]]; then + echo "path exists but is not a symlink: ${BASE_DIR}/current" >&2 + exit 1 +else + ln -s "${BASE_DIR}" "${BASE_DIR}/current" + echo "created symlink: ${BASE_DIR}/current -> ${BASE_DIR}" +fi + +echo "[4/6] install systemd drop-ins" +install_dropin "domaincheck-worker" "[Service] +WorkingDirectory=${BASE_DIR}/current/domainCheck +ExecStart= +ExecStart=${BASE_DIR}/domainCheck/.venv/bin/python ${BASE_DIR}/current/domainCheck/detect_worker.py" + +install_dropin "domaincheck-node-agent" "[Service] +User=root +Group=root +WorkingDirectory=${BASE_DIR}/current/domain-api +ExecStart= +ExecStart=${BASE_DIR}/domainCheck/.venv/bin/python -m app.node_agent" + +if [[ "${ROLE}" == "control" ]]; then + install_dropin "domaincheck-api" "[Service] +WorkingDirectory=${BASE_DIR}/current/domain-api +ExecStart= +ExecStart=${BASE_DIR}/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100" + + install_dropin "domaincheck-sync-agent" "[Service] +WorkingDirectory=${BASE_DIR}/current/domain-api +ExecStart= +ExecStart=${BASE_DIR}/domainCheck/.venv/bin/python -m app.sync_agent" +fi + +echo "[5/6] reload systemd and restart node agent" +systemctl daemon-reload +systemctl restart domaincheck-node-agent + +echo "[6/6] summary" +echo "role=${ROLE}" +echo "base_dir=${BASE_DIR}" +echo "current_target=$(readlink -f "${BASE_DIR}/current" || true)" +echo +echo "recommended next checks:" +echo " systemctl status domaincheck-node-agent --no-pager -l" +echo " systemctl cat domaincheck-worker" +if [[ "${ROLE}" == "control" ]]; then + echo " systemctl cat domaincheck-api" + echo " systemctl cat domaincheck-sync-agent" +fi diff --git a/domain-api/deploy/multi-region/install_mainland_controller_quick.sh b/domain-api/deploy/multi-region/install_mainland_controller_quick.sh index 797abba..b970be4 100755 --- a/domain-api/deploy/multi-region/install_mainland_controller_quick.sh +++ b/domain-api/deploy/multi-region/install_mainland_controller_quick.sh @@ -125,8 +125,8 @@ SYNC_SOURCE_REGION=mainland SYNC_TARGET_REGION=overseas SYNC_TARGET_API_BASE_URL=${TARGET_API_BASE_URL} SYNC_SHARED_TOKEN=${SYNC_SHARED_TOKEN} -SYNC_BATCH_SIZE=200 -SYNC_POLL_INTERVAL_SECONDS=30 +SYNC_BATCH_SIZE=5000 +SYNC_POLL_INTERVAL_SECONDS=2 EOF systemctl daemon-reload diff --git a/domain-api/deploy/multi-region/install_mainland_worker_quick.sh b/domain-api/deploy/multi-region/install_mainland_worker_quick.sh index 8312cde..cd5b010 100755 --- a/domain-api/deploy/multi-region/install_mainland_worker_quick.sh +++ b/domain-api/deploy/multi-region/install_mainland_worker_quick.sh @@ -126,8 +126,8 @@ SYNC_SOURCE_REGION=mainland SYNC_TARGET_REGION=overseas SYNC_TARGET_API_BASE_URL=${TARGET_API_BASE_URL} SYNC_SHARED_TOKEN=${SYNC_SHARED_TOKEN} -SYNC_BATCH_SIZE=200 -SYNC_POLL_INTERVAL_SECONDS=30 +SYNC_BATCH_SIZE=5000 +SYNC_POLL_INTERVAL_SECONDS=2 EOF systemctl daemon-reload diff --git a/domain-api/deploy/multi-region/night_shift_until_20260420_1200.sh b/domain-api/deploy/multi-region/night_shift_until_20260420_1200.sh new file mode 100755 index 0000000..f1b998c --- /dev/null +++ b/domain-api/deploy/multi-region/night_shift_until_20260420_1200.sh @@ -0,0 +1,303 @@ +#!/usr/bin/env bash +set -u -o pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)" +BASE_URL="${1:-http://127.0.0.1:8100}" +DEADLINE_TEXT="${2:-2026-04-20 12:00:00 +0800}" +RUNTIME_DIR="${REPO_ROOT}/docs/ops_center_runtime/night_runs" +START_TS="$(date +%Y%m%d_%H%M%S)" +RUN_ID="night_run_${START_TS}" +LOG_FILE="${RUNTIME_DIR}/${RUN_ID}.log" +STATE_DIR="${RUNTIME_DIR}/${RUN_ID}" +PID_FILE="${RUNTIME_DIR}/${RUN_ID}.pid" +REPORT_FILE="${RUNTIME_DIR}/${RUN_ID}_report.md" +SUMMARY_JSON="${RUNTIME_DIR}/${RUN_ID}_summary.json" + +mkdir -p "${RUNTIME_DIR}" "${STATE_DIR}" +echo "$$" > "${PID_FILE}" + +DEADLINE_EPOCH="$(python3 - <<'PY' "${DEADLINE_TEXT}" +import datetime +import sys + +text = sys.argv[1] +dt = datetime.datetime.strptime(text, "%Y-%m-%d %H:%M:%S %z") +print(int(dt.timestamp())) +PY +)" + +cycles=0 +log_sync_recover_runs=0 +inspection_runs=0 +inspection_churn_runs=0 +quick_rechecks=0 +stop_reason="deadline_reached" +last_go_live_status="" +last_publish_ready="" +last_log_sync_state="" +last_issue_total="" +last_problem_runs_total="" +last_launchpad_status="" +last_launchpad_action="" +last_problem_run_code="" + +log() { + printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S %z')" "$*" | tee -a "${LOG_FILE}" +} + +run_cmd() { + log "RUN $*" + if "$@" >> "${LOG_FILE}" 2>&1; then + log "OK $*" + return 0 + fi + log "ERR $*" + return 1 +} + +fetch_json() { + local url="$1" + local target="$2" + if curl -m 20 -s "${url}" > "${target}.tmp"; then + mv "${target}.tmp" "${target}" + return 0 + fi + rm -f "${target}.tmp" + return 1 +} + +snapshot_state() { + local cycle_dir="$1" + mkdir -p "${cycle_dir}" + + fetch_json "${BASE_URL}/api/v1/ops/go-live-summary" "${cycle_dir}/go_live.json" || true + fetch_json "${BASE_URL}/api/v1/ops/stack-diagnosis" "${cycle_dir}/stack.json" || true + fetch_json "${BASE_URL}/api/v1/ops/releases/launchpad" "${cycle_dir}/launchpad.json" || true + fetch_json "${BASE_URL}/api/v1/ops/playbook-runs" "${cycle_dir}/playbook_runs.json" || true + fetch_json "${BASE_URL}/api/v1/ops/nodes/overseas-control-01/scene-log?limit=120&mode=full" "${cycle_dir}/scene_overseas_control_01.json" || true + + python3 - <<'PY' \ + "${cycle_dir}/go_live.json" \ + "${cycle_dir}/stack.json" \ + "${cycle_dir}/launchpad.json" \ + "${cycle_dir}/playbook_runs.json" \ + "${cycle_dir}/scene_overseas_control_01.json" \ + "${SUMMARY_JSON}" \ + "${cycles}" \ + > "${cycle_dir}/summary.env" +import json +import os +import sys + +go_path, stack_path, launch_path, runs_path, scene_path, summary_path, cycles = sys.argv[1:8] + +def load_json(path): + try: + with open(path, "r", encoding="utf-8") as fh: + return json.load(fh) + except Exception: + return {} + +go_data = load_json(go_path).get("data", {}) +stack_data = load_json(stack_path).get("data", {}).get("diagnosis", {}) +launch_data = load_json(launch_path).get("data", {}) +runs_data = load_json(runs_path).get("data", {}) +scene_data = load_json(scene_path).get("data", {}) + +issues = stack_data.get("issues") or [] +issue_codes = [str(item.get("code") or "") for item in issues if item.get("code")] +problem_runs = runs_data.get("problem_runs") or [] +problem_run_code = "" +if problem_runs: + problem_run_code = str(problem_runs[0].get("run_code") or "") + +summary = { + "cycles": int(cycles), + "go_live_status": str(go_data.get("go_live_status") or ""), + "publish_ready": bool(go_data.get("publish_ready")), + "log_sync_state": str(go_data.get("log_sync_state") or ""), + "log_sync_missing_node_codes": go_data.get("log_sync_missing_node_codes") or [], + "stack_status": str(stack_data.get("stack_status") or ""), + "issue_total": int(stack_data.get("issue_total") or 0), + "blocking_issue_total": int(stack_data.get("blocking_issue_total") or 0), + "issue_codes": issue_codes, + "launchpad_status": str((launch_data.get("launchpad_status") or {}).get("status") or ""), + "launchpad_recommended_action": str((launch_data.get("launchpad_status") or {}).get("recommended_action_code") or ""), + "problem_runs_total": int(runs_data.get("problem_runs_total") or 0), + "problem_run_code": problem_run_code, + "scene_status": str(scene_data.get("status") or ""), + "scene_line_count": int((scene_data.get("source_summary") or {}).get("line_count") or 0), + "generated_at": str(go_data.get("generated_at") or stack_data.get("generated_at") or ""), +} + +with open(summary_path, "w", encoding="utf-8") as fh: + json.dump(summary, fh, ensure_ascii=False, indent=2) + +def emit(key, value): + if isinstance(value, bool): + value = "true" if value else "false" + elif isinstance(value, list): + value = ",".join(str(item) for item in value) + else: + value = str(value) + print(f'{key}="{value}"') + +for key, value in summary.items(): + emit(key.upper(), value) +PY + + # shellcheck disable=SC1090 + source "${cycle_dir}/summary.env" + last_go_live_status="${GO_LIVE_STATUS}" + last_publish_ready="${PUBLISH_READY}" + last_log_sync_state="${LOG_SYNC_STATE}" + last_issue_total="${ISSUE_TOTAL}" + last_problem_runs_total="${PROBLEM_RUNS_TOTAL}" + last_launchpad_status="${LAUNCHPAD_STATUS}" + last_launchpad_action="${LAUNCHPAD_RECOMMENDED_ACTION}" + last_problem_run_code="${PROBLEM_RUN_CODE}" +} + +write_report() { + python3 - <<'PY' \ + "${REPORT_FILE}" \ + "${RUN_ID}" \ + "${BASE_URL}" \ + "${DEADLINE_TEXT}" \ + "${stop_reason}" \ + "${cycles}" \ + "${log_sync_recover_runs}" \ + "${inspection_runs}" \ + "${inspection_churn_runs}" \ + "${quick_rechecks}" \ + "${last_go_live_status}" \ + "${last_publish_ready}" \ + "${last_log_sync_state}" \ + "${last_issue_total}" \ + "${last_problem_runs_total}" \ + "${last_launchpad_status}" \ + "${last_launchpad_action}" \ + "${last_problem_run_code}" \ + "${SUMMARY_JSON}" +import json +import sys + +( + report_path, + run_id, + base_url, + deadline_text, + stop_reason, + cycles, + log_sync_recover_runs, + inspection_runs, + inspection_churn_runs, + quick_rechecks, + last_go_live_status, + last_publish_ready, + last_log_sync_state, + last_issue_total, + last_problem_runs_total, + last_launchpad_status, + last_launchpad_action, + last_problem_run_code, + summary_json_path, +) = sys.argv[1:20] + +summary = {} +try: + with open(summary_json_path, "r", encoding="utf-8") as fh: + summary = json.load(fh) +except Exception: + summary = {} + +lines = [ + f"# NIGHT RUN REPORT {run_id}", + "", + f"- Base URL: `{base_url}`", + f"- Deadline: `{deadline_text}`", + f"- Stop Reason: `{stop_reason}`", + f"- Cycles: `{cycles}`", + f"- Log Sync Recover Runs: `{log_sync_recover_runs}`", + f"- Inspection Runs: `{inspection_runs}`", + f"- Inspection Churn Runs: `{inspection_churn_runs}`", + f"- Quick Rechecks: `{quick_rechecks}`", + "", + "## Final Snapshot", + "", + f"- `go_live_status = {last_go_live_status}`", + f"- `publish_ready = {last_publish_ready}`", + f"- `log_sync_state = {last_log_sync_state}`", + f"- `issue_total = {last_issue_total}`", + f"- `problem_runs_total = {last_problem_runs_total}`", + f"- `launchpad_status = {last_launchpad_status}`", + f"- `launchpad_recommended_action = {last_launchpad_action}`", + f"- `problem_run_code = {last_problem_run_code}`", + "", + "## Summary JSON", + "", + "```json", + json.dumps(summary, ensure_ascii=False, indent=2), + "```", +] + +with open(report_path, "w", encoding="utf-8") as fh: + fh.write("\n".join(lines) + "\n") +PY +} + +cleanup() { + write_report + log "night run stopped: reason=${stop_reason}" + log "report: ${REPORT_FILE}" + rm -f "${PID_FILE}" +} + +trap cleanup EXIT + +log "night run started: run_id=${RUN_ID}" +log "base_url=${BASE_URL}" +log "deadline=${DEADLINE_TEXT}" +log "state_dir=${STATE_DIR}" + +while true; do + now_epoch="$(date +%s)" + if [[ "${now_epoch}" -ge "${DEADLINE_EPOCH}" ]]; then + stop_reason="deadline_reached" + break + fi + + cycles=$((cycles + 1)) + cycle_dir="${STATE_DIR}/cycle_${cycles}" + snapshot_state "${cycle_dir}" + + log "cycle=${cycles} go_live=${last_go_live_status} log_sync=${last_log_sync_state} issue_total=${last_issue_total} problem_runs=${last_problem_runs_total} launchpad=${last_launchpad_status}/${last_launchpad_action} focus_run=${last_problem_run_code}" + + if [[ "${last_log_sync_state}" != "full_capture" ]]; then + log "action: recover log sync coverage" + run_cmd bash "${SCRIPT_DIR}/drive_ops_center.sh" log-sync-recover "${BASE_URL}" full confirm cli/night-pack || true + log_sync_recover_runs=$((log_sync_recover_runs + 1)) + run_cmd bash "${SCRIPT_DIR}/drive_ops_center.sh" driver-run "${BASE_URL}" run_inspection_participating '{}' confirm cli/night-pack || true + inspection_runs=$((inspection_runs + 1)) + quick_rechecks=$((quick_rechecks + 1)) + sleep 45 + continue + fi + + if [[ "${last_issue_total}" != "0" && "${last_problem_runs_total}" != "0" && "${inspection_churn_runs}" -lt 3 ]]; then + log "action: create fresh safe inspection run to dilute stale problem run window" + run_cmd bash "${SCRIPT_DIR}/drive_ops_center.sh" driver-run "${BASE_URL}" run_inspection_participating '{}' confirm cli/night-pack || true + inspection_runs=$((inspection_runs + 1)) + inspection_churn_runs=$((inspection_churn_runs + 1)) + sleep 1800 + continue + fi + + if [[ "${last_issue_total}" == "0" && "${last_log_sync_state}" == "full_capture" && "${last_problem_runs_total}" == "0" ]]; then + stop_reason="signoff_ready_candidate" + break + fi + + sleep 3600 +done diff --git a/domain-api/deploy/multi-region/templates/domaincheck-api.env.example b/domain-api/deploy/multi-region/templates/domaincheck-api.env.example index 06b3b6f..9bd74c1 100644 --- a/domain-api/deploy/multi-region/templates/domaincheck-api.env.example +++ b/domain-api/deploy/multi-region/templates/domaincheck-api.env.example @@ -26,5 +26,5 @@ SYNC_SOURCE_REGION=overseas SYNC_TARGET_REGION=overseas SYNC_TARGET_API_BASE_URL= SYNC_SHARED_TOKEN= -SYNC_BATCH_SIZE=200 -SYNC_POLL_INTERVAL_SECONDS=30 +SYNC_BATCH_SIZE=5000 +SYNC_POLL_INTERVAL_SECONDS=2 diff --git a/domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example b/domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example index 458174b..1adf123 100644 --- a/domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example +++ b/domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example @@ -11,5 +11,5 @@ API_SERVICE_NAME=domaincheck-api SYNC_AGENT_SERVICE_NAME=domaincheck-sync-agent NODE_AGENT_SERVICE_NAME=domaincheck-node-agent -OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.start_detection","runtime.stop_detection","runtime.pull_tasks","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","health.snapshot","logs.collect","diagnostics.collect","deploy.release"] +OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.start_detection","runtime.stop_detection","runtime.pull_tasks","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","runtime.reset_lab_state","health.snapshot","logs.collect","diagnostics.collect","deploy.release"] OPS_AGENT_LABELS={} diff --git a/domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example b/domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example index f0c8107..b1fc4f0 100644 --- a/domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example +++ b/domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example @@ -23,5 +23,5 @@ SYNC_SOURCE_REGION=mainland SYNC_TARGET_REGION=overseas SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1 SYNC_SHARED_TOKEN= -SYNC_BATCH_SIZE=200 -SYNC_POLL_INTERVAL_SECONDS=30 +SYNC_BATCH_SIZE=5000 +SYNC_POLL_INTERVAL_SECONDS=2 diff --git a/domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example b/domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example index cfe80c2..084e511 100644 --- a/domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example +++ b/domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example @@ -23,5 +23,5 @@ SYNC_SOURCE_REGION=mainland SYNC_TARGET_REGION=overseas SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1 SYNC_SHARED_TOKEN= -SYNC_BATCH_SIZE=200 -SYNC_POLL_INTERVAL_SECONDS=30 +SYNC_BATCH_SIZE=5000 +SYNC_POLL_INTERVAL_SECONDS=2 diff --git a/domain-api/deploy/systemd/domain-node-agent.service b/domain-api/deploy/systemd/domain-node-agent.service index 84300cd..ed9d991 100644 --- a/domain-api/deploy/systemd/domain-node-agent.service +++ b/domain-api/deploy/systemd/domain-node-agent.service @@ -6,13 +6,14 @@ After=network.target Type=simple WorkingDirectory=/opt/domaincheck/domain-api EnvironmentFile=-/etc/default/domaincheck-api +EnvironmentFile=-/etc/default/domaincheck-worker EnvironmentFile=-/etc/default/domaincheck-node-agent Environment="PATH=/home/www/.nvm/versions/node/v20.20.2/bin:/home/www/.local/bin:/home/www/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin" ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m app.node_agent Restart=always RestartSec=5 -User=www -Group=www +User=root +Group=root SupplementaryGroups=systemd-journal [Install] diff --git a/domain-api/tests/test_cluster_runtime_service.py b/domain-api/tests/test_cluster_runtime_service.py new file mode 100644 index 0000000..009c8b9 --- /dev/null +++ b/domain-api/tests/test_cluster_runtime_service.py @@ -0,0 +1,93 @@ +from __future__ import annotations + +import unittest +from unittest.mock import MagicMock, patch + +from app.services import cluster_runtime_service + + +class RuntimeSchemaGuardTests(unittest.TestCase): + def test_ensure_runtime_schema_executes_only_once_per_process(self) -> None: + conn = MagicMock() + cursor_cm = MagicMock() + cursor = MagicMock() + conn.cursor.return_value = cursor_cm + cursor_cm.__enter__.return_value = cursor + db_cm = MagicMock() + db_cm.__enter__.return_value = conn + + with patch.object(cluster_runtime_service, "_RUNTIME_SCHEMA_READY", False): + with patch.object(cluster_runtime_service, "get_db", return_value=db_cm) as mocked_get_db: + cluster_runtime_service.ensure_runtime_schema() + cluster_runtime_service.ensure_runtime_schema() + + mocked_get_db.assert_called_once() + self.assertEqual(cursor.execute.call_count, 2) + cursor.execute.assert_any_call( + "SELECT pg_advisory_xact_lock(%s)", + (cluster_runtime_service._RUNTIME_SCHEMA_ADVISORY_LOCK_ID,), + ) + cursor.execute.assert_any_call(cluster_runtime_service._RUNTIME_SCHEMA_SQL) + conn.commit.assert_called_once() + + def test_control_node_supports_worker_only_on_mainland_with_worker_signals(self) -> None: + self.assertFalse( + cluster_runtime_service._control_node_supports_worker( + region="overseas", + metadata={ + "worker_online": False, + "detect_participating": True, + "active_threads": 0, + "max_threads": 0, + }, + ) + ) + self.assertTrue( + cluster_runtime_service._control_node_supports_worker( + region="mainland", + metadata={ + "worker_online": True, + "detect_participating": False, + "active_threads": 0, + "max_threads": 0, + }, + ) + ) + + def test_metadata_idle_without_runtime_work_detects_stale_idle_heartbeat(self) -> None: + self.assertTrue( + cluster_runtime_service._metadata_idle_without_runtime_work( + { + "phase_label": "idle", + "phase_detail": "Worker 已启动,等待检测指令", + "active_threads": 323, + "max_threads": 4, + "active_job_code": "", + "job_items_total": 0, + "job_items_claimed": 0, + "job_items_running": 0, + "job_items_completed": 0, + "job_items_failed": 0, + } + ) + ) + self.assertFalse( + cluster_runtime_service._metadata_idle_without_runtime_work( + { + "phase_label": "running", + "active_threads": 12, + } + ) + ) + self.assertFalse( + cluster_runtime_service._metadata_idle_without_runtime_work( + { + "phase_label": "idle", + "active_job_code": "sync-overseas-1", + } + ) + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/domain-api/tests/test_core_db_retry.py b/domain-api/tests/test_core_db_retry.py new file mode 100644 index 0000000..44f263f --- /dev/null +++ b/domain-api/tests/test_core_db_retry.py @@ -0,0 +1,34 @@ +from psycopg2 import errors + +from app.core.db import db_read_retry + + +def test_db_read_retry_retries_retryable_error_once() -> None: + attempts = {"count": 0} + + @db_read_retry(attempts=3, initial_delay_seconds=0) + def flaky() -> str: + attempts["count"] += 1 + if attempts["count"] == 1: + raise errors.DeadlockDetected() + return "ok" + + assert flaky() == "ok" + assert attempts["count"] == 2 + + +def test_db_read_retry_does_not_swallow_non_retryable_error() -> None: + attempts = {"count": 0} + + @db_read_retry(attempts=3, initial_delay_seconds=0) + def broken() -> str: + attempts["count"] += 1 + raise ValueError("boom") + + try: + broken() + except ValueError as exc: + assert str(exc) == "boom" + else: + raise AssertionError("expected ValueError") + assert attempts["count"] == 1 diff --git a/domain-api/tests/test_core_files_runtime_root.py b/domain-api/tests/test_core_files_runtime_root.py new file mode 100644 index 0000000..d6ef633 --- /dev/null +++ b/domain-api/tests/test_core_files_runtime_root.py @@ -0,0 +1,32 @@ +from __future__ import annotations + +import os +import tempfile +import unittest +from pathlib import Path +from unittest.mock import patch + +from app.core import files + + +class RuntimeRootTests(unittest.TestCase): + def test_runtime_root_prefers_shared_runtime_outside_release_tree(self) -> None: + fake_file = "/opt/domaincheck/releases/domaincheck_release_20260420_222047/domain-api/app/core/files.py" + with patch.object(files, "__file__", fake_file): + with patch.object(Path, "mkdir", autospec=True, return_value=None) as mocked_mkdir: + runtime_root = files.runtime_root() + + self.assertEqual(Path("/opt/domaincheck/runtime/domain-api"), runtime_root) + mocked_mkdir.assert_called_once() + + def test_runtime_root_respects_env_override(self) -> None: + with tempfile.TemporaryDirectory() as tempdir: + override = Path(tempdir) / "custom-runtime" + with patch.dict(os.environ, {"DOMAIN_API_RUNTIME_ROOT": str(override)}, clear=False): + runtime_root = files.runtime_root() + self.assertEqual(override, runtime_root) + self.assertTrue(runtime_root.exists()) + + +if __name__ == "__main__": + unittest.main() diff --git a/domain-api/tests/test_dashboard_service.py b/domain-api/tests/test_dashboard_service.py new file mode 100644 index 0000000..0c6c451 --- /dev/null +++ b/domain-api/tests/test_dashboard_service.py @@ -0,0 +1,237 @@ +import unittest +from unittest.mock import patch + +from app.services.dashboard import fetch_overview + + +class _FakeCursor: + def __init__(self, responses): + self._responses = list(responses) + + def execute(self, sql, params=None): + self._last_sql = sql + self._last_params = params + + def fetchone(self): + if self._responses: + return self._responses.pop(0) + return (0,) + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + +class _FakeConnection: + def __init__(self, responses): + self._cursor = _FakeCursor(responses) + + def cursor(self): + return self._cursor + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + +class DashboardServiceTests(unittest.TestCase): + @patch("app.services.dashboard._fetch_active_jobs_aggregate") + @patch("app.services.dashboard.get_detect_capacity_plan") + @patch("app.services.dashboard.get_detect_queue_health") + @patch("app.services.dashboard.get_runtime_status") + @patch("app.services.dashboard.get_active_detect_job_summary") + @patch("app.services.dashboard.get_db") + def test_fetch_overview_includes_ops_metrics( + self, + mock_get_db, + mock_get_active_detect_job_summary, + mock_get_runtime_status, + mock_get_detect_queue_health, + mock_get_detect_capacity_plan, + mock_fetch_active_jobs_aggregate, + ) -> None: + mock_get_db.return_value = _FakeConnection( + responses=[ + (1000,), + (900,), + (10,), + (5,), + (0,), + (1,), + (430,), + (420,), + (17,), + ] + ) + mock_fetch_active_jobs_aggregate.return_value = { + "active_jobs_total": 3, + "queue": { + "items_total": 5200, + "pending": 4700, + "claimed": 11, + "running": 165, + "completed": 300, + "blacklisted": 12, + "failed": 7, + }, + "throughput": { + "processed_recent": 30, + "processed_per_minute": 2.0, + "completed_recent": 20, + "blacklisted_recent": 1, + "failed_recent": 2, + }, + "retry_total": 12, + "steps": [ + { + "step_code": "detect_register", + "step_name": "注册状态检测", + "items_pending": 185, + "items_running": 3, + "items_claimed": 0, + "items_completed": 7, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 20, + "processed_per_minute": 1.33, + } + ], + "nodes": [ + { + "node_code": "mainland-controller-01", + "items_running": 120, + "items_claimed": 0, + "processed_recent": 20, + "processed_per_minute": 1.33, + "completed_recent": 15, + "failed_recent": 1, + "blacklisted_recent": 0, + } + ], + } + mock_get_active_detect_job_summary.return_value = { + "job_id": 76, + "job_code": "sync-overseas-3612", + "items_pending": 185, + "items_running": 8, + "items_completed": 7, + "items_blacklisted": 0, + "items_failed": 0, + } + mock_get_runtime_status.return_value = { + "worker": {"running": True, "mode": "linux-systemd", "expected_on_this_node": True}, + "node": {"region": "overseas", "role": "control"}, + "cluster": {"summary": {"online_worker_nodes": 2, "dedicated_online_worker_nodes": 1, "online_control_nodes": 1}}, + "detect": { + "backlog": { + "pending_total": 9438, + "claimed_total": 410, + "running_total": 15, + "completed_total": 951, + "blacklisted_total": 0, + "failed_total": 137, + "register_pending": 8487, + "downstream_pending": 951, + } + }, + } + mock_get_detect_queue_health.return_value = { + "has_active_job": True, + "job": {"job_id": 76, "job_code": "sync-overseas-3612", "status": "running", "progress_percent": 3.5}, + "queue": { + "items_total": 200, + "pending": 185, + "claimed": 0, + "display_claimed": 0, + "running": 8, + "display_running": 165, + "completed": 7, + "blacklisted": 0, + "failed": 0, + }, + "throughput": {"processed_recent": 0, "processed_per_minute": 0}, + "steps": [ + { + "step_code": "detect_register", + "step_name": "注册状态检测", + "items_pending": 185, + "items_running": 165, + "items_claimed": 0, + "items_completed": 7, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0.0, + } + ], + "runtime_activity": { + "step_stats": { + "detect_360_site": { + "step_code": "detect_360_site", + "started_recent": 269, + "processed_recent": 6, + "completed_recent": 6, + "failed_recent": 0, + "blacklisted_recent": 0, + } + } + }, + "nodes": [ + { + "node_code": "mainland-controller-01", + "items_running": 165, + "items_claimed": 0, + "processed_recent": 0, + "processed_per_minute": 0.0, + "completed_recent": 0, + "failed_recent": 0, + "blacklisted_recent": 0, + } + ], + } + mock_get_detect_capacity_plan.return_value = { + "estimated_hours_remaining": 1.5, + "remaining_items": 193, + "recommended_additional_workers": 1, + } + + data = fetch_overview() + + self.assertEqual("sync-overseas-3612", data["active_job"]["job_code"]) + self.assertEqual(3, data["active_jobs_aggregate"]["active_jobs_total"]) + self.assertEqual(9438, data["active_jobs_aggregate"]["queue"]["pending"]) + self.assertEqual(410, data["active_jobs_aggregate"]["queue"]["claimed"]) + self.assertEqual(15, data["active_jobs_aggregate"]["queue"]["running"]) + self.assertEqual(951, data["active_jobs_aggregate"]["queue"]["completed"]) + self.assertEqual(137, data["active_jobs_aggregate"]["queue"]["failed"]) + self.assertEqual("mainland-controller-01", data["active_jobs_aggregate"]["nodes"][0]["node_code"]) + self.assertEqual(900, data["pending_total"]) + self.assertEqual(10, data["completed_total"]) + self.assertEqual(430, data["registerable_total"]) + self.assertEqual(420, data["purchasable_total"]) + self.assertEqual(185, data["queue_pending_total"]) + self.assertEqual(8, data["queue_running_total"]) + self.assertEqual(165, data["queue_display_running_total"]) + self.assertEqual(7, data["queue_completed_total"]) + self.assertEqual(9438, data["backlog_pending_total"]) + self.assertEqual(8487, data["backlog_register_pending_total"]) + self.assertEqual(951, data["backlog_downstream_pending_total"]) + self.assertEqual(12, data["retry_total"]) + self.assertEqual("注册状态检测", data["bottleneck_step"]["step_name"]) + self.assertEqual(0.0, data["processed_per_minute"]) + self.assertEqual(0, data["processed_recent"]) + self.assertEqual(0.0, data["ops_summary"]["processed_per_minute"]) + self.assertEqual(1.5, data["ops_summary"]["estimated_hours_remaining"]) + self.assertEqual(1, data["active_execution_nodes"]) + self.assertEqual(1, data["ops_summary"]["active_execution_nodes"]) + self.assertEqual(2, len(data["step_queue"])) + self.assertTrue(any(item["step_code"] == "detect_360_site" for item in data["step_queue"])) + self.assertEqual(1, len(data["node_throughput"])) + + +if __name__ == "__main__": + unittest.main() diff --git a/domain-api/tests/test_detect_api_routes.py b/domain-api/tests/test_detect_api_routes.py new file mode 100644 index 0000000..08102bc --- /dev/null +++ b/domain-api/tests/test_detect_api_routes.py @@ -0,0 +1,75 @@ +from __future__ import annotations + +import unittest +from unittest.mock import patch + +from app.api.routes import detect as detect_route + + +class DetectApiRoutesTestCase(unittest.TestCase): + @patch("app.api.routes.detect.create_detect_run_snapshot") + @patch("app.api.routes.detect._dispatch_remote_detect_start") + @patch("app.api.routes.detect.get_settings_payload") + @patch("app.api.routes.detect.get_detect_status") + @patch("app.api.routes.detect.send_worker_command") + @patch("app.api.routes.detect.start_worker") + @patch("app.api.routes.detect.append_detect_job_event") + @patch("app.api.routes.detect.create_detect_job_if_needed") + def test_start_detect_skips_local_worker_on_overseas_control( + self, + mock_create_job, + mock_append_event, + mock_start_worker, + mock_send_worker_command, + mock_get_detect_status, + mock_get_settings_payload, + mock_dispatch_remote, + mock_create_snapshot, + ) -> None: + mock_create_job.return_value = { + "job_id": 29, + "job_code": "sync-overseas-29", + "status": "running", + "items_pending": 100, + "items_claimed": 0, + "items_running": 0, + "task_mode": "single_step", + "step_code": "detect_register", + } + mock_get_detect_status.return_value = { + "worker_mode": "linux-systemd", + "worker_online": False, + "worker_process_count": 0, + "worker_latest_start_time": "", + "worker_runtime_message": "not-applicable", + "progress": {}, + } + mock_get_settings_payload.return_value = { + "runtime": {"thread_count": 2000}, + "proxy_config": {"proxy_enable": True, "allow_direct": False, "proxy_urls": ["a"]}, + } + mock_dispatch_remote.return_value = { + "queued_jobs": [{"node_code": "mainland-controller-01"}], + "queued_total": 1, + "failed_total": 0, + "target_summary": {"controller_nodes": ["mainland-controller-01"], "worker_nodes": ["mainland-worker-01"]}, + } + + with patch.object(detect_route.settings, "node_region", "overseas"), patch.object( + detect_route.settings, "node_role", "control" + ): + response = detect_route.start_detect() + + self.assertEqual(0, response.code) + self.assertIn("海外控制面", response.message) + mock_start_worker.assert_not_called() + mock_send_worker_command.assert_not_called() + mock_dispatch_remote.assert_called_once() + mock_create_snapshot.assert_called_once() + event_types = [call.kwargs.get("event_type") for call in mock_append_event.call_args_list] + self.assertIn("job_dispatch_requested", event_types) + self.assertIn("job_dispatch_skipped_local", event_types) + + +if __name__ == "__main__": + unittest.main() diff --git a/domain-api/tests/test_detect_job_service.py b/domain-api/tests/test_detect_job_service.py index ab06d84..9a92f98 100644 --- a/domain-api/tests/test_detect_job_service.py +++ b/domain-api/tests/test_detect_job_service.py @@ -1,121 +1,791 @@ import unittest +from unittest.mock import MagicMock, patch -from app.services.detect_job_service import _build_effective_node_stats, _build_effective_summary +from psycopg2 import errors + +from app.services.detect_job_service import ( + append_detect_job_event, + _build_step_payload, + _classify_pipeline_item_outcome, + _classify_runtime_debug_event, + _enrich_active_job_summary_with_runtime, + _build_display_summary, + _build_runtime_display_bucket, + normalize_detect_step_code, + get_detect_queue_health, + process_detect_pipeline_now, + resolve_initial_domain_pipeline_item, + resolve_domain_pipeline_step, + resolve_detect_job_definition, +) class DetectJobServiceTests(unittest.TestCase): - def test_build_effective_node_stats_keeps_runtime_nodes_and_backfills_unassigned(self) -> None: - distributed_node_stats = [ - { - "node_code": "mainland-controller-01", - "items_total": 200, - "items_pending": 195, - "items_claimed": 0, - "items_running": 5, - "items_completed": 0, - "items_blacklisted": 0, - "items_failed": 0, - }, - { - "node_code": "mainland-worker-01", - "items_total": 70, - "items_pending": 56, - "items_claimed": 9, - "items_running": 5, - "items_completed": 0, - "items_blacklisted": 0, - "items_failed": 0, - }, - { - "node_code": "overseas-control-01", - "items_total": 50, - "items_pending": 0, - "items_claimed": 25, - "items_running": 4, - "items_completed": 21, - "items_blacklisted": 0, - "items_failed": 0, - }, - { - "node_code": "unassigned", - "items_total": 950, - "items_pending": 950, - "items_claimed": 0, - "items_running": 0, - "items_completed": 0, - "items_blacklisted": 0, - "items_failed": 0, - }, - ] + def test_append_detect_job_event_skips_missing_job_fk(self) -> None: + class FakeCursor: + def __init__(self) -> None: + self.exec_calls = [] + self.selects = 0 - node_stats = _build_effective_node_stats( - distributed_node_stats=distributed_node_stats, - raw_items_total=1000, + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + def execute(self, sql, params=None): + self.exec_calls.append((sql, params)) + if "INSERT INTO detect_run_events" in sql: + raise AssertionError("should not insert detect_run_events when job row is missing") + if "SELECT 1 FROM detect_jobs" in sql: + self.selects += 1 + + def fetchone(self): + return None + + class FakeConn: + def __init__(self) -> None: + self.cursor_obj = FakeCursor() + self.commit_calls = 0 + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + def cursor(self): + return self.cursor_obj + + def commit(self): + self.commit_calls += 1 + + fake_conn = FakeConn() + + with patch("app.services.detect_job_service.get_db", return_value=fake_conn): + with patch("app.services.detect_job_service.push_debug_event") as mock_push: + append_detect_job_event( + 5, + event_type="job_dispatch_requested", + message="控制面已发送检测启动请求", + ) + + self.assertEqual(1, fake_conn.cursor_obj.selects) + self.assertEqual(0, fake_conn.commit_calls) + mock_push.assert_called_once() + + def test_process_detect_pipeline_now_retries_deadlock_once(self) -> None: + calls = {"count": 0} + + def _run_pipeline(*, limit, job_id): + calls["count"] += 1 + if calls["count"] == 1: + raise errors.DeadlockDetected() + return { + "processed_items": 7, + "advanced_items": 3, + "retried_items": 1, + } + + with patch("app.services.detect_job_service.process_detect_pipeline", side_effect=_run_pipeline): + with patch("app.services.detect_job_service.time.sleep") as mock_sleep: + ok, message, data = process_detect_pipeline_now(limit=123, job_id=45) + + self.assertTrue(ok) + self.assertIn("deadlock 自动重试 1 次后成功", message) + self.assertEqual(1, data["retry_attempts"]) + self.assertEqual(7, data["processed_items"]) + self.assertEqual(2, calls["count"]) + mock_sleep.assert_called_once() + + def test_classify_runtime_debug_event_maps_single_step_finalized_completed(self) -> None: + classified = _classify_runtime_debug_event( + event_type="worker_log", + message="检测步骤跟踪: domain=example.com | step=注册状态检测 | stage=single_step_finalized | elapsed_ms=2700 | ok=1 | detect_key=detect_register | final_status=completed | result_state=passed", + payload={"job_code": "sync-overseas-27456", "node_code": "mainland-worker-01"}, ) - self.assertEqual(4, len(node_stats)) - unassigned = next(item for item in node_stats if item["node_code"] == "unassigned") - self.assertEqual(680, unassigned["items_total"]) - self.assertEqual(680, unassigned["items_pending"]) - self.assertEqual("central_queue", unassigned["metrics_source"]) + self.assertEqual("sync-overseas-27456", classified["job_code"]) + self.assertTrue(classified["terminal"]) + self.assertEqual("completed", classified["terminal_status"]) + self.assertEqual("detect_register", classified["step_code"]) - def test_build_effective_summary_uses_effective_node_stats_and_raw_blacklisted(self) -> None: - node_stats = [ - { - "node_code": "mainland-controller-01", - "items_total": 200, - "items_pending": 195, - "items_claimed": 0, - "items_running": 5, - "items_completed": 0, - "items_blacklisted": 0, - "items_failed": 0, - }, - { - "node_code": "mainland-worker-01", - "items_total": 70, - "items_pending": 56, - "items_claimed": 9, - "items_running": 5, - "items_completed": 0, - "items_blacklisted": 0, - "items_failed": 0, - }, - { - "node_code": "overseas-control-01", - "items_total": 50, - "items_pending": 0, - "items_claimed": 25, - "items_running": 4, - "items_completed": 21, - "items_blacklisted": 0, - "items_failed": 0, - }, - { - "node_code": "unassigned", - "items_total": 680, - "items_pending": 680, - "items_claimed": 0, - "items_running": 0, - "items_completed": 0, - "items_blacklisted": 0, - "items_failed": 0, - }, - ] - - summary = _build_effective_summary( - node_stats=node_stats, - raw_items_total=1000, - raw_items_blacklisted=3, + def test_build_runtime_display_bucket_prefers_real_active_threads_over_raw_current_load(self) -> None: + bucket = _build_runtime_display_bucket( + ( + "mainland-worker-01", + "mainland", + "worker", + "busy", + 753, + { + "job_items_total": 1000, + "job_items_claimed": 0, + "job_items_running": 0, + "job_items_completed": 342, + "active_threads": 61, + "max_threads": 400, + "detect_participating": True, + }, + None, + ) ) + self.assertIsNotNone(bucket) + self.assertEqual(61, bucket["current_load"]) + self.assertEqual(61, bucket["display_running"]) + self.assertEqual(61, bucket["active_threads"]) + + def test_build_runtime_display_bucket_skips_local_overseas_control_plane_load(self) -> None: + with patch("app.services.detect_job_service.settings.node_region", "overseas"): + with patch("app.services.detect_job_service.settings.node_role", "control"): + with patch("app.services.detect_job_service.settings.node_code", "overseas-control-01"): + bucket = _build_runtime_display_bucket( + ( + "overseas-control-01", + "overseas", + "control", + "busy", + 371, + { + "active_threads": 371, + "max_threads": 3200, + "detect_participating": False, + }, + None, + ) + ) + + self.assertIsNone(bucket) + + def test_build_runtime_display_bucket_skips_any_control_node_without_worker_support(self) -> None: + bucket = _build_runtime_display_bucket( + ( + "overseas-control-01", + "overseas", + "control", + "busy", + 378, + { + "active_threads": 378, + "max_threads": 3200, + "detect_participating": False, + "worker_online": False, + }, + None, + ) + ) + + self.assertIsNone(bucket) + + def test_build_display_summary_ignores_inflated_raw_current_load(self) -> None: + summary = _build_display_summary( + [ + { + "node_code": "mainland-controller-01", + "items_claimed": 0, + "items_running": 0, + "display_running": 729, + "current_load": 729, + "active_threads": 729, + "max_threads": 800, + "items_completed": 196, + "items_failed": 0, + }, + { + "node_code": "mainland-worker-01", + "items_claimed": 0, + "items_running": 0, + "display_running": 753, + "current_load": 753, + "active_threads": 61, + "max_threads": 400, + "items_completed": 146, + "items_failed": 0, + }, + ] + ) + + self.assertEqual(790, summary["display_running"]) + self.assertEqual(790, summary["current_load"]) + self.assertEqual(790, summary["active_threads"]) + + def test_enrich_active_job_summary_with_runtime_adds_runtime_job_code_and_recent_events(self) -> None: + summary = _enrich_active_job_summary_with_runtime( + { + "job_id": 275, + "job_code": "sync-overseas-5297", + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "failed_recent": 0, + "blacklisted_recent": 0, + "recent_domain_events": [], + }, + event_limit=20, + window_minutes=15, + runtime_activity={ + "focus_job_code": "sync-overseas-9506", + "job_codes": ["sync-overseas-9506", "sync-overseas-27456"], + "processed_recent": 120, + "completed_recent": 118, + "failed_recent": 1, + "blacklisted_recent": 1, + }, + runtime_snapshot={ + "job": { + "job_id": 1835, + "job_code": "sync-overseas-9506", + "progress_percent": 44.2, + }, + "queue_health": { + "queue": { + "items_total": 1000, + "pending": 176, + "claimed": 223, + "running": 159, + "completed": 442, + "display_running": 220, + } + }, + }, + recent_domain_events=[ + { + "event_type": "domain_completed", + "message": "域名检测完成: example.com", + "created_at": "2026-04-21 04:10:00", + } + ], + ) + + self.assertEqual("sync-overseas-9506", summary["runtime_job_code"]) + self.assertEqual(["sync-overseas-9506", "sync-overseas-27456"], summary["runtime_job_codes"]) + self.assertEqual(120, summary["processed_recent"]) + self.assertEqual(8.0, summary["processed_per_minute"]) + self.assertEqual(118, summary["completed_recent"]) + self.assertEqual(1, summary["failed_recent"]) + self.assertEqual(1, summary["blacklisted_recent"]) + self.assertEqual(1, len(summary["recent_domain_events"])) + self.assertEqual("sync-overseas-9506", summary["runtime_snapshot_job_code"]) + self.assertEqual(1000, summary["runtime_snapshot_queue"]["items_total"]) self.assertEqual(1000, summary["items_total"]) - self.assertEqual(931, summary["items_pending"]) - self.assertEqual(34, summary["items_claimed"]) - self.assertEqual(14, summary["items_running"]) - self.assertEqual(21, summary["items_completed"]) - self.assertEqual(3, summary["items_blacklisted"]) - self.assertEqual(24, summary["items_terminal"]) + self.assertEqual(176, summary["items_pending"]) + self.assertEqual(223, summary["items_claimed"]) + self.assertEqual(159, summary["items_running"]) + self.assertEqual(442, summary["items_completed"]) + self.assertEqual(220, summary["display_items_running"]) + + def test_enrich_active_job_summary_with_runtime_prefers_snapshot_identity_and_events(self) -> None: + summary = _enrich_active_job_summary_with_runtime( + { + "job_id": 275, + "job_code": "sync-overseas-5297", + "status": "running", + "recent_events": [ + { + "node_code": "mainland-worker-01", + "event_type": "domain_started", + "message": "开始检测域名: stale.com", + "payload": {"job_code": "sync-overseas-5297"}, + "created_at": "2026-04-21 11:58:00", + } + ], + "current_cycle_events": [], + "latest_event": None, + "display_items_running": 0, + "display_current_load": 0, + "display_active_threads": 0, + "display_max_threads": 0, + "display_active_node_codes": [], + }, + event_limit=10, + window_minutes=15, + runtime_activity={ + "focus_job_code": "sync-overseas-31437", + "job_codes": ["sync-overseas-31437"], + "processed_recent": 10, + "completed_recent": 10, + "failed_recent": 0, + "blacklisted_recent": 0, + }, + runtime_snapshot={ + "job": { + "job_id": 1902, + "job_code": "sync-overseas-31437", + "status": "running", + "progress_percent": 34.3, + "node_stats": [ + { + "node_code": "mainland-controller-01", + "items_claimed": 230, + "items_running": 67, + "items_completed": 343, + "items_failed": 0, + "active_threads": 170, + "max_threads": 2000, + }, + { + "node_code": "mainland-worker-01", + "items_claimed": 75, + "items_running": 0, + "items_completed": 0, + "items_failed": 0, + "active_threads": 19, + "max_threads": 1200, + }, + ], + }, + "queue_health": { + "queue": { + "items_total": 1000, + "pending": 285, + "claimed": 305, + "running": 67, + "completed": 343, + "display_running": 189, + } + }, + "recent_events": [ + { + "job_id": 2137, + "node_code": "mainland-controller-01", + "event_type": "job_created", + "message": "同步拉取待检测批次 sync-overseas-35461,共 1000 个任务项", + "payload": {"source_record_id": 35461}, + "created_at": "2026-04-22 01:21:29", + }, + { + "job_id": 1902, + "node_code": "mainland-controller-01", + "event_type": "worker_log", + "message": "从任务队列获取到 125 个需要检测的域名", + "payload": {"job_code": "sync-overseas-31437"}, + "created_at": "2026-04-22 01:19:25", + }, + ], + }, + ) + + self.assertEqual(1902, summary["job_id"]) + self.assertEqual("sync-overseas-31437", summary["job_code"]) + self.assertEqual("sync-overseas-31437", summary["runtime_job_code"]) + self.assertEqual(189, summary["display_items_running"]) + self.assertEqual(189, summary["display_current_load"]) + self.assertEqual(189, summary["display_active_threads"]) + self.assertEqual(3200, summary["display_max_threads"]) + self.assertEqual(["mainland-controller-01", "mainland-worker-01"], summary["display_active_node_codes"]) + self.assertEqual(1, len(summary["recent_events"])) + self.assertEqual("worker_log", summary["latest_event"]["event_type"]) + self.assertEqual("sync-overseas-31437", summary["recent_events"][0]["payload"]["job_code"]) + + def test_get_detect_queue_health_preserves_runtime_node_throughput_after_snapshot_override(self) -> None: + lease_row = (None, None, 0, 0) + throughput_rows = [] + step_throughput_rows = [] + runtime_display_rows = [ + ( + "mainland-controller-01", + "mainland", + "control", + "busy", + 170, + { + "job_items_total": 265, + "job_items_claimed": 85, + "job_items_running": 0, + "job_items_completed": 105, + "active_threads": 170, + "max_threads": 2000, + "detect_participating": True, + }, + None, + ), + ( + "mainland-worker-01", + "mainland", + "worker", + "busy", + 19, + { + "job_items_total": 139, + "job_items_claimed": 64, + "job_items_running": 0, + "job_items_completed": 4, + "active_threads": 19, + "max_threads": 1200, + "detect_participating": True, + }, + None, + ), + ( + "overseas-control-01", + "overseas", + "control", + "busy", + 371, + { + "active_threads": 371, + "max_threads": 3200, + "detect_participating": False, + }, + None, + ), + ] + + conn = MagicMock() + cursor_cm = MagicMock() + cursor = MagicMock() + conn.cursor.return_value = cursor_cm + cursor_cm.__enter__.return_value = cursor + db_cm = MagicMock() + db_cm.__enter__.return_value = conn + cursor.fetchone.return_value = lease_row + cursor.fetchall.side_effect = [throughput_rows, step_throughput_rows, runtime_display_rows] + + active_job = { + "job_id": 1937, + "job_code": "sync-overseas-31987", + "status": "running", + "items_total": 1000, + "items_pending": 596, + "items_claimed": 149, + "items_running": 146, + "items_completed": 109, + "items_blacklisted": 0, + "items_failed": 0, + "progress_percent": 10.9, + "node_stats": [], + "distributed_node_stats": [ + { + "node_code": "mainland-controller-01", + "items_total": 265, + "items_pending": 0, + "items_claimed": 85, + "items_running": 75, + "items_completed": 105, + "items_blacklisted": 0, + "items_failed": 0, + "metrics_source": "runtime", + }, + { + "node_code": "mainland-worker-01", + "items_total": 139, + "items_pending": 0, + "items_claimed": 64, + "items_running": 71, + "items_completed": 4, + "items_blacklisted": 0, + "items_failed": 0, + "metrics_source": "runtime", + }, + ], + "step_stats": [ + { + "step_code": "detect_register", + "items_total": 1000, + "items_pending": 449, + "items_claimed": 131, + "items_running": 273, + "items_completed": 147, + "items_blacklisted": 0, + "items_failed": 0, + } + ], + } + runtime_activity = { + "processed_recent": 133, + "completed_recent": 113, + "failed_recent": 20, + "blacklisted_recent": 0, + "focus_job_code": "sync-overseas-31563", + "job_codes": ["sync-overseas-31563"], + "step_code": "detect_register", + "nodes": { + "mainland-controller-01": { + "node_code": "mainland-controller-01", + "processed_recent": 74, + "completed_recent": 54, + "failed_recent": 20, + "blacklisted_recent": 0, + }, + "mainland-worker-01": { + "node_code": "mainland-worker-01", + "processed_recent": 59, + "completed_recent": 59, + "failed_recent": 0, + "blacklisted_recent": 0, + }, + }, + } + runtime_snapshot = { + "job": { + "job_id": 1937, + "job_code": "sync-overseas-31987", + "progress_percent": 10.9, + }, + "queue_health": { + "queue": { + "items_total": 1000, + "pending": 596, + "claimed": 149, + "running": 146, + "completed": 109, + "blacklisted": 0, + "failed": 0, + "display_claimed": 149, + "display_running": 403, + }, + "nodes": [ + { + "node_code": "mainland-controller-01", + "items_total": 265, + "items_pending": 0, + "items_claimed": 85, + "items_running": 75, + "items_completed": 105, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + "metrics_source": "runtime", + }, + { + "node_code": "mainland-worker-01", + "items_total": 139, + "items_pending": 0, + "items_claimed": 64, + "items_running": 71, + "items_completed": 4, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + "metrics_source": "runtime", + }, + ], + "steps": [], + }, + } + + with patch("app.services.detect_job_service.settings.node_region", "overseas"): + with patch("app.services.detect_job_service.settings.node_role", "control"): + with patch("app.services.detect_job_service.settings.node_code", "overseas-control-01"): + with patch("app.services.detect_job_service.get_active_detect_job_summary", return_value=active_job): + with patch("app.services.detect_job_service._load_runtime_activity_snapshot", return_value=runtime_activity): + with patch("app.services.detect_job_service._load_latest_runtime_active_job_snapshot", return_value=runtime_snapshot): + with patch("app.services.detect_job_service.get_db", return_value=db_cm): + health = get_detect_queue_health(window_minutes=15) + + node_map = {item["node_code"]: item for item in health["nodes"]} + self.assertEqual(74, node_map["mainland-controller-01"]["processed_recent"]) + self.assertEqual(54, node_map["mainland-controller-01"]["completed_recent"]) + self.assertEqual(20, node_map["mainland-controller-01"]["failed_recent"]) + self.assertEqual(170, node_map["mainland-controller-01"]["items_running"]) + self.assertEqual(59, node_map["mainland-worker-01"]["processed_recent"]) + self.assertEqual(59, node_map["mainland-worker-01"]["completed_recent"]) + self.assertEqual(19, node_map["mainland-worker-01"]["items_running"]) + self.assertEqual(113, health["throughput"]["completed_recent"]) + self.assertEqual(189, health["queue"]["display_running"]) + self.assertNotIn("overseas-control-01", node_map) + + def test_normalize_detect_step_code_accepts_supported_single_step(self) -> None: + self.assertEqual("detect_baidu_site", normalize_detect_step_code("detect_baidu_site")) + self.assertEqual("detect_wayback", normalize_detect_step_code("detect_wayback")) + + def test_normalize_detect_step_code_rejects_unknown_step(self) -> None: + self.assertEqual("", normalize_detect_step_code("detect_unknown")) + + def test_resolve_detect_job_definition_builds_single_step_job(self) -> None: + definition = resolve_detect_job_definition("detect_baidu_site") + + self.assertTrue(definition["is_single_step"]) + self.assertEqual("single_step", definition["task_mode"]) + self.assertEqual("detect_baidu_site", definition["step_code"]) + + def test_resolve_detect_job_definition_builds_wayback_single_step_job(self) -> None: + definition = resolve_detect_job_definition("detect_wayback") + + self.assertTrue(definition["is_single_step"]) + self.assertEqual("single_step", definition["task_mode"]) + self.assertEqual("detect_wayback", definition["step_code"]) + + def test_build_step_payload_adds_wayback_recent_years_strategy(self) -> None: + payload = _build_step_payload( + step_code="detect_wayback", + domain_snapshot={"domain": "example.com", "source_type": 2}, + settings_payload={"detect_options": {"detect_wayback": True}}, + ) + + self.assertEqual("detect_wayback", payload["step_code"]) + self.assertEqual("recent_years", payload["wayback_strategy"]) + self.assertEqual(5, payload["wayback_recent_years"]) + self.assertTrue(payload["wayback_stop_on_first_hit"]) + + def test_resolve_detect_job_definition_defaults_to_domain_pipeline(self) -> None: + definition = resolve_detect_job_definition(None) + + self.assertFalse(definition["is_single_step"]) + self.assertEqual("domain_pipeline", definition["task_mode"]) + self.assertEqual("", definition["step_code"]) + + def test_resolve_domain_pipeline_step_skips_yikoujia_register(self) -> None: + step_code = resolve_domain_pipeline_step( + { + "source_type": 1, + "register_status": 0, + "baidu_site": {}, + "qihu360_site": {}, + "chinaz_info": {}, + "aizhan_info": {}, + "wayback_info": {}, + "jucha_info": {}, + "juziseo_info": {}, + }, + settings_payload={ + "detect_options": { + "detect_register": True, + "detect_baidu_site": True, + "detect_360_site": False, + "detect_chinaz": False, + "detect_aizhan": False, + "detect_wayback": False, + "detect_jucha": False, + "detect_juziseo": False, + "detect_order": ["detect_register", "detect_baidu_site"], + } + }, + ) + + self.assertEqual("detect_baidu_site", step_code) + + def test_resolve_domain_pipeline_step_moves_to_next_incomplete_step(self) -> None: + step_code = resolve_domain_pipeline_step( + { + "source_type": 2, + "register_status": 3, + "baidu_site": {"state": "passed"}, + "qihu360_site": {}, + "chinaz_info": {}, + "aizhan_info": {}, + "wayback_info": {}, + "jucha_info": {}, + "juziseo_info": {}, + }, + settings_payload={ + "detect_options": { + "detect_register": True, + "detect_baidu_site": True, + "detect_360_site": True, + "detect_chinaz": False, + "detect_aizhan": False, + "detect_wayback": False, + "detect_jucha": False, + "detect_juziseo": False, + "detect_order": ["detect_register", "detect_baidu_site", "detect_360_site"], + } + }, + after_step_code="detect_baidu_site", + ) + + self.assertEqual("detect_360_site", step_code) + + def test_resolve_initial_domain_pipeline_item_builds_first_step_payload(self) -> None: + step_code, payload = resolve_initial_domain_pipeline_item( + { + "id": 10, + "domain": "example.com", + "source_type": 2, + "register_status": 0, + "baidu_site": {}, + "qihu360_site": {}, + "chinaz_info": {}, + "aizhan_info": {}, + "wayback_info": {}, + "jucha_info": {}, + "juziseo_info": {}, + }, + settings_payload={ + "detect_options": { + "detect_register": True, + "detect_baidu_site": True, + "detect_order": ["detect_register", "detect_baidu_site"], + } + }, + ) + + self.assertEqual("detect_register", step_code) + self.assertIsNotNone(payload) + self.assertEqual("detect_register", payload["step_code"]) + self.assertEqual("example.com", payload["domain"]) + + def test_resolve_initial_domain_pipeline_item_returns_empty_when_pipeline_already_done(self) -> None: + step_code, payload = resolve_initial_domain_pipeline_item( + { + "id": 11, + "domain": "done.com", + "source_type": 2, + "register_status": 3, + "baidu_site": {"state": "passed"}, + "qihu360_site": {}, + "chinaz_info": {}, + "aizhan_info": {}, + "wayback_info": {}, + "jucha_info": {}, + "juziseo_info": {}, + }, + settings_payload={ + "detect_options": { + "detect_register": True, + "detect_baidu_site": True, + "detect_order": ["detect_register", "detect_baidu_site"], + } + }, + ) + + self.assertEqual("", step_code) + self.assertIsNone(payload) + + def test_classify_pipeline_item_outcome_retries_external_failure(self) -> None: + outcome = _classify_pipeline_item_outcome( + item_status="failed", + result_payload={"state": "degraded", "message": "timeout", "retry_recommended": True}, + step_code="detect_baidu_site", + attempt_count=0, + ) + + self.assertEqual("retry", outcome["action"]) + self.assertTrue(outcome["should_retry"]) + self.assertEqual("external_retry", outcome["reason_code"]) + + def test_classify_pipeline_item_outcome_rejects_business_failure(self) -> None: + outcome = _classify_pipeline_item_outcome( + item_status="failed", + result_payload={"state": "rejected", "message": "title contains forbidden keyword"}, + step_code="detect_chinaz", + attempt_count=0, + ) + + self.assertEqual("reject", outcome["action"]) + self.assertFalse(outcome["should_retry"]) + self.assertEqual("business_reject", outcome["reason_code"]) + + def test_classify_pipeline_item_outcome_marks_blacklisted_terminal(self) -> None: + outcome = _classify_pipeline_item_outcome( + item_status="blacklisted", + result_payload={"state": "blacklisted", "message": "risk hit"}, + step_code="detect_baidu_site", + attempt_count=0, + ) + + self.assertEqual("black_hit", outcome["action"]) + self.assertFalse(outcome["should_retry"]) + self.assertEqual("blacklisted", outcome["reason_code"]) if __name__ == "__main__": diff --git a/domain-api/tests/test_detect_service_status_fallback.py b/domain-api/tests/test_detect_service_status_fallback.py new file mode 100644 index 0000000..cde0391 --- /dev/null +++ b/domain-api/tests/test_detect_service_status_fallback.py @@ -0,0 +1,69 @@ +import unittest +from unittest.mock import patch + +from app.services import detect_service + + +class DetectServiceStatusFallbackTests(unittest.TestCase): + def test_get_detect_status_keeps_runtime_snapshot_when_db_is_unreachable(self) -> None: + runtime_state = { + "service_running": True, + "detecting": True, + "active_threads": 7, + "max_threads": 120, + "phase": "running", + "detail": "Worker 正在处理 7 个检测任务", + "updated_at": "2026-04-20 23:59:00", + "available_proxy_count": 18, + } + + with patch("app.services.detect_service.ensure_runtime_schema"), \ + patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")), \ + patch("app.services.detect_service.get_settings_payload", return_value={"proxy_config": {"proxy_enable": True, "allow_direct": False, "proxy_urls": ["a"]}}), \ + patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "full"}), \ + patch("app.services.detect_service._load_recent_worker_lines", return_value=(True, "", [])), \ + patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": True, "process_count": 1, "latest_start_time": "2026-04-20 23:58:00", "message": "active/running"}), \ + patch("app.services.detect_service._load_runtime_state", return_value=runtime_state), \ + patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}), \ + patch("app.services.detect_service._extract_available_proxy_count", return_value=0), \ + patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}), \ + patch("app.services.detect_service._normalize_recent_warning", return_value=""), \ + patch("app.services.detect_service._build_proxy_runtime_snapshot", return_value={"state": "healthy", "label": "代理正常", "detail": "healthy", "direct_fallback_active": False, "reason": "healthy", "last_refresh_status": "ok", "last_refresh_time": "", "source_count": 2, "raw_items": 18, "validated_count": 18, "available_count": 18, "source_stats": [], "supplier_empty": False}), \ + patch("app.services.detect_service.resolve_thread_count", return_value={"effective_thread_count": 120, "default_thread_count": 5, "source": "node_override", "override_thread_count": 120, "node_code": "mainland-worker-01"}), \ + patch("app.services.detect_service.get_active_detect_job_summary", side_effect=RuntimeError("db down")), \ + patch("app.services.detect_service.sync_detect_runs", return_value=[]), \ + patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}), \ + patch("app.services.detect_service._extract_dependency_alerts", return_value=[]), \ + patch("app.services.detect_service.append_detect_result_projection_if_changed"): + payload = detect_service.get_detect_status() + + self.assertTrue(payload["worker_online"]) + self.assertTrue(payload["detecting"]) + self.assertEqual(7, payload["active_thread_count"]) + self.assertEqual(120, payload["max_thread_count"]) + self.assertEqual(0, payload["progress"]["pending"]) + self.assertEqual(0, payload["progress"]["completed"]) + + def test_filter_lines_since_supports_journalctl_syslog_timestamps(self) -> None: + lines = [ + "Apr 21 20:12:42 mainland-controller python[1]: 当前实际线程数量: 323/4", + "Apr 21 20:17:20 mainland-controller python[2]: Worker 已启动,等待检测指令", + ] + + filtered = detect_service._filter_lines_since(lines, "2026-04-21 20:17:00") + + self.assertEqual( + ["Apr 21 20:17:20 mainland-controller python[2]: Worker 已启动,等待检测指令"], + filtered, + ) + + def test_filter_lines_since_falls_back_when_no_timestamp_is_parseable(self) -> None: + lines = ["no timestamp line 1", "no timestamp line 2"] + + filtered = detect_service._filter_lines_since(lines, "2026-04-21 20:17:00") + + self.assertEqual(lines, filtered) + + +if __name__ == "__main__": + unittest.main() diff --git a/domain-api/tests/test_juming_services.py b/domain-api/tests/test_juming_services.py new file mode 100644 index 0000000..c2a4ecb --- /dev/null +++ b/domain-api/tests/test_juming_services.py @@ -0,0 +1,205 @@ +from __future__ import annotations + +import json +import os +import tempfile +import unittest +from contextlib import contextmanager +from unittest.mock import patch + +from requests.cookies import RequestsCookieJar + +from app.services import juming_service, juming_task_service + + +class JumingServiceTests(unittest.TestCase): + def test_crawl_juming_rejects_invalid_cookie(self) -> None: + cookie_jar = RequestsCookieJar() + cookie_jar.set("sid", "expired") + + with patch.object(juming_service, "_load_juming_cookie", return_value=(cookie_jar, "local")): + with patch.object(juming_service, "_validate_juming_cookie", return_value=(False, "聚名登录态已失效,请重新登录")): + with self.assertRaisesRegex(ValueError, "已失效"): + juming_service.crawl_juming({"mode": "delete_list"}) + + def test_get_juming_status_exposes_remote_validation(self) -> None: + cookie_jar = RequestsCookieJar() + cookie_jar.set("sid", "alive") + + with patch.object(juming_service, "_load_juming_cookie", return_value=(cookie_jar, "local")): + with patch.object(juming_service, "_validate_juming_cookie", return_value=(False, "聚名登录态已失效,请重新登录")): + payload = juming_service.get_juming_status() + + self.assertTrue(payload["cookie_present"]) + self.assertFalse(payload["cookie_valid"]) + self.assertFalse(payload["cookie_ready"]) + self.assertIn("已失效", payload["cookie_message"]) + + def test_insert_domains_uses_copy_stage_import_path(self) -> None: + class FakeCursor: + def __init__(self) -> None: + self.executed: list[tuple[str, object]] = [] + self.copy_calls: list[tuple[str, tuple[str, ...], str]] = [] + + def __enter__(self): + return self + + def __exit__(self, exc_type, exc, tb): + return False + + def execute(self, sql: str, params: object = None) -> None: + self.executed.append((sql, params)) + + def copy_from(self, file_obj, table: str, columns: tuple[str, ...]) -> None: + self.copy_calls.append((table, columns, file_obj.read())) + + def fetchone(self): + return (2, 2, 0) + + class FakeConn: + def __init__(self) -> None: + self.cursor_obj = FakeCursor() + self.commit_calls = 0 + + def cursor(self): + return self.cursor_obj + + def commit(self) -> None: + self.commit_calls += 1 + + fake_conn = FakeConn() + + @contextmanager + def fake_get_db(): + yield fake_conn + + with patch.object(juming_service, "get_db", fake_get_db): + with patch.object(juming_service, "IMPORT_BATCH_SIZE", 2): + stats = juming_service._insert_domains( + ["alpha.com", "beta.net"], + juming_service.DELETE_LIST_SOURCE_TYPE, + ) + + self.assertEqual(2, stats["added"]) + self.assertEqual(1, fake_conn.commit_calls) + self.assertTrue(fake_conn.cursor_obj.copy_calls) + table, columns, payload = fake_conn.cursor_obj.copy_calls[0] + self.assertEqual("juming_import_stage", table) + self.assertEqual(("domain", "tld"), columns) + self.assertIn("alpha.com\tcom", payload) + self.assertIn("beta.net\tnet", payload) + executed_sql = "\n".join(sql for sql, _params in fake_conn.cursor_obj.executed) + self.assertIn("create temporary table if not exists juming_import_stage", executed_sql.lower()) + self.assertIn("inserted as", executed_sql.lower()) + self.assertIn("left join domains existing", executed_sql.lower()) + + def test_delete_list_import_skips_already_imported_same_signature(self) -> None: + cookie_jar = RequestsCookieJar() + cookie_jar.set("sid", "alive") + + class FakeJM: + cookie = cookie_jar + + def new_cha_del(self, current_date: str): + if current_date == "2026-03-21": + return ["alpha.com", "beta.net"] + return [] + + logs: list[str] = [] + signature = juming_service._compute_domains_signature(["alpha.com", "beta.net"]) + cached_state = { + "2026-03-21": { + "signature": signature, + "total": 2, + "valid": 2, + "invalid": 0, + } + } + + with patch.object(juming_service, "_load_juming_cookie", return_value=(cookie_jar, "local")): + with patch.object(juming_service, "JM", return_value=FakeJM()): + with patch.object(juming_service, "_load_delete_import_state", return_value=cached_state): + with patch.object(juming_service, "_insert_domains") as mock_insert: + result = juming_service._crawl_delete_list_and_import( + "2026-03-21", + False, + log=logs.append, + ) + + mock_insert.assert_not_called() + self.assertEqual(0, result["stats"]["added"]) + self.assertEqual(2, result["stats"]["exists"]) + self.assertTrue(any("跳过重复入库" in line for line in logs)) + + +class JumingTaskServiceTests(unittest.TestCase): + def setUp(self) -> None: + self._tmpdir = tempfile.TemporaryDirectory() + self._old_runtime_root = os.environ.get("DOMAIN_API_RUNTIME_ROOT") + os.environ["DOMAIN_API_RUNTIME_ROOT"] = self._tmpdir.name + juming_task_service._ACTIVE_TASK_IDS.clear() + + def tearDown(self) -> None: + juming_task_service._ACTIVE_TASK_IDS.clear() + if self._old_runtime_root is None: + os.environ.pop("DOMAIN_API_RUNTIME_ROOT", None) + else: + os.environ["DOMAIN_API_RUNTIME_ROOT"] = self._old_runtime_root + self._tmpdir.cleanup() + + def _write_tasks(self, records: list[dict]) -> None: + path = os.path.join(self._tmpdir.name, "juming_tasks.json") + with open(path, "w", encoding="utf-8") as handle: + json.dump(records, handle, ensure_ascii=False, indent=2) + + def test_cleanup_orphaned_tasks_marks_running_task_failed(self) -> None: + self._write_tasks( + [ + { + "task_id": "task-1", + "status": "running", + "phase": "importing", + "phase_label": "入库中", + "cancel_requested": False, + "message": "开始入库处理", + "created_at": "2026-04-21 21:00:00", + "updated_at": "2026-04-21 21:00:00", + "started_at": "2026-04-21 21:00:00", + "completed_at": "", + "result": None, + "logs": [], + } + ] + ) + + tasks = juming_task_service.list_juming_tasks() + self.assertEqual("failed", tasks[0]["status"]) + self.assertIn("中断", tasks[0]["message"]) + + def test_create_task_rejects_parallel_active_task(self) -> None: + self._write_tasks( + [ + { + "task_id": "task-1", + "status": "running", + "phase": "fetching", + "phase_label": "抓取中", + "cancel_requested": False, + "message": "正在抓取", + "created_at": "2026-04-21 21:00:00", + "updated_at": "2026-04-21 21:00:00", + "started_at": "2026-04-21 21:00:00", + "completed_at": "", + "result": None, + "logs": [], + } + ] + ) + juming_task_service._ACTIVE_TASK_IDS.add("task-1") + + with self.assertRaisesRegex(ValueError, "已有聚名采集任务正在运行"): + juming_task_service.create_juming_task({"mode": "delete_list"}) + + +if __name__ == "__main__": + unittest.main() diff --git a/domain-api/tests/test_node_agent_delivery_queue.py b/domain-api/tests/test_node_agent_delivery_queue.py index 960843a..8f652b5 100644 --- a/domain-api/tests/test_node_agent_delivery_queue.py +++ b/domain-api/tests/test_node_agent_delivery_queue.py @@ -20,6 +20,111 @@ class NodeAgentDeliveryQueueTests(unittest.TestCase): self.assertEqual(0, delivery_queue["pending_count"]) self.assertEqual(0, delivery_queue["dead_letter_count"]) + def test_base_payload_prefers_non_loopback_identity(self) -> None: + with tempfile.TemporaryDirectory() as temp_dir: + with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir), patch.object( + node_agent, + "CONTROL_PLANE_BASE_URL", + "http://152.53.37.118:8100", + ), patch.object( + node_agent.socket, + "gethostname", + return_value="localhost", + ), patch.object( + node_agent.socket, + "getfqdn", + return_value="localhost.localdomain", + ), patch.object( + node_agent.os, + "uname", + return_value=type("Uname", (), {"nodename": "localhost"})(), + ), patch.object( + node_agent, + "NODE_CODE", + "mainland-controller-01", + ), patch.object( + node_agent.socket, + "getaddrinfo", + return_value=[(None, None, None, None, ("127.0.0.1", 0))], + ), patch.object( + node_agent.socket, + "gethostbyname", + return_value="127.0.0.1", + ): + class FakeSocket: + def connect(self, target): + self.target = target + + def getsockname(self): + return ("121.204.244.188", 12345) + + def close(self): + return None + + with patch.object(node_agent.socket, "socket", return_value=FakeSocket()): + payload = node_agent._base_payload() + + self.assertNotIn(payload["hostname"], {"", "localhost", "localhost.localdomain"}) + self.assertEqual("121.204.244.188", payload["ip"]) + + def test_detect_runtime_snapshot_degrades_to_worker_runtime_when_detect_status_fails(self) -> None: + with patch.dict(os.environ, {}, clear=False): + with patch( + "app.services.worker_control_service.detect_worker_runtime", + return_value={ + "running": True, + "process_count": 1, + "latest_start_time": "2026-04-20 18:00:00", + "message": "active/running", + }, + ), patch( + "app.services.detect_service.get_detect_status", + side_effect=RuntimeError('connection to server at "127.0.0.1", port 5432 failed'), + ): + snapshot = node_agent._detect_runtime_snapshot() + + self.assertTrue(snapshot["worker_online"]) + self.assertTrue(snapshot["service_running"]) + self.assertFalse(snapshot["detecting"]) + self.assertEqual("active/running", snapshot["phase_detail"]) + self.assertEqual("2026-04-20 18:00:00", snapshot["updated_at"]) + self.assertIn("127.0.0.1", snapshot["error"]) + + def test_detect_runtime_snapshot_infers_worker_online_from_active_threads(self) -> None: + with patch.dict(os.environ, {}, clear=False): + with patch( + "app.services.worker_control_service.detect_worker_runtime", + return_value={ + "running": False, + "process_count": 0, + "latest_start_time": "", + "message": "", + }, + ), patch( + "app.services.detect_service.get_detect_status", + return_value={ + "worker_online": False, + "detecting": False, + "active_thread_count": 19, + "max_thread_count": 1200, + "phase_label": "检测中", + "phase_detail": "Worker 正在处理 162 个检测任务", + "runtime_state": { + "service_running": False, + "updated_at": "2026-04-21 00:00:06", + }, + "active_job": {"items_running": 0}, + }, + ): + snapshot = node_agent._detect_runtime_snapshot() + + self.assertTrue(snapshot["worker_online"]) + self.assertTrue(snapshot["service_running"]) + self.assertTrue(snapshot["detecting"]) + self.assertTrue(snapshot["detect_participating"]) + self.assertEqual(19, snapshot["active_threads"]) + self.assertEqual(19, snapshot["current_load"]) + def test_job_event_network_failure_is_queued_for_retry(self) -> None: with tempfile.TemporaryDirectory() as temp_dir: with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir), patch.object( @@ -279,6 +384,63 @@ class NodeAgentDeliveryQueueTests(unittest.TestCase): self.assertEqual("failed_local", event_payload["start_delivery_state"]) self.assertIn("temporary offline", event_payload["start_delivery_error"]) + def test_register_heartbeat_and_pull_use_configured_timeouts(self) -> None: + with patch.object(node_agent, "_post", return_value={"code": 0, "message": "ok", "data": {"jobs": []}}) as mock_post, patch.object( + node_agent, + "AGENT_REGISTER_TIMEOUT_SECONDS", + 91, + ), patch.object( + node_agent, + "AGENT_HEARTBEAT_TIMEOUT_SECONDS", + 92, + ), patch.object( + node_agent, + "AGENT_PULL_TIMEOUT_SECONDS", + 93, + ): + node_agent._register() + node_agent._heartbeat() + jobs = node_agent._pull_jobs() + + self.assertEqual([], jobs) + self.assertEqual(3, mock_post.call_count) + self.assertEqual(91, mock_post.call_args_list[0].kwargs["timeout"]) + self.assertEqual(92, mock_post.call_args_list[1].kwargs["timeout"]) + self.assertEqual(93, mock_post.call_args_list[2].kwargs["timeout"]) + + def test_job_delivery_uses_configured_timeouts(self) -> None: + with patch.object( + node_agent, + "_deliver_or_queue", + side_effect=lambda **kwargs: {"state": "queued", "timeout": kwargs["timeout"]}, + ) as mock_deliver, patch.object( + node_agent, + "AGENT_JOB_COMPLETE_TIMEOUT_SECONDS", + 94, + ), patch.object( + node_agent, + "AGENT_JOB_EVENT_TIMEOUT_SECONDS", + 47, + ): + complete_result = node_agent._job_complete( + 11, + status="success", + stdout="ok", + stderr="", + result={"ok": True}, + ) + event_result = node_agent._job_event( + 11, + event_type="executor_received", + message="accepted", + ) + + self.assertEqual("queued", complete_result["state"]) + self.assertEqual("queued", event_result["state"]) + self.assertEqual(2, mock_deliver.call_count) + self.assertEqual(94, mock_deliver.call_args_list[0].kwargs["timeout"]) + self.assertEqual(47, mock_deliver.call_args_list[1].kwargs["timeout"]) + if __name__ == "__main__": unittest.main() diff --git a/domain-api/tests/test_ops_action_executor_core.py b/domain-api/tests/test_ops_action_executor_core.py index 55d94e2..7f40ecd 100644 --- a/domain-api/tests/test_ops_action_executor_core.py +++ b/domain-api/tests/test_ops_action_executor_core.py @@ -110,6 +110,34 @@ class OpsActionExecutorCoreTests(unittest.TestCase): self.assertTrue(data["fallback_used"]) self.assertIn("detect_worker.log", data["log_paths"][0]) + @patch("app.services.ops_action_executor_core._reset_lab_state") + def test_runtime_reset_lab_state_returns_helper_result_with_host_context(self, mock_reset_lab_state) -> None: + runner = _FakeRunner({}) + mock_reset_lab_state.return_value = ( + True, + "lab runtime reset completed", + {"database": {"include_domains": True}}, + ) + + ok, message, data = execute_structured_action( + "runtime.reset_lab_state", + {"clear_database": True, "clear_domains": True}, + service_names=build_service_name_map( + api_service_name="domaincheck-api", + worker_service_name="domaincheck-worker", + sync_agent_service_name="domaincheck-sync-agent", + ), + runner=runner, + host_context={"hostname": "mainland-controller-01", "ip": "121.204.244.188"}, + ) + + self.assertTrue(ok) + self.assertEqual("lab runtime reset completed", message) + self.assertEqual("mainland-controller-01", data["hostname"]) + self.assertEqual("121.204.244.188", data["ip"]) + self.assertTrue(data["database"]["include_domains"]) + mock_reset_lab_state.assert_called_once() + if __name__ == "__main__": unittest.main() diff --git a/domain-api/tests/test_ops_agent_service.py b/domain-api/tests/test_ops_agent_service.py index 1d4d52d..80de80c 100644 --- a/domain-api/tests/test_ops_agent_service.py +++ b/domain-api/tests/test_ops_agent_service.py @@ -4,8 +4,11 @@ from unittest.mock import patch from app.api.routes.ops_agent import _build_agent_response from app.services.ops_agent_service import ( + _build_agent_runtime_config_bundle, + _runtime_config_bundle_hash_payload, agent_append_job_event, agent_complete_job, + agent_mark_job_started, agent_pull_jobs, agent_register, build_node_agent_bootstrap_plan, @@ -102,6 +105,64 @@ class _SequenceConnection: class OpsAgentServiceTests(unittest.TestCase): + def test_runtime_config_bundle_hash_ignores_generated_at(self) -> None: + bundle_a = { + "node_code": "mainland-worker-01", + "thread_count": 100, + "node_thread_counts": {"mainland-worker-01": 100}, + "runtime_settings": {"worker_log_sync_enabled": True}, + "generated_at": "2026-04-19 17:30:00", + "config_hash": "stale-hash", + } + bundle_b = { + "node_code": "mainland-worker-01", + "thread_count": 100, + "node_thread_counts": {"mainland-worker-01": 100}, + "runtime_settings": {"worker_log_sync_enabled": True}, + "generated_at": "2026-04-19 17:35:00", + "config_hash": "other-stale-hash", + } + + self.assertEqual( + _runtime_config_bundle_hash_payload(bundle_a), + _runtime_config_bundle_hash_payload(bundle_b), + ) + + @patch("app.services.ops_agent_service.get_sensitive_words_payload") + @patch("app.services.ops_agent_service.get_runtime_settings") + @patch("app.services.ops_agent_service.get_settings_payload") + def test_build_agent_runtime_config_bundle_keeps_hash_stable_for_same_config( + self, + mock_get_settings_payload, + mock_get_runtime_settings, + mock_get_sensitive_words_payload, + ) -> None: + mock_get_settings_payload.return_value = { + "detect_options": {"detect_wayback": True}, + "proxy_config": {"enabled": True}, + "thread_count": 100, + "node_thread_counts": {"mainland-worker-01": 100}, + } + mock_get_runtime_settings.return_value = { + "worker_log_sync_enabled": True, + "worker_log_sync_mode": "full", + } + mock_get_sensitive_words_payload.return_value = { + "text": "foo\nbar", + "total": 2, + "items": ["foo", "bar"], + } + + with patch( + "app.services.ops_agent_service._format_time", + side_effect=["2026-04-19 17:30:00", "2026-04-19 17:35:00"], + ): + first_bundle = _build_agent_runtime_config_bundle("mainland-worker-01") + second_bundle = _build_agent_runtime_config_bundle("mainland-worker-01") + + self.assertNotEqual(first_bundle["generated_at"], second_bundle["generated_at"]) + self.assertEqual(first_bundle["config_hash"], second_bundle["config_hash"]) + def test_build_agent_response_extracts_detail_code(self) -> None: response = _build_agent_response(False, "bad request", {"detail_code": "agent_token_invalid", "foo": "bar"}) @@ -730,6 +791,76 @@ class OpsAgentServiceTests(unittest.TestCase): self.assertEqual(0, payload["summary"]["agent_ready"]) self.assertEqual(1, payload["summary"]["remote_access_ready"]) + @patch("app.services.runtime_status_service.get_runtime_status") + @patch("app.services.cluster_runtime_service.get_cluster_snapshot") + @patch("app.services.ops_job_service.list_managed_nodes") + @patch("app.services.ops_agent_service.get_db") + @patch("app.services.ops_agent_service.ensure_ops_agent_schema") + def test_list_managed_nodes_with_agent_state_prefers_non_loopback_cluster_identity_and_infers_worker_online( + self, + mock_ensure_ops_agent_schema, + mock_get_db, + mock_list_managed_nodes, + mock_get_cluster_snapshot, + mock_get_runtime_status, + ) -> None: + now = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + mock_ensure_ops_agent_schema.return_value = None + mock_get_db.return_value = _EmptyConnection() + mock_list_managed_nodes.return_value = [ + { + "node_code": "mainland-worker-01", + "region": "mainland", + "role": "worker", + "title": "Mainland Worker 01", + "is_enabled": True, + "metadata": { + "hostname": "S244-248", + "ip": "121.204.244.248", + }, + "last_seen_at": now, + } + ] + mock_get_cluster_snapshot.return_value = { + "nodes": [ + { + "node_code": "mainland-worker-01", + "region": "mainland", + "role": "worker", + "status": "busy", + "current_load": 19, + "hostname": "localhost", + "ip": "127.0.0.1", + "last_heartbeat_at": now, + "is_effective_worker": True, + "detect_participating": False, + "metadata": { + "worker_online": False, + "active_threads": 19, + "max_threads": 1200, + "phase_label": "检测中", + "phase_detail": "Worker 正在处理 162 个检测任务", + }, + } + ] + } + mock_get_runtime_status.return_value = { + "detect": { + "participating_nodes": [], + "non_participating_nodes": [], + } + } + + payload = list_managed_nodes_with_agent_state() + + self.assertEqual(1, len(payload["nodes"])) + row = payload["nodes"][0] + self.assertEqual("S244-248", row["cluster_hostname"]) + self.assertEqual("121.204.244.248", row["cluster_ip"]) + self.assertTrue(row["detect_runtime"]["worker_online"]) + self.assertTrue(row["detect_runtime"]["detect_participating"]) + self.assertEqual(19, row["detect_runtime"]["active_threads"]) + def test_agent_register_requires_node_code_with_detail_code(self) -> None: ok, message, data = agent_register({}, token="ignored") @@ -1176,6 +1307,119 @@ class OpsAgentServiceTests(unittest.TestCase): self.assertEqual("当前控制面仅支持对可见头部记录执行单条动作", message) self.assertEqual("evt-hidden", data["record_id"]) + @patch("app.services.ops_agent_service.get_ops_job") + @patch("app.services.ops_agent_service.append_ops_job_event") + @patch("app.services.ops_agent_service._authenticate_agent_token") + @patch("app.services.ops_agent_service.get_db") + def test_agent_pull_jobs_commits_before_appending_event( + self, + mock_get_db, + mock_authenticate_agent_token, + mock_append_ops_job_event, + mock_get_ops_job, + ) -> None: + cursor = _SequenceCursor(fetchall_values=[[(36,)]] ) + connection = _SequenceConnection(cursor) + mock_get_db.return_value = connection + mock_authenticate_agent_token.return_value = (True, "ok", {"node_code": "mainland-worker-01"}) + mock_get_ops_job.return_value = {"id": 36, "job_code": "ops-demo", "action": "health.snapshot"} + + def _assert_after_commit(**kwargs): + self.assertTrue(connection.committed) + self.assertEqual(36, kwargs["job_id"]) + self.assertEqual("agent_dispatched", kwargs["event_type"]) + + mock_append_ops_job_event.side_effect = _assert_after_commit + + ok, message, data = agent_pull_jobs({"node_code": "mainland-worker-01"}, token="agent-token", limit=1) + + self.assertTrue(ok) + self.assertEqual("ok", message) + self.assertEqual(1, data["count"]) + self.assertTrue(connection.committed) + mock_append_ops_job_event.assert_called_once() + + @patch("app.services.ops_agent_service.get_ops_job") + @patch("app.services.ops_agent_service.append_ops_job_event") + @patch("app.services.ops_agent_service._authenticate_agent_token") + @patch("app.services.ops_agent_service.get_db") + def test_agent_mark_job_started_commits_before_appending_event( + self, + mock_get_db, + mock_authenticate_agent_token, + mock_append_ops_job_event, + mock_get_ops_job, + ) -> None: + cursor = _SequenceCursor(fetchone_values=[(36,)], fetchall_values=[[(101,), (102,)]]) + connection = _SequenceConnection(cursor) + mock_get_db.return_value = connection + mock_authenticate_agent_token.return_value = (True, "ok", {"node_code": "mainland-worker-01"}) + mock_get_ops_job.return_value = {"id": 36, "job_code": "ops-demo", "action": "health.snapshot"} + + def _assert_after_commit(**kwargs): + self.assertTrue(connection.committed) + self.assertEqual(36, kwargs["job_id"]) + self.assertEqual("agent_started", kwargs["event_type"]) + self.assertEqual({"step_ids": [101, 102]}, kwargs["payload"]) + + mock_append_ops_job_event.side_effect = _assert_after_commit + + ok, message, data = agent_mark_job_started(36, {"node_code": "mainland-worker-01"}, token="agent-token") + + self.assertTrue(ok) + self.assertEqual("任务已标记为运行中", message) + self.assertEqual(36, data["job"]["id"]) + self.assertTrue(connection.committed) + mock_append_ops_job_event.assert_called_once() + + @patch("app.services.ops_release_service.refresh_release_rollout_for_job") + @patch("app.services.ops_agent_service.get_ops_job") + @patch("app.services.ops_agent_service.append_ops_job_event") + @patch("app.services.ops_agent_service._authenticate_agent_token") + @patch("app.services.ops_agent_service.get_db") + def test_agent_complete_job_commits_before_appending_event( + self, + mock_get_db, + mock_authenticate_agent_token, + mock_append_ops_job_event, + mock_get_ops_job, + mock_refresh_release_rollout_for_job, + ) -> None: + cursor = _SequenceCursor(fetchone_values=[(36, "")], fetchall_values=[[(101,)]] ) + connection = _SequenceConnection(cursor) + mock_get_db.return_value = connection + mock_authenticate_agent_token.return_value = (True, "ok", {"node_code": "mainland-worker-01"}) + mock_get_ops_job.return_value = {"id": 36, "job_code": "ops-demo", "status": "success"} + + def _assert_after_commit(**kwargs): + self.assertTrue(connection.committed) + self.assertEqual(36, kwargs["job_id"]) + self.assertEqual("agent_completed", kwargs["event_type"]) + self.assertEqual("info", kwargs["level"]) + self.assertEqual([101], kwargs["payload"]["step_ids"]) + + mock_append_ops_job_event.side_effect = _assert_after_commit + + ok, message, data = agent_complete_job( + 36, + { + "node_code": "mainland-worker-01", + "status": "success", + "stdout": "ok", + "stderr": "", + "result": {"summary": "done"}, + "duration_ms": 120, + }, + token="agent-token", + ) + + self.assertTrue(ok) + self.assertEqual("任务结果已回写", message) + self.assertEqual(36, data["job"]["id"]) + self.assertTrue(connection.committed) + mock_append_ops_job_event.assert_called_once() + mock_refresh_release_rollout_for_job.assert_called_once_with(36) + if __name__ == "__main__": unittest.main() diff --git a/domain-api/tests/test_ops_job_service.py b/domain-api/tests/test_ops_job_service.py index 4813d5f..8beeb6e 100644 --- a/domain-api/tests/test_ops_job_service.py +++ b/domain-api/tests/test_ops_job_service.py @@ -4,11 +4,13 @@ from datetime import datetime from unittest.mock import MagicMock, patch from app.core.config import settings +from app.core.db import get_db from app.services.ops_job_service import ( _execute_control_plane_job, _serialize_job_row, create_ops_job, dispatch_ops_job, + list_managed_nodes, upsert_managed_node, ) @@ -109,6 +111,39 @@ class OpsJobServiceTests(unittest.TestCase): self.assertTrue(job["steps_loaded"]) self.assertIn("mainland-worker-02", job["summary_text"]) + def test_upsert_managed_node_supports_ssh_entry_and_secret_flags(self) -> None: + node_code = "test-managed-node-secret" + ok, message, data = upsert_managed_node( + { + "node_code": node_code, + "region": "mainland", + "role": "worker", + "ssh_entry": "121.204.244.188 root ", + "is_enabled": True, + } + ) + + self.assertTrue(ok, message) + node = data["node"] + self.assertEqual("121.204.244.188", node["ssh_host"]) + self.assertEqual("root", node["ssh_user"]) + self.assertEqual(22, node["ssh_port"]) + self.assertEqual("password", node["auth_mode"]) + self.assertTrue(node["ssh_password_configured"]) + + managed = next(item for item in list_managed_nodes() if item["node_code"] == node_code) + self.assertTrue(managed["ssh_password_configured"]) + + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + "SELECT ssh_password FROM ops_managed_node_secrets WHERE node_code = %s", + (node_code,), + ) + row = cur.fetchone() + self.assertIsNotNone(row) + self.assertEqual("demo-password", str(row[0] or "")) + @patch("app.services.ops_job_service.list_managed_nodes") @patch("app.services.ops_agent_service.build_node_agent_bootstrap_plan") def test_execute_control_plane_bootstrap_uses_managed_node_defaults( @@ -237,7 +272,11 @@ class OpsJobServiceTests(unittest.TestCase): self.assertEqual("keep-me", node["metadata"]["operator_note"]) self.assertEqual("online", node["metadata"]["cluster_status"]) - insert_call = cursor.execute.call_args_list[-1] + insert_call = next( + call + for call in cursor.execute.call_args_list + if "INSERT INTO ops_managed_nodes" in str(call.args[0]) + ) insert_params = insert_call.args[1] self.assertEqual("121.204.244.248", insert_params[4]) self.assertEqual(22, insert_params[5]) diff --git a/domain-api/tests/test_ops_policy_service.py b/domain-api/tests/test_ops_policy_service.py index a1766e5..d63460d 100644 --- a/domain-api/tests/test_ops_policy_service.py +++ b/domain-api/tests/test_ops_policy_service.py @@ -248,6 +248,42 @@ class OpsPolicyServiceTests(unittest.TestCase): self.assertTrue(preview["blocked"]) self.assertIn("local-runtime 仅支持当前控制面本机节点", " ".join(preview["blocking_reasons"])) + @patch("app.services.ops_policy_service.get_cluster_snapshot") + def test_runtime_reset_lab_state_is_critical_and_requires_approval(self, mock_cluster_snapshot) -> None: + mock_cluster_snapshot.return_value = { + "nodes": [ + { + "node_code": "mainland-controller-01", + "region": "mainland", + "role": "control", + "status": "online", + "current_load": 0, + "is_effective_worker": True, + "detect_participating": False, + } + ], + "summary": { + "online_worker_nodes": 2, + "dedicated_online_worker_nodes": 1, + "online_control_nodes": 2, + "busy_nodes": [], + }, + } + + preview = preview_ops_job_policy( + { + "action": "runtime.reset_lab_state", + "target_type": "node", + "target_node_code": "mainland-controller-01", + "execution_mode": "remote-agent", + "payload": {"clear_database": True, "clear_redis": True}, + } + ) + + self.assertEqual("critical", preview["risk_level"]) + self.assertTrue(preview["approval_required"]) + self.assertFalse(preview["blocked"]) + if __name__ == "__main__": unittest.main() diff --git a/domain-api/tests/test_ops_release_executor_core.py b/domain-api/tests/test_ops_release_executor_core.py index d5da2eb..bcdaf1e 100644 --- a/domain-api/tests/test_ops_release_executor_core.py +++ b/domain-api/tests/test_ops_release_executor_core.py @@ -1,10 +1,18 @@ import io +import grp +import os +import pwd import tarfile import tempfile import unittest from pathlib import Path +from unittest.mock import patch -from app.services.ops_release_executor_core import build_remote_release_action_script, execute_release_action +from app.services.ops_release_executor_core import ( + _pick_release_owner_group, + build_remote_release_action_script, + execute_release_action, +) class _BytesResponse: @@ -24,20 +32,45 @@ class _BytesResponse: def _build_release_archive() -> bytes: buffer = io.BytesIO() with tarfile.open(fileobj=buffer, mode="w:gz") as archive: - content = b"hello-release" - info = tarfile.TarInfo(name="bundle/README.txt") - info.size = len(content) - archive.addfile(info, io.BytesIO(content)) + files = { + "README.txt": b"hello-release", + "domain-api/deploy/systemd/domain-node-agent.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-worker\n", + "domain-api/deploy/systemd/domain-worker.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-worker\n", + "domain-api/deploy/systemd/domain-api.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-api\n", + "domain-api/deploy/systemd/domain-sync-agent.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-worker\n", + } + for name, content in files.items(): + info = tarfile.TarInfo(name=name) + info.size = len(content) + archive.addfile(info, io.BytesIO(content)) return buffer.getvalue() class OpsReleaseExecutorCoreTests(unittest.TestCase): + def test_pick_release_owner_group_prefers_service_identity_over_path_owner(self) -> None: + with tempfile.TemporaryDirectory() as tmpdir: + with patch( + "app.services.ops_release_executor_core._resolve_path_owner_group", + return_value=("root", "root"), + ): + owner_user, owner_group = _pick_release_owner_group( + Path(tmpdir), + [{"service_name": "domaincheck-worker", "user": "www", "group": "www"}], + ) + self.assertEqual(("www", "www"), (owner_user, owner_group)) + def test_execute_release_action_deploys_archive_and_switches_current(self) -> None: archive_bytes = _build_release_archive() commands: list[tuple[str, ...]] = [] def run_command(command: list[str], *, timeout: int = 60): commands.append(tuple(command)) + if command[:3] == ["systemctl", "show", "domaincheck-worker"] and "-p" in command: + current_user = pwd.getpwuid(os.getuid()).pw_name + current_group = grp.getgrgid(os.getgid()).gr_name + return 0, f"{current_user}\n{current_group}\n", "" + if command[:2] == ["chown", "-R"]: + return 0, "", "" if command[:2] == ["systemctl", "restart"]: return 0, "", "" if command[:2] == ["systemctl", "is-active"]: @@ -45,11 +78,13 @@ class OpsReleaseExecutorCoreTests(unittest.TestCase): return 0, "", "" with tempfile.TemporaryDirectory() as tmpdir: + systemd_root = Path(tmpdir) / "systemd" ok, message, data = execute_release_action( { "release_version": "2026.04.18-rc1", "artifact_url": "https://example.com/domaincheck.tar.gz", "install_root": tmpdir, + "systemd_unit_root": str(systemd_root), "restart_services": ["domaincheck-worker"], "health_check_services": ["domaincheck-worker"], "health_check_urls": [], @@ -70,19 +105,29 @@ class OpsReleaseExecutorCoreTests(unittest.TestCase): release_dir = Path(data["release_dir"]) self.assertTrue(release_dir.exists()) self.assertTrue((release_dir / ".release-meta.json").exists()) + self.assertTrue(any(cmd[:2] == ("chown", "-R") for cmd in commands)) + self.assertIn(("systemctl", "daemon-reload"), commands) self.assertIn(("systemctl", "restart", "domaincheck-worker"), commands) self.assertIn(("systemctl", "is-active", "domaincheck-worker"), commands) + self.assertTrue((systemd_root / "domaincheck-node-agent.service").exists()) + self.assertTrue((systemd_root / "domaincheck-node-agent.service.d" / "current-path.conf").exists()) def test_build_remote_release_action_script_is_valid_python(self) -> None: script = build_remote_release_action_script( { "release_version": "2026.04.18-rc1", "artifact_url": "https://example.com/domaincheck.tar.gz", + "switch_current": True, + "rollback_on_failure": False, } ) compile(script, "", "exec") self.assertIn("execute_release_action(", script) + self.assertIn("True", script) + self.assertIn("False", script) + self.assertIn("def collect_service_identity(", script) + self.assertIn("def apply_release_permissions(", script) if __name__ == "__main__": diff --git a/domain-api/tests/test_ops_runtime_executor_service.py b/domain-api/tests/test_ops_runtime_executor_service.py index e4e4f93..b080ebe 100644 --- a/domain-api/tests/test_ops_runtime_executor_service.py +++ b/domain-api/tests/test_ops_runtime_executor_service.py @@ -2,14 +2,17 @@ import subprocess import tempfile import unittest from pathlib import Path -from unittest.mock import patch +from unittest.mock import MagicMock, patch +from app.services import ops_runtime_executor_service from app.services.ops_runtime_executor_service import execute_local_support_action, execute_ssh_action class OpsRuntimeExecutorServiceTests(unittest.TestCase): @patch("app.services.ops_runtime_executor_service.subprocess.run") - def test_execute_ssh_action_parses_structured_json_result(self, mock_run) -> None: + @patch("app.services.ops_runtime_executor_service._load_ssh_secret") + def test_execute_ssh_action_parses_structured_json_result(self, mock_load_secret, mock_run) -> None: + mock_load_secret.return_value = {"ssh_password": "", "ssh_private_key": ""} mock_run.return_value = subprocess.CompletedProcess( args=["ssh"], returncode=0, @@ -113,6 +116,105 @@ class OpsRuntimeExecutorServiceTests(unittest.TestCase): issued_command = mock_run.call_args.args[0] self.assertEqual(["systemctl", "restart", "domaincheck-worker"], issued_command) + @patch("app.services.ops_runtime_executor_service.subprocess.run") + @patch("app.services.ops_runtime_executor_service._load_ssh_secret") + @patch("app.services.ops_runtime_executor_service.paramiko.SSHClient") + def test_execute_ssh_action_uses_paramiko_for_password_auth(self, mock_ssh_client_cls, mock_load_secret, mock_run) -> None: + mock_load_secret.return_value = {"ssh_password": "demo-pass", "ssh_private_key": ""} + mock_client = MagicMock() + stdout = MagicMock() + stderr = MagicMock() + stdout.channel.recv_exit_status.return_value = 0 + stdout.read.return_value = ( + b'{"ok": true, "message": "health snapshot collected", "result": {"checks": {"api": {"state": "active"}}}}\n' + ) + stderr.read.return_value = b"" + mock_client.exec_command.return_value = (None, stdout, stderr) + mock_ssh_client_cls.return_value = mock_client + + ok, message, data = execute_ssh_action( + { + "node_code": "mainland-worker-01", + "ssh_host": "121.204.244.248", + "ssh_port": 22, + "ssh_user": "root", + "auth_mode": "password", + }, + "health.snapshot", + {}, + ) + + self.assertTrue(ok) + self.assertEqual("health snapshot collected", message) + self.assertEqual("active", data["checks"]["api"]["state"]) + self.assertEqual("password", data["transport"]["auth_mode"]) + mock_client.connect.assert_called_once() + self.assertFalse(mock_run.called) + + @patch("app.services.ops_runtime_executor_service.subprocess.run") + @patch("app.services.ops_runtime_executor_service._load_ssh_secret") + @patch("app.services.ops_runtime_executor_service.paramiko.SSHClient") + def test_execute_ssh_action_falls_back_to_password_when_node_auth_mode_is_key_but_only_password_exists( + self, + mock_ssh_client_cls, + mock_load_secret, + mock_run, + ) -> None: + mock_load_secret.return_value = {"ssh_password": "demo-pass", "ssh_private_key": ""} + mock_client = MagicMock() + stdout = MagicMock() + stderr = MagicMock() + stdout.channel.recv_exit_status.return_value = 0 + stdout.read.return_value = ( + b'{"ok": true, "message": "health snapshot collected", "result": {"checks": {"api": {"state": "active"}}}}\n' + ) + stderr.read.return_value = b"" + mock_client.exec_command.return_value = (None, stdout, stderr) + mock_ssh_client_cls.return_value = mock_client + + ok, message, data = execute_ssh_action( + { + "node_code": "mainland-controller-01", + "ssh_host": "121.204.244.188", + "ssh_port": 22, + "ssh_user": "root", + "auth_mode": "key", + }, + "health.snapshot", + {}, + ) + + self.assertTrue(ok) + self.assertEqual("health snapshot collected", message) + self.assertEqual("password", data["transport"]["auth_mode"]) + mock_client.connect.assert_called_once() + self.assertFalse(mock_run.called) + + @patch("app.services.ops_runtime_executor_service._load_ssh_secret") + def test_execute_ssh_action_returns_graceful_error_when_paramiko_missing_for_password_auth( + self, + mock_load_secret, + ) -> None: + mock_load_secret.return_value = {"ssh_password": "demo-pass", "ssh_private_key": ""} + + with patch.object(ops_runtime_executor_service, "paramiko", None): + ok, message, data = execute_ssh_action( + { + "node_code": "mainland-controller-01", + "ssh_host": "121.204.244.188", + "ssh_port": 22, + "ssh_user": "root", + "auth_mode": "password", + }, + "health.snapshot", + {}, + ) + + self.assertFalse(ok) + self.assertIn("paramiko", message) + self.assertEqual("password", data["transport"]["auth_mode"]) + self.assertEqual("121.204.244.188", data["transport"]["ssh_host"]) + if __name__ == "__main__": unittest.main() diff --git a/domain-api/tests/test_runtime_status_service.py b/domain-api/tests/test_runtime_status_service.py new file mode 100644 index 0000000..172b22a --- /dev/null +++ b/domain-api/tests/test_runtime_status_service.py @@ -0,0 +1,131 @@ +import unittest + +from app.services.detect_service import _slice_remote_log_lines_fairly +from app.services.runtime_status_service import _align_queue_health_with_backlog, _build_detect_node_row + + +class RuntimeStatusServiceTests(unittest.TestCase): + def test_build_detect_node_row_merges_queue_running_metrics(self) -> None: + row = _build_detect_node_row( + node_code="mainland-worker-01", + cluster_node={ + "node_code": "mainland-worker-01", + "region": "mainland", + "role": "worker", + "status": "busy", + "is_effective_worker": True, + "current_load": 61, + "metadata": { + "job_items_total": 1000, + "job_items_claimed": 0, + "job_items_running": 61, + }, + }, + job_node={ + "node_code": "mainland-worker-01", + "items_total": 0, + "items_pending": 0, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_failed": 0, + }, + queue_node={ + "node_code": "mainland-worker-01", + "items_total": 1000, + "items_pending": 739, + "items_claimed": 200, + "items_running": 61, + "items_completed": 191, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 632, + "processed_per_minute": 42.13, + }, + ) + + self.assertEqual(1000, row["items_total"]) + self.assertEqual(200, row["items_claimed"]) + self.assertEqual(61, row["items_running"]) + self.assertEqual("running", row["participation_state"]) + self.assertTrue(row["is_current_participant"]) + self.assertTrue(row["is_dispatch_active"]) + + def test_build_detect_node_row_uses_active_threads_as_current_execution_signal(self) -> None: + row = _build_detect_node_row( + node_code="mainland-worker-01", + cluster_node={ + "node_code": "mainland-worker-01", + "region": "mainland", + "role": "worker", + "status": "busy", + "is_effective_worker": True, + "current_load": 61, + "metadata": { + "active_threads": 61, + "max_threads": 400, + }, + }, + job_node={}, + queue_node={ + "node_code": "mainland-worker-01", + "processed_recent": 632, + "processed_per_minute": 42.13, + }, + ) + + self.assertEqual(61, row["active_threads"]) + self.assertEqual(400, row["max_threads"]) + self.assertEqual("runtime_active", row["participation_state"]) + self.assertEqual("执行中", row["participation_label"]) + self.assertTrue(row["is_dispatch_active"]) + + def test_slice_remote_log_lines_fairly_keeps_secondary_node_visible(self) -> None: + lines = [f"[2026-04-19 17:30:{i:02d}] [mainland-controller-01] controller-{i}" for i in range(20)] + lines.extend( + [f"[2026-04-19 17:31:{i:02d}] [mainland-worker-01] worker-{i}" for i in range(2)] + ) + + sliced = _slice_remote_log_lines_fairly(lines, limit=6, min_per_node=2) + + self.assertEqual(6, len(sliced)) + self.assertTrue(any("[mainland-controller-01]" in line for line in sliced)) + self.assertTrue(any("[mainland-worker-01]" in line for line in sliced)) + + def test_align_queue_health_with_backlog_prefers_larger_runtime_snapshot(self) -> None: + aligned = _align_queue_health_with_backlog( + { + "has_active_job": True, + "queue": { + "items_total": 120, + "pending": 100, + "claimed": 10, + "running": 5, + "completed": 5, + "blacklisted": 0, + "failed": 0, + "terminal": 5, + }, + }, + { + "pending_total": 1200, + "claimed_total": 230, + "running_total": 40, + "completed_total": 300, + "blacklisted_total": 12, + "failed_total": 8, + }, + ) + + self.assertTrue(aligned["has_active_job"]) + self.assertEqual(1200, aligned["queue"]["pending"]) + self.assertEqual(230, aligned["queue"]["claimed"]) + self.assertEqual(40, aligned["queue"]["running"]) + self.assertEqual(300, aligned["queue"]["completed"]) + self.assertEqual(12, aligned["queue"]["blacklisted"]) + self.assertEqual(8, aligned["queue"]["failed"]) + self.assertEqual(1790, aligned["queue"]["items_total"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/domain-api/tests/test_settings_service.py b/domain-api/tests/test_settings_service.py new file mode 100644 index 0000000..7cb1a67 --- /dev/null +++ b/domain-api/tests/test_settings_service.py @@ -0,0 +1,31 @@ +import unittest + +from app.services.settings_service import _normalize_thread_count, resolve_thread_count + + +class SettingsServiceTests(unittest.TestCase): + def test_normalize_thread_count_allows_values_above_256(self) -> None: + self.assertEqual(512, _normalize_thread_count(512)) + + def test_normalize_thread_count_rejects_non_positive_values(self) -> None: + with self.assertRaises(ValueError): + _normalize_thread_count(0) + + def test_resolve_thread_count_uses_large_node_override(self) -> None: + payload = { + "thread_count": 8, + "node_thread_counts": { + "mainland-controller-01": 512, + }, + } + + resolved = resolve_thread_count(node_code="mainland-controller-01", settings_payload=payload) + + self.assertEqual(8, resolved["default_thread_count"]) + self.assertEqual(512, resolved["override_thread_count"]) + self.assertEqual(512, resolved["effective_thread_count"]) + self.assertEqual("node_override", resolved["source"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/domain-api/tests/test_sync_agent.py b/domain-api/tests/test_sync_agent.py index 82952a3..6ec1b24 100644 --- a/domain-api/tests/test_sync_agent.py +++ b/domain-api/tests/test_sync_agent.py @@ -1,7 +1,15 @@ import unittest from unittest.mock import patch -from app.sync_agent import _append_detect_result_projection_snapshot +from app.sync_agent import ( + _append_detect_result_projection_snapshot, + _build_aligned_queue_health_snapshot, + _emit_structured_tick, + _emit_sync_result_breakdown, + _filter_runtime_events_for_job, + _run_pipeline_stage_processor, + _select_projection_job_snapshots, +) class SyncAgentTests(unittest.TestCase): @@ -30,6 +38,155 @@ class SyncAgentTests(unittest.TestCase): self.assertEqual(1, payload["progress"]["blacklisted"]) self.assertEqual(2, payload["progress"]["failed"]) + @patch("app.sync_agent.get_latest_unprojected_detect_job_summary") + @patch("app.sync_agent.get_active_detect_job_summary") + def test_select_projection_job_snapshots_returns_active_and_latest_finished( + self, + mock_get_active_detect_job_summary, + mock_get_latest_unprojected_detect_job_summary, + ) -> None: + mock_get_active_detect_job_summary.return_value = {"job_id": 101, "job_code": "running-job"} + mock_get_latest_unprojected_detect_job_summary.return_value = {"job_id": 102, "job_code": "finished-job"} + + snapshots = _select_projection_job_snapshots() + + self.assertEqual(["running-job", "finished-job"], [item["job_code"] for item in snapshots]) + + @patch("app.sync_agent.get_latest_unprojected_detect_job_summary") + @patch("app.sync_agent.get_active_detect_job_summary") + def test_select_projection_job_snapshots_deduplicates_same_job( + self, + mock_get_active_detect_job_summary, + mock_get_latest_unprojected_detect_job_summary, + ) -> None: + mock_get_active_detect_job_summary.return_value = {"job_id": 101, "job_code": "same-job"} + mock_get_latest_unprojected_detect_job_summary.return_value = {"job_id": 101, "job_code": "same-job"} + + snapshots = _select_projection_job_snapshots() + + self.assertEqual(1, len(snapshots)) + self.assertEqual("same-job", snapshots[0]["job_code"]) + + def test_build_aligned_queue_health_snapshot_overrides_mismatched_job(self) -> None: + active_job = { + "job_id": 1891, + "job_code": "sync-overseas-31230", + "status": "running", + "progress_percent": 75.9, + "items_total": 1000, + "items_pending": 155, + "items_claimed": 45, + "items_running": 41, + "items_completed": 759, + "items_blacklisted": 0, + "items_failed": 0, + "node_stats": [ + { + "node_code": "mainland-worker-01", + "items_total": 710, + "items_claimed": 45, + "items_running": 41, + "items_completed": 624, + } + ], + } + queue_health = { + "job": {"job_code": "sync-overseas-31273"}, + "queue": {"items_total": 1000, "pending": 838, "claimed": 162, "running": 0, "completed": 0}, + "nodes": [{"node_code": "mainland-controller-01", "items_total": 162}], + } + + data = _build_aligned_queue_health_snapshot(active_job, queue_health) + + self.assertEqual("sync-overseas-31230", data["job"]["job_code"]) + self.assertEqual(155, data["queue"]["pending"]) + self.assertEqual(45, data["queue"]["claimed"]) + self.assertEqual(41, data["queue"]["running"]) + self.assertEqual(759, data["queue"]["completed"]) + self.assertEqual("mainland-worker-01", data["nodes"][0]["node_code"]) + self.assertEqual("unassigned", data["nodes"][-1]["node_code"]) + + @patch("app.sync_agent.push_debug_event") + def test_emit_sync_result_breakdown_marks_idle_projection_as_idle(self, mock_push_debug_event) -> None: + _emit_sync_result_breakdown( + { + "results": [ + { + "sync_type": "detect_result_projection", + "ok": False, + "message": "当前没有可推送的detect_result_projection", + "data": {}, + } + ] + } + ) + + mock_push_debug_event.assert_called_once() + self.assertEqual("detect_result_projection_sync_idle", mock_push_debug_event.call_args.kwargs["event_type"]) + self.assertEqual("info", mock_push_debug_event.call_args.kwargs["level"]) + + @patch("app.sync_agent.push_debug_event") + def test_emit_structured_tick_marks_idle_sync_push_as_idle(self, mock_push_debug_event) -> None: + _emit_structured_tick( + base_event_type="sync_push", + ok=False, + message="当前没有需要立即推送的同步投影", + data={"results": []}, + ) + + mock_push_debug_event.assert_called_once() + self.assertEqual("sync_push_idle", mock_push_debug_event.call_args.kwargs["event_type"]) + self.assertEqual("info", mock_push_debug_event.call_args.kwargs["level"]) + + def test_filter_runtime_events_for_job_discards_mismatched_jobs(self) -> None: + events = _filter_runtime_events_for_job( + [ + { + "job_id": 2137, + "node_code": "mainland-controller-01", + "event_type": "job_created", + "message": "new job", + "payload": {"source_record_id": 35461}, + "created_at": "2026-04-22 01:21:29", + }, + { + "job_id": 1902, + "node_code": "mainland-controller-01", + "event_type": "worker_log", + "message": "claim batch", + "payload": {"job_code": "sync-overseas-31437"}, + "created_at": "2026-04-22 01:19:25", + }, + ], + job_code="sync-overseas-31437", + job_id=1902, + limit=8, + ) + + self.assertEqual(1, len(events)) + self.assertEqual("worker_log", events[0]["event_type"]) + + @patch("app.sync_agent.push_debug_event") + @patch("app.sync_agent.process_detect_pipeline_now") + def test_run_pipeline_stage_processor_uses_high_default_limit( + self, + mock_process_detect_pipeline_now, + mock_push_debug_event, + ) -> None: + mock_process_detect_pipeline_now.return_value = ( + True, + "pipeline ok", + {"processed_items": 512}, + ) + + ok, message, data = _run_pipeline_stage_processor() + + self.assertTrue(ok) + self.assertEqual("pipeline ok", message) + self.assertEqual({"processed_items": 512}, data) + mock_process_detect_pipeline_now.assert_called_once_with(limit=5000) + self.assertEqual("pipeline_tick_success", mock_push_debug_event.call_args.kwargs["event_type"]) + if __name__ == "__main__": unittest.main() diff --git a/domain-api/tests/test_sync_push_service.py b/domain-api/tests/test_sync_push_service.py index d1b0ae8..0565fd2 100644 --- a/domain-api/tests/test_sync_push_service.py +++ b/domain-api/tests/test_sync_push_service.py @@ -2,7 +2,10 @@ import unittest from unittest.mock import patch from app.services.sync_push_service import ( + _build_task_pull_backlog_limits, _extract_detect_result_projection_events, + _resolve_detect_result_target_job_id, + _should_throttle_task_pull, ingest_runtime_projection, ) @@ -46,6 +49,61 @@ class _FakeConnection: class SyncPushServiceTests(unittest.TestCase): + def test_build_task_pull_backlog_limits_scales_with_thread_configuration(self) -> None: + limits = _build_task_pull_backlog_limits( + 5000, + settings_payload={ + "thread_count": 100, + "node_thread_counts": { + "mainland-controller-01": 2000, + "mainland-worker-01": 1200, + }, + }, + ) + + self.assertEqual(3200, limits["estimated_total_threads"]) + self.assertEqual(6400, limits["max_pending_total"]) + self.assertEqual(3200, limits["max_register_pending"]) + self.assertEqual(800, limits["max_downstream_pending"]) + + def test_should_throttle_task_pull_when_register_backlog_overwhelms_downstream(self) -> None: + should_throttle, reason = _should_throttle_task_pull( + { + "pending_total": 9438, + "claimed_total": 410, + "running_total": 15, + "register_pending": 8487, + "downstream_pending": 951, + }, + { + "max_pending_total": 12000, + "max_register_pending": 3200, + "max_downstream_pending": 800, + }, + ) + + self.assertTrue(should_throttle) + self.assertEqual("register_pending", reason) + + def test_should_not_throttle_task_pull_when_backlog_is_within_limits(self) -> None: + should_throttle, reason = _should_throttle_task_pull( + { + "pending_total": 1800, + "claimed_total": 100, + "running_total": 300, + "register_pending": 600, + "downstream_pending": 120, + }, + { + "max_pending_total": 6400, + "max_register_pending": 3200, + "max_downstream_pending": 800, + }, + ) + + self.assertFalse(should_throttle) + self.assertEqual("", reason) + def test_extract_detect_result_projection_events_adds_import_metadata(self) -> None: projection = { "job": { @@ -79,6 +137,43 @@ class SyncPushServiceTests(unittest.TestCase): self.assertTrue(event["payload"]["imported_from_projection"]) self.assertTrue(event["payload"]["import_fingerprint"]) + @patch("app.services.sync_push_service.get_db") + def test_resolve_detect_result_target_job_id_prefers_matching_job_code(self, mock_get_db) -> None: + fake_conn = _FakeConnection(rows=[(456,)]) + mock_get_db.return_value = fake_conn + + target_job_id = _resolve_detect_result_target_job_id( + projection={ + "job": { + "job_code": "sync-mainland-123", + } + } + ) + + self.assertEqual(456, target_job_id) + self.assertIn("WHERE job_code = %s", fake_conn.cursor_obj.executed[0][0]) + + @patch("app.services.detect_job_service.get_active_detect_job_summary") + @patch("app.services.sync_push_service.get_db") + def test_resolve_detect_result_target_job_id_falls_back_to_active_job( + self, + mock_get_db, + mock_get_active_detect_job_summary, + ) -> None: + fake_conn = _FakeConnection(rows=[None]) + mock_get_db.return_value = fake_conn + mock_get_active_detect_job_summary.return_value = {"job_id": 789} + + target_job_id = _resolve_detect_result_target_job_id( + projection={ + "job": { + "job_code": "sync-mainland-999", + } + } + ) + + self.assertEqual(789, target_job_id) + @patch("app.services.sync_push_service._import_detect_result_projection_events") @patch("app.services.sync_push_service.get_db") def test_ingest_runtime_projection_imports_detect_result_events_on_new_record( diff --git a/domain-api/tests/test_sync_record_service.py b/domain-api/tests/test_sync_record_service.py index c1dad47..fb4e4c3 100644 --- a/domain-api/tests/test_sync_record_service.py +++ b/domain-api/tests/test_sync_record_service.py @@ -1,6 +1,10 @@ import unittest +from unittest.mock import patch -from app.services.sync_record_service import _collect_recent_domain_events +from app.services.sync_record_service import ( + _build_runtime_projection_payload, + _collect_recent_domain_events, +) class SyncRecordServiceTests(unittest.TestCase): @@ -50,6 +54,82 @@ class SyncRecordServiceTests(unittest.TestCase): self.assertEqual("domain_completed", events[1]["event_type"]) self.assertEqual("a.com", events[1]["payload"]["domain"]) + @patch("app.services.sync_record_service._resolve_local_ip", return_value="152.53.37.118") + @patch("app.services.sync_record_service.socket.gethostname", return_value="overseas-control-01") + @patch("app.services.sync_record_service.settings") + def test_build_runtime_projection_payload_zeros_execution_for_overseas_control( + self, + mock_settings, + _mock_hostname, + _mock_resolve_ip, + ) -> None: + mock_settings.node_code = "overseas-control-01" + mock_settings.node_region = "overseas" + mock_settings.node_role = "control" + mock_settings.sync_source_region = "overseas" + mock_settings.sync_target_region = "mainland" + + payload = _build_runtime_projection_payload( + detect={ + "worker_online": False, + "worker_mode": "linux-systemd", + "active_thread_count": 0, + "max_thread_count": 0, + "phase_label": "当前节点不承载", + "phase_detail": "当前节点为海外控制面,仅承载 API 控制与同步接收,不执行本机检测任务。", + "proxy_runtime_label": "不适用", + "proxy_runtime_reason": "not_applicable", + "progress": { + "pending": 125, + "running": 293, + "completed": 71, + "blacklisted": 0, + "failed": 1, + }, + "active_job": { + "job_id": 2027, + "job_code": "sync-overseas-33501", + "status": "running", + "progress_percent": 7.2, + "items_total": 1000, + "items_terminal": 72, + "items_pending": 125, + "items_running": 293, + "items_failed": 1, + "node_stats": [{"node_code": "mainland-controller-01", "items_running": 293}], + }, + }, + cluster={ + "nodes_total": 3, + "nodes": [ + { + "node_code": "overseas-control-01", + "current_load": 293, + "detect_participating": True, + } + ], + "summary": { + "online_worker_nodes": 3, + "dedicated_online_worker_nodes": 1, + "online_control_nodes": 2, + "busy_nodes": ["overseas-control-01"], + "stale_nodes": [], + "offline_nodes": [], + }, + }, + source_region="overseas", + target_region="mainland", + ) + + projection = payload["projection"] + self.assertFalse(projection["worker_online"]) + self.assertFalse(projection["detect_participating"]) + self.assertEqual(0, projection["active_thread_count"]) + self.assertEqual(0, projection["max_thread_count"]) + self.assertEqual(0, projection["progress"]["running"]) + self.assertEqual("", projection["active_job"]["job_code"]) + self.assertEqual([], projection["active_job"]["node_stats"]) + if __name__ == "__main__": unittest.main() diff --git a/domain-web/src/layouts/MainLayout.vue b/domain-web/src/layouts/MainLayout.vue index 38155c1..59efe8d 100644 --- a/domain-web/src/layouts/MainLayout.vue +++ b/domain-web/src/layouts/MainLayout.vue @@ -16,7 +16,7 @@

{{ title }}

-

当前保持桌面版不变,Web 管理后台进入可持续运维阶段。

+

{{ pageDescription }}

@@ -68,6 +68,7 @@ const menuItems = [ { path: "/runtime", label: "运行中心" }, { path: "/runtime-debug", label: "运行调试" }, { path: "/ops-center", label: "运维中枢" }, + { path: "/managed-nodes", label: "托管节点" }, { path: "/settings", label: "系统设置" }, { path: "/imports", label: "域名导入" }, { path: "/sensitive-words", label: "敏感词配置" }, @@ -79,6 +80,13 @@ const menuItems = [ ]; const title = computed(() => String(route.meta?.title || "概览")); +const pageDescription = computed( + () => + String( + route.meta?.description || + "当前保持桌面版不变,Web 管理后台进入可持续运维阶段。" + ) +); const loadRuntimeSummary = async () => { try { @@ -258,7 +266,9 @@ onBeforeUnmount(() => { flex: 1; min-height: 0; padding: 0 28px 28px; - overflow: hidden; + overflow-x: hidden; + overflow-y: auto; + -webkit-overflow-scrolling: touch; } @media (max-width: 960px) { diff --git a/domain-web/src/router/index.ts b/domain-web/src/router/index.ts index 5d44774..83f9da6 100644 --- a/domain-web/src/router/index.ts +++ b/domain-web/src/router/index.ts @@ -13,18 +13,19 @@ const routes: RouteRecordRaw[] = [ component: () => import("@/layouts/MainLayout.vue"), children: [ { path: "", redirect: "/dashboard" }, - { path: "dashboard", name: "dashboard", component: () => import("@/views/dashboard/DashboardView.vue"), meta: { title: "概览" } }, - { path: "runtime", name: "runtime", component: () => import("@/views/runtime/RuntimeView.vue"), meta: { title: "运行中心" } }, - { path: "runtime-debug", name: "runtime-debug", component: () => import("@/views/runtime/RuntimeDebugView.vue"), meta: { title: "运行调试" } }, - { path: "ops-center", name: "ops-center", component: () => import("@/views/ops/OpsCenterView.vue"), meta: { title: "运维中枢" } }, - { path: "settings", name: "settings", component: () => import("@/views/settings/SettingsView.vue"), meta: { title: "系统设置" } }, - { path: "imports", name: "imports", component: () => import("@/views/imports/ImportsView.vue"), meta: { title: "域名导入" } }, - { path: "sensitive-words", name: "sensitive-words", component: () => import("@/views/sensitive-words/SensitiveWordsView.vue"), meta: { title: "敏感词配置" } }, - { path: "juming", name: "juming", component: () => import("@/views/juming/JumingView.vue"), meta: { title: "聚名采集" } }, - { path: "detect", name: "detect", component: () => import("@/views/detect/DetectView.vue"), meta: { title: "检测控制" } }, - { path: "domains", name: "domains", component: () => import("@/views/domains/DomainsView.vue"), meta: { title: "域名筛选" } }, - { path: "exports", name: "exports", component: () => import("@/views/exports/ExportsView.vue"), meta: { title: "导出中心" } }, - { path: "logs", name: "logs", component: () => import("@/views/logs/LogsView.vue"), meta: { title: "日志诊断" } } + { path: "dashboard", name: "dashboard", component: () => import("@/views/dashboard/DashboardView.vue"), meta: { title: "概览", description: "统一查看当前任务、吞吐、瓶颈步骤和整体运行态。" } }, + { path: "runtime", name: "runtime", component: () => import("@/views/runtime/RuntimeView.vue"), meta: { title: "运行中心", description: "看检测任务有没有真正跑起来,重点盯 running、ppm、步骤队列和节点负载。" } }, + { path: "runtime-debug", name: "runtime-debug", component: () => import("@/views/runtime/RuntimeDebugView.vue"), meta: { title: "运行调试", description: "排查步骤卡点、事件流、调试日志和运行时异常。" } }, + { path: "ops-center", name: "ops-center", component: () => import("@/views/ops/OpsCenterView.vue"), meta: { title: "运维中枢", description: "这里是总驾驶舱,负责发布、运维任务、巡检、接管进度和整体运维视角。" } }, + { path: "managed-nodes", name: "managed-nodes", component: () => import("@/views/ops/ManagedNodesView.vue"), meta: { title: "托管节点", description: "这里专门管理机器本身:新增节点、填 SSH、启用停用、维护接入信息。" } }, + { path: "settings", name: "settings", component: () => import("@/views/settings/SettingsView.vue"), meta: { title: "系统设置", description: "维护系统基础配置、运行参数和默认行为。" } }, + { path: "imports", name: "imports", component: () => import("@/views/imports/ImportsView.vue"), meta: { title: "域名导入", description: "导入待检测域名数据,建立任务源。" } }, + { path: "sensitive-words", name: "sensitive-words", component: () => import("@/views/sensitive-words/SensitiveWordsView.vue"), meta: { title: "敏感词配置", description: "维护过滤词、黑名单词和筛选规则。" } }, + { path: "juming", name: "juming", component: () => import("@/views/juming/JumingView.vue"), meta: { title: "聚名采集", description: "抓取聚名删除列表,跟踪采集、去重和入库进度。" } }, + { path: "detect", name: "detect", component: () => import("@/views/detect/DetectView.vue"), meta: { title: "检测控制", description: "启动检测、观察步骤推进,并排查并发和补位问题。" } }, + { path: "domains", name: "domains", component: () => import("@/views/domains/DomainsView.vue"), meta: { title: "域名筛选", description: "查看域名状态、筛选结果和流程产出。" } }, + { path: "exports", name: "exports", component: () => import("@/views/exports/ExportsView.vue"), meta: { title: "导出中心", description: "管理导出任务和结果文件。" } }, + { path: "logs", name: "logs", component: () => import("@/views/logs/LogsView.vue"), meta: { title: "日志诊断", description: "集中查看系统日志、错误信息和诊断输出。" } } ] } ]; diff --git a/domain-web/src/views/dashboard/DashboardView.vue b/domain-web/src/views/dashboard/DashboardView.vue index bba85b5..6441b06 100644 --- a/domain-web/src/views/dashboard/DashboardView.vue +++ b/domain-web/src/views/dashboard/DashboardView.vue @@ -24,18 +24,107 @@
- - - 已创建 Web 后台、API、运行中心和导入/导出/筛选主链路。 - 继续补 Linux Worker 控制闭环和更完整的运行诊断。 - 把检测端逐步迁为 Linux Worker,前后端彻底分离。 - + +
+
+
当前任务
+
{{ activeJob.job_code || "-" }}
+
+ {{ + activeJob.job_code + ? `状态 ${activeJob.status || "-"} / 当前任务总数 ${activeJob.items_total || 0} / 当前累计活跃任务 ${opsSummary.active_jobs_total || 0}` + : "当前没有活跃检测任务" + }} +
+
+
+
当前速度
+
{{ processedPerMinuteText }}
+
近窗完成 {{ opsSummary.completed_recent || 0 }} / 失败 {{ opsSummary.failed_recent || 0 }} / 黑名单 {{ opsSummary.blacklisted_recent || 0 }}
+
+
+
预计剩余
+
{{ etaText }}
+
按当前吞吐推算剩余 {{ opsSummary.remaining_items || 0 }} 项
+
+
+
瓶颈步骤
+
{{ bottleneckStep.step_name || "-" }}
+
待处理 {{ bottleneckStep.items_pending || 0 }} / 执行中 {{ bottleneckStep.items_running || 0 }}
+
+
+
真实积压
+
{{ backlogPendingText }}
+
注册待处理 {{ backlogRegisterPendingText }} / 后续步骤 {{ backlogDownstreamPendingText }}
+
+
+
有效执行节点
+
{{ opsSummary.active_execution_nodes || 0 }}
+
在线 Worker {{ opsSummary.online_worker_nodes || 0 }} / 独立 Worker {{ opsSummary.dedicated_online_worker_nodes || 0 }} / 当前线程负载 {{ queueDisplayRunningText }}
+
+
+ + + + + + + + + + + + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/domain-web/src/views/ops/OpsCenterView.vue b/domain-web/src/views/ops/OpsCenterView.vue index e50a35d..d2f8a91 100644 --- a/domain-web/src/views/ops/OpsCenterView.vue +++ b/domain-web/src/views/ops/OpsCenterView.vue @@ -6168,6 +6168,12 @@ + + + @@ -6183,6 +6189,38 @@ + + +
+ 当前状态:{{ managedNodeForm.ssh_password_configured && !managedNodeForm.clear_ssh_password ? "已保存密码" : "未保存密码" }} +
+ +
+ + +
+ 当前状态:{{ managedNodeForm.ssh_private_key_configured && !managedNodeForm.clear_ssh_private_key ? "已保存私钥" : "未保存私钥" }} +
+ +
@@ -6652,10 +6690,17 @@ const managedNodeForm = ref({ title: "", region: "mainland", role: "worker", + ssh_entry: "", ssh_host: "", ssh_port: 22, ssh_user: "root", auth_mode: "key", + ssh_password: "", + ssh_private_key: "", + ssh_password_configured: false, + ssh_private_key_configured: false, + clear_ssh_password: false, + clear_ssh_private_key: false, deploy_channel: "stable", is_enabled: true, metadata: {} as GenericRecord @@ -12712,10 +12757,17 @@ const resetManagedNodeForm = () => { title: "", region: "mainland", role: "worker", + ssh_entry: "", ssh_host: "", ssh_port: 22, ssh_user: "root", auth_mode: "key", + ssh_password: "", + ssh_private_key: "", + ssh_password_configured: false, + ssh_private_key_configured: false, + clear_ssh_password: false, + clear_ssh_private_key: false, deploy_channel: "stable", is_enabled: true, metadata: {} @@ -13696,10 +13748,17 @@ const openManagedNodeDialog = ( title: String(row.title || row.node_code || ""), region: String(row.region || "mainland"), role: String(row.role || "worker"), + ssh_entry: "", ssh_host: String(row.ssh_host || row.cluster_ip || row.agent_ip || row.metadata?.ip || ""), ssh_port: Number(row.ssh_port || 22) || 22, ssh_user: String(row.ssh_user || "root"), auth_mode: String(row.auth_mode || "key"), + ssh_password: "", + ssh_private_key: "", + ssh_password_configured: Boolean(row.ssh_password_configured), + ssh_private_key_configured: Boolean(row.ssh_private_key_configured), + clear_ssh_password: false, + clear_ssh_private_key: false, deploy_channel: String(row.deploy_channel || "stable"), is_enabled: Boolean(row.is_managed ? row.is_enabled : true), metadata: { @@ -13866,10 +13925,15 @@ const saveManagedNode = async () => { title: managedNodeForm.value.title.trim() || managedNodeForm.value.node_code.trim(), region: managedNodeForm.value.region, role: managedNodeForm.value.role, + ssh_entry: managedNodeForm.value.ssh_entry.trim(), ssh_host: managedNodeForm.value.ssh_host.trim(), ssh_port: managedNodeForm.value.ssh_port, ssh_user: managedNodeForm.value.ssh_user.trim(), auth_mode: managedNodeForm.value.auth_mode, + ssh_password: managedNodeForm.value.auth_mode === "password" ? managedNodeForm.value.ssh_password : "", + ssh_private_key: managedNodeForm.value.auth_mode === "key" ? managedNodeForm.value.ssh_private_key : "", + clear_ssh_password: Boolean(managedNodeForm.value.clear_ssh_password), + clear_ssh_private_key: Boolean(managedNodeForm.value.clear_ssh_private_key), deploy_channel: managedNodeForm.value.deploy_channel, is_enabled: managedNodeForm.value.is_enabled, metadata: { @@ -13885,6 +13949,14 @@ const saveManagedNode = async () => { ssh_port: managedNodeForm.value.ssh_port, ssh_user: managedNodeForm.value.ssh_user.trim(), auth_mode: managedNodeForm.value.auth_mode, + ssh_password_configured: + managedNodeForm.value.auth_mode === "password" + ? Boolean(managedNodeForm.value.ssh_password || (managedNodeForm.value.ssh_password_configured && !managedNodeForm.value.clear_ssh_password)) + : false, + ssh_private_key_configured: + managedNodeForm.value.auth_mode === "key" + ? Boolean(managedNodeForm.value.ssh_private_key || (managedNodeForm.value.ssh_private_key_configured && !managedNodeForm.value.clear_ssh_private_key)) + : false, deploy_channel: managedNodeForm.value.deploy_channel, is_enabled: managedNodeForm.value.is_enabled, metadata: { ...(managedNodeForm.value.metadata || {}) }, diff --git a/domain-web/src/views/runtime/RuntimeView.vue b/domain-web/src/views/runtime/RuntimeView.vue index f133f7a..7cfcf04 100644 --- a/domain-web/src/views/runtime/RuntimeView.vue +++ b/domain-web/src/views/runtime/RuntimeView.vue @@ -750,7 +750,7 @@ - + - + @@ -27,7 +27,7 @@
- + 删除
diff --git a/domainCheck/app/config.py b/domainCheck/app/config.py index 13993a6..b68134a 100644 --- a/domainCheck/app/config.py +++ b/domainCheck/app/config.py @@ -34,14 +34,56 @@ else: # 开发环境目录 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -# 加载环境变量 -env_path = os.path.join(BASE_DIR, '.env') -if os.path.exists(env_path): - load_dotenv(env_path) - _safe_echo(f"成功加载环境变量文件: {env_path}") -else: + +def _detect_install_root(base_dir): + normalized = os.path.abspath(str(base_dir or "")) + marker = f"{os.sep}releases{os.sep}" + if marker in normalized: + return normalized.split(marker, 1)[0] + return "" + + +def _resolve_runtime_root(base_dir): + explicit_root = str(os.getenv('DOMAINCHECK_RUNTIME_ROOT', '') or '').strip() + if explicit_root: + return os.path.abspath(explicit_root) + install_root = _detect_install_root(base_dir) + if install_root: + return os.path.join(install_root, 'runtime', 'domainCheck') + return os.path.abspath(base_dir) + + +def _load_environment(base_dir): + install_root = _detect_install_root(base_dir) + runtime_root = _resolve_runtime_root(base_dir) + candidate_paths = [ + str(os.getenv('DOMAINCHECK_ENV_FILE', '') or '').strip(), + os.path.join(base_dir, '.env'), + os.path.join(runtime_root, '.env'), + os.path.join(os.getcwd(), '.env'), + ] + if install_root: + candidate_paths.append(os.path.join(install_root, 'domainCheck', '.env')) + + seen = set() + for path in candidate_paths: + normalized = os.path.abspath(path) if path else '' + if not normalized or normalized in seen: + continue + seen.add(normalized) + if os.path.exists(normalized): + load_dotenv(normalized) + _safe_echo(f"成功加载环境变量文件: {normalized}") + return normalized + load_dotenv() - _safe_echo(f"环境变量文件不存在: {env_path},使用默认环境变量") + fallback_path = os.path.join(base_dir, '.env') + _safe_echo(f"环境变量文件不存在: {fallback_path},使用默认环境变量") + return "" + +# 加载环境变量 +env_path = _load_environment(BASE_DIR) +RUNTIME_ROOT = _resolve_runtime_root(BASE_DIR) class Config: @@ -55,7 +97,11 @@ class Config: DB_DATABASE = os.getenv('DB_DATABASE', 'domain_scan_db') DB_USER = os.getenv('DB_USER', 'postgres') DB_PASSWORD = os.getenv('DB_PASSWORD', 'postgres') - DB_POOL_SIZE = int(os.getenv('DB_POOL_SIZE', 5)) + DB_POOL_SIZE = int(os.getenv('DB_POOL_SIZE', 64)) + DB_POOL_WARM_SIZE = int(os.getenv('DB_POOL_WARM_SIZE', 8)) + DB_POOL_IDLE_KEEP_MAX = int(os.getenv('DB_POOL_IDLE_KEEP_MAX', DB_POOL_SIZE)) + DB_POOL_HEALTHCHECK_INTERVAL = float(os.getenv('DB_POOL_HEALTHCHECK_INTERVAL', 30)) + DB_POOL_ACQUIRE_TIMEOUT = float(os.getenv('DB_POOL_ACQUIRE_TIMEOUT', 20)) # 消息队列配置 RABBITMQ_HOST = os.getenv('RABBITMQ_HOST', 'localhost') @@ -86,12 +132,14 @@ class Config: DETECT_TIMEOUT = int(os.getenv('DETECT_TIMEOUT', 30)) DETECT_RETRY_COUNT = int(os.getenv('DETECT_RETRY_COUNT', 3)) DETECT_CONCURRENCY = int(os.getenv('DETECT_CONCURRENCY', 10)) - WAYBACK_CDX_TIMEOUT = int(os.getenv('WAYBACK_CDX_TIMEOUT', 15)) - WAYBACK_SNAPSHOT_TIMEOUT = int(os.getenv('WAYBACK_SNAPSHOT_TIMEOUT', 12)) - WAYBACK_RETRY_COUNT = int(os.getenv('WAYBACK_RETRY_COUNT', 2)) + WAYBACK_CDX_TIMEOUT = int(os.getenv('WAYBACK_CDX_TIMEOUT', 2)) + WAYBACK_SNAPSHOT_TIMEOUT = int(os.getenv('WAYBACK_SNAPSHOT_TIMEOUT', 2)) + WAYBACK_RETRY_COUNT = int(os.getenv('WAYBACK_RETRY_COUNT', 0)) WAYBACK_REQUEST_DELAY = float(os.getenv('WAYBACK_REQUEST_DELAY', 0)) WAYBACK_PROGRESS_INTERVAL = int(os.getenv('WAYBACK_PROGRESS_INTERVAL', 500)) - WAYBACK_DOMAIN_CONCURRENCY = int(os.getenv('WAYBACK_DOMAIN_CONCURRENCY', 3)) + WAYBACK_DOMAIN_CONCURRENCY = int(os.getenv('WAYBACK_DOMAIN_CONCURRENCY', 2)) + WAYBACK_MAX_RECORDS = int(os.getenv('WAYBACK_MAX_RECORDS', 4)) + WAYBACK_TRANSIENT_BACKOFF_SECONDS = float(os.getenv('WAYBACK_TRANSIENT_BACKOFF_SECONDS', 8)) WAYBACK_TITLE_MAX_BYTES = int(os.getenv('WAYBACK_TITLE_MAX_BYTES', 65536)) WAYBACK_TIMESTAMP_CACHE_TTL = int(os.getenv('WAYBACK_TIMESTAMP_CACHE_TTL', 86400)) WAYBACK_TITLE_CACHE_TTL = int(os.getenv('WAYBACK_TITLE_CACHE_TTL', 2592000)) @@ -139,8 +187,9 @@ class Config: # 目录配置 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) - DATA_DIR = os.path.join(BASE_DIR, 'data') - LOG_DIR = os.path.join(BASE_DIR, 'logs') + RUNTIME_ROOT = RUNTIME_ROOT + DATA_DIR = os.path.join(RUNTIME_ROOT, 'data') + LOG_DIR = os.path.join(RUNTIME_ROOT, 'logs') # 确保目录存在 os.makedirs(DATA_DIR, exist_ok=True) diff --git a/domainCheck/app/detectors/wayback_detector.py b/domainCheck/app/detectors/wayback_detector.py index c95c067..561f042 100644 --- a/domainCheck/app/detectors/wayback_detector.py +++ b/domainCheck/app/detectors/wayback_detector.py @@ -32,6 +32,8 @@ class WaybackDetector(BaseDetector): """ TITLE_PATTERN = re.compile(r']*>(.*?)', re.IGNORECASE | re.DOTALL) + _transient_backoff_until = 0.0 + _transient_backoff_lock = threading.Lock() def __init__(self): """ @@ -56,7 +58,7 @@ class WaybackDetector(BaseDetector): allowed_methods=frozenset(["GET"]), raise_on_status=False, ) - adapter = HTTPAdapter(max_retries=retry, pool_connections=10, pool_maxsize=10) + adapter = HTTPAdapter(max_retries=retry, pool_connections=128, pool_maxsize=256) session.mount('http://', adapter) session.mount('https://', adapter) session.headers.update({ @@ -65,6 +67,43 @@ class WaybackDetector(BaseDetector): }) return session + @staticmethod + def _resolve_timeout(total_timeout, connect_cap=3): + total = max(1.0, float(total_timeout or 0)) + return (min(float(connect_cap), total), total) + + @staticmethod + def _is_transient_request_error(message): + lowered = str(message or '').lower() + keywords = ( + 'timeout', + 'timed out', + 'connecttimeout', + 'read timeout', + 'connection reset', + 'connection refused', + 'connection aborted', + 'name or service not known', + 'temporary failure', + 'max retries exceeded', + 'ssl', + ) + return any(keyword in lowered for keyword in keywords) + + @classmethod + def _transient_backoff_remaining_seconds(cls): + with cls._transient_backoff_lock: + remaining = float(cls._transient_backoff_until or 0) - time.time() + return max(0.0, remaining) + + @classmethod + def _trip_transient_backoff(cls): + backoff_seconds = max(0.0, float(getattr(config, "WAYBACK_TRANSIENT_BACKOFF_SECONDS", 0) or 0)) + if backoff_seconds <= 0: + return + with cls._transient_backoff_lock: + cls._transient_backoff_until = max(float(cls._transient_backoff_until or 0), time.time() + backoff_seconds) + def _build_redis_client(self): try: client = redis.Redis( @@ -237,7 +276,7 @@ class WaybackDetector(BaseDetector): response = self.session.get( self.cdx_api_url, params=params, - timeout=config.WAYBACK_CDX_TIMEOUT, + timeout=self._resolve_timeout(config.WAYBACK_CDX_TIMEOUT), stream=True, ) if response.status_code != 200: @@ -328,17 +367,27 @@ class WaybackDetector(BaseDetector): try: response = self.session.get( snapshot_url, - timeout=config.WAYBACK_SNAPSHOT_TIMEOUT, + timeout=self._resolve_timeout(config.WAYBACK_SNAPSHOT_TIMEOUT), stream=True, ) if response.status_code != 200: - data = {'timestamp': timestamp, 'title': '', 'ok': False} + data = { + 'timestamp': timestamp, + 'title': '', + 'ok': False, + 'error': f'HTTP {response.status_code}', + } self._save_cached_title(domain, timestamp, data) return data content_type = (response.headers.get('Content-Type') or '').lower() if content_type and 'text/html' not in content_type and 'application/xhtml+xml' not in content_type: - data = {'timestamp': timestamp, 'title': '', 'ok': False} + data = { + 'timestamp': timestamp, + 'title': '', + 'ok': False, + 'error': f'content_type:{content_type}', + } self._save_cached_title(domain, timestamp, data) return data @@ -359,11 +408,11 @@ class WaybackDetector(BaseDetector): content = ''.join(chunks) title = self._extract_title(content) if found_title or content else '' - data = {'timestamp': timestamp, 'title': title, 'ok': True} + data = {'timestamp': timestamp, 'title': title, 'ok': True, 'error': ''} self._save_cached_title(domain, timestamp, data) return data - except Exception: - data = {'timestamp': timestamp, 'title': '', 'ok': False} + except Exception as exc: + data = {'timestamp': timestamp, 'title': '', 'ok': False, 'error': str(exc)} self._save_cached_title(domain, timestamp, data) return data finally: @@ -404,15 +453,83 @@ class WaybackDetector(BaseDetector): self._handle_exception(e, domain) return False - def scan_snapshots(self, domain, sensitive_words=None, stop_on_first_hit=True): + @staticmethod + def _resolve_recent_year_cutoff(recent_years): + try: + normalized_recent_years = int(recent_years or 0) + except (TypeError, ValueError): + return None + if normalized_recent_years <= 0: + return None + return time.gmtime().tm_year - normalized_recent_years + 1 + + def _filter_records_recent_years(self, records, recent_years=None): + cutoff_year = self._resolve_recent_year_cutoff(recent_years) + if cutoff_year is None: + return list(records or []) + filtered = [] + for item in list(records or []): + timestamp = str((item or {}).get('timestamp') or '').strip() + if len(timestamp) < 4: + continue + try: + year = int(timestamp[:4]) + except (TypeError, ValueError): + continue + if year >= cutoff_year: + filtered.append(item) + return filtered + + def _limit_records_for_scan(self, records): + max_records = max(1, int(getattr(config, "WAYBACK_MAX_RECORDS", 8) or 8)) + trimmed = [] + for item in list(records or []): + if len(trimmed) >= max_records: + break + trimmed.append(item) + return trimmed + + def scan_snapshots(self, domain, sensitive_words=None, stop_on_first_hit=True, recent_years=None): sensitive_words = sensitive_words or config.load_sensitive_words() request_errors = [] + transient_backoff_remaining = self._transient_backoff_remaining_seconds() + if transient_backoff_remaining > 0: + return { + 'snapshot_years': [], + 'has_sensitive_content': False, + 'matched_word': None, + 'matched_timestamp': None, + 'matched_title': None, + 'backlink_count': 0, + 'backlink_count_gt_10': False, + 'checked_snapshot_count': 0, + 'fetched_snapshot_count': 0, + 'failed_snapshot_count': 1, + 'unique_title_count': 0, + 'duplicate_title_skipped': 0, + 'digest_duplicate_skipped': 0, + 'request_error_count': 1, + 'request_errors': [f"wayback_backoff_active:{round(transient_backoff_remaining, 2)}s"], + 'elapsed_seconds': 0.0, + } latest_fetch = self._fetch_cdx_records_with_meta(domain, limit=-1, fast_latest=True) latest_record = (latest_fetch.get('records') or [None])[0] if latest_fetch.get('error'): request_errors.append(f"latest_cdx: {latest_fetch.get('error')}") + if self._is_transient_request_error(latest_fetch.get('error')): + self._trip_transient_backoff() latest_timestamp = (latest_record or {}).get('timestamp') latest_digest = (latest_record or {}).get('digest', '') + cutoff_year = self._resolve_recent_year_cutoff(recent_years) + if latest_timestamp and cutoff_year is not None: + try: + latest_year = int(str(latest_timestamp)[:4]) + except (TypeError, ValueError): + latest_year = 0 + if latest_year < cutoff_year: + latest_record = None + latest_timestamp = None + latest_digest = '' matched_word = None matched_timestamp = None matched_title = None @@ -427,6 +544,28 @@ class WaybackDetector(BaseDetector): digest_seen = set() checked_snapshot_count = 0 domain_concurrency = max(1, config.WAYBACK_DOMAIN_CONCURRENCY) + transient_snapshot_failures = 0 + transient_snapshot_failure_threshold = max(2, domain_concurrency) + + if (not latest_timestamp) and latest_fetch.get('error') and self._is_transient_request_error(latest_fetch.get('error')): + return { + 'snapshot_years': [], + 'has_sensitive_content': False, + 'matched_word': None, + 'matched_timestamp': None, + 'matched_title': None, + 'backlink_count': 0, + 'backlink_count_gt_10': False, + 'checked_snapshot_count': 0, + 'fetched_snapshot_count': 0, + 'failed_snapshot_count': max(1, failed_snapshot_count), + 'unique_title_count': 0, + 'duplicate_title_skipped': 0, + 'digest_duplicate_skipped': 0, + 'request_error_count': len(request_errors), + 'request_errors': request_errors, + 'elapsed_seconds': round(time.time() - started_at, 2), + } if latest_timestamp: latest_result = self._fetch_snapshot_title(domain, latest_timestamp) @@ -462,6 +601,30 @@ class WaybackDetector(BaseDetector): } else: failed_snapshot_count += 1 + latest_error = str((latest_result or {}).get('error') or '').strip() + if latest_error: + request_errors.append(f"latest_snapshot: {latest_error}") + if latest_error and self._is_transient_request_error(latest_error): + transient_snapshot_failures += 1 + self._trip_transient_backoff() + return { + 'snapshot_years': [], + 'has_sensitive_content': False, + 'matched_word': None, + 'matched_timestamp': None, + 'matched_title': None, + 'backlink_count': 0, + 'backlink_count_gt_10': False, + 'checked_snapshot_count': checked_snapshot_count, + 'fetched_snapshot_count': fetched_snapshot_count, + 'failed_snapshot_count': failed_snapshot_count, + 'unique_title_count': unique_title_count, + 'duplicate_title_skipped': duplicate_title_skipped, + 'digest_duplicate_skipped': digest_duplicate_skipped, + 'request_error_count': len(request_errors), + 'request_errors': request_errors, + 'elapsed_seconds': round(time.time() - started_at, 2), + } cached_records = self._load_cached_records(domain) if cached_records is not None: @@ -470,12 +633,16 @@ class WaybackDetector(BaseDetector): records_fetch = self._fetch_cdx_records_with_meta(domain) if records_fetch.get('error'): request_errors.append(f"records_cdx: {records_fetch.get('error')}") + if self._is_transient_request_error(records_fetch.get('error')): + self._trip_transient_backoff() records = records_fetch.get('records') or [] if records: self._save_cached_records(domain, records) self._save_cached_timestamps(domain, [item['timestamp'] for item in records]) + records = self._filter_records_recent_years(records, recent_years=recent_years) records = sorted(records, key=lambda item: item.get('timestamp', ''), reverse=True) + records = self._limit_records_for_scan(records) years = sorted({int(item['timestamp'][:4]) for item in records if len(item.get('timestamp', '')) >= 4}) checked_snapshot_count = len(records) pending_records = [] @@ -522,9 +689,18 @@ class WaybackDetector(BaseDetector): if not result or not result.get('ok'): failed_snapshot_count += 1 + error_message = str((result or {}).get('error') or '').strip() + if error_message: + request_errors.append(f"snapshot:{timestamp}: {error_message}") + if error_message and self._is_transient_request_error(error_message): + transient_snapshot_failures += 1 + self._trip_transient_backoff() + if transient_snapshot_failures >= transient_snapshot_failure_threshold: + stop_requested = True continue fetched_snapshot_count += 1 + transient_snapshot_failures = 0 title = result.get('title', '') normalized_title = self._normalize_title(title) if normalized_title: diff --git a/domainCheck/app/ui/system_settings.py b/domainCheck/app/ui/system_settings.py index b040cf8..33d460a 100644 --- a/domainCheck/app/ui/system_settings.py +++ b/domainCheck/app/ui/system_settings.py @@ -1355,7 +1355,7 @@ class SystemSettingsWidget(QWidget): return raw_value = self.thread_count_input.text().strip() try: - thread_count = str(min(20, max(1, int(raw_value or '10')))) + thread_count = str(max(1, int(raw_value or '10'))) except Exception: thread_count = '10' if self.thread_count_input.text().strip() != thread_count: @@ -1390,7 +1390,7 @@ class SystemSettingsWidget(QWidget): if os.path.exists('thread_count.json'): with open('thread_count.json', 'r', encoding='utf-8') as f: thread_config = json.load(f) - thread_count = str(min(20, max(1, int(thread_config.get('thread_count', '10') or '10')))) + thread_count = str(max(1, int(thread_config.get('thread_count', '10') or '10'))) self.thread_count_input.setText(thread_count) logger.info("检测线程数配置加载成功") else: diff --git a/domainCheck/app/utils/database.py b/domainCheck/app/utils/database.py index f8ff941..804a270 100644 --- a/domainCheck/app/utils/database.py +++ b/domainCheck/app/utils/database.py @@ -8,13 +8,17 @@ @explain : 数据库操作类 ''' -import psycopg2 -from psycopg2.extras import Json import json -import redis -import threading +import os import socket +import threading +import time + +import psycopg2 +import redis from loguru import logger +from psycopg2 import extensions +from psycopg2.extras import Json from app.config import config from app.utils.status_codes import ( DETECT_STATUS_BLACKLISTED, @@ -29,6 +33,51 @@ from app.utils.status_codes import ( ) +_POOL_VERBOSE_LOGS = bool(int(os.getenv("DOMAINCHECK_DB_POOL_VERBOSE_LOGS", "0") or 0)) + + +_STEP_CLAIM_PRIORITY = { + # Prefer deeper pipeline steps first so domains that already passed an + # earlier gate can keep advancing instead of being starved behind the + # oldest first-step backlog. This keeps later steps like 360/chinaz/aizhan + # from sitting pending forever while baidu/register continue to dominate + # the queue. + "detect_juziseo": 10, + "detect_jucha": 20, + "detect_wayback": 30, + "detect_aizhan": 40, + "detect_chinaz": 50, + "detect_360_site": 60, + "detect_baidu_site": 70, + "detect_register": 100, +} + + +def _detect_job_item_step_priority(step_code: str) -> int: + normalized = str(step_code or "").strip() + if not normalized: + return 999 + return int(_STEP_CLAIM_PRIORITY.get(normalized, 10)) + + +def _ordered_step_claim_codes() -> list[str]: + return [ + step_code + for step_code, _ in sorted( + _STEP_CLAIM_PRIORITY.items(), + key=lambda item: (int(item[1] or 0), str(item[0] or "")), + ) + ] + + +def _resolve_step_claim_quota(limit: int) -> int: + configured_quota = int(os.getenv("DOMAINCHECK_STEP_CLAIM_QUOTA", "0") or 0) + if configured_quota > 0: + return max(1, configured_quota) + normalized_limit = max(1, int(limit or 1)) + return max(64, min(normalized_limit, max(1, normalized_limit // 4))) + + class Database: """ 数据库操作类 @@ -52,8 +101,22 @@ class Database: # 数据库连接池 self.connection_pool = [] - self.pool_size = config.DB_POOL_SIZE # 连接池大小 + self.pool_size = max(1, int(config.DB_POOL_SIZE)) # 连接池大小 + self.pool_warm_size = max(1, min(self.pool_size, int(getattr(config, 'DB_POOL_WARM_SIZE', 16) or 16))) + self.pool_idle_keep_max = max( + self.pool_warm_size, + min(self.pool_size, int(getattr(config, 'DB_POOL_IDLE_KEEP_MAX', self.pool_size) or self.pool_size)), + ) + self.pool_healthcheck_interval = max( + 0.0, + float(getattr(config, 'DB_POOL_HEALTHCHECK_INTERVAL', 30) or 30), + ) + self.pool_acquire_timeout = max(1.0, float(getattr(config, 'DB_POOL_ACQUIRE_TIMEOUT', 20))) self.pool_lock = threading.Lock() + self.pool_condition = threading.Condition(self.pool_lock) + self._pool_initialized = False + self.total_connections = 0 + self._connection_last_healthcheck = {} # 初始化连接池 self._init_connection_pool() @@ -85,18 +148,39 @@ class Database: 初始化数据库连接池 """ try: - for i in range(self.pool_size): - conn = psycopg2.connect( - host=self.host, - port=self.port, - database=self.database, - user=self.user, - password=self.password - ) - self.connection_pool.append(conn) - logger.info(f"数据库连接池初始化成功,大小: {self.pool_size}") + with self.pool_condition: + missing = self.pool_warm_size - self.total_connections + if missing <= 0: + return + + created = [] + for i in range(missing): + created.append(self._create_connection()) + + with self.pool_condition: + self.connection_pool.extend(created) + self.total_connections += len(created) + self._pool_initialized = True + now = time.monotonic() + for conn in created: + self._connection_last_healthcheck[id(conn)] = now + self.pool_condition.notify_all() + logger.info( + f"数据库连接池初始化成功,预热: {len(created)},池中空闲: {len(self.connection_pool)},总连接: {self.total_connections}/{self.pool_size}" + ) except Exception as e: logger.error(f"初始化数据库连接池失败: {e}") + + def _create_connection(self): + return psycopg2.connect( + host=self.host, + port=self.port, + database=self.database, + user=self.user, + password=self.password, + connect_timeout=5, + application_name="domaincheck-worker", + ) def _init_bloom_filter(self): """ @@ -134,45 +218,72 @@ class Database: thread_id = thread_id or threading.current_thread().ident try: - with self.pool_lock: - if not self.connection_pool: - # 连接池为空,尝试重新初始化 - self._init_connection_pool() - - if self.connection_pool: - # 从连接池获取连接 - conn = self.connection_pool.pop() - # 检查连接是否有效 - if conn and not conn.closed: - try: - # 测试连接 - cur = conn.cursor() - cur.execute("SELECT 1") - cur.fetchone() - cur.close() - logger.debug(f"线程 {thread_id} 从连接池获取连接成功") - return conn, conn.cursor() - except: - # 连接无效,关闭并重新获取 + if not self._pool_initialized: + self._init_connection_pool() + + deadline = time.monotonic() + self.pool_acquire_timeout + warned_pool_empty = False + + while True: + create_new = False + with self.pool_condition: + while self.connection_pool: + conn = self.connection_pool.pop() + if conn and not conn.closed: try: - conn.close() - except: - pass - if self.connection_pool: - conn = self.connection_pool.pop() - if conn and not conn.closed: - return conn, conn.cursor() - - # 连接池为空或所有连接都无效,创建新连接 - logger.warning(f"连接池为空,线程 {thread_id} 创建新连接") - conn = psycopg2.connect( - host=self.host, - port=self.port, - database=self.database, - user=self.user, - password=self.password - ) - return conn, conn.cursor() + if conn.get_transaction_status() != extensions.TRANSACTION_STATUS_IDLE: + conn.rollback() + last_healthcheck = float(self._connection_last_healthcheck.get(id(conn), 0.0) or 0.0) + now = time.monotonic() + if self.pool_healthcheck_interval > 0 and now - last_healthcheck >= self.pool_healthcheck_interval: + cur = conn.cursor() + cur.execute("SELECT 1") + cur.fetchone() + cur.close() + self._connection_last_healthcheck[id(conn)] = now + if _POOL_VERBOSE_LOGS: + logger.debug(f"线程 {thread_id} 从连接池获取连接成功") + return conn, conn.cursor() + except Exception: + self._connection_last_healthcheck.pop(id(conn), None) + try: + conn.close() + except Exception: + pass + self.total_connections = max(0, self.total_connections - 1) + continue + + if self.total_connections < self.pool_size: + self.total_connections += 1 + create_new = True + else: + remaining = deadline - time.monotonic() + if remaining <= 0: + logger.warning( + f"连接池耗尽,线程 {thread_id} 等待超时 {self.pool_acquire_timeout}s,返回空连接" + ) + return None, None + + if not warned_pool_empty: + warned_pool_empty = True + logger.warning( + f"连接池耗尽,线程 {thread_id} 等待可复用连接,池大小 {self.pool_size}" + ) + self.pool_condition.wait(timeout=min(0.5, remaining)) + + if create_new: + try: + conn = self._create_connection() + self._connection_last_healthcheck[id(conn)] = time.monotonic() + if _POOL_VERBOSE_LOGS: + logger.debug(f"线程 {thread_id} 新建数据库连接成功,总连接 {self.total_connections}/{self.pool_size}") + return conn, conn.cursor() + except Exception as e: + with self.pool_condition: + self.total_connections = max(0, self.total_connections - 1) + self.pool_condition.notify() + logger.error(f"线程 {thread_id} 新建数据库连接失败: {e}") + return None, None except Exception as e: logger.error(f"线程 {thread_id} 获取数据库连接失败: {e}") return None, None @@ -201,18 +312,38 @@ class Database: pass if conn and not conn.closed: - with self.pool_lock: - if len(self.connection_pool) < self.pool_size: + with self.pool_condition: + try: + if conn.get_transaction_status() != extensions.TRANSACTION_STATUS_IDLE: + conn.rollback() + except Exception: + try: + self._connection_last_healthcheck.pop(id(conn), None) + conn.close() + finally: + self.total_connections = max(0, self.total_connections - 1) + self.pool_condition.notify() + logger.warning("连接归还前回滚失败,已关闭连接") + return + if len(self.connection_pool) < self.pool_idle_keep_max: self.connection_pool.append(conn) - logger.debug("连接已放回连接池") + self.pool_condition.notify() + if _POOL_VERBOSE_LOGS: + logger.debug("连接已放回连接池") else: - # 连接池已满,关闭连接 + # 启动阶段只预热少量连接,高并发跑起来后允许保留更多空闲连接, + # 避免本地数据库在“建连/关连”之间来回抖动。 + self._connection_last_healthcheck.pop(id(conn), None) conn.close() - logger.debug("连接池已满,关闭连接") + self.total_connections = max(0, self.total_connections - 1) + self.pool_condition.notify() + if _POOL_VERBOSE_LOGS: + logger.debug(f"空闲连接超过保留阈值({self.pool_idle_keep_max}),已关闭多余连接") except Exception as e: logger.error(f"关闭数据库连接失败: {e}") try: if conn and not conn.closed: + self._connection_last_healthcheck.pop(id(conn), None) conn.close() except: pass @@ -394,6 +525,8 @@ class Database: job_code VARCHAR(64) NOT NULL UNIQUE, source VARCHAR(64) NOT NULL DEFAULT 'manual', plan_hash VARCHAR(128) NOT NULL DEFAULT '', + task_mode VARCHAR(32) NOT NULL DEFAULT 'domain_pipeline', + step_code VARCHAR(64) NOT NULL DEFAULT '', status VARCHAR(32) NOT NULL DEFAULT 'pending', remark TEXT NOT NULL DEFAULT '', created_by VARCHAR(64) NOT NULL DEFAULT '', @@ -406,6 +539,7 @@ class Database: id BIGSERIAL PRIMARY KEY, job_id BIGINT NOT NULL REFERENCES detect_jobs(id) ON DELETE CASCADE, domain_id BIGINT NOT NULL, + step_code VARCHAR(64) NOT NULL DEFAULT '', status VARCHAR(32) NOT NULL DEFAULT 'pending', claimed_by VARCHAR(64) NOT NULL DEFAULT '', claim_token VARCHAR(64) NOT NULL DEFAULT '', @@ -413,16 +547,32 @@ class Database: attempt_count INTEGER NOT NULL DEFAULT 0, last_error TEXT NOT NULL DEFAULT '', result_version VARCHAR(64) NOT NULL DEFAULT '', + step_payload_json JSONB, + result_payload_json JSONB, started_at TIMESTAMP, finished_at TIMESTAMP, updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, - CONSTRAINT uq_detect_job_items_job_domain UNIQUE (job_id, domain_id) + create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ); CREATE INDEX IF NOT EXISTS idx_detect_job_items_status_lease ON detect_job_items(status, lease_expires_at); + ALTER TABLE detect_jobs + ADD COLUMN IF NOT EXISTS task_mode VARCHAR(32) NOT NULL DEFAULT 'domain_pipeline', + ADD COLUMN IF NOT EXISTS step_code VARCHAR(64) NOT NULL DEFAULT ''; + + ALTER TABLE detect_job_items + ADD COLUMN IF NOT EXISTS step_code VARCHAR(64) NOT NULL DEFAULT '', + ADD COLUMN IF NOT EXISTS step_payload_json JSONB, + ADD COLUMN IF NOT EXISTS result_payload_json JSONB; + + ALTER TABLE detect_job_items + DROP CONSTRAINT IF EXISTS uq_detect_job_items_job_domain; + + CREATE UNIQUE INDEX IF NOT EXISTS idx_detect_job_items_job_domain_step + ON detect_job_items(job_id, domain_id, step_code); + CREATE TABLE IF NOT EXISTS detect_run_events ( id BIGSERIAL PRIMARY KEY, job_id BIGINT REFERENCES detect_jobs(id) ON DELETE SET NULL, @@ -490,7 +640,7 @@ class Database: logger.warning(f"注册检测节点失败: {e}") return False - def claim_detect_job_items(self, node_code, limit=1000, lease_seconds=3600): + def claim_detect_job_items(self, node_code, limit=1000, lease_seconds=3600, job_id=None): """ 领取一批待执行的任务项。 """ @@ -502,50 +652,94 @@ class Database: if not conn or not cur: logger.error("领取检测任务失败: 无法获取数据库连接") return [] - cur.execute( - """ - WITH picked AS ( - SELECT id - FROM detect_job_items - WHERE status IN ('pending', 'failed') - AND (lease_expires_at IS NULL OR lease_expires_at < CURRENT_TIMESTAMP) - ORDER BY create_time ASC, id ASC - FOR UPDATE SKIP LOCKED - LIMIT %s - ), - updated AS ( - UPDATE detect_job_items AS item - SET status = 'claimed', - claimed_by = %s, - claim_token = %s, - lease_expires_at = CURRENT_TIMESTAMP + (%s || ' seconds')::interval, - attempt_count = item.attempt_count + 1, - started_at = COALESCE(item.started_at, CURRENT_TIMESTAMP), - updated_at = CURRENT_TIMESTAMP - FROM picked - WHERE item.id = picked.id - RETURNING item.id, item.job_id, item.domain_id, item.claim_token + normalized_job_id = int(job_id) if job_id not in (None, "", 0, "0") else None + normalized_limit = max(1, int(limit or 1)) + normalized_lease_seconds = max(60, int(lease_seconds or 3600)) + step_claim_quota = _resolve_step_claim_quota(normalized_limit) + rows = [] + + def _claim_rows(batch_limit: int, step_condition_sql: str = "", step_condition_params: tuple = ()) -> list: + safe_batch_limit = max(1, int(batch_limit or 0)) + if safe_batch_limit <= 0: + return [] + extra_condition = f"\n AND {step_condition_sql}" if step_condition_sql else "" + cur.execute( + f""" + WITH picked AS ( + SELECT id + FROM detect_job_items + WHERE status IN ('pending', 'failed') + AND COALESCE(step_code, '') <> '' + AND (%s IS NULL OR job_id = %s) + AND (lease_expires_at IS NULL OR lease_expires_at < CURRENT_TIMESTAMP){extra_condition} + ORDER BY create_time ASC, id ASC + FOR UPDATE SKIP LOCKED + LIMIT %s + ), + updated AS ( + UPDATE detect_job_items AS item + SET status = 'claimed', + claimed_by = %s, + claim_token = %s, + lease_expires_at = CURRENT_TIMESTAMP + (%s || ' seconds')::interval, + attempt_count = item.attempt_count + 1, + started_at = COALESCE(item.started_at, CURRENT_TIMESTAMP), + updated_at = CURRENT_TIMESTAMP + FROM picked + WHERE item.id = picked.id + RETURNING item.id, item.job_id, item.domain_id, item.claim_token, item.step_code, item.step_payload_json + ) + SELECT + updated.id, + updated.job_id, + updated.domain_id, + updated.claim_token, + updated.step_code, + detect_jobs.task_mode, + detect_jobs.job_code, + detect_jobs.step_code, + updated.step_payload_json, + domains.domain, + domains.source_type, + domains.register_status, + domains.detect_status, + domains.use_status, + domains.expire_date, + domains.jucha_status, + domains.juziseo_status + FROM updated + JOIN detect_jobs ON detect_jobs.id = updated.job_id + JOIN domains ON domains.id = updated.domain_id + ORDER BY updated.id ASC + """, + ( + normalized_job_id, + normalized_job_id, + *step_condition_params, + safe_batch_limit, + node_code, + claim_token, + normalized_lease_seconds, + ), ) - SELECT - updated.id, - updated.job_id, - updated.domain_id, - updated.claim_token, - domains.domain, - domains.source_type, - domains.register_status, - domains.detect_status, - domains.use_status, - domains.expire_date, - domains.jucha_status, - domains.juziseo_status - FROM updated - JOIN domains ON domains.id = updated.domain_id - ORDER BY updated.id ASC - """, - (limit, node_code, claim_token, max(60, int(lease_seconds or 3600))), - ) - rows = cur.fetchall() or [] + return cur.fetchall() or [] + + remaining = normalized_limit + for step_code in _ordered_step_claim_codes(): + if remaining <= 0: + break + claimed_rows = _claim_rows( + min(remaining, step_claim_quota), + "COALESCE(step_code, '') = %s", + (str(step_code or "").strip(),), + ) + if claimed_rows: + rows.extend(claimed_rows) + remaining -= len(claimed_rows) + + if remaining > 0: + rows.extend(_claim_rows(remaining)) + touched_job_ids = sorted({row[1] for row in rows}) for job_id in touched_job_ids: cur.execute( @@ -564,14 +758,19 @@ class Database: "job_id": row[1], "id": row[2], "claim_token": row[3], - "domain": row[4], - "source_type": row[5], - "register_status": row[6], - "detect_status": row[7], - "use_status": row[8], - "expire_date": row[9], - "jucha_status": row[10], - "juziseo_status": row[11], + "item_step_code": row[4], + "task_mode": row[5], + "job_code": row[6], + "step_code": row[4] or row[7], + "step_payload": row[8], + "domain": row[9], + "source_type": row[10], + "register_status": row[11], + "detect_status": row[12], + "use_status": row[13], + "expire_date": row[14], + "jucha_status": row[15], + "juziseo_status": row[16], } for row in rows ] @@ -586,6 +785,31 @@ class Database: finally: self.close(conn, cur) + def has_dispatchable_detect_job_items(self): + """ + 是否仍存在可继续派发的标准步骤任务项。 + + 这里显式排除空 step_code 的旧兼容链路项,避免 worker 在 single_step + 任务已经大量积压时继续回退到旧 domains 链路补位。 + """ + row = self.fetch_one( + """ + SELECT EXISTS ( + SELECT 1 + FROM detect_job_items + WHERE status IN ('pending', 'failed') + AND COALESCE(step_code, '') <> '' + AND (lease_expires_at IS NULL OR lease_expires_at < CURRENT_TIMESTAMP) + LIMIT 1 + ) AS has_items + """ + ) + if isinstance(row, dict): + return bool(row.get("has_items")) + if isinstance(row, (list, tuple)) and row: + return bool(row[0]) + return False + def recycle_expired_detect_job_items(self): """ 回收租约过期但未完成的任务项,重新放回 pending。 @@ -688,18 +912,104 @@ class Database: finally: self.close(conn, cur) + def get_active_detect_job(self): + """ + 获取当前活动中的检测任务摘要,供 Worker 重启后自动回挂。 + """ + return self.fetch_one( + """ + SELECT + job.id, + job.job_code, + job.task_mode, + job.status, + count(item.*) FILTER (WHERE item.status = 'pending') AS items_pending, + count(item.*) FILTER (WHERE item.status = 'claimed') AS items_claimed, + count(item.*) FILTER (WHERE item.status = 'running') AS items_running, + count(item.*) FILTER (WHERE item.status IN ('completed', 'blacklisted')) AS items_completed, + count(item.*) FILTER (WHERE item.status = 'failed') AS items_failed, + max(item.updated_at) AS latest_item_updated_at, + max(item.create_time) AS latest_item_created_at + FROM detect_jobs AS job + LEFT JOIN detect_job_items AS item + ON item.job_id = job.id + WHERE job.status IN ('pending', 'running') + GROUP BY job.id, job.job_code, job.task_mode, job.status + ORDER BY + CASE + WHEN count(item.*) FILTER (WHERE item.status IN ('claimed', 'running')) > 0 THEN 0 + WHEN count(item.*) FILTER (WHERE item.status = 'pending') > 0 THEN 1 + ELSE 2 + END ASC, + CASE WHEN job.status = 'running' THEN 0 ELSE 1 END ASC, + COALESCE(max(item.updated_at), max(item.create_time), job.started_at, job.created_at) DESC, + job.id DESC + LIMIT 1 + """ + ) + def mark_detect_job_item_running(self, job_item_id, claim_token): return self.execute( """ UPDATE detect_job_items SET status = 'running', - updated_at = CURRENT_TIMESTAMP, - lease_expires_at = CURRENT_TIMESTAMP + interval '1 hour' - WHERE id = %s AND claim_token = %s + updated_at = CURRENT_TIMESTAMP + WHERE id = %s AND claim_token = %s AND status = 'claimed' """, (job_item_id, claim_token), ) + def mark_detect_job_items_running_batch(self, items): + normalized_items = [] + for item in list(items or []): + if not isinstance(item, (list, tuple)) or len(item) < 2: + continue + try: + job_item_id = int(item[0] or 0) + except Exception: + continue + claim_token = str(item[1] or "").strip() + if job_item_id <= 0 or not claim_token: + continue + normalized_items.append((job_item_id, claim_token)) + if not normalized_items: + return 0 + + conn = None + cur = None + try: + conn, cur = self.connect() + if not conn or not cur: + return -1 + values_sql = ",".join( + cur.mogrify("(%s, %s)", item).decode("utf-8") + for item in normalized_items + ) + cur.execute( + f""" + UPDATE detect_job_items AS item + SET status = 'running', + updated_at = CURRENT_TIMESTAMP + FROM (VALUES {values_sql}) AS batch(id, claim_token) + WHERE item.id = batch.id + AND item.claim_token = batch.claim_token + AND item.status = 'claimed' + """ + ) + updated_count = int(cur.rowcount or 0) + conn.commit() + return updated_count + except Exception as e: + try: + if conn: + conn.rollback() + except Exception: + pass + logger.error(f"批量标记检测任务项为运行中失败: {e}") + return -1 + finally: + self.close(conn, cur) + def renew_detect_job_item_lease(self, job_item_id, claim_token, lease_seconds=3600): return self.execute( """ @@ -720,27 +1030,40 @@ class Database: (job_id, job_item_id, node_code, event_type, level, message, Json(payload or {})), ) - def complete_detect_job_item(self, job_item_id, claim_token, final_status='completed'): + def complete_detect_job_item(self, job_item_id, claim_token, final_status='completed', result_payload=None, result_version='v1', refresh_job_status=True): conn = None cur = None try: conn, cur = self.connect() if not conn or not cur: return False + payload_json = json.dumps(result_payload, ensure_ascii=False) if result_payload is not None else None cur.execute( """ UPDATE detect_job_items SET status = %s, + result_payload_json = COALESCE(%s::jsonb, result_payload_json), + result_version = CASE + WHEN %s <> '' THEN %s + ELSE result_version + END, finished_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP, lease_expires_at = NULL WHERE id = %s AND claim_token = %s RETURNING job_id """, - (final_status, job_item_id, claim_token), + ( + final_status, + payload_json, + str(result_version or '').strip(), + str(result_version or '').strip(), + job_item_id, + claim_token, + ), ) row = cur.fetchone() - if row: + if row and refresh_job_status: self._refresh_detect_job_status_with_cursor(cur, row[0]) conn.commit() return bool(row) @@ -755,28 +1078,41 @@ class Database: finally: self.close(conn, cur) - def fail_detect_job_item(self, job_item_id, claim_token, reason=''): + def fail_detect_job_item(self, job_item_id, claim_token, reason='', result_payload=None, result_version='v1', refresh_job_status=True): conn = None cur = None try: conn, cur = self.connect() if not conn or not cur: return False + payload_json = json.dumps(result_payload, ensure_ascii=False) if result_payload is not None else None cur.execute( """ UPDATE detect_job_items SET status = 'failed', last_error = %s, + result_payload_json = COALESCE(%s::jsonb, result_payload_json), + result_version = CASE + WHEN %s <> '' THEN %s + ELSE result_version + END, finished_at = CURRENT_TIMESTAMP, updated_at = CURRENT_TIMESTAMP, lease_expires_at = NULL WHERE id = %s AND claim_token = %s RETURNING job_id """, - (str(reason or '')[:1000], job_item_id, claim_token), + ( + str(reason or '')[:1000], + payload_json, + str(result_version or '').strip(), + str(result_version or '').strip(), + job_item_id, + claim_token, + ), ) row = cur.fetchone() - if row: + if row and refresh_job_status: self._refresh_detect_job_status_with_cursor(cur, row[0]) conn.commit() return bool(row) @@ -791,28 +1127,180 @@ class Database: finally: self.close(conn, cur) + def finalize_detect_job_items_batch(self, items): + normalized_items = [] + event_rows = [] + for item in list(items or []): + if not isinstance(item, dict): + continue + try: + job_item_id = int(item.get("job_item_id") or 0) + except Exception: + continue + claim_token = str(item.get("claim_token") or "").strip() + final_status = str(item.get("final_status") or "").strip() or "failed" + message = str(item.get("message") or "").strip() + result_payload = item.get("result_payload") + result_version = str(item.get("result_version") or "v1").strip() + if job_item_id <= 0 or not claim_token: + continue + payload_json = "" + if result_payload is not None: + try: + payload_json = json.dumps(result_payload, ensure_ascii=False) + except Exception: + payload_json = "" + normalized_items.append( + ( + job_item_id, + claim_token, + final_status, + message[:1000], + payload_json, + result_version, + ) + ) + try: + event_job_id = int(item.get("job_id") or 0) + except Exception: + event_job_id = 0 + event_type = str(item.get("event_type") or "").strip() + event_message = str(item.get("event_message") or "").strip() + if event_job_id > 0 and event_type and event_message: + event_payload_json = "{}" + try: + event_payload_json = json.dumps(item.get("event_payload") or {}, ensure_ascii=False) + except Exception: + event_payload_json = "{}" + event_rows.append( + ( + job_item_id, + event_job_id, + str(item.get("node_code") or "").strip(), + event_type, + str(item.get("event_level") or "info").strip() or "info", + event_message[:1000], + event_payload_json, + ) + ) + if not normalized_items: + return 0 + + conn = None + cur = None + try: + conn, cur = self.connect() + if not conn or not cur: + return -1 + values_sql = ",".join( + cur.mogrify("(%s, %s, %s, %s, %s, %s)", item).decode("utf-8") + for item in normalized_items + ) + cur.execute( + f""" + UPDATE detect_job_items AS item + SET status = batch.final_status, + last_error = CASE + WHEN batch.final_status = 'failed' AND batch.message <> '' THEN batch.message + ELSE item.last_error + END, + result_payload_json = COALESCE(NULLIF(batch.payload_json, '')::jsonb, item.result_payload_json), + result_version = CASE + WHEN batch.result_version <> '' THEN batch.result_version + ELSE item.result_version + END, + finished_at = CURRENT_TIMESTAMP, + updated_at = CURRENT_TIMESTAMP, + lease_expires_at = NULL + FROM ( + VALUES {values_sql} + ) AS batch(id, claim_token, final_status, message, payload_json, result_version) + WHERE item.id = batch.id + AND item.claim_token = batch.claim_token + AND item.status IN ('claimed', 'running') + RETURNING item.id + """ + ) + updated_item_ids = {int(row[0]) for row in (cur.fetchall() or [])} + updated_count = len(updated_item_ids) + filtered_event_rows = [row for row in event_rows if int(row[0]) in updated_item_ids] + if filtered_event_rows: + values_sql = ",".join( + cur.mogrify("(%s, %s, %s, %s, %s, %s, %s::jsonb)", row).decode("utf-8") + for row in filtered_event_rows + ) + cur.execute( + f""" + INSERT INTO detect_run_events ( + job_item_id, + job_id, + node_code, + event_type, + level, + message, + payload_json + ) + VALUES {values_sql} + """ + ) + conn.commit() + return updated_count + except Exception as e: + try: + if conn: + conn.rollback() + except Exception: + pass + logger.error(f"批量完成检测任务项失败: {e}") + return -1 + finally: + self.close(conn, cur) + def _refresh_detect_job_status_with_cursor(self, cur, job_id): cur.execute( """ SELECT - count(*) FILTER (WHERE status IN ('pending', 'claimed', 'running')) AS active_count, - count(*) FILTER (WHERE status = 'failed') AS failed_count, - count(*) FILTER (WHERE status IN ('completed', 'blacklisted')) AS done_count + job.task_mode, + count(*) FILTER (WHERE detect_job_items.status = 'pending') AS pending_count, + count(*) FILTER (WHERE detect_job_items.status IN ('claimed', 'running')) AS dispatch_active_count, + count(*) FILTER (WHERE detect_job_items.status = 'failed') AS failed_count, + count(*) FILTER (WHERE detect_job_items.status IN ('completed', 'blacklisted')) AS done_count, + count(*) FILTER ( + WHERE detect_job_items.status IN ('completed', 'blacklisted', 'failed') + AND COALESCE(detect_job_items.step_code, '') <> '' + AND COALESCE(result_payload_json->>'controller_processed', 'false') <> 'true' + ) AS unprocessed_terminal_count FROM detect_job_items + JOIN detect_jobs AS job ON job.id = detect_job_items.job_id WHERE job_id = %s + GROUP BY job.task_mode """, (job_id,), ) row = cur.fetchone() - active_count = int(row[0] or 0) - failed_count = int(row[1] or 0) - done_count = int(row[2] or 0) - if active_count > 0: + task_mode = str((row or [""])[0] or "").strip() + pending_count = int((row or ["", 0])[1] or 0) + dispatch_active_count = int((row or ["", 0, 0])[2] or 0) + failed_count = int((row or ["", 0, 0, 0])[3] or 0) + done_count = int((row or ["", 0, 0, 0, 0])[4] or 0) + unprocessed_terminal_count = int((row or ["", 0, 0, 0, 0, 0])[5] or 0) + if dispatch_active_count > 0 or (task_mode == 'domain_pipeline' and unprocessed_terminal_count > 0): cur.execute( "UPDATE detect_jobs SET status = 'running', started_at = COALESCE(started_at, CURRENT_TIMESTAMP) WHERE id = %s", (job_id,), ) return + if pending_count > 0: + cur.execute( + """ + UPDATE detect_jobs + SET status = 'pending', + finished_at = NULL + WHERE id = %s + """, + (job_id,), + ) + return final_status = 'completed' if failed_count > 0 and done_count > 0: final_status = 'partial_failed' @@ -828,6 +1316,27 @@ class Database: """, (final_status, job_id), ) + + def refresh_detect_job_status(self, job_id): + conn = None + cur = None + try: + conn, cur = self.connect() + if not conn or not cur: + return False + self._refresh_detect_job_status_with_cursor(cur, job_id) + conn.commit() + return True + except Exception as e: + try: + if conn: + conn.rollback() + except Exception: + pass + logger.error(f"刷新检测任务状态失败: {e}") + return False + finally: + self.close(conn, cur) def fetch_one(self, sql, params=None): """ @@ -1271,6 +1780,40 @@ class Database: sql = "UPDATE domains SET detect_status = %s WHERE id = %s" return self.execute(sql, (status, domain_id)) + def update_domain_detect_status_batch(self, items): + """ + 批量更新域名检测状态,减少高并发失败风暴时的写库往返。 + """ + normalized_items = [] + for item in items or []: + try: + domain_id, status = item + normalized_items.append((int(domain_id), int(status))) + except Exception: + continue + if not normalized_items: + return 0 + + values_sql = ", ".join(["(%s, %s)"] * len(normalized_items)) + params = [] + for domain_id, status in normalized_items: + params.extend([domain_id, status]) + + sql = f""" + UPDATE domains AS d + SET detect_status = v.detect_status, + detect_time = CASE + WHEN v.detect_status = %s THEN CURRENT_TIMESTAMP + ELSE d.detect_time + END + FROM ( + VALUES {values_sql} + ) AS v(domain_id, detect_status) + WHERE d.id = v.domain_id + """ + params = [DETECT_STATUS_COMPLETED] + params + return len(normalized_items) if self.execute(sql, tuple(params)) else -1 + def recycle_running_domains(self, target_status): """ 回收异常中断后遗留的“检测中”状态。 @@ -1410,6 +1953,43 @@ class Database: SET reason = %s, updated_at = NOW() """ return self.execute(sql, (domain, reason, reason)) + + def mark_domain_blacklisted(self, domain_id, domain, reason): + """ + 在一个事务中同时更新域名黑名单状态和黑名单表,减少热路径往返。 + """ + conn = None + cur = None + try: + conn, cur = self.connect() + if not conn or not cur: + logger.error("标记域名黑名单失败: 无法获取数据库连接") + return False + cur.execute( + "UPDATE domains SET detect_status = %s WHERE id = %s", + (DETECT_STATUS_BLACKLISTED, domain_id), + ) + cur.execute( + """ + INSERT INTO blacklist (domain, reason, created_at) + VALUES (%s, %s, NOW()) + ON CONFLICT (domain) DO UPDATE + SET reason = %s, updated_at = NOW() + """, + (domain, reason, reason), + ) + conn.commit() + return True + except Exception as e: + try: + if conn: + conn.rollback() + except Exception: + pass + logger.error(f"标记域名黑名单失败: {e}") + return False + finally: + self.close(conn, cur) def update_domain_register_status(self, domain_id, status): """ @@ -1421,6 +2001,24 @@ class Database: """ sql = "UPDATE domains SET register_status = %s WHERE id = %s" return self.execute(sql, (status, domain_id)) + + def update_domain_register_result(self, domain_id, status, expire_date=None): + """ + 一次性更新注册状态及过期时间,减少热路径往返次数。 + + :param domain_id: 域名ID + :param status: 注册状态 + :param expire_date: 过期时间;为空时保持原值 + :return: bool - 是否更新成功 + """ + normalized_expire_date = str(expire_date).strip() if expire_date not in (None, "") else None + sql = """ + UPDATE domains + SET register_status = %s, + expire_date = COALESCE(%s, expire_date) + WHERE id = %s + """ + return self.execute(sql, (status, normalized_expire_date, domain_id)) def update_domain_beian_info(self, domain_id, company_type, website_url, has_beian, beian_year): """ @@ -1435,6 +2033,31 @@ class Database: """ sql = "UPDATE domains SET company_type = %s, website_url = %s, has_beian = %s, beian_year = %s WHERE id = %s" return self.execute(sql, (company_type, website_url, has_beian, beian_year, domain_id)) + + def update_domain_beian_info_and_mark_jucha_detected( + self, + domain_id, + company_type=None, + website_url=None, + has_beian=None, + beian_year=None, + ): + """ + 一次性更新备案信息并标记聚查已完成,减少热路径写库次数。 + """ + sql = """ + UPDATE domains + SET company_type = COALESCE(%s, company_type), + website_url = COALESCE(%s, website_url), + has_beian = COALESCE(%s, has_beian), + beian_year = COALESCE(%s, beian_year), + jucha_status = %s + WHERE id = %s + """ + return self.execute( + sql, + (company_type, website_url, has_beian, beian_year, THIRD_PARTY_STATUS_DONE, domain_id), + ) def update_domain_review_status(self, domain_id, review_status): """ @@ -1446,6 +2069,35 @@ class Database: """ sql = "UPDATE domains SET review_status = %s WHERE id = %s" return self.execute(sql, (review_status, domain_id)) + + def update_domain_review_status_batch(self, items): + """ + 批量更新域名复核状态。 + """ + normalized_items = [] + for item in items or []: + try: + domain_id, review_status = item + normalized_items.append((int(domain_id), int(review_status))) + except Exception: + continue + if not normalized_items: + return 0 + + values_sql = ", ".join(["(%s, %s)"] * len(normalized_items)) + params = [] + for domain_id, review_status in normalized_items: + params.extend([domain_id, review_status]) + + sql = f""" + UPDATE domains AS d + SET review_status = v.review_status + FROM ( + VALUES {values_sql} + ) AS v(domain_id, review_status) + WHERE d.id = v.domain_id + """ + return len(normalized_items) if self.execute(sql, tuple(params)) else -1 def update_domain_snapshot_years(self, domain_id, years): """ @@ -1457,6 +2109,109 @@ class Database: """ sql = "UPDATE domains SET snapshot_years = %s WHERE id = %s" return self.execute(sql, (years, domain_id)) + + def update_domain_wayback_summary(self, domain_id, years=None, backlink_count=None): + """ + 一次性更新时光机摘要字段,减少 domains 表写入次数。 + """ + sql = """ + UPDATE domains + SET snapshot_years = COALESCE(%s, snapshot_years), + backlink_count = COALESCE(%s, backlink_count) + WHERE id = %s + """ + return self.execute(sql, (years, backlink_count, domain_id)) + + def complete_domain_detection(self, domain_id, *, register_status, use_status, expire_date): + """ + 一次性完成域名完成态、待复核态、过期时间归零逻辑。 + + :param domain_id: 域名ID + :param register_status: 当前注册状态 + :param use_status: 当前使用状态 + :param expire_date: 当前过期时间 + :return: bool - 是否更新成功 + """ + has_expire_date = bool(expire_date) + sql = """ + UPDATE domains + SET detect_status = %s, + detect_time = CURRENT_TIMESTAMP, + expire_date = CASE + WHEN %s = %s AND %s = 0 AND %s THEN NULL + ELSE expire_date + END, + review_status = CASE + WHEN %s = %s THEN %s + ELSE review_status + END + WHERE id = %s + """ + return self.execute( + sql, + ( + DETECT_STATUS_COMPLETED, + register_status, + REGISTER_STATUS_AVAILABLE, + use_status, + has_expire_date, + register_status, + REGISTER_STATUS_AVAILABLE, + REVIEW_STATUS_PENDING, + domain_id, + ), + ) + + def complete_domain_detection_batch(self, items): + """ + 批量更新域名完成态,减少 completed 尾部的单条写库往返。 + """ + normalized_items = [] + for item in items or []: + try: + domain_id, register_status, use_status, has_expire_date = item + normalized_items.append( + ( + int(domain_id), + int(register_status), + int(use_status), + bool(has_expire_date), + ) + ) + except Exception: + continue + if not normalized_items: + return 0 + + values_sql = ", ".join(["(%s, %s, %s, %s)"] * len(normalized_items)) + params = [] + for domain_id, register_status, use_status, has_expire_date in normalized_items: + params.extend([domain_id, register_status, use_status, has_expire_date]) + + sql = f""" + UPDATE domains AS d + SET detect_status = %s, + detect_time = CURRENT_TIMESTAMP, + expire_date = CASE + WHEN v.register_status = %s AND v.use_status = 0 AND v.has_expire_date THEN NULL + ELSE d.expire_date + END, + review_status = CASE + WHEN v.register_status = %s THEN %s + ELSE d.review_status + END + FROM ( + VALUES {values_sql} + ) AS v(domain_id, register_status, use_status, has_expire_date) + WHERE d.id = v.domain_id + """ + params = [ + DETECT_STATUS_COMPLETED, + REGISTER_STATUS_AVAILABLE, + REGISTER_STATUS_AVAILABLE, + REVIEW_STATUS_PENDING, + ] + params + return len(normalized_items) if self.execute(sql, tuple(params)) else -1 def create_detect_task(self, domain_id, task_type, priority=0): """ diff --git a/domainCheck/detect/aizhan.py b/domainCheck/detect/aizhan.py index 10982c5..def1347 100644 --- a/domainCheck/detect/aizhan.py +++ b/domainCheck/detect/aizhan.py @@ -9,14 +9,41 @@ ''' import re # 正则表达式模块,用于从HTML中提取网站标题 +import os import requests # HTTP请求库,用于发送网络请求 from loguru import logger # 日志记录模块,用于记录程序运行日志 +from requests.adapters import HTTPAdapter from typing import List, Optional # 类型提示 -def check_aizhan(domain: str, sensitive_words: Optional[List[str]] = None, proxies: dict = None): +_SESSION = requests.Session() +_SESSION.mount("http://", HTTPAdapter(pool_connections=256, pool_maxsize=512, max_retries=0)) +_SESSION.mount("https://", HTTPAdapter(pool_connections=256, pool_maxsize=512, max_retries=0)) +AIZHAN_TIMEOUT_PROXY = max( + 0.8, + float(os.getenv("DOMAINCHECK_AIZHAN_TIMEOUT_PROXY", "1.6") or 1.6), +) +AIZHAN_TIMEOUT_DIRECT = max( + 1.0, + float(os.getenv("DOMAINCHECK_AIZHAN_TIMEOUT_DIRECT", "2.2") or 2.2), +) + + +def _resolve_aizhan_timeout(proxies: dict = None, budget_seconds: Optional[float] = None) -> float: + timeout = float(AIZHAN_TIMEOUT_PROXY if proxies else AIZHAN_TIMEOUT_DIRECT) + if budget_seconds not in (None, "", 0, "0"): + timeout = min(timeout, max(0.6, float(budget_seconds or 0.0))) + return max(0.6, timeout) + + +def check_aizhan( + domain: str, + sensitive_words: Optional[List[str]] = None, + proxies: dict = None, + budget_seconds: Optional[float] = None, +): ''' 查询域名的网站标题信息是否存在敏感词 @@ -43,86 +70,43 @@ def check_aizhan(domain: str, sensitive_words: Optional[List[str]] = None, proxi 'host': 'www.aizhan.com', # 目标主机地址 } - # 添加重试机制 - max_retries = 3 - for retry in range(max_retries): - try: - # 发送GET请求获取页面内容 - # url: 请求URL - # headers: HTTP请求头 - # timeout=10: 设置超时时间10秒 - # proxies: 代理配置(如需使用代理) - response = requests.get(url, headers=headers, timeout=10, proxies=proxies) + try: + # 发送GET请求获取页面内容 + # url: 请求URL + # headers: HTTP请求头 + # timeout=10: 设置超时时间10秒 + # proxies: 代理配置(如需使用代理) + response = _SESSION.get( + url, + headers=headers, + timeout=_resolve_aizhan_timeout(proxies, budget_seconds=budget_seconds), + proxies=proxies, + ) - # 判断请求是否成功(HTTP 200表示成功) - if response.status_code == 200: - # 使用正则表达式提取网站标题 - # 正则解释:匹配 id="webpage_title"> 开头,
结尾,中间的内容 - # 匹配模式:id="webpage_title">标题内容 - # ([^<]+) 表示匹配一个或多个非<字符,作为捕获组 - match = re.search(r'id="webpage_title">([^<]+)', response.text) + # 判断请求是否成功(HTTP 200表示成功) + if response.status_code == 200: + # 使用正则表达式提取网站标题 + # 正则解释:匹配 id="webpage_title"> 开头, 结尾,中间的内容 + # 匹配模式:id="webpage_title">标题内容 + # ([^<]+) 表示匹配一个或多个非<字符,作为捕获组 + match = re.search(r'id="webpage_title">([^<]+)', response.text) - # 如果找到匹配则返回标题文本,否则返回空字符串 - title = match.group(1) if match else '' - if title: - for sensitive_word in sensitive_words: - # 使用正则表达式检查是否包含敏感词 - if re.search(sensitive_word, title, re.IGNORECASE): - logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}') - return False, f'检测到敏感词: {sensitive_word} 在结果: {title}' - return True, title + # 如果找到匹配则返回标题文本,否则返回空字符串 + title = match.group(1) if match else '' + if title: + for sensitive_word in sensitive_words: + # 使用正则表达式检查是否包含敏感词 + if re.search(sensitive_word, title, re.IGNORECASE): + logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}') + return False, f'检测到敏感词: {sensitive_word} 在结果: {title}' + return True, title - # 请求失败,继续重试 - logger.warning(f"爱站网检测第{retry+1}次失败: {domain}, 状态码: {response.status_code}") - if retry < max_retries - 1: - import time - time.sleep(2) # 等待2秒后重试 - else: - # 达到最大重试次数,尝试不使用代理 - if proxies: - logger.info('爱站网检测失败,尝试不使用代理') - try: - response = requests.get(url, headers=headers, timeout=10, proxies=None) - if response.status_code == 200: - match = re.search(r'id="webpage_title">([^<]+)', response.text) - title = match.group(1) if match else '' - if title: - for sensitive_word in sensitive_words: - if re.search(sensitive_word, title, re.IGNORECASE): - logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}') - return False, f'检测到敏感词: {sensitive_word} 在结果: {title}' - return True, title - return True, '' - except Exception as e2: - logger.error(f"不使用代理的爱站网检测也失败: {domain}, 错误: {e2}") - return False, str(e2) - return True, '' + logger.debug(f"爱站网检测失败: {domain}, 状态码: {response.status_code}") + return True, '' - except Exception as e: # 捕获所有异常(网络错误、超时等) - logger.warning(f"爱站网检测第{retry+1}次失败: {domain}, 错误: {e}") - if retry < max_retries - 1: - import time - time.sleep(2) # 等待2秒后重试 - else: - # 达到最大重试次数,尝试不使用代理 - if proxies: - logger.info('爱站网检测失败,尝试不使用代理') - try: - response = requests.get(url, headers=headers, timeout=10, proxies=None) - if response.status_code == 200: - match = re.search(r'id="webpage_title">([^<]+)', response.text) - title = match.group(1) if match else '' - if title: - for sensitive_word in sensitive_words: - if re.search(sensitive_word, title, re.IGNORECASE): - logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}') - return False, f'检测到敏感词: {sensitive_word} 在结果: {title}' - return True, title - return True, '' - except Exception as e2: - logger.error(f"不使用代理的爱站网检测也失败: {domain}, 错误: {e2}") - return False, str(e2) - return False, str(e) + except Exception as e: # 捕获所有异常(网络错误、超时等) + logger.debug(f"爱站网检测失败: {domain}, 错误: {e}") + return False, str(e) # # ========================= 【测试代码】 ========================= # if __name__ == '__main__': @@ -152,4 +136,4 @@ def check_aizhan(domain: str, sensitive_words: Optional[List[str]] = None, proxi # logger.info(f"查询域名: {test_domain_4}") # # result_4 = check_aizhan(test_domain_4) -# logger.info(f"网站标题: {result_4 if result_4 else '(无结果)'}") \ No newline at end of file +# logger.info(f"网站标题: {result_4 if result_4 else '(无结果)'}") diff --git a/domainCheck/detect/baidu.py b/domainCheck/detect/baidu.py index cce6a66..42d539d 100644 --- a/domainCheck/detect/baidu.py +++ b/domainCheck/detect/baidu.py @@ -8,15 +8,121 @@ @explain : 百度域名检测工具 - 包含百度搜索结果查询和百度安全API检测功能 ''' +import os import random # 随机数生成模块,用于随机选择浏览器类型 import re # 正则表达式模块,用于从HTML中提取搜索结果 +import threading import time from loguru import logger # 日志记录模块,用于记录程序运行日志 from curl_cffi import requests # curl_cffi库,支持浏览器模拟的HTTP请求 from typing import List, Optional, Union, Any # 类型提示 -def check_site(domain: str, sensitive_words: Optional[List[str]] = None, proxies: dict = None) -> Union[ +BAIDU_SITE_TIMEOUT_PROXY = max( + 0.8, + float(os.getenv("DOMAINCHECK_BAIDU_TIMEOUT_PROXY", "1.4") or 1.4), +) +BAIDU_SITE_TIMEOUT_DIRECT = max( + 1.0, + float(os.getenv("DOMAINCHECK_BAIDU_TIMEOUT_DIRECT", "1.8") or 1.8), +) +BAIDU_SCAN_TIMEOUT_PROXY = max( + 1.0, + float(os.getenv("DOMAINCHECK_BAIDU_SCAN_TIMEOUT_PROXY", "1.6") or 1.6), +) +BAIDU_SCAN_TIMEOUT_DIRECT = max( + 1.2, + float(os.getenv("DOMAINCHECK_BAIDU_SCAN_TIMEOUT_DIRECT", "2.0") or 2.0), +) +BAIDU_SITE_URL = 'https://m.baidu.com/s' +BAIDU_SAFE_SCAN_URL = 'https://mobsec-sec.baidu.com/3.1/scanurl' +BAIDU_SITE_PATTERN = r'([^<]+)' +_THREAD_LOCAL = threading.local() + + +def _resolve_baidu_timeout(proxies: dict = None, budget_seconds: Optional[float] = None, *, scan: bool = False) -> float: + if scan: + timeout = BAIDU_SCAN_TIMEOUT_PROXY if proxies else BAIDU_SCAN_TIMEOUT_DIRECT + else: + timeout = BAIDU_SITE_TIMEOUT_PROXY if proxies else BAIDU_SITE_TIMEOUT_DIRECT + timeout = float(timeout or 0.0) + if budget_seconds not in (None, "", 0, "0"): + timeout = min(timeout, max(0.6, float(budget_seconds or 0.0))) + return max(0.6, timeout) + +BAIDU_SITE_COOKIES = { + 'BIDUPSID': '053DBE4D820C6EFB729DC7B13B1F82B2', + 'PSTM': '1775657119', + 'H_PS_PSSID': '63148_67862_67986_68002_68142_68148_68152_68141_68165_68189_68226_68267_68296_68336_68369_68453_68438_68464_68541_68546_68558_68520_68589_68621_68615_68606_68601_68682_68671_68735_68544_68733_68766_68807_68901_68918_68836_68921_68955_68976_68997_69007_69010_69018_69024_69014', + 'BAIDUID': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1', + 'BDORZ': 'B490B5EBF6F3CD402E515D22BCDA1598', + 'BAIDUID_BFESS': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1', + 'BA_HECTOR': 'ah0l24210la4050l0g0k0g2g0h20071ktcocj27', + 'ZFY': 'Qxyr75Xm7o8zUxYzuFnYoW7cnS:AnVp:BpnrNVkr3usno:C', + 'delPer': '0', + 'BAIDUID_REF': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1', + 'H_WISE_SIDS': '110085_661771_667681_673676_683089_682564_685373_660925_687556_686285_690306_690303_690478_690576_690334_691545_685595_690883_692754_692777_692118_692910_693221_693364_693403_693656_693941_694124_693392_694171_694236_693996_694316_694324_693886_694409_694379_694696_694780_694783_692006_694884_694918_694996_694991_695032_695118_694840_695140_695193_695110_695262_688610_694933_695278_694865_695214_695374_690651_694577_692378_695452_695457_695480_695603_695642_695631_695668_695824_695843_695866_695888_695892_695897_695975_694177_695900_695939_696145_696150_696153_696069_696078_696066_694359_696288_696308_694985_696297_695715_696128_696313_696457_696431_696473_696492_696590_693385_696610_696682_696679_696662_696651_696655_696658_696673_696643_696110_696729_696733_696772_8000116_8000133_8000138_8000159_8000163_8000167_8000176_8000186_8000190_8000204', + 'H_WISE_SIDS_BFESS': '110085_661771_667681_673676_683089_682564_685373_660925_687556_686285_690306_690303_690478_690576_690334_691545_685595_690883_692754_692777_692118_692910_693221_693364_693403_693656_693941_694124_693392_694171_694236_693996_694316_694324_693886_694409_694379_694696_694780_694783_692006_694884_694918_694996_694991_695032_695118_694840_695140_695193_695110_695262_688610_694933_695278_694865_695214_695374_690651_694577_692378_695452_695457_695480_695603_695642_695631_695668_695824_695843_695866_695888_695892_695897_695975_694177_695900_695939_696145_696150_696153_696069_696078_696066_694359_696288_696308_694985_696297_695715_696128_696313_696457_696431_696473_696492_696590_693385_696610_696682_696679_696662_696651_696655_696658_696673_696643_696110_696729_696733_696772_8000116_8000133_8000138_8000159_8000163_8000167_8000176_8000186_8000190_8000204', + 'MSA_PBT': '147', + 'MSA_ZOOM': '1000', + 'wpr': '0', + 'COOKIE_SESSION': '0_0_0_0_0_0_0_0_0_0_0_0_0_1775657396%7C1%230_0_0_0_0_0_0_0_1775657396%7C1', + 'MSA_PHY_WH': '1440_3440', + 'POLYFILL': '0', + 'MSA_WH': '1254_940', + 'kleck': '7ce2abac229d2e8ba435a8bcc6256f57f53e9d08954443bf', + 'PSCBD': '16%3A1%3A3', + 'SE_LAUNCH': '5%3A1775657396_16%3A29594323%3A3', + 'BDSVRTM': '3', + 'PSINO': '6', + '__bsi': '17976785662214065461_00_7_R_R_6_0303_c02f_Y', +} + +BAIDU_SITE_HEADERS = { + 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7', + 'Accept-Language': 'zh-CN,zh;q=0.9', + 'Connection': 'keep-alive', + 'Sec-Fetch-Dest': 'document', + 'Sec-Fetch-Mode': 'navigate', + 'Sec-Fetch-Site': 'none', + 'Sec-Fetch-User': '?1', + 'Upgrade-Insecure-Requests': '1', + 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0', + 'sec-ch-ua': '"Chromium";v="146", "Not-A.Brand";v="24", "Microsoft Edge";v="146"', + 'sec-ch-ua-mobile': '?0', + 'sec-ch-ua-platform': '"Windows"', +} + +BAIDU_SAFE_SCAN_HEADERS = { + 'Cache-Control': 'no-cache', + 'Content-Type': 'application/json; charset=utf-8', + 'Host': 'mobsec-sec.baidu.com', + 'User-Agent': 'okhttp/3.12.12', +} + +BAIDU_SAFE_SCAN_PARAMS = { + 'auth_ver': '2', + 'appkey': '4665fd2c6b0922a551e8ae74', + 'nonce': '1751309255340', + 'lc': '77qHTv4VtmRiXYtd', + 'pkg': 'com.baidu.searchbox', + 'vc': '-1', + 'cuid': 'CF0E6FCCD824D146605C16AC1C8BAC95%7CVFWAO56TC', + 'tk': '', + 'type': '3', + 'vn': '2.6.0', + 's': 'fdfef2ce96c1c7193f9b80425020f63e', +} + + +def _get_thread_session(): + session = getattr(_THREAD_LOCAL, "session", None) + if session is None: + session = requests.Session() + _THREAD_LOCAL.session = session + return session + +def check_site(domain: str, sensitive_words: Optional[List[str]] = None, proxies: dict = None, budget_seconds: Optional[float] = None) -> Union[ None, tuple[bool, str], list[Any]]: ''' 通过百度搜索查询域名的相关信息 @@ -31,80 +137,41 @@ def check_site(domain: str, sensitive_words: Optional[List[str]] = None, proxies if sensitive_words is None: sensitive_words = [] - # 添加重试机制 - max_retries = 3 - for retry in range(max_retries): - try: # try-except块,用于捕获网络请求中的异常 + try: # try-except块,用于捕获网络请求中的异常 # 构建搜索查询参数 # wd: 搜索关键词,格式为site:域名 params = { 'wd': f'site:{domain}' # 搜索site:domain,限制搜索结果为指定域名 } - cookies = { - 'BIDUPSID': '053DBE4D820C6EFB729DC7B13B1F82B2', - 'PSTM': '1775657119', - 'H_PS_PSSID': '63148_67862_67986_68002_68142_68148_68152_68141_68165_68189_68226_68267_68296_68336_68369_68453_68438_68464_68541_68546_68558_68520_68589_68621_68615_68606_68601_68682_68671_68735_68544_68733_68766_68807_68901_68918_68836_68921_68955_68976_68997_69007_69010_69018_69024_69014', - 'BAIDUID': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1', - 'BDORZ': 'B490B5EBF6F3CD402E515D22BCDA1598', - 'BAIDUID_BFESS': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1', - 'BA_HECTOR': 'ah0l24210la4050l0g0k0g2g0h20071ktcocj27', - 'ZFY': 'Qxyr75Xm7o8zUxYzuFnYoW7cnS:AnVp:BpnrNVkr3usno:C', - 'delPer': '0', - 'BAIDUID_REF': '053DBE4D820C6EFB729DC7B13B1F82B2:FG=1', - 'H_WISE_SIDS': '110085_661771_667681_673676_683089_682564_685373_660925_687556_686285_690306_690303_690478_690576_690334_691545_685595_690883_692754_692777_692118_692910_693221_693364_693403_693656_693941_694124_693392_694171_694236_693996_694316_694324_693886_694409_694379_694696_694780_694783_692006_694884_694918_694996_694991_695032_695118_694840_695140_695193_695110_695262_688610_694933_695278_694865_695214_695374_690651_694577_692378_695452_695457_695480_695603_695642_695631_695668_695824_695843_695866_695888_695892_695897_695975_694177_695900_695939_696145_696150_696153_696069_696078_696066_694359_696288_696308_694985_696297_695715_696128_696313_696457_696431_696473_696492_696590_693385_696610_696682_696679_696662_696651_696655_696658_696673_696643_696110_696729_696733_696772_8000116_8000133_8000138_8000159_8000163_8000167_8000176_8000186_8000190_8000204', - 'H_WISE_SIDS_BFESS': '110085_661771_667681_673676_683089_682564_685373_660925_687556_686285_690306_690303_690478_690576_690334_691545_685595_690883_692754_692777_692118_692910_693221_693364_693403_693656_693941_694124_693392_694171_694236_693996_694316_694324_693886_694409_694379_694696_694780_694783_692006_694884_694918_694996_694991_695032_695118_694840_695140_695193_695110_695262_688610_694933_695278_694865_695214_695374_690651_694577_692378_695452_695457_695480_695603_695642_695631_695668_695824_695843_695866_695888_695892_695897_695975_694177_695900_695939_696145_696150_696153_696069_696078_696066_694359_696288_696308_694985_696297_695715_696128_696313_696457_696431_696473_696492_696590_693385_696610_696682_696679_696662_696651_696655_696658_696673_696643_696110_696729_696733_696772_8000116_8000133_8000138_8000159_8000163_8000167_8000176_8000186_8000190_8000204', - 'MSA_PBT': '147', - 'MSA_ZOOM': '1000', - 'wpr': '0', - 'COOKIE_SESSION': '0_0_0_0_0_0_0_0_0_0_0_0_0_1775657396%7C1%230_0_0_0_0_0_0_0_1775657396%7C1', - 'MSA_PHY_WH': '1440_3440', - 'POLYFILL': '0', - 'MSA_WH': '1254_940', - 'kleck': '7ce2abac229d2e8ba435a8bcc6256f57f53e9d08954443bf', - 'PSCBD': '16%3A1%3A3', - 'SE_LAUNCH': '5%3A1775657396_16%3A29594323%3A3', - 'BDSVRTM': '3', - 'PSINO': '6', - '__bsi': '17976785662214065461_00_7_R_R_6_0303_c02f_Y', - } - - headers = { - 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7', - 'Accept-Language': 'zh-CN,zh;q=0.9', - 'Connection': 'keep-alive', - 'Sec-Fetch-Dest': 'document', - 'Sec-Fetch-Mode': 'navigate', - 'Sec-Fetch-Site': 'none', - 'Sec-Fetch-User': '?1', - 'Upgrade-Insecure-Requests': '1', - 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0', - 'sec-ch-ua': '"Chromium";v="146", "Not-A.Brand";v="24", "Microsoft Edge";v="146"', - 'sec-ch-ua-mobile': '?0', - 'sec-ch-ua-platform': '"Windows"', - } - - # 百度搜索URL - url = 'https://m.baidu.com/s' - - # 发送GET请求获取搜索结果 - # url: 请求URL - # params: 查询参数 - # headers: HTTP请求头 - # timeout=15: 超时时间15秒 - # proxies: 代理配置 - # impersonate: 模拟的浏览器类型 - # .content: 获取响应内容(字节流) - # .decode('utf-8'): 解码为UTF-8字符串 - response_text = requests.get(url, params=params, headers=headers, cookies=cookies, timeout=15, - proxies=proxies).content.decode('utf-8') + session = _get_thread_session() + request_started_at = time.perf_counter() + logger.info( + f"百度site阶段: domain={domain} | stage=request_start | proxy={'yes' if proxies else 'no'}" + ) + response = session.get( + BAIDU_SITE_URL, + params=params, + headers=BAIDU_SITE_HEADERS, + cookies=BAIDU_SITE_COOKIES, + timeout=_resolve_baidu_timeout(proxies, budget_seconds=budget_seconds, scan=False), + proxies=proxies, + ) + logger.info( + f"百度site阶段: domain={domain} | stage=request_done | status={response.status_code} " + f"| elapsed_ms={int((time.perf_counter() - request_started_at) * 1000)}" + ) + response_text = response.content.decode('utf-8', errors='ignore') # 使用正则表达式提取搜索结果标题 # re.findall()返回所有匹配的结果列表 # 正则解释:匹配 之间的内容 - match = re.findall(r'([^<]+)', response_text,re.S) + match = re.findall(BAIDU_SITE_PATTERN, response_text, re.S) # 返回匹配结果列表,如果没有匹配则返回空列表 search_results= match if match else [] + logger.info( + f"百度site阶段: domain={domain} | stage=parsed | result_count={len(search_results)}" + ) # 如果没有搜索结果,直接返回通过 if not search_results: @@ -120,19 +187,12 @@ def check_site(domain: str, sensitive_words: Optional[List[str]] = None, proxies # 所有搜索结果都不包含敏感词,返回通过 return True, '' - except Exception as e: # 捕获所有异常 - # logger.warning(f"百度site检测第{retry+1}次失败: {domain}, 错误: {e}") - if retry < max_retries - 1: - import time - time.sleep(2) # 等待2秒后重试 - proxies=None - else: - # 达到最大重试次数,返回检测失败 - logger.error(f"百度site检测失败: {domain}, 已达到最大重试次数") - return False, str(e) # 返回错误信息 + except Exception as e: # 捕获所有异常 + logger.error(f"百度site检测失败: {domain}, 错误: {e}") + return False, str(e) -def baidu(domain: str, proxies: dict = None) -> str: +def baidu(domain: str, proxies: dict = None, budget_seconds: Optional[float] = None) -> str: ''' 通过百度安全API检测域名安全状态 @@ -142,29 +202,6 @@ def baidu(domain: str, proxies: dict = None) -> str: :param proxies: 代理配置(如:{'http': 'http://127.0.0.1:7890', 'https': 'http://127.0.0.1:7890'} :return: str - 安全状态描述("风险网站提示"、"高危网站提示"或"正常"),如果检测失败则返回错误信息 ''' - # 构建HTTP请求头 - headers = { - 'Cache-Control': 'no-cache', # 不使用缓存 - 'Content-Type': 'application/json; charset=utf-8', # 内容类型为JSON - 'Host': 'mobsec-sec.baidu.com', # 目标主机 - 'User-Agent': 'okhttp/3.12.12', # 模拟OKHttp客户端 - } - - # 构建请求参数 - params = { - 'auth_ver': '2', # 认证版本 - 'appkey': '4665fd2c6b0922a551e8ae74', # API应用密钥 - 'nonce': '1751309255340', # 随机数(注意:实际使用时应该动态生成) - 'lc': '77qHTv4VtmRiXYtd', # 本地配置参数 - 'pkg': 'com.baidu.searchbox', # 应用包名 - 'vc': '-1', # 版本码 - 'cuid': 'CF0E6FCCD824D146605C16AC1C8BAC95%7CVFWAO56TC', # 设备ID(注意:实际使用时应该动态生成) - 'tk': '', # 令牌(空) - 'type': '3', # 检测类型 - 'vn': '2.6.0', # 版本号 - 's': 'fdfef2ce96c1c7193f9b80425020f63e', # 签名(注意:实际使用时应该动态生成) - } - # 构建JSON请求数据 json_data = { 'url': f'https://{domain}/', # 待检测的URL(添加https协议头) @@ -173,10 +210,7 @@ def baidu(domain: str, proxies: dict = None) -> str: # 提取URL作为后续获取结果的键 key = json_data['url'] - # 添加重试机制 - max_retries = 3 - for retry in range(max_retries): - try: # try-except块,用于捕获API调用中的异常 + try: # try-except块,用于捕获API调用中的异常 # 发送POST请求到百度安全API # url: API接口地址 # params: 查询参数 @@ -184,8 +218,24 @@ def baidu(domain: str, proxies: dict = None) -> str: # json: JSON请求数据 # proxies: 代理配置 # .json(): 解析JSON响应 - response = requests.post('https://mobsec-sec.baidu.com/3.1/scanurl', params=params, headers=headers, - json=json_data, proxies=proxies, timeout=15).json() + session = _get_thread_session() + request_started_at = time.perf_counter() + logger.info( + f"百度安全阶段: domain={domain} | stage=request_start | proxy={'yes' if proxies else 'no'}" + ) + response = session.post( + BAIDU_SAFE_SCAN_URL, + params=BAIDU_SAFE_SCAN_PARAMS, + headers=BAIDU_SAFE_SCAN_HEADERS, + json=json_data, + proxies=proxies, + timeout=_resolve_baidu_timeout(proxies, budget_seconds=budget_seconds, scan=True), + ) + logger.info( + f"百度安全阶段: domain={domain} | stage=request_done | status={response.status_code} " + f"| elapsed_ms={int((time.perf_counter() - request_started_at) * 1000)}" + ) + response = response.json() # # 记录API响应(调试用) # logger.info(response) @@ -202,27 +252,9 @@ def baidu(domain: str, proxies: dict = None) -> str: # msgs.get(): 根据安全等级获取对应的状态描述,默认值为'正常' return msgs.get(response['response']['datas'][key].get('grand_level', 'level'), '正常') - except Exception as e: # 捕获所有异常 - logger.warning(f"百度网页安全中心检测第{retry+1}次失败: {domain}, 错误: {e}") - if retry < max_retries - 1: - time.sleep(2) # 等待2秒后重试 - else: - # 达到最大重试次数,尝试不使用代理 - if proxies: - logger.info('百度网页安全中心检测失败,尝试不使用代理') - try: - response = requests.post('https://mobsec-sec.baidu.com/3.1/scanurl', params=params, headers=headers, - json=json_data, proxies=None, timeout=15).json() - msgs = { - 2193: '风险网站提示', # 风险网站 - 2243: '高危网站提示', # 高危网站 - '_': '正常' # 默认正常 - } - return msgs.get(response['response']['datas'][key].get('grand_level', 'level'), '正常') - except Exception as e2: - logger.error(f"不使用代理的百度网页安全中心检测也失败: {domain}, 错误: {e2}") - return str(e2) # 返回错误信息 - return str(e) # 返回错误信息 + except Exception as e: # 捕获所有异常 + logger.error(f"百度网页安全中心检测失败: {domain}, 错误: {e}") + return str(e) # # ========================= 【测试代码】 ========================= @@ -260,4 +292,4 @@ def baidu(domain: str, proxies: dict = None) -> str: # logger.info(f"搜索结果标题数: {len(result_1)}") # for i, title in enumerate(result_1, 1): # logger.info(f" 结果{i}: {title}") - # time.sleep(1) \ No newline at end of file + # time.sleep(1) diff --git a/domainCheck/detect/c360.py b/domainCheck/detect/c360.py index 3899cea..aa0ac95 100644 --- a/domainCheck/detect/c360.py +++ b/domainCheck/detect/c360.py @@ -9,8 +9,12 @@ ''' import re # 正则表达式模块 +import os +import threading +import time import requests # HTTP请求库 from loguru import logger # 日志记录 +from requests.adapters import HTTPAdapter from typing import List, Tuple, Optional, Dict # 类型提示 @@ -19,12 +23,40 @@ SO_SEARCH_URL = 'https://www.so.com/s' # 360搜索URL SO_REFERER_TEMPLATE = 'https://www.so.com/s?ie=utf-8&q=site%3A{domain}' # Referer模板 SEARCH_PATTERN = r'target="_blank">([^<]+)' # 搜索结果匹配模式 BLOCKED_CODE = 3 # 拦截状态码 +_SESSION_LOCAL = threading.local() +SO_TIMEOUT_PROXY = max( + 1.0, + float(os.getenv("DOMAINCHECK_360_TIMEOUT_PROXY", "1.4") or 1.4), +) +SO_TIMEOUT_DIRECT = max( + 1.2, + float(os.getenv("DOMAINCHECK_360_TIMEOUT_DIRECT", "1.8") or 1.8), +) + + +def _resolve_360_timeout(proxies: Optional[Dict] = None, budget_seconds: Optional[float] = None) -> float: + timeout = float(SO_TIMEOUT_PROXY if proxies else SO_TIMEOUT_DIRECT) + if budget_seconds not in (None, "", 0, "0"): + timeout = min(timeout, max(0.6, float(budget_seconds or 0.0))) + return max(0.6, timeout) + + +def _get_session(): + session = getattr(_SESSION_LOCAL, "session", None) + if session is not None: + return session + session = requests.Session() + session.mount("http://", HTTPAdapter(pool_connections=64, pool_maxsize=128, max_retries=0)) + session.mount("https://", HTTPAdapter(pool_connections=64, pool_maxsize=128, max_retries=0)) + _SESSION_LOCAL.session = session + return session def check_domain( domain: str, sensitive_words: Optional[List[str]] = None, - proxies: Optional[Dict] = None + proxies: Optional[Dict] = None, + budget_seconds: Optional[float] = None, ) -> Tuple[bool, str]: """ 检查域名是否包含敏感词 @@ -60,125 +92,65 @@ def check_domain( 'q': f'site:{domain}', # 搜索查询 } - # 添加重试机制 - max_retries = 3 - for retry in range(max_retries): - try: - # 发送GET请求获取搜索结果 - response = requests.get( - SO_SEARCH_URL, - params=params, - headers=headers, - proxies=proxies, - timeout=15 # 增加超时时间到15秒 - ) + try: + session_started_at = time.perf_counter() + logger.info(f'360阶段: domain={domain} | stage=session_init_start') + session = _get_session() + logger.info( + f'360阶段: domain={domain} | stage=session_init_done ' + f'| elapsed_ms={int((time.perf_counter() - session_started_at) * 1000)}' + ) + # 发送GET请求获取搜索结果 + request_started_at = time.perf_counter() + logger.info(f'360阶段: domain={domain} | stage=request_start | proxy={"yes" if proxies else "no"}') + response = session.get( + SO_SEARCH_URL, + params=params, + headers=headers, + proxies=proxies, + timeout=_resolve_360_timeout(proxies, budget_seconds=budget_seconds), + ) + logger.info( + f'360阶段: domain={domain} | stage=request_done | status={response.status_code} ' + f'| elapsed_ms={int((time.perf_counter() - request_started_at) * 1000)}' + ) - # 解析响应内容 - response_html = response.content.decode('utf-8', errors='ignore') + # 解析响应内容 + response_html = response.content.decode('utf-8', errors='ignore') - # 使用正则表达式提取搜索结果 - search_results = re.findall(SEARCH_PATTERN, response_html) + # 使用正则表达式提取搜索结果 + search_results = re.findall(SEARCH_PATTERN, response_html) + logger.info(f'360阶段: domain={domain} | stage=parsed | result_count={len(search_results)}') - # 记录搜索结果数量 - logger.info(f'360搜索结果数量: {len(search_results)}') + # 记录搜索结果数量 + logger.info(f'360搜索结果数量: {len(search_results)}') - # 如果没有搜索结果,直接返回通过 - if not search_results: - return True, '' - - # 检查每个搜索结果是否包含敏感词 - for result in search_results: - for sensitive_word in sensitive_words: - # 使用正则表达式检查是否包含敏感词 - if re.search(sensitive_word, result, re.IGNORECASE): - logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {result}') - return False, f'检测到敏感词: {sensitive_word} 在结果: {result}' - - # 所有搜索结果都不包含敏感词,返回通过 + # 如果没有搜索结果,直接返回通过 + if not search_results: return True, '' - except requests.exceptions.Timeout: - logger.warning(f'360搜索请求第{retry+1}次超时') - if retry < max_retries - 1: - import time - time.sleep(2) # 等待2秒后重试 - else: - # 达到最大重试次数,尝试不使用代理 - if proxies: - logger.info('360搜索请求超时,尝试不使用代理') - try: - response = requests.get( - SO_SEARCH_URL, - params=params, - headers=headers, - proxies=None, - timeout=15 - ) - response_html = response.content.decode('utf-8', errors='ignore') - search_results = re.findall(SEARCH_PATTERN, response_html) - logger.info(f'不使用代理的360搜索结果数量: {len(search_results)}') - if not search_results: - return True, '' - for result in search_results: - for sensitive_word in sensitive_words: - if re.search(sensitive_word, result, re.IGNORECASE): - logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {result}') - return False, f'检测到敏感词: {sensitive_word} 在结果: {result}' - return True, '' - except Exception as e: - logger.error(f'不使用代理的360搜索请求也失败: {str(e)}') - return False, f'360搜索请求超时' - return False, '360搜索请求超时' - except requests.exceptions.RequestException as e: - error_str = str(e) - logger.warning(f'360搜索请求第{retry+1}次失败: {error_str}') - # 检查是否是"Too many open files"错误 - if 'Too many open files' in error_str: - if retry < max_retries - 1: - import time - time.sleep(2) # 等待2秒后重试 - else: - # 达到最大重试次数,返回failure标记,不拉黑域名 - logger.error(f'360搜索请求失败,已达到最大重试次数: {error_str}') - return True, 'failure' - else: - if retry < max_retries - 1: - import time - time.sleep(2) # 等待2秒后重试 - else: - # 达到最大重试次数,尝试不使用代理 - if proxies: - logger.info('360搜索请求失败,尝试不使用代理') - try: - response = requests.get( - SO_SEARCH_URL, - params=params, - headers=headers, - proxies=None, - timeout=15 - ) - response_html = response.content.decode('utf-8', errors='ignore') - search_results = re.findall(SEARCH_PATTERN, response_html) - logger.info(f'不使用代理的360搜索结果数量: {len(search_results)}') - if not search_results: - return True, '' - for result in search_results: - for sensitive_word in sensitive_words: - if re.search(sensitive_word, result, re.IGNORECASE): - logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {result}') - return False, f'检测到敏感词: {sensitive_word} 在结果: {result}' - return True, '' - except Exception as e2: - logger.error(f'不使用代理的360搜索请求也失败: {str(e2)}') - return False, f'360搜索请求失败: {str(e)}' - return False, f'360搜索请求失败: {str(e)}' - except Exception as e: - logger.warning(f'360搜索第{retry+1}次未知错误: {str(e)}') - if retry < max_retries - 1: - import time - time.sleep(2) # 等待2秒后重试 - else: - return False, f'未知错误: {str(e)}' + # 检查每个搜索结果是否包含敏感词 + for result in search_results: + for sensitive_word in sensitive_words: + # 使用正则表达式检查是否包含敏感词 + if re.search(sensitive_word, result, re.IGNORECASE): + logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {result}') + return False, f'检测到敏感词: {sensitive_word} 在结果: {result}' + + # 所有搜索结果都不包含敏感词,返回通过 + return True, '' + + except requests.exceptions.Timeout: + return False, '360搜索请求超时' + except requests.exceptions.RequestException as e: + error_str = str(e) + logger.debug(f'360搜索请求失败: {error_str}') + # 检查是否是"Too many open files"错误 + if 'Too many open files' in error_str: + return True, 'failure' + return False, f'360搜索请求失败: {error_str}' + except Exception as e: + return False, f'未知错误: {str(e)}' # if __name__ == '__main__': @@ -188,4 +160,4 @@ def check_domain( # # logger.info(f'开始检测域名: {test_domain}') # result = check_domain(test_domain, test_sensitive_words) -# logger.info(f'检测结果: {result}') \ No newline at end of file +# logger.info(f'检测结果: {result}') diff --git a/domainCheck/detect/chinaz.py b/domainCheck/detect/chinaz.py index 24cfec1..e034049 100644 --- a/domainCheck/detect/chinaz.py +++ b/domainCheck/detect/chinaz.py @@ -11,15 +11,50 @@ import json # JSON处理模块,用于解析API响应数据 import os # 操作系统接口模块,用于获取脚本所在目录路径 import re # 正则表达式模块,用于提取网页中的标题和加密密钥 +import threading import time # 时间处理模块,用于生成时间戳 from typing import List, Optional # 类型提示 import quickjs # JavaScript执行引擎,用于执行JS加密代码生成签名 import requests # HTTP请求库,用于发送网络请求 from loguru import logger # 日志记录模块,用于记录程序运行日志 +from requests.adapters import HTTPAdapter -def check_title(domain: str,sensitive_words: Optional[List[str]] = None, proxies: dict = None): +_CHINAZ_JS_CACHE = None +_SESSION_LOCAL = threading.local() + + +def _resolve_chinaz_timeout(proxies: dict = None, budget_seconds: Optional[float] = None) -> float: + timeout = float(1.8 if proxies else 2.8) + if budget_seconds not in (None, "", 0, "0"): + timeout = min(timeout, max(0.6, float(budget_seconds or 0.0))) + return max(0.6, timeout) + + +def _get_session(): + session = getattr(_SESSION_LOCAL, "session", None) + if session is not None: + return session + session = requests.Session() + session.mount("http://", HTTPAdapter(pool_connections=32, pool_maxsize=64, max_retries=0)) + session.mount("https://", HTTPAdapter(pool_connections=32, pool_maxsize=64, max_retries=0)) + _SESSION_LOCAL.session = session + return session + + +def _load_chinaz_js(): + global _CHINAZ_JS_CACHE + if _CHINAZ_JS_CACHE is not None: + return _CHINAZ_JS_CACHE + current_dir = os.path.dirname(__file__) + js_path = os.path.join(current_dir, "chinaz.js") + with open(js_path, "r", encoding="utf-8") as f: + _CHINAZ_JS_CACHE = f.read() + return _CHINAZ_JS_CACHE + + +def check_title(domain: str, sensitive_words: Optional[List[str]] = None, proxies: dict = None, budget_seconds: Optional[float] = None): ''' 检测域名标题并获取SEO数据 @@ -31,105 +66,133 @@ def check_title(domain: str,sensitive_words: Optional[List[str]] = None, proxies if sensitive_words is None: sensitive_words = [] - # 添加重试机制 - max_retries = 3 - for retry in range(max_retries): - try: - # 构建SEO查询URL(站长工具网站) - url = f'https://seo.chinaz.com/{domain}' - - # 发送GET请求获取页面内容,设置超时10秒 - response = requests.get(url, timeout=10, proxies=proxies) - - # 判断请求是否成功(HTTP 200表示成功) - if response.status_code == 200: - # 使用正则表达式提取网站标题(从id="site_title"的div标签中提取文本内容) - # 正则解释:匹配 id="site_title"> 开头, 结尾,中间的内容 - match = re.search(r'id="site_title">([^<]+)', response.text) - - # 使用正则表达式提取加密密钥enkey(用于后续API请求的身份验证) - # 正则解释:匹配 var enkey = '...' 中的单引号内容 - pattern = r"var enkey = '([^']+)''" + try: + step_started_at = time.perf_counter() + print( + f"[chinaz-entry] domain={domain} proxy={'yes' if proxies else 'no'}", + flush=True, + ) + session_started_at = time.perf_counter() + logger.info(f"站长之家阶段: domain={domain} | stage=session_init_start") + session = _get_session() + logger.info( + f"站长之家阶段: domain={domain} | stage=session_init_done " + f"| elapsed_ms={int((time.perf_counter() - session_started_at) * 1000)}" + ) + # 构建SEO查询URL(站长工具网站) + url = f'https://seo.chinaz.com/{domain}' + page_started_at = time.perf_counter() + logger.info( + f"站长之家阶段: domain={domain} | stage=page_request_start | proxy={'yes' if proxies else 'no'}" + ) + + # 发送GET请求获取页面内容,设置超时并收口到步骤剩余预算内 + response = session.get( + url, + timeout=_resolve_chinaz_timeout(proxies, budget_seconds=budget_seconds), + proxies=proxies, + ) + logger.info( + f"站长之家阶段: domain={domain} | stage=page_request_done | status={response.status_code} " + f"| elapsed_ms={int((time.perf_counter() - page_started_at) * 1000)}" + ) + + # 判断请求是否成功(HTTP 200表示成功) + if response.status_code == 200: + # 使用正则表达式提取网站标题(从id="site_title"的div标签中提取文本内容) + # 正则解释:匹配 id="site_title"> 开头, 结尾,中间的内容 + match = re.search(r'id="site_title">([^<]+)', response.text) + + # 使用正则表达式提取加密密钥enkey(用于后续API请求的身份验证) + # 正则解释:匹配 var enkey = '...' 中的单引号内容 + pattern = r"var enkey = '([^']+)''" + match_enkey = re.search(pattern, response.text) + + # 如果上面的正则匹配失败,尝试匹配正确的单引号版本 + if not match_enkey: + pattern = r"var enkey = '([^']+)" # 匹配 var enkey = '...' 格式 match_enkey = re.search(pattern, response.text) - - # 如果上面的正则匹配失败,尝试匹配正确的单引号版本 - if not match_enkey: - pattern = r"var enkey = '([^']+)" # 匹配 var enkey = '...' 格式 - match_enkey = re.search(pattern, response.text) - - # 如果找到enkey则提取并去除首尾空格,否则为空字符串 - enkey = match_enkey.group(1).strip() if match_enkey else '' - # 提取网站标题,如果找到则返回,否则返回空字符串 - title = match.group(1).strip() if match else '' + # 如果找到enkey则提取并去除首尾空格,否则为空字符串 + enkey = match_enkey.group(1).strip() if match_enkey else '' - if title: - for sensitive_word in sensitive_words: - # 使用正则表达式检查是否包含敏感词 - if re.search(sensitive_word, title, re.IGNORECASE): - logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}') - return False, f'检测到敏感词: {sensitive_word} 在结果: {title}', () + # 提取网站标题,如果找到则返回,否则返回空字符串 + title = match.group(1).strip() if match else '' + logger.info( + f"站长之家阶段: domain={domain} | stage=page_parsed | title={'yes' if bool(title) else 'no'} " + f"| enkey={'yes' if bool(enkey) else 'no'}" + ) - # 调用get_site_data获取详细SEO数据 - success, result = get_site_data(domain, enkey, proxies=proxies) - - # 检查网站分类 - if success and result: - # 定义需要拉黑的分类 - blacklist_categories = ['视频电影', '体育运动', '常用查询', '游戏网站', '游戏', '视频', '体育'] - - # 检查分类是否在黑名单中 - if isinstance(result, dict): - # 检查可能的分类字段 - category_fields = ['Category', 'category', '分类', '网站分类'] - for field in category_fields: - if field in result: - category = result[field] - if isinstance(category, str): - for blacklist_category in blacklist_categories: - if blacklist_category in category: - logger.warning(f'检测到黑名单分类: {category} 在域名: {domain}') - return False, f'网站分类: {category} 属于黑名单分类', () - - # 检查Result是否为列表 - elif isinstance(result, list): - for item in result: - if isinstance(item, dict): - category_fields = ['Category', 'category', '分类', '网站分类'] - for field in category_fields: - if field in item: - category = item[field] - if isinstance(category, str): - for blacklist_category in blacklist_categories: - if blacklist_category in category: - logger.warning(f'检测到黑名单分类: {category} 在域名: {domain}') - return False, f'网站分类: {category} 属于黑名单分类', () - - return True, 'success', (success, result) - - # 请求失败返回空标题和空元组 - return True, '', () - - except Exception as e: # 捕获所有异常 - error_str = str(e) - # 检查是否是需要重试的错误类型 - if any(error_type in error_str for error_type in ['Too many open files', 'EOF occurred in violation of protocol', 'Max retries exceeded', 'Read timed out']): - logger.warning(f"站长工具检测第{retry+1}次失败: {domain}, 错误: {e}") - if retry < max_retries - 1: - time.sleep(2) # 等待2秒后重试 - continue - else: - # 达到最大重试次数,返回失败但不拉黑 - logger.error(f"站长工具检测失败,已达到最大重试次数: {domain}, 错误: {e}") - return True, 'failure', (False, str(e)) - else: - # 其他错误直接返回 - logger.error(f"站长工具检测失败: {domain}, 错误: {e}") - return False, str(e), () + if title: + for sensitive_word in sensitive_words: + # 使用正则表达式检查是否包含敏感词 + if re.search(sensitive_word, title, re.IGNORECASE): + logger.warning(f'检测到敏感词: {sensitive_word} 在结果: {title}') + return False, f'检测到敏感词: {sensitive_word} 在结果: {title}', () + + # 调用get_site_data获取详细SEO数据 + seo_started_at = time.perf_counter() + logger.info(f"站长之家阶段: domain={domain} | stage=seo_request_start") + remaining_budget_seconds = None + if budget_seconds not in (None, "", 0, "0"): + remaining_budget_seconds = max(0.6, float(budget_seconds or 0.0) - (time.perf_counter() - step_started_at)) + success, result = get_site_data(domain, enkey, proxies=proxies, budget_seconds=remaining_budget_seconds) + logger.info( + f"站长之家阶段: domain={domain} | stage=seo_request_done | success={success} " + f"| elapsed_ms={int((time.perf_counter() - seo_started_at) * 1000)}" + ) + if not success: + return False, str(result or '站长工具 SEO 数据获取失败'), () + + # 检查网站分类 + if success and result: + # 定义需要拉黑的分类 + blacklist_categories = ['视频电影', '体育运动', '常用查询', '游戏网站', '游戏', '视频', '体育'] + + # 检查分类是否在黑名单中 + if isinstance(result, dict): + # 检查可能的分类字段 + category_fields = ['Category', 'category', '分类', '网站分类'] + for field in category_fields: + if field in result: + category = result[field] + if isinstance(category, str): + for blacklist_category in blacklist_categories: + if blacklist_category in category: + logger.warning(f'检测到黑名单分类: {category} 在域名: {domain}') + return False, f'网站分类: {category} 属于黑名单分类', () + + # 检查Result是否为列表 + elif isinstance(result, list): + for item in result: + if isinstance(item, dict): + category_fields = ['Category', 'category', '分类', '网站分类'] + for field in category_fields: + if field in item: + category = item[field] + if isinstance(category, str): + for blacklist_category in blacklist_categories: + if blacklist_category in category: + logger.warning(f'检测到黑名单分类: {category} 在域名: {domain}') + return False, f'网站分类: {category} 属于黑名单分类', () + + return True, 'success', result + + logger.debug(f"站长工具检测失败: {domain}, 状态码: {response.status_code}") + return False, f"站长工具页面返回状态码: {response.status_code}", () + + except Exception as e: # 捕获所有异常 + error_str = str(e) + # 检查是否是需要重试的错误类型 + if any(error_type in error_str for error_type in ['Too many open files', 'EOF occurred in violation of protocol', 'Max retries exceeded', 'Read timed out']): + logger.error(f"站长工具检测失败,重试链路已耗尽: {domain}, 错误: {e}") + return True, 'failure', (False, str(e)) + logger.error(f"站长工具检测失败: {domain}, 错误: {e}") + return False, str(e), () # ========================= 【最终请求】 ========================= -def get_site_data(domain, enkey: str, proxies: dict = None): +def get_site_data(domain, enkey: str, proxies: dict = None, budget_seconds: Optional[float] = None): ''' 获取站点的详细SEO数据 @@ -143,15 +206,10 @@ def get_site_data(domain, enkey: str, proxies: dict = None): - (False, 错误信息): 查询失败 ''' try: - # 获取当前脚本所在目录(用于构建JS文件的完整路径) - current_dir = os.path.dirname(__file__) - - # 构建JS加密代码文件的完整路径 - js_path = os.path.join(current_dir, "chinaz.js") - - # 读取JS加密代码文件(包含生成签名的函数) - with open(js_path, "r", encoding="utf-8") as f: - js_code = f.read() + session = _get_session() + js_code = _load_chinaz_js() + js_started_at = time.perf_counter() + logger.info(f"站长之家阶段: domain={domain} | stage=js_context_start") # 创建JavaScript执行上下文(QuickJS引擎) ctx = quickjs.Context() @@ -166,6 +224,10 @@ def get_site_data(domain, enkey: str, proxies: dict = None): # 获取当前时间戳(毫秒),用于防止缓存和重放攻击 ts = str(int(time.time() * 1000)) + logger.info( + f"站长之家阶段: domain={domain} | stage=js_context_done " + f"| elapsed_ms={int((time.perf_counter() - js_started_at) * 1000)}" + ) # 构建请求参数字典(包含API所需的全部参数) params = { @@ -198,28 +260,43 @@ def get_site_data(domain, enkey: str, proxies: dict = None): url = 'https://othertool.chinaz.com/GetTopRanked.ashx' # 发送GET请求获取SEO数据 - resp = requests.get( + api_started_at = time.perf_counter() + logger.info( + f"站长之家阶段: domain={domain} | stage=seo_api_request_start | proxy={'yes' if proxies else 'no'}" + ) + resp = session.get( url, # 请求URL params=params, # 查询参数(自动拼接为URL参数) headers=headers, # 请求头 - timeout=10, # 超时时间10秒 + timeout=_resolve_chinaz_timeout(proxies, budget_seconds=budget_seconds), # 进一步缩短超时,避免长时间卡住线程 proxies=proxies # 代理配置 ) + logger.info( + f"站长之家阶段: domain={domain} | stage=seo_api_request_done | status={resp.status_code} " + f"| elapsed_ms={int((time.perf_counter() - api_started_at) * 1000)}" + ) # 判断请求是否成功 if resp.status_code == 200: # 使用正则表达式提取JSON数据(API返回格式:callback(json_data)) # 正则解释:匹配括号内的JSON内容 match = re.search(r'\((.*?)\)', resp.text) + if not match: + return False, "站长工具接口返回格式异常" # 提取JSON字符串 json_str = match.group(1) # 解析JSON数据为Python字典 data = json.loads(json_str) + logger.info( + f"站长之家阶段: domain={domain} | stage=seo_api_parsed " + f"| state_code={data.get('StateCode')}" + ) # 返回状态码和结果(StateCode为1表示成功) return data['StateCode'] == 1, data['Result'] + return False, f"站长工具接口返回状态码: {resp.status_code}" except Exception as e: # 捕获所有异常 logger.error(e) # 记录错误日志 @@ -239,4 +316,4 @@ def get_site_data(domain, enkey: str, proxies: dict = None): # # title_3, seo_data_3 = check_title(test_domain_3) # logger.info(f"域名标题: {title_3}") -# logger.info(f"SEO数据: {seo_data_3}") \ No newline at end of file +# logger.info(f"SEO数据: {seo_data_3}") diff --git a/domainCheck/detect/register.py b/domainCheck/detect/register.py index 871997b..68c7549 100644 --- a/domainCheck/detect/register.py +++ b/domainCheck/detect/register.py @@ -8,13 +8,78 @@ @explain : 域名注册状态检测工具 ''' +import json +import os +import threading from datetime import datetime, timezone, timedelta # 日期时间处理,用于时区转换 +from typing import Optional -import requests # HTTP请求库 from loguru import logger # 日志记录 +import requests +import urllib3 + +_DIRECT_HTTP = requests.Session() +_DIRECT_HTTP.trust_env = False +_DIRECT_HTTP.mount("http://", requests.adapters.HTTPAdapter(pool_connections=256, pool_maxsize=512, max_retries=0)) +_DIRECT_HTTP.mount("https://", requests.adapters.HTTPAdapter(pool_connections=256, pool_maxsize=512, max_retries=0)) +_PROXY_MANAGERS = {} +_PROXY_MANAGER_LOCK = threading.Lock() -def check_register(domain: str, postfix: str = 'com',proxies: dict = None): +def _get_http_manager(proxies: Optional[dict] = None): + proxy_url = "" + if proxies: + proxy_url = str(proxies.get("https") or proxies.get("http") or "").strip() + if not proxy_url: + return _DIRECT_HTTP + with _PROXY_MANAGER_LOCK: + manager = _PROXY_MANAGERS.get(proxy_url) + if manager is None: + manager = requests.Session() + manager.trust_env = False + manager.mount("http://", requests.adapters.HTTPAdapter(pool_connections=128, pool_maxsize=256, max_retries=0)) + manager.mount("https://", requests.adapters.HTTPAdapter(pool_connections=128, pool_maxsize=256, max_retries=0)) + _PROXY_MANAGERS[proxy_url] = manager + return manager + + +def _request_register(url: str, *, proxies: dict = None, timeout=None): + manager = _get_http_manager(proxies) + request_timeout = timeout + if isinstance(timeout, urllib3.Timeout): + request_timeout = (timeout.connect_timeout, timeout.read_timeout) + return manager.get(url, timeout=request_timeout, proxies=proxies, allow_redirects=True) + + +def _resolve_register_timeout(proxies: Optional[dict], budget_seconds: Optional[float] = None): + if proxies: + connect_timeout = float(os.getenv("DOMAINCHECK_REGISTER_PROXY_CONNECT_TIMEOUT", "1.0") or 1.0) + read_timeout = float(os.getenv("DOMAINCHECK_REGISTER_PROXY_READ_TIMEOUT", "1.5") or 1.5) + total_timeout = float(os.getenv("DOMAINCHECK_REGISTER_PROXY_TOTAL_TIMEOUT", "2.2") or 2.2) + else: + connect_timeout = float(os.getenv("DOMAINCHECK_REGISTER_DIRECT_CONNECT_TIMEOUT", "1.2") or 1.2) + read_timeout = float(os.getenv("DOMAINCHECK_REGISTER_DIRECT_READ_TIMEOUT", "1.8") or 1.8) + total_timeout = float(os.getenv("DOMAINCHECK_REGISTER_DIRECT_TOTAL_TIMEOUT", "2.6") or 2.6) + connect_timeout = max(0.2, connect_timeout) + read_timeout = max(0.3, read_timeout) + total_timeout = max(max(connect_timeout, read_timeout), float(total_timeout or 0.0)) + if budget_seconds not in (None, "", 0, "0"): + remaining_budget = max(0.6, float(budget_seconds or 0.0)) + total_timeout = min(total_timeout, remaining_budget) + # requests 只原生支持 (connect, read),这里把剩余预算重新切成更短的 + # connect/read,避免单次 RDAP 请求把整个步骤预算一次性吃掉。 + if total_timeout <= 1.0: + connect_timeout = min(connect_timeout, 0.35) + read_timeout = min(read_timeout, max(0.3, total_timeout - 0.2)) + else: + connect_timeout = min(connect_timeout, max(0.35, total_timeout * 0.35)) + read_timeout = min(read_timeout, max(0.45, total_timeout - connect_timeout)) + connect_timeout = max(0.2, min(connect_timeout, total_timeout)) + read_timeout = max(0.3, min(read_timeout, total_timeout)) + return urllib3.Timeout(connect=connect_timeout, read=read_timeout, total=total_timeout) + + +def check_register(domain: str, postfix: str = 'com', proxies: dict = None, budget_seconds: Optional[float] = None): ''' 检测注册状态 :param domain: 待检测域名(不包含后缀) @@ -25,34 +90,31 @@ def check_register(domain: str, postfix: str = 'com',proxies: dict = None): url = f'https://rdap.verisign.com/{postfix}/v1/domain/{domain}' # 构建RDAP查询URL # url = f'https://www.baidu.com' # 构建RDAP查询URL - # 添加重试机制 - max_retries = 3 - for retry in range(max_retries): - try: - response = requests.get(url, timeout=20, proxies=proxies) # 发送GET请求,设置超时5秒 - if response.status_code == 200: # HTTP 200表示域名已注册 - json_data = response.json() # 解析JSON响应 - for item in json_data['events']: # 遍历事件列表 - if item['eventAction'] == 'expiration': # 如果是过期时间事件 - utc_time = datetime.fromisoformat(item['eventDate'].replace('Z', '+00:00')) # 解析UTC时间 - beijing_tz = timezone(timedelta(hours=8)) # 创建北京时区(UTC+8) - beijing_time = utc_time.astimezone(beijing_tz) # 将UTC时间转换为北京时间 - return 3, beijing_time.strftime("%Y-%m-%d %H:%M:%S") # 返回已注册状态和过期时间 - - elif response.status_code == 404: # HTTP 404表示域名不存在,可注册 - # 可注册状态 - return 2, '' # 返回可注册状态,过期时间为空 - logger.error(response.status_code) - except requests.exceptions.RequestException as e: - # logger.warning(f"注册状态检测第{retry+1}次失败: {domain}, 错误: {e}") - if retry < max_retries - 1: - import time - time.sleep(2) # 等待2秒后重试 - proxies=None - else: - # 达到最大重试次数,返回检测失败 - logger.error(f"注册状态检测失败: {domain}, 已达到最大重试次数") - return -1, '' # -1表示检测失败,过期时间为空 + mode = "proxy" if proxies else "direct" + timeout = _resolve_register_timeout(proxies, budget_seconds=budget_seconds) + response = None + try: + response = _request_register(url, proxies=proxies, timeout=timeout) + except Exception as e: + logger.warning(f"注册状态检测请求异常: {domain}, 模式: {mode}, 错误: {e}") + raise + + if int(response.status_code) == 200: # HTTP 200表示域名已注册 + json_data = response.json() if response.content else {} # 解析JSON响应 + for item in json_data['events']: # 遍历事件列表 + if item['eventAction'] == 'expiration': # 如果是过期时间事件 + utc_time = datetime.fromisoformat(item['eventDate'].replace('Z', '+00:00')) # 解析UTC时间 + beijing_tz = timezone(timedelta(hours=8)) # 创建北京时区(UTC+8) + beijing_time = utc_time.astimezone(beijing_tz) # 将UTC时间转换为北京时间 + return 3, beijing_time.strftime("%Y-%m-%d %H:%M:%S") # 返回已注册状态和过期时间 + return 3, '' + + if int(response.status_code) == 404: # HTTP 404表示域名不存在,可注册 + return 2, '' # 返回可注册状态,过期时间为空 + + error_message = f"rdap unexpected status {response.status_code}" + logger.warning(f"注册状态检测返回异常状态码: {domain}, 模式: {mode}, 状态码: {response.status_code}") + raise RuntimeError(error_message) diff --git a/domainCheck/detect_worker.py b/domainCheck/detect_worker.py index df4a5b6..c8eb3ed 100644 --- a/domainCheck/detect_worker.py +++ b/domainCheck/detect_worker.py @@ -12,12 +12,18 @@ import os import sys import json import time +import socket import threading import collections +import resource import schedule +import urllib.error +import urllib.parse +import urllib.request +from concurrent.futures import ThreadPoolExecutor, as_completed from datetime import datetime from loguru import logger -from queue import Queue +from queue import Empty, Full, Queue from PySide6.QtWidgets import QApplication, QMainWindow, QTextEdit, QPushButton, QVBoxLayout, QHBoxLayout, QWidget, QLabel, QProgressBar, QGroupBox, QScrollArea, QFrame from PySide6.QtCore import Qt, QThread, Signal, QMetaObject, Q_ARG, QCoreApplication, QEvent, QTimer from PySide6.QtGui import QIcon @@ -42,6 +48,41 @@ CONTROL_CHANNEL = "domain_tool:worker_control" RUNTIME_STATE_KEY = "domain_tool:detect_runtime_state" PENDING_CONTROL_KEY = "domain_tool:worker_pending_command" RUNTIME_SETTINGS_KEY = "domain_tool:runtime_settings" +REMOTE_DEBUG_EVENT_TIMEOUT = 5 + + +def _raise_nofile_soft_limit(): + try: + soft_limit, hard_limit = resource.getrlimit(resource.RLIMIT_NOFILE) + target_limit = hard_limit if hard_limit not in (-1, resource.RLIM_INFINITY) else soft_limit + if target_limit and soft_limit < target_limit: + resource.setrlimit(resource.RLIMIT_NOFILE, (target_limit, hard_limit)) + logger.info(f"已提升 NOFILE 软限制: {soft_limit} -> {target_limit}") + except Exception as exc: + logger.warning(f"提升 NOFILE 软限制失败: {exc}") + + +def _resolve_worker_log_file() -> str: + candidate_dirs = [ + str(getattr(config, "LOG_DIR", "") or "").strip(), + os.path.join("/opt/domaincheck", "runtime", "domainCheck", "logs"), + os.path.join("/tmp", "domaincheck", "logs"), + ] + seen = set() + for raw_dir in candidate_dirs: + log_dir = os.path.abspath(raw_dir) if raw_dir else "" + if not log_dir or log_dir in seen: + continue + seen.add(log_dir) + try: + os.makedirs(log_dir, exist_ok=True) + log_file_path = os.path.join(log_dir, "detect_worker.log") + with open(log_file_path, "a", encoding="utf-8"): + pass + return log_file_path + except Exception: + continue + return os.path.join("/tmp", "detect_worker.log") class DetectThread(QThread): """ @@ -437,13 +478,8 @@ class DetectMainWindow(QMainWindow): # 初始化检测端 self.worker = DetectWorker(self) - # 启动Redis订阅线程(如果使用Redis) - if self.worker.use_redis: - self.worker.running = True - self.worker.redis_sub_thread = threading.Thread(target=self.worker.start_redis_subscription) - self.worker.redis_sub_thread.daemon = True - self.worker.redis_sub_thread.start() - logger.debug("Redis配置更新订阅已启动") + # 激活服务态运行时:启动控制链路,并在服务重启后立即尝试回挂活动检测任务。 + self.worker.activate_service_runtime() # 连接信号 self.worker.connect_signals() @@ -621,6 +657,7 @@ class DetectMainWindow(QMainWindow): new_detect_options = self.worker.load_detect_options() new_proxy_config = self.worker.load_proxy_config() new_thread_count = self.worker.load_thread_count() + new_sensitive_words = self.worker.db.get_all_sensitive_words() self.worker.load_cookies_from_remote() # 检查配置是否发生变化 @@ -634,6 +671,9 @@ class DetectMainWindow(QMainWindow): if new_thread_count != self.worker.thread_count: self.worker.thread_count = new_thread_count config_changed = True + if new_sensitive_words != self.worker.sensitive_words: + self.worker.sensitive_words = new_sensitive_words + config_changed = True # 如果配置发生变化,更新标签 if config_changed: @@ -789,6 +829,13 @@ class ConfigUpdateSignal(QObject): """ config_updated = Signal() +def runtime_state_key(node_code=None): + normalized_node_code = str(node_code or config.NODE_CODE or "").strip() + if not normalized_node_code: + return RUNTIME_STATE_KEY + return f"{RUNTIME_STATE_KEY}:{normalized_node_code}" + + class DetectWorker: """ 域名检测端工作类 @@ -807,19 +854,63 @@ class DetectWorker: self.detect_thread = None # 检测线程实例 self.detect_command_thread = None self.detect_lock = threading.Lock() - self.runtime_heartbeat_interval = 30 + self._domain_thread_counter_lock = threading.Lock() + self._task_local = threading.local() + self._step_result_cache_lock = threading.Lock() + self._step_result_cache = {} + self._active_domain_threads = 0 + self.detect_command_started_at = 0.0 + self.detect_command_last_activity_at = 0.0 + self._last_domain_started_at = 0.0 + self._last_domain_result_at = 0.0 + self._pending_restart_source = "" + self._pending_restart_payload = None + self._pending_restart_reason = "" + self._detect_session_seq = 0 + self._detect_session_owner = 0 + self._last_thread_count_refresh_at = 0.0 + self.runtime_heartbeat_interval = max( + 3.0, + float(os.getenv("DOMAINCHECK_RUNTIME_HEARTBEAT_INTERVAL", "5") or 5), + ) self._runtime_heartbeat_stop = threading.Event() self._last_runtime_phase = "idle" self._last_runtime_detail = "Worker 初始化中" self._last_runtime_extra = {} + self._last_runtime_state_push_at = 0.0 + self._last_runtime_state_push_phase = "" self.current_cycle_token = "" self.current_job_id = None self.current_job_code = "" + self.current_job_task_mode = "" self.runtime_settings = { "worker_log_sync_enabled": False, "worker_log_sync_mode": "key", + "worker_step_trace_enabled": True, + "worker_step_trace_sync_full": True, } self._last_synced_worker_log = "" + self._worker_log_sync_queue = Queue(maxsize=50000) + self._worker_log_sync_stop = threading.Event() + self._worker_log_sync_drop_count = 0 + self._last_worker_log_sync_drop_notice_at = 0.0 + self._running_mark_lock = threading.Lock() + self._pending_running_marks = collections.deque() + self._last_running_mark_flush_at = 0.0 + self._job_finalize_lock = threading.Lock() + self._pending_job_finalizations = collections.deque() + self._last_job_finalize_flush_at = 0.0 + self._domain_status_update_lock = threading.Lock() + self._pending_domain_status_updates = collections.deque() + self._last_domain_status_flush_at = 0.0 + self._domain_completion_lock = threading.Lock() + self._pending_domain_completions = collections.deque() + self._last_domain_completion_flush_at = 0.0 + self._review_status_update_lock = threading.Lock() + self._pending_review_status_updates = collections.deque() + self._last_review_status_flush_at = 0.0 + self._completed_future_lock = threading.Lock() + self._completed_futures = collections.deque() # 初始化数据库连接 self.db = Database() @@ -890,13 +981,98 @@ class DetectWorker: self.proxy_refresh_cooldown_seconds = 30 self.proxy_next_refresh_time = 0.0 self.proxy_max_reuse_count = max(8, self.thread_count * 4) + self.proxy_step_wait_timeout_seconds = max( + 0.0, + float(os.getenv("DOMAINCHECK_PROXY_WAIT_TIMEOUT", "1.5") or 1.5), + ) + self.proxy_direct_fallback_grace_seconds = max( + 0.0, + float(os.getenv("DOMAINCHECK_PROXY_DIRECT_FALLBACK_GRACE", "0.35") or 0.35), + ) + self.proxy_step_retry_max_attempts = max( + 1, + int(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS", "12") or 12), + ) + self.proxy_step_retry_max_seconds = max( + 5.0, + float(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_SECONDS", "45") or 45), + ) + self.proxy_step_retry_budget_overrides = { + "注册状态检测": { + "max_attempts": max( + 1, + int(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_REGISTER", "3") or 3), + ), + "max_seconds": max( + 5.0, + float(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_SECONDS_REGISTER", "8") or 8), + ), + }, + "百度site检测": { + "max_attempts": max( + 1, + int(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_BAIDU", "3") or 3), + ), + "max_seconds": max( + 5.0, + float(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_SECONDS_BAIDU", "6") or 6), + ), + }, + "360检测": { + "max_attempts": max( + 1, + int(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_360", "2") or 2), + ), + "max_seconds": max( + 5.0, + float(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_SECONDS_360", "5") or 5), + ), + }, + "站长之家检测": { + "max_attempts": max( + 1, + int(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_CHINAZ", "2") or 2), + ), + "max_seconds": max( + 5.0, + float(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_SECONDS_CHINAZ", "5") or 5), + ), + }, + "爱站网检测": { + "max_attempts": max( + 1, + int(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_AIZHAN", "2") or 2), + ), + "max_seconds": max( + 5.0, + float(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_SECONDS_AIZHAN", "5") or 5), + ), + }, + "时光机检测": { + "max_attempts": max( + 1, + int(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_WAYBACK", "3") or 3), + ), + "max_seconds": max( + 5.0, + float(os.getenv("DOMAINCHECK_PROXY_STEP_MAX_SECONDS_WAYBACK", "12") or 12), + ), + }, + } + self._last_no_proxy_notice_at = 0.0 self.proxy_failure_lock = threading.Lock() self.proxy_failure_counts = {} self.proxy_quarantine_until = {} + self.proxy_source_by_key = {} + self.proxy_source_failure_counts = {} + self.proxy_source_quarantine_until = {} + self.redis_sub_thread = None + self._last_autoresume_skip_log_at = 0.0 + self._last_autoresume_success_log_at = 0.0 # 加载敏感词(只加载一次,所有线程共享) try: - self.sensitive_words = self.db.get_all_sensitive_words() + self.sensitive_words = self._load_sensitive_words_runtime() logger.info(f"加载敏感词完成,共 {len(self.sensitive_words)} 个敏感词") except Exception as e: logger.error(f"加载敏感词失败: {e}") @@ -925,6 +1101,8 @@ class DetectWorker: # Redis订阅线程将在start方法中启动 self._update_runtime_state("idle", "Worker 已启动,等待检测指令") self._start_runtime_heartbeat_loop() + self._start_worker_log_sync_loop() + self.trigger_proxy_refresh(reason="worker_startup", reset_cooldown=True) def _start_runtime_heartbeat_loop(self): existing = getattr(self, "_runtime_heartbeat_thread", None) @@ -944,6 +1122,10 @@ class DetectWorker: self._consume_pending_control_command() except Exception as e: logger.debug(f"运行态心跳补偿消费待执行控制指令失败: {e}") + try: + self._resume_active_detect_job_if_needed(reason="heartbeat_autoresume") + except Exception as e: + logger.debug(f"运行态心跳自动回挂检测任务失败: {e}") try: self._update_runtime_state( self._last_runtime_phase, @@ -953,6 +1135,320 @@ class DetectWorker: except Exception as e: logger.debug(f"运行态心跳续期失败: {e}") + def _start_worker_log_sync_loop(self): + existing = getattr(self, "_worker_log_sync_thread", None) + if existing and existing.is_alive(): + return + self._worker_log_sync_stop.clear() + self._worker_log_sync_thread = threading.Thread( + target=self._worker_log_sync_loop, + name="WorkerLogSyncThread", + daemon=True, + ) + self._worker_log_sync_thread.start() + + def _worker_log_sync_loop(self): + while True: + if self._worker_log_sync_stop.is_set() and self._worker_log_sync_queue.empty(): + break + try: + item = self._worker_log_sync_queue.get(timeout=0.5) + except Empty: + continue + if item is None: + self._worker_log_sync_queue.task_done() + if self._worker_log_sync_stop.is_set(): + break + continue + + if item.get("write_db", True): + try: + self.db.append_detect_run_event( + item["job_id"], + item["job_item_id"], + config.NODE_CODE, + event_type='worker_log', + message=item["message"], + level=item["level"], + payload=item["payload"], + ) + except Exception as e: + logger.debug(f"Worker 日志事件写库失败: {e}") + + try: + self._push_remote_debug_event( + message=item["message"], + level=item["level"], + payload=item["payload"], + event_type="worker_log", + ) + except Exception as e: + logger.debug(f"Worker 日志事件远端回传失败: {e}") + finally: + self._worker_log_sync_queue.task_done() + + def _enqueue_running_mark(self, job_item_id, claim_token): + normalized_claim_token = str(claim_token or "").strip() + try: + normalized_job_item_id = int(job_item_id or 0) + except Exception: + normalized_job_item_id = 0 + if normalized_job_item_id <= 0 or not normalized_claim_token: + return + with self._running_mark_lock: + self._pending_running_marks.append((normalized_job_item_id, normalized_claim_token)) + + def _flush_pending_running_marks(self, *, force=False, batch_limit=512): + now_ts = time.time() + if not force and now_ts - float(getattr(self, "_last_running_mark_flush_at", 0.0) or 0.0) < 0.05: + return 0 + batch = [] + with self._running_mark_lock: + if not self._pending_running_marks: + if force: + self._last_running_mark_flush_at = now_ts + return 0 + while self._pending_running_marks and len(batch) < max(1, int(batch_limit or 1)): + batch.append(self._pending_running_marks.popleft()) + updated_count = 0 + try: + updated_count = int(self.db.mark_detect_job_items_running_batch(batch) or 0) + except Exception as e: + logger.debug(f"批量刷新运行中任务项失败: {e}") + updated_count = -1 + if updated_count < 0: + with self._running_mark_lock: + for item in reversed(batch): + self._pending_running_marks.appendleft(item) + logger.warning( + f"批量刷新运行中任务项失败,已回退待重试队列: batch={len(batch)} | force={1 if force else 0}" + ) + updated_count = 0 + self._last_running_mark_flush_at = now_ts + return updated_count + + def _enqueue_job_finalization( + self, + *, + job_item_id, + claim_token, + final_status, + message="", + result_payload=None, + result_version="v1", + job_id=None, + node_code="", + event_type="", + event_level="info", + event_message="", + event_payload=None, + ): + normalized_claim_token = str(claim_token or "").strip() + try: + normalized_job_item_id = int(job_item_id or 0) + except Exception: + normalized_job_item_id = 0 + if normalized_job_item_id <= 0 or not normalized_claim_token: + return + try: + normalized_job_id = int(job_id or 0) + except Exception: + normalized_job_id = 0 + with self._job_finalize_lock: + self._pending_job_finalizations.append( + { + "job_item_id": normalized_job_item_id, + "claim_token": normalized_claim_token, + "final_status": str(final_status or "").strip() or "failed", + "message": str(message or "").strip(), + "result_payload": result_payload, + "result_version": str(result_version or "v1").strip() or "v1", + "job_id": normalized_job_id, + "node_code": str(node_code or "").strip(), + "event_type": str(event_type or "").strip(), + "event_level": str(event_level or "info").strip() or "info", + "event_message": str(event_message or "").strip(), + "event_payload": dict(event_payload or {}), + } + ) + + def _flush_pending_job_finalizations(self, *, force=False, batch_limit=512): + now_ts = time.time() + if not force and now_ts - float(getattr(self, "_last_job_finalize_flush_at", 0.0) or 0.0) < 0.05: + return 0 + batch = [] + with self._job_finalize_lock: + if not self._pending_job_finalizations: + if force: + self._last_job_finalize_flush_at = now_ts + return 0 + while self._pending_job_finalizations and len(batch) < max(1, int(batch_limit or 1)): + batch.append(self._pending_job_finalizations.popleft()) + updated_count = 0 + flush_started_at = time.perf_counter() + try: + updated_count = int(self.db.finalize_detect_job_items_batch(batch) or 0) + except Exception as e: + logger.debug(f"批量回写任务项完成态失败: {e}") + updated_count = -1 + if updated_count < 0: + with self._job_finalize_lock: + for item in reversed(batch): + self._pending_job_finalizations.appendleft(item) + logger.warning( + f"批量回写任务项完成态失败,已回退待重试队列: batch={len(batch)} | force={1 if force else 0}" + ) + updated_count = 0 + flush_elapsed_ms = int((time.perf_counter() - flush_started_at) * 1000) + if batch and (flush_elapsed_ms >= 20 or len(batch) >= 64 or force): + logger.info( + f"任务项完成态批量回写: batch={len(batch)} | updated={updated_count} " + f"| elapsed_ms={flush_elapsed_ms} | force={1 if force else 0}" + ) + self._last_job_finalize_flush_at = now_ts + return updated_count + + def _enqueue_domain_status_update(self, domain_id, status): + try: + normalized_domain_id = int(domain_id or 0) + normalized_status = int(status) + except Exception: + return + if normalized_domain_id <= 0: + return + with self._domain_status_update_lock: + self._pending_domain_status_updates.append((normalized_domain_id, normalized_status)) + + def _enqueue_domain_completion(self, domain_id, *, register_status, use_status, expire_date): + try: + normalized_domain_id = int(domain_id or 0) + normalized_register_status = int(register_status or 0) + normalized_use_status = int(use_status or 0) + except Exception: + return + if normalized_domain_id <= 0: + return + with self._domain_completion_lock: + self._pending_domain_completions.append( + ( + normalized_domain_id, + normalized_register_status, + normalized_use_status, + bool(expire_date), + ) + ) + + def _flush_pending_domain_completions(self, *, force=False, batch_limit=512): + now_ts = time.time() + if not force and now_ts - float(getattr(self, "_last_domain_completion_flush_at", 0.0) or 0.0) < 0.05: + return 0 + batch = [] + with self._domain_completion_lock: + if not self._pending_domain_completions: + if force: + self._last_domain_completion_flush_at = now_ts + return 0 + while self._pending_domain_completions and len(batch) < max(1, int(batch_limit or 1)): + batch.append(self._pending_domain_completions.popleft()) + updated_count = 0 + flush_started_at = time.perf_counter() + try: + updated_count = int(self.db.complete_domain_detection_batch(batch) or 0) + except Exception as e: + logger.debug(f"批量刷新域名完成态失败: {e}") + updated_count = -1 + if updated_count < 0: + with self._domain_completion_lock: + for item in reversed(batch): + self._pending_domain_completions.appendleft(item) + logger.warning( + f"批量刷新域名完成态失败,已回退待重试队列: batch={len(batch)} | force={1 if force else 0}" + ) + updated_count = 0 + flush_elapsed_ms = int((time.perf_counter() - flush_started_at) * 1000) + if batch and (flush_elapsed_ms >= 20 or len(batch) >= 64 or force): + logger.info( + f"域名完成态批量回写: batch={len(batch)} | updated={updated_count} " + f"| elapsed_ms={flush_elapsed_ms} | force={1 if force else 0}" + ) + self._last_domain_completion_flush_at = now_ts + return updated_count + + def _flush_pending_domain_status_updates(self, *, force=False, batch_limit=512): + now_ts = time.time() + if not force and now_ts - float(getattr(self, "_last_domain_status_flush_at", 0.0) or 0.0) < 0.05: + return 0 + batch = [] + with self._domain_status_update_lock: + if not self._pending_domain_status_updates: + if force: + self._last_domain_status_flush_at = now_ts + return 0 + while self._pending_domain_status_updates and len(batch) < max(1, int(batch_limit or 1)): + batch.append(self._pending_domain_status_updates.popleft()) + updated_count = 0 + try: + updated_count = int(self.db.update_domain_detect_status_batch(batch) or 0) + except Exception as e: + logger.debug(f"批量刷新域名检测状态失败: {e}") + updated_count = -1 + if updated_count < 0: + with self._domain_status_update_lock: + for item in reversed(batch): + self._pending_domain_status_updates.appendleft(item) + logger.warning( + f"批量刷新域名检测状态失败,已回退待重试队列: batch={len(batch)} | force={1 if force else 0}" + ) + updated_count = 0 + self._last_domain_status_flush_at = now_ts + return updated_count + + def _enqueue_review_status_update(self, domain_id, review_status): + try: + normalized_domain_id = int(domain_id or 0) + normalized_review_status = int(review_status) + except Exception: + return + if normalized_domain_id <= 0: + return + with self._review_status_update_lock: + self._pending_review_status_updates.append((normalized_domain_id, normalized_review_status)) + + def _flush_pending_review_status_updates(self, *, force=False, batch_limit=512): + now_ts = time.time() + if not force and now_ts - float(getattr(self, "_last_review_status_flush_at", 0.0) or 0.0) < 0.05: + return 0 + batch = [] + with self._review_status_update_lock: + if not self._pending_review_status_updates: + if force: + self._last_review_status_flush_at = now_ts + return 0 + while self._pending_review_status_updates and len(batch) < max(1, int(batch_limit or 1)): + batch.append(self._pending_review_status_updates.popleft()) + updated_count = 0 + try: + updated_count = int(self.db.update_domain_review_status_batch(batch) or 0) + except Exception as e: + logger.debug(f"批量刷新域名复核状态失败: {e}") + updated_count = -1 + if updated_count < 0: + with self._review_status_update_lock: + for item in reversed(batch): + self._pending_review_status_updates.appendleft(item) + logger.warning( + f"批量刷新域名复核状态失败,已回退待重试队列: batch={len(batch)} | force={1 if force else 0}" + ) + updated_count = 0 + self._last_review_status_flush_at = now_ts + return updated_count + + def _enqueue_completed_future(self, future): + if future is None: + return + with self._completed_future_lock: + self._completed_futures.append(future) + def _consume_pending_control_command(self): if not self.use_redis or self.redis_client is None: return @@ -971,13 +1467,136 @@ class DetectWorker: logger.info(f"发现待执行 Worker 控制指令: {payload}") self._handle_control_message(payload) + def _resume_active_detect_job_if_needed(self, reason: str = "worker_bootstrap"): + now_ts = time.time() + stale_detect = False + + def log_skip(detail: str): + if now_ts - float(getattr(self, "_last_autoresume_skip_log_at", 0.0) or 0.0) >= 15.0: + logger.info(f"自动回挂跳过: reason={reason}, detail={detail}") + self._last_autoresume_skip_log_at = now_ts + + if not self.running: + log_skip("worker_not_running") + return False + if self._consume_pending_restart_request(trigger_reason=reason): + return True + if self.stop_requested: + log_skip("stop_requested") + return False + if self.detecting: + stale_detect, stale_reason = self._is_stale_detect_session() + if not stale_detect: + log_skip("already_detecting") + return False + logger.warning(f"检测运行态疑似空转,允许自动回挂接管: {stale_reason}") + thread = self.detect_command_thread + if thread and thread.is_alive() and not stale_detect: + log_skip("detect_command_thread_alive") + return False + try: + active_job = self.db.get_active_detect_job() or {} + except Exception as e: + logger.debug(f"读取当前检测任务失败,跳过自动回挂: {e}") + return False + job_id = active_job.get("id") + job_code = str(active_job.get("job_code") or "").strip() + task_mode = str(active_job.get("task_mode") or "").strip() + if not job_id or not job_code: + log_skip("active_job_missing_id_or_code") + return False + items_pending = int(active_job.get("items_pending", 0) or 0) + items_claimed = int(active_job.get("items_claimed", 0) or 0) + items_running = int(active_job.get("items_running", 0) or 0) + if items_pending <= 0 and items_claimed <= 0 and items_running <= 0: + log_skip("active_job_empty") + return False + logger.warning( + f"检测 Worker 空闲但发现活动任务,准备自动回挂: " + f"job_code={job_code}, pending={items_pending}, claimed={items_claimed}, running={items_running}, reason={reason}" + ) + started = self.start_detection_async( + source="auto-resume", + control_payload={ + "source": reason, + "job_id": int(job_id), + "job_code": job_code, + "task_mode": task_mode, + "target_job_id": int(job_id), + "target_job_code": job_code, + "target_task_mode": task_mode, + }, + ) + if started and now_ts - float(getattr(self, "_last_autoresume_success_log_at", 0.0) or 0.0) >= 5.0: + logger.warning( + f"自动回挂已触发: reason={reason}, job_code={job_code}, " + f"pending={items_pending}, claimed={items_claimed}, running={items_running}" + ) + self._last_autoresume_success_log_at = now_ts + return started + + def activate_service_runtime(self): + if not self.running: + self.running = True + logger.info( + f"激活 Worker 服务态运行时: node={config.NODE_CODE}, " + f"thread_count={self.thread_count}, proxy_enabled={1 if self.proxy_config.get('proxy_enable', False) else 0}" + ) + if self.use_redis: + thread = getattr(self, "redis_sub_thread", None) + if not thread or not thread.is_alive(): + self.redis_sub_thread = threading.Thread( + target=self.start_redis_subscription, + name="RedisSubscriptionThread", + daemon=True, + ) + self.redis_sub_thread.start() + logger.info("Redis 配置/控制订阅线程已启动") + try: + self._consume_pending_control_command() + except Exception as e: + logger.warning(f"服务态启动时消费待执行控制指令失败: {e}") + try: + self._resume_active_detect_job_if_needed(reason="service_runtime_bootstrap") + except Exception as e: + logger.warning(f"服务态启动时自动回挂活动任务失败: {e}") + def _update_runtime_state(self, phase: str, detail: str, **extra): """ 更新 Redis 中的检测运行态,供 Web 后台读取。 """ + volatile_keys = { + "active_threads", + "current_load", + "cycle_token", + "job_id", + "job_code", + } + previous_extra = dict(getattr(self, "_last_runtime_extra", {}) or {}) + merged_extra = {key: value for key, value in previous_extra.items() if key not in volatile_keys} + merged_extra.update({key: value for key, value in dict(extra or {}).items() if key not in volatile_keys}) + if "max_threads" not in merged_extra: + merged_extra["max_threads"] = int(getattr(self, "thread_count", 0) or 0) + with self._domain_thread_counter_lock: + live_active_threads = int(self._active_domain_threads or 0) + explicit_active_threads = (extra or {}).get("active_threads") + if not bool(self.detecting): + effective_active_threads = 0 + elif explicit_active_threads in (None, ""): + effective_active_threads = live_active_threads + else: + effective_active_threads = max(int(explicit_active_threads or 0), live_active_threads) + merged_extra["active_threads"] = effective_active_threads + + explicit_current_load = (extra or {}).get("current_load") + default_current_load = effective_active_threads if effective_active_threads > 0 else (1 if self.detecting else 0) + if explicit_current_load in (None, ""): + merged_extra["current_load"] = default_current_load + else: + merged_extra["current_load"] = max(int(explicit_current_load or 0), default_current_load) self._last_runtime_phase = phase self._last_runtime_detail = detail - self._last_runtime_extra = dict(extra or {}) + self._last_runtime_extra = merged_extra if not self.use_redis or self.redis_client is None: return with self.proxy_pool_lock: @@ -991,6 +1610,7 @@ class DetectWorker: ): payload_warning = self.proxy_last_refresh_status payload = { + "node_code": config.NODE_CODE, "phase": phase, "detail": detail, "service_running": bool(self.running), @@ -1007,19 +1627,35 @@ class DetectWorker: "recent_warning": payload_warning, "updated_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), } - payload.update(extra) + payload.update(merged_extra) if self.current_cycle_token and not payload.get("cycle_token"): payload["cycle_token"] = self.current_cycle_token if self.current_job_id and not payload.get("job_id"): payload["job_id"] = self.current_job_id if self.current_job_code and not payload.get("job_code"): payload["job_code"] = self.current_job_code + now_ts = time.time() + runtime_state_busy_phases = {"preparing", "refreshing_proxy", "fetching", "running", "completing"} + runtime_state_min_interval = max( + 0.05, + float(os.getenv("DOMAINCHECK_RUNTIME_STATE_MIN_INTERVAL", "0.25") or 0.25), + ) + last_push_phase = str(getattr(self, "_last_runtime_state_push_phase", "") or "") + last_push_at = float(getattr(self, "_last_runtime_state_push_at", 0.0) or 0.0) + if ( + phase in runtime_state_busy_phases + and phase == last_push_phase + and now_ts - last_push_at < runtime_state_min_interval + ): + return try: - self.redis_client.set(RUNTIME_STATE_KEY, json.dumps(payload, ensure_ascii=False)) + self.redis_client.set(runtime_state_key(), json.dumps(payload, ensure_ascii=False)) except Exception as e: logger.debug(f"写入检测运行态失败: {e}") try: - current_load = int(extra.get("active_threads", payload.get("active_threads", 0)) or 0) + actual_active_threads = int(payload.get("active_threads", 0) or 0) + max_threads = int(payload.get("max_threads", getattr(self, "thread_count", 0)) or 0) + current_load = actual_active_threads if bool(payload.get("detecting")) and current_load <= 0: current_load = 1 self.db.register_cluster_node( @@ -1033,7 +1669,8 @@ class DetectWorker: "detail": detail, "detecting": bool(payload.get("detecting")), "available_proxy_count": int(payload.get("available_proxy_count", 0) or 0), - "active_threads": current_load, + "active_threads": actual_active_threads, + "max_threads": max_threads, "cycle_token": str(payload.get("cycle_token") or ""), "job_id": payload.get("job_id"), "job_code": str(payload.get("job_code") or ""), @@ -1047,26 +1684,188 @@ class DetectWorker: ) except Exception as e: logger.debug(f"更新检测节点心跳失败: {e}") + self._last_runtime_state_push_at = now_ts + self._last_runtime_state_push_phase = phase def _mark_detection_phase(self, phase: str, detail: str, **extra): + if phase not in {"idle", "completed", "stopped"}: + self._note_detection_activity() self._update_runtime_state(phase, detail, **extra) + def _change_active_domain_threads(self, delta: int): + with self._domain_thread_counter_lock: + self._active_domain_threads = max(0, int(self._active_domain_threads or 0) + int(delta)) + if self._active_domain_threads > 0: + self._note_detection_activity() + return self._active_domain_threads + + def _get_active_domain_threads(self): + with self._domain_thread_counter_lock: + return int(self._active_domain_threads or 0) + + def _note_detection_activity(self): + self.detect_command_last_activity_at = time.time() + + def _note_domain_started(self): + now_ts = time.time() + self._last_domain_started_at = now_ts + self.detect_command_last_activity_at = now_ts + + def _note_domain_result(self): + now_ts = time.time() + self._last_domain_result_at = now_ts + self.detect_command_last_activity_at = now_ts + + def _schedule_pending_restart(self, source: str, control_payload=None, reason: str = ""): + self._pending_restart_source = str(source or "deferred-restart").strip() or "deferred-restart" + self._pending_restart_payload = dict(control_payload or {}) + self._pending_restart_reason = str(reason or "").strip() + + def _should_force_takeover_stale_session(self, stale_reason: str = "") -> bool: + thread = self.detect_command_thread + if not thread or not thread.is_alive(): + return False + with self._domain_thread_counter_lock: + live_active_threads = int(self._active_domain_threads or 0) + if live_active_threads > 0: + return False + + last_activity_at = max( + float(self.detect_command_last_activity_at or 0.0), + float(self.detect_command_started_at or 0.0), + float(getattr(self, "_last_domain_started_at", 0.0) or 0.0), + float(getattr(self, "_last_domain_result_at", 0.0) or 0.0), + ) + idle_seconds = max(0.0, time.time() - last_activity_at) if last_activity_at > 0 else 0.0 + thread_age_seconds = max(0.0, time.time() - float(self.detect_command_started_at or 0.0)) + if idle_seconds >= 10 or thread_age_seconds >= 20: + return True + return bool(stale_reason) + + def _consume_pending_restart_request(self, trigger_reason: str = ""): + with self.detect_lock: + if self.detect_command_thread and self.detect_command_thread.is_alive(): + return False + source = str(self._pending_restart_source or "").strip() + payload = dict(self._pending_restart_payload or {}) + pending_reason = str(self._pending_restart_reason or "").strip() + if not source and not payload: + return False + self._pending_restart_source = "" + self._pending_restart_payload = None + self._pending_restart_reason = "" + logger.warning( + "检测任务执行延期重启: " + f"trigger={trigger_reason or 'unknown'}, source={source or 'deferred-restart'}, " + f"detail={pending_reason or 'none'}" + ) + return self.start_detection_async(source=source or "deferred-restart", control_payload=payload) + + def _incoming_job_code(self, control_payload=None) -> str: + payload = control_payload or {} + job_code = str(payload.get("job_code") or "").strip() + if not job_code: + job_code = str(payload.get("target_job_code") or "").strip() + return job_code + + def _is_stale_detect_session(self, control_payload=None): + thread = self.detect_command_thread + if not thread or not thread.is_alive(): + return False, "" + + live_active_threads = 0 + with self._domain_thread_counter_lock: + live_active_threads = int(self._active_domain_threads or 0) + + last_activity_at = max( + float(self.detect_command_last_activity_at or 0.0), + float(self.detect_command_started_at or 0.0), + float(getattr(self, "_last_domain_started_at", 0.0) or 0.0), + float(getattr(self, "_last_domain_result_at", 0.0) or 0.0), + ) + if last_activity_at <= 0: + return False, "" + + idle_seconds = max(0.0, time.time() - last_activity_at) + thread_age_seconds = max(0.0, time.time() - float(self.detect_command_started_at or 0.0)) + last_result_at = float(getattr(self, "_last_domain_result_at", 0.0) or 0.0) + result_idle_seconds = max(0.0, time.time() - last_result_at) if last_result_at > 0 else idle_seconds + incoming_job_code = self._incoming_job_code(control_payload) + current_job_code = str(self.current_job_code or "").strip() + job_switched = bool(incoming_job_code and current_job_code and incoming_job_code != current_job_code) + incoming_job_changed = bool(incoming_job_code and incoming_job_code != current_job_code) + low_activity_recoverable = live_active_threads <= 1 + multi_thread_force_recover = incoming_job_changed and thread_age_seconds >= 30 and result_idle_seconds >= 30 + if not low_activity_recoverable and not multi_thread_force_recover: + return False, "" + + if job_switched and idle_seconds >= 15: + return True, ( + f"检测指令线程仍停留在旧任务 {current_job_code}," + f"但新任务已切换为 {incoming_job_code},且已空转 {int(idle_seconds)} 秒" + ) + if incoming_job_changed and idle_seconds >= 20: + return True, ( + f"检测指令线程未挂载新任务 {incoming_job_code}," + f"当前任务上下文为 {current_job_code or 'empty'},且已空转 {int(idle_seconds)} 秒" + ) + if multi_thread_force_recover: + return True, ( + f"检测指令线程已运行 {int(thread_age_seconds)} 秒,最近 {int(result_idle_seconds)} 秒无域名产出," + f"且新任务 {incoming_job_code} 已到达,当前活跃线程 {live_active_threads}" + ) + if idle_seconds >= 45: + return True, f"检测指令线程已空转 {int(idle_seconds)} 秒,当前活跃线程 {live_active_threads}" + if thread_age_seconds >= 30 and idle_seconds >= 20: + return True, ( + f"检测指令线程已启动 {int(thread_age_seconds)} 秒,但最近 {int(idle_seconds)} 秒没有任何有效检测活动,当前活跃线程 {live_active_threads}" + ) + return False, "" + def _set_active_cycle_context(self, control_payload=None): payload = control_payload or {} self._last_synced_worker_log = "" self.current_cycle_token = str(payload.get("cycle_token") or "").strip() job_id = payload.get("job_id") + if job_id in (None, "", 0, "0"): + job_id = payload.get("target_job_id") try: self.current_job_id = int(job_id) if job_id not in (None, "", 0, "0") else None except Exception: self.current_job_id = None - self.current_job_code = str(payload.get("job_code") or "").strip() + job_code = str(payload.get("job_code") or "").strip() + if not job_code: + job_code = str(payload.get("target_job_code") or "").strip() + self.current_job_code = job_code + task_mode = str(payload.get("task_mode") or payload.get("target_task_mode") or "").strip() + if not task_mode and str(job_code).startswith("step-"): + task_mode = "single_step" + self.current_job_task_mode = task_mode def _clear_active_cycle_context(self): self._last_synced_worker_log = "" self.current_cycle_token = "" self.current_job_id = None self.current_job_code = "" + self.current_job_task_mode = "" + self.detect_command_started_at = 0.0 + self.detect_command_last_activity_at = 0.0 + self._last_domain_started_at = 0.0 + self._last_domain_result_at = 0.0 + + def _is_single_step_session_active(self): + task_mode = str(getattr(self, "current_job_task_mode", "") or "").strip() + if task_mode == "single_step": + return True + return str(getattr(self, "current_job_code", "") or "").strip().startswith("step-") + + def _should_scope_claims_to_current_job(self): + """ + single_step 会话必须严格绑定当前 job; + domain_pipeline 会话则应持续从全局队列补位,避免旧 job 占住 worker、 + 新 job 长时间堆在 pending。 + """ + return self._is_single_step_session_active() def start_detection_async(self, source: str = "remote", control_payload=None): """ @@ -1074,14 +1873,61 @@ class DetectWorker: """ with self.detect_lock: if self.detect_command_thread and self.detect_command_thread.is_alive(): - logger.warning("收到启动检测指令,但检测任务已在运行,忽略重复启动") - self._update_runtime_state("running", "检测任务已在运行,忽略重复启动") - return False + stale_detect, stale_reason = self._is_stale_detect_session(control_payload) + if not stale_detect: + incoming_job_code = self._incoming_job_code(control_payload) + current_job_code = str(self.current_job_code or "").strip() + incoming_task_mode = str( + (control_payload or {}).get("task_mode") + or (control_payload or {}).get("target_task_mode") + or "" + ).strip() + if ( + incoming_task_mode != "single_step" + and not self._is_single_step_session_active() + and incoming_job_code + and incoming_job_code != current_job_code + ): + message = ( + "检测任务已在运行,已接收新的 pipeline 唤醒指令," + f"继续由当前 worker pool 补位处理: current={current_job_code or 'none'} -> incoming={incoming_job_code}" + ) + logger.info(message) + self._update_runtime_state("running", message) + return True + logger.warning("收到启动检测指令,但检测任务已在运行,忽略重复启动") + self._update_runtime_state("running", "检测任务已在运行,忽略重复启动") + return False + logger.warning(f"检测任务线程疑似卡死,准备回收后重新启动: {stale_reason}") + self.stop_requested = True + recovery_deadline = time.time() + 8 + while self.detect_command_thread and self.detect_command_thread.is_alive() and time.time() < recovery_deadline: + time.sleep(0.2) + if self.detect_command_thread and self.detect_command_thread.is_alive(): + if self._should_force_takeover_stale_session(stale_reason): + logger.warning(f"检测任务线程超时未退出,执行强制接管: {stale_reason}") + self.detect_command_thread = None + self.detecting = False + self._clear_active_cycle_context() + self._update_runtime_state("restarting", f"检测线程疑似卡死,执行强制接管: {stale_reason}") + else: + self._schedule_pending_restart(source, control_payload, stale_reason) + self._update_runtime_state("restarting", f"检测线程疑似卡死,已登记延期重启: {stale_reason}") + return True self.stop_requested = False + self.detect_command_started_at = time.time() + self.detect_command_last_activity_at = self.detect_command_started_at + self._detect_session_seq = int(self._detect_session_seq or 0) + 1 + session_id = int(self._detect_session_seq or 0) + self._detect_session_owner = session_id self.detect_command_thread = threading.Thread( target=self._run_detection_session, - kwargs={"source": source, "control_payload": dict(control_payload or {})}, + kwargs={ + "source": source, + "control_payload": dict(control_payload or {}), + "session_id": session_id, + }, name="RemoteDetectCommandThread", daemon=True, ) @@ -1098,9 +1944,15 @@ class DetectWorker: ) return True - def _run_detection_session(self, source: str = "remote", control_payload=None): + def _run_detection_session(self, source: str = "remote", control_payload=None, session_id: int = 0): self._set_active_cycle_context(control_payload) with self.detect_lock: + current_owner = int(getattr(self, "_detect_session_owner", 0) or 0) + if session_id and current_owner and session_id != current_owner: + logger.warning( + f"检测任务会话启动时发现 owner 已切换,忽略旧会话: session_id={session_id}, current_owner={current_owner}" + ) + return self.detecting = True try: logger.info(f"开始执行远程检测任务,来源: {source}") @@ -1127,10 +1979,32 @@ class DetectWorker: job_code=self.current_job_code, ) finally: + current_thread = threading.current_thread() with self.detect_lock: - self.detecting = False - self.detect_command_thread = None - if self.running and not self.stop_requested: + current_owner = int(getattr(self, "_detect_session_owner", 0) or 0) + is_owner = not session_id or current_owner in {0, session_id} + if is_owner: + self.detecting = False + if session_id: + self._detect_session_owner = 0 + if self.detect_command_thread is current_thread or not ( + self.detect_command_thread and self.detect_command_thread.is_alive() + ): + self.detect_command_thread = None + if not is_owner: + logger.warning( + f"检测任务旧会话结束,忽略清理: session_id={session_id}, current_owner={current_owner}" + ) + return + restarted_from_pending = False + if self.running: + try: + restarted_from_pending = bool( + self._consume_pending_restart_request(trigger_reason="session_finalize") + ) + except Exception as restart_error: + logger.warning(f"检测任务延期重启触发失败: {restart_error}") + if self.running and not self.stop_requested and not restarted_from_pending: self._update_runtime_state( "idle", "检测任务结束,Worker 保持待命", @@ -1329,10 +2203,26 @@ class DetectWorker: """ 加载检测线程数 """ - max_recommended_threads = 20 try: node_code = str(getattr(config, "NODE_CODE", "") or "").strip() node_thread_counts = {} + redis_global_thread_count = None + + def _warn_missing_node_override(source_name, overrides, fallback_value): + if not node_code or not isinstance(overrides, dict) or not overrides: + return + if node_code in overrides: + return + available_codes = ", ".join(sorted(str(code) for code in overrides.keys() if str(code).strip())) + logger.warning( + "节点线程覆盖未命中,回退到%s通用线程数: node_code=%s, fallback=%s, available_node_codes=%s" + % ( + source_name, + node_code, + fallback_value, + available_codes or "-", + ) + ) # 从Redis获取配置 if self.use_redis: @@ -1346,13 +2236,15 @@ class DetectWorker: if node_code and isinstance(node_thread_counts, dict): node_thread_count_str = node_thread_counts.get(node_code) if node_thread_count_str is not None: - thread_count = min(max_recommended_threads, max(1, int(node_thread_count_str))) + thread_count = max(1, int(node_thread_count_str)) logger.info(f"从Redis加载节点专属检测线程数成功: {node_code} -> {thread_count}") return thread_count thread_count_str = self.redis_client.get('domain_tool:thread_count') if thread_count_str: - thread_count = min(max_recommended_threads, max(1, int(thread_count_str))) + thread_count = max(1, int(thread_count_str)) + redis_global_thread_count = thread_count + _warn_missing_node_override("Redis", node_thread_counts, thread_count) logger.info(f"从Redis加载检测线程数成功: {thread_count}") return thread_count @@ -1363,7 +2255,7 @@ class DetectWorker: if node_code and isinstance(node_thread_counts, dict): node_thread_count = node_thread_counts.get(node_code) if node_thread_count is not None: - thread_count = min(max_recommended_threads, max(1, int(node_thread_count))) + thread_count = max(1, int(node_thread_count)) logger.info(f"从本地文件加载节点专属检测线程数成功: {node_code} -> {thread_count}") return thread_count @@ -1371,7 +2263,9 @@ class DetectWorker: with open('thread_count.json', 'r', encoding='utf-8') as f: thread_config = json.load(f) thread_count = thread_config.get('thread_count', '10') - thread_count = min(max_recommended_threads, max(1, int(thread_count))) + thread_count = max(1, int(thread_count)) + if redis_global_thread_count is None: + _warn_missing_node_override("本地文件", node_thread_counts, thread_count) logger.info(f"从本地文件加载检测线程数成功: {thread_count}") return thread_count else: @@ -1384,10 +2278,190 @@ class DetectWorker: logger.info(f"使用默认检测线程数: {default_thread_count}") return default_thread_count + def refresh_thread_count_runtime(self, *, force=False, min_interval=2.0): + """ + 运行中热刷新线程数配置。 + + Web 后台改了节点并发后,不应该等整批任务跑完才生效。 + """ + now_ts = time.time() + last_refresh_at = float(getattr(self, "_last_thread_count_refresh_at", 0.0) or 0.0) + if not force and now_ts - last_refresh_at < max(0.2, float(min_interval or 0.0)): + return max(1, int(getattr(self, "thread_count", 1) or 1)) + + previous = max(1, int(getattr(self, "thread_count", 1) or 1)) + latest = max(1, int(self.load_thread_count() or previous)) + self._last_thread_count_refresh_at = now_ts + if latest != previous: + self.thread_count = latest + self.proxy_max_reuse_count = max(8, self.thread_count * 4) + logger.info(f"检测线程数热更新: {previous} -> {latest}") + self.update_config_labels() + return max(1, int(getattr(self, "thread_count", latest) or latest)) + + def _pull_sync_tasks_until_available(self, *, thread_limit: int) -> list[dict]: + """ + mainland-controller 在本地任务吃空时,不要立即退出检测循环,而是主动向海外控制面 + 连续补货几轮,尽量把本地队列重新喂满。 + """ + if str(getattr(config, "NODE_REGION", "") or "").strip() != "mainland": + return [] + if str(getattr(config, "NODE_ROLE", "") or "").strip() != "control": + return [] + + max_rounds = max(1, int(os.getenv("DOMAINCHECK_SYNC_PULL_BURST_ROUNDS", "4") or 4)) + current_thread_limit = max(1, int(thread_limit or getattr(self, "thread_count", 1) or 1)) + configured_pull_limit = int(os.getenv("DOMAINCHECK_SYNC_PULL_LIMIT", "0") or 0) + pull_limit = max( + 1000, + configured_pull_limit if configured_pull_limit > 0 else min(20000, current_thread_limit * 4), + ) + claim_batch_size = max(20, current_thread_limit) + claim_lease_seconds = max(300, min(1800, claim_batch_size * 30)) + local_api_base_url = str( + os.getenv("DOMAINCHECK_LOCAL_API_BASE_URL", "http://127.0.0.1:8100/api/v1") or "http://127.0.0.1:8100/api/v1" + ).strip().rstrip("/") + + for round_index in range(1, max_rounds + 1): + try: + request = urllib.request.Request( + f"{local_api_base_url}/runtime/actions/pull_tasks", + data=json.dumps({"limit": pull_limit}, ensure_ascii=False).encode("utf-8"), + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(request, timeout=max(15, min(60, 10 + pull_limit // 100))) as response: + raw = response.read().decode("utf-8") + response_data = json.loads(raw) if raw else {} + pull_ok = int(response_data.get("code", 1) or 1) == 0 + pull_message = str(response_data.get("message") or "").strip() + pull_data = response_data.get("data") or {} + except Exception as exc: + logger.warning(f"本地队列补货失败: round={round_index}/{max_rounds}, error={exc}") + self._sync_worker_log_event( + "本地队列补货失败", + level='warning', + payload={ + "round": round_index, + "max_rounds": max_rounds, + "pull_limit": pull_limit, + "error": str(exc), + }, + mode='full', + ) + break + + pull_state = str((pull_data or {}).get("pull_state") or "").strip() + queued_count = int((pull_data or {}).get("queued_count", 0) or 0) + worker_start_message = str((pull_data or {}).get("worker_start_message") or "").strip() + refill_message = ( + f"本地队列主动补货: round={round_index}/{max_rounds} | " + f"pull_limit={pull_limit} | ok={1 if pull_ok else 0} | " + f"state={pull_state or 'unknown'} | queued_count={queued_count}" + ) + if worker_start_message: + refill_message = f"{refill_message} | worker_start={worker_start_message}" + logger.info(f"{refill_message} | message={pull_message}") + self._sync_worker_log_event( + refill_message, + payload={ + "round": round_index, + "max_rounds": max_rounds, + "pull_limit": pull_limit, + "pull_ok": bool(pull_ok), + "pull_state": pull_state, + "queued_count": queued_count, + "message": str(pull_message or "").strip(), + "worker_start_message": worker_start_message, + }, + mode='full', + ) + + domains = self.db.claim_detect_job_items( + config.NODE_CODE, + limit=claim_batch_size, + lease_seconds=claim_lease_seconds, + ) + if domains: + logger.info( + f"主动补货后已重新领取任务: round={round_index}/{max_rounds}, " + f"claim_batch_size={claim_batch_size}, domains={len(domains)}" + ) + return domains + + if pull_state == "idle": + break + if not pull_ok and pull_state not in {"ack_warning", "worker_start_warning"}: + break + time.sleep(0.2) + + return [] + + def _process_pipeline_tasks_until_available(self, *, thread_limit: int) -> list[dict]: + """ + 当本地任务队列空了时,主动让 controller 处理已完成步骤,尽量把下一步任务补出来。 + """ + local_api_base_url = str( + os.getenv("DOMAINCHECK_LOCAL_API_BASE_URL", "http://127.0.0.1:8100/api/v1") or "http://127.0.0.1:8100/api/v1" + ).strip().rstrip("/") + current_thread_limit = max(1, int(thread_limit or getattr(self, "thread_count", 1) or 1)) + configured_process_limit = int(os.getenv("DOMAINCHECK_PIPELINE_PROCESS_LIMIT", "0") or 0) + process_limit = max( + 20, + configured_process_limit if configured_process_limit > 0 else current_thread_limit * 4, + ) + claim_batch_size = max(20, current_thread_limit) + claim_lease_seconds = max(300, min(1800, claim_batch_size * 30)) + + try: + request = urllib.request.Request( + f"{local_api_base_url}/runtime/actions/process_pipeline", + data=json.dumps({"limit": process_limit}, ensure_ascii=False).encode("utf-8"), + headers={"Content-Type": "application/json"}, + method="POST", + ) + with urllib.request.urlopen(request, timeout=20) as response: + raw = response.read().decode("utf-8") + response_data = json.loads(raw) if raw else {} + process_ok = int(response_data.get("code", 1) or 1) == 0 + process_message = str(response_data.get("message") or "").strip() + process_data = response_data.get("data") or {} + logger.info( + f"本地 pipeline 推进: ok={1 if process_ok else 0} | " + f"processed={int(process_data.get('processed_items', 0) or 0)} | " + f"advanced={int(process_data.get('advanced_items', 0) or 0)} | " + f"retried={int(process_data.get('retried_items', 0) or 0)} | " + f"message={process_message}" + ) + except Exception as exc: + logger.warning(f"本地 pipeline 推进失败: {exc}") + self._sync_worker_log_event( + "本地 pipeline 推进失败", + level='warning', + payload={"error": str(exc), "process_limit": process_limit}, + mode='full', + ) + return [] + + domains = self.db.claim_detect_job_items( + config.NODE_CODE, + limit=claim_batch_size, + lease_seconds=claim_lease_seconds, + ) + if domains: + self._sync_worker_log_event( + "本地 pipeline 推进后已重新领取任务", + payload={"domains": len(domains), "claim_batch_size": claim_batch_size}, + mode='full', + ) + return domains + def load_runtime_settings(self): default_settings = { "worker_log_sync_enabled": False, "worker_log_sync_mode": "key", + "worker_step_trace_enabled": True, + "worker_step_trace_sync_full": True, } try: if self.use_redis: @@ -1397,12 +2471,15 @@ class DetectWorker: runtime_settings.update(json.loads(runtime_settings_raw)) runtime_settings["worker_log_sync_enabled"] = bool(runtime_settings.get("worker_log_sync_enabled", False)) runtime_settings["worker_log_sync_mode"] = "full" if str(runtime_settings.get("worker_log_sync_mode", "key")).strip().lower() == "full" else "key" + runtime_settings["worker_step_trace_enabled"] = bool(runtime_settings.get("worker_step_trace_enabled", True)) + runtime_settings["worker_step_trace_sync_full"] = bool(runtime_settings.get("worker_step_trace_sync_full", True)) logger.info(f"从Redis加载运行时设置成功: {runtime_settings}") return runtime_settings candidate_paths = [ 'runtime_settings.json', os.path.join('runtime', 'runtime_settings.json'), + os.path.join('..', 'domain-api', 'runtime', 'runtime_settings.json'), ] for candidate_path in candidate_paths: if os.path.exists(candidate_path): @@ -1411,17 +2488,250 @@ class DetectWorker: runtime_settings.update(json.load(f)) runtime_settings["worker_log_sync_enabled"] = bool(runtime_settings.get("worker_log_sync_enabled", False)) runtime_settings["worker_log_sync_mode"] = "full" if str(runtime_settings.get("worker_log_sync_mode", "key")).strip().lower() == "full" else "key" + runtime_settings["worker_step_trace_enabled"] = bool(runtime_settings.get("worker_step_trace_enabled", True)) + runtime_settings["worker_step_trace_sync_full"] = bool(runtime_settings.get("worker_step_trace_sync_full", True)) logger.info(f"从本地文件加载运行时设置成功: {runtime_settings}") return runtime_settings except Exception as e: logger.error(f"加载运行时设置失败: {e}") return default_settings + def _load_sensitive_words_runtime(self): + try: + if self.use_redis and self.redis_client is not None: + sensitive_words_raw = self.redis_client.get("domain_tool:sensitive_words") + if sensitive_words_raw is not None: + try: + parsed = json.loads(sensitive_words_raw) + except Exception: + parsed = sensitive_words_raw + if isinstance(parsed, list): + words = [str(item).strip() for item in parsed if str(item).strip()] + else: + words = [line.strip() for line in str(parsed or "").splitlines() if line.strip()] + logger.info(f"从Redis加载敏感词成功,共 {len(words)} 个敏感词") + return words + + candidate_paths = [ + 'runtime/sensitive_words.json', + 'sensitive_words.json', + 'sensitive_words.txt', + ] + for candidate_path in candidate_paths: + if not os.path.exists(candidate_path): + continue + if candidate_path.endswith('.json'): + with open(candidate_path, 'r', encoding='utf-8') as f: + payload = json.load(f) + if isinstance(payload, dict): + if isinstance(payload.get('items'), list): + words = [] + for item in payload.get('items') or []: + if isinstance(item, dict): + word = str(item.get('word') or '').strip() + else: + word = str(item).strip() + if word: + words.append(word) + else: + words = [line.strip() for line in str(payload.get('text') or '').splitlines() if line.strip()] + elif isinstance(payload, list): + words = [str(item).strip() for item in payload if str(item).strip()] + else: + words = [] + else: + with open(candidate_path, 'r', encoding='utf-8') as f: + words = [line.strip() for line in f.read().splitlines() if line.strip()] + logger.info(f"从本地文件加载敏感词成功: {candidate_path},共 {len(words)} 个敏感词") + return words + except Exception as e: + logger.error(f"从运行时配置加载敏感词失败: {e}") + + words = self.db.get_all_sensitive_words() + logger.info(f"从数据库加载敏感词成功,共 {len(words)} 个敏感词") + return words + + @staticmethod + def _remote_debug_ingest_url(): + base_url = str(os.getenv("SYNC_TARGET_API_BASE_URL", "") or "").strip().rstrip("/") + if not base_url: + return "" + if base_url.endswith("/api/v1"): + return f"{base_url}/runtime/debug-ingest" + if base_url.endswith("/api/v1/runtime"): + return f"{base_url}/debug-ingest" + return f"{base_url}/api/v1/runtime/debug-ingest" + + def _push_remote_debug_event(self, *, message, level='info', payload=None, event_type='worker_log'): + ingest_url = self._remote_debug_ingest_url() + if not ingest_url: + return + request_payload = { + "source_region": config.NODE_REGION, + "node_code": config.NODE_CODE, + "hostname": socket.gethostname(), + "service": "worker-event", + "event_type": str(event_type or "worker_log"), + "level": str(level or "info"), + "message": str(message or "").strip()[:2000], + "payload": payload or {}, + } + headers = { + "Content-Type": "application/json", + } + shared_token = str(os.getenv("SYNC_SHARED_TOKEN", "") or "").strip() + if shared_token: + headers["X-Domaincheck-Sync-Token"] = shared_token + request = urllib.request.Request( + ingest_url, + data=json.dumps(request_payload, ensure_ascii=False, sort_keys=True).encode("utf-8"), + headers=headers, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=REMOTE_DEBUG_EVENT_TIMEOUT) as response: + raw = response.read().decode("utf-8", errors="ignore") + if raw: + parsed = json.loads(raw) + if isinstance(parsed, dict) and parsed.get("code") not in (0, "0", None, ""): + logger.debug(f"远端调试事件返回非成功: {parsed}") + except Exception as e: + logger.debug(f"远端调试事件回传失败: {e}") + def _worker_log_sync_mode(self): if not bool((self.runtime_settings or {}).get("worker_log_sync_enabled", False)): return "off" return "full" if str((self.runtime_settings or {}).get("worker_log_sync_mode", "key")).strip().lower() == "full" else "key" + def _worker_step_trace_enabled(self): + return bool((self.runtime_settings or {}).get("worker_step_trace_enabled", True)) + + def _worker_step_trace_sync_full_enabled(self): + return bool((self.runtime_settings or {}).get("worker_step_trace_sync_full", True)) + + @staticmethod + def _single_step_cache_key(domain_id, field_name): + return f"domain_tool:single_step_result:{int(domain_id or 0)}:{str(field_name or '').strip()}" + + def _set_current_task_context(self, *, domain_id, is_step_task, task_mode, job_item_id=None): + self._task_local.domain_id = int(domain_id or 0) + self._task_local.is_step_task = bool(is_step_task) + self._task_local.task_mode = str(task_mode or "").strip() + self._task_local.job_item_id = int(job_item_id or 0) if job_item_id else None + self._task_local.detect_status = None + self._task_local.step_proxy_retry_waits = {} + self._task_local.step_force_direct_once = {} + + def _clear_current_task_context(self): + for attr in ("domain_id", "is_step_task", "task_mode", "job_item_id", "detect_status", "step_proxy_retry_waits", "step_force_direct_once"): + try: + delattr(self._task_local, attr) + except Exception: + pass + + def _is_current_single_step_task(self): + return bool(getattr(self._task_local, "is_step_task", False)) + + def _set_task_detect_status(self, status): + try: + normalized_status = int(status) + except Exception: + normalized_status = None + self._task_local.detect_status = normalized_status + + def _get_task_detect_status(self): + try: + value = getattr(self._task_local, "detect_status", None) + return int(value) if value is not None else None + except Exception: + return None + + def _resolve_current_task_final_status(self): + current_status = self._get_task_detect_status() + if current_status == DETECT_STATUS_BLACKLISTED: + return 'blacklisted' + return 'failed' + + def _set_step_proxy_retry_wait(self, step_name, wait_seconds): + normalized_step_name = str(step_name or "").strip() + if not normalized_step_name: + return + try: + normalized_wait_seconds = max(0.0, float(wait_seconds or 0.0)) + except Exception: + normalized_wait_seconds = 0.0 + retry_waits = getattr(self._task_local, "step_proxy_retry_waits", None) + if not isinstance(retry_waits, dict): + retry_waits = {} + self._task_local.step_proxy_retry_waits = retry_waits + retry_waits[normalized_step_name] = normalized_wait_seconds + + def _consume_step_proxy_retry_wait(self, step_name): + normalized_step_name = str(step_name or "").strip() + if not normalized_step_name: + return 0.0 + retry_waits = getattr(self._task_local, "step_proxy_retry_waits", None) + if not isinstance(retry_waits, dict): + return 0.0 + try: + return max(0.0, float(retry_waits.pop(normalized_step_name, 0.0) or 0.0)) + except Exception: + return 0.0 + + def _set_step_force_direct_once(self, step_name, enabled=True): + normalized_step_name = str(step_name or "").strip() + if not normalized_step_name: + return + force_map = getattr(self._task_local, "step_force_direct_once", None) + if not isinstance(force_map, dict): + force_map = {} + self._task_local.step_force_direct_once = force_map + force_map[normalized_step_name] = bool(enabled) + + def _consume_step_force_direct_once(self, step_name): + normalized_step_name = str(step_name or "").strip() + if not normalized_step_name: + return False + force_map = getattr(self._task_local, "step_force_direct_once", None) + if not isinstance(force_map, dict): + return False + return bool(force_map.pop(normalized_step_name, False)) + + def _cache_single_step_result_payload(self, domain_id, field_name, payload): + normalized_field_name = str(field_name or "").strip() + if not normalized_field_name: + return + cache_key = self._single_step_cache_key(domain_id, normalized_field_name) + with self._step_result_cache_lock: + self._step_result_cache[(int(domain_id or 0), normalized_field_name)] = dict(payload or {}) + if self.use_redis and self.redis_client is not None: + try: + self.redis_client.setex(cache_key, 600, json.dumps(payload or {}, ensure_ascii=False)) + except Exception as e: + logger.debug(f"写入 Redis single_step 结果缓存失败: {e}") + + def _load_cached_single_step_result_payload(self, domain_id, field_name): + normalized_field_name = str(field_name or "").strip() + if not normalized_field_name: + return None + cache_tuple_key = (int(domain_id or 0), normalized_field_name) + with self._step_result_cache_lock: + cached_payload = self._step_result_cache.pop(cache_tuple_key, None) + if isinstance(cached_payload, dict) and cached_payload: + return dict(cached_payload) + if self.use_redis and self.redis_client is not None: + try: + redis_key = self._single_step_cache_key(domain_id, normalized_field_name) + raw_payload = self.redis_client.get(redis_key) + if raw_payload: + self.redis_client.delete(redis_key) + parsed = json.loads(raw_payload) + if isinstance(parsed, dict): + return parsed + except Exception as e: + logger.debug(f"读取 Redis single_step 结果缓存失败: {e}") + return None + def _sync_worker_log_event(self, message, level='info', payload=None, *, mode='key', job_item_id=None): current_mode = self._worker_log_sync_mode() if current_mode == "off": @@ -1433,10 +2743,14 @@ class DetectWorker: return if normalized_message == self._last_synced_worker_log and mode != 'full': return - job_id = self.current_job_id + event_payload = dict(payload or {}) + explicit_job_id = event_payload.get("job_id") + try: + job_id = int(explicit_job_id) if explicit_job_id not in (None, "", 0, "0") else self.current_job_id + except Exception: + job_id = self.current_job_id if not job_id: return - event_payload = dict(payload or {}) if self.current_cycle_token and not event_payload.get("cycle_token"): event_payload["cycle_token"] = self.current_cycle_token if self.current_job_code and not event_payload.get("job_code"): @@ -1444,18 +2758,157 @@ class DetectWorker: event_payload["log_mode"] = mode event_payload["synced_by"] = "worker_log_callback" try: - self.db.append_detect_run_event( - job_id, - job_item_id, - config.NODE_CODE, - event_type='worker_log', - message=normalized_message, - level=level, - payload=event_payload, - ) + queue_item = { + "job_id": job_id, + "job_item_id": job_item_id, + "message": normalized_message, + "level": level, + "payload": event_payload, + "write_db": not (mode == 'full' and self._is_current_single_step_task()), + } + try: + self._worker_log_sync_queue.put_nowait(queue_item) + except Full: + if mode != 'full': + self._worker_log_sync_queue.put(queue_item, timeout=0.2) + else: + raise self._last_synced_worker_log = normalized_message except Exception as e: - logger.debug(f"回传Worker日志事件失败: {e}") + if mode == 'full': + self._worker_log_sync_drop_count += 1 + now_ts = time.time() + if now_ts - self._last_worker_log_sync_drop_notice_at >= 10: + logger.warning( + f"Worker 全量日志同步队列拥堵,已丢弃 {self._worker_log_sync_drop_count} 条 full 日志" + ) + self._last_worker_log_sync_drop_notice_at = now_ts + else: + logger.debug(f"回传Worker日志事件失败: {e}") + + @staticmethod + def _step_display_name(detect_key): + mapping = { + 'detect_register': '注册状态检测', + 'detect_baidu_site': '百度site检测', + 'detect_360_site': '360 site检测', + 'detect_chinaz': '站长之家检测', + 'detect_aizhan': '爱站检测', + 'detect_wayback': '时光机检测', + 'detect_jucha': '聚查检测', + 'detect_juziseo': '桔子检测', + } + return mapping.get(str(detect_key or '').strip(), str(detect_key or '').strip() or 'unknown') + + @staticmethod + def _step_result_field_name(detect_key): + mapping = { + 'detect_register': 'register_status', + 'detect_baidu_site': 'baidu_site', + 'detect_360_site': 'qihu360_site', + 'detect_chinaz': 'chinaz_info', + 'detect_aizhan': 'aizhan_info', + 'detect_wayback': 'wayback_info', + } + return mapping.get(str(detect_key or '').strip(), '') + + def _build_single_step_job_result(self, *, domain_id, detect_key, step_ok, step_name): + field_name = self._step_result_field_name(detect_key) + cached_payload = self._load_cached_single_step_result_payload(domain_id, field_name) + if isinstance(cached_payload, dict) and cached_payload: + payload = dict(cached_payload) + payload["step_code"] = str(detect_key or "").strip() + payload["step_name"] = str(step_name or "").strip() + payload["field_name"] = field_name + payload["task_mode"] = "single_step" + payload["domain_id"] = int(domain_id or 0) + if payload.get("state") == "degraded": + payload["retry_recommended"] = True + return payload + try: + domain_snapshot = self.db.get_domain_by_id(domain_id) or {} + except Exception: + domain_snapshot = {} + raw_payload = domain_snapshot.get(field_name) if field_name else None + payload = dict(raw_payload) if isinstance(raw_payload, dict) else {} + if not payload: + payload = { + "status": bool(step_ok), + "state": "passed" if step_ok else "failed", + "message": "success" if step_ok else f"{step_name} 未通过", + } + payload["step_code"] = str(detect_key or "").strip() + payload["step_name"] = str(step_name or "").strip() + payload["field_name"] = field_name + payload["task_mode"] = "single_step" + payload["domain_id"] = int(domain_id or 0) + if payload.get("state") == "degraded": + payload["retry_recommended"] = True + return payload + + @staticmethod + def _resolve_single_step_finalization(result_payload): + payload = dict(result_payload or {}) + state = str(payload.get("state") or "").strip().lower() + message = str(payload.get("message") or "").strip() + status = payload.get("status") + if state == "blacklisted": + return "blacklisted", message or "步骤命中黑名单" + if state == "degraded": + return "failed", message or "步骤外部依赖异常,建议重试" + if state in {"failed", "error", "rejected"}: + return "failed", message or "步骤执行失败" + if status is False: + return "failed", message or "步骤执行失败" + return "completed", message or "步骤执行完成" + + def _emit_step_trace(self, domain_name, step_name, stage, *, elapsed_ms=None, ok=None, level='info', job_item_id=None, sync_mode=None, **extra): + if not self._worker_step_trace_enabled(): + return + payload = { + "domain": str(domain_name or "").strip(), + "step": str(step_name or "").strip(), + "stage": str(stage or "").strip(), + } + if elapsed_ms is not None: + payload["elapsed_ms"] = int(elapsed_ms or 0) + if ok is not None: + payload["ok"] = bool(ok) + for key, value in dict(extra or {}).items(): + if value in (None, ""): + continue + payload[key] = value + + message_parts = [ + f"domain={payload.get('domain') or '-'}", + f"step={payload.get('step') or '-'}", + f"stage={payload.get('stage') or '-'}", + ] + if "elapsed_ms" in payload: + message_parts.append(f"elapsed_ms={payload['elapsed_ms']}") + if "ok" in payload: + message_parts.append(f"ok={1 if payload['ok'] else 0}") + for key in sorted(payload.keys()): + if key in {"domain", "step", "stage", "elapsed_ms", "ok"}: + continue + value = payload.get(key) + if isinstance(value, (dict, list, tuple)): + value = json.dumps(value, ensure_ascii=False, sort_keys=True) + message_parts.append(f"{key}={value}") + message = "检测步骤跟踪: " + " | ".join(message_parts) + logger.log(level.upper(), message) + + should_sync_full = self._worker_step_trace_sync_full_enabled() + if sync_mode == 'off': + should_sync_full = False + if should_sync_full: + self._sync_worker_log_event( + message, + level=level, + payload=payload, + mode='full', + job_item_id=job_item_id, + ) def test_proxy(self, proxy_item, result_queue): """ @@ -1491,6 +2944,8 @@ class DetectWorker: test_targets = [ ("http://www.baidu.com", 8), ("https://m.baidu.com", 8), + ("http://www.qq.com", 8), + ("http://www.360.cn", 8), ] last_reason = "unknown" session = requests.Session() @@ -1512,7 +2967,7 @@ class DetectWorker: allow_redirects=True, verify=False if target_url.startswith("https://") else True, ) - if test_response.status_code in (200, 301, 302): + if test_response.status_code in (200, 204, 301, 302, 403): result_payload["ok"] = True result_payload["reason"] = f"ok:{target_url}:{test_response.status_code}" break @@ -1539,6 +2994,177 @@ class DetectWorker: return proxy_data return [] + def _proxy_source_tag_from_url(self, raw_url): + normalized_url = str(raw_url or "").strip() + if not normalized_url: + return "unknown" + try: + parsed = urllib.parse.urlsplit(normalized_url) + query = urllib.parse.parse_qs(parsed.query, keep_blank_values=True) + group_value = str((query.get("group") or [""])[0] or "").strip() + if group_value: + return group_value.upper() + if parsed.netloc: + return parsed.netloc + except Exception: + pass + return normalized_url + + def _interleave_proxy_entries(self, proxy_entries): + if not proxy_entries: + return [] + grouped_entries = collections.OrderedDict() + for proxy_entry in proxy_entries: + source_tag = str((proxy_entry or {}).get("source_tag") or "unknown").strip() or "unknown" + grouped_entries.setdefault(source_tag, collections.deque()).append(proxy_entry) + merged_entries = [] + while grouped_entries: + exhausted_tags = [] + for source_tag, items in grouped_entries.items(): + if not items: + exhausted_tags.append(source_tag) + continue + merged_entries.append(items.popleft()) + if not items: + exhausted_tags.append(source_tag) + for source_tag in exhausted_tags: + grouped_entries.pop(source_tag, None) + return merged_entries + + def _merge_proxy_entries(self, new_entries, existing_entries): + merged = [] + seen = set() + for proxy_entry in list(new_entries or []) + list(existing_entries or []): + if isinstance(proxy_entry, dict) and proxy_entry.get("proxy"): + signature = self._proxy_key(proxy_entry.get("proxy")) + else: + signature = self._proxy_key(proxy_entry) + if not signature or signature in seen: + continue + seen.add(signature) + merged.append(proxy_entry) + if merged and all(isinstance(item, dict) and item.get("proxy") for item in merged): + return self._interleave_proxy_entries(merged) + return merged + + def _proxy_signature_from_item(self, proxy_item): + if not isinstance(proxy_item, dict): + return "" + ip = str(proxy_item.get("ip", "") or "").strip() + port = str(proxy_item.get("port", "") or "").strip() + username = str(proxy_item.get("username", "") or "").strip() + password = str(proxy_item.get("password", "") or "").strip() + if not ip or not port: + return "" + return json.dumps( + { + "ip": ip, + "port": port, + "username": username, + "password": password, + }, + sort_keys=True, + ensure_ascii=False, + ) + + def _is_proxy_item_expired(self, proxy_item): + if not isinstance(proxy_item, dict): + return True + now_ts = time.time() + expire_at_ms = proxy_item.get("expire_at_ms") + expire_at = proxy_item.get("expire_at") + try: + if expire_at_ms not in (None, "", 0, "0"): + return float(expire_at_ms) / 1000.0 <= now_ts + if expire_at not in (None, "", 0, "0"): + return float(expire_at) <= now_ts + except Exception: + return False + return False + + def _set_proxy_request_count(self, raw_url, count): + normalized_url = str(raw_url or "").strip() + if not normalized_url: + return "" + try: + parsed = urllib.parse.urlsplit(normalized_url) + query = urllib.parse.parse_qs(parsed.query, keep_blank_values=True) + query["count"] = [str(max(1, int(count or 0)))] + return urllib.parse.urlunsplit( + ( + parsed.scheme, + parsed.netloc, + parsed.path, + urllib.parse.urlencode(query, doseq=True), + parsed.fragment, + ) + ) + except Exception: + return normalized_url + + def _prepare_proxy_fetch_plan(self, proxy_api_urls, current_pool_size=0): + source_urls = [str(url or "").strip() for url in proxy_api_urls if str(url or "").strip()] + if not source_urls: + return [], { + "target_total": 0, + "batch_size": 0, + "rounds": 0, + "source_count": 0, + } + + demand_threads = self._proxy_demand_threads() + source_count = len(source_urls) + provider_batch_cap = max(20, int(os.getenv("DOMAINCHECK_PROXY_PROVIDER_BATCH_CAP", "140") or 140)) + current_pool_size = max(0, int(current_pool_size or 0)) + refresh_threshold = self._proxy_refresh_threshold() + shortage = max(0, refresh_threshold - current_pool_size) + headroom = max(source_count * 20, int(refresh_threshold * 0.15)) + active_round_cap = max(1, int(os.getenv("DOMAINCHECK_PROXY_MAX_ROUNDS_ACTIVE", "2") or 2)) + idle_round_cap = max(2, int(os.getenv("DOMAINCHECK_PROXY_MAX_ROUNDS_IDLE", "4") or 4)) + round_cap = active_round_cap if self.detecting else idle_round_cap + target_total = max( + source_count * provider_batch_cap, + shortage + headroom, + ) + per_round_capacity = max(1, source_count * provider_batch_cap) + rounds = max(1, min(round_cap, (target_total + per_round_capacity - 1) // per_round_capacity)) + batch_size = max( + 20, + min( + provider_batch_cap, + (target_total + (source_count * rounds) - 1) // (source_count * rounds), + ), + ) + fetch_plan = [] + for round_index in range(rounds): + for source_url in source_urls: + fetch_plan.append( + { + "source_url": source_url, + "request_url": self._set_proxy_request_count(source_url, batch_size), + "batch_size": batch_size, + "round": round_index + 1, + } + ) + + return fetch_plan, { + "target_total": target_total, + "batch_size": batch_size, + "rounds": rounds, + "source_count": source_count, + "demand_threads": demand_threads, + "current_pool_size": current_pool_size, + "refresh_threshold": refresh_threshold, + "shortage": shortage, + } + + def _proxy_source_fetch_timeout_seconds(self): + default_timeout = "4.0" if self.detecting else "6.0" + return max( + 1.5, + float(os.getenv("DOMAINCHECK_PROXY_SOURCE_FETCH_TIMEOUT", default_timeout) or default_timeout), + ) + def _proxy_key(self, proxy): if isinstance(proxy, dict): return json.dumps(proxy, sort_keys=True, ensure_ascii=False) @@ -1573,13 +3199,174 @@ class DetectWorker: self.proxy_failure_counts.pop(proxy_key, None) self.proxy_quarantine_until.pop(proxy_key, None) - def _schedule_proxy_refresh_if_needed(self, available_count=None): + def _proxy_source_tag_for_proxy(self, proxy): + proxy_key = self._proxy_key(proxy) + with self.proxy_failure_lock: + source_map = getattr(self, "proxy_source_by_key", None) or {} + return str(source_map.get(proxy_key, "") or "").strip() + + def _is_proxy_source_quarantined(self, source_tag): + normalized_source_tag = str(source_tag or "").strip() + if not normalized_source_tag: + return False + now_ts = time.time() + with self.proxy_failure_lock: + quarantine_map = getattr(self, "proxy_source_quarantine_until", None) or {} + retry_after = float(quarantine_map.get(normalized_source_tag, 0) or 0) + if retry_after <= now_ts: + if normalized_source_tag in quarantine_map: + quarantine_map.pop(normalized_source_tag, None) + (getattr(self, "proxy_source_failure_counts", None) or {}).pop(normalized_source_tag, None) + return False + return True + + def _is_proxy_auth_issue(self, message): + lowered = str(message or "").lower() + return any( + keyword in lowered + for keyword in ( + "407 proxy authentication required", + "proxy authentication required", + "tunnel connection failed: 407", + "status code 407", + ) + ) + + def _remove_proxy_source_from_pool(self, source_tag): + normalized_source_tag = str(source_tag or "").strip() + if not normalized_source_tag: + return 0 + removed = 0 with self.proxy_pool_lock: - current_count = len(self.proxy_pool) if available_count is None else available_count - threshold = max(2, self.thread_count // 2) + filtered_pool = [] + for proxy_item in list(getattr(self, "proxy_pool", []) or []): + item_source_tag = "" + if isinstance(proxy_item, dict): + item_source_tag = str(proxy_item.get("source_tag") or "").strip() + if item_source_tag and item_source_tag == normalized_source_tag: + removed += 1 + continue + filtered_pool.append(proxy_item) + if removed: + self.proxy_pool = filtered_pool + return removed + + def _mark_proxy_source_failure(self, proxy, reason=""): + source_tag = self._proxy_source_tag_for_proxy(proxy) + if not source_tag: + return "", 0, 0, 0 + + with self.proxy_failure_lock: + failure_counts = getattr(self, "proxy_source_failure_counts", None) + if failure_counts is None: + self.proxy_source_failure_counts = {} + failure_counts = self.proxy_source_failure_counts + quarantine_map = getattr(self, "proxy_source_quarantine_until", None) + if quarantine_map is None: + self.proxy_source_quarantine_until = {} + quarantine_map = self.proxy_source_quarantine_until + failure_count = int(failure_counts.get(source_tag, 0) or 0) + 1 + failure_counts[source_tag] = failure_count + if self._is_proxy_auth_issue(reason): + cooldown_seconds = min(600, max(90, 90 * failure_count)) + else: + cooldown_seconds = min(300, max(30, 30 * failure_count)) + quarantine_map[source_tag] = time.time() + cooldown_seconds + + removed = self._remove_proxy_source_from_pool(source_tag) + logger.warning( + f"代理来源已临时隔离 {cooldown_seconds}s: source={source_tag}, failures={failure_count}, removed={removed}, reason={reason}" + ) + return source_tag, failure_count, cooldown_seconds, removed + + def _clear_proxy_source_failure(self, proxy): + source_tag = self._proxy_source_tag_for_proxy(proxy) + if not source_tag: + return + with self.proxy_failure_lock: + (getattr(self, "proxy_source_failure_counts", None) or {}).pop(source_tag, None) + (getattr(self, "proxy_source_quarantine_until", None) or {}).pop(source_tag, None) + + def _schedule_proxy_refresh_if_needed(self, available_count=None): + if available_count is None: + with self.proxy_pool_lock: + current_count = len(self.proxy_pool) + else: + # 调用方可能已经持有 proxy_pool_lock;避免对同一把非重入锁再次加锁。 + current_count = int(available_count or 0) + threshold = self._proxy_refresh_threshold() if current_count < threshold and not self.proxy_refresh_lock.locked(): threading.Thread(target=self.refresh_proxy_pool, daemon=True).start() + def _has_proxy_sources(self): + proxy_urls = self.proxy_config.get('proxy_urls') or [] + if not proxy_urls and self.proxy_config.get('proxy_url'): + proxy_urls = [self.proxy_config.get('proxy_url', '')] + return bool([url for url in proxy_urls if str(url or '').strip()]) + + def trigger_proxy_refresh(self, reason="", reset_cooldown=False): + """ + 代理配置一旦下发,就允许在非检测态主动补一次刷新,避免页面长期停在“未刷新”。 + """ + if not self.proxy_config.get('proxy_enable', False): + return False + if not self._has_proxy_sources(): + return False + if reset_cooldown: + self.proxy_next_refresh_time = 0.0 + if self.proxy_refresh_lock.locked(): + logger.debug(f"代理池刷新已在进行中,跳过主动触发: {reason or 'manual'}") + return False + logger.info(f"主动触发代理池刷新: {reason or 'manual'}") + threading.Thread(target=self.refresh_proxy_pool, daemon=True).start() + return True + + def _proxy_refresh_threshold(self): + """ + 代理池的补货阈值只用于触发后台刷新,不应反向限制并发。 + """ + demand_threads = self._proxy_demand_threads() + return max(840, demand_threads * 3) + + def _proxy_demand_threads(self): + configured_threads = max(1, int(getattr(self, "thread_count", 0) or 0)) + live_active_threads = max(0, int(self._get_active_domain_threads() or 0)) + if self.detecting: + warm_floor = min(configured_threads, max(400, configured_threads // 2)) + demand_threads = max(live_active_threads, warm_floor) + else: + demand_threads = min(configured_threads, 64) + return max(1, demand_threads) + + def _should_emit_thread_progress(self, current_active, max_threads, last_emit_count, last_emit_at): + """ + 线程创建/回收阶段的活跃数上报做节流,避免每起一个线程就同步 + 一次远端日志和运行态,反向把并发拉低。 + """ + now = time.time() + step = max(1, min(10, max_threads // 10)) + if current_active == 1: + return True, now + if current_active >= max_threads: + return True, now + if current_active == 0 and last_emit_count != 0: + return True, now + if current_active - last_emit_count >= step: + return True, now + if current_active > 0 and now - last_emit_at >= 5.0: + return True, now + return False, now + + def _wait_for_proxy_refresh_settle(self, timeout_seconds=2.0): + deadline = time.time() + max(0.0, float(timeout_seconds or 0.0)) + while self.proxy_refresh_lock.locked() and time.time() < deadline: + with self.proxy_pool_lock: + if self.proxy_pool: + return True + time.sleep(0.05) + with self.proxy_pool_lock: + return bool(self.proxy_pool) + def refresh_proxy_pool(self): """ 刷新代理池 @@ -1607,8 +3394,10 @@ class DetectWorker: try: now_ts = time.time() + retry_delay_seconds = 0.15 if self.detecting else min(self.proxy_refresh_cooldown_seconds, 10) with self.proxy_pool_lock: has_cached_proxies = bool(self.proxy_pool) + cached_proxy_pool = list(self.proxy_pool) if not has_cached_proxies and self.proxy_next_refresh_time and now_ts < self.proxy_next_refresh_time: wait_seconds = int(max(1, self.proxy_next_refresh_time - now_ts)) self.proxy_last_refresh_status = f"冷却中,{wait_seconds} 秒后再试" @@ -1624,146 +3413,287 @@ class DetectWorker: if not proxy_api_urls and self.proxy_config.get('proxy_url'): proxy_api_urls = [self.proxy_config.get('proxy_url', '')] proxy_api_urls = [url for url in proxy_api_urls if url] - self.proxy_last_refresh_source_count = len(proxy_api_urls) + proxy_fetch_plan, fetch_plan_meta = self._prepare_proxy_fetch_plan( + proxy_api_urls, + current_pool_size=len(cached_proxy_pool), + ) + source_count = int(fetch_plan_meta.get("source_count", 0) or 0) + batch_size = int(fetch_plan_meta.get("batch_size", 0) or 0) + target_total = int(fetch_plan_meta.get("target_total", 0) or 0) + rounds = int(fetch_plan_meta.get("rounds", 0) or 0) + self.proxy_last_refresh_source_count = source_count self.proxy_last_source_stats = [] - if proxy_api_urls: + if proxy_fetch_plan: import requests - import threading - import random - from queue import Queue - proxy_list = [] - for proxy_api_url in proxy_api_urls: + cooldown_detected = False + + aggressive_proxy_rotation = str(os.getenv("DOMAINCHECK_PROXY_FAST_ROTATION", "1") or "1").strip().lower() not in { + "", + "0", + "false", + "no", + "off", + } + + def fetch_proxy_source(plan_item): + source_url = str(plan_item.get("source_url") or "").strip() + requested_batch_size = max(1, int(plan_item.get("batch_size", 0) or batch_size or 1)) + round_index = int(plan_item.get("round", 1) or 1) source_stat = { - "url": proxy_api_url, + "url": source_url, + "source_url": source_url, "status": "unknown", "http_status": 0, "raw_items": 0, "error": "", + "requested_batch_size": requested_batch_size, + "effective_batch_size": requested_batch_size, + "round": round_index, } - try: - response = requests.get(proxy_api_url, timeout=10) - source_stat["http_status"] = int(response.status_code or 0) - if response.status_code == 200: + current_items = [] + batch_attempts = [] + if self.detecting and aggressive_proxy_rotation: + dynamic_fallbacks = [ + requested_batch_size, + max(80, requested_batch_size // 2), + 80, + 50, + 20, + 10, + ] + else: + dynamic_fallbacks = [ + requested_batch_size, + max(500, requested_batch_size // 2), + 200, + 100, + 50, + 20, + 10, + ] + for candidate in dynamic_fallbacks: + candidate = max(1, int(candidate or 0)) + if candidate not in batch_attempts: + batch_attempts.append(candidate) + + for candidate_batch_size in batch_attempts: + request_url = self._set_proxy_request_count(source_url, candidate_batch_size) + source_stat["effective_batch_size"] = candidate_batch_size + try: + response = requests.get( + request_url, + timeout=self._proxy_source_fetch_timeout_seconds(), + ) + source_stat["http_status"] = int(response.status_code or 0) + if response.status_code != 200: + source_stat["status"] = "http_error" + source_stat["error"] = f"HTTP {response.status_code}" + logger.warning(f"代理池链接响应异常: {request_url}, 状态码: {response.status_code}") + continue + proxy_data = response.json() current_items = self._extract_proxy_items(proxy_data) - proxy_list.extend(current_items) + source_tag = self._proxy_source_tag_from_url(source_url) + annotated_items = [] + for proxy_item in current_items: + if isinstance(proxy_item, dict): + item_copy = dict(proxy_item) + item_copy.setdefault("source_url", source_url) + item_copy.setdefault("source_tag", source_tag) + annotated_items.append(item_copy) + current_items = annotated_items source_stat["raw_items"] = len(current_items) - source_stat["status"] = "ok" - logger.info(f"代理池链接拉取成功: {proxy_api_url}, 原始代理数: {len(current_items)}") - else: - source_stat["status"] = "http_error" - source_stat["error"] = f"HTTP {response.status_code}" - logger.warning(f"代理池链接响应异常: {proxy_api_url}, 状态码: {response.status_code}") - except Exception as api_error: - source_stat["status"] = "request_error" - source_stat["error"] = str(api_error) - logger.warning(f"代理池链接拉取失败: {proxy_api_url}, 错误: {api_error}") - self.proxy_last_source_stats.append(source_stat) + if current_items: + source_stat["status"] = "ok" + logger.info( + f"代理池链接拉取成功: {request_url}, 原始代理数: {len(current_items)}, 轮次: {round_index}" + ) + break + + error_message = "" + if isinstance(proxy_data, dict): + error_message = str(proxy_data.get("error") or "").strip() + source_stat["status"] = "cooldown" if "cooldown" in error_message.lower() else "empty" + source_stat["error"] = error_message or "empty" + logger.info( + f"代理池链接空载,降批重试: {request_url}, 轮次: {round_index}, 提示: {source_stat['error']}" + ) + if source_stat["status"] == "cooldown": + break + except Exception as api_error: + source_stat["status"] = "request_error" + source_stat["error"] = str(api_error) + logger.warning(f"代理池链接拉取失败: {request_url}, 错误: {api_error}") + source_stat["url"] = self._set_proxy_request_count(source_url, source_stat["effective_batch_size"]) + return source_stat, current_items + + round_plan_map = {} + for plan_item in proxy_fetch_plan: + round_key = max(1, int(plan_item.get("round", 1) or 1)) + round_plan_map.setdefault(round_key, []).append(plan_item) + + round_keys = sorted(round_plan_map.keys()) + per_round_workers = max(1, min(source_count or len(proxy_api_urls) or 1, 12)) + cooled_source_urls = set() + for round_index in round_keys: + round_plan_items = [ + item + for item in (round_plan_map.get(round_index) or []) + if str(item.get("source_url") or "").strip() not in cooled_source_urls + ] + if not round_plan_items: + continue + with ThreadPoolExecutor(max_workers=min(per_round_workers, len(round_plan_items))) as executor: + futures = [executor.submit(fetch_proxy_source, plan_item) for plan_item in round_plan_items] + for future in as_completed(futures): + source_stat, current_items = future.result() + if "cooldown" in str(source_stat.get("error") or "").lower() or source_stat.get("status") == "cooldown": + cooldown_detected = True + cooled_source_urls.add(str(source_stat.get("source_url") or "").strip()) + self.proxy_last_source_stats.append(source_stat) + proxy_list.extend(current_items) + if len(proxy_list) >= target_total: + break + if self.detecting and aggressive_proxy_rotation and round_index < len(round_keys): + time.sleep(0.15 if cooldown_detected else 0.05) if proxy_list: self.proxy_last_refresh_total_items = len(proxy_list) - max_validate_count = max(24, self.thread_count * 3) - if len(proxy_list) > max_validate_count: - proxy_list = random.sample(proxy_list, max_validate_count) - logger.info(f"代理池验证已启用抽样模式,本次抽样 {len(proxy_list)} 个代理进行可用性验证") - self.proxy_last_validated_count = len(proxy_list) - result_queue = Queue() - threads = [] - - for proxy_item in proxy_list: - thread = threading.Thread(target=self.test_proxy, args=(proxy_item, result_queue)) - thread.daemon = True - thread.start() - threads.append(thread) + self.proxy_last_validated_count = 0 new_proxies = [] + fallback_stale_proxies = [] seen_proxy_strings = set() - completed = 0 - start_time = time.time() - timeout = 30 - failure_reason_stats = collections.Counter() - failure_samples = [] + expired_count = 0 + invalid_count = 0 - while completed < len(threads) and time.time() - start_time < timeout: - try: - result = result_queue.get(timeout=1) - if result is not None: - if isinstance(result, dict) and result.get("ok"): - proxy_value = result.get("proxy") - proxy_signature = json.dumps(proxy_value, sort_keys=True) - if proxy_signature not in seen_proxy_strings: - seen_proxy_strings.add(proxy_signature) - new_proxies.append({'proxy': proxy_value, 'usage_count': 0}) - else: - reason = "" - if isinstance(result, dict): - reason = str(result.get("reason", "")).strip() - proxy_value = result.get("proxy") - if proxy_value and len(failure_samples) < 3: - failure_samples.append(f"{reason} -> {proxy_value}") - failure_reason_stats[reason or "unknown"] += 1 - completed += 1 - except Exception: - pass + for proxy_item in proxy_list: + signature = self._proxy_signature_from_item(proxy_item) + if not signature: + invalid_count += 1 + continue + if signature in seen_proxy_strings: + continue + seen_proxy_strings.add(signature) + ip = str(proxy_item.get("ip", "") or "").strip() + port = str(proxy_item.get("port", "") or "").strip() + username = str(proxy_item.get("username", "") or "").strip() + password = str(proxy_item.get("password", "") or "").strip() + proxy_url = f"http://{username}:{password}@{ip}:{port}" if username and password else f"http://{ip}:{port}" + source_tag = self._proxy_source_tag_from_url(proxy_item.get("source_url") or proxy_item.get("source_tag") or "") + if self._is_proxy_source_quarantined(source_tag): + continue + proxy_entry = { + 'proxy': { + 'http': proxy_url, + 'https': proxy_url, + }, + 'usage_count': 0, + 'source_tag': source_tag, + } - for thread in threads: - try: - thread.join(timeout=5) - except Exception: - pass + if self._is_proxy_item_expired(proxy_item): + expired_count += 1 + fallback_stale_proxies.append(proxy_entry) + continue + + new_proxies.append(proxy_entry) + + fallback_stale_count = 0 + if not new_proxies and fallback_stale_proxies and not cached_proxy_pool: + new_proxies = fallback_stale_proxies + fallback_stale_count = len(new_proxies) + + merged_proxy_pool = [] + if new_proxies: + merged_proxy_pool = self._merge_proxy_entries(new_proxies, cached_proxy_pool) + elif cached_proxy_pool: + merged_proxy_pool = list(cached_proxy_pool) with self.proxy_pool_lock: - self.proxy_pool = new_proxies + self.proxy_pool = merged_proxy_pool + source_map = {} + for proxy_entry in merged_proxy_pool: + if isinstance(proxy_entry, dict) and proxy_entry.get("proxy"): + source_map[self._proxy_key(proxy_entry.get("proxy"))] = str( + proxy_entry.get("source_tag") or "" + ).strip() + with self.proxy_failure_lock: + self.proxy_source_by_key = source_map self.proxy_last_refresh_time = datetime.now() - self.proxy_last_available_count = len(new_proxies) - self.proxy_last_refresh_status = f"刷新成功,可用 {len(new_proxies)} 个" - self.proxy_next_refresh_time = 0.0 if new_proxies else time.time() + self.proxy_refresh_cooldown_seconds - logger.info(f"代理池刷新完成,共 {len(new_proxies)} 个可用代理,来源链接 {len(proxy_api_urls)} 个") - if failure_reason_stats: - logger.warning( - f"代理校验失败汇总: {dict(failure_reason_stats.most_common(5))}" - ) - if failure_samples: - logger.warning(f"代理校验失败样例: {' | '.join(failure_samples)}") + self.proxy_last_available_count = len(self.proxy_pool) + if new_proxies: + if fallback_stale_count > 0: + self.proxy_last_refresh_status = ( + f"疑似过期回退增量入池 {len(new_proxies)} 个,当前池 {len(self.proxy_pool)} 个(跳过预验证)" + ) + else: + self.proxy_last_refresh_status = ( + f"增量入池 {len(new_proxies)} 个,当前池 {len(self.proxy_pool)} 个(跳过预验证)" + ) + self.proxy_next_refresh_time = 0.0 + elif cached_proxy_pool: + self.proxy_last_refresh_status = f"本轮未拿到新代理,继续沿用缓存 {len(cached_proxy_pool)} 个" + next_retry_delay = 0.5 if (self.detecting and aggressive_proxy_rotation and cooldown_detected) else (2 if cooldown_detected else retry_delay_seconds) + self.proxy_next_refresh_time = time.time() + next_retry_delay + else: + self.proxy_last_refresh_status = "代理源暂时冷却中,稍后继续补货" if cooldown_detected else "未取到可用代理数据" + next_retry_delay = 0.5 if (self.detecting and aggressive_proxy_rotation and cooldown_detected) else (2 if cooldown_detected else retry_delay_seconds) + self.proxy_next_refresh_time = time.time() + next_retry_delay + logger.info( + f"代理池刷新完成,共 {len(self.proxy_pool)} 个可用代理,来源链接 {source_count} 个," + f"原始 {self.proxy_last_refresh_total_items} 个,去过期 {expired_count} 个,非法 {invalid_count} 个," + f"目标总量 {target_total},单批 {batch_size},轮次 {rounds}" + ) self._update_runtime_state( "running" if self.detecting else "idle", - f"代理池刷新完成,共 {len(new_proxies)} 个可用代理,来源链接 {len(proxy_api_urls)} 个,原始 {self.proxy_last_refresh_total_items} 个,验证 {self.proxy_last_validated_count} 个", + f"代理池刷新完成,共 {len(self.proxy_pool)} 个可用代理,来源链接 {source_count} 个,原始 {self.proxy_last_refresh_total_items} 个,单批 {batch_size},轮次 {rounds}", ) self._sync_worker_log_event( - f"代理池刷新完成,共 {len(new_proxies)} 个可用代理,来源链接 {len(proxy_api_urls)} 个,原始 {self.proxy_last_refresh_total_items} 个,验证 {self.proxy_last_validated_count} 个", + f"代理池刷新完成,共 {len(self.proxy_pool)} 个可用代理,来源链接 {source_count} 个,原始 {self.proxy_last_refresh_total_items} 个,单批 {batch_size},轮次 {rounds}", payload={ - "available_proxy_count": len(new_proxies), - "source_count": len(proxy_api_urls), + "available_proxy_count": len(self.proxy_pool), + "source_count": source_count, "raw_items": self.proxy_last_refresh_total_items, - "validated_count": self.proxy_last_validated_count, + "validated_count": 0, + "expired_count": expired_count, + "invalid_count": invalid_count, + "fallback_stale_count": fallback_stale_count, + "target_total": target_total, + "batch_size": batch_size, + "rounds": rounds, }, mode='key', ) else: with self.proxy_pool_lock: - self.proxy_pool = [] + self.proxy_pool = cached_proxy_pool self.proxy_last_refresh_time = datetime.now() self.proxy_last_refresh_total_items = 0 self.proxy_last_validated_count = 0 - self.proxy_last_available_count = 0 - self.proxy_last_refresh_status = "未取到可用代理数据" - self.proxy_next_refresh_time = time.time() + self.proxy_refresh_cooldown_seconds + self.proxy_last_available_count = len(self.proxy_pool) + if cached_proxy_pool: + self.proxy_last_refresh_status = f"未取到新代理,继续沿用缓存 {len(cached_proxy_pool)} 个" + else: + self.proxy_last_refresh_status = "代理源暂时冷却中,稍后继续补货" if cooldown_detected else "未取到可用代理数据" + next_retry_delay = 0.5 if (self.detecting and aggressive_proxy_rotation and cooldown_detected) else (2 if cooldown_detected else retry_delay_seconds) + self.proxy_next_refresh_time = time.time() + next_retry_delay logger.warning("所有代理池链接均未返回可用代理数据") self._update_runtime_state( "refreshing_proxy" if self.detecting else "idle", - "所有代理池链接均未返回可用代理数据", + self.proxy_last_refresh_status, ) - self._sync_worker_log_event("所有代理池链接均未返回可用代理数据", level='warning', mode='key') + self._sync_worker_log_event(self.proxy_last_refresh_status, level='warning', mode='key') else: with self.proxy_pool_lock: - self.proxy_pool = [] + self.proxy_pool = cached_proxy_pool self.proxy_last_refresh_time = datetime.now() self.proxy_last_refresh_total_items = 0 self.proxy_last_validated_count = 0 - self.proxy_last_available_count = 0 + self.proxy_last_available_count = len(self.proxy_pool) self.proxy_last_refresh_status = "未配置代理池链接" - self.proxy_next_refresh_time = time.time() + self.proxy_refresh_cooldown_seconds + self.proxy_next_refresh_time = time.time() + (0.5 if (self.detecting and aggressive_proxy_rotation) else retry_delay_seconds) self._update_runtime_state( "idle" if not self.detecting else "running", "未配置代理池链接", @@ -1771,19 +3701,23 @@ class DetectWorker: self._sync_worker_log_event("未配置代理池链接", level='warning', mode='key') except Exception as e: with self.proxy_pool_lock: - self.proxy_pool = [] + cached_proxy_pool = list(self.proxy_pool) + self.proxy_pool = cached_proxy_pool self.proxy_last_refresh_time = datetime.now() self.proxy_last_refresh_total_items = 0 self.proxy_last_validated_count = 0 - self.proxy_last_available_count = 0 - self.proxy_last_refresh_status = f"刷新失败: {e}" - self.proxy_next_refresh_time = time.time() + self.proxy_refresh_cooldown_seconds + self.proxy_last_available_count = len(self.proxy_pool) + if cached_proxy_pool: + self.proxy_last_refresh_status = f"刷新失败,继续沿用缓存 {len(cached_proxy_pool)} 个: {e}" + else: + self.proxy_last_refresh_status = f"刷新失败: {e}" + self.proxy_next_refresh_time = time.time() + retry_delay_seconds logger.error(f"刷新代理池失败: {e}") self._update_runtime_state( "refreshing_proxy" if self.detecting else "failed", - f"刷新代理池失败: {e}", + self.proxy_last_refresh_status, ) - self._sync_worker_log_event(f"刷新代理池失败: {e}", level='error', mode='key') + self._sync_worker_log_event(self.proxy_last_refresh_status, level='error', mode='key') finally: self.proxy_refresh_lock.release() @@ -1796,50 +3730,54 @@ class DetectWorker: failure_count, cooldown_seconds = self._mark_proxy_failure(proxy) with self.proxy_pool_lock: remaining_count = len(self.proxy_pool) - # 检查代理池中的代理结构 + removed = 0 if self.proxy_pool and isinstance(self.proxy_pool[0], dict) and 'proxy' in self.proxy_pool[0]: - # 新的代理池结构 - for i, proxy_item in enumerate(self.proxy_pool): - if proxy_item['proxy'] == proxy: - proxy_item['usage_count'] = 0 - self.proxy_pool.append(self.proxy_pool.pop(i)) - remaining_count = len(self.proxy_pool) - logger.info(f"代理进入隔离观察,暂不硬删除: {proxy}") - break - else: - # 旧的代理池结构 - if proxy in self.proxy_pool: + filtered_pool = [] + for proxy_item in self.proxy_pool: + if proxy_item.get('proxy') == proxy: + removed += 1 + continue + filtered_pool.append(proxy_item) + if removed: + self.proxy_pool = filtered_pool remaining_count = len(self.proxy_pool) - logger.info(f"代理进入隔离观察,暂不硬删除: {proxy}") + logger.info(f"代理失效已从当前池移除 {removed} 个: {proxy}") + else: + filtered_pool = [] + for existing_proxy in self.proxy_pool: + if existing_proxy == proxy: + removed += 1 + continue + filtered_pool.append(existing_proxy) + if removed: + self.proxy_pool = filtered_pool + remaining_count = len(self.proxy_pool) + logger.info(f"代理失效已从当前池移除 {removed} 个: {proxy}") logger.info( f"代理失败计数已更新: 第 {failure_count} 次失败,隔离 {cooldown_seconds}s,当前池内代理 {remaining_count} 个" ) self._schedule_proxy_refresh_if_needed(remaining_count) - def get_proxies(self): + def get_proxies(self, excluded_proxy_keys=None): """ 获取代理配置 """ + excluded_proxy_keys = set(excluded_proxy_keys or []) if self.proxy_config.get('proxy_enable', False): with self.proxy_pool_lock: current_pool_size = len(self.proxy_pool) is_empty = current_pool_size == 0 - is_insufficient = current_pool_size < max(2, self.thread_count // 2) + is_insufficient = current_pool_size < self._proxy_refresh_threshold() - if is_empty or is_insufficient: - if is_empty: - logger.info("代理池为空,刷新代理池") - self.refresh_proxy_pool() - if is_empty: - wait_deadline = time.time() + 2.0 - while time.time() < wait_deadline: - with self.proxy_pool_lock: - if self.proxy_pool: - break - if not self.proxy_refresh_lock.locked(): - break - time.sleep(0.1) + if is_empty: + # 不要在检测线程里同步阻塞等待代理刷新,否则线程数上去了也会卡死在这里。 + self._schedule_proxy_refresh_if_needed(current_pool_size) + elif is_insufficient: + logger.debug( + f"代理池低水位,触发后台补货: 当前 {current_pool_size},阈值 {self._proxy_refresh_threshold()}" + ) + self._schedule_proxy_refresh_if_needed(current_pool_size) with self.proxy_pool_lock: if self.proxy_pool: @@ -1848,9 +3786,17 @@ class DetectWorker: if isinstance(self.proxy_pool[0], dict) and 'proxy' in self.proxy_pool[0]: proxy_item = self.proxy_pool.pop(0) proxy = proxy_item['proxy'] + proxy_key = self._proxy_key(proxy) if self._is_proxy_quarantined(proxy): self.proxy_pool.append(proxy_item) continue + source_tag = str(proxy_item.get("source_tag") or "").strip() + if self._is_proxy_source_quarantined(source_tag): + self.proxy_pool.append(proxy_item) + continue + if proxy_key in excluded_proxy_keys: + self.proxy_pool.append(proxy_item) + continue usage_count = int(proxy_item.get('usage_count', 0)) + 1 if usage_count >= self.proxy_max_reuse_count: @@ -1858,30 +3804,41 @@ class DetectWorker: else: proxy_item['usage_count'] = usage_count self.proxy_pool.append(proxy_item) - self._clear_proxy_failure(proxy) + with self.proxy_failure_lock: + source_map = getattr(self, "proxy_source_by_key", None) + if source_map is None: + self.proxy_source_by_key = {} + source_map = self.proxy_source_by_key + source_map[proxy_key] = source_tag self._schedule_proxy_refresh_if_needed(len(self.proxy_pool)) return proxy proxy = self.proxy_pool.pop(0) + proxy_key = self._proxy_key(proxy) if self._is_proxy_quarantined(proxy): self.proxy_pool.append(proxy) continue + if proxy_key in excluded_proxy_keys: + self.proxy_pool.append(proxy) + continue logger.info(f"从代理池选择代理: {proxy}") self.proxy_pool.append(proxy) - self._clear_proxy_failure(proxy) self._schedule_proxy_refresh_if_needed(len(self.proxy_pool)) return proxy if self.proxy_config.get('proxy_enable', False): - logger.warning("代理已启用,但当前无可用代理") - fallback_detail = ( - f"代理已启用,但当前无可用代理;已允许直连兜底,最近代理状态:{self.proxy_last_refresh_status}" - if self.allow_direct_connection() - else f"代理已启用,但当前无可用代理;且未允许直连,最近代理状态:{self.proxy_last_refresh_status}" - ) - self._update_runtime_state( - "running" if self.detecting else "idle", - fallback_detail, - ) + now_ts = time.time() + if now_ts - float(getattr(self, "_last_no_proxy_notice_at", 0.0) or 0.0) >= 5.0: + self._last_no_proxy_notice_at = now_ts + logger.warning("代理已启用,但当前无可用代理") + fallback_detail = ( + f"代理已启用,但当前无可用代理;已允许直连兜底,最近代理状态:{self.proxy_last_refresh_status}" + if self.allow_direct_connection() + else f"代理已启用,但当前无可用代理;且未允许直连,最近代理状态:{self.proxy_last_refresh_status}" + ) + self._update_runtime_state( + "running" if self.detecting else "idle", + fallback_detail, + ) else: logger.info("未启用代理,使用直接连接") return None @@ -1889,20 +3846,173 @@ class DetectWorker: def allow_direct_connection(self): return bool(self.proxy_config.get('allow_direct', False)) - def _get_proxy_for_step(self, domain_id, domain_name, step_name): - proxy = self.get_proxies() - if proxy: - return proxy - if self.proxy_config.get('proxy_enable', False) and not self.allow_direct_connection(): - detail = self.proxy_last_refresh_status - logger.error(f"{step_name} 无可用代理,且当前不允许直连兜底: {domain_name},最近代理状态: {detail}") + def _get_proxy_for_step(self, domain_id, domain_name, step_name, excluded_proxy_keys=None): + started_at = time.perf_counter() + excluded_proxy_keys = set(excluded_proxy_keys or []) + proxy_enabled = bool(self.proxy_config.get('proxy_enable', False)) + direct_allowed = self.allow_direct_connection() + force_direct_once = self._consume_step_force_direct_once(step_name) + if proxy_enabled and direct_allowed and force_direct_once: + self._emit_step_trace( + domain_name, + step_name, + "proxy_direct_fallback", + elapsed_ms=0, + proxy_mode="direct", + recent_proxy_status=self.proxy_last_refresh_status, + retry_attempts=0, + reason="external_retry_direct_fallback", + ) + return None + direct_retry_wait_seconds = self._consume_step_proxy_retry_wait(step_name) + wait_window_seconds = 0.0 + if proxy_enabled and not direct_allowed: + wait_window_seconds = self.proxy_step_wait_timeout_seconds + elif proxy_enabled and direct_allowed: + wait_window_seconds = max(self.proxy_direct_fallback_grace_seconds, direct_retry_wait_seconds) + wait_deadline = time.time() + wait_window_seconds + wait_logged_at = 0.0 + attempt = 0 + + while True: + proxy = self.get_proxies(excluded_proxy_keys=excluded_proxy_keys) + elapsed_ms = int((time.perf_counter() - started_at) * 1000) + if proxy: + self._emit_step_trace( + domain_name, + step_name, + "proxy_selected", + elapsed_ms=elapsed_ms, + proxy_mode="proxy", + excluded_proxy_count=len(excluded_proxy_keys), + retry_attempts=attempt, + ) + return proxy + + if not proxy_enabled: + self._emit_step_trace( + domain_name, + step_name, + "proxy_direct_fallback", + elapsed_ms=elapsed_ms, + proxy_mode="direct", + recent_proxy_status=self.proxy_last_refresh_status, + ) + return None + + now_ts = time.time() + if now_ts >= wait_deadline: + break + + attempt += 1 + if not self.proxy_refresh_lock.locked(): + self.trigger_proxy_refresh( + reason=f"step_wait:{step_name}", + reset_cooldown=bool(direct_retry_wait_seconds) or step_name == '注册状态检测', + ) + if now_ts - wait_logged_at >= 1.0: + wait_logged_at = now_ts + logger.info( + f"{step_name} 暂无可用代理,继续等待补货: {domain_name}, " + f"attempt={attempt}, waited_ms={elapsed_ms}, 最近状态: {self.proxy_last_refresh_status}" + ) + time.sleep(0.05 if direct_retry_wait_seconds > 0 else 0.1) + + if proxy_enabled and direct_allowed: + self._emit_step_trace( + domain_name, + step_name, + "proxy_direct_fallback", + elapsed_ms=int((time.perf_counter() - started_at) * 1000), + proxy_mode="direct", + recent_proxy_status=self.proxy_last_refresh_status, + retry_attempts=attempt, + ) + return None + + detail = self.proxy_last_refresh_status + elapsed_ms = int((time.perf_counter() - started_at) * 1000) + logger.error( + f"{step_name} 无可用代理,且当前不允许直连兜底: {domain_name}," + f"waited_ms={elapsed_ms}, attempts={attempt}, 最近代理状态: {detail}" + ) + self._emit_step_trace( + domain_name, + step_name, + "proxy_unavailable", + elapsed_ms=elapsed_ms, + level='warning', + proxy_mode="required", + recent_proxy_status=detail, + retry_attempts=attempt, + ) + if not self._is_current_single_step_task(): self.db.update_domain_detect_status(domain_id, DETECT_STATUS_FAILED) - return '__NO_PROXY__' - return None + return '__NO_PROXY__' + + def _should_rotate_proxy_on_error(self, proxy, reason, step_name, domain_name): + if not proxy or not self.proxy_config.get('proxy_enable', False): + return False + normalized_reason = str(reason or "") + if not self._is_external_dependency_issue(normalized_reason): + return False + if self._is_proxy_auth_issue(normalized_reason): + self._mark_proxy_source_failure(proxy, normalized_reason) + self.remove_proxy(proxy) + try: + self.trigger_proxy_refresh(reason=f"proxy_rotate:{step_name}", reset_cooldown=True) + except Exception as refresh_error: + logger.debug(f"{step_name} 触发代理快速补货失败: {refresh_error}") + logger.warning(f"{step_name} 当前代理异常,切换下一个代理继续: {domain_name}, 原因: {normalized_reason}") + return True + + def _should_retry_external_issue(self, proxy, reason, step_name, domain_name): + normalized_reason = str(reason or "") + if not self._is_external_dependency_issue(normalized_reason): + return False + if proxy and self.allow_direct_connection(): + if self._is_proxy_auth_issue(normalized_reason): + self._mark_proxy_source_failure(proxy, normalized_reason) + self.remove_proxy(proxy) + self._set_step_force_direct_once(step_name, True) + try: + self.trigger_proxy_refresh(reason=f"direct_fallback:{step_name}", reset_cooldown=True) + except Exception as refresh_error: + logger.debug(f"{step_name} 触发直连兜底前代理补货失败: {refresh_error}") + logger.warning( + f"{step_name} 代理链路异常,下一次尝试直连兜底: " + f"{domain_name}, 原因: {normalized_reason}" + ) + return True + if self._should_rotate_proxy_on_error(proxy, normalized_reason, step_name, domain_name): + return True + if self.proxy_config.get('proxy_enable', False): + if not proxy and self.allow_direct_connection(): + forced_wait_seconds = max( + self.proxy_direct_fallback_grace_seconds, + min( + self.proxy_step_wait_timeout_seconds or 0.0, + float(os.getenv("DOMAINCHECK_DIRECT_RETRY_PROXY_WAIT", "1.2") or 1.2), + ) if self.proxy_step_wait_timeout_seconds > 0 else float(os.getenv("DOMAINCHECK_DIRECT_RETRY_PROXY_WAIT", "1.2") or 1.2), + ) + self._set_step_proxy_retry_wait(step_name, forced_wait_seconds) + try: + self.trigger_proxy_refresh(reason=f"external_retry:{step_name}", reset_cooldown=True) + except Exception as refresh_error: + logger.debug(f"{step_name} 触发代理补货失败: {refresh_error}") + logger.warning( + f"{step_name} 外部链路异常,已触发代理补货并继续重试: " + f"{domain_name}, 原因: {normalized_reason}" + ) + return True + return False def _mark_blacklisted(self, domain_id, domain_name, reason): - self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED) - self.db.add_to_blacklist(domain_name, reason) + self._set_task_detect_status(DETECT_STATUS_BLACKLISTED) + if self._is_current_single_step_task(): + logger.info(f"single_step 域名命中黑名单,跳过本地域名表/黑名单表写入: {domain_name}, 原因: {reason}") + return + self.db.mark_domain_blacklisted(domain_id, domain_name, reason) logger.info(f"域名已加入黑名单: {domain_name}, 原因: {reason}") def _step_result_payload(self, *, ok, state, message="", **extra): @@ -1918,7 +4028,13 @@ class DetectWorker: def _record_step_result(self, domain_id, field_name, *, ok, state, message="", **extra): if not field_name: return - self._upsert_json_detection(domain_id, **{field_name: self._step_result_payload(ok=ok, state=state, message=message, **extra)}) + payload = self._step_result_payload(ok=ok, state=state, message=message, **extra) + if self._is_current_single_step_task(): + self._cache_single_step_result_payload(domain_id, field_name, payload) + return + with self._step_result_cache_lock: + self._step_result_cache[(int(domain_id or 0), field_name)] = dict(payload) + self._upsert_json_detection(domain_id, **{field_name: payload}) def _should_blacklist_result(self, message): if not message: @@ -1938,18 +4054,43 @@ class DetectWorker: def _mark_detection_failed(self, domain_id, domain_name, step_name, reason, field_name=None): reason = reason or "未知错误" - self.db.update_domain_detect_status(domain_id, DETECT_STATUS_FAILED) + if self._is_current_single_step_task(): + self._set_task_detect_status(DETECT_STATUS_FAILED) + if field_name: + self._record_step_result(domain_id, field_name, ok=False, state="failed", message=reason, step=step_name) + logger.warning(f"{step_name} single_step 技术失败,跳过本地域名状态写入: {domain_name}, 原因: {reason}") + return + current_status = self._get_task_detect_status() or 0 + if current_status == DETECT_STATUS_BLACKLISTED: + logger.warning(f"{step_name} 技术失败,但域名已是黑名单状态,保持黑名单不回退: {domain_name}, 原因: {reason}") + else: + self._enqueue_domain_status_update(domain_id, DETECT_STATUS_FAILED) + self._set_task_detect_status(DETECT_STATUS_FAILED) if field_name: self._record_step_result(domain_id, field_name, ok=False, state="failed", message=reason, step=step_name) logger.warning(f"{step_name} 技术失败,域名标记为检测失败: {domain_name}, 原因: {reason}") + def _mark_detection_rejected(self, domain_id, domain_name, step_name, reason, field_name=None): + reason = reason or "业务判定未通过" + self._set_task_detect_status(DETECT_STATUS_FAILED) + if field_name: + self._record_step_result(domain_id, field_name, ok=False, state="rejected", message=reason, step=step_name) + if self._is_current_single_step_task(): + logger.warning(f"{step_name} single_step 业务不通过,跳过本地域名状态写入: {domain_name}, 原因: {reason}") + return + self._enqueue_domain_status_update(domain_id, DETECT_STATUS_FAILED) + logger.warning(f"{step_name} 业务判定未通过,终止后续步骤: {domain_name}, 原因: {reason}") + def _mark_detection_degraded(self, domain_id, domain_name, step_name, reason, field_name=None, review_required=True): reason = reason or "外部依赖异常" if field_name: self._record_step_result(domain_id, field_name, ok=False, state="degraded", message=reason, step=step_name) + if self._is_current_single_step_task(): + logger.warning(f"{step_name} single_step 外部依赖异常,跳过本地 review 状态写入: {domain_name}, 原因: {reason}") + return if review_required: try: - self.db.update_domain_review_status(domain_id, REVIEW_STATUS_PENDING) + self._enqueue_review_status_update(domain_id, REVIEW_STATUS_PENDING) except Exception as review_error: logger.warning(f"{step_name} 降级后设置待人工复核失败: {domain_name}, 错误: {review_error}") logger.warning(f"{step_name} 外部依赖异常,步骤降级继续执行: {domain_name}, 原因: {reason}") @@ -1961,6 +4102,17 @@ class DetectWorker: keywords = [ "timeout", "timed out", + "超时", + "请求超时", + "连接超时", + "读取超时", + "网络超时", + "请求失败", + "网络异常", + "连接失败", + "连接中断", + "连接被重置", + "服务暂不可用", "connection", "connection reset", "connection refused", @@ -1976,22 +4128,152 @@ class DetectWorker: "ssl", "read timeout", "connecttimeout", + "operation timed out", + "connection timed out", + "resolving timed out", + "http 403", + "http 429", + "status code 403", + "status code 429", + "状态码: 403", + "状态码: 429", + "状态码: 502", + "状态码: 503", + "状态码: 504", + "forbidden", + "empty reply from server", + "recv failure", + "curl: (28)", + "curl:(28)", + "curl: (52)", + "curl:(52)", + "响应格式异常", + "cannot unpack non-iterable nonetype object", ] return any(keyword in lowered for keyword in keywords) - def _complete_detection(self, domain_id, domain_name): - self.db.update_domain_detect_status(domain_id, DETECT_STATUS_COMPLETED) - domain_info = self.db.get_domain_by_id(domain_id) - if not domain_info: - return - use_status = domain_info.get('use_status', 0) - register_status = domain_info.get('register_status', 0) - expire_date = domain_info.get('expire_date') + def _resolve_proxy_step_retry_budget(self, step_name): + overrides = (getattr(self, "proxy_step_retry_budget_overrides", None) or {}).get(step_name) or {} + max_attempts = max( + 1, + int(overrides.get("max_attempts", getattr(self, "proxy_step_retry_max_attempts", 12)) or 1), + ) + max_seconds = max( + 5.0, + float(overrides.get("max_seconds", getattr(self, "proxy_step_retry_max_seconds", 45)) or 5.0), + ) + return max_attempts, max_seconds + + def _check_proxy_step_retry_budget(self, step_name, domain_name, started_at, attempt_count, tried_proxy_keys, last_reason=""): + elapsed_seconds = max(0.0, time.perf_counter() - float(started_at or 0.0)) + max_attempts, max_seconds = self._resolve_proxy_step_retry_budget(step_name) + if attempt_count < max_attempts and elapsed_seconds < max_seconds: + return False, "" + + reason = ( + f"{step_name} 重试预算耗尽: attempts={attempt_count}, " + f"elapsed_ms={int(elapsed_seconds * 1000)}, tried_proxies={len(tried_proxy_keys)}" + ) + if last_reason: + reason = f"{reason}, last_reason={str(last_reason)[:200]}" + logger.warning(f"{reason}: {domain_name}") + self._emit_step_trace( + domain_name, + step_name, + "retry_budget_exhausted", + level='warning', + elapsed_ms=int(elapsed_seconds * 1000), + retry_attempts=attempt_count, + tried_proxy_count=len(tried_proxy_keys), + last_reason=str(last_reason or "")[:200], + ) + return True, reason + + def _remaining_proxy_step_retry_budget_seconds(self, step_name, started_at, reserve_seconds=0.0): + elapsed_seconds = max(0.0, time.perf_counter() - float(started_at or 0.0)) + _, max_seconds = self._resolve_proxy_step_retry_budget(step_name) + return max(0.0, float(max_seconds or 0.0) - elapsed_seconds - max(0.0, float(reserve_seconds or 0.0))) + + def _resolve_submit_backlog_limit(self, max_threads, live_active): + normalized_threads = max(1, int(max_threads or 1)) + live_active = max(0, int(live_active or 0)) + configured_backlog_floor = int(os.getenv("DOMAINCHECK_SUBMIT_BACKLOG_FLOOR", "0") or 0) + configured_backlog_ceiling = int(os.getenv("DOMAINCHECK_SUBMIT_BACKLOG_CEIL", "0") or 0) + # Default to a full-thread warm window. Large machines should be allowed + # to keep enough local backlog to actually fill the configured worker + # pool instead of self-throttling to a few hundred runnable items. + default_backlog_floor = normalized_threads + backlog_floor = max(16, configured_backlog_floor if configured_backlog_floor > 0 else default_backlog_floor) + backlog_ceiling = max( + backlog_floor, + configured_backlog_ceiling if configured_backlog_ceiling > 0 else normalized_threads, + ) + dynamic_target = max( + backlog_floor, + live_active + max(32, normalized_threads // 4), + ) + return min(backlog_ceiling, dynamic_target) + + def _resolve_dispatch_capacity(self, max_threads, live_active, backlog_limit): + normalized_threads = max(1, int(max_threads or 1)) + live_active = max(0, int(live_active or 0)) + backlog_limit = max(1, int(backlog_limit or 1)) + dispatch_cap_multiplier = max( + 1, + int(os.getenv("DOMAINCHECK_DISPATCH_CAP_MULTIPLIER", "1") or 1), + ) + dispatch_capacity_cap = max(1, normalized_threads * dispatch_cap_multiplier) + return min(dispatch_capacity_cap, max(backlog_limit, live_active + backlog_limit)) + + def _resolve_claim_batch_size(self, current_limit, available_slots, live_active=None): + normalized_limit = max(1, int(current_limit or 1)) + available_slots = max(1, int(available_slots or 0)) + live_active = max( + 0, + int(self._get_active_domain_threads() or 0) if live_active is None else int(live_active or 0), + ) + claim_cap_multiplier = max( + 1, + int(os.getenv("DOMAINCHECK_CLAIM_CAP_MULTIPLIER", "1") or 1), + ) + configured_claim_floor = int(os.getenv("DOMAINCHECK_CLAIM_BATCH_FLOOR", "0") or 0) + configured_claim_ceiling = int(os.getenv("DOMAINCHECK_CLAIM_BATCH_CEIL", "0") or 0) + # Default to a full-thread claim window so large worker pools can + # actually fill their slots. Operators can still shrink this via env + # overrides if they explicitly want a tighter rolling buffer. + default_claim_floor = normalized_limit + claim_floor = max(4, configured_claim_floor if configured_claim_floor > 0 else default_claim_floor) + claim_ceiling = max( + claim_floor, + configured_claim_ceiling if configured_claim_ceiling > 0 else normalized_limit, + ) + warm_window = max( + claim_floor, + live_active + max(32, normalized_limit // 4), + ) + return max( + 1, + min( + available_slots, + normalized_limit * claim_cap_multiplier, + max(claim_floor, min(claim_ceiling, warm_window)), + ), + ) + + def _complete_detection(self, domain_id, domain_name, domain): + register_status = int((domain or {}).get('register_status', 0) or 0) + use_status = int((domain or {}).get('use_status', 0) or 0) + expire_date = (domain or {}).get('expire_date') + self._enqueue_domain_completion( + domain_id, + register_status=register_status, + use_status=use_status, + expire_date=expire_date, + ) if use_status == 0 and register_status == REGISTER_STATUS_AVAILABLE and expire_date: - self.db.update_domain_expire_date(domain_id, None) logger.info(f"域名 {domain_name} 满足条件,已将expire_date置空") + domain['expire_date'] = None if register_status == REGISTER_STATUS_AVAILABLE: - self.db.update_domain_review_status(domain_id, REVIEW_STATUS_PENDING) logger.info(f"域名 {domain_name} 满足条件,已设置为待人工复核") def _should_run_jucha(self, domain): @@ -2034,54 +4316,152 @@ class DetectWorker: is_ykj = domain.get('source_type', 0) == 1 if is_ykj: logger.info(f"一口价域名跳过注册状态检测: {domain_name}") + if self._is_current_single_step_task(): + self._cache_single_step_result_payload( + domain_id, + 'register_status', + self._step_result_payload( + ok=True, + state="passed", + message="一口价域名跳过注册状态检测", + source_type=domain.get('source_type'), + skipped=True, + ), + ) return True logger.info(f"检测注册状态: {domain_name}") - proxy = self._get_proxy_for_step(domain_id, domain_name, '注册状态检测') - if proxy == '__NO_PROXY__': - return False - try: - tld = domain_name.split('.')[-1] - status, expire_date = register.check_register(domain_name, tld, proxy if proxy else None) - if status != -1: - self.db.update_domain_register_status(domain_id, status) - if expire_date: - from datetime import datetime, timedelta - try: - date_part = expire_date.split(' ')[0] - expire_date_obj = datetime.strptime(date_part, "%Y-%m-%d") - new_expire_date = expire_date_obj + timedelta(days=75) - self.db.update_domain_expire_date(domain_id, new_expire_date.strftime("%Y-%m-%d")) - except Exception as e: - logger.error(f"处理过期日期失败: {e}") - self.db.update_domain_expire_date(domain_id, expire_date) - logger.info(f"注册状态检测完成: {domain_name}, 状态: {status}, 过期日期: {expire_date}") + tried_proxy_keys = set() + started_at = time.perf_counter() + attempt_count = 0 + last_reason = "" + # 高并发下如果注册检测仍然直连优先,会先把 controller 本机的外连资源打爆, + # 导致大量线程卡在 direct request error,而不是尽快让代理承接流量。 + # 这里默认改成“代理优先,直连兜底”;只有显式打开环境变量时才允许直连优先。 + register_direct_first = ( + self.allow_direct_connection() + and str(os.getenv("DOMAINCHECK_REGISTER_DIRECT_FIRST", "0") or "0").strip().lower() + not in {"", "0", "false", "no", "off"} + ) + direct_attempt_consumed = False + while True: + budget_exhausted, budget_reason = self._check_proxy_step_retry_budget( + '注册状态检测', + domain_name, + started_at, + attempt_count, + tried_proxy_keys, + last_reason=last_reason, + ) + if budget_exhausted: + self._mark_detection_failed(domain_id, domain_name, '注册状态检测', budget_reason) + return False + + proxy = None + if register_direct_first and not direct_attempt_consumed: + direct_attempt_consumed = True + self._emit_step_trace( + domain_name, + '注册状态检测', + 'proxy_direct_fallback', + proxy_mode='direct_first', + retry_attempts=attempt_count, + ) else: - logger.warning(f"注册状态检测失败,不更新数据: {domain_name}") + proxy = self._get_proxy_for_step(domain_id, domain_name, '注册状态检测', excluded_proxy_keys=tried_proxy_keys) + if proxy == '__NO_PROXY__': + return False if proxy: - self.remove_proxy(proxy) + tried_proxy_keys.add(self._proxy_key(proxy)) + attempt_count += 1 + try: + tld = domain_name.split('.')[-1] + remaining_budget_seconds = self._remaining_proxy_step_retry_budget_seconds( + '注册状态检测', + started_at, + reserve_seconds=0.2, + ) + status, expire_date = register.check_register( + domain_name, + tld, + proxy if proxy else None, + budget_seconds=remaining_budget_seconds, + ) + if status != -1: + if proxy: + self._clear_proxy_failure(proxy) + self._clear_proxy_source_failure(proxy) + normalized_expire_date = expire_date + if expire_date: + from datetime import datetime, timedelta + try: + date_part = expire_date.split(' ')[0] + expire_date_obj = datetime.strptime(date_part, "%Y-%m-%d") + normalized_expire_date = (expire_date_obj + timedelta(days=75)).strftime("%Y-%m-%d") + except Exception as e: + logger.error(f"处理过期日期失败: {e}") + normalized_expire_date = expire_date + if self._is_current_single_step_task(): + self._cache_single_step_result_payload( + domain_id, + 'register_status', + self._step_result_payload( + ok=True, + state="passed", + message="success", + register_status=int(status), + expire_date=normalized_expire_date or "", + ), + ) + else: + self.db.update_domain_register_result(domain_id, status, normalized_expire_date) + domain['register_status'] = status + if normalized_expire_date: + domain['expire_date'] = normalized_expire_date + logger.info(f"注册状态检测完成: {domain_name}, 状态: {status}, 过期日期: {expire_date}") + return True + logger.warning(f"注册状态检测失败,不更新数据: {domain_name}") + last_reason = 'register lookup failed' + if self._should_rotate_proxy_on_error(proxy, 'register lookup failed', '注册状态检测', domain_name): + continue self._mark_detection_failed(domain_id, domain_name, '注册状态检测', '注册状态检测返回未知状态') return False - except Exception as e: - logger.error(f"注册状态检测失败: {domain_name}, 错误: {e}") - if proxy: - self.remove_proxy(proxy) - self._mark_detection_failed(domain_id, domain_name, '注册状态检测', str(e)) - return False + except Exception as e: + logger.error(f"注册状态检测失败: {domain_name}, 错误: {e}") + last_reason = str(e) + if proxy is None and self.proxy_config.get('proxy_enable', False): + logger.warning(f"注册状态检测直连失败,切换代理继续: {domain_name}, 原因: {e}") + continue + if self._should_rotate_proxy_on_error(proxy, str(e), '注册状态检测', domain_name): + continue + self._mark_detection_failed(domain_id, domain_name, '注册状态检测', str(e)) + return False return True - def _run_detect_wayback(self, domain_id, domain_name, sensitive_words): + def _run_detect_wayback(self, domain_id, domain_name, sensitive_words, step_payload=None): logger.info(f"检测时光机: {domain_name}") try: - wayback_result = self.wayback_detector.scan_snapshots(domain_name, sensitive_words=sensitive_words) - snapshot_years = wayback_result.get('snapshot_years') or [] - if snapshot_years: - self.db.update_domain_snapshot_years(domain_id, ",".join(str(year) for year in snapshot_years)) - backlink_count = int(wayback_result.get('backlink_count', 0) or 0) - self.db.execute("UPDATE domains SET backlink_count = %s WHERE id = %s", (backlink_count, domain_id)) - self._upsert_json_detection( - domain_id, - backlink_count_gt_10=bool(wayback_result.get('backlink_count_gt_10')), + normalized_step_payload = dict(step_payload or {}) if isinstance(step_payload, dict) else {} + recent_years = normalized_step_payload.get("wayback_recent_years") + stop_on_first_hit = normalized_step_payload.get("wayback_stop_on_first_hit") + wayback_result = self.wayback_detector.scan_snapshots( + domain_name, + sensitive_words=sensitive_words, + stop_on_first_hit=bool(True if stop_on_first_hit is None else stop_on_first_hit), + recent_years=int(recent_years) if str(recent_years or "").strip() else None, ) + snapshot_years = wayback_result.get('snapshot_years') or [] + is_step_task = self._is_current_single_step_task() + backlink_count = int(wayback_result.get('backlink_count', 0) or 0) + if not is_step_task: + self.db.update_domain_wayback_summary( + domain_id, + years=",".join(str(year) for year in snapshot_years) if snapshot_years else None, + backlink_count=backlink_count, + ) + self._upsert_json_detection( + domain_id, + backlink_count_gt_10=bool(wayback_result.get('backlink_count_gt_10')), + ) if wayback_result.get('has_sensitive_content'): reason = f"时光机快照命中敏感词: {wayback_result.get('matched_word', '')}".strip() self._record_step_result(domain_id, 'wayback_info', ok=False, state="blacklisted", message=reason) @@ -2109,6 +4489,8 @@ class DetectWorker: fetched_snapshot_count=int(wayback_result.get('fetched_snapshot_count', 0) or 0), failed_snapshot_count=int(wayback_result.get('failed_snapshot_count', 0) or 0), backlink_count=int(wayback_result.get('backlink_count', 0) or 0), + backlink_count_gt_10=bool(wayback_result.get('backlink_count_gt_10')), + snapshot_years=list(snapshot_years), ) logger.info( "时光机检测完成: %s, 检查快照 %s 个, 成功抓取 %s 个, 失败 %s 个, 最大友链 %s, 耗时 %ss" @@ -2129,132 +4511,200 @@ class DetectWorker: def _run_detect_chinaz(self, domain_id, domain_name, sensitive_words): logger.info(f"检测站长之家: {domain_name}") - proxy = self._get_proxy_for_step(domain_id, domain_name, '站长之家检测') - if proxy == '__NO_PROXY__': - return False - try: - success, message, seo_data = chinaz.check_title(domain_name, sensitive_words, proxy) - if not success: - logger.warning(f"站长之家检测未通过: {domain_name}, 原因: {message}") - if self._should_blacklist_result(message): - self._record_step_result(domain_id, 'chinaz_info', ok=False, state="blacklisted", message=message) - self._mark_blacklisted(domain_id, domain_name, message) - return False - if self._is_external_dependency_issue(message): - self._mark_detection_degraded(domain_id, domain_name, '站长之家检测', message or 'external dependency issue', 'chinaz_info') - return True - self._mark_detection_failed(domain_id, domain_name, '站长之家检测', message or 'failure', 'chinaz_info') - return False - self._record_step_result(domain_id, 'chinaz_info', ok=True, state="passed", message=message or "success", seo=seo_data) - logger.info(f"站长之家检测完成: {domain_name}") - except Exception as e: - logger.error(f"站长之家检测失败: {domain_name}, 错误: {e}") - if proxy: - self.remove_proxy(proxy) - if self._is_external_dependency_issue(e): - self._mark_detection_degraded(domain_id, domain_name, '站长之家检测', str(e), 'chinaz_info') + tried_proxy_keys = set() + started_at = time.perf_counter() + attempt_count = 0 + last_reason = "" + while True: + budget_exhausted, budget_reason = self._check_proxy_step_retry_budget( + '站长之家检测', + domain_name, + started_at, + attempt_count, + tried_proxy_keys, + last_reason=last_reason, + ) + if budget_exhausted: + self._mark_detection_degraded(domain_id, domain_name, '站长之家检测', budget_reason, 'chinaz_info') return True - self._mark_detection_failed(domain_id, domain_name, '站长之家检测', str(e), 'chinaz_info') - return False + proxy = self._get_proxy_for_step(domain_id, domain_name, '站长之家检测', excluded_proxy_keys=tried_proxy_keys) + if proxy == '__NO_PROXY__': + return False + if proxy: + tried_proxy_keys.add(self._proxy_key(proxy)) + attempt_count += 1 + try: + remaining_budget_seconds = self._remaining_proxy_step_retry_budget_seconds( + '站长之家检测', + started_at, + reserve_seconds=0.2, + ) + logger.info( + f"站长之家调用边界: domain={domain_name} | attempt={attempt_count} " + f"| module={getattr(chinaz, '__file__', 'unknown')} | proxy={'yes' if proxy else 'no'}" + ) + success, message, seo_data = chinaz.check_title( + domain_name, + sensitive_words, + proxy, + budget_seconds=remaining_budget_seconds, + ) + logger.info( + f"站长之家返回边界: domain={domain_name} | attempt={attempt_count} " + f"| success={success} | message={message}" + ) + if not success: + logger.warning(f"站长之家检测未通过: {domain_name}, 原因: {message}") + last_reason = str(message or "") + if self._should_blacklist_result(message): + self._record_step_result(domain_id, 'chinaz_info', ok=False, state="blacklisted", message=message) + self._mark_blacklisted(domain_id, domain_name, message) + return False + if self._should_rotate_proxy_on_error(proxy, message, '站长之家检测', domain_name): + continue + if self._is_external_dependency_issue(message): + self._mark_detection_degraded(domain_id, domain_name, '站长之家检测', message or 'external dependency issue', 'chinaz_info') + return True + self._mark_detection_rejected(domain_id, domain_name, '站长之家检测', message or 'business rejected', 'chinaz_info') + return False + self._record_step_result(domain_id, 'chinaz_info', ok=True, state="passed", message=message or "success", seo=seo_data) + logger.info(f"站长之家检测完成: {domain_name}") + return True + except Exception as e: + logger.error(f"站长之家检测失败: {domain_name}, 错误: {e}") + last_reason = str(e) + if self._should_rotate_proxy_on_error(proxy, str(e), '站长之家检测', domain_name): + continue + if self._is_external_dependency_issue(e): + self._mark_detection_degraded(domain_id, domain_name, '站长之家检测', str(e), 'chinaz_info') + return True + self._mark_detection_failed(domain_id, domain_name, '站长之家检测', str(e), 'chinaz_info') + return False return True def _run_detect_aizhan(self, domain_id, domain_name, sensitive_words): logger.info(f"检测爱站网: {domain_name}") - proxy = self._get_proxy_for_step(domain_id, domain_name, '爱站网检测') - if proxy == '__NO_PROXY__': - return False - try: - success, message = aizhan.check_aizhan(domain_name, sensitive_words, proxy) - if not success: - logger.warning(f"爱站网检测未通过: {domain_name}, 原因: {message}") - if self._should_blacklist_result(message): - self._record_step_result(domain_id, 'aizhan_info', ok=False, state="blacklisted", message=message) - self._mark_blacklisted(domain_id, domain_name, message) - return False - if self._is_external_dependency_issue(message): - self._mark_detection_degraded(domain_id, domain_name, '爱站网检测', message or 'external dependency issue', 'aizhan_info') - return True - self._mark_detection_failed(domain_id, domain_name, '爱站网检测', message or 'failure', 'aizhan_info') - return False - self._record_step_result(domain_id, 'aizhan_info', ok=True, state="passed", message=message or "success") - logger.info(f"爱站网检测完成: {domain_name}") - except Exception as e: - logger.error(f"爱站网检测失败: {domain_name}, 错误: {e}") - if proxy: - self.remove_proxy(proxy) - if self._is_external_dependency_issue(e): - self._mark_detection_degraded(domain_id, domain_name, '爱站网检测', str(e), 'aizhan_info') + tried_proxy_keys = set() + started_at = time.perf_counter() + attempt_count = 0 + last_reason = "" + while True: + budget_exhausted, budget_reason = self._check_proxy_step_retry_budget( + '爱站网检测', + domain_name, + started_at, + attempt_count, + tried_proxy_keys, + last_reason=last_reason, + ) + if budget_exhausted: + self._mark_detection_degraded(domain_id, domain_name, '爱站网检测', budget_reason, 'aizhan_info') return True - self._mark_detection_failed(domain_id, domain_name, '爱站网检测', str(e), 'aizhan_info') - return False + proxy = self._get_proxy_for_step(domain_id, domain_name, '爱站网检测', excluded_proxy_keys=tried_proxy_keys) + if proxy == '__NO_PROXY__': + return False + if proxy: + tried_proxy_keys.add(self._proxy_key(proxy)) + attempt_count += 1 + try: + remaining_budget_seconds = self._remaining_proxy_step_retry_budget_seconds( + '爱站网检测', + started_at, + reserve_seconds=0.2, + ) + success, message = aizhan.check_aizhan( + domain_name, + sensitive_words, + proxy, + budget_seconds=remaining_budget_seconds, + ) + if not success: + logger.warning(f"爱站网检测未通过: {domain_name}, 原因: {message}") + last_reason = str(message or "") + if self._should_blacklist_result(message): + self._record_step_result(domain_id, 'aizhan_info', ok=False, state="blacklisted", message=message) + self._mark_blacklisted(domain_id, domain_name, message) + return False + if self._should_rotate_proxy_on_error(proxy, message, '爱站网检测', domain_name): + continue + if self._is_external_dependency_issue(message): + self._mark_detection_degraded(domain_id, domain_name, '爱站网检测', message or 'external dependency issue', 'aizhan_info') + return True + self._mark_detection_rejected(domain_id, domain_name, '爱站网检测', message or 'business rejected', 'aizhan_info') + return False + self._record_step_result(domain_id, 'aizhan_info', ok=True, state="passed", message=message or "success") + logger.info(f"爱站网检测完成: {domain_name}") + return True + except Exception as e: + logger.error(f"爱站网检测失败: {domain_name}, 错误: {e}") + last_reason = str(e) + if self._should_rotate_proxy_on_error(proxy, str(e), '爱站网检测', domain_name): + continue + if self._is_external_dependency_issue(e): + self._mark_detection_degraded(domain_id, domain_name, '爱站网检测', str(e), 'aizhan_info') + return True + self._mark_detection_failed(domain_id, domain_name, '爱站网检测', str(e), 'aizhan_info') + return False return True def _run_detect_baidu(self, domain_id, domain_name, sensitive_words): logger.info(f"检测百度: {domain_name}") - proxy = self._get_proxy_for_step(domain_id, domain_name, '百度site检测') - if proxy == '__NO_PROXY__': - return False - try: - success, message = baidu.check_site(domain_name, sensitive_words, proxy) - if not success: - logger.warning(f"百度site检测未通过: {domain_name}, 原因: {message}") - if proxy and ('timeout' in message.lower() or 'connection' in message.lower()) and self.allow_direct_connection(): - logger.info(f"代理检测失败,尝试不使用代理重新检测: {domain_name}") - success, message = baidu.check_site(domain_name, sensitive_words, None) - if success: - logger.info(f"不使用代理检测百度成功: {domain_name}") - else: - logger.warning(f"不使用代理检测百度仍未通过: {domain_name}, 原因: {message}") - if self._should_blacklist_result(message): - self._record_step_result(domain_id, 'baidu_site', ok=False, state="blacklisted", message=message) - self._mark_blacklisted(domain_id, domain_name, message) - elif self._is_external_dependency_issue(message): - self._mark_detection_degraded(domain_id, domain_name, '百度site检测', message, 'baidu_site') - return True - else: - self._mark_detection_failed(domain_id, domain_name, '百度site检测', message, 'baidu_site') - return False - else: + tried_proxy_keys = set() + started_at = time.perf_counter() + attempt_count = 0 + last_reason = "" + while True: + budget_exhausted, budget_reason = self._check_proxy_step_retry_budget( + '百度site检测', + domain_name, + started_at, + attempt_count, + tried_proxy_keys, + last_reason=last_reason, + ) + if budget_exhausted: + self._mark_detection_degraded(domain_id, domain_name, '百度site检测', budget_reason, 'baidu_site') + return True + proxy = self._get_proxy_for_step(domain_id, domain_name, '百度site检测', excluded_proxy_keys=tried_proxy_keys) + if proxy == '__NO_PROXY__': + return False + if proxy: + tried_proxy_keys.add(self._proxy_key(proxy)) + attempt_count += 1 + try: + remaining_budget_seconds = self._remaining_proxy_step_retry_budget_seconds( + '百度site检测', + started_at, + reserve_seconds=0.2, + ) + success, message = baidu.check_site( + domain_name, + sensitive_words, + proxy, + budget_seconds=remaining_budget_seconds, + ) + if not success: + logger.warning(f"百度site检测未通过: {domain_name}, 原因: {message}") + last_reason = str(message or "") if self._should_blacklist_result(message): self._record_step_result(domain_id, 'baidu_site', ok=False, state="blacklisted", message=message) self._mark_blacklisted(domain_id, domain_name, message) - elif self._is_external_dependency_issue(message): + return False + if self._should_retry_external_issue(proxy, message, '百度site检测', domain_name): + continue + if self._is_external_dependency_issue(message): self._mark_detection_degraded(domain_id, domain_name, '百度site检测', message, 'baidu_site') return True - else: - self._mark_detection_failed(domain_id, domain_name, '百度site检测', message, 'baidu_site') + self._mark_detection_rejected(domain_id, domain_name, '百度site检测', message or 'business rejected', 'baidu_site') return False - self._record_step_result(domain_id, 'baidu_site', ok=True, state="passed", message=message or "success") - logger.info(f"百度site检测完成: {domain_name}") - except Exception as e: - logger.error(f"百度site检测失败: {domain_name}, 错误: {e}") - if proxy: - self.remove_proxy(proxy) - if self.allow_direct_connection(): - try: - logger.info(f"尝试不使用代理重新检测百度: {domain_name}") - success, message = baidu.check_site(domain_name, sensitive_words, None) - if success: - logger.info(f"不使用代理检测百度成功: {domain_name}") - else: - logger.warning(f"不使用代理检测百度仍未通过: {domain_name}, 原因: {message}") - if self._should_blacklist_result(message): - self._record_step_result(domain_id, 'baidu_site', ok=False, state="blacklisted", message=message) - self._mark_blacklisted(domain_id, domain_name, message) - elif self._is_external_dependency_issue(message): - self._mark_detection_degraded(domain_id, domain_name, '百度site检测', message, 'baidu_site') - return True - else: - self._mark_detection_failed(domain_id, domain_name, '百度site检测', message, 'baidu_site') - return False - except Exception as e2: - logger.error(f"不使用代理检测百度也失败: {domain_name}, 错误: {e2}") - if self._is_external_dependency_issue(e2): - self._mark_detection_degraded(domain_id, domain_name, '百度site检测', str(e2), 'baidu_site') - return True - self._mark_detection_failed(domain_id, domain_name, '百度site检测', str(e2), 'baidu_site') - return False - else: + self._record_step_result(domain_id, 'baidu_site', ok=True, state="passed", message=message or "success") + logger.info(f"百度site检测完成: {domain_name}") + return True + except Exception as e: + logger.error(f"百度site检测失败: {domain_name}, 错误: {e}") + last_reason = str(e) + if self._should_retry_external_issue(proxy, str(e), '百度site检测', domain_name): + continue if self._is_external_dependency_issue(e): self._mark_detection_degraded(domain_id, domain_name, '百度site检测', str(e), 'baidu_site') return True @@ -2264,31 +4714,67 @@ class DetectWorker: def _run_detect_360(self, domain_id, domain_name, sensitive_words): logger.info(f"检测360: {domain_name}") - proxy = self._get_proxy_for_step(domain_id, domain_name, '360检测') - if proxy == '__NO_PROXY__': - return False - try: - passed, message = c360.check_domain(domain_name, sensitive_words, proxy) - if not passed: - logger.warning(f"360检测未通过: {domain_name}, 原因: {message}") - if self._should_blacklist_result(message): - self._record_step_result(domain_id, 'qihu360_site', ok=False, state="blacklisted", message=message) - self._mark_blacklisted(domain_id, domain_name, message) - return False - if self._is_external_dependency_issue(message): - self._mark_detection_degraded(domain_id, domain_name, '360检测', message or 'external dependency issue', 'qihu360_site') - return True - self._mark_detection_failed(domain_id, domain_name, '360检测', message or 'failure', 'qihu360_site') - return False - self._record_step_result(domain_id, 'qihu360_site', ok=True, state="passed", message=message or "success") - logger.info(f"360检测完成: {domain_name}") - except Exception as e: - logger.error(f"360检测失败: {domain_name}, 错误: {e}") - if self._is_external_dependency_issue(e): - self._mark_detection_degraded(domain_id, domain_name, '360检测', str(e), 'qihu360_site') + tried_proxy_keys = set() + started_at = time.perf_counter() + attempt_count = 0 + last_reason = "" + while True: + budget_exhausted, budget_reason = self._check_proxy_step_retry_budget( + '360检测', + domain_name, + started_at, + attempt_count, + tried_proxy_keys, + last_reason=last_reason, + ) + if budget_exhausted: + self._mark_detection_degraded(domain_id, domain_name, '360检测', budget_reason, 'qihu360_site') return True - self._mark_detection_failed(domain_id, domain_name, '360检测', str(e), 'qihu360_site') - return False + proxy = self._get_proxy_for_step(domain_id, domain_name, '360检测', excluded_proxy_keys=tried_proxy_keys) + if proxy == '__NO_PROXY__': + return False + if proxy: + tried_proxy_keys.add(self._proxy_key(proxy)) + attempt_count += 1 + try: + remaining_budget_seconds = self._remaining_proxy_step_retry_budget_seconds( + '360检测', + started_at, + reserve_seconds=0.2, + ) + passed, message = c360.check_domain( + domain_name, + sensitive_words, + proxy, + budget_seconds=remaining_budget_seconds, + ) + if not passed: + logger.warning(f"360检测未通过: {domain_name}, 原因: {message}") + last_reason = str(message or "") + if self._should_blacklist_result(message): + self._record_step_result(domain_id, 'qihu360_site', ok=False, state="blacklisted", message=message) + self._mark_blacklisted(domain_id, domain_name, message) + return False + if self._should_rotate_proxy_on_error(proxy, message, '360检测', domain_name): + continue + if self._is_external_dependency_issue(message): + self._mark_detection_degraded(domain_id, domain_name, '360检测', message or 'external dependency issue', 'qihu360_site') + return True + self._mark_detection_rejected(domain_id, domain_name, '360检测', message or 'business rejected', 'qihu360_site') + return False + self._record_step_result(domain_id, 'qihu360_site', ok=True, state="passed", message=message or "success") + logger.info(f"360检测完成: {domain_name}") + return True + except Exception as e: + logger.error(f"360检测失败: {domain_name}, 错误: {e}") + last_reason = str(e) + if self._should_rotate_proxy_on_error(proxy, str(e), '360检测', domain_name): + continue + if self._is_external_dependency_issue(e): + self._mark_detection_degraded(domain_id, domain_name, '360检测', str(e), 'qihu360_site') + return True + self._mark_detection_failed(domain_id, domain_name, '360检测', str(e), 'qihu360_site') + return False return True def _run_detect_jucha(self, domain_id, domain, domain_name): @@ -2302,6 +4788,7 @@ class DetectWorker: if proxy == '__NO_PROXY__': return False self.jc_instance.session.proxies = proxy or {} + beian_update_payload = {} logger.info(f"检测聚查WHOIS: {domain_name}") try: success, message, whois_info = self.jc_instance.check_whois_domain(domain_name) @@ -2333,7 +4820,12 @@ class DetectWorker: beian_year = beian_time.split('-')[0] except Exception: pass - self.db.update_domain_beian_info(domain_id, company_type, website_url, has_beian_flag, beian_year) + beian_update_payload = { + "company_type": company_type, + "website_url": website_url, + "has_beian": has_beian_flag, + "beian_year": beian_year, + } logger.info(f"聚查备案检测完成: {domain_name}") except Exception as e: logger.error(f"聚查备案检测失败: {domain_name}, 错误: {e}") @@ -2363,7 +4855,13 @@ class DetectWorker: except Exception as e: logger.error(f"聚查拦截检测失败: {domain_name}, 错误: {e}") finally: - self.db.mark_jucha_detected(domain_id) + self.db.update_domain_beian_info_and_mark_jucha_detected( + domain_id, + company_type=beian_update_payload.get("company_type"), + website_url=beian_update_payload.get("website_url"), + has_beian=beian_update_payload.get("has_beian"), + beian_year=beian_update_payload.get("beian_year"), + ) return True def _run_detect_juziseo(self, domain_id, domain, domain_name, sensitive_words): @@ -2438,43 +4936,80 @@ class DetectWorker: job_item_id = job_context.get('job_item_id') job_id = job_context.get('job_id') claim_token = job_context.get('claim_token') + task_mode = str(job_context.get('task_mode') or '').strip() or 'domain_pipeline' + step_payload = job_context.get('step_payload') if isinstance(job_context.get('step_payload'), dict) else {} + requested_step_code = str(step_payload.get('step_code') or job_context.get('step_code') or '').strip() + is_step_task = bool(requested_step_code) cycle_token = str(job_context.get('cycle_token') or self.current_cycle_token or '').strip() job_code = str(job_context.get('job_code') or self.current_job_code or '').strip() + domain_started_at = time.perf_counter() + self._set_current_task_context( + domain_id=domain_id, + is_step_task=is_step_task, + task_mode=task_mode, + job_item_id=job_item_id, + ) + current_active_threads = self._change_active_domain_threads(1) + self._note_domain_started() + if self.detecting and ( + current_active_threads <= 16 + or current_active_threads % 10 == 0 + or current_active_threads >= max(1, int(getattr(self, "thread_count", 1) or 1)) + ): + self._mark_detection_phase( + "running", + f"当前实际线程数量: {current_active_threads}/{self.thread_count}", + active_threads=current_active_threads, + max_threads=self.thread_count, + ) + self._emit_step_trace( + domain_name, + "domain", + "started", + job_item_id=job_item_id, + thread_id=thread_id, + cycle_token=cycle_token, + job_code=job_code, + ) - def finalize_job_item(final_status, message=""): + def finalize_job_item(final_status, message="", result_payload=None): + self._note_domain_result() if not job_item_id or not claim_token: return try: - if final_status in {'completed', 'blacklisted'}: - self.db.complete_detect_job_item(job_item_id, claim_token, final_status) - else: - self.db.fail_detect_job_item(job_item_id, claim_token, message or final_status) - self.db.append_detect_run_event( - job_id, - job_item_id, - config.NODE_CODE, - event_type=f"domain_{final_status}", - message=message or f"{domain_name} -> {final_status}", - level='error' if final_status == 'failed' else 'info', - payload={ - "domain_id": domain_id, - "domain": domain_name, - "status": final_status, - "cycle_token": cycle_token, - "job_code": job_code, - }, + event_payload = { + "domain_id": domain_id, + "domain": domain_name, + "status": final_status, + "cycle_token": cycle_token, + "job_code": job_code, + } + self._enqueue_job_finalization( + job_item_id=job_item_id, + claim_token=claim_token, + final_status=final_status, + message=message or final_status, + result_payload=result_payload, + job_id=job_id, + node_code=config.NODE_CODE, + event_type="" if is_step_task else f"domain_{final_status}", + event_level='error' if final_status == 'failed' else 'info', + event_message=message or f"{domain_name} -> {final_status}", + event_payload=event_payload, ) except Exception as finalize_error: logger.warning(f"回写任务项状态失败: {domain_name}, 错误: {finalize_error}") def renew_job_item_lease(detail=""): - if not job_item_id or not claim_token: + if is_step_task or not job_item_id or not claim_token: return try: self.db.renew_detect_job_item_lease( job_item_id, claim_token, - lease_seconds=max(1800, self.thread_count * 600), + # 租约续期必须和线程数解耦;线程数越高不代表单任务应该锁更久, + # 否则节点重启或线程异常后会把大量 running 项锁死数天。 + lease_seconds=900, ) except Exception as renew_error: logger.warning(f"续租任务项失败: {domain_name}, 错误: {renew_error}") @@ -2487,23 +5022,26 @@ class DetectWorker: return logger.info(f"开始检测域名: {domain_name}") - self.db.update_domain_detect_status(domain_id, DETECT_STATUS_RUNNING) + if not is_step_task: + self.db.update_domain_detect_status(domain_id, DETECT_STATUS_RUNNING) + self._set_task_detect_status(DETECT_STATUS_RUNNING) if job_item_id and claim_token: - self.db.mark_detect_job_item_running(job_item_id, claim_token) + self._enqueue_running_mark(job_item_id, claim_token) renew_job_item_lease("domain_started") - self.db.append_detect_run_event( - job_id, - job_item_id, - config.NODE_CODE, - event_type='domain_started', - message=f"开始检测域名: {domain_name}", - payload={ - "domain_id": domain_id, - "domain": domain_name, - "cycle_token": cycle_token, - "job_code": job_code, - }, - ) + if not is_step_task: + self.db.append_detect_run_event( + job_id, + job_item_id, + config.NODE_CODE, + event_type='domain_started', + message=f"开始检测域名: {domain_name}", + payload={ + "domain_id": domain_id, + "domain": domain_name, + "cycle_token": cycle_token, + "job_code": job_code, + }, + ) self._sync_worker_log_event( f"开始检测域名: {domain_name}", payload={ @@ -2516,53 +5054,278 @@ class DetectWorker: # 使用共享的敏感词列表 sensitive_words = self.sensitive_words - logger.debug(f"线程 {thread_id} 使用共享敏感词,共 {len(sensitive_words)} 个敏感词") - for detect_key in self._get_detect_execution_order(): - renew_job_item_lease(detect_key) - if detect_key == 'detect_register': - if not self._run_detect_register(domain_id, domain, domain_name): - domain_snapshot = self.db.get_domain_by_id(domain_id) or {} - finalize_job_item('blacklisted' if int(domain_snapshot.get('detect_status', 0) or 0) == DETECT_STATUS_BLACKLISTED else 'failed', '注册状态检测未通过') - return - elif detect_key == 'detect_baidu_site': - if not self._run_detect_baidu(domain_id, domain_name, sensitive_words): - domain_snapshot = self.db.get_domain_by_id(domain_id) or {} - finalize_job_item('blacklisted' if int(domain_snapshot.get('detect_status', 0) or 0) == DETECT_STATUS_BLACKLISTED else 'failed', '百度检测未通过') - return - elif detect_key == 'detect_360_site': - if not self._run_detect_360(domain_id, domain_name, sensitive_words): - domain_snapshot = self.db.get_domain_by_id(domain_id) or {} - finalize_job_item('blacklisted' if int(domain_snapshot.get('detect_status', 0) or 0) == DETECT_STATUS_BLACKLISTED else 'failed', '360检测未通过') - return - elif detect_key == 'detect_chinaz': - if not self._run_detect_chinaz(domain_id, domain_name, sensitive_words): - domain_snapshot = self.db.get_domain_by_id(domain_id) or {} - finalize_job_item('blacklisted' if int(domain_snapshot.get('detect_status', 0) or 0) == DETECT_STATUS_BLACKLISTED else 'failed', '站长之家检测未通过') - return - elif detect_key == 'detect_aizhan': - if not self._run_detect_aizhan(domain_id, domain_name, sensitive_words): - domain_snapshot = self.db.get_domain_by_id(domain_id) or {} - finalize_job_item('blacklisted' if int(domain_snapshot.get('detect_status', 0) or 0) == DETECT_STATUS_BLACKLISTED else 'failed', '爱站检测未通过') - return - elif detect_key == 'detect_wayback': - if not self._run_detect_wayback(domain_id, domain_name, sensitive_words): - domain_snapshot = self.db.get_domain_by_id(domain_id) or {} - finalize_job_item('blacklisted' if int(domain_snapshot.get('detect_status', 0) or 0) == DETECT_STATUS_BLACKLISTED else 'failed', '时光机检测未通过') - return - elif detect_key == 'detect_jucha': - if not self._run_detect_jucha(domain_id, domain, domain_name): - domain_snapshot = self.db.get_domain_by_id(domain_id) or {} - finalize_job_item('blacklisted' if int(domain_snapshot.get('detect_status', 0) or 0) == DETECT_STATUS_BLACKLISTED else 'failed', '聚查检测未通过') - return - elif detect_key == 'detect_juziseo': - if not self._run_detect_juziseo(domain_id, domain, domain_name, sensitive_words): - domain_snapshot = self.db.get_domain_by_id(domain_id) or {} - finalize_job_item('blacklisted' if int(domain_snapshot.get('detect_status', 0) or 0) == DETECT_STATUS_BLACKLISTED else 'failed', '桔子检测未通过') - return + execution_order = self._get_detect_execution_order() - self._complete_detection(domain_id, domain_name) + parallel_free_steps_enabled = bool( + int(os.getenv("DOMAINCHECK_PARALLEL_FREE_STEPS", "1") or 1) + ) + parallel_free_step_keys = ( + 'detect_baidu_site', + 'detect_360_site', + 'detect_chinaz', + 'detect_aizhan', + ) + parallel_free_step_set = set(parallel_free_step_keys) + parallel_free_step_workers = max( + 1, + int(os.getenv("DOMAINCHECK_PARALLEL_FREE_STEP_WORKERS", "1") or 1), + ) + + def execute_detect_step(detect_key): + step_name = self._step_display_name(detect_key) + renew_job_item_lease(detect_key) + step_started_at = time.perf_counter() + self._emit_step_trace( + domain_name, + step_name, + "started", + job_item_id=job_item_id, + detect_key=detect_key, + ) + step_ok = True + if detect_key == 'detect_register': + step_ok = self._run_detect_register(domain_id, domain, domain_name) + elif detect_key == 'detect_baidu_site': + step_ok = self._run_detect_baidu(domain_id, domain_name, sensitive_words) + elif detect_key == 'detect_360_site': + step_ok = self._run_detect_360(domain_id, domain_name, sensitive_words) + elif detect_key == 'detect_chinaz': + step_ok = self._run_detect_chinaz(domain_id, domain_name, sensitive_words) + elif detect_key == 'detect_aizhan': + step_ok = self._run_detect_aizhan(domain_id, domain_name, sensitive_words) + elif detect_key == 'detect_wayback': + step_ok = self._run_detect_wayback( + domain_id, + domain_name, + sensitive_words, + step_payload=step_payload, + ) + elif detect_key == 'detect_jucha': + step_ok = self._run_detect_jucha(domain_id, domain, domain_name) + elif detect_key == 'detect_juziseo': + step_ok = self._run_detect_juziseo(domain_id, domain, domain_name, sensitive_words) + else: + logger.error(f"不支持的检测步骤: {detect_key}") + step_ok = False + + step_elapsed_ms = int((time.perf_counter() - step_started_at) * 1000) + self._emit_step_trace( + domain_name, + step_name, + "finished", + job_item_id=job_item_id, + detect_key=detect_key, + elapsed_ms=step_elapsed_ms, + ok=step_ok, + ) + return { + "detect_key": detect_key, + "step_name": step_name, + "step_ok": bool(step_ok), + "step_elapsed_ms": step_elapsed_ms, + } + + if is_step_task: + self._emit_step_trace( + domain_name, + "domain", + "plan_ready", + job_item_id=job_item_id, + step_count=1, + detect_order=requested_step_code, + sensitive_word_count=len(sensitive_words), + task_mode=task_mode, + ) + step_result = execute_detect_step(requested_step_code) + result_payload = self._build_single_step_job_result( + domain_id=domain_id, + detect_key=requested_step_code, + step_ok=step_result["step_ok"], + step_name=step_result["step_name"], + ) + final_status, final_message = self._resolve_single_step_finalization(result_payload) + self._emit_step_trace( + domain_name, + step_result["step_name"], + "single_step_finalized", + job_item_id=job_item_id, + detect_key=requested_step_code, + elapsed_ms=step_result["step_elapsed_ms"], + ok=final_status in {"completed", "blacklisted"}, + final_status=final_status, + result_state=result_payload.get("state"), + ) + finalize_job_item(final_status, final_message, result_payload=result_payload) + return + + self._emit_step_trace( + domain_name, + "domain", + "plan_ready", + job_item_id=job_item_id, + step_count=len(execution_order), + detect_order=",".join(execution_order), + sensitive_word_count=len(sensitive_words), + task_mode=task_mode, + ) + + step_index = 0 + while step_index < len(execution_order): + detect_key = execution_order[step_index] + if parallel_free_steps_enabled and detect_key in parallel_free_step_set: + parallel_group = [] + while step_index < len(execution_order) and execution_order[step_index] in parallel_free_step_set: + parallel_group.append(execution_order[step_index]) + step_index += 1 + if len(parallel_group) > 1 and parallel_free_step_workers > 1: + parallel_message = ( + f"域名步骤并行组启动: domain={domain_name} | " + f"steps={','.join(parallel_group)} | workers={min(parallel_free_step_workers, len(parallel_group))}" + ) + logger.info(parallel_message) + self._sync_worker_log_event( + parallel_message, + payload={ + "domain_id": domain_id, + "domain": domain_name, + "steps": parallel_group, + "workers": min(parallel_free_step_workers, len(parallel_group)), + }, + mode='full', + job_item_id=job_item_id, + ) + parallel_results = {} + with ThreadPoolExecutor(max_workers=min(parallel_free_step_workers, len(parallel_group))) as executor: + future_map = {executor.submit(execute_detect_step, key): key for key in parallel_group} + for future in as_completed(future_map): + key = future_map[future] + parallel_results[key] = future.result() + for ordered_key in parallel_group: + step_result = parallel_results.get(ordered_key) or { + "detect_key": ordered_key, + "step_name": self._step_display_name(ordered_key), + "step_ok": False, + "step_elapsed_ms": 0, + } + if not step_result["step_ok"]: + final_status = self._resolve_current_task_final_status() + self._emit_step_trace( + domain_name, + step_result["step_name"], + "aborted", + job_item_id=job_item_id, + detect_key=ordered_key, + elapsed_ms=step_result["step_elapsed_ms"], + ok=False, + final_status=final_status, + ) + failure_reason_map = { + 'detect_register': '注册状态检测未通过', + 'detect_baidu_site': '百度检测未通过', + 'detect_360_site': '360检测未通过', + 'detect_chinaz': '站长之家检测未通过', + 'detect_aizhan': '爱站检测未通过', + 'detect_wayback': '时光机检测未通过', + 'detect_jucha': '聚查检测未通过', + 'detect_juziseo': '桔子检测未通过', + } + finalize_job_item(final_status, failure_reason_map.get(ordered_key, f'{step_result["step_name"]} 未通过')) + return + continue + if len(parallel_group) > 1: + serial_message = ( + f"域名步骤串行组推进: domain={domain_name} | " + f"steps={','.join(parallel_group)} | workers=1" + ) + logger.info(serial_message) + self._sync_worker_log_event( + serial_message, + payload={ + "domain_id": domain_id, + "domain": domain_name, + "steps": parallel_group, + "workers": 1, + }, + mode='full', + job_item_id=job_item_id, + ) + for ordered_key in parallel_group: + step_result = execute_detect_step(ordered_key) + if not step_result["step_ok"]: + final_status = self._resolve_current_task_final_status() + self._emit_step_trace( + domain_name, + step_result["step_name"], + "aborted", + job_item_id=job_item_id, + detect_key=ordered_key, + elapsed_ms=step_result["step_elapsed_ms"], + ok=False, + final_status=final_status, + ) + failure_reason_map = { + 'detect_register': '注册状态检测未通过', + 'detect_baidu_site': '百度检测未通过', + 'detect_360_site': '360检测未通过', + 'detect_chinaz': '站长之家检测未通过', + 'detect_aizhan': '爱站检测未通过', + 'detect_wayback': '时光机检测未通过', + 'detect_jucha': '聚查检测未通过', + 'detect_juziseo': '桔子检测未通过', + } + finalize_job_item(final_status, failure_reason_map.get(ordered_key, f'{step_result["step_name"]} 未通过')) + return + continue + + step_index += 1 + step_result = execute_detect_step(detect_key) + if not step_result["step_ok"]: + final_status = self._resolve_current_task_final_status() + self._emit_step_trace( + domain_name, + step_result["step_name"], + "aborted", + job_item_id=job_item_id, + detect_key=detect_key, + elapsed_ms=step_result["step_elapsed_ms"], + ok=False, + final_status=final_status, + ) + failure_reason_map = { + 'detect_register': '注册状态检测未通过', + 'detect_baidu_site': '百度检测未通过', + 'detect_360_site': '360检测未通过', + 'detect_chinaz': '站长之家检测未通过', + 'detect_aizhan': '爱站检测未通过', + 'detect_wayback': '时光机检测未通过', + 'detect_jucha': '聚查检测未通过', + 'detect_juziseo': '桔子检测未通过', + } + finalize_job_item(final_status, failure_reason_map.get(detect_key, f'{step_result["step_name"]} 未通过')) + return + + completion_tail_started_at = time.perf_counter() + complete_detection_started_at = time.perf_counter() + self._complete_detection(domain_id, domain_name, domain) + complete_detection_elapsed_ms = int((time.perf_counter() - complete_detection_started_at) * 1000) + self._set_task_detect_status(DETECT_STATUS_COMPLETED) + finalize_enqueue_started_at = time.perf_counter() finalize_job_item('completed', f"域名检测完成: {domain_name}") + finalize_enqueue_elapsed_ms = int((time.perf_counter() - finalize_enqueue_started_at) * 1000) + trace_emit_started_at = time.perf_counter() + if str(os.getenv("DOMAINCHECK_DOMAIN_COMPLETED_TRACE", "0") or "0").strip().lower() not in {"", "0", "false", "no", "off"}: + self._emit_step_trace( + domain_name, + "domain", + "completed", + job_item_id=job_item_id, + elapsed_ms=int((time.perf_counter() - domain_started_at) * 1000), + ok=True, + sync_mode='off', + ) + trace_emit_elapsed_ms = int((time.perf_counter() - trace_emit_started_at) * 1000) logger.info(f"域名检测完成: {domain_name}") + completion_log_sync_started_at = time.perf_counter() self._sync_worker_log_event( f"域名检测完成: {domain_name}", payload={ @@ -2573,11 +5336,37 @@ class DetectWorker: mode='full', job_item_id=job_item_id, ) + completion_log_sync_elapsed_ms = int((time.perf_counter() - completion_log_sync_started_at) * 1000) + completion_tail_elapsed_ms = int((time.perf_counter() - completion_tail_started_at) * 1000) + if ( + completion_tail_elapsed_ms >= 20 + or complete_detection_elapsed_ms >= 10 + or trace_emit_elapsed_ms >= 10 + or completion_log_sync_elapsed_ms >= 10 + ): + logger.info( + f"域名完成尾部耗时: domain={domain_name} | total_ms={completion_tail_elapsed_ms} " + f"| complete_detection_ms={complete_detection_elapsed_ms} " + f"| finalize_enqueue_ms={finalize_enqueue_elapsed_ms} " + f"| trace_emit_ms={trace_emit_elapsed_ms} " + f"| completion_log_enqueue_ms={completion_log_sync_elapsed_ms}" + ) except Exception as e: logger.error(f"检测域名出错: {domain_name}, 错误: {e}") - self.db.update_domain_detect_status(domain_id, DETECT_STATUS_FAILED) + if not is_step_task: + self.db.update_domain_detect_status(domain_id, DETECT_STATUS_FAILED) + self._set_task_detect_status(DETECT_STATUS_FAILED) finalize_job_item('failed', str(e)) + self._emit_step_trace( + domain_name, + "domain", + "failed", + job_item_id=job_item_id, + elapsed_ms=int((time.perf_counter() - domain_started_at) * 1000), + ok=False, + error=str(e)[:300], + ) self._sync_worker_log_event( f"检测域名出错: {domain_name}, 错误: {e}", level='error', @@ -2589,6 +5378,16 @@ class DetectWorker: mode='full', job_item_id=job_item_id, ) + finally: + self._clear_current_task_context() + remaining_active_threads = self._change_active_domain_threads(-1) + if self.detecting: + self._mark_detection_phase( + "running", + f"当前实际线程数量: {remaining_active_threads}/{self.thread_count}", + active_threads=remaining_active_threads, + max_threads=self.thread_count, + ) def start_detection(self): """ @@ -2597,6 +5396,9 @@ class DetectWorker: logger.info("开始执行域名检测任务") self.detecting = True self.stop_requested = False + total_processed = 0 + final_phase = "idle" + final_detail = "Worker 等待下一次启动" self._mark_detection_phase("preparing", "开始执行域名检测任务,正在加载配置") self._sync_worker_log_event("开始执行域名检测任务,正在加载配置", mode='key') @@ -2605,6 +5407,7 @@ class DetectWorker: self.detect_options = self.load_detect_options() self.proxy_config = self.load_proxy_config() self.thread_count = self.load_thread_count() + self._last_thread_count_refresh_at = time.time() self.proxy_max_reuse_count = max(8, self.thread_count * 4) logger.info(f"检测线程数设置为: {self.thread_count}") self.load_cookies_from_remote() @@ -2621,6 +5424,8 @@ class DetectWorker: logger.error(traceback.format_exc()) self._mark_detection_phase("failed", f"加载配置失败: {e}") self._sync_worker_log_event(f"加载配置失败: {e}", level='error', mode='key') + final_phase = "failed" + final_detail = f"加载配置失败: {e}" return # 重新加载cookies @@ -2640,7 +5445,14 @@ class DetectWorker: logger.info("开始检测,刷新代理池") self._mark_detection_phase("refreshing_proxy", "开始检测,正在刷新代理池") self._sync_worker_log_event("开始检测,正在刷新代理池", mode='key') - self.refresh_proxy_pool() + if self.proxy_refresh_lock.locked(): + if self._wait_for_proxy_refresh_settle(timeout_seconds=2.0): + logger.info("检测启动前等待到了可用代理,直接复用当前代理池") + else: + logger.info("检测启动前代理池仍未就绪,继续主动刷新一次") + self.refresh_proxy_pool() + else: + self.refresh_proxy_pool() # 更新JC和Juziseo实例的代理设置 if self.detect_options.get('detect_jucha') and self.jc_instance is not None: @@ -2650,7 +5462,7 @@ class DetectWorker: # 重新加载敏感词,确保获取最新的敏感词列表 try: - self.sensitive_words = self.db.get_all_sensitive_words() + self.sensitive_words = self._load_sensitive_words_runtime() logger.info(f"重新加载敏感词完成,共 {len(self.sensitive_words)} 个敏感词") except Exception as e: logger.error(f"重新加载敏感词失败: {e}") @@ -2662,240 +5474,419 @@ class DetectWorker: self.update_config_labels() try: - batch_size = 1000 - total_processed = 0 - claim_batch_size = min(batch_size, max(50, self.thread_count * 8)) - - while self.running: - if self.stop_requested: - logger.info("检测任务收到停止请求,停止继续领取任务") - self._mark_detection_phase("stopping", "检测任务收到停止请求,准备安全退出") - self._sync_worker_log_event("检测任务收到停止请求,准备安全退出", level='warning', mode='key') - break - recycled_job_items = self.db.recycle_expired_detect_job_items() - if recycled_job_items: - logger.warning(f"已回收 {recycled_job_items} 个租约过期的任务项,重新回到 pending") - # 获取需要检测的域名 - domains = self.db.claim_detect_job_items( - config.NODE_CODE, - limit=claim_batch_size, - lease_seconds=max(1800, self.thread_count * 600), + runtime_mode_summary = { + "qt_platform": str(os.environ.get("QT_QPA_PLATFORM", "") or "").strip() or "desktop", + "worker_mode": str(os.environ.get("WORKER_MODE", "") or "").strip() or "default", + "redis_control": bool(self.use_redis), + "log_sync_mode": self._worker_log_sync_mode(), + "step_trace_enabled": self._worker_step_trace_enabled(), + } + runtime_mode_message = ( + "检测运行模式摘要: " + f"core=detect_worker.py | qt_platform={runtime_mode_summary['qt_platform']} | " + f"worker_mode={runtime_mode_summary['worker_mode']} | redis_control={1 if runtime_mode_summary['redis_control'] else 0} | " + f"log_sync_mode={runtime_mode_summary['log_sync_mode']} | " + f"step_trace_enabled={1 if runtime_mode_summary['step_trace_enabled'] else 0}" + ) + logger.info(runtime_mode_message) + self._sync_worker_log_event(runtime_mode_message, payload=runtime_mode_summary, mode='full') + pending_domains = collections.deque() + inflight_futures = set() + legacy_tail_reached = False + last_thread_progress_emit_count = 0 + last_thread_progress_emit_at = 0.0 + last_job_status_refresh_at = 0.0 + last_expired_job_recycle_at = 0.0 + last_future_fallback_scan_at = 0.0 + worker_pool_size = max(1, int(self.refresh_thread_count_runtime(force=True, min_interval=0.0) or 1)) + worker_pool = ThreadPoolExecutor( + max_workers=worker_pool_size, + thread_name_prefix="detect-domain", + ) + last_worker_pool_size = worker_pool_size + + def prune_finished_futures(): + nonlocal inflight_futures, last_future_fallback_scan_at + if not inflight_futures: + return 0 + finished = [] + with self._completed_future_lock: + while self._completed_futures and len(finished) < 4096: + finished.append(self._completed_futures.popleft()) + if not finished: + now_ts = time.time() + if now_ts - last_future_fallback_scan_at < 0.5: + return 0 + last_future_fallback_scan_at = now_ts + finished = [future for future in list(inflight_futures) if future.done()] + if not finished: + return 0 + pruned_count = 0 + for future in dict.fromkeys(finished): + if future not in inflight_futures: + continue + inflight_futures.discard(future) + pruned_count += 1 + try: + future.result() + except Exception as future_error: + logger.error(f"检测任务 future 执行异常: {future_error}") + import traceback + logger.error(traceback.format_exc()) + return pruned_count + + def emit_thread_progress(max_threads, *, batch_size=0): + nonlocal last_thread_progress_emit_count, last_thread_progress_emit_at + current_active = max(len(inflight_futures), self._get_active_domain_threads()) + should_emit_progress, emit_now = self._should_emit_thread_progress( + current_active, + max_threads, + last_thread_progress_emit_count, + last_thread_progress_emit_at, ) - using_job_queue = bool(domains) - if not domains: - domains = self.db.get_domains_to_detect(limit=batch_size, detect_options=self.detect_options) + if should_emit_progress: + logger.info(f"当前实际线程数量: {current_active}/{max_threads}") + self._mark_detection_phase( + "running", + f"当前实际线程数量: {current_active}/{max_threads}", + active_threads=current_active, + max_threads=max_threads, + batch_size=batch_size, + processed=total_processed, + pending_buffer=len(pending_domains), + ) + last_thread_progress_emit_count = current_active + last_thread_progress_emit_at = emit_now + if self.detect_thread: + try: + self.detect_thread.thread_count_signal.emit(current_active, max_threads) + except Exception as e: + logger.error(f"发送线程数量更新信号失败: {e}") + import traceback + logger.error(traceback.format_exc()) + + def inflight_submission_metrics(max_threads): + live_active = max(0, int(self._get_active_domain_threads() or 0)) + submitted_total = len(inflight_futures) + queued_backlog = max(0, submitted_total - live_active) + backlog_limit = self._resolve_submit_backlog_limit(max_threads, live_active) + dispatch_capacity = self._resolve_dispatch_capacity(max_threads, live_active, backlog_limit) + effective_available = max(0, dispatch_capacity - submitted_total) + return live_active, submitted_total, queued_backlog, backlog_limit, dispatch_capacity, effective_available + + def sync_worker_pool_size(desired_threads): + nonlocal last_worker_pool_size + normalized_threads = max(1, int(desired_threads or 1)) + current_pool_size = max(1, int(getattr(worker_pool, "_max_workers", normalized_threads) or normalized_threads)) + if current_pool_size != normalized_threads: + worker_pool._max_workers = normalized_threads + logger.info(f"动态调整 worker_pool 大小: {current_pool_size} -> {normalized_threads}") + self._sync_worker_log_event( + f"动态调整 worker_pool 大小: {current_pool_size} -> {normalized_threads}", + payload={ + "previous_max_workers": current_pool_size, + "current_max_workers": normalized_threads, + }, + mode='full', + ) + last_worker_pool_size = normalized_threads + elif last_worker_pool_size != normalized_threads: + last_worker_pool_size = normalized_threads + return normalized_threads + + def append_pending_domains(domains, using_job_queue, queue_label, claim_elapsed_ms, legacy_fetch_elapsed_ms, thread_limit, claim_batch_size, claim_lease_seconds): current_batch_size = len(domains) - queue_label = "任务队列" if using_job_queue else "兼容旧链路" - logger.info(f"从{queue_label}获取到 {current_batch_size} 个需要检测的域名") + if current_batch_size <= 0: + return + for domain in domains: + pending_domains.append( + { + "domain": domain, + "using_job_queue": bool(using_job_queue), + } + ) + fetch_trace_payload = { + "queue_label": queue_label, + "batch_size": current_batch_size, + "claim_elapsed_ms": claim_elapsed_ms, + "legacy_fetch_elapsed_ms": legacy_fetch_elapsed_ms, + "thread_limit": thread_limit, + "claim_batch_size": claim_batch_size, + "claim_lease_seconds": claim_lease_seconds, + "pending_buffer": len(pending_domains), + } + fetch_trace_message = ( + f"批次领取跟踪: queue={queue_label} | batch_size={current_batch_size} | " + f"claim_elapsed_ms={claim_elapsed_ms} | legacy_fetch_elapsed_ms={legacy_fetch_elapsed_ms} | " + f"thread_limit={thread_limit} | claim_batch_size={claim_batch_size} | " + f"claim_lease_seconds={claim_lease_seconds} | pending_buffer={len(pending_domains)}" + ) + logger.info(fetch_trace_message) + self._sync_worker_log_event(fetch_trace_message, payload=fetch_trace_payload, mode='full') self._mark_detection_phase( "fetching", f"从{queue_label}获取到 {current_batch_size} 个需要检测的域名", batch_size=current_batch_size, queue_source="detect_job_items" if using_job_queue else "domains", + pending_buffer=len(pending_domains), ) self._sync_worker_log_event( f"从{queue_label}获取到 {current_batch_size} 个需要检测的域名", payload={ "batch_size": current_batch_size, "queue_source": "detect_job_items" if using_job_queue else "domains", + "pending_buffer": len(pending_domains), }, mode='full', ) - - if not domains: - logger.info("没有需要检测的域名") - self._mark_detection_phase("idle", "当前没有需要检测的域名,Worker 等待下一次启动") - self._sync_worker_log_event("当前没有需要检测的域名,Worker 等待下一次启动", mode='key') - break - - # 旧链路直接扫 domains 表时,少于 batch_size 代表已接近尾批; - # 任务队列模式会按小批量持续领取,不能把“小批量正常领取”误判为整轮结束。 - should_stop = (not using_job_queue) and current_batch_size < batch_size - - # 创建线程池,限制同时运行的线程数量 - active_threads = [] - max_threads = self.thread_count - + + def compute_claim_batch_size(current_limit, available_slots): + return self._resolve_claim_batch_size(current_limit, available_slots) + + last_dispatchable_step_items_check_at = 0.0 + dispatchable_step_items_cached = False + legacy_domain_fallback_enabled = bool( + int(os.getenv("DOMAINCHECK_ENABLE_LEGACY_DOMAIN_FALLBACK", "0") or 0) + ) + last_legacy_fallback_skip_log_at = 0.0 + + def has_dispatchable_step_items(): + nonlocal last_dispatchable_step_items_check_at, dispatchable_step_items_cached + now_ts = time.time() + if now_ts - float(last_dispatchable_step_items_check_at or 0.0) < 1.0: + return dispatchable_step_items_cached try: - logger.info(f"开始创建线程,当前批次域名数: {current_batch_size},最大线程数: {max_threads}") - self._mark_detection_phase( - "creating_threads", - f"开始创建线程,当前批次域名数: {current_batch_size},最大线程数: {max_threads}", - batch_size=current_batch_size, - max_threads=max_threads, - ) - self._sync_worker_log_event( - f"开始创建线程,当前批次域名数: {current_batch_size},最大线程数: {max_threads}", - payload={"batch_size": current_batch_size, "max_threads": max_threads}, - mode='full', - ) - for i, domain in enumerate(domains): - # 检查是否需要停止 - if not self.running or self.stop_requested: - logger.info("检测已停止,停止创建新线程") - self._mark_detection_phase("stopping", "检测已停止,停止创建新线程") - break - - # 等待线程数量降到最大值以下 - while len([t for t in active_threads if t.is_alive()]) >= max_threads: - # 清理已完成的线程 - active_threads = [t for t in active_threads if t.is_alive()] - # 短暂休眠,避免CPU占用过高 - import time - time.sleep(0.1) - - domain_id = domain['id'] - domain_name = domain['domain'] - - # 更新进度 - if self.detect_thread: - try: - # 通过信号发送进度更新 - self.detect_thread.progress_signal.emit(total_processed + i, total_processed + current_batch_size) - except Exception as e: - logger.error(f"发送进度更新信号失败: {e}") - import traceback - logger.error(traceback.format_exc()) - - # 创建线程 + dispatchable_step_items_cached = bool(self.db.has_dispatchable_detect_job_items()) + except Exception as e: + logger.debug(f"检查标准步骤待派发任务失败: {e}") + last_dispatchable_step_items_check_at = now_ts + return dispatchable_step_items_cached + + def start_domain_thread(domain, using_job_queue, max_threads): + nonlocal total_processed + domain_id = domain['id'] + try: + job_context = { + 'job_item_id': domain.get('job_item_id'), + 'job_id': domain.get('job_id'), + 'claim_token': domain.get('claim_token'), + 'task_mode': domain.get('task_mode'), + 'step_code': domain.get('step_code'), + 'step_payload': domain.get('step_payload'), + 'cycle_token': self.current_cycle_token, + 'job_code': domain.get('job_code') or self.current_job_code, + } if using_job_queue else None + future = worker_pool.submit(self.detect_domain, domain_id, domain, job_context) + future.add_done_callback(self._enqueue_completed_future) + inflight_futures.add(future) + total_processed += 1 + if self.detect_thread: try: - job_context = { - 'job_item_id': domain.get('job_item_id'), - 'job_id': domain.get('job_id'), - 'claim_token': domain.get('claim_token'), - 'cycle_token': self.current_cycle_token, - 'job_code': self.current_job_code, - } if using_job_queue else None - thread = threading.Thread(target=self.detect_domain, args=(domain_id, domain, job_context)) - active_threads.append(thread) - logger.debug(f"创建线程 {i+1} 成功,域名: {domain_name}") + self.detect_thread.progress_signal.emit(total_processed, total_processed + len(pending_domains) + len(inflight_futures)) except Exception as e: - logger.error(f"创建线程失败: {e}") + logger.error(f"发送进度更新信号失败: {e}") import traceback logger.error(traceback.format_exc()) - continue - - # 启动线程 - try: - thread.start() - logger.debug(f"启动线程 {i+1} 成功,域名: {domain_name}") - except Exception as e: - logger.error(f"启动线程失败: {e}") - import traceback - logger.error(traceback.format_exc()) - active_threads.remove(thread) - continue - - # 显示当前实际线程数量 - current_active = len([t for t in active_threads if t.is_alive()]) - logger.info(f"当前实际线程数量: {current_active}/{max_threads}") - self._mark_detection_phase( - "running", - f"当前实际线程数量: {current_active}/{max_threads}", - active_threads=current_active, - max_threads=max_threads, - batch_size=current_batch_size, - ) - self._sync_worker_log_event( - f"当前实际线程数量: {current_active}/{max_threads}", - payload={ - "active_threads": current_active, - "max_threads": max_threads, - "batch_size": current_batch_size, - }, - mode='full', - ) - - # 更新GUI显示 - if self.detect_thread: - try: - # 通过信号发送线程数量更新 - self.detect_thread.thread_count_signal.emit(current_active, max_threads) - except Exception as e: - logger.error(f"发送线程数量更新信号失败: {e}") - import traceback - logger.error(traceback.format_exc()) + emit_thread_progress(max_threads, batch_size=len(pending_domains) + len(inflight_futures)) + return True except Exception as e: logger.error(f"创建或启动线程失败: {e}") import traceback logger.error(traceback.format_exc()) - - # 等待所有线程完成 - logger.info(f"等待剩余 {len(active_threads)} 个线程完成") - for i, thread in enumerate(active_threads): - # 检查是否需要停止 - if not self.running or self.stop_requested: - logger.info("检测已停止,停止等待线程完成") - self._mark_detection_phase("stopping", "检测已停止,停止等待线程完成") + return False + + try: + while self.running: + if self.stop_requested: + logger.info("检测任务收到停止请求,停止继续领取任务") + self._mark_detection_phase("stopping", "检测任务收到停止请求,准备安全退出") + self._sync_worker_log_event("检测任务收到停止请求,准备安全退出", level='warning', mode='key') + final_phase = "stopped" + final_detail = "检测任务收到停止请求,已停止继续领取任务" break - try: - thread.join(timeout=30) # 添加超时,避免线程阻塞 - current_active = len([t for t in active_threads if t.is_alive()]) - self._mark_detection_phase( - "running", - f"当前实际线程数量: {current_active}/{max_threads}", - active_threads=current_active, - max_threads=max_threads, - batch_size=current_batch_size, - processed=total_processed + i, + prune_finished_futures() + self._flush_pending_running_marks() + self._flush_pending_job_finalizations() + self._flush_pending_domain_completions() + self._flush_pending_domain_status_updates() + self._flush_pending_review_status_updates() + current_thread_limit = self.refresh_thread_count_runtime(force=True, min_interval=0.0) + max_threads = sync_worker_pool_size(current_thread_limit) + live_active, submitted_total, queued_backlog, backlog_limit, dispatch_capacity, available_slots = inflight_submission_metrics(max_threads) + now_ts = time.time() + if now_ts - last_expired_job_recycle_at >= 2.0: + recycled_job_items = self.db.recycle_expired_detect_job_items() + last_expired_job_recycle_at = now_ts + if recycled_job_items: + logger.warning(f"已回收 {recycled_job_items} 个租约过期的任务项,重新回到 pending") + if self.current_job_id and now_ts - last_job_status_refresh_at >= 2.0: + self.db.refresh_detect_job_status(self.current_job_id) + last_job_status_refresh_at = now_ts + pending_buffer_cap_multiplier = max( + 1, + int(os.getenv("DOMAINCHECK_PENDING_BUFFER_CAP_MULTIPLIER", "1") or 1), + ) + pending_buffer_target = min( + max(4, backlog_limit), + max(4, int(max_threads or 0) * pending_buffer_cap_multiplier), + ) + single_step_session_active = self._is_single_step_session_active() + refill_slots = max(0, min(int(available_slots or 0), pending_buffer_target - len(pending_domains))) + if refill_slots > 0: + claim_batch_size = compute_claim_batch_size(max_threads, refill_slots) + claim_lease_seconds = max(300, min(1800, claim_batch_size * 30)) + # single_step 必须绑定当前 job;domain_pipeline 则持续从全局任务队列补位, + # 否则旧 job 未完全结束时,新 job 会长期堆在 pending。 + scoped_job_id = ( + self.current_job_id + if self.current_job_id and self._should_scope_claims_to_current_job() + else None ) - # 更新进度 - if self.detect_thread: - try: - # 通过信号发送进度更新 - self.detect_thread.progress_signal.emit(total_processed + i + 1, total_processed + current_batch_size) - except Exception as e: - logger.error(f"发送进度更新信号失败: {e}") - import traceback - logger.error(traceback.format_exc()) - except Exception as e: - logger.error(f"等待线程完成失败: {e}") - import traceback - logger.error(traceback.format_exc()) - - # 发送线程数量更新信号 - if self.detect_thread: - try: - active_count = threading.active_count() - self.detect_thread.thread_count_signal.emit(active_count, max_threads) - except Exception as e: - logger.error(f"发送线程数量更新信号失败: {e}") - import traceback - logger.error(traceback.format_exc()) - - # 完成当前批次进度 - if self.detect_thread: - try: - # 通过信号发送进度更新 - self.detect_thread.progress_signal.emit(total_processed + current_batch_size, total_processed + current_batch_size) - except Exception as e: - logger.error(f"发送进度更新信号失败: {e}") - import traceback - logger.error(traceback.format_exc()) - - total_processed += current_batch_size - - # 如果当前批次数量少于batch_size,检测完成后停止获取域名 - if should_stop: - logger.info(f"当前批次域名数量 ({current_batch_size}) 少于 {batch_size},停止获取域名") - self._mark_detection_phase( - "completing", - f"当前批次域名数量 ({current_batch_size}) 少于 {batch_size},本轮检测即将完成", - processed=total_processed, - ) - self._sync_worker_log_event( - f"当前批次域名数量 ({current_batch_size}) 少于 {batch_size},本轮检测即将完成", - payload={"processed": total_processed}, - mode='key', - ) - break - logger.info(f"当前批次检测完成,累计处理 {total_processed} 个域名") - self._mark_detection_phase("batch_completed", f"当前批次检测完成,累计处理 {total_processed} 个域名", processed=total_processed) - self._sync_worker_log_event( - f"当前批次检测完成,累计处理 {total_processed} 个域名", - payload={"processed": total_processed}, - mode='key', - ) + claim_started_at = time.perf_counter() + domains = self.db.claim_detect_job_items( + config.NODE_CODE, + limit=claim_batch_size, + lease_seconds=claim_lease_seconds, + job_id=scoped_job_id, + ) + claim_elapsed_ms = int((time.perf_counter() - claim_started_at) * 1000) + legacy_fetch_elapsed_ms = 0 + if domains: + append_pending_domains( + domains, + True, + "任务队列", + claim_elapsed_ms, + legacy_fetch_elapsed_ms, + dispatch_capacity, + claim_batch_size, + claim_lease_seconds, + ) + elif not single_step_session_active: + pipeline_domains = self._process_pipeline_tasks_until_available(thread_limit=max_threads) + if pipeline_domains: + append_pending_domains( + pipeline_domains, + True, + "任务队列(pipeline推进)", + claim_elapsed_ms, + legacy_fetch_elapsed_ms, + dispatch_capacity, + claim_batch_size, + claim_lease_seconds, + ) + else: + refill_domains = self._pull_sync_tasks_until_available(thread_limit=max_threads) + if refill_domains: + append_pending_domains( + refill_domains, + True, + "任务队列(主动补货)", + claim_elapsed_ms, + legacy_fetch_elapsed_ms, + dispatch_capacity, + claim_batch_size, + claim_lease_seconds, + ) + elif has_dispatchable_step_items(): + logger.info("标准步骤队列仍有待派发任务,跳过兼容旧链路补位") + elif legacy_domain_fallback_enabled and not legacy_tail_reached: + legacy_fetch_limit = max(1, claim_batch_size) + legacy_fetch_started_at = time.perf_counter() + legacy_domains = self.db.get_domains_to_detect(limit=legacy_fetch_limit, detect_options=self.detect_options) + legacy_fetch_elapsed_ms = int((time.perf_counter() - legacy_fetch_started_at) * 1000) + if legacy_domains: + append_pending_domains( + legacy_domains, + False, + "兼容旧链路", + claim_elapsed_ms, + legacy_fetch_elapsed_ms, + dispatch_capacity, + claim_batch_size, + claim_lease_seconds, + ) + if len(legacy_domains) < legacy_fetch_limit: + legacy_tail_reached = True + elif not legacy_domain_fallback_enabled: + current_ts = time.time() + if current_ts - float(last_legacy_fallback_skip_log_at or 0.0) >= 5.0: + logger.info("兼容旧链路补位已禁用,当前仅允许 detect_job_items 标准步骤补位") + last_legacy_fallback_skip_log_at = current_ts + else: + logger.info("single_step 会话进行中,跳过 pipeline 推进 / 主动补货 / 兼容旧链路补位") + + dispatched_this_round = 0 + while pending_domains: + live_active, submitted_total, queued_backlog, backlog_limit, dispatch_capacity, effective_available = inflight_submission_metrics(max_threads) + if effective_available <= 0: + break + if not self.running or self.stop_requested: + logger.info("检测已停止,停止创建新线程") + self._mark_detection_phase("stopping", "检测已停止,停止创建新线程") + final_phase = "stopped" + final_detail = "检测已停止,停止创建新线程" + break + pending_entry = pending_domains.popleft() + domain = pending_entry.get("domain") or {} + using_job_queue = bool(pending_entry.get("using_job_queue")) + if start_domain_thread(domain, using_job_queue, max_threads): + dispatched_this_round += 1 + # 派发大批量任务时,尽早把刚启动的任务项从 claimed 刷到 running, + # 避免数据库状态长期滞后,影响 controller 对真实活跃量的判断。 + if dispatched_this_round % 64 == 0: + self._flush_pending_running_marks() + + if dispatched_this_round: + dispatch_summary_message = ( + f"持续补位派发完成: dispatched={dispatched_this_round} | " + f"active_threads={len(inflight_futures)} | pending_buffer={len(pending_domains)} | " + f"max_threads={max_threads} | queued_backlog={queued_backlog} | backlog_limit={backlog_limit}" + ) + logger.info(dispatch_summary_message) + self._sync_worker_log_event( + dispatch_summary_message, + payload={ + "dispatched": dispatched_this_round, + "active_threads": len(inflight_futures), + "pending_buffer": len(pending_domains), + "max_threads": max_threads, + "queued_backlog": queued_backlog, + "backlog_limit": backlog_limit, + }, + mode='full', + ) + + if not pending_domains and not inflight_futures: + if legacy_tail_reached: + logger.info("兼容旧链路已到尾批,且当前没有活跃线程,本轮检测完成") + self._mark_detection_phase( + "completing", + "兼容旧链路已到尾批,且当前没有活跃线程,本轮检测完成", + processed=total_processed, + ) + self._sync_worker_log_event( + "兼容旧链路已到尾批,且当前没有活跃线程,本轮检测完成", + payload={"processed": total_processed}, + mode='key', + ) + break + logger.info("没有需要检测的域名") + self._mark_detection_phase("idle", "当前没有需要检测的域名,Worker 等待下一次启动") + self._sync_worker_log_event("当前没有需要检测的域名,Worker 等待下一次启动", mode='key') + break + + emit_thread_progress(max_threads, batch_size=len(pending_domains) + len(inflight_futures)) + time.sleep(0.05 if pending_domains or inflight_futures else 0.2) + finally: + self._flush_pending_running_marks(force=True, batch_limit=5000) + self._flush_pending_job_finalizations(force=True, batch_limit=5000) + self._flush_pending_domain_completions(force=True, batch_limit=5000) + self._flush_pending_domain_status_updates(force=True, batch_limit=5000) + self._flush_pending_review_status_updates(force=True, batch_limit=5000) + if self.current_job_id: + self.db.refresh_detect_job_status(self.current_job_id) + worker_pool.shutdown(wait=False, cancel_futures=False) # 完成进度 if self.detect_thread: @@ -2910,13 +5901,27 @@ class DetectWorker: logger.info("域名检测任务完成") self._mark_detection_phase("completed", "域名检测任务完成") self._sync_worker_log_event("域名检测任务完成", payload={"processed": total_processed}, mode='key') + final_phase = "completed" + final_detail = "域名检测任务完成" except Exception as e: logger.error(f"执行检测任务出错: {e}") self._mark_detection_phase("failed", f"执行检测任务出错: {e}") self._sync_worker_log_event(f"执行检测任务出错: {e}", level='error', mode='key') + final_phase = "failed" + final_detail = f"执行检测任务出错: {e}" finally: self.detecting = False + if not self.running or self.stop_requested: + final_phase = "stopped" + final_detail = "检测任务已停止" + self._update_runtime_state( + final_phase, + final_detail, + active_threads=0, + max_threads=max(1, int(getattr(self, "thread_count", 1) or 1)), + processed=total_processed, + ) def run_daily_task(self): """ @@ -3097,6 +6102,7 @@ class DetectWorker: # 使用logger.debug输出到文件日志,不输出到GUI日志框 logger.debug("开始监听配置更新...") self._consume_pending_control_command() + self._resume_active_detect_job_if_needed(reason="redis_subscription_bootstrap") # 循环监听消息 while self.running: @@ -3109,12 +6115,27 @@ class DetectWorker: if channel == CONFIG_UPDATE_CHANNEL: config_type = payload logger.debug(f"收到配置更新消息: {config_type}") + previous_proxy_config = dict(self.proxy_config or {}) self.detect_options = self.load_detect_options() self.proxy_config = self.load_proxy_config() self.thread_count = self.load_thread_count() + self.sensitive_words = self._load_sensitive_words_runtime() + self.proxy_max_reuse_count = max(8, self.thread_count * 4) self.runtime_settings = self.load_runtime_settings() self.load_cookies_from_remote() self.update_config_labels() + previous_urls = tuple(previous_proxy_config.get('proxy_urls') or []) + current_urls = tuple(self.proxy_config.get('proxy_urls') or []) + proxy_changed = ( + bool(previous_proxy_config.get('proxy_enable', False)) != bool(self.proxy_config.get('proxy_enable', False)) + or bool(previous_proxy_config.get('allow_direct', False)) != bool(self.proxy_config.get('allow_direct', False)) + or previous_urls != current_urls + ) + if config_type in {"proxy_config", "thread_count", "node_thread_counts", "runtime_settings"} or proxy_changed: + self.trigger_proxy_refresh( + reason=f"config_update:{config_type}", + reset_cooldown=proxy_changed or config_type == "proxy_config", + ) logger.debug("配置已更新") continue @@ -3141,6 +6162,11 @@ class DetectWorker: """ logger.info("停止域名检测端") self._runtime_heartbeat_stop.set() + self._worker_log_sync_stop.set() + try: + self._worker_log_sync_queue.put_nowait(None) + except Exception: + pass self.stop_requested = True self.running = False self.detecting = False @@ -3149,7 +6175,9 @@ class DetectWorker: if __name__ == "__main__": try: # 配置日志 - logger.add("detect_worker.log", rotation="1 day", level="DEBUG") + log_file_path = _resolve_worker_log_file() + logger.add(log_file_path, rotation="1 day", level="DEBUG") + _raise_nofile_soft_limit() logger.info("程序开始运行") # 创建应用程序 diff --git a/domainCheck/node_thread_counts.json b/domainCheck/node_thread_counts.json index a4e3c54..e570b3c 100644 --- a/domainCheck/node_thread_counts.json +++ b/domainCheck/node_thread_counts.json @@ -1,4 +1,4 @@ { - "mainland-controller-01": 100, - "mainland-worker-01": 50 + "mainland-controller-01": 2000, + "mainland-worker-01": 1200 } \ No newline at end of file diff --git a/domainCheck/proxy_config.json b/domainCheck/proxy_config.json index 613c29a..6985fda 100644 --- a/domainCheck/proxy_config.json +++ b/domainCheck/proxy_config.json @@ -1,13 +1,13 @@ { "proxy_enable": true, "allow_direct": true, - "proxy_url": "http://211.101.244.154:18003/getProxy_batch.php?group=B&count=50", + "proxy_url": "http://211.101.244.154:18003/getProxy_batch.php?group=B&count=140", "proxy_urls": [ - "http://211.101.244.154:18003/getProxy_batch.php?group=B&count=50", - "http://211.101.244.154:18003/getProxy_batch.php?group=C&count=50", - "http://211.101.244.154:18003/getProxy_batch.php?group=D&count=50", - "http://211.101.244.154:18003/getProxy_batch.php?group=E&count=50", - "http://211.101.244.154:18008/getProxy_batch.php?group=F&count=50", - "http://211.101.244.154:18003/getProxy_batch.php?group=G&count=50" + "http://211.101.244.154:18003/getProxy_batch.php?group=B&count=140", + "http://211.101.244.154:18003/getProxy_batch.php?group=C&count=140", + "http://211.101.244.154:18003/getProxy_batch.php?group=D&count=140", + "http://211.101.244.154:18003/getProxy_batch.php?group=E&count=140", + "http://211.101.244.154:18003/getProxy_batch.php?group=F&count=140", + "http://211.101.244.154:18003/getProxy_batch.php?group=G&count=140" ] } \ No newline at end of file diff --git a/domainCheck/runtime/runtime_settings.json b/domainCheck/runtime/runtime_settings.json new file mode 100644 index 0000000..320f5f3 --- /dev/null +++ b/domainCheck/runtime/runtime_settings.json @@ -0,0 +1,8 @@ +{ + "worker_mode": "linux-systemd", + "worker_service_name": "domaincheck-worker", + "api_service_name": "domaincheck-api", + "sync_agent_service_name": "domaincheck-sync-agent", + "worker_log_sync_enabled": true, + "worker_log_sync_mode": "full" +} \ No newline at end of file diff --git a/domainCheck/runtime/sensitive_words.json b/domainCheck/runtime/sensitive_words.json new file mode 100644 index 0000000..1192d12 --- /dev/null +++ b/domainCheck/runtime/sensitive_words.json @@ -0,0 +1,2156 @@ +{ + "items": [ + { + "word": "11选5", + "category": "default", + "priority": 1 + }, + { + "word": "12bet", + "category": "default", + "priority": 1 + }, + { + "word": "1669k推", + "category": "default", + "priority": 1 + }, + { + "word": "5g", + "category": "default", + "priority": 1 + }, + { + "word": "91制片", + "category": "default", + "priority": 1 + }, + { + "word": "9码论坛", + "category": "default", + "priority": 1 + }, + { + "word": "adult", + "category": "default", + "priority": 1 + }, + { + "word": "AG", + "category": "default", + "priority": 1 + }, + { + "word": "ai", + "category": "default", + "priority": 1 + }, + { + "word": "App", + "category": "default", + "priority": 1 + }, + { + "word": "av", + "category": "default", + "priority": 1 + }, + { + "word": "AV", + "category": "default", + "priority": 1 + }, + { + "word": "av大片", + "category": "default", + "priority": 1 + }, + { + "word": "AV女优", + "category": "default", + "priority": 1 + }, + { + "word": "A片", + "category": "default", + "priority": 1 + }, + { + "word": "a级", + "category": "default", + "priority": 1 + }, + { + "word": "bc", + "category": "default", + "priority": 1 + }, + { + "word": "beplay", + "category": "default", + "priority": 1 + }, + { + "word": "Bet365", + "category": "default", + "priority": 1 + }, + { + "word": "BOB", + "category": "default", + "priority": 1 + }, + { + "word": "bocai", + "category": "default", + "priority": 1 + }, + { + "word": "cp", + "category": "default", + "priority": 1 + }, + { + "word": "d88", + "category": "default", + "priority": 1 + }, + { + "word": "DDoS", + "category": "default", + "priority": 1 + }, + { + "word": "DeFi博彩", + "category": "default", + "priority": 1 + }, + { + "word": "duchang", + "category": "default", + "priority": 1 + }, + { + "word": "EK娱乐", + "category": "default", + "priority": 1 + }, + { + "word": "erotic", + "category": "default", + "priority": 1 + }, + { + "word": "gb", + "category": "default", + "priority": 1 + }, + { + "word": "hentai", + "category": "default", + "priority": 1 + }, + { + "word": "iOS", + "category": "default", + "priority": 1 + }, + { + "word": "Job", + "category": "default", + "priority": 1 + }, + { + "word": "K8", + "category": "default", + "priority": 1 + }, + { + "word": "k8国际", + "category": "default", + "priority": 1 + }, + { + "word": "k8娱乐", + "category": "default", + "priority": 1 + }, + { + "word": "kok", + "category": "default", + "priority": 1 + }, + { + "word": "K粉", + "category": "default", + "priority": 1 + }, + { + "word": "Nab", + "category": "default", + "priority": 1 + }, + { + "word": "nude", + "category": "default", + "priority": 1 + }, + { + "word": "P2P理财", + "category": "default", + "priority": 1 + }, + { + "word": "Pk", + "category": "default", + "priority": 1 + }, + { + "word": "Pk10", + "category": "default", + "priority": 1 + }, + { + "word": "porn", + "category": "default", + "priority": 1 + }, + { + "word": "qq", + "category": "default", + "priority": 1 + }, + { + "word": "sex", + "category": "default", + "priority": 1 + }, + { + "word": "twitch", + "category": "default", + "priority": 1 + }, + { + "word": "U乐", + "category": "default", + "priority": 1 + }, + { + "word": "VIP", + "category": "default", + "priority": 1 + }, + { + "word": "v片", + "category": "default", + "priority": 1 + }, + { + "word": "xxx", + "category": "default", + "priority": 1 + }, + { + "word": "一区二区", + "category": "default", + "priority": 1 + }, + { + "word": "一夜情", + "category": "default", + "priority": 1 + }, + { + "word": "一夜晴", + "category": "default", + "priority": 1 + }, + { + "word": "一夜暴富", + "category": "default", + "priority": 1 + }, + { + "word": "一级", + "category": "default", + "priority": 1 + }, + { + "word": "一肖一码", + "category": "default", + "priority": 1 + }, + { + "word": "一键", + "category": "default", + "priority": 1 + }, + { + "word": "七步诗", + "category": "default", + "priority": 1 + }, + { + "word": "三级", + "category": "default", + "priority": 1 + }, + { + "word": "三级片", + "category": "default", + "priority": 1 + }, + { + "word": "三级视频", + "category": "default", + "priority": 1 + }, + { + "word": "上线", + "category": "default", + "priority": 1 + }, + { + "word": "下注", + "category": "default", + "priority": 1 + }, + { + "word": "下载", + "category": "default", + "priority": 1 + }, + { + "word": "不卡", + "category": "default", + "priority": 1 + }, + { + "word": "世界杯", + "category": "default", + "priority": 1 + }, + { + "word": "丝瓜", + "category": "default", + "priority": 1 + }, + { + "word": "丝瓜视频", + "category": "default", + "priority": 1 + }, + { + "word": "丝袜", + "category": "default", + "priority": 1 + }, + { + "word": "中介", + "category": "default", + "priority": 1 + }, + { + "word": "中奖通知", + "category": "default", + "priority": 1 + }, + { + "word": "中文字幕", + "category": "default", + "priority": 1 + }, + { + "word": "久草", + "category": "default", + "priority": 1 + }, + { + "word": "乐虎", + "category": "default", + "priority": 1 + }, + { + "word": "乐鱼", + "category": "default", + "priority": 1 + }, + { + "word": "九州", + "category": "default", + "priority": 1 + }, + { + "word": "九游", + "category": "default", + "priority": 1 + }, + { + "word": "乱伦", + "category": "default", + "priority": 1 + }, + { + "word": "二维码", + "category": "default", + "priority": 1 + }, + { + "word": "云博", + "category": "default", + "priority": 1 + }, + { + "word": "云顶国际", + "category": "default", + "priority": 1 + }, + { + "word": "五月色", + "category": "default", + "priority": 1 + }, + { + "word": "亚太网", + "category": "default", + "priority": 1 + }, + { + "word": "亚洲", + "category": "default", + "priority": 1 + }, + { + "word": "亚洲美女", + "category": "default", + "priority": 1 + }, + { + "word": "亚美", + "category": "default", + "priority": 1 + }, + { + "word": "交易", + "category": "default", + "priority": 1 + }, + { + "word": "人妻", + "category": "default", + "priority": 1 + }, + { + "word": "人才市场", + "category": "default", + "priority": 1 + }, + { + "word": "今日", + "category": "default", + "priority": 1 + }, + { + "word": "代刷", + "category": "default", + "priority": 1 + }, + { + "word": "代收", + "category": "default", + "priority": 1 + }, + { + "word": "代考", + "category": "default", + "priority": 1 + }, + { + "word": "传奇", + "category": "default", + "priority": 1 + }, + { + "word": "传媒", + "category": "default", + "priority": 1 + }, + { + "word": "体彩", + "category": "default", + "priority": 1 + }, + { + "word": "体育", + "category": "default", + "priority": 1 + }, + { + "word": "体育app", + "category": "default", + "priority": 1 + }, + { + "word": "体育博彩", + "category": "default", + "priority": 1 + }, + { + "word": "体育投注", + "category": "default", + "priority": 1 + }, + { + "word": "体验", + "category": "default", + "priority": 1 + }, + { + "word": "体验金", + "category": "default", + "priority": 1 + }, + { + "word": "信用卡套现", + "category": "default", + "priority": 1 + }, + { + "word": "做爱", + "category": "default", + "priority": 1 + }, + { + "word": "偷拍", + "category": "default", + "priority": 1 + }, + { + "word": "免费", + "category": "default", + "priority": 1 + }, + { + "word": "免费人成视频", + "category": "default", + "priority": 1 + }, + { + "word": "入口", + "category": "default", + "priority": 1 + }, + { + "word": "全本", + "category": "default", + "priority": 1 + }, + { + "word": "全讯", + "category": "default", + "priority": 1 + }, + { + "word": "全集", + "category": "default", + "priority": 1 + }, + { + "word": "六合", + "category": "default", + "priority": 1 + }, + { + "word": "六合彩", + "category": "default", + "priority": 1 + }, + { + "word": "共产", + "category": "default", + "priority": 1 + }, + { + "word": "养号", + "category": "default", + "priority": 1 + }, + { + "word": "冒充客服", + "category": "default", + "priority": 1 + }, + { + "word": "军委", + "category": "default", + "priority": 1 + }, + { + "word": "军火", + "category": "default", + "priority": 1 + }, + { + "word": "冰毒", + "category": "default", + "priority": 1 + }, + { + "word": "凯发", + "category": "default", + "priority": 1 + }, + { + "word": "分分彩", + "category": "default", + "priority": 1 + }, + { + "word": "利高", + "category": "default", + "priority": 1 + }, + { + "word": "刷单", + "category": "default", + "priority": 1 + }, + { + "word": "刷单任务", + "category": "default", + "priority": 1 + }, + { + "word": "加密彩票", + "category": "default", + "priority": 1 + }, + { + "word": "加载中", + "category": "default", + "priority": 1 + }, + { + "word": "动漫", + "category": "default", + "priority": 1 + }, + { + "word": "勒索软件", + "category": "default", + "priority": 1 + }, + { + "word": "北京快乐8", + "category": "default", + "priority": 1 + }, + { + "word": "北京赛车", + "category": "default", + "priority": 1 + }, + { + "word": "区块链博彩", + "category": "default", + "priority": 1 + }, + { + "word": "十八彩", + "category": "default", + "priority": 1 + }, + { + "word": "千亿体育", + "category": "default", + "priority": 1 + }, + { + "word": "千亿国际", + "category": "default", + "priority": 1 + }, + { + "word": "午夜", + "category": "default", + "priority": 1 + }, + { + "word": "午夜成年", + "category": "default", + "priority": 1 + }, + { + "word": "华体", + "category": "default", + "priority": 1 + }, + { + "word": "单双", + "category": "default", + "priority": 1 + }, + { + "word": "博彩", + "category": "default", + "priority": 1 + }, + { + "word": "博彩网站", + "category": "default", + "priority": 1 + }, + { + "word": "博彩预测", + "category": "default", + "priority": 1 + }, + { + "word": "博狗", + "category": "default", + "priority": 1 + }, + { + "word": "博鱼", + "category": "default", + "priority": 1 + }, + { + "word": "卡商", + "category": "default", + "priority": 1 + }, + { + "word": "卡盟", + "category": "default", + "priority": 1 + }, + { + "word": "发牌", + "category": "default", + "priority": 1 + }, + { + "word": "发财网", + "category": "default", + "priority": 1 + }, + { + "word": "同城聊天室", + "category": "default", + "priority": 1 + }, + { + "word": "吸精", + "category": "default", + "priority": 1 + }, + { + "word": "咪咕", + "category": "default", + "priority": 1 + }, + { + "word": "哥哥撸", + "category": "default", + "priority": 1 + }, + { + "word": "商城", + "category": "default", + "priority": 1 + }, + { + "word": "啪啪", + "category": "default", + "priority": 1 + }, + { + "word": "四方支付", + "category": "default", + "priority": 1 + }, + { + "word": "国产", + "category": "default", + "priority": 1 + }, + { + "word": "国产专区", + "category": "default", + "priority": 1 + }, + { + "word": "国产福利", + "category": "default", + "priority": 1 + }, + { + "word": "国产精品", + "category": "default", + "priority": 1 + }, + { + "word": "国语", + "category": "default", + "priority": 1 + }, + { + "word": "国际", + "category": "default", + "priority": 1 + }, + { + "word": "图片", + "category": "default", + "priority": 1 + }, + { + "word": "在线", + "category": "default", + "priority": 1 + }, + { + "word": "在线博彩", + "category": "default", + "priority": 1 + }, + { + "word": "在线无码", + "category": "default", + "priority": 1 + }, + { + "word": "地主", + "category": "default", + "priority": 1 + }, + { + "word": "地址", + "category": "default", + "priority": 1 + }, + { + "word": "外围", + "category": "default", + "priority": 1 + }, + { + "word": "外挂", + "category": "default", + "priority": 1 + }, + { + "word": "大乐透", + "category": "default", + "priority": 1 + }, + { + "word": "大发888", + "category": "default", + "priority": 1 + }, + { + "word": "大小", + "category": "default", + "priority": 1 + }, + { + "word": "天天", + "category": "default", + "priority": 1 + }, + { + "word": "天天彩", + "category": "default", + "priority": 1 + }, + { + "word": "天美传媒", + "category": "default", + "priority": 1 + }, + { + "word": "太阳城", + "category": "default", + "priority": 1 + }, + { + "word": "太阳备用", + "category": "default", + "priority": 1 + }, + { + "word": "太阳成集团", + "category": "default", + "priority": 1 + }, + { + "word": "套现", + "category": "default", + "priority": 1 + }, + { + "word": "女优", + "category": "default", + "priority": 1 + }, + { + "word": "好浪", + "category": "default", + "priority": 1 + }, + { + "word": "好爽", + "category": "default", + "priority": 1 + }, + { + "word": "好紧", + "category": "default", + "priority": 1 + }, + { + "word": "威尼斯", + "category": "default", + "priority": 1 + }, + { + "word": "娱乐", + "category": "default", + "priority": 1 + }, + { + "word": "娱乐在线", + "category": "default", + "priority": 1 + }, + { + "word": "娱乐场", + "category": "default", + "priority": 1 + }, + { + "word": "娱乐官网", + "category": "default", + "priority": 1 + }, + { + "word": "媒体", + "category": "default", + "priority": 1 + }, + { + "word": "安卓", + "category": "default", + "priority": 1 + }, + { + "word": "宝博", + "category": "default", + "priority": 1 + }, + { + "word": "寂寞", + "category": "default", + "priority": 1 + }, + { + "word": "小姐姐", + "category": "default", + "priority": 1 + }, + { + "word": "小蝌蚪", + "category": "default", + "priority": 1 + }, + { + "word": "小说", + "category": "default", + "priority": 1 + }, + { + "word": "岛国", + "category": "default", + "priority": 1 + }, + { + "word": "币博", + "category": "default", + "priority": 1 + }, + { + "word": "平台", + "category": "default", + "priority": 1 + }, + { + "word": "开心", + "category": "default", + "priority": 1 + }, + { + "word": "开户", + "category": "default", + "priority": 1 + }, + { + "word": "开锁", + "category": "default", + "priority": 1 + }, + { + "word": "引流变现", + "category": "default", + "priority": 1 + }, + { + "word": "强奸", + "category": "default", + "priority": 1 + }, + { + "word": "录像", + "category": "default", + "priority": 1 + }, + { + "word": "彩神", + "category": "default", + "priority": 1 + }, + { + "word": "彩票", + "category": "default", + "priority": 1 + }, + { + "word": "彩票预测", + "category": "default", + "priority": 1 + }, + { + "word": "彩金", + "category": "default", + "priority": 1 + }, + { + "word": "影视", + "category": "default", + "priority": 1 + }, + { + "word": "影院", + "category": "default", + "priority": 1 + }, + { + "word": "很黄很黄", + "category": "default", + "priority": 1 + }, + { + "word": "德州", + "category": "default", + "priority": 1 + }, + { + "word": "必赢", + "category": "default", + "priority": 1 + }, + { + "word": "快3", + "category": "default", + "priority": 1 + }, + { + "word": "快三", + "category": "default", + "priority": 1 + }, + { + "word": "快乐", + "category": "default", + "priority": 1 + }, + { + "word": "快播", + "category": "default", + "priority": 1 + }, + { + "word": "快速致富", + "category": "default", + "priority": 1 + }, + { + "word": "性", + "category": "default", + "priority": 1 + }, + { + "word": "性爱", + "category": "default", + "priority": 1 + }, + { + "word": "性用品", + "category": "default", + "priority": 1 + }, + { + "word": "性病", + "category": "default", + "priority": 1 + }, + { + "word": "恐怖主义", + "category": "default", + "priority": 1 + }, + { + "word": "情人交友网", + "category": "default", + "priority": 1 + }, + { + "word": "情色", + "category": "default", + "priority": 1 + }, + { + "word": "情色小说", + "category": "default", + "priority": 1 + }, + { + "word": "情色聊天室", + "category": "default", + "priority": 1 + }, + { + "word": "成人", + "category": "default", + "priority": 1 + }, + { + "word": "成人直播", + "category": "default", + "priority": 1 + }, + { + "word": "成人看片", + "category": "default", + "priority": 1 + }, + { + "word": "成人网站", + "category": "default", + "priority": 1 + }, + { + "word": "成人视频", + "category": "default", + "priority": 1 + }, + { + "word": "成本人视频", + "category": "default", + "priority": 1 + }, + { + "word": "房", + "category": "default", + "priority": 1 + }, + { + "word": "扑克", + "category": "default", + "priority": 1 + }, + { + "word": "打开", + "category": "default", + "priority": 1 + }, + { + "word": "投注", + "category": "default", + "priority": 1 + }, + { + "word": "投资理财骗局", + "category": "default", + "priority": 1 + }, + { + "word": "报告", + "category": "default", + "priority": 1 + }, + { + "word": "抽搐", + "category": "default", + "priority": 1 + }, + { + "word": "接码", + "category": "default", + "priority": 1 + }, + { + "word": "提示", + "category": "default", + "priority": 1 + }, + { + "word": "摄像", + "category": "default", + "priority": 1 + }, + { + "word": "播放", + "category": "default", + "priority": 1 + }, + { + "word": "收码平台", + "category": "default", + "priority": 1 + }, + { + "word": "新蒲京", + "category": "default", + "priority": 1 + }, + { + "word": "新闻", + "category": "default", + "priority": 1 + }, + { + "word": "旅行", + "category": "default", + "priority": 1 + }, + { + "word": "旗舰", + "category": "default", + "priority": 1 + }, + { + "word": "无法找到该页", + "category": "default", + "priority": 1 + }, + { + "word": "无码", + "category": "default", + "priority": 1 + }, + { + "word": "无码视频", + "category": "default", + "priority": 1 + }, + { + "word": "无限", + "category": "default", + "priority": 1 + }, + { + "word": "日出水", + "category": "default", + "priority": 1 + }, + { + "word": "日博", + "category": "default", + "priority": 1 + }, + { + "word": "日本", + "category": "default", + "priority": 1 + }, + { + "word": "日本免费", + "category": "default", + "priority": 1 + }, + { + "word": "日韩", + "category": "default", + "priority": 1 + }, + { + "word": "时时", + "category": "default", + "priority": 1 + }, + { + "word": "时时彩", + "category": "default", + "priority": 1 + }, + { + "word": "星际", + "category": "default", + "priority": 1 + }, + { + "word": "春情", + "category": "default", + "priority": 1 + }, + { + "word": "暴利", + "category": "default", + "priority": 1 + }, + { + "word": "更新", + "category": "default", + "priority": 1 + }, + { + "word": "木马", + "category": "default", + "priority": 1 + }, + { + "word": "未删", + "category": "default", + "priority": 1 + }, + { + "word": "杀猪盘", + "category": "default", + "priority": 1 + }, + { + "word": "枪支", + "category": "default", + "priority": 1 + }, + { + "word": "标题", + "category": "default", + "priority": 1 + }, + { + "word": "棋牌", + "category": "default", + "priority": 1 + }, + { + "word": "棋牌游戏", + "category": "default", + "priority": 1 + }, + { + "word": "棋盘", + "category": "default", + "priority": 1 + }, + { + "word": "欧冠", + "category": "default", + "priority": 1 + }, + { + "word": "欧博", + "category": "default", + "priority": 1 + }, + { + "word": "欧宝", + "category": "default", + "priority": 1 + }, + { + "word": "欧洲杯", + "category": "default", + "priority": 1 + }, + { + "word": "欧美", + "category": "default", + "priority": 1 + }, + { + "word": "欧美成人看", + "category": "default", + "priority": 1 + }, + { + "word": "欧美精品", + "category": "default", + "priority": 1 + }, + { + "word": "毛片", + "category": "default", + "priority": 1 + }, + { + "word": "水蜜桃", + "category": "default", + "priority": 1 + }, + { + "word": "永久", + "category": "default", + "priority": 1 + }, + { + "word": "永利", + "category": "default", + "priority": 1 + }, + { + "word": "江南娱乐", + "category": "default", + "priority": 1 + }, + { + "word": "污片", + "category": "default", + "priority": 1 + }, + { + "word": "污视频", + "category": "default", + "priority": 1 + }, + { + "word": "沙巴", + "category": "default", + "priority": 1 + }, + { + "word": "注册", + "category": "default", + "priority": 1 + }, + { + "word": "注册-", + "category": "default", + "priority": 1 + }, + { + "word": "洗钱", + "category": "default", + "priority": 1 + }, + { + "word": "浦京", + "category": "default", + "priority": 1 + }, + { + "word": "游戏", + "category": "default", + "priority": 1 + }, + { + "word": "澳门", + "category": "default", + "priority": 1 + }, + { + "word": "澳门赌场", + "category": "default", + "priority": 1 + }, + { + "word": "激情", + "category": "default", + "priority": 1 + }, + { + "word": "灰色产业", + "category": "default", + "priority": 1 + }, + { + "word": "炒币", + "category": "default", + "priority": 1 + }, + { + "word": "熟女", + "category": "default", + "priority": 1 + }, + { + "word": "爆破", + "category": "default", + "priority": 1 + }, + { + "word": "版本", + "category": "default", + "priority": 1 + }, + { + "word": "特别黄的视频", + "category": "default", + "priority": 1 + }, + { + "word": "特黄", + "category": "default", + "priority": 1 + }, + { + "word": "特黄毛片", + "category": "default", + "priority": 1 + }, + { + "word": "猫咪", + "category": "default", + "priority": 1 + }, + { + "word": "现金放心", + "category": "default", + "priority": 1 + }, + { + "word": "球", + "category": "default", + "priority": 1 + }, + { + "word": "申博", + "category": "default", + "priority": 1 + }, + { + "word": "电影", + "category": "default", + "priority": 1 + }, + { + "word": "电玩城", + "category": "default", + "priority": 1 + }, + { + "word": "电竞", + "category": "default", + "priority": 1 + }, + { + "word": "登录", + "category": "default", + "priority": 1 + }, + { + "word": "白粉", + "category": "default", + "priority": 1 + }, + { + "word": "百家", + "category": "default", + "priority": 1 + }, + { + "word": "百家乐", + "category": "default", + "priority": 1 + }, + { + "word": "百科", + "category": "default", + "priority": 1 + }, + { + "word": "皇冠", + "category": "default", + "priority": 1 + }, + { + "word": "皇朝娛樂", + "category": "default", + "priority": 1 + }, + { + "word": "盗号", + "category": "default", + "priority": 1 + }, + { + "word": "直播", + "category": "default", + "priority": 1 + }, + { + "word": "真人", + "category": "default", + "priority": 1 + }, + { + "word": "真人网娱乐", + "category": "default", + "priority": 1 + }, + { + "word": "真实", + "category": "default", + "priority": 1 + }, + { + "word": "真钱游戏", + "category": "default", + "priority": 1 + }, + { + "word": "知乎", + "category": "default", + "priority": 1 + }, + { + "word": "研究", + "category": "default", + "priority": 1 + }, + { + "word": "破解", + "category": "default", + "priority": 1 + }, + { + "word": "社工库", + "category": "default", + "priority": 1 + }, + { + "word": "福利", + "category": "default", + "priority": 1 + }, + { + "word": "私家", + "category": "default", + "priority": 1 + }, + { + "word": "私拍", + "category": "default", + "priority": 1 + }, + { + "word": "私服", + "category": "default", + "priority": 1 + }, + { + "word": "秋葵", + "category": "default", + "priority": 1 + }, + { + "word": "科普一下", + "category": "default", + "priority": 1 + }, + { + "word": "秒速pk10", + "category": "default", + "priority": 1 + }, + { + "word": "移民", + "category": "default", + "priority": 1 + }, + { + "word": "程序", + "category": "default", + "priority": 1 + }, + { + "word": "稳赢", + "category": "default", + "priority": 1 + }, + { + "word": "章节", + "category": "default", + "priority": 1 + }, + { + "word": "米乐体育", + "category": "default", + "priority": 1 + }, + { + "word": "精品视频", + "category": "default", + "priority": 1 + }, + { + "word": "约会", + "category": "default", + "priority": 1 + }, + { + "word": "约炮", + "category": "default", + "priority": 1 + }, + { + "word": "线路导航", + "category": "default", + "priority": 1 + }, + { + "word": "网上", + "category": "default", + "priority": 1 + }, + { + "word": "网络兼职", + "category": "default", + "priority": 1 + }, + { + "word": "网贷", + "category": "default", + "priority": 1 + }, + { + "word": "网赚", + "category": "default", + "priority": 1 + }, + { + "word": "网页", + "category": "default", + "priority": 1 + }, + { + "word": "美高梅", + "category": "default", + "priority": 1 + }, + { + "word": "老虎机", + "category": "default", + "priority": 1 + }, + { + "word": "联网", + "category": "default", + "priority": 1 + }, + { + "word": "自拍", + "category": "default", + "priority": 1 + }, + { + "word": "色", + "category": "default", + "priority": 1 + }, + { + "word": "色情", + "category": "default", + "priority": 1 + }, + { + "word": "色站", + "category": "default", + "priority": 1 + }, + { + "word": "色群", + "category": "default", + "priority": 1 + }, + { + "word": "色视", + "category": "default", + "priority": 1 + }, + { + "word": "色视频", + "category": "default", + "priority": 1 + }, + { + "word": "艳舞", + "category": "default", + "priority": 1 + }, + { + "word": "芭莎", + "category": "default", + "priority": 1 + }, + { + "word": "茄子", + "category": "default", + "priority": 1 + }, + { + "word": "草莓视频", + "category": "default", + "priority": 1 + }, + { + "word": "菠菜", + "category": "default", + "priority": 1 + }, + { + "word": "葡京", + "category": "default", + "priority": 1 + }, + { + "word": "葡萄京", + "category": "default", + "priority": 1 + }, + { + "word": "虚假招聘", + "category": "default", + "priority": 1 + }, + { + "word": "虚拟货币", + "category": "default", + "priority": 1 + }, + { + "word": "蜜桃传媒", + "category": "default", + "priority": 1 + }, + { + "word": "裸照", + "category": "default", + "priority": 1 + }, + { + "word": "裸聊", + "category": "default", + "priority": 1 + }, + { + "word": "观看", + "category": "default", + "priority": 1 + }, + { + "word": "视频", + "category": "default", + "priority": 1 + }, + { + "word": "视频裸聊", + "category": "default", + "priority": 1 + }, + { + "word": "认证", + "category": "default", + "priority": 1 + }, + { + "word": "论理", + "category": "default", + "priority": 1 + }, + { + "word": "证券", + "category": "default", + "priority": 1 + }, + { + "word": "诈金花", + "category": "default", + "priority": 1 + }, + { + "word": "语音", + "category": "default", + "priority": 1 + }, + { + "word": "请等待", + "category": "default", + "priority": 1 + }, + { + "word": "财神娱乐", + "category": "default", + "priority": 1 + }, + { + "word": "贩毒", + "category": "default", + "priority": 1 + }, + { + "word": "贷款", + "category": "default", + "priority": 1 + }, + { + "word": "贷款秒批", + "category": "default", + "priority": 1 + }, + { + "word": "贸易", + "category": "default", + "priority": 1 + }, + { + "word": "资源", + "category": "default", + "priority": 1 + }, + { + "word": "资金盘", + "category": "default", + "priority": 1 + }, + { + "word": "赌博", + "category": "default", + "priority": 1 + }, + { + "word": "赌场", + "category": "default", + "priority": 1 + }, + { + "word": "赌球", + "category": "default", + "priority": 1 + }, + { + "word": "赔率", + "category": "default", + "priority": 1 + }, + { + "word": "赚快钱", + "category": "default", + "priority": 1 + }, + { + "word": "赛车pk", + "category": "default", + "priority": 1 + }, + { + "word": "赛车计划", + "category": "default", + "priority": 1 + }, + { + "word": "走私", + "category": "default", + "priority": 1 + }, + { + "word": "足球", + "category": "default", + "priority": 1 + }, + { + "word": "跑分", + "category": "default", + "priority": 1 + }, + { + "word": "转码", + "category": "default", + "priority": 1 + }, + { + "word": "软件", + "category": "default", + "priority": 1 + }, + { + "word": "违禁品", + "category": "default", + "priority": 1 + }, + { + "word": "迷药", + "category": "default", + "priority": 1 + }, + { + "word": "退款诈骗", + "category": "default", + "priority": 1 + }, + { + "word": "送68", + "category": "default", + "priority": 1 + }, + { + "word": "造假", + "category": "default", + "priority": 1 + }, + { + "word": "邪教", + "category": "default", + "priority": 1 + }, + { + "word": "金彩国际", + "category": "default", + "priority": 1 + }, + { + "word": "金沙", + "category": "default", + "priority": 1 + }, + { + "word": "金沙娱乐", + "category": "default", + "priority": 1 + }, + { + "word": "金洋", + "category": "default", + "priority": 1 + }, + { + "word": "钓鱼", + "category": "default", + "priority": 1 + }, + { + "word": "钱", + "category": "default", + "priority": 1 + }, + { + "word": "银河", + "category": "default", + "priority": 1 + }, + { + "word": "银河总站", + "category": "default", + "priority": 1 + }, + { + "word": "银行", + "category": "default", + "priority": 1 + }, + { + "word": "错误", + "category": "default", + "priority": 1 + }, + { + "word": "防火墙", + "category": "default", + "priority": 1 + }, + { + "word": "露脸", + "category": "default", + "priority": 1 + }, + { + "word": "韩国", + "category": "default", + "priority": 1 + }, + { + "word": "香港", + "category": "default", + "priority": 1 + }, + { + "word": "香港六合", + "category": "default", + "priority": 1 + }, + { + "word": "香港马会", + "category": "default", + "priority": 1 + }, + { + "word": "香蕉", + "category": "default", + "priority": 1 + }, + { + "word": "高清", + "category": "default", + "priority": 1 + }, + { + "word": "高潮", + "category": "default", + "priority": 1 + }, + { + "word": "鱼", + "category": "default", + "priority": 1 + }, + { + "word": "麻豆", + "category": "default", + "priority": 1 + }, + { + "word": "黄色", + "category": "default", + "priority": 1 + }, + { + "word": "黄色片", + "category": "default", + "priority": 1 + }, + { + "word": "黄金彩", + "category": "default", + "priority": 1 + }, + { + "word": "黑客", + "category": "default", + "priority": 1 + }, + { + "word": "黑市", + "category": "default", + "priority": 1 + }, + { + "word": "龙珠", + "category": "default", + "priority": 1 + } + ], + "text": "11选5\n12bet\n1669k推\n5g\n91制片\n9码论坛\nadult\nAG\nai\nApp\nav\nAV\nav大片\nAV女优\nA片\na级\nbc\nbeplay\nBet365\nBOB\nbocai\ncp\nd88\nDDoS\nDeFi博彩\nduchang\nEK娱乐\nerotic\ngb\nhentai\niOS\nJob\nK8\nk8国际\nk8娱乐\nkok\nK粉\nNab\nnude\nP2P理财\nPk\nPk10\nporn\nqq\nsex\ntwitch\nU乐\nVIP\nv片\nxxx\n一区二区\n一夜情\n一夜晴\n一夜暴富\n一级\n一肖一码\n一键\n七步诗\n三级\n三级片\n三级视频\n上线\n下注\n下载\n不卡\n世界杯\n丝瓜\n丝瓜视频\n丝袜\n中介\n中奖通知\n中文字幕\n久草\n乐虎\n乐鱼\n九州\n九游\n乱伦\n二维码\n云博\n云顶国际\n五月色\n亚太网\n亚洲\n亚洲美女\n亚美\n交易\n人妻\n人才市场\n今日\n代刷\n代收\n代考\n传奇\n传媒\n体彩\n体育\n体育app\n体育博彩\n体育投注\n体验\n体验金\n信用卡套现\n做爱\n偷拍\n免费\n免费人成视频\n入口\n全本\n全讯\n全集\n六合\n六合彩\n共产\n养号\n冒充客服\n军委\n军火\n冰毒\n凯发\n分分彩\n利高\n刷单\n刷单任务\n加密彩票\n加载中\n动漫\n勒索软件\n北京快乐8\n北京赛车\n区块链博彩\n十八彩\n千亿体育\n千亿国际\n午夜\n午夜成年\n华体\n单双\n博彩\n博彩网站\n博彩预测\n博狗\n博鱼\n卡商\n卡盟\n发牌\n发财网\n同城聊天室\n吸精\n咪咕\n哥哥撸\n商城\n啪啪\n四方支付\n国产\n国产专区\n国产福利\n国产精品\n国语\n国际\n图片\n在线\n在线博彩\n在线无码\n地主\n地址\n外围\n外挂\n大乐透\n大发888\n大小\n天天\n天天彩\n天美传媒\n太阳城\n太阳备用\n太阳成集团\n套现\n女优\n好浪\n好爽\n好紧\n威尼斯\n娱乐\n娱乐在线\n娱乐场\n娱乐官网\n媒体\n安卓\n宝博\n寂寞\n小姐姐\n小蝌蚪\n小说\n岛国\n币博\n平台\n开心\n开户\n开锁\n引流变现\n强奸\n录像\n彩神\n彩票\n彩票预测\n彩金\n影视\n影院\n很黄很黄\n德州\n必赢\n快3\n快三\n快乐\n快播\n快速致富\n性\n性爱\n性用品\n性病\n恐怖主义\n情人交友网\n情色\n情色小说\n情色聊天室\n成人\n成人直播\n成人看片\n成人网站\n成人视频\n成本人视频\n房\n扑克\n打开\n投注\n投资理财骗局\n报告\n抽搐\n接码\n提示\n摄像\n播放\n收码平台\n新蒲京\n新闻\n旅行\n旗舰\n无法找到该页\n无码\n无码视频\n无限\n日出水\n日博\n日本\n日本免费\n日韩\n时时\n时时彩\n星际\n春情\n暴利\n更新\n木马\n未删\n杀猪盘\n枪支\n标题\n棋牌\n棋牌游戏\n棋盘\n欧冠\n欧博\n欧宝\n欧洲杯\n欧美\n欧美成人看\n欧美精品\n毛片\n水蜜桃\n永久\n永利\n江南娱乐\n污片\n污视频\n沙巴\n注册\n注册-\n洗钱\n浦京\n游戏\n澳门\n澳门赌场\n激情\n灰色产业\n炒币\n熟女\n爆破\n版本\n特别黄的视频\n特黄\n特黄毛片\n猫咪\n现金放心\n球\n申博\n电影\n电玩城\n电竞\n登录\n白粉\n百家\n百家乐\n百科\n皇冠\n皇朝娛樂\n盗号\n直播\n真人\n真人网娱乐\n真实\n真钱游戏\n知乎\n研究\n破解\n社工库\n福利\n私家\n私拍\n私服\n秋葵\n科普一下\n秒速pk10\n移民\n程序\n稳赢\n章节\n米乐体育\n精品视频\n约会\n约炮\n线路导航\n网上\n网络兼职\n网贷\n网赚\n网页\n美高梅\n老虎机\n联网\n自拍\n色\n色情\n色站\n色群\n色视\n色视频\n艳舞\n芭莎\n茄子\n草莓视频\n菠菜\n葡京\n葡萄京\n虚假招聘\n虚拟货币\n蜜桃传媒\n裸照\n裸聊\n观看\n视频\n视频裸聊\n认证\n论理\n证券\n诈金花\n语音\n请等待\n财神娱乐\n贩毒\n贷款\n贷款秒批\n贸易\n资源\n资金盘\n赌博\n赌场\n赌球\n赔率\n赚快钱\n赛车pk\n赛车计划\n走私\n足球\n跑分\n转码\n软件\n违禁品\n迷药\n退款诈骗\n送68\n造假\n邪教\n金彩国际\n金沙\n金沙娱乐\n金洋\n钓鱼\n钱\n银河\n银河总站\n银行\n错误\n防火墙\n露脸\n韩国\n香港\n香港六合\n香港马会\n香蕉\n高清\n高潮\n鱼\n麻豆\n黄色\n黄色片\n黄金彩\n黑客\n黑市\n龙珠", + "total": 430 +} \ No newline at end of file diff --git a/domainCheck/runtime_settings.json b/domainCheck/runtime_settings.json new file mode 100644 index 0000000..320f5f3 --- /dev/null +++ b/domainCheck/runtime_settings.json @@ -0,0 +1,8 @@ +{ + "worker_mode": "linux-systemd", + "worker_service_name": "domaincheck-worker", + "api_service_name": "domaincheck-api", + "sync_agent_service_name": "domaincheck-sync-agent", + "worker_log_sync_enabled": true, + "worker_log_sync_mode": "full" +} \ No newline at end of file diff --git a/domainCheck/tests/test_database_claim_priority.py b/domainCheck/tests/test_database_claim_priority.py new file mode 100644 index 0000000..818cf35 --- /dev/null +++ b/domainCheck/tests/test_database_claim_priority.py @@ -0,0 +1,55 @@ +import unittest +import sys + + +sys.path.insert(0, "/www/wwwroot/getDomain/domainCheck") + +from app.utils.database import _detect_job_item_step_priority +from app.utils.database import _ordered_step_claim_codes +from app.utils.database import _resolve_step_claim_quota + + +class DetectJobItemClaimPriorityTestCase(unittest.TestCase): + def test_ordered_step_claim_codes_prioritize_deeper_steps_first(self): + ordered = _ordered_step_claim_codes() + self.assertLess(ordered.index("detect_360_site"), ordered.index("detect_baidu_site")) + self.assertLess(ordered.index("detect_chinaz"), ordered.index("detect_360_site")) + self.assertGreater(ordered.index("detect_register"), ordered.index("detect_baidu_site")) + + def test_later_pipeline_steps_are_prioritized_ahead_of_earlier_steps(self): + self.assertLess( + _detect_job_item_step_priority("detect_360_site"), + _detect_job_item_step_priority("detect_baidu_site"), + ) + self.assertLess( + _detect_job_item_step_priority("detect_chinaz"), + _detect_job_item_step_priority("detect_360_site"), + ) + self.assertLess( + _detect_job_item_step_priority("detect_aizhan"), + _detect_job_item_step_priority("detect_chinaz"), + ) + + def test_non_register_steps_are_prioritized_ahead_of_register(self): + self.assertLess( + _detect_job_item_step_priority("detect_baidu_site"), + _detect_job_item_step_priority("detect_register"), + ) + self.assertLess( + _detect_job_item_step_priority("detect_wayback"), + _detect_job_item_step_priority("detect_register"), + ) + + def test_blank_step_code_has_lowest_priority(self): + self.assertGreater( + _detect_job_item_step_priority(""), + _detect_job_item_step_priority("detect_register"), + ) + + def test_step_claim_quota_defaults_to_quarter_window_with_floor(self): + self.assertEqual(64, _resolve_step_claim_quota(120)) + self.assertEqual(300, _resolve_step_claim_quota(1200)) + + +if __name__ == "__main__": + unittest.main() diff --git a/domainCheck/tests/test_detect_worker_flush_retry.py b/domainCheck/tests/test_detect_worker_flush_retry.py new file mode 100644 index 0000000..fef0b4a --- /dev/null +++ b/domainCheck/tests/test_detect_worker_flush_retry.py @@ -0,0 +1,225 @@ +import collections +import threading +import unittest +import sys +from unittest.mock import MagicMock +from unittest.mock import patch + + +sys.path.insert(0, "/www/wwwroot/getDomain/domainCheck") + +from detect_worker import DetectWorker # noqa: E402 + + +class _FakeDb: + def __init__(self, method_name: str): + self.method_name = method_name + self.calls = 0 + + def _run(self, batch): + self.calls += 1 + return -1 if self.calls == 1 else len(batch) + + def mark_detect_job_items_running_batch(self, batch): + return self._run(batch) + + def finalize_detect_job_items_batch(self, batch): + return self._run(batch) + + +class DetectWorkerFlushRetryTest(unittest.TestCase): + def _build_worker(self, db): + worker = DetectWorker.__new__(DetectWorker) + worker.db = db + worker._task_local = threading.local() + worker._running_mark_lock = threading.Lock() + worker._pending_running_marks = collections.deque() + worker._last_running_mark_flush_at = 0.0 + worker._job_finalize_lock = threading.Lock() + worker._pending_job_finalizations = collections.deque() + worker._last_job_finalize_flush_at = 0.0 + worker.proxy_pool = [] + worker.proxy_pool_lock = threading.Lock() + worker.proxy_failure_lock = threading.Lock() + worker.proxy_failure_counts = {} + worker.proxy_quarantine_until = {} + worker.proxy_source_by_key = {} + worker.proxy_source_failure_counts = {} + worker.proxy_source_quarantine_until = {} + worker.proxy_config = {"proxy_enable": True, "allow_direct": True} + worker.proxy_last_refresh_status = "test" + worker.trigger_proxy_refresh = MagicMock() + worker.runtime_settings = {} + worker.thread_count = 600 + worker.detecting = True + worker._get_active_domain_threads = MagicMock(return_value=0) + return worker + + def test_running_mark_flush_requeues_batch_when_db_write_fails(self): + worker = self._build_worker(_FakeDb("running")) + worker._pending_running_marks.extend([(101, "token-a"), (102, "token-b")]) + + updated = worker._flush_pending_running_marks(force=True, batch_limit=10) + self.assertEqual(0, updated) + self.assertEqual([(101, "token-a"), (102, "token-b")], list(worker._pending_running_marks)) + + updated = worker._flush_pending_running_marks(force=True, batch_limit=10) + self.assertEqual(2, updated) + self.assertEqual([], list(worker._pending_running_marks)) + + def test_job_finalization_flush_requeues_batch_when_db_write_fails(self): + worker = self._build_worker(_FakeDb("finalize")) + worker._pending_job_finalizations.extend( + [ + {"job_item_id": 201, "claim_token": "token-a", "final_status": "completed"}, + {"job_item_id": 202, "claim_token": "token-b", "final_status": "failed"}, + ] + ) + + updated = worker._flush_pending_job_finalizations(force=True, batch_limit=10) + self.assertEqual(0, updated) + self.assertEqual(2, len(worker._pending_job_finalizations)) + + updated = worker._flush_pending_job_finalizations(force=True, batch_limit=10) + self.assertEqual(2, updated) + self.assertEqual(0, len(worker._pending_job_finalizations)) + + def test_proxy_auth_failure_quarantines_whole_source_group(self): + worker = self._build_worker(_FakeDb("noop")) + bad_proxy = {"http": "http://bad-proxy", "https": "http://bad-proxy"} + good_proxy = {"http": "http://good-proxy", "https": "http://good-proxy"} + worker.proxy_pool = [ + {"proxy": bad_proxy, "usage_count": 0, "source_tag": "D"}, + {"proxy": good_proxy, "usage_count": 0, "source_tag": "E"}, + ] + worker.proxy_source_by_key[worker._proxy_key(bad_proxy)] = "D" + worker.proxy_source_by_key[worker._proxy_key(good_proxy)] = "E" + + source_tag, failure_count, cooldown_seconds, removed = worker._mark_proxy_source_failure( + bad_proxy, + "Tunnel connection failed: 407 Proxy Authentication Required", + ) + + self.assertEqual("D", source_tag) + self.assertEqual(1, failure_count) + self.assertGreaterEqual(cooldown_seconds, 90) + self.assertEqual(1, removed) + self.assertTrue(worker._is_proxy_source_quarantined("D")) + self.assertEqual( + [{"proxy": good_proxy, "usage_count": 0, "source_tag": "E"}], + worker.proxy_pool, + ) + + def test_prepare_proxy_fetch_plan_respects_pool_shortage_instead_of_overfetching(self): + worker = self._build_worker(_FakeDb("noop")) + + plan, meta = worker._prepare_proxy_fetch_plan( + [ + "http://provider.local/get?group=B", + "http://provider.local/get?group=C", + "http://provider.local/get?group=D", + "http://provider.local/get?group=E", + "http://provider.local/get?group=F", + "http://provider.local/get?group=G", + ], + current_pool_size=1000, + ) + + self.assertEqual(1, meta["rounds"]) + self.assertGreaterEqual(meta["target_total"], 840) + self.assertLessEqual(meta["target_total"], 1200) + self.assertEqual(6, len(plan)) + + def test_prepare_proxy_fetch_plan_caps_active_rounds(self): + worker = self._build_worker(_FakeDb("noop")) + + plan, meta = worker._prepare_proxy_fetch_plan( + [ + "http://provider.local/get?group=B", + "http://provider.local/get?group=C", + "http://provider.local/get?group=D", + "http://provider.local/get?group=E", + "http://provider.local/get?group=F", + "http://provider.local/get?group=G", + ], + current_pool_size=0, + ) + + self.assertLessEqual(meta["rounds"], 2) + self.assertEqual(meta["rounds"] * 6, len(plan)) + + def test_submit_backlog_limit_defaults_to_thread_count_window(self): + worker = self._build_worker(_FakeDb("noop")) + + self.assertEqual(2000, worker._resolve_submit_backlog_limit(2000, 0)) + self.assertEqual(2000, worker._resolve_submit_backlog_limit(2000, 1200)) + + def test_dispatch_capacity_can_ramp_to_full_thread_limit(self): + worker = self._build_worker(_FakeDb("noop")) + + self.assertEqual(2000, worker._resolve_dispatch_capacity(2000, 0, 2000)) + self.assertEqual(2000, worker._resolve_dispatch_capacity(2000, 1000, 1000)) + + def test_claim_batch_size_defaults_to_thread_count_window(self): + worker = self._build_worker(_FakeDb("noop")) + + self.assertEqual(2000, worker._resolve_claim_batch_size(2000, 2000, live_active=0)) + self.assertEqual(1500, worker._resolve_claim_batch_size(2000, 1500, live_active=800)) + + def test_get_proxy_for_step_can_force_single_direct_fallback_once(self): + worker = self._build_worker(_FakeDb("noop")) + worker._set_current_task_context(domain_id=1, is_step_task=True, task_mode="single_step") + worker._set_step_force_direct_once("百度site检测", True) + + proxy = worker._get_proxy_for_step(1, "example.com", "百度site检测") + + self.assertIsNone(proxy) + self.assertFalse(worker._consume_step_force_direct_once("百度site检测")) + + def test_retry_external_issue_with_proxy_forces_next_attempt_direct_when_allowed(self): + worker = self._build_worker(_FakeDb("noop")) + worker._set_current_task_context(domain_id=1, is_step_task=True, task_mode="single_step") + worker.remove_proxy = MagicMock() + proxy = {"http": "http://bad-proxy", "https": "http://bad-proxy"} + + should_retry = worker._should_retry_external_issue( + proxy, + "curl: (28) Connection timed out after 552 milliseconds", + "百度site检测", + "example.com", + ) + + self.assertTrue(should_retry) + worker.remove_proxy.assert_called_once_with(proxy) + worker.trigger_proxy_refresh.assert_called_once() + self.assertTrue(worker._consume_step_force_direct_once("百度site检测")) + + def test_external_dependency_issue_recognizes_common_chinese_timeout_messages(self): + worker = self._build_worker(_FakeDb("noop")) + + self.assertTrue(worker._is_external_dependency_issue("360搜索请求超时")) + self.assertTrue(worker._is_external_dependency_issue("百度site检测失败: 连接超时")) + self.assertTrue(worker._is_external_dependency_issue("站长之家检测失败: 网络异常")) + + def test_run_detect_aizhan_passes_remaining_budget_to_detector(self): + worker = self._build_worker(_FakeDb("noop")) + worker._check_proxy_step_retry_budget = MagicMock(return_value=(False, "")) + worker._remaining_proxy_step_retry_budget_seconds = MagicMock(return_value=1.7) + worker._get_proxy_for_step = MagicMock(return_value=None) + worker._mark_detection_degraded = MagicMock() + worker._should_blacklist_result = MagicMock(return_value=False) + worker._record_step_result = MagicMock() + worker._should_rotate_proxy_on_error = MagicMock(return_value=False) + worker._is_external_dependency_issue = MagicMock(return_value=False) + worker._mark_detection_rejected = MagicMock() + worker._mark_detection_failed = MagicMock() + + with patch("detect_worker.aizhan.check_aizhan", return_value=(True, "")) as mock_check: + result = worker._run_detect_aizhan(1, "example.com", []) + + self.assertTrue(result) + self.assertEqual(1.7, mock_check.call_args.kwargs["budget_seconds"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/domainCheck/tests/test_register_timeout_config.py b/domainCheck/tests/test_register_timeout_config.py new file mode 100644 index 0000000..ac1ad87 --- /dev/null +++ b/domainCheck/tests/test_register_timeout_config.py @@ -0,0 +1,64 @@ +import unittest +from unittest.mock import MagicMock, patch + +import urllib3 + +from domainCheck.detect import register + + +class RegisterTimeoutConfigTests(unittest.TestCase): + def test_proxy_timeout_uses_bounded_total(self): + with patch.dict("os.environ", {}, clear=False): + timeout = register._resolve_register_timeout({"http": "http://127.0.0.1:8080"}) + self.assertIsInstance(timeout, urllib3.Timeout) + self.assertEqual(timeout.connect_timeout, 1.0) + self.assertEqual(timeout.read_timeout, 1.5) + self.assertEqual(timeout.total, 2.2) + + def test_direct_timeout_respects_override(self): + with patch.dict( + "os.environ", + { + "DOMAINCHECK_REGISTER_DIRECT_CONNECT_TIMEOUT": "0.9", + "DOMAINCHECK_REGISTER_DIRECT_READ_TIMEOUT": "1.4", + "DOMAINCHECK_REGISTER_DIRECT_TOTAL_TIMEOUT": "1.7", + }, + clear=False, + ): + timeout = register._resolve_register_timeout(None) + self.assertEqual(timeout.connect_timeout, 0.9) + self.assertEqual(timeout.read_timeout, 1.4) + self.assertEqual(timeout.total, 1.7) + + def test_budget_clamps_proxy_timeout_into_remaining_window(self): + with patch.dict("os.environ", {}, clear=False): + timeout = register._resolve_register_timeout( + {"http": "http://127.0.0.1:8080"}, + budget_seconds=0.8, + ) + self.assertEqual(timeout.total, 0.8) + self.assertLessEqual(timeout.connect_timeout, 0.35) + self.assertLessEqual(timeout.read_timeout, 0.61) + + def test_request_register_converts_urllib3_timeout_for_requests(self): + session = MagicMock() + response = MagicMock() + session.get.return_value = response + timeout = urllib3.Timeout(connect=1.1, read=1.7, total=2.4) + with patch("domainCheck.detect.register._get_http_manager", return_value=session): + result = register._request_register( + "https://rdap.verisign.com/com/v1/domain/example", + proxies={"http": "http://127.0.0.1:8080", "https": "http://127.0.0.1:8080"}, + timeout=timeout, + ) + self.assertIs(result, response) + session.get.assert_called_once_with( + "https://rdap.verisign.com/com/v1/domain/example", + timeout=(1.1, 1.7), + proxies={"http": "http://127.0.0.1:8080", "https": "http://127.0.0.1:8080"}, + allow_redirects=True, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/domainCheck/tests/test_step_timeout_budgets.py b/domainCheck/tests/test_step_timeout_budgets.py new file mode 100644 index 0000000..7f778f2 --- /dev/null +++ b/domainCheck/tests/test_step_timeout_budgets.py @@ -0,0 +1,33 @@ +import unittest +from unittest.mock import patch + +from domainCheck.detect import aizhan, baidu, c360, chinaz + + +class StepTimeoutBudgetTests(unittest.TestCase): + def test_baidu_timeout_respects_remaining_budget(self): + with patch.dict("os.environ", {}, clear=False): + timeout = baidu._resolve_baidu_timeout({"http": "http://127.0.0.1:8080"}, budget_seconds=1.1) + self.assertLessEqual(timeout, 1.1) + self.assertGreaterEqual(timeout, 0.6) + + def test_360_timeout_respects_remaining_budget(self): + with patch.dict("os.environ", {}, clear=False): + timeout = c360._resolve_360_timeout({"http": "http://127.0.0.1:8080"}, budget_seconds=0.9) + self.assertLessEqual(timeout, 0.9) + self.assertGreaterEqual(timeout, 0.6) + + def test_chinaz_timeout_respects_remaining_budget(self): + timeout = chinaz._resolve_chinaz_timeout({"http": "http://127.0.0.1:8080"}, budget_seconds=1.3) + self.assertLessEqual(timeout, 1.3) + self.assertGreaterEqual(timeout, 0.6) + + def test_aizhan_timeout_respects_remaining_budget(self): + with patch.dict("os.environ", {}, clear=False): + timeout = aizhan._resolve_aizhan_timeout({"http": "http://127.0.0.1:8080"}, budget_seconds=1.0) + self.assertLessEqual(timeout, 1.0) + self.assertGreaterEqual(timeout, 0.6) + + +if __name__ == "__main__": + unittest.main() diff --git a/domainCheck/tests/test_wayback_detector_recent_years.py b/domainCheck/tests/test_wayback_detector_recent_years.py new file mode 100644 index 0000000..5c5ef62 --- /dev/null +++ b/domainCheck/tests/test_wayback_detector_recent_years.py @@ -0,0 +1,173 @@ +import sys +import time +import unittest +from unittest.mock import patch + + +sys.path.insert(0, "/www/wwwroot/getDomain/domainCheck") + +from app.detectors.wayback_detector import WaybackDetector # noqa: E402 + + +class WaybackDetectorRecentYearsTest(unittest.TestCase): + def setUp(self): + WaybackDetector._transient_backoff_until = 0.0 + + def test_filter_records_recent_years_drops_old_snapshots(self): + detector = WaybackDetector.__new__(WaybackDetector) + current_year = time.gmtime().tm_year + records = [ + {"timestamp": f"{current_year}0101000000", "digest": "new"}, + {"timestamp": f"{current_year - 2}0101000000", "digest": "mid"}, + {"timestamp": f"{current_year - 6}0101000000", "digest": "old"}, + ] + + filtered = detector._filter_records_recent_years(records, recent_years=5) + + self.assertEqual( + [ + f"{current_year}0101000000", + f"{current_year - 2}0101000000", + ], + [item["timestamp"] for item in filtered], + ) + + def test_scan_snapshots_only_counts_recent_years_window(self): + detector = WaybackDetector.__new__(WaybackDetector) + current_year = time.gmtime().tm_year + detector._fetch_cdx_records_with_meta = lambda domain, limit=None, fast_latest=False: { + "records": [ + {"timestamp": f"{current_year}0101000000", "digest": "latest"}, + {"timestamp": f"{current_year - 1}0101000000", "digest": "prev"}, + {"timestamp": f"{current_year - 6}0101000000", "digest": "old"}, + ] + } + detector._load_cached_records = lambda domain: None + detector._save_cached_records = lambda domain, records: None + detector._save_cached_timestamps = lambda domain, timestamps: None + detector._fetch_snapshot_title = lambda domain, timestamp: { + "timestamp": timestamp, + "title": f"title-{timestamp}", + "ok": True, + } + detector._normalize_title = lambda title: title + detector._find_sensitive_word = lambda title, words: None + detector._log_info = lambda message: None + detector._handle_exception = lambda exc, domain: None + + result = detector.scan_snapshots("example.com", sensitive_words=[], recent_years=5) + + self.assertEqual(2, result["checked_snapshot_count"]) + self.assertEqual(2, result["fetched_snapshot_count"]) + self.assertEqual(sorted([current_year - 1, current_year]), result["snapshot_years"]) + + def test_scan_snapshots_limits_records_per_domain(self): + detector = WaybackDetector.__new__(WaybackDetector) + current_year = time.gmtime().tm_year + records = [ + {"timestamp": f"{current_year}01010{i}0000", "digest": f"d{i}"} + for i in range(6) + ] + detector._fetch_cdx_records_with_meta = lambda domain, limit=None, fast_latest=False: { + "records": records, + "error": None, + } + detector._load_cached_records = lambda domain: None + detector._save_cached_records = lambda domain, records: None + detector._save_cached_timestamps = lambda domain, timestamps: None + detector._fetch_snapshot_title = lambda domain, timestamp: { + "timestamp": timestamp, + "title": f"title-{timestamp}", + "ok": True, + } + detector._normalize_title = lambda title: title + detector._find_sensitive_word = lambda title, words: None + detector._log_info = lambda message: None + detector._handle_exception = lambda exc, domain: None + + with patch("app.detectors.wayback_detector.config.WAYBACK_MAX_RECORDS", 3): + result = detector.scan_snapshots("example.com", sensitive_words=[], recent_years=5) + + self.assertEqual(3, result["checked_snapshot_count"]) + # 最新快照会先独立尝试一次,再进入裁剪后的扫描窗口。 + self.assertEqual(4, result["fetched_snapshot_count"]) + + def test_scan_snapshots_fast_degrades_when_latest_cdx_is_transient_failure(self): + detector = WaybackDetector.__new__(WaybackDetector) + detector._fetch_cdx_records_with_meta = lambda domain, limit=None, fast_latest=False: { + "records": [], + "error": "HTTPSConnectionPool(host='web.archive.org', port=443): Max retries exceeded", + } + detector._load_cached_records = lambda domain: [ + {"timestamp": "20260101000000", "digest": "d1"}, + {"timestamp": "20250101000000", "digest": "d2"}, + ] + detector._save_cached_records = lambda domain, records: None + detector._save_cached_timestamps = lambda domain, timestamps: None + detector._fetch_snapshot_title = lambda domain, timestamp: self.fail("should not fetch snapshot titles") + detector._normalize_title = lambda title: title + detector._find_sensitive_word = lambda title, words: None + detector._log_info = lambda message: None + detector._handle_exception = lambda exc, domain: None + + result = detector.scan_snapshots("example.com", sensitive_words=[], recent_years=5) + + self.assertEqual(0, result["checked_snapshot_count"]) + self.assertGreaterEqual(result["failed_snapshot_count"], 1) + self.assertGreaterEqual(result["request_error_count"], 1) + + def test_scan_snapshots_fast_degrades_when_latest_snapshot_is_transient_failure(self): + detector = WaybackDetector.__new__(WaybackDetector) + detector._fetch_cdx_records_with_meta = lambda domain, limit=None, fast_latest=False: { + "records": [{"timestamp": "20260101000000", "digest": "latest"}] if fast_latest else [ + {"timestamp": "20260101000000", "digest": "latest"}, + {"timestamp": "20250101000000", "digest": "older"}, + ], + "error": None, + } + detector._load_cached_records = lambda domain: None + detector._save_cached_records = lambda domain, records: None + detector._save_cached_timestamps = lambda domain, timestamps: None + detector._fetch_snapshot_title = lambda domain, timestamp: { + "timestamp": timestamp, + "title": "", + "ok": False, + "error": "HTTPSConnectionPool(host='web.archive.org', port=443): Max retries exceeded", + } + detector._normalize_title = lambda title: title + detector._find_sensitive_word = lambda title, words: None + detector._log_info = lambda message: None + detector._handle_exception = lambda exc, domain: None + + result = detector.scan_snapshots("example.com", sensitive_words=[], recent_years=5) + + self.assertEqual(1, result["checked_snapshot_count"]) + self.assertEqual(0, result["fetched_snapshot_count"]) + self.assertGreaterEqual(result["failed_snapshot_count"], 1) + self.assertTrue( + any("latest_snapshot:" in item for item in result["request_errors"]) + ) + + def test_scan_snapshots_short_circuits_when_transient_backoff_active(self): + detector = WaybackDetector.__new__(WaybackDetector) + WaybackDetector._transient_backoff_until = time.time() + 5 + detector._fetch_cdx_records_with_meta = lambda *args, **kwargs: self.fail("should not request cdx during backoff") + detector._load_cached_records = lambda domain: None + detector._save_cached_records = lambda domain, records: None + detector._save_cached_timestamps = lambda domain, timestamps: None + detector._fetch_snapshot_title = lambda domain, timestamp: self.fail("should not request snapshot during backoff") + detector._normalize_title = lambda title: title + detector._find_sensitive_word = lambda title, words: None + detector._log_info = lambda message: None + detector._handle_exception = lambda exc, domain: None + + result = detector.scan_snapshots("example.com", sensitive_words=[], recent_years=5) + + self.assertEqual(0, result["checked_snapshot_count"]) + self.assertEqual(0, result["fetched_snapshot_count"]) + self.assertEqual(1, result["request_error_count"]) + self.assertTrue(any("wayback_backoff_active:" in item for item in result["request_errors"])) + + +if __name__ == "__main__": + unittest.main() diff --git a/domainCheck/thread_count.json b/domainCheck/thread_count.json index aab0c6b..0ba7557 100644 --- a/domainCheck/thread_count.json +++ b/domainCheck/thread_count.json @@ -1,3 +1,3 @@ { - "thread_count": "5" + "thread_count": "100" } \ No newline at end of file diff --git a/package_domain_release.ps1 b/package_domain_release.ps1 index cdd122c..406f333 100644 --- a/package_domain_release.ps1 +++ b/package_domain_release.ps1 @@ -125,6 +125,40 @@ foreach ($item in $webItems) { } Copy-OptionalItem -Source (Join-Path $root "domain-web\dist") -Destination $webTarget +$domainCheckTarget = Join-Path $stagingRoot "domainCheck" +Copy-OptionalItem -Source (Join-Path $root "domainCheck") -Destination $stagingRoot +if (Test-Path -LiteralPath $domainCheckTarget) { + $excludedDomainCheckPaths = @( + ".venv", + ".pytest_cache", + "__pycache__", + "logs", + "data", + "runtime", + ".env", + "credentials.json", + "juming_cookies.pkl", + "detect_options.json", + "proxy_config.json", + "thread_count.json", + "node_thread_counts.json", + "runtime_settings.json", + "detect_worker.log" + ) + foreach ($relativePath in $excludedDomainCheckPaths) { + $targetPath = Join-Path $domainCheckTarget $relativePath + if (Test-Path -LiteralPath $targetPath) { + Remove-Item -LiteralPath $targetPath -Recurse -Force + } + } + Get-ChildItem -Path $domainCheckTarget -Directory -Recurse -Force -Filter "__pycache__" | ForEach-Object { + Remove-Item -LiteralPath $_.FullName -Recurse -Force + } + Get-ChildItem -Path $domainCheckTarget -File -Recurse -Force -Include "*.pyc", "detect_worker*.log" | ForEach-Object { + Remove-Item -LiteralPath $_.FullName -Force + } +} + $smokeOk = $false $smokeMessage = "" if ((Test-Path -LiteralPath $smokeScript) -and (Test-Path -LiteralPath $smokeRunner)) { @@ -185,6 +219,7 @@ $manifest = [ordered]@{ scripts = Get-ChildItem -Path $scriptsTarget -File | Select-Object -ExpandProperty Name domain_api = Get-ChildItem -Path $apiTarget | Select-Object -ExpandProperty Name domain_web = Get-ChildItem -Path $webTarget | Select-Object -ExpandProperty Name + domain_check = if (Test-Path -LiteralPath $domainCheckTarget) { Get-ChildItem -Path $domainCheckTarget | Select-Object -ExpandProperty Name } else { @() } } } @@ -201,6 +236,7 @@ $readmePath = Join-Path $stagingRoot "README_RELEASE.txt" "- scripts", "- domain-api", "- domain-web", + "- domainCheck", "- release_manifest.json", "- smoke_test_report.json (if generated)", "", diff --git a/package_domain_release.sh b/package_domain_release.sh index f0cf50b..b359880 100755 --- a/package_domain_release.sh +++ b/package_domain_release.sh @@ -146,6 +146,30 @@ for item in "${WEB_ITEMS[@]}"; do done copy_optional_item "${ROOT_DIR}/domain-web/dist" "${WEB_TARGET}/" +DOMAINCHECK_TARGET="${STAGING_ROOT}/domainCheck" +copy_optional_item "${ROOT_DIR}/domainCheck" "${STAGING_ROOT}/" +if [[ -d "${DOMAINCHECK_TARGET}" ]]; then + rm -rf \ + "${DOMAINCHECK_TARGET}/.venv" \ + "${DOMAINCHECK_TARGET}/.pytest_cache" \ + "${DOMAINCHECK_TARGET}/__pycache__" \ + "${DOMAINCHECK_TARGET}/logs" \ + "${DOMAINCHECK_TARGET}/data" \ + "${DOMAINCHECK_TARGET}/runtime" + rm -f \ + "${DOMAINCHECK_TARGET}/.env" \ + "${DOMAINCHECK_TARGET}/credentials.json" \ + "${DOMAINCHECK_TARGET}/juming_cookies.pkl" \ + "${DOMAINCHECK_TARGET}/detect_options.json" \ + "${DOMAINCHECK_TARGET}/proxy_config.json" \ + "${DOMAINCHECK_TARGET}/thread_count.json" \ + "${DOMAINCHECK_TARGET}/node_thread_counts.json" \ + "${DOMAINCHECK_TARGET}/runtime_settings.json" \ + "${DOMAINCHECK_TARGET}/detect_worker.log" + find "${DOMAINCHECK_TARGET}" -type d -name "__pycache__" -prune -exec rm -rf {} + + find "${DOMAINCHECK_TARGET}" -type f \( -name "*.pyc" -o -name "detect_worker*.log" \) -delete +fi + SMOKE_OK="false" SMOKE_MESSAGE="" if [[ "${SKIP_SMOKE_TEST}" == "1" ]]; then @@ -213,6 +237,7 @@ manifest = { "scripts": sorted(item.name for item in scripts_target.iterdir() if item.is_file()) if scripts_target.exists() else [], "domain_api": sorted(item.name for item in api_target.iterdir()) if api_target.exists() else [], "domain_web": sorted(item.name for item in web_target.iterdir()) if web_target.exists() else [], + "domain_check": sorted(item.name for item in (staging_root / "domainCheck").iterdir()) if (staging_root / "domainCheck").exists() else [], }, } @@ -231,6 +256,7 @@ Contents: - scripts - domain-api - domain-web +- domainCheck - release_manifest.json - smoke_test_report.json (if generated) diff --git a/verify_domain_release.ps1 b/verify_domain_release.ps1 index d32639a..6662ff7 100644 --- a/verify_domain_release.ps1 +++ b/verify_domain_release.ps1 @@ -66,6 +66,10 @@ try { "release_manifest.json", "domain-api/README.md", "domain-web/README.md", + "domainCheck/detect_worker.py", + "domainCheck/app/config.py", + "domainCheck/detect/register.py", + "domainCheck/.env.example", "scripts/package_domain_release.ps1", "scripts/verify_domain_release.ps1", "scripts/package_domain_release.sh", diff --git a/verify_domain_release.sh b/verify_domain_release.sh index af9e2bb..85d8b47 100755 --- a/verify_domain_release.sh +++ b/verify_domain_release.sh @@ -116,6 +116,10 @@ required_entries = [ "release_manifest.json", "domain-api/README.md", "domain-web/README.md", + "domainCheck/detect_worker.py", + "domainCheck/app/config.py", + "domainCheck/detect/register.py", + "domainCheck/.env.example", "scripts/package_domain_release.ps1", "scripts/verify_domain_release.ps1", "scripts/package_domain_release.sh",