d
This commit is contained in:
Your Name
287
docs/base.md
Normal file
287
docs/base.md
Normal file
@@ -0,0 +1,287 @@
|
||||
# 主线任务验收总表
|
||||
|
||||
更新时间：2026-04-21
|
||||
|
||||
## 当前执行原则
|
||||
|
||||
当前先只做一件事：
|
||||
|
||||
- 跑通主任务流程
|
||||
- 把 7 个大任务按“可测试、可验收”的标准推进
|
||||
- 并发、代理、展示层、CPU 打满这类细节优化先封存，不再抢主线
|
||||
|
||||
当前不再把“性能抠点”当主目标。
|
||||
后续额度恢复后，再继续做细节优化专项。
|
||||
|
||||
## 当前主线优先级
|
||||
|
||||
按下面顺序推进，不跳步：
|
||||
|
||||
1. 大任务 1 真实验收闭环
|
||||
2. 大任务 2 controller 编排闭环
|
||||
3. 大任务 3 统一结果状态机闭环
|
||||
4. 大任务 4 本地控制状态 + syncer/finalizer 闭环
|
||||
5. 大任务 5 固定 worker pool + 持续补位替换旧模型
|
||||
6. 大任务 6 时光机一期接入标准步骤
|
||||
7. 大任务 7 运营视角指标面板落地
|
||||
|
||||
## 7 个大任务当前状态
|
||||
|
||||
### 大任务 1
|
||||
单步骤任务底座落地
|
||||
|
||||
目标：
|
||||
先打通 `controller -> queue -> claim -> worker -> result -> controller` 的完整闭环。
|
||||
|
||||
当前状态：
|
||||
- 已有较完整代码底座
|
||||
- `single_step` / `step_code` 模型已基本成形
|
||||
- 现在缺的是“真实运行验收”而不是继续堆骨架
|
||||
|
||||
本任务验收只认下面 5 件事：
|
||||
|
||||
1. controller 能生成 `baidu_check` 任务
|
||||
2. worker 能拉到 `baidu_check` 任务并执行
|
||||
3. worker 能回传标准化结果
|
||||
4. controller 能更新本地步骤状态
|
||||
5. worker 超时未回传时，任务会回收重投
|
||||
|
||||
当前结论：
|
||||
- 代码方向已基本对齐
|
||||
- 2026-04-20 已完成一轮 live 验收打钩：
|
||||
- `controller` 已生成 `single_step / detect_baidu_site` 任务（例：`job_id=6`, `job_code=step-20260420153527-9fc7b5`）
|
||||
- `worker` 已精准领取当前 `job_id` 并执行，不再被旧 `pipeline / sync / legacy fallback` 污染
|
||||
- `worker` 已回传标准化结果到 `detect_job_items.result_payload_json`
|
||||
- `controller` 已可按 `job_id` 定向执行 `process_pipeline`，并把结果落到本地 `domain_detections.baidu_site`
|
||||
- “超时/遗留未回传”链路已验证会被节点启动释放并重新领取执行（`job_id=5` 在遗留 `running` 后被重新释放、重新领取并完成）
|
||||
- 当前主线结论：大任务 1 已达到“可验收通过”状态，可以继续把主精力切到大任务 2 / 3
|
||||
|
||||
### 大任务 2
|
||||
pipeline 编排器落地
|
||||
|
||||
目标：
|
||||
把“下一步跑什么”彻底收回 controller。
|
||||
|
||||
当前状态：
|
||||
- 基础方向已落到 controller 驱动
|
||||
- 还需要继续做“按后台勾选顺序推进 / 按域名属性跳过”的真实验收
|
||||
|
||||
本任务必须确认：
|
||||
- 同一个域名不会在 worker 里串完整条链
|
||||
- controller 是唯一推进者
|
||||
- 勾选顺序变化能影响下一步投递
|
||||
- 跳过规则生效
|
||||
|
||||
当前结论：
|
||||
- 已进入可验收阶段
|
||||
- 2026-04-20 已补上关键收口：
|
||||
- `single_step` 会话下，worker 只按当前 `job_id` 领取任务
|
||||
- `single_step` 会话下，旧的 `pipeline 推进 / sync pull / legacy fallback` 已被显式跳过
|
||||
- `process_pipeline` 已支持按 `job_id` 定向处理，方便 controller 精准推进当前步骤
|
||||
- 2026-04-20 又补完一轮 live 验收：
|
||||
- `pass`：`detect_baidu_site` 完成后，controller 已为同一 `job` 创建下一步 `detect_360_site`
|
||||
- `skip`：一口价域名在 `detect_register + detect_baidu_site` 配置下，会直接解析到 `detect_baidu_site`
|
||||
- 同时修掉了一个真实阻塞点：`process_pipeline()` 事务内再开第二连接写 `detect_run_events` 会把 controller 自己锁住；现已改成同事务同 cursor 写事件
|
||||
- 当前主线结论：大任务 2 的 controller 编排主干已可验收，剩下更多是扩步骤和补更细跳过规则
|
||||
|
||||
### 大任务 3
|
||||
步骤结果判定与重试策略落地
|
||||
|
||||
目标：
|
||||
统一 `pass / retry / black_hit / reject`。
|
||||
|
||||
当前状态：
|
||||
- 结果结构、重试入口、黑名单终止方向已基本进入主链
|
||||
- 还需要补 live 验收，重点是 TTL 回收、重投、终止规则
|
||||
|
||||
本任务必须确认：
|
||||
- 外部失败会重投当前步骤
|
||||
- 黑名单命中会终止后续步骤
|
||||
- 非黑名单业务不通过按规则终止
|
||||
- 不会出现同一步无限重试
|
||||
|
||||
当前结论：
|
||||
- 代码已明显推进
|
||||
- 2026-04-20 已完成 controller 侧 live 验收：
|
||||
- `retry`：`state=degraded` 会重投当前步骤，不推进下一步
|
||||
- `black_hit`：`state=blacklisted` 会终止后续步骤并结束当前 job
|
||||
- `reject`：`state=rejected` 会终止当前流程，不再重试
|
||||
- worker 侧也已补上业务失败 -> `rejected` 的结果态，不再把业务不通过和技术失败都混成 `failed`
|
||||
- 当前主线结论：大任务 3 的统一结果状态机已基本闭环，可继续往大任务 4 / 5 推进
|
||||
|
||||
### 大任务 4
|
||||
本地控制状态与海外主库同步落地
|
||||
|
||||
目标：
|
||||
controller 本地维护高频状态，syncer 批量同步海外主库，finalizer 标记流程完成。
|
||||
|
||||
当前状态：
|
||||
- 本地状态和部分同步链路已经在跑
|
||||
- 已补上“最近完成任务快照也继续产出结果投影”
|
||||
- 已补上结果导入后按 job_code 优先定位本地 job，并刷新本地 job 收尾状态
|
||||
- syncer/finalizer 还缺少一轮 live 闭环验收
|
||||
|
||||
本任务必须确认：
|
||||
- 高频链路不依赖 worker 频繁直写海外主库
|
||||
- controller 本地状态完整
|
||||
- 批量同步成功
|
||||
- 流程完成状态准确
|
||||
|
||||
当前结论：
|
||||
- 代码主链已进一步收口
|
||||
- 2026-04-21 已确认 `detect_result_projection / runtime_projection` 在 `detect_sync_records` 中持续产出且状态为 `projected`
|
||||
- 当前说明 syncer 主链已恢复，但“本地 job 收尾状态 + 主库最终账本完全一致”的终验还需要继续盯现场
|
||||
|
||||
### 大任务 5
|
||||
worker 池化与持续补位调度落地
|
||||
|
||||
目标：
|
||||
真正替掉“批量认领 + 批量等待”的旧模型。
|
||||
|
||||
当前状态：
|
||||
- 已经做了多轮并发热路径优化
|
||||
- 已进一步压缩 executor 内部 backlog，补位更接近固定槽位模型
|
||||
- 但还需要一轮 live 运行观察 claimed/running 曲线，确认旧的批量认领惯性已被压住
|
||||
|
||||
本任务必须确认：
|
||||
- 活跃槽位稳定贴近配置上限
|
||||
- 并发不再大起大落
|
||||
- 吞吐明显提升
|
||||
- 调度器自耗下降
|
||||
|
||||
当前结论：
|
||||
- 这是主线里仍然偏重的未完项
|
||||
- 2026-04-21 已继续做现场修正：
|
||||
- `claim_detect_job_items()` 已显式排除空 `step_code` 的 legacy 项，避免 worker 从标准队列入口继续误吞旧 whole-domain 项
|
||||
- `mainland-controller-01 / 121.204.244.188` 与 `mainland-worker-01 / 121.204.244.248` 均已确认在真大陆节点参与执行
|
||||
- 海外控制面 `queue_health / dashboard` 已补上“读取大陆 runtime/debug 近窗执行流”的兜底口径
|
||||
- 当前首页已能看到真实近窗吞吐，例如 `10-15 项/分钟` 量级、并能拆到 controller/worker 两个节点
|
||||
- 当前主线判断：
|
||||
- “高 claimed 假活跃”问题已继续缓解
|
||||
- “活吞吐不可见”问题已明显改善
|
||||
- 但“总 completed 账本持续增长”和“running 贴近线程上限”仍未彻底验收，所以大任务 5 仍未签字通过
|
||||
|
||||
### 大任务 6
|
||||
时光机一期流程落地
|
||||
|
||||
目标：
|
||||
时光机按“最近 5 年”方案接成标准步骤任务。
|
||||
|
||||
当前状态：
|
||||
- 现有项目里已有时光机相关检测逻辑
|
||||
- 但还没有完全按标准步骤任务方式接进新 pipeline 验收
|
||||
|
||||
本任务必须确认：
|
||||
- wayback 作为标准步骤任务接入
|
||||
- 最近 5 年快照策略稳定
|
||||
- 返回标准化结果
|
||||
- controller 能把它当普通步骤推进/终止
|
||||
|
||||
当前结论：
|
||||
- 2026-04-21 已落代码并通过测试：
|
||||
- `detect_wayback` 已作为标准 `single_step` 步骤接入 controller / worker 主链
|
||||
- 时光机一期已按“最近 5 年 + 命中即停”策略落地到 payload 和 detector
|
||||
- 焦点测试已通过
|
||||
- 但还缺 live 现场验收：
|
||||
- 真实任务投递
|
||||
- worker 执行
|
||||
- 标准化结果回传
|
||||
- controller 按普通步骤推进/终止
|
||||
- 当前主线结论：大任务 6 已进入“代码落地完成、待现场验收”状态
|
||||
|
||||
### 大任务 7
|
||||
运营视角指标与验收面板落地
|
||||
|
||||
目标：
|
||||
让后台能判断“有没有跑起来、卡在哪一步、多久跑完”。
|
||||
|
||||
当前状态：
|
||||
- 后端已有部分 runtime / detect status / debug event 统计基础
|
||||
- 但完整的运营视角指标面板还没有完全落地验收
|
||||
|
||||
本任务必须确认：
|
||||
- 每步骤队列数
|
||||
- 每步骤吞吐
|
||||
- 每分钟完成量
|
||||
- 当前 pipeline 分布
|
||||
- 重试数 / 黑名单数 / 失败数
|
||||
- 预估剩余时间
|
||||
|
||||
当前结论：
|
||||
- 2026-04-21 已落地首页最小运营面板，并补了第二层口径修正：
|
||||
- `completed / pending / running` 已切到“全活跃任务累计口径”
|
||||
- `步骤队列 / 节点吞吐 / 重试压力 / 有效执行节点` 已可直接展示
|
||||
- `ETA` 在无真实近窗完成量时会显示“待计算”，避免误报 0 小时
|
||||
- 2026-04-21 又补上“大陆 runtime/debug 近窗执行流”兜底：
|
||||
- `active_job` 已可显示 `runtime_job_code`
|
||||
- `processed_per_minute / completed_recent / node_throughput / step_queue` 已能贴近大陆真实执行
|
||||
- 首页已能直接区分 `db_job_code` 与 `runtime_job_code`
|
||||
- 当前结论：大任务 7 已进入“可运营分析并能指导现场排障”阶段，但仍需继续把“总 completed 账本”和“最终验收面板”完全统一
|
||||
|
||||
## 现在只做什么
|
||||
|
||||
当前只盯主线，不跑偏：
|
||||
|
||||
1. 把大任务 1 做成可真实验收
|
||||
2. 验收过后立刻推进大任务 2
|
||||
3. 再按顺序推进 3、4、5、6、7
|
||||
|
||||
如果某个问题只是：
|
||||
|
||||
- CPU 没吃满
|
||||
- 某个 timeout 还能再抠
|
||||
- 代理池还能更激进
|
||||
- 页面还能再改得更好看
|
||||
|
||||
都先不打断主线。
|
||||
|
||||
## 已封存的细节优化 backlog
|
||||
|
||||
下面这些不是不做，而是先封存：
|
||||
|
||||
### A. 并发/调度优化
|
||||
|
||||
- 固定 worker pool 彻底替换旧批量认领模型
|
||||
- claimed 回弹继续压缩
|
||||
- executor 内部排队继续收紧
|
||||
- 活跃 running 继续往上抬
|
||||
- controller / worker 双节点吞吐平衡
|
||||
|
||||
### B. DB 往返优化
|
||||
|
||||
- 继续合并 `domains` 表高频更新
|
||||
- 继续减少 `mark_running / finalize` 这类必要写库点开销
|
||||
- 能走 Redis 或本地状态的尽量不走高频 DB
|
||||
|
||||
### C. 外部请求链路优化
|
||||
|
||||
- 代理失败后的重试链继续压缩
|
||||
- 直连失败后回代理的等待窗口再收紧
|
||||
- 无代理窗口等待策略再优化
|
||||
- 外部请求 timeout 再按真实成功率调优
|
||||
|
||||
### D. 代理池策略优化
|
||||
|
||||
- 代理池刷新频率与补位策略继续增强
|
||||
- 失败代理淘汰与新代理拉取节奏继续优化
|
||||
- 控制“不过度预验证”和“不过度浪费线程”之间的平衡
|
||||
|
||||
### E. 运营展示层优化
|
||||
|
||||
- 更细的 runtime 面板
|
||||
- ETA / 每分钟吞吐更精细展示
|
||||
- 日志窗口布局继续优化
|
||||
- 非主线的页面交互增强
|
||||
|
||||
## 接下来执行口径
|
||||
|
||||
接下来统一按这个口径推进：
|
||||
|
||||
- 先验收主流程
|
||||
- 再补主流程缺口
|
||||
- 性能细节先记账，不抢主线
|
||||
- 每做完一个大任务，就给出“是否验收通过”的明确结论
|
||||
|
||||
一句话定调：
|
||||
|
||||
当前阶段不是“继续无限抠并发”，而是“先把 7 个大任务按主线流程逐个打通并验收”。
|
||||
49
docs/codex接手.md
Normal file
49
docs/codex接手.md
Normal file
@@ -0,0 +1,49 @@
|
||||
把下面这段直接发给新的 Codex 就行：
|
||||
|
||||
```text
|
||||
接手这个项目，请先不要发散，也不要先动展示层。你先完整阅读并基于现状继续推进主线。
|
||||
|
||||
项目路径：
|
||||
`/www/wwwroot/getDomain`
|
||||
|
||||
必须先看这几份文档：
|
||||
1. `docs/ops_center_runtime/HANDOFF_20260420_1920.md`
|
||||
2. `docs/base.md`
|
||||
3. `docs/test.md`
|
||||
|
||||
这次接手的硬性要求：
|
||||
1. 先以 `docs/ops_center_runtime/HANDOFF_20260420_1920.md` 为准建立上下文。
|
||||
2. 不要把当前这台机器当成真正的 mainland controller。
|
||||
3. 当前工作机是海外 12 核测试控制面，真正的 `mainland-controller-01` 是 `121.204.244.188`。
|
||||
4. 不要再优先动页面、展示层、面板文案。
|
||||
5. 不要用破坏性 git 命令，不要回滚现有脏工作区改动。
|
||||
6. 新增或修改代码前，先确认你改的是主线瓶颈，而不是辅助功能。
|
||||
|
||||
当前已经确认的事实：
|
||||
1. 真正的 `mainland-controller-01` 已重新对正。
|
||||
2. controller 的 node-agent 身份上报已经修好，现在控制面里应显示：
|
||||
- `agent_hostname = mainland-controller-01`
|
||||
- `agent_ip = 121.204.244.188`
|
||||
3. controller 远端发布链已经成功跑通过一次。
|
||||
4. 当前真正吃任务的是 `mainland-controller-01`。
|
||||
5. `mainland-worker-01` 目前是 agent 在线，但检测没有真正参与，现象是本地检测态里出现 `127.0.0.1:5432 connection refused`。
|
||||
6. 当前主线已经不是“节点身份问题”，而是“真 controller 吞吐”和“worker 恢复”。
|
||||
|
||||
你接手后只做两条主线：
|
||||
1. 恢复 `mainland-worker-01`，让它重新进入可参与检测状态。
|
||||
2. 继续压 `mainland-controller-01` 的真实吞吐，只盯 `claimed -> running -> completed` 的推进，不回展示层。
|
||||
|
||||
你要避免的坑：
|
||||
1. 不要再把本机当成 `mainland-controller-01`。
|
||||
2. 不要优先用本机 Python service 入口直接造 deploy job，优先通过运行中的 API HTTP 接口。
|
||||
3. 不要把 `job 332` 这种“node-agent 重启自己导致状态未优雅回写”的现象直接误判为真正失败。
|
||||
4. 不要跑偏到 UI、导出、日志窗口样式这些支线。
|
||||
|
||||
你开始后先做这三件事，再继续动手：
|
||||
1. 复述你理解的当前真实环境拓扑。
|
||||
2. 复述当前两条唯一主线任务。
|
||||
3. 给出你准备先验证的 3 个现场指标，再开始执行。
|
||||
|
||||
目标只有一个：
|
||||
先把主流程和真实并发跑起来，让 controller 真机和 worker 真正参与检测，再谈细节优化。
|
||||
```
|
||||
96
docs/ops_center_runtime/HANDOFF_20260419_0308.md
Normal file
96
docs/ops_center_runtime/HANDOFF_20260419_0308.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# HANDOFF 2026-04-19 03:08
|
||||
|
||||
## 本轮做了什么
|
||||
|
||||
本轮没有进入新实现，只做了最小链路核查：
|
||||
|
||||
1. 复查中央 `detect/job/active`
|
||||
2. 复查中央 `ops/nodes`
|
||||
3. 复查 `go-live-summary` / `stack-diagnosis` / `release-launchpad`
|
||||
4. 复查最新 `onboarding.acceptance`
|
||||
5. 通过远端 Agent 对 `mainland-controller-01` 执行：
|
||||
- `service.status`
|
||||
- `logs.collect`
|
||||
目标服务：`domaincheck-sync-agent`
|
||||
|
||||
## 本轮结论
|
||||
|
||||
当前状态已经进一步推进：
|
||||
|
||||
- 检测任务在跑
|
||||
- 节点接管在跑
|
||||
- 最新 acceptance 已成功
|
||||
- `mainland-controller-01` 的 `domaincheck-sync-agent` 已重启成功
|
||||
- 重启后首轮已把 mainland 逐条结果推回中央
|
||||
|
||||
## 关键证据
|
||||
|
||||
### 1. 检测任务确实在跑
|
||||
|
||||
- 活跃任务：`detect-20260417170546-96023e`
|
||||
- 参与节点：
|
||||
- `mainland-worker-01`
|
||||
- `mainland-controller-01`
|
||||
- `overseas-control-01`
|
||||
|
||||
### 2. 最新 acceptance 已成功
|
||||
|
||||
- `pbr-9ce5c85f17`：成功
|
||||
- `pbr-389abd618c`：成功
|
||||
|
||||
说明：
|
||||
|
||||
- 接管验收不再是当前唯一主阻塞
|
||||
|
||||
### 3. sync-agent 服务已重启到新进程
|
||||
|
||||
远端只读结果：
|
||||
|
||||
- 服务：`domaincheck-sync-agent`
|
||||
- 状态：`active (running)`
|
||||
- 启动时间：`2026-04-19 16:09:52 CST`
|
||||
- 运行命令：
|
||||
- `/opt/domaincheck/domainCheck/.venv/bin/python -m app.sync_agent`
|
||||
|
||||
### 4. 重启后首轮结果投影已推送成功
|
||||
|
||||
最新日志显示：
|
||||
|
||||
- `detect_result_projection`
|
||||
- `batch_count = 1`
|
||||
- `success_count = 1`
|
||||
- `event_import.imported_count = 10`
|
||||
|
||||
### 5. 中央 mainland 逐条结果已出现
|
||||
|
||||
- 新增 `detect_result_ingest`
|
||||
- `id = 5382`
|
||||
- `created_at = 2026-04-19 03:10:14`
|
||||
- 近期 mainland `domain_*`
|
||||
- `count = 10`
|
||||
|
||||
## 下一轮唯一剩余动作
|
||||
|
||||
下一轮不要发散实现，只做持续性复查：
|
||||
|
||||
1. `/api/v1/runtime/sync-summary`
|
||||
2. `/api/v1/detect/job/active`
|
||||
3. mainland `domain_started/domain_completed/domain_failed/domain_blacklisted`
|
||||
4. `/api/v1/ops/go-live-summary`
|
||||
5. `/api/v1/ops/stack-diagnosis`
|
||||
|
||||
重点判断：
|
||||
|
||||
- mainland `domain_*` 是否持续增长
|
||||
- 总检 attention 是否已主要退化为历史残留
|
||||
|
||||
## 继续不要做什么
|
||||
|
||||
- 不做新页面
|
||||
- 不做控制面增强
|
||||
- 不做发布动作
|
||||
- 不做 item 级最终回写
|
||||
|
||||
## 一句话结论
|
||||
|
||||
`mainland-controller-01` 的 `domaincheck-sync-agent` 重启后，逐条结果同步已经打通；当前工作重点从“修链路”切换到“验证持续性与上线签收口径”。
|
||||
128
docs/ops_center_runtime/HANDOFF_20260419_0318.md
Normal file
128
docs/ops_center_runtime/HANDOFF_20260419_0318.md
Normal file
@@ -0,0 +1,128 @@
|
||||
# HANDOFF 2026-04-19 03:18
|
||||
|
||||
## 本轮做了什么
|
||||
|
||||
本轮没有进入新实现，只做了运行态只读复查：
|
||||
|
||||
1. 对比一个完整 40 秒观察窗口前后的：
|
||||
- `/api/v1/detect/job/active`
|
||||
- `/api/v1/runtime/sync-summary`
|
||||
2. 复查中央 `detect_run_events`
|
||||
3. 复查：
|
||||
- `/api/v1/ops/go-live-summary`
|
||||
- `/api/v1/ops/stack-diagnosis`
|
||||
- `/api/v1/ops/activity-stream`
|
||||
4. 复查节点现场日志：
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
|
||||
## 本轮结论
|
||||
|
||||
这轮结论要纠偏：
|
||||
|
||||
- 接管链路基本完成
|
||||
- 同步链路已经打通过一次
|
||||
- 但检测执行面当前没有继续出新结果
|
||||
|
||||
因此当前主阻塞不再是“接管/同步未通”，而是：
|
||||
|
||||
- 检测执行停滞
|
||||
- 外部站点依赖或代理池可用性异常
|
||||
|
||||
## 关键证据
|
||||
|
||||
### 1. 三台节点都在参与，但近窗没有吞吐
|
||||
|
||||
活跃任务仍是：
|
||||
|
||||
- `detect-20260417170546-96023e`
|
||||
|
||||
参与节点：
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
- `overseas-control-01`
|
||||
|
||||
但 40 秒观察窗口前后完全一致：
|
||||
|
||||
- `progress_percent = 2.1`
|
||||
- `completed = 21`
|
||||
- `running = 14`
|
||||
- `claimed = 34`
|
||||
- `pending = 931`
|
||||
|
||||
### 2. mainland 逐条结果没有继续增长
|
||||
|
||||
中央查询结果：
|
||||
|
||||
- mainland `domain_*` 事件数仍为 `10`
|
||||
- 最新 mainland `detect_result_ingest` 仍是：
|
||||
- `id = 5382`
|
||||
- `created_at = 2026-04-19 03:10:14`
|
||||
|
||||
说明：
|
||||
|
||||
- 首批同步成功过
|
||||
- 但后续没有继续流入新结果
|
||||
|
||||
### 3. go-live 与 stack 口径已经比之前更完整
|
||||
|
||||
当前：
|
||||
|
||||
- `remote_access_ready = 3/3`
|
||||
- `log_sync_state = full_capture`
|
||||
- `go_live_status = attention`
|
||||
|
||||
这说明：
|
||||
|
||||
- 基础运维骨架已经起来
|
||||
- attention 现在不能只归因于接管未完成
|
||||
|
||||
### 4. controller 现场日志已直接指向代理/外部依赖问题
|
||||
|
||||
`mainland-controller-01` 现场日志显示：
|
||||
|
||||
- `当前可用代理数: 0`
|
||||
- `最近结果: 刷新成功，可用 0 个`
|
||||
- 检测链包含：
|
||||
- 注册
|
||||
- 百度 site
|
||||
- 360 site
|
||||
- 站长之家
|
||||
- 爱站
|
||||
- 时光机
|
||||
|
||||
### 5. 页面上的“外部站点异常”与现场证据一致
|
||||
|
||||
从当前现场判断：
|
||||
|
||||
- 这不是页面误报
|
||||
- 而是执行面确实卡在外部依赖/代理可用性上
|
||||
|
||||
## 下一轮唯一应该做什么
|
||||
|
||||
下一轮不要发散实现，只做 `J2-检测执行停滞收口批`：
|
||||
|
||||
1. 继续只读确认：
|
||||
- `/api/v1/detect/job/active`
|
||||
- `/api/v1/runtime/sync-summary`
|
||||
- `/api/v1/ops/activity-stream`
|
||||
- `/api/v1/ops/nodes/{node_code}/scene-log`
|
||||
2. 必要时补一轮：
|
||||
- `domaincheck-worker` 运行日志取证
|
||||
3. 只判断三件事：
|
||||
- 是否仍然 `近窗吞吐 0`
|
||||
- 是否仍然 `可用代理数 0`
|
||||
- 是否仍然没有新 `domain_*` 事件
|
||||
|
||||
## 现在不要做什么
|
||||
|
||||
- 不做新页面
|
||||
- 不做控制面增强
|
||||
- 不做发布动作
|
||||
- 不做 item 级最终回写
|
||||
- 不把当前状态误判成“已稳定可签收”
|
||||
|
||||
## 一句话结论
|
||||
|
||||
当前项目已经不是“接不管、看不见、不同步”，而是“接管和同步都基本打通了，但检测执行卡在外部站点/代理可用性问题上，导致任务挂起且没有继续产出”。
|
||||
117
docs/ops_center_runtime/HANDOFF_20260419_0324.md
Normal file
117
docs/ops_center_runtime/HANDOFF_20260419_0324.md
Normal file
@@ -0,0 +1,117 @@
|
||||
# HANDOFF 2026-04-19 03:24
|
||||
|
||||
## 本轮做了什么
|
||||
|
||||
本轮没有进入新实现，只做了执行面根因收紧：
|
||||
|
||||
1. 继续观察活跃检测任务是否前进
|
||||
2. 继续观察 mainland `domain_*` 是否增长
|
||||
3. 通过正式 `ops job` 远端采集：
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
的 `domaincheck-worker` 日志
|
||||
4. 再盯一个 35 秒窗口，看 full capture 打开后，源日志时间是否继续前进
|
||||
|
||||
## 本轮结论
|
||||
|
||||
这轮已经能把问题说得更准：
|
||||
|
||||
- 不是“日志没回来”
|
||||
- 不是“全量日志没开”
|
||||
- 而是日志链路已经通了，但执行进程没有继续产生日志
|
||||
|
||||
当前第一主阻塞：
|
||||
|
||||
- `mainland-controller-01` 代理池可用数为 0
|
||||
|
||||
次级问题：
|
||||
|
||||
- `mainland-worker-01` 的时光机依赖异常会降级继续执行
|
||||
|
||||
## 关键证据
|
||||
|
||||
### 1. full capture 是开的，但源日志没继续前进
|
||||
|
||||
当前：
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `capture_at = 2026-04-19 03:21:59`
|
||||
- 源日志时间仍停在 `Apr 19 01:00:23`
|
||||
- `mainland-worker-01`
|
||||
- `capture_at = 2026-04-19 03:22:01`
|
||||
- 源日志时间仍停在 `Apr 19 02:19:56`
|
||||
|
||||
35 秒后再次复查：
|
||||
|
||||
- `capture_at` 没变化
|
||||
- `source_msg` 也没变化
|
||||
|
||||
说明：
|
||||
|
||||
- 日志回传本身不是主问题
|
||||
- 执行进程这段时间没有继续产生日志
|
||||
|
||||
### 2. controller 代理源能拉到数据，但所有代理都验不过
|
||||
|
||||
`mainland-controller-01` 的 `domaincheck-worker` 日志显示：
|
||||
|
||||
- 6 个代理源都能拉到原始代理
|
||||
- 抽样验证 24 个代理后：
|
||||
- `代理池刷新完成，共 0 个可用代理`
|
||||
- 失败集中在：
|
||||
- `ProxyError@https://m.baidu.com`
|
||||
- `Unable to connect to proxy`
|
||||
- `ConnectTimeoutError`
|
||||
|
||||
说明：
|
||||
|
||||
- 不是代理接口挂了
|
||||
- 是代理名单本身不可用
|
||||
|
||||
### 3. worker 还能跑，但时光机依赖异常会降级
|
||||
|
||||
`mainland-worker-01` 的 `domaincheck-worker` 日志显示：
|
||||
|
||||
- `时光机检测 外部依赖异常，步骤降级继续执行`
|
||||
- 同时仍可见：
|
||||
- `域名检测完成`
|
||||
- 收到新的控制消息时：
|
||||
- `收到启动检测指令，但检测任务已在运行，忽略重复启动`
|
||||
|
||||
说明：
|
||||
|
||||
- worker 不是完全不可用
|
||||
- 时光机异常存在，但不是最核心阻塞
|
||||
|
||||
### 4. 活跃任务仍然没有前进
|
||||
|
||||
- `progress_percent = 2.1`
|
||||
- `completed = 21`
|
||||
- `running = 14`
|
||||
- `claimed = 34`
|
||||
- `pending = 931`
|
||||
|
||||
并且 mainland `domain_*` 仍固定在 `10`
|
||||
|
||||
## 下一轮唯一应该做什么
|
||||
|
||||
继续只做 `J2-检测执行停滞收口批`：
|
||||
|
||||
1. 不扩功能
|
||||
2. 不改页面
|
||||
3. 只围绕 controller 代理池问题取证和恢复验证
|
||||
|
||||
唯一要确认的是：
|
||||
|
||||
- controller 代理池是否仍然 `可用 0`
|
||||
- 一旦代理恢复，`domain_*` 是否会继续增长
|
||||
|
||||
## 现在不要做什么
|
||||
|
||||
- 不把问题继续泛化成“外部站点都异常”
|
||||
- 不把问题误判为“日志回传没开”
|
||||
- 不做新页面、新模块、发布动作
|
||||
|
||||
## 一句话结论
|
||||
|
||||
当前项目不是“接管失败”，也不是“日志没回来”，而是“controller 侧拉到了代理名单，但代理全部校验失败，导致检测执行没有继续产生新结果”。
|
||||
66
docs/ops_center_runtime/HANDOFF_20260419_0426.md
Normal file
66
docs/ops_center_runtime/HANDOFF_20260419_0426.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# HANDOFF 2026-04-19 04:26 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已修复 `runtime/status` 兼容层，恢复旧字段：
|
||||
- `api_online`
|
||||
- `worker_online`
|
||||
- `cluster_summary`
|
||||
- `thread_count`
|
||||
- 已重启中央 `domaincheck-api`
|
||||
- 已重新构建 `domain-web` 前端静态包
|
||||
- 已确认大陆双节点日志回传同时进入中央：
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
- 已确认大陆两台线程配置真实生效：
|
||||
- `mainland-controller-01 -> 100`
|
||||
- `mainland-worker-01 -> 50`
|
||||
|
||||
## 当前真实状态
|
||||
|
||||
- `/api/v1/runtime/status`
|
||||
- `api_online = true`
|
||||
- `worker_online = true`
|
||||
- `cluster_summary.online_worker_nodes = 3`
|
||||
- `/api/v1/detect/status`
|
||||
- `progress.pending = 163815`
|
||||
- `progress.completed = 51`
|
||||
- `progress.running = 10`
|
||||
- `remote_log_line_count = 240`
|
||||
- `remote_log_node_count = 2`
|
||||
- 近 5 分钟中央 `detect_debug_events`
|
||||
- `mainland-controller-01 -> worker_log`
|
||||
- `mainland-worker-01 -> worker_log`
|
||||
|
||||
## 当前结论
|
||||
|
||||
- “API 离线 / 本机 Worker 离线 / 有效执行节点 0” 已不是后端真实状态
|
||||
- “全量日志没回来” 已不是后端真实状态
|
||||
- “100/50 并发没有下发” 也不是后端真实状态
|
||||
- 当前剩余主问题已经收紧为：
|
||||
- 检测实际吞吐仍偏低
|
||||
- 代理可用性与任务分发节奏仍在限制体感并发
|
||||
|
||||
## 下一轮唯一任务
|
||||
|
||||
只做：`J2-检测执行吞吐收口`
|
||||
|
||||
顺序：
|
||||
|
||||
1. 强刷浏览器，确认新前端包已经生效
|
||||
2. 复查 Detect 页是否恢复：
|
||||
- API 在线
|
||||
- 本机 Worker 在线
|
||||
- 有效执行节点 > 0
|
||||
- 日志窗口出现双节点日志
|
||||
3. 若显示已恢复，再继续只排查：
|
||||
- 为什么实际执行吞吐仍低于 `100/50`
|
||||
- 重点看代理可用性、任务领取节奏、线程实际活跃数
|
||||
|
||||
## 现在不要做
|
||||
|
||||
- 不扩新页面
|
||||
- 不扩控制面功能
|
||||
- 不新增专题文档
|
||||
- 不进入发布动作
|
||||
- 不切新大方向
|
||||
104
docs/ops_center_runtime/HANDOFF_20260419_1334.md
Normal file
104
docs/ops_center_runtime/HANDOFF_20260419_1334.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# HANDOFF 2026-04-19 13:34 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已修复 controller `sync-agent pull_tasks` 只导 `domains`、不导本地执行队列的问题
|
||||
- 已让 controller 持续创建本地镜像任务：
|
||||
- `sync-overseas-7380`
|
||||
- `sync-overseas-7383`
|
||||
- `sync-overseas-7392`
|
||||
- 后续批次持续增加
|
||||
- 已重启两台大陆 `domaincheck-worker`
|
||||
- 已确认两台大陆节点都进入真实任务队列：
|
||||
- `mainland-controller-01 -> 从任务队列获取到 800 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- 已修复 controller `runtime/detect_runs.json` 属主错误：
|
||||
- `root:root -> www:www`
|
||||
- 已确认 `runtime_projection` 最新同步恢复成功：
|
||||
- `sync_state = success`
|
||||
- `runtime_projection -> 投影推送成功`
|
||||
|
||||
## 当前真实状态
|
||||
|
||||
当前 `/api/v1/ops/nodes` 已显示：
|
||||
|
||||
- `remote_access_ready = 3`
|
||||
- `participating = 3`
|
||||
- `dispatch_active = 1`
|
||||
|
||||
大陆两台当前状态：
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `agent_state = online_busy`
|
||||
- `detect_runtime.max_threads = 100`
|
||||
- `processed_recent = 115`
|
||||
- `processed_per_minute = 7.67`
|
||||
- `is_current_participant = true`
|
||||
- `mainland-worker-01`
|
||||
- `agent_state = online_busy`
|
||||
- `detect_runtime.max_threads = 50`
|
||||
- `processed_recent = 632`
|
||||
- `processed_per_minute = 42.13`
|
||||
- `is_current_participant = true`
|
||||
|
||||
## 当前结论
|
||||
|
||||
- 大陆两台不是“在线但没干活”
|
||||
- 而是已经进入真实镜像队列执行
|
||||
- `100/50` 并发也不是停留在配置层，而是已经开始真实消费任务
|
||||
- 后台运行态同步链已经恢复
|
||||
|
||||
现在剩余问题已经收敛到：
|
||||
|
||||
- Detect 页面主计数 / 日志窗口是否完全跟上新的运行态
|
||||
- 恢复后的吞吐是否能持续稳定
|
||||
|
||||
## 一个关键口径说明
|
||||
|
||||
当前大陆执行是“镜像队列”模式：
|
||||
|
||||
- 海外待检测批次先被 controller 拉回大陆本地
|
||||
- 在大陆本地形成 `sync-overseas-*` 的 `detect_jobs/detect_job_items`
|
||||
- 再由大陆 worker 真正执行
|
||||
- 运行态和结果再同步回海外
|
||||
|
||||
所以现在不要再只盯中央原生 `detect_job_items.claimed/running` 判断大陆有没有参与。
|
||||
|
||||
应该优先看：
|
||||
|
||||
- `/api/v1/ops/nodes`
|
||||
- `processed_recent`
|
||||
- `processed_per_minute`
|
||||
- `detect_runtime.active_threads/max_threads`
|
||||
|
||||
## 下一轮唯一任务
|
||||
|
||||
只做：`J2-页面口径与吞吐稳定性收口`
|
||||
|
||||
顺序：
|
||||
|
||||
1. 继续观察 Detect 页面
|
||||
2. 确认日志窗口是否已经持续刷新
|
||||
3. 确认主计数是否开始跟随最新运行态
|
||||
4. 继续观察 1 到 2 个同步周期内：
|
||||
- `mainland-controller-01 processed_recent`
|
||||
- `mainland-worker-01 processed_recent`
|
||||
- `processed_per_minute`
|
||||
5. 若页面仍不跟，优先修页面读取口径，不扩功能
|
||||
|
||||
## 现在不要做
|
||||
|
||||
- 不扩新页面
|
||||
- 不扩控制面功能
|
||||
- 不新增模块
|
||||
- 不进入发布动作
|
||||
- 不切新方向
|
||||
|
||||
## 推荐模型
|
||||
|
||||
- 当前最合适：`GPT-5.4 + high`
|
||||
|
||||
原因：
|
||||
|
||||
- 现在主要是收口、验证、局部修复
|
||||
- 需要稳定推理，但不需要切到超高
|
||||
109
docs/ops_center_runtime/HANDOFF_20260419_1348.md
Normal file
109
docs/ops_center_runtime/HANDOFF_20260419_1348.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# HANDOFF 2026-04-19 13:48 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已定位并修复 `domainCheck/detect_worker.py` 的上下文绑定缺口：
|
||||
- `sync-pull` 控制消息只有
|
||||
- `target_job_id`
|
||||
- `target_job_code`
|
||||
- 旧逻辑只读取
|
||||
- `job_id`
|
||||
- `job_code`
|
||||
- 结果是 `mainland-worker-01` 明明已经在跑，但 `current_job_id` 为空，`worker_log` 被整段短路
|
||||
- 已把补丁同步到：
|
||||
- `mainland-controller-01:/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- `mainland-worker-01:/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- 已在两台大陆机完成：
|
||||
- `python -m py_compile /opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- `systemctl restart domaincheck-worker`
|
||||
|
||||
## 当前硬证据
|
||||
|
||||
### 1. mainland worker 日志回传已恢复
|
||||
|
||||
当前 `/api/v1/runtime/debug-events` 已出现：
|
||||
|
||||
- `mainland-worker-01 -> 开始执行检测任务，来源: redis-control`
|
||||
- `mainland-worker-01 -> 开始执行域名检测任务，正在加载配置`
|
||||
- `mainland-worker-01 -> 开始检测，正在刷新代理池`
|
||||
- `mainland-worker-01 -> 代理池刷新完成，共 7 个可用代理，来源链接 6 个，原始 265 个，验证 150 个`
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 1/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 2/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 3/50`
|
||||
|
||||
### 2. Detect 页面远端日志已经恢复双节点
|
||||
|
||||
当前 `/api/v1/detect/status` 已显示：
|
||||
|
||||
- `remote_log_node_count = 2`
|
||||
- `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]`
|
||||
|
||||
最近日志样本：
|
||||
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 1/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 2/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 3/50`
|
||||
- `mainland-controller-01 -> 当前实际线程数量: 1/100 ... 3/100`
|
||||
|
||||
### 3. 三台节点仍保持参与态
|
||||
|
||||
当前 `/api/v1/ops/nodes` 摘要：
|
||||
|
||||
- `online = 3`
|
||||
- `agent_ready = 3`
|
||||
- `remote_access_ready = 3`
|
||||
- `participating = 3`
|
||||
- `dispatch_active = 1`
|
||||
|
||||
## 当前判断
|
||||
|
||||
已经闭合：
|
||||
|
||||
- 大陆节点真实执行
|
||||
- mainland worker 日志回传
|
||||
- Detect 页面远端日志窗口只显示 controller 的问题
|
||||
|
||||
仍待收口：
|
||||
|
||||
- Detect 页面主计数：
|
||||
- `pending = 163678`
|
||||
- `completed = 195`
|
||||
- `running = 3`
|
||||
- 当前这些主计数还没有和这轮大陆执行立即同步推进
|
||||
|
||||
因此当前剩余唯一主问题已经缩成：
|
||||
|
||||
- `detect_result_projection / 结果统计回推` 是否还存在延迟或聚合口径缺口
|
||||
|
||||
## 下一轮唯一任务
|
||||
|
||||
只做：`J2-结果计数收口批`
|
||||
|
||||
顺序：
|
||||
|
||||
1. 复查 `detect_result_projection` 最新推送是否持续成功
|
||||
2. 复查 mainland 共享库里 `sync-overseas-*` 任务的完成/失败数量是否增长
|
||||
3. 复查 overseas `detect/status.progress` 是否跟着推进
|
||||
4. 若日志继续推进但主计数仍不动，只修结果统计回推口径，不扩功能
|
||||
|
||||
## 现在不要做
|
||||
|
||||
- 不扩新页面
|
||||
- 不扩控制面功能
|
||||
- 不新增模块
|
||||
- 不切新方向
|
||||
- 不直接进入发布动作
|
||||
|
||||
## 推荐模型
|
||||
|
||||
- 当前继续用：`GPT-5.4 + high`
|
||||
|
||||
原因：
|
||||
|
||||
- 现在是收口型问题
|
||||
- 需要稳定排查与小范围补丁
|
||||
- 不需要切超高推理
|
||||
65
docs/ops_center_runtime/HANDOFF_20260419_1416.md
Normal file
65
docs/ops_center_runtime/HANDOFF_20260419_1416.md
Normal file
@@ -0,0 +1,65 @@
|
||||
# HANDOFF 2026-04-19 14:16 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已修复 `detect_worker.py` 中“代理不足反向限制并发”的热路径：
|
||||
- 代理池不足时改为后台补货
|
||||
- 不再因为 `current_pool_size < thread_count/2` 而同步刷新、拖慢线程拉起
|
||||
- 已对线程创建阶段的运行态 / worker_log 做节流：
|
||||
- 不再每起一个线程就同步打一轮重日志
|
||||
- 避免远端日志回传本身把并发拉低
|
||||
- 已把上述修复部署到：
|
||||
- overseas 当前 live 运行目录 `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- repo 目录 `/www/wwwroot/getDomain/domainCheck/detect_worker.py`
|
||||
- `mainland-controller-01`
|
||||
- `mainland-worker-01`
|
||||
- 已补上“真实活跃检测线程计数”逻辑到代码文件中
|
||||
|
||||
## 当前结果
|
||||
|
||||
- `mainland-controller-01`
|
||||
- 海外后台当前已能看到：
|
||||
- `active_threads ~= 99~100`
|
||||
- `max_threads = 100`
|
||||
- 结论：
|
||||
- controller 的 `100` 并发已经真实跑起来
|
||||
- `mainland-worker-01`
|
||||
- 已重新接到新批次：
|
||||
- `source_record_id = 7679`
|
||||
- `target_job_code = sync-overseas-7679`
|
||||
- 最新日志已出现：
|
||||
- `开始执行域名检测任务`
|
||||
- `开始检测，刷新代理池`
|
||||
- `代理池刷新完成，共 23 个可用代理`
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- 本机进程线程量：
|
||||
- `152`
|
||||
- 结论：
|
||||
- worker 已重新进入真实执行
|
||||
- 但 overseas 后台对 `mainland-worker-01.active_threads/current_load` 的显示仍偏低
|
||||
|
||||
## 当前剩余问题
|
||||
|
||||
- 不是 controller 并发限制问题
|
||||
- 当前唯一剩余的运行面收口点是：
|
||||
- `mainland-worker-01` 的运行态上报 / 后台显示口径仍未完全对齐
|
||||
- 同时仍需继续观察：
|
||||
- Detect 页面主计数 `pending/completed/running`
|
||||
- 结果统计回推是否继续推进
|
||||
|
||||
## 下一轮建议
|
||||
|
||||
只继续做这两件事：
|
||||
|
||||
- 追 `mainland-worker-01` 的运行态采集链：
|
||||
- 为什么本机已进入新批次执行，但海外后台仍显示 `active_threads = 2`
|
||||
- 继续核对 Detect 主计数与结果回推：
|
||||
- 判断是统计延迟、投影延迟，还是任务结果尚未进入中央口径
|
||||
|
||||
不要做：
|
||||
|
||||
- 新页面
|
||||
- 新模块
|
||||
- 控制面增强
|
||||
- 发布动作
|
||||
- 新专题文档
|
||||
52
docs/ops_center_runtime/HANDOFF_20260419_1421.md
Normal file
52
docs/ops_center_runtime/HANDOFF_20260419_1421.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# HANDOFF_20260419_1421
|
||||
|
||||
更新时间：2026-04-19 14:21 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已完成检测页前端布局收口并上线验证：
|
||||
- `任务日志控制台` 已上移到事件列表前
|
||||
- `检测事件流` 已改成独立滚动区域
|
||||
- 事件表已限制最大高度，避免页面被长列表持续撑长
|
||||
- 已在真实线上静态目录重新构建：
|
||||
- `/www/wwwroot/getDomain/domain-web/dist`
|
||||
- 已验证线上 Nginx 实际指向该目录：
|
||||
- `/www/server/panel/vhost/nginx/domaincheck_3201.conf`
|
||||
- `/www/server/panel/vhost/nginx/domaincheck_152.53.37.118.conf`
|
||||
- 已验证本机线上端口可返回新页面：
|
||||
- `curl -I http://127.0.0.1:3201/ -> 200`
|
||||
- `index.html` 最后修改时间已更新到本轮
|
||||
- 新资源：
|
||||
- `/assets/DetectView-B9S1WMRT.js`
|
||||
- `/assets/DetectView-BFxMNRNR.css`
|
||||
|
||||
## 当前判断
|
||||
|
||||
- 这轮前端布局任务已经闭环
|
||||
- 如果浏览器仍显示旧布局，优先判断为缓存未刷新
|
||||
- 当前更值得继续追的不是页面结构，而是：
|
||||
- `mainland-worker-01` 运行态上报口径
|
||||
- Detect 主计数推进
|
||||
- 并发真实吞吐继续拉升
|
||||
|
||||
## 下一步建议
|
||||
|
||||
下一轮继续只围绕检测执行收口：
|
||||
|
||||
- 复核 `mainland-worker-01` 为什么本机线程已拉起，但后台 `active_threads/current_load` 仍偏低
|
||||
- 继续比对：
|
||||
- `/api/v1/detect/status`
|
||||
- `/api/v1/runtime/debug-events`
|
||||
- `/api/v1/ops/nodes`
|
||||
- 继续验证 Detect 页面主计数是否跟随真实执行推进
|
||||
|
||||
## 本轮关键命令证据
|
||||
|
||||
```bash
|
||||
cd /opt/domaincheck/domain-web && npm run build
|
||||
curl -I http://127.0.0.1:3201/
|
||||
curl http://127.0.0.1:3201/
|
||||
rg -n "event-stream-panel|任务日志控制台|检测事件流" \
|
||||
/www/wwwroot/getDomain/domain-web/dist/assets/DetectView-BFxMNRNR.css \
|
||||
/www/wwwroot/getDomain/domain-web/dist/assets/DetectView-B9S1WMRT.js
|
||||
```
|
||||
54
docs/ops_center_runtime/HANDOFF_20260419_1429.md
Normal file
54
docs/ops_center_runtime/HANDOFF_20260419_1429.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# HANDOFF_20260419_1429
|
||||
|
||||
更新时间：2026-04-19 14:29 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已修复代理链“配置已下发但长期停留未刷新”的问题
|
||||
- `domaincheck-worker` 现在会在以下时机主动刷新代理池：
|
||||
- worker 启动后
|
||||
- `proxy_config`
|
||||
- `thread_count`
|
||||
- `node_thread_counts`
|
||||
- `runtime_settings`
|
||||
- 以上配置更新后
|
||||
- 已重启：
|
||||
- `domaincheck-worker`
|
||||
- `domaincheck-api`
|
||||
- 已完成一次真实代理池刷新并拿到明确结果：
|
||||
- 原始代理数：`202`
|
||||
- 抽样验证数：`24`
|
||||
- 可用代理数：`0`
|
||||
|
||||
## 当前状态
|
||||
|
||||
- 当前后台不再显示误导性的“未刷新”
|
||||
- 当前真实状态为：
|
||||
- `proxy_runtime_label = 降级直连`
|
||||
- `proxy_runtime_reason = proxy_validation_zero`
|
||||
- `proxy_runtime_detail = 代理源最近返回了 202 个代理，已验证 24 个，但当前 0 个可用；系统已自动降级为直连继续执行；最近状态：刷新成功，可用 0 个`
|
||||
|
||||
## 关键判断
|
||||
|
||||
- 这批代理当前的主要问题不是“系统没去用”
|
||||
- 而是：
|
||||
- 代理源会返回很多 IP
|
||||
- 但 worker 实测校验时全部失败
|
||||
- 失败以 `ProxyError / ConnectTimeout` 为主
|
||||
|
||||
## 下一步建议
|
||||
|
||||
- 优先不要再纠结“为什么页面写未刷新”
|
||||
- 这一层已经修好
|
||||
- 下一轮如果继续优化代理，应只做以下两种之一：
|
||||
- 更换/补充代理供应组，重新验证可用率
|
||||
- 调整代理校验策略，但要接受更高的脏代理混入风险
|
||||
|
||||
## 现场证据
|
||||
|
||||
```text
|
||||
主动触发代理池刷新: worker_startup
|
||||
代理池链接拉取成功 ... 原始代理数: 50 / 50 / 0 / 50 / 2 / 50
|
||||
代理池验证已启用抽样模式，本次抽样 24 个代理进行可用性验证
|
||||
代理池刷新完成，共 0 个可用代理，来源链接 6 个
|
||||
```
|
||||
42
docs/ops_center_runtime/HANDOFF_20260419_1450.md
Normal file
42
docs/ops_center_runtime/HANDOFF_20260419_1450.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# HANDOFF_20260419_1450
|
||||
|
||||
更新时间：2026-04-19 14:50 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 已继续优化代理链，不只停留在“看见 0 可用”
|
||||
- 已新增两项优化：
|
||||
- 扩展严格验证目标：
|
||||
- `baidu`
|
||||
- `m.baidu`
|
||||
- `qq`
|
||||
- `360`
|
||||
- 当严格验证 `0` 命中时，启用：
|
||||
- `宽松入池 + 快速隔离`
|
||||
|
||||
## 当前结果
|
||||
|
||||
- 当前最新代理刷新结果：
|
||||
- `proxy_last_refresh_total_items = 270`
|
||||
- `proxy_last_validated_count = 24`
|
||||
- `proxy_last_available_count = 2`
|
||||
- `proxy_last_refresh_status = 宽松入池 2 个（严格校验 0 命中）`
|
||||
- 后台当前已显示：
|
||||
- `available_proxy_count = 2`
|
||||
- `proxy_runtime_detail = 代理池当前可用 2 个代理，配置来源 6 个；最近状态：宽松入池 2 个（严格校验 0 命中）`
|
||||
|
||||
## 关键判断
|
||||
|
||||
- 严格校验下，这批代理整体质量依然偏差
|
||||
- 但现在已经不再是完全 `0` 可用
|
||||
- 系统已成功放出少量试跑代理，可以继续观察：
|
||||
- 是否带动检测吞吐
|
||||
- 是否很快被失败隔离重新打回 `0`
|
||||
|
||||
## 下一步
|
||||
|
||||
- 优先观察这 `2` 个试跑代理是否真的参与检测执行
|
||||
- 如果能参与并带来吞吐，再继续逐步放宽
|
||||
- 如果很快再次掉回 `0`，下一轮就应转到：
|
||||
- 补代理供应组
|
||||
- 或按地区/分组拆分代理质量统计
|
||||
64
docs/ops_center_runtime/HANDOFF_20260419_1454.md
Normal file
64
docs/ops_center_runtime/HANDOFF_20260419_1454.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# HANDOFF 2026-04-19 14:54 CST
|
||||
|
||||
## 本轮目标
|
||||
|
||||
把代理链从“预验证优先”切到用户指定的最小高效路径：
|
||||
|
||||
- 只去掉过期 / 非法代理
|
||||
- 不再单独做可用性预验证
|
||||
- 直接投入真实检测
|
||||
- 失败即淘汰并触发补刷
|
||||
|
||||
## 本轮已完成
|
||||
|
||||
- 已修改 [detect_worker.py](/www/wwwroot/getDomain/domainCheck/detect_worker.py)
|
||||
- `refresh_proxy_pool()`
|
||||
- 移除预验证入池逻辑
|
||||
- 仅做过期 / 非法 / 去重过滤
|
||||
- 直接把代理源返回结果入池
|
||||
- `remove_proxy()`
|
||||
- 代理失败后不再留在当前池尾部
|
||||
- 直接从当前池移除
|
||||
- 保留失败标记并在低水位时触发补刷
|
||||
- 已重启 `domaincheck-worker`
|
||||
- 已确认 worker 新日志进入运行态
|
||||
|
||||
## 最新运行证据
|
||||
|
||||
- worker 最新日志：
|
||||
- `代理池刷新完成，共 270 个入池代理，来源链接 6 个，原始 270 个，去过期 0 个，非法 0 个`
|
||||
- API 最新状态：
|
||||
- `available_proxy_count = 270`
|
||||
- `proxy_last_refresh_status = 直入池 270 个（跳过预验证）`
|
||||
- `proxy_last_validated_count = 0`
|
||||
- `proxy_runtime_reason = healthy`
|
||||
|
||||
## 当前判断
|
||||
|
||||
- 这轮目标已经完成
|
||||
- 代理链现在已经符合“直接跑任务，失败就丢弃代理并换新”的要求
|
||||
- 下一步不该再回头做单独代理预验证
|
||||
|
||||
## 下一步主任务
|
||||
|
||||
唯一主任务：
|
||||
|
||||
- 继续观察真实检测吞吐是否随直入池策略抬升
|
||||
- 重点看：
|
||||
- `/api/v1/detect/status`
|
||||
- `/api/v1/ops/nodes`
|
||||
- 检测页主计数 `pending/completed/running`
|
||||
- 失败代理淘汰后是否能持续自动补池
|
||||
|
||||
## 若下一轮继续
|
||||
|
||||
优先处理：
|
||||
|
||||
- “并发已下发但主计数不明显推进”的统计 / 调度口径问题
|
||||
|
||||
不要处理：
|
||||
|
||||
- 不要重新加回重预验证逻辑
|
||||
- 不要扩展新页面
|
||||
- 不要扩展控制面新模块
|
||||
- 不要发散到发布链或新专题文档
|
||||
107
docs/ops_center_runtime/HANDOFF_20260419_2053.md
Normal file
107
docs/ops_center_runtime/HANDOFF_20260419_2053.md
Normal file
@@ -0,0 +1,107 @@
|
||||
# HANDOFF_20260419_2053
|
||||
|
||||
更新时间：2026-04-19 20:53 CST
|
||||
|
||||
## 本轮完成
|
||||
|
||||
- 修复发布打包遗漏：
|
||||
- 最新发布包现在已包含 `domainCheck/`
|
||||
- 最新正式签收包：
|
||||
- `domaincheck_release_20260419_205437`
|
||||
- 修复发布任务观测性：
|
||||
- `domain-api/app/services/ops_release_executor_core.py`
|
||||
- 发布目录不可写时显式失败
|
||||
- 增加 `ExecStart` 与 `current` 软链对齐观测
|
||||
- `domain-api/app/node_agent.py`
|
||||
- 增加 job 级兜底异常回写，避免任务假性卡死
|
||||
- 将卡死的 worker 发布任务纠正为真实失败：
|
||||
- `release_id = 6`
|
||||
- `rollout_id = 5`
|
||||
- `job_id = 204`
|
||||
- 当前已从假 `running` 回正为 `failed`
|
||||
- 本次失败对应的 Release 版本：
|
||||
- `domaincheck_release_20260419_203933`
|
||||
|
||||
## 今晚新增关键结论
|
||||
|
||||
### 1. worker 发布失败的真实原因已明确
|
||||
|
||||
- `mainland-worker-01`
|
||||
- node-agent 日志：
|
||||
- `2026-04-19 20:40:15 [node-agent] loop error: [Errno 13] Permission denied: '/opt/domaincheck/downloads'`
|
||||
- 对应发布任务：
|
||||
- `job 204 = failed`
|
||||
- `rollout 5 = failed`
|
||||
- 当前失败原因已经正式回写：
|
||||
- `install_root not writable: /opt/domaincheck/downloads`
|
||||
|
||||
### 2. mainland 两台节点都不符合当前 ReleaseHub 发布模型
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `domaincheck-worker` 当前启动路径：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- `mainland-controller-01`
|
||||
- `domaincheck-worker` 当前启动路径：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
|
||||
这意味着：
|
||||
|
||||
- 当前线上服务不是跑在
|
||||
- `/opt/domaincheck/current/domainCheck/...`
|
||||
- 所以即使发布成功切了 `current` 软链
|
||||
- 也不会自动让现有 systemd 服务切到新版本
|
||||
|
||||
### 3. worker 当前不是新版本高吞吐样本
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `Active since = 2026-04-19 18:18:50 CST`
|
||||
- `Main PID = 635924`
|
||||
- `CPU = 18.994s`
|
||||
- 最近日志仍停留在旧代码行号：
|
||||
- `_run_detect_register:2465`
|
||||
- `detect_domain:2947`
|
||||
- `start_redis_subscription:3590`
|
||||
|
||||
说明：
|
||||
|
||||
- worker 仍是旧进程
|
||||
- 当前不能把它当成“已切新包并真实参与高吞吐”的有效证据
|
||||
|
||||
## 结论
|
||||
|
||||
当前唯一主阻塞不是前端、不是统计口径、也不是单纯并发参数。
|
||||
|
||||
当前唯一主阻塞是：
|
||||
|
||||
- mainland 节点 `deploy.release` 目录权限不满足
|
||||
- mainland 节点 `domaincheck-worker` 的 systemd `ExecStart` 与 ReleaseHub 的 `current` 发布模型不一致
|
||||
|
||||
在这两个问题修正前：
|
||||
|
||||
- 不建议继续对 mainland 节点做正式 rollout
|
||||
- 不建议继续把 worker 统计口径偏差当成纯展示层问题
|
||||
|
||||
## 下一步唯一建议
|
||||
|
||||
1. 先修 mainland 节点发布基座
|
||||
- 让 node-agent 对 `/opt/domaincheck/downloads`、`/opt/domaincheck/releases` 有写权限
|
||||
- 或明确改成一个 node-agent 真正可写的发布根目录
|
||||
2. 再统一 `domaincheck-worker.service`
|
||||
- 让 `ExecStart` 指向 `/opt/domaincheck/current/domainCheck/detect_worker.py`
|
||||
- 不再直指 `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
3. 完成后重新发起：
|
||||
- worker rollout
|
||||
- 验证 PID 切换
|
||||
- 验证代码行号切换到当前版本
|
||||
- 再看吞吐和 CPU 利用率
|
||||
|
||||
## 本轮改动文件
|
||||
|
||||
- `package_domain_release.sh`
|
||||
- `verify_domain_release.sh`
|
||||
- `package_domain_release.ps1`
|
||||
- `verify_domain_release.ps1`
|
||||
- `domain-api/app/services/ops_release_executor_core.py`
|
||||
- `domain-api/app/node_agent.py`
|
||||
- `docs/ops_center_runtime/TASK_BOARD.md`
|
||||
- `docs/ops_center_runtime/IMPLEMENTATION_STATUS.md`
|
||||
102
docs/ops_center_runtime/HANDOFF_20260419_2117.md
Normal file
102
docs/ops_center_runtime/HANDOFF_20260419_2117.md
Normal file
@@ -0,0 +1,102 @@
|
||||
# HANDOFF_20260419_2117
|
||||
|
||||
更新时间：2026-04-19 21:17 CST
|
||||
|
||||
## 本轮结论
|
||||
|
||||
`mainland-worker-01` 的最新正式 rollout 已完成根因收口：
|
||||
|
||||
- `release_id = 7`
|
||||
- `rollout_id = 6`
|
||||
- `job_id = 211`
|
||||
|
||||
执行结果：
|
||||
|
||||
- 发布包下载成功
|
||||
- `/opt/domaincheck/downloads/domaincheck_release_20260419_205437.tar.gz`
|
||||
- release 解压成功
|
||||
- `/opt/domaincheck/releases/domaincheck_release_20260419_205437`
|
||||
- `current` 软链切换成功
|
||||
- `/opt/domaincheck/current -> /opt/domaincheck/releases/domaincheck_release_20260419_205437`
|
||||
- 最终失败在：
|
||||
- `systemctl restart domaincheck-worker`
|
||||
|
||||
控制面失败结果原文：
|
||||
|
||||
- `restart failed: domaincheck-worker`
|
||||
- `Failed to restart domaincheck-worker.service: Interactive authentication required.`
|
||||
|
||||
## 真实根因
|
||||
|
||||
不是发布包问题，也不是目录权限问题了。
|
||||
|
||||
当前根因是：
|
||||
|
||||
- `domaincheck-node-agent` 运行身份仍是 `www`
|
||||
- 它具备下载、解压、切换 `current` 的权限
|
||||
- 但不具备执行 systemd 服务重启的权限
|
||||
|
||||
所以当前所有以下动作在 mainland 机器上都会有同类风险：
|
||||
|
||||
- `deploy.release`
|
||||
- `service.restart`
|
||||
- 任何依赖 node-agent 直接操作 systemd 的动作
|
||||
|
||||
## 本轮已做代码修正
|
||||
|
||||
已修改：
|
||||
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
- `domain-api/deploy/multi-region/fix_mainland_release_base.sh`
|
||||
- 安装 node-agent drop-in 时补齐：
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
|
||||
已验证：
|
||||
|
||||
- `bash -n domain-api/deploy/multi-region/fix_mainland_release_base.sh`
|
||||
|
||||
## 现场下一步
|
||||
|
||||
下一步不要再重复发起新的 rollout，先把 `mainland-worker-01` 的 node-agent 切到 root。
|
||||
|
||||
建议现场执行：
|
||||
|
||||
```bash
|
||||
mkdir -p /etc/systemd/system/domaincheck-node-agent.service.d
|
||||
cat >/etc/systemd/system/domaincheck-node-agent.service.d/runtime-user.conf <<'EOF'
|
||||
[Service]
|
||||
User=root
|
||||
Group=root
|
||||
EOF
|
||||
systemctl daemon-reload
|
||||
systemctl restart domaincheck-node-agent
|
||||
systemctl status domaincheck-node-agent --no-pager -l
|
||||
```
|
||||
|
||||
完成后应立即复查：
|
||||
|
||||
```bash
|
||||
systemctl show domaincheck-node-agent -p User -p Group
|
||||
journalctl -u domaincheck-node-agent -n 80 --no-pager -l
|
||||
```
|
||||
|
||||
预期结果：
|
||||
|
||||
- `domaincheck-node-agent` 以 `root` 身份运行
|
||||
- 心跳恢复，`mainland-worker-01` 不再是 `stale`
|
||||
- 后续再发起 `deploy.release` 时，能够闭环到 `systemctl restart domaincheck-worker`
|
||||
|
||||
## 当前判断
|
||||
|
||||
当前主阻塞只剩一个：
|
||||
|
||||
- mainland node-agent 权限模型未切到 root
|
||||
|
||||
这个问题修完后，再重新发起 worker rollout，才有资格继续看：
|
||||
|
||||
- `domaincheck-worker` 是否切到新 PID
|
||||
- health check 是否通过
|
||||
- rollout 6 之后的 acceptance 是否可继续
|
||||
95
docs/ops_center_runtime/HANDOFF_20260419_2125.md
Normal file
95
docs/ops_center_runtime/HANDOFF_20260419_2125.md
Normal file
@@ -0,0 +1,95 @@
|
||||
# HANDOFF_20260419_2125
|
||||
|
||||
更新时间：2026-04-19 21:25 CST
|
||||
|
||||
## 本轮结果
|
||||
|
||||
`mainland-worker-01` 的 release rollout 已正式成功收口。
|
||||
|
||||
关键信息：
|
||||
|
||||
- `release_id = 7`
|
||||
- `rollout_id = 7`
|
||||
- `job_id = 214`
|
||||
|
||||
最终状态：
|
||||
|
||||
- `job 214 = success`
|
||||
- `rollout 7 = completed`
|
||||
- `mainland-worker-01.agent_state = online_busy`
|
||||
|
||||
## 成功证据
|
||||
|
||||
控制面 `job events` 已记录：
|
||||
|
||||
- `event_type = agent_completed`
|
||||
- `summary_text = release deployed`
|
||||
|
||||
结果载荷中已确认：
|
||||
|
||||
- `release_version = domaincheck_release_20260419_205437`
|
||||
- `restart_results`
|
||||
- `domaincheck-worker.returncode = 0`
|
||||
- `health_check.ok = true`
|
||||
- `services_checked = ["domaincheck-worker"]`
|
||||
- `execstart_alignment.mismatched_services = []`
|
||||
|
||||
说明这次已经完整走通：
|
||||
|
||||
1. 下载发布包
|
||||
2. 解压 release
|
||||
3. 切换 `/opt/domaincheck/current`
|
||||
4. 重启 `domaincheck-worker`
|
||||
5. 健康检查通过
|
||||
|
||||
## 为什么这次成功
|
||||
|
||||
不是发布逻辑变化了，而是现网权限问题被修掉了。
|
||||
|
||||
上一轮失败根因：
|
||||
|
||||
- `domaincheck-node-agent` 以 `www` 身份运行
|
||||
- 无法执行 `systemctl restart domaincheck-worker`
|
||||
- systemd 返回：
|
||||
- `Interactive authentication required`
|
||||
|
||||
本轮修复：
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `domaincheck-node-agent` 已切为：
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
|
||||
## 当前剩余项
|
||||
|
||||
唯一建议立即同步的预防动作：
|
||||
|
||||
- 在 `mainland-controller-01` 上，也把 `domaincheck-node-agent` 切成 `root`
|
||||
|
||||
原因：
|
||||
|
||||
- worker 已经验证这就是 release 闭环最后一层权限门槛
|
||||
- controller 如果后续参与自身发布 / restart，同样会遇到同类 systemd 权限问题
|
||||
|
||||
建议执行：
|
||||
|
||||
```bash
|
||||
mkdir -p /etc/systemd/system/domaincheck-node-agent.service.d
|
||||
cat >/etc/systemd/system/domaincheck-node-agent.service.d/runtime-user.conf <<'EOF'
|
||||
[Service]
|
||||
User=root
|
||||
Group=root
|
||||
EOF
|
||||
systemctl daemon-reload
|
||||
systemctl restart domaincheck-node-agent
|
||||
systemctl show domaincheck-node-agent -p User -p Group
|
||||
```
|
||||
|
||||
## 当前判断
|
||||
|
||||
关于 mainland worker 的发布链，已经不再是阻塞项。
|
||||
|
||||
现在可以进入下一步：
|
||||
|
||||
- 再决定是否对 `mainland-controller-01` 做同样 root 切换
|
||||
- 然后继续 controller acceptance / rollout 收口
|
||||
563
docs/ops_center_runtime/HANDOFF_20260420_1920.md
Normal file
563
docs/ops_center_runtime/HANDOFF_20260420_1920.md
Normal file
@@ -0,0 +1,563 @@
|
||||
# HANDOFF_20260420_1920
|
||||
|
||||
更新时间：2026-04-20 19:20 CST
|
||||
|
||||
## 本轮结论
|
||||
|
||||
这轮最重要的收口已经完成：
|
||||
|
||||
- 已确认当前工作机 `/www/wwwroot/getDomain` 所在主机是海外测试控制面，不是国内 `mainland-controller-01`
|
||||
- 已把“假冒 mainland-controller-01”的本机配置纠正为 `overseas-control-01`
|
||||
- 已把真正的 `mainland-controller-01` 重新接回控制面，并完成一次成功的远端发布
|
||||
- 已修正 `mainland-controller-01` 的 node-agent 身份上报问题
|
||||
- 现在控制面里看到的 `mainland-controller-01` 已经是真实主机：
|
||||
- `hostname = mainland-controller-01`
|
||||
- `ip = 121.204.244.188`
|
||||
|
||||
一句话说当前状态：
|
||||
|
||||
“主线已经从‘节点身份混乱’推进到‘真 controller 已经对正并重新进场’，下一步应只盯 controller 真机吞吐和 worker 恢复，不要再回展示层。”
|
||||
|
||||
## 当前环境认知
|
||||
|
||||
### 1. 当前 Codex 所在机器不是大陆 controller
|
||||
|
||||
本地执行结果：
|
||||
|
||||
- `hostname = v2202604268673447256`
|
||||
- `nproc = 12`
|
||||
|
||||
这台机器是海外控制面测试机，不是用户说的 112 核 controller。
|
||||
|
||||
真正的国内 controller 是：
|
||||
|
||||
- `node_code = mainland-controller-01`
|
||||
- `ssh_host = 121.204.244.188`
|
||||
|
||||
真正的国内 worker 是：
|
||||
|
||||
- `node_code = mainland-worker-01`
|
||||
- `ssh_host = 121.204.244.248`
|
||||
|
||||
### 2. 本机服务身份已经纠正
|
||||
|
||||
为避免海外测试机继续伪装成 mainland controller，已经改过这些环境文件：
|
||||
|
||||
- `/etc/default/domaincheck-worker`
|
||||
- `/etc/default/domaincheck-node-agent`
|
||||
- `/etc/default/domaincheck-api`
|
||||
|
||||
修正后的核心值：
|
||||
|
||||
- `NODE_CODE=overseas-control-01`
|
||||
- `NODE_REGION=overseas`
|
||||
- `SYNC_PUSH_ENABLED=false`
|
||||
|
||||
并且已经执行过：
|
||||
|
||||
- 停止本机 `domaincheck-worker`
|
||||
- 停止本机 `domaincheck-node-agent`
|
||||
- 重启本机 `domaincheck-api`
|
||||
|
||||
当前海外控制面 API 进程环境已确认是：
|
||||
|
||||
- `NODE_CODE=overseas-control-01`
|
||||
- `NODE_REGION=overseas`
|
||||
|
||||
## 本轮已完成的关键修复
|
||||
|
||||
## A. 修复 controller node-agent 上报 localhost / 127.0.0.1
|
||||
|
||||
修改文件：
|
||||
|
||||
- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py)
|
||||
|
||||
修复内容：
|
||||
|
||||
- `_hostname()` 不再优先相信 `localhost`
|
||||
- `_ip()` 不再使用容易得到 `127.0.0.1` 的旧逻辑
|
||||
- 优先通过控制面地址推导本机出口 IP
|
||||
- 回退时也会过滤 loopback
|
||||
|
||||
本轮新增/补过的测试：
|
||||
|
||||
- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py)
|
||||
|
||||
注意：
|
||||
|
||||
- 当前仓库里这份修复已经在真 controller 上生效
|
||||
- handover 已显示：
|
||||
- `agent_hostname = mainland-controller-01`
|
||||
- `agent_ip = 121.204.244.188`
|
||||
- `cluster_hostname = mainland-controller-01`
|
||||
- `cluster_ip = 121.204.244.188`
|
||||
|
||||
## B. 修复 control 节点发布健康检查窗口过短
|
||||
|
||||
修改文件：
|
||||
|
||||
- [domain-api/app/services/ops_release_service.py](/www/wwwroot/getDomain/domain-api/app/services/ops_release_service.py)
|
||||
|
||||
修改目标：
|
||||
|
||||
- control 节点发布时，`domaincheck-api` 启动偏慢，旧健康检查窗口太短，会误判失败并回滚
|
||||
|
||||
代码里已改成：
|
||||
|
||||
- `health_check_timeout_seconds = 20`
|
||||
- `health_check_retries = 6`
|
||||
- `health_check_interval_seconds = 3`
|
||||
|
||||
但要注意一个坑：
|
||||
|
||||
- 这份代码虽然已改进源码
|
||||
- 海外控制面的运行中 API 进程还没有通过这份新源码重新部署
|
||||
- 所以 `smart-rollout-preview` 里看到的默认值一度仍然是旧的 `10 / 2 / 2`
|
||||
|
||||
因此本轮实际是通过“手工 remote-agent deploy job 显式带长窗口 payload”打通了 controller 发布。
|
||||
|
||||
## C. 真 controller 发布已经成功一次
|
||||
|
||||
成功任务：
|
||||
|
||||
- `job_id = 331`
|
||||
- `job_code = ops-20260420183116-c33723`
|
||||
- `action = deploy.release`
|
||||
- `target_node_code = mainland-controller-01`
|
||||
- `execution_mode = remote-agent`
|
||||
- `status = success`
|
||||
|
||||
本次使用的 release：
|
||||
|
||||
- `release_id = 27`
|
||||
- `release_version = domaincheck_release_20260420_182916`
|
||||
|
||||
发布结果要点：
|
||||
|
||||
- 发布包下载成功
|
||||
- checksum 校验成功
|
||||
- `/opt/domaincheck/current` 切换成功
|
||||
- `domaincheck-api / domaincheck-worker / domaincheck-sync-agent` 重启成功
|
||||
- 健康检查最终通过
|
||||
|
||||
尤其要记住：
|
||||
|
||||
- 这次成功不是靠 smart rollout 默认值
|
||||
- 是通过显式下发以下健康窗口打通的：
|
||||
- `health_check_timeout_seconds = 20`
|
||||
- `health_check_retries = 6`
|
||||
- `health_check_interval_seconds = 3`
|
||||
|
||||
## D. controller node-agent 已经重新连回
|
||||
|
||||
后续又触发了：
|
||||
|
||||
- `job_id = 332`
|
||||
- `action = service.restart`
|
||||
- `payload.service_name = domaincheck-node-agent`
|
||||
|
||||
这个任务本身还停留在 `running`，原因很正常：
|
||||
|
||||
- node-agent 在“执行重启自己”的过程中会打断自身回执链
|
||||
- 所以作业状态可能不会自然收尾
|
||||
|
||||
但实际效果已经发生：
|
||||
|
||||
- API 日志已看到 `121.204.244.188` 在 `2026-04-20 18:35:16` 重新开始：
|
||||
- `POST /api/v1/ops/agent/heartbeat`
|
||||
- `POST /api/v1/ops/agent/pull?limit=1`
|
||||
|
||||
因此这条 job 332 可以视为“结果已生效，但状态未优雅回写”的典型自重启任务。
|
||||
|
||||
## 当前实机状态
|
||||
|
||||
按最新控制面查询：
|
||||
|
||||
### mainland-controller-01
|
||||
|
||||
- `agent_online = true`
|
||||
- `cluster_status = busy`
|
||||
- `agent_hostname = mainland-controller-01`
|
||||
- `agent_ip = 121.204.244.188`
|
||||
- `current_load` 在本轮观察中约 `210 - 233`
|
||||
- `detect_runtime.active_threads` 在本轮观察中约 `210 - 233 / 2000`
|
||||
- 已经有持续日志回传
|
||||
|
||||
### mainland-worker-01
|
||||
|
||||
- node-agent 在线
|
||||
- 但当前没有真正参与检测
|
||||
- 控制面返回的 `detect_runtime` 错误为：
|
||||
- `connection to server at "127.0.0.1", port 5432 failed: Connection refused`
|
||||
|
||||
这意味着：
|
||||
|
||||
- worker 机现在不是主要算力来源
|
||||
- 目前真正吃任务的是 `mainland-controller-01`
|
||||
|
||||
## 当前主线瓶颈
|
||||
|
||||
目前主线已经不是“谁是 controller”了，当前瓶颈明确是下面两个：
|
||||
|
||||
1. `mainland-worker-01` 未恢复到可参与检测状态
|
||||
2. `mainland-controller-01` 虽然真实线程已抬到 200+，但控制面队列视角仍存在：
|
||||
- `claimed` 偏高
|
||||
- `running/completed` 推进不够理想
|
||||
- 吞吐没有完全吃透机器
|
||||
|
||||
也就是说：
|
||||
|
||||
- “节点身份问题”已基本打穿
|
||||
- “发布链路问题”已基本打穿
|
||||
- 下一步该只盯“真 controller 吞吐”和“worker 恢复”
|
||||
|
||||
## 本轮新增发现
|
||||
|
||||
### 1. mainland-worker-01 的 `127.0.0.1:5432` 更像是 node-agent 侧遥测链路问题
|
||||
|
||||
本轮继续排查后，发现这个问题至少有两层：
|
||||
|
||||
1. `domainCheck` 默认配置本身就是：
|
||||
- `DB_HOST = localhost`
|
||||
2. worker 机上的 `domaincheck-node-agent` systemd unit 当前只加载：
|
||||
- `/etc/default/domaincheck-api`
|
||||
- `/etc/default/domaincheck-node-agent`
|
||||
|
||||
但大陆 worker 快速安装脚本真正写入数据库与 Redis 指向的是：
|
||||
|
||||
- `/etc/default/domaincheck-worker`
|
||||
|
||||
也就是说，worker 机上很可能出现这种情况：
|
||||
|
||||
- `domaincheck-worker` 进程拿到的是正确的 `DB_HOST=${MAINLAND_CONTROLLER_IP}`
|
||||
- 但 `domaincheck-node-agent` 没有继承 `/etc/default/domaincheck-worker`
|
||||
- node-agent 内部去跑 `get_detect_status()` 时，就会落回 `domain-api` 默认配置：
|
||||
- `db_host = 127.0.0.1`
|
||||
|
||||
于是控制面上看到的现象就变成：
|
||||
|
||||
- `mainland-worker-01 agent 在线`
|
||||
- 但 `detect_runtime` 里报：
|
||||
- `connection to server at "127.0.0.1", port 5432 failed: Connection refused`
|
||||
|
||||
### 2. node-agent 当前把“DB 查询失败”和“worker 进程离线”混成了一种失败
|
||||
|
||||
`domain-api/app/node_agent.py` 里的 `_detect_runtime_snapshot()` 原来是：
|
||||
|
||||
- `get_detect_status()` 和 `detect_worker_runtime()` 放在同一个总 `try` 里
|
||||
|
||||
这会导致：
|
||||
|
||||
- 只要前面的 DB 查询失败
|
||||
- 后面的 worker systemd 运行态也一起被吞掉
|
||||
- 最终上报成：
|
||||
- `worker_online = false`
|
||||
- `service_running = false`
|
||||
|
||||
即使真实情况其实可能只是：
|
||||
|
||||
- worker 服务还活着
|
||||
- 只是 node-agent 在采集 detect status 时查错 DB 了
|
||||
|
||||
### 3. 本轮已在源码里补的最小修复
|
||||
|
||||
已修改：
|
||||
|
||||
- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py)
|
||||
- [domain-api/deploy/systemd/domain-node-agent.service](/www/wwwroot/getDomain/domain-api/deploy/systemd/domain-node-agent.service)
|
||||
- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py)
|
||||
|
||||
修复内容：
|
||||
|
||||
1. `_detect_runtime_snapshot()` 改成分层降级：
|
||||
- 先拿 `detect_worker_runtime()`
|
||||
- 再单独尝试 `get_detect_status()`
|
||||
- 即使 detect status 因 DB 异常失败，也保留真实的 worker service 运行态
|
||||
2. `domaincheck-node-agent.service` 追加：
|
||||
- `EnvironmentFile=-/etc/default/domaincheck-worker`
|
||||
|
||||
这样 worker 机上的 node-agent 也能直接继承 worker 真实使用的 `DB_HOST / REDIS_HOST`。
|
||||
|
||||
### 4. 这次修复的生效边界
|
||||
|
||||
要特别注意：
|
||||
|
||||
- `node_agent.py` 代码修复可以通过常规 release 进入真实运行目录并生效
|
||||
- 但 `domaincheck-node-agent.service` 属于 `/etc/systemd/system/` 下的系统级 unit 文件
|
||||
- 常规 `deploy.release` 只会切换 `/opt/domaincheck/current`，不会自动重装 systemd unit
|
||||
|
||||
所以这两项应分开看：
|
||||
|
||||
1. `node_agent.py`：
|
||||
- 可以走最小 release 先上真机
|
||||
2. `domain-node-agent.service`：
|
||||
- 需要后续补 systemd unit 落地动作
|
||||
- 至少要包含：
|
||||
- 覆盖 unit 文件或 drop-in
|
||||
- `systemctl daemon-reload`
|
||||
- `systemctl restart domaincheck-node-agent`
|
||||
|
||||
### 5. 本轮测试结果
|
||||
|
||||
已通过的焦点测试：
|
||||
|
||||
- `PYTHONPATH=/www/wwwroot/getDomain/domain-api /opt/domaincheck/domainCheck/.venv/bin/python -m unittest tests.test_node_agent_delivery_queue -v`
|
||||
|
||||
结果：
|
||||
|
||||
- `11 tests`
|
||||
- `OK`
|
||||
|
||||
新增验证点：
|
||||
|
||||
- `test_detect_runtime_snapshot_degrades_to_worker_runtime_when_detect_status_fails`
|
||||
|
||||
它验证了：
|
||||
|
||||
- 即使 `get_detect_status()` 抛出
|
||||
- `connection to server at "127.0.0.1", port 5432 failed`
|
||||
- node-agent 依然会保留：
|
||||
- `worker_online = true`
|
||||
- `service_running = true`
|
||||
- `phase_detail = active/running`
|
||||
|
||||
## 本轮最小发布动作
|
||||
|
||||
### 1. 已生成新发布包
|
||||
|
||||
本轮最小修复已重新打包：
|
||||
|
||||
- `release_version = domaincheck_release_20260420_215734`
|
||||
- `release_id = 28`
|
||||
|
||||
生成结果：
|
||||
|
||||
- `archive = /www/wwwroot/getDomain/release/domaincheck_release_20260420_215734.tar.gz`
|
||||
- `sha256 = ad2c41625503dab1efaeadbf0641c8f6875a3cad0581a3b9c7835ccc37cdc718`
|
||||
|
||||
### 2. 已向真节点发起最小 release 下发
|
||||
|
||||
本轮只为把 `node_agent.py` 先推进真实运行目录，已创建：
|
||||
|
||||
- `job_id = 335`
|
||||
- `target = mainland-controller-01`
|
||||
- `action = deploy.release`
|
||||
- `job_id = 336`
|
||||
- `target = mainland-worker-01`
|
||||
- `action = deploy.release`
|
||||
|
||||
这两条 job 都已经越过“等待拉取”，进入了 agent 执行阶段，并至少记录到：
|
||||
|
||||
- `executor_received`
|
||||
- `deploy_download_started`
|
||||
|
||||
说明：
|
||||
|
||||
- 真 controller 和真 worker 都已经接到了这版最小 release
|
||||
- 当前至少已经在真实节点上进入下载/发布链
|
||||
|
||||
### 3. 这次发布的真实目标
|
||||
|
||||
这次发布不是为了一次性解决所有 worker 恢复问题，而是为了先把下面这条代码修复推到真节点：
|
||||
|
||||
- `domain-api/app/node_agent.py`
|
||||
|
||||
目的：
|
||||
|
||||
- 即使 worker 节点的 `detect_status` 查询因 DB 指向错误失败
|
||||
- node-agent 也不再把真实的 worker service 运行态一并吞掉
|
||||
- 控制面能先看到更接近真实的 `worker_online / service_running`
|
||||
|
||||
### 4. 这次发布的限制
|
||||
|
||||
这次最小 release 暂时还不能自动完成下面这件事：
|
||||
|
||||
- 把 `/etc/systemd/system/domaincheck-node-agent.service` 更新为新版本
|
||||
|
||||
原因：
|
||||
|
||||
- 常规 `deploy.release` 切的是 `/opt/domaincheck/current`
|
||||
- 不会自动重装 systemd unit
|
||||
|
||||
因此当前判断是：
|
||||
|
||||
1. `node_agent.py` 代码修复：
|
||||
- 已经进入真实节点发布链
|
||||
2. `domaincheck-node-agent.service` 的 `EnvironmentFile=-/etc/default/domaincheck-worker`：
|
||||
- 仍需要后续单独补系统级落地动作
|
||||
|
||||
### 5. 当前发布观察结论
|
||||
|
||||
截至本次交接整理时：
|
||||
|
||||
- `job 335 / 336` 已启动执行
|
||||
- 但尚未在本轮记录中拿到最终 `success / failed` 收尾结论
|
||||
- 因此后续接手时，先做的第一件事之一，就是复查这两条 job 的最终状态与事件流
|
||||
|
||||
## 当前不要再踩的坑
|
||||
|
||||
### 1. 不要再把本机当成 mainland-controller-01
|
||||
|
||||
当前工作机是海外测试控制面，只负责：
|
||||
|
||||
- API
|
||||
- 控制面
|
||||
- 同步接收
|
||||
|
||||
不是 112 核大陆 controller。
|
||||
|
||||
### 2. 不要直接用本机 Python service 函数创建 deploy job
|
||||
|
||||
坑点：
|
||||
|
||||
- 直接在 shell 里跑本地 Python 服务函数时，读取到的 `settings.node_code` 可能不是运行中 API 进程的真实环境
|
||||
- 之前就出现过把 job 错判为 `local-runtime` 的情况
|
||||
|
||||
正确做法：
|
||||
|
||||
- 优先通过“正在运行的 API HTTP 接口”创建 job
|
||||
- 不要优先走本机 Python service 入口
|
||||
|
||||
推荐路径：
|
||||
|
||||
- `POST /api/v1/ops/releases/from-package/latest`
|
||||
- `POST /api/v1/ops/jobs`
|
||||
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
|
||||
|
||||
### 3. 不要把 job 332 当成硬故障
|
||||
|
||||
`job 332` 是 `service.restart domaincheck-node-agent`。
|
||||
|
||||
它卡在 `running` 不代表没生效，反而更像：
|
||||
|
||||
- node-agent 重启了自己
|
||||
- 回执链没能把 job 收尾
|
||||
|
||||
判断是否真的生效，应看：
|
||||
|
||||
- API 日志里有没有来自 `121.204.244.188` 的 heartbeat/pull
|
||||
- handover 里 `agent_hostname / agent_ip` 是否已刷新
|
||||
|
||||
### 4. 不要再回展示层
|
||||
|
||||
当前最值钱的推进方向仍然是：
|
||||
|
||||
- controller 真机吞吐
|
||||
- claim/running/completed 推进
|
||||
- worker 恢复
|
||||
|
||||
不要把额度再耗在页面、文案、展示层结构上。
|
||||
|
||||
## 这轮动过的关键文件
|
||||
|
||||
最关键的源码文件：
|
||||
|
||||
- [domain-api/app/node_agent.py](/www/wwwroot/getDomain/domain-api/app/node_agent.py)
|
||||
- [domain-api/app/services/ops_release_service.py](/www/wwwroot/getDomain/domain-api/app/services/ops_release_service.py)
|
||||
- [domain-api/tests/test_node_agent_delivery_queue.py](/www/wwwroot/getDomain/domain-api/tests/test_node_agent_delivery_queue.py)
|
||||
|
||||
此外，工作区还存在大量其他未提交改动，不要随意回滚：
|
||||
|
||||
- `domain-api/`
|
||||
- `domain-web/`
|
||||
- `domainCheck/`
|
||||
- `docs/ops_center_runtime/`
|
||||
|
||||
这是一个脏工作区，接手时必须小心，不要用破坏性 git 命令。
|
||||
|
||||
## 建议下一个 Codex 只做的两件事
|
||||
|
||||
### 任务 1：恢复 mainland-worker-01
|
||||
|
||||
目标：
|
||||
|
||||
- 让 `mainland-worker-01` 从“agent 在线但不参与检测”恢复到可执行检测
|
||||
|
||||
先查方向：
|
||||
|
||||
- 为什么它在本地检测态里访问 `127.0.0.1:5432` 失败
|
||||
- 是本机 PostgreSQL 没启
|
||||
- 还是 worker 的运行配置仍然错误指向本地 DB
|
||||
- 还是它本来就不该走本地 PostgreSQL，而应走远端/统一控制链
|
||||
|
||||
验收标准：
|
||||
|
||||
- `mainland-worker-01.detect_runtime.worker_online = true`
|
||||
- `mainland-worker-01.detect_runtime.active_threads > 0`
|
||||
- 能稳定进入参与节点
|
||||
|
||||
### 任务 2：继续抬 mainland-controller-01 真吞吐
|
||||
|
||||
目标：
|
||||
|
||||
- 只盯真 controller 的任务推进链，不碰页面
|
||||
|
||||
重点看：
|
||||
|
||||
- `claimed -> running -> completed` 是否持续推进
|
||||
- `items_claimed` 是否能下降
|
||||
- `processed_recent / processed_per_minute` 是否能抬起来
|
||||
- `active_threads` 与 `completed` 是否成正相关
|
||||
|
||||
验收标准：
|
||||
|
||||
- controller 持续有真实 completed 增长
|
||||
- running/claimed 更贴近“持续补位”而不是堆积
|
||||
- 机器性能使用能继续往上抬
|
||||
|
||||
## 可直接复用的验证点
|
||||
|
||||
### 1. 看节点 handover
|
||||
|
||||
接口：
|
||||
|
||||
- `GET /api/v1/ops/nodes/mainland-controller-01/handover`
|
||||
|
||||
这一项已经能确认：
|
||||
|
||||
- 当前是不是对到了真 controller
|
||||
- agent/ip/hostname 是否正确
|
||||
- 当前 load / detect_runtime 是否在动
|
||||
|
||||
### 2. 看节点列表
|
||||
|
||||
接口：
|
||||
|
||||
- `GET /api/v1/ops/nodes`
|
||||
|
||||
重点字段：
|
||||
|
||||
- `agent_hostname`
|
||||
- `agent_ip`
|
||||
- `cluster_hostname`
|
||||
- `cluster_ip`
|
||||
- `detect_runtime.active_threads`
|
||||
- `current_load`
|
||||
|
||||
### 3. 看海外控制面 API 日志
|
||||
|
||||
已验证有效：
|
||||
|
||||
- `journalctl -u domaincheck-api -n 200 --no-pager`
|
||||
|
||||
尤其可以筛：
|
||||
|
||||
- `121.204.244.188`
|
||||
- `/api/v1/ops/agent/heartbeat`
|
||||
- `/api/v1/ops/agent/pull`
|
||||
|
||||
### 4. 发布 controller 的正确方式
|
||||
|
||||
推荐继续使用运行中的 API HTTP 接口，而不是本地 Python service 调用。
|
||||
|
||||
## 交接结论
|
||||
|
||||
这一轮真正解决掉的，不是“性能问题”本身，而是它前面最大的认知阻塞：
|
||||
|
||||
- 之前一直有一部分判断建立在“本机就是 controller”这个错误前提上
|
||||
- 现在这个前提已经纠正
|
||||
- 真 controller 已经重新纳入控制面，而且身份上报正确、远端发布成功、线程也已经真实抬起来
|
||||
|
||||
接手人从这里继续时，应该把主线收缩成一句话：
|
||||
|
||||
“只盯 `mainland-controller-01` 的真实吞吐推进，并恢复 `mainland-worker-01`，不要再回展示层，也不要再把海外 12 核测试机当成主算力机。”
|
||||
@@ -1,23 +1,189 @@
|
||||
# IMPLEMENTATION_STATUS
|
||||
|
||||
更新时间：2026-04-19 03:41 CST
|
||||
更新时间：2026-04-19 21:25 CST
|
||||
|
||||
## 当前真实状态
|
||||
|
||||
阶段判断：
|
||||
|
||||
- 海外单脑接管能力：约 `95%`
|
||||
- 分布式检测真实执行能力：约 `88%~90%`
|
||||
- 距离“可稳定上线并放心用后台发起检测”：约 `86%~89%`
|
||||
- 海外单脑接管能力：约 `97%`
|
||||
- 发布闭环真实可用度：约 `65%~70%`
|
||||
- 分布式检测真实执行能力：约 `80%~85%`
|
||||
- 距离“可稳定上线并放心用后台发起检测”：约 `78%~82%`
|
||||
|
||||
## 21:17 最新校正
|
||||
|
||||
刚完成的 `mainland-worker-01` 正式 rollout 已给出最终根因：
|
||||
|
||||
- `job 211 / rollout 6 / release 7`
|
||||
- 发布包下载成功
|
||||
- release 解压成功
|
||||
- `current` 软链切换成功
|
||||
- 最终失败在服务重启：
|
||||
- `restart failed: domaincheck-worker`
|
||||
- `Failed to restart domaincheck-worker.service: Interactive authentication required.`
|
||||
|
||||
这说明：
|
||||
|
||||
- 当前 ReleaseHub 主链路本身已可推进到“切换 current”
|
||||
- 真正未闭环的是 mainland node-agent 的 systemd 权限
|
||||
- 只要 node-agent 仍以 `www` 运行，后续 `deploy.release / service.restart` 都会在 systemd 这一跳失败
|
||||
|
||||
已完成的代码侧修正：
|
||||
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- 改为 `User=root`
|
||||
- 改为 `Group=root`
|
||||
- `domain-api/deploy/multi-region/fix_mainland_release_base.sh`
|
||||
- 改为给 node-agent drop-in 写入 `User=root` / `Group=root`
|
||||
|
||||
因此当前真实上线完成度需要再补一句校正：
|
||||
|
||||
- 发布闭环真实可用度：
|
||||
- 不是“发布包模型还没修”
|
||||
- 而是“node-agent 权限模型还差最后一次现网切换”
|
||||
|
||||
## 21:25 最新进展
|
||||
|
||||
worker 侧的现网切换已经完成验证成功：
|
||||
|
||||
- `mainland-worker-01`
|
||||
- 已把 `domaincheck-node-agent` 切为 `root`
|
||||
- 随后重新发起：
|
||||
- `rollout_id = 7`
|
||||
- `job_id = 214`
|
||||
- 最终结果：
|
||||
- `job 214 = success`
|
||||
- `rollout 7 = completed`
|
||||
|
||||
正式成功证据：
|
||||
|
||||
- `agent_completed`
|
||||
- `summary_text = release deployed`
|
||||
- `restart_results[domaincheck-worker].returncode = 0`
|
||||
- `health_check.ok = true`
|
||||
- `systemd ExecStart` 已对齐：
|
||||
- `/opt/domaincheck/current/domainCheck/detect_worker.py`
|
||||
|
||||
因此当前关于发布闭环的真实判断应更新为：
|
||||
|
||||
- worker 发布闭环：
|
||||
- 已从“卡在 systemd restart 权限”推进到“真实成功”
|
||||
- 当前剩余风险不再在 worker
|
||||
- 当前剩余预防项在 controller：
|
||||
- `domaincheck-node-agent` 也应同步切为 `root`
|
||||
|
||||
## 顶部校正
|
||||
|
||||
今天晚上的最新结论，需要覆盖前面一部分偏乐观判断：
|
||||
|
||||
- 最新发布包问题已经定位并修复：
|
||||
- 之前发布包漏掉了 `domainCheck/`
|
||||
- 现在最新签收包 `domaincheck_release_20260419_205437` 已经包含 `domainCheck/`
|
||||
- 但对 `mainland-worker-01` 的正式发布验证证明：
|
||||
- 线上 mainland 节点目前还不满足现有 ReleaseHub 发布模型
|
||||
- 已确认的真实阻塞有两个：
|
||||
- `deploy.release` 在 `mainland-worker-01` 上会因为
|
||||
- `Permission denied: /opt/domaincheck/downloads`
|
||||
- 而直接失败
|
||||
- mainland 两台节点当前 `domaincheck-worker` 的 `ExecStart` 都直接指向：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- 而不是 `/opt/domaincheck/current/domainCheck/detect_worker.py`
|
||||
|
||||
这两个事实组合起来说明：
|
||||
|
||||
- 当前发布链虽然在控制面上已经成型
|
||||
- 但 mainland 线上节点的安装形态还是旧模式
|
||||
- 所以当前不能再把“已能稳定 rollout 新版本”算进上线完成度
|
||||
|
||||
## 今晚新增硬证据
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `job 204 / rollout 5` 已正式失败回写
|
||||
- 失败原因已收口为：
|
||||
- `install_root not writable: /opt/domaincheck/downloads`
|
||||
- 只读核验得到的实际服务状态仍是旧进程：
|
||||
- `Active since Sun 2026-04-19 18:18:50 CST`
|
||||
- `Main PID = 635924`
|
||||
- `CPU = 18.994s`
|
||||
- 说明 worker 当前并没有切到新包，也没有形成可信的新吞吐样本
|
||||
- `mainland-controller-01`
|
||||
- 当前 `domaincheck-worker` 虽然在高负载运行
|
||||
- 但服务启动路径同样是：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- 说明 controller 也没有对齐 `current` 软链发布模型
|
||||
|
||||
## 本轮代码侧新增收口
|
||||
|
||||
- 已补齐发布包构建：
|
||||
- `package_domain_release.sh`
|
||||
- `verify_domain_release.sh`
|
||||
- `package_domain_release.ps1`
|
||||
- `verify_domain_release.ps1`
|
||||
- 现在发布包会包含 `domainCheck/`
|
||||
- 已补齐发布执行器的失败可观测性：
|
||||
- `domain-api/app/services/ops_release_executor_core.py`
|
||||
- 现在会在发布目录不可写时显式返回失败结果
|
||||
- 同时补充 `ExecStart` 与 `current` 软链的对齐观测
|
||||
- 已补齐 node-agent 的兜底失败回写：
|
||||
- `domain-api/app/node_agent.py`
|
||||
- 避免以后再出现任务已经炸掉但控制面一直卡在 `running`
|
||||
|
||||
## 本轮最新结论
|
||||
|
||||
这轮结论需要更新为四段：
|
||||
新增结论：
|
||||
|
||||
- 接管与同步能力已经明显趋于完成
|
||||
- worker 控制消息补偿链已经完成线上验证
|
||||
- controller 运行环境漂移已经被现场修正
|
||||
- 当前唯一剩余主阻塞已经收紧到 controller 代理池无可用代理
|
||||
- 检测页观察面已完成一轮线上收口：
|
||||
- `任务日志控制台` 已移到事件表上方
|
||||
- `检测事件流` 已改成独立滚动区
|
||||
- 已在真实线上静态目录重建生产包并通过本机 `3201` 端口验证
|
||||
- 当前如果用户仍看到旧布局，优先判断为浏览器缓存而不是部署未生效
|
||||
- 代理链已完成策略切换：
|
||||
- worker 启动后会主动刷新代理池
|
||||
- 配置更新后也会主动刷新代理池
|
||||
- 后台已能区分：
|
||||
- `等待首刷`
|
||||
- `proxy_validation_zero`
|
||||
- `直入池`
|
||||
- 当前本机最新真实结果是：
|
||||
- 原始代理 `270`
|
||||
- 预验证 `0`
|
||||
- 直入池 `270`
|
||||
- 说明当前问题已经从“代理校验过严导致池为空”切换为“真实任务内动态淘汰失效代理”
|
||||
|
||||
- `mainland-controller-01`
|
||||
- 已确认解除“代理不足即同步刷新、反向压死并发”的旧限制
|
||||
- 海外后台当前已能看到：
|
||||
- `active_threads = 99~100`
|
||||
- `max_threads = 100`
|
||||
- 说明 controller 的 `100` 并发已经真实生效
|
||||
- `mainland-worker-01`
|
||||
- 已重新部署最新版 `detect_worker.py`
|
||||
- 已重新接收新的 `sync-pull` 批次并重进检测：
|
||||
- `开始执行域名检测任务`
|
||||
- `开始检测，刷新代理池`
|
||||
- `代理池刷新完成，共 23 个可用代理`
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- 当前 worker 进程本机线程量已到：
|
||||
- `152` 个 OS 线程
|
||||
- 说明它并不是没启动，而是已经进入新批次执行
|
||||
- 当前剩余偏差：
|
||||
- overseas 后台对 `mainland-worker-01.active_threads/current_load` 的显示仍偏低
|
||||
- 这已经从“真实执行问题”收敛为“运行态上报 / 显示口径问题”
|
||||
|
||||
这一轮不是只修了显示问题，而是把“大陆节点真实执行 + worker 日志回传”一起补齐了。
|
||||
|
||||
当前最新结论：
|
||||
|
||||
- controller 的 `pull_tasks -> 本地队列` 断点已经修通
|
||||
- 两台大陆 worker 都已经进入真实镜像队列执行
|
||||
- `mainland-worker-01` 的 `worker_log` 已开始直接回灌 overseas `detect_debug_events`
|
||||
- `runtime_projection` 权限问题已经修复，后台运行态同步恢复
|
||||
- `/api/v1/ops/nodes` 已能证明 controller 的高并发真实生效
|
||||
- 当前剩余问题已经收敛为：
|
||||
- `mainland-worker-01` 运行态显示口径仍需继续对齐
|
||||
- 检测页主计数口径是否完全跟上
|
||||
- 结果统计回推是否持续稳定
|
||||
|
||||
## 当前证据拆分
|
||||
|
||||
@@ -25,210 +191,273 @@
|
||||
|
||||
已经完成：
|
||||
|
||||
- `detect_result_projection` 支持 `recent_domain_events`
|
||||
- 中央 ingest 会把逐条事件写入 `detect_run_events`
|
||||
- `sync_agent` 会自动产出 `detect_result_projection`
|
||||
- worker 控制消息新增 `request_id`
|
||||
- worker 运行态心跳会补偿消费 pending 控制消息
|
||||
- worker 收到并处理控制消息后会按 `request_id` 清理 pending 指令
|
||||
- `sync_push_service.ingest_detect_task_projection`
|
||||
- 不再只导入 `domains`
|
||||
- 已改为同时创建本地 `detect_jobs`
|
||||
- 已改为同时创建本地 `detect_job_items`
|
||||
- 已把 `target_job_id / target_job_code / queued_count / worker_start_ok` 回写到同步结果
|
||||
- `sync_agent`
|
||||
- controller 现在会在每轮 `pull_tasks` 后自动唤起本地 worker
|
||||
- `detect_worker._set_active_cycle_context`
|
||||
- 已兼容 `sync-pull` 控制消息
|
||||
- 不再只读取 `job_id / job_code`
|
||||
- 现在会回退读取 `target_job_id / target_job_code`
|
||||
- 运行态同步链
|
||||
- controller `runtime/detect_runs.json` 权限已修正为 `www:www`
|
||||
- `runtime_projection` 已恢复成功推送
|
||||
|
||||
本地代码验证已通过：
|
||||
|
||||
- `unittest domain-api/tests/test_worker_control_service.py`
|
||||
- `python -m py_compile domainCheck/detect_worker.py domain-api/app/services/worker_control_service.py`
|
||||
- `python -m py_compile domain-api/app/services/sync_push_service.py`
|
||||
- `python -m py_compile domain-api/app/services/runtime_control_service.py`
|
||||
- `python -m py_compile domain-api/app/sync_agent.py`
|
||||
- `python -m py_compile domainCheck/detect_worker.py`
|
||||
|
||||
线上运行验证也已经出现正向证据：
|
||||
### 2. 接管 / 外部条件闭环
|
||||
|
||||
- `mainland-worker-01`
|
||||
- 启动后发现待执行控制指令
|
||||
- 接受 `start_detection`
|
||||
- 开始执行远程检测任务
|
||||
- `mainland-controller-01`
|
||||
- 修正 Redis 环境后
|
||||
- 重新接受 `start_detection`
|
||||
- 开始执行远程检测任务
|
||||
|
||||
### 2. 接管/同步闭环
|
||||
|
||||
当前已完成：
|
||||
已经完成：
|
||||
|
||||
- `remote_access_ready = 3/3`
|
||||
- `log_sync_state = full_capture`
|
||||
- `mainland-controller-01` 的 `domaincheck-sync-agent` 已重启到新进程
|
||||
- 首轮 `detect_result_projection` 推送成功过一次
|
||||
- 中央已收到 mainland 首批 `domain_*` 事件
|
||||
- `ssh_ready = 2`
|
||||
- 大陆 controller / worker 都可远程运维
|
||||
- controller `domaincheck-sync-agent` 在线
|
||||
- mainland 两台 `domaincheck-node-agent` 在线
|
||||
|
||||
这说明：
|
||||
当前仍依赖你额外输入的部分：
|
||||
|
||||
- mainland 到中央的基础同步链是活的
|
||||
- 结果投影链至少成功打通过一次
|
||||
- 暂无新的 SSH / 接管前置输入缺口
|
||||
- 如果页面仍显示旧状态，最多只需要你浏览器强刷确认，不需要再补 SSH 信息
|
||||
|
||||
### 2.1 前端部署闭环
|
||||
|
||||
已经完成：
|
||||
|
||||
- 在线静态根目录确认：
|
||||
- `/www/wwwroot/getDomain/domain-web/dist`
|
||||
- 在线 Nginx 配置确认：
|
||||
- `/www/server/panel/vhost/nginx/domaincheck_3201.conf`
|
||||
- `/www/server/panel/vhost/nginx/domaincheck_152.53.37.118.conf`
|
||||
- 生产包重建确认：
|
||||
- `vite build` 成功
|
||||
- 线上资源确认：
|
||||
- `DetectView-B9S1WMRT.js`
|
||||
- `DetectView-BFxMNRNR.css`
|
||||
|
||||
说明：
|
||||
|
||||
- 检测页布局调整不是只停留在源码
|
||||
- 已经进入线上可访问静态产物
|
||||
|
||||
### 2.2 代理运行态闭环
|
||||
|
||||
已经完成：
|
||||
|
||||
- worker 启动即触发代理池首刷
|
||||
- 配置变更后自动补刷
|
||||
- API 状态页已能准确显示：
|
||||
- `proxy_not_refreshed_yet`
|
||||
- `proxy_validation_zero`
|
||||
- `宽松入池`
|
||||
|
||||
当前最新证据：
|
||||
|
||||
- `proxy_last_refresh_status = 直入池 270 个（跳过预验证）`
|
||||
- `proxy_last_refresh_total_items = 270`
|
||||
- `proxy_last_validated_count = 0`
|
||||
- `proxy_last_available_count = 270`
|
||||
|
||||
说明：
|
||||
|
||||
- 代理配置本身已成功下发
|
||||
- 代理源接口本身也能返回大量原始代理
|
||||
- 当前不再把“预验证是否通过”作为入池门槛
|
||||
- 现在改为让真实检测来完成失效代理淘汰
|
||||
|
||||
### 3. 检测执行闭环
|
||||
|
||||
当前未完成：
|
||||
这一块现在已经从“怀疑恢复”进入“确认恢复”：
|
||||
|
||||
- 短观察窗口内：
|
||||
- `progress_percent` 仍是 `2.1`
|
||||
- `items_completed` 仍是 `21`
|
||||
- `items_claimed` 仍是 `34`
|
||||
- `items_pending` 仍是 `931`
|
||||
- 中央 recent events 已刷新到更晚时间
|
||||
- `runtime/sync-summary` 最新记录已继续增长
|
||||
- 但 completed 尚未继续上涨
|
||||
- `mainland-controller-01`
|
||||
- 已出现：
|
||||
- `从任务队列获取到 800 个需要检测的域名`
|
||||
- `最大线程数: 100`
|
||||
- `当前实际线程数量: 1/100 ... 6/100`
|
||||
- `mainland-worker-01`
|
||||
- 已出现：
|
||||
- `从任务队列获取到 400 个需要检测的域名`
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `当前实际线程数量: 1/50 ... 3/50`
|
||||
- `开始检测域名: 0-demagogo.com`
|
||||
- `开始检测域名: 00123321.com`
|
||||
- `开始检测域名: 001dm.com`
|
||||
|
||||
这说明：
|
||||
|
||||
- 当前不是单纯“页面没刷新”
|
||||
- 而是执行现场这段时间没有继续出结果
|
||||
- `100/50` 已不是“配置已写入但没生效”
|
||||
- 它已经进入真实任务消费
|
||||
- worker 的远端日志也已经跟着真实执行一起回传
|
||||
|
||||
### 4. 海外后台可视证据
|
||||
|
||||
`/api/v1/ops/nodes` 当前已显示：
|
||||
|
||||
- `summary.remote_access_ready = 3`
|
||||
- `summary.participating = 3`
|
||||
- `summary.dispatch_active = 1`
|
||||
|
||||
并且大陆两台都已经被判定为真实参与者：
|
||||
|
||||
- `mainland-controller-01`
|
||||
- `agent_state = online_busy`
|
||||
- `participation_state = recent_throughput`
|
||||
- `processed_recent = 115`
|
||||
- `processed_per_minute = 7.67`
|
||||
- `detect_runtime.max_threads = 100`
|
||||
- `mainland-worker-01`
|
||||
- `agent_state = online_busy`
|
||||
- `participation_state = recent_throughput`
|
||||
- `processed_recent = 632`
|
||||
- `processed_per_minute = 42.13`
|
||||
- `detect_runtime.max_threads = 50`
|
||||
|
||||
这说明：
|
||||
|
||||
- 海外后台已经不只是看到“在线”
|
||||
- 现在已经能看到大陆节点的实际吞吐
|
||||
|
||||
## 本轮新增硬证据
|
||||
|
||||
通过中央观测面、节点现场日志和远端 `domaincheck-worker` 日志，已确认：
|
||||
### 证据 1：controller 的镜像队列已经真正落库
|
||||
|
||||
- `mainland-controller-01` 现场日志显示：
|
||||
- `当前可用代理数: 0`
|
||||
- `最近结果: 刷新成功，可用 0 个`
|
||||
- controller 新增远端日志显示：
|
||||
- 代理源拉取成功
|
||||
- 抽样校验后 `共 0 个可用代理`
|
||||
- 失败集中在：
|
||||
- `ProxyError@https://m.baidu.com`
|
||||
- `Unable to connect to proxy`
|
||||
- `ConnectTimeoutError`
|
||||
- worker 新增远端日志显示：
|
||||
- `发现待执行 Worker 控制指令`
|
||||
- `已接受检测启动指令`
|
||||
- `开始执行远程检测任务`
|
||||
- 说明线上补偿消费链已真正工作
|
||||
- controller 新增远端日志显示：
|
||||
- 初次重启后：
|
||||
- `Authentication required`
|
||||
- `maximum recursion depth exceeded`
|
||||
- 进一步排查确认：
|
||||
- `/etc/default/domaincheck-worker` 的 `REDIS_PASSWORD` 为空
|
||||
- 修正后再次重启：
|
||||
- `Redis 连接成功: 127.0.0.1:6379`
|
||||
- `已接受检测启动指令`
|
||||
- `开始执行远程检测任务`
|
||||
- 后续日志继续收紧到：
|
||||
- `代理已启用，但当前无可用代理`
|
||||
- 两台大陆节点 full capture 已开启，但源日志时间没有继续前进
|
||||
`sync-agent` 现场日志已出现：
|
||||
|
||||
这说明：
|
||||
- `target_job_code = sync-overseas-7380`
|
||||
- `queued_count = 200`
|
||||
- `worker_start_ok = True`
|
||||
|
||||
- worker 控制消息链不再是主阻塞
|
||||
- controller Redis 环境漂移也不再是主阻塞
|
||||
- 当前第一主阻塞已经进一步收紧到 controller 代理池不可用
|
||||
- worker 的时光机异常是客观存在的次级问题
|
||||
- 不是控制面未接管
|
||||
- 不是同步链未打通
|
||||
并持续产生：
|
||||
|
||||
## 本轮新增代码修复
|
||||
- `sync-overseas-7383`
|
||||
- `sync-overseas-7386`
|
||||
- `sync-overseas-7392`
|
||||
- `sync-overseas-7417`
|
||||
|
||||
本轮不是只停留在诊断，还补了一处运行态最小修复：
|
||||
说明：
|
||||
|
||||
### 修复点 1：控制消息唯一标识
|
||||
- 大陆 controller 现在不是只拉数据
|
||||
- 而是在持续形成可执行批次
|
||||
|
||||
- 文件：
|
||||
- `domain-api/app/services/worker_control_service.py`
|
||||
- 变更：
|
||||
- 每次 `send_worker_command(...)` 都附带 `request_id`
|
||||
- Redis `publish` 与 pending fallback 使用同一份消息体
|
||||
### 证据 2：两台大陆 worker 都已转入真实队列
|
||||
|
||||
### 修复点 2：worker 运行态补偿消费 pending 指令
|
||||
现场日志已明确出现：
|
||||
|
||||
- 文件：
|
||||
- `domainCheck/detect_worker.py`
|
||||
- 变更：
|
||||
- 心跳线程每轮会补偿尝试消费 pending 控制消息
|
||||
- 解决“worker 在线但 pubsub 消息漏收，导致 pending 指令长期不被消费”的风险
|
||||
- controller：
|
||||
- `从任务队列获取到 800 个需要检测的域名`
|
||||
- worker：
|
||||
- `从任务队列获取到 400 个需要检测的域名`
|
||||
|
||||
### 修复点 3：worker 收到指令后确认清理 pending
|
||||
说明：
|
||||
|
||||
- 文件：
|
||||
- `domainCheck/detect_worker.py`
|
||||
- 变更：
|
||||
- worker 实际收到控制消息后，会按 `request_id` 清理对应 pending 指令
|
||||
- 避免修复后又产生重复回放
|
||||
- 当前不是兼容旧链路假运行
|
||||
- 是镜像队列真运行
|
||||
|
||||
### 当前判断
|
||||
### 证据 3：runtime_projection 已恢复成功
|
||||
|
||||
这组修复解决的是：
|
||||
修复前：
|
||||
|
||||
- “检测启动已经发布，但 worker 可能静默漏收”的代码风险
|
||||
- `refresh runtime_projection failed: [Errno 13] Permission denied`
|
||||
|
||||
所以当前最准确的状态是：
|
||||
修复后：
|
||||
|
||||
- 代码级控制链缺口已补上
|
||||
- 线上部署验证已通过
|
||||
- 但 `controller` 代理池校验后 `0 available` 的现场阻塞仍然存在
|
||||
- 最新 `sync-agent` 日志已出现：
|
||||
- `sync_state = success`
|
||||
- `runtime_projection -> 投影推送成功`
|
||||
|
||||
说明：
|
||||
|
||||
- 后台运行态刷新和日志窗口不再被 controller 本地权限卡死
|
||||
|
||||
### 证据 4：`mainland-worker-01` 的 `worker_log` 已进入 overseas
|
||||
|
||||
`/api/v1/runtime/debug-events` 当前已能看到：
|
||||
|
||||
- `mainland-worker-01 -> 开始执行检测任务，来源: redis-control`
|
||||
- `mainland-worker-01 -> 开始执行域名检测任务，正在加载配置`
|
||||
- `mainland-worker-01 -> 开始检测，正在刷新代理池`
|
||||
- `mainland-worker-01 -> 代理池刷新完成，共 7 个可用代理，来源链接 6 个，原始 265 个，验证 150 个`
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 1/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 2/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 3/50`
|
||||
|
||||
说明：
|
||||
|
||||
- worker 的真实执行链已经进入 overseas 的日志视图
|
||||
- “后台看不到大陆 worker 在干活”这一层已经闭合
|
||||
|
||||
## 当前口径说明
|
||||
|
||||
这里有一个非常关键的判断口径已经变化：
|
||||
|
||||
- 大陆节点当前执行的是“海外任务在大陆本地落镜像队列，再把运行态和结果同步回海外”
|
||||
- 因此不能再只盯中央原生 `detect_job_items.claimed/running`
|
||||
- 现在应该同时看：
|
||||
- `/api/v1/ops/nodes`
|
||||
- `processed_recent`
|
||||
- `processed_per_minute`
|
||||
- `detect_runtime.active_threads/max_threads`
|
||||
- 大陆 controller 本地 `sync-overseas-*` 队列
|
||||
|
||||
如果只看中央原生队列，会误判“大陆没有参与”。
|
||||
|
||||
## 当前已闭合的问题
|
||||
|
||||
已闭合：
|
||||
|
||||
- 大陆 controller 无法 `pull_tasks`
|
||||
- Detect 页面误判“大陆没跑”
|
||||
- Detect 主统计口径不一致
|
||||
- Runtime / Queue / Detect 主摘要不统一
|
||||
- 中央逐条事件接收缺口
|
||||
- `sync_agent` 自动产出结果投影缺口
|
||||
- controller `pull_tasks` 只导入 domain、不导入本地队列
|
||||
- 大陆 worker 长时间卡在旧检测会话导致忽略新启动指令
|
||||
- controller `runtime_projection` 因文件属主错误无法推送
|
||||
- 大陆两台“在线但不确定是否真实执行”的状态模糊
|
||||
|
||||
## 当前唯一剩余问题
|
||||
## 当前剩余问题
|
||||
|
||||
当前唯一主问题仍然是：
|
||||
当前剩余问题已经缩成两个最小项：
|
||||
|
||||
- 检测执行面没有恢复到持续产出
|
||||
|
||||
但现在已经不需要再拆成“代码待验证”和“现场阻塞”两层。
|
||||
|
||||
当前唯一剩余现场阻塞就是：
|
||||
|
||||
- controller 代理池可用性为 0
|
||||
- 因此没有持续产生新的 domain 级结果
|
||||
- Detect 页面主计数 `pending/completed/running` 是否继续推进
|
||||
- 结果统计回推是否稳定，而不只是日志链路先恢复
|
||||
|
||||
换句话说：
|
||||
|
||||
- 现在不是逻辑未实现
|
||||
- 不是中央映射失败
|
||||
- 不是节点未接管
|
||||
- 而是执行现场没有继续产出，且 controller 侧卡在代理校验失败
|
||||
|
||||
## acceptance 当前状态
|
||||
|
||||
最新接管验收结果仍然成立：
|
||||
|
||||
- `pbr-9ce5c85f17`：`onboarding.acceptance` 成功
|
||||
- `pbr-389abd618c`：`onboarding.acceptance` 成功
|
||||
|
||||
旧的 `attention` run 依然是历史残留，但它们已经不是当前最真实的生产阻塞。
|
||||
- 现在不是接管问题
|
||||
- 不是链路不通问题
|
||||
- 不是日志完全回不来问题
|
||||
- 而是最后一层“主计数推进 + 结果统计收口”问题
|
||||
|
||||
## 当前是否可以继续跑检测测试
|
||||
|
||||
当前结论：
|
||||
|
||||
- 可以继续做最小运行态排查
|
||||
- 但不需要再优先验证 worker 控制消息链
|
||||
- 当前还不能把状态视作“后台检测已经稳定恢复”
|
||||
- 可以继续跑检测测试
|
||||
- 而且现在已经具备“后台可观测 + 大陆真实参与”的条件
|
||||
|
||||
## 当前是否建议直接上线
|
||||
|
||||
当前结论：
|
||||
|
||||
- 不建议现在按“可稳定上线”判断
|
||||
- 已接近可上线状态
|
||||
- 但我仍建议把它视为“准上线收口态”，不是最终完全签收态
|
||||
|
||||
原因不是接管面，而是执行面：
|
||||
原因：
|
||||
|
||||
- 三台节点都已接入
|
||||
- worker / controller 都已重新接上控制链
|
||||
- 但当前任务没有持续吞吐
|
||||
- controller 代理池全部验不过会直接影响检测产出
|
||||
- 主执行链已经恢复
|
||||
- 但还需要再确认 1 到 2 个同步周期内页面口径与吞吐稳定性
|
||||
|
||||
## 当前优先级判断
|
||||
|
||||
最高优先级：
|
||||
|
||||
- `J2-检测执行停滞收口批`
|
||||
- `J2-检测执行收口批`
|
||||
|
||||
当前不应继续推进：
|
||||
|
||||
@@ -236,18 +465,19 @@
|
||||
- 新模块
|
||||
- 新页面
|
||||
- 发布动作
|
||||
- 与检测执行停滞无关的工作
|
||||
- 与检测执行收口无关的工作
|
||||
|
||||
## 完成下一轮后的预期
|
||||
|
||||
如果下一轮确认：
|
||||
|
||||
- controller 代理池恢复可用
|
||||
- `domain_*` 开始继续增长
|
||||
- `items_completed` 和近窗吞吐重新前进
|
||||
- Detect 页面主计数已跟上
|
||||
- 日志窗口持续刷新
|
||||
- `detect_result_projection` 持续推进
|
||||
- `processed_recent / processed_per_minute` 持续增长
|
||||
|
||||
则整体可上线程度预计可回升到：
|
||||
则整体可上线程度预计可提升到：
|
||||
|
||||
- `92%~94%`
|
||||
- `96%~98%`
|
||||
|
||||
在那之前，当前口径应保持保守。
|
||||
|
||||
@@ -1,177 +1,456 @@
|
||||
# TASK_BOARD
|
||||
|
||||
更新时间：2026-04-19 03:41 CST
|
||||
更新时间：2026-04-19 21:25 CST
|
||||
|
||||
## 2026-04-20 主线切换说明
|
||||
|
||||
从这一刻开始，当前工作重心正式切回：
|
||||
|
||||
- 先跑通主任务流程
|
||||
- 先推进 7 个大任务的测试与验收
|
||||
- 并发、代理、DB 往返、展示层等细节优化先封存，不再抢主线
|
||||
|
||||
细节优化不删除，统一视为 backlog：
|
||||
|
||||
- worker pool 持续补位
|
||||
- claimed 回弹压缩
|
||||
- 高频 DB 更新继续合并
|
||||
- 代理失败链和等待窗口继续压缩
|
||||
- 运营视角页面继续细化
|
||||
|
||||
这些后续继续做，但不再打断主任务验收顺序。
|
||||
|
||||
当前主口径：
|
||||
|
||||
1. 大任务 1 真实验收闭环
|
||||
2. 大任务 2 controller 编排闭环
|
||||
3. 大任务 3 统一结果状态机闭环
|
||||
4. 大任务 4 本地控制状态 + syncer/finalizer 闭环
|
||||
5. 大任务 5 固定 worker pool 落地
|
||||
6. 大任务 6 时光机一期接入
|
||||
7. 大任务 7 运营视角面板闭环
|
||||
|
||||
## 21:17 最新补充
|
||||
|
||||
刚刚这轮真实 rollout 已经把 mainland worker 的最后一层阻塞拿实：
|
||||
|
||||
- `mainland-worker-01`
|
||||
- `release_id = 7`
|
||||
- `rollout_id = 6`
|
||||
- `job_id = 211`
|
||||
- 发布链路并不是卡在下载或解压：
|
||||
- 发布包 `domaincheck_release_20260419_205437` 已下载到 `/opt/domaincheck/downloads`
|
||||
- release 已解压到 `/opt/domaincheck/releases/domaincheck_release_20260419_205437`
|
||||
- `/opt/domaincheck/current` 已切到新 release
|
||||
- 最终失败点已经明确：
|
||||
- `restart failed: domaincheck-worker`
|
||||
- systemd 原因：
|
||||
- `Interactive authentication required`
|
||||
- 这说明当前 mainland 节点的 `domaincheck-node-agent` 虽然能写发布目录，但因为以 `www` 身份运行，无法执行：
|
||||
- `systemctl restart domaincheck-worker`
|
||||
|
||||
当前新的唯一主阻塞已经进一步收敛为：
|
||||
|
||||
- node-agent systemd 运行身份不对
|
||||
- 需要把 `domaincheck-node-agent` 改成 `root` 运行，发布链最后一跳才能闭环
|
||||
|
||||
本轮已在仓库内同步修正：
|
||||
|
||||
- `domain-api/deploy/systemd/domain-node-agent.service`
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
- `domain-api/deploy/multi-region/fix_mainland_release_base.sh`
|
||||
- node-agent drop-in 现在也会显式写入：
|
||||
- `User=root`
|
||||
- `Group=root`
|
||||
|
||||
## 21:25 worker rollout 收口
|
||||
|
||||
`mainland-worker-01` 已完成新一轮正式 rollout 收口成功：
|
||||
|
||||
- `release_id = 7`
|
||||
- `rollout_id = 7`
|
||||
- `job_id = 214`
|
||||
|
||||
结果确认：
|
||||
|
||||
- `job 214 = success`
|
||||
- `rollout 7 = completed`
|
||||
- `mainland-worker-01` 已恢复为：
|
||||
- `agent_state = online_busy`
|
||||
- 最近心跳恢复正常
|
||||
|
||||
控制面完成证据：
|
||||
|
||||
- `agent_completed`
|
||||
- `summary_text = release deployed`
|
||||
- `restart_results`
|
||||
- `domaincheck-worker.returncode = 0`
|
||||
- `health_check.ok = true`
|
||||
- `execstart_alignment.mismatched_services = []`
|
||||
|
||||
这说明 worker 当前已经真正完成：
|
||||
|
||||
- 新包下载
|
||||
- release 解压
|
||||
- `current` 切换
|
||||
- `domaincheck-worker` 重启
|
||||
- 健康检查通过
|
||||
|
||||
当前关于 mainland rollout 的唯一剩余预防项变成：
|
||||
|
||||
- `mainland-controller-01` 也应该同步把 `domaincheck-node-agent` 切到 `root`
|
||||
- 否则后续 controller 自身执行 `deploy.release / service.restart` 时还会踩到同类 systemd 权限问题
|
||||
|
||||
## 顶部校正
|
||||
|
||||
今天晚上的最新排查已经把当前主阻塞重新定性，之前“大陆两台已经完全进入统一新镜像执行”的判断需要收紧。
|
||||
|
||||
当前新增硬结论：
|
||||
|
||||
- 最新正式发布包已经重打并签收通过：
|
||||
- `domaincheck_release_20260419_205437`
|
||||
- 发布包现在已经确实包含 `domainCheck/`
|
||||
- 对 `mainland-worker-01` 发起正式 smart rollout 后，真实失败原因已经拿到：
|
||||
- `job 204 / rollout 5`
|
||||
- 本次失败对应的正式 Release 版本仍是：
|
||||
- `domaincheck_release_20260419_203933`
|
||||
- 失败原因：
|
||||
- `Permission denied: /opt/domaincheck/downloads`
|
||||
- 该失败已通过正式 agent complete 回写，后台不再是假 `running`
|
||||
- `mainland-worker-01` 当前运行中的 `domaincheck-worker` 仍是旧进程：
|
||||
- `Main PID = 635924`
|
||||
- `Active since = 2026-04-19 18:18:50 CST`
|
||||
- `CPU = 18.994s`
|
||||
- 说明它不是高吞吐新进程，而是老进程长期挂着
|
||||
- `mainland-controller-01` 当前运行中的 `domaincheck-worker` 虽然活跃，但服务启动路径同样是：
|
||||
- `/opt/domaincheck/domainCheck/detect_worker.py`
|
||||
- 两台 mainland 节点当前 `domaincheck-worker` 的 systemd 启动路径都不是：
|
||||
- `/opt/domaincheck/current/domainCheck/detect_worker.py`
|
||||
|
||||
这意味着：
|
||||
|
||||
- 当前 ReleaseHub 的 `current -> release_version` 切换模型，与线上 mainland 节点的实际服务启动路径不一致
|
||||
- 即使发布包和签收链已经修好，线上节点也还没有具备“按当前发布模型热切版本”的条件
|
||||
- 当前唯一主阻塞已经从“并发参数是否下发”切换成：
|
||||
- mainland 节点发布权限/目录模型不匹配
|
||||
- mainland 节点服务启动路径与发布模型不匹配
|
||||
|
||||
## 当前主批次
|
||||
|
||||
唯一主批次：`J2-检测执行停滞收口批`
|
||||
唯一主批次：`J2-检测执行收口批`
|
||||
|
||||
目标：
|
||||
|
||||
- 不进入新页面
|
||||
- 不扩展控制面
|
||||
- 不新增发布动作
|
||||
- 只收口当前唯一真实阻塞：
|
||||
- 节点已接管
|
||||
- 同步已打通
|
||||
- 但检测执行没有继续产出新结果
|
||||
- 只收口三件已经缩小到运行面的事情：
|
||||
- 大陆节点必须真正进入统一镜像队列，而不是停留在兼容旧链路
|
||||
- 后台必须能看到大陆节点的实时运行态与日志
|
||||
- `100/50` 并发配置要从“已下发”推进到“真实有参与吞吐”
|
||||
|
||||
## 本轮最新状态
|
||||
|
||||
本轮已经完成“代码修复 -> 两台大陆节点部署 -> 运行态复查”的完整一轮验证。
|
||||
### 20:53 最新阻塞结论
|
||||
|
||||
### 已完成的最小修复
|
||||
- `mainland-worker-01` smart rollout 已真实失败，不再继续误判为执行中
|
||||
- 失败证据：
|
||||
- node-agent 日志：
|
||||
- `2026-04-19 20:40:15 [node-agent] loop error: [Errno 13] Permission denied: '/opt/domaincheck/downloads'`
|
||||
- 发布任务：
|
||||
- `job 204 = failed`
|
||||
- 发布批次：
|
||||
- `rollout 5 = failed`
|
||||
- 当前不应该继续把精力放在前端展示或并发口径微调上
|
||||
- 当前必须先收口：
|
||||
- mainland 节点 `deploy.release` 所需目录权限
|
||||
- mainland 节点 systemd `ExecStart` 与 `/opt/domaincheck/current` 的一致性
|
||||
|
||||
- `worker_control_service.send_worker_command(...)`
|
||||
- 为每条 worker 控制消息补上 `request_id`
|
||||
- 保证 Redis 发布与 pending fallback 使用同一份负载
|
||||
- `detect_worker`
|
||||
- 在运行态心跳里周期性补偿消费 pending 控制消息
|
||||
- 在真正收到控制消息后，按 `request_id` 清理 pending 指令
|
||||
这轮已经完成从“控制链打通”到“真实执行恢复 + worker 日志回传恢复”的关键跨越。
|
||||
|
||||
本地验证已通过：
|
||||
### 本轮前端已上线校验
|
||||
|
||||
- `unittest domain-api/tests/test_worker_control_service.py`
|
||||
- `python -m py_compile domainCheck/detect_worker.py domain-api/app/services/worker_control_service.py`
|
||||
- 检测页 `任务日志控制台` 已调整到事件列表上方
|
||||
- `检测事件流` 已改成独立滚动区：
|
||||
- 表格 `max-height = 320`
|
||||
- 避免事件越积越多把整个页面继续向下撑长
|
||||
- 已在线上实际静态目录重新构建：
|
||||
- `/www/wwwroot/getDomain/domain-web/dist`
|
||||
- 已确认线上 Nginx 指向该目录：
|
||||
- `domaincheck_3201.conf`
|
||||
- `domaincheck_152.53.37.118.conf`
|
||||
- 已通过 `http://127.0.0.1:3201/` 返回 `200` 且加载新资源：
|
||||
- `DetectView-B9S1WMRT.js`
|
||||
- `DetectView-BFxMNRNR.css`
|
||||
|
||||
### 已完成的线上验证
|
||||
### 本轮代理链已收口
|
||||
|
||||
- `mainland-worker-01`
|
||||
- 已拉到 `main` 最新提交 `c33f4f1`
|
||||
- 已重启 `domaincheck-worker`
|
||||
- 启动后明确出现：
|
||||
- `发现待执行 Worker 控制指令`
|
||||
- `已接受检测启动指令`
|
||||
- `开始执行远程检测任务`
|
||||
- 已修复 worker 在“代理配置已下发但尚未开始检测”时长期停留 `未刷新` 的问题
|
||||
- `domaincheck-worker` 现在会在以下时机主动触发代理池刷新：
|
||||
- worker 启动后
|
||||
- `proxy_config / thread_count / node_thread_counts / runtime_settings` 更新后
|
||||
- 当前后台状态已不再误报“未刷新”
|
||||
- 代理策略已切到：
|
||||
- 只去掉过期 / 非法代理
|
||||
- 跳过预验证
|
||||
- 直接入池执行
|
||||
- 真实失败后立即淘汰并补刷
|
||||
- 最新实测结果已经明确：
|
||||
- 原始代理 `270` 个
|
||||
- 预验证 `0` 个
|
||||
- 当前可用代理 `270` 个
|
||||
- 最近状态：
|
||||
- `proxy_last_refresh_status = 直入池 270 个（跳过预验证）`
|
||||
- `available_proxy_count = 270`
|
||||
|
||||
### 本轮已完成
|
||||
|
||||
- `sync_push_service`
|
||||
- 已修复 `detect_task_ingest` 只落 `domains`、不落本地 `detect_jobs/detect_job_items` 的缺口
|
||||
- controller 现在会为拉回来的海外批次持续创建本地镜像任务：
|
||||
- `sync-overseas-7380`
|
||||
- `sync-overseas-7383`
|
||||
- `sync-overseas-7392`
|
||||
- 持续增长中
|
||||
- `mainland-controller-01`
|
||||
- 已拉到 `main` 最新提交 `c33f4f1`
|
||||
- 已重启 `domaincheck-worker`
|
||||
- 发现运行环境漂移：
|
||||
- `/etc/default/domaincheck-worker` 中 `REDIS_PASSWORD` 为空
|
||||
- 已最小修正该节点运行环境后再次重启
|
||||
- 修正后明确出现：
|
||||
- `Redis 连接成功: 127.0.0.1:6379`
|
||||
- `已接受检测启动指令`
|
||||
- `开始执行远程检测任务`
|
||||
- 已明确进入真实队列：
|
||||
- `从任务队列获取到 800 个需要检测的域名`
|
||||
- `最大线程数: 100`
|
||||
- `当前实际线程数量: 1/100 ... 6/100`
|
||||
- `mainland-worker-01`
|
||||
- 已修复 `sync-pull` 控制消息上下文绑定缺口：
|
||||
- `detect_worker._set_active_cycle_context` 现在会读取
|
||||
- `target_job_id`
|
||||
- `target_job_code`
|
||||
- 已重启 `domaincheck-worker`
|
||||
- 已明确进入真实队列：
|
||||
- `从任务队列获取到 400 个需要检测的域名`
|
||||
- `最大线程数: 50`
|
||||
- `当前实际线程数量: 1/50 ... 3/50`
|
||||
- 已明确把运行日志回传到 overseas：
|
||||
- `开始执行检测任务，来源: redis-control`
|
||||
- `开始执行域名检测任务，正在加载配置`
|
||||
- `开始检测，正在刷新代理池`
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `当前实际线程数量: 1/50`
|
||||
- `当前实际线程数量: 2/50`
|
||||
- `当前实际线程数量: 3/50`
|
||||
- `runtime_projection`
|
||||
- 已定位 controller `domain-api/runtime/detect_runs.json` 权限错误
|
||||
- 已把 `runtime` 目录和 `detect_runs.json` 改回 `www:www`
|
||||
- `domaincheck-sync-agent` 最新日志已从
|
||||
- `partial_success`
|
||||
- 变成：
|
||||
- `sync_state = success`
|
||||
- `runtime_projection -> 投影推送成功`
|
||||
- 海外控制面 `/api/v1/ops/nodes`
|
||||
- 现已明确看到三台节点都在参与
|
||||
- `remote_access_ready = 3`
|
||||
- `participating = 3`
|
||||
- `mainland-controller-01.is_current_participant = true`
|
||||
- `mainland-worker-01.is_current_participant = true`
|
||||
|
||||
### 本轮仍未完成的事情
|
||||
### 当前最关键的新证据
|
||||
|
||||
- 中央 `items_completed` 仍未在短观察窗口内继续增长
|
||||
- `mainland-controller-01` 仍然持续报：
|
||||
- `代理已启用，但当前无可用代理`
|
||||
- 因此当前剩余阻塞已经进一步收紧到：
|
||||
- controller 现场代理池没有可用代理
|
||||
- `mainland-controller-01`
|
||||
- `processed_recent = 115`
|
||||
- `processed_per_minute = 7.67`
|
||||
- `detect_runtime.max_threads = 100`
|
||||
- `mainland-worker-01`
|
||||
- `processed_recent = 632`
|
||||
- `processed_per_minute = 42.13`
|
||||
- `detect_runtime.max_threads = 50`
|
||||
- `overseas-control-01`
|
||||
- 仍在执行中央原生队列
|
||||
- 当前判断：
|
||||
- 大陆节点已经不是“纸面在线”
|
||||
- 已经是真实参与检测执行
|
||||
|
||||
## 本轮最新复查结果
|
||||
## 当前主判断
|
||||
|
||||
本轮在完成修复部署后，中央与节点现场出现了新的运行证据：
|
||||
### 本轮新增结论
|
||||
|
||||
- 活跃任务仍是 `detect-20260417170546-96023e`
|
||||
- `mainland-worker-01` 最新事件时间已经从旧窗口推进到：
|
||||
- `2026-04-19 16:37:00`
|
||||
- `runtime/sync-summary` 最新记录已继续增长到：
|
||||
- `id = 5491`
|
||||
- `created_at = 2026-04-19 03:40:45`
|
||||
- 说明中央已经重新收到新的运行侧同步流量
|
||||
- 但短窗口内主进度仍未松动：
|
||||
- `completed = 21`
|
||||
- `running = 14`
|
||||
- `claimed = 34`
|
||||
- `pending = 931`
|
||||
- controller 现场最新日志已收紧为：
|
||||
- `代理已启用，但当前无可用代理`
|
||||
- `mainland-controller-01`
|
||||
- 已修复“代理数量不足反向限制并发”的热路径问题
|
||||
- 当前海外后台已稳定看到：
|
||||
- `active_threads ~= 99~100`
|
||||
- `max_threads = 100`
|
||||
- 说明 `100` 并发不再只是配置已下发，而是已进入真实运行态
|
||||
- `mainland-worker-01`
|
||||
- 已重新部署同版 `detect_worker.py`
|
||||
- 已重新接收到新的 `sync-pull` 批次：
|
||||
- `source_record_id = 7679`
|
||||
- `target_job_code = sync-overseas-7679`
|
||||
- 已完成代理抽样校验并进入：
|
||||
- `开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- 当前剩余问题已经缩成：
|
||||
- worker 本机已启动新批次并创建线程
|
||||
- 但海外后台对 `mainland-worker-01.active_threads` 的显示仍偏低，和本机进程线程量不完全一致
|
||||
|
||||
### 当前唯一剩余收口点
|
||||
|
||||
- 不再是 controller 并发限制问题
|
||||
- 当前唯一剩余收口点变成：
|
||||
- `mainland-worker-01` 的运行态上报口径仍需继续和真实执行量对齐
|
||||
- 以及检测主计数 `pending/completed/running` 继续推进
|
||||
|
||||
### 已经收口的部分
|
||||
|
||||
- 大陆节点不再停留在 `pending_bootstrap`
|
||||
- Agent + SSH 接管已经完成
|
||||
- `sync-agent pull_tasks` 已经真正生成本地镜像队列
|
||||
- 两台大陆 worker 已经真正吃到 `detect_job_items`
|
||||
- 后台运行态同步链已经恢复
|
||||
|
||||
### 当前剩余问题
|
||||
|
||||
- Detect 页面日志窗口已经不再是单节点
|
||||
- `/api/v1/detect/status` 已出现：
|
||||
- `remote_log_node_count = 2`
|
||||
- `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]`
|
||||
- 当前剩余问题缩成一个最小点：
|
||||
- 检测页主计数 `pending/completed/running` 还没有跟着这轮大陆执行立即推进
|
||||
- 前端布局问题已收口，后续不再停留在“页面结构挡住观察”
|
||||
- 代理链当前也已收口到真实口径，不再停留在“配置有了但没刷新”
|
||||
- 下一步应继续观察这 `270` 个直入池代理在真实检测里的淘汰速度与吞吐提升
|
||||
- 现在更应该同时看：
|
||||
- `/api/v1/detect/status`
|
||||
- `/api/v1/runtime/debug-events`
|
||||
- `/api/v1/ops/nodes`
|
||||
- worker `current actual threads` 日志
|
||||
|
||||
## 下一步唯一主批次
|
||||
|
||||
唯一主批次保持为：`J2-检测执行收口批`
|
||||
|
||||
下一步只做：
|
||||
|
||||
- 继续观察 Detect 页面主计数是否跟上最新运行态
|
||||
- 继续确认 `detect_result_projection` / 结果统计回推是否稳定推进
|
||||
- 继续收口 `mainland-worker-01` 的运行态上报，使后台显示与本机真实线程量一致
|
||||
- 若仍有显示偏差，只修统计/上报口径，不新扩功能
|
||||
|
||||
## 候选批次
|
||||
|
||||
### Candidate J3
|
||||
|
||||
名称：结果计数收口批
|
||||
|
||||
进入条件：
|
||||
|
||||
- 节点已经真实执行
|
||||
- 日志窗口已经恢复
|
||||
- 但 Detect 页面主计数仍不前进
|
||||
|
||||
只做：
|
||||
|
||||
- 复核 `detect_result_projection`
|
||||
- 复核结果统计回推
|
||||
- 不改控制面结构
|
||||
|
||||
### Candidate J4
|
||||
|
||||
名称：吞吐稳定性观察批
|
||||
|
||||
进入条件：
|
||||
|
||||
- 页面口径已恢复
|
||||
- 继续确认大陆两节点吞吐是否稳定，不再回落
|
||||
|
||||
只做：
|
||||
|
||||
- 继续观察 `processed_recent`
|
||||
- 继续观察代理池质量
|
||||
- 继续观察 `active_threads/max_threads`
|
||||
|
||||
## 暂停项
|
||||
|
||||
以下任务现在不应继续推进：
|
||||
|
||||
- 新页面
|
||||
- 新模块
|
||||
- 新发布动作
|
||||
- 新专题文档
|
||||
- 与检测执行收口无关的控制面增强
|
||||
|
||||
## 关键证据
|
||||
|
||||
### 证据 1：接管与同步已通
|
||||
### 证据 1：镜像队列已经落地
|
||||
|
||||
当前中央状态：
|
||||
controller `sync-agent` 最新 `task pull tick` 已出现：
|
||||
|
||||
- `remote_access_ready = 3/3`
|
||||
- `log_sync_state = full_capture`
|
||||
- mainland controller 的 `domaincheck-sync-agent` 已在新进程上运行
|
||||
- `target_job_id`
|
||||
- `target_job_code`
|
||||
- `queued_count = 200`
|
||||
- `worker_start_ok = True`
|
||||
|
||||
说明：
|
||||
|
||||
- 当前不是接管问题
|
||||
- 也不是日志回传问题
|
||||
- 更不是同步链完全断开
|
||||
- controller 现在不是只同步域名
|
||||
- 而是在本地持续创建可执行队列
|
||||
|
||||
### 证据 2：检测任务当前没有继续出新结果
|
||||
### 证据 2：大陆 worker 已进入真实队列
|
||||
|
||||
连续 40 秒前后对比结果完全一致：
|
||||
现场日志已经明确出现：
|
||||
|
||||
- `JOB_PROGRESS = 2.1`
|
||||
- `items_completed = 21`
|
||||
- `items_running = 14`
|
||||
- `items_claimed = 34`
|
||||
- `items_pending = 931`
|
||||
- `mainland-controller-01 -> 从任务队列获取到 800 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
|
||||
说明：
|
||||
|
||||
- 当前不是“页面慢一拍”
|
||||
- 而是执行面这段时间确实没有继续产出
|
||||
- “并发没起来”的主要根因已经修正
|
||||
- 它们不是在跑旧兼容链路
|
||||
|
||||
### 证据 3：中央 mainland 逐条结果没有继续增长
|
||||
### 证据 3：运行态同步已恢复
|
||||
|
||||
当前中央查询结果：
|
||||
controller `sync-agent` 最新日志已经出现：
|
||||
|
||||
- mainland `domain_started/domain_completed/domain_failed/domain_blacklisted`
|
||||
- 仍为 `10`
|
||||
- 最新 mainland `detect_result_ingest`
|
||||
- 仍为 `5382`
|
||||
- `sync_state = success`
|
||||
- `runtime_projection -> 投影推送成功`
|
||||
|
||||
说明：
|
||||
|
||||
- 首批同步成功过
|
||||
- 但后续并没有继续流入新逐条结果
|
||||
- 后台运行态/日志窗口链路已经不再被权限错误卡死
|
||||
|
||||
### 证据 4：controller 现场日志已指向代理池可用性为 0
|
||||
### 证据 4：海外 `/ops/nodes` 已把大陆节点判定为真实参与者
|
||||
|
||||
`mainland-controller-01` 现场日志显示：
|
||||
当前海外控制面已经显示：
|
||||
|
||||
- `当前可用代理数: 0`
|
||||
- `最近结果: 刷新成功，可用 0 个`
|
||||
- 免费检测链包含：
|
||||
- 注册查询
|
||||
- 百度 site
|
||||
- 360 site
|
||||
- 站长之家
|
||||
- 爱站
|
||||
- 时光机
|
||||
|
||||
进一步的远端 `logs.collect(domaincheck-worker)` 结果显示：
|
||||
|
||||
- controller 能从 6 个代理源成功拉到原始代理
|
||||
- 但在抽样验证后：
|
||||
- `代理池刷新完成，共 0 个可用代理`
|
||||
- 失败原因集中在：
|
||||
- `ProxyError@https://m.baidu.com`
|
||||
- `Unable to connect to proxy`
|
||||
- `ConnectTimeoutError`
|
||||
- `mainland-controller-01`
|
||||
- `online_busy`
|
||||
- `is_current_participant = true`
|
||||
- `processed_recent = 115`
|
||||
- `mainland-worker-01`
|
||||
- `online_busy`
|
||||
- `is_current_participant = true`
|
||||
- `processed_recent = 632`
|
||||
|
||||
说明：
|
||||
|
||||
- 当前不是代理源接口没返回
|
||||
- 而是“拿到的代理全部验不过”
|
||||
- 主阻塞已经可以精确收紧到 controller 代理池不可用
|
||||
- 现在大陆两台都已进入真实参与态
|
||||
- 下一步不再是接管问题，而是页面口径与吞吐稳定性问题
|
||||
|
||||
### 证据 5：worker 的时光机异常存在，但不是第一主因
|
||||
### 证据 5：Detect 页面远端日志已恢复双节点
|
||||
|
||||
`mainland-worker-01` 的远端 `domaincheck-worker` 日志显示：
|
||||
当前 `/api/v1/detect/status` 已显示：
|
||||
|
||||
- 存在：
|
||||
- `时光机检测 外部依赖异常，步骤降级继续执行`
|
||||
- 同时仍可见：
|
||||
- `域名检测完成`
|
||||
- 最近收到控制消息后：
|
||||
- `收到启动检测指令，但检测任务已在运行，忽略重复启动`
|
||||
- `remote_log_node_count = 2`
|
||||
- `remote_log_nodes = ["mainland-controller-01", "mainland-worker-01"]`
|
||||
|
||||
最近日志样本已出现：
|
||||
|
||||
- `mainland-worker-01 -> 从任务队列获取到 400 个需要检测的域名`
|
||||
- `mainland-worker-01 -> 开始创建线程，当前批次域名数: 400，最大线程数: 50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 1/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 2/50`
|
||||
- `mainland-worker-01 -> 当前实际线程数量: 3/50`
|
||||
|
||||
说明：
|
||||
|
||||
- worker 并不是完全不能执行
|
||||
- 时光机异常是客观存在的次级问题
|
||||
- 但它不像 controller 代理池为 0 那样直接卡住整体吞吐
|
||||
- worker 不再是“只在节点本地运行、页面看不到”
|
||||
- 检测页日志链已经真正接上 mainland worker
|
||||
|
||||
### 证据 6：full_capture 已开启，但源日志时间没有继续前进
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
683962
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 02:29:55","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":3,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"full_capture","status_label":"全量观察","status_type":"success","summary":"节点当前正在参与检测，已保留 1 条现场日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":1,"key_line_count":0,"full_line_count":1,"last_at":"2026-04-19 01:20:35","last_line":"[2026-04-19 01:20:35] [overseas-control-01] 2026-04-19 01:00:18.866 | INFO | __main__:start_detection:2575 - 开始执行域名检测任务"},"missing_reason_code":"","missing_reason":"","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 02:30:02"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract，供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,16 @@
|
||||
CYCLES="2"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="full_capture"
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS="attention"
|
||||
ISSUE_TOTAL="1"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES="playbook_runs_need_attention"
|
||||
LAUNCHPAD_STATUS="blocked"
|
||||
LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance"
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS="full_capture"
|
||||
SCENE_LINE_COUNT="1"
|
||||
GENERATED_AT="2026-04-19 02:29:55"
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 03:30:12","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"partial_coverage","log_sync_mode":"full","log_sync_covered_nodes":2,"log_sync_missing_node_codes":["overseas-control-01"],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"missing_sample","status_label":"缺少样本","status_type":"warning","summary":"当前还没有收到该参与节点的远端日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":0,"key_line_count":0,"full_line_count":0,"last_at":"","last_line":""},"missing_reason_code":"no_sample","missing_reason":"当前还没有收到该参与节点的远端日志样本。","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 03:30:19"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract，供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,16 @@
|
||||
CYCLES="3"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="partial_coverage"
|
||||
LOG_SYNC_MISSING_NODE_CODES="overseas-control-01"
|
||||
STACK_STATUS="attention"
|
||||
ISSUE_TOTAL="2"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES="remote_log_sync_waiting_sample,playbook_runs_need_attention"
|
||||
LAUNCHPAD_STATUS="blocked"
|
||||
LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance"
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS="missing_sample"
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT="2026-04-19 03:30:12"
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 03:31:26","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":3,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"full_capture","status_label":"全量观察","status_type":"success","summary":"节点当前正在参与检测，已保留 3 条现场日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":3,"key_line_count":0,"full_line_count":3,"last_at":"2026-04-19 03:30:30","last_line":"[2026-04-19 03:30:30] [overseas-control-01] 2026-04-19 01:59:40.686 | WARNING | __main__:start_detection_async:1073 - 收到启动检测指令，但检测任务已在运行，忽略重复启动"},"missing_reason_code":"","missing_reason":"","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 03:31:33"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract，供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,16 @@
|
||||
CYCLES="4"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="full_capture"
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS="attention"
|
||||
ISSUE_TOTAL="1"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES="playbook_runs_need_attention"
|
||||
LAUNCHPAD_STATUS="blocked"
|
||||
LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance"
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS="full_capture"
|
||||
SCENE_LINE_COUNT="3"
|
||||
GENERATED_AT="2026-04-19 03:31:26"
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 04:31:45","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":3,"log_sync_enabled":true,"log_sync_state":"partial_coverage","log_sync_mode":"full","log_sync_covered_nodes":2,"log_sync_missing_node_codes":["overseas-control-01"],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention","log_sync_partial=2/3"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"node_code":"overseas-control-01","available":true,"status":"missing_sample","status_label":"缺少样本","status_type":"warning","summary":"当前还没有收到该参与节点的远端日志样本。","log_sync_enabled":true,"mode":"full","mode_label":"全量回传","records_total":0,"records_visible":0,"records_truncated":false,"records":[],"latest_record":{},"source_summary":{"node_code":"overseas-control-01","line_count":0,"key_line_count":0,"full_line_count":0,"last_at":"","last_line":""},"missing_reason_code":"no_sample","missing_reason":"当前还没有收到该参与节点的远端日志样本。","node":{"node_code":"overseas-control-01","region":"overseas","role":"control","status":"busy","current_load":25,"last_heartbeat_at":"2026-04-19 04:31:52"},"participation":{"detect_participating":true,"participation_state":"running","participation_label":"执行中","participation_reason":"当前正在执行 4 项检测任务。","participation_bucket":"dispatch_active","participation_bucket_label":"执行/已领","is_dispatch_active":true},"contract_navigation":{"detail_endpoint_pattern":"/api/v1/ops/contracts/{contract_key}","primary_contract_key":"ops_observability_contract","contract_keys":["ops_observability_contract","ops_stack_diagnosis_contract"],"contracts":[{"key":"ops_observability_contract","title":"Ops Observability Contract","status":"active","version":"v1","summary":"冻结 execution scene / inspection overview / activity stream / delivery queue 的正式观察面 contract。","primary_endpoint":"/api/v1/ops/overview","schema_doc_path":"docs/schemas/ops_observability_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_observability_contract","discovery_endpoints":["/api/v1/ops/overview","/api/v1/ops/inspection-overview","/api/v1/ops/activity-stream","/api/v1/ops/nodes/{node_code}/scene-log","/api/v1/ops/nodes/{node_code}/delivery-queue","/api/v1/ops/nodes/{node_code}/delivery-queue/records","/api/v1/ops/nodes/{node_code}/delivery-queue/flush","/api/v1/ops/nodes/{node_code}/delivery-queue/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay","/api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","ops_driver_contract","ops_playbook_contract","ops_stack_diagnosis_contract"]},{"key":"ops_stack_diagnosis_contract","title":"Ops Stack Diagnosis Contract","status":"active","version":"v1","summary":"冻结海外单脑总检入口的统一诊断 contract，供页面、CLI、Codex、按钮共享同一份第一现场判断。","primary_endpoint":"/api/v1/ops/stack-diagnosis","schema_doc_path":"docs/schemas/ops_stack_diagnosis_contract.md","detail_endpoint":"/api/v1/ops/contracts/ops_stack_diagnosis_contract","discovery_endpoints":["/api/v1/ops/go-live-summary","/api/v1/ops/stack-diagnosis","/api/v1/ops/contracts","/api/v1/ops/link-snapshot","/api/v1/ops/overview","/api/v1/ops/nodes","/api/v1/ops/releases/launchpad","/api/v1/ops/playbook-runs","/api/v1/ops/activity-stream"],"related_contract_keys":["ops_job_contract","ops_agent_protocol","release_hub_contract","ops_driver_contract","ops_playbook_contract","ops_observability_contract"]}]}},"detail_code":null}
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,16 @@
|
||||
CYCLES="5"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="partial_coverage"
|
||||
LOG_SYNC_MISSING_NODE_CODES="overseas-control-01"
|
||||
STACK_STATUS="attention"
|
||||
ISSUE_TOTAL="2"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES="remote_log_sync_waiting_sample,playbook_runs_need_attention"
|
||||
LAUNCHPAD_STATUS="blocked"
|
||||
LAUNCHPAD_RECOMMENDED_ACTION="run_acceptance"
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS="missing_sample"
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT="2026-04-19 04:31:45"
|
||||
@@ -0,0 +1,16 @@
|
||||
CYCLES="6"
|
||||
GO_LIVE_STATUS=""
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE=""
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS=""
|
||||
ISSUE_TOTAL="0"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES=""
|
||||
LAUNCHPAD_STATUS=""
|
||||
LAUNCHPAD_RECOMMENDED_ACTION=""
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS=""
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT=""
|
||||
@@ -0,0 +1,16 @@
|
||||
CYCLES="7"
|
||||
GO_LIVE_STATUS=""
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE=""
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS=""
|
||||
ISSUE_TOTAL="0"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES=""
|
||||
LAUNCHPAD_STATUS=""
|
||||
LAUNCHPAD_RECOMMENDED_ACTION=""
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS=""
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT=""
|
||||
@@ -0,0 +1 @@
|
||||
{"code":0,"message":"ok","data":{"base_url":"http://127.0.0.1:8100","generated_at":"2026-04-19 04:34:28","go_live_status":"attention","publish_ready":false,"publish_status":"attention","publish_status_label":"可发布但建议先复核","publish_summary":"当前没有硬阻断，但仍有上线前关注项，建议先完成复核再正式发版。","stack_status":"attention","contracts_ready":true,"contracts_total":11,"launchpad_status":"attention","launchpad_status_label":"待补执行面","launchpad_recommended_action_code":"fix_managed_nodes","launchpad_recommended_target_node_code":"","launchpad_recommended_recovery_label":"","launchpad_recommended_recovery_summary":"来自 overview.recommendation.primary_action_code","launchpad_onboarding_bootstrap_pending_nodes":0,"launchpad_onboarding_acceptance_ready_nodes":0,"route_surface_complete":true,"route_surface_missing_keys":[],"route_surface_declares_bootstrap_plan":true,"runtime_schema_stale":false,"repository_capabilities":{"supports_install_command_block":true,"supports_multi_layout_bootstrap":true},"managed_enabled":3,"remote_access_ready":3,"queue_dead_letter_nodes":0,"activity_start_delivery_issue_total":0,"participating_nodes_total":1,"log_sync_enabled":true,"log_sync_state":"full_capture","log_sync_mode":"full","log_sync_covered_nodes":1,"log_sync_missing_node_codes":[],"next_step_action_code":"focus_playbook_run","next_step_reason":"来自 overview.recommendation.primary_action_code","operator_lane":"ops_jobs","operator_title":"按总检默认下一步继续处理","operator_primary_command_key":"focus_playbook_run","publish_blocking_reasons":[],"publish_warnings":["stack_diagnosis=attention","release_launchpad=attention"],"blocking_reasons":[],"warnings":["stack_diagnosis=attention","release_launchpad=attention"],"recommended_commands":{"stack_summary":"bash domain-api/deploy/multi-region/check_ops_center_stack.sh http://127.0.0.1:8100 summary","contracts":"bash domain-api/deploy/multi-region/check_ops_contracts.sh http://127.0.0.1:8100","ops_plane":"bash domain-api/deploy/multi-region/check_ops_plane.sh http://127.0.0.1:8100","release_hub":"bash domain-api/deploy/multi-region/check_release_hub.sh http://127.0.0.1:8100","inspection":"bash domain-api/deploy/multi-region/check_ops_inspection.sh http://127.0.0.1:8100","overview":"bash domain-api/deploy/multi-region/drive_ops_center.sh overview http://127.0.0.1:8100","go_live_recover":"bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover http://127.0.0.1:8100","doctor_export":"bash domain-api/deploy/multi-region/drive_ops_center.sh doctor-export /tmp/domaincheck-go-live http://127.0.0.1:8100","next_step":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 focus_playbook_run","log_sync_logs":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 open_worker_logs_participating","log_sync_inspection":"bash domain-api/deploy/multi-region/drive_ops_center.sh driver-resolve http://127.0.0.1:8100 run_inspection_participating"},"source_refs":{"stack_diagnosis_contract_key":"ops_stack_diagnosis_contract","contracts_registry_version":"2026-04-18","runtime_build_commit_sha":"246838ae4c07","release_focus_ref":{"kind":"release_hub","release_id":2,"release_version":"domaincheck_release_20260418_013833","channel":"stable","rollout_id":0,"rollout_code":"","section":"release_launchpad"}}},"detail_code":null}
|
||||
@@ -0,0 +1,16 @@
|
||||
CYCLES="8"
|
||||
GO_LIVE_STATUS="attention"
|
||||
PUBLISH_READY="false"
|
||||
LOG_SYNC_STATE="full_capture"
|
||||
LOG_SYNC_MISSING_NODE_CODES=""
|
||||
STACK_STATUS=""
|
||||
ISSUE_TOTAL="0"
|
||||
BLOCKING_ISSUE_TOTAL="0"
|
||||
ISSUE_CODES=""
|
||||
LAUNCHPAD_STATUS=""
|
||||
LAUNCHPAD_RECOMMENDED_ACTION=""
|
||||
PROBLEM_RUNS_TOTAL="0"
|
||||
PROBLEM_RUN_CODE=""
|
||||
SCENE_STATUS=""
|
||||
SCENE_LINE_COUNT="0"
|
||||
GENERATED_AT="2026-04-19 04:34:28"
|
||||
@@ -0,0 +1,59 @@
|
||||
# night_run_20260419_012929 初步分析
|
||||
|
||||
## 结论
|
||||
|
||||
- 夜跑不是完全空跑，`cycle_1` 到 `cycle_8` 期间持续执行了巡检、收口和日志回传恢复动作。
|
||||
- 真正的中断点出现在 `2026-04-19 04:32:49` 到 `04:33:35`，本机 API `127.0.0.1:8100` 短时不可达，导致两轮自动动作直接失败。
|
||||
- 最终停止原因是 `signoff_ready_candidate`，这是“候选可签收”型收口，不等同于“整轮全绿、无异常结束”。
|
||||
|
||||
## 关键时间点
|
||||
|
||||
- `2026-04-19 01:29:29 +0800`
|
||||
- 夜跑启动。
|
||||
- `cycle_1` 到 `cycle_5`
|
||||
- 持续产出 `go_live.json`、`launchpad.json`、`playbook_runs.json`、`stack.json`、`scene_overseas_control_01.json`、`summary.env`。
|
||||
- `2026-04-19 04:32:04 +0800`
|
||||
- `run_inspection_participating` 成功创建 playbook，回执 `pbr-6d77b32f40`。
|
||||
- `2026-04-19 04:32:49 +0800`
|
||||
- `cycle=6`
|
||||
- `log-sync-recover` 调用失败。
|
||||
- `driver-run run_inspection_participating` 调用失败。
|
||||
- 错误为 `curl: (7) Failed to connect to 127.0.0.1 port 8100: Connection refused`。
|
||||
- `2026-04-19 04:33:34 +0800`
|
||||
- `cycle=7`
|
||||
- 同类动作再次失败，错误相同。
|
||||
- `2026-04-19 04:34:28 +0800`
|
||||
- `cycle=8`
|
||||
- `go_live=attention`
|
||||
- `log_sync=full_capture`
|
||||
- 夜跑停止，`reason=signoff_ready_candidate`。
|
||||
|
||||
## 已确认的问题
|
||||
|
||||
- 夜跑期间存在控制面 API 短时离线或重启窗口。
|
||||
- 自动恢复逻辑在 API 不可达时会直接失败，但日志里没有看到进一步的退避、跳过本轮、等待 API 恢复后的再确认闭环。
|
||||
- 当时的“可签收候选”判断，掺杂了 API 短时不可达窗口，所以不能把这次夜跑结论直接当成正式签收证据。
|
||||
|
||||
## 这轮修复后的关联状态
|
||||
|
||||
- 当前中央控制面已经恢复正常。
|
||||
- `runtime/cluster` 已恢复为 3 台有效执行节点。
|
||||
- `detect/status` 已恢复远端日志回传。
|
||||
- `mainland-controller-01` 当前已恢复 `100/100`。
|
||||
- `mainland-worker-01` 当前已进入活跃参与，中央已看到 `2/50`。
|
||||
|
||||
## 明天继续看时，优先检查
|
||||
|
||||
- `docs/ops_center_runtime/night_runs/night_run_20260419_012929.log`
|
||||
- 重点看 `04:32:49` 到 `04:34:28` 这段 API 拒绝连接窗口。
|
||||
- `docs/ops_center_runtime/night_runs/night_run_20260419_012929/cycle_8/go_live.json`
|
||||
- 确认 `go_live=attention` 的具体触发项。
|
||||
- `docs/ops_center_runtime/night_runs/night_run_20260419_012929_report.md`
|
||||
- 对照夜跑最终报告和原始日志，确认是否把“候选可签收”误当成“正式通过”。
|
||||
|
||||
## 下一步建议
|
||||
|
||||
- 补一条夜跑期间的 API 可用性守护：
|
||||
- 发现 `127.0.0.1:8100` 不可达时，不立刻继续推进收口动作，先等待 API 恢复后重试。
|
||||
- 把“候选可签收”和“正式可签收”拆开：
|
||||
- 避免在 API 短时重启窗口里出现假阳性收口。
|
||||
@@ -0,0 +1,44 @@
|
||||
# NIGHT RUN REPORT night_run_20260419_012929
|
||||
|
||||
- Base URL: `http://127.0.0.1:8100`
|
||||
- Deadline: `2026-04-20 12:00:00 +0800`
|
||||
- Stop Reason: `signoff_ready_candidate`
|
||||
- Cycles: `8`
|
||||
- Log Sync Recover Runs: `4`
|
||||
- Inspection Runs: `4`
|
||||
- Inspection Churn Runs: `0`
|
||||
- Quick Rechecks: `4`
|
||||
|
||||
## Final Snapshot
|
||||
|
||||
- `go_live_status = attention`
|
||||
- `publish_ready = false`
|
||||
- `log_sync_state = full_capture`
|
||||
- `issue_total = 0`
|
||||
- `problem_runs_total = 0`
|
||||
- `launchpad_status = `
|
||||
- `launchpad_recommended_action = `
|
||||
- `problem_run_code = `
|
||||
|
||||
## Summary JSON
|
||||
|
||||
```json
|
||||
{
|
||||
"cycles": 8,
|
||||
"go_live_status": "attention",
|
||||
"publish_ready": false,
|
||||
"log_sync_state": "full_capture",
|
||||
"log_sync_missing_node_codes": [],
|
||||
"stack_status": "",
|
||||
"issue_total": 0,
|
||||
"blocking_issue_total": 0,
|
||||
"issue_codes": [],
|
||||
"launchpad_status": "",
|
||||
"launchpad_recommended_action": "",
|
||||
"problem_runs_total": 0,
|
||||
"problem_run_code": "",
|
||||
"scene_status": "",
|
||||
"scene_line_count": 0,
|
||||
"generated_at": "2026-04-19 04:34:28"
|
||||
}
|
||||
```
|
||||
@@ -1,20 +1,18 @@
|
||||
{
|
||||
"cycles": 1,
|
||||
"cycles": 8,
|
||||
"go_live_status": "attention",
|
||||
"publish_ready": false,
|
||||
"log_sync_state": "full_capture",
|
||||
"log_sync_missing_node_codes": [],
|
||||
"stack_status": "attention",
|
||||
"issue_total": 1,
|
||||
"stack_status": "",
|
||||
"issue_total": 0,
|
||||
"blocking_issue_total": 0,
|
||||
"issue_codes": [
|
||||
"playbook_runs_need_attention"
|
||||
],
|
||||
"launchpad_status": "blocked",
|
||||
"launchpad_recommended_action": "run_acceptance",
|
||||
"issue_codes": [],
|
||||
"launchpad_status": "",
|
||||
"launchpad_recommended_action": "",
|
||||
"problem_runs_total": 0,
|
||||
"problem_run_code": "",
|
||||
"scene_status": "full_capture",
|
||||
"scene_line_count": 1,
|
||||
"generated_at": "2026-04-19 01:29:38"
|
||||
"scene_status": "",
|
||||
"scene_line_count": 0,
|
||||
"generated_at": "2026-04-19 04:34:28"
|
||||
}
|
||||
114
docs/test.md
Normal file
114
docs/test.md
Normal file
@@ -0,0 +1,114 @@
|
||||
开个定时任务不停扫数据库，未检测的 快到期删除的，全部扫出来推送给注册任务队列；
|
||||
|
||||
注册检测：筛选出可以注册的，这个完成后，不管成功失败，返会标准化结果给 controller，告诉他，这个流程我走完了，如果是外部原因导致未能出结果就失败的，controller 会重新投入注册检测，更新数据库，如果是成功的，把对应的状态更新数据库，跟新注册状态；
|
||||
|
||||
controller 收到返回注册完成后， 以标准化格式下一步任务队列里面 ，等worker 来啦取，
|
||||
worker 只负责去controller 啦取的任务，处理任务，根据任务标准标识，安排对于的函数处理，所以woker只复制啦和跑，我没任务了，我很有空，我就去controller 获取任务，只要你给，我就跑，跑完返回结果给你；
|
||||
controller 在收到worker的啦取请求后，按后台勾选的配置任务队列，按顺序返回给woker，不是随便返回
|
||||
controller 返回任务时，要同时完成认领
|
||||
也就是：
|
||||
标记该 task 已分配给某个 worker
|
||||
进入 running 状态
|
||||
设置超时 TTL
|
||||
超时未回传则回收重投
|
||||
否则会出现：
|
||||
worker 拿了任务挂了
|
||||
controller 以为还在跑
|
||||
任务永远丢了
|
||||
|
||||
|
||||
xx检测：当前 xx 步骤执行并返回判定结果，这个完成后，不管成功失败，返会标准化结果给controller，告诉他，这个xx流程我走完了，如果是外部原因导致失败的，还没有出结果的，非业务判定不通过被跑到为黑名单的，controller 会重新投入当前xx任务队列，如果名中黑名单，直接跟新黑名单状态，不在分发到后面所有步骤，更新数据库，如果是成功的，跟新当前xx检测状态，
|
||||
|
||||
controller 收到某步骤返回结果后，先根据当前 后台 配置和该域名已完成步骤状态，解析该域名在本次流程中的下一勾选步骤；若存在下一步骤，则以标准化任务格式投入对应任务队列；若不存在，则标记本次流程完成
|
||||
|
||||
只按顺序处理后台勾选的任务
|
||||
注册检测
|
||||
百度检测
|
||||
站长检测
|
||||
爱站检测
|
||||
时光机检测
|
||||
聚查检测
|
||||
桔子检测
|
||||
|
||||
备注：
|
||||
时光机具体还要细分方案 ，目前项目内已经又对于的方案，加一个直晒前最近5年的快照，先跑通，后面可以继续细优化
|
||||
步骤都是按后台勾选，把勾选的跑完就算流程跑完，第一次没勾选的，下次勾选，可以直接跑够选的步骤，其他跳过；
|
||||
|
||||
|
||||
|
||||
海外主库(域名源/最终账本)
|
||||
↓
|
||||
controller dispatcher 拉取待处理域名
|
||||
↓
|
||||
按 pipeline 配置生成首个待执行步骤任务
|
||||
↓
|
||||
推入 Redis 对应步骤队列
|
||||
↓
|
||||
worker 向 controller 拉取任务
|
||||
↓
|
||||
controller 认领并分配任务给 worker
|
||||
↓
|
||||
worker 执行检测并回传标准化结果
|
||||
↓
|
||||
controller stage-processor 更新本地控制状态
|
||||
↓
|
||||
根据结果判断：
|
||||
- retry -> 重投当前步骤
|
||||
- black_hit -> 拉黑并终止
|
||||
- reject -> 终止/复核
|
||||
- pass -> 解析下一勾选步骤并投递
|
||||
↓
|
||||
controller syncer 批量同步海外主库
|
||||
↓
|
||||
controller finalizer 标记本次流程完成
|
||||
|
||||
|
||||
|
||||
|
||||
域名检测流程设计
|
||||
1. 总体原则
|
||||
海外主库负责域名源数据与最终账本持久化，不参与高频实时调度
|
||||
controller 负责任务编排、任务分发、结果处理、状态推进与批量同步
|
||||
Redis 负责各步骤待执行队列、运行中任务、重试任务与去重控制
|
||||
worker 仅负责向 controller 拉取任务、执行对应检测函数、回传标准化结果
|
||||
2. 注册任务投递
|
||||
定时任务持续扫描海外主库中的待检测域名，包括未检测域名、快到期删除域名等
|
||||
controller dispatcher 将符合条件的域名按标准化任务格式推入注册任务队列
|
||||
3. 步骤执行规则
|
||||
|
||||
每个检测步骤执行完成后，无论结果如何，worker 均需返回标准化结果给 controller。
|
||||
controller 根据结果类型做统一处理：
|
||||
|
||||
若为外部原因导致未得到有效结果，则根据重试策略重新投入当前步骤队列
|
||||
若为业务判定不通过，则更新当前步骤状态，并按策略终止本次流程
|
||||
若命中黑名单，则更新黑名单状态并终止后续所有步骤
|
||||
若业务判定通过，则更新当前步骤状态，并根据本次 pipeline 配置解析下一勾选步骤，投入对应任务队列
|
||||
4. pipeline 推进规则
|
||||
所有步骤按后台勾选生成本次 pipeline
|
||||
controller 仅按勾选顺序为单个域名推进下一步骤
|
||||
若某步骤在本次 pipeline 中未勾选，则直接跳过
|
||||
若前次未勾选、后次新增勾选，则可直接从已完成状态之后继续补跑，无需重跑已完成步骤
|
||||
5. worker 拉取规则
|
||||
worker 空闲时向 controller 发起拉取任务请求
|
||||
controller 根据任务队列状态、任务顺序与后台配置返回当前可执行任务
|
||||
worker 只负责执行任务，不负责流程判断、不负责决定下一步骤、不直接高频写海外主库
|
||||
6. 状态更新规则
|
||||
controller stage-processor 实时更新本地控制状态
|
||||
controller syncer 以批量方式将步骤状态、黑名单状态、流程状态同步至海外主库
|
||||
controller finalizer 在本次 pipeline 所有勾选步骤完成或流程被终止后，标记流程完成
|
||||
7. 当前步骤顺序
|
||||
|
||||
按后台勾选顺序处理以下任务：
|
||||
|
||||
注册检测
|
||||
百度检测
|
||||
站长检测
|
||||
爱站检测
|
||||
时光机检测
|
||||
聚查检测
|
||||
桔子检测
|
||||
8. 时光机一期方案
|
||||
当前先按最近 5 年快照执行简化方案，先跑通主流程
|
||||
后续再继续细化为更完整的时光机子流程
|
||||
|
||||
controller 机器如果性能足够剩余也可以部署worker 跑，
|
||||
@@ -4,6 +4,7 @@ from uuid import uuid4
|
||||
|
||||
from fastapi import APIRouter
|
||||
|
||||
from app.core.config import settings
|
||||
from app.schemas.common import ApiResponse
|
||||
from app.services.detect_job_service import (
|
||||
append_detect_job_event,
|
||||
@@ -12,6 +13,7 @@ from app.services.detect_job_service import (
|
||||
get_detect_job_summary,
|
||||
get_detect_queue_health,
|
||||
list_detect_jobs,
|
||||
normalize_detect_step_code,
|
||||
)
|
||||
from app.services.detect_service import get_detect_status
|
||||
from app.services.detect_run_service import create_detect_run_snapshot, finalize_detect_run, mark_detect_run_stopping
|
||||
@@ -22,6 +24,13 @@ from app.services.worker_control_service import send_worker_command, start_worke
|
||||
router = APIRouter(tags=["detect"])
|
||||
|
||||
|
||||
def _local_worker_expected_on_this_node() -> bool:
|
||||
return not (
|
||||
str(settings.node_region or "").strip() == "overseas"
|
||||
and str(settings.node_role or "").strip() == "control"
|
||||
)
|
||||
|
||||
|
||||
def _build_detect_action_result(
|
||||
*,
|
||||
action: str,
|
||||
@@ -255,6 +264,11 @@ def detect_queue_summary(window_minutes: int = 15) -> ApiResponse:
|
||||
return ApiResponse(data=get_detect_queue_health(window_minutes=window_minutes))
|
||||
|
||||
|
||||
@router.get("/detect/queue-health", response_model=ApiResponse)
|
||||
def detect_queue_health_alias(window_minutes: int = 15) -> ApiResponse:
|
||||
return ApiResponse(data=get_detect_queue_health(window_minutes=window_minutes))
|
||||
|
||||
|
||||
@router.get("/detect/jobs/{job_id}", response_model=ApiResponse)
|
||||
def detect_job_detail(job_id: int) -> ApiResponse:
|
||||
data = get_detect_job_summary(job_id, event_limit=100)
|
||||
@@ -264,8 +278,17 @@ def detect_job_detail(job_id: int) -> ApiResponse:
|
||||
|
||||
|
||||
@router.post("/detect/start", response_model=ApiResponse)
|
||||
def start_detect() -> ApiResponse:
|
||||
job_summary = create_detect_job_if_needed(limit=1000, created_by="api")
|
||||
def start_detect(step_code: str | None = None) -> ApiResponse:
|
||||
normalized_step_code = normalize_detect_step_code(step_code)
|
||||
if step_code and not normalized_step_code:
|
||||
result = _build_detect_action_result(
|
||||
action="start",
|
||||
ok=False,
|
||||
message=f"暂不支持步骤任务: {step_code}",
|
||||
data={"job": None, "step_code": str(step_code or "").strip()},
|
||||
)
|
||||
return ApiResponse(code=1, message=f"暂不支持步骤任务: {step_code}", data=result)
|
||||
job_summary = create_detect_job_if_needed(limit=10000, created_by="api", step_code=step_code)
|
||||
if not job_summary:
|
||||
result = _build_detect_action_result(
|
||||
action="start",
|
||||
@@ -292,42 +315,58 @@ def start_detect() -> ApiResponse:
|
||||
},
|
||||
)
|
||||
|
||||
ok, message = start_worker()
|
||||
if not ok:
|
||||
result = _build_detect_action_result(
|
||||
action="start",
|
||||
ok=False,
|
||||
message=message,
|
||||
data={"job": job_summary},
|
||||
local_worker_expected = _local_worker_expected_on_this_node()
|
||||
if local_worker_expected:
|
||||
ok, message = start_worker()
|
||||
if not ok:
|
||||
result = _build_detect_action_result(
|
||||
action="start",
|
||||
ok=False,
|
||||
message=message,
|
||||
data={"job": job_summary},
|
||||
)
|
||||
append_detect_job_event(
|
||||
job_summary["job_id"],
|
||||
event_type="job_dispatch_failed",
|
||||
level="error",
|
||||
message=f"启动 Worker 失败: {message}",
|
||||
payload={"cycle_token": cycle_token},
|
||||
)
|
||||
return ApiResponse(
|
||||
code=1,
|
||||
message=message,
|
||||
data=result,
|
||||
)
|
||||
|
||||
command_ok, command_message = send_worker_command(
|
||||
"start_detection",
|
||||
payload={
|
||||
"cycle_token": cycle_token,
|
||||
"job_id": job_summary["job_id"],
|
||||
"job_code": job_summary["job_code"],
|
||||
"task_mode": job_summary.get("task_mode", ""),
|
||||
"step_code": job_summary.get("step_code", ""),
|
||||
},
|
||||
)
|
||||
append_detect_job_event(
|
||||
job_summary["job_id"],
|
||||
event_type="job_dispatch_failed",
|
||||
level="error",
|
||||
message=f"启动 Worker 失败: {message}",
|
||||
event_type="job_dispatch_sent" if command_ok else "job_dispatch_rejected",
|
||||
level="info" if command_ok else "error",
|
||||
message=command_message,
|
||||
payload={"cycle_token": cycle_token},
|
||||
)
|
||||
return ApiResponse(
|
||||
code=1,
|
||||
message=message,
|
||||
data=result,
|
||||
else:
|
||||
ok = True
|
||||
message = "当前节点为海外控制面，仅负责派单，不启动本机 Worker"
|
||||
command_ok = True
|
||||
command_message = "已跳过本机 Worker 启动，转为仅向大陆执行节点派发检测动作"
|
||||
append_detect_job_event(
|
||||
job_summary["job_id"],
|
||||
event_type="job_dispatch_skipped_local",
|
||||
level="info",
|
||||
message=command_message,
|
||||
payload={"cycle_token": cycle_token},
|
||||
)
|
||||
|
||||
command_ok, command_message = send_worker_command(
|
||||
"start_detection",
|
||||
payload={
|
||||
"cycle_token": cycle_token,
|
||||
"job_id": job_summary["job_id"],
|
||||
"job_code": job_summary["job_code"],
|
||||
},
|
||||
)
|
||||
append_detect_job_event(
|
||||
job_summary["job_id"],
|
||||
event_type="job_dispatch_sent" if command_ok else "job_dispatch_rejected",
|
||||
level="info" if command_ok else "error",
|
||||
message=command_message,
|
||||
payload={"cycle_token": cycle_token},
|
||||
)
|
||||
snapshot = get_detect_status()
|
||||
settings_payload = get_settings_payload()
|
||||
settings_summary = _build_settings_summary(settings_payload)
|
||||
|
||||
@@ -9,6 +9,7 @@ from app.services.ops_agent_service import (
|
||||
agent_heartbeat,
|
||||
agent_mark_job_started,
|
||||
agent_pull_jobs,
|
||||
agent_pull_runtime_config,
|
||||
agent_register,
|
||||
build_node_agent_bootstrap_plan,
|
||||
issue_node_agent_token,
|
||||
@@ -72,6 +73,12 @@ def ops_agent_pull(payload: dict, limit: int = 1, x_domaincheck_agent_token: Opt
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/runtime-config", response_model=ApiResponse)
|
||||
def ops_agent_runtime_config(payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = agent_pull_runtime_config(payload, token=_resolve_agent_token(x_domaincheck_agent_token))
|
||||
return _build_agent_response(ok, message, data)
|
||||
|
||||
|
||||
@router.post("/ops/agent/jobs/{job_id}/start", response_model=ApiResponse)
|
||||
def ops_agent_job_start(job_id: int, payload: dict, x_domaincheck_agent_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = agent_mark_job_started(job_id, payload, token=_resolve_agent_token(x_domaincheck_agent_token))
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Header
|
||||
from fastapi import APIRouter, Body, Header
|
||||
|
||||
from app.schemas.common import ApiResponse
|
||||
from app.services.build_info_service import get_runtime_build_info
|
||||
from app.services.cluster_runtime_service import get_cluster_snapshot
|
||||
from app.services.debug_event_service import get_debug_diagnosis, get_debug_event_overview, get_debug_handoff_report, ingest_debug_event, list_debug_events
|
||||
from app.services.detect_job_service import get_detect_queue_health
|
||||
from app.services.runtime_control_service import runtime_action
|
||||
from app.services.runtime_status_service import get_runtime_preflight, get_runtime_status
|
||||
from app.services.sync_push_service import (
|
||||
@@ -103,6 +104,36 @@ def runtime_debug_handoff(window_minutes: int = 10, source_region: Optional[str]
|
||||
)
|
||||
|
||||
|
||||
@router.get("/runtime/queue-health", response_model=ApiResponse)
|
||||
def runtime_queue_health(window_minutes: int = 15) -> ApiResponse:
|
||||
return ApiResponse(data=get_detect_queue_health(window_minutes=window_minutes))
|
||||
|
||||
|
||||
@router.get("/runtime/health-handover", response_model=ApiResponse)
|
||||
def runtime_health_handover(
|
||||
node_code: Optional[str] = None,
|
||||
window_minutes: int = 10,
|
||||
source_region: Optional[str] = None,
|
||||
) -> ApiResponse:
|
||||
sync_summary = get_sync_summary()
|
||||
runtime_status_payload = get_runtime_status()
|
||||
data = get_debug_handoff_report(
|
||||
window_minutes=window_minutes,
|
||||
source_region=source_region,
|
||||
sync_summary=sync_summary,
|
||||
readiness=runtime_status_payload.get("readiness") or {},
|
||||
)
|
||||
normalized_node_code = str(node_code or "").strip()
|
||||
if normalized_node_code:
|
||||
data = dict(data)
|
||||
data["nodes"] = [
|
||||
item
|
||||
for item in list(data.get("nodes") or [])
|
||||
if str(item.get("node_code") or "").strip() == normalized_node_code
|
||||
]
|
||||
return ApiResponse(data=data)
|
||||
|
||||
|
||||
@router.post("/runtime/sync-ingest", response_model=ApiResponse)
|
||||
def runtime_sync_ingest(payload: dict, x_domaincheck_sync_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = ingest_runtime_projection(payload, shared_token=x_domaincheck_sync_token)
|
||||
@@ -110,7 +141,7 @@ def runtime_sync_ingest(payload: dict, x_domaincheck_sync_token: Optional[str] =
|
||||
|
||||
|
||||
@router.get("/runtime/task-export", response_model=ApiResponse)
|
||||
def runtime_task_export(limit: int = 200, x_domaincheck_sync_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
def runtime_task_export(limit: int = 1000, x_domaincheck_sync_token: Optional[str] = Header(default=None)) -> ApiResponse:
|
||||
ok, message, data = export_detect_task_projection(limit=limit, shared_token=x_domaincheck_sync_token)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
@@ -128,6 +159,6 @@ def runtime_debug_ingest(payload: dict, x_domaincheck_sync_token: Optional[str]
|
||||
|
||||
|
||||
@router.post("/runtime/actions/{action}", response_model=ApiResponse)
|
||||
def runtime_action_trigger(action: str) -> ApiResponse:
|
||||
ok, message, data = runtime_action(action)
|
||||
def runtime_action_trigger(action: str, payload: Optional[dict] = Body(default=None)) -> ApiResponse:
|
||||
ok, message, data = runtime_action(action, payload=payload)
|
||||
return ApiResponse(code=0 if ok else 1, message=message, data=data)
|
||||
|
||||
@@ -40,8 +40,12 @@ class Settings(BaseSettings):
|
||||
sync_target_region: str = "overseas"
|
||||
sync_target_api_base_url: str = ""
|
||||
sync_shared_token: str = ""
|
||||
sync_batch_size: int = 200
|
||||
sync_poll_interval_seconds: int = 30
|
||||
sync_batch_size: int = 5000
|
||||
sync_poll_interval_seconds: int = 2
|
||||
sync_pipeline_process_limit: int = 5000
|
||||
sync_pull_max_pending_items: int = 0
|
||||
sync_pull_max_register_pending_items: int = 0
|
||||
sync_pull_max_downstream_pending_items: int = 0
|
||||
build_manifest_path: str = ""
|
||||
build_commit_sha: str = ""
|
||||
build_commit_ref: str = ""
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
from contextlib import contextmanager
|
||||
from functools import wraps
|
||||
import time
|
||||
|
||||
import psycopg2
|
||||
from psycopg2 import errors
|
||||
|
||||
from app.core.config import settings
|
||||
|
||||
@@ -18,3 +21,39 @@ def get_db():
|
||||
yield conn
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
_RETRYABLE_READ_ERRORS = (
|
||||
errors.DeadlockDetected,
|
||||
errors.SerializationFailure,
|
||||
errors.LockNotAvailable,
|
||||
)
|
||||
|
||||
|
||||
def is_retryable_read_error(exc: Exception) -> bool:
|
||||
return isinstance(exc, _RETRYABLE_READ_ERRORS)
|
||||
|
||||
|
||||
def is_retryable_db_error(exc: Exception) -> bool:
|
||||
return isinstance(exc, _RETRYABLE_READ_ERRORS)
|
||||
|
||||
|
||||
def db_read_retry(*, attempts: int = 3, initial_delay_seconds: float = 0.05, backoff: float = 2.0):
|
||||
def decorator(func):
|
||||
@wraps(func)
|
||||
def wrapper(*args, **kwargs):
|
||||
delay = max(0.0, float(initial_delay_seconds or 0.0))
|
||||
for attempt in range(1, max(1, int(attempts or 1)) + 1):
|
||||
try:
|
||||
return func(*args, **kwargs)
|
||||
except Exception as exc:
|
||||
if not is_retryable_read_error(exc) or attempt >= max(1, int(attempts or 1)):
|
||||
raise
|
||||
if delay > 0:
|
||||
time.sleep(delay)
|
||||
delay *= max(1.0, float(backoff or 1.0))
|
||||
return func(*args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
return decorator
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
@@ -63,9 +64,35 @@ def tail_lines(relative_path: str, max_lines: int = 120) -> list[str]:
|
||||
|
||||
|
||||
def runtime_root() -> Path:
|
||||
path = Path(__file__).resolve().parents[2] / "runtime"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
env_override = str(os.getenv("DOMAIN_API_RUNTIME_ROOT", "") or "").strip()
|
||||
candidates: list[Path] = []
|
||||
if env_override:
|
||||
candidates.append(Path(env_override))
|
||||
|
||||
base_dir = Path(__file__).resolve().parents[2]
|
||||
for parent in base_dir.parents:
|
||||
if parent.name != "releases":
|
||||
continue
|
||||
# Released builds live under /opt/domaincheck/releases/<release>/domain-api.
|
||||
# Runtime state must not be written back into the immutable release tree,
|
||||
# otherwise sync-agent / detect runtime snapshots fail with permission errors.
|
||||
candidates.append(parent.parent / "runtime" / "domain-api")
|
||||
break
|
||||
|
||||
candidates.append(base_dir / "runtime")
|
||||
|
||||
last_error: OSError | None = None
|
||||
for candidate in candidates:
|
||||
try:
|
||||
candidate.mkdir(parents=True, exist_ok=True)
|
||||
return candidate
|
||||
except OSError as exc:
|
||||
last_error = exc
|
||||
continue
|
||||
|
||||
if last_error is not None:
|
||||
raise last_error
|
||||
raise RuntimeError("failed to resolve runtime root")
|
||||
|
||||
|
||||
def read_runtime_json(filename: str, default: dict | list | None = None):
|
||||
@@ -164,7 +191,16 @@ def load_detect_records() -> list[dict]:
|
||||
try:
|
||||
with path.open("r", encoding="utf-8") as handle:
|
||||
return json.load(handle)
|
||||
except (json.JSONDecodeError, OSError):
|
||||
except (json.JSONDecodeError, UnicodeDecodeError, OSError):
|
||||
try:
|
||||
raw = path.read_bytes()
|
||||
text = raw.decode("utf-8", errors="replace")
|
||||
decoder = json.JSONDecoder()
|
||||
payload, _ = decoder.raw_decode(text)
|
||||
if isinstance(payload, list):
|
||||
return payload
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
|
||||
|
||||
|
||||
@@ -1,13 +1,16 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import traceback
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime
|
||||
from urllib.parse import urlparse
|
||||
from uuid import uuid4
|
||||
|
||||
from app.services.ops_action_executor_core import (
|
||||
@@ -28,6 +31,44 @@ NODE_CODE = str(os.getenv("NODE_CODE", "")).strip()
|
||||
NODE_REGION = str(os.getenv("NODE_REGION", "mainland")).strip() or "mainland"
|
||||
NODE_ROLE = str(os.getenv("NODE_ROLE", "worker")).strip() or "worker"
|
||||
AGENT_POLL_INTERVAL_SECONDS = max(2, int(os.getenv("OPS_AGENT_POLL_INTERVAL_SECONDS", "5") or 5))
|
||||
AGENT_RUNTIME_CONFIG_SYNC_INTERVAL_SECONDS = max(
|
||||
10,
|
||||
int(os.getenv("OPS_AGENT_RUNTIME_CONFIG_SYNC_INTERVAL_SECONDS", "30") or 30),
|
||||
)
|
||||
AGENT_HTTP_TIMEOUT_SECONDS = max(10, int(os.getenv("OPS_AGENT_HTTP_TIMEOUT_SECONDS", "60") or 60))
|
||||
AGENT_REGISTER_TIMEOUT_SECONDS = max(
|
||||
10,
|
||||
int(os.getenv("OPS_AGENT_REGISTER_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS)) or AGENT_HTTP_TIMEOUT_SECONDS),
|
||||
)
|
||||
AGENT_HEARTBEAT_TIMEOUT_SECONDS = max(
|
||||
10,
|
||||
int(os.getenv("OPS_AGENT_HEARTBEAT_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS)) or AGENT_HTTP_TIMEOUT_SECONDS),
|
||||
)
|
||||
AGENT_PULL_TIMEOUT_SECONDS = max(
|
||||
10,
|
||||
int(os.getenv("OPS_AGENT_PULL_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS)) or AGENT_HTTP_TIMEOUT_SECONDS),
|
||||
)
|
||||
AGENT_RUNTIME_CONFIG_TIMEOUT_SECONDS = max(
|
||||
10,
|
||||
int(
|
||||
os.getenv("OPS_AGENT_RUNTIME_CONFIG_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS))
|
||||
or AGENT_HTTP_TIMEOUT_SECONDS
|
||||
),
|
||||
)
|
||||
AGENT_JOB_COMPLETE_TIMEOUT_SECONDS = max(
|
||||
10,
|
||||
int(
|
||||
os.getenv("OPS_AGENT_JOB_COMPLETE_TIMEOUT_SECONDS", str(AGENT_HTTP_TIMEOUT_SECONDS))
|
||||
or AGENT_HTTP_TIMEOUT_SECONDS
|
||||
),
|
||||
)
|
||||
AGENT_JOB_EVENT_TIMEOUT_SECONDS = max(
|
||||
10,
|
||||
int(
|
||||
os.getenv("OPS_AGENT_JOB_EVENT_TIMEOUT_SECONDS", str(max(15, AGENT_HTTP_TIMEOUT_SECONDS // 2)))
|
||||
or max(15, AGENT_HTTP_TIMEOUT_SECONDS // 2)
|
||||
),
|
||||
)
|
||||
WORKER_SERVICE_NAME = str(os.getenv("WORKER_SERVICE_NAME", os.getenv("WORKER_SERVICE", "domaincheck-worker"))).strip() or "domaincheck-worker"
|
||||
API_SERVICE_NAME = str(os.getenv("API_SERVICE_NAME", "domaincheck-api")).strip() or "domaincheck-api"
|
||||
SYNC_AGENT_SERVICE_NAME = str(os.getenv("SYNC_AGENT_SERVICE_NAME", "domaincheck-sync-agent")).strip() or "domaincheck-sync-agent"
|
||||
@@ -50,6 +91,7 @@ _LAST_QUEUE_FLUSH_SUMMARY = {
|
||||
"dead_letter": 0,
|
||||
"last_flush_at": "",
|
||||
}
|
||||
_LAST_RUNTIME_CONFIG_HASH = ""
|
||||
|
||||
|
||||
def _normalize_text_list(raw_value: object) -> list[str]:
|
||||
@@ -98,6 +140,7 @@ AGENT_CAPABILITIES = _json_env(
|
||||
"runtime.restart_api",
|
||||
"runtime.start_sync_agent",
|
||||
"runtime.stop_sync_agent",
|
||||
"runtime.reset_lab_state",
|
||||
"health.snapshot",
|
||||
"logs.collect",
|
||||
"diagnostics.collect",
|
||||
@@ -601,7 +644,7 @@ def _headers() -> dict[str, str]:
|
||||
}
|
||||
|
||||
|
||||
def _request(method: str, path: str, payload: dict | None = None, timeout: int = 30) -> dict:
|
||||
def _request(method: str, path: str, payload: dict | None = None, timeout: int = AGENT_HTTP_TIMEOUT_SECONDS) -> dict:
|
||||
if not CONTROL_PLANE_BASE_URL:
|
||||
raise RuntimeError("OPS_CONTROL_PLANE_BASE_URL 未配置")
|
||||
if not AGENT_TOKEN:
|
||||
@@ -614,22 +657,286 @@ def _request(method: str, path: str, payload: dict | None = None, timeout: int =
|
||||
return json.loads(body or "{}")
|
||||
|
||||
|
||||
def _post(path: str, payload: dict, timeout: int = 30) -> dict:
|
||||
def _post(path: str, payload: dict, timeout: int = AGENT_HTTP_TIMEOUT_SECONDS) -> dict:
|
||||
return _request("POST", path, payload, timeout=timeout)
|
||||
|
||||
|
||||
def _hostname() -> str:
|
||||
def _pull_runtime_config(timeout: int = AGENT_RUNTIME_CONFIG_TIMEOUT_SECONDS) -> dict:
|
||||
response = _post("/api/v1/ops/agent/runtime-config", {"node_code": NODE_CODE}, timeout=timeout)
|
||||
_ensure_ok_response(response, "agent runtime config pull failed")
|
||||
data = response.get("data") or {}
|
||||
return dict(data.get("bundle") or {})
|
||||
|
||||
|
||||
def _publish_local_config_update(config_type: str) -> None:
|
||||
try:
|
||||
return socket.gethostname()
|
||||
from app.core.redis_client import get_redis
|
||||
|
||||
redis_client = get_redis()
|
||||
redis_client.publish("domain_tool:config_update", str(config_type or "").strip() or "config")
|
||||
except Exception:
|
||||
return ""
|
||||
return
|
||||
|
||||
|
||||
def _apply_runtime_config(bundle: dict) -> bool:
|
||||
global _LAST_RUNTIME_CONFIG_HASH
|
||||
|
||||
normalized_bundle = dict(bundle or {})
|
||||
bundle_hash = str(normalized_bundle.get("config_hash") or "").strip()
|
||||
if not bundle_hash:
|
||||
bundle_hash = hashlib.sha256(
|
||||
json.dumps(normalized_bundle, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
).hexdigest()
|
||||
if bundle_hash and bundle_hash == _LAST_RUNTIME_CONFIG_HASH:
|
||||
return False
|
||||
|
||||
from app.core.files import write_json
|
||||
from app.services.runtime_settings_service import update_runtime_settings
|
||||
from app.services.sensitive_words_service import save_sensitive_words_payload
|
||||
|
||||
detect_options = dict(normalized_bundle.get("detect_options") or {})
|
||||
proxy_config = dict(normalized_bundle.get("proxy_config") or {})
|
||||
thread_count = int(normalized_bundle.get("thread_count", 2) or 2)
|
||||
node_thread_counts = dict(normalized_bundle.get("node_thread_counts") or {})
|
||||
runtime_settings = dict(normalized_bundle.get("runtime_settings") or {})
|
||||
sensitive_words = dict(normalized_bundle.get("sensitive_words") or {})
|
||||
sensitive_words_text = str(sensitive_words.get("text") or "")
|
||||
sensitive_word_items = list(sensitive_words.get("items") or [])
|
||||
|
||||
write_json("detect_options.json", detect_options)
|
||||
write_json("proxy_config.json", proxy_config)
|
||||
write_json("thread_count.json", {"thread_count": str(thread_count)})
|
||||
write_json("node_thread_counts.json", node_thread_counts)
|
||||
write_json("runtime_settings.json", runtime_settings)
|
||||
write_json("runtime/runtime_settings.json", runtime_settings)
|
||||
write_json(
|
||||
"runtime/sensitive_words.json",
|
||||
{
|
||||
"items": sensitive_word_items,
|
||||
"text": sensitive_words_text,
|
||||
"total": int(sensitive_words.get("total", 0) or 0),
|
||||
},
|
||||
)
|
||||
|
||||
try:
|
||||
update_runtime_settings(runtime_settings)
|
||||
except Exception as exc:
|
||||
_log(f"runtime settings local api sync skipped: {exc}")
|
||||
|
||||
try:
|
||||
save_sensitive_words_payload({"text": sensitive_words_text})
|
||||
except Exception as exc:
|
||||
_log(f"sensitive words db sync skipped: {exc}")
|
||||
|
||||
try:
|
||||
from app.core.redis_client import get_redis
|
||||
|
||||
redis_client = get_redis()
|
||||
redis_client.set("domain_tool:detect_options", json.dumps(detect_options, ensure_ascii=False))
|
||||
redis_client.set("domain_tool:proxy_config", json.dumps(proxy_config, ensure_ascii=False))
|
||||
redis_client.set("domain_tool:thread_count", thread_count)
|
||||
redis_client.set("domain_tool:node_thread_counts", json.dumps(node_thread_counts, ensure_ascii=False))
|
||||
redis_client.set("domain_tool:runtime_settings", json.dumps(runtime_settings, ensure_ascii=False))
|
||||
redis_client.set("domain_tool:sensitive_words", json.dumps(sensitive_word_items, ensure_ascii=False))
|
||||
redis_client.publish("domain_tool:config_update", "detect_options")
|
||||
redis_client.publish("domain_tool:config_update", "proxy_config")
|
||||
redis_client.publish("domain_tool:config_update", "thread_count")
|
||||
redis_client.publish("domain_tool:config_update", "node_thread_counts")
|
||||
redis_client.publish("domain_tool:config_update", "runtime_settings")
|
||||
redis_client.publish("domain_tool:config_update", "sensitive_words")
|
||||
except Exception:
|
||||
_publish_local_config_update("detect_options")
|
||||
_publish_local_config_update("proxy_config")
|
||||
_publish_local_config_update("thread_count")
|
||||
_publish_local_config_update("node_thread_counts")
|
||||
_publish_local_config_update("runtime_settings")
|
||||
_publish_local_config_update("sensitive_words")
|
||||
|
||||
_LAST_RUNTIME_CONFIG_HASH = bundle_hash
|
||||
_log(
|
||||
"runtime config applied: "
|
||||
f"thread_count={thread_count} "
|
||||
f"node_override={node_thread_counts.get(NODE_CODE)} "
|
||||
f"sensitive_words={int(sensitive_words.get('total', 0) or 0)}"
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def _hostname() -> str:
|
||||
generic_values = {"localhost", "localhost.localdomain", "ip6-localhost", "localhost6"}
|
||||
candidates: list[str] = []
|
||||
try:
|
||||
candidates.append(socket.gethostname())
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
candidates.append(socket.getfqdn())
|
||||
except Exception:
|
||||
pass
|
||||
try:
|
||||
candidates.append(os.uname().nodename)
|
||||
except Exception:
|
||||
pass
|
||||
candidates.append(str(os.getenv("HOSTNAME", "")).strip())
|
||||
|
||||
for value in candidates:
|
||||
normalized = str(value or "").strip()
|
||||
if not normalized:
|
||||
continue
|
||||
if normalized.lower() in generic_values:
|
||||
continue
|
||||
return normalized
|
||||
return NODE_CODE or ""
|
||||
|
||||
|
||||
def _control_plane_host_port() -> tuple[str, int]:
|
||||
parsed = urlparse(CONTROL_PLANE_BASE_URL if "://" in CONTROL_PLANE_BASE_URL else f"http://{CONTROL_PLANE_BASE_URL}")
|
||||
host = str(parsed.hostname or "").strip()
|
||||
if not host:
|
||||
return "", 0
|
||||
port = int(parsed.port or (443 if parsed.scheme == "https" else 80))
|
||||
return host, port
|
||||
|
||||
|
||||
def _first_non_loopback_ip(values: list[str]) -> str:
|
||||
for value in values:
|
||||
normalized = str(value or "").strip()
|
||||
if not normalized or normalized.startswith("127.") or normalized == "::1":
|
||||
continue
|
||||
return normalized
|
||||
return ""
|
||||
|
||||
|
||||
def _ip() -> str:
|
||||
explicit_ip = str(os.getenv("OPS_AGENT_IP", "")).strip()
|
||||
if explicit_ip:
|
||||
return explicit_ip
|
||||
|
||||
host, port = _control_plane_host_port()
|
||||
if host and port:
|
||||
for family in (socket.AF_INET, socket.AF_INET6):
|
||||
sock = None
|
||||
try:
|
||||
sock = socket.socket(family, socket.SOCK_DGRAM)
|
||||
sock.connect((host, port))
|
||||
local_ip = str(sock.getsockname()[0] or "").strip()
|
||||
if local_ip and not local_ip.startswith("127.") and local_ip != "::1":
|
||||
return local_ip
|
||||
except Exception:
|
||||
pass
|
||||
finally:
|
||||
if sock is not None:
|
||||
try:
|
||||
sock.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
hostname = _hostname()
|
||||
if hostname:
|
||||
try:
|
||||
addrinfo = socket.getaddrinfo(hostname, None)
|
||||
resolved_ips = [str(item[4][0] or "").strip() for item in addrinfo if item and len(item) >= 5]
|
||||
best_ip = _first_non_loopback_ip(resolved_ips)
|
||||
if best_ip:
|
||||
return best_ip
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
return socket.gethostbyname(socket.gethostname())
|
||||
fallback_ip = str(socket.gethostbyname(socket.gethostname()) or "").strip()
|
||||
if fallback_ip and not fallback_ip.startswith("127."):
|
||||
return fallback_ip
|
||||
except Exception:
|
||||
return ""
|
||||
pass
|
||||
return ""
|
||||
|
||||
|
||||
def _detect_runtime_snapshot() -> dict:
|
||||
worker_runtime = {}
|
||||
try:
|
||||
from app.services.worker_control_service import detect_worker_runtime
|
||||
|
||||
worker_runtime = detect_worker_runtime() or {}
|
||||
except Exception as exc:
|
||||
worker_runtime = {
|
||||
"running": False,
|
||||
"process_count": 0,
|
||||
"latest_start_time": "",
|
||||
"message": "",
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
try:
|
||||
from app.services.detect_service import get_detect_status
|
||||
|
||||
detect_status = get_detect_status() or {}
|
||||
runtime_state = detect_status.get("runtime_state") if isinstance(detect_status.get("runtime_state"), dict) else {}
|
||||
active_job = detect_status.get("active_job") if isinstance(detect_status.get("active_job"), dict) else {}
|
||||
|
||||
active_threads = max(0, int(detect_status.get("active_thread_count", 0) or 0))
|
||||
max_threads = max(0, int(detect_status.get("max_thread_count", 0) or 0))
|
||||
current_load = max(
|
||||
active_threads,
|
||||
int(active_job.get("items_running", 0) or 0),
|
||||
)
|
||||
phase_label = str(detect_status.get("phase_label") or "").strip()
|
||||
phase_detail = str(
|
||||
detect_status.get("phase_detail")
|
||||
or detect_status.get("recent_event")
|
||||
or detect_status.get("worker_runtime_message")
|
||||
or ""
|
||||
).strip()
|
||||
|
||||
inferred_worker_online = bool(
|
||||
worker_runtime.get("running", False)
|
||||
or detect_status.get("worker_online", False)
|
||||
or runtime_state.get("service_running", False)
|
||||
or current_load > 0
|
||||
or active_threads > 0
|
||||
)
|
||||
return {
|
||||
"worker_online": inferred_worker_online,
|
||||
"service_running": bool(
|
||||
runtime_state.get("service_running", False)
|
||||
or worker_runtime.get("running", False)
|
||||
or inferred_worker_online
|
||||
),
|
||||
"detecting": bool(
|
||||
detect_status.get("detecting", False)
|
||||
or runtime_state.get("detecting", False)
|
||||
or current_load > 0
|
||||
or active_threads > 0
|
||||
),
|
||||
"active_threads": active_threads,
|
||||
"max_threads": max_threads,
|
||||
"current_load": current_load,
|
||||
"phase_label": phase_label,
|
||||
"phase_detail": phase_detail,
|
||||
"recent_warning": str(detect_status.get("recent_warning") or "").strip(),
|
||||
"updated_at": str(runtime_state.get("updated_at") or "").strip(),
|
||||
"detect_participating": bool(
|
||||
detect_status.get("detect_participating", False)
|
||||
or current_load > 0
|
||||
or active_threads > 0
|
||||
),
|
||||
}
|
||||
except Exception as exc:
|
||||
worker_online = bool(worker_runtime.get("running", False))
|
||||
worker_message = str(worker_runtime.get("message") or "").strip()
|
||||
return {
|
||||
"worker_online": worker_online,
|
||||
"service_running": worker_online,
|
||||
"detecting": False,
|
||||
"active_threads": 0,
|
||||
"max_threads": 0,
|
||||
"current_load": 0,
|
||||
"phase_label": "",
|
||||
"phase_detail": worker_message,
|
||||
"recent_warning": "",
|
||||
"updated_at": str(worker_runtime.get("latest_start_time") or "").strip(),
|
||||
"detect_participating": False,
|
||||
"error": str(exc),
|
||||
}
|
||||
|
||||
|
||||
def _base_payload() -> dict:
|
||||
@@ -651,12 +958,37 @@ def _base_payload() -> dict:
|
||||
"node_agent": NODE_AGENT_SERVICE_NAME,
|
||||
},
|
||||
"delivery_queue": _delivery_queue_snapshot(),
|
||||
"detect_runtime": _detect_runtime_snapshot(),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _run(command: list[str], timeout: int = 60) -> tuple[int, str, str]:
|
||||
completed = subprocess.run(command, capture_output=True, text=True, timeout=timeout)
|
||||
normalized_command = [str(part or "").strip() for part in command]
|
||||
combined_output = f"{completed.stdout or ''}\n{completed.stderr or ''}".lower()
|
||||
needs_sudo_retry = (
|
||||
normalized_command
|
||||
and normalized_command[0] == "systemctl"
|
||||
and completed.returncode != 0
|
||||
and "sudo" not in normalized_command
|
||||
and any(
|
||||
marker in combined_output
|
||||
for marker in (
|
||||
"interactive authentication required",
|
||||
"authentication is required",
|
||||
"authorization not available",
|
||||
"polkit",
|
||||
)
|
||||
)
|
||||
)
|
||||
if needs_sudo_retry:
|
||||
completed = subprocess.run(
|
||||
["sudo", "-n", *normalized_command],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout,
|
||||
)
|
||||
return completed.returncode, completed.stdout.strip(), completed.stderr.strip()
|
||||
|
||||
|
||||
@@ -727,18 +1059,22 @@ def _execute_action(
|
||||
|
||||
|
||||
def _register() -> None:
|
||||
response = _post("/api/v1/ops/agent/register", _base_payload())
|
||||
response = _post("/api/v1/ops/agent/register", _base_payload(), timeout=AGENT_REGISTER_TIMEOUT_SECONDS)
|
||||
_ensure_ok_response(response, "agent register failed")
|
||||
_log(f"registered: {response.get('message')}")
|
||||
|
||||
|
||||
def _heartbeat() -> None:
|
||||
response = _post("/api/v1/ops/agent/heartbeat", _base_payload())
|
||||
response = _post("/api/v1/ops/agent/heartbeat", _base_payload(), timeout=AGENT_HEARTBEAT_TIMEOUT_SECONDS)
|
||||
_ensure_ok_response(response, "agent heartbeat failed")
|
||||
|
||||
|
||||
def _pull_jobs() -> list[dict]:
|
||||
response = _post(f"/api/v1/ops/agent/pull?limit=1", {"node_code": NODE_CODE})
|
||||
response = _post(
|
||||
f"/api/v1/ops/agent/pull?limit=1",
|
||||
{"node_code": NODE_CODE},
|
||||
timeout=AGENT_PULL_TIMEOUT_SECONDS,
|
||||
)
|
||||
_ensure_ok_response(response, "agent pull failed")
|
||||
data = response.get("data") or {}
|
||||
return list(data.get("jobs") or [])
|
||||
@@ -847,7 +1183,7 @@ def _job_complete(
|
||||
path=f"/api/v1/ops/agent/jobs/{job_id}/complete",
|
||||
payload=payload,
|
||||
request_id=request_id,
|
||||
timeout=30,
|
||||
timeout=AGENT_JOB_COMPLETE_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
|
||||
@@ -883,7 +1219,7 @@ def _job_event(
|
||||
path=f"/api/v1/ops/agent/jobs/{job_id}/events",
|
||||
payload=delivery_payload,
|
||||
request_id=request_id,
|
||||
timeout=15,
|
||||
timeout=AGENT_JOB_EVENT_TIMEOUT_SECONDS,
|
||||
)
|
||||
|
||||
|
||||
@@ -931,7 +1267,22 @@ def _process_job(job: dict) -> None:
|
||||
"start_delivery_error": start_delivery_error,
|
||||
},
|
||||
)
|
||||
ok, message, result = _execute_action(action, payload, job_id=job_id, job_context=normalized_job)
|
||||
try:
|
||||
ok, message, result = _execute_action(action, payload, job_id=job_id, job_context=normalized_job)
|
||||
except Exception as exc:
|
||||
ok = False
|
||||
message = f"executor exception: {exc}"
|
||||
result = {
|
||||
"stdout": "",
|
||||
"stderr": traceback.format_exc(),
|
||||
"summary_text": message,
|
||||
"exception_type": exc.__class__.__name__,
|
||||
}
|
||||
_log(
|
||||
"job execute exception: "
|
||||
f"id={job_id} code={normalized_job.get('job_code') or '-'} "
|
||||
f"action={action} error={exc}"
|
||||
)
|
||||
stdout = str(result.get("stdout") or "")
|
||||
stderr = str(result.get("stderr") or "")
|
||||
duration_ms = max(0, int((time.monotonic() - started_at) * 1000))
|
||||
@@ -956,6 +1307,57 @@ def _process_job(job: dict) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _process_job_with_guard(job: dict) -> None:
|
||||
normalized_job = _normalize_agent_job(job)
|
||||
job_id = int(normalized_job.get("job_id") or 0)
|
||||
action = str(normalized_job.get("action") or "").strip()
|
||||
started_at = time.monotonic()
|
||||
try:
|
||||
_process_job(normalized_job)
|
||||
except Exception as exc:
|
||||
duration_ms = max(0, int((time.monotonic() - started_at) * 1000))
|
||||
traceback_text = traceback.format_exc()
|
||||
summary_text = f"node-agent fatal exception: {exc}"
|
||||
_log(
|
||||
"job fatal exception: "
|
||||
f"id={job_id} code={normalized_job.get('job_code') or '-'} "
|
||||
f"action={action or '-'} error={exc}"
|
||||
)
|
||||
if job_id <= 0:
|
||||
raise
|
||||
try:
|
||||
delivery = _job_complete(
|
||||
job_id,
|
||||
status="failed",
|
||||
stdout="",
|
||||
stderr=traceback_text,
|
||||
result={
|
||||
"stdout": "",
|
||||
"stderr": traceback_text,
|
||||
"summary_text": summary_text,
|
||||
"exception_type": exc.__class__.__name__,
|
||||
"fatal_loop_exception": True,
|
||||
},
|
||||
error_message=summary_text,
|
||||
duration_ms=duration_ms,
|
||||
summary_text=summary_text,
|
||||
focus_ref=dict(normalized_job.get("focus_ref") or {}),
|
||||
step_ref=dict(normalized_job.get("step_ref") or {}),
|
||||
release_context=dict(normalized_job.get("release_context") or {}),
|
||||
)
|
||||
_log(
|
||||
"job fatal exception completion: "
|
||||
f"id={job_id} code={normalized_job.get('job_code') or '-'} "
|
||||
f"action={action or '-'} delivery={delivery.get('state')}"
|
||||
)
|
||||
except Exception as completion_exc:
|
||||
_log(
|
||||
"job fatal exception completion failed: "
|
||||
f"id={job_id} code={normalized_job.get('job_code') or '-'} "
|
||||
f"action={action or '-'} error={completion_exc}"
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
if not NODE_CODE:
|
||||
raise RuntimeError("NODE_CODE 未配置")
|
||||
@@ -963,6 +1365,7 @@ def main() -> None:
|
||||
_ensure_queue_dirs()
|
||||
_register()
|
||||
last_heartbeat_at = 0.0
|
||||
last_runtime_config_sync_at = 0.0
|
||||
|
||||
while True:
|
||||
now = time.time()
|
||||
@@ -970,13 +1373,17 @@ def main() -> None:
|
||||
delivery_summary = _flush_delivery_queue(limit=AGENT_QUEUE_FLUSH_LIMIT)
|
||||
if delivery_summary["delivered"] or delivery_summary["dead_letter"]:
|
||||
_log(f"delivery queue flush: {delivery_summary}")
|
||||
if now - last_runtime_config_sync_at >= AGENT_RUNTIME_CONFIG_SYNC_INTERVAL_SECONDS:
|
||||
bundle = _pull_runtime_config()
|
||||
_apply_runtime_config(bundle)
|
||||
last_runtime_config_sync_at = now
|
||||
if now - last_heartbeat_at >= 15:
|
||||
_heartbeat()
|
||||
last_heartbeat_at = now
|
||||
jobs = _pull_jobs()
|
||||
if jobs:
|
||||
for job in jobs:
|
||||
_process_job(job)
|
||||
_process_job_with_guard(job)
|
||||
else:
|
||||
time.sleep(AGENT_POLL_INTERVAL_SECONDS)
|
||||
except urllib.error.HTTPError as exc:
|
||||
|
||||
@@ -2,10 +2,11 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import socket
|
||||
import threading
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.db import get_db
|
||||
from app.core.db import db_read_retry, get_db
|
||||
|
||||
|
||||
_RUNTIME_SCHEMA_SQL = """
|
||||
@@ -29,6 +30,8 @@ CREATE TABLE IF NOT EXISTS detect_jobs (
|
||||
job_code VARCHAR(64) NOT NULL UNIQUE,
|
||||
source VARCHAR(64) NOT NULL DEFAULT 'manual',
|
||||
plan_hash VARCHAR(128) NOT NULL DEFAULT '',
|
||||
task_mode VARCHAR(32) NOT NULL DEFAULT 'domain_pipeline',
|
||||
step_code VARCHAR(64) NOT NULL DEFAULT '',
|
||||
status VARCHAR(32) NOT NULL DEFAULT 'pending',
|
||||
remark TEXT NOT NULL DEFAULT '',
|
||||
created_by VARCHAR(64) NOT NULL DEFAULT '',
|
||||
@@ -41,6 +44,7 @@ CREATE TABLE IF NOT EXISTS detect_job_items (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
job_id BIGINT NOT NULL REFERENCES detect_jobs(id) ON DELETE CASCADE,
|
||||
domain_id BIGINT NOT NULL,
|
||||
step_code VARCHAR(64) NOT NULL DEFAULT '',
|
||||
status VARCHAR(32) NOT NULL DEFAULT 'pending',
|
||||
claimed_by VARCHAR(64) NOT NULL DEFAULT '',
|
||||
claim_token VARCHAR(64) NOT NULL DEFAULT '',
|
||||
@@ -48,16 +52,32 @@ CREATE TABLE IF NOT EXISTS detect_job_items (
|
||||
attempt_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT NOT NULL DEFAULT '',
|
||||
result_version VARCHAR(64) NOT NULL DEFAULT '',
|
||||
step_payload_json JSONB,
|
||||
result_payload_json JSONB,
|
||||
started_at TIMESTAMP,
|
||||
finished_at TIMESTAMP,
|
||||
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
CONSTRAINT uq_detect_job_items_job_domain UNIQUE (job_id, domain_id)
|
||||
create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_detect_job_items_status_lease
|
||||
ON detect_job_items(status, lease_expires_at);
|
||||
|
||||
ALTER TABLE detect_jobs
|
||||
ADD COLUMN IF NOT EXISTS task_mode VARCHAR(32) NOT NULL DEFAULT 'domain_pipeline',
|
||||
ADD COLUMN IF NOT EXISTS step_code VARCHAR(64) NOT NULL DEFAULT '';
|
||||
|
||||
ALTER TABLE detect_job_items
|
||||
ADD COLUMN IF NOT EXISTS step_code VARCHAR(64) NOT NULL DEFAULT '',
|
||||
ADD COLUMN IF NOT EXISTS step_payload_json JSONB,
|
||||
ADD COLUMN IF NOT EXISTS result_payload_json JSONB;
|
||||
|
||||
ALTER TABLE detect_job_items
|
||||
DROP CONSTRAINT IF EXISTS uq_detect_job_items_job_domain;
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_detect_job_items_job_domain_step
|
||||
ON detect_job_items(job_id, domain_id, step_code);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS detect_run_events (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
job_id BIGINT REFERENCES detect_jobs(id) ON DELETE SET NULL,
|
||||
@@ -90,6 +110,9 @@ _STALE_AFTER_SECONDS = 90
|
||||
_OFFLINE_AFTER_MINUTES = 5
|
||||
_PRUNE_IMPORTED_AFTER_MINUTES = 30
|
||||
_PRUNE_GENERAL_AFTER_HOURS = 6
|
||||
_RUNTIME_SCHEMA_READY = False
|
||||
_RUNTIME_SCHEMA_LOCK = threading.Lock()
|
||||
_RUNTIME_SCHEMA_ADVISORY_LOCK_ID = 62021001
|
||||
|
||||
|
||||
def _resolve_local_ip() -> str:
|
||||
@@ -110,12 +133,85 @@ def _decode_json(value: object) -> dict:
|
||||
return {}
|
||||
|
||||
|
||||
def _control_node_supports_worker(*, region: object, metadata: dict | None) -> bool:
|
||||
normalized_region = str(region or "").strip()
|
||||
runtime_metadata = dict(metadata or {})
|
||||
active_threads = int(runtime_metadata.get("active_threads", 0) or 0)
|
||||
max_threads = int(runtime_metadata.get("max_threads", 0) or 0)
|
||||
if normalized_region != "mainland":
|
||||
return False
|
||||
return bool(
|
||||
runtime_metadata.get("worker_online", False)
|
||||
or runtime_metadata.get("detect_participating", False)
|
||||
or active_threads > 0
|
||||
or max_threads > 0
|
||||
)
|
||||
|
||||
|
||||
def _metadata_idle_without_runtime_work(metadata: dict | None) -> bool:
|
||||
runtime_metadata = dict(metadata or {})
|
||||
phase = str(
|
||||
runtime_metadata.get("phase")
|
||||
or runtime_metadata.get("phase_label")
|
||||
or ""
|
||||
).strip().lower()
|
||||
if phase not in {"idle", "completed", "stopped"}:
|
||||
return False
|
||||
|
||||
active_job_code = str(runtime_metadata.get("active_job_code") or "").strip()
|
||||
counters = (
|
||||
int(runtime_metadata.get("job_items_total", 0) or 0),
|
||||
int(runtime_metadata.get("job_items_claimed", 0) or 0),
|
||||
int(runtime_metadata.get("job_items_running", 0) or 0),
|
||||
int(runtime_metadata.get("job_items_completed", 0) or 0),
|
||||
int(runtime_metadata.get("job_items_failed", 0) or 0),
|
||||
)
|
||||
if active_job_code:
|
||||
return False
|
||||
return not any(value > 0 for value in counters)
|
||||
|
||||
|
||||
def _load_managed_node_overlays() -> dict[str, dict]:
|
||||
try:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT node_code, metadata_json, last_seen_at
|
||||
FROM ops_managed_nodes
|
||||
WHERE is_enabled = TRUE
|
||||
"""
|
||||
)
|
||||
rows = cur.fetchall()
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
overlays: dict[str, dict] = {}
|
||||
for row in rows:
|
||||
node_code = str(row[0] or "").strip()
|
||||
if not node_code:
|
||||
continue
|
||||
overlays[node_code] = {
|
||||
"metadata": _decode_json(row[1]),
|
||||
"last_seen_at": row[2],
|
||||
}
|
||||
return overlays
|
||||
|
||||
|
||||
def ensure_runtime_schema() -> None:
|
||||
with get_db() as conn:
|
||||
conn.autocommit = False
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(_RUNTIME_SCHEMA_SQL)
|
||||
conn.commit()
|
||||
global _RUNTIME_SCHEMA_READY
|
||||
if _RUNTIME_SCHEMA_READY:
|
||||
return
|
||||
with _RUNTIME_SCHEMA_LOCK:
|
||||
if _RUNTIME_SCHEMA_READY:
|
||||
return
|
||||
with get_db() as conn:
|
||||
conn.autocommit = False
|
||||
with conn.cursor() as cur:
|
||||
cur.execute("SELECT pg_advisory_xact_lock(%s)", (_RUNTIME_SCHEMA_ADVISORY_LOCK_ID,))
|
||||
cur.execute(_RUNTIME_SCHEMA_SQL)
|
||||
conn.commit()
|
||||
_RUNTIME_SCHEMA_READY = True
|
||||
|
||||
|
||||
def register_node_heartbeat(
|
||||
@@ -195,6 +291,43 @@ def cleanup_imported_runtime_nodes(*, region: str, role: str, keep_node_code: st
|
||||
conn.commit()
|
||||
|
||||
|
||||
def cleanup_imported_runtime_nodes_many(*, region: str, role: str, keep_node_codes: list[str] | tuple[str, ...] | set[str]) -> None:
|
||||
normalized_region = str(region or "").strip() or "unknown"
|
||||
normalized_role = str(role or "").strip() or "unknown"
|
||||
preserved_node_codes = sorted(
|
||||
{
|
||||
str(node_code or "").strip()
|
||||
for node_code in (keep_node_codes or [])
|
||||
if str(node_code or "").strip()
|
||||
}
|
||||
)
|
||||
if not preserved_node_codes:
|
||||
return
|
||||
|
||||
placeholders = ", ".join(["%s"] * len(preserved_node_codes))
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
f"""
|
||||
DELETE FROM detect_worker_nodes
|
||||
WHERE region = %s
|
||||
AND role = %s
|
||||
AND (
|
||||
node_code = %s
|
||||
OR (metadata_json->>'service') = 'runtime-ingest'
|
||||
)
|
||||
AND node_code NOT IN ({placeholders})
|
||||
""",
|
||||
(
|
||||
normalized_region,
|
||||
normalized_role,
|
||||
f"{normalized_region}-{normalized_role}-imported",
|
||||
*preserved_node_codes,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def prune_expired_runtime_nodes() -> None:
|
||||
imported_cutoff = datetime.now() - timedelta(minutes=_PRUNE_IMPORTED_AFTER_MINUTES)
|
||||
general_cutoff = datetime.now() - timedelta(hours=_PRUNE_GENERAL_AFTER_HOURS)
|
||||
@@ -219,10 +352,12 @@ def prune_expired_runtime_nodes() -> None:
|
||||
|
||||
def register_local_control_heartbeat() -> None:
|
||||
from app.services.detect_job_service import get_active_detect_job_summary
|
||||
from app.services.detect_service import get_detect_status
|
||||
from app.services.worker_control_service import detect_worker_runtime
|
||||
|
||||
worker_runtime = detect_worker_runtime()
|
||||
worker_online = bool(worker_runtime.get("running", False))
|
||||
detect_status = get_detect_status()
|
||||
active_job = get_active_detect_job_summary(event_limit=5) or {}
|
||||
node_stats = list(active_job.get("node_stats") or [])
|
||||
local_bucket = next(
|
||||
@@ -233,7 +368,9 @@ def register_local_control_heartbeat() -> None:
|
||||
items_claimed = int(local_bucket.get("items_claimed", 0) or 0)
|
||||
items_running = int(local_bucket.get("items_running", 0) or 0)
|
||||
items_completed = int(local_bucket.get("items_completed", 0) or 0)
|
||||
current_load = max(items_running, items_claimed, 0)
|
||||
active_threads = int(detect_status.get("active_thread_count", 0) or 0)
|
||||
max_threads = int(detect_status.get("max_thread_count", 0) or 0)
|
||||
current_load = max(items_running, active_threads, 0)
|
||||
detect_participating = bool(worker_online and (items_total > 0 or current_load > 0))
|
||||
node_status = "busy" if current_load > 0 else "online"
|
||||
register_node_heartbeat(
|
||||
@@ -256,6 +393,10 @@ def register_local_control_heartbeat() -> None:
|
||||
"job_items_claimed": items_claimed,
|
||||
"job_items_running": items_running,
|
||||
"job_items_completed": items_completed,
|
||||
"active_threads": active_threads,
|
||||
"max_threads": max_threads,
|
||||
"phase_label": str(detect_status.get("phase_label") or ""),
|
||||
"phase_detail": str(detect_status.get("phase_detail") or ""),
|
||||
"updated_at": datetime.now().isoformat(timespec="seconds"),
|
||||
},
|
||||
)
|
||||
@@ -274,9 +415,11 @@ def _normalize_node_status(raw_status: str, last_heartbeat_at: datetime | None)
|
||||
return status
|
||||
|
||||
|
||||
@db_read_retry()
|
||||
def get_cluster_snapshot() -> dict:
|
||||
prune_expired_runtime_nodes()
|
||||
register_local_control_heartbeat()
|
||||
managed_overlays = _load_managed_node_overlays()
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
@@ -297,21 +440,50 @@ def get_cluster_snapshot() -> dict:
|
||||
cur.execute("SELECT count(*) FROM detect_job_items WHERE status IN ('pending', 'claimed', 'running')")
|
||||
active_items = cur.fetchone()[0]
|
||||
|
||||
nodes = [
|
||||
{
|
||||
"node_code": row[0],
|
||||
"region": row[1],
|
||||
"role": row[2],
|
||||
"hostname": row[3],
|
||||
"ip": row[4],
|
||||
"status": _normalize_node_status(row[5], row[9]),
|
||||
"worker_version": row[6],
|
||||
"current_load": row[7],
|
||||
"metadata": _decode_json(row[8]),
|
||||
"last_heartbeat_at": row[9].isoformat(sep=" ", timespec="seconds") if row[9] else "",
|
||||
}
|
||||
for row in rows
|
||||
]
|
||||
nodes = []
|
||||
for row in rows:
|
||||
node_code = str(row[0] or "").strip()
|
||||
metadata = _decode_json(row[8])
|
||||
current_load = int(row[7] or 0)
|
||||
sanitized_idle_runtime = _metadata_idle_without_runtime_work(metadata)
|
||||
if sanitized_idle_runtime:
|
||||
current_load = 0
|
||||
metadata["active_threads"] = 0
|
||||
metadata["detect_participating"] = False
|
||||
metadata["sanitized_runtime_state"] = "idle_phase_zeroed"
|
||||
runtime_last_heartbeat = row[9]
|
||||
managed_overlay = managed_overlays.get(node_code) or {}
|
||||
managed_last_seen = managed_overlay.get("last_seen_at")
|
||||
overlay_is_newer = bool(
|
||||
managed_last_seen
|
||||
and (not runtime_last_heartbeat or managed_last_seen > runtime_last_heartbeat)
|
||||
)
|
||||
effective_last_heartbeat = managed_last_seen if overlay_is_newer else runtime_last_heartbeat
|
||||
normalized_status = _normalize_node_status(row[5], effective_last_heartbeat)
|
||||
if sanitized_idle_runtime and normalized_status == "busy":
|
||||
normalized_status = "online"
|
||||
if overlay_is_newer and normalized_status in {"offline", "stale"}:
|
||||
normalized_status = "busy" if current_load > 0 else "online"
|
||||
if managed_last_seen:
|
||||
metadata["agent_last_seen_at"] = managed_last_seen.isoformat(sep=" ", timespec="seconds")
|
||||
if overlay_is_newer:
|
||||
metadata["cluster_status_source"] = "managed-agent-overlay"
|
||||
nodes.append(
|
||||
{
|
||||
"node_code": node_code,
|
||||
"region": row[1],
|
||||
"role": row[2],
|
||||
"hostname": row[3],
|
||||
"ip": row[4],
|
||||
"status": normalized_status,
|
||||
"worker_version": row[6],
|
||||
"current_load": current_load,
|
||||
"metadata": metadata,
|
||||
"last_heartbeat_at": effective_last_heartbeat.isoformat(sep=" ", timespec="seconds")
|
||||
if effective_last_heartbeat
|
||||
else "",
|
||||
}
|
||||
)
|
||||
status_counts: dict[str, int] = {}
|
||||
role_counts: dict[str, int] = {}
|
||||
region_counts: dict[str, int] = {}
|
||||
@@ -330,6 +502,12 @@ def get_cluster_snapshot() -> dict:
|
||||
metadata = node.get("metadata") or {}
|
||||
node_current_load = int(node.get("current_load", 0) or 0)
|
||||
effective_worker = False
|
||||
if node_role == "control" and not _control_node_supports_worker(region=node_region, metadata=metadata):
|
||||
node_current_load = 0
|
||||
node["current_load"] = 0
|
||||
if node_status == "busy":
|
||||
node_status = "online"
|
||||
node["status"] = "online"
|
||||
|
||||
status_counts[node_status] = status_counts.get(node_status, 0) + 1
|
||||
role_counts[node_role] = role_counts.get(node_role, 0) + 1
|
||||
@@ -345,14 +523,16 @@ def get_cluster_snapshot() -> dict:
|
||||
dedicated_online_worker_nodes += 1
|
||||
effective_worker = True
|
||||
elif node_role == "control" and node_status in {"online", "busy"}:
|
||||
if bool(metadata.get("worker_online", False)) or bool(metadata.get("detect_participating", False)) or node_current_load > 0:
|
||||
if _control_node_supports_worker(region=node_region, metadata=metadata):
|
||||
effective_worker = True
|
||||
if effective_worker:
|
||||
online_worker_nodes += 1
|
||||
if node_role == "control" and node_status in {"online", "busy"}:
|
||||
online_control_nodes += 1
|
||||
node["is_effective_worker"] = effective_worker
|
||||
node["detect_participating"] = bool(metadata.get("detect_participating", False) or node_current_load > 0)
|
||||
node["detect_participating"] = bool(
|
||||
effective_worker and (metadata.get("detect_participating", False) or node_current_load > 0)
|
||||
)
|
||||
|
||||
return {
|
||||
"nodes": nodes,
|
||||
|
||||
@@ -1,10 +1,390 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from app.core.db import get_db
|
||||
from app.services.detect_job_service import (
|
||||
_build_step_bucket,
|
||||
get_active_detect_job_summary,
|
||||
get_detect_capacity_plan,
|
||||
get_detect_queue_health,
|
||||
)
|
||||
from app.services.runtime_status_service import get_runtime_status
|
||||
|
||||
|
||||
def _empty_active_jobs_aggregate(window_minutes: int) -> dict:
|
||||
return {
|
||||
"window_minutes": int(window_minutes or 15),
|
||||
"active_jobs_total": 0,
|
||||
"queue": {
|
||||
"items_total": 0,
|
||||
"pending": 0,
|
||||
"claimed": 0,
|
||||
"running": 0,
|
||||
"completed": 0,
|
||||
"blacklisted": 0,
|
||||
"failed": 0,
|
||||
"terminal": 0,
|
||||
},
|
||||
"throughput": {
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0,
|
||||
"completed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
"failed_recent": 0,
|
||||
},
|
||||
"steps": [],
|
||||
"nodes": [],
|
||||
"retry_total": 0,
|
||||
}
|
||||
|
||||
|
||||
def _merge_step_queues_with_runtime_activity(
|
||||
base_steps: list[dict] | None,
|
||||
*,
|
||||
runtime_activity: dict | None = None,
|
||||
window_minutes: int = 15,
|
||||
limit: int = 8,
|
||||
) -> list[dict]:
|
||||
safe_window_minutes = max(1, int(window_minutes or 15))
|
||||
normalized_limit = max(1, int(limit or 8))
|
||||
step_map: dict[str, dict] = {}
|
||||
|
||||
for item in list(base_steps or []):
|
||||
step_code = str(item.get("step_code") or "").strip()
|
||||
if not step_code:
|
||||
continue
|
||||
bucket = _build_step_bucket(step_code)
|
||||
bucket.update(
|
||||
{
|
||||
"items_total": int(item.get("items_total", 0) or 0),
|
||||
"items_pending": int(item.get("items_pending", 0) or 0),
|
||||
"items_claimed": int(item.get("items_claimed", 0) or 0),
|
||||
"items_running": int(item.get("items_running", 0) or 0),
|
||||
"items_completed": int(item.get("items_completed", 0) or 0),
|
||||
"items_blacklisted": int(item.get("items_blacklisted", 0) or 0),
|
||||
"items_failed": int(item.get("items_failed", 0) or 0),
|
||||
"started_recent": int(item.get("started_recent", 0) or 0),
|
||||
"processed_recent": int(item.get("processed_recent", 0) or 0),
|
||||
"processed_per_minute": float(item.get("processed_per_minute", 0) or 0),
|
||||
"completed_recent": int(item.get("completed_recent", 0) or 0),
|
||||
"blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0),
|
||||
"failed_recent": int(item.get("failed_recent", 0) or 0),
|
||||
}
|
||||
)
|
||||
step_map[step_code] = bucket
|
||||
|
||||
runtime_step_stats = dict((runtime_activity or {}).get("step_stats") or {})
|
||||
for step_code, stats in runtime_step_stats.items():
|
||||
normalized_step_code = str(step_code or "").strip()
|
||||
if not normalized_step_code:
|
||||
continue
|
||||
bucket = step_map.setdefault(normalized_step_code, _build_step_bucket(normalized_step_code))
|
||||
started_recent = int((stats or {}).get("started_recent", 0) or 0)
|
||||
processed_recent = int((stats or {}).get("processed_recent", 0) or 0)
|
||||
completed_recent = int((stats or {}).get("completed_recent", 0) or 0)
|
||||
blacklisted_recent = int((stats or {}).get("blacklisted_recent", 0) or 0)
|
||||
failed_recent = int((stats or {}).get("failed_recent", 0) or 0)
|
||||
bucket["started_recent"] = max(int(bucket.get("started_recent", 0) or 0), started_recent)
|
||||
bucket["processed_recent"] = max(int(bucket.get("processed_recent", 0) or 0), processed_recent)
|
||||
bucket["completed_recent"] = max(int(bucket.get("completed_recent", 0) or 0), completed_recent)
|
||||
bucket["blacklisted_recent"] = max(int(bucket.get("blacklisted_recent", 0) or 0), blacklisted_recent)
|
||||
bucket["failed_recent"] = max(int(bucket.get("failed_recent", 0) or 0), failed_recent)
|
||||
bucket["processed_per_minute"] = max(
|
||||
float(bucket.get("processed_per_minute", 0) or 0),
|
||||
round(processed_recent / safe_window_minutes, 2),
|
||||
)
|
||||
|
||||
return sorted(
|
||||
step_map.values(),
|
||||
key=lambda item: (
|
||||
-int(item.get("items_pending", 0) or 0),
|
||||
-int(item.get("items_running", 0) or 0),
|
||||
-int(item.get("started_recent", 0) or 0),
|
||||
-int(item.get("processed_recent", 0) or 0),
|
||||
str(item.get("step_code") or ""),
|
||||
),
|
||||
)[:normalized_limit]
|
||||
|
||||
|
||||
def _align_active_jobs_aggregate_with_runtime(
|
||||
aggregate: dict,
|
||||
*,
|
||||
runtime: dict,
|
||||
queue_health: dict,
|
||||
) -> dict:
|
||||
normalized = dict(aggregate or {})
|
||||
node_payload = dict((runtime or {}).get("node") or {})
|
||||
if str(node_payload.get("region") or "").strip() != "overseas" or str(node_payload.get("role") or "").strip() != "control":
|
||||
return normalized
|
||||
|
||||
backlog = dict(((runtime or {}).get("detect") or {}).get("backlog") or {})
|
||||
snapshot_backlog = dict(queue_health.get("runtime_snapshot_backlog") or {})
|
||||
|
||||
def _backlog_value(key: str) -> int:
|
||||
return max(int(backlog.get(key, 0) or 0), int(snapshot_backlog.get(key, 0) or 0))
|
||||
|
||||
pending_total = _backlog_value("pending_total")
|
||||
claimed_total = _backlog_value("claimed_total")
|
||||
running_total = _backlog_value("running_total")
|
||||
completed_total = _backlog_value("completed_total")
|
||||
blacklisted_total = _backlog_value("blacklisted_total")
|
||||
failed_total = _backlog_value("failed_total")
|
||||
|
||||
queue = dict(queue_health.get("queue") or {})
|
||||
throughput = dict(queue_health.get("throughput") or {})
|
||||
terminal_total = max(
|
||||
completed_total + blacklisted_total + failed_total,
|
||||
int(queue.get("completed", 0) or 0) + int(queue.get("blacklisted", 0) or 0) + int(queue.get("failed", 0) or 0),
|
||||
)
|
||||
items_total = pending_total + claimed_total + running_total + terminal_total
|
||||
has_runtime_work = items_total > 0 or bool(queue_health.get("has_active_job"))
|
||||
|
||||
normalized["active_jobs_total"] = max(
|
||||
int(normalized.get("active_jobs_total", 0) or 0),
|
||||
1 if has_runtime_work else 0,
|
||||
)
|
||||
normalized["queue"] = {
|
||||
"items_total": items_total,
|
||||
"pending": pending_total,
|
||||
"claimed": claimed_total,
|
||||
"running": running_total,
|
||||
"completed": max(completed_total, int(queue.get("completed", 0) or 0)),
|
||||
"blacklisted": max(blacklisted_total, int(queue.get("blacklisted", 0) or 0)),
|
||||
"failed": max(failed_total, int(queue.get("failed", 0) or 0)),
|
||||
"terminal": terminal_total,
|
||||
}
|
||||
normalized["throughput"] = {
|
||||
"processed_recent": int(throughput.get("processed_recent", 0) or 0),
|
||||
"processed_per_minute": float(throughput.get("processed_per_minute", 0) or 0),
|
||||
"completed_recent": int(throughput.get("completed_recent", 0) or 0),
|
||||
"blacklisted_recent": int(throughput.get("blacklisted_recent", 0) or 0),
|
||||
"failed_recent": int(throughput.get("failed_recent", 0) or 0),
|
||||
}
|
||||
normalized["steps"] = _merge_step_queues_with_runtime_activity(
|
||||
list(queue_health.get("steps") or []),
|
||||
runtime_activity=dict(queue_health.get("runtime_activity") or {}),
|
||||
window_minutes=int(queue_health.get("window_minutes", normalized.get("window_minutes", 15)) or 15),
|
||||
limit=8,
|
||||
)
|
||||
normalized["nodes"] = list(queue_health.get("nodes") or [])
|
||||
return normalized
|
||||
|
||||
|
||||
def _fetch_active_jobs_aggregate(window_minutes: int = 15) -> dict:
|
||||
safe_window_minutes = max(5, min(int(window_minutes or 15), 120))
|
||||
payload = _empty_active_jobs_aggregate(safe_window_minutes)
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT id
|
||||
FROM detect_jobs
|
||||
WHERE status IN ('pending', 'running', 'partial_failed')
|
||||
ORDER BY
|
||||
CASE
|
||||
WHEN status = 'running' THEN 0
|
||||
WHEN status = 'pending' THEN 1
|
||||
WHEN status = 'partial_failed' THEN 2
|
||||
ELSE 3
|
||||
END ASC,
|
||||
COALESCE(started_at, created_at) DESC,
|
||||
id DESC
|
||||
"""
|
||||
)
|
||||
job_ids = [int(row[0]) for row in cur.fetchall() if row and row[0] is not None]
|
||||
if not job_ids:
|
||||
return payload
|
||||
|
||||
payload["active_jobs_total"] = len(job_ids)
|
||||
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
COUNT(*) AS items_total,
|
||||
COUNT(*) FILTER (WHERE status = 'pending') AS items_pending,
|
||||
COUNT(*) FILTER (WHERE status = 'claimed') AS items_claimed,
|
||||
COUNT(*) FILTER (WHERE status = 'running') AS items_running,
|
||||
COUNT(*) FILTER (WHERE status = 'completed') AS items_completed,
|
||||
COUNT(*) FILTER (WHERE status = 'blacklisted') AS items_blacklisted,
|
||||
COUNT(*) FILTER (WHERE status = 'failed') AS items_failed
|
||||
FROM detect_job_items
|
||||
WHERE job_id = ANY(%s)
|
||||
""",
|
||||
(job_ids,),
|
||||
)
|
||||
queue_row = cur.fetchone() or (0, 0, 0, 0, 0, 0, 0)
|
||||
payload["queue"] = {
|
||||
"items_total": int(queue_row[0] or 0),
|
||||
"pending": int(queue_row[1] or 0),
|
||||
"claimed": int(queue_row[2] or 0),
|
||||
"running": int(queue_row[3] or 0),
|
||||
"completed": int(queue_row[4] or 0),
|
||||
"blacklisted": int(queue_row[5] or 0),
|
||||
"failed": int(queue_row[6] or 0),
|
||||
"terminal": int(queue_row[4] or 0) + int(queue_row[5] or 0) + int(queue_row[6] or 0),
|
||||
}
|
||||
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
COUNT(*) AS processed_recent,
|
||||
COUNT(*) FILTER (WHERE event_type = 'domain_completed') AS completed_recent,
|
||||
COUNT(*) FILTER (WHERE event_type = 'domain_blacklisted') AS blacklisted_recent,
|
||||
COUNT(*) FILTER (WHERE event_type = 'domain_failed') AS failed_recent
|
||||
FROM detect_run_events
|
||||
WHERE job_id = ANY(%s)
|
||||
AND event_type IN ('domain_completed', 'domain_blacklisted', 'domain_failed')
|
||||
AND created_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval
|
||||
""",
|
||||
(job_ids, safe_window_minutes),
|
||||
)
|
||||
throughput_row = cur.fetchone() or (0, 0, 0, 0)
|
||||
processed_recent = int(throughput_row[0] or 0)
|
||||
payload["throughput"] = {
|
||||
"processed_recent": processed_recent,
|
||||
"processed_per_minute": round(processed_recent / safe_window_minutes, 2),
|
||||
"completed_recent": int(throughput_row[1] or 0),
|
||||
"blacklisted_recent": int(throughput_row[2] or 0),
|
||||
"failed_recent": int(throughput_row[3] or 0),
|
||||
}
|
||||
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT COUNT(*)
|
||||
FROM detect_job_items
|
||||
WHERE job_id = ANY(%s)
|
||||
AND attempt_count > 1
|
||||
""",
|
||||
(job_ids,),
|
||||
)
|
||||
payload["retry_total"] = int((cur.fetchone() or [0])[0] or 0)
|
||||
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
COALESCE(NULLIF(step_code, ''), 'domain_pipeline') AS step_code,
|
||||
COUNT(*) AS items_total,
|
||||
COUNT(*) FILTER (WHERE status = 'pending') AS items_pending,
|
||||
COUNT(*) FILTER (WHERE status = 'claimed') AS items_claimed,
|
||||
COUNT(*) FILTER (WHERE status = 'running') AS items_running,
|
||||
COUNT(*) FILTER (WHERE status = 'completed') AS items_completed,
|
||||
COUNT(*) FILTER (WHERE status = 'blacklisted') AS items_blacklisted,
|
||||
COUNT(*) FILTER (WHERE status = 'failed') AS items_failed,
|
||||
COUNT(*) FILTER (
|
||||
WHERE status IN ('completed', 'blacklisted', 'failed')
|
||||
AND finished_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval
|
||||
) AS processed_recent
|
||||
FROM detect_job_items
|
||||
WHERE job_id = ANY(%s)
|
||||
GROUP BY COALESCE(NULLIF(step_code, ''), 'domain_pipeline')
|
||||
ORDER BY
|
||||
COUNT(*) FILTER (WHERE status = 'pending') DESC,
|
||||
COUNT(*) FILTER (WHERE status = 'running') DESC,
|
||||
COUNT(*) DESC,
|
||||
COALESCE(NULLIF(step_code, ''), 'domain_pipeline') ASC
|
||||
LIMIT 8
|
||||
""",
|
||||
(safe_window_minutes, job_ids),
|
||||
)
|
||||
steps: list[dict] = []
|
||||
for row in cur.fetchall():
|
||||
step_code = str(row[0] or "domain_pipeline")
|
||||
bucket = _build_step_bucket(step_code)
|
||||
bucket.update(
|
||||
{
|
||||
"items_total": int(row[1] or 0),
|
||||
"items_pending": int(row[2] or 0),
|
||||
"items_claimed": int(row[3] or 0),
|
||||
"items_running": int(row[4] or 0),
|
||||
"items_completed": int(row[5] or 0),
|
||||
"items_blacklisted": int(row[6] or 0),
|
||||
"items_failed": int(row[7] or 0),
|
||||
"processed_recent": int(row[8] or 0),
|
||||
"processed_per_minute": round(int(row[8] or 0) / safe_window_minutes, 2),
|
||||
}
|
||||
)
|
||||
steps.append(bucket)
|
||||
payload["steps"] = steps
|
||||
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
COALESCE(NULLIF(claimed_by, ''), 'unassigned') AS node_code,
|
||||
COUNT(*) FILTER (WHERE status = 'claimed') AS items_claimed,
|
||||
COUNT(*) FILTER (WHERE status = 'running') AS items_running,
|
||||
COUNT(*) FILTER (WHERE status = 'completed') AS items_completed_total
|
||||
FROM detect_job_items
|
||||
WHERE job_id = ANY(%s)
|
||||
GROUP BY COALESCE(NULLIF(claimed_by, ''), 'unassigned')
|
||||
""",
|
||||
(job_ids,),
|
||||
)
|
||||
node_map = {
|
||||
str(row[0] or "unassigned"): {
|
||||
"node_code": str(row[0] or "unassigned"),
|
||||
"items_running": int(row[2] or 0),
|
||||
"items_claimed": int(row[1] or 0),
|
||||
"items_completed": int(row[3] or 0),
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0,
|
||||
"completed_recent": 0,
|
||||
"failed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
}
|
||||
for row in cur.fetchall()
|
||||
}
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
COALESCE(NULLIF(node_code, ''), 'unassigned') AS node_code,
|
||||
COUNT(*) AS processed_recent,
|
||||
COUNT(*) FILTER (WHERE event_type = 'domain_completed') AS completed_recent,
|
||||
COUNT(*) FILTER (WHERE event_type = 'domain_blacklisted') AS blacklisted_recent,
|
||||
COUNT(*) FILTER (WHERE event_type = 'domain_failed') AS failed_recent
|
||||
FROM detect_run_events
|
||||
WHERE job_id = ANY(%s)
|
||||
AND event_type IN ('domain_completed', 'domain_blacklisted', 'domain_failed')
|
||||
AND created_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval
|
||||
GROUP BY COALESCE(NULLIF(node_code, ''), 'unassigned')
|
||||
""",
|
||||
(job_ids, safe_window_minutes),
|
||||
)
|
||||
for row in cur.fetchall():
|
||||
node_code = str(row[0] or "unassigned")
|
||||
bucket = node_map.setdefault(
|
||||
node_code,
|
||||
{
|
||||
"node_code": node_code,
|
||||
"items_running": 0,
|
||||
"items_claimed": 0,
|
||||
"items_completed": 0,
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0,
|
||||
"completed_recent": 0,
|
||||
"failed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
},
|
||||
)
|
||||
processed_recent = int(row[1] or 0)
|
||||
bucket["processed_recent"] = processed_recent
|
||||
bucket["processed_per_minute"] = round(processed_recent / safe_window_minutes, 2)
|
||||
bucket["completed_recent"] = int(row[2] or 0)
|
||||
bucket["blacklisted_recent"] = int(row[3] or 0)
|
||||
bucket["failed_recent"] = int(row[4] or 0)
|
||||
|
||||
payload["nodes"] = sorted(
|
||||
node_map.values(),
|
||||
key=lambda item: (
|
||||
-int(item.get("items_running", 0) or 0),
|
||||
-int(item.get("processed_recent", 0) or 0),
|
||||
-int(item.get("items_claimed", 0) or 0),
|
||||
str(item.get("node_code") or ""),
|
||||
),
|
||||
)[:8]
|
||||
return payload
|
||||
|
||||
|
||||
def fetch_overview() -> dict:
|
||||
window_minutes = 15
|
||||
queries = {
|
||||
"domains_total": "select count(*) from domains",
|
||||
"pending_total": "select count(*) from domains where detect_status = 0",
|
||||
@@ -12,6 +392,8 @@ def fetch_overview() -> dict:
|
||||
"running_total": "select count(*) from domains where detect_status = 2",
|
||||
"blacklist_total": "select count(*) from domains where detect_status = 3",
|
||||
"failed_total": "select count(*) from domains where detect_status = 4",
|
||||
"registerable_total": "select count(*) from domains where detect_status = 1 and register_status = 2",
|
||||
"purchasable_total": "select count(*) from domains where detect_status = 1 and register_status = 2 and coalesce(use_status, 0) = 0",
|
||||
"sensitive_words_total": "select count(*) from sensitive_words",
|
||||
}
|
||||
result: dict[str, int | str] = {}
|
||||
@@ -23,6 +405,11 @@ def fetch_overview() -> dict:
|
||||
result[key] = cur.fetchone()[0]
|
||||
except Exception:
|
||||
result[key] = 0
|
||||
|
||||
active_jobs_aggregate = _fetch_active_jobs_aggregate(window_minutes=window_minutes)
|
||||
active_job = get_active_detect_job_summary(event_limit=20) or {}
|
||||
aggregate_queue = active_jobs_aggregate.get("queue") or {}
|
||||
|
||||
runtime = get_runtime_status()
|
||||
cluster_summary = ((runtime.get("cluster") or {}).get("summary") or {})
|
||||
online_worker_nodes = int(cluster_summary.get("online_worker_nodes", 0) or 0)
|
||||
@@ -37,4 +424,274 @@ def fetch_overview() -> dict:
|
||||
result["worker_mode"] = runtime["worker"]["mode"]
|
||||
result["node_region"] = runtime["node"]["region"]
|
||||
result["node_role"] = runtime["node"]["role"]
|
||||
|
||||
queue_health = get_detect_queue_health(window_minutes=window_minutes)
|
||||
active_jobs_aggregate = _align_active_jobs_aggregate_with_runtime(
|
||||
active_jobs_aggregate,
|
||||
runtime=runtime,
|
||||
queue_health=queue_health,
|
||||
)
|
||||
runtime_snapshot_backlog = dict(queue_health.get("runtime_snapshot_backlog") or {})
|
||||
aggregate_queue = active_jobs_aggregate.get("queue") or {}
|
||||
aggregate_queue_health = {
|
||||
"has_active_job": bool(int(active_jobs_aggregate.get("active_jobs_total", 0) or 0) > 0),
|
||||
"queue": aggregate_queue,
|
||||
"throughput": active_jobs_aggregate.get("throughput") or {},
|
||||
}
|
||||
selected_queue_health = aggregate_queue_health if aggregate_queue_health["has_active_job"] else queue_health
|
||||
if float((queue_health.get("throughput") or {}).get("processed_per_minute", 0) or 0) > float(
|
||||
(selected_queue_health.get("throughput") or {}).get("processed_per_minute", 0) or 0
|
||||
):
|
||||
selected_queue_health = queue_health
|
||||
capacity_plan = get_detect_capacity_plan(
|
||||
queue_health=selected_queue_health,
|
||||
online_worker_nodes=online_worker_nodes,
|
||||
)
|
||||
|
||||
retry_total = int(active_jobs_aggregate.get("retry_total", 0) or 0)
|
||||
step_queue: list[dict] = []
|
||||
node_throughput: list[dict] = []
|
||||
bottleneck_step: dict | None = None
|
||||
active_job_summary: dict | None = None
|
||||
|
||||
if queue_health.get("has_active_job"):
|
||||
job_payload = queue_health.get("job") or {}
|
||||
queue_payload = queue_health.get("queue") or {}
|
||||
throughput_payload = queue_health.get("throughput") or {}
|
||||
runtime_job_code = str(job_payload.get("runtime_job_code") or "").strip()
|
||||
display_job_code = runtime_job_code or str(job_payload.get("job_code") or "")
|
||||
active_job_summary = {
|
||||
"job_id": int(job_payload.get("job_id", 0) or 0),
|
||||
"job_code": display_job_code,
|
||||
"db_job_code": str(job_payload.get("job_code") or ""),
|
||||
"runtime_job_code": runtime_job_code,
|
||||
"status": str(job_payload.get("status") or ""),
|
||||
"progress_percent": float(job_payload.get("progress_percent", 0) or 0),
|
||||
"items_total": int(queue_payload.get("items_total", 0) or 0),
|
||||
"items_pending": int(queue_payload.get("pending", 0) or 0),
|
||||
"items_claimed": int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0),
|
||||
"items_running": int(queue_payload.get("running", 0) or 0),
|
||||
"items_display_running": int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0),
|
||||
"items_completed": int(queue_payload.get("completed", 0) or 0),
|
||||
"items_blacklisted": int(queue_payload.get("blacklisted", 0) or 0),
|
||||
"items_failed": int(queue_payload.get("failed", 0) or 0),
|
||||
"processed_per_minute": float(throughput_payload.get("processed_per_minute", 0) or 0),
|
||||
"processed_recent": int(throughput_payload.get("processed_recent", 0) or 0),
|
||||
"completed_recent": int(throughput_payload.get("completed_recent", 0) or 0),
|
||||
"failed_recent": int(throughput_payload.get("failed_recent", 0) or 0),
|
||||
"blacklisted_recent": int(throughput_payload.get("blacklisted_recent", 0) or 0),
|
||||
"active_jobs_total": int(active_jobs_aggregate.get("active_jobs_total", 0) or 0),
|
||||
}
|
||||
|
||||
queue_pending_total = 0
|
||||
queue_claimed_total = 0
|
||||
queue_running_total = 0
|
||||
queue_display_running_total = 0
|
||||
queue_completed_total = 0
|
||||
queue_blacklist_total = 0
|
||||
queue_failed_total = 0
|
||||
backlog_payload = ((runtime.get("detect") or {}).get("backlog") or {})
|
||||
backlog_pending_total = max(
|
||||
int(backlog_payload.get("pending_total", 0) or 0),
|
||||
int(runtime_snapshot_backlog.get("pending_total", 0) or 0),
|
||||
)
|
||||
backlog_claimed_total = max(
|
||||
int(backlog_payload.get("claimed_total", 0) or 0),
|
||||
int(runtime_snapshot_backlog.get("claimed_total", 0) or 0),
|
||||
)
|
||||
backlog_running_total = max(
|
||||
int(backlog_payload.get("running_total", 0) or 0),
|
||||
int(runtime_snapshot_backlog.get("running_total", 0) or 0),
|
||||
)
|
||||
backlog_register_pending_total = max(
|
||||
int(backlog_payload.get("register_pending", 0) or 0),
|
||||
int(runtime_snapshot_backlog.get("register_pending", 0) or 0),
|
||||
)
|
||||
backlog_downstream_pending_total = max(
|
||||
int(backlog_payload.get("downstream_pending", 0) or 0),
|
||||
int(runtime_snapshot_backlog.get("downstream_pending", 0) or 0),
|
||||
)
|
||||
if queue_health.get("has_active_job"):
|
||||
queue_payload = queue_health.get("queue") or {}
|
||||
queue_pending_total = int(queue_payload.get("pending", 0) or 0)
|
||||
queue_claimed_total = int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0)
|
||||
queue_running_total = int(queue_payload.get("running", 0) or 0)
|
||||
queue_display_running_total = int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0)
|
||||
queue_completed_total = int(queue_payload.get("completed", 0) or 0)
|
||||
queue_blacklist_total = int(queue_payload.get("blacklisted", 0) or 0)
|
||||
queue_failed_total = int(queue_payload.get("failed", 0) or 0)
|
||||
elif int(active_jobs_aggregate.get("active_jobs_total", 0) or 0) > 0:
|
||||
queue_pending_total = int(aggregate_queue.get("pending", 0) or 0)
|
||||
queue_claimed_total = int(aggregate_queue.get("claimed", 0) or 0)
|
||||
queue_running_total = int(aggregate_queue.get("running", 0) or 0)
|
||||
queue_display_running_total = queue_running_total
|
||||
queue_completed_total = int(aggregate_queue.get("completed", 0) or 0)
|
||||
queue_blacklist_total = int(aggregate_queue.get("blacklisted", 0) or 0)
|
||||
queue_failed_total = int(aggregate_queue.get("failed", 0) or 0)
|
||||
elif active_job:
|
||||
queue_pending_total = int(active_job.get("items_pending", 0) or 0)
|
||||
queue_claimed_total = int(active_job.get("items_claimed", 0) or 0)
|
||||
queue_running_total = int(active_job.get("items_running", 0) or 0)
|
||||
queue_display_running_total = int(
|
||||
active_job.get("display_items_running", active_job.get("items_running", 0)) or 0
|
||||
)
|
||||
queue_completed_total = int(active_job.get("items_completed", 0) or 0)
|
||||
queue_blacklist_total = int(active_job.get("items_blacklisted", 0) or 0)
|
||||
queue_failed_total = int(active_job.get("items_failed", 0) or 0)
|
||||
|
||||
aggregate_step_queue = [
|
||||
{
|
||||
"step_code": str(item.get("step_code") or ""),
|
||||
"step_name": str(item.get("step_name") or ""),
|
||||
"items_pending": int(item.get("items_pending", 0) or 0),
|
||||
"items_running": int(item.get("items_running", 0) or 0),
|
||||
"items_claimed": int(item.get("items_claimed", 0) or 0),
|
||||
"items_completed": int(item.get("items_completed", 0) or 0),
|
||||
"items_blacklisted": int(item.get("items_blacklisted", 0) or 0),
|
||||
"items_failed": int(item.get("items_failed", 0) or 0),
|
||||
"started_recent": int(item.get("started_recent", 0) or 0),
|
||||
"processed_per_minute": float(item.get("processed_per_minute", 0) or 0),
|
||||
"processed_recent": int(item.get("processed_recent", 0) or 0),
|
||||
"completed_recent": int(item.get("completed_recent", 0) or 0),
|
||||
"blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0),
|
||||
"failed_recent": int(item.get("failed_recent", 0) or 0),
|
||||
}
|
||||
for item in list(active_jobs_aggregate.get("steps") or [])[:8]
|
||||
]
|
||||
aggregate_node_throughput = [
|
||||
{
|
||||
"node_code": str(item.get("node_code") or ""),
|
||||
"items_pending": int(item.get("items_pending", 0) or 0),
|
||||
"items_running": int(item.get("items_running", 0) or 0),
|
||||
"display_running": int(item.get("display_running", item.get("items_running", 0)) or 0),
|
||||
"items_claimed": int(item.get("items_claimed", 0) or 0),
|
||||
"current_load": int(item.get("current_load", item.get("display_running", 0)) or 0),
|
||||
"active_threads": int(item.get("active_threads", 0) or 0),
|
||||
"max_threads": int(item.get("max_threads", 0) or 0),
|
||||
"processed_recent": int(item.get("processed_recent", 0) or 0),
|
||||
"processed_per_minute": float(item.get("processed_per_minute", 0) or 0),
|
||||
"completed_recent": int(item.get("completed_recent", 0) or 0),
|
||||
"failed_recent": int(item.get("failed_recent", 0) or 0),
|
||||
"blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0),
|
||||
}
|
||||
for item in list(active_jobs_aggregate.get("nodes") or [])[:8]
|
||||
]
|
||||
queue_step_queue = [
|
||||
{
|
||||
"step_code": str(item.get("step_code") or ""),
|
||||
"step_name": str(item.get("step_name") or ""),
|
||||
"items_pending": int(item.get("items_pending", 0) or 0),
|
||||
"items_running": int(item.get("items_running", 0) or 0),
|
||||
"items_claimed": int(item.get("items_claimed", 0) or 0),
|
||||
"items_completed": int(item.get("items_completed", 0) or 0),
|
||||
"items_blacklisted": int(item.get("items_blacklisted", 0) or 0),
|
||||
"items_failed": int(item.get("items_failed", 0) or 0),
|
||||
"started_recent": int(item.get("started_recent", 0) or 0),
|
||||
"processed_per_minute": float(item.get("processed_per_minute", 0) or 0),
|
||||
"processed_recent": int(item.get("processed_recent", 0) or 0),
|
||||
"completed_recent": int(item.get("completed_recent", 0) or 0),
|
||||
"blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0),
|
||||
"failed_recent": int(item.get("failed_recent", 0) or 0),
|
||||
}
|
||||
for item in _merge_step_queues_with_runtime_activity(
|
||||
list(queue_health.get("steps") or []),
|
||||
runtime_activity=dict(queue_health.get("runtime_activity") or {}),
|
||||
window_minutes=window_minutes,
|
||||
limit=8,
|
||||
)
|
||||
]
|
||||
queue_node_throughput = [
|
||||
{
|
||||
"node_code": str(item.get("node_code") or ""),
|
||||
"items_pending": int(item.get("items_pending", 0) or 0),
|
||||
"items_running": int(item.get("items_running", 0) or 0),
|
||||
"display_running": int(item.get("display_running", item.get("items_running", 0)) or 0),
|
||||
"items_claimed": int(item.get("items_claimed", 0) or 0),
|
||||
"current_load": int(item.get("current_load", item.get("display_running", 0)) or 0),
|
||||
"active_threads": int(item.get("active_threads", 0) or 0),
|
||||
"max_threads": int(item.get("max_threads", 0) or 0),
|
||||
"processed_recent": int(item.get("processed_recent", 0) or 0),
|
||||
"processed_per_minute": float(item.get("processed_per_minute", 0) or 0),
|
||||
"completed_recent": int(item.get("completed_recent", 0) or 0),
|
||||
"failed_recent": int(item.get("failed_recent", 0) or 0),
|
||||
"blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0),
|
||||
}
|
||||
for item in list(queue_health.get("nodes") or [])[:8]
|
||||
]
|
||||
step_queue = aggregate_step_queue
|
||||
node_throughput = aggregate_node_throughput
|
||||
aggregate_ppm = float((active_jobs_aggregate.get("throughput") or {}).get("processed_per_minute", 0) or 0)
|
||||
queue_ppm = float((queue_health.get("throughput") or {}).get("processed_per_minute", 0) or 0)
|
||||
if queue_health.get("has_active_job") or queue_ppm > aggregate_ppm:
|
||||
step_queue = queue_step_queue
|
||||
node_throughput = queue_node_throughput
|
||||
if step_queue:
|
||||
bottleneck_step = max(
|
||||
step_queue,
|
||||
key=lambda item: (
|
||||
int(item.get("items_pending", 0) or 0),
|
||||
int(item.get("items_running", 0) or 0),
|
||||
-float(item.get("processed_per_minute", 0) or 0),
|
||||
),
|
||||
)
|
||||
|
||||
result["active_job"] = active_job_summary or {}
|
||||
result["queue_health"] = queue_health
|
||||
result["active_jobs_aggregate"] = active_jobs_aggregate
|
||||
result["capacity_plan"] = capacity_plan
|
||||
result["step_queue"] = step_queue
|
||||
result["node_throughput"] = node_throughput
|
||||
result["retry_total"] = retry_total
|
||||
result["bottleneck_step"] = bottleneck_step or {}
|
||||
aggregate_throughput = active_jobs_aggregate.get("throughput") or {}
|
||||
queue_throughput = queue_health.get("throughput") or {}
|
||||
ops_processed_per_minute = float(aggregate_throughput.get("processed_per_minute", 0) or 0)
|
||||
ops_processed_recent = int(aggregate_throughput.get("processed_recent", 0) or 0)
|
||||
ops_completed_recent = int(aggregate_throughput.get("completed_recent", 0) or 0)
|
||||
ops_failed_recent = int(aggregate_throughput.get("failed_recent", 0) or 0)
|
||||
ops_blacklisted_recent = int(aggregate_throughput.get("blacklisted_recent", 0) or 0)
|
||||
if queue_health.get("has_active_job") or float(queue_throughput.get("processed_per_minute", 0) or 0) > ops_processed_per_minute:
|
||||
ops_processed_per_minute = float(queue_throughput.get("processed_per_minute", 0) or 0)
|
||||
ops_processed_recent = int(queue_throughput.get("processed_recent", 0) or 0)
|
||||
ops_completed_recent = int(queue_throughput.get("completed_recent", 0) or 0)
|
||||
ops_failed_recent = int(queue_throughput.get("failed_recent", 0) or 0)
|
||||
ops_blacklisted_recent = int(queue_throughput.get("blacklisted_recent", 0) or 0)
|
||||
result["ops_summary"] = {
|
||||
"active_jobs_total": int(active_jobs_aggregate.get("active_jobs_total", 0) or 0),
|
||||
"processed_per_minute": ops_processed_per_minute,
|
||||
"processed_recent": ops_processed_recent,
|
||||
"completed_recent": ops_completed_recent,
|
||||
"failed_recent": ops_failed_recent,
|
||||
"blacklisted_recent": ops_blacklisted_recent,
|
||||
"estimated_hours_remaining": float(capacity_plan.get("estimated_hours_remaining", 0) or 0),
|
||||
"remaining_items": int(capacity_plan.get("remaining_items", 0) or 0),
|
||||
"recommended_additional_workers": int(capacity_plan.get("recommended_additional_workers", 0) or 0),
|
||||
"online_worker_nodes": online_worker_nodes,
|
||||
"dedicated_online_worker_nodes": dedicated_online_worker_nodes,
|
||||
"active_execution_nodes": sum(
|
||||
1
|
||||
for item in node_throughput
|
||||
if int(item.get("items_running", 0) or 0) > 0
|
||||
or int(item.get("items_claimed", 0) or 0) > 0
|
||||
or int(item.get("processed_recent", 0) or 0) > 0
|
||||
),
|
||||
}
|
||||
result["processed_per_minute"] = ops_processed_per_minute
|
||||
result["processed_recent"] = ops_processed_recent
|
||||
result["completed_recent"] = ops_completed_recent
|
||||
result["failed_recent"] = ops_failed_recent
|
||||
result["blacklisted_recent"] = ops_blacklisted_recent
|
||||
result["active_execution_nodes"] = int(result["ops_summary"]["active_execution_nodes"] or 0)
|
||||
result["queue_pending_total"] = queue_pending_total
|
||||
result["queue_claimed_total"] = queue_claimed_total
|
||||
result["queue_running_total"] = queue_running_total
|
||||
result["queue_display_running_total"] = max(queue_display_running_total, queue_running_total)
|
||||
result["queue_completed_total"] = queue_completed_total
|
||||
result["queue_blacklist_total"] = queue_blacklist_total
|
||||
result["queue_failed_total"] = queue_failed_total
|
||||
result["backlog_pending_total"] = max(backlog_pending_total, queue_pending_total)
|
||||
result["backlog_claimed_total"] = max(backlog_claimed_total, queue_claimed_total)
|
||||
result["backlog_running_total"] = max(backlog_running_total, queue_running_total)
|
||||
result["backlog_register_pending_total"] = backlog_register_pending_total
|
||||
result["backlog_downstream_pending_total"] = backlog_downstream_pending_total
|
||||
return result
|
||||
|
||||
@@ -2,12 +2,13 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import socket
|
||||
import threading
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.db import get_db
|
||||
from app.core.db import db_read_retry, get_db
|
||||
|
||||
|
||||
_DEBUG_SCHEMA_SQL = """
|
||||
@@ -27,6 +28,9 @@ CREATE INDEX IF NOT EXISTS idx_detect_debug_events_created
|
||||
ON detect_debug_events(created_at DESC);
|
||||
"""
|
||||
|
||||
_DEBUG_SCHEMA_READY = False
|
||||
_DEBUG_SCHEMA_LOCK = threading.Lock()
|
||||
|
||||
|
||||
def _format_time(value: datetime | None) -> str:
|
||||
return value.isoformat(sep=" ", timespec="seconds") if value else ""
|
||||
@@ -82,11 +86,18 @@ def _debug_ingest_url(base_url: str) -> str:
|
||||
|
||||
|
||||
def ensure_debug_event_schema() -> None:
|
||||
with get_db() as conn:
|
||||
conn.autocommit = False
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(_DEBUG_SCHEMA_SQL)
|
||||
conn.commit()
|
||||
global _DEBUG_SCHEMA_READY
|
||||
if _DEBUG_SCHEMA_READY:
|
||||
return
|
||||
with _DEBUG_SCHEMA_LOCK:
|
||||
if _DEBUG_SCHEMA_READY:
|
||||
return
|
||||
with get_db() as conn:
|
||||
conn.autocommit = False
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(_DEBUG_SCHEMA_SQL)
|
||||
conn.commit()
|
||||
_DEBUG_SCHEMA_READY = True
|
||||
|
||||
|
||||
def append_debug_event(
|
||||
@@ -159,12 +170,185 @@ def append_debug_event(
|
||||
return record_id
|
||||
|
||||
|
||||
def _load_debug_event_record(record_id: int) -> dict | None:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT id, source_region, node_code, service, event_type, level, message, payload_json, created_at
|
||||
FROM detect_debug_events
|
||||
WHERE id = %s
|
||||
LIMIT 1
|
||||
""",
|
||||
(int(record_id),),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if not row:
|
||||
return None
|
||||
return {
|
||||
"id": int(row[0]),
|
||||
"source_region": str(row[1] or ""),
|
||||
"node_code": str(row[2] or ""),
|
||||
"service": str(row[3] or ""),
|
||||
"event_type": str(row[4] or ""),
|
||||
"level": str(row[5] or "info"),
|
||||
"message": str(row[6] or ""),
|
||||
"payload": row[7] if isinstance(row[7], dict) else {},
|
||||
"created_at": _format_time(row[8]),
|
||||
}
|
||||
|
||||
|
||||
def _normalize_worker_log_event(debug_event: dict) -> dict | None:
|
||||
payload = dict(debug_event.get("payload") or {})
|
||||
message = _normalize_message(debug_event.get("message"), fallback="worker_log")
|
||||
domain = str(payload.get("domain") or "").strip().lower()
|
||||
status = str(payload.get("status") or "").strip().lower()
|
||||
if not domain and ":" in message:
|
||||
domain = message.rsplit(":", 1)[-1].strip().lower()
|
||||
|
||||
event_type = ""
|
||||
if status == "completed":
|
||||
event_type = "domain_completed"
|
||||
elif status == "failed":
|
||||
event_type = "domain_failed"
|
||||
elif status == "blacklisted":
|
||||
event_type = "domain_blacklisted"
|
||||
elif "开始检测域名" in message:
|
||||
event_type = "domain_started"
|
||||
elif "域名检测完成" in message:
|
||||
event_type = "domain_completed"
|
||||
payload.setdefault("status", "completed")
|
||||
elif "域名已命中黑名单" in message or "命中黑名单" in message:
|
||||
event_type = "domain_blacklisted"
|
||||
payload.setdefault("status", "blacklisted")
|
||||
elif "域名检测失败" in message:
|
||||
event_type = "domain_failed"
|
||||
payload.setdefault("status", "failed")
|
||||
|
||||
if not event_type or not domain:
|
||||
return None
|
||||
|
||||
payload.setdefault("domain", domain)
|
||||
payload["imported_from_debug_event"] = True
|
||||
payload["debug_event_record_id"] = int(debug_event.get("id") or 0)
|
||||
payload["debug_event_source_region"] = str(debug_event.get("source_region") or "")
|
||||
return {
|
||||
"node_code": str(debug_event.get("node_code") or "").strip(),
|
||||
"event_type": event_type,
|
||||
"level": str(debug_event.get("level") or "info"),
|
||||
"message": message,
|
||||
"payload": payload,
|
||||
"created_at": str(debug_event.get("created_at") or "").strip(),
|
||||
}
|
||||
|
||||
|
||||
def _ingest_worker_log_into_active_job(debug_event: dict) -> dict:
|
||||
if str(debug_event.get("event_type") or "").strip() != "worker_log":
|
||||
return {"imported": False, "reason": "not_worker_log"}
|
||||
|
||||
normalized_event = _normalize_worker_log_event(debug_event)
|
||||
if not normalized_event:
|
||||
return {"imported": False, "reason": "not_domain_progress_event"}
|
||||
|
||||
from app.services.detect_job_service import get_active_detect_job_summary
|
||||
from app.services.sync_push_service import (
|
||||
_apply_detect_result_event_to_domain,
|
||||
_apply_detect_result_event_to_job_item,
|
||||
)
|
||||
|
||||
active_job = get_active_detect_job_summary(event_limit=1) or {}
|
||||
target_job_id = int(active_job.get("job_id") or 0)
|
||||
if target_job_id <= 0:
|
||||
return {"imported": False, "reason": "no_active_job"}
|
||||
|
||||
debug_event_record_id = int(debug_event.get("id") or 0)
|
||||
updated_job_items = 0
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT id
|
||||
FROM detect_run_events
|
||||
WHERE job_id = %s
|
||||
AND (payload_json->>'debug_event_record_id') = %s
|
||||
ORDER BY id DESC
|
||||
LIMIT 1
|
||||
""",
|
||||
(target_job_id, str(debug_event_record_id)),
|
||||
)
|
||||
existing = cur.fetchone()
|
||||
if existing:
|
||||
return {
|
||||
"imported": False,
|
||||
"reason": "deduplicated",
|
||||
"target_job_id": target_job_id,
|
||||
"detect_run_event_id": int(existing[0]),
|
||||
}
|
||||
|
||||
created_at = _parse_time(normalized_event.get("created_at"))
|
||||
payload_json = _safe_json_dumps(normalized_event.get("payload") or {})
|
||||
if created_at:
|
||||
cur.execute(
|
||||
"""
|
||||
INSERT INTO detect_run_events (
|
||||
job_id, node_code, event_type, level, message, payload_json, created_at
|
||||
) VALUES (%s, %s, %s, %s, %s, %s::jsonb, %s)
|
||||
RETURNING id
|
||||
""",
|
||||
(
|
||||
target_job_id,
|
||||
normalized_event["node_code"],
|
||||
normalized_event["event_type"],
|
||||
normalized_event["level"],
|
||||
normalized_event["message"],
|
||||
payload_json,
|
||||
created_at,
|
||||
),
|
||||
)
|
||||
else:
|
||||
cur.execute(
|
||||
"""
|
||||
INSERT INTO detect_run_events (
|
||||
job_id, node_code, event_type, level, message, payload_json
|
||||
) VALUES (%s, %s, %s, %s, %s, %s::jsonb)
|
||||
RETURNING id
|
||||
""",
|
||||
(
|
||||
target_job_id,
|
||||
normalized_event["node_code"],
|
||||
normalized_event["event_type"],
|
||||
normalized_event["level"],
|
||||
normalized_event["message"],
|
||||
payload_json,
|
||||
),
|
||||
)
|
||||
detect_run_event_id = int((cur.fetchone() or [0])[0] or 0)
|
||||
_apply_detect_result_event_to_domain(cur, normalized_event)
|
||||
updated_job_items = _apply_detect_result_event_to_job_item(
|
||||
cur,
|
||||
target_job_id=target_job_id,
|
||||
event=normalized_event,
|
||||
)
|
||||
conn.commit()
|
||||
return {
|
||||
"imported": True,
|
||||
"reason": "imported",
|
||||
"target_job_id": target_job_id,
|
||||
"detect_run_event_id": detect_run_event_id,
|
||||
"updated_job_items": updated_job_items,
|
||||
"event_type": normalized_event["event_type"],
|
||||
"domain": str((normalized_event.get("payload") or {}).get("domain") or ""),
|
||||
}
|
||||
|
||||
|
||||
@db_read_retry()
|
||||
def list_debug_events(
|
||||
limit: int = 50,
|
||||
*,
|
||||
service: str | None = None,
|
||||
event_type: str | None = None,
|
||||
source_region: str | None = None,
|
||||
node_code: str | None = None,
|
||||
level: str | None = None,
|
||||
before_id: int | None = None,
|
||||
after_id: int | None = None,
|
||||
@@ -183,6 +367,9 @@ def list_debug_events(
|
||||
if str(source_region or "").strip():
|
||||
conditions.append("source_region = %s")
|
||||
params.append(str(source_region).strip())
|
||||
if str(node_code or "").strip():
|
||||
conditions.append("node_code = %s")
|
||||
params.append(str(node_code).strip())
|
||||
if str(level or "").strip():
|
||||
conditions.append("level = %s")
|
||||
params.append(str(level).strip())
|
||||
@@ -237,6 +424,7 @@ def list_debug_events(
|
||||
}
|
||||
|
||||
|
||||
@db_read_retry()
|
||||
def get_debug_event_overview(*, window_minutes: int = 10, source_region: str | None = None) -> dict:
|
||||
ensure_debug_event_schema()
|
||||
safe_window = max(1, min(int(window_minutes or 10), 180))
|
||||
@@ -671,7 +859,24 @@ def ingest_debug_event(payload: dict, *, shared_token: str | None = None) -> tup
|
||||
message=_normalize_message(payload.get("message"), fallback="remote debug event"),
|
||||
payload=payload.get("payload") or {},
|
||||
)
|
||||
return True, "调试事件接收成功", {"record_id": record_id}
|
||||
debug_event = _load_debug_event_record(record_id) or {
|
||||
"id": int(record_id),
|
||||
"source_region": str(payload.get("source_region") or settings.node_region),
|
||||
"node_code": str(payload.get("node_code") or ""),
|
||||
"service": str(payload.get("service") or "unknown"),
|
||||
"event_type": str(payload.get("event_type") or "event"),
|
||||
"level": str(payload.get("level") or "info"),
|
||||
"message": _normalize_message(payload.get("message"), fallback="remote debug event"),
|
||||
"payload": payload.get("payload") or {},
|
||||
"created_at": "",
|
||||
}
|
||||
job_import = {}
|
||||
if str(debug_event.get("service") or "").strip() == "worker-event":
|
||||
try:
|
||||
job_import = _ingest_worker_log_into_active_job(debug_event)
|
||||
except Exception as exc:
|
||||
job_import = {"imported": False, "reason": f"job_import_failed: {exc}"}
|
||||
return True, "调试事件接收成功", {"record_id": record_id, "job_import": job_import}
|
||||
|
||||
|
||||
def push_debug_event(
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -2,10 +2,14 @@ from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
import subprocess
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from app.core.config import settings
|
||||
from app.core.db import get_db
|
||||
from app.core.files import resolve_domain_path, tail_lines
|
||||
from app.core.redis_client import get_redis
|
||||
from app.services.debug_event_service import list_debug_events
|
||||
from app.services.cluster_runtime_service import ensure_runtime_schema
|
||||
from app.services.runtime_settings_service import get_runtime_settings
|
||||
from app.services.detect_run_service import sync_detect_runs
|
||||
from app.services.detect_job_service import get_active_detect_job_summary
|
||||
@@ -16,9 +20,79 @@ from app.services.worker_control_service import detect_worker_runtime
|
||||
|
||||
_PROXY_COUNT_RE = re.compile(r"当前可用代理数[:：]\s*(\d+)")
|
||||
_THREAD_COUNT_RE = re.compile(r"当前实际线程数量[:：]\s*(\d+)\s*/\s*(\d+)")
|
||||
_STEP_TRACE_DOMAIN_RE = re.compile(r"domain=([^\s|]+)")
|
||||
_REGISTER_DOMAIN_RE = re.compile(r"检测注册状态[:：]\s*([^\s]+)")
|
||||
_RUNTIME_STATE_KEY = "domain_tool:detect_runtime_state"
|
||||
_TIMESTAMP_FORMATS = ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S")
|
||||
_SYSLOG_TIMESTAMP_FORMAT = "%b %d %H:%M:%S"
|
||||
_REMOTE_LOG_MAX_CHARS = 500
|
||||
_REMOTE_DEBUG_EVENT_TYPES = {
|
||||
"worker_log",
|
||||
"active_job_snapshot",
|
||||
"domain_started",
|
||||
"domain_completed",
|
||||
"domain_failed",
|
||||
"domain_blacklisted",
|
||||
"task_pull_success",
|
||||
"task_pull_partial",
|
||||
"task_pull_failed",
|
||||
"queue_overdue_leases",
|
||||
}
|
||||
|
||||
|
||||
def _extract_remote_log_node_code(line: str) -> str:
|
||||
text = str(line or "").strip()
|
||||
if not text.startswith("["):
|
||||
return ""
|
||||
first_close = text.find("]")
|
||||
if first_close < 0:
|
||||
return ""
|
||||
second_open = text.find("[", first_close + 1)
|
||||
second_close = text.find("]", second_open + 1) if second_open >= 0 else -1
|
||||
if second_open < 0 or second_close < 0:
|
||||
return ""
|
||||
return text[second_open + 1:second_close].strip()
|
||||
|
||||
|
||||
def _slice_remote_log_lines_fairly(lines: list[str], *, limit: int = 240, min_per_node: int = 12) -> list[str]:
|
||||
safe_limit = max(1, int(limit or 240))
|
||||
if len(lines) <= safe_limit:
|
||||
return list(lines or [])
|
||||
|
||||
normalized_lines = [str(line or "").strip() for line in list(lines or []) if str(line or "").strip()]
|
||||
if len(normalized_lines) <= safe_limit:
|
||||
return normalized_lines
|
||||
|
||||
if min_per_node <= 0:
|
||||
return normalized_lines[-safe_limit:]
|
||||
|
||||
kept_indexes: set[int] = set()
|
||||
per_node_counts: dict[str, int] = {}
|
||||
for index in range(len(normalized_lines) - 1, -1, -1):
|
||||
node_code = _extract_remote_log_node_code(normalized_lines[index])
|
||||
if not node_code:
|
||||
continue
|
||||
current_count = int(per_node_counts.get(node_code, 0) or 0)
|
||||
if current_count >= min_per_node:
|
||||
continue
|
||||
kept_indexes.add(index)
|
||||
per_node_counts[node_code] = current_count + 1
|
||||
if len(kept_indexes) >= safe_limit:
|
||||
break
|
||||
|
||||
for index in range(len(normalized_lines) - 1, -1, -1):
|
||||
if len(kept_indexes) >= safe_limit:
|
||||
break
|
||||
kept_indexes.add(index)
|
||||
|
||||
return [normalized_lines[index] for index in sorted(kept_indexes)]
|
||||
|
||||
|
||||
def _runtime_state_key(node_code: str | None = None) -> str:
|
||||
normalized_node_code = str(node_code or settings.node_code or "").strip()
|
||||
if not normalized_node_code:
|
||||
return _RUNTIME_STATE_KEY
|
||||
return f"{_RUNTIME_STATE_KEY}:{normalized_node_code}"
|
||||
|
||||
|
||||
def _extract_dependency_alerts(lines: list[str]) -> list[dict]:
|
||||
@@ -83,6 +157,26 @@ def _extract_active_thread_snapshot(lines: list[str]) -> dict:
|
||||
return {"active": 0, "max": 0}
|
||||
|
||||
|
||||
def _estimate_active_threads_from_recent_lines(lines: list[str], *, limit: int) -> int:
|
||||
if not lines:
|
||||
return 0
|
||||
|
||||
active_domains: list[str] = []
|
||||
seen_domains: set[str] = set()
|
||||
for line in reversed(lines[-80:]):
|
||||
match = _STEP_TRACE_DOMAIN_RE.search(line) or _REGISTER_DOMAIN_RE.search(line)
|
||||
if not match:
|
||||
continue
|
||||
domain = str(match.group(1) or "").strip()
|
||||
if not domain or domain in seen_domains:
|
||||
continue
|
||||
seen_domains.add(domain)
|
||||
active_domains.append(domain)
|
||||
if len(active_domains) >= int(limit):
|
||||
break
|
||||
return len(active_domains)
|
||||
|
||||
|
||||
def _parse_time(raw: str | None) -> datetime | None:
|
||||
if not raw:
|
||||
return None
|
||||
@@ -101,10 +195,12 @@ def _parse_time(raw: str | None) -> datetime | None:
|
||||
return None
|
||||
|
||||
|
||||
def _extract_log_time(line: str) -> datetime | None:
|
||||
def _extract_log_time(line: str, *, reference_year: int | None = None) -> datetime | None:
|
||||
if len(line) < 19:
|
||||
return None
|
||||
candidates = [line[:26], line[:19]]
|
||||
text = str(line or "").strip()
|
||||
else:
|
||||
text = str(line or "")
|
||||
candidates = [text[:26], text[:19]]
|
||||
for candidate in candidates:
|
||||
for fmt in _TIMESTAMP_FORMATS:
|
||||
if len(candidate) != len(datetime.now().strftime(fmt)):
|
||||
@@ -113,14 +209,79 @@ def _extract_log_time(line: str) -> datetime | None:
|
||||
return datetime.strptime(candidate, fmt)
|
||||
except ValueError:
|
||||
continue
|
||||
syslog_candidate = str(text[:15] or "").strip()
|
||||
if syslog_candidate:
|
||||
try:
|
||||
parsed = datetime.strptime(syslog_candidate, _SYSLOG_TIMESTAMP_FORMAT)
|
||||
return parsed.replace(year=int(reference_year or datetime.now().year))
|
||||
except ValueError:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _read_worker_journal_lines(service_name: str, *, max_lines: int) -> tuple[list[str], str | None]:
|
||||
normalized_service_name = str(service_name or "").strip()
|
||||
if not normalized_service_name:
|
||||
return [], None
|
||||
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
["journalctl", "-u", normalized_service_name, "-n", str(max_lines), "--no-pager"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=12,
|
||||
)
|
||||
except Exception:
|
||||
return [], None
|
||||
|
||||
output = str(completed.stdout or "").strip()
|
||||
if completed.returncode != 0 or not output:
|
||||
return [], None
|
||||
|
||||
lines = [str(line or "").rstrip() for line in output.splitlines() if str(line or "").strip()]
|
||||
if not lines:
|
||||
return [], None
|
||||
return lines[-max_lines:], datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def _load_recent_worker_lines(runtime_settings: dict, *, max_lines: int = 160) -> tuple[bool, str | None, list[str]]:
|
||||
worker_log = resolve_domain_path("detect_worker.log", "logs/detect_worker.log")
|
||||
|
||||
worker_online = False
|
||||
last_log_time: str | None = None
|
||||
recent_lines = tail_lines("detect_worker.log", max_lines=max_lines)
|
||||
if worker_log and worker_log.exists():
|
||||
modified = datetime.fromtimestamp(worker_log.stat().st_mtime, tz=timezone.utc)
|
||||
last_log_time = modified.isoformat()
|
||||
worker_online = (datetime.now(timezone.utc) - modified).total_seconds() < 180
|
||||
|
||||
if str(runtime_settings.get("worker_mode") or "").strip() == "linux-systemd":
|
||||
service_name = str(runtime_settings.get("worker_service_name") or "").strip() or "domaincheck-worker"
|
||||
journal_lines, journal_last_time = _read_worker_journal_lines(service_name, max_lines=max_lines)
|
||||
if journal_lines:
|
||||
recent_lines = journal_lines
|
||||
worker_online = True
|
||||
if journal_last_time:
|
||||
last_log_time = journal_last_time
|
||||
|
||||
return worker_online, last_log_time, recent_lines
|
||||
|
||||
|
||||
def _filter_lines_since(lines: list[str], started_at: str | None) -> list[str]:
|
||||
started_time = _parse_time(started_at)
|
||||
if not started_time:
|
||||
return lines
|
||||
filtered = [line for line in lines if (_extract_log_time(line) or started_time) >= started_time]
|
||||
filtered: list[str] = []
|
||||
parsed_any = False
|
||||
for line in lines:
|
||||
line_time = _extract_log_time(line, reference_year=started_time.year)
|
||||
if line_time is None:
|
||||
continue
|
||||
parsed_any = True
|
||||
if line_time >= started_time:
|
||||
filtered.append(line)
|
||||
if not parsed_any:
|
||||
return lines
|
||||
return filtered or lines
|
||||
|
||||
|
||||
@@ -258,6 +419,204 @@ def _build_remote_log_snapshot(
|
||||
}
|
||||
|
||||
|
||||
def _build_remote_log_snapshot_from_debug_events(
|
||||
active_job: dict | None,
|
||||
*,
|
||||
enabled: bool,
|
||||
mode: str,
|
||||
limit: int = 240,
|
||||
) -> dict:
|
||||
if not enabled:
|
||||
return {
|
||||
"lines": [],
|
||||
"line_count": 0,
|
||||
"last_at": "",
|
||||
"last_line": "",
|
||||
"source_nodes": [],
|
||||
"source_node_count": 0,
|
||||
"source_node_summaries": [],
|
||||
}
|
||||
|
||||
normalized_mode = str(mode or "key").strip().lower()
|
||||
if normalized_mode not in {"key", "full"}:
|
||||
normalized_mode = "key"
|
||||
|
||||
participating_node_codes = {
|
||||
str(item.get("node_code") or "").strip()
|
||||
for item in list((active_job or {}).get("node_stats") or [])
|
||||
if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned"
|
||||
}
|
||||
created_after = (datetime.now() - timedelta(hours=6)).strftime("%Y-%m-%d %H:%M:%S")
|
||||
safe_limit = max(1, int(limit or 240))
|
||||
node_limit = max(40, min(200, safe_limit))
|
||||
records: list[dict] = []
|
||||
if participating_node_codes:
|
||||
for node_code in sorted(participating_node_codes):
|
||||
payload = list_debug_events(
|
||||
limit=node_limit,
|
||||
created_after=created_after,
|
||||
node_code=node_code,
|
||||
)
|
||||
records.extend(list(payload.get("records") or []))
|
||||
records.sort(
|
||||
key=lambda item: (
|
||||
str(item.get("created_at") or ""),
|
||||
int(item.get("id") or 0),
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
else:
|
||||
payload = list_debug_events(limit=max(safe_limit * 4, 240), created_after=created_after)
|
||||
records = list(payload.get("records") or [])
|
||||
if not records:
|
||||
return {
|
||||
"lines": [],
|
||||
"line_count": 0,
|
||||
"last_at": "",
|
||||
"last_line": "",
|
||||
"source_nodes": [],
|
||||
"source_node_count": 0,
|
||||
"source_node_summaries": [],
|
||||
}
|
||||
|
||||
lines: list[str] = []
|
||||
source_nodes: set[str] = set()
|
||||
source_node_summaries: dict[str, dict] = {}
|
||||
last_at = ""
|
||||
last_line = ""
|
||||
|
||||
for record in reversed(records):
|
||||
event_type = str(record.get("event_type") or "").strip()
|
||||
if event_type not in _REMOTE_DEBUG_EVENT_TYPES:
|
||||
continue
|
||||
node_code = str(record.get("node_code") or "").strip() or "unknown"
|
||||
if participating_node_codes and node_code not in participating_node_codes:
|
||||
continue
|
||||
message = str(record.get("message") or "").strip()
|
||||
if not message:
|
||||
continue
|
||||
created_at = str(record.get("created_at") or "").strip()
|
||||
payload = record.get("payload") if isinstance(record.get("payload"), dict) else {}
|
||||
event_mode = str(payload.get("log_mode") or "key").strip().lower()
|
||||
if event_mode not in {"key", "full"}:
|
||||
event_mode = "key"
|
||||
if normalized_mode != "full" and event_mode == "full":
|
||||
continue
|
||||
if len(message) > _REMOTE_LOG_MAX_CHARS:
|
||||
message = f"{message[:_REMOTE_LOG_MAX_CHARS]}..."
|
||||
formatted_line = f"[{created_at}] [{node_code}] {message}"
|
||||
lines.append(formatted_line)
|
||||
source_nodes.add(node_code)
|
||||
node_summary = source_node_summaries.setdefault(
|
||||
node_code,
|
||||
{
|
||||
"node_code": node_code,
|
||||
"line_count": 0,
|
||||
"key_line_count": 0,
|
||||
"full_line_count": 0,
|
||||
"last_at": "",
|
||||
"last_line": "",
|
||||
},
|
||||
)
|
||||
node_summary["line_count"] += 1
|
||||
if event_mode == "full":
|
||||
node_summary["full_line_count"] += 1
|
||||
else:
|
||||
node_summary["key_line_count"] += 1
|
||||
node_summary["last_at"] = created_at
|
||||
node_summary["last_line"] = formatted_line
|
||||
last_at = created_at
|
||||
last_line = formatted_line
|
||||
|
||||
sliced_lines = _slice_remote_log_lines_fairly(lines, limit=safe_limit)
|
||||
sorted_source_node_summaries = sorted(
|
||||
source_node_summaries.values(),
|
||||
key=lambda item: (
|
||||
str(item.get("last_at") or ""),
|
||||
str(item.get("node_code") or ""),
|
||||
),
|
||||
reverse=True,
|
||||
)
|
||||
return {
|
||||
"lines": sliced_lines,
|
||||
"line_count": len(sliced_lines),
|
||||
"last_at": last_at,
|
||||
"last_line": last_line,
|
||||
"source_nodes": sorted(source_nodes),
|
||||
"source_node_count": len(source_nodes),
|
||||
"source_node_summaries": sorted_source_node_summaries,
|
||||
}
|
||||
|
||||
|
||||
def _merge_remote_log_snapshots(primary: dict, secondary: dict, *, limit: int = 240) -> dict:
|
||||
merged_lines: list[str] = []
|
||||
seen_lines: set[str] = set()
|
||||
for raw_line in list(primary.get("lines") or []) + list(secondary.get("lines") or []):
|
||||
line = str(raw_line or "").strip()
|
||||
if not line or line in seen_lines:
|
||||
continue
|
||||
seen_lines.add(line)
|
||||
merged_lines.append(line)
|
||||
if limit > 0:
|
||||
merged_lines = _slice_remote_log_lines_fairly(merged_lines, limit=limit)
|
||||
|
||||
summaries: dict[str, dict] = {}
|
||||
for snapshot in (primary, secondary):
|
||||
for raw_summary in list(snapshot.get("source_node_summaries") or []):
|
||||
if not isinstance(raw_summary, dict):
|
||||
continue
|
||||
node_code = str(raw_summary.get("node_code") or "").strip()
|
||||
if not node_code:
|
||||
continue
|
||||
summary = summaries.setdefault(
|
||||
node_code,
|
||||
{
|
||||
"node_code": node_code,
|
||||
"line_count": 0,
|
||||
"key_line_count": 0,
|
||||
"full_line_count": 0,
|
||||
"last_at": "",
|
||||
"last_line": "",
|
||||
},
|
||||
)
|
||||
summary["line_count"] = max(int(summary.get("line_count", 0) or 0), int(raw_summary.get("line_count", 0) or 0))
|
||||
summary["key_line_count"] = max(int(summary.get("key_line_count", 0) or 0), int(raw_summary.get("key_line_count", 0) or 0))
|
||||
summary["full_line_count"] = max(int(summary.get("full_line_count", 0) or 0), int(raw_summary.get("full_line_count", 0) or 0))
|
||||
raw_last_at = str(raw_summary.get("last_at") or "")
|
||||
if raw_last_at >= str(summary.get("last_at") or ""):
|
||||
summary["last_at"] = raw_last_at
|
||||
summary["last_line"] = str(raw_summary.get("last_line") or "")
|
||||
|
||||
source_nodes = sorted(
|
||||
{
|
||||
str(node_code or "").strip()
|
||||
for node_code in list(primary.get("source_nodes") or []) + list(secondary.get("source_nodes") or [])
|
||||
if str(node_code or "").strip()
|
||||
}
|
||||
)
|
||||
last_at = max(str(primary.get("last_at") or ""), str(secondary.get("last_at") or ""))
|
||||
last_line = str(primary.get("last_line") or "")
|
||||
if str(secondary.get("last_at") or "") >= str(primary.get("last_at") or ""):
|
||||
last_line = str(secondary.get("last_line") or last_line)
|
||||
|
||||
return {
|
||||
"lines": merged_lines,
|
||||
"line_count": len(merged_lines),
|
||||
"last_at": last_at,
|
||||
"last_line": last_line,
|
||||
"source_nodes": source_nodes,
|
||||
"source_node_count": len(source_nodes),
|
||||
"source_node_summaries": sorted(
|
||||
summaries.values(),
|
||||
key=lambda item: (
|
||||
str(item.get("last_at") or ""),
|
||||
str(item.get("node_code") or ""),
|
||||
),
|
||||
reverse=True,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _resolve_remote_log_lines(
|
||||
active_job: dict | None,
|
||||
runs: list[dict],
|
||||
@@ -277,17 +636,72 @@ def _resolve_remote_log_snapshot(
|
||||
mode: str,
|
||||
limit: int = 240,
|
||||
) -> dict:
|
||||
return _build_remote_log_snapshot(active_job, enabled=enabled, mode=mode, limit=limit)
|
||||
primary_snapshot = _build_remote_log_snapshot(active_job, enabled=enabled, mode=mode, limit=limit)
|
||||
debug_snapshot = _build_remote_log_snapshot_from_debug_events(active_job, enabled=enabled, mode=mode, limit=limit)
|
||||
if int(primary_snapshot.get("line_count", 0) or 0) <= 0:
|
||||
return debug_snapshot
|
||||
if int(debug_snapshot.get("line_count", 0) or 0) <= 0:
|
||||
return primary_snapshot
|
||||
return _merge_remote_log_snapshots(primary_snapshot, debug_snapshot, limit=limit)
|
||||
|
||||
|
||||
def _load_runtime_state() -> dict:
|
||||
try:
|
||||
redis_client = get_redis()
|
||||
raw = redis_client.get(_RUNTIME_STATE_KEY)
|
||||
if not raw:
|
||||
for key in (_runtime_state_key(), _RUNTIME_STATE_KEY):
|
||||
raw = redis_client.get(key)
|
||||
if not raw:
|
||||
continue
|
||||
data = json.loads(raw)
|
||||
if not isinstance(data, dict):
|
||||
continue
|
||||
if key == _RUNTIME_STATE_KEY:
|
||||
payload_node_code = str(data.get("node_code") or "").strip()
|
||||
if payload_node_code and payload_node_code != str(settings.node_code or "").strip():
|
||||
continue
|
||||
return data
|
||||
return {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _load_runtime_state_from_cluster_node() -> dict:
|
||||
try:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT current_load, metadata_json, last_heartbeat_at
|
||||
FROM detect_worker_nodes
|
||||
WHERE node_code = %s
|
||||
LIMIT 1
|
||||
""",
|
||||
(settings.node_code,),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if not row:
|
||||
return {}
|
||||
data = json.loads(raw)
|
||||
return data if isinstance(data, dict) else {}
|
||||
current_load, metadata_json, last_heartbeat_at = row
|
||||
metadata = metadata_json if isinstance(metadata_json, dict) else {}
|
||||
if not metadata:
|
||||
return {}
|
||||
runtime_state = {
|
||||
"node_code": settings.node_code,
|
||||
"phase": str(metadata.get("phase") or metadata.get("phase_label") or "").strip(),
|
||||
"detail": str(metadata.get("detail") or metadata.get("phase_detail") or "").strip(),
|
||||
"service_running": True,
|
||||
"detecting": bool(metadata.get("detecting", False) or int(current_load or 0) > 0),
|
||||
"stop_requested": False,
|
||||
"available_proxy_count": int(metadata.get("available_proxy_count", 0) or 0),
|
||||
"active_threads": int(metadata.get("active_threads", 0) or 0),
|
||||
"max_threads": int(metadata.get("max_threads", 0) or 0),
|
||||
"job_id": metadata.get("job_id"),
|
||||
"job_code": str(metadata.get("job_code") or metadata.get("active_job_code") or "").strip(),
|
||||
"updated_at": str(metadata.get("updated_at") or (_format_time(last_heartbeat_at) if last_heartbeat_at else "")).strip(),
|
||||
}
|
||||
if runtime_state["detail"] or runtime_state["active_threads"] > 0 or runtime_state["max_threads"] > 0:
|
||||
return runtime_state
|
||||
return {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
@@ -361,12 +775,51 @@ def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, a
|
||||
"supplier_empty": False,
|
||||
}
|
||||
|
||||
if refresh_status in {"", "未刷新"} and source_count > 0:
|
||||
if allow_direct:
|
||||
return {
|
||||
"state": "degraded_direct",
|
||||
"label": "等待首刷",
|
||||
"detail": f"代理配置已下发，但代理池尚未完成首轮刷新；当前先按直连继续执行；最近状态：{refresh_status or '未刷新'}",
|
||||
"direct_fallback_active": True,
|
||||
"reason": "proxy_not_refreshed_yet",
|
||||
"last_refresh_status": refresh_status or "未刷新",
|
||||
"last_refresh_time": refresh_time,
|
||||
"source_count": source_count,
|
||||
"raw_items": raw_items,
|
||||
"validated_count": validated,
|
||||
"available_count": available,
|
||||
"source_stats": source_stats,
|
||||
"supplier_empty": False,
|
||||
}
|
||||
return {
|
||||
"state": "warming_up",
|
||||
"label": "等待首刷",
|
||||
"detail": "代理配置已下发，但代理池尚未完成首轮刷新；由于未允许直连，检测链路会等待代理刷新完成",
|
||||
"direct_fallback_active": False,
|
||||
"reason": "proxy_not_refreshed_yet",
|
||||
"last_refresh_status": refresh_status or "未刷新",
|
||||
"last_refresh_time": refresh_time,
|
||||
"source_count": source_count,
|
||||
"raw_items": raw_items,
|
||||
"validated_count": validated,
|
||||
"available_count": available,
|
||||
"source_stats": source_stats,
|
||||
"supplier_empty": False,
|
||||
}
|
||||
|
||||
if allow_direct:
|
||||
detail = "代理池当前无可用代理，已自动降级为直连继续执行"
|
||||
reason = "no_available_proxy"
|
||||
if supplier_empty:
|
||||
reason = "supplier_empty_pool"
|
||||
detail = "代理源最近都返回正常响应，但原始代理数为 0，当前判断为供应池为空；系统已自动降级为直连继续执行"
|
||||
elif raw_items > 0 and validated > 0:
|
||||
reason = "proxy_validation_zero"
|
||||
detail = (
|
||||
f"代理源最近返回了 {raw_items} 个代理，已验证 {validated} 个，但当前 0 个可用；"
|
||||
"系统已自动降级为直连继续执行"
|
||||
)
|
||||
if refresh_status:
|
||||
detail = f"{detail}；最近状态：{refresh_status}"
|
||||
return {
|
||||
@@ -390,6 +843,12 @@ def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, a
|
||||
if supplier_empty:
|
||||
reason = "supplier_empty_pool"
|
||||
detail = "代理源最近都返回正常响应，但原始代理数为 0，当前判断为供应池为空；由于未允许直连，检测链路会等待代理恢复"
|
||||
elif raw_items > 0 and validated > 0:
|
||||
reason = "proxy_validation_zero"
|
||||
detail = (
|
||||
f"代理源最近返回了 {raw_items} 个代理，已验证 {validated} 个，但当前 0 个可用；"
|
||||
"由于未允许直连，检测链路会等待代理恢复"
|
||||
)
|
||||
if refresh_status:
|
||||
detail = f"{detail}；最近状态：{refresh_status}"
|
||||
return {
|
||||
@@ -410,36 +869,45 @@ def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, a
|
||||
|
||||
|
||||
def get_detect_status() -> dict:
|
||||
try:
|
||||
ensure_runtime_schema()
|
||||
except Exception:
|
||||
# Node agent heartbeats should degrade gracefully even if runtime schema
|
||||
# initialization is temporarily unavailable.
|
||||
pass
|
||||
|
||||
queries = {
|
||||
"pending": "select count(*) from domains where detect_status = 0",
|
||||
"completed": "select count(*) from domains where detect_status = 1",
|
||||
"running": "select count(*) from domains where detect_status = 2",
|
||||
"blacklisted": "select count(*) from domains where detect_status = 3",
|
||||
"failed": "select count(*) from domains where detect_status = 4",
|
||||
"registerable": "select count(*) from domains where detect_status = 1 and register_status = 2",
|
||||
}
|
||||
progress: dict[str, int] = {}
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
for key, query in queries.items():
|
||||
try:
|
||||
cur.execute(query)
|
||||
progress[key] = cur.fetchone()[0]
|
||||
except Exception:
|
||||
progress[key] = 0
|
||||
progress: dict[str, int] = {key: 0 for key in queries}
|
||||
try:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
for key, query in queries.items():
|
||||
try:
|
||||
cur.execute(query)
|
||||
progress[key] = cur.fetchone()[0]
|
||||
except Exception:
|
||||
progress[key] = 0
|
||||
except Exception:
|
||||
# Worker runtime/status pages should still render using runtime-state and
|
||||
# cluster fallbacks even when the local DB endpoint is temporarily wrong
|
||||
# or unreachable (for example remote worker nodes without direct DB access).
|
||||
pass
|
||||
|
||||
settings_payload = get_settings_payload()
|
||||
worker_log = resolve_domain_path("detect_worker.log", "logs/detect_worker.log")
|
||||
|
||||
worker_online = False
|
||||
last_log_time = None
|
||||
if worker_log and worker_log.exists():
|
||||
modified = datetime.fromtimestamp(worker_log.stat().st_mtime, tz=timezone.utc)
|
||||
last_log_time = modified.isoformat()
|
||||
worker_online = (datetime.now(timezone.utc) - modified).total_seconds() < 180
|
||||
|
||||
recent_lines = tail_lines("detect_worker.log", max_lines=160)
|
||||
runtime_settings = get_runtime_settings()
|
||||
worker_online, last_log_time, recent_lines = _load_recent_worker_lines(runtime_settings, max_lines=160)
|
||||
runtime = detect_worker_runtime()
|
||||
runtime_state = _load_runtime_state()
|
||||
runtime_started_at = runtime.get("latest_start_time", "") if 'runtime' in locals() else ""
|
||||
if not runtime_state:
|
||||
runtime_state = _load_runtime_state_from_cluster_node()
|
||||
runtime_started_at = runtime.get("latest_start_time", "")
|
||||
recent_lines = _filter_lines_since(recent_lines, runtime_started_at)
|
||||
available_proxy_count = _extract_available_proxy_count(recent_lines)
|
||||
active_thread_snapshot = _extract_active_thread_snapshot(recent_lines)
|
||||
@@ -449,21 +917,35 @@ def get_detect_status() -> dict:
|
||||
"active": int(runtime_state.get("active_threads", active_thread_snapshot["active"]) or active_thread_snapshot["active"]),
|
||||
"max": int(runtime_state.get("max_threads", active_thread_snapshot["max"]) or active_thread_snapshot["max"]),
|
||||
}
|
||||
progress_total = sum(progress.values())
|
||||
registerable_count = int(progress.get("registerable", 0) or 0)
|
||||
progress_total = sum(
|
||||
int(progress.get(key, 0) or 0)
|
||||
for key in ("pending", "completed", "running", "blacklisted", "failed")
|
||||
)
|
||||
progress_done = progress.get("completed", 0) + progress.get("blacklisted", 0) + progress.get("failed", 0)
|
||||
progress_percent = round((progress_done / progress_total) * 100, 2) if progress_total > 0 else 0
|
||||
runtime = detect_worker_runtime()
|
||||
runtime_started_at = runtime.get("latest_start_time", "")
|
||||
recent_lines = _filter_lines_since(recent_lines, runtime_started_at)
|
||||
recent_proxy_warning = _normalize_recent_warning(runtime_state, recent_lines, available_proxy_count)
|
||||
proxy_runtime = _build_proxy_runtime_snapshot(settings_payload, runtime_state, available_proxy_count)
|
||||
thread_count_resolution = resolve_thread_count(settings_payload=settings_payload)
|
||||
effective_thread_count = int(thread_count_resolution["effective_thread_count"])
|
||||
runtime_settings = get_runtime_settings()
|
||||
worker_online = worker_online or runtime.get("running", False)
|
||||
if runtime_state.get("service_running") is True:
|
||||
worker_online = True
|
||||
if not runtime_state.get("detecting", False) and not progress.get("running", 0):
|
||||
if active_thread_snapshot["active"] <= 0 and (runtime_state.get("detecting", False) or runtime.get("running", False)):
|
||||
estimated_active_threads = _estimate_active_threads_from_recent_lines(
|
||||
recent_lines,
|
||||
limit=max(1, effective_thread_count),
|
||||
)
|
||||
if estimated_active_threads > 0:
|
||||
active_thread_snapshot["active"] = estimated_active_threads
|
||||
inferred_detecting = bool(
|
||||
runtime_state.get("detecting", False)
|
||||
or int(progress.get("running", 0) or 0) > 0
|
||||
or int(active_thread_snapshot.get("active", 0) or 0) > 0
|
||||
)
|
||||
if not inferred_detecting:
|
||||
active_thread_snapshot = {"active": 0, "max": active_thread_snapshot["max"] or effective_thread_count}
|
||||
settings_summary = {
|
||||
"thread_count": effective_thread_count,
|
||||
@@ -475,10 +957,59 @@ def get_detect_status() -> dict:
|
||||
"allow_direct": settings_payload["proxy_config"].get("allow_direct", False),
|
||||
"proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])),
|
||||
}
|
||||
active_job = get_active_detect_job_summary(event_limit=240)
|
||||
try:
|
||||
active_job = get_active_detect_job_summary(event_limit=240)
|
||||
except Exception:
|
||||
active_job = None
|
||||
if settings.node_region == "overseas" and settings.node_role == "control" and active_job:
|
||||
progress = {
|
||||
"pending": int(active_job.get("items_pending", 0) or 0),
|
||||
"running": int(
|
||||
active_job.get("display_active_threads", active_job.get("display_items_running", active_job.get("items_running", 0)))
|
||||
or 0
|
||||
),
|
||||
"completed": int(active_job.get("items_completed", 0) or 0),
|
||||
"failed": int(active_job.get("items_failed", 0) or 0),
|
||||
"blacklisted": int(active_job.get("items_blacklisted", 0) or 0),
|
||||
"registerable": registerable_count,
|
||||
}
|
||||
progress_percent = float(active_job.get("progress_percent", 0) or 0)
|
||||
local_node_bucket = {}
|
||||
for item in list((active_job or {}).get("node_stats") or []):
|
||||
if str(item.get("node_code") or "").strip() == str(settings.node_code or "").strip():
|
||||
local_node_bucket = item
|
||||
break
|
||||
local_runtime_load = int(
|
||||
local_node_bucket.get("active_threads")
|
||||
or local_node_bucket.get("items_running")
|
||||
or 0
|
||||
)
|
||||
local_runtime_max_threads = int(local_node_bucket.get("max_threads", 0) or 0)
|
||||
if active_thread_snapshot["active"] <= 0 and local_runtime_load > 0:
|
||||
active_thread_snapshot["active"] = local_runtime_load
|
||||
if active_thread_snapshot["max"] <= 0:
|
||||
active_thread_snapshot["max"] = local_runtime_max_threads or effective_thread_count
|
||||
if settings.node_region == "overseas" and settings.node_role == "control" and active_job:
|
||||
distributed_node_stats = list(active_job.get("distributed_node_stats") or active_job.get("node_stats") or [])
|
||||
aggregated_active_threads = 0
|
||||
aggregated_max_threads = 0
|
||||
for item in distributed_node_stats:
|
||||
node_code = str(item.get("node_code") or "").strip()
|
||||
if not node_code or node_code == "unassigned":
|
||||
continue
|
||||
aggregated_active_threads += int(
|
||||
item.get("active_threads")
|
||||
or item.get("items_running")
|
||||
or 0
|
||||
)
|
||||
aggregated_max_threads += int(item.get("max_threads", 0) or 0)
|
||||
if aggregated_active_threads > 0:
|
||||
active_thread_snapshot["active"] = aggregated_active_threads
|
||||
if aggregated_max_threads > 0:
|
||||
active_thread_snapshot["max"] = aggregated_max_threads
|
||||
runtime_snapshot = {
|
||||
**runtime,
|
||||
"detecting": runtime_state.get("detecting", False),
|
||||
"detecting": inferred_detecting,
|
||||
"proxy_runtime_state": proxy_runtime["state"],
|
||||
"proxy_runtime_label": proxy_runtime["label"],
|
||||
"proxy_runtime_detail": proxy_runtime["detail"],
|
||||
@@ -518,7 +1049,7 @@ def get_detect_status() -> dict:
|
||||
"runtime_state": runtime_state,
|
||||
"phase_label": runtime_state.get("phase", ""),
|
||||
"phase_detail": runtime_state.get("detail", ""),
|
||||
"detecting": runtime_state.get("detecting", False),
|
||||
"detecting": inferred_detecting,
|
||||
"thread_count": effective_thread_count,
|
||||
"thread_count_default": int(thread_count_resolution["default_thread_count"]),
|
||||
"thread_count_source": str(thread_count_resolution["source"]),
|
||||
@@ -549,6 +1080,7 @@ def get_detect_status() -> dict:
|
||||
"progress_percent": progress_percent,
|
||||
"recent_event": runtime_state.get("detail") or _recent_event(recent_lines),
|
||||
"recent_warning": recent_proxy_warning,
|
||||
"aggregate_detect_view": bool(settings.node_region == "overseas" and settings.node_role == "control" and active_job),
|
||||
"log_lines": recent_lines,
|
||||
"remote_log_lines": remote_log_lines,
|
||||
"remote_log_line_count": int(remote_log_snapshot.get("line_count", 0) or 0),
|
||||
|
||||
@@ -145,7 +145,8 @@ def _write_txt(path, rows: list[dict]) -> None:
|
||||
|
||||
|
||||
def _write_csv(path, rows: list[dict]) -> None:
|
||||
with path.open("w", encoding="utf-8", newline="") as handle:
|
||||
# Add BOM so Excel on Chinese Windows opens CSV without mojibake.
|
||||
with path.open("w", encoding="utf-8-sig", newline="") as handle:
|
||||
writer = csv.writer(handle)
|
||||
writer.writerow([label for _, label in EXPORT_HEADERS])
|
||||
for row in rows:
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
import hashlib
|
||||
from io import StringIO
|
||||
import pickle
|
||||
import re
|
||||
import sys
|
||||
@@ -35,6 +37,9 @@ LEGACY_JUMING_COOKIE_FILES = [
|
||||
DELETE_LIST_SOURCE_TYPE = 2
|
||||
FIXED_PRICE_SOURCE_TYPE = 1
|
||||
JUMING_PREFERENCES_FILE = "juming_preferences.json"
|
||||
JUMING_DELETE_IMPORT_STATE_FILE = "juming_delete_import_state.json"
|
||||
IMPORT_BATCH_SIZE = 50000
|
||||
IMPORT_PROGRESS_EVERY = 100000
|
||||
|
||||
|
||||
class TaskStoppedError(RuntimeError):
|
||||
@@ -125,6 +130,118 @@ def _persist_juming_cookie(cookie_jar: RequestsCookieJar) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def _empty_import_stats() -> dict[str, int]:
|
||||
return {
|
||||
"total": 0,
|
||||
"valid": 0,
|
||||
"added": 0,
|
||||
"exists": 0,
|
||||
"invalid": 0,
|
||||
"failed": 0,
|
||||
}
|
||||
|
||||
|
||||
def _merge_import_stats(base: dict[str, int], delta: dict[str, int]) -> dict[str, int]:
|
||||
merged = dict(base or _empty_import_stats())
|
||||
for key in ("total", "valid", "added", "exists", "invalid", "failed"):
|
||||
merged[key] = int(merged.get(key, 0) or 0) + int((delta or {}).get(key, 0) or 0)
|
||||
return merged
|
||||
|
||||
|
||||
def _load_delete_import_state() -> dict[str, dict]:
|
||||
payload = read_runtime_json(JUMING_DELETE_IMPORT_STATE_FILE, default={})
|
||||
return payload if isinstance(payload, dict) else {}
|
||||
|
||||
|
||||
def _save_delete_import_state(payload: dict[str, dict]) -> None:
|
||||
write_runtime_json(JUMING_DELETE_IMPORT_STATE_FILE, payload)
|
||||
|
||||
|
||||
def _compute_domains_signature(domains: list[str]) -> str:
|
||||
digest = hashlib.sha1()
|
||||
for domain in domains:
|
||||
digest.update(str(domain).strip().encode("utf-8", errors="ignore"))
|
||||
digest.update(b"\n")
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _looks_like_login_redirect(location: str) -> bool:
|
||||
normalized = str(location or "").strip().lower()
|
||||
if not normalized:
|
||||
return False
|
||||
return any(
|
||||
marker in normalized
|
||||
for marker in (
|
||||
"/login",
|
||||
"user_zh",
|
||||
"p_login",
|
||||
"passport",
|
||||
"sign",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _looks_like_login_body(body: str) -> bool:
|
||||
normalized = str(body or "").strip().lower()
|
||||
if not normalized:
|
||||
return False
|
||||
return any(
|
||||
marker in normalized
|
||||
for marker in (
|
||||
"账号登录",
|
||||
"请先登录",
|
||||
"登录后查看",
|
||||
"登录聚名",
|
||||
"user_zh",
|
||||
"p_login",
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _validate_juming_cookie(
|
||||
cookie_jar: RequestsCookieJar | None,
|
||||
*,
|
||||
probe_date: str | None = None,
|
||||
) -> tuple[bool, str]:
|
||||
if cookie_jar is None or not _cookie_jar_to_dict(cookie_jar):
|
||||
return False, "未检测到有效 Cookie"
|
||||
|
||||
jm = JM()
|
||||
jm.cookie = cookie_jar
|
||||
probe_date = str(probe_date or date.today().isoformat())
|
||||
url = f"{jm.base_url}/newcha/del_down?scsj={probe_date}"
|
||||
|
||||
try:
|
||||
response = jm.session.get(
|
||||
url,
|
||||
headers=jm.headers,
|
||||
cookies=jm.cookie,
|
||||
allow_redirects=False,
|
||||
timeout=10,
|
||||
)
|
||||
except Exception as exc:
|
||||
return False, f"登录态校验失败: {exc}"
|
||||
|
||||
location = str(response.headers.get("Location") or "").strip()
|
||||
if response.status_code in {301, 302, 303, 307, 308}:
|
||||
if _looks_like_login_redirect(location):
|
||||
return False, "聚名登录态已失效，请重新登录"
|
||||
if location:
|
||||
return True, f"删除列表下载链路校验通过: {probe_date}"
|
||||
|
||||
try:
|
||||
body = response.text
|
||||
except Exception:
|
||||
body = ""
|
||||
|
||||
if _looks_like_login_body(body):
|
||||
return False, "聚名登录态已失效，请重新登录"
|
||||
|
||||
if response.ok:
|
||||
return True, f"聚名 Cookie 已通过远端校验: {probe_date}"
|
||||
return False, f"聚名登录态校验失败，HTTP {response.status_code}"
|
||||
|
||||
|
||||
def _jucha_cookie_status() -> dict:
|
||||
if JUCHA_COOKIE_FILE.exists():
|
||||
return {
|
||||
@@ -186,11 +303,16 @@ def _load_juming_cookie() -> tuple[RequestsCookieJar | None, str]:
|
||||
|
||||
def get_juming_status() -> dict:
|
||||
cookie_jar, storage = _load_juming_cookie()
|
||||
cookie_valid, cookie_message = _validate_juming_cookie(cookie_jar)
|
||||
cookie_count = len(_cookie_jar_to_dict(cookie_jar)) if cookie_jar is not None else 0
|
||||
status = {
|
||||
"cookie_ready": cookie_jar is not None,
|
||||
"cookie_ready": bool(cookie_jar is not None and cookie_valid),
|
||||
"cookie_present": cookie_jar is not None,
|
||||
"cookie_valid": cookie_valid,
|
||||
"cookie_message": cookie_message,
|
||||
"cookie_storage": storage,
|
||||
"cookie_file": str(JUMING_COOKIE_FILE),
|
||||
"cookie_count": len(_cookie_jar_to_dict(cookie_jar)) if cookie_jar is not None else 0,
|
||||
"cookie_count": cookie_count,
|
||||
"jucha": _jucha_cookie_status(),
|
||||
"supported_modes": [
|
||||
{"label": "聚名一口价", "value": "fixed_price", "source_type": FIXED_PRICE_SOURCE_TYPE},
|
||||
@@ -308,80 +430,146 @@ def _insert_domains(
|
||||
source_type: int,
|
||||
log: Callable[[str], None] | None = None,
|
||||
should_stop: Callable[[], bool] | None = None,
|
||||
*,
|
||||
announce_total: bool = True,
|
||||
progress_label: str = "",
|
||||
) -> dict:
|
||||
total = len(domains)
|
||||
normalized_rows: list[tuple[str, str]] = []
|
||||
invalid = 0
|
||||
|
||||
_emit_log(log, f"开始入库处理，共收到 {total} 个原始域名")
|
||||
for value in domains:
|
||||
_check_stop(should_stop)
|
||||
normalized = normalize_domain(value)
|
||||
if not normalized:
|
||||
invalid += 1
|
||||
continue
|
||||
tld = normalized.rsplit(".", 1)[-1]
|
||||
normalized_rows.append((normalized, tld))
|
||||
|
||||
existing_set: set[str] = set()
|
||||
progress_prefix = f"{progress_label} " if str(progress_label or "").strip() else ""
|
||||
if announce_total:
|
||||
_emit_log(log, f"{progress_prefix}开始入库处理，共收到 {total} 个原始域名")
|
||||
inserted = 0
|
||||
existing = 0
|
||||
processed = 0
|
||||
valid = 0
|
||||
last_progress_at = 0
|
||||
pending_batch: list[tuple[str, str]] = []
|
||||
pending_seen: set[str] = set()
|
||||
stage_ready = False
|
||||
|
||||
def emit_progress(force: bool = False) -> None:
|
||||
nonlocal last_progress_at
|
||||
if not force and processed - last_progress_at < IMPORT_PROGRESS_EVERY:
|
||||
return
|
||||
last_progress_at = processed
|
||||
_emit_log(
|
||||
log,
|
||||
(
|
||||
f"{progress_prefix}入库进度：已处理 {processed}/{total}，"
|
||||
f"有效 {valid}，新增 {inserted}，已存在 {existing}，无效 {invalid}"
|
||||
),
|
||||
)
|
||||
|
||||
def ensure_stage_table(cur) -> None:
|
||||
nonlocal stage_ready
|
||||
if stage_ready:
|
||||
return
|
||||
cur.execute(
|
||||
"""
|
||||
create temporary table if not exists juming_import_stage (
|
||||
domain text primary key,
|
||||
tld text not null
|
||||
) on commit preserve rows
|
||||
"""
|
||||
)
|
||||
stage_ready = True
|
||||
|
||||
def stage_rows(cur, rows: list[tuple[str, str]]) -> None:
|
||||
buffer = StringIO()
|
||||
for domain, tld in rows:
|
||||
buffer.write(f"{domain}\t{tld}\n")
|
||||
buffer.seek(0)
|
||||
cur.copy_from(buffer, "juming_import_stage", columns=("domain", "tld"))
|
||||
|
||||
def flush_batch(cur, conn) -> None:
|
||||
nonlocal inserted, existing
|
||||
if not pending_batch:
|
||||
return
|
||||
ensure_stage_table(cur)
|
||||
cur.execute("set local synchronous_commit = off")
|
||||
cur.execute("truncate table juming_import_stage")
|
||||
stage_rows(cur, pending_batch)
|
||||
cur.execute(
|
||||
"""
|
||||
with existing_rows as (
|
||||
select count(*)
|
||||
from juming_import_stage stage
|
||||
join domains existing on existing.domain = stage.domain
|
||||
),
|
||||
inserted as (
|
||||
insert into domains (
|
||||
domain, tld, source_type, use_status, detect_status, register_status,
|
||||
has_beian, company_type, website_url, beian_year, snapshot_years,
|
||||
expire_date, create_time, update_time, review_status, detect_time,
|
||||
backlink_count, jucha_status, juziseo_status
|
||||
)
|
||||
select
|
||||
stage.domain,
|
||||
stage.tld,
|
||||
%s,
|
||||
0, 0, 0,
|
||||
1, null, null, null, null,
|
||||
null, now(), now(), 0, null,
|
||||
0, 0, 0
|
||||
from juming_import_stage stage
|
||||
left join domains existing on existing.domain = stage.domain
|
||||
where existing.id is null
|
||||
returning id
|
||||
),
|
||||
task_insert as (
|
||||
insert into detect_tasks (
|
||||
domain_id, task_type, status, priority, retry_count, create_time, update_time
|
||||
)
|
||||
select id, 1, 1, 5, 0, now(), now()
|
||||
from inserted
|
||||
returning 1
|
||||
)
|
||||
select
|
||||
(select count(*) from inserted),
|
||||
(select count(*) from task_insert),
|
||||
(select count(*) from existing_rows)
|
||||
""",
|
||||
(source_type,),
|
||||
)
|
||||
inserted_count, _task_count, existing_count = cur.fetchone()
|
||||
inserted += int(inserted_count or 0)
|
||||
existing += int(existing_count or 0)
|
||||
conn.commit()
|
||||
pending_batch.clear()
|
||||
pending_seen.clear()
|
||||
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
normalized_domains = [row[0] for row in normalized_rows]
|
||||
if normalized_domains:
|
||||
cur.execute("select domain from domains where domain = any(%s)", (normalized_domains,))
|
||||
existing_set = {row[0] for row in cur.fetchall()}
|
||||
if existing_set:
|
||||
_emit_log(log, f"检测到 {len(existing_set)} 个已存在域名，将自动跳过")
|
||||
|
||||
inserted_since_commit = 0
|
||||
for domain, tld in normalized_rows:
|
||||
for value in domains:
|
||||
_check_stop(should_stop)
|
||||
if domain in existing_set:
|
||||
processed += 1
|
||||
normalized = normalize_domain(value)
|
||||
if not normalized:
|
||||
invalid += 1
|
||||
emit_progress()
|
||||
continue
|
||||
cur.execute(
|
||||
"""
|
||||
insert into domains (
|
||||
domain, tld, source_type, use_status, detect_status, register_status,
|
||||
has_beian, company_type, website_url, beian_year, snapshot_years,
|
||||
expire_date, create_time, update_time, review_status, detect_time,
|
||||
backlink_count, jucha_status, juziseo_status
|
||||
) values (
|
||||
%s, %s, %s, 0, 0, 0,
|
||||
1, null, null, null, null,
|
||||
null, now(), now(), 0, null,
|
||||
0, 0, 0
|
||||
)
|
||||
returning id
|
||||
""",
|
||||
(domain, tld, source_type),
|
||||
)
|
||||
domain_id = cur.fetchone()[0]
|
||||
cur.execute(
|
||||
"""
|
||||
insert into detect_tasks (domain_id, task_type, status, priority, retry_count, create_time, update_time)
|
||||
values (%s, 1, 1, 5, 0, now(), now())
|
||||
""",
|
||||
(domain_id,),
|
||||
)
|
||||
inserted += 1
|
||||
inserted_since_commit += 1
|
||||
if inserted_since_commit >= 500:
|
||||
conn.commit()
|
||||
inserted_since_commit = 0
|
||||
conn.commit()
|
||||
valid += 1
|
||||
if normalized in pending_seen:
|
||||
existing += 1
|
||||
emit_progress()
|
||||
continue
|
||||
pending_seen.add(normalized)
|
||||
pending_batch.append((normalized, normalized.rsplit(".", 1)[-1]))
|
||||
if len(pending_batch) >= IMPORT_BATCH_SIZE:
|
||||
flush_batch(cur, conn)
|
||||
emit_progress(force=True)
|
||||
flush_batch(cur, conn)
|
||||
emit_progress(force=True)
|
||||
|
||||
valid = len(normalized_rows)
|
||||
exists = len(existing_set)
|
||||
_emit_log(log, f"入库完成：有效 {valid}，新增 {inserted}，已存在 {exists}，无效 {invalid}")
|
||||
_emit_log(log, f"{progress_prefix}入库完成：有效 {valid}，新增 {inserted}，已存在 {existing}，无效 {invalid}")
|
||||
return {
|
||||
"total": total,
|
||||
"valid": valid,
|
||||
"added": inserted,
|
||||
"exists": exists,
|
||||
"exists": existing,
|
||||
"invalid": invalid,
|
||||
"failed": max(valid - exists - inserted, 0),
|
||||
"failed": max(valid - existing - inserted, 0),
|
||||
}
|
||||
|
||||
|
||||
@@ -450,6 +638,98 @@ def _crawl_delete_list(
|
||||
return domains, dates
|
||||
|
||||
|
||||
def _crawl_delete_list_and_import(
|
||||
crawl_date: str,
|
||||
auto_date: bool,
|
||||
log: Callable[[str], None] | None = None,
|
||||
should_stop: Callable[[], bool] | None = None,
|
||||
) -> dict:
|
||||
cookie_jar, _ = _load_juming_cookie()
|
||||
jm = JM()
|
||||
jm.cookie = cookie_jar or RequestsCookieJar()
|
||||
|
||||
start_date = datetime.strptime(crawl_date, "%Y-%m-%d").date()
|
||||
end_date = date.today() + timedelta(days=4)
|
||||
current_date = start_date
|
||||
dates: list[dict[str, int]] = []
|
||||
domains_found = 0
|
||||
sample_domains: list[str] = []
|
||||
stats = _empty_import_stats()
|
||||
import_state = _load_delete_import_state()
|
||||
_emit_log(log, f"开始采集删除列表：起始日期 {crawl_date}，自动追加日期 {'开启' if auto_date else '关闭'}")
|
||||
|
||||
while current_date <= end_date:
|
||||
_check_stop(should_stop)
|
||||
current_date_text = current_date.isoformat()
|
||||
_emit_log(log, f"正在抓取 {current_date_text} 的删除列表")
|
||||
domains_for_date = [item.strip() for item in jm.new_cha_del(current_date_text) if item.strip()]
|
||||
domains_found += len(domains_for_date)
|
||||
dates.append({"date": current_date_text, "count": len(domains_for_date)})
|
||||
_emit_log(log, f"{current_date_text} 抓取到 {len(domains_for_date)} 个域名，累计 {domains_found} 个")
|
||||
if domains_for_date:
|
||||
if len(sample_domains) < 20:
|
||||
sample_domains.extend(domains_for_date[: max(0, 20 - len(sample_domains))])
|
||||
signature = _compute_domains_signature(domains_for_date)
|
||||
cached = import_state.get(current_date_text) or {}
|
||||
if (
|
||||
cached.get("signature") == signature
|
||||
and int(cached.get("total", 0) or 0) == len(domains_for_date)
|
||||
):
|
||||
cached_valid = int(cached.get("valid", 0) or 0)
|
||||
cached_invalid = int(cached.get("invalid", 0) or 0)
|
||||
date_stats = {
|
||||
"total": len(domains_for_date),
|
||||
"valid": cached_valid,
|
||||
"added": 0,
|
||||
"exists": cached_valid,
|
||||
"invalid": cached_invalid,
|
||||
"failed": 0,
|
||||
}
|
||||
_emit_log(
|
||||
log,
|
||||
(
|
||||
f"{current_date_text} 删除列表内容未变化，跳过重复入库："
|
||||
f"有效 {cached_valid}，视为已存在 {cached_valid}，无效 {cached_invalid}"
|
||||
),
|
||||
)
|
||||
else:
|
||||
date_stats = _insert_domains(
|
||||
domains_for_date,
|
||||
DELETE_LIST_SOURCE_TYPE,
|
||||
log=log,
|
||||
should_stop=should_stop,
|
||||
announce_total=False,
|
||||
progress_label=current_date_text,
|
||||
)
|
||||
import_state[current_date_text] = {
|
||||
"signature": signature,
|
||||
"total": int(date_stats.get("total", 0) or 0),
|
||||
"valid": int(date_stats.get("valid", 0) or 0),
|
||||
"invalid": int(date_stats.get("invalid", 0) or 0),
|
||||
"updated_at": datetime.now().isoformat(sep=" ", timespec="seconds"),
|
||||
}
|
||||
_save_delete_import_state(import_state)
|
||||
stats = _merge_import_stats(stats, date_stats)
|
||||
if not auto_date:
|
||||
break
|
||||
current_date += timedelta(days=1)
|
||||
|
||||
_emit_log(
|
||||
log,
|
||||
(
|
||||
f"删除列表采集+入库完成：抓取 {domains_found} 个域名，"
|
||||
f"新增 {stats['added']}，已存在 {stats['exists']}，无效 {stats['invalid']}"
|
||||
),
|
||||
)
|
||||
return {
|
||||
"mode": "delete_list",
|
||||
"dates": dates,
|
||||
"domains_found": domains_found,
|
||||
"stats": stats,
|
||||
"sample_domains": sample_domains[:20],
|
||||
}
|
||||
|
||||
|
||||
def crawl_juming(
|
||||
payload: dict,
|
||||
log: Callable[[str], None] | None = None,
|
||||
@@ -462,7 +742,10 @@ def crawl_juming(
|
||||
cookie_jar, storage = _load_juming_cookie()
|
||||
if cookie_jar is None:
|
||||
raise ValueError("未找到聚名 Cookie，请先在桌面版系统设置完成聚名登录，或将 Cookie 同步到服务器")
|
||||
_emit_log(log, f"检测到聚名登录态，来源：{storage}")
|
||||
cookie_valid, cookie_message = _validate_juming_cookie(cookie_jar)
|
||||
if not cookie_valid:
|
||||
raise ValueError(cookie_message)
|
||||
_emit_log(log, f"检测到聚名登录态，来源：{storage}，远端校验通过")
|
||||
_check_stop(should_stop)
|
||||
|
||||
if mode == "fixed_price":
|
||||
@@ -482,13 +765,6 @@ def crawl_juming(
|
||||
|
||||
crawl_date = str(payload.get("crawl_date") or date.today().isoformat())
|
||||
auto_date = bool(payload.get("auto_date", True))
|
||||
domains, dates = _crawl_delete_list(crawl_date, auto_date, log=log, should_stop=should_stop)
|
||||
stats = _insert_domains(domains, DELETE_LIST_SOURCE_TYPE, log=log, should_stop=should_stop)
|
||||
return {
|
||||
"mode": mode,
|
||||
"cookie_storage": storage,
|
||||
"dates": dates,
|
||||
"domains_found": len(domains),
|
||||
"stats": stats,
|
||||
"sample_domains": domains[:20],
|
||||
}
|
||||
result = _crawl_delete_list_and_import(crawl_date, auto_date, log=log, should_stop=should_stop)
|
||||
result["cookie_storage"] = storage
|
||||
return result
|
||||
|
||||
@@ -10,6 +10,7 @@ from app.services.juming_service import TaskStoppedError, crawl_juming
|
||||
|
||||
_JUMING_TASK_LOCK = threading.Lock()
|
||||
_MAX_LOG_LINES = 400
|
||||
_ACTIVE_TASK_IDS: set[str] = set()
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
@@ -17,6 +18,7 @@ def _now() -> str:
|
||||
|
||||
|
||||
def list_juming_tasks() -> list[dict]:
|
||||
_cleanup_orphaned_tasks()
|
||||
return load_juming_records()
|
||||
|
||||
|
||||
@@ -55,6 +57,43 @@ def _is_stop_requested(task_id: str) -> bool:
|
||||
return bool(target and target.get("cancel_requested"))
|
||||
|
||||
|
||||
def _cleanup_orphaned_tasks() -> None:
|
||||
with _JUMING_TASK_LOCK:
|
||||
tasks = load_juming_records()
|
||||
changed = False
|
||||
for item in tasks:
|
||||
status = str(item.get("status") or "").strip()
|
||||
task_id = str(item.get("task_id") or "").strip()
|
||||
if status == "running" and task_id and task_id not in _ACTIVE_TASK_IDS:
|
||||
item["status"] = "failed"
|
||||
item["phase"] = "failed"
|
||||
item["phase_label"] = "失败"
|
||||
item["cancel_requested"] = False
|
||||
item["message"] = "任务因服务重启或进程中断而终止，请重新执行"
|
||||
item["updated_at"] = _now()
|
||||
logs = list(item.get("logs") or [])
|
||||
logs.append(f"[{_now()}] 任务因服务重启或进程中断而终止，请重新执行")
|
||||
item["logs"] = logs[-_MAX_LOG_LINES:]
|
||||
changed = True
|
||||
if changed:
|
||||
_save_tasks(tasks)
|
||||
|
||||
|
||||
def _ensure_no_active_task() -> None:
|
||||
_cleanup_orphaned_tasks()
|
||||
tasks = load_juming_records()
|
||||
active = next(
|
||||
(
|
||||
item
|
||||
for item in tasks
|
||||
if str(item.get("status") or "").strip() == "running"
|
||||
),
|
||||
None,
|
||||
)
|
||||
if active:
|
||||
raise ValueError(f"已有聚名采集任务正在运行：{active.get('task_id')}")
|
||||
|
||||
|
||||
def _set_phase(task_id: str, phase: str, message: str | None = None) -> None:
|
||||
phase_labels = {
|
||||
"queued": "排队中",
|
||||
@@ -78,12 +117,18 @@ def _set_phase(task_id: str, phase: str, message: str | None = None) -> None:
|
||||
def _log_and_track_phase(task_id: str, message: str) -> None:
|
||||
if "开始采集" in message or "正在抓取" in message:
|
||||
_set_phase(task_id, "fetching", message)
|
||||
elif "开始入库处理" in message or "入库完成" in message or "已存在域名" in message:
|
||||
elif (
|
||||
"开始入库处理" in message
|
||||
or "入库进度" in message
|
||||
or "入库完成" in message
|
||||
or "已存在域名" in message
|
||||
):
|
||||
_set_phase(task_id, "importing", message)
|
||||
_append_log(task_id, message)
|
||||
|
||||
|
||||
def _run_juming_task(task_id: str, payload: dict) -> None:
|
||||
_ACTIVE_TASK_IDS.add(task_id)
|
||||
_update_task(task_id, status="running", started_at=_now(), message="聚名采集任务开始执行", cancel_requested=False)
|
||||
_set_phase(task_id, "starting", "聚名采集任务开始执行")
|
||||
_append_log(task_id, "任务已启动，正在准备读取聚名登录态")
|
||||
@@ -127,9 +172,12 @@ def _run_juming_task(task_id: str, payload: dict) -> None:
|
||||
)
|
||||
_set_phase(task_id, "failed")
|
||||
_append_log(task_id, f"任务执行失败：{exc}")
|
||||
finally:
|
||||
_ACTIVE_TASK_IDS.discard(task_id)
|
||||
|
||||
|
||||
def create_juming_task(payload: dict) -> dict:
|
||||
_ensure_no_active_task()
|
||||
task_id = uuid4().hex
|
||||
mode = str(payload.get("mode") or "delete_list").strip() or "delete_list"
|
||||
record = {
|
||||
@@ -154,12 +202,14 @@ def create_juming_task(payload: dict) -> dict:
|
||||
tasks.insert(0, record)
|
||||
_save_tasks(tasks)
|
||||
|
||||
_ACTIVE_TASK_IDS.add(task_id)
|
||||
worker = threading.Thread(target=_run_juming_task, args=(task_id, dict(payload or {})), daemon=True)
|
||||
worker.start()
|
||||
return record
|
||||
|
||||
|
||||
def retry_juming_task(task_id: str) -> dict:
|
||||
_ensure_no_active_task()
|
||||
with _JUMING_TASK_LOCK:
|
||||
tasks = load_juming_records()
|
||||
target = next((item for item in tasks if item["task_id"] == task_id), None)
|
||||
|
||||
@@ -1,9 +1,15 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
import psycopg2
|
||||
import redis
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.files import runtime_root as api_runtime_root
|
||||
|
||||
|
||||
STRUCTURED_ACTIONS = {
|
||||
@@ -22,6 +28,7 @@ STRUCTURED_ACTIONS = {
|
||||
"runtime.restart_api",
|
||||
"runtime.start_sync_agent",
|
||||
"runtime.stop_sync_agent",
|
||||
"runtime.reset_lab_state",
|
||||
}
|
||||
|
||||
|
||||
@@ -140,6 +147,233 @@ def systemctl_action_name(action: str) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _safe_bool(value: object, default: bool) -> bool:
|
||||
if value is None:
|
||||
return bool(default)
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
text = str(value).strip().lower()
|
||||
if not text:
|
||||
return bool(default)
|
||||
if text in {"1", "true", "yes", "on"}:
|
||||
return True
|
||||
if text in {"0", "false", "no", "off"}:
|
||||
return False
|
||||
return bool(default)
|
||||
|
||||
|
||||
def _node_agent_queue_dir() -> Path:
|
||||
explicit = str(os.getenv("OPS_AGENT_QUEUE_DIR", "") or "").strip()
|
||||
if explicit:
|
||||
return Path(explicit)
|
||||
project_dir = Path(__file__).resolve().parents[2]
|
||||
return project_dir / "runtime" / "node-agent-queue" / (settings.node_code or "unbound")
|
||||
|
||||
|
||||
def _detect_install_root(base_path: Path) -> Path | None:
|
||||
normalized = str(base_path.resolve())
|
||||
for marker in (f"{os.sep}releases{os.sep}", f"{os.sep}current{os.sep}"):
|
||||
if marker in normalized:
|
||||
return Path(normalized.split(marker, 1)[0])
|
||||
return None
|
||||
|
||||
|
||||
def _domaincheck_runtime_root() -> Path:
|
||||
explicit = str(os.getenv("DOMAINCHECK_RUNTIME_ROOT", "") or "").strip()
|
||||
if explicit:
|
||||
return Path(explicit)
|
||||
domain_root = Path(settings.domain_root).resolve()
|
||||
install_root = _detect_install_root(domain_root)
|
||||
if install_root is not None:
|
||||
return install_root / "runtime" / "domainCheck"
|
||||
return domain_root
|
||||
|
||||
|
||||
def _clear_path_contents(path: Path, *, preserve_names: set[str] | None = None) -> list[str]:
|
||||
if not path.exists():
|
||||
return []
|
||||
cleared: list[str] = []
|
||||
preserved = preserve_names or set()
|
||||
for child in path.iterdir():
|
||||
if child.name in preserved:
|
||||
continue
|
||||
if child.is_dir() and not child.is_symlink():
|
||||
shutil.rmtree(child, ignore_errors=False)
|
||||
else:
|
||||
child.unlink(missing_ok=True)
|
||||
cleared.append(str(child))
|
||||
return cleared
|
||||
|
||||
|
||||
def _reset_runtime_files(
|
||||
*,
|
||||
include_api_runtime: bool,
|
||||
include_worker_runtime: bool,
|
||||
include_node_agent_queue: bool,
|
||||
) -> dict:
|
||||
cleared_targets: dict[str, list[str]] = {}
|
||||
if include_api_runtime:
|
||||
api_root = api_runtime_root()
|
||||
cleared_targets["api_runtime"] = _clear_path_contents(api_root, preserve_names={".env"})
|
||||
|
||||
if include_worker_runtime:
|
||||
worker_root = _domaincheck_runtime_root()
|
||||
worker_targets: dict[str, list[str]] = {}
|
||||
for name in ("data", "logs"):
|
||||
target = worker_root / name
|
||||
worker_targets[name] = _clear_path_contents(target, preserve_names={".env"}) if target.exists() else []
|
||||
detect_worker_log = Path(settings.domain_root) / "detect_worker.log"
|
||||
if detect_worker_log.exists():
|
||||
detect_worker_log.unlink(missing_ok=True)
|
||||
worker_targets["files"] = [str(detect_worker_log)]
|
||||
cleared_targets["worker_runtime"] = [
|
||||
item for values in worker_targets.values() for item in values
|
||||
]
|
||||
|
||||
if include_node_agent_queue:
|
||||
queue_dir = _node_agent_queue_dir()
|
||||
cleared_targets["node_agent_queue"] = _clear_path_contents(queue_dir) if queue_dir.exists() else []
|
||||
|
||||
return cleared_targets
|
||||
|
||||
|
||||
def _truncate_detect_runtime_tables(*, include_domains: bool) -> dict:
|
||||
table_names = [
|
||||
"detect_debug_events",
|
||||
"detect_run_events",
|
||||
"detect_job_items",
|
||||
"detect_jobs",
|
||||
"detect_sync_records",
|
||||
"detect_tasks",
|
||||
"detect_worker_nodes",
|
||||
"domain_detections",
|
||||
"domain_blacklist",
|
||||
]
|
||||
if include_domains:
|
||||
table_names.append("domains")
|
||||
|
||||
conn = psycopg2.connect(
|
||||
host=settings.db_host,
|
||||
port=settings.db_port,
|
||||
dbname=settings.db_database,
|
||||
user=settings.db_user,
|
||||
password=settings.db_password,
|
||||
)
|
||||
try:
|
||||
conn.autocommit = False
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
f"TRUNCATE TABLE {', '.join(table_names)} RESTART IDENTITY CASCADE"
|
||||
)
|
||||
conn.commit()
|
||||
finally:
|
||||
conn.close()
|
||||
return {"tables": table_names, "include_domains": bool(include_domains)}
|
||||
|
||||
|
||||
def _flush_runtime_redis() -> dict:
|
||||
client = redis.Redis(
|
||||
host=settings.redis_host,
|
||||
port=settings.redis_port,
|
||||
password=settings.redis_password or None,
|
||||
db=settings.redis_db,
|
||||
decode_responses=True,
|
||||
socket_connect_timeout=5,
|
||||
socket_timeout=5,
|
||||
)
|
||||
try:
|
||||
size_before = int(client.dbsize() or 0)
|
||||
client.flushdb()
|
||||
size_after = int(client.dbsize() or 0)
|
||||
finally:
|
||||
try:
|
||||
client.close()
|
||||
except Exception:
|
||||
pass
|
||||
return {"db": int(settings.redis_db), "size_before": size_before, "size_after": size_after}
|
||||
|
||||
|
||||
def _run_service_command(
|
||||
runner: CommandRunner,
|
||||
service_name: str,
|
||||
action: str,
|
||||
) -> dict:
|
||||
code, stdout, stderr = runner(["systemctl", action, service_name], timeout=60)
|
||||
return {
|
||||
"service_name": service_name,
|
||||
"action": action,
|
||||
"returncode": int(code or 0),
|
||||
"stdout": trim_output(stdout, 4000),
|
||||
"stderr": trim_output(stderr, 4000),
|
||||
"ok": int(code or 0) == 0,
|
||||
}
|
||||
|
||||
|
||||
def _reset_lab_state(
|
||||
payload: dict,
|
||||
*,
|
||||
runner: CommandRunner,
|
||||
) -> tuple[bool, str, dict]:
|
||||
normalized_payload = dict(payload or {})
|
||||
stop_services = [
|
||||
str(item).strip()
|
||||
for item in list(normalized_payload.get("stop_services") or [])
|
||||
if str(item).strip()
|
||||
]
|
||||
start_services = [
|
||||
str(item).strip()
|
||||
for item in list(normalized_payload.get("start_services") or [])
|
||||
if str(item).strip()
|
||||
]
|
||||
|
||||
clear_database = _safe_bool(normalized_payload.get("clear_database"), False)
|
||||
clear_domains = _safe_bool(normalized_payload.get("clear_domains"), False)
|
||||
clear_redis = _safe_bool(normalized_payload.get("clear_redis"), False)
|
||||
clear_api_runtime = _safe_bool(normalized_payload.get("clear_api_runtime"), False)
|
||||
clear_worker_runtime = _safe_bool(normalized_payload.get("clear_worker_runtime"), True)
|
||||
clear_node_agent_queue = _safe_bool(normalized_payload.get("clear_node_agent_queue"), True)
|
||||
|
||||
result: dict[str, object] = {
|
||||
"stop_services": [],
|
||||
"start_services": [],
|
||||
"database": {},
|
||||
"redis": {},
|
||||
"runtime": {},
|
||||
}
|
||||
errors: list[str] = []
|
||||
|
||||
for service_name in stop_services:
|
||||
service_result = _run_service_command(runner, service_name, "stop")
|
||||
result["stop_services"].append(service_result)
|
||||
if not service_result["ok"]:
|
||||
errors.append(f"stop {service_name} failed")
|
||||
|
||||
try:
|
||||
if clear_database:
|
||||
result["database"] = _truncate_detect_runtime_tables(include_domains=clear_domains)
|
||||
if clear_redis:
|
||||
result["redis"] = _flush_runtime_redis()
|
||||
result["runtime"] = _reset_runtime_files(
|
||||
include_api_runtime=clear_api_runtime,
|
||||
include_worker_runtime=clear_worker_runtime,
|
||||
include_node_agent_queue=clear_node_agent_queue,
|
||||
)
|
||||
except Exception as exc:
|
||||
errors.append(str(exc))
|
||||
|
||||
for service_name in start_services:
|
||||
service_result = _run_service_command(runner, service_name, "start")
|
||||
result["start_services"].append(service_result)
|
||||
if not service_result["ok"]:
|
||||
errors.append(f"start {service_name} failed")
|
||||
|
||||
ok = not errors
|
||||
message = "lab runtime reset completed" if ok else "lab runtime reset finished with errors"
|
||||
if errors:
|
||||
result["errors"] = errors
|
||||
return ok, message, result
|
||||
|
||||
|
||||
def execute_structured_action(
|
||||
action: str,
|
||||
payload: dict | None,
|
||||
@@ -291,4 +525,12 @@ def execute_structured_action(
|
||||
}
|
||||
return True, "diagnostics collected", diagnostics
|
||||
|
||||
if normalized_action == "runtime.reset_lab_state":
|
||||
ok, message, result = _reset_lab_state(
|
||||
normalized_payload,
|
||||
runner=runner,
|
||||
)
|
||||
result.update(host_details)
|
||||
return ok, message, result
|
||||
|
||||
return False, f"unsupported action: {normalized_action}", {"action": normalized_action}
|
||||
|
||||
@@ -12,6 +12,9 @@ from app.core.db import get_db
|
||||
from app.services.ops_command_service import build_bash_command
|
||||
from app.services.ops_job_service import create_ops_job, ensure_ops_schema, get_ops_job
|
||||
from app.services.ops_template_service import build_ops_template_payload, get_ops_action_template
|
||||
from app.services.runtime_settings_service import get_runtime_settings
|
||||
from app.services.sensitive_words_service import get_sensitive_words_payload
|
||||
from app.services.settings_service import get_settings_payload
|
||||
|
||||
|
||||
_AGENT_SCHEMA_SQL = """
|
||||
@@ -117,6 +120,48 @@ def _format_time(value: object) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _prefer_non_loopback_identity(primary: object, fallback: object) -> str:
|
||||
primary_value = str(primary or "").strip()
|
||||
fallback_value = str(fallback or "").strip()
|
||||
invalid_values = {"", "localhost", "localhost.localdomain", "127.0.0.1", "::1"}
|
||||
if primary_value and primary_value.lower() not in invalid_values:
|
||||
return primary_value
|
||||
return fallback_value
|
||||
|
||||
|
||||
def _merge_detect_runtime_snapshot(cluster_metadata: dict, metadata: dict, current_load: int) -> dict:
|
||||
active_threads = int(cluster_metadata.get("active_threads", metadata.get("active_threads", 0)) or 0)
|
||||
max_threads = int(cluster_metadata.get("max_threads", metadata.get("max_threads", 0)) or 0)
|
||||
inferred_worker_online = bool(
|
||||
cluster_metadata.get("worker_online", metadata.get("worker_online", False))
|
||||
or cluster_metadata.get("service_running", metadata.get("service_running", False))
|
||||
or current_load > 0
|
||||
or active_threads > 0
|
||||
)
|
||||
inferred_detect_participating = bool(
|
||||
cluster_metadata.get("detect_participating", metadata.get("detect_participating", False))
|
||||
or current_load > 0
|
||||
or active_threads > 0
|
||||
)
|
||||
return {
|
||||
"worker_online": inferred_worker_online,
|
||||
"detect_participating": inferred_detect_participating,
|
||||
"active_threads": active_threads,
|
||||
"max_threads": max_threads,
|
||||
"current_load": current_load,
|
||||
"phase_label": str(cluster_metadata.get("phase_label", metadata.get("phase_label", "")) or "").strip(),
|
||||
"phase_detail": str(cluster_metadata.get("phase_detail", metadata.get("phase_detail", "")) or "").strip(),
|
||||
"recent_warning": str(cluster_metadata.get("recent_warning", metadata.get("recent_warning", "")) or "").strip(),
|
||||
"proxy_runtime_label": str(
|
||||
cluster_metadata.get("proxy_runtime_label", metadata.get("proxy_runtime_label", "")) or ""
|
||||
).strip(),
|
||||
"proxy_runtime_reason": str(
|
||||
cluster_metadata.get("proxy_runtime_reason", metadata.get("proxy_runtime_reason", "")) or ""
|
||||
).strip(),
|
||||
"updated_at": str(cluster_metadata.get("updated_at", metadata.get("updated_at", "")) or "").strip(),
|
||||
}
|
||||
|
||||
|
||||
def _ops_job_event_level_label(level: object) -> str:
|
||||
normalized_level = str(level or "").strip().lower()
|
||||
mapping = {
|
||||
@@ -588,6 +633,7 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None =
|
||||
seen_node_codes.add(node_code)
|
||||
metadata = dict(node.get("metadata") or {})
|
||||
cluster_node = cluster_map.get(node_code, {})
|
||||
cluster_metadata = dict(cluster_node.get("metadata") or {})
|
||||
latest_token = latest_tokens.get(node_code, {})
|
||||
latest_job = latest_jobs.get(node_code, {})
|
||||
participation_row = dict(detect_participation_map.get(node_code) or {})
|
||||
@@ -607,6 +653,15 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None =
|
||||
current_load=current_load,
|
||||
)
|
||||
capabilities = list(metadata.get("capabilities") or [])
|
||||
detect_runtime = _merge_detect_runtime_snapshot(cluster_metadata, metadata, current_load)
|
||||
active_job_snapshot = {
|
||||
"job_code": str(cluster_metadata.get("active_job_code", metadata.get("active_job_code", "")) or "").strip(),
|
||||
"status": str(cluster_metadata.get("active_job_status", metadata.get("active_job_status", "")) or "").strip(),
|
||||
"items_total": int(cluster_metadata.get("job_items_total", metadata.get("job_items_total", 0)) or 0),
|
||||
"items_claimed": int(cluster_metadata.get("job_items_claimed", metadata.get("job_items_claimed", 0)) or 0),
|
||||
"items_running": int(cluster_metadata.get("job_items_running", metadata.get("job_items_running", 0)) or 0),
|
||||
"items_completed": int(cluster_metadata.get("job_items_completed", metadata.get("job_items_completed", 0)) or 0),
|
||||
}
|
||||
merged_node = {
|
||||
**node,
|
||||
"is_managed": True,
|
||||
@@ -620,10 +675,14 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None =
|
||||
"agent_version": str(metadata.get("agent_version") or "").strip(),
|
||||
"agent_hostname": str(metadata.get("hostname") or "").strip(),
|
||||
"agent_ip": str(metadata.get("ip") or "").strip(),
|
||||
"cluster_hostname": str(cluster_node.get("hostname") or "").strip(),
|
||||
"cluster_ip": str(cluster_node.get("ip") or "").strip(),
|
||||
"cluster_hostname": _prefer_non_loopback_identity(cluster_node.get("hostname"), metadata.get("hostname")),
|
||||
"cluster_ip": _prefer_non_loopback_identity(cluster_node.get("ip"), metadata.get("ip")),
|
||||
"cluster_status": cluster_status,
|
||||
"cluster_current_load": current_load,
|
||||
"current_load": current_load,
|
||||
"detect_runtime": detect_runtime,
|
||||
"runtime_state": detect_runtime,
|
||||
"active_job": active_job_snapshot,
|
||||
"cluster_last_heartbeat_at": str(cluster_node.get("last_heartbeat_at") or metadata.get("last_heartbeat_at") or "").strip(),
|
||||
"cluster_is_effective_worker": bool(cluster_node.get("is_effective_worker", metadata.get("is_effective_worker", False))),
|
||||
"cluster_detect_participating": bool(
|
||||
@@ -688,6 +747,7 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None =
|
||||
if not node_code or node_code in seen_node_codes:
|
||||
continue
|
||||
metadata = dict(cluster_node.get("metadata") or {})
|
||||
cluster_metadata = dict(cluster_node.get("metadata") or {})
|
||||
latest_token = latest_tokens.get(node_code, {})
|
||||
latest_job = latest_jobs.get(node_code, {})
|
||||
participation_row = dict(detect_participation_map.get(node_code) or {})
|
||||
@@ -706,6 +766,15 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None =
|
||||
current_load=current_load,
|
||||
)
|
||||
capabilities = list(metadata.get("capabilities") or [])
|
||||
detect_runtime = _merge_detect_runtime_snapshot(cluster_metadata, metadata, current_load)
|
||||
active_job_snapshot = {
|
||||
"job_code": str(cluster_metadata.get("active_job_code", metadata.get("active_job_code", "")) or "").strip(),
|
||||
"status": str(cluster_metadata.get("active_job_status", metadata.get("active_job_status", "")) or "").strip(),
|
||||
"items_total": int(cluster_metadata.get("job_items_total", metadata.get("job_items_total", 0)) or 0),
|
||||
"items_claimed": int(cluster_metadata.get("job_items_claimed", metadata.get("job_items_claimed", 0)) or 0),
|
||||
"items_running": int(cluster_metadata.get("job_items_running", metadata.get("job_items_running", 0)) or 0),
|
||||
"items_completed": int(cluster_metadata.get("job_items_completed", metadata.get("job_items_completed", 0)) or 0),
|
||||
}
|
||||
fallback_node = {
|
||||
"node_code": node_code,
|
||||
"region": str(cluster_node.get("region") or "").strip(),
|
||||
@@ -732,12 +801,16 @@ def list_managed_nodes_with_agent_state(*, participation_payload: dict | None =
|
||||
"capabilities": capabilities,
|
||||
"capabilities_count": len(capabilities),
|
||||
"agent_version": str(metadata.get("agent_version") or "").strip(),
|
||||
"agent_hostname": str(cluster_node.get("hostname") or metadata.get("hostname") or "").strip(),
|
||||
"agent_ip": str(cluster_node.get("ip") or metadata.get("ip") or "").strip(),
|
||||
"cluster_hostname": str(cluster_node.get("hostname") or "").strip(),
|
||||
"cluster_ip": str(cluster_node.get("ip") or "").strip(),
|
||||
"agent_hostname": _prefer_non_loopback_identity(cluster_node.get("hostname"), metadata.get("hostname")),
|
||||
"agent_ip": _prefer_non_loopback_identity(cluster_node.get("ip"), metadata.get("ip")),
|
||||
"cluster_hostname": _prefer_non_loopback_identity(cluster_node.get("hostname"), metadata.get("hostname")),
|
||||
"cluster_ip": _prefer_non_loopback_identity(cluster_node.get("ip"), metadata.get("ip")),
|
||||
"cluster_status": cluster_status,
|
||||
"cluster_current_load": current_load,
|
||||
"current_load": current_load,
|
||||
"detect_runtime": detect_runtime,
|
||||
"runtime_state": detect_runtime,
|
||||
"active_job": active_job_snapshot,
|
||||
"cluster_last_heartbeat_at": str(cluster_node.get("last_heartbeat_at") or "").strip(),
|
||||
"cluster_is_effective_worker": bool(cluster_node.get("is_effective_worker", False)),
|
||||
"cluster_detect_participating": bool(
|
||||
@@ -2359,7 +2432,7 @@ def build_node_agent_bootstrap_plan(
|
||||
f"SYNC_AGENT_SERVICE_NAME={settings.sync_agent_service_name}",
|
||||
"NODE_AGENT_SERVICE_NAME=domaincheck-node-agent",
|
||||
"",
|
||||
'OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","health.snapshot","logs.collect","diagnostics.collect","deploy.release"]',
|
||||
'OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.start_detection","runtime.stop_detection","runtime.pull_tasks","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","runtime.reset_lab_state","health.snapshot","logs.collect","diagnostics.collect","deploy.release"]',
|
||||
"OPS_AGENT_LABELS={}",
|
||||
]
|
||||
env_content = "\n".join(env_lines)
|
||||
@@ -2529,6 +2602,135 @@ def _upsert_agent_runtime(node_code: str, payload: dict) -> None:
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _load_existing_detect_node_runtime(node_code: str) -> dict:
|
||||
normalized_node_code = str(node_code or "").strip()
|
||||
if not normalized_node_code:
|
||||
return {}
|
||||
|
||||
try:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT status, current_load, metadata_json, last_heartbeat_at
|
||||
FROM detect_worker_nodes
|
||||
WHERE node_code = %s
|
||||
LIMIT 1
|
||||
""",
|
||||
(normalized_node_code,),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
if not row:
|
||||
return {}
|
||||
metadata = row[2] if isinstance(row[2], dict) else {}
|
||||
return {
|
||||
"status": str(row[0] or "").strip(),
|
||||
"current_load": int(row[1] or 0),
|
||||
"metadata": metadata,
|
||||
"last_heartbeat_at": row[3],
|
||||
}
|
||||
|
||||
|
||||
def _upsert_agent_detect_runtime(node_code: str, payload: dict) -> None:
|
||||
normalized_node_code = str(node_code or "").strip()
|
||||
if not normalized_node_code:
|
||||
return
|
||||
|
||||
metadata = payload.get("metadata") if isinstance(payload.get("metadata"), dict) else {}
|
||||
detect_runtime = metadata.get("detect_runtime") if isinstance(metadata.get("detect_runtime"), dict) else {}
|
||||
if not detect_runtime:
|
||||
return
|
||||
|
||||
try:
|
||||
from app.services.cluster_runtime_service import register_node_heartbeat
|
||||
|
||||
region = str(payload.get("region") or "unknown").strip() or "unknown"
|
||||
role = str(payload.get("role") or "worker").strip() or "worker"
|
||||
active_threads = max(0, int(detect_runtime.get("active_threads") or 0))
|
||||
max_threads = max(0, int(detect_runtime.get("max_threads") or 0))
|
||||
current_load = max(
|
||||
0,
|
||||
int(detect_runtime.get("current_load") or 0),
|
||||
active_threads,
|
||||
)
|
||||
worker_online = bool(detect_runtime.get("worker_online", False) or detect_runtime.get("service_running", False))
|
||||
detect_participating = bool(detect_runtime.get("detect_participating", False) or current_load > 0)
|
||||
phase_label = str(detect_runtime.get("phase_label") or "").strip()
|
||||
phase_detail = str(detect_runtime.get("phase_detail") or "").strip()
|
||||
recent_warning = str(detect_runtime.get("recent_warning") or "").strip()
|
||||
has_runtime_signal = any(
|
||||
[
|
||||
worker_online,
|
||||
detect_participating,
|
||||
current_load > 0,
|
||||
active_threads > 0,
|
||||
max_threads > 0,
|
||||
bool(phase_label),
|
||||
bool(phase_detail),
|
||||
bool(recent_warning),
|
||||
]
|
||||
)
|
||||
if not has_runtime_signal:
|
||||
return
|
||||
existing_runtime = _load_existing_detect_node_runtime(normalized_node_code)
|
||||
existing_metadata = existing_runtime.get("metadata") if isinstance(existing_runtime.get("metadata"), dict) else {}
|
||||
existing_active_threads = max(0, int(existing_metadata.get("active_threads", 0) or 0))
|
||||
existing_current_load = max(0, int(existing_runtime.get("current_load", 0) or 0), existing_active_threads)
|
||||
existing_status = str(existing_runtime.get("status") or "").strip()
|
||||
existing_last_heartbeat_at = existing_runtime.get("last_heartbeat_at")
|
||||
existing_is_recent = False
|
||||
if isinstance(existing_last_heartbeat_at, datetime):
|
||||
current_time = (
|
||||
datetime.now(existing_last_heartbeat_at.tzinfo)
|
||||
if existing_last_heartbeat_at.tzinfo
|
||||
else datetime.now()
|
||||
)
|
||||
existing_is_recent = (current_time - existing_last_heartbeat_at) <= timedelta(seconds=120)
|
||||
normalized_phase_detail = phase_detail.lower()
|
||||
generic_phase_detail = bool(normalized_phase_detail) and "/" in normalized_phase_detail and " " not in normalized_phase_detail
|
||||
weak_agent_snapshot = bool(
|
||||
worker_online
|
||||
and current_load <= 0
|
||||
and active_threads <= 0
|
||||
and max_threads > 0
|
||||
and not phase_label
|
||||
and (not phase_detail or generic_phase_detail)
|
||||
and not recent_warning
|
||||
)
|
||||
if weak_agent_snapshot and existing_is_recent and (
|
||||
existing_current_load > 0 or existing_status == "busy"
|
||||
):
|
||||
return
|
||||
status = "busy" if current_load > 0 else ("online" if worker_online else "unknown")
|
||||
heartbeat_metadata = {
|
||||
"service": "agent-heartbeat",
|
||||
"worker_online": worker_online,
|
||||
"detect_participating": detect_participating,
|
||||
"active_threads": active_threads,
|
||||
"max_threads": max_threads,
|
||||
"phase_label": phase_label,
|
||||
"phase_detail": phase_detail,
|
||||
"recent_warning": recent_warning,
|
||||
"updated_at": str(detect_runtime.get("updated_at") or "").strip(),
|
||||
"agent_heartbeat_at": datetime.now().isoformat(timespec="seconds"),
|
||||
}
|
||||
register_node_heartbeat(
|
||||
node_code=normalized_node_code,
|
||||
region=region,
|
||||
role=role,
|
||||
status=status,
|
||||
current_load=current_load,
|
||||
metadata=heartbeat_metadata,
|
||||
hostname_override=str(payload.get("hostname") or "").strip(),
|
||||
ip_override=str(payload.get("ip") or "").strip(),
|
||||
)
|
||||
except Exception:
|
||||
return
|
||||
|
||||
|
||||
def agent_register(payload: dict, *, token: str) -> tuple[bool, str, dict]:
|
||||
node_code = str(payload.get("node_code") or "").strip()
|
||||
if not node_code:
|
||||
@@ -2537,6 +2739,7 @@ def agent_register(payload: dict, *, token: str) -> tuple[bool, str, dict]:
|
||||
if not ok:
|
||||
return False, message, auth
|
||||
_upsert_agent_runtime(node_code, payload)
|
||||
_upsert_agent_detect_runtime(node_code, payload)
|
||||
return True, "Agent 注册成功", {
|
||||
"node_code": node_code,
|
||||
"expires_at": auth.get("expires_at", ""),
|
||||
@@ -2552,6 +2755,7 @@ def agent_heartbeat(payload: dict, *, token: str) -> tuple[bool, str, dict]:
|
||||
if not ok:
|
||||
return False, message, auth
|
||||
_upsert_agent_runtime(node_code, payload)
|
||||
_upsert_agent_detect_runtime(node_code, payload)
|
||||
return True, "heartbeat ok", {
|
||||
"node_code": node_code,
|
||||
"server_time": _format_time(datetime.now()),
|
||||
@@ -2559,6 +2763,57 @@ def agent_heartbeat(payload: dict, *, token: str) -> tuple[bool, str, dict]:
|
||||
}
|
||||
|
||||
|
||||
def _runtime_config_bundle_hash_payload(bundle: dict | None) -> dict:
|
||||
normalized = dict(bundle or {})
|
||||
normalized.pop("config_hash", None)
|
||||
normalized.pop("generated_at", None)
|
||||
return normalized
|
||||
|
||||
|
||||
def _build_agent_runtime_config_bundle(node_code: str) -> dict:
|
||||
settings_payload = get_settings_payload()
|
||||
runtime_settings = get_runtime_settings()
|
||||
sensitive_words_payload = get_sensitive_words_payload()
|
||||
bundle = {
|
||||
"node_code": str(node_code or "").strip(),
|
||||
"detect_options": dict(settings_payload.get("detect_options") or {}),
|
||||
"proxy_config": dict(settings_payload.get("proxy_config") or {}),
|
||||
"thread_count": int(settings_payload.get("thread_count", 2) or 2),
|
||||
"node_thread_counts": dict(settings_payload.get("node_thread_counts") or {}),
|
||||
"runtime_settings": dict(runtime_settings or {}),
|
||||
"sensitive_words": {
|
||||
"text": str(sensitive_words_payload.get("text") or ""),
|
||||
"total": int(sensitive_words_payload.get("total", 0) or 0),
|
||||
"items": list(sensitive_words_payload.get("items") or []),
|
||||
},
|
||||
"generated_at": _format_time(datetime.now()),
|
||||
}
|
||||
bundle["config_hash"] = hashlib.sha256(
|
||||
json.dumps(
|
||||
_runtime_config_bundle_hash_payload(bundle),
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
).encode("utf-8")
|
||||
).hexdigest()
|
||||
return bundle
|
||||
|
||||
|
||||
def agent_pull_runtime_config(payload: dict, *, token: str) -> tuple[bool, str, dict]:
|
||||
node_code = str(payload.get("node_code") or "").strip()
|
||||
if not node_code:
|
||||
return _agent_error("node_code 不能为空", "agent_node_code_required")
|
||||
ok, message, auth = _authenticate_agent_token(token, expected_node_code=node_code)
|
||||
if not ok:
|
||||
return False, message, auth
|
||||
bundle = _build_agent_runtime_config_bundle(node_code)
|
||||
return True, "runtime config ok", {
|
||||
"node_code": node_code,
|
||||
"server_time": _format_time(datetime.now()),
|
||||
"expires_at": auth.get("expires_at", ""),
|
||||
"bundle": bundle,
|
||||
}
|
||||
|
||||
|
||||
def agent_pull_jobs(payload: dict, *, token: str, limit: int = 1) -> tuple[bool, str, dict]:
|
||||
node_code = str(payload.get("node_code") or "").strip()
|
||||
if not node_code:
|
||||
@@ -2568,6 +2823,7 @@ def agent_pull_jobs(payload: dict, *, token: str, limit: int = 1) -> tuple[bool,
|
||||
return False, message, _auth
|
||||
|
||||
safe_limit = min(max(int(limit or 1), 1), 10)
|
||||
dispatched_events: list[dict] = []
|
||||
with get_db() as conn:
|
||||
conn.autocommit = False
|
||||
with conn.cursor() as cur:
|
||||
@@ -2604,15 +2860,19 @@ def agent_pull_jobs(payload: dict, *, token: str, limit: int = 1) -> tuple[bool,
|
||||
""",
|
||||
(job_id,),
|
||||
)
|
||||
append_ops_job_event(
|
||||
job_id=job_id,
|
||||
node_code=node_code,
|
||||
event_type="agent_dispatched",
|
||||
message=f"任务已派发给节点 {node_code}",
|
||||
payload={"node_code": node_code},
|
||||
dispatched_events.append(
|
||||
{
|
||||
"job_id": job_id,
|
||||
"node_code": node_code,
|
||||
"event_type": "agent_dispatched",
|
||||
"message": f"任务已派发给节点 {node_code}",
|
||||
"payload": {"node_code": node_code},
|
||||
}
|
||||
)
|
||||
jobs.append(_agent_job_envelope(get_ops_job(job_id)))
|
||||
conn.commit()
|
||||
for event in dispatched_events:
|
||||
append_ops_job_event(**event)
|
||||
return True, "ok", {
|
||||
"jobs": jobs,
|
||||
"count": len(jobs),
|
||||
@@ -2631,6 +2891,7 @@ def agent_mark_job_started(job_id: int, payload: dict, *, token: str) -> tuple[b
|
||||
if not ok:
|
||||
return False, message, _auth
|
||||
|
||||
started_event: dict | None = None
|
||||
with get_db() as conn:
|
||||
conn.autocommit = False
|
||||
with conn.cursor() as cur:
|
||||
@@ -2662,14 +2923,16 @@ def agent_mark_job_started(job_id: int, payload: dict, *, token: str) -> tuple[b
|
||||
)
|
||||
step_rows = cur.fetchall()
|
||||
step_ids = [int(item[0]) for item in step_rows]
|
||||
append_ops_job_event(
|
||||
job_id=int(job_id),
|
||||
node_code=node_code,
|
||||
event_type="agent_started",
|
||||
message=f"节点 {node_code} 已开始执行任务",
|
||||
payload={"step_ids": step_ids},
|
||||
)
|
||||
started_event = {
|
||||
"job_id": int(job_id),
|
||||
"node_code": node_code,
|
||||
"event_type": "agent_started",
|
||||
"message": f"节点 {node_code} 已开始执行任务",
|
||||
"payload": {"step_ids": step_ids},
|
||||
}
|
||||
conn.commit()
|
||||
if started_event:
|
||||
append_ops_job_event(**started_event)
|
||||
job = get_ops_job(int(job_id))
|
||||
return True, "任务已标记为运行中", {
|
||||
"job": job,
|
||||
@@ -2710,6 +2973,7 @@ def agent_complete_job(job_id: int, payload: dict, *, token: str) -> tuple[bool,
|
||||
result["focus_ref"] = focus_ref
|
||||
event_level = "info" if job_status == "success" else ("warning" if job_status == "partially_succeeded" else "error")
|
||||
|
||||
completed_event: dict | None = None
|
||||
with get_db() as conn:
|
||||
conn.autocommit = False
|
||||
with conn.cursor() as cur:
|
||||
@@ -2789,22 +3053,24 @@ def agent_complete_job(job_id: int, payload: dict, *, token: str) -> tuple[bool,
|
||||
)
|
||||
step_rows = cur.fetchall()
|
||||
step_ids = [int(item[0]) for item in step_rows]
|
||||
append_ops_job_event(
|
||||
job_id=int(job_id),
|
||||
node_code=node_code,
|
||||
client_event_id=(f"complete:{client_request_id}" if client_request_id else ""),
|
||||
event_type="agent_completed",
|
||||
message=f"节点 {node_code} 已完成任务，状态: {job_status}",
|
||||
level=event_level,
|
||||
payload={
|
||||
completed_event = {
|
||||
"job_id": int(job_id),
|
||||
"node_code": node_code,
|
||||
"client_event_id": (f"complete:{client_request_id}" if client_request_id else ""),
|
||||
"event_type": "agent_completed",
|
||||
"message": f"节点 {node_code} 已完成任务，状态: {job_status}",
|
||||
"level": event_level,
|
||||
"payload": {
|
||||
"step_ids": step_ids,
|
||||
"result": result,
|
||||
"duration_ms": duration_ms,
|
||||
"summary_text": str(result.get("summary_text") or result.get("summary") or summary_text).strip(),
|
||||
"focus_ref": focus_ref,
|
||||
},
|
||||
)
|
||||
}
|
||||
conn.commit()
|
||||
if completed_event:
|
||||
append_ops_job_event(**completed_event)
|
||||
from app.services.ops_release_service import refresh_release_rollout_for_job
|
||||
|
||||
refresh_release_rollout_for_job(int(job_id))
|
||||
|
||||
@@ -39,6 +39,13 @@ CREATE TABLE IF NOT EXISTS ops_managed_nodes (
|
||||
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_managed_node_secrets (
|
||||
node_code VARCHAR(64) PRIMARY KEY REFERENCES ops_managed_nodes(node_code) ON DELETE CASCADE,
|
||||
ssh_password TEXT NOT NULL DEFAULT '',
|
||||
ssh_private_key TEXT NOT NULL DEFAULT '',
|
||||
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS ops_jobs (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
job_code VARCHAR(64) NOT NULL UNIQUE,
|
||||
@@ -321,6 +328,107 @@ def _serialize_node_row(row: tuple) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _load_node_secret_flags(node_codes: list[str]) -> dict[str, dict]:
|
||||
normalized_codes = [str(item or "").strip() for item in node_codes if str(item or "").strip()]
|
||||
if not normalized_codes:
|
||||
return {}
|
||||
result: dict[str, dict] = {}
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT node_code, ssh_password, ssh_private_key
|
||||
FROM ops_managed_node_secrets
|
||||
WHERE node_code = ANY(%s)
|
||||
""",
|
||||
(normalized_codes,),
|
||||
)
|
||||
rows = cur.fetchall()
|
||||
for row in rows:
|
||||
node_code = str(row[0] or "").strip()
|
||||
result[node_code] = {
|
||||
"ssh_password_configured": bool(str(row[1] or "").strip()),
|
||||
"ssh_private_key_configured": bool(str(row[2] or "").strip()),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def _parse_ssh_entry(raw_value: object) -> dict:
|
||||
raw = str(raw_value or "").strip()
|
||||
if not raw:
|
||||
return {}
|
||||
parts = raw.split(maxsplit=2)
|
||||
if len(parts) < 2:
|
||||
return {}
|
||||
host_port = str(parts[0] or "").strip()
|
||||
ssh_user = str(parts[1] or "").strip()
|
||||
secret = str(parts[2] or "").strip() if len(parts) >= 3 else ""
|
||||
ssh_host = host_port
|
||||
ssh_port = 22
|
||||
if ":" in host_port:
|
||||
host_candidate, port_candidate = host_port.rsplit(":", 1)
|
||||
if host_candidate and port_candidate.isdigit():
|
||||
ssh_host = host_candidate
|
||||
ssh_port = max(int(port_candidate), 1)
|
||||
if secret.startswith("<") and secret.endswith(">") and len(secret) >= 2:
|
||||
secret = secret[1:-1].strip()
|
||||
payload = {
|
||||
"ssh_host": ssh_host,
|
||||
"ssh_port": ssh_port,
|
||||
"ssh_user": ssh_user,
|
||||
}
|
||||
if secret:
|
||||
payload["auth_mode"] = "password"
|
||||
payload["ssh_password"] = secret
|
||||
return payload
|
||||
|
||||
|
||||
def _upsert_managed_node_secret(
|
||||
*,
|
||||
node_code: str,
|
||||
ssh_password: str = "",
|
||||
ssh_private_key: str = "",
|
||||
clear_ssh_password: bool = False,
|
||||
clear_ssh_private_key: bool = False,
|
||||
) -> None:
|
||||
normalized_node_code = str(node_code or "").strip()
|
||||
if not normalized_node_code:
|
||||
return
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
INSERT INTO ops_managed_node_secrets (
|
||||
node_code, ssh_password, ssh_private_key, updated_at
|
||||
) VALUES (%s, %s, %s, CURRENT_TIMESTAMP)
|
||||
ON CONFLICT (node_code) DO UPDATE SET
|
||||
ssh_password = CASE
|
||||
WHEN %s THEN ''
|
||||
WHEN %s <> '' THEN %s
|
||||
ELSE ops_managed_node_secrets.ssh_password
|
||||
END,
|
||||
ssh_private_key = CASE
|
||||
WHEN %s THEN ''
|
||||
WHEN %s <> '' THEN %s
|
||||
ELSE ops_managed_node_secrets.ssh_private_key
|
||||
END,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
""",
|
||||
(
|
||||
normalized_node_code,
|
||||
"" if clear_ssh_password else ssh_password,
|
||||
"" if clear_ssh_private_key else ssh_private_key,
|
||||
clear_ssh_password,
|
||||
ssh_password,
|
||||
ssh_password,
|
||||
clear_ssh_private_key,
|
||||
ssh_private_key,
|
||||
ssh_private_key,
|
||||
),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _pick_text_value(payload: dict, key: str, fallback: str = "", *, default: str = "") -> str:
|
||||
if key in payload:
|
||||
normalized = str(payload.get(key) or "").strip()
|
||||
@@ -397,6 +505,15 @@ def upsert_managed_node(payload: dict) -> tuple[bool, str, dict]:
|
||||
node_code = str(payload.get("node_code") or "").strip()
|
||||
if not node_code:
|
||||
return False, "node_code 不能为空", {}
|
||||
parsed_ssh_entry = _parse_ssh_entry(payload.get("ssh_entry"))
|
||||
merged_payload = {
|
||||
**dict(payload or {}),
|
||||
**{key: value for key, value in parsed_ssh_entry.items() if value not in ("", None)},
|
||||
}
|
||||
ssh_password = str(merged_payload.get("ssh_password") or "").strip()
|
||||
ssh_private_key = str(merged_payload.get("ssh_private_key") or "")
|
||||
clear_ssh_password = bool(merged_payload.get("clear_ssh_password", False))
|
||||
clear_ssh_private_key = bool(merged_payload.get("clear_ssh_private_key", False))
|
||||
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
@@ -413,22 +530,22 @@ def upsert_managed_node(payload: dict) -> tuple[bool, str, dict]:
|
||||
existing_row = cur.fetchone()
|
||||
existing_node = _serialize_node_row(existing_row) if existing_row else {}
|
||||
existing_metadata = dict(existing_node.get("metadata") or {})
|
||||
incoming_metadata = dict(payload.get("metadata") or {})
|
||||
incoming_metadata = dict(merged_payload.get("metadata") or {})
|
||||
|
||||
region = _pick_text_value(payload, "region", str(existing_node.get("region") or ""), default="unknown") or "unknown"
|
||||
role = _pick_text_value(payload, "role", str(existing_node.get("role") or ""), default="worker") or "worker"
|
||||
title = _pick_text_value(payload, "title", str(existing_node.get("title") or ""), default=node_code) or node_code
|
||||
ssh_host = _pick_text_value(payload, "ssh_host", str(existing_node.get("ssh_host") or ""))
|
||||
ssh_port = _pick_int_value(payload, "ssh_port", int(existing_node.get("ssh_port") or 22), default=22, minimum=1)
|
||||
ssh_user = _pick_text_value(payload, "ssh_user", str(existing_node.get("ssh_user") or ""))
|
||||
auth_mode = _pick_text_value(payload, "auth_mode", str(existing_node.get("auth_mode") or ""), default="key") or "key"
|
||||
region = _pick_text_value(merged_payload, "region", str(existing_node.get("region") or ""), default="unknown") or "unknown"
|
||||
role = _pick_text_value(merged_payload, "role", str(existing_node.get("role") or ""), default="worker") or "worker"
|
||||
title = _pick_text_value(merged_payload, "title", str(existing_node.get("title") or ""), default=node_code) or node_code
|
||||
ssh_host = _pick_text_value(merged_payload, "ssh_host", str(existing_node.get("ssh_host") or ""))
|
||||
ssh_port = _pick_int_value(merged_payload, "ssh_port", int(existing_node.get("ssh_port") or 22), default=22, minimum=1)
|
||||
ssh_user = _pick_text_value(merged_payload, "ssh_user", str(existing_node.get("ssh_user") or ""))
|
||||
auth_mode = _pick_text_value(merged_payload, "auth_mode", str(existing_node.get("auth_mode") or ""), default="key") or "key"
|
||||
deploy_channel = _pick_text_value(
|
||||
payload,
|
||||
merged_payload,
|
||||
"deploy_channel",
|
||||
str(existing_node.get("deploy_channel") or ""),
|
||||
default="stable",
|
||||
) or "stable"
|
||||
is_enabled = bool(payload["is_enabled"]) if "is_enabled" in payload else bool(existing_node.get("is_enabled", True))
|
||||
is_enabled = bool(merged_payload["is_enabled"]) if "is_enabled" in merged_payload else bool(existing_node.get("is_enabled", True))
|
||||
metadata = {
|
||||
**existing_metadata,
|
||||
**incoming_metadata,
|
||||
@@ -469,7 +586,16 @@ def upsert_managed_node(payload: dict) -> tuple[bool, str, dict]:
|
||||
)
|
||||
row = cur.fetchone()
|
||||
conn.commit()
|
||||
return True, "托管节点已保存", {"node": _serialize_node_row(row)}
|
||||
_upsert_managed_node_secret(
|
||||
node_code=node_code,
|
||||
ssh_password=ssh_password,
|
||||
ssh_private_key=ssh_private_key,
|
||||
clear_ssh_password=clear_ssh_password,
|
||||
clear_ssh_private_key=clear_ssh_private_key,
|
||||
)
|
||||
node = _serialize_node_row(row)
|
||||
node.update(_load_node_secret_flags([node_code]).get(node_code, {}))
|
||||
return True, "托管节点已保存", {"node": node}
|
||||
|
||||
|
||||
def list_managed_nodes() -> list[dict]:
|
||||
@@ -485,7 +611,11 @@ def list_managed_nodes() -> list[dict]:
|
||||
"""
|
||||
)
|
||||
rows = cur.fetchall()
|
||||
return [_serialize_node_row(row) for row in rows]
|
||||
items = [_serialize_node_row(row) for row in rows]
|
||||
secret_flags = _load_node_secret_flags([str(item.get("node_code") or "") for item in items])
|
||||
for item in items:
|
||||
item.update(secret_flags.get(str(item.get("node_code") or "").strip(), {}))
|
||||
return items
|
||||
|
||||
|
||||
def sync_managed_nodes_from_cluster(*, dry_run: bool = False) -> dict:
|
||||
|
||||
@@ -42,6 +42,7 @@ _CRITICAL_RISK_ACTIONS = {
|
||||
"deploy.rollback",
|
||||
"node.bootstrap",
|
||||
"cluster.reconfigure",
|
||||
"runtime.reset_lab_state",
|
||||
}
|
||||
|
||||
|
||||
@@ -166,11 +167,23 @@ def _preview_single_node_policy(
|
||||
node_detect_participating = bool(target_node.get("detect_participating", False))
|
||||
|
||||
if target_node:
|
||||
if node_status in {"busy"} and action in {"runtime.stop_worker", "runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
|
||||
blocking_reasons.append("目标节点当前处于 busy 状态，不适合直接执行中断类动作。")
|
||||
interrupt_actions = {"runtime.stop_worker", "runtime.restart_api", "service.restart"}
|
||||
restart_like_actions = {"service.restart"}
|
||||
rolling_deploy_actions = {"deploy.release", "deploy.rollback"}
|
||||
|
||||
if node_detect_participating and action in {"runtime.stop_worker", "runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
|
||||
if node_status in {"busy"} and action in interrupt_actions - restart_like_actions:
|
||||
blocking_reasons.append("目标节点当前处于 busy 状态，不适合直接执行中断类动作。")
|
||||
elif node_status in {"busy"} and action in restart_like_actions:
|
||||
warnings.append("目标节点当前处于 busy 状态，重启会带来瞬时抖动，请确认当前窗口可接受。")
|
||||
elif node_status in {"busy"} and action in rolling_deploy_actions:
|
||||
warnings.append("目标节点当前处于 busy 状态，滚动发布会触发服务重启，请确认当前窗口可接受短暂抖动。")
|
||||
|
||||
if node_detect_participating and action in interrupt_actions - restart_like_actions:
|
||||
blocking_reasons.append("目标节点正在参与检测，需先迁移负载或人工确认后再执行。")
|
||||
elif node_detect_participating and action in restart_like_actions:
|
||||
warnings.append("目标节点正在参与检测，重启会中断当前任务，请确认剩余节点仍可承接负载。")
|
||||
elif node_detect_participating and action in rolling_deploy_actions:
|
||||
warnings.append("目标节点正在参与检测，建议优先采用单节点滚动发布，并确认其余节点仍可承接负载。")
|
||||
|
||||
if node_role == "control" and action in {"runtime.restart_api", "deploy.release", "deploy.rollback", "service.restart"}:
|
||||
approval_reasons.append("目标节点是 control 节点，建议强制走审批或维护窗口。")
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import grp
|
||||
import hashlib
|
||||
import inspect
|
||||
import json
|
||||
import os
|
||||
import pwd
|
||||
import shutil
|
||||
import tarfile
|
||||
import textwrap
|
||||
@@ -13,6 +16,22 @@ from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
_SYSTEMD_TEMPLATE_SPECS = {
|
||||
"domaincheck-api": {
|
||||
"template": Path("domain-api/deploy/systemd/domain-api.service"),
|
||||
},
|
||||
"domaincheck-worker": {
|
||||
"template": Path("domain-api/deploy/systemd/domain-worker.service"),
|
||||
},
|
||||
"domaincheck-sync-agent": {
|
||||
"template": Path("domain-api/deploy/systemd/domain-sync-agent.service"),
|
||||
},
|
||||
"domaincheck-node-agent": {
|
||||
"template": Path("domain-api/deploy/systemd/domain-node-agent.service"),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def normalize_text_list(raw_value: object) -> list[str]:
|
||||
if isinstance(raw_value, list):
|
||||
return [str(item).strip() for item in raw_value if str(item).strip()]
|
||||
@@ -147,6 +166,219 @@ def run_release_health_checks(
|
||||
}
|
||||
|
||||
|
||||
def ensure_directory_ready(path: Path) -> tuple[bool, dict]:
|
||||
normalized_path = Path(path).resolve()
|
||||
try:
|
||||
normalized_path.mkdir(parents=True, exist_ok=True)
|
||||
except Exception as exc:
|
||||
return False, {
|
||||
"path": str(normalized_path),
|
||||
"error": str(exc),
|
||||
"exception_type": exc.__class__.__name__,
|
||||
}
|
||||
return True, {"path": str(normalized_path)}
|
||||
|
||||
|
||||
def _resolve_path_owner_group(path: Path) -> tuple[str, str]:
|
||||
normalized_path = Path(path).resolve()
|
||||
stat_info = normalized_path.stat()
|
||||
owner_user = ""
|
||||
owner_group = ""
|
||||
try:
|
||||
owner_user = pwd.getpwuid(stat_info.st_uid).pw_name
|
||||
except Exception:
|
||||
owner_user = ""
|
||||
try:
|
||||
owner_group = grp.getgrgid(stat_info.st_gid).gr_name
|
||||
except Exception:
|
||||
owner_group = ""
|
||||
return owner_user, owner_group
|
||||
|
||||
|
||||
def collect_service_identity(run_command, service_name: str) -> dict:
|
||||
code, stdout, stderr = run_command(
|
||||
["systemctl", "show", service_name, "-p", "User", "-p", "Group", "--value"],
|
||||
timeout=15,
|
||||
)
|
||||
lines = [line.strip() for line in (stdout or stderr or "").splitlines()]
|
||||
user = lines[0] if len(lines) >= 1 else ""
|
||||
group = lines[1] if len(lines) >= 2 else ""
|
||||
return {
|
||||
"service_name": service_name,
|
||||
"returncode": int(code or 0),
|
||||
"user": user,
|
||||
"group": group,
|
||||
"ok": int(code or 0) == 0,
|
||||
}
|
||||
|
||||
|
||||
def _pick_release_owner_group(
|
||||
releases_dir: Path,
|
||||
service_identities: list[dict],
|
||||
) -> tuple[str, str]:
|
||||
for item in service_identities:
|
||||
user = str(item.get("user") or "").strip()
|
||||
group = str(item.get("group") or "").strip()
|
||||
if user or group:
|
||||
return user, group
|
||||
owner_user, owner_group = _resolve_path_owner_group(releases_dir)
|
||||
if owner_user or owner_group:
|
||||
return owner_user, owner_group
|
||||
return "", ""
|
||||
|
||||
|
||||
def apply_release_permissions(
|
||||
run_command,
|
||||
*,
|
||||
release_dir: Path,
|
||||
owner_user: str,
|
||||
owner_group: str,
|
||||
) -> dict:
|
||||
normalized_release_dir = Path(release_dir).resolve()
|
||||
normalized_user = str(owner_user or "").strip()
|
||||
normalized_group = str(owner_group or "").strip()
|
||||
if not normalized_user and not normalized_group:
|
||||
return {
|
||||
"attempted": False,
|
||||
"release_dir": str(normalized_release_dir),
|
||||
"owner_user": normalized_user,
|
||||
"owner_group": normalized_group,
|
||||
"returncode": 0,
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
"ok": True,
|
||||
}
|
||||
owner_spec = f"{normalized_user}:{normalized_group}" if normalized_group else normalized_user
|
||||
code, stdout, stderr = run_command(
|
||||
["chown", "-R", owner_spec, str(normalized_release_dir)],
|
||||
timeout=180,
|
||||
)
|
||||
return {
|
||||
"attempted": True,
|
||||
"release_dir": str(normalized_release_dir),
|
||||
"owner_user": normalized_user,
|
||||
"owner_group": normalized_group,
|
||||
"owner_spec": owner_spec,
|
||||
"returncode": int(code or 0),
|
||||
"stdout": stdout,
|
||||
"stderr": stderr,
|
||||
"ok": int(code or 0) == 0,
|
||||
}
|
||||
|
||||
|
||||
def collect_service_execstart(run_command, service_name: str) -> dict:
|
||||
code, stdout, stderr = run_command(
|
||||
["systemctl", "show", service_name, "-p", "ExecStart", "--value"],
|
||||
timeout=15,
|
||||
)
|
||||
execstart_value = stdout or stderr
|
||||
return {
|
||||
"service_name": service_name,
|
||||
"returncode": int(code or 0),
|
||||
"execstart": execstart_value,
|
||||
"ok": int(code or 0) == 0,
|
||||
}
|
||||
|
||||
|
||||
def _write_text_file(path: Path, content: str) -> None:
|
||||
normalized_path = Path(path).resolve()
|
||||
normalized_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
normalized_path.write_text(str(content or "").rstrip() + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _systemd_dropin_content(service_name: str, install_root: str) -> str:
|
||||
normalized_service_name = str(service_name or "").strip()
|
||||
normalized_install_root = str(install_root or "").rstrip("/")
|
||||
if normalized_service_name == "domaincheck-api":
|
||||
return "\n".join(
|
||||
[
|
||||
"[Service]",
|
||||
f"WorkingDirectory={normalized_install_root}/current/domain-api",
|
||||
"ExecStart=",
|
||||
f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100",
|
||||
]
|
||||
)
|
||||
if normalized_service_name == "domaincheck-worker":
|
||||
return "\n".join(
|
||||
[
|
||||
"[Service]",
|
||||
f"WorkingDirectory={normalized_install_root}/current/domainCheck",
|
||||
"ExecStart=",
|
||||
f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python {normalized_install_root}/current/domainCheck/detect_worker.py",
|
||||
]
|
||||
)
|
||||
if normalized_service_name == "domaincheck-sync-agent":
|
||||
return "\n".join(
|
||||
[
|
||||
"[Service]",
|
||||
f"WorkingDirectory={normalized_install_root}/current/domain-api",
|
||||
"ExecStart=",
|
||||
f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python -m app.sync_agent",
|
||||
]
|
||||
)
|
||||
if normalized_service_name == "domaincheck-node-agent":
|
||||
return "\n".join(
|
||||
[
|
||||
"[Service]",
|
||||
"User=root",
|
||||
"Group=root",
|
||||
f"WorkingDirectory={normalized_install_root}/current/domain-api",
|
||||
"ExecStart=",
|
||||
f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python -m app.node_agent",
|
||||
]
|
||||
)
|
||||
return ""
|
||||
|
||||
|
||||
def _sync_release_systemd_units(
|
||||
*,
|
||||
release_dir: Path,
|
||||
install_root: Path,
|
||||
systemd_unit_root: Path,
|
||||
switch_current: bool,
|
||||
) -> dict:
|
||||
normalized_release_dir = Path(release_dir).resolve()
|
||||
normalized_systemd_root = Path(systemd_unit_root).resolve()
|
||||
results: list[dict] = []
|
||||
synced_units: list[str] = []
|
||||
|
||||
for service_name, spec in _SYSTEMD_TEMPLATE_SPECS.items():
|
||||
template_path = normalized_release_dir / Path(spec["template"])
|
||||
if not template_path.exists():
|
||||
continue
|
||||
|
||||
target_unit_path = normalized_systemd_root / f"{service_name}.service"
|
||||
target_unit_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(template_path, target_unit_path)
|
||||
os.chmod(target_unit_path, 0o644)
|
||||
|
||||
dropin_path = normalized_systemd_root / f"{service_name}.service.d" / "current-path.conf"
|
||||
dropin_written = False
|
||||
if switch_current:
|
||||
dropin_content = _systemd_dropin_content(service_name, str(install_root))
|
||||
if dropin_content:
|
||||
_write_text_file(dropin_path, dropin_content)
|
||||
dropin_written = True
|
||||
|
||||
results.append(
|
||||
{
|
||||
"service_name": service_name,
|
||||
"template_path": str(template_path),
|
||||
"target_unit_path": str(target_unit_path),
|
||||
"dropin_path": str(dropin_path) if dropin_written else "",
|
||||
"dropin_written": dropin_written,
|
||||
}
|
||||
)
|
||||
synced_units.append(service_name)
|
||||
|
||||
return {
|
||||
"systemd_unit_root": str(normalized_systemd_root),
|
||||
"synced_units": synced_units,
|
||||
"results": results,
|
||||
"daemon_reload_required": bool(synced_units),
|
||||
}
|
||||
|
||||
|
||||
def safe_extract_tar(archive: tarfile.TarFile, target_dir: Path) -> None:
|
||||
target_dir_resolved = target_dir.resolve()
|
||||
members = archive.getmembers()
|
||||
@@ -189,6 +421,7 @@ def execute_release_action(
|
||||
health_check_retries = max(0, int(normalized_payload.get("health_check_retries") or 2))
|
||||
health_check_interval_seconds = max(0, int(normalized_payload.get("health_check_interval_seconds") or 2))
|
||||
rollback_on_failure = coerce_bool(normalized_payload.get("rollback_on_failure", True), default=True)
|
||||
systemd_unit_root = Path(str(normalized_payload.get("systemd_unit_root") or "/etc/systemd/system")).resolve()
|
||||
|
||||
if not release_version:
|
||||
return False, "release_version missing", {}
|
||||
@@ -202,8 +435,57 @@ def execute_release_action(
|
||||
artifact_path = downloads_dir / f"{release_version}.tar.gz"
|
||||
current_link = install_root / "current"
|
||||
previous_current_target = ""
|
||||
downloads_dir.mkdir(parents=True, exist_ok=True)
|
||||
releases_dir.mkdir(parents=True, exist_ok=True)
|
||||
runtime_dirs = [downloads_dir, releases_dir]
|
||||
prepared_dirs: list[dict] = []
|
||||
for directory in runtime_dirs:
|
||||
ok, preparation = ensure_directory_ready(directory)
|
||||
prepared_dirs.append(preparation)
|
||||
if not ok:
|
||||
event_callback(
|
||||
"deploy_preflight_failed",
|
||||
f"发布目录不可写: {preparation.get('path') or directory}",
|
||||
level="error",
|
||||
payload={
|
||||
"release_version": release_version,
|
||||
"install_root": str(install_root),
|
||||
"prepared_dirs": prepared_dirs,
|
||||
},
|
||||
)
|
||||
return False, f"install_root not writable: {preparation.get('path') or directory}", {
|
||||
"release_version": release_version,
|
||||
"install_root": str(install_root),
|
||||
"prepared_dirs": prepared_dirs,
|
||||
}
|
||||
|
||||
service_execstarts = [
|
||||
collect_service_execstart(run_command, service_name)
|
||||
for service_name in restart_services
|
||||
if str(service_name or "").strip()
|
||||
]
|
||||
service_identities = [
|
||||
collect_service_identity(run_command, service_name)
|
||||
for service_name in restart_services
|
||||
if str(service_name or "").strip()
|
||||
]
|
||||
current_link_text = str(current_link)
|
||||
execstart_alignment = {
|
||||
"switch_current": switch_current,
|
||||
"current_link": current_link_text,
|
||||
"services": service_execstarts,
|
||||
"mismatched_services": [
|
||||
item.get("service_name")
|
||||
for item in service_execstarts
|
||||
if str(item.get("execstart") or "").strip()
|
||||
and current_link_text not in str(item.get("execstart") or "")
|
||||
],
|
||||
}
|
||||
if switch_current and execstart_alignment["mismatched_services"]:
|
||||
event_callback(
|
||||
"deploy_execstart_mismatch",
|
||||
"检测到目标服务 ExecStart 未引用 current 软链，发布后可能不会切到新版本",
|
||||
level="warning",
|
||||
payload=execstart_alignment,
|
||||
)
|
||||
|
||||
if current_link.exists():
|
||||
try:
|
||||
@@ -259,6 +541,38 @@ def execute_release_action(
|
||||
temp_dir.rename(target_dir)
|
||||
extracted_target = target_dir
|
||||
|
||||
release_owner_user, release_owner_group = _pick_release_owner_group(releases_dir, service_identities)
|
||||
permission_result = apply_release_permissions(
|
||||
run_command,
|
||||
release_dir=extracted_target,
|
||||
owner_user=release_owner_user,
|
||||
owner_group=release_owner_group,
|
||||
)
|
||||
if not permission_result.get("ok", False):
|
||||
event_callback(
|
||||
"deploy_permission_fix_failed",
|
||||
f"发布目录权限修正失败: {release_version}",
|
||||
level="error",
|
||||
payload=permission_result,
|
||||
)
|
||||
return False, "release permission fix failed", {
|
||||
"release_version": release_version,
|
||||
"release_dir": str(extracted_target),
|
||||
"artifact_path": str(artifact_path),
|
||||
"checksum": calculated_checksum,
|
||||
"previous_current_target": previous_current_target,
|
||||
"prepared_dirs": prepared_dirs,
|
||||
"execstart_alignment": execstart_alignment,
|
||||
"service_identities": service_identities,
|
||||
"permission_result": permission_result,
|
||||
}
|
||||
if permission_result.get("attempted"):
|
||||
event_callback(
|
||||
"deploy_permissions_aligned",
|
||||
f"发布目录权限已对齐: {release_version}",
|
||||
payload=permission_result,
|
||||
)
|
||||
|
||||
meta_path = extracted_target / ".release-meta.json"
|
||||
meta_path.write_text(
|
||||
json.dumps(
|
||||
@@ -274,6 +588,50 @@ def execute_release_action(
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
systemd_sync_result = _sync_release_systemd_units(
|
||||
release_dir=extracted_target,
|
||||
install_root=install_root,
|
||||
systemd_unit_root=systemd_unit_root,
|
||||
switch_current=switch_current,
|
||||
)
|
||||
if systemd_sync_result.get("synced_units"):
|
||||
event_callback(
|
||||
"deploy_systemd_units_synced",
|
||||
f"systemd 单元已同步: {', '.join(systemd_sync_result.get('synced_units') or [])}",
|
||||
payload=systemd_sync_result,
|
||||
)
|
||||
daemon_reload_result = {
|
||||
"returncode": 0,
|
||||
"stdout": "",
|
||||
"stderr": "",
|
||||
}
|
||||
if systemd_sync_result.get("daemon_reload_required"):
|
||||
code, stdout, stderr = run_command(["systemctl", "daemon-reload"], timeout=45)
|
||||
daemon_reload_result = {
|
||||
"returncode": int(code or 0),
|
||||
"stdout": stdout,
|
||||
"stderr": stderr,
|
||||
}
|
||||
if int(code or 0) != 0:
|
||||
return False, "systemd daemon-reload failed", {
|
||||
"release_version": release_version,
|
||||
"release_dir": str(extracted_target),
|
||||
"artifact_path": str(artifact_path),
|
||||
"checksum": calculated_checksum,
|
||||
"previous_current_target": previous_current_target,
|
||||
"prepared_dirs": prepared_dirs,
|
||||
"execstart_alignment": execstart_alignment,
|
||||
"service_identities": service_identities,
|
||||
"permission_result": permission_result,
|
||||
"systemd_sync": systemd_sync_result,
|
||||
"daemon_reload": daemon_reload_result,
|
||||
}
|
||||
event_callback(
|
||||
"deploy_systemd_reloaded",
|
||||
"systemd daemon-reload 完成",
|
||||
payload=daemon_reload_result,
|
||||
)
|
||||
|
||||
if switch_current:
|
||||
if current_link.is_symlink() or current_link.is_file():
|
||||
current_link.unlink(missing_ok=True)
|
||||
@@ -312,6 +670,12 @@ def execute_release_action(
|
||||
"artifact_path": str(artifact_path),
|
||||
"checksum": calculated_checksum,
|
||||
"previous_current_target": previous_current_target,
|
||||
"prepared_dirs": prepared_dirs,
|
||||
"execstart_alignment": execstart_alignment,
|
||||
"service_identities": service_identities,
|
||||
"permission_result": permission_result,
|
||||
"systemd_sync": systemd_sync_result,
|
||||
"daemon_reload": daemon_reload_result,
|
||||
"restart_results": restarted,
|
||||
}
|
||||
|
||||
@@ -380,6 +744,10 @@ def execute_release_action(
|
||||
"checksum": calculated_checksum,
|
||||
"current_link": str(current_link),
|
||||
"previous_current_target": previous_current_target,
|
||||
"prepared_dirs": prepared_dirs,
|
||||
"execstart_alignment": execstart_alignment,
|
||||
"systemd_sync": systemd_sync_result,
|
||||
"daemon_reload": daemon_reload_result,
|
||||
"restart_results": restarted,
|
||||
"health_check": health_result,
|
||||
"rollback": rollback_result,
|
||||
@@ -397,6 +765,10 @@ def execute_release_action(
|
||||
"checksum": calculated_checksum,
|
||||
"current_link": str(current_link),
|
||||
"previous_current_target": previous_current_target,
|
||||
"prepared_dirs": prepared_dirs,
|
||||
"execstart_alignment": execstart_alignment,
|
||||
"systemd_sync": systemd_sync_result,
|
||||
"daemon_reload": daemon_reload_result,
|
||||
"restart_results": restarted,
|
||||
"health_check": health_result,
|
||||
}
|
||||
@@ -415,6 +787,15 @@ def build_remote_release_action_script(
|
||||
collect_service_state,
|
||||
check_health_url,
|
||||
run_release_health_checks,
|
||||
ensure_directory_ready,
|
||||
_resolve_path_owner_group,
|
||||
collect_service_identity,
|
||||
_pick_release_owner_group,
|
||||
apply_release_permissions,
|
||||
collect_service_execstart,
|
||||
_write_text_file,
|
||||
_systemd_dropin_content,
|
||||
_sync_release_systemd_units,
|
||||
safe_extract_tar,
|
||||
execute_release_action,
|
||||
]
|
||||
@@ -422,6 +803,17 @@ def build_remote_release_action_script(
|
||||
textwrap.dedent(inspect.getsource(func)).strip("\n")
|
||||
for func in helper_functions
|
||||
)
|
||||
systemd_template_specs_source = (
|
||||
"_SYSTEMD_TEMPLATE_SPECS = "
|
||||
+ repr(
|
||||
{
|
||||
service_name: {
|
||||
"template": str(spec["template"]),
|
||||
}
|
||||
for service_name, spec in _SYSTEMD_TEMPLATE_SPECS.items()
|
||||
}
|
||||
)
|
||||
)
|
||||
return f"""from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
@@ -435,12 +827,15 @@ from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
{systemd_template_specs_source}
|
||||
|
||||
|
||||
{helper_source}
|
||||
|
||||
|
||||
PAYLOAD = {json.dumps(dict(payload or {{}}), ensure_ascii=False)}
|
||||
DEFAULT_API_SERVICE_NAME = {json.dumps(str(default_api_service_name or 'domaincheck-api'), ensure_ascii=False)}
|
||||
USER_AGENT = {json.dumps(str(user_agent or 'domaincheck-ssh/0.1'), ensure_ascii=False)}
|
||||
PAYLOAD = {repr(dict(payload or {}))}
|
||||
DEFAULT_API_SERVICE_NAME = {repr(str(default_api_service_name or 'domaincheck-api'))}
|
||||
USER_AGENT = {repr(str(user_agent or 'domaincheck-ssh/0.1'))}
|
||||
|
||||
|
||||
def _run(command, timeout=60):
|
||||
|
||||
@@ -3,11 +3,13 @@ from __future__ import annotations
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
from datetime import datetime
|
||||
from math import ceil
|
||||
from pathlib import Path
|
||||
from threading import Lock
|
||||
from urllib.parse import urlsplit, urlunsplit
|
||||
from uuid import uuid4
|
||||
|
||||
from app.core.db import get_db
|
||||
@@ -490,7 +492,119 @@ def _normalize_public_base_url(raw_value: str) -> str:
|
||||
normalized = str(raw_value or "").strip().rstrip("/")
|
||||
if normalized.endswith("/api/v1"):
|
||||
normalized = normalized[: -len("/api/v1")]
|
||||
return normalized
|
||||
return _rewrite_loopback_control_plane_url(normalized)
|
||||
|
||||
|
||||
def _is_loopback_hostname(hostname: str) -> bool:
|
||||
normalized = str(hostname or "").strip().lower().strip("[]")
|
||||
return normalized in {"127.0.0.1", "localhost", "0.0.0.0", "::1"}
|
||||
|
||||
|
||||
def _build_url_with_host(raw_url: str, *, host: str, scheme: str = "", port: int | None = None) -> str:
|
||||
normalized_url = str(raw_url or "").strip()
|
||||
if not normalized_url:
|
||||
return ""
|
||||
parsed = urlsplit(normalized_url)
|
||||
if not parsed.scheme or not parsed.netloc:
|
||||
return normalized_url
|
||||
normalized_host = str(host or "").strip().strip("[]")
|
||||
if not normalized_host:
|
||||
return normalized_url
|
||||
final_scheme = str(scheme or parsed.scheme or "http").strip() or "http"
|
||||
final_port = parsed.port if port is None else int(port)
|
||||
netloc = f"{normalized_host}:{final_port}" if final_port else normalized_host
|
||||
return urlunsplit((final_scheme, netloc, parsed.path, parsed.query, parsed.fragment))
|
||||
|
||||
|
||||
def _resolve_public_control_plane_origin(loopback_url: str) -> str:
|
||||
normalized_loopback_url = str(loopback_url or "").strip()
|
||||
if not normalized_loopback_url:
|
||||
return ""
|
||||
parsed_loopback = urlsplit(normalized_loopback_url)
|
||||
default_scheme = str(parsed_loopback.scheme or "http").strip() or "http"
|
||||
default_port = parsed_loopback.port
|
||||
|
||||
env_candidates = [
|
||||
os.getenv("OPS_CONTROL_PLANE_PUBLIC_BASE_URL", ""),
|
||||
os.getenv("CONTROL_PLANE_PUBLIC_BASE_URL", ""),
|
||||
os.getenv("OPS_CONTROL_PLANE_BASE_URL", ""),
|
||||
]
|
||||
for candidate in env_candidates:
|
||||
normalized_candidate = str(candidate or "").strip().rstrip("/")
|
||||
if not normalized_candidate:
|
||||
continue
|
||||
parsed_candidate = urlsplit(
|
||||
normalized_candidate if "://" in normalized_candidate else f"{default_scheme}://{normalized_candidate}"
|
||||
)
|
||||
candidate_host = str(parsed_candidate.hostname or "").strip()
|
||||
if candidate_host and not _is_loopback_hostname(candidate_host):
|
||||
return _build_url_with_host(
|
||||
normalized_loopback_url,
|
||||
host=candidate_host,
|
||||
scheme=str(parsed_candidate.scheme or default_scheme),
|
||||
port=parsed_candidate.port if parsed_candidate.port is not None else default_port,
|
||||
)
|
||||
|
||||
try:
|
||||
from app.services.cluster_runtime_service import get_cluster_snapshot
|
||||
|
||||
snapshot = get_cluster_snapshot()
|
||||
local_hostnames = {
|
||||
str(socket.gethostname() or "").strip().lower(),
|
||||
str(socket.getfqdn() or "").strip().lower(),
|
||||
}
|
||||
fallback_control_hosts: list[str] = []
|
||||
for item in list(snapshot.get("nodes") or []):
|
||||
if str(item.get("role") or "").strip() != "control":
|
||||
continue
|
||||
control_host = str(item.get("hostname") or "").strip().lower()
|
||||
control_ip = str(item.get("ip") or "").strip()
|
||||
if not control_ip or _is_loopback_hostname(control_ip):
|
||||
continue
|
||||
if control_host and control_host in local_hostnames:
|
||||
return _build_url_with_host(
|
||||
normalized_loopback_url,
|
||||
host=control_ip,
|
||||
scheme=default_scheme,
|
||||
port=default_port,
|
||||
)
|
||||
fallback_control_hosts.append(control_ip)
|
||||
for control_ip in fallback_control_hosts:
|
||||
if control_ip and not _is_loopback_hostname(control_ip):
|
||||
return _build_url_with_host(
|
||||
normalized_loopback_url,
|
||||
host=control_ip,
|
||||
scheme=default_scheme,
|
||||
port=default_port,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
resolved_host = str(socket.gethostbyname(socket.gethostname()) or "").strip()
|
||||
if resolved_host and not _is_loopback_hostname(resolved_host):
|
||||
return _build_url_with_host(
|
||||
normalized_loopback_url,
|
||||
host=resolved_host,
|
||||
scheme=default_scheme,
|
||||
port=default_port,
|
||||
)
|
||||
except Exception:
|
||||
pass
|
||||
return ""
|
||||
|
||||
|
||||
def _rewrite_loopback_control_plane_url(raw_url: str) -> str:
|
||||
normalized = str(raw_url or "").strip()
|
||||
if not normalized:
|
||||
return ""
|
||||
parsed = urlsplit(normalized)
|
||||
if not parsed.scheme or not parsed.netloc:
|
||||
return normalized
|
||||
if not _is_loopback_hostname(str(parsed.hostname or "").strip()):
|
||||
return normalized
|
||||
resolved = _resolve_public_control_plane_origin(normalized)
|
||||
return resolved or normalized
|
||||
|
||||
|
||||
def _build_absolute_release_package_url(base_url: str, raw_path: str) -> str:
|
||||
@@ -1684,7 +1798,7 @@ def _resolve_rollout_targets(selector: dict) -> list[dict]:
|
||||
continue
|
||||
if only_effective_workers and not bool(item.get("is_effective_worker", False)):
|
||||
continue
|
||||
if only_online and str(item.get("status") or "") != "online":
|
||||
if only_online and str(item.get("status") or "") not in {"online", "busy"}:
|
||||
continue
|
||||
targets.append(item)
|
||||
|
||||
@@ -2037,9 +2151,12 @@ def _build_smart_rollout_role_policy(mode: str, *, execution_mode: str = "remote
|
||||
"restart_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"],
|
||||
"health_check_urls": ["http://127.0.0.1:8100/health"],
|
||||
"health_check_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"],
|
||||
"health_check_timeout_seconds": 10,
|
||||
"health_check_retries": 2,
|
||||
"health_check_interval_seconds": 2,
|
||||
# Control 节点启动期间会先经历较长的 import / startup hook，
|
||||
# systemd 已经 active 但 /health 仍可能在 15-20 秒内拒绝连接。
|
||||
# 这里把健康检查窗口放宽到约 40 秒，避免被误回滚。
|
||||
"health_check_timeout_seconds": 20,
|
||||
"health_check_retries": 9,
|
||||
"health_check_interval_seconds": 4,
|
||||
"rollback_on_failure": True,
|
||||
"switch_current": True,
|
||||
}
|
||||
@@ -3254,18 +3371,56 @@ def refresh_release_rollout_for_job(job_id: int) -> dict:
|
||||
return refresh_release_rollout(rollout_id)
|
||||
|
||||
|
||||
def _build_release_job_payload(release: dict, rollout: dict) -> dict:
|
||||
def _default_release_deploy_payload_for_target(target: dict) -> dict:
|
||||
role = str((target or {}).get("role") or "").strip().lower()
|
||||
if role == "control":
|
||||
return {
|
||||
"restart_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"],
|
||||
"health_check_urls": ["http://127.0.0.1:8100/health"],
|
||||
"health_check_services": ["domaincheck-api", "domaincheck-worker", "domaincheck-sync-agent"],
|
||||
"health_check_timeout_seconds": 20,
|
||||
"health_check_retries": 9,
|
||||
"health_check_interval_seconds": 4,
|
||||
}
|
||||
return {
|
||||
"restart_services": ["domaincheck-worker"],
|
||||
"health_check_urls": [],
|
||||
"health_check_services": ["domaincheck-worker"],
|
||||
"health_check_timeout_seconds": 10,
|
||||
"health_check_retries": 2,
|
||||
"health_check_interval_seconds": 2,
|
||||
}
|
||||
|
||||
|
||||
def _build_release_job_payload(release: dict, rollout: dict, *, target: dict | None = None) -> dict:
|
||||
policy = dict(rollout.get("policy") or {})
|
||||
deploy_payload = dict(policy.get("deploy_payload") or {})
|
||||
target_defaults = _default_release_deploy_payload_for_target(target or {})
|
||||
|
||||
if not [str(item).strip() for item in list(deploy_payload.get("restart_services") or []) if str(item).strip()]:
|
||||
deploy_payload["restart_services"] = list(target_defaults.get("restart_services") or [])
|
||||
if not [str(item).strip() for item in list(deploy_payload.get("health_check_services") or []) if str(item).strip()]:
|
||||
deploy_payload["health_check_services"] = list(target_defaults.get("health_check_services") or [])
|
||||
if not [str(item).strip() for item in list(deploy_payload.get("health_check_urls") or []) if str(item).strip()]:
|
||||
deploy_payload["health_check_urls"] = list(target_defaults.get("health_check_urls") or [])
|
||||
if deploy_payload.get("health_check_timeout_seconds") in (None, "", 0, "0"):
|
||||
deploy_payload["health_check_timeout_seconds"] = int(target_defaults.get("health_check_timeout_seconds") or 10)
|
||||
if deploy_payload.get("health_check_retries") in (None, "", 0, "0"):
|
||||
deploy_payload["health_check_retries"] = int(target_defaults.get("health_check_retries") or 2)
|
||||
if deploy_payload.get("health_check_interval_seconds") in (None, "", 0, "0"):
|
||||
deploy_payload["health_check_interval_seconds"] = int(target_defaults.get("health_check_interval_seconds") or 2)
|
||||
|
||||
artifact_url = _rewrite_loopback_control_plane_url(str(release.get("artifact_url") or "").strip())
|
||||
return {
|
||||
"release_id": int(release.get("id") or 0),
|
||||
"rollout_id": int(rollout.get("id") or 0),
|
||||
"release_version": str(release.get("release_version") or ""),
|
||||
"artifact_url": str(release.get("artifact_url") or ""),
|
||||
"artifact_url": artifact_url,
|
||||
"checksum": str(release.get("checksum") or ""),
|
||||
"channel": str(release.get("channel") or ""),
|
||||
"commit_sha": str(release.get("commit_sha") or ""),
|
||||
"notes": str(release.get("notes") or ""),
|
||||
"target_node_role": str((target or {}).get("role") or "").strip(),
|
||||
**deploy_payload,
|
||||
}
|
||||
|
||||
@@ -3308,12 +3463,11 @@ def _enqueue_rollout_batch(rollout_id: int, *, created_by: str, reason: str = "m
|
||||
auto_dispatch = bool(policy.get("auto_dispatch", False))
|
||||
auto_approve = bool(policy.get("auto_approve", False))
|
||||
execution_mode = str(policy.get("execution_mode") or "remote-agent").strip() or "remote-agent"
|
||||
job_payload = _build_release_job_payload(release, rollout)
|
||||
|
||||
for target in batch_targets:
|
||||
target_node_code = str(target.get("node_code") or "").strip()
|
||||
if not target_node_code:
|
||||
continue
|
||||
job_payload = _build_release_job_payload(release, rollout, target=target)
|
||||
job_ok, _job_message, job_data = create_ops_job(
|
||||
{
|
||||
"action": "deploy.release",
|
||||
|
||||
@@ -1,8 +1,16 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shlex
|
||||
import subprocess
|
||||
from io import StringIO
|
||||
|
||||
try:
|
||||
import paramiko
|
||||
except ImportError: # pragma: no cover - exercised via graceful fallback tests
|
||||
paramiko = None
|
||||
|
||||
from app.core.db import get_db
|
||||
from app.core.config import settings
|
||||
from app.services.ops_action_executor_core import (
|
||||
STRUCTURED_ACTIONS,
|
||||
@@ -76,6 +84,7 @@ def execute_ssh_action(node: dict, action: str, payload: dict | None = None) ->
|
||||
}
|
||||
|
||||
normalized_payload = dict(payload or {})
|
||||
node_secret = _load_ssh_secret(node_code)
|
||||
service_names = _service_name_map()
|
||||
if normalized_action == "deploy.release":
|
||||
remote_script = build_remote_release_action_script(
|
||||
@@ -105,28 +114,69 @@ def execute_ssh_action(node: dict, action: str, payload: dict | None = None) ->
|
||||
"PY",
|
||||
]
|
||||
)
|
||||
ssh_command = [
|
||||
"ssh",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"PreferredAuthentications=publickey",
|
||||
"-o",
|
||||
"StrictHostKeyChecking=accept-new",
|
||||
"-o",
|
||||
f"ConnectTimeout={_SSH_CONNECT_TIMEOUT_SECONDS}",
|
||||
"-p",
|
||||
str(ssh_port),
|
||||
f"{ssh_user}@{ssh_host}",
|
||||
remote_command,
|
||||
]
|
||||
timeout_seconds = int(_SSH_REMOTE_TIMEOUT_SECONDS.get(normalized_action, 45) or 45) + _SSH_CONNECT_TIMEOUT_SECONDS
|
||||
completed = subprocess.run(
|
||||
ssh_command,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout_seconds,
|
||||
)
|
||||
auth_mode = str(node.get("auth_mode") or "").strip() or ("password" if node_secret.get("ssh_password") else "key")
|
||||
ssh_password = str(node_secret.get("ssh_password") or "").strip()
|
||||
ssh_private_key = str(node_secret.get("ssh_private_key") or "").strip()
|
||||
if auth_mode == "key" and not ssh_private_key and ssh_password:
|
||||
auth_mode = "password"
|
||||
elif auth_mode == "password" and not ssh_password and ssh_private_key:
|
||||
auth_mode = "key"
|
||||
if (auth_mode == "password" and ssh_password) or ssh_private_key:
|
||||
if paramiko is None:
|
||||
return False, "当前环境未安装 paramiko，无法使用密码或私钥 SSH 执行", {
|
||||
"executor": "ssh",
|
||||
"action": normalized_action,
|
||||
"transport": {
|
||||
"executor": "ssh",
|
||||
"node_code": node_code,
|
||||
"ssh_host": ssh_host,
|
||||
"ssh_user": ssh_user,
|
||||
"ssh_port": ssh_port,
|
||||
"auth_mode": auth_mode,
|
||||
"action": normalized_action,
|
||||
},
|
||||
}
|
||||
if auth_mode == "password" and ssh_password:
|
||||
completed = _run_paramiko_command(
|
||||
ssh_host=ssh_host,
|
||||
ssh_port=ssh_port,
|
||||
ssh_user=ssh_user,
|
||||
remote_command=remote_command,
|
||||
timeout_seconds=timeout_seconds,
|
||||
ssh_password=ssh_password,
|
||||
)
|
||||
elif ssh_private_key:
|
||||
completed = _run_paramiko_command(
|
||||
ssh_host=ssh_host,
|
||||
ssh_port=ssh_port,
|
||||
ssh_user=ssh_user,
|
||||
remote_command=remote_command,
|
||||
timeout_seconds=timeout_seconds,
|
||||
ssh_private_key=ssh_private_key,
|
||||
)
|
||||
else:
|
||||
ssh_command = [
|
||||
"ssh",
|
||||
"-o",
|
||||
"BatchMode=yes",
|
||||
"-o",
|
||||
"PreferredAuthentications=publickey",
|
||||
"-o",
|
||||
"StrictHostKeyChecking=accept-new",
|
||||
"-o",
|
||||
f"ConnectTimeout={_SSH_CONNECT_TIMEOUT_SECONDS}",
|
||||
"-p",
|
||||
str(ssh_port),
|
||||
f"{ssh_user}@{ssh_host}",
|
||||
remote_command,
|
||||
]
|
||||
completed = subprocess.run(
|
||||
ssh_command,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=timeout_seconds,
|
||||
)
|
||||
|
||||
transport = {
|
||||
"executor": "ssh",
|
||||
@@ -134,6 +184,7 @@ def execute_ssh_action(node: dict, action: str, payload: dict | None = None) ->
|
||||
"ssh_host": ssh_host,
|
||||
"ssh_user": ssh_user,
|
||||
"ssh_port": ssh_port,
|
||||
"auth_mode": auth_mode,
|
||||
"action": normalized_action,
|
||||
"returncode": int(completed.returncode or 0),
|
||||
}
|
||||
@@ -166,6 +217,93 @@ def execute_ssh_action(node: dict, action: str, payload: dict | None = None) ->
|
||||
}
|
||||
|
||||
|
||||
def _load_ssh_secret(node_code: str) -> dict:
|
||||
normalized_node_code = str(node_code or "").strip()
|
||||
if not normalized_node_code:
|
||||
return {}
|
||||
try:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT ssh_password, ssh_private_key
|
||||
FROM ops_managed_node_secrets
|
||||
WHERE node_code = %s
|
||||
LIMIT 1
|
||||
""",
|
||||
(normalized_node_code,),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
except Exception:
|
||||
return {}
|
||||
if not row:
|
||||
return {}
|
||||
return {
|
||||
"ssh_password": str(row[0] or ""),
|
||||
"ssh_private_key": str(row[1] or ""),
|
||||
}
|
||||
|
||||
|
||||
def _load_private_key(private_key_text: str) -> paramiko.PKey:
|
||||
if paramiko is None:
|
||||
raise RuntimeError("paramiko is not installed")
|
||||
key_text = str(private_key_text or "")
|
||||
for key_cls in (paramiko.Ed25519Key, paramiko.RSAKey, paramiko.ECDSAKey, paramiko.DSSKey):
|
||||
try:
|
||||
return key_cls.from_private_key(StringIO(key_text))
|
||||
except Exception:
|
||||
continue
|
||||
raise ValueError("无法识别 SSH 私钥格式")
|
||||
|
||||
|
||||
def _run_paramiko_command(
|
||||
*,
|
||||
ssh_host: str,
|
||||
ssh_port: int,
|
||||
ssh_user: str,
|
||||
remote_command: str,
|
||||
timeout_seconds: int,
|
||||
ssh_password: str = "",
|
||||
ssh_private_key: str = "",
|
||||
) -> subprocess.CompletedProcess:
|
||||
if paramiko is None:
|
||||
raise RuntimeError("paramiko is not installed")
|
||||
client = paramiko.SSHClient()
|
||||
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
connect_kwargs = {
|
||||
"hostname": ssh_host,
|
||||
"port": int(ssh_port),
|
||||
"username": ssh_user,
|
||||
"timeout": _SSH_CONNECT_TIMEOUT_SECONDS,
|
||||
"banner_timeout": _SSH_CONNECT_TIMEOUT_SECONDS,
|
||||
"auth_timeout": _SSH_CONNECT_TIMEOUT_SECONDS,
|
||||
"look_for_keys": False,
|
||||
"allow_agent": False,
|
||||
}
|
||||
if ssh_password:
|
||||
connect_kwargs["password"] = ssh_password
|
||||
elif ssh_private_key:
|
||||
connect_kwargs["pkey"] = _load_private_key(ssh_private_key)
|
||||
else:
|
||||
connect_kwargs["look_for_keys"] = True
|
||||
connect_kwargs["allow_agent"] = True
|
||||
try:
|
||||
client.connect(**connect_kwargs)
|
||||
wrapped_command = f"bash -lc {shlex.quote(remote_command)}"
|
||||
_, stdout, stderr = client.exec_command(wrapped_command, timeout=timeout_seconds)
|
||||
returncode = int(stdout.channel.recv_exit_status())
|
||||
stdout_text = stdout.read().decode("utf-8", errors="replace")
|
||||
stderr_text = stderr.read().decode("utf-8", errors="replace")
|
||||
return subprocess.CompletedProcess(
|
||||
args=["paramiko", f"{ssh_user}@{ssh_host}"],
|
||||
returncode=returncode,
|
||||
stdout=stdout_text,
|
||||
stderr=stderr_text,
|
||||
)
|
||||
finally:
|
||||
client.close()
|
||||
|
||||
|
||||
def _service_name_map() -> dict[str, str]:
|
||||
return build_service_name_map(
|
||||
api_service_name=settings.api_service_name,
|
||||
@@ -199,6 +337,25 @@ def trim_output(text, limit):
|
||||
|
||||
def run(cmd, timeout=60):
|
||||
completed = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
|
||||
normalized_cmd = [str(part or "").strip() for part in cmd]
|
||||
combined_output = f"{{completed.stdout or ''}}\\n{{completed.stderr or ''}}".lower()
|
||||
needs_sudo_retry = (
|
||||
normalized_cmd
|
||||
and normalized_cmd[0] == "systemctl"
|
||||
and completed.returncode != 0
|
||||
and "sudo" not in normalized_cmd
|
||||
and any(
|
||||
marker in combined_output
|
||||
for marker in (
|
||||
"interactive authentication required",
|
||||
"authentication is required",
|
||||
"authorization not available",
|
||||
"polkit",
|
||||
)
|
||||
)
|
||||
)
|
||||
if needs_sudo_retry:
|
||||
completed = subprocess.run(["sudo", "-n", *normalized_cmd], capture_output=True, text=True, timeout=timeout)
|
||||
return int(completed.returncode or 0), str(completed.stdout or "").strip(), str(completed.stderr or "").strip()
|
||||
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ from pathlib import Path
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.debug_event_service import push_debug_event
|
||||
from app.services.detect_job_service import process_detect_pipeline_now
|
||||
from app.services.sync_push_service import pull_detect_task_batch_now, push_runtime_projection_now
|
||||
from app.services.runtime_settings_service import get_runtime_settings
|
||||
from app.services.worker_control_service import _run_systemctl, normalize_systemctl_error, send_worker_command, start_worker, stop_worker
|
||||
@@ -180,6 +181,16 @@ def runtime_action(action: str, payload: dict | None = None) -> tuple[bool, str,
|
||||
result = _build_runtime_action_result(action=normalized_action, poll_after_seconds=2, refresh_runtime=True, ok=ok, message=message, data=data)
|
||||
_emit_runtime_action_event(normalized_action, stage="finished", ok=ok, message=message, data=result)
|
||||
return ok, message, result
|
||||
if normalized_action == "process_pipeline":
|
||||
process_limit = normalized_payload.get("limit")
|
||||
process_job_id = normalized_payload.get("job_id")
|
||||
ok, message, data = process_detect_pipeline_now(
|
||||
limit=int(process_limit or 0) or None,
|
||||
job_id=int(process_job_id or 0) or None,
|
||||
)
|
||||
result = _build_runtime_action_result(action=normalized_action, poll_after_seconds=1, refresh_runtime=True, ok=ok, message=message, data=data)
|
||||
_emit_runtime_action_event(normalized_action, stage="finished", ok=ok, message=message, data=result)
|
||||
return ok, message, result
|
||||
if normalized_action == "start_detection":
|
||||
command_ok, command_message = send_worker_command(
|
||||
"start_detection",
|
||||
|
||||
@@ -10,12 +10,151 @@ from app.core.redis_client import get_redis
|
||||
from app.services.build_info_service import get_runtime_build_info
|
||||
from app.services.cluster_runtime_service import get_cluster_snapshot
|
||||
from app.services.detect_service import get_detect_status
|
||||
from app.services.detect_job_service import get_detect_capacity_plan, get_detect_queue_health
|
||||
from app.services.detect_job_service import (
|
||||
_load_latest_runtime_active_job_snapshot,
|
||||
get_detect_capacity_plan,
|
||||
get_detect_queue_health,
|
||||
)
|
||||
from app.services.sync_record_service import append_runtime_projection_if_changed, get_sync_summary
|
||||
from app.services.runtime_settings_service import get_runtime_settings
|
||||
from app.services.worker_control_service import detect_sync_agent_runtime, detect_worker_runtime
|
||||
|
||||
|
||||
def _align_queue_health_with_backlog(queue_health: dict | None, backlog_snapshot: dict | None) -> dict:
|
||||
normalized = dict(queue_health or {})
|
||||
queue = dict(normalized.get("queue") or {})
|
||||
backlog = dict(backlog_snapshot or {})
|
||||
|
||||
pending_total = max(int(queue.get("pending", 0) or 0), int(backlog.get("pending_total", 0) or 0))
|
||||
claimed_total = max(int(queue.get("claimed", 0) or 0), int(backlog.get("claimed_total", 0) or 0))
|
||||
running_total = max(int(queue.get("running", 0) or 0), int(backlog.get("running_total", 0) or 0))
|
||||
completed_total = max(int(queue.get("completed", 0) or 0), int(backlog.get("completed_total", 0) or 0))
|
||||
blacklisted_total = max(int(queue.get("blacklisted", 0) or 0), int(backlog.get("blacklisted_total", 0) or 0))
|
||||
failed_total = max(int(queue.get("failed", 0) or 0), int(backlog.get("failed_total", 0) or 0))
|
||||
terminal_total = max(
|
||||
int(queue.get("terminal", 0) or 0),
|
||||
completed_total + blacklisted_total + failed_total,
|
||||
)
|
||||
|
||||
normalized["has_active_job"] = bool(
|
||||
normalized.get("has_active_job")
|
||||
or pending_total > 0
|
||||
or claimed_total > 0
|
||||
or running_total > 0
|
||||
or terminal_total > 0
|
||||
)
|
||||
normalized["queue"] = {
|
||||
**queue,
|
||||
"items_total": pending_total + claimed_total + running_total + terminal_total,
|
||||
"pending": pending_total,
|
||||
"claimed": claimed_total,
|
||||
"running": running_total,
|
||||
"completed": completed_total,
|
||||
"blacklisted": blacklisted_total,
|
||||
"failed": failed_total,
|
||||
"terminal": terminal_total,
|
||||
}
|
||||
return normalized
|
||||
|
||||
|
||||
def _decode_projection_payload(value: object) -> dict:
|
||||
if isinstance(value, dict):
|
||||
return dict(value)
|
||||
if value in (None, ""):
|
||||
return {}
|
||||
try:
|
||||
import json
|
||||
|
||||
return dict(json.loads(value))
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _load_detect_backlog_snapshot() -> dict:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
COUNT(*) FILTER (WHERE item.status = 'pending') AS pending_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'claimed') AS claimed_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'running') AS running_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'completed') AS completed_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'blacklisted') AS blacklisted_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'failed') AS failed_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code = 'detect_register') AS register_pending,
|
||||
COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code <> 'detect_register') AS downstream_pending
|
||||
FROM detect_job_items item
|
||||
JOIN detect_jobs job ON job.id = item.job_id
|
||||
WHERE job.status IN ('pending', 'running')
|
||||
"""
|
||||
)
|
||||
row = cur.fetchone() or (0, 0, 0, 0, 0, 0, 0, 0)
|
||||
return {
|
||||
"pending_total": int(row[0] or 0),
|
||||
"claimed_total": int(row[1] or 0),
|
||||
"running_total": int(row[2] or 0),
|
||||
"completed_total": int(row[3] or 0),
|
||||
"blacklisted_total": int(row[4] or 0),
|
||||
"failed_total": int(row[5] or 0),
|
||||
"register_pending": int(row[6] or 0),
|
||||
"downstream_pending": int(row[7] or 0),
|
||||
}
|
||||
|
||||
|
||||
def _load_latest_remote_runtime_projection_backlog() -> dict:
|
||||
if not (settings.node_region == "overseas" and settings.node_role == "control"):
|
||||
return {}
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT payload_json
|
||||
FROM detect_sync_records
|
||||
WHERE sync_type = 'runtime_projection'
|
||||
AND source_region = 'mainland'
|
||||
AND target_region = 'overseas'
|
||||
AND status IN ('projected', 'pushing', 'synced')
|
||||
ORDER BY updated_at DESC, id DESC
|
||||
LIMIT 1
|
||||
"""
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if not row:
|
||||
return {}
|
||||
payload = _decode_projection_payload(row[0])
|
||||
projection = payload.get("projection") if isinstance(payload, dict) else {}
|
||||
backlog = projection.get("backlog") if isinstance(projection, dict) else {}
|
||||
if not isinstance(backlog, dict):
|
||||
return {}
|
||||
return {
|
||||
"pending_total": int(backlog.get("pending_total", 0) or 0),
|
||||
"claimed_total": int(backlog.get("claimed_total", 0) or 0),
|
||||
"running_total": int(backlog.get("running_total", 0) or 0),
|
||||
"completed_total": int(backlog.get("completed_total", 0) or 0),
|
||||
"blacklisted_total": int(backlog.get("blacklisted_total", 0) or 0),
|
||||
"failed_total": int(backlog.get("failed_total", 0) or 0),
|
||||
"register_pending": int(backlog.get("register_pending", 0) or 0),
|
||||
"downstream_pending": int(backlog.get("downstream_pending", 0) or 0),
|
||||
}
|
||||
|
||||
|
||||
def _merge_backlog_snapshots(primary: dict, secondary: dict) -> dict:
|
||||
merged = dict(primary or {})
|
||||
for key in (
|
||||
"pending_total",
|
||||
"claimed_total",
|
||||
"running_total",
|
||||
"completed_total",
|
||||
"blacklisted_total",
|
||||
"failed_total",
|
||||
"register_pending",
|
||||
"downstream_pending",
|
||||
):
|
||||
merged[key] = max(int(merged.get(key, 0) or 0), int((secondary or {}).get(key, 0) or 0))
|
||||
return merged
|
||||
|
||||
|
||||
def _runtime_log_path(filename: str) -> str:
|
||||
path = Path(__file__).resolve().parents[2] / "runtime" / "logs" / filename
|
||||
return str(path)
|
||||
@@ -191,6 +330,8 @@ def _build_multi_region_readiness(
|
||||
def _detect_participation_snapshot(*, row: dict) -> dict:
|
||||
items_running = int(row.get("items_running", 0) or 0)
|
||||
items_claimed = int(row.get("items_claimed", 0) or 0)
|
||||
active_threads = int(row.get("active_threads", 0) or 0)
|
||||
max_threads = int(row.get("max_threads", 0) or 0)
|
||||
processed_recent = int(row.get("processed_recent", 0) or 0)
|
||||
current_load = int(row.get("current_load", 0) or 0)
|
||||
status = str(row.get("status") or "").strip().lower()
|
||||
@@ -211,6 +352,17 @@ def _detect_participation_snapshot(*, row: dict) -> dict:
|
||||
"is_current_participant": True,
|
||||
"is_dispatch_active": True,
|
||||
}
|
||||
if active_threads > 0:
|
||||
detail = f"当前活跃线程 {active_threads}"
|
||||
if max_threads > 0:
|
||||
detail = f"{detail}/{max_threads}"
|
||||
return {
|
||||
"participation_state": "runtime_active",
|
||||
"participation_label": "执行中",
|
||||
"participation_reason": detail,
|
||||
"is_current_participant": True,
|
||||
"is_dispatch_active": True,
|
||||
}
|
||||
if processed_recent > 0:
|
||||
return {
|
||||
"participation_state": "recent_throughput",
|
||||
@@ -242,11 +394,52 @@ def _build_detect_node_row(*, node_code: str, cluster_node: dict, job_node: dict
|
||||
role = str(cluster_node.get("role") or job_node.get("role") or "worker")
|
||||
region = str(cluster_node.get("region") or settings.node_region)
|
||||
is_effective_worker = bool(cluster_node.get("is_effective_worker", False) or role == "worker")
|
||||
items_total = int(job_node.get("items_total", metadata.get("job_items_total", 0)) or 0)
|
||||
items_claimed = int(job_node.get("items_claimed", metadata.get("job_items_claimed", 0)) or 0)
|
||||
items_running = int(job_node.get("items_running", metadata.get("job_items_running", 0)) or 0)
|
||||
items_completed = int(job_node.get("items_completed", metadata.get("job_items_completed", 0)) or 0)
|
||||
items_failed = int(job_node.get("items_failed", metadata.get("job_items_failed", 0)) or 0)
|
||||
items_total = max(
|
||||
int(job_node.get("items_total", 0) or 0),
|
||||
int(queue_node.get("items_total", 0) or 0),
|
||||
int(metadata.get("job_items_total", 0) or 0),
|
||||
)
|
||||
items_claimed = max(
|
||||
int(job_node.get("items_claimed", 0) or 0),
|
||||
int(queue_node.get("items_claimed", 0) or 0),
|
||||
int(metadata.get("job_items_claimed", 0) or 0),
|
||||
)
|
||||
items_running = max(
|
||||
int(job_node.get("items_running", 0) or 0),
|
||||
int(queue_node.get("items_running", 0) or 0),
|
||||
int(metadata.get("job_items_running", 0) or 0),
|
||||
)
|
||||
items_completed = max(
|
||||
int(job_node.get("items_completed", 0) or 0),
|
||||
int(queue_node.get("items_completed", 0) or 0),
|
||||
int(metadata.get("job_items_completed", 0) or 0),
|
||||
)
|
||||
items_failed = max(
|
||||
int(job_node.get("items_failed", 0) or 0),
|
||||
int(queue_node.get("items_failed", 0) or 0),
|
||||
int(metadata.get("job_items_failed", 0) or 0),
|
||||
)
|
||||
items_blacklisted = max(
|
||||
int(job_node.get("items_blacklisted", 0) or 0),
|
||||
int(queue_node.get("items_blacklisted", 0) or 0),
|
||||
)
|
||||
active_threads = max(
|
||||
int((cluster_node.get("metadata") or {}).get("active_threads", 0) or 0),
|
||||
int(job_node.get("active_threads", 0) or 0),
|
||||
int(queue_node.get("active_threads", 0) or 0),
|
||||
)
|
||||
max_threads = max(
|
||||
int((cluster_node.get("metadata") or {}).get("max_threads", 0) or 0),
|
||||
int(job_node.get("max_threads", 0) or 0),
|
||||
int(queue_node.get("max_threads", 0) or 0),
|
||||
)
|
||||
current_load = max(items_running, active_threads, 0)
|
||||
derived_pending = max(items_total - items_claimed - items_running - items_completed - items_failed - items_blacklisted, 0)
|
||||
items_pending = max(
|
||||
int(job_node.get("items_pending", 0) or 0),
|
||||
int(queue_node.get("items_pending", 0) or 0),
|
||||
derived_pending,
|
||||
)
|
||||
row = {
|
||||
"node_code": node_code,
|
||||
"role": role,
|
||||
@@ -254,13 +447,16 @@ def _build_detect_node_row(*, node_code: str, cluster_node: dict, job_node: dict
|
||||
"status": str(cluster_node.get("status") or "unknown"),
|
||||
"is_effective_worker": is_effective_worker,
|
||||
"detect_participating": False,
|
||||
"current_load": int(cluster_node.get("current_load", 0) or 0),
|
||||
"current_load": current_load,
|
||||
"items_total": items_total,
|
||||
"items_pending": int(job_node.get("items_pending", max(items_total - items_claimed - items_completed - items_failed, 0)) or 0),
|
||||
"items_pending": items_pending,
|
||||
"items_claimed": items_claimed,
|
||||
"items_running": items_running,
|
||||
"items_completed": items_completed,
|
||||
"items_blacklisted": items_blacklisted,
|
||||
"items_failed": items_failed,
|
||||
"active_threads": active_threads,
|
||||
"max_threads": max_threads,
|
||||
"processed_recent": int(queue_node.get("processed_recent", 0) or 0),
|
||||
"processed_per_minute": float(queue_node.get("processed_per_minute", 0) or 0),
|
||||
"last_heartbeat_at": str(cluster_node.get("last_heartbeat_at") or ""),
|
||||
@@ -465,8 +661,13 @@ def get_runtime_status() -> dict:
|
||||
effective_online_worker_nodes = int((cluster_snapshot.get("summary") or {}).get("online_worker_nodes", 0) or 0)
|
||||
if effective_online_worker_nodes <= 0 and worker_runtime.get("running", False):
|
||||
effective_online_worker_nodes = max(1, worker_runtime.get("process_count", 1) or 1)
|
||||
backlog_snapshot = _load_detect_backlog_snapshot()
|
||||
remote_backlog_snapshot = _load_latest_remote_runtime_projection_backlog()
|
||||
runtime_snapshot_backlog = dict(_load_latest_runtime_active_job_snapshot(15).get("backlog") or {})
|
||||
backlog_snapshot = _merge_backlog_snapshots(backlog_snapshot, remote_backlog_snapshot)
|
||||
backlog_snapshot = _merge_backlog_snapshots(backlog_snapshot, runtime_snapshot_backlog)
|
||||
capacity_plan = get_detect_capacity_plan(
|
||||
queue_health=queue_health,
|
||||
queue_health=_align_queue_health_with_backlog(queue_health, backlog_snapshot),
|
||||
online_worker_nodes=effective_online_worker_nodes,
|
||||
target_finish_hours=6,
|
||||
)
|
||||
@@ -498,6 +699,7 @@ def get_runtime_status() -> dict:
|
||||
"worker_online": worker_runtime.get("running", False),
|
||||
"worker_mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
|
||||
"queue_health": queue_health,
|
||||
"backlog": backlog_snapshot,
|
||||
"capacity_plan": capacity_plan,
|
||||
"log_sync": {
|
||||
"enabled": bool(runtime_settings.get("worker_log_sync_enabled", False)),
|
||||
@@ -565,8 +767,7 @@ def get_runtime_status() -> dict:
|
||||
)
|
||||
build_info = get_runtime_build_info()
|
||||
|
||||
return {
|
||||
"api": {
|
||||
api_payload = {
|
||||
"service": "domain-api",
|
||||
"version": "0.1.0",
|
||||
"api_prefix": settings.api_prefix,
|
||||
@@ -579,13 +780,13 @@ def get_runtime_status() -> dict:
|
||||
"stdout_log": _runtime_log_path("domain-api.stdout.log"),
|
||||
"stderr_log": _runtime_log_path("domain-api.stderr.log"),
|
||||
"build": build_info,
|
||||
},
|
||||
"node": {
|
||||
}
|
||||
node_payload = {
|
||||
"code": settings.node_code,
|
||||
"region": settings.node_region,
|
||||
"role": settings.node_role,
|
||||
},
|
||||
"worker": {
|
||||
}
|
||||
worker_payload = {
|
||||
"mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
|
||||
"service_name": runtime_settings.get("worker_service_name", settings.worker_service_name),
|
||||
"running": worker_runtime.get("running", False),
|
||||
@@ -594,8 +795,8 @@ def get_runtime_status() -> dict:
|
||||
"latest_start_time": worker_runtime.get("latest_start_time", ""),
|
||||
"message": worker_runtime.get("message", ""),
|
||||
"log_path": str(Path(settings.domain_root) / "detect_worker.log"),
|
||||
},
|
||||
"sync_agent": {
|
||||
}
|
||||
sync_agent_payload = {
|
||||
"mode": sync_agent_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
|
||||
"service_name": runtime_settings.get("sync_agent_service_name", settings.sync_agent_service_name),
|
||||
"running": sync_agent_runtime.get("running", False),
|
||||
@@ -603,7 +804,44 @@ def get_runtime_status() -> dict:
|
||||
"latest_start_time": sync_agent_runtime.get("latest_start_time", ""),
|
||||
"message": sync_agent_runtime.get("message", ""),
|
||||
"expected_on_this_node": settings.node_region == "mainland" and settings.node_role == "control",
|
||||
},
|
||||
}
|
||||
compatibility_payload = {
|
||||
# Backward-compatible flat fields for older pages / stale built assets.
|
||||
"api_online": bool(api_payload.get("pid")),
|
||||
"api_service_name": api_payload.get("service_name", ""),
|
||||
"worker_online": bool(worker_payload.get("running", False)),
|
||||
"worker_mode": worker_payload.get("mode", ""),
|
||||
"worker_service_name": worker_payload.get("service_name", ""),
|
||||
"worker_process_count": worker_payload.get("process_count", 0),
|
||||
"worker_latest_start_time": worker_payload.get("latest_start_time", ""),
|
||||
"worker_runtime_message": worker_payload.get("message", ""),
|
||||
"thread_count": detect_snapshot.get("thread_count", 0),
|
||||
"thread_count_default": detect_snapshot.get("thread_count_default", 0),
|
||||
"thread_count_source": detect_snapshot.get("thread_count_source", ""),
|
||||
"thread_count_override": detect_snapshot.get("thread_count_override"),
|
||||
"active_thread_count": detect_payload.get("active_thread_count", 0),
|
||||
"max_thread_count": detect_payload.get("max_thread_count", 0),
|
||||
"progress": detect_payload.get("progress", {}),
|
||||
"backlog": detect_payload.get("backlog", {}),
|
||||
"progress_percent": detect_payload.get("progress_percent", 0),
|
||||
"available_proxy_count": detect_payload.get("available_proxy_count", 0),
|
||||
"proxy_pool_count": detect_payload.get("proxy_pool_count", 0),
|
||||
"proxy_runtime_label": detect_payload.get("proxy_runtime_label", ""),
|
||||
"proxy_runtime_detail": detect_payload.get("proxy_runtime_detail", ""),
|
||||
"proxy_runtime_reason": detect_payload.get("proxy_runtime_reason", ""),
|
||||
"proxy_last_refresh_time": detect_payload.get("proxy_last_refresh_time", ""),
|
||||
"recent_event": detect_payload.get("recent_event", ""),
|
||||
"recent_warning": detect_payload.get("recent_warning", ""),
|
||||
"runtime_state": worker_runtime.get("runtime_state") or {},
|
||||
"cluster_summary": cluster_snapshot.get("summary") or {},
|
||||
}
|
||||
|
||||
return {
|
||||
**compatibility_payload,
|
||||
"api": api_payload,
|
||||
"node": node_payload,
|
||||
"worker": worker_payload,
|
||||
"sync_agent": sync_agent_payload,
|
||||
"detect": detect_payload,
|
||||
"cluster": cluster_snapshot,
|
||||
"sync": sync_summary,
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
from app.core.db import get_db
|
||||
from app.core.redis_client import get_redis
|
||||
|
||||
|
||||
def get_sensitive_words_payload() -> dict:
|
||||
@@ -54,6 +57,13 @@ def save_sensitive_words_payload(payload: dict) -> dict:
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
try:
|
||||
redis_client = get_redis()
|
||||
redis_client.set("domain_tool:sensitive_words", json.dumps(words, ensure_ascii=False))
|
||||
redis_client.publish("domain_tool:config_update", "sensitive_words")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return {
|
||||
"total": len(words),
|
||||
"text": "\n".join(words),
|
||||
|
||||
@@ -35,8 +35,8 @@ def _normalize_thread_count(value: object, *, field_name: str = "thread_count")
|
||||
thread_count = int(value)
|
||||
except Exception as exc:
|
||||
raise ValueError(f"{field_name} must be an integer") from exc
|
||||
if thread_count < 1 or thread_count > 256:
|
||||
raise ValueError(f"{field_name} out of range")
|
||||
if thread_count < 1:
|
||||
raise ValueError(f"{field_name} must be >= 1")
|
||||
return thread_count
|
||||
|
||||
|
||||
|
||||
@@ -11,7 +11,16 @@ from uuid import uuid4
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.db import get_db
|
||||
from app.services.cluster_runtime_service import cleanup_imported_runtime_nodes, register_node_heartbeat
|
||||
from app.services.cluster_runtime_service import (
|
||||
cleanup_imported_runtime_nodes,
|
||||
cleanup_imported_runtime_nodes_many,
|
||||
register_node_heartbeat,
|
||||
)
|
||||
from app.services.detect_job_service import (
|
||||
_load_domain_pipeline_snapshot,
|
||||
resolve_initial_domain_pipeline_item,
|
||||
)
|
||||
from app.services.settings_service import get_settings_payload
|
||||
from app.services.sync_record_service import _decode_json, _normalize_region
|
||||
|
||||
|
||||
@@ -80,10 +89,16 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv
|
||||
if not node_code:
|
||||
node_code = f"{region}-{role}-imported"
|
||||
|
||||
controller_current_load = max(
|
||||
int(projection.get("active_thread_count", 0) or 0),
|
||||
int(((projection.get("active_job") or {}).get("items_running", 0) or 0)),
|
||||
)
|
||||
metadata = {
|
||||
"service": "runtime-ingest",
|
||||
"projection_source_region": source_region,
|
||||
"worker_mode": projection.get("worker_mode", ""),
|
||||
"active_threads": int(projection.get("active_thread_count", 0) or 0),
|
||||
"max_threads": int(projection.get("max_thread_count", 0) or 0),
|
||||
"phase_label": projection.get("phase_label", ""),
|
||||
"phase_detail": projection.get("phase_detail", ""),
|
||||
"proxy_runtime_label": projection.get("proxy_runtime_label", ""),
|
||||
@@ -102,8 +117,8 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv
|
||||
node_code=node_code,
|
||||
region=region,
|
||||
role=role,
|
||||
status="online",
|
||||
current_load=int(((projection.get("progress") or {}).get("running", 0) or 0)),
|
||||
status="busy" if controller_current_load > 0 else "online",
|
||||
current_load=controller_current_load,
|
||||
metadata=metadata,
|
||||
hostname_override=hostname,
|
||||
ip_override=ip,
|
||||
@@ -111,15 +126,21 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv
|
||||
cleanup_imported_runtime_nodes(region=region, role=role, keep_node_code=node_code)
|
||||
|
||||
active_job = projection.get("active_job") or {}
|
||||
worker_node_codes: list[str] = []
|
||||
for node_stat in list(active_job.get("node_stats") or []):
|
||||
worker_node_code = str(node_stat.get("node_code") or "").strip()
|
||||
if not worker_node_code or worker_node_code == "unassigned":
|
||||
continue
|
||||
if worker_node_code == node_code:
|
||||
continue
|
||||
items_running = int(node_stat.get("items_running", 0) or 0)
|
||||
items_claimed = int(node_stat.get("items_claimed", 0) or 0)
|
||||
items_total = int(node_stat.get("items_total", 0) or 0)
|
||||
worker_status = "busy" if (items_running > 0 or items_claimed > 0) else "online"
|
||||
worker_load = max(items_running, items_claimed, 0)
|
||||
worker_runtime_load = int(node_stat.get("current_load", 0) or 0)
|
||||
worker_active_threads = int(node_stat.get("active_threads", worker_runtime_load) or 0)
|
||||
worker_max_threads = int(node_stat.get("max_threads", 0) or 0)
|
||||
worker_load = max(worker_active_threads, items_running, 0)
|
||||
worker_status = "busy" if worker_load > 0 else "online"
|
||||
worker_metadata = {
|
||||
"service": "runtime-ingest",
|
||||
"projection_source_region": source_region,
|
||||
@@ -128,10 +149,18 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv
|
||||
"phase_detail": projection.get("phase_detail", ""),
|
||||
"proxy_runtime_label": projection.get("proxy_runtime_label", ""),
|
||||
"proxy_runtime_reason": projection.get("proxy_runtime_reason", ""),
|
||||
"active_threads": worker_active_threads,
|
||||
"max_threads": worker_max_threads,
|
||||
"updated_at": _format_time(received_at or datetime.now()),
|
||||
"job_items_total": items_total,
|
||||
"job_items_running": items_running,
|
||||
"job_items_claimed": items_claimed,
|
||||
"job_items_completed": int(node_stat.get("items_completed", 0) or 0),
|
||||
"job_items_failed": int(node_stat.get("items_failed", 0) or 0),
|
||||
"job_items_blacklisted": int(node_stat.get("items_blacklisted", 0) or 0),
|
||||
"metrics_source": str(node_stat.get("metrics_source") or "runtime").strip() or "runtime",
|
||||
"source_status": str(node_stat.get("status") or "").strip(),
|
||||
"source_role": str(node_stat.get("role") or "worker").strip() or "worker",
|
||||
"derived_from": node_code,
|
||||
}
|
||||
register_node_heartbeat(
|
||||
@@ -144,7 +173,9 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv
|
||||
hostname_override=hostname,
|
||||
ip_override=ip,
|
||||
)
|
||||
cleanup_imported_runtime_nodes(region=region, role="worker", keep_node_code=worker_node_code)
|
||||
worker_node_codes.append(worker_node_code)
|
||||
if worker_node_codes:
|
||||
cleanup_imported_runtime_nodes_many(region=region, role="worker", keep_node_codes=worker_node_codes)
|
||||
|
||||
|
||||
def _load_latest_projection(sync_type: str) -> dict | None:
|
||||
@@ -226,6 +257,88 @@ def _load_pushable_projections(sync_type: str, limit: int) -> list[dict]:
|
||||
return selected
|
||||
|
||||
|
||||
def _estimate_total_worker_threads(settings_payload: dict | None = None) -> int:
|
||||
payload = settings_payload if isinstance(settings_payload, dict) else get_settings_payload()
|
||||
default_threads = max(1, int(payload.get("thread_count", 100) or 100))
|
||||
node_thread_counts = payload.get("node_thread_counts") if isinstance(payload.get("node_thread_counts"), dict) else {}
|
||||
total_threads = 0
|
||||
for raw_value in node_thread_counts.values():
|
||||
try:
|
||||
total_threads += max(0, int(raw_value or 0))
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
return max(total_threads, default_threads)
|
||||
|
||||
|
||||
def _load_local_detect_backlog_snapshot() -> dict:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT
|
||||
COUNT(*) FILTER (WHERE item.status = 'pending') AS pending_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'claimed') AS claimed_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'running') AS running_total,
|
||||
COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code = 'detect_register') AS register_pending,
|
||||
COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code <> 'detect_register') AS downstream_pending
|
||||
FROM detect_job_items item
|
||||
JOIN detect_jobs job ON job.id = item.job_id
|
||||
WHERE job.status IN ('pending', 'running')
|
||||
"""
|
||||
)
|
||||
row = cur.fetchone() or (0, 0, 0, 0, 0)
|
||||
return {
|
||||
"pending_total": int(row[0] or 0),
|
||||
"claimed_total": int(row[1] or 0),
|
||||
"running_total": int(row[2] or 0),
|
||||
"register_pending": int(row[3] or 0),
|
||||
"downstream_pending": int(row[4] or 0),
|
||||
}
|
||||
|
||||
|
||||
def _build_task_pull_backlog_limits(configured_limit: int, settings_payload: dict | None = None) -> dict:
|
||||
estimated_total_threads = _estimate_total_worker_threads(settings_payload)
|
||||
max_pending_total = int(settings.sync_pull_max_pending_items or 0)
|
||||
if max_pending_total <= 0:
|
||||
max_pending_total = max(int(configured_limit or 0), estimated_total_threads * 2)
|
||||
|
||||
max_register_pending = int(settings.sync_pull_max_register_pending_items or 0)
|
||||
if max_register_pending <= 0:
|
||||
max_register_pending = max(max(500, int(configured_limit or 0) // 2), estimated_total_threads)
|
||||
|
||||
max_downstream_pending = int(settings.sync_pull_max_downstream_pending_items or 0)
|
||||
if max_downstream_pending <= 0:
|
||||
max_downstream_pending = max(250, estimated_total_threads // 4)
|
||||
|
||||
return {
|
||||
"estimated_total_threads": estimated_total_threads,
|
||||
"max_pending_total": max_pending_total,
|
||||
"max_register_pending": max_register_pending,
|
||||
"max_downstream_pending": max_downstream_pending,
|
||||
}
|
||||
|
||||
|
||||
def _should_throttle_task_pull(backlog_snapshot: dict, backlog_limits: dict) -> tuple[bool, str]:
|
||||
pending_total = int(backlog_snapshot.get("pending_total", 0) or 0)
|
||||
register_pending = int(backlog_snapshot.get("register_pending", 0) or 0)
|
||||
downstream_pending = int(backlog_snapshot.get("downstream_pending", 0) or 0)
|
||||
max_pending_total = int(backlog_limits.get("max_pending_total", 0) or 0)
|
||||
max_register_pending = int(backlog_limits.get("max_register_pending", 0) or 0)
|
||||
max_downstream_pending = int(backlog_limits.get("max_downstream_pending", 0) or 0)
|
||||
|
||||
if max_pending_total > 0 and pending_total >= max_pending_total:
|
||||
return True, "pending_total"
|
||||
if (
|
||||
downstream_pending > 0
|
||||
and max_register_pending > 0
|
||||
and register_pending >= max_register_pending
|
||||
):
|
||||
return True, "register_pending"
|
||||
if max_downstream_pending > 0 and downstream_pending >= max_downstream_pending:
|
||||
return True, "downstream_pending"
|
||||
return False, ""
|
||||
|
||||
|
||||
def _latest_push_attempt(source_record_id: int, target_region: str, sync_type: str) -> dict | None:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
@@ -308,10 +421,51 @@ def _task_selection_sql() -> str:
|
||||
"""
|
||||
|
||||
|
||||
def _task_projection_limit(limit: int | None) -> int:
|
||||
requested = max(1, int(limit or 5000))
|
||||
configured = max(5000, int(settings.sync_batch_size or 200))
|
||||
cap = max(10000, configured, 5000)
|
||||
return max(1, min(requested, cap))
|
||||
|
||||
|
||||
def _task_projection_items_total(projection: dict) -> int:
|
||||
payload = projection.get("payload") or {}
|
||||
projection_payload = payload.get("projection") or {}
|
||||
try:
|
||||
return int(projection_payload.get("items_total", 0) or 0)
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
def _task_projection_selection_limit(projection: dict) -> int:
|
||||
payload = projection.get("payload") or {}
|
||||
projection_payload = payload.get("projection") or {}
|
||||
try:
|
||||
return int(projection_payload.get("selection_limit", 0) or 0)
|
||||
except Exception:
|
||||
return 0
|
||||
|
||||
|
||||
def _mark_task_projection_superseded(record_id: int, *, reason: str) -> None:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
UPDATE detect_sync_records
|
||||
SET status = 'superseded',
|
||||
error_message = %s,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = %s
|
||||
""",
|
||||
(str(reason or "").strip()[:500], int(record_id)),
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
|
||||
def _load_pending_task_projection(limit: int) -> dict | None:
|
||||
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
|
||||
target_region = _normalize_region(settings.sync_target_region, "overseas")
|
||||
safe_limit = max(1, min(int(limit or 1000), max(1, int(settings.sync_batch_size or 200))))
|
||||
safe_limit = _task_projection_limit(limit)
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
@@ -340,6 +494,22 @@ def _load_pending_task_projection(limit: int) -> dict | None:
|
||||
latest_ingest = _latest_ingest_attempt(projection["id"], projection["target_region"], "detect_task_projection")
|
||||
if latest_ingest and latest_ingest["status"] == "received":
|
||||
continue
|
||||
items_total = _task_projection_items_total(projection)
|
||||
selection_limit = _task_projection_selection_limit(projection)
|
||||
created_at = projection.get("created_at")
|
||||
if (
|
||||
safe_limit >= 1000
|
||||
and max(items_total, selection_limit) > 0
|
||||
and max(items_total, selection_limit) < safe_limit
|
||||
and isinstance(created_at, datetime)
|
||||
):
|
||||
now = datetime.now(created_at.tzinfo) if created_at.tzinfo else datetime.now()
|
||||
if now - created_at >= timedelta(minutes=10):
|
||||
_mark_task_projection_superseded(
|
||||
projection["id"],
|
||||
reason=f"stale small task projection skipped: items_total={items_total}, selection_limit={selection_limit}, requested_limit={safe_limit}",
|
||||
)
|
||||
continue
|
||||
return projection
|
||||
return None
|
||||
|
||||
@@ -361,7 +531,7 @@ def export_detect_task_projection(limit: int = 1000, *, shared_token: str | None
|
||||
|
||||
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
|
||||
target_region = _normalize_region(settings.sync_target_region, "overseas")
|
||||
safe_limit = max(1, min(int(limit or 1000), max(1, int(settings.sync_batch_size or 200))))
|
||||
safe_limit = _task_projection_limit(limit)
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(_task_selection_sql(), (safe_limit,))
|
||||
@@ -528,6 +698,7 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N
|
||||
|
||||
inserted_count = 0
|
||||
updated_count = 0
|
||||
domain_ids: list[int] = []
|
||||
for item in items:
|
||||
domain = str(item.get("domain") or "").strip().lower()
|
||||
if not domain:
|
||||
@@ -562,7 +733,7 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N
|
||||
ELSE EXCLUDED.detect_status
|
||||
END,
|
||||
update_time = CURRENT_TIMESTAMP
|
||||
RETURNING (xmax = 0) AS inserted
|
||||
RETURNING id, (xmax = 0) AS inserted
|
||||
""",
|
||||
(
|
||||
domain,
|
||||
@@ -576,12 +747,120 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N
|
||||
int(item.get("juziseo_status") or 0),
|
||||
),
|
||||
)
|
||||
inserted = bool((cur.fetchone() or [False])[0])
|
||||
row = cur.fetchone() or [0, False]
|
||||
domain_id = int(row[0] or 0)
|
||||
inserted = bool(row[1])
|
||||
if domain_id > 0:
|
||||
domain_ids.append(domain_id)
|
||||
if inserted:
|
||||
inserted_count += 1
|
||||
else:
|
||||
updated_count += 1
|
||||
|
||||
target_job_code = f"sync-{source_region}-{source_record_id}"
|
||||
target_job_remark = (
|
||||
f"同步拉取待检测批次 {str(projection.get('batch_code') or '').strip() or source_record_id}，"
|
||||
f"共 {len(domain_ids)} 个域名"
|
||||
)
|
||||
cur.execute(
|
||||
"""
|
||||
INSERT INTO detect_jobs (job_code, source, plan_hash, task_mode, step_code, status, remark, created_by)
|
||||
VALUES (%s, %s, %s, 'domain_pipeline', '', 'pending', %s, %s)
|
||||
ON CONFLICT (job_code) DO UPDATE SET
|
||||
source = EXCLUDED.source,
|
||||
plan_hash = EXCLUDED.plan_hash,
|
||||
task_mode = EXCLUDED.task_mode,
|
||||
step_code = EXCLUDED.step_code,
|
||||
remark = EXCLUDED.remark,
|
||||
created_by = EXCLUDED.created_by,
|
||||
status = CASE
|
||||
WHEN detect_jobs.status IN ('completed', 'failed', 'cancelled') THEN 'pending'
|
||||
ELSE detect_jobs.status
|
||||
END,
|
||||
started_at = CASE
|
||||
WHEN detect_jobs.status IN ('completed', 'failed', 'cancelled') THEN NULL
|
||||
ELSE detect_jobs.started_at
|
||||
END,
|
||||
finished_at = CASE
|
||||
WHEN detect_jobs.status IN ('completed', 'failed', 'cancelled') THEN NULL
|
||||
ELSE detect_jobs.finished_at
|
||||
END
|
||||
RETURNING id
|
||||
""",
|
||||
(
|
||||
target_job_code,
|
||||
"sync-pull",
|
||||
projection_hash,
|
||||
target_job_remark,
|
||||
"sync-agent",
|
||||
),
|
||||
)
|
||||
target_job_id = int((cur.fetchone() or [0])[0] or 0)
|
||||
|
||||
settings_payload = get_settings_payload()
|
||||
queued_count = 0
|
||||
deduplicated_job_items = 0
|
||||
skipped_job_items = 0
|
||||
if target_job_id > 0:
|
||||
for domain_id in domain_ids:
|
||||
domain_snapshot = _load_domain_pipeline_snapshot(cur, int(domain_id))
|
||||
if not domain_snapshot:
|
||||
skipped_job_items += 1
|
||||
continue
|
||||
item_step_code, step_payload = resolve_initial_domain_pipeline_item(
|
||||
domain_snapshot,
|
||||
settings_payload=settings_payload,
|
||||
)
|
||||
if not item_step_code or not step_payload:
|
||||
skipped_job_items += 1
|
||||
continue
|
||||
cur.execute(
|
||||
"""
|
||||
INSERT INTO detect_job_items (job_id, domain_id, step_code, status, step_payload_json)
|
||||
VALUES (%s, %s, %s, 'pending', %s::jsonb)
|
||||
ON CONFLICT (job_id, domain_id, step_code) DO NOTHING
|
||||
RETURNING id
|
||||
""",
|
||||
(
|
||||
target_job_id,
|
||||
domain_id,
|
||||
item_step_code,
|
||||
json.dumps(step_payload, ensure_ascii=False),
|
||||
),
|
||||
)
|
||||
inserted_job_item = cur.fetchone()
|
||||
if inserted_job_item:
|
||||
queued_count += 1
|
||||
else:
|
||||
deduplicated_job_items += 1
|
||||
|
||||
if queued_count > 0:
|
||||
cur.execute(
|
||||
"""
|
||||
INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json)
|
||||
VALUES (%s, %s, %s, %s, %s, %s::jsonb)
|
||||
""",
|
||||
(
|
||||
target_job_id,
|
||||
settings.node_code,
|
||||
"job_created",
|
||||
"info",
|
||||
f"同步拉取待检测批次 {target_job_code}，共 {queued_count} 个任务项",
|
||||
json.dumps(
|
||||
{
|
||||
"source_region": source_region,
|
||||
"source_record_id": source_record_id,
|
||||
"projection_hash": projection_hash,
|
||||
"batch_code": str(projection.get("batch_code") or "").strip(),
|
||||
"queued_count": queued_count,
|
||||
"deduplicated_job_items": deduplicated_job_items,
|
||||
"skipped_job_items": skipped_job_items,
|
||||
},
|
||||
ensure_ascii=False,
|
||||
),
|
||||
),
|
||||
)
|
||||
|
||||
cur.execute(
|
||||
"""
|
||||
INSERT INTO detect_sync_records (
|
||||
@@ -602,6 +881,11 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N
|
||||
"items_total": len(items),
|
||||
"inserted_count": inserted_count,
|
||||
"updated_count": updated_count,
|
||||
"target_job_id": target_job_id,
|
||||
"target_job_code": target_job_code,
|
||||
"queued_count": queued_count,
|
||||
"deduplicated_job_items": deduplicated_job_items,
|
||||
"skipped_job_items": skipped_job_items,
|
||||
"received_at": _format_time(received_at),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
@@ -617,6 +901,11 @@ def ingest_detect_task_projection(payload: dict, *, shared_token: str | None = N
|
||||
"items_total": len(items),
|
||||
"inserted_count": inserted_count,
|
||||
"updated_count": updated_count,
|
||||
"target_job_id": target_job_id,
|
||||
"target_job_code": target_job_code,
|
||||
"queued_count": queued_count,
|
||||
"deduplicated_job_items": deduplicated_job_items,
|
||||
"skipped_job_items": skipped_job_items,
|
||||
"deduplicated": False,
|
||||
}
|
||||
|
||||
@@ -742,7 +1031,27 @@ def _extract_detect_result_projection_events(
|
||||
return events
|
||||
|
||||
|
||||
def _resolve_detect_result_target_job_id() -> int:
|
||||
def _resolve_detect_result_target_job_id(*, projection: dict) -> int:
|
||||
source_job = projection.get("job") or {}
|
||||
source_job_code = str(source_job.get("job_code") or "").strip()
|
||||
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
if source_job_code:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT id
|
||||
FROM detect_jobs
|
||||
WHERE job_code = %s
|
||||
ORDER BY id DESC
|
||||
LIMIT 1
|
||||
""",
|
||||
(source_job_code,),
|
||||
)
|
||||
row = cur.fetchone()
|
||||
if row:
|
||||
return int(row[0] or 0)
|
||||
|
||||
from app.services.detect_job_service import get_active_detect_job_summary
|
||||
|
||||
active_job = get_active_detect_job_summary(event_limit=1) or {}
|
||||
@@ -759,6 +1068,163 @@ def _parse_event_created_at(value: str) -> datetime | None:
|
||||
return None
|
||||
|
||||
|
||||
def _extract_event_domain(payload: dict, message: str) -> str:
|
||||
domain = str(payload.get("domain") or "").strip().lower()
|
||||
if domain:
|
||||
return domain
|
||||
text = str(message or "").strip()
|
||||
if ":" in text:
|
||||
candidate = text.rsplit(":", 1)[-1].strip().lower()
|
||||
if candidate:
|
||||
return candidate
|
||||
return ""
|
||||
|
||||
|
||||
def _apply_detect_result_event_to_domain(cur, event: dict) -> None:
|
||||
payload = _decode_json(event.get("payload"))
|
||||
domain = _extract_event_domain(payload, str(event.get("message") or ""))
|
||||
if not domain:
|
||||
return
|
||||
|
||||
event_type = str(event.get("event_type") or "").strip()
|
||||
created_at = _parse_event_created_at(str(event.get("created_at") or ""))
|
||||
effective_time = created_at or datetime.now()
|
||||
|
||||
if event_type == "domain_started":
|
||||
cur.execute(
|
||||
"""
|
||||
UPDATE domains
|
||||
SET detect_status = CASE
|
||||
WHEN detect_status IN (1, 3) THEN detect_status
|
||||
ELSE 2
|
||||
END,
|
||||
update_time = CURRENT_TIMESTAMP
|
||||
WHERE domain = %s
|
||||
""",
|
||||
(domain,),
|
||||
)
|
||||
return
|
||||
|
||||
if event_type == "domain_completed":
|
||||
cur.execute(
|
||||
"""
|
||||
UPDATE domains
|
||||
SET detect_status = 1,
|
||||
detect_time = COALESCE(detect_time, %s),
|
||||
update_time = CURRENT_TIMESTAMP
|
||||
WHERE domain = %s
|
||||
""",
|
||||
(effective_time, domain),
|
||||
)
|
||||
return
|
||||
|
||||
if event_type == "domain_blacklisted":
|
||||
cur.execute(
|
||||
"""
|
||||
UPDATE domains
|
||||
SET detect_status = 3,
|
||||
update_time = CURRENT_TIMESTAMP
|
||||
WHERE domain = %s
|
||||
""",
|
||||
(domain,),
|
||||
)
|
||||
return
|
||||
|
||||
if event_type == "domain_failed":
|
||||
cur.execute(
|
||||
"""
|
||||
UPDATE domains
|
||||
SET detect_status = CASE
|
||||
WHEN detect_status IN (1, 3) THEN detect_status
|
||||
ELSE 4
|
||||
END,
|
||||
update_time = CURRENT_TIMESTAMP
|
||||
WHERE domain = %s
|
||||
""",
|
||||
(domain,),
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
def _apply_detect_result_event_to_job_item(cur, *, target_job_id: int, event: dict) -> int:
|
||||
if int(target_job_id or 0) <= 0:
|
||||
return 0
|
||||
|
||||
payload = _decode_json(event.get("payload"))
|
||||
domain = _extract_event_domain(payload, str(event.get("message") or ""))
|
||||
if not domain:
|
||||
return 0
|
||||
|
||||
event_type = str(event.get("event_type") or "").strip()
|
||||
node_code = str(event.get("node_code") or "").strip()
|
||||
message = str(event.get("message") or "").strip()
|
||||
|
||||
if event_type == "domain_started":
|
||||
cur.execute(
|
||||
"""
|
||||
UPDATE detect_job_items AS item
|
||||
SET status = CASE
|
||||
WHEN item.status IN ('completed', 'blacklisted', 'failed') THEN item.status
|
||||
ELSE 'running'
|
||||
END,
|
||||
claimed_by = CASE
|
||||
WHEN %s <> '' THEN %s
|
||||
ELSE item.claimed_by
|
||||
END,
|
||||
started_at = COALESCE(item.started_at, CURRENT_TIMESTAMP),
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
FROM domains AS d
|
||||
WHERE item.job_id = %s
|
||||
AND item.domain_id = d.id
|
||||
AND d.domain = %s
|
||||
AND item.status IN ('pending', 'claimed', 'running')
|
||||
""",
|
||||
(node_code, node_code, int(target_job_id), domain),
|
||||
)
|
||||
return int(cur.rowcount or 0)
|
||||
|
||||
if event_type not in {"domain_completed", "domain_blacklisted", "domain_failed"}:
|
||||
return 0
|
||||
|
||||
final_status = {
|
||||
"domain_completed": "completed",
|
||||
"domain_blacklisted": "blacklisted",
|
||||
"domain_failed": "failed",
|
||||
}[event_type]
|
||||
cur.execute(
|
||||
"""
|
||||
UPDATE detect_job_items AS item
|
||||
SET status = %s,
|
||||
claimed_by = CASE
|
||||
WHEN %s <> '' THEN %s
|
||||
ELSE item.claimed_by
|
||||
END,
|
||||
finished_at = COALESCE(item.finished_at, CURRENT_TIMESTAMP),
|
||||
updated_at = CURRENT_TIMESTAMP,
|
||||
lease_expires_at = NULL,
|
||||
last_error = CASE
|
||||
WHEN %s = 'failed' THEN LEFT(%s, 1000)
|
||||
ELSE item.last_error
|
||||
END
|
||||
FROM domains AS d
|
||||
WHERE item.job_id = %s
|
||||
AND item.domain_id = d.id
|
||||
AND d.domain = %s
|
||||
AND item.status IN ('pending', 'claimed', 'running')
|
||||
""",
|
||||
(
|
||||
final_status,
|
||||
node_code,
|
||||
node_code,
|
||||
final_status,
|
||||
message,
|
||||
int(target_job_id),
|
||||
domain,
|
||||
),
|
||||
)
|
||||
return int(cur.rowcount or 0)
|
||||
|
||||
|
||||
def _import_detect_result_projection_events(
|
||||
*,
|
||||
source_region: str,
|
||||
@@ -773,12 +1239,11 @@ def _import_detect_result_projection_events(
|
||||
if not events:
|
||||
return {"imported_count": 0, "deduplicated_count": 0, "target_job_id": 0}
|
||||
|
||||
target_job_id = _resolve_detect_result_target_job_id()
|
||||
if target_job_id <= 0:
|
||||
return {"imported_count": 0, "deduplicated_count": 0, "target_job_id": 0}
|
||||
target_job_id = _resolve_detect_result_target_job_id(projection=projection)
|
||||
|
||||
imported_count = 0
|
||||
deduplicated_count = 0
|
||||
updated_job_items = 0
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
for event in events:
|
||||
@@ -831,12 +1296,27 @@ def _import_detect_result_projection_events(
|
||||
json.dumps(event["payload"], ensure_ascii=False),
|
||||
),
|
||||
)
|
||||
_apply_detect_result_event_to_domain(cur, event)
|
||||
if target_job_id > 0:
|
||||
updated_job_items += _apply_detect_result_event_to_job_item(
|
||||
cur,
|
||||
target_job_id=target_job_id,
|
||||
event=event,
|
||||
)
|
||||
imported_count += 1
|
||||
conn.commit()
|
||||
if target_job_id > 0:
|
||||
from app.services.detect_job_service import refresh_detect_job_status
|
||||
|
||||
try:
|
||||
refresh_detect_job_status(target_job_id)
|
||||
except Exception:
|
||||
pass
|
||||
return {
|
||||
"imported_count": imported_count,
|
||||
"deduplicated_count": deduplicated_count,
|
||||
"target_job_id": target_job_id,
|
||||
"updated_job_items": updated_job_items,
|
||||
}
|
||||
|
||||
|
||||
@@ -941,6 +1421,16 @@ def ingest_runtime_projection(payload: dict, *, shared_token: str | None = None)
|
||||
|
||||
|
||||
def _push_projection_now(sync_type: str, ingest_url: str) -> tuple[bool, str, dict]:
|
||||
if sync_type == "runtime_projection":
|
||||
# Regenerate the runtime snapshot before every push so the sync agent
|
||||
# does not keep replaying a stale projection record while the worker
|
||||
# thread count / phase is still changing.
|
||||
from app.services.runtime_status_service import get_runtime_status
|
||||
|
||||
try:
|
||||
get_runtime_status()
|
||||
except Exception as exc:
|
||||
return False, f"刷新 runtime_projection 失败: {exc}", {"action": "push_sync", "sync_type": sync_type}
|
||||
source_record = _load_latest_projection(sync_type)
|
||||
if not source_record:
|
||||
return False, f"当前没有可推送的{sync_type}", {"action": "push_sync", "sync_type": sync_type}
|
||||
@@ -1177,15 +1667,33 @@ def pull_detect_task_batch_now(limit: int | None = None) -> tuple[bool, str, dic
|
||||
if not export_url or not ack_url:
|
||||
return False, "未配置任务拉取目标地址", {"action": "pull_tasks", "pull_state": "misconfigured", "ui_level": "warning", "poll_schedule_seconds": []}
|
||||
|
||||
safe_limit = max(1, min(int(limit or settings.sync_batch_size or 200), max(1, int(settings.sync_batch_size or 200))))
|
||||
configured_limit = max(5000, int(settings.sync_batch_size or 200))
|
||||
requested_limit = int(limit or configured_limit)
|
||||
safe_limit = max(1, min(requested_limit, max(10000, configured_limit)))
|
||||
settings_payload = get_settings_payload()
|
||||
backlog_snapshot = _load_local_detect_backlog_snapshot()
|
||||
backlog_limits = _build_task_pull_backlog_limits(configured_limit, settings_payload=settings_payload)
|
||||
should_throttle, throttle_reason = _should_throttle_task_pull(backlog_snapshot, backlog_limits)
|
||||
if should_throttle:
|
||||
return True, "本地待处理积压较高，暂停拉取新批次", {
|
||||
"action": "pull_tasks",
|
||||
"pull_state": "throttled",
|
||||
"ui_level": "info",
|
||||
"poll_schedule_seconds": [1, 3],
|
||||
"reason": throttle_reason,
|
||||
**backlog_snapshot,
|
||||
**backlog_limits,
|
||||
}
|
||||
|
||||
request_url = f"{export_url}?limit={safe_limit}"
|
||||
export_timeout = max(20, min(90, 15 + safe_limit // 40))
|
||||
request = urllib.request.Request(
|
||||
request_url,
|
||||
headers={**({"X-Domaincheck-Sync-Token": settings.sync_shared_token} if settings.sync_shared_token else {})},
|
||||
method="GET",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(request, timeout=20) as response:
|
||||
with urllib.request.urlopen(request, timeout=export_timeout) as response:
|
||||
raw = response.read().decode("utf-8")
|
||||
data = json.loads(raw) if raw else {}
|
||||
except json.JSONDecodeError as exc:
|
||||
@@ -1280,7 +1788,7 @@ def pull_detect_task_batch_now(limit: int | None = None) -> tuple[bool, str, dic
|
||||
**(ingest_data or {}),
|
||||
}
|
||||
|
||||
return True, "待检测任务批次拉取并入库成功", {
|
||||
result = {
|
||||
"action": "pull_tasks",
|
||||
"pull_state": "success",
|
||||
"ui_level": "success",
|
||||
@@ -1291,3 +1799,27 @@ def pull_detect_task_batch_now(limit: int | None = None) -> tuple[bool, str, dic
|
||||
**(ingest_data or {}),
|
||||
"ack": ack_data,
|
||||
}
|
||||
queued_count = int(result.get("queued_count", 0) or 0)
|
||||
if queued_count > 0:
|
||||
try:
|
||||
from app.services.worker_control_service import send_worker_command
|
||||
|
||||
start_ok, start_message = send_worker_command(
|
||||
"start_detection",
|
||||
payload={
|
||||
"source": "sync-pull",
|
||||
"source_record_id": source_record_id,
|
||||
"target_job_id": int(result.get("target_job_id", 0) or 0),
|
||||
"target_job_code": str(result.get("target_job_code") or "").strip(),
|
||||
},
|
||||
)
|
||||
result["worker_start_ok"] = bool(start_ok)
|
||||
result["worker_start_message"] = str(start_message or "").strip()
|
||||
except Exception as exc:
|
||||
result["worker_start_ok"] = False
|
||||
result["worker_start_message"] = f"同步入库后自动唤起 Worker 失败: {exc}"
|
||||
result["ui_level"] = "warning"
|
||||
result["pull_state"] = "worker_start_warning"
|
||||
return True, "待检测任务批次拉取并入库成功；但自动唤起 Worker 失败", result
|
||||
|
||||
return True, "待检测任务批次拉取并入库成功", result
|
||||
|
||||
@@ -6,7 +6,7 @@ import socket
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from app.core.config import settings
|
||||
from app.core.db import get_db
|
||||
from app.core.db import db_read_retry, get_db
|
||||
|
||||
|
||||
def _format_time(value: datetime | None) -> str:
|
||||
@@ -45,28 +45,94 @@ _DETECT_RESULT_EVENT_TYPES = {
|
||||
"domain_blacklisted",
|
||||
}
|
||||
|
||||
_TERMINAL_DETECT_RESULT_EVENT_TYPES = {
|
||||
"domain_completed",
|
||||
"domain_failed",
|
||||
"domain_blacklisted",
|
||||
}
|
||||
|
||||
|
||||
def _collect_recent_domain_events(active_job: dict, limit: int = 30) -> list[dict]:
|
||||
events = list(active_job.get("current_cycle_events") or active_job.get("recent_events") or [])
|
||||
safe_limit = max(1, int(limit or 30))
|
||||
seen: set[tuple[str, str, str, str]] = set()
|
||||
normalized: list[dict] = []
|
||||
for event in reversed(events):
|
||||
event_type = str(event.get("event_type") or "").strip()
|
||||
|
||||
def _append_event(raw_event: dict) -> None:
|
||||
event_type = str(raw_event.get("event_type") or "").strip()
|
||||
if event_type not in _DETECT_RESULT_EVENT_TYPES:
|
||||
continue
|
||||
payload = _decode_json(event.get("payload"))
|
||||
normalized.append(
|
||||
{
|
||||
"node_code": str(event.get("node_code") or "").strip(),
|
||||
"event_type": event_type,
|
||||
"level": str(event.get("level") or "info").strip() or "info",
|
||||
"message": str(event.get("message") or "").strip(),
|
||||
"created_at": str(event.get("created_at") or "").strip(),
|
||||
"payload": payload,
|
||||
}
|
||||
return
|
||||
normalized_event = {
|
||||
"node_code": str(raw_event.get("node_code") or "").strip(),
|
||||
"event_type": event_type,
|
||||
"level": str(raw_event.get("level") or "info").strip() or "info",
|
||||
"message": str(raw_event.get("message") or "").strip(),
|
||||
"created_at": str(raw_event.get("created_at") or "").strip(),
|
||||
"payload": _decode_json(raw_event.get("payload")),
|
||||
}
|
||||
event_key = (
|
||||
normalized_event["node_code"],
|
||||
normalized_event["event_type"],
|
||||
normalized_event["message"],
|
||||
normalized_event["created_at"],
|
||||
)
|
||||
if limit <= 0:
|
||||
return normalized
|
||||
return normalized[-int(limit):]
|
||||
if event_key in seen:
|
||||
return
|
||||
seen.add(event_key)
|
||||
normalized.append(normalized_event)
|
||||
|
||||
# Keep a small slice of the current-cycle `domain_started` events so the
|
||||
# remote log / live activity view still reflects the node's latest work.
|
||||
for event in reversed(list(active_job.get("current_cycle_events") or active_job.get("recent_events") or [])):
|
||||
if str(event.get("event_type") or "").strip() != "domain_started":
|
||||
continue
|
||||
_append_event(event)
|
||||
if len(normalized) >= min(10, max(1, safe_limit // 3)):
|
||||
break
|
||||
|
||||
# Always pull the most recent terminal result events from the full job
|
||||
# history. Otherwise a flood of newer `domain_started` events can hide
|
||||
# terminal completions, and overseas will never advance completed counts.
|
||||
job_id = int(active_job.get("job_id") or 0)
|
||||
if job_id > 0:
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT node_code, event_type, level, message, payload_json, created_at
|
||||
FROM detect_run_events
|
||||
WHERE job_id = %s
|
||||
AND event_type IN ('domain_completed', 'domain_failed', 'domain_blacklisted')
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT %s
|
||||
""",
|
||||
(job_id, max(safe_limit * 4, 60)),
|
||||
)
|
||||
rows = cur.fetchall()
|
||||
for row in reversed(rows):
|
||||
_append_event(
|
||||
{
|
||||
"node_code": row[0] or "",
|
||||
"event_type": row[1] or "",
|
||||
"level": row[2] or "info",
|
||||
"message": row[3] or "",
|
||||
"payload": _decode_json(row[4]),
|
||||
"created_at": _format_time(row[5]),
|
||||
}
|
||||
)
|
||||
else:
|
||||
for event in reversed(list(active_job.get("current_cycle_events") or active_job.get("recent_events") or [])):
|
||||
if str(event.get("event_type") or "").strip() in _TERMINAL_DETECT_RESULT_EVENT_TYPES:
|
||||
_append_event(event)
|
||||
|
||||
normalized.sort(
|
||||
key=lambda item: (
|
||||
str(item.get("created_at") or ""),
|
||||
str(item.get("node_code") or ""),
|
||||
str(item.get("event_type") or ""),
|
||||
str(item.get("message") or ""),
|
||||
)
|
||||
)
|
||||
return normalized[-safe_limit:]
|
||||
|
||||
|
||||
def _build_detect_result_batch_digest(batch: dict | None) -> dict:
|
||||
@@ -154,6 +220,7 @@ def _should_append_runtime_projection(previous_payload: dict, current_projection
|
||||
return now - previous_created_at >= timedelta(seconds=45)
|
||||
|
||||
|
||||
@db_read_retry()
|
||||
def list_sync_records(limit: int = 20) -> list[dict]:
|
||||
safe_limit = max(1, min(int(limit or 20), 200))
|
||||
with get_db() as conn:
|
||||
@@ -221,6 +288,7 @@ def _latest_sync_record_by_source(
|
||||
}
|
||||
|
||||
|
||||
@db_read_retry()
|
||||
def get_detect_result_sync_batches(limit: int = 5) -> dict:
|
||||
safe_limit = max(1, min(int(limit or 5), 20))
|
||||
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
|
||||
@@ -364,6 +432,7 @@ def get_detect_result_sync_batches(limit: int = 5) -> dict:
|
||||
}
|
||||
|
||||
|
||||
@db_read_retry()
|
||||
def get_sync_summary(record_limit: int = 10) -> dict:
|
||||
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
|
||||
target_region = _normalize_region(settings.sync_target_region, "overseas")
|
||||
@@ -465,56 +534,43 @@ def append_sync_record(
|
||||
return record_id
|
||||
|
||||
|
||||
def append_runtime_projection_if_changed(
|
||||
def _local_node_expected_to_execute_worker() -> bool:
|
||||
node_role = str(settings.node_role or "").strip()
|
||||
node_region = str(settings.node_region or "").strip()
|
||||
return node_role == "worker" or (node_region == "mainland" and node_role == "control")
|
||||
|
||||
|
||||
def _build_runtime_projection_payload(
|
||||
*,
|
||||
detect: dict,
|
||||
cluster: dict,
|
||||
source_region: str | None = None,
|
||||
target_region: str | None = None,
|
||||
) -> int | None:
|
||||
normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region))
|
||||
normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas"))
|
||||
source_region: str,
|
||||
target_region: str,
|
||||
) -> dict:
|
||||
active_job = detect.get("active_job") or {}
|
||||
local_worker_expected = _local_node_expected_to_execute_worker()
|
||||
local_participating = False
|
||||
for node in list(cluster.get("nodes") or []):
|
||||
if str(node.get("node_code") or "").strip() != settings.node_code:
|
||||
continue
|
||||
local_participating = bool(node.get("detect_participating", False) or node.get("current_load", 0))
|
||||
if local_worker_expected:
|
||||
local_participating = bool(node.get("detect_participating", False) or node.get("current_load", 0))
|
||||
break
|
||||
local_job_bucket = {}
|
||||
for item in list(active_job.get("node_stats") or []):
|
||||
if str(item.get("node_code") or "").strip() != settings.node_code:
|
||||
continue
|
||||
local_job_bucket = item
|
||||
break
|
||||
if not local_participating:
|
||||
if local_worker_expected:
|
||||
for item in list(active_job.get("node_stats") or []):
|
||||
if str(item.get("node_code") or "").strip() != settings.node_code:
|
||||
continue
|
||||
local_job_bucket = item
|
||||
break
|
||||
if local_worker_expected and not local_participating:
|
||||
local_participating = bool(
|
||||
int(local_job_bucket.get("items_running", 0) or 0) > 0
|
||||
or int(local_job_bucket.get("items_claimed", 0) or 0) > 0
|
||||
)
|
||||
projection = {
|
||||
"node": {
|
||||
"node_code": settings.node_code,
|
||||
"region": settings.node_region,
|
||||
"role": settings.node_role,
|
||||
"hostname": socket.gethostname(),
|
||||
"ip": _resolve_local_ip(),
|
||||
},
|
||||
"worker_online": bool(detect.get("worker_online", False)),
|
||||
"detect_participating": local_participating,
|
||||
"worker_mode": detect.get("worker_mode", ""),
|
||||
"phase_label": detect.get("phase_label", ""),
|
||||
"phase_detail": detect.get("phase_detail", ""),
|
||||
"proxy_runtime_label": detect.get("proxy_runtime_label", ""),
|
||||
"proxy_runtime_reason": detect.get("proxy_runtime_reason", ""),
|
||||
"progress": {
|
||||
"pending": int((detect.get("progress") or {}).get("pending", 0) or 0),
|
||||
"running": int((detect.get("progress") or {}).get("running", 0) or 0),
|
||||
"completed": int((detect.get("progress") or {}).get("completed", 0) or 0),
|
||||
"blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0),
|
||||
"failed": int((detect.get("progress") or {}).get("failed", 0) or 0),
|
||||
},
|
||||
"active_job": {
|
||||
|
||||
projection_active_job = (
|
||||
{
|
||||
"job_id": active_job.get("job_id"),
|
||||
"job_code": active_job.get("job_code", ""),
|
||||
"status": active_job.get("status", ""),
|
||||
@@ -525,7 +581,58 @@ def append_runtime_projection_if_changed(
|
||||
"items_running": active_job.get("items_running", 0),
|
||||
"items_failed": active_job.get("items_failed", 0),
|
||||
"node_stats": list(active_job.get("node_stats") or []),
|
||||
}
|
||||
if local_worker_expected
|
||||
else {
|
||||
"job_id": None,
|
||||
"job_code": "",
|
||||
"status": "",
|
||||
"progress_percent": 0,
|
||||
"items_total": 0,
|
||||
"items_terminal": 0,
|
||||
"items_pending": 0,
|
||||
"items_running": 0,
|
||||
"items_failed": 0,
|
||||
"node_stats": [],
|
||||
}
|
||||
)
|
||||
progress_payload = (
|
||||
{
|
||||
"pending": int((detect.get("progress") or {}).get("pending", 0) or 0),
|
||||
"running": int((detect.get("progress") or {}).get("running", 0) or 0),
|
||||
"completed": int((detect.get("progress") or {}).get("completed", 0) or 0),
|
||||
"blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0),
|
||||
"failed": int((detect.get("progress") or {}).get("failed", 0) or 0),
|
||||
}
|
||||
if local_worker_expected
|
||||
else {
|
||||
"pending": 0,
|
||||
"running": 0,
|
||||
"completed": 0,
|
||||
"blacklisted": 0,
|
||||
"failed": 0,
|
||||
}
|
||||
)
|
||||
projection = {
|
||||
"node": {
|
||||
"node_code": settings.node_code,
|
||||
"region": settings.node_region,
|
||||
"role": settings.node_role,
|
||||
"hostname": socket.gethostname(),
|
||||
"ip": _resolve_local_ip(),
|
||||
},
|
||||
"worker_online": bool(detect.get("worker_online", False)) if local_worker_expected else False,
|
||||
"detect_participating": local_participating if local_worker_expected else False,
|
||||
"worker_mode": detect.get("worker_mode", ""),
|
||||
"active_thread_count": int(detect.get("active_thread_count", 0) or 0) if local_worker_expected else 0,
|
||||
"max_thread_count": int(detect.get("max_thread_count", 0) or 0) if local_worker_expected else 0,
|
||||
"phase_label": detect.get("phase_label", ""),
|
||||
"phase_detail": detect.get("phase_detail", ""),
|
||||
"proxy_runtime_label": detect.get("proxy_runtime_label", ""),
|
||||
"proxy_runtime_reason": detect.get("proxy_runtime_reason", ""),
|
||||
"progress": progress_payload,
|
||||
"backlog": dict(detect.get("backlog") or {}) if local_worker_expected else {},
|
||||
"active_job": projection_active_job,
|
||||
"cluster_summary": {
|
||||
"nodes_total": int(cluster.get("nodes_total", 0) or 0),
|
||||
"online_worker_nodes": int((cluster.get("summary") or {}).get("online_worker_nodes", 0) or 0),
|
||||
@@ -544,13 +651,32 @@ def append_runtime_projection_if_changed(
|
||||
for item in (detect.get("dependency_alerts") or [])[:3]
|
||||
],
|
||||
}
|
||||
payload = {
|
||||
return {
|
||||
"projection": projection,
|
||||
"projection_hash": hashlib.sha1(
|
||||
json.dumps(projection, ensure_ascii=False, sort_keys=True).encode("utf-8")
|
||||
).hexdigest(),
|
||||
"source_region": source_region,
|
||||
"target_region": target_region,
|
||||
}
|
||||
|
||||
|
||||
def append_runtime_projection_if_changed(
|
||||
*,
|
||||
detect: dict,
|
||||
cluster: dict,
|
||||
source_region: str | None = None,
|
||||
target_region: str | None = None,
|
||||
) -> int | None:
|
||||
normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region))
|
||||
normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas"))
|
||||
payload = _build_runtime_projection_payload(
|
||||
detect=detect,
|
||||
cluster=cluster,
|
||||
source_region=normalized_source_region,
|
||||
target_region=normalized_target_region,
|
||||
)
|
||||
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
@@ -570,7 +696,7 @@ def append_runtime_projection_if_changed(
|
||||
latest_created_at = latest[1] if latest else None
|
||||
if latest_payload.get("projection_hash") == payload["projection_hash"]:
|
||||
return None
|
||||
if not _should_append_runtime_projection(latest_payload, projection, latest_created_at):
|
||||
if not _should_append_runtime_projection(latest_payload, payload["projection"], latest_created_at):
|
||||
return None
|
||||
cur.execute(
|
||||
"""
|
||||
@@ -650,6 +776,7 @@ def append_detect_result_projection_if_changed(
|
||||
|
||||
with get_db() as conn:
|
||||
with conn.cursor() as cur:
|
||||
current_job_id = int((projection.get("job") or {}).get("job_id") or 0)
|
||||
cur.execute(
|
||||
"""
|
||||
SELECT payload_json, created_at
|
||||
@@ -657,10 +784,19 @@ def append_detect_result_projection_if_changed(
|
||||
WHERE sync_type = 'detect_result_projection'
|
||||
AND source_region = %s
|
||||
AND target_region = %s
|
||||
AND (
|
||||
%s <= 0
|
||||
OR (payload_json->'projection'->'job'->>'job_id') = %s
|
||||
)
|
||||
ORDER BY created_at DESC, id DESC
|
||||
LIMIT 1
|
||||
""",
|
||||
(normalized_source_region, normalized_target_region),
|
||||
(
|
||||
normalized_source_region,
|
||||
normalized_target_region,
|
||||
current_job_id,
|
||||
str(current_job_id),
|
||||
),
|
||||
)
|
||||
latest = cur.fetchone()
|
||||
latest_payload = _decode_json(latest[0]) if latest else {}
|
||||
|
||||
@@ -60,7 +60,12 @@ def normalize_systemctl_error(raw_message: str, *, service_name: str = "") -> st
|
||||
return f"{target} 控制失败，未返回可用错误信息"
|
||||
|
||||
lowered = message.lower()
|
||||
if "sudo: a password is required" in lowered or "authentication is required" in lowered:
|
||||
if (
|
||||
"sudo: a password is required" in lowered
|
||||
or "authentication is required" in lowered
|
||||
or "interactive authentication required" in lowered
|
||||
or "authorization not available" in lowered
|
||||
):
|
||||
target = normalized_service_name or "systemd 服务"
|
||||
return f"{target} 控制失败：当前运行用户没有免密 systemctl 权限，请为 API 进程授予对应 sudo/systemd 权限"
|
||||
if "unit " in lowered and " could not be found" in lowered:
|
||||
|
||||
@@ -5,13 +5,34 @@ import time
|
||||
|
||||
from app.core.config import settings
|
||||
from app.services.debug_event_service import push_debug_event
|
||||
from app.services.detect_job_service import get_active_detect_job_summary, get_detect_queue_health, list_recent_detect_run_events
|
||||
from app.services.detect_job_service import (
|
||||
get_active_detect_job_summary,
|
||||
get_detect_queue_health,
|
||||
get_latest_detect_job_summary,
|
||||
get_latest_unprojected_detect_job_summary,
|
||||
list_recent_detect_run_events,
|
||||
process_detect_pipeline_now,
|
||||
)
|
||||
from app.services.sync_record_service import append_detect_result_projection_if_changed
|
||||
from app.services.sync_push_service import pull_detect_task_batch_now, push_runtime_projection_now
|
||||
from app.services.sync_push_service import (
|
||||
_load_local_detect_backlog_snapshot,
|
||||
pull_detect_task_batch_now,
|
||||
push_runtime_projection_now,
|
||||
)
|
||||
|
||||
|
||||
logger = logging.getLogger("domaincheck.sync_agent")
|
||||
|
||||
_IDLE_SYNC_KEYWORDS = (
|
||||
"当前没有可推送",
|
||||
"当前没有需要立即推送",
|
||||
"已全部同步完成",
|
||||
"无需重复发送",
|
||||
"进行中",
|
||||
"等待下个重试窗口",
|
||||
"暂停拉取",
|
||||
)
|
||||
|
||||
|
||||
def _append_detect_result_projection_snapshot(active_job: dict) -> None:
|
||||
if not active_job:
|
||||
@@ -32,6 +53,159 @@ def _append_detect_result_projection_snapshot(active_job: dict) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _is_idle_sync_message(message: str) -> bool:
|
||||
normalized = str(message or "").strip()
|
||||
return any(keyword in normalized for keyword in _IDLE_SYNC_KEYWORDS)
|
||||
|
||||
|
||||
def _filter_runtime_events_for_job(events: list[dict], *, job_code: str = "", job_id: int = 0, limit: int = 8) -> list[dict]:
|
||||
target_job_code = str(job_code or "").strip()
|
||||
target_job_id = int(job_id or 0)
|
||||
safe_limit = max(1, min(int(limit or 8), 50))
|
||||
filtered: list[dict] = []
|
||||
for raw_event in list(events or []):
|
||||
if not isinstance(raw_event, dict):
|
||||
continue
|
||||
payload = raw_event.get("payload") if isinstance(raw_event.get("payload"), dict) else {}
|
||||
event_job_code = str(payload.get("job_code") or "").strip()
|
||||
event_job_id = int(raw_event.get("job_id") or 0)
|
||||
if target_job_code and event_job_code != target_job_code and (target_job_id <= 0 or event_job_id != target_job_id):
|
||||
continue
|
||||
filtered.append(raw_event)
|
||||
if len(filtered) >= safe_limit:
|
||||
break
|
||||
return filtered
|
||||
|
||||
|
||||
def _build_aligned_queue_health_snapshot(active_job: dict, queue_health: dict | None) -> dict:
|
||||
snapshot = dict(queue_health or {})
|
||||
if not active_job:
|
||||
return snapshot
|
||||
|
||||
active_job_code = str(active_job.get("runtime_job_code") or active_job.get("job_code") or "").strip()
|
||||
queue_job = dict(snapshot.get("job") or {})
|
||||
queue_job_code = str(queue_job.get("runtime_job_code") or queue_job.get("job_code") or "").strip()
|
||||
|
||||
if active_job_code and queue_job_code and active_job_code == queue_job_code:
|
||||
return snapshot
|
||||
|
||||
active_job_items_total = int(active_job.get("items_total", 0) or 0)
|
||||
active_job_pending = int(active_job.get("items_pending", 0) or 0)
|
||||
active_job_claimed = int(active_job.get("items_claimed", 0) or 0)
|
||||
active_job_running = int(active_job.get("items_running", 0) or 0)
|
||||
active_job_completed = int(active_job.get("items_completed", 0) or 0)
|
||||
active_job_blacklisted = int(active_job.get("items_blacklisted", 0) or 0)
|
||||
active_job_failed = int(active_job.get("items_failed", 0) or 0)
|
||||
active_job_terminal = int(
|
||||
active_job.get("items_terminal", active_job_completed + active_job_blacklisted + active_job_failed) or 0
|
||||
)
|
||||
display_claimed = int(active_job.get("display_items_claimed", active_job_claimed) or active_job_claimed)
|
||||
display_running = int(active_job.get("display_items_running", active_job_running) or active_job_running)
|
||||
|
||||
node_entries: list[dict] = []
|
||||
for node in list(active_job.get("node_stats") or []):
|
||||
node_entries.append(
|
||||
{
|
||||
"node_code": str(node.get("node_code") or "").strip(),
|
||||
"items_total": int(node.get("items_total", 0) or 0),
|
||||
"items_pending": int(node.get("items_pending", 0) or 0),
|
||||
"items_claimed": int(node.get("items_claimed", 0) or 0),
|
||||
"items_running": int(node.get("items_running", 0) or 0),
|
||||
"items_completed": int(node.get("items_completed", 0) or 0),
|
||||
"items_blacklisted": int(node.get("items_blacklisted", 0) or 0),
|
||||
"items_failed": int(node.get("items_failed", 0) or 0),
|
||||
"processed_recent": int(node.get("processed_recent", 0) or 0),
|
||||
"processed_per_minute": float(node.get("processed_per_minute", 0) or 0),
|
||||
"completed_recent": int(node.get("completed_recent", 0) or 0),
|
||||
"blacklisted_recent": int(node.get("blacklisted_recent", 0) or 0),
|
||||
"failed_recent": int(node.get("failed_recent", 0) or 0),
|
||||
"metrics_source": str(node.get("metrics_source") or "runtime"),
|
||||
}
|
||||
)
|
||||
|
||||
assigned_total = sum(int(item.get("items_total", 0) or 0) for item in node_entries)
|
||||
unassigned_total = max(0, active_job_items_total - assigned_total)
|
||||
if unassigned_total > 0:
|
||||
node_entries.append(
|
||||
{
|
||||
"node_code": "unassigned",
|
||||
"items_total": unassigned_total,
|
||||
"items_pending": active_job_pending,
|
||||
"items_claimed": 0,
|
||||
"items_running": 0,
|
||||
"items_completed": 0,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0.0,
|
||||
"completed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
"failed_recent": 0,
|
||||
"metrics_source": "central_queue",
|
||||
}
|
||||
)
|
||||
|
||||
snapshot["job"] = {
|
||||
"job_id": active_job.get("job_id"),
|
||||
"job_code": str(active_job.get("job_code") or "").strip(),
|
||||
"runtime_job_code": active_job_code,
|
||||
"status": str(active_job.get("status") or "").strip(),
|
||||
"progress_percent": float(active_job.get("progress_percent", 0) or 0),
|
||||
}
|
||||
snapshot["queue"] = {
|
||||
**dict(snapshot.get("queue") or {}),
|
||||
"items_total": active_job_items_total,
|
||||
"pending": active_job_pending,
|
||||
"claimed": active_job_claimed,
|
||||
"running": active_job_running,
|
||||
"display_claimed": display_claimed,
|
||||
"display_running": display_running,
|
||||
"completed": active_job_completed,
|
||||
"blacklisted": active_job_blacklisted,
|
||||
"failed": active_job_failed,
|
||||
"terminal": active_job_terminal,
|
||||
"terminal_percent": round((active_job_terminal / active_job_items_total) * 100, 2) if active_job_items_total else 0.0,
|
||||
}
|
||||
snapshot["nodes"] = node_entries
|
||||
return snapshot
|
||||
|
||||
|
||||
def _select_projection_job_snapshot() -> dict | None:
|
||||
active_job = get_active_detect_job_summary(event_limit=10)
|
||||
if active_job:
|
||||
return active_job
|
||||
return get_latest_detect_job_summary(
|
||||
event_limit=10,
|
||||
statuses=("completed", "partial_failed", "failed"),
|
||||
recent_minutes=20,
|
||||
)
|
||||
|
||||
|
||||
def _select_projection_job_snapshots() -> list[dict]:
|
||||
snapshots: list[dict] = []
|
||||
seen_job_ids: set[int] = set()
|
||||
|
||||
active_job = get_active_detect_job_summary(event_limit=10)
|
||||
if active_job:
|
||||
active_job_id = int(active_job.get("job_id") or 0)
|
||||
if active_job_id > 0 and active_job_id not in seen_job_ids:
|
||||
snapshots.append(active_job)
|
||||
seen_job_ids.add(active_job_id)
|
||||
|
||||
latest_finished_job = get_latest_unprojected_detect_job_summary(
|
||||
event_limit=10,
|
||||
statuses=("completed", "partial_failed", "failed"),
|
||||
recent_minutes=180,
|
||||
)
|
||||
if latest_finished_job:
|
||||
latest_finished_job_id = int(latest_finished_job.get("job_id") or 0)
|
||||
if latest_finished_job_id > 0 and latest_finished_job_id not in seen_job_ids:
|
||||
snapshots.append(latest_finished_job)
|
||||
seen_job_ids.add(latest_finished_job_id)
|
||||
|
||||
return snapshots
|
||||
|
||||
|
||||
def _emit_structured_tick(
|
||||
*,
|
||||
base_event_type: str,
|
||||
@@ -42,7 +216,13 @@ def _emit_structured_tick(
|
||||
payload = {"ok": ok, "data": data or {}}
|
||||
event_type = f"{base_event_type}_failed"
|
||||
level = "warning"
|
||||
if ok:
|
||||
if isinstance(data, dict) and str(data.get("pull_state") or "").strip() == "throttled":
|
||||
event_type = f"{base_event_type}_idle"
|
||||
level = "info"
|
||||
elif not ok and _is_idle_sync_message(message):
|
||||
event_type = f"{base_event_type}_idle"
|
||||
level = "info"
|
||||
elif ok:
|
||||
event_type = f"{base_event_type}_success"
|
||||
level = "info"
|
||||
if "但远端确认失败" in str(message or ""):
|
||||
@@ -70,7 +250,10 @@ def _emit_sync_result_breakdown(data: dict | None) -> None:
|
||||
result_data = item.get("data") or {}
|
||||
event_type = f"{sync_type}_sync_failed"
|
||||
level = "warning"
|
||||
if ok:
|
||||
if not ok and _is_idle_sync_message(message):
|
||||
event_type = f"{sync_type}_sync_idle"
|
||||
level = "info"
|
||||
elif ok:
|
||||
event_type = f"{sync_type}_sync_success"
|
||||
level = "info"
|
||||
if isinstance(result_data, dict) and result_data.get("success_count") is not None:
|
||||
@@ -95,12 +278,31 @@ def _emit_sync_result_breakdown(data: dict | None) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _run_pipeline_stage_processor() -> tuple[bool, str, dict]:
|
||||
process_limit = max(500, min(int(settings.sync_pipeline_process_limit or 5000), 5000))
|
||||
ok, message, data = process_detect_pipeline_now(limit=process_limit)
|
||||
push_debug_event(
|
||||
service="sync-agent",
|
||||
event_type="pipeline_tick_success" if ok else "pipeline_tick_failed",
|
||||
level="info" if ok else "warning",
|
||||
message=message,
|
||||
payload={
|
||||
"ok": ok,
|
||||
"limit": process_limit,
|
||||
"data": data or {},
|
||||
},
|
||||
)
|
||||
return ok, message, data
|
||||
|
||||
|
||||
def main() -> None:
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
|
||||
)
|
||||
interval = max(10, int(settings.sync_poll_interval_seconds or 30))
|
||||
# Old env files still ship SYNC_POLL_INTERVAL_SECONDS=30. Cap the interval
|
||||
# so controller pull/pipeline ticks cannot be throttled into starvation.
|
||||
interval = max(2, min(int(settings.sync_poll_interval_seconds or 2), 5))
|
||||
logger.info(
|
||||
"sync agent started: node=%s source=%s target=%s interval=%ss enabled=%s",
|
||||
settings.node_code,
|
||||
@@ -111,9 +313,16 @@ def main() -> None:
|
||||
)
|
||||
while True:
|
||||
try:
|
||||
pipeline_ok, pipeline_message, pipeline_data = _run_pipeline_stage_processor()
|
||||
logger.info(
|
||||
"pipeline tick: ok=%s message=%s data=%s",
|
||||
pipeline_ok,
|
||||
pipeline_message,
|
||||
pipeline_data,
|
||||
)
|
||||
active_job = get_active_detect_job_summary(event_limit=10)
|
||||
if active_job:
|
||||
_append_detect_result_projection_snapshot(active_job)
|
||||
for projection_job in _select_projection_job_snapshots():
|
||||
_append_detect_result_projection_snapshot(projection_job)
|
||||
ok, message, data = push_runtime_projection_now()
|
||||
logger.info("sync tick: ok=%s message=%s data=%s", ok, message, data)
|
||||
push_debug_event(
|
||||
@@ -136,8 +345,16 @@ def main() -> None:
|
||||
)
|
||||
_emit_structured_tick(base_event_type="task_pull", ok=pull_ok, message=pull_message, data=pull_data)
|
||||
if active_job:
|
||||
queue_health = get_detect_queue_health(window_minutes=15)
|
||||
recent_events = list_recent_detect_run_events(limit=8)
|
||||
queue_health = _build_aligned_queue_health_snapshot(
|
||||
active_job,
|
||||
get_detect_queue_health(window_minutes=15),
|
||||
)
|
||||
recent_events = _filter_runtime_events_for_job(
|
||||
list_recent_detect_run_events(limit=24),
|
||||
job_code=str(active_job.get("runtime_job_code") or active_job.get("job_code") or "").strip(),
|
||||
job_id=int(active_job.get("job_id", 0) or 0),
|
||||
limit=8,
|
||||
)
|
||||
push_debug_event(
|
||||
service="detect-runtime",
|
||||
event_type="active_job_snapshot",
|
||||
@@ -158,6 +375,7 @@ def main() -> None:
|
||||
"node_stats": list(active_job.get("node_stats") or []),
|
||||
},
|
||||
"queue_health": queue_health,
|
||||
"backlog": _load_local_detect_backlog_snapshot(),
|
||||
"recent_events": recent_events,
|
||||
},
|
||||
)
|
||||
|
||||
112
domain-api/deploy/multi-region/fix_mainland_release_base.sh
Executable file
112
domain-api/deploy/multi-region/fix_mainland_release_base.sh
Executable file
@@ -0,0 +1,112 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROLE="${1:-}"
|
||||
BASE_DIR="${2:-/opt/domaincheck}"
|
||||
WWW_USER="${WWW_USER:-www}"
|
||||
WWW_GROUP="${WWW_GROUP:-www}"
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
usage:
|
||||
bash domain-api/deploy/multi-region/fix_mainland_release_base.sh <worker|control> [base_dir]
|
||||
|
||||
examples:
|
||||
bash domain-api/deploy/multi-region/fix_mainland_release_base.sh worker
|
||||
bash domain-api/deploy/multi-region/fix_mainland_release_base.sh control
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${ROLE}" != "worker" && "${ROLE}" != "control" ]]; then
|
||||
usage >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$(id -u)" != "0" ]]; then
|
||||
echo "please run as root" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
require_path() {
|
||||
local path="$1"
|
||||
if [[ ! -e "${path}" ]]; then
|
||||
echo "required path missing: ${path}" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
install_dropin() {
|
||||
local service_name="$1"
|
||||
local content="$2"
|
||||
local dropin_dir="/etc/systemd/system/${service_name}.d"
|
||||
local dropin_file="${dropin_dir}/current-path.conf"
|
||||
mkdir -p "${dropin_dir}"
|
||||
printf '%s\n' "${content}" > "${dropin_file}"
|
||||
echo "installed ${dropin_file}"
|
||||
}
|
||||
|
||||
echo "[1/6] validate existing runtime"
|
||||
require_path "${BASE_DIR}"
|
||||
require_path "${BASE_DIR}/domainCheck"
|
||||
require_path "${BASE_DIR}/domainCheck/.venv/bin/python"
|
||||
require_path "${BASE_DIR}/domainCheck/detect_worker.py"
|
||||
require_path "${BASE_DIR}/domain-api"
|
||||
|
||||
echo "[2/6] make release root writable for ${WWW_USER}:${WWW_GROUP}"
|
||||
chgrp "${WWW_GROUP}" "${BASE_DIR}"
|
||||
chmod 2775 "${BASE_DIR}"
|
||||
mkdir -p "${BASE_DIR}/downloads" "${BASE_DIR}/releases"
|
||||
chown -R "${WWW_USER}:${WWW_GROUP}" "${BASE_DIR}/downloads" "${BASE_DIR}/releases"
|
||||
|
||||
echo "[3/6] ensure current link exists"
|
||||
if [[ -L "${BASE_DIR}/current" ]]; then
|
||||
echo "keep existing symlink: ${BASE_DIR}/current -> $(readlink -f "${BASE_DIR}/current" || true)"
|
||||
elif [[ -e "${BASE_DIR}/current" ]]; then
|
||||
echo "path exists but is not a symlink: ${BASE_DIR}/current" >&2
|
||||
exit 1
|
||||
else
|
||||
ln -s "${BASE_DIR}" "${BASE_DIR}/current"
|
||||
echo "created symlink: ${BASE_DIR}/current -> ${BASE_DIR}"
|
||||
fi
|
||||
|
||||
echo "[4/6] install systemd drop-ins"
|
||||
install_dropin "domaincheck-worker" "[Service]
|
||||
WorkingDirectory=${BASE_DIR}/current/domainCheck
|
||||
ExecStart=
|
||||
ExecStart=${BASE_DIR}/domainCheck/.venv/bin/python ${BASE_DIR}/current/domainCheck/detect_worker.py"
|
||||
|
||||
install_dropin "domaincheck-node-agent" "[Service]
|
||||
User=root
|
||||
Group=root
|
||||
WorkingDirectory=${BASE_DIR}/current/domain-api
|
||||
ExecStart=
|
||||
ExecStart=${BASE_DIR}/domainCheck/.venv/bin/python -m app.node_agent"
|
||||
|
||||
if [[ "${ROLE}" == "control" ]]; then
|
||||
install_dropin "domaincheck-api" "[Service]
|
||||
WorkingDirectory=${BASE_DIR}/current/domain-api
|
||||
ExecStart=
|
||||
ExecStart=${BASE_DIR}/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100"
|
||||
|
||||
install_dropin "domaincheck-sync-agent" "[Service]
|
||||
WorkingDirectory=${BASE_DIR}/current/domain-api
|
||||
ExecStart=
|
||||
ExecStart=${BASE_DIR}/domainCheck/.venv/bin/python -m app.sync_agent"
|
||||
fi
|
||||
|
||||
echo "[5/6] reload systemd and restart node agent"
|
||||
systemctl daemon-reload
|
||||
systemctl restart domaincheck-node-agent
|
||||
|
||||
echo "[6/6] summary"
|
||||
echo "role=${ROLE}"
|
||||
echo "base_dir=${BASE_DIR}"
|
||||
echo "current_target=$(readlink -f "${BASE_DIR}/current" || true)"
|
||||
echo
|
||||
echo "recommended next checks:"
|
||||
echo " systemctl status domaincheck-node-agent --no-pager -l"
|
||||
echo " systemctl cat domaincheck-worker"
|
||||
if [[ "${ROLE}" == "control" ]]; then
|
||||
echo " systemctl cat domaincheck-api"
|
||||
echo " systemctl cat domaincheck-sync-agent"
|
||||
fi
|
||||
@@ -125,8 +125,8 @@ SYNC_SOURCE_REGION=mainland
|
||||
SYNC_TARGET_REGION=overseas
|
||||
SYNC_TARGET_API_BASE_URL=${TARGET_API_BASE_URL}
|
||||
SYNC_SHARED_TOKEN=${SYNC_SHARED_TOKEN}
|
||||
SYNC_BATCH_SIZE=200
|
||||
SYNC_POLL_INTERVAL_SECONDS=30
|
||||
SYNC_BATCH_SIZE=5000
|
||||
SYNC_POLL_INTERVAL_SECONDS=2
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
|
||||
@@ -126,8 +126,8 @@ SYNC_SOURCE_REGION=mainland
|
||||
SYNC_TARGET_REGION=overseas
|
||||
SYNC_TARGET_API_BASE_URL=${TARGET_API_BASE_URL}
|
||||
SYNC_SHARED_TOKEN=${SYNC_SHARED_TOKEN}
|
||||
SYNC_BATCH_SIZE=200
|
||||
SYNC_POLL_INTERVAL_SECONDS=30
|
||||
SYNC_BATCH_SIZE=5000
|
||||
SYNC_POLL_INTERVAL_SECONDS=2
|
||||
EOF
|
||||
|
||||
systemctl daemon-reload
|
||||
|
||||
303
domain-api/deploy/multi-region/night_shift_until_20260420_1200.sh
Executable file
303
domain-api/deploy/multi-region/night_shift_until_20260420_1200.sh
Executable file
@@ -0,0 +1,303 @@
|
||||
#!/usr/bin/env bash
|
||||
set -u -o pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
|
||||
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||
DEADLINE_TEXT="${2:-2026-04-20 12:00:00 +0800}"
|
||||
RUNTIME_DIR="${REPO_ROOT}/docs/ops_center_runtime/night_runs"
|
||||
START_TS="$(date +%Y%m%d_%H%M%S)"
|
||||
RUN_ID="night_run_${START_TS}"
|
||||
LOG_FILE="${RUNTIME_DIR}/${RUN_ID}.log"
|
||||
STATE_DIR="${RUNTIME_DIR}/${RUN_ID}"
|
||||
PID_FILE="${RUNTIME_DIR}/${RUN_ID}.pid"
|
||||
REPORT_FILE="${RUNTIME_DIR}/${RUN_ID}_report.md"
|
||||
SUMMARY_JSON="${RUNTIME_DIR}/${RUN_ID}_summary.json"
|
||||
|
||||
mkdir -p "${RUNTIME_DIR}" "${STATE_DIR}"
|
||||
echo "$$" > "${PID_FILE}"
|
||||
|
||||
DEADLINE_EPOCH="$(python3 - <<'PY' "${DEADLINE_TEXT}"
|
||||
import datetime
|
||||
import sys
|
||||
|
||||
text = sys.argv[1]
|
||||
dt = datetime.datetime.strptime(text, "%Y-%m-%d %H:%M:%S %z")
|
||||
print(int(dt.timestamp()))
|
||||
PY
|
||||
)"
|
||||
|
||||
cycles=0
|
||||
log_sync_recover_runs=0
|
||||
inspection_runs=0
|
||||
inspection_churn_runs=0
|
||||
quick_rechecks=0
|
||||
stop_reason="deadline_reached"
|
||||
last_go_live_status=""
|
||||
last_publish_ready=""
|
||||
last_log_sync_state=""
|
||||
last_issue_total=""
|
||||
last_problem_runs_total=""
|
||||
last_launchpad_status=""
|
||||
last_launchpad_action=""
|
||||
last_problem_run_code=""
|
||||
|
||||
log() {
|
||||
printf '[%s] %s\n' "$(date '+%Y-%m-%d %H:%M:%S %z')" "$*" | tee -a "${LOG_FILE}"
|
||||
}
|
||||
|
||||
run_cmd() {
|
||||
log "RUN $*"
|
||||
if "$@" >> "${LOG_FILE}" 2>&1; then
|
||||
log "OK $*"
|
||||
return 0
|
||||
fi
|
||||
log "ERR $*"
|
||||
return 1
|
||||
}
|
||||
|
||||
fetch_json() {
|
||||
local url="$1"
|
||||
local target="$2"
|
||||
if curl -m 20 -s "${url}" > "${target}.tmp"; then
|
||||
mv "${target}.tmp" "${target}"
|
||||
return 0
|
||||
fi
|
||||
rm -f "${target}.tmp"
|
||||
return 1
|
||||
}
|
||||
|
||||
snapshot_state() {
|
||||
local cycle_dir="$1"
|
||||
mkdir -p "${cycle_dir}"
|
||||
|
||||
fetch_json "${BASE_URL}/api/v1/ops/go-live-summary" "${cycle_dir}/go_live.json" || true
|
||||
fetch_json "${BASE_URL}/api/v1/ops/stack-diagnosis" "${cycle_dir}/stack.json" || true
|
||||
fetch_json "${BASE_URL}/api/v1/ops/releases/launchpad" "${cycle_dir}/launchpad.json" || true
|
||||
fetch_json "${BASE_URL}/api/v1/ops/playbook-runs" "${cycle_dir}/playbook_runs.json" || true
|
||||
fetch_json "${BASE_URL}/api/v1/ops/nodes/overseas-control-01/scene-log?limit=120&mode=full" "${cycle_dir}/scene_overseas_control_01.json" || true
|
||||
|
||||
python3 - <<'PY' \
|
||||
"${cycle_dir}/go_live.json" \
|
||||
"${cycle_dir}/stack.json" \
|
||||
"${cycle_dir}/launchpad.json" \
|
||||
"${cycle_dir}/playbook_runs.json" \
|
||||
"${cycle_dir}/scene_overseas_control_01.json" \
|
||||
"${SUMMARY_JSON}" \
|
||||
"${cycles}" \
|
||||
> "${cycle_dir}/summary.env"
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
go_path, stack_path, launch_path, runs_path, scene_path, summary_path, cycles = sys.argv[1:8]
|
||||
|
||||
def load_json(path):
|
||||
try:
|
||||
with open(path, "r", encoding="utf-8") as fh:
|
||||
return json.load(fh)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
go_data = load_json(go_path).get("data", {})
|
||||
stack_data = load_json(stack_path).get("data", {}).get("diagnosis", {})
|
||||
launch_data = load_json(launch_path).get("data", {})
|
||||
runs_data = load_json(runs_path).get("data", {})
|
||||
scene_data = load_json(scene_path).get("data", {})
|
||||
|
||||
issues = stack_data.get("issues") or []
|
||||
issue_codes = [str(item.get("code") or "") for item in issues if item.get("code")]
|
||||
problem_runs = runs_data.get("problem_runs") or []
|
||||
problem_run_code = ""
|
||||
if problem_runs:
|
||||
problem_run_code = str(problem_runs[0].get("run_code") or "")
|
||||
|
||||
summary = {
|
||||
"cycles": int(cycles),
|
||||
"go_live_status": str(go_data.get("go_live_status") or ""),
|
||||
"publish_ready": bool(go_data.get("publish_ready")),
|
||||
"log_sync_state": str(go_data.get("log_sync_state") or ""),
|
||||
"log_sync_missing_node_codes": go_data.get("log_sync_missing_node_codes") or [],
|
||||
"stack_status": str(stack_data.get("stack_status") or ""),
|
||||
"issue_total": int(stack_data.get("issue_total") or 0),
|
||||
"blocking_issue_total": int(stack_data.get("blocking_issue_total") or 0),
|
||||
"issue_codes": issue_codes,
|
||||
"launchpad_status": str((launch_data.get("launchpad_status") or {}).get("status") or ""),
|
||||
"launchpad_recommended_action": str((launch_data.get("launchpad_status") or {}).get("recommended_action_code") or ""),
|
||||
"problem_runs_total": int(runs_data.get("problem_runs_total") or 0),
|
||||
"problem_run_code": problem_run_code,
|
||||
"scene_status": str(scene_data.get("status") or ""),
|
||||
"scene_line_count": int((scene_data.get("source_summary") or {}).get("line_count") or 0),
|
||||
"generated_at": str(go_data.get("generated_at") or stack_data.get("generated_at") or ""),
|
||||
}
|
||||
|
||||
with open(summary_path, "w", encoding="utf-8") as fh:
|
||||
json.dump(summary, fh, ensure_ascii=False, indent=2)
|
||||
|
||||
def emit(key, value):
|
||||
if isinstance(value, bool):
|
||||
value = "true" if value else "false"
|
||||
elif isinstance(value, list):
|
||||
value = ",".join(str(item) for item in value)
|
||||
else:
|
||||
value = str(value)
|
||||
print(f'{key}="{value}"')
|
||||
|
||||
for key, value in summary.items():
|
||||
emit(key.upper(), value)
|
||||
PY
|
||||
|
||||
# shellcheck disable=SC1090
|
||||
source "${cycle_dir}/summary.env"
|
||||
last_go_live_status="${GO_LIVE_STATUS}"
|
||||
last_publish_ready="${PUBLISH_READY}"
|
||||
last_log_sync_state="${LOG_SYNC_STATE}"
|
||||
last_issue_total="${ISSUE_TOTAL}"
|
||||
last_problem_runs_total="${PROBLEM_RUNS_TOTAL}"
|
||||
last_launchpad_status="${LAUNCHPAD_STATUS}"
|
||||
last_launchpad_action="${LAUNCHPAD_RECOMMENDED_ACTION}"
|
||||
last_problem_run_code="${PROBLEM_RUN_CODE}"
|
||||
}
|
||||
|
||||
write_report() {
|
||||
python3 - <<'PY' \
|
||||
"${REPORT_FILE}" \
|
||||
"${RUN_ID}" \
|
||||
"${BASE_URL}" \
|
||||
"${DEADLINE_TEXT}" \
|
||||
"${stop_reason}" \
|
||||
"${cycles}" \
|
||||
"${log_sync_recover_runs}" \
|
||||
"${inspection_runs}" \
|
||||
"${inspection_churn_runs}" \
|
||||
"${quick_rechecks}" \
|
||||
"${last_go_live_status}" \
|
||||
"${last_publish_ready}" \
|
||||
"${last_log_sync_state}" \
|
||||
"${last_issue_total}" \
|
||||
"${last_problem_runs_total}" \
|
||||
"${last_launchpad_status}" \
|
||||
"${last_launchpad_action}" \
|
||||
"${last_problem_run_code}" \
|
||||
"${SUMMARY_JSON}"
|
||||
import json
|
||||
import sys
|
||||
|
||||
(
|
||||
report_path,
|
||||
run_id,
|
||||
base_url,
|
||||
deadline_text,
|
||||
stop_reason,
|
||||
cycles,
|
||||
log_sync_recover_runs,
|
||||
inspection_runs,
|
||||
inspection_churn_runs,
|
||||
quick_rechecks,
|
||||
last_go_live_status,
|
||||
last_publish_ready,
|
||||
last_log_sync_state,
|
||||
last_issue_total,
|
||||
last_problem_runs_total,
|
||||
last_launchpad_status,
|
||||
last_launchpad_action,
|
||||
last_problem_run_code,
|
||||
summary_json_path,
|
||||
) = sys.argv[1:20]
|
||||
|
||||
summary = {}
|
||||
try:
|
||||
with open(summary_json_path, "r", encoding="utf-8") as fh:
|
||||
summary = json.load(fh)
|
||||
except Exception:
|
||||
summary = {}
|
||||
|
||||
lines = [
|
||||
f"# NIGHT RUN REPORT {run_id}",
|
||||
"",
|
||||
f"- Base URL: `{base_url}`",
|
||||
f"- Deadline: `{deadline_text}`",
|
||||
f"- Stop Reason: `{stop_reason}`",
|
||||
f"- Cycles: `{cycles}`",
|
||||
f"- Log Sync Recover Runs: `{log_sync_recover_runs}`",
|
||||
f"- Inspection Runs: `{inspection_runs}`",
|
||||
f"- Inspection Churn Runs: `{inspection_churn_runs}`",
|
||||
f"- Quick Rechecks: `{quick_rechecks}`",
|
||||
"",
|
||||
"## Final Snapshot",
|
||||
"",
|
||||
f"- `go_live_status = {last_go_live_status}`",
|
||||
f"- `publish_ready = {last_publish_ready}`",
|
||||
f"- `log_sync_state = {last_log_sync_state}`",
|
||||
f"- `issue_total = {last_issue_total}`",
|
||||
f"- `problem_runs_total = {last_problem_runs_total}`",
|
||||
f"- `launchpad_status = {last_launchpad_status}`",
|
||||
f"- `launchpad_recommended_action = {last_launchpad_action}`",
|
||||
f"- `problem_run_code = {last_problem_run_code}`",
|
||||
"",
|
||||
"## Summary JSON",
|
||||
"",
|
||||
"```json",
|
||||
json.dumps(summary, ensure_ascii=False, indent=2),
|
||||
"```",
|
||||
]
|
||||
|
||||
with open(report_path, "w", encoding="utf-8") as fh:
|
||||
fh.write("\n".join(lines) + "\n")
|
||||
PY
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
write_report
|
||||
log "night run stopped: reason=${stop_reason}"
|
||||
log "report: ${REPORT_FILE}"
|
||||
rm -f "${PID_FILE}"
|
||||
}
|
||||
|
||||
trap cleanup EXIT
|
||||
|
||||
log "night run started: run_id=${RUN_ID}"
|
||||
log "base_url=${BASE_URL}"
|
||||
log "deadline=${DEADLINE_TEXT}"
|
||||
log "state_dir=${STATE_DIR}"
|
||||
|
||||
while true; do
|
||||
now_epoch="$(date +%s)"
|
||||
if [[ "${now_epoch}" -ge "${DEADLINE_EPOCH}" ]]; then
|
||||
stop_reason="deadline_reached"
|
||||
break
|
||||
fi
|
||||
|
||||
cycles=$((cycles + 1))
|
||||
cycle_dir="${STATE_DIR}/cycle_${cycles}"
|
||||
snapshot_state "${cycle_dir}"
|
||||
|
||||
log "cycle=${cycles} go_live=${last_go_live_status} log_sync=${last_log_sync_state} issue_total=${last_issue_total} problem_runs=${last_problem_runs_total} launchpad=${last_launchpad_status}/${last_launchpad_action} focus_run=${last_problem_run_code}"
|
||||
|
||||
if [[ "${last_log_sync_state}" != "full_capture" ]]; then
|
||||
log "action: recover log sync coverage"
|
||||
run_cmd bash "${SCRIPT_DIR}/drive_ops_center.sh" log-sync-recover "${BASE_URL}" full confirm cli/night-pack || true
|
||||
log_sync_recover_runs=$((log_sync_recover_runs + 1))
|
||||
run_cmd bash "${SCRIPT_DIR}/drive_ops_center.sh" driver-run "${BASE_URL}" run_inspection_participating '{}' confirm cli/night-pack || true
|
||||
inspection_runs=$((inspection_runs + 1))
|
||||
quick_rechecks=$((quick_rechecks + 1))
|
||||
sleep 45
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "${last_issue_total}" != "0" && "${last_problem_runs_total}" != "0" && "${inspection_churn_runs}" -lt 3 ]]; then
|
||||
log "action: create fresh safe inspection run to dilute stale problem run window"
|
||||
run_cmd bash "${SCRIPT_DIR}/drive_ops_center.sh" driver-run "${BASE_URL}" run_inspection_participating '{}' confirm cli/night-pack || true
|
||||
inspection_runs=$((inspection_runs + 1))
|
||||
inspection_churn_runs=$((inspection_churn_runs + 1))
|
||||
sleep 1800
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "${last_issue_total}" == "0" && "${last_log_sync_state}" == "full_capture" && "${last_problem_runs_total}" == "0" ]]; then
|
||||
stop_reason="signoff_ready_candidate"
|
||||
break
|
||||
fi
|
||||
|
||||
sleep 3600
|
||||
done
|
||||
@@ -26,5 +26,5 @@ SYNC_SOURCE_REGION=overseas
|
||||
SYNC_TARGET_REGION=overseas
|
||||
SYNC_TARGET_API_BASE_URL=
|
||||
SYNC_SHARED_TOKEN=
|
||||
SYNC_BATCH_SIZE=200
|
||||
SYNC_POLL_INTERVAL_SECONDS=30
|
||||
SYNC_BATCH_SIZE=5000
|
||||
SYNC_POLL_INTERVAL_SECONDS=2
|
||||
|
||||
@@ -11,5 +11,5 @@ API_SERVICE_NAME=domaincheck-api
|
||||
SYNC_AGENT_SERVICE_NAME=domaincheck-sync-agent
|
||||
NODE_AGENT_SERVICE_NAME=domaincheck-node-agent
|
||||
|
||||
OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.start_detection","runtime.stop_detection","runtime.pull_tasks","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","health.snapshot","logs.collect","diagnostics.collect","deploy.release"]
|
||||
OPS_AGENT_CAPABILITIES=["service.start","service.stop","service.restart","service.status","runtime.start_worker","runtime.stop_worker","runtime.start_detection","runtime.stop_detection","runtime.pull_tasks","runtime.restart_api","runtime.start_sync_agent","runtime.stop_sync_agent","runtime.reset_lab_state","health.snapshot","logs.collect","diagnostics.collect","deploy.release"]
|
||||
OPS_AGENT_LABELS={}
|
||||
|
||||
@@ -23,5 +23,5 @@ SYNC_SOURCE_REGION=mainland
|
||||
SYNC_TARGET_REGION=overseas
|
||||
SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
|
||||
SYNC_SHARED_TOKEN=
|
||||
SYNC_BATCH_SIZE=200
|
||||
SYNC_POLL_INTERVAL_SECONDS=30
|
||||
SYNC_BATCH_SIZE=5000
|
||||
SYNC_POLL_INTERVAL_SECONDS=2
|
||||
|
||||
@@ -23,5 +23,5 @@ SYNC_SOURCE_REGION=mainland
|
||||
SYNC_TARGET_REGION=overseas
|
||||
SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
|
||||
SYNC_SHARED_TOKEN=
|
||||
SYNC_BATCH_SIZE=200
|
||||
SYNC_POLL_INTERVAL_SECONDS=30
|
||||
SYNC_BATCH_SIZE=5000
|
||||
SYNC_POLL_INTERVAL_SECONDS=2
|
||||
|
||||
@@ -6,13 +6,14 @@ After=network.target
|
||||
Type=simple
|
||||
WorkingDirectory=/opt/domaincheck/domain-api
|
||||
EnvironmentFile=-/etc/default/domaincheck-api
|
||||
EnvironmentFile=-/etc/default/domaincheck-worker
|
||||
EnvironmentFile=-/etc/default/domaincheck-node-agent
|
||||
Environment="PATH=/home/www/.nvm/versions/node/v20.20.2/bin:/home/www/.local/bin:/home/www/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
|
||||
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m app.node_agent
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
User=www
|
||||
Group=www
|
||||
User=root
|
||||
Group=root
|
||||
SupplementaryGroups=systemd-journal
|
||||
|
||||
[Install]
|
||||
|
||||
93
domain-api/tests/test_cluster_runtime_service.py
Normal file
93
domain-api/tests/test_cluster_runtime_service.py
Normal file
@@ -0,0 +1,93 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from app.services import cluster_runtime_service
|
||||
|
||||
|
||||
class RuntimeSchemaGuardTests(unittest.TestCase):
|
||||
def test_ensure_runtime_schema_executes_only_once_per_process(self) -> None:
|
||||
conn = MagicMock()
|
||||
cursor_cm = MagicMock()
|
||||
cursor = MagicMock()
|
||||
conn.cursor.return_value = cursor_cm
|
||||
cursor_cm.__enter__.return_value = cursor
|
||||
db_cm = MagicMock()
|
||||
db_cm.__enter__.return_value = conn
|
||||
|
||||
with patch.object(cluster_runtime_service, "_RUNTIME_SCHEMA_READY", False):
|
||||
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm) as mocked_get_db:
|
||||
cluster_runtime_service.ensure_runtime_schema()
|
||||
cluster_runtime_service.ensure_runtime_schema()
|
||||
|
||||
mocked_get_db.assert_called_once()
|
||||
self.assertEqual(cursor.execute.call_count, 2)
|
||||
cursor.execute.assert_any_call(
|
||||
"SELECT pg_advisory_xact_lock(%s)",
|
||||
(cluster_runtime_service._RUNTIME_SCHEMA_ADVISORY_LOCK_ID,),
|
||||
)
|
||||
cursor.execute.assert_any_call(cluster_runtime_service._RUNTIME_SCHEMA_SQL)
|
||||
conn.commit.assert_called_once()
|
||||
|
||||
def test_control_node_supports_worker_only_on_mainland_with_worker_signals(self) -> None:
|
||||
self.assertFalse(
|
||||
cluster_runtime_service._control_node_supports_worker(
|
||||
region="overseas",
|
||||
metadata={
|
||||
"worker_online": False,
|
||||
"detect_participating": True,
|
||||
"active_threads": 0,
|
||||
"max_threads": 0,
|
||||
},
|
||||
)
|
||||
)
|
||||
self.assertTrue(
|
||||
cluster_runtime_service._control_node_supports_worker(
|
||||
region="mainland",
|
||||
metadata={
|
||||
"worker_online": True,
|
||||
"detect_participating": False,
|
||||
"active_threads": 0,
|
||||
"max_threads": 0,
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
def test_metadata_idle_without_runtime_work_detects_stale_idle_heartbeat(self) -> None:
|
||||
self.assertTrue(
|
||||
cluster_runtime_service._metadata_idle_without_runtime_work(
|
||||
{
|
||||
"phase_label": "idle",
|
||||
"phase_detail": "Worker 已启动，等待检测指令",
|
||||
"active_threads": 323,
|
||||
"max_threads": 4,
|
||||
"active_job_code": "",
|
||||
"job_items_total": 0,
|
||||
"job_items_claimed": 0,
|
||||
"job_items_running": 0,
|
||||
"job_items_completed": 0,
|
||||
"job_items_failed": 0,
|
||||
}
|
||||
)
|
||||
)
|
||||
self.assertFalse(
|
||||
cluster_runtime_service._metadata_idle_without_runtime_work(
|
||||
{
|
||||
"phase_label": "running",
|
||||
"active_threads": 12,
|
||||
}
|
||||
)
|
||||
)
|
||||
self.assertFalse(
|
||||
cluster_runtime_service._metadata_idle_without_runtime_work(
|
||||
{
|
||||
"phase_label": "idle",
|
||||
"active_job_code": "sync-overseas-1",
|
||||
}
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
34
domain-api/tests/test_core_db_retry.py
Normal file
34
domain-api/tests/test_core_db_retry.py
Normal file
@@ -0,0 +1,34 @@
|
||||
from psycopg2 import errors
|
||||
|
||||
from app.core.db import db_read_retry
|
||||
|
||||
|
||||
def test_db_read_retry_retries_retryable_error_once() -> None:
|
||||
attempts = {"count": 0}
|
||||
|
||||
@db_read_retry(attempts=3, initial_delay_seconds=0)
|
||||
def flaky() -> str:
|
||||
attempts["count"] += 1
|
||||
if attempts["count"] == 1:
|
||||
raise errors.DeadlockDetected()
|
||||
return "ok"
|
||||
|
||||
assert flaky() == "ok"
|
||||
assert attempts["count"] == 2
|
||||
|
||||
|
||||
def test_db_read_retry_does_not_swallow_non_retryable_error() -> None:
|
||||
attempts = {"count": 0}
|
||||
|
||||
@db_read_retry(attempts=3, initial_delay_seconds=0)
|
||||
def broken() -> str:
|
||||
attempts["count"] += 1
|
||||
raise ValueError("boom")
|
||||
|
||||
try:
|
||||
broken()
|
||||
except ValueError as exc:
|
||||
assert str(exc) == "boom"
|
||||
else:
|
||||
raise AssertionError("expected ValueError")
|
||||
assert attempts["count"] == 1
|
||||
32
domain-api/tests/test_core_files_runtime_root.py
Normal file
32
domain-api/tests/test_core_files_runtime_root.py
Normal file
@@ -0,0 +1,32 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.core import files
|
||||
|
||||
|
||||
class RuntimeRootTests(unittest.TestCase):
|
||||
def test_runtime_root_prefers_shared_runtime_outside_release_tree(self) -> None:
|
||||
fake_file = "/opt/domaincheck/releases/domaincheck_release_20260420_222047/domain-api/app/core/files.py"
|
||||
with patch.object(files, "__file__", fake_file):
|
||||
with patch.object(Path, "mkdir", autospec=True, return_value=None) as mocked_mkdir:
|
||||
runtime_root = files.runtime_root()
|
||||
|
||||
self.assertEqual(Path("/opt/domaincheck/runtime/domain-api"), runtime_root)
|
||||
mocked_mkdir.assert_called_once()
|
||||
|
||||
def test_runtime_root_respects_env_override(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as tempdir:
|
||||
override = Path(tempdir) / "custom-runtime"
|
||||
with patch.dict(os.environ, {"DOMAIN_API_RUNTIME_ROOT": str(override)}, clear=False):
|
||||
runtime_root = files.runtime_root()
|
||||
self.assertEqual(override, runtime_root)
|
||||
self.assertTrue(runtime_root.exists())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
237
domain-api/tests/test_dashboard_service.py
Normal file
237
domain-api/tests/test_dashboard_service.py
Normal file
@@ -0,0 +1,237 @@
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.services.dashboard import fetch_overview
|
||||
|
||||
|
||||
class _FakeCursor:
|
||||
def __init__(self, responses):
|
||||
self._responses = list(responses)
|
||||
|
||||
def execute(self, sql, params=None):
|
||||
self._last_sql = sql
|
||||
self._last_params = params
|
||||
|
||||
def fetchone(self):
|
||||
if self._responses:
|
||||
return self._responses.pop(0)
|
||||
return (0,)
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
|
||||
class _FakeConnection:
|
||||
def __init__(self, responses):
|
||||
self._cursor = _FakeCursor(responses)
|
||||
|
||||
def cursor(self):
|
||||
return self._cursor
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
|
||||
class DashboardServiceTests(unittest.TestCase):
|
||||
@patch("app.services.dashboard._fetch_active_jobs_aggregate")
|
||||
@patch("app.services.dashboard.get_detect_capacity_plan")
|
||||
@patch("app.services.dashboard.get_detect_queue_health")
|
||||
@patch("app.services.dashboard.get_runtime_status")
|
||||
@patch("app.services.dashboard.get_active_detect_job_summary")
|
||||
@patch("app.services.dashboard.get_db")
|
||||
def test_fetch_overview_includes_ops_metrics(
|
||||
self,
|
||||
mock_get_db,
|
||||
mock_get_active_detect_job_summary,
|
||||
mock_get_runtime_status,
|
||||
mock_get_detect_queue_health,
|
||||
mock_get_detect_capacity_plan,
|
||||
mock_fetch_active_jobs_aggregate,
|
||||
) -> None:
|
||||
mock_get_db.return_value = _FakeConnection(
|
||||
responses=[
|
||||
(1000,),
|
||||
(900,),
|
||||
(10,),
|
||||
(5,),
|
||||
(0,),
|
||||
(1,),
|
||||
(430,),
|
||||
(420,),
|
||||
(17,),
|
||||
]
|
||||
)
|
||||
mock_fetch_active_jobs_aggregate.return_value = {
|
||||
"active_jobs_total": 3,
|
||||
"queue": {
|
||||
"items_total": 5200,
|
||||
"pending": 4700,
|
||||
"claimed": 11,
|
||||
"running": 165,
|
||||
"completed": 300,
|
||||
"blacklisted": 12,
|
||||
"failed": 7,
|
||||
},
|
||||
"throughput": {
|
||||
"processed_recent": 30,
|
||||
"processed_per_minute": 2.0,
|
||||
"completed_recent": 20,
|
||||
"blacklisted_recent": 1,
|
||||
"failed_recent": 2,
|
||||
},
|
||||
"retry_total": 12,
|
||||
"steps": [
|
||||
{
|
||||
"step_code": "detect_register",
|
||||
"step_name": "注册状态检测",
|
||||
"items_pending": 185,
|
||||
"items_running": 3,
|
||||
"items_claimed": 0,
|
||||
"items_completed": 7,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
"processed_recent": 20,
|
||||
"processed_per_minute": 1.33,
|
||||
}
|
||||
],
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"items_running": 120,
|
||||
"items_claimed": 0,
|
||||
"processed_recent": 20,
|
||||
"processed_per_minute": 1.33,
|
||||
"completed_recent": 15,
|
||||
"failed_recent": 1,
|
||||
"blacklisted_recent": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
mock_get_active_detect_job_summary.return_value = {
|
||||
"job_id": 76,
|
||||
"job_code": "sync-overseas-3612",
|
||||
"items_pending": 185,
|
||||
"items_running": 8,
|
||||
"items_completed": 7,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
}
|
||||
mock_get_runtime_status.return_value = {
|
||||
"worker": {"running": True, "mode": "linux-systemd", "expected_on_this_node": True},
|
||||
"node": {"region": "overseas", "role": "control"},
|
||||
"cluster": {"summary": {"online_worker_nodes": 2, "dedicated_online_worker_nodes": 1, "online_control_nodes": 1}},
|
||||
"detect": {
|
||||
"backlog": {
|
||||
"pending_total": 9438,
|
||||
"claimed_total": 410,
|
||||
"running_total": 15,
|
||||
"completed_total": 951,
|
||||
"blacklisted_total": 0,
|
||||
"failed_total": 137,
|
||||
"register_pending": 8487,
|
||||
"downstream_pending": 951,
|
||||
}
|
||||
},
|
||||
}
|
||||
mock_get_detect_queue_health.return_value = {
|
||||
"has_active_job": True,
|
||||
"job": {"job_id": 76, "job_code": "sync-overseas-3612", "status": "running", "progress_percent": 3.5},
|
||||
"queue": {
|
||||
"items_total": 200,
|
||||
"pending": 185,
|
||||
"claimed": 0,
|
||||
"display_claimed": 0,
|
||||
"running": 8,
|
||||
"display_running": 165,
|
||||
"completed": 7,
|
||||
"blacklisted": 0,
|
||||
"failed": 0,
|
||||
},
|
||||
"throughput": {"processed_recent": 0, "processed_per_minute": 0},
|
||||
"steps": [
|
||||
{
|
||||
"step_code": "detect_register",
|
||||
"step_name": "注册状态检测",
|
||||
"items_pending": 185,
|
||||
"items_running": 165,
|
||||
"items_claimed": 0,
|
||||
"items_completed": 7,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0.0,
|
||||
}
|
||||
],
|
||||
"runtime_activity": {
|
||||
"step_stats": {
|
||||
"detect_360_site": {
|
||||
"step_code": "detect_360_site",
|
||||
"started_recent": 269,
|
||||
"processed_recent": 6,
|
||||
"completed_recent": 6,
|
||||
"failed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
}
|
||||
}
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"items_running": 165,
|
||||
"items_claimed": 0,
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0.0,
|
||||
"completed_recent": 0,
|
||||
"failed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
mock_get_detect_capacity_plan.return_value = {
|
||||
"estimated_hours_remaining": 1.5,
|
||||
"remaining_items": 193,
|
||||
"recommended_additional_workers": 1,
|
||||
}
|
||||
|
||||
data = fetch_overview()
|
||||
|
||||
self.assertEqual("sync-overseas-3612", data["active_job"]["job_code"])
|
||||
self.assertEqual(3, data["active_jobs_aggregate"]["active_jobs_total"])
|
||||
self.assertEqual(9438, data["active_jobs_aggregate"]["queue"]["pending"])
|
||||
self.assertEqual(410, data["active_jobs_aggregate"]["queue"]["claimed"])
|
||||
self.assertEqual(15, data["active_jobs_aggregate"]["queue"]["running"])
|
||||
self.assertEqual(951, data["active_jobs_aggregate"]["queue"]["completed"])
|
||||
self.assertEqual(137, data["active_jobs_aggregate"]["queue"]["failed"])
|
||||
self.assertEqual("mainland-controller-01", data["active_jobs_aggregate"]["nodes"][0]["node_code"])
|
||||
self.assertEqual(900, data["pending_total"])
|
||||
self.assertEqual(10, data["completed_total"])
|
||||
self.assertEqual(430, data["registerable_total"])
|
||||
self.assertEqual(420, data["purchasable_total"])
|
||||
self.assertEqual(185, data["queue_pending_total"])
|
||||
self.assertEqual(8, data["queue_running_total"])
|
||||
self.assertEqual(165, data["queue_display_running_total"])
|
||||
self.assertEqual(7, data["queue_completed_total"])
|
||||
self.assertEqual(9438, data["backlog_pending_total"])
|
||||
self.assertEqual(8487, data["backlog_register_pending_total"])
|
||||
self.assertEqual(951, data["backlog_downstream_pending_total"])
|
||||
self.assertEqual(12, data["retry_total"])
|
||||
self.assertEqual("注册状态检测", data["bottleneck_step"]["step_name"])
|
||||
self.assertEqual(0.0, data["processed_per_minute"])
|
||||
self.assertEqual(0, data["processed_recent"])
|
||||
self.assertEqual(0.0, data["ops_summary"]["processed_per_minute"])
|
||||
self.assertEqual(1.5, data["ops_summary"]["estimated_hours_remaining"])
|
||||
self.assertEqual(1, data["active_execution_nodes"])
|
||||
self.assertEqual(1, data["ops_summary"]["active_execution_nodes"])
|
||||
self.assertEqual(2, len(data["step_queue"]))
|
||||
self.assertTrue(any(item["step_code"] == "detect_360_site" for item in data["step_queue"]))
|
||||
self.assertEqual(1, len(data["node_throughput"]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
75
domain-api/tests/test_detect_api_routes.py
Normal file
75
domain-api/tests/test_detect_api_routes.py
Normal file
@@ -0,0 +1,75 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.api.routes import detect as detect_route
|
||||
|
||||
|
||||
class DetectApiRoutesTestCase(unittest.TestCase):
|
||||
@patch("app.api.routes.detect.create_detect_run_snapshot")
|
||||
@patch("app.api.routes.detect._dispatch_remote_detect_start")
|
||||
@patch("app.api.routes.detect.get_settings_payload")
|
||||
@patch("app.api.routes.detect.get_detect_status")
|
||||
@patch("app.api.routes.detect.send_worker_command")
|
||||
@patch("app.api.routes.detect.start_worker")
|
||||
@patch("app.api.routes.detect.append_detect_job_event")
|
||||
@patch("app.api.routes.detect.create_detect_job_if_needed")
|
||||
def test_start_detect_skips_local_worker_on_overseas_control(
|
||||
self,
|
||||
mock_create_job,
|
||||
mock_append_event,
|
||||
mock_start_worker,
|
||||
mock_send_worker_command,
|
||||
mock_get_detect_status,
|
||||
mock_get_settings_payload,
|
||||
mock_dispatch_remote,
|
||||
mock_create_snapshot,
|
||||
) -> None:
|
||||
mock_create_job.return_value = {
|
||||
"job_id": 29,
|
||||
"job_code": "sync-overseas-29",
|
||||
"status": "running",
|
||||
"items_pending": 100,
|
||||
"items_claimed": 0,
|
||||
"items_running": 0,
|
||||
"task_mode": "single_step",
|
||||
"step_code": "detect_register",
|
||||
}
|
||||
mock_get_detect_status.return_value = {
|
||||
"worker_mode": "linux-systemd",
|
||||
"worker_online": False,
|
||||
"worker_process_count": 0,
|
||||
"worker_latest_start_time": "",
|
||||
"worker_runtime_message": "not-applicable",
|
||||
"progress": {},
|
||||
}
|
||||
mock_get_settings_payload.return_value = {
|
||||
"runtime": {"thread_count": 2000},
|
||||
"proxy_config": {"proxy_enable": True, "allow_direct": False, "proxy_urls": ["a"]},
|
||||
}
|
||||
mock_dispatch_remote.return_value = {
|
||||
"queued_jobs": [{"node_code": "mainland-controller-01"}],
|
||||
"queued_total": 1,
|
||||
"failed_total": 0,
|
||||
"target_summary": {"controller_nodes": ["mainland-controller-01"], "worker_nodes": ["mainland-worker-01"]},
|
||||
}
|
||||
|
||||
with patch.object(detect_route.settings, "node_region", "overseas"), patch.object(
|
||||
detect_route.settings, "node_role", "control"
|
||||
):
|
||||
response = detect_route.start_detect()
|
||||
|
||||
self.assertEqual(0, response.code)
|
||||
self.assertIn("海外控制面", response.message)
|
||||
mock_start_worker.assert_not_called()
|
||||
mock_send_worker_command.assert_not_called()
|
||||
mock_dispatch_remote.assert_called_once()
|
||||
mock_create_snapshot.assert_called_once()
|
||||
event_types = [call.kwargs.get("event_type") for call in mock_append_event.call_args_list]
|
||||
self.assertIn("job_dispatch_requested", event_types)
|
||||
self.assertIn("job_dispatch_skipped_local", event_types)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,121 +1,791 @@
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from app.services.detect_job_service import _build_effective_node_stats, _build_effective_summary
|
||||
from psycopg2 import errors
|
||||
|
||||
from app.services.detect_job_service import (
|
||||
append_detect_job_event,
|
||||
_build_step_payload,
|
||||
_classify_pipeline_item_outcome,
|
||||
_classify_runtime_debug_event,
|
||||
_enrich_active_job_summary_with_runtime,
|
||||
_build_display_summary,
|
||||
_build_runtime_display_bucket,
|
||||
normalize_detect_step_code,
|
||||
get_detect_queue_health,
|
||||
process_detect_pipeline_now,
|
||||
resolve_initial_domain_pipeline_item,
|
||||
resolve_domain_pipeline_step,
|
||||
resolve_detect_job_definition,
|
||||
)
|
||||
|
||||
|
||||
class DetectJobServiceTests(unittest.TestCase):
|
||||
def test_build_effective_node_stats_keeps_runtime_nodes_and_backfills_unassigned(self) -> None:
|
||||
distributed_node_stats = [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"items_total": 200,
|
||||
"items_pending": 195,
|
||||
"items_claimed": 0,
|
||||
"items_running": 5,
|
||||
"items_completed": 0,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"items_total": 70,
|
||||
"items_pending": 56,
|
||||
"items_claimed": 9,
|
||||
"items_running": 5,
|
||||
"items_completed": 0,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
},
|
||||
{
|
||||
"node_code": "overseas-control-01",
|
||||
"items_total": 50,
|
||||
"items_pending": 0,
|
||||
"items_claimed": 25,
|
||||
"items_running": 4,
|
||||
"items_completed": 21,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
},
|
||||
{
|
||||
"node_code": "unassigned",
|
||||
"items_total": 950,
|
||||
"items_pending": 950,
|
||||
"items_claimed": 0,
|
||||
"items_running": 0,
|
||||
"items_completed": 0,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
},
|
||||
]
|
||||
def test_append_detect_job_event_skips_missing_job_fk(self) -> None:
|
||||
class FakeCursor:
|
||||
def __init__(self) -> None:
|
||||
self.exec_calls = []
|
||||
self.selects = 0
|
||||
|
||||
node_stats = _build_effective_node_stats(
|
||||
distributed_node_stats=distributed_node_stats,
|
||||
raw_items_total=1000,
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def execute(self, sql, params=None):
|
||||
self.exec_calls.append((sql, params))
|
||||
if "INSERT INTO detect_run_events" in sql:
|
||||
raise AssertionError("should not insert detect_run_events when job row is missing")
|
||||
if "SELECT 1 FROM detect_jobs" in sql:
|
||||
self.selects += 1
|
||||
|
||||
def fetchone(self):
|
||||
return None
|
||||
|
||||
class FakeConn:
|
||||
def __init__(self) -> None:
|
||||
self.cursor_obj = FakeCursor()
|
||||
self.commit_calls = 0
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def cursor(self):
|
||||
return self.cursor_obj
|
||||
|
||||
def commit(self):
|
||||
self.commit_calls += 1
|
||||
|
||||
fake_conn = FakeConn()
|
||||
|
||||
with patch("app.services.detect_job_service.get_db", return_value=fake_conn):
|
||||
with patch("app.services.detect_job_service.push_debug_event") as mock_push:
|
||||
append_detect_job_event(
|
||||
5,
|
||||
event_type="job_dispatch_requested",
|
||||
message="控制面已发送检测启动请求",
|
||||
)
|
||||
|
||||
self.assertEqual(1, fake_conn.cursor_obj.selects)
|
||||
self.assertEqual(0, fake_conn.commit_calls)
|
||||
mock_push.assert_called_once()
|
||||
|
||||
def test_process_detect_pipeline_now_retries_deadlock_once(self) -> None:
|
||||
calls = {"count": 0}
|
||||
|
||||
def _run_pipeline(*, limit, job_id):
|
||||
calls["count"] += 1
|
||||
if calls["count"] == 1:
|
||||
raise errors.DeadlockDetected()
|
||||
return {
|
||||
"processed_items": 7,
|
||||
"advanced_items": 3,
|
||||
"retried_items": 1,
|
||||
}
|
||||
|
||||
with patch("app.services.detect_job_service.process_detect_pipeline", side_effect=_run_pipeline):
|
||||
with patch("app.services.detect_job_service.time.sleep") as mock_sleep:
|
||||
ok, message, data = process_detect_pipeline_now(limit=123, job_id=45)
|
||||
|
||||
self.assertTrue(ok)
|
||||
self.assertIn("deadlock 自动重试 1 次后成功", message)
|
||||
self.assertEqual(1, data["retry_attempts"])
|
||||
self.assertEqual(7, data["processed_items"])
|
||||
self.assertEqual(2, calls["count"])
|
||||
mock_sleep.assert_called_once()
|
||||
|
||||
def test_classify_runtime_debug_event_maps_single_step_finalized_completed(self) -> None:
|
||||
classified = _classify_runtime_debug_event(
|
||||
event_type="worker_log",
|
||||
message="检测步骤跟踪: domain=example.com | step=注册状态检测 | stage=single_step_finalized | elapsed_ms=2700 | ok=1 | detect_key=detect_register | final_status=completed | result_state=passed",
|
||||
payload={"job_code": "sync-overseas-27456", "node_code": "mainland-worker-01"},
|
||||
)
|
||||
|
||||
self.assertEqual(4, len(node_stats))
|
||||
unassigned = next(item for item in node_stats if item["node_code"] == "unassigned")
|
||||
self.assertEqual(680, unassigned["items_total"])
|
||||
self.assertEqual(680, unassigned["items_pending"])
|
||||
self.assertEqual("central_queue", unassigned["metrics_source"])
|
||||
self.assertEqual("sync-overseas-27456", classified["job_code"])
|
||||
self.assertTrue(classified["terminal"])
|
||||
self.assertEqual("completed", classified["terminal_status"])
|
||||
self.assertEqual("detect_register", classified["step_code"])
|
||||
|
||||
def test_build_effective_summary_uses_effective_node_stats_and_raw_blacklisted(self) -> None:
|
||||
node_stats = [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"items_total": 200,
|
||||
"items_pending": 195,
|
||||
"items_claimed": 0,
|
||||
"items_running": 5,
|
||||
"items_completed": 0,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"items_total": 70,
|
||||
"items_pending": 56,
|
||||
"items_claimed": 9,
|
||||
"items_running": 5,
|
||||
"items_completed": 0,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
},
|
||||
{
|
||||
"node_code": "overseas-control-01",
|
||||
"items_total": 50,
|
||||
"items_pending": 0,
|
||||
"items_claimed": 25,
|
||||
"items_running": 4,
|
||||
"items_completed": 21,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
},
|
||||
{
|
||||
"node_code": "unassigned",
|
||||
"items_total": 680,
|
||||
"items_pending": 680,
|
||||
"items_claimed": 0,
|
||||
"items_running": 0,
|
||||
"items_completed": 0,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
},
|
||||
]
|
||||
|
||||
summary = _build_effective_summary(
|
||||
node_stats=node_stats,
|
||||
raw_items_total=1000,
|
||||
raw_items_blacklisted=3,
|
||||
def test_build_runtime_display_bucket_prefers_real_active_threads_over_raw_current_load(self) -> None:
|
||||
bucket = _build_runtime_display_bucket(
|
||||
(
|
||||
"mainland-worker-01",
|
||||
"mainland",
|
||||
"worker",
|
||||
"busy",
|
||||
753,
|
||||
{
|
||||
"job_items_total": 1000,
|
||||
"job_items_claimed": 0,
|
||||
"job_items_running": 0,
|
||||
"job_items_completed": 342,
|
||||
"active_threads": 61,
|
||||
"max_threads": 400,
|
||||
"detect_participating": True,
|
||||
},
|
||||
None,
|
||||
)
|
||||
)
|
||||
|
||||
self.assertIsNotNone(bucket)
|
||||
self.assertEqual(61, bucket["current_load"])
|
||||
self.assertEqual(61, bucket["display_running"])
|
||||
self.assertEqual(61, bucket["active_threads"])
|
||||
|
||||
def test_build_runtime_display_bucket_skips_local_overseas_control_plane_load(self) -> None:
|
||||
with patch("app.services.detect_job_service.settings.node_region", "overseas"):
|
||||
with patch("app.services.detect_job_service.settings.node_role", "control"):
|
||||
with patch("app.services.detect_job_service.settings.node_code", "overseas-control-01"):
|
||||
bucket = _build_runtime_display_bucket(
|
||||
(
|
||||
"overseas-control-01",
|
||||
"overseas",
|
||||
"control",
|
||||
"busy",
|
||||
371,
|
||||
{
|
||||
"active_threads": 371,
|
||||
"max_threads": 3200,
|
||||
"detect_participating": False,
|
||||
},
|
||||
None,
|
||||
)
|
||||
)
|
||||
|
||||
self.assertIsNone(bucket)
|
||||
|
||||
def test_build_runtime_display_bucket_skips_any_control_node_without_worker_support(self) -> None:
|
||||
bucket = _build_runtime_display_bucket(
|
||||
(
|
||||
"overseas-control-01",
|
||||
"overseas",
|
||||
"control",
|
||||
"busy",
|
||||
378,
|
||||
{
|
||||
"active_threads": 378,
|
||||
"max_threads": 3200,
|
||||
"detect_participating": False,
|
||||
"worker_online": False,
|
||||
},
|
||||
None,
|
||||
)
|
||||
)
|
||||
|
||||
self.assertIsNone(bucket)
|
||||
|
||||
def test_build_display_summary_ignores_inflated_raw_current_load(self) -> None:
|
||||
summary = _build_display_summary(
|
||||
[
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"items_claimed": 0,
|
||||
"items_running": 0,
|
||||
"display_running": 729,
|
||||
"current_load": 729,
|
||||
"active_threads": 729,
|
||||
"max_threads": 800,
|
||||
"items_completed": 196,
|
||||
"items_failed": 0,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"items_claimed": 0,
|
||||
"items_running": 0,
|
||||
"display_running": 753,
|
||||
"current_load": 753,
|
||||
"active_threads": 61,
|
||||
"max_threads": 400,
|
||||
"items_completed": 146,
|
||||
"items_failed": 0,
|
||||
},
|
||||
]
|
||||
)
|
||||
|
||||
self.assertEqual(790, summary["display_running"])
|
||||
self.assertEqual(790, summary["current_load"])
|
||||
self.assertEqual(790, summary["active_threads"])
|
||||
|
||||
def test_enrich_active_job_summary_with_runtime_adds_runtime_job_code_and_recent_events(self) -> None:
|
||||
summary = _enrich_active_job_summary_with_runtime(
|
||||
{
|
||||
"job_id": 275,
|
||||
"job_code": "sync-overseas-5297",
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0,
|
||||
"completed_recent": 0,
|
||||
"failed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
"recent_domain_events": [],
|
||||
},
|
||||
event_limit=20,
|
||||
window_minutes=15,
|
||||
runtime_activity={
|
||||
"focus_job_code": "sync-overseas-9506",
|
||||
"job_codes": ["sync-overseas-9506", "sync-overseas-27456"],
|
||||
"processed_recent": 120,
|
||||
"completed_recent": 118,
|
||||
"failed_recent": 1,
|
||||
"blacklisted_recent": 1,
|
||||
},
|
||||
runtime_snapshot={
|
||||
"job": {
|
||||
"job_id": 1835,
|
||||
"job_code": "sync-overseas-9506",
|
||||
"progress_percent": 44.2,
|
||||
},
|
||||
"queue_health": {
|
||||
"queue": {
|
||||
"items_total": 1000,
|
||||
"pending": 176,
|
||||
"claimed": 223,
|
||||
"running": 159,
|
||||
"completed": 442,
|
||||
"display_running": 220,
|
||||
}
|
||||
},
|
||||
},
|
||||
recent_domain_events=[
|
||||
{
|
||||
"event_type": "domain_completed",
|
||||
"message": "域名检测完成: example.com",
|
||||
"created_at": "2026-04-21 04:10:00",
|
||||
}
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual("sync-overseas-9506", summary["runtime_job_code"])
|
||||
self.assertEqual(["sync-overseas-9506", "sync-overseas-27456"], summary["runtime_job_codes"])
|
||||
self.assertEqual(120, summary["processed_recent"])
|
||||
self.assertEqual(8.0, summary["processed_per_minute"])
|
||||
self.assertEqual(118, summary["completed_recent"])
|
||||
self.assertEqual(1, summary["failed_recent"])
|
||||
self.assertEqual(1, summary["blacklisted_recent"])
|
||||
self.assertEqual(1, len(summary["recent_domain_events"]))
|
||||
self.assertEqual("sync-overseas-9506", summary["runtime_snapshot_job_code"])
|
||||
self.assertEqual(1000, summary["runtime_snapshot_queue"]["items_total"])
|
||||
self.assertEqual(1000, summary["items_total"])
|
||||
self.assertEqual(931, summary["items_pending"])
|
||||
self.assertEqual(34, summary["items_claimed"])
|
||||
self.assertEqual(14, summary["items_running"])
|
||||
self.assertEqual(21, summary["items_completed"])
|
||||
self.assertEqual(3, summary["items_blacklisted"])
|
||||
self.assertEqual(24, summary["items_terminal"])
|
||||
self.assertEqual(176, summary["items_pending"])
|
||||
self.assertEqual(223, summary["items_claimed"])
|
||||
self.assertEqual(159, summary["items_running"])
|
||||
self.assertEqual(442, summary["items_completed"])
|
||||
self.assertEqual(220, summary["display_items_running"])
|
||||
|
||||
def test_enrich_active_job_summary_with_runtime_prefers_snapshot_identity_and_events(self) -> None:
|
||||
summary = _enrich_active_job_summary_with_runtime(
|
||||
{
|
||||
"job_id": 275,
|
||||
"job_code": "sync-overseas-5297",
|
||||
"status": "running",
|
||||
"recent_events": [
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"event_type": "domain_started",
|
||||
"message": "开始检测域名: stale.com",
|
||||
"payload": {"job_code": "sync-overseas-5297"},
|
||||
"created_at": "2026-04-21 11:58:00",
|
||||
}
|
||||
],
|
||||
"current_cycle_events": [],
|
||||
"latest_event": None,
|
||||
"display_items_running": 0,
|
||||
"display_current_load": 0,
|
||||
"display_active_threads": 0,
|
||||
"display_max_threads": 0,
|
||||
"display_active_node_codes": [],
|
||||
},
|
||||
event_limit=10,
|
||||
window_minutes=15,
|
||||
runtime_activity={
|
||||
"focus_job_code": "sync-overseas-31437",
|
||||
"job_codes": ["sync-overseas-31437"],
|
||||
"processed_recent": 10,
|
||||
"completed_recent": 10,
|
||||
"failed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
},
|
||||
runtime_snapshot={
|
||||
"job": {
|
||||
"job_id": 1902,
|
||||
"job_code": "sync-overseas-31437",
|
||||
"status": "running",
|
||||
"progress_percent": 34.3,
|
||||
"node_stats": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"items_claimed": 230,
|
||||
"items_running": 67,
|
||||
"items_completed": 343,
|
||||
"items_failed": 0,
|
||||
"active_threads": 170,
|
||||
"max_threads": 2000,
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"items_claimed": 75,
|
||||
"items_running": 0,
|
||||
"items_completed": 0,
|
||||
"items_failed": 0,
|
||||
"active_threads": 19,
|
||||
"max_threads": 1200,
|
||||
},
|
||||
],
|
||||
},
|
||||
"queue_health": {
|
||||
"queue": {
|
||||
"items_total": 1000,
|
||||
"pending": 285,
|
||||
"claimed": 305,
|
||||
"running": 67,
|
||||
"completed": 343,
|
||||
"display_running": 189,
|
||||
}
|
||||
},
|
||||
"recent_events": [
|
||||
{
|
||||
"job_id": 2137,
|
||||
"node_code": "mainland-controller-01",
|
||||
"event_type": "job_created",
|
||||
"message": "同步拉取待检测批次 sync-overseas-35461，共 1000 个任务项",
|
||||
"payload": {"source_record_id": 35461},
|
||||
"created_at": "2026-04-22 01:21:29",
|
||||
},
|
||||
{
|
||||
"job_id": 1902,
|
||||
"node_code": "mainland-controller-01",
|
||||
"event_type": "worker_log",
|
||||
"message": "从任务队列获取到 125 个需要检测的域名",
|
||||
"payload": {"job_code": "sync-overseas-31437"},
|
||||
"created_at": "2026-04-22 01:19:25",
|
||||
},
|
||||
],
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual(1902, summary["job_id"])
|
||||
self.assertEqual("sync-overseas-31437", summary["job_code"])
|
||||
self.assertEqual("sync-overseas-31437", summary["runtime_job_code"])
|
||||
self.assertEqual(189, summary["display_items_running"])
|
||||
self.assertEqual(189, summary["display_current_load"])
|
||||
self.assertEqual(189, summary["display_active_threads"])
|
||||
self.assertEqual(3200, summary["display_max_threads"])
|
||||
self.assertEqual(["mainland-controller-01", "mainland-worker-01"], summary["display_active_node_codes"])
|
||||
self.assertEqual(1, len(summary["recent_events"]))
|
||||
self.assertEqual("worker_log", summary["latest_event"]["event_type"])
|
||||
self.assertEqual("sync-overseas-31437", summary["recent_events"][0]["payload"]["job_code"])
|
||||
|
||||
def test_get_detect_queue_health_preserves_runtime_node_throughput_after_snapshot_override(self) -> None:
|
||||
lease_row = (None, None, 0, 0)
|
||||
throughput_rows = []
|
||||
step_throughput_rows = []
|
||||
runtime_display_rows = [
|
||||
(
|
||||
"mainland-controller-01",
|
||||
"mainland",
|
||||
"control",
|
||||
"busy",
|
||||
170,
|
||||
{
|
||||
"job_items_total": 265,
|
||||
"job_items_claimed": 85,
|
||||
"job_items_running": 0,
|
||||
"job_items_completed": 105,
|
||||
"active_threads": 170,
|
||||
"max_threads": 2000,
|
||||
"detect_participating": True,
|
||||
},
|
||||
None,
|
||||
),
|
||||
(
|
||||
"mainland-worker-01",
|
||||
"mainland",
|
||||
"worker",
|
||||
"busy",
|
||||
19,
|
||||
{
|
||||
"job_items_total": 139,
|
||||
"job_items_claimed": 64,
|
||||
"job_items_running": 0,
|
||||
"job_items_completed": 4,
|
||||
"active_threads": 19,
|
||||
"max_threads": 1200,
|
||||
"detect_participating": True,
|
||||
},
|
||||
None,
|
||||
),
|
||||
(
|
||||
"overseas-control-01",
|
||||
"overseas",
|
||||
"control",
|
||||
"busy",
|
||||
371,
|
||||
{
|
||||
"active_threads": 371,
|
||||
"max_threads": 3200,
|
||||
"detect_participating": False,
|
||||
},
|
||||
None,
|
||||
),
|
||||
]
|
||||
|
||||
conn = MagicMock()
|
||||
cursor_cm = MagicMock()
|
||||
cursor = MagicMock()
|
||||
conn.cursor.return_value = cursor_cm
|
||||
cursor_cm.__enter__.return_value = cursor
|
||||
db_cm = MagicMock()
|
||||
db_cm.__enter__.return_value = conn
|
||||
cursor.fetchone.return_value = lease_row
|
||||
cursor.fetchall.side_effect = [throughput_rows, step_throughput_rows, runtime_display_rows]
|
||||
|
||||
active_job = {
|
||||
"job_id": 1937,
|
||||
"job_code": "sync-overseas-31987",
|
||||
"status": "running",
|
||||
"items_total": 1000,
|
||||
"items_pending": 596,
|
||||
"items_claimed": 149,
|
||||
"items_running": 146,
|
||||
"items_completed": 109,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
"progress_percent": 10.9,
|
||||
"node_stats": [],
|
||||
"distributed_node_stats": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"items_total": 265,
|
||||
"items_pending": 0,
|
||||
"items_claimed": 85,
|
||||
"items_running": 75,
|
||||
"items_completed": 105,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
"metrics_source": "runtime",
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"items_total": 139,
|
||||
"items_pending": 0,
|
||||
"items_claimed": 64,
|
||||
"items_running": 71,
|
||||
"items_completed": 4,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
"metrics_source": "runtime",
|
||||
},
|
||||
],
|
||||
"step_stats": [
|
||||
{
|
||||
"step_code": "detect_register",
|
||||
"items_total": 1000,
|
||||
"items_pending": 449,
|
||||
"items_claimed": 131,
|
||||
"items_running": 273,
|
||||
"items_completed": 147,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
}
|
||||
],
|
||||
}
|
||||
runtime_activity = {
|
||||
"processed_recent": 133,
|
||||
"completed_recent": 113,
|
||||
"failed_recent": 20,
|
||||
"blacklisted_recent": 0,
|
||||
"focus_job_code": "sync-overseas-31563",
|
||||
"job_codes": ["sync-overseas-31563"],
|
||||
"step_code": "detect_register",
|
||||
"nodes": {
|
||||
"mainland-controller-01": {
|
||||
"node_code": "mainland-controller-01",
|
||||
"processed_recent": 74,
|
||||
"completed_recent": 54,
|
||||
"failed_recent": 20,
|
||||
"blacklisted_recent": 0,
|
||||
},
|
||||
"mainland-worker-01": {
|
||||
"node_code": "mainland-worker-01",
|
||||
"processed_recent": 59,
|
||||
"completed_recent": 59,
|
||||
"failed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
},
|
||||
},
|
||||
}
|
||||
runtime_snapshot = {
|
||||
"job": {
|
||||
"job_id": 1937,
|
||||
"job_code": "sync-overseas-31987",
|
||||
"progress_percent": 10.9,
|
||||
},
|
||||
"queue_health": {
|
||||
"queue": {
|
||||
"items_total": 1000,
|
||||
"pending": 596,
|
||||
"claimed": 149,
|
||||
"running": 146,
|
||||
"completed": 109,
|
||||
"blacklisted": 0,
|
||||
"failed": 0,
|
||||
"display_claimed": 149,
|
||||
"display_running": 403,
|
||||
},
|
||||
"nodes": [
|
||||
{
|
||||
"node_code": "mainland-controller-01",
|
||||
"items_total": 265,
|
||||
"items_pending": 0,
|
||||
"items_claimed": 85,
|
||||
"items_running": 75,
|
||||
"items_completed": 105,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0,
|
||||
"completed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
"failed_recent": 0,
|
||||
"metrics_source": "runtime",
|
||||
},
|
||||
{
|
||||
"node_code": "mainland-worker-01",
|
||||
"items_total": 139,
|
||||
"items_pending": 0,
|
||||
"items_claimed": 64,
|
||||
"items_running": 71,
|
||||
"items_completed": 4,
|
||||
"items_blacklisted": 0,
|
||||
"items_failed": 0,
|
||||
"processed_recent": 0,
|
||||
"processed_per_minute": 0,
|
||||
"completed_recent": 0,
|
||||
"blacklisted_recent": 0,
|
||||
"failed_recent": 0,
|
||||
"metrics_source": "runtime",
|
||||
},
|
||||
],
|
||||
"steps": [],
|
||||
},
|
||||
}
|
||||
|
||||
with patch("app.services.detect_job_service.settings.node_region", "overseas"):
|
||||
with patch("app.services.detect_job_service.settings.node_role", "control"):
|
||||
with patch("app.services.detect_job_service.settings.node_code", "overseas-control-01"):
|
||||
with patch("app.services.detect_job_service.get_active_detect_job_summary", return_value=active_job):
|
||||
with patch("app.services.detect_job_service._load_runtime_activity_snapshot", return_value=runtime_activity):
|
||||
with patch("app.services.detect_job_service._load_latest_runtime_active_job_snapshot", return_value=runtime_snapshot):
|
||||
with patch("app.services.detect_job_service.get_db", return_value=db_cm):
|
||||
health = get_detect_queue_health(window_minutes=15)
|
||||
|
||||
node_map = {item["node_code"]: item for item in health["nodes"]}
|
||||
self.assertEqual(74, node_map["mainland-controller-01"]["processed_recent"])
|
||||
self.assertEqual(54, node_map["mainland-controller-01"]["completed_recent"])
|
||||
self.assertEqual(20, node_map["mainland-controller-01"]["failed_recent"])
|
||||
self.assertEqual(170, node_map["mainland-controller-01"]["items_running"])
|
||||
self.assertEqual(59, node_map["mainland-worker-01"]["processed_recent"])
|
||||
self.assertEqual(59, node_map["mainland-worker-01"]["completed_recent"])
|
||||
self.assertEqual(19, node_map["mainland-worker-01"]["items_running"])
|
||||
self.assertEqual(113, health["throughput"]["completed_recent"])
|
||||
self.assertEqual(189, health["queue"]["display_running"])
|
||||
self.assertNotIn("overseas-control-01", node_map)
|
||||
|
||||
def test_normalize_detect_step_code_accepts_supported_single_step(self) -> None:
|
||||
self.assertEqual("detect_baidu_site", normalize_detect_step_code("detect_baidu_site"))
|
||||
self.assertEqual("detect_wayback", normalize_detect_step_code("detect_wayback"))
|
||||
|
||||
def test_normalize_detect_step_code_rejects_unknown_step(self) -> None:
|
||||
self.assertEqual("", normalize_detect_step_code("detect_unknown"))
|
||||
|
||||
def test_resolve_detect_job_definition_builds_single_step_job(self) -> None:
|
||||
definition = resolve_detect_job_definition("detect_baidu_site")
|
||||
|
||||
self.assertTrue(definition["is_single_step"])
|
||||
self.assertEqual("single_step", definition["task_mode"])
|
||||
self.assertEqual("detect_baidu_site", definition["step_code"])
|
||||
|
||||
def test_resolve_detect_job_definition_builds_wayback_single_step_job(self) -> None:
|
||||
definition = resolve_detect_job_definition("detect_wayback")
|
||||
|
||||
self.assertTrue(definition["is_single_step"])
|
||||
self.assertEqual("single_step", definition["task_mode"])
|
||||
self.assertEqual("detect_wayback", definition["step_code"])
|
||||
|
||||
def test_build_step_payload_adds_wayback_recent_years_strategy(self) -> None:
|
||||
payload = _build_step_payload(
|
||||
step_code="detect_wayback",
|
||||
domain_snapshot={"domain": "example.com", "source_type": 2},
|
||||
settings_payload={"detect_options": {"detect_wayback": True}},
|
||||
)
|
||||
|
||||
self.assertEqual("detect_wayback", payload["step_code"])
|
||||
self.assertEqual("recent_years", payload["wayback_strategy"])
|
||||
self.assertEqual(5, payload["wayback_recent_years"])
|
||||
self.assertTrue(payload["wayback_stop_on_first_hit"])
|
||||
|
||||
def test_resolve_detect_job_definition_defaults_to_domain_pipeline(self) -> None:
|
||||
definition = resolve_detect_job_definition(None)
|
||||
|
||||
self.assertFalse(definition["is_single_step"])
|
||||
self.assertEqual("domain_pipeline", definition["task_mode"])
|
||||
self.assertEqual("", definition["step_code"])
|
||||
|
||||
def test_resolve_domain_pipeline_step_skips_yikoujia_register(self) -> None:
|
||||
step_code = resolve_domain_pipeline_step(
|
||||
{
|
||||
"source_type": 1,
|
||||
"register_status": 0,
|
||||
"baidu_site": {},
|
||||
"qihu360_site": {},
|
||||
"chinaz_info": {},
|
||||
"aizhan_info": {},
|
||||
"wayback_info": {},
|
||||
"jucha_info": {},
|
||||
"juziseo_info": {},
|
||||
},
|
||||
settings_payload={
|
||||
"detect_options": {
|
||||
"detect_register": True,
|
||||
"detect_baidu_site": True,
|
||||
"detect_360_site": False,
|
||||
"detect_chinaz": False,
|
||||
"detect_aizhan": False,
|
||||
"detect_wayback": False,
|
||||
"detect_jucha": False,
|
||||
"detect_juziseo": False,
|
||||
"detect_order": ["detect_register", "detect_baidu_site"],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual("detect_baidu_site", step_code)
|
||||
|
||||
def test_resolve_domain_pipeline_step_moves_to_next_incomplete_step(self) -> None:
|
||||
step_code = resolve_domain_pipeline_step(
|
||||
{
|
||||
"source_type": 2,
|
||||
"register_status": 3,
|
||||
"baidu_site": {"state": "passed"},
|
||||
"qihu360_site": {},
|
||||
"chinaz_info": {},
|
||||
"aizhan_info": {},
|
||||
"wayback_info": {},
|
||||
"jucha_info": {},
|
||||
"juziseo_info": {},
|
||||
},
|
||||
settings_payload={
|
||||
"detect_options": {
|
||||
"detect_register": True,
|
||||
"detect_baidu_site": True,
|
||||
"detect_360_site": True,
|
||||
"detect_chinaz": False,
|
||||
"detect_aizhan": False,
|
||||
"detect_wayback": False,
|
||||
"detect_jucha": False,
|
||||
"detect_juziseo": False,
|
||||
"detect_order": ["detect_register", "detect_baidu_site", "detect_360_site"],
|
||||
}
|
||||
},
|
||||
after_step_code="detect_baidu_site",
|
||||
)
|
||||
|
||||
self.assertEqual("detect_360_site", step_code)
|
||||
|
||||
def test_resolve_initial_domain_pipeline_item_builds_first_step_payload(self) -> None:
|
||||
step_code, payload = resolve_initial_domain_pipeline_item(
|
||||
{
|
||||
"id": 10,
|
||||
"domain": "example.com",
|
||||
"source_type": 2,
|
||||
"register_status": 0,
|
||||
"baidu_site": {},
|
||||
"qihu360_site": {},
|
||||
"chinaz_info": {},
|
||||
"aizhan_info": {},
|
||||
"wayback_info": {},
|
||||
"jucha_info": {},
|
||||
"juziseo_info": {},
|
||||
},
|
||||
settings_payload={
|
||||
"detect_options": {
|
||||
"detect_register": True,
|
||||
"detect_baidu_site": True,
|
||||
"detect_order": ["detect_register", "detect_baidu_site"],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual("detect_register", step_code)
|
||||
self.assertIsNotNone(payload)
|
||||
self.assertEqual("detect_register", payload["step_code"])
|
||||
self.assertEqual("example.com", payload["domain"])
|
||||
|
||||
def test_resolve_initial_domain_pipeline_item_returns_empty_when_pipeline_already_done(self) -> None:
|
||||
step_code, payload = resolve_initial_domain_pipeline_item(
|
||||
{
|
||||
"id": 11,
|
||||
"domain": "done.com",
|
||||
"source_type": 2,
|
||||
"register_status": 3,
|
||||
"baidu_site": {"state": "passed"},
|
||||
"qihu360_site": {},
|
||||
"chinaz_info": {},
|
||||
"aizhan_info": {},
|
||||
"wayback_info": {},
|
||||
"jucha_info": {},
|
||||
"juziseo_info": {},
|
||||
},
|
||||
settings_payload={
|
||||
"detect_options": {
|
||||
"detect_register": True,
|
||||
"detect_baidu_site": True,
|
||||
"detect_order": ["detect_register", "detect_baidu_site"],
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual("", step_code)
|
||||
self.assertIsNone(payload)
|
||||
|
||||
def test_classify_pipeline_item_outcome_retries_external_failure(self) -> None:
|
||||
outcome = _classify_pipeline_item_outcome(
|
||||
item_status="failed",
|
||||
result_payload={"state": "degraded", "message": "timeout", "retry_recommended": True},
|
||||
step_code="detect_baidu_site",
|
||||
attempt_count=0,
|
||||
)
|
||||
|
||||
self.assertEqual("retry", outcome["action"])
|
||||
self.assertTrue(outcome["should_retry"])
|
||||
self.assertEqual("external_retry", outcome["reason_code"])
|
||||
|
||||
def test_classify_pipeline_item_outcome_rejects_business_failure(self) -> None:
|
||||
outcome = _classify_pipeline_item_outcome(
|
||||
item_status="failed",
|
||||
result_payload={"state": "rejected", "message": "title contains forbidden keyword"},
|
||||
step_code="detect_chinaz",
|
||||
attempt_count=0,
|
||||
)
|
||||
|
||||
self.assertEqual("reject", outcome["action"])
|
||||
self.assertFalse(outcome["should_retry"])
|
||||
self.assertEqual("business_reject", outcome["reason_code"])
|
||||
|
||||
def test_classify_pipeline_item_outcome_marks_blacklisted_terminal(self) -> None:
|
||||
outcome = _classify_pipeline_item_outcome(
|
||||
item_status="blacklisted",
|
||||
result_payload={"state": "blacklisted", "message": "risk hit"},
|
||||
step_code="detect_baidu_site",
|
||||
attempt_count=0,
|
||||
)
|
||||
|
||||
self.assertEqual("black_hit", outcome["action"])
|
||||
self.assertFalse(outcome["should_retry"])
|
||||
self.assertEqual("blacklisted", outcome["reason_code"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
69
domain-api/tests/test_detect_service_status_fallback.py
Normal file
69
domain-api/tests/test_detect_service_status_fallback.py
Normal file
@@ -0,0 +1,69 @@
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from app.services import detect_service
|
||||
|
||||
|
||||
class DetectServiceStatusFallbackTests(unittest.TestCase):
|
||||
def test_get_detect_status_keeps_runtime_snapshot_when_db_is_unreachable(self) -> None:
|
||||
runtime_state = {
|
||||
"service_running": True,
|
||||
"detecting": True,
|
||||
"active_threads": 7,
|
||||
"max_threads": 120,
|
||||
"phase": "running",
|
||||
"detail": "Worker 正在处理 7 个检测任务",
|
||||
"updated_at": "2026-04-20 23:59:00",
|
||||
"available_proxy_count": 18,
|
||||
}
|
||||
|
||||
with patch("app.services.detect_service.ensure_runtime_schema"), \
|
||||
patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")), \
|
||||
patch("app.services.detect_service.get_settings_payload", return_value={"proxy_config": {"proxy_enable": True, "allow_direct": False, "proxy_urls": ["a"]}}), \
|
||||
patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "full"}), \
|
||||
patch("app.services.detect_service._load_recent_worker_lines", return_value=(True, "", [])), \
|
||||
patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": True, "process_count": 1, "latest_start_time": "2026-04-20 23:58:00", "message": "active/running"}), \
|
||||
patch("app.services.detect_service._load_runtime_state", return_value=runtime_state), \
|
||||
patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}), \
|
||||
patch("app.services.detect_service._extract_available_proxy_count", return_value=0), \
|
||||
patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}), \
|
||||
patch("app.services.detect_service._normalize_recent_warning", return_value=""), \
|
||||
patch("app.services.detect_service._build_proxy_runtime_snapshot", return_value={"state": "healthy", "label": "代理正常", "detail": "healthy", "direct_fallback_active": False, "reason": "healthy", "last_refresh_status": "ok", "last_refresh_time": "", "source_count": 2, "raw_items": 18, "validated_count": 18, "available_count": 18, "source_stats": [], "supplier_empty": False}), \
|
||||
patch("app.services.detect_service.resolve_thread_count", return_value={"effective_thread_count": 120, "default_thread_count": 5, "source": "node_override", "override_thread_count": 120, "node_code": "mainland-worker-01"}), \
|
||||
patch("app.services.detect_service.get_active_detect_job_summary", side_effect=RuntimeError("db down")), \
|
||||
patch("app.services.detect_service.sync_detect_runs", return_value=[]), \
|
||||
patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}), \
|
||||
patch("app.services.detect_service._extract_dependency_alerts", return_value=[]), \
|
||||
patch("app.services.detect_service.append_detect_result_projection_if_changed"):
|
||||
payload = detect_service.get_detect_status()
|
||||
|
||||
self.assertTrue(payload["worker_online"])
|
||||
self.assertTrue(payload["detecting"])
|
||||
self.assertEqual(7, payload["active_thread_count"])
|
||||
self.assertEqual(120, payload["max_thread_count"])
|
||||
self.assertEqual(0, payload["progress"]["pending"])
|
||||
self.assertEqual(0, payload["progress"]["completed"])
|
||||
|
||||
def test_filter_lines_since_supports_journalctl_syslog_timestamps(self) -> None:
|
||||
lines = [
|
||||
"Apr 21 20:12:42 mainland-controller python[1]: 当前实际线程数量: 323/4",
|
||||
"Apr 21 20:17:20 mainland-controller python[2]: Worker 已启动，等待检测指令",
|
||||
]
|
||||
|
||||
filtered = detect_service._filter_lines_since(lines, "2026-04-21 20:17:00")
|
||||
|
||||
self.assertEqual(
|
||||
["Apr 21 20:17:20 mainland-controller python[2]: Worker 已启动，等待检测指令"],
|
||||
filtered,
|
||||
)
|
||||
|
||||
def test_filter_lines_since_falls_back_when_no_timestamp_is_parseable(self) -> None:
|
||||
lines = ["no timestamp line 1", "no timestamp line 2"]
|
||||
|
||||
filtered = detect_service._filter_lines_since(lines, "2026-04-21 20:17:00")
|
||||
|
||||
self.assertEqual(lines, filtered)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
205
domain-api/tests/test_juming_services.py
Normal file
205
domain-api/tests/test_juming_services.py
Normal file
@@ -0,0 +1,205 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import contextmanager
|
||||
from unittest.mock import patch
|
||||
|
||||
from requests.cookies import RequestsCookieJar
|
||||
|
||||
from app.services import juming_service, juming_task_service
|
||||
|
||||
|
||||
class JumingServiceTests(unittest.TestCase):
|
||||
def test_crawl_juming_rejects_invalid_cookie(self) -> None:
|
||||
cookie_jar = RequestsCookieJar()
|
||||
cookie_jar.set("sid", "expired")
|
||||
|
||||
with patch.object(juming_service, "_load_juming_cookie", return_value=(cookie_jar, "local")):
|
||||
with patch.object(juming_service, "_validate_juming_cookie", return_value=(False, "聚名登录态已失效，请重新登录")):
|
||||
with self.assertRaisesRegex(ValueError, "已失效"):
|
||||
juming_service.crawl_juming({"mode": "delete_list"})
|
||||
|
||||
def test_get_juming_status_exposes_remote_validation(self) -> None:
|
||||
cookie_jar = RequestsCookieJar()
|
||||
cookie_jar.set("sid", "alive")
|
||||
|
||||
with patch.object(juming_service, "_load_juming_cookie", return_value=(cookie_jar, "local")):
|
||||
with patch.object(juming_service, "_validate_juming_cookie", return_value=(False, "聚名登录态已失效，请重新登录")):
|
||||
payload = juming_service.get_juming_status()
|
||||
|
||||
self.assertTrue(payload["cookie_present"])
|
||||
self.assertFalse(payload["cookie_valid"])
|
||||
self.assertFalse(payload["cookie_ready"])
|
||||
self.assertIn("已失效", payload["cookie_message"])
|
||||
|
||||
def test_insert_domains_uses_copy_stage_import_path(self) -> None:
|
||||
class FakeCursor:
|
||||
def __init__(self) -> None:
|
||||
self.executed: list[tuple[str, object]] = []
|
||||
self.copy_calls: list[tuple[str, tuple[str, ...], str]] = []
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
return False
|
||||
|
||||
def execute(self, sql: str, params: object = None) -> None:
|
||||
self.executed.append((sql, params))
|
||||
|
||||
def copy_from(self, file_obj, table: str, columns: tuple[str, ...]) -> None:
|
||||
self.copy_calls.append((table, columns, file_obj.read()))
|
||||
|
||||
def fetchone(self):
|
||||
return (2, 2, 0)
|
||||
|
||||
class FakeConn:
|
||||
def __init__(self) -> None:
|
||||
self.cursor_obj = FakeCursor()
|
||||
self.commit_calls = 0
|
||||
|
||||
def cursor(self):
|
||||
return self.cursor_obj
|
||||
|
||||
def commit(self) -> None:
|
||||
self.commit_calls += 1
|
||||
|
||||
fake_conn = FakeConn()
|
||||
|
||||
@contextmanager
|
||||
def fake_get_db():
|
||||
yield fake_conn
|
||||
|
||||
with patch.object(juming_service, "get_db", fake_get_db):
|
||||
with patch.object(juming_service, "IMPORT_BATCH_SIZE", 2):
|
||||
stats = juming_service._insert_domains(
|
||||
["alpha.com", "beta.net"],
|
||||
juming_service.DELETE_LIST_SOURCE_TYPE,
|
||||
)
|
||||
|
||||
self.assertEqual(2, stats["added"])
|
||||
self.assertEqual(1, fake_conn.commit_calls)
|
||||
self.assertTrue(fake_conn.cursor_obj.copy_calls)
|
||||
table, columns, payload = fake_conn.cursor_obj.copy_calls[0]
|
||||
self.assertEqual("juming_import_stage", table)
|
||||
self.assertEqual(("domain", "tld"), columns)
|
||||
self.assertIn("alpha.com\tcom", payload)
|
||||
self.assertIn("beta.net\tnet", payload)
|
||||
executed_sql = "\n".join(sql for sql, _params in fake_conn.cursor_obj.executed)
|
||||
self.assertIn("create temporary table if not exists juming_import_stage", executed_sql.lower())
|
||||
self.assertIn("inserted as", executed_sql.lower())
|
||||
self.assertIn("left join domains existing", executed_sql.lower())
|
||||
|
||||
def test_delete_list_import_skips_already_imported_same_signature(self) -> None:
|
||||
cookie_jar = RequestsCookieJar()
|
||||
cookie_jar.set("sid", "alive")
|
||||
|
||||
class FakeJM:
|
||||
cookie = cookie_jar
|
||||
|
||||
def new_cha_del(self, current_date: str):
|
||||
if current_date == "2026-03-21":
|
||||
return ["alpha.com", "beta.net"]
|
||||
return []
|
||||
|
||||
logs: list[str] = []
|
||||
signature = juming_service._compute_domains_signature(["alpha.com", "beta.net"])
|
||||
cached_state = {
|
||||
"2026-03-21": {
|
||||
"signature": signature,
|
||||
"total": 2,
|
||||
"valid": 2,
|
||||
"invalid": 0,
|
||||
}
|
||||
}
|
||||
|
||||
with patch.object(juming_service, "_load_juming_cookie", return_value=(cookie_jar, "local")):
|
||||
with patch.object(juming_service, "JM", return_value=FakeJM()):
|
||||
with patch.object(juming_service, "_load_delete_import_state", return_value=cached_state):
|
||||
with patch.object(juming_service, "_insert_domains") as mock_insert:
|
||||
result = juming_service._crawl_delete_list_and_import(
|
||||
"2026-03-21",
|
||||
False,
|
||||
log=logs.append,
|
||||
)
|
||||
|
||||
mock_insert.assert_not_called()
|
||||
self.assertEqual(0, result["stats"]["added"])
|
||||
self.assertEqual(2, result["stats"]["exists"])
|
||||
self.assertTrue(any("跳过重复入库" in line for line in logs))
|
||||
|
||||
|
||||
class JumingTaskServiceTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self._tmpdir = tempfile.TemporaryDirectory()
|
||||
self._old_runtime_root = os.environ.get("DOMAIN_API_RUNTIME_ROOT")
|
||||
os.environ["DOMAIN_API_RUNTIME_ROOT"] = self._tmpdir.name
|
||||
juming_task_service._ACTIVE_TASK_IDS.clear()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
juming_task_service._ACTIVE_TASK_IDS.clear()
|
||||
if self._old_runtime_root is None:
|
||||
os.environ.pop("DOMAIN_API_RUNTIME_ROOT", None)
|
||||
else:
|
||||
os.environ["DOMAIN_API_RUNTIME_ROOT"] = self._old_runtime_root
|
||||
self._tmpdir.cleanup()
|
||||
|
||||
def _write_tasks(self, records: list[dict]) -> None:
|
||||
path = os.path.join(self._tmpdir.name, "juming_tasks.json")
|
||||
with open(path, "w", encoding="utf-8") as handle:
|
||||
json.dump(records, handle, ensure_ascii=False, indent=2)
|
||||
|
||||
def test_cleanup_orphaned_tasks_marks_running_task_failed(self) -> None:
|
||||
self._write_tasks(
|
||||
[
|
||||
{
|
||||
"task_id": "task-1",
|
||||
"status": "running",
|
||||
"phase": "importing",
|
||||
"phase_label": "入库中",
|
||||
"cancel_requested": False,
|
||||
"message": "开始入库处理",
|
||||
"created_at": "2026-04-21 21:00:00",
|
||||
"updated_at": "2026-04-21 21:00:00",
|
||||
"started_at": "2026-04-21 21:00:00",
|
||||
"completed_at": "",
|
||||
"result": None,
|
||||
"logs": [],
|
||||
}
|
||||
]
|
||||
)
|
||||
|
||||
tasks = juming_task_service.list_juming_tasks()
|
||||
self.assertEqual("failed", tasks[0]["status"])
|
||||
self.assertIn("中断", tasks[0]["message"])
|
||||
|
||||
def test_create_task_rejects_parallel_active_task(self) -> None:
|
||||
self._write_tasks(
|
||||
[
|
||||
{
|
||||
"task_id": "task-1",
|
||||
"status": "running",
|
||||
"phase": "fetching",
|
||||
"phase_label": "抓取中",
|
||||
"cancel_requested": False,
|
||||
"message": "正在抓取",
|
||||
"created_at": "2026-04-21 21:00:00",
|
||||
"updated_at": "2026-04-21 21:00:00",
|
||||
"started_at": "2026-04-21 21:00:00",
|
||||
"completed_at": "",
|
||||
"result": None,
|
||||
"logs": [],
|
||||
}
|
||||
]
|
||||
)
|
||||
juming_task_service._ACTIVE_TASK_IDS.add("task-1")
|
||||
|
||||
with self.assertRaisesRegex(ValueError, "已有聚名采集任务正在运行"):
|
||||
juming_task_service.create_juming_task({"mode": "delete_list"})
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -20,6 +20,111 @@ class NodeAgentDeliveryQueueTests(unittest.TestCase):
|
||||
self.assertEqual(0, delivery_queue["pending_count"])
|
||||
self.assertEqual(0, delivery_queue["dead_letter_count"])
|
||||
|
||||
def test_base_payload_prefers_non_loopback_identity(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir), patch.object(
|
||||
node_agent,
|
||||
"CONTROL_PLANE_BASE_URL",
|
||||
"http://152.53.37.118:8100",
|
||||
), patch.object(
|
||||
node_agent.socket,
|
||||
"gethostname",
|
||||
return_value="localhost",
|
||||
), patch.object(
|
||||
node_agent.socket,
|
||||
"getfqdn",
|
||||
return_value="localhost.localdomain",
|
||||
), patch.object(
|
||||
node_agent.os,
|
||||
"uname",
|
||||
return_value=type("Uname", (), {"nodename": "localhost"})(),
|
||||
), patch.object(
|
||||
node_agent,
|
||||
"NODE_CODE",
|
||||
"mainland-controller-01",
|
||||
), patch.object(
|
||||
node_agent.socket,
|
||||
"getaddrinfo",
|
||||
return_value=[(None, None, None, None, ("127.0.0.1", 0))],
|
||||
), patch.object(
|
||||
node_agent.socket,
|
||||
"gethostbyname",
|
||||
return_value="127.0.0.1",
|
||||
):
|
||||
class FakeSocket:
|
||||
def connect(self, target):
|
||||
self.target = target
|
||||
|
||||
def getsockname(self):
|
||||
return ("121.204.244.188", 12345)
|
||||
|
||||
def close(self):
|
||||
return None
|
||||
|
||||
with patch.object(node_agent.socket, "socket", return_value=FakeSocket()):
|
||||
payload = node_agent._base_payload()
|
||||
|
||||
self.assertNotIn(payload["hostname"], {"", "localhost", "localhost.localdomain"})
|
||||
self.assertEqual("121.204.244.188", payload["ip"])
|
||||
|
||||
def test_detect_runtime_snapshot_degrades_to_worker_runtime_when_detect_status_fails(self) -> None:
|
||||
with patch.dict(os.environ, {}, clear=False):
|
||||
with patch(
|
||||
"app.services.worker_control_service.detect_worker_runtime",
|
||||
return_value={
|
||||
"running": True,
|
||||
"process_count": 1,
|
||||
"latest_start_time": "2026-04-20 18:00:00",
|
||||
"message": "active/running",
|
||||
},
|
||||
), patch(
|
||||
"app.services.detect_service.get_detect_status",
|
||||
side_effect=RuntimeError('connection to server at "127.0.0.1", port 5432 failed'),
|
||||
):
|
||||
snapshot = node_agent._detect_runtime_snapshot()
|
||||
|
||||
self.assertTrue(snapshot["worker_online"])
|
||||
self.assertTrue(snapshot["service_running"])
|
||||
self.assertFalse(snapshot["detecting"])
|
||||
self.assertEqual("active/running", snapshot["phase_detail"])
|
||||
self.assertEqual("2026-04-20 18:00:00", snapshot["updated_at"])
|
||||
self.assertIn("127.0.0.1", snapshot["error"])
|
||||
|
||||
def test_detect_runtime_snapshot_infers_worker_online_from_active_threads(self) -> None:
|
||||
with patch.dict(os.environ, {}, clear=False):
|
||||
with patch(
|
||||
"app.services.worker_control_service.detect_worker_runtime",
|
||||
return_value={
|
||||
"running": False,
|
||||
"process_count": 0,
|
||||
"latest_start_time": "",
|
||||
"message": "",
|
||||
},
|
||||
), patch(
|
||||
"app.services.detect_service.get_detect_status",
|
||||
return_value={
|
||||
"worker_online": False,
|
||||
"detecting": False,
|
||||
"active_thread_count": 19,
|
||||
"max_thread_count": 1200,
|
||||
"phase_label": "检测中",
|
||||
"phase_detail": "Worker 正在处理 162 个检测任务",
|
||||
"runtime_state": {
|
||||
"service_running": False,
|
||||
"updated_at": "2026-04-21 00:00:06",
|
||||
},
|
||||
"active_job": {"items_running": 0},
|
||||
},
|
||||
):
|
||||
snapshot = node_agent._detect_runtime_snapshot()
|
||||
|
||||
self.assertTrue(snapshot["worker_online"])
|
||||
self.assertTrue(snapshot["service_running"])
|
||||
self.assertTrue(snapshot["detecting"])
|
||||
self.assertTrue(snapshot["detect_participating"])
|
||||
self.assertEqual(19, snapshot["active_threads"])
|
||||
self.assertEqual(19, snapshot["current_load"])
|
||||
|
||||
def test_job_event_network_failure_is_queued_for_retry(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as temp_dir:
|
||||
with patch.object(node_agent, "AGENT_QUEUE_DIR", temp_dir), patch.object(
|
||||
@@ -279,6 +384,63 @@ class NodeAgentDeliveryQueueTests(unittest.TestCase):
|
||||
self.assertEqual("failed_local", event_payload["start_delivery_state"])
|
||||
self.assertIn("temporary offline", event_payload["start_delivery_error"])
|
||||
|
||||
def test_register_heartbeat_and_pull_use_configured_timeouts(self) -> None:
|
||||
with patch.object(node_agent, "_post", return_value={"code": 0, "message": "ok", "data": {"jobs": []}}) as mock_post, patch.object(
|
||||
node_agent,
|
||||
"AGENT_REGISTER_TIMEOUT_SECONDS",
|
||||
91,
|
||||
), patch.object(
|
||||
node_agent,
|
||||
"AGENT_HEARTBEAT_TIMEOUT_SECONDS",
|
||||
92,
|
||||
), patch.object(
|
||||
node_agent,
|
||||
"AGENT_PULL_TIMEOUT_SECONDS",
|
||||
93,
|
||||
):
|
||||
node_agent._register()
|
||||
node_agent._heartbeat()
|
||||
jobs = node_agent._pull_jobs()
|
||||
|
||||
self.assertEqual([], jobs)
|
||||
self.assertEqual(3, mock_post.call_count)
|
||||
self.assertEqual(91, mock_post.call_args_list[0].kwargs["timeout"])
|
||||
self.assertEqual(92, mock_post.call_args_list[1].kwargs["timeout"])
|
||||
self.assertEqual(93, mock_post.call_args_list[2].kwargs["timeout"])
|
||||
|
||||
def test_job_delivery_uses_configured_timeouts(self) -> None:
|
||||
with patch.object(
|
||||
node_agent,
|
||||
"_deliver_or_queue",
|
||||
side_effect=lambda **kwargs: {"state": "queued", "timeout": kwargs["timeout"]},
|
||||
) as mock_deliver, patch.object(
|
||||
node_agent,
|
||||
"AGENT_JOB_COMPLETE_TIMEOUT_SECONDS",
|
||||
94,
|
||||
), patch.object(
|
||||
node_agent,
|
||||
"AGENT_JOB_EVENT_TIMEOUT_SECONDS",
|
||||
47,
|
||||
):
|
||||
complete_result = node_agent._job_complete(
|
||||
11,
|
||||
status="success",
|
||||
stdout="ok",
|
||||
stderr="",
|
||||
result={"ok": True},
|
||||
)
|
||||
event_result = node_agent._job_event(
|
||||
11,
|
||||
event_type="executor_received",
|
||||
message="accepted",
|
||||
)
|
||||
|
||||
self.assertEqual("queued", complete_result["state"])
|
||||
self.assertEqual("queued", event_result["state"])
|
||||
self.assertEqual(2, mock_deliver.call_count)
|
||||
self.assertEqual(94, mock_deliver.call_args_list[0].kwargs["timeout"])
|
||||
self.assertEqual(47, mock_deliver.call_args_list[1].kwargs["timeout"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user