admin/getDomain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: stabilize multi-region runtime sync and worker orchestration

Browse Source
This commit is contained in:
root
2026-04-27 15:48:12 +08:00
parent 7cbde2aa78
commit 215a364891
137 changed files with 31931 additions and 1943 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
.gitignore vendored
View File

@@ -21,6 +21,7 @@ dist/
# Logs and runtime data
*.log
*.pid
/runtime/
/diagnostics/
/domain-api/runtime/
@@ -51,8 +52,27 @@ domain-api/deploy/multi-region/*.conf
domainCheck/tools/node-v20.19.4-win-x64/
domainCheck/app/credentials.json
domainCheck/credentials.json
domainCheck/runtime/
domainCheck/**/*.pkl
domainCheck/domains.txt
thread_count.json
node_thread_counts.json
runtime_settings.json
runtime/runtime_settings.json
runtime/sensitive_words.json
# Ops center generated night-run artifacts
docs/ops_center_runtime/night_runs/night_run_*/
docs/ops_center_runtime/night_runs/night_run_*_summary.json
docs/ops_center_runtime/night_runs/*.log
docs/ops_center_runtime/night_runs/*.pid
docs/ops_center_runtime/night_runs/step_mix_*/
docs/ops_center_runtime/chinaz_gray_runs/
docs/_tmp_regression/
# Local probe / backup artifacts
.codex-release-probe.txt
*.bak_*
# OS / editor
.DS_Store

84
docs/shenhe.md Normal file
View File

@@ -0,0 +1,84 @@
可以，按“尽量省额度但不丢高风险”的思路，我建议你把全项目审核拆成 3 轮，目标控制在 `800万 ~ 1200万 tokens`。
**先排除**
第一轮先不要审这些，不然额度会被白白吃掉：
- `release/`
- `.venv/`
- `node_modules/`
- `domain-web/package-lock.json`
- `domainCheck/app/sdk_leg.js`
- `domainCheck/detect/sdk_leg.js`
- 运行时产物、日志、快照、历史 night runs
- 大 JSON 词库这类静态数据，除非代码直接依赖逻辑可疑
**三轮顺序**
1. 控制面与数据正确性
预算：`300万 ~ 450万`
- `domain-api/app/services/detect_job_service.py`
- `domain-api/app/services/detect_service.py`
- `domain-api/app/services/runtime_status_service.py`
- `domain-api/app/services/dashboard.py`
- `domain-api/app/services/sync_record_service.py`
- `domain-api/app/services/sync_push_service.py`
- `domain-api/app/services/worker_control_service.py`
- `domain-api/app/services/settings_service.py`
- `domain-api/app/services/cluster_runtime_service.py`
- 对应 `routes/` 和关键测试
这一轮最值钱，因为它直接查：
- 页面显示为什么和现场不一致
- sync 为什么会把旧状态盖新状态
- runtime projection / queue health / active job 是否互相打架
- 配置下发和节点实际执行是否一致
2. Worker 并发与执行链
预算：`300万 ~ 400万`
- `domainCheck/detect_worker.py`
- `domainCheck/app/utils/database.py`
- `domainCheck/app/detectors/`
- `domainCheck/detect/`
- `domainCheck/tests/` 里和并发、连接池、超时、代理有关的测试
这一轮重点查：
- 多进程 / 多线程是否真能提升吞吐
- DB 连接池、代理池、任务领取链有没有硬瓶颈
- 内存为什么高、进程为什么空转
- 超时、重试、降级逻辑是否会拖垮吞吐
3. 发布、运维、前端展示
预算：`200万 ~ 300万`
- `domain-api/deploy/`
- `domain-api/app/node_agent.py`
- `domain-api/app/services/ops_*`
- `domain-web/src/views/detect/`
- `domain-web/src/views/runtime/`
- `domain-web/src/views/settings/`
- systemd 模板、迁移/发布脚本
这一轮重点查：
- 发布链和实际运行是否一致
- 多实例 worker 的部署是否完整
- 前端是否误导运维判断
- 迁移、接管、rollout 有没有高风险坑
**模型建议**
- 第 1 轮：`gpt-5.4 + xhigh`
- 第 2 轮：`gpt-5.4 + xhigh`
- 第 3 轮：`gpt-5.4 + high`
这样通常能把额度压在你要的区间里。
**输出方式**
每轮都只要这 3 类结果，最省额度：
- `P0/P1` 真实问题
- 影响面
- 修复建议
不要第一轮就让模型写大篇架构说明，不然额度会烧很快。
一句话版：
先审 `domain-api` 的状态/同步链，再审 `domainCheck` 的并发执行链，最后审 `deploy + ops + 前端展示`；按这个顺序，`800万~1200万 tokens` 是有机会压住的。
如果你要，我下一步可以直接给你生成一份“第一轮审核清单”，精确到文件名单。

58
docs/test.md
View File

@@ -62,7 +62,65 @@ controller syncer 批量同步海外主库
↓
controller finalizer 标记本次流程完成
发现问题
主体功能
1.去后台点击 聚名获取删除域名入库
2.按后台勾选 的检测选项，按顺序逐个去跑
3.正常逻辑是按顺序 一步一步前面的处理完了，再往下一个处理，最大进程跑起来，直到所有任务跑完
现在遇到的问题是：
1.后台展示文案的歧义非常大；运营很难理解
比喻托管节点：这个应该就是 服务器管理，现在你把进程也算是一个节点，全部放到这个列表，我看起来都蒙，如果需要你可以加多一个进程管理不就好了，不要混一起
现在后台应该有 机器/进程/线程，节点到底是啥？现在已经乱了，
运行配置这块也是 节点独立线程覆盖 你把所有进程都列出来配置 线程数量，这个不需要的，所有进程的线程数量全部走默认的，进程那么大不会人工管理的，设计很不合理，
检测管理页面：日志输出窗口 这块也是
参与节点：3 这个应该改成 参与 服务器
参与进程：62 参与 进程
参与 线程
运行中：127
线程：127 / 74000（参与节点汇总）
要让人一眼看明白
2.最重要一点，后台页面的数据展示很多都是不符合实际的，很多一点，为啥黑名单一直是0，这块不肯定的，这个正常清空最少命中80%；
进程和线程一直跑不起来，这个最致命，优化了好几日了，知道目前还是没跑通完整流程
3.概览
步骤队列
看每一步堆积、吞吐和失败，快速判断到底卡在注册、百度、360、爱站还是站长之家。 这块应该按后台勾选设置的顺序拍下来，正常任务完成也是，一个跑完才会往下推，才会往下一个跑
4.检测的进程和线程 机器 不稳定，不会自动检测 自动跑起来，就是最大性能没有跑起来
5.如果大陆controller 的DB链接数量是瓶颈，那可以每个大陆 机器都开启db+redis 反正每个机器的配置都很高的，只要有效率能提速
6.还有一个但海外机器绝对不参与worker 检测
一句话结论：
线上 hotfix 收口基本完成
下一步最该继续的是代理供给优化，不是回到代码审核
我下一步建议就直接转到代理链路，继续收：
为什么多实例同时刷新时会被代理源限流
是否要继续压低单实例补货批量/频率
是否要做更强的跨进程代理补货协调
大陆处理好的跑完流程域名是否返回海外机器勾选状态
域名检测流程设计

229
docs/后台运行观察口径.md Normal file
View File

@@ -0,0 +1,229 @@
# 后台运行观察口径
> 说明：这份文档主要解释页面怎么看。当前线上执行口径与判断顺序，统一以 [当前线上最终Runbook.md](/www/wwwroot/getDomain/docs/当前线上最终Runbook.md) 为准。
更新时间：`2026-04-24 14:24`
## 先说结论
以后不要先盯散乱日志。
日志只是辅助定位。
现在优先看后台 `运行中心` 第一屏。
那里已经会直接告诉你：
- 全库总盘子有多少
- 当前活跃批次到底有多少
- 近 15 分钟真实推进了多少
- 当前到底算不算真跑起来了
- 还有多少待处理
- 真正有多少进程和线程在干活
- 最近 15 分钟处理了多少
- 失败多不多
- 黑名单有没有推进
- 当前最忙的是哪些进程
- 主要积压卡在哪几个步骤
只有在后台页面临时不可用，或者你想在终端里持续盯时，才再用下面这条命令：
```bash
cd /www/wwwroot/getDomain
./.venv/bin/python tools/runtime_observer.py
```
如果想持续盯着看，就用：
```bash
cd /www/wwwroot/getDomain
./.venv/bin/python tools/runtime_observer.py --watch 5
```
它会把和后台首屏同一套核心数字收成一屏，不用再自己去拼：
- 当前任务是谁
- 还有多少待处理
- 真正有多少进程和线程在干活
- 最近 15 分钟到底处理了多少
- 失败多不多
- 黑名单有没有推进
- 哪几个节点真的在跑
- 每一步卡在哪
## 先看哪 4 组数
脚本里最重要的是这 4 行：
### 1. 任务积压
看：
- `pending`
- `claimed`
- `running`
这组数回答的是：
- 还有多少活没做
- 有没有任务已经被领走
- 当前有多少任务处于运行中
### 2. 结果产出
看：
- `completed`
- `failed`
- `blacklisted`
这组数回答的是：
- 成功推进了多少
- 失败了多少
- 黑名单命中了多少
### 3. 执行面
看：
- `active_processes`
- `active_threads`
- `max_threads`
这组数回答的是：
- 现在到底有多少执行实例真的在动
- 当前用了多少线程
- 当前理论上限是多少
### 4. 最近吞吐
看：
- `processed_recent`
- `per_minute`
- `failed_recent`
- `blacklisted_recent`
这组数回答的是：
- 最近 15 分钟有没有真实推进
- 大概每分钟能跑多少
- 最近失败是不是很多
- 最近有没有新的黑名单命中
## 怎么判断“真跑起来了”
满足下面这 3 条，才算真跑：
1. `pending` 还有积压
2. `active_threads` 不是 0
3. `processed_recent` 持续增长
如果只是页面上显示 `running`，但：
- `active_threads` 接近 0
- `processed_recent` 也接近 0
那更像是：
- 旧状态残影
- 口径没收准
- 或者任务挂着但没真正消化
## 怎么快速判断卡在哪
### 情况 1：`pending` 很高，`active_threads` 很低
说明更像：
- 没真正拉起执行面
- worker 没开始干活
- 或者任务没正确分发进去
### 情况 2：`active_threads` 不低，但 `failed_recent` 很高
说明更像：
- 不是没跑
- 而是外部步骤在大量失败
- 常见就是代理、RDAP、爱站、时光机这类链路超时
### 情况 3：`completed_recent` 很低，`blacklisted_recent` 也很低
说明更像：
- 线程虽然在跑
- 但结果大多没形成有效推进
- 更像在外部失败里空转
### 情况 4：节点列表里只有一台在有动作
说明更像：
- 当前真正承担执行的就一台
- 其他节点可能只是在线
- 或者只是历史残影，不是真正参与执行
## 为什么不要先盯日志
因为日志只能回答：
- 某一步报了什么错
- 某个实例刚才做了什么
但它回答不了：
- 现在到底有多少进程真在跑
- 一共跑成功了多少
- 黑名单推进了多少
- 当前整体吞吐是上升还是下降
所以顺序应该固定成：
1. 先看后台 `运行中心` 第一屏
2. 再看节点列表和步骤分布
3. 只有需要深挖时，再看 `runtime_observer.py` 或具体日志
## 一句话记法
先看：
- 有没有积压
- 有没有执行面
- 最近有没有吞吐
这 3 个一起动，才算真的在跑。
## 现在先看哪 3 块
后台第一屏已经拆成这 3 个核心口径：
### 1. 总盘子（全库）
回答的是：
- 海外主库现在一共有多少域名
- 全库还有多少待检测
- 全库已经通过、失败、黑名单各多少
### 2. 当前活跃批次
回答的是：
- 当前正在跑的是哪一批
- 这一批的展示口径有多少
- 这一批的原始口径有多少
这一块非常重要。
它不是全库总量。
### 3. 近窗吞吐
回答的是：
- 最近 15 分钟到底推进了多少
- 每分钟大概多少
- 最近完成、失败、黑名单各多少

103
docs/审核顺序.md Normal file
View File

@@ -0,0 +1,103 @@
# 第一轮审核清单
第一轮目标：先查控制面、运行态聚合、同步链、节点控制和配置分发。
建议模型与强度：
- `gpt-5.4`
- `xhigh`
建议额度：
- `300万 ~ 450万 tokens`
本轮不看：
- `release/`
- `.venv/`
- `node_modules/`
- 运行时日志、快照、night runs
- 大型静态资源和锁文件
## 审核顺序
### P0：运行态与页面口径
1. [domain-api/app/services/detect_job_service.py](/www/wwwroot/getDomain/domain-api/app/services/detect_job_service.py)
重点看：`active_job`、`runtime_snapshot`、`runtime_ingest`、`distributed_node_stats`、`display_*` 字段是否会互相覆盖。
2. [domain-api/app/services/detect_service.py](/www/wwwroot/getDomain/domain-api/app/services/detect_service.py)
重点看：检测控制页口径、聚合线程数、参与节点/参与进程统计、零值回退逻辑。
3. [domain-api/app/services/runtime_status_service.py](/www/wwwroot/getDomain/domain-api/app/services/runtime_status_service.py)
重点看：运行中心聚合、`queue_health` 与 backlog 对齐、活跃任务判定。
4. [domain-api/app/services/dashboard.py](/www/wwwroot/getDomain/domain-api/app/services/dashboard.py)
重点看：首页总览是否复用旧快照、`queue_display_*` 和 `active_job` 是否一致。
5. [domain-api/app/api/routes/detect.py](/www/wwwroot/getDomain/domain-api/app/api/routes/detect.py)
重点看：检测控制接口有没有直接透传脏口径。
6. [domain-api/app/api/routes/runtime.py](/www/wwwroot/getDomain/domain-api/app/api/routes/runtime.py)
重点看：运行中心接口是否二次加工错误。
7. [domain-api/app/api/routes/dashboard.py](/www/wwwroot/getDomain/domain-api/app/api/routes/dashboard.py)
重点看：首页口径是否和服务层一致。
### P0：同步链与投影链
8. [domain-api/app/services/sync_record_service.py](/www/wwwroot/getDomain/domain-api/app/services/sync_record_service.py)
重点看：`runtime_projection`、`runtime_ingest`、未来时间记录、重复投影、自愈逻辑。
9. [domain-api/app/services/sync_push_service.py](/www/wwwroot/getDomain/domain-api/app/services/sync_push_service.py)
重点看：同步触发、推送条件、失败重试、是否会把旧状态推成新状态。
10. [domain-api/app/services/cluster_runtime_service.py](/www/wwwroot/getDomain/domain-api/app/services/cluster_runtime_service.py)
重点看：集群节点汇总、心跳、stale/offline 判定。
11. [domain-api/app/services/debug_event_service.py](/www/wwwroot/getDomain/domain-api/app/services/debug_event_service.py)
重点看：调试事件是否参与运行态推断，是否会误导当前任务。
### P0：节点控制与配置分发
12. [domain-api/app/services/worker_control_service.py](/www/wwwroot/getDomain/domain-api/app/services/worker_control_service.py)
重点看：worker 进程探测、当前状态读取、systemd 口径和实际进程口径是否一致。
13. [domain-api/app/services/settings_service.py](/www/wwwroot/getDomain/domain-api/app/services/settings_service.py)
重点看：`process_count`、`thread_count`、节点级覆盖、多实例父子节点映射。
14. [domain-api/app/services/runtime_settings_service.py](/www/wwwroot/getDomain/domain-api/app/services/runtime_settings_service.py)
重点看：运行态配置来源、热配置优先级、页面修改后是否真生效。
15. [domain-api/app/services/runtime_control_service.py](/www/wwwroot/getDomain/domain-api/app/services/runtime_control_service.py)
重点看：启动/停止/恢复对多实例 worker 是否安全。
### P1：检测任务主链
16. [domain-api/app/services/detect_run_service.py](/www/wwwroot/getDomain/domain-api/app/services/detect_run_service.py)
重点看：运行记录、cycle 事件、页面日志来源。
17. [domain-api/app/services/domains_service.py](/www/wwwroot/getDomain/domain-api/app/services/domains_service.py)
重点看：域名主数据是否和检测任务状态有交叉写入风险。
18. [domain-api/app/services/import_task_service.py](/www/wwwroot/getDomain/domain-api/app/services/import_task_service.py)
重点看：导入任务是否影响 backlog、是否能造成页面统计失真。
## 必看测试
### 直接对应运行态/聚合
1. [domain-api/tests/test_detect_job_service.py](/www/wwwroot/getDomain/domain-api/tests/test_detect_job_service.py)
2. [domain-api/tests/test_detect_service_status_fallback.py](/www/wwwroot/getDomain/domain-api/tests/test_detect_service_status_fallback.py)
3. [domain-api/tests/test_runtime_status_service.py](/www/wwwroot/getDomain/domain-api/tests/test_runtime_status_service.py)
4. [domain-api/tests/test_dashboard_service.py](/www/wwwroot/getDomain/domain-api/tests/test_dashboard_service.py)
### 直接对应同步链
5. [domain-api/tests/test_sync_record_service.py](/www/wwwroot/getDomain/domain-api/tests/test_sync_record_service.py)
6. [domain-api/tests/test_sync_push_service.py](/www/wwwroot/getDomain/domain-api/tests/test_sync_push_service.py)
7. [domain-api/tests/test_cluster_runtime_service.py](/www/wwwroot/getDomain/domain-api/tests/test_cluster_runtime_service.py)
### 直接对应节点控制与配置
8. [domain-api/tests/test_worker_control_service.py](/www/wwwroot/getDomain/domain-api/tests/test_worker_control_service.py)
9. [domain-api/tests/test_settings_service.py](/www/wwwroot/getDomain/domain-api/tests/test_settings_service.py)
10. [domain-api/tests/test_detect_api_routes.py](/www/wwwroot/getDomain/domain-api/tests/test_detect_api_routes.py)
11. [domain-api/tests/test_ops_api_routes.py](/www/wwwroot/getDomain/domain-api/tests/test_ops_api_routes.py)
## 本轮输出要求
只输出这 3 类内容，避免烧额度：
- `P0 / P1` 真实问题，带文件和行号
- 影响范围
- 修复建议
不要在第一轮做这些：
- 大篇架构说明
- 文件逐段复述
- UI 细节优化建议
- 发布脚本和迁移功能深挖
## 第一轮完成标准
满足以下条件就可以结束第一轮，转第二轮：
- 能回答“为什么页面显示会和现场不一致”
- 能回答“为什么 sync 会把旧运行态覆盖成新页面口径”
- 能回答“进程/线程配置、页面显示、节点实际执行三者有没有断层”
- 能列出前 `10` 个最值得先修的 `P0/P1` 问题

180
docs/当前已验证有效的线上参数与改动清单.md Normal file
View File

@@ -0,0 +1,180 @@
# 当前已验证有效的线上参数与改动清单
> 说明：这份文档保留作阶段留档。后续执行和观察，统一以 [当前线上最终Runbook.md](/www/wwwroot/getDomain/docs/当前线上最终Runbook.md) 为准。
最后更新：2026-04-25 17:30 左右
适用节点：`mainland-controller-01`
目标：给后续观察和继续优化留一个“当前已经验证有效”的固定基线，不再靠聊天记录回忆。
## 1. 当前线上基底
- 当前 `current` 指向：
`/opt/domaincheck/releases/domaincheck_release_20260424_220158`
- 当前核心服务状态：
- `domaincheck-worker.service = active`
- `domaincheck-sync-agent.service = active`
- `domaincheck-api.service = active`
说明：
- 这台机器现在不是只跑旧 release 原样代码，而是“`220158` 基底 + 多轮热补”。
- 后续如果要正式固化，应该把当前热补内容重新打一版正式 release。
## 2. 当前已确认生效的远端环境参数
以下参数已经在运行中的 `detect_worker.py` 进程环境里确认过，不是只改了文件没重启：
### 单机性能/爱站相关
- `DOMAINCHECK_REGISTER_SINGLE_MACHINE_MODE=1`
- `DOMAINCHECK_SINGLE_MACHINE_AIZHAN_DIRECT_FIRST=1`
- `DOMAINCHECK_AIZHAN_REMOTE_DISCONNECT_DEGRADE=1`
- `DOMAINCHECK_AIZHAN_EXTERNAL_FAST_DEGRADE=1`
- `DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_AIZHAN=2`
- `DOMAINCHECK_PROXY_STEP_MAX_SECONDS_AIZHAN=10`
- `DOMAINCHECK_AIZHAN_TIMEOUT_PROXY=1.8`
- `DOMAINCHECK_AIZHAN_TIMEOUT_DIRECT=2.4`
### Wayback 相关
- `DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_WAYBACK=2`
- `DOMAINCHECK_PROXY_STEP_MAX_SECONDS_WAYBACK=8`
- `WAYBACK_CDX_TIMEOUT=3`
- `WAYBACK_SNAPSHOT_TIMEOUT=2`
- `WAYBACK_RETRY_COUNT=1`
- `WAYBACK_DOMAIN_CONCURRENCY=1`
- `WAYBACK_MAX_RECORDS=2`
- `WAYBACK_TRANSIENT_BACKOFF_SECONDS=0.5`
## 3. 当前已确认上线的代码改动方向
这些不是“计划中”，而是已经热补到 `mainland-controller-01` 并跑起来的：
### 调度/分发侧
- 预补货提前触发，不再等批次快空了才补。
- 预补货时不让拿到锁的那个 worker 先独吞下一批任务。
- `pipeline` 推进和主动 `pull_tasks` 做了共享锁，减少 60 个 worker 同时拆批次。
- `session_replaced -> queued_before_start` 这条重复唤起链已经加护栏压住。
- `claim / finalize / release` 这些本地 DB 重链做过多轮减重，分钟级长尾已经明显掉下来。
### 注册/站点检测侧
- 注册检测保留单机性能模式。
- 百度、360、爱站都已经不再停留在过于激进的超时口径。
- `爱站` 已经补到：
- 单机模式下首轮可直连优先
- `RemoteDisconnected` 可直接降级
- 现在又进一步支持“外部依赖异常快速降级”，不再把整轮预算白白耗完
### Wayback 侧
- `wayback` 已经从“自己 backoff 把自己打死”收住。
- 当前远端新版逻辑里：
- `latest_cdx` transient 时，不再硬打额外的 `records_cdx`
- backoff 已收短到 `0.5s`
- 当前剩下主要是 `web.archive.org` 本身慢/拒绝连接
## 4. 这轮真正验证出来的效果
下面这些是已经实测到的，不是预估：
### 4.1 机器确实吃起来了
较早一轮稳定快照里，`mainland-controller-01` 已经出现：
- `60` 个节点全在线
- `53` 个子 worker 的 `active_threads > 0`
- `active_thread_count = 13531`
后续观察里，最近 2 分钟也长期能看到：
- `59` 个 fresh worker
- 大约 `51 ~ 56` 个 worker 有真实线程数大于 `0`
- 真实线程总和大约在 `3.8 万 ~ 4.2 万`
这说明当前不是“进程活着但没吃活”，而是确实在跑。
### 4.2 内存高位问题先被压下来了
滚动重启并替换旧 RSS 后，现场有过一轮明显回落：
- `Mem used ≈ 31Gi`
- `available ≈ 93Gi`
- `60` 个 worker 总 RSS 约 `23.9Gi`
说明“旧 RSS 不退”这层已经不是最早那种危险状态。
### 4.3 爱站与 Wayback 的尾部外部链，已经从“卡住不产出”变成“持续产出”
观察窗：`2026-04-25 17:17:18` 到 `17:29:23`
- `completed_total: 50786 -> 51298`，净增 `512`
- `aizhan_completed: 7718 -> 7936`，净增 `218`
- `wayback_completed: 3068 -> 3362`，净增 `294`
- 这段窗口里：
- `aizhan_failed = 0`
- `wayback_failed = 0`
这个结论很重要：
- 现在已经不是“压住 failed，但 completed 不动”
- 而是“failed 压住了，同时 completed 也在持续增长”
### 4.4 爱站快速降级这刀确实有作用
更早一段观察里，`爱站 failed` 还是会堆。
在把 `DOMAINCHECK_AIZHAN_EXTERNAL_FAST_DEGRADE=1` 灰到远端后，后面的观察窗里：
- `aizhan_failed` 被压到 `0`
- `aizhan_completed` 持续增长
这说明当前主链里，对爱站这类外部依赖异常，“快速降级继续”是有效的。
## 5. 当前怎么理解这些数字
当前最应该盯的不是：
- 单次 `claimed`
- 单次 `pending`
- 某一个瞬间的 `running`
因为这是 `domain_pipeline`，前一步跑完会继续生成下一步，数字天然会波动。
当前更应该盯的是：
1. `completed_total` 是否持续增长
2. `aizhan_completed / wayback_completed` 是否持续增长
3. `aizhan_failed / wayback_failed` 是否继续维持低位
如果这三组数继续保持当前趋势，就说明这套配置是对的。
## 6. 当前不要再乱动的东西
在下一轮更长观察结束前，建议先不要再继续频繁改：
- worker 进程数
- 每进程线程上限
- claim / backlog 批量参数
- wayback / 爱站超时参数
原因很简单：
现在已经进入“参数开始生效、completed 在涨”的阶段，再频繁动，会把已验证有效的窗口打碎。
## 7. 下一步建议
当前建议先停手观察，不再继续大改参数。
优先做两件事：
1. 继续观察更长窗口
建议至少再看 `30 ~ 60` 分钟，确认：
- `completed_total` 继续增长
- `aizhan_completed / wayback_completed` 继续增长
- `aizhan_failed / wayback_failed` 不重新抬头
2. 之后再做正式固化
把当前这些热补过的代码和远端有效参数，整理进正式 release，避免后续机器重启或重新发版时丢失。
## 8. 当前一句话结论
**这轮已经验证到：当前 `mainland-controller-01` 上这套“单机性能模式 + 外部依赖快速降级继续”的组合是有效的，主链已经从“线程忙但不出结果”切到“线程忙，同时持续产出 completed”。**

273
docs/当前线上最终Runbook.md Normal file
View File

@@ -0,0 +1,273 @@
# 当前线上最终 Runbook
最后更新：2026-04-26 18:12 左右
适用节点：`mainland-controller-01`
这份文档的目标很简单：
后面不再靠聊天记录回忆，也不在多份文档里来回翻。
只看这一份，就知道：
1. 当前线上到底跑在什么基线上
2. 现在先看什么数
3. 什么情况继续观察
4. 什么情况才值得继续动刀
## 1. 当前结论
当前这套线上方案，已经从：
- 线程忙但不出结果
推进到了：
- 线程忙，同时持续产出 `completed`
所以现在的正确策略不是继续乱改参数，而是：
**先稳住，先观察，确认这套已经验证有效的配置能不能持续出结果。**
## 2. 当前线上基线
### 当前基底版本
- `current` 指向：
`/opt/domaincheck/releases/domaincheck_release_20260424_220158`
说明：
- 当前不是纯旧 release 原样运行
- 真实线上状态是：
`220158 基底 + 多轮已热补代码 + 已生效 env`
### 当前服务状态
- `domaincheck-worker.service = active`
- `domaincheck-sync-agent.service = active`
- `domaincheck-api.service = active`
## 3. 当前已确认生效的关键参数
这些参数是**已经在运行进程环境里确认过**的，不是只改了文件。
### 单机性能与爱站相关
- `DOMAINCHECK_REGISTER_SINGLE_MACHINE_MODE=1`
- `DOMAINCHECK_SINGLE_MACHINE_AIZHAN_DIRECT_FIRST=1`
- `DOMAINCHECK_AIZHAN_REMOTE_DISCONNECT_DEGRADE=1`
- `DOMAINCHECK_AIZHAN_EXTERNAL_FAST_DEGRADE=1`
- `DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_AIZHAN=2`
- `DOMAINCHECK_PROXY_STEP_MAX_SECONDS_AIZHAN=10`
- `DOMAINCHECK_AIZHAN_TIMEOUT_PROXY=1.8`
- `DOMAINCHECK_AIZHAN_TIMEOUT_DIRECT=2.4`
### Wayback 相关
- `DOMAINCHECK_PROXY_STEP_MAX_ATTEMPTS_WAYBACK=2`
- `DOMAINCHECK_PROXY_STEP_MAX_SECONDS_WAYBACK=8`
- `WAYBACK_CDX_TIMEOUT=3`
- `WAYBACK_SNAPSHOT_TIMEOUT=2`
- `WAYBACK_RETRY_COUNT=1`
- `WAYBACK_DOMAIN_CONCURRENCY=1`
- `WAYBACK_MAX_RECORDS=2`
- `WAYBACK_TRANSIENT_BACKOFF_SECONDS=0.5`
## 4. 当前已确认上线的代码方向
### 调度 / 分发侧
- 预补货提前触发
- 预补货时不再让补货 worker 先独吞下一批
- `pipeline` 推进和主动 `pull_tasks` 做了共享锁
- `session_replaced -> queued_before_start` 这条重复唤起链已压住
- `claim / finalize / release` 这些本地 DB 重链已经做过减重
### 外部步骤侧
- 注册检测保留单机性能模式
- 百度、360、爱站都不再是早期那套过于激进的超时口径
- `爱站` 当前已经具备：
- 首轮可直连优先
- `RemoteDisconnected` 可直接降级
- 外部依赖异常可快速降级继续
- `wayback` 当前已经具备：
- `latest_cdx` transient 时不再硬打额外 `records_cdx`
- backoff 已收短
- 主要剩下 `web.archive.org` 本身慢/拒绝连接
## 5. 当前已经验证出来的效果
### 5.1 机器已经真正吃起来
已观察到：
- `60` 个节点全在线
- 多次快照里 `51 ~ 56` 个 worker 有真实线程
- 真实线程和长期在 `3.8 万 ~ 4.2 万`
所以当前不是“进程活着但没干活”，而是确实在跑。
### 5.2 内存高位问题先被压下来
滚动重启释放旧 RSS 后，现场出现过一轮明显回落：
- `Mem used ≈ 31Gi`
- `available ≈ 93Gi`
- `60` 个 worker 总 RSS ≈ `23.9Gi`
说明“旧 RSS 不退”这层已经先被压住。
### 5.3 当前真正最重要的验证结果
观察窗：`2026-04-25 17:17:18` 到 `17:29:23`
- `completed_total: 50786 -> 51298`，净增 `512`
- `aizhan_completed: 7718 -> 7936`，净增 `218`
- `wayback_completed: 3068 -> 3362`，净增 `294`
- 这段窗口里：
- `aizhan_failed = 0`
- `wayback_failed = 0`
这说明：
- 现在已经不是“压住 failed，但 completed 不动”
- 而是“failed 被压住，同时 completed 持续增长”
## 6. 当前最该看的数
后面观察时，优先级固定按这个顺序：
1. `completed_total`
2. `aizhan_completed`
3. `wayback_completed`
4. `aizhan_failed`
5. `wayback_failed`
6. `running / claimed / pending`
顺序不要反过来。
原因：
- `claimed / pending` 在 `domain_pipeline` 下天然会波动
- `completed` 才是最真实的产出
## 7. 当前正确的判断方法
### 7.1 什么叫“继续观察就行”
只要看到下面这种趋势，就先不要再改：
- `completed_total` 持续增长
- `aizhan_completed` 持续增长
- `wayback_completed` 持续增长
- `aizhan_failed / wayback_failed` 维持低位或归零
这说明：
- 当前主链是通的
- 当前配置方向是对的
- 现在最应该做的是稳住，不是继续乱调
### 7.2 什么叫“又卡住了”
只有出现下面这些情况，才值得继续下刀：
#### 情况一
`completed_total` 在 `30 ~ 60` 分钟窗口里基本不动
说明：
- 虽然线程在跑
- 但结果没有真正落地
#### 情况二
`aizhan_failed` 或 `wayback_failed` 又明显抬头
说明：
- 当前“快速降级继续”的策略还不够稳
#### 情况三
`running` 长时间很高，但 `aizhan_completed / wayback_completed` 不再增长
说明：
- 新的主瓶颈又出现了
## 8. 当前建议观察窗口
建议至少继续看 `30 ~ 60` 分钟。
正确做法：
1. 记录一组起始值
2. 每隔 `10 ~ 15` 分钟看一次
3. 至少看 `3 ~ 4` 个点
不要只看一两个瞬间。
## 9. 当前先不要再碰的东西
在这轮观察结束前，先不要继续改：
- worker 进程数
- 每进程线程数
- claim 批量
- backlog 批量
- 爱站超时
- wayback 超时
- 其他步骤的重试窗口
原因很简单：
- 现在已经进入“开始稳定产出”的阶段
- 再频繁改参数，只会把已经验证有效的窗口打碎
## 10. 当前一句话执行原则
**先稳住，先看 `completed` 能不能持续增长；只要结果还在持续出来，就先不要再乱改。**
## 11. `job 876` 尾项释放的当前判断
观察时间：`2026-04-26 18:12` 左右
当前不是“又卡死了”，而是：
- `idx_detect_job_items_release_node_job` 还没转成 `valid=1`
- 但它已经稳定进入 `index validation: scanning table`
- 最新验证进度已经到 `229991 / 6197384`
`codex-job876-tail-watch.service` 这条后台链当前是活的，而且在连续记进度：
- `18:09:51 -> 126725`
- `18:10:52 -> 169564`
- `18:11:52 -> 210067`
- `18:12:22 -> 227856`
同时现场另外两组状态也正常：
- `job 876 = running`
- worker 已恢复到：
- `base=active`
- `templated=199`
- `node_agent=active`
这次现场更准确的判断应该是：
- 已经从“卡住”切成“后台验证扫描中”
- 当前最合理的动作不是再人工乱动
- 应该继续让索引自己扫完
一旦 `idx_detect_job_items_release_node_job` 转成 `valid=1`，`codex-job876-tail-watch.service` 就会自动继续释放 `876` 的尾项。
## 12. 如果后面还要继续优化，下一刀顺序
只有在更长观察窗口里确认又卡住时，才按这个顺序继续：
1. 先看 `completed_total` 为什么不涨
2. 再看 `aizhan_completed / wayback_completed` 哪个先停
3. 再决定是继续打外部链，还是回头查回写 / 聚合语义
不要一上来就回去改线程、claim、并发。

149
docs/当前线上运维执行清单.md Normal file
View File

@@ -0,0 +1,149 @@
# 当前线上运维执行清单
> 说明：这份文档保留作阶段执行稿。后续执行和观察，统一以 [当前线上最终Runbook.md](/www/wwwroot/getDomain/docs/当前线上最终Runbook.md) 为准。
最后更新：2026-04-25 17:30 左右
适用节点：`mainland-controller-01`
## 1. 当前目标
当前先不要继续乱调参数。
先确认这一套已经生效的配置，能不能稳定把结果持续跑出来。
现在最重要的不是：
- 再加线程
- 再改 claim
- 再改批量
- 再改超时
现在最重要的是：
1. `completed_total` 能不能持续增长
2. `aizhan_completed / wayback_completed` 能不能继续增长
3. `aizhan_failed / wayback_failed` 会不会重新抬头
## 2. 当前已经验证有效的方向
这一轮已经确认有效的不是单点，而是整套思路：
- 单机性能模式保留
- 任务已经能铺到更多 worker 上
- `爱站` 支持更快降级继续
- `wayback` 已经减少了额外的 `records_cdx` 压力
- 当前主链已经从“线程忙但不出结果”变成“线程忙，同时持续产出 completed”
## 3. 当前观察时先看什么
优先看这 6 组数：
1. `completed_total`
2. `aizhan_completed`
3. `wayback_completed`
4. `aizhan_failed`
5. `wayback_failed`
6. `running / claimed / pending`
判断顺序不要乱：
1. 先看 `completed_total` 有没有涨
2. 再看 `aizhan_completed / wayback_completed` 有没有涨
3. 最后才看 `failed` 和 `pending`
原因：
- `claimed / pending` 在 `domain_pipeline` 下会波动，不能单独看
- `completed` 才是最真实的产出
## 4. 当前正确的判断口径
### 说明一
如果看到：
- `running` 很高
- `claimed / pending` 在波动
- 但 `completed_total` 也在持续涨
这不是问题。
这说明当前机器正在持续消化任务。
### 说明二
如果看到：
- `aizhan_failed = 0`
- `wayback_failed = 0`
- `aizhan_completed / wayback_completed` 继续涨
这说明现在这套“外部依赖快速降级继续”是对的。
### 说明三
如果看到：
- `running` 很高
- 但 `completed_total` 长时间完全不涨
这才说明主链又卡住了，需要继续排。
## 5. 当前建议观察窗口
建议至少看 `30 ~ 60` 分钟，不要只看一两个瞬间。
推荐做法：
1. 先记一组起始值
2. 每隔 `10 ~ 15` 分钟看一次
3. 连看至少 `3 ~ 4` 个点
只看单点，很容易误判。
## 6. 什么情况说明继续观察就行
只要出现下面这种趋势，就先不要动参数：
- `completed_total` 持续增长
- `aizhan_completed` 持续增长
- `wayback_completed` 持续增长
- `aizhan_failed / wayback_failed` 维持低位或归零
这说明当前主链是通的，应该先稳住。
## 7. 什么情况才需要继续下刀
只有出现下面这些情况，才值得继续改：
### 情况一
`completed_total` 在 `30 ~ 60` 分钟窗口里基本不动
说明虽然线程在跑，但结果没有真正落地。
### 情况二
`aizhan_failed` 或 `wayback_failed` 又明显抬头
说明当前“快速降级继续”还不够稳。
### 情况三
`running` 长时间很高，但 `aizhan_completed / wayback_completed` 不再增长
说明新的主瓶颈又出现了。
## 8. 当前不要再碰的东西
在这轮观察结束前，先不要继续改：
- worker 进程数
- 每进程线程数
- claim 批量
- backlog 批量
- wayback 参数
- 爱站参数
原因很简单：
现在已经进到“开始稳定产出”的阶段，频繁动参数，只会把已验证有效的窗口打碎。
## 9. 当前一句话执行原则
**先稳住，先观察 completed 是否持续增长；只要结果还在持续出来，就先不要再乱改。**

162
docs/新机器接手交接_20260427.md Normal file
View File

@@ -0,0 +1,162 @@
# 新机器接手交接
## 1. 目标
这次不是继续在当前机器上排障，而是：
- 把当前代码整理后提交到 Git
- 在新服务器上重新部署验证
- 让新的 Codex 接手继续推进
当前最重要的未完成主线只有一条：
- `domain-api/app/services/sync_push_service.py` 中 `_load_pushable_projections()` 的修复，需要在远端稳定部署后验证：
- `detect_result_projection` push 是否恢复
- `detect_result_ingest` 是否出现
## 2. 当前代码结论
已经确认并完成的代码侧修复：
- `domain-api/app/services/sync_push_service.py`
- 修复 `_load_pushable_projections()` 选择逻辑
- 旧问题：最新 `detect_result_projection` 明明存在且没有 push attempt，但候选集为空
- 现修复：候选改为优先按最新 projection 选择
- `domain-api/tests/test_sync_push_service.py`
- 已补回归测试
本地验证已完成：
- `py_compile` 通过
- `python -m unittest tests.test_sync_push_service` 通过
## 3. 当前卡点
不是代码不清楚，而是当前远端 SSH 不稳定：
- `121.204.244.188:22` TCP 可连接
- 但 SSH banner 经常超时
- Paramiko 常见报错：
- `SSHException('No existing session')`
- `Error reading SSH protocol banner`
因此当前没有拿到“远端已发布并生效”的硬结果。
## 4. 新机器接手后的首要动作
新机器接手后，先不要扩散排查面，只做这一个最小动作：
1. 发布 `domain-api/app/services/sync_push_service.py`
2. 重启 `domaincheck-sync-agent.service`
3. 只验证两项：
- `detect_result_projection` push 是否恢复
- `detect_result_ingest` 是否出现
## 5. 关键背景
项目根目录：
- `/www/wwwroot/getDomain`
主要子目录：
- `domain-api/`
- `domain-web/`
- `domainCheck/`
- `docs/`
当前远端主机：
- `mainland-controller-01`
- `121.204.244.188`
已确认的远端运行时 region 配置：
- `NODE_REGION=mainland`
- `SYNC_SOURCE_REGION=mainland`
- `SYNC_TARGET_REGION=overseas`
已确认的数据事实：
- mainland 本地已经生成了 `detect_result_projection`
- surrogate job 例如：
- `job_id=909`
- `job_code=sync-overseas-28618`
- worker 事件已写入 mainland 本地库
- 真正未打通的是 `detect_result_projection` 的 push 选择阶段
## 6. 建议提交范围
建议优先提交真正的代码改动，不要把运行产物一起带上。
建议提交：
- `domain-api/`
- `domain-web/`
- `domainCheck/`
- `tools/`
- 需要保留的文档 `.md`
建议重点确认本次必须包含：
- `domain-api/app/services/sync_push_service.py`
- `domain-api/tests/test_sync_push_service.py`
## 7. 明确不要提交的内容
以下属于运行产物、临时文件或本地探针，不建议提交：
- `docs/ops_center_runtime/night_runs/step_mix_*/`
- `docs/ops_center_runtime/chinaz_gray_runs/`
- `docs/_tmp_regression/`
- `.codex-release-probe.txt`
- `*.bak_*`
- `*.log`
- `*.pid`
当前工作区里特别要排除的项目：
- `domainCheck/app/utils/database.py.bak_20260426_220818_active_job_cache_tune`
- `domainCheck/detect_worker.py.bak_20260426_220818_active_job_cache_tune`
- `docs/ops_center_runtime/night_runs/step_mix_20260426_051718_smoke_p200/`
- `docs/ops_center_runtime/night_runs/step_mix_20260426_051754_smoke_p200/`
- `docs/ops_center_runtime/night_runs/step_mix_20260426_052641_debug_stepmix/`
- `docs/ops_center_runtime/night_runs/step_mix_20260426_053252_mainland_step_mix_p200_t800/`
- `docs/ops_center_runtime/chinaz_gray_runs/`
- `.codex-release-probe.txt`
## 8. 已跟踪但不建议把这次删除提交进去的文件
这些文件目前显示为已删除，但更像本地运行态/配置文件，不建议在这次“迁移到新服务器”的提交里带上删除：
- `domainCheck/app/thread_count.json`
- `domainCheck/node_thread_counts.json`
- `domainCheck/runtime/runtime_settings.json`
- `domainCheck/runtime/sensitive_words.json`
- `domainCheck/runtime_settings.json`
- `domainCheck/thread_count.json`
处理建议：
- 如果这些删除不是你明确想做的配置收口，请在提交前恢复
- 不要把“本地运行时删掉了某个 JSON”误当作产品代码改动提交
## 9. 提交前建议
提交前建议至少做这几步：
1. `git status --short`
2. 把运行产物和备份文件排除掉
3. 只保留代码、测试、必要文档
4. 单独复查：
- `sync_push_service.py`
- `test_sync_push_service.py`
- `.gitignore`
- 本交接文档
## 10. 给新机器上的 Codex 的一句话
接手后不要重新发散排查 worker、projection 生成、事件落库链；这些已经基本收敛。优先完成 `sync_push_service.py` 的远端生效验证，只盯：
- `detect_result_projection` push
- `detect_result_ingest`

417
docs/测试优化分析.md Normal file
View File

@@ -0,0 +1,417 @@
# 测试优化分析
> 说明：这份文档保留作历史分析记录。当前线上执行基线与观察方法，统一以 [当前线上最终Runbook.md](/www/wwwroot/getDomain/docs/当前线上最终Runbook.md) 为准。
更新时间：`2026-04-24 14:00`
## 这份文档是干什么的
这份文档不是讲理想情况，而是把这次已经做过的单机压测结果，和当前现场“根本没真正跑起来”的现状，放在一起说清楚。
目的很直接：
- 先把已经验证过的参数结论留档
- 再把现在为什么看起来线程上限很高、实际只跑了很少线程说清楚
- 给后面继续优化的人一个明确顺序，别再一边主链没跑通，一边继续盲目加并发
## 一句话结论
早上那轮“只测一台大陆机器”的压测，已经得出过一个单机最优值：
- `mainland-controller-01` 单机跑时，当前最优参数是 `35 进程 / 900 线程`
但这不等于项目现在已经能按这个参数稳定跑。
按 `2026-04-24 13:22` 左右控制面看到的实时状态，这套链路目前更像是：
- 任务还挂着
- 页面也还能显示“运行中”
- 但真正参与执行的只有很少几个桶
- 总体上属于“没有真正跑起来”
所以现在的主矛盾不是“继续把线程调大”，而是“先让主流程真的跑起来”。
## 这次单机压测是怎么测的
这轮压测不是全链路压测，而是一个故意收缩后的测试环境。
当时做了这几件事：
- 先停掉 `121.204.244.248`，也就是 `mainland-worker-01`
- 只保留 `mainland-controller-01` 单机承担检测
- 停掉 controller 上的 `domaincheck-sync-agent.service`
- 不再继续额外增加新的 Redis 限制动作
- 直接在 controller 本地数据库里看 60 秒窗口真实完成量和更新量
这么做的目的，不是模拟线上真实规模，而是先排除多机互相干扰，看单机能跑到什么水平。
## 单机压测结果
压测时间段大致是 `2026-04-24 05:14 - 05:28`。
当时 controller 本地 60 秒窗口结果如下：
| 参数 | 60 秒完成量 | 60 秒更新量 | Redis 连接数 | 内存已用 | 可用内存 | 实际活跃 worker |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `35/900` | `4092` | `22267` | `751` | `73.6G` | `54.3G` | `35` |
| `40/1000` | `1885` | `1544` | `793` | `75.1G` | `52.9G` | `40` |
| `45/1100` | `388` | `11347` | `818` | `75.7G` | `52.3G` | `45` |
| `50/1200` | `0` | `12118` | `823` | `75.9G` | `52.1G` | `45` |
## 压测结论
从这轮结果里，可以直接得出 4 个结论：
### 1. 单机最优值不是越大越好
当前这套链路下，`35/900` 明显优于后面的更大参数。
也就是说，继续加进程、加线程，并没有带来更高完成量，反而更差。
### 2. 当前单机稳定上限大约在 `45/1100`
`45/1100` 还能勉强活住。
但它已经不是“跑得更快”，只是“还能挂着不死”。
### 3. `50/1200` 已经没有意义
虽然当时把参数抬到了 `50/1200`，但最终实际只活成了 `45` 个 worker，而且完成量已经掉到 `0`。
这说明不是配置没写进去，而是链路已经先撞到别的瓶颈了。
### 4. 当时真正的瓶颈已经不是 Redis
从这轮压测看，Redis 连接数有上升，但没有先爆。
更像是外部检测链路本身先撑不住了，尤其是：
- 注册检测
- 代理池质量
- 会话切换抖动
## 为什么当时测出来 `35/900`，现在页面却只剩 `52` 线程
这就是最容易误判的地方。
早上压测结论成立的前提是：
- 只测 controller 单机
- `mainland-worker-01` 被停掉
- `sync-agent` 被停掉
- 现场是专门为压测收缩过的
而现在你看到的实时状态，已经不是那个压测场景了。
按 `2026-04-24 13:22` 左右接口返回的状态，现在的现场是：
- `detect/job/active` 里的活动任务还是 `job_id=1`
- 这条任务从 `2026-04-21 23:30:42` 就已经是 `running`
- `raw_items_pending=9478`
- `raw_items_running=1`
- `raw_items_completed=4996`
- `raw_items_failed=486`
同时，`runtime/status` 给出的聚合状态是：
- `参与服务器：4`
- `参与进程：4`
- `参与线程：52`
- `运行中：52`
- `线程上限：52 / 2701`
这几组数字合起来说明的不是“系统已经高效跑起来”，而是：
- 页面还能看到一些参与桶
- 但真正持续工作的执行面很薄
- 大量任务还躺在队列里，没有被健康地持续消化
## 当前为什么可以直接判断“根本跑不起来”
不是因为页面数字小，而是因为几条关键事实同时出现了：
### 1. 活动任务太老
当前活动任务还是 `2026-04-21` 开出来的老任务。
如果主链真的健康，它不应该拖这么久还挂在 `running`。
### 2. 待处理量还很大
当前还剩：
- `9478` 条原始 pending
- `3779` 条 claimed
- `1028` 条 running backlog
但真正展示出来的活跃线程只有 `52`。
这个比例说明队列不是空了，而是消化不动。
### 3. 当前节点自己根本不跑本机 worker
当前 `overseas-control-01` 的 `runtime/status` 里明确写着：
- `worker_online=false`
- `worker_process_count=0`
- `worker_runtime_message=inactive/dead`
也就是说，现在这个页面看到的“运行中”，本来就不是本机在跑，而是靠远端汇总。
### 4. 远端参与桶很少，而且口径不稳
现在控制面聚合出的参与节点只有：
- `mainland-worker-01`
- `mainland-controller-01-k`
- `mainland-controller-01-a`
- `mainland-controller-01-s`
这和早上单机压测时的 `35` 个 worker，根本不是一个级别。
所以你现在看到“线程上限 2701”，不要被这个数字迷惑。
它只是汇总上限，不代表这些线程真的都在跑。
## 现在最像什么问题
如果用人话说，现在最像下面这种情况：
- 任务队列里还有很多活
- 但是执行面只剩很薄一层还在动
- 页面还能看到“运行中”，所以看起来像没完全死
- 可真正吞吐已经低到接近“没跑起来”
所以这不是单纯的参数问题，而是主流程状态已经歪了。
## 当前最该先做的事
下一步优化顺序建议固定成这样：
### 第 1 步：先恢复“能持续跑”
先不要继续加进程、加线程。
先确认 4 件事：
- 当前到底是哪几个节点真的在跑
- `mainland-worker-01` 是不是还应该继续停着
- controller 上当前实际活着多少 worker
- 老任务里这些 `claimed/running` 有没有大量卡死项
### 第 2 步：把老任务和会话问题理顺
现在这条 `2026-04-21` 的老任务已经拖太久了。
要优先确认：
- 是不是有很多旧 `session_replaced`
- 是不是有大量任务项一直卡在 `claimed/running`
- 是不是页面显示还在跑，但实际 worker 已经不工作了
### 第 3 步：确认代理链是不是主瓶颈
早上单机压测已经说明，Redis 不是先撞墙的那个点。
下一步更值得盯的是：
- `detect_register` 的真实完成率
- 代理池是否经常空
- RDAP 访问是否大量超时
### 第 4 步：只有主链恢复后，才重新做性能优化
## 13:48 补充结论
这轮又确认了一件非常关键的事：
- `mainland-worker-01` 不是“已经彻底消失”，而是“被标记为停用，但之前还在继续上报运行态”
后来实际做了两步处理：
- 先通过远端 `ops job` 让 `mainland-worker-01` 本机执行 `systemctl stop domaincheck-worker`
- 再把控制面的状态汇总改成：只要节点在 `ops_managed_nodes` 里是 `is_enabled=false`，就不再算进在线 Worker、参与节点和活跃线程
处理完成后，控制面实时口径已经从“2 台参与节点 / 2 个检测进程 / 2 线程”收到了：
- `参与节点：1`
- `参与进程：1`
- `运行中：1`
- 只剩 `mainland-controller-01`
这说明前面那种“明明已经准备单机测了，页面还一直把停用节点算进去”的问题，确实是状态口径 bug，不是现场真的还有两台健康执行机一起在跑。
所以从现在开始再看单机测试结果时，要以这条新口径为准：
- `mainland-worker-01` 已停用，不再参与单机压测统计
- 当前有效执行面只剩 `mainland-controller-01`
等主流程恢复成“真的能持续吃队列”之后，再拿 `35/900` 作为第一版基线。
到那时可以按下面顺序再调：
1. 先验证 `35/900` 在恢复后的现场还能不能稳定
2. 如果稳定，再测 `40/1000`
3. 只有吞吐确实提高了，才继续往上加
## 这份文档的最终结论
可以把这次结果浓缩成两句话：
第一句：
- 单机压测已经证明，当前代码和链路下，`mainland-controller-01` 的最佳参数是 `35 进程 / 900 线程`
第二句：
- 但当前线上真正的问题不是“参数不够大”，而是“主流程根本没有持续跑起来”，所以现在继续加线程没有意义
后面再优化时，应该先解决“为什么只剩 52 线程在动”，再谈如何把吞吐重新抬高。
## 2026-04-24 13:35 补充结论
这份文档写完以后，又继续往下排了一轮，现场有两个非常关键的新结论。
### 1. `52` 线程里有明显口径错配
后来继续核对后发现，页面里那组：
- `参与服务器：4`
- `参与进程：4`
- `参与线程：52`
并不是来自 `detect_job_items` 真表本身。
真实数据库里，`job_id=1` 其实只剩：
- `1` 条 `running`
而且这条记录最后更新时间还停在 `2026-04-22 15:00:56`，本质上已经是老僵死项。
真正把页面抬到 `52` 的，是一条口径 bug：
- 当前 active job 还是老任务 `detect-20260421232649-acfa3d`
- 但控制面又拿到了别的 sync job 的 runtime overlay
- 两套数据被硬合到了一起
简单说就是：
- 任务表是老 job
- 线程数却借用了别的 job 的运行态
### 2. 这条口径 bug 已经修掉了
现在代码已经补成：
- 只有 runtime overlay 的 `job_code / job_id` 和当前 active job 对得上时，才允许覆盖 `queue_health`
- 对不上时，宁可退回真实任务表，也不再把别的 job 的运行态硬套到当前页面上
修完并重启海外控制面 API 之后，实时口径已经从之前的 `52` 收回到了：
- `参与节点：2`
- `参与进程：2`
- `参与线程：2`
当前这 `2` 个真实信号分别来自：
- `mainland-controller-01` 那条老 `running`
- `mainland-worker-01` 当前 heartbeat 上报的 `1` 条活跃线程
这说明一件事：
- 之前那组 `52` 的确主要是错口径，不是实际吞吐
## 目前还剩的真实问题
虽然 `52 -> 2` 这一步已经收准了，但项目还是没有真正恢复健康。
当前还剩 2 个真实问题：
### 1. `mainland-worker-01` 明明已停用，却还在持续推运行态
当前控制面日志里还能持续看到：
- `121.204.244.248` 往海外控制面打 `runtime/debug-ingest`
- 同时也还在打 `ops/agent/pull`
这说明这台机器不是“旧残影”，而是还真的在线、还真的在报活。
所以“已停用”这件事，目前只是在托管配置层停了，但没有真正把它从 Agent / runtime 上报链里摘干净。
### 2. `job_id=1` 这条老任务本身也还没收口
当前老任务依然挂着：
- `job_id=1`
- `job_code=detect-20260421232649-acfa3d`
- `started_at=2026-04-21 23:30:42`
而真实任务表里，它现在已经不是“很多线程在跑”，而是：
- 大部分还在 `pending`
- 只剩 1 条老 `running`
这说明这条老任务本身也需要后续专门处理，不能继续长期挂在 `running`。
## 现在更准确的下一步
到这里为止，下一步就更明确了：
1. 先把 `mainland-worker-01` 真的摘掉。
2. 再处理 `job_id=1` 这条老任务的遗留 `running` 项。
3. 等这两件事收完，再重新看当前真实执行面到底还有没有持续吞吐。
也就是说，现在已经不是“继续猜线程参数”的阶段，而是“先把错误执行面和老任务残留清掉”的阶段。
## 新增一版“单机性能模式”给注册检测
这次为了后面继续压单机吞吐，我已经在代码里补了一版只影响“注册状态检测”的性能模式。
它的目标不是改全局架构，而是先把当前最像瓶颈的那一段链缩短：
- 只改 `detect_register`
- 不碰百度、360、站长、爱站、时光机这些步骤
- 保留当前版的连接复用和失败闭环
- 只把注册检测改得更像老版本那种“先尽快打出去，再说”
### 这版模式做了什么
打开后，注册检测会变成下面这个思路：
- 即使全局 `allow_direct=false`，注册检测这一步也允许直连
- 默认先连续直连 `2` 次
- 这两次之间不再等代理补货
- 只有前面的直连没打通，后面才继续走代理兜底
简单说就是：
- 当前默认模式：代理优先，直连兜底
- 单机性能模式：注册检测直连优先，代理兜底
### 怎么开
这版先做成环境变量开关：
- `DOMAINCHECK_REGISTER_SINGLE_MACHINE_MODE=1`
- `DOMAINCHECK_REGISTER_DIRECT_STREAK_ATTEMPTS=2`
当前默认建议先用这组：
- `进程：35`
- `线程：900`
- `DOMAINCHECK_REGISTER_SINGLE_MACHINE_MODE=1`
- `DOMAINCHECK_REGISTER_DIRECT_STREAK_ATTEMPTS=2`
如果后面继续试更激进一点，再考虑把直连连打次数从 `2` 提到 `3`，但不建议一上来就抬。
### 为什么不是直接回退老版本
老版本真正有参考价值的是“链短、切得快”，不是它的实现本身更先进。
所以这次没有回退这几样东西：
- 没回退到 `20s * 3` 的长超时
- 没回退到“注册失败也继续往后跑”
- 没丢掉当前的 `Session` 复用
换句话说，这次是“借老策略”，不是“退老实现”。

405
docs/项目当前运行流程说明.md Normal file
View File

@@ -0,0 +1,405 @@
# 项目当前运行流程说明
> 说明：这份文档主要回答“项目现在怎么跑”。当前线上执行基线、观察顺序和是否继续调参，统一以 [当前线上最终Runbook.md](/www/wwwroot/getDomain/docs/当前线上最终Runbook.md) 为准。
更新时间：`2026-04-24`
## 这份文档是干什么的
这不是一份“理想设计稿”，而是按项目现在的真实跑法整理出来的说明。
目标只有一个：让人用人话看明白，这个项目现在到底怎么跑，任务从哪里来，谁在执行，结果又是怎么回来的。
## 先用一句话讲明白
这套系统现在不是“点一下开始，然后一台机器自己跑完”。
它现在的实际跑法是：
海外控制面先挑出要检测的域名，打成一批任务，然后把这批任务交给大陆节点；大陆节点把任务拉下来后，由 worker 一步一步去跑检测；跑出来的结果，再同步回海外控制面，最后由页面统一展示。
## 先认识 5 个角色
### 1. 海外控制面
可以把它理解成“总调度台”。
它主要负责：
- 创建检测任务
- 给大陆节点派动作
- 接收大陆回传的运行状态和检测结果
- 在页面上展示当前进度
当前本机 `overseas-control-01` 就是这个角色。
它本机一般不直接跑检测。
### 2. 大陆控制节点
可以把它理解成“大陆现场调度员”。
它主要负责：
- 去海外把待检测批次拉下来
- 启动大陆这边的 sync-agent、worker
- 把大陆现场的运行情况回传出去
### 3. 大陆 worker 节点
这才是真正“干活”的机器。
它负责：
- 领取待执行任务
- 开线程跑检测
- 把每一步的结果写回本地
### 4. sync-agent
它就是“搬运工”。
负责在海外和大陆之间搬 3 类东西：
- 待检测任务批次
- 当前运行状态
- 检测结果
### 5. node-agent
它就是“远程执行员”。
它会在节点上定时来领控制命令，然后执行，比如：
- 启动 worker
- 开始检测
- 拉取任务
- 重启服务
## 当前现场快照
下面这段，是按 `2026-04-24 01:44 - 01:45` 左右控制面看到的实时数据整理的。
- 当前控制面节点是 `overseas-control-01`
- 角色是 `overseas / control`
- 本机 API 在线
- 本机 worker 不承担实际检测
当时控制面看到的活动任务是：
- `job_id = 208`
- `job_code = sync-overseas-1430`
- 状态是 `running`
- 这一批一共 `5000` 个任务项
- 其中 `4435` 个已经被领取
- `575` 个正在执行
当时控制面汇总看到的执行规模大致是：
- 约 `28` 个参与节点
- 约 `28` 个检测进程
- 约 `575` 个活跃线程
当时积压里最大头还是“注册状态检测”：
- `detect_register` 待处理约 `8235`
- 后续步骤待处理约 `1243`
同一时间段里，系统的 readiness 仍然提示：
- 大陆节点心跳并不稳定
- 还有结果批次待推送
这说明了一件很关键的事：
不是完全没跑，而是“链路在跑，但跑得不稳”。所以现在的主矛盾，确实还是稳定性，不是单纯把线程数继续往上调。
## 整个项目现在是怎么跑的
下面按真实流程，一步一步讲。
### 第 1 步：海外控制面先挑出“需要再检测”的域名
简单理解就是：
- 还没检测过的
- 之前检测失败过的
- 或者状态需要重新确认的
这些域名会先被挑出来，形成一份“待处理名单”。
这时候只是“选名单”，还没有真正开始跑检测。
### 第 2 步：海外控制面创建一轮检测任务
系统会先创建一条“本轮任务”记录。
然后把这批域名拆成很多个“任务项”。
这里要特别注意：
它不是一次就给某个域名下发“整套检测”。
它做的是：
- 先判断这个域名下一步该做什么
- 只给它安排“下一步”
所以这个系统现在跑的是“分步骤推进”，不是“单个域名一次性从头跑到尾”。
### 第 3 步：海外控制面自己不跑，而是把动作派到大陆
当你在页面上点“开始检测”后，海外控制面会去排队发送几类动作给大陆节点：
- 让大陆启动 sync-agent
- 让大陆去拉取待检测批次
- 让大陆启动 worker
- 让大陆开始执行检测
这些动作不是直接 ssh 过去硬敲命令。
而是先进入远程动作队列，再由大陆节点上的 node-agent 定时来领。
所以你可以把它理解成：
海外控制面负责“发指令”，大陆节点负责“取指令并执行”。
### 第 4 步：大陆控制节点先把任务批次拉下来
大陆控制节点会去海外控制面拿一批待检测域名。
拿到以后，会做几件事：
- 把这批域名落到大陆本地库里
- 在大陆本地创建一条对应的 job
- 给每个域名生成“下一步要做什么”的任务项
- 再回头告诉海外：这批任务我已经收到了
这个“确认收到”很重要。
因为如果不确认，海外会以为这批任务还没被接走，后面就可能重复下发。
另外，大陆控制节点不会无脑一直拉新任务。
如果它发现本地已经堆了很多待处理任务，它会先停一下，不再继续拉新批次，避免越堆越多。
### 第 5 步：worker 启动后，先做准备，不会立刻开跑
worker 真正开始检测前，会先做一轮准备动作。
大致包括：
- 重新读取最新配置
- 重新读取线程数和进程数
- 重新加载 cookies
- 刷新代理池
- 回收上次异常退出留下来的遗留任务
所以你看到“worker 已经启动”，并不等于“已经开始稳定出结果”。
它中间还有一个准备阶段。
### 第 6 步：worker 真正领的，是任务队列里的“下一步”
worker 现在不是直接扫整张域名表。
它主要是从任务队列里领取待执行任务。
领取后的状态大致可以这样理解：
- `pending`：还没被谁接手
- `claimed`：已经被某个节点领走了
- `running`：已经开始跑了
- `completed / failed / blacklisted`：这一小步跑完了
也就是说，系统现在盯的不是“这个域名整体做完没”，而是“这个域名现在跑到哪一步了”。
### 第 7 步：一个域名不是一次跑完，而是一小步一小步推进
当前默认的主流程顺序，大致是：
1. 注册状态检测
2. 百度 site 检测
3. 360 site 检测
4. 站长之家检测
5. 爱站检测
6. 时光机检测
有些来源的域名，会跳过注册状态这一步，直接从后面的步骤开始。
所以你不能把它理解成“所有域名一定都从第一步开始”。
更准确地说，是系统会先判断这个域名“现在最该补哪一步”。
### 第 8 步：流程顺序，和 worker 实际优先领什么，不完全一样
这个地方很容易误会，所以单独说一下。
流程顺序上，域名一般是先过前面的关，再去后面的关。
但 worker 实际领任务时，会优先照顾已经走到后面的域名。
为什么要这样做？
因为如果完全按最前面的步骤一直领，后面的域名会永远被堵住，怎么都跑不到尾部。
所以现在的真实策略更像是：
- 流程上按顺序推进
- 调度上优先让已经走到后面的域名尽快跑完
这也是为什么你现在会看到“注册状态检测积压很大”，但后面的步骤也还在继续跑。
### 第 9 步：每一步跑完后，系统会决定下一步怎么走
某一步做完后，系统不会简单地只记一个“成功 / 失败”。
它还会决定后面怎么走。
大致有几种情况：
- 这一步通过了：给这个域名创建“下一步”的任务项
- 这一步命中黑名单：后面的步骤就不再继续
- 这一步属于外部站点异常、超时、代理问题：可能会重试，也可能降级，也可能转成人工复核
- 这一步明确不通过：流程在这里终止
所以这个项目现在不是“跑完就完”，而是“每走完一步，再决定下一步”。
### 第 10 步：结果先写回大陆本地
worker 跑完某一步后，会先把结果写回大陆本地。
写回去的内容包括：
- 这一个任务项是什么结果
- 这个域名当前是什么状态
- 这一步的详细信息
- 是否需要人工复核
所以大陆本地库，先是“第一落点”。
### 第 11 步：大陆再把结果和运行状态同步回海外
大陆这边不是只回传“最终结果”。
它还会把两类信息往海外送：
- 当前运行状态
- 最近有哪些域名开始了、完成了、失败了、进黑名单了
然后海外控制面收到后，再去更新自己的展示和汇总。
这也是为什么页面上看到的数字，不是某一台机器的原始数字。
它其实是“控制面汇总后的结果”。
### 第 12 步：海外页面最后展示出来的，是一份汇总快照
所以现在页面上的数据，本质上是几部分拼起来的：
- 控制面自己知道的任务信息
- 大陆回传的运行状态
- 大陆回传的结果事件
- 当前 job 的统计汇总
这就会带来一个现实情况：
如果大陆心跳断一下，或者结果同步晚一点，页面看起来就会突然不稳定，甚至和现场有一点时间差。
这不一定代表 worker 完全没跑。
很多时候，只是“页面依赖的回传链路断了一下”。
## 你可以把现在的项目理解成两条并行链
### 第一条：任务执行链
海外选域名 -> 建任务 -> 大陆拉批次 -> worker 领任务 -> 按步骤推进 -> 写结果
### 第二条：运行状态和结果同步链
大陆上报运行状态 -> 大陆回推结果事件 -> 海外接收并汇总 -> 页面刷新展示
这两条链只要有一条不稳，使用感受就会变差。
如果任务执行链慢，你会感觉“没速度”。
如果同步链不稳，你会感觉“页面不准、看不清到底跑没跑”。
## 为什么你会一直觉得“没有速度、没有效率”
因为现在影响体验的，不只是“worker 够不够快”。
这套系统至少要同时经过下面这些环节：
- 海外选任务
- 海外派动作
- 大陆领动作
- 大陆拉批次
- worker 领任务
- 外部站点检测
- 结果回传
- 页面汇总展示
任何一段不稳，最后给人的感觉都会像是“整套系统没跑起来”。
所以从当前现场来看，先把主链路稳定跑顺，仍然比继续往上冲性能更重要。
## 如果你只想用最简单的方法判断现在卡在哪
可以按下面这个顺序看：
1. 先看有没有新的活动 job
- 没有的话，说明卡在“任务还没真正建起来”
2. 再看大陆有没有把任务批次收下
- 没收下的话，说明卡在“海外发过去了，但大陆没接住”
3. 再看 `claimed` 和 `running` 有没有开始增长
- 不增长的话，说明 worker 没真正开始领任务
4. 再看 `completed / failed / blacklisted` 有没有开始变化
- 长时间不变的话，说明任务虽然在跑，但结果没持续产出
5. 最后看海外页面有没有跟着更新
- 大陆明明在跑，海外不更新，通常就是“结果同步链”有问题
## 最后只记住 4 句话就够了
1. 现在这套项目是“海外控制，大陆执行”，不是单机直跑。
2. 一个域名不是一次跑完，而是按步骤一小步一小步推进。
3. 页面看到的是汇总快照，不是某台机器的原始现场数字。
4. 当前最大的矛盾仍然是链路稳定，不是单纯把性能参数继续调大。
继续看完了，当前是稳推进，不是又卡死。
现在现场是：
- `idx_detect_job_items_release_node_job` 还没转 `valid=1`
- 但已经稳定在 `index validation: scanning table`
- 最新进度到：
- `229991 / 6197384`
- `watch` 这条后台链是活的，而且一直在连续记进度：
- `18:09:51 -> 126725`
- `18:10:52 -> 169564`
- `18:11:52 -> 210067`
- `18:12:22 -> 227856`
另外两点也正常：
- `876` 现在还显示 `running`
- worker 已经全恢复：
- `base=active`
- `templated=199`
- `node_agent=active`
一句话说：
**现在已经从“卡住”切成“后台验证扫描中”，而且进度在稳定往前走。**
下一步最合理的动作不是再人工乱动，而是继续让它扫；一旦索引转成 `valid=1`，`codex-job876-tail-watch.service` 就会自动接着去释放 `876` 的尾项。
发布 sync_push_service.py
重启 domaincheck-sync-agent.service
验证：
detect_result_projection push 是否恢复
detect_result_ingest 是否出现

152
domain-api/app/api/routes/detect.py
View File

@@ -2,7 +2,7 @@ from __future__ import annotations
from uuid import uuid4
from fastapi import APIRouter
from fastapi import APIRouter, Body
from app.core.config import settings
from app.schemas.common import ApiResponse
@@ -18,7 +18,7 @@ from app.services.detect_job_service import (
from app.services.detect_service import get_detect_status
from app.services.detect_run_service import create_detect_run_snapshot, finalize_detect_run, mark_detect_run_stopping
from app.services.ops_job_service import create_ops_job, list_managed_nodes
from app.services.settings_service import get_settings_payload, resolve_thread_count
from app.services.settings_service import get_settings_payload, resolve_process_count, resolve_thread_count
from app.services.worker_control_service import send_worker_command, start_worker
router = APIRouter(tags=["detect"])
@@ -66,7 +66,13 @@ def _build_detect_action_result(
def _build_settings_summary(settings_payload: dict) -> dict:
thread_count_resolution = resolve_thread_count(settings_payload=settings_payload)
process_count_resolution = resolve_process_count(settings_payload=settings_payload)
return {
"process_count": int(process_count_resolution["effective_process_count"]),
"process_count_default": int(process_count_resolution["default_process_count"]),
"process_count_source": str(process_count_resolution["source"]),
"process_count_override": process_count_resolution["override_process_count"],
"process_count_node_code": str(process_count_resolution["node_code"]),
"thread_count": int(thread_count_resolution["effective_thread_count"]),
"thread_count_default": int(thread_count_resolution["default_thread_count"]),
"thread_count_source": str(thread_count_resolution["source"]),
@@ -78,6 +84,38 @@ def _build_settings_summary(settings_payload: dict) -> dict:
}
def _normalize_target_node_codes(payload: dict | None) -> list[str]:
if not isinstance(payload, dict):
return []
normalized_targets: list[str] = []
def append_target(raw_value: object) -> None:
normalized_value = str(raw_value or "").strip()
if normalized_value and normalized_value not in normalized_targets:
normalized_targets.append(normalized_value)
for key in ("target_node_codes", "node_codes"):
raw_value = payload.get(key)
if isinstance(raw_value, (list, tuple, set)):
for item in raw_value:
append_target(item)
elif isinstance(raw_value, str) and raw_value.strip():
for item in raw_value.split(","):
append_target(item)
if normalized_targets:
return normalized_targets
for key in ("target_node_code", "node_code"):
raw_value = payload.get(key)
if raw_value not in (None, ""):
append_target(raw_value)
if normalized_targets:
return normalized_targets
return normalized_targets
def _mainland_detect_targets() -> dict[str, list[dict]]:
controllers: list[dict] = []
workers: list[dict] = []
@@ -212,15 +250,18 @@ def _dispatch_remote_detect_start(*, job_summary: dict, cycle_token: str) -> dic
}
def _dispatch_remote_detect_stop(*, active_job: dict | None, cycle_token: str = "") -> dict:
def _dispatch_remote_detect_stop(*, active_job: dict | None, cycle_token: str = "", payload: dict | None = None) -> dict:
targets = _mainland_detect_targets()
job_summary = active_job or {}
queued: list[dict] = []
target_node_codes = _normalize_target_node_codes(payload)
for node in [*targets["controllers"], *targets["workers"]]:
node_code = str(node.get("node_code") or "").strip()
if not node_code:
continue
if target_node_codes and node_code not in target_node_codes:
continue
queued.append(
_queue_remote_detect_job(
node_code=node_code,
@@ -317,26 +358,63 @@ def start_detect(step_code: str | None = None) -> ApiResponse:
local_worker_expected = _local_worker_expected_on_this_node()
if local_worker_expected:
ok, message = start_worker()
if not ok:
result = _build_detect_action_result(
action="start",
ok=False,
message=message,
data={"job": job_summary},
)
prestart_snapshot = get_detect_status()
worker_already_running = bool(prestart_snapshot.get("worker_online", False)) or int(
prestart_snapshot.get("worker_process_count", 0) or 0
) > 0
if worker_already_running:
ok = True
message = "检测端已在运行，跳过重复启动，直接发送控制指令"
append_detect_job_event(
job_summary["job_id"],
event_type="job_dispatch_failed",
level="error",
message=f"启动 Worker 失败: {message}",
payload={"cycle_token": cycle_token},
)
return ApiResponse(
code=1,
event_type="job_dispatch_start_skipped",
level="info",
message=message,
data=result,
payload={
"cycle_token": cycle_token,
"worker_process_count": int(prestart_snapshot.get("worker_process_count", 0) or 0),
},
)
else:
ok, message = start_worker()
if not ok:
degraded_snapshot = get_detect_status()
worker_already_running = bool(degraded_snapshot.get("worker_online", False)) or int(
degraded_snapshot.get("worker_process_count", 0) or 0
) > 0
if not worker_already_running:
result = _build_detect_action_result(
action="start",
ok=False,
message=message,
data={"job": job_summary},
)
append_detect_job_event(
job_summary["job_id"],
event_type="job_dispatch_failed",
level="error",
message=f"启动 Worker 失败: {message}",
payload={"cycle_token": cycle_token},
)
return ApiResponse(
code=1,
message=message,
data=result,
)
degraded_message = f"{message}；检测端已在运行，改为直接发送控制指令"
append_detect_job_event(
job_summary["job_id"],
event_type="job_dispatch_start_degraded",
level="warning",
message=degraded_message,
payload={
"cycle_token": cycle_token,
"worker_process_count": int(degraded_snapshot.get("worker_process_count", 0) or 0),
},
)
ok = True
message = degraded_message
command_ok, command_message = send_worker_command(
"start_detection",
@@ -372,18 +450,19 @@ def start_detect(step_code: str | None = None) -> ApiResponse:
settings_summary = _build_settings_summary(settings_payload)
if command_ok:
remote_dispatch = _dispatch_remote_detect_start(job_summary=job_summary, cycle_token=cycle_token)
create_detect_run_snapshot(
message=f"{message}；{command_message}",
runtime={
"mode": snapshot.get("worker_mode", ""),
"running": snapshot.get("worker_online", False),
"process_count": snapshot.get("worker_process_count", 0),
"latest_start_time": snapshot.get("worker_latest_start_time", ""),
"message": snapshot.get("worker_runtime_message", ""),
},
progress=snapshot.get("progress", {}),
settings_summary=settings_summary,
)
if local_worker_expected:
create_detect_run_snapshot(
message=f"{message}；{command_message}",
runtime={
"mode": snapshot.get("worker_mode", ""),
"running": snapshot.get("worker_online", False),
"process_count": snapshot.get("worker_process_count", 0),
"latest_start_time": snapshot.get("worker_latest_start_time", ""),
"message": snapshot.get("worker_runtime_message", ""),
},
progress=snapshot.get("progress", {}),
settings_summary=settings_summary,
)
append_detect_job_event(
job_summary["job_id"],
event_type="job_dispatch_remote_queued",
@@ -411,11 +490,16 @@ def start_detect(step_code: str | None = None) -> ApiResponse:
@router.post("/detect/stop", response_model=ApiResponse)
def stop_detect() -> ApiResponse:
def stop_detect(payload: dict | None = Body(default=None)) -> ApiResponse:
active_job = get_active_detect_job_summary(event_limit=10)
ok, message = send_worker_command("stop_detection")
normalized_payload = {
key: value
for key, value in dict(payload or {}).items()
if value not in (None, "")
}
ok, message = send_worker_command("stop_detection", payload=normalized_payload)
cycle_token = str((active_job or {}).get("current_cycle_token") or "").strip()
remote_dispatch = _dispatch_remote_detect_stop(active_job=active_job, cycle_token=cycle_token)
remote_dispatch = _dispatch_remote_detect_stop(active_job=active_job, cycle_token=cycle_token, payload=normalized_payload)
if active_job:
append_detect_job_event(
active_job["job_id"],

22
domain-api/app/api/routes/ops.py
View File

@@ -31,6 +31,11 @@ from app.services.ops_job_service import (
sync_managed_nodes_from_cluster,
upsert_managed_node,
)
from app.services.ops_migration_service import (
execute_ops_migration,
get_ops_migration_source_profile,
preview_ops_migration,
)
from app.services.ops_playbook_service import (
cancel_ops_playbook_run,
execute_ops_playbook,
@@ -80,6 +85,23 @@ def ops_overview() -> ApiResponse:
return ApiResponse(data=get_ops_overview())
@router.get("/ops/migration/source-profile", response_model=ApiResponse)
def ops_migration_source_profile() -> ApiResponse:
return ApiResponse(data=get_ops_migration_source_profile())
@router.post("/ops/migration/preview", response_model=ApiResponse)
def ops_migration_preview(payload: dict | None = None) -> ApiResponse:
ok, message, data = preview_ops_migration(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/ops/migration/execute", response_model=ApiResponse)
def ops_migration_execute(payload: dict | None = None) -> ApiResponse:
ok, message, data = execute_ops_migration(payload or {})
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.get("/ops/link-snapshot", response_model=ApiResponse)
def ops_link_snapshot() -> ApiResponse:
return ApiResponse(data=get_ops_link_snapshot())

58
domain-api/app/api/routes/runtime.py
View File

@@ -19,6 +19,55 @@ from app.services.sync_record_service import get_sync_summary, list_sync_records
router = APIRouter(tags=["runtime"])
def _filter_debug_handover_by_node_code(payload: dict, node_code: str) -> dict:
normalized_node_code = str(node_code or "").strip()
if not normalized_node_code:
return dict(payload or {})
normalized_payload = dict(payload or {})
def _matches_node(item: object) -> bool:
if not isinstance(item, dict):
return False
return str(item.get("node_code") or "").strip() == normalized_node_code
overview = dict(normalized_payload.get("overview") or {})
if overview:
overview["recent_issues"] = [
item
for item in list(overview.get("recent_issues") or [])
if _matches_node(item)
]
normalized_payload["overview"] = overview
normalized_payload["recent_issues"] = [
item
for item in list(normalized_payload.get("recent_issues") or [])
if _matches_node(item)
]
normalized_payload["issue_groups"] = [
item
for item in list(normalized_payload.get("issue_groups") or [])
if _matches_node(item)
]
failure_handoff = dict(normalized_payload.get("failure_handoff") or {})
if failure_handoff:
failure_handoff["recent_issues"] = [
item
for item in list(failure_handoff.get("recent_issues") or [])
if _matches_node(item)
]
failure_handoff["issue_groups"] = [
item
for item in list(failure_handoff.get("issue_groups") or [])
if _matches_node(item)
]
normalized_payload["failure_handoff"] = failure_handoff
return normalized_payload
@router.get("/runtime/status", response_model=ApiResponse)
def runtime_status() -> ApiResponse:
return ApiResponse(data=get_runtime_status())
@@ -60,6 +109,7 @@ def runtime_debug_events(
service: Optional[str] = None,
event_type: Optional[str] = None,
source_region: Optional[str] = None,
node_code: Optional[str] = None,
level: Optional[str] = None,
before_id: Optional[int] = None,
after_id: Optional[int] = None,
@@ -71,6 +121,7 @@ def runtime_debug_events(
service=service,
event_type=event_type,
source_region=source_region,
node_code=node_code,
level=level,
before_id=before_id,
after_id=after_id,
@@ -125,12 +176,7 @@ def runtime_health_handover(
)
normalized_node_code = str(node_code or "").strip()
if normalized_node_code:
data = dict(data)
data["nodes"] = [
item
for item in list(data.get("nodes") or [])
if str(item.get("node_code") or "").strip() == normalized_node_code
]
data = _filter_debug_handover_by_node_code(data, normalized_node_code)
return ApiResponse(data=data)

65
domain-api/app/core/redis_client.py
View File

@@ -1,17 +1,64 @@
from __future__ import annotations
import os
import threading
import redis
from app.core.config import settings
_REDIS_CLIENT: redis.Redis | None = None
_LOCK = threading.Lock()
def _safe_int(raw_value: object, default: int, minimum: int) -> int:
try:
parsed = int(raw_value)
except Exception:
parsed = default
return max(minimum, parsed)
def _safe_float(raw_value: object, default: float, minimum: float) -> float:
try:
parsed = float(raw_value)
except Exception:
parsed = default
return max(minimum, parsed)
def get_redis() -> redis.Redis:
return redis.Redis(
host=settings.redis_host,
port=settings.redis_port,
password=settings.redis_password or None,
db=settings.redis_db,
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
)
global _REDIS_CLIENT
with _LOCK:
if _REDIS_CLIENT is not None:
return _REDIS_CLIENT
pool = redis.BlockingConnectionPool(
host=settings.redis_host,
port=settings.redis_port,
password=settings.redis_password or None,
db=settings.redis_db,
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
health_check_interval=_safe_int(os.getenv("DOMAIN_API_REDIS_HEALTH_CHECK_INTERVAL", "30"), 30, 0),
retry_on_timeout=True,
max_connections=_safe_int(os.getenv("DOMAIN_API_REDIS_MAX_CONNECTIONS", "32"), 32, 1),
timeout=_safe_float(os.getenv("DOMAIN_API_REDIS_POOL_TIMEOUT", "1.5"), 1.5, 0.1),
client_name=f"domain-api:{settings.node_code}:{os.getpid()}",
)
_REDIS_CLIENT = redis.Redis(connection_pool=pool)
return _REDIS_CLIENT
def reset_redis_client_for_tests() -> None:
global _REDIS_CLIENT
with _LOCK:
client = _REDIS_CLIENT
_REDIS_CLIENT = None
if client is not None:
try:
client.close()
except Exception:
pass

33
domain-api/app/main.py
View File

@@ -11,6 +11,23 @@ from app.services.ops_agent_service import ensure_ops_agent_schema
from app.services.ops_job_service import ensure_ops_schema
from app.services.ops_release_service import ensure_ops_release_schema
_heartbeat_stop_event = threading.Event()
_bootstrap_started = False
_bootstrap_lock = threading.Lock()
def _bootstrap_runtime_prerequisites() -> None:
for task in (
ensure_runtime_schema,
ensure_ops_schema,
ensure_ops_agent_schema,
ensure_ops_release_schema,
register_local_control_heartbeat,
):
try:
task()
except Exception:
# Startup must not block API listening on slow schema checks or DB stalls.
pass
def _control_heartbeat_loop() -> None:
@@ -39,14 +56,18 @@ app.add_middleware(
@app.on_event("startup")
def on_startup() -> None:
ensure_runtime_schema()
ensure_ops_schema()
ensure_ops_agent_schema()
ensure_ops_release_schema()
global _bootstrap_started
remember_registered_route_paths(route.path for route in app.routes)
register_local_control_heartbeat()
_heartbeat_stop_event.clear()
threading.Thread(target=_control_heartbeat_loop, name="control-heartbeat", daemon=True).start()
with _bootstrap_lock:
if not _bootstrap_started:
threading.Thread(
target=_bootstrap_runtime_prerequisites,
name="api-bootstrap",
daemon=True,
).start()
threading.Thread(target=_control_heartbeat_loop, name="control-heartbeat", daemon=True).start()
_bootstrap_started = True
@app.on_event("shutdown")

309
domain-api/app/node_agent.py
View File

@@ -10,6 +10,7 @@ import time
import urllib.error
import urllib.request
from datetime import datetime
from pathlib import Path
from urllib.parse import urlparse
from uuid import uuid4
@@ -678,6 +679,265 @@ def _publish_local_config_update(config_type: str) -> None:
return
def _normalize_process_count(value: object, default: int = 1) -> int:
try:
normalized = int(value)
except Exception:
return max(1, int(default or 1))
return max(1, normalized)
def _resolve_desired_process_count(bundle: dict) -> int:
normalized_bundle = dict(bundle or {})
node_process_counts = normalized_bundle.get("node_process_counts")
if isinstance(node_process_counts, dict):
override_value = node_process_counts.get(NODE_CODE)
if override_value not in (None, ""):
return _normalize_process_count(override_value, default=80)
return _normalize_process_count(normalized_bundle.get("process_count", 80), default=80)
def _worker_instance_suffixes(extra_count: int) -> list[str]:
alphabet = "abcdefghijklmnopqrstuvwxyz"
suffixes: list[str] = []
normalized_extra_count = max(0, int(extra_count or 0))
if normalized_extra_count <= 0:
return suffixes
for char in alphabet:
suffixes.append(char)
if len(suffixes) >= normalized_extra_count:
return suffixes
for first in alphabet:
for second in alphabet:
suffixes.append(f"{first}{second}")
if len(suffixes) >= normalized_extra_count:
return suffixes
return suffixes[:normalized_extra_count]
def _worker_env_path(service_name: str, suffix: str = "") -> Path:
normalized_service_name = str(service_name or "").strip() or WORKER_SERVICE_NAME
suffix_text = f"-{suffix}" if str(suffix or "").strip() else ""
return Path("/etc/default") / f"{normalized_service_name}{suffix_text}"
def _worker_instance_unit_path(service_name: str) -> Path:
normalized_service_name = str(service_name or "").strip() or WORKER_SERVICE_NAME
return Path("/etc/systemd/system") / f"{normalized_service_name}@.service"
def _worker_instance_unit_name(service_name: str, suffix: str) -> str:
normalized_service_name = str(service_name or "").strip() or WORKER_SERVICE_NAME
return f"{normalized_service_name}@{suffix}"
def _template_worker_unit_source_path() -> Path:
return Path(_PROJECT_DIR) / "deploy" / "systemd" / "domain-worker@.service"
def _render_worker_instance_env(base_env_text: str, *, instance_node_code: str, parent_node_code: str) -> str:
lines: list[str] = []
saw_node_code = False
saw_parent = False
for raw_line in str(base_env_text or "").splitlines():
if raw_line.startswith("NODE_CODE="):
lines.append(f"NODE_CODE={instance_node_code}")
saw_node_code = True
continue
if raw_line.startswith("WORKER_PARENT_NODE_CODE="):
lines.append(f"WORKER_PARENT_NODE_CODE={parent_node_code}")
saw_parent = True
continue
lines.append(raw_line)
if not saw_node_code:
lines.append(f"NODE_CODE={instance_node_code}")
if not saw_parent:
lines.append(f"WORKER_PARENT_NODE_CODE={parent_node_code}")
return "\n".join(lines).rstrip() + "\n"
def _ensure_worker_instance_unit_template(service_name: str) -> tuple[bool, str]:
unit_path = _worker_instance_unit_path(service_name)
source_path = _template_worker_unit_source_path()
if not source_path.exists():
return False, f"worker instance template missing: {source_path}"
try:
template_text = source_path.read_text(encoding="utf-8")
normalized_service_name = str(service_name or "").strip() or WORKER_SERVICE_NAME
rendered = (
template_text
.replace("domaincheck-worker-%i", f"{normalized_service_name}-%i")
.replace("domaincheck-worker@%i", f"{normalized_service_name}@%i")
)
if unit_path.exists():
existing = unit_path.read_text(encoding="utf-8")
if existing == rendered:
return True, str(unit_path)
unit_path.write_text(rendered, encoding="utf-8")
return True, str(unit_path)
except Exception as exc:
return False, f"write worker instance template failed: {exc}"
def _worker_instance_start_batch_size() -> int:
raw_value = str(os.getenv("NODE_AGENT_WORKER_RECONCILE_BATCH_SIZE", "") or "").strip()
try:
return max(1, min(32, int(raw_value or 6)))
except Exception:
return 6
def _worker_instance_start_batch_delay_seconds() -> float:
raw_value = str(os.getenv("NODE_AGENT_WORKER_RECONCILE_BATCH_DELAY_SECONDS", "") or "").strip()
try:
return max(0.0, min(30.0, float(raw_value or 1.0)))
except Exception:
return 1.0
def _chunked_units(units: list[str], size: int) -> list[list[str]]:
batch_size = max(1, int(size or 1))
return [units[index:index + batch_size] for index in range(0, len(units), batch_size)]
def _reconcile_worker_instances(bundle: dict) -> dict:
runtime_settings = dict(bundle.get("runtime_settings") or {})
worker_mode = str(runtime_settings.get("worker_mode") or "").strip() or "windows-local"
worker_service_name = str(runtime_settings.get("worker_service_name") or WORKER_SERVICE_NAME).strip() or WORKER_SERVICE_NAME
desired_process_count = _resolve_desired_process_count(bundle)
if worker_mode != "linux-systemd":
return {
"applied": False,
"reason": f"worker_mode={worker_mode}",
"desired_process_count": desired_process_count,
}
if NODE_REGION != "mainland":
return {
"applied": False,
"reason": f"region={NODE_REGION}",
"desired_process_count": desired_process_count,
}
base_env_path = _worker_env_path(worker_service_name)
if not base_env_path.exists():
return {
"applied": False,
"reason": f"base env missing: {base_env_path}",
"desired_process_count": desired_process_count,
}
ok, template_message = _ensure_worker_instance_unit_template(worker_service_name)
if not ok:
return {
"applied": False,
"reason": template_message,
"desired_process_count": desired_process_count,
}
desired_suffixes = _worker_instance_suffixes(max(0, desired_process_count - 1))
desired_units = [_worker_instance_unit_name(worker_service_name, suffix) for suffix in desired_suffixes]
desired_env_paths = {_worker_env_path(worker_service_name, suffix) for suffix in desired_suffixes}
managed_prefix = f"{worker_service_name}-"
try:
base_env_text = base_env_path.read_text(encoding="utf-8")
for suffix in desired_suffixes:
env_path = _worker_env_path(worker_service_name, suffix)
env_path.write_text(
_render_worker_instance_env(
base_env_text,
instance_node_code=f"{NODE_CODE}-{suffix}",
parent_node_code=NODE_CODE,
),
encoding="utf-8",
)
except Exception as exc:
return {
"applied": False,
"reason": f"write worker env failed: {exc}",
"desired_process_count": desired_process_count,
}
existing_env_paths: list[Path] = []
try:
for candidate in Path("/etc/default").iterdir():
if not candidate.is_file():
continue
if not candidate.name.startswith(managed_prefix):
continue
existing_env_paths.append(candidate)
except Exception:
existing_env_paths = []
stale_env_paths = [
candidate
for candidate in existing_env_paths
if candidate not in desired_env_paths
]
rc, stdout, stderr = _run(["systemctl", "daemon-reload"], timeout=90)
if rc != 0:
return {
"applied": False,
"reason": stderr or stdout or "systemctl daemon-reload failed",
"desired_process_count": desired_process_count,
}
if desired_units:
rc, stdout, stderr = _run(["systemctl", "enable", *desired_units], timeout=180)
if rc != 0:
return {
"applied": False,
"reason": stderr or stdout or "systemctl enable worker instances failed",
"desired_process_count": desired_process_count,
}
batch_size = _worker_instance_start_batch_size()
batch_delay_seconds = _worker_instance_start_batch_delay_seconds()
for batch_index, unit_batch in enumerate(_chunked_units(desired_units, batch_size), start=1):
rc, stdout, stderr = _run(
["systemctl", "start", *unit_batch],
timeout=max(120, 30 * len(unit_batch)),
)
if rc != 0:
return {
"applied": False,
"reason": (
stderr
or stdout
or f"systemctl start worker instances failed at batch {batch_index}"
),
"desired_process_count": desired_process_count,
}
if batch_delay_seconds > 0 and batch_index * batch_size < len(desired_units):
time.sleep(batch_delay_seconds)
stale_units = [
f"{worker_service_name}@{candidate.name[len(managed_prefix):]}"
for candidate in stale_env_paths
if candidate.name[len(managed_prefix):]
]
if stale_units:
_run(["systemctl", "stop", *stale_units], timeout=180)
_run(["systemctl", "disable", *stale_units], timeout=180)
for candidate in stale_env_paths:
try:
candidate.unlink()
except Exception:
continue
return {
"applied": True,
"reason": "reconciled",
"desired_process_count": desired_process_count,
"instance_service_name": f"{worker_service_name}@.service",
"extra_instances": len(desired_suffixes),
"stale_instances_removed": len(stale_units),
"template_path": template_message,
}
def _apply_runtime_config(bundle: dict) -> bool:
global _LAST_RUNTIME_CONFIG_HASH
@@ -688,6 +948,13 @@ def _apply_runtime_config(bundle: dict) -> bool:
json.dumps(normalized_bundle, ensure_ascii=False, sort_keys=True).encode("utf-8")
).hexdigest()
if bundle_hash and bundle_hash == _LAST_RUNTIME_CONFIG_HASH:
reconcile_summary = _reconcile_worker_instances(normalized_bundle)
if reconcile_summary.get("applied"):
_log(
"worker instance reconcile refreshed: "
f"desired={reconcile_summary.get('desired_process_count', 1)} "
f"extra={reconcile_summary.get('extra_instances', 0)}"
)
return False
from app.core.files import write_json
@@ -696,8 +963,10 @@ def _apply_runtime_config(bundle: dict) -> bool:
detect_options = dict(normalized_bundle.get("detect_options") or {})
proxy_config = dict(normalized_bundle.get("proxy_config") or {})
thread_count = int(normalized_bundle.get("thread_count", 2) or 2)
thread_count = int(normalized_bundle.get("thread_count", 1000) or 1000)
node_thread_counts = dict(normalized_bundle.get("node_thread_counts") or {})
process_count = int(normalized_bundle.get("process_count", 80) or 80)
node_process_counts = dict(normalized_bundle.get("node_process_counts") or {})
runtime_settings = dict(normalized_bundle.get("runtime_settings") or {})
sensitive_words = dict(normalized_bundle.get("sensitive_words") or {})
sensitive_words_text = str(sensitive_words.get("text") or "")
@@ -707,6 +976,8 @@ def _apply_runtime_config(bundle: dict) -> bool:
write_json("proxy_config.json", proxy_config)
write_json("thread_count.json", {"thread_count": str(thread_count)})
write_json("node_thread_counts.json", node_thread_counts)
write_json("process_count.json", {"process_count": str(process_count)})
write_json("node_process_counts.json", node_process_counts)
write_json("runtime_settings.json", runtime_settings)
write_json("runtime/runtime_settings.json", runtime_settings)
write_json(
@@ -736,12 +1007,16 @@ def _apply_runtime_config(bundle: dict) -> bool:
redis_client.set("domain_tool:proxy_config", json.dumps(proxy_config, ensure_ascii=False))
redis_client.set("domain_tool:thread_count", thread_count)
redis_client.set("domain_tool:node_thread_counts", json.dumps(node_thread_counts, ensure_ascii=False))
redis_client.set("domain_tool:process_count", process_count)
redis_client.set("domain_tool:node_process_counts", json.dumps(node_process_counts, ensure_ascii=False))
redis_client.set("domain_tool:runtime_settings", json.dumps(runtime_settings, ensure_ascii=False))
redis_client.set("domain_tool:sensitive_words", json.dumps(sensitive_word_items, ensure_ascii=False))
redis_client.publish("domain_tool:config_update", "detect_options")
redis_client.publish("domain_tool:config_update", "proxy_config")
redis_client.publish("domain_tool:config_update", "thread_count")
redis_client.publish("domain_tool:config_update", "node_thread_counts")
redis_client.publish("domain_tool:config_update", "process_count")
redis_client.publish("domain_tool:config_update", "node_process_counts")
redis_client.publish("domain_tool:config_update", "runtime_settings")
redis_client.publish("domain_tool:config_update", "sensitive_words")
except Exception:
@@ -749,16 +1024,30 @@ def _apply_runtime_config(bundle: dict) -> bool:
_publish_local_config_update("proxy_config")
_publish_local_config_update("thread_count")
_publish_local_config_update("node_thread_counts")
_publish_local_config_update("process_count")
_publish_local_config_update("node_process_counts")
_publish_local_config_update("runtime_settings")
_publish_local_config_update("sensitive_words")
reconcile_summary = _reconcile_worker_instances(normalized_bundle)
_LAST_RUNTIME_CONFIG_HASH = bundle_hash
_log(
"runtime config applied: "
f"thread_count={thread_count} "
f"process_count={process_count} "
f"node_override={node_thread_counts.get(NODE_CODE)} "
f"process_override={node_process_counts.get(NODE_CODE)} "
f"sensitive_words={int(sensitive_words.get('total', 0) or 0)}"
)
if reconcile_summary.get("applied"):
_log(
"worker instance reconcile: "
f"desired={reconcile_summary.get('desired_process_count', 1)} "
f"extra={reconcile_summary.get('extra_instances', 0)} "
f"removed={reconcile_summary.get('stale_instances_removed', 0)}"
)
elif reconcile_summary.get("reason"):
_log(f"worker instance reconcile skipped: {reconcile_summary.get('reason')}")
return True
@@ -914,6 +1203,15 @@ def _detect_runtime_snapshot() -> dict:
"phase_detail": phase_detail,
"recent_warning": str(detect_status.get("recent_warning") or "").strip(),
"updated_at": str(runtime_state.get("updated_at") or "").strip(),
"available_proxy_count": int(detect_status.get("available_proxy_count", 0) or 0),
"proxy_runtime_label": str(detect_status.get("proxy_runtime_label") or "").strip(),
"proxy_runtime_reason": str(detect_status.get("proxy_runtime_reason") or "").strip(),
"proxy_last_refresh_status": str(detect_status.get("proxy_last_refresh_status") or "").strip(),
"proxy_last_refresh_time": str(detect_status.get("proxy_last_refresh_time") or "").strip(),
"proxy_last_refresh_source_count": int(detect_status.get("proxy_last_refresh_source_count", 0) or 0),
"proxy_last_refresh_total_items": int(detect_status.get("proxy_last_refresh_total_items", 0) or 0),
"proxy_last_validated_count": int(detect_status.get("proxy_last_validated_count", 0) or 0),
"proxy_last_available_count": int(detect_status.get("proxy_last_available_count", 0) or 0),
"detect_participating": bool(
detect_status.get("detect_participating", False)
or current_load > 0
@@ -934,6 +1232,15 @@ def _detect_runtime_snapshot() -> dict:
"phase_detail": worker_message,
"recent_warning": "",
"updated_at": str(worker_runtime.get("latest_start_time") or "").strip(),
"available_proxy_count": 0,
"proxy_runtime_label": "",
"proxy_runtime_reason": "",
"proxy_last_refresh_status": "",
"proxy_last_refresh_time": "",
"proxy_last_refresh_source_count": 0,
"proxy_last_refresh_total_items": 0,
"proxy_last_validated_count": 0,
"proxy_last_available_count": 0,
"detect_participating": False,
"error": str(exc),
}

3
domain-api/app/services/build_info_service.py
View File

@@ -164,6 +164,9 @@ def _expected_route_paths() -> dict[str, str]:
"ops_contracts": f"{prefix}/ops/contracts",
"ops_contract_detail": f"{prefix}/ops/contracts/{{contract_key}}",
"ops_stack_diagnosis": f"{prefix}/ops/stack-diagnosis",
"ops_migration_source_profile": f"{prefix}/ops/migration/source-profile",
"ops_migration_preview": f"{prefix}/ops/migration/preview",
"ops_migration_execute": f"{prefix}/ops/migration/execute",
"ops_node_handover": f"{prefix}/ops/nodes/{{node_code}}/handover",
"ops_node_onboarding": f"{prefix}/ops/nodes/{{node_code}}/onboarding",
"ops_node_onboarding_bootstrap_preview": f"{prefix}/ops/nodes/{{node_code}}/onboarding/bootstrap/preview",

416
domain-api/app/services/cluster_runtime_service.py
View File

@@ -5,6 +5,8 @@ import socket
import threading
from datetime import datetime, timedelta
from psycopg2 import errors
from app.core.config import settings
from app.core.db import db_read_retry, get_db
@@ -63,6 +65,22 @@ CREATE TABLE IF NOT EXISTS detect_job_items (
CREATE INDEX IF NOT EXISTS idx_detect_job_items_status_lease
ON detect_job_items(status, lease_expires_at);
CREATE INDEX IF NOT EXISTS idx_detect_job_items_claim_ready
ON detect_job_items(status, create_time, id)
WHERE step_code <> '';
CREATE INDEX IF NOT EXISTS idx_detect_job_items_claim_job_ready
ON detect_job_items(job_id, status, create_time, id)
WHERE step_code <> '';
CREATE INDEX IF NOT EXISTS idx_detect_job_items_claim_step_ready
ON detect_job_items(status, step_code, lease_expires_at, create_time, id)
WHERE step_code <> '' AND status IN ('pending', 'failed');
CREATE INDEX IF NOT EXISTS idx_detect_job_items_claim_job_step_ready
ON detect_job_items(job_id, status, step_code, lease_expires_at, create_time, id)
WHERE step_code <> '' AND status IN ('pending', 'failed');
ALTER TABLE detect_jobs
ADD COLUMN IF NOT EXISTS task_mode VARCHAR(32) NOT NULL DEFAULT 'domain_pipeline',
ADD COLUMN IF NOT EXISTS step_code VARCHAR(64) NOT NULL DEFAULT '';
@@ -108,11 +126,100 @@ CREATE TABLE IF NOT EXISTS detect_sync_records (
_STALE_AFTER_SECONDS = 90
_OFFLINE_AFTER_MINUTES = 5
_IMPORTED_RUNTIME_STALE_AFTER_MINUTES = 10
_IMPORTED_RUNTIME_OFFLINE_AFTER_MINUTES = 30
_PRUNE_IMPORTED_AFTER_MINUTES = 30
_PRUNE_GENERAL_AFTER_HOURS = 6
_RUNTIME_SCHEMA_READY = False
_RUNTIME_SCHEMA_LOCK = threading.Lock()
_RUNTIME_SCHEMA_ADVISORY_LOCK_ID = 62021001
_RUNTIME_SCHEMA_INDEX_ADVISORY_LOCK_ID = 62021002
_DISABLED_MANAGED_NODE_CACHE: set[str] = set()
_DISABLED_MANAGED_NODE_CACHE_EXPIRES_AT = 0.0
_RUNTIME_REQUIRED_TABLES = (
"detect_worker_nodes",
"detect_jobs",
"detect_job_items",
"detect_run_events",
"detect_sync_records",
)
_RUNTIME_REQUIRED_COLUMNS = {
"detect_jobs": {"task_mode", "step_code"},
"detect_job_items": {"step_code", "step_payload_json", "result_payload_json"},
}
_RUNTIME_REQUIRED_INDEXES = (
"idx_detect_job_items_job_domain_step",
"idx_detect_job_items_claim_step_ready",
"idx_detect_job_items_claim_job_step_ready",
"idx_detect_sync_records_scope_created",
"idx_detect_sync_records_source_record_created",
"idx_detect_sync_records_source_record_hash_created",
"idx_detect_sync_records_runtime_push_lookup",
)
_RUNTIME_REQUIRED_INDEX_TABLES = {
"idx_detect_job_items_job_domain_step": "detect_job_items",
"idx_detect_job_items_claim_step_ready": "detect_job_items",
"idx_detect_job_items_claim_job_step_ready": "detect_job_items",
"idx_detect_sync_records_scope_created": "detect_sync_records",
"idx_detect_sync_records_source_record_created": "detect_sync_records",
"idx_detect_sync_records_source_record_hash_created": "detect_sync_records",
"idx_detect_sync_records_runtime_push_lookup": "detect_sync_records",
}
_RUNTIME_REQUIRED_INDEX_DDL = {
"idx_detect_job_items_job_domain_step": """
CREATE UNIQUE INDEX CONCURRENTLY IF NOT EXISTS idx_detect_job_items_job_domain_step
ON detect_job_items(job_id, domain_id, step_code)
""",
"idx_detect_job_items_claim_step_ready": """
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_detect_job_items_claim_step_ready
ON detect_job_items(status, step_code, lease_expires_at, create_time, id)
WHERE step_code <> '' AND status IN ('pending', 'failed')
""",
"idx_detect_job_items_claim_job_step_ready": """
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_detect_job_items_claim_job_step_ready
ON detect_job_items(job_id, status, step_code, lease_expires_at, create_time, id)
WHERE step_code <> '' AND status IN ('pending', 'failed')
""",
"idx_detect_sync_records_scope_created": """
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_detect_sync_records_scope_created
ON detect_sync_records(sync_type, source_region, target_region, created_at DESC, id DESC)
""",
"idx_detect_sync_records_source_record_created": """
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_detect_sync_records_source_record_created
ON detect_sync_records(
sync_type,
source_region,
target_region,
((payload_json->>'source_record_id')),
created_at DESC,
id DESC
)
""",
"idx_detect_sync_records_source_record_hash_created": """
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_detect_sync_records_source_record_hash_created
ON detect_sync_records(
sync_type,
source_region,
target_region,
((payload_json->>'source_record_id')),
((payload_json->>'projection_hash')),
created_at DESC,
id DESC
)
""",
"idx_detect_sync_records_runtime_push_lookup": """
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_detect_sync_records_runtime_push_lookup
ON detect_sync_records(
source_region,
target_region,
((payload_json->>'sync_type')),
((payload_json->>'source_record_id')),
created_at DESC,
id DESC
)
WHERE sync_type = 'runtime_push'
""",
}
def _resolve_local_ip() -> str:
@@ -133,6 +240,17 @@ def _decode_json(value: object) -> dict:
return {}
def _parse_runtime_timestamp(value: object) -> datetime | None:
raw = str(value or "").strip()
if not raw:
return None
normalized = raw.replace("Z", "+00:00")
try:
return datetime.fromisoformat(normalized)
except Exception:
return None
def _control_node_supports_worker(*, region: object, metadata: dict | None) -> bool:
normalized_region = str(region or "").strip()
runtime_metadata = dict(metadata or {})
@@ -198,6 +316,35 @@ def _load_managed_node_overlays() -> dict[str, dict]:
return overlays
def _load_disabled_managed_node_codes() -> set[str]:
global _DISABLED_MANAGED_NODE_CACHE, _DISABLED_MANAGED_NODE_CACHE_EXPIRES_AT
now_ts = datetime.now().timestamp()
if now_ts < _DISABLED_MANAGED_NODE_CACHE_EXPIRES_AT:
return set(_DISABLED_MANAGED_NODE_CACHE)
try:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT node_code
FROM ops_managed_nodes
WHERE is_enabled = FALSE
"""
)
rows = cur.fetchall()
except Exception:
return set(_DISABLED_MANAGED_NODE_CACHE)
disabled_codes = {
str(row[0] or "").strip()
for row in list(rows or [])
if str(row[0] or "").strip()
}
_DISABLED_MANAGED_NODE_CACHE = disabled_codes
_DISABLED_MANAGED_NODE_CACHE_EXPIRES_AT = now_ts + 5.0
return set(disabled_codes)
def ensure_runtime_schema() -> None:
global _RUNTIME_SCHEMA_READY
if _RUNTIME_SCHEMA_READY:
@@ -206,14 +353,131 @@ def ensure_runtime_schema() -> None:
if _RUNTIME_SCHEMA_READY:
return
with get_db() as conn:
conn.autocommit = False
with conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (_RUNTIME_SCHEMA_ADVISORY_LOCK_ID,))
cur.execute(_RUNTIME_SCHEMA_SQL)
conn.commit()
if _runtime_schema_basics_present(cur):
missing_indexes = list(_runtime_missing_indexes(cur))
if not missing_indexes:
_RUNTIME_SCHEMA_READY = True
return
else:
missing_indexes = []
if missing_indexes:
_ensure_runtime_schema_indexes(missing_indexes)
with conn.cursor() as cur:
if _runtime_schema_basics_present(cur) and not list(_runtime_missing_indexes(cur)):
_RUNTIME_SCHEMA_READY = True
return
else:
try:
with conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (_RUNTIME_SCHEMA_ADVISORY_LOCK_ID,))
cur.execute(_RUNTIME_SCHEMA_SQL)
conn.commit()
except Exception as exc:
recoverable = isinstance(exc, (errors.DeadlockDetected, errors.LockNotAvailable))
try:
conn.rollback()
except Exception:
pass
if not recoverable:
raise
with conn.cursor() as cur:
if not _runtime_schema_basics_present(cur):
raise
_RUNTIME_SCHEMA_READY = True
def _runtime_schema_basics_present(cur) -> bool:
for table_name in _RUNTIME_REQUIRED_TABLES:
cur.execute("SELECT to_regclass(%s)", (f"public.{table_name}",))
row = cur.fetchone()
if not row or not row[0]:
return False
for table_name, required_columns in _RUNTIME_REQUIRED_COLUMNS.items():
cur.execute(
"""
SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = %s
""",
(table_name,),
)
existing_columns = {str(row[0] or "").strip() for row in list(cur.fetchall() or [])}
if not set(required_columns).issubset(existing_columns):
return False
return True
def _runtime_missing_indexes(cur):
states = _runtime_index_states(cur)
for index_name in _RUNTIME_REQUIRED_INDEXES:
if not bool((states.get(index_name) or {}).get("valid")):
yield index_name
def _runtime_index_states(cur) -> dict[str, dict[str, bool]]:
table_names = sorted(set(_RUNTIME_REQUIRED_INDEX_TABLES.values()))
cur.execute(
"""
SELECT
idx.relname AS index_name,
pg_index.indisvalid AS is_valid,
pg_index.indisready AS is_ready,
pg_index.indislive AS is_live
FROM pg_class AS idx
JOIN pg_index ON pg_index.indexrelid = idx.oid
JOIN pg_class AS tbl ON tbl.oid = pg_index.indrelid
JOIN pg_namespace AS ns ON ns.oid = tbl.relnamespace
WHERE ns.nspname = 'public'
AND tbl.relname = ANY(%s)
AND idx.relname = ANY(%s)
""",
(table_names, list(_RUNTIME_REQUIRED_INDEXES)),
)
states = {
index_name: {"valid": False, "ready": False, "live": False}
for index_name in _RUNTIME_REQUIRED_INDEXES
}
for row in list(cur.fetchall() or []):
index_name = str(row[0] or "").strip()
if index_name not in states:
continue
states[index_name] = {
"valid": bool(row[1]),
"ready": bool(row[2]),
"live": bool(row[3]),
}
return states
def _ensure_runtime_schema_indexes(index_names: list[str] | tuple[str, ...]) -> None:
normalized_indexes = [
index_name
for index_name in list(index_names or [])
if str(index_name or "").strip() in _RUNTIME_REQUIRED_INDEX_DDL
]
if not normalized_indexes:
return
with get_db() as conn:
conn.autocommit = True
with conn.cursor() as cur:
cur.execute("SELECT pg_try_advisory_lock(%s)", (_RUNTIME_SCHEMA_INDEX_ADVISORY_LOCK_ID,))
row = cur.fetchone()
if not bool((row or [False])[0]):
return
try:
current_states = _runtime_index_states(cur)
for index_name in normalized_indexes:
index_state = current_states.get(index_name) or {}
if bool(index_state.get("valid")):
continue
if bool(index_state.get("ready")) or bool(index_state.get("live")):
cur.execute(f"DROP INDEX CONCURRENTLY IF EXISTS {index_name}")
cur.execute(_RUNTIME_REQUIRED_INDEX_DDL[index_name])
finally:
cur.execute("SELECT pg_advisory_unlock(%s)", (_RUNTIME_SCHEMA_INDEX_ADVISORY_LOCK_ID,))
def register_node_heartbeat(
*,
node_code: str,
@@ -351,14 +615,11 @@ def prune_expired_runtime_nodes() -> None:
def register_local_control_heartbeat() -> None:
from app.services.detect_job_service import get_active_detect_job_summary
from app.services.detect_service import get_detect_status
from app.services.worker_control_service import detect_worker_runtime
worker_runtime = detect_worker_runtime()
worker_online = bool(worker_runtime.get("running", False))
detect_status = get_detect_status()
active_job = get_active_detect_job_summary(event_limit=5) or {}
worker_online = bool(detect_status.get("worker_online", False))
active_job = dict(detect_status.get("active_job") or {})
node_stats = list(active_job.get("node_stats") or [])
local_bucket = next(
(item for item in node_stats if str(item.get("node_code") or "").strip() == settings.node_code),
@@ -385,7 +646,7 @@ def register_local_control_heartbeat() -> None:
"api_port": settings.api_port,
"worker_mode": settings.worker_mode,
"worker_online": worker_online,
"worker_process_count": int(worker_runtime.get("process_count", 0) or 0),
"worker_process_count": int(detect_status.get("worker_process_count", 0) or 0),
"detect_participating": detect_participating,
"active_job_code": str(active_job.get("job_code") or ""),
"active_job_status": str(active_job.get("status") or ""),
@@ -395,6 +656,15 @@ def register_local_control_heartbeat() -> None:
"job_items_completed": items_completed,
"active_threads": active_threads,
"max_threads": max_threads,
"available_proxy_count": int(detect_status.get("available_proxy_count", 0) or 0),
"proxy_runtime_label": str(detect_status.get("proxy_runtime_label") or "").strip(),
"proxy_runtime_reason": str(detect_status.get("proxy_runtime_reason") or "").strip(),
"proxy_last_refresh_status": str(detect_status.get("proxy_last_refresh_status") or "").strip(),
"proxy_last_refresh_time": str(detect_status.get("proxy_last_refresh_time") or "").strip(),
"proxy_last_refresh_source_count": int(detect_status.get("proxy_last_refresh_source_count", 0) or 0),
"proxy_last_refresh_total_items": int(detect_status.get("proxy_last_refresh_total_items", 0) or 0),
"proxy_last_validated_count": int(detect_status.get("proxy_last_validated_count", 0) or 0),
"proxy_last_available_count": int(detect_status.get("proxy_last_available_count", 0) or 0),
"phase_label": str(detect_status.get("phase_label") or ""),
"phase_detail": str(detect_status.get("phase_detail") or ""),
"updated_at": datetime.now().isoformat(timespec="seconds"),
@@ -402,15 +672,42 @@ def register_local_control_heartbeat() -> None:
)
def _normalize_node_status(raw_status: str, last_heartbeat_at: datetime | None) -> str:
def _resolve_effective_runtime_heartbeat(
*,
metadata: dict | None,
last_heartbeat_at: datetime | None,
update_time: datetime | None,
) -> datetime | None:
effective_last_heartbeat = last_heartbeat_at
runtime_metadata = dict(metadata or {})
if str(runtime_metadata.get("service") or "").strip() != "runtime-ingest":
return effective_last_heartbeat
metadata_updated_at = _parse_runtime_timestamp(runtime_metadata.get("updated_at"))
for candidate in (update_time, metadata_updated_at):
if not candidate:
continue
if effective_last_heartbeat is None or candidate > effective_last_heartbeat:
effective_last_heartbeat = candidate
return effective_last_heartbeat
def _normalize_node_status(raw_status: str, last_heartbeat_at: datetime | None, *, metadata: dict | None = None) -> str:
status = str(raw_status or "").strip() or "unknown"
if not last_heartbeat_at:
return status
now = datetime.now(last_heartbeat_at.tzinfo) if last_heartbeat_at.tzinfo else datetime.now()
age = now - last_heartbeat_at
if age > timedelta(minutes=_OFFLINE_AFTER_MINUTES):
runtime_metadata = dict(metadata or {})
service_name = str(runtime_metadata.get("service") or "").strip()
stale_after = timedelta(seconds=_STALE_AFTER_SECONDS)
offline_after = timedelta(minutes=_OFFLINE_AFTER_MINUTES)
if service_name == "runtime-ingest":
stale_after = timedelta(minutes=_IMPORTED_RUNTIME_STALE_AFTER_MINUTES)
offline_after = timedelta(minutes=_IMPORTED_RUNTIME_OFFLINE_AFTER_MINUTES)
if age > offline_after:
return "offline"
if age > timedelta(seconds=_STALE_AFTER_SECONDS):
if age > stale_after:
return "stale"
return status
@@ -420,11 +717,12 @@ def get_cluster_snapshot() -> dict:
prune_expired_runtime_nodes()
register_local_control_heartbeat()
managed_overlays = _load_managed_node_overlays()
disabled_node_codes = _load_disabled_managed_node_codes()
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT node_code, region, role, hostname, ip, status, worker_version, current_load, metadata_json, last_heartbeat_at
SELECT node_code, region, role, hostname, ip, status, worker_version, current_load, metadata_json, last_heartbeat_at, update_time
FROM detect_worker_nodes
ORDER BY
CASE WHEN status = 'busy' THEN 0 WHEN status = 'online' THEN 1 ELSE 2 END,
@@ -434,14 +732,33 @@ def get_cluster_snapshot() -> dict:
LIMIT 100
"""
)
rows = cur.fetchall()
rows = [
row
for row in list(cur.fetchall() or [])
if str((row or [""])[0] or "").strip() not in disabled_node_codes
]
cur.execute(
"""
SELECT node_code, region, role, hostname, ip, status, worker_version, current_load, metadata_json, last_heartbeat_at, update_time
FROM detect_worker_nodes
ORDER BY
CASE WHEN status = 'busy' THEN 0 WHEN status = 'online' THEN 1 ELSE 2 END,
region ASC,
role ASC,
node_code ASC
"""
)
summary_rows = [
row
for row in list(cur.fetchall() or [])
if str((row or [""])[0] or "").strip() not in disabled_node_codes
]
cur.execute("SELECT count(*) FROM detect_jobs")
jobs_total = cur.fetchone()[0]
cur.execute("SELECT count(*) FROM detect_job_items WHERE status IN ('pending', 'claimed', 'running')")
active_items = cur.fetchone()[0]
nodes = []
for row in rows:
def _build_node_payload(row: tuple) -> dict:
node_code = str(row[0] or "").strip()
metadata = _decode_json(row[8])
current_load = int(row[7] or 0)
@@ -452,14 +769,20 @@ def get_cluster_snapshot() -> dict:
metadata["detect_participating"] = False
metadata["sanitized_runtime_state"] = "idle_phase_zeroed"
runtime_last_heartbeat = row[9]
runtime_update_time = row[10]
managed_overlay = managed_overlays.get(node_code) or {}
managed_last_seen = managed_overlay.get("last_seen_at")
effective_last_heartbeat = _resolve_effective_runtime_heartbeat(
metadata=metadata,
last_heartbeat_at=runtime_last_heartbeat,
update_time=runtime_update_time,
)
overlay_is_newer = bool(
managed_last_seen
and (not runtime_last_heartbeat or managed_last_seen > runtime_last_heartbeat)
and (not effective_last_heartbeat or managed_last_seen > effective_last_heartbeat)
)
effective_last_heartbeat = managed_last_seen if overlay_is_newer else runtime_last_heartbeat
normalized_status = _normalize_node_status(row[5], effective_last_heartbeat)
effective_last_heartbeat = managed_last_seen if overlay_is_newer else effective_last_heartbeat
normalized_status = _normalize_node_status(row[5], effective_last_heartbeat, metadata=metadata)
if sanitized_idle_runtime and normalized_status == "busy":
normalized_status = "online"
if overlay_is_newer and normalized_status in {"offline", "stale"}:
@@ -468,22 +791,23 @@ def get_cluster_snapshot() -> dict:
metadata["agent_last_seen_at"] = managed_last_seen.isoformat(sep=" ", timespec="seconds")
if overlay_is_newer:
metadata["cluster_status_source"] = "managed-agent-overlay"
nodes.append(
{
"node_code": node_code,
"region": row[1],
"role": row[2],
"hostname": row[3],
"ip": row[4],
"status": normalized_status,
"worker_version": row[6],
"current_load": current_load,
"metadata": metadata,
"last_heartbeat_at": effective_last_heartbeat.isoformat(sep=" ", timespec="seconds")
if effective_last_heartbeat
else "",
}
)
return {
"node_code": node_code,
"region": row[1],
"role": row[2],
"hostname": row[3],
"ip": row[4],
"status": normalized_status,
"worker_version": row[6],
"current_load": current_load,
"metadata": metadata,
"last_heartbeat_at": effective_last_heartbeat.isoformat(sep=" ", timespec="seconds")
if effective_last_heartbeat
else "",
}
nodes = [_build_node_payload(row) for row in rows]
summary_nodes = [_build_node_payload(row) for row in summary_rows]
status_counts: dict[str, int] = {}
role_counts: dict[str, int] = {}
region_counts: dict[str, int] = {}
@@ -494,7 +818,7 @@ def get_cluster_snapshot() -> dict:
dedicated_online_worker_nodes = 0
online_control_nodes = 0
for node in nodes:
for node in summary_nodes:
node_status = str(node.get("status") or "unknown")
node_role = str(node.get("role") or "unknown")
node_region = str(node.get("region") or "unknown")
@@ -534,9 +858,25 @@ def get_cluster_snapshot() -> dict:
effective_worker and (metadata.get("detect_participating", False) or node_current_load > 0)
)
summary_node_map = {
str(item.get("node_code") or "").strip(): item
for item in summary_nodes
if str(item.get("node_code") or "").strip()
}
for node in nodes:
summary_node = summary_node_map.get(str(node.get("node_code") or "").strip())
if not summary_node:
continue
node["current_load"] = summary_node.get("current_load", node.get("current_load", 0))
node["status"] = summary_node.get("status", node.get("status", "unknown"))
node["metadata"] = summary_node.get("metadata", node.get("metadata") or {})
node["last_heartbeat_at"] = summary_node.get("last_heartbeat_at", node.get("last_heartbeat_at", ""))
node["is_effective_worker"] = bool(summary_node.get("is_effective_worker", False))
node["detect_participating"] = bool(summary_node.get("detect_participating", False))
return {
"nodes": nodes,
"nodes_total": len(nodes),
"nodes_total": len(summary_nodes),
"jobs_total": jobs_total,
"active_job_items": active_items,
"summary": {

203
domain-api/app/services/dashboard.py
View File

@@ -1,13 +1,18 @@
from __future__ import annotations
from app.core.config import settings
from app.core.db import get_db
from app.services.cluster_runtime_service import get_cluster_snapshot
from app.services.detect_service import get_detect_status
from app.services.detect_job_service import (
_build_step_bucket,
order_step_buckets,
get_active_detect_job_summary,
get_detect_capacity_plan,
get_detect_queue_health,
)
from app.services.runtime_status_service import get_runtime_status
from app.services.runtime_settings_service import get_runtime_settings
from app.services.worker_control_service import detect_worker_runtime
def _empty_active_jobs_aggregate(window_minutes: int) -> dict:
@@ -93,16 +98,7 @@ def _merge_step_queues_with_runtime_activity(
round(processed_recent / safe_window_minutes, 2),
)
return sorted(
step_map.values(),
key=lambda item: (
-int(item.get("items_pending", 0) or 0),
-int(item.get("items_running", 0) or 0),
-int(item.get("started_recent", 0) or 0),
-int(item.get("processed_recent", 0) or 0),
str(item.get("step_code") or ""),
),
)[:normalized_limit]
return order_step_buckets(list(step_map.values()), limit=normalized_limit)
def _align_active_jobs_aggregate_with_runtime(
@@ -169,6 +165,58 @@ def _align_active_jobs_aggregate_with_runtime(
return normalized
def _build_dashboard_runtime_summary(*, queue_health: dict) -> dict:
runtime_settings = get_runtime_settings()
worker_runtime = detect_worker_runtime()
worker_expected_on_this_node = not (
str(settings.node_region or "").strip() == "overseas"
and str(settings.node_role or "").strip() == "control"
)
return {
"node": {
"region": settings.node_region,
"role": settings.node_role,
},
"worker": {
"running": bool(worker_runtime.get("running", False)),
"mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
"expected_on_this_node": worker_expected_on_this_node,
},
"cluster": get_cluster_snapshot(),
"detect": {
"backlog": dict((queue_health or {}).get("runtime_snapshot_backlog") or {}),
},
}
def _resolve_server_code(node_code: str | None) -> str:
normalized_node_code = str(node_code or "").strip()
if not normalized_node_code:
return ""
parent_node_code, separator, suffix = normalized_node_code.rpartition("-")
if separator and parent_node_code and suffix.isalpha() and len(suffix) <= 3:
if any(char.isdigit() for char in parent_node_code):
return parent_node_code
return normalized_node_code
def _count_active_execution_servers(rows: list[dict] | None) -> int:
active_servers: set[str] = set()
for item in list(rows or []):
if not isinstance(item, dict):
continue
server_code = _resolve_server_code(item.get("node_code"))
if not server_code:
continue
if (
int(item.get("items_running", 0) or 0) > 0
or int(item.get("items_claimed", 0) or 0) > 0
or int(item.get("processed_recent", 0) or 0) > 0
):
active_servers.add(server_code)
return len(active_servers)
def _fetch_active_jobs_aggregate(window_minutes: int = 15) -> dict:
safe_window_minutes = max(5, min(int(window_minutes or 15), 120))
payload = _empty_active_jobs_aggregate(safe_window_minutes)
@@ -303,7 +351,7 @@ def _fetch_active_jobs_aggregate(window_minutes: int = 15) -> dict:
}
)
steps.append(bucket)
payload["steps"] = steps
payload["steps"] = order_step_buckets(steps, limit=8)
cur.execute(
"""
@@ -409,8 +457,28 @@ def fetch_overview() -> dict:
active_jobs_aggregate = _fetch_active_jobs_aggregate(window_minutes=window_minutes)
active_job = get_active_detect_job_summary(event_limit=20) or {}
aggregate_queue = active_jobs_aggregate.get("queue") or {}
active_job_display_claimed = int(active_job.get("display_items_claimed", active_job.get("items_claimed", 0)) or 0)
active_job_display_running = int(
active_job.get("display_items_running", active_job.get("display_active_threads", active_job.get("items_running", 0)))
or 0
)
active_job_display_active_threads = int(
active_job.get("display_active_threads", active_job.get("display_items_running", active_job.get("items_running", 0)))
or 0
)
active_job_display_max_threads = int(active_job.get("display_max_threads", 0) or 0)
active_job_distributed_node_stats = [
dict(item)
for item in list(active_job.get("distributed_node_stats") or [])
if isinstance(item, dict)
]
runtime = get_runtime_status()
queue_health = get_detect_queue_health(window_minutes=window_minutes)
runtime = _build_dashboard_runtime_summary(queue_health=queue_health)
try:
detect_status = get_detect_status()
except Exception:
detect_status = {}
cluster_summary = ((runtime.get("cluster") or {}).get("summary") or {})
online_worker_nodes = int(cluster_summary.get("online_worker_nodes", 0) or 0)
dedicated_online_worker_nodes = int(cluster_summary.get("dedicated_online_worker_nodes", 0) or 0)
@@ -425,7 +493,6 @@ def fetch_overview() -> dict:
result["node_region"] = runtime["node"]["region"]
result["node_role"] = runtime["node"]["role"]
queue_health = get_detect_queue_health(window_minutes=window_minutes)
active_jobs_aggregate = _align_active_jobs_aggregate_with_runtime(
active_jobs_aggregate,
runtime=runtime,
@@ -460,6 +527,25 @@ def fetch_overview() -> dict:
throughput_payload = queue_health.get("throughput") or {}
runtime_job_code = str(job_payload.get("runtime_job_code") or "").strip()
display_job_code = runtime_job_code or str(job_payload.get("job_code") or "")
active_job_matches_display_job = display_job_code == str(active_job.get("job_code") or "").strip()
queue_nodes = [
dict(item)
for item in list(queue_health.get("nodes") or [])
if isinstance(item, dict)
]
summary_display_running = max(
int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0),
active_job_display_running if active_job_matches_display_job else 0,
)
summary_display_active_threads = max(
int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0),
active_job_display_active_threads if active_job_matches_display_job else 0,
)
summary_display_max_threads = max(
sum(int(item.get("max_threads", 0) or 0) for item in queue_nodes),
active_job_display_max_threads if active_job_matches_display_job else 0,
)
summary_distributed_node_stats = active_job_distributed_node_stats if active_job_matches_display_job else queue_nodes
active_job_summary = {
"job_id": int(job_payload.get("job_id", 0) or 0),
"job_code": display_job_code,
@@ -469,9 +555,19 @@ def fetch_overview() -> dict:
"progress_percent": float(job_payload.get("progress_percent", 0) or 0),
"items_total": int(queue_payload.get("items_total", 0) or 0),
"items_pending": int(queue_payload.get("pending", 0) or 0),
"items_claimed": int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0),
"items_running": int(queue_payload.get("running", 0) or 0),
"items_display_running": int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0),
"items_claimed": max(
int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0),
active_job_display_claimed if active_job_matches_display_job else 0,
),
"items_running": max(
int(queue_payload.get("running", 0) or 0),
int(active_job.get("items_running", 0) or 0) if active_job_matches_display_job else 0,
),
"items_display_running": summary_display_running,
"display_items_running": summary_display_running,
"display_active_threads": summary_display_active_threads,
"display_max_threads": summary_display_max_threads,
"distributed_node_stats": summary_distributed_node_stats,
"items_completed": int(queue_payload.get("completed", 0) or 0),
"items_blacklisted": int(queue_payload.get("blacklisted", 0) or 0),
"items_failed": int(queue_payload.get("failed", 0) or 0),
@@ -482,6 +578,33 @@ def fetch_overview() -> dict:
"blacklisted_recent": int(throughput_payload.get("blacklisted_recent", 0) or 0),
"active_jobs_total": int(active_jobs_aggregate.get("active_jobs_total", 0) or 0),
}
elif active_job:
active_job_summary = {
"job_id": int(active_job.get("job_id", 0) or 0),
"job_code": str(active_job.get("job_code") or active_job.get("runtime_job_code") or ""),
"db_job_code": str(active_job.get("job_code") or ""),
"runtime_job_code": str(active_job.get("runtime_job_code") or ""),
"status": str(active_job.get("status") or ""),
"progress_percent": float(active_job.get("progress_percent", 0) or 0),
"items_total": int(active_job.get("items_total", 0) or 0),
"items_pending": int(active_job.get("items_pending", 0) or 0),
"items_claimed": active_job_display_claimed,
"items_running": int(active_job.get("items_running", 0) or 0),
"items_display_running": active_job_display_running,
"display_items_running": active_job_display_running,
"display_active_threads": active_job_display_active_threads,
"display_max_threads": active_job_display_max_threads,
"distributed_node_stats": active_job_distributed_node_stats,
"items_completed": int(active_job.get("items_completed", 0) or 0),
"items_blacklisted": int(active_job.get("items_blacklisted", 0) or 0),
"items_failed": int(active_job.get("items_failed", 0) or 0),
"processed_per_minute": float((active_jobs_aggregate.get("throughput") or {}).get("processed_per_minute", 0) or 0),
"processed_recent": int((active_jobs_aggregate.get("throughput") or {}).get("processed_recent", 0) or 0),
"completed_recent": int((active_jobs_aggregate.get("throughput") or {}).get("completed_recent", 0) or 0),
"failed_recent": int((active_jobs_aggregate.get("throughput") or {}).get("failed_recent", 0) or 0),
"blacklisted_recent": int((active_jobs_aggregate.get("throughput") or {}).get("blacklisted_recent", 0) or 0),
"active_jobs_total": int(active_jobs_aggregate.get("active_jobs_total", 0) or 0),
}
queue_pending_total = 0
queue_claimed_total = 0
@@ -514,9 +637,15 @@ def fetch_overview() -> dict:
if queue_health.get("has_active_job"):
queue_payload = queue_health.get("queue") or {}
queue_pending_total = int(queue_payload.get("pending", 0) or 0)
queue_claimed_total = int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0)
queue_running_total = int(queue_payload.get("running", 0) or 0)
queue_display_running_total = int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0)
queue_claimed_total = max(
int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0),
active_job_display_claimed,
)
queue_running_total = max(int(queue_payload.get("running", 0) or 0), int(active_job.get("items_running", 0) or 0))
queue_display_running_total = max(
int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0),
active_job_display_running,
)
queue_completed_total = int(queue_payload.get("completed", 0) or 0)
queue_blacklist_total = int(queue_payload.get("blacklisted", 0) or 0)
queue_failed_total = int(queue_payload.get("failed", 0) or 0)
@@ -558,7 +687,7 @@ def fetch_overview() -> dict:
}
for item in list(active_jobs_aggregate.get("steps") or [])[:8]
]
aggregate_node_throughput = [
aggregate_node_throughput_all = [
{
"node_code": str(item.get("node_code") or ""),
"items_pending": int(item.get("items_pending", 0) or 0),
@@ -574,8 +703,9 @@ def fetch_overview() -> dict:
"failed_recent": int(item.get("failed_recent", 0) or 0),
"blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0),
}
for item in list(active_jobs_aggregate.get("nodes") or [])[:8]
for item in list(active_jobs_aggregate.get("nodes") or [])
]
aggregate_node_throughput = aggregate_node_throughput_all[:8]
queue_step_queue = [
{
"step_code": str(item.get("step_code") or ""),
@@ -600,7 +730,7 @@ def fetch_overview() -> dict:
limit=8,
)
]
queue_node_throughput = [
queue_node_throughput_all = [
{
"node_code": str(item.get("node_code") or ""),
"items_pending": int(item.get("items_pending", 0) or 0),
@@ -616,15 +746,18 @@ def fetch_overview() -> dict:
"failed_recent": int(item.get("failed_recent", 0) or 0),
"blacklisted_recent": int(item.get("blacklisted_recent", 0) or 0),
}
for item in list(queue_health.get("nodes") or [])[:8]
for item in list(queue_health.get("nodes") or [])
]
queue_node_throughput = queue_node_throughput_all[:8]
step_queue = aggregate_step_queue
node_throughput = aggregate_node_throughput
active_execution_node_source = aggregate_node_throughput_all
aggregate_ppm = float((active_jobs_aggregate.get("throughput") or {}).get("processed_per_minute", 0) or 0)
queue_ppm = float((queue_health.get("throughput") or {}).get("processed_per_minute", 0) or 0)
if queue_health.get("has_active_job") or queue_ppm > aggregate_ppm:
step_queue = queue_step_queue
node_throughput = queue_node_throughput
active_execution_node_source = queue_node_throughput_all
if step_queue:
bottleneck_step = max(
step_queue,
@@ -668,13 +801,7 @@ def fetch_overview() -> dict:
"recommended_additional_workers": int(capacity_plan.get("recommended_additional_workers", 0) or 0),
"online_worker_nodes": online_worker_nodes,
"dedicated_online_worker_nodes": dedicated_online_worker_nodes,
"active_execution_nodes": sum(
1
for item in node_throughput
if int(item.get("items_running", 0) or 0) > 0
or int(item.get("items_claimed", 0) or 0) > 0
or int(item.get("processed_recent", 0) or 0) > 0
),
"active_execution_nodes": _count_active_execution_servers(active_execution_node_source),
}
result["processed_per_minute"] = ops_processed_per_minute
result["processed_recent"] = ops_processed_recent
@@ -686,12 +813,26 @@ def fetch_overview() -> dict:
result["queue_claimed_total"] = queue_claimed_total
result["queue_running_total"] = queue_running_total
result["queue_display_running_total"] = max(queue_display_running_total, queue_running_total)
result["queue_display_max_threads"] = max(
int((active_job_summary or {}).get("display_max_threads", 0) or 0),
sum(int(item.get("max_threads", 0) or 0) for item in list(queue_health.get("nodes") or []) if isinstance(item, dict)),
)
result["queue_completed_total"] = queue_completed_total
result["queue_blacklist_total"] = queue_blacklist_total
result["queue_failed_total"] = queue_failed_total
result["current_job_blacklisted"] = queue_blacklist_total
result["recent_blacklisted_total"] = ops_blacklisted_recent
result["cumulative_blacklisted_total"] = int(result.get("blacklist_total", 0) or 0)
result["backlog_pending_total"] = max(backlog_pending_total, queue_pending_total)
result["backlog_claimed_total"] = max(backlog_claimed_total, queue_claimed_total)
result["backlog_running_total"] = max(backlog_running_total, queue_running_total)
result["backlog_register_pending_total"] = backlog_register_pending_total
result["backlog_downstream_pending_total"] = backlog_downstream_pending_total
result["cluster_proxy_available_count"] = int(detect_status.get("available_proxy_count", 0) or 0)
result["cluster_proxy_runtime_label"] = str(detect_status.get("proxy_runtime_label") or "").strip()
result["cluster_proxy_runtime_detail"] = str(detect_status.get("proxy_runtime_detail") or "").strip()
result["cluster_proxy_last_refresh_status"] = str(detect_status.get("proxy_last_refresh_status") or "").strip()
result["aggregate_process_count"] = int(detect_status.get("aggregate_process_count", 0) or 0)
result["aggregate_participating_node_count"] = int(detect_status.get("aggregate_participating_node_count", 0) or 0)
result["aggregate_active_thread_count"] = int(detect_status.get("active_thread_count", 0) or 0)
return result

113
domain-api/app/services/debug_event_service.py
View File

@@ -242,6 +242,105 @@ def _normalize_worker_log_event(debug_event: dict) -> dict | None:
}
def _normalize_debug_event_job_identity(payload: dict | None) -> dict:
normalized_payload = dict(payload or {}) if isinstance(payload, dict) else {}
nested_job = normalized_payload.get("job") if isinstance(normalized_payload.get("job"), dict) else {}
raw_job_id = normalized_payload.get("job_id")
if raw_job_id in (None, "", 0, "0"):
raw_job_id = normalized_payload.get("target_job_id")
if raw_job_id in (None, "", 0, "0"):
raw_job_id = nested_job.get("job_id")
try:
job_id = int(raw_job_id or 0)
except Exception:
job_id = 0
job_code = str(
normalized_payload.get("job_code")
or normalized_payload.get("target_job_code")
or nested_job.get("job_code")
or ""
).strip()
cycle_token = str(normalized_payload.get("cycle_token") or nested_job.get("cycle_token") or "").strip()
return {
"job_id": job_id,
"job_code": job_code,
"cycle_token": cycle_token,
"has_identity": bool(job_id > 0 or job_code),
}
def _load_detect_job_summary_by_job_code(job_code: str, *, event_limit: int = 1) -> dict | None:
normalized_job_code = str(job_code or "").strip()
if not normalized_job_code:
return None
from app.services.detect_job_service import get_detect_job_summary
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id
FROM detect_jobs
WHERE job_code = %s
ORDER BY id DESC
LIMIT 1
""",
(normalized_job_code,),
)
row = cur.fetchone()
if not row:
return None
return get_detect_job_summary(int(row[0]), event_limit=event_limit)
def _resolve_target_job_for_debug_event(event_payload: dict | None) -> tuple[dict | None, str]:
from app.services.detect_job_service import get_active_detect_job_summary, get_detect_job_summary
identity = _normalize_debug_event_job_identity(event_payload)
if not identity["has_identity"]:
return None, "missing_job_identity"
payload_job_id = int(identity["job_id"] or 0)
payload_job_code = str(identity["job_code"] or "").strip()
payload_cycle_token = str(identity["cycle_token"] or "").strip()
active_job = get_active_detect_job_summary(event_limit=1) or {}
active_job_id = int(active_job.get("job_id") or 0)
active_job_code = str(active_job.get("job_code") or active_job.get("runtime_job_code") or "").strip()
target_job: dict | None = None
if payload_job_id > 0:
if active_job_id == payload_job_id:
target_job = active_job
else:
target_job = get_detect_job_summary(payload_job_id, event_limit=1)
elif payload_job_code:
if active_job_code and active_job_code == payload_job_code:
target_job = active_job
else:
target_job = _load_detect_job_summary_by_job_code(payload_job_code, event_limit=1)
if not target_job:
return None, "job_not_found"
target_job_id = int(target_job.get("job_id") or 0)
target_job_code = str(target_job.get("job_code") or target_job.get("runtime_job_code") or "").strip()
target_cycle_token = str(target_job.get("current_cycle_token") or "").strip()
if payload_job_id > 0 and target_job_id > 0 and target_job_id != payload_job_id:
return None, "job_mismatch"
if payload_job_code and target_job_code and target_job_code != payload_job_code:
return None, "job_mismatch"
if payload_cycle_token and target_cycle_token and payload_cycle_token != target_cycle_token:
return None, "cycle_mismatch"
return target_job, "matched"
def _ingest_worker_log_into_active_job(debug_event: dict) -> dict:
if str(debug_event.get("event_type") or "").strip() != "worker_log":
return {"imported": False, "reason": "not_worker_log"}
@@ -249,15 +348,16 @@ def _ingest_worker_log_into_active_job(debug_event: dict) -> dict:
normalized_event = _normalize_worker_log_event(debug_event)
if not normalized_event:
return {"imported": False, "reason": "not_domain_progress_event"}
from app.services.detect_job_service import get_active_detect_job_summary
from app.services.sync_push_service import (
_apply_detect_result_event_to_domain,
_apply_detect_result_event_to_job_item,
)
active_job = get_active_detect_job_summary(event_limit=1) or {}
target_job_id = int(active_job.get("job_id") or 0)
target_job, resolve_reason = _resolve_target_job_for_debug_event(normalized_event.get("payload"))
if not target_job:
return {"imported": False, "reason": resolve_reason}
target_job_id = int(target_job.get("job_id") or 0)
if target_job_id <= 0:
return {"imported": False, "reason": "no_active_job"}
@@ -334,6 +434,7 @@ def _ingest_worker_log_into_active_job(debug_event: dict) -> dict:
"imported": True,
"reason": "imported",
"target_job_id": target_job_id,
"target_job_code": str(target_job.get("job_code") or ""),
"detect_run_event_id": detect_run_event_id,
"updated_job_items": updated_job_items,
"event_type": normalized_event["event_type"],
@@ -847,7 +948,9 @@ def get_debug_handoff_report(
def ingest_debug_event(payload: dict, *, shared_token: str | None = None) -> tuple[bool, str, dict]:
configured_token = str(settings.sync_shared_token or "").strip()
incoming_token = str(shared_token or "").strip()
if configured_token and incoming_token != configured_token:
if not configured_token:
return False, "调试事件共享 token 未配置，拒绝远端写入", {"configuration_required": True}
if incoming_token != configured_token:
return False, "调试事件 token 校验失败", {}
record_id = append_debug_event(

1947
domain-api/app/services/detect_job_service.py
View File

File diff suppressed because it is too large Load Diff

338
domain-api/app/services/detect_run_service.py
View File

@@ -1,5 +1,6 @@
from __future__ import annotations
import threading
from datetime import datetime
from uuid import uuid4
@@ -10,6 +11,7 @@ _MAX_LOG_LINES = 240
_LOG_TAIL_LINES = 1200
_ACTIVE_STATUSES = {"starting", "running", "stopping"}
_TIMESTAMP_FORMATS = ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S")
_DETECT_RUNS_LOCK = threading.RLock()
def _now() -> str:
@@ -234,192 +236,196 @@ def _sync_record(
def create_detect_run_snapshot(message: str, runtime: dict, progress: dict, settings_summary: dict) -> dict:
records = _load()
active = _find_active(records)
current_logs = _capture_worker_logs()
if active:
if active.get("status") == "stopping" and runtime.get("running"):
active["status"] = "running"
with _DETECT_RUNS_LOCK:
records = _load()
active = _find_active(records)
current_logs = _capture_worker_logs()
if active:
if active.get("status") == "stopping" and runtime.get("running"):
active["status"] = "running"
_sync_record(
active,
status=active.get("status", "starting"),
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
)
_save(records)
return dict(active)
initial_started_at = runtime.get("latest_start_time") or _now()
record = {
"run_id": uuid4().hex,
"status": "starting",
"message": message,
"created_at": _now(),
"updated_at": _now(),
"started_at": initial_started_at,
"completed_at": "",
"runtime": runtime,
"progress": progress,
"settings_summary": settings_summary,
"phase_label": "",
"phase_detail": "",
"phase_history": [],
"logs": [],
}
_sync_record(
active,
status=active.get("status", "starting"),
record,
status="starting",
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
)
records.insert(0, record)
_save(records)
return dict(active)
initial_started_at = runtime.get("latest_start_time") or _now()
record = {
"run_id": uuid4().hex,
"status": "starting",
"message": message,
"created_at": _now(),
"updated_at": _now(),
"started_at": initial_started_at,
"completed_at": "",
"runtime": runtime,
"progress": progress,
"settings_summary": settings_summary,
"phase_label": "",
"phase_detail": "",
"phase_history": [],
"logs": [],
}
_sync_record(
record,
status="starting",
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
)
records.insert(0, record)
_save(records)
return dict(record)
return dict(record)
def finalize_detect_run(message: str, runtime: dict, progress: dict, settings_summary: dict, active_job: dict | None = None) -> dict | None:
records = _load()
target = _find_active(records)
if not target:
return None
final_status = "stopped" if target.get("status") == "stopping" else "failed"
_sync_record(
target,
status=final_status,
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=_capture_worker_logs(),
active_job=active_job,
)
_save(records)
return dict(target)
def sync_detect_runs(runtime: dict, progress: dict, settings_summary: dict, active_job: dict | None = None) -> list[dict]:
records = _load()
active = _find_active(records)
current_logs = _capture_worker_logs()
active_job = active_job or {}
runtime_detecting = bool(runtime.get("detecting", False))
active_job_status = str(active_job.get("status", "") or "").strip()
active_job_open = active_job_status in {"pending", "running"}
execution_active = runtime_detecting or active_job_open or int((progress or {}).get("running", 0) or 0) > 0
if runtime.get("running") and execution_active:
if active and not _same_session(active, runtime):
_sync_record(
active,
status="stopped",
message="检测服务已重启，上一轮会话已归档",
runtime=active.get("runtime") or runtime,
progress=active.get("progress") or progress,
settings_summary=active.get("settings_summary") or settings_summary,
log_lines=current_logs,
active_job=active.get("active_job") or active_job,
)
active = None
if active:
next_status = "running" if active.get("status") != "stopping" else "stopping"
_sync_record(
active,
status=next_status,
message=runtime.get("message") or active.get("message") or "检测服务运行中",
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
active_job=active_job,
)
else:
started_at = runtime.get("latest_start_time") or _now()
record = {
"run_id": uuid4().hex,
"status": "running",
"message": runtime.get("message") or "检测服务运行中",
"created_at": _now(),
"updated_at": _now(),
"started_at": started_at,
"completed_at": "",
"runtime": runtime,
"progress": progress,
"settings_summary": settings_summary,
"phase_label": "",
"phase_detail": "",
"phase_history": [],
"logs": [],
"active_job": active_job,
}
_sync_record(
record,
status="running",
message=record["message"],
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
active_job=active_job,
)
records.insert(0, record)
elif active:
if runtime.get("running") and not execution_active:
if active.get("status") == "stopping":
final_status = "stopped"
final_message = runtime.get("message") or "检测任务已停止，Worker 保持待命"
elif active_job_status == "partial_failed":
final_status = "partial_failed"
final_message = "检测任务已结束，存在部分失败项"
elif active_job_status == "failed":
final_status = "failed"
final_message = "检测任务已结束，任务结果为失败"
else:
final_status = "completed"
final_message = "检测任务已自然完成，Worker 保持待命"
else:
final_status = "stopped" if active.get("status") == "stopping" else "failed"
final_message = runtime.get("message") or ("检测服务已停止" if final_status == "stopped" else "检测服务异常退出")
with _DETECT_RUNS_LOCK:
records = _load()
target = _find_active(records)
if not target:
return None
final_status = "stopped" if target.get("status") == "stopping" else "failed"
_sync_record(
active,
target,
status=final_status,
message=final_message,
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
log_lines=_capture_worker_logs(),
active_job=active_job,
)
_save(records)
return dict(target)
if records:
records[0]["logs"] = _merge_logs(
records[0].get("logs"),
_filter_logs_since(current_logs, records[0].get("started_at")),
)
_save(records)
return records
def sync_detect_runs(runtime: dict, progress: dict, settings_summary: dict, active_job: dict | None = None) -> list[dict]:
with _DETECT_RUNS_LOCK:
records = _load()
active = _find_active(records)
current_logs = _capture_worker_logs()
active_job = active_job or {}
runtime_detecting = bool(runtime.get("detecting", False))
active_job_status = str(active_job.get("status", "") or "").strip()
active_job_open = active_job_status in {"pending", "running"}
execution_active = runtime_detecting or active_job_open or int((progress or {}).get("running", 0) or 0) > 0
if runtime.get("running") and execution_active:
if active and not _same_session(active, runtime):
_sync_record(
active,
status="stopped",
message="检测服务已重启，上一轮会话已归档",
runtime=active.get("runtime") or runtime,
progress=active.get("progress") or progress,
settings_summary=active.get("settings_summary") or settings_summary,
log_lines=current_logs,
active_job=active.get("active_job") or active_job,
)
active = None
if active:
next_status = "running" if active.get("status") != "stopping" else "stopping"
_sync_record(
active,
status=next_status,
message=runtime.get("message") or active.get("message") or "检测服务运行中",
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
active_job=active_job,
)
else:
started_at = runtime.get("latest_start_time") or _now()
record = {
"run_id": uuid4().hex,
"status": "running",
"message": runtime.get("message") or "检测服务运行中",
"created_at": _now(),
"updated_at": _now(),
"started_at": started_at,
"completed_at": "",
"runtime": runtime,
"progress": progress,
"settings_summary": settings_summary,
"phase_label": "",
"phase_detail": "",
"phase_history": [],
"logs": [],
"active_job": active_job,
}
_sync_record(
record,
status="running",
message=record["message"],
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
active_job=active_job,
)
records.insert(0, record)
elif active:
if runtime.get("running") and not execution_active:
if active.get("status") == "stopping":
final_status = "stopped"
final_message = runtime.get("message") or "检测任务已停止，Worker 保持待命"
elif active_job_status == "partial_failed":
final_status = "partial_failed"
final_message = "检测任务已结束，存在部分失败项"
elif active_job_status == "failed":
final_status = "failed"
final_message = "检测任务已结束，任务结果为失败"
else:
final_status = "completed"
final_message = "检测任务已自然完成，Worker 保持待命"
else:
final_status = "stopped" if active.get("status") == "stopping" else "failed"
final_message = runtime.get("message") or ("检测服务已停止" if final_status == "stopped" else "检测服务异常退出")
_sync_record(
active,
status=final_status,
message=final_message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
active_job=active_job,
)
if records:
records[0]["logs"] = _merge_logs(
records[0].get("logs"),
_filter_logs_since(current_logs, records[0].get("started_at")),
)
_save(records)
return records
def mark_detect_run_stopping(message: str, runtime: dict, progress: dict, settings_summary: dict) -> dict | None:
records = _load()
target = _find_active(records)
if not target:
return None
_sync_record(
target,
status="stopping",
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=_capture_worker_logs(),
active_job=target.get("active_job") or {},
)
_save(records)
return dict(target)
with _DETECT_RUNS_LOCK:
records = _load()
target = _find_active(records)
if not target:
return None
_sync_record(
target,
status="stopping",
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=_capture_worker_logs(),
active_job=target.get("active_job") or {},
)
_save(records)
return dict(target)

878
domain-api/app/services/detect_service.py
View File

@@ -3,6 +3,8 @@ from __future__ import annotations
import json
import re
import subprocess
import threading
import time
from datetime import datetime, timedelta, timezone
from app.core.config import settings
from app.core.db import get_db
@@ -12,13 +14,16 @@ from app.services.debug_event_service import list_debug_events
from app.services.cluster_runtime_service import ensure_runtime_schema
from app.services.runtime_settings_service import get_runtime_settings
from app.services.detect_run_service import sync_detect_runs
from app.services.detect_job_service import get_active_detect_job_summary
from app.services.settings_service import get_settings_payload, resolve_thread_count
from app.services.detect_job_service import get_active_detect_job_summary, get_detect_queue_health
from app.services.settings_service import get_settings_payload, resolve_process_count, resolve_thread_count
from app.services.sync_record_service import append_detect_result_projection_if_changed
from app.services.worker_control_service import detect_worker_runtime
_PROXY_COUNT_RE = re.compile(r"当前可用代理数[:：]\s*(\d+)")
_PROXY_REFRESH_COUNT_RE = re.compile(r"代理池刷新完成，共\s*(\d+)\s*个可用代理")
_PROXY_CACHE_COUNT_RE = re.compile(r"继续沿用缓存\s*(\d+)\s*个")
_PROXY_SHARED_SNAPSHOT_COUNT_RE = re.compile(r"(?:复用共享代理快照|共享代理快照)\s*(\d+)\s*个")
_THREAD_COUNT_RE = re.compile(r"当前实际线程数量[:：]\s*(\d+)\s*/\s*(\d+)")
_STEP_TRACE_DOMAIN_RE = re.compile(r"domain=([^\s|]+)")
_REGISTER_DOMAIN_RE = re.compile(r"检测注册状态[:：]\s*([^\s]+)")
@@ -38,6 +43,89 @@ _REMOTE_DEBUG_EVENT_TYPES = {
"task_pull_failed",
"queue_overdue_leases",
}
_DETECT_STATUS_CACHE_LOCK = threading.Lock()
_DETECT_STATUS_CACHE_TTL_SECONDS = 3.0
_DETECT_STATUS_CACHE_VALUE: dict | None = None
_DETECT_STATUS_CACHE_EXPIRES_AT = 0.0
_AGGREGATE_RUNTIME_NODE_STALE_AFTER = timedelta(seconds=90)
_DISABLED_MANAGED_NODE_CACHE: set[str] = set()
_DISABLED_MANAGED_NODE_CACHE_EXPIRES_AT = 0.0
def _clone_detect_status_payload(value: dict | None) -> dict:
try:
return json.loads(json.dumps(dict(value or {}), ensure_ascii=False))
except Exception:
return dict(value or {})
def _extract_debug_event_job_identity(payload: dict | None) -> dict:
normalized_payload = dict(payload or {}) if isinstance(payload, dict) else {}
nested_job = normalized_payload.get("job") if isinstance(normalized_payload.get("job"), dict) else {}
raw_job_id = normalized_payload.get("job_id")
if raw_job_id in (None, "", 0, "0"):
raw_job_id = normalized_payload.get("target_job_id")
if raw_job_id in (None, "", 0, "0"):
raw_job_id = nested_job.get("job_id")
try:
job_id = int(raw_job_id or 0)
except Exception:
job_id = 0
return {
"job_id": job_id,
"job_code": str(
normalized_payload.get("job_code")
or normalized_payload.get("target_job_code")
or nested_job.get("job_code")
or ""
).strip(),
"cycle_token": str(normalized_payload.get("cycle_token") or nested_job.get("cycle_token") or "").strip(),
"has_identity": bool(job_id > 0 or str(
normalized_payload.get("job_code")
or normalized_payload.get("target_job_code")
or nested_job.get("job_code")
or ""
).strip()),
}
def _debug_event_matches_active_job(record: dict, active_job: dict | None) -> bool:
normalized_active_job = dict(active_job or {})
active_job_id = int(normalized_active_job.get("job_id") or 0)
active_job_code = str(
normalized_active_job.get("runtime_job_code")
or normalized_active_job.get("job_code")
or ""
).strip()
active_cycle_token = str(normalized_active_job.get("current_cycle_token") or "").strip()
if active_job_id <= 0 and not active_job_code and not active_cycle_token:
return True
identity = _extract_debug_event_job_identity(record.get("payload"))
if not identity["has_identity"]:
return False
event_job_id = int(identity["job_id"] or 0)
event_job_code = str(identity["job_code"] or "").strip()
event_cycle_token = str(identity["cycle_token"] or "").strip()
if active_cycle_token and event_cycle_token and event_cycle_token != active_cycle_token:
return False
if active_job_id > 0 and event_job_id > 0 and event_job_id != active_job_id:
return False
if active_job_code and event_job_code and event_job_code != active_job_code:
return False
if active_job_id > 0 and event_job_id == active_job_id:
return True
if active_job_code and event_job_code and event_job_code == active_job_code:
return True
if active_cycle_token and event_cycle_token and event_cycle_token == active_cycle_token:
return True
return False
def _extract_remote_log_node_code(line: str) -> str:
@@ -95,6 +183,225 @@ def _runtime_state_key(node_code: str | None = None) -> str:
return f"{_RUNTIME_STATE_KEY}:{normalized_node_code}"
def _local_worker_expected_on_this_node() -> bool:
return not (
str(settings.node_region or "").strip() == "overseas"
and str(settings.node_role or "").strip() == "control"
)
def _int_value(value: object) -> int:
try:
return int(value or 0)
except Exception:
return 0
def _max_runtime_metric(*values: object) -> int:
return max((_int_value(value) for value in values), default=0)
def _resolve_capacity_node_code(node_code: str, settings_payload: dict) -> tuple[str, bool]:
normalized_node_code = str(node_code or "").strip()
if not normalized_node_code:
return "", False
node_thread_counts = dict(settings_payload.get("node_thread_counts") or {})
node_process_counts = dict(settings_payload.get("node_process_counts") or {})
parent_node_code, separator, suffix = normalized_node_code.rpartition("-")
if (
separator
and parent_node_code
and suffix.isalpha()
and len(suffix) <= 3
and (
parent_node_code in node_thread_counts
or parent_node_code in node_process_counts
or bool(re.search(r"\d$", parent_node_code))
)
):
return parent_node_code, True
if normalized_node_code in node_thread_counts or normalized_node_code in node_process_counts:
return normalized_node_code, False
return normalized_node_code, False
def _is_current_participant_bucket(item: dict | None) -> bool:
payload = dict(item or {})
node_code = str(payload.get("node_code") or "").strip()
if not node_code or node_code == "unassigned":
return False
return any(
_int_value(payload.get(field)) > 0
for field in ("items_claimed", "items_running", "display_running", "current_load", "active_threads")
)
def _aggregate_runtime_node_is_live(item: dict | None) -> bool:
payload = dict(item or {})
node_code = str(payload.get("node_code") or "").strip()
if not node_code:
return False
if node_code == "unassigned":
return True
status = str(payload.get("status") or "").strip().lower()
if status in {"stale", "offline"}:
return False
last_heartbeat_at = _parse_time(payload.get("last_heartbeat_at"))
if last_heartbeat_at is None:
return True
reference_now = datetime.now(last_heartbeat_at.tzinfo) if last_heartbeat_at.tzinfo else datetime.now()
return (reference_now - last_heartbeat_at) <= _AGGREGATE_RUNTIME_NODE_STALE_AFTER
def _filter_live_aggregate_runtime_nodes(node_rows: list[dict] | None) -> list[dict]:
disabled_node_codes = _load_disabled_managed_node_codes(
[
str(item.get("node_code") or "").strip()
for item in list(node_rows or [])
if isinstance(item, dict)
]
)
return [
dict(item)
for item in list(node_rows or [])
if isinstance(item, dict)
and str(item.get("node_code") or "").strip() not in disabled_node_codes
and _aggregate_runtime_node_is_live(item)
]
def _build_aggregate_detect_capacity(*, active_job: dict | None, settings_payload: dict) -> dict:
node_rows = _filter_live_aggregate_runtime_nodes(
list((active_job or {}).get("distributed_node_stats") or (active_job or {}).get("node_stats") or [])
)
node_thread_counts = dict(settings_payload.get("node_thread_counts") or {})
node_process_counts = dict(settings_payload.get("node_process_counts") or {})
participant_node_codes: list[str] = []
process_count_total = 0
max_threads_total = 0
representative_thread_count = 0
child_parent_codes: set[str] = set()
for raw_item in node_rows:
if not isinstance(raw_item, dict) or not _is_current_participant_bucket(raw_item):
continue
node_code = str(raw_item.get("node_code") or "").strip()
capacity_node_code, is_child_instance = _resolve_capacity_node_code(node_code, settings_payload)
if is_child_instance and capacity_node_code:
child_parent_codes.add(capacity_node_code)
for raw_item in node_rows:
if not isinstance(raw_item, dict) or not _is_current_participant_bucket(raw_item):
continue
node_code = str(raw_item.get("node_code") or "").strip()
capacity_node_code, is_child_instance = _resolve_capacity_node_code(node_code, settings_payload)
if not capacity_node_code:
continue
if not is_child_instance and node_code in child_parent_codes:
continue
if node_code not in participant_node_codes:
participant_node_codes.append(node_code)
thread_resolution_node_code = node_code if node_code in node_thread_counts else capacity_node_code
thread_resolution = resolve_thread_count(node_code=thread_resolution_node_code, settings_payload=settings_payload)
per_process_thread_count = max(1, int(thread_resolution["effective_thread_count"] or 1))
if representative_thread_count <= 0:
representative_thread_count = per_process_thread_count
if is_child_instance:
process_count = 1
else:
if node_code in node_process_counts:
process_resolution = resolve_process_count(node_code=node_code, settings_payload=settings_payload)
process_count = max(1, int(process_resolution["effective_process_count"] or 1))
else:
process_count = 1
process_count_total += process_count
max_threads_total += process_count * per_process_thread_count
return {
"participant_node_codes": participant_node_codes,
"participant_node_count": len(participant_node_codes),
"process_count": process_count_total,
"max_threads": max_threads_total,
"per_process_thread_count": representative_thread_count,
}
def _merge_aggregate_active_job_with_queue_health(active_job: dict | None, queue_health: dict | None) -> dict | None:
normalized_active_job = dict(active_job or {})
normalized_queue_health = dict(queue_health or {})
if not normalized_queue_health.get("has_active_job"):
return normalized_active_job or active_job
queue_payload = dict(normalized_queue_health.get("queue") or {})
queue_job = dict(normalized_queue_health.get("job") or {})
raw_queue_nodes = [dict(item) for item in list(normalized_queue_health.get("nodes") or []) if isinstance(item, dict)]
queue_nodes = _filter_live_aggregate_runtime_nodes(raw_queue_nodes)
if not queue_payload and not queue_nodes and not queue_job:
return normalized_active_job or active_job
queue_display_running = sum(
_max_runtime_metric(
item.get("display_running"),
item.get("current_load"),
item.get("active_threads"),
item.get("items_running"),
)
for item in queue_nodes
if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned"
)
if raw_queue_nodes:
display_max_threads = sum(_int_value(item.get("max_threads")) for item in queue_nodes)
display_items_running = queue_display_running
display_active_threads = queue_display_running
else:
display_max_threads = _max_runtime_metric(
sum(_int_value(item.get("max_threads")) for item in queue_nodes),
normalized_active_job.get("display_max_threads"),
)
display_items_running = _max_runtime_metric(
queue_display_running,
queue_payload.get("display_running"),
normalized_active_job.get("display_items_running"),
normalized_active_job.get("display_active_threads"),
)
display_active_threads = _max_runtime_metric(
queue_display_running,
queue_payload.get("display_running"),
normalized_active_job.get("display_active_threads"),
normalized_active_job.get("display_items_running"),
)
merged = dict(normalized_active_job)
merged.update(
{
"job_id": queue_job.get("job_id", merged.get("job_id")),
"job_code": queue_job.get("job_code", merged.get("job_code")),
"status": queue_job.get("status", merged.get("status")),
"progress_percent": queue_job.get("progress_percent", merged.get("progress_percent", 0)),
"items_total": _int_value(queue_payload.get("items_total", merged.get("items_total"))),
"items_pending": _int_value(queue_payload.get("pending", merged.get("items_pending"))),
"items_claimed": _int_value(queue_payload.get("claimed", merged.get("items_claimed"))),
"items_running": _int_value(queue_payload.get("running", merged.get("items_running"))),
"items_completed": _int_value(queue_payload.get("completed", merged.get("items_completed"))),
"items_blacklisted": _int_value(queue_payload.get("blacklisted", merged.get("items_blacklisted"))),
"items_failed": _int_value(queue_payload.get("failed", merged.get("items_failed"))),
"display_items_running": display_items_running,
"display_active_threads": display_active_threads,
"display_max_threads": display_max_threads,
}
)
if raw_queue_nodes:
merged["node_stats"] = list(queue_nodes)
merged["distributed_node_stats"] = list(queue_nodes)
return merged
def _extract_dependency_alerts(lines: list[str]) -> list[dict]:
alerts: list[dict] = []
recent_lines = lines[-120:] if lines else []
@@ -140,9 +447,29 @@ def _extract_dependency_alerts(lines: list[str]) -> list[dict]:
def _extract_available_proxy_count(lines: list[str]) -> int:
for line in reversed(lines):
match = _PROXY_COUNT_RE.search(line)
if match:
return int(match.group(1))
count = _extract_available_proxy_count_from_text(line)
if count > 0:
return count
return 0
def _extract_available_proxy_count_from_text(text: str) -> int:
normalized_text = str(text or "").strip()
if not normalized_text:
return 0
for pattern in (
_PROXY_COUNT_RE,
_PROXY_REFRESH_COUNT_RE,
_PROXY_CACHE_COUNT_RE,
_PROXY_SHARED_SNAPSHOT_COUNT_RE,
):
match = pattern.search(normalized_text)
if not match:
continue
try:
return int(match.group(1) or 0)
except Exception:
continue
return 0
@@ -492,6 +819,8 @@ def _build_remote_log_snapshot_from_debug_events(
node_code = str(record.get("node_code") or "").strip() or "unknown"
if participating_node_codes and node_code not in participating_node_codes:
continue
if not _debug_event_matches_active_job(record, active_job):
continue
message = str(record.get("message") or "").strip()
if not message:
continue
@@ -665,6 +994,55 @@ def _load_runtime_state() -> dict:
return {}
def _load_disabled_managed_node_codes(node_codes: list[str] | None = None) -> set[str]:
global _DISABLED_MANAGED_NODE_CACHE, _DISABLED_MANAGED_NODE_CACHE_EXPIRES_AT
normalized_codes = [
str(item or "").strip()
for item in list(node_codes or [])
if str(item or "").strip()
]
now_ts = time.time()
if not normalized_codes and now_ts < _DISABLED_MANAGED_NODE_CACHE_EXPIRES_AT:
return set(_DISABLED_MANAGED_NODE_CACHE)
try:
with get_db() as conn:
with conn.cursor() as cur:
if normalized_codes:
cur.execute(
"""
SELECT node_code
FROM ops_managed_nodes
WHERE is_enabled = FALSE
AND node_code = ANY(%s)
""",
(normalized_codes,),
)
else:
cur.execute(
"""
SELECT node_code
FROM ops_managed_nodes
WHERE is_enabled = FALSE
"""
)
rows = list(cur.fetchall() or [])
except Exception:
if normalized_codes:
return {code for code in normalized_codes if code in _DISABLED_MANAGED_NODE_CACHE}
return set(_DISABLED_MANAGED_NODE_CACHE)
disabled_codes = {
str(row[0] or "").strip()
for row in rows
if str(row[0] or "").strip()
}
if normalized_codes:
return disabled_codes
_DISABLED_MANAGED_NODE_CACHE = disabled_codes
_DISABLED_MANAGED_NODE_CACHE_EXPIRES_AT = now_ts + 5.0
return set(disabled_codes)
def _load_runtime_state_from_cluster_node() -> dict:
try:
with get_db() as conn:
@@ -706,6 +1084,359 @@ def _load_runtime_state_from_cluster_node() -> dict:
return {}
def _load_runtime_states_from_cluster_nodes(node_codes: list[str] | tuple[str, ...]) -> dict[str, dict]:
normalized_node_codes = [
str(item or "").strip()
for item in list(node_codes or [])
if str(item or "").strip()
]
if not normalized_node_codes:
return {}
try:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT node_code, current_load, metadata_json, last_heartbeat_at
FROM detect_worker_nodes
WHERE node_code = ANY(%s)
""",
(normalized_node_codes,),
)
rows = list(cur.fetchall() or [])
except Exception:
return {}
payload: dict[str, dict] = {}
for row in rows:
node_code = str(row[0] or "").strip()
if not node_code:
continue
current_load = int(row[1] or 0)
metadata_json = row[2]
last_heartbeat_at = row[3]
metadata = metadata_json if isinstance(metadata_json, dict) else {}
payload[node_code] = {
"node_code": node_code,
"current_load": current_load,
"last_heartbeat_at": (
last_heartbeat_at.isoformat(sep=" ", timespec="seconds")
if hasattr(last_heartbeat_at, "isoformat")
else str(last_heartbeat_at or "").strip()
),
"available_proxy_count": int(
metadata.get("available_proxy_count", metadata.get("proxy_last_available_count", 0)) or 0
),
"proxy_runtime_label": str(metadata.get("proxy_runtime_label") or "").strip(),
"proxy_runtime_reason": str(metadata.get("proxy_runtime_reason") or "").strip(),
"proxy_last_refresh_status": str(metadata.get("proxy_last_refresh_status") or "").strip(),
"proxy_last_refresh_time": str(metadata.get("proxy_last_refresh_time") or "").strip(),
"proxy_last_refresh_source_count": int(metadata.get("proxy_last_refresh_source_count", 0) or 0),
"proxy_last_refresh_total_items": int(metadata.get("proxy_last_refresh_total_items", 0) or 0),
"proxy_last_validated_count": int(metadata.get("proxy_last_validated_count", 0) or 0),
"active_threads": int(metadata.get("active_threads", 0) or 0),
"max_threads": int(metadata.get("max_threads", 0) or 0),
"detect_participating": bool(metadata.get("detect_participating", False) or current_load > 0),
}
return payload
def _load_recent_proxy_debug_events(node_codes: list[str] | tuple[str, ...], *, window_minutes: int = 20) -> list[dict]:
normalized_node_codes = [
str(item or "").strip()
for item in list(node_codes or [])
if str(item or "").strip() and str(item or "").strip() != "unassigned"
]
if not normalized_node_codes:
return []
safe_window_minutes = max(5, min(int(window_minutes or 20), 120))
created_after = datetime.now() - timedelta(minutes=safe_window_minutes)
safe_limit = max(80, min(len(normalized_node_codes) * 20, 800))
try:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT node_code, message, payload_json, created_at
FROM detect_debug_events
WHERE event_type = 'worker_log'
AND node_code = ANY(%s)
AND created_at >= %s
ORDER BY created_at DESC, id DESC
LIMIT %s
""",
(normalized_node_codes, created_after, safe_limit),
)
rows = list(cur.fetchall() or [])
except Exception:
return []
return [
{
"event_type": "worker_log",
"node_code": str(row[0] or "").strip(),
"message": str(row[1] or "").strip(),
"payload": row[2] if isinstance(row[2], dict) else {},
"created_at": (
row[3].isoformat(sep=" ", timespec="seconds")
if hasattr(row[3], "isoformat")
else str(row[3] or "").strip()
),
}
for row in rows
if str(row[0] or "").strip() and str(row[1] or "").strip()
]
def _is_proxy_runtime_message(message: str) -> bool:
normalized_message = str(message or "").strip()
if not normalized_message:
return False
if _extract_available_proxy_count_from_text(normalized_message) > 0:
return True
lowered_message = normalized_message.lower()
return any(
keyword in normalized_message or keyword in lowered_message
for keyword in (
"代理",
"proxy",
"cooldown",
"rate limited",
)
)
def _infer_proxy_runtime_label_from_message(message: str, *, available_proxy_count: int) -> tuple[str, str]:
normalized_message = str(message or "").strip()
lowered_message = normalized_message.lower()
if available_proxy_count > 0:
return "代理正常", "aggregate_log_healthy"
if "rate limited" in lowered_message or "cooldown" in lowered_message or "冷却" in normalized_message:
return "代理源暂时冷却中", "aggregate_log_cooldown"
if any(keyword in normalized_message for keyword in ("未取到新代理", "未取到可用代理数据", "无可用代理", "未返回可用代理数据")):
return "代理待补货", "aggregate_log_empty"
if "等待首刷" in normalized_message:
return "等待首刷", "aggregate_log_waiting"
return "", ""
def _build_aggregate_proxy_runtime_rows_from_events(
*,
active_job: dict | None,
settings_payload: dict,
participant_node_codes: list[str],
participant_server_codes: list[str],
) -> dict[str, dict]:
normalized_active_job = dict(active_job or {})
events = list(normalized_active_job.get("current_cycle_events") or normalized_active_job.get("recent_events") or [])
events.extend(_load_recent_proxy_debug_events(participant_node_codes))
if not events:
return {}
current_cycle_token = str(normalized_active_job.get("current_cycle_token") or "").strip()
allowed_server_codes = {str(item or "").strip() for item in list(participant_server_codes or []) if str(item or "").strip()}
rows: dict[str, dict] = {}
for event in events:
if not isinstance(event, dict):
continue
if str(event.get("event_type") or "").strip() != "worker_log":
continue
payload = event.get("payload") if isinstance(event.get("payload"), dict) else {}
event_cycle_token = str(payload.get("cycle_token") or "").strip()
if current_cycle_token and event_cycle_token and event_cycle_token != current_cycle_token:
continue
node_code = str(event.get("node_code") or "").strip()
if not node_code or node_code == "unassigned":
continue
capacity_node_code, _ = _resolve_capacity_node_code(node_code, settings_payload)
server_code = str(capacity_node_code or node_code or "").strip()
if not server_code or (allowed_server_codes and server_code not in allowed_server_codes):
continue
message = str(event.get("message") or "").strip()
if not _is_proxy_runtime_message(message):
continue
created_at = str(event.get("created_at") or "").strip()
available_proxy_count = _extract_available_proxy_count_from_text(message)
label, reason = _infer_proxy_runtime_label_from_message(
message,
available_proxy_count=available_proxy_count,
)
row = rows.setdefault(
server_code,
{
"node_code": server_code,
"available_proxy_count": 0,
"proxy_runtime_label": "",
"proxy_runtime_reason": "",
"proxy_last_refresh_status": "",
"proxy_last_refresh_time": "",
"proxy_last_refresh_source_count": 0,
"proxy_last_refresh_total_items": 0,
"proxy_last_validated_count": 0,
"_count_seen_at": "",
"_status_seen_at": "",
},
)
if available_proxy_count > 0 and created_at >= str(row.get("_count_seen_at") or ""):
row["available_proxy_count"] = int(available_proxy_count or 0)
row["_count_seen_at"] = created_at
if created_at >= str(row.get("_status_seen_at") or ""):
row["proxy_last_refresh_status"] = message
row["proxy_last_refresh_time"] = created_at
row["_status_seen_at"] = created_at
if label:
row["proxy_runtime_label"] = label
if reason:
row["proxy_runtime_reason"] = reason
return {
server_code: {
key: value
for key, value in row.items()
if not str(key).startswith("_")
}
for server_code, row in rows.items()
}
def _build_aggregate_proxy_runtime(
*,
active_job: dict | None,
settings_payload: dict,
fallback_available_proxy_count: int,
fallback_proxy_runtime: dict,
) -> tuple[int, dict]:
node_rows = list((active_job or {}).get("distributed_node_stats") or (active_job or {}).get("node_stats") or [])
participant_node_codes: list[str] = []
participant_server_codes: list[str] = []
for raw_item in node_rows:
if not isinstance(raw_item, dict) or not _is_current_participant_bucket(raw_item):
continue
node_code = str(raw_item.get("node_code") or "").strip()
if node_code and node_code != "unassigned" and node_code not in participant_node_codes:
participant_node_codes.append(node_code)
capacity_node_code, _ = _resolve_capacity_node_code(node_code, settings_payload)
normalized_server_code = str(capacity_node_code or node_code or "").strip()
if not normalized_server_code or normalized_server_code == "unassigned":
continue
if normalized_server_code not in participant_server_codes:
participant_server_codes.append(normalized_server_code)
if not participant_server_codes:
return fallback_available_proxy_count, fallback_proxy_runtime
event_runtime = _build_aggregate_proxy_runtime_rows_from_events(
active_job=active_job,
settings_payload=settings_payload,
participant_node_codes=participant_node_codes,
participant_server_codes=participant_server_codes,
)
cluster_runtime = _load_runtime_states_from_cluster_nodes(participant_server_codes)
rows: list[dict] = []
for code in participant_server_codes:
runtime_row = dict(cluster_runtime.get(code) or {})
event_row = dict(event_runtime.get(code) or {})
if not runtime_row and not event_row:
continue
merged_row = {
"node_code": code,
"available_proxy_count": int(runtime_row.get("available_proxy_count", 0) or 0),
"proxy_runtime_label": str(runtime_row.get("proxy_runtime_label") or "").strip(),
"proxy_runtime_reason": str(runtime_row.get("proxy_runtime_reason") or "").strip(),
"proxy_last_refresh_status": str(runtime_row.get("proxy_last_refresh_status") or "").strip(),
"proxy_last_refresh_time": str(runtime_row.get("proxy_last_refresh_time") or runtime_row.get("last_heartbeat_at") or "").strip(),
"proxy_last_refresh_source_count": int(runtime_row.get("proxy_last_refresh_source_count", 0) or 0),
"proxy_last_refresh_total_items": int(runtime_row.get("proxy_last_refresh_total_items", 0) or 0),
"proxy_last_validated_count": int(runtime_row.get("proxy_last_validated_count", 0) or 0),
}
if int(merged_row.get("available_proxy_count", 0) or 0) <= 0 and int(event_row.get("available_proxy_count", 0) or 0) > 0:
merged_row["available_proxy_count"] = int(event_row.get("available_proxy_count", 0) or 0)
if not str(merged_row.get("proxy_runtime_label") or "").strip():
merged_row["proxy_runtime_label"] = str(event_row.get("proxy_runtime_label") or "").strip()
if not str(merged_row.get("proxy_runtime_reason") or "").strip():
merged_row["proxy_runtime_reason"] = str(event_row.get("proxy_runtime_reason") or "").strip()
if not str(merged_row.get("proxy_last_refresh_status") or "").strip():
merged_row["proxy_last_refresh_status"] = str(event_row.get("proxy_last_refresh_status") or "").strip()
if not str(merged_row.get("proxy_last_refresh_time") or "").strip():
merged_row["proxy_last_refresh_time"] = str(event_row.get("proxy_last_refresh_time") or "").strip()
if int(merged_row.get("proxy_last_refresh_source_count", 0) or 0) <= 0:
merged_row["proxy_last_refresh_source_count"] = int(event_row.get("proxy_last_refresh_source_count", 0) or 0)
if int(merged_row.get("proxy_last_refresh_total_items", 0) or 0) <= 0:
merged_row["proxy_last_refresh_total_items"] = int(event_row.get("proxy_last_refresh_total_items", 0) or 0)
if int(merged_row.get("proxy_last_validated_count", 0) or 0) <= 0:
merged_row["proxy_last_validated_count"] = int(event_row.get("proxy_last_validated_count", 0) or 0)
rows.append(merged_row)
if not rows:
return fallback_available_proxy_count, fallback_proxy_runtime
total_available_proxy_count = sum(max(0, int(item.get("available_proxy_count", 0) or 0)) for item in rows)
latest_refresh_time = max((str(item.get("proxy_last_refresh_time") or "") for item in rows), default="")
source_count = sum(int(item.get("proxy_last_refresh_source_count", 0) or 0) for item in rows)
raw_items = sum(int(item.get("proxy_last_refresh_total_items", 0) or 0) for item in rows)
validated_count = sum(int(item.get("proxy_last_validated_count", 0) or 0) for item in rows)
refresh_status_parts = [
f"{str(item.get('node_code') or '')}:{str(item.get('proxy_last_refresh_status') or '').strip()}"
for item in rows
if str(item.get("proxy_last_refresh_status") or "").strip()
]
refresh_status = "；".join(refresh_status_parts[:6])
if len(refresh_status_parts) > 6:
refresh_status = f"{refresh_status} 等 {len(refresh_status_parts)} 台"
if total_available_proxy_count > 0:
return total_available_proxy_count, {
"state": "healthy",
"label": "集群代理正常",
"detail": (
f"参与服务器 {len(rows)} 台，共可用 {total_available_proxy_count} 个代理"
+ (f"；最近状态：{refresh_status}" if refresh_status else "")
),
"direct_fallback_active": False,
"reason": "aggregate_healthy",
"last_refresh_status": refresh_status,
"last_refresh_time": latest_refresh_time,
"source_count": source_count,
"raw_items": raw_items,
"validated_count": validated_count,
"available_count": total_available_proxy_count,
"source_stats": [],
"supplier_empty": False,
}
fallback_label = next((str(item.get("proxy_runtime_label") or "").strip() for item in rows if str(item.get("proxy_runtime_label") or "").strip()), "")
fallback_reason = next((str(item.get("proxy_runtime_reason") or "").strip() for item in rows if str(item.get("proxy_runtime_reason") or "").strip()), "")
if fallback_label:
return 0, {
"state": "warming_up",
"label": fallback_label,
"detail": (
f"参与服务器 {len(rows)} 台，当前尚未汇总到可用代理"
+ (f"；最近状态：{refresh_status}" if refresh_status else "")
),
"direct_fallback_active": bool(fallback_proxy_runtime.get("direct_fallback_active", False)),
"reason": fallback_reason or "aggregate_proxy_unavailable",
"last_refresh_status": refresh_status,
"last_refresh_time": latest_refresh_time,
"source_count": source_count,
"raw_items": raw_items,
"validated_count": validated_count,
"available_count": 0,
"source_stats": [],
"supplier_empty": bool(fallback_proxy_runtime.get("supplier_empty", False)),
}
return fallback_available_proxy_count, fallback_proxy_runtime
def _normalize_recent_warning(runtime_state: dict, recent_lines: list[str], available_proxy_count: int) -> str:
runtime_warning = str(runtime_state.get("recent_warning", "") or "").strip()
if runtime_warning:
@@ -869,6 +1600,13 @@ def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, a
def get_detect_status() -> dict:
global _DETECT_STATUS_CACHE_EXPIRES_AT, _DETECT_STATUS_CACHE_VALUE
now_ts = time.monotonic()
with _DETECT_STATUS_CACHE_LOCK:
if _DETECT_STATUS_CACHE_VALUE is not None and now_ts < _DETECT_STATUS_CACHE_EXPIRES_AT:
return _clone_detect_status_payload(_DETECT_STATUS_CACHE_VALUE)
try:
ensure_runtime_schema()
except Exception:
@@ -901,12 +1639,18 @@ def get_detect_status() -> dict:
pass
settings_payload = get_settings_payload()
worker_expected_on_this_node = _local_worker_expected_on_this_node()
runtime_settings = get_runtime_settings()
worker_online, last_log_time, recent_lines = _load_recent_worker_lines(runtime_settings, max_lines=160)
runtime = detect_worker_runtime()
runtime_state = _load_runtime_state()
if not runtime_state:
runtime_state = _load_runtime_state_from_cluster_node()
if not worker_expected_on_this_node:
worker_online = False
last_log_time = ""
recent_lines = []
runtime_state = {}
runtime_started_at = runtime.get("latest_start_time", "")
recent_lines = _filter_lines_since(recent_lines, runtime_started_at)
available_proxy_count = _extract_available_proxy_count(recent_lines)
@@ -961,13 +1705,49 @@ def get_detect_status() -> dict:
active_job = get_active_detect_job_summary(event_limit=240)
except Exception:
active_job = None
if settings.node_region == "overseas" and settings.node_role == "control" and active_job:
aggregate_detect_view = bool(settings.node_region == "overseas" and settings.node_role == "control" and active_job)
aggregate_queue_health = {}
if aggregate_detect_view:
try:
aggregate_queue_health = get_detect_queue_health(window_minutes=15)
except Exception:
aggregate_queue_health = {}
active_job = _merge_aggregate_active_job_with_queue_health(active_job, aggregate_queue_health)
aggregate_capacity = (
_build_aggregate_detect_capacity(active_job=active_job, settings_payload=settings_payload)
if aggregate_detect_view
else {
"participant_node_codes": [],
"participant_node_count": 0,
"process_count": 0,
"max_threads": 0,
"per_process_thread_count": 0,
}
)
if aggregate_detect_view:
raw_aggregate_node_rows = list(active_job.get("distributed_node_stats") or active_job.get("node_stats") or [])
aggregate_node_rows = _filter_live_aggregate_runtime_nodes(raw_aggregate_node_rows)
aggregate_display_running = sum(
_max_runtime_metric(
item.get("display_running"),
item.get("current_load"),
item.get("active_threads"),
item.get("items_running"),
)
for item in aggregate_node_rows
if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned"
)
if raw_aggregate_node_rows:
display_running = aggregate_display_running
else:
display_running = _max_runtime_metric(
active_job.get("display_active_threads"),
active_job.get("display_items_running"),
(aggregate_queue_health.get("queue") or {}).get("display_running"),
)
progress = {
"pending": int(active_job.get("items_pending", 0) or 0),
"running": int(
active_job.get("display_active_threads", active_job.get("display_items_running", active_job.get("items_running", 0)))
or 0
),
"running": display_running,
"completed": int(active_job.get("items_completed", 0) or 0),
"failed": int(active_job.get("items_failed", 0) or 0),
"blacklisted": int(active_job.get("items_blacklisted", 0) or 0),
@@ -989,8 +1769,20 @@ def get_detect_status() -> dict:
active_thread_snapshot["active"] = local_runtime_load
if active_thread_snapshot["max"] <= 0:
active_thread_snapshot["max"] = local_runtime_max_threads or effective_thread_count
if settings.node_region == "overseas" and settings.node_role == "control" and active_job:
distributed_node_stats = list(active_job.get("distributed_node_stats") or active_job.get("node_stats") or [])
if aggregate_detect_view:
raw_distributed_node_stats = list(active_job.get("distributed_node_stats") or active_job.get("node_stats") or [])
distributed_node_stats = _filter_live_aggregate_runtime_nodes(raw_distributed_node_stats)
raw_participant_count = sum(
1
for item in raw_distributed_node_stats
if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned"
)
live_participant_count = sum(
1
for item in distributed_node_stats
if str(item.get("node_code") or "").strip() and str(item.get("node_code") or "").strip() != "unassigned"
)
dropped_aggregate_node_count = max(0, raw_participant_count - live_participant_count)
aggregated_active_threads = 0
aggregated_max_threads = 0
for item in distributed_node_stats:
@@ -1005,8 +1797,38 @@ def get_detect_status() -> dict:
aggregated_max_threads += int(item.get("max_threads", 0) or 0)
if aggregated_active_threads > 0:
active_thread_snapshot["active"] = aggregated_active_threads
elif dropped_aggregate_node_count <= 0 and _max_runtime_metric(
active_job.get("display_active_threads"),
active_job.get("display_items_running"),
progress.get("running"),
) > 0:
active_thread_snapshot["active"] = _max_runtime_metric(
active_job.get("display_active_threads"),
active_job.get("display_items_running"),
progress.get("running"),
)
if aggregated_max_threads > 0:
active_thread_snapshot["max"] = aggregated_max_threads
elif dropped_aggregate_node_count <= 0 and _max_runtime_metric(
active_job.get("display_max_threads"),
aggregate_capacity.get("max_threads"),
) > 0:
active_thread_snapshot["max"] = _max_runtime_metric(
active_job.get("display_max_threads"),
aggregate_capacity.get("max_threads"),
)
configured_max_threads = int(aggregate_capacity.get("max_threads", 0) or 0)
if configured_max_threads > 0:
active_thread_snapshot["max"] = max(active_thread_snapshot["max"], configured_max_threads)
available_proxy_count, proxy_runtime = _build_aggregate_proxy_runtime(
active_job=active_job,
settings_payload=settings_payload,
fallback_available_proxy_count=available_proxy_count,
fallback_proxy_runtime=proxy_runtime,
)
display_worker_process_count = int(runtime.get("process_count", 0) or 0)
if aggregate_detect_view and int(aggregate_capacity.get("process_count", 0) or 0) > 0:
display_worker_process_count = int(aggregate_capacity.get("process_count", 0) or 0)
runtime_snapshot = {
**runtime,
"detecting": inferred_detecting,
@@ -1029,21 +1851,22 @@ def get_detect_status() -> dict:
)
remote_log_lines = list(remote_log_snapshot.get("lines") or [])
dependency_alerts = _extract_dependency_alerts(recent_lines)
append_detect_result_projection_if_changed(
detect={
"active_job": active_job,
"progress": progress,
"phase_label": runtime_state.get("phase", ""),
"phase_detail": runtime_state.get("detail", ""),
}
)
if _local_worker_expected_on_this_node():
append_detect_result_projection_if_changed(
detect={
"active_job": active_job,
"progress": progress,
"phase_label": runtime_state.get("phase", ""),
"phase_detail": runtime_state.get("detail", ""),
}
)
return {
result = {
"worker_online": worker_online,
"worker_mode": runtime.get("mode", "windows-local"),
"worker_service_name": runtime_settings.get("worker_service_name", ""),
"api_service_name": runtime_settings.get("api_service_name", ""),
"worker_process_count": runtime.get("process_count", 0),
"worker_process_count": display_worker_process_count,
"worker_latest_start_time": runtime.get("latest_start_time", ""),
"worker_runtime_message": runtime.get("message", ""),
"runtime_state": runtime_state,
@@ -1057,6 +1880,11 @@ def get_detect_status() -> dict:
"thread_count_node_code": str(thread_count_resolution["node_code"]),
"active_thread_count": active_thread_snapshot["active"],
"max_thread_count": active_thread_snapshot["max"] or effective_thread_count,
"aggregate_process_count": int(aggregate_capacity.get("process_count", 0) or 0),
"aggregate_participating_node_count": int(aggregate_capacity.get("participant_node_count", 0) or 0),
"aggregate_participating_node_codes": list(aggregate_capacity.get("participant_node_codes") or []),
"aggregate_max_thread_count": int(aggregate_capacity.get("max_threads", 0) or 0),
"aggregate_thread_count_per_process": int(aggregate_capacity.get("per_process_thread_count", 0) or 0),
"proxy_enable": settings_payload["proxy_config"].get("proxy_enable", False),
"allow_direct": settings_payload["proxy_config"].get("allow_direct", False),
"proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])),
@@ -1080,7 +1908,7 @@ def get_detect_status() -> dict:
"progress_percent": progress_percent,
"recent_event": runtime_state.get("detail") or _recent_event(recent_lines),
"recent_warning": recent_proxy_warning,
"aggregate_detect_view": bool(settings.node_region == "overseas" and settings.node_role == "control" and active_job),
"aggregate_detect_view": aggregate_detect_view,
"log_lines": recent_lines,
"remote_log_lines": remote_log_lines,
"remote_log_line_count": int(remote_log_snapshot.get("line_count", 0) or 0),
@@ -1094,3 +1922,7 @@ def get_detect_status() -> dict:
"worker_log_sync_enabled": worker_log_sync_enabled,
"worker_log_sync_mode": worker_log_sync_mode,
}
with _DETECT_STATUS_CACHE_LOCK:
_DETECT_STATUS_CACHE_VALUE = _clone_detect_status_payload(result)
_DETECT_STATUS_CACHE_EXPIRES_AT = time.monotonic() + _DETECT_STATUS_CACHE_TTL_SECONDS
return result

99
domain-api/app/services/domains_service.py
View File

@@ -1,5 +1,6 @@
from __future__ import annotations
from datetime import datetime
from math import ceil
from app.core.db import get_db
@@ -113,6 +114,8 @@ def _build_step_details(row: tuple) -> list[dict]:
_normalize_step_detail("时光机", row[21]),
_normalize_step_detail("站长之家", row[22]),
_normalize_step_detail("爱站网", row[23]),
_normalize_step_detail("桔子SEO", row[24]),
_normalize_step_detail("聚查", row[25]),
]
@@ -152,6 +155,28 @@ def _normalize_detection_update(value) -> bool | None:
raise ValueError("检测结果字段仅支持“是”或“否”")
def _now_text() -> str:
return datetime.now().isoformat(sep=" ", timespec="seconds")
def _merge_detection_step_update(field: str, value: bool, existing_payload) -> dict | bool:
if field == "is_chinese_title":
return bool(value)
existing = dict(existing_payload or {}) if isinstance(existing_payload, dict) else {}
previous_status = existing.get("status") if isinstance(existing.get("status"), bool) else None
merged = dict(existing)
merged["status"] = bool(value)
merged["state"] = "passed" if bool(value) else "failed"
merged["step"] = str(merged.get("step") or field)
if previous_status != bool(value) or not str(merged.get("message") or "").strip():
merged["message"] = "人工批量更新"
if previous_status != bool(value) or not str(merged.get("checked_at") or "").strip():
merged["checked_at"] = _now_text()
merged["manual_override"] = True
return merged
def _build_domain_query_parts(filters: dict | None = None) -> tuple[str, str, list[object]]:
filters = filters or {}
conditions: list[str] = []
@@ -190,8 +215,9 @@ def _build_domain_query_parts(filters: dict | None = None) -> tuple[str, str, li
if filters.get("source_type") is not None:
conditions.append("d.source_type = %s")
params.append(int(filters["source_type"]))
if filters.get("backlink_gt_10"):
conditions.append("coalesce(dd.backlink_count_gt_10, false) = true")
if filters.get("backlink_gt_10") is not None:
conditions.append("coalesce(dd.backlink_count_gt_10, false) = %s")
params.append(bool(filters["backlink_gt_10"]))
from_clause = """
from domains d
@@ -264,7 +290,9 @@ def fetch_domains(
dd.google_site,
dd.wayback_info,
dd.chinaz_info,
dd.aizhan_info
dd.aizhan_info,
dd.juziseo_info,
dd.jucha_info
{from_clause}
{where_clause}
order by d.id desc
@@ -360,26 +388,7 @@ def fetch_domain_detail(domain_id: int) -> dict | None:
if not row:
return None
step_details = [
_normalize_step_detail("百度历史收录", row[16]),
_normalize_step_detail("百度Site收录", row[17]),
{
"label": "标题为中文",
"state": "passed" if bool(row[18]) else "",
"status": bool(row[18]),
"message": "标题含中文" if bool(row[18]) else "",
"checked_at": "",
"step": "中文标题",
"raw": row[18],
},
_normalize_step_detail("360 Site收录", row[19]),
_normalize_step_detail("Google Site收录", row[20]),
_normalize_step_detail("时光机", row[21]),
_normalize_step_detail("站长之家", row[22]),
_normalize_step_detail("爱站网", row[23]),
_normalize_step_detail("桔子SEO", row[24]),
_normalize_step_detail("聚查", row[25]),
]
step_details = _build_step_details(row)
step_summary = _summarize_step_details(step_details)
return {
"id": row[0],
@@ -523,6 +532,34 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict:
tuple(params),
)
existing_detection_payloads: dict[str, object] = {}
existing_detection_row = None
if "backlink_count" in payload or detection_fields.intersection(payload.keys()):
cur.execute(
"""
select
id,
baidu_history,
baidu_site,
is_chinese_title,
qihu360_site,
google_site,
backlink_count_gt_10
from domain_detections
where domain_id = %s
""",
(domain_id,),
)
existing_detection_row = cur.fetchone()
if existing_detection_row:
existing_detection_payloads = {
"baidu_history": existing_detection_row[1],
"baidu_site": existing_detection_row[2],
"is_chinese_title": existing_detection_row[3],
"qihu360_site": existing_detection_row[4],
"google_site": existing_detection_row[5],
}
detection_payload: dict[str, object] = {}
for field in detection_fields:
if field not in payload:
@@ -530,15 +567,15 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict:
normalized = _normalize_detection_update(payload[field])
if normalized is None:
continue
if field == "is_chinese_title":
detection_payload[field] = normalized
else:
detection_payload[field] = {"status": normalized}
detection_payload[field] = _merge_detection_step_update(
field,
normalized,
existing_detection_payloads.get(field),
)
if "backlink_count" in payload:
backlink_gt_10 = int(payload["backlink_count"]) > 10
cur.execute("select id from domain_detections where domain_id = %s", (domain_id,))
if cur.fetchone():
if existing_detection_row:
cur.execute(
"update domain_detections set backlink_count_gt_10 = %s, update_time = now() where domain_id = %s",
(backlink_gt_10, domain_id),
@@ -553,9 +590,7 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict:
)
if detection_payload:
cur.execute("select id from domain_detections where domain_id = %s", (domain_id,))
existing_detection = cur.fetchone()
if existing_detection:
if existing_detection_row:
detection_set_parts: list[str] = []
detection_params: list[object] = []
for field, value in detection_payload.items():

104
domain-api/app/services/import_task_service.py
View File

@@ -10,6 +10,7 @@ from app.services.import_worker_service import import_domains_from_path
_IMPORT_TASK_LOCK = threading.Lock()
_IMPORT_EXECUTION_LOCK = threading.Lock()
_SOURCE_TYPE_LABELS = {
6: "手工录入",
7: "TXT 导入",
@@ -77,63 +78,64 @@ def _source_label(source_type: int) -> str:
def _run_import_task(task_id: str, file_path: str, source_type: int = 7) -> None:
_update_task_with_log(
task_id,
f"导入任务开始执行，来源类型：{_source_label(source_type)}",
status="running",
started_at=_now(),
message=f"导入任务开始执行，来源类型：{_source_label(source_type)}",
phase="reading",
phase_label=_phase_label("reading"),
)
try:
path = Path(file_path)
with _IMPORT_EXECUTION_LOCK:
_update_task_with_log(
task_id,
f"开始读取文件：{path.name}",
f"导入任务开始执行，来源类型：{_source_label(source_type)}",
status="running",
started_at=_now(),
message=f"导入任务开始执行，来源类型：{_source_label(source_type)}",
phase="reading",
phase_label=_phase_label("reading"),
)
raw_lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
total_lines = len(raw_lines)
non_empty = sum(1 for line in raw_lines if line.strip())
_update_task_with_log(
task_id,
f"文件读取完成，共 {total_lines} 行，非空 {non_empty} 行",
phase="normalizing",
phase_label=_phase_label("normalizing"),
message=f"文件读取完成，准备清洗 {non_empty} 条域名",
)
try:
path = Path(file_path)
_update_task_with_log(
task_id,
f"开始读取文件：{path.name}",
phase="reading",
phase_label=_phase_label("reading"),
)
raw_lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
total_lines = len(raw_lines)
non_empty = sum(1 for line in raw_lines if line.strip())
_update_task_with_log(
task_id,
f"文件读取完成，共 {total_lines} 行，非空 {non_empty} 行",
phase="normalizing",
phase_label=_phase_label("normalizing"),
message=f"文件读取完成，准备清洗 {non_empty} 条域名",
)
result = import_domains_from_path(path, source_type=source_type)
stats = result.get("stats", {})
_update_task_with_log(
task_id,
(
f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，"
f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}，"
f"来源类型 {result.get('source_label') or _source_label(source_type)}"
),
status="completed",
completed_at=_now(),
result=result,
message=(
f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，"
f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}"
),
phase="completed",
phase_label=_phase_label("completed"),
)
except Exception as exc:
_update_task_with_log(
task_id,
f"导入失败：{exc}",
status="failed",
completed_at=_now(),
message=f"导入失败：{exc}",
phase="failed",
phase_label=_phase_label("failed"),
)
result = import_domains_from_path(path, source_type=source_type)
stats = result.get("stats", {})
_update_task_with_log(
task_id,
(
f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，"
f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}，"
f"来源类型 {result.get('source_label') or _source_label(source_type)}"
),
status="completed",
completed_at=_now(),
result=result,
message=(
f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，"
f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}"
),
phase="completed",
phase_label=_phase_label("completed"),
)
except Exception as exc:
_update_task_with_log(
task_id,
f"导入失败：{exc}",
status="failed",
completed_at=_now(),
message=f"导入失败：{exc}",
phase="failed",
phase_label=_phase_label("failed"),
)
def create_import_task(content: bytes, filename: str, source_type: int = 7) -> dict:

17
domain-api/app/services/import_worker_service.py
View File

@@ -47,6 +47,8 @@ def import_domains_from_path(file_path: Path, source_type: int = 7) -> dict:
domains = [row[0] for row in normalized_rows]
existing_set: set[str] = set()
inserted = 0
exists = 0
seen_in_batch: set[str] = set()
with get_db() as conn:
with conn.cursor() as cur:
@@ -55,7 +57,12 @@ def import_domains_from_path(file_path: Path, source_type: int = 7) -> dict:
existing_set = {row[0] for row in cur.fetchall()}
for domain, tld in normalized_rows:
if domain in seen_in_batch:
exists += 1
continue
seen_in_batch.add(domain)
if domain in existing_set:
exists += 1
continue
cur.execute(
"""
@@ -70,11 +77,17 @@ def import_domains_from_path(file_path: Path, source_type: int = 7) -> dict:
null, now(), now(), 0, null,
0, 0, 0
)
on conflict (domain) do nothing
returning id
""",
(domain, tld, source_type),
)
domain_id = cur.fetchone()[0]
inserted_row = cur.fetchone()
if not inserted_row:
existing_set.add(domain)
exists += 1
continue
domain_id = inserted_row[0]
cur.execute(
"""
insert into detect_tasks (domain_id, task_type, status, priority, retry_count, create_time, update_time)
@@ -82,10 +95,10 @@ def import_domains_from_path(file_path: Path, source_type: int = 7) -> dict:
""",
(domain_id,),
)
existing_set.add(domain)
inserted += 1
conn.commit()
exists = len(existing_set)
valid = len(normalized_rows)
stats = {
"total": total,

24
domain-api/app/services/ops_action_executor_core.py
View File

@@ -6,10 +6,10 @@ from pathlib import Path
from typing import Protocol
import psycopg2
import redis
from app.core.config import settings
from app.core.files import runtime_root as api_runtime_root
from app.core.redis_client import get_redis
STRUCTURED_ACTIONS = {
@@ -272,24 +272,10 @@ def _truncate_detect_runtime_tables(*, include_domains: bool) -> dict:
def _flush_runtime_redis() -> dict:
client = redis.Redis(
host=settings.redis_host,
port=settings.redis_port,
password=settings.redis_password or None,
db=settings.redis_db,
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
)
try:
size_before = int(client.dbsize() or 0)
client.flushdb()
size_after = int(client.dbsize() or 0)
finally:
try:
client.close()
except Exception:
pass
client = get_redis()
size_before = int(client.dbsize() or 0)
client.flushdb()
size_after = int(client.dbsize() or 0)
return {"db": int(settings.redis_db), "size_before": size_before, "size_after": size_after}

96
domain-api/app/services/ops_agent_service.py
View File

@@ -7,6 +7,8 @@ import shlex
import threading
from datetime import datetime, timedelta
from psycopg2 import errors
from app.core.config import settings
from app.core.db import get_db
from app.services.ops_command_service import build_bash_command
@@ -80,6 +82,32 @@ ALTER TABLE ops_jobs ADD COLUMN IF NOT EXISTS last_agent_complete_request_id VAR
_OPS_AGENT_SCHEMA_LOCK = threading.Lock()
_OPS_AGENT_SCHEMA_READY = False
_OPS_AGENT_SCHEMA_ADVISORY_LOCK_KEY = 90421802
_OPS_AGENT_REQUIRED_TABLES = ("ops_node_tokens", "ops_job_events")
_OPS_AGENT_REQUIRED_COLUMNS = {
"ops_node_tokens": (
"node_code",
"purpose",
"issued_by",
"is_enabled",
"expires_at",
"last_used_at",
"metadata_json",
"created_at",
"updated_at",
),
"ops_job_events": (
"job_id",
"step_id",
"node_code",
"client_event_id",
"event_type",
"level",
"message",
"payload_json",
"created_at",
),
"ops_jobs": ("last_agent_complete_request_id",),
}
def ensure_ops_agent_schema() -> None:
@@ -91,14 +119,52 @@ def ensure_ops_agent_schema() -> None:
if _OPS_AGENT_SCHEMA_READY:
return
with get_db() as conn:
conn.autocommit = False
with conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (_OPS_AGENT_SCHEMA_ADVISORY_LOCK_KEY,))
cur.execute(_AGENT_SCHEMA_SQL)
conn.commit()
if _ops_agent_schema_basics_present(cur):
_OPS_AGENT_SCHEMA_READY = True
return
conn.autocommit = False
try:
with conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (_OPS_AGENT_SCHEMA_ADVISORY_LOCK_KEY,))
cur.execute(_AGENT_SCHEMA_SQL)
conn.commit()
except Exception as exc:
recoverable = isinstance(exc, (errors.DeadlockDetected, errors.LockNotAvailable))
try:
conn.rollback()
except Exception:
pass
if not recoverable:
raise
with conn.cursor() as cur:
if not _ops_agent_schema_basics_present(cur):
raise
_OPS_AGENT_SCHEMA_READY = True
def _ops_agent_schema_basics_present(cur) -> bool:
for table_name in _OPS_AGENT_REQUIRED_TABLES:
cur.execute("SELECT to_regclass(%s)", (f"public.{table_name}",))
row = cur.fetchone()
if not row or not row[0]:
return False
for table_name, required_columns in _OPS_AGENT_REQUIRED_COLUMNS.items():
cur.execute(
"""
SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = %s
""",
(table_name,),
)
existing_columns = {str(row[0] or "").strip() for row in list(cur.fetchall() or [])}
if not set(required_columns).issubset(existing_columns):
return False
return True
def _hash_token(token: str) -> str:
return hashlib.sha256(str(token or "").encode("utf-8")).hexdigest()
@@ -152,12 +218,21 @@ def _merge_detect_runtime_snapshot(cluster_metadata: dict, metadata: dict, curre
"phase_label": str(cluster_metadata.get("phase_label", metadata.get("phase_label", "")) or "").strip(),
"phase_detail": str(cluster_metadata.get("phase_detail", metadata.get("phase_detail", "")) or "").strip(),
"recent_warning": str(cluster_metadata.get("recent_warning", metadata.get("recent_warning", "")) or "").strip(),
"available_proxy_count": int(
cluster_metadata.get("available_proxy_count", metadata.get("available_proxy_count", 0)) or 0
),
"proxy_runtime_label": str(
cluster_metadata.get("proxy_runtime_label", metadata.get("proxy_runtime_label", "")) or ""
).strip(),
"proxy_runtime_reason": str(
cluster_metadata.get("proxy_runtime_reason", metadata.get("proxy_runtime_reason", "")) or ""
).strip(),
"proxy_last_refresh_status": str(
cluster_metadata.get("proxy_last_refresh_status", metadata.get("proxy_last_refresh_status", "")) or ""
).strip(),
"proxy_last_refresh_time": str(
cluster_metadata.get("proxy_last_refresh_time", metadata.get("proxy_last_refresh_time", "")) or ""
).strip(),
"updated_at": str(cluster_metadata.get("updated_at", metadata.get("updated_at", "")) or "").strip(),
}
@@ -2714,6 +2789,15 @@ def _upsert_agent_detect_runtime(node_code: str, payload: dict) -> None:
"phase_label": phase_label,
"phase_detail": phase_detail,
"recent_warning": recent_warning,
"available_proxy_count": max(0, int(detect_runtime.get("available_proxy_count") or 0)),
"proxy_runtime_label": str(detect_runtime.get("proxy_runtime_label") or "").strip(),
"proxy_runtime_reason": str(detect_runtime.get("proxy_runtime_reason") or "").strip(),
"proxy_last_refresh_status": str(detect_runtime.get("proxy_last_refresh_status") or "").strip(),
"proxy_last_refresh_time": str(detect_runtime.get("proxy_last_refresh_time") or "").strip(),
"proxy_last_refresh_source_count": max(0, int(detect_runtime.get("proxy_last_refresh_source_count") or 0)),
"proxy_last_refresh_total_items": max(0, int(detect_runtime.get("proxy_last_refresh_total_items") or 0)),
"proxy_last_validated_count": max(0, int(detect_runtime.get("proxy_last_validated_count") or 0)),
"proxy_last_available_count": max(0, int(detect_runtime.get("proxy_last_available_count") or 0)),
"updated_at": str(detect_runtime.get("updated_at") or "").strip(),
"agent_heartbeat_at": datetime.now().isoformat(timespec="seconds"),
}
@@ -2778,8 +2862,10 @@ def _build_agent_runtime_config_bundle(node_code: str) -> dict:
"node_code": str(node_code or "").strip(),
"detect_options": dict(settings_payload.get("detect_options") or {}),
"proxy_config": dict(settings_payload.get("proxy_config") or {}),
"thread_count": int(settings_payload.get("thread_count", 2) or 2),
"thread_count": int(settings_payload.get("thread_count", 1000) or 1000),
"node_thread_counts": dict(settings_payload.get("node_thread_counts") or {}),
"process_count": int(settings_payload.get("process_count", 80) or 80),
"node_process_counts": dict(settings_payload.get("node_process_counts") or {}),
"runtime_settings": dict(runtime_settings or {}),
"sensitive_words": {
"text": str(sensitive_words_payload.get("text") or ""),

1
domain-api/app/services/ops_execution_capability_service.py
View File

@@ -19,6 +19,7 @@ _SSH_ACTIONS = set(STRUCTURED_ACTIONS) | {"deploy.release"}
_REMOTE_AGENT_ACTIONS = set(STRUCTURED_ACTIONS) | _REMOTE_AGENT_ONLY_ACTIONS | {"deploy.release"}
_CONTROL_PLANE_ACTIONS = {
"node.bootstrap",
"migration.execute",
}

84
domain-api/app/services/ops_job_service.py
View File

@@ -5,6 +5,8 @@ import threading
from datetime import datetime
from uuid import uuid4
from psycopg2 import errors
from app.core.config import settings
from app.core.db import get_db
from app.services.ops_execution_capability_service import (
@@ -116,6 +118,28 @@ _LOCAL_RUNTIME_ACTIONS = {
_OPS_SCHEMA_LOCK = threading.Lock()
_OPS_SCHEMA_READY = False
_OPS_SCHEMA_ADVISORY_LOCK_KEY = 90421801
_OPS_REQUIRED_TABLES = (
"ops_managed_nodes",
"ops_managed_node_secrets",
"ops_jobs",
"ops_job_steps",
)
_OPS_REQUIRED_COLUMNS = {
"ops_jobs": (
"risk_level",
"approval_required",
"approval_status",
"approved_by",
"approved_at",
"blocked_reason",
"cancellation_reason",
"dispatched_at",
"target_selector_json",
"policy_json",
"rollout_id",
),
"ops_job_steps": ("stdout_text", "stderr_text", "result_json"),
}
def ensure_ops_schema() -> None:
@@ -126,14 +150,52 @@ def ensure_ops_schema() -> None:
if _OPS_SCHEMA_READY:
return
with get_db() as conn:
conn.autocommit = False
with conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (_OPS_SCHEMA_ADVISORY_LOCK_KEY,))
cur.execute(_OPS_SCHEMA_SQL)
conn.commit()
if _ops_schema_basics_present(cur):
_OPS_SCHEMA_READY = True
return
conn.autocommit = False
try:
with conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (_OPS_SCHEMA_ADVISORY_LOCK_KEY,))
cur.execute(_OPS_SCHEMA_SQL)
conn.commit()
except Exception as exc:
recoverable = isinstance(exc, (errors.DeadlockDetected, errors.LockNotAvailable))
try:
conn.rollback()
except Exception:
pass
if not recoverable:
raise
with conn.cursor() as cur:
if not _ops_schema_basics_present(cur):
raise
_OPS_SCHEMA_READY = True
def _ops_schema_basics_present(cur) -> bool:
for table_name in _OPS_REQUIRED_TABLES:
cur.execute("SELECT to_regclass(%s)", (f"public.{table_name}",))
row = cur.fetchone()
if not row or not row[0]:
return False
for table_name, required_columns in _OPS_REQUIRED_COLUMNS.items():
cur.execute(
"""
SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = %s
""",
(table_name,),
)
existing_columns = {str(row[0] or "").strip() for row in list(cur.fetchall() or [])}
if not set(required_columns).issubset(existing_columns):
return False
return True
def _decode_json(value: object) -> dict:
if isinstance(value, dict):
return value
@@ -819,6 +881,7 @@ def _resolve_bootstrap_target_defaults(target_node_code: str) -> dict:
def _execute_control_plane_job(
action: str,
*,
job_id: int = 0,
target_node_code: str,
payload: dict | None = None,
requested_by: str = "api",
@@ -864,6 +927,17 @@ def _execute_control_plane_job(
}
return False, message, data
if action == "migration.execute":
from app.services.ops_migration_service import execute_ops_migration_job
return execute_ops_migration_job(
job_id=int(job_id or 0),
target_node_code=str(target_node_code or "").strip(),
payload=dict(normalized_payload or {}),
requested_by=str(requested_by or "api").strip() or "api",
metadata=dict(normalized_metadata or {}),
)
return False, f"当前未实现控制面执行动作: {action}", {}
@@ -1006,6 +1080,7 @@ def _execute_control_plane_job_record(job_id: int) -> tuple[bool, str, dict]:
ok, message, result = _execute_control_plane_job(
str(job.get("action") or ""),
job_id=int(job_id),
target_node_code=str(job.get("target_node_code") or ""),
payload=dict(job.get("payload") or {}),
requested_by=str(job.get("requested_by") or "api"),
@@ -1339,6 +1414,7 @@ def create_ops_job(payload: dict) -> tuple[bool, str, dict]:
if execution_mode == "control-plane":
ok, message, result = _execute_control_plane_job(
action,
job_id=job_id,
target_node_code=target_node_code,
payload=dict(input_payload or {}),
requested_by=requested_by,

1559
domain-api/app/services/ops_migration_service.py Normal file
View File

File diff suppressed because it is too large Load Diff

8
domain-api/app/services/ops_policy_service.py
View File

@@ -41,6 +41,7 @@ _HIGH_RISK_ACTIONS = {
_CRITICAL_RISK_ACTIONS = {
"deploy.rollback",
"node.bootstrap",
"migration.execute",
"cluster.reconfigure",
"runtime.reset_lab_state",
}
@@ -114,6 +115,13 @@ def _preview_action_payload_guardrails(
if target_nodes_total > 1:
recommendations.append("接管动作建议按单节点节奏推进，先确认首台节点接入成功后再继续放量。")
if action == "migration.execute":
if execution_mode != "control-plane":
blocking_reasons.append("migration.execute 仅支持 control-plane 执行方式。")
if bool(payload.get("overwrite_database", False)):
approval_reasons.append("迁移任务包含数据库覆盖，正式环境必须显式确认后再执行。")
recommendations.append("迁移属于长任务，建议通过后台任务窗口持续观察日志与健康检查结果。")
if action in _CRITICAL_RISK_ACTIONS:
approval_reasons.append("该动作属于 critical 风险动作，正式环境必须审批。")
elif action in _HIGH_RISK_ACTIONS:

102
domain-api/app/services/ops_release_executor_core.py
View File

@@ -11,6 +11,7 @@ import tarfile
import textwrap
import time
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime
from pathlib import Path
@@ -23,6 +24,9 @@ _SYSTEMD_TEMPLATE_SPECS = {
"domaincheck-worker": {
"template": Path("domain-api/deploy/systemd/domain-worker.service"),
},
"domaincheck-worker@": {
"template": Path("domain-api/deploy/systemd/domain-worker@.service"),
},
"domaincheck-sync-agent": {
"template": Path("domain-api/deploy/systemd/domain-sync-agent.service"),
},
@@ -71,6 +75,78 @@ def normalize_release_health_check_services(
return normalize_text_list(health_check_service_source)
def normalize_release_health_check_urls(raw_value: object) -> list[str]:
normalized_urls: list[str] = []
for raw_item in normalize_text_list(raw_value):
normalized_item = str(raw_item or "").strip()
if not normalized_item:
continue
try:
parsed = urllib.parse.urlsplit(normalized_item)
except Exception:
normalized_urls.append(normalized_item)
continue
normalized_path = str(parsed.path or "").rstrip("/")
if normalized_path in {"/api/v1/runtime/status", "/runtime/status"}:
parsed = parsed._replace(path="/health", query="", fragment="")
normalized_item = urllib.parse.urlunsplit(parsed)
normalized_urls.append(normalized_item)
return normalized_urls
def _dedupe_service_names(service_names: list[str]) -> list[str]:
deduped: list[str] = []
seen: set[str] = set()
for item in service_names:
normalized = str(item or "").strip()
if not normalized or normalized in seen:
continue
seen.add(normalized)
deduped.append(normalized)
return deduped
def _list_worker_instance_units(run_command, base_service_name: str) -> list[str]:
normalized_base = str(base_service_name or "").strip()
if normalized_base != "domaincheck-worker":
return []
code, stdout, stderr = run_command(
[
"systemctl",
"list-units",
"--type=service",
"--all",
"domaincheck-worker@*",
"--no-legend",
"--plain",
],
timeout=30,
)
raw_output = stdout if stdout.strip() else stderr
if int(code or 0) != 0 and not str(raw_output or "").strip():
return []
units: list[str] = []
for line in str(raw_output or "").splitlines():
parts = line.strip().split()
if not parts:
continue
unit_name = str(parts[0] or "").strip()
if unit_name:
units.append(unit_name)
return _dedupe_service_names(units)
def expand_release_service_units(run_command, service_names: list[str]) -> list[str]:
expanded: list[str] = []
for item in list(service_names or []):
normalized = str(item or "").strip()
if not normalized:
continue
expanded.append(normalized)
expanded.extend(_list_worker_instance_units(run_command, normalized))
return _dedupe_service_names(expanded)
def collect_service_state(run_command, service_name: str) -> dict:
code, stdout, stderr = run_command(["systemctl", "is-active", service_name], timeout=15)
state = stdout or stderr
@@ -295,7 +371,7 @@ def _systemd_dropin_content(service_name: str, install_root: str) -> str:
"[Service]",
f"WorkingDirectory={normalized_install_root}/current/domain-api",
"ExecStart=",
f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100",
f"ExecStart={normalized_install_root}/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100 --timeout-graceful-shutdown 15",
]
)
if normalized_service_name == "domaincheck-worker":
@@ -411,7 +487,7 @@ def execute_release_action(
install_root = Path(str(normalized_payload.get("install_root") or "/opt/domaincheck")).resolve()
switch_current = coerce_bool(normalized_payload.get("switch_current", True), default=True)
restart_services = normalize_text_list(normalized_payload.get("restart_services"))
health_check_urls = normalize_text_list(normalized_payload.get("health_check_urls"))
health_check_urls = normalize_release_health_check_urls(normalized_payload.get("health_check_urls"))
health_check_services = normalize_release_health_check_services(
normalized_payload,
restart_services,
@@ -457,14 +533,17 @@ def execute_release_action(
"prepared_dirs": prepared_dirs,
}
expanded_restart_services = expand_release_service_units(run_command, restart_services)
expanded_health_check_services = expand_release_service_units(run_command, health_check_services)
service_execstarts = [
collect_service_execstart(run_command, service_name)
for service_name in restart_services
for service_name in expanded_restart_services
if str(service_name or "").strip()
]
service_identities = [
collect_service_identity(run_command, service_name)
for service_name in restart_services
for service_name in expanded_restart_services
if str(service_name or "").strip()
]
current_link_text = str(current_link)
@@ -645,7 +724,7 @@ def execute_release_action(
)
restarted: list[dict] = []
for service_name in restart_services:
for service_name in expanded_restart_services:
normalized_service_name = str(service_name or "").strip()
if not normalized_service_name:
continue
@@ -681,7 +760,7 @@ def execute_release_action(
health_ok, health_result = run_release_health_checks(
urls=health_check_urls,
services=health_check_services,
services=expanded_health_check_services,
timeout=health_check_timeout_seconds,
retries=health_check_retries,
interval_seconds=health_check_interval_seconds,
@@ -723,7 +802,7 @@ def execute_release_action(
)
rollback_result["post_rollback_health"] = run_release_health_checks(
urls=health_check_urls,
services=health_check_services,
services=expanded_health_check_services,
timeout=health_check_timeout_seconds,
retries=0,
interval_seconds=0,
@@ -766,6 +845,8 @@ def execute_release_action(
"current_link": str(current_link),
"previous_current_target": previous_current_target,
"prepared_dirs": prepared_dirs,
"expanded_restart_services": expanded_restart_services,
"expanded_health_check_services": expanded_health_check_services,
"execstart_alignment": execstart_alignment,
"systemd_sync": systemd_sync_result,
"daemon_reload": daemon_reload_result,
@@ -784,6 +865,7 @@ def build_remote_release_action_script(
normalize_text_list,
coerce_bool,
normalize_release_health_check_services,
normalize_release_health_check_urls,
collect_service_state,
check_health_url,
run_release_health_checks,
@@ -793,6 +875,9 @@ def build_remote_release_action_script(
_pick_release_owner_group,
apply_release_permissions,
collect_service_execstart,
_dedupe_service_names,
_list_worker_instance_units,
expand_release_service_units,
_write_text_file,
_systemd_dropin_content,
_sync_release_systemd_units,
@@ -816,8 +901,11 @@ def build_remote_release_action_script(
)
return f"""from __future__ import annotations
import grp
import hashlib
import json
import os
import pwd
import shutil
import tarfile
import time

73
domain-api/app/services/ops_release_service.py
View File

@@ -12,6 +12,8 @@ from threading import Lock
from urllib.parse import urlsplit, urlunsplit
from uuid import uuid4
from psycopg2 import errors
from app.core.db import get_db
from app.services.build_info_service import get_runtime_build_info
from app.services.ops_command_service import build_bash_command
@@ -74,6 +76,19 @@ _ROLLOUT_INSPECTION_ACTION_KEYS = ("health.snapshot", "logs.collect", "diagnosti
_SAFE_RELEASE_PACKAGE_NAME_RE = re.compile(r"^[A-Za-z0-9._-]+$")
_RELEASE_SCHEMA_LOCK = Lock()
_RELEASE_SCHEMA_READY = False
_RELEASE_SCHEMA_ADVISORY_LOCK_KEY = 90421803
_RELEASE_REQUIRED_TABLES = ("ops_releases", "ops_release_rollouts")
_RELEASE_REQUIRED_COLUMNS = {
"ops_release_rollouts": (
"rollout_code",
"target_nodes_json",
"batch_cursor",
"batches_total",
"jobs_total",
"jobs_created",
"result_summary_json",
),
}
def ensure_ops_release_schema() -> None:
@@ -86,13 +101,52 @@ def ensure_ops_release_schema() -> None:
if _RELEASE_SCHEMA_READY:
return
with get_db() as conn:
conn.autocommit = False
with conn.cursor() as cur:
cur.execute(_RELEASE_SCHEMA_SQL)
conn.commit()
if _ops_release_schema_basics_present(cur):
_RELEASE_SCHEMA_READY = True
return
conn.autocommit = False
try:
with conn.cursor() as cur:
cur.execute("SELECT pg_advisory_xact_lock(%s)", (_RELEASE_SCHEMA_ADVISORY_LOCK_KEY,))
cur.execute(_RELEASE_SCHEMA_SQL)
conn.commit()
except Exception as exc:
recoverable = isinstance(exc, (errors.DeadlockDetected, errors.LockNotAvailable))
try:
conn.rollback()
except Exception:
pass
if not recoverable:
raise
with conn.cursor() as cur:
if not _ops_release_schema_basics_present(cur):
raise
_RELEASE_SCHEMA_READY = True
def _ops_release_schema_basics_present(cur) -> bool:
for table_name in _RELEASE_REQUIRED_TABLES:
cur.execute("SELECT to_regclass(%s)", (f"public.{table_name}",))
row = cur.fetchone()
if not row or not row[0]:
return False
for table_name, required_columns in _RELEASE_REQUIRED_COLUMNS.items():
cur.execute(
"""
SELECT column_name
FROM information_schema.columns
WHERE table_schema = 'public' AND table_name = %s
""",
(table_name,),
)
existing_columns = {str(row[0] or "").strip() for row in list(cur.fetchall() or [])}
if not set(required_columns).issubset(existing_columns):
return False
return True
def _decode_json(value: object) -> dict:
if isinstance(value, dict):
return value
@@ -1091,7 +1145,11 @@ def build_rollout_target_operational_readiness(
desired_release: dict | None = None,
) -> dict:
from app.services.cluster_runtime_service import get_cluster_snapshot
from app.services.ops_agent_service import ensure_ops_agent_schema, get_managed_node_onboarding
from app.services.ops_agent_service import (
ensure_ops_agent_schema,
get_managed_node_onboarding,
list_managed_nodes_with_agent_state,
)
from app.services.ops_job_service import list_managed_nodes
ensure_ops_agent_schema()
@@ -1132,6 +1190,7 @@ def build_rollout_target_operational_readiness(
for item in list(cluster_snapshot.get("nodes") or [])
if str(item.get("node_code") or "").strip()
}
managed_nodes_payload = list_managed_nodes_with_agent_state()
managed_nodes = list_managed_nodes()
managed_map = {
str(item.get("node_code") or "").strip(): item
@@ -1239,7 +1298,11 @@ def build_rollout_target_operational_readiness(
last_seen_at = str(managed.get("last_seen_at") or "").strip() or str(metadata.get("last_seen_at") or "").strip()
cluster_status = str(cluster_node.get("status") or target.get("status") or "").strip()
current_load = int(cluster_node.get("current_load", target.get("current_load", 0)) or 0)
onboarding = get_managed_node_onboarding(node_code) if node_code else {}
onboarding = (
get_managed_node_onboarding(node_code, nodes_payload=managed_nodes_payload)
if node_code
else {}
)
onboarding_stage = dict(onboarding.get("onboarding_stage") or {})
recovery_decision = dict(onboarding.get("recovery_decision") or {})
onboarding_stage_code = str(onboarding_stage.get("code") or "").strip()

54
domain-api/app/services/ops_service.py
View File

@@ -4180,6 +4180,8 @@ def get_ops_activity_stream(
status: str = "",
execution_mode: str = "",
query: str = "",
runtime_status: dict | None = None,
managed_nodes_payload: dict | None = None,
) -> dict:
safe_limit = min(max(int(limit or _OPS_ACTIVITY_FETCH_LIMIT), 1), 100)
safe_scan_limit = min(max(int(scan_limit or (safe_limit * 4)), safe_limit), 400)
@@ -4187,12 +4189,18 @@ def get_ops_activity_stream(
normalized_status = str(status or "").strip()
normalized_execution_mode = str(execution_mode or "").strip()
normalized_query = str(query or "").strip()
runtime_status = get_runtime_status()
execution_scene = _build_ops_execution_scene(runtime_status.get("detect") or {})
managed_nodes_payload = list_managed_nodes_with_agent_state()
resolved_runtime_status = dict(runtime_status or {})
if not resolved_runtime_status:
resolved_runtime_status = get_runtime_status()
execution_scene = _build_ops_execution_scene(resolved_runtime_status.get("detect") or {})
resolved_managed_nodes_payload = dict(managed_nodes_payload or {})
if not resolved_managed_nodes_payload:
resolved_managed_nodes_payload = list_managed_nodes_with_agent_state(
participation_payload=resolved_runtime_status.get("detect") or {}
)
managed_node_map = {
str(item.get("node_code") or "").strip(): dict(item or {})
for item in list(managed_nodes_payload.get("nodes") or [])
for item in list(resolved_managed_nodes_payload.get("nodes") or [])
if str(item.get("node_code") or "").strip()
}
@@ -4219,7 +4227,10 @@ def get_ops_activity_stream(
rollouts = list_release_rollouts(limit=safe_scan_limit)
rollout_items = [_build_rollout_activity(rollout) for rollout in rollouts if int(rollout.get("id") or 0) > 0]
runbook = get_ops_runbook()
runbook = get_ops_runbook(
runtime_status=resolved_runtime_status,
managed_nodes_payload=resolved_managed_nodes_payload,
)
runbook_items = [
_build_runbook_sequence_activity(sequence)
for sequence in list(runbook.get("control_sequences") or [])
@@ -8099,7 +8110,11 @@ def get_ops_overview() -> dict:
managed_nodes = list(managed_nodes_payload.get("nodes") or [])
managed_nodes_summary = managed_nodes_payload.get("summary") or {}
inspection_overview = get_ops_inspection_overview(managed_nodes=managed_nodes)
activity_stream = get_ops_activity_stream(limit=8)
activity_stream = get_ops_activity_stream(
limit=8,
runtime_status=runtime,
managed_nodes_payload=managed_nodes_payload,
)
release_summary = get_release_summary()
preferred_release = _preferred_release_for_ops()
release_launchpad = get_release_launchpad()
@@ -11555,17 +11570,30 @@ def get_ops_blueprint() -> dict:
}
def get_ops_runbook() -> dict:
runtime = get_runtime_status()
def get_ops_runbook(
*,
runtime_status: dict | None = None,
managed_nodes_payload: dict | None = None,
release_launchpad: dict | None = None,
) -> dict:
runtime = dict(runtime_status or {})
if not runtime:
runtime = get_runtime_status()
readiness = runtime.get("readiness") or {}
worker_runtime = runtime.get("worker") or {}
sync_agent_runtime = runtime.get("sync_agent") or {}
managed_nodes_payload = list_managed_nodes_with_agent_state(participation_payload=runtime.get("detect") or {})
release_launchpad = get_release_launchpad()
resolved_managed_nodes_payload = dict(managed_nodes_payload or {})
if not resolved_managed_nodes_payload:
resolved_managed_nodes_payload = list_managed_nodes_with_agent_state(
participation_payload=runtime.get("detect") or {}
)
resolved_release_launchpad = dict(release_launchpad or {})
if not resolved_release_launchpad:
resolved_release_launchpad = get_release_launchpad()
control_sequences = _attach_ops_runbook_sequence_resolutions(
_build_ops_runbook_control_sequences(
managed_nodes_payload=managed_nodes_payload,
release_launchpad=release_launchpad,
managed_nodes_payload=resolved_managed_nodes_payload,
release_launchpad=resolved_release_launchpad,
),
requested_by="api/runbook",
)
@@ -11606,6 +11634,6 @@ def get_ops_runbook() -> dict:
"status": str(readiness.get("status") or ""),
"summary": str(readiness.get("summary") or ""),
},
"release_launchpad": release_launchpad,
"release_launchpad": resolved_release_launchpad,
"control_sequences": control_sequences,
}

11
domain-api/app/services/runtime_control_service.py
View File

@@ -211,14 +211,21 @@ def runtime_action(action: str, payload: dict | None = None) -> tuple[bool, str,
_emit_runtime_action_event(normalized_action, stage="finished", ok=command_ok, message=command_message, data=result)
return command_ok, command_message, result
if normalized_action == "stop_detection":
command_ok, command_message = send_worker_command("stop_detection")
command_ok, command_message = send_worker_command(
"stop_detection",
payload={
key: value
for key, value in normalized_payload.items()
if value not in (None, "")
},
)
result = _build_runtime_action_result(
action=normalized_action,
poll_after_seconds=2,
refresh_runtime=True,
ok=command_ok,
message=command_message,
data={},
data={"payload": normalized_payload},
)
_emit_runtime_action_event(normalized_action, stage="finished", ok=command_ok, message=command_message, data=result)
return command_ok, command_message, result

72
domain-api/app/services/runtime_settings_service.py
View File

@@ -1,7 +1,10 @@
from __future__ import annotations
import json
from app.core.config import settings
from app.core.files import read_runtime_json, write_runtime_json
from app.core.redis_client import get_redis
DEFAULT_RUNTIME_SETTINGS = {
@@ -11,13 +14,50 @@ DEFAULT_RUNTIME_SETTINGS = {
"sync_agent_service_name": settings.sync_agent_service_name,
"worker_log_sync_enabled": False,
"worker_log_sync_mode": "key",
"control_node_autoresume_enabled": False,
"claim_recent_jobs_first": False,
"claim_recent_jobs_limit": 0,
"claim_recent_jobs_window_hours": 0,
"claim_batch_floor": 0,
"claim_batch_ceil": 0,
"submit_backlog_floor": 0,
"submit_backlog_ceil": 0,
"dispatch_cap_multiplier": 1,
"pending_buffer_cap_multiplier": 1,
}
RUNTIME_SETTINGS_REDIS_KEY = "domain_tool:runtime_settings"
CONFIG_UPDATE_CHANNEL = "domain_tool:config_update"
def _normalize_worker_log_sync_mode(value: object) -> str:
return "full" if str(value or "").strip().lower() == "full" else "key"
def _normalize_bool(value: object, default: bool = False) -> bool:
if value is None:
return bool(default)
if isinstance(value, bool):
return value
return str(value or "").strip().lower() not in {"", "0", "false", "no", "off"}
def _normalize_non_negative_int(value: object, default: int = 0) -> int:
try:
normalized = int(value)
except (TypeError, ValueError):
normalized = int(default)
return max(0, normalized)
def _normalize_positive_int(value: object, default: int = 1) -> int:
try:
normalized = int(value)
except (TypeError, ValueError):
normalized = int(default)
return max(1, normalized)
def normalize_runtime_settings(payload: dict | None) -> dict:
merged = dict(DEFAULT_RUNTIME_SETTINGS)
if isinstance(payload, dict):
@@ -33,8 +73,28 @@ def normalize_runtime_settings(payload: dict | None) -> dict:
value = str(merged.get(key) or "").strip()
merged[key] = value or DEFAULT_RUNTIME_SETTINGS[key]
merged["worker_log_sync_enabled"] = bool(merged.get("worker_log_sync_enabled", False))
merged["worker_log_sync_enabled"] = _normalize_bool(merged.get("worker_log_sync_enabled", False), default=False)
merged["worker_log_sync_mode"] = _normalize_worker_log_sync_mode(merged.get("worker_log_sync_mode"))
merged["control_node_autoresume_enabled"] = _normalize_bool(
merged.get("control_node_autoresume_enabled", False),
default=False,
)
merged["claim_recent_jobs_first"] = _normalize_bool(
merged.get("claim_recent_jobs_first", False),
default=False,
)
for key in ("claim_batch_floor", "claim_batch_ceil", "submit_backlog_floor", "submit_backlog_ceil"):
merged[key] = _normalize_non_negative_int(merged.get(key), DEFAULT_RUNTIME_SETTINGS[key])
for key in (
"claim_recent_jobs_limit",
"claim_recent_jobs_window_hours",
"dispatch_cap_multiplier",
"pending_buffer_cap_multiplier",
):
if key in {"claim_recent_jobs_limit", "claim_recent_jobs_window_hours"}:
merged[key] = _normalize_non_negative_int(merged.get(key), DEFAULT_RUNTIME_SETTINGS[key])
continue
merged[key] = _normalize_positive_int(merged.get(key), DEFAULT_RUNTIME_SETTINGS[key])
return merged
@@ -43,7 +103,17 @@ def get_runtime_settings() -> dict:
return normalize_runtime_settings(stored)
def _sync_runtime_settings_update(runtime_settings: dict) -> None:
try:
redis_client = get_redis()
redis_client.set(RUNTIME_SETTINGS_REDIS_KEY, json.dumps(runtime_settings, ensure_ascii=False))
redis_client.publish(CONFIG_UPDATE_CHANNEL, "runtime_settings")
except Exception:
pass
def update_runtime_settings(payload: dict) -> dict:
merged = normalize_runtime_settings({**get_runtime_settings(), **(payload or {})})
write_runtime_json("runtime_settings.json", merged)
_sync_runtime_settings_update(merged)
return merged

932
domain-api/app/services/runtime_status_service.py
View File

File diff suppressed because it is too large Load Diff

118
domain-api/app/services/settings_service.py
View File

@@ -14,6 +14,8 @@ REDIS_KEYS = {
"proxy_config": "domain_tool:proxy_config",
"thread_count": "domain_tool:thread_count",
"node_thread_counts": "domain_tool:node_thread_counts",
"process_count": "domain_tool:process_count",
"node_process_counts": "domain_tool:node_process_counts",
"credentials": "domain_tool:credentials",
"runtime_settings": "domain_tool:runtime_settings",
}
@@ -40,6 +42,16 @@ def _normalize_thread_count(value: object, *, field_name: str = "thread_count")
return thread_count
def _normalize_process_count(value: object, *, field_name: str = "process_count") -> int:
try:
process_count = int(value)
except Exception as exc:
raise ValueError(f"{field_name} must be an integer") from exc
if process_count < 1:
raise ValueError(f"{field_name} must be >= 1")
return process_count
def _normalize_node_thread_counts(payload: object) -> dict[str, int]:
if payload in (None, ""):
return {}
@@ -55,14 +67,32 @@ def _normalize_node_thread_counts(payload: object) -> dict[str, int]:
return normalized
def _normalize_node_process_counts(payload: object) -> dict[str, int]:
if payload in (None, ""):
return {}
if not isinstance(payload, dict):
raise ValueError("node_process_counts must be an object")
normalized: dict[str, int] = {}
for raw_node_code, raw_process_count in payload.items():
node_code = str(raw_node_code or "").strip()
if not node_code:
raise ValueError("node_process_counts contains empty node code")
normalized[node_code] = _normalize_process_count(
raw_process_count,
field_name=f"node_process_counts.{node_code}",
)
return normalized
def _load_thread_count_config() -> tuple[int, dict[str, int]]:
thread_count_payload = read_json("thread_count.json", default={"thread_count": "2"})
thread_count_payload = read_json("thread_count.json", default={"thread_count": "1000"})
node_thread_counts_payload = read_json("node_thread_counts.json", default={})
try:
default_thread_count = _normalize_thread_count(thread_count_payload.get("thread_count", 2))
default_thread_count = _normalize_thread_count(thread_count_payload.get("thread_count", 1000))
except ValueError:
default_thread_count = 2
default_thread_count = 1000
try:
node_thread_counts = _normalize_node_thread_counts(node_thread_counts_payload)
except ValueError:
@@ -86,9 +116,40 @@ def _load_thread_count_config() -> tuple[int, dict[str, int]]:
return default_thread_count, node_thread_counts
def _load_process_count_config() -> tuple[int, dict[str, int]]:
process_count_payload = read_json("process_count.json", default={"process_count": "80"})
node_process_counts_payload = read_json("node_process_counts.json", default={})
try:
default_process_count = _normalize_process_count(process_count_payload.get("process_count", 80))
except ValueError:
default_process_count = 80
try:
node_process_counts = _normalize_node_process_counts(node_process_counts_payload)
except ValueError:
node_process_counts = {}
redis_client = get_redis()
try:
if redis_process_count := redis_client.get(REDIS_KEYS["process_count"]):
try:
default_process_count = _normalize_process_count(redis_process_count)
except ValueError:
pass
if redis_node_process_counts := redis_client.get(REDIS_KEYS["node_process_counts"]):
try:
node_process_counts = _normalize_node_process_counts(json.loads(redis_node_process_counts))
except ValueError:
pass
except Exception:
pass
return default_process_count, node_process_counts
def resolve_thread_count(node_code: str | None = None, settings_payload: dict | None = None) -> dict:
payload = settings_payload or get_settings_payload()
default_thread_count = int(payload.get("thread_count", 2))
default_thread_count = int(payload.get("thread_count", 1000))
node_thread_counts = _normalize_node_thread_counts(payload.get("node_thread_counts", {}))
normalized_node_code = str(node_code or app_settings.node_code or "").strip()
@@ -110,10 +171,35 @@ def resolve_thread_count(node_code: str | None = None, settings_payload: dict |
}
def resolve_process_count(node_code: str | None = None, settings_payload: dict | None = None) -> dict:
payload = settings_payload or get_settings_payload()
default_process_count = int(payload.get("process_count", 80))
node_process_counts = _normalize_node_process_counts(payload.get("node_process_counts", {}))
normalized_node_code = str(node_code or app_settings.node_code or "").strip()
override_process_count = None
source = "default"
effective_process_count = default_process_count
if normalized_node_code and normalized_node_code in node_process_counts:
override_process_count = node_process_counts[normalized_node_code]
effective_process_count = override_process_count
source = "node_override"
return {
"node_code": normalized_node_code,
"default_process_count": default_process_count,
"effective_process_count": effective_process_count,
"override_process_count": override_process_count,
"source": source,
"node_process_counts": node_process_counts,
}
def get_settings_payload() -> dict:
detect_options = read_json("detect_options.json", default={})
proxy_config = read_json("proxy_config.json", default={})
thread_count, node_thread_counts = _load_thread_count_config()
process_count, node_process_counts = _load_process_count_config()
redis_client = get_redis()
try:
@@ -129,6 +215,8 @@ def get_settings_payload() -> dict:
"proxy_config": proxy_config,
"thread_count": thread_count,
"node_thread_counts": node_thread_counts,
"process_count": process_count,
"node_process_counts": node_process_counts,
"current_node_code": app_settings.node_code,
"runtime_settings": get_runtime_settings(),
}
@@ -193,12 +281,18 @@ def update_settings_payload(payload: dict) -> dict:
proxy_config = payload.get("proxy_config", current["proxy_config"])
thread_count = _normalize_thread_count(payload.get("thread_count", current["thread_count"]))
node_thread_counts = _normalize_node_thread_counts(payload.get("node_thread_counts", current.get("node_thread_counts", {})))
process_count = _normalize_process_count(payload.get("process_count", current.get("process_count", 80)))
node_process_counts = _normalize_node_process_counts(
payload.get("node_process_counts", current.get("node_process_counts", {}))
)
runtime_settings = update_runtime_settings(payload.get("runtime_settings", current["runtime_settings"]))
write_json("detect_options.json", detect_options)
write_json("proxy_config.json", proxy_config)
write_json("thread_count.json", {"thread_count": str(thread_count)})
write_json("node_thread_counts.json", node_thread_counts)
write_json("process_count.json", {"process_count": str(process_count)})
write_json("node_process_counts.json", node_process_counts)
redis_client = get_redis()
try:
redis_client.set(REDIS_KEYS["detect_options"], json.dumps(detect_options, ensure_ascii=False))
@@ -209,12 +303,16 @@ def update_settings_payload(payload: dict) -> dict:
redis_client.publish("domain_tool:thread_count:update", str(thread_count))
redis_client.set(REDIS_KEYS["node_thread_counts"], json.dumps(node_thread_counts, ensure_ascii=False))
redis_client.publish("domain_tool:node_thread_counts:update", json.dumps(node_thread_counts, ensure_ascii=False))
redis_client.set(REDIS_KEYS["runtime_settings"], json.dumps(runtime_settings, ensure_ascii=False))
redis_client.publish("domain_tool:config_update", "runtime_settings")
redis_client.set(REDIS_KEYS["process_count"], process_count)
redis_client.publish("domain_tool:process_count:update", str(process_count))
redis_client.set(REDIS_KEYS["node_process_counts"], json.dumps(node_process_counts, ensure_ascii=False))
redis_client.publish("domain_tool:node_process_counts:update", json.dumps(node_process_counts, ensure_ascii=False))
redis_client.publish("domain_tool:config_update", "node_thread_counts")
redis_client.publish("domain_tool:config_update", "node_process_counts")
redis_client.publish("domain_tool:config_update", "detect_options")
redis_client.publish("domain_tool:config_update", "proxy_config")
redis_client.publish("domain_tool:config_update", "thread_count")
redis_client.publish("domain_tool:config_update", "process_count")
except Exception:
pass
@@ -223,6 +321,8 @@ def update_settings_payload(payload: dict) -> dict:
"proxy_config": proxy_config,
"thread_count": thread_count,
"node_thread_counts": node_thread_counts,
"process_count": process_count,
"node_process_counts": node_process_counts,
"current_node_code": app_settings.node_code,
"runtime_settings": runtime_settings,
}
@@ -254,6 +354,12 @@ def validate_settings_payload(payload: dict) -> None:
if "node_thread_counts" in payload:
_normalize_node_thread_counts(payload["node_thread_counts"])
if "process_count" in payload:
_normalize_process_count(payload["process_count"])
if "node_process_counts" in payload:
_normalize_node_process_counts(payload["node_process_counts"])
if "detect_options" in payload:
detect_options = payload["detect_options"]
if not isinstance(detect_options, dict):

621
domain-api/app/services/sync_push_service.py
View File

@@ -2,6 +2,7 @@ from __future__ import annotations
import hashlib
import json
import os
import socket
import urllib.error
import urllib.parse
@@ -11,17 +12,30 @@ from uuid import uuid4
from app.core.config import settings
from app.core.db import get_db
from app.core.redis_client import get_redis
from app.services.cluster_runtime_service import (
cleanup_imported_runtime_nodes,
cleanup_imported_runtime_nodes_many,
get_cluster_snapshot,
register_node_heartbeat,
)
from app.services.detect_job_service import (
_load_domain_pipeline_snapshot,
get_active_detect_job_summary,
resolve_initial_domain_pipeline_item,
)
from app.services.settings_service import get_settings_payload
from app.services.sync_record_service import _decode_json, _normalize_region
from app.services.settings_service import (
get_settings_payload,
resolve_process_count,
resolve_thread_count,
)
from app.services.sync_record_service import (
_RUNTIME_PROJECTION_FUTURE_SKEW_GRACE,
_decode_json,
_normalize_region,
_pick_latest_projection_row,
append_runtime_projection_if_changed,
)
_DETECT_RESULT_EVENT_TYPES = {
@@ -30,6 +44,222 @@ _DETECT_RESULT_EVENT_TYPES = {
"domain_failed",
"domain_blacklisted",
}
_LOCAL_BACKLOG_PENDING_FRESHNESS_HOURS = 6
_LOCAL_BACKLOG_MAX_JOBS = 4
_SYNC_PULL_WORKER_WAKE_TTL_SECONDS = 20
_SYNC_PULL_WORKER_WAKE_KEY_PREFIX = "domain_tool:sync_pull_worker_wake"
def _flag_enabled(raw_value: object, *, default: bool = False) -> bool:
if raw_value is None:
return bool(default)
if isinstance(raw_value, bool):
return raw_value
return str(raw_value or "").strip().lower() not in {"", "0", "false", "no", "off"}
def _fast_runtime_projection_enabled() -> bool:
return _flag_enabled(
os.getenv("DOMAINCHECK_SYNC_RUNTIME_FAST_PROJECTION"),
default=False,
)
def _local_projection_node_code(node_code: str) -> bool:
normalized_node_code = str(node_code or "").strip()
local_node_code = str(settings.node_code or "").strip()
if not normalized_node_code or not local_node_code:
return False
return normalized_node_code == local_node_code or normalized_node_code.startswith(f"{local_node_code}-")
def _append_fast_runtime_projection_snapshot() -> int | None:
local_node_code = str(settings.node_code or "").strip()
if not local_node_code:
return None
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT node_code, region, role, status, current_load, metadata_json
FROM detect_worker_nodes
WHERE node_code = %s OR node_code LIKE %s
ORDER BY node_code ASC
""",
(local_node_code, f"{local_node_code}-%"),
)
raw_rows = list(cur.fetchall() or [])
if not raw_rows:
return None
cluster_nodes: list[dict] = []
queue_nodes: list[dict] = []
busy_nodes: list[str] = []
stale_nodes: list[str] = []
offline_nodes: list[str] = []
online_worker_nodes = 0
dedicated_online_worker_nodes = 0
online_control_nodes = 0
display_running = 0
display_max_threads = 0
controller_metadata: dict = {}
for node_code, region, role, status, current_load, metadata_json in raw_rows:
metadata = dict(metadata_json or {})
normalized_node_code = str(node_code or "").strip()
normalized_role = str(role or metadata.get("source_role") or "").strip() or "worker"
normalized_status = str(status or metadata.get("source_status") or "").strip() or "unknown"
normalized_region = _normalize_region(region, settings.node_region)
normalized_current_load = int(current_load or 0)
active_threads = int(metadata.get("active_threads", 0) or 0)
max_threads = int(metadata.get("max_threads", 0) or 0)
detect_participating = bool(metadata.get("detect_participating", False) or normalized_current_load > 0 or active_threads > 0)
runtime_running = max(active_threads, normalized_current_load)
if normalized_status == "busy":
busy_nodes.append(normalized_node_code)
elif normalized_status == "stale":
stale_nodes.append(normalized_node_code)
elif normalized_status == "offline":
offline_nodes.append(normalized_node_code)
if normalized_status not in {"stale", "offline"}:
if normalized_role == "worker":
online_worker_nodes += 1
if normalized_node_code != local_node_code:
dedicated_online_worker_nodes += 1
elif normalized_role == "control":
online_control_nodes += 1
cluster_nodes.append(
{
"node_code": normalized_node_code,
"role": normalized_role,
"status": normalized_status,
"current_load": normalized_current_load,
"active_threads": active_threads,
"max_threads": max_threads,
"detect_participating": detect_participating,
}
)
queue_nodes.append(
{
"node_code": normalized_node_code,
"items_total": 0,
"items_pending": 0,
"items_claimed": 0,
"items_running": runtime_running,
"items_completed": 0,
"items_blacklisted": 0,
"items_failed": 0,
"display_running": runtime_running,
"display_claimed": 0,
"current_load": normalized_current_load,
"active_threads": active_threads,
"max_threads": max_threads,
"role": normalized_role,
"status": normalized_status,
"detect_participating": detect_participating,
}
)
display_running += runtime_running
display_max_threads += max_threads
if normalized_node_code == local_node_code:
controller_metadata = metadata
controller_job_code = str(controller_metadata.get("active_job_code") or "").strip()
controller_job_status = str(controller_metadata.get("active_job_status") or "").strip()
detect_payload = {
"worker_online": True,
"worker_mode": str(controller_metadata.get("worker_mode") or "linux-systemd").strip() or "linux-systemd",
"phase_label": str(controller_metadata.get("phase_label") or "集群执行中").strip() or "集群执行中",
"phase_detail": str(controller_metadata.get("phase_detail") or "").strip(),
"proxy_runtime_label": str(controller_metadata.get("proxy_runtime_label") or "").strip(),
"proxy_runtime_reason": str(controller_metadata.get("proxy_runtime_reason") or "").strip(),
"detect_participating": bool(display_running > 0),
"progress": {
"pending": 0,
"running": display_running,
"completed": 0,
"blacklisted": 0,
"failed": 0,
},
"queue_health": {
"queue": {
"items_total": 0,
"pending": 0,
"claimed": 0,
"running": display_running,
"completed": 0,
"blacklisted": 0,
"failed": 0,
"terminal": 0,
"display_claimed": 0,
"display_running": display_running,
"display_max_threads": display_max_threads,
},
"nodes": list(queue_nodes),
},
"active_job": {
"job_id": None,
"job_code": controller_job_code,
"status": controller_job_status,
"progress_percent": 0,
"items_total": 0,
"items_terminal": 0,
"items_pending": 0,
"items_claimed": 0,
"items_running": display_running,
"items_failed": 0,
"items_completed": 0,
"display_items_claimed": 0,
"display_items_running": display_running,
"display_active_threads": display_running,
"display_max_threads": display_max_threads,
"node_stats": list(queue_nodes),
"distributed_node_stats": list(queue_nodes),
},
"backlog": {},
"dependency_alerts": [],
}
cluster_payload = {
"nodes": [
{
"node_code": item["node_code"],
"role": item["role"],
"status": item["status"],
"current_load": item["current_load"],
"metadata": {
"active_threads": item["active_threads"],
"max_threads": item["max_threads"],
"detect_participating": item["detect_participating"],
"source_role": item["role"],
"source_status": item["status"],
},
}
for item in cluster_nodes
],
"nodes_total": len(cluster_nodes),
"summary": {
"busy_nodes": busy_nodes,
"stale_nodes": stale_nodes,
"offline_nodes": offline_nodes,
"online_worker_nodes": online_worker_nodes,
"dedicated_online_worker_nodes": dedicated_online_worker_nodes,
"online_control_nodes": online_control_nodes,
},
}
return append_runtime_projection_if_changed(
detect=detect_payload,
cluster=cluster_payload,
source_region=_normalize_region(settings.sync_source_region, settings.node_region),
target_region=_normalize_region(settings.sync_target_region, "overseas"),
)
def _format_time(value: datetime | None) -> str:
@@ -69,6 +299,42 @@ def _task_ack_url(base_url: str) -> str:
return f"{text}/api/v1/runtime/task-ack"
def _build_sync_pull_worker_wake_key(
*,
projection_job_code: str = "",
projection_cycle_token: str = "",
target_job_code: str = "",
source_record_id: int = 0,
) -> str:
scope = (
str(projection_cycle_token or "").strip()
or str(projection_job_code or "").strip()
or str(target_job_code or "").strip()
or f"record-{int(source_record_id or 0)}"
)
return f"{_SYNC_PULL_WORKER_WAKE_KEY_PREFIX}:{scope}"
def _acquire_sync_pull_worker_wake_guard(key: str, ttl_seconds: int = _SYNC_PULL_WORKER_WAKE_TTL_SECONDS) -> bool:
normalized_key = str(key or "").strip()
if not normalized_key:
return True
try:
redis_client = get_redis()
return bool(
redis_client.set(
normalized_key,
datetime.now().isoformat(timespec="seconds"),
ex=max(1, int(ttl_seconds or 1)),
nx=True,
)
)
except Exception:
# Wake dedupe is a throughput optimization; fall back to legacy behavior
# if Redis is temporarily unavailable.
return True
def _projection_ingest_type(sync_type: str) -> str:
if sync_type == "runtime_projection":
return "runtime_ingest"
@@ -126,21 +392,82 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv
cleanup_imported_runtime_nodes(region=region, role=role, keep_node_code=node_code)
active_job = projection.get("active_job") or {}
worker_node_codes: list[str] = []
worker_rows_by_code: dict[str, dict] = {}
for cluster_node in list(projection.get("cluster_nodes") or []):
if not isinstance(cluster_node, dict):
continue
worker_node_code = str(cluster_node.get("node_code") or "").strip()
if not worker_node_code or worker_node_code == node_code:
continue
worker_rows_by_code[worker_node_code] = {
"node_code": worker_node_code,
"role": str(cluster_node.get("role") or "worker").strip() or "worker",
"status": str(cluster_node.get("status") or "").strip(),
"current_load": int(cluster_node.get("current_load", 0) or 0),
"active_threads": int(cluster_node.get("active_threads", 0) or 0),
"max_threads": int(cluster_node.get("max_threads", 0) or 0),
"detect_participating": bool(cluster_node.get("detect_participating", False)),
"items_total": 0,
"items_running": 0,
"items_claimed": 0,
"items_completed": 0,
"items_failed": 0,
"items_blacklisted": 0,
"metrics_source": "runtime",
}
for node_stat in list(active_job.get("node_stats") or []):
worker_node_code = str(node_stat.get("node_code") or "").strip()
if not worker_node_code or worker_node_code == "unassigned":
if not worker_node_code or worker_node_code == "unassigned" or worker_node_code == node_code:
continue
if worker_node_code == node_code:
continue
items_running = int(node_stat.get("items_running", 0) or 0)
items_claimed = int(node_stat.get("items_claimed", 0) or 0)
items_total = int(node_stat.get("items_total", 0) or 0)
worker_runtime_load = int(node_stat.get("current_load", 0) or 0)
worker_active_threads = int(node_stat.get("active_threads", worker_runtime_load) or 0)
worker_max_threads = int(node_stat.get("max_threads", 0) or 0)
worker_load = max(worker_active_threads, items_running, 0)
worker_status = "busy" if worker_load > 0 else "online"
worker_row = worker_rows_by_code.setdefault(
worker_node_code,
{
"node_code": worker_node_code,
"role": str(node_stat.get("role") or "worker").strip() or "worker",
"status": str(node_stat.get("status") or "").strip(),
"current_load": int(node_stat.get("current_load", 0) or 0),
"active_threads": int(node_stat.get("active_threads", 0) or 0),
"max_threads": int(node_stat.get("max_threads", 0) or 0),
"detect_participating": False,
"items_total": 0,
"items_running": 0,
"items_claimed": 0,
"items_completed": 0,
"items_failed": 0,
"items_blacklisted": 0,
"metrics_source": "runtime",
},
)
worker_row["role"] = str(node_stat.get("role") or worker_row.get("role") or "worker").strip() or "worker"
worker_row["status"] = str(node_stat.get("status") or worker_row.get("status") or "").strip()
worker_row["current_load"] = max(int(worker_row.get("current_load", 0) or 0), int(node_stat.get("current_load", 0) or 0))
worker_row["active_threads"] = max(int(worker_row.get("active_threads", 0) or 0), int(node_stat.get("active_threads", 0) or 0))
worker_row["max_threads"] = max(int(worker_row.get("max_threads", 0) or 0), int(node_stat.get("max_threads", 0) or 0))
worker_row["detect_participating"] = bool(
worker_row.get("detect_participating", False)
or int(node_stat.get("items_running", 0) or 0) > 0
or int(node_stat.get("items_claimed", 0) or 0) > 0
or int(node_stat.get("active_threads", 0) or 0) > 0
)
worker_row["items_total"] = int(node_stat.get("items_total", worker_row.get("items_total", 0)) or 0)
worker_row["items_running"] = int(node_stat.get("items_running", worker_row.get("items_running", 0)) or 0)
worker_row["items_claimed"] = int(node_stat.get("items_claimed", worker_row.get("items_claimed", 0)) or 0)
worker_row["items_completed"] = int(node_stat.get("items_completed", worker_row.get("items_completed", 0)) or 0)
worker_row["items_failed"] = int(node_stat.get("items_failed", worker_row.get("items_failed", 0)) or 0)
worker_row["items_blacklisted"] = int(node_stat.get("items_blacklisted", worker_row.get("items_blacklisted", 0)) or 0)
worker_row["metrics_source"] = str(node_stat.get("metrics_source") or worker_row.get("metrics_source") or "runtime").strip() or "runtime"
worker_node_codes: list[str] = []
for worker_node_code, worker_row in sorted(worker_rows_by_code.items()):
items_running = int(worker_row.get("items_running", 0) or 0)
items_claimed = int(worker_row.get("items_claimed", 0) or 0)
items_total = int(worker_row.get("items_total", 0) or 0)
worker_runtime_load = int(worker_row.get("current_load", 0) or 0)
worker_active_threads = int(worker_row.get("active_threads", worker_runtime_load) or 0)
worker_max_threads = int(worker_row.get("max_threads", 0) or 0)
worker_load = max(worker_active_threads, items_running, worker_runtime_load, 0)
worker_status = str(worker_row.get("status") or "").strip() or ("busy" if worker_load > 0 else "online")
worker_metadata = {
"service": "runtime-ingest",
"projection_source_region": source_region,
@@ -155,12 +482,13 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv
"job_items_total": items_total,
"job_items_running": items_running,
"job_items_claimed": items_claimed,
"job_items_completed": int(node_stat.get("items_completed", 0) or 0),
"job_items_failed": int(node_stat.get("items_failed", 0) or 0),
"job_items_blacklisted": int(node_stat.get("items_blacklisted", 0) or 0),
"metrics_source": str(node_stat.get("metrics_source") or "runtime").strip() or "runtime",
"source_status": str(node_stat.get("status") or "").strip(),
"source_role": str(node_stat.get("role") or "worker").strip() or "worker",
"job_items_completed": int(worker_row.get("items_completed", 0) or 0),
"job_items_failed": int(worker_row.get("items_failed", 0) or 0),
"job_items_blacklisted": int(worker_row.get("items_blacklisted", 0) or 0),
"metrics_source": str(worker_row.get("metrics_source") or "runtime").strip() or "runtime",
"source_status": str(worker_row.get("status") or "").strip(),
"source_role": str(worker_row.get("role") or "worker").strip() or "worker",
"detect_participating": bool(worker_row.get("detect_participating", False)),
"derived_from": node_code,
}
register_node_heartbeat(
@@ -181,6 +509,7 @@ def _refresh_remote_runtime_node(*, source_region: str, projection: dict, receiv
def _load_latest_projection(sync_type: str) -> dict | None:
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
target_region = _normalize_region(settings.sync_target_region, "overseas")
future_cutoff = datetime.now() + _RUNTIME_PROJECTION_FUTURE_SKEW_GRACE
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
@@ -190,12 +519,15 @@ def _load_latest_projection(sync_type: str) -> dict | None:
WHERE sync_type = %s
AND source_region = %s
AND target_region = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
ORDER BY
CASE WHEN created_at <= %s THEN 0 ELSE 1 END ASC,
created_at DESC,
id DESC
LIMIT 200
""",
(sync_type, source_region, target_region),
(sync_type, source_region, target_region, future_cutoff),
)
row = cur.fetchone()
row = _pick_latest_projection_row(list(cur.fetchall() or []), created_at_index=5)
if not row:
return None
return {
@@ -222,7 +554,7 @@ def _load_pushable_projections(sync_type: str, limit: int) -> list[dict]:
WHERE sync_type = %s
AND source_region = %s
AND target_region = %s
ORDER BY created_at ASC, id ASC
ORDER BY created_at DESC, id DESC
LIMIT %s
""",
(sync_type, source_region, target_region, safe_limit * 5),
@@ -259,20 +591,158 @@ def _load_pushable_projections(sync_type: str, limit: int) -> list[dict]:
def _estimate_total_worker_threads(settings_payload: dict | None = None) -> int:
payload = settings_payload if isinstance(settings_payload, dict) else get_settings_payload()
default_threads = max(1, int(payload.get("thread_count", 100) or 100))
node_thread_counts = payload.get("node_thread_counts") if isinstance(payload.get("node_thread_counts"), dict) else {}
total_threads = 0
for raw_value in node_thread_counts.values():
thread_info = resolve_thread_count(settings.node_code, settings_payload=payload)
process_info = resolve_process_count(settings.node_code, settings_payload=payload)
effective_threads = max(
1,
int(thread_info.get("effective_thread_count", payload.get("thread_count", 100)) or 100),
)
effective_process_count = max(
1,
int(process_info.get("effective_process_count", payload.get("process_count", 1)) or 1),
)
return effective_threads * effective_process_count
def _sync_task_projection_limit_cap() -> int:
configured_cap = int(os.getenv("DOMAINCHECK_SYNC_TASK_LIMIT_CAP", "200000") or 200000)
return max(10000, configured_cap)
def _resolve_task_pull_request_limit(limit: int | None, settings_payload: dict | None = None) -> int:
configured = max(5000, int(settings.sync_batch_size or 200))
estimated_total_threads = _estimate_total_worker_threads(settings_payload)
cap = _sync_task_projection_limit_cap()
default_limit = max(
configured,
min(cap, max(10000, estimated_total_threads * 2)),
)
requested = int(limit or default_limit)
return max(1, min(requested, cap))
def _select_relevant_backlog_job_ids_from_rows(
job_rows: list[tuple[object, object, object]] | tuple[tuple[object, object, object], ...],
*,
freshness_hours: int = _LOCAL_BACKLOG_PENDING_FRESHNESS_HOURS,
limit: int = _LOCAL_BACKLOG_MAX_JOBS,
) -> list[int]:
safe_limit = max(1, min(int(limit or _LOCAL_BACKLOG_MAX_JOBS), 16))
safe_freshness_hours = max(1, min(int(freshness_hours or _LOCAL_BACKLOG_PENDING_FRESHNESS_HOURS), 168))
selected: list[int] = []
fallback_job_id = 0
for raw_job_id, raw_status, raw_activity_at in list(job_rows or []):
try:
total_threads += max(0, int(raw_value or 0))
job_id = int(raw_job_id or 0)
except (TypeError, ValueError):
continue
return max(total_threads, default_threads)
if job_id <= 0:
continue
if fallback_job_id <= 0:
fallback_job_id = job_id
if job_id in selected:
continue
status = str(raw_status or "").strip().lower()
keep = status == "running"
if not keep and raw_activity_at is not None:
now = datetime.now(raw_activity_at.tzinfo) if getattr(raw_activity_at, "tzinfo", None) else datetime.now()
keep = now - raw_activity_at <= timedelta(hours=safe_freshness_hours)
if not keep:
continue
selected.append(job_id)
if len(selected) >= safe_limit:
break
if not selected and fallback_job_id > 0:
selected.append(fallback_job_id)
return selected
def _build_backlog_snapshot_from_active_job(active_job: dict | None) -> dict:
normalized_job = dict(active_job or {})
if not normalized_job:
return {}
pending_total = max(0, int(normalized_job.get("items_pending", 0) or 0))
claimed_total = max(
max(
int(normalized_job.get("items_claimed", 0) or 0),
int(normalized_job.get("display_items_claimed", 0) or 0),
),
0,
)
running_total = max(
max(
int(normalized_job.get("items_running", 0) or 0),
int(normalized_job.get("display_items_running", 0) or 0),
),
int(normalized_job.get("display_active_threads", 0) or 0),
0,
)
register_pending = 0
downstream_pending = 0
for raw_step in list(normalized_job.get("step_stats") or normalized_job.get("raw_step_stats") or []):
if not isinstance(raw_step, dict):
continue
step_code = str(raw_step.get("step_code") or raw_step.get("code") or "").strip()
step_pending = max(
int(raw_step.get("items_pending", raw_step.get("pending", 0)) or 0),
0,
)
if step_pending <= 0:
continue
if step_code == "detect_register":
register_pending += step_pending
else:
downstream_pending += step_pending
if register_pending <= 0 and downstream_pending <= 0 and pending_total > 0:
downstream_pending = pending_total
if pending_total <= 0 and claimed_total <= 0 and running_total <= 0:
return {}
return {
"pending_total": pending_total,
"claimed_total": claimed_total,
"running_total": running_total,
"register_pending": register_pending,
"downstream_pending": downstream_pending,
}
def _load_local_detect_backlog_snapshot() -> dict:
active_job_snapshot = _build_backlog_snapshot_from_active_job(
get_active_detect_job_summary(event_limit=1)
)
if active_job_snapshot:
return active_job_snapshot
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, status, COALESCE(started_at, created_at) AS activity_at
FROM detect_jobs
WHERE status IN ('pending', 'running')
ORDER BY
CASE WHEN status = 'running' THEN 0 ELSE 1 END,
COALESCE(started_at, created_at) DESC,
id DESC
LIMIT %s
""",
(_LOCAL_BACKLOG_MAX_JOBS * 8,),
)
selected_job_ids = _select_relevant_backlog_job_ids_from_rows(cur.fetchall())
if not selected_job_ids:
return {
"pending_total": 0,
"claimed_total": 0,
"running_total": 0,
"register_pending": 0,
"downstream_pending": 0,
}
cur.execute(
"""
SELECT
@@ -282,9 +752,9 @@ def _load_local_detect_backlog_snapshot() -> dict:
COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code = 'detect_register') AS register_pending,
COUNT(*) FILTER (WHERE item.status = 'pending' AND item.step_code <> 'detect_register') AS downstream_pending
FROM detect_job_items item
JOIN detect_jobs job ON job.id = item.job_id
WHERE job.status IN ('pending', 'running')
"""
WHERE item.job_id = ANY(%s)
""",
(selected_job_ids,),
)
row = cur.fetchone() or (0, 0, 0, 0, 0)
return {
@@ -422,9 +892,9 @@ def _task_selection_sql() -> str:
def _task_projection_limit(limit: int | None) -> int:
requested = max(1, int(limit or 5000))
requested = max(1, int(limit or max(5000, int(settings.sync_batch_size or 200))))
configured = max(5000, int(settings.sync_batch_size or 200))
cap = max(10000, configured, 5000)
cap = max(configured, _sync_task_projection_limit_cap())
return max(1, min(requested, cap))
@@ -1422,15 +1892,21 @@ def ingest_runtime_projection(payload: dict, *, shared_token: str | None = None)
def _push_projection_now(sync_type: str, ingest_url: str) -> tuple[bool, str, dict]:
if sync_type == "runtime_projection":
# Regenerate the runtime snapshot before every push so the sync agent
# does not keep replaying a stale projection record while the worker
# thread count / phase is still changing.
from app.services.runtime_status_service import get_runtime_status
if _fast_runtime_projection_enabled():
try:
_append_fast_runtime_projection_snapshot()
except Exception as exc:
return False, f"快速刷新 runtime_projection 失败: {exc}", {"action": "push_sync", "sync_type": sync_type}
else:
# Refresh only the lightweight runtime projection snapshot before every
# push so the sync agent does not keep replaying a stale record while
# avoiding the full runtime/status assembly cost.
from app.services.runtime_status_service import refresh_runtime_projection_snapshot
try:
get_runtime_status()
except Exception as exc:
return False, f"刷新 runtime_projection 失败: {exc}", {"action": "push_sync", "sync_type": sync_type}
try:
refresh_runtime_projection_snapshot(window_minutes=15)
except Exception as exc:
return False, f"刷新 runtime_projection 失败: {exc}", {"action": "push_sync", "sync_type": sync_type}
source_record = _load_latest_projection(sync_type)
if not source_record:
return False, f"当前没有可推送的{sync_type}", {"action": "push_sync", "sync_type": sync_type}
@@ -1667,12 +2143,10 @@ def pull_detect_task_batch_now(limit: int | None = None) -> tuple[bool, str, dic
if not export_url or not ack_url:
return False, "未配置任务拉取目标地址", {"action": "pull_tasks", "pull_state": "misconfigured", "ui_level": "warning", "poll_schedule_seconds": []}
configured_limit = max(5000, int(settings.sync_batch_size or 200))
requested_limit = int(limit or configured_limit)
safe_limit = max(1, min(requested_limit, max(10000, configured_limit)))
settings_payload = get_settings_payload()
safe_limit = _resolve_task_pull_request_limit(limit, settings_payload=settings_payload)
backlog_snapshot = _load_local_detect_backlog_snapshot()
backlog_limits = _build_task_pull_backlog_limits(configured_limit, settings_payload=settings_payload)
backlog_limits = _build_task_pull_backlog_limits(safe_limit, settings_payload=settings_payload)
should_throttle, throttle_reason = _should_throttle_task_pull(backlog_snapshot, backlog_limits)
if should_throttle:
return True, "本地待处理积压较高，暂停拉取新批次", {
@@ -1804,17 +2278,48 @@ def pull_detect_task_batch_now(limit: int | None = None) -> tuple[bool, str, dic
try:
from app.services.worker_control_service import send_worker_command
start_ok, start_message = send_worker_command(
"start_detection",
payload={
"source": "sync-pull",
"source_record_id": source_record_id,
"target_job_id": int(result.get("target_job_id", 0) or 0),
"target_job_code": str(result.get("target_job_code") or "").strip(),
},
projection_active_job = dict(projection.get("active_job") or {})
projection_job_id = int(projection_active_job.get("job_id", 0) or 0)
projection_job_code = str(projection_active_job.get("job_code") or "").strip()
projection_cycle_token = str(
projection_active_job.get("current_cycle_token")
or projection_active_job.get("cycle_token")
or ""
).strip()
start_payload = {
"source": "sync-pull",
"source_record_id": source_record_id,
"target_job_id": int(result.get("target_job_id", 0) or 0),
"target_job_code": str(result.get("target_job_code") or "").strip(),
}
# Mainland ingest creates a local target_job_* for queue ownership, but
# worker runtime/log identity should still follow the upstream active
# detect job so cluster aggregation keeps controller activity attached
# to the real pipeline job instead of the local sync-pull surrogate.
if projection_job_id > 0:
start_payload["job_id"] = projection_job_id
if projection_job_code:
start_payload["job_code"] = projection_job_code
if projection_cycle_token:
start_payload["cycle_token"] = projection_cycle_token
wake_guard_key = _build_sync_pull_worker_wake_key(
projection_job_code=projection_job_code,
projection_cycle_token=projection_cycle_token,
target_job_code=str(result.get("target_job_code") or "").strip(),
source_record_id=source_record_id,
)
result["worker_start_ok"] = bool(start_ok)
result["worker_start_message"] = str(start_message or "").strip()
if _acquire_sync_pull_worker_wake_guard(wake_guard_key):
start_ok, start_message = send_worker_command(
"start_detection",
payload=start_payload,
)
result["worker_start_ok"] = bool(start_ok)
result["worker_start_message"] = str(start_message or "").strip()
result["worker_start_skipped"] = False
else:
result["worker_start_ok"] = True
result["worker_start_skipped"] = True
result["worker_start_message"] = "已跳过同任务短窗内重复 Worker 唤起"
except Exception as exc:
result["worker_start_ok"] = False
result["worker_start_message"] = f"同步入库后自动唤起 Worker 失败: {exc}"

288
domain-api/app/services/sync_record_service.py
View File

@@ -51,6 +51,80 @@ _TERMINAL_DETECT_RESULT_EVENT_TYPES = {
"domain_blacklisted",
}
_RUNTIME_PROJECTION_HEARTBEAT_INTERVAL = timedelta(seconds=45)
_RUNTIME_PROJECTION_FUTURE_SKEW_GRACE = timedelta(minutes=5)
def _runtime_projection_activity_signature(projection: dict) -> dict:
normalized_projection = dict(projection or {})
active_job = dict(normalized_projection.get("active_job") or {})
normalized_nodes: list[tuple] = []
for raw_item in list(active_job.get("distributed_node_stats") or active_job.get("node_stats") or []):
if not isinstance(raw_item, dict):
continue
node_code = str(raw_item.get("node_code") or "").strip()
if not node_code:
continue
normalized_nodes.append(
(
node_code,
int(raw_item.get("display_running", raw_item.get("current_load", 0)) or 0),
int(raw_item.get("active_threads", 0) or 0),
int(raw_item.get("max_threads", 0) or 0),
int(raw_item.get("items_claimed", 0) or 0),
int(raw_item.get("items_running", 0) or 0),
int(raw_item.get("items_total", 0) or 0),
str(raw_item.get("status") or "").strip(),
)
)
normalized_cluster_nodes: list[tuple] = []
for raw_item in list(normalized_projection.get("cluster_nodes") or []):
if not isinstance(raw_item, dict):
continue
node_code = str(raw_item.get("node_code") or "").strip()
if not node_code:
continue
normalized_cluster_nodes.append(
(
node_code,
str(raw_item.get("role") or "").strip(),
str(raw_item.get("status") or "").strip(),
int(raw_item.get("current_load", 0) or 0),
int(raw_item.get("active_threads", 0) or 0),
int(raw_item.get("max_threads", 0) or 0),
bool(raw_item.get("detect_participating", False)),
)
)
return {
"active_thread_count": int(normalized_projection.get("active_thread_count", 0) or 0),
"max_thread_count": int(normalized_projection.get("max_thread_count", 0) or 0),
"job_display_running": int(active_job.get("display_items_running", 0) or 0),
"job_display_claimed": int(active_job.get("display_items_claimed", 0) or 0),
"job_display_max_threads": int(active_job.get("display_max_threads", 0) or 0),
"job_items_total": int(active_job.get("items_total", 0) or 0),
"job_items_running": int(active_job.get("items_running", 0) or 0),
"job_items_claimed": int(active_job.get("items_claimed", 0) or 0),
"node_stats": normalized_nodes,
"cluster_nodes": normalized_cluster_nodes,
}
def _pick_latest_projection_row(rows: list[tuple], *, created_at_index: int) -> tuple | None:
candidates = list(rows or [])
if not candidates:
return None
fallback = candidates[0]
for row in candidates:
if len(row) <= int(created_at_index):
return row
created_at = row[created_at_index]
if not isinstance(created_at, datetime):
return row
now = datetime.now(created_at.tzinfo) if created_at.tzinfo else datetime.now()
if created_at <= now + _RUNTIME_PROJECTION_FUTURE_SKEW_GRACE:
return row
return fallback
def _collect_recent_domain_events(active_job: dict, limit: int = 30) -> list[dict]:
safe_limit = max(1, int(limit or 30))
@@ -214,10 +288,15 @@ def _should_append_runtime_projection(previous_payload: dict, current_projection
if previous_alerts != current_alerts:
return True
if _runtime_projection_activity_signature(previous_projection) != _runtime_projection_activity_signature(current_projection):
return True
if not previous_created_at:
return True
now = datetime.now(previous_created_at.tzinfo) if previous_created_at.tzinfo else datetime.now()
return now - previous_created_at >= timedelta(seconds=45)
if previous_created_at > now + _RUNTIME_PROJECTION_FUTURE_SKEW_GRACE:
return True
return now - previous_created_at >= _RUNTIME_PROJECTION_HEARTBEAT_INTERVAL
@db_read_retry()
@@ -293,6 +372,25 @@ def get_detect_result_sync_batches(limit: int = 5) -> dict:
safe_limit = max(1, min(int(limit or 5), 20))
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
target_region = _normalize_region(settings.sync_target_region, "overseas")
local_worker_expected = _local_node_expected_to_execute_worker()
if not local_worker_expected:
return {
"applicable": False,
"local_worker_expected": False,
"reason": "当前节点不承载本地检测执行，结果批次推送概览不适用。",
"source_region": source_region,
"target_region": target_region,
"jobs_total": 0,
"state_counts": {
"synced": 0,
"delivered": 0,
"pushing": 0,
"projected": 0,
"failed": 0,
"unsynced": 0,
},
"batches": [],
}
batches: list[dict] = []
with get_db() as conn:
@@ -424,6 +522,9 @@ def get_detect_result_sync_batches(limit: int = 5) -> dict:
state_counts[state] = state_counts.get(state, 0) + 1
return {
"applicable": True,
"local_worker_expected": local_worker_expected,
"reason": "",
"source_region": source_region,
"target_region": target_region,
"jobs_total": len(batches),
@@ -436,6 +537,11 @@ def get_detect_result_sync_batches(limit: int = 5) -> dict:
def get_sync_summary(record_limit: int = 10) -> dict:
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
target_region = _normalize_region(settings.sync_target_region, "overseas")
local_worker_expected = _local_node_expected_to_execute_worker()
push_expected_on_this_node = bool(
str(settings.node_region or "").strip() == "mainland"
and str(settings.node_role or "").strip() == "control"
)
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
@@ -486,6 +592,8 @@ def get_sync_summary(record_limit: int = 10) -> dict:
return {
"enabled": bool(settings.sync_push_enabled),
"push_expected_on_this_node": push_expected_on_this_node,
"local_worker_expected_on_this_node": local_worker_expected,
"source_region": source_region,
"target_region": target_region,
"target_api_base_url": settings.sync_target_api_base_url,
@@ -540,6 +648,107 @@ def _local_node_expected_to_execute_worker() -> bool:
return node_role == "worker" or (node_region == "mainland" and node_role == "control")
def _projection_node_rows(*, detect: dict, active_job: dict) -> list[dict]:
queue_health = dict(detect.get("queue_health") or {})
queue_nodes = [
dict(item)
for item in list(queue_health.get("nodes") or [])
if isinstance(item, dict) and str(item.get("node_code") or "").strip()
]
if queue_nodes:
return queue_nodes
return [
dict(item)
for item in list(active_job.get("distributed_node_stats") or active_job.get("node_stats") or [])
if isinstance(item, dict) and str(item.get("node_code") or "").strip()
]
def _projection_cluster_node_rows(*, cluster: dict) -> list[dict]:
normalized_rows: list[dict] = []
for raw_item in list(cluster.get("nodes") or []):
if not isinstance(raw_item, dict):
continue
node_code = str(raw_item.get("node_code") or "").strip()
if not _is_local_projection_node(node_code):
continue
metadata = dict(raw_item.get("metadata") or {})
normalized_rows.append(
{
"node_code": node_code,
"role": str(raw_item.get("role") or metadata.get("source_role") or "").strip(),
"status": str(raw_item.get("status") or metadata.get("source_status") or "").strip(),
"current_load": int(raw_item.get("current_load", 0) or 0),
"active_threads": int(metadata.get("active_threads", 0) or 0),
"max_threads": int(metadata.get("max_threads", raw_item.get("max_threads", 0)) or 0),
"detect_participating": bool(
raw_item.get("detect_participating", metadata.get("detect_participating", False))
),
}
)
normalized_rows.sort(key=lambda item: str(item.get("node_code") or ""))
return normalized_rows
def _projection_display_summary(*, detect: dict, active_job: dict, node_rows: list[dict]) -> dict:
queue_payload = dict((detect.get("queue_health") or {}).get("queue") or {})
display_running = 0
display_max_threads = 0
display_claimed = 0
running_items = 0
for raw_item in list(node_rows or []):
item = dict(raw_item or {})
running_items += int(item.get("items_running", 0) or 0)
display_running += max(
int(item.get("display_running", 0) or 0),
int(item.get("current_load", 0) or 0),
int(item.get("active_threads", 0) or 0),
int(item.get("items_running", 0) or 0),
)
display_max_threads += max(0, int(item.get("max_threads", 0) or 0))
display_claimed += max(
int(item.get("items_claimed", 0) or 0),
int(item.get("display_claimed", 0) or 0),
)
display_running = max(
display_running,
int(queue_payload.get("display_running", queue_payload.get("running", 0)) or 0),
int(active_job.get("display_active_threads", active_job.get("display_items_running", active_job.get("items_running", 0))) or 0),
)
running_items = max(
running_items,
int(queue_payload.get("running", 0) or 0),
int(active_job.get("items_running", 0) or 0),
)
display_claimed = max(
display_claimed,
int(queue_payload.get("display_claimed", queue_payload.get("claimed", 0)) or 0),
int(active_job.get("display_items_claimed", active_job.get("items_claimed", 0)) or 0),
)
display_max_threads = max(
display_max_threads,
int(active_job.get("display_max_threads", 0) or 0),
int(detect.get("aggregate_max_thread_count", 0) or 0),
int(detect.get("max_thread_count", 0) or 0),
)
return {
"items_running": running_items,
"display_running": display_running,
"display_claimed": display_claimed,
"display_max_threads": display_max_threads,
}
def _is_local_projection_node(node_code: str) -> bool:
normalized_node_code = str(node_code or "").strip()
local_node_code = str(settings.node_code or "").strip()
if not normalized_node_code or not local_node_code:
return False
return normalized_node_code == local_node_code or normalized_node_code.startswith(f"{local_node_code}-")
def _build_runtime_projection_payload(
*,
detect: dict,
@@ -549,6 +758,19 @@ def _build_runtime_projection_payload(
) -> dict:
active_job = detect.get("active_job") or {}
local_worker_expected = _local_node_expected_to_execute_worker()
projection_node_rows = _projection_node_rows(detect=detect, active_job=active_job) if local_worker_expected else []
projection_cluster_rows = _projection_cluster_node_rows(cluster=cluster) if local_worker_expected else []
display_summary = (
_projection_display_summary(detect=detect, active_job=active_job, node_rows=projection_node_rows)
if local_worker_expected
else {
"items_running": 0,
"display_running": 0,
"display_claimed": 0,
"display_max_threads": 0,
}
)
queue_payload = dict((detect.get("queue_health") or {}).get("queue") or {}) if local_worker_expected else {}
local_participating = False
for node in list(cluster.get("nodes") or []):
if str(node.get("node_code") or "").strip() != settings.node_code:
@@ -558,15 +780,22 @@ def _build_runtime_projection_payload(
break
local_job_bucket = {}
if local_worker_expected:
for item in list(active_job.get("node_stats") or []):
if str(item.get("node_code") or "").strip() != settings.node_code:
for item in projection_node_rows:
if not _is_local_projection_node(str(item.get("node_code") or "").strip()):
continue
local_job_bucket = item
local_participating = local_participating or bool(
int(item.get("display_running", 0) or 0) > 0
or int(item.get("active_threads", 0) or 0) > 0
or int(item.get("items_running", 0) or 0) > 0
or int(item.get("items_claimed", 0) or 0) > 0
)
break
if local_worker_expected and not local_participating:
local_participating = bool(
int(local_job_bucket.get("items_running", 0) or 0) > 0
or int(local_job_bucket.get("items_claimed", 0) or 0) > 0
or int(display_summary.get("display_running", 0) or 0) > 0
)
projection_active_job = (
@@ -575,12 +804,18 @@ def _build_runtime_projection_payload(
"job_code": active_job.get("job_code", ""),
"status": active_job.get("status", ""),
"progress_percent": active_job.get("progress_percent", 0),
"items_total": active_job.get("items_total", 0),
"items_terminal": active_job.get("items_terminal", 0),
"items_pending": active_job.get("items_pending", 0),
"items_running": active_job.get("items_running", 0),
"items_failed": active_job.get("items_failed", 0),
"node_stats": list(active_job.get("node_stats") or []),
"items_total": int(queue_payload.get("items_total", active_job.get("items_total", 0)) or 0),
"items_terminal": int(queue_payload.get("terminal", active_job.get("items_terminal", 0)) or 0),
"items_pending": int(queue_payload.get("pending", active_job.get("items_pending", 0)) or 0),
"items_claimed": int(queue_payload.get("claimed", active_job.get("items_claimed", 0)) or 0),
"items_running": int(display_summary.get("items_running", 0) or 0),
"items_failed": int(queue_payload.get("failed", active_job.get("items_failed", 0)) or 0),
"display_items_claimed": int(display_summary.get("display_claimed", 0) or 0),
"display_items_running": int(display_summary.get("display_running", 0) or 0),
"display_active_threads": int(display_summary.get("display_running", 0) or 0),
"display_max_threads": int(display_summary.get("display_max_threads", 0) or 0),
"node_stats": list(projection_node_rows),
"distributed_node_stats": list(projection_node_rows),
}
if local_worker_expected
else {
@@ -591,18 +826,24 @@ def _build_runtime_projection_payload(
"items_total": 0,
"items_terminal": 0,
"items_pending": 0,
"items_claimed": 0,
"items_running": 0,
"items_failed": 0,
"display_items_claimed": 0,
"display_items_running": 0,
"display_active_threads": 0,
"display_max_threads": 0,
"node_stats": [],
"distributed_node_stats": [],
}
)
progress_payload = (
{
"pending": int((detect.get("progress") or {}).get("pending", 0) or 0),
"running": int((detect.get("progress") or {}).get("running", 0) or 0),
"completed": int((detect.get("progress") or {}).get("completed", 0) or 0),
"blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0),
"failed": int((detect.get("progress") or {}).get("failed", 0) or 0),
"pending": int(queue_payload.get("pending", (detect.get("progress") or {}).get("pending", 0)) or 0),
"running": int(display_summary.get("display_running", 0) or 0),
"completed": int(queue_payload.get("completed", (detect.get("progress") or {}).get("completed", 0)) or 0),
"blacklisted": int(queue_payload.get("blacklisted", (detect.get("progress") or {}).get("blacklisted", 0)) or 0),
"failed": int(queue_payload.get("failed", (detect.get("progress") or {}).get("failed", 0)) or 0),
}
if local_worker_expected
else {
@@ -624,8 +865,8 @@ def _build_runtime_projection_payload(
"worker_online": bool(detect.get("worker_online", False)) if local_worker_expected else False,
"detect_participating": local_participating if local_worker_expected else False,
"worker_mode": detect.get("worker_mode", ""),
"active_thread_count": int(detect.get("active_thread_count", 0) or 0) if local_worker_expected else 0,
"max_thread_count": int(detect.get("max_thread_count", 0) or 0) if local_worker_expected else 0,
"active_thread_count": int(display_summary.get("display_running", 0) or 0) if local_worker_expected else 0,
"max_thread_count": int(display_summary.get("display_max_threads", 0) or 0) if local_worker_expected else 0,
"phase_label": detect.get("phase_label", ""),
"phase_detail": detect.get("phase_detail", ""),
"proxy_runtime_label": detect.get("proxy_runtime_label", ""),
@@ -633,6 +874,7 @@ def _build_runtime_projection_payload(
"progress": progress_payload,
"backlog": dict(detect.get("backlog") or {}) if local_worker_expected else {},
"active_job": projection_active_job,
"cluster_nodes": list(projection_cluster_rows),
"cluster_summary": {
"nodes_total": int(cluster.get("nodes_total", 0) or 0),
"online_worker_nodes": int((cluster.get("summary") or {}).get("online_worker_nodes", 0) or 0),
@@ -670,6 +912,7 @@ def append_runtime_projection_if_changed(
) -> int | None:
normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region))
normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas"))
future_cutoff = datetime.now() + _RUNTIME_PROJECTION_FUTURE_SKEW_GRACE
payload = _build_runtime_projection_payload(
detect=detect,
cluster=cluster,
@@ -686,16 +929,17 @@ def append_runtime_projection_if_changed(
WHERE sync_type = 'runtime_projection'
AND source_region = %s
AND target_region = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
ORDER BY
CASE WHEN created_at <= %s THEN 0 ELSE 1 END ASC,
created_at DESC,
id DESC
LIMIT 200
""",
(normalized_source_region, normalized_target_region),
(normalized_source_region, normalized_target_region, future_cutoff),
)
latest = cur.fetchone()
latest = _pick_latest_projection_row(list(cur.fetchall() or []), created_at_index=1)
latest_payload = _decode_json(latest[0]) if latest else {}
latest_created_at = latest[1] if latest else None
if latest_payload.get("projection_hash") == payload["projection_hash"]:
return None
if not _should_append_runtime_projection(latest_payload, payload["projection"], latest_created_at):
return None
cur.execute(

304
domain-api/app/services/worker_control_service.py
View File

@@ -7,6 +7,8 @@ from datetime import datetime
from pathlib import Path
from uuid import uuid4
import redis
from app.core.config import settings
from app.core.redis_client import get_redis
from app.services.runtime_settings_service import get_runtime_settings
@@ -16,6 +18,101 @@ WORKER_CONTROL_CHANNEL = "domain_tool:worker_control"
WORKER_PENDING_COMMAND_KEY = "domain_tool:worker_pending_command"
def _normalize_target_node_codes(payload: dict | None) -> list[str]:
if not isinstance(payload, dict):
return []
normalized_targets: list[str] = []
def append_target(raw_value: object) -> None:
normalized_value = str(raw_value or "").strip()
if normalized_value and normalized_value not in normalized_targets:
normalized_targets.append(normalized_value)
for key in ("target_node_codes", "node_codes"):
raw_value = payload.get(key)
if isinstance(raw_value, (list, tuple, set)):
for item in raw_value:
append_target(item)
elif isinstance(raw_value, str) and raw_value.strip():
for item in raw_value.split(","):
append_target(item)
if normalized_targets:
return normalized_targets
for key in ("target_node_code", "node_code"):
raw_value = payload.get(key)
if raw_value not in (None, ""):
append_target(raw_value)
if normalized_targets:
return normalized_targets
return normalized_targets
def _pending_command_keys(command_payload: dict) -> list[str]:
target_node_codes = _normalize_target_node_codes(command_payload)
if not target_node_codes:
return [WORKER_PENDING_COMMAND_KEY]
return [f"{WORKER_PENDING_COMMAND_KEY}:{node_code}" for node_code in target_node_codes]
def _dedupe_target_node_codes(node_codes: list[str]) -> list[str]:
deduped: list[str] = []
seen: set[str] = set()
for raw_value in list(node_codes or []):
normalized_value = str(raw_value or "").strip()
if not normalized_value or normalized_value in seen:
continue
seen.add(normalized_value)
deduped.append(normalized_value)
return deduped
def _expand_local_linux_worker_target_node_codes(service_name: str) -> list[str]:
base_node_code = str(settings.node_code or "").strip()
normalized_service_name = str(service_name or "").strip()
if not base_node_code or not normalized_service_name:
return []
target_node_codes = [base_node_code]
for unit in _expand_linux_worker_control_units(normalized_service_name):
normalized_unit = str(unit or "").strip()
if not normalized_unit:
continue
if normalized_unit.endswith(".service"):
normalized_unit = normalized_unit[:-8]
if normalized_unit == normalized_service_name:
continue
template_prefix = f"{normalized_service_name}@"
if not normalized_unit.startswith(template_prefix):
continue
instance_suffix = str(normalized_unit.split("@", 1)[1] or "").strip()
if instance_suffix:
target_node_codes.append(f"{base_node_code}-{instance_suffix}")
return _dedupe_target_node_codes(target_node_codes)
def _publish_worker_command(redis_client, *, serialized: str, pending_keys: list[str]) -> None:
for key in pending_keys:
redis_client.set(key, serialized, ex=120)
redis_client.publish(WORKER_CONTROL_CHANNEL, serialized)
def _build_direct_redis_client() -> redis.Redis:
return redis.Redis(
host=settings.redis_host,
port=settings.redis_port,
password=settings.redis_password or None,
db=settings.redis_db,
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
retry_on_timeout=True,
client_name=f"domain-api-workerctl:{settings.node_code}:{os.getpid()}",
)
def _domain_root() -> Path:
return Path(settings.domain_root)
@@ -37,6 +134,125 @@ def _run_shell(command: list[str], timeout: int = 20) -> subprocess.CompletedPro
return subprocess.run(command, capture_output=True, text=True, timeout=timeout)
def _parse_process_count_output(result: subprocess.CompletedProcess[str]) -> int | None:
raw_output = (result.stdout or "").strip()
if raw_output.isdigit():
return max(0, int(raw_output or 0))
return None
def _probe_linux_worker_process_count() -> int:
probe_commands = (
(["bash", "-lc", "pgrep -fc '[d]etect_worker.py' || true"], 8),
(["bash", "-lc", "ps -eo args= | grep '[d]etect_worker.py' | wc -l"], 12),
)
for command, timeout in probe_commands:
try:
result = _run_shell(command, timeout=timeout)
except Exception:
continue
parsed_count = _parse_process_count_output(result)
if parsed_count is not None:
return parsed_count
return 0
def _probe_linux_worker_instance_count(service_name: str) -> int:
normalized_service_name = str(service_name or "").strip()
if not normalized_service_name:
return 0
template_prefix = normalized_service_name[:-8] if normalized_service_name.endswith(".service") else normalized_service_name
try:
result = _run_systemctl(
[
"list-units",
f"{template_prefix}@*",
"--type=service",
"--all",
"--no-legend",
"--plain",
],
timeout=12,
require_sudo=False,
)
except Exception:
return 0
if result.returncode != 0:
return 0
count = 0
for raw_line in (result.stdout or "").splitlines():
line = str(raw_line or "").strip()
if not line:
continue
parts = line.split()
if len(parts) < 4:
continue
if parts[2] != "active" or parts[3] != "running":
continue
count += 1
return count
def _dedupe_units(units: list[str]) -> list[str]:
deduped: list[str] = []
seen: set[str] = set()
for item in units:
normalized = str(item or "").strip()
if not normalized or normalized in seen:
continue
seen.add(normalized)
deduped.append(normalized)
return deduped
def _list_linux_worker_instance_units(service_name: str) -> list[str]:
normalized_service_name = str(service_name or "").strip()
if not normalized_service_name:
return []
template_prefix = normalized_service_name[:-8] if normalized_service_name.endswith(".service") else normalized_service_name
units: list[str] = []
try:
result = _run_systemctl(
[
"list-units",
f"{template_prefix}@*",
"--type=service",
"--all",
"--no-legend",
"--plain",
],
timeout=12,
require_sudo=False,
)
for raw_line in (result.stdout or "").splitlines():
parts = str(raw_line or "").strip().split()
if parts:
units.append(str(parts[0] or "").strip())
except Exception:
pass
managed_prefix = f"{normalized_service_name}-"
try:
for candidate in Path("/etc/default").iterdir():
if not candidate.is_file():
continue
if not candidate.name.startswith(managed_prefix):
continue
suffix = str(candidate.name[len(managed_prefix):] or "").strip()
if suffix:
units.append(f"{normalized_service_name}@{suffix}")
except Exception:
pass
return _dedupe_units(units)
def _expand_linux_worker_control_units(service_name: str) -> list[str]:
normalized_service_name = str(service_name or "").strip()
if not normalized_service_name:
return []
return _dedupe_units([normalized_service_name, *_list_linux_worker_instance_units(normalized_service_name)])
def _run_systemctl(
command: list[str],
timeout: int = 20,
@@ -192,7 +408,36 @@ def _windows_runtime() -> dict:
def _linux_runtime() -> dict:
runtime = _runtime_config()
service_name = runtime["worker_service_name"]
return probe_systemd_service(service_name, mode="linux-systemd")
service_probe = probe_systemd_service(service_name, mode="linux-systemd")
instance_count = _probe_linux_worker_instance_count(service_name)
process_count = _probe_linux_worker_process_count()
if process_count <= 0 and instance_count <= 0:
return service_probe
latest_start_time = str(service_probe.get("latest_start_time") or "").strip()
if service_probe.get("running", False) or instance_count > 0:
message = service_probe.get("message") or f"multi-instance active ({process_count})"
if instance_count > 0 and not service_probe.get("running", False):
message = f"template instances active ({instance_count})"
return {
**service_probe,
"running": True,
"process_count": process_count,
"latest_start_time": latest_start_time,
"message": message,
}
message = str(service_probe.get("message") or "").strip()
if message:
message = f"{message}; detected {process_count} unmanaged worker processes"
else:
message = f"detected {process_count} unmanaged worker processes"
return {
**service_probe,
"process_count": process_count,
"latest_start_time": latest_start_time,
"message": message,
}
def detect_sync_agent_runtime() -> dict:
@@ -232,9 +477,13 @@ def start_worker() -> tuple[bool, str]:
worker_mode = runtime["worker_mode"]
service_name = runtime["worker_service_name"]
if worker_mode == "linux-systemd":
result = _run_systemctl(["start", service_name], timeout=30)
units = _expand_linux_worker_control_units(service_name)
result = _run_systemctl(["start", *units], timeout=max(30, 15 * max(1, len(units))))
if result.returncode != 0:
return False, normalize_systemctl_error(result.stderr or result.stdout or "启动 Linux Worker 失败", service_name=service_name)
extra_units = max(0, len(units) - 1)
if extra_units > 0:
return True, f"Linux Worker 启动命令已发送: {service_name}，附带 {extra_units} 个实例"
return True, f"Linux Worker 启动命令已发送: {service_name}"
if os.name != "nt":
@@ -260,9 +509,13 @@ def stop_worker() -> tuple[bool, str]:
worker_mode = runtime["worker_mode"]
service_name = runtime["worker_service_name"]
if worker_mode == "linux-systemd":
result = _run_systemctl(["stop", service_name], timeout=30)
units = _expand_linux_worker_control_units(service_name)
result = _run_systemctl(["stop", *units], timeout=max(30, 15 * max(1, len(units))))
if result.returncode != 0:
return False, normalize_systemctl_error(result.stderr or result.stdout or "停止 Linux Worker 失败", service_name=service_name)
extra_units = max(0, len(units) - 1)
if extra_units > 0:
return True, f"Linux Worker 停止命令已发送: {service_name}，附带 {extra_units} 个实例"
return True, f"Linux Worker 停止命令已发送: {service_name}"
if os.name != "nt":
@@ -289,15 +542,42 @@ def stop_worker() -> tuple[bool, str]:
def send_worker_command(action: str, payload: dict | None = None) -> tuple[bool, str]:
command_payload = {"action": action}
if payload:
command_payload.update(payload)
runtime = _runtime_config()
explicit_targets = _normalize_target_node_codes(command_payload)
if not explicit_targets and str(runtime.get("worker_mode") or "").strip() == "linux-systemd":
expanded_targets = _expand_local_linux_worker_target_node_codes(
str(runtime.get("worker_service_name") or "").strip()
)
if expanded_targets:
command_payload["target_node_codes"] = expanded_targets
command_payload["request_id"] = str(command_payload.get("request_id") or f"workerctl-{uuid4().hex[:12]}")
serialized = json.dumps(command_payload, ensure_ascii=False)
pending_keys = _pending_command_keys(command_payload)
target_node_codes = _normalize_target_node_codes(command_payload)
direct_client = None
try:
redis_client = get_redis()
command_payload = {"action": action}
if payload:
command_payload.update(payload)
command_payload["request_id"] = str(command_payload.get("request_id") or f"workerctl-{uuid4().hex[:12]}")
serialized = json.dumps(command_payload, ensure_ascii=False)
redis_client.set(WORKER_PENDING_COMMAND_KEY, serialized, ex=120)
redis_client.publish(WORKER_CONTROL_CHANNEL, serialized)
return True, f"已发送 Worker 控制指令: {action}"
try:
_publish_worker_command(get_redis(), serialized=serialized, pending_keys=pending_keys)
except Exception:
direct_client = _build_direct_redis_client()
_publish_worker_command(direct_client, serialized=serialized, pending_keys=pending_keys)
if pending_keys == [WORKER_PENDING_COMMAND_KEY]:
return True, f"已发送 Worker 控制指令: {action}"
if len(target_node_codes) <= 4:
target_summary = ",".join(target_node_codes)
else:
target_summary = f"{len(target_node_codes)} targets"
return True, f"已发送 Worker 控制指令: {action} -> {target_summary}"
except Exception as exc:
return False, f"发送 Worker 控制指令失败: {exc}"
finally:
if direct_client is not None:
try:
direct_client.close()
except Exception:
pass

405
domain-api/app/sync_agent.py
View File

@@ -1,8 +1,10 @@
from __future__ import annotations
import logging
import os
import time
from app.core.db import db_read_retry, get_db
from app.core.config import settings
from app.services.debug_event_service import push_debug_event
from app.services.detect_job_service import (
@@ -19,6 +21,10 @@ from app.services.sync_push_service import (
pull_detect_task_batch_now,
push_runtime_projection_now,
)
from app.services.worker_control_service import (
_expand_local_linux_worker_target_node_codes,
send_worker_command,
)
logger = logging.getLogger("domaincheck.sync_agent")
@@ -33,6 +39,9 @@ _IDLE_SYNC_KEYWORDS = (
"暂停拉取",
)
_LAST_OVERLAP_JOB_ID = 0
_LAST_OVERLAP_TRIGGERED_AT = 0.0
def _append_detect_result_projection_snapshot(active_job: dict) -> None:
if not active_job:
@@ -278,6 +287,196 @@ def _emit_sync_result_breakdown(data: dict | None) -> None:
)
def _bool_env(name: str, default: bool) -> bool:
raw = str(os.getenv(name, "1" if default else "0") or ("1" if default else "0")).strip().lower()
return raw not in {"0", "false", "off", "no"}
def _int_env(name: str, default: int, *, minimum: int = 0, maximum: int | None = None) -> int:
try:
value = int(os.getenv(name, str(default)) or default)
except (TypeError, ValueError):
value = int(default)
value = max(minimum, value)
if maximum is not None:
value = min(maximum, value)
return value
@db_read_retry(attempts=3, initial_delay_seconds=0.05, backoff=2.0)
def _select_overlap_start_candidate() -> dict | None:
if settings.node_region != "mainland" or settings.node_role != "control":
return None
if not _bool_env("DOMAINCHECK_OVERLAP_HANDOFF_ENABLED", True):
return None
min_running_jobs = _int_env("DOMAINCHECK_OVERLAP_HANDOFF_MIN_RUNNING_JOBS", 1, minimum=1, maximum=16)
running_age_seconds = _int_env("DOMAINCHECK_OVERLAP_HANDOFF_MIN_RUNNING_AGE_SECONDS", 300, minimum=30, maximum=7200)
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
WITH running_state AS (
SELECT
COUNT(*) AS running_jobs,
COALESCE(
MAX(
EXTRACT(
EPOCH FROM (CURRENT_TIMESTAMP - COALESCE(job.started_at, job.created_at))
)
),
0
) AS max_running_age_seconds
FROM detect_jobs AS job
WHERE job.status = 'running'
),
pending_candidate AS (
SELECT
job.id,
job.job_code,
job.task_mode,
COALESCE(job.started_at, job.created_at) AS activity_at
FROM detect_jobs AS job
WHERE job.status = 'pending'
ORDER BY
COALESCE(job.started_at, job.created_at) DESC,
job.id DESC
LIMIT 1
)
SELECT
pending.id,
pending.job_code,
pending.task_mode,
running_state.running_jobs,
running_state.max_running_age_seconds,
'overlap_tail_handoff' AS selection_reason
FROM pending_candidate AS pending
CROSS JOIN running_state
WHERE running_state.running_jobs >= %s
AND running_state.max_running_age_seconds >= %s
""",
(
min_running_jobs,
running_age_seconds,
),
)
row = cur.fetchone()
if not row:
return None
return {
"job_id": int(row[0] or 0),
"job_code": str(row[1] or "").strip(),
"task_mode": str(row[2] or "domain_pipeline").strip() or "domain_pipeline",
"running_jobs": int(row[3] or 0),
"max_running_age_seconds": int(float(row[4] or 0) or 0),
"selection_reason": str(row[5] or "overlap_tail_handoff").strip() or "overlap_tail_handoff",
}
@db_read_retry(attempts=3, initial_delay_seconds=0.05, backoff=2.0)
def _select_overlap_target_node_codes() -> list[str]:
if settings.node_region != "mainland" or settings.node_role != "control":
return []
target_limit = _int_env("DOMAINCHECK_OVERLAP_HANDOFF_TARGETS", 12, minimum=1, maximum=32)
target_scan_limit = _int_env(
"DOMAINCHECK_OVERLAP_HANDOFF_TARGET_SCAN_LIMIT",
max(target_limit * 4, 16),
minimum=target_limit,
maximum=256,
)
max_current_load = _int_env("DOMAINCHECK_OVERLAP_HANDOFF_TARGET_MAX_CURRENT_LOAD", 0, minimum=0, maximum=4096)
expanded_targets = _expand_local_linux_worker_target_node_codes(str(settings.worker_service_name or "").strip())
if not expanded_targets:
return []
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT node_code, current_load
FROM detect_worker_nodes
WHERE node_code = ANY(%s)
ORDER BY
CASE WHEN COALESCE(current_load, 0) <= %s THEN 0 ELSE 1 END ASC,
COALESCE(current_load, 0) ASC,
COALESCE(update_time, last_heartbeat_at) DESC,
node_code ASC
LIMIT %s
""",
(
expanded_targets,
max_current_load,
target_scan_limit,
),
)
rows = list(cur.fetchall() or [])
preferred: list[str] = []
fallback: list[str] = []
seen: set[str] = set()
for raw_node_code, raw_current_load in rows:
node_code = str(raw_node_code or "").strip()
if not node_code or node_code in seen:
continue
seen.add(node_code)
fallback.append(node_code)
if int(raw_current_load or 0) <= max_current_load:
preferred.append(node_code)
ordered = preferred + [node_code for node_code in fallback if node_code not in set(preferred)]
if not ordered:
ordered = list(expanded_targets or [])
return ordered[:target_limit]
def _maybe_trigger_overlap_start() -> tuple[bool, str, dict]:
global _LAST_OVERLAP_JOB_ID, _LAST_OVERLAP_TRIGGERED_AT
candidate = _select_overlap_start_candidate()
if not candidate:
return False, "当前没有满足尾盘接棒条件的 pending job", {}
cooldown_seconds = _int_env("DOMAINCHECK_OVERLAP_HANDOFF_COOLDOWN_SECONDS", 120, minimum=15, maximum=1800)
job_id = int(candidate.get("job_id") or 0)
now_ts = time.time()
if (
job_id > 0
and job_id == _LAST_OVERLAP_JOB_ID
and now_ts - float(_LAST_OVERLAP_TRIGGERED_AT or 0.0) < cooldown_seconds
):
return False, f"overlap handoff 冷却中: job_id={job_id}", candidate
target_node_codes = _select_overlap_target_node_codes()
if not target_node_codes:
return False, f"overlap handoff 未找到可接棒 worker: job_id={job_id}", candidate
payload = {
"job_id": job_id,
"job_code": str(candidate.get("job_code") or "").strip(),
"target_job_id": job_id,
"target_job_code": str(candidate.get("job_code") or "").strip(),
"task_mode": str(candidate.get("task_mode") or "domain_pipeline").strip() or "domain_pipeline",
"source": "overlap-handoff",
"selection_reason": str(candidate.get("selection_reason") or "overlap_tail_handoff").strip() or "overlap_tail_handoff",
"tail_handoff_candidate": True,
"target_node_codes": target_node_codes,
}
ok, message = send_worker_command("start_detection", payload=payload)
if ok and job_id > 0:
_LAST_OVERLAP_JOB_ID = job_id
_LAST_OVERLAP_TRIGGERED_AT = now_ts
push_debug_event(
service="sync-agent",
event_type="overlap_handoff_started",
level="warning",
message=message,
payload={**candidate, "cooldown_seconds": cooldown_seconds, "target_node_codes": target_node_codes},
)
return ok, message, candidate
def _run_pipeline_stage_processor() -> tuple[bool, str, dict]:
process_limit = max(500, min(int(settings.sync_pipeline_process_limit or 5000), 5000))
ok, message, data = process_detect_pipeline_now(limit=process_limit)
@@ -295,6 +494,94 @@ def _run_pipeline_stage_processor() -> tuple[bool, str, dict]:
return ok, message, data
def _emit_runtime_debug_snapshots() -> None:
active_job = get_active_detect_job_summary(event_limit=10)
for projection_job in _select_projection_job_snapshots():
_append_detect_result_projection_snapshot(projection_job)
if not active_job:
return
queue_health = _build_aligned_queue_health_snapshot(
active_job,
get_detect_queue_health(window_minutes=15),
)
recent_events = _filter_runtime_events_for_job(
list_recent_detect_run_events(limit=24),
job_code=str(active_job.get("runtime_job_code") or active_job.get("job_code") or "").strip(),
job_id=int(active_job.get("job_id", 0) or 0),
limit=8,
)
push_debug_event(
service="detect-runtime",
event_type="active_job_snapshot",
level="info",
message=f"active job {active_job.get('job_code', '')} status={active_job.get('status', '')}",
payload={
"job": {
"job_id": active_job.get("job_id"),
"job_code": active_job.get("job_code", ""),
"status": active_job.get("status", ""),
"items_total": active_job.get("items_total", 0),
"items_pending": active_job.get("items_pending", 0),
"items_claimed": active_job.get("items_claimed", 0),
"items_running": active_job.get("items_running", 0),
"items_completed": active_job.get("items_completed", 0),
"items_failed": active_job.get("items_failed", 0),
"progress_percent": active_job.get("progress_percent", 0),
"node_stats": list(active_job.get("node_stats") or []),
},
"queue_health": queue_health,
"backlog": _load_local_detect_backlog_snapshot(),
"recent_events": recent_events,
},
)
if queue_health.get("queue", {}).get("overdue_leases", 0):
push_debug_event(
service="detect-runtime",
event_type="queue_overdue_leases",
level="warning",
message=f"检测队列存在过期租约 {queue_health.get('queue', {}).get('overdue_leases', 0)} 个",
payload=queue_health,
)
for event in recent_events:
event_type = str(event.get("event_type") or "").strip()
if event_type not in {"domain_started", "domain_completed", "domain_failed", "domain_blacklisted"}:
continue
push_debug_event(
service="worker-event",
event_type=event_type,
level=str(event.get("level") or "info"),
message=str(event.get("message") or "").strip(),
payload={
"job_id": event.get("job_id"),
"node_code": event.get("node_code", ""),
"created_at": event.get("created_at", ""),
**(event.get("payload") or {}),
},
)
def _run_sync_tick_once() -> dict:
overlap_ok, overlap_message, overlap_data = _maybe_trigger_overlap_start()
sync_ok, sync_message, sync_data = push_runtime_projection_now()
pull_ok, pull_message, pull_data = pull_detect_task_batch_now()
pipeline_ok, pipeline_message, pipeline_data = _run_pipeline_stage_processor()
try:
_emit_runtime_debug_snapshots()
debug_snapshot_error = ""
except Exception as debug_exc: # pragma: no cover - logged by caller path
logger.warning("runtime debug snapshot skipped: %s", debug_exc)
debug_snapshot_error = str(debug_exc)
return {
"sync": {"ok": sync_ok, "message": sync_message, "data": sync_data},
"pull": {"ok": pull_ok, "message": pull_message, "data": pull_data},
"pipeline": {"ok": pipeline_ok, "message": pipeline_message, "data": pipeline_data},
"overlap": {"ok": overlap_ok, "message": overlap_message, "data": overlap_data},
"debug_snapshot_error": debug_snapshot_error,
}
def main() -> None:
logging.basicConfig(
level=logging.INFO,
@@ -313,28 +600,24 @@ def main() -> None:
)
while True:
try:
pipeline_ok, pipeline_message, pipeline_data = _run_pipeline_stage_processor()
logger.info(
"pipeline tick: ok=%s message=%s data=%s",
pipeline_ok,
pipeline_message,
pipeline_data,
)
active_job = get_active_detect_job_summary(event_limit=10)
for projection_job in _select_projection_job_snapshots():
_append_detect_result_projection_snapshot(projection_job)
ok, message, data = push_runtime_projection_now()
logger.info("sync tick: ok=%s message=%s data=%s", ok, message, data)
tick = _run_sync_tick_once()
sync_ok = bool((tick.get("sync") or {}).get("ok"))
sync_message = str((tick.get("sync") or {}).get("message") or "")
sync_data = (tick.get("sync") or {}).get("data") or {}
logger.info("sync tick: ok=%s message=%s data=%s", sync_ok, sync_message, sync_data)
push_debug_event(
service="sync-agent",
event_type="sync_tick",
level="info" if ok else "warning",
message=message,
payload={"ok": ok, "data": data},
level="info" if sync_ok else "warning",
message=sync_message,
payload={"ok": sync_ok, "data": sync_data},
)
_emit_structured_tick(base_event_type="sync_push", ok=ok, message=message, data=data)
_emit_sync_result_breakdown(data)
pull_ok, pull_message, pull_data = pull_detect_task_batch_now()
_emit_structured_tick(base_event_type="sync_push", ok=sync_ok, message=sync_message, data=sync_data)
_emit_sync_result_breakdown(sync_data)
pull_ok = bool((tick.get("pull") or {}).get("ok"))
pull_message = str((tick.get("pull") or {}).get("message") or "")
pull_data = (tick.get("pull") or {}).get("data") or {}
logger.info("task pull tick: ok=%s message=%s data=%s", pull_ok, pull_message, pull_data)
push_debug_event(
service="sync-agent",
@@ -344,65 +627,33 @@ def main() -> None:
payload={"ok": pull_ok, "data": pull_data},
)
_emit_structured_tick(base_event_type="task_pull", ok=pull_ok, message=pull_message, data=pull_data)
if active_job:
queue_health = _build_aligned_queue_health_snapshot(
active_job,
get_detect_queue_health(window_minutes=15),
)
recent_events = _filter_runtime_events_for_job(
list_recent_detect_run_events(limit=24),
job_code=str(active_job.get("runtime_job_code") or active_job.get("job_code") or "").strip(),
job_id=int(active_job.get("job_id", 0) or 0),
limit=8,
)
push_debug_event(
service="detect-runtime",
event_type="active_job_snapshot",
level="info",
message=f"active job {active_job.get('job_code', '')} status={active_job.get('status', '')}",
payload={
"job": {
"job_id": active_job.get("job_id"),
"job_code": active_job.get("job_code", ""),
"status": active_job.get("status", ""),
"items_total": active_job.get("items_total", 0),
"items_pending": active_job.get("items_pending", 0),
"items_claimed": active_job.get("items_claimed", 0),
"items_running": active_job.get("items_running", 0),
"items_completed": active_job.get("items_completed", 0),
"items_failed": active_job.get("items_failed", 0),
"progress_percent": active_job.get("progress_percent", 0),
"node_stats": list(active_job.get("node_stats") or []),
},
"queue_health": queue_health,
"backlog": _load_local_detect_backlog_snapshot(),
"recent_events": recent_events,
},
)
if queue_health.get("queue", {}).get("overdue_leases", 0):
push_debug_event(
service="detect-runtime",
event_type="queue_overdue_leases",
level="warning",
message=f"检测队列存在过期租约 {queue_health.get('queue', {}).get('overdue_leases', 0)} 个",
payload=queue_health,
)
for event in recent_events:
event_type = str(event.get("event_type") or "").strip()
if event_type not in {"domain_started", "domain_completed", "domain_failed", "domain_blacklisted"}:
continue
push_debug_event(
service="worker-event",
event_type=event_type,
level=str(event.get("level") or "info"),
message=str(event.get("message") or "").strip(),
payload={
"job_id": event.get("job_id"),
"node_code": event.get("node_code", ""),
"created_at": event.get("created_at", ""),
**(event.get("payload") or {}),
},
)
pipeline_ok = bool((tick.get("pipeline") or {}).get("ok"))
pipeline_message = str((tick.get("pipeline") or {}).get("message") or "")
pipeline_data = (tick.get("pipeline") or {}).get("data") or {}
logger.info(
"pipeline tick: ok=%s message=%s data=%s",
pipeline_ok,
pipeline_message,
pipeline_data,
)
overlap_ok = bool((tick.get("overlap") or {}).get("ok"))
overlap_message = str((tick.get("overlap") or {}).get("message") or "")
overlap_data = (tick.get("overlap") or {}).get("data") or {}
logger.info(
"overlap tick: ok=%s message=%s data=%s",
overlap_ok,
overlap_message,
overlap_data,
)
push_debug_event(
service="sync-agent",
event_type="overlap_tick",
level="info" if overlap_ok else "warning",
message=overlap_message,
payload={"ok": overlap_ok, "data": overlap_data},
)
except Exception as exc:
logger.exception("sync tick failed: %s", exc)
push_debug_event(

28
domain-api/deploy/linux/smoke_test.py
View File

@@ -8,7 +8,7 @@ from datetime import datetime
import requests
def _fetch_json(session: requests.Session, url: str, timeout: int = 15) -> tuple[bool, str, dict | None]:
def _fetch_json(session: requests.Session, url: str, timeout: int = 25) -> tuple[bool, str, dict | None]:
try:
response = session.get(url, timeout=timeout)
response.raise_for_status()
@@ -45,20 +45,22 @@ def main() -> int:
checks: list[dict[str, object]] = []
endpoints = [
("health", f"{base_url}/health"),
("runtime_status", f"{base_url}/api/v1/runtime/status"),
("runtime_preflight", f"{base_url}/api/v1/runtime/preflight"),
("dashboard_overview", f"{base_url}/api/v1/dashboard/overview"),
("settings_export", f"{base_url}/api/v1/settings/export"),
("settings_backups", f"{base_url}/api/v1/settings/backups"),
("detect_status", f"{base_url}/api/v1/detect/status"),
("imports_summary", f"{base_url}/api/v1/imports/summary"),
("exports", f"{base_url}/api/v1/exports"),
("logs_latest", f"{base_url}/api/v1/logs/latest"),
("health", f"{base_url}/health", 25),
("runtime_status", f"{base_url}/api/v1/runtime/status", 25),
("runtime_preflight", f"{base_url}/api/v1/runtime/preflight", 25),
# overview merges multiple heavy runtime aggregates and can legitimately
# take longer than lightweight health/readiness endpoints on live load.
("dashboard_overview", f"{base_url}/api/v1/dashboard/overview", 25),
("settings_export", f"{base_url}/api/v1/settings/export", 25),
("settings_backups", f"{base_url}/api/v1/settings/backups", 25),
("detect_status", f"{base_url}/api/v1/detect/status", 25),
("imports_summary", f"{base_url}/api/v1/imports/summary", 25),
("exports", f"{base_url}/api/v1/exports", 25),
("logs_latest", f"{base_url}/api/v1/logs/latest", 25),
]
for name, url in endpoints:
ok, message, payload = _fetch_json(session, url)
for name, url, timeout in endpoints:
ok, message, payload = _fetch_json(session, url, timeout=timeout)
checks.append(
{
"name": name,

27
domain-api/deploy/multi-region/README.md
View File

@@ -39,6 +39,7 @@
- `domain-api/deploy/multi-region/drive_ops_action.sh`
- `domain-api/deploy/multi-region/templates/domaincheck-node-agent.env.example`
- `domain-api/deploy/multi-region/templates/domaincheck-ops-center.env.example`
- `domain-api/deploy/multi-region/templates/domaincheck-worker.instance.env.example`
- `domain-api/deploy/multi-region/check_cluster.sh`
- `domain-api/deploy/multi-region/check_node_agent.sh`
- `domain-api/deploy/multi-region/check_ops_center_stack.sh`
@@ -218,6 +219,32 @@ bash domain-api/deploy/multi-region/drive_ops_center.sh go-live-recover
- `runtime/build-info`
- 一份最终 `status / headline / missing_items / tooling_items` 收口摘要
## 一点六、单机多进程 Worker 起步方式
如果大陆执行机是高核大内存机器，而单个 `detect_worker.py` 进程还吃不满机器，可以先不要继续堆单进程线程数，优先改成“同机多实例 Worker”：
```bash
# 1. 复制实例模板，按实例编号准备独立 env
cp domain-api/deploy/multi-region/templates/domaincheck-worker.instance.env.example /etc/default/domaincheck-worker-a
cp domain-api/deploy/multi-region/templates/domaincheck-worker.instance.env.example /etc/default/domaincheck-worker-b
# 2. 为每个实例设置不同的 NODE_CODE
# mainland-controller-01-a
# mainland-controller-01-b
# 3. 启用 systemd 模板实例
cp domain-api/deploy/systemd/domain-worker@.service /etc/systemd/system/domaincheck-worker@.service
systemctl daemon-reload
systemctl enable --now domaincheck-worker@a
systemctl enable --now domaincheck-worker@b
```
注意：
- 同一台机器上的每个实例必须使用不同的 `NODE_CODE`
- 每个实例建议先用中等线程数压测，不要直接把单进程线程数拉到极限
- 这版控制指令已经支持按 `NODE_CODE` 定向，不同实例不会再一起响应同一条本地 Worker 指令
说明：
- `missing_items` 只保留真正影响部署或运行面的缺口

3
domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example
View File

@@ -1,5 +1,8 @@
WORKER_MODE=linux-systemd
QT_QPA_PLATFORM=offscreen
# 单实例控制节点参考。若同一台大陆大机需要起多个 Worker 进程，
# 请改用 domaincheck-worker.instance.env.example + domain-worker@.service。
NODE_CODE=mainland-controller-01
NODE_REGION=mainland
NODE_ROLE=control

40
domain-api/deploy/multi-region/templates/domaincheck-worker.instance.env.example Normal file
View File

@@ -0,0 +1,40 @@
WORKER_MODE=linux-systemd
QT_QPA_PLATFORM=offscreen
# 对应 systemd 模板:
# cp domaincheck-worker.instance.env.example /etc/default/domaincheck-worker-a
# cp domain-api/deploy/systemd/domain-worker@.service /etc/systemd/system/domaincheck-worker@.service
# systemctl daemon-reload
# systemctl enable --now domaincheck-worker@a
#
# 同一台机器上起多个 worker 实例时，每个实例必须使用不同的 NODE_CODE，
# 否则运行态、控制指令和集群心跳会互相覆盖。
NODE_CODE=mainland-controller-01-a
WORKER_PARENT_NODE_CODE=mainland-controller-01
NODE_REGION=mainland
NODE_ROLE=control
# DB
# DB_HOST=127.0.0.1
# DB_PORT=5432
# DB_DATABASE=domain
# DB_USER=postgres
# DB_PASSWORD=postgres
# REDIS
# REDIS_HOST=127.0.0.1
# REDIS_PORT=6379
# REDIS_PASSWORD=
# REDIS_DB=0
# SYNC
SYNC_PUSH_ENABLED=true
SYNC_SOURCE_REGION=mainland
SYNC_TARGET_REGION=overseas
SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
SYNC_SHARED_TOKEN=
SYNC_BATCH_SIZE=5000
SYNC_POLL_INTERVAL_SECONDS=2
# 第一版建议每实例先用中等线程数压测，不要直接把单进程线程拉到极限。
# THREAD_COUNT=1000

3
domain-api/deploy/systemd/domain-api.service
View File

@@ -11,9 +11,10 @@ Environment="WORKER_MODE=linux-systemd"
Environment="API_HOST=0.0.0.0"
Environment="API_PORT=8100"
Environment="DOMAIN_ROOT=/opt/domaincheck/domainCheck"
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100 --timeout-graceful-shutdown 15
Restart=always
RestartSec=5
TimeoutStopSec=20
User=www
Group=www

6
domain-api/deploy/systemd/domain-worker.service
View File

@@ -4,13 +4,15 @@ After=network.target redis.service postgresql.service
[Service]
Type=simple
WorkingDirectory=/opt/domaincheck/domainCheck
WorkingDirectory=/opt/domaincheck/current/domainCheck
EnvironmentFile=-/etc/default/domaincheck-worker
Environment="WORKER_MODE=linux-systemd"
Environment="QT_QPA_PLATFORM=offscreen"
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python /opt/domaincheck/domainCheck/detect_worker.py
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python /opt/domaincheck/current/domainCheck/detect_worker.py
Restart=always
RestartSec=5
LimitNOFILE=65535
TasksMax=infinity
User=www
Group=www

21
domain-api/deploy/systemd/domain-worker@.service Normal file
View File

@@ -0,0 +1,21 @@
[Unit]
Description=domainCheck Worker Instance %i
After=network.target redis.service postgresql.service
[Service]
Type=simple
WorkingDirectory=/opt/domaincheck/current/domainCheck
EnvironmentFile=-/etc/default/domaincheck-worker-%i
Environment="WORKER_MODE=linux-systemd"
Environment="QT_QPA_PLATFORM=offscreen"
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python /opt/domaincheck/current/domainCheck/detect_worker.py
Restart=always
RestartSec=5
LimitNOFILE=65535
TasksMax=infinity
SyslogIdentifier=domaincheck-worker@%i
User=www
Group=www
[Install]
WantedBy=multi-user.target

3
domain-api/tests/test_build_info_service.py
View File

@@ -81,6 +81,9 @@ class BuildInfoServiceTests(unittest.TestCase):
self.assertEqual("env-sha256", payload["checksum"])
self.assertFalse(payload["route_surface"]["surface_complete"])
self.assertIn("ops_stack_diagnosis", payload["route_surface"]["missing_keys"])
self.assertIn("ops_migration_source_profile", payload["route_surface"]["missing_keys"])
self.assertIn("ops_migration_preview", payload["route_surface"]["missing_keys"])
self.assertIn("ops_migration_execute", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_bootstrap_preview", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_bootstrap_execute", payload["route_surface"]["missing_keys"])
self.assertIn("ops_node_onboarding_acceptance_preview", payload["route_surface"]["missing_keys"])

352
domain-api/tests/test_cluster_runtime_service.py
View File

@@ -1,12 +1,65 @@
from __future__ import annotations
import unittest
from datetime import datetime, timedelta
from unittest.mock import MagicMock, patch
from psycopg2 import errors
from app.services import cluster_runtime_service
class RuntimeSchemaGuardTests(unittest.TestCase):
def test_ensure_runtime_schema_does_not_toggle_autocommit_after_queries(self) -> None:
class _Connection:
def __init__(self) -> None:
self._autocommit = False
self.touched = False
self.committed = False
@property
def autocommit(self):
return self._autocommit
@autocommit.setter
def autocommit(self, value):
if self.touched:
raise AssertionError("autocommit should not be reassigned after queries start")
self._autocommit = value
def cursor(self):
conn = self
class _CursorContext:
def __enter__(self_inner):
conn.touched = True
return cursor
def __exit__(self_inner, exc_type, exc, tb):
return False
return _CursorContext()
def commit(self):
self.committed = True
cursor = MagicMock()
conn = _Connection()
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
with patch.object(cluster_runtime_service, "_RUNTIME_SCHEMA_READY", False):
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm):
with patch.object(cluster_runtime_service, "_runtime_schema_basics_present", return_value=False):
cluster_runtime_service.ensure_runtime_schema()
cursor.execute.assert_any_call(
"SELECT pg_advisory_xact_lock(%s)",
(cluster_runtime_service._RUNTIME_SCHEMA_ADVISORY_LOCK_ID,),
)
cursor.execute.assert_any_call(cluster_runtime_service._RUNTIME_SCHEMA_SQL)
self.assertTrue(conn.committed)
def test_ensure_runtime_schema_executes_only_once_per_process(self) -> None:
conn = MagicMock()
cursor_cm = MagicMock()
@@ -18,11 +71,11 @@ class RuntimeSchemaGuardTests(unittest.TestCase):
with patch.object(cluster_runtime_service, "_RUNTIME_SCHEMA_READY", False):
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm) as mocked_get_db:
cluster_runtime_service.ensure_runtime_schema()
cluster_runtime_service.ensure_runtime_schema()
with patch.object(cluster_runtime_service, "_runtime_schema_basics_present", return_value=False):
cluster_runtime_service.ensure_runtime_schema()
cluster_runtime_service.ensure_runtime_schema()
mocked_get_db.assert_called_once()
self.assertEqual(cursor.execute.call_count, 2)
cursor.execute.assert_any_call(
"SELECT pg_advisory_xact_lock(%s)",
(cluster_runtime_service._RUNTIME_SCHEMA_ADVISORY_LOCK_ID,),
@@ -30,6 +83,118 @@ class RuntimeSchemaGuardTests(unittest.TestCase):
cursor.execute.assert_any_call(cluster_runtime_service._RUNTIME_SCHEMA_SQL)
conn.commit.assert_called_once()
def test_ensure_runtime_schema_skips_ddl_when_required_schema_already_exists(self) -> None:
conn = MagicMock()
cursor_cm = MagicMock()
cursor = MagicMock()
conn.cursor.return_value = cursor_cm
cursor_cm.__enter__.return_value = cursor
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
with patch.object(cluster_runtime_service, "_RUNTIME_SCHEMA_READY", False):
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm):
with patch.object(cluster_runtime_service, "_runtime_schema_basics_present", return_value=True):
with patch.object(cluster_runtime_service, "_runtime_missing_indexes", return_value=iter(())):
cluster_runtime_service.ensure_runtime_schema()
self.assertFalse(
any(sql == cluster_runtime_service._RUNTIME_SCHEMA_SQL for sql, _params in cursor.execute.call_args_list)
)
conn.commit.assert_not_called()
def test_ensure_runtime_schema_accepts_deadlock_when_required_schema_already_exists(self) -> None:
class _Cursor:
def __init__(self, *, raise_on_schema=False, fetchone_values=None, fetchall_values=None) -> None:
self.raise_on_schema = raise_on_schema
self.fetchone_values = list(fetchone_values or [])
self.fetchall_values = list(fetchall_values or [])
self.executed = []
def execute(self, sql, params=None):
self.executed.append((sql, params))
if self.raise_on_schema and sql == cluster_runtime_service._RUNTIME_SCHEMA_SQL:
raise errors.DeadlockDetected()
def fetchone(self):
if self.fetchone_values:
return self.fetchone_values.pop(0)
return None
def fetchall(self):
if self.fetchall_values:
return self.fetchall_values.pop(0)
return []
class _CursorContext:
def __init__(self, cursor) -> None:
self.cursor = cursor
def __enter__(self):
return self.cursor
def __exit__(self, exc_type, exc, tb):
return False
conn = MagicMock()
first_cursor = _Cursor()
second_cursor = _Cursor(raise_on_schema=True)
third_cursor = _Cursor()
conn.cursor.side_effect = [
_CursorContext(first_cursor),
_CursorContext(second_cursor),
_CursorContext(third_cursor),
]
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
with patch.object(cluster_runtime_service, "_RUNTIME_SCHEMA_READY", False):
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm):
with patch.object(cluster_runtime_service, "_runtime_schema_basics_present", side_effect=[False, True]):
with patch.object(cluster_runtime_service, "_runtime_missing_indexes", return_value=iter(())):
cluster_runtime_service.ensure_runtime_schema()
conn.rollback.assert_called_once()
conn.commit.assert_not_called()
def test_ensure_runtime_schema_repairs_missing_indexes_without_full_ddl(self) -> None:
conn = MagicMock()
cursor_cm = MagicMock()
cursor = MagicMock()
conn.cursor.return_value = cursor_cm
cursor_cm.__enter__.return_value = cursor
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
with patch.object(cluster_runtime_service, "_RUNTIME_SCHEMA_READY", False):
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm):
with patch.object(cluster_runtime_service, "_runtime_schema_basics_present", return_value=True):
with patch.object(
cluster_runtime_service,
"_runtime_missing_indexes",
side_effect=[iter(("idx_detect_job_items_claim_step_ready",)), iter(())],
):
with patch.object(cluster_runtime_service, "_ensure_runtime_schema_indexes") as mocked_ensure_indexes:
cluster_runtime_service.ensure_runtime_schema()
mocked_ensure_indexes.assert_called_once()
self.assertFalse(
any(sql == cluster_runtime_service._RUNTIME_SCHEMA_SQL for sql, _params in cursor.execute.call_args_list)
)
def test_runtime_missing_indexes_treats_invalid_indexes_as_missing(self) -> None:
cur = MagicMock()
cur.fetchall.return_value = [
("idx_detect_job_items_job_domain_step", True, True, True),
("idx_detect_job_items_claim_step_ready", False, True, True),
]
missing = list(cluster_runtime_service._runtime_missing_indexes(cur))
self.assertIn("idx_detect_job_items_claim_step_ready", missing)
self.assertIn("idx_detect_job_items_claim_job_step_ready", missing)
self.assertNotIn("idx_detect_job_items_job_domain_step", missing)
def test_control_node_supports_worker_only_on_mainland_with_worker_signals(self) -> None:
self.assertFalse(
cluster_runtime_service._control_node_supports_worker(
@@ -88,6 +253,187 @@ class RuntimeSchemaGuardTests(unittest.TestCase):
)
)
def test_get_cluster_snapshot_counts_all_rows_even_when_display_nodes_are_limited(self) -> None:
now = datetime.now()
display_rows = [
(
f"mainland-worker-{index:03d}",
"mainland",
"worker",
f"worker-{index:03d}",
f"10.0.0.{index}",
"online",
"test",
0,
{},
now,
now,
)
for index in range(1, 101)
]
summary_rows = display_rows + [
(
"mainland-worker-101",
"mainland",
"worker",
"worker-101",
"10.0.0.101",
"online",
"test",
0,
{},
now,
now,
)
]
conn = MagicMock()
cursor_cm = MagicMock()
cursor = MagicMock()
cursor.fetchall.side_effect = [display_rows, summary_rows]
cursor.fetchone.side_effect = [(12,), (34,)]
conn.cursor.return_value = cursor_cm
cursor_cm.__enter__.return_value = cursor
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm):
with patch.object(cluster_runtime_service, "prune_expired_runtime_nodes"):
with patch.object(cluster_runtime_service, "register_local_control_heartbeat"):
with patch.object(cluster_runtime_service, "_load_managed_node_overlays", return_value={}):
with patch.object(cluster_runtime_service, "_load_disabled_managed_node_codes", return_value=set()):
snapshot = cluster_runtime_service.get_cluster_snapshot()
self.assertEqual(100, len(snapshot["nodes"]))
self.assertEqual(101, snapshot["nodes_total"])
self.assertEqual(101, snapshot["summary"]["online_worker_nodes"])
self.assertEqual(12, snapshot["jobs_total"])
self.assertEqual(34, snapshot["active_job_items"])
def test_get_cluster_snapshot_excludes_disabled_managed_nodes(self) -> None:
now = datetime.now()
display_rows = [
("mainland-controller-01", "mainland", "control", "controller", "10.0.0.1", "online", "test", 1, {}, now, now),
("mainland-worker-01", "mainland", "worker", "worker-01", "10.0.0.2", "busy", "test", 1, {}, now, now),
]
summary_rows = list(display_rows)
conn = MagicMock()
cursor_cm = MagicMock()
cursor = MagicMock()
cursor.fetchall.side_effect = [display_rows, summary_rows]
cursor.fetchone.side_effect = [(1,), (2,)]
conn.cursor.return_value = cursor_cm
cursor_cm.__enter__.return_value = cursor
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm):
with patch.object(cluster_runtime_service, "prune_expired_runtime_nodes"):
with patch.object(cluster_runtime_service, "register_local_control_heartbeat"):
with patch.object(cluster_runtime_service, "_load_managed_node_overlays", return_value={}):
with patch.object(cluster_runtime_service, "_load_disabled_managed_node_codes", return_value={"mainland-worker-01"}):
snapshot = cluster_runtime_service.get_cluster_snapshot()
self.assertEqual(["mainland-controller-01"], [item["node_code"] for item in snapshot["nodes"]])
self.assertEqual(1, snapshot["nodes_total"])
self.assertEqual(1, snapshot["summary"]["online_control_nodes"])
self.assertEqual(0, snapshot["summary"]["online_worker_nodes"])
def test_get_cluster_snapshot_prefers_imported_runtime_update_time_for_freshness(self) -> None:
now = datetime.now()
stale_heartbeat = now - timedelta(minutes=8)
fresh_ingest = now - timedelta(seconds=20)
display_rows = [
(
"mainland-controller-01-da",
"mainland",
"worker",
"controller",
"10.0.0.1",
"busy",
"test",
42,
{
"service": "runtime-ingest",
"updated_at": fresh_ingest.isoformat(timespec="seconds"),
"active_threads": 42,
},
stale_heartbeat,
fresh_ingest,
)
]
summary_rows = list(display_rows)
conn = MagicMock()
cursor_cm = MagicMock()
cursor = MagicMock()
cursor.fetchall.side_effect = [display_rows, summary_rows]
cursor.fetchone.side_effect = [(1,), (1,)]
conn.cursor.return_value = cursor_cm
cursor_cm.__enter__.return_value = cursor
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
with patch.object(cluster_runtime_service, "get_db", return_value=db_cm):
with patch.object(cluster_runtime_service, "prune_expired_runtime_nodes"):
with patch.object(cluster_runtime_service, "register_local_control_heartbeat"):
with patch.object(cluster_runtime_service, "_load_managed_node_overlays", return_value={}):
with patch.object(cluster_runtime_service, "_load_disabled_managed_node_codes", return_value=set()):
snapshot = cluster_runtime_service.get_cluster_snapshot()
self.assertEqual(1, snapshot["summary"]["online_worker_nodes"])
self.assertEqual([], snapshot["summary"]["offline_nodes"])
self.assertEqual("busy", snapshot["nodes"][0]["status"])
def test_register_local_control_heartbeat_uses_detect_status_snapshot(self) -> None:
detect_status = {
"worker_online": True,
"active_thread_count": 17,
"max_thread_count": 320,
"available_proxy_count": 41,
"proxy_runtime_label": "代理可用",
"proxy_runtime_reason": "pool_ready",
"proxy_last_refresh_status": "ok",
"proxy_last_refresh_time": "2026-04-24 01:30:00",
"proxy_last_refresh_source_count": 3,
"proxy_last_refresh_total_items": 120,
"proxy_last_validated_count": 110,
"proxy_last_available_count": 41,
"phase_label": "running",
"phase_detail": "正在执行检测",
"active_job": {
"job_code": "detect-20260424013000-abcd12",
"status": "running",
"node_stats": [
{
"node_code": "mainland-controller-01",
"items_total": 56,
"items_claimed": 5,
"items_running": 9,
"items_completed": 42,
}
],
},
}
with patch.object(cluster_runtime_service.settings, "node_code", "mainland-controller-01"):
with patch.object(cluster_runtime_service.settings, "node_region", "mainland"):
with patch.object(cluster_runtime_service.settings, "node_role", "control"):
with patch("app.services.detect_service.get_detect_status", return_value=detect_status):
with patch.object(cluster_runtime_service, "register_node_heartbeat") as mock_register:
cluster_runtime_service.register_local_control_heartbeat()
mock_register.assert_called_once()
payload = mock_register.call_args.kwargs
self.assertEqual("mainland-controller-01", payload["node_code"])
self.assertEqual("busy", payload["status"])
self.assertEqual(17, payload["current_load"])
self.assertTrue(payload["metadata"]["worker_online"])
self.assertTrue(payload["metadata"]["detect_participating"])
self.assertEqual("detect-20260424013000-abcd12", payload["metadata"]["active_job_code"])
self.assertEqual(17, payload["metadata"]["active_threads"])
self.assertEqual(320, payload["metadata"]["max_threads"])
self.assertEqual(41, payload["metadata"]["available_proxy_count"])
self.assertEqual("running", payload["metadata"]["phase_label"])
if __name__ == "__main__":
unittest.main()

267
domain-api/tests/test_dashboard_service.py
View File

@@ -39,20 +39,22 @@ class _FakeConnection:
class DashboardServiceTests(unittest.TestCase):
@patch("app.services.dashboard.get_detect_status")
@patch("app.services.dashboard._fetch_active_jobs_aggregate")
@patch("app.services.dashboard.get_detect_capacity_plan")
@patch("app.services.dashboard.get_detect_queue_health")
@patch("app.services.dashboard.get_runtime_status")
@patch("app.services.dashboard._build_dashboard_runtime_summary")
@patch("app.services.dashboard.get_active_detect_job_summary")
@patch("app.services.dashboard.get_db")
def test_fetch_overview_includes_ops_metrics(
self,
mock_get_db,
mock_get_active_detect_job_summary,
mock_get_runtime_status,
mock_build_dashboard_runtime_summary,
mock_get_detect_queue_health,
mock_get_detect_capacity_plan,
mock_fetch_active_jobs_aggregate,
mock_get_detect_status,
) -> None:
mock_get_db.return_value = _FakeConnection(
responses=[
@@ -122,7 +124,7 @@ class DashboardServiceTests(unittest.TestCase):
"items_blacklisted": 0,
"items_failed": 0,
}
mock_get_runtime_status.return_value = {
mock_build_dashboard_runtime_summary.return_value = {
"worker": {"running": True, "mode": "linux-systemd", "expected_on_this_node": True},
"node": {"region": "overseas", "role": "control"},
"cluster": {"summary": {"online_worker_nodes": 2, "dedicated_online_worker_nodes": 1, "online_control_nodes": 1}},
@@ -198,6 +200,15 @@ class DashboardServiceTests(unittest.TestCase):
"remaining_items": 193,
"recommended_additional_workers": 1,
}
mock_get_detect_status.return_value = {
"available_proxy_count": 1011,
"proxy_runtime_label": "集群代理正常",
"proxy_runtime_detail": "参与服务器 2 台，共可用 1011 个代理",
"proxy_last_refresh_status": "mainland-controller-01:201；mainland-worker-01:810",
"aggregate_process_count": 10,
"aggregate_participating_node_count": 10,
"active_thread_count": 165,
}
data = fetch_overview()
@@ -218,6 +229,10 @@ class DashboardServiceTests(unittest.TestCase):
self.assertEqual(165, data["queue_display_running_total"])
self.assertEqual(7, data["queue_completed_total"])
self.assertEqual(9438, data["backlog_pending_total"])
self.assertEqual(1011, data["cluster_proxy_available_count"])
self.assertEqual("集群代理正常", data["cluster_proxy_runtime_label"])
self.assertEqual(10, data["aggregate_process_count"])
self.assertEqual(1, data["ops_summary"]["active_execution_nodes"])
self.assertEqual(8487, data["backlog_register_pending_total"])
self.assertEqual(951, data["backlog_downstream_pending_total"])
self.assertEqual(12, data["retry_total"])
@@ -228,10 +243,256 @@ class DashboardServiceTests(unittest.TestCase):
self.assertEqual(1.5, data["ops_summary"]["estimated_hours_remaining"])
self.assertEqual(1, data["active_execution_nodes"])
self.assertEqual(1, data["ops_summary"]["active_execution_nodes"])
self.assertEqual(0, data["current_job_blacklisted"])
self.assertEqual(0, data["recent_blacklisted_total"])
self.assertEqual(0, data["cumulative_blacklisted_total"])
self.assertEqual(2, len(data["step_queue"]))
self.assertTrue(any(item["step_code"] == "detect_360_site" for item in data["step_queue"]))
self.assertEqual(1, len(data["node_throughput"]))
@patch("app.services.dashboard._fetch_active_jobs_aggregate")
@patch("app.services.dashboard.get_detect_capacity_plan")
@patch("app.services.dashboard.get_detect_queue_health")
@patch("app.services.dashboard._build_dashboard_runtime_summary")
@patch("app.services.dashboard.get_active_detect_job_summary")
@patch("app.services.dashboard.get_db")
def test_fetch_overview_prefers_active_job_display_running_when_queue_snapshot_is_stale(
self,
mock_get_db,
mock_get_active_detect_job_summary,
mock_build_dashboard_runtime_summary,
mock_get_detect_queue_health,
mock_get_detect_capacity_plan,
mock_fetch_active_jobs_aggregate,
) -> None:
mock_get_db.return_value = _FakeConnection(
responses=[
(1000,),
(900,),
(10,),
(5,),
(0,),
(1,),
(430,),
(420,),
(17,),
]
)
mock_fetch_active_jobs_aggregate.return_value = {
"active_jobs_total": 1,
"queue": {"items_total": 8402, "pending": 5001, "claimed": 0, "running": 1, "completed": 0, "blacklisted": 0, "failed": 0},
"throughput": {"processed_recent": 0, "processed_per_minute": 0.0},
"steps": [],
"nodes": [],
"retry_total": 0,
}
mock_get_active_detect_job_summary.return_value = {
"job_id": 11,
"job_code": "sync-overseas-51",
"status": "running",
"items_total": 8402,
"items_pending": 5001,
"items_claimed": 0,
"items_running": 0,
"items_completed": 0,
"items_blacklisted": 0,
"items_failed": 0,
"display_items_running": 1432,
"display_active_threads": 138,
"display_max_threads": 80000,
"distributed_node_stats": [
{"node_code": "mainland-controller-01-a", "display_running": 1000, "active_threads": 1000, "max_threads": 1000},
{"node_code": "mainland-controller-01-b", "display_running": 432, "active_threads": 432, "max_threads": 1000},
],
}
mock_build_dashboard_runtime_summary.return_value = {
"worker": {"running": False, "mode": "linux-systemd", "expected_on_this_node": False},
"node": {"region": "overseas", "role": "control"},
"cluster": {"summary": {"online_worker_nodes": 1, "dedicated_online_worker_nodes": 0, "online_control_nodes": 2}},
"detect": {
"backlog": {
"pending_total": 5001,
"claimed_total": 0,
"running_total": 0,
"completed_total": 0,
"blacklisted_total": 0,
"failed_total": 0,
"register_pending": 3495,
"downstream_pending": 1506,
}
},
}
mock_get_detect_queue_health.return_value = {
"has_active_job": True,
"job": {"job_id": 11, "job_code": "sync-overseas-51", "runtime_job_code": "sync-overseas-51", "status": "running", "progress_percent": 40.49},
"queue": {
"items_total": 8402,
"pending": 5001,
"claimed": 0,
"display_claimed": 0,
"running": 1,
"display_running": 1,
"completed": 0,
"blacklisted": 0,
"failed": 0,
},
"throughput": {"processed_recent": 0, "processed_per_minute": 0.0},
"steps": [],
"runtime_activity": {},
"nodes": [],
}
mock_get_detect_capacity_plan.return_value = {
"estimated_hours_remaining": 0,
"remaining_items": 5001,
"recommended_additional_workers": 0,
}
data = fetch_overview()
self.assertEqual(1432, data["queue_display_running_total"])
self.assertEqual(1432, data["active_job"]["items_display_running"])
self.assertEqual(1432, data["active_job"]["display_items_running"])
self.assertEqual(138, data["active_job"]["display_active_threads"])
self.assertEqual(80000, data["active_job"]["display_max_threads"])
self.assertEqual(2, len(data["active_job"]["distributed_node_stats"]))
self.assertEqual(80000, data["queue_display_max_threads"])
self.assertEqual(0, data["current_job_blacklisted"])
self.assertEqual(0, data["recent_blacklisted_total"])
self.assertEqual(0, data["cumulative_blacklisted_total"])
@patch("app.services.dashboard._fetch_active_jobs_aggregate")
@patch("app.services.dashboard.get_detect_capacity_plan")
@patch("app.services.dashboard.get_detect_queue_health")
@patch("app.services.dashboard._build_dashboard_runtime_summary")
@patch("app.services.dashboard.get_active_detect_job_summary")
@patch("app.services.dashboard.get_db")
def test_fetch_overview_counts_active_execution_nodes_from_full_node_set(
self,
mock_get_db,
mock_get_active_detect_job_summary,
mock_build_dashboard_runtime_summary,
mock_get_detect_queue_health,
mock_get_detect_capacity_plan,
mock_fetch_active_jobs_aggregate,
) -> None:
mock_get_db.return_value = _FakeConnection(responses=[(0,)] * 9)
mock_fetch_active_jobs_aggregate.return_value = {
"active_jobs_total": 1,
"queue": {"items_total": 9, "pending": 0, "claimed": 0, "running": 9, "completed": 0, "blacklisted": 0, "failed": 0},
"throughput": {"processed_recent": 0, "processed_per_minute": 0.0, "completed_recent": 0, "blacklisted_recent": 0, "failed_recent": 0},
"steps": [],
"nodes": [
{
"node_code": f"mainland-controller-01-{index:02d}",
"items_running": 1,
"items_claimed": 0,
"processed_recent": 0,
"processed_per_minute": 0.0,
"completed_recent": 0,
"failed_recent": 0,
"blacklisted_recent": 0,
}
for index in range(9)
],
"retry_total": 0,
}
mock_get_active_detect_job_summary.return_value = {}
mock_build_dashboard_runtime_summary.return_value = {
"worker": {"running": True, "mode": "linux-systemd", "expected_on_this_node": True},
"node": {"region": "mainland", "role": "worker"},
"cluster": {"summary": {"online_worker_nodes": 1, "dedicated_online_worker_nodes": 0, "online_control_nodes": 1}},
"detect": {"backlog": {}},
}
mock_get_detect_queue_health.return_value = {
"has_active_job": False,
"queue": {},
"throughput": {"processed_recent": 0, "processed_per_minute": 0.0},
"steps": [],
"runtime_activity": {},
"nodes": [],
"runtime_snapshot_backlog": {},
}
mock_get_detect_capacity_plan.return_value = {
"estimated_hours_remaining": 0,
"remaining_items": 0,
"recommended_additional_workers": 0,
}
data = fetch_overview()
self.assertEqual(8, len(data["node_throughput"]))
self.assertEqual(9, data["active_execution_nodes"])
self.assertEqual(9, data["ops_summary"]["active_execution_nodes"])
@patch("app.services.dashboard._fetch_active_jobs_aggregate")
@patch("app.services.dashboard.get_detect_capacity_plan")
@patch("app.services.dashboard.get_detect_queue_health")
@patch("app.services.dashboard._build_dashboard_runtime_summary")
@patch("app.services.dashboard.get_active_detect_job_summary")
@patch("app.services.dashboard.get_db")
def test_fetch_overview_keeps_active_job_summary_when_queue_health_temporarily_empty(
self,
mock_get_db,
mock_get_active_detect_job_summary,
mock_build_dashboard_runtime_summary,
mock_get_detect_queue_health,
mock_get_detect_capacity_plan,
mock_fetch_active_jobs_aggregate,
) -> None:
mock_get_db.return_value = _FakeConnection(responses=[(0,)] * 9)
mock_fetch_active_jobs_aggregate.return_value = {
"active_jobs_total": 1,
"queue": {"items_total": 8402, "pending": 5001, "claimed": 0, "running": 537, "completed": 0, "blacklisted": 0, "failed": 0},
"throughput": {"processed_recent": 12, "processed_per_minute": 0.8, "completed_recent": 10, "blacklisted_recent": 1, "failed_recent": 1},
"steps": [],
"nodes": [],
"retry_total": 0,
}
mock_get_active_detect_job_summary.return_value = {
"job_id": 255,
"job_code": "sync-overseas-255",
"runtime_job_code": "sync-overseas-255",
"status": "running",
"progress_percent": 22.4,
"items_total": 8402,
"items_pending": 5001,
"items_claimed": 0,
"items_running": 537,
"items_completed": 0,
"items_blacklisted": 0,
"items_failed": 0,
"display_items_running": 11799,
"display_active_threads": 11799,
"display_max_threads": 75400,
"distributed_node_stats": [{"node_code": "mainland-controller-01-a", "display_running": 537, "active_threads": 537, "max_threads": 1000}],
}
mock_build_dashboard_runtime_summary.return_value = {
"worker": {"running": False, "mode": "linux-systemd", "expected_on_this_node": False},
"node": {"region": "overseas", "role": "control"},
"cluster": {"summary": {"online_worker_nodes": 1, "dedicated_online_worker_nodes": 0, "online_control_nodes": 1}},
"detect": {"backlog": {}},
}
mock_get_detect_queue_health.return_value = {
"has_active_job": False,
"queue": {},
"throughput": {"processed_recent": 0, "processed_per_minute": 0.0},
"steps": [],
"runtime_activity": {},
"nodes": [],
"runtime_snapshot_backlog": {},
}
mock_get_detect_capacity_plan.return_value = {
"estimated_hours_remaining": 0,
"remaining_items": 5001,
"recommended_additional_workers": 0,
}
data = fetch_overview()
self.assertEqual("sync-overseas-255", data["active_job"]["job_code"])
self.assertEqual(11799, data["active_job"]["display_items_running"])
self.assertEqual(75400, data["active_job"]["display_max_threads"])
if __name__ == "__main__":
unittest.main()

83
domain-api/tests/test_debug_event_service.py Normal file
View File

@@ -0,0 +1,83 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
from app.services import debug_event_service
class DebugEventServiceTests(unittest.TestCase):
def test_ingest_debug_event_requires_configured_shared_token(self) -> None:
with patch.object(debug_event_service.settings, "sync_shared_token", ""):
ok, message, data = debug_event_service.ingest_debug_event(
{
"source_region": "mainland",
"node_code": "mainland-controller-01",
"service": "worker-event",
"event_type": "worker_log",
"message": "开始检测域名: a.com",
"payload": {"job_id": 1, "job_code": "sync-overseas-1"},
},
shared_token=None,
)
self.assertFalse(ok)
self.assertIn("token 未配置", message)
self.assertTrue(data["configuration_required"])
def test_resolve_target_job_for_debug_event_rejects_missing_identity(self) -> None:
job, reason = debug_event_service._resolve_target_job_for_debug_event({"domain": "a.com"})
self.assertIsNone(job)
self.assertEqual("missing_job_identity", reason)
@patch("app.services.debug_event_service._load_detect_job_summary_by_job_code")
@patch("app.services.detect_job_service.get_detect_job_summary")
@patch("app.services.detect_job_service.get_active_detect_job_summary")
def test_resolve_target_job_for_debug_event_prefers_payload_job_id_over_current_active_job(
self,
mock_get_active_detect_job_summary,
mock_get_detect_job_summary,
mock_load_detect_job_summary_by_job_code,
) -> None:
mock_get_active_detect_job_summary.return_value = {
"job_id": 12,
"job_code": "sync-overseas-12",
"current_cycle_token": "cycle-12",
}
mock_get_detect_job_summary.return_value = {
"job_id": 11,
"job_code": "sync-overseas-11",
"current_cycle_token": "cycle-11",
}
job, reason = debug_event_service._resolve_target_job_for_debug_event(
{"job_id": 11, "job_code": "sync-overseas-11", "cycle_token": "cycle-11"}
)
self.assertEqual("matched", reason)
self.assertEqual(11, job["job_id"])
mock_get_detect_job_summary.assert_called_once_with(11, event_limit=1)
mock_load_detect_job_summary_by_job_code.assert_not_called()
@patch("app.services.detect_job_service.get_active_detect_job_summary")
def test_resolve_target_job_for_debug_event_rejects_cycle_mismatch(
self,
mock_get_active_detect_job_summary,
) -> None:
mock_get_active_detect_job_summary.return_value = {
"job_id": 12,
"job_code": "sync-overseas-12",
"current_cycle_token": "cycle-current",
}
job, reason = debug_event_service._resolve_target_job_for_debug_event(
{"job_id": 12, "job_code": "sync-overseas-12", "cycle_token": "cycle-old"}
)
self.assertIsNone(job)
self.assertEqual("cycle_mismatch", reason)
if __name__ == "__main__":
unittest.main()

232
domain-api/tests/test_detect_api_routes.py
View File

@@ -7,6 +7,153 @@ from app.api.routes import detect as detect_route
class DetectApiRoutesTestCase(unittest.TestCase):
@patch("app.api.routes.detect.create_detect_run_snapshot")
@patch("app.api.routes.detect._dispatch_remote_detect_start")
@patch("app.api.routes.detect.get_settings_payload")
@patch("app.api.routes.detect.get_detect_status")
@patch("app.api.routes.detect.send_worker_command")
@patch("app.api.routes.detect.start_worker")
@patch("app.api.routes.detect.append_detect_job_event")
@patch("app.api.routes.detect.create_detect_job_if_needed")
def test_start_detect_skips_redundant_systemctl_when_worker_already_online(
self,
mock_create_job,
mock_append_event,
mock_start_worker,
mock_send_worker_command,
mock_get_detect_status,
mock_get_settings_payload,
mock_dispatch_remote,
mock_create_snapshot,
) -> None:
mock_create_job.return_value = {
"job_id": 30,
"job_code": "sync-overseas-30",
"status": "running",
"items_pending": 100,
"items_claimed": 0,
"items_running": 0,
"task_mode": "domain_pipeline",
"step_code": "",
}
mock_get_detect_status.return_value = {
"worker_mode": "linux-systemd",
"worker_online": True,
"worker_process_count": 60,
"worker_latest_start_time": "2026-04-24 23:30:00",
"worker_runtime_message": "running",
"progress": {},
}
mock_send_worker_command.return_value = (True, "已发送 Worker 控制指令: start_detection")
mock_get_settings_payload.return_value = {
"runtime": {"thread_count": 1000},
"proxy_config": {"proxy_enable": True, "allow_direct": True, "proxy_urls": ["a"]},
}
mock_dispatch_remote.return_value = {
"queued_jobs": [],
"queued_total": 0,
"failed_total": 0,
"target_summary": {"controller_nodes": [], "worker_nodes": []},
}
with patch.object(detect_route.settings, "node_region", "mainland"), patch.object(
detect_route.settings, "node_role", "control"
):
response = detect_route.start_detect()
self.assertEqual(0, response.code)
self.assertIn("检测端已在运行，跳过重复启动", response.message)
mock_start_worker.assert_not_called()
mock_send_worker_command.assert_called_once()
event_types = [call.kwargs.get("event_type") for call in mock_append_event.call_args_list]
self.assertIn("job_dispatch_start_skipped", event_types)
self.assertIn("job_dispatch_sent", event_types)
mock_create_snapshot.assert_called_once()
@patch("app.api.routes.detect.create_detect_run_snapshot")
@patch("app.api.routes.detect._dispatch_remote_detect_start")
@patch("app.api.routes.detect.get_settings_payload")
@patch("app.api.routes.detect.get_detect_status")
@patch("app.api.routes.detect.send_worker_command")
@patch("app.api.routes.detect.start_worker")
@patch("app.api.routes.detect.append_detect_job_event")
@patch("app.api.routes.detect.create_detect_job_if_needed")
def test_start_detect_falls_back_to_direct_command_when_worker_already_running(
self,
mock_create_job,
mock_append_event,
mock_start_worker,
mock_send_worker_command,
mock_get_detect_status,
mock_get_settings_payload,
mock_dispatch_remote,
mock_create_snapshot,
) -> None:
mock_create_job.return_value = {
"job_id": 31,
"job_code": "sync-overseas-31",
"status": "running",
"items_pending": 100,
"items_claimed": 0,
"items_running": 0,
"task_mode": "domain_pipeline",
"step_code": "",
}
mock_start_worker.return_value = (
False,
"domaincheck-worker 控制失败：当前运行用户没有免密 systemctl 权限，请为 API 进程授予对应 sudo/systemd 权限",
)
mock_send_worker_command.return_value = (True, "已发送 Worker 控制指令: start_detection")
mock_get_detect_status.side_effect = [
{
"worker_mode": "linux-systemd",
"worker_online": False,
"worker_process_count": 0,
"worker_latest_start_time": "",
"worker_runtime_message": "starting",
"progress": {},
},
{
"worker_mode": "linux-systemd",
"worker_online": True,
"worker_process_count": 60,
"worker_latest_start_time": "2026-04-24 23:30:00",
"worker_runtime_message": "running",
"progress": {},
},
{
"worker_mode": "linux-systemd",
"worker_online": True,
"worker_process_count": 60,
"worker_latest_start_time": "2026-04-24 23:30:00",
"worker_runtime_message": "running",
"progress": {},
},
]
mock_get_settings_payload.return_value = {
"runtime": {"thread_count": 1000},
"proxy_config": {"proxy_enable": True, "allow_direct": True, "proxy_urls": ["a"]},
}
mock_dispatch_remote.return_value = {
"queued_jobs": [],
"queued_total": 0,
"failed_total": 0,
"target_summary": {"controller_nodes": [], "worker_nodes": []},
}
with patch.object(detect_route.settings, "node_region", "mainland"), patch.object(
detect_route.settings, "node_role", "control"
):
response = detect_route.start_detect()
self.assertEqual(0, response.code)
self.assertIn("检测端已在运行，改为直接发送控制指令", response.message)
mock_send_worker_command.assert_called_once()
event_types = [call.kwargs.get("event_type") for call in mock_append_event.call_args_list]
self.assertIn("job_dispatch_start_degraded", event_types)
self.assertIn("job_dispatch_sent", event_types)
mock_create_snapshot.assert_called_once()
@patch("app.api.routes.detect.create_detect_run_snapshot")
@patch("app.api.routes.detect._dispatch_remote_detect_start")
@patch("app.api.routes.detect.get_settings_payload")
@@ -65,11 +212,94 @@ class DetectApiRoutesTestCase(unittest.TestCase):
mock_start_worker.assert_not_called()
mock_send_worker_command.assert_not_called()
mock_dispatch_remote.assert_called_once()
mock_create_snapshot.assert_called_once()
mock_create_snapshot.assert_not_called()
event_types = [call.kwargs.get("event_type") for call in mock_append_event.call_args_list]
self.assertIn("job_dispatch_requested", event_types)
self.assertIn("job_dispatch_skipped_local", event_types)
@patch("app.api.routes.detect.finalize_detect_run")
@patch("app.api.routes.detect.mark_detect_run_stopping")
@patch("app.api.routes.detect.get_settings_payload")
@patch("app.api.routes.detect.get_detect_status")
@patch("app.api.routes.detect._dispatch_remote_detect_stop")
@patch("app.api.routes.detect.send_worker_command")
@patch("app.api.routes.detect.get_active_detect_job_summary")
def test_stop_detect_forwards_target_payload(
self,
mock_get_active_job,
mock_send_worker_command,
mock_dispatch_remote_stop,
mock_get_detect_status,
mock_get_settings_payload,
mock_mark_detect_run_stopping,
mock_finalize_detect_run,
) -> None:
mock_get_active_job.return_value = {
"job_id": 29,
"job_code": "sync-overseas-29",
"status": "running",
"current_cycle_token": "cycle-29",
}
mock_send_worker_command.return_value = (True, "已发送 Worker 控制指令")
mock_dispatch_remote_stop.return_value = {
"queued_jobs": [{"node_code": "mainland-controller-01"}],
"queued_total": 1,
"failed_total": 0,
"target_summary": {"controller_nodes": ["mainland-controller-01"], "worker_nodes": []},
}
mock_get_detect_status.return_value = {
"worker_mode": "linux-systemd",
"worker_online": True,
"worker_process_count": 1,
"worker_latest_start_time": "",
"worker_runtime_message": "running",
"progress": {},
}
mock_get_settings_payload.return_value = {
"thread_count": 1000,
"node_thread_counts": {},
"process_count": 80,
"node_process_counts": {},
"proxy_config": {"proxy_enable": True, "allow_direct": False, "proxy_urls": ["a"]},
}
response = detect_route.stop_detect(payload={"target_node_codes": ["mainland-controller-01"]})
self.assertEqual(0, response.code)
mock_send_worker_command.assert_called_once_with(
"stop_detection",
payload={"target_node_codes": ["mainland-controller-01"]},
)
mock_dispatch_remote_stop.assert_called_once_with(
active_job=mock_get_active_job.return_value,
cycle_token="cycle-29",
payload={"target_node_codes": ["mainland-controller-01"]},
)
mock_mark_detect_run_stopping.assert_called_once()
mock_finalize_detect_run.assert_not_called()
@patch("app.api.routes.detect.create_ops_job")
@patch("app.api.routes.detect.list_managed_nodes")
def test_dispatch_remote_detect_stop_filters_target_nodes(self, mock_list_managed_nodes, mock_create_ops_job) -> None:
mock_list_managed_nodes.return_value = [
{"node_code": "mainland-controller-01", "region": "mainland", "role": "control", "last_seen_at": "2026-04-23T14:00:00", "is_enabled": True},
{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker", "last_seen_at": "2026-04-23T14:00:00", "is_enabled": True},
]
mock_create_ops_job.return_value = (True, "queued", {})
result = detect_route._dispatch_remote_detect_stop(
active_job={"job_id": 29, "job_code": "sync-overseas-29"},
cycle_token="cycle-29",
payload={"target_node_codes": ["mainland-worker-01"]},
)
self.assertEqual(1, result["queued_total"])
self.assertEqual(["mainland-controller-01"], result["target_summary"]["controller_nodes"])
self.assertEqual(["mainland-worker-01"], result["target_summary"]["worker_nodes"])
mock_create_ops_job.assert_called_once()
create_payload = mock_create_ops_job.call_args.args[0]
self.assertEqual("mainland-worker-01", create_payload["target_node_code"])
if __name__ == "__main__":
unittest.main()

1981
domain-api/tests/test_detect_job_service.py
View File

File diff suppressed because it is too large Load Diff

65
domain-api/tests/test_detect_run_service.py Normal file
View File

@@ -0,0 +1,65 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
import app.services.detect_run_service as detect_run_service
class _TraceLock:
def __init__(self, order: list[str]) -> None:
self.order = order
def __enter__(self):
self.order.append("enter")
return self
def __exit__(self, exc_type, exc, tb):
self.order.append("exit")
return False
class DetectRunServiceTests(unittest.TestCase):
def test_create_detect_run_snapshot_acquires_lock_before_loading_records(self) -> None:
order: list[str] = []
with patch.object(detect_run_service, "_DETECT_RUNS_LOCK", _TraceLock(order)):
with patch.object(detect_run_service, "_load", side_effect=lambda: order.append("load") or []):
with patch.object(detect_run_service, "_capture_worker_logs", return_value=[]):
with patch.object(detect_run_service, "_save", side_effect=lambda records: order.append("save")):
record = detect_run_service.create_detect_run_snapshot(
"start",
{"latest_start_time": "2026-04-23 15:00:00", "running": True},
{"running": 1},
{"thread_count": 1000},
)
self.assertEqual("enter", order[0])
self.assertIn("load", order)
self.assertIn("save", order)
self.assertEqual("exit", order[-1])
self.assertEqual("starting", record["status"])
def test_sync_detect_runs_acquires_lock_before_mutating_records(self) -> None:
order: list[str] = []
with patch.object(detect_run_service, "_DETECT_RUNS_LOCK", _TraceLock(order)):
with patch.object(detect_run_service, "_load", side_effect=lambda: order.append("load") or []):
with patch.object(detect_run_service, "_capture_worker_logs", return_value=[]):
with patch.object(detect_run_service, "_save", side_effect=lambda records: order.append("save")):
records = detect_run_service.sync_detect_runs(
{"running": True, "detecting": True, "latest_start_time": "2026-04-23 15:00:00"},
{"running": 1, "pending": 0},
{"thread_count": 1000},
active_job={"status": "running", "items_running": 1},
)
self.assertEqual("enter", order[0])
self.assertIn("load", order)
self.assertIn("save", order)
self.assertEqual("exit", order[-1])
self.assertEqual("running", records[0]["status"])
if __name__ == "__main__":
unittest.main()

62
domain-api/tests/test_detect_service_remote_logs.py Normal file
View File

@@ -0,0 +1,62 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
from app.services import detect_service
class DetectServiceRemoteLogsTests(unittest.TestCase):
@patch("app.services.detect_service.list_debug_events")
def test_build_remote_log_snapshot_from_debug_events_filters_to_active_job_identity(self, mock_list_debug_events) -> None:
mock_list_debug_events.return_value = {
"records": [
{
"id": 3,
"node_code": "mainland-controller-01-a",
"event_type": "worker_log",
"message": "旧任务日志",
"created_at": "2026-04-23 16:00:00",
"payload": {
"job_id": 10,
"job_code": "sync-overseas-10",
"cycle_token": "cycle-old",
"log_mode": "key",
},
},
{
"id": 4,
"node_code": "mainland-controller-01-a",
"event_type": "worker_log",
"message": "当前任务日志",
"created_at": "2026-04-23 16:01:00",
"payload": {
"job_id": 12,
"job_code": "sync-overseas-12",
"cycle_token": "cycle-12",
"log_mode": "key",
},
},
]
}
snapshot = detect_service._build_remote_log_snapshot_from_debug_events(
{
"job_id": 12,
"job_code": "sync-overseas-12",
"runtime_job_code": "sync-overseas-12",
"current_cycle_token": "cycle-12",
"node_stats": [{"node_code": "mainland-controller-01-a"}],
},
enabled=True,
mode="key",
limit=20,
)
self.assertEqual(1, snapshot["line_count"])
self.assertIn("当前任务日志", snapshot["lines"][0])
self.assertNotIn("旧任务日志", "\n".join(snapshot["lines"]))
if __name__ == "__main__":
unittest.main()

916
domain-api/tests/test_detect_service_status_fallback.py
View File

@@ -1,10 +1,47 @@
import unittest
from contextlib import ExitStack
from datetime import datetime, timedelta
from unittest.mock import patch
from app.services import detect_service
class DetectServiceStatusFallbackTests(unittest.TestCase):
def test_filter_live_aggregate_runtime_nodes_excludes_disabled_managed_nodes(self) -> None:
now_text = datetime.now().isoformat(sep=" ", timespec="seconds")
rows = [
{
"node_code": "mainland-controller-01",
"status": "busy",
"last_heartbeat_at": now_text,
"display_running": 1,
},
{
"node_code": "mainland-worker-01",
"status": "busy",
"last_heartbeat_at": now_text,
"display_running": 1,
},
]
with patch("app.services.detect_service._load_disabled_managed_node_codes", return_value={"mainland-worker-01"}):
filtered = detect_service._filter_live_aggregate_runtime_nodes(rows)
self.assertEqual(["mainland-controller-01"], [item["node_code"] for item in filtered])
def setUp(self) -> None:
detect_service._DETECT_STATUS_CACHE_VALUE = None
detect_service._DETECT_STATUS_CACHE_EXPIRES_AT = 0.0
def test_resolve_capacity_node_code_treats_child_instance_suffix_as_single_process(self) -> None:
capacity_node_code, is_child_instance = detect_service._resolve_capacity_node_code(
"mainland-controller-01-ae",
{"node_thread_counts": {"overseas-control-01": 1}, "node_process_counts": {}},
)
self.assertEqual("mainland-controller-01", capacity_node_code)
self.assertTrue(is_child_instance)
def test_get_detect_status_keeps_runtime_snapshot_when_db_is_unreachable(self) -> None:
runtime_state = {
"service_running": True,
@@ -17,24 +54,39 @@ class DetectServiceStatusFallbackTests(unittest.TestCase):
"available_proxy_count": 18,
}
with patch("app.services.detect_service.ensure_runtime_schema"), \
patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")), \
patch("app.services.detect_service.get_settings_payload", return_value={"proxy_config": {"proxy_enable": True, "allow_direct": False, "proxy_urls": ["a"]}}), \
patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "full"}), \
patch("app.services.detect_service._load_recent_worker_lines", return_value=(True, "", [])), \
patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": True, "process_count": 1, "latest_start_time": "2026-04-20 23:58:00", "message": "active/running"}), \
patch("app.services.detect_service._load_runtime_state", return_value=runtime_state), \
patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}), \
patch("app.services.detect_service._extract_available_proxy_count", return_value=0), \
patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}), \
patch("app.services.detect_service._normalize_recent_warning", return_value=""), \
patch("app.services.detect_service._build_proxy_runtime_snapshot", return_value={"state": "healthy", "label": "代理正常", "detail": "healthy", "direct_fallback_active": False, "reason": "healthy", "last_refresh_status": "ok", "last_refresh_time": "", "source_count": 2, "raw_items": 18, "validated_count": 18, "available_count": 18, "source_stats": [], "supplier_empty": False}), \
patch("app.services.detect_service.resolve_thread_count", return_value={"effective_thread_count": 120, "default_thread_count": 5, "source": "node_override", "override_thread_count": 120, "node_code": "mainland-worker-01"}), \
patch("app.services.detect_service.get_active_detect_job_summary", side_effect=RuntimeError("db down")), \
patch("app.services.detect_service.sync_detect_runs", return_value=[]), \
patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}), \
patch("app.services.detect_service._extract_dependency_alerts", return_value=[]), \
patch("app.services.detect_service.append_detect_result_projection_if_changed"):
with ExitStack() as stack:
stack.enter_context(patch.object(detect_service.settings, "node_region", "mainland"))
stack.enter_context(patch.object(detect_service.settings, "node_role", "worker"))
stack.enter_context(patch.object(detect_service.settings, "node_code", "mainland-worker-01"))
stack.enter_context(patch("app.services.detect_service.ensure_runtime_schema"))
stack.enter_context(patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")))
stack.enter_context(
patch("app.services.detect_service.get_settings_payload", return_value={"proxy_config": {"proxy_enable": True, "allow_direct": False, "proxy_urls": ["a"]}})
)
stack.enter_context(patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "full"}))
stack.enter_context(patch("app.services.detect_service._load_recent_worker_lines", return_value=(True, "", [])))
stack.enter_context(
patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": True, "process_count": 1, "latest_start_time": "2026-04-20 23:58:00", "message": "active/running"})
)
stack.enter_context(patch("app.services.detect_service._load_runtime_state", return_value=runtime_state))
stack.enter_context(patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_available_proxy_count", return_value=0))
stack.enter_context(patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}))
stack.enter_context(patch("app.services.detect_service._normalize_recent_warning", return_value=""))
stack.enter_context(
patch(
"app.services.detect_service._build_proxy_runtime_snapshot",
return_value={"state": "healthy", "label": "代理正常", "detail": "healthy", "direct_fallback_active": False, "reason": "healthy", "last_refresh_status": "ok", "last_refresh_time": "", "source_count": 2, "raw_items": 18, "validated_count": 18, "available_count": 18, "source_stats": [], "supplier_empty": False},
)
)
stack.enter_context(
patch("app.services.detect_service.resolve_thread_count", return_value={"effective_thread_count": 120, "default_thread_count": 5, "source": "node_override", "override_thread_count": 120, "node_code": "mainland-worker-01"})
)
stack.enter_context(patch("app.services.detect_service.get_active_detect_job_summary", side_effect=RuntimeError("db down")))
stack.enter_context(patch("app.services.detect_service.sync_detect_runs", return_value=[]))
stack.enter_context(patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_dependency_alerts", return_value=[]))
stack.enter_context(patch("app.services.detect_service.append_detect_result_projection_if_changed"))
payload = detect_service.get_detect_status()
self.assertTrue(payload["worker_online"])
@@ -44,6 +96,56 @@ class DetectServiceStatusFallbackTests(unittest.TestCase):
self.assertEqual(0, payload["progress"]["pending"])
self.assertEqual(0, payload["progress"]["completed"])
def test_get_detect_status_skips_local_result_projection_on_overseas_control(self) -> None:
runtime_state = {
"service_running": False,
"detecting": False,
"active_threads": 0,
"max_threads": 1,
"phase": "idle",
"detail": "当前节点不承载本地检测执行",
"updated_at": "2026-04-24 02:10:00",
"available_proxy_count": 0,
}
with ExitStack() as stack:
stack.enter_context(patch.object(detect_service.settings, "node_region", "overseas"))
stack.enter_context(patch.object(detect_service.settings, "node_role", "control"))
stack.enter_context(patch.object(detect_service.settings, "node_code", "overseas-control-01"))
stack.enter_context(patch("app.services.detect_service.ensure_runtime_schema"))
stack.enter_context(patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")))
stack.enter_context(
patch("app.services.detect_service.get_settings_payload", return_value={"proxy_config": {"proxy_enable": False, "allow_direct": True, "proxy_urls": []}})
)
stack.enter_context(patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "key"}))
stack.enter_context(patch("app.services.detect_service._load_recent_worker_lines", return_value=(False, "", [])))
stack.enter_context(
patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": False, "process_count": 0, "latest_start_time": "", "message": "inactive"})
)
stack.enter_context(patch("app.services.detect_service._load_runtime_state", return_value=runtime_state))
stack.enter_context(patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_available_proxy_count", return_value=0))
stack.enter_context(patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}))
stack.enter_context(patch("app.services.detect_service._normalize_recent_warning", return_value=""))
stack.enter_context(
patch(
"app.services.detect_service._build_proxy_runtime_snapshot",
return_value={"state": "disabled", "label": "不适用", "detail": "-", "direct_fallback_active": False, "reason": "not_applicable", "last_refresh_status": "", "last_refresh_time": "", "source_count": 0, "raw_items": 0, "validated_count": 0, "available_count": 0, "source_stats": [], "supplier_empty": False},
)
)
stack.enter_context(
patch("app.services.detect_service.resolve_thread_count", return_value={"effective_thread_count": 1, "default_thread_count": 1, "source": "default", "override_thread_count": None, "node_code": "overseas-control-01"})
)
stack.enter_context(patch("app.services.detect_service.get_active_detect_job_summary", return_value=None))
stack.enter_context(patch("app.services.detect_service.sync_detect_runs", return_value=[]))
stack.enter_context(patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_dependency_alerts", return_value=[]))
mock_append = stack.enter_context(patch("app.services.detect_service.append_detect_result_projection_if_changed"))
detect_service.get_detect_status()
mock_append.assert_not_called()
def test_filter_lines_since_supports_journalctl_syslog_timestamps(self) -> None:
lines = [
"Apr 21 20:12:42 mainland-controller python[1]: 当前实际线程数量: 323/4",
@@ -64,6 +166,784 @@ class DetectServiceStatusFallbackTests(unittest.TestCase):
self.assertEqual(lines, filtered)
def test_get_detect_status_uses_aggregate_capacity_for_overseas_control(self) -> None:
active_job = {
"job_id": 11,
"job_code": "sync-overseas-51",
"status": "running",
"items_pending": 5001,
"items_completed": 1788,
"items_failed": 0,
"items_blacklisted": 0,
"progress_percent": 40.49,
"display_items_running": 1432,
"display_active_threads": 138,
"distributed_node_stats": [
{
"node_code": "mainland-controller-01",
"items_claimed": 73,
"items_running": 1432,
"display_running": 1432,
"active_threads": 138,
"max_threads": 2000,
},
{
"node_code": "mainland-worker-01",
"items_completed": 1614,
"items_running": 0,
"active_threads": 0,
"max_threads": 0,
},
{
"node_code": "unassigned",
"items_pending": 3495,
},
],
}
aggregate_queue_health = {
"has_active_job": True,
"job": {
"job_id": 11,
"job_code": "sync-overseas-51",
"status": "running",
"progress_percent": 40.49,
},
"queue": {
"items_total": 8402,
"pending": 5001,
"claimed": 0,
"running": 5909,
"display_running": 5909,
"completed": 0,
"blacklisted": 0,
"failed": 0,
},
"nodes": [
{
"node_code": "mainland-controller-01-a",
"items_running": 1000,
"display_running": 1000,
"active_threads": 1000,
"max_threads": 1000,
},
{
"node_code": "mainland-controller-01-b",
"items_running": 972,
"display_running": 972,
"active_threads": 972,
"max_threads": 1000,
},
{
"node_code": "mainland-controller-01-c",
"items_running": 3937,
"display_running": 3937,
"active_threads": 3937,
"max_threads": 36000,
},
],
}
def _resolve_thread_count(*, node_code=None, settings_payload=None):
if node_code == "mainland-controller-01" or str(node_code or "").startswith("mainland-controller-01-"):
return {
"effective_thread_count": 1000,
"default_thread_count": 1000,
"source": "default",
"override_thread_count": None,
"node_code": str(node_code or "mainland-controller-01"),
}
return {
"effective_thread_count": 1,
"default_thread_count": 1000,
"source": "node_override",
"override_thread_count": 1,
"node_code": "overseas-control-01",
}
def _resolve_process_count(*, node_code=None, settings_payload=None):
if node_code == "mainland-controller-01":
return {
"effective_process_count": 80,
"default_process_count": 80,
"source": "default",
"override_process_count": None,
"node_code": "mainland-controller-01",
}
if str(node_code or "").startswith("mainland-controller-01-"):
return {
"effective_process_count": 1,
"default_process_count": 80,
"source": "child_instance",
"override_process_count": None,
"node_code": str(node_code or ""),
}
return {
"effective_process_count": 1,
"default_process_count": 80,
"source": "node_override",
"override_process_count": 1,
"node_code": "overseas-control-01",
}
with ExitStack() as stack:
stack.enter_context(patch.object(detect_service.settings, "node_region", "overseas"))
stack.enter_context(patch.object(detect_service.settings, "node_role", "control"))
stack.enter_context(patch.object(detect_service.settings, "node_code", "overseas-control-01"))
stack.enter_context(patch("app.services.detect_service.ensure_runtime_schema"))
stack.enter_context(patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")))
stack.enter_context(
patch(
"app.services.detect_service.get_settings_payload",
return_value={
"proxy_config": {"proxy_enable": False, "allow_direct": True, "proxy_urls": []},
"process_count": 80,
"node_process_counts": {},
"thread_count": 1000,
"node_thread_counts": {"overseas-control-01": 1},
},
)
)
stack.enter_context(
patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "key"})
)
stack.enter_context(
patch("app.services.detect_service._load_recent_worker_lines", return_value=(False, "2026-04-23 00:00:00", ["stale line"]))
)
stack.enter_context(
patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": False, "process_count": 0, "latest_start_time": "", "message": "inactive"})
)
stack.enter_context(
patch("app.services.detect_service._load_runtime_state", return_value={"service_running": True, "detecting": True, "active_threads": 2, "max_threads": 1, "detail": "当前实际线程数量: 2/1"})
)
stack.enter_context(patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_available_proxy_count", return_value=0))
stack.enter_context(patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}))
stack.enter_context(patch("app.services.detect_service._normalize_recent_warning", return_value=""))
stack.enter_context(
patch(
"app.services.detect_service._build_proxy_runtime_snapshot",
return_value={
"state": "disabled",
"label": "未启用代理",
"detail": "-",
"direct_fallback_active": True,
"reason": "proxy_disabled",
"last_refresh_status": "",
"last_refresh_time": "",
"source_count": 0,
"raw_items": 0,
"validated_count": 0,
"available_count": 0,
"source_stats": [],
"supplier_empty": False,
},
)
)
stack.enter_context(patch("app.services.detect_service.resolve_thread_count", side_effect=_resolve_thread_count))
stack.enter_context(patch("app.services.detect_service.resolve_process_count", side_effect=_resolve_process_count))
stack.enter_context(patch("app.services.detect_service.get_active_detect_job_summary", return_value=active_job))
stack.enter_context(patch("app.services.detect_service.get_detect_queue_health", return_value=aggregate_queue_health))
stack.enter_context(
patch(
"app.services.detect_service._load_runtime_states_from_cluster_nodes",
return_value={
"mainland-controller-01": {
"node_code": "mainland-controller-01",
"available_proxy_count": 486,
"proxy_runtime_label": "代理正常",
"proxy_runtime_reason": "healthy",
"proxy_last_refresh_status": "复用共享代理快照 486 个",
"proxy_last_refresh_time": "2026-04-23 19:07:12",
"proxy_last_refresh_source_count": 6,
"proxy_last_refresh_total_items": 120,
"proxy_last_validated_count": 0,
}
},
)
)
stack.enter_context(patch("app.services.detect_service.sync_detect_runs", return_value=[]))
stack.enter_context(patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_dependency_alerts", return_value=[]))
stack.enter_context(patch("app.services.detect_service.append_detect_result_projection_if_changed"))
payload = detect_service.get_detect_status()
self.assertFalse(payload["worker_online"])
self.assertEqual({}, payload["runtime_state"])
self.assertEqual(3, payload["worker_process_count"])
self.assertEqual(3, payload["aggregate_process_count"])
self.assertEqual(3, payload["aggregate_participating_node_count"])
self.assertEqual(
["mainland-controller-01-a", "mainland-controller-01-b", "mainland-controller-01-c"],
payload["aggregate_participating_node_codes"],
)
self.assertEqual(3000, payload["aggregate_max_thread_count"])
self.assertEqual(38000, payload["max_thread_count"])
self.assertEqual(5909, payload["active_thread_count"])
self.assertEqual(486, payload["available_proxy_count"])
self.assertEqual("集群代理正常", payload["proxy_runtime_label"])
self.assertIn("486", payload["proxy_last_refresh_status"])
def test_get_detect_status_uses_queue_running_when_aggregate_display_fields_are_zero(self) -> None:
active_job = {
"job_id": 60,
"job_code": "sync-overseas-255",
"status": "running",
"items_total": 8402,
"items_pending": 7865,
"items_running": 537,
"items_completed": 0,
"items_failed": 0,
"items_blacklisted": 0,
"progress_percent": 6.39,
"display_items_running": 0,
"display_active_threads": 0,
"display_max_threads": 0,
"distributed_node_stats": [],
}
aggregate_queue_health = {
"has_active_job": True,
"job": {
"job_id": 60,
"job_code": "sync-overseas-255",
"status": "running",
"progress_percent": 6.39,
},
"queue": {
"items_total": 8402,
"pending": 7865,
"claimed": 0,
"running": 537,
"display_running": 0,
"completed": 0,
"blacklisted": 0,
"failed": 0,
},
"nodes": [
{
"node_code": "mainland-controller-01-a",
"items_running": 537,
"display_running": 537,
"active_threads": 537,
"max_threads": 1000,
}
],
}
with ExitStack() as stack:
stack.enter_context(patch.object(detect_service.settings, "node_region", "overseas"))
stack.enter_context(patch.object(detect_service.settings, "node_role", "control"))
stack.enter_context(patch.object(detect_service.settings, "node_code", "overseas-control-01"))
stack.enter_context(patch("app.services.detect_service.ensure_runtime_schema"))
stack.enter_context(patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")))
stack.enter_context(
patch(
"app.services.detect_service.get_settings_payload",
return_value={
"proxy_config": {"proxy_enable": False, "allow_direct": True, "proxy_urls": []},
"process_count": 80,
"node_process_counts": {"mainland-controller-01": 24},
"thread_count": 1000,
"node_thread_counts": {"overseas-control-01": 1, "mainland-controller-01-a": 1000},
},
)
)
stack.enter_context(
patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "key"})
)
stack.enter_context(
patch("app.services.detect_service._load_recent_worker_lines", return_value=(False, "2026-04-23 00:00:00", ["stale line"]))
)
stack.enter_context(
patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": False, "process_count": 0, "latest_start_time": "", "message": "inactive"})
)
stack.enter_context(
patch("app.services.detect_service._load_runtime_state", return_value={"service_running": False, "detecting": False, "active_threads": 0, "max_threads": 1, "detail": ""})
)
stack.enter_context(patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_available_proxy_count", return_value=0))
stack.enter_context(patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}))
stack.enter_context(patch("app.services.detect_service._normalize_recent_warning", return_value=""))
stack.enter_context(
patch(
"app.services.detect_service._build_proxy_runtime_snapshot",
return_value={
"state": "disabled",
"label": "未启用代理",
"detail": "-",
"direct_fallback_active": True,
"reason": "proxy_disabled",
"last_refresh_status": "",
"last_refresh_time": "",
"source_count": 0,
"raw_items": 0,
"validated_count": 0,
"available_count": 0,
"source_stats": [],
"supplier_empty": False,
},
)
)
stack.enter_context(
patch(
"app.services.detect_service.resolve_thread_count",
return_value={
"effective_thread_count": 1,
"default_thread_count": 1000,
"source": "node_override",
"override_thread_count": 1,
"node_code": "overseas-control-01",
},
)
)
stack.enter_context(
patch(
"app.services.detect_service.resolve_process_count",
return_value={
"effective_process_count": 1,
"default_process_count": 80,
"source": "node_override",
"override_process_count": 1,
"node_code": "overseas-control-01",
},
)
)
stack.enter_context(patch("app.services.detect_service.get_active_detect_job_summary", return_value=active_job))
stack.enter_context(patch("app.services.detect_service.get_detect_queue_health", return_value=aggregate_queue_health))
stack.enter_context(patch("app.services.detect_service.sync_detect_runs", return_value=[]))
stack.enter_context(patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_dependency_alerts", return_value=[]))
stack.enter_context(patch("app.services.detect_service.append_detect_result_projection_if_changed"))
payload = detect_service.get_detect_status()
self.assertEqual(537, payload["progress"]["running"])
self.assertEqual(537, payload["active_thread_count"])
self.assertEqual(1000, payload["max_thread_count"])
self.assertEqual(1, payload["aggregate_process_count"])
def test_get_detect_status_drops_stale_aggregate_nodes_from_running_and_process_counts(self) -> None:
live_heartbeat = datetime.now().isoformat(sep=" ", timespec="seconds")
stale_heartbeat = (datetime.now() - timedelta(minutes=8)).isoformat(sep=" ", timespec="seconds")
active_job = {
"job_id": 88,
"job_code": "sync-overseas-688",
"status": "running",
"items_total": 2600,
"items_pending": 500,
"items_running": 2100,
"items_completed": 0,
"items_failed": 0,
"items_blacklisted": 0,
"progress_percent": 80.77,
"display_items_running": 2100,
"display_active_threads": 2100,
"display_max_threads": 2000,
"distributed_node_stats": [
{
"node_code": "mainland-controller-01-a",
"items_running": 300,
"display_running": 300,
"active_threads": 300,
"max_threads": 1000,
"status": "busy",
"last_heartbeat_at": live_heartbeat,
},
{
"node_code": "mainland-controller-01-b",
"items_running": 1800,
"display_running": 1800,
"active_threads": 1800,
"max_threads": 1000,
"status": "stale",
"last_heartbeat_at": stale_heartbeat,
},
],
}
aggregate_queue_health = {
"has_active_job": True,
"job": {
"job_id": 88,
"job_code": "sync-overseas-688",
"status": "running",
"progress_percent": 80.77,
},
"queue": {
"items_total": 2600,
"pending": 500,
"claimed": 0,
"running": 2100,
"display_running": 2100,
"completed": 0,
"blacklisted": 0,
"failed": 0,
},
"nodes": [
{
"node_code": "mainland-controller-01-a",
"items_running": 300,
"display_running": 300,
"active_threads": 300,
"max_threads": 1000,
"status": "busy",
"last_heartbeat_at": live_heartbeat,
},
{
"node_code": "mainland-controller-01-b",
"items_running": 1800,
"display_running": 1800,
"active_threads": 1800,
"max_threads": 1000,
"status": "stale",
"last_heartbeat_at": stale_heartbeat,
},
],
}
with ExitStack() as stack:
stack.enter_context(patch.object(detect_service.settings, "node_region", "overseas"))
stack.enter_context(patch.object(detect_service.settings, "node_role", "control"))
stack.enter_context(patch.object(detect_service.settings, "node_code", "overseas-control-01"))
stack.enter_context(patch("app.services.detect_service.ensure_runtime_schema"))
stack.enter_context(patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")))
stack.enter_context(
patch(
"app.services.detect_service.get_settings_payload",
return_value={
"proxy_config": {"proxy_enable": False, "allow_direct": True, "proxy_urls": []},
"process_count": 80,
"node_process_counts": {"mainland-controller-01": 70},
"thread_count": 1000,
"node_thread_counts": {"overseas-control-01": 1},
},
)
)
stack.enter_context(
patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "key"})
)
stack.enter_context(
patch("app.services.detect_service._load_recent_worker_lines", return_value=(False, "2026-04-23 00:00:00", ["stale line"]))
)
stack.enter_context(
patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": False, "process_count": 0, "latest_start_time": "", "message": "inactive"})
)
stack.enter_context(
patch("app.services.detect_service._load_runtime_state", return_value={"service_running": False, "detecting": False, "active_threads": 0, "max_threads": 1, "detail": ""})
)
stack.enter_context(patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_available_proxy_count", return_value=0))
stack.enter_context(patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}))
stack.enter_context(patch("app.services.detect_service._normalize_recent_warning", return_value=""))
stack.enter_context(
patch(
"app.services.detect_service._build_proxy_runtime_snapshot",
return_value={
"state": "disabled",
"label": "未启用代理",
"detail": "-",
"direct_fallback_active": True,
"reason": "proxy_disabled",
"last_refresh_status": "",
"last_refresh_time": "",
"source_count": 0,
"raw_items": 0,
"validated_count": 0,
"available_count": 0,
"source_stats": [],
"supplier_empty": False,
},
)
)
stack.enter_context(
patch(
"app.services.detect_service.resolve_thread_count",
return_value={
"effective_thread_count": 1,
"default_thread_count": 1000,
"source": "node_override",
"override_thread_count": 1,
"node_code": "overseas-control-01",
},
)
)
stack.enter_context(
patch(
"app.services.detect_service.resolve_process_count",
side_effect=[
{
"effective_process_count": 1,
"default_process_count": 80,
"source": "child_instance",
"override_process_count": None,
"node_code": "mainland-controller-01-a",
}
],
)
)
stack.enter_context(patch("app.services.detect_service.get_active_detect_job_summary", return_value=active_job))
stack.enter_context(patch("app.services.detect_service.get_detect_queue_health", return_value=aggregate_queue_health))
stack.enter_context(patch("app.services.detect_service.sync_detect_runs", return_value=[]))
stack.enter_context(patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_dependency_alerts", return_value=[]))
stack.enter_context(patch("app.services.detect_service.append_detect_result_projection_if_changed"))
payload = detect_service.get_detect_status()
self.assertEqual(300, payload["progress"]["running"])
self.assertEqual(300, payload["active_thread_count"])
self.assertEqual(1000, payload["max_thread_count"])
self.assertEqual(1, payload["aggregate_process_count"])
self.assertEqual(1, payload["aggregate_participating_node_count"])
self.assertEqual(["mainland-controller-01-a"], payload["aggregate_participating_node_codes"])
def test_get_detect_status_uses_event_proxy_counts_when_cluster_runtime_proxy_counts_are_missing(self) -> None:
active_job = {
"job_id": 77,
"job_code": "sync-overseas-482",
"status": "running",
"items_pending": 410,
"items_completed": 100,
"items_failed": 0,
"items_blacklisted": 0,
"progress_percent": 55.0,
"display_items_running": 807,
"display_active_threads": 807,
"current_cycle_token": "cycle-1",
"distributed_node_stats": [
{
"node_code": "mainland-controller-01-u",
"items_running": 300,
"display_running": 300,
"active_threads": 300,
"max_threads": 1000,
},
{
"node_code": "mainland-controller-01-v",
"items_running": 301,
"display_running": 301,
"active_threads": 301,
"max_threads": 1000,
},
{
"node_code": "mainland-worker-01-a",
"items_running": 206,
"display_running": 206,
"active_threads": 206,
"max_threads": 1000,
},
],
"current_cycle_events": [
{
"event_type": "worker_log",
"created_at": "2026-04-23 19:40:00",
"node_code": "mainland-controller-01-u",
"message": "当前可用代理数: 486",
"payload": {"cycle_token": "cycle-1", "log_mode": "key"},
},
{
"event_type": "worker_log",
"created_at": "2026-04-23 19:40:10",
"node_code": "mainland-controller-01-v",
"message": "代理池刷新完成，共 486 个可用代理，来源链接 6 个，原始 520 个",
"payload": {"cycle_token": "cycle-1", "log_mode": "key"},
},
{
"event_type": "worker_log",
"created_at": "2026-04-23 19:40:20",
"node_code": "mainland-worker-01-a",
"message": "共享刷新进行中，继续沿用缓存 321 个",
"payload": {"cycle_token": "cycle-1", "log_mode": "key"},
},
],
}
aggregate_queue_health = {
"has_active_job": True,
"job": {
"job_id": 77,
"job_code": "sync-overseas-482",
"status": "running",
"progress_percent": 55.0,
},
"queue": {
"items_total": 1317,
"pending": 410,
"claimed": 0,
"running": 807,
"display_running": 807,
"completed": 100,
"blacklisted": 0,
"failed": 0,
},
"nodes": [
{
"node_code": "mainland-controller-01-u",
"items_running": 300,
"display_running": 300,
"active_threads": 300,
"max_threads": 1000,
},
{
"node_code": "mainland-controller-01-v",
"items_running": 301,
"display_running": 301,
"active_threads": 301,
"max_threads": 1000,
},
{
"node_code": "mainland-worker-01-a",
"items_running": 206,
"display_running": 206,
"active_threads": 206,
"max_threads": 1000,
},
],
}
def _resolve_thread_count(*, node_code=None, settings_payload=None):
if str(node_code or "").startswith("mainland-"):
return {
"effective_thread_count": 1000,
"default_thread_count": 1000,
"source": "default",
"override_thread_count": None,
"node_code": str(node_code or ""),
}
return {
"effective_thread_count": 1,
"default_thread_count": 1000,
"source": "node_override",
"override_thread_count": 1,
"node_code": "overseas-control-01",
}
def _resolve_process_count(*, node_code=None, settings_payload=None):
normalized_node_code = str(node_code or "")
if normalized_node_code == "mainland-controller-01":
return {
"effective_process_count": 80,
"default_process_count": 80,
"source": "default",
"override_process_count": None,
"node_code": normalized_node_code,
}
if normalized_node_code == "mainland-worker-01":
return {
"effective_process_count": 60,
"default_process_count": 60,
"source": "default",
"override_process_count": None,
"node_code": normalized_node_code,
}
if normalized_node_code.startswith("mainland-"):
return {
"effective_process_count": 1,
"default_process_count": 80,
"source": "child_instance",
"override_process_count": None,
"node_code": normalized_node_code,
}
return {
"effective_process_count": 1,
"default_process_count": 80,
"source": "node_override",
"override_process_count": 1,
"node_code": "overseas-control-01",
}
with ExitStack() as stack:
stack.enter_context(patch.object(detect_service.settings, "node_region", "overseas"))
stack.enter_context(patch.object(detect_service.settings, "node_role", "control"))
stack.enter_context(patch.object(detect_service.settings, "node_code", "overseas-control-01"))
stack.enter_context(patch("app.services.detect_service.ensure_runtime_schema"))
stack.enter_context(patch("app.services.detect_service.get_db", side_effect=RuntimeError("db down")))
stack.enter_context(
patch(
"app.services.detect_service.get_settings_payload",
return_value={
"proxy_config": {"proxy_enable": False, "allow_direct": True, "proxy_urls": []},
"process_count": 80,
"node_process_counts": {"mainland-worker-01": 60},
"thread_count": 1000,
"node_thread_counts": {"overseas-control-01": 1},
},
)
)
stack.enter_context(
patch("app.services.detect_service.get_runtime_settings", return_value={"worker_log_sync_enabled": False, "worker_log_sync_mode": "key"})
)
stack.enter_context(
patch("app.services.detect_service._load_recent_worker_lines", return_value=(False, "2026-04-23 00:00:00", ["stale line"]))
)
stack.enter_context(
patch("app.services.detect_service.detect_worker_runtime", return_value={"mode": "linux-systemd", "running": False, "process_count": 0, "latest_start_time": "", "message": "inactive"})
)
stack.enter_context(
patch("app.services.detect_service._load_runtime_state", return_value={"service_running": False, "detecting": False, "active_threads": 0, "max_threads": 1, "detail": ""})
)
stack.enter_context(patch("app.services.detect_service._load_runtime_state_from_cluster_node", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_available_proxy_count", return_value=0))
stack.enter_context(patch("app.services.detect_service._extract_active_thread_snapshot", return_value={"active": 0, "max": 0}))
stack.enter_context(patch("app.services.detect_service._normalize_recent_warning", return_value=""))
stack.enter_context(
patch(
"app.services.detect_service._build_proxy_runtime_snapshot",
return_value={
"state": "warming_up",
"label": "等待首刷",
"detail": "-",
"direct_fallback_active": False,
"reason": "waiting_for_first_refresh",
"last_refresh_status": "未刷新",
"last_refresh_time": "",
"source_count": 0,
"raw_items": 0,
"validated_count": 0,
"available_count": 0,
"source_stats": [],
"supplier_empty": False,
},
)
)
stack.enter_context(patch("app.services.detect_service.resolve_thread_count", side_effect=_resolve_thread_count))
stack.enter_context(patch("app.services.detect_service.resolve_process_count", side_effect=_resolve_process_count))
stack.enter_context(patch("app.services.detect_service.get_active_detect_job_summary", return_value=active_job))
stack.enter_context(patch("app.services.detect_service.get_detect_queue_health", return_value=aggregate_queue_health))
stack.enter_context(
patch(
"app.services.detect_service._load_runtime_states_from_cluster_nodes",
return_value={
"mainland-controller-01": {
"node_code": "mainland-controller-01",
"available_proxy_count": 0,
"proxy_runtime_label": "",
"proxy_runtime_reason": "",
"proxy_last_refresh_status": "",
"proxy_last_refresh_time": "",
"proxy_last_refresh_source_count": 0,
"proxy_last_refresh_total_items": 0,
"proxy_last_validated_count": 0,
},
"mainland-worker-01": {
"node_code": "mainland-worker-01",
"available_proxy_count": 0,
"proxy_runtime_label": "",
"proxy_runtime_reason": "",
"proxy_last_refresh_status": "",
"proxy_last_refresh_time": "",
"proxy_last_refresh_source_count": 0,
"proxy_last_refresh_total_items": 0,
"proxy_last_validated_count": 0,
},
},
)
)
stack.enter_context(patch("app.services.detect_service.sync_detect_runs", return_value=[]))
stack.enter_context(patch("app.services.detect_service._resolve_remote_log_snapshot", return_value={}))
stack.enter_context(patch("app.services.detect_service._extract_dependency_alerts", return_value=[]))
stack.enter_context(patch("app.services.detect_service.append_detect_result_projection_if_changed"))
payload = detect_service.get_detect_status()
self.assertEqual(807, payload["available_proxy_count"])
self.assertEqual("集群代理正常", payload["proxy_runtime_label"])
self.assertIn("参与服务器 2 台", payload["proxy_runtime_detail"])
if __name__ == "__main__":
unittest.main()

143
domain-api/tests/test_domains_service.py Normal file
View File

@@ -0,0 +1,143 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
from app.services import domains_service
class _FakeDomainsCursor:
def __init__(self) -> None:
self._fetchone_result = None
self._fetchall_result = []
self.executed: list[tuple[str, tuple]] = []
self.updated_detection_params = None
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def execute(self, sql: str, params=None) -> None:
normalized_sql = " ".join(str(sql or "").split()).lower()
tuple_params = tuple(params or ())
self.executed.append((normalized_sql, tuple_params))
if normalized_sql.startswith("select count(*)"):
self._fetchone_result = (1,)
return
if normalized_sql.startswith("select d.id, d.domain"):
self._fetchall_result = [
(
1,
"a.com",
0,
0,
0,
0,
1,
"",
None,
"",
0,
None,
7,
False,
None,
"",
{"status": True, "state": "passed"},
{"status": True, "state": "passed"},
True,
{"status": True, "state": "passed"},
{"status": True, "state": "passed"},
{"status": True, "state": "passed"},
{"status": True, "state": "passed"},
{"status": True, "state": "passed"},
{"status": False, "state": "failed", "message": "juziseo failed"},
{"status": False, "state": "blacklisted", "message": "jucha blacklisted"},
)
]
return
if normalized_sql.startswith("update domains set"):
return
if normalized_sql.startswith("select id, baidu_history"):
self._fetchone_result = (
7,
{"status": False, "state": "failed", "message": "timeout", "checked_at": "2026-04-20 12:00:00", "step": "baidu_site"},
{"status": True, "state": "passed", "message": "ok", "checked_at": "2026-04-20 12:00:00", "step": "baidu_site"},
False,
{"status": False, "state": "failed", "message": "old", "checked_at": "2026-04-20 12:00:00", "step": "qihu360_site"},
{"status": False, "state": "failed", "message": "old", "checked_at": "2026-04-20 12:00:00", "step": "google_site"},
False,
)
return
if normalized_sql.startswith("update domain_detections set"):
self.updated_detection_params = tuple_params
return
raise AssertionError(f"unexpected sql: {sql}")
def fetchone(self):
return self._fetchone_result
def fetchall(self):
return list(self._fetchall_result)
class _FakeDomainsConnection:
def __init__(self) -> None:
self.cursor_instance = _FakeDomainsCursor()
self.committed = False
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def cursor(self):
return self.cursor_instance
def commit(self) -> None:
self.committed = True
class DomainsServiceTests(unittest.TestCase):
def test_build_domain_query_parts_supports_false_backlink_filter(self) -> None:
_from_clause, where_clause, params = domains_service._build_domain_query_parts({"backlink_gt_10": False})
self.assertIn("coalesce(dd.backlink_count_gt_10, false) = %s", where_clause)
self.assertEqual([False], params)
@patch("app.services.domains_service.get_db")
def test_fetch_domains_step_summary_counts_juziseo_and_jucha_results(self, mock_get_db) -> None:
fake_conn = _FakeDomainsConnection()
mock_get_db.return_value = fake_conn
result = domains_service.fetch_domains(page=1, page_size=20)
summary = result["list"][0]["step_summary"]
self.assertEqual(1, summary["failed_count"])
self.assertEqual(1, summary["blacklisted_count"])
self.assertTrue(summary["has_failed"])
self.assertTrue(summary["has_blacklisted_step"])
@patch("app.services.domains_service.get_db")
def test_batch_update_domains_preserves_detection_metadata_shape(self, mock_get_db) -> None:
fake_conn = _FakeDomainsConnection()
mock_get_db.return_value = fake_conn
with patch("app.services.domains_service._now_text", return_value="2026-04-23 15:30:00"):
result = domains_service.batch_update_domains([42], {"baidu_site": "否"})
self.assertEqual(1, result["updated_count"])
self.assertIsNotNone(fake_conn.cursor_instance.updated_detection_params)
updated_payload = fake_conn.cursor_instance.updated_detection_params[0]
self.assertEqual(False, updated_payload["status"])
self.assertEqual("failed", updated_payload["state"])
self.assertEqual("人工批量更新", updated_payload["message"])
self.assertEqual("2026-04-23 15:30:00", updated_payload["checked_at"])
self.assertTrue(updated_payload["manual_override"])
if __name__ == "__main__":
unittest.main()

52
domain-api/tests/test_import_task_service.py Normal file
View File

@@ -0,0 +1,52 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
import app.services.import_task_service as import_task_service
class _TraceLock:
def __init__(self, order: list[str]) -> None:
self.order = order
def __enter__(self):
self.order.append("enter")
return self
def __exit__(self, exc_type, exc, tb):
self.order.append("exit")
return False
class ImportTaskServiceTests(unittest.TestCase):
@patch("app.services.import_task_service.import_domains_from_path")
@patch("app.services.import_task_service._update_task_with_log")
def test_run_import_task_acquires_execution_lock_before_marking_running(
self,
mock_update_task_with_log,
mock_import_domains_from_path,
) -> None:
order: list[str] = []
mock_update_task_with_log.side_effect = lambda *args, **kwargs: order.append("update")
mock_import_domains_from_path.return_value = {
"source_label": "TXT 导入",
"stats": {"total": 1, "valid": 1, "added": 1, "exists": 0, "invalid": 0},
}
with tempfile.TemporaryDirectory() as tmpdir:
path = Path(tmpdir) / "domains.txt"
path.write_text("a.com\n", encoding="utf-8")
with patch.object(import_task_service, "_IMPORT_EXECUTION_LOCK", _TraceLock(order)):
import_task_service._run_import_task("task-1", str(path), source_type=7)
self.assertEqual("enter", order[0])
self.assertIn("update", order[1:])
self.assertEqual("exit", order[-1])
if __name__ == "__main__":
unittest.main()

94
domain-api/tests/test_import_worker_service.py Normal file
View File

@@ -0,0 +1,94 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from app.services.import_worker_service import import_domains_from_path
class _FakeCursor:
def __init__(self) -> None:
self._fetchall_result = []
self._fetchone_result = None
self.inserted_domains: list[str] = []
self.inserted_detect_tasks: list[int] = []
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def execute(self, sql: str, params=None) -> None:
normalized_sql = " ".join(str(sql or "").split()).lower()
params = params or ()
if normalized_sql.startswith("select domain from domains where domain = any"):
self._fetchall_result = []
return
if normalized_sql.startswith("insert into domains"):
domain = params[0]
self.inserted_domains.append(domain)
self._fetchone_result = (len(self.inserted_domains),)
return
if normalized_sql.startswith("insert into detect_tasks"):
self.inserted_detect_tasks.append(int(params[0]))
return
raise AssertionError(f"unexpected sql: {sql}")
def fetchall(self):
return list(self._fetchall_result)
def fetchone(self):
return self._fetchone_result
class _FakeConnection:
def __init__(self) -> None:
self.cursor_instance = _FakeCursor()
self.committed = False
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
def cursor(self):
return self.cursor_instance
def commit(self) -> None:
self.committed = True
class ImportWorkerServiceTests(unittest.TestCase):
@patch("app.services.import_worker_service.get_db")
def test_import_domains_from_path_skips_duplicate_domains_in_same_batch(self, mock_get_db) -> None:
fake_conn = _FakeConnection()
mock_get_db.return_value = fake_conn
with tempfile.TemporaryDirectory() as tmpdir:
path = Path(tmpdir) / "domains.txt"
path.write_text("a.com\na.com\nb.net\ninvalid-domain\n", encoding="utf-8")
result = import_domains_from_path(path, source_type=7)
self.assertEqual(["a.com", "b.net"], fake_conn.cursor_instance.inserted_domains)
self.assertEqual([1, 2], fake_conn.cursor_instance.inserted_detect_tasks)
self.assertTrue(fake_conn.committed)
self.assertEqual(
{
"total": 4,
"valid": 3,
"added": 2,
"exists": 1,
"invalid": 1,
"failed": 0,
},
result["stats"],
)
if __name__ == "__main__":
unittest.main()

36
domain-api/tests/test_ops_api_routes.py
View File

@@ -4,6 +4,9 @@ import unittest
from unittest.mock import patch
from app.api.routes.ops import (
ops_migration_execute,
ops_migration_preview,
ops_migration_source_profile,
ops_doctor_decision,
ops_go_live_bundle,
ops_go_live_review,
@@ -13,6 +16,39 @@ from app.api.routes.ops import (
class OpsApiRoutesTestCase(unittest.TestCase):
@patch("app.api.routes.ops.get_ops_migration_source_profile")
def test_migration_source_profile_route_returns_payload(self, mock_source_profile) -> None:
mock_source_profile.return_value = {"source_node": {"node_code": "overseas-control-01"}}
response = ops_migration_source_profile()
self.assertEqual(0, response.code)
self.assertEqual("overseas-control-01", response.data["source_node"]["node_code"])
mock_source_profile.assert_called_once_with()
@patch("app.api.routes.ops.preview_ops_migration")
def test_migration_preview_route_passes_payload(self, mock_preview_ops_migration) -> None:
mock_preview_ops_migration.return_value = (True, "ok", {"plan_steps": [{"key": "sync_env_files"}]})
payload = {"target_node_code": "node-a"}
response = ops_migration_preview(payload)
self.assertEqual(0, response.code)
self.assertEqual("sync_env_files", response.data["plan_steps"][0]["key"])
mock_preview_ops_migration.assert_called_once_with(payload)
@patch("app.api.routes.ops.execute_ops_migration")
def test_migration_execute_route_returns_error_payload(self, mock_execute_ops_migration) -> None:
mock_execute_ops_migration.return_value = (False, "failed", {"blocking_reasons": ["ssh missing"]})
payload = {"target_node_code": "node-a"}
response = ops_migration_execute(payload)
self.assertEqual(1, response.code)
self.assertEqual("failed", response.message)
self.assertEqual(["ssh missing"], response.data["blocking_reasons"])
mock_execute_ops_migration.assert_called_once_with(payload)
@patch("app.api.routes.ops.get_ops_go_live_signoff")
def test_go_live_signoff_route_uses_service_payload(self, mock_get_ops_go_live_signoff) -> None:
mock_get_ops_go_live_signoff.return_value = {

290
domain-api/tests/test_ops_migration_service.py Normal file
View File

@@ -0,0 +1,290 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
from app.services.ops_migration_service import execute_ops_migration, preview_ops_migration
class OpsMigrationServiceTestCase(unittest.TestCase):
@patch("app.services.ops_migration_service._collect_remote_checks")
@patch("app.services.ops_migration_service._inspect_target_database")
@patch("app.services.ops_migration_service._find_managed_node")
def test_preview_returns_blocking_reason_when_remote_tools_missing(
self,
mock_find_managed_node,
mock_inspect_target_database,
mock_collect_remote_checks,
) -> None:
mock_find_managed_node.return_value = {
"node_code": "target-a",
"ssh_host": "10.0.0.8",
"ssh_user": "root",
"ssh_port": 22,
"auth_mode": "key",
}
mock_collect_remote_checks.return_value = {
"tools": {
"python3": True,
"node": False,
"npm": False,
"systemctl": True,
"psql": True,
"pg_dump": True,
"curl": True,
},
"paths": {
"repo_exists": True,
"repo_git": True,
"domain_root_exists": True,
"api_root_exists": True,
"web_root_exists": True,
},
"remote_db_config": {
"DB_HOST": "127.0.0.1",
"DB_PORT": "5432",
"DB_DATABASE": "domain",
"DB_USER": "domainuser",
},
"blocking_reasons": ["目标机缺少 node。", "目标机缺少 npm。"],
"warnings": [],
}
mock_inspect_target_database.return_value = {"available": False}
ok, message, data = preview_ops_migration({"target_node_code": "target-a"})
self.assertFalse(ok)
self.assertEqual("迁移预检查未通过", message)
self.assertIn("目标机缺少 node。", data["blocking_reasons"])
self.assertIn("目标机缺少 npm。", data["blocking_reasons"])
@patch("app.services.ops_migration_service._collect_remote_checks")
@patch("app.services.ops_migration_service._inspect_target_database")
@patch("app.services.ops_migration_service._find_managed_node")
def test_preview_uses_remote_env_as_target_db_default(
self,
mock_find_managed_node,
mock_inspect_target_database,
mock_collect_remote_checks,
) -> None:
mock_find_managed_node.return_value = {
"node_code": "target-a",
"ssh_host": "10.0.0.8",
"ssh_user": "root",
"ssh_port": 22,
"auth_mode": "key",
}
mock_collect_remote_checks.return_value = {
"tools": {
"python3": True,
"node": True,
"npm": True,
"systemctl": True,
"psql": True,
"pg_dump": True,
"curl": True,
},
"paths": {
"repo_exists": True,
"repo_git": True,
"domain_root_exists": True,
"api_root_exists": True,
"web_root_exists": True,
},
"git_commit": "abc123",
"remote_db_config": {
"DB_HOST": "127.0.0.1",
"DB_PORT": "5433",
"DB_DATABASE": "domain_remote",
"DB_USER": "remote_user",
},
"blocking_reasons": [],
"warnings": [],
}
mock_inspect_target_database.return_value = {
"available": True,
"has_business_data": False,
}
ok, message, data = preview_ops_migration({"target_node_code": "target-a", "overwrite_database": True})
self.assertTrue(ok)
self.assertEqual("迁移预检查完成", message)
self.assertEqual("domain_remote", data["target_db_config"]["database"])
self.assertEqual("remote_user", data["target_db_config"]["user"])
self.assertEqual(5433, data["target_db_config"]["port"])
self.assertTrue(bool(data["execution_guard"]["token"]))
@patch("app.services.ops_migration_service._run_remote_health_check")
@patch("app.services.ops_migration_service._restart_remote_services")
@patch("app.services.ops_migration_service._build_remote_frontend")
@patch("app.services.ops_migration_service._inspect_target_database")
@patch("app.services.ops_migration_service._collect_remote_checks")
@patch("app.services.ops_migration_service._find_managed_node")
def test_execute_rejects_missing_confirmation_text_for_nonempty_target_db(
self,
mock_find_managed_node,
mock_collect_remote_checks,
mock_inspect_target_database,
mock_build_remote_frontend,
mock_restart_remote_services,
mock_run_remote_health_check,
) -> None:
mock_find_managed_node.return_value = {
"node_code": "target-a",
"ssh_host": "10.0.0.8",
"ssh_user": "root",
"ssh_port": 22,
"auth_mode": "key",
}
mock_collect_remote_checks.return_value = {
"tools": {
"python3": True,
"node": True,
"npm": True,
"systemctl": True,
"psql": True,
"pg_dump": True,
"curl": True,
},
"paths": {
"repo_exists": True,
"repo_git": True,
"domain_root_exists": True,
"api_root_exists": True,
"web_root_exists": True,
},
"git_commit": "abc123",
"remote_db_config": {
"DB_HOST": "127.0.0.1",
"DB_PORT": "5432",
"DB_DATABASE": "domain_remote",
"DB_USER": "remote_user",
},
"blocking_reasons": [],
"warnings": [],
}
mock_inspect_target_database.return_value = {
"available": True,
"database": "domain_remote",
"has_business_data": True,
"public_table_count": 10,
"business_table_count": 5,
}
mock_build_remote_frontend.return_value = (True, "ok", {})
mock_restart_remote_services.return_value = (True, "ok", {})
mock_run_remote_health_check.return_value = (True, "ok", {})
preview_ok, _preview_message, preview_data = preview_ops_migration(
{"target_node_code": "target-a", "overwrite_database": True}
)
self.assertTrue(preview_ok)
token = preview_data["execution_guard"]["token"]
required_confirmation_text = preview_data["execution_guard"]["required_confirmation_text"]
self.assertEqual("OVERWRITE domain_remote", required_confirmation_text)
execute_ok, execute_message, execute_data = execute_ops_migration(
{
"target_node_code": "target-a",
"overwrite_database": True,
"execute_confirmation_token": token,
}
)
self.assertFalse(execute_ok)
self.assertEqual("缺少数据库覆盖确认文案，执行被拒绝。", execute_message)
self.assertIn("missing execute_confirmation_text", execute_data["blocking_reasons"])
@patch("app.services.ops_migration_service._start_migration_dispatch_thread")
@patch("app.services.ops_migration_service.create_ops_job")
@patch("app.services.ops_migration_service._collect_remote_checks")
@patch("app.services.ops_migration_service._inspect_target_database")
@patch("app.services.ops_migration_service._find_managed_node")
def test_execute_creates_async_job_for_long_running_migration(
self,
mock_find_managed_node,
mock_inspect_target_database,
mock_collect_remote_checks,
mock_create_ops_job,
mock_start_thread,
) -> None:
mock_find_managed_node.return_value = {
"node_code": "target-a",
"ssh_host": "10.0.0.8",
"ssh_user": "root",
"ssh_port": 22,
"auth_mode": "key",
}
mock_collect_remote_checks.return_value = {
"tools": {
"python3": True,
"node": True,
"npm": True,
"systemctl": True,
"psql": True,
"pg_dump": True,
"curl": True,
},
"paths": {
"repo_exists": True,
"repo_git": True,
"domain_root_exists": True,
"api_root_exists": True,
"web_root_exists": True,
},
"git_commit": "abc123",
"remote_db_config": {
"DB_HOST": "127.0.0.1",
"DB_PORT": "5432",
"DB_DATABASE": "domain_remote",
"DB_USER": "remote_user",
},
"blocking_reasons": [],
"warnings": [],
}
mock_inspect_target_database.return_value = {
"available": True,
"database": "domain_remote",
"has_business_data": False,
"public_table_count": 0,
"business_table_count": 0,
}
mock_create_ops_job.return_value = (
True,
"ok",
{
"job": {
"id": 88,
"job_code": "ops-20260422160000-abc123",
"action": "migration.execute",
"status": "queued",
"target_node_code": "target-a",
}
},
)
preview_ok, _preview_message, preview_data = preview_ops_migration({"target_node_code": "target-a"})
self.assertTrue(preview_ok)
token = preview_data["execution_guard"]["token"]
execute_ok, execute_message, execute_data = execute_ops_migration(
{
"target_node_code": "target-a",
"execute_confirmation_token": token,
}
)
self.assertTrue(execute_ok)
self.assertEqual("迁移任务已创建，后台开始执行。", execute_message)
self.assertEqual(88, execute_data["job"]["id"])
self.assertEqual("migration.execute", mock_create_ops_job.call_args.args[0]["action"])
self.assertFalse(bool(mock_create_ops_job.call_args.args[0]["run_now"]))
self.assertEqual("control-plane", mock_create_ops_job.call_args.args[0]["execution_mode"])
self.assertEqual("", mock_create_ops_job.call_args.args[0]["payload"]["target_db_password"])
mock_start_thread.assert_called_once_with(88)
if __name__ == "__main__":
unittest.main()

62
domain-api/tests/test_ops_release_executor_core.py
View File

@@ -9,9 +9,11 @@ from pathlib import Path
from unittest.mock import patch
from app.services.ops_release_executor_core import (
_systemd_dropin_content,
_pick_release_owner_group,
build_remote_release_action_script,
execute_release_action,
normalize_release_health_check_urls,
)
@@ -36,6 +38,7 @@ def _build_release_archive() -> bytes:
"README.txt": b"hello-release",
"domain-api/deploy/systemd/domain-node-agent.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-worker\n",
"domain-api/deploy/systemd/domain-worker.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-worker\n",
"domain-api/deploy/systemd/domain-worker@.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-worker-%i\n",
"domain-api/deploy/systemd/domain-api.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-api\n",
"domain-api/deploy/systemd/domain-sync-agent.service": b"[Service]\nEnvironmentFile=-/etc/default/domaincheck-worker\n",
}
@@ -47,6 +50,42 @@ def _build_release_archive() -> bytes:
class OpsReleaseExecutorCoreTests(unittest.TestCase):
def test_normalize_release_health_check_urls_rewrites_runtime_status_probe(self) -> None:
self.assertEqual(
[
"http://127.0.0.1:8100/health",
"http://127.0.0.1:8100/health",
"https://example.com/custom-health",
],
normalize_release_health_check_urls(
[
"http://127.0.0.1:8100/api/v1/runtime/status",
"http://127.0.0.1:8100/runtime/status?full=1",
"https://example.com/custom-health",
]
),
)
def test_api_service_template_limits_graceful_shutdown(self) -> None:
service_text = Path("domain-api/deploy/systemd/domain-api.service").read_text(encoding="utf-8")
self.assertIn("--timeout-graceful-shutdown 15", service_text)
self.assertIn("TimeoutStopSec=20", service_text)
def test_worker_service_template_uses_current_symlink(self) -> None:
service_text = Path("domain-api/deploy/systemd/domain-worker.service").read_text(encoding="utf-8")
self.assertIn("WorkingDirectory=/opt/domaincheck/current/domainCheck", service_text)
self.assertIn(
"ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python /opt/domaincheck/current/domainCheck/detect_worker.py",
service_text,
)
def test_api_service_dropin_uses_graceful_shutdown_timeout(self) -> None:
dropin_text = _systemd_dropin_content("domaincheck-api", "/opt/domaincheck")
self.assertIn("--timeout-graceful-shutdown 15", dropin_text)
def test_pick_release_owner_group_prefers_service_identity_over_path_owner(self) -> None:
with tempfile.TemporaryDirectory() as tmpdir:
with patch(
@@ -65,16 +104,29 @@ class OpsReleaseExecutorCoreTests(unittest.TestCase):
def run_command(command: list[str], *, timeout: int = 60):
commands.append(tuple(command))
if command[:5] == ["systemctl", "list-units", "--type=service", "--all", "domaincheck-worker@*"]:
return 0, (
"domaincheck-worker@a.service loaded active running domainCheck Worker Instance a\n"
"domaincheck-worker@ah.service loaded inactive dead domainCheck Worker Instance ah\n"
), ""
if command[:3] == ["systemctl", "show", "domaincheck-worker"] and "-p" in command:
current_user = pwd.getpwuid(os.getuid()).pw_name
current_group = grp.getgrgid(os.getgid()).gr_name
return 0, f"{current_user}\n{current_group}\n", ""
if command[:3] == ["systemctl", "show", "domaincheck-worker@a.service"] and "-p" in command:
current_user = pwd.getpwuid(os.getuid()).pw_name
current_group = grp.getgrgid(os.getgid()).gr_name
return 0, f"{current_user}\n{current_group}\n", ""
if command[:3] == ["systemctl", "show", "domaincheck-worker@ah.service"] and "-p" in command:
current_user = pwd.getpwuid(os.getuid()).pw_name
current_group = grp.getgrgid(os.getgid()).gr_name
return 0, f"{current_user}\n{current_group}\n", ""
if command[:2] == ["chown", "-R"]:
return 0, "", ""
if command[:2] == ["systemctl", "restart"]:
return 0, "", ""
if command[:2] == ["systemctl", "is-active"]:
return 0, "active", ""
return 0, "\n".join("active" for _ in command[2:]), ""
return 0, "", ""
with tempfile.TemporaryDirectory() as tmpdir:
@@ -108,8 +160,13 @@ class OpsReleaseExecutorCoreTests(unittest.TestCase):
self.assertTrue(any(cmd[:2] == ("chown", "-R") for cmd in commands))
self.assertIn(("systemctl", "daemon-reload"), commands)
self.assertIn(("systemctl", "restart", "domaincheck-worker"), commands)
self.assertIn(("systemctl", "restart", "domaincheck-worker@a.service"), commands)
self.assertIn(("systemctl", "restart", "domaincheck-worker@ah.service"), commands)
self.assertIn(("systemctl", "is-active", "domaincheck-worker"), commands)
self.assertIn(("systemctl", "is-active", "domaincheck-worker@a.service"), commands)
self.assertIn(("systemctl", "is-active", "domaincheck-worker@ah.service"), commands)
self.assertTrue((systemd_root / "domaincheck-node-agent.service").exists())
self.assertTrue((systemd_root / "domaincheck-worker@.service").exists())
self.assertTrue((systemd_root / "domaincheck-node-agent.service.d" / "current-path.conf").exists())
def test_build_remote_release_action_script_is_valid_python(self) -> None:
@@ -128,6 +185,9 @@ class OpsReleaseExecutorCoreTests(unittest.TestCase):
self.assertIn("False", script)
self.assertIn("def collect_service_identity(", script)
self.assertIn("def apply_release_permissions(", script)
self.assertIn("import grp", script)
self.assertIn("import os", script)
self.assertIn("import pwd", script)
if __name__ == "__main__":

91
domain-api/tests/test_ops_release_service_execution_modes.py
View File

@@ -38,6 +38,97 @@ class _EmptyConnection:
class OpsReleaseServiceExecutionModeTests(unittest.TestCase):
@patch("app.services.ops_release_service.get_db")
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
@patch("app.services.ops_agent_service.get_managed_node_onboarding")
@patch("app.services.ops_agent_service.list_managed_nodes_with_agent_state")
@patch("app.services.ops_job_service.list_managed_nodes")
@patch("app.services.cluster_runtime_service.get_cluster_snapshot")
def test_build_rollout_target_operational_readiness_reuses_managed_node_snapshot_for_onboarding(
self,
mock_get_cluster_snapshot,
mock_list_managed_nodes,
mock_list_managed_nodes_with_agent_state,
mock_get_managed_node_onboarding,
mock_ensure_ops_agent_schema,
mock_get_db,
) -> None:
captured_nodes_payloads = []
def _fake_onboarding(node_code, **kwargs):
captured_nodes_payloads.append(kwargs.get("nodes_payload"))
return {
"onboarding_stage": {"code": "ready", "label": "已接管"},
"summary": f"{node_code} ready",
"recovery_decision": {
"action": "noop",
"label": "当前无需额外恢复动作",
"summary": "当前节点暂无需要执行的接管恢复动作。",
"command_hint": "",
"window": "none",
},
}
mock_ensure_ops_agent_schema.return_value = None
mock_get_db.return_value = _EmptyConnection()
mock_get_cluster_snapshot.return_value = {
"nodes": [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
},
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"status": "online",
"current_load": 0,
"is_effective_worker": True,
},
]
}
mock_list_managed_nodes.return_value = [
{
"node_code": "mainland-worker-01",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"ssh_host": "121.204.244.248",
"ssh_user": "root",
"metadata": {},
"last_seen_at": "",
},
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"is_enabled": True,
"ssh_host": "121.204.244.249",
"ssh_user": "root",
"metadata": {},
"last_seen_at": "",
},
]
mock_list_managed_nodes_with_agent_state.return_value = {"nodes": [{"node_code": "mainland-worker-01"}, {"node_code": "mainland-worker-02"}]}
mock_get_managed_node_onboarding.side_effect = _fake_onboarding
readiness = build_rollout_target_operational_readiness(
[
{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker"},
{"node_code": "mainland-worker-02", "region": "mainland", "role": "worker"},
],
execution_mode="remote-agent",
)
self.assertEqual(2, len(readiness["rows"]))
mock_list_managed_nodes_with_agent_state.assert_called_once()
self.assertEqual(2, len(captured_nodes_payloads))
self.assertTrue(all(payload == {"nodes": [{"node_code": "mainland-worker-01"}, {"node_code": "mainland-worker-02"}]} for payload in captured_nodes_payloads))
@patch("app.services.ops_release_service.get_db")
@patch("app.services.ops_agent_service.ensure_ops_agent_schema")
@patch("app.services.ops_agent_service.get_managed_node_onboarding")

310
domain-api/tests/test_ops_schema_init.py
View File

@@ -3,6 +3,8 @@ from unittest.mock import MagicMock, patch
import app.services.ops_agent_service as ops_agent_service
import app.services.ops_job_service as ops_job_service
import app.services.ops_release_service as ops_release_service
from psycopg2 import errors
class OpsSchemaInitTests(unittest.TestCase):
@@ -10,6 +12,7 @@ class OpsSchemaInitTests(unittest.TestCase):
def test_ensure_ops_schema_uses_advisory_lock_and_skips_after_ready(self, mock_get_db) -> None:
conn = MagicMock()
cursor = MagicMock()
cursor.fetchone.return_value = None
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
@@ -35,6 +38,94 @@ class OpsSchemaInitTests(unittest.TestCase):
cursor.execute.assert_any_call(ops_job_service._OPS_SCHEMA_SQL)
conn.commit.assert_called_once()
@patch("app.services.ops_job_service.get_db")
def test_ensure_ops_schema_skips_ddl_when_required_schema_already_exists(self, mock_get_db) -> None:
conn = MagicMock()
cursor = MagicMock()
cursor.fetchone.side_effect = [(f"public.{name}",) for name in ops_job_service._OPS_REQUIRED_TABLES]
cursor.fetchall.side_effect = [
[(column,) for column in ops_job_service._OPS_REQUIRED_COLUMNS["ops_jobs"]],
[(column,) for column in ops_job_service._OPS_REQUIRED_COLUMNS["ops_job_steps"]],
]
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
db_ctx.__exit__.return_value = False
cursor_ctx.__enter__.return_value = cursor
cursor_ctx.__exit__.return_value = False
conn.cursor.return_value = cursor_ctx
mock_get_db.return_value = db_ctx
previous_ready = ops_job_service._OPS_SCHEMA_READY
ops_job_service._OPS_SCHEMA_READY = False
try:
ops_job_service.ensure_ops_schema()
finally:
ops_job_service._OPS_SCHEMA_READY = previous_ready
self.assertFalse(any(call.args[0] == ops_job_service._OPS_SCHEMA_SQL for call in cursor.execute.call_args_list))
conn.commit.assert_not_called()
@patch("app.services.ops_job_service.get_db")
def test_ensure_ops_schema_accepts_deadlock_when_required_schema_already_exists(self, mock_get_db) -> None:
class _Cursor:
def __init__(self, *, raise_on_schema=False, fetchone_values=None, fetchall_values=None) -> None:
self.raise_on_schema = raise_on_schema
self.fetchone_values = list(fetchone_values or [])
self.fetchall_values = list(fetchall_values or [])
def execute(self, sql, params=None):
if self.raise_on_schema and sql == ops_job_service._OPS_SCHEMA_SQL:
raise errors.DeadlockDetected()
def fetchone(self):
if self.fetchone_values:
return self.fetchone_values.pop(0)
return None
def fetchall(self):
if self.fetchall_values:
return self.fetchall_values.pop(0)
return []
class _CursorContext:
def __init__(self, cursor) -> None:
self.cursor = cursor
def __enter__(self):
return self.cursor
def __exit__(self, exc_type, exc, tb):
return False
conn = MagicMock()
conn.cursor.side_effect = [
_CursorContext(_Cursor(fetchone_values=[None])),
_CursorContext(_Cursor(raise_on_schema=True)),
_CursorContext(
_Cursor(
fetchone_values=[(f"public.{name}",) for name in ops_job_service._OPS_REQUIRED_TABLES],
fetchall_values=[
[(column,) for column in ops_job_service._OPS_REQUIRED_COLUMNS["ops_jobs"]],
[(column,) for column in ops_job_service._OPS_REQUIRED_COLUMNS["ops_job_steps"]],
],
)
),
]
db_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
mock_get_db.return_value = db_ctx
previous_ready = ops_job_service._OPS_SCHEMA_READY
ops_job_service._OPS_SCHEMA_READY = False
try:
ops_job_service.ensure_ops_schema()
finally:
ops_job_service._OPS_SCHEMA_READY = previous_ready
conn.rollback.assert_called_once()
conn.commit.assert_not_called()
@patch("app.services.ops_agent_service.ensure_ops_schema")
@patch("app.services.ops_agent_service.get_db")
def test_ensure_ops_agent_schema_uses_advisory_lock_and_skips_after_ready(
@@ -44,6 +135,7 @@ class OpsSchemaInitTests(unittest.TestCase):
) -> None:
conn = MagicMock()
cursor = MagicMock()
cursor.fetchone.return_value = None
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
@@ -70,6 +162,224 @@ class OpsSchemaInitTests(unittest.TestCase):
cursor.execute.assert_any_call(ops_agent_service._AGENT_SCHEMA_SQL)
conn.commit.assert_called_once()
@patch("app.services.ops_agent_service.ensure_ops_schema")
@patch("app.services.ops_agent_service.get_db")
def test_ensure_ops_agent_schema_skips_ddl_when_required_schema_already_exists(
self,
mock_get_db,
mock_ensure_ops_schema,
) -> None:
conn = MagicMock()
cursor = MagicMock()
cursor.fetchone.side_effect = [(f"public.{name}",) for name in ops_agent_service._OPS_AGENT_REQUIRED_TABLES]
cursor.fetchall.side_effect = [
[(column,) for column in ops_agent_service._OPS_AGENT_REQUIRED_COLUMNS["ops_node_tokens"]],
[(column,) for column in ops_agent_service._OPS_AGENT_REQUIRED_COLUMNS["ops_job_events"]],
[(column,) for column in ops_agent_service._OPS_AGENT_REQUIRED_COLUMNS["ops_jobs"]],
]
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
db_ctx.__exit__.return_value = False
cursor_ctx.__enter__.return_value = cursor
cursor_ctx.__exit__.return_value = False
conn.cursor.return_value = cursor_ctx
mock_get_db.return_value = db_ctx
previous_ready = ops_agent_service._OPS_AGENT_SCHEMA_READY
ops_agent_service._OPS_AGENT_SCHEMA_READY = False
try:
ops_agent_service.ensure_ops_agent_schema()
finally:
ops_agent_service._OPS_AGENT_SCHEMA_READY = previous_ready
self.assertFalse(any(call.args[0] == ops_agent_service._AGENT_SCHEMA_SQL for call in cursor.execute.call_args_list))
conn.commit.assert_not_called()
self.assertEqual(1, mock_ensure_ops_schema.call_count)
@patch("app.services.ops_agent_service.ensure_ops_schema")
@patch("app.services.ops_agent_service.get_db")
def test_ensure_ops_agent_schema_accepts_deadlock_when_required_schema_already_exists(
self,
mock_get_db,
mock_ensure_ops_schema,
) -> None:
class _Cursor:
def __init__(self, *, raise_on_schema=False, fetchone_values=None, fetchall_values=None) -> None:
self.raise_on_schema = raise_on_schema
self.fetchone_values = list(fetchone_values or [])
self.fetchall_values = list(fetchall_values or [])
def execute(self, sql, params=None):
if self.raise_on_schema and sql == ops_agent_service._AGENT_SCHEMA_SQL:
raise errors.DeadlockDetected()
def fetchone(self):
if self.fetchone_values:
return self.fetchone_values.pop(0)
return None
def fetchall(self):
if self.fetchall_values:
return self.fetchall_values.pop(0)
return []
class _CursorContext:
def __init__(self, cursor) -> None:
self.cursor = cursor
def __enter__(self):
return self.cursor
def __exit__(self, exc_type, exc, tb):
return False
conn = MagicMock()
conn.cursor.side_effect = [
_CursorContext(_Cursor(fetchone_values=[None, None])),
_CursorContext(_Cursor(raise_on_schema=True)),
_CursorContext(
_Cursor(
fetchone_values=[(f"public.{name}",) for name in ops_agent_service._OPS_AGENT_REQUIRED_TABLES],
fetchall_values=[
[(column,) for column in ops_agent_service._OPS_AGENT_REQUIRED_COLUMNS["ops_node_tokens"]],
[(column,) for column in ops_agent_service._OPS_AGENT_REQUIRED_COLUMNS["ops_job_events"]],
[(column,) for column in ops_agent_service._OPS_AGENT_REQUIRED_COLUMNS["ops_jobs"]],
],
)
),
]
db_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
mock_get_db.return_value = db_ctx
previous_ready = ops_agent_service._OPS_AGENT_SCHEMA_READY
ops_agent_service._OPS_AGENT_SCHEMA_READY = False
try:
ops_agent_service.ensure_ops_agent_schema()
finally:
ops_agent_service._OPS_AGENT_SCHEMA_READY = previous_ready
conn.rollback.assert_called_once()
conn.commit.assert_not_called()
self.assertEqual(1, mock_ensure_ops_schema.call_count)
@patch("app.services.ops_release_service.get_db")
def test_ensure_ops_release_schema_uses_advisory_lock_and_skips_after_ready(self, mock_get_db) -> None:
conn = MagicMock()
cursor = MagicMock()
cursor.fetchone.return_value = None
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
db_ctx.__exit__.return_value = False
cursor_ctx.__enter__.return_value = cursor
cursor_ctx.__exit__.return_value = False
conn.cursor.return_value = cursor_ctx
mock_get_db.return_value = db_ctx
previous_ready = ops_release_service._RELEASE_SCHEMA_READY
ops_release_service._RELEASE_SCHEMA_READY = False
try:
ops_release_service.ensure_ops_release_schema()
ops_release_service.ensure_ops_release_schema()
finally:
ops_release_service._RELEASE_SCHEMA_READY = previous_ready
self.assertEqual(1, mock_get_db.call_count)
cursor.execute.assert_any_call(
"SELECT pg_advisory_xact_lock(%s)",
(ops_release_service._RELEASE_SCHEMA_ADVISORY_LOCK_KEY,),
)
cursor.execute.assert_any_call(ops_release_service._RELEASE_SCHEMA_SQL)
conn.commit.assert_called_once()
@patch("app.services.ops_release_service.get_db")
def test_ensure_ops_release_schema_skips_ddl_when_required_schema_already_exists(self, mock_get_db) -> None:
conn = MagicMock()
cursor = MagicMock()
cursor.fetchone.side_effect = [(f"public.{name}",) for name in ops_release_service._RELEASE_REQUIRED_TABLES]
cursor.fetchall.side_effect = [
[(column,) for column in ops_release_service._RELEASE_REQUIRED_COLUMNS["ops_release_rollouts"]],
]
db_ctx = MagicMock()
cursor_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
db_ctx.__exit__.return_value = False
cursor_ctx.__enter__.return_value = cursor
cursor_ctx.__exit__.return_value = False
conn.cursor.return_value = cursor_ctx
mock_get_db.return_value = db_ctx
previous_ready = ops_release_service._RELEASE_SCHEMA_READY
ops_release_service._RELEASE_SCHEMA_READY = False
try:
ops_release_service.ensure_ops_release_schema()
finally:
ops_release_service._RELEASE_SCHEMA_READY = previous_ready
self.assertFalse(any(call.args[0] == ops_release_service._RELEASE_SCHEMA_SQL for call in cursor.execute.call_args_list))
conn.commit.assert_not_called()
@patch("app.services.ops_release_service.get_db")
def test_ensure_ops_release_schema_accepts_deadlock_when_required_schema_already_exists(self, mock_get_db) -> None:
class _Cursor:
def __init__(self, *, raise_on_schema=False, fetchone_values=None, fetchall_values=None) -> None:
self.raise_on_schema = raise_on_schema
self.fetchone_values = list(fetchone_values or [])
self.fetchall_values = list(fetchall_values or [])
def execute(self, sql, params=None):
if self.raise_on_schema and sql == ops_release_service._RELEASE_SCHEMA_SQL:
raise errors.DeadlockDetected()
def fetchone(self):
if self.fetchone_values:
return self.fetchone_values.pop(0)
return None
def fetchall(self):
if self.fetchall_values:
return self.fetchall_values.pop(0)
return []
class _CursorContext:
def __init__(self, cursor) -> None:
self.cursor = cursor
def __enter__(self):
return self.cursor
def __exit__(self, exc_type, exc, tb):
return False
conn = MagicMock()
conn.cursor.side_effect = [
_CursorContext(_Cursor(fetchone_values=[None])),
_CursorContext(_Cursor(raise_on_schema=True)),
_CursorContext(
_Cursor(
fetchone_values=[(f"public.{name}",) for name in ops_release_service._RELEASE_REQUIRED_TABLES],
fetchall_values=[
[(column,) for column in ops_release_service._RELEASE_REQUIRED_COLUMNS["ops_release_rollouts"]],
],
)
),
]
db_ctx = MagicMock()
db_ctx.__enter__.return_value = conn
mock_get_db.return_value = db_ctx
previous_ready = ops_release_service._RELEASE_SCHEMA_READY
ops_release_service._RELEASE_SCHEMA_READY = False
try:
ops_release_service.ensure_ops_release_schema()
finally:
ops_release_service._RELEASE_SCHEMA_READY = previous_ready
conn.rollback.assert_called_once()
conn.commit.assert_not_called()
if __name__ == "__main__":
unittest.main()

29
domain-api/tests/test_ops_service_activity.py
View File

@@ -10,6 +10,35 @@ class OpsServiceActivityTests(unittest.TestCase):
def _runtime_status(detect: Optional[dict] = None) -> dict:
return {"detect": dict(detect or {})}
@patch("app.services.ops_service.get_ops_runbook", return_value={"control_sequences": []})
@patch("app.services.ops_service.list_release_rollouts", return_value=[])
@patch("app.services.ops_service.list_ops_jobs", return_value=[])
@patch("app.services.ops_service.get_recent_ops_playbook_runs", return_value={"runs": []})
@patch("app.services.ops_service.list_managed_nodes_with_agent_state", side_effect=AssertionError("should reuse managed nodes"))
@patch("app.services.ops_service.get_runtime_status", side_effect=AssertionError("should reuse runtime status"))
def test_activity_stream_reuses_provided_runtime_and_managed_snapshots(
self,
_mock_get_runtime_status,
_mock_list_managed_nodes_with_agent_state,
_mock_get_recent_ops_playbook_runs,
_mock_list_ops_jobs,
_mock_list_release_rollouts,
mock_get_ops_runbook,
) -> None:
runtime_snapshot = self._runtime_status()
managed_snapshot = {"nodes": []}
payload = get_ops_activity_stream(
limit=10,
scan_limit=20,
runtime_status=runtime_snapshot,
managed_nodes_payload=managed_snapshot,
)
self.assertEqual([], payload["items"])
self.assertEqual(0, payload["summary"]["total"])
self.assertEqual(runtime_snapshot, mock_get_ops_runbook.call_args.kwargs["runtime_status"])
self.assertEqual(managed_snapshot, mock_get_ops_runbook.call_args.kwargs["managed_nodes_payload"])
@patch("app.services.ops_service.get_ops_runbook")
@patch("app.services.ops_service.list_release_rollouts")
@patch("app.services.ops_service.list_ops_jobs")

27
domain-api/tests/test_redis_client.py Normal file
View File

@@ -0,0 +1,27 @@
import unittest
from unittest.mock import patch
from app.core.redis_client import get_redis, reset_redis_client_for_tests
class ApiRedisClientTests(unittest.TestCase):
def tearDown(self) -> None:
reset_redis_client_for_tests()
@patch("app.core.redis_client.redis.Redis")
@patch("app.core.redis_client.redis.BlockingConnectionPool")
def test_get_redis_reuses_singleton_client(self, mock_pool, mock_redis) -> None:
singleton = object()
mock_redis.return_value = singleton
client_a = get_redis()
client_b = get_redis()
self.assertIs(client_a, singleton)
self.assertIs(client_b, singleton)
mock_pool.assert_called_once()
mock_redis.assert_called_once()
if __name__ == "__main__":
unittest.main()

93
domain-api/tests/test_runtime_api_routes.py Normal file
View File

@@ -0,0 +1,93 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
from app.api.routes import runtime as runtime_route
class RuntimeApiRoutesTests(unittest.TestCase):
@patch("app.api.routes.runtime.list_debug_events")
def test_runtime_debug_events_forwards_node_code_filter(self, mock_list_debug_events) -> None:
mock_list_debug_events.return_value = {"records": [], "has_more": False}
response = runtime_route.runtime_debug_events(node_code="mainland-controller-01-a", limit=20)
self.assertEqual(0, response.code)
mock_list_debug_events.assert_called_once_with(
limit=20,
service=None,
event_type=None,
source_region=None,
node_code="mainland-controller-01-a",
level=None,
before_id=None,
after_id=None,
created_after=None,
)
@patch("app.api.routes.runtime.get_runtime_status")
@patch("app.api.routes.runtime.get_sync_summary")
@patch("app.api.routes.runtime.get_debug_handoff_report")
def test_runtime_health_handover_filters_recent_issues_and_issue_groups_by_node_code(
self,
mock_get_debug_handoff_report,
mock_get_sync_summary,
mock_get_runtime_status,
) -> None:
mock_get_sync_summary.return_value = {"latest_record": {}}
mock_get_runtime_status.return_value = {"readiness": {"status": "ready"}}
mock_get_debug_handoff_report.return_value = {
"overview": {
"recent_issues": [
{"node_code": "mainland-controller-01", "message": "keep"},
{"node_code": "mainland-worker-01", "message": "drop"},
]
},
"recent_issues": [
{"node_code": "mainland-controller-01", "message": "keep"},
{"node_code": "mainland-worker-01", "message": "drop"},
],
"issue_groups": [
{"node_code": "mainland-controller-01", "message": "keep"},
{"node_code": "mainland-worker-01", "message": "drop"},
],
"failure_handoff": {
"recent_issues": [
{"node_code": "mainland-controller-01", "message": "keep"},
{"node_code": "mainland-worker-01", "message": "drop"},
],
"issue_groups": [
{"node_code": "mainland-controller-01", "message": "keep"},
{"node_code": "mainland-worker-01", "message": "drop"},
],
},
}
response = runtime_route.runtime_health_handover(node_code="mainland-controller-01")
self.assertEqual(0, response.code)
self.assertEqual(
[{"node_code": "mainland-controller-01", "message": "keep"}],
response.data["recent_issues"],
)
self.assertEqual(
[{"node_code": "mainland-controller-01", "message": "keep"}],
response.data["issue_groups"],
)
self.assertEqual(
[{"node_code": "mainland-controller-01", "message": "keep"}],
response.data["overview"]["recent_issues"],
)
self.assertEqual(
[{"node_code": "mainland-controller-01", "message": "keep"}],
response.data["failure_handoff"]["recent_issues"],
)
self.assertEqual(
[{"node_code": "mainland-controller-01", "message": "keep"}],
response.data["failure_handoff"]["issue_groups"],
)
if __name__ == "__main__":
unittest.main()

37
domain-api/tests/test_runtime_control_service.py Normal file
View File

@@ -0,0 +1,37 @@
from __future__ import annotations
import unittest
from unittest.mock import patch
from app.services.runtime_control_service import runtime_action
class RuntimeControlServiceTests(unittest.TestCase):
@patch("app.services.runtime_control_service._emit_runtime_action_event")
@patch("app.services.runtime_control_service.send_worker_command")
def test_runtime_action_stop_detection_forwards_target_payload(
self,
mock_send_worker_command,
_mock_emit_runtime_action_event,
) -> None:
mock_send_worker_command.return_value = (True, "已发送 Worker 控制指令")
ok, message, data = runtime_action(
"stop_detection",
payload={"target_node_codes": ["mainland-controller-01-a", "mainland-controller-01-b"]},
)
self.assertTrue(ok)
self.assertEqual("已发送 Worker 控制指令", message)
mock_send_worker_command.assert_called_once_with(
"stop_detection",
payload={"target_node_codes": ["mainland-controller-01-a", "mainland-controller-01-b"]},
)
self.assertEqual(
["mainland-controller-01-a", "mainland-controller-01-b"],
data["payload"]["target_node_codes"],
)
if __name__ == "__main__":
unittest.main()

77
domain-api/tests/test_runtime_settings_service.py Normal file
View File

@@ -0,0 +1,77 @@
from __future__ import annotations
import json
import unittest
from unittest.mock import Mock, patch
from app.services.runtime_settings_service import update_runtime_settings
class RuntimeSettingsServiceTests(unittest.TestCase):
@patch("app.services.runtime_settings_service.get_redis")
@patch("app.services.runtime_settings_service.write_runtime_json")
@patch("app.services.runtime_settings_service.get_runtime_settings")
def test_update_runtime_settings_publishes_runtime_settings_refresh(
self,
mock_get_runtime_settings,
mock_write_runtime_json,
mock_get_redis,
) -> None:
redis_client = Mock()
mock_get_redis.return_value = redis_client
mock_get_runtime_settings.return_value = {
"worker_mode": "linux-systemd",
"worker_service_name": "domaincheck-worker",
"api_service_name": "domaincheck-api",
"sync_agent_service_name": "domaincheck-sync-agent",
"worker_log_sync_enabled": False,
"worker_log_sync_mode": "key",
"control_node_autoresume_enabled": False,
"claim_recent_jobs_first": False,
"claim_recent_jobs_limit": 8,
"claim_recent_jobs_window_hours": 24,
"claim_batch_floor": 0,
"claim_batch_ceil": 0,
"submit_backlog_floor": 0,
"submit_backlog_ceil": 0,
"dispatch_cap_multiplier": 1,
"pending_buffer_cap_multiplier": 1,
}
updated = update_runtime_settings(
{
"worker_log_sync_enabled": True,
"worker_log_sync_mode": "full",
"control_node_autoresume_enabled": True,
"claim_recent_jobs_first": True,
"claim_recent_jobs_limit": 6,
"claim_recent_jobs_window_hours": 72,
"claim_batch_floor": 200,
"claim_batch_ceil": 800,
"submit_backlog_floor": 500,
"submit_backlog_ceil": 1500,
"dispatch_cap_multiplier": 2,
"pending_buffer_cap_multiplier": 2,
}
)
mock_write_runtime_json.assert_called_once_with("runtime_settings.json", updated)
redis_client.set.assert_called_once()
redis_key, serialized = redis_client.set.call_args.args
self.assertEqual("domain_tool:runtime_settings", redis_key)
self.assertEqual(updated, json.loads(serialized))
self.assertTrue(updated["control_node_autoresume_enabled"])
self.assertTrue(updated["claim_recent_jobs_first"])
self.assertEqual(6, updated["claim_recent_jobs_limit"])
self.assertEqual(72, updated["claim_recent_jobs_window_hours"])
self.assertEqual(200, updated["claim_batch_floor"])
self.assertEqual(800, updated["claim_batch_ceil"])
self.assertEqual(500, updated["submit_backlog_floor"])
self.assertEqual(1500, updated["submit_backlog_ceil"])
self.assertEqual(2, updated["dispatch_cap_multiplier"])
self.assertEqual(2, updated["pending_buffer_cap_multiplier"])
redis_client.publish.assert_called_once_with("domain_tool:config_update", "runtime_settings")
if __name__ == "__main__":
unittest.main()

522
domain-api/tests/test_runtime_status_service.py
View File

@@ -1,10 +1,273 @@
import unittest
from datetime import datetime, timedelta
from unittest.mock import MagicMock, patch
import app.services.runtime_status_service as runtime_status_service_module
from app.services.detect_service import _slice_remote_log_lines_fairly
from app.services.runtime_status_service import _align_queue_health_with_backlog, _build_detect_node_row
from app.services.runtime_status_service import (
_align_queue_health_with_backlog,
_build_detect_node_row,
_build_detect_observation_summary,
refresh_runtime_projection_snapshot,
get_runtime_status,
)
class RuntimeStatusServiceTests(unittest.TestCase):
def setUp(self) -> None:
runtime_status_service_module._DOMAIN_INVENTORY_CACHE_VALUE = None
runtime_status_service_module._DOMAIN_INVENTORY_CACHE_EXPIRES_AT = 0.0
runtime_status_service_module._RUNTIME_STATUS_CACHE_VALUE = None
runtime_status_service_module._RUNTIME_STATUS_CACHE_EXPIRES_AT = 0.0
runtime_status_service_module._RUNTIME_STATUS_CACHE_SIGNATURE = ()
def test_build_detect_observation_summary_marks_real_running_state(self) -> None:
summary = _build_detect_observation_summary(
detect_payload={
"active_job": {
"job_id": 308,
"job_code": "sync-overseas-16432",
"status": "running",
"progress_percent": 24.6,
"items_pending": 3999,
"items_claimed": 52,
"items_running": 136,
"items_completed": 971,
"items_failed": 0,
"items_blacklisted": 0,
"display_items_running": 222,
"display_active_threads": 222,
"display_max_threads": 2000,
"processed_recent": 172,
"processed_per_minute": 11.47,
"completed_recent": 2,
"failed_recent": 169,
"blacklisted_recent": 1,
},
"active_thread_count": 222,
"max_thread_count": 2000,
"queue_health": {
"queue": {
"pending": 3999,
"claimed": 52,
"running": 136,
"display_running": 222,
"completed": 971,
"failed": 0,
"blacklisted": 0,
},
"throughput": {
"processed_recent": 172,
"processed_per_minute": 11.47,
"completed_recent": 2,
"failed_recent": 169,
"blacklisted_recent": 1,
},
"steps": [
{
"step_code": "detect_register",
"step_name": "注册检测",
"items_pending": 2100,
"items_running": 40,
"processed_recent": 88,
"failed_recent": 100,
"blacklisted_recent": 0,
}
],
},
"participating_nodes": [
{
"node_code": "mainland-controller-01-a",
"participation_label": "执行中",
"is_dispatch_active": True,
"items_claimed": 12,
"items_running": 100,
"active_threads": 100,
"max_threads": 1000,
"processed_recent": 80,
"processed_per_minute": 5.33,
},
{
"node_code": "mainland-controller-01-b",
"participation_label": "执行中",
"is_dispatch_active": True,
"items_claimed": 8,
"items_running": 122,
"active_threads": 122,
"max_threads": 1000,
"processed_recent": 92,
"processed_per_minute": 6.13,
},
],
"participation_summary": {
"dispatch_active_nodes": 2,
"participating_nodes": 2,
},
"backlog": {
"pending_total": 3999,
"claimed_total": 52,
"running_total": 136,
"completed_total": 971,
},
},
cluster_snapshot={"summary": {"online_worker_nodes": 1}},
inventory_summary={
"scope_label": "海外主库总盘子",
"authoritative": True,
"domains_total": 5690937,
"pending_total": 5690918,
"completed_total": 0,
"running_total": 7,
"blacklist_total": 0,
"failed_total": 12,
"processed_total": 12,
"remaining_total": 5690925,
},
)
self.assertEqual("running", summary["state"])
self.assertEqual("真跑中", summary["state_label"])
self.assertEqual(2, summary["execution"]["active_processes"])
self.assertEqual(222, summary["execution"]["active_threads"])
self.assertEqual(169, summary["throughput"]["failed_recent"])
self.assertEqual("detect_register", summary["top_steps"][0]["step_code"])
self.assertEqual(5690937, summary["source_inventory"]["domains_total"])
self.assertEqual(5244, summary["active_batch"]["effective_items_total"])
self.assertIn("不等于全盘累计", summary["scope_hint"])
def test_build_detect_observation_summary_marks_not_running_when_only_backlog_left(self) -> None:
summary = _build_detect_observation_summary(
detect_payload={
"active_job": {
"job_id": 401,
"job_code": "sync-overseas-404",
"status": "running",
"items_pending": 5000,
"items_claimed": 0,
"items_running": 0,
"items_completed": 0,
"items_failed": 0,
"items_blacklisted": 0,
},
"queue_health": {
"queue": {
"pending": 5000,
"claimed": 0,
"running": 0,
"completed": 0,
"failed": 0,
"blacklisted": 0,
},
"throughput": {
"processed_recent": 0,
"processed_per_minute": 0,
"completed_recent": 0,
"failed_recent": 0,
"blacklisted_recent": 0,
},
"steps": [
{
"step_code": "detect_register",
"step_name": "注册检测",
"items_pending": 5000,
"items_running": 0,
}
],
},
"participating_nodes": [],
"participation_summary": {
"dispatch_active_nodes": 0,
"participating_nodes": 0,
},
"backlog": {
"pending_total": 5000,
"claimed_total": 0,
"running_total": 0,
},
},
cluster_snapshot={"summary": {"online_worker_nodes": 1}},
inventory_summary={
"scope_label": "海外主库总盘子",
"authoritative": True,
"domains_total": 5690937,
"pending_total": 5690918,
"completed_total": 0,
"running_total": 7,
"blacklist_total": 0,
"failed_total": 12,
"processed_total": 12,
"remaining_total": 5690925,
},
)
self.assertEqual("not_running", summary["state"])
self.assertEqual("没跑起来", summary["state_label"])
self.assertEqual(0, summary["execution"]["active_processes"])
self.assertIn("队列", summary["focus_hint"])
self.assertEqual(5000, summary["active_batch"]["effective_items_total"])
self.assertEqual(5690937, summary["source_inventory"]["domains_total"])
@patch("app.services.runtime_status_service.get_active_detect_job_summary")
@patch("app.services.runtime_status_service.get_db")
def test_load_detect_backlog_snapshot_prefers_active_job_snapshot(
self,
mock_get_db,
mock_get_active_detect_job_summary,
) -> None:
mock_get_active_detect_job_summary.return_value = {
"items_pending": 321,
"items_claimed": 2,
"items_running": 9,
"items_completed": 50,
"items_failed": 3,
"display_active_threads": 11,
"step_stats": [
{"step_code": "detect_register", "items_pending": 210},
{"step_code": "detect_baidu", "items_pending": 111},
],
}
backlog = runtime_status_service_module._load_detect_backlog_snapshot()
self.assertEqual(321, backlog["pending_total"])
self.assertEqual(2, backlog["claimed_total"])
self.assertEqual(11, backlog["running_total"])
self.assertEqual(50, backlog["completed_total"])
self.assertEqual(3, backlog["failed_total"])
self.assertEqual(210, backlog["register_pending"])
self.assertEqual(111, backlog["downstream_pending"])
mock_get_db.assert_not_called()
@patch("app.services.runtime_status_service.get_active_detect_job_summary", return_value={})
@patch("app.services.runtime_status_service.get_db")
def test_load_detect_backlog_snapshot_ignores_stale_pending_jobs(
self,
mock_get_db,
_mock_get_active_detect_job_summary,
) -> None:
conn = MagicMock()
cursor_cm = MagicMock()
cursor = MagicMock()
cursor.fetchall.return_value = [
(11, "running", datetime.now() - timedelta(hours=2)),
(10, "pending", datetime.now() - timedelta(minutes=30)),
(9, "pending", datetime.now() - timedelta(hours=8)),
]
cursor.fetchone.return_value = (321, 2, 9, 50, 1, 3, 210, 111)
conn.cursor.return_value = cursor_cm
cursor_cm.__enter__.return_value = cursor
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
mock_get_db.return_value = db_cm
backlog = runtime_status_service_module._load_detect_backlog_snapshot()
self.assertEqual(321, backlog["pending_total"])
self.assertEqual(210, backlog["register_pending"])
executed_sql, executed_params = cursor.execute.call_args_list[1][0]
self.assertIn("WHERE item.job_id = ANY(%s)", executed_sql)
self.assertEqual([11, 10], list(executed_params[0]))
def test_build_detect_node_row_merges_queue_running_metrics(self) -> None:
row = _build_detect_node_row(
node_code="mainland-worker-01",
@@ -92,6 +355,42 @@ class RuntimeStatusServiceTests(unittest.TestCase):
self.assertTrue(any("[mainland-controller-01]" in line for line in sliced))
self.assertTrue(any("[mainland-worker-01]" in line for line in sliced))
def test_build_multi_region_readiness_skips_local_batch_warning_when_batches_are_not_applicable(self) -> None:
with patch("app.services.runtime_status_service.settings.node_region", "overseas"), \
patch("app.services.runtime_status_service.settings.node_role", "control"):
readiness = runtime_status_service_module._build_multi_region_readiness(
cluster_snapshot={
"nodes": [
{"node_code": "overseas-control-01", "region": "overseas", "role": "control", "status": "online"},
{"node_code": "mainland-controller-01", "region": "mainland", "role": "control", "status": "online"},
{"node_code": "mainland-worker-01", "region": "mainland", "role": "worker", "status": "busy", "is_effective_worker": True},
],
"summary": {
"online_control_nodes": 2,
"online_worker_nodes": 1,
},
},
sync_summary={
"enabled": False,
"source_region": "overseas",
"target_region": "mainland",
"detect_result_batches": {
"applicable": False,
"reason": "当前节点不承载本地检测执行，结果批次推送概览不适用。",
"state_counts": {"projected": 3, "failed": 1, "pushing": 1, "synced": 0},
},
},
worker_runtime={"running": False},
sync_agent_runtime={"running": False},
)
warning_text = " ".join(readiness["warnings"])
info_text = " ".join(readiness["info"])
self.assertNotIn("结果批次仍待推送", warning_text)
self.assertNotIn("结果批次同步失败", warning_text)
self.assertIn("不适用", info_text)
self.assertFalse(readiness["sync"]["applicable"])
def test_align_queue_health_with_backlog_prefers_larger_runtime_snapshot(self) -> None:
aligned = _align_queue_health_with_backlog(
{
@@ -126,6 +425,227 @@ class RuntimeStatusServiceTests(unittest.TestCase):
self.assertEqual(8, aligned["queue"]["failed"])
self.assertEqual(1790, aligned["queue"]["items_total"])
def test_load_latest_remote_runtime_projection_backlog_skips_future_dated_rows(self) -> None:
now = datetime.now()
conn = MagicMock()
cursor_cm = MagicMock()
cursor = MagicMock()
cursor.fetchall.return_value = [
({"projection": {"backlog": {"pending_total": 9999}}}, now + timedelta(hours=4)),
({"projection": {"backlog": {"pending_total": 321, "running_total": 8}}}, now - timedelta(minutes=2)),
]
conn.cursor.return_value = cursor_cm
cursor_cm.__enter__.return_value = cursor
db_cm = MagicMock()
db_cm.__enter__.return_value = conn
with patch("app.services.runtime_status_service.settings.node_region", "overseas"), \
patch("app.services.runtime_status_service.settings.node_role", "control"), \
patch("app.services.runtime_status_service.get_db", return_value=db_cm):
backlog = runtime_status_service_module._load_latest_remote_runtime_projection_backlog()
self.assertEqual(321, backlog["pending_total"])
self.assertEqual(8, backlog["running_total"])
def test_get_runtime_status_keeps_remote_aggregate_detect_metrics_on_overseas_control(self) -> None:
with patch("app.services.runtime_status_service.settings.node_region", "overseas"), \
patch("app.services.runtime_status_service.settings.node_role", "control"), \
patch("app.services.runtime_status_service.detect_worker_runtime", return_value={"running": False, "mode": "linux-systemd", "process_count": 0, "message": "inactive", "latest_start_time": ""}), \
patch("app.services.runtime_status_service.detect_sync_agent_runtime", return_value={"running": False, "mode": "linux-systemd", "process_count": 0, "message": "inactive", "latest_start_time": ""}), \
patch("app.services.runtime_status_service.get_runtime_settings", return_value={"worker_mode": "linux-systemd"}), \
patch("app.services.runtime_status_service.get_detect_status", return_value={
"phase_label": "",
"phase_detail": "",
"recent_event": "远端执行中",
"recent_warning": "",
"progress_percent": 40.49,
"progress": {"pending": 5001, "running": 138, "completed": 0, "failed": 0, "blacklisted": 0},
"active_thread_count": 138,
"max_thread_count": 80000,
"aggregate_process_count": 80,
"aggregate_participating_node_count": 1,
"aggregate_participating_node_codes": ["mainland-controller-01"],
"aggregate_max_thread_count": 80000,
"aggregate_thread_count_per_process": 1000,
"available_proxy_count": 0,
"proxy_pool_count": 0,
"proxy_runtime_label": "",
"proxy_runtime_detail": "",
"proxy_runtime_reason": "",
"proxy_supplier_empty": False,
"proxy_last_refresh_status": "",
"proxy_last_refresh_time": "",
"proxy_last_refresh_source_count": 0,
"proxy_last_refresh_total_items": 0,
"proxy_last_validated_count": 0,
"proxy_last_available_count": 0,
"proxy_source_stats": [],
"dependency_alerts": [],
"active_job": {"job_code": "sync-overseas-51", "items_pending": 5001, "items_running": 0},
"runs": [],
"remote_log_line_count": 0,
"remote_log_node_count": 0,
"remote_log_nodes": [],
"remote_log_node_summaries": [],
"remote_log_last_at": "",
"remote_log_last_line": "",
"remote_log_lines": [],
"aggregate_detect_view": True,
}), \
patch("app.services.runtime_status_service.get_cluster_snapshot", return_value={
"nodes": [{"node_code": "mainland-controller-01", "role": "control", "region": "mainland", "status": "busy", "is_effective_worker": True, "current_load": 1432, "metadata": {"active_threads": 138, "max_threads": 80000}}],
"summary": {"online_worker_nodes": 1, "dedicated_online_worker_nodes": 0, "online_control_nodes": 2},
}), \
patch("app.services.runtime_status_service.get_detect_queue_health", return_value={
"has_active_job": True,
"job": {"job_id": 11, "job_code": "sync-overseas-51", "status": "running", "progress_percent": 40.49},
"queue": {"items_total": 8402, "pending": 5001, "claimed": 0, "running": 1432, "display_running": 1432},
"nodes": [{"node_code": "mainland-controller-01-a", "items_running": 1432, "display_running": 1432, "active_threads": 1432, "max_threads": 1000}],
"steps": [],
"runtime_activity": {},
"window_minutes": 15,
}), \
patch("app.services.runtime_status_service._load_detect_backlog_snapshot", return_value={"pending_total": 5001, "claimed_total": 0, "running_total": 0, "completed_total": 0, "blacklisted_total": 0, "failed_total": 0, "register_pending": 3495, "downstream_pending": 1506}), \
patch("app.services.runtime_status_service._load_latest_remote_runtime_projection_backlog", return_value={}), \
patch("app.services.runtime_status_service._load_latest_runtime_active_job_snapshot", return_value={}), \
patch("app.services.runtime_status_service._load_domain_inventory_summary", return_value={"scope_label": "海外主库总盘子", "authoritative": True, "domains_total": 5690937, "pending_total": 5690918, "completed_total": 0, "running_total": 7, "blacklist_total": 0, "failed_total": 12, "processed_total": 12, "remaining_total": 5690925}), \
patch("app.services.runtime_status_service.get_detect_capacity_plan", return_value={}), \
patch("app.services.runtime_status_service.append_runtime_projection_if_changed"), \
patch("app.services.runtime_status_service.get_sync_summary", return_value={}), \
patch("app.services.runtime_status_service._build_multi_region_readiness", return_value={"status": "ready", "ready": True}), \
patch("app.services.runtime_status_service.get_runtime_build_info", return_value={}):
payload = get_runtime_status()
self.assertEqual(1, payload["detect"]["aggregate_process_count"])
self.assertEqual(1432, payload["detect"]["active_thread_count"])
self.assertEqual(1000, payload["detect"]["max_thread_count"])
self.assertEqual("集群执行中", payload["detect"]["phase_label"])
self.assertFalse(payload["detect"]["worker_online"])
self.assertEqual("sync-overseas-51", payload["detect"]["active_job"]["job_code"])
self.assertIn("observation_summary", payload["detect"])
self.assertEqual("在跑但偏慢", payload["detect"]["observation_summary"]["state_label"])
self.assertEqual(5690937, payload["detect"]["observation_summary"]["source_inventory"]["domains_total"])
def test_get_runtime_status_reuses_short_ttl_cache(self) -> None:
context_payload = {
"runtime_settings": {
"worker_mode": "linux-systemd",
"api_service_name": "domaincheck-api",
"worker_service_name": "domaincheck-worker",
"sync_agent_service_name": "domaincheck-sync-agent",
},
"worker_runtime": {
"running": True,
"mode": "linux-systemd",
"process_count": 60,
"latest_start_time": "2026-04-24 17:10:28",
"message": "running",
"runtime_state": {"phase": "detecting"},
},
"worker_expected_on_this_node": True,
"detect_payload": {
"_detect_snapshot": {"thread_count": 1000},
"active_thread_count": 321,
"max_thread_count": 60000,
"progress": {"pending": 10, "running": 3},
"backlog": {"pending_total": 10, "running_total": 3},
"progress_percent": 25.5,
"available_proxy_count": 12,
"proxy_pool_count": 20,
"proxy_runtime_label": "代理正常",
"proxy_runtime_detail": "ok",
"proxy_runtime_reason": "healthy",
"proxy_last_refresh_time": "2026-04-24 17:12:55",
"recent_event": "running",
"recent_warning": "",
},
"cluster_snapshot": {"summary": {"online_worker_nodes": 1}},
}
sync_summary = {"enabled": True}
readiness = {"status": "ready", "ready": True}
with patch("app.services.runtime_status_service._build_runtime_detect_context", return_value=context_payload) as mock_context, \
patch("app.services.runtime_status_service.detect_sync_agent_runtime", return_value={"running": True, "mode": "linux-systemd", "process_count": 1, "latest_start_time": "", "message": "ok"}), \
patch("app.services.runtime_status_service.append_runtime_projection_if_changed") as mock_append, \
patch("app.services.runtime_status_service.get_sync_summary", return_value=sync_summary) as mock_sync_summary, \
patch("app.services.runtime_status_service._build_multi_region_readiness", return_value=readiness), \
patch("app.services.runtime_status_service.get_runtime_build_info", return_value={"package_name": "pkg"}), \
patch("app.services.runtime_status_service.time.monotonic", side_effect=[100.0, 100.1, 100.2]):
first = get_runtime_status()
second = get_runtime_status()
self.assertEqual(first, second)
self.assertEqual(1, mock_context.call_count)
self.assertEqual(1, mock_append.call_count)
self.assertEqual(1, mock_sync_summary.call_count)
@patch("app.services.runtime_status_service.append_runtime_projection_if_changed", return_value=321)
@patch(
"app.services.runtime_status_service.get_cluster_snapshot",
return_value={"nodes": [], "summary": {"online_worker_nodes": 3}},
)
@patch(
"app.services.runtime_status_service.get_active_detect_job_summary",
return_value={
"job_id": 867,
"job_code": "sync-overseas-19835",
"status": "running",
"progress_percent": 99.1,
"items_total": 10000,
"items_pending": 4,
"items_claimed": 0,
"items_running": 5,
"items_completed": 55402,
"items_blacklisted": 0,
"items_failed": 0,
"display_items_running": 5,
"display_active_threads": 5,
"display_items_claimed": 0,
"display_max_threads": 1000,
"node_stats": [{"node_code": "mainland-controller-01", "items_running": 5}],
"raw_step_stats": [{"step_code": "detect_wayback", "items_pending": 4}],
},
)
@patch(
"app.services.runtime_status_service.detect_worker_runtime",
return_value={"running": True, "mode": "linux-systemd", "process_count": 60, "thread_count": 1000, "max_threads": 60000, "message": "ok"},
)
@patch("app.services.runtime_status_service.get_runtime_settings", return_value={"thread_count": 1000, "worker_mode": "linux-systemd"})
@patch(
"app.services.runtime_status_service._load_detect_backlog_snapshot",
return_value={
"pending_total": 4,
"claimed_total": 0,
"running_total": 5,
"completed_total": 55402,
"blacklisted_total": 0,
"failed_total": 0,
"register_pending": 0,
"downstream_pending": 4,
},
)
@patch("app.services.runtime_status_service._build_runtime_detect_context", side_effect=AssertionError("heavy path should not run"))
def test_refresh_runtime_projection_snapshot_uses_lightweight_context(
self,
mock_heavy_context,
mock_load_detect_backlog_snapshot,
mock_get_runtime_settings,
mock_detect_worker_runtime,
mock_get_active_detect_job_summary,
mock_get_cluster_snapshot,
mock_append_runtime_projection_if_changed,
) -> None:
result = refresh_runtime_projection_snapshot()
self.assertEqual(321, result["record_id"])
mock_append_runtime_projection_if_changed.assert_called_once()
detect = mock_append_runtime_projection_if_changed.call_args.kwargs["detect"]
self.assertEqual("sync-overseas-19835", detect["active_job"]["job_code"])
self.assertEqual(5, detect["active_thread_count"])
self.assertEqual(4, detect["backlog"]["pending_total"])
self.assertEqual(5, detect["queue_health"]["queue"]["running"])
mock_heavy_context.assert_not_called()
if __name__ == "__main__":
unittest.main()

26
domain-api/tests/test_settings_service.py
View File

@@ -1,6 +1,11 @@
import unittest
from app.services.settings_service import _normalize_thread_count, resolve_thread_count
from app.services.settings_service import (
_normalize_process_count,
_normalize_thread_count,
resolve_process_count,
resolve_thread_count,
)
class SettingsServiceTests(unittest.TestCase):
@@ -26,6 +31,25 @@ class SettingsServiceTests(unittest.TestCase):
self.assertEqual(512, resolved["effective_thread_count"])
self.assertEqual("node_override", resolved["source"])
def test_normalize_process_count_rejects_non_positive_values(self) -> None:
with self.assertRaises(ValueError):
_normalize_process_count(0)
def test_resolve_process_count_uses_large_node_override(self) -> None:
payload = {
"process_count": 80,
"node_process_counts": {
"mainland-controller-01": 96,
},
}
resolved = resolve_process_count(node_code="mainland-controller-01", settings_payload=payload)
self.assertEqual(80, resolved["default_process_count"])
self.assertEqual(96, resolved["override_process_count"])
self.assertEqual(96, resolved["effective_process_count"])
self.assertEqual("node_override", resolved["source"])
if __name__ == "__main__":
unittest.main()

137
domain-api/tests/test_sync_agent.py
View File

@@ -4,6 +4,9 @@ from unittest.mock import patch
from app.sync_agent import (
_append_detect_result_projection_snapshot,
_build_aligned_queue_health_snapshot,
_emit_runtime_debug_snapshots,
_maybe_trigger_overlap_start,
_run_sync_tick_once,
_emit_structured_tick,
_emit_sync_result_breakdown,
_filter_runtime_events_for_job,
@@ -187,6 +190,140 @@ class SyncAgentTests(unittest.TestCase):
mock_process_detect_pipeline_now.assert_called_once_with(limit=5000)
self.assertEqual("pipeline_tick_success", mock_push_debug_event.call_args.kwargs["event_type"])
@patch("app.sync_agent._load_local_detect_backlog_snapshot", return_value={"pending_total": 12})
@patch("app.sync_agent.list_recent_detect_run_events", return_value=[])
@patch("app.sync_agent.get_detect_queue_health", return_value={"queue": {"overdue_leases": 0}})
@patch("app.sync_agent.push_debug_event")
@patch("app.sync_agent._append_detect_result_projection_snapshot")
@patch("app.sync_agent._select_projection_job_snapshots", return_value=[{"job_id": 1, "job_code": "finished-job"}])
@patch(
"app.sync_agent.get_active_detect_job_summary",
return_value={
"job_id": 2,
"job_code": "running-job",
"status": "running",
"items_total": 100,
"items_pending": 10,
"items_claimed": 5,
"items_running": 7,
"items_completed": 70,
"items_failed": 8,
"progress_percent": 70.0,
"node_stats": [{"node_code": "mainland-controller-01", "items_total": 90}],
},
)
def test_emit_runtime_debug_snapshots_emits_projection_and_active_job(
self,
mock_get_active_detect_job_summary,
mock_select_projection_job_snapshots,
mock_append_detect_result_projection_snapshot,
mock_push_debug_event,
mock_get_detect_queue_health,
mock_list_recent_detect_run_events,
mock_load_local_detect_backlog_snapshot,
) -> None:
_emit_runtime_debug_snapshots()
mock_get_active_detect_job_summary.assert_called_once_with(event_limit=10)
mock_select_projection_job_snapshots.assert_called_once_with()
mock_append_detect_result_projection_snapshot.assert_called_once()
self.assertTrue(mock_push_debug_event.called)
self.assertEqual("active_job_snapshot", mock_push_debug_event.call_args_list[0].kwargs["event_type"])
@patch("app.sync_agent._emit_runtime_debug_snapshots")
@patch("app.sync_agent._run_pipeline_stage_processor", return_value=(True, "pipeline ok", {"stage": "pipeline"}))
@patch("app.sync_agent.pull_detect_task_batch_now", return_value=(True, "pull ok", {"stage": "pull"}))
@patch("app.sync_agent.push_runtime_projection_now", return_value=(True, "sync ok", {"stage": "sync"}))
def test_run_sync_tick_once_prioritizes_sync_before_pipeline(
self,
mock_push_runtime_projection_now,
mock_pull_detect_task_batch_now,
mock_run_pipeline_stage_processor,
mock_emit_runtime_debug_snapshots,
) -> None:
with patch("app.sync_agent._maybe_trigger_overlap_start", return_value=(False, "no overlap", {})) as mock_overlap:
tick = _run_sync_tick_once()
self.assertTrue(tick["sync"]["ok"])
self.assertEqual("sync ok", tick["sync"]["message"])
mock_push_runtime_projection_now.assert_called_once_with()
mock_pull_detect_task_batch_now.assert_called_once_with()
mock_run_pipeline_stage_processor.assert_called_once_with()
mock_overlap.assert_called_once_with()
mock_emit_runtime_debug_snapshots.assert_called_once_with()
self.assertEqual("no overlap", tick["overlap"]["message"])
@patch("app.sync_agent.send_worker_command")
@patch("app.sync_agent._select_overlap_target_node_codes")
@patch("app.sync_agent._select_overlap_start_candidate")
@patch("app.sync_agent.push_debug_event")
def test_maybe_trigger_overlap_start_dispatches_pending_job(
self,
mock_push_debug_event,
mock_select_overlap_start_candidate,
mock_select_overlap_target_node_codes,
mock_send_worker_command,
) -> None:
mock_select_overlap_start_candidate.return_value = {
"job_id": 885,
"job_code": "sync-overseas-20592",
"task_mode": "domain_pipeline",
"items_pending": 64000,
"items_claimed": 0,
"items_running": 0,
"selection_reason": "overlap_tail_handoff",
}
mock_select_overlap_target_node_codes.return_value = [
"mainland-controller-01-a",
"mainland-controller-01-da",
]
mock_send_worker_command.return_value = (True, "已发送 Worker 控制指令: start_detection -> mainland-controller-01-a,mainland-controller-01-da")
with patch("app.sync_agent._LAST_OVERLAP_JOB_ID", 0), patch("app.sync_agent._LAST_OVERLAP_TRIGGERED_AT", 0.0):
ok, message, data = _maybe_trigger_overlap_start()
self.assertTrue(ok)
self.assertIn("mainland-controller-01-a", message)
self.assertEqual(885, data["job_id"])
mock_send_worker_command.assert_called_once_with(
"start_detection",
payload={
"job_id": 885,
"job_code": "sync-overseas-20592",
"target_job_id": 885,
"target_job_code": "sync-overseas-20592",
"task_mode": "domain_pipeline",
"source": "overlap-handoff",
"selection_reason": "overlap_tail_handoff",
"tail_handoff_candidate": True,
"target_node_codes": [
"mainland-controller-01-a",
"mainland-controller-01-da",
],
},
)
self.assertEqual("overlap_handoff_started", mock_push_debug_event.call_args.kwargs["event_type"])
@patch("app.sync_agent._select_overlap_start_candidate")
def test_maybe_trigger_overlap_start_honors_cooldown_for_same_job(
self,
mock_select_overlap_start_candidate,
) -> None:
mock_select_overlap_start_candidate.return_value = {
"job_id": 885,
"job_code": "sync-overseas-20592",
"task_mode": "domain_pipeline",
"items_pending": 64000,
"selection_reason": "overlap_tail_handoff",
}
with patch("app.sync_agent._LAST_OVERLAP_JOB_ID", 885), patch("app.sync_agent._LAST_OVERLAP_TRIGGERED_AT", __import__('time').time()):
ok, message, data = _maybe_trigger_overlap_start()
self.assertFalse(ok)
self.assertIn("冷却中", message)
self.assertEqual(885, data["job_id"])
if __name__ == "__main__":
unittest.main()

607
domain-api/tests/test_sync_push_service.py
View File

@@ -1,10 +1,20 @@
import unittest
from datetime import datetime, timedelta
from unittest.mock import patch
from app.services.sync_push_service import (
_build_task_pull_backlog_limits,
_extract_detect_result_projection_events,
_load_latest_projection,
_load_pushable_projections,
_refresh_remote_runtime_node,
_load_local_detect_backlog_snapshot,
_push_projection_now,
_resolve_task_pull_request_limit,
_task_projection_limit,
pull_detect_task_batch_now,
_resolve_detect_result_target_job_id,
_select_relevant_backlog_job_ids_from_rows,
_should_throttle_task_pull,
ingest_runtime_projection,
)
@@ -23,6 +33,12 @@ class _FakeCursor:
return self._rows.pop(0)
return None
def fetchall(self):
if self._rows:
value = self._rows.pop(0)
return list(value or [])
return []
def __enter__(self):
return self
@@ -48,23 +64,311 @@ class _FakeConnection:
return False
class _FakeUrlopenResponse:
def __init__(self, payload):
self._payload = payload
def read(self):
import json
return json.dumps(self._payload, ensure_ascii=False).encode("utf-8")
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
class SyncPushServiceTests(unittest.TestCase):
def test_build_task_pull_backlog_limits_scales_with_thread_configuration(self) -> None:
@patch("app.services.sync_push_service.cleanup_imported_runtime_nodes_many")
@patch("app.services.sync_push_service.cleanup_imported_runtime_nodes")
@patch("app.services.sync_push_service.register_node_heartbeat")
def test_refresh_remote_runtime_node_imports_cluster_worker_rows_without_active_job(
self,
mock_register_node_heartbeat,
_mock_cleanup_imported_runtime_nodes,
mock_cleanup_imported_runtime_nodes_many,
) -> None:
_refresh_remote_runtime_node(
source_region="mainland",
projection={
"node": {
"node_code": "mainland-controller-01",
"region": "mainland",
"role": "control",
"hostname": "localhost",
"ip": "127.0.0.1",
},
"worker_mode": "linux-systemd",
"phase_label": "等待中",
"phase_detail": "等待任务",
"proxy_runtime_label": "正常",
"proxy_runtime_reason": "healthy",
"active_thread_count": 0,
"max_thread_count": 60000,
"worker_online": True,
"detect_participating": False,
"active_job": {
"job_code": "",
"status": "",
"node_stats": [],
},
"cluster_nodes": [
{
"node_code": "mainland-controller-01",
"role": "control",
"status": "online",
"current_load": 0,
"active_threads": 0,
"max_threads": 60000,
"detect_participating": False,
},
{
"node_code": "mainland-controller-01-a",
"role": "worker",
"status": "online",
"current_load": 0,
"active_threads": 0,
"max_threads": 1000,
"detect_participating": False,
},
{
"node_code": "mainland-controller-01-b",
"role": "worker",
"status": "busy",
"current_load": 12,
"active_threads": 12,
"max_threads": 1000,
"detect_participating": True,
},
],
},
)
self.assertEqual(3, mock_register_node_heartbeat.call_count)
worker_calls = [call.kwargs for call in mock_register_node_heartbeat.call_args_list[1:]]
self.assertEqual(
["mainland-controller-01-a", "mainland-controller-01-b"],
[item["node_code"] for item in worker_calls],
)
self.assertEqual(12, worker_calls[1]["current_load"])
self.assertEqual(
["mainland-controller-01-a", "mainland-controller-01-b"],
mock_cleanup_imported_runtime_nodes_many.call_args.kwargs["keep_node_codes"],
)
@patch("app.services.sync_push_service.settings")
def test_task_pull_request_limit_scales_with_estimated_threads(self, mock_settings) -> None:
mock_settings.node_code = "mainland-controller-01"
mock_settings.sync_batch_size = 5000
safe_limit = _resolve_task_pull_request_limit(
None,
settings_payload={
"thread_count": 1000,
"process_count": 1,
"node_thread_counts": {"mainland-controller-01": 1000},
"node_process_counts": {"mainland-controller-01": 60},
},
)
self.assertEqual(120000, safe_limit)
@patch("app.services.sync_push_service.settings")
def test_task_projection_limit_allows_large_explicit_pull_request(self, mock_settings) -> None:
mock_settings.sync_batch_size = 5000
self.assertEqual(120000, _task_projection_limit(120000))
@patch("app.services.sync_push_service.settings")
@patch("app.services.sync_push_service.get_db")
def test_load_latest_projection_prefers_non_future_row_over_clock_skewed_history(
self,
mock_get_db,
mock_settings,
) -> None:
now = datetime.now()
future_row = (
10081,
"mainland",
"overseas",
"projected",
{"projection": {"active_thread_count": 138}},
now + timedelta(hours=4),
now + timedelta(hours=4),
)
valid_row = (
11633,
"mainland",
"overseas",
"projected",
{"projection": {"active_thread_count": 6431}},
now - timedelta(minutes=2),
now - timedelta(minutes=2),
)
fake_conn = _FakeConnection(rows=[[future_row, valid_row]])
mock_get_db.return_value = fake_conn
mock_settings.sync_source_region = "mainland"
mock_settings.node_region = "mainland"
mock_settings.sync_target_region = "overseas"
with patch("app.services.sync_push_service.datetime") as mock_datetime:
mock_datetime.now.return_value = now
record = _load_latest_projection("runtime_projection")
self.assertIsNotNone(record)
self.assertEqual(11633, record["id"])
executed_sql, executed_params = fake_conn.cursor_obj.executed[0]
self.assertIn("CASE WHEN created_at <= %s THEN 0 ELSE 1 END", executed_sql)
self.assertEqual("runtime_projection", executed_params[0])
self.assertEqual("mainland", executed_params[1])
self.assertEqual("overseas", executed_params[2])
self.assertEqual(now + timedelta(minutes=5), executed_params[3])
@patch("app.services.sync_push_service._latest_push_attempt")
@patch("app.services.sync_push_service.settings")
@patch("app.services.sync_push_service.get_db")
def test_load_pushable_projections_prefers_latest_unsent_records(
self,
mock_get_db,
mock_settings,
mock_latest_push_attempt,
) -> None:
now = datetime.now()
old_success_rows = [
(
10000 + index,
"mainland",
"overseas",
"projected",
{"projection_hash": f"old-{index}", "projection": {"job": {"job_id": index}}},
now - timedelta(minutes=120 - index),
now - timedelta(minutes=120 - index),
)
for index in range(100)
]
newest_unsent = (
23360,
"mainland",
"overseas",
"projected",
{"projection_hash": "new-hash", "projection": {"job": {"job_id": 909, "job_code": "sync-overseas-28618"}}},
now,
now,
)
fake_conn = _FakeConnection(rows=[old_success_rows + [newest_unsent]])
mock_get_db.return_value = fake_conn
mock_settings.sync_source_region = "mainland"
mock_settings.node_region = "mainland"
mock_settings.sync_target_region = "overseas"
mock_settings.sync_batch_size = 20
def _attempt_side_effect(source_record_id, target_region, sync_type):
if source_record_id == 23360:
return None
return {"status": "success", "created_at": now}
mock_latest_push_attempt.side_effect = _attempt_side_effect
records = _load_pushable_projections("detect_result_projection", limit=20)
self.assertEqual([23360], [item["id"] for item in records])
executed_sql, _ = fake_conn.cursor_obj.executed[0]
self.assertIn("ORDER BY created_at DESC, id DESC", executed_sql)
@patch("app.services.sync_push_service.settings.node_code", "mainland-controller-01")
def test_build_task_pull_backlog_limits_scales_with_local_server_capacity(self) -> None:
limits = _build_task_pull_backlog_limits(
5000,
settings_payload={
"thread_count": 100,
"process_count": 80,
"node_thread_counts": {
"mainland-controller-01": 2000,
"mainland-controller-01": 1000,
"mainland-controller-01-a": 1000,
"mainland-worker-01": 1200,
},
"node_process_counts": {
"mainland-controller-01": 80,
"mainland-worker-01": 60,
},
},
)
self.assertEqual(3200, limits["estimated_total_threads"])
self.assertEqual(6400, limits["max_pending_total"])
self.assertEqual(3200, limits["max_register_pending"])
self.assertEqual(800, limits["max_downstream_pending"])
self.assertEqual(80000, limits["estimated_total_threads"])
self.assertEqual(160000, limits["max_pending_total"])
self.assertEqual(80000, limits["max_register_pending"])
self.assertEqual(20000, limits["max_downstream_pending"])
def test_select_relevant_backlog_job_ids_from_rows_skips_stale_pending_jobs(self) -> None:
now = datetime.now()
job_ids = _select_relevant_backlog_job_ids_from_rows(
[
(11, "running", now - timedelta(hours=10)),
(10, "pending", now - timedelta(minutes=30)),
(9, "pending", now - timedelta(hours=7)),
],
freshness_hours=6,
limit=4,
)
self.assertEqual([11, 10], job_ids)
@patch("app.services.sync_push_service.get_active_detect_job_summary")
@patch("app.services.sync_push_service.get_db")
def test_load_local_detect_backlog_snapshot_prefers_active_job_snapshot(
self,
mock_get_db,
mock_get_active_detect_job_summary,
) -> None:
mock_get_active_detect_job_summary.return_value = {
"items_pending": 123,
"items_claimed": 4,
"items_running": 7,
"display_items_running": 9,
"step_stats": [
{"step_code": "detect_register", "items_pending": 90},
{"step_code": "detect_baidu", "items_pending": 33},
],
}
backlog = _load_local_detect_backlog_snapshot()
self.assertEqual(123, backlog["pending_total"])
self.assertEqual(4, backlog["claimed_total"])
self.assertEqual(9, backlog["running_total"])
self.assertEqual(90, backlog["register_pending"])
self.assertEqual(33, backlog["downstream_pending"])
mock_get_db.assert_not_called()
@patch("app.services.sync_push_service.get_active_detect_job_summary", return_value={})
@patch("app.services.sync_push_service.get_db")
def test_load_local_detect_backlog_snapshot_only_counts_relevant_jobs(
self,
mock_get_db,
_mock_get_active_detect_job_summary,
) -> None:
fake_conn = _FakeConnection(
rows=[
[
(11, "running", datetime.now() - timedelta(hours=2)),
(10, "pending", datetime.now() - timedelta(minutes=30)),
(9, "pending", datetime.now() - timedelta(hours=7)),
],
(123, 4, 7, 90, 33),
]
)
mock_get_db.return_value = fake_conn
backlog = _load_local_detect_backlog_snapshot()
self.assertEqual(123, backlog["pending_total"])
self.assertEqual(90, backlog["register_pending"])
executed_sql, executed_params = fake_conn.cursor_obj.executed[1]
self.assertIn("WHERE item.job_id = ANY(%s)", executed_sql)
self.assertEqual([11, 10], list(executed_params[0]))
def test_should_throttle_task_pull_when_register_backlog_overwhelms_downstream(self) -> None:
should_throttle, reason = _should_throttle_task_pull(
@@ -104,6 +408,224 @@ class SyncPushServiceTests(unittest.TestCase):
self.assertFalse(should_throttle)
self.assertEqual("", reason)
@patch("app.services.sync_push_service._should_throttle_task_pull", return_value=(False, ""))
@patch("app.services.sync_push_service._build_task_pull_backlog_limits", return_value={})
@patch("app.services.sync_push_service._load_local_detect_backlog_snapshot", return_value={})
@patch("app.services.sync_push_service._acquire_sync_pull_worker_wake_guard", return_value=True)
@patch("app.services.sync_push_service.ingest_detect_task_projection")
@patch("app.services.worker_control_service.send_worker_command")
@patch("app.services.sync_push_service.urllib.request.urlopen")
@patch("app.services.sync_push_service.settings")
@patch("app.services.sync_push_service.get_settings_payload")
def test_pull_detect_task_batch_now_starts_worker_with_projection_active_job_identity(
self,
mock_get_settings_payload,
mock_settings,
mock_urlopen,
mock_send_worker_command,
mock_ingest_detect_task_projection,
_mock_acquire_sync_pull_worker_wake_guard,
_mock_load_local_detect_backlog_snapshot,
_mock_build_task_pull_backlog_limits,
_mock_should_throttle_task_pull,
) -> None:
mock_settings.node_region = "mainland"
mock_settings.node_role = "control"
mock_settings.node_code = "mainland-controller-01"
mock_settings.sync_target_api_base_url = "http://example.com"
mock_settings.sync_target_region = "overseas"
mock_settings.sync_batch_size = 200
mock_settings.sync_shared_token = ""
mock_get_settings_payload.return_value = {
"thread_count": 1000,
"process_count": 1,
"node_thread_counts": {"mainland-controller-01": 1000},
"node_process_counts": {"mainland-controller-01": 60},
}
projection = {
"batch_code": "task-20260423210000-aa11bb",
"active_job": {
"job_id": 376,
"job_code": "sync-overseas-376",
"current_cycle_token": "cycle-376",
},
}
export_response = {
"code": 0,
"data": {
"source_record_id": 15164,
"projection_hash": "hash-15164",
"projection": projection,
},
}
ack_response = {"code": 0, "data": {"acknowledged": True}}
mock_urlopen.side_effect = [
_FakeUrlopenResponse(export_response),
_FakeUrlopenResponse(ack_response),
]
mock_ingest_detect_task_projection.return_value = (
True,
"任务批次接收成功",
{
"target_job_id": 1902,
"target_job_code": "sync-overseas-15164",
"queued_count": 1000,
},
)
mock_send_worker_command.return_value = (True, "started")
ok, message, payload = pull_detect_task_batch_now(limit=1000)
self.assertTrue(ok)
self.assertEqual("待检测任务批次拉取并入库成功", message)
self.assertTrue(payload["worker_start_ok"])
export_request = mock_urlopen.call_args_list[0][0][0]
self.assertIn("limit=1000", export_request.full_url)
mock_send_worker_command.assert_called_once()
args, kwargs = mock_send_worker_command.call_args
self.assertEqual("start_detection", args[0])
sent_payload = kwargs["payload"]
self.assertEqual(15164, sent_payload["source_record_id"])
self.assertEqual(1902, sent_payload["target_job_id"])
self.assertEqual("sync-overseas-15164", sent_payload["target_job_code"])
self.assertEqual(376, sent_payload["job_id"])
self.assertEqual("sync-overseas-376", sent_payload["job_code"])
self.assertEqual("cycle-376", sent_payload["cycle_token"])
@patch("app.services.sync_push_service._should_throttle_task_pull", return_value=(False, ""))
@patch("app.services.sync_push_service._build_task_pull_backlog_limits", return_value={})
@patch("app.services.sync_push_service._load_local_detect_backlog_snapshot", return_value={})
@patch("app.services.sync_push_service._acquire_sync_pull_worker_wake_guard", return_value=False)
@patch("app.services.sync_push_service.ingest_detect_task_projection")
@patch("app.services.worker_control_service.send_worker_command")
@patch("app.services.sync_push_service.urllib.request.urlopen")
@patch("app.services.sync_push_service.settings")
@patch("app.services.sync_push_service.get_settings_payload")
def test_pull_detect_task_batch_now_skips_duplicate_worker_wake_within_short_window(
self,
mock_get_settings_payload,
mock_settings,
mock_urlopen,
mock_send_worker_command,
mock_ingest_detect_task_projection,
_mock_acquire_sync_pull_worker_wake_guard,
_mock_load_local_detect_backlog_snapshot,
_mock_build_task_pull_backlog_limits,
_mock_should_throttle_task_pull,
) -> None:
mock_settings.node_region = "mainland"
mock_settings.node_role = "control"
mock_settings.node_code = "mainland-controller-01"
mock_settings.sync_target_api_base_url = "http://example.com"
mock_settings.sync_target_region = "overseas"
mock_settings.sync_batch_size = 200
mock_settings.sync_shared_token = ""
mock_get_settings_payload.return_value = {
"thread_count": 1000,
"process_count": 1,
"node_thread_counts": {"mainland-controller-01": 1000},
"node_process_counts": {"mainland-controller-01": 60},
}
projection = {
"batch_code": "task-20260423210000-aa11bb",
"active_job": {
"job_id": 376,
"job_code": "sync-overseas-376",
"current_cycle_token": "cycle-376",
},
}
export_response = {
"code": 0,
"data": {
"source_record_id": 15164,
"projection_hash": "hash-15164",
"projection": projection,
},
}
ack_response = {"code": 0, "data": {"acknowledged": True}}
mock_urlopen.side_effect = [
_FakeUrlopenResponse(export_response),
_FakeUrlopenResponse(ack_response),
]
mock_ingest_detect_task_projection.return_value = (
True,
"任务批次接收成功",
{
"target_job_id": 1902,
"target_job_code": "sync-overseas-15164",
"queued_count": 1000,
},
)
ok, message, payload = pull_detect_task_batch_now(limit=1000)
self.assertTrue(ok)
self.assertEqual("待检测任务批次拉取并入库成功", message)
self.assertTrue(payload["worker_start_ok"])
self.assertTrue(payload["worker_start_skipped"])
self.assertIn("重复 Worker 唤起", payload["worker_start_message"])
mock_send_worker_command.assert_not_called()
@patch("app.services.sync_push_service._should_throttle_task_pull", return_value=(False, ""))
@patch("app.services.sync_push_service._build_task_pull_backlog_limits", return_value={})
@patch("app.services.sync_push_service._load_local_detect_backlog_snapshot", return_value={})
@patch("app.services.sync_push_service._acquire_sync_pull_worker_wake_guard", return_value=False)
@patch("app.services.sync_push_service.ingest_detect_task_projection")
@patch("app.services.sync_push_service.urllib.request.urlopen")
@patch("app.services.sync_push_service.settings")
@patch("app.services.sync_push_service.get_settings_payload")
def test_pull_detect_task_batch_now_uses_adaptive_limit_when_unspecified(
self,
mock_get_settings_payload,
mock_settings,
mock_urlopen,
mock_ingest_detect_task_projection,
_mock_acquire_sync_pull_worker_wake_guard,
_mock_load_local_detect_backlog_snapshot,
_mock_build_task_pull_backlog_limits,
_mock_should_throttle_task_pull,
) -> None:
mock_settings.node_region = "mainland"
mock_settings.node_role = "control"
mock_settings.node_code = "mainland-controller-01"
mock_settings.sync_target_api_base_url = "http://example.com"
mock_settings.sync_target_region = "overseas"
mock_settings.sync_batch_size = 5000
mock_settings.sync_shared_token = ""
mock_get_settings_payload.return_value = {
"thread_count": 1000,
"process_count": 1,
"node_thread_counts": {"mainland-controller-01": 1000},
"node_process_counts": {"mainland-controller-01": 60},
}
export_response = {
"code": 0,
"data": {
"source_record_id": 15164,
"projection_hash": "hash-15164",
"projection": {"batch_code": "task-20260423210000-aa11bb", "active_job": {}},
},
}
ack_response = {"code": 0, "data": {"acknowledged": True}}
mock_urlopen.side_effect = [
_FakeUrlopenResponse(export_response),
_FakeUrlopenResponse(ack_response),
]
mock_ingest_detect_task_projection.return_value = (
True,
"任务批次接收成功",
{"target_job_id": 1902, "target_job_code": "sync-overseas-15164", "queued_count": 1000},
)
ok, _message, _payload = pull_detect_task_batch_now(limit=None)
self.assertTrue(ok)
export_request = mock_urlopen.call_args_list[0][0][0]
self.assertIn("limit=120000", export_request.full_url)
def test_extract_detect_result_projection_events_adds_import_metadata(self) -> None:
projection = {
"job": {
@@ -137,6 +659,79 @@ class SyncPushServiceTests(unittest.TestCase):
self.assertTrue(event["payload"]["imported_from_projection"])
self.assertTrue(event["payload"]["import_fingerprint"])
@patch("app.services.sync_push_service._push_projection_record")
@patch("app.services.sync_push_service._load_latest_projection")
@patch("app.services.runtime_status_service.refresh_runtime_projection_snapshot")
def test_push_projection_now_refreshes_runtime_projection_with_lightweight_snapshot(
self,
mock_refresh_runtime_projection_snapshot,
mock_load_latest_projection,
mock_push_projection_record,
) -> None:
mock_refresh_runtime_projection_snapshot.return_value = {
"record_id": 10082,
"active_thread_count": 1972,
"max_thread_count": 80000,
"queue_display_running": 1972,
}
mock_load_latest_projection.return_value = {
"id": 10082,
"source_region": "mainland",
"target_region": "overseas",
"created_at": None,
"payload": {"projection_hash": "hash-10082", "projection": {}},
}
mock_push_projection_record.return_value = (
True,
"投影推送成功",
{"action": "push_sync", "sync_type": "runtime_projection", "source_record_id": 10082},
)
ok, message, data = _push_projection_now("runtime_projection", "https://example.com/api/v1/runtime/sync-ingest")
self.assertTrue(ok)
self.assertEqual("投影推送成功", message)
self.assertEqual(10082, data["source_record_id"])
mock_refresh_runtime_projection_snapshot.assert_called_once_with(window_minutes=15)
mock_load_latest_projection.assert_called_once_with("runtime_projection")
mock_push_projection_record.assert_called_once()
@patch("app.services.sync_push_service._push_projection_record")
@patch("app.services.sync_push_service._load_latest_projection")
@patch("app.services.sync_push_service._append_fast_runtime_projection_snapshot")
@patch("app.services.runtime_status_service.refresh_runtime_projection_snapshot")
def test_push_projection_now_can_use_fast_runtime_projection_path(
self,
mock_refresh_runtime_projection_snapshot,
mock_append_fast_runtime_projection_snapshot,
mock_load_latest_projection,
mock_push_projection_record,
) -> None:
mock_append_fast_runtime_projection_snapshot.return_value = 10091
mock_load_latest_projection.return_value = {
"id": 10091,
"source_region": "mainland",
"target_region": "overseas",
"created_at": None,
"payload": {"projection_hash": "hash-10091", "projection": {}},
}
mock_push_projection_record.return_value = (
True,
"投影推送成功",
{"action": "push_sync", "sync_type": "runtime_projection", "source_record_id": 10091},
)
with patch.dict("os.environ", {"DOMAINCHECK_SYNC_RUNTIME_FAST_PROJECTION": "1"}, clear=False):
ok, message, data = _push_projection_now("runtime_projection", "https://example.com/api/v1/runtime/sync-ingest")
self.assertTrue(ok)
self.assertEqual("投影推送成功", message)
self.assertEqual(10091, data["source_record_id"])
mock_append_fast_runtime_projection_snapshot.assert_called_once()
mock_refresh_runtime_projection_snapshot.assert_not_called()
mock_load_latest_projection.assert_called_once_with("runtime_projection")
mock_push_projection_record.assert_called_once()
@patch("app.services.sync_push_service.get_db")
def test_resolve_detect_result_target_job_id_prefers_matching_job_code(self, mock_get_db) -> None:
fake_conn = _FakeConnection(rows=[(456,)])

725
domain-api/tests/test_sync_record_service.py
View File

@@ -1,12 +1,61 @@
import json
import unittest
from datetime import datetime, timedelta
from unittest.mock import patch
from app.services.sync_record_service import (
_build_runtime_projection_payload,
_collect_recent_domain_events,
_pick_latest_projection_row,
append_runtime_projection_if_changed,
get_detect_result_sync_batches,
)
class _FakeCursor:
def __init__(self, fetchone_values):
self.fetchone_values = list(fetchone_values or [])
self.executed = []
def execute(self, sql, params=None):
self.executed.append((sql, params))
def fetchone(self):
if self.fetchone_values:
return self.fetchone_values.pop(0)
return None
def fetchall(self):
if self.fetchone_values:
value = self.fetchone_values.pop(0)
return list(value or [])
return []
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
class _FakeConnection:
def __init__(self, fetchone_values):
self.cursor_obj = _FakeCursor(fetchone_values)
self.committed = False
def cursor(self):
return self.cursor_obj
def commit(self):
self.committed = True
def __enter__(self):
return self
def __exit__(self, exc_type, exc, tb):
return False
class SyncRecordServiceTests(unittest.TestCase):
def test_collect_recent_domain_events_filters_and_keeps_latest_slice(self) -> None:
active_job = {
@@ -130,6 +179,682 @@ class SyncRecordServiceTests(unittest.TestCase):
self.assertEqual("", projection["active_job"]["job_code"])
self.assertEqual([], projection["active_job"]["node_stats"])
@patch("app.services.sync_record_service._resolve_local_ip", return_value="121.204.244.188")
@patch("app.services.sync_record_service.socket.gethostname", return_value="mainland-controller-01")
@patch("app.services.sync_record_service.settings")
def test_build_runtime_projection_payload_prefers_runtime_queue_nodes_for_multi_process_controller(
self,
mock_settings,
_mock_hostname,
_mock_resolve_ip,
) -> None:
mock_settings.node_code = "mainland-controller-01"
mock_settings.node_region = "mainland"
mock_settings.node_role = "control"
mock_settings.sync_source_region = "mainland"
mock_settings.sync_target_region = "overseas"
payload = _build_runtime_projection_payload(
detect={
"worker_online": True,
"worker_mode": "linux-systemd",
"active_thread_count": 1,
"max_thread_count": 1000,
"aggregate_max_thread_count": 80000,
"phase_label": "运行中",
"phase_detail": "80 实例运行",
"proxy_runtime_label": "正常",
"proxy_runtime_reason": "healthy",
"progress": {
"pending": 5001,
"running": 0,
"completed": 0,
"blacklisted": 0,
"failed": 0,
},
"backlog": {"pending_total": 1157325},
"queue_health": {
"queue": {
"items_total": 5292,
"pending": 4578,
"claimed": 0,
"display_claimed": 120,
"running": 1972,
"display_running": 1972,
"completed": 690,
"blacklisted": 0,
"failed": 24,
"terminal": 714,
},
"nodes": [
{
"node_code": "mainland-controller-01-a",
"items_running": 1000,
"items_claimed": 0,
"display_running": 1000,
"active_threads": 1000,
"max_threads": 1000,
"region": "mainland",
"role": "control",
"status": "busy",
},
{
"node_code": "mainland-controller-01-b",
"items_running": 972,
"items_claimed": 120,
"display_running": 972,
"active_threads": 972,
"max_threads": 1000,
"region": "mainland",
"role": "control",
"status": "busy",
},
],
},
"active_job": {
"job_id": 55,
"job_code": "sync-overseas-55",
"status": "running",
"progress_percent": 13.49,
"items_total": 5292,
"items_terminal": 714,
"items_pending": 4578,
"items_running": 0,
"items_failed": 24,
"node_stats": [
{
"node_code": "mainland-controller-01",
"items_running": 1,
"items_claimed": 0,
"items_total": 5292,
}
],
},
"dependency_alerts": [],
},
cluster={
"nodes_total": 80,
"nodes": [
{
"node_code": "mainland-controller-01",
"current_load": 1,
"detect_participating": True,
}
],
"summary": {
"online_worker_nodes": 65,
"dedicated_online_worker_nodes": 1,
"online_control_nodes": 65,
"busy_nodes": ["mainland-controller-01-a", "mainland-controller-01-b"],
"stale_nodes": [],
"offline_nodes": [],
},
},
source_region="mainland",
target_region="overseas",
)
projection = payload["projection"]
self.assertTrue(projection["detect_participating"])
self.assertEqual(1972, projection["active_thread_count"])
self.assertEqual(80000, projection["max_thread_count"])
self.assertEqual(1972, projection["progress"]["running"])
self.assertEqual(2, len(projection["active_job"]["node_stats"]))
self.assertEqual(2, len(projection["active_job"]["distributed_node_stats"]))
self.assertEqual(1, len(projection["cluster_nodes"]))
self.assertEqual("mainland-controller-01", projection["cluster_nodes"][0]["node_code"])
self.assertEqual(1972, projection["active_job"]["items_running"])
self.assertEqual(1972, projection["active_job"]["display_items_running"])
self.assertEqual(80000, projection["active_job"]["display_max_threads"])
@patch("app.services.sync_record_service._resolve_local_ip", return_value="121.204.244.188")
@patch("app.services.sync_record_service.socket.gethostname", return_value="mainland-controller-01")
@patch("app.services.sync_record_service.settings")
@patch("app.services.sync_record_service.get_db")
def test_append_runtime_projection_if_changed_writes_heartbeat_for_unchanged_projection_after_interval(
self,
mock_get_db,
mock_settings,
_mock_hostname,
_mock_resolve_ip,
) -> None:
mock_settings.node_code = "mainland-controller-01"
mock_settings.node_region = "mainland"
mock_settings.node_role = "control"
mock_settings.sync_source_region = "mainland"
mock_settings.sync_target_region = "overseas"
detect = {
"worker_online": True,
"worker_mode": "linux-systemd",
"active_thread_count": 3200,
"max_thread_count": 80000,
"phase_label": "运行中",
"phase_detail": "80 实例运行",
"proxy_runtime_label": "正常",
"proxy_runtime_reason": "healthy",
"progress": {
"pending": 1200000,
"running": 6400,
"completed": 50000,
"blacklisted": 1200,
"failed": 88,
},
"backlog": {"pending_total": 1200000},
"active_job": {
"job_id": 55,
"job_code": "sync-overseas-55",
"status": "running",
"progress_percent": 12.5,
"items_total": 1300000,
"items_terminal": 51288,
"items_pending": 1200000,
"items_running": 6400,
"items_failed": 88,
"node_stats": [
{
"node_code": "mainland-controller-01",
"items_running": 6400,
"items_claimed": 7000,
"items_total": 1300000,
}
],
},
"dependency_alerts": [],
}
cluster = {
"nodes_total": 1,
"nodes": [
{
"node_code": "mainland-controller-01",
"current_load": 6400,
"detect_participating": True,
}
],
"summary": {
"online_worker_nodes": 1,
"dedicated_online_worker_nodes": 0,
"online_control_nodes": 1,
"busy_nodes": ["mainland-controller-01"],
"stale_nodes": [],
"offline_nodes": [],
},
}
previous_payload = _build_runtime_projection_payload(
detect=detect,
cluster=cluster,
source_region="mainland",
target_region="overseas",
)
fake_conn = _FakeConnection(
[
[(previous_payload, datetime.now() - timedelta(seconds=90))],
(321,),
]
)
mock_get_db.return_value = fake_conn
record_id = append_runtime_projection_if_changed(
detect=detect,
cluster=cluster,
source_region="mainland",
target_region="overseas",
)
self.assertEqual(321, record_id)
self.assertTrue(fake_conn.committed)
self.assertTrue(
any("INSERT INTO detect_sync_records" in sql for sql, _params in fake_conn.cursor_obj.executed)
)
@patch("app.services.sync_record_service._resolve_local_ip", return_value="121.204.244.188")
@patch("app.services.sync_record_service.socket.gethostname", return_value="mainland-controller-01")
@patch("app.services.sync_record_service.settings")
@patch("app.services.sync_record_service.get_db")
def test_append_runtime_projection_if_changed_writes_when_cluster_nodes_change_within_window(
self,
mock_get_db,
mock_settings,
_mock_hostname,
_mock_resolve_ip,
) -> None:
mock_settings.node_code = "mainland-controller-01"
mock_settings.node_region = "mainland"
mock_settings.node_role = "control"
mock_settings.sync_source_region = "mainland"
mock_settings.sync_target_region = "overseas"
previous_payload = {
"projection": {
"worker_online": True,
"worker_mode": "linux-systemd",
"phase_label": "运行中",
"phase_detail": "等待中",
"proxy_runtime_label": "正常",
"proxy_runtime_reason": "healthy",
"active_thread_count": 0,
"max_thread_count": 60000,
"progress": {"pending": 10000, "running": 0, "completed": 0, "blacklisted": 0, "failed": 0},
"active_job": {
"job_id": None,
"job_code": "",
"status": "",
"items_total": 0,
"items_running": 0,
"items_claimed": 0,
"display_items_running": 0,
"display_items_claimed": 0,
"display_max_threads": 0,
"node_stats": [],
"distributed_node_stats": [],
},
"cluster_summary": {
"nodes_total": 43,
"online_worker_nodes": 43,
"dedicated_online_worker_nodes": 42,
"online_control_nodes": 1,
"busy_nodes": [],
"stale_nodes": [],
"offline_nodes": [],
},
"cluster_nodes": [
{
"node_code": "mainland-controller-01",
"role": "control",
"status": "online",
"current_load": 0,
"active_threads": 0,
"max_threads": 60000,
"detect_participating": False,
}
],
"dependency_alerts": [],
}
}
fake_conn = _FakeConnection(
[
[(previous_payload, datetime.now() - timedelta(seconds=10))],
(911,),
]
)
mock_get_db.return_value = fake_conn
record_id = append_runtime_projection_if_changed(
detect={
"worker_online": True,
"worker_mode": "linux-systemd",
"phase_label": "运行中",
"phase_detail": "等待中",
"proxy_runtime_label": "正常",
"proxy_runtime_reason": "healthy",
"progress": {"pending": 10000, "running": 0, "completed": 0, "blacklisted": 0, "failed": 0},
"backlog": {},
"active_job": {},
"dependency_alerts": [],
},
cluster={
"nodes_total": 61,
"nodes": [
{
"node_code": "mainland-controller-01",
"role": "control",
"status": "online",
"current_load": 0,
"metadata": {"active_threads": 0, "max_threads": 60000},
},
{
"node_code": "mainland-controller-01-a",
"role": "worker",
"status": "online",
"current_load": 0,
"metadata": {"active_threads": 0, "max_threads": 1000},
},
],
"summary": {
"online_worker_nodes": 61,
"dedicated_online_worker_nodes": 60,
"online_control_nodes": 1,
"busy_nodes": [],
"stale_nodes": [],
"offline_nodes": [],
},
},
source_region="mainland",
target_region="overseas",
)
self.assertEqual(911, record_id)
self.assertTrue(fake_conn.committed)
self.assertTrue(
any("INSERT INTO detect_sync_records" in sql for sql, _params in fake_conn.cursor_obj.executed)
)
@patch("app.services.sync_record_service._resolve_local_ip", return_value="121.204.244.188")
@patch("app.services.sync_record_service.socket.gethostname", return_value="mainland-controller-01")
@patch("app.services.sync_record_service.settings")
@patch("app.services.sync_record_service.get_db")
def test_append_runtime_projection_if_changed_skips_unchanged_projection_within_heartbeat_window(
self,
mock_get_db,
mock_settings,
_mock_hostname,
_mock_resolve_ip,
) -> None:
mock_settings.node_code = "mainland-controller-01"
mock_settings.node_region = "mainland"
mock_settings.node_role = "control"
mock_settings.sync_source_region = "mainland"
mock_settings.sync_target_region = "overseas"
detect = {
"worker_online": True,
"worker_mode": "linux-systemd",
"active_thread_count": 3200,
"max_thread_count": 80000,
"phase_label": "运行中",
"phase_detail": "80 实例运行",
"proxy_runtime_label": "正常",
"proxy_runtime_reason": "healthy",
"progress": {
"pending": 1200000,
"running": 6400,
"completed": 50000,
"blacklisted": 1200,
"failed": 88,
},
"backlog": {"pending_total": 1200000},
"active_job": {
"job_id": 55,
"job_code": "sync-overseas-55",
"status": "running",
"progress_percent": 12.5,
"items_total": 1300000,
"items_terminal": 51288,
"items_pending": 1200000,
"items_running": 6400,
"items_failed": 88,
"node_stats": [
{
"node_code": "mainland-controller-01",
"items_running": 6400,
"items_claimed": 7000,
"items_total": 1300000,
}
],
},
"dependency_alerts": [],
}
cluster = {
"nodes_total": 1,
"nodes": [
{
"node_code": "mainland-controller-01",
"current_load": 6400,
"detect_participating": True,
}
],
"summary": {
"online_worker_nodes": 1,
"dedicated_online_worker_nodes": 0,
"online_control_nodes": 1,
"busy_nodes": ["mainland-controller-01"],
"stale_nodes": [],
"offline_nodes": [],
},
}
previous_payload = _build_runtime_projection_payload(
detect=detect,
cluster=cluster,
source_region="mainland",
target_region="overseas",
)
fake_conn = _FakeConnection(
[
[(previous_payload, datetime.now() - timedelta(seconds=10))],
]
)
mock_get_db.return_value = fake_conn
record_id = append_runtime_projection_if_changed(
detect=detect,
cluster=cluster,
source_region="mainland",
target_region="overseas",
)
self.assertIsNone(record_id)
self.assertFalse(fake_conn.committed)
self.assertFalse(
any("INSERT INTO detect_sync_records" in sql for sql, _params in fake_conn.cursor_obj.executed)
)
@patch("app.services.sync_record_service._resolve_local_ip", return_value="121.204.244.188")
@patch("app.services.sync_record_service.socket.gethostname", return_value="mainland-controller-01")
@patch("app.services.sync_record_service.settings")
@patch("app.services.sync_record_service.get_db")
def test_append_runtime_projection_if_changed_writes_when_activity_signature_changes_within_window(
self,
mock_get_db,
mock_settings,
_mock_hostname,
_mock_resolve_ip,
) -> None:
mock_settings.node_code = "mainland-controller-01"
mock_settings.node_region = "mainland"
mock_settings.node_role = "control"
mock_settings.sync_source_region = "mainland"
mock_settings.sync_target_region = "overseas"
previous_detect = {
"worker_online": True,
"worker_mode": "linux-systemd",
"active_thread_count": 300,
"max_thread_count": 60000,
"phase_label": "运行中",
"phase_detail": "60 实例运行",
"proxy_runtime_label": "正常",
"proxy_runtime_reason": "healthy",
"progress": {
"pending": 5000,
"running": 600,
"completed": 1000,
"blacklisted": 0,
"failed": 10,
},
"queue_health": {
"queue": {
"items_total": 6610,
"pending": 5000,
"claimed": 10,
"display_claimed": 10,
"running": 600,
"display_running": 600,
"completed": 1000,
"blacklisted": 0,
"failed": 10,
"terminal": 1010,
"display_max_threads": 60000,
},
"nodes": [
{
"node_code": "mainland-controller-01-a",
"items_running": 300,
"items_claimed": 10,
"display_running": 300,
"active_threads": 300,
"max_threads": 1000,
"region": "mainland",
"role": "control",
"status": "busy",
}
],
},
"active_job": {
"job_id": 867,
"job_code": "sync-overseas-19835",
"status": "running",
"progress_percent": 15.0,
"items_total": 6610,
"items_terminal": 1010,
"items_pending": 5000,
"items_claimed": 10,
"items_running": 600,
"items_failed": 10,
"node_stats": [
{
"node_code": "mainland-controller-01-a",
"items_running": 300,
"items_claimed": 10,
"items_total": 6610,
}
],
},
"dependency_alerts": [],
}
current_detect = {
**previous_detect,
"active_thread_count": 900,
"queue_health": {
"queue": {
"items_total": 6610,
"pending": 4300,
"claimed": 30,
"display_claimed": 30,
"running": 900,
"display_running": 900,
"completed": 1370,
"blacklisted": 0,
"failed": 10,
"terminal": 1380,
"display_max_threads": 60000,
},
"nodes": [
{
"node_code": "mainland-controller-01-a",
"items_running": 420,
"items_claimed": 20,
"display_running": 420,
"active_threads": 420,
"max_threads": 1000,
"region": "mainland",
"role": "control",
"status": "busy",
},
{
"node_code": "mainland-controller-01-b",
"items_running": 480,
"items_claimed": 10,
"display_running": 480,
"active_threads": 480,
"max_threads": 1000,
"region": "mainland",
"role": "control",
"status": "busy",
},
],
},
"active_job": {
**previous_detect["active_job"],
"items_pending": 4300,
"items_claimed": 30,
"items_running": 900,
"node_stats": [
{
"node_code": "mainland-controller-01-a",
"items_running": 420,
"items_claimed": 20,
"items_total": 6610,
},
{
"node_code": "mainland-controller-01-b",
"items_running": 480,
"items_claimed": 10,
"items_total": 6610,
},
],
},
}
cluster = {
"nodes_total": 2,
"nodes": [
{
"node_code": "mainland-controller-01",
"current_load": 900,
"detect_participating": True,
}
],
"summary": {
"online_worker_nodes": 60,
"dedicated_online_worker_nodes": 0,
"online_control_nodes": 60,
"busy_nodes": ["mainland-controller-01-a", "mainland-controller-01-b"],
"stale_nodes": [],
"offline_nodes": [],
},
}
previous_payload = _build_runtime_projection_payload(
detect=previous_detect,
cluster=cluster,
source_region="mainland",
target_region="overseas",
)
fake_conn = _FakeConnection(
[
[(previous_payload, datetime.now() - timedelta(seconds=10))],
(654,),
]
)
mock_get_db.return_value = fake_conn
record_id = append_runtime_projection_if_changed(
detect=current_detect,
cluster=cluster,
source_region="mainland",
target_region="overseas",
)
self.assertEqual(654, record_id)
self.assertTrue(fake_conn.committed)
self.assertTrue(
any("INSERT INTO detect_sync_records" in sql for sql, _params in fake_conn.cursor_obj.executed)
)
def test_pick_latest_projection_row_skips_future_dated_rows(self) -> None:
rows = [
("future", datetime.now() + timedelta(hours=6)),
("recent", datetime.now() - timedelta(seconds=10)),
("older", datetime.now() - timedelta(minutes=2)),
]
selected = _pick_latest_projection_row(rows, created_at_index=1)
self.assertEqual("recent", selected[0])
@patch("app.services.sync_record_service.get_db")
@patch("app.services.sync_record_service.settings")
def test_get_detect_result_sync_batches_marks_overseas_control_as_not_applicable(
self,
mock_settings,
mock_get_db,
) -> None:
mock_settings.node_code = "overseas-control-01"
mock_settings.node_region = "overseas"
mock_settings.node_role = "control"
mock_settings.sync_source_region = "overseas"
mock_settings.sync_target_region = "mainland"
payload = get_detect_result_sync_batches(limit=5)
self.assertFalse(payload["applicable"])
self.assertFalse(payload["local_worker_expected"])
self.assertEqual(0, payload["jobs_total"])
self.assertEqual([], payload["batches"])
self.assertIn("不适用", payload["reason"])
mock_get_db.assert_not_called()
if __name__ == "__main__":
unittest.main()

323
domain-api/tests/test_worker_control_service.py
View File

@@ -1,8 +1,17 @@
import json
import subprocess
import unittest
from unittest.mock import Mock, patch
from app.services.worker_control_service import WORKER_CONTROL_CHANNEL, WORKER_PENDING_COMMAND_KEY, send_worker_command
from app.core.config import settings
from app.services.worker_control_service import (
WORKER_CONTROL_CHANNEL,
WORKER_PENDING_COMMAND_KEY,
detect_worker_runtime,
send_worker_command,
start_worker,
stop_worker,
)
class WorkerControlServiceTests(unittest.TestCase):
@@ -11,10 +20,11 @@ class WorkerControlServiceTests(unittest.TestCase):
redis_client = Mock()
mock_get_redis.return_value = redis_client
ok, message = send_worker_command(
"start_detection",
payload={"job_id": 1, "job_code": "detect-20260419030000-abc123"},
)
with patch("app.services.worker_control_service._runtime_config", return_value={"worker_mode": "windows-local", "worker_service_name": "domaincheck-worker"}):
ok, message = send_worker_command(
"start_detection",
payload={"job_id": 1, "job_code": "detect-20260419030000-abc123"},
)
self.assertTrue(ok)
self.assertIn("已发送 Worker 控制指令", message)
@@ -33,6 +43,309 @@ class WorkerControlServiceTests(unittest.TestCase):
redis_client.publish.assert_called_once_with(WORKER_CONTROL_CHANNEL, serialized)
@patch("app.services.worker_control_service.get_redis")
def test_send_worker_command_scopes_pending_command_to_target_worker_instances(self, mock_get_redis) -> None:
redis_client = Mock()
mock_get_redis.return_value = redis_client
with patch("app.services.worker_control_service._runtime_config", return_value={"worker_mode": "linux-systemd", "worker_service_name": "domaincheck-worker"}):
ok, message = send_worker_command(
"start_detection",
payload={
"job_id": 2,
"target_node_codes": ["mainland-controller-01-a", "mainland-controller-01-b"],
},
)
self.assertTrue(ok)
self.assertIn("mainland-controller-01-a,mainland-controller-01-b", message)
self.assertEqual(2, redis_client.set.call_count)
set_keys = [call.args[0] for call in redis_client.set.call_args_list]
self.assertEqual(
[
f"{WORKER_PENDING_COMMAND_KEY}:mainland-controller-01-a",
f"{WORKER_PENDING_COMMAND_KEY}:mainland-controller-01-b",
],
set_keys,
)
serialized = redis_client.set.call_args_list[0].args[1]
payload = json.loads(serialized)
self.assertEqual(["mainland-controller-01-a", "mainland-controller-01-b"], payload["target_node_codes"])
redis_client.publish.assert_called_once_with(WORKER_CONTROL_CHANNEL, serialized)
@patch("app.services.worker_control_service._expand_linux_worker_control_units")
@patch("app.services.worker_control_service.get_redis")
def test_send_worker_command_expands_local_linux_worker_instances_when_targets_unspecified(
self,
mock_get_redis,
mock_expand_linux_worker_control_units,
) -> None:
redis_client = Mock()
mock_get_redis.return_value = redis_client
mock_expand_linux_worker_control_units.return_value = [
"domaincheck-worker",
"domaincheck-worker@a.service",
"domaincheck-worker@b.service",
]
with patch("app.services.worker_control_service._runtime_config", return_value={"worker_mode": "linux-systemd", "worker_service_name": "domaincheck-worker"}), \
patch.object(settings, "node_code", "mainland-controller-01"):
ok, message = send_worker_command("start_detection", payload={"job_id": 3})
self.assertTrue(ok)
self.assertIn("mainland-controller-01,mainland-controller-01-a,mainland-controller-01-b", message)
self.assertEqual(3, redis_client.set.call_count)
set_keys = [call.args[0] for call in redis_client.set.call_args_list]
self.assertEqual(
[
f"{WORKER_PENDING_COMMAND_KEY}:mainland-controller-01",
f"{WORKER_PENDING_COMMAND_KEY}:mainland-controller-01-a",
f"{WORKER_PENDING_COMMAND_KEY}:mainland-controller-01-b",
],
set_keys,
)
serialized = redis_client.set.call_args_list[0].args[1]
payload = json.loads(serialized)
self.assertEqual(
["mainland-controller-01", "mainland-controller-01-a", "mainland-controller-01-b"],
payload["target_node_codes"],
)
redis_client.publish.assert_called_once_with(WORKER_CONTROL_CHANNEL, serialized)
@patch("app.services.worker_control_service._build_direct_redis_client")
@patch("app.services.worker_control_service.get_redis")
def test_send_worker_command_falls_back_to_direct_redis_client(
self,
mock_get_redis,
mock_build_direct_redis_client,
) -> None:
mock_get_redis.side_effect = RecursionError("maximum recursion depth exceeded")
direct_client = Mock()
mock_build_direct_redis_client.return_value = direct_client
with patch("app.services.worker_control_service._runtime_config", return_value={"worker_mode": "windows-local", "worker_service_name": "domaincheck-worker"}):
ok, message = send_worker_command("start_detection", payload={"job_id": 9})
self.assertTrue(ok)
self.assertIn("已发送 Worker 控制指令", message)
direct_client.set.assert_called_once()
serialized = direct_client.set.call_args.args[1]
payload = json.loads(serialized)
self.assertEqual("start_detection", payload["action"])
self.assertEqual(9, payload["job_id"])
direct_client.publish.assert_called_once_with(WORKER_CONTROL_CHANNEL, serialized)
direct_client.close.assert_called_once()
@patch("app.services.worker_control_service._probe_linux_worker_instance_count", return_value=0)
@patch("app.services.worker_control_service._run_shell")
@patch("app.services.worker_control_service.probe_systemd_service")
@patch("app.services.worker_control_service._runtime_config")
def test_detect_worker_runtime_prefers_fast_pgrep_probe(
self,
mock_runtime_config,
mock_probe_systemd_service,
mock_run_shell,
_mock_instance_count,
) -> None:
mock_runtime_config.return_value = {
"worker_mode": "linux-systemd",
"worker_service_name": "domaincheck-worker",
}
mock_probe_systemd_service.return_value = {
"mode": "linux-systemd",
"service_name": "domaincheck-worker",
"running": True,
"process_count": 1,
"latest_start_time": "2026-04-22 23:00:00",
"message": "active/running",
}
mock_run_shell.return_value = subprocess.CompletedProcess(
args=["bash", "-lc", "pgrep -fc '[d]etect_worker.py' || true"],
returncode=0,
stdout="80\n",
stderr="",
)
runtime = detect_worker_runtime()
self.assertTrue(runtime["running"])
self.assertEqual(80, runtime["process_count"])
self.assertEqual(1, mock_run_shell.call_count)
@patch("app.services.worker_control_service._probe_linux_worker_instance_count", return_value=0)
@patch("app.services.worker_control_service._run_shell")
@patch("app.services.worker_control_service.probe_systemd_service")
@patch("app.services.worker_control_service._runtime_config")
def test_detect_worker_runtime_falls_back_when_pgrep_probe_is_unavailable(
self,
mock_runtime_config,
mock_probe_systemd_service,
mock_run_shell,
_mock_instance_count,
) -> None:
mock_runtime_config.return_value = {
"worker_mode": "linux-systemd",
"worker_service_name": "domaincheck-worker",
}
mock_probe_systemd_service.return_value = {
"mode": "linux-systemd",
"service_name": "domaincheck-worker",
"running": True,
"process_count": 1,
"latest_start_time": "2026-04-22 23:00:00",
"message": "active/running",
}
mock_run_shell.side_effect = [
subprocess.CompletedProcess(
args=["bash", "-lc", "pgrep -fc '[d]etect_worker.py' || true"],
returncode=0,
stdout="",
stderr="pgrep: command not found\n",
),
subprocess.CompletedProcess(
args=["bash", "-lc", "ps -eo args= | grep '[d]etect_worker.py' | wc -l"],
returncode=0,
stdout="12\n",
stderr="",
),
]
runtime = detect_worker_runtime()
self.assertTrue(runtime["running"])
self.assertEqual(12, runtime["process_count"])
self.assertEqual(2, mock_run_shell.call_count)
@patch("app.services.worker_control_service._probe_linux_worker_process_count", return_value=7)
@patch("app.services.worker_control_service._probe_linux_worker_instance_count", return_value=0)
@patch("app.services.worker_control_service.probe_systemd_service")
@patch("app.services.worker_control_service._runtime_config")
def test_detect_worker_runtime_keeps_service_offline_when_only_unmanaged_processes_exist(
self,
mock_runtime_config,
mock_probe_systemd_service,
_mock_instance_count,
_mock_process_count,
) -> None:
mock_runtime_config.return_value = {
"worker_mode": "linux-systemd",
"worker_service_name": "domaincheck-worker",
}
mock_probe_systemd_service.return_value = {
"mode": "linux-systemd",
"service_name": "domaincheck-worker",
"running": False,
"process_count": 0,
"latest_start_time": "",
"message": "inactive/dead",
}
runtime = detect_worker_runtime()
self.assertFalse(runtime["running"])
self.assertEqual(7, runtime["process_count"])
self.assertIn("unmanaged worker processes", runtime["message"])
@patch("app.services.worker_control_service._probe_linux_worker_process_count", return_value=30)
@patch("app.services.worker_control_service._probe_linux_worker_instance_count", return_value=3)
@patch("app.services.worker_control_service.probe_systemd_service")
@patch("app.services.worker_control_service._runtime_config")
def test_detect_worker_runtime_accepts_active_template_instances_when_base_service_is_inactive(
self,
mock_runtime_config,
mock_probe_systemd_service,
_mock_instance_count,
_mock_process_count,
) -> None:
mock_runtime_config.return_value = {
"worker_mode": "linux-systemd",
"worker_service_name": "domaincheck-worker",
}
mock_probe_systemd_service.return_value = {
"mode": "linux-systemd",
"service_name": "domaincheck-worker",
"running": False,
"process_count": 0,
"latest_start_time": "",
"message": "inactive/dead",
}
runtime = detect_worker_runtime()
self.assertTrue(runtime["running"])
self.assertEqual(30, runtime["process_count"])
self.assertEqual("template instances active (3)", runtime["message"])
@patch("app.services.worker_control_service._expand_linux_worker_control_units")
@patch("app.services.worker_control_service._run_systemctl")
@patch("app.services.worker_control_service._runtime_config")
def test_start_worker_includes_template_instances(
self,
mock_runtime_config,
mock_run_systemctl,
mock_expand_units,
) -> None:
mock_runtime_config.return_value = {
"worker_mode": "linux-systemd",
"worker_service_name": "domaincheck-worker",
}
mock_expand_units.return_value = [
"domaincheck-worker",
"domaincheck-worker@a",
"domaincheck-worker@b",
]
mock_run_systemctl.return_value = subprocess.CompletedProcess(
args=["systemctl", "start", "domaincheck-worker", "domaincheck-worker@a", "domaincheck-worker@b"],
returncode=0,
stdout="",
stderr="",
)
ok, message = start_worker()
self.assertTrue(ok)
self.assertIn("附带 2 个实例", message)
mock_run_systemctl.assert_called_once_with(
["start", "domaincheck-worker", "domaincheck-worker@a", "domaincheck-worker@b"],
timeout=45,
)
@patch("app.services.worker_control_service._expand_linux_worker_control_units")
@patch("app.services.worker_control_service._run_systemctl")
@patch("app.services.worker_control_service._runtime_config")
def test_stop_worker_includes_template_instances(
self,
mock_runtime_config,
mock_run_systemctl,
mock_expand_units,
) -> None:
mock_runtime_config.return_value = {
"worker_mode": "linux-systemd",
"worker_service_name": "domaincheck-worker",
}
mock_expand_units.return_value = [
"domaincheck-worker",
"domaincheck-worker@a",
"domaincheck-worker@b",
]
mock_run_systemctl.return_value = subprocess.CompletedProcess(
args=["systemctl", "stop", "domaincheck-worker", "domaincheck-worker@a", "domaincheck-worker@b"],
returncode=0,
stdout="",
stderr="",
)
ok, message = stop_worker()
self.assertTrue(ok)
self.assertIn("附带 2 个实例", message)
mock_run_systemctl.assert_called_once_with(
["stop", "domaincheck-worker", "domaincheck-worker@a", "domaincheck-worker@b"],
timeout=45,
)
if __name__ == "__main__":
unittest.main()

3
domain-web/src/api/modules.ts
View File

@@ -103,6 +103,9 @@ export const opsApi = {
dispatchJob: (jobId: number) => http.post(`/ops/jobs/${jobId}/dispatch`),
issueAgentToken: (payload: Record<string, unknown>) => http.post("/ops/agent/tokens", payload),
buildAgentBootstrapPlan: (payload: Record<string, unknown>) => http.post("/ops/agent/bootstrap-plan", payload),
migrationSourceProfile: () => http.get("/ops/migration/source-profile"),
previewMigration: (payload: Record<string, unknown>) => http.post("/ops/migration/preview", payload),
executeMigration: (payload: Record<string, unknown>) => http.post("/ops/migration/execute", payload),
releases: (params?: Record<string, unknown>) => http.get("/ops/releases", { params }),
latestRelease: (channel = "stable") => http.get("/ops/releases/latest", { params: { channel } }),
latestReleasePackageMetadata: () => http.get("/ops/releases/package-metadata/latest"),

3
domain-web/src/layouts/MainLayout.vue
View File

@@ -68,7 +68,8 @@ const menuItems = [
{ path: "/runtime", label: "运行中心" },
{ path: "/runtime-debug", label: "运行调试" },
{ path: "/ops-center", label: "运维中枢" },
{ path: "/managed-nodes", label: "托管节点" },
{ path: "/managed-nodes", label: "服务器管理" },
{ path: "/migration", label: "迁移向导" },
{ path: "/settings", label: "系统设置" },
{ path: "/imports", label: "域名导入" },
{ path: "/sensitive-words", label: "敏感词配置" },

3
domain-web/src/router/index.ts
View File

@@ -17,7 +17,8 @@ const routes: RouteRecordRaw[] = [
{ path: "runtime", name: "runtime", component: () => import("@/views/runtime/RuntimeView.vue"), meta: { title: "运行中心", description: "看检测任务有没有真正跑起来，重点盯 running、ppm、步骤队列和节点负载。" } },
{ path: "runtime-debug", name: "runtime-debug", component: () => import("@/views/runtime/RuntimeDebugView.vue"), meta: { title: "运行调试", description: "排查步骤卡点、事件流、调试日志和运行时异常。" } },
{ path: "ops-center", name: "ops-center", component: () => import("@/views/ops/OpsCenterView.vue"), meta: { title: "运维中枢", description: "这里是总驾驶舱，负责发布、运维任务、巡检、接管进度和整体运维视角。" } },
{ path: "managed-nodes", name: "managed-nodes", component: () => import("@/views/ops/ManagedNodesView.vue"), meta: { title: "托管节点", description: "这里专门管理机器本身：新增节点、填 SSH、启用停用、维护接入信息。" } },
{ path: "managed-nodes", name: "managed-nodes", component: () => import("@/views/ops/ManagedNodesView.vue"), meta: { title: "服务器管理", description: "这里专门管理服务器本身：新增机器、填 SSH、启用停用、维护接入信息。" } },
{ path: "migration", name: "migration", component: () => import("@/views/ops/MigrationView.vue"), meta: { title: "迁移向导", description: "面向已 clone 代码的新机器，先预检，再预览计划，最后按选项同步配置、数据库和服务。" } },
{ path: "settings", name: "settings", component: () => import("@/views/settings/SettingsView.vue"), meta: { title: "系统设置", description: "维护系统基础配置、运行参数和默认行为。" } },
{ path: "imports", name: "imports", component: () => import("@/views/imports/ImportsView.vue"), meta: { title: "域名导入", description: "导入待检测域名数据，建立任务源。" } },
{ path: "sensitive-words", name: "sensitive-words", component: () => import("@/views/sensitive-words/SensitiveWordsView.vue"), meta: { title: "敏感词配置", description: "维护过滤词、黑名单词和筛选规则。" } },

127
domain-web/src/views/dashboard/DashboardView.vue
View File

@@ -43,7 +43,7 @@
<div class="summary-card">
<div class="summary-title">当前速度</div>
<div class="summary-value">{{ processedPerMinuteText }}</div>
<div class="summary-note">近窗完成 {{ opsSummary.completed_recent || 0 }} / 失败 {{ opsSummary.failed_recent || 0 }} / 黑名单 {{ opsSummary.blacklisted_recent || 0 }}</div>
<div class="summary-note">近窗完成 {{ opsSummary.completed_recent || 0 }} / 失败 {{ opsSummary.failed_recent || 0 }} / 近窗黑名单 {{ opsSummary.blacklisted_recent || 0 }}</div>
</div>
<div class="summary-card">
<div class="summary-title">预计剩余</div>
@@ -61,18 +61,25 @@
<div class="summary-note">注册待处理 {{ backlogRegisterPendingText }} / 后续步骤 {{ backlogDownstreamPendingText }}</div>
</div>
<div class="summary-card">
<div class="summary-title">有效执行节点</div>
<div class="summary-value">{{ opsSummary.active_execution_nodes || 0 }}</div>
<div class="summary-note">在线 Worker {{ opsSummary.online_worker_nodes || 0 }} / 独立 Worker {{ opsSummary.dedicated_online_worker_nodes || 0 }} / 当前线程负载 {{ queueDisplayRunningText }}</div>
<div class="summary-title">有效执行服务器</div>
<div class="summary-value">{{ activeExecutionServerCount }}</div>
<div class="summary-note">在线 Worker 服务器 {{ opsSummary.online_worker_nodes || 0 }} / 当前参与进程 {{ participatingProcessCountText }} / 当前参与线程 {{ queueDisplayRunningText }}</div>
</div>
<div class="summary-card">
<div class="summary-title">集群代理</div>
<div class="summary-value">{{ clusterProxyAvailableText }}</div>
<div class="summary-note">{{ clusterProxySummaryText }}</div>
</div>
</div>
<el-row v-if="activeJob.job_code" :gutter="12" class="stat-row">
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="当前任务待处理" :value="activeJob.items_pending || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="当前线程负载" :value="activeJobDisplayRunning" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="当前参与线程" :value="activeJobDisplayRunning" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="当前任务执行中" :value="activeJob.items_running || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="当前任务通过" :value="activeJob.items_completed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="当前任务黑名单" :value="activeJob.items_blacklisted || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="近窗黑名单" :value="opsSummary.blacklisted_recent || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="累计黑名单" :value="inventoryBlacklistedTotal" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="当前任务失败" :value="activeJob.items_failed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="6"><el-statistic title="当前进度" :value="activeJob.progress_percent || 0" suffix="%" /></el-col>
</el-row>
@@ -85,29 +92,36 @@
/>
</PageCard>
<PageCard title="步骤队列" description="看每一步堆积、吞吐和失败，快速判断到底卡在注册、百度、360、爱站还是站长之家。">
<PageCard title="步骤队列" description="按后台勾选顺序展示每一步的堆积、吞吐和失败，方便直接判断当前流程推进到哪一步。">
<el-table :data="stepQueue" border empty-text="当前没有活跃步骤队列">
<el-table-column label="顺序" width="70">
<template #default="{ $index }">
{{ $index + 1 }}
</template>
</el-table-column>
<el-table-column prop="step_name" label="步骤" min-width="150" />
<el-table-column prop="items_pending" label="待处理" min-width="90" />
<el-table-column prop="items_claimed" label="已领" min-width="90" />
<el-table-column prop="items_running" label="执行中" min-width="90" />
<el-table-column prop="items_completed" label="完成" min-width="90" />
<el-table-column prop="items_blacklisted" label="黑名单" min-width="90" />
<el-table-column prop="items_completed" label="当前完成" min-width="90" />
<el-table-column prop="items_blacklisted" label="当前黑名单" min-width="90" />
<el-table-column prop="items_failed" label="失败" min-width="90" />
<el-table-column prop="started_recent" label="近窗启动" min-width="100" />
<el-table-column prop="processed_recent" label="近窗处理" min-width="100" />
<el-table-column prop="completed_recent" label="近窗完成" min-width="100" />
<el-table-column prop="blacklisted_recent" label="近窗黑名单" min-width="100" />
<el-table-column prop="failed_recent" label="近窗失败" min-width="100" />
<el-table-column prop="processed_per_minute" label="项/分钟" min-width="100" />
</el-table>
</PageCard>
<PageCard title="节点吞吐" description="看 controller 和 worker 是不是都真在跑，谁在空转，谁在出结果。">
<el-table :data="nodeThroughput" border empty-text="当前没有节点吞吐数据">
<el-table-column prop="node_code" label="节点" min-width="180" />
<PageCard title="服务器吞吐" description="这里按服务器聚合展示，不再把每个进程实例混成节点，方便运营直接看哪台机器真在跑。">
<el-table :data="serverThroughput" border empty-text="当前没有服务器吞吐数据">
<el-table-column prop="server_code" label="服务器" min-width="180" />
<el-table-column prop="process_count" label="参与进程" min-width="100" />
<el-table-column prop="items_running" label="任务执行中" min-width="110" />
<el-table-column prop="display_running" label="线程负载" min-width="100" />
<el-table-column label="活跃线程" min-width="120">
<el-table-column prop="display_running" label="参与线程" min-width="100" />
<el-table-column label="线程状态" min-width="120">
<template #default="{ row }">
{{ formatThreadState(row) }}
</template>
@@ -117,7 +131,7 @@
<el-table-column prop="processed_per_minute" label="项/分钟" min-width="100" />
<el-table-column prop="completed_recent" label="完成" min-width="90" />
<el-table-column prop="failed_recent" label="失败" min-width="90" />
<el-table-column prop="blacklisted_recent" label="黑名单" min-width="90" />
<el-table-column prop="blacklisted_recent" label="近窗黑名单" min-width="100" />
</el-table>
</PageCard>
</div>
@@ -160,6 +174,21 @@ const opsSummary = ref<Record<string, any>>({});
const bottleneckStep = ref<Record<string, any>>({});
const stepQueue = ref<Record<string, any>[]>([]);
const nodeThroughput = ref<Record<string, any>[]>([]);
const inventoryBlacklistedTotal = ref(0);
const resolveServerCode = (rawNodeCode?: string) => {
const normalized = String(rawNodeCode || "").trim();
if (!normalized) return "";
const parts = normalized.split("-");
if (parts.length >= 2) {
const suffix = parts[parts.length - 1];
const parent = parts.slice(0, -1).join("-");
if (/^[a-z]{1,3}$/i.test(suffix) && /\d$/.test(parent)) {
return parent;
}
}
return normalized;
};
const clearRefreshTimer = () => {
if (timer) {
@@ -190,6 +219,58 @@ const processedPerMinuteText = computed(() => {
const activeJobDisplayRunning = computed(() => Number(activeJob.value.items_display_running || activeJob.value.items_running || 0));
const queueDisplayRunningText = computed(() => String(Number(opsSummary.value.queue_display_running_total || 0)));
const participatingProcessCountText = computed(() => String(Number(opsSummary.value.aggregate_process_count || 0)));
const clusterProxyAvailableText = computed(() => String(Number(opsSummary.value.cluster_proxy_available_count || 0)));
const clusterProxySummaryText = computed(() => {
const label = String(opsSummary.value.cluster_proxy_runtime_label || "未知");
const detail = String(opsSummary.value.cluster_proxy_runtime_detail || opsSummary.value.cluster_proxy_last_refresh_status || "").trim();
return detail ? `${label} / ${detail}` : label;
});
const serverThroughput = computed(() => {
const grouped = new Map<string, Record<string, any>>();
for (const rawRow of nodeThroughput.value || []) {
const serverCode = resolveServerCode(String(rawRow?.node_code || ""));
if (!serverCode) continue;
const bucket = grouped.get(serverCode) || {
server_code: serverCode,
process_count: 0,
items_running: 0,
display_running: 0,
items_claimed: 0,
current_load: 0,
active_threads: 0,
max_threads: 0,
processed_recent: 0,
processed_per_minute: 0,
completed_recent: 0,
failed_recent: 0,
blacklisted_recent: 0
};
bucket.process_count += 1;
bucket.items_running += Number(rawRow?.items_running || 0);
bucket.display_running += Number(rawRow?.display_running || 0);
bucket.items_claimed += Number(rawRow?.items_claimed || 0);
bucket.current_load += Number(rawRow?.current_load || 0);
bucket.active_threads += Number(rawRow?.active_threads || 0);
bucket.max_threads += Number(rawRow?.max_threads || 0);
bucket.processed_recent += Number(rawRow?.processed_recent || 0);
bucket.processed_per_minute = Number((bucket.processed_per_minute + Number(rawRow?.processed_per_minute || 0)).toFixed(2));
bucket.completed_recent += Number(rawRow?.completed_recent || 0);
bucket.failed_recent += Number(rawRow?.failed_recent || 0);
bucket.blacklisted_recent += Number(rawRow?.blacklisted_recent || 0);
grouped.set(serverCode, bucket);
}
return [...grouped.values()].sort((a, b) => (
Number(b.processed_recent || 0) - Number(a.processed_recent || 0)
|| Number(b.display_running || 0) - Number(a.display_running || 0)
|| String(a.server_code || "").localeCompare(String(b.server_code || ""))
));
});
const activeExecutionServerCount = computed(() => serverThroughput.value.filter((item) =>
Number(item.items_running || 0) > 0
|| Number(item.items_claimed || 0) > 0
|| Number(item.processed_recent || 0) > 0
).length);
const backlogPendingText = computed(() => String(Number(opsSummary.value.backlog_pending_total || 0)));
@@ -240,11 +321,18 @@ const loadOverview = async (showError = true) => {
backlog_running_total: data.backlog_running_total || 0,
backlog_register_pending_total: data.backlog_register_pending_total || 0,
backlog_downstream_pending_total: data.backlog_downstream_pending_total || 0,
queue_display_running_total: data.queue_display_running_total || 0
queue_display_running_total: data.queue_display_running_total || 0,
aggregate_process_count: data.aggregate_process_count || 0,
aggregate_active_thread_count: data.aggregate_active_thread_count || 0,
cluster_proxy_available_count: data.cluster_proxy_available_count || 0,
cluster_proxy_runtime_label: data.cluster_proxy_runtime_label || "",
cluster_proxy_runtime_detail: data.cluster_proxy_runtime_detail || "",
cluster_proxy_last_refresh_status: data.cluster_proxy_last_refresh_status || ""
};
bottleneckStep.value = data.bottleneck_step || {};
stepQueue.value = Array.isArray(data.step_queue) ? data.step_queue : [];
nodeThroughput.value = Array.isArray(data.node_throughput) ? data.node_throughput : [];
inventoryBlacklistedTotal.value = Number(data.blacklist_total || 0);
stats.value = [
{ label: "总域名", value: String(data.domains_total), note: "数据库域名总量" },
{ label: "库存待检测", value: String(data.pending_total), note: "全库尚未跑完的域名" },
@@ -266,9 +354,14 @@ const loadOverview = async (showError = true) => {
{ label: "API", value: data.api_status, note: "当前 API 运行状态" },
{ label: "本机 Worker", value: localWorkerValue, note: localWorkerNote },
{
label: "有效执行节点",
label: "有效执行服务器",
value: `${data.cluster_worker_status || "offline"} / ${data.cluster_online_worker_nodes || 0}`,
note: `可执行检测的在线节点，独立 Worker ${data.cluster_dedicated_online_worker_nodes || 0}，控制面 ${data.cluster_online_control_nodes || 0}`
note: `可执行检测的在线服务器，当前参与进程 ${data.aggregate_process_count || 0}，当前参与线程 ${data.queue_display_running_total || 0}`
},
{
label: "集群代理",
value: String(data.cluster_proxy_available_count || 0),
note: String(data.cluster_proxy_runtime_label || "未知")
},
{ label: "模式", value: data.worker_mode, note: "当前 Worker 运行模式" }
];

198
domain-web/src/views/detect/DetectView.vue
View File

@@ -31,8 +31,10 @@
<el-descriptions-item label="检测进程数">{{ status.worker_process_count }}</el-descriptions-item>
<el-descriptions-item label="本机线程配置">{{ status.thread_count }}</el-descriptions-item>
<el-descriptions-item label="线程来源">{{ threadCountSummary }}</el-descriptions-item>
<el-descriptions-item v-if="isAggregateDetectView" label="当前汇总线程">{{ threadSnapshotActive }} / {{ threadSnapshotMax }}</el-descriptions-item>
<el-descriptions-item v-if="isAggregateDetectView" label="当前参与节点">{{ participatingNodeCount }}</el-descriptions-item>
<el-descriptions-item v-if="isAggregateDetectView" label="当前参与线程">{{ threadSnapshotActive }} / {{ threadSnapshotMax }}</el-descriptions-item>
<el-descriptions-item v-if="isAggregateDetectView" label="当前参与服务器">{{ participatingServerCount }}</el-descriptions-item>
<el-descriptions-item v-if="isAggregateDetectView" label="当前参与进程">{{ aggregateProcessCount }}</el-descriptions-item>
<el-descriptions-item v-if="isAggregateDetectView" label="参与线程上限">{{ aggregateMaxThreadCount }}</el-descriptions-item>
<el-descriptions-item label="代理启用">{{ status.proxy_enable ? "是" : "否" }}</el-descriptions-item>
<el-descriptions-item label="允许直连">{{ status.allow_direct ? "是" : "否" }}</el-descriptions-item>
<el-descriptions-item label="代理池数量">{{ status.proxy_pool_count }}</el-descriptions-item>
@@ -114,7 +116,8 @@
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="待检测" :value="status.progress.pending || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测中" :value="displayRunningStat" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测通过" :value="status.progress.completed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="黑名单" :value="status.progress.blacklisted || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="当前任务黑名单" :value="status.progress.blacklisted || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="近窗黑名单" :value="queueSummary.throughput?.blacklisted_recent || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测失败" :value="status.progress.failed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="可注册" :value="registerableStat" /></el-col>
</el-row>
@@ -136,9 +139,9 @@
<div class="summary-note">{{ currentPhaseDetail }}</div>
</div>
<div class="summary-card">
<div class="summary-title">活跃线程</div>
<div class="summary-title">参与线程</div>
<div class="summary-value">{{ threadSnapshotActive }} / {{ threadSnapshotMax }}</div>
<div class="summary-note">当前参与节点实际活跃线程 / 配置线程上限</div>
<div class="summary-note">当前参与服务器汇总线程 / 配置线程上限</div>
</div>
<div class="summary-card">
<div class="summary-title">当前速度</div>
@@ -161,9 +164,10 @@
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务待领" :value="activeJob.items_pending || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务已领" :value="activeJobDisplayClaimed" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务执行中" :value="activeJobDisplayRunning" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="活跃线程" :value="threadSnapshotActive" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="参与线程" :value="threadSnapshotActive" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务完成" :value="activeJob.items_completed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务黑名单" :value="activeJob.items_blacklisted || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="近窗黑名单" :value="queueSummary.throughput?.blacklisted_recent || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务失败" :value="activeJob.items_failed || 0" /></el-col>
</el-row>
@@ -216,25 +220,39 @@
:title="queueAlertText"
/>
<el-table :data="queueSummary.nodes || []" border style="margin-bottom: 16px" empty-text="当前没有可展示的节点吞吐摘要">
<el-table-column prop="node_code" label="节点" min-width="180" />
<el-table
:data="queueSummaryServerRows"
border
:max-height="sixRowTableMaxHeight"
style="margin-bottom: 16px"
empty-text="当前没有可展示的服务器吞吐摘要"
>
<el-table-column prop="server_code" label="服务器" min-width="180" />
<el-table-column prop="process_count" label="参与进程" min-width="100" />
<el-table-column prop="items_running" label="任务执行中" min-width="100" />
<el-table-column prop="items_claimed" label="已领未跑" min-width="100" />
<el-table-column prop="processed_recent" label="近窗处理" min-width="100" />
<el-table-column prop="processed_per_minute" label="项/分钟" min-width="100" />
<el-table-column prop="completed_recent" label="完成" min-width="90" />
<el-table-column prop="blacklisted_recent" label="黑名单" min-width="90" />
<el-table-column prop="blacklisted_recent" label="近窗黑名单" min-width="100" />
<el-table-column prop="failed_recent" label="失败" min-width="90" />
</el-table>
</div>
<el-table v-if="activeJobDisplayNodeStats.length" :data="activeJobDisplayNodeStats" border style="margin-bottom: 16px">
<el-table-column prop="node_code" label="节点" min-width="180" />
<el-table
v-if="activeJobDisplayServerStats.length"
:data="activeJobDisplayServerStats"
border
:max-height="sixRowTableMaxHeight"
style="margin-bottom: 16px"
>
<el-table-column prop="server_code" label="服务器" min-width="180" />
<el-table-column prop="process_count" label="参与进程" min-width="100" />
<el-table-column prop="items_total" label="任务总数" min-width="100" />
<el-table-column prop="items_pending" label="待领" min-width="90" />
<el-table-column prop="items_claimed" label="已领" min-width="90" />
<el-table-column prop="items_running" label="任务执行中" min-width="100" />
<el-table-column label="活跃线程" min-width="130">
<el-table-column label="参与线程" min-width="130">
<template #default="{ row }">
<span>{{ row.active_threads || 0 }} / {{ row.max_threads || 0 }}</span>
</template>
@@ -287,10 +305,12 @@
<div class="task-detail-meta">
<el-switch v-model="logAutoFollow" inline-prompt active-text="跟随日志" inactive-text="暂停跟随" size="small" />
<el-button text size="small" @click="scrollLogToBottom(true)">回到底部</el-button>
<span>{{ isAggregateDetectView ? "参与节点" : "本机 Worker" }}：{{ isAggregateDetectView ? participatingNodeCount : (status.worker_online ? "在线" : "离线") }}</span>
<span>{{ isAggregateDetectView ? "参与服务器" : "本机 Worker" }}：{{ isAggregateDetectView ? participatingServerCount : (status.worker_online ? "在线" : "离线") }}</span>
<span v-if="isAggregateDetectView">参与进程：{{ aggregateProcessCount }}</span>
<span v-if="isAggregateDetectView">参与线程：{{ threadSnapshotActive }}</span>
<span v-if="selectedRun?.phase_label">阶段：{{ selectedRun.phase_label }}</span>
<span>运行中：{{ displayRunningStat }}</span>
<span>线程：{{ runtimeThreadSummary }}</span>
<span>{{ isAggregateDetectView ? "线程上限" : "线程配置" }}：{{ runtimeThreadSummary }}</span>
</div>
</div>
<div v-if="selectedRun?.phase_detail" class="phase-detail">{{ selectedRun.phase_detail }}</div>
@@ -322,7 +342,13 @@
</el-table>
</div>
<el-table v-if="phaseHistory.length" :data="phaseHistory" border style="margin-top: 16px">
<el-table
v-if="phaseHistory.length"
:data="phaseHistory"
border
:max-height="sixRowTableMaxHeight"
style="margin-top: 16px"
>
<el-table-column prop="at" label="切换时间" min-width="180" />
<el-table-column prop="label" label="阶段" min-width="120" />
<el-table-column prop="detail" label="阶段说明" min-width="360" show-overflow-tooltip />
@@ -338,6 +364,7 @@ import PageCard from "@/components/PageCard.vue";
import { detectApi, settingsApi } from "@/api/modules";
const DETECT_LAST_ACTION_KEY = "domaincheck:detect:last-action";
const sixRowTableMaxHeight = 336;
const router = useRouter();
const status = ref({
worker_online: false,
@@ -354,6 +381,10 @@ const status = ref({
thread_count_node_code: "",
active_thread_count: 0,
max_thread_count: 0,
aggregate_process_count: 0,
aggregate_participating_node_count: 0,
aggregate_participating_node_codes: [] as string[],
aggregate_max_thread_count: 0,
proxy_enable: false,
allow_direct: false,
proxy_pool_count: 0,
@@ -392,12 +423,77 @@ const status = ref({
}
});
const resolveServerCode = (rawNodeCode?: string) => {
const normalized = String(rawNodeCode || "").trim();
if (!normalized) return "";
const parts = normalized.split("-");
if (parts.length >= 2) {
const suffix = parts[parts.length - 1];
const parent = parts.slice(0, -1).join("-");
if (/^[a-z]{1,3}$/i.test(suffix) && /\d$/.test(parent)) {
return parent;
}
}
return normalized;
};
const aggregateServerRows = (rows: any[]) => {
const grouped = new Map<string, Record<string, any>>();
for (const rawItem of rows || []) {
const serverCode = resolveServerCode(String(rawItem?.node_code || ""));
if (!serverCode || serverCode === "unassigned") continue;
const bucket = grouped.get(serverCode) || {
server_code: serverCode,
process_count: 0,
items_total: 0,
items_pending: 0,
items_claimed: 0,
items_running: 0,
items_completed: 0,
items_blacklisted: 0,
items_failed: 0,
processed_recent: 0,
processed_per_minute: 0,
completed_recent: 0,
blacklisted_recent: 0,
failed_recent: 0,
display_running: 0,
current_load: 0,
active_threads: 0,
max_threads: 0
};
bucket.process_count += 1;
bucket.items_total += Number(rawItem?.items_total || 0);
bucket.items_pending += Number(rawItem?.items_pending || 0);
bucket.items_claimed += Number(rawItem?.items_claimed || 0);
bucket.items_running += Number(rawItem?.items_running || 0);
bucket.items_completed += Number(rawItem?.items_completed || 0);
bucket.items_blacklisted += Number(rawItem?.items_blacklisted || 0);
bucket.items_failed += Number(rawItem?.items_failed || 0);
bucket.processed_recent += Number(rawItem?.processed_recent || 0);
bucket.processed_per_minute = Number((bucket.processed_per_minute + Number(rawItem?.processed_per_minute || 0)).toFixed(2));
bucket.completed_recent += Number(rawItem?.completed_recent || 0);
bucket.blacklisted_recent += Number(rawItem?.blacklisted_recent || 0);
bucket.failed_recent += Number(rawItem?.failed_recent || 0);
bucket.display_running += Number(rawItem?.display_running || 0);
bucket.current_load += Number(rawItem?.current_load || 0);
bucket.active_threads += Number(rawItem?.active_threads || 0);
bucket.max_threads += Number(rawItem?.max_threads || 0);
grouped.set(serverCode, bucket);
}
return [...grouped.values()].sort((a, b) => (
Number(b.processed_recent || 0) - Number(a.processed_recent || 0)
|| Number(b.display_running || 0) - Number(a.display_running || 0)
|| String(a.server_code || "").localeCompare(String(b.server_code || ""))
));
};
const threadCountSummary = computed(() => {
const effective = Number(status.value.thread_count || 0);
const defaultValue = Number(status.value.thread_count_default || 0);
const nodeCode = String(status.value.thread_count_node_code || "").trim();
if (status.value.thread_count_source === "node_override" && status.value.thread_count_override) {
return `${effective}（节点 ${nodeCode || "-"} 单独覆盖，默认 ${defaultValue}）`;
return `${effective}（服务器 ${nodeCode || "-"} 单独覆盖，默认 ${defaultValue}）`;
}
return `${effective}（默认）`;
});
@@ -415,7 +511,8 @@ const queueSummary = ref({
},
throughput: {
processed_recent: 0,
processed_per_minute: 0
processed_per_minute: 0,
blacklisted_recent: 0
},
nodes: [] as any[]
});
@@ -466,7 +563,20 @@ const activeJobDisplayClaimed = computed(() => Number(activeJob.value?.display_i
const activeJobDisplayRunning = computed(() => Math.max(
Number(activeJob.value?.items_running ?? 0),
Number(activeJob.value?.display_active_threads ?? 0),
Number(activeJob.value?.display_items_running ?? 0),
));
const aggregateProcessCount = computed(() => {
if (isAggregateDetectView.value) {
return currentParticipatingProcessRows.value.length || Number(status.value.aggregate_process_count || status.value.worker_process_count || 0);
}
return Number(status.value.worker_process_count || 0);
});
const aggregateMaxThreadCount = computed(() => {
if (isAggregateDetectView.value) {
return Number(status.value.aggregate_max_thread_count || 0);
}
return Number(status.value.max_thread_count || status.value.thread_count || 0);
});
const threadSnapshotActive = computed(() => {
const jobValue = Number(activeJob.value?.display_active_threads ?? 0);
if (jobValue > 0) {
@@ -476,6 +586,10 @@ const threadSnapshotActive = computed(() => {
});
const threadSnapshotMax = computed(() => {
const jobValue = Number(activeJob.value?.display_max_threads ?? 0);
const aggregateValue = Number(status.value.aggregate_max_thread_count || 0);
if (aggregateValue > 0) {
return Math.max(jobValue, aggregateValue);
}
if (jobValue > 0) {
return jobValue;
}
@@ -490,9 +604,23 @@ const activeJobDisplayNodeStats = computed(() => {
}
return Array.isArray(activeJob.value?.node_stats) ? activeJob.value.node_stats : [];
});
const participatingNodeCount = computed(() => {
const rows = activeJobDisplayNodeStats.value.filter((item) => String(item?.node_code || "").trim() && String(item?.node_code || "").trim() !== "unassigned");
return rows.length || 0;
const currentParticipatingProcessRows = computed(() =>
activeJobDisplayNodeStats.value.filter((item) => {
const nodeCode = String(item?.node_code || "").trim();
if (!nodeCode || nodeCode === "unassigned") return false;
return [
Number(item?.items_claimed || 0),
Number(item?.items_running || 0),
Number(item?.display_running || 0),
Number(item?.current_load || 0),
Number(item?.active_threads || 0)
].some((value) => value > 0);
})
);
const activeJobDisplayServerStats = computed(() => aggregateServerRows(activeJobDisplayNodeStats.value));
const queueSummaryServerRows = computed(() => aggregateServerRows(Array.isArray(queueSummary.value.nodes) ? queueSummary.value.nodes : []));
const participatingServerCount = computed(() => {
return activeJobDisplayServerStats.value.length || Number(status.value.aggregate_participating_node_count || 0) || 0;
});
const queueProcessedPerMinute = computed(() => Number(queueSummary.value.throughput?.processed_per_minute || 0));
const queueProcessedPerHour = computed(() => Number((queueProcessedPerMinute.value * 60).toFixed(2)));
@@ -521,7 +649,7 @@ const estimatedRemainingText = computed(() => {
});
const runtimeThreadSummary = computed(() => {
if (isAggregateDetectView.value) {
return `${threadSnapshotActive.value} / ${threadSnapshotMax.value}（参与节点汇总）`;
return `${threadSnapshotActive.value} / ${threadSnapshotMax.value}（参与服务器汇总）`;
}
return threadCountSummary.value;
});
@@ -532,16 +660,32 @@ const activeJobSummaryText = computed(() => {
const job = activeJob.value;
const displayClaimed = Number(job.display_items_claimed ?? job.items_claimed ?? 0);
const displayRunning = activeJobDisplayRunning.value;
const displayActiveThreads = Number(job.display_active_threads ?? 0);
const displayMaxThreads = Number(job.display_max_threads ?? 0);
return `状态：${jobStatusText(job.status)} / 总数 ${job.items_total || 0} / 待领 ${job.items_pending || 0} / 已领 ${displayClaimed} / 实时执行 ${displayRunning} / 活跃线程 ${displayActiveThreads}/${displayMaxThreads} / 速度 ${queueProcessedPerMinute.value} 项/分钟 / 预计剩余 ${estimatedRemainingText.value}`;
const displayActiveThreads = threadSnapshotActive.value;
const displayMaxThreads = threadSnapshotMax.value;
return `状态：${jobStatusText(job.status)} / 总数 ${job.items_total || 0} / 待领 ${job.items_pending || 0} / 已领 ${displayClaimed} / 实时执行 ${displayRunning} / 参与线程 ${displayActiveThreads}/${displayMaxThreads} / 速度 ${queueProcessedPerMinute.value} 项/分钟 / 预计剩余 ${estimatedRemainingText.value}`;
});
const selectedRun = computed(() => {
if (isAggregateDetectView.value) {
return null;
}
return runs.value.find((item) => item.run_id === selectedRunId.value) || runs.value[0] || null;
});
const currentPhaseLabel = computed(() => {
if (selectedRun.value?.phase_label) {
return selectedRun.value.phase_label;
}
if (isAggregateDetectView.value && activeJob.value?.job_code) {
return "集群执行中";
}
return status.value.worker_online ? "运行中" : "未启动";
});
const selectedRun = computed(() => runs.value.find((item) => item.run_id === selectedRunId.value) || runs.value[0] || null);
const currentPhaseLabel = computed(() => selectedRun.value?.phase_label || (status.value.worker_online ? "运行中" : "未启动"));
const currentPhaseDetail = computed(() => {
if (selectedRun.value?.phase_detail) {
return selectedRun.value.phase_detail;
}
if (isAggregateDetectView.value && activeJob.value?.job_code) {
return status.value.recent_event || `当前汇总执行由 ${participatingServerCount.value} 台服务器、${aggregateProcessCount.value} 个进程承担。`;
}
if (status.value.recent_event) {
return status.value.recent_event;
}
@@ -629,7 +773,7 @@ const remoteLogMirrorText = computed(() => {
});
const detectScopeSummaryText = computed(() => {
if (isAggregateDetectView.value) {
return `当前检测状态：${currentPhaseLabel}；当前页展示的是多节点汇总运行态，参与节点 ${participatingNodeCount.value} 台，实时活跃线程 ${threadSnapshotActive.value}/${threadSnapshotMax.value}。`;
return `当前检测状态：${currentPhaseLabel}；当前页展示的是多服务器汇总运行态，参与服务器 ${participatingServerCount.value} 台，参与进程 ${aggregateProcessCount.value} 个，参与线程 ${threadSnapshotActive.value}/${threadSnapshotMax.value}。`;
}
return `当前检测状态：${currentPhaseLabel}；本机 Worker ${status.value.worker_online ? "在线" : "离线"}。`;
});

12
domain-web/src/views/ops/ManagedNodesView.vue
View File

@@ -1,5 +1,5 @@
<template>
<PageCard title="托管节点" description="专门用于纳管机器、维护 SSH 信息、启停节点。运维中枢继续负责总驾驶舱、任务、发布和巡检。">
<PageCard title="服务器管理" description="专门用于纳管服务器、维护 SSH 信息、启停机器。运维中枢继续负责总驾驶舱、任务、发布和巡检。">
<template #header-extra>
<div class="header-actions">
<el-button plain :loading="loading.syncing" @click="syncNodesFromCluster">同步节点</el-button>
@@ -14,12 +14,12 @@
show-icon
type="info"
style="margin-bottom: 16px"
title="这里是独立的节点管理入口。后面你新增机器、填 SSH、启用/停用节点，都从这里做，不用再去运维中枢长页面里找。"
title="这里是独立的服务器管理入口。后面你新增机器、填 SSH、启用/停用服务器，都从这里做，不用再去运维中枢长页面里找。"
/>
<div class="summary-grid">
<div class="summary-card">
<span class="summary-label">总节点</span>
<span class="summary-label">总服务器</span>
<strong>{{ summary.total }}</strong>
<span class="summary-note">已纳管 {{ summary.managedTotal }} / 已启用 {{ summary.enabled }}</span>
</div>
@@ -29,14 +29,14 @@
<span class="summary-note">stale {{ summary.stale }} / 待接入 {{ summary.pendingBootstrap }}</span>
</div>
<div class="summary-card">
<span class="summary-label">执行节点</span>
<span class="summary-label">执行服务器</span>
<strong>{{ summary.participating }}</strong>
<span class="summary-note">在线待命 {{ summary.standby }} / 死信 {{ summary.queueDeadLetterNodes }}</span>
</div>
</div>
<el-table :data="nodes" stripe size="small" height="100%">
<el-table-column label="节点" min-width="220">
<el-table-column label="服务器" min-width="220">
<template #default="{ row }">
<div class="node-cell">
<strong>{{ row.node_code || "-" }}</strong>
@@ -89,7 +89,7 @@
</el-table-column>
</el-table>
<el-dialog v-model="managedNodeVisible" :title="managedNodeEditing ? '编辑托管节点' : '新增托管节点'" width="760px">
<el-dialog v-model="managedNodeVisible" :title="managedNodeEditing ? '编辑服务器' : '新增服务器'" width="760px">
<el-form label-position="top" :model="managedNodeForm" class="dialog-form">
<div class="dialog-grid">
<el-form-item label="节点编码">

818
domain-web/src/views/ops/MigrationView.vue Normal file
View File

@@ -0,0 +1,818 @@
<template>
<PageCard title="迁移向导" description="目标机已先手动 clone 代码后，在这里复用托管节点 SSH 信息做预检查、计划预览和迁移执行。">
<template #header-extra>
<div class="header-actions">
<el-button plain :loading="loading.nodes" @click="loadNodes">刷新节点</el-button>
<el-button plain :loading="loading.source" @click="loadSourceProfile">刷新源信息</el-button>
</div>
</template>
<el-alert
:closable="false"
type="info"
show-icon
title="当前 MVP 只覆盖“海外中控机迁到一台已 clone 代码的新机器”场景。数据库覆盖默认关闭，建议先预检、看计划，再执行。"
style="margin-bottom: 16px"
/>
<el-alert
v-if="!migrationApiSupported"
:closable="false"
type="warning"
show-icon
:title="backendCompatibility.message || defaultUnavailableMessage"
style="margin-bottom: 16px"
/>
<div class="section-grid">
<div class="summary-card">
<span class="summary-label">源节点</span>
<strong>{{ sourceProfile.source_node?.node_code || "-" }}</strong>
<span class="summary-note">
{{ sourceProfile.source_node?.region || "-" }} / {{ sourceProfile.source_node?.role || "-" }} / {{ sourceProfile.source_node?.worker_mode || "-" }}
</span>
</div>
<div class="summary-card">
<span class="summary-label">源数据库</span>
<strong>{{ sourceProfile.database?.database || "-" }}</strong>
<span class="summary-note">
{{ sourceProfile.database?.user || "-" }} @ {{ sourceProfile.database?.host || "-" }}:{{ sourceProfile.database?.port || "-" }}
</span>
</div>
<div class="summary-card">
<span class="summary-label">目标节点</span>
<strong>{{ currentTargetSummary }}</strong>
<span class="summary-note">{{ currentTargetSshSummary }}</span>
</div>
</div>
<el-form label-position="top" :model="form" class="migration-form">
<div class="form-grid">
<el-form-item label="目标托管节点">
<el-select v-model="form.target_node_code" filterable placeholder="请选择已配置 SSH 的托管节点">
<el-option
v-for="item in targetNodes"
:key="item.node_code"
:label="`${item.node_code} (${item.ssh_user || '-'}@${item.ssh_host || '-'})`"
:value="item.node_code"
/>
</el-select>
</el-form-item>
<el-form-item label="目标仓库路径">
<el-input v-model="form.target_repo_path" placeholder="/www/wwwroot/getDomain" />
</el-form-item>
<el-form-item label="目标 domainRoot">
<el-input v-model="form.target_domain_root" placeholder="/opt/domaincheck/domainCheck" />
</el-form-item>
<el-form-item label="目标 apiRoot">
<el-input v-model="form.target_api_root" placeholder="/opt/domaincheck/domain-api" />
</el-form-item>
<el-form-item label="目标 webRoot">
<el-input v-model="form.target_web_root" placeholder="/opt/domaincheck/domain-web" />
</el-form-item>
</div>
<div class="toggle-grid">
<el-switch v-model="form.sync_env_files" active-text="同步 env 配置" inactive-text="跳过 env 配置" />
<el-switch v-model="form.sync_systemd_units" active-text="同步 systemd" inactive-text="跳过 systemd" />
<el-switch v-model="form.build_frontend" active-text="远端重建前端" inactive-text="跳过前端构建" />
<el-switch v-model="form.restart_services" active-text="执行后重启服务" inactive-text="执行后不重启服务" />
<el-switch v-model="form.overwrite_database" active-text="覆盖目标数据库" inactive-text="不覆盖目标数据库" />
<el-switch
v-model="form.backup_target_database"
:disabled="!form.overwrite_database"
active-text="覆盖前先备份目标库"
inactive-text="跳过目标库备份"
/>
</div>
<div v-if="form.overwrite_database" class="form-grid" style="margin-top: 8px">
<el-form-item label="目标 DB Host">
<el-input v-model="form.target_db_host" placeholder="留空则尝试从目标机 .env 自动识别" />
</el-form-item>
<el-form-item label="目标 DB Port">
<el-input-number v-model="form.target_db_port" :min="0" :max="65535" />
</el-form-item>
<el-form-item label="目标 DB Name">
<el-input v-model="form.target_db_name" placeholder="留空则尝试从目标机 .env 自动识别" />
</el-form-item>
<el-form-item label="目标 DB User">
<el-input v-model="form.target_db_user" placeholder="留空则尝试从目标机 .env 自动识别" />
</el-form-item>
<el-form-item label="目标 DB Password" class="full-width">
<el-input v-model="form.target_db_password" type="password" show-password placeholder="留空则尝试从目标机 .env 自动识别" />
</el-form-item>
</div>
</el-form>
<el-alert
v-if="previewStale"
:closable="false"
type="warning"
show-icon
title="表单参数已变化，当前预检令牌已失效，请重新执行“预检查并生成计划”。"
style="margin-bottom: 16px"
/>
<div class="action-row">
<el-button type="primary" plain :disabled="!migrationApiSupported" :loading="loading.preview" @click="previewMigration">预检查并生成计划</el-button>
<el-button type="danger" plain :disabled="!migrationApiSupported || !canExecute" :loading="loading.execute" @click="executeMigration">执行迁移</el-button>
</div>
<div v-if="previewResult" class="result-section">
<div class="result-header">
<strong>预检查结果</strong>
<el-tag :type="previewResult.code === 0 ? 'success' : 'danger'">{{ previewResult.message || "预检查完成" }}</el-tag>
</div>
<div v-if="previewBlockingReasons.length" class="preview-list">
<div v-for="item in previewBlockingReasons" :key="`blocking-${item}`" class="preview-list__item preview-list__item--danger">{{ item }}</div>
</div>
<div v-if="previewWarnings.length" class="preview-list">
<div v-for="item in previewWarnings" :key="`warning-${item}`" class="preview-list__item preview-list__item--warning">{{ item }}</div>
</div>
<div class="inspection-summary-grid" style="margin-top: 12px">
<div class="mini-card">
<span class="summary-label">目标 Git Commit</span>
<strong>{{ previewData.remote_checks?.git_commit || "-" }}</strong>
<span class="summary-note">预检读取目标仓库 HEAD</span>
</div>
<div class="mini-card">
<span class="summary-label">目标数据库</span>
<strong>{{ previewData.target_db_config?.database || "-" }}</strong>
<span class="summary-note">
{{ previewData.target_db_config?.user || "-" }} @ {{ previewData.target_db_config?.host || "-" }}:{{ previewData.target_db_config?.port || "-" }}
</span>
</div>
<div class="mini-card">
<span class="summary-label">目标库现状</span>
<strong>{{ targetDbInspection.available ? (targetDbInspection.has_business_data ? "非空库" : "近似空库") : "未识别" }}</strong>
<span class="summary-note">
表 {{ targetDbInspection.public_table_count || 0 }} / 业务表 {{ targetDbInspection.business_table_count || 0 }} / 估算行 {{ targetDbInspection.approx_total_rows || 0 }}
</span>
</div>
<div class="mini-card">
<span class="summary-label">工具面</span>
<strong>{{ readyToolCount }} / {{ totalToolCount }}</strong>
<span class="summary-note">python3 / node / npm / systemctl / psql / pg_dump / curl</span>
</div>
</div>
<div class="result-grid">
<div class="preview-box">
<div class="preview-box__title">执行计划</div>
<div class="preview-list">
<div v-for="step in previewPlanSteps" :key="step.key" class="preview-list__item">{{ step.title }}</div>
</div>
</div>
<div class="preview-box">
<div class="preview-box__title">目标机检查</div>
<div class="preview-list">
<div class="preview-list__item">repo_exists: {{ boolLabel(previewData.remote_checks?.paths?.repo_exists) }}</div>
<div class="preview-list__item">repo_git: {{ boolLabel(previewData.remote_checks?.paths?.repo_git) }}</div>
<div class="preview-list__item">domain_root_exists: {{ boolLabel(previewData.remote_checks?.paths?.domain_root_exists) }}</div>
<div class="preview-list__item">api_root_exists: {{ boolLabel(previewData.remote_checks?.paths?.api_root_exists) }}</div>
<div class="preview-list__item">web_root_exists: {{ boolLabel(previewData.remote_checks?.paths?.web_root_exists) }}</div>
</div>
</div>
</div>
<div v-if="requiredConfirmationText" class="preview-box" style="margin-top: 12px">
<div class="preview-box__title">高风险确认</div>
<div class="preview-list" style="margin-bottom: 12px">
<div class="preview-list__item preview-list__item--warning">
目标库探测到已有业务数据。执行前需要输入确认文案：<strong>{{ requiredConfirmationText }}</strong>
</div>
</div>
<el-input
v-model="form.execute_confirmation_text"
:placeholder="`请输入 ${requiredConfirmationText}`"
clearable
/>
</div>
</div>
<div v-if="executeResult" class="result-section">
<div class="result-header">
<strong>提交结果</strong>
<el-tag :type="executeResult.code === 0 ? 'success' : 'danger'">{{ executeResult.message || "执行完成" }}</el-tag>
</div>
<el-descriptions :column="2" border size="small">
<el-descriptions-item label="任务号">{{ activeMigrationJob.job_code || executeResult.data?.job?.job_code || "-" }}</el-descriptions-item>
<el-descriptions-item label="任务状态">
<el-tag :type="jobStatusType(activeMigrationJob.status || executeResult.data?.job?.status || '')" size="small">
{{ jobStatusLabel(activeMigrationJob.status || executeResult.data?.job?.status || "") }}
</el-tag>
</el-descriptions-item>
<el-descriptions-item label="目标节点">{{ activeMigrationJob.target_node_code || executeResult.data?.job?.target_node_code || "-" }}</el-descriptions-item>
<el-descriptions-item label="最近更新">{{ activeMigrationJob.updated_at || activeMigrationJob.finished_at || "-" }}</el-descriptions-item>
</el-descriptions>
</div>
<div v-if="activeMigrationJob.id" class="result-section">
<div class="result-header">
<strong>后台迁移任务</strong>
<div class="header-actions">
<el-button plain size="small" :loading="loading.jobDetail || loading.jobEvents" @click="refreshMigrationJobProgress">刷新日志</el-button>
<el-button type="primary" plain size="small" @click="migrationLogVisible = true">查看日志窗口</el-button>
</div>
</div>
<div class="inspection-summary-grid" style="margin-bottom: 12px">
<div class="mini-card">
<span class="summary-label">任务号</span>
<strong>{{ activeMigrationJob.job_code || "-" }}</strong>
<span class="summary-note">ID {{ activeMigrationJob.id || "-" }}</span>
</div>
<div class="mini-card">
<span class="summary-label">状态</span>
<strong>{{ jobStatusLabel(activeMigrationJob.status || "") }}</strong>
<span class="summary-note">{{ activeMigrationJob.updated_at || activeMigrationJob.finished_at || activeMigrationJob.created_at || "-" }}</span>
</div>
<div class="mini-card">
<span class="summary-label">事件流</span>
<strong>{{ migrationJobEventStream.length }}</strong>
<span class="summary-note">最近 {{ migrationJobEventsSummary.latest_at || "-" }}</span>
</div>
<div class="mini-card">
<span class="summary-label">自动刷新</span>
<strong>{{ migrationJobIsTerminal ? "已停止" : "运行中" }}</strong>
<span class="summary-note">{{ migrationJobIsTerminal ? "任务已收口" : "每 2 秒刷新一次" }}</span>
</div>
</div>
<el-alert
v-if="activeMigrationJob.error_message"
:closable="false"
type="error"
show-icon
style="margin-bottom: 12px"
>
{{ activeMigrationJob.error_message }}
</el-alert>
<el-table v-if="executionSteps.length" :data="executionSteps" stripe size="small">
<el-table-column prop="title" label="步骤" min-width="180" />
<el-table-column label="结果" width="100">
<template #default="{ row }">
<el-tag :type="row.ok ? 'success' : 'danger'" size="small">{{ row.ok ? "成功" : "失败" }}</el-tag>
</template>
</el-table-column>
<el-table-column prop="message" label="说明" min-width="240" />
</el-table>
</div>
<el-drawer v-model="migrationLogVisible" title="迁移执行日志" size="880px">
<template #default>
<div class="drawer-meta">
<span>任务：{{ activeMigrationJob.job_code || "-" }}</span>
<span>状态：{{ jobStatusLabel(activeMigrationJob.status || "") }}</span>
<span>节点：{{ activeMigrationJob.target_node_code || "-" }}</span>
<span>事件：{{ migrationJobEventStream.length }}</span>
</div>
<div class="log-console">
<div v-for="item in migrationJobEventStream" :key="item.id || `${item.event_type}-${item.created_at}`" class="log-entry">
<div class="log-entry__meta">
<span>{{ item.created_at || "-" }}</span>
<el-tag :type="eventLevelType(item.level || 'info')" size="small">{{ item.level_label || item.level || "-" }}</el-tag>
<span>{{ item.event_type || "-" }}</span>
</div>
<div class="log-entry__message">{{ item.summary || item.message || "-" }}</div>
<pre v-if="migrationEventDetailText(item)" class="log-entry__detail">{{ migrationEventDetailText(item) }}</pre>
</div>
<div v-if="!migrationJobEventStream.length" class="preview-box">
<div class="preview-list__item">当前还没有新的迁移事件日志。</div>
</div>
</div>
</template>
</el-drawer>
</PageCard>
</template>
<script setup lang="ts">
import { computed, onBeforeUnmount, onMounted, ref } from "vue";
import { ElMessage } from "element-plus";
import PageCard from "@/components/PageCard.vue";
import { opsApi } from "@/api/modules";
type GenericRecord = Record<string, any>;
const loading = ref({
nodes: false,
source: false,
preview: false,
execute: false,
jobDetail: false,
jobEvents: false
});
const sourceProfile = ref<GenericRecord>({});
const targetNodes = ref<GenericRecord[]>([]);
const previewResult = ref<GenericRecord | null>(null);
const executeResult = ref<GenericRecord | null>(null);
const activeMigrationJob = ref<GenericRecord>({});
const migrationJobEvents = ref<GenericRecord[]>([]);
const migrationJobEventsSummary = ref<GenericRecord>({});
const migrationLogVisible = ref(false);
const lastPreviewPayloadSignature = ref("");
const defaultUnavailableMessage = "当前后端还没部署迁移向导接口，请先发布或重启 domaincheck-api 到最新版本。";
const backendCompatibility = ref({
migrationApiSupported: true,
message: ""
});
let migrationJobPollTimer: ReturnType<typeof setInterval> | null = null;
const form = ref({
target_node_code: "",
target_repo_path: "/www/wwwroot/getDomain",
target_domain_root: "/opt/domaincheck/domainCheck",
target_api_root: "/opt/domaincheck/domain-api",
target_web_root: "/opt/domaincheck/domain-web",
sync_env_files: true,
sync_systemd_units: true,
build_frontend: true,
restart_services: true,
overwrite_database: false,
backup_target_database: true,
target_db_host: "",
target_db_port: 0,
target_db_name: "",
target_db_user: "",
target_db_password: "",
execute_confirmation_text: ""
});
const currentTarget = computed(
() => targetNodes.value.find((item) => String(item.node_code || "") === String(form.value.target_node_code || "")) || {}
);
const migrationApiSupported = computed(() => backendCompatibility.value.migrationApiSupported);
const currentTargetSummary = computed(() => String(currentTarget.value.node_code || "-"));
const currentTargetSshSummary = computed(() => {
const user = String(currentTarget.value.ssh_user || "-").trim() || "-";
const host = String(currentTarget.value.ssh_host || "-").trim() || "-";
const port = Number(currentTarget.value.ssh_port || 22) || 22;
return `${user}@${host}:${port}`;
});
const previewData = computed(() => previewResult.value?.data || {});
const previewPlanSteps = computed(() => Array.isArray(previewData.value?.plan_steps) ? previewData.value.plan_steps : []);
const previewBlockingReasons = computed(() => Array.isArray(previewData.value?.blocking_reasons) ? previewData.value.blocking_reasons : []);
const previewWarnings = computed(() => Array.isArray(previewData.value?.warnings) ? previewData.value.warnings : []);
const readyToolCount = computed(() => Object.values(previewData.value?.remote_checks?.tools || {}).filter(Boolean).length);
const totalToolCount = computed(() => Object.keys(previewData.value?.remote_checks?.tools || {}).length);
const requiredConfirmationText = computed(() => String(previewData.value?.execution_guard?.required_confirmation_text || "").trim());
const previewToken = computed(() => String(previewData.value?.execution_guard?.token || "").trim());
const targetDbInspection = computed(() => previewData.value?.target_db_inspection || {});
const basePayloadSignature = computed(() => JSON.stringify(buildBasePayload()));
const previewStale = computed(
() => Boolean(previewResult.value) && lastPreviewPayloadSignature.value !== "" && lastPreviewPayloadSignature.value !== basePayloadSignature.value
);
const canExecute = computed(
() =>
previewResult.value?.code === 0 &&
migrationApiSupported.value &&
!loading.value.execute &&
Boolean(previewToken.value) &&
!previewStale.value &&
(!requiredConfirmationText.value || form.value.execute_confirmation_text.trim() === requiredConfirmationText.value)
);
const executionSteps = computed(() =>
Array.isArray(activeMigrationJob.value?.result?.execution_steps)
? activeMigrationJob.value.result.execution_steps
: (Array.isArray(executeResult.value?.data?.execution_steps) ? executeResult.value?.data.execution_steps : [])
);
const migrationJobEventStream = computed(() => [...migrationJobEvents.value].reverse());
const migrationJobIsTerminal = computed(() => {
const status = String(activeMigrationJob.value?.status || "").trim();
return ["success", "failed", "blocked", "cancelled", "completed_with_issues", "partially_succeeded"].includes(status);
});
const boolLabel = (value: unknown) => (value ? "是" : "否");
const formatJsonBlock = (value: unknown) => {
if (value === null || typeof value === "undefined" || value === "") {
return "";
}
if (typeof value === "string") {
return value;
}
try {
return JSON.stringify(value, null, 2);
} catch {
return String(value);
}
};
const jobStatusType = (status: string) => {
if (status === "success") return "success";
if (status === "failed" || status === "blocked" || status === "cancelled") return "danger";
if (status === "running" || status === "dispatching" || status === "awaiting_approval") return "warning";
return "info";
};
const jobStatusLabel = (status: string) => {
if (status === "success") return "成功";
if (status === "failed") return "失败";
if (status === "blocked") return "阻断";
if (status === "cancelled") return "已取消";
if (status === "running") return "执行中";
if (status === "dispatching") return "派发中";
if (status === "awaiting_approval") return "待审批";
if (status === "queued") return "排队中";
if (status === "completed_with_issues") return "带问题完成";
if (status === "partially_succeeded") return "部分成功";
return status || "未知";
};
const eventLevelType = (level: string) => {
if (level === "success") return "success";
if (level === "warning") return "warning";
if (level === "error" || level === "critical") return "danger";
return "info";
};
const migrationEventDetailText = (row: GenericRecord) => {
const payload = row?.payload || {};
const chunks: string[] = [];
if (payload?.step_key || payload?.title) {
chunks.push(`step: ${payload.step_key || "-"} / ${payload.title || "-"}`);
}
if (payload?.stdout) {
chunks.push(`stdout:\n${String(payload.stdout).trim()}`);
}
if (payload?.stderr) {
chunks.push(`stderr:\n${String(payload.stderr).trim()}`);
}
if (!payload?.stdout && !payload?.stderr) {
const compact = { ...payload };
delete compact.occurred_at;
delete compact.summary_text;
if (Object.keys(compact).length) {
chunks.push(formatJsonBlock(compact));
}
}
return chunks.filter(Boolean).join("\n\n").trim();
};
const clearMigrationApiCompatibilityWarning = () => {
backendCompatibility.value = {
migrationApiSupported: true,
message: ""
};
};
const markMigrationApiUnavailable = (message = defaultUnavailableMessage) => {
backendCompatibility.value = {
migrationApiSupported: false,
message
};
};
const extractMigrationErrorMessage = (error: any, fallback: string) => {
const responseStatus = Number(error?.response?.status || 0);
if (responseStatus === 404) {
return defaultUnavailableMessage;
}
const apiMessage = String(error?.response?.data?.message || error?.message || "").trim();
if (apiMessage) {
return apiMessage;
}
return fallback;
};
const buildBasePayload = () => ({
target_node_code: form.value.target_node_code,
target_repo_path: form.value.target_repo_path,
target_domain_root: form.value.target_domain_root,
target_api_root: form.value.target_api_root,
target_web_root: form.value.target_web_root,
sync_env_files: form.value.sync_env_files,
sync_systemd_units: form.value.sync_systemd_units,
build_frontend: form.value.build_frontend,
restart_services: form.value.restart_services,
overwrite_database: form.value.overwrite_database,
backup_target_database: form.value.backup_target_database,
target_db_host: form.value.target_db_host,
target_db_port: form.value.target_db_port,
target_db_name: form.value.target_db_name,
target_db_user: form.value.target_db_user,
target_db_password: form.value.target_db_password
});
const buildPayload = () => ({
...buildBasePayload(),
execute_confirmation_token: previewToken.value,
execute_confirmation_text: form.value.execute_confirmation_text.trim()
});
const stopMigrationJobPolling = () => {
if (migrationJobPollTimer) {
clearInterval(migrationJobPollTimer);
migrationJobPollTimer = null;
}
};
const refreshMigrationJobProgress = async (options: { silent?: boolean } = {}) => {
const normalizedJobId = Number(activeMigrationJob.value?.id || executeResult.value?.data?.job?.id || 0);
if (!normalizedJobId) {
return;
}
const { silent = false } = options;
loading.value.jobDetail = true;
loading.value.jobEvents = true;
try {
const [detailResponse, eventsResponse] = await Promise.all([
opsApi.jobDetail(normalizedJobId),
opsApi.jobEvents(normalizedJobId, { limit: 200 })
]);
activeMigrationJob.value = detailResponse.data || {};
migrationJobEvents.value = Array.isArray(eventsResponse.data?.events) ? eventsResponse.data.events : [];
migrationJobEventsSummary.value = eventsResponse.data?.summary || {};
if (migrationJobIsTerminal.value) {
stopMigrationJobPolling();
}
} catch (error: any) {
if (!silent) {
ElMessage.error(error?.message || "读取迁移任务日志失败");
}
} finally {
loading.value.jobDetail = false;
loading.value.jobEvents = false;
}
};
const startMigrationJobPolling = () => {
stopMigrationJobPolling();
void refreshMigrationJobProgress({ silent: true });
migrationJobPollTimer = setInterval(() => {
void refreshMigrationJobProgress({ silent: true });
}, 2000);
};
const loadNodes = async () => {
loading.value.nodes = true;
try {
const response = await opsApi.nodes();
const nodes = Array.isArray(response.data?.nodes) ? response.data.nodes : [];
targetNodes.value = nodes.filter((item: GenericRecord) => Boolean(item.ssh_host) && Boolean(item.ssh_user));
if (!form.value.target_node_code && targetNodes.value.length) {
form.value.target_node_code = String(targetNodes.value[0].node_code || "");
}
} finally {
loading.value.nodes = false;
}
};
const loadSourceProfile = async () => {
loading.value.source = true;
try {
const response = await opsApi.migrationSourceProfile();
sourceProfile.value = response.data || {};
clearMigrationApiCompatibilityWarning();
} catch (error: any) {
const message = extractMigrationErrorMessage(error, "读取迁移源信息失败");
sourceProfile.value = {};
if (Number(error?.response?.status || 0) === 404) {
markMigrationApiUnavailable(message);
ElMessage.warning(message);
return;
}
ElMessage.error(message);
} finally {
loading.value.source = false;
}
};
const previewMigration = async () => {
loading.value.preview = true;
executeResult.value = null;
activeMigrationJob.value = {};
migrationJobEvents.value = [];
migrationJobEventsSummary.value = {};
stopMigrationJobPolling();
form.value.execute_confirmation_text = "";
try {
previewResult.value = await opsApi.previewMigration(buildBasePayload());
lastPreviewPayloadSignature.value = basePayloadSignature.value;
clearMigrationApiCompatibilityWarning();
if (previewResult.value.code === 0) {
ElMessage.success("迁移计划已生成");
}
} catch (error: any) {
const message = extractMigrationErrorMessage(error, "预检查未通过");
previewResult.value = error?.code
? error
: { code: 1, message, data: { blocking_reasons: [message] } };
lastPreviewPayloadSignature.value = basePayloadSignature.value;
if (Number(error?.response?.status || 0) === 404) {
markMigrationApiUnavailable(message);
}
ElMessage.warning(message);
} finally {
loading.value.preview = false;
}
};
const executeMigration = async () => {
loading.value.execute = true;
try {
executeResult.value = await opsApi.executeMigration(buildPayload());
clearMigrationApiCompatibilityWarning();
if (executeResult.value.code === 0) {
activeMigrationJob.value = executeResult.value.data?.job || {};
migrationLogVisible.value = true;
startMigrationJobPolling();
ElMessage.success(executeResult.value.message || "迁移任务已提交");
}
} catch (error: any) {
const message = extractMigrationErrorMessage(error, "迁移执行失败");
executeResult.value = error?.code
? error
: { code: 1, message, data: { blocking_reasons: [message] } };
if (Number(error?.response?.status || 0) === 404) {
markMigrationApiUnavailable(message);
ElMessage.warning(message);
return;
}
ElMessage.error(message);
} finally {
loading.value.execute = false;
}
};
onMounted(async () => {
await Promise.all([loadNodes(), loadSourceProfile()]);
});
onBeforeUnmount(() => {
stopMigrationJobPolling();
});
</script>
<style scoped lang="scss">
.header-actions,
.action-row {
display: flex;
gap: 10px;
flex-wrap: wrap;
}
.section-grid,
.result-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
gap: 16px;
margin-bottom: 16px;
}
.inspection-summary-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(180px, 1fr));
gap: 16px;
}
.summary-card,
.mini-card {
padding: 16px;
border-radius: 16px;
background: linear-gradient(180deg, #ffffff 0%, #f8fafc 100%);
border: 1px solid #e2e8f0;
display: flex;
flex-direction: column;
gap: 6px;
}
.summary-label {
font-size: 12px;
color: #64748b;
text-transform: uppercase;
letter-spacing: 0.04em;
}
.summary-note {
color: #64748b;
font-size: 12px;
line-height: 1.5;
}
.migration-form {
margin-bottom: 18px;
}
.form-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
gap: 16px;
}
.toggle-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr));
gap: 14px;
margin-top: 6px;
}
.full-width {
grid-column: 1 / -1;
}
.result-section {
margin-top: 18px;
}
.result-header {
display: flex;
justify-content: space-between;
align-items: center;
gap: 12px;
margin-bottom: 12px;
}
.preview-box {
padding: 16px;
border-radius: 16px;
background: #f8fafc;
border: 1px solid #e2e8f0;
}
.preview-box__title {
font-weight: 600;
margin-bottom: 10px;
}
.preview-list {
display: flex;
flex-direction: column;
gap: 8px;
}
.preview-list__item {
padding: 10px 12px;
border-radius: 12px;
background: #ffffff;
border: 1px solid #e2e8f0;
color: #334155;
}
.preview-list__item--danger {
border-color: #fecaca;
background: #fff1f2;
color: #991b1b;
}
.preview-list__item--warning {
border-color: #fde68a;
background: #fffbeb;
color: #92400e;
}
.drawer-meta {
display: flex;
flex-wrap: wrap;
gap: 12px;
color: #64748b;
font-size: 13px;
margin-bottom: 12px;
}
.log-console {
display: flex;
flex-direction: column;
gap: 12px;
max-height: calc(100vh - 180px);
overflow: auto;
padding-right: 4px;
}
.log-entry {
padding: 14px;
border-radius: 14px;
border: 1px solid #dbe3ef;
background: #0f172a;
color: #e2e8f0;
}
.log-entry__meta {
display: flex;
align-items: center;
flex-wrap: wrap;
gap: 8px;
margin-bottom: 8px;
color: #94a3b8;
font-size: 12px;
}
.log-entry__message {
white-space: pre-wrap;
line-height: 1.6;
}
.log-entry__detail {
margin: 10px 0 0;
padding: 12px;
border-radius: 10px;
background: rgba(15, 23, 42, 0.72);
border: 1px solid rgba(148, 163, 184, 0.24);
color: #cbd5e1;
white-space: pre-wrap;
word-break: break-word;
font-size: 12px;
line-height: 1.55;
}
</style>

634
domain-web/src/views/runtime/RuntimeView.vue
View File

File diff suppressed because it is too large Load Diff

112
domain-web/src/views/settings/SettingsView.vue
View File

@@ -9,29 +9,49 @@
<PageCard title="运行配置">
<el-form label-position="top">
<el-alert
:title="currentNodeCode ? `支持默认并发和节点单独覆盖；当前后台节点编码：${currentNodeCode}` : '支持默认并发和节点单独覆盖：未单独配置的机器自动走默认线程数。'"
:title="currentNodeCode ? `支持默认进程/线程和服务器级覆盖；当前后台节点编码：${currentNodeCode}` : '支持默认进程/线程和服务器级覆盖：未单独配置的服务器自动走默认值。'"
type="info"
:closable="false"
show-icon
style="margin-bottom: 12px"
/>
<el-form-item label="默认 Worker 进程数">
<el-input-number v-model="processCount" :min="1" />
</el-form-item>
<el-form-item label="服务器独立进程覆盖">
<div class="node-thread-overrides">
<div v-if="!nodeProcessOverrides.length" class="node-thread-empty">
当前没有服务器级单独覆盖配置，所有服务器都会走默认进程数。
</div>
<div v-for="(item, index) in nodeProcessOverrides" :key="`${item.node_code}-process-${index}`" class="node-thread-row">
<el-input v-model="item.node_code" placeholder="服务器编码，例如 mainland-controller-01" />
<el-input-number v-model="item.process_count" :min="1" />
<el-button type="danger" plain @click="removeNodeProcessOverride(index)">删除</el-button>
</div>
<div class="node-thread-actions">
<el-button plain @click="addNodeProcessOverride">新增服务器覆盖</el-button>
</div>
</div>
</el-form-item>
<el-form-item label="默认检测线程数">
<el-input-number v-model="threadCount" :min="1" />
</el-form-item>
<el-form-item label="节点独立线程覆盖">
<el-form-item label="服务器独立线程覆盖">
<div class="node-thread-overrides">
<div v-if="!nodeThreadOverrides.length" class="node-thread-empty">
当前没有单独覆盖配置，所有节点都会走默认线程数。
当前没有服务器级单独覆盖配置，所有服务器都会走默认线程数。
</div>
<div v-for="(item, index) in nodeThreadOverrides" :key="`${item.node_code}-${index}`" class="node-thread-row">
<el-input v-model="item.node_code" placeholder="节点编码，例如 mainland-worker-01" />
<el-input v-model="item.node_code" placeholder="服务器编码，例如 mainland-worker-01" />
<el-input-number v-model="item.thread_count" :min="1" />
<el-button type="danger" plain @click="removeNodeThreadOverride(index)">删除</el-button>
</div>
<div class="node-thread-actions">
<el-button plain @click="addNodeThreadOverride">新增节点覆盖</el-button>
<el-button plain @click="addNodeThreadOverride">新增服务器覆盖</el-button>
</div>
</div>
</el-form-item>
@@ -71,7 +91,7 @@
type="info"
show-icon
style="margin-top: 12px"
title="关闭时不追加远端日志镜像；关键模式回传阶段/代理/异常等关键过程；全量模式会追加更多执行过程，便于临时分析。"
title="默认会按服务器级别统一下发：80 进程、每进程 1000 线程；这里只保留服务器级覆盖，不建议人工管理进程实例级并发。关闭时不追加远端日志镜像；关键模式回传阶段/代理/异常等关键过程；全量模式会追加更多执行过程，便于临时分析。"
/>
</el-form>
</PageCard>
@@ -223,6 +243,25 @@ type NodeThreadOverrideRow = {
thread_count: number;
};
type NodeProcessOverrideRow = {
node_code: string;
process_count: number;
};
const resolveServerCode = (rawNodeCode: string) => {
const normalized = String(rawNodeCode || "").trim();
if (!normalized) return "";
const parts = normalized.split("-");
if (parts.length >= 2) {
const suffix = parts[parts.length - 1];
const parent = parts.slice(0, -1).join("-");
if (/^[a-z]{1,3}$/i.test(suffix) && /\d$/.test(parent)) {
return parent;
}
}
return normalized;
};
const DETECT_LABELS: Record<string, string> = {
detect_register: "检查注册",
detect_baidu_site: "百度 site 查询",
@@ -235,9 +274,11 @@ const DETECT_LABELS: Record<string, string> = {
};
const loading = ref(true);
const threadCount = ref(2);
const processCount = ref(80);
const threadCount = ref(1000);
const currentNodeCode = ref("");
const nodeThreadOverrides = ref<NodeThreadOverrideRow[]>([]);
const nodeProcessOverrides = ref<NodeProcessOverrideRow[]>([]);
const detectItems = ref<DetectItem[]>([]);
const backups = ref<BackupRecord[]>([]);
const proxyConfig = ref<Record<string, any>>({
@@ -328,12 +369,32 @@ const moveItem = (index: number, offset: -1 | 1) => {
};
const normalizeNodeThreadOverrides = (payload: Record<string, number | string>) => {
nodeThreadOverrides.value = Object.entries(payload || {})
.map(([node_code, thread_count]) => ({
node_code: String(node_code || "").trim(),
thread_count: Number(thread_count || 0)
}))
.filter((item) => item.node_code && item.thread_count > 0)
const collapsed = Object.entries(payload || {}).reduce<Record<string, number>>((result, [rawNodeCode, threadCountValue]) => {
const node_code = resolveServerCode(String(rawNodeCode || "").trim());
const thread_count = Number(threadCountValue || 0);
if (!node_code || thread_count <= 0) {
return result;
}
result[node_code] = Math.max(Number(result[node_code] || 0), thread_count);
return result;
}, {});
nodeThreadOverrides.value = Object.entries(collapsed)
.map(([node_code, thread_count]) => ({ node_code, thread_count }))
.sort((a, b) => a.node_code.localeCompare(b.node_code));
};
const normalizeNodeProcessOverrides = (payload: Record<string, number | string>) => {
const collapsed = Object.entries(payload || {}).reduce<Record<string, number>>((result, [rawNodeCode, processCountValue]) => {
const node_code = resolveServerCode(String(rawNodeCode || "").trim());
const process_count = Number(processCountValue || 0);
if (!node_code || process_count <= 0) {
return result;
}
result[node_code] = Math.max(Number(result[node_code] || 0), process_count);
return result;
}, {});
nodeProcessOverrides.value = Object.entries(collapsed)
.map(([node_code, process_count]) => ({ node_code, process_count }))
.sort((a, b) => a.node_code.localeCompare(b.node_code));
};
@@ -348,10 +409,21 @@ const addNodeThreadOverride = () => {
});
};
const addNodeProcessOverride = () => {
nodeProcessOverrides.value.push({
node_code: "",
process_count: processCount.value || 80
});
};
const removeNodeThreadOverride = (index: number) => {
nodeThreadOverrides.value.splice(index, 1);
};
const removeNodeProcessOverride = (index: number) => {
nodeProcessOverrides.value.splice(index, 1);
};
const loadBackups = async () => {
try {
const response = await settingsApi.getSettingsBackups();
@@ -379,8 +451,10 @@ const loadSettings = async () => {
loading.value = true;
try {
const response = await settingsApi.getSettings();
processCount.value = response.data.process_count || 80;
threadCount.value = response.data.thread_count;
currentNodeCode.value = response.data.current_node_code || "";
normalizeNodeProcessOverrides(response.data.node_process_counts || {});
normalizeNodeThreadOverrides(response.data.node_thread_counts || {});
normalizeDetectItems(response.data.detect_options || {});
@@ -430,7 +504,19 @@ const buildSettingsPayload = () => {
return result;
}, {});
const nodeProcessCounts = nodeProcessOverrides.value.reduce<Record<string, number>>((result, item) => {
const nodeCode = String(item.node_code || "").trim();
const processCountValue = Number(item.process_count || 0);
if (!nodeCode || processCountValue <= 0) {
return result;
}
result[nodeCode] = processCountValue;
return result;
}, {});
return {
process_count: processCount.value,
node_process_counts: nodeProcessCounts,
thread_count: threadCount.value,
node_thread_counts: nodeThreadCounts,
detect_options: detectOptions,

135
domainCheck/app/core/detect_engine.py
View File

@@ -11,6 +11,7 @@
import time
from loguru import logger
from app.utils.database import Database
from app.utils.detection_results import normalize_detector_result
from app.detectors.rdap_detector import RDAPDetector
from app.detectors.wayback_detector import WaybackDetector
from app.detectors.baidu_detector import BaiduDetector
@@ -32,6 +33,10 @@ class DetectEngine:
"""
检测引擎
"""
OUTCOME_SUCCESS = "success"
OUTCOME_BLACKLISTED = "blacklisted"
OUTCOME_FAILED = "failed"
def __init__(self):
"""
@@ -49,6 +54,9 @@ class DetectEngine:
self.jucha_detector = JuchaDetector()
def detect_domain(self, domain_id):
return self._detect_domain_with_outcome(domain_id) == self.OUTCOME_SUCCESS
def _detect_domain_with_outcome(self, domain_id):
"""
检测域名
@@ -69,24 +77,30 @@ class DetectEngine:
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_RUNNING)
# 1. 基础检测
if not self._basic_detect(domain_id, domain):
basic_outcome = self._basic_detect(domain_id, domain)
if basic_outcome != self.OUTCOME_SUCCESS:
logger.info(f"基础检测失败，停止后续检测: {domain}")
return False
if basic_outcome == self.OUTCOME_FAILED:
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_FAILED)
return basic_outcome
# 2. 深度检测
if not self._deep_detect(domain_id, domain):
deep_outcome = self._deep_detect(domain_id, domain)
if deep_outcome != self.OUTCOME_SUCCESS:
logger.info(f"深度检测失败: {domain}")
return False
if deep_outcome == self.OUTCOME_FAILED:
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_FAILED)
return deep_outcome
# 更新检测状态为正常
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_COMPLETED)
logger.info(f"域名检测完成: {domain}")
return True
return self.OUTCOME_SUCCESS
except Exception as e:
logger.error(f"检测域名出错: {e}")
# 更新检测状态为检测失败
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_FAILED)
return False
return self.OUTCOME_FAILED
def _basic_detect(self, domain_id, domain):
"""
@@ -94,7 +108,7 @@ class DetectEngine:
:param domain_id: 域名ID
:param domain: 域名
:return: bool - 是否检测通过
:return: str - 检测结果
"""
# 1. 检查是否为一口价域名
is_ykj = self.db.is_ykj_domain(domain_id)
@@ -108,7 +122,7 @@ class DetectEngine:
if self.db.is_blacklisted(domain):
logger.info(f"域名在黑名单中: {domain}")
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
return False
return self.OUTCOME_BLACKLISTED
# 4. 时光机快照年份采集
snapshot_years = self.wayback_detector.get_snapshot_years(domain)
@@ -120,9 +134,9 @@ class DetectEngine:
logger.info(f"域名包含敏感词: {domain}")
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
self.db.add_to_blacklist(domain, "快照包含敏感词")
return False
return self.OUTCOME_BLACKLISTED
return True
return self.OUTCOME_SUCCESS
def _deep_detect(self, domain_id, domain):
"""
@@ -130,38 +144,52 @@ class DetectEngine:
:param domain_id: 域名ID
:param domain: 域名
:return: bool - 是否检测通过
:return: str - 检测结果
"""
# 1. 百度历史/Site
baidu_history = self.baidu_detector.check_history(domain)
baidu_site = self.baidu_detector.check_site(domain)
# 2. 360 Site
qihu360_site = self.qihu360_detector.check_site(domain)
# 3. Google Site
google_site = self.google_detector.check_site(domain)
# 4. 站长之家
chinaz_info = self.chinaz_detector.check_domain(domain)
# 5. 爱站网
aizhan_info = self.aizhan_detector.check_domain(domain)
# 6. 桔子SEO
juziseo_info = self.juziseo_detector.check_domain(domain)
# 7. 聚查
jucha_info = self.jucha_detector.check_domain(domain)
# 检查是否有风险
if self._check_risk(domain_id, domain, baidu_history, baidu_site, qihu360_site, google_site, chinaz_info, aizhan_info, juziseo_info, jucha_info):
return False
detector_results = {}
detector_steps = [
("baidu_history", lambda: self.baidu_detector.check_history(domain)),
("baidu_site", lambda: self.baidu_detector.check_site(domain)),
("qihu360_site", lambda: self.qihu360_detector.check_site(domain)),
("google_site", lambda: self.google_detector.check_site(domain)),
("chinaz_info", lambda: self.chinaz_detector.check_domain(domain)),
("aizhan_info", lambda: self.aizhan_detector.check_domain(domain)),
("juziseo_info", lambda: self.juziseo_detector.check_domain(domain)),
("jucha_info", lambda: self.jucha_detector.check_domain(domain)),
]
for detector_name, runner in detector_steps:
detector_results[detector_name] = normalize_detector_result(detector_name, runner())
detector_error = detector_results[detector_name].get("error")
if detector_error:
logger.error(
f"深度检测存在第三方检测错误: {domain}, detector={detector_name}, error={detector_error}"
)
return self.OUTCOME_FAILED
if self._check_risk(
domain_id,
domain,
detector_results.get("baidu_history"),
detector_results.get("baidu_site"),
detector_results.get("qihu360_site"),
detector_results.get("google_site"),
detector_results.get("chinaz_info"),
detector_results.get("aizhan_info"),
detector_results.get("juziseo_info"),
detector_results.get("jucha_info"),
):
self._persist_detection_results(domain_id, detector_results)
return self.OUTCOME_BLACKLISTED
# 保存检测结果
self.db.add_detection_result(domain_id, baidu_history, baidu_site, qihu360_site, google_site, chinaz_info, aizhan_info, juziseo_info, jucha_info)
persisted = self._persist_detection_results(domain_id, detector_results)
if not persisted:
logger.error(f"保存检测结果失败: {domain}")
return self.OUTCOME_FAILED
return True
return self.OUTCOME_SUCCESS
def _check_risk(self, domain_id, domain, baidu_history, baidu_site, qihu360_site, google_site, chinaz_info, aizhan_info, juziseo_info, jucha_info):
"""
@@ -180,7 +208,9 @@ class DetectEngine:
:return: bool - 是否有风险
"""
# 检查百度历史过灰
if baidu_history and '灰' in str(baidu_history):
if (
isinstance(baidu_history, dict) and baidu_history.get('has_gray')
) or (baidu_history and '灰' in str(baidu_history)):
logger.info(f"百度历史过灰: {domain}")
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
self.db.add_to_blacklist(domain, "百度历史过灰")
@@ -215,7 +245,7 @@ class DetectEngine:
# 检查WHOIS状态
if jucha_info and 'whois' in jucha_info:
if jucha_info['whois'].get('status') in ['clientHold', 'serverHold']:
if jucha_info['whois'].get('whois_status') in ['clientHold', 'serverHold']:
logger.info(f"WHOIS状态异常: {domain}")
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
self.db.add_to_blacklist(domain, "WHOIS状态异常")
@@ -225,11 +255,24 @@ class DetectEngine:
if jucha_info and 'intercept' in jucha_info:
if not jucha_info['intercept'].get('normal', True):
logger.info(f"拦截检测异常: {domain}")
self.db.update_domain_detect_status(domain_id, 4) # 4 表示黑名单
self.db.update_domain_detect_status(domain_id, DETECT_STATUS_BLACKLISTED)
self.db.add_to_blacklist(domain, "拦截检测异常")
return True
return False
def _persist_detection_results(self, domain_id, detector_results):
return self.db.add_detection_result(
domain_id,
detector_results.get("baidu_history"),
detector_results.get("baidu_site"),
detector_results.get("qihu360_site"),
detector_results.get("google_site"),
detector_results.get("chinaz_info"),
detector_results.get("aizhan_info"),
detector_results.get("juziseo_info"),
detector_results.get("jucha_info"),
)
def _contains_sensitive_words(self, text):
"""
@@ -264,11 +307,12 @@ class DetectEngine:
self.db.update_task_status(task_id, 1) # 1 表示执行中
# 执行检测
success = self.detect_domain(domain_id)
outcome = self._detect_domain_with_outcome(domain_id)
# 更新任务状态
if success:
if outcome in (self.OUTCOME_SUCCESS, self.OUTCOME_BLACKLISTED):
self.db.update_task_status(task_id, 2) # 2 表示完成
return True
else:
# 增加重试次数
retry_count = task.get('retry_count', 0) + 1
@@ -277,8 +321,7 @@ class DetectEngine:
self.db.update_task_status(task_id, 0) # 0 表示待执行
else:
self.db.update_task_status(task_id, 3) # 3 表示失败
return success
return False
except Exception as e:
logger.error(f"处理任务出错: {e}")
# 更新任务状态为失败

4
domainCheck/app/detectors/aizhan_detector.py
View File

@@ -64,7 +64,7 @@ class AizhanDetector(BaseDetector):
}
else:
self._log_warning(f"爱站网查询失败: {response.status_code}")
return {'title': '', 'risk': '', 'has_sensitive': False}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
return self._handle_exception(e, domain)
@@ -127,4 +127,4 @@ class AizhanDetector(BaseDetector):
if word in title:
return True
return False
return False

12
domainCheck/app/detectors/baidu_detector.py
View File

@@ -83,10 +83,9 @@ class BaiduDetector(BaseDetector):
}
else:
self._log_warning(f"百度site查询失败: {response.status_code}")
return {'has_收录': False, 'subdomains': []}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'has_收录': False, 'subdomains': []}
return self._handle_exception(e, domain)
def check_history(self, domain):
"""
@@ -123,10 +122,9 @@ class BaiduDetector(BaseDetector):
}
else:
self._log_warning(f"百度历史查询失败: {response.status_code}")
return {'has_history': False, 'has_gray': False}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'has_history': False, 'has_gray': False}
return self._handle_exception(e, domain)
def _extract_subdomains(self, content, domain):
"""
@@ -148,4 +146,4 @@ class BaiduDetector(BaseDetector):
return subdomains
except Exception as e:
self._handle_exception(e, domain)
return []
return []

4
domainCheck/app/detectors/chinaz_detector.py
View File

@@ -64,7 +64,7 @@ class ChinazDetector(BaseDetector):
}
else:
self._log_warning(f"站长之家查询失败: {response.status_code}")
return {'title': '', 'category': '', 'has_sensitive': False}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
return self._handle_exception(e, domain)
@@ -127,4 +127,4 @@ class ChinazDetector(BaseDetector):
if word in title:
return True
return False
return False

5
domainCheck/app/detectors/google_detector.py
View File

@@ -74,7 +74,6 @@ class GoogleDetector(BaseDetector):
}
else:
self._log_warning(f"Google site查询失败: {response.status_code}")
return {'has_收录': False}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'has_收录': False}
return self._handle_exception(e, domain)

17
domainCheck/app/detectors/jucha_detector.py
View File

@@ -81,10 +81,9 @@ class JuchaDetector(BaseDetector):
return whois_info
else:
self._log_warning(f"聚查WHOIS查询失败: {response.status_code}")
return {'status': ''}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'status': ''}
return self._handle_exception(e, domain)
def check_beian(self, domain):
"""
@@ -113,10 +112,9 @@ class JuchaDetector(BaseDetector):
return beian_info
else:
self._log_warning(f"聚查备案查询失败: {response.status_code}")
return {'has_beian': False, 'beian_year': '', 'is_enterprise': False, 'beian_match': False}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'has_beian': False, 'beian_year': '', 'is_enterprise': False, 'beian_match': False}
return self._handle_exception(e, domain)
def check_intercept(self, domain):
"""
@@ -147,10 +145,9 @@ class JuchaDetector(BaseDetector):
}
else:
self._log_warning(f"聚查拦截查询失败: {response.status_code}")
return {'normal': False}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'normal': False}
return self._handle_exception(e, domain)
def _extract_whois_info(self, content):
"""
@@ -225,4 +222,4 @@ class JuchaDetector(BaseDetector):
return False
except Exception as e:
self._handle_exception(e, 'check_intercept_status')
return False
return False

12
domainCheck/app/detectors/juziseo_detector.py
View File

@@ -93,10 +93,9 @@ class JuziseoDetector(BaseDetector):
}
else:
self._log_warning(f"桔子SEO历史查询失败: {response.status_code}")
return {'has_sensitive': False, 'has_baidu_history': False, 'has_subdomains': False, 'is_simplified': True}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'has_sensitive': False, 'has_baidu_history': False, 'has_subdomains': False, 'is_simplified': True}
return self._handle_exception(e, domain)
def check_backlink(self, domain):
"""
@@ -131,10 +130,9 @@ class JuziseoDetector(BaseDetector):
}
else:
self._log_warning(f"桔子SEO外链查询失败: {response.status_code}")
return {'has_sensitive': False, 'has_subdomains': False}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'has_sensitive': False, 'has_subdomains': False}
return self._handle_exception(e, domain)
def _extract_history_info(self, content):
"""
@@ -211,4 +209,4 @@ class JuziseoDetector(BaseDetector):
if word in content:
return True
return False
return False

7
domainCheck/app/detectors/qihu360_detector.py
View File

@@ -79,10 +79,9 @@ class Qihu360Detector(BaseDetector):
}
else:
self._log_warning(f"360 site查询失败: {response.status_code}")
return {'has_收录': False, 'subdomains': []}
return {'error': f'HTTP {response.status_code}'}
except Exception as e:
self._handle_exception(e, domain)
return {'has_收录': False, 'subdomains': []}
return self._handle_exception(e, domain)
def _extract_subdomains(self, content, domain):
"""
@@ -104,4 +103,4 @@ class Qihu360Detector(BaseDetector):
return subdomains
except Exception as e:
self._handle_exception(e, domain)
return []
return []

4
domainCheck/app/detectors/rdap_detector.py
View File

@@ -73,7 +73,7 @@ class RDAPDetector(BaseDetector):
return 2 # 可注册
# 检查域名状态
statuses = result.get('status', [])
statuses = result.get('statuses', [])
if 'clientHold' in statuses:
return 7 # clientHold
elif 'serverHold' in statuses:
@@ -125,4 +125,4 @@ class RDAPDetector(BaseDetector):
elif event_action == 'last update':
result['last_update'] = event_date
return result
return result

142
domainCheck/app/detectors/wayback_detector.py
View File

@@ -17,13 +17,13 @@ import zlib
from base64 import b64decode, b64encode
from concurrent.futures import FIRST_COMPLETED, ThreadPoolExecutor, wait
import redis
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from app.config import config
from app.detectors.base import BaseDetector
from app.utils.redis_client import get_redis_client
class WaybackDetector(BaseDetector):
@@ -51,6 +51,7 @@ class WaybackDetector(BaseDetector):
def _build_session(self):
session = requests.Session()
session.trust_env = False
retry = Retry(
total=max(0, config.WAYBACK_RETRY_COUNT),
backoff_factor=0.5,
@@ -106,15 +107,7 @@ class WaybackDetector(BaseDetector):
def _build_redis_client(self):
try:
client = redis.Redis(
host=config.REDIS_HOST,
port=config.REDIS_PORT,
password=config.REDIS_PASSWORD,
db=config.REDIS_DB,
decode_responses=True,
socket_connect_timeout=5,
socket_timeout=5,
)
client = get_redis_client(role="standard")
client.ping()
return client
except Exception:
@@ -489,6 +482,14 @@ class WaybackDetector(BaseDetector):
trimmed.append(item)
return trimmed
def _resolve_scan_record_fetch_limit(self):
max_records = max(1, int(getattr(config, "WAYBACK_MAX_RECORDS", 8) or 8))
# 扫描阶段最终只会保留最近的少量快照；如果每次都把整个 CDX 历史拉下来，
# 在快照特别多的域名上会白白浪费很多秒。这里改成“最近一小窗”，同时留出
# 重复 digest/标题的缓冲空间，避免把最新几条里重复记录全部裁没了。
recent_window = max(12, max_records * 6)
return -recent_window
def scan_snapshots(self, domain, sensitive_words=None, stop_on_first_hit=True, recent_years=None):
sensitive_words = sensitive_words or config.load_sensitive_words()
request_errors = []
@@ -512,12 +513,17 @@ class WaybackDetector(BaseDetector):
'request_errors': [f"wayback_backoff_active:{round(transient_backoff_remaining, 2)}s"],
'elapsed_seconds': 0.0,
}
transient_request_failures = 0
transient_request_failure_threshold = 2
latest_fetch = self._fetch_cdx_records_with_meta(domain, limit=-1, fast_latest=True)
latest_record = (latest_fetch.get('records') or [None])[0]
latest_fetch_transient_failure = False
if latest_fetch.get('error'):
request_errors.append(f"latest_cdx: {latest_fetch.get('error')}")
if self._is_transient_request_error(latest_fetch.get('error')):
self._trip_transient_backoff()
latest_fetch_transient_failure = True
transient_request_failures += 1
latest_timestamp = (latest_record or {}).get('timestamp')
latest_digest = (latest_record or {}).get('digest', '')
cutoff_year = self._resolve_recent_year_cutoff(recent_years)
@@ -547,26 +553,6 @@ class WaybackDetector(BaseDetector):
transient_snapshot_failures = 0
transient_snapshot_failure_threshold = max(2, domain_concurrency)
if (not latest_timestamp) and latest_fetch.get('error') and self._is_transient_request_error(latest_fetch.get('error')):
return {
'snapshot_years': [],
'has_sensitive_content': False,
'matched_word': None,
'matched_timestamp': None,
'matched_title': None,
'backlink_count': 0,
'backlink_count_gt_10': False,
'checked_snapshot_count': 0,
'fetched_snapshot_count': 0,
'failed_snapshot_count': max(1, failed_snapshot_count),
'unique_title_count': 0,
'duplicate_title_skipped': 0,
'digest_duplicate_skipped': 0,
'request_error_count': len(request_errors),
'request_errors': request_errors,
'elapsed_seconds': round(time.time() - started_at, 2),
}
if latest_timestamp:
latest_result = self._fetch_snapshot_title(domain, latest_timestamp)
checked_snapshot_count = 1
@@ -605,40 +591,55 @@ class WaybackDetector(BaseDetector):
if latest_error:
request_errors.append(f"latest_snapshot: {latest_error}")
if latest_error and self._is_transient_request_error(latest_error):
transient_snapshot_failures += 1
self._trip_transient_backoff()
return {
'snapshot_years': [],
'has_sensitive_content': False,
'matched_word': None,
'matched_timestamp': None,
'matched_title': None,
'backlink_count': 0,
'backlink_count_gt_10': False,
'checked_snapshot_count': checked_snapshot_count,
'fetched_snapshot_count': fetched_snapshot_count,
'failed_snapshot_count': failed_snapshot_count,
'unique_title_count': unique_title_count,
'duplicate_title_skipped': duplicate_title_skipped,
'digest_duplicate_skipped': digest_duplicate_skipped,
'request_error_count': len(request_errors),
'request_errors': request_errors,
'elapsed_seconds': round(time.time() - started_at, 2),
}
transient_request_failures += 1
cached_records = self._load_cached_records(domain)
if cached_records is not None:
records = cached_records
else:
records_fetch = self._fetch_cdx_records_with_meta(domain)
if records_fetch.get('error'):
request_errors.append(f"records_cdx: {records_fetch.get('error')}")
if self._is_transient_request_error(records_fetch.get('error')):
self._trip_transient_backoff()
records = records_fetch.get('records') or []
if records:
self._save_cached_records(domain, records)
self._save_cached_timestamps(domain, [item['timestamp'] for item in records])
if latest_fetch_transient_failure and not latest_timestamp:
records = []
else:
records_fetch = self._fetch_cdx_records_with_meta(
domain,
limit=self._resolve_scan_record_fetch_limit(),
)
if records_fetch.get('error'):
request_errors.append(f"records_cdx: {records_fetch.get('error')}")
if self._is_transient_request_error(records_fetch.get('error')):
transient_request_failures += 1
records = records_fetch.get('records') or []
if records:
self._save_cached_records(domain, records)
self._save_cached_timestamps(domain, [item['timestamp'] for item in records])
if (
not latest_timestamp
and not records
and (
transient_request_failures >= transient_request_failure_threshold
or latest_fetch_transient_failure
)
):
self._trip_transient_backoff()
return {
'snapshot_years': [],
'has_sensitive_content': False,
'matched_word': None,
'matched_timestamp': None,
'matched_title': None,
'backlink_count': 0,
'backlink_count_gt_10': False,
'checked_snapshot_count': checked_snapshot_count,
'fetched_snapshot_count': fetched_snapshot_count,
'failed_snapshot_count': max(1, failed_snapshot_count),
'unique_title_count': 0,
'duplicate_title_skipped': duplicate_title_skipped,
'digest_duplicate_skipped': digest_duplicate_skipped,
'request_error_count': len(request_errors),
'request_errors': request_errors,
'elapsed_seconds': round(time.time() - started_at, 2),
}
records = self._filter_records_recent_years(records, recent_years=recent_years)
records = sorted(records, key=lambda item: item.get('timestamp', ''), reverse=True)
@@ -660,12 +661,12 @@ class WaybackDetector(BaseDetector):
digest_seen.add(digest)
pending_records.append(item)
with ThreadPoolExecutor(max_workers=domain_concurrency) as executor:
pending = {}
index = 0
finished_count = 1 if latest_timestamp else 0
stop_requested = False
executor = ThreadPoolExecutor(max_workers=domain_concurrency)
pending = {}
index = 0
finished_count = 1 if latest_timestamp else 0
stop_requested = False
try:
while (index < len(pending_records) or pending) and not stop_requested:
while index < len(pending_records) and len(pending) < domain_concurrency and not stop_requested:
timestamp = pending_records[index]['timestamp']
@@ -694,8 +695,8 @@ class WaybackDetector(BaseDetector):
request_errors.append(f"snapshot:{timestamp}: {error_message}")
if error_message and self._is_transient_request_error(error_message):
transient_snapshot_failures += 1
self._trip_transient_backoff()
if transient_snapshot_failures >= transient_snapshot_failure_threshold:
self._trip_transient_backoff()
stop_requested = True
continue
@@ -722,10 +723,13 @@ class WaybackDetector(BaseDetector):
)
if config.WAYBACK_REQUEST_DELAY > 0:
time.sleep(config.WAYBACK_REQUEST_DELAY)
finally:
if stop_requested:
for future in pending:
for future in list(pending.keys()):
future.cancel()
executor.shutdown(wait=False, cancel_futures=True)
else:
executor.shutdown(wait=True)
return {
'snapshot_years': years,

75
domainCheck/app/ui/domain_filter.py
View File

@@ -8,6 +8,8 @@
@explain : 域名筛选界面
'''
import json
from PySide6.QtWidgets import QWidget, QVBoxLayout, QHBoxLayout, QGridLayout, QPushButton, QLabel, QComboBox, QDateEdit, QCheckBox, QTableWidget, QTableWidgetItem, QHeaderView, QFileDialog, QLineEdit, QSpinBox, QInputDialog
from PySide6.QtGui import QIntValidator
from PySide6.QtCore import Qt, QDate, QThread, Signal
@@ -15,6 +17,11 @@ from loguru import logger
from app.core.export_manager import ExportManager
from app.utils.database import Database
from app.utils.detection_results import (
build_manual_detection_result,
load_detection_result,
resolve_detection_status,
)
from app.utils.status_codes import (
DETECT_STATUS_BLACKLISTED,
DETECT_STATUS_COMPLETED,
@@ -205,9 +212,9 @@ class UpdateThread(QThread):
baidu_history_value = self.update_values['baidu_history']
if baidu_history_value != '不更新':
status_value = status_mappings['百度历史收录状态'][baidu_history_value]
# 转换为JSON格式
import json
json_value = json.dumps({"status": status_value})
json_value = json.dumps(
build_manual_detection_result(status_value, legacy_key='has_history')
)
if detection_id:
cur.execute("UPDATE domain_detections SET baidu_history = %s WHERE domain_id = %s", (json_value, domain_info['id']))
else:
@@ -219,9 +226,9 @@ class UpdateThread(QThread):
baidu_site_value = self.update_values['baidu_site']
if baidu_site_value != '不更新':
status_value = status_mappings['百度site收录状态'][baidu_site_value]
# 转换为JSON格式
import json
json_value = json.dumps({"status": status_value})
json_value = json.dumps(
build_manual_detection_result(status_value, legacy_key='has_收录')
)
if detection_id:
cur.execute("UPDATE domain_detections SET baidu_site = %s WHERE domain_id = %s", (json_value, domain_info['id']))
else:
@@ -244,9 +251,9 @@ class UpdateThread(QThread):
qihu360_site_value = self.update_values['qihu360_site']
if qihu360_site_value != '不更新':
status_value = status_mappings['360 site收录状态'][qihu360_site_value]
# 转换为JSON格式
import json
json_value = json.dumps({"status": status_value})
json_value = json.dumps(
build_manual_detection_result(status_value, legacy_key='has_收录')
)
if detection_id:
cur.execute("UPDATE domain_detections SET qihu360_site = %s WHERE domain_id = %s", (json_value, domain_info['id']))
else:
@@ -258,9 +265,9 @@ class UpdateThread(QThread):
google_site_value = self.update_values['google_site']
if google_site_value != '不更新':
status_value = status_mappings['Google site收录状态'][google_site_value]
# 转换为JSON格式
import json
json_value = json.dumps({"status": status_value})
json_value = json.dumps(
build_manual_detection_result(status_value, legacy_key='has_收录')
)
if detection_id:
cur.execute("UPDATE domain_detections SET google_site = %s WHERE domain_id = %s", (json_value, domain_info['id']))
else:
@@ -1082,28 +1089,14 @@ class DomainFilterWidget(QWidget):
# 百度历史收录状态
baidu_history = domain.get('baidu_history')
if baidu_history is None:
baidu_history = {}
elif isinstance(baidu_history, str):
import json
try:
baidu_history = json.loads(baidu_history)
except:
baidu_history = {}
baidu_history_status = '是' if baidu_history.get('status') else '否'
baidu_history = load_detection_result(baidu_history)
baidu_history_status = '是' if resolve_detection_status(baidu_history, 'has_history') else '否'
self.table_widget.setItem(row, 12, QTableWidgetItem(baidu_history_status))
# 百度site收录状态
baidu_site = domain.get('baidu_site')
if baidu_site is None:
baidu_site = {}
elif isinstance(baidu_site, str):
import json
try:
baidu_site = json.loads(baidu_site)
except:
baidu_site = {}
baidu_site_status = '是' if baidu_site.get('status') else '否'
baidu_site = load_detection_result(baidu_site)
baidu_site_status = '是' if resolve_detection_status(baidu_site, 'has_收录') else '否'
self.table_widget.setItem(row, 13, QTableWidgetItem(baidu_site_status))
# title是否有中文
@@ -1113,28 +1106,14 @@ class DomainFilterWidget(QWidget):
# 360site收录
qihu360_site = domain.get('qihu360_site')
if qihu360_site is None:
qihu360_site = {}
elif isinstance(qihu360_site, str):
import json
try:
qihu360_site = json.loads(qihu360_site)
except:
qihu360_site = {}
qihu360_site_status = '是' if qihu360_site.get('status') else '否'
qihu360_site = load_detection_result(qihu360_site)
qihu360_site_status = '是' if resolve_detection_status(qihu360_site, 'has_收录') else '否'
self.table_widget.setItem(row, 15, QTableWidgetItem(qihu360_site_status))
# Google site收录状态
google_site = domain.get('google_site')
if google_site is None:
google_site = {}
elif isinstance(google_site, str):
import json
try:
google_site = json.loads(google_site)
except:
google_site = {}
google_site_status = '是' if google_site.get('status') else '否'
google_site = load_detection_result(google_site)
google_site_status = '是' if resolve_detection_status(google_site, 'has_收录') else '否'
self.table_widget.setItem(row, 16, QTableWidgetItem(google_site_status))
# 友情链接数量

1913
domainCheck/app/utils/database.py
View File

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More