diff --git a/.gitignore b/.gitignore index 3fbf09c..83820c9 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,7 @@ dist/ *.log logs/ runtime/ +diagnostics/ # Build and release artifacts release/ @@ -38,6 +39,10 @@ tmp/ domains.txt credentials.json +# Local deploy config copies +domain-api/deploy/multi-region/*.conf +!domain-api/deploy/multi-region/*.conf.example + # domainCheck local runtime / bundled tools domainCheck/tools/node-v20.19.4-win-x64/ domainCheck/app/credentials.json diff --git a/docs/07_domainCheck_WebLinux发布验收清单.md b/docs/07_domainCheck_WebLinux发布验收清单.md index 5730fda..6f58ac9 100644 --- a/docs/07_domainCheck_WebLinux发布验收清单.md +++ b/docs/07_domainCheck_WebLinux发布验收清单.md @@ -1,90 +1,231 @@ # 07 domainCheck Web/Linux 发布验收清单 -## 一、基础连通 +## 一、文档定位 -- `domain-api` 已启动 -- `domainCheck Worker` 已启动 -- `http://服务器IP:8100/health` 返回 `status=ok` -- `runtime/preflight` 返回 `ok=true` +本文档用于回答两个问题： -## 二、Web 后台 +1. 当前这套 Web/Linux 方案，到底哪些项已经验过 +2. 发布前还需要按什么维度再核一遍 + +阅读建议： + +- 想看测试服真实联调结果，优先看 `docs/13_domainCheck_Linux测试服交接文档.md` +- 想看正式上线前最后一次操作顺序，优先看 `docs/14_domainCheck_正式上线前最终检查单.md` + +本文档更适合作为“发布验收维度总表”。 + +## 二、基础连通验收 + +以下项目应视为发布验收的第一层： + +- `domaincheck-api` 已启动 +- `domaincheck-worker` 已启动 +- `http://127.0.0.1:8100/health` 返回 `status=ok` +- `/health` 返回 `worker_mode=linux-systemd` +- `/api/v1/runtime/preflight` 返回 `ok=true` +- `/api/v1/runtime/status` 返回 `worker.running=true` + +当前测试服状态： + +- 已验证通过 + +## 三、Web 后台验收 + +### 1. 运行与展示 - 登录正常 -- 顶部可看到 `API / Worker / 运行模式` -- 概览页能正常读取统计 -- 运行中心能正常读取： +- 顶部状态可展示 `API / Worker / 运行模式` +- 概览页可正常读取统计 +- 运行中心可正常读取： - API 版本 - API 前缀 - PID - Worker 进程数 - 自检结果 -## 三、配置能力 +### 2. 配置能力 -- 系统设置能正常读取 -- 线程数修改后可保存 +- 系统设置可正常读取 +- 线程数可修改并保存 - 检测顺序可调整并保存 - 代理池列表可编辑并保存 -- `worker_mode / service_name` 可保存 +- `worker_mode / worker_service_name / api_service_name` 可保存 - 可手动创建配置备份 - 可下载配置备份 - 可导出配置快照 - 可导入配置快照 - 导入配置时会自动生成导入前备份 -## 四、业务能力 +当前测试服状态： + +- 核心接口与配置链路已验证通过 + +## 四、业务能力验收 + +发布前建议至少覆盖下面 5 类能力： + +- 导入 +- 列表筛选 +- 批量更新 +- 导出 +- 日志与诊断 + +### 1. 导入链路 - 导入任务可创建 - 导入任务列表可刷新 -- 导入任务失败时可重试 -- 域名筛选可查询 -- 批量更新可执行 -- 导出记录可生成 -- 导出文件可下载 -- 日志诊断页可查看日志 -- 诊断包可下载 +- 导入结果统计正确 +- 导入成功后会写入 `domains` +- 导入成功后会自动创建 `detect_tasks` -## 五、Linux 特有项 +当前测试服状态： + +- 已使用样本 TXT 实际验证通过 +- 当前已确认： + - 总数 `5` + - 有效 `3` + - 新增 `3` + - 无效 `2` + +### 2. 列表与筛选 + +- 域名列表可查询 +- 筛选项接口正常返回 +- 页面结果与数据库记录一致 + +当前测试服状态： + +- 已验证通过 + +### 3. 批量更新 + +- 批量更新接口可执行 +- 页面展示与数据库字段一致 +- 关联检测字段会同步更新 + +当前测试服状态： + +- 已验证通过 +- 已确认 `backlink_count_gt_10` 可随批量更新同步生效 + +### 4. 导出链路 + +- 导出任务可创建 +- 导出记录可查看 +- 导出文件可下载 +- TXT / CSV 至少一种格式已回归 + +当前测试服状态： + +- TXT、CSV 均已验证通过 + +### 5. 日志与诊断 + +- 日志接口可正常读取 +- 诊断包可正常导出 + +当前测试服状态： + +- 已验证通过 +- 当前已生成正式服务态诊断包： + - `/opt/domaincheck/diagnostics/diag_20260416_134923.tar.gz` + +## 五、Linux 特有项验收 + +以下项目是 Web/Linux 交付里最容易在正式环境出问题的部分： - `domaincheck-api.service` 可正常启动/停止 - `domaincheck-worker.service` 可正常启动/停止 - `journalctl` 可查看两边日志 -- Nginx 反代正常 -- Web 静态文件 `dist` 已正确发布 +- 运行中心可调用 `start_worker / stop_worker / restart_api` +- Worker 以 `QT_QPA_PLATFORM=offscreen` 正常运行 +- API 自重启不会再因为同步等待自身停机而误报 `500` -## 六、建议发布前命令 +当前测试服状态： -### 1. 运行 API 自测 +- 已验证通过 + +## 六、数据库与权限验收 + +Linux 新环境发布前，下面两项必须显式确认： + +### 1. 数据库初始化 + +如果 PostgreSQL 使用的是新库，必须先执行： ```bash -cd /opt/domaincheck/domain-api -python deploy/linux/smoke_test.py --base-url http://127.0.0.1:8100 +cd /opt/domaincheck/domainCheck +python3 init_database.py +``` + +否则至少这些接口会直接失败： + +- `/api/v1/dashboard/overview` +- `/api/v1/detect/status` +- `/api/v1/imports/summary` + +### 2. 服务用户写权限 + +正式 `systemd` 服务用户必须可写： + +- `domain-api/runtime/` +- `domainCheck/detect_worker.log` + +否则可能出现： + +- `/api/v1/imports/upload` 返回 `500` +- Worker 循环重启 + +当前测试服状态： + +- 两项都已实际踩坑并修复 + +## 七、发布前建议命令 + +### 1. 健康检查 + +```bash +curl http://127.0.0.1:8100/health +curl http://127.0.0.1:8100/api/v1/runtime/preflight +curl http://127.0.0.1:8100/api/v1/runtime/status +``` + +### 2. systemd 状态 + +```bash +systemctl status domaincheck-api --no-pager -l +systemctl status domaincheck-worker --no-pager -l +``` + +### 3. smoke test + +```bash +cd /opt/domaincheck/domain-api/deploy/linux +python3 smoke_test.py --base-url http://127.0.0.1:8100 ``` 如需同时校验 Web 首页： ```bash -python deploy/linux/smoke_test.py --base-url http://127.0.0.1:8100 --web-url http://127.0.0.1 +python3 smoke_test.py --base-url http://127.0.0.1:8100 --web-url http://127.0.0.1 ``` -### 2. 查看 API 健康状态 +### 4. 诊断包导出 ```bash -curl http://127.0.0.1:8100/health -curl http://127.0.0.1:8100/api/v1/runtime/preflight +cd /opt/domaincheck/domain-api/deploy/linux +bash collect_diagnostics.sh /opt/domaincheck ``` -### 3. 查看 systemd 状态 +## 八、当前发布验收结论 -```bash -systemctl status domaincheck-api -systemctl status domaincheck-worker -``` - -## 七、当前结论 - -如果以上检查项全部通过，则可以认为： +结合当前 Linux 测试服已经完成的联调结果，可以给出下面的结论： - Web 管理后台已达到可交付状态 -- Linux 部署环境已达到可联调状态 -- 可以进入真实服务器联调或灰度上线阶段 +- Linux 正式 `systemd` 服务态已验证通过 +- 核心业务链路已完成最小闭环回归 +- 当前剩余工作主要是正式环境发布、灰度观察和持续稳定性观察 + +一句话结论： + +> 当前项目已经通过 Web/Linux 发布所需的核心验收项，后续重点不再是功能开发，而是正式环境收口与上线后观察。 diff --git a/docs/10_domainCheck_最终交付结论.md b/docs/10_domainCheck_最终交付结论.md index b5731aa..0449323 100644 --- a/docs/10_domainCheck_最终交付结论.md +++ b/docs/10_domainCheck_最终交付结论.md @@ -55,13 +55,20 @@ - 上线前灰度发布 - 发布后观察期 +补充说明： + +- Linux 测试服基础联调已完成 +- 正式 `systemd` 服务态已验证通过 +- 当前进入的是“正式上线前最后检查与观察”阶段 + ## 五、建议下一步 1. 把最新交付包发送到目标 Linux 服务器 -2. 按 `docs/05`、`docs/07`、`docs/08` 的顺序执行部署与验收 -3. 部署完成后再做一次真实环境 smoke test -4. 进入灰度上线 +2. 按 `docs/05`、`docs/13`、`docs/14` 的顺序执行部署、核对与收口 +3. 部署完成后再做一次正式服务态 `smoke test` +4. 导出一份最终诊断包 +5. 进入灰度上线 ## 六、最终判断 -当前这套项目，已经达到“工程交付完成，待真实环境上线联调”的状态。 +当前这套项目，已经达到“工程交付完成，Linux 测试服联调闭环，待正式环境上线收口”的状态。 diff --git a/docs/12_domainCheck_导航索引.md b/docs/12_domainCheck_导航索引.md index f598b3c..9b265c8 100644 --- a/docs/12_domainCheck_导航索引.md +++ b/docs/12_domainCheck_导航索引.md @@ -20,6 +20,22 @@ - `docs/13_domainCheck_Linux测试服交接文档.md` +### 5. 正式上线前最终检查单 + +- `docs/14_domainCheck_正式上线前最终检查单.md` + +### 6. 多机检测与跨地域部署设计 + +- `docs/16_domainCheck_多机检测与跨地域部署设计.md` + +### 7. 全流程部署实操手册 + +- `docs/17_domainCheck_全流程部署实操手册.md` + +### 8. CentOS9 一键复制部署与更新文档 + +- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md` + ## 二、文档阅读顺序 ### 1. 先看总体方案 @@ -33,6 +49,10 @@ - `docs/07_domainCheck_WebLinux发布验收清单.md` - `docs/09_domainCheck_交付打包说明.md` - `docs/13_domainCheck_Linux测试服交接文档.md` +- `docs/14_domainCheck_正式上线前最终检查单.md` +- `docs/16_domainCheck_多机检测与跨地域部署设计.md` +- `docs/17_domainCheck_全流程部署实操手册.md` +- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md` ### 3. 如果要回溯历史需求与问题 @@ -63,6 +83,9 @@ - `domain-api/deploy/systemd/domain-api.service` - `domain-api/deploy/systemd/domain-worker.service` - `domain-api/deploy/linux/README.md` +- `domain-api/deploy/multi-region/README.md` +- `domain-api/deploy/multi-region/bootstrap_overseas.sh` +- `domain-api/deploy/multi-region/bootstrap_mainland.sh` - `domain-api/deploy/linux/smoke_test.py` - `domain-api/deploy/linux/collect_diagnostics.sh` diff --git a/docs/13_domainCheck_Linux测试服交接文档.md b/docs/13_domainCheck_Linux测试服交接文档.md index 68d9efb..440ba5a 100644 --- a/docs/13_domainCheck_Linux测试服交接文档.md +++ b/docs/13_domainCheck_Linux测试服交接文档.md @@ -23,17 +23,21 @@ - 交付包、验包、SHA256、最新交付指针已完成 - Linux 测试服数据库初始化问题已定位并修复 - Linux 测试服在“临时 API 进程 + 已初始化数据库”的模式下，`smoke test` 已通过 +- Linux 测试服已经完成正式 `systemd` 服务化联调，`domaincheck-api` 与 `domaincheck-worker` 均已拉起 +- 正式服务态下 `/health` 已确认 `worker_mode = linux-systemd` +- 正式服务态 `smoke test` 已通过 +- 已补做一轮真实导入回归，确认“导入域名 -> 写入 `domains` -> 自动创建 `detect_tasks`”链路正常 ### 2. 未完全完成 -- Linux 测试服当前还不是最终正式部署形态 -- 这次通过的是“临时启动 API 进程”的验证，不是正式 `systemd` 服务托管态 -- `worker_mode` 当前仍表现为 `windows-local` -- `domaincheck-api` / `domaincheck-worker` 还没有按正式生产口径落成 Linux `systemd` 服务闭环 +- Linux 测试服虽然已完成正式服务化联调，但仍属于“测试服验证通过”，不等于生产观察期已经完成 +- Worker 当前通过 `QT_QPA_PLATFORM=offscreen` 运行，属于“无头 Qt 托管态”，后续仍建议继续观察稳定性 +- 真实业务网络环境下的长时检测、代理池质量、Wayback 首次全量列表耗时，还需要继续压测和观察 +- 当前数据库里仅导入了少量回归样本，不代表真实大批量数据已完成验收 一句话结论： -> 功能和数据库问题已经打通；正式 Linux 上线态还差最后一段服务化收口。 +> 功能、数据库和正式服务化都已经打通；后续工作转入真实业务回归、稳定性观察和上线前优化。 ## 三、这次 Linux 测试服已验证通过的内容 @@ -79,7 +83,7 @@ python init_database.py - 库结构已正常 - 只是当前还没有正式业务数据导入 -### 3. smoke test 已通过 +### 3. 临时验证态 smoke test 已通过 新版 `smoke test` 结果： @@ -96,7 +100,7 @@ python init_database.py - Redis 连接可用 - 缺表问题已解除 -### 4. 诊断包已导出 +### 4. 临时验证态诊断包已导出 本次测试服诊断产物： @@ -105,36 +109,95 @@ python init_database.py 可用于后续继续排障或归档。 -## 四、当前测试服不是正式上线态的原因 +## 四、正式服务态已补验证通过 -虽然 `smoke test` 已通过，但当前仍不是正式上线态，原因如下： +在后续继续收口过程中，已经额外完成了正式 `systemd` 服务态验证，结果如下： -### 1. 当前通过的是临时 API 进程验证 +### 1. 正式服务已落地 -本次验证是通过“当前工作区里手动启动的 API 进程”完成的，不是通过正式服务方式完成的。 - -### 2. worker_mode 仍不是 Linux 正式模式 - -当前表现仍是： - -- `worker_mode = windows-local` - -这意味着： - -- 运行中心、配置项、控制逻辑还没有真正切到 Linux 正式托管模式 -- 当前仍属于“测试验证态” - -### 3. systemd 服务还未正式落地闭环 - -还没有最终确认以下两项处于正式可用状态： +已安装并启用： - `domaincheck-api` - `domaincheck-worker` -也就是说： +并已确认两者为 `active (running)`。 -- 现在能证明代码跑得通 -- 但还没有证明“服务化部署后也稳定可用” +### 2. 运行模式已切换成功 + +正式服务态下接口返回已确认： + +- `/health` 中 `worker_mode = linux-systemd` +- `/api/v1/runtime/preflight` 中 `worker_mode = linux-systemd` +- `/api/v1/runtime/status` 中： + - `api.service_name = domaincheck-api` + - `worker.service_name = domaincheck-worker` + - `worker.running = true` + - `process_count = 1` + +### 3. 正式服务态 smoke test 已通过 + +正式服务态下再次执行： + +```bash +python smoke_test.py --base-url http://127.0.0.1:8100 +``` + +结果仍为： + +```json +{ + "ok": true +} +``` + +### 4. 正式服务态诊断包已导出 + +正式服务态诊断产物： + +- 目录：`/opt/domaincheck/diagnostics/diag_20260416_134923` +- 压缩包：`/opt/domaincheck/diagnostics/diag_20260416_134923.tar.gz` + +### 5. 正式服务态中已确认的兼容与权限问题 + +本轮继续联调还确认并处理了两类真实 Linux 问题： + +- `detect/jucha.py`、`detect/juming.py` 中存在 Windows 专属 `subprocess.STARTUPINFO()` 写法，已改为跨平台兼容处理 +- 若曾以 `root` 手工运行过 API/Worker，可能会留下 `root` 所有者的运行文件，导致正式 `systemd` 服务用户无法写入 + +其中已实际踩到并修复的权限点包括： + +- `domain-api/runtime/` 目录权限 +- `domainCheck/detect_worker.log` 文件权限 + +这个问题的表现是： + +- `/api/v1/imports/upload` 因无法创建 `runtime/imports/` 返回 `500` +- Worker 因无法写 `detect_worker.log` 进入循环重启 + +### 6. 运行中心控制链路已补通 + +本轮继续验证后，运行中心里的 Linux 控制动作也已经打通： + +- Worker `start` / `stop` 已可通过 `sudo -n systemctl` 正常执行 +- API 自重启已改为 `systemctl --no-block restart domaincheck-api` + +这样处理后，`systemd` 会异步接管重启流程，接口可以先返回成功，避免出现“服务其实已经重启成功，但调用方因为等待自身停机而收到 `500`”的误导性现象。 + +### 7. 服务态日志噪音已进一步收敛 + +本轮还额外处理了两类不会阻断功能、但会影响正式服务观察体验的日志噪音： + +- Redis 配置订阅从阻塞 `listen()` 改为短轮询 `get_message()`，Linux 空闲时不再每分钟刷 `Redis订阅失败: Timeout reading from socket` +- `offscreen` 模式下去掉 Qt 不支持的按钮样式属性，Worker 启动时不再刷 `Unknown property transition/transform/box-shadow` +- Worker 图标资源定位改为优先使用 `detect_worker.py` 同目录，Linux 服务态不再误报 `/opt/new_logo.svg`、`/opt/favicon2.ico` 不存在 +- 付费检测器改为按需初始化，默认关闭 `detect_jucha` / `detect_juziseo` 时，不再在启动阶段报 cookie 文件缺失 +- Redis 未加载 Bloom 模块时保留为降级说明，继续使用普通缓存，不再作为故障级告警处理 + +这几项修改的结果是： + +- `domaincheck-worker` 仍保持 `active (running)` +- `/api/v1/runtime/status` 中 `worker.running = true` +- 正式测试服日志更适合持续观察和上线前留档 ## 五、这次联调后已经明确固化的部署规则 @@ -169,6 +232,7 @@ python init_database.py - `domain-web` 主要页面已完成 - `domain-api` 主要接口已完成 - 运行中心、系统设置、导入、导出、日志诊断、自检、自测均已具备 +- 正式 `systemd` 服务态下 `/health`、`/runtime/preflight`、`/runtime/status`、`smoke test` 已全部通过 ### 3. 交付层 @@ -185,56 +249,23 @@ python init_database.py - Linux 联调输入清单已完成 - 导航索引文档已完成 +### 5. 真实业务回归层 + +- 已通过 API 上传样本 TXT +- 已确认导入结果： + - 总数 `5` + - 有效 `3` + - 新增 `3` + - 无效 `2` +- 已确认 `domains_total = 3` +- 已确认 `detect_tasks_total = 3` +- 已确认导入后会自动创建 `detect_tasks` + ## 七、当前未完成项清单 以下内容仍属于“后续要做”： -### 1. 正式 Linux 目录落地 - -需要确认正式部署目录结构为： - -```text -/opt/domaincheck -├── domainCheck -├── domain-api -├── domain-web -``` - -如果测试服当前目录不是这一套，需要统一。 - -### 2. 正式 systemd 服务化 - -需要把以下服务真正落好并验证： - -- `domaincheck-api` -- `domaincheck-worker` - -至少要完成： - -- 安装 service 文件 -- `daemon-reload` -- `enable` -- `start` -- `status` -- `journalctl` - -### 3. worker_mode 切换为 linux-systemd - -需要确认： - -- Web 系统设置中运行模式改为 `linux-systemd` -- API `/health` 或运行中心能正确反映： - - `worker_mode = linux-systemd` - -### 4. 正式服务态再跑一次 smoke test - -不是临时 API 进程跑通就结束，还要在正式服务态下再跑一次： - -```bash -python smoke_test.py --base-url http://127.0.0.1:8100 -``` - -### 5. Worker 真实联动验证 +### 1. Worker 真实联动验证 还应继续确认： @@ -243,17 +274,26 @@ python smoke_test.py --base-url http://127.0.0.1:8100 - 检测控制页读取是否正常 - `detect_worker.log` 是否正常写入 -### 6. 导入真实数据后的业务复测 +### 2. 导入更多真实数据后的业务复测 -当前 `domains_count = 0`，说明库表正常，但业务数据还没开始导入。 +当前已经完成一轮小样本回归，不再是空库空表态。 后续应至少补一次真实业务复测： -- 导入域名 -- 查看 `domains` 增长 -- 查看 `detect_tasks` 生成 +- 导入更接近真实业务规模的域名样本 +- 查看 `domains` 持续增长 +- 查看 `detect_tasks` 持续生成 - 再验证筛选与导出 +### 3. 长时间运行与代理池观察 + +仍建议继续验证： + +- Worker 长时运行稳定性 +- Redis 订阅超时后的重连是否持续稳定 +- 国内网络环境下代理池真实可用率 +- Wayback 首次全量快照列表的耗时表现 + ## 八、后续继续收口的推荐顺序 建议 Linux 上的下一位接手人严格按下面顺序执行。 @@ -291,6 +331,11 @@ order by tablename; - `domain-api/deploy/systemd/domain-api.service` - `domain-api/deploy/systemd/domain-worker.service` +如果此前用 `root` 手工跑过 API 或 Worker，建议先确认下面这些路径对正式服务用户可写： + +- `domain-api/runtime/` +- `domainCheck/detect_worker.log` + ### 第四步：启动正式服务 ```bash @@ -318,6 +363,16 @@ curl http://127.0.0.1:8100/api/v1/runtime/preflight curl http://127.0.0.1:8100/api/v1/runtime/status ``` +### 第七步：做一轮导入回归 + +至少验证： + +- `/api/v1/imports/upload` +- `/api/v1/imports/tasks` +- `/api/v1/imports/summary` +- `domains` 增长 +- `detect_tasks` 自动创建 + 目标是确认： - `status = ok` diff --git a/docs/14_domainCheck_正式上线前最终检查单.md b/docs/14_domainCheck_正式上线前最终检查单.md new file mode 100644 index 0000000..36bea7c --- /dev/null +++ b/docs/14_domainCheck_正式上线前最终检查单.md @@ -0,0 +1,244 @@ +# 14 domainCheck 正式上线前最终检查单 + +## 一、使用场景 + +本文档用于正式上线前最后一次收口。 + +适用前提： + +- 代码已更新到最新版本 +- Linux 测试服已完成基础联调 +- PostgreSQL 已执行过 `domainCheck/init_database.py` +- `domaincheck-api` 与 `domaincheck-worker` 已按 `systemd` 托管 + +如果当前仍处于“新机器首次部署”，请先回看： + +- `docs/05_domainCheck_Linux部署清单.md` +- `domain-api/deploy/linux/README.md` +- `docs/13_domainCheck_Linux测试服交接文档.md` + +如果后续进入“国外控制面 + 大陆执行面 + 多 Worker 扩容”阶段，请直接补充阅读： + +- `docs/16_domainCheck_多机检测与跨地域部署设计.md` +- `domain-api/deploy/multi-region/README.md` + +如果当前已经进入“大陆 controller + 海外 control”双地域正式联调，还应额外确认： + +- 大陆 `controller` 节点的 `/etc/default/domaincheck-worker` + - `NODE_ROLE=control` +- 大陆 `domaincheck-sync-agent` + - 已启动并稳定运行 +- 海外控制面 `runtime/sync-summary` + - 能看到 `detect_result_batches` + +## 二、上线前必须确认的结论 + +上线前至少要确认下面这些结论同时成立： + +- `domaincheck-api` 为 `active (running)` +- `domaincheck-worker` 为 `active (running)` +- `/health` 返回 `worker_mode=linux-systemd` +- `/api/v1/runtime/preflight` 返回 `ok=true` +- `smoke test` 返回 `ok=true` +- 运行中心里的 `start_worker / stop_worker / restart_api` 控制链路可用 +- 导入、筛选、批量更新、导出四条核心业务链路至少各回归一次 +- 诊断包可正常导出 + +## 三、正式上线前执行顺序 + +### 1. 核对服务状态 + +```bash +systemctl status domaincheck-api --no-pager -l +systemctl status domaincheck-worker --no-pager -l +``` + +期望： + +- 两个服务都为 `active (running)` +- `domaincheck-worker` 不再循环重启 + +如果当前是大陆 controller 节点，还要补一条： + +```bash +systemctl status domaincheck-sync-agent --no-pager -l +``` + +期望： + +- `domaincheck-sync-agent` 为 `active (running)` +- 不循环重启 + +### 2. 核对健康接口 + +```bash +curl http://127.0.0.1:8100/health +curl http://127.0.0.1:8100/api/v1/runtime/preflight +curl http://127.0.0.1:8100/api/v1/runtime/readiness +curl http://127.0.0.1:8100/api/v1/runtime/status +``` + +期望： + +- `/health` 返回 `status=ok` +- `/health` 返回 `worker_mode=linux-systemd` +- `/runtime/preflight` 返回 `ok=true` +- `/runtime/readiness` 不应返回 `blocking` +- `/runtime/status` 返回 `worker.running=true` + +如果当前已经接入跨地域同步，还建议补看： + +```bash +curl http://127.0.0.1:8100/api/v1/runtime/sync-summary +``` + +期望： + +- 返回里能看到 `detect_result_batches` +- 至少能区分： + - `synced` + - `projected` + - `failed` +- 若大陆 `sync-agent` 已接上，则最近执行过的检测任务不应长期停留在 `projected` + +当前 `runtime/preflight` 还会额外展示： + +- RedisBloom 是否安装，若未安装会明确提示“降级为普通缓存” +- `detect_jucha` / `detect_juziseo` 是否启用 +- 若启用了付费检测，对应本地 cookie 文件是否已就绪 + +当前 `detect/status` / `runtime/status` 还会明确展示代理运行态： + +- `代理正常` +- `降级直连` +- `等待代理` + +解释： + +- `降级直连` 表示代理池暂时无可用代理，但当前允许直连兜底，任务不会因此中断 +- `等待代理` 表示代理池无可用代理，且当前未允许直连，这会直接影响检测吞吐或导致步骤失败 + +当前还补充了两类更细的诊断语义： + +- `proxy_runtime_reason=supplier_empty_pool` + - 表示代理源最近都返回了正常 HTTP 响应，但原始代理数为 `0` + - 这类问题更偏向供应侧空池，不是程序拉取失败 +- `dependency_alerts` + - 表示外部依赖站点当前存在可观测异常 + - 例如 `web.archive.org` 拒连、超时、连接池异常等 + +当前免费检测链路中的外部依赖异常还采用了“降级继续”策略： + +- 例如 `时光机 / 站长之家 / 爱站 / 百度 / 360` 这类外部站点出现明显网络波动、拒连、超时 +- 系统会优先记录为步骤级 `degraded` +- 并将域名置为 `待人工复核` +- 同时继续执行后续检测步骤，避免把整条任务链路直接污染成 `检测失败` + +### 3. 跑正式服务态 smoke test + +```bash +cd /opt/domaincheck/domain-api/deploy/linux +python3 smoke_test.py --base-url http://127.0.0.1:8100 +``` + +如需把 Web 首页一起纳入检查： + +```bash +python3 smoke_test.py --base-url http://127.0.0.1:8100 --web-url http://127.0.0.1 +``` + +期望： + +- 输出中 `ok=true` + +### 4. 回归运行中心控制动作 + +建议至少各执行一次： + +- `start_worker` +- `stop_worker` +- `restart_api` + +期望： + +- `stop_worker` 后，`runtime/status` 中 `worker.running=false` +- `start_worker` 后，`runtime/status` 中 `worker.running=true` +- `restart_api` 调用返回成功，随后 API 能重新恢复 + +说明： + +- 当前 Linux 正式服务态已改为 `systemctl --no-block restart domaincheck-api` +- 因此 `restart_api` 会先返回成功，再由 `systemd` 异步完成 API 切换 + +### 5. 回归核心业务链路 + +建议至少确认下面 4 条： + +- 导入域名成功，且会落到 `domains` +- 导入后自动创建 `detect_tasks` +- 筛选接口 `/api/v1/domains` 返回正常 +- 批量更新后数据库字段与页面展示一致 +- 导出任务可创建、导出文件可下载 + +如果时间紧，最小回归顺序建议为： + +1. 导入一份小样本 +2. 查看导入任务结果 +3. 查看域名列表 +4. 执行一次批量更新 +5. 生成一次 TXT 或 CSV 导出 + +## 四、建议保留的上线证据 + +正式上线前，建议至少保留下面这些结果： + +- `systemctl status domaincheck-api --no-pager -l` +- `systemctl status domaincheck-worker --no-pager -l` +- `journalctl -u domaincheck-api -n 200 --no-pager` +- `journalctl -u domaincheck-worker -n 200 --no-pager` +- `systemctl status domaincheck-sync-agent --no-pager -l` +- `journalctl -u domaincheck-sync-agent -n 200 --no-pager` +- `/health` 返回 +- `/api/v1/runtime/preflight` 返回 +- `/api/v1/runtime/sync-summary` 返回 +- `smoke_test.py` 输出 +- 一份最新诊断包 + +诊断包导出命令： + +```bash +cd /opt/domaincheck/domain-api/deploy/linux +bash collect_diagnostics.sh /opt/domaincheck +``` + +## 五、当前可接受的非阻断项 + +以下项目目前已确认不会阻断正式服务运行： + +- Redis 未安装 Bloom 模块时，会降级为普通缓存 +- `worker` 在 `QT_QPA_PLATFORM=offscreen` 下无头运行 +- 付费检测默认关闭时，不要求本机预置 `jucha` / `juziseo` cookie 文件 + +这些项可以在后续优化阶段继续增强，但不应再作为当前上线阻塞条件。 + +## 六、仍建议继续观察的项 + +以下内容不属于当前功能阻断，但上线后建议继续观察： + +- 真实代理池可用率 +- 代理为空时是否长期停留在 `降级直连` +- Wayback 首次全量检测耗时 +- 长时间运行下的 Worker 稳定性 +- 真实业务数据量上升后的数据库与导出耗时 + +## 七、最终上线判定 + +如果下面三类检查都通过，就可以视为已经具备正式上线条件： + +- 服务状态检查通过 +- 核心接口与 `smoke test` 通过 +- 核心业务链路回归通过 + +一句话判定： + +> 当前项目已经完成开发、Linux 测试服联调、正式服务态验证和关键日志降噪；剩余工作主要是正式环境发布与上线后观察，不再是功能性大修。 diff --git a/docs/15_domainCheck_Web对齐旧桌面功能核对表.md b/docs/15_domainCheck_Web对齐旧桌面功能核对表.md new file mode 100644 index 0000000..9792b07 --- /dev/null +++ b/docs/15_domainCheck_Web对齐旧桌面功能核对表.md @@ -0,0 +1,59 @@ +# domainCheck Web 对齐旧桌面功能核对表 + +更新时间：2026-04-16 + +## 一、旧桌面主标签页对齐 + +| 旧桌面标签 | Web 当前状态 | 说明 | +| --- | --- | --- | +| 聚名爬取 | 已补齐 | 已支持聚名账号密码登录、聚查自动联名、过期删除/一口价采集、Cookie 兜底上传 | +| 域名筛选 | 已补齐主干 | 已补“全部”默认选项，支持查询、分页、批量更新、筛选页导出范围，并继续补回旧桌面明细列 | +| 域名导入 | 已补齐主干 | 已支持 TXT 上传导入，也已补回手工粘贴域名创建导入任务 | +| 敏感词配置 | 已补齐 | 本轮新增独立页面，支持加载、编辑、导入、导出、保存 | +| 系统设置 | 已补齐主干 | 已补桔子SEO 登录入口；聚名登录入口放在“聚名采集”页 | + +## 二、旧桌面登录能力对齐 + +| 能力 | Web 当前状态 | 说明 | +| --- | --- | --- | +| 聚名账号密码登录 | 已补齐 | 位于 `#/juming` | +| 聚查联名登录 | 已补齐 | 聚名登录后自动执行，同时提供手动“重新联名登录聚查” | +| 桔子SEO 登录 | 已补齐 | 位于 `#/settings` 的“第三方登录”卡片 | + +## 三、已确认补齐的体验项 + +- 域名筛选下拉框显式提供“全部” +- 桔子SEO 不再遗漏登录入口 +- 敏感词配置不再缺页 +- 聚查不只依赖自动联名，也可手动重试 +- 域名导入已补回“粘贴域名后直接导入” +- 筛选页已补回“当前页 / 导出几页 / 全部”的导出范围能力 +- 筛选页已补回旧桌面常用明细列：过期时间、单位性质、百度历史、百度Site、中文标题、360 Site、Google Site +- 批量更新已补回上述检测明细字段的人工修正入口 +- CSV / Excel 导出已补回旧桌面常用明细列，避免导出结果比桌面版缩水 +- 聚名页面已显式展示“已保存账号 / 聚名登录态 / 聚查联名状态”，避免运营误判 +- 系统设置页已显式展示“已保存桔子SEO账号 / Cookie 就绪状态”，并同步持久化到本地与 Redis +- 聚名采集、域名导入、检测控制三条长任务链路均已统一为“任务中心”形态，支持阶段态、日志常驻、切页后回看 +- 检测控制页已补齐更细的阶段态展示，可识别“准备检测 / 刷新代理池 / 取任务中 / 建线程中 / 检测中 / 批次完成 / 完成归档” +- 检测控制页已补齐阶段切换时间线，便于回看长任务在何时进入哪个阶段 +- 聚名采集页已补齐中文状态标签、当前采集阶段与任务概览卡片，避免运营面对英文状态误判 +- 域名导入页已补齐“选中文件预估行数”和“当前导入阶段/任务概览”摘要，更接近旧桌面即时反馈体验 +- 域名导入页已补齐“前 10 行本地预览”和“大文件任务提示”，降低误传错文件的概率 +- 域名导入页已补齐提交前本地预检，可即时展示“有效 / 重复 / 非法”统计 +- 域名导入页已补齐“来源类型”选择与任务展示，导入记录可区分 TXT 导入 / 手工录入 / 其它 +- 域名筛选页已补齐“单位性质”筛选，进一步对齐旧桌面常用条件 +- 域名筛选页已补齐“来源类型”筛选；导入页完成后可一键跳转到筛选页查看对应来源结果 +- 聚名采集、域名导入、检测控制三页的任务中心文案已统一，均按“阶段 / 摘要 / 日志 / 回看”同一口径表达 + +## 四、下一轮继续核对的次级项 + +- 旧桌面导入页的少量细节交互，是否还需要补更细的批次说明 +- 旧桌面检测控制页的少量状态提示文案，是否还需要继续压词统一 +- 旧桌面批量筛选中的少量边缘筛选项，是否还有遗漏 + +## 五、当前可直接验证的页面 + +- `http://152.53.37.118:3201/#/juming` +- `http://152.53.37.118:3201/#/domains` +- `http://152.53.37.118:3201/#/settings` +- `http://152.53.37.118:3201/#/sensitive-words` diff --git a/docs/16_domainCheck_多机检测与跨地域部署设计.md b/docs/16_domainCheck_多机检测与跨地域部署设计.md new file mode 100644 index 0000000..e185c1d --- /dev/null +++ b/docs/16_domainCheck_多机检测与跨地域部署设计.md @@ -0,0 +1,796 @@ +# 16 domainCheck 多机检测与跨地域部署设计 + +## 一、目标 + +本文档用于固化 `domainCheck` 的正式多机落地方案，满足下面这些约束： + +- Web 后台、管理 API 必须部署在国外机器 +- 检测执行、代理、Redis、运行态存储部署在大陆机器 +- 最终结果、配置、归档、备份以国外机器为主 +- 初期部署不复杂，支持从 `国外 1 台 + 大陆 1 台` 起步 +- 后期不够时，优先横向增加大陆 Worker，不推翻整体架构 + +一句话原则： + +> 国外负责“管、看、存、备份”，大陆负责“跑、调度、抢任务、贴近代理资源”。 + +## 二、当前落地状态 + +截至 `2026-04-16`，第一阶段多机运行骨架已经完成，当前代码已具备： + +- 控制面节点注册与心跳 +- 执行面节点注册与心跳 +- 运行库骨架表自动初始化 +- 集群节点快照接口 +- 国外/大陆两套部署脚手架 + +当前已落地的运行态表： + +- `detect_worker_nodes` +- `detect_jobs` +- `detect_job_items` +- `detect_run_events` +- `detect_sync_records` + +当前可直接验证的接口： + +- `GET /api/v1/runtime/status` +- `GET /api/v1/runtime/readiness` +- `GET /api/v1/runtime/cluster` +- `GET /api/v1/detect/job/active` +- `GET /api/v1/detect/jobs` +- `GET /api/v1/detect/queue-summary` + +当前 `runtime/cluster` 还补充了 `summary` 摘要，至少会返回： + +- `status_counts` +- `role_counts` +- `region_counts` +- `online_worker_nodes` +- `online_control_nodes` +- `busy_nodes` +- `stale_nodes` +- `offline_nodes` + +当前还补充了同步观测骨架： + +- `GET /api/v1/runtime/sync-summary` +- `GET /api/v1/runtime/sync-records` + +可用于提前观察： + +- 当前节点是否启用同步推送 +- 源地域 / 目标地域配置是否正确 +- `detect_sync_records` 最近状态分布 + +当前代码还补了一层“运行态自动投影”： + +- 控制面在读取 `runtime/status` 时，会把当前运行摘要按去重策略写入 `detect_sync_records` +- 当前记录类型为： + - `runtime_projection` +- 当前还新增了“结果摘要投影”： + - `detect_result_projection` +- 用于把当前活跃任务的摘要、最近阶段事件、任务收敛进度先沉淀到统一同步面板 +- 其作用不是替代后续真正的 `sync-service` +- 而是先保证跨地域部署阶段已经有统一可观察的“同步记录面板” + +当前还补上了正式同步最小闭环骨架： + +- 大陆执行面可通过 `SYNC_TARGET_API_BASE_URL` 指向国外控制面 +- 若 `SYNC_PUSH_ENABLED=true`，应由大陆 `domaincheck-sync-agent` 按轮询间隔自动推送最新运行态投影 +- 国外控制面通过： + - `POST /api/v1/runtime/sync-ingest` + 接收投影 +- 若配置了 `SYNC_SHARED_TOKEN`，接收端会校验： + - `X-Domaincheck-Sync-Token` + +因此当前虽然还没有完整业务结果同步服务，但“同步协议、发送端、接收端、记录面板、独立同步代理”都已经有了第一版可运行骨架 + +当前同步骨架已经支持的记录类型包括： + +- `runtime_projection` +- `runtime_ingest` +- `detect_result_projection` +- `detect_result_ingest` +- `runtime_push` + +当前这一轮还补上了“结果批次同步可观测”能力： + +- `GET /api/v1/runtime/sync-summary` 现在会额外返回： + - `detect_result_batches` +- 可直接按最近检测任务查看： + - 是否已生成结果投影 + - 是否已推送 + - 是否已被目标地域接收 + - 最近一次失败原因 +- 当前同步策略也已细化为： + - `runtime_projection` + 只推最新一条，避免把历史运行态投影整批补传 + - `detect_result_projection` + 按批次补推，确保检测结果积压可以自动追平 +- 大陆 `controller` 的环境模板也已固定为： + - `NODE_ROLE=control` + 避免把 controller 误注册成普通 Worker，导致运行中心和 sync-agent 承载判断失真 + +当前这一轮还额外补齐了“检测会话追踪”能力： + +- `POST /api/v1/detect/start` 会生成本轮 `cycle_token` +- `cycle_token` 会进入 Worker 控制指令 +- Worker 心跳元数据会回写 `cycle_token / job_id / job_code` +- `domain_started / domain_completed / domain_failed` 事件会附带 `cycle_token` +- `GET /api/v1/detect/job/active` 已支持按当前 `cycle_token` 聚合“本轮事件” + +当前运行态还补齐了“代理执行语义”： + +- 代理可用时显示 `代理正常` +- 代理暂时不可用但允许直连时显示 `降级直连` +- 代理暂时不可用且不允许直连时显示 `等待代理` +- 当代理源全部 `HTTP 200` 但 `raw_items=0` 时，会额外标记为“供应池为空”，避免误判成程序异常 + +当前阶段的准确表述应为： + +> 多机运行骨架已经落地，分布式调度与跨地域结果同步进入下一阶段。 + +截至当前这一轮代码，最小任务闭环也已经接上： + +- `POST /api/v1/detect/start` 会先创建 `detect_jobs` +- Worker 会优先从 `detect_job_items` 领取任务 +- 运行中心已能显示当前活跃任务摘要 +- 节点状态里已能反映 `busy / current_load` +- Worker 已支持小批量领取、任务续租、过期租约回收 +- API 控制面节点已支持定时心跳，避免集群视图把控制面误判为离线 +- 活跃任务接口已支持节点分布、进度百分比和最近事件回看 +- Worker 进程重启后会主动回收当前节点遗留的 `running / claimed` 运行态，避免后台出现“进程已重启但旧任务仍显示运行中”的残影 + +这意味着后续新增第二台大陆 Worker 时，已经不再是“完全从零设计”的状态，而是可以在现有骨架上继续细化调度策略。 + +当前最小联调命令也已经固定为： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100 +``` + +这条命令会连续检查： + +- `/health` +- `/api/v1/runtime/cluster` +- `/api/v1/detect/status` +- `/api/v1/detect/job/active` + +如需单独检查“跨地域同步骨架是否已接上”，建议再补两条： + +```bash +curl http://127.0.0.1:8100/api/v1/runtime/sync-summary +curl http://127.0.0.1:8100/api/v1/runtime/sync-records?limit=10 +``` + +如需快速拿到“当前这套多机部署是否已进入可联调 / 待处理 / 阻断”结论，当前也已经固定了： + +```bash +curl http://127.0.0.1:8100/api/v1/runtime/readiness +``` + +该接口会直接给出： + +- `status` + - `ready` + - `attention` + - `blocking` +- `summary` +- `blocking_issues` +- `warnings` +- `info` + +如果这两条接口已经能看到： + +- `source_region / target_region` +- `records_total > 0` +- 最近记录里出现 `runtime_projection` + +则说明当前测试服已经具备： + +- 同步配置观测入口 +- 同步记录落库入口 +- 后续接入正式 `sync-service` 的最小运行骨架 + +如果要在大陆 controller 节点本机确认“这台机器本身是否配成了 controller，而不是普通 Worker”，当前也已经固定了一条本机检查命令： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_mainland_controller.sh +``` + +这条命令会直接校验： + +- `/etc/default/domaincheck-worker` +- `NODE_ROLE=control` +- `SYNC_PUSH_ENABLED=true` +- `SYNC_TARGET_API_BASE_URL` 是否已填写 +- `domaincheck-worker` +- `domaincheck-sync-agent` + 是否已经启动 + +如果当前客观条件下还不能把代码真正部署到大陆机器，当前也支持先在国外测试机上做“模拟多节点联调”： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/simulate_multi_region.sh http://127.0.0.1:8100 +``` + +这条命令会临时模拟： + +- 一个大陆 `controller` +- 一个大陆 `worker` + +它的目的不是替代真实部署，而是先把下面这些提前跑通： + +- `runtime/readiness` +- `runtime/cluster` +- 运行中心的多机就绪度提示 +- 文档、脚本、状态面板三者是否一致 + +如果测试服之前已经跑过旧节点、旧 worker，当前还残留陈旧心跳记录，也可以先清一次 `detect_worker_nodes` 里的历史残影，再观察 readiness： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 --dry-run +bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 +``` + +它的作用是： + +- 清掉很久没心跳的旧节点记录 +- 避免运行中心长期被历史离线节点污染成 `attention` +- 让单机模拟多节点联调时，看到更接近真实接入后的状态 + +## 三点五、当前实测结论 + +截至 `2026-04-16 18:56`，测试服已经验证通过： + +- `domaincheck-api` 与 `domaincheck-worker` 可在 `systemd` 下稳定重启 +- `worker_mode=linux-systemd` +- `runtime/cluster` 可同时看到： + - `overseas-control-01` + - `mainland-worker-01` +- 大陆 Worker 节点元数据已显示： + - `phase` + - `detail` + - `job_id` + - `job_code` + - `cycle_token` +- `detect/job/active` 的 `current_cycle_events` 已只回看当前这一轮检测事件，不再混入旧轮次历史 + +当前测试服最新一轮实测 `cycle_token` 示例： + +- `aec105737f` + +该轮已确认能看到： + +- `job_dispatch_requested` +- `job_dispatch_sent` +- `domain_started` +- `domain_completed` + +这说明“控制面发起一次检测”到“执行面逐域名回写事件”的会话闭环已经打通。 +也说明当前后台已经不再只会显示“有告警”，而是能明确说明任务到底是在正常代理、降级直连，还是被代理卡住。 + +## 四、推荐架构 + +### 1. 国外控制面 + +国外机器负责： + +- `domain-web` +- `domain-api` +- 国外主 PostgreSQL +- 配置中心 +- 导出、审核、运营后台 +- 最终结果归档 +- 国外备份与日志归档 + +国外控制面的职责： + +- 对外提供访问入口 +- 保存最终业务结果 +- 保存系统配置和审计记录 +- 接收大陆执行面的结果同步 +- 为后续灾备提供稳定的权威数据源 + +### 2. 大陆执行面 + +大陆机器负责： + +- `detect-worker` +- 大陆 Redis +- 大陆运行库 PostgreSQL +- 代理池配置与网络环境 +- 本地运行日志 +- 调度和任务租约 + +大陆执行面的职责： + +- 抢任务 +- 续租 +- 运行检测链路 +- 写入本地运行态 +- 汇总并同步结果到国外 + +### 3. 初期物理部署 + +当前最优起步方案： + +- 国外 `1` 台： + - `domain-web` + - `domain-api` + - `postgresql_main` +- 大陆 `1` 台： + - `redis` + - `postgresql_runtime` + - `scheduler` + - `detect-worker` + +说明： + +- 这个起步方案的优先目标是先把跨地域职责边界定清楚 +- 大陆机器当前可以先把 Redis、运行库、调度和 Worker 放在同一台 +- 但逻辑上必须按不同角色设计，避免后续扩容时重新返工 +- 各节点身份建议统一通过 `/etc/default/domaincheck-api` 与 `/etc/default/domaincheck-worker` 管理，不直接修改 service 文件正文 + +### 4. 后续扩容 + +当大陆执行压力不够时，扩容顺序建议为： + +1. 新增大陆 Worker 机器 +2. 如果 Redis 压力增大，再拆 Redis +3. 如果运行库写入压力增大，再拆运行库 +4. 国外主库继续作为最终结果主库，不参与高频任务调度 + +后续结构示意： + +```text +国外 +├── web + api + postgresql_main +└── backup / archive + +大陆 +├── redis + postgresql_runtime + scheduler +├── worker-01 +├── worker-02 +├── worker-03 +└── worker-N +``` + +## 五、数据分层 + +### 1. 结果态数据 + +这类数据保存在国外主库： + +- 域名主数据 +- 检测最终结果 +- 黑名单结论 +- 导出记录 +- 审核记录 +- 系统配置 +- 操作审计 + +特点： + +- 最终一致 +- 生命周期长 +- 提供给后台页面和运营使用 + +### 2. 运行态数据 + +这类数据优先保存在大陆执行面： + +- 任务领取状态 +- 任务租约 +- 节点心跳 +- 当前阶段 +- 实时日志事件 +- 重试信息 +- 本地暂存结果 + +特点： + +- 高频读写 +- 延迟敏感 +- 主要服务于 Worker 协作和调度 + +## 六、任务模型 + +当前项目还偏向单机 Worker 直接扫描 `domains` 表的模式。正式多机化后，建议改为显式任务模型。 + +### 1. 核心表建议 + +建议新增： + +- `detect_jobs` +- `detect_job_items` +- `detect_worker_nodes` +- `detect_run_events` +- `detect_sync_records` + +### 2. detect_jobs + +表示一轮检测任务。 + +建议字段： + +- `id` +- `job_code` +- `source` +- `plan_hash` +- `status` +- `created_at` +- `started_at` +- `finished_at` +- `created_by` +- `remark` + +说明： + +- `plan_hash` 用于标识本轮检测所使用的规则版本和参数快照 +- 一轮任务可以对应大量域名任务项 + +### 3. detect_job_items + +表示单个域名的具体任务项。 + +建议字段： + +- `id` +- `job_id` +- `domain_id` +- `status` +- `claimed_by` +- `claim_token` +- `lease_expires_at` +- `attempt_count` +- `last_error` +- `started_at` +- `finished_at` +- `result_version` +- `updated_at` + +建议唯一约束： + +- `unique(job_id, domain_id)` + +说明： + +- 多台 Worker 只抢 `status=pending` 且未被占用或租约已过期的任务 +- `claim_token` 用于避免状态回写串单 +- `lease_expires_at` 用于机器故障后的任务回收 + +### 4. detect_worker_nodes + +表示 Worker 节点。 + +建议字段： + +- `node_code` +- `region` +- `role` +- `hostname` +- `ip` +- `status` +- `worker_version` +- `last_heartbeat_at` +- `current_load` +- `remark` + +说明： + +- 后台后续可以直接展示“哪个节点在线、哪个节点在跑、当前负载多少” + +### 5. detect_run_events + +表示高价值阶段日志。 + +建议字段： + +- `id` +- `job_id` +- `job_item_id` +- `node_code` +- `event_type` +- `level` +- `message` +- `payload_json` +- `created_at` + +说明： + +- 页面展示优先依赖事件流，不直接硬依赖某一台机器本地日志文件 +- 原始大日志仍保留在大陆 Worker 本地 + +## 七、多机抢任务与去重 + +### 1. 抢任务方式 + +推荐使用数据库租约模型： + +- Worker 从 `detect_job_items` 中领取一批待处理任务 +- 使用 `FOR UPDATE SKIP LOCKED` 或原子更新方式抢占 +- 领取后写入： + - `claimed_by` + - `claim_token` + - `lease_expires_at` + +### 2. 去重原则 + +多机环境下，去重不能依赖“约定不重复”，必须依赖数据库约束和租约。 + +至少保证： + +- 同一轮 `job_id` 下，同一域名只出现一条任务项 +- 同一任务项同一时刻只允许被一个 `claim_token` 持有 +- 回写结果时校验 `claim_token` + +### 3. 续租机制 + +Worker 执行过程中应定期续租： + +- 每 `30-60` 秒续租一次 +- 续租时更新 `lease_expires_at` +- 如果节点挂掉，租约到期后其他 Worker 可重新领取 + +当前代码已实现的原则是： + +- 任务被领取后进入 `claimed / running` +- 检测步骤执行过程中会持续续租 +- 若任务长期未续租且租约过期，会自动回收到 `pending` + +这样可以避免： + +- 长任务跑到一半租约失效 +- Worker 异常退出后任务永远卡死 +- 后续多机扩容时出现大量“假占用” + +### 4. 失败重试 + +建议对失败进行分类： + +- 网络失败 +- 代理失败 +- 第三方目标站异常 +- 程序逻辑失败 + +可按类型控制重试次数，不要无限重试。 + +### 5. 当前代码与目标模型的关系 + +当前代码里，运行态表和节点心跳已经具备，但主检测流程仍以现有 `domains` / `detect_status` 链路为主。 + +这意味着： + +- 现在已经可以先把“多机可见性”和“节点骨架”跑起来 +- 下一步要把“检测入口”从直接扫 `domains`，逐步切到 `detect_jobs + detect_job_items` +- 切换时必须保留旧链路回退能力，避免一次性重写导致现网不稳 + +## 八、同步策略 + +### 1. 国外到大陆 + +同步内容： + +- 系统配置 +- 检测规则 +- 敏感词版本 +- 任务创建命令 + +特点： + +- 低频 +- 要求可靠 +- 可版本化 + +### 2. 大陆到国外 + +同步内容： + +- 任务进度摘要 +- 关键阶段事件 +- 最终检测结果 +- 黑名单结论 +- 归档日志包索引 + +特点： + +- 高频部分只传摘要和关键事件 +- 最终结果必须幂等 + +### 3. 幂等要求 + +结果同步到国外时，必须支持幂等： + +- 同一 `job_item_id` 重复上报不应产生重复结果 +- 同一域名同一轮任务只保留最终有效状态 + +### 4. 推荐同步落地方式 + +第一阶段不建议直接做数据库双向复制，更推荐独立同步服务： + +- 大陆执行面负责写本地运行态 +- `sync-service` 负责批量汇总结果与事件 +- 国外控制面负责接收、落库、归档 + +推荐第一阶段同步内容： + +- 最终检测结果 +- 关键阶段事件 +- 失败摘要 +- 节点健康摘要 +- 配置快照版本 + +不建议第一阶段同步： + +- 全量原始日志 +- 高频心跳明细 +- 临时抓取中间产物 + +原因： + +- 成本高 +- 跨地域网络抖动会放大耦合 +- 对运营后台价值不成比例 + +## 九、部署建议 + +### 1. 国外机器 + +目录建议： + +```text +/opt/domaincheck +├── domain-api +├── domain-web +└── shared +``` + +部署角色： + +- `web` +- `api` +- `postgresql_main` + +### 2. 大陆调度中心 + +目录建议： + +```text +/opt/domaincheck +├── domainCheck +├── runtime-db +├── redis +└── shared +``` + +部署角色： + +- `scheduler` +- `redis` +- `postgresql_runtime` +- `worker` + +### 3. 大陆纯 Worker 节点 + +目录建议： + +```text +/opt/domaincheck +├── domainCheck +└── shared +``` + +部署角色： + +- `worker` + +### 4. 一键化原则 + +部署脚本必须满足： + +- 新机器只需要少量环境变量 +- 不要求现场手工改很多路径 +- 国外机和大陆机各自有独立入口脚本 +- 后续新增大陆 Worker 节点继续复用同一脚本 + +### 5. 当前建议的最小部署 + +当前最适合正式起步的形态： + +- 国外 `1` 台： + - `domain-web` + - `domain-api` + - 国外主 PostgreSQL +- 大陆 `1` 台： + - `redis` + - 大陆运行库 PostgreSQL + - `detect-worker` + - 后续可补 `sync-service` + +扩容时优先： + +1. 新增大陆 Worker +2. 再视压力拆分大陆运行库与 Redis +3. 国外控制面保持稳定，不跟着高频扩缩 + +## 十、阶段性落地建议 + +### 第一阶段 + +先完成： + +- 国外控制面部署 +- 大陆单机执行面部署 +- 节点注册与心跳 +- 当前后台继续可用 + +### 第二阶段 + +再完成： + +- 任务项模型 +- 多 Worker 抢任务 +- 事件日志流 +- 结果异步同步回国外 + +### 第三阶段 + +最后完成： + +- 大陆多 Worker 横向扩容 +- 运行库与调度中心独立拆分 +- 国外归档和监控补齐 + +## 十一、运行与观测建议 + +多机之后，不能再只靠“登录某台机器 tail 日志”来判断系统状态。 + +建议分层： + +- 大陆 Worker 本地保留完整原始日志 +- `detect_run_events` 保存关键阶段事件 +- 国外后台优先展示事件流、节点负载和同步状态 +- 真排障时再下钻到具体节点原始日志 + +后台建议优先展示： + +- 当前在线节点数 +- 每节点最近心跳 +- 每节点当前负载 +- 当前活跃任务数 +- 队列积压与最老待领年龄 +- 租约是否过期、是否即将到期 +- 近 15 分钟每节点吞吐 +- 最近失败摘要 +- 最近同步结果 +- 最近结果摘要投影 + +## 十二、下一阶段开发清单 + +接下来建议按下面顺序继续落地： + +1. 后台创建 `detect_jobs / detect_job_items` +2. Worker 实现批量 claim / renew / finish / fail +3. 增加租约超时回收 +4. 将检测阶段事件写入 `detect_run_events` +5. 增加 `sync-service` +6. 完成第二台大陆 Worker 接入演练 + +## 十三、最终建议 + +对当前项目来说，最优解不是“全部放国外”，也不是“全部放大陆”，而是： + +- 国外做控制面和主结果库 +- 大陆做执行面和运行态调度 +- 先从 `国外 1 台 + 大陆 1 台` 起步 +- 后续优先横向增加大陆 Worker + +一句话总结： + +> 先把职责边界设计对，再让部署简单；后续扩容时只加 Worker，不再重构核心架构。 diff --git a/docs/17_domainCheck_全流程部署实操手册.md b/docs/17_domainCheck_全流程部署实操手册.md new file mode 100644 index 0000000..6043322 --- /dev/null +++ b/docs/17_domainCheck_全流程部署实操手册.md @@ -0,0 +1,647 @@ +# 17 domainCheck 全流程部署实操手册 + +## 一、文档目标 + +这份文档不是设计说明，也不是零散检查单，而是给实际部署时直接照着做的“全流程操作手册”。 + +适用场景： + +- 先在国外机器完成单机部署与联调 +- 后续再扩到“国外控制面 + 大陆执行面” +- 当前暂时无法把代码真正部署到大陆机器时，先在国外机器模拟多节点联调 +- 最后再把同样的部署方式复制到其他机器 + +一句话理解： + +> 先把国外主控机部署好，再按同样套路扩第二台、第三台，不靠临场猜。 + +并发配置补充： + +- 系统支持 `默认线程数 + 节点单独覆盖` +- 节点单独覆盖按机器自己的 `NODE_CODE` 命中 +- 某台机器没配置覆盖值时，自动回退到默认线程数 + +## 二、推荐部署形态 + +### 1. 当前最推荐起步形态 + +- 国外机器 `1` 台： + - `domain-web` + - `domain-api` + - PostgreSQL + - Redis + - `domaincheck-worker` +- 后续扩容时： + - 新增大陆 `controller` + - 新增大陆 `worker` + +说明： + +- 因为你当前还不能直接把代码稳定部署到大陆机器，所以第一阶段先把国外机器单机跑稳 +- 这台国外机器同时承担： + - 后台 + - API + - 数据库 + - Redis + - Worker +- 等这套稳定后，再把多机脚本复制到其他机器 + +### 2. 最终推荐形态 + +- 国外 `control`： + - `domain-web` + - `domain-api` + - 主 PostgreSQL + - 结果归档 +- 大陆 `controller`： + - Redis + - 运行态 PostgreSQL + - `domaincheck-worker` + - `domaincheck-sync-agent` +- 大陆 `worker`： + - `domaincheck-worker` + +## 三、服务器准备 + +### 1. 系统要求 + +- CentOS Stream 9 +- Python `3.11` +- PostgreSQL `14+` +- Redis `6+` +- Nginx + +### 2. 目录约定 + +统一使用： + +```text +/opt/domaincheck +├── domain-api +├── domain-web +└── domainCheck +``` + +### 3. 当前仓库与运行态关系 + +如果你像当前测试机一样，用仓库目录做源码源头，也可以用软链接： + +```bash +ln -s /www/wwwroot/getDomain/domain-api /opt/domaincheck/domain-api +ln -s /www/wwwroot/getDomain/domain-web /opt/domaincheck/domain-web +ln -s /www/wwwroot/getDomain/domainCheck /opt/domaincheck/domainCheck +``` + +如果你是完整复制代码到目标机，也可以直接把目录上传到 `/opt/domaincheck/`。 + +## 四、第一阶段：国外单机部署 + +这一阶段的目标是： + +- 后台可访问 +- API 正常 +- 数据库已初始化 +- Worker 可跑 +- 运行中心正常 +- smoke test 通过 + +### 1. 拉取代码 + +当前建议统一走 `git` 管理和更新，不再手工散传目录。 + +第一次部署建议： + +```bash +mkdir -p /www/wwwroot +cd /www/wwwroot +git clone 你的仓库地址 getDomain +cd getDomain +git checkout main +git pull origin main +``` + +然后建立运行目录软链接： + +```bash +mkdir -p /opt/domaincheck +ln -s /www/wwwroot/getDomain/domain-api /opt/domaincheck/domain-api +ln -s /www/wwwroot/getDomain/domain-web /opt/domaincheck/domain-web +ln -s /www/wwwroot/getDomain/domainCheck /opt/domaincheck/domainCheck +``` + +后续更新统一使用： + +```bash +cd /www/wwwroot/getDomain +git fetch --all +git checkout main +git pull --ff-only origin main +``` + +### 2. 创建 Python 虚拟环境 + +```bash +cd /opt/domaincheck/domainCheck +python3.11 -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +``` + +然后补 API 依赖： + +```bash +cd /opt/domaincheck/domain-api +/opt/domaincheck/domainCheck/.venv/bin/pip install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart +``` + +### 3. 配置数据库和 Redis + +确保 PostgreSQL 和 Redis 可用。 + +建议先确认： + +```bash +psql -h 127.0.0.1 -U postgres -d domain -c "select 1;" +redis-cli ping +``` + +### 4. 配置 `domainCheck/.env` + +至少确认这些项： + +```env +DB_HOST=127.0.0.1 +DB_PORT=5432 +DB_DATABASE=domain +DB_USER=postgres +DB_PASSWORD=你的密码 + +REDIS_HOST=127.0.0.1 +REDIS_PORT=6379 +REDIS_PASSWORD= +REDIS_DB=0 +``` + +### 5. 初始化数据库 + +这是最重要的一步，新库必须先做： + +```bash +cd /opt/domaincheck/domainCheck +python3 init_database.py +``` + +如果不先执行，至少这些接口会直接 `500`： + +- `/api/v1/dashboard/overview` +- `/api/v1/detect/status` +- `/api/v1/imports/summary` + +### 6. 构建前端 + +进入 `domain-web`，配置生产环境 API 地址： + +```bash +cd /opt/domaincheck/domain-web +cp .env.production.example .env.production +``` + +把 `VITE_API_BASE_URL` 改成目标 API，例如： + +```env +VITE_API_BASE_URL=https://api.domain.com/api/v1 +``` + +然后构建： + +```bash +npm install +npm run build +``` + +### 7. 配置 Nginx + +参考： + +- `domain-web/deploy/nginx/domain-web.conf` + +把静态目录指到： + +```text +/opt/domaincheck/domain-web/dist +``` + +常见做法： + +- `admin.domain` 对外提供后台页面 +- `api.domain.com` 对外提供 API + +### 8. 安装 systemd 服务 + +复制模板： + +```bash +cp /opt/domaincheck/domain-api/deploy/systemd/domain-api.service /etc/systemd/system/domaincheck-api.service +cp /opt/domaincheck/domain-api/deploy/systemd/domain-worker.service /etc/systemd/system/domaincheck-worker.service +``` + +### 9. 配置 API 环境变量 + +建议使用： + +```bash +cp /opt/domaincheck/domain-api/deploy/multi-region/templates/domaincheck-api.env.example /etc/default/domaincheck-api +``` + +至少改这些： + +```env +WORKER_MODE=linux-systemd +API_HOST=0.0.0.0 +API_PORT=8100 +DOMAIN_ROOT=/opt/domaincheck/domainCheck +NODE_CODE=overseas-control-01 +NODE_REGION=overseas +NODE_ROLE=control +CORS_ORIGINS=http://127.0.0.1:3201,http://localhost:3201,http://你的服务器IP:3201 +SYNC_PUSH_ENABLED=false +``` + +如果你已经固定域名，建议直接改成： + +```env +CORS_ORIGINS=https://admin.domain,http://127.0.0.1:3201,http://localhost:3201 +``` + +### 10. 修正权限 + +```bash +mkdir -p /opt/domaincheck/domain-api/runtime +touch /opt/domaincheck/domainCheck/detect_worker.log +chown -R www:www /opt/domaincheck/domain-api/runtime +chown www:www /opt/domaincheck/domainCheck/detect_worker.log +chmod 664 /opt/domaincheck/domainCheck/detect_worker.log +``` + +### 11. 启动服务 + +```bash +systemctl daemon-reload +systemctl enable domaincheck-api +systemctl enable domaincheck-worker +systemctl restart domaincheck-api +systemctl restart domaincheck-worker +``` + +### 12. 检查服务状态 + +```bash +systemctl status domaincheck-api --no-pager -l +systemctl status domaincheck-worker --no-pager -l +``` + +## 五、第一阶段验收 + +### 1. 接口检查 + +```bash +curl http://127.0.0.1:8100/health +curl http://127.0.0.1:8100/api/v1/runtime/preflight +curl http://127.0.0.1:8100/api/v1/runtime/status +curl http://127.0.0.1:8100/api/v1/runtime/readiness +``` + +期望： + +- `/health.status=ok` +- `/health.worker_mode=linux-systemd` +- `/runtime/preflight.ok=true` +- `/runtime/status.worker.running=true` +- `/runtime/readiness.status` 至少不是 `blocking` + +### 2. smoke test + +```bash +cd /opt/domaincheck/domain-api/deploy/linux +python3 smoke_test.py --base-url http://127.0.0.1:8100 +``` + +如果要连 Web 一起检查： + +```bash +python3 smoke_test.py --base-url http://127.0.0.1:8100 --web-url http://127.0.0.1:3201 +``` + +期望： + +- `ok=true` + +### 3. 后台页面检查 + +浏览器打开： + +```text +https://admin.domain +``` + +至少检查： + +- 登录 +- 运行中心 +- 系统设置 +- 域名筛选 +- 导入 +- 导出 + +## 六、第二阶段：国外机器模拟多机联调 + +如果你暂时还不能把代码部署到大陆机器，就先在国外机器做这一步。 + +### 1. 一键多机演练 + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/rehearse_multi_region.sh http://127.0.0.1:8100 +``` + +它会自动模拟： + +- 一个大陆 controller +- 两个大陆 worker + +并自动检查： + +- `runtime/readiness` +- `runtime/cluster` +- online control / online worker 数量 +- 节点状态是否符合预期 + +### 2. 通过标准 + +如果输出里出现： + +```text +rehearsal passed +``` + +说明当前这台国外机器上的多机模拟联调已通过，可以继续部署到其他机器。 + +### 3. 如果想手工模拟 + +也可以单独跑： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/simulate_multi_region.sh http://127.0.0.1:8100 +``` + +停止时按： + +```text +Ctrl+C +``` + +### 4. 清理旧节点残影 + +如果之前测试过多轮，集群里可能残留老节点： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 --dry-run +bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 +``` + +如果只删某个旧节点： + +```bash +bash deploy/multi-region/prune_cluster_nodes.sh --node-code mainland-worker-01 +``` + +## 七、第三阶段：部署到其他机器 + +等国外单机和模拟多机都通过后，就可以把同样的代码与脚本部署到其他机器。 + +### A. 新海外控制机 + +在目标机执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_overseas.sh /opt/domaincheck +``` + +然后完善： + +- `/etc/default/domaincheck-api` +- Nginx +- PostgreSQL +- Redis + +最后检查： + +```bash +bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100 +``` + +### B. 大陆 controller + +在目标机执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller +``` + +然后检查： + +```bash +bash deploy/multi-region/check_mainland_controller.sh +``` + +关键配置是： + +```env +NODE_CODE=mainland-controller-01 +NODE_REGION=mainland +NODE_ROLE=control +SYNC_PUSH_ENABLED=true +SYNC_SOURCE_REGION=mainland +SYNC_TARGET_REGION=overseas +SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1 +SYNC_SHARED_TOKEN=你自己的共享令牌 +``` + +### C. 大陆 worker + +在目标机执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker +``` + +关键配置是： + +```env +NODE_CODE=mainland-worker-01 +NODE_REGION=mainland +NODE_ROLE=worker +SYNC_PUSH_ENABLED=true +SYNC_SOURCE_REGION=mainland +SYNC_TARGET_REGION=overseas +SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1 +``` + +## 八、部署到其他机器后的联调顺序 + +建议按这个顺序，不容易乱： + +1. 先让海外控制机稳定 +2. 再接大陆 controller +3. 再接第一台大陆 worker +4. 再接第二台、第三台 worker + +每接一台都执行： + +```bash +curl http://海外控制机:8100/api/v1/runtime/readiness +curl http://海外控制机:8100/api/v1/runtime/cluster +curl http://海外控制机:8100/api/v1/runtime/sync-summary +``` + +如果海外 API 已经挂到正式域名，建议直接写成： + +```bash +curl https://api.domain.com/api/v1/runtime/readiness +curl https://api.domain.com/api/v1/runtime/cluster +curl https://api.domain.com/api/v1/runtime/sync-summary +``` + +### 联调通过标准 + +- `runtime/cluster` 能看到新节点 +- `last_heartbeat_at` 持续刷新 +- 大陆 controller 为 `role=control` +- 大陆 worker 为 `role=worker` +- 执行任务时 worker 状态能变成 `busy` +- `runtime/readiness` 至少不是 `blocking` +- `runtime/sync-summary` 能看到 `detect_result_batches` + +## 九、上线前最后检查 + +按这份文档部署完成后，再回看： + +- [14_domainCheck_正式上线前最终检查单.md](/www/wwwroot/getDomain/docs/14_domainCheck_正式上线前最终检查单.md:1) + +重点保留： + +- `systemctl status` +- `journalctl` +- `/health` +- `/runtime/preflight` +- `/runtime/readiness` +- `/runtime/sync-summary` +- `smoke test` +- 诊断包 + +## 十、常见问题 + +### 1. API 启动了，但接口 500 + +优先检查有没有先执行： + +```bash +cd /opt/domaincheck/domainCheck +python3 init_database.py +``` + +### 2. Worker 一直重启 + +优先检查： + +- `/opt/domaincheck/domainCheck/detect_worker.log` 权限 +- `/opt/domaincheck/domain-api/runtime/` 权限 +- `domainCheck/.env` 数据库和 Redis 配置 + +### 3. readiness 一直是 `attention` + +优先检查： + +- 集群里是否残留旧离线节点 +- 是否还没接入大陆 controller +- 是否还没有在线 worker + +必要时先清理旧节点： + +```bash +bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 +``` + +### 4. 模拟多机通过了，真实机器还没接上 + +这是正常的。 + +模拟多机的意义是： + +- 验证代码、脚本、页面、状态口径一致 +- 不代表真实大陆网络、代理、同步链路已经完成 + +真实机器接入时，重点要再看： + +- 节点心跳 +- 同步目标地址 +- 共享 token +- 真实 Redis / PostgreSQL 连接 + +## 十一、最短执行版本 + +如果你只想看最短版，可以照这个跑： + +### 国外单机先跑通 + +```bash +cd /opt/domaincheck/domainCheck +python3.11 -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +python3 init_database.py + +cd /opt/domaincheck/domain-api +/opt/domaincheck/domainCheck/.venv/bin/pip install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart + +systemctl daemon-reload +systemctl enable domaincheck-api domaincheck-worker +systemctl restart domaincheck-api domaincheck-worker + +curl http://127.0.0.1:8100/health +curl http://127.0.0.1:8100/api/v1/runtime/preflight +curl http://127.0.0.1:8100/api/v1/runtime/readiness +``` + +### 再跑多机模拟验收 + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/rehearse_multi_region.sh http://127.0.0.1:8100 +``` + +### 通过后再上其他机器 + +```bash +bash deploy/multi-region/bootstrap_overseas.sh /opt/domaincheck +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker +``` + +## 十二、最终结论 + +当前最稳的推进方式是： + +1. 先在国外机器完成单机部署 +2. 再在国外机器完成多机模拟演练 +3. 演练通过后，再复制到其他机器 +4. 最后再做真实跨地域联调 + +一句话总结： + +> 先把单机跑稳，再把多机脚本跑通，最后再扩机器；每一步都有现成脚本，不靠现场猜。 diff --git a/docs/18_domainCheck_CentOS9一键复制部署与更新文档.md b/docs/18_domainCheck_CentOS9一键复制部署与更新文档.md new file mode 100644 index 0000000..1b6e682 --- /dev/null +++ b/docs/18_domainCheck_CentOS9一键复制部署与更新文档.md @@ -0,0 +1,1015 @@ +# 18 domainCheck CentOS9 一键复制部署与更新文档 + +## 一、文档用途 + +这份文档只做一件事： + +> 给你一套可以直接复制执行的命令，完成 `1 台国外机器 + 2 台国内机器` 的傻瓜式部署和后续更新。 + +统一前提： + +- 系统：`CentOS Stream 9` +- 代码管理：统一 `git` +- API 域名：`api.domain.com` +- 后台域名：`admin.domain` +- 分支：默认 `main` + +目标机器： + +- 国外机器 `1` 台 + - 角色：`overseas-control` +- 国内机器 `2` 台 + - `mainland-controller-01` + - `mainland-worker-01` + +一句话部署口径： + +- 国外机器：`宝塔管理 Nginx / PostgreSQL / Redis + systemd 管理项目服务` +- 国内机器：`纯 systemd 执行节点，不装宝塔` + +## 二、先改这里 + +正式执行本文命令前，先把下面这些变量替换成你自己的真实值。 + +```text +GIT_REPO_URL=你的仓库地址 +GIT_BRANCH=main + +API_DOMAIN=api.domain.com +ADMIN_DOMAIN=admin.domain + +DB_NAME=domain +DB_USER=domainuser +DB_PASSWORD=请改成你的强密码 + +SYNC_SHARED_TOKEN=请改成你的共享令牌 + +OVERSEAS_NODE_CODE=overseas-control-01 +MAINLAND_CONTROLLER_NODE_CODE=mainland-controller-01 +MAINLAND_WORKER_NODE_CODE=mainland-worker-01 + +MAINLAND_CONTROLLER_IP=国内controller机器IP + +NODE_VERSION=20 +``` + +最少必须先替换这 6 个： + +- `GIT_REPO_URL` +- `DB_PASSWORD` +- `SYNC_SHARED_TOKEN` +- `API_DOMAIN` +- `ADMIN_DOMAIN` +- `MAINLAND_CONTROLLER_IP` + +## 三、统一约定 + +### 1. 仓库目录 + +```text +/www/wwwroot/getDomain +``` + +### 2. 运行目录 + +```text +/opt/domaincheck +``` + +### 3. 统一更新命令 + +后续所有机器代码更新统一执行： + +```bash +cd /www/wwwroot/getDomain +git fetch --all +git checkout main +git pull --ff-only origin main +``` + +### 4. 统一软链接 + +所有机器都统一建立： + +```bash +mkdir -p /opt/domaincheck +ln -sfn /www/wwwroot/getDomain/domain-api /opt/domaincheck/domain-api +ln -sfn /www/wwwroot/getDomain/domain-web /opt/domaincheck/domain-web +ln -sfn /www/wwwroot/getDomain/domainCheck /opt/domaincheck/domainCheck +``` + +### 5. 并发配置约定 + +- 默认检测并发由后台系统设置里的 `默认检测线程数` 控制 +- 若某台机器需要单独并发，可在后台系统设置里增加 `节点独立线程覆盖` +- 规则是： + - 节点已单独配置：走该节点自己的线程数 + - 节点未单独配置：回退到默认线程数 +- 节点编码必须和该机器 `/etc/default/domaincheck-worker` 里的 `NODE_CODE` 一致 +- 配置会同时写入： + - Redis：`domain_tool:thread_count` + - Redis：`domain_tool:node_thread_counts` + - 本地文件：`thread_count.json` + - 本地文件：`node_thread_counts.json` + +### 6. 宝塔与 `nvm` 口径 + +- 国外主控机： + - 你如果已经长期用宝塔管理 `PostgreSQL`、`Redis`、`Nginx`，可以继续用 + - 但 `domain-api`、`domaincheck-worker`、`domaincheck-sync-agent` 仍建议走 `systemd` +- 国内机器： + - 不建议为了纯 `worker` 节点额外安装宝塔 + - 原因不是不能用，而是会多一层面板进程、端口、权限和故障面 + - 国内机器的最佳形态是： + - 只装 Python 运行环境 + - 只装项目代码 + - 只跑 `domaincheck-worker` + - 若是 controller，再加 `domaincheck-sync-agent` +- Node： + - 继续用 `nvm` 没问题，不需要改成系统全局安装 + - 建议统一到 `Node 20` + - 生产构建时只要保证执行构建命令前已经 `nvm use 20` + +### 7. 你实际应该怎么选 + +- 如果国外主控机已经是宝塔机： + - 继续沿用宝塔即可 + - 宝塔负责： + - `Nginx` + - `PostgreSQL` + - `Redis` + - SSL 证书 + - 项目自身服务继续交给： + - `domaincheck-api.service` + - `domaincheck-worker.service` +- 如果国内机器只是跑检测： + - 不要装宝塔 + - 直接按本文的纯命令部署 + - 这样最轻、最稳、最容易横向复制 + +## 四、最短复制块 + +这一节给你三段最短命令。 + +用法： + +- 急着部署时，先用这里 +- 想看每一步解释，再看后面的完整章节 + +### 1. 国外主控机最短复制块 + +先把下面 8 个值替换掉再执行： + +```text +GIT_REPO_URL +GIT_BRANCH +API_DOMAIN +ADMIN_DOMAIN +DB_PASSWORD +SYNC_SHARED_TOKEN +OVERSEAS_NODE_CODE +NODE_VERSION +``` + +也可以直接改脚本头部变量后执行： + +```bash +cp /www/wwwroot/getDomain/domain-api/deploy/multi-region/install_overseas_quick.conf.example \ + /www/wwwroot/getDomain/domain-api/deploy/multi-region/install_overseas_quick.conf + +# 先编辑 install_overseas_quick.conf +bash /www/wwwroot/getDomain/domain-api/deploy/multi-region/install_overseas_quick.sh +``` + +```bash +mkdir -p /www/wwwroot +cd /www/wwwroot +git clone GIT_REPO_URL getDomain || true +cd getDomain +git fetch --all +git checkout GIT_BRANCH +git pull --ff-only origin GIT_BRANCH + +mkdir -p /opt/domaincheck +ln -sfn /www/wwwroot/getDomain/domain-api /opt/domaincheck/domain-api +ln -sfn /www/wwwroot/getDomain/domain-web /opt/domaincheck/domain-web +ln -sfn /www/wwwroot/getDomain/domainCheck /opt/domaincheck/domainCheck + +cd /opt/domaincheck/domainCheck +python3.11 -m venv .venv || true +source .venv/bin/activate +pip install --upgrade pip +pip install -r requirements.txt +/opt/domaincheck/domainCheck/.venv/bin/pip install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart + +cat >/opt/domaincheck/domainCheck/.env </opt/domaincheck/domain-web/.env.production </opt/domaincheck/domainCheck/.env </etc/default/domaincheck-worker </opt/domaincheck/domainCheck/.env </etc/default/domaincheck-worker </opt/domaincheck/domainCheck/.env <<'EOF' +DB_HOST=127.0.0.1 +DB_PORT=5432 +DB_DATABASE=domain +DB_USER=domainuser +DB_PASSWORD=请改成你的强密码 + +REDIS_HOST=127.0.0.1 +REDIS_PORT=6379 +REDIS_PASSWORD= +REDIS_DB=0 +EOF +``` + +### 5. 初始化数据库结构 + +```bash +cd /opt/domaincheck/domainCheck +/opt/domaincheck/domainCheck/.venv/bin/python init_database.py +``` + +### 6. 配置前端生产环境 + +```bash +cat >/opt/domaincheck/domain-web/.env.production <<'EOF' +VITE_API_BASE_URL=https://api.domain.com/api/v1 +EOF +``` + +### 7. 用 `nvm` 构建前端 + +如果你平时就是 `nvm`，建议直接固定： + +```bash +export NVM_DIR="$HOME/.nvm" +[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" +nvm install 20 +nvm use 20 +node -v +``` + +然后再构建： + +```bash +cd /opt/domaincheck/domain-web +npm install +npm run build +``` + +### 8. 配置 API 环境变量 + +```bash +cp /opt/domaincheck/domain-api/deploy/multi-region/templates/domaincheck-api.env.example /etc/default/domaincheck-api +sed -i 's#^DOMAIN_ROOT=.*#DOMAIN_ROOT=/opt/domaincheck/domainCheck#' /etc/default/domaincheck-api +sed -i 's#^API_HOST=.*#API_HOST=0.0.0.0#' /etc/default/domaincheck-api +sed -i 's#^API_PORT=.*#API_PORT=8100#' /etc/default/domaincheck-api +sed -i 's#^NODE_CODE=.*#NODE_CODE=overseas-control-01#' /etc/default/domaincheck-api +sed -i 's#^NODE_REGION=.*#NODE_REGION=overseas#' /etc/default/domaincheck-api +sed -i 's#^NODE_ROLE=.*#NODE_ROLE=control#' /etc/default/domaincheck-api +sed -i 's#^SYNC_PUSH_ENABLED=.*#SYNC_PUSH_ENABLED=false#' /etc/default/domaincheck-api +sed -i 's#^CORS_ORIGINS=.*#CORS_ORIGINS=https://admin.domain,http://127.0.0.1:3201,http://localhost:3201#' /etc/default/domaincheck-api +``` + +### 9. 安装 systemd + +```bash +cp /opt/domaincheck/domain-api/deploy/systemd/domain-api.service /etc/systemd/system/domaincheck-api.service +cp /opt/domaincheck/domain-api/deploy/systemd/domain-worker.service /etc/systemd/system/domaincheck-worker.service +``` + +### 10. 修正权限 + +```bash +mkdir -p /opt/domaincheck/domain-api/runtime +touch /opt/domaincheck/domainCheck/detect_worker.log +chown -R www:www /opt/domaincheck/domain-api/runtime +chown www:www /opt/domaincheck/domainCheck/detect_worker.log +chmod 664 /opt/domaincheck/domainCheck/detect_worker.log +``` + +### 11. 启动国外服务 + +```bash +systemctl daemon-reload +systemctl enable domaincheck-api +systemctl enable domaincheck-worker +systemctl restart domaincheck-api +systemctl restart domaincheck-worker +``` + +### 12. 配置 Nginx + +如果你已经用宝塔建好了： + +- `admin.domain` +- `api.domain.com` + +并且已经在宝塔里配好了反代和静态站点，那么这一节可以不执行。 + +如果你不用宝塔，就直接执行下面命令。 + +后台站点： + +```bash +cat >/etc/nginx/conf.d/admin.domain.conf <<'EOF' +server { + listen 80; + server_name admin.domain; + + root /opt/domaincheck/domain-web/dist; + index index.html; + + location / { + try_files $uri $uri/ /index.html; + } +} +EOF +``` + +API 站点： + +```bash +cat >/etc/nginx/conf.d/api.domain.com.conf <<'EOF' +server { + listen 80; + server_name api.domain.com; + + location / { + proxy_pass http://127.0.0.1:8100; + proxy_http_version 1.1; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +EOF +``` + +启动 Nginx： + +```bash +nginx -t +systemctl enable nginx +systemctl restart nginx +``` + +### 13. 国外机器验收 + +```bash +curl http://127.0.0.1:8100/health +curl http://127.0.0.1:8100/api/v1/runtime/preflight +curl http://127.0.0.1:8100/api/v1/runtime/readiness + +cd /opt/domaincheck/domain-api/deploy/linux +python3 smoke_test.py --base-url http://127.0.0.1:8100 +``` + +## 七、国内 controller 一键部署 + +这一节默认你是： + +- 国内 controller +- 不装宝塔 +- 只跑 `domaincheck-worker + domaincheck-sync-agent` +- 本机承载 Redis / PostgreSQL 控制面 + +### 1. 先准备 Redis 和 PostgreSQL + +```bash +postgresql-setup --initdb +systemctl enable postgresql +systemctl start postgresql + +systemctl enable redis +systemctl start redis +``` + +创建数据库： + +```bash +sudo -u postgres psql <<'SQL' +CREATE USER domainuser WITH PASSWORD '请改成你的强密码'; +CREATE DATABASE domain OWNER domainuser; +GRANT ALL PRIVILEGES ON DATABASE domain TO domainuser; +SQL +``` + +### 2. 配置 `domainCheck/.env` + +```bash +cat >/opt/domaincheck/domainCheck/.env <<'EOF' +DB_HOST=127.0.0.1 +DB_PORT=5432 +DB_DATABASE=domain +DB_USER=domainuser +DB_PASSWORD=请改成你的强密码 + +REDIS_HOST=127.0.0.1 +REDIS_PORT=6379 +REDIS_PASSWORD= +REDIS_DB=0 +EOF +``` + +### 3. 执行 bootstrap + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller +``` + +### 4. 配置 `/etc/default/domaincheck-worker` + +```bash +cat >/etc/default/domaincheck-worker <<'EOF' +WORKER_MODE=linux-systemd +QT_QPA_PLATFORM=offscreen +NODE_CODE=mainland-controller-01 +NODE_REGION=mainland +NODE_ROLE=control + +DB_HOST=127.0.0.1 +DB_PORT=5432 +DB_DATABASE=domain +DB_USER=domainuser +DB_PASSWORD=请改成你的强密码 + +REDIS_HOST=127.0.0.1 +REDIS_PORT=6379 +REDIS_PASSWORD= +REDIS_DB=0 + +SYNC_PUSH_ENABLED=true +SYNC_SOURCE_REGION=mainland +SYNC_TARGET_REGION=overseas +SYNC_TARGET_API_BASE_URL=https://api.domain.com/api/v1 +SYNC_SHARED_TOKEN=请改成你的共享令牌 +SYNC_BATCH_SIZE=200 +SYNC_POLL_INTERVAL_SECONDS=30 +EOF +``` + +### 5. 启动 controller 服务 + +```bash +systemctl daemon-reload +systemctl enable domaincheck-worker +systemctl enable domaincheck-sync-agent +systemctl restart domaincheck-worker +systemctl restart domaincheck-sync-agent +``` + +### 6. 本机检查 + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_mainland_controller.sh +``` + +## 八、国内 worker 一键部署 + +这一节默认你是： + +- 国内纯 worker 节点 +- 不装宝塔 +- 不承载后台 +- 不承载 API +- 不承载 Nginx + +### 1. 配置 `domainCheck/.env` + +如果它连接 controller 的同一套 Redis / PostgreSQL，就改成 controller 地址： + +```bash +cat >/opt/domaincheck/domainCheck/.env <<'EOF' +DB_HOST=国内controller机器IP +DB_PORT=5432 +DB_DATABASE=domain +DB_USER=domainuser +DB_PASSWORD=请改成你的强密码 + +REDIS_HOST=国内controller机器IP +REDIS_PORT=6379 +REDIS_PASSWORD= +REDIS_DB=0 +EOF +``` + +### 2. 执行 bootstrap + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker +``` + +### 3. 配置 `/etc/default/domaincheck-worker` + +```bash +cat >/etc/default/domaincheck-worker <<'EOF' +WORKER_MODE=linux-systemd +QT_QPA_PLATFORM=offscreen +NODE_CODE=mainland-worker-01 +NODE_REGION=mainland +NODE_ROLE=worker + +DB_HOST=国内controller机器IP +DB_PORT=5432 +DB_DATABASE=domain +DB_USER=domainuser +DB_PASSWORD=请改成你的强密码 + +REDIS_HOST=国内controller机器IP +REDIS_PORT=6379 +REDIS_PASSWORD= +REDIS_DB=0 + +SYNC_PUSH_ENABLED=true +SYNC_SOURCE_REGION=mainland +SYNC_TARGET_REGION=overseas +SYNC_TARGET_API_BASE_URL=https://api.domain.com/api/v1 +SYNC_SHARED_TOKEN=请改成你的共享令牌 +SYNC_BATCH_SIZE=200 +SYNC_POLL_INTERVAL_SECONDS=30 +EOF +``` + +### 4. 启动 worker + +```bash +systemctl daemon-reload +systemctl enable domaincheck-worker +systemctl restart domaincheck-worker +``` + +## 九、统一更新命令 + +以后你更新代码，三台机器统一执行： + +```bash +cd /www/wwwroot/getDomain +git fetch --all +git checkout main +git pull --ff-only origin main +``` + +然后： + +### 国外机器 + +如果国外机器前端构建使用 `nvm`，更新时建议固定这么跑： + +```bash +export NVM_DIR="$HOME/.nvm" +[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" +nvm use 20 + +cd /opt/domaincheck/domain-web +npm install +npm run build + +systemctl restart domaincheck-api +systemctl restart domaincheck-worker +systemctl restart nginx +``` + +### 国内 controller + +```bash +systemctl restart domaincheck-worker +systemctl restart domaincheck-sync-agent +``` + +### 国内 worker + +```bash +systemctl restart domaincheck-worker +``` + +## 十、最终联调检查 + +在国外机器执行： + +```bash +curl https://api.domain.com/api/v1/runtime/readiness +curl https://api.domain.com/api/v1/runtime/cluster +curl https://api.domain.com/api/v1/runtime/sync-summary +``` + +期望： + +- `runtime/readiness.status` 不是 `blocking` +- `runtime/cluster` 能看到： + - `overseas-control-01` + - `mainland-controller-01` + - `mainland-worker-01` +- `runtime/sync-summary` 能看到 `detect_result_batches` + +## 十一、如果当前还不能上国内机器 + +先在国外机器演练： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/rehearse_multi_region.sh http://127.0.0.1:8100 +``` + +如果它输出： + +```text +rehearsal passed +``` + +说明： + +- 多机脚本没问题 +- readiness 判断没问题 +- cluster 视图没问题 +- 可以开始把同样方式复制到其他机器 + +## 十二、最推荐的真实落地顺序 + +如果按你的使用习惯，我建议你就按下面走： + +### 方案 A：最符合你现在的习惯 + +- 国外主控机： + - 宝塔管理 `Nginx / PostgreSQL / Redis / SSL` + - `systemd` 管理 `domaincheck-api / domaincheck-worker` + - 前端构建使用 `nvm` +- 国内 controller： + - 不装宝塔 + - 跑 `PostgreSQL / Redis / domaincheck-worker / domaincheck-sync-agent` +- 国内 worker： + - 不装宝塔 + - 只跑 `domaincheck-worker` + +### 方案 B：如果以后你想进一步简化 + +- 国外机继续保持不变 +- 国内 controller 也可以后续去掉本地 PostgreSQL，只保留： + - `Redis` + - `domaincheck-worker` + - `domaincheck-sync-agent` +- 国内 worker 全部连 controller 的 Redis 和数据库 + +### 当前阶段我建议你先不要改的 + +- 不要为了统一而让国内 worker 也装宝塔 +- 不要为了“更标准”把 `nvm` 改成全局 Node +- 不要让项目服务交给宝塔守护进程 + +## 十三、最终建议 + +你现在最稳的部署顺序就是： + +1. 先部署国外机器 +2. 国外机器 smoke test 通过 +3. 国外机器多机模拟演练通过 +4. 再部署国内 controller +5. 再部署国内 worker +6. 最后看 `api.domain.com` 的 readiness / cluster / sync-summary + +一句话总结： + +> 先把国外主控机跑稳，再用同样脚本扩到 2 台国内机器；代码更新统一走 git，部署执行直接复制这份文档里的命令。 diff --git a/domain-api/app/api/routes/detect.py b/domain-api/app/api/routes/detect.py index 24f46a6..6d6ddfc 100644 --- a/domain-api/app/api/routes/detect.py +++ b/domain-api/app/api/routes/detect.py @@ -1,25 +1,151 @@ +from uuid import uuid4 + from fastapi import APIRouter from app.schemas.common import ApiResponse +from app.services.detect_job_service import ( + append_detect_job_event, + create_detect_job_if_needed, + get_active_detect_job_summary, + get_detect_job_summary, + get_detect_queue_health, + list_detect_jobs, +) from app.services.detect_service import get_detect_status -from app.services.worker_control_service import start_worker, stop_worker +from app.services.detect_run_service import create_detect_run_snapshot, finalize_detect_run, mark_detect_run_stopping +from app.services.settings_service import get_settings_payload, resolve_thread_count +from app.services.worker_control_service import send_worker_command, start_worker router = APIRouter(tags=["detect"]) +def _build_settings_summary(settings_payload: dict) -> dict: + thread_count_resolution = resolve_thread_count(settings_payload=settings_payload) + return { + "thread_count": int(thread_count_resolution["effective_thread_count"]), + "thread_count_default": int(thread_count_resolution["default_thread_count"]), + "thread_count_source": str(thread_count_resolution["source"]), + "thread_count_override": thread_count_resolution["override_thread_count"], + "thread_count_node_code": str(thread_count_resolution["node_code"]), + "proxy_enable": settings_payload["proxy_config"].get("proxy_enable", False), + "allow_direct": settings_payload["proxy_config"].get("allow_direct", False), + "proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])), + } + + @router.get("/detect/status", response_model=ApiResponse) def detect_status() -> ApiResponse: return ApiResponse(data=get_detect_status()) +@router.get("/detect/job/active", response_model=ApiResponse) +def detect_active_job() -> ApiResponse: + return ApiResponse(data=get_active_detect_job_summary(event_limit=50)) + + +@router.get("/detect/jobs", response_model=ApiResponse) +def detect_jobs(limit: int = 20) -> ApiResponse: + return ApiResponse(data={"items": list_detect_jobs(limit=limit), "limit": max(1, min(int(limit or 20), 100))}) + + +@router.get("/detect/queue-summary", response_model=ApiResponse) +def detect_queue_summary(window_minutes: int = 15) -> ApiResponse: + return ApiResponse(data=get_detect_queue_health(window_minutes=window_minutes)) + + +@router.get("/detect/jobs/{job_id}", response_model=ApiResponse) +def detect_job_detail(job_id: int) -> ApiResponse: + data = get_detect_job_summary(job_id, event_limit=100) + if not data: + return ApiResponse(code=1, message="检测任务不存在", data=None) + return ApiResponse(data=data) + + @router.post("/detect/start", response_model=ApiResponse) def start_detect() -> ApiResponse: + job_summary = create_detect_job_if_needed(limit=1000, created_by="api") + if not job_summary: + return ApiResponse( + code=0, + message="当前没有可创建的检测任务", + data={ + "action": "start", + "job": None, + "poll_after_seconds": 2, + "refresh_status": True, + }, + ) + cycle_token = uuid4().hex[:10] + append_detect_job_event( + job_summary["job_id"], + event_type="job_dispatch_requested", + message="控制面已发送检测启动请求", + payload={ + "cycle_token": cycle_token, + "status": job_summary.get("status"), + "items_pending": job_summary.get("items_pending", 0), + "items_claimed": job_summary.get("items_claimed", 0), + "items_running": job_summary.get("items_running", 0), + }, + ) + ok, message = start_worker() + if not ok: + append_detect_job_event( + job_summary["job_id"], + event_type="job_dispatch_failed", + level="error", + message=f"启动 Worker 失败: {message}", + payload={"cycle_token": cycle_token}, + ) + return ApiResponse( + code=1, + message=message, + data={ + "action": "start", + "job": job_summary, + "poll_after_seconds": 2, + "refresh_status": True, + }, + ) + + command_ok, command_message = send_worker_command( + "start_detection", + payload={ + "cycle_token": cycle_token, + "job_id": job_summary["job_id"], + "job_code": job_summary["job_code"], + }, + ) + append_detect_job_event( + job_summary["job_id"], + event_type="job_dispatch_sent" if command_ok else "job_dispatch_rejected", + level="info" if command_ok else "error", + message=command_message, + payload={"cycle_token": cycle_token}, + ) + snapshot = get_detect_status() + settings_payload = get_settings_payload() + settings_summary = _build_settings_summary(settings_payload) + if command_ok: + create_detect_run_snapshot( + message=f"{message}；{command_message}", + runtime={ + "mode": snapshot.get("worker_mode", ""), + "running": snapshot.get("worker_online", False), + "process_count": snapshot.get("worker_process_count", 0), + "latest_start_time": snapshot.get("worker_latest_start_time", ""), + "message": snapshot.get("worker_runtime_message", ""), + }, + progress=snapshot.get("progress", {}), + settings_summary=settings_summary, + ) return ApiResponse( - code=0 if ok else 1, - message=message, + code=0 if command_ok else 1, + message=f"{message}；{command_message}" if command_ok else command_message, data={ "action": "start", + "job": job_summary, "poll_after_seconds": 2, "refresh_status": True, }, @@ -28,7 +154,45 @@ def start_detect() -> ApiResponse: @router.post("/detect/stop", response_model=ApiResponse) def stop_detect() -> ApiResponse: - ok, message = stop_worker() + active_job = get_active_detect_job_summary(event_limit=10) + ok, message = send_worker_command("stop_detection") + if active_job: + append_detect_job_event( + active_job["job_id"], + event_type="job_stop_requested" if ok else "job_stop_request_failed", + level="info" if ok else "error", + message=message, + payload={"cycle_token": active_job.get("current_cycle_token", "")}, + ) + snapshot = get_detect_status() + settings_payload = get_settings_payload() + settings_summary = _build_settings_summary(settings_payload) + mark_detect_run_stopping( + message=message, + runtime={ + "mode": snapshot.get("worker_mode", ""), + "running": snapshot.get("worker_online", False), + "process_count": snapshot.get("worker_process_count", 0), + "latest_start_time": snapshot.get("worker_latest_start_time", ""), + "message": snapshot.get("worker_runtime_message", ""), + }, + progress=snapshot.get("progress", {}), + settings_summary=settings_summary, + ) + if not snapshot.get("worker_online", False): + finalize_detect_run( + message=message, + runtime={ + "mode": snapshot.get("worker_mode", ""), + "running": snapshot.get("worker_online", False), + "process_count": snapshot.get("worker_process_count", 0), + "latest_start_time": snapshot.get("worker_latest_start_time", ""), + "message": snapshot.get("worker_runtime_message", ""), + }, + progress=snapshot.get("progress", {}), + settings_summary=settings_summary, + active_job=active_job, + ) return ApiResponse( code=0 if ok else 1, message=message, diff --git a/domain-api/app/api/routes/domains.py b/domain-api/app/api/routes/domains.py index bf0511a..1ce8b2e 100644 --- a/domain-api/app/api/routes/domains.py +++ b/domain-api/app/api/routes/domains.py @@ -1,7 +1,9 @@ +from __future__ import annotations + from fastapi import APIRouter, Query from app.schemas.common import ApiResponse -from app.services.domains_service import batch_update_domains, domain_filter_options, fetch_domains +from app.services.domains_service import batch_update_domains, domain_filter_options, fetch_domain_detail, fetch_domains router = APIRouter(tags=["domains"]) @@ -24,6 +26,8 @@ def domain_list( beian_year: int | None = Query(default=None), snapshot_year: str | None = Query(default=None), website_url: str | None = Query(default=None), + company_type: str | None = Query(default=None), + source_type: int | None = Query(default=None), backlink_gt_10: bool | None = Query(default=None), ) -> ApiResponse: return ApiResponse( @@ -39,11 +43,21 @@ def domain_list( beian_year=beian_year, snapshot_year=snapshot_year, website_url=website_url, + company_type=company_type, + source_type=source_type, backlink_gt_10=backlink_gt_10, ) ) +@router.get("/domains/{domain_id}", response_model=ApiResponse) +def domain_detail(domain_id: int) -> ApiResponse: + data = fetch_domain_detail(domain_id) + if not data: + return ApiResponse(code=1, message="域名不存在", data=None) + return ApiResponse(data=data) + + @router.post("/domains/batch-update", response_model=ApiResponse) def domain_batch_update(payload: dict) -> ApiResponse: result = batch_update_domains(payload.get("domain_ids", []), payload.get("updates", {})) diff --git a/domain-api/app/api/routes/imports.py b/domain-api/app/api/routes/imports.py index f3f1e66..7d8957b 100644 --- a/domain-api/app/api/routes/imports.py +++ b/domain-api/app/api/routes/imports.py @@ -1,4 +1,4 @@ -from fastapi import APIRouter, File, UploadFile +from fastapi import APIRouter, File, Form, UploadFile from app.schemas.common import ApiResponse from app.services.import_task_service import create_import_task, list_import_tasks, retry_import_task @@ -18,12 +18,24 @@ def import_tasks() -> ApiResponse: @router.post("/imports/upload", response_model=ApiResponse) -async def upload_import(file: UploadFile = File(...)) -> ApiResponse: +async def upload_import( + file: UploadFile = File(...), + source_type: int = Form(default=7), +) -> ApiResponse: content = await file.read() - task = create_import_task(content, file.filename or "domains.txt") + task = create_import_task(content, file.filename or "domains.txt", source_type=source_type) return ApiResponse(message="导入任务已创建", data=task) +@router.post("/imports/text", response_model=ApiResponse) +def upload_import_text(payload: dict) -> ApiResponse: + text = str(payload.get("text") or "") + filename = str(payload.get("filename") or "manual_domains.txt") + source_type = int(payload.get("source_type") or 6) + task = create_import_task(text.encode("utf-8"), filename, source_type=source_type) + return ApiResponse(message="文本导入任务已创建", data=task) + + @router.post("/imports/tasks/{task_id}/retry", response_model=ApiResponse) def import_task_retry(task_id: str) -> ApiResponse: task = retry_import_task(task_id) diff --git a/domain-api/app/api/routes/juming.py b/domain-api/app/api/routes/juming.py new file mode 100644 index 0000000..6d570f6 --- /dev/null +++ b/domain-api/app/api/routes/juming.py @@ -0,0 +1,93 @@ +from fastapi import APIRouter, File, HTTPException, UploadFile + +from app.schemas.common import ApiResponse +from app.services.juming_service import ( + get_juming_preferences, + get_juming_status, + login_jucha_with_juming_cookie, + login_juming, + update_juming_preferences, + upload_juming_cookie, +) +from app.services.juming_task_service import create_juming_task, list_juming_tasks, request_stop_juming_task, retry_juming_task + +router = APIRouter(tags=["juming"]) + + +@router.get("/juming/status", response_model=ApiResponse) +def juming_status() -> ApiResponse: + return ApiResponse(data=get_juming_status()) + + +@router.get("/juming/preferences", response_model=ApiResponse) +def juming_preferences() -> ApiResponse: + return ApiResponse(data=get_juming_preferences()) + + +@router.put("/juming/preferences", response_model=ApiResponse) +def update_juming_preferences_route(payload: dict) -> ApiResponse: + try: + data = update_juming_preferences(payload) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return ApiResponse(message="聚名采集偏好已保存", data=data) + + +@router.post("/juming/crawl", response_model=ApiResponse) +def juming_crawl(payload: dict) -> ApiResponse: + try: + data = create_juming_task(payload) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return ApiResponse(message="聚名采集任务已创建", data=data) + + +@router.get("/juming/tasks", response_model=ApiResponse) +def juming_tasks() -> ApiResponse: + return ApiResponse(data=list_juming_tasks()) + + +@router.post("/juming/tasks/{task_id}/retry", response_model=ApiResponse) +def juming_task_retry(task_id: str) -> ApiResponse: + try: + data = retry_juming_task(task_id) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return ApiResponse(message="聚名采集任务已重新加入队列", data=data) + + +@router.post("/juming/tasks/{task_id}/stop", response_model=ApiResponse) +def juming_task_stop(task_id: str) -> ApiResponse: + try: + data = request_stop_juming_task(task_id) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return ApiResponse(message="聚名采集任务已发送停止请求", data=data) + + +@router.post("/juming/login", response_model=ApiResponse) +def juming_login(payload: dict) -> ApiResponse: + try: + data = login_juming(str(payload.get("email") or ""), str(payload.get("password") or "")) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return ApiResponse(message="聚名登录成功", data=data) + + +@router.post("/juming/jucha-login", response_model=ApiResponse) +def juming_jucha_login() -> ApiResponse: + try: + data = login_jucha_with_juming_cookie() + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return ApiResponse(message="聚查联名登录成功", data=data) + + +@router.post("/juming/cookie/upload", response_model=ApiResponse) +async def juming_cookie_upload(file: UploadFile = File(...)) -> ApiResponse: + content = await file.read() + try: + data = upload_juming_cookie(file.filename or "juming_cookies.pkl", content) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return ApiResponse(message="聚名 Cookie 已上传", data=data) diff --git a/domain-api/app/api/routes/runtime.py b/domain-api/app/api/routes/runtime.py index 236f640..ecb7c73 100644 --- a/domain-api/app/api/routes/runtime.py +++ b/domain-api/app/api/routes/runtime.py @@ -1,8 +1,13 @@ -from fastapi import APIRouter +from typing import Optional + +from fastapi import APIRouter, Header from app.schemas.common import ApiResponse +from app.services.cluster_runtime_service import get_cluster_snapshot from app.services.runtime_control_service import runtime_action from app.services.runtime_status_service import get_runtime_preflight, get_runtime_status +from app.services.sync_push_service import ingest_runtime_projection +from app.services.sync_record_service import get_sync_summary, list_sync_records router = APIRouter(tags=["runtime"]) @@ -12,11 +17,37 @@ def runtime_status() -> ApiResponse: return ApiResponse(data=get_runtime_status()) +@router.get("/runtime/readiness", response_model=ApiResponse) +def runtime_readiness() -> ApiResponse: + return ApiResponse(data=(get_runtime_status().get("readiness") or {})) + + @router.get("/runtime/preflight", response_model=ApiResponse) def runtime_preflight() -> ApiResponse: return ApiResponse(data=get_runtime_preflight()) +@router.get("/runtime/cluster", response_model=ApiResponse) +def runtime_cluster() -> ApiResponse: + return ApiResponse(data=get_cluster_snapshot()) + + +@router.get("/runtime/sync-summary", response_model=ApiResponse) +def runtime_sync_summary() -> ApiResponse: + return ApiResponse(data=get_sync_summary()) + + +@router.get("/runtime/sync-records", response_model=ApiResponse) +def runtime_sync_records(limit: int = 20) -> ApiResponse: + return ApiResponse(data={"records": list_sync_records(limit=limit)}) + + +@router.post("/runtime/sync-ingest", response_model=ApiResponse) +def runtime_sync_ingest(payload: dict, x_domaincheck_sync_token: Optional[str] = Header(default=None)) -> ApiResponse: + ok, message, data = ingest_runtime_projection(payload, shared_token=x_domaincheck_sync_token) + return ApiResponse(code=0 if ok else 1, message=message, data=data) + + @router.post("/runtime/actions/{action}", response_model=ApiResponse) def runtime_action_trigger(action: str) -> ApiResponse: ok, message, data = runtime_action(action) diff --git a/domain-api/app/api/routes/sensitive_words.py b/domain-api/app/api/routes/sensitive_words.py new file mode 100644 index 0000000..7737400 --- /dev/null +++ b/domain-api/app/api/routes/sensitive_words.py @@ -0,0 +1,17 @@ +from fastapi import APIRouter + +from app.schemas.common import ApiResponse +from app.services.sensitive_words_service import get_sensitive_words_payload, save_sensitive_words_payload + +router = APIRouter(tags=["sensitive_words"]) + + +@router.get("/sensitive-words", response_model=ApiResponse) +def get_sensitive_words() -> ApiResponse: + return ApiResponse(data=get_sensitive_words_payload()) + + +@router.put("/sensitive-words", response_model=ApiResponse) +def update_sensitive_words(payload: dict) -> ApiResponse: + data = save_sensitive_words_payload(payload) + return ApiResponse(message=f"已保存 {data['total']} 个敏感词", data=data) diff --git a/domain-api/app/api/routes/settings.py b/domain-api/app/api/routes/settings.py index 1397980..cbd5430 100644 --- a/domain-api/app/api/routes/settings.py +++ b/domain-api/app/api/routes/settings.py @@ -5,12 +5,15 @@ from fastapi.responses import FileResponse from app.core.files import settings_backup_root from app.schemas.common import ApiResponse +from app.services.juziseo_service import get_juziseo_status, login_juziseo from app.services.settings_service import ( backup_current_settings, export_settings_snapshot, + get_credentials_payload, get_settings_payload, import_settings_snapshot, list_settings_backups, + update_credentials_payload, update_settings_payload, validate_settings_payload, ) @@ -61,6 +64,30 @@ def get_settings_backups() -> ApiResponse: return ApiResponse(data=list_settings_backups()) +@router.get("/settings/credentials", response_model=ApiResponse) +def get_settings_credentials() -> ApiResponse: + return ApiResponse(data=get_credentials_payload()) + + +@router.put("/settings/credentials", response_model=ApiResponse) +def update_settings_credentials(payload: dict) -> ApiResponse: + return ApiResponse(message="credentials updated", data=update_credentials_payload(payload)) + + +@router.get("/settings/juziseo/status", response_model=ApiResponse) +def get_settings_juziseo_status() -> ApiResponse: + return ApiResponse(data=get_juziseo_status()) + + +@router.post("/settings/juziseo/login", response_model=ApiResponse) +def settings_juziseo_login(payload: dict) -> ApiResponse: + try: + data = login_juziseo(str(payload.get("email") or ""), str(payload.get("password") or "")) + except ValueError as exc: + raise HTTPException(status_code=400, detail=str(exc)) from exc + return ApiResponse(message="桔子SEO登录成功", data=data) + + @router.get("/settings/backups/download/{filename}") def download_settings_backup(filename: str): path = settings_backup_root() / Path(filename).name diff --git a/domain-api/app/core/config.py b/domain-api/app/core/config.py index 1236b93..90b0909 100644 --- a/domain-api/app/core/config.py +++ b/domain-api/app/core/config.py @@ -31,6 +31,17 @@ class Settings(BaseSettings): worker_mode: str = "windows-local" worker_service_name: str = "domaincheck-worker" api_service_name: str = "domaincheck-api" + sync_agent_service_name: str = "domaincheck-sync-agent" + node_code: str = "overseas-control-01" + node_region: str = "overseas" + node_role: str = "control" + sync_push_enabled: bool = False + sync_source_region: str = "unknown" + sync_target_region: str = "overseas" + sync_target_api_base_url: str = "" + sync_shared_token: str = "" + sync_batch_size: int = 200 + sync_poll_interval_seconds: int = 30 @field_validator("cors_origins", mode="before") @classmethod diff --git a/domain-api/app/core/files.py b/domain-api/app/core/files.py index f83e2db..9045bed 100644 --- a/domain-api/app/core/files.py +++ b/domain-api/app/core/files.py @@ -15,8 +15,11 @@ def read_json(relative_path: str, default: dict | list | None = None): path = domain_root() / relative_path if not path.exists(): return {} if default is None else default - with path.open("r", encoding="utf-8") as handle: - return json.load(handle) + try: + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + except (json.JSONDecodeError, OSError): + return {} if default is None else default def write_json(relative_path: str, payload) -> None: @@ -44,8 +47,11 @@ def read_runtime_json(filename: str, default: dict | list | None = None): path = runtime_root() / filename if not path.exists(): return {} if default is None else default - with path.open("r", encoding="utf-8") as handle: - return json.load(handle) + try: + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + except (json.JSONDecodeError, OSError): + return {} if default is None else default def write_runtime_json(filename: str, payload) -> None: @@ -77,8 +83,11 @@ def load_export_records() -> list[dict]: path = runtime_root() / "export_tasks.json" if not path.exists(): return [] - with path.open("r", encoding="utf-8") as handle: - return json.load(handle) + try: + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + except (json.JSONDecodeError, OSError): + return [] def save_export_record(record: dict) -> None: @@ -93,8 +102,11 @@ def load_import_records() -> list[dict]: path = runtime_root() / "import_tasks.json" if not path.exists(): return [] - with path.open("r", encoding="utf-8") as handle: - return json.load(handle) + try: + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + except (json.JSONDecodeError, OSError): + return [] def save_import_records(records: list[dict]) -> None: @@ -103,5 +115,39 @@ def save_import_records(records: list[dict]) -> None: json.dump(records[:200], handle, ensure_ascii=False, indent=2, default=str) +def load_juming_records() -> list[dict]: + path = runtime_root() / "juming_tasks.json" + if not path.exists(): + return [] + try: + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + except (json.JSONDecodeError, OSError): + return [] + + +def save_juming_records(records: list[dict]) -> None: + path = runtime_root() / "juming_tasks.json" + with path.open("w", encoding="utf-8") as handle: + json.dump(records[:200], handle, ensure_ascii=False, indent=2, default=str) + + +def load_detect_records() -> list[dict]: + path = runtime_root() / "detect_runs.json" + if not path.exists(): + return [] + try: + with path.open("r", encoding="utf-8") as handle: + return json.load(handle) + except (json.JSONDecodeError, OSError): + return [] + + +def save_detect_records(records: list[dict]) -> None: + path = runtime_root() / "detect_runs.json" + with path.open("w", encoding="utf-8") as handle: + json.dump(records[:200], handle, ensure_ascii=False, indent=2, default=str) + + def timestamp_filename(prefix: str, ext: str) -> str: return f"{prefix}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.{ext}" diff --git a/domain-api/app/main.py b/domain-api/app/main.py index 89b5b87..8a9989e 100644 --- a/domain-api/app/main.py +++ b/domain-api/app/main.py @@ -1,8 +1,22 @@ +import threading + from fastapi import FastAPI from fastapi.middleware.cors import CORSMiddleware -from app.api.routes import auth, dashboard, settings as settings_routes, imports, detect, domains, exports, logs, runtime +from app.api.routes import auth, dashboard, settings as settings_routes, imports, detect, domains, exports, logs, runtime, juming, sensitive_words from app.core.config import settings as app_settings +from app.services.cluster_runtime_service import ensure_runtime_schema, register_local_control_heartbeat +_heartbeat_stop_event = threading.Event() + + +def _control_heartbeat_loop() -> None: + while not _heartbeat_stop_event.is_set(): + try: + register_local_control_heartbeat() + except Exception: + pass + _heartbeat_stop_event.wait(30) + app = FastAPI( title="domainCheck API", @@ -19,6 +33,19 @@ app.add_middleware( ) +@app.on_event("startup") +def on_startup() -> None: + ensure_runtime_schema() + register_local_control_heartbeat() + _heartbeat_stop_event.clear() + threading.Thread(target=_control_heartbeat_loop, name="control-heartbeat", daemon=True).start() + + +@app.on_event("shutdown") +def on_shutdown() -> None: + _heartbeat_stop_event.set() + + @app.get("/health") def health() -> dict: return { @@ -41,3 +68,5 @@ app.include_router(domains.router, prefix=app_settings.api_prefix) app.include_router(exports.router, prefix=app_settings.api_prefix) app.include_router(logs.router, prefix=app_settings.api_prefix) app.include_router(runtime.router, prefix=app_settings.api_prefix) +app.include_router(juming.router, prefix=app_settings.api_prefix) +app.include_router(sensitive_words.router, prefix=app_settings.api_prefix) diff --git a/domain-api/app/services/cluster_runtime_service.py b/domain-api/app/services/cluster_runtime_service.py new file mode 100644 index 0000000..89f8daf --- /dev/null +++ b/domain-api/app/services/cluster_runtime_service.py @@ -0,0 +1,272 @@ +from __future__ import annotations + +import json +import socket +from datetime import datetime, timedelta + +from app.core.config import settings +from app.core.db import get_db + + +_RUNTIME_SCHEMA_SQL = """ +CREATE TABLE IF NOT EXISTS detect_worker_nodes ( + node_code VARCHAR(64) PRIMARY KEY, + region VARCHAR(32) NOT NULL DEFAULT 'unknown', + role VARCHAR(32) NOT NULL DEFAULT 'worker', + hostname VARCHAR(255) NOT NULL DEFAULT '', + ip VARCHAR(64) NOT NULL DEFAULT '', + status VARCHAR(32) NOT NULL DEFAULT 'unknown', + worker_version VARCHAR(32) NOT NULL DEFAULT '', + current_load INTEGER NOT NULL DEFAULT 0, + metadata_json JSONB, + last_heartbeat_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS detect_jobs ( + id BIGSERIAL PRIMARY KEY, + job_code VARCHAR(64) NOT NULL UNIQUE, + source VARCHAR(64) NOT NULL DEFAULT 'manual', + plan_hash VARCHAR(128) NOT NULL DEFAULT '', + status VARCHAR(32) NOT NULL DEFAULT 'pending', + remark TEXT NOT NULL DEFAULT '', + created_by VARCHAR(64) NOT NULL DEFAULT '', + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + started_at TIMESTAMP, + finished_at TIMESTAMP +); + +CREATE TABLE IF NOT EXISTS detect_job_items ( + id BIGSERIAL PRIMARY KEY, + job_id BIGINT NOT NULL REFERENCES detect_jobs(id) ON DELETE CASCADE, + domain_id BIGINT NOT NULL, + status VARCHAR(32) NOT NULL DEFAULT 'pending', + claimed_by VARCHAR(64) NOT NULL DEFAULT '', + claim_token VARCHAR(64) NOT NULL DEFAULT '', + lease_expires_at TIMESTAMP, + attempt_count INTEGER NOT NULL DEFAULT 0, + last_error TEXT NOT NULL DEFAULT '', + result_version VARCHAR(64) NOT NULL DEFAULT '', + started_at TIMESTAMP, + finished_at TIMESTAMP, + updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + CONSTRAINT uq_detect_job_items_job_domain UNIQUE (job_id, domain_id) +); + +CREATE INDEX IF NOT EXISTS idx_detect_job_items_status_lease +ON detect_job_items(status, lease_expires_at); + +CREATE TABLE IF NOT EXISTS detect_run_events ( + id BIGSERIAL PRIMARY KEY, + job_id BIGINT REFERENCES detect_jobs(id) ON DELETE SET NULL, + job_item_id BIGINT REFERENCES detect_job_items(id) ON DELETE SET NULL, + node_code VARCHAR(64) NOT NULL DEFAULT '', + event_type VARCHAR(64) NOT NULL DEFAULT '', + level VARCHAR(16) NOT NULL DEFAULT 'info', + message TEXT NOT NULL DEFAULT '', + payload_json JSONB, + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX IF NOT EXISTS idx_detect_run_events_job_created +ON detect_run_events(job_id, created_at DESC); + +CREATE TABLE IF NOT EXISTS detect_sync_records ( + id BIGSERIAL PRIMARY KEY, + sync_type VARCHAR(32) NOT NULL DEFAULT '', + source_region VARCHAR(32) NOT NULL DEFAULT '', + target_region VARCHAR(32) NOT NULL DEFAULT '', + status VARCHAR(32) NOT NULL DEFAULT 'pending', + payload_json JSONB, + error_message TEXT NOT NULL DEFAULT '', + created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP +); +""" + + +def _resolve_local_ip() -> str: + try: + return socket.gethostbyname(socket.gethostname()) + except Exception: + return "" + + +def _decode_json(value: object) -> dict: + if isinstance(value, dict): + return value + if value in (None, ""): + return {} + try: + return json.loads(value) + except Exception: + return {} + + +def ensure_runtime_schema() -> None: + with get_db() as conn: + conn.autocommit = False + with conn.cursor() as cur: + cur.execute(_RUNTIME_SCHEMA_SQL) + conn.commit() + + +def register_node_heartbeat( + *, + node_code: str, + region: str, + role: str, + status: str, + current_load: int = 0, + worker_version: str = "0.1.0", + metadata: dict | None = None, +) -> None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + INSERT INTO detect_worker_nodes ( + node_code, region, role, hostname, ip, status, worker_version, current_load, metadata_json, last_heartbeat_at, update_time + ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) + ON CONFLICT (node_code) DO UPDATE SET + region = EXCLUDED.region, + role = EXCLUDED.role, + hostname = EXCLUDED.hostname, + ip = EXCLUDED.ip, + status = EXCLUDED.status, + worker_version = EXCLUDED.worker_version, + current_load = EXCLUDED.current_load, + metadata_json = EXCLUDED.metadata_json, + last_heartbeat_at = CURRENT_TIMESTAMP, + update_time = CURRENT_TIMESTAMP + """, + ( + node_code, + region, + role, + socket.gethostname(), + _resolve_local_ip(), + status, + worker_version, + max(0, int(current_load or 0)), + json.dumps(metadata or {}, ensure_ascii=False), + ), + ) + conn.commit() + + +def register_local_control_heartbeat() -> None: + register_node_heartbeat( + node_code=settings.node_code, + region=settings.node_region, + role=settings.node_role, + status="online", + current_load=0, + metadata={ + "service": "domain-api", + "api_host": settings.api_host, + "api_port": settings.api_port, + "worker_mode": settings.worker_mode, + "updated_at": datetime.now().isoformat(timespec="seconds"), + }, + ) + + +def _normalize_node_status(raw_status: str, last_heartbeat_at: datetime | None) -> str: + status = str(raw_status or "").strip() or "unknown" + if not last_heartbeat_at: + return status + now = datetime.now(last_heartbeat_at.tzinfo) if last_heartbeat_at.tzinfo else datetime.now() + age = now - last_heartbeat_at + if age > timedelta(minutes=5): + return "offline" + if age > timedelta(seconds=90): + return "stale" + return status + + +def get_cluster_snapshot() -> dict: + register_local_control_heartbeat() + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT node_code, region, role, hostname, ip, status, worker_version, current_load, metadata_json, last_heartbeat_at + FROM detect_worker_nodes + ORDER BY + CASE WHEN status = 'busy' THEN 0 WHEN status = 'online' THEN 1 ELSE 2 END, + region ASC, + role ASC, + node_code ASC + LIMIT 100 + """ + ) + rows = cur.fetchall() + cur.execute("SELECT count(*) FROM detect_jobs") + jobs_total = cur.fetchone()[0] + cur.execute("SELECT count(*) FROM detect_job_items WHERE status IN ('pending', 'claimed', 'running')") + active_items = cur.fetchone()[0] + + nodes = [ + { + "node_code": row[0], + "region": row[1], + "role": row[2], + "hostname": row[3], + "ip": row[4], + "status": _normalize_node_status(row[5], row[9]), + "worker_version": row[6], + "current_load": row[7], + "metadata": _decode_json(row[8]), + "last_heartbeat_at": row[9].isoformat(sep=" ", timespec="seconds") if row[9] else "", + } + for row in rows + ] + status_counts: dict[str, int] = {} + role_counts: dict[str, int] = {} + region_counts: dict[str, int] = {} + busy_nodes: list[str] = [] + stale_nodes: list[str] = [] + offline_nodes: list[str] = [] + online_worker_nodes = 0 + online_control_nodes = 0 + + for node in nodes: + node_status = str(node.get("status") or "unknown") + node_role = str(node.get("role") or "unknown") + node_region = str(node.get("region") or "unknown") + node_code = str(node.get("node_code") or "") + + status_counts[node_status] = status_counts.get(node_status, 0) + 1 + role_counts[node_role] = role_counts.get(node_role, 0) + 1 + region_counts[node_region] = region_counts.get(node_region, 0) + 1 + + if node_status == "busy": + busy_nodes.append(node_code) + if node_status == "stale": + stale_nodes.append(node_code) + if node_status == "offline": + offline_nodes.append(node_code) + if node_role == "worker" and node_status in {"online", "busy"}: + online_worker_nodes += 1 + if node_role == "control" and node_status in {"online", "busy"}: + online_control_nodes += 1 + + return { + "nodes": nodes, + "nodes_total": len(nodes), + "jobs_total": jobs_total, + "active_job_items": active_items, + "summary": { + "status_counts": status_counts, + "role_counts": role_counts, + "region_counts": region_counts, + "busy_nodes": busy_nodes, + "stale_nodes": stale_nodes, + "offline_nodes": offline_nodes, + "online_worker_nodes": online_worker_nodes, + "online_control_nodes": online_control_nodes, + }, + } diff --git a/domain-api/app/services/detect_job_service.py b/domain-api/app/services/detect_job_service.py new file mode 100644 index 0000000..d0c3eff --- /dev/null +++ b/domain-api/app/services/detect_job_service.py @@ -0,0 +1,558 @@ +from __future__ import annotations + +import json +from datetime import datetime +import math +from uuid import uuid4 + +from app.core.config import settings +from app.core.db import get_db + + +ACTIVE_JOB_STATUSES = ("pending", "running") + + +def _selection_sql() -> str: + return """ + SELECT id + FROM domains + WHERE + detect_status IN (0, 4) + OR (use_status = 0 AND detect_status = 1 AND register_status = 3 AND expire_date < CURRENT_DATE) + ORDER BY id ASC + LIMIT %s + """ + + +def _format_time(value: datetime | None) -> str: + return value.isoformat(sep=" ", timespec="seconds") if value else "" + + +def _decode_payload(value: object) -> dict: + if isinstance(value, dict): + return value + if value in (None, ""): + return {} + try: + return json.loads(value) + except Exception: + return {} + + +def _extract_current_cycle_events(events: list[dict]) -> tuple[str, list[dict]]: + if not events: + return "", [] + + latest_payload = events[0].get("payload") or {} + latest_cycle_token = str(latest_payload.get("cycle_token") or "").strip() + if latest_cycle_token: + current_cycle = [] + for event in events: + payload = event.get("payload") or {} + event_cycle = str(payload.get("cycle_token") or "").strip() + if event_cycle == latest_cycle_token: + current_cycle.append(event) + return latest_cycle_token, current_cycle + + cycle_token = "" + anchor_index = -1 + for index, event in enumerate(events): + payload = event.get("payload") or {} + event_type = str(event.get("event_type") or "") + candidate = str(payload.get("cycle_token") or "").strip() + if candidate and event_type in {"job_dispatch_sent", "job_dispatch_requested", "job_dispatch_failed", "job_dispatch_rejected"}: + cycle_token = candidate + anchor_index = index + break + if not cycle_token: + return "", events + current_cycle = [] + for index, event in enumerate(events[: anchor_index + 1]): + payload = event.get("payload") or {} + event_cycle = str(payload.get("cycle_token") or "").strip() + if index == anchor_index or event_cycle == cycle_token: + current_cycle.append(event) + return cycle_token, current_cycle + + +def _fetch_job_summary(cur, job_row, event_limit: int = 20) -> dict: + job_id = job_row[0] + cur.execute( + """ + SELECT status, count(*) + FROM detect_job_items + WHERE job_id = %s + GROUP BY status + """, + (job_id,), + ) + counts = {status: int(count) for status, count in cur.fetchall()} + cur.execute( + """ + SELECT COALESCE(NULLIF(claimed_by, ''), 'unassigned') AS node_code, + status, + count(*) + FROM detect_job_items + WHERE job_id = %s + GROUP BY COALESCE(NULLIF(claimed_by, ''), 'unassigned'), status + ORDER BY node_code ASC, status ASC + """, + (job_id,), + ) + node_buckets: dict[str, dict] = {} + for node_code, status, count in cur.fetchall(): + bucket = node_buckets.setdefault( + node_code, + { + "node_code": node_code, + "items_total": 0, + "items_pending": 0, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + }, + ) + field_name = f"items_{status}" + if field_name in bucket: + bucket[field_name] += int(count) + bucket["items_total"] += int(count) + + cur.execute( + """ + SELECT node_code, event_type, level, message, payload_json, created_at + FROM detect_run_events + WHERE job_id = %s + ORDER BY created_at DESC, id DESC + LIMIT %s + """, + (job_id, max(1, int(event_limit or 20))), + ) + events = [ + { + "node_code": item[0], + "event_type": item[1], + "level": item[2], + "message": item[3], + "payload": _decode_payload(item[4]), + "created_at": _format_time(item[5]), + } + for item in cur.fetchall() + ] + cycle_token, current_cycle_events = _extract_current_cycle_events(events) + total = sum(counts.values()) + terminal = int(counts.get("completed", 0)) + int(counts.get("blacklisted", 0)) + int(counts.get("failed", 0)) + return { + "job_id": job_id, + "job_code": job_row[1], + "source": job_row[2], + "status": job_row[3], + "created_by": job_row[4], + "created_at": _format_time(job_row[5]), + "started_at": _format_time(job_row[6]), + "finished_at": _format_time(job_row[7]), + "items_total": total, + "items_pending": int(counts.get("pending", 0)), + "items_claimed": int(counts.get("claimed", 0)), + "items_running": int(counts.get("running", 0)), + "items_completed": int(counts.get("completed", 0)), + "items_blacklisted": int(counts.get("blacklisted", 0)), + "items_failed": int(counts.get("failed", 0)), + "items_terminal": terminal, + "progress_percent": round((terminal / total) * 100, 2) if total else 0, + "node_stats": list(node_buckets.values()), + "recent_events": events, + "latest_event": events[0] if events else None, + "current_cycle_token": cycle_token, + "current_cycle_events": current_cycle_events, + "latest_cycle_event": current_cycle_events[0] if current_cycle_events else None, + } + + +def get_detect_job_summary(job_id: int, event_limit: int = 20) -> dict | None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at + FROM detect_jobs + WHERE id = %s + LIMIT 1 + """, + (int(job_id),), + ) + row = cur.fetchone() + if not row: + return None + return _fetch_job_summary(cur, row, event_limit=event_limit) + + +def get_active_detect_job_summary(event_limit: int = 20) -> dict | None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at + FROM detect_jobs + WHERE status IN ('pending', 'running') + ORDER BY created_at DESC, id DESC + LIMIT 1 + """ + ) + row = cur.fetchone() + if not row: + return None + return _fetch_job_summary(cur, row, event_limit=event_limit) + + +def list_detect_jobs(limit: int = 20) -> list[dict]: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at + FROM detect_jobs + ORDER BY created_at DESC, id DESC + LIMIT %s + """, + (max(1, min(int(limit or 20), 100)),), + ) + rows = cur.fetchall() + return [_fetch_job_summary(cur, row, event_limit=10) for row in rows] + + +def get_detect_queue_health(window_minutes: int = 15) -> dict: + window_minutes = max(5, min(int(window_minutes or 15), 120)) + active_job = get_active_detect_job_summary(event_limit=10) + if not active_job: + return { + "window_minutes": window_minutes, + "has_active_job": False, + "job": None, + "queue": { + "items_total": 0, + "pending": 0, + "claimed": 0, + "running": 0, + "completed": 0, + "blacklisted": 0, + "failed": 0, + "terminal": 0, + "terminal_percent": 0, + "oldest_pending_at": "", + "oldest_pending_age_minutes": 0, + "nearest_lease_expiry_at": "", + "overdue_leases": 0, + "expiring_soon_leases": 0, + }, + "throughput": { + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + }, + "nodes": [], + } + + job_id = int(active_job["job_id"]) + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT + MIN(create_time) FILTER (WHERE status = 'pending') AS oldest_pending_at, + MIN(lease_expires_at) FILTER (WHERE status IN ('claimed', 'running') AND lease_expires_at IS NOT NULL) AS nearest_lease_expiry_at, + COUNT(*) FILTER (WHERE status IN ('claimed', 'running') AND lease_expires_at IS NOT NULL AND lease_expires_at < CURRENT_TIMESTAMP) AS overdue_leases, + COUNT(*) FILTER ( + WHERE status IN ('claimed', 'running') + AND lease_expires_at IS NOT NULL + AND lease_expires_at >= CURRENT_TIMESTAMP + AND lease_expires_at < CURRENT_TIMESTAMP + interval '5 minutes' + ) AS expiring_soon_leases + FROM detect_job_items + WHERE job_id = %s + """, + (job_id,), + ) + lease_row = cur.fetchone() + + cur.execute( + """ + SELECT + COALESCE(NULLIF(node_code, ''), 'unknown') AS node_code, + COUNT(*) AS processed_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_completed') AS completed_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_blacklisted') AS blacklisted_recent, + COUNT(*) FILTER (WHERE event_type = 'domain_failed') AS failed_recent + FROM detect_run_events + WHERE job_id = %s + AND event_type IN ('domain_completed', 'domain_blacklisted', 'domain_failed') + AND created_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval + GROUP BY COALESCE(NULLIF(node_code, ''), 'unknown') + ORDER BY processed_recent DESC, node_code ASC + """, + (job_id, window_minutes), + ) + throughput_rows = cur.fetchall() + + oldest_pending_at = _format_time(lease_row[0]) if lease_row and lease_row[0] else "" + nearest_lease_expiry_at = _format_time(lease_row[1]) if lease_row and lease_row[1] else "" + oldest_pending_age_minutes = 0 + if lease_row and lease_row[0]: + oldest_pending_age_minutes = max(0, int((datetime.now() - lease_row[0]).total_seconds() // 60)) + + node_map = { + str(item.get("node_code") or "unknown"): { + "node_code": str(item.get("node_code") or "unknown"), + "items_total": int(item.get("items_total", 0) or 0), + "items_pending": int(item.get("items_pending", 0) or 0), + "items_claimed": int(item.get("items_claimed", 0) or 0), + "items_running": int(item.get("items_running", 0) or 0), + "items_completed": int(item.get("items_completed", 0) or 0), + "items_blacklisted": int(item.get("items_blacklisted", 0) or 0), + "items_failed": int(item.get("items_failed", 0) or 0), + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + } + for item in active_job.get("node_stats") or [] + } + total_processed_recent = 0 + total_completed_recent = 0 + total_blacklisted_recent = 0 + total_failed_recent = 0 + for row in throughput_rows: + node_code = str(row[0] or "unknown") + bucket = node_map.setdefault( + node_code, + { + "node_code": node_code, + "items_total": 0, + "items_pending": 0, + "items_claimed": 0, + "items_running": 0, + "items_completed": 0, + "items_blacklisted": 0, + "items_failed": 0, + "processed_recent": 0, + "processed_per_minute": 0, + "completed_recent": 0, + "blacklisted_recent": 0, + "failed_recent": 0, + }, + ) + processed_recent = int(row[1] or 0) + completed_recent = int(row[2] or 0) + blacklisted_recent = int(row[3] or 0) + failed_recent = int(row[4] or 0) + bucket["processed_recent"] = processed_recent + bucket["processed_per_minute"] = round(processed_recent / window_minutes, 2) + bucket["completed_recent"] = completed_recent + bucket["blacklisted_recent"] = blacklisted_recent + bucket["failed_recent"] = failed_recent + total_processed_recent += processed_recent + total_completed_recent += completed_recent + total_blacklisted_recent += blacklisted_recent + total_failed_recent += failed_recent + + nodes = sorted( + node_map.values(), + key=lambda item: ( + -int(item.get("processed_recent", 0) or 0), + -int(item.get("items_running", 0) or 0), + str(item.get("node_code") or ""), + ), + ) + items_total = int(active_job.get("items_total", 0) or 0) + terminal = int(active_job.get("items_terminal", 0) or 0) + return { + "window_minutes": window_minutes, + "has_active_job": True, + "job": { + "job_id": job_id, + "job_code": active_job.get("job_code", ""), + "status": active_job.get("status", ""), + "progress_percent": active_job.get("progress_percent", 0), + }, + "queue": { + "items_total": items_total, + "pending": int(active_job.get("items_pending", 0) or 0), + "claimed": int(active_job.get("items_claimed", 0) or 0), + "running": int(active_job.get("items_running", 0) or 0), + "completed": int(active_job.get("items_completed", 0) or 0), + "blacklisted": int(active_job.get("items_blacklisted", 0) or 0), + "failed": int(active_job.get("items_failed", 0) or 0), + "terminal": terminal, + "terminal_percent": round((terminal / items_total) * 100, 2) if items_total else 0, + "oldest_pending_at": oldest_pending_at, + "oldest_pending_age_minutes": oldest_pending_age_minutes, + "nearest_lease_expiry_at": nearest_lease_expiry_at, + "overdue_leases": int(lease_row[2] or 0) if lease_row else 0, + "expiring_soon_leases": int(lease_row[3] or 0) if lease_row else 0, + }, + "throughput": { + "processed_recent": total_processed_recent, + "processed_per_minute": round(total_processed_recent / window_minutes, 2), + "completed_recent": total_completed_recent, + "blacklisted_recent": total_blacklisted_recent, + "failed_recent": total_failed_recent, + }, + "nodes": nodes, + } + + +def get_detect_capacity_plan(*, queue_health: dict | None = None, online_worker_nodes: int = 0, target_finish_hours: int = 6) -> dict: + queue_health = queue_health or get_detect_queue_health(window_minutes=15) + target_finish_hours = max(1, min(int(target_finish_hours or 6), 72)) + online_worker_nodes = max(0, int(online_worker_nodes or 0)) + + if not queue_health.get("has_active_job"): + return { + "has_active_job": False, + "online_worker_nodes": online_worker_nodes, + "target_finish_hours": target_finish_hours, + "estimated_hours_remaining": 0, + "recommended_total_workers": max(1, online_worker_nodes), + "recommended_additional_workers": 0, + "current_processed_per_hour": 0, + "pending_items": 0, + "terminal_items": 0, + "summary": "当前没有活跃任务，无需扩容建议。", + } + + queue = queue_health.get("queue") or {} + throughput = queue_health.get("throughput") or {} + pending_items = int(queue.get("pending", 0) or 0) + claimed_items = int(queue.get("claimed", 0) or 0) + running_items = int(queue.get("running", 0) or 0) + remaining_items = pending_items + claimed_items + running_items + current_processed_per_hour = round(float(throughput.get("processed_per_minute", 0) or 0) * 60, 2) + estimated_hours_remaining = round((remaining_items / current_processed_per_hour), 2) if current_processed_per_hour > 0 else 0 + + recommended_total_workers = max(1, online_worker_nodes or 1) + recommended_additional_workers = 0 + if remaining_items > 0 and target_finish_hours > 0: + required_per_hour = remaining_items / target_finish_hours + if current_processed_per_hour > 0 and max(1, online_worker_nodes) > 0: + per_worker_per_hour = current_processed_per_hour / max(1, online_worker_nodes) + recommended_total_workers = max(1, int(math.ceil(required_per_hour / per_worker_per_hour))) + recommended_additional_workers = max(0, recommended_total_workers - online_worker_nodes) + elif remaining_items > 0: + recommended_total_workers = max(1, online_worker_nodes or 1) + recommended_additional_workers = 0 + + summary = ( + f"当前在线 Worker {online_worker_nodes} 台，近窗吞吐约 {current_processed_per_hour} 项/小时，" + f"剩余待处理约 {remaining_items} 项，预计还需 {estimated_hours_remaining} 小时。" + ) + if recommended_additional_workers > 0: + summary = ( + f"{summary} 若希望在 {target_finish_hours} 小时内收敛，建议总 Worker 数达到 " + f"{recommended_total_workers} 台，至少再加 {recommended_additional_workers} 台。" + ) + else: + summary = f"{summary} 按当前目标 {target_finish_hours} 小时看，现有 Worker 数量暂时够用。" + + return { + "has_active_job": True, + "online_worker_nodes": online_worker_nodes, + "target_finish_hours": target_finish_hours, + "estimated_hours_remaining": estimated_hours_remaining, + "recommended_total_workers": recommended_total_workers, + "recommended_additional_workers": recommended_additional_workers, + "current_processed_per_hour": current_processed_per_hour, + "pending_items": pending_items, + "remaining_items": remaining_items, + "terminal_items": int(queue.get("terminal", 0) or 0), + "summary": summary, + } + + +def append_detect_job_event( + job_id: int, + *, + event_type: str, + message: str, + level: str = "info", + payload: dict | None = None, + node_code: str | None = None, +) -> None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json) + VALUES (%s, %s, %s, %s, %s, %s::jsonb) + """, + ( + int(job_id), + node_code or settings.node_code, + str(event_type or "").strip() or "info", + str(level or "info").strip() or "info", + str(message or "").strip(), + json.dumps(payload or {}, ensure_ascii=False), + ), + ) + conn.commit() + + +def create_detect_job_if_needed(limit: int = 1000, created_by: str = "system") -> dict | None: + existing = get_active_detect_job_summary() + if existing: + return existing + + with get_db() as conn: + conn.autocommit = False + with conn.cursor() as cur: + cur.execute(_selection_sql(), (max(1, int(limit or 1000)),)) + domain_ids = [row[0] for row in cur.fetchall()] + if not domain_ids: + conn.rollback() + return None + + job_code = f"detect-{datetime.now().strftime('%Y%m%d%H%M%S')}-{uuid4().hex[:6]}" + plan_hash = uuid4().hex + cur.execute( + """ + INSERT INTO detect_jobs (job_code, source, plan_hash, status, remark, created_by) + VALUES (%s, %s, %s, 'pending', %s, %s) + RETURNING id + """, + ( + job_code, + "api-start", + plan_hash, + f"API 创建检测任务，待检测域名 {len(domain_ids)} 个", + created_by, + ), + ) + job_id = cur.fetchone()[0] + item_rows = [(job_id, domain_id) for domain_id in domain_ids] + cur.executemany( + """ + INSERT INTO detect_job_items (job_id, domain_id, status) + VALUES (%s, %s, 'pending') + ON CONFLICT (job_id, domain_id) DO NOTHING + """, + item_rows, + ) + cur.execute( + """ + INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json) + VALUES (%s, %s, %s, %s, %s, %s::jsonb) + """, + ( + job_id, + settings.node_code, + "job_created", + "info", + f"创建检测任务 {job_code}，共 {len(domain_ids)} 个域名", + '{"count": %s}' % len(domain_ids), + ), + ) + conn.commit() + return get_active_detect_job_summary() diff --git a/domain-api/app/services/detect_run_service.py b/domain-api/app/services/detect_run_service.py new file mode 100644 index 0000000..3ad0ebd --- /dev/null +++ b/domain-api/app/services/detect_run_service.py @@ -0,0 +1,425 @@ +from __future__ import annotations + +from datetime import datetime +from uuid import uuid4 + +from app.core.files import load_detect_records, save_detect_records, tail_lines + + +_MAX_LOG_LINES = 240 +_LOG_TAIL_LINES = 1200 +_ACTIVE_STATUSES = {"starting", "running", "stopping"} +_TIMESTAMP_FORMATS = ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S") + + +def _now() -> str: + return datetime.now().isoformat(sep=" ", timespec="seconds") + + +def _save(records: list[dict]) -> None: + save_detect_records(records) + + +def _load() -> list[dict]: + return load_detect_records() + + +def _capture_worker_logs(max_lines: int = _LOG_TAIL_LINES) -> list[str]: + lines = tail_lines("detect_worker.log", max_lines=max_lines) + return lines[-max_lines:] + + +def _parse_time(raw: str | None) -> datetime | None: + if not raw: + return None + text = str(raw).strip() + if not text: + return None + try: + return datetime.fromisoformat(text) + except ValueError: + pass + for fmt in _TIMESTAMP_FORMATS: + try: + return datetime.strptime(text, fmt) + except ValueError: + continue + return None + + +def _extract_log_time(line: str) -> datetime | None: + if len(line) < 19: + return None + candidates = [line[:26], line[:19]] + for candidate in candidates: + for fmt in _TIMESTAMP_FORMATS: + if len(candidate) != len(datetime.now().strftime(fmt)): + continue + try: + return datetime.strptime(candidate, fmt) + except ValueError: + continue + return None + + +def _filter_logs_since(lines: list[str], started_at: str | None) -> list[str]: + if not started_at: + return lines[-_MAX_LOG_LINES:] + started_time = _parse_time(started_at) + if not started_time: + return lines[-_MAX_LOG_LINES:] + filtered = [line for line in lines if (_extract_log_time(line) or started_time) >= started_time] + if filtered: + return filtered[-_MAX_LOG_LINES:] + return lines[-_MAX_LOG_LINES:] + + +def _merge_logs(existing: list[str] | None, current: list[str]) -> list[str]: + merged = list(existing or []) + for line in current: + if line not in merged[-40:]: + merged.append(line) + continue + if not merged or merged[-1] != line: + merged.append(line) + return merged[-_MAX_LOG_LINES:] + + +def _find_active(records: list[dict]) -> dict | None: + return next((item for item in records if item.get("status") in _ACTIVE_STATUSES), None) + + +def _same_session(active: dict | None, runtime: dict) -> bool: + if not active: + return False + runtime_started_at = str(runtime.get("latest_start_time", "") or "").strip() + active_started_at = str(active.get("started_at", "") or "").strip() + if not runtime_started_at or not active_started_at: + return True + runtime_started = _parse_time(runtime_started_at) + active_started = _parse_time(active_started_at) + if not runtime_started or not active_started: + return True + return abs((runtime_started - active_started).total_seconds()) < 3 + + +def _latest_matching_log(log_lines: list[str], keywords: tuple[str, ...]) -> str: + for line in reversed(log_lines or []): + if any(keyword in line for keyword in keywords): + return line + return "" + + +def _sync_phase_history(record: dict, phase_label: str, phase_detail: str) -> None: + history = list(record.get("phase_history") or []) + current = { + "at": _now(), + "label": phase_label or "-", + "detail": phase_detail or "", + } + if history: + latest = history[-1] + if latest.get("label") == current["label"] and latest.get("detail") == current["detail"]: + return + history.append(current) + record["phase_history"] = history[-20:] + + +def _phase_from_runtime(status: str, runtime: dict, progress: dict, log_lines: list[str], active_job: dict | None = None) -> tuple[str, str]: + active_job = active_job or {} + if status == "starting": + return "启动中", "正在拉起检测服务并等待 Worker 就绪" + if status == "stopping": + return "停止中", "已发送停止请求，等待 Worker 退出并归档日志" + if status == "completed": + terminal = int(active_job.get("items_terminal", 0) or 0) + return "已完成", f"本轮检测已自然完成，本次累计处理 {terminal} 个任务项" + if status == "partial_failed": + failed = int(active_job.get("items_failed", 0) or 0) + terminal = int(active_job.get("items_terminal", 0) or 0) + return "部分失败", f"本轮检测已结束，其中失败 {failed} 个，累计处理 {terminal} 个任务项" + if status == "failed": + return "失败", runtime.get("message") or "Worker 异常退出，请检查日志" + if status == "stopped": + return "已停止", runtime.get("message") or "检测服务已停止" + if runtime.get("running"): + proxy_runtime_state = str(runtime.get("proxy_runtime_state", "") or "").strip() + proxy_runtime_detail = str(runtime.get("proxy_runtime_detail", "") or "").strip() + running = (progress or {}).get("running", 0) + pending = (progress or {}).get("pending", 0) + phase_log = _latest_matching_log( + log_lines, + ( + "Connection refused", + "Read timed out", + "ConnectTimeout", + "HTTPSConnectionPool", + "WaybackDetector", + "域名检测任务完成", + "当前批次检测完成", + "当前实际线程数量", + "开始创建线程", + "获取到", + "开始检测，刷新代理池", + "刷新代理池", + "没有需要检测的域名", + "开始执行域名检测任务", + ), + ) + if proxy_runtime_state == "blocked_no_proxy": + return "等待代理", proxy_runtime_detail or "代理池当前无可用代理，且未允许直连" + if proxy_runtime_state == "degraded_direct": + return "降级直连", proxy_runtime_detail or "代理池暂无可用代理，当前使用直连继续执行" + if "外部依赖异常，步骤降级继续执行" in phase_log: + return "外部站点异常", "外部依赖当前波动，系统已按降级策略继续执行并保留人工复核" + if "WaybackDetector" in phase_log or "web.archive.org" in phase_log: + return "外部站点异常", "时光机依赖当前访问异常，任务仍在继续，建议关注网络或代理策略" + if any(keyword in phase_log for keyword in ("Connection refused", "Read timed out", "ConnectTimeout", "HTTPSConnectionPool")): + return "网络波动", phase_log + if "域名检测任务完成" in phase_log: + return "完成归档", "本轮检测已处理完成，正在等待下一轮任务或归档最终日志" + if "当前批次检测完成" in phase_log: + return "批次完成", phase_log + if "当前实际线程数量" in phase_log or running > 0: + if running > 0: + return "检测中", f"Worker 正在处理 {running} 个检测任务" + return "检测中", phase_log + if "开始创建线程" in phase_log: + return "建线程中", phase_log + if "获取到" in phase_log: + return "取任务中", phase_log + if "刷新代理池" in phase_log: + return "刷新代理池", phase_log + if "没有需要检测的域名" in phase_log: + return "空闲等待", "Worker 在线，当前没有待检测域名" + if "开始执行域名检测任务" in phase_log: + return "准备检测", phase_log + if running > 0: + return "检测中", f"Worker 正在处理 {running} 个检测任务" + if pending > 0: + return "取任务中", f"Worker 在线，等待或领取待检测域名，当前剩余 {pending} 个" + return "运行中", "Worker 在线，当前没有活跃检测任务" + return "已停止", runtime.get("message") or "检测服务已停止" + + +def _sync_record( + record: dict, + *, + status: str, + message: str, + runtime: dict, + progress: dict, + settings_summary: dict, + log_lines: list[str], + active_job: dict | None = None, +) -> dict: + record["status"] = status + record["message"] = message + record["runtime"] = runtime + record["progress"] = progress + record["settings_summary"] = settings_summary + record["updated_at"] = _now() + if not record.get("started_at"): + record["started_at"] = runtime.get("latest_start_time") or record["updated_at"] + session_logs = _merge_logs(record.get("logs"), _filter_logs_since(log_lines, record.get("started_at"))) + record["logs"] = session_logs + record["active_job"] = active_job or {} + record["phase_label"], record["phase_detail"] = _phase_from_runtime(status, runtime, progress, session_logs, active_job) + _sync_phase_history(record, record.get("phase_label", ""), record.get("phase_detail", "")) + if status in {"stopped", "failed", "completed", "partial_failed"} and not record.get("completed_at"): + record["completed_at"] = _now() + if status in _ACTIVE_STATUSES: + record["completed_at"] = "" + return record + + +def create_detect_run_snapshot(message: str, runtime: dict, progress: dict, settings_summary: dict) -> dict: + records = _load() + active = _find_active(records) + current_logs = _capture_worker_logs() + if active: + if active.get("status") == "stopping" and runtime.get("running"): + active["status"] = "running" + _sync_record( + active, + status=active.get("status", "starting"), + message=message, + runtime=runtime, + progress=progress, + settings_summary=settings_summary, + log_lines=current_logs, + ) + _save(records) + return dict(active) + + initial_started_at = runtime.get("latest_start_time") or _now() + record = { + "run_id": uuid4().hex, + "status": "starting", + "message": message, + "created_at": _now(), + "updated_at": _now(), + "started_at": initial_started_at, + "completed_at": "", + "runtime": runtime, + "progress": progress, + "settings_summary": settings_summary, + "phase_label": "", + "phase_detail": "", + "phase_history": [], + "logs": [], + } + _sync_record( + record, + status="starting", + message=message, + runtime=runtime, + progress=progress, + settings_summary=settings_summary, + log_lines=current_logs, + ) + records.insert(0, record) + _save(records) + return dict(record) + + +def finalize_detect_run(message: str, runtime: dict, progress: dict, settings_summary: dict, active_job: dict | None = None) -> dict | None: + records = _load() + target = _find_active(records) + if not target: + return None + final_status = "stopped" if target.get("status") == "stopping" else "failed" + _sync_record( + target, + status=final_status, + message=message, + runtime=runtime, + progress=progress, + settings_summary=settings_summary, + log_lines=_capture_worker_logs(), + active_job=active_job, + ) + _save(records) + return dict(target) + + +def sync_detect_runs(runtime: dict, progress: dict, settings_summary: dict, active_job: dict | None = None) -> list[dict]: + records = _load() + active = _find_active(records) + current_logs = _capture_worker_logs() + active_job = active_job or {} + runtime_detecting = bool(runtime.get("detecting", False)) + active_job_status = str(active_job.get("status", "") or "").strip() + active_job_open = active_job_status in {"pending", "running"} + execution_active = runtime_detecting or active_job_open or int((progress or {}).get("running", 0) or 0) > 0 + + if runtime.get("running") and execution_active: + if active and not _same_session(active, runtime): + _sync_record( + active, + status="stopped", + message="检测服务已重启，上一轮会话已归档", + runtime=active.get("runtime") or runtime, + progress=active.get("progress") or progress, + settings_summary=active.get("settings_summary") or settings_summary, + log_lines=current_logs, + active_job=active.get("active_job") or active_job, + ) + active = None + if active: + next_status = "running" if active.get("status") != "stopping" else "stopping" + _sync_record( + active, + status=next_status, + message=runtime.get("message") or active.get("message") or "检测服务运行中", + runtime=runtime, + progress=progress, + settings_summary=settings_summary, + log_lines=current_logs, + active_job=active_job, + ) + else: + started_at = runtime.get("latest_start_time") or _now() + record = { + "run_id": uuid4().hex, + "status": "running", + "message": runtime.get("message") or "检测服务运行中", + "created_at": _now(), + "updated_at": _now(), + "started_at": started_at, + "completed_at": "", + "runtime": runtime, + "progress": progress, + "settings_summary": settings_summary, + "phase_label": "", + "phase_detail": "", + "phase_history": [], + "logs": [], + "active_job": active_job, + } + _sync_record( + record, + status="running", + message=record["message"], + runtime=runtime, + progress=progress, + settings_summary=settings_summary, + log_lines=current_logs, + active_job=active_job, + ) + records.insert(0, record) + elif active: + if runtime.get("running") and not execution_active: + if active.get("status") == "stopping": + final_status = "stopped" + final_message = runtime.get("message") or "检测任务已停止，Worker 保持待命" + elif active_job_status == "partial_failed": + final_status = "partial_failed" + final_message = "检测任务已结束，存在部分失败项" + elif active_job_status == "failed": + final_status = "failed" + final_message = "检测任务已结束，任务结果为失败" + else: + final_status = "completed" + final_message = "检测任务已自然完成，Worker 保持待命" + else: + final_status = "stopped" if active.get("status") == "stopping" else "failed" + final_message = runtime.get("message") or ("检测服务已停止" if final_status == "stopped" else "检测服务异常退出") + _sync_record( + active, + status=final_status, + message=final_message, + runtime=runtime, + progress=progress, + settings_summary=settings_summary, + log_lines=current_logs, + active_job=active_job, + ) + + if records: + records[0]["logs"] = _merge_logs( + records[0].get("logs"), + _filter_logs_since(current_logs, records[0].get("started_at")), + ) + + _save(records) + return records + + +def mark_detect_run_stopping(message: str, runtime: dict, progress: dict, settings_summary: dict) -> dict | None: + records = _load() + target = _find_active(records) + if not target: + return None + _sync_record( + target, + status="stopping", + message=message, + runtime=runtime, + progress=progress, + settings_summary=settings_summary, + log_lines=_capture_worker_logs(), + active_job=target.get("active_job") or {}, + ) + _save(records) + return dict(target) diff --git a/domain-api/app/services/detect_service.py b/domain-api/app/services/detect_service.py index 867dcfc..c66d609 100644 --- a/domain-api/app/services/detect_service.py +++ b/domain-api/app/services/detect_service.py @@ -1,15 +1,275 @@ from __future__ import annotations +import json +import re from datetime import datetime, timezone from pathlib import Path from app.core.db import get_db from app.core.files import tail_lines +from app.core.redis_client import get_redis from app.services.runtime_settings_service import get_runtime_settings -from app.services.settings_service import get_settings_payload +from app.services.detect_run_service import sync_detect_runs +from app.services.detect_job_service import get_active_detect_job_summary +from app.services.settings_service import get_settings_payload, resolve_thread_count +from app.services.sync_record_service import append_detect_result_projection_if_changed from app.services.worker_control_service import detect_worker_runtime +_PROXY_COUNT_RE = re.compile(r"当前可用代理数[:：]\s*(\d+)") +_THREAD_COUNT_RE = re.compile(r"当前实际线程数量[:：]\s*(\d+)\s*/\s*(\d+)") +_RUNTIME_STATE_KEY = "domain_tool:detect_runtime_state" +_TIMESTAMP_FORMATS = ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S") + + +def _extract_dependency_alerts(lines: list[str]) -> list[dict]: + alerts: list[dict] = [] + recent_lines = lines[-120:] if lines else [] + degraded_line = "" + for line in reversed(recent_lines): + if "外部依赖异常，步骤降级继续执行" in line: + degraded_line = line + alerts.append( + { + "kind": "dependency_degraded", + "level": "warning", + "title": "外部依赖降级继续", + "detail": line, + } + ) + break + if degraded_line: + return alerts + for line in reversed(recent_lines): + if "WaybackDetector" in line or "web.archive.org" in line: + alerts.append( + { + "kind": "wayback", + "level": "warning", + "title": "时光机依赖异常", + "detail": line, + } + ) + break + for line in reversed(recent_lines): + if any(keyword in line for keyword in ("HTTPSConnectionPool", "Connection refused", "Read timed out", "ConnectTimeout")): + alerts.append( + { + "kind": "network", + "level": "warning", + "title": "外部网络波动", + "detail": line, + } + ) + break + return alerts + + +def _extract_available_proxy_count(lines: list[str]) -> int: + for line in reversed(lines): + match = _PROXY_COUNT_RE.search(line) + if match: + return int(match.group(1)) + return 0 + + +def _extract_active_thread_snapshot(lines: list[str]) -> dict: + for line in reversed(lines): + match = _THREAD_COUNT_RE.search(line) + if match: + return { + "active": int(match.group(1)), + "max": int(match.group(2)), + } + return {"active": 0, "max": 0} + + +def _parse_time(raw: str | None) -> datetime | None: + if not raw: + return None + text = str(raw).strip() + if not text: + return None + try: + return datetime.fromisoformat(text) + except ValueError: + pass + for fmt in _TIMESTAMP_FORMATS: + try: + return datetime.strptime(text, fmt) + except ValueError: + continue + return None + + +def _extract_log_time(line: str) -> datetime | None: + if len(line) < 19: + return None + candidates = [line[:26], line[:19]] + for candidate in candidates: + for fmt in _TIMESTAMP_FORMATS: + if len(candidate) != len(datetime.now().strftime(fmt)): + continue + try: + return datetime.strptime(candidate, fmt) + except ValueError: + continue + return None + + +def _filter_lines_since(lines: list[str], started_at: str | None) -> list[str]: + started_time = _parse_time(started_at) + if not started_time: + return lines + filtered = [line for line in lines if (_extract_log_time(line) or started_time) >= started_time] + return filtered or lines + + +def _recent_event(lines: list[str]) -> str: + interesting_keywords = ( + "开始检测", + "获取到", + "当前批次检测完成", + "域名检测任务完成", + "代理池刷新完成", + "没有需要检测的域名", + "检测已停止", + ) + for line in reversed(lines): + if any(keyword in line for keyword in interesting_keywords): + return line + return "" + + +def _load_runtime_state() -> dict: + try: + redis_client = get_redis() + raw = redis_client.get(_RUNTIME_STATE_KEY) + if not raw: + return {} + data = json.loads(raw) + return data if isinstance(data, dict) else {} + except Exception: + return {} + + +def _normalize_recent_warning(runtime_state: dict, recent_lines: list[str], available_proxy_count: int) -> str: + runtime_warning = str(runtime_state.get("recent_warning", "") or "").strip() + if runtime_warning: + if runtime_warning in {"未刷新", "代理未启用", "未配置代理池链接"}: + return "" + if available_proxy_count > 0 and ("无可用代理" in runtime_warning or "未取到可用代理数据" in runtime_warning): + return "" + return runtime_warning + + recent_proxy_warning = next( + (line for line in reversed(recent_lines) if "代理池刷新失败" in line or "无可用代理" in line or "Redis订阅失败" in line), + "", + ) + if available_proxy_count > 0 and "无可用代理" in recent_proxy_warning: + return "" + return recent_proxy_warning + + +def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, available_proxy_count: int) -> dict: + proxy_config = settings_payload.get("proxy_config") or {} + proxy_enable = bool(proxy_config.get("proxy_enable", False)) + allow_direct = bool(proxy_config.get("allow_direct", False)) + refresh_status = str(runtime_state.get("proxy_last_refresh_status", "") or "").strip() + refresh_time = str(runtime_state.get("proxy_last_refresh_time", "") or "").strip() + source_count = len(proxy_config.get("proxy_urls", [])) + source_stats = runtime_state.get("proxy_last_source_stats") or [] + raw_items = int(runtime_state.get("proxy_last_refresh_total_items", 0) or 0) + validated = int(runtime_state.get("proxy_last_validated_count", 0) or 0) + available = int(runtime_state.get("proxy_last_available_count", available_proxy_count) or available_proxy_count) + source_ok_count = sum(1 for item in source_stats if str(item.get("status", "") or "").strip() == "ok") + supplier_empty = bool(source_stats) and source_ok_count == len(source_stats) and raw_items <= 0 + + if not proxy_enable: + return { + "state": "disabled", + "label": "未启用代理", + "detail": "当前使用直连模式，未启用代理池", + "direct_fallback_active": True, + "reason": "proxy_disabled", + "last_refresh_status": refresh_status or "代理未启用", + "last_refresh_time": refresh_time, + "source_count": source_count, + "raw_items": raw_items, + "validated_count": validated, + "available_count": available, + "source_stats": source_stats, + "supplier_empty": False, + } + + if available_proxy_count > 0: + detail = f"代理池当前可用 {available_proxy_count} 个代理，配置来源 {source_count} 个" + if refresh_status: + detail = f"{detail}；最近状态：{refresh_status}" + return { + "state": "healthy", + "label": "代理正常", + "detail": detail, + "direct_fallback_active": False, + "reason": "healthy", + "last_refresh_status": refresh_status, + "last_refresh_time": refresh_time, + "source_count": source_count, + "raw_items": raw_items, + "validated_count": validated, + "available_count": available, + "source_stats": source_stats, + "supplier_empty": False, + } + + if allow_direct: + detail = "代理池当前无可用代理，已自动降级为直连继续执行" + reason = "no_available_proxy" + if supplier_empty: + reason = "supplier_empty_pool" + detail = "代理源最近都返回正常响应，但原始代理数为 0，当前判断为供应池为空；系统已自动降级为直连继续执行" + if refresh_status: + detail = f"{detail}；最近状态：{refresh_status}" + return { + "state": "degraded_direct", + "label": "降级直连", + "detail": detail, + "direct_fallback_active": True, + "reason": reason, + "last_refresh_status": refresh_status or "当前无可用代理", + "last_refresh_time": refresh_time, + "source_count": source_count, + "raw_items": raw_items, + "validated_count": validated, + "available_count": available, + "source_stats": source_stats, + "supplier_empty": supplier_empty, + } + + detail = "代理池当前无可用代理，且未允许直连，检测链路会等待代理恢复" + reason = "no_available_proxy" + if supplier_empty: + reason = "supplier_empty_pool" + detail = "代理源最近都返回正常响应，但原始代理数为 0，当前判断为供应池为空；由于未允许直连，检测链路会等待代理恢复" + if refresh_status: + detail = f"{detail}；最近状态：{refresh_status}" + return { + "state": "blocked_no_proxy", + "label": "等待代理", + "detail": detail, + "direct_fallback_active": False, + "reason": reason, + "last_refresh_status": refresh_status or "当前无可用代理", + "last_refresh_time": refresh_time, + "source_count": source_count, + "raw_items": raw_items, + "validated_count": validated, + "available_count": available, + "source_stats": source_stats, + "supplier_empty": supplier_empty, + } + + def get_detect_status() -> dict: queries = { "pending": "select count(*) from domains where detect_status = 0", @@ -37,14 +297,65 @@ def get_detect_status() -> dict: last_log_time = modified.isoformat() worker_online = (datetime.now(timezone.utc) - modified).total_seconds() < 180 - recent_lines = tail_lines("detect_worker.log", max_lines=80) - recent_proxy_warning = next( - (line for line in reversed(recent_lines) if "代理" in line or "Redis订阅失败" in line), - "", - ) + recent_lines = tail_lines("detect_worker.log", max_lines=160) + runtime_state = _load_runtime_state() + runtime_started_at = runtime.get("latest_start_time", "") if 'runtime' in locals() else "" + recent_lines = _filter_lines_since(recent_lines, runtime_started_at) + available_proxy_count = _extract_available_proxy_count(recent_lines) + active_thread_snapshot = _extract_active_thread_snapshot(recent_lines) + if runtime_state: + available_proxy_count = int(runtime_state.get("available_proxy_count", available_proxy_count) or available_proxy_count) + active_thread_snapshot = { + "active": int(runtime_state.get("active_threads", active_thread_snapshot["active"]) or active_thread_snapshot["active"]), + "max": int(runtime_state.get("max_threads", active_thread_snapshot["max"]) or active_thread_snapshot["max"]), + } + progress_total = sum(progress.values()) + progress_done = progress.get("completed", 0) + progress.get("blacklisted", 0) + progress.get("failed", 0) + progress_percent = round((progress_done / progress_total) * 100, 2) if progress_total > 0 else 0 runtime = detect_worker_runtime() + runtime_started_at = runtime.get("latest_start_time", "") + recent_lines = _filter_lines_since(recent_lines, runtime_started_at) + recent_proxy_warning = _normalize_recent_warning(runtime_state, recent_lines, available_proxy_count) + proxy_runtime = _build_proxy_runtime_snapshot(settings_payload, runtime_state, available_proxy_count) + thread_count_resolution = resolve_thread_count(settings_payload=settings_payload) + effective_thread_count = int(thread_count_resolution["effective_thread_count"]) runtime_settings = get_runtime_settings() worker_online = worker_online or runtime.get("running", False) + if runtime_state.get("service_running") is True: + worker_online = True + if not runtime_state.get("detecting", False) and not progress.get("running", 0): + active_thread_snapshot = {"active": 0, "max": active_thread_snapshot["max"] or effective_thread_count} + settings_summary = { + "thread_count": effective_thread_count, + "thread_count_default": int(thread_count_resolution["default_thread_count"]), + "thread_count_source": str(thread_count_resolution["source"]), + "thread_count_override": thread_count_resolution["override_thread_count"], + "thread_count_node_code": str(thread_count_resolution["node_code"]), + "proxy_enable": settings_payload["proxy_config"].get("proxy_enable", False), + "allow_direct": settings_payload["proxy_config"].get("allow_direct", False), + "proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])), + } + active_job = get_active_detect_job_summary() + runtime_snapshot = { + **runtime, + "detecting": runtime_state.get("detecting", False), + "proxy_runtime_state": proxy_runtime["state"], + "proxy_runtime_label": proxy_runtime["label"], + "proxy_runtime_detail": proxy_runtime["detail"], + "proxy_direct_fallback_active": proxy_runtime["direct_fallback_active"], + "proxy_runtime_reason": proxy_runtime["reason"], + "proxy_supplier_empty": proxy_runtime["supplier_empty"], + } + runs = sync_detect_runs(runtime_snapshot, progress, settings_summary, active_job=active_job) + dependency_alerts = _extract_dependency_alerts(recent_lines) + append_detect_result_projection_if_changed( + detect={ + "active_job": active_job, + "progress": progress, + "phase_label": runtime_state.get("phase", ""), + "phase_detail": runtime_state.get("detail", ""), + } + ) return { "worker_online": worker_online, @@ -54,12 +365,41 @@ def get_detect_status() -> dict: "worker_process_count": runtime.get("process_count", 0), "worker_latest_start_time": runtime.get("latest_start_time", ""), "worker_runtime_message": runtime.get("message", ""), - "thread_count": settings_payload["thread_count"], + "runtime_state": runtime_state, + "phase_label": runtime_state.get("phase", ""), + "phase_detail": runtime_state.get("detail", ""), + "detecting": runtime_state.get("detecting", False), + "thread_count": effective_thread_count, + "thread_count_default": int(thread_count_resolution["default_thread_count"]), + "thread_count_source": str(thread_count_resolution["source"]), + "thread_count_override": thread_count_resolution["override_thread_count"], + "thread_count_node_code": str(thread_count_resolution["node_code"]), + "active_thread_count": active_thread_snapshot["active"], + "max_thread_count": active_thread_snapshot["max"] or effective_thread_count, "proxy_enable": settings_payload["proxy_config"].get("proxy_enable", False), "allow_direct": settings_payload["proxy_config"].get("allow_direct", False), "proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])), - "available_proxy_count": 0, + "available_proxy_count": available_proxy_count, + "proxy_runtime_state": proxy_runtime["state"], + "proxy_runtime_label": proxy_runtime["label"], + "proxy_runtime_detail": proxy_runtime["detail"], + "proxy_direct_fallback_active": proxy_runtime["direct_fallback_active"], + "proxy_runtime_reason": proxy_runtime["reason"], + "proxy_supplier_empty": proxy_runtime["supplier_empty"], + "proxy_last_refresh_status": proxy_runtime["last_refresh_status"], + "proxy_last_refresh_time": proxy_runtime["last_refresh_time"], + "proxy_last_refresh_source_count": proxy_runtime["source_count"], + "proxy_last_refresh_total_items": proxy_runtime["raw_items"], + "proxy_last_validated_count": proxy_runtime["validated_count"], + "proxy_last_available_count": proxy_runtime["available_count"], + "proxy_source_stats": proxy_runtime["source_stats"], + "dependency_alerts": dependency_alerts, "last_worker_log_time": last_log_time, "progress": progress, + "progress_percent": progress_percent, + "recent_event": runtime_state.get("detail") or _recent_event(recent_lines), "recent_warning": recent_proxy_warning, + "log_lines": recent_lines, + "runs": runs, + "active_job": active_job, } diff --git a/domain-api/app/services/domains_service.py b/domain-api/app/services/domains_service.py index 1affcdb..ef016a1 100644 --- a/domain-api/app/services/domains_service.py +++ b/domain-api/app/services/domains_service.py @@ -7,7 +7,7 @@ from app.core.db import get_db DETECT_STATUS_LABELS = { 0: "待检测", - 1: "检测完成", + 1: "检测通过", 2: "检测中", 3: "黑名单", 4: "检测失败", @@ -40,6 +40,14 @@ REVIEW_STATUS_LABELS = { 3: "人工拒绝", } +SOURCE_TYPE_LABELS = { + 1: "聚名一口价", + 2: "聚名过期删除", + 6: "手工录入", + 7: "TXT 导入", + 9: "其它", +} + BEIAN_STATUS_LABELS = { 1: "未检测", 2: "有备案", @@ -47,6 +55,103 @@ BEIAN_STATUS_LABELS = { } +def _format_timestamp(value) -> str | None: + if not value: + return None + return value.isoformat(sep=" ", timespec="seconds") + + +def _json_status_to_text(value) -> str: + if isinstance(value, dict): + status = value.get("status") + else: + status = None + return "是" if status else "否" + + +def _json_state(value) -> str: + if isinstance(value, dict): + return str(value.get("state") or "").strip() + return "" + + +def _json_message(value) -> str: + if isinstance(value, dict): + return str(value.get("message") or "").strip() + return "" + + +def _normalize_step_detail(label: str, value) -> dict: + payload = value if isinstance(value, dict) else {} + state = str(payload.get("state") or "").strip() + return { + "label": label, + "state": state or ("passed" if bool(payload.get("status")) else ""), + "status": bool(payload.get("status")) if isinstance(payload.get("status"), bool) else None, + "message": str(payload.get("message") or "").strip(), + "checked_at": str(payload.get("checked_at") or "").strip(), + "step": str(payload.get("step") or "").strip(), + "raw": payload, + } + + +def _build_step_details(row: tuple) -> list[dict]: + return [ + _normalize_step_detail("百度历史收录", row[16]), + _normalize_step_detail("百度Site收录", row[17]), + { + "label": "标题为中文", + "state": "passed" if bool(row[18]) else "", + "status": bool(row[18]), + "message": "标题含中文" if bool(row[18]) else "", + "checked_at": "", + "step": "中文标题", + "raw": row[18], + }, + _normalize_step_detail("360 Site收录", row[19]), + _normalize_step_detail("Google Site收录", row[20]), + _normalize_step_detail("时光机", row[21]), + _normalize_step_detail("站长之家", row[22]), + _normalize_step_detail("爱站网", row[23]), + ] + + +def _summarize_step_details(step_details: list[dict]) -> dict: + degraded = [item for item in step_details if item.get("state") == "degraded"] + failed = [item for item in step_details if item.get("state") == "failed"] + blacklisted = [item for item in step_details if item.get("state") == "blacklisted"] + return { + "degraded_count": len(degraded), + "failed_count": len(failed), + "blacklisted_count": len(blacklisted), + "has_degraded": bool(degraded), + "has_failed": bool(failed), + "has_blacklisted_step": bool(blacklisted), + "summary_text": ( + f"降级 {len(degraded)} / 失败 {len(failed)} / 命中 {len(blacklisted)}" + if degraded or failed or blacklisted + else "步骤正常" + ), + } + + +def _bool_to_text(value) -> str: + return "是" if bool(value) else "否" + + +def _normalize_detection_update(value) -> bool | None: + if value in (None, "", "skip"): + return None + if isinstance(value, bool): + return value + text = str(value).strip().lower() + if text in {"是", "true", "1", "yes"}: + return True + if text in {"否", "false", "0", "no"}: + return False + raise ValueError("检测结果字段仅支持“是”或“否”") + + def _build_domain_query_parts(filters: dict | None = None) -> tuple[str, str, list[object]]: filters = filters or {} conditions: list[str] = [] @@ -79,6 +184,12 @@ def _build_domain_query_parts(filters: dict | None = None) -> tuple[str, str, li if filters.get("website_url"): conditions.append("coalesce(d.website_url, '') ilike %s") params.append(f"%{str(filters['website_url']).strip()}%") + if filters.get("company_type"): + conditions.append("coalesce(d.company_type, '') = %s") + params.append(str(filters["company_type"]).strip()) + if filters.get("source_type") is not None: + conditions.append("d.source_type = %s") + params.append(int(filters["source_type"])) if filters.get("backlink_gt_10"): conditions.append("coalesce(dd.backlink_count_gt_10, false) = true") @@ -102,6 +213,8 @@ def fetch_domains( beian_year: int | None = None, snapshot_year: str | None = None, website_url: str | None = None, + company_type: str | None = None, + source_type: int | None = None, backlink_gt_10: bool | None = None, ) -> dict: offset = (page - 1) * page_size @@ -115,6 +228,8 @@ def fetch_domains( "beian_year": beian_year, "snapshot_year": snapshot_year, "website_url": website_url, + "company_type": company_type, + "source_type": source_type, "backlink_gt_10": backlink_gt_10, } from_clause, where_clause, params = _build_domain_query_parts(filters) @@ -138,7 +253,18 @@ def fetch_domains( d.snapshot_years, d.backlink_count, d.detect_time, - coalesce(dd.backlink_count_gt_10, false) as backlink_gt_10 + d.source_type, + coalesce(dd.backlink_count_gt_10, false) as backlink_gt_10, + d.expire_date, + d.company_type, + dd.baidu_history, + dd.baidu_site, + dd.is_chinese_title, + dd.qihu360_site, + dd.google_site, + dd.wayback_info, + dd.chinaz_info, + dd.aizhan_info {from_clause} {where_clause} order by d.id desc @@ -148,8 +274,11 @@ def fetch_domains( ) rows = cur.fetchall() - items = [ - { + items = [] + for row in rows: + step_details = _build_step_details(row) + step_summary = _summarize_step_details(step_details) + items.append({ "id": row[0], "domain": row[1], "register_status": REGISTER_STATUS_LABELS.get(row[2], str(row[2])), @@ -166,11 +295,19 @@ def fetch_domains( "beian_year": row[8], "snapshot_years": row[9] or "", "backlink_count": row[10], - "detect_time": row[11].isoformat() if row[11] else None, - "backlink_gt_10": row[12], - } - for row in rows - ] + "detect_time": _format_timestamp(row[11]), + "source_type": row[12], + "source_label": SOURCE_TYPE_LABELS.get(row[12], str(row[12])), + "backlink_gt_10": row[13], + "expire_date": _format_timestamp(row[14]), + "company_type": row[15] or "", + "baidu_history": _json_status_to_text(row[16]), + "baidu_site": _json_status_to_text(row[17]), + "is_chinese_title": _bool_to_text(row[18]), + "qihu360_site": _json_status_to_text(row[19]), + "google_site": _json_status_to_text(row[20]), + "step_summary": step_summary, + }) return { "list": items, "page": page, @@ -180,12 +317,116 @@ def fetch_domains( } +def fetch_domain_detail(domain_id: int) -> dict | None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + select + d.id, + d.domain, + d.register_status, + d.use_status, + d.detect_status, + d.review_status, + d.has_beian, + d.website_url, + d.beian_year, + d.snapshot_years, + d.backlink_count, + d.detect_time, + d.source_type, + coalesce(dd.backlink_count_gt_10, false) as backlink_gt_10, + d.expire_date, + d.company_type, + dd.baidu_history, + dd.baidu_site, + dd.is_chinese_title, + dd.qihu360_site, + dd.google_site, + dd.wayback_info, + dd.chinaz_info, + dd.aizhan_info, + dd.juziseo_info, + dd.jucha_info + from domains d + left join domain_detections dd on dd.domain_id = d.id + where d.id = %s + limit 1 + """, + (int(domain_id),), + ) + row = cur.fetchone() + if not row: + return None + + step_details = [ + _normalize_step_detail("百度历史收录", row[16]), + _normalize_step_detail("百度Site收录", row[17]), + { + "label": "标题为中文", + "state": "passed" if bool(row[18]) else "", + "status": bool(row[18]), + "message": "标题含中文" if bool(row[18]) else "", + "checked_at": "", + "step": "中文标题", + "raw": row[18], + }, + _normalize_step_detail("360 Site收录", row[19]), + _normalize_step_detail("Google Site收录", row[20]), + _normalize_step_detail("时光机", row[21]), + _normalize_step_detail("站长之家", row[22]), + _normalize_step_detail("爱站网", row[23]), + _normalize_step_detail("桔子SEO", row[24]), + _normalize_step_detail("聚查", row[25]), + ] + step_summary = _summarize_step_details(step_details) + return { + "id": row[0], + "domain": row[1], + "register_status": REGISTER_STATUS_LABELS.get(row[2], str(row[2])), + "register_status_code": row[2], + "use_status": USE_STATUS_LABELS.get(row[3], str(row[3])), + "use_status_code": row[3], + "detect_status": DETECT_STATUS_LABELS.get(row[4], str(row[4])), + "detect_status_code": row[4], + "review_status": REVIEW_STATUS_LABELS.get(row[5], str(row[5])), + "review_status_code": row[5], + "has_beian": BEIAN_STATUS_LABELS.get(row[6], str(row[6])), + "has_beian_code": row[6], + "website_url": row[7] or "", + "beian_year": row[8], + "snapshot_years": row[9] or "", + "backlink_count": row[10], + "detect_time": _format_timestamp(row[11]), + "source_type": row[12], + "source_label": SOURCE_TYPE_LABELS.get(row[12], str(row[12])), + "backlink_gt_10": row[13], + "expire_date": _format_timestamp(row[14]), + "company_type": row[15] or "", + "step_summary": step_summary, + "step_details": step_details, + "raw_detection": { + "baidu_history": row[16], + "baidu_site": row[17], + "is_chinese_title": row[18], + "qihu360_site": row[19], + "google_site": row[20], + "wayback_info": row[21], + "chinaz_info": row[22], + "aizhan_info": row[23], + "juziseo_info": row[24], + "jucha_info": row[25], + }, + } + + def domain_filter_options() -> dict: return { "register_status": [ {"label": label, "value": value} for value, label in REGISTER_STATUS_LABELS.items() - if value in (2, 3, 4, 5, 6, 7, 8, 10) + if value in (0, 2, 3, 4, 5, 6, 7, 8, 9, 10) ], "detect_status": [ {"label": label, "value": value} @@ -204,6 +445,10 @@ def domain_filter_options() -> dict: {"label": "有备案", "value": 2}, {"label": "无备案", "value": 3}, ], + "source_type": [ + {"label": label, "value": value} + for value, label in SOURCE_TYPE_LABELS.items() + ], "supports_backlink_gt_10": True, "supports_txt_export": True, "supports_excel_export": True, @@ -225,11 +470,35 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict: "detect_time", "website_url", "backlink_count", + "baidu_history", + "baidu_site", + "is_chinese_title", + "qihu360_site", + "google_site", } payload = {key: value for key, value in updates.items() if key in allowed_fields and value not in (None, "", "skip")} if not payload: raise ValueError("没有可更新的字段") + domain_fields = { + "review_status", + "expire_date", + "has_beian", + "beian_year", + "snapshot_years", + "company_type", + "detect_time", + "website_url", + "backlink_count", + } + detection_fields = { + "baidu_history", + "baidu_site", + "is_chinese_title", + "qihu360_site", + "google_site", + } + updated_count = 0 with get_db() as conn: with conn.cursor() as cur: @@ -238,6 +507,8 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict: params: list[object] = [] for field, value in payload.items(): + if field not in domain_fields: + continue if field == "backlink_count": set_parts.append("backlink_count = %s") params.append(int(value)) @@ -245,11 +516,24 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict: set_parts.append(f"{field} = %s") params.append(value) - params.append(domain_id) - cur.execute( - f"update domains set {', '.join(set_parts)}, update_time = now() where id = %s", - tuple(params), - ) + if set_parts: + params.append(domain_id) + cur.execute( + f"update domains set {', '.join(set_parts)}, update_time = now() where id = %s", + tuple(params), + ) + + detection_payload: dict[str, object] = {} + for field in detection_fields: + if field not in payload: + continue + normalized = _normalize_detection_update(payload[field]) + if normalized is None: + continue + if field == "is_chinese_title": + detection_payload[field] = normalized + else: + detection_payload[field] = {"status": normalized} if "backlink_count" in payload: backlink_gt_10 = int(payload["backlink_count"]) > 10 @@ -268,6 +552,36 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict: (domain_id, backlink_gt_10), ) + if detection_payload: + cur.execute("select id from domain_detections where domain_id = %s", (domain_id,)) + existing_detection = cur.fetchone() + if existing_detection: + detection_set_parts: list[str] = [] + detection_params: list[object] = [] + for field, value in detection_payload.items(): + detection_set_parts.append(f"{field} = %s") + detection_params.append(value) + detection_params.append(domain_id) + cur.execute( + f""" + update domain_detections + set {', '.join(detection_set_parts)}, update_time = now() + where domain_id = %s + """, + tuple(detection_params), + ) + else: + insert_fields = ["domain_id", *detection_payload.keys(), "create_time", "update_time"] + placeholders = ["%s"] * (1 + len(detection_payload)) + ["now()", "now()"] + insert_params = [domain_id, *detection_payload.values()] + cur.execute( + f""" + insert into domain_detections ({', '.join(insert_fields)}) + values ({', '.join(placeholders)}) + """, + tuple(insert_params), + ) + updated_count += 1 conn.commit() diff --git a/domain-api/app/services/export_service.py b/domain-api/app/services/export_service.py index 435ba43..3756e60 100644 --- a/domain-api/app/services/export_service.py +++ b/domain-api/app/services/export_service.py @@ -23,8 +23,15 @@ EXPORT_HEADERS = [ ("use_status", "使用状态"), ("detect_status", "检测状态"), ("review_status", "复核状态"), + ("expire_date", "过期时间"), + ("company_type", "单位性质"), ("has_beian", "备案状态"), ("website_url", "首页网址"), + ("baidu_history", "百度历史收录"), + ("baidu_site", "百度Site收录"), + ("is_chinese_title", "标题为中文"), + ("qihu360_site", "360 Site收录"), + ("google_site", "Google Site收录"), ("beian_year", "备案年份"), ("snapshot_years", "快照年份"), ("backlink_count", "友链数"), @@ -33,6 +40,18 @@ EXPORT_HEADERS = [ ] +def _json_status_to_text(value) -> str: + if isinstance(value, dict): + status = value.get("status") + else: + status = None + return "是" if status else "否" + + +def _bool_to_text(value) -> str: + return "是" if bool(value) else "否" + + def _normalize_payload(payload: dict) -> dict: data = dict(payload or {}) data["page"] = int(data.get("page", 1) or 1) @@ -68,8 +87,15 @@ def _query_export_rows(payload: dict) -> list[dict]: d.use_status, d.detect_status, d.review_status, + d.expire_date, + d.company_type, d.has_beian, d.website_url, + dd.baidu_history, + dd.baidu_site, + dd.is_chinese_title, + dd.qihu360_site, + dd.google_site, d.beian_year, d.snapshot_years, d.backlink_count, @@ -93,13 +119,20 @@ def _query_export_rows(payload: dict) -> list[dict]: "use_status": USE_STATUS_LABELS.get(row[2], str(row[2])), "detect_status": DETECT_STATUS_LABELS.get(row[3], str(row[3])), "review_status": REVIEW_STATUS_LABELS.get(row[4], str(row[4])), - "has_beian": BEIAN_STATUS_LABELS.get(row[5], str(row[5])), - "website_url": row[6] or "", - "beian_year": row[7] or "", - "snapshot_years": row[8] or "", - "backlink_count": row[9] or 0, - "backlink_gt_10": "是" if row[10] else "否", - "detect_time": row[11].isoformat(sep=" ", timespec="seconds") if row[11] else "", + "expire_date": row[5].isoformat(sep=" ", timespec="seconds") if row[5] else "", + "company_type": row[6] or "", + "has_beian": BEIAN_STATUS_LABELS.get(row[7], str(row[7])), + "website_url": row[8] or "", + "baidu_history": _json_status_to_text(row[9]), + "baidu_site": _json_status_to_text(row[10]), + "is_chinese_title": _bool_to_text(row[11]), + "qihu360_site": _json_status_to_text(row[12]), + "google_site": _json_status_to_text(row[13]), + "beian_year": row[14] or "", + "snapshot_years": row[15] or "", + "backlink_count": row[16] or 0, + "backlink_gt_10": "是" if row[17] else "否", + "detect_time": row[18].isoformat(sep=" ", timespec="seconds") if row[18] else "", } ) return result diff --git a/domain-api/app/services/import_task_service.py b/domain-api/app/services/import_task_service.py index bf4ce35..98e1a5f 100644 --- a/domain-api/app/services/import_task_service.py +++ b/domain-api/app/services/import_task_service.py @@ -10,6 +10,11 @@ from app.services.import_worker_service import import_domains_from_path _IMPORT_TASK_LOCK = threading.Lock() +_SOURCE_TYPE_LABELS = { + 6: "手工录入", + 7: "TXT 导入", + 9: "其它", +} def _now() -> str: @@ -24,6 +29,12 @@ def _save_tasks(tasks: list[dict]) -> None: save_import_records(tasks) +def _append_log_locked(target: dict, message: str) -> None: + target.setdefault("logs", []) + target["logs"].append(f"[{_now()}] {message}") + target["logs"] = target["logs"][-200:] + + def _update_task(task_id: str, **patch: object) -> dict | None: with _IMPORT_TASK_LOCK: tasks = load_import_records() @@ -36,13 +47,73 @@ def _update_task(task_id: str, **patch: object) -> dict | None: return dict(target) +def _update_task_with_log(task_id: str, log_message: str, **patch: object) -> dict | None: + with _IMPORT_TASK_LOCK: + tasks = load_import_records() + target = next((item for item in tasks if item["task_id"] == task_id), None) + if not target: + return None + target.update(patch) + _append_log_locked(target, log_message) + target["updated_at"] = _now() + _save_tasks(tasks) + return dict(target) + + +def _phase_label(phase: str) -> str: + mapping = { + "queued": "排队中", + "reading": "读取文件中", + "normalizing": "清洗中", + "importing": "入库中", + "completed": "已完成", + "failed": "失败", + } + return mapping.get(phase, phase) + + +def _source_label(source_type: int) -> str: + return _SOURCE_TYPE_LABELS.get(int(source_type or 7), "未知") + + def _run_import_task(task_id: str, file_path: str, source_type: int = 7) -> None: - _update_task(task_id, status="running", started_at=_now(), message="导入任务开始执行") + _update_task_with_log( + task_id, + f"导入任务开始执行，来源类型：{_source_label(source_type)}", + status="running", + started_at=_now(), + message=f"导入任务开始执行，来源类型：{_source_label(source_type)}", + phase="reading", + phase_label=_phase_label("reading"), + ) try: - result = import_domains_from_path(Path(file_path), source_type=source_type) - stats = result.get("stats", {}) - _update_task( + path = Path(file_path) + _update_task_with_log( task_id, + f"开始读取文件：{path.name}", + phase="reading", + phase_label=_phase_label("reading"), + ) + raw_lines = path.read_text(encoding="utf-8", errors="replace").splitlines() + total_lines = len(raw_lines) + non_empty = sum(1 for line in raw_lines if line.strip()) + _update_task_with_log( + task_id, + f"文件读取完成，共 {total_lines} 行，非空 {non_empty} 行", + phase="normalizing", + phase_label=_phase_label("normalizing"), + message=f"文件读取完成，准备清洗 {non_empty} 条域名", + ) + + result = import_domains_from_path(path, source_type=source_type) + stats = result.get("stats", {}) + _update_task_with_log( + task_id, + ( + f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，" + f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}，" + f"来源类型 {result.get('source_label') or _source_label(source_type)}" + ), status="completed", completed_at=_now(), result=result, @@ -50,13 +121,18 @@ def _run_import_task(task_id: str, file_path: str, source_type: int = 7) -> None f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，" f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}" ), + phase="completed", + phase_label=_phase_label("completed"), ) except Exception as exc: - _update_task( + _update_task_with_log( task_id, + f"导入失败：{exc}", status="failed", completed_at=_now(), message=f"导入失败：{exc}", + phase="failed", + phase_label=_phase_label("failed"), ) @@ -71,12 +147,16 @@ def create_import_task(content: bytes, filename: str, source_type: int = 7) -> d "filename": safe_name, "stored_path": str(target), "source_type": source_type, + "source_label": _source_label(source_type), "status": "queued", - "message": "文件已接收，等待处理", + "message": f"文件已接收，等待后台处理，来源类型：{_source_label(source_type)}", "created_at": _now(), "updated_at": _now(), "started_at": "", "completed_at": "", + "phase": "queued", + "phase_label": _phase_label("queued"), + "logs": [f"[{_now()}] 文件已接收，等待后台处理，来源类型：{_source_label(source_type)}"], "result": None, } @@ -104,6 +184,9 @@ def retry_import_task(task_id: str) -> dict: target["completed_at"] = "" target["updated_at"] = _now() target["result"] = None + target["phase"] = "queued" + target["phase_label"] = _phase_label("queued") + target["logs"] = [f"[{_now()}] 任务已重新加入队列，等待后台执行"] _save_tasks(tasks) stored_path = target["stored_path"] source_type = int(target.get("source_type", 7)) diff --git a/domain-api/app/services/import_worker_service.py b/domain-api/app/services/import_worker_service.py index 0bce79f..2846974 100644 --- a/domain-api/app/services/import_worker_service.py +++ b/domain-api/app/services/import_worker_service.py @@ -8,6 +8,11 @@ from app.core.files import import_root DOMAIN_PATTERN = re.compile(r"^(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+(com|net)$", re.IGNORECASE) +SOURCE_TYPE_LABELS = { + 6: "手工录入", + 7: "TXT 导入", + 9: "其它", +} def normalize_domain(value: str) -> str | None: @@ -92,6 +97,8 @@ def import_domains_from_path(file_path: Path, source_type: int = 7) -> dict: } return { "filename": file_path.name, + "source_type": source_type, + "source_label": SOURCE_TYPE_LABELS.get(int(source_type or 7), "未知"), "stats": stats, } diff --git a/domain-api/app/services/juming_service.py b/domain-api/app/services/juming_service.py new file mode 100644 index 0000000..be8c9ed --- /dev/null +++ b/domain-api/app/services/juming_service.py @@ -0,0 +1,494 @@ +from __future__ import annotations + +import ast +import pickle +import re +import sys +from collections.abc import Callable +from datetime import date, datetime, timedelta +from pathlib import Path + +from requests.cookies import RequestsCookieJar + +from app.core.db import get_db +from app.core.redis_client import get_redis +from app.core.config import settings +from app.core.files import read_runtime_json, write_runtime_json +from app.services.import_worker_service import normalize_domain + + +DOMAIN_ROOT = Path(settings.domain_root) +if str(DOMAIN_ROOT) not in sys.path: + sys.path.insert(0, str(DOMAIN_ROOT)) + +from detect.juming import JM # type: ignore # noqa: E402 +from detect.jucha import JC # type: ignore # noqa: E402 + + +YKJ_DOMAIN_PATTERN = re.compile(r" dict: + return { + "mode": "delete_list", + "page_start": 1, + "page_size": 500, + "page_count": 1, + "crawl_date": date.today().isoformat(), + "auto_date": True, + } + + +def get_juming_preferences() -> dict: + defaults = _default_juming_preferences() + stored = read_runtime_json(JUMING_PREFERENCES_FILE, default={}) + payload = { + "mode": str(stored.get("mode", defaults["mode"])) if stored else defaults["mode"], + "page_start": int(stored.get("page_start", defaults["page_start"])) if stored else defaults["page_start"], + "page_size": int(stored.get("page_size", defaults["page_size"])) if stored else defaults["page_size"], + "page_count": int(stored.get("page_count", defaults["page_count"])) if stored else defaults["page_count"], + "crawl_date": str(stored.get("crawl_date", defaults["crawl_date"])) if stored else defaults["crawl_date"], + "auto_date": bool(stored.get("auto_date", defaults["auto_date"])) if stored else defaults["auto_date"], + } + if payload["mode"] not in {"delete_list", "fixed_price"}: + payload["mode"] = defaults["mode"] + payload["page_start"] = max(payload["page_start"], 1) + payload["page_size"] = min(max(payload["page_size"], 1), 1000) + payload["page_count"] = min(max(payload["page_count"], 1), 20) + if not payload["crawl_date"]: + payload["crawl_date"] = defaults["crawl_date"] + return payload + + +def update_juming_preferences(payload: dict) -> dict: + current = get_juming_preferences() + next_payload = { + "mode": str(payload.get("mode", current["mode"]) or current["mode"]), + "page_start": int(payload.get("page_start", current["page_start"]) or current["page_start"]), + "page_size": int(payload.get("page_size", current["page_size"]) or current["page_size"]), + "page_count": int(payload.get("page_count", current["page_count"]) or current["page_count"]), + "crawl_date": str(payload.get("crawl_date", current["crawl_date"]) or current["crawl_date"]), + "auto_date": bool(payload.get("auto_date", current["auto_date"])), + } + if next_payload["mode"] not in {"delete_list", "fixed_price"}: + raise ValueError("无效的聚名采集类型") + next_payload["page_start"] = max(next_payload["page_start"], 1) + next_payload["page_size"] = min(max(next_payload["page_size"], 1), 1000) + next_payload["page_count"] = min(max(next_payload["page_count"], 1), 20) + write_runtime_json(JUMING_PREFERENCES_FILE, next_payload) + return next_payload + + +def _cookie_dict_to_jar(cookie_dict: dict[str, str]) -> RequestsCookieJar: + cookie_jar = RequestsCookieJar() + for name, value in cookie_dict.items(): + cookie_jar.set(name, value) + return cookie_jar + + +def _emit_log(log: Callable[[str], None] | None, message: str) -> None: + if log: + log(message) + + +def _check_stop(should_stop: Callable[[], bool] | None) -> None: + if should_stop and should_stop(): + raise TaskStoppedError("任务已停止") + + +def _cookie_jar_to_dict(cookie_jar: RequestsCookieJar) -> dict[str, str]: + return {str(cookie.name): str(cookie.value) for cookie in cookie_jar} + + +def _persist_juming_cookie(cookie_jar: RequestsCookieJar) -> None: + JUMING_COOKIE_FILE.parent.mkdir(parents=True, exist_ok=True) + with JUMING_COOKIE_FILE.open("wb") as handle: + pickle.dump(cookie_jar, handle) + + try: + redis_client = get_redis() + redis_client.set("domain_tool:juming_cookies", str(_cookie_jar_to_dict(cookie_jar))) + except Exception: + pass + + +def _jucha_cookie_status() -> dict: + if JUCHA_COOKIE_FILE.exists(): + return { + "cookie_ready": True, + "cookie_file": str(JUCHA_COOKIE_FILE), + } + return { + "cookie_ready": False, + "cookie_file": str(JUCHA_COOKIE_FILE), + } + + +def _load_juming_cookie() -> tuple[RequestsCookieJar | None, str]: + if JUMING_COOKIE_FILE.exists(): + try: + with JUMING_COOKIE_FILE.open("rb") as handle: + loaded = pickle.load(handle) + if isinstance(loaded, RequestsCookieJar): + return loaded, "local" + if isinstance(loaded, dict): + return _cookie_dict_to_jar({str(k): str(v) for k, v in loaded.items()}), "local" + except Exception: + pass + + for legacy_path in LEGACY_JUMING_COOKIE_FILES: + if not legacy_path.exists() or legacy_path == JUMING_COOKIE_FILE: + continue + try: + with legacy_path.open("rb") as handle: + loaded = pickle.load(handle) + if isinstance(loaded, RequestsCookieJar): + _persist_juming_cookie(loaded) + return loaded, f"migrated:{legacy_path}" + if isinstance(loaded, dict): + cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in loaded.items()}) + _persist_juming_cookie(cookie_jar) + return cookie_jar, f"migrated:{legacy_path}" + except Exception: + continue + + try: + redis_client = get_redis() + raw = redis_client.get("domain_tool:juming_cookies") + if raw: + parsed = ast.literal_eval(raw) + if isinstance(parsed, dict) and parsed: + cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in parsed.items()}) + try: + with JUMING_COOKIE_FILE.open("wb") as handle: + pickle.dump(cookie_jar, handle) + except Exception: + pass + return cookie_jar, "redis" + except Exception: + pass + + return None, "missing" + + +def get_juming_status() -> dict: + cookie_jar, storage = _load_juming_cookie() + status = { + "cookie_ready": cookie_jar is not None, + "cookie_storage": storage, + "cookie_file": str(JUMING_COOKIE_FILE), + "cookie_count": len(_cookie_jar_to_dict(cookie_jar)) if cookie_jar is not None else 0, + "jucha": _jucha_cookie_status(), + "supported_modes": [ + {"label": "聚名一口价", "value": "fixed_price", "source_type": FIXED_PRICE_SOURCE_TYPE}, + {"label": "聚名过期删除", "value": "delete_list", "source_type": DELETE_LIST_SOURCE_TYPE}, + ], + "defaults": get_juming_preferences(), + } + status["linked_jucha"] = { + "attempted": False, + "ok": bool(status["jucha"]["cookie_ready"]), + "message": "聚查登录态已就绪" if status["jucha"]["cookie_ready"] else "尚未检测到聚查 Cookie", + } + return status + + +def login_juming(email: str, password: str) -> dict: + account = str(email or "").strip() + secret = str(password or "").strip() + if not account or not secret: + raise ValueError("请输入聚名账号和密码") + + jm = JM() + jm.load_cookies() + login_result = jm.user_zh_p_login(account, secret) + if not login_result[0]: + raise ValueError(f"聚名登录失败: {login_result[1]}") + + jm.save_cookies() + _persist_juming_cookie(jm.cookie) + + linked_jucha = { + "attempted": True, + "ok": False, + "message": "未执行", + } + try: + jc = JC() + jc.load_juming_cookies() + linked_ok, linked_message = jc.auth_login() + linked_jucha["ok"] = bool(linked_ok) + linked_jucha["message"] = str(linked_message) + if linked_ok: + jc.save_cookies() + except Exception as exc: + linked_jucha["message"] = f"聚查联名登录失败: {exc}" + + status = get_juming_status() + status["linked_jucha"] = linked_jucha + return status + + +def login_jucha_with_juming_cookie() -> dict: + if not JUMING_COOKIE_FILE.exists(): + raise ValueError("请先完成聚名登录，当前未检测到聚名 Cookie") + + jc = JC() + jc.load_juming_cookies() + linked_ok, linked_message = jc.auth_login() + if not linked_ok: + raise ValueError(f"聚查登录失败: {linked_message}") + jc.save_cookies() + + status = get_juming_status() + status["linked_jucha"] = { + "attempted": True, + "ok": True, + "message": str(linked_message), + } + return status + + +def upload_juming_cookie(filename: str, content: bytes) -> dict: + name = Path(filename or "juming_cookies.pkl").name.lower() + if not (name.endswith(".pkl") or name.endswith(".pickle") or name.endswith(".json") or name.endswith(".txt")): + raise ValueError("仅支持上传 .pkl / .pickle / .json / .txt 格式的聚名 Cookie 文件") + + cookie_jar: RequestsCookieJar | None = None + parse_error: str | None = None + + if name.endswith(".pkl") or name.endswith(".pickle"): + try: + loaded = pickle.loads(content) + if isinstance(loaded, RequestsCookieJar): + cookie_jar = loaded + elif isinstance(loaded, dict): + cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in loaded.items()}) + except Exception as exc: + parse_error = str(exc) + else: + try: + text = content.decode("utf-8") + parsed = ast.literal_eval(text) + if isinstance(parsed, dict): + cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in parsed.items()}) + except Exception as exc: + parse_error = str(exc) + + if cookie_jar is None: + raise ValueError(f"聚名 Cookie 文件解析失败: {parse_error or '内容不符合预期'}") + + if not _cookie_jar_to_dict(cookie_jar): + raise ValueError("聚名 Cookie 文件为空，未检测到有效 Cookie") + + _persist_juming_cookie(cookie_jar) + return { + "cookie_ready": True, + "cookie_storage": "upload", + "cookie_file": str(JUMING_COOKIE_FILE), + "cookie_count": len(_cookie_jar_to_dict(cookie_jar)), + } + + +def _insert_domains( + domains: list[str], + source_type: int, + log: Callable[[str], None] | None = None, + should_stop: Callable[[], bool] | None = None, +) -> dict: + total = len(domains) + normalized_rows: list[tuple[str, str]] = [] + invalid = 0 + + _emit_log(log, f"开始入库处理，共收到 {total} 个原始域名") + for value in domains: + _check_stop(should_stop) + normalized = normalize_domain(value) + if not normalized: + invalid += 1 + continue + tld = normalized.rsplit(".", 1)[-1] + normalized_rows.append((normalized, tld)) + + existing_set: set[str] = set() + inserted = 0 + + with get_db() as conn: + with conn.cursor() as cur: + normalized_domains = [row[0] for row in normalized_rows] + if normalized_domains: + cur.execute("select domain from domains where domain = any(%s)", (normalized_domains,)) + existing_set = {row[0] for row in cur.fetchall()} + if existing_set: + _emit_log(log, f"检测到 {len(existing_set)} 个已存在域名，将自动跳过") + + inserted_since_commit = 0 + for domain, tld in normalized_rows: + _check_stop(should_stop) + if domain in existing_set: + continue + cur.execute( + """ + insert into domains ( + domain, tld, source_type, use_status, detect_status, register_status, + has_beian, company_type, website_url, beian_year, snapshot_years, + expire_date, create_time, update_time, review_status, detect_time, + backlink_count, jucha_status, juziseo_status + ) values ( + %s, %s, %s, 0, 0, 0, + 1, null, null, null, null, + null, now(), now(), 0, null, + 0, 0, 0 + ) + returning id + """, + (domain, tld, source_type), + ) + domain_id = cur.fetchone()[0] + cur.execute( + """ + insert into detect_tasks (domain_id, task_type, status, priority, retry_count, create_time, update_time) + values (%s, 1, 1, 5, 0, now(), now()) + """, + (domain_id,), + ) + inserted += 1 + inserted_since_commit += 1 + if inserted_since_commit >= 500: + conn.commit() + inserted_since_commit = 0 + conn.commit() + + valid = len(normalized_rows) + exists = len(existing_set) + _emit_log(log, f"入库完成：有效 {valid}，新增 {inserted}，已存在 {exists}，无效 {invalid}") + return { + "total": total, + "valid": valid, + "added": inserted, + "exists": exists, + "invalid": invalid, + "failed": max(valid - exists - inserted, 0), + } + + +def _crawl_fixed_price( + page_start: int, + page_size: int, + page_count: int, + log: Callable[[str], None] | None = None, + should_stop: Callable[[], bool] | None = None, +) -> tuple[list[str], list[dict[str, int]]]: + cookie_jar, _ = _load_juming_cookie() + jm = JM() + jm.cookie = cookie_jar or RequestsCookieJar() + + domains: list[str] = [] + pages: list[dict[str, int]] = [] + current_page = page_start + _emit_log(log, f"开始采集一口价域名：起始页 {page_start}，每页 {page_size}，最多 {page_count} 页") + + for _ in range(page_count): + _check_stop(should_stop) + _emit_log(log, f"正在抓取第 {current_page} 页") + success, html = jm.ykj_get_list(page=current_page, page_size=page_size) + if not success: + raise RuntimeError(str(html)) + page_domains = [item.strip() for item in YKJ_DOMAIN_PATTERN.findall(html) if item.strip()] + domains.extend(page_domains) + pages.append({"page": current_page, "count": len(page_domains)}) + _emit_log(log, f"第 {current_page} 页抓取到 {len(page_domains)} 个域名，累计 {len(domains)} 个") + if len(page_domains) < page_size: + _emit_log(log, "当前页返回数量小于分页数量，判定已到末页，停止继续抓取") + break + current_page += 1 + + return domains, pages + + +def _crawl_delete_list( + crawl_date: str, + auto_date: bool, + log: Callable[[str], None] | None = None, + should_stop: Callable[[], bool] | None = None, +) -> tuple[list[str], list[dict[str, int]]]: + cookie_jar, _ = _load_juming_cookie() + jm = JM() + jm.cookie = cookie_jar or RequestsCookieJar() + + start_date = datetime.strptime(crawl_date, "%Y-%m-%d").date() + end_date = date.today() + timedelta(days=4) + current_date = start_date + domains: list[str] = [] + dates: list[dict[str, int]] = [] + _emit_log(log, f"开始采集删除列表：起始日期 {crawl_date}，自动追加日期 {'开启' if auto_date else '关闭'}") + + while current_date <= end_date: + _check_stop(should_stop) + _emit_log(log, f"正在抓取 {current_date.isoformat()} 的删除列表") + domains_for_date = [item.strip() for item in jm.new_cha_del(current_date.isoformat()) if item.strip()] + domains.extend(domains_for_date) + dates.append({"date": current_date.isoformat(), "count": len(domains_for_date)}) + _emit_log(log, f"{current_date.isoformat()} 抓取到 {len(domains_for_date)} 个域名，累计 {len(domains)} 个") + if not auto_date: + break + current_date += timedelta(days=1) + + return domains, dates + + +def crawl_juming( + payload: dict, + log: Callable[[str], None] | None = None, + should_stop: Callable[[], bool] | None = None, +) -> dict: + mode = str(payload.get("mode") or "delete_list").strip() + if mode not in {"fixed_price", "delete_list"}: + raise ValueError("仅支持 fixed_price 或 delete_list") + + cookie_jar, storage = _load_juming_cookie() + if cookie_jar is None: + raise ValueError("未找到聚名 Cookie，请先在桌面版系统设置完成聚名登录，或将 Cookie 同步到服务器") + _emit_log(log, f"检测到聚名登录态，来源：{storage}") + _check_stop(should_stop) + + if mode == "fixed_price": + page_start = max(int(payload.get("page_start") or 1), 1) + page_size = min(max(int(payload.get("page_size") or 500), 1), 1000) + page_count = min(max(int(payload.get("page_count") or 1), 1), 20) + domains, pages = _crawl_fixed_price(page_start, page_size, page_count, log=log, should_stop=should_stop) + stats = _insert_domains(domains, FIXED_PRICE_SOURCE_TYPE, log=log, should_stop=should_stop) + return { + "mode": mode, + "cookie_storage": storage, + "pages": pages, + "domains_found": len(domains), + "stats": stats, + "sample_domains": domains[:20], + } + + crawl_date = str(payload.get("crawl_date") or date.today().isoformat()) + auto_date = bool(payload.get("auto_date", True)) + domains, dates = _crawl_delete_list(crawl_date, auto_date, log=log, should_stop=should_stop) + stats = _insert_domains(domains, DELETE_LIST_SOURCE_TYPE, log=log, should_stop=should_stop) + return { + "mode": mode, + "cookie_storage": storage, + "dates": dates, + "domains_found": len(domains), + "stats": stats, + "sample_domains": domains[:20], + } diff --git a/domain-api/app/services/juming_task_service.py b/domain-api/app/services/juming_task_service.py new file mode 100644 index 0000000..4ba7e05 --- /dev/null +++ b/domain-api/app/services/juming_task_service.py @@ -0,0 +1,206 @@ +from __future__ import annotations + +import threading +from datetime import datetime +from uuid import uuid4 + +from app.core.files import load_juming_records, save_juming_records +from app.services.juming_service import TaskStoppedError, crawl_juming + + +_JUMING_TASK_LOCK = threading.Lock() +_MAX_LOG_LINES = 400 + + +def _now() -> str: + return datetime.now().isoformat(sep=" ", timespec="seconds") + + +def list_juming_tasks() -> list[dict]: + return load_juming_records() + + +def _save_tasks(tasks: list[dict]) -> None: + save_juming_records(tasks) + + +def _append_log(task_id: str, message: str) -> None: + with _JUMING_TASK_LOCK: + tasks = load_juming_records() + target = next((item for item in tasks if item["task_id"] == task_id), None) + if not target: + return + logs = list(target.get("logs") or []) + logs.append(f"[{_now()}] {message}") + target["logs"] = logs[-_MAX_LOG_LINES:] + target["updated_at"] = _now() + _save_tasks(tasks) + + +def _update_task(task_id: str, **patch: object) -> dict | None: + with _JUMING_TASK_LOCK: + tasks = load_juming_records() + target = next((item for item in tasks if item["task_id"] == task_id), None) + if not target: + return None + target.update(patch) + target["updated_at"] = _now() + _save_tasks(tasks) + return dict(target) + + +def _is_stop_requested(task_id: str) -> bool: + tasks = load_juming_records() + target = next((item for item in tasks if item["task_id"] == task_id), None) + return bool(target and target.get("cancel_requested")) + + +def _set_phase(task_id: str, phase: str, message: str | None = None) -> None: + phase_labels = { + "queued": "排队中", + "starting": "启动中", + "fetching": "抓取中", + "importing": "入库中", + "completed": "已完成", + "failed": "失败", + "stopping": "停止中", + "stopped": "已停止", + } + patch: dict[str, object] = { + "phase": phase, + "phase_label": phase_labels.get(phase, phase), + } + if message: + patch["message"] = message + _update_task(task_id, **patch) + + +def _log_and_track_phase(task_id: str, message: str) -> None: + if "开始采集" in message or "正在抓取" in message: + _set_phase(task_id, "fetching", message) + elif "开始入库处理" in message or "入库完成" in message or "已存在域名" in message: + _set_phase(task_id, "importing", message) + _append_log(task_id, message) + + +def _run_juming_task(task_id: str, payload: dict) -> None: + _update_task(task_id, status="running", started_at=_now(), message="聚名采集任务开始执行", cancel_requested=False) + _set_phase(task_id, "starting", "聚名采集任务开始执行") + _append_log(task_id, "任务已启动，正在准备读取聚名登录态") + try: + result = crawl_juming( + payload, + log=lambda message: _log_and_track_phase(task_id, str(message)), + should_stop=lambda: _is_stop_requested(task_id), + ) + stats = result.get("stats", {}) + _update_task( + task_id, + status="completed", + completed_at=_now(), + result=result, + cancel_requested=False, + message=( + f"采集完成：抓取 {result.get('domains_found', 0)} 个域名，" + f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}" + ), + ) + _set_phase(task_id, "completed") + _append_log(task_id, "任务执行完成") + except TaskStoppedError as exc: + _update_task( + task_id, + status="stopped", + completed_at=_now(), + cancel_requested=False, + message=str(exc), + ) + _set_phase(task_id, "stopped") + _append_log(task_id, "任务已按请求停止") + except Exception as exc: + _update_task( + task_id, + status="failed", + completed_at=_now(), + cancel_requested=False, + message=f"采集失败：{exc}", + ) + _set_phase(task_id, "failed") + _append_log(task_id, f"任务执行失败：{exc}") + + +def create_juming_task(payload: dict) -> dict: + task_id = uuid4().hex + mode = str(payload.get("mode") or "delete_list").strip() or "delete_list" + record = { + "task_id": task_id, + "mode": mode, + "payload": dict(payload or {}), + "status": "queued", + "phase": "queued", + "phase_label": "排队中", + "cancel_requested": False, + "message": "采集任务已创建，等待后台执行", + "created_at": _now(), + "updated_at": _now(), + "started_at": "", + "completed_at": "", + "result": None, + "logs": [f"[{_now()}] 已创建聚名采集任务，等待后台执行"], + } + + with _JUMING_TASK_LOCK: + tasks = load_juming_records() + tasks.insert(0, record) + _save_tasks(tasks) + + worker = threading.Thread(target=_run_juming_task, args=(task_id, dict(payload or {})), daemon=True) + worker.start() + return record + + +def retry_juming_task(task_id: str) -> dict: + with _JUMING_TASK_LOCK: + tasks = load_juming_records() + target = next((item for item in tasks if item["task_id"] == task_id), None) + if not target: + raise ValueError("聚名采集任务不存在") + if target.get("status") == "running": + raise ValueError("聚名采集任务正在运行，不能重复执行") + target["status"] = "queued" + target["phase"] = "queued" + target["phase_label"] = "排队中" + target["cancel_requested"] = False + target["message"] = "采集任务已重新加入队列" + target["started_at"] = "" + target["completed_at"] = "" + target["updated_at"] = _now() + target["result"] = None + target["logs"] = [f"[{_now()}] 已重新加入队列，等待后台执行"] + payload = dict(target.get("payload") or {}) + record = dict(target) + _save_tasks(tasks) + + worker = threading.Thread(target=_run_juming_task, args=(task_id, payload), daemon=True) + worker.start() + return record + + +def request_stop_juming_task(task_id: str) -> dict: + with _JUMING_TASK_LOCK: + tasks = load_juming_records() + target = next((item for item in tasks if item["task_id"] == task_id), None) + if not target: + raise ValueError("聚名采集任务不存在") + if target.get("status") in {"completed", "failed", "stopped"}: + raise ValueError("当前任务已结束，无需停止") + target["cancel_requested"] = True + target["phase"] = "stopping" + target["phase_label"] = "停止中" + target["message"] = "已发送停止请求，等待当前步骤安全退出" + target["updated_at"] = _now() + logs = list(target.get("logs") or []) + logs.append(f"[{_now()}] 已收到停止请求，等待当前步骤安全退出") + target["logs"] = logs[-_MAX_LOG_LINES:] + _save_tasks(tasks) + return dict(target) diff --git a/domain-api/app/services/juziseo_service.py b/domain-api/app/services/juziseo_service.py new file mode 100644 index 0000000..31e21dd --- /dev/null +++ b/domain-api/app/services/juziseo_service.py @@ -0,0 +1,111 @@ +from __future__ import annotations + +import ast +import pickle +import sys +from pathlib import Path + +from requests.cookies import RequestsCookieJar + +from app.core.config import settings +from app.core.redis_client import get_redis + + +DOMAIN_ROOT = Path(settings.domain_root) +if str(DOMAIN_ROOT) not in sys.path: + sys.path.insert(0, str(DOMAIN_ROOT)) + +from detect.juziseo import Juziseo # type: ignore # noqa: E402 + + +JUZISEO_COOKIE_FILE = DOMAIN_ROOT / "juziseo_cookies.pkl" + + +def _cookie_dict_to_jar(cookie_dict: dict[str, str]) -> RequestsCookieJar: + cookie_jar = RequestsCookieJar() + for name, value in cookie_dict.items(): + cookie_jar.set(name, value) + return cookie_jar + + +def _cookie_jar_to_dict(cookie_jar: RequestsCookieJar) -> dict[str, str]: + return {str(cookie.name): str(cookie.value) for cookie in cookie_jar} + + +def _persist_juziseo_cookie(cookie_jar: RequestsCookieJar) -> None: + JUZISEO_COOKIE_FILE.parent.mkdir(parents=True, exist_ok=True) + with JUZISEO_COOKIE_FILE.open("wb") as handle: + pickle.dump(cookie_jar, handle) + + try: + redis_client = get_redis() + redis_client.set("domain_tool:juziseo_cookies", str(_cookie_jar_to_dict(cookie_jar))) + except Exception: + pass + + +def _load_juziseo_cookie() -> tuple[RequestsCookieJar | None, str]: + if JUZISEO_COOKIE_FILE.exists(): + try: + with JUZISEO_COOKIE_FILE.open("rb") as handle: + loaded = pickle.load(handle) + if isinstance(loaded, RequestsCookieJar): + return loaded, "local" + if isinstance(loaded, dict): + return _cookie_dict_to_jar({str(k): str(v) for k, v in loaded.items()}), "local" + except Exception: + pass + + try: + redis_client = get_redis() + raw = redis_client.get("domain_tool:juziseo_cookies") + if raw: + parsed = ast.literal_eval(raw) + if isinstance(parsed, dict) and parsed: + cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in parsed.items()}) + try: + with JUZISEO_COOKIE_FILE.open("wb") as handle: + pickle.dump(cookie_jar, handle) + except Exception: + pass + return cookie_jar, "redis" + except Exception: + pass + + return None, "missing" + + +def get_juziseo_status() -> dict: + cookie_jar, storage = _load_juziseo_cookie() + return { + "cookie_ready": cookie_jar is not None, + "cookie_storage": storage, + "cookie_file": str(JUZISEO_COOKIE_FILE), + "cookie_count": len(_cookie_jar_to_dict(cookie_jar)) if cookie_jar is not None else 0, + } + + +def login_juziseo(email: str, password: str) -> dict: + account = str(email or "").strip() + secret = str(password or "").strip() + if not account or not secret: + raise ValueError("请输入桔子SEO账号和密码") + + juziseo = Juziseo() + juziseo.load_cookies(str(JUZISEO_COOKIE_FILE)) + ok, message = juziseo.login(account, secret) + if not ok: + raise ValueError(f"桔子SEO登录失败: {message}") + + try: + juziseo.save_cookies(str(JUZISEO_COOKIE_FILE)) + except Exception: + pass + try: + _persist_juziseo_cookie(juziseo.cookie) + except Exception: + pass + + status = get_juziseo_status() + status["message"] = str(message) + return status diff --git a/domain-api/app/services/runtime_control_service.py b/domain-api/app/services/runtime_control_service.py index c385d1a..508a05c 100644 --- a/domain-api/app/services/runtime_control_service.py +++ b/domain-api/app/services/runtime_control_service.py @@ -5,8 +5,9 @@ import subprocess from pathlib import Path from app.core.config import settings +from app.services.sync_push_service import push_runtime_projection_now from app.services.runtime_settings_service import get_runtime_settings -from app.services.worker_control_service import start_worker, stop_worker +from app.services.worker_control_service import _run_systemctl, start_worker, stop_worker def _workspace_root() -> Path: @@ -23,7 +24,9 @@ def restart_api() -> tuple[bool, str]: worker_mode = runtime.get("worker_mode", settings.worker_mode) if worker_mode == "linux-systemd": - result = _run_shell(["systemctl", "restart", api_service_name], timeout=30) + # When the API restarts itself under systemd, wait-free restart avoids + # blocking the HTTP request until uvicorn is torn down. + result = _run_systemctl(["restart", api_service_name], timeout=5, no_block=True) if result.returncode != 0: return False, (result.stderr or result.stdout or "重启 Linux API 失败").strip() return True, f"Linux API 重启命令已发送: {api_service_name}" @@ -48,6 +51,35 @@ def restart_api() -> tuple[bool, str]: return True, "API 重启命令已发送" +def _run_systemd_action(service_name: str, action: str, *, no_block: bool = False) -> tuple[bool, str]: + systemctl_command = [] + if no_block: + systemctl_command.append("--no-block") + systemctl_command.extend([action, service_name]) + result = _run_systemctl(systemctl_command, timeout=10 if no_block else 30) + if result.returncode != 0: + return False, (result.stderr or result.stdout or f"{action} {service_name} 失败").strip() + return True, f"{service_name} {action} 命令已发送" + + +def start_sync_agent() -> tuple[bool, str]: + runtime = get_runtime_settings() + worker_mode = runtime.get("worker_mode", settings.worker_mode) + service_name = runtime.get("sync_agent_service_name", settings.sync_agent_service_name) + if worker_mode != "linux-systemd": + return False, "sync-agent 仅在 Linux systemd 多机部署中使用。" + return _run_systemd_action(service_name, "start") + + +def stop_sync_agent() -> tuple[bool, str]: + runtime = get_runtime_settings() + worker_mode = runtime.get("worker_mode", settings.worker_mode) + service_name = runtime.get("sync_agent_service_name", settings.sync_agent_service_name) + if worker_mode != "linux-systemd": + return False, "sync-agent 仅在 Linux systemd 多机部署中使用。" + return _run_systemd_action(service_name, "stop") + + def runtime_action(action: str) -> tuple[bool, str, dict]: if action == "start_worker": ok, message = start_worker() @@ -58,6 +90,20 @@ def runtime_action(action: str) -> tuple[bool, str, dict]: if action == "restart_api": ok, message = restart_api() return ok, message, {"action": action, "poll_after_seconds": 4, "refresh_runtime": True} + if action == "start_sync_agent": + ok, message = start_sync_agent() + return ok, message, {"action": action, "poll_after_seconds": 2, "refresh_runtime": True} + if action == "stop_sync_agent": + ok, message = stop_sync_agent() + return ok, message, {"action": action, "poll_after_seconds": 2, "refresh_runtime": True} + if action == "push_sync": + ok, message, data = push_runtime_projection_now() + return ok, message, { + "action": action, + "poll_after_seconds": 2, + "refresh_runtime": True, + **(data or {}), + } return False, f"不支持的运行时动作: {action}", { "action": action, "poll_after_seconds": 0, diff --git a/domain-api/app/services/runtime_settings_service.py b/domain-api/app/services/runtime_settings_service.py index 7005e43..e21b8ed 100644 --- a/domain-api/app/services/runtime_settings_service.py +++ b/domain-api/app/services/runtime_settings_service.py @@ -8,6 +8,7 @@ DEFAULT_RUNTIME_SETTINGS = { "worker_mode": settings.worker_mode, "worker_service_name": settings.worker_service_name, "api_service_name": settings.api_service_name, + "sync_agent_service_name": settings.sync_agent_service_name, } diff --git a/domain-api/app/services/runtime_status_service.py b/domain-api/app/services/runtime_status_service.py index 6dfefa3..17180a9 100644 --- a/domain-api/app/services/runtime_status_service.py +++ b/domain-api/app/services/runtime_status_service.py @@ -5,9 +5,14 @@ from pathlib import Path from app.core.config import settings from app.core.db import get_db +from app.core.files import read_json from app.core.redis_client import get_redis +from app.services.cluster_runtime_service import get_cluster_snapshot +from app.services.detect_service import get_detect_status +from app.services.detect_job_service import get_detect_capacity_plan, get_detect_queue_health +from app.services.sync_record_service import append_runtime_projection_if_changed, get_sync_summary from app.services.runtime_settings_service import get_runtime_settings -from app.services.worker_control_service import detect_worker_runtime +from app.services.worker_control_service import detect_sync_agent_runtime, detect_worker_runtime def _runtime_log_path(filename: str) -> str: @@ -15,10 +20,195 @@ def _runtime_log_path(filename: str) -> str: return str(path) +def _domain_cookie_status(filename: str) -> tuple[bool, str]: + path = Path(settings.domain_root) / filename + return path.exists(), str(path) + + +def _bloom_filter_status() -> tuple[bool, str]: + try: + redis_client = get_redis() + modules = redis_client.execute_command("MODULE", "LIST") + for module in modules or []: + module_parts = module[::2] + module_values = module[1::2] + module_info = dict(zip(module_parts, module_values)) + module_name = str(module_info.get("name", "")).lower() + if module_name in {"bf", "redisbloom"}: + return True, "RedisBloom 已安装" + return False, "Redis 未安装 RedisBloom，当前将使用普通缓存" + except Exception as exc: + return False, f"RedisBloom 检查失败: {exc}" + + +def _build_multi_region_readiness( + *, + cluster_snapshot: dict, + sync_summary: dict, + worker_runtime: dict, + sync_agent_runtime: dict, +) -> dict: + nodes = list(cluster_snapshot.get("nodes") or []) + summary = cluster_snapshot.get("summary") or {} + batch_summary = (sync_summary.get("detect_result_batches") or {}) + batch_states = batch_summary.get("state_counts") or {} + + online_control_nodes = int(summary.get("online_control_nodes", 0) or 0) + online_worker_nodes = int(summary.get("online_worker_nodes", 0) or 0) + mainland_control_nodes = [ + node for node in nodes + if str(node.get("region") or "") == "mainland" + and str(node.get("role") or "") == "control" + and str(node.get("status") or "") in {"online", "busy"} + ] + mainland_worker_nodes = [ + node for node in nodes + if str(node.get("region") or "") == "mainland" + and str(node.get("role") or "") == "worker" + and str(node.get("status") or "") in {"online", "busy"} + ] + + blocking_issues: list[str] = [] + warning_issues: list[str] = [] + info_items: list[str] = [] + + if online_control_nodes <= 0: + blocking_issues.append("当前没有在线控制面节点，无法视为正式可用集群。") + + if settings.node_region == "mainland" and settings.node_role == "control": + if not sync_agent_runtime.get("running", False): + blocking_issues.append("当前节点应承载 sync-agent，但服务未运行。") + if str(settings.sync_target_api_base_url or "").strip() == "": + blocking_issues.append("当前节点未配置 SYNC_TARGET_API_BASE_URL，无法向海外控制面推送。") + if not bool(settings.sync_push_enabled): + blocking_issues.append("当前节点未启用 SYNC_PUSH_ENABLED，结果同步不会自动执行。") + + if settings.node_region == "overseas" and online_control_nodes > 0 and not mainland_control_nodes: + warning_issues.append("当前尚未观察到在线的大陆 controller 节点，后续自动结果同步仍未进入正式双地域态。") + + if online_worker_nodes <= 0: + warning_issues.append("当前没有在线 Worker 节点，检测任务无法在多机状态下继续推进。") + + offline_nodes = list(summary.get("offline_nodes") or []) + stale_nodes = list(summary.get("stale_nodes") or []) + if stale_nodes: + warning_issues.append(f"存在失活节点: {'、'.join(stale_nodes)}") + if offline_nodes: + warning_issues.append(f"存在离线节点: {'、'.join(offline_nodes)}") + + failed_batches = int(batch_states.get("failed", 0) or 0) + projected_batches = int(batch_states.get("projected", 0) or 0) + pushing_batches = int(batch_states.get("pushing", 0) or 0) + synced_batches = int(batch_states.get("synced", 0) or 0) + if failed_batches > 0: + warning_issues.append(f"存在 {failed_batches} 个结果批次同步失败，需要检查 sync-agent 或目标接收面。") + if projected_batches > 0: + warning_issues.append(f"存在 {projected_batches} 个结果批次仍待推送。") + if pushing_batches > 0: + info_items.append(f"当前有 {pushing_batches} 个结果批次正在推送。") + if synced_batches > 0: + info_items.append(f"最近已接收 {synced_batches} 个结果批次。") + + if worker_runtime.get("running", False): + info_items.append("当前节点本机 Worker 进程在线。") + if settings.node_region == "mainland" and settings.node_role == "control" and sync_agent_runtime.get("running", False): + info_items.append("当前节点本机 sync-agent 在线。") + if mainland_worker_nodes: + info_items.append(f"在线大陆 Worker {len(mainland_worker_nodes)} 台。") + if mainland_control_nodes: + info_items.append(f"在线大陆 controller {len(mainland_control_nodes)} 台。") + + if blocking_issues: + status = "blocking" + summary_text = blocking_issues[0] + elif warning_issues: + status = "attention" + summary_text = warning_issues[0] + else: + status = "ready" + summary_text = "当前多机与跨地域骨架已进入可联调、可持续观察状态。" + + return { + "status": status, + "ready": status == "ready", + "summary": summary_text, + "blocking_issues": blocking_issues, + "warnings": warning_issues, + "info": info_items, + "cluster": { + "online_control_nodes": online_control_nodes, + "online_worker_nodes": online_worker_nodes, + "mainland_control_nodes": len(mainland_control_nodes), + "mainland_worker_nodes": len(mainland_worker_nodes), + }, + "sync": { + "enabled": bool(sync_summary.get("enabled", False)), + "source_region": sync_summary.get("source_region", ""), + "target_region": sync_summary.get("target_region", ""), + "projected_batches": projected_batches, + "pushing_batches": pushing_batches, + "failed_batches": failed_batches, + "synced_batches": synced_batches, + }, + } + + def get_runtime_status() -> dict: runtime_settings = get_runtime_settings() worker_runtime = detect_worker_runtime() + sync_agent_runtime = detect_sync_agent_runtime() api_pid = os.getpid() + detect_snapshot = get_detect_status() + latest_run = (detect_snapshot.get("runs") or [None])[0] or {} + + cluster_snapshot = get_cluster_snapshot() + queue_health = get_detect_queue_health(window_minutes=15) + effective_online_worker_nodes = int((cluster_snapshot.get("summary") or {}).get("online_worker_nodes", 0) or 0) + if effective_online_worker_nodes <= 0 and worker_runtime.get("running", False): + effective_online_worker_nodes = max(1, worker_runtime.get("process_count", 1) or 1) + capacity_plan = get_detect_capacity_plan( + queue_health=queue_health, + online_worker_nodes=effective_online_worker_nodes, + target_finish_hours=6, + ) + detect_payload = { + "phase_label": latest_run.get("phase_label", ""), + "phase_detail": latest_run.get("phase_detail", ""), + "recent_event": detect_snapshot.get("recent_event", ""), + "recent_warning": detect_snapshot.get("recent_warning", ""), + "progress_percent": detect_snapshot.get("progress_percent", 0), + "progress": detect_snapshot.get("progress", {}), + "active_thread_count": detect_snapshot.get("active_thread_count", 0), + "max_thread_count": detect_snapshot.get("max_thread_count", 0), + "available_proxy_count": detect_snapshot.get("available_proxy_count", 0), + "proxy_pool_count": detect_snapshot.get("proxy_pool_count", 0), + "proxy_runtime_label": detect_snapshot.get("proxy_runtime_label", ""), + "proxy_runtime_detail": detect_snapshot.get("proxy_runtime_detail", ""), + "proxy_runtime_reason": detect_snapshot.get("proxy_runtime_reason", ""), + "proxy_supplier_empty": detect_snapshot.get("proxy_supplier_empty", False), + "proxy_last_refresh_status": detect_snapshot.get("proxy_last_refresh_status", ""), + "proxy_last_refresh_time": detect_snapshot.get("proxy_last_refresh_time", ""), + "proxy_last_refresh_source_count": detect_snapshot.get("proxy_last_refresh_source_count", 0), + "proxy_last_refresh_total_items": detect_snapshot.get("proxy_last_refresh_total_items", 0), + "proxy_last_validated_count": detect_snapshot.get("proxy_last_validated_count", 0), + "proxy_last_available_count": detect_snapshot.get("proxy_last_available_count", 0), + "proxy_source_stats": detect_snapshot.get("proxy_source_stats", []), + "dependency_alerts": detect_snapshot.get("dependency_alerts", []), + "active_job": detect_snapshot.get("active_job"), + "runs_count": len(detect_snapshot.get("runs") or []), + "worker_online": worker_runtime.get("running", False), + "worker_mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")), + "queue_health": queue_health, + "capacity_plan": capacity_plan, + } + append_runtime_projection_if_changed(detect=detect_payload, cluster=cluster_snapshot) + sync_summary = get_sync_summary(record_limit=5) + readiness = _build_multi_region_readiness( + cluster_snapshot=cluster_snapshot, + sync_summary=sync_summary, + worker_runtime=worker_runtime, + sync_agent_runtime=sync_agent_runtime, + ) return { "api": { @@ -34,6 +224,11 @@ def get_runtime_status() -> dict: "stdout_log": _runtime_log_path("domain-api.stdout.log"), "stderr_log": _runtime_log_path("domain-api.stderr.log"), }, + "node": { + "code": settings.node_code, + "region": settings.node_region, + "role": settings.node_role, + }, "worker": { "mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")), "service_name": runtime_settings.get("worker_service_name", settings.worker_service_name), @@ -43,12 +238,26 @@ def get_runtime_status() -> dict: "message": worker_runtime.get("message", ""), "log_path": str(Path(settings.domain_root) / "detect_worker.log"), }, + "sync_agent": { + "mode": sync_agent_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")), + "service_name": runtime_settings.get("sync_agent_service_name", settings.sync_agent_service_name), + "running": sync_agent_runtime.get("running", False), + "process_count": sync_agent_runtime.get("process_count", 0), + "latest_start_time": sync_agent_runtime.get("latest_start_time", ""), + "message": sync_agent_runtime.get("message", ""), + "expected_on_this_node": settings.node_region == "mainland" and settings.node_role == "control", + }, + "detect": detect_payload, + "cluster": cluster_snapshot, + "sync": sync_summary, + "readiness": readiness, } def get_runtime_preflight() -> dict: runtime_settings = get_runtime_settings() checks: list[dict[str, object]] = [] + detect_options = read_json("detect_options.json", default={}) domain_root = Path(settings.domain_root) checks.append( @@ -75,10 +284,62 @@ def get_runtime_preflight() -> dict: checks.append({"key": "redis", "label": "Redis", "ok": True, "message": f"{settings.redis_host}:{settings.redis_port}/{settings.redis_db}"}) except Exception as exc: checks.append({"key": "redis", "label": "Redis", "ok": False, "message": str(exc)}) + else: + bloom_ok, bloom_message = _bloom_filter_status() + checks.append( + { + "key": "redis_bloom", + "label": "RedisBloom", + "ok": True, + "message": bloom_message, + "level": "info" if bloom_ok else "warn", + "degraded": not bloom_ok, + } + ) worker_mode = runtime_settings.get("worker_mode", "windows-local") checks.append({"key": "worker_mode", "label": "运行模式", "ok": True, "message": worker_mode}) + jucha_enabled = bool(detect_options.get("detect_jucha")) + jucha_cookie_ok, jucha_cookie_path = _domain_cookie_status("jucha_cookies.pkl") + checks.append( + { + "key": "detect_jucha", + "label": "聚查检测", + "ok": True if not jucha_enabled else jucha_cookie_ok, + "message": "已启用" if jucha_enabled else "未启用", + } + ) + checks.append( + { + "key": "jucha_cookie", + "label": "聚查 Cookie", + "ok": True if not jucha_enabled else jucha_cookie_ok, + "message": jucha_cookie_path if jucha_enabled else "未启用聚查检测，无需本地 Cookie", + "level": "info" if (not jucha_enabled or jucha_cookie_ok) else "warn", + } + ) + + juziseo_enabled = bool(detect_options.get("detect_juziseo")) + juziseo_cookie_ok, juziseo_cookie_path = _domain_cookie_status("juziseo_cookies.pkl") + checks.append( + { + "key": "detect_juziseo", + "label": "桔子SEO检测", + "ok": True if not juziseo_enabled else juziseo_cookie_ok, + "message": "已启用" if juziseo_enabled else "未启用", + } + ) + checks.append( + { + "key": "juziseo_cookie", + "label": "桔子SEO Cookie", + "ok": True if not juziseo_enabled else juziseo_cookie_ok, + "message": juziseo_cookie_path if juziseo_enabled else "未启用桔子SEO检测，无需本地 Cookie", + "level": "info" if (not juziseo_enabled or juziseo_cookie_ok) else "warn", + } + ) + if worker_mode == "linux-systemd": checks.append( { @@ -96,6 +357,14 @@ def get_runtime_preflight() -> dict: "message": runtime_settings.get("api_service_name", ""), } ) + checks.append( + { + "key": "sync_agent_service_name", + "label": "Sync agent service 名", + "ok": bool(runtime_settings.get("sync_agent_service_name")), + "message": runtime_settings.get("sync_agent_service_name", ""), + } + ) else: checks.append( { diff --git a/domain-api/app/services/sensitive_words_service.py b/domain-api/app/services/sensitive_words_service.py new file mode 100644 index 0000000..401644b --- /dev/null +++ b/domain-api/app/services/sensitive_words_service.py @@ -0,0 +1,60 @@ +from __future__ import annotations + +from app.core.db import get_db + + +def get_sensitive_words_payload() -> dict: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + select word, category, priority + from sensitive_words + order by priority desc, word asc + """ + ) + rows = cur.fetchall() + + items = [ + { + "word": row[0], + "category": row[1] or "default", + "priority": row[2] or 1, + } + for row in rows + ] + return { + "items": items, + "text": "\n".join(item["word"] for item in items), + "total": len(items), + } + + +def save_sensitive_words_payload(payload: dict) -> dict: + raw_text = str(payload.get("text") or "") + words = [] + seen: set[str] = set() + for line in raw_text.splitlines(): + word = line.strip() + if not word or word in seen: + continue + seen.add(word) + words.append(word) + + with get_db() as conn: + with conn.cursor() as cur: + cur.execute("delete from sensitive_words") + if words: + cur.executemany( + """ + insert into sensitive_words (word, category, priority, create_time) + values (%s, 'default', 1, now()) + """, + [(word,) for word in words], + ) + conn.commit() + + return { + "total": len(words), + "text": "\n".join(words), + } diff --git a/domain-api/app/services/settings_service.py b/domain-api/app/services/settings_service.py index a3588f3..a6302e3 100644 --- a/domain-api/app/services/settings_service.py +++ b/domain-api/app/services/settings_service.py @@ -3,6 +3,7 @@ from __future__ import annotations import json from datetime import datetime +from app.core.config import settings as app_settings from app.core.files import read_json, settings_backup_root, write_json from app.core.redis_client import get_redis from app.services.runtime_settings_service import get_runtime_settings, update_runtime_settings @@ -12,6 +13,8 @@ REDIS_KEYS = { "detect_options": "domain_tool:detect_options", "proxy_config": "domain_tool:proxy_config", "thread_count": "domain_tool:thread_count", + "node_thread_counts": "domain_tool:node_thread_counts", + "credentials": "domain_tool:credentials", } DETECT_OPTION_KEYS = { @@ -26,10 +29,90 @@ DETECT_OPTION_KEYS = { } +def _normalize_thread_count(value: object, *, field_name: str = "thread_count") -> int: + try: + thread_count = int(value) + except Exception as exc: + raise ValueError(f"{field_name} must be an integer") from exc + if thread_count < 1 or thread_count > 256: + raise ValueError(f"{field_name} out of range") + return thread_count + + +def _normalize_node_thread_counts(payload: object) -> dict[str, int]: + if payload in (None, ""): + return {} + if not isinstance(payload, dict): + raise ValueError("node_thread_counts must be an object") + + normalized: dict[str, int] = {} + for raw_node_code, raw_thread_count in payload.items(): + node_code = str(raw_node_code or "").strip() + if not node_code: + raise ValueError("node_thread_counts contains empty node code") + normalized[node_code] = _normalize_thread_count(raw_thread_count, field_name=f"node_thread_counts.{node_code}") + return normalized + + +def _load_thread_count_config() -> tuple[int, dict[str, int]]: + thread_count_payload = read_json("thread_count.json", default={"thread_count": "2"}) + node_thread_counts_payload = read_json("node_thread_counts.json", default={}) + + try: + default_thread_count = _normalize_thread_count(thread_count_payload.get("thread_count", 2)) + except ValueError: + default_thread_count = 2 + try: + node_thread_counts = _normalize_node_thread_counts(node_thread_counts_payload) + except ValueError: + node_thread_counts = {} + + redis_client = get_redis() + try: + if redis_thread_count := redis_client.get(REDIS_KEYS["thread_count"]): + try: + default_thread_count = _normalize_thread_count(redis_thread_count) + except ValueError: + pass + if redis_node_thread_counts := redis_client.get(REDIS_KEYS["node_thread_counts"]): + try: + node_thread_counts = _normalize_node_thread_counts(json.loads(redis_node_thread_counts)) + except ValueError: + pass + except Exception: + pass + + return default_thread_count, node_thread_counts + + +def resolve_thread_count(node_code: str | None = None, settings_payload: dict | None = None) -> dict: + payload = settings_payload or get_settings_payload() + default_thread_count = int(payload.get("thread_count", 2)) + node_thread_counts = _normalize_node_thread_counts(payload.get("node_thread_counts", {})) + normalized_node_code = str(node_code or app_settings.node_code or "").strip() + + override_thread_count = None + source = "default" + effective_thread_count = default_thread_count + if normalized_node_code and normalized_node_code in node_thread_counts: + override_thread_count = node_thread_counts[normalized_node_code] + effective_thread_count = override_thread_count + source = "node_override" + + return { + "node_code": normalized_node_code, + "default_thread_count": default_thread_count, + "effective_thread_count": effective_thread_count, + "override_thread_count": override_thread_count, + "source": source, + "node_thread_counts": node_thread_counts, + } + + def get_settings_payload() -> dict: detect_options = read_json("detect_options.json", default={}) proxy_config = read_json("proxy_config.json", default={}) - thread_count = read_json("thread_count.json", default={"thread_count": "2"}) + thread_count, node_thread_counts = _load_thread_count_config() redis_client = get_redis() try: @@ -37,29 +120,84 @@ def get_settings_payload() -> dict: detect_options = json.loads(redis_detect_options) if redis_proxy_config := redis_client.get(REDIS_KEYS["proxy_config"]): proxy_config = json.loads(redis_proxy_config) - if redis_thread_count := redis_client.get(REDIS_KEYS["thread_count"]): - thread_count = {"thread_count": str(redis_thread_count)} except Exception: pass return { "detect_options": detect_options, "proxy_config": proxy_config, - "thread_count": int(thread_count.get("thread_count", 2)), + "thread_count": thread_count, + "node_thread_counts": node_thread_counts, + "current_node_code": app_settings.node_code, "runtime_settings": get_runtime_settings(), } +def get_credentials_payload() -> dict: + credentials = read_json( + "credentials.json", + default={ + "juming": {"email": "", "password": ""}, + "juziseo": {"email": "", "password": ""}, + }, + ) + + redis_client = get_redis() + try: + if redis_credentials := redis_client.get(REDIS_KEYS["credentials"]): + credentials = json.loads(redis_credentials) + except Exception: + pass + + return { + "juming": { + "email": str(credentials.get("juming", {}).get("email", "")), + "password": str(credentials.get("juming", {}).get("password", "")), + }, + "juziseo": { + "email": str(credentials.get("juziseo", {}).get("email", "")), + "password": str(credentials.get("juziseo", {}).get("password", "")), + }, + } + + +def update_credentials_payload(payload: dict) -> dict: + current = get_credentials_payload() + credentials = { + "juming": { + "email": str(payload.get("juming", {}).get("email", current["juming"]["email"])), + "password": str(payload.get("juming", {}).get("password", current["juming"]["password"])), + }, + "juziseo": { + "email": str(payload.get("juziseo", {}).get("email", current["juziseo"]["email"])), + "password": str(payload.get("juziseo", {}).get("password", current["juziseo"]["password"])), + }, + } + + write_json("credentials.json", credentials) + + redis_client = get_redis() + try: + redis_client.set(REDIS_KEYS["credentials"], json.dumps(credentials, ensure_ascii=False)) + redis_client.publish("domain_tool:credentials:update", json.dumps(credentials, ensure_ascii=False)) + except Exception: + pass + + return credentials + + def update_settings_payload(payload: dict) -> dict: current = get_settings_payload() detect_options = payload.get("detect_options", current["detect_options"]) proxy_config = payload.get("proxy_config", current["proxy_config"]) - thread_count = int(payload.get("thread_count", current["thread_count"])) + thread_count = _normalize_thread_count(payload.get("thread_count", current["thread_count"])) + node_thread_counts = _normalize_node_thread_counts(payload.get("node_thread_counts", current.get("node_thread_counts", {}))) runtime_settings = update_runtime_settings(payload.get("runtime_settings", current["runtime_settings"])) write_json("detect_options.json", detect_options) write_json("proxy_config.json", proxy_config) write_json("thread_count.json", {"thread_count": str(thread_count)}) + write_json("node_thread_counts.json", node_thread_counts) redis_client = get_redis() try: @@ -69,6 +207,8 @@ def update_settings_payload(payload: dict) -> dict: redis_client.publish("domain_tool:proxy_config:update", json.dumps(proxy_config, ensure_ascii=False)) redis_client.set(REDIS_KEYS["thread_count"], thread_count) redis_client.publish("domain_tool:thread_count:update", str(thread_count)) + redis_client.set(REDIS_KEYS["node_thread_counts"], json.dumps(node_thread_counts, ensure_ascii=False)) + redis_client.publish("domain_tool:node_thread_counts:update", json.dumps(node_thread_counts, ensure_ascii=False)) except Exception: pass @@ -76,6 +216,8 @@ def update_settings_payload(payload: dict) -> dict: "detect_options": detect_options, "proxy_config": proxy_config, "thread_count": thread_count, + "node_thread_counts": node_thread_counts, + "current_node_code": app_settings.node_code, "runtime_settings": runtime_settings, } @@ -101,12 +243,10 @@ def validate_settings_payload(payload: dict) -> None: raise ValueError("invalid settings payload") if "thread_count" in payload: - try: - thread_count = int(payload["thread_count"]) - except Exception as exc: - raise ValueError("thread_count must be an integer") from exc - if thread_count < 1 or thread_count > 256: - raise ValueError("thread_count out of range") + _normalize_thread_count(payload["thread_count"]) + + if "node_thread_counts" in payload: + _normalize_node_thread_counts(payload["node_thread_counts"]) if "detect_options" in payload: detect_options = payload["detect_options"] @@ -135,6 +275,9 @@ def validate_settings_payload(payload: dict) -> None: worker_mode = runtime_settings.get("worker_mode") if worker_mode and worker_mode not in {"windows-local", "linux-systemd"}: raise ValueError("worker_mode must be windows-local or linux-systemd") + for key in ("worker_service_name", "api_service_name", "sync_agent_service_name"): + if key in runtime_settings and runtime_settings[key] is not None and not str(runtime_settings[key]).strip(): + raise ValueError(f"{key} must not be empty") def backup_current_settings(reason: str = "manual") -> dict: diff --git a/domain-api/app/services/sync_push_service.py b/domain-api/app/services/sync_push_service.py new file mode 100644 index 0000000..c03aa56 --- /dev/null +++ b/domain-api/app/services/sync_push_service.py @@ -0,0 +1,423 @@ +from __future__ import annotations + +import json +import urllib.error +import urllib.parse +import urllib.request +from datetime import datetime, timedelta + +from app.core.config import settings +from app.core.db import get_db +from app.services.sync_record_service import _decode_json, _normalize_region + + +def _format_time(value: datetime | None) -> str: + return value.isoformat(sep=" ", timespec="seconds") if value else "" + + +def _ingest_url(base_url: str) -> str: + text = str(base_url or "").strip().rstrip("/") + if not text: + return "" + if text.endswith("/api/v1"): + return f"{text}/runtime/sync-ingest" + if text.endswith("/api/v1/runtime"): + return f"{text}/sync-ingest" + return f"{text}/api/v1/runtime/sync-ingest" + + +def _projection_ingest_type(sync_type: str) -> str: + if sync_type == "runtime_projection": + return "runtime_ingest" + if sync_type == "detect_result_projection": + return "detect_result_ingest" + return "sync_ingest" + + +def _load_latest_projection(sync_type: str) -> dict | None: + source_region = _normalize_region(settings.sync_source_region, settings.node_region) + target_region = _normalize_region(settings.sync_target_region, "overseas") + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, source_region, target_region, status, payload_json, created_at, updated_at + FROM detect_sync_records + WHERE sync_type = %s + AND source_region = %s + AND target_region = %s + ORDER BY created_at DESC, id DESC + LIMIT 1 + """, + (sync_type, source_region, target_region), + ) + row = cur.fetchone() + if not row: + return None + return { + "id": row[0], + "source_region": row[1], + "target_region": row[2], + "status": row[3], + "payload": _decode_json(row[4]), + "created_at": row[5], + "updated_at": row[6], + } + + +def _load_pushable_projections(sync_type: str, limit: int) -> list[dict]: + safe_limit = max(1, min(int(limit or 1), max(1, int(settings.sync_batch_size or 200)))) + source_region = _normalize_region(settings.sync_source_region, settings.node_region) + target_region = _normalize_region(settings.sync_target_region, "overseas") + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, source_region, target_region, status, payload_json, created_at, updated_at + FROM detect_sync_records + WHERE sync_type = %s + AND source_region = %s + AND target_region = %s + ORDER BY created_at ASC, id ASC + LIMIT %s + """, + (sync_type, source_region, target_region, safe_limit * 5), + ) + rows = cur.fetchall() + + selected: list[dict] = [] + for row in rows: + projection = { + "id": row[0], + "source_region": row[1], + "target_region": row[2], + "status": row[3], + "payload": _decode_json(row[4]), + "created_at": row[5], + "updated_at": row[6], + } + latest_attempt = _latest_push_attempt(projection["id"], projection["target_region"], sync_type) + if latest_attempt and latest_attempt["status"] == "success": + continue + if latest_attempt and latest_attempt["status"] == "pending": + continue + if latest_attempt and latest_attempt["status"] == "failed": + last_created_at = latest_attempt.get("created_at") + if isinstance(last_created_at, datetime): + now = datetime.now(last_created_at.tzinfo) if last_created_at.tzinfo else datetime.now() + if now - last_created_at < timedelta(seconds=max(10, int(settings.sync_poll_interval_seconds or 30))): + continue + selected.append(projection) + if len(selected) >= safe_limit: + break + return selected + + +def _latest_push_attempt(source_record_id: int, target_region: str, sync_type: str) -> dict | None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, status, payload_json, error_message, created_at, updated_at + FROM detect_sync_records + WHERE sync_type = 'runtime_push' + AND source_region = %s + AND target_region = %s + AND (payload_json->>'sync_type') = %s + AND (payload_json->>'source_record_id') = %s + ORDER BY created_at DESC, id DESC + LIMIT 1 + """, + ( + _normalize_region(settings.sync_source_region, settings.node_region), + target_region, + sync_type, + str(int(source_record_id)), + ), + ) + row = cur.fetchone() + if not row: + return None + return { + "id": row[0], + "status": row[1], + "payload": _decode_json(row[2]), + "error_message": row[3] or "", + "created_at": row[4], + "updated_at": row[5], + } + + +def _create_push_attempt(*, source_record: dict, ingest_url: str, sync_type: str) -> int: + payload = { + "sync_type": sync_type, + "source_record_id": source_record["id"], + "projection_hash": (source_record.get("payload") or {}).get("projection_hash", ""), + "ingest_url": ingest_url, + } + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + INSERT INTO detect_sync_records ( + sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at + ) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) + RETURNING id + """, + ( + "runtime_push", + source_record["source_region"], + source_record["target_region"], + "pending", + json.dumps(payload, ensure_ascii=False), + "", + ), + ) + record_id = int(cur.fetchone()[0]) + conn.commit() + return record_id + + +def _update_push_attempt(record_id: int, *, status: str, payload: dict | None = None, error_message: str = "") -> None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + UPDATE detect_sync_records + SET status = %s, + payload_json = %s::jsonb, + error_message = %s, + updated_at = CURRENT_TIMESTAMP + WHERE id = %s + """, + ( + str(status or "").strip() or "pending", + json.dumps(payload or {}, ensure_ascii=False), + str(error_message or "").strip(), + int(record_id), + ), + ) + conn.commit() + + +def ingest_runtime_projection(payload: dict, *, shared_token: str | None = None) -> tuple[bool, str, dict]: + configured_token = str(settings.sync_shared_token or "").strip() + incoming_token = str(shared_token or "").strip() + if configured_token and incoming_token != configured_token: + return False, "同步 token 校验失败", {} + + sync_type = str(payload.get("sync_type") or "runtime_projection").strip() or "runtime_projection" + ingest_sync_type = _projection_ingest_type(sync_type) + source_region = _normalize_region(payload.get("source_region"), "unknown") + source_record_id = int(payload.get("source_record_id") or 0) + projection_hash = str(payload.get("projection_hash") or "").strip() + projection = payload.get("projection") or {} + target_region = _normalize_region(settings.node_region, "overseas") + + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id + FROM detect_sync_records + WHERE sync_type = %s + AND source_region = %s + AND target_region = %s + AND (payload_json->>'source_record_id') = %s + AND (payload_json->>'projection_hash') = %s + ORDER BY created_at DESC, id DESC + LIMIT 1 + """, + (ingest_sync_type, source_region, target_region, str(source_record_id), projection_hash), + ) + existing = cur.fetchone() + if existing: + return True, "同步投影已存在，已按幂等处理", {"record_id": int(existing[0]), "deduplicated": True} + + stored_payload = { + "sync_type": sync_type, + "source_record_id": source_record_id, + "projection_hash": projection_hash, + "projection": projection, + "received_at": _format_time(datetime.now()), + } + cur.execute( + """ + INSERT INTO detect_sync_records ( + sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at + ) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) + RETURNING id + """, + ( + ingest_sync_type, + source_region, + target_region, + "received", + json.dumps(stored_payload, ensure_ascii=False), + "", + ), + ) + record_id = int(cur.fetchone()[0]) + conn.commit() + return True, "同步投影接收成功", {"record_id": record_id, "deduplicated": False} + + +def _push_projection_now(sync_type: str, ingest_url: str) -> tuple[bool, str, dict]: + source_record = _load_latest_projection(sync_type) + if not source_record: + return False, f"当前没有可推送的{sync_type}", {"action": "push_sync", "sync_type": sync_type} + return _push_projection_record(source_record, sync_type, ingest_url) + + +def _push_projection_batch(sync_type: str, ingest_url: str) -> tuple[bool, str, dict]: + pending_records = _load_pushable_projections(sync_type, limit=max(1, int(settings.sync_batch_size or 200))) + if not pending_records: + latest = _load_latest_projection(sync_type) + if latest: + latest_attempt = _latest_push_attempt(latest["id"], latest["target_region"], sync_type) + if latest_attempt and latest_attempt["status"] == "success": + return True, "当前批次已全部同步完成", { + "action": "push_sync", + "sync_type": sync_type, + "deduplicated": True, + "batch_count": 0, + } + return False, f"当前没有可推送的{sync_type}", {"action": "push_sync", "sync_type": sync_type, "batch_count": 0} + + results: list[dict] = [] + success_count = 0 + for record in pending_records: + ok, message, data = _push_projection_record(record, sync_type, ingest_url) + results.append( + { + "source_record_id": record["id"], + "ok": ok, + "message": message, + "data": data, + } + ) + if ok: + success_count += 1 + + overall_ok = success_count > 0 + message = f"{sync_type} 批量推送完成，成功 {success_count}/{len(results)}" + return overall_ok, message, { + "action": "push_sync", + "sync_type": sync_type, + "batch_count": len(results), + "success_count": success_count, + "results": results, + } + + +def _push_projection_record(source_record: dict, sync_type: str, ingest_url: str) -> tuple[bool, str, dict]: + latest_attempt = _latest_push_attempt(source_record["id"], source_record["target_region"], sync_type) + if latest_attempt and latest_attempt["status"] == "success": + return True, "该投影已推送，无需重复发送", { + "action": "push_sync", + "sync_type": sync_type, + "source_record_id": source_record["id"], + "deduplicated": True, + } + if latest_attempt and latest_attempt["status"] == "pending": + return True, "该投影已有同步推送进行中，暂不重复发送", { + "action": "push_sync", + "sync_type": sync_type, + "source_record_id": source_record["id"], + "deduplicated": True, + } + if latest_attempt and latest_attempt["status"] == "failed": + last_created_at = latest_attempt.get("created_at") + if isinstance(last_created_at, datetime): + now = datetime.now(last_created_at.tzinfo) if last_created_at.tzinfo else datetime.now() + if now - last_created_at < timedelta(seconds=max(10, int(settings.sync_poll_interval_seconds or 30))): + return False, "最近一次同步推送刚失败，等待下个重试窗口", { + "action": "push_sync", + "sync_type": sync_type, + "source_record_id": source_record["id"], + "deduplicated": True, + } + + attempt_id = _create_push_attempt(source_record=source_record, ingest_url=ingest_url, sync_type=sync_type) + request_payload = { + "sync_type": sync_type, + "source_region": source_record["source_region"], + "source_record_id": source_record["id"], + "projection_hash": (source_record.get("payload") or {}).get("projection_hash", ""), + "projection": (source_record.get("payload") or {}).get("projection", {}), + "created_at": _format_time(source_record.get("created_at")), + } + request_body = json.dumps(request_payload, ensure_ascii=False).encode("utf-8") + request = urllib.request.Request( + ingest_url, + data=request_body, + headers={ + "Content-Type": "application/json", + **({"X-Domaincheck-Sync-Token": settings.sync_shared_token} if settings.sync_shared_token else {}), + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=15) as response: + raw = response.read().decode("utf-8") + data = json.loads(raw) if raw else {} + status_code = getattr(response, "status", 200) + except urllib.error.HTTPError as exc: + error_body = exc.read().decode("utf-8", errors="ignore") if hasattr(exc, "read") else "" + payload = { + "sync_type": sync_type, + "source_record_id": source_record["id"], + "projection_hash": request_payload["projection_hash"], + "ingest_url": ingest_url, + "http_status": getattr(exc, "code", 500), + "response_text": error_body[:1000], + } + _update_push_attempt(attempt_id, status="failed", payload=payload, error_message=f"HTTP {getattr(exc, 'code', 500)}") + return False, f"同步推送失败: HTTP {getattr(exc, 'code', 500)}", {"action": "push_sync", "sync_type": sync_type, "attempt_id": attempt_id} + except Exception as exc: + payload = { + "sync_type": sync_type, + "source_record_id": source_record["id"], + "projection_hash": request_payload["projection_hash"], + "ingest_url": ingest_url, + } + _update_push_attempt(attempt_id, status="failed", payload=payload, error_message=str(exc)) + return False, f"同步推送失败: {exc}", {"action": "push_sync", "sync_type": sync_type, "attempt_id": attempt_id} + + payload = { + "sync_type": sync_type, + "source_record_id": source_record["id"], + "projection_hash": request_payload["projection_hash"], + "ingest_url": ingest_url, + "http_status": status_code, + "response": data, + } + _update_push_attempt(attempt_id, status="success", payload=payload, error_message="") + return True, "投影推送成功", { + "action": "push_sync", + "sync_type": sync_type, + "attempt_id": attempt_id, + "source_record_id": source_record["id"], + "response": data, + } + + +def push_runtime_projection_now() -> tuple[bool, str, dict]: + if not settings.sync_push_enabled: + return False, "未启用同步推送", {"action": "push_sync"} + + ingest_url = _ingest_url(settings.sync_target_api_base_url) + if not ingest_url: + return False, "未配置同步目标地址", {"action": "push_sync"} + + results = [] + ok, message, data = _push_projection_now("runtime_projection", ingest_url) + results.append({"sync_type": "runtime_projection", "ok": ok, "message": message, "data": data}) + + ok, message, data = _push_projection_batch("detect_result_projection", ingest_url) + results.append({"sync_type": "detect_result_projection", "ok": ok, "message": message, "data": data}) + + success_count = sum(1 for item in results if item["ok"]) + if success_count == 0: + return False, "当前没有成功推送的同步投影", {"action": "push_sync", "results": results} + return True, f"同步推送完成，成功 {success_count}/{len(results)}", {"action": "push_sync", "results": results} diff --git a/domain-api/app/services/sync_record_service.py b/domain-api/app/services/sync_record_service.py new file mode 100644 index 0000000..ef793cd --- /dev/null +++ b/domain-api/app/services/sync_record_service.py @@ -0,0 +1,592 @@ +from __future__ import annotations + +import hashlib +import json +from datetime import datetime, timedelta + +from app.core.config import settings +from app.core.db import get_db + + +def _format_time(value: datetime | None) -> str: + return value.isoformat(sep=" ", timespec="seconds") if value else "" + + +def _decode_json(value: object) -> dict: + if isinstance(value, dict): + return value + if value in (None, ""): + return {} + try: + return json.loads(value) + except Exception: + return {} + + +def _normalize_region(value: str | None, fallback: str) -> str: + text = str(value or "").strip() + if not text or text == "unknown": + return str(fallback or "unknown").strip() or "unknown" + return text + + +def _should_append_runtime_projection(previous_payload: dict, current_projection: dict, previous_created_at: datetime | None) -> bool: + if not previous_payload: + return True + previous_projection = previous_payload.get("projection") or {} + if not previous_projection: + return True + + keys_requiring_immediate_write = ( + "worker_online", + "worker_mode", + "phase_label", + "phase_detail", + "proxy_runtime_label", + "proxy_runtime_reason", + ) + for key in keys_requiring_immediate_write: + if previous_projection.get(key) != current_projection.get(key): + return True + + previous_job = previous_projection.get("active_job") or {} + current_job = current_projection.get("active_job") or {} + for key in ("job_id", "job_code", "status"): + if previous_job.get(key) != current_job.get(key): + return True + + previous_cluster = previous_projection.get("cluster_summary") or {} + current_cluster = current_projection.get("cluster_summary") or {} + for key in ("online_worker_nodes", "online_control_nodes", "busy_nodes", "stale_nodes", "offline_nodes"): + if previous_cluster.get(key) != current_cluster.get(key): + return True + + previous_progress = previous_projection.get("progress") or {} + current_progress = current_projection.get("progress") or {} + failed_delta = abs(int(current_progress.get("failed", 0) or 0) - int(previous_progress.get("failed", 0) or 0)) + blacklisted_delta = abs(int(current_progress.get("blacklisted", 0) or 0) - int(previous_progress.get("blacklisted", 0) or 0)) + completed_delta = abs(int(current_progress.get("completed", 0) or 0) - int(previous_progress.get("completed", 0) or 0)) + running_delta = abs(int(current_progress.get("running", 0) or 0) - int(previous_progress.get("running", 0) or 0)) + if failed_delta > 0 or blacklisted_delta >= 10 or completed_delta >= 20 or running_delta >= 5: + return True + + previous_alerts = previous_projection.get("dependency_alerts") or [] + current_alerts = current_projection.get("dependency_alerts") or [] + if previous_alerts != current_alerts: + return True + + if not previous_created_at: + return True + now = datetime.now(previous_created_at.tzinfo) if previous_created_at.tzinfo else datetime.now() + return now - previous_created_at >= timedelta(seconds=45) + + +def list_sync_records(limit: int = 20) -> list[dict]: + safe_limit = max(1, min(int(limit or 20), 200)) + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at + FROM detect_sync_records + ORDER BY created_at DESC, id DESC + LIMIT %s + """, + (safe_limit,), + ) + rows = cur.fetchall() + return [ + { + "id": row[0], + "sync_type": row[1], + "source_region": row[2], + "target_region": row[3], + "status": row[4], + "payload": _decode_json(row[5]), + "error_message": row[6] or "", + "created_at": _format_time(row[7]), + "updated_at": _format_time(row[8]), + } + for row in rows + ] + + +def _latest_sync_record_by_source( + cur, + *, + sync_type: str, + source_region: str, + target_region: str, + source_record_id: int, +) -> dict | None: + cur.execute( + """ + SELECT id, sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at + FROM detect_sync_records + WHERE sync_type = %s + AND source_region = %s + AND target_region = %s + AND (payload_json->>'source_record_id') = %s + ORDER BY created_at DESC, id DESC + LIMIT 1 + """, + (sync_type, source_region, target_region, str(int(source_record_id))), + ) + row = cur.fetchone() + if not row: + return None + return { + "id": row[0], + "sync_type": row[1], + "source_region": row[2], + "target_region": row[3], + "status": row[4], + "payload": _decode_json(row[5]), + "error_message": row[6] or "", + "created_at": _format_time(row[7]), + "updated_at": _format_time(row[8]), + } + + +def get_detect_result_sync_batches(limit: int = 5) -> dict: + safe_limit = max(1, min(int(limit or 5), 20)) + source_region = _normalize_region(settings.sync_source_region, settings.node_region) + target_region = _normalize_region(settings.sync_target_region, "overseas") + batches: list[dict] = [] + + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT id, job_code, status, created_at, started_at, finished_at + FROM detect_jobs + ORDER BY created_at DESC, id DESC + LIMIT %s + """, + (safe_limit,), + ) + jobs = cur.fetchall() + + for row in jobs: + job_id = int(row[0]) + job_code = str(row[1] or "") + job_status = str(row[2] or "") + created_at = _format_time(row[3]) + started_at = _format_time(row[4]) + finished_at = _format_time(row[5]) + + cur.execute( + """ + SELECT status, count(*) + FROM detect_job_items + WHERE job_id = %s + GROUP BY status + """, + (job_id,), + ) + item_counts = {str(status or ""): int(count) for status, count in cur.fetchall()} + items_total = sum(item_counts.values()) + items_terminal = int(item_counts.get("completed", 0)) + int(item_counts.get("blacklisted", 0)) + int(item_counts.get("failed", 0)) + + cur.execute( + """ + SELECT id, status, payload_json, created_at, updated_at + FROM detect_sync_records + WHERE sync_type = 'detect_result_projection' + AND source_region = %s + AND target_region = %s + AND (payload_json->'projection'->'job'->>'job_id') = %s + ORDER BY created_at DESC, id DESC + LIMIT 1 + """, + (source_region, target_region, str(job_id)), + ) + projection_row = cur.fetchone() + projection = None + latest_push = None + latest_ingest = None + sync_state = "unsynced" + sync_message = "该任务还没有生成结果投影" + + if projection_row: + projection_payload = _decode_json(projection_row[2]) + projection = { + "id": int(projection_row[0]), + "status": projection_row[1], + "payload": projection_payload, + "created_at": _format_time(projection_row[3]), + "updated_at": _format_time(projection_row[4]), + } + latest_push = _latest_sync_record_by_source( + cur, + sync_type="runtime_push", + source_region=source_region, + target_region=target_region, + source_record_id=projection["id"], + ) + latest_ingest = _latest_sync_record_by_source( + cur, + sync_type="detect_result_ingest", + source_region=source_region, + target_region=target_region, + source_record_id=projection["id"], + ) + + if latest_ingest: + sync_state = "synced" + sync_message = "最近一条结果投影已被目标地域接收" + elif latest_push and latest_push.get("status") == "success": + sync_state = "delivered" + sync_message = "结果投影已推送成功，等待目标侧回看接收记录" + elif latest_push and latest_push.get("status") == "pending": + sync_state = "pushing" + sync_message = "结果投影正在推送中" + elif latest_push and latest_push.get("status") == "failed": + sync_state = "failed" + sync_message = latest_push.get("error_message") or "最近一次结果投影推送失败" + else: + sync_state = "projected" + sync_message = "已生成结果投影，等待同步代理推送" + + batches.append( + { + "job_id": job_id, + "job_code": job_code, + "job_status": job_status, + "created_at": created_at, + "started_at": started_at, + "finished_at": finished_at, + "items_total": items_total, + "items_terminal": items_terminal, + "items_pending": int(item_counts.get("pending", 0)), + "items_running": int(item_counts.get("running", 0)) + int(item_counts.get("claimed", 0)), + "items_failed": int(item_counts.get("failed", 0)), + "progress_percent": round((items_terminal / items_total) * 100, 2) if items_total else 0, + "sync_state": sync_state, + "sync_message": sync_message, + "projection": projection, + "latest_push": latest_push, + "latest_ingest": latest_ingest, + } + ) + + state_counts = { + "synced": 0, + "delivered": 0, + "pushing": 0, + "projected": 0, + "failed": 0, + "unsynced": 0, + } + for item in batches: + state = str(item.get("sync_state") or "unsynced") + state_counts[state] = state_counts.get(state, 0) + 1 + + return { + "source_region": source_region, + "target_region": target_region, + "jobs_total": len(batches), + "state_counts": state_counts, + "batches": batches, + } + + +def get_sync_summary(record_limit: int = 10) -> dict: + source_region = _normalize_region(settings.sync_source_region, settings.node_region) + target_region = _normalize_region(settings.sync_target_region, "overseas") + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT status, count(*) + FROM detect_sync_records + GROUP BY status + """ + ) + status_counts = {str(status or "unknown"): int(count) for status, count in cur.fetchall()} + cur.execute( + """ + SELECT sync_type, count(*) + FROM detect_sync_records + GROUP BY sync_type + """ + ) + type_counts = {str(sync_type or "unknown"): int(count) for sync_type, count in cur.fetchall()} + cur.execute("SELECT count(*) FROM detect_sync_records") + total = int(cur.fetchone()[0] or 0) + cur.execute( + """ + SELECT id, sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at + FROM detect_sync_records + ORDER BY created_at DESC, id DESC + LIMIT 1 + """ + ) + latest = cur.fetchone() + + latest_record = None + if latest: + latest_record = { + "id": latest[0], + "sync_type": latest[1], + "source_region": latest[2], + "target_region": latest[3], + "status": latest[4], + "payload": _decode_json(latest[5]), + "error_message": latest[6] or "", + "created_at": _format_time(latest[7]), + "updated_at": _format_time(latest[8]), + } + + return { + "enabled": bool(settings.sync_push_enabled), + "source_region": source_region, + "target_region": target_region, + "target_api_base_url": settings.sync_target_api_base_url, + "batch_size": max(1, int(settings.sync_batch_size or 200)), + "poll_interval_seconds": max(5, int(settings.sync_poll_interval_seconds or 30)), + "records_total": total, + "status_counts": status_counts, + "type_counts": type_counts, + "latest_record": latest_record, + "detect_result_batches": get_detect_result_sync_batches(limit=min(5, record_limit)), + "recent_records": list_sync_records(limit=record_limit), + } + + +def append_sync_record( + *, + sync_type: str, + source_region: str, + target_region: str, + status: str, + payload: dict | None = None, + error_message: str = "", +) -> int: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + INSERT INTO detect_sync_records ( + sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at + ) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) + RETURNING id + """, + ( + str(sync_type or "").strip() or "unknown", + _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region)), + _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas")), + str(status or "").strip() or "pending", + json.dumps(payload or {}, ensure_ascii=False), + str(error_message or "").strip(), + ), + ) + record_id = int(cur.fetchone()[0]) + conn.commit() + return record_id + + +def append_runtime_projection_if_changed( + *, + detect: dict, + cluster: dict, + source_region: str | None = None, + target_region: str | None = None, +) -> int | None: + normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region)) + normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas")) + active_job = detect.get("active_job") or {} + projection = { + "worker_online": bool(detect.get("worker_online", False)), + "worker_mode": detect.get("worker_mode", ""), + "phase_label": detect.get("phase_label", ""), + "phase_detail": detect.get("phase_detail", ""), + "proxy_runtime_label": detect.get("proxy_runtime_label", ""), + "proxy_runtime_reason": detect.get("proxy_runtime_reason", ""), + "progress": { + "pending": int((detect.get("progress") or {}).get("pending", 0) or 0), + "running": int((detect.get("progress") or {}).get("running", 0) or 0), + "completed": int((detect.get("progress") or {}).get("completed", 0) or 0), + "blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0), + "failed": int((detect.get("progress") or {}).get("failed", 0) or 0), + }, + "active_job": { + "job_id": active_job.get("job_id"), + "job_code": active_job.get("job_code", ""), + "status": active_job.get("status", ""), + "progress_percent": active_job.get("progress_percent", 0), + "items_total": active_job.get("items_total", 0), + "items_terminal": active_job.get("items_terminal", 0), + "items_pending": active_job.get("items_pending", 0), + "items_running": active_job.get("items_running", 0), + "items_failed": active_job.get("items_failed", 0), + }, + "cluster_summary": { + "nodes_total": int(cluster.get("nodes_total", 0) or 0), + "online_worker_nodes": int((cluster.get("summary") or {}).get("online_worker_nodes", 0) or 0), + "online_control_nodes": int((cluster.get("summary") or {}).get("online_control_nodes", 0) or 0), + "busy_nodes": list((cluster.get("summary") or {}).get("busy_nodes") or []), + "stale_nodes": list((cluster.get("summary") or {}).get("stale_nodes") or []), + "offline_nodes": list((cluster.get("summary") or {}).get("offline_nodes") or []), + }, + "dependency_alerts": [ + { + "kind": item.get("kind", ""), + "title": item.get("title", ""), + "level": item.get("level", ""), + } + for item in (detect.get("dependency_alerts") or [])[:3] + ], + } + payload = { + "projection": projection, + "projection_hash": hashlib.sha1( + json.dumps(projection, ensure_ascii=False, sort_keys=True).encode("utf-8") + ).hexdigest(), + } + + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT payload_json, created_at + FROM detect_sync_records + WHERE sync_type = 'runtime_projection' + AND source_region = %s + AND target_region = %s + ORDER BY created_at DESC, id DESC + LIMIT 1 + """, + (normalized_source_region, normalized_target_region), + ) + latest = cur.fetchone() + latest_payload = _decode_json(latest[0]) if latest else {} + latest_created_at = latest[1] if latest else None + if latest_payload.get("projection_hash") == payload["projection_hash"]: + return None + if not _should_append_runtime_projection(latest_payload, projection, latest_created_at): + return None + cur.execute( + """ + INSERT INTO detect_sync_records ( + sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at + ) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) + RETURNING id + """, + ( + "runtime_projection", + normalized_source_region, + normalized_target_region, + "projected", + json.dumps(payload, ensure_ascii=False), + "", + ), + ) + record_id = int(cur.fetchone()[0]) + conn.commit() + return record_id + + +def append_detect_result_projection_if_changed( + *, + detect: dict, + source_region: str | None = None, + target_region: str | None = None, +) -> int | None: + normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region)) + normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas")) + active_job = detect.get("active_job") or {} + if not active_job: + return None + + latest_cycle_event = active_job.get("latest_cycle_event") or active_job.get("latest_event") or {} + projection = { + "job": { + "job_id": active_job.get("job_id"), + "job_code": active_job.get("job_code", ""), + "status": active_job.get("status", ""), + "progress_percent": active_job.get("progress_percent", 0), + "items_total": active_job.get("items_total", 0), + "items_pending": active_job.get("items_pending", 0), + "items_claimed": active_job.get("items_claimed", 0), + "items_running": active_job.get("items_running", 0), + "items_completed": active_job.get("items_completed", 0), + "items_blacklisted": active_job.get("items_blacklisted", 0), + "items_failed": active_job.get("items_failed", 0), + "items_terminal": active_job.get("items_terminal", 0), + "current_cycle_token": active_job.get("current_cycle_token", ""), + }, + "latest_event": { + "node_code": latest_cycle_event.get("node_code", ""), + "event_type": latest_cycle_event.get("event_type", ""), + "message": latest_cycle_event.get("message", ""), + "created_at": latest_cycle_event.get("created_at", ""), + }, + "queue": { + "pending": int((detect.get("progress") or {}).get("pending", 0) or 0), + "running": int((detect.get("progress") or {}).get("running", 0) or 0), + "completed": int((detect.get("progress") or {}).get("completed", 0) or 0), + "blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0), + "failed": int((detect.get("progress") or {}).get("failed", 0) or 0), + }, + "phase": { + "label": detect.get("phase_label", ""), + "detail": detect.get("phase_detail", ""), + }, + } + payload = { + "projection": projection, + "projection_hash": hashlib.sha1( + json.dumps(projection, ensure_ascii=False, sort_keys=True).encode("utf-8") + ).hexdigest(), + } + + with get_db() as conn: + with conn.cursor() as cur: + cur.execute( + """ + SELECT payload_json, created_at + FROM detect_sync_records + WHERE sync_type = 'detect_result_projection' + AND source_region = %s + AND target_region = %s + ORDER BY created_at DESC, id DESC + LIMIT 1 + """, + (normalized_source_region, normalized_target_region), + ) + latest = cur.fetchone() + latest_payload = _decode_json(latest[0]) if latest else {} + latest_created_at = latest[1] if latest else None + if latest_payload.get("projection_hash") == payload["projection_hash"]: + return None + + latest_projection = latest_payload.get("projection") or {} + latest_job = latest_projection.get("job") or {} + current_job = projection.get("job") or {} + latest_event = latest_projection.get("latest_event") or {} + current_event = projection.get("latest_event") or {} + if latest_job.get("status") == current_job.get("status") and latest_event == current_event and latest_created_at: + now = datetime.now(latest_created_at.tzinfo) if latest_created_at.tzinfo else datetime.now() + if now - latest_created_at < timedelta(seconds=30): + return None + + cur.execute( + """ + INSERT INTO detect_sync_records ( + sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at + ) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP) + RETURNING id + """, + ( + "detect_result_projection", + normalized_source_region, + normalized_target_region, + "projected", + json.dumps(payload, ensure_ascii=False), + "", + ), + ) + record_id = int(cur.fetchone()[0]) + conn.commit() + return record_id diff --git a/domain-api/app/services/worker_control_service.py b/domain-api/app/services/worker_control_service.py index e31278c..03bc098 100644 --- a/domain-api/app/services/worker_control_service.py +++ b/domain-api/app/services/worker_control_service.py @@ -3,12 +3,18 @@ from __future__ import annotations import json import os import subprocess +from datetime import datetime from pathlib import Path from app.core.config import settings +from app.core.redis_client import get_redis from app.services.runtime_settings_service import get_runtime_settings +WORKER_CONTROL_CHANNEL = "domain_tool:worker_control" +WORKER_PENDING_COMMAND_KEY = "domain_tool:worker_pending_command" + + def _domain_root() -> Path: return Path(settings.domain_root) @@ -30,6 +36,73 @@ def _run_shell(command: list[str], timeout: int = 20) -> subprocess.CompletedPro return subprocess.run(command, capture_output=True, text=True, timeout=timeout) +def _run_systemctl(command: list[str], timeout: int = 20, require_sudo: bool = True) -> subprocess.CompletedProcess[str]: + systemctl_command = ["systemctl", *command] + if os.geteuid() == 0 or not require_sudo: + return _run_shell(systemctl_command, timeout=timeout) + return _run_shell(["sudo", "-n", *systemctl_command], timeout=timeout) + + +def _parse_systemd_timestamp(raw_timestamp: str) -> str: + raw_timestamp = (raw_timestamp or "").strip() + if not raw_timestamp: + return "" + try: + parsed = datetime.strptime(raw_timestamp, "%a %Y-%m-%d %H:%M:%S %Z") + return parsed.strftime("%Y-%m-%d %H:%M:%S") + except ValueError: + pass + try: + parsed = datetime.strptime(raw_timestamp.rsplit(" ", 1)[0], "%a %Y-%m-%d %H:%M:%S") + return parsed.strftime("%Y-%m-%d %H:%M:%S") + except ValueError: + return "" + + +def probe_systemd_service(service_name: str, *, mode: str = "linux-systemd") -> dict: + result = _run_systemctl( + [ + "show", + service_name, + "--no-page", + "--property=ActiveState,SubState,MainPID,ExecMainStartTimestamp,ActiveEnterTimestamp", + ], + require_sudo=False, + ) + output = (result.stdout or result.stderr or "").strip() + if result.returncode != 0: + return { + "mode": mode, + "service_name": service_name, + "running": False, + "process_count": 0, + "latest_start_time": "", + "message": output or f"systemd service {service_name} not available", + } + + data: dict[str, str] = {} + for line in output.splitlines(): + if "=" in line: + key, value = line.split("=", 1) + data[key] = value + main_pid = int(data.get("MainPID", "0") or 0) + active_state = data.get("ActiveState", "") + sub_state = data.get("SubState", "") + latest_start_time = "" + for raw_timestamp in (data.get("ExecMainStartTimestamp", ""), data.get("ActiveEnterTimestamp", "")): + latest_start_time = _parse_systemd_timestamp(raw_timestamp) + if latest_start_time: + break + return { + "mode": mode, + "service_name": service_name, + "running": active_state == "active", + "process_count": 1 if main_pid > 0 else 0, + "latest_start_time": latest_start_time, + "message": f"{active_state}/{sub_state}" if active_state else "", + } + + def _windows_runtime() -> dict: command = """ $targets = Get-CimInstance Win32_Process -Filter "name='python.exe'" | @@ -83,32 +156,23 @@ def _windows_runtime() -> dict: def _linux_runtime() -> dict: runtime = _runtime_config() service_name = runtime["worker_service_name"] - result = _run_shell(["systemctl", "show", service_name, "--no-page", "--property=ActiveState,SubState,MainPID"]) - output = (result.stdout or result.stderr or "").strip() - if result.returncode != 0: + return probe_systemd_service(service_name, mode="linux-systemd") + + +def detect_sync_agent_runtime() -> dict: + runtime = _runtime_config() + worker_mode = runtime.get("worker_mode", settings.worker_mode) + service_name = runtime.get("sync_agent_service_name", settings.sync_agent_service_name) + if worker_mode != "linux-systemd": return { - "mode": "linux-systemd", + "mode": worker_mode, + "service_name": service_name, "running": False, "process_count": 0, "latest_start_time": "", - "message": output or f"systemd service {service_name} not available", + "message": "sync-agent 仅在 Linux systemd 多机部署中使用", } - - data: dict[str, str] = {} - for line in output.splitlines(): - if "=" in line: - key, value = line.split("=", 1) - data[key] = value - main_pid = int(data.get("MainPID", "0") or 0) - active_state = data.get("ActiveState", "") - sub_state = data.get("SubState", "") - return { - "mode": "linux-systemd", - "running": active_state == "active", - "process_count": 1 if main_pid > 0 else 0, - "latest_start_time": "", - "message": f"{active_state}/{sub_state}" if active_state else "", - } + return probe_systemd_service(service_name, mode="linux-systemd") def detect_worker_runtime() -> dict: @@ -132,7 +196,7 @@ def start_worker() -> tuple[bool, str]: worker_mode = runtime["worker_mode"] service_name = runtime["worker_service_name"] if worker_mode == "linux-systemd": - result = _run_shell(["systemctl", "start", service_name], timeout=30) + result = _run_systemctl(["start", service_name], timeout=30) if result.returncode != 0: return False, (result.stderr or result.stdout or "启动 Linux Worker 失败").strip() return True, f"Linux Worker 启动命令已发送: {service_name}" @@ -160,7 +224,7 @@ def stop_worker() -> tuple[bool, str]: worker_mode = runtime["worker_mode"] service_name = runtime["worker_service_name"] if worker_mode == "linux-systemd": - result = _run_shell(["systemctl", "stop", service_name], timeout=30) + result = _run_systemctl(["stop", service_name], timeout=30) if result.returncode != 0: return False, (result.stderr or result.stdout or "停止 Linux Worker 失败").strip() return True, f"Linux Worker 停止命令已发送: {service_name}" @@ -186,3 +250,17 @@ def stop_worker() -> tuple[bool, str]: if "NO_PROCESS" in output: return True, "当前没有运行中的检测端进程" return True, output or "检测端已停止" + + +def send_worker_command(action: str, payload: dict | None = None) -> tuple[bool, str]: + try: + redis_client = get_redis() + command_payload = {"action": action} + if payload: + command_payload.update(payload) + serialized = json.dumps(command_payload, ensure_ascii=False) + redis_client.set(WORKER_PENDING_COMMAND_KEY, serialized, ex=120) + redis_client.publish(WORKER_CONTROL_CHANNEL, serialized) + return True, f"已发送 Worker 控制指令: {action}" + except Exception as exc: + return False, f"发送 Worker 控制指令失败: {exc}" diff --git a/domain-api/app/sync_agent.py b/domain-api/app/sync_agent.py new file mode 100644 index 0000000..3a00a31 --- /dev/null +++ b/domain-api/app/sync_agent.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +import logging +import time + +from app.core.config import settings +from app.services.sync_push_service import push_runtime_projection_now + + +logger = logging.getLogger("domaincheck.sync_agent") + + +def main() -> None: + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", + ) + interval = max(10, int(settings.sync_poll_interval_seconds or 30)) + logger.info( + "sync agent started: node=%s source=%s target=%s interval=%ss enabled=%s", + settings.node_code, + settings.sync_source_region, + settings.sync_target_region, + interval, + settings.sync_push_enabled, + ) + while True: + try: + ok, message, data = push_runtime_projection_now() + logger.info("sync tick: ok=%s message=%s data=%s", ok, message, data) + except Exception as exc: + logger.exception("sync tick failed: %s", exc) + time.sleep(interval) + + +if __name__ == "__main__": + main() diff --git a/domain-api/deploy/linux/README.md b/domain-api/deploy/linux/README.md index 069bd91..6488fe5 100644 --- a/domain-api/deploy/linux/README.md +++ b/domain-api/deploy/linux/README.md @@ -111,6 +111,27 @@ sudo cp /opt/domaincheck/domain-api/deploy/systemd/domain-worker.service /etc/sy - `Group` - `Environment` +### 权限注意 + +正式 `systemd` 服务用户必须对下面这些路径有写权限： + +- `domain-api/runtime/` +- `domainCheck/detect_worker.log` + +如果之前曾用 `root` 手工跑过 API 或 Worker，可能会留下 `root` 所有者的运行目录或日志文件，导致正式服务态出现下面这类问题： + +- `/api/v1/imports/upload` 因无法创建 `runtime/imports/` 返回 `500` +- Worker 因无法写 `detect_worker.log` 进入循环重启 + +必要时先修正所有者： + +```bash +sudo chown -R www:www /opt/domaincheck/domain-api/runtime +sudo touch /opt/domaincheck/domainCheck/detect_worker.log +sudo chown www:www /opt/domaincheck/domainCheck/detect_worker.log +sudo chmod 664 /opt/domaincheck/domainCheck/detect_worker.log +``` + ## 六、启动顺序 ```bash @@ -135,6 +156,18 @@ journalctl -u domaincheck-api -n 200 --no-pager journalctl -u domaincheck-worker -n 200 --no-pager ``` +如果计划让 Web 后台里的“运行中心”控制 Worker 启停或触发 API 重启，还需要给服务用户补一条免密 sudo 规则。测试服当前使用的是： + +```text +Defaults:www !requiretty +Cmnd_Alias DOMAINCHECK_WORKER_CTL = /usr/bin/systemctl start domaincheck-worker, /usr/bin/systemctl stop domaincheck-worker +Cmnd_Alias DOMAINCHECK_WORKER_SHOW = /usr/bin/systemctl show domaincheck-worker --no-page --property=ActiveState\,SubState\,MainPID +Cmnd_Alias DOMAINCHECK_API_RESTART = /usr/bin/systemctl restart domaincheck-api, /usr/bin/systemctl --no-block restart domaincheck-api +www ALL=(root) NOPASSWD: DOMAINCHECK_WORKER_CTL, DOMAINCHECK_WORKER_SHOW, DOMAINCHECK_API_RESTART +``` + +其中 API 自重启建议使用 `systemctl --no-block restart domaincheck-api`，这样接口可以先返回成功，再由 `systemd` 异步完成服务切换，避免调用方因为等待自身停机而拿到 `500`。 + ## 七、联调检查 ### 1. API 健康检查 diff --git a/domain-api/deploy/multi-region/README.md b/domain-api/deploy/multi-region/README.md new file mode 100644 index 0000000..614438a --- /dev/null +++ b/domain-api/deploy/multi-region/README.md @@ -0,0 +1,354 @@ +# domainCheck 跨地域部署入口 + +本文档对应当前推荐的最小可用部署形态： + +- 国外 `1` 台：`domain-web + domain-api + postgresql_main` +- 大陆 `1` 台：`redis + postgresql_runtime + scheduler + detect-worker` + +目标： + +- 后台和 API 放国外 +- 检测执行放大陆 +- 后续新增大陆 Worker 时，不再改整体部署方式 + +## 一、目录约定 + +统一使用： + +```text +/opt/domaincheck +``` + +仓库内部署入口： + +- `deploy/multi-region/bootstrap_overseas.sh` +- `deploy/multi-region/bootstrap_mainland.sh` +- `deploy/multi-region/check_cluster.sh` +- `deploy/multi-region/check_mainland_controller.sh` +- `deploy/multi-region/simulate_cluster_node.py` +- `deploy/multi-region/simulate_multi_region.sh` +- `deploy/multi-region/prune_cluster_nodes.py` +- `deploy/multi-region/prune_cluster_nodes.sh` +- `deploy/multi-region/templates/*.env.example` + +## 二、国外机器部署 + +在国外机器执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_overseas.sh /opt/domaincheck +``` + +脚本会： + +- 创建运行目录 +- 检查 Python 虚拟环境 +- 安装 API 依赖 +- 安装 systemd 服务模板 +- 在 `/etc/default/domaincheck-api` 生成环境变量模板 +- 给出后续启动命令 + +## 三、大陆机器部署 + +在大陆机器执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller +``` + +如果后面新增纯 Worker 节点： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker +``` + +其中： + +- `controller` + 表示这台机器承担 `redis + runtime-db + scheduler + worker + sync-agent` +- `worker` + 表示这台机器只承担 `detect-worker` + +脚本还会在大陆机器生成： + +- `/etc/default/domaincheck-worker` + +当前模板已内置同步相关占位配置： + +- `SYNC_PUSH_ENABLED` +- `SYNC_SOURCE_REGION` +- `SYNC_TARGET_REGION` +- `SYNC_TARGET_API_BASE_URL` +- `SYNC_SHARED_TOKEN` +- `SYNC_BATCH_SIZE` +- `SYNC_POLL_INTERVAL_SECONDS` + +建议把每台大陆节点自己的身份信息放在这里，而不是直接修改 service 文件正文： + +- `NODE_CODE` +- `NODE_REGION=mainland` +- `NODE_ROLE=control` + - 仅 `controller` 模板使用 +- `NODE_ROLE=worker` + - 仅纯 Worker 模板使用 +- `DB_*` +- `REDIS_*` + +## 四、当前脚本定位 + +当前入口脚本是第一版“标准化部署脚手架”，优先解决： + +- 目录统一 +- systemd 模板统一 +- 节点环境变量入口统一 +- 新增节点时操作步骤统一 + +当前大陆 bootstrap 还会自动补齐一组最小 Python 依赖： + +- `fastapi` +- `uvicorn` +- `pydantic-settings` +- `psycopg2-binary` +- `redis` + +这样 `domaincheck-sync-agent` 在大陆 controller 节点上可以直接启动，不会因为 `app.sync_agent` 缺少 API 侧依赖而失败。 + +当前脚手架还额外区分了“自动同步由谁跑”： + +- 海外控制面： + - `domain-api` + - 负责接收 `runtime/sync-ingest` +- 内地 controller 节点： + - `domaincheck-sync-agent` + - 负责按轮询周期把本地最新投影推送到海外控制面 + +当前还不会自动安装数据库主从或自动创建跨地域同步链路。 + +原因： + +- 当前项目仍处于单 Worker 改造向多 Worker 任务模型过渡阶段 +- 真正的任务调度与跨地域结果同步需要后续代码配合落地 +- 但当前已经预留同步配置模板与同步记录查询接口，便于后续接入 `sync-service` +- 当前控制面还会自动把运行态摘要写入 `detect_sync_records` + - `sync_type=runtime_projection` + - 仅在关键状态变化或达到最小采样间隔时写入 + - 作用是先把“同步观测面”跑起来，而不是替代最终的跨地域结果同步服务 +- 当 `SYNC_PUSH_ENABLED=true` 且配置了 `SYNC_TARGET_API_BASE_URL` 后： + - 大陆 `domaincheck-sync-agent` 会按 `SYNC_POLL_INTERVAL_SECONDS` 自动尝试推送： + - 最新一条 `runtime_projection` + - 当前积压的 `detect_result_projection` 批次 + - 目标控制面通过 `POST /api/v1/runtime/sync-ingest` 接收 + - 若配置了 `SYNC_SHARED_TOKEN`，接收端会校验 `X-Domaincheck-Sync-Token` + - `GET /api/v1/runtime/sync-summary` 还会返回： + - `detect_result_batches` + 用于直接观察最近检测任务的结果同步是否已接收、仍待推送或发生失败 + +## 五、当前已落地能力 + +截至 `2026-04-16`，当前代码已经具备： + +- 节点注册与心跳 +- 运行态表自动初始化 +- 集群节点状态接口 +- 检测任务主表与任务项表 +- 活跃任务摘要与任务详情接口 +- Worker 小批量领取、租约续租、重启释放和过期回收 + +可验证接口： + +```bash +curl http://127.0.0.1:8100/api/v1/runtime/status +curl http://127.0.0.1:8100/api/v1/runtime/readiness +curl http://127.0.0.1:8100/api/v1/runtime/cluster +curl http://127.0.0.1:8100/api/v1/runtime/sync-summary +curl http://127.0.0.1:8100/api/v1/runtime/sync-records?limit=10 +curl http://127.0.0.1:8100/api/v1/detect/job/active +curl http://127.0.0.1:8100/api/v1/detect/jobs +``` + +也可以直接执行一键联调： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100 +``` + +这条命令当前除了原始接口输出，还会额外给出一段压缩摘要，直接汇总： + +- `ready / attention / blocking` +- 在线控制面 / Worker 数 +- `busy / stale / offline` 节点 +- 同步投影 / 接收计数 +- 结果批次的： + - `synced` + - `delivered` + - `pushing` + - `projected` +- `failed` +- `unsynced` + +如果要在大陆 controller 节点本机确认“这台机器本身是否已经具备 controller 身份”，还可以执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_mainland_controller.sh +``` + +这条命令会直接检查： + +- `/etc/default/domaincheck-worker` 是否存在 +- `NODE_ROLE=control` 是否正确 +- `SYNC_PUSH_ENABLED` / `SYNC_TARGET_API_BASE_URL` 是否已配置 +- `domaincheck-worker` +- `domaincheck-sync-agent` + 是否已启用并处于运行态 + +如果当前还无法真正部署到大陆机器，也可以先在国外测试机上做“单机模拟多节点联调”： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/simulate_multi_region.sh http://127.0.0.1:8100 +``` + +这条命令会临时模拟： + +- `mainland-controller-sim` +- `mainland-worker-sim-01` + +用于提前验证： + +- `runtime/readiness` +- `runtime/cluster` +- 运行中心顶部的多机就绪度结论 + +结束时按 `Ctrl+C`，脚本会自动清理模拟节点记录。 + +如果测试服里已经残留了很久没心跳的旧节点记录，导致 `runtime/readiness` 一直被离线节点拖成 `attention`，可以先做清理： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 --dry-run +bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 +``` + +如果只想清理某个确定已经废弃的节点： + +```bash +bash deploy/multi-region/prune_cluster_nodes.sh --node-code mainland-worker-01 +``` + +如果要判断大陆 Worker 是否已经真正接入，不要只看 `systemctl`，还要看： + +- `/api/v1/runtime/cluster` 中是否出现对应 `node_code` +- `last_heartbeat_at` 是否持续刷新 +- `role` / `region` 是否符合预期 +- `metadata.job_id / metadata.cycle_token` 是否能在执行时出现 +- `summary.online_worker_nodes` 是否大于 `0` +- `summary.status_counts.busy` 是否会在执行中增加 + +## 六、第二台大陆 Worker 接入建议 + +后续新增大陆 Worker 时，建议顺序为： + +1. 拉最新代码到新大陆机器 +2. 准备与主执行面一致的 Python 环境 +3. 配置该节点自己的： + - `NODE_CODE` + - `NODE_REGION=mainland` + - `NODE_ROLE=worker` +4. 执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker +``` + +5. 启动后在国外控制面检查： + +```bash +curl http://127.0.0.1:8100/api/v1/runtime/cluster +``` + +如果接口里出现新节点，并且心跳持续更新，说明节点接入成功。 + +如果是第二台及以上大陆 Worker，最少还要确认： + +- `NODE_CODE` 与其它节点不重复 +- 连接的是同一套大陆 Redis / runtime-db +- `/etc/default/domaincheck-worker` 已按该机器单独填写 +- 若该机器不是 controller，则不要额外启 `domaincheck-sync-agent` + +## 七、当前推荐联调命令 + +国外控制面建议至少保留下面这组命令： + +```bash +curl http://127.0.0.1:8100/health +curl http://127.0.0.1:8100/api/v1/runtime/preflight +curl http://127.0.0.1:8100/api/v1/runtime/readiness +curl http://127.0.0.1:8100/api/v1/runtime/cluster +curl http://127.0.0.1:8100/api/v1/detect/status +curl http://127.0.0.1:8100/api/v1/detect/job/active +curl http://127.0.0.1:8100/api/v1/detect/jobs?limit=5 +``` + +如需一次性确认控制面和执行面都接通，可以直接执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100 +``` + +观察重点： + +- 控制面节点心跳应持续刷新，而不是只在 API 启动时更新一次 +- 大陆 Worker 节点在运行检测时应显示 `busy` +- `detect/job/active` 应能看到当前任务总量、完成量、节点分布和最近事件 +- `detect/jobs` 可用于回看最近几轮任务是否正常收敛 +- 当代理池暂时为空但允许直连时，检测控制页会显示 `降级直连` +- 当代理池为空且不允许直连时，检测会话阶段会显示 `等待代理` +- `runtime/cluster.summary` 中应能直接看出： + - 当前在线控制面节点数 + - 当前在线 Worker 节点数 + - 当前 `busy / stale / offline` 节点清单 +- `runtime/sync-summary` 中可直接查看： + - 是否启用同步推送 + - 当前配置的源地域 / 目标地域 + - 最近同步记录和状态分布 + - 若当前还未接入真正的 `sync-service`，也应至少能看到： + - `runtime_projection` + - `detect_result_projection` + - 接入自动推送后，还应能看到： + - `runtime_push` + - `runtime_ingest` + - `detect_result_ingest` + +大陆 controller 节点建议额外确认： + +```bash +systemctl status domaincheck-sync-agent --no-pager -l +``` + +如果同步配置已填写完整，则期望： + +- `domaincheck-sync-agent` 为 `active (running)` +- 海外控制面的 `runtime/sync-summary` 中能同时看到： + - `runtime_projection` + - `runtime_push` + - `runtime_ingest` + +## 八、后续演进 + +后续会继续补： + +- 运行库初始化脚本 +- 节点配置模板 +- 结果同步服务 +- 多 Worker 调度服务模板 + +配套设计文档见： + +- `docs/16_domainCheck_多机检测与跨地域部署设计.md` diff --git a/domain-api/deploy/multi-region/bootstrap_mainland.sh b/domain-api/deploy/multi-region/bootstrap_mainland.sh new file mode 100755 index 0000000..c6bdc3b --- /dev/null +++ b/domain-api/deploy/multi-region/bootstrap_mainland.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +set -euo pipefail + +BASE_DIR="${1:-/opt/domaincheck}" +ROLE="${2:-controller}" +API_DIR="$BASE_DIR/domain-api" +DOMAIN_DIR="$BASE_DIR/domainCheck" +SERVICE_NAME="domaincheck-worker" +SYNC_AGENT_SERVICE_NAME="domaincheck-sync-agent" +ENV_FILE="/etc/default/domaincheck-worker" +TEMPLATE_DIR="$API_DIR/deploy/multi-region/templates" + +if [ "$ROLE" != "controller" ] && [ "$ROLE" != "worker" ]; then + echo "invalid role: $ROLE" + echo "usage: bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck [controller|worker]" + exit 1 +fi + +echo "[1/7] preparing directories" +mkdir -p "$BASE_DIR" "$DOMAIN_DIR" "$API_DIR/runtime" + +if [ "$ROLE" = "controller" ]; then + mkdir -p /var/lib/domaincheck-runtime-db /var/lib/domaincheck-redis +fi + +echo "[2/7] checking python environment" +if [ ! -x "$DOMAIN_DIR/.venv/bin/python" ]; then + echo "missing python venv: $DOMAIN_DIR/.venv/bin/python" + echo "please deploy domainCheck first and create .venv there" + exit 1 +fi + +echo "[2.5/7] installing mainland runtime dependencies" +"$DOMAIN_DIR/.venv/bin/pip" install -q fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart + +echo "[3/7] fixing runtime permissions" +touch "$DOMAIN_DIR/detect_worker.log" +chmod 664 "$DOMAIN_DIR/detect_worker.log" + +echo "[4/7] installing worker service" +install -m 0644 "$API_DIR/deploy/systemd/domain-worker.service" "/etc/systemd/system/${SERVICE_NAME}.service" +if [ "$ROLE" = "controller" ]; then + install -m 0644 "$API_DIR/deploy/systemd/domain-sync-agent.service" "/etc/systemd/system/${SYNC_AGENT_SERVICE_NAME}.service" +fi + +echo "[4.5/7] preparing environment file" +if [ ! -f "$ENV_FILE" ]; then + if [ "$ROLE" = "controller" ]; then + TEMPLATE_FILE="$TEMPLATE_DIR/domaincheck-worker.controller.env.example" + else + TEMPLATE_FILE="$TEMPLATE_DIR/domaincheck-worker.worker.env.example" + fi + install -m 0644 "$TEMPLATE_FILE" "$ENV_FILE" + sed -i "s#__BASE_DIR__#${BASE_DIR}#g" "$ENV_FILE" + chmod 0644 "$ENV_FILE" + echo "created env template: $ENV_FILE" +else + echo "env file already exists: $ENV_FILE" +fi + +echo "[5/7] reloading systemd" +systemctl daemon-reload + +if [ "$ROLE" = "controller" ]; then + echo "[6/7] controller notes" + echo "controller mode selected" + echo "please ensure local redis and postgresql_runtime are installed and listening on localhost" + echo "sync agent service: ${SYNC_AGENT_SERVICE_NAME}" +else + echo "[6/7] worker node notes" + echo "worker mode selected" + echo "this node is expected to connect to mainland redis and runtime-db" +fi + +echo "[7/7] summary" +echo "mainland bootstrap completed" +echo "role: ${ROLE}" +echo "service: ${SERVICE_NAME}" +echo "next commands:" +echo " systemctl enable ${SERVICE_NAME}" +echo " systemctl restart ${SERVICE_NAME}" +echo " systemctl status ${SERVICE_NAME} --no-pager -l" +if [ "$ROLE" = "controller" ]; then + echo " systemctl enable ${SYNC_AGENT_SERVICE_NAME}" + echo " systemctl restart ${SYNC_AGENT_SERVICE_NAME}" + echo " systemctl status ${SYNC_AGENT_SERVICE_NAME} --no-pager -l" +fi +echo " vi ${ENV_FILE}" diff --git a/domain-api/deploy/multi-region/bootstrap_overseas.sh b/domain-api/deploy/multi-region/bootstrap_overseas.sh new file mode 100755 index 0000000..0c5ede7 --- /dev/null +++ b/domain-api/deploy/multi-region/bootstrap_overseas.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +set -euo pipefail + +BASE_DIR="${1:-/opt/domaincheck}" +API_DIR="$BASE_DIR/domain-api" +WEB_DIR="$BASE_DIR/domain-web" +RUNTIME_DIR="$API_DIR/runtime" +SERVICE_NAME="domaincheck-api" +ENV_FILE="/etc/default/domaincheck-api" +TEMPLATE_FILE="$API_DIR/deploy/multi-region/templates/domaincheck-api.env.example" + +echo "[1/6] preparing directories" +mkdir -p "$BASE_DIR" "$API_DIR" "$WEB_DIR" "$RUNTIME_DIR" + +echo "[2/6] checking python environment" +if [ ! -x "$BASE_DIR/domainCheck/.venv/bin/python" ]; then + echo "missing python venv: $BASE_DIR/domainCheck/.venv/bin/python" + echo "please deploy domainCheck first and create .venv there" + exit 1 +fi + +echo "[3/6] installing api dependencies" +"$BASE_DIR/domainCheck/.venv/bin/pip" install -q fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart + +echo "[4/6] installing systemd service" +install -m 0644 "$API_DIR/deploy/systemd/domain-api.service" "/etc/systemd/system/${SERVICE_NAME}.service" + +echo "[4.5/6] preparing environment file" +if [ ! -f "$ENV_FILE" ]; then + install -m 0644 "$TEMPLATE_FILE" "$ENV_FILE" + sed -i "s#__BASE_DIR__#${BASE_DIR}#g" "$ENV_FILE" + chmod 0644 "$ENV_FILE" + echo "created env template: $ENV_FILE" +else + echo "env file already exists: $ENV_FILE" +fi + +echo "[5/6] reloading systemd" +systemctl daemon-reload + +echo "[6/6] summary" +echo "overseas bootstrap completed" +echo "service: ${SERVICE_NAME}" +echo "next commands:" +echo " systemctl enable ${SERVICE_NAME}" +echo " systemctl restart ${SERVICE_NAME}" +echo " systemctl status ${SERVICE_NAME} --no-pager -l" +echo " vi ${ENV_FILE}" diff --git a/domain-api/deploy/multi-region/check_cluster.sh b/domain-api/deploy/multi-region/check_cluster.sh new file mode 100755 index 0000000..8dd98a4 --- /dev/null +++ b/domain-api/deploy/multi-region/check_cluster.sh @@ -0,0 +1,89 @@ +#!/usr/bin/env bash +set -euo pipefail + +BASE_URL="${1:-http://127.0.0.1:8100}" +PYTHON_BIN="${PYTHON_BIN:-python3}" + +echo "[1/7] health" +curl -fsS "${BASE_URL}/health" +echo +echo + +echo "[2/7] runtime readiness" +curl -fsS "${BASE_URL}/api/v1/runtime/readiness" +echo +echo + +echo "[3/7] runtime cluster" +curl -fsS "${BASE_URL}/api/v1/runtime/cluster" +echo +echo + +echo "[4/7] runtime sync summary" +curl -fsS "${BASE_URL}/api/v1/runtime/sync-summary" +echo +echo + +echo "[5/7] detect status" +curl -fsS "${BASE_URL}/api/v1/detect/status" +echo +echo + +echo "[6/7] active detect job" +curl -fsS "${BASE_URL}/api/v1/detect/job/active" +echo +echo + +echo "[7/7] condensed summary" +READINESS_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/readiness")" +SYNC_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/sync-summary")" +CLUSTER_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/cluster")" +"${PYTHON_BIN}" - <<'PY' "$READINESS_JSON" "$SYNC_JSON" "$CLUSTER_JSON" +import json +import sys + +readiness = json.loads(sys.argv[1]).get("data", {}) +sync = json.loads(sys.argv[2]).get("data", {}) +cluster = json.loads(sys.argv[3]).get("data", {}) +batches = (sync.get("detect_result_batches") or {}) +states = batches.get("state_counts") or {} +summary = cluster.get("summary") or {} +print( + json.dumps( + { + "readiness": { + "status": readiness.get("status"), + "ready": bool(readiness.get("ready", False)), + "summary": readiness.get("summary", ""), + }, + "cluster": { + "online_control_nodes": summary.get("online_control_nodes", 0), + "online_worker_nodes": summary.get("online_worker_nodes", 0), + "busy_nodes": summary.get("busy_nodes", []), + "stale_nodes": summary.get("stale_nodes", []), + "offline_nodes": summary.get("offline_nodes", []), + }, + "sync": { + "enabled": bool(sync.get("enabled", False)), + "source_region": sync.get("source_region"), + "target_region": sync.get("target_region"), + "runtime_projection": (sync.get("type_counts") or {}).get("runtime_projection", 0), + "detect_result_projection": (sync.get("type_counts") or {}).get("detect_result_projection", 0), + "runtime_ingest": (sync.get("type_counts") or {}).get("runtime_ingest", 0), + "detect_result_ingest": (sync.get("type_counts") or {}).get("detect_result_ingest", 0), + }, + "detect_result_batches": { + "jobs_total": batches.get("jobs_total", 0), + "synced": states.get("synced", 0), + "delivered": states.get("delivered", 0), + "pushing": states.get("pushing", 0), + "projected": states.get("projected", 0), + "failed": states.get("failed", 0), + "unsynced": states.get("unsynced", 0), + }, + }, + ensure_ascii=False, + indent=2, + ) +) +PY diff --git a/domain-api/deploy/multi-region/check_mainland_controller.sh b/domain-api/deploy/multi-region/check_mainland_controller.sh new file mode 100755 index 0000000..e28ff7b --- /dev/null +++ b/domain-api/deploy/multi-region/check_mainland_controller.sh @@ -0,0 +1,76 @@ +#!/usr/bin/env bash +set -euo pipefail + +ENV_FILE="${1:-/etc/default/domaincheck-worker}" +WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}" +SYNC_AGENT_SERVICE="${SYNC_AGENT_SERVICE:-domaincheck-sync-agent}" +PYTHON_BIN="${PYTHON_BIN:-python3}" + +echo "[1/5] environment file" +if [ ! -f "$ENV_FILE" ]; then + echo "missing env file: $ENV_FILE" + exit 1 +fi +echo "env file: $ENV_FILE" +echo + +echo "[2/5] key env summary" +"${PYTHON_BIN}" - <<'PY' "$ENV_FILE" +import json +import sys +from pathlib import Path + +env_path = Path(sys.argv[1]) +values = {} +for line in env_path.read_text(encoding="utf-8").splitlines(): + text = line.strip() + if not text or text.startswith("#") or "=" not in text: + continue + key, value = text.split("=", 1) + values[key.strip()] = value.strip() + +required = { + "NODE_CODE": values.get("NODE_CODE", ""), + "NODE_REGION": values.get("NODE_REGION", ""), + "NODE_ROLE": values.get("NODE_ROLE", ""), + "SYNC_PUSH_ENABLED": values.get("SYNC_PUSH_ENABLED", ""), + "SYNC_SOURCE_REGION": values.get("SYNC_SOURCE_REGION", ""), + "SYNC_TARGET_REGION": values.get("SYNC_TARGET_REGION", ""), + "SYNC_TARGET_API_BASE_URL": values.get("SYNC_TARGET_API_BASE_URL", ""), +} +print(json.dumps(required, ensure_ascii=False, indent=2)) + +errors = [] +if required["NODE_REGION"] != "mainland": + errors.append("NODE_REGION 必须为 mainland") +if required["NODE_ROLE"] != "control": + errors.append("NODE_ROLE 必须为 control") +if str(required["SYNC_PUSH_ENABLED"]).lower() != "true": + errors.append("SYNC_PUSH_ENABLED 必须为 true") +if required["SYNC_SOURCE_REGION"] != "mainland": + errors.append("SYNC_SOURCE_REGION 必须为 mainland") +if not required["SYNC_TARGET_API_BASE_URL"]: + errors.append("必须配置 SYNC_TARGET_API_BASE_URL") + +if errors: + print() + print("env validation failed:") + for item in errors: + print(f"- {item}") + sys.exit(1) +PY +echo + +echo "[3/5] systemd status" +systemctl is-enabled "$WORKER_SERVICE" || true +systemctl is-active "$WORKER_SERVICE" +systemctl is-enabled "$SYNC_AGENT_SERVICE" || true +systemctl is-active "$SYNC_AGENT_SERVICE" +echo + +echo "[4/5] worker service detail" +systemctl status "$WORKER_SERVICE" --no-pager -l | sed -n '1,25p' +echo + +echo "[5/5] sync-agent detail" +systemctl status "$SYNC_AGENT_SERVICE" --no-pager -l | sed -n '1,25p' diff --git a/domain-api/deploy/multi-region/install_mainland_controller_quick.conf.example b/domain-api/deploy/multi-region/install_mainland_controller_quick.conf.example new file mode 100644 index 0000000..60283e8 --- /dev/null +++ b/domain-api/deploy/multi-region/install_mainland_controller_quick.conf.example @@ -0,0 +1,8 @@ +GIT_REPO_URL="YOUR_GIT_REPO_URL" +GIT_BRANCH="main" +DB_NAME="domain" +DB_USER="domainuser" +DB_PASSWORD="CHANGE_ME_DB_PASSWORD" +SYNC_SHARED_TOKEN="CHANGE_ME_SYNC_SHARED_TOKEN" +MAINLAND_CONTROLLER_NODE_CODE="mainland-controller-01" +TARGET_API_BASE_URL="https://api.domain.com/api/v1" diff --git a/domain-api/deploy/multi-region/install_mainland_controller_quick.sh b/domain-api/deploy/multi-region/install_mainland_controller_quick.sh new file mode 100755 index 0000000..01233fe --- /dev/null +++ b/domain-api/deploy/multi-region/install_mainland_controller_quick.sh @@ -0,0 +1,110 @@ +#!/usr/bin/env bash +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +CONFIG_FILE="${SCRIPT_DIR}/install_mainland_controller_quick.conf" +CONFIG_EXAMPLE="${SCRIPT_DIR}/install_mainland_controller_quick.conf.example" + +if [[ ! -f "${CONFIG_FILE}" ]]; then + echo "未找到配置文件：${CONFIG_FILE}" + echo "请先执行：cp ${CONFIG_EXAMPLE} ${CONFIG_FILE}" + exit 1 +fi + +# shellcheck disable=SC1090 +source "${CONFIG_FILE}" + +: "${GIT_REPO_URL:=YOUR_GIT_REPO_URL}" +: "${GIT_BRANCH:=main}" +: "${DB_NAME:=domain}" +: "${DB_USER:=domainuser}" +: "${DB_PASSWORD:=CHANGE_ME_DB_PASSWORD}" +: "${SYNC_SHARED_TOKEN:=CHANGE_ME_SYNC_SHARED_TOKEN}" +: "${MAINLAND_CONTROLLER_NODE_CODE:=mainland-controller-01}" +: "${TARGET_API_BASE_URL:=https://api.domain.com/api/v1}" + +WORKSPACE_ROOT="/www/wwwroot" +REPO_DIR="${WORKSPACE_ROOT}/getDomain" +BASE_DIR="/opt/domaincheck" +DOMAINCHECK_DIR="${BASE_DIR}/domainCheck" +DOMAIN_API_DIR="${BASE_DIR}/domain-api" + +if [[ "${GIT_REPO_URL}" == "YOUR_GIT_REPO_URL" || "${DB_PASSWORD}" == "CHANGE_ME_DB_PASSWORD" || "${SYNC_SHARED_TOKEN}" == "CHANGE_ME_SYNC_SHARED_TOKEN" ]]; then + echo "请先修改配置文件中的必填项：GIT_REPO_URL / DB_PASSWORD / SYNC_SHARED_TOKEN" + exit 1 +fi + +mkdir -p "${WORKSPACE_ROOT}" +cd "${WORKSPACE_ROOT}" +if [[ ! -d "${REPO_DIR}/.git" ]]; then + git clone "${GIT_REPO_URL}" getDomain +fi +cd "${REPO_DIR}" +git fetch --all +git checkout "${GIT_BRANCH}" +git pull --ff-only origin "${GIT_BRANCH}" + +mkdir -p "${BASE_DIR}" +ln -sfn "${REPO_DIR}/domain-api" "${DOMAIN_API_DIR}" +ln -sfn "${REPO_DIR}/domain-web" "${BASE_DIR}/domain-web" +ln -sfn "${REPO_DIR}/domainCheck" "${DOMAINCHECK_DIR}" + +cd "${DOMAINCHECK_DIR}" +python3.11 -m venv .venv || true +source .venv/bin/activate +pip install --upgrade pip +pip install -r requirements.txt +"${DOMAINCHECK_DIR}/.venv/bin/pip" install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart + +cat >"${DOMAINCHECK_DIR}/.env" </etc/default/domaincheck-worker <"${DOMAINCHECK_DIR}/.env" </etc/default/domaincheck-worker <"${DOMAINCHECK_DIR}/.env" <"${DOMAIN_WEB_DIR}/.env.production" < None: + parser = argparse.ArgumentParser(description="Prune stale/offline cluster node records from detect_worker_nodes.") + parser.add_argument("--minutes", type=int, default=30, help="delete nodes whose heartbeat is older than this many minutes") + parser.add_argument("--node-code", action="append", default=[], help="delete a specific node code; can be passed multiple times") + parser.add_argument("--dry-run", action="store_true") + args = parser.parse_args() + + cutoff = datetime.now() - timedelta(minutes=max(1, int(args.minutes or 30))) + removed: list[dict] = [] + with get_db() as conn: + with conn.cursor() as cur: + if args.node_code: + cur.execute( + """ + SELECT node_code, region, role, status, last_heartbeat_at + FROM detect_worker_nodes + WHERE node_code = ANY(%s) + ORDER BY node_code ASC + """, + (list(args.node_code),), + ) + else: + cur.execute( + """ + SELECT node_code, region, role, status, last_heartbeat_at + FROM detect_worker_nodes + WHERE last_heartbeat_at < %s + ORDER BY last_heartbeat_at ASC, node_code ASC + """, + (cutoff,), + ) + rows = cur.fetchall() + for row in rows: + removed.append( + { + "node_code": row[0], + "region": row[1], + "role": row[2], + "status": row[3], + "last_heartbeat_at": row[4].isoformat(sep=" ", timespec="seconds") if row[4] else "", + } + ) + if removed and not args.dry_run: + cur.execute( + "DELETE FROM detect_worker_nodes WHERE node_code = ANY(%s)", + ([item["node_code"] for item in removed],), + ) + if removed and not args.dry_run: + conn.commit() + + if not removed: + print("no cluster nodes matched prune conditions") + return + + print("matched cluster nodes:") + for item in removed: + print( + f"- {item['node_code']} | {item['region']} | {item['role']} | {item['status']} | {item['last_heartbeat_at']}" + ) + if args.dry_run: + print("dry-run only, nothing deleted") + else: + print(f"deleted {len(removed)} cluster node record(s)") + + +if __name__ == "__main__": + main() diff --git a/domain-api/deploy/multi-region/prune_cluster_nodes.sh b/domain-api/deploy/multi-region/prune_cluster_nodes.sh new file mode 100755 index 0000000..deec7f8 --- /dev/null +++ b/domain-api/deploy/multi-region/prune_cluster_nodes.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +set -euo pipefail + +API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}" + +exec "$PYTHON_BIN" "$API_DIR/deploy/multi-region/prune_cluster_nodes.py" "$@" diff --git a/domain-api/deploy/multi-region/rehearse_multi_region.sh b/domain-api/deploy/multi-region/rehearse_multi_region.sh new file mode 100755 index 0000000..2b98eef --- /dev/null +++ b/domain-api/deploy/multi-region/rehearse_multi_region.sh @@ -0,0 +1,135 @@ +#!/usr/bin/env bash +set -euo pipefail + +BASE_URL="${1:-http://127.0.0.1:8100}" +API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}" +SIM_SCRIPT="$API_DIR/deploy/multi-region/simulate_cluster_node.py" +PRUNE_SCRIPT="$API_DIR/deploy/multi-region/prune_cluster_nodes.sh" +RUNTIME_DIR="$API_DIR/runtime" +mkdir -p "$RUNTIME_DIR" + +CTRL_LOG="$RUNTIME_DIR/rehearse-mainland-controller.log" +WORKER1_LOG="$RUNTIME_DIR/rehearse-mainland-worker-01.log" +WORKER2_LOG="$RUNTIME_DIR/rehearse-mainland-worker-02.log" + +CTRL_NODE="mainland-controller-rehearsal" +WORKER1_NODE="mainland-worker-rehearsal-01" +WORKER2_NODE="mainland-worker-rehearsal-02" + +cleanup() { + set +e + if [ -n "${CTRL_PID:-}" ]; then kill "$CTRL_PID" >/dev/null 2>&1 || true; fi + if [ -n "${WORKER1_PID:-}" ]; then kill "$WORKER1_PID" >/dev/null 2>&1 || true; fi + if [ -n "${WORKER2_PID:-}" ]; then kill "$WORKER2_PID" >/dev/null 2>&1 || true; fi + bash "$PRUNE_SCRIPT" --node-code "$CTRL_NODE" --node-code "$WORKER1_NODE" --node-code "$WORKER2_NODE" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +echo "[0/6] cleaning possible stale rehearsal nodes" +bash "$PRUNE_SCRIPT" --node-code "$CTRL_NODE" --node-code "$WORKER1_NODE" --node-code "$WORKER2_NODE" >/dev/null 2>&1 || true + +echo "[1/6] start simulated mainland controller" +"$PYTHON_BIN" "$SIM_SCRIPT" \ + --node-code "$CTRL_NODE" \ + --region mainland \ + --role control \ + --status online \ + --detail "多机演练 controller" \ + --interval 15 >"$CTRL_LOG" 2>&1 & +CTRL_PID=$! + +echo "[2/6] start simulated mainland workers" +"$PYTHON_BIN" "$SIM_SCRIPT" \ + --node-code "$WORKER1_NODE" \ + --region mainland \ + --role worker \ + --status busy \ + --current-load 1 \ + --phase "running" \ + --detail "多机演练 worker-01" \ + --job-id 1 \ + --job-code "detect-rehearsal-cluster" \ + --cycle-token "rehearsal001" \ + --active-threads 5 \ + --available-proxy-count 12 \ + --interval 15 >"$WORKER1_LOG" 2>&1 & +WORKER1_PID=$! + +"$PYTHON_BIN" "$SIM_SCRIPT" \ + --node-code "$WORKER2_NODE" \ + --region mainland \ + --role worker \ + --status online \ + --current-load 0 \ + --phase "idle" \ + --detail "多机演练 worker-02" \ + --job-id 0 \ + --job-code "" \ + --cycle-token "" \ + --active-threads 0 \ + --available-proxy-count 8 \ + --interval 15 >"$WORKER2_LOG" 2>&1 & +WORKER2_PID=$! + +sleep 2 + +echo "[3/6] fetch runtime readiness" +READINESS_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/readiness")" +echo "$READINESS_JSON" +echo + +echo "[4/6] fetch runtime cluster" +CLUSTER_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/cluster")" +echo "$CLUSTER_JSON" +echo + +echo "[5/6] assert rehearsal result" +"${PYTHON_BIN}" - <<'PY' "$READINESS_JSON" "$CLUSTER_JSON" "$CTRL_NODE" "$WORKER1_NODE" "$WORKER2_NODE" +import json +import sys + +readiness = json.loads(sys.argv[1]).get("data", {}) +cluster = json.loads(sys.argv[2]).get("data", {}) +ctrl_node, worker1_node, worker2_node = sys.argv[3:6] + +errors = [] +if readiness.get("status") != "ready": + errors.append(f"readiness.status expected ready, got {readiness.get('status')!r}") +if not readiness.get("ready", False): + errors.append("readiness.ready expected true") + +summary = cluster.get("summary") or {} +if int(summary.get("online_control_nodes", 0) or 0) < 2: + errors.append(f"online_control_nodes expected >= 2, got {summary.get('online_control_nodes')}") +if int(summary.get("online_worker_nodes", 0) or 0) < 2: + errors.append(f"online_worker_nodes expected >= 2, got {summary.get('online_worker_nodes')}") + +nodes = {str(item.get('node_code') or ''): item for item in (cluster.get("nodes") or [])} +for node_code in (ctrl_node, worker1_node, worker2_node): + if node_code not in nodes: + errors.append(f"missing simulated node in cluster: {node_code}") + +if worker1_node in nodes and str(nodes[worker1_node].get("status")) != "busy": + errors.append(f"{worker1_node} expected busy, got {nodes[worker1_node].get('status')!r}") +if worker2_node in nodes and str(nodes[worker2_node].get("status")) != "online": + errors.append(f"{worker2_node} expected online, got {nodes[worker2_node].get('status')!r}") + +if errors: + print("rehearsal failed:") + for item in errors: + print(f"- {item}") + raise SystemExit(1) + +print("rehearsal passed") +print(json.dumps({ + "readiness": readiness, + "cluster_summary": summary, +}, ensure_ascii=False, indent=2)) +PY + +echo +echo "[6/6] rehearsal completed" +echo "controller log: $CTRL_LOG" +echo "worker1 log: $WORKER1_LOG" +echo "worker2 log: $WORKER2_LOG" diff --git a/domain-api/deploy/multi-region/simulate_cluster_node.py b/domain-api/deploy/multi-region/simulate_cluster_node.py new file mode 100755 index 0000000..2c45812 --- /dev/null +++ b/domain-api/deploy/multi-region/simulate_cluster_node.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path + +API_ROOT = Path(__file__).resolve().parents[2] +if str(API_ROOT) not in sys.path: + sys.path.insert(0, str(API_ROOT)) + +from app.core.db import get_db +from app.services.cluster_runtime_service import ensure_runtime_schema, register_node_heartbeat + + +def _metadata_from_args(args: argparse.Namespace) -> dict: + metadata = { + "service": "cluster-simulator", + "simulated": True, + "phase": args.phase or "", + "detail": args.detail or "", + } + if args.job_id: + metadata["job_id"] = args.job_id + if args.job_code: + metadata["job_code"] = args.job_code + if args.cycle_token: + metadata["cycle_token"] = args.cycle_token + if args.active_threads is not None: + metadata["active_threads"] = args.active_threads + if args.available_proxy_count is not None: + metadata["available_proxy_count"] = args.available_proxy_count + if args.metadata_json: + try: + custom = json.loads(args.metadata_json) + if isinstance(custom, dict): + metadata.update(custom) + except Exception as exc: + raise SystemExit(f"invalid --metadata-json: {exc}") from exc + return metadata + + +def _delete_node(node_code: str) -> None: + with get_db() as conn: + with conn.cursor() as cur: + cur.execute("DELETE FROM detect_worker_nodes WHERE node_code = %s", (str(node_code),)) + conn.commit() + + +def main() -> None: + parser = argparse.ArgumentParser(description="Simulate a cluster node heartbeat for multi-region rehearsal.") + parser.add_argument("--node-code", required=True) + parser.add_argument("--region", required=True) + parser.add_argument("--role", required=True, choices=("control", "worker")) + parser.add_argument("--status", default="online") + parser.add_argument("--current-load", type=int, default=0) + parser.add_argument("--phase", default="") + parser.add_argument("--detail", default="") + parser.add_argument("--job-id", type=int, default=0) + parser.add_argument("--job-code", default="") + parser.add_argument("--cycle-token", default="") + parser.add_argument("--active-threads", type=int, default=None) + parser.add_argument("--available-proxy-count", type=int, default=None) + parser.add_argument("--metadata-json", default="") + parser.add_argument("--interval", type=int, default=20) + parser.add_argument("--iterations", type=int, default=0, help="0 means run forever") + parser.add_argument("--clear", action="store_true", help="delete the node record and exit") + args = parser.parse_args() + + ensure_runtime_schema() + if args.clear: + _delete_node(args.node_code) + print(f"cleared simulated node: {args.node_code}") + return + + metadata = _metadata_from_args(args) + interval = max(5, int(args.interval or 20)) + iterations = max(0, int(args.iterations or 0)) + current = 0 + while True: + current += 1 + register_node_heartbeat( + node_code=args.node_code, + region=args.region, + role=args.role, + status=args.status, + current_load=max(0, int(args.current_load or 0)), + metadata=metadata, + ) + print( + json.dumps( + { + "node_code": args.node_code, + "region": args.region, + "role": args.role, + "status": args.status, + "current_load": max(0, int(args.current_load or 0)), + "iteration": current, + "metadata": metadata, + }, + ensure_ascii=False, + ), + flush=True, + ) + if iterations and current >= iterations: + break + time.sleep(interval) + + +if __name__ == "__main__": + main() diff --git a/domain-api/deploy/multi-region/simulate_multi_region.sh b/domain-api/deploy/multi-region/simulate_multi_region.sh new file mode 100755 index 0000000..ebb4788 --- /dev/null +++ b/domain-api/deploy/multi-region/simulate_multi_region.sh @@ -0,0 +1,65 @@ +#!/usr/bin/env bash +set -euo pipefail + +BASE_URL="${1:-http://127.0.0.1:8100}" +API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}" +SIM_SCRIPT="$API_DIR/deploy/multi-region/simulate_cluster_node.py" +RUNTIME_DIR="$API_DIR/runtime" +mkdir -p "$RUNTIME_DIR" + +CONTROLLER_LOG="$RUNTIME_DIR/sim-mainland-controller.log" +WORKER_LOG="$RUNTIME_DIR/sim-mainland-worker.log" + +cleanup() { + set +e + if [ -n "${CTRL_PID:-}" ]; then kill "$CTRL_PID" >/dev/null 2>&1 || true; fi + if [ -n "${WORKER_PID:-}" ]; then kill "$WORKER_PID" >/dev/null 2>&1 || true; fi + "$PYTHON_BIN" "$SIM_SCRIPT" --node-code mainland-controller-sim --region mainland --role control --clear >/dev/null 2>&1 || true + "$PYTHON_BIN" "$SIM_SCRIPT" --node-code mainland-worker-sim-01 --region mainland --role worker --clear >/dev/null 2>&1 || true +} +trap cleanup EXIT + +echo "[1/4] start simulated mainland controller" +"$PYTHON_BIN" "$SIM_SCRIPT" \ + --node-code mainland-controller-sim \ + --region mainland \ + --role control \ + --status online \ + --detail "模拟大陆 controller 联调节点" \ + --interval 15 >"$CONTROLLER_LOG" 2>&1 & +CTRL_PID=$! + +echo "[2/4] start simulated mainland worker" +"$PYTHON_BIN" "$SIM_SCRIPT" \ + --node-code mainland-worker-sim-01 \ + --region mainland \ + --role worker \ + --status busy \ + --current-load 1 \ + --phase "running" \ + --detail "模拟大陆 worker 执行中" \ + --job-id 1 \ + --job-code "detect-simulated-cluster" \ + --cycle-token "simcycle001" \ + --active-threads 5 \ + --available-proxy-count 12 \ + --interval 15 >"$WORKER_LOG" 2>&1 & +WORKER_PID=$! + +sleep 2 + +echo "[3/4] runtime readiness" +curl -fsS "${BASE_URL}/api/v1/runtime/readiness" +echo +echo + +echo "[4/4] runtime cluster summary" +curl -fsS "${BASE_URL}/api/v1/runtime/cluster" +echo +echo +echo "simulated nodes running:" +echo " controller log: $CONTROLLER_LOG" +echo " worker log: $WORKER_LOG" +echo "press Ctrl+C to stop simulation and clean up" +wait diff --git a/domain-api/deploy/multi-region/templates/domaincheck-api.env.example b/domain-api/deploy/multi-region/templates/domaincheck-api.env.example new file mode 100644 index 0000000..444859e --- /dev/null +++ b/domain-api/deploy/multi-region/templates/domaincheck-api.env.example @@ -0,0 +1,30 @@ +WORKER_MODE=linux-systemd +API_HOST=0.0.0.0 +API_PORT=8100 +DOMAIN_ROOT=__BASE_DIR__/domainCheck +NODE_CODE=overseas-control-01 +NODE_REGION=overseas +NODE_ROLE=control +CORS_ORIGINS=http://127.0.0.1:3201,http://localhost:3201 + +# DB +# DB_HOST=127.0.0.1 +# DB_PORT=5432 +# DB_DATABASE=domain +# DB_USER=postgres +# DB_PASSWORD=postgres + +# REDIS +# REDIS_HOST=127.0.0.1 +# REDIS_PORT=6379 +# REDIS_PASSWORD= +# REDIS_DB=0 + +# SYNC +SYNC_PUSH_ENABLED=false +SYNC_SOURCE_REGION=overseas +SYNC_TARGET_REGION=overseas +SYNC_TARGET_API_BASE_URL= +SYNC_SHARED_TOKEN= +SYNC_BATCH_SIZE=200 +SYNC_POLL_INTERVAL_SECONDS=30 diff --git a/domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example b/domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example new file mode 100644 index 0000000..f0c8107 --- /dev/null +++ b/domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example @@ -0,0 +1,27 @@ +WORKER_MODE=linux-systemd +QT_QPA_PLATFORM=offscreen +NODE_CODE=mainland-controller-01 +NODE_REGION=mainland +NODE_ROLE=control + +# DB +# DB_HOST=127.0.0.1 +# DB_PORT=5432 +# DB_DATABASE=domain +# DB_USER=postgres +# DB_PASSWORD=postgres + +# REDIS +# REDIS_HOST=127.0.0.1 +# REDIS_PORT=6379 +# REDIS_PASSWORD= +# REDIS_DB=0 + +# SYNC +SYNC_PUSH_ENABLED=true +SYNC_SOURCE_REGION=mainland +SYNC_TARGET_REGION=overseas +SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1 +SYNC_SHARED_TOKEN= +SYNC_BATCH_SIZE=200 +SYNC_POLL_INTERVAL_SECONDS=30 diff --git a/domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example b/domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example new file mode 100644 index 0000000..cfe80c2 --- /dev/null +++ b/domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example @@ -0,0 +1,27 @@ +WORKER_MODE=linux-systemd +QT_QPA_PLATFORM=offscreen +NODE_CODE=mainland-worker-01 +NODE_REGION=mainland +NODE_ROLE=worker + +# DB +# DB_HOST=127.0.0.1 +# DB_PORT=5432 +# DB_DATABASE=domain +# DB_USER=postgres +# DB_PASSWORD=postgres + +# REDIS +# REDIS_HOST=127.0.0.1 +# REDIS_PORT=6379 +# REDIS_PASSWORD= +# REDIS_DB=0 + +# SYNC +SYNC_PUSH_ENABLED=true +SYNC_SOURCE_REGION=mainland +SYNC_TARGET_REGION=overseas +SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1 +SYNC_SHARED_TOKEN= +SYNC_BATCH_SIZE=200 +SYNC_POLL_INTERVAL_SECONDS=30 diff --git a/domain-api/deploy/systemd/domain-api.service b/domain-api/deploy/systemd/domain-api.service index 7e7e8a2..50973e5 100644 --- a/domain-api/deploy/systemd/domain-api.service +++ b/domain-api/deploy/systemd/domain-api.service @@ -5,14 +5,17 @@ After=network.target redis.service postgresql.service [Service] Type=simple WorkingDirectory=/opt/domaincheck/domain-api +EnvironmentFile=-/etc/default/domaincheck-api +Environment="PATH=/home/www/.nvm/versions/node/v20.20.2/bin:/home/www/.local/bin:/home/www/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin" Environment="WORKER_MODE=linux-systemd" Environment="API_HOST=0.0.0.0" Environment="API_PORT=8100" +Environment="DOMAIN_ROOT=/opt/domaincheck/domainCheck" ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100 Restart=always RestartSec=5 -User=www-data -Group=www-data +User=www +Group=www [Install] WantedBy=multi-user.target diff --git a/domain-api/deploy/systemd/domain-sync-agent.service b/domain-api/deploy/systemd/domain-sync-agent.service new file mode 100644 index 0000000..666d09f --- /dev/null +++ b/domain-api/deploy/systemd/domain-sync-agent.service @@ -0,0 +1,17 @@ +[Unit] +Description=domainCheck Sync Agent +After=network.target postgresql.service + +[Service] +Type=simple +WorkingDirectory=/opt/domaincheck/domain-api +EnvironmentFile=-/etc/default/domaincheck-worker +Environment="PATH=/home/www/.nvm/versions/node/v20.20.2/bin:/home/www/.local/bin:/home/www/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin" +ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m app.sync_agent +Restart=always +RestartSec=5 +User=www +Group=www + +[Install] +WantedBy=multi-user.target diff --git a/domain-api/deploy/systemd/domain-worker.service b/domain-api/deploy/systemd/domain-worker.service index 8441558..c55c1cd 100644 --- a/domain-api/deploy/systemd/domain-worker.service +++ b/domain-api/deploy/systemd/domain-worker.service @@ -5,12 +5,14 @@ After=network.target redis.service postgresql.service [Service] Type=simple WorkingDirectory=/opt/domaincheck/domainCheck +EnvironmentFile=-/etc/default/domaincheck-worker Environment="WORKER_MODE=linux-systemd" +Environment="QT_QPA_PLATFORM=offscreen" ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python /opt/domaincheck/domainCheck/detect_worker.py Restart=always RestartSec=5 -User=www-data -Group=www-data +User=www +Group=www [Install] WantedBy=multi-user.target diff --git a/domain-web/src/api/modules.ts b/domain-web/src/api/modules.ts index fac5a3c..f00f500 100644 --- a/domain-web/src/api/modules.ts +++ b/domain-web/src/api/modules.ts @@ -10,18 +10,24 @@ export const dashboardApi = { export const runtimeApi = { status: () => http.get("/runtime/status"), + readiness: () => http.get("/runtime/readiness"), preflight: () => http.get("/runtime/preflight"), + cluster: () => http.get("/runtime/cluster"), action: (action: string) => http.post(`/runtime/actions/${action}`) }; export const settingsApi = { getSettings: () => http.get("/settings"), updateSettings: (payload: Record) => http.put("/settings", payload), + getCredentials: () => http.get("/settings/credentials"), + updateCredentials: (payload: Record) => http.put("/settings/credentials", payload), exportSettings: () => http.get("/settings/export"), importSettings: (payload: Record) => http.post("/settings/import", payload), validateImportSettings: (payload: Record) => http.post("/settings/validate-import", payload), backupSettings: () => http.post("/settings/backup"), getSettingsBackups: () => http.get("/settings/backups"), + getJuziseoStatus: () => http.get("/settings/juziseo/status"), + loginJuziseo: (payload: { email: string; password: string }) => http.post("/settings/juziseo/login", payload), backupDownloadUrl: (filename: string) => `${(import.meta.env.VITE_API_BASE_URL || "http://127.0.0.1:8100/api/v1").replace(/\/api\/v1\/?$/, "")}/api/v1/settings/backups/download/${encodeURIComponent(filename)}` }; @@ -30,6 +36,7 @@ export const importsApi = { summary: () => http.get("/imports/summary"), tasks: () => http.get("/imports/tasks"), retry: (taskId: string) => http.post(`/imports/tasks/${taskId}/retry`), + createTextTask: (payload: { text: string; filename?: string }) => http.post("/imports/text", payload), upload: (formData: FormData) => http.post("/imports/upload", formData, { headers: { @@ -40,6 +47,10 @@ export const importsApi = { export const detectApi = { status: () => http.get("/detect/status"), + activeJob: () => http.get("/detect/job/active"), + queueSummary: (params?: Record) => http.get("/detect/queue-summary", { params }), + jobs: (params?: Record) => http.get("/detect/jobs", { params }), + jobDetail: (jobId: number) => http.get(`/detect/jobs/${jobId}`), start: () => http.post("/detect/start"), stop: () => http.post("/detect/stop") }; @@ -47,6 +58,7 @@ export const detectApi = { export const domainsApi = { filters: () => http.get("/domains/filters"), list: (params?: Record) => http.get("/domains", { params }), + detail: (domainId: number) => http.get(`/domains/${domainId}`), batchUpdate: (payload: Record) => http.post("/domains/batch-update", payload) }; @@ -59,3 +71,26 @@ export const logsApi = { latest: () => http.get("/logs/latest"), bundleUrl: () => `${(import.meta.env.VITE_API_BASE_URL || "http://127.0.0.1:8100/api/v1").replace(/\/api\/v1\/?$/, "")}/api/v1/logs/bundle` }; + +export const sensitiveWordsApi = { + get: () => http.get("/sensitive-words"), + save: (payload: { text: string }) => http.put("/sensitive-words", payload) +}; + +export const jumingApi = { + status: () => http.get("/juming/status"), + preferences: () => http.get("/juming/preferences"), + updatePreferences: (payload: Record) => http.put("/juming/preferences", payload), + login: (payload: { email: string; password: string }) => http.post("/juming/login", payload), + loginJucha: () => http.post("/juming/jucha-login"), + crawl: (payload: Record) => http.post("/juming/crawl", payload), + tasks: () => http.get("/juming/tasks"), + retryTask: (taskId: string) => http.post(`/juming/tasks/${taskId}/retry`), + stopTask: (taskId: string) => http.post(`/juming/tasks/${taskId}/stop`), + uploadCookie: (formData: FormData) => + http.post("/juming/cookie/upload", formData, { + headers: { + "Content-Type": "multipart/form-data" + } + }) +}; diff --git a/domain-web/src/components/PageCard.vue b/domain-web/src/components/PageCard.vue index cddabde..919d5c6 100644 --- a/domain-web/src/components/PageCard.vue +++ b/domain-web/src/components/PageCard.vue @@ -27,6 +27,11 @@ defineProps<{ border-radius: 20px; box-shadow: 0 18px 40px rgba(15, 23, 42, 0.05); padding: 22px; + height: 100%; + display: flex; + flex-direction: column; + min-height: 0; + overflow: hidden; } .page-card__header { @@ -35,6 +40,7 @@ defineProps<{ align-items: flex-start; gap: 16px; margin-bottom: 18px; + flex-shrink: 0; } .page-card__header h3 { @@ -46,4 +52,10 @@ defineProps<{ margin: 8px 0 0; color: #64748b; } + +.page-card__content { + flex: 1; + min-height: 0; + overflow: auto; +} diff --git a/domain-web/src/layouts/MainLayout.vue b/domain-web/src/layouts/MainLayout.vue index a901ebe..7d80805 100644 --- a/domain-web/src/layouts/MainLayout.vue +++ b/domain-web/src/layouts/MainLayout.vue @@ -63,6 +63,8 @@ const menuItems = [ { path: "/runtime", label: "运行中心" }, { path: "/settings", label: "系统设置" }, { path: "/imports", label: "域名导入" }, + { path: "/sensitive-words", label: "敏感词配置" }, + { path: "/juming", label: "聚名采集" }, { path: "/detect", label: "检测控制" }, { path: "/domains", label: "域名筛选" }, { path: "/exports", label: "导出中心" }, @@ -119,7 +121,9 @@ onBeforeUnmount(() => { .layout-shell { display: grid; grid-template-columns: 240px 1fr; - min-height: 100vh; + height: 100vh; + height: 100dvh; + overflow: hidden; } .sidebar { @@ -129,6 +133,8 @@ onBeforeUnmount(() => { display: flex; flex-direction: column; gap: 28px; + min-height: 0; + overflow: auto; } .brand { @@ -169,6 +175,8 @@ onBeforeUnmount(() => { display: flex; flex-direction: column; min-width: 0; + min-height: 0; + overflow: hidden; } .topbar { @@ -235,12 +243,16 @@ onBeforeUnmount(() => { } .content { + flex: 1; + min-height: 0; padding: 0 28px 28px; + overflow: hidden; } @media (max-width: 960px) { .layout-shell { grid-template-columns: 1fr; + grid-template-rows: auto 1fr; } .sidebar { diff --git a/domain-web/src/router/index.ts b/domain-web/src/router/index.ts index 77fd42b..f757021 100644 --- a/domain-web/src/router/index.ts +++ b/domain-web/src/router/index.ts @@ -17,6 +17,8 @@ const routes: RouteRecordRaw[] = [ { path: "runtime", name: "runtime", component: () => import("@/views/runtime/RuntimeView.vue"), meta: { title: "运行中心" } }, { path: "settings", name: "settings", component: () => import("@/views/settings/SettingsView.vue"), meta: { title: "系统设置" } }, { path: "imports", name: "imports", component: () => import("@/views/imports/ImportsView.vue"), meta: { title: "域名导入" } }, + { path: "sensitive-words", name: "sensitive-words", component: () => import("@/views/sensitive-words/SensitiveWordsView.vue"), meta: { title: "敏感词配置" } }, + { path: "juming", name: "juming", component: () => import("@/views/juming/JumingView.vue"), meta: { title: "聚名采集" } }, { path: "detect", name: "detect", component: () => import("@/views/detect/DetectView.vue"), meta: { title: "检测控制" } }, { path: "domains", name: "domains", component: () => import("@/views/domains/DomainsView.vue"), meta: { title: "域名筛选" } }, { path: "exports", name: "exports", component: () => import("@/views/exports/ExportsView.vue"), meta: { title: "导出中心" } }, diff --git a/domain-web/src/styles/main.scss b/domain-web/src/styles/main.scss index 0cfa0d4..799f330 100644 --- a/domain-web/src/styles/main.scss +++ b/domain-web/src/styles/main.scss @@ -13,6 +13,7 @@ html, body, #app { margin: 0; + height: 100%; min-height: 100vh; } @@ -21,6 +22,7 @@ body { radial-gradient(circle at top left, rgba(37, 99, 235, 0.08), transparent 28%), radial-gradient(circle at top right, rgba(14, 165, 233, 0.08), transparent 22%), #f4f6fb; + overflow: hidden; } a { diff --git a/domain-web/src/views/detect/DetectView.vue b/domain-web/src/views/detect/DetectView.vue index d7e1b0f..5fc1c96 100644 --- a/domain-web/src/views/detect/DetectView.vue +++ b/domain-web/src/views/detect/DetectView.vue @@ -1,15 +1,17 @@ diff --git a/domain-web/src/views/imports/ImportsView.vue b/domain-web/src/views/imports/ImportsView.vue index dd3e9ed..ea62514 100644 --- a/domain-web/src/views/imports/ImportsView.vue +++ b/domain-web/src/views/imports/ImportsView.vue @@ -1,18 +1,80 @@ + + diff --git a/domain-web/src/views/sensitive-words/SensitiveWordsView.vue b/domain-web/src/views/sensitive-words/SensitiveWordsView.vue new file mode 100644 index 0000000..35e6da8 --- /dev/null +++ b/domain-web/src/views/sensitive-words/SensitiveWordsView.vue @@ -0,0 +1,114 @@ + + + + + diff --git a/domain-web/src/views/settings/SettingsView.vue b/domain-web/src/views/settings/SettingsView.vue index 29a8822..e003190 100644 --- a/domain-web/src/views/settings/SettingsView.vue +++ b/domain-web/src/views/settings/SettingsView.vue @@ -8,8 +8,32 @@
- - + + + + + + + +
+
+ 当前没有单独覆盖配置，所有节点都会走默认线程数。 +
+
+ + + 删除 +
+
+ 新增节点覆盖 +
+
@@ -26,6 +50,10 @@ + + + + @@ -54,6 +82,47 @@ + + + + + + + + + + + + + + + {{ juziseoStatus.cookie_file || "-" }} + + + + 登录桔子SEO + 刷新状态 + + +
+ 聚名登录入口在“聚名采集”页面；桔子SEO 登录入口已按旧桌面版补回到系统设置。 + 当前会同时保存账号和 Cookie，方便 Linux 长驻服务直接复用。 +
+ + +
@@ -111,7 +180,7 @@