admin/getDomain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

first

Browse Source
This commit is contained in:
Your Name
2026-04-16 21:35:47 +08:00
parent ff32aa50bf
commit ebf632e651
86 changed files with 14097 additions and 585 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
.gitignore vendored
View File

@@ -23,6 +23,7 @@ dist/
*.log *.log
logs/ logs/
runtime/ runtime/
diagnostics/
# Build and release artifacts # Build and release artifacts
release/ release/
@@ -38,6 +39,10 @@ tmp/
domains.txt domains.txt
credentials.json credentials.json
# Local deploy config copies
domain-api/deploy/multi-region/*.conf
!domain-api/deploy/multi-region/*.conf.example
# domainCheck local runtime / bundled tools # domainCheck local runtime / bundled tools
domainCheck/tools/node-v20.19.4-win-x64/ domainCheck/tools/node-v20.19.4-win-x64/
domainCheck/app/credentials.json domainCheck/app/credentials.json

227
docs/07_domainCheck_WebLinux发布验收清单.md
View File

@@ -1,90 +1,231 @@
# 07 domainCheck Web/Linux 发布验收清单 # 07 domainCheck Web/Linux 发布验收清单
## 一、基础连通 ## 一、文档定位
- `domain-api` 已启动 本文档用于回答两个问题：
- `domainCheck Worker` 已启动
- `http://服务器IP:8100/health` 返回 `status=ok`
- `runtime/preflight` 返回 `ok=true`
## 二、Web 后台 1. 当前这套 Web/Linux 方案，到底哪些项已经验过
2. 发布前还需要按什么维度再核一遍
阅读建议：
- 想看测试服真实联调结果，优先看 `docs/13_domainCheck_Linux测试服交接文档.md`
- 想看正式上线前最后一次操作顺序，优先看 `docs/14_domainCheck_正式上线前最终检查单.md`
本文档更适合作为“发布验收维度总表”。
## 二、基础连通验收
以下项目应视为发布验收的第一层：
- `domaincheck-api` 已启动
- `domaincheck-worker` 已启动
- `http://127.0.0.1:8100/health` 返回 `status=ok`
- `/health` 返回 `worker_mode=linux-systemd`
- `/api/v1/runtime/preflight` 返回 `ok=true`
- `/api/v1/runtime/status` 返回 `worker.running=true`
当前测试服状态：
- 已验证通过
## 三、Web 后台验收
### 1. 运行与展示
- 登录正常 - 登录正常
- 顶部可看到 `API / Worker / 运行模式` - 顶部状态可展示 `API / Worker / 运行模式`
- 概览页能正常读取统计 - 概览页可正常读取统计
- 运行中心能正常读取： - 运行中心可正常读取：
- API 版本 - API 版本
- API 前缀 - API 前缀
- PID - PID
- Worker 进程数 - Worker 进程数
- 自检结果 - 自检结果
## 三、配置能力 ### 2. 配置能力
- 系统设置能正常读取 - 系统设置可正常读取
- 线程数修改后可保存 - 线程数可修改并保存
- 检测顺序可调整并保存 - 检测顺序可调整并保存
- 代理池列表可编辑并保存 - 代理池列表可编辑并保存
- `worker_mode / service_name` 可保存 - `worker_mode / worker_service_name / api_service_name` 可保存
- 可手动创建配置备份 - 可手动创建配置备份
- 可下载配置备份 - 可下载配置备份
- 可导出配置快照 - 可导出配置快照
- 可导入配置快照 - 可导入配置快照
- 导入配置时会自动生成导入前备份 - 导入配置时会自动生成导入前备份
## 四、业务能力 当前测试服状态：
- 核心接口与配置链路已验证通过
## 四、业务能力验收
发布前建议至少覆盖下面 5 类能力：
- 导入
- 列表筛选
- 批量更新
- 导出
- 日志与诊断
### 1. 导入链路
- 导入任务可创建 - 导入任务可创建
- 导入任务列表可刷新 - 导入任务列表可刷新
- 导入任务失败时可重试 - 导入结果统计正确
- 域名筛选可查询 - 导入成功后会写入 `domains`
- 批量更新可执行 - 导入成功后会自动创建 `detect_tasks`
- 导出记录可生成
- 导出文件可下载
- 日志诊断页可查看日志
- 诊断包可下载
## 五、Linux 特有项 当前测试服状态：
- 已使用样本 TXT 实际验证通过
- 当前已确认：
- 总数 `5`
- 有效 `3`
- 新增 `3`
- 无效 `2`
### 2. 列表与筛选
- 域名列表可查询
- 筛选项接口正常返回
- 页面结果与数据库记录一致
当前测试服状态：
- 已验证通过
### 3. 批量更新
- 批量更新接口可执行
- 页面展示与数据库字段一致
- 关联检测字段会同步更新
当前测试服状态：
- 已验证通过
- 已确认 `backlink_count_gt_10` 可随批量更新同步生效
### 4. 导出链路
- 导出任务可创建
- 导出记录可查看
- 导出文件可下载
- TXT / CSV 至少一种格式已回归
当前测试服状态：
- TXT、CSV 均已验证通过
### 5. 日志与诊断
- 日志接口可正常读取
- 诊断包可正常导出
当前测试服状态：
- 已验证通过
- 当前已生成正式服务态诊断包：
- `/opt/domaincheck/diagnostics/diag_20260416_134923.tar.gz`
## 五、Linux 特有项验收
以下项目是 Web/Linux 交付里最容易在正式环境出问题的部分：
- `domaincheck-api.service` 可正常启动/停止 - `domaincheck-api.service` 可正常启动/停止
- `domaincheck-worker.service` 可正常启动/停止 - `domaincheck-worker.service` 可正常启动/停止
- `journalctl` 可查看两边日志 - `journalctl` 可查看两边日志
- Nginx 反代正常 - 运行中心可调用 `start_worker / stop_worker / restart_api`
- Web 静态文件 `dist` 已正确发布 - Worker 以 `QT_QPA_PLATFORM=offscreen` 正常运行
- API 自重启不会再因为同步等待自身停机而误报 `500`
## 六、建议发布前命令 当前测试服状态：
### 1. 运行 API 自测 - 已验证通过
## 六、数据库与权限验收
Linux 新环境发布前，下面两项必须显式确认：
### 1. 数据库初始化
如果 PostgreSQL 使用的是新库，必须先执行：
```bash ```bash
cd /opt/domaincheck/domain-api cd /opt/domaincheck/domainCheck
python deploy/linux/smoke_test.py --base-url http://127.0.0.1:8100 python3 init_database.py
```
否则至少这些接口会直接失败：
- `/api/v1/dashboard/overview`
- `/api/v1/detect/status`
- `/api/v1/imports/summary`
### 2. 服务用户写权限
正式 `systemd` 服务用户必须可写：
- `domain-api/runtime/`
- `domainCheck/detect_worker.log`
否则可能出现：
- `/api/v1/imports/upload` 返回 `500`
- Worker 循环重启
当前测试服状态：
- 两项都已实际踩坑并修复
## 七、发布前建议命令
### 1. 健康检查
```bash
curl http://127.0.0.1:8100/health
curl http://127.0.0.1:8100/api/v1/runtime/preflight
curl http://127.0.0.1:8100/api/v1/runtime/status
```
### 2. systemd 状态
```bash
systemctl status domaincheck-api --no-pager -l
systemctl status domaincheck-worker --no-pager -l
```
### 3. smoke test
```bash
cd /opt/domaincheck/domain-api/deploy/linux
python3 smoke_test.py --base-url http://127.0.0.1:8100
``` ```
如需同时校验 Web 首页： 如需同时校验 Web 首页：
```bash ```bash
python deploy/linux/smoke_test.py --base-url http://127.0.0.1:8100 --web-url http://127.0.0.1 python3 smoke_test.py --base-url http://127.0.0.1:8100 --web-url http://127.0.0.1
``` ```
### 2. 查看 API 健康状态 ### 4. 诊断包导出
```bash ```bash
curl http://127.0.0.1:8100/health cd /opt/domaincheck/domain-api/deploy/linux
curl http://127.0.0.1:8100/api/v1/runtime/preflight bash collect_diagnostics.sh /opt/domaincheck
``` ```
### 3. 查看 systemd 状态 ## 八、当前发布验收结论
```bash 结合当前 Linux 测试服已经完成的联调结果，可以给出下面的结论：
systemctl status domaincheck-api
systemctl status domaincheck-worker
```
## 七、当前结论
如果以上检查项全部通过，则可以认为：
- Web 管理后台已达到可交付状态 - Web 管理后台已达到可交付状态
- Linux 部署环境已达到可联调状态 - Linux 正式 `systemd` 服务态已验证通过
- 可以进入真实服务器联调或灰度上线阶段 - 核心业务链路已完成最小闭环回归
- 当前剩余工作主要是正式环境发布、灰度观察和持续稳定性观察
一句话结论：
> 当前项目已经通过 Web/Linux 发布所需的核心验收项，后续重点不再是功能开发，而是正式环境收口与上线后观察。

15
docs/10_domainCheck_最终交付结论.md
View File

@@ -55,13 +55,20 @@
- 上线前灰度发布 - 上线前灰度发布
- 发布后观察期 - 发布后观察期
补充说明：
- Linux 测试服基础联调已完成
- 正式 `systemd` 服务态已验证通过
- 当前进入的是“正式上线前最后检查与观察”阶段
## 五、建议下一步 ## 五、建议下一步
1. 把最新交付包发送到目标 Linux 服务器 1. 把最新交付包发送到目标 Linux 服务器
2. 按 `docs/05`、`docs/07`、`docs/08` 的顺序执行部署与验收 2. 按 `docs/05`、`docs/13`、`docs/14` 的顺序执行部署、核对与收口
3. 部署完成后再做一次真实环境 smoke test 3. 部署完成后再做一次正式服务态 `smoke test`
4. 进入灰度上线 4. 导出一份最终诊断包
5. 进入灰度上线
## 六、最终判断 ## 六、最终判断
当前这套项目，已经达到“工程交付完成，待真实环境上线联调”的状态。 当前这套项目，已经达到“工程交付完成，Linux 测试服联调闭环，待正式环境上线收口”的状态。

23
docs/12_domainCheck_导航索引.md
View File

@@ -20,6 +20,22 @@
- `docs/13_domainCheck_Linux测试服交接文档.md` - `docs/13_domainCheck_Linux测试服交接文档.md`
### 5. 正式上线前最终检查单
- `docs/14_domainCheck_正式上线前最终检查单.md`
### 6. 多机检测与跨地域部署设计
- `docs/16_domainCheck_多机检测与跨地域部署设计.md`
### 7. 全流程部署实操手册
- `docs/17_domainCheck_全流程部署实操手册.md`
### 8. CentOS9 一键复制部署与更新文档
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
## 二、文档阅读顺序 ## 二、文档阅读顺序
### 1. 先看总体方案 ### 1. 先看总体方案
@@ -33,6 +49,10 @@
- `docs/07_domainCheck_WebLinux发布验收清单.md` - `docs/07_domainCheck_WebLinux发布验收清单.md`
- `docs/09_domainCheck_交付打包说明.md` - `docs/09_domainCheck_交付打包说明.md`
- `docs/13_domainCheck_Linux测试服交接文档.md` - `docs/13_domainCheck_Linux测试服交接文档.md`
- `docs/14_domainCheck_正式上线前最终检查单.md`
- `docs/16_domainCheck_多机检测与跨地域部署设计.md`
- `docs/17_domainCheck_全流程部署实操手册.md`
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
### 3. 如果要回溯历史需求与问题 ### 3. 如果要回溯历史需求与问题
@@ -63,6 +83,9 @@
- `domain-api/deploy/systemd/domain-api.service` - `domain-api/deploy/systemd/domain-api.service`
- `domain-api/deploy/systemd/domain-worker.service` - `domain-api/deploy/systemd/domain-worker.service`
- `domain-api/deploy/linux/README.md` - `domain-api/deploy/linux/README.md`
- `domain-api/deploy/multi-region/README.md`
- `domain-api/deploy/multi-region/bootstrap_overseas.sh`
- `domain-api/deploy/multi-region/bootstrap_mainland.sh`
- `domain-api/deploy/linux/smoke_test.py` - `domain-api/deploy/linux/smoke_test.py`
- `domain-api/deploy/linux/collect_diagnostics.sh` - `domain-api/deploy/linux/collect_diagnostics.sh`

215
docs/13_domainCheck_Linux测试服交接文档.md
View File

@@ -23,17 +23,21 @@
- 交付包、验包、SHA256、最新交付指针已完成 - 交付包、验包、SHA256、最新交付指针已完成
- Linux 测试服数据库初始化问题已定位并修复 - Linux 测试服数据库初始化问题已定位并修复
- Linux 测试服在“临时 API 进程 + 已初始化数据库”的模式下，`smoke test` 已通过 - Linux 测试服在“临时 API 进程 + 已初始化数据库”的模式下，`smoke test` 已通过
- Linux 测试服已经完成正式 `systemd` 服务化联调，`domaincheck-api` 与 `domaincheck-worker` 均已拉起
- 正式服务态下 `/health` 已确认 `worker_mode = linux-systemd`
- 正式服务态 `smoke test` 已通过
- 已补做一轮真实导入回归，确认“导入域名 -> 写入 `domains` -> 自动创建 `detect_tasks`”链路正常
### 2. 未完全完成 ### 2. 未完全完成
- Linux 测试服当前还不是最终正式部署形态 - Linux 测试服虽然已完成正式服务化联调，但仍属于“测试服验证通过”，不等于生产观察期已经完成
- 这次通过的是“临时启动 API 进程”的验证，不是正式 `systemd` 服务托管态 - Worker 当前通过 `QT_QPA_PLATFORM=offscreen` 运行，属于“无头 Qt 托管态”，后续仍建议继续观察稳定性
- `worker_mode` 当前仍表现为 `windows-local` - 真实业务网络环境下的长时检测、代理池质量、Wayback 首次全量列表耗时，还需要继续压测和观察
- `domaincheck-api` / `domaincheck-worker` 还没有按正式生产口径落成 Linux `systemd` 服务闭环 - 当前数据库里仅导入了少量回归样本，不代表真实大批量数据已完成验收
一句话结论： 一句话结论：
> 功能和数据库问题已经打通；正式 Linux 上线态还差最后一段服务化收口。 > 功能、数据库和正式服务化都已经打通；后续工作转入真实业务回归、稳定性观察和上线前优化。
## 三、这次 Linux 测试服已验证通过的内容 ## 三、这次 Linux 测试服已验证通过的内容
@@ -79,7 +83,7 @@ python init_database.py
- 库结构已正常 - 库结构已正常
- 只是当前还没有正式业务数据导入 - 只是当前还没有正式业务数据导入
### 3. smoke test 已通过 ### 3. 临时验证态 smoke test 已通过
新版 `smoke test` 结果： 新版 `smoke test` 结果：
@@ -96,7 +100,7 @@ python init_database.py
- Redis 连接可用 - Redis 连接可用
- 缺表问题已解除 - 缺表问题已解除
### 4. 诊断包已导出 ### 4. 临时验证态诊断包已导出
本次测试服诊断产物： 本次测试服诊断产物：
@@ -105,36 +109,95 @@ python init_database.py
可用于后续继续排障或归档。 可用于后续继续排障或归档。
## 四、当前测试服不是正式上线态的原因 ## 四、正式服务态已补验证通过
虽然 `smoke test` 已通过，但当前仍不是正式上线态，原因如下： 在后续继续收口过程中，已经额外完成了正式 `systemd` 服务态验证，结果如下：
### 1. 当前通过的是临时 API 进程验证 ### 1. 正式服务已落地
本次验证是通过“当前工作区里手动启动的 API 进程”完成的，不是通过正式服务方式完成的。 已安装并启用：
### 2. worker_mode 仍不是 Linux 正式模式
当前表现仍是：
- `worker_mode = windows-local`
这意味着：
- 运行中心、配置项、控制逻辑还没有真正切到 Linux 正式托管模式
- 当前仍属于“测试验证态”
### 3. systemd 服务还未正式落地闭环
还没有最终确认以下两项处于正式可用状态：
- `domaincheck-api` - `domaincheck-api`
- `domaincheck-worker` - `domaincheck-worker`
也就是说： 并已确认两者为 `active (running)`。
- 现在能证明代码跑得通 ### 2. 运行模式已切换成功
- 但还没有证明“服务化部署后也稳定可用”
正式服务态下接口返回已确认：
- `/health` 中 `worker_mode = linux-systemd`
- `/api/v1/runtime/preflight` 中 `worker_mode = linux-systemd`
- `/api/v1/runtime/status` 中：
- `api.service_name = domaincheck-api`
- `worker.service_name = domaincheck-worker`
- `worker.running = true`
- `process_count = 1`
### 3. 正式服务态 smoke test 已通过
正式服务态下再次执行：
```bash
python smoke_test.py --base-url http://127.0.0.1:8100
```
结果仍为：
```json
{
"ok": true
}
```
### 4. 正式服务态诊断包已导出
正式服务态诊断产物：
- 目录：`/opt/domaincheck/diagnostics/diag_20260416_134923`
- 压缩包：`/opt/domaincheck/diagnostics/diag_20260416_134923.tar.gz`
### 5. 正式服务态中已确认的兼容与权限问题
本轮继续联调还确认并处理了两类真实 Linux 问题：
- `detect/jucha.py`、`detect/juming.py` 中存在 Windows 专属 `subprocess.STARTUPINFO()` 写法，已改为跨平台兼容处理
- 若曾以 `root` 手工运行过 API/Worker，可能会留下 `root` 所有者的运行文件，导致正式 `systemd` 服务用户无法写入
其中已实际踩到并修复的权限点包括：
- `domain-api/runtime/` 目录权限
- `domainCheck/detect_worker.log` 文件权限
这个问题的表现是：
- `/api/v1/imports/upload` 因无法创建 `runtime/imports/` 返回 `500`
- Worker 因无法写 `detect_worker.log` 进入循环重启
### 6. 运行中心控制链路已补通
本轮继续验证后，运行中心里的 Linux 控制动作也已经打通：
- Worker `start` / `stop` 已可通过 `sudo -n systemctl` 正常执行
- API 自重启已改为 `systemctl --no-block restart domaincheck-api`
这样处理后，`systemd` 会异步接管重启流程，接口可以先返回成功，避免出现“服务其实已经重启成功，但调用方因为等待自身停机而收到 `500`”的误导性现象。
### 7. 服务态日志噪音已进一步收敛
本轮还额外处理了两类不会阻断功能、但会影响正式服务观察体验的日志噪音：
- Redis 配置订阅从阻塞 `listen()` 改为短轮询 `get_message()`，Linux 空闲时不再每分钟刷 `Redis订阅失败: Timeout reading from socket`
- `offscreen` 模式下去掉 Qt 不支持的按钮样式属性，Worker 启动时不再刷 `Unknown property transition/transform/box-shadow`
- Worker 图标资源定位改为优先使用 `detect_worker.py` 同目录，Linux 服务态不再误报 `/opt/new_logo.svg`、`/opt/favicon2.ico` 不存在
- 付费检测器改为按需初始化，默认关闭 `detect_jucha` / `detect_juziseo` 时，不再在启动阶段报 cookie 文件缺失
- Redis 未加载 Bloom 模块时保留为降级说明，继续使用普通缓存，不再作为故障级告警处理
这几项修改的结果是：
- `domaincheck-worker` 仍保持 `active (running)`
- `/api/v1/runtime/status` 中 `worker.running = true`
- 正式测试服日志更适合持续观察和上线前留档
## 五、这次联调后已经明确固化的部署规则 ## 五、这次联调后已经明确固化的部署规则
@@ -169,6 +232,7 @@ python init_database.py
- `domain-web` 主要页面已完成 - `domain-web` 主要页面已完成
- `domain-api` 主要接口已完成 - `domain-api` 主要接口已完成
- 运行中心、系统设置、导入、导出、日志诊断、自检、自测均已具备 - 运行中心、系统设置、导入、导出、日志诊断、自检、自测均已具备
- 正式 `systemd` 服务态下 `/health`、`/runtime/preflight`、`/runtime/status`、`smoke test` 已全部通过
### 3. 交付层 ### 3. 交付层
@@ -185,56 +249,23 @@ python init_database.py
- Linux 联调输入清单已完成 - Linux 联调输入清单已完成
- 导航索引文档已完成 - 导航索引文档已完成
### 5. 真实业务回归层
- 已通过 API 上传样本 TXT
- 已确认导入结果：
- 总数 `5`
- 有效 `3`
- 新增 `3`
- 无效 `2`
- 已确认 `domains_total = 3`
- 已确认 `detect_tasks_total = 3`
- 已确认导入后会自动创建 `detect_tasks`
## 七、当前未完成项清单 ## 七、当前未完成项清单
以下内容仍属于“后续要做”： 以下内容仍属于“后续要做”：
### 1. 正式 Linux 目录落地 ### 1. Worker 真实联动验证
需要确认正式部署目录结构为：
```text
/opt/domaincheck
├── domainCheck
├── domain-api
├── domain-web
```
如果测试服当前目录不是这一套，需要统一。
### 2. 正式 systemd 服务化
需要把以下服务真正落好并验证：
- `domaincheck-api`
- `domaincheck-worker`
至少要完成：
- 安装 service 文件
- `daemon-reload`
- `enable`
- `start`
- `status`
- `journalctl`
### 3. worker_mode 切换为 linux-systemd
需要确认：
- Web 系统设置中运行模式改为 `linux-systemd`
- API `/health` 或运行中心能正确反映：
- `worker_mode = linux-systemd`
### 4. 正式服务态再跑一次 smoke test
不是临时 API 进程跑通就结束，还要在正式服务态下再跑一次：
```bash
python smoke_test.py --base-url http://127.0.0.1:8100
```
### 5. Worker 真实联动验证
还应继续确认： 还应继续确认：
@@ -243,17 +274,26 @@ python smoke_test.py --base-url http://127.0.0.1:8100
- 检测控制页读取是否正常 - 检测控制页读取是否正常
- `detect_worker.log` 是否正常写入 - `detect_worker.log` 是否正常写入
### 6. 导入真实数据后的业务复测 ### 2. 导入更多真实数据后的业务复测
当前 `domains_count = 0`，说明库表正常，但业务数据还没开始导入。 当前已经完成一轮小样本回归，不再是空库空表态。
后续应至少补一次真实业务复测： 后续应至少补一次真实业务复测：
- 导入域名 - 导入更接近真实业务规模的域名样本
- 查看 `domains` 增长 - 查看 `domains` 持续增长
- 查看 `detect_tasks` 生成 - 查看 `detect_tasks` 持续生成
- 再验证筛选与导出 - 再验证筛选与导出
### 3. 长时间运行与代理池观察
仍建议继续验证：
- Worker 长时运行稳定性
- Redis 订阅超时后的重连是否持续稳定
- 国内网络环境下代理池真实可用率
- Wayback 首次全量快照列表的耗时表现
## 八、后续继续收口的推荐顺序 ## 八、后续继续收口的推荐顺序
建议 Linux 上的下一位接手人严格按下面顺序执行。 建议 Linux 上的下一位接手人严格按下面顺序执行。
@@ -291,6 +331,11 @@ order by tablename;
- `domain-api/deploy/systemd/domain-api.service` - `domain-api/deploy/systemd/domain-api.service`
- `domain-api/deploy/systemd/domain-worker.service` - `domain-api/deploy/systemd/domain-worker.service`
如果此前用 `root` 手工跑过 API 或 Worker，建议先确认下面这些路径对正式服务用户可写：
- `domain-api/runtime/`
- `domainCheck/detect_worker.log`
### 第四步：启动正式服务 ### 第四步：启动正式服务
```bash ```bash
@@ -318,6 +363,16 @@ curl http://127.0.0.1:8100/api/v1/runtime/preflight
curl http://127.0.0.1:8100/api/v1/runtime/status curl http://127.0.0.1:8100/api/v1/runtime/status
``` ```
### 第七步：做一轮导入回归
至少验证：
- `/api/v1/imports/upload`
- `/api/v1/imports/tasks`
- `/api/v1/imports/summary`
- `domains` 增长
- `detect_tasks` 自动创建
目标是确认： 目标是确认：
- `status = ok` - `status = ok`

244
docs/14_domainCheck_正式上线前最终检查单.md Normal file
View File

@@ -0,0 +1,244 @@
# 14 domainCheck 正式上线前最终检查单
## 一、使用场景
本文档用于正式上线前最后一次收口。
适用前提：
- 代码已更新到最新版本
- Linux 测试服已完成基础联调
- PostgreSQL 已执行过 `domainCheck/init_database.py`
- `domaincheck-api` 与 `domaincheck-worker` 已按 `systemd` 托管
如果当前仍处于“新机器首次部署”，请先回看：
- `docs/05_domainCheck_Linux部署清单.md`
- `domain-api/deploy/linux/README.md`
- `docs/13_domainCheck_Linux测试服交接文档.md`
如果后续进入“国外控制面 + 大陆执行面 + 多 Worker 扩容”阶段，请直接补充阅读：
- `docs/16_domainCheck_多机检测与跨地域部署设计.md`
- `domain-api/deploy/multi-region/README.md`
如果当前已经进入“大陆 controller + 海外 control”双地域正式联调，还应额外确认：
- 大陆 `controller` 节点的 `/etc/default/domaincheck-worker`
- `NODE_ROLE=control`
- 大陆 `domaincheck-sync-agent`
- 已启动并稳定运行
- 海外控制面 `runtime/sync-summary`
- 能看到 `detect_result_batches`
## 二、上线前必须确认的结论
上线前至少要确认下面这些结论同时成立：
- `domaincheck-api` 为 `active (running)`
- `domaincheck-worker` 为 `active (running)`
- `/health` 返回 `worker_mode=linux-systemd`
- `/api/v1/runtime/preflight` 返回 `ok=true`
- `smoke test` 返回 `ok=true`
- 运行中心里的 `start_worker / stop_worker / restart_api` 控制链路可用
- 导入、筛选、批量更新、导出四条核心业务链路至少各回归一次
- 诊断包可正常导出
## 三、正式上线前执行顺序
### 1. 核对服务状态
```bash
systemctl status domaincheck-api --no-pager -l
systemctl status domaincheck-worker --no-pager -l
```
期望：
- 两个服务都为 `active (running)`
- `domaincheck-worker` 不再循环重启
如果当前是大陆 controller 节点，还要补一条：
```bash
systemctl status domaincheck-sync-agent --no-pager -l
```
期望：
- `domaincheck-sync-agent` 为 `active (running)`
- 不循环重启
### 2. 核对健康接口
```bash
curl http://127.0.0.1:8100/health
curl http://127.0.0.1:8100/api/v1/runtime/preflight
curl http://127.0.0.1:8100/api/v1/runtime/readiness
curl http://127.0.0.1:8100/api/v1/runtime/status
```
期望：
- `/health` 返回 `status=ok`
- `/health` 返回 `worker_mode=linux-systemd`
- `/runtime/preflight` 返回 `ok=true`
- `/runtime/readiness` 不应返回 `blocking`
- `/runtime/status` 返回 `worker.running=true`
如果当前已经接入跨地域同步，还建议补看：
```bash
curl http://127.0.0.1:8100/api/v1/runtime/sync-summary
```
期望：
- 返回里能看到 `detect_result_batches`
- 至少能区分：
- `synced`
- `projected`
- `failed`
- 若大陆 `sync-agent` 已接上，则最近执行过的检测任务不应长期停留在 `projected`
当前 `runtime/preflight` 还会额外展示：
- RedisBloom 是否安装，若未安装会明确提示“降级为普通缓存”
- `detect_jucha` / `detect_juziseo` 是否启用
- 若启用了付费检测，对应本地 cookie 文件是否已就绪
当前 `detect/status` / `runtime/status` 还会明确展示代理运行态：
- `代理正常`
- `降级直连`
- `等待代理`
解释：
- `降级直连` 表示代理池暂时无可用代理，但当前允许直连兜底，任务不会因此中断
- `等待代理` 表示代理池无可用代理，且当前未允许直连，这会直接影响检测吞吐或导致步骤失败
当前还补充了两类更细的诊断语义：
- `proxy_runtime_reason=supplier_empty_pool`
- 表示代理源最近都返回了正常 HTTP 响应，但原始代理数为 `0`
- 这类问题更偏向供应侧空池，不是程序拉取失败
- `dependency_alerts`
- 表示外部依赖站点当前存在可观测异常
- 例如 `web.archive.org` 拒连、超时、连接池异常等
当前免费检测链路中的外部依赖异常还采用了“降级继续”策略：
- 例如 `时光机 / 站长之家 / 爱站 / 百度 / 360` 这类外部站点出现明显网络波动、拒连、超时
- 系统会优先记录为步骤级 `degraded`
- 并将域名置为 `待人工复核`
- 同时继续执行后续检测步骤，避免把整条任务链路直接污染成 `检测失败`
### 3. 跑正式服务态 smoke test
```bash
cd /opt/domaincheck/domain-api/deploy/linux
python3 smoke_test.py --base-url http://127.0.0.1:8100
```
如需把 Web 首页一起纳入检查：
```bash
python3 smoke_test.py --base-url http://127.0.0.1:8100 --web-url http://127.0.0.1
```
期望：
- 输出中 `ok=true`
### 4. 回归运行中心控制动作
建议至少各执行一次：
- `start_worker`
- `stop_worker`
- `restart_api`
期望：
- `stop_worker` 后，`runtime/status` 中 `worker.running=false`
- `start_worker` 后，`runtime/status` 中 `worker.running=true`
- `restart_api` 调用返回成功，随后 API 能重新恢复
说明：
- 当前 Linux 正式服务态已改为 `systemctl --no-block restart domaincheck-api`
- 因此 `restart_api` 会先返回成功，再由 `systemd` 异步完成 API 切换
### 5. 回归核心业务链路
建议至少确认下面 4 条：
- 导入域名成功，且会落到 `domains`
- 导入后自动创建 `detect_tasks`
- 筛选接口 `/api/v1/domains` 返回正常
- 批量更新后数据库字段与页面展示一致
- 导出任务可创建、导出文件可下载
如果时间紧，最小回归顺序建议为：
1. 导入一份小样本
2. 查看导入任务结果
3. 查看域名列表
4. 执行一次批量更新
5. 生成一次 TXT 或 CSV 导出
## 四、建议保留的上线证据
正式上线前，建议至少保留下面这些结果：
- `systemctl status domaincheck-api --no-pager -l`
- `systemctl status domaincheck-worker --no-pager -l`
- `journalctl -u domaincheck-api -n 200 --no-pager`
- `journalctl -u domaincheck-worker -n 200 --no-pager`
- `systemctl status domaincheck-sync-agent --no-pager -l`
- `journalctl -u domaincheck-sync-agent -n 200 --no-pager`
- `/health` 返回
- `/api/v1/runtime/preflight` 返回
- `/api/v1/runtime/sync-summary` 返回
- `smoke_test.py` 输出
- 一份最新诊断包
诊断包导出命令：
```bash
cd /opt/domaincheck/domain-api/deploy/linux
bash collect_diagnostics.sh /opt/domaincheck
```
## 五、当前可接受的非阻断项
以下项目目前已确认不会阻断正式服务运行：
- Redis 未安装 Bloom 模块时，会降级为普通缓存
- `worker` 在 `QT_QPA_PLATFORM=offscreen` 下无头运行
- 付费检测默认关闭时，不要求本机预置 `jucha` / `juziseo` cookie 文件
这些项可以在后续优化阶段继续增强，但不应再作为当前上线阻塞条件。
## 六、仍建议继续观察的项
以下内容不属于当前功能阻断，但上线后建议继续观察：
- 真实代理池可用率
- 代理为空时是否长期停留在 `降级直连`
- Wayback 首次全量检测耗时
- 长时间运行下的 Worker 稳定性
- 真实业务数据量上升后的数据库与导出耗时
## 七、最终上线判定
如果下面三类检查都通过，就可以视为已经具备正式上线条件：
- 服务状态检查通过
- 核心接口与 `smoke test` 通过
- 核心业务链路回归通过
一句话判定：
> 当前项目已经完成开发、Linux 测试服联调、正式服务态验证和关键日志降噪；剩余工作主要是正式环境发布与上线后观察，不再是功能性大修。

59
docs/15_domainCheck_Web对齐旧桌面功能核对表.md Normal file
View File

@@ -0,0 +1,59 @@
# domainCheck Web 对齐旧桌面功能核对表
更新时间：2026-04-16
## 一、旧桌面主标签页对齐
| 旧桌面标签 | Web 当前状态 | 说明 |
| --- | --- | --- |
| 聚名爬取 | 已补齐 | 已支持聚名账号密码登录、聚查自动联名、过期删除/一口价采集、Cookie 兜底上传 |
| 域名筛选 | 已补齐主干 | 已补“全部”默认选项，支持查询、分页、批量更新、筛选页导出范围，并继续补回旧桌面明细列 |
| 域名导入 | 已补齐主干 | 已支持 TXT 上传导入，也已补回手工粘贴域名创建导入任务 |
| 敏感词配置 | 已补齐 | 本轮新增独立页面，支持加载、编辑、导入、导出、保存 |
| 系统设置 | 已补齐主干 | 已补桔子SEO 登录入口；聚名登录入口放在“聚名采集”页 |
## 二、旧桌面登录能力对齐
| 能力 | Web 当前状态 | 说明 |
| --- | --- | --- |
| 聚名账号密码登录 | 已补齐 | 位于 `#/juming` |
| 聚查联名登录 | 已补齐 | 聚名登录后自动执行，同时提供手动“重新联名登录聚查” |
| 桔子SEO 登录 | 已补齐 | 位于 `#/settings` 的“第三方登录”卡片 |
## 三、已确认补齐的体验项
- 域名筛选下拉框显式提供“全部”
- 桔子SEO 不再遗漏登录入口
- 敏感词配置不再缺页
- 聚查不只依赖自动联名，也可手动重试
- 域名导入已补回“粘贴域名后直接导入”
- 筛选页已补回“当前页 / 导出几页 / 全部”的导出范围能力
- 筛选页已补回旧桌面常用明细列：过期时间、单位性质、百度历史、百度Site、中文标题、360 Site、Google Site
- 批量更新已补回上述检测明细字段的人工修正入口
- CSV / Excel 导出已补回旧桌面常用明细列，避免导出结果比桌面版缩水
- 聚名页面已显式展示“已保存账号 / 聚名登录态 / 聚查联名状态”，避免运营误判
- 系统设置页已显式展示“已保存桔子SEO账号 / Cookie 就绪状态”，并同步持久化到本地与 Redis
- 聚名采集、域名导入、检测控制三条长任务链路均已统一为“任务中心”形态，支持阶段态、日志常驻、切页后回看
- 检测控制页已补齐更细的阶段态展示，可识别“准备检测 / 刷新代理池 / 取任务中 / 建线程中 / 检测中 / 批次完成 / 完成归档”
- 检测控制页已补齐阶段切换时间线，便于回看长任务在何时进入哪个阶段
- 聚名采集页已补齐中文状态标签、当前采集阶段与任务概览卡片，避免运营面对英文状态误判
- 域名导入页已补齐“选中文件预估行数”和“当前导入阶段/任务概览”摘要，更接近旧桌面即时反馈体验
- 域名导入页已补齐“前 10 行本地预览”和“大文件任务提示”，降低误传错文件的概率
- 域名导入页已补齐提交前本地预检，可即时展示“有效 / 重复 / 非法”统计
- 域名导入页已补齐“来源类型”选择与任务展示，导入记录可区分 TXT 导入 / 手工录入 / 其它
- 域名筛选页已补齐“单位性质”筛选，进一步对齐旧桌面常用条件
- 域名筛选页已补齐“来源类型”筛选；导入页完成后可一键跳转到筛选页查看对应来源结果
- 聚名采集、域名导入、检测控制三页的任务中心文案已统一，均按“阶段 / 摘要 / 日志 / 回看”同一口径表达
## 四、下一轮继续核对的次级项
- 旧桌面导入页的少量细节交互，是否还需要补更细的批次说明
- 旧桌面检测控制页的少量状态提示文案，是否还需要继续压词统一
- 旧桌面批量筛选中的少量边缘筛选项，是否还有遗漏
## 五、当前可直接验证的页面
- `http://152.53.37.118:3201/#/juming`
- `http://152.53.37.118:3201/#/domains`
- `http://152.53.37.118:3201/#/settings`
- `http://152.53.37.118:3201/#/sensitive-words`

796
docs/16_domainCheck_多机检测与跨地域部署设计.md Normal file
View File

@@ -0,0 +1,796 @@
# 16 domainCheck 多机检测与跨地域部署设计
## 一、目标
本文档用于固化 `domainCheck` 的正式多机落地方案，满足下面这些约束：
- Web 后台、管理 API 必须部署在国外机器
- 检测执行、代理、Redis、运行态存储部署在大陆机器
- 最终结果、配置、归档、备份以国外机器为主
- 初期部署不复杂，支持从 `国外 1 台 + 大陆 1 台` 起步
- 后期不够时，优先横向增加大陆 Worker，不推翻整体架构
一句话原则：
> 国外负责“管、看、存、备份”，大陆负责“跑、调度、抢任务、贴近代理资源”。
## 二、当前落地状态
截至 `2026-04-16`，第一阶段多机运行骨架已经完成，当前代码已具备：
- 控制面节点注册与心跳
- 执行面节点注册与心跳
- 运行库骨架表自动初始化
- 集群节点快照接口
- 国外/大陆两套部署脚手架
当前已落地的运行态表：
- `detect_worker_nodes`
- `detect_jobs`
- `detect_job_items`
- `detect_run_events`
- `detect_sync_records`
当前可直接验证的接口：
- `GET /api/v1/runtime/status`
- `GET /api/v1/runtime/readiness`
- `GET /api/v1/runtime/cluster`
- `GET /api/v1/detect/job/active`
- `GET /api/v1/detect/jobs`
- `GET /api/v1/detect/queue-summary`
当前 `runtime/cluster` 还补充了 `summary` 摘要，至少会返回：
- `status_counts`
- `role_counts`
- `region_counts`
- `online_worker_nodes`
- `online_control_nodes`
- `busy_nodes`
- `stale_nodes`
- `offline_nodes`
当前还补充了同步观测骨架：
- `GET /api/v1/runtime/sync-summary`
- `GET /api/v1/runtime/sync-records`
可用于提前观察：
- 当前节点是否启用同步推送
- 源地域 / 目标地域配置是否正确
- `detect_sync_records` 最近状态分布
当前代码还补了一层“运行态自动投影”：
- 控制面在读取 `runtime/status` 时，会把当前运行摘要按去重策略写入 `detect_sync_records`
- 当前记录类型为：
- `runtime_projection`
- 当前还新增了“结果摘要投影”：
- `detect_result_projection`
- 用于把当前活跃任务的摘要、最近阶段事件、任务收敛进度先沉淀到统一同步面板
- 其作用不是替代后续真正的 `sync-service`
- 而是先保证跨地域部署阶段已经有统一可观察的“同步记录面板”
当前还补上了正式同步最小闭环骨架：
- 大陆执行面可通过 `SYNC_TARGET_API_BASE_URL` 指向国外控制面
- 若 `SYNC_PUSH_ENABLED=true`，应由大陆 `domaincheck-sync-agent` 按轮询间隔自动推送最新运行态投影
- 国外控制面通过：
- `POST /api/v1/runtime/sync-ingest`
接收投影
- 若配置了 `SYNC_SHARED_TOKEN`，接收端会校验：
- `X-Domaincheck-Sync-Token`
因此当前虽然还没有完整业务结果同步服务，但“同步协议、发送端、接收端、记录面板、独立同步代理”都已经有了第一版可运行骨架
当前同步骨架已经支持的记录类型包括：
- `runtime_projection`
- `runtime_ingest`
- `detect_result_projection`
- `detect_result_ingest`
- `runtime_push`
当前这一轮还补上了“结果批次同步可观测”能力：
- `GET /api/v1/runtime/sync-summary` 现在会额外返回：
- `detect_result_batches`
- 可直接按最近检测任务查看：
- 是否已生成结果投影
- 是否已推送
- 是否已被目标地域接收
- 最近一次失败原因
- 当前同步策略也已细化为：
- `runtime_projection`
只推最新一条，避免把历史运行态投影整批补传
- `detect_result_projection`
按批次补推，确保检测结果积压可以自动追平
- 大陆 `controller` 的环境模板也已固定为：
- `NODE_ROLE=control`
避免把 controller 误注册成普通 Worker，导致运行中心和 sync-agent 承载判断失真
当前这一轮还额外补齐了“检测会话追踪”能力：
- `POST /api/v1/detect/start` 会生成本轮 `cycle_token`
- `cycle_token` 会进入 Worker 控制指令
- Worker 心跳元数据会回写 `cycle_token / job_id / job_code`
- `domain_started / domain_completed / domain_failed` 事件会附带 `cycle_token`
- `GET /api/v1/detect/job/active` 已支持按当前 `cycle_token` 聚合“本轮事件”
当前运行态还补齐了“代理执行语义”：
- 代理可用时显示 `代理正常`
- 代理暂时不可用但允许直连时显示 `降级直连`
- 代理暂时不可用且不允许直连时显示 `等待代理`
- 当代理源全部 `HTTP 200` 但 `raw_items=0` 时，会额外标记为“供应池为空”，避免误判成程序异常
当前阶段的准确表述应为：
> 多机运行骨架已经落地，分布式调度与跨地域结果同步进入下一阶段。
截至当前这一轮代码，最小任务闭环也已经接上：
- `POST /api/v1/detect/start` 会先创建 `detect_jobs`
- Worker 会优先从 `detect_job_items` 领取任务
- 运行中心已能显示当前活跃任务摘要
- 节点状态里已能反映 `busy / current_load`
- Worker 已支持小批量领取、任务续租、过期租约回收
- API 控制面节点已支持定时心跳，避免集群视图把控制面误判为离线
- 活跃任务接口已支持节点分布、进度百分比和最近事件回看
- Worker 进程重启后会主动回收当前节点遗留的 `running / claimed` 运行态，避免后台出现“进程已重启但旧任务仍显示运行中”的残影
这意味着后续新增第二台大陆 Worker 时，已经不再是“完全从零设计”的状态，而是可以在现有骨架上继续细化调度策略。
当前最小联调命令也已经固定为：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100
```
这条命令会连续检查：
- `/health`
- `/api/v1/runtime/cluster`
- `/api/v1/detect/status`
- `/api/v1/detect/job/active`
如需单独检查“跨地域同步骨架是否已接上”，建议再补两条：
```bash
curl http://127.0.0.1:8100/api/v1/runtime/sync-summary
curl http://127.0.0.1:8100/api/v1/runtime/sync-records?limit=10
```
如需快速拿到“当前这套多机部署是否已进入可联调 / 待处理 / 阻断”结论，当前也已经固定了：
```bash
curl http://127.0.0.1:8100/api/v1/runtime/readiness
```
该接口会直接给出：
- `status`
- `ready`
- `attention`
- `blocking`
- `summary`
- `blocking_issues`
- `warnings`
- `info`
如果这两条接口已经能看到：
- `source_region / target_region`
- `records_total > 0`
- 最近记录里出现 `runtime_projection`
则说明当前测试服已经具备：
- 同步配置观测入口
- 同步记录落库入口
- 后续接入正式 `sync-service` 的最小运行骨架
如果要在大陆 controller 节点本机确认“这台机器本身是否配成了 controller，而不是普通 Worker”，当前也已经固定了一条本机检查命令：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/check_mainland_controller.sh
```
这条命令会直接校验：
- `/etc/default/domaincheck-worker`
- `NODE_ROLE=control`
- `SYNC_PUSH_ENABLED=true`
- `SYNC_TARGET_API_BASE_URL` 是否已填写
- `domaincheck-worker`
- `domaincheck-sync-agent`
是否已经启动
如果当前客观条件下还不能把代码真正部署到大陆机器，当前也支持先在国外测试机上做“模拟多节点联调”：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/simulate_multi_region.sh http://127.0.0.1:8100
```
这条命令会临时模拟：
- 一个大陆 `controller`
- 一个大陆 `worker`
它的目的不是替代真实部署，而是先把下面这些提前跑通：
- `runtime/readiness`
- `runtime/cluster`
- 运行中心的多机就绪度提示
- 文档、脚本、状态面板三者是否一致
如果测试服之前已经跑过旧节点、旧 worker，当前还残留陈旧心跳记录，也可以先清一次 `detect_worker_nodes` 里的历史残影，再观察 readiness：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 --dry-run
bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30
```
它的作用是：
- 清掉很久没心跳的旧节点记录
- 避免运行中心长期被历史离线节点污染成 `attention`
- 让单机模拟多节点联调时，看到更接近真实接入后的状态
## 三点五、当前实测结论
截至 `2026-04-16 18:56`，测试服已经验证通过：
- `domaincheck-api` 与 `domaincheck-worker` 可在 `systemd` 下稳定重启
- `worker_mode=linux-systemd`
- `runtime/cluster` 可同时看到：
- `overseas-control-01`
- `mainland-worker-01`
- 大陆 Worker 节点元数据已显示：
- `phase`
- `detail`
- `job_id`
- `job_code`
- `cycle_token`
- `detect/job/active` 的 `current_cycle_events` 已只回看当前这一轮检测事件，不再混入旧轮次历史
当前测试服最新一轮实测 `cycle_token` 示例：
- `aec105737f`
该轮已确认能看到：
- `job_dispatch_requested`
- `job_dispatch_sent`
- `domain_started`
- `domain_completed`
这说明“控制面发起一次检测”到“执行面逐域名回写事件”的会话闭环已经打通。
也说明当前后台已经不再只会显示“有告警”，而是能明确说明任务到底是在正常代理、降级直连，还是被代理卡住。
## 四、推荐架构
### 1. 国外控制面
国外机器负责：
- `domain-web`
- `domain-api`
- 国外主 PostgreSQL
- 配置中心
- 导出、审核、运营后台
- 最终结果归档
- 国外备份与日志归档
国外控制面的职责：
- 对外提供访问入口
- 保存最终业务结果
- 保存系统配置和审计记录
- 接收大陆执行面的结果同步
- 为后续灾备提供稳定的权威数据源
### 2. 大陆执行面
大陆机器负责：
- `detect-worker`
- 大陆 Redis
- 大陆运行库 PostgreSQL
- 代理池配置与网络环境
- 本地运行日志
- 调度和任务租约
大陆执行面的职责：
- 抢任务
- 续租
- 运行检测链路
- 写入本地运行态
- 汇总并同步结果到国外
### 3. 初期物理部署
当前最优起步方案：
- 国外 `1` 台：
- `domain-web`
- `domain-api`
- `postgresql_main`
- 大陆 `1` 台：
- `redis`
- `postgresql_runtime`
- `scheduler`
- `detect-worker`
说明：
- 这个起步方案的优先目标是先把跨地域职责边界定清楚
- 大陆机器当前可以先把 Redis、运行库、调度和 Worker 放在同一台
- 但逻辑上必须按不同角色设计，避免后续扩容时重新返工
- 各节点身份建议统一通过 `/etc/default/domaincheck-api` 与 `/etc/default/domaincheck-worker` 管理，不直接修改 service 文件正文
### 4. 后续扩容
当大陆执行压力不够时，扩容顺序建议为：
1. 新增大陆 Worker 机器
2. 如果 Redis 压力增大，再拆 Redis
3. 如果运行库写入压力增大，再拆运行库
4. 国外主库继续作为最终结果主库，不参与高频任务调度
后续结构示意：
```text
国外
├── web + api + postgresql_main
└── backup / archive
大陆
├── redis + postgresql_runtime + scheduler
├── worker-01
├── worker-02
├── worker-03
└── worker-N
```
## 五、数据分层
### 1. 结果态数据
这类数据保存在国外主库：
- 域名主数据
- 检测最终结果
- 黑名单结论
- 导出记录
- 审核记录
- 系统配置
- 操作审计
特点：
- 最终一致
- 生命周期长
- 提供给后台页面和运营使用
### 2. 运行态数据
这类数据优先保存在大陆执行面：
- 任务领取状态
- 任务租约
- 节点心跳
- 当前阶段
- 实时日志事件
- 重试信息
- 本地暂存结果
特点：
- 高频读写
- 延迟敏感
- 主要服务于 Worker 协作和调度
## 六、任务模型
当前项目还偏向单机 Worker 直接扫描 `domains` 表的模式。正式多机化后，建议改为显式任务模型。
### 1. 核心表建议
建议新增：
- `detect_jobs`
- `detect_job_items`
- `detect_worker_nodes`
- `detect_run_events`
- `detect_sync_records`
### 2. detect_jobs
表示一轮检测任务。
建议字段：
- `id`
- `job_code`
- `source`
- `plan_hash`
- `status`
- `created_at`
- `started_at`
- `finished_at`
- `created_by`
- `remark`
说明：
- `plan_hash` 用于标识本轮检测所使用的规则版本和参数快照
- 一轮任务可以对应大量域名任务项
### 3. detect_job_items
表示单个域名的具体任务项。
建议字段：
- `id`
- `job_id`
- `domain_id`
- `status`
- `claimed_by`
- `claim_token`
- `lease_expires_at`
- `attempt_count`
- `last_error`
- `started_at`
- `finished_at`
- `result_version`
- `updated_at`
建议唯一约束：
- `unique(job_id, domain_id)`
说明：
- 多台 Worker 只抢 `status=pending` 且未被占用或租约已过期的任务
- `claim_token` 用于避免状态回写串单
- `lease_expires_at` 用于机器故障后的任务回收
### 4. detect_worker_nodes
表示 Worker 节点。
建议字段：
- `node_code`
- `region`
- `role`
- `hostname`
- `ip`
- `status`
- `worker_version`
- `last_heartbeat_at`
- `current_load`
- `remark`
说明：
- 后台后续可以直接展示“哪个节点在线、哪个节点在跑、当前负载多少”
### 5. detect_run_events
表示高价值阶段日志。
建议字段：
- `id`
- `job_id`
- `job_item_id`
- `node_code`
- `event_type`
- `level`
- `message`
- `payload_json`
- `created_at`
说明：
- 页面展示优先依赖事件流，不直接硬依赖某一台机器本地日志文件
- 原始大日志仍保留在大陆 Worker 本地
## 七、多机抢任务与去重
### 1. 抢任务方式
推荐使用数据库租约模型：
- Worker 从 `detect_job_items` 中领取一批待处理任务
- 使用 `FOR UPDATE SKIP LOCKED` 或原子更新方式抢占
- 领取后写入：
- `claimed_by`
- `claim_token`
- `lease_expires_at`
### 2. 去重原则
多机环境下，去重不能依赖“约定不重复”，必须依赖数据库约束和租约。
至少保证：
- 同一轮 `job_id` 下，同一域名只出现一条任务项
- 同一任务项同一时刻只允许被一个 `claim_token` 持有
- 回写结果时校验 `claim_token`
### 3. 续租机制
Worker 执行过程中应定期续租：
- 每 `30-60` 秒续租一次
- 续租时更新 `lease_expires_at`
- 如果节点挂掉，租约到期后其他 Worker 可重新领取
当前代码已实现的原则是：
- 任务被领取后进入 `claimed / running`
- 检测步骤执行过程中会持续续租
- 若任务长期未续租且租约过期，会自动回收到 `pending`
这样可以避免：
- 长任务跑到一半租约失效
- Worker 异常退出后任务永远卡死
- 后续多机扩容时出现大量“假占用”
### 4. 失败重试
建议对失败进行分类：
- 网络失败
- 代理失败
- 第三方目标站异常
- 程序逻辑失败
可按类型控制重试次数，不要无限重试。
### 5. 当前代码与目标模型的关系
当前代码里，运行态表和节点心跳已经具备，但主检测流程仍以现有 `domains` / `detect_status` 链路为主。
这意味着：
- 现在已经可以先把“多机可见性”和“节点骨架”跑起来
- 下一步要把“检测入口”从直接扫 `domains`，逐步切到 `detect_jobs + detect_job_items`
- 切换时必须保留旧链路回退能力，避免一次性重写导致现网不稳
## 八、同步策略
### 1. 国外到大陆
同步内容：
- 系统配置
- 检测规则
- 敏感词版本
- 任务创建命令
特点：
- 低频
- 要求可靠
- 可版本化
### 2. 大陆到国外
同步内容：
- 任务进度摘要
- 关键阶段事件
- 最终检测结果
- 黑名单结论
- 归档日志包索引
特点：
- 高频部分只传摘要和关键事件
- 最终结果必须幂等
### 3. 幂等要求
结果同步到国外时，必须支持幂等：
- 同一 `job_item_id` 重复上报不应产生重复结果
- 同一域名同一轮任务只保留最终有效状态
### 4. 推荐同步落地方式
第一阶段不建议直接做数据库双向复制，更推荐独立同步服务：
- 大陆执行面负责写本地运行态
- `sync-service` 负责批量汇总结果与事件
- 国外控制面负责接收、落库、归档
推荐第一阶段同步内容：
- 最终检测结果
- 关键阶段事件
- 失败摘要
- 节点健康摘要
- 配置快照版本
不建议第一阶段同步：
- 全量原始日志
- 高频心跳明细
- 临时抓取中间产物
原因：
- 成本高
- 跨地域网络抖动会放大耦合
- 对运营后台价值不成比例
## 九、部署建议
### 1. 国外机器
目录建议：
```text
/opt/domaincheck
├── domain-api
├── domain-web
└── shared
```
部署角色：
- `web`
- `api`
- `postgresql_main`
### 2. 大陆调度中心
目录建议：
```text
/opt/domaincheck
├── domainCheck
├── runtime-db
├── redis
└── shared
```
部署角色：
- `scheduler`
- `redis`
- `postgresql_runtime`
- `worker`
### 3. 大陆纯 Worker 节点
目录建议：
```text
/opt/domaincheck
├── domainCheck
└── shared
```
部署角色：
- `worker`
### 4. 一键化原则
部署脚本必须满足：
- 新机器只需要少量环境变量
- 不要求现场手工改很多路径
- 国外机和大陆机各自有独立入口脚本
- 后续新增大陆 Worker 节点继续复用同一脚本
### 5. 当前建议的最小部署
当前最适合正式起步的形态：
- 国外 `1` 台：
- `domain-web`
- `domain-api`
- 国外主 PostgreSQL
- 大陆 `1` 台：
- `redis`
- 大陆运行库 PostgreSQL
- `detect-worker`
- 后续可补 `sync-service`
扩容时优先：
1. 新增大陆 Worker
2. 再视压力拆分大陆运行库与 Redis
3. 国外控制面保持稳定，不跟着高频扩缩
## 十、阶段性落地建议
### 第一阶段
先完成：
- 国外控制面部署
- 大陆单机执行面部署
- 节点注册与心跳
- 当前后台继续可用
### 第二阶段
再完成：
- 任务项模型
- 多 Worker 抢任务
- 事件日志流
- 结果异步同步回国外
### 第三阶段
最后完成：
- 大陆多 Worker 横向扩容
- 运行库与调度中心独立拆分
- 国外归档和监控补齐
## 十一、运行与观测建议
多机之后，不能再只靠“登录某台机器 tail 日志”来判断系统状态。
建议分层：
- 大陆 Worker 本地保留完整原始日志
- `detect_run_events` 保存关键阶段事件
- 国外后台优先展示事件流、节点负载和同步状态
- 真排障时再下钻到具体节点原始日志
后台建议优先展示：
- 当前在线节点数
- 每节点最近心跳
- 每节点当前负载
- 当前活跃任务数
- 队列积压与最老待领年龄
- 租约是否过期、是否即将到期
- 近 15 分钟每节点吞吐
- 最近失败摘要
- 最近同步结果
- 最近结果摘要投影
## 十二、下一阶段开发清单
接下来建议按下面顺序继续落地：
1. 后台创建 `detect_jobs / detect_job_items`
2. Worker 实现批量 claim / renew / finish / fail
3. 增加租约超时回收
4. 将检测阶段事件写入 `detect_run_events`
5. 增加 `sync-service`
6. 完成第二台大陆 Worker 接入演练
## 十三、最终建议
对当前项目来说，最优解不是“全部放国外”，也不是“全部放大陆”，而是：
- 国外做控制面和主结果库
- 大陆做执行面和运行态调度
- 先从 `国外 1 台 + 大陆 1 台` 起步
- 后续优先横向增加大陆 Worker
一句话总结：
> 先把职责边界设计对，再让部署简单；后续扩容时只加 Worker，不再重构核心架构。

647
docs/17_domainCheck_全流程部署实操手册.md Normal file
View File

@@ -0,0 +1,647 @@
# 17 domainCheck 全流程部署实操手册
## 一、文档目标
这份文档不是设计说明，也不是零散检查单，而是给实际部署时直接照着做的“全流程操作手册”。
适用场景：
- 先在国外机器完成单机部署与联调
- 后续再扩到“国外控制面 + 大陆执行面”
- 当前暂时无法把代码真正部署到大陆机器时，先在国外机器模拟多节点联调
- 最后再把同样的部署方式复制到其他机器
一句话理解：
> 先把国外主控机部署好，再按同样套路扩第二台、第三台，不靠临场猜。
并发配置补充：
- 系统支持 `默认线程数 + 节点单独覆盖`
- 节点单独覆盖按机器自己的 `NODE_CODE` 命中
- 某台机器没配置覆盖值时，自动回退到默认线程数
## 二、推荐部署形态
### 1. 当前最推荐起步形态
- 国外机器 `1` 台：
- `domain-web`
- `domain-api`
- PostgreSQL
- Redis
- `domaincheck-worker`
- 后续扩容时：
- 新增大陆 `controller`
- 新增大陆 `worker`
说明：
- 因为你当前还不能直接把代码稳定部署到大陆机器，所以第一阶段先把国外机器单机跑稳
- 这台国外机器同时承担：
- 后台
- API
- 数据库
- Redis
- Worker
- 等这套稳定后，再把多机脚本复制到其他机器
### 2. 最终推荐形态
- 国外 `control`：
- `domain-web`
- `domain-api`
- 主 PostgreSQL
- 结果归档
- 大陆 `controller`：
- Redis
- 运行态 PostgreSQL
- `domaincheck-worker`
- `domaincheck-sync-agent`
- 大陆 `worker`：
- `domaincheck-worker`
## 三、服务器准备
### 1. 系统要求
- CentOS Stream 9
- Python `3.11`
- PostgreSQL `14+`
- Redis `6+`
- Nginx
### 2. 目录约定
统一使用：
```text
/opt/domaincheck
├── domain-api
├── domain-web
└── domainCheck
```
### 3. 当前仓库与运行态关系
如果你像当前测试机一样，用仓库目录做源码源头，也可以用软链接：
```bash
ln -s /www/wwwroot/getDomain/domain-api /opt/domaincheck/domain-api
ln -s /www/wwwroot/getDomain/domain-web /opt/domaincheck/domain-web
ln -s /www/wwwroot/getDomain/domainCheck /opt/domaincheck/domainCheck
```
如果你是完整复制代码到目标机，也可以直接把目录上传到 `/opt/domaincheck/`。
## 四、第一阶段：国外单机部署
这一阶段的目标是：
- 后台可访问
- API 正常
- 数据库已初始化
- Worker 可跑
- 运行中心正常
- smoke test 通过
### 1. 拉取代码
当前建议统一走 `git` 管理和更新，不再手工散传目录。
第一次部署建议：
```bash
mkdir -p /www/wwwroot
cd /www/wwwroot
git clone 你的仓库地址 getDomain
cd getDomain
git checkout main
git pull origin main
```
然后建立运行目录软链接：
```bash
mkdir -p /opt/domaincheck
ln -s /www/wwwroot/getDomain/domain-api /opt/domaincheck/domain-api
ln -s /www/wwwroot/getDomain/domain-web /opt/domaincheck/domain-web
ln -s /www/wwwroot/getDomain/domainCheck /opt/domaincheck/domainCheck
```
后续更新统一使用：
```bash
cd /www/wwwroot/getDomain
git fetch --all
git checkout main
git pull --ff-only origin main
```
### 2. 创建 Python 虚拟环境
```bash
cd /opt/domaincheck/domainCheck
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```
然后补 API 依赖：
```bash
cd /opt/domaincheck/domain-api
/opt/domaincheck/domainCheck/.venv/bin/pip install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
```
### 3. 配置数据库和 Redis
确保 PostgreSQL 和 Redis 可用。
建议先确认：
```bash
psql -h 127.0.0.1 -U postgres -d domain -c "select 1;"
redis-cli ping
```
### 4. 配置 `domainCheck/.env`
至少确认这些项：
```env
DB_HOST=127.0.0.1
DB_PORT=5432
DB_DATABASE=domain
DB_USER=postgres
DB_PASSWORD=你的密码
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
```
### 5. 初始化数据库
这是最重要的一步，新库必须先做：
```bash
cd /opt/domaincheck/domainCheck
python3 init_database.py
```
如果不先执行，至少这些接口会直接 `500`：
- `/api/v1/dashboard/overview`
- `/api/v1/detect/status`
- `/api/v1/imports/summary`
### 6. 构建前端
进入 `domain-web`，配置生产环境 API 地址：
```bash
cd /opt/domaincheck/domain-web
cp .env.production.example .env.production
```
把 `VITE_API_BASE_URL` 改成目标 API，例如：
```env
VITE_API_BASE_URL=https://api.domain.com/api/v1
```
然后构建：
```bash
npm install
npm run build
```
### 7. 配置 Nginx
参考：
- `domain-web/deploy/nginx/domain-web.conf`
把静态目录指到：
```text
/opt/domaincheck/domain-web/dist
```
常见做法：
- `admin.domain` 对外提供后台页面
- `api.domain.com` 对外提供 API
### 8. 安装 systemd 服务
复制模板：
```bash
cp /opt/domaincheck/domain-api/deploy/systemd/domain-api.service /etc/systemd/system/domaincheck-api.service
cp /opt/domaincheck/domain-api/deploy/systemd/domain-worker.service /etc/systemd/system/domaincheck-worker.service
```
### 9. 配置 API 环境变量
建议使用：
```bash
cp /opt/domaincheck/domain-api/deploy/multi-region/templates/domaincheck-api.env.example /etc/default/domaincheck-api
```
至少改这些：
```env
WORKER_MODE=linux-systemd
API_HOST=0.0.0.0
API_PORT=8100
DOMAIN_ROOT=/opt/domaincheck/domainCheck
NODE_CODE=overseas-control-01
NODE_REGION=overseas
NODE_ROLE=control
CORS_ORIGINS=http://127.0.0.1:3201,http://localhost:3201,http://你的服务器IP:3201
SYNC_PUSH_ENABLED=false
```
如果你已经固定域名，建议直接改成：
```env
CORS_ORIGINS=https://admin.domain,http://127.0.0.1:3201,http://localhost:3201
```
### 10. 修正权限
```bash
mkdir -p /opt/domaincheck/domain-api/runtime
touch /opt/domaincheck/domainCheck/detect_worker.log
chown -R www:www /opt/domaincheck/domain-api/runtime
chown www:www /opt/domaincheck/domainCheck/detect_worker.log
chmod 664 /opt/domaincheck/domainCheck/detect_worker.log
```
### 11. 启动服务
```bash
systemctl daemon-reload
systemctl enable domaincheck-api
systemctl enable domaincheck-worker
systemctl restart domaincheck-api
systemctl restart domaincheck-worker
```
### 12. 检查服务状态
```bash
systemctl status domaincheck-api --no-pager -l
systemctl status domaincheck-worker --no-pager -l
```
## 五、第一阶段验收
### 1. 接口检查
```bash
curl http://127.0.0.1:8100/health
curl http://127.0.0.1:8100/api/v1/runtime/preflight
curl http://127.0.0.1:8100/api/v1/runtime/status
curl http://127.0.0.1:8100/api/v1/runtime/readiness
```
期望：
- `/health.status=ok`
- `/health.worker_mode=linux-systemd`
- `/runtime/preflight.ok=true`
- `/runtime/status.worker.running=true`
- `/runtime/readiness.status` 至少不是 `blocking`
### 2. smoke test
```bash
cd /opt/domaincheck/domain-api/deploy/linux
python3 smoke_test.py --base-url http://127.0.0.1:8100
```
如果要连 Web 一起检查：
```bash
python3 smoke_test.py --base-url http://127.0.0.1:8100 --web-url http://127.0.0.1:3201
```
期望：
- `ok=true`
### 3. 后台页面检查
浏览器打开：
```text
https://admin.domain
```
至少检查：
- 登录
- 运行中心
- 系统设置
- 域名筛选
- 导入
- 导出
## 六、第二阶段：国外机器模拟多机联调
如果你暂时还不能把代码部署到大陆机器，就先在国外机器做这一步。
### 1. 一键多机演练
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/rehearse_multi_region.sh http://127.0.0.1:8100
```
它会自动模拟：
- 一个大陆 controller
- 两个大陆 worker
并自动检查：
- `runtime/readiness`
- `runtime/cluster`
- online control / online worker 数量
- 节点状态是否符合预期
### 2. 通过标准
如果输出里出现：
```text
rehearsal passed
```
说明当前这台国外机器上的多机模拟联调已通过，可以继续部署到其他机器。
### 3. 如果想手工模拟
也可以单独跑：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/simulate_multi_region.sh http://127.0.0.1:8100
```
停止时按：
```text
Ctrl+C
```
### 4. 清理旧节点残影
如果之前测试过多轮，集群里可能残留老节点：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 --dry-run
bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30
```
如果只删某个旧节点：
```bash
bash deploy/multi-region/prune_cluster_nodes.sh --node-code mainland-worker-01
```
## 七、第三阶段：部署到其他机器
等国外单机和模拟多机都通过后，就可以把同样的代码与脚本部署到其他机器。
### A. 新海外控制机
在目标机执行：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/bootstrap_overseas.sh /opt/domaincheck
```
然后完善：
- `/etc/default/domaincheck-api`
- Nginx
- PostgreSQL
- Redis
最后检查：
```bash
bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100
```
### B. 大陆 controller
在目标机执行：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
```
然后检查：
```bash
bash deploy/multi-region/check_mainland_controller.sh
```
关键配置是：
```env
NODE_CODE=mainland-controller-01
NODE_REGION=mainland
NODE_ROLE=control
SYNC_PUSH_ENABLED=true
SYNC_SOURCE_REGION=mainland
SYNC_TARGET_REGION=overseas
SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
SYNC_SHARED_TOKEN=你自己的共享令牌
```
### C. 大陆 worker
在目标机执行：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
```
关键配置是：
```env
NODE_CODE=mainland-worker-01
NODE_REGION=mainland
NODE_ROLE=worker
SYNC_PUSH_ENABLED=true
SYNC_SOURCE_REGION=mainland
SYNC_TARGET_REGION=overseas
SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
```
## 八、部署到其他机器后的联调顺序
建议按这个顺序，不容易乱：
1. 先让海外控制机稳定
2. 再接大陆 controller
3. 再接第一台大陆 worker
4. 再接第二台、第三台 worker
每接一台都执行：
```bash
curl http://海外控制机:8100/api/v1/runtime/readiness
curl http://海外控制机:8100/api/v1/runtime/cluster
curl http://海外控制机:8100/api/v1/runtime/sync-summary
```
如果海外 API 已经挂到正式域名，建议直接写成：
```bash
curl https://api.domain.com/api/v1/runtime/readiness
curl https://api.domain.com/api/v1/runtime/cluster
curl https://api.domain.com/api/v1/runtime/sync-summary
```
### 联调通过标准
- `runtime/cluster` 能看到新节点
- `last_heartbeat_at` 持续刷新
- 大陆 controller 为 `role=control`
- 大陆 worker 为 `role=worker`
- 执行任务时 worker 状态能变成 `busy`
- `runtime/readiness` 至少不是 `blocking`
- `runtime/sync-summary` 能看到 `detect_result_batches`
## 九、上线前最后检查
按这份文档部署完成后，再回看：
- [14_domainCheck_正式上线前最终检查单.md](/www/wwwroot/getDomain/docs/14_domainCheck_正式上线前最终检查单.md:1)
重点保留：
- `systemctl status`
- `journalctl`
- `/health`
- `/runtime/preflight`
- `/runtime/readiness`
- `/runtime/sync-summary`
- `smoke test`
- 诊断包
## 十、常见问题
### 1. API 启动了，但接口 500
优先检查有没有先执行：
```bash
cd /opt/domaincheck/domainCheck
python3 init_database.py
```
### 2. Worker 一直重启
优先检查：
- `/opt/domaincheck/domainCheck/detect_worker.log` 权限
- `/opt/domaincheck/domain-api/runtime/` 权限
- `domainCheck/.env` 数据库和 Redis 配置
### 3. readiness 一直是 `attention`
优先检查：
- 集群里是否残留旧离线节点
- 是否还没接入大陆 controller
- 是否还没有在线 worker
必要时先清理旧节点：
```bash
bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30
```
### 4. 模拟多机通过了，真实机器还没接上
这是正常的。
模拟多机的意义是：
- 验证代码、脚本、页面、状态口径一致
- 不代表真实大陆网络、代理、同步链路已经完成
真实机器接入时，重点要再看：
- 节点心跳
- 同步目标地址
- 共享 token
- 真实 Redis / PostgreSQL 连接
## 十一、最短执行版本
如果你只想看最短版，可以照这个跑：
### 国外单机先跑通
```bash
cd /opt/domaincheck/domainCheck
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python3 init_database.py
cd /opt/domaincheck/domain-api
/opt/domaincheck/domainCheck/.venv/bin/pip install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
systemctl daemon-reload
systemctl enable domaincheck-api domaincheck-worker
systemctl restart domaincheck-api domaincheck-worker
curl http://127.0.0.1:8100/health
curl http://127.0.0.1:8100/api/v1/runtime/preflight
curl http://127.0.0.1:8100/api/v1/runtime/readiness
```
### 再跑多机模拟验收
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/rehearse_multi_region.sh http://127.0.0.1:8100
```
### 通过后再上其他机器
```bash
bash deploy/multi-region/bootstrap_overseas.sh /opt/domaincheck
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
```
## 十二、最终结论
当前最稳的推进方式是：
1. 先在国外机器完成单机部署
2. 再在国外机器完成多机模拟演练
3. 演练通过后，再复制到其他机器
4. 最后再做真实跨地域联调
一句话总结：
> 先把单机跑稳，再把多机脚本跑通，最后再扩机器；每一步都有现成脚本，不靠现场猜。

1015
docs/18_domainCheck_CentOS9一键复制部署与更新文档.md Normal file
View File

File diff suppressed because it is too large Load Diff

172
domain-api/app/api/routes/detect.py
View File

@@ -1,25 +1,151 @@
from uuid import uuid4
from fastapi import APIRouter from fastapi import APIRouter
from app.schemas.common import ApiResponse from app.schemas.common import ApiResponse
from app.services.detect_job_service import (
append_detect_job_event,
create_detect_job_if_needed,
get_active_detect_job_summary,
get_detect_job_summary,
get_detect_queue_health,
list_detect_jobs,
)
from app.services.detect_service import get_detect_status from app.services.detect_service import get_detect_status
from app.services.worker_control_service import start_worker, stop_worker from app.services.detect_run_service import create_detect_run_snapshot, finalize_detect_run, mark_detect_run_stopping
from app.services.settings_service import get_settings_payload, resolve_thread_count
from app.services.worker_control_service import send_worker_command, start_worker
router = APIRouter(tags=["detect"]) router = APIRouter(tags=["detect"])
def _build_settings_summary(settings_payload: dict) -> dict:
thread_count_resolution = resolve_thread_count(settings_payload=settings_payload)
return {
"thread_count": int(thread_count_resolution["effective_thread_count"]),
"thread_count_default": int(thread_count_resolution["default_thread_count"]),
"thread_count_source": str(thread_count_resolution["source"]),
"thread_count_override": thread_count_resolution["override_thread_count"],
"thread_count_node_code": str(thread_count_resolution["node_code"]),
"proxy_enable": settings_payload["proxy_config"].get("proxy_enable", False),
"allow_direct": settings_payload["proxy_config"].get("allow_direct", False),
"proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])),
}
@router.get("/detect/status", response_model=ApiResponse) @router.get("/detect/status", response_model=ApiResponse)
def detect_status() -> ApiResponse: def detect_status() -> ApiResponse:
return ApiResponse(data=get_detect_status()) return ApiResponse(data=get_detect_status())
@router.get("/detect/job/active", response_model=ApiResponse)
def detect_active_job() -> ApiResponse:
return ApiResponse(data=get_active_detect_job_summary(event_limit=50))
@router.get("/detect/jobs", response_model=ApiResponse)
def detect_jobs(limit: int = 20) -> ApiResponse:
return ApiResponse(data={"items": list_detect_jobs(limit=limit), "limit": max(1, min(int(limit or 20), 100))})
@router.get("/detect/queue-summary", response_model=ApiResponse)
def detect_queue_summary(window_minutes: int = 15) -> ApiResponse:
return ApiResponse(data=get_detect_queue_health(window_minutes=window_minutes))
@router.get("/detect/jobs/{job_id}", response_model=ApiResponse)
def detect_job_detail(job_id: int) -> ApiResponse:
data = get_detect_job_summary(job_id, event_limit=100)
if not data:
return ApiResponse(code=1, message="检测任务不存在", data=None)
return ApiResponse(data=data)
@router.post("/detect/start", response_model=ApiResponse) @router.post("/detect/start", response_model=ApiResponse)
def start_detect() -> ApiResponse: def start_detect() -> ApiResponse:
job_summary = create_detect_job_if_needed(limit=1000, created_by="api")
if not job_summary:
return ApiResponse(
code=0,
message="当前没有可创建的检测任务",
data={
"action": "start",
"job": None,
"poll_after_seconds": 2,
"refresh_status": True,
},
)
cycle_token = uuid4().hex[:10]
append_detect_job_event(
job_summary["job_id"],
event_type="job_dispatch_requested",
message="控制面已发送检测启动请求",
payload={
"cycle_token": cycle_token,
"status": job_summary.get("status"),
"items_pending": job_summary.get("items_pending", 0),
"items_claimed": job_summary.get("items_claimed", 0),
"items_running": job_summary.get("items_running", 0),
},
)
ok, message = start_worker() ok, message = start_worker()
if not ok:
append_detect_job_event(
job_summary["job_id"],
event_type="job_dispatch_failed",
level="error",
message=f"启动 Worker 失败: {message}",
payload={"cycle_token": cycle_token},
)
return ApiResponse(
code=1,
message=message,
data={
"action": "start",
"job": job_summary,
"poll_after_seconds": 2,
"refresh_status": True,
},
)
command_ok, command_message = send_worker_command(
"start_detection",
payload={
"cycle_token": cycle_token,
"job_id": job_summary["job_id"],
"job_code": job_summary["job_code"],
},
)
append_detect_job_event(
job_summary["job_id"],
event_type="job_dispatch_sent" if command_ok else "job_dispatch_rejected",
level="info" if command_ok else "error",
message=command_message,
payload={"cycle_token": cycle_token},
)
snapshot = get_detect_status()
settings_payload = get_settings_payload()
settings_summary = _build_settings_summary(settings_payload)
if command_ok:
create_detect_run_snapshot(
message=f"{message}；{command_message}",
runtime={
"mode": snapshot.get("worker_mode", ""),
"running": snapshot.get("worker_online", False),
"process_count": snapshot.get("worker_process_count", 0),
"latest_start_time": snapshot.get("worker_latest_start_time", ""),
"message": snapshot.get("worker_runtime_message", ""),
},
progress=snapshot.get("progress", {}),
settings_summary=settings_summary,
)
return ApiResponse( return ApiResponse(
code=0 if ok else 1, code=0 if command_ok else 1,
message=message, message=f"{message}；{command_message}" if command_ok else command_message,
data={ data={
"action": "start", "action": "start",
"job": job_summary,
"poll_after_seconds": 2, "poll_after_seconds": 2,
"refresh_status": True, "refresh_status": True,
}, },
@@ -28,7 +154,45 @@ def start_detect() -> ApiResponse:
@router.post("/detect/stop", response_model=ApiResponse) @router.post("/detect/stop", response_model=ApiResponse)
def stop_detect() -> ApiResponse: def stop_detect() -> ApiResponse:
ok, message = stop_worker() active_job = get_active_detect_job_summary(event_limit=10)
ok, message = send_worker_command("stop_detection")
if active_job:
append_detect_job_event(
active_job["job_id"],
event_type="job_stop_requested" if ok else "job_stop_request_failed",
level="info" if ok else "error",
message=message,
payload={"cycle_token": active_job.get("current_cycle_token", "")},
)
snapshot = get_detect_status()
settings_payload = get_settings_payload()
settings_summary = _build_settings_summary(settings_payload)
mark_detect_run_stopping(
message=message,
runtime={
"mode": snapshot.get("worker_mode", ""),
"running": snapshot.get("worker_online", False),
"process_count": snapshot.get("worker_process_count", 0),
"latest_start_time": snapshot.get("worker_latest_start_time", ""),
"message": snapshot.get("worker_runtime_message", ""),
},
progress=snapshot.get("progress", {}),
settings_summary=settings_summary,
)
if not snapshot.get("worker_online", False):
finalize_detect_run(
message=message,
runtime={
"mode": snapshot.get("worker_mode", ""),
"running": snapshot.get("worker_online", False),
"process_count": snapshot.get("worker_process_count", 0),
"latest_start_time": snapshot.get("worker_latest_start_time", ""),
"message": snapshot.get("worker_runtime_message", ""),
},
progress=snapshot.get("progress", {}),
settings_summary=settings_summary,
active_job=active_job,
)
return ApiResponse( return ApiResponse(
code=0 if ok else 1, code=0 if ok else 1,
message=message, message=message,

16
domain-api/app/api/routes/domains.py
View File

@@ -1,7 +1,9 @@
from __future__ import annotations
from fastapi import APIRouter, Query from fastapi import APIRouter, Query
from app.schemas.common import ApiResponse from app.schemas.common import ApiResponse
from app.services.domains_service import batch_update_domains, domain_filter_options, fetch_domains from app.services.domains_service import batch_update_domains, domain_filter_options, fetch_domain_detail, fetch_domains
router = APIRouter(tags=["domains"]) router = APIRouter(tags=["domains"])
@@ -24,6 +26,8 @@ def domain_list(
beian_year: int | None = Query(default=None), beian_year: int | None = Query(default=None),
snapshot_year: str | None = Query(default=None), snapshot_year: str | None = Query(default=None),
website_url: str | None = Query(default=None), website_url: str | None = Query(default=None),
company_type: str | None = Query(default=None),
source_type: int | None = Query(default=None),
backlink_gt_10: bool | None = Query(default=None), backlink_gt_10: bool | None = Query(default=None),
) -> ApiResponse: ) -> ApiResponse:
return ApiResponse( return ApiResponse(
@@ -39,11 +43,21 @@ def domain_list(
beian_year=beian_year, beian_year=beian_year,
snapshot_year=snapshot_year, snapshot_year=snapshot_year,
website_url=website_url, website_url=website_url,
company_type=company_type,
source_type=source_type,
backlink_gt_10=backlink_gt_10, backlink_gt_10=backlink_gt_10,
) )
) )
@router.get("/domains/{domain_id}", response_model=ApiResponse)
def domain_detail(domain_id: int) -> ApiResponse:
data = fetch_domain_detail(domain_id)
if not data:
return ApiResponse(code=1, message="域名不存在", data=None)
return ApiResponse(data=data)
@router.post("/domains/batch-update", response_model=ApiResponse) @router.post("/domains/batch-update", response_model=ApiResponse)
def domain_batch_update(payload: dict) -> ApiResponse: def domain_batch_update(payload: dict) -> ApiResponse:
result = batch_update_domains(payload.get("domain_ids", []), payload.get("updates", {})) result = batch_update_domains(payload.get("domain_ids", []), payload.get("updates", {}))

18
domain-api/app/api/routes/imports.py
View File

@@ -1,4 +1,4 @@
from fastapi import APIRouter, File, UploadFile from fastapi import APIRouter, File, Form, UploadFile
from app.schemas.common import ApiResponse from app.schemas.common import ApiResponse
from app.services.import_task_service import create_import_task, list_import_tasks, retry_import_task from app.services.import_task_service import create_import_task, list_import_tasks, retry_import_task
@@ -18,12 +18,24 @@ def import_tasks() -> ApiResponse:
@router.post("/imports/upload", response_model=ApiResponse) @router.post("/imports/upload", response_model=ApiResponse)
async def upload_import(file: UploadFile = File(...)) -> ApiResponse: async def upload_import(
file: UploadFile = File(...),
source_type: int = Form(default=7),
) -> ApiResponse:
content = await file.read() content = await file.read()
task = create_import_task(content, file.filename or "domains.txt") task = create_import_task(content, file.filename or "domains.txt", source_type=source_type)
return ApiResponse(message="导入任务已创建", data=task) return ApiResponse(message="导入任务已创建", data=task)
@router.post("/imports/text", response_model=ApiResponse)
def upload_import_text(payload: dict) -> ApiResponse:
text = str(payload.get("text") or "")
filename = str(payload.get("filename") or "manual_domains.txt")
source_type = int(payload.get("source_type") or 6)
task = create_import_task(text.encode("utf-8"), filename, source_type=source_type)
return ApiResponse(message="文本导入任务已创建", data=task)
@router.post("/imports/tasks/{task_id}/retry", response_model=ApiResponse) @router.post("/imports/tasks/{task_id}/retry", response_model=ApiResponse)
def import_task_retry(task_id: str) -> ApiResponse: def import_task_retry(task_id: str) -> ApiResponse:
task = retry_import_task(task_id) task = retry_import_task(task_id)

93
domain-api/app/api/routes/juming.py Normal file
View File

@@ -0,0 +1,93 @@
from fastapi import APIRouter, File, HTTPException, UploadFile
from app.schemas.common import ApiResponse
from app.services.juming_service import (
get_juming_preferences,
get_juming_status,
login_jucha_with_juming_cookie,
login_juming,
update_juming_preferences,
upload_juming_cookie,
)
from app.services.juming_task_service import create_juming_task, list_juming_tasks, request_stop_juming_task, retry_juming_task
router = APIRouter(tags=["juming"])
@router.get("/juming/status", response_model=ApiResponse)
def juming_status() -> ApiResponse:
return ApiResponse(data=get_juming_status())
@router.get("/juming/preferences", response_model=ApiResponse)
def juming_preferences() -> ApiResponse:
return ApiResponse(data=get_juming_preferences())
@router.put("/juming/preferences", response_model=ApiResponse)
def update_juming_preferences_route(payload: dict) -> ApiResponse:
try:
data = update_juming_preferences(payload)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return ApiResponse(message="聚名采集偏好已保存", data=data)
@router.post("/juming/crawl", response_model=ApiResponse)
def juming_crawl(payload: dict) -> ApiResponse:
try:
data = create_juming_task(payload)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return ApiResponse(message="聚名采集任务已创建", data=data)
@router.get("/juming/tasks", response_model=ApiResponse)
def juming_tasks() -> ApiResponse:
return ApiResponse(data=list_juming_tasks())
@router.post("/juming/tasks/{task_id}/retry", response_model=ApiResponse)
def juming_task_retry(task_id: str) -> ApiResponse:
try:
data = retry_juming_task(task_id)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return ApiResponse(message="聚名采集任务已重新加入队列", data=data)
@router.post("/juming/tasks/{task_id}/stop", response_model=ApiResponse)
def juming_task_stop(task_id: str) -> ApiResponse:
try:
data = request_stop_juming_task(task_id)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return ApiResponse(message="聚名采集任务已发送停止请求", data=data)
@router.post("/juming/login", response_model=ApiResponse)
def juming_login(payload: dict) -> ApiResponse:
try:
data = login_juming(str(payload.get("email") or ""), str(payload.get("password") or ""))
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return ApiResponse(message="聚名登录成功", data=data)
@router.post("/juming/jucha-login", response_model=ApiResponse)
def juming_jucha_login() -> ApiResponse:
try:
data = login_jucha_with_juming_cookie()
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return ApiResponse(message="聚查联名登录成功", data=data)
@router.post("/juming/cookie/upload", response_model=ApiResponse)
async def juming_cookie_upload(file: UploadFile = File(...)) -> ApiResponse:
content = await file.read()
try:
data = upload_juming_cookie(file.filename or "juming_cookies.pkl", content)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return ApiResponse(message="聚名 Cookie 已上传", data=data)

33
domain-api/app/api/routes/runtime.py
View File

@@ -1,8 +1,13 @@
from fastapi import APIRouter from typing import Optional
from fastapi import APIRouter, Header
from app.schemas.common import ApiResponse from app.schemas.common import ApiResponse
from app.services.cluster_runtime_service import get_cluster_snapshot
from app.services.runtime_control_service import runtime_action from app.services.runtime_control_service import runtime_action
from app.services.runtime_status_service import get_runtime_preflight, get_runtime_status from app.services.runtime_status_service import get_runtime_preflight, get_runtime_status
from app.services.sync_push_service import ingest_runtime_projection
from app.services.sync_record_service import get_sync_summary, list_sync_records
router = APIRouter(tags=["runtime"]) router = APIRouter(tags=["runtime"])
@@ -12,11 +17,37 @@ def runtime_status() -> ApiResponse:
return ApiResponse(data=get_runtime_status()) return ApiResponse(data=get_runtime_status())
@router.get("/runtime/readiness", response_model=ApiResponse)
def runtime_readiness() -> ApiResponse:
return ApiResponse(data=(get_runtime_status().get("readiness") or {}))
@router.get("/runtime/preflight", response_model=ApiResponse) @router.get("/runtime/preflight", response_model=ApiResponse)
def runtime_preflight() -> ApiResponse: def runtime_preflight() -> ApiResponse:
return ApiResponse(data=get_runtime_preflight()) return ApiResponse(data=get_runtime_preflight())
@router.get("/runtime/cluster", response_model=ApiResponse)
def runtime_cluster() -> ApiResponse:
return ApiResponse(data=get_cluster_snapshot())
@router.get("/runtime/sync-summary", response_model=ApiResponse)
def runtime_sync_summary() -> ApiResponse:
return ApiResponse(data=get_sync_summary())
@router.get("/runtime/sync-records", response_model=ApiResponse)
def runtime_sync_records(limit: int = 20) -> ApiResponse:
return ApiResponse(data={"records": list_sync_records(limit=limit)})
@router.post("/runtime/sync-ingest", response_model=ApiResponse)
def runtime_sync_ingest(payload: dict, x_domaincheck_sync_token: Optional[str] = Header(default=None)) -> ApiResponse:
ok, message, data = ingest_runtime_projection(payload, shared_token=x_domaincheck_sync_token)
return ApiResponse(code=0 if ok else 1, message=message, data=data)
@router.post("/runtime/actions/{action}", response_model=ApiResponse) @router.post("/runtime/actions/{action}", response_model=ApiResponse)
def runtime_action_trigger(action: str) -> ApiResponse: def runtime_action_trigger(action: str) -> ApiResponse:
ok, message, data = runtime_action(action) ok, message, data = runtime_action(action)

17
domain-api/app/api/routes/sensitive_words.py Normal file
View File

@@ -0,0 +1,17 @@
from fastapi import APIRouter
from app.schemas.common import ApiResponse
from app.services.sensitive_words_service import get_sensitive_words_payload, save_sensitive_words_payload
router = APIRouter(tags=["sensitive_words"])
@router.get("/sensitive-words", response_model=ApiResponse)
def get_sensitive_words() -> ApiResponse:
return ApiResponse(data=get_sensitive_words_payload())
@router.put("/sensitive-words", response_model=ApiResponse)
def update_sensitive_words(payload: dict) -> ApiResponse:
data = save_sensitive_words_payload(payload)
return ApiResponse(message=f"已保存 {data['total']} 个敏感词", data=data)

27
domain-api/app/api/routes/settings.py
View File

@@ -5,12 +5,15 @@ from fastapi.responses import FileResponse
from app.core.files import settings_backup_root from app.core.files import settings_backup_root
from app.schemas.common import ApiResponse from app.schemas.common import ApiResponse
from app.services.juziseo_service import get_juziseo_status, login_juziseo
from app.services.settings_service import ( from app.services.settings_service import (
backup_current_settings, backup_current_settings,
export_settings_snapshot, export_settings_snapshot,
get_credentials_payload,
get_settings_payload, get_settings_payload,
import_settings_snapshot, import_settings_snapshot,
list_settings_backups, list_settings_backups,
update_credentials_payload,
update_settings_payload, update_settings_payload,
validate_settings_payload, validate_settings_payload,
) )
@@ -61,6 +64,30 @@ def get_settings_backups() -> ApiResponse:
return ApiResponse(data=list_settings_backups()) return ApiResponse(data=list_settings_backups())
@router.get("/settings/credentials", response_model=ApiResponse)
def get_settings_credentials() -> ApiResponse:
return ApiResponse(data=get_credentials_payload())
@router.put("/settings/credentials", response_model=ApiResponse)
def update_settings_credentials(payload: dict) -> ApiResponse:
return ApiResponse(message="credentials updated", data=update_credentials_payload(payload))
@router.get("/settings/juziseo/status", response_model=ApiResponse)
def get_settings_juziseo_status() -> ApiResponse:
return ApiResponse(data=get_juziseo_status())
@router.post("/settings/juziseo/login", response_model=ApiResponse)
def settings_juziseo_login(payload: dict) -> ApiResponse:
try:
data = login_juziseo(str(payload.get("email") or ""), str(payload.get("password") or ""))
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
return ApiResponse(message="桔子SEO登录成功", data=data)
@router.get("/settings/backups/download/{filename}") @router.get("/settings/backups/download/{filename}")
def download_settings_backup(filename: str): def download_settings_backup(filename: str):
path = settings_backup_root() / Path(filename).name path = settings_backup_root() / Path(filename).name

11
domain-api/app/core/config.py
View File

@@ -31,6 +31,17 @@ class Settings(BaseSettings):
worker_mode: str = "windows-local" worker_mode: str = "windows-local"
worker_service_name: str = "domaincheck-worker" worker_service_name: str = "domaincheck-worker"
api_service_name: str = "domaincheck-api" api_service_name: str = "domaincheck-api"
sync_agent_service_name: str = "domaincheck-sync-agent"
node_code: str = "overseas-control-01"
node_region: str = "overseas"
node_role: str = "control"
sync_push_enabled: bool = False
sync_source_region: str = "unknown"
sync_target_region: str = "overseas"
sync_target_api_base_url: str = ""
sync_shared_token: str = ""
sync_batch_size: int = 200
sync_poll_interval_seconds: int = 30
@field_validator("cors_origins", mode="before") @field_validator("cors_origins", mode="before")
@classmethod @classmethod

62
domain-api/app/core/files.py
View File

@@ -15,8 +15,11 @@ def read_json(relative_path: str, default: dict | list | None = None):
path = domain_root() / relative_path path = domain_root() / relative_path
if not path.exists(): if not path.exists():
return {} if default is None else default return {} if default is None else default
with path.open("r", encoding="utf-8") as handle: try:
return json.load(handle) with path.open("r", encoding="utf-8") as handle:
return json.load(handle)
except (json.JSONDecodeError, OSError):
return {} if default is None else default
def write_json(relative_path: str, payload) -> None: def write_json(relative_path: str, payload) -> None:
@@ -44,8 +47,11 @@ def read_runtime_json(filename: str, default: dict | list | None = None):
path = runtime_root() / filename path = runtime_root() / filename
if not path.exists(): if not path.exists():
return {} if default is None else default return {} if default is None else default
with path.open("r", encoding="utf-8") as handle: try:
return json.load(handle) with path.open("r", encoding="utf-8") as handle:
return json.load(handle)
except (json.JSONDecodeError, OSError):
return {} if default is None else default
def write_runtime_json(filename: str, payload) -> None: def write_runtime_json(filename: str, payload) -> None:
@@ -77,8 +83,11 @@ def load_export_records() -> list[dict]:
path = runtime_root() / "export_tasks.json" path = runtime_root() / "export_tasks.json"
if not path.exists(): if not path.exists():
return [] return []
with path.open("r", encoding="utf-8") as handle: try:
return json.load(handle) with path.open("r", encoding="utf-8") as handle:
return json.load(handle)
except (json.JSONDecodeError, OSError):
return []
def save_export_record(record: dict) -> None: def save_export_record(record: dict) -> None:
@@ -93,8 +102,11 @@ def load_import_records() -> list[dict]:
path = runtime_root() / "import_tasks.json" path = runtime_root() / "import_tasks.json"
if not path.exists(): if not path.exists():
return [] return []
with path.open("r", encoding="utf-8") as handle: try:
return json.load(handle) with path.open("r", encoding="utf-8") as handle:
return json.load(handle)
except (json.JSONDecodeError, OSError):
return []
def save_import_records(records: list[dict]) -> None: def save_import_records(records: list[dict]) -> None:
@@ -103,5 +115,39 @@ def save_import_records(records: list[dict]) -> None:
json.dump(records[:200], handle, ensure_ascii=False, indent=2, default=str) json.dump(records[:200], handle, ensure_ascii=False, indent=2, default=str)
def load_juming_records() -> list[dict]:
path = runtime_root() / "juming_tasks.json"
if not path.exists():
return []
try:
with path.open("r", encoding="utf-8") as handle:
return json.load(handle)
except (json.JSONDecodeError, OSError):
return []
def save_juming_records(records: list[dict]) -> None:
path = runtime_root() / "juming_tasks.json"
with path.open("w", encoding="utf-8") as handle:
json.dump(records[:200], handle, ensure_ascii=False, indent=2, default=str)
def load_detect_records() -> list[dict]:
path = runtime_root() / "detect_runs.json"
if not path.exists():
return []
try:
with path.open("r", encoding="utf-8") as handle:
return json.load(handle)
except (json.JSONDecodeError, OSError):
return []
def save_detect_records(records: list[dict]) -> None:
path = runtime_root() / "detect_runs.json"
with path.open("w", encoding="utf-8") as handle:
json.dump(records[:200], handle, ensure_ascii=False, indent=2, default=str)
def timestamp_filename(prefix: str, ext: str) -> str: def timestamp_filename(prefix: str, ext: str) -> str:
return f"{prefix}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.{ext}" return f"{prefix}_{datetime.now().strftime('%Y%m%d_%H%M%S')}.{ext}"

31
domain-api/app/main.py
View File

@@ -1,8 +1,22 @@
import threading
from fastapi import FastAPI from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.cors import CORSMiddleware
from app.api.routes import auth, dashboard, settings as settings_routes, imports, detect, domains, exports, logs, runtime from app.api.routes import auth, dashboard, settings as settings_routes, imports, detect, domains, exports, logs, runtime, juming, sensitive_words
from app.core.config import settings as app_settings from app.core.config import settings as app_settings
from app.services.cluster_runtime_service import ensure_runtime_schema, register_local_control_heartbeat
_heartbeat_stop_event = threading.Event()
def _control_heartbeat_loop() -> None:
while not _heartbeat_stop_event.is_set():
try:
register_local_control_heartbeat()
except Exception:
pass
_heartbeat_stop_event.wait(30)
app = FastAPI( app = FastAPI(
title="domainCheck API", title="domainCheck API",
@@ -19,6 +33,19 @@ app.add_middleware(
) )
@app.on_event("startup")
def on_startup() -> None:
ensure_runtime_schema()
register_local_control_heartbeat()
_heartbeat_stop_event.clear()
threading.Thread(target=_control_heartbeat_loop, name="control-heartbeat", daemon=True).start()
@app.on_event("shutdown")
def on_shutdown() -> None:
_heartbeat_stop_event.set()
@app.get("/health") @app.get("/health")
def health() -> dict: def health() -> dict:
return { return {
@@ -41,3 +68,5 @@ app.include_router(domains.router, prefix=app_settings.api_prefix)
app.include_router(exports.router, prefix=app_settings.api_prefix) app.include_router(exports.router, prefix=app_settings.api_prefix)
app.include_router(logs.router, prefix=app_settings.api_prefix) app.include_router(logs.router, prefix=app_settings.api_prefix)
app.include_router(runtime.router, prefix=app_settings.api_prefix) app.include_router(runtime.router, prefix=app_settings.api_prefix)
app.include_router(juming.router, prefix=app_settings.api_prefix)
app.include_router(sensitive_words.router, prefix=app_settings.api_prefix)

272
domain-api/app/services/cluster_runtime_service.py Normal file
View File

@@ -0,0 +1,272 @@
from __future__ import annotations
import json
import socket
from datetime import datetime, timedelta
from app.core.config import settings
from app.core.db import get_db
_RUNTIME_SCHEMA_SQL = """
CREATE TABLE IF NOT EXISTS detect_worker_nodes (
node_code VARCHAR(64) PRIMARY KEY,
region VARCHAR(32) NOT NULL DEFAULT 'unknown',
role VARCHAR(32) NOT NULL DEFAULT 'worker',
hostname VARCHAR(255) NOT NULL DEFAULT '',
ip VARCHAR(64) NOT NULL DEFAULT '',
status VARCHAR(32) NOT NULL DEFAULT 'unknown',
worker_version VARCHAR(32) NOT NULL DEFAULT '',
current_load INTEGER NOT NULL DEFAULT 0,
metadata_json JSONB,
last_heartbeat_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
update_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS detect_jobs (
id BIGSERIAL PRIMARY KEY,
job_code VARCHAR(64) NOT NULL UNIQUE,
source VARCHAR(64) NOT NULL DEFAULT 'manual',
plan_hash VARCHAR(128) NOT NULL DEFAULT '',
status VARCHAR(32) NOT NULL DEFAULT 'pending',
remark TEXT NOT NULL DEFAULT '',
created_by VARCHAR(64) NOT NULL DEFAULT '',
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
started_at TIMESTAMP,
finished_at TIMESTAMP
);
CREATE TABLE IF NOT EXISTS detect_job_items (
id BIGSERIAL PRIMARY KEY,
job_id BIGINT NOT NULL REFERENCES detect_jobs(id) ON DELETE CASCADE,
domain_id BIGINT NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'pending',
claimed_by VARCHAR(64) NOT NULL DEFAULT '',
claim_token VARCHAR(64) NOT NULL DEFAULT '',
lease_expires_at TIMESTAMP,
attempt_count INTEGER NOT NULL DEFAULT 0,
last_error TEXT NOT NULL DEFAULT '',
result_version VARCHAR(64) NOT NULL DEFAULT '',
started_at TIMESTAMP,
finished_at TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uq_detect_job_items_job_domain UNIQUE (job_id, domain_id)
);
CREATE INDEX IF NOT EXISTS idx_detect_job_items_status_lease
ON detect_job_items(status, lease_expires_at);
CREATE TABLE IF NOT EXISTS detect_run_events (
id BIGSERIAL PRIMARY KEY,
job_id BIGINT REFERENCES detect_jobs(id) ON DELETE SET NULL,
job_item_id BIGINT REFERENCES detect_job_items(id) ON DELETE SET NULL,
node_code VARCHAR(64) NOT NULL DEFAULT '',
event_type VARCHAR(64) NOT NULL DEFAULT '',
level VARCHAR(16) NOT NULL DEFAULT 'info',
message TEXT NOT NULL DEFAULT '',
payload_json JSONB,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX IF NOT EXISTS idx_detect_run_events_job_created
ON detect_run_events(job_id, created_at DESC);
CREATE TABLE IF NOT EXISTS detect_sync_records (
id BIGSERIAL PRIMARY KEY,
sync_type VARCHAR(32) NOT NULL DEFAULT '',
source_region VARCHAR(32) NOT NULL DEFAULT '',
target_region VARCHAR(32) NOT NULL DEFAULT '',
status VARCHAR(32) NOT NULL DEFAULT 'pending',
payload_json JSONB,
error_message TEXT NOT NULL DEFAULT '',
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
"""
def _resolve_local_ip() -> str:
try:
return socket.gethostbyname(socket.gethostname())
except Exception:
return ""
def _decode_json(value: object) -> dict:
if isinstance(value, dict):
return value
if value in (None, ""):
return {}
try:
return json.loads(value)
except Exception:
return {}
def ensure_runtime_schema() -> None:
with get_db() as conn:
conn.autocommit = False
with conn.cursor() as cur:
cur.execute(_RUNTIME_SCHEMA_SQL)
conn.commit()
def register_node_heartbeat(
*,
node_code: str,
region: str,
role: str,
status: str,
current_load: int = 0,
worker_version: str = "0.1.0",
metadata: dict | None = None,
) -> None:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO detect_worker_nodes (
node_code, region, role, hostname, ip, status, worker_version, current_load, metadata_json, last_heartbeat_at, update_time
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
ON CONFLICT (node_code) DO UPDATE SET
region = EXCLUDED.region,
role = EXCLUDED.role,
hostname = EXCLUDED.hostname,
ip = EXCLUDED.ip,
status = EXCLUDED.status,
worker_version = EXCLUDED.worker_version,
current_load = EXCLUDED.current_load,
metadata_json = EXCLUDED.metadata_json,
last_heartbeat_at = CURRENT_TIMESTAMP,
update_time = CURRENT_TIMESTAMP
""",
(
node_code,
region,
role,
socket.gethostname(),
_resolve_local_ip(),
status,
worker_version,
max(0, int(current_load or 0)),
json.dumps(metadata or {}, ensure_ascii=False),
),
)
conn.commit()
def register_local_control_heartbeat() -> None:
register_node_heartbeat(
node_code=settings.node_code,
region=settings.node_region,
role=settings.node_role,
status="online",
current_load=0,
metadata={
"service": "domain-api",
"api_host": settings.api_host,
"api_port": settings.api_port,
"worker_mode": settings.worker_mode,
"updated_at": datetime.now().isoformat(timespec="seconds"),
},
)
def _normalize_node_status(raw_status: str, last_heartbeat_at: datetime | None) -> str:
status = str(raw_status or "").strip() or "unknown"
if not last_heartbeat_at:
return status
now = datetime.now(last_heartbeat_at.tzinfo) if last_heartbeat_at.tzinfo else datetime.now()
age = now - last_heartbeat_at
if age > timedelta(minutes=5):
return "offline"
if age > timedelta(seconds=90):
return "stale"
return status
def get_cluster_snapshot() -> dict:
register_local_control_heartbeat()
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT node_code, region, role, hostname, ip, status, worker_version, current_load, metadata_json, last_heartbeat_at
FROM detect_worker_nodes
ORDER BY
CASE WHEN status = 'busy' THEN 0 WHEN status = 'online' THEN 1 ELSE 2 END,
region ASC,
role ASC,
node_code ASC
LIMIT 100
"""
)
rows = cur.fetchall()
cur.execute("SELECT count(*) FROM detect_jobs")
jobs_total = cur.fetchone()[0]
cur.execute("SELECT count(*) FROM detect_job_items WHERE status IN ('pending', 'claimed', 'running')")
active_items = cur.fetchone()[0]
nodes = [
{
"node_code": row[0],
"region": row[1],
"role": row[2],
"hostname": row[3],
"ip": row[4],
"status": _normalize_node_status(row[5], row[9]),
"worker_version": row[6],
"current_load": row[7],
"metadata": _decode_json(row[8]),
"last_heartbeat_at": row[9].isoformat(sep=" ", timespec="seconds") if row[9] else "",
}
for row in rows
]
status_counts: dict[str, int] = {}
role_counts: dict[str, int] = {}
region_counts: dict[str, int] = {}
busy_nodes: list[str] = []
stale_nodes: list[str] = []
offline_nodes: list[str] = []
online_worker_nodes = 0
online_control_nodes = 0
for node in nodes:
node_status = str(node.get("status") or "unknown")
node_role = str(node.get("role") or "unknown")
node_region = str(node.get("region") or "unknown")
node_code = str(node.get("node_code") or "")
status_counts[node_status] = status_counts.get(node_status, 0) + 1
role_counts[node_role] = role_counts.get(node_role, 0) + 1
region_counts[node_region] = region_counts.get(node_region, 0) + 1
if node_status == "busy":
busy_nodes.append(node_code)
if node_status == "stale":
stale_nodes.append(node_code)
if node_status == "offline":
offline_nodes.append(node_code)
if node_role == "worker" and node_status in {"online", "busy"}:
online_worker_nodes += 1
if node_role == "control" and node_status in {"online", "busy"}:
online_control_nodes += 1
return {
"nodes": nodes,
"nodes_total": len(nodes),
"jobs_total": jobs_total,
"active_job_items": active_items,
"summary": {
"status_counts": status_counts,
"role_counts": role_counts,
"region_counts": region_counts,
"busy_nodes": busy_nodes,
"stale_nodes": stale_nodes,
"offline_nodes": offline_nodes,
"online_worker_nodes": online_worker_nodes,
"online_control_nodes": online_control_nodes,
},
}

558
domain-api/app/services/detect_job_service.py Normal file
View File

@@ -0,0 +1,558 @@
from __future__ import annotations
import json
from datetime import datetime
import math
from uuid import uuid4
from app.core.config import settings
from app.core.db import get_db
ACTIVE_JOB_STATUSES = ("pending", "running")
def _selection_sql() -> str:
return """
SELECT id
FROM domains
WHERE
detect_status IN (0, 4)
OR (use_status = 0 AND detect_status = 1 AND register_status = 3 AND expire_date < CURRENT_DATE)
ORDER BY id ASC
LIMIT %s
"""
def _format_time(value: datetime | None) -> str:
return value.isoformat(sep=" ", timespec="seconds") if value else ""
def _decode_payload(value: object) -> dict:
if isinstance(value, dict):
return value
if value in (None, ""):
return {}
try:
return json.loads(value)
except Exception:
return {}
def _extract_current_cycle_events(events: list[dict]) -> tuple[str, list[dict]]:
if not events:
return "", []
latest_payload = events[0].get("payload") or {}
latest_cycle_token = str(latest_payload.get("cycle_token") or "").strip()
if latest_cycle_token:
current_cycle = []
for event in events:
payload = event.get("payload") or {}
event_cycle = str(payload.get("cycle_token") or "").strip()
if event_cycle == latest_cycle_token:
current_cycle.append(event)
return latest_cycle_token, current_cycle
cycle_token = ""
anchor_index = -1
for index, event in enumerate(events):
payload = event.get("payload") or {}
event_type = str(event.get("event_type") or "")
candidate = str(payload.get("cycle_token") or "").strip()
if candidate and event_type in {"job_dispatch_sent", "job_dispatch_requested", "job_dispatch_failed", "job_dispatch_rejected"}:
cycle_token = candidate
anchor_index = index
break
if not cycle_token:
return "", events
current_cycle = []
for index, event in enumerate(events[: anchor_index + 1]):
payload = event.get("payload") or {}
event_cycle = str(payload.get("cycle_token") or "").strip()
if index == anchor_index or event_cycle == cycle_token:
current_cycle.append(event)
return cycle_token, current_cycle
def _fetch_job_summary(cur, job_row, event_limit: int = 20) -> dict:
job_id = job_row[0]
cur.execute(
"""
SELECT status, count(*)
FROM detect_job_items
WHERE job_id = %s
GROUP BY status
""",
(job_id,),
)
counts = {status: int(count) for status, count in cur.fetchall()}
cur.execute(
"""
SELECT COALESCE(NULLIF(claimed_by, ''), 'unassigned') AS node_code,
status,
count(*)
FROM detect_job_items
WHERE job_id = %s
GROUP BY COALESCE(NULLIF(claimed_by, ''), 'unassigned'), status
ORDER BY node_code ASC, status ASC
""",
(job_id,),
)
node_buckets: dict[str, dict] = {}
for node_code, status, count in cur.fetchall():
bucket = node_buckets.setdefault(
node_code,
{
"node_code": node_code,
"items_total": 0,
"items_pending": 0,
"items_claimed": 0,
"items_running": 0,
"items_completed": 0,
"items_blacklisted": 0,
"items_failed": 0,
},
)
field_name = f"items_{status}"
if field_name in bucket:
bucket[field_name] += int(count)
bucket["items_total"] += int(count)
cur.execute(
"""
SELECT node_code, event_type, level, message, payload_json, created_at
FROM detect_run_events
WHERE job_id = %s
ORDER BY created_at DESC, id DESC
LIMIT %s
""",
(job_id, max(1, int(event_limit or 20))),
)
events = [
{
"node_code": item[0],
"event_type": item[1],
"level": item[2],
"message": item[3],
"payload": _decode_payload(item[4]),
"created_at": _format_time(item[5]),
}
for item in cur.fetchall()
]
cycle_token, current_cycle_events = _extract_current_cycle_events(events)
total = sum(counts.values())
terminal = int(counts.get("completed", 0)) + int(counts.get("blacklisted", 0)) + int(counts.get("failed", 0))
return {
"job_id": job_id,
"job_code": job_row[1],
"source": job_row[2],
"status": job_row[3],
"created_by": job_row[4],
"created_at": _format_time(job_row[5]),
"started_at": _format_time(job_row[6]),
"finished_at": _format_time(job_row[7]),
"items_total": total,
"items_pending": int(counts.get("pending", 0)),
"items_claimed": int(counts.get("claimed", 0)),
"items_running": int(counts.get("running", 0)),
"items_completed": int(counts.get("completed", 0)),
"items_blacklisted": int(counts.get("blacklisted", 0)),
"items_failed": int(counts.get("failed", 0)),
"items_terminal": terminal,
"progress_percent": round((terminal / total) * 100, 2) if total else 0,
"node_stats": list(node_buckets.values()),
"recent_events": events,
"latest_event": events[0] if events else None,
"current_cycle_token": cycle_token,
"current_cycle_events": current_cycle_events,
"latest_cycle_event": current_cycle_events[0] if current_cycle_events else None,
}
def get_detect_job_summary(job_id: int, event_limit: int = 20) -> dict | None:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at
FROM detect_jobs
WHERE id = %s
LIMIT 1
""",
(int(job_id),),
)
row = cur.fetchone()
if not row:
return None
return _fetch_job_summary(cur, row, event_limit=event_limit)
def get_active_detect_job_summary(event_limit: int = 20) -> dict | None:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at
FROM detect_jobs
WHERE status IN ('pending', 'running')
ORDER BY created_at DESC, id DESC
LIMIT 1
"""
)
row = cur.fetchone()
if not row:
return None
return _fetch_job_summary(cur, row, event_limit=event_limit)
def list_detect_jobs(limit: int = 20) -> list[dict]:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, job_code, source, status, created_by, created_at, started_at, finished_at
FROM detect_jobs
ORDER BY created_at DESC, id DESC
LIMIT %s
""",
(max(1, min(int(limit or 20), 100)),),
)
rows = cur.fetchall()
return [_fetch_job_summary(cur, row, event_limit=10) for row in rows]
def get_detect_queue_health(window_minutes: int = 15) -> dict:
window_minutes = max(5, min(int(window_minutes or 15), 120))
active_job = get_active_detect_job_summary(event_limit=10)
if not active_job:
return {
"window_minutes": window_minutes,
"has_active_job": False,
"job": None,
"queue": {
"items_total": 0,
"pending": 0,
"claimed": 0,
"running": 0,
"completed": 0,
"blacklisted": 0,
"failed": 0,
"terminal": 0,
"terminal_percent": 0,
"oldest_pending_at": "",
"oldest_pending_age_minutes": 0,
"nearest_lease_expiry_at": "",
"overdue_leases": 0,
"expiring_soon_leases": 0,
},
"throughput": {
"processed_recent": 0,
"processed_per_minute": 0,
"completed_recent": 0,
"blacklisted_recent": 0,
"failed_recent": 0,
},
"nodes": [],
}
job_id = int(active_job["job_id"])
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT
MIN(create_time) FILTER (WHERE status = 'pending') AS oldest_pending_at,
MIN(lease_expires_at) FILTER (WHERE status IN ('claimed', 'running') AND lease_expires_at IS NOT NULL) AS nearest_lease_expiry_at,
COUNT(*) FILTER (WHERE status IN ('claimed', 'running') AND lease_expires_at IS NOT NULL AND lease_expires_at < CURRENT_TIMESTAMP) AS overdue_leases,
COUNT(*) FILTER (
WHERE status IN ('claimed', 'running')
AND lease_expires_at IS NOT NULL
AND lease_expires_at >= CURRENT_TIMESTAMP
AND lease_expires_at < CURRENT_TIMESTAMP + interval '5 minutes'
) AS expiring_soon_leases
FROM detect_job_items
WHERE job_id = %s
""",
(job_id,),
)
lease_row = cur.fetchone()
cur.execute(
"""
SELECT
COALESCE(NULLIF(node_code, ''), 'unknown') AS node_code,
COUNT(*) AS processed_recent,
COUNT(*) FILTER (WHERE event_type = 'domain_completed') AS completed_recent,
COUNT(*) FILTER (WHERE event_type = 'domain_blacklisted') AS blacklisted_recent,
COUNT(*) FILTER (WHERE event_type = 'domain_failed') AS failed_recent
FROM detect_run_events
WHERE job_id = %s
AND event_type IN ('domain_completed', 'domain_blacklisted', 'domain_failed')
AND created_at >= CURRENT_TIMESTAMP - (%s || ' minutes')::interval
GROUP BY COALESCE(NULLIF(node_code, ''), 'unknown')
ORDER BY processed_recent DESC, node_code ASC
""",
(job_id, window_minutes),
)
throughput_rows = cur.fetchall()
oldest_pending_at = _format_time(lease_row[0]) if lease_row and lease_row[0] else ""
nearest_lease_expiry_at = _format_time(lease_row[1]) if lease_row and lease_row[1] else ""
oldest_pending_age_minutes = 0
if lease_row and lease_row[0]:
oldest_pending_age_minutes = max(0, int((datetime.now() - lease_row[0]).total_seconds() // 60))
node_map = {
str(item.get("node_code") or "unknown"): {
"node_code": str(item.get("node_code") or "unknown"),
"items_total": int(item.get("items_total", 0) or 0),
"items_pending": int(item.get("items_pending", 0) or 0),
"items_claimed": int(item.get("items_claimed", 0) or 0),
"items_running": int(item.get("items_running", 0) or 0),
"items_completed": int(item.get("items_completed", 0) or 0),
"items_blacklisted": int(item.get("items_blacklisted", 0) or 0),
"items_failed": int(item.get("items_failed", 0) or 0),
"processed_recent": 0,
"processed_per_minute": 0,
"completed_recent": 0,
"blacklisted_recent": 0,
"failed_recent": 0,
}
for item in active_job.get("node_stats") or []
}
total_processed_recent = 0
total_completed_recent = 0
total_blacklisted_recent = 0
total_failed_recent = 0
for row in throughput_rows:
node_code = str(row[0] or "unknown")
bucket = node_map.setdefault(
node_code,
{
"node_code": node_code,
"items_total": 0,
"items_pending": 0,
"items_claimed": 0,
"items_running": 0,
"items_completed": 0,
"items_blacklisted": 0,
"items_failed": 0,
"processed_recent": 0,
"processed_per_minute": 0,
"completed_recent": 0,
"blacklisted_recent": 0,
"failed_recent": 0,
},
)
processed_recent = int(row[1] or 0)
completed_recent = int(row[2] or 0)
blacklisted_recent = int(row[3] or 0)
failed_recent = int(row[4] or 0)
bucket["processed_recent"] = processed_recent
bucket["processed_per_minute"] = round(processed_recent / window_minutes, 2)
bucket["completed_recent"] = completed_recent
bucket["blacklisted_recent"] = blacklisted_recent
bucket["failed_recent"] = failed_recent
total_processed_recent += processed_recent
total_completed_recent += completed_recent
total_blacklisted_recent += blacklisted_recent
total_failed_recent += failed_recent
nodes = sorted(
node_map.values(),
key=lambda item: (
-int(item.get("processed_recent", 0) or 0),
-int(item.get("items_running", 0) or 0),
str(item.get("node_code") or ""),
),
)
items_total = int(active_job.get("items_total", 0) or 0)
terminal = int(active_job.get("items_terminal", 0) or 0)
return {
"window_minutes": window_minutes,
"has_active_job": True,
"job": {
"job_id": job_id,
"job_code": active_job.get("job_code", ""),
"status": active_job.get("status", ""),
"progress_percent": active_job.get("progress_percent", 0),
},
"queue": {
"items_total": items_total,
"pending": int(active_job.get("items_pending", 0) or 0),
"claimed": int(active_job.get("items_claimed", 0) or 0),
"running": int(active_job.get("items_running", 0) or 0),
"completed": int(active_job.get("items_completed", 0) or 0),
"blacklisted": int(active_job.get("items_blacklisted", 0) or 0),
"failed": int(active_job.get("items_failed", 0) or 0),
"terminal": terminal,
"terminal_percent": round((terminal / items_total) * 100, 2) if items_total else 0,
"oldest_pending_at": oldest_pending_at,
"oldest_pending_age_minutes": oldest_pending_age_minutes,
"nearest_lease_expiry_at": nearest_lease_expiry_at,
"overdue_leases": int(lease_row[2] or 0) if lease_row else 0,
"expiring_soon_leases": int(lease_row[3] or 0) if lease_row else 0,
},
"throughput": {
"processed_recent": total_processed_recent,
"processed_per_minute": round(total_processed_recent / window_minutes, 2),
"completed_recent": total_completed_recent,
"blacklisted_recent": total_blacklisted_recent,
"failed_recent": total_failed_recent,
},
"nodes": nodes,
}
def get_detect_capacity_plan(*, queue_health: dict | None = None, online_worker_nodes: int = 0, target_finish_hours: int = 6) -> dict:
queue_health = queue_health or get_detect_queue_health(window_minutes=15)
target_finish_hours = max(1, min(int(target_finish_hours or 6), 72))
online_worker_nodes = max(0, int(online_worker_nodes or 0))
if not queue_health.get("has_active_job"):
return {
"has_active_job": False,
"online_worker_nodes": online_worker_nodes,
"target_finish_hours": target_finish_hours,
"estimated_hours_remaining": 0,
"recommended_total_workers": max(1, online_worker_nodes),
"recommended_additional_workers": 0,
"current_processed_per_hour": 0,
"pending_items": 0,
"terminal_items": 0,
"summary": "当前没有活跃任务，无需扩容建议。",
}
queue = queue_health.get("queue") or {}
throughput = queue_health.get("throughput") or {}
pending_items = int(queue.get("pending", 0) or 0)
claimed_items = int(queue.get("claimed", 0) or 0)
running_items = int(queue.get("running", 0) or 0)
remaining_items = pending_items + claimed_items + running_items
current_processed_per_hour = round(float(throughput.get("processed_per_minute", 0) or 0) * 60, 2)
estimated_hours_remaining = round((remaining_items / current_processed_per_hour), 2) if current_processed_per_hour > 0 else 0
recommended_total_workers = max(1, online_worker_nodes or 1)
recommended_additional_workers = 0
if remaining_items > 0 and target_finish_hours > 0:
required_per_hour = remaining_items / target_finish_hours
if current_processed_per_hour > 0 and max(1, online_worker_nodes) > 0:
per_worker_per_hour = current_processed_per_hour / max(1, online_worker_nodes)
recommended_total_workers = max(1, int(math.ceil(required_per_hour / per_worker_per_hour)))
recommended_additional_workers = max(0, recommended_total_workers - online_worker_nodes)
elif remaining_items > 0:
recommended_total_workers = max(1, online_worker_nodes or 1)
recommended_additional_workers = 0
summary = (
f"当前在线 Worker {online_worker_nodes} 台，近窗吞吐约 {current_processed_per_hour} 项/小时，"
f"剩余待处理约 {remaining_items} 项，预计还需 {estimated_hours_remaining} 小时。"
)
if recommended_additional_workers > 0:
summary = (
f"{summary} 若希望在 {target_finish_hours} 小时内收敛，建议总 Worker 数达到 "
f"{recommended_total_workers} 台，至少再加 {recommended_additional_workers} 台。"
)
else:
summary = f"{summary} 按当前目标 {target_finish_hours} 小时看，现有 Worker 数量暂时够用。"
return {
"has_active_job": True,
"online_worker_nodes": online_worker_nodes,
"target_finish_hours": target_finish_hours,
"estimated_hours_remaining": estimated_hours_remaining,
"recommended_total_workers": recommended_total_workers,
"recommended_additional_workers": recommended_additional_workers,
"current_processed_per_hour": current_processed_per_hour,
"pending_items": pending_items,
"remaining_items": remaining_items,
"terminal_items": int(queue.get("terminal", 0) or 0),
"summary": summary,
}
def append_detect_job_event(
job_id: int,
*,
event_type: str,
message: str,
level: str = "info",
payload: dict | None = None,
node_code: str | None = None,
) -> None:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json)
VALUES (%s, %s, %s, %s, %s, %s::jsonb)
""",
(
int(job_id),
node_code or settings.node_code,
str(event_type or "").strip() or "info",
str(level or "info").strip() or "info",
str(message or "").strip(),
json.dumps(payload or {}, ensure_ascii=False),
),
)
conn.commit()
def create_detect_job_if_needed(limit: int = 1000, created_by: str = "system") -> dict | None:
existing = get_active_detect_job_summary()
if existing:
return existing
with get_db() as conn:
conn.autocommit = False
with conn.cursor() as cur:
cur.execute(_selection_sql(), (max(1, int(limit or 1000)),))
domain_ids = [row[0] for row in cur.fetchall()]
if not domain_ids:
conn.rollback()
return None
job_code = f"detect-{datetime.now().strftime('%Y%m%d%H%M%S')}-{uuid4().hex[:6]}"
plan_hash = uuid4().hex
cur.execute(
"""
INSERT INTO detect_jobs (job_code, source, plan_hash, status, remark, created_by)
VALUES (%s, %s, %s, 'pending', %s, %s)
RETURNING id
""",
(
job_code,
"api-start",
plan_hash,
f"API 创建检测任务，待检测域名 {len(domain_ids)} 个",
created_by,
),
)
job_id = cur.fetchone()[0]
item_rows = [(job_id, domain_id) for domain_id in domain_ids]
cur.executemany(
"""
INSERT INTO detect_job_items (job_id, domain_id, status)
VALUES (%s, %s, 'pending')
ON CONFLICT (job_id, domain_id) DO NOTHING
""",
item_rows,
)
cur.execute(
"""
INSERT INTO detect_run_events (job_id, node_code, event_type, level, message, payload_json)
VALUES (%s, %s, %s, %s, %s, %s::jsonb)
""",
(
job_id,
settings.node_code,
"job_created",
"info",
f"创建检测任务 {job_code}，共 {len(domain_ids)} 个域名",
'{"count": %s}' % len(domain_ids),
),
)
conn.commit()
return get_active_detect_job_summary()

425
domain-api/app/services/detect_run_service.py Normal file
View File

@@ -0,0 +1,425 @@
from __future__ import annotations
from datetime import datetime
from uuid import uuid4
from app.core.files import load_detect_records, save_detect_records, tail_lines
_MAX_LOG_LINES = 240
_LOG_TAIL_LINES = 1200
_ACTIVE_STATUSES = {"starting", "running", "stopping"}
_TIMESTAMP_FORMATS = ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S")
def _now() -> str:
return datetime.now().isoformat(sep=" ", timespec="seconds")
def _save(records: list[dict]) -> None:
save_detect_records(records)
def _load() -> list[dict]:
return load_detect_records()
def _capture_worker_logs(max_lines: int = _LOG_TAIL_LINES) -> list[str]:
lines = tail_lines("detect_worker.log", max_lines=max_lines)
return lines[-max_lines:]
def _parse_time(raw: str | None) -> datetime | None:
if not raw:
return None
text = str(raw).strip()
if not text:
return None
try:
return datetime.fromisoformat(text)
except ValueError:
pass
for fmt in _TIMESTAMP_FORMATS:
try:
return datetime.strptime(text, fmt)
except ValueError:
continue
return None
def _extract_log_time(line: str) -> datetime | None:
if len(line) < 19:
return None
candidates = [line[:26], line[:19]]
for candidate in candidates:
for fmt in _TIMESTAMP_FORMATS:
if len(candidate) != len(datetime.now().strftime(fmt)):
continue
try:
return datetime.strptime(candidate, fmt)
except ValueError:
continue
return None
def _filter_logs_since(lines: list[str], started_at: str | None) -> list[str]:
if not started_at:
return lines[-_MAX_LOG_LINES:]
started_time = _parse_time(started_at)
if not started_time:
return lines[-_MAX_LOG_LINES:]
filtered = [line for line in lines if (_extract_log_time(line) or started_time) >= started_time]
if filtered:
return filtered[-_MAX_LOG_LINES:]
return lines[-_MAX_LOG_LINES:]
def _merge_logs(existing: list[str] | None, current: list[str]) -> list[str]:
merged = list(existing or [])
for line in current:
if line not in merged[-40:]:
merged.append(line)
continue
if not merged or merged[-1] != line:
merged.append(line)
return merged[-_MAX_LOG_LINES:]
def _find_active(records: list[dict]) -> dict | None:
return next((item for item in records if item.get("status") in _ACTIVE_STATUSES), None)
def _same_session(active: dict | None, runtime: dict) -> bool:
if not active:
return False
runtime_started_at = str(runtime.get("latest_start_time", "") or "").strip()
active_started_at = str(active.get("started_at", "") or "").strip()
if not runtime_started_at or not active_started_at:
return True
runtime_started = _parse_time(runtime_started_at)
active_started = _parse_time(active_started_at)
if not runtime_started or not active_started:
return True
return abs((runtime_started - active_started).total_seconds()) < 3
def _latest_matching_log(log_lines: list[str], keywords: tuple[str, ...]) -> str:
for line in reversed(log_lines or []):
if any(keyword in line for keyword in keywords):
return line
return ""
def _sync_phase_history(record: dict, phase_label: str, phase_detail: str) -> None:
history = list(record.get("phase_history") or [])
current = {
"at": _now(),
"label": phase_label or "-",
"detail": phase_detail or "",
}
if history:
latest = history[-1]
if latest.get("label") == current["label"] and latest.get("detail") == current["detail"]:
return
history.append(current)
record["phase_history"] = history[-20:]
def _phase_from_runtime(status: str, runtime: dict, progress: dict, log_lines: list[str], active_job: dict | None = None) -> tuple[str, str]:
active_job = active_job or {}
if status == "starting":
return "启动中", "正在拉起检测服务并等待 Worker 就绪"
if status == "stopping":
return "停止中", "已发送停止请求，等待 Worker 退出并归档日志"
if status == "completed":
terminal = int(active_job.get("items_terminal", 0) or 0)
return "已完成", f"本轮检测已自然完成，本次累计处理 {terminal} 个任务项"
if status == "partial_failed":
failed = int(active_job.get("items_failed", 0) or 0)
terminal = int(active_job.get("items_terminal", 0) or 0)
return "部分失败", f"本轮检测已结束，其中失败 {failed} 个，累计处理 {terminal} 个任务项"
if status == "failed":
return "失败", runtime.get("message") or "Worker 异常退出，请检查日志"
if status == "stopped":
return "已停止", runtime.get("message") or "检测服务已停止"
if runtime.get("running"):
proxy_runtime_state = str(runtime.get("proxy_runtime_state", "") or "").strip()
proxy_runtime_detail = str(runtime.get("proxy_runtime_detail", "") or "").strip()
running = (progress or {}).get("running", 0)
pending = (progress or {}).get("pending", 0)
phase_log = _latest_matching_log(
log_lines,
(
"Connection refused",
"Read timed out",
"ConnectTimeout",
"HTTPSConnectionPool",
"WaybackDetector",
"域名检测任务完成",
"当前批次检测完成",
"当前实际线程数量",
"开始创建线程",
"获取到",
"开始检测，刷新代理池",
"刷新代理池",
"没有需要检测的域名",
"开始执行域名检测任务",
),
)
if proxy_runtime_state == "blocked_no_proxy":
return "等待代理", proxy_runtime_detail or "代理池当前无可用代理，且未允许直连"
if proxy_runtime_state == "degraded_direct":
return "降级直连", proxy_runtime_detail or "代理池暂无可用代理，当前使用直连继续执行"
if "外部依赖异常，步骤降级继续执行" in phase_log:
return "外部站点异常", "外部依赖当前波动，系统已按降级策略继续执行并保留人工复核"
if "WaybackDetector" in phase_log or "web.archive.org" in phase_log:
return "外部站点异常", "时光机依赖当前访问异常，任务仍在继续，建议关注网络或代理策略"
if any(keyword in phase_log for keyword in ("Connection refused", "Read timed out", "ConnectTimeout", "HTTPSConnectionPool")):
return "网络波动", phase_log
if "域名检测任务完成" in phase_log:
return "完成归档", "本轮检测已处理完成，正在等待下一轮任务或归档最终日志"
if "当前批次检测完成" in phase_log:
return "批次完成", phase_log
if "当前实际线程数量" in phase_log or running > 0:
if running > 0:
return "检测中", f"Worker 正在处理 {running} 个检测任务"
return "检测中", phase_log
if "开始创建线程" in phase_log:
return "建线程中", phase_log
if "获取到" in phase_log:
return "取任务中", phase_log
if "刷新代理池" in phase_log:
return "刷新代理池", phase_log
if "没有需要检测的域名" in phase_log:
return "空闲等待", "Worker 在线，当前没有待检测域名"
if "开始执行域名检测任务" in phase_log:
return "准备检测", phase_log
if running > 0:
return "检测中", f"Worker 正在处理 {running} 个检测任务"
if pending > 0:
return "取任务中", f"Worker 在线，等待或领取待检测域名，当前剩余 {pending} 个"
return "运行中", "Worker 在线，当前没有活跃检测任务"
return "已停止", runtime.get("message") or "检测服务已停止"
def _sync_record(
record: dict,
*,
status: str,
message: str,
runtime: dict,
progress: dict,
settings_summary: dict,
log_lines: list[str],
active_job: dict | None = None,
) -> dict:
record["status"] = status
record["message"] = message
record["runtime"] = runtime
record["progress"] = progress
record["settings_summary"] = settings_summary
record["updated_at"] = _now()
if not record.get("started_at"):
record["started_at"] = runtime.get("latest_start_time") or record["updated_at"]
session_logs = _merge_logs(record.get("logs"), _filter_logs_since(log_lines, record.get("started_at")))
record["logs"] = session_logs
record["active_job"] = active_job or {}
record["phase_label"], record["phase_detail"] = _phase_from_runtime(status, runtime, progress, session_logs, active_job)
_sync_phase_history(record, record.get("phase_label", ""), record.get("phase_detail", ""))
if status in {"stopped", "failed", "completed", "partial_failed"} and not record.get("completed_at"):
record["completed_at"] = _now()
if status in _ACTIVE_STATUSES:
record["completed_at"] = ""
return record
def create_detect_run_snapshot(message: str, runtime: dict, progress: dict, settings_summary: dict) -> dict:
records = _load()
active = _find_active(records)
current_logs = _capture_worker_logs()
if active:
if active.get("status") == "stopping" and runtime.get("running"):
active["status"] = "running"
_sync_record(
active,
status=active.get("status", "starting"),
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
)
_save(records)
return dict(active)
initial_started_at = runtime.get("latest_start_time") or _now()
record = {
"run_id": uuid4().hex,
"status": "starting",
"message": message,
"created_at": _now(),
"updated_at": _now(),
"started_at": initial_started_at,
"completed_at": "",
"runtime": runtime,
"progress": progress,
"settings_summary": settings_summary,
"phase_label": "",
"phase_detail": "",
"phase_history": [],
"logs": [],
}
_sync_record(
record,
status="starting",
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
)
records.insert(0, record)
_save(records)
return dict(record)
def finalize_detect_run(message: str, runtime: dict, progress: dict, settings_summary: dict, active_job: dict | None = None) -> dict | None:
records = _load()
target = _find_active(records)
if not target:
return None
final_status = "stopped" if target.get("status") == "stopping" else "failed"
_sync_record(
target,
status=final_status,
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=_capture_worker_logs(),
active_job=active_job,
)
_save(records)
return dict(target)
def sync_detect_runs(runtime: dict, progress: dict, settings_summary: dict, active_job: dict | None = None) -> list[dict]:
records = _load()
active = _find_active(records)
current_logs = _capture_worker_logs()
active_job = active_job or {}
runtime_detecting = bool(runtime.get("detecting", False))
active_job_status = str(active_job.get("status", "") or "").strip()
active_job_open = active_job_status in {"pending", "running"}
execution_active = runtime_detecting or active_job_open or int((progress or {}).get("running", 0) or 0) > 0
if runtime.get("running") and execution_active:
if active and not _same_session(active, runtime):
_sync_record(
active,
status="stopped",
message="检测服务已重启，上一轮会话已归档",
runtime=active.get("runtime") or runtime,
progress=active.get("progress") or progress,
settings_summary=active.get("settings_summary") or settings_summary,
log_lines=current_logs,
active_job=active.get("active_job") or active_job,
)
active = None
if active:
next_status = "running" if active.get("status") != "stopping" else "stopping"
_sync_record(
active,
status=next_status,
message=runtime.get("message") or active.get("message") or "检测服务运行中",
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
active_job=active_job,
)
else:
started_at = runtime.get("latest_start_time") or _now()
record = {
"run_id": uuid4().hex,
"status": "running",
"message": runtime.get("message") or "检测服务运行中",
"created_at": _now(),
"updated_at": _now(),
"started_at": started_at,
"completed_at": "",
"runtime": runtime,
"progress": progress,
"settings_summary": settings_summary,
"phase_label": "",
"phase_detail": "",
"phase_history": [],
"logs": [],
"active_job": active_job,
}
_sync_record(
record,
status="running",
message=record["message"],
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
active_job=active_job,
)
records.insert(0, record)
elif active:
if runtime.get("running") and not execution_active:
if active.get("status") == "stopping":
final_status = "stopped"
final_message = runtime.get("message") or "检测任务已停止，Worker 保持待命"
elif active_job_status == "partial_failed":
final_status = "partial_failed"
final_message = "检测任务已结束，存在部分失败项"
elif active_job_status == "failed":
final_status = "failed"
final_message = "检测任务已结束，任务结果为失败"
else:
final_status = "completed"
final_message = "检测任务已自然完成，Worker 保持待命"
else:
final_status = "stopped" if active.get("status") == "stopping" else "failed"
final_message = runtime.get("message") or ("检测服务已停止" if final_status == "stopped" else "检测服务异常退出")
_sync_record(
active,
status=final_status,
message=final_message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=current_logs,
active_job=active_job,
)
if records:
records[0]["logs"] = _merge_logs(
records[0].get("logs"),
_filter_logs_since(current_logs, records[0].get("started_at")),
)
_save(records)
return records
def mark_detect_run_stopping(message: str, runtime: dict, progress: dict, settings_summary: dict) -> dict | None:
records = _load()
target = _find_active(records)
if not target:
return None
_sync_record(
target,
status="stopping",
message=message,
runtime=runtime,
progress=progress,
settings_summary=settings_summary,
log_lines=_capture_worker_logs(),
active_job=target.get("active_job") or {},
)
_save(records)
return dict(target)

356
domain-api/app/services/detect_service.py
View File

@@ -1,15 +1,275 @@
from __future__ import annotations from __future__ import annotations
import json
import re
from datetime import datetime, timezone from datetime import datetime, timezone
from pathlib import Path from pathlib import Path
from app.core.db import get_db from app.core.db import get_db
from app.core.files import tail_lines from app.core.files import tail_lines
from app.core.redis_client import get_redis
from app.services.runtime_settings_service import get_runtime_settings from app.services.runtime_settings_service import get_runtime_settings
from app.services.settings_service import get_settings_payload from app.services.detect_run_service import sync_detect_runs
from app.services.detect_job_service import get_active_detect_job_summary
from app.services.settings_service import get_settings_payload, resolve_thread_count
from app.services.sync_record_service import append_detect_result_projection_if_changed
from app.services.worker_control_service import detect_worker_runtime from app.services.worker_control_service import detect_worker_runtime
_PROXY_COUNT_RE = re.compile(r"当前可用代理数[:：]\s*(\d+)")
_THREAD_COUNT_RE = re.compile(r"当前实际线程数量[:：]\s*(\d+)\s*/\s*(\d+)")
_RUNTIME_STATE_KEY = "domain_tool:detect_runtime_state"
_TIMESTAMP_FORMATS = ("%Y-%m-%d %H:%M:%S.%f", "%Y-%m-%d %H:%M:%S")
def _extract_dependency_alerts(lines: list[str]) -> list[dict]:
alerts: list[dict] = []
recent_lines = lines[-120:] if lines else []
degraded_line = ""
for line in reversed(recent_lines):
if "外部依赖异常，步骤降级继续执行" in line:
degraded_line = line
alerts.append(
{
"kind": "dependency_degraded",
"level": "warning",
"title": "外部依赖降级继续",
"detail": line,
}
)
break
if degraded_line:
return alerts
for line in reversed(recent_lines):
if "WaybackDetector" in line or "web.archive.org" in line:
alerts.append(
{
"kind": "wayback",
"level": "warning",
"title": "时光机依赖异常",
"detail": line,
}
)
break
for line in reversed(recent_lines):
if any(keyword in line for keyword in ("HTTPSConnectionPool", "Connection refused", "Read timed out", "ConnectTimeout")):
alerts.append(
{
"kind": "network",
"level": "warning",
"title": "外部网络波动",
"detail": line,
}
)
break
return alerts
def _extract_available_proxy_count(lines: list[str]) -> int:
for line in reversed(lines):
match = _PROXY_COUNT_RE.search(line)
if match:
return int(match.group(1))
return 0
def _extract_active_thread_snapshot(lines: list[str]) -> dict:
for line in reversed(lines):
match = _THREAD_COUNT_RE.search(line)
if match:
return {
"active": int(match.group(1)),
"max": int(match.group(2)),
}
return {"active": 0, "max": 0}
def _parse_time(raw: str | None) -> datetime | None:
if not raw:
return None
text = str(raw).strip()
if not text:
return None
try:
return datetime.fromisoformat(text)
except ValueError:
pass
for fmt in _TIMESTAMP_FORMATS:
try:
return datetime.strptime(text, fmt)
except ValueError:
continue
return None
def _extract_log_time(line: str) -> datetime | None:
if len(line) < 19:
return None
candidates = [line[:26], line[:19]]
for candidate in candidates:
for fmt in _TIMESTAMP_FORMATS:
if len(candidate) != len(datetime.now().strftime(fmt)):
continue
try:
return datetime.strptime(candidate, fmt)
except ValueError:
continue
return None
def _filter_lines_since(lines: list[str], started_at: str | None) -> list[str]:
started_time = _parse_time(started_at)
if not started_time:
return lines
filtered = [line for line in lines if (_extract_log_time(line) or started_time) >= started_time]
return filtered or lines
def _recent_event(lines: list[str]) -> str:
interesting_keywords = (
"开始检测",
"获取到",
"当前批次检测完成",
"域名检测任务完成",
"代理池刷新完成",
"没有需要检测的域名",
"检测已停止",
)
for line in reversed(lines):
if any(keyword in line for keyword in interesting_keywords):
return line
return ""
def _load_runtime_state() -> dict:
try:
redis_client = get_redis()
raw = redis_client.get(_RUNTIME_STATE_KEY)
if not raw:
return {}
data = json.loads(raw)
return data if isinstance(data, dict) else {}
except Exception:
return {}
def _normalize_recent_warning(runtime_state: dict, recent_lines: list[str], available_proxy_count: int) -> str:
runtime_warning = str(runtime_state.get("recent_warning", "") or "").strip()
if runtime_warning:
if runtime_warning in {"未刷新", "代理未启用", "未配置代理池链接"}:
return ""
if available_proxy_count > 0 and ("无可用代理" in runtime_warning or "未取到可用代理数据" in runtime_warning):
return ""
return runtime_warning
recent_proxy_warning = next(
(line for line in reversed(recent_lines) if "代理池刷新失败" in line or "无可用代理" in line or "Redis订阅失败" in line),
"",
)
if available_proxy_count > 0 and "无可用代理" in recent_proxy_warning:
return ""
return recent_proxy_warning
def _build_proxy_runtime_snapshot(settings_payload: dict, runtime_state: dict, available_proxy_count: int) -> dict:
proxy_config = settings_payload.get("proxy_config") or {}
proxy_enable = bool(proxy_config.get("proxy_enable", False))
allow_direct = bool(proxy_config.get("allow_direct", False))
refresh_status = str(runtime_state.get("proxy_last_refresh_status", "") or "").strip()
refresh_time = str(runtime_state.get("proxy_last_refresh_time", "") or "").strip()
source_count = len(proxy_config.get("proxy_urls", []))
source_stats = runtime_state.get("proxy_last_source_stats") or []
raw_items = int(runtime_state.get("proxy_last_refresh_total_items", 0) or 0)
validated = int(runtime_state.get("proxy_last_validated_count", 0) or 0)
available = int(runtime_state.get("proxy_last_available_count", available_proxy_count) or available_proxy_count)
source_ok_count = sum(1 for item in source_stats if str(item.get("status", "") or "").strip() == "ok")
supplier_empty = bool(source_stats) and source_ok_count == len(source_stats) and raw_items <= 0
if not proxy_enable:
return {
"state": "disabled",
"label": "未启用代理",
"detail": "当前使用直连模式，未启用代理池",
"direct_fallback_active": True,
"reason": "proxy_disabled",
"last_refresh_status": refresh_status or "代理未启用",
"last_refresh_time": refresh_time,
"source_count": source_count,
"raw_items": raw_items,
"validated_count": validated,
"available_count": available,
"source_stats": source_stats,
"supplier_empty": False,
}
if available_proxy_count > 0:
detail = f"代理池当前可用 {available_proxy_count} 个代理，配置来源 {source_count} 个"
if refresh_status:
detail = f"{detail}；最近状态：{refresh_status}"
return {
"state": "healthy",
"label": "代理正常",
"detail": detail,
"direct_fallback_active": False,
"reason": "healthy",
"last_refresh_status": refresh_status,
"last_refresh_time": refresh_time,
"source_count": source_count,
"raw_items": raw_items,
"validated_count": validated,
"available_count": available,
"source_stats": source_stats,
"supplier_empty": False,
}
if allow_direct:
detail = "代理池当前无可用代理，已自动降级为直连继续执行"
reason = "no_available_proxy"
if supplier_empty:
reason = "supplier_empty_pool"
detail = "代理源最近都返回正常响应，但原始代理数为 0，当前判断为供应池为空；系统已自动降级为直连继续执行"
if refresh_status:
detail = f"{detail}；最近状态：{refresh_status}"
return {
"state": "degraded_direct",
"label": "降级直连",
"detail": detail,
"direct_fallback_active": True,
"reason": reason,
"last_refresh_status": refresh_status or "当前无可用代理",
"last_refresh_time": refresh_time,
"source_count": source_count,
"raw_items": raw_items,
"validated_count": validated,
"available_count": available,
"source_stats": source_stats,
"supplier_empty": supplier_empty,
}
detail = "代理池当前无可用代理，且未允许直连，检测链路会等待代理恢复"
reason = "no_available_proxy"
if supplier_empty:
reason = "supplier_empty_pool"
detail = "代理源最近都返回正常响应，但原始代理数为 0，当前判断为供应池为空；由于未允许直连，检测链路会等待代理恢复"
if refresh_status:
detail = f"{detail}；最近状态：{refresh_status}"
return {
"state": "blocked_no_proxy",
"label": "等待代理",
"detail": detail,
"direct_fallback_active": False,
"reason": reason,
"last_refresh_status": refresh_status or "当前无可用代理",
"last_refresh_time": refresh_time,
"source_count": source_count,
"raw_items": raw_items,
"validated_count": validated,
"available_count": available,
"source_stats": source_stats,
"supplier_empty": supplier_empty,
}
def get_detect_status() -> dict: def get_detect_status() -> dict:
queries = { queries = {
"pending": "select count(*) from domains where detect_status = 0", "pending": "select count(*) from domains where detect_status = 0",
@@ -37,14 +297,65 @@ def get_detect_status() -> dict:
last_log_time = modified.isoformat() last_log_time = modified.isoformat()
worker_online = (datetime.now(timezone.utc) - modified).total_seconds() < 180 worker_online = (datetime.now(timezone.utc) - modified).total_seconds() < 180
recent_lines = tail_lines("detect_worker.log", max_lines=80) recent_lines = tail_lines("detect_worker.log", max_lines=160)
recent_proxy_warning = next( runtime_state = _load_runtime_state()
(line for line in reversed(recent_lines) if "代理" in line or "Redis订阅失败" in line), runtime_started_at = runtime.get("latest_start_time", "") if 'runtime' in locals() else ""
"", recent_lines = _filter_lines_since(recent_lines, runtime_started_at)
) available_proxy_count = _extract_available_proxy_count(recent_lines)
active_thread_snapshot = _extract_active_thread_snapshot(recent_lines)
if runtime_state:
available_proxy_count = int(runtime_state.get("available_proxy_count", available_proxy_count) or available_proxy_count)
active_thread_snapshot = {
"active": int(runtime_state.get("active_threads", active_thread_snapshot["active"]) or active_thread_snapshot["active"]),
"max": int(runtime_state.get("max_threads", active_thread_snapshot["max"]) or active_thread_snapshot["max"]),
}
progress_total = sum(progress.values())
progress_done = progress.get("completed", 0) + progress.get("blacklisted", 0) + progress.get("failed", 0)
progress_percent = round((progress_done / progress_total) * 100, 2) if progress_total > 0 else 0
runtime = detect_worker_runtime() runtime = detect_worker_runtime()
runtime_started_at = runtime.get("latest_start_time", "")
recent_lines = _filter_lines_since(recent_lines, runtime_started_at)
recent_proxy_warning = _normalize_recent_warning(runtime_state, recent_lines, available_proxy_count)
proxy_runtime = _build_proxy_runtime_snapshot(settings_payload, runtime_state, available_proxy_count)
thread_count_resolution = resolve_thread_count(settings_payload=settings_payload)
effective_thread_count = int(thread_count_resolution["effective_thread_count"])
runtime_settings = get_runtime_settings() runtime_settings = get_runtime_settings()
worker_online = worker_online or runtime.get("running", False) worker_online = worker_online or runtime.get("running", False)
if runtime_state.get("service_running") is True:
worker_online = True
if not runtime_state.get("detecting", False) and not progress.get("running", 0):
active_thread_snapshot = {"active": 0, "max": active_thread_snapshot["max"] or effective_thread_count}
settings_summary = {
"thread_count": effective_thread_count,
"thread_count_default": int(thread_count_resolution["default_thread_count"]),
"thread_count_source": str(thread_count_resolution["source"]),
"thread_count_override": thread_count_resolution["override_thread_count"],
"thread_count_node_code": str(thread_count_resolution["node_code"]),
"proxy_enable": settings_payload["proxy_config"].get("proxy_enable", False),
"allow_direct": settings_payload["proxy_config"].get("allow_direct", False),
"proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])),
}
active_job = get_active_detect_job_summary()
runtime_snapshot = {
**runtime,
"detecting": runtime_state.get("detecting", False),
"proxy_runtime_state": proxy_runtime["state"],
"proxy_runtime_label": proxy_runtime["label"],
"proxy_runtime_detail": proxy_runtime["detail"],
"proxy_direct_fallback_active": proxy_runtime["direct_fallback_active"],
"proxy_runtime_reason": proxy_runtime["reason"],
"proxy_supplier_empty": proxy_runtime["supplier_empty"],
}
runs = sync_detect_runs(runtime_snapshot, progress, settings_summary, active_job=active_job)
dependency_alerts = _extract_dependency_alerts(recent_lines)
append_detect_result_projection_if_changed(
detect={
"active_job": active_job,
"progress": progress,
"phase_label": runtime_state.get("phase", ""),
"phase_detail": runtime_state.get("detail", ""),
}
)
return { return {
"worker_online": worker_online, "worker_online": worker_online,
@@ -54,12 +365,41 @@ def get_detect_status() -> dict:
"worker_process_count": runtime.get("process_count", 0), "worker_process_count": runtime.get("process_count", 0),
"worker_latest_start_time": runtime.get("latest_start_time", ""), "worker_latest_start_time": runtime.get("latest_start_time", ""),
"worker_runtime_message": runtime.get("message", ""), "worker_runtime_message": runtime.get("message", ""),
"thread_count": settings_payload["thread_count"], "runtime_state": runtime_state,
"phase_label": runtime_state.get("phase", ""),
"phase_detail": runtime_state.get("detail", ""),
"detecting": runtime_state.get("detecting", False),
"thread_count": effective_thread_count,
"thread_count_default": int(thread_count_resolution["default_thread_count"]),
"thread_count_source": str(thread_count_resolution["source"]),
"thread_count_override": thread_count_resolution["override_thread_count"],
"thread_count_node_code": str(thread_count_resolution["node_code"]),
"active_thread_count": active_thread_snapshot["active"],
"max_thread_count": active_thread_snapshot["max"] or effective_thread_count,
"proxy_enable": settings_payload["proxy_config"].get("proxy_enable", False), "proxy_enable": settings_payload["proxy_config"].get("proxy_enable", False),
"allow_direct": settings_payload["proxy_config"].get("allow_direct", False), "allow_direct": settings_payload["proxy_config"].get("allow_direct", False),
"proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])), "proxy_pool_count": len(settings_payload["proxy_config"].get("proxy_urls", [])),
"available_proxy_count": 0, "available_proxy_count": available_proxy_count,
"proxy_runtime_state": proxy_runtime["state"],
"proxy_runtime_label": proxy_runtime["label"],
"proxy_runtime_detail": proxy_runtime["detail"],
"proxy_direct_fallback_active": proxy_runtime["direct_fallback_active"],
"proxy_runtime_reason": proxy_runtime["reason"],
"proxy_supplier_empty": proxy_runtime["supplier_empty"],
"proxy_last_refresh_status": proxy_runtime["last_refresh_status"],
"proxy_last_refresh_time": proxy_runtime["last_refresh_time"],
"proxy_last_refresh_source_count": proxy_runtime["source_count"],
"proxy_last_refresh_total_items": proxy_runtime["raw_items"],
"proxy_last_validated_count": proxy_runtime["validated_count"],
"proxy_last_available_count": proxy_runtime["available_count"],
"proxy_source_stats": proxy_runtime["source_stats"],
"dependency_alerts": dependency_alerts,
"last_worker_log_time": last_log_time, "last_worker_log_time": last_log_time,
"progress": progress, "progress": progress,
"progress_percent": progress_percent,
"recent_event": runtime_state.get("detail") or _recent_event(recent_lines),
"recent_warning": recent_proxy_warning, "recent_warning": recent_proxy_warning,
"log_lines": recent_lines,
"runs": runs,
"active_job": active_job,
} }

344
domain-api/app/services/domains_service.py
View File

@@ -7,7 +7,7 @@ from app.core.db import get_db
DETECT_STATUS_LABELS = { DETECT_STATUS_LABELS = {
0: "待检测", 0: "待检测",
1: "检测完成", 1: "检测通过",
2: "检测中", 2: "检测中",
3: "黑名单", 3: "黑名单",
4: "检测失败", 4: "检测失败",
@@ -40,6 +40,14 @@ REVIEW_STATUS_LABELS = {
3: "人工拒绝", 3: "人工拒绝",
} }
SOURCE_TYPE_LABELS = {
1: "聚名一口价",
2: "聚名过期删除",
6: "手工录入",
7: "TXT 导入",
9: "其它",
}
BEIAN_STATUS_LABELS = { BEIAN_STATUS_LABELS = {
1: "未检测", 1: "未检测",
2: "有备案", 2: "有备案",
@@ -47,6 +55,103 @@ BEIAN_STATUS_LABELS = {
} }
def _format_timestamp(value) -> str | None:
if not value:
return None
return value.isoformat(sep=" ", timespec="seconds")
def _json_status_to_text(value) -> str:
if isinstance(value, dict):
status = value.get("status")
else:
status = None
return "是" if status else "否"
def _json_state(value) -> str:
if isinstance(value, dict):
return str(value.get("state") or "").strip()
return ""
def _json_message(value) -> str:
if isinstance(value, dict):
return str(value.get("message") or "").strip()
return ""
def _normalize_step_detail(label: str, value) -> dict:
payload = value if isinstance(value, dict) else {}
state = str(payload.get("state") or "").strip()
return {
"label": label,
"state": state or ("passed" if bool(payload.get("status")) else ""),
"status": bool(payload.get("status")) if isinstance(payload.get("status"), bool) else None,
"message": str(payload.get("message") or "").strip(),
"checked_at": str(payload.get("checked_at") or "").strip(),
"step": str(payload.get("step") or "").strip(),
"raw": payload,
}
def _build_step_details(row: tuple) -> list[dict]:
return [
_normalize_step_detail("百度历史收录", row[16]),
_normalize_step_detail("百度Site收录", row[17]),
{
"label": "标题为中文",
"state": "passed" if bool(row[18]) else "",
"status": bool(row[18]),
"message": "标题含中文" if bool(row[18]) else "",
"checked_at": "",
"step": "中文标题",
"raw": row[18],
},
_normalize_step_detail("360 Site收录", row[19]),
_normalize_step_detail("Google Site收录", row[20]),
_normalize_step_detail("时光机", row[21]),
_normalize_step_detail("站长之家", row[22]),
_normalize_step_detail("爱站网", row[23]),
]
def _summarize_step_details(step_details: list[dict]) -> dict:
degraded = [item for item in step_details if item.get("state") == "degraded"]
failed = [item for item in step_details if item.get("state") == "failed"]
blacklisted = [item for item in step_details if item.get("state") == "blacklisted"]
return {
"degraded_count": len(degraded),
"failed_count": len(failed),
"blacklisted_count": len(blacklisted),
"has_degraded": bool(degraded),
"has_failed": bool(failed),
"has_blacklisted_step": bool(blacklisted),
"summary_text": (
f"降级 {len(degraded)} / 失败 {len(failed)} / 命中 {len(blacklisted)}"
if degraded or failed or blacklisted
else "步骤正常"
),
}
def _bool_to_text(value) -> str:
return "是" if bool(value) else "否"
def _normalize_detection_update(value) -> bool | None:
if value in (None, "", "skip"):
return None
if isinstance(value, bool):
return value
text = str(value).strip().lower()
if text in {"是", "true", "1", "yes"}:
return True
if text in {"否", "false", "0", "no"}:
return False
raise ValueError("检测结果字段仅支持“是”或“否”")
def _build_domain_query_parts(filters: dict | None = None) -> tuple[str, str, list[object]]: def _build_domain_query_parts(filters: dict | None = None) -> tuple[str, str, list[object]]:
filters = filters or {} filters = filters or {}
conditions: list[str] = [] conditions: list[str] = []
@@ -79,6 +184,12 @@ def _build_domain_query_parts(filters: dict | None = None) -> tuple[str, str, li
if filters.get("website_url"): if filters.get("website_url"):
conditions.append("coalesce(d.website_url, '') ilike %s") conditions.append("coalesce(d.website_url, '') ilike %s")
params.append(f"%{str(filters['website_url']).strip()}%") params.append(f"%{str(filters['website_url']).strip()}%")
if filters.get("company_type"):
conditions.append("coalesce(d.company_type, '') = %s")
params.append(str(filters["company_type"]).strip())
if filters.get("source_type") is not None:
conditions.append("d.source_type = %s")
params.append(int(filters["source_type"]))
if filters.get("backlink_gt_10"): if filters.get("backlink_gt_10"):
conditions.append("coalesce(dd.backlink_count_gt_10, false) = true") conditions.append("coalesce(dd.backlink_count_gt_10, false) = true")
@@ -102,6 +213,8 @@ def fetch_domains(
beian_year: int | None = None, beian_year: int | None = None,
snapshot_year: str | None = None, snapshot_year: str | None = None,
website_url: str | None = None, website_url: str | None = None,
company_type: str | None = None,
source_type: int | None = None,
backlink_gt_10: bool | None = None, backlink_gt_10: bool | None = None,
) -> dict: ) -> dict:
offset = (page - 1) * page_size offset = (page - 1) * page_size
@@ -115,6 +228,8 @@ def fetch_domains(
"beian_year": beian_year, "beian_year": beian_year,
"snapshot_year": snapshot_year, "snapshot_year": snapshot_year,
"website_url": website_url, "website_url": website_url,
"company_type": company_type,
"source_type": source_type,
"backlink_gt_10": backlink_gt_10, "backlink_gt_10": backlink_gt_10,
} }
from_clause, where_clause, params = _build_domain_query_parts(filters) from_clause, where_clause, params = _build_domain_query_parts(filters)
@@ -138,7 +253,18 @@ def fetch_domains(
d.snapshot_years, d.snapshot_years,
d.backlink_count, d.backlink_count,
d.detect_time, d.detect_time,
coalesce(dd.backlink_count_gt_10, false) as backlink_gt_10 d.source_type,
coalesce(dd.backlink_count_gt_10, false) as backlink_gt_10,
d.expire_date,
d.company_type,
dd.baidu_history,
dd.baidu_site,
dd.is_chinese_title,
dd.qihu360_site,
dd.google_site,
dd.wayback_info,
dd.chinaz_info,
dd.aizhan_info
{from_clause} {from_clause}
{where_clause} {where_clause}
order by d.id desc order by d.id desc
@@ -148,8 +274,11 @@ def fetch_domains(
) )
rows = cur.fetchall() rows = cur.fetchall()
items = [ items = []
{ for row in rows:
step_details = _build_step_details(row)
step_summary = _summarize_step_details(step_details)
items.append({
"id": row[0], "id": row[0],
"domain": row[1], "domain": row[1],
"register_status": REGISTER_STATUS_LABELS.get(row[2], str(row[2])), "register_status": REGISTER_STATUS_LABELS.get(row[2], str(row[2])),
@@ -166,11 +295,19 @@ def fetch_domains(
"beian_year": row[8], "beian_year": row[8],
"snapshot_years": row[9] or "", "snapshot_years": row[9] or "",
"backlink_count": row[10], "backlink_count": row[10],
"detect_time": row[11].isoformat() if row[11] else None, "detect_time": _format_timestamp(row[11]),
"backlink_gt_10": row[12], "source_type": row[12],
} "source_label": SOURCE_TYPE_LABELS.get(row[12], str(row[12])),
for row in rows "backlink_gt_10": row[13],
] "expire_date": _format_timestamp(row[14]),
"company_type": row[15] or "",
"baidu_history": _json_status_to_text(row[16]),
"baidu_site": _json_status_to_text(row[17]),
"is_chinese_title": _bool_to_text(row[18]),
"qihu360_site": _json_status_to_text(row[19]),
"google_site": _json_status_to_text(row[20]),
"step_summary": step_summary,
})
return { return {
"list": items, "list": items,
"page": page, "page": page,
@@ -180,12 +317,116 @@ def fetch_domains(
} }
def fetch_domain_detail(domain_id: int) -> dict | None:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
select
d.id,
d.domain,
d.register_status,
d.use_status,
d.detect_status,
d.review_status,
d.has_beian,
d.website_url,
d.beian_year,
d.snapshot_years,
d.backlink_count,
d.detect_time,
d.source_type,
coalesce(dd.backlink_count_gt_10, false) as backlink_gt_10,
d.expire_date,
d.company_type,
dd.baidu_history,
dd.baidu_site,
dd.is_chinese_title,
dd.qihu360_site,
dd.google_site,
dd.wayback_info,
dd.chinaz_info,
dd.aizhan_info,
dd.juziseo_info,
dd.jucha_info
from domains d
left join domain_detections dd on dd.domain_id = d.id
where d.id = %s
limit 1
""",
(int(domain_id),),
)
row = cur.fetchone()
if not row:
return None
step_details = [
_normalize_step_detail("百度历史收录", row[16]),
_normalize_step_detail("百度Site收录", row[17]),
{
"label": "标题为中文",
"state": "passed" if bool(row[18]) else "",
"status": bool(row[18]),
"message": "标题含中文" if bool(row[18]) else "",
"checked_at": "",
"step": "中文标题",
"raw": row[18],
},
_normalize_step_detail("360 Site收录", row[19]),
_normalize_step_detail("Google Site收录", row[20]),
_normalize_step_detail("时光机", row[21]),
_normalize_step_detail("站长之家", row[22]),
_normalize_step_detail("爱站网", row[23]),
_normalize_step_detail("桔子SEO", row[24]),
_normalize_step_detail("聚查", row[25]),
]
step_summary = _summarize_step_details(step_details)
return {
"id": row[0],
"domain": row[1],
"register_status": REGISTER_STATUS_LABELS.get(row[2], str(row[2])),
"register_status_code": row[2],
"use_status": USE_STATUS_LABELS.get(row[3], str(row[3])),
"use_status_code": row[3],
"detect_status": DETECT_STATUS_LABELS.get(row[4], str(row[4])),
"detect_status_code": row[4],
"review_status": REVIEW_STATUS_LABELS.get(row[5], str(row[5])),
"review_status_code": row[5],
"has_beian": BEIAN_STATUS_LABELS.get(row[6], str(row[6])),
"has_beian_code": row[6],
"website_url": row[7] or "",
"beian_year": row[8],
"snapshot_years": row[9] or "",
"backlink_count": row[10],
"detect_time": _format_timestamp(row[11]),
"source_type": row[12],
"source_label": SOURCE_TYPE_LABELS.get(row[12], str(row[12])),
"backlink_gt_10": row[13],
"expire_date": _format_timestamp(row[14]),
"company_type": row[15] or "",
"step_summary": step_summary,
"step_details": step_details,
"raw_detection": {
"baidu_history": row[16],
"baidu_site": row[17],
"is_chinese_title": row[18],
"qihu360_site": row[19],
"google_site": row[20],
"wayback_info": row[21],
"chinaz_info": row[22],
"aizhan_info": row[23],
"juziseo_info": row[24],
"jucha_info": row[25],
},
}
def domain_filter_options() -> dict: def domain_filter_options() -> dict:
return { return {
"register_status": [ "register_status": [
{"label": label, "value": value} {"label": label, "value": value}
for value, label in REGISTER_STATUS_LABELS.items() for value, label in REGISTER_STATUS_LABELS.items()
if value in (2, 3, 4, 5, 6, 7, 8, 10) if value in (0, 2, 3, 4, 5, 6, 7, 8, 9, 10)
], ],
"detect_status": [ "detect_status": [
{"label": label, "value": value} {"label": label, "value": value}
@@ -204,6 +445,10 @@ def domain_filter_options() -> dict:
{"label": "有备案", "value": 2}, {"label": "有备案", "value": 2},
{"label": "无备案", "value": 3}, {"label": "无备案", "value": 3},
], ],
"source_type": [
{"label": label, "value": value}
for value, label in SOURCE_TYPE_LABELS.items()
],
"supports_backlink_gt_10": True, "supports_backlink_gt_10": True,
"supports_txt_export": True, "supports_txt_export": True,
"supports_excel_export": True, "supports_excel_export": True,
@@ -225,11 +470,35 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict:
"detect_time", "detect_time",
"website_url", "website_url",
"backlink_count", "backlink_count",
"baidu_history",
"baidu_site",
"is_chinese_title",
"qihu360_site",
"google_site",
} }
payload = {key: value for key, value in updates.items() if key in allowed_fields and value not in (None, "", "skip")} payload = {key: value for key, value in updates.items() if key in allowed_fields and value not in (None, "", "skip")}
if not payload: if not payload:
raise ValueError("没有可更新的字段") raise ValueError("没有可更新的字段")
domain_fields = {
"review_status",
"expire_date",
"has_beian",
"beian_year",
"snapshot_years",
"company_type",
"detect_time",
"website_url",
"backlink_count",
}
detection_fields = {
"baidu_history",
"baidu_site",
"is_chinese_title",
"qihu360_site",
"google_site",
}
updated_count = 0 updated_count = 0
with get_db() as conn: with get_db() as conn:
with conn.cursor() as cur: with conn.cursor() as cur:
@@ -238,6 +507,8 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict:
params: list[object] = [] params: list[object] = []
for field, value in payload.items(): for field, value in payload.items():
if field not in domain_fields:
continue
if field == "backlink_count": if field == "backlink_count":
set_parts.append("backlink_count = %s") set_parts.append("backlink_count = %s")
params.append(int(value)) params.append(int(value))
@@ -245,11 +516,24 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict:
set_parts.append(f"{field} = %s") set_parts.append(f"{field} = %s")
params.append(value) params.append(value)
params.append(domain_id) if set_parts:
cur.execute( params.append(domain_id)
f"update domains set {', '.join(set_parts)}, update_time = now() where id = %s", cur.execute(
tuple(params), f"update domains set {', '.join(set_parts)}, update_time = now() where id = %s",
) tuple(params),
)
detection_payload: dict[str, object] = {}
for field in detection_fields:
if field not in payload:
continue
normalized = _normalize_detection_update(payload[field])
if normalized is None:
continue
if field == "is_chinese_title":
detection_payload[field] = normalized
else:
detection_payload[field] = {"status": normalized}
if "backlink_count" in payload: if "backlink_count" in payload:
backlink_gt_10 = int(payload["backlink_count"]) > 10 backlink_gt_10 = int(payload["backlink_count"]) > 10
@@ -268,6 +552,36 @@ def batch_update_domains(domain_ids: list[int], updates: dict) -> dict:
(domain_id, backlink_gt_10), (domain_id, backlink_gt_10),
) )
if detection_payload:
cur.execute("select id from domain_detections where domain_id = %s", (domain_id,))
existing_detection = cur.fetchone()
if existing_detection:
detection_set_parts: list[str] = []
detection_params: list[object] = []
for field, value in detection_payload.items():
detection_set_parts.append(f"{field} = %s")
detection_params.append(value)
detection_params.append(domain_id)
cur.execute(
f"""
update domain_detections
set {', '.join(detection_set_parts)}, update_time = now()
where domain_id = %s
""",
tuple(detection_params),
)
else:
insert_fields = ["domain_id", *detection_payload.keys(), "create_time", "update_time"]
placeholders = ["%s"] * (1 + len(detection_payload)) + ["now()", "now()"]
insert_params = [domain_id, *detection_payload.values()]
cur.execute(
f"""
insert into domain_detections ({', '.join(insert_fields)})
values ({', '.join(placeholders)})
""",
tuple(insert_params),
)
updated_count += 1 updated_count += 1
conn.commit() conn.commit()

47
domain-api/app/services/export_service.py
View File

@@ -23,8 +23,15 @@ EXPORT_HEADERS = [
("use_status", "使用状态"), ("use_status", "使用状态"),
("detect_status", "检测状态"), ("detect_status", "检测状态"),
("review_status", "复核状态"), ("review_status", "复核状态"),
("expire_date", "过期时间"),
("company_type", "单位性质"),
("has_beian", "备案状态"), ("has_beian", "备案状态"),
("website_url", "首页网址"), ("website_url", "首页网址"),
("baidu_history", "百度历史收录"),
("baidu_site", "百度Site收录"),
("is_chinese_title", "标题为中文"),
("qihu360_site", "360 Site收录"),
("google_site", "Google Site收录"),
("beian_year", "备案年份"), ("beian_year", "备案年份"),
("snapshot_years", "快照年份"), ("snapshot_years", "快照年份"),
("backlink_count", "友链数"), ("backlink_count", "友链数"),
@@ -33,6 +40,18 @@ EXPORT_HEADERS = [
] ]
def _json_status_to_text(value) -> str:
if isinstance(value, dict):
status = value.get("status")
else:
status = None
return "是" if status else "否"
def _bool_to_text(value) -> str:
return "是" if bool(value) else "否"
def _normalize_payload(payload: dict) -> dict: def _normalize_payload(payload: dict) -> dict:
data = dict(payload or {}) data = dict(payload or {})
data["page"] = int(data.get("page", 1) or 1) data["page"] = int(data.get("page", 1) or 1)
@@ -68,8 +87,15 @@ def _query_export_rows(payload: dict) -> list[dict]:
d.use_status, d.use_status,
d.detect_status, d.detect_status,
d.review_status, d.review_status,
d.expire_date,
d.company_type,
d.has_beian, d.has_beian,
d.website_url, d.website_url,
dd.baidu_history,
dd.baidu_site,
dd.is_chinese_title,
dd.qihu360_site,
dd.google_site,
d.beian_year, d.beian_year,
d.snapshot_years, d.snapshot_years,
d.backlink_count, d.backlink_count,
@@ -93,13 +119,20 @@ def _query_export_rows(payload: dict) -> list[dict]:
"use_status": USE_STATUS_LABELS.get(row[2], str(row[2])), "use_status": USE_STATUS_LABELS.get(row[2], str(row[2])),
"detect_status": DETECT_STATUS_LABELS.get(row[3], str(row[3])), "detect_status": DETECT_STATUS_LABELS.get(row[3], str(row[3])),
"review_status": REVIEW_STATUS_LABELS.get(row[4], str(row[4])), "review_status": REVIEW_STATUS_LABELS.get(row[4], str(row[4])),
"has_beian": BEIAN_STATUS_LABELS.get(row[5], str(row[5])), "expire_date": row[5].isoformat(sep=" ", timespec="seconds") if row[5] else "",
"website_url": row[6] or "", "company_type": row[6] or "",
"beian_year": row[7] or "", "has_beian": BEIAN_STATUS_LABELS.get(row[7], str(row[7])),
"snapshot_years": row[8] or "", "website_url": row[8] or "",
"backlink_count": row[9] or 0, "baidu_history": _json_status_to_text(row[9]),
"backlink_gt_10": "是" if row[10] else "否", "baidu_site": _json_status_to_text(row[10]),
"detect_time": row[11].isoformat(sep=" ", timespec="seconds") if row[11] else "", "is_chinese_title": _bool_to_text(row[11]),
"qihu360_site": _json_status_to_text(row[12]),
"google_site": _json_status_to_text(row[13]),
"beian_year": row[14] or "",
"snapshot_years": row[15] or "",
"backlink_count": row[16] or 0,
"backlink_gt_10": "是" if row[17] else "否",
"detect_time": row[18].isoformat(sep=" ", timespec="seconds") if row[18] else "",
} }
) )
return result return result

95
domain-api/app/services/import_task_service.py
View File

@@ -10,6 +10,11 @@ from app.services.import_worker_service import import_domains_from_path
_IMPORT_TASK_LOCK = threading.Lock() _IMPORT_TASK_LOCK = threading.Lock()
_SOURCE_TYPE_LABELS = {
6: "手工录入",
7: "TXT 导入",
9: "其它",
}
def _now() -> str: def _now() -> str:
@@ -24,6 +29,12 @@ def _save_tasks(tasks: list[dict]) -> None:
save_import_records(tasks) save_import_records(tasks)
def _append_log_locked(target: dict, message: str) -> None:
target.setdefault("logs", [])
target["logs"].append(f"[{_now()}] {message}")
target["logs"] = target["logs"][-200:]
def _update_task(task_id: str, **patch: object) -> dict | None: def _update_task(task_id: str, **patch: object) -> dict | None:
with _IMPORT_TASK_LOCK: with _IMPORT_TASK_LOCK:
tasks = load_import_records() tasks = load_import_records()
@@ -36,13 +47,73 @@ def _update_task(task_id: str, **patch: object) -> dict | None:
return dict(target) return dict(target)
def _update_task_with_log(task_id: str, log_message: str, **patch: object) -> dict | None:
with _IMPORT_TASK_LOCK:
tasks = load_import_records()
target = next((item for item in tasks if item["task_id"] == task_id), None)
if not target:
return None
target.update(patch)
_append_log_locked(target, log_message)
target["updated_at"] = _now()
_save_tasks(tasks)
return dict(target)
def _phase_label(phase: str) -> str:
mapping = {
"queued": "排队中",
"reading": "读取文件中",
"normalizing": "清洗中",
"importing": "入库中",
"completed": "已完成",
"failed": "失败",
}
return mapping.get(phase, phase)
def _source_label(source_type: int) -> str:
return _SOURCE_TYPE_LABELS.get(int(source_type or 7), "未知")
def _run_import_task(task_id: str, file_path: str, source_type: int = 7) -> None: def _run_import_task(task_id: str, file_path: str, source_type: int = 7) -> None:
_update_task(task_id, status="running", started_at=_now(), message="导入任务开始执行") _update_task_with_log(
task_id,
f"导入任务开始执行，来源类型：{_source_label(source_type)}",
status="running",
started_at=_now(),
message=f"导入任务开始执行，来源类型：{_source_label(source_type)}",
phase="reading",
phase_label=_phase_label("reading"),
)
try: try:
result = import_domains_from_path(Path(file_path), source_type=source_type) path = Path(file_path)
stats = result.get("stats", {}) _update_task_with_log(
_update_task(
task_id, task_id,
f"开始读取文件：{path.name}",
phase="reading",
phase_label=_phase_label("reading"),
)
raw_lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
total_lines = len(raw_lines)
non_empty = sum(1 for line in raw_lines if line.strip())
_update_task_with_log(
task_id,
f"文件读取完成，共 {total_lines} 行，非空 {non_empty} 行",
phase="normalizing",
phase_label=_phase_label("normalizing"),
message=f"文件读取完成，准备清洗 {non_empty} 条域名",
)
result = import_domains_from_path(path, source_type=source_type)
stats = result.get("stats", {})
_update_task_with_log(
task_id,
(
f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，"
f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}，"
f"来源类型 {result.get('source_label') or _source_label(source_type)}"
),
status="completed", status="completed",
completed_at=_now(), completed_at=_now(),
result=result, result=result,
@@ -50,13 +121,18 @@ def _run_import_task(task_id: str, file_path: str, source_type: int = 7) -> None
f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，" f"导入完成：总数 {stats.get('total', 0)}，有效 {stats.get('valid', 0)}，"
f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}" f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}"
), ),
phase="completed",
phase_label=_phase_label("completed"),
) )
except Exception as exc: except Exception as exc:
_update_task( _update_task_with_log(
task_id, task_id,
f"导入失败：{exc}",
status="failed", status="failed",
completed_at=_now(), completed_at=_now(),
message=f"导入失败：{exc}", message=f"导入失败：{exc}",
phase="failed",
phase_label=_phase_label("failed"),
) )
@@ -71,12 +147,16 @@ def create_import_task(content: bytes, filename: str, source_type: int = 7) -> d
"filename": safe_name, "filename": safe_name,
"stored_path": str(target), "stored_path": str(target),
"source_type": source_type, "source_type": source_type,
"source_label": _source_label(source_type),
"status": "queued", "status": "queued",
"message": "文件已接收，等待处理", "message": f"文件已接收，等待后台处理，来源类型：{_source_label(source_type)}",
"created_at": _now(), "created_at": _now(),
"updated_at": _now(), "updated_at": _now(),
"started_at": "", "started_at": "",
"completed_at": "", "completed_at": "",
"phase": "queued",
"phase_label": _phase_label("queued"),
"logs": [f"[{_now()}] 文件已接收，等待后台处理，来源类型：{_source_label(source_type)}"],
"result": None, "result": None,
} }
@@ -104,6 +184,9 @@ def retry_import_task(task_id: str) -> dict:
target["completed_at"] = "" target["completed_at"] = ""
target["updated_at"] = _now() target["updated_at"] = _now()
target["result"] = None target["result"] = None
target["phase"] = "queued"
target["phase_label"] = _phase_label("queued")
target["logs"] = [f"[{_now()}] 任务已重新加入队列，等待后台执行"]
_save_tasks(tasks) _save_tasks(tasks)
stored_path = target["stored_path"] stored_path = target["stored_path"]
source_type = int(target.get("source_type", 7)) source_type = int(target.get("source_type", 7))

7
domain-api/app/services/import_worker_service.py
View File

@@ -8,6 +8,11 @@ from app.core.files import import_root
DOMAIN_PATTERN = re.compile(r"^(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+(com|net)$", re.IGNORECASE) DOMAIN_PATTERN = re.compile(r"^(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+(com|net)$", re.IGNORECASE)
SOURCE_TYPE_LABELS = {
6: "手工录入",
7: "TXT 导入",
9: "其它",
}
def normalize_domain(value: str) -> str | None: def normalize_domain(value: str) -> str | None:
@@ -92,6 +97,8 @@ def import_domains_from_path(file_path: Path, source_type: int = 7) -> dict:
} }
return { return {
"filename": file_path.name, "filename": file_path.name,
"source_type": source_type,
"source_label": SOURCE_TYPE_LABELS.get(int(source_type or 7), "未知"),
"stats": stats, "stats": stats,
} }

494
domain-api/app/services/juming_service.py Normal file
View File

@@ -0,0 +1,494 @@
from __future__ import annotations
import ast
import pickle
import re
import sys
from collections.abc import Callable
from datetime import date, datetime, timedelta
from pathlib import Path
from requests.cookies import RequestsCookieJar
from app.core.db import get_db
from app.core.redis_client import get_redis
from app.core.config import settings
from app.core.files import read_runtime_json, write_runtime_json
from app.services.import_worker_service import normalize_domain
DOMAIN_ROOT = Path(settings.domain_root)
if str(DOMAIN_ROOT) not in sys.path:
sys.path.insert(0, str(DOMAIN_ROOT))
from detect.juming import JM # type: ignore # noqa: E402
from detect.jucha import JC # type: ignore # noqa: E402
YKJ_DOMAIN_PATTERN = re.compile(r"<a class='yda1 ydz' ym='([^']*)'")
JUMING_COOKIE_FILE = DOMAIN_ROOT / "juming_cookies.pkl"
JUCHA_COOKIE_FILE = DOMAIN_ROOT / "jucha_cookies.pkl"
LEGACY_JUMING_COOKIE_FILES = [
Path.cwd() / "juming_cookies.pkl",
Path(__file__).resolve().parents[2] / "juming_cookies.pkl",
]
DELETE_LIST_SOURCE_TYPE = 2
FIXED_PRICE_SOURCE_TYPE = 1
JUMING_PREFERENCES_FILE = "juming_preferences.json"
class TaskStoppedError(RuntimeError):
pass
def _default_juming_preferences() -> dict:
return {
"mode": "delete_list",
"page_start": 1,
"page_size": 500,
"page_count": 1,
"crawl_date": date.today().isoformat(),
"auto_date": True,
}
def get_juming_preferences() -> dict:
defaults = _default_juming_preferences()
stored = read_runtime_json(JUMING_PREFERENCES_FILE, default={})
payload = {
"mode": str(stored.get("mode", defaults["mode"])) if stored else defaults["mode"],
"page_start": int(stored.get("page_start", defaults["page_start"])) if stored else defaults["page_start"],
"page_size": int(stored.get("page_size", defaults["page_size"])) if stored else defaults["page_size"],
"page_count": int(stored.get("page_count", defaults["page_count"])) if stored else defaults["page_count"],
"crawl_date": str(stored.get("crawl_date", defaults["crawl_date"])) if stored else defaults["crawl_date"],
"auto_date": bool(stored.get("auto_date", defaults["auto_date"])) if stored else defaults["auto_date"],
}
if payload["mode"] not in {"delete_list", "fixed_price"}:
payload["mode"] = defaults["mode"]
payload["page_start"] = max(payload["page_start"], 1)
payload["page_size"] = min(max(payload["page_size"], 1), 1000)
payload["page_count"] = min(max(payload["page_count"], 1), 20)
if not payload["crawl_date"]:
payload["crawl_date"] = defaults["crawl_date"]
return payload
def update_juming_preferences(payload: dict) -> dict:
current = get_juming_preferences()
next_payload = {
"mode": str(payload.get("mode", current["mode"]) or current["mode"]),
"page_start": int(payload.get("page_start", current["page_start"]) or current["page_start"]),
"page_size": int(payload.get("page_size", current["page_size"]) or current["page_size"]),
"page_count": int(payload.get("page_count", current["page_count"]) or current["page_count"]),
"crawl_date": str(payload.get("crawl_date", current["crawl_date"]) or current["crawl_date"]),
"auto_date": bool(payload.get("auto_date", current["auto_date"])),
}
if next_payload["mode"] not in {"delete_list", "fixed_price"}:
raise ValueError("无效的聚名采集类型")
next_payload["page_start"] = max(next_payload["page_start"], 1)
next_payload["page_size"] = min(max(next_payload["page_size"], 1), 1000)
next_payload["page_count"] = min(max(next_payload["page_count"], 1), 20)
write_runtime_json(JUMING_PREFERENCES_FILE, next_payload)
return next_payload
def _cookie_dict_to_jar(cookie_dict: dict[str, str]) -> RequestsCookieJar:
cookie_jar = RequestsCookieJar()
for name, value in cookie_dict.items():
cookie_jar.set(name, value)
return cookie_jar
def _emit_log(log: Callable[[str], None] | None, message: str) -> None:
if log:
log(message)
def _check_stop(should_stop: Callable[[], bool] | None) -> None:
if should_stop and should_stop():
raise TaskStoppedError("任务已停止")
def _cookie_jar_to_dict(cookie_jar: RequestsCookieJar) -> dict[str, str]:
return {str(cookie.name): str(cookie.value) for cookie in cookie_jar}
def _persist_juming_cookie(cookie_jar: RequestsCookieJar) -> None:
JUMING_COOKIE_FILE.parent.mkdir(parents=True, exist_ok=True)
with JUMING_COOKIE_FILE.open("wb") as handle:
pickle.dump(cookie_jar, handle)
try:
redis_client = get_redis()
redis_client.set("domain_tool:juming_cookies", str(_cookie_jar_to_dict(cookie_jar)))
except Exception:
pass
def _jucha_cookie_status() -> dict:
if JUCHA_COOKIE_FILE.exists():
return {
"cookie_ready": True,
"cookie_file": str(JUCHA_COOKIE_FILE),
}
return {
"cookie_ready": False,
"cookie_file": str(JUCHA_COOKIE_FILE),
}
def _load_juming_cookie() -> tuple[RequestsCookieJar | None, str]:
if JUMING_COOKIE_FILE.exists():
try:
with JUMING_COOKIE_FILE.open("rb") as handle:
loaded = pickle.load(handle)
if isinstance(loaded, RequestsCookieJar):
return loaded, "local"
if isinstance(loaded, dict):
return _cookie_dict_to_jar({str(k): str(v) for k, v in loaded.items()}), "local"
except Exception:
pass
for legacy_path in LEGACY_JUMING_COOKIE_FILES:
if not legacy_path.exists() or legacy_path == JUMING_COOKIE_FILE:
continue
try:
with legacy_path.open("rb") as handle:
loaded = pickle.load(handle)
if isinstance(loaded, RequestsCookieJar):
_persist_juming_cookie(loaded)
return loaded, f"migrated:{legacy_path}"
if isinstance(loaded, dict):
cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in loaded.items()})
_persist_juming_cookie(cookie_jar)
return cookie_jar, f"migrated:{legacy_path}"
except Exception:
continue
try:
redis_client = get_redis()
raw = redis_client.get("domain_tool:juming_cookies")
if raw:
parsed = ast.literal_eval(raw)
if isinstance(parsed, dict) and parsed:
cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in parsed.items()})
try:
with JUMING_COOKIE_FILE.open("wb") as handle:
pickle.dump(cookie_jar, handle)
except Exception:
pass
return cookie_jar, "redis"
except Exception:
pass
return None, "missing"
def get_juming_status() -> dict:
cookie_jar, storage = _load_juming_cookie()
status = {
"cookie_ready": cookie_jar is not None,
"cookie_storage": storage,
"cookie_file": str(JUMING_COOKIE_FILE),
"cookie_count": len(_cookie_jar_to_dict(cookie_jar)) if cookie_jar is not None else 0,
"jucha": _jucha_cookie_status(),
"supported_modes": [
{"label": "聚名一口价", "value": "fixed_price", "source_type": FIXED_PRICE_SOURCE_TYPE},
{"label": "聚名过期删除", "value": "delete_list", "source_type": DELETE_LIST_SOURCE_TYPE},
],
"defaults": get_juming_preferences(),
}
status["linked_jucha"] = {
"attempted": False,
"ok": bool(status["jucha"]["cookie_ready"]),
"message": "聚查登录态已就绪" if status["jucha"]["cookie_ready"] else "尚未检测到聚查 Cookie",
}
return status
def login_juming(email: str, password: str) -> dict:
account = str(email or "").strip()
secret = str(password or "").strip()
if not account or not secret:
raise ValueError("请输入聚名账号和密码")
jm = JM()
jm.load_cookies()
login_result = jm.user_zh_p_login(account, secret)
if not login_result[0]:
raise ValueError(f"聚名登录失败: {login_result[1]}")
jm.save_cookies()
_persist_juming_cookie(jm.cookie)
linked_jucha = {
"attempted": True,
"ok": False,
"message": "未执行",
}
try:
jc = JC()
jc.load_juming_cookies()
linked_ok, linked_message = jc.auth_login()
linked_jucha["ok"] = bool(linked_ok)
linked_jucha["message"] = str(linked_message)
if linked_ok:
jc.save_cookies()
except Exception as exc:
linked_jucha["message"] = f"聚查联名登录失败: {exc}"
status = get_juming_status()
status["linked_jucha"] = linked_jucha
return status
def login_jucha_with_juming_cookie() -> dict:
if not JUMING_COOKIE_FILE.exists():
raise ValueError("请先完成聚名登录，当前未检测到聚名 Cookie")
jc = JC()
jc.load_juming_cookies()
linked_ok, linked_message = jc.auth_login()
if not linked_ok:
raise ValueError(f"聚查登录失败: {linked_message}")
jc.save_cookies()
status = get_juming_status()
status["linked_jucha"] = {
"attempted": True,
"ok": True,
"message": str(linked_message),
}
return status
def upload_juming_cookie(filename: str, content: bytes) -> dict:
name = Path(filename or "juming_cookies.pkl").name.lower()
if not (name.endswith(".pkl") or name.endswith(".pickle") or name.endswith(".json") or name.endswith(".txt")):
raise ValueError("仅支持上传 .pkl / .pickle / .json / .txt 格式的聚名 Cookie 文件")
cookie_jar: RequestsCookieJar | None = None
parse_error: str | None = None
if name.endswith(".pkl") or name.endswith(".pickle"):
try:
loaded = pickle.loads(content)
if isinstance(loaded, RequestsCookieJar):
cookie_jar = loaded
elif isinstance(loaded, dict):
cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in loaded.items()})
except Exception as exc:
parse_error = str(exc)
else:
try:
text = content.decode("utf-8")
parsed = ast.literal_eval(text)
if isinstance(parsed, dict):
cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in parsed.items()})
except Exception as exc:
parse_error = str(exc)
if cookie_jar is None:
raise ValueError(f"聚名 Cookie 文件解析失败: {parse_error or '内容不符合预期'}")
if not _cookie_jar_to_dict(cookie_jar):
raise ValueError("聚名 Cookie 文件为空，未检测到有效 Cookie")
_persist_juming_cookie(cookie_jar)
return {
"cookie_ready": True,
"cookie_storage": "upload",
"cookie_file": str(JUMING_COOKIE_FILE),
"cookie_count": len(_cookie_jar_to_dict(cookie_jar)),
}
def _insert_domains(
domains: list[str],
source_type: int,
log: Callable[[str], None] | None = None,
should_stop: Callable[[], bool] | None = None,
) -> dict:
total = len(domains)
normalized_rows: list[tuple[str, str]] = []
invalid = 0
_emit_log(log, f"开始入库处理，共收到 {total} 个原始域名")
for value in domains:
_check_stop(should_stop)
normalized = normalize_domain(value)
if not normalized:
invalid += 1
continue
tld = normalized.rsplit(".", 1)[-1]
normalized_rows.append((normalized, tld))
existing_set: set[str] = set()
inserted = 0
with get_db() as conn:
with conn.cursor() as cur:
normalized_domains = [row[0] for row in normalized_rows]
if normalized_domains:
cur.execute("select domain from domains where domain = any(%s)", (normalized_domains,))
existing_set = {row[0] for row in cur.fetchall()}
if existing_set:
_emit_log(log, f"检测到 {len(existing_set)} 个已存在域名，将自动跳过")
inserted_since_commit = 0
for domain, tld in normalized_rows:
_check_stop(should_stop)
if domain in existing_set:
continue
cur.execute(
"""
insert into domains (
domain, tld, source_type, use_status, detect_status, register_status,
has_beian, company_type, website_url, beian_year, snapshot_years,
expire_date, create_time, update_time, review_status, detect_time,
backlink_count, jucha_status, juziseo_status
) values (
%s, %s, %s, 0, 0, 0,
1, null, null, null, null,
null, now(), now(), 0, null,
0, 0, 0
)
returning id
""",
(domain, tld, source_type),
)
domain_id = cur.fetchone()[0]
cur.execute(
"""
insert into detect_tasks (domain_id, task_type, status, priority, retry_count, create_time, update_time)
values (%s, 1, 1, 5, 0, now(), now())
""",
(domain_id,),
)
inserted += 1
inserted_since_commit += 1
if inserted_since_commit >= 500:
conn.commit()
inserted_since_commit = 0
conn.commit()
valid = len(normalized_rows)
exists = len(existing_set)
_emit_log(log, f"入库完成：有效 {valid}，新增 {inserted}，已存在 {exists}，无效 {invalid}")
return {
"total": total,
"valid": valid,
"added": inserted,
"exists": exists,
"invalid": invalid,
"failed": max(valid - exists - inserted, 0),
}
def _crawl_fixed_price(
page_start: int,
page_size: int,
page_count: int,
log: Callable[[str], None] | None = None,
should_stop: Callable[[], bool] | None = None,
) -> tuple[list[str], list[dict[str, int]]]:
cookie_jar, _ = _load_juming_cookie()
jm = JM()
jm.cookie = cookie_jar or RequestsCookieJar()
domains: list[str] = []
pages: list[dict[str, int]] = []
current_page = page_start
_emit_log(log, f"开始采集一口价域名：起始页 {page_start}，每页 {page_size}，最多 {page_count} 页")
for _ in range(page_count):
_check_stop(should_stop)
_emit_log(log, f"正在抓取第 {current_page} 页")
success, html = jm.ykj_get_list(page=current_page, page_size=page_size)
if not success:
raise RuntimeError(str(html))
page_domains = [item.strip() for item in YKJ_DOMAIN_PATTERN.findall(html) if item.strip()]
domains.extend(page_domains)
pages.append({"page": current_page, "count": len(page_domains)})
_emit_log(log, f"第 {current_page} 页抓取到 {len(page_domains)} 个域名，累计 {len(domains)} 个")
if len(page_domains) < page_size:
_emit_log(log, "当前页返回数量小于分页数量，判定已到末页，停止继续抓取")
break
current_page += 1
return domains, pages
def _crawl_delete_list(
crawl_date: str,
auto_date: bool,
log: Callable[[str], None] | None = None,
should_stop: Callable[[], bool] | None = None,
) -> tuple[list[str], list[dict[str, int]]]:
cookie_jar, _ = _load_juming_cookie()
jm = JM()
jm.cookie = cookie_jar or RequestsCookieJar()
start_date = datetime.strptime(crawl_date, "%Y-%m-%d").date()
end_date = date.today() + timedelta(days=4)
current_date = start_date
domains: list[str] = []
dates: list[dict[str, int]] = []
_emit_log(log, f"开始采集删除列表：起始日期 {crawl_date}，自动追加日期 {'开启' if auto_date else '关闭'}")
while current_date <= end_date:
_check_stop(should_stop)
_emit_log(log, f"正在抓取 {current_date.isoformat()} 的删除列表")
domains_for_date = [item.strip() for item in jm.new_cha_del(current_date.isoformat()) if item.strip()]
domains.extend(domains_for_date)
dates.append({"date": current_date.isoformat(), "count": len(domains_for_date)})
_emit_log(log, f"{current_date.isoformat()} 抓取到 {len(domains_for_date)} 个域名，累计 {len(domains)} 个")
if not auto_date:
break
current_date += timedelta(days=1)
return domains, dates
def crawl_juming(
payload: dict,
log: Callable[[str], None] | None = None,
should_stop: Callable[[], bool] | None = None,
) -> dict:
mode = str(payload.get("mode") or "delete_list").strip()
if mode not in {"fixed_price", "delete_list"}:
raise ValueError("仅支持 fixed_price 或 delete_list")
cookie_jar, storage = _load_juming_cookie()
if cookie_jar is None:
raise ValueError("未找到聚名 Cookie，请先在桌面版系统设置完成聚名登录，或将 Cookie 同步到服务器")
_emit_log(log, f"检测到聚名登录态，来源：{storage}")
_check_stop(should_stop)
if mode == "fixed_price":
page_start = max(int(payload.get("page_start") or 1), 1)
page_size = min(max(int(payload.get("page_size") or 500), 1), 1000)
page_count = min(max(int(payload.get("page_count") or 1), 1), 20)
domains, pages = _crawl_fixed_price(page_start, page_size, page_count, log=log, should_stop=should_stop)
stats = _insert_domains(domains, FIXED_PRICE_SOURCE_TYPE, log=log, should_stop=should_stop)
return {
"mode": mode,
"cookie_storage": storage,
"pages": pages,
"domains_found": len(domains),
"stats": stats,
"sample_domains": domains[:20],
}
crawl_date = str(payload.get("crawl_date") or date.today().isoformat())
auto_date = bool(payload.get("auto_date", True))
domains, dates = _crawl_delete_list(crawl_date, auto_date, log=log, should_stop=should_stop)
stats = _insert_domains(domains, DELETE_LIST_SOURCE_TYPE, log=log, should_stop=should_stop)
return {
"mode": mode,
"cookie_storage": storage,
"dates": dates,
"domains_found": len(domains),
"stats": stats,
"sample_domains": domains[:20],
}

206
domain-api/app/services/juming_task_service.py Normal file
View File

@@ -0,0 +1,206 @@
from __future__ import annotations
import threading
from datetime import datetime
from uuid import uuid4
from app.core.files import load_juming_records, save_juming_records
from app.services.juming_service import TaskStoppedError, crawl_juming
_JUMING_TASK_LOCK = threading.Lock()
_MAX_LOG_LINES = 400
def _now() -> str:
return datetime.now().isoformat(sep=" ", timespec="seconds")
def list_juming_tasks() -> list[dict]:
return load_juming_records()
def _save_tasks(tasks: list[dict]) -> None:
save_juming_records(tasks)
def _append_log(task_id: str, message: str) -> None:
with _JUMING_TASK_LOCK:
tasks = load_juming_records()
target = next((item for item in tasks if item["task_id"] == task_id), None)
if not target:
return
logs = list(target.get("logs") or [])
logs.append(f"[{_now()}] {message}")
target["logs"] = logs[-_MAX_LOG_LINES:]
target["updated_at"] = _now()
_save_tasks(tasks)
def _update_task(task_id: str, **patch: object) -> dict | None:
with _JUMING_TASK_LOCK:
tasks = load_juming_records()
target = next((item for item in tasks if item["task_id"] == task_id), None)
if not target:
return None
target.update(patch)
target["updated_at"] = _now()
_save_tasks(tasks)
return dict(target)
def _is_stop_requested(task_id: str) -> bool:
tasks = load_juming_records()
target = next((item for item in tasks if item["task_id"] == task_id), None)
return bool(target and target.get("cancel_requested"))
def _set_phase(task_id: str, phase: str, message: str | None = None) -> None:
phase_labels = {
"queued": "排队中",
"starting": "启动中",
"fetching": "抓取中",
"importing": "入库中",
"completed": "已完成",
"failed": "失败",
"stopping": "停止中",
"stopped": "已停止",
}
patch: dict[str, object] = {
"phase": phase,
"phase_label": phase_labels.get(phase, phase),
}
if message:
patch["message"] = message
_update_task(task_id, **patch)
def _log_and_track_phase(task_id: str, message: str) -> None:
if "开始采集" in message or "正在抓取" in message:
_set_phase(task_id, "fetching", message)
elif "开始入库处理" in message or "入库完成" in message or "已存在域名" in message:
_set_phase(task_id, "importing", message)
_append_log(task_id, message)
def _run_juming_task(task_id: str, payload: dict) -> None:
_update_task(task_id, status="running", started_at=_now(), message="聚名采集任务开始执行", cancel_requested=False)
_set_phase(task_id, "starting", "聚名采集任务开始执行")
_append_log(task_id, "任务已启动，正在准备读取聚名登录态")
try:
result = crawl_juming(
payload,
log=lambda message: _log_and_track_phase(task_id, str(message)),
should_stop=lambda: _is_stop_requested(task_id),
)
stats = result.get("stats", {})
_update_task(
task_id,
status="completed",
completed_at=_now(),
result=result,
cancel_requested=False,
message=(
f"采集完成：抓取 {result.get('domains_found', 0)} 个域名，"
f"新增 {stats.get('added', 0)}，已存在 {stats.get('exists', 0)}，无效 {stats.get('invalid', 0)}"
),
)
_set_phase(task_id, "completed")
_append_log(task_id, "任务执行完成")
except TaskStoppedError as exc:
_update_task(
task_id,
status="stopped",
completed_at=_now(),
cancel_requested=False,
message=str(exc),
)
_set_phase(task_id, "stopped")
_append_log(task_id, "任务已按请求停止")
except Exception as exc:
_update_task(
task_id,
status="failed",
completed_at=_now(),
cancel_requested=False,
message=f"采集失败：{exc}",
)
_set_phase(task_id, "failed")
_append_log(task_id, f"任务执行失败：{exc}")
def create_juming_task(payload: dict) -> dict:
task_id = uuid4().hex
mode = str(payload.get("mode") or "delete_list").strip() or "delete_list"
record = {
"task_id": task_id,
"mode": mode,
"payload": dict(payload or {}),
"status": "queued",
"phase": "queued",
"phase_label": "排队中",
"cancel_requested": False,
"message": "采集任务已创建，等待后台执行",
"created_at": _now(),
"updated_at": _now(),
"started_at": "",
"completed_at": "",
"result": None,
"logs": [f"[{_now()}] 已创建聚名采集任务，等待后台执行"],
}
with _JUMING_TASK_LOCK:
tasks = load_juming_records()
tasks.insert(0, record)
_save_tasks(tasks)
worker = threading.Thread(target=_run_juming_task, args=(task_id, dict(payload or {})), daemon=True)
worker.start()
return record
def retry_juming_task(task_id: str) -> dict:
with _JUMING_TASK_LOCK:
tasks = load_juming_records()
target = next((item for item in tasks if item["task_id"] == task_id), None)
if not target:
raise ValueError("聚名采集任务不存在")
if target.get("status") == "running":
raise ValueError("聚名采集任务正在运行，不能重复执行")
target["status"] = "queued"
target["phase"] = "queued"
target["phase_label"] = "排队中"
target["cancel_requested"] = False
target["message"] = "采集任务已重新加入队列"
target["started_at"] = ""
target["completed_at"] = ""
target["updated_at"] = _now()
target["result"] = None
target["logs"] = [f"[{_now()}] 已重新加入队列，等待后台执行"]
payload = dict(target.get("payload") or {})
record = dict(target)
_save_tasks(tasks)
worker = threading.Thread(target=_run_juming_task, args=(task_id, payload), daemon=True)
worker.start()
return record
def request_stop_juming_task(task_id: str) -> dict:
with _JUMING_TASK_LOCK:
tasks = load_juming_records()
target = next((item for item in tasks if item["task_id"] == task_id), None)
if not target:
raise ValueError("聚名采集任务不存在")
if target.get("status") in {"completed", "failed", "stopped"}:
raise ValueError("当前任务已结束，无需停止")
target["cancel_requested"] = True
target["phase"] = "stopping"
target["phase_label"] = "停止中"
target["message"] = "已发送停止请求，等待当前步骤安全退出"
target["updated_at"] = _now()
logs = list(target.get("logs") or [])
logs.append(f"[{_now()}] 已收到停止请求，等待当前步骤安全退出")
target["logs"] = logs[-_MAX_LOG_LINES:]
_save_tasks(tasks)
return dict(target)

111
domain-api/app/services/juziseo_service.py Normal file
View File

@@ -0,0 +1,111 @@
from __future__ import annotations
import ast
import pickle
import sys
from pathlib import Path
from requests.cookies import RequestsCookieJar
from app.core.config import settings
from app.core.redis_client import get_redis
DOMAIN_ROOT = Path(settings.domain_root)
if str(DOMAIN_ROOT) not in sys.path:
sys.path.insert(0, str(DOMAIN_ROOT))
from detect.juziseo import Juziseo # type: ignore # noqa: E402
JUZISEO_COOKIE_FILE = DOMAIN_ROOT / "juziseo_cookies.pkl"
def _cookie_dict_to_jar(cookie_dict: dict[str, str]) -> RequestsCookieJar:
cookie_jar = RequestsCookieJar()
for name, value in cookie_dict.items():
cookie_jar.set(name, value)
return cookie_jar
def _cookie_jar_to_dict(cookie_jar: RequestsCookieJar) -> dict[str, str]:
return {str(cookie.name): str(cookie.value) for cookie in cookie_jar}
def _persist_juziseo_cookie(cookie_jar: RequestsCookieJar) -> None:
JUZISEO_COOKIE_FILE.parent.mkdir(parents=True, exist_ok=True)
with JUZISEO_COOKIE_FILE.open("wb") as handle:
pickle.dump(cookie_jar, handle)
try:
redis_client = get_redis()
redis_client.set("domain_tool:juziseo_cookies", str(_cookie_jar_to_dict(cookie_jar)))
except Exception:
pass
def _load_juziseo_cookie() -> tuple[RequestsCookieJar | None, str]:
if JUZISEO_COOKIE_FILE.exists():
try:
with JUZISEO_COOKIE_FILE.open("rb") as handle:
loaded = pickle.load(handle)
if isinstance(loaded, RequestsCookieJar):
return loaded, "local"
if isinstance(loaded, dict):
return _cookie_dict_to_jar({str(k): str(v) for k, v in loaded.items()}), "local"
except Exception:
pass
try:
redis_client = get_redis()
raw = redis_client.get("domain_tool:juziseo_cookies")
if raw:
parsed = ast.literal_eval(raw)
if isinstance(parsed, dict) and parsed:
cookie_jar = _cookie_dict_to_jar({str(k): str(v) for k, v in parsed.items()})
try:
with JUZISEO_COOKIE_FILE.open("wb") as handle:
pickle.dump(cookie_jar, handle)
except Exception:
pass
return cookie_jar, "redis"
except Exception:
pass
return None, "missing"
def get_juziseo_status() -> dict:
cookie_jar, storage = _load_juziseo_cookie()
return {
"cookie_ready": cookie_jar is not None,
"cookie_storage": storage,
"cookie_file": str(JUZISEO_COOKIE_FILE),
"cookie_count": len(_cookie_jar_to_dict(cookie_jar)) if cookie_jar is not None else 0,
}
def login_juziseo(email: str, password: str) -> dict:
account = str(email or "").strip()
secret = str(password or "").strip()
if not account or not secret:
raise ValueError("请输入桔子SEO账号和密码")
juziseo = Juziseo()
juziseo.load_cookies(str(JUZISEO_COOKIE_FILE))
ok, message = juziseo.login(account, secret)
if not ok:
raise ValueError(f"桔子SEO登录失败: {message}")
try:
juziseo.save_cookies(str(JUZISEO_COOKIE_FILE))
except Exception:
pass
try:
_persist_juziseo_cookie(juziseo.cookie)
except Exception:
pass
status = get_juziseo_status()
status["message"] = str(message)
return status

50
domain-api/app/services/runtime_control_service.py
View File

@@ -5,8 +5,9 @@ import subprocess
from pathlib import Path from pathlib import Path
from app.core.config import settings from app.core.config import settings
from app.services.sync_push_service import push_runtime_projection_now
from app.services.runtime_settings_service import get_runtime_settings from app.services.runtime_settings_service import get_runtime_settings
from app.services.worker_control_service import start_worker, stop_worker from app.services.worker_control_service import _run_systemctl, start_worker, stop_worker
def _workspace_root() -> Path: def _workspace_root() -> Path:
@@ -23,7 +24,9 @@ def restart_api() -> tuple[bool, str]:
worker_mode = runtime.get("worker_mode", settings.worker_mode) worker_mode = runtime.get("worker_mode", settings.worker_mode)
if worker_mode == "linux-systemd": if worker_mode == "linux-systemd":
result = _run_shell(["systemctl", "restart", api_service_name], timeout=30) # When the API restarts itself under systemd, wait-free restart avoids
# blocking the HTTP request until uvicorn is torn down.
result = _run_systemctl(["restart", api_service_name], timeout=5, no_block=True)
if result.returncode != 0: if result.returncode != 0:
return False, (result.stderr or result.stdout or "重启 Linux API 失败").strip() return False, (result.stderr or result.stdout or "重启 Linux API 失败").strip()
return True, f"Linux API 重启命令已发送: {api_service_name}" return True, f"Linux API 重启命令已发送: {api_service_name}"
@@ -48,6 +51,35 @@ def restart_api() -> tuple[bool, str]:
return True, "API 重启命令已发送" return True, "API 重启命令已发送"
def _run_systemd_action(service_name: str, action: str, *, no_block: bool = False) -> tuple[bool, str]:
systemctl_command = []
if no_block:
systemctl_command.append("--no-block")
systemctl_command.extend([action, service_name])
result = _run_systemctl(systemctl_command, timeout=10 if no_block else 30)
if result.returncode != 0:
return False, (result.stderr or result.stdout or f"{action} {service_name} 失败").strip()
return True, f"{service_name} {action} 命令已发送"
def start_sync_agent() -> tuple[bool, str]:
runtime = get_runtime_settings()
worker_mode = runtime.get("worker_mode", settings.worker_mode)
service_name = runtime.get("sync_agent_service_name", settings.sync_agent_service_name)
if worker_mode != "linux-systemd":
return False, "sync-agent 仅在 Linux systemd 多机部署中使用。"
return _run_systemd_action(service_name, "start")
def stop_sync_agent() -> tuple[bool, str]:
runtime = get_runtime_settings()
worker_mode = runtime.get("worker_mode", settings.worker_mode)
service_name = runtime.get("sync_agent_service_name", settings.sync_agent_service_name)
if worker_mode != "linux-systemd":
return False, "sync-agent 仅在 Linux systemd 多机部署中使用。"
return _run_systemd_action(service_name, "stop")
def runtime_action(action: str) -> tuple[bool, str, dict]: def runtime_action(action: str) -> tuple[bool, str, dict]:
if action == "start_worker": if action == "start_worker":
ok, message = start_worker() ok, message = start_worker()
@@ -58,6 +90,20 @@ def runtime_action(action: str) -> tuple[bool, str, dict]:
if action == "restart_api": if action == "restart_api":
ok, message = restart_api() ok, message = restart_api()
return ok, message, {"action": action, "poll_after_seconds": 4, "refresh_runtime": True} return ok, message, {"action": action, "poll_after_seconds": 4, "refresh_runtime": True}
if action == "start_sync_agent":
ok, message = start_sync_agent()
return ok, message, {"action": action, "poll_after_seconds": 2, "refresh_runtime": True}
if action == "stop_sync_agent":
ok, message = stop_sync_agent()
return ok, message, {"action": action, "poll_after_seconds": 2, "refresh_runtime": True}
if action == "push_sync":
ok, message, data = push_runtime_projection_now()
return ok, message, {
"action": action,
"poll_after_seconds": 2,
"refresh_runtime": True,
**(data or {}),
}
return False, f"不支持的运行时动作: {action}", { return False, f"不支持的运行时动作: {action}", {
"action": action, "action": action,
"poll_after_seconds": 0, "poll_after_seconds": 0,

1
domain-api/app/services/runtime_settings_service.py
View File

@@ -8,6 +8,7 @@ DEFAULT_RUNTIME_SETTINGS = {
"worker_mode": settings.worker_mode, "worker_mode": settings.worker_mode,
"worker_service_name": settings.worker_service_name, "worker_service_name": settings.worker_service_name,
"api_service_name": settings.api_service_name, "api_service_name": settings.api_service_name,
"sync_agent_service_name": settings.sync_agent_service_name,
} }

271
domain-api/app/services/runtime_status_service.py
View File

@@ -5,9 +5,14 @@ from pathlib import Path
from app.core.config import settings from app.core.config import settings
from app.core.db import get_db from app.core.db import get_db
from app.core.files import read_json
from app.core.redis_client import get_redis from app.core.redis_client import get_redis
from app.services.cluster_runtime_service import get_cluster_snapshot
from app.services.detect_service import get_detect_status
from app.services.detect_job_service import get_detect_capacity_plan, get_detect_queue_health
from app.services.sync_record_service import append_runtime_projection_if_changed, get_sync_summary
from app.services.runtime_settings_service import get_runtime_settings from app.services.runtime_settings_service import get_runtime_settings
from app.services.worker_control_service import detect_worker_runtime from app.services.worker_control_service import detect_sync_agent_runtime, detect_worker_runtime
def _runtime_log_path(filename: str) -> str: def _runtime_log_path(filename: str) -> str:
@@ -15,10 +20,195 @@ def _runtime_log_path(filename: str) -> str:
return str(path) return str(path)
def _domain_cookie_status(filename: str) -> tuple[bool, str]:
path = Path(settings.domain_root) / filename
return path.exists(), str(path)
def _bloom_filter_status() -> tuple[bool, str]:
try:
redis_client = get_redis()
modules = redis_client.execute_command("MODULE", "LIST")
for module in modules or []:
module_parts = module[::2]
module_values = module[1::2]
module_info = dict(zip(module_parts, module_values))
module_name = str(module_info.get("name", "")).lower()
if module_name in {"bf", "redisbloom"}:
return True, "RedisBloom 已安装"
return False, "Redis 未安装 RedisBloom，当前将使用普通缓存"
except Exception as exc:
return False, f"RedisBloom 检查失败: {exc}"
def _build_multi_region_readiness(
*,
cluster_snapshot: dict,
sync_summary: dict,
worker_runtime: dict,
sync_agent_runtime: dict,
) -> dict:
nodes = list(cluster_snapshot.get("nodes") or [])
summary = cluster_snapshot.get("summary") or {}
batch_summary = (sync_summary.get("detect_result_batches") or {})
batch_states = batch_summary.get("state_counts") or {}
online_control_nodes = int(summary.get("online_control_nodes", 0) or 0)
online_worker_nodes = int(summary.get("online_worker_nodes", 0) or 0)
mainland_control_nodes = [
node for node in nodes
if str(node.get("region") or "") == "mainland"
and str(node.get("role") or "") == "control"
and str(node.get("status") or "") in {"online", "busy"}
]
mainland_worker_nodes = [
node for node in nodes
if str(node.get("region") or "") == "mainland"
and str(node.get("role") or "") == "worker"
and str(node.get("status") or "") in {"online", "busy"}
]
blocking_issues: list[str] = []
warning_issues: list[str] = []
info_items: list[str] = []
if online_control_nodes <= 0:
blocking_issues.append("当前没有在线控制面节点，无法视为正式可用集群。")
if settings.node_region == "mainland" and settings.node_role == "control":
if not sync_agent_runtime.get("running", False):
blocking_issues.append("当前节点应承载 sync-agent，但服务未运行。")
if str(settings.sync_target_api_base_url or "").strip() == "":
blocking_issues.append("当前节点未配置 SYNC_TARGET_API_BASE_URL，无法向海外控制面推送。")
if not bool(settings.sync_push_enabled):
blocking_issues.append("当前节点未启用 SYNC_PUSH_ENABLED，结果同步不会自动执行。")
if settings.node_region == "overseas" and online_control_nodes > 0 and not mainland_control_nodes:
warning_issues.append("当前尚未观察到在线的大陆 controller 节点，后续自动结果同步仍未进入正式双地域态。")
if online_worker_nodes <= 0:
warning_issues.append("当前没有在线 Worker 节点，检测任务无法在多机状态下继续推进。")
offline_nodes = list(summary.get("offline_nodes") or [])
stale_nodes = list(summary.get("stale_nodes") or [])
if stale_nodes:
warning_issues.append(f"存在失活节点: {'、'.join(stale_nodes)}")
if offline_nodes:
warning_issues.append(f"存在离线节点: {'、'.join(offline_nodes)}")
failed_batches = int(batch_states.get("failed", 0) or 0)
projected_batches = int(batch_states.get("projected", 0) or 0)
pushing_batches = int(batch_states.get("pushing", 0) or 0)
synced_batches = int(batch_states.get("synced", 0) or 0)
if failed_batches > 0:
warning_issues.append(f"存在 {failed_batches} 个结果批次同步失败，需要检查 sync-agent 或目标接收面。")
if projected_batches > 0:
warning_issues.append(f"存在 {projected_batches} 个结果批次仍待推送。")
if pushing_batches > 0:
info_items.append(f"当前有 {pushing_batches} 个结果批次正在推送。")
if synced_batches > 0:
info_items.append(f"最近已接收 {synced_batches} 个结果批次。")
if worker_runtime.get("running", False):
info_items.append("当前节点本机 Worker 进程在线。")
if settings.node_region == "mainland" and settings.node_role == "control" and sync_agent_runtime.get("running", False):
info_items.append("当前节点本机 sync-agent 在线。")
if mainland_worker_nodes:
info_items.append(f"在线大陆 Worker {len(mainland_worker_nodes)} 台。")
if mainland_control_nodes:
info_items.append(f"在线大陆 controller {len(mainland_control_nodes)} 台。")
if blocking_issues:
status = "blocking"
summary_text = blocking_issues[0]
elif warning_issues:
status = "attention"
summary_text = warning_issues[0]
else:
status = "ready"
summary_text = "当前多机与跨地域骨架已进入可联调、可持续观察状态。"
return {
"status": status,
"ready": status == "ready",
"summary": summary_text,
"blocking_issues": blocking_issues,
"warnings": warning_issues,
"info": info_items,
"cluster": {
"online_control_nodes": online_control_nodes,
"online_worker_nodes": online_worker_nodes,
"mainland_control_nodes": len(mainland_control_nodes),
"mainland_worker_nodes": len(mainland_worker_nodes),
},
"sync": {
"enabled": bool(sync_summary.get("enabled", False)),
"source_region": sync_summary.get("source_region", ""),
"target_region": sync_summary.get("target_region", ""),
"projected_batches": projected_batches,
"pushing_batches": pushing_batches,
"failed_batches": failed_batches,
"synced_batches": synced_batches,
},
}
def get_runtime_status() -> dict: def get_runtime_status() -> dict:
runtime_settings = get_runtime_settings() runtime_settings = get_runtime_settings()
worker_runtime = detect_worker_runtime() worker_runtime = detect_worker_runtime()
sync_agent_runtime = detect_sync_agent_runtime()
api_pid = os.getpid() api_pid = os.getpid()
detect_snapshot = get_detect_status()
latest_run = (detect_snapshot.get("runs") or [None])[0] or {}
cluster_snapshot = get_cluster_snapshot()
queue_health = get_detect_queue_health(window_minutes=15)
effective_online_worker_nodes = int((cluster_snapshot.get("summary") or {}).get("online_worker_nodes", 0) or 0)
if effective_online_worker_nodes <= 0 and worker_runtime.get("running", False):
effective_online_worker_nodes = max(1, worker_runtime.get("process_count", 1) or 1)
capacity_plan = get_detect_capacity_plan(
queue_health=queue_health,
online_worker_nodes=effective_online_worker_nodes,
target_finish_hours=6,
)
detect_payload = {
"phase_label": latest_run.get("phase_label", ""),
"phase_detail": latest_run.get("phase_detail", ""),
"recent_event": detect_snapshot.get("recent_event", ""),
"recent_warning": detect_snapshot.get("recent_warning", ""),
"progress_percent": detect_snapshot.get("progress_percent", 0),
"progress": detect_snapshot.get("progress", {}),
"active_thread_count": detect_snapshot.get("active_thread_count", 0),
"max_thread_count": detect_snapshot.get("max_thread_count", 0),
"available_proxy_count": detect_snapshot.get("available_proxy_count", 0),
"proxy_pool_count": detect_snapshot.get("proxy_pool_count", 0),
"proxy_runtime_label": detect_snapshot.get("proxy_runtime_label", ""),
"proxy_runtime_detail": detect_snapshot.get("proxy_runtime_detail", ""),
"proxy_runtime_reason": detect_snapshot.get("proxy_runtime_reason", ""),
"proxy_supplier_empty": detect_snapshot.get("proxy_supplier_empty", False),
"proxy_last_refresh_status": detect_snapshot.get("proxy_last_refresh_status", ""),
"proxy_last_refresh_time": detect_snapshot.get("proxy_last_refresh_time", ""),
"proxy_last_refresh_source_count": detect_snapshot.get("proxy_last_refresh_source_count", 0),
"proxy_last_refresh_total_items": detect_snapshot.get("proxy_last_refresh_total_items", 0),
"proxy_last_validated_count": detect_snapshot.get("proxy_last_validated_count", 0),
"proxy_last_available_count": detect_snapshot.get("proxy_last_available_count", 0),
"proxy_source_stats": detect_snapshot.get("proxy_source_stats", []),
"dependency_alerts": detect_snapshot.get("dependency_alerts", []),
"active_job": detect_snapshot.get("active_job"),
"runs_count": len(detect_snapshot.get("runs") or []),
"worker_online": worker_runtime.get("running", False),
"worker_mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
"queue_health": queue_health,
"capacity_plan": capacity_plan,
}
append_runtime_projection_if_changed(detect=detect_payload, cluster=cluster_snapshot)
sync_summary = get_sync_summary(record_limit=5)
readiness = _build_multi_region_readiness(
cluster_snapshot=cluster_snapshot,
sync_summary=sync_summary,
worker_runtime=worker_runtime,
sync_agent_runtime=sync_agent_runtime,
)
return { return {
"api": { "api": {
@@ -34,6 +224,11 @@ def get_runtime_status() -> dict:
"stdout_log": _runtime_log_path("domain-api.stdout.log"), "stdout_log": _runtime_log_path("domain-api.stdout.log"),
"stderr_log": _runtime_log_path("domain-api.stderr.log"), "stderr_log": _runtime_log_path("domain-api.stderr.log"),
}, },
"node": {
"code": settings.node_code,
"region": settings.node_region,
"role": settings.node_role,
},
"worker": { "worker": {
"mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")), "mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
"service_name": runtime_settings.get("worker_service_name", settings.worker_service_name), "service_name": runtime_settings.get("worker_service_name", settings.worker_service_name),
@@ -43,12 +238,26 @@ def get_runtime_status() -> dict:
"message": worker_runtime.get("message", ""), "message": worker_runtime.get("message", ""),
"log_path": str(Path(settings.domain_root) / "detect_worker.log"), "log_path": str(Path(settings.domain_root) / "detect_worker.log"),
}, },
"sync_agent": {
"mode": sync_agent_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
"service_name": runtime_settings.get("sync_agent_service_name", settings.sync_agent_service_name),
"running": sync_agent_runtime.get("running", False),
"process_count": sync_agent_runtime.get("process_count", 0),
"latest_start_time": sync_agent_runtime.get("latest_start_time", ""),
"message": sync_agent_runtime.get("message", ""),
"expected_on_this_node": settings.node_region == "mainland" and settings.node_role == "control",
},
"detect": detect_payload,
"cluster": cluster_snapshot,
"sync": sync_summary,
"readiness": readiness,
} }
def get_runtime_preflight() -> dict: def get_runtime_preflight() -> dict:
runtime_settings = get_runtime_settings() runtime_settings = get_runtime_settings()
checks: list[dict[str, object]] = [] checks: list[dict[str, object]] = []
detect_options = read_json("detect_options.json", default={})
domain_root = Path(settings.domain_root) domain_root = Path(settings.domain_root)
checks.append( checks.append(
@@ -75,10 +284,62 @@ def get_runtime_preflight() -> dict:
checks.append({"key": "redis", "label": "Redis", "ok": True, "message": f"{settings.redis_host}:{settings.redis_port}/{settings.redis_db}"}) checks.append({"key": "redis", "label": "Redis", "ok": True, "message": f"{settings.redis_host}:{settings.redis_port}/{settings.redis_db}"})
except Exception as exc: except Exception as exc:
checks.append({"key": "redis", "label": "Redis", "ok": False, "message": str(exc)}) checks.append({"key": "redis", "label": "Redis", "ok": False, "message": str(exc)})
else:
bloom_ok, bloom_message = _bloom_filter_status()
checks.append(
{
"key": "redis_bloom",
"label": "RedisBloom",
"ok": True,
"message": bloom_message,
"level": "info" if bloom_ok else "warn",
"degraded": not bloom_ok,
}
)
worker_mode = runtime_settings.get("worker_mode", "windows-local") worker_mode = runtime_settings.get("worker_mode", "windows-local")
checks.append({"key": "worker_mode", "label": "运行模式", "ok": True, "message": worker_mode}) checks.append({"key": "worker_mode", "label": "运行模式", "ok": True, "message": worker_mode})
jucha_enabled = bool(detect_options.get("detect_jucha"))
jucha_cookie_ok, jucha_cookie_path = _domain_cookie_status("jucha_cookies.pkl")
checks.append(
{
"key": "detect_jucha",
"label": "聚查检测",
"ok": True if not jucha_enabled else jucha_cookie_ok,
"message": "已启用" if jucha_enabled else "未启用",
}
)
checks.append(
{
"key": "jucha_cookie",
"label": "聚查 Cookie",
"ok": True if not jucha_enabled else jucha_cookie_ok,
"message": jucha_cookie_path if jucha_enabled else "未启用聚查检测，无需本地 Cookie",
"level": "info" if (not jucha_enabled or jucha_cookie_ok) else "warn",
}
)
juziseo_enabled = bool(detect_options.get("detect_juziseo"))
juziseo_cookie_ok, juziseo_cookie_path = _domain_cookie_status("juziseo_cookies.pkl")
checks.append(
{
"key": "detect_juziseo",
"label": "桔子SEO检测",
"ok": True if not juziseo_enabled else juziseo_cookie_ok,
"message": "已启用" if juziseo_enabled else "未启用",
}
)
checks.append(
{
"key": "juziseo_cookie",
"label": "桔子SEO Cookie",
"ok": True if not juziseo_enabled else juziseo_cookie_ok,
"message": juziseo_cookie_path if juziseo_enabled else "未启用桔子SEO检测，无需本地 Cookie",
"level": "info" if (not juziseo_enabled or juziseo_cookie_ok) else "warn",
}
)
if worker_mode == "linux-systemd": if worker_mode == "linux-systemd":
checks.append( checks.append(
{ {
@@ -96,6 +357,14 @@ def get_runtime_preflight() -> dict:
"message": runtime_settings.get("api_service_name", ""), "message": runtime_settings.get("api_service_name", ""),
} }
) )
checks.append(
{
"key": "sync_agent_service_name",
"label": "Sync agent service 名",
"ok": bool(runtime_settings.get("sync_agent_service_name")),
"message": runtime_settings.get("sync_agent_service_name", ""),
}
)
else: else:
checks.append( checks.append(
{ {

60
domain-api/app/services/sensitive_words_service.py Normal file
View File

@@ -0,0 +1,60 @@
from __future__ import annotations
from app.core.db import get_db
def get_sensitive_words_payload() -> dict:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
select word, category, priority
from sensitive_words
order by priority desc, word asc
"""
)
rows = cur.fetchall()
items = [
{
"word": row[0],
"category": row[1] or "default",
"priority": row[2] or 1,
}
for row in rows
]
return {
"items": items,
"text": "\n".join(item["word"] for item in items),
"total": len(items),
}
def save_sensitive_words_payload(payload: dict) -> dict:
raw_text = str(payload.get("text") or "")
words = []
seen: set[str] = set()
for line in raw_text.splitlines():
word = line.strip()
if not word or word in seen:
continue
seen.add(word)
words.append(word)
with get_db() as conn:
with conn.cursor() as cur:
cur.execute("delete from sensitive_words")
if words:
cur.executemany(
"""
insert into sensitive_words (word, category, priority, create_time)
values (%s, 'default', 1, now())
""",
[(word,) for word in words],
)
conn.commit()
return {
"total": len(words),
"text": "\n".join(words),
}

165
domain-api/app/services/settings_service.py
View File

@@ -3,6 +3,7 @@ from __future__ import annotations
import json import json
from datetime import datetime from datetime import datetime
from app.core.config import settings as app_settings
from app.core.files import read_json, settings_backup_root, write_json from app.core.files import read_json, settings_backup_root, write_json
from app.core.redis_client import get_redis from app.core.redis_client import get_redis
from app.services.runtime_settings_service import get_runtime_settings, update_runtime_settings from app.services.runtime_settings_service import get_runtime_settings, update_runtime_settings
@@ -12,6 +13,8 @@ REDIS_KEYS = {
"detect_options": "domain_tool:detect_options", "detect_options": "domain_tool:detect_options",
"proxy_config": "domain_tool:proxy_config", "proxy_config": "domain_tool:proxy_config",
"thread_count": "domain_tool:thread_count", "thread_count": "domain_tool:thread_count",
"node_thread_counts": "domain_tool:node_thread_counts",
"credentials": "domain_tool:credentials",
} }
DETECT_OPTION_KEYS = { DETECT_OPTION_KEYS = {
@@ -26,10 +29,90 @@ DETECT_OPTION_KEYS = {
} }
def _normalize_thread_count(value: object, *, field_name: str = "thread_count") -> int:
try:
thread_count = int(value)
except Exception as exc:
raise ValueError(f"{field_name} must be an integer") from exc
if thread_count < 1 or thread_count > 256:
raise ValueError(f"{field_name} out of range")
return thread_count
def _normalize_node_thread_counts(payload: object) -> dict[str, int]:
if payload in (None, ""):
return {}
if not isinstance(payload, dict):
raise ValueError("node_thread_counts must be an object")
normalized: dict[str, int] = {}
for raw_node_code, raw_thread_count in payload.items():
node_code = str(raw_node_code or "").strip()
if not node_code:
raise ValueError("node_thread_counts contains empty node code")
normalized[node_code] = _normalize_thread_count(raw_thread_count, field_name=f"node_thread_counts.{node_code}")
return normalized
def _load_thread_count_config() -> tuple[int, dict[str, int]]:
thread_count_payload = read_json("thread_count.json", default={"thread_count": "2"})
node_thread_counts_payload = read_json("node_thread_counts.json", default={})
try:
default_thread_count = _normalize_thread_count(thread_count_payload.get("thread_count", 2))
except ValueError:
default_thread_count = 2
try:
node_thread_counts = _normalize_node_thread_counts(node_thread_counts_payload)
except ValueError:
node_thread_counts = {}
redis_client = get_redis()
try:
if redis_thread_count := redis_client.get(REDIS_KEYS["thread_count"]):
try:
default_thread_count = _normalize_thread_count(redis_thread_count)
except ValueError:
pass
if redis_node_thread_counts := redis_client.get(REDIS_KEYS["node_thread_counts"]):
try:
node_thread_counts = _normalize_node_thread_counts(json.loads(redis_node_thread_counts))
except ValueError:
pass
except Exception:
pass
return default_thread_count, node_thread_counts
def resolve_thread_count(node_code: str | None = None, settings_payload: dict | None = None) -> dict:
payload = settings_payload or get_settings_payload()
default_thread_count = int(payload.get("thread_count", 2))
node_thread_counts = _normalize_node_thread_counts(payload.get("node_thread_counts", {}))
normalized_node_code = str(node_code or app_settings.node_code or "").strip()
override_thread_count = None
source = "default"
effective_thread_count = default_thread_count
if normalized_node_code and normalized_node_code in node_thread_counts:
override_thread_count = node_thread_counts[normalized_node_code]
effective_thread_count = override_thread_count
source = "node_override"
return {
"node_code": normalized_node_code,
"default_thread_count": default_thread_count,
"effective_thread_count": effective_thread_count,
"override_thread_count": override_thread_count,
"source": source,
"node_thread_counts": node_thread_counts,
}
def get_settings_payload() -> dict: def get_settings_payload() -> dict:
detect_options = read_json("detect_options.json", default={}) detect_options = read_json("detect_options.json", default={})
proxy_config = read_json("proxy_config.json", default={}) proxy_config = read_json("proxy_config.json", default={})
thread_count = read_json("thread_count.json", default={"thread_count": "2"}) thread_count, node_thread_counts = _load_thread_count_config()
redis_client = get_redis() redis_client = get_redis()
try: try:
@@ -37,29 +120,84 @@ def get_settings_payload() -> dict:
detect_options = json.loads(redis_detect_options) detect_options = json.loads(redis_detect_options)
if redis_proxy_config := redis_client.get(REDIS_KEYS["proxy_config"]): if redis_proxy_config := redis_client.get(REDIS_KEYS["proxy_config"]):
proxy_config = json.loads(redis_proxy_config) proxy_config = json.loads(redis_proxy_config)
if redis_thread_count := redis_client.get(REDIS_KEYS["thread_count"]):
thread_count = {"thread_count": str(redis_thread_count)}
except Exception: except Exception:
pass pass
return { return {
"detect_options": detect_options, "detect_options": detect_options,
"proxy_config": proxy_config, "proxy_config": proxy_config,
"thread_count": int(thread_count.get("thread_count", 2)), "thread_count": thread_count,
"node_thread_counts": node_thread_counts,
"current_node_code": app_settings.node_code,
"runtime_settings": get_runtime_settings(), "runtime_settings": get_runtime_settings(),
} }
def get_credentials_payload() -> dict:
credentials = read_json(
"credentials.json",
default={
"juming": {"email": "", "password": ""},
"juziseo": {"email": "", "password": ""},
},
)
redis_client = get_redis()
try:
if redis_credentials := redis_client.get(REDIS_KEYS["credentials"]):
credentials = json.loads(redis_credentials)
except Exception:
pass
return {
"juming": {
"email": str(credentials.get("juming", {}).get("email", "")),
"password": str(credentials.get("juming", {}).get("password", "")),
},
"juziseo": {
"email": str(credentials.get("juziseo", {}).get("email", "")),
"password": str(credentials.get("juziseo", {}).get("password", "")),
},
}
def update_credentials_payload(payload: dict) -> dict:
current = get_credentials_payload()
credentials = {
"juming": {
"email": str(payload.get("juming", {}).get("email", current["juming"]["email"])),
"password": str(payload.get("juming", {}).get("password", current["juming"]["password"])),
},
"juziseo": {
"email": str(payload.get("juziseo", {}).get("email", current["juziseo"]["email"])),
"password": str(payload.get("juziseo", {}).get("password", current["juziseo"]["password"])),
},
}
write_json("credentials.json", credentials)
redis_client = get_redis()
try:
redis_client.set(REDIS_KEYS["credentials"], json.dumps(credentials, ensure_ascii=False))
redis_client.publish("domain_tool:credentials:update", json.dumps(credentials, ensure_ascii=False))
except Exception:
pass
return credentials
def update_settings_payload(payload: dict) -> dict: def update_settings_payload(payload: dict) -> dict:
current = get_settings_payload() current = get_settings_payload()
detect_options = payload.get("detect_options", current["detect_options"]) detect_options = payload.get("detect_options", current["detect_options"])
proxy_config = payload.get("proxy_config", current["proxy_config"]) proxy_config = payload.get("proxy_config", current["proxy_config"])
thread_count = int(payload.get("thread_count", current["thread_count"])) thread_count = _normalize_thread_count(payload.get("thread_count", current["thread_count"]))
node_thread_counts = _normalize_node_thread_counts(payload.get("node_thread_counts", current.get("node_thread_counts", {})))
runtime_settings = update_runtime_settings(payload.get("runtime_settings", current["runtime_settings"])) runtime_settings = update_runtime_settings(payload.get("runtime_settings", current["runtime_settings"]))
write_json("detect_options.json", detect_options) write_json("detect_options.json", detect_options)
write_json("proxy_config.json", proxy_config) write_json("proxy_config.json", proxy_config)
write_json("thread_count.json", {"thread_count": str(thread_count)}) write_json("thread_count.json", {"thread_count": str(thread_count)})
write_json("node_thread_counts.json", node_thread_counts)
redis_client = get_redis() redis_client = get_redis()
try: try:
@@ -69,6 +207,8 @@ def update_settings_payload(payload: dict) -> dict:
redis_client.publish("domain_tool:proxy_config:update", json.dumps(proxy_config, ensure_ascii=False)) redis_client.publish("domain_tool:proxy_config:update", json.dumps(proxy_config, ensure_ascii=False))
redis_client.set(REDIS_KEYS["thread_count"], thread_count) redis_client.set(REDIS_KEYS["thread_count"], thread_count)
redis_client.publish("domain_tool:thread_count:update", str(thread_count)) redis_client.publish("domain_tool:thread_count:update", str(thread_count))
redis_client.set(REDIS_KEYS["node_thread_counts"], json.dumps(node_thread_counts, ensure_ascii=False))
redis_client.publish("domain_tool:node_thread_counts:update", json.dumps(node_thread_counts, ensure_ascii=False))
except Exception: except Exception:
pass pass
@@ -76,6 +216,8 @@ def update_settings_payload(payload: dict) -> dict:
"detect_options": detect_options, "detect_options": detect_options,
"proxy_config": proxy_config, "proxy_config": proxy_config,
"thread_count": thread_count, "thread_count": thread_count,
"node_thread_counts": node_thread_counts,
"current_node_code": app_settings.node_code,
"runtime_settings": runtime_settings, "runtime_settings": runtime_settings,
} }
@@ -101,12 +243,10 @@ def validate_settings_payload(payload: dict) -> None:
raise ValueError("invalid settings payload") raise ValueError("invalid settings payload")
if "thread_count" in payload: if "thread_count" in payload:
try: _normalize_thread_count(payload["thread_count"])
thread_count = int(payload["thread_count"])
except Exception as exc: if "node_thread_counts" in payload:
raise ValueError("thread_count must be an integer") from exc _normalize_node_thread_counts(payload["node_thread_counts"])
if thread_count < 1 or thread_count > 256:
raise ValueError("thread_count out of range")
if "detect_options" in payload: if "detect_options" in payload:
detect_options = payload["detect_options"] detect_options = payload["detect_options"]
@@ -135,6 +275,9 @@ def validate_settings_payload(payload: dict) -> None:
worker_mode = runtime_settings.get("worker_mode") worker_mode = runtime_settings.get("worker_mode")
if worker_mode and worker_mode not in {"windows-local", "linux-systemd"}: if worker_mode and worker_mode not in {"windows-local", "linux-systemd"}:
raise ValueError("worker_mode must be windows-local or linux-systemd") raise ValueError("worker_mode must be windows-local or linux-systemd")
for key in ("worker_service_name", "api_service_name", "sync_agent_service_name"):
if key in runtime_settings and runtime_settings[key] is not None and not str(runtime_settings[key]).strip():
raise ValueError(f"{key} must not be empty")
def backup_current_settings(reason: str = "manual") -> dict: def backup_current_settings(reason: str = "manual") -> dict:

423
domain-api/app/services/sync_push_service.py Normal file
View File

@@ -0,0 +1,423 @@
from __future__ import annotations
import json
import urllib.error
import urllib.parse
import urllib.request
from datetime import datetime, timedelta
from app.core.config import settings
from app.core.db import get_db
from app.services.sync_record_service import _decode_json, _normalize_region
def _format_time(value: datetime | None) -> str:
return value.isoformat(sep=" ", timespec="seconds") if value else ""
def _ingest_url(base_url: str) -> str:
text = str(base_url or "").strip().rstrip("/")
if not text:
return ""
if text.endswith("/api/v1"):
return f"{text}/runtime/sync-ingest"
if text.endswith("/api/v1/runtime"):
return f"{text}/sync-ingest"
return f"{text}/api/v1/runtime/sync-ingest"
def _projection_ingest_type(sync_type: str) -> str:
if sync_type == "runtime_projection":
return "runtime_ingest"
if sync_type == "detect_result_projection":
return "detect_result_ingest"
return "sync_ingest"
def _load_latest_projection(sync_type: str) -> dict | None:
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
target_region = _normalize_region(settings.sync_target_region, "overseas")
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, source_region, target_region, status, payload_json, created_at, updated_at
FROM detect_sync_records
WHERE sync_type = %s
AND source_region = %s
AND target_region = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
""",
(sync_type, source_region, target_region),
)
row = cur.fetchone()
if not row:
return None
return {
"id": row[0],
"source_region": row[1],
"target_region": row[2],
"status": row[3],
"payload": _decode_json(row[4]),
"created_at": row[5],
"updated_at": row[6],
}
def _load_pushable_projections(sync_type: str, limit: int) -> list[dict]:
safe_limit = max(1, min(int(limit or 1), max(1, int(settings.sync_batch_size or 200))))
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
target_region = _normalize_region(settings.sync_target_region, "overseas")
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, source_region, target_region, status, payload_json, created_at, updated_at
FROM detect_sync_records
WHERE sync_type = %s
AND source_region = %s
AND target_region = %s
ORDER BY created_at ASC, id ASC
LIMIT %s
""",
(sync_type, source_region, target_region, safe_limit * 5),
)
rows = cur.fetchall()
selected: list[dict] = []
for row in rows:
projection = {
"id": row[0],
"source_region": row[1],
"target_region": row[2],
"status": row[3],
"payload": _decode_json(row[4]),
"created_at": row[5],
"updated_at": row[6],
}
latest_attempt = _latest_push_attempt(projection["id"], projection["target_region"], sync_type)
if latest_attempt and latest_attempt["status"] == "success":
continue
if latest_attempt and latest_attempt["status"] == "pending":
continue
if latest_attempt and latest_attempt["status"] == "failed":
last_created_at = latest_attempt.get("created_at")
if isinstance(last_created_at, datetime):
now = datetime.now(last_created_at.tzinfo) if last_created_at.tzinfo else datetime.now()
if now - last_created_at < timedelta(seconds=max(10, int(settings.sync_poll_interval_seconds or 30))):
continue
selected.append(projection)
if len(selected) >= safe_limit:
break
return selected
def _latest_push_attempt(source_record_id: int, target_region: str, sync_type: str) -> dict | None:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, status, payload_json, error_message, created_at, updated_at
FROM detect_sync_records
WHERE sync_type = 'runtime_push'
AND source_region = %s
AND target_region = %s
AND (payload_json->>'sync_type') = %s
AND (payload_json->>'source_record_id') = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
""",
(
_normalize_region(settings.sync_source_region, settings.node_region),
target_region,
sync_type,
str(int(source_record_id)),
),
)
row = cur.fetchone()
if not row:
return None
return {
"id": row[0],
"status": row[1],
"payload": _decode_json(row[2]),
"error_message": row[3] or "",
"created_at": row[4],
"updated_at": row[5],
}
def _create_push_attempt(*, source_record: dict, ingest_url: str, sync_type: str) -> int:
payload = {
"sync_type": sync_type,
"source_record_id": source_record["id"],
"projection_hash": (source_record.get("payload") or {}).get("projection_hash", ""),
"ingest_url": ingest_url,
}
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO detect_sync_records (
sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at
) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
RETURNING id
""",
(
"runtime_push",
source_record["source_region"],
source_record["target_region"],
"pending",
json.dumps(payload, ensure_ascii=False),
"",
),
)
record_id = int(cur.fetchone()[0])
conn.commit()
return record_id
def _update_push_attempt(record_id: int, *, status: str, payload: dict | None = None, error_message: str = "") -> None:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
UPDATE detect_sync_records
SET status = %s,
payload_json = %s::jsonb,
error_message = %s,
updated_at = CURRENT_TIMESTAMP
WHERE id = %s
""",
(
str(status or "").strip() or "pending",
json.dumps(payload or {}, ensure_ascii=False),
str(error_message or "").strip(),
int(record_id),
),
)
conn.commit()
def ingest_runtime_projection(payload: dict, *, shared_token: str | None = None) -> tuple[bool, str, dict]:
configured_token = str(settings.sync_shared_token or "").strip()
incoming_token = str(shared_token or "").strip()
if configured_token and incoming_token != configured_token:
return False, "同步 token 校验失败", {}
sync_type = str(payload.get("sync_type") or "runtime_projection").strip() or "runtime_projection"
ingest_sync_type = _projection_ingest_type(sync_type)
source_region = _normalize_region(payload.get("source_region"), "unknown")
source_record_id = int(payload.get("source_record_id") or 0)
projection_hash = str(payload.get("projection_hash") or "").strip()
projection = payload.get("projection") or {}
target_region = _normalize_region(settings.node_region, "overseas")
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id
FROM detect_sync_records
WHERE sync_type = %s
AND source_region = %s
AND target_region = %s
AND (payload_json->>'source_record_id') = %s
AND (payload_json->>'projection_hash') = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
""",
(ingest_sync_type, source_region, target_region, str(source_record_id), projection_hash),
)
existing = cur.fetchone()
if existing:
return True, "同步投影已存在，已按幂等处理", {"record_id": int(existing[0]), "deduplicated": True}
stored_payload = {
"sync_type": sync_type,
"source_record_id": source_record_id,
"projection_hash": projection_hash,
"projection": projection,
"received_at": _format_time(datetime.now()),
}
cur.execute(
"""
INSERT INTO detect_sync_records (
sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at
) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
RETURNING id
""",
(
ingest_sync_type,
source_region,
target_region,
"received",
json.dumps(stored_payload, ensure_ascii=False),
"",
),
)
record_id = int(cur.fetchone()[0])
conn.commit()
return True, "同步投影接收成功", {"record_id": record_id, "deduplicated": False}
def _push_projection_now(sync_type: str, ingest_url: str) -> tuple[bool, str, dict]:
source_record = _load_latest_projection(sync_type)
if not source_record:
return False, f"当前没有可推送的{sync_type}", {"action": "push_sync", "sync_type": sync_type}
return _push_projection_record(source_record, sync_type, ingest_url)
def _push_projection_batch(sync_type: str, ingest_url: str) -> tuple[bool, str, dict]:
pending_records = _load_pushable_projections(sync_type, limit=max(1, int(settings.sync_batch_size or 200)))
if not pending_records:
latest = _load_latest_projection(sync_type)
if latest:
latest_attempt = _latest_push_attempt(latest["id"], latest["target_region"], sync_type)
if latest_attempt and latest_attempt["status"] == "success":
return True, "当前批次已全部同步完成", {
"action": "push_sync",
"sync_type": sync_type,
"deduplicated": True,
"batch_count": 0,
}
return False, f"当前没有可推送的{sync_type}", {"action": "push_sync", "sync_type": sync_type, "batch_count": 0}
results: list[dict] = []
success_count = 0
for record in pending_records:
ok, message, data = _push_projection_record(record, sync_type, ingest_url)
results.append(
{
"source_record_id": record["id"],
"ok": ok,
"message": message,
"data": data,
}
)
if ok:
success_count += 1
overall_ok = success_count > 0
message = f"{sync_type} 批量推送完成，成功 {success_count}/{len(results)}"
return overall_ok, message, {
"action": "push_sync",
"sync_type": sync_type,
"batch_count": len(results),
"success_count": success_count,
"results": results,
}
def _push_projection_record(source_record: dict, sync_type: str, ingest_url: str) -> tuple[bool, str, dict]:
latest_attempt = _latest_push_attempt(source_record["id"], source_record["target_region"], sync_type)
if latest_attempt and latest_attempt["status"] == "success":
return True, "该投影已推送，无需重复发送", {
"action": "push_sync",
"sync_type": sync_type,
"source_record_id": source_record["id"],
"deduplicated": True,
}
if latest_attempt and latest_attempt["status"] == "pending":
return True, "该投影已有同步推送进行中，暂不重复发送", {
"action": "push_sync",
"sync_type": sync_type,
"source_record_id": source_record["id"],
"deduplicated": True,
}
if latest_attempt and latest_attempt["status"] == "failed":
last_created_at = latest_attempt.get("created_at")
if isinstance(last_created_at, datetime):
now = datetime.now(last_created_at.tzinfo) if last_created_at.tzinfo else datetime.now()
if now - last_created_at < timedelta(seconds=max(10, int(settings.sync_poll_interval_seconds or 30))):
return False, "最近一次同步推送刚失败，等待下个重试窗口", {
"action": "push_sync",
"sync_type": sync_type,
"source_record_id": source_record["id"],
"deduplicated": True,
}
attempt_id = _create_push_attempt(source_record=source_record, ingest_url=ingest_url, sync_type=sync_type)
request_payload = {
"sync_type": sync_type,
"source_region": source_record["source_region"],
"source_record_id": source_record["id"],
"projection_hash": (source_record.get("payload") or {}).get("projection_hash", ""),
"projection": (source_record.get("payload") or {}).get("projection", {}),
"created_at": _format_time(source_record.get("created_at")),
}
request_body = json.dumps(request_payload, ensure_ascii=False).encode("utf-8")
request = urllib.request.Request(
ingest_url,
data=request_body,
headers={
"Content-Type": "application/json",
**({"X-Domaincheck-Sync-Token": settings.sync_shared_token} if settings.sync_shared_token else {}),
},
method="POST",
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
raw = response.read().decode("utf-8")
data = json.loads(raw) if raw else {}
status_code = getattr(response, "status", 200)
except urllib.error.HTTPError as exc:
error_body = exc.read().decode("utf-8", errors="ignore") if hasattr(exc, "read") else ""
payload = {
"sync_type": sync_type,
"source_record_id": source_record["id"],
"projection_hash": request_payload["projection_hash"],
"ingest_url": ingest_url,
"http_status": getattr(exc, "code", 500),
"response_text": error_body[:1000],
}
_update_push_attempt(attempt_id, status="failed", payload=payload, error_message=f"HTTP {getattr(exc, 'code', 500)}")
return False, f"同步推送失败: HTTP {getattr(exc, 'code', 500)}", {"action": "push_sync", "sync_type": sync_type, "attempt_id": attempt_id}
except Exception as exc:
payload = {
"sync_type": sync_type,
"source_record_id": source_record["id"],
"projection_hash": request_payload["projection_hash"],
"ingest_url": ingest_url,
}
_update_push_attempt(attempt_id, status="failed", payload=payload, error_message=str(exc))
return False, f"同步推送失败: {exc}", {"action": "push_sync", "sync_type": sync_type, "attempt_id": attempt_id}
payload = {
"sync_type": sync_type,
"source_record_id": source_record["id"],
"projection_hash": request_payload["projection_hash"],
"ingest_url": ingest_url,
"http_status": status_code,
"response": data,
}
_update_push_attempt(attempt_id, status="success", payload=payload, error_message="")
return True, "投影推送成功", {
"action": "push_sync",
"sync_type": sync_type,
"attempt_id": attempt_id,
"source_record_id": source_record["id"],
"response": data,
}
def push_runtime_projection_now() -> tuple[bool, str, dict]:
if not settings.sync_push_enabled:
return False, "未启用同步推送", {"action": "push_sync"}
ingest_url = _ingest_url(settings.sync_target_api_base_url)
if not ingest_url:
return False, "未配置同步目标地址", {"action": "push_sync"}
results = []
ok, message, data = _push_projection_now("runtime_projection", ingest_url)
results.append({"sync_type": "runtime_projection", "ok": ok, "message": message, "data": data})
ok, message, data = _push_projection_batch("detect_result_projection", ingest_url)
results.append({"sync_type": "detect_result_projection", "ok": ok, "message": message, "data": data})
success_count = sum(1 for item in results if item["ok"])
if success_count == 0:
return False, "当前没有成功推送的同步投影", {"action": "push_sync", "results": results}
return True, f"同步推送完成，成功 {success_count}/{len(results)}", {"action": "push_sync", "results": results}

592
domain-api/app/services/sync_record_service.py Normal file
View File

@@ -0,0 +1,592 @@
from __future__ import annotations
import hashlib
import json
from datetime import datetime, timedelta
from app.core.config import settings
from app.core.db import get_db
def _format_time(value: datetime | None) -> str:
return value.isoformat(sep=" ", timespec="seconds") if value else ""
def _decode_json(value: object) -> dict:
if isinstance(value, dict):
return value
if value in (None, ""):
return {}
try:
return json.loads(value)
except Exception:
return {}
def _normalize_region(value: str | None, fallback: str) -> str:
text = str(value or "").strip()
if not text or text == "unknown":
return str(fallback or "unknown").strip() or "unknown"
return text
def _should_append_runtime_projection(previous_payload: dict, current_projection: dict, previous_created_at: datetime | None) -> bool:
if not previous_payload:
return True
previous_projection = previous_payload.get("projection") or {}
if not previous_projection:
return True
keys_requiring_immediate_write = (
"worker_online",
"worker_mode",
"phase_label",
"phase_detail",
"proxy_runtime_label",
"proxy_runtime_reason",
)
for key in keys_requiring_immediate_write:
if previous_projection.get(key) != current_projection.get(key):
return True
previous_job = previous_projection.get("active_job") or {}
current_job = current_projection.get("active_job") or {}
for key in ("job_id", "job_code", "status"):
if previous_job.get(key) != current_job.get(key):
return True
previous_cluster = previous_projection.get("cluster_summary") or {}
current_cluster = current_projection.get("cluster_summary") or {}
for key in ("online_worker_nodes", "online_control_nodes", "busy_nodes", "stale_nodes", "offline_nodes"):
if previous_cluster.get(key) != current_cluster.get(key):
return True
previous_progress = previous_projection.get("progress") or {}
current_progress = current_projection.get("progress") or {}
failed_delta = abs(int(current_progress.get("failed", 0) or 0) - int(previous_progress.get("failed", 0) or 0))
blacklisted_delta = abs(int(current_progress.get("blacklisted", 0) or 0) - int(previous_progress.get("blacklisted", 0) or 0))
completed_delta = abs(int(current_progress.get("completed", 0) or 0) - int(previous_progress.get("completed", 0) or 0))
running_delta = abs(int(current_progress.get("running", 0) or 0) - int(previous_progress.get("running", 0) or 0))
if failed_delta > 0 or blacklisted_delta >= 10 or completed_delta >= 20 or running_delta >= 5:
return True
previous_alerts = previous_projection.get("dependency_alerts") or []
current_alerts = current_projection.get("dependency_alerts") or []
if previous_alerts != current_alerts:
return True
if not previous_created_at:
return True
now = datetime.now(previous_created_at.tzinfo) if previous_created_at.tzinfo else datetime.now()
return now - previous_created_at >= timedelta(seconds=45)
def list_sync_records(limit: int = 20) -> list[dict]:
safe_limit = max(1, min(int(limit or 20), 200))
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at
FROM detect_sync_records
ORDER BY created_at DESC, id DESC
LIMIT %s
""",
(safe_limit,),
)
rows = cur.fetchall()
return [
{
"id": row[0],
"sync_type": row[1],
"source_region": row[2],
"target_region": row[3],
"status": row[4],
"payload": _decode_json(row[5]),
"error_message": row[6] or "",
"created_at": _format_time(row[7]),
"updated_at": _format_time(row[8]),
}
for row in rows
]
def _latest_sync_record_by_source(
cur,
*,
sync_type: str,
source_region: str,
target_region: str,
source_record_id: int,
) -> dict | None:
cur.execute(
"""
SELECT id, sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at
FROM detect_sync_records
WHERE sync_type = %s
AND source_region = %s
AND target_region = %s
AND (payload_json->>'source_record_id') = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
""",
(sync_type, source_region, target_region, str(int(source_record_id))),
)
row = cur.fetchone()
if not row:
return None
return {
"id": row[0],
"sync_type": row[1],
"source_region": row[2],
"target_region": row[3],
"status": row[4],
"payload": _decode_json(row[5]),
"error_message": row[6] or "",
"created_at": _format_time(row[7]),
"updated_at": _format_time(row[8]),
}
def get_detect_result_sync_batches(limit: int = 5) -> dict:
safe_limit = max(1, min(int(limit or 5), 20))
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
target_region = _normalize_region(settings.sync_target_region, "overseas")
batches: list[dict] = []
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT id, job_code, status, created_at, started_at, finished_at
FROM detect_jobs
ORDER BY created_at DESC, id DESC
LIMIT %s
""",
(safe_limit,),
)
jobs = cur.fetchall()
for row in jobs:
job_id = int(row[0])
job_code = str(row[1] or "")
job_status = str(row[2] or "")
created_at = _format_time(row[3])
started_at = _format_time(row[4])
finished_at = _format_time(row[5])
cur.execute(
"""
SELECT status, count(*)
FROM detect_job_items
WHERE job_id = %s
GROUP BY status
""",
(job_id,),
)
item_counts = {str(status or ""): int(count) for status, count in cur.fetchall()}
items_total = sum(item_counts.values())
items_terminal = int(item_counts.get("completed", 0)) + int(item_counts.get("blacklisted", 0)) + int(item_counts.get("failed", 0))
cur.execute(
"""
SELECT id, status, payload_json, created_at, updated_at
FROM detect_sync_records
WHERE sync_type = 'detect_result_projection'
AND source_region = %s
AND target_region = %s
AND (payload_json->'projection'->'job'->>'job_id') = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
""",
(source_region, target_region, str(job_id)),
)
projection_row = cur.fetchone()
projection = None
latest_push = None
latest_ingest = None
sync_state = "unsynced"
sync_message = "该任务还没有生成结果投影"
if projection_row:
projection_payload = _decode_json(projection_row[2])
projection = {
"id": int(projection_row[0]),
"status": projection_row[1],
"payload": projection_payload,
"created_at": _format_time(projection_row[3]),
"updated_at": _format_time(projection_row[4]),
}
latest_push = _latest_sync_record_by_source(
cur,
sync_type="runtime_push",
source_region=source_region,
target_region=target_region,
source_record_id=projection["id"],
)
latest_ingest = _latest_sync_record_by_source(
cur,
sync_type="detect_result_ingest",
source_region=source_region,
target_region=target_region,
source_record_id=projection["id"],
)
if latest_ingest:
sync_state = "synced"
sync_message = "最近一条结果投影已被目标地域接收"
elif latest_push and latest_push.get("status") == "success":
sync_state = "delivered"
sync_message = "结果投影已推送成功，等待目标侧回看接收记录"
elif latest_push and latest_push.get("status") == "pending":
sync_state = "pushing"
sync_message = "结果投影正在推送中"
elif latest_push and latest_push.get("status") == "failed":
sync_state = "failed"
sync_message = latest_push.get("error_message") or "最近一次结果投影推送失败"
else:
sync_state = "projected"
sync_message = "已生成结果投影，等待同步代理推送"
batches.append(
{
"job_id": job_id,
"job_code": job_code,
"job_status": job_status,
"created_at": created_at,
"started_at": started_at,
"finished_at": finished_at,
"items_total": items_total,
"items_terminal": items_terminal,
"items_pending": int(item_counts.get("pending", 0)),
"items_running": int(item_counts.get("running", 0)) + int(item_counts.get("claimed", 0)),
"items_failed": int(item_counts.get("failed", 0)),
"progress_percent": round((items_terminal / items_total) * 100, 2) if items_total else 0,
"sync_state": sync_state,
"sync_message": sync_message,
"projection": projection,
"latest_push": latest_push,
"latest_ingest": latest_ingest,
}
)
state_counts = {
"synced": 0,
"delivered": 0,
"pushing": 0,
"projected": 0,
"failed": 0,
"unsynced": 0,
}
for item in batches:
state = str(item.get("sync_state") or "unsynced")
state_counts[state] = state_counts.get(state, 0) + 1
return {
"source_region": source_region,
"target_region": target_region,
"jobs_total": len(batches),
"state_counts": state_counts,
"batches": batches,
}
def get_sync_summary(record_limit: int = 10) -> dict:
source_region = _normalize_region(settings.sync_source_region, settings.node_region)
target_region = _normalize_region(settings.sync_target_region, "overseas")
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT status, count(*)
FROM detect_sync_records
GROUP BY status
"""
)
status_counts = {str(status or "unknown"): int(count) for status, count in cur.fetchall()}
cur.execute(
"""
SELECT sync_type, count(*)
FROM detect_sync_records
GROUP BY sync_type
"""
)
type_counts = {str(sync_type or "unknown"): int(count) for sync_type, count in cur.fetchall()}
cur.execute("SELECT count(*) FROM detect_sync_records")
total = int(cur.fetchone()[0] or 0)
cur.execute(
"""
SELECT id, sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at
FROM detect_sync_records
ORDER BY created_at DESC, id DESC
LIMIT 1
"""
)
latest = cur.fetchone()
latest_record = None
if latest:
latest_record = {
"id": latest[0],
"sync_type": latest[1],
"source_region": latest[2],
"target_region": latest[3],
"status": latest[4],
"payload": _decode_json(latest[5]),
"error_message": latest[6] or "",
"created_at": _format_time(latest[7]),
"updated_at": _format_time(latest[8]),
}
return {
"enabled": bool(settings.sync_push_enabled),
"source_region": source_region,
"target_region": target_region,
"target_api_base_url": settings.sync_target_api_base_url,
"batch_size": max(1, int(settings.sync_batch_size or 200)),
"poll_interval_seconds": max(5, int(settings.sync_poll_interval_seconds or 30)),
"records_total": total,
"status_counts": status_counts,
"type_counts": type_counts,
"latest_record": latest_record,
"detect_result_batches": get_detect_result_sync_batches(limit=min(5, record_limit)),
"recent_records": list_sync_records(limit=record_limit),
}
def append_sync_record(
*,
sync_type: str,
source_region: str,
target_region: str,
status: str,
payload: dict | None = None,
error_message: str = "",
) -> int:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
INSERT INTO detect_sync_records (
sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at
) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
RETURNING id
""",
(
str(sync_type or "").strip() or "unknown",
_normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region)),
_normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas")),
str(status or "").strip() or "pending",
json.dumps(payload or {}, ensure_ascii=False),
str(error_message or "").strip(),
),
)
record_id = int(cur.fetchone()[0])
conn.commit()
return record_id
def append_runtime_projection_if_changed(
*,
detect: dict,
cluster: dict,
source_region: str | None = None,
target_region: str | None = None,
) -> int | None:
normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region))
normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas"))
active_job = detect.get("active_job") or {}
projection = {
"worker_online": bool(detect.get("worker_online", False)),
"worker_mode": detect.get("worker_mode", ""),
"phase_label": detect.get("phase_label", ""),
"phase_detail": detect.get("phase_detail", ""),
"proxy_runtime_label": detect.get("proxy_runtime_label", ""),
"proxy_runtime_reason": detect.get("proxy_runtime_reason", ""),
"progress": {
"pending": int((detect.get("progress") or {}).get("pending", 0) or 0),
"running": int((detect.get("progress") or {}).get("running", 0) or 0),
"completed": int((detect.get("progress") or {}).get("completed", 0) or 0),
"blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0),
"failed": int((detect.get("progress") or {}).get("failed", 0) or 0),
},
"active_job": {
"job_id": active_job.get("job_id"),
"job_code": active_job.get("job_code", ""),
"status": active_job.get("status", ""),
"progress_percent": active_job.get("progress_percent", 0),
"items_total": active_job.get("items_total", 0),
"items_terminal": active_job.get("items_terminal", 0),
"items_pending": active_job.get("items_pending", 0),
"items_running": active_job.get("items_running", 0),
"items_failed": active_job.get("items_failed", 0),
},
"cluster_summary": {
"nodes_total": int(cluster.get("nodes_total", 0) or 0),
"online_worker_nodes": int((cluster.get("summary") or {}).get("online_worker_nodes", 0) or 0),
"online_control_nodes": int((cluster.get("summary") or {}).get("online_control_nodes", 0) or 0),
"busy_nodes": list((cluster.get("summary") or {}).get("busy_nodes") or []),
"stale_nodes": list((cluster.get("summary") or {}).get("stale_nodes") or []),
"offline_nodes": list((cluster.get("summary") or {}).get("offline_nodes") or []),
},
"dependency_alerts": [
{
"kind": item.get("kind", ""),
"title": item.get("title", ""),
"level": item.get("level", ""),
}
for item in (detect.get("dependency_alerts") or [])[:3]
],
}
payload = {
"projection": projection,
"projection_hash": hashlib.sha1(
json.dumps(projection, ensure_ascii=False, sort_keys=True).encode("utf-8")
).hexdigest(),
}
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT payload_json, created_at
FROM detect_sync_records
WHERE sync_type = 'runtime_projection'
AND source_region = %s
AND target_region = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
""",
(normalized_source_region, normalized_target_region),
)
latest = cur.fetchone()
latest_payload = _decode_json(latest[0]) if latest else {}
latest_created_at = latest[1] if latest else None
if latest_payload.get("projection_hash") == payload["projection_hash"]:
return None
if not _should_append_runtime_projection(latest_payload, projection, latest_created_at):
return None
cur.execute(
"""
INSERT INTO detect_sync_records (
sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at
) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
RETURNING id
""",
(
"runtime_projection",
normalized_source_region,
normalized_target_region,
"projected",
json.dumps(payload, ensure_ascii=False),
"",
),
)
record_id = int(cur.fetchone()[0])
conn.commit()
return record_id
def append_detect_result_projection_if_changed(
*,
detect: dict,
source_region: str | None = None,
target_region: str | None = None,
) -> int | None:
normalized_source_region = _normalize_region(source_region, _normalize_region(settings.sync_source_region, settings.node_region))
normalized_target_region = _normalize_region(target_region, _normalize_region(settings.sync_target_region, "overseas"))
active_job = detect.get("active_job") or {}
if not active_job:
return None
latest_cycle_event = active_job.get("latest_cycle_event") or active_job.get("latest_event") or {}
projection = {
"job": {
"job_id": active_job.get("job_id"),
"job_code": active_job.get("job_code", ""),
"status": active_job.get("status", ""),
"progress_percent": active_job.get("progress_percent", 0),
"items_total": active_job.get("items_total", 0),
"items_pending": active_job.get("items_pending", 0),
"items_claimed": active_job.get("items_claimed", 0),
"items_running": active_job.get("items_running", 0),
"items_completed": active_job.get("items_completed", 0),
"items_blacklisted": active_job.get("items_blacklisted", 0),
"items_failed": active_job.get("items_failed", 0),
"items_terminal": active_job.get("items_terminal", 0),
"current_cycle_token": active_job.get("current_cycle_token", ""),
},
"latest_event": {
"node_code": latest_cycle_event.get("node_code", ""),
"event_type": latest_cycle_event.get("event_type", ""),
"message": latest_cycle_event.get("message", ""),
"created_at": latest_cycle_event.get("created_at", ""),
},
"queue": {
"pending": int((detect.get("progress") or {}).get("pending", 0) or 0),
"running": int((detect.get("progress") or {}).get("running", 0) or 0),
"completed": int((detect.get("progress") or {}).get("completed", 0) or 0),
"blacklisted": int((detect.get("progress") or {}).get("blacklisted", 0) or 0),
"failed": int((detect.get("progress") or {}).get("failed", 0) or 0),
},
"phase": {
"label": detect.get("phase_label", ""),
"detail": detect.get("phase_detail", ""),
},
}
payload = {
"projection": projection,
"projection_hash": hashlib.sha1(
json.dumps(projection, ensure_ascii=False, sort_keys=True).encode("utf-8")
).hexdigest(),
}
with get_db() as conn:
with conn.cursor() as cur:
cur.execute(
"""
SELECT payload_json, created_at
FROM detect_sync_records
WHERE sync_type = 'detect_result_projection'
AND source_region = %s
AND target_region = %s
ORDER BY created_at DESC, id DESC
LIMIT 1
""",
(normalized_source_region, normalized_target_region),
)
latest = cur.fetchone()
latest_payload = _decode_json(latest[0]) if latest else {}
latest_created_at = latest[1] if latest else None
if latest_payload.get("projection_hash") == payload["projection_hash"]:
return None
latest_projection = latest_payload.get("projection") or {}
latest_job = latest_projection.get("job") or {}
current_job = projection.get("job") or {}
latest_event = latest_projection.get("latest_event") or {}
current_event = projection.get("latest_event") or {}
if latest_job.get("status") == current_job.get("status") and latest_event == current_event and latest_created_at:
now = datetime.now(latest_created_at.tzinfo) if latest_created_at.tzinfo else datetime.now()
if now - latest_created_at < timedelta(seconds=30):
return None
cur.execute(
"""
INSERT INTO detect_sync_records (
sync_type, source_region, target_region, status, payload_json, error_message, created_at, updated_at
) VALUES (%s, %s, %s, %s, %s::jsonb, %s, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
RETURNING id
""",
(
"detect_result_projection",
normalized_source_region,
normalized_target_region,
"projected",
json.dumps(payload, ensure_ascii=False),
"",
),
)
record_id = int(cur.fetchone()[0])
conn.commit()
return record_id

124
domain-api/app/services/worker_control_service.py
View File

@@ -3,12 +3,18 @@ from __future__ import annotations
import json import json
import os import os
import subprocess import subprocess
from datetime import datetime
from pathlib import Path from pathlib import Path
from app.core.config import settings from app.core.config import settings
from app.core.redis_client import get_redis
from app.services.runtime_settings_service import get_runtime_settings from app.services.runtime_settings_service import get_runtime_settings
WORKER_CONTROL_CHANNEL = "domain_tool:worker_control"
WORKER_PENDING_COMMAND_KEY = "domain_tool:worker_pending_command"
def _domain_root() -> Path: def _domain_root() -> Path:
return Path(settings.domain_root) return Path(settings.domain_root)
@@ -30,6 +36,73 @@ def _run_shell(command: list[str], timeout: int = 20) -> subprocess.CompletedPro
return subprocess.run(command, capture_output=True, text=True, timeout=timeout) return subprocess.run(command, capture_output=True, text=True, timeout=timeout)
def _run_systemctl(command: list[str], timeout: int = 20, require_sudo: bool = True) -> subprocess.CompletedProcess[str]:
systemctl_command = ["systemctl", *command]
if os.geteuid() == 0 or not require_sudo:
return _run_shell(systemctl_command, timeout=timeout)
return _run_shell(["sudo", "-n", *systemctl_command], timeout=timeout)
def _parse_systemd_timestamp(raw_timestamp: str) -> str:
raw_timestamp = (raw_timestamp or "").strip()
if not raw_timestamp:
return ""
try:
parsed = datetime.strptime(raw_timestamp, "%a %Y-%m-%d %H:%M:%S %Z")
return parsed.strftime("%Y-%m-%d %H:%M:%S")
except ValueError:
pass
try:
parsed = datetime.strptime(raw_timestamp.rsplit(" ", 1)[0], "%a %Y-%m-%d %H:%M:%S")
return parsed.strftime("%Y-%m-%d %H:%M:%S")
except ValueError:
return ""
def probe_systemd_service(service_name: str, *, mode: str = "linux-systemd") -> dict:
result = _run_systemctl(
[
"show",
service_name,
"--no-page",
"--property=ActiveState,SubState,MainPID,ExecMainStartTimestamp,ActiveEnterTimestamp",
],
require_sudo=False,
)
output = (result.stdout or result.stderr or "").strip()
if result.returncode != 0:
return {
"mode": mode,
"service_name": service_name,
"running": False,
"process_count": 0,
"latest_start_time": "",
"message": output or f"systemd service {service_name} not available",
}
data: dict[str, str] = {}
for line in output.splitlines():
if "=" in line:
key, value = line.split("=", 1)
data[key] = value
main_pid = int(data.get("MainPID", "0") or 0)
active_state = data.get("ActiveState", "")
sub_state = data.get("SubState", "")
latest_start_time = ""
for raw_timestamp in (data.get("ExecMainStartTimestamp", ""), data.get("ActiveEnterTimestamp", "")):
latest_start_time = _parse_systemd_timestamp(raw_timestamp)
if latest_start_time:
break
return {
"mode": mode,
"service_name": service_name,
"running": active_state == "active",
"process_count": 1 if main_pid > 0 else 0,
"latest_start_time": latest_start_time,
"message": f"{active_state}/{sub_state}" if active_state else "",
}
def _windows_runtime() -> dict: def _windows_runtime() -> dict:
command = """ command = """
$targets = Get-CimInstance Win32_Process -Filter "name='python.exe'" | $targets = Get-CimInstance Win32_Process -Filter "name='python.exe'" |
@@ -83,32 +156,23 @@ def _windows_runtime() -> dict:
def _linux_runtime() -> dict: def _linux_runtime() -> dict:
runtime = _runtime_config() runtime = _runtime_config()
service_name = runtime["worker_service_name"] service_name = runtime["worker_service_name"]
result = _run_shell(["systemctl", "show", service_name, "--no-page", "--property=ActiveState,SubState,MainPID"]) return probe_systemd_service(service_name, mode="linux-systemd")
output = (result.stdout or result.stderr or "").strip()
if result.returncode != 0:
def detect_sync_agent_runtime() -> dict:
runtime = _runtime_config()
worker_mode = runtime.get("worker_mode", settings.worker_mode)
service_name = runtime.get("sync_agent_service_name", settings.sync_agent_service_name)
if worker_mode != "linux-systemd":
return { return {
"mode": "linux-systemd", "mode": worker_mode,
"service_name": service_name,
"running": False, "running": False,
"process_count": 0, "process_count": 0,
"latest_start_time": "", "latest_start_time": "",
"message": output or f"systemd service {service_name} not available", "message": "sync-agent 仅在 Linux systemd 多机部署中使用",
} }
return probe_systemd_service(service_name, mode="linux-systemd")
data: dict[str, str] = {}
for line in output.splitlines():
if "=" in line:
key, value = line.split("=", 1)
data[key] = value
main_pid = int(data.get("MainPID", "0") or 0)
active_state = data.get("ActiveState", "")
sub_state = data.get("SubState", "")
return {
"mode": "linux-systemd",
"running": active_state == "active",
"process_count": 1 if main_pid > 0 else 0,
"latest_start_time": "",
"message": f"{active_state}/{sub_state}" if active_state else "",
}
def detect_worker_runtime() -> dict: def detect_worker_runtime() -> dict:
@@ -132,7 +196,7 @@ def start_worker() -> tuple[bool, str]:
worker_mode = runtime["worker_mode"] worker_mode = runtime["worker_mode"]
service_name = runtime["worker_service_name"] service_name = runtime["worker_service_name"]
if worker_mode == "linux-systemd": if worker_mode == "linux-systemd":
result = _run_shell(["systemctl", "start", service_name], timeout=30) result = _run_systemctl(["start", service_name], timeout=30)
if result.returncode != 0: if result.returncode != 0:
return False, (result.stderr or result.stdout or "启动 Linux Worker 失败").strip() return False, (result.stderr or result.stdout or "启动 Linux Worker 失败").strip()
return True, f"Linux Worker 启动命令已发送: {service_name}" return True, f"Linux Worker 启动命令已发送: {service_name}"
@@ -160,7 +224,7 @@ def stop_worker() -> tuple[bool, str]:
worker_mode = runtime["worker_mode"] worker_mode = runtime["worker_mode"]
service_name = runtime["worker_service_name"] service_name = runtime["worker_service_name"]
if worker_mode == "linux-systemd": if worker_mode == "linux-systemd":
result = _run_shell(["systemctl", "stop", service_name], timeout=30) result = _run_systemctl(["stop", service_name], timeout=30)
if result.returncode != 0: if result.returncode != 0:
return False, (result.stderr or result.stdout or "停止 Linux Worker 失败").strip() return False, (result.stderr or result.stdout or "停止 Linux Worker 失败").strip()
return True, f"Linux Worker 停止命令已发送: {service_name}" return True, f"Linux Worker 停止命令已发送: {service_name}"
@@ -186,3 +250,17 @@ def stop_worker() -> tuple[bool, str]:
if "NO_PROCESS" in output: if "NO_PROCESS" in output:
return True, "当前没有运行中的检测端进程" return True, "当前没有运行中的检测端进程"
return True, output or "检测端已停止" return True, output or "检测端已停止"
def send_worker_command(action: str, payload: dict | None = None) -> tuple[bool, str]:
try:
redis_client = get_redis()
command_payload = {"action": action}
if payload:
command_payload.update(payload)
serialized = json.dumps(command_payload, ensure_ascii=False)
redis_client.set(WORKER_PENDING_COMMAND_KEY, serialized, ex=120)
redis_client.publish(WORKER_CONTROL_CHANNEL, serialized)
return True, f"已发送 Worker 控制指令: {action}"
except Exception as exc:
return False, f"发送 Worker 控制指令失败: {exc}"

37
domain-api/app/sync_agent.py Normal file
View File

@@ -0,0 +1,37 @@
from __future__ import annotations
import logging
import time
from app.core.config import settings
from app.services.sync_push_service import push_runtime_projection_now
logger = logging.getLogger("domaincheck.sync_agent")
def main() -> None:
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
)
interval = max(10, int(settings.sync_poll_interval_seconds or 30))
logger.info(
"sync agent started: node=%s source=%s target=%s interval=%ss enabled=%s",
settings.node_code,
settings.sync_source_region,
settings.sync_target_region,
interval,
settings.sync_push_enabled,
)
while True:
try:
ok, message, data = push_runtime_projection_now()
logger.info("sync tick: ok=%s message=%s data=%s", ok, message, data)
except Exception as exc:
logger.exception("sync tick failed: %s", exc)
time.sleep(interval)
if __name__ == "__main__":
main()

33
domain-api/deploy/linux/README.md
View File

@@ -111,6 +111,27 @@ sudo cp /opt/domaincheck/domain-api/deploy/systemd/domain-worker.service /etc/sy
- `Group` - `Group`
- `Environment` - `Environment`
### 权限注意
正式 `systemd` 服务用户必须对下面这些路径有写权限：
- `domain-api/runtime/`
- `domainCheck/detect_worker.log`
如果之前曾用 `root` 手工跑过 API 或 Worker，可能会留下 `root` 所有者的运行目录或日志文件，导致正式服务态出现下面这类问题：
- `/api/v1/imports/upload` 因无法创建 `runtime/imports/` 返回 `500`
- Worker 因无法写 `detect_worker.log` 进入循环重启
必要时先修正所有者：
```bash
sudo chown -R www:www /opt/domaincheck/domain-api/runtime
sudo touch /opt/domaincheck/domainCheck/detect_worker.log
sudo chown www:www /opt/domaincheck/domainCheck/detect_worker.log
sudo chmod 664 /opt/domaincheck/domainCheck/detect_worker.log
```
## 六、启动顺序 ## 六、启动顺序
```bash ```bash
@@ -135,6 +156,18 @@ journalctl -u domaincheck-api -n 200 --no-pager
journalctl -u domaincheck-worker -n 200 --no-pager journalctl -u domaincheck-worker -n 200 --no-pager
``` ```
如果计划让 Web 后台里的“运行中心”控制 Worker 启停或触发 API 重启，还需要给服务用户补一条免密 sudo 规则。测试服当前使用的是：
```text
Defaults:www !requiretty
Cmnd_Alias DOMAINCHECK_WORKER_CTL = /usr/bin/systemctl start domaincheck-worker, /usr/bin/systemctl stop domaincheck-worker
Cmnd_Alias DOMAINCHECK_WORKER_SHOW = /usr/bin/systemctl show domaincheck-worker --no-page --property=ActiveState\,SubState\,MainPID
Cmnd_Alias DOMAINCHECK_API_RESTART = /usr/bin/systemctl restart domaincheck-api, /usr/bin/systemctl --no-block restart domaincheck-api
www ALL=(root) NOPASSWD: DOMAINCHECK_WORKER_CTL, DOMAINCHECK_WORKER_SHOW, DOMAINCHECK_API_RESTART
```
其中 API 自重启建议使用 `systemctl --no-block restart domaincheck-api`，这样接口可以先返回成功，再由 `systemd` 异步完成服务切换，避免调用方因为等待自身停机而拿到 `500`。
## 七、联调检查 ## 七、联调检查
### 1. API 健康检查 ### 1. API 健康检查

354
domain-api/deploy/multi-region/README.md Normal file
View File

@@ -0,0 +1,354 @@
# domainCheck 跨地域部署入口
本文档对应当前推荐的最小可用部署形态：
- 国外 `1` 台：`domain-web + domain-api + postgresql_main`
- 大陆 `1` 台：`redis + postgresql_runtime + scheduler + detect-worker`
目标：
- 后台和 API 放国外
- 检测执行放大陆
- 后续新增大陆 Worker 时，不再改整体部署方式
## 一、目录约定
统一使用：
```text
/opt/domaincheck
```
仓库内部署入口：
- `deploy/multi-region/bootstrap_overseas.sh`
- `deploy/multi-region/bootstrap_mainland.sh`
- `deploy/multi-region/check_cluster.sh`
- `deploy/multi-region/check_mainland_controller.sh`
- `deploy/multi-region/simulate_cluster_node.py`
- `deploy/multi-region/simulate_multi_region.sh`
- `deploy/multi-region/prune_cluster_nodes.py`
- `deploy/multi-region/prune_cluster_nodes.sh`
- `deploy/multi-region/templates/*.env.example`
## 二、国外机器部署
在国外机器执行：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/bootstrap_overseas.sh /opt/domaincheck
```
脚本会：
- 创建运行目录
- 检查 Python 虚拟环境
- 安装 API 依赖
- 安装 systemd 服务模板
- 在 `/etc/default/domaincheck-api` 生成环境变量模板
- 给出后续启动命令
## 三、大陆机器部署
在大陆机器执行：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
```
如果后面新增纯 Worker 节点：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
```
其中：
- `controller`
表示这台机器承担 `redis + runtime-db + scheduler + worker + sync-agent`
- `worker`
表示这台机器只承担 `detect-worker`
脚本还会在大陆机器生成：
- `/etc/default/domaincheck-worker`
当前模板已内置同步相关占位配置：
- `SYNC_PUSH_ENABLED`
- `SYNC_SOURCE_REGION`
- `SYNC_TARGET_REGION`
- `SYNC_TARGET_API_BASE_URL`
- `SYNC_SHARED_TOKEN`
- `SYNC_BATCH_SIZE`
- `SYNC_POLL_INTERVAL_SECONDS`
建议把每台大陆节点自己的身份信息放在这里，而不是直接修改 service 文件正文：
- `NODE_CODE`
- `NODE_REGION=mainland`
- `NODE_ROLE=control`
- 仅 `controller` 模板使用
- `NODE_ROLE=worker`
- 仅纯 Worker 模板使用
- `DB_*`
- `REDIS_*`
## 四、当前脚本定位
当前入口脚本是第一版“标准化部署脚手架”，优先解决：
- 目录统一
- systemd 模板统一
- 节点环境变量入口统一
- 新增节点时操作步骤统一
当前大陆 bootstrap 还会自动补齐一组最小 Python 依赖：
- `fastapi`
- `uvicorn`
- `pydantic-settings`
- `psycopg2-binary`
- `redis`
这样 `domaincheck-sync-agent` 在大陆 controller 节点上可以直接启动，不会因为 `app.sync_agent` 缺少 API 侧依赖而失败。
当前脚手架还额外区分了“自动同步由谁跑”：
- 海外控制面：
- `domain-api`
- 负责接收 `runtime/sync-ingest`
- 内地 controller 节点：
- `domaincheck-sync-agent`
- 负责按轮询周期把本地最新投影推送到海外控制面
当前还不会自动安装数据库主从或自动创建跨地域同步链路。
原因：
- 当前项目仍处于单 Worker 改造向多 Worker 任务模型过渡阶段
- 真正的任务调度与跨地域结果同步需要后续代码配合落地
- 但当前已经预留同步配置模板与同步记录查询接口，便于后续接入 `sync-service`
- 当前控制面还会自动把运行态摘要写入 `detect_sync_records`
- `sync_type=runtime_projection`
- 仅在关键状态变化或达到最小采样间隔时写入
- 作用是先把“同步观测面”跑起来，而不是替代最终的跨地域结果同步服务
- 当 `SYNC_PUSH_ENABLED=true` 且配置了 `SYNC_TARGET_API_BASE_URL` 后：
- 大陆 `domaincheck-sync-agent` 会按 `SYNC_POLL_INTERVAL_SECONDS` 自动尝试推送：
- 最新一条 `runtime_projection`
- 当前积压的 `detect_result_projection` 批次
- 目标控制面通过 `POST /api/v1/runtime/sync-ingest` 接收
- 若配置了 `SYNC_SHARED_TOKEN`，接收端会校验 `X-Domaincheck-Sync-Token`
- `GET /api/v1/runtime/sync-summary` 还会返回：
- `detect_result_batches`
用于直接观察最近检测任务的结果同步是否已接收、仍待推送或发生失败
## 五、当前已落地能力
截至 `2026-04-16`，当前代码已经具备：
- 节点注册与心跳
- 运行态表自动初始化
- 集群节点状态接口
- 检测任务主表与任务项表
- 活跃任务摘要与任务详情接口
- Worker 小批量领取、租约续租、重启释放和过期回收
可验证接口：
```bash
curl http://127.0.0.1:8100/api/v1/runtime/status
curl http://127.0.0.1:8100/api/v1/runtime/readiness
curl http://127.0.0.1:8100/api/v1/runtime/cluster
curl http://127.0.0.1:8100/api/v1/runtime/sync-summary
curl http://127.0.0.1:8100/api/v1/runtime/sync-records?limit=10
curl http://127.0.0.1:8100/api/v1/detect/job/active
curl http://127.0.0.1:8100/api/v1/detect/jobs
```
也可以直接执行一键联调：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100
```
这条命令当前除了原始接口输出，还会额外给出一段压缩摘要，直接汇总：
- `ready / attention / blocking`
- 在线控制面 / Worker 数
- `busy / stale / offline` 节点
- 同步投影 / 接收计数
- 结果批次的：
- `synced`
- `delivered`
- `pushing`
- `projected`
- `failed`
- `unsynced`
如果要在大陆 controller 节点本机确认“这台机器本身是否已经具备 controller 身份”，还可以执行：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/check_mainland_controller.sh
```
这条命令会直接检查：
- `/etc/default/domaincheck-worker` 是否存在
- `NODE_ROLE=control` 是否正确
- `SYNC_PUSH_ENABLED` / `SYNC_TARGET_API_BASE_URL` 是否已配置
- `domaincheck-worker`
- `domaincheck-sync-agent`
是否已启用并处于运行态
如果当前还无法真正部署到大陆机器，也可以先在国外测试机上做“单机模拟多节点联调”：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/simulate_multi_region.sh http://127.0.0.1:8100
```
这条命令会临时模拟：
- `mainland-controller-sim`
- `mainland-worker-sim-01`
用于提前验证：
- `runtime/readiness`
- `runtime/cluster`
- 运行中心顶部的多机就绪度结论
结束时按 `Ctrl+C`，脚本会自动清理模拟节点记录。
如果测试服里已经残留了很久没心跳的旧节点记录，导致 `runtime/readiness` 一直被离线节点拖成 `attention`，可以先做清理：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 --dry-run
bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30
```
如果只想清理某个确定已经废弃的节点：
```bash
bash deploy/multi-region/prune_cluster_nodes.sh --node-code mainland-worker-01
```
如果要判断大陆 Worker 是否已经真正接入，不要只看 `systemctl`，还要看：
- `/api/v1/runtime/cluster` 中是否出现对应 `node_code`
- `last_heartbeat_at` 是否持续刷新
- `role` / `region` 是否符合预期
- `metadata.job_id / metadata.cycle_token` 是否能在执行时出现
- `summary.online_worker_nodes` 是否大于 `0`
- `summary.status_counts.busy` 是否会在执行中增加
## 六、第二台大陆 Worker 接入建议
后续新增大陆 Worker 时，建议顺序为：
1. 拉最新代码到新大陆机器
2. 准备与主执行面一致的 Python 环境
3. 配置该节点自己的：
- `NODE_CODE`
- `NODE_REGION=mainland`
- `NODE_ROLE=worker`
4. 执行：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
```
5. 启动后在国外控制面检查：
```bash
curl http://127.0.0.1:8100/api/v1/runtime/cluster
```
如果接口里出现新节点，并且心跳持续更新，说明节点接入成功。
如果是第二台及以上大陆 Worker，最少还要确认：
- `NODE_CODE` 与其它节点不重复
- 连接的是同一套大陆 Redis / runtime-db
- `/etc/default/domaincheck-worker` 已按该机器单独填写
- 若该机器不是 controller，则不要额外启 `domaincheck-sync-agent`
## 七、当前推荐联调命令
国外控制面建议至少保留下面这组命令：
```bash
curl http://127.0.0.1:8100/health
curl http://127.0.0.1:8100/api/v1/runtime/preflight
curl http://127.0.0.1:8100/api/v1/runtime/readiness
curl http://127.0.0.1:8100/api/v1/runtime/cluster
curl http://127.0.0.1:8100/api/v1/detect/status
curl http://127.0.0.1:8100/api/v1/detect/job/active
curl http://127.0.0.1:8100/api/v1/detect/jobs?limit=5
```
如需一次性确认控制面和执行面都接通，可以直接执行：
```bash
cd /opt/domaincheck/domain-api
bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100
```
观察重点：
- 控制面节点心跳应持续刷新，而不是只在 API 启动时更新一次
- 大陆 Worker 节点在运行检测时应显示 `busy`
- `detect/job/active` 应能看到当前任务总量、完成量、节点分布和最近事件
- `detect/jobs` 可用于回看最近几轮任务是否正常收敛
- 当代理池暂时为空但允许直连时，检测控制页会显示 `降级直连`
- 当代理池为空且不允许直连时，检测会话阶段会显示 `等待代理`
- `runtime/cluster.summary` 中应能直接看出：
- 当前在线控制面节点数
- 当前在线 Worker 节点数
- 当前 `busy / stale / offline` 节点清单
- `runtime/sync-summary` 中可直接查看：
- 是否启用同步推送
- 当前配置的源地域 / 目标地域
- 最近同步记录和状态分布
- 若当前还未接入真正的 `sync-service`，也应至少能看到：
- `runtime_projection`
- `detect_result_projection`
- 接入自动推送后，还应能看到：
- `runtime_push`
- `runtime_ingest`
- `detect_result_ingest`
大陆 controller 节点建议额外确认：
```bash
systemctl status domaincheck-sync-agent --no-pager -l
```
如果同步配置已填写完整，则期望：
- `domaincheck-sync-agent` 为 `active (running)`
- 海外控制面的 `runtime/sync-summary` 中能同时看到：
- `runtime_projection`
- `runtime_push`
- `runtime_ingest`
## 八、后续演进
后续会继续补：
- 运行库初始化脚本
- 节点配置模板
- 结果同步服务
- 多 Worker 调度服务模板
配套设计文档见：
- `docs/16_domainCheck_多机检测与跨地域部署设计.md`

88
domain-api/deploy/multi-region/bootstrap_mainland.sh Executable file
View File

@@ -0,0 +1,88 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_DIR="${1:-/opt/domaincheck}"
ROLE="${2:-controller}"
API_DIR="$BASE_DIR/domain-api"
DOMAIN_DIR="$BASE_DIR/domainCheck"
SERVICE_NAME="domaincheck-worker"
SYNC_AGENT_SERVICE_NAME="domaincheck-sync-agent"
ENV_FILE="/etc/default/domaincheck-worker"
TEMPLATE_DIR="$API_DIR/deploy/multi-region/templates"
if [ "$ROLE" != "controller" ] && [ "$ROLE" != "worker" ]; then
echo "invalid role: $ROLE"
echo "usage: bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck [controller|worker]"
exit 1
fi
echo "[1/7] preparing directories"
mkdir -p "$BASE_DIR" "$DOMAIN_DIR" "$API_DIR/runtime"
if [ "$ROLE" = "controller" ]; then
mkdir -p /var/lib/domaincheck-runtime-db /var/lib/domaincheck-redis
fi
echo "[2/7] checking python environment"
if [ ! -x "$DOMAIN_DIR/.venv/bin/python" ]; then
echo "missing python venv: $DOMAIN_DIR/.venv/bin/python"
echo "please deploy domainCheck first and create .venv there"
exit 1
fi
echo "[2.5/7] installing mainland runtime dependencies"
"$DOMAIN_DIR/.venv/bin/pip" install -q fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
echo "[3/7] fixing runtime permissions"
touch "$DOMAIN_DIR/detect_worker.log"
chmod 664 "$DOMAIN_DIR/detect_worker.log"
echo "[4/7] installing worker service"
install -m 0644 "$API_DIR/deploy/systemd/domain-worker.service" "/etc/systemd/system/${SERVICE_NAME}.service"
if [ "$ROLE" = "controller" ]; then
install -m 0644 "$API_DIR/deploy/systemd/domain-sync-agent.service" "/etc/systemd/system/${SYNC_AGENT_SERVICE_NAME}.service"
fi
echo "[4.5/7] preparing environment file"
if [ ! -f "$ENV_FILE" ]; then
if [ "$ROLE" = "controller" ]; then
TEMPLATE_FILE="$TEMPLATE_DIR/domaincheck-worker.controller.env.example"
else
TEMPLATE_FILE="$TEMPLATE_DIR/domaincheck-worker.worker.env.example"
fi
install -m 0644 "$TEMPLATE_FILE" "$ENV_FILE"
sed -i "s#__BASE_DIR__#${BASE_DIR}#g" "$ENV_FILE"
chmod 0644 "$ENV_FILE"
echo "created env template: $ENV_FILE"
else
echo "env file already exists: $ENV_FILE"
fi
echo "[5/7] reloading systemd"
systemctl daemon-reload
if [ "$ROLE" = "controller" ]; then
echo "[6/7] controller notes"
echo "controller mode selected"
echo "please ensure local redis and postgresql_runtime are installed and listening on localhost"
echo "sync agent service: ${SYNC_AGENT_SERVICE_NAME}"
else
echo "[6/7] worker node notes"
echo "worker mode selected"
echo "this node is expected to connect to mainland redis and runtime-db"
fi
echo "[7/7] summary"
echo "mainland bootstrap completed"
echo "role: ${ROLE}"
echo "service: ${SERVICE_NAME}"
echo "next commands:"
echo " systemctl enable ${SERVICE_NAME}"
echo " systemctl restart ${SERVICE_NAME}"
echo " systemctl status ${SERVICE_NAME} --no-pager -l"
if [ "$ROLE" = "controller" ]; then
echo " systemctl enable ${SYNC_AGENT_SERVICE_NAME}"
echo " systemctl restart ${SYNC_AGENT_SERVICE_NAME}"
echo " systemctl status ${SYNC_AGENT_SERVICE_NAME} --no-pager -l"
fi
echo " vi ${ENV_FILE}"

48
domain-api/deploy/multi-region/bootstrap_overseas.sh Executable file
View File

@@ -0,0 +1,48 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_DIR="${1:-/opt/domaincheck}"
API_DIR="$BASE_DIR/domain-api"
WEB_DIR="$BASE_DIR/domain-web"
RUNTIME_DIR="$API_DIR/runtime"
SERVICE_NAME="domaincheck-api"
ENV_FILE="/etc/default/domaincheck-api"
TEMPLATE_FILE="$API_DIR/deploy/multi-region/templates/domaincheck-api.env.example"
echo "[1/6] preparing directories"
mkdir -p "$BASE_DIR" "$API_DIR" "$WEB_DIR" "$RUNTIME_DIR"
echo "[2/6] checking python environment"
if [ ! -x "$BASE_DIR/domainCheck/.venv/bin/python" ]; then
echo "missing python venv: $BASE_DIR/domainCheck/.venv/bin/python"
echo "please deploy domainCheck first and create .venv there"
exit 1
fi
echo "[3/6] installing api dependencies"
"$BASE_DIR/domainCheck/.venv/bin/pip" install -q fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
echo "[4/6] installing systemd service"
install -m 0644 "$API_DIR/deploy/systemd/domain-api.service" "/etc/systemd/system/${SERVICE_NAME}.service"
echo "[4.5/6] preparing environment file"
if [ ! -f "$ENV_FILE" ]; then
install -m 0644 "$TEMPLATE_FILE" "$ENV_FILE"
sed -i "s#__BASE_DIR__#${BASE_DIR}#g" "$ENV_FILE"
chmod 0644 "$ENV_FILE"
echo "created env template: $ENV_FILE"
else
echo "env file already exists: $ENV_FILE"
fi
echo "[5/6] reloading systemd"
systemctl daemon-reload
echo "[6/6] summary"
echo "overseas bootstrap completed"
echo "service: ${SERVICE_NAME}"
echo "next commands:"
echo " systemctl enable ${SERVICE_NAME}"
echo " systemctl restart ${SERVICE_NAME}"
echo " systemctl status ${SERVICE_NAME} --no-pager -l"
echo " vi ${ENV_FILE}"

89
domain-api/deploy/multi-region/check_cluster.sh Executable file
View File

@@ -0,0 +1,89 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
PYTHON_BIN="${PYTHON_BIN:-python3}"
echo "[1/7] health"
curl -fsS "${BASE_URL}/health"
echo
echo
echo "[2/7] runtime readiness"
curl -fsS "${BASE_URL}/api/v1/runtime/readiness"
echo
echo
echo "[3/7] runtime cluster"
curl -fsS "${BASE_URL}/api/v1/runtime/cluster"
echo
echo
echo "[4/7] runtime sync summary"
curl -fsS "${BASE_URL}/api/v1/runtime/sync-summary"
echo
echo
echo "[5/7] detect status"
curl -fsS "${BASE_URL}/api/v1/detect/status"
echo
echo
echo "[6/7] active detect job"
curl -fsS "${BASE_URL}/api/v1/detect/job/active"
echo
echo
echo "[7/7] condensed summary"
READINESS_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/readiness")"
SYNC_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/sync-summary")"
CLUSTER_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/cluster")"
"${PYTHON_BIN}" - <<'PY' "$READINESS_JSON" "$SYNC_JSON" "$CLUSTER_JSON"
import json
import sys
readiness = json.loads(sys.argv[1]).get("data", {})
sync = json.loads(sys.argv[2]).get("data", {})
cluster = json.loads(sys.argv[3]).get("data", {})
batches = (sync.get("detect_result_batches") or {})
states = batches.get("state_counts") or {}
summary = cluster.get("summary") or {}
print(
json.dumps(
{
"readiness": {
"status": readiness.get("status"),
"ready": bool(readiness.get("ready", False)),
"summary": readiness.get("summary", ""),
},
"cluster": {
"online_control_nodes": summary.get("online_control_nodes", 0),
"online_worker_nodes": summary.get("online_worker_nodes", 0),
"busy_nodes": summary.get("busy_nodes", []),
"stale_nodes": summary.get("stale_nodes", []),
"offline_nodes": summary.get("offline_nodes", []),
},
"sync": {
"enabled": bool(sync.get("enabled", False)),
"source_region": sync.get("source_region"),
"target_region": sync.get("target_region"),
"runtime_projection": (sync.get("type_counts") or {}).get("runtime_projection", 0),
"detect_result_projection": (sync.get("type_counts") or {}).get("detect_result_projection", 0),
"runtime_ingest": (sync.get("type_counts") or {}).get("runtime_ingest", 0),
"detect_result_ingest": (sync.get("type_counts") or {}).get("detect_result_ingest", 0),
},
"detect_result_batches": {
"jobs_total": batches.get("jobs_total", 0),
"synced": states.get("synced", 0),
"delivered": states.get("delivered", 0),
"pushing": states.get("pushing", 0),
"projected": states.get("projected", 0),
"failed": states.get("failed", 0),
"unsynced": states.get("unsynced", 0),
},
},
ensure_ascii=False,
indent=2,
)
)
PY

76
domain-api/deploy/multi-region/check_mainland_controller.sh Executable file
View File

@@ -0,0 +1,76 @@
#!/usr/bin/env bash
set -euo pipefail
ENV_FILE="${1:-/etc/default/domaincheck-worker}"
WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}"
SYNC_AGENT_SERVICE="${SYNC_AGENT_SERVICE:-domaincheck-sync-agent}"
PYTHON_BIN="${PYTHON_BIN:-python3}"
echo "[1/5] environment file"
if [ ! -f "$ENV_FILE" ]; then
echo "missing env file: $ENV_FILE"
exit 1
fi
echo "env file: $ENV_FILE"
echo
echo "[2/5] key env summary"
"${PYTHON_BIN}" - <<'PY' "$ENV_FILE"
import json
import sys
from pathlib import Path
env_path = Path(sys.argv[1])
values = {}
for line in env_path.read_text(encoding="utf-8").splitlines():
text = line.strip()
if not text or text.startswith("#") or "=" not in text:
continue
key, value = text.split("=", 1)
values[key.strip()] = value.strip()
required = {
"NODE_CODE": values.get("NODE_CODE", ""),
"NODE_REGION": values.get("NODE_REGION", ""),
"NODE_ROLE": values.get("NODE_ROLE", ""),
"SYNC_PUSH_ENABLED": values.get("SYNC_PUSH_ENABLED", ""),
"SYNC_SOURCE_REGION": values.get("SYNC_SOURCE_REGION", ""),
"SYNC_TARGET_REGION": values.get("SYNC_TARGET_REGION", ""),
"SYNC_TARGET_API_BASE_URL": values.get("SYNC_TARGET_API_BASE_URL", ""),
}
print(json.dumps(required, ensure_ascii=False, indent=2))
errors = []
if required["NODE_REGION"] != "mainland":
errors.append("NODE_REGION 必须为 mainland")
if required["NODE_ROLE"] != "control":
errors.append("NODE_ROLE 必须为 control")
if str(required["SYNC_PUSH_ENABLED"]).lower() != "true":
errors.append("SYNC_PUSH_ENABLED 必须为 true")
if required["SYNC_SOURCE_REGION"] != "mainland":
errors.append("SYNC_SOURCE_REGION 必须为 mainland")
if not required["SYNC_TARGET_API_BASE_URL"]:
errors.append("必须配置 SYNC_TARGET_API_BASE_URL")
if errors:
print()
print("env validation failed:")
for item in errors:
print(f"- {item}")
sys.exit(1)
PY
echo
echo "[3/5] systemd status"
systemctl is-enabled "$WORKER_SERVICE" || true
systemctl is-active "$WORKER_SERVICE"
systemctl is-enabled "$SYNC_AGENT_SERVICE" || true
systemctl is-active "$SYNC_AGENT_SERVICE"
echo
echo "[4/5] worker service detail"
systemctl status "$WORKER_SERVICE" --no-pager -l | sed -n '1,25p'
echo
echo "[5/5] sync-agent detail"
systemctl status "$SYNC_AGENT_SERVICE" --no-pager -l | sed -n '1,25p'

8
domain-api/deploy/multi-region/install_mainland_controller_quick.conf.example Normal file
View File

@@ -0,0 +1,8 @@
GIT_REPO_URL="YOUR_GIT_REPO_URL"
GIT_BRANCH="main"
DB_NAME="domain"
DB_USER="domainuser"
DB_PASSWORD="CHANGE_ME_DB_PASSWORD"
SYNC_SHARED_TOKEN="CHANGE_ME_SYNC_SHARED_TOKEN"
MAINLAND_CONTROLLER_NODE_CODE="mainland-controller-01"
TARGET_API_BASE_URL="https://api.domain.com/api/v1"

110
domain-api/deploy/multi-region/install_mainland_controller_quick.sh Executable file
View File

@@ -0,0 +1,110 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CONFIG_FILE="${SCRIPT_DIR}/install_mainland_controller_quick.conf"
CONFIG_EXAMPLE="${SCRIPT_DIR}/install_mainland_controller_quick.conf.example"
if [[ ! -f "${CONFIG_FILE}" ]]; then
echo "未找到配置文件：${CONFIG_FILE}"
echo "请先执行：cp ${CONFIG_EXAMPLE} ${CONFIG_FILE}"
exit 1
fi
# shellcheck disable=SC1090
source "${CONFIG_FILE}"
: "${GIT_REPO_URL:=YOUR_GIT_REPO_URL}"
: "${GIT_BRANCH:=main}"
: "${DB_NAME:=domain}"
: "${DB_USER:=domainuser}"
: "${DB_PASSWORD:=CHANGE_ME_DB_PASSWORD}"
: "${SYNC_SHARED_TOKEN:=CHANGE_ME_SYNC_SHARED_TOKEN}"
: "${MAINLAND_CONTROLLER_NODE_CODE:=mainland-controller-01}"
: "${TARGET_API_BASE_URL:=https://api.domain.com/api/v1}"
WORKSPACE_ROOT="/www/wwwroot"
REPO_DIR="${WORKSPACE_ROOT}/getDomain"
BASE_DIR="/opt/domaincheck"
DOMAINCHECK_DIR="${BASE_DIR}/domainCheck"
DOMAIN_API_DIR="${BASE_DIR}/domain-api"
if [[ "${GIT_REPO_URL}" == "YOUR_GIT_REPO_URL" || "${DB_PASSWORD}" == "CHANGE_ME_DB_PASSWORD" || "${SYNC_SHARED_TOKEN}" == "CHANGE_ME_SYNC_SHARED_TOKEN" ]]; then
echo "请先修改配置文件中的必填项：GIT_REPO_URL / DB_PASSWORD / SYNC_SHARED_TOKEN"
exit 1
fi
mkdir -p "${WORKSPACE_ROOT}"
cd "${WORKSPACE_ROOT}"
if [[ ! -d "${REPO_DIR}/.git" ]]; then
git clone "${GIT_REPO_URL}" getDomain
fi
cd "${REPO_DIR}"
git fetch --all
git checkout "${GIT_BRANCH}"
git pull --ff-only origin "${GIT_BRANCH}"
mkdir -p "${BASE_DIR}"
ln -sfn "${REPO_DIR}/domain-api" "${DOMAIN_API_DIR}"
ln -sfn "${REPO_DIR}/domain-web" "${BASE_DIR}/domain-web"
ln -sfn "${REPO_DIR}/domainCheck" "${DOMAINCHECK_DIR}"
cd "${DOMAINCHECK_DIR}"
python3.11 -m venv .venv || true
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
"${DOMAINCHECK_DIR}/.venv/bin/pip" install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
cat >"${DOMAINCHECK_DIR}/.env" <<EOF
DB_HOST=127.0.0.1
DB_PORT=5432
DB_DATABASE=${DB_NAME}
DB_USER=${DB_USER}
DB_PASSWORD=${DB_PASSWORD}
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
EOF
cd "${DOMAIN_API_DIR}"
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
cat >/etc/default/domaincheck-worker <<EOF
WORKER_MODE=linux-systemd
QT_QPA_PLATFORM=offscreen
NODE_CODE=${MAINLAND_CONTROLLER_NODE_CODE}
NODE_REGION=mainland
NODE_ROLE=control
DB_HOST=127.0.0.1
DB_PORT=5432
DB_DATABASE=${DB_NAME}
DB_USER=${DB_USER}
DB_PASSWORD=${DB_PASSWORD}
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
SYNC_PUSH_ENABLED=true
SYNC_SOURCE_REGION=mainland
SYNC_TARGET_REGION=overseas
SYNC_TARGET_API_BASE_URL=${TARGET_API_BASE_URL}
SYNC_SHARED_TOKEN=${SYNC_SHARED_TOKEN}
SYNC_BATCH_SIZE=200
SYNC_POLL_INTERVAL_SECONDS=30
EOF
systemctl daemon-reload
systemctl enable domaincheck-worker
systemctl enable domaincheck-sync-agent
systemctl restart domaincheck-worker
systemctl restart domaincheck-sync-agent
echo
echo "国内 controller 部署完成，建议继续执行："
echo "cd ${DOMAIN_API_DIR} && bash deploy/multi-region/check_mainland_controller.sh"

9
domain-api/deploy/multi-region/install_mainland_worker_quick.conf.example Normal file
View File

@@ -0,0 +1,9 @@
GIT_REPO_URL="YOUR_GIT_REPO_URL"
GIT_BRANCH="main"
DB_NAME="domain"
DB_USER="domainuser"
DB_PASSWORD="CHANGE_ME_DB_PASSWORD"
SYNC_SHARED_TOKEN="CHANGE_ME_SYNC_SHARED_TOKEN"
MAINLAND_WORKER_NODE_CODE="mainland-worker-01"
MAINLAND_CONTROLLER_IP="127.0.0.1"
TARGET_API_BASE_URL="https://api.domain.com/api/v1"

109
domain-api/deploy/multi-region/install_mainland_worker_quick.sh Executable file
View File

@@ -0,0 +1,109 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CONFIG_FILE="${SCRIPT_DIR}/install_mainland_worker_quick.conf"
CONFIG_EXAMPLE="${SCRIPT_DIR}/install_mainland_worker_quick.conf.example"
if [[ ! -f "${CONFIG_FILE}" ]]; then
echo "未找到配置文件：${CONFIG_FILE}"
echo "请先执行：cp ${CONFIG_EXAMPLE} ${CONFIG_FILE}"
exit 1
fi
# shellcheck disable=SC1090
source "${CONFIG_FILE}"
: "${GIT_REPO_URL:=YOUR_GIT_REPO_URL}"
: "${GIT_BRANCH:=main}"
: "${DB_NAME:=domain}"
: "${DB_USER:=domainuser}"
: "${DB_PASSWORD:=CHANGE_ME_DB_PASSWORD}"
: "${SYNC_SHARED_TOKEN:=CHANGE_ME_SYNC_SHARED_TOKEN}"
: "${MAINLAND_WORKER_NODE_CODE:=mainland-worker-01}"
: "${MAINLAND_CONTROLLER_IP:=127.0.0.1}"
: "${TARGET_API_BASE_URL:=https://api.domain.com/api/v1}"
WORKSPACE_ROOT="/www/wwwroot"
REPO_DIR="${WORKSPACE_ROOT}/getDomain"
BASE_DIR="/opt/domaincheck"
DOMAINCHECK_DIR="${BASE_DIR}/domainCheck"
DOMAIN_API_DIR="${BASE_DIR}/domain-api"
if [[ "${GIT_REPO_URL}" == "YOUR_GIT_REPO_URL" || "${DB_PASSWORD}" == "CHANGE_ME_DB_PASSWORD" || "${SYNC_SHARED_TOKEN}" == "CHANGE_ME_SYNC_SHARED_TOKEN" || "${MAINLAND_CONTROLLER_IP}" == "127.0.0.1" ]]; then
echo "请先修改配置文件中的必填项：GIT_REPO_URL / DB_PASSWORD / SYNC_SHARED_TOKEN / MAINLAND_CONTROLLER_IP"
exit 1
fi
mkdir -p "${WORKSPACE_ROOT}"
cd "${WORKSPACE_ROOT}"
if [[ ! -d "${REPO_DIR}/.git" ]]; then
git clone "${GIT_REPO_URL}" getDomain
fi
cd "${REPO_DIR}"
git fetch --all
git checkout "${GIT_BRANCH}"
git pull --ff-only origin "${GIT_BRANCH}"
mkdir -p "${BASE_DIR}"
ln -sfn "${REPO_DIR}/domain-api" "${DOMAIN_API_DIR}"
ln -sfn "${REPO_DIR}/domain-web" "${BASE_DIR}/domain-web"
ln -sfn "${REPO_DIR}/domainCheck" "${DOMAINCHECK_DIR}"
cd "${DOMAINCHECK_DIR}"
python3.11 -m venv .venv || true
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
"${DOMAINCHECK_DIR}/.venv/bin/pip" install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
cat >"${DOMAINCHECK_DIR}/.env" <<EOF
DB_HOST=${MAINLAND_CONTROLLER_IP}
DB_PORT=5432
DB_DATABASE=${DB_NAME}
DB_USER=${DB_USER}
DB_PASSWORD=${DB_PASSWORD}
REDIS_HOST=${MAINLAND_CONTROLLER_IP}
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
EOF
cd "${DOMAIN_API_DIR}"
bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
cat >/etc/default/domaincheck-worker <<EOF
WORKER_MODE=linux-systemd
QT_QPA_PLATFORM=offscreen
NODE_CODE=${MAINLAND_WORKER_NODE_CODE}
NODE_REGION=mainland
NODE_ROLE=worker
DB_HOST=${MAINLAND_CONTROLLER_IP}
DB_PORT=5432
DB_DATABASE=${DB_NAME}
DB_USER=${DB_USER}
DB_PASSWORD=${DB_PASSWORD}
REDIS_HOST=${MAINLAND_CONTROLLER_IP}
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
SYNC_PUSH_ENABLED=true
SYNC_SOURCE_REGION=mainland
SYNC_TARGET_REGION=overseas
SYNC_TARGET_API_BASE_URL=${TARGET_API_BASE_URL}
SYNC_SHARED_TOKEN=${SYNC_SHARED_TOKEN}
SYNC_BATCH_SIZE=200
SYNC_POLL_INTERVAL_SECONDS=30
EOF
systemctl daemon-reload
systemctl enable domaincheck-worker
systemctl restart domaincheck-worker
echo
echo "国内 worker 部署完成，建议继续在国外机查看："
echo "curl https://api.domain.com/api/v1/runtime/cluster"

9
domain-api/deploy/multi-region/install_overseas_quick.conf.example Normal file
View File

@@ -0,0 +1,9 @@
GIT_REPO_URL="YOUR_GIT_REPO_URL"
GIT_BRANCH="main"
API_DOMAIN="api.domain.com"
ADMIN_DOMAIN="admin.domain"
DB_NAME="domain"
DB_USER="domainuser"
DB_PASSWORD="CHANGE_ME_DB_PASSWORD"
OVERSEAS_NODE_CODE="overseas-control-01"
NODE_VERSION="20"

124
domain-api/deploy/multi-region/install_overseas_quick.sh Executable file
View File

@@ -0,0 +1,124 @@
#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CONFIG_FILE="${SCRIPT_DIR}/install_overseas_quick.conf"
CONFIG_EXAMPLE="${SCRIPT_DIR}/install_overseas_quick.conf.example"
if [[ ! -f "${CONFIG_FILE}" ]]; then
echo "未找到配置文件：${CONFIG_FILE}"
echo "请先执行：cp ${CONFIG_EXAMPLE} ${CONFIG_FILE}"
exit 1
fi
# shellcheck disable=SC1090
source "${CONFIG_FILE}"
: "${GIT_REPO_URL:=YOUR_GIT_REPO_URL}"
: "${GIT_BRANCH:=main}"
: "${API_DOMAIN:=api.domain.com}"
: "${ADMIN_DOMAIN:=admin.domain}"
: "${DB_NAME:=domain}"
: "${DB_USER:=domainuser}"
: "${DB_PASSWORD:=CHANGE_ME_DB_PASSWORD}"
: "${OVERSEAS_NODE_CODE:=overseas-control-01}"
: "${NODE_VERSION:=20}"
WORKSPACE_ROOT="/www/wwwroot"
REPO_DIR="${WORKSPACE_ROOT}/getDomain"
BASE_DIR="/opt/domaincheck"
DOMAINCHECK_DIR="${BASE_DIR}/domainCheck"
DOMAIN_API_DIR="${BASE_DIR}/domain-api"
DOMAIN_WEB_DIR="${BASE_DIR}/domain-web"
if [[ "${GIT_REPO_URL}" == "YOUR_GIT_REPO_URL" || "${DB_PASSWORD}" == "CHANGE_ME_DB_PASSWORD" ]]; then
echo "请先修改配置文件中的必填项：GIT_REPO_URL / DB_PASSWORD"
exit 1
fi
mkdir -p "${WORKSPACE_ROOT}"
cd "${WORKSPACE_ROOT}"
if [[ ! -d "${REPO_DIR}/.git" ]]; then
git clone "${GIT_REPO_URL}" getDomain
fi
cd "${REPO_DIR}"
git fetch --all
git checkout "${GIT_BRANCH}"
git pull --ff-only origin "${GIT_BRANCH}"
mkdir -p "${BASE_DIR}"
ln -sfn "${REPO_DIR}/domain-api" "${DOMAIN_API_DIR}"
ln -sfn "${REPO_DIR}/domain-web" "${DOMAIN_WEB_DIR}"
ln -sfn "${REPO_DIR}/domainCheck" "${DOMAINCHECK_DIR}"
cd "${DOMAINCHECK_DIR}"
python3.11 -m venv .venv || true
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
"${DOMAINCHECK_DIR}/.venv/bin/pip" install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
cat >"${DOMAINCHECK_DIR}/.env" <<EOF
DB_HOST=127.0.0.1
DB_PORT=5432
DB_DATABASE=${DB_NAME}
DB_USER=${DB_USER}
DB_PASSWORD=${DB_PASSWORD}
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
EOF
"${DOMAINCHECK_DIR}/.venv/bin/python" "${DOMAINCHECK_DIR}/init_database.py"
cat >"${DOMAIN_WEB_DIR}/.env.production" <<EOF
VITE_API_BASE_URL=https://${API_DOMAIN}/api/v1
EOF
export NVM_DIR="${HOME}/.nvm"
if [[ -s "${NVM_DIR}/nvm.sh" ]]; then
# shellcheck disable=SC1090
source "${NVM_DIR}/nvm.sh"
else
echo "未找到 nvm，请先安装 nvm"
exit 1
fi
nvm install "${NODE_VERSION}"
nvm use "${NODE_VERSION}"
cd "${DOMAIN_WEB_DIR}"
npm install
npm run build
cp "${DOMAIN_API_DIR}/deploy/multi-region/templates/domaincheck-api.env.example" /etc/default/domaincheck-api
sed -i "s#^DOMAIN_ROOT=.*#DOMAIN_ROOT=${DOMAINCHECK_DIR}#" /etc/default/domaincheck-api
sed -i 's#^API_HOST=.*#API_HOST=0.0.0.0#' /etc/default/domaincheck-api
sed -i 's#^API_PORT=.*#API_PORT=8100#' /etc/default/domaincheck-api
sed -i "s#^NODE_CODE=.*#NODE_CODE=${OVERSEAS_NODE_CODE}#" /etc/default/domaincheck-api
sed -i 's#^NODE_REGION=.*#NODE_REGION=overseas#' /etc/default/domaincheck-api
sed -i 's#^NODE_ROLE=.*#NODE_ROLE=control#' /etc/default/domaincheck-api
sed -i 's#^SYNC_PUSH_ENABLED=.*#SYNC_PUSH_ENABLED=false#' /etc/default/domaincheck-api
sed -i "s#^CORS_ORIGINS=.*#CORS_ORIGINS=https://${ADMIN_DOMAIN},http://127.0.0.1:3201,http://localhost:3201#" /etc/default/domaincheck-api
cp "${DOMAIN_API_DIR}/deploy/systemd/domain-api.service" /etc/systemd/system/domaincheck-api.service
cp "${DOMAIN_API_DIR}/deploy/systemd/domain-worker.service" /etc/systemd/system/domaincheck-worker.service
mkdir -p "${DOMAIN_API_DIR}/runtime"
touch "${DOMAINCHECK_DIR}/detect_worker.log"
chown -R www:www "${DOMAIN_API_DIR}/runtime"
chown www:www "${DOMAINCHECK_DIR}/detect_worker.log"
chmod 664 "${DOMAINCHECK_DIR}/detect_worker.log"
systemctl daemon-reload
systemctl enable domaincheck-api
systemctl enable domaincheck-worker
systemctl restart domaincheck-api
systemctl restart domaincheck-worker
echo
echo "国外主控机部署完成，建议继续执行："
echo "curl http://127.0.0.1:8100/health"
echo "curl http://127.0.0.1:8100/api/v1/runtime/preflight"
echo "curl http://127.0.0.1:8100/api/v1/runtime/readiness"

82
domain-api/deploy/multi-region/prune_cluster_nodes.py Normal file
View File

@@ -0,0 +1,82 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import sys
from datetime import datetime, timedelta
from pathlib import Path
API_ROOT = Path(__file__).resolve().parents[2]
if str(API_ROOT) not in sys.path:
sys.path.insert(0, str(API_ROOT))
from app.core.db import get_db
def main() -> None:
parser = argparse.ArgumentParser(description="Prune stale/offline cluster node records from detect_worker_nodes.")
parser.add_argument("--minutes", type=int, default=30, help="delete nodes whose heartbeat is older than this many minutes")
parser.add_argument("--node-code", action="append", default=[], help="delete a specific node code; can be passed multiple times")
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
cutoff = datetime.now() - timedelta(minutes=max(1, int(args.minutes or 30)))
removed: list[dict] = []
with get_db() as conn:
with conn.cursor() as cur:
if args.node_code:
cur.execute(
"""
SELECT node_code, region, role, status, last_heartbeat_at
FROM detect_worker_nodes
WHERE node_code = ANY(%s)
ORDER BY node_code ASC
""",
(list(args.node_code),),
)
else:
cur.execute(
"""
SELECT node_code, region, role, status, last_heartbeat_at
FROM detect_worker_nodes
WHERE last_heartbeat_at < %s
ORDER BY last_heartbeat_at ASC, node_code ASC
""",
(cutoff,),
)
rows = cur.fetchall()
for row in rows:
removed.append(
{
"node_code": row[0],
"region": row[1],
"role": row[2],
"status": row[3],
"last_heartbeat_at": row[4].isoformat(sep=" ", timespec="seconds") if row[4] else "",
}
)
if removed and not args.dry_run:
cur.execute(
"DELETE FROM detect_worker_nodes WHERE node_code = ANY(%s)",
([item["node_code"] for item in removed],),
)
if removed and not args.dry_run:
conn.commit()
if not removed:
print("no cluster nodes matched prune conditions")
return
print("matched cluster nodes:")
for item in removed:
print(
f"- {item['node_code']} | {item['region']} | {item['role']} | {item['status']} | {item['last_heartbeat_at']}"
)
if args.dry_run:
print("dry-run only, nothing deleted")
else:
print(f"deleted {len(removed)} cluster node record(s)")
if __name__ == "__main__":
main()

7
domain-api/deploy/multi-region/prune_cluster_nodes.sh Executable file
View File

@@ -0,0 +1,7 @@
#!/usr/bin/env bash
set -euo pipefail
API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}"
exec "$PYTHON_BIN" "$API_DIR/deploy/multi-region/prune_cluster_nodes.py" "$@"

135
domain-api/deploy/multi-region/rehearse_multi_region.sh Executable file
View File

@@ -0,0 +1,135 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}"
SIM_SCRIPT="$API_DIR/deploy/multi-region/simulate_cluster_node.py"
PRUNE_SCRIPT="$API_DIR/deploy/multi-region/prune_cluster_nodes.sh"
RUNTIME_DIR="$API_DIR/runtime"
mkdir -p "$RUNTIME_DIR"
CTRL_LOG="$RUNTIME_DIR/rehearse-mainland-controller.log"
WORKER1_LOG="$RUNTIME_DIR/rehearse-mainland-worker-01.log"
WORKER2_LOG="$RUNTIME_DIR/rehearse-mainland-worker-02.log"
CTRL_NODE="mainland-controller-rehearsal"
WORKER1_NODE="mainland-worker-rehearsal-01"
WORKER2_NODE="mainland-worker-rehearsal-02"
cleanup() {
set +e
if [ -n "${CTRL_PID:-}" ]; then kill "$CTRL_PID" >/dev/null 2>&1 || true; fi
if [ -n "${WORKER1_PID:-}" ]; then kill "$WORKER1_PID" >/dev/null 2>&1 || true; fi
if [ -n "${WORKER2_PID:-}" ]; then kill "$WORKER2_PID" >/dev/null 2>&1 || true; fi
bash "$PRUNE_SCRIPT" --node-code "$CTRL_NODE" --node-code "$WORKER1_NODE" --node-code "$WORKER2_NODE" >/dev/null 2>&1 || true
}
trap cleanup EXIT
echo "[0/6] cleaning possible stale rehearsal nodes"
bash "$PRUNE_SCRIPT" --node-code "$CTRL_NODE" --node-code "$WORKER1_NODE" --node-code "$WORKER2_NODE" >/dev/null 2>&1 || true
echo "[1/6] start simulated mainland controller"
"$PYTHON_BIN" "$SIM_SCRIPT" \
--node-code "$CTRL_NODE" \
--region mainland \
--role control \
--status online \
--detail "多机演练 controller" \
--interval 15 >"$CTRL_LOG" 2>&1 &
CTRL_PID=$!
echo "[2/6] start simulated mainland workers"
"$PYTHON_BIN" "$SIM_SCRIPT" \
--node-code "$WORKER1_NODE" \
--region mainland \
--role worker \
--status busy \
--current-load 1 \
--phase "running" \
--detail "多机演练 worker-01" \
--job-id 1 \
--job-code "detect-rehearsal-cluster" \
--cycle-token "rehearsal001" \
--active-threads 5 \
--available-proxy-count 12 \
--interval 15 >"$WORKER1_LOG" 2>&1 &
WORKER1_PID=$!
"$PYTHON_BIN" "$SIM_SCRIPT" \
--node-code "$WORKER2_NODE" \
--region mainland \
--role worker \
--status online \
--current-load 0 \
--phase "idle" \
--detail "多机演练 worker-02" \
--job-id 0 \
--job-code "" \
--cycle-token "" \
--active-threads 0 \
--available-proxy-count 8 \
--interval 15 >"$WORKER2_LOG" 2>&1 &
WORKER2_PID=$!
sleep 2
echo "[3/6] fetch runtime readiness"
READINESS_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/readiness")"
echo "$READINESS_JSON"
echo
echo "[4/6] fetch runtime cluster"
CLUSTER_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/cluster")"
echo "$CLUSTER_JSON"
echo
echo "[5/6] assert rehearsal result"
"${PYTHON_BIN}" - <<'PY' "$READINESS_JSON" "$CLUSTER_JSON" "$CTRL_NODE" "$WORKER1_NODE" "$WORKER2_NODE"
import json
import sys
readiness = json.loads(sys.argv[1]).get("data", {})
cluster = json.loads(sys.argv[2]).get("data", {})
ctrl_node, worker1_node, worker2_node = sys.argv[3:6]
errors = []
if readiness.get("status") != "ready":
errors.append(f"readiness.status expected ready, got {readiness.get('status')!r}")
if not readiness.get("ready", False):
errors.append("readiness.ready expected true")
summary = cluster.get("summary") or {}
if int(summary.get("online_control_nodes", 0) or 0) < 2:
errors.append(f"online_control_nodes expected >= 2, got {summary.get('online_control_nodes')}")
if int(summary.get("online_worker_nodes", 0) or 0) < 2:
errors.append(f"online_worker_nodes expected >= 2, got {summary.get('online_worker_nodes')}")
nodes = {str(item.get('node_code') or ''): item for item in (cluster.get("nodes") or [])}
for node_code in (ctrl_node, worker1_node, worker2_node):
if node_code not in nodes:
errors.append(f"missing simulated node in cluster: {node_code}")
if worker1_node in nodes and str(nodes[worker1_node].get("status")) != "busy":
errors.append(f"{worker1_node} expected busy, got {nodes[worker1_node].get('status')!r}")
if worker2_node in nodes and str(nodes[worker2_node].get("status")) != "online":
errors.append(f"{worker2_node} expected online, got {nodes[worker2_node].get('status')!r}")
if errors:
print("rehearsal failed:")
for item in errors:
print(f"- {item}")
raise SystemExit(1)
print("rehearsal passed")
print(json.dumps({
"readiness": readiness,
"cluster_summary": summary,
}, ensure_ascii=False, indent=2))
PY
echo
echo "[6/6] rehearsal completed"
echo "controller log: $CTRL_LOG"
echo "worker1 log: $WORKER1_LOG"
echo "worker2 log: $WORKER2_LOG"

113
domain-api/deploy/multi-region/simulate_cluster_node.py Executable file
View File

@@ -0,0 +1,113 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import json
import sys
import time
from pathlib import Path
API_ROOT = Path(__file__).resolve().parents[2]
if str(API_ROOT) not in sys.path:
sys.path.insert(0, str(API_ROOT))
from app.core.db import get_db
from app.services.cluster_runtime_service import ensure_runtime_schema, register_node_heartbeat
def _metadata_from_args(args: argparse.Namespace) -> dict:
metadata = {
"service": "cluster-simulator",
"simulated": True,
"phase": args.phase or "",
"detail": args.detail or "",
}
if args.job_id:
metadata["job_id"] = args.job_id
if args.job_code:
metadata["job_code"] = args.job_code
if args.cycle_token:
metadata["cycle_token"] = args.cycle_token
if args.active_threads is not None:
metadata["active_threads"] = args.active_threads
if args.available_proxy_count is not None:
metadata["available_proxy_count"] = args.available_proxy_count
if args.metadata_json:
try:
custom = json.loads(args.metadata_json)
if isinstance(custom, dict):
metadata.update(custom)
except Exception as exc:
raise SystemExit(f"invalid --metadata-json: {exc}") from exc
return metadata
def _delete_node(node_code: str) -> None:
with get_db() as conn:
with conn.cursor() as cur:
cur.execute("DELETE FROM detect_worker_nodes WHERE node_code = %s", (str(node_code),))
conn.commit()
def main() -> None:
parser = argparse.ArgumentParser(description="Simulate a cluster node heartbeat for multi-region rehearsal.")
parser.add_argument("--node-code", required=True)
parser.add_argument("--region", required=True)
parser.add_argument("--role", required=True, choices=("control", "worker"))
parser.add_argument("--status", default="online")
parser.add_argument("--current-load", type=int, default=0)
parser.add_argument("--phase", default="")
parser.add_argument("--detail", default="")
parser.add_argument("--job-id", type=int, default=0)
parser.add_argument("--job-code", default="")
parser.add_argument("--cycle-token", default="")
parser.add_argument("--active-threads", type=int, default=None)
parser.add_argument("--available-proxy-count", type=int, default=None)
parser.add_argument("--metadata-json", default="")
parser.add_argument("--interval", type=int, default=20)
parser.add_argument("--iterations", type=int, default=0, help="0 means run forever")
parser.add_argument("--clear", action="store_true", help="delete the node record and exit")
args = parser.parse_args()
ensure_runtime_schema()
if args.clear:
_delete_node(args.node_code)
print(f"cleared simulated node: {args.node_code}")
return
metadata = _metadata_from_args(args)
interval = max(5, int(args.interval or 20))
iterations = max(0, int(args.iterations or 0))
current = 0
while True:
current += 1
register_node_heartbeat(
node_code=args.node_code,
region=args.region,
role=args.role,
status=args.status,
current_load=max(0, int(args.current_load or 0)),
metadata=metadata,
)
print(
json.dumps(
{
"node_code": args.node_code,
"region": args.region,
"role": args.role,
"status": args.status,
"current_load": max(0, int(args.current_load or 0)),
"iteration": current,
"metadata": metadata,
},
ensure_ascii=False,
),
flush=True,
)
if iterations and current >= iterations:
break
time.sleep(interval)
if __name__ == "__main__":
main()

65
domain-api/deploy/multi-region/simulate_multi_region.sh Executable file
View File

@@ -0,0 +1,65 @@
#!/usr/bin/env bash
set -euo pipefail
BASE_URL="${1:-http://127.0.0.1:8100}"
API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}"
SIM_SCRIPT="$API_DIR/deploy/multi-region/simulate_cluster_node.py"
RUNTIME_DIR="$API_DIR/runtime"
mkdir -p "$RUNTIME_DIR"
CONTROLLER_LOG="$RUNTIME_DIR/sim-mainland-controller.log"
WORKER_LOG="$RUNTIME_DIR/sim-mainland-worker.log"
cleanup() {
set +e
if [ -n "${CTRL_PID:-}" ]; then kill "$CTRL_PID" >/dev/null 2>&1 || true; fi
if [ -n "${WORKER_PID:-}" ]; then kill "$WORKER_PID" >/dev/null 2>&1 || true; fi
"$PYTHON_BIN" "$SIM_SCRIPT" --node-code mainland-controller-sim --region mainland --role control --clear >/dev/null 2>&1 || true
"$PYTHON_BIN" "$SIM_SCRIPT" --node-code mainland-worker-sim-01 --region mainland --role worker --clear >/dev/null 2>&1 || true
}
trap cleanup EXIT
echo "[1/4] start simulated mainland controller"
"$PYTHON_BIN" "$SIM_SCRIPT" \
--node-code mainland-controller-sim \
--region mainland \
--role control \
--status online \
--detail "模拟大陆 controller 联调节点" \
--interval 15 >"$CONTROLLER_LOG" 2>&1 &
CTRL_PID=$!
echo "[2/4] start simulated mainland worker"
"$PYTHON_BIN" "$SIM_SCRIPT" \
--node-code mainland-worker-sim-01 \
--region mainland \
--role worker \
--status busy \
--current-load 1 \
--phase "running" \
--detail "模拟大陆 worker 执行中" \
--job-id 1 \
--job-code "detect-simulated-cluster" \
--cycle-token "simcycle001" \
--active-threads 5 \
--available-proxy-count 12 \
--interval 15 >"$WORKER_LOG" 2>&1 &
WORKER_PID=$!
sleep 2
echo "[3/4] runtime readiness"
curl -fsS "${BASE_URL}/api/v1/runtime/readiness"
echo
echo
echo "[4/4] runtime cluster summary"
curl -fsS "${BASE_URL}/api/v1/runtime/cluster"
echo
echo
echo "simulated nodes running:"
echo " controller log: $CONTROLLER_LOG"
echo " worker log: $WORKER_LOG"
echo "press Ctrl+C to stop simulation and clean up"
wait

30
domain-api/deploy/multi-region/templates/domaincheck-api.env.example Normal file
View File

@@ -0,0 +1,30 @@
WORKER_MODE=linux-systemd
API_HOST=0.0.0.0
API_PORT=8100
DOMAIN_ROOT=__BASE_DIR__/domainCheck
NODE_CODE=overseas-control-01
NODE_REGION=overseas
NODE_ROLE=control
CORS_ORIGINS=http://127.0.0.1:3201,http://localhost:3201
# DB
# DB_HOST=127.0.0.1
# DB_PORT=5432
# DB_DATABASE=domain
# DB_USER=postgres
# DB_PASSWORD=postgres
# REDIS
# REDIS_HOST=127.0.0.1
# REDIS_PORT=6379
# REDIS_PASSWORD=
# REDIS_DB=0
# SYNC
SYNC_PUSH_ENABLED=false
SYNC_SOURCE_REGION=overseas
SYNC_TARGET_REGION=overseas
SYNC_TARGET_API_BASE_URL=
SYNC_SHARED_TOKEN=
SYNC_BATCH_SIZE=200
SYNC_POLL_INTERVAL_SECONDS=30

27
domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example Normal file
View File

@@ -0,0 +1,27 @@
WORKER_MODE=linux-systemd
QT_QPA_PLATFORM=offscreen
NODE_CODE=mainland-controller-01
NODE_REGION=mainland
NODE_ROLE=control
# DB
# DB_HOST=127.0.0.1
# DB_PORT=5432
# DB_DATABASE=domain
# DB_USER=postgres
# DB_PASSWORD=postgres
# REDIS
# REDIS_HOST=127.0.0.1
# REDIS_PORT=6379
# REDIS_PASSWORD=
# REDIS_DB=0
# SYNC
SYNC_PUSH_ENABLED=true
SYNC_SOURCE_REGION=mainland
SYNC_TARGET_REGION=overseas
SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
SYNC_SHARED_TOKEN=
SYNC_BATCH_SIZE=200
SYNC_POLL_INTERVAL_SECONDS=30

27
domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example Normal file
View File

@@ -0,0 +1,27 @@
WORKER_MODE=linux-systemd
QT_QPA_PLATFORM=offscreen
NODE_CODE=mainland-worker-01
NODE_REGION=mainland
NODE_ROLE=worker
# DB
# DB_HOST=127.0.0.1
# DB_PORT=5432
# DB_DATABASE=domain
# DB_USER=postgres
# DB_PASSWORD=postgres
# REDIS
# REDIS_HOST=127.0.0.1
# REDIS_PORT=6379
# REDIS_PASSWORD=
# REDIS_DB=0
# SYNC
SYNC_PUSH_ENABLED=true
SYNC_SOURCE_REGION=mainland
SYNC_TARGET_REGION=overseas
SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
SYNC_SHARED_TOKEN=
SYNC_BATCH_SIZE=200
SYNC_POLL_INTERVAL_SECONDS=30

7
domain-api/deploy/systemd/domain-api.service
View File

@@ -5,14 +5,17 @@ After=network.target redis.service postgresql.service
[Service] [Service]
Type=simple Type=simple
WorkingDirectory=/opt/domaincheck/domain-api WorkingDirectory=/opt/domaincheck/domain-api
EnvironmentFile=-/etc/default/domaincheck-api
Environment="PATH=/home/www/.nvm/versions/node/v20.20.2/bin:/home/www/.local/bin:/home/www/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
Environment="WORKER_MODE=linux-systemd" Environment="WORKER_MODE=linux-systemd"
Environment="API_HOST=0.0.0.0" Environment="API_HOST=0.0.0.0"
Environment="API_PORT=8100" Environment="API_PORT=8100"
Environment="DOMAIN_ROOT=/opt/domaincheck/domainCheck"
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100 ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m uvicorn app.main:app --host 0.0.0.0 --port 8100
Restart=always Restart=always
RestartSec=5 RestartSec=5
User=www-data User=www
Group=www-data Group=www
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target

17
domain-api/deploy/systemd/domain-sync-agent.service Normal file
View File

@@ -0,0 +1,17 @@
[Unit]
Description=domainCheck Sync Agent
After=network.target postgresql.service
[Service]
Type=simple
WorkingDirectory=/opt/domaincheck/domain-api
EnvironmentFile=-/etc/default/domaincheck-worker
Environment="PATH=/home/www/.nvm/versions/node/v20.20.2/bin:/home/www/.local/bin:/home/www/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin"
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python -m app.sync_agent
Restart=always
RestartSec=5
User=www
Group=www
[Install]
WantedBy=multi-user.target

6
domain-api/deploy/systemd/domain-worker.service
View File

@@ -5,12 +5,14 @@ After=network.target redis.service postgresql.service
[Service] [Service]
Type=simple Type=simple
WorkingDirectory=/opt/domaincheck/domainCheck WorkingDirectory=/opt/domaincheck/domainCheck
EnvironmentFile=-/etc/default/domaincheck-worker
Environment="WORKER_MODE=linux-systemd" Environment="WORKER_MODE=linux-systemd"
Environment="QT_QPA_PLATFORM=offscreen"
ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python /opt/domaincheck/domainCheck/detect_worker.py ExecStart=/opt/domaincheck/domainCheck/.venv/bin/python /opt/domaincheck/domainCheck/detect_worker.py
Restart=always Restart=always
RestartSec=5 RestartSec=5
User=www-data User=www
Group=www-data Group=www
[Install] [Install]
WantedBy=multi-user.target WantedBy=multi-user.target

35
domain-web/src/api/modules.ts
View File

@@ -10,18 +10,24 @@ export const dashboardApi = {
export const runtimeApi = { export const runtimeApi = {
status: () => http.get("/runtime/status"), status: () => http.get("/runtime/status"),
readiness: () => http.get("/runtime/readiness"),
preflight: () => http.get("/runtime/preflight"), preflight: () => http.get("/runtime/preflight"),
cluster: () => http.get("/runtime/cluster"),
action: (action: string) => http.post(`/runtime/actions/${action}`) action: (action: string) => http.post(`/runtime/actions/${action}`)
}; };
export const settingsApi = { export const settingsApi = {
getSettings: () => http.get("/settings"), getSettings: () => http.get("/settings"),
updateSettings: (payload: Record<string, unknown>) => http.put("/settings", payload), updateSettings: (payload: Record<string, unknown>) => http.put("/settings", payload),
getCredentials: () => http.get("/settings/credentials"),
updateCredentials: (payload: Record<string, unknown>) => http.put("/settings/credentials", payload),
exportSettings: () => http.get("/settings/export"), exportSettings: () => http.get("/settings/export"),
importSettings: (payload: Record<string, unknown>) => http.post("/settings/import", payload), importSettings: (payload: Record<string, unknown>) => http.post("/settings/import", payload),
validateImportSettings: (payload: Record<string, unknown>) => http.post("/settings/validate-import", payload), validateImportSettings: (payload: Record<string, unknown>) => http.post("/settings/validate-import", payload),
backupSettings: () => http.post("/settings/backup"), backupSettings: () => http.post("/settings/backup"),
getSettingsBackups: () => http.get("/settings/backups"), getSettingsBackups: () => http.get("/settings/backups"),
getJuziseoStatus: () => http.get("/settings/juziseo/status"),
loginJuziseo: (payload: { email: string; password: string }) => http.post("/settings/juziseo/login", payload),
backupDownloadUrl: (filename: string) => backupDownloadUrl: (filename: string) =>
`${(import.meta.env.VITE_API_BASE_URL || "http://127.0.0.1:8100/api/v1").replace(/\/api\/v1\/?$/, "")}/api/v1/settings/backups/download/${encodeURIComponent(filename)}` `${(import.meta.env.VITE_API_BASE_URL || "http://127.0.0.1:8100/api/v1").replace(/\/api\/v1\/?$/, "")}/api/v1/settings/backups/download/${encodeURIComponent(filename)}`
}; };
@@ -30,6 +36,7 @@ export const importsApi = {
summary: () => http.get("/imports/summary"), summary: () => http.get("/imports/summary"),
tasks: () => http.get("/imports/tasks"), tasks: () => http.get("/imports/tasks"),
retry: (taskId: string) => http.post(`/imports/tasks/${taskId}/retry`), retry: (taskId: string) => http.post(`/imports/tasks/${taskId}/retry`),
createTextTask: (payload: { text: string; filename?: string }) => http.post("/imports/text", payload),
upload: (formData: FormData) => upload: (formData: FormData) =>
http.post("/imports/upload", formData, { http.post("/imports/upload", formData, {
headers: { headers: {
@@ -40,6 +47,10 @@ export const importsApi = {
export const detectApi = { export const detectApi = {
status: () => http.get("/detect/status"), status: () => http.get("/detect/status"),
activeJob: () => http.get("/detect/job/active"),
queueSummary: (params?: Record<string, unknown>) => http.get("/detect/queue-summary", { params }),
jobs: (params?: Record<string, unknown>) => http.get("/detect/jobs", { params }),
jobDetail: (jobId: number) => http.get(`/detect/jobs/${jobId}`),
start: () => http.post("/detect/start"), start: () => http.post("/detect/start"),
stop: () => http.post("/detect/stop") stop: () => http.post("/detect/stop")
}; };
@@ -47,6 +58,7 @@ export const detectApi = {
export const domainsApi = { export const domainsApi = {
filters: () => http.get("/domains/filters"), filters: () => http.get("/domains/filters"),
list: (params?: Record<string, unknown>) => http.get("/domains", { params }), list: (params?: Record<string, unknown>) => http.get("/domains", { params }),
detail: (domainId: number) => http.get(`/domains/${domainId}`),
batchUpdate: (payload: Record<string, unknown>) => http.post("/domains/batch-update", payload) batchUpdate: (payload: Record<string, unknown>) => http.post("/domains/batch-update", payload)
}; };
@@ -59,3 +71,26 @@ export const logsApi = {
latest: () => http.get("/logs/latest"), latest: () => http.get("/logs/latest"),
bundleUrl: () => `${(import.meta.env.VITE_API_BASE_URL || "http://127.0.0.1:8100/api/v1").replace(/\/api\/v1\/?$/, "")}/api/v1/logs/bundle` bundleUrl: () => `${(import.meta.env.VITE_API_BASE_URL || "http://127.0.0.1:8100/api/v1").replace(/\/api\/v1\/?$/, "")}/api/v1/logs/bundle`
}; };
export const sensitiveWordsApi = {
get: () => http.get("/sensitive-words"),
save: (payload: { text: string }) => http.put("/sensitive-words", payload)
};
export const jumingApi = {
status: () => http.get("/juming/status"),
preferences: () => http.get("/juming/preferences"),
updatePreferences: (payload: Record<string, unknown>) => http.put("/juming/preferences", payload),
login: (payload: { email: string; password: string }) => http.post("/juming/login", payload),
loginJucha: () => http.post("/juming/jucha-login"),
crawl: (payload: Record<string, unknown>) => http.post("/juming/crawl", payload),
tasks: () => http.get("/juming/tasks"),
retryTask: (taskId: string) => http.post(`/juming/tasks/${taskId}/retry`),
stopTask: (taskId: string) => http.post(`/juming/tasks/${taskId}/stop`),
uploadCookie: (formData: FormData) =>
http.post("/juming/cookie/upload", formData, {
headers: {
"Content-Type": "multipart/form-data"
}
})
};

12
domain-web/src/components/PageCard.vue
View File

@@ -27,6 +27,11 @@ defineProps<{
border-radius: 20px; border-radius: 20px;
box-shadow: 0 18px 40px rgba(15, 23, 42, 0.05); box-shadow: 0 18px 40px rgba(15, 23, 42, 0.05);
padding: 22px; padding: 22px;
height: 100%;
display: flex;
flex-direction: column;
min-height: 0;
overflow: hidden;
} }
.page-card__header { .page-card__header {
@@ -35,6 +40,7 @@ defineProps<{
align-items: flex-start; align-items: flex-start;
gap: 16px; gap: 16px;
margin-bottom: 18px; margin-bottom: 18px;
flex-shrink: 0;
} }
.page-card__header h3 { .page-card__header h3 {
@@ -46,4 +52,10 @@ defineProps<{
margin: 8px 0 0; margin: 8px 0 0;
color: #64748b; color: #64748b;
} }
.page-card__content {
flex: 1;
min-height: 0;
overflow: auto;
}
</style> </style>

14
domain-web/src/layouts/MainLayout.vue
View File

@@ -63,6 +63,8 @@ const menuItems = [
{ path: "/runtime", label: "运行中心" }, { path: "/runtime", label: "运行中心" },
{ path: "/settings", label: "系统设置" }, { path: "/settings", label: "系统设置" },
{ path: "/imports", label: "域名导入" }, { path: "/imports", label: "域名导入" },
{ path: "/sensitive-words", label: "敏感词配置" },
{ path: "/juming", label: "聚名采集" },
{ path: "/detect", label: "检测控制" }, { path: "/detect", label: "检测控制" },
{ path: "/domains", label: "域名筛选" }, { path: "/domains", label: "域名筛选" },
{ path: "/exports", label: "导出中心" }, { path: "/exports", label: "导出中心" },
@@ -119,7 +121,9 @@ onBeforeUnmount(() => {
.layout-shell { .layout-shell {
display: grid; display: grid;
grid-template-columns: 240px 1fr; grid-template-columns: 240px 1fr;
min-height: 100vh; height: 100vh;
height: 100dvh;
overflow: hidden;
} }
.sidebar { .sidebar {
@@ -129,6 +133,8 @@ onBeforeUnmount(() => {
display: flex; display: flex;
flex-direction: column; flex-direction: column;
gap: 28px; gap: 28px;
min-height: 0;
overflow: auto;
} }
.brand { .brand {
@@ -169,6 +175,8 @@ onBeforeUnmount(() => {
display: flex; display: flex;
flex-direction: column; flex-direction: column;
min-width: 0; min-width: 0;
min-height: 0;
overflow: hidden;
} }
.topbar { .topbar {
@@ -235,12 +243,16 @@ onBeforeUnmount(() => {
} }
.content { .content {
flex: 1;
min-height: 0;
padding: 0 28px 28px; padding: 0 28px 28px;
overflow: hidden;
} }
@media (max-width: 960px) { @media (max-width: 960px) {
.layout-shell { .layout-shell {
grid-template-columns: 1fr; grid-template-columns: 1fr;
grid-template-rows: auto 1fr;
} }
.sidebar { .sidebar {

2
domain-web/src/router/index.ts
View File

@@ -17,6 +17,8 @@ const routes: RouteRecordRaw[] = [
{ path: "runtime", name: "runtime", component: () => import("@/views/runtime/RuntimeView.vue"), meta: { title: "运行中心" } }, { path: "runtime", name: "runtime", component: () => import("@/views/runtime/RuntimeView.vue"), meta: { title: "运行中心" } },
{ path: "settings", name: "settings", component: () => import("@/views/settings/SettingsView.vue"), meta: { title: "系统设置" } }, { path: "settings", name: "settings", component: () => import("@/views/settings/SettingsView.vue"), meta: { title: "系统设置" } },
{ path: "imports", name: "imports", component: () => import("@/views/imports/ImportsView.vue"), meta: { title: "域名导入" } }, { path: "imports", name: "imports", component: () => import("@/views/imports/ImportsView.vue"), meta: { title: "域名导入" } },
{ path: "sensitive-words", name: "sensitive-words", component: () => import("@/views/sensitive-words/SensitiveWordsView.vue"), meta: { title: "敏感词配置" } },
{ path: "juming", name: "juming", component: () => import("@/views/juming/JumingView.vue"), meta: { title: "聚名采集" } },
{ path: "detect", name: "detect", component: () => import("@/views/detect/DetectView.vue"), meta: { title: "检测控制" } }, { path: "detect", name: "detect", component: () => import("@/views/detect/DetectView.vue"), meta: { title: "检测控制" } },
{ path: "domains", name: "domains", component: () => import("@/views/domains/DomainsView.vue"), meta: { title: "域名筛选" } }, { path: "domains", name: "domains", component: () => import("@/views/domains/DomainsView.vue"), meta: { title: "域名筛选" } },
{ path: "exports", name: "exports", component: () => import("@/views/exports/ExportsView.vue"), meta: { title: "导出中心" } }, { path: "exports", name: "exports", component: () => import("@/views/exports/ExportsView.vue"), meta: { title: "导出中心" } },

2
domain-web/src/styles/main.scss
View File

@@ -13,6 +13,7 @@ html,
body, body,
#app { #app {
margin: 0; margin: 0;
height: 100%;
min-height: 100vh; min-height: 100vh;
} }
@@ -21,6 +22,7 @@ body {
radial-gradient(circle at top left, rgba(37, 99, 235, 0.08), transparent 28%), radial-gradient(circle at top left, rgba(37, 99, 235, 0.08), transparent 28%),
radial-gradient(circle at top right, rgba(14, 165, 233, 0.08), transparent 22%), radial-gradient(circle at top right, rgba(14, 165, 233, 0.08), transparent 22%),
#f4f6fb; #f4f6fb;
overflow: hidden;
} }
a { a {

633
domain-web/src/views/detect/DetectView.vue
View File

@@ -1,15 +1,17 @@
<template> <template>
<PageCard title="检测控制" description="当前已接入真实状态读取和本地 Worker 启停，后续 Linux 版会沿用同一页接 systemd。"> <PageCard title="检测控制" description="已升级为任务中心：支持启停控制、阶段查看、日志常驻和会话回看，并沉淀线程/代理/进度快照。">
<div class="actions"> <div class="actions">
<el-button type="primary" :loading="actionLoading === 'start'" @click="invoke('start')">启动检测</el-button> <el-button type="primary" :loading="actionLoading === 'start'" @click="invoke('start')">启动检测</el-button>
<el-button :loading="actionLoading === 'stop'" @click="invoke('stop')">停止检测</el-button> <el-button :loading="actionLoading === 'stop'" @click="invoke('stop')">停止检测</el-button>
<el-button plain :loading="loading" @click="loadStatus">刷新状态</el-button> <el-button plain :loading="loading" @click="loadStatus">刷新状态</el-button>
<el-button plain @click="goRuntime">前往运行中心</el-button>
<el-switch v-model="autoRefresh" inline-prompt active-text="自动刷新" inactive-text="手动" /> <el-switch v-model="autoRefresh" inline-prompt active-text="自动刷新" inactive-text="手动" />
<span class="updated-at">最近同步：{{ lastUpdatedAt || "暂无" }}</span>
</div> </div>
<el-alert <el-alert
v-if="lastAction.message" v-if="lastAction.message"
:title="`${lastAction.label}：${lastAction.message}`" :title="`${lastAction.label}：${lastAction.message}${lastAction.at ? `（${lastAction.at}）` : ''}`"
:type="lastAction.type" :type="lastAction.type"
:closable="false" :closable="false"
show-icon show-icon
@@ -23,14 +25,34 @@
<el-descriptions-item label="API 服务">{{ status.api_service_name || "-" }}</el-descriptions-item> <el-descriptions-item label="API 服务">{{ status.api_service_name || "-" }}</el-descriptions-item>
<el-descriptions-item label="检测进程数">{{ status.worker_process_count }}</el-descriptions-item> <el-descriptions-item label="检测进程数">{{ status.worker_process_count }}</el-descriptions-item>
<el-descriptions-item label="线程数">{{ status.thread_count }}</el-descriptions-item> <el-descriptions-item label="线程数">{{ status.thread_count }}</el-descriptions-item>
<el-descriptions-item label="线程来源">{{ threadCountSummary }}</el-descriptions-item>
<el-descriptions-item label="代理启用">{{ status.proxy_enable ? "是" : "否" }}</el-descriptions-item> <el-descriptions-item label="代理启用">{{ status.proxy_enable ? "是" : "否" }}</el-descriptions-item>
<el-descriptions-item label="允许直连">{{ status.allow_direct ? "是" : "否" }}</el-descriptions-item> <el-descriptions-item label="允许直连">{{ status.allow_direct ? "是" : "否" }}</el-descriptions-item>
<el-descriptions-item label="代理池数量">{{ status.proxy_pool_count }}</el-descriptions-item> <el-descriptions-item label="代理池数量">{{ status.proxy_pool_count }}</el-descriptions-item>
<el-descriptions-item label="可用代理数">{{ status.available_proxy_count }}</el-descriptions-item> <el-descriptions-item label="可用代理数">{{ status.available_proxy_count }}</el-descriptions-item>
<el-descriptions-item label="代理状态">{{ status.proxy_runtime_label || "-" }}</el-descriptions-item>
<el-descriptions-item label="代理诊断">{{ proxyDiagnosisText }}</el-descriptions-item>
<el-descriptions-item label="最近代理刷新">{{ status.proxy_last_refresh_time || "暂无" }}</el-descriptions-item>
<el-descriptions-item label="最近 Worker 日志时间">{{ status.last_worker_log_time || "暂无" }}</el-descriptions-item> <el-descriptions-item label="最近 Worker 日志时间">{{ status.last_worker_log_time || "暂无" }}</el-descriptions-item>
<el-descriptions-item label="最近启动时间">{{ status.worker_latest_start_time || "暂无" }}</el-descriptions-item> <el-descriptions-item label="最近启动时间">{{ status.worker_latest_start_time || "暂无" }}</el-descriptions-item>
</el-descriptions> </el-descriptions>
<el-alert
v-if="status.proxy_runtime_detail"
:type="proxyAlertType"
:closable="false"
show-icon
:title="status.proxy_runtime_detail"
style="margin-top: 16px"
/>
<el-alert
v-if="status.proxy_supplier_empty"
type="warning"
:closable="false"
show-icon
title="代理供应侧最近返回 200，但原始代理数为 0；当前判断是供应池为空，不是程序拉取失败。"
style="margin-top: 12px"
/>
<el-alert <el-alert
v-if="status.recent_warning" v-if="status.recent_warning"
type="warning" type="warning"
@@ -39,6 +61,15 @@
:title="status.recent_warning" :title="status.recent_warning"
style="margin-top: 16px" style="margin-top: 16px"
/> />
<el-alert
v-for="alert in status.dependency_alerts || []"
:key="`${alert.kind}-${alert.detail}`"
:type="alert.level === 'error' ? 'error' : 'warning'"
:closable="false"
show-icon
:title="`${alert.title}：${alert.detail}`"
style="margin-top: 12px"
/>
<el-alert <el-alert
v-if="status.worker_runtime_message" v-if="status.worker_runtime_message"
type="info" type="info"
@@ -47,23 +78,216 @@
:title="status.worker_runtime_message" :title="status.worker_runtime_message"
style="margin-top: 12px" style="margin-top: 12px"
/> />
<el-alert
v-if="status.recent_event"
type="success"
:closable="false"
show-icon
:title="status.recent_event"
style="margin-top: 12px"
/>
<el-alert
:closable="false"
show-icon
type="info"
:title="`当前检测状态：${currentPhaseLabel}；Worker ${status.worker_online ? '在线' : '离线'}。`"
style="margin-top: 12px"
/>
<el-row :gutter="12" style="margin-top: 16px"> <el-row :gutter="12" style="margin-top: 16px; margin-bottom: 16px">
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="待检测" :value="status.progress.pending || 0" /></el-col> <el-col :xs="12" :sm="8" :md="4"><el-statistic title="待检测" :value="status.progress.pending || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测中" :value="status.progress.running || 0" /></el-col> <el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测中" :value="status.progress.running || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测完成" :value="status.progress.completed || 0" /></el-col> <el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测通过" :value="status.progress.completed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="黑名单" :value="status.progress.blacklisted || 0" /></el-col> <el-col :xs="12" :sm="8" :md="4"><el-statistic title="黑名单" :value="status.progress.blacklisted || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测失败" :value="status.progress.failed || 0" /></el-col> <el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测失败" :value="status.progress.failed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="检测会话" :value="runs.length" /></el-col>
</el-row> </el-row>
<div class="summary-grid">
<div class="summary-card">
<div class="summary-title">当前任务</div>
<div class="summary-value task-code">{{ activeJob?.job_code || "-" }}</div>
<div class="summary-note">{{ activeJobSummaryText }}</div>
</div>
<div class="summary-card">
<div class="summary-title">检测进度</div>
<div class="summary-value">{{ status.progress_percent || 0 }}%</div>
<el-progress :percentage="Number(status.progress_percent || 0)" :stroke-width="10" />
</div>
<div class="summary-card">
<div class="summary-title">当前阶段</div>
<div class="summary-value phase-value">{{ currentPhaseLabel }}</div>
<div class="summary-note">{{ currentPhaseDetail }}</div>
</div>
<div class="summary-card">
<div class="summary-title">线程快照</div>
<div class="summary-value">{{ status.active_thread_count || 0 }} / {{ status.max_thread_count || status.thread_count || 0 }}</div>
<div class="summary-note">当前活跃线程 / 配置线程上限</div>
</div>
<div class="summary-card">
<div class="summary-title">代理快照</div>
<div class="summary-value">{{ status.available_proxy_count || 0 }} / {{ status.proxy_pool_count || 0 }}</div>
<div class="summary-note">{{ proxyDiagnosisText }}</div>
</div>
</div>
<el-row v-if="activeJob" :gutter="12" style="margin-bottom: 16px">
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务待领" :value="activeJob.items_pending || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务已领" :value="activeJob.items_claimed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务执行中" :value="activeJob.items_running || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务完成" :value="activeJob.items_completed || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务黑名单" :value="activeJob.items_blacklisted || 0" /></el-col>
<el-col :xs="12" :sm="8" :md="4"><el-statistic title="任务失败" :value="activeJob.items_failed || 0" /></el-col>
</el-row>
<div v-if="activeJob" class="summary-grid summary-grid-tight">
<div class="summary-card">
<div class="summary-title">任务进度</div>
<div class="summary-value">{{ activeJob.progress_percent || 0 }}%</div>
<el-progress :percentage="Number(activeJob.progress_percent || 0)" :stroke-width="10" />
</div>
<div class="summary-card">
<div class="summary-title">最近任务事件</div>
<div class="summary-value event-value">{{ activeJob.latest_event?.event_type || "-" }}</div>
<div class="summary-note">{{ activeJob.latest_event?.message || "当前还没有任务事件" }}</div>
</div>
</div>
<div class="history-panel">
<div class="history-header">
<h3>队列健康度</h3>
<span class="history-note">聚合近 {{ queueSummary.window_minutes || 15 }} 分钟吞吐、租约和积压，便于后续多机扩容判断</span>
</div>
<div class="summary-grid summary-grid-tight">
<div class="summary-card">
<div class="summary-title">近窗吞吐</div>
<div class="summary-value">{{ queueSummary.throughput?.processed_recent || 0 }}</div>
<div class="summary-note">近 {{ queueSummary.window_minutes || 15 }} 分钟处理 {{ queueSummary.throughput?.processed_per_minute || 0 }} 项/分钟</div>
</div>
<div class="summary-card">
<div class="summary-title">租约风险</div>
<div class="summary-value">{{ queueSummary.queue?.overdue_leases || 0 }}</div>
<div class="summary-note">过期 {{ queueSummary.queue?.overdue_leases || 0 }} / 5 分钟内到期 {{ queueSummary.queue?.expiring_soon_leases || 0 }}</div>
</div>
<div class="summary-card">
<div class="summary-title">最老待领</div>
<div class="summary-value">{{ queueSummary.queue?.oldest_pending_age_minutes || 0 }} 分钟</div>
<div class="summary-note">{{ queueSummary.queue?.oldest_pending_at || "当前没有待领取任务项" }}</div>
</div>
<div class="summary-card">
<div class="summary-title">任务收敛</div>
<div class="summary-value">{{ queueSummary.queue?.terminal_percent || 0 }}%</div>
<div class="summary-note">已收敛 {{ queueSummary.queue?.terminal || 0 }} / {{ queueSummary.queue?.items_total || 0 }}</div>
</div>
</div>
<el-alert
:closable="false"
show-icon
:type="queueAlertType"
style="margin-bottom: 12px"
:title="queueAlertText"
/>
<el-table :data="queueSummary.nodes || []" border style="margin-bottom: 16px" empty-text="当前没有可展示的节点吞吐摘要">
<el-table-column prop="node_code" label="节点" min-width="180" />
<el-table-column prop="items_running" label="执行中" min-width="90" />
<el-table-column prop="items_claimed" label="已领未跑" min-width="100" />
<el-table-column prop="processed_recent" label="近窗处理" min-width="100" />
<el-table-column prop="processed_per_minute" label="项/分钟" min-width="100" />
<el-table-column prop="completed_recent" label="完成" min-width="90" />
<el-table-column prop="blacklisted_recent" label="黑名单" min-width="90" />
<el-table-column prop="failed_recent" label="失败" min-width="90" />
</el-table>
</div>
<el-table v-if="activeJob?.node_stats?.length" :data="activeJob.node_stats" border style="margin-bottom: 16px">
<el-table-column prop="node_code" label="节点" min-width="180" />
<el-table-column prop="items_total" label="任务总数" min-width="100" />
<el-table-column prop="items_pending" label="待领" min-width="90" />
<el-table-column prop="items_claimed" label="已领" min-width="90" />
<el-table-column prop="items_running" label="执行中" min-width="90" />
<el-table-column prop="items_completed" label="完成" min-width="90" />
<el-table-column prop="items_failed" label="失败" min-width="90" />
</el-table>
<el-table v-if="(activeJob?.current_cycle_events || activeJob?.recent_events)?.length" :data="activeJob.current_cycle_events || activeJob.recent_events" border style="margin-bottom: 16px">
<el-table-column prop="created_at" label="事件时间" min-width="160" />
<el-table-column prop="node_code" label="节点" min-width="160" />
<el-table-column prop="event_type" label="事件类型" min-width="160" />
<el-table-column prop="message" label="事件说明" min-width="320" show-overflow-tooltip />
</el-table>
<el-table :data="runs" border highlight-current-row @current-change="handleCurrentRunChange">
<el-table-column label="状态" min-width="100">
<template #default="{ row }">
<el-tag :type="statusTagType(row.status)" effect="dark">{{ statusText(row.status) }}</el-tag>
</template>
</el-table-column>
<el-table-column label="阶段" min-width="110">
<template #default="{ row }">
<el-tag :type="phaseTagType(row.phase_label, row.status)" plain>{{ row.phase_label || "-" }}</el-tag>
</template>
</el-table-column>
<el-table-column prop="message" label="会话信息" min-width="280" show-overflow-tooltip />
<el-table-column prop="started_at" label="开始时间" min-width="160" />
<el-table-column prop="completed_at" label="结束时间" min-width="160" />
<el-table-column label="线程/代理" min-width="160">
<template #default="{ row }">
<span>线程 {{ row.settings_summary?.thread_count || 0 }} / 代理 {{ row.settings_summary?.proxy_pool_count || 0 }}</span>
</template>
</el-table-column>
<el-table-column label="进度快照" min-width="220">
<template #default="{ row }">
<span>
待 {{ row.progress?.pending || 0 }}
/ 中 {{ row.progress?.running || 0 }}
/ 完 {{ row.progress?.completed || 0 }}
/ 黑 {{ row.progress?.blacklisted || 0 }}
/ 失败 {{ row.progress?.failed || 0 }}
</span>
</template>
</el-table-column>
</el-table>
<div class="task-detail">
<div class="task-detail-header">
<div>
<div class="task-detail-title">任务日志控制台</div>
<div class="task-detail-subtitle">
{{ selectedRun ? `当前查看会话：${selectedRun.run_id}` : "当前暂无检测会话，日志窗口会显示最近 Worker 输出" }}
</div>
</div>
<div class="task-detail-meta">
<el-switch v-model="logAutoFollow" inline-prompt active-text="跟随日志" inactive-text="暂停跟随" size="small" />
<el-button text size="small" @click="scrollLogToBottom(true)">回到底部</el-button>
<span>Worker：{{ status.worker_online ? "在线" : "离线" }}</span>
<span v-if="selectedRun?.phase_label">阶段：{{ selectedRun.phase_label }}</span>
<span>运行中：{{ status.progress.running || 0 }}</span>
<span>线程：{{ threadCountSummary }}</span>
</div>
</div>
<div v-if="selectedRun?.phase_detail" class="phase-detail">{{ selectedRun.phase_detail }}</div>
<pre ref="logConsoleRef" class="log-console" @scroll="handleLogScroll">{{ selectedLogText }}</pre>
</div>
<el-table v-if="phaseHistory.length" :data="phaseHistory" border style="margin-top: 16px">
<el-table-column prop="at" label="切换时间" min-width="180" />
<el-table-column prop="label" label="阶段" min-width="120" />
<el-table-column prop="detail" label="阶段说明" min-width="360" show-overflow-tooltip />
</el-table>
</PageCard> </PageCard>
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { onBeforeUnmount, onMounted, ref, watch } from "vue"; import { computed, nextTick, onBeforeUnmount, onMounted, ref, watch } from "vue";
import { useRouter } from "vue-router";
import { ElMessage } from "element-plus"; import { ElMessage } from "element-plus";
import PageCard from "@/components/PageCard.vue"; import PageCard from "@/components/PageCard.vue";
import { detectApi } from "@/api/modules"; import { detectApi } from "@/api/modules";
const DETECT_LAST_ACTION_KEY = "domaincheck:detect:last-action";
const router = useRouter();
const status = ref({ const status = ref({
worker_online: false, worker_online: false,
worker_mode: "windows-local", worker_mode: "windows-local",
@@ -73,12 +297,32 @@ const status = ref({
worker_latest_start_time: "", worker_latest_start_time: "",
worker_runtime_message: "", worker_runtime_message: "",
thread_count: 0, thread_count: 0,
thread_count_default: 0,
thread_count_source: "default",
thread_count_override: null as number | null,
thread_count_node_code: "",
active_thread_count: 0,
max_thread_count: 0,
proxy_enable: false, proxy_enable: false,
allow_direct: false, allow_direct: false,
proxy_pool_count: 0, proxy_pool_count: 0,
available_proxy_count: 0, available_proxy_count: 0,
last_worker_log_time: "", last_worker_log_time: "",
recent_event: "",
recent_warning: "", recent_warning: "",
proxy_runtime_state: "",
proxy_runtime_label: "",
proxy_runtime_detail: "",
proxy_runtime_reason: "",
proxy_supplier_empty: false,
proxy_direct_fallback_active: false,
proxy_last_refresh_status: "",
proxy_last_refresh_time: "",
dependency_alerts: [] as any[],
log_lines: [] as string[],
runs: [] as any[],
progress_percent: 0,
active_job: null as any,
progress: { progress: {
pending: 0, pending: 0,
running: 0, running: 0,
@@ -88,15 +332,209 @@ const status = ref({
} }
}); });
const threadCountSummary = computed(() => {
const effective = Number(status.value.thread_count || 0);
const defaultValue = Number(status.value.thread_count_default || 0);
const nodeCode = String(status.value.thread_count_node_code || "").trim();
if (status.value.thread_count_source === "node_override" && status.value.thread_count_override) {
return `${effective}（节点 ${nodeCode || "-"} 单独覆盖，默认 ${defaultValue}）`;
}
return `${effective}（默认）`;
});
const queueSummary = ref({
window_minutes: 15,
has_active_job: false,
queue: {
items_total: 0,
terminal: 0,
terminal_percent: 0,
overdue_leases: 0,
expiring_soon_leases: 0,
oldest_pending_at: "",
oldest_pending_age_minutes: 0
},
throughput: {
processed_recent: 0,
processed_per_minute: 0
},
nodes: [] as any[]
});
const autoRefresh = ref(true); const autoRefresh = ref(true);
const logAutoFollow = ref(true);
const loading = ref(false); const loading = ref(false);
const actionLoading = ref<"" | "start" | "stop">(""); const actionLoading = ref<"" | "start" | "stop">("");
const lastUpdatedAt = ref("");
const selectedRunId = ref("");
const lastAction = ref({ const lastAction = ref({
label: "", label: "",
message: "", message: "",
at: "",
type: "success" as "success" | "warning" | "info" | "error" type: "success" as "success" | "warning" | "info" | "error"
}); });
let timer: number | null = null; let timer: number | null = null;
const logConsoleRef = ref<HTMLElement | null>(null);
const persistLastAction = () => {
if (!lastAction.value.message) {
window.localStorage.removeItem(DETECT_LAST_ACTION_KEY);
return;
}
window.localStorage.setItem(DETECT_LAST_ACTION_KEY, JSON.stringify(lastAction.value));
};
const restoreLastAction = () => {
try {
const raw = window.localStorage.getItem(DETECT_LAST_ACTION_KEY);
if (!raw) return;
const parsed = JSON.parse(raw);
lastAction.value = {
label: parsed.label || "",
message: parsed.message || "",
at: parsed.at || "",
type: parsed.type || "info"
};
} catch {
window.localStorage.removeItem(DETECT_LAST_ACTION_KEY);
}
};
const runs = computed(() => status.value.runs || []);
const activeJob = computed(() => status.value.active_job || null);
const activeJobSummaryText = computed(() => {
if (!activeJob.value) {
return "当前暂无活跃检测任务";
}
const job = activeJob.value;
return `状态：${jobStatusText(job.status)} / 总数 ${job.items_total || 0} / 待领 ${job.items_pending || 0} / 执行中 ${job.items_running || 0}`;
});
const selectedRun = computed(() => runs.value.find((item) => item.run_id === selectedRunId.value) || runs.value[0] || null);
const currentPhaseLabel = computed(() => selectedRun.value?.phase_label || (status.value.worker_online ? "运行中" : "未启动"));
const currentPhaseDetail = computed(() => {
if (selectedRun.value?.phase_detail) {
return selectedRun.value.phase_detail;
}
if (status.value.recent_event) {
return status.value.recent_event;
}
return status.value.worker_runtime_message || "当前暂无阶段明细";
});
const phaseHistory = computed(() => {
const items = selectedRun.value?.phase_history;
if (!Array.isArray(items)) return [];
return [...items].reverse();
});
const selectedLogText = computed(() => {
const runLogs = selectedRun.value?.logs;
if (Array.isArray(runLogs) && runLogs.length) {
return runLogs.join("\n");
}
const liveLogs = status.value.log_lines || [];
return liveLogs.length ? liveLogs.join("\n") : "暂无检测日志";
});
const proxyAlertType = computed<"success" | "warning" | "info" | "error">(() => {
if (status.value.proxy_runtime_state === "healthy") return "success";
if (status.value.proxy_runtime_state === "degraded_direct") return "warning";
if (status.value.proxy_runtime_state === "blocked_no_proxy") return "error";
return "info";
});
const proxyDiagnosisText = computed(() => {
if (status.value.proxy_supplier_empty) {
return "供应池为空，系统已按配置继续运行";
}
return status.value.proxy_runtime_label || "当前可用代理 / 配置代理池链接数";
});
const queueAlertType = computed<"success" | "warning" | "info" | "error">(() => {
if (!queueSummary.value.has_active_job) return "info";
if ((queueSummary.value.queue?.overdue_leases || 0) > 0) return "error";
if ((queueSummary.value.queue?.expiring_soon_leases || 0) > 0 || (queueSummary.value.queue?.oldest_pending_age_minutes || 0) >= 30) return "warning";
return "success";
});
const queueAlertText = computed(() => {
if (!queueSummary.value.has_active_job) {
return "当前没有活跃检测任务，队列健康度会在下一轮任务创建后开始统计。";
}
const queue = queueSummary.value.queue || {};
const throughput = queueSummary.value.throughput || {};
const parts = [
`当前队列待领 ${queue.pending || 0} / 已领 ${queue.claimed || 0} / 执行中 ${queue.running || 0}`,
`近 ${queueSummary.value.window_minutes || 15} 分钟处理 ${throughput.processed_recent || 0} 项，约 ${throughput.processed_per_minute || 0} 项/分钟`
];
if ((queue.overdue_leases || 0) > 0) {
parts.push(`存在 ${queue.overdue_leases} 个过期租约，建议立即检查 Worker 心跳和数据库续租`);
} else if ((queue.expiring_soon_leases || 0) > 0) {
parts.push(`有 ${queue.expiring_soon_leases} 个任务租约将在 5 分钟内到期`);
}
if ((queue.oldest_pending_age_minutes || 0) > 0) {
parts.push(`最老待领任务已等待 ${queue.oldest_pending_age_minutes} 分钟`);
}
return parts.join("；");
});
const scrollLogToBottom = async (force = false) => {
if (!force && !logAutoFollow.value) {
return;
}
await nextTick();
const element = logConsoleRef.value;
if (!element) {
return;
}
element.scrollTop = element.scrollHeight;
if (force) {
logAutoFollow.value = true;
}
};
const handleLogScroll = () => {
const element = logConsoleRef.value;
if (!element) {
return;
}
const distanceToBottom = element.scrollHeight - element.scrollTop - element.clientHeight;
logAutoFollow.value = distanceToBottom < 24;
};
const statusText = (value: string) => {
if (value === "starting") return "启动中";
if (value === "running") return "运行中";
if (value === "stopping") return "停止中";
if (value === "completed") return "已完成";
if (value === "stopped") return "已停止";
if (value === "failed") return "失败";
if (value === "partial_failed") return "部分失败";
return value || "-";
};
const jobStatusText = (value: string) => {
if (value === "pending") return "待执行";
if (value === "running") return "执行中";
if (value === "completed") return "已完成";
if (value === "partial_failed") return "部分失败";
if (value === "failed") return "失败";
return value || "-";
};
const statusTagType = (value: string) => {
if (value === "running") return "success";
if (value === "starting" || value === "stopping") return "warning";
if (value === "completed") return "success";
if (value === "partial_failed") return "warning";
if (value === "failed") return "danger";
return "info";
};
const phaseTagType = (phaseLabel: string, statusValue: string) => {
if (statusValue === "failed") return "danger";
if (statusValue === "partial_failed") return "warning";
if (statusValue === "starting" || statusValue === "stopping") return "warning";
if (["检测中", "完成归档", "批次完成", "已完成"].includes(phaseLabel)) return "success";
if (["刷新代理池", "取任务中", "建线程中", "准备检测", "外部站点异常", "网络波动", "部分失败"].includes(phaseLabel)) return "warning";
return "info";
};
const clearRefreshTimer = () => { const clearRefreshTimer = () => {
if (timer) { if (timer) {
@@ -110,15 +548,25 @@ const ensureRefreshTimer = () => {
if (autoRefresh.value) { if (autoRefresh.value) {
timer = window.setInterval(() => { timer = window.setInterval(() => {
loadStatus(false); loadStatus(false);
}, 10000); }, 5000);
} }
}; };
const loadStatus = async (showError = true) => { const loadStatus = async (showError = true) => {
loading.value = true; loading.value = true;
try { try {
const response = await detectApi.status(); const [statusResponse, queueResponse] = await Promise.all([
status.value = response.data; detectApi.status(),
detectApi.queueSummary({ window_minutes: 15 })
]);
status.value = statusResponse.data;
queueSummary.value = queueResponse.data;
if (!selectedRunId.value && runs.value.length) {
selectedRunId.value = runs.value[0].run_id;
} else if (selectedRunId.value && !runs.value.some((item) => item.run_id === selectedRunId.value) && runs.value.length) {
selectedRunId.value = runs.value[0].run_id;
}
lastUpdatedAt.value = new Date().toLocaleString("zh-CN", { hour12: false });
} catch { } catch {
if (showError) { if (showError) {
ElMessage.error("读取检测状态失败"); ElMessage.error("读取检测状态失败");
@@ -136,34 +584,58 @@ const invoke = async (action: "start" | "stop") => {
lastAction.value = { lastAction.value = {
label: action === "start" ? "检测启动" : "检测停止", label: action === "start" ? "检测启动" : "检测停止",
message: response.message, message: response.message,
at: new Date().toLocaleString("zh-CN", { hour12: false }),
type: "success" type: "success"
}; };
persistLastAction();
const pollDelaySeconds = Number(response.data?.poll_after_seconds || 2); const pollDelaySeconds = Number(response.data?.poll_after_seconds || 2);
if (response.data?.refresh_status) { window.setTimeout(() => {
window.setTimeout(() => { loadStatus(false);
loadStatus(false); }, pollDelaySeconds * 1000);
}, pollDelaySeconds * 1000);
return;
}
await loadStatus(false);
} catch (error: any) { } catch (error: any) {
const message = error?.message || `${action === "start" ? "启动" : "停止"}操作失败`; const message = error?.message || `${action === "start" ? "启动" : "停止"}操作失败`;
lastAction.value = { lastAction.value = {
label: action === "start" ? "检测启动" : "检测停止", label: action === "start" ? "检测启动" : "检测停止",
message, message,
at: new Date().toLocaleString("zh-CN", { hour12: false }),
type: "error" type: "error"
}; };
persistLastAction();
ElMessage.error(message); ElMessage.error(message);
} finally { } finally {
actionLoading.value = ""; actionLoading.value = "";
} }
}; };
const handleCurrentRunChange = (row: any | null) => {
selectedRunId.value = row?.run_id || "";
};
const goRuntime = () => {
router.push("/runtime");
};
watch(autoRefresh, ensureRefreshTimer); watch(autoRefresh, ensureRefreshTimer);
watch(selectedLogText, async (current, previous) => {
if (current === previous) {
return;
}
await scrollLogToBottom();
});
watch(selectedRunId, async () => {
await scrollLogToBottom(true);
});
watch(logAutoFollow, async (enabled) => {
if (enabled) {
await scrollLogToBottom(true);
}
});
onMounted(async () => { onMounted(async () => {
restoreLastAction();
await loadStatus(); await loadStatus();
ensureRefreshTimer(); ensureRefreshTimer();
await scrollLogToBottom(true);
}); });
onBeforeUnmount(clearRefreshTimer); onBeforeUnmount(clearRefreshTimer);
@@ -175,9 +647,140 @@ onBeforeUnmount(clearRefreshTimer);
gap: 12px; gap: 12px;
margin-bottom: 16px; margin-bottom: 16px;
flex-wrap: wrap; flex-wrap: wrap;
align-items: center;
} }
.action-alert { .action-alert {
margin-bottom: 16px; margin-bottom: 16px;
} }
.updated-at {
color: #64748b;
font-size: 13px;
}
.summary-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
gap: 12px;
margin-bottom: 16px;
}
.summary-card {
border: 1px solid #dbe4ee;
border-radius: 14px;
padding: 14px 16px;
background: #f8fbff;
}
.summary-title {
color: #64748b;
font-size: 13px;
}
.summary-value {
margin: 8px 0 10px;
color: #0f172a;
font-size: 24px;
font-weight: 700;
}
.phase-value {
font-size: 20px;
}
.summary-note {
margin-top: 8px;
color: #64748b;
font-size: 12px;
}
.summary-grid-tight {
margin-top: -4px;
}
.history-panel {
margin: 16px 0;
}
.history-header {
display: flex;
justify-content: space-between;
gap: 12px;
align-items: center;
margin-bottom: 12px;
}
.history-header h3 {
margin: 0;
}
.history-note {
color: #64748b;
font-size: 12px;
}
.task-code,
.event-value {
font-size: 18px;
word-break: break-all;
}
.task-detail {
margin: 16px 0;
border-radius: 10px;
overflow: hidden;
border: 1px solid #1f2937;
background: #020617;
}
.task-detail-header {
display: flex;
justify-content: space-between;
gap: 12px;
align-items: center;
padding: 12px 16px;
background: #111827;
color: #e5e7eb;
}
.task-detail-title {
font-size: 15px;
font-weight: 600;
}
.task-detail-subtitle,
.task-detail-meta {
color: #94a3b8;
font-size: 12px;
}
.task-detail-meta {
display: flex;
gap: 12px;
flex-wrap: wrap;
align-items: center;
}
.log-console {
margin: 0;
min-height: 320px;
max-height: 520px;
overflow: auto;
padding: 16px;
color: #e0e7ff;
background: #020617;
font-size: 12px;
line-height: 1.7;
font-family: Consolas, Monaco, monospace;
white-space: pre-wrap;
word-break: break-word;
}
.phase-detail {
padding: 10px 16px 0;
color: #93c5fd;
font-size: 12px;
background: #020617;
}
</style> </style>

717
domain-web/src/views/domains/DomainsView.vue
View File

@@ -1,86 +1,214 @@
<template> <template>
<PageCard title="域名筛选" description="已接入真实分页、联表筛选、检测时间展示和批量更新。"> <PageCard title="域名筛选" description="已接入真实分页、联表筛选、检测时间展示和批量更新。">
<el-form label-width="88px" class="toolbar-grid"> <template #header-extra>
<el-form-item label="域名关键字"> <div class="header-actions">
<el-input v-model="filters.domain_keyword" placeholder="支持模糊查询" clearable /> <el-button text @click="showAdvancedFilters = !showAdvancedFilters">
</el-form-item> {{ showAdvancedFilters ? "收起筛选" : "展开筛选" }}
<el-form-item label="注册状态"> </el-button>
<el-select v-model="filters.register_status" clearable>
<el-option v-for="item in registerOptions" :key="item.value" :label="item.label" :value="item.value" />
</el-select>
</el-form-item>
<el-form-item label="检测状态">
<el-select v-model="filters.detect_status" clearable>
<el-option v-for="item in detectOptions" :key="item.value" :label="item.label" :value="item.value" />
</el-select>
</el-form-item>
<el-form-item label="使用状态">
<el-select v-model="filters.use_status" clearable>
<el-option v-for="item in useOptions" :key="item.value" :label="item.label" :value="item.value" />
</el-select>
</el-form-item>
<el-form-item label="复核状态">
<el-select v-model="filters.review_status" clearable>
<el-option v-for="item in reviewOptions" :key="item.value" :label="item.label" :value="item.value" />
</el-select>
</el-form-item>
<el-form-item label="备案状态">
<el-select v-model="filters.has_beian" clearable>
<el-option label="有备案" :value="2" />
<el-option label="无备案" :value="3" />
<el-option label="未检测" :value="1" />
</el-select>
</el-form-item>
<el-form-item label="备案年份">
<el-input v-model="filters.beian_year" placeholder="如 2023" clearable />
</el-form-item>
<el-form-item label="快照年份">
<el-input v-model="filters.snapshot_year" placeholder="如 2024" clearable />
</el-form-item>
<el-form-item label="首页网址">
<el-input v-model="filters.website_url" placeholder="模糊匹配网址" clearable />
</el-form-item>
<el-form-item label="友链 > 10">
<el-switch v-model="filters.backlink_gt_10" />
</el-form-item>
<el-form-item class="actions">
<el-button type="primary" @click="search">查询</el-button> <el-button type="primary" @click="search">查询</el-button>
<el-button @click="reset">重置</el-button> <el-button @click="reset">重置</el-button>
<el-button plain @click="exportDialogVisible = true">导出</el-button>
<el-button type="warning" :disabled="selectedIds.length === 0" @click="batchDialogVisible = true"> <el-button type="warning" :disabled="selectedIds.length === 0" @click="batchDialogVisible = true">
批量更新选中域名 批量更新
</el-button> </el-button>
</el-form-item> </div>
</el-form> </template>
<el-table :data="rows" border @selection-change="handleSelectionChange"> <div class="domains-layout">
<el-table-column type="selection" width="48" fixed="left" /> <el-form label-width="80px" size="small" class="toolbar-grid">
<el-table-column prop="domain" label="域名" min-width="180" fixed="left" /> <el-form-item label="域名关键字">
<el-table-column prop="register_status" label="注册状态" min-width="100" /> <el-input v-model="filters.domain_keyword" placeholder="支持模糊查询" clearable />
<el-table-column prop="use_status" label="使用状态" min-width="100" /> </el-form-item>
<el-table-column prop="detect_status" label="检测状态" min-width="100" /> <el-form-item label="注册状态">
<el-table-column prop="review_status" label="复核状态" min-width="100" /> <el-select v-model="filters.register_status">
<el-table-column prop="has_beian" label="备案" min-width="80" /> <el-option v-for="item in registerOptions" :key="String(item.value)" :label="item.label" :value="item.value" />
<el-table-column prop="website_url" label="首页网址" min-width="200" show-overflow-tooltip /> </el-select>
<el-table-column prop="beian_year" label="备案年份" min-width="100" /> </el-form-item>
<el-table-column prop="snapshot_years" label="快照年份" min-width="120" /> <el-form-item label="检测状态">
<el-table-column prop="backlink_count" label="友链数" min-width="90" /> <el-select v-model="filters.detect_status">
<el-table-column prop="backlink_gt_10" label="友链>10" min-width="90"> <el-option v-for="item in detectOptions" :key="String(item.value)" :label="item.label" :value="item.value" />
<template #default="{ row }">{{ row.backlink_gt_10 ? "是" : "否" }}</template> </el-select>
</el-table-column> </el-form-item>
<el-table-column prop="detect_time" label="检测时间" min-width="180" /> <el-form-item label="使用状态">
</el-table> <el-select v-model="filters.use_status">
<el-option v-for="item in useOptions" :key="String(item.value)" :label="item.label" :value="item.value" />
</el-select>
</el-form-item>
<el-form-item v-show="showAdvancedFilters" label="复核状态">
<el-select v-model="filters.review_status">
<el-option v-for="item in reviewOptions" :key="String(item.value)" :label="item.label" :value="item.value" />
</el-select>
</el-form-item>
<el-form-item v-show="showAdvancedFilters" label="备案状态">
<el-select v-model="filters.has_beian">
<el-option label="全部" :value="null" />
<el-option label="有备案" :value="2" />
<el-option label="无备案" :value="3" />
<el-option label="未检测" :value="1" />
</el-select>
</el-form-item>
<el-form-item v-show="showAdvancedFilters" label="备案年份">
<el-input v-model="filters.beian_year" placeholder="如 2023" clearable />
</el-form-item>
<el-form-item v-show="showAdvancedFilters" label="快照年份">
<el-input v-model="filters.snapshot_year" placeholder="如 2024" clearable />
</el-form-item>
<el-form-item v-show="showAdvancedFilters" label="首页网址">
<el-input v-model="filters.website_url" placeholder="模糊匹配网址" clearable />
</el-form-item>
<el-form-item v-show="showAdvancedFilters" label="单位性质">
<el-select v-model="filters.company_type">
<el-option label="全部" value="" />
<el-option label="企业" value="企业" />
<el-option label="个人" value="个人" />
</el-select>
</el-form-item>
<el-form-item v-show="showAdvancedFilters" label="来源类型">
<el-select v-model="filters.source_type">
<el-option v-for="item in sourceOptions" :key="String(item.value)" :label="item.label" :value="item.value" />
</el-select>
</el-form-item>
<el-form-item v-show="showAdvancedFilters" label="友链 > 10">
<el-switch v-model="filters.backlink_gt_10" />
</el-form-item>
</el-form>
<div class="pager"> <div class="active-filters">
<el-pagination <span class="active-filters__label">当前筛选</span>
background <template v-if="activeFilterChips.length">
layout="total, sizes, prev, pager, next" <span v-for="chip in activeFilterChips" :key="chip.label" class="active-filter-chip">
:current-page="page" {{ chip.label }}
:page-size="pageSize" </span>
:page-sizes="[20, 50, 100, 200]" </template>
:total="total" <span v-else class="active-filters__empty">未设置额外筛选，当前展示全量结果。</span>
@current-change="changePage" </div>
@size-change="changePageSize"
/> <div class="table-region">
<el-table :data="rows" border size="small" height="100%" @selection-change="handleSelectionChange">
<el-table-column type="selection" width="48" fixed="left" />
<el-table-column label="域名" min-width="200" fixed="left" show-overflow-tooltip>
<template #default="{ row }">
<el-button text class="domain-link" @click="openDomainDetail(row)">
{{ row.domain }}
</el-button>
</template>
</el-table-column>
<el-table-column prop="register_status" min-width="100" show-overflow-tooltip>
<template #header>
<el-tooltip content="表示域名当前是否仍处于可注册/已注册等注册态。" placement="top">
<span class="th-label">注册状态</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column prop="use_status" min-width="100" show-overflow-tooltip>
<template #header>
<el-tooltip content="表示该域名当前是否已被业务侧占用、可用或待处理。" placement="top">
<span class="th-label">使用状态</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column prop="detect_status" min-width="100" show-overflow-tooltip>
<template #header>
<el-tooltip content="检测通过表示本轮已跑完启用流程，且未进黑名单、未落失败。" placement="top">
<span class="th-label">检测状态</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column prop="review_status" label="复核状态" min-width="100" show-overflow-tooltip />
<el-table-column prop="expire_date" label="过期时间" min-width="170" show-overflow-tooltip />
<el-table-column prop="company_type" label="单位性质" min-width="100" show-overflow-tooltip />
<el-table-column prop="source_label" label="来源类型" min-width="110" show-overflow-tooltip />
<el-table-column prop="has_beian" label="备案" min-width="80" show-overflow-tooltip />
<el-table-column prop="website_url" label="首页网址" min-width="200" show-overflow-tooltip />
<el-table-column prop="beian_year" label="备案年份" min-width="100" show-overflow-tooltip />
<el-table-column prop="snapshot_years" label="快照年份" min-width="120" show-overflow-tooltip />
<el-table-column prop="baidu_history" min-width="110" show-overflow-tooltip>
<template #header>
<el-tooltip content="是：历史快照或历史收录结果中命中过；否：未命中或未发现。" placement="top">
<span class="th-label">百度历史收录</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column prop="baidu_site" min-width="110" show-overflow-tooltip>
<template #header>
<el-tooltip content="是：当前百度 site 查询有收录；否：无收录或未命中。" placement="top">
<span class="th-label">百度Site收录</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column prop="is_chinese_title" min-width="100" show-overflow-tooltip>
<template #header>
<el-tooltip content="是：首页标题含中文；否：标题不是中文或未取到标题。" placement="top">
<span class="th-label">标题为中文</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column prop="qihu360_site" min-width="110" show-overflow-tooltip>
<template #header>
<el-tooltip content="是：360 搜索 site 查询有收录；否：无收录或未命中。" placement="top">
<span class="th-label">360 Site收录</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column prop="google_site" min-width="120" show-overflow-tooltip>
<template #header>
<el-tooltip content="是：Google site 查询有收录；否：无收录或未命中。" placement="top">
<span class="th-label">Google Site收录</span>
</el-tooltip>
</template>
</el-table-column>
<el-table-column label="步骤诊断" min-width="150">
<template #default="{ row }">
<el-tag v-if="row.step_summary?.has_degraded" type="warning" effect="light">
{{ row.step_summary.summary_text }}
</el-tag>
<el-tag v-else-if="row.step_summary?.has_failed" type="danger" effect="light">
{{ row.step_summary.summary_text }}
</el-tag>
<span v-else class="step-summary-ok">{{ row.step_summary?.summary_text || "步骤正常" }}</span>
</template>
</el-table-column>
<el-table-column prop="backlink_count" label="友链数" min-width="90" show-overflow-tooltip />
<el-table-column prop="backlink_gt_10" label="友链>10" min-width="90">
<template #default="{ row }">{{ row.backlink_gt_10 ? "是" : "否" }}</template>
</el-table-column>
<el-table-column prop="detect_time" label="检测时间" min-width="180" show-overflow-tooltip />
</el-table>
</div>
<div class="domains-footer">
<div class="pager-meta">
<span class="footer-summary">共 {{ total }} 条，第 {{ page }} 页 / 每页 {{ pageSize }} 条 / 已选 {{ selectedIds.length }} 条</span>
<el-tooltip
effect="dark"
placement="top"
content="当前筛选条件、页码和每页数量会自动记住；刷新页面后会恢复上次工作面。"
>
<span class="hint-chip">筛选状态已自动记忆</span>
</el-tooltip>
<el-tooltip
effect="dark"
placement="top"
content="检测通过=本轮已跑完当前启用的检测流程，且未进黑名单、未落检测失败；百度历史/百度Site/360 Site/Google Site 的“是”表示该搜索源有收录，“否”表示未收录或未命中；标题为中文 的“是”表示首页标题含中文，“否”表示标题不是中文。"
>
<span class="hint-chip">字段说明</span>
</el-tooltip>
</div>
<div class="pager">
<el-pagination
background
small
layout="total, sizes, prev, pager, next"
:current-page="page"
:page-size="pageSize"
:page-sizes="[20, 50, 100, 200]"
:total="total"
@current-change="changePage"
@size-change="changePageSize"
/>
</div>
</div>
</div> </div>
<el-dialog v-model="batchDialogVisible" title="批量更新选中域名" width="720px"> <el-dialog v-model="batchDialogVisible" title="批量更新选中域名" width="720px">
@@ -90,6 +218,9 @@
<el-option v-for="item in reviewOptions" :key="item.value" :label="item.label" :value="item.value" /> <el-option v-for="item in reviewOptions" :key="item.value" :label="item.label" :value="item.value" />
</el-select> </el-select>
</el-form-item> </el-form-item>
<el-form-item label="过期时间">
<el-input v-model="batchForm.expire_date" placeholder="如 2026-12-31 00:00:00" clearable />
</el-form-item>
<el-form-item label="备案状态"> <el-form-item label="备案状态">
<el-select v-model="batchForm.has_beian" clearable> <el-select v-model="batchForm.has_beian" clearable>
<el-option label="有备案" :value="2" /> <el-option label="有备案" :value="2" />
@@ -118,22 +249,122 @@
<el-form-item label="友链数"> <el-form-item label="友链数">
<el-input-number v-model="batchForm.backlink_count" :min="0" :max="1000000" /> <el-input-number v-model="batchForm.backlink_count" :min="0" :max="1000000" />
</el-form-item> </el-form-item>
<el-form-item label="百度历史收录">
<el-select v-model="batchForm.baidu_history" clearable>
<el-option label="是" value="是" />
<el-option label="否" value="否" />
</el-select>
</el-form-item>
<el-form-item label="百度Site收录">
<el-select v-model="batchForm.baidu_site" clearable>
<el-option label="是" value="是" />
<el-option label="否" value="否" />
</el-select>
</el-form-item>
<el-form-item label="标题为中文">
<el-select v-model="batchForm.is_chinese_title" clearable>
<el-option label="是" value="是" />
<el-option label="否" value="否" />
</el-select>
</el-form-item>
<el-form-item label="360 Site收录">
<el-select v-model="batchForm.qihu360_site" clearable>
<el-option label="是" value="是" />
<el-option label="否" value="否" />
</el-select>
</el-form-item>
<el-form-item label="Google Site收录">
<el-select v-model="batchForm.google_site" clearable>
<el-option label="是" value="是" />
<el-option label="否" value="否" />
</el-select>
</el-form-item>
</el-form> </el-form>
<template #footer> <template #footer>
<el-button @click="closeBatchDialog">取消</el-button> <el-button @click="closeBatchDialog">取消</el-button>
<el-button type="primary" @click="submitBatchUpdate">确认更新</el-button> <el-button type="primary" @click="submitBatchUpdate">确认更新</el-button>
</template> </template>
</el-dialog> </el-dialog>
<el-dialog v-model="exportDialogVisible" title="导出筛选结果" width="520px">
<el-form label-width="100px">
<el-form-item label="导出格式">
<el-radio-group v-model="exportForm.type">
<el-radio-button label="txt">TXT</el-radio-button>
<el-radio-button label="csv">CSV</el-radio-button>
<el-radio-button label="xlsx">Excel</el-radio-button>
</el-radio-group>
</el-form-item>
<el-form-item label="导出范围">
<el-radio-group v-model="exportForm.scope">
<el-radio-button label="page">当前页</el-radio-button>
<el-radio-button label="pages">导出几页</el-radio-button>
<el-radio-button label="all">全部</el-radio-button>
</el-radio-group>
</el-form-item>
<el-form-item v-if="exportForm.scope === 'pages'" label="导出页数">
<el-input-number v-model="exportForm.page_count" :min="1" :max="1000" />
</el-form-item>
<el-alert
:closable="false"
show-icon
type="info"
:title="`将按当前筛选条件导出，当前页码 ${page}，每页 ${pageSize} 条。`"
/>
</el-form>
<template #footer>
<el-button @click="exportDialogVisible = false">取消</el-button>
<el-button type="primary" :loading="exporting" @click="submitExport">生成导出文件</el-button>
</template>
</el-dialog>
<el-drawer v-model="detailVisible" title="域名检测详情" size="720px">
<template v-if="detailLoading">
<div class="detail-loading">正在加载域名详情...</div>
</template>
<template v-else-if="detailRow">
<el-descriptions :column="2" border>
<el-descriptions-item label="域名">{{ detailRow.domain }}</el-descriptions-item>
<el-descriptions-item label="检测状态">{{ detailRow.detect_status }}</el-descriptions-item>
<el-descriptions-item label="复核状态">{{ detailRow.review_status }}</el-descriptions-item>
<el-descriptions-item label="来源类型">{{ detailRow.source_label }}</el-descriptions-item>
<el-descriptions-item label="步骤诊断">{{ detailRow.step_summary?.summary_text || "步骤正常" }}</el-descriptions-item>
<el-descriptions-item label="检测时间">{{ detailRow.detect_time || "-" }}</el-descriptions-item>
<el-descriptions-item label="首页网址">{{ detailRow.website_url || "-" }}</el-descriptions-item>
<el-descriptions-item label="快照年份">{{ detailRow.snapshot_years || "-" }}</el-descriptions-item>
</el-descriptions>
<el-table :data="detailRow.step_details || []" border size="small" style="margin-top: 16px">
<el-table-column prop="label" label="步骤" min-width="140" />
<el-table-column label="状态" min-width="110">
<template #default="{ row }">
<el-tag :type="stepStateTagType(row.state)" effect="light">
{{ stepStateText(row.state) }}
</el-tag>
</template>
</el-table-column>
<el-table-column prop="message" label="说明" min-width="320" show-overflow-tooltip />
<el-table-column prop="checked_at" label="检查时间" min-width="170" show-overflow-tooltip />
</el-table>
</template>
<template v-else>
<div class="detail-loading">当前没有可展示的详情。</div>
</template>
</el-drawer>
</PageCard> </PageCard>
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { onMounted, ref } from "vue"; import { computed, onMounted, ref, watch } from "vue";
import { useRoute } from "vue-router";
import { ElMessage } from "element-plus"; import { ElMessage } from "element-plus";
import PageCard from "@/components/PageCard.vue"; import PageCard from "@/components/PageCard.vue";
import { domainsApi } from "@/api/modules"; import { domainsApi, exportsApi } from "@/api/modules";
type Option = { label: string; value: number }; type Option = { label: string; value: number | null };
const DOMAINS_VIEW_STORAGE_KEY = "domaincheck:domains:view-state";
const route = useRoute();
const rows = ref<any[]>([]); const rows = ref<any[]>([]);
const page = ref(1); const page = ref(1);
@@ -141,52 +372,132 @@ const pageSize = ref(20);
const total = ref(0); const total = ref(0);
const selectedIds = ref<number[]>([]); const selectedIds = ref<number[]>([]);
const batchDialogVisible = ref(false); const batchDialogVisible = ref(false);
const exportDialogVisible = ref(false);
const exporting = ref(false);
const restoredState = ref(false);
const showAdvancedFilters = ref(false);
const detailVisible = ref(false);
const detailLoading = ref(false);
const detailRow = ref<any | null>(null);
const registerOptions = ref<Option[]>([]); const registerOptions = ref<Option[]>([]);
const detectOptions = ref<Option[]>([]); const detectOptions = ref<Option[]>([]);
const useOptions = ref<Option[]>([]); const useOptions = ref<Option[]>([]);
const reviewOptions = ref<Option[]>([]); const reviewOptions = ref<Option[]>([]);
const sourceOptions = ref<Option[]>([]);
const defaultFilters = () => ({ const defaultFilters = () => ({
domain_keyword: "", domain_keyword: "",
register_status: undefined as number | undefined, register_status: null as number | null,
detect_status: undefined as number | undefined, detect_status: null as number | null,
use_status: undefined as number | undefined, use_status: null as number | null,
review_status: undefined as number | undefined, review_status: null as number | null,
has_beian: undefined as number | undefined, has_beian: null as number | null,
beian_year: "", beian_year: "",
snapshot_year: "", snapshot_year: "",
website_url: "", website_url: "",
company_type: "",
source_type: null as number | null,
backlink_gt_10: false backlink_gt_10: false
}); });
const defaultBatchForm = () => ({ const defaultBatchForm = () => ({
review_status: undefined as number | undefined, review_status: undefined as number | undefined,
expire_date: "",
has_beian: undefined as number | undefined, has_beian: undefined as number | undefined,
beian_year: "", beian_year: "",
snapshot_years: "", snapshot_years: "",
company_type: "", company_type: "",
website_url: "", website_url: "",
detect_time: "", detect_time: "",
backlink_count: undefined as number | undefined backlink_count: undefined as number | undefined,
baidu_history: "",
baidu_site: "",
is_chinese_title: "",
qihu360_site: "",
google_site: ""
}); });
const filters = ref(defaultFilters()); const filters = ref(defaultFilters());
const batchForm = ref(defaultBatchForm()); const batchForm = ref(defaultBatchForm());
const exportForm = ref({
type: "txt",
scope: "page",
page_count: 1
});
const findOptionLabel = (options: Option[], value: number | null) => {
if (value === null || value === undefined) return "";
return options.find((item) => item.value === value)?.label || String(value);
};
const activeFilterChips = computed(() => {
const chips: Array<{ label: string }> = [];
if (filters.value.domain_keyword.trim()) chips.push({ label: `域名：${filters.value.domain_keyword.trim()}` });
if (filters.value.register_status !== null) chips.push({ label: `注册：${findOptionLabel(registerOptions.value, filters.value.register_status)}` });
if (filters.value.detect_status !== null) chips.push({ label: `检测：${findOptionLabel(detectOptions.value, filters.value.detect_status)}` });
if (filters.value.use_status !== null) chips.push({ label: `使用：${findOptionLabel(useOptions.value, filters.value.use_status)}` });
if (filters.value.review_status !== null) chips.push({ label: `复核：${findOptionLabel(reviewOptions.value, filters.value.review_status)}` });
if (filters.value.has_beian !== null) {
const beianText = { 1: "未检测", 2: "有备案", 3: "无备案" }[filters.value.has_beian as 1 | 2 | 3] || String(filters.value.has_beian);
chips.push({ label: `备案：${beianText}` });
}
if (filters.value.beian_year.trim()) chips.push({ label: `备案年份：${filters.value.beian_year.trim()}` });
if (filters.value.snapshot_year.trim()) chips.push({ label: `快照年份：${filters.value.snapshot_year.trim()}` });
if (filters.value.website_url.trim()) chips.push({ label: `首页网址：${filters.value.website_url.trim()}` });
if (filters.value.company_type.trim()) chips.push({ label: `单位性质：${filters.value.company_type.trim()}` });
if (filters.value.source_type !== null) chips.push({ label: `来源：${findOptionLabel(sourceOptions.value, filters.value.source_type)}` });
if (filters.value.backlink_gt_10) chips.push({ label: "友链 > 10" });
return chips;
});
const restoreViewState = () => {
try {
const raw = window.localStorage.getItem(DOMAINS_VIEW_STORAGE_KEY);
if (!raw) return;
const parsed = JSON.parse(raw);
filters.value = {
...defaultFilters(),
...(parsed.filters || {})
};
page.value = Math.max(Number(parsed.page || 1), 1);
pageSize.value = [20, 50, 100, 200].includes(Number(parsed.pageSize)) ? Number(parsed.pageSize) : 20;
showAdvancedFilters.value = Boolean(parsed.showAdvancedFilters);
} catch {
window.localStorage.removeItem(DOMAINS_VIEW_STORAGE_KEY);
} finally {
restoredState.value = true;
}
};
const persistViewState = () => {
if (!restoredState.value) return;
window.localStorage.setItem(
DOMAINS_VIEW_STORAGE_KEY,
JSON.stringify({
filters: filters.value,
page: page.value,
pageSize: pageSize.value,
showAdvancedFilters: showAdvancedFilters.value
})
);
};
const buildParams = () => { const buildParams = () => {
const params: Record<string, unknown> = { const params: Record<string, unknown> = {
page: page.value, page: page.value,
page_size: pageSize.value, page_size: pageSize.value,
domain_keyword: filters.value.domain_keyword || undefined, domain_keyword: filters.value.domain_keyword || undefined,
register_status: filters.value.register_status, register_status: filters.value.register_status ?? undefined,
detect_status: filters.value.detect_status, detect_status: filters.value.detect_status ?? undefined,
use_status: filters.value.use_status, use_status: filters.value.use_status ?? undefined,
review_status: filters.value.review_status, review_status: filters.value.review_status ?? undefined,
has_beian: filters.value.has_beian, has_beian: filters.value.has_beian ?? undefined,
beian_year: filters.value.beian_year || undefined, beian_year: filters.value.beian_year || undefined,
snapshot_year: filters.value.snapshot_year || undefined, snapshot_year: filters.value.snapshot_year || undefined,
website_url: filters.value.website_url || undefined website_url: filters.value.website_url || undefined,
company_type: filters.value.company_type || undefined,
source_type: filters.value.source_type ?? undefined
}; };
if (filters.value.backlink_gt_10) { if (filters.value.backlink_gt_10) {
params.backlink_gt_10 = true; params.backlink_gt_10 = true;
@@ -197,10 +508,11 @@ const buildParams = () => {
const loadFilterOptions = async () => { const loadFilterOptions = async () => {
try { try {
const response = await domainsApi.filters(); const response = await domainsApi.filters();
registerOptions.value = response.data.register_status || []; registerOptions.value = [{ label: "全部", value: null }, ...(response.data.register_status || [])];
detectOptions.value = response.data.detect_status || []; detectOptions.value = [{ label: "全部", value: null }, ...(response.data.detect_status || [])];
useOptions.value = response.data.use_status || []; useOptions.value = [{ label: "全部", value: null }, ...(response.data.use_status || [])];
reviewOptions.value = response.data.review_status || []; reviewOptions.value = [{ label: "全部", value: null }, ...(response.data.review_status || [])];
sourceOptions.value = [{ label: "全部", value: null }, ...(response.data.source_type || [])];
} catch { } catch {
ElMessage.error("读取筛选选项失败"); ElMessage.error("读取筛选选项失败");
} }
@@ -217,6 +529,37 @@ const loadDomains = async () => {
} }
}; };
const stepStateText = (value: string) => {
if (value === "passed") return "正常";
if (value === "degraded") return "降级";
if (value === "failed") return "失败";
if (value === "blacklisted") return "命中";
return value || "-";
};
const stepStateTagType = (value: string) => {
if (value === "passed") return "success";
if (value === "degraded") return "warning";
if (value === "failed") return "danger";
if (value === "blacklisted") return "danger";
return "info";
};
const openDomainDetail = async (row: any) => {
detailVisible.value = true;
detailLoading.value = true;
detailRow.value = null;
try {
const response = await domainsApi.detail(row.id);
detailRow.value = response.data;
} catch {
ElMessage.error("读取域名详情失败");
detailVisible.value = false;
} finally {
detailLoading.value = false;
}
};
const handleSelectionChange = (selection: any[]) => { const handleSelectionChange = (selection: any[]) => {
selectedIds.value = selection.map((item) => item.id); selectedIds.value = selection.map((item) => item.id);
}; };
@@ -241,6 +584,7 @@ const reset = () => {
filters.value = defaultFilters(); filters.value = defaultFilters();
page.value = 1; page.value = 1;
pageSize.value = 20; pageSize.value = 20;
window.localStorage.removeItem(DOMAINS_VIEW_STORAGE_KEY);
loadDomains(); loadDomains();
}; };
@@ -249,6 +593,26 @@ const closeBatchDialog = () => {
batchForm.value = defaultBatchForm(); batchForm.value = defaultBatchForm();
}; };
const submitExport = async () => {
exporting.value = true;
try {
const response = await exportsApi.run({
...buildParams(),
type: exportForm.value.type,
scope: exportForm.value.scope,
page: page.value,
page_size: pageSize.value,
page_count: exportForm.value.page_count
});
exportDialogVisible.value = false;
ElMessage.success(response.message || "导出文件已生成");
} catch (error: any) {
ElMessage.error(error?.message || "生成导出文件失败");
} finally {
exporting.value = false;
}
};
const submitBatchUpdate = async () => { const submitBatchUpdate = async () => {
const updates: Record<string, unknown> = {}; const updates: Record<string, unknown> = {};
Object.entries(batchForm.value).forEach(([key, value]) => { Object.entries(batchForm.value).forEach(([key, value]) => {
@@ -273,32 +637,161 @@ const submitBatchUpdate = async () => {
} }
}; };
watch(filters, persistViewState, { deep: true });
watch(page, persistViewState);
watch(pageSize, persistViewState);
watch(showAdvancedFilters, persistViewState);
onMounted(async () => { onMounted(async () => {
restoreViewState();
if (route.query.source_type) {
const parsed = Number(route.query.source_type);
if (!Number.isNaN(parsed)) {
filters.value.source_type = parsed;
page.value = 1;
}
}
await loadFilterOptions(); await loadFilterOptions();
await loadDomains(); await loadDomains();
}); });
</script> </script>
<style scoped> <style scoped>
.domains-layout {
height: 100%;
display: flex;
flex-direction: column;
min-height: 0;
gap: 12px;
}
.header-actions {
display: flex;
gap: 8px;
align-items: center;
flex-wrap: nowrap;
}
.th-label {
display: inline-flex;
align-items: center;
cursor: help;
}
.domain-link {
padding: 0;
font-weight: 600;
}
.toolbar-grid { .toolbar-grid {
display: grid; display: grid;
grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
gap: 8px 16px; gap: 6px 14px;
margin-bottom: 16px; flex-shrink: 0;
} }
.toolbar-grid :deep(.el-form-item) { .toolbar-grid :deep(.el-form-item) {
margin-bottom: 8px; margin-bottom: 6px;
} }
.actions { .active-filters {
align-self: end; display: flex;
align-items: center;
gap: 8px;
min-height: 30px;
margin-top: -2px;
padding: 2px 2px 0;
color: #64748b;
font-size: 12px;
flex-wrap: wrap;
}
.active-filters__label {
color: #334155;
font-weight: 600;
}
.active-filters__empty {
color: #94a3b8;
}
.step-summary-ok {
color: #64748b;
font-size: 12px;
}
.active-filter-chip {
display: inline-flex;
align-items: center;
max-width: 240px;
min-height: 24px;
padding: 0 10px;
border-radius: 999px;
background: #f1f5f9;
color: #334155;
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}
.table-region {
flex: 1;
min-height: 0;
overflow: hidden;
}
.table-region :deep(.el-table__row td) {
height: 38px;
}
.table-region :deep(.cell) {
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
line-height: 22px;
}
.domains-footer {
flex-shrink: 0;
display: flex;
align-items: center;
justify-content: space-between;
gap: 12px;
min-height: 40px;
}
.pager-meta {
display: flex;
align-items: center;
gap: 10px;
min-height: 32px;
padding: 0 2px 0 0;
color: #64748b;
font-size: 12px;
flex-wrap: wrap;
min-width: 0;
}
.footer-summary {
color: #334155;
font-weight: 500;
}
.hint-chip {
display: inline-flex;
align-items: center;
min-height: 26px;
padding: 0 10px;
border-radius: 999px;
background: #eef4ff;
color: #31558f;
cursor: help;
} }
.pager { .pager {
display: flex; display: flex;
justify-content: flex-end; justify-content: flex-end;
margin-top: 16px; flex-shrink: 0;
} }
.batch-grid { .batch-grid {
@@ -306,4 +799,26 @@ onMounted(async () => {
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr)); grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
gap: 8px 16px; gap: 8px 16px;
} }
.detail-loading {
padding: 24px 4px;
color: #64748b;
}
@media (max-width: 1100px) {
.header-actions {
flex-wrap: wrap;
justify-content: flex-end;
}
.domains-footer {
align-items: flex-start;
flex-direction: column;
}
.pager {
width: 100%;
justify-content: flex-end;
}
}
</style> </style>

451
domain-web/src/views/imports/ImportsView.vue
View File

@@ -1,18 +1,80 @@
<template> <template>
<PageCard title="域名导入" description="已改成导入任务模式，适合后续大文件上传和 Linux 后端持续处理。"> <PageCard title="域名导入" description="已升级为任务中心：适合大文件上传、文本粘贴导入，并支持阶段查看、日志常驻和切页回看。">
<div class="toolbar"> <div class="toolbar">
<el-upload :show-file-list="false" :auto-upload="false" :on-change="handleChange" accept=".txt"> <el-upload :show-file-list="false" :auto-upload="false" :on-change="handleChange" accept=".txt">
<template #trigger> <template #trigger>
<el-button type="primary">选择 TXT 文件</el-button> <el-button type="primary">选择 TXT 文件</el-button>
</template> </template>
</el-upload> </el-upload>
<el-select v-model="sourceType" style="width: 150px">
<el-option v-for="item in sourceTypeOptions" :key="item.value" :label="item.label" :value="item.value" />
</el-select>
<span class="filename">{{ selectedName || "尚未选择文件" }}</span> <span class="filename">{{ selectedName || "尚未选择文件" }}</span>
<el-button :loading="uploading" :disabled="!selectedFile" @click="submitUpload">创建导入任务</el-button> <el-button :loading="uploading" :disabled="!selectedFile" @click="submitUpload">创建导入任务</el-button>
<el-button plain @click="downloadTemplate">下载示例模板</el-button>
<el-button plain :loading="loading" @click="loadAll">刷新</el-button> <el-button plain :loading="loading" @click="loadAll">刷新</el-button>
<el-switch v-model="autoRefresh" inline-prompt active-text="自动刷新" inactive-text="手动" /> <el-switch v-model="autoRefresh" inline-prompt active-text="自动刷新" inactive-text="手动" />
<span class="updated-at">最近刷新：{{ lastUpdatedAt || "暂无" }}</span> <span class="updated-at">最近同步：{{ lastUpdatedAt || "暂无" }}</span>
</div> </div>
<el-alert
v-if="selectedName"
:closable="false"
show-icon
type="info"
style="margin-bottom: 12px"
:title="selectedFileSummary"
/>
<el-row v-if="selectedName && fileStats.total > 0" :gutter="12" style="margin-bottom: 12px">
<el-col :xs="12" :md="6"><el-statistic title="总行数" :value="fileStats.total" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="有效域名" :value="fileStats.valid" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="重复域名" :value="fileStats.duplicates" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="非法域名" :value="fileStats.invalid" /></el-col>
</el-row>
<el-alert
v-if="selectedName && selectedFileLineCount !== null && selectedFileLineCount > 1000"
:closable="false"
show-icon
type="warning"
style="margin-bottom: 12px"
title="当前文件行数较多，建议直接创建后台导入任务并在下方任务中心查看日志，不必等待页面停留。"
/>
<el-table
v-if="selectedFilePreview.length"
:data="selectedFilePreview"
border
size="small"
style="margin-bottom: 12px"
>
<el-table-column prop="line" label="行号" width="90" />
<el-table-column prop="domain" label="预览内容" min-width="320" show-overflow-tooltip />
</el-table>
<el-input
v-model="manualText"
type="textarea"
:rows="8"
placeholder="支持手工粘贴域名，一行一个。对齐旧桌面版的文本导入方式。"
style="margin-bottom: 12px"
/>
<div class="toolbar" style="margin-top: 0">
<el-button type="primary" plain :loading="submittingText" @click="submitTextImport">粘贴内容创建导入任务</el-button>
<el-button plain :disabled="!manualText.trim()" @click="clearManualText">清空内容</el-button>
<span class="updated-at">当前行数：{{ manualCount }}</span>
<span class="updated-at">{{ draftHint }}</span>
</div>
<el-row v-if="manualStats.total > 0" :gutter="12" style="margin-bottom: 16px">
<el-col :xs="12" :md="6"><el-statistic title="文本总行数" :value="manualStats.total" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="有效域名" :value="manualStats.valid" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="重复域名" :value="manualStats.duplicates" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="非法域名" :value="manualStats.invalid" /></el-col>
</el-row>
<el-descriptions :column="2" border> <el-descriptions :column="2" border>
<el-descriptions-item label="已导入域名总数">{{ summary.domains_total }}</el-descriptions-item> <el-descriptions-item label="已导入域名总数">{{ summary.domains_total }}</el-descriptions-item>
<el-descriptions-item label="detect_tasks 总数">{{ summary.detect_tasks_total }}</el-descriptions-item> <el-descriptions-item label="detect_tasks 总数">{{ summary.detect_tasks_total }}</el-descriptions-item>
@@ -21,16 +83,60 @@
<el-descriptions-item :span="2" label="最近导入时间">{{ summary.last_import_time || "暂无" }}</el-descriptions-item> <el-descriptions-item :span="2" label="最近导入时间">{{ summary.last_import_time || "暂无" }}</el-descriptions-item>
</el-descriptions> </el-descriptions>
<el-row :gutter="12" style="margin-top: 16px; margin-bottom: 16px">
<el-col :xs="24" :md="12">
<div class="summary-card">
<div class="summary-title">当前导入阶段</div>
<div class="summary-value">{{ currentPhaseLabel }}</div>
<div class="summary-note">{{ currentPhaseDetail }}</div>
</div>
</el-col>
<el-col :xs="24" :md="12">
<div class="summary-card">
<div class="summary-title">当前任务概览</div>
<div class="summary-value">{{ currentTaskSummary }}</div>
<div class="summary-note">支持大文件上传和文本粘贴导入，离开页面后回来仍可继续回看。</div>
<el-button
v-if="selectedTask?.source_type"
plain
size="small"
style="margin-top: 10px"
@click="goDomainsBySource(selectedTask.source_type)"
>
前往域名筛选查看结果
</el-button>
</div>
</el-col>
</el-row>
<el-alert v-if="lastResult" style="margin-top: 16px" type="success" :closable="false" show-icon :title="lastResult" /> <el-alert v-if="lastResult" style="margin-top: 16px" type="success" :closable="false" show-icon :title="lastResult" />
<el-alert v-if="loadError" style="margin-top: 12px" type="warning" :closable="false" show-icon title="读取导入任务失败，已保留上一次结果。" /> <el-alert v-if="loadError" style="margin-top: 12px" type="warning" :closable="false" show-icon title="读取导入任务失败，已保留上一次结果。" />
<el-table :data="tasks" border style="margin-top: 16px"> <el-table :data="tasks" border style="margin-top: 16px" highlight-current-row @current-change="handleCurrentTaskChange">
<el-table-column prop="status" label="状态" min-width="100" /> <el-table-column label="状态" min-width="100">
<template #default="{ row }">
<el-tag :type="statusTagType(row.status)" effect="dark">{{ statusText(row.status) }}</el-tag>
</template>
</el-table-column>
<el-table-column label="阶段" min-width="110">
<template #default="{ row }">
<el-tag :type="phaseTagType(row.status)" plain>{{ row.phase_label || "-" }}</el-tag>
</template>
</el-table-column>
<el-table-column prop="source_label" label="来源类型" min-width="110" />
<el-table-column prop="filename" label="文件名" min-width="220" show-overflow-tooltip /> <el-table-column prop="filename" label="文件名" min-width="220" show-overflow-tooltip />
<el-table-column prop="message" label="任务信息" min-width="260" show-overflow-tooltip /> <el-table-column prop="message" label="任务信息" min-width="260" show-overflow-tooltip />
<el-table-column prop="created_at" label="创建时间" min-width="160" /> <el-table-column prop="created_at" label="创建时间" min-width="160" />
<el-table-column prop="started_at" label="开始时间" min-width="160" /> <el-table-column prop="started_at" label="开始时间" min-width="160" />
<el-table-column prop="completed_at" label="完成时间" min-width="160" /> <el-table-column prop="completed_at" label="完成时间" min-width="160" />
<el-table-column label="结果" min-width="240">
<template #default="{ row }">
<span v-if="row.result?.stats">
总数 {{ row.result.stats.total }} / 新增 {{ row.result.stats.added }} / 已存在 {{ row.result.stats.exists }} / 无效 {{ row.result.stats.invalid }}
</span>
<span v-else>-</span>
</template>
</el-table-column>
<el-table-column label="操作" min-width="110" fixed="right"> <el-table-column label="操作" min-width="110" fixed="right">
<template #default="{ row }"> <template #default="{ row }">
<el-button <el-button
@@ -44,24 +150,37 @@
<span v-else>-</span> <span v-else>-</span>
</template> </template>
</el-table-column> </el-table-column>
<el-table-column label="结果" min-width="220">
<template #default="{ row }">
<span v-if="row.result?.stats">
总数 {{ row.result.stats.total }} / 新增 {{ row.result.stats.added }} / 已存在 {{ row.result.stats.exists }} / 无效 {{ row.result.stats.invalid }}
</span>
<span v-else>-</span>
</template>
</el-table-column>
</el-table> </el-table>
<div class="task-detail">
<div class="task-detail-header">
<div>
<div class="task-detail-title">导入日志控制台</div>
<div class="task-detail-subtitle">
{{ selectedTask ? `当前查看任务：${selectedTask.task_id}` : "请选择一条任务记录查看日志" }}
</div>
</div>
<div class="task-detail-meta">
<span v-if="selectedTask">状态：{{ statusText(selectedTask.status) }}</span>
<span v-if="selectedTask?.phase_label">阶段：{{ selectedTask.phase_label }}</span>
<span v-if="selectedTask?.result?.stats">新增：{{ selectedTask.result.stats.added || 0 }}</span>
</div>
</div>
<pre class="log-console">{{ selectedTaskLogs }}</pre>
</div>
</PageCard> </PageCard>
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { onBeforeUnmount, onMounted, ref, watch } from "vue"; import { computed, onBeforeUnmount, onMounted, ref, watch } from "vue";
import { useRouter } from "vue-router";
import { ElMessage } from "element-plus"; import { ElMessage } from "element-plus";
import PageCard from "@/components/PageCard.vue"; import PageCard from "@/components/PageCard.vue";
import { importsApi } from "@/api/modules"; import { importsApi } from "@/api/modules";
const IMPORT_DRAFT_STORAGE_KEY = "domaincheck:imports:manual-draft";
const router = useRouter();
const summary = ref({ const summary = ref({
domains_total: 0, domains_total: 0,
detect_tasks_total: 0, detect_tasks_total: 0,
@@ -70,16 +189,105 @@ const summary = ref({
running_import_tasks: 0 running_import_tasks: 0
}); });
const tasks = ref<any[]>([]); const tasks = ref<any[]>([]);
const selectedTaskId = ref("");
const selectedFile = ref<File | null>(null); const selectedFile = ref<File | null>(null);
const sourceType = ref(7);
const selectedName = ref(""); const selectedName = ref("");
const selectedFileLineCount = ref<number | null>(null);
const selectedFilePreview = ref<Array<{ line: number; domain: string }>>([]);
const selectedFileRawRows = ref<string[]>([]);
const lastResult = ref(""); const lastResult = ref("");
const loading = ref(false); const loading = ref(false);
const uploading = ref(false); const uploading = ref(false);
const loadError = ref(false); const loadError = ref(false);
const lastUpdatedAt = ref(""); const lastUpdatedAt = ref("");
const autoRefresh = ref(true); const autoRefresh = ref(true);
const manualText = ref("");
const submittingText = ref(false);
const draftSavedAt = ref("");
let timer: number | null = null; let timer: number | null = null;
const sourceTypeOptions = [
{ label: "TXT 导入", value: 7 },
{ label: "手工录入", value: 6 },
{ label: "其它", value: 9 }
];
const manualCount = computed(() =>
manualText.value
.split(/\r?\n/)
.map((item) => item.trim())
.filter(Boolean).length
);
const DOMAIN_PATTERN = /^(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+(com|net)$/i;
const normalizeDomain = (value: string) => {
let candidate = String(value || "").trim().toLowerCase();
candidate = candidate.replace(/^https?:\/\//, "");
candidate = candidate.split("/")[0].replace(/^\.+|\.+$/g, "");
if (candidate.startsWith("www.")) {
candidate = candidate.slice(4);
}
return DOMAIN_PATTERN.test(candidate) ? candidate : "";
};
const summarizeRows = (rows: string[]) => {
const normalizedList = rows
.map((item) => normalizeDomain(item))
.filter(Boolean);
const unique = new Set(normalizedList);
return {
total: rows.length,
valid: normalizedList.length,
duplicates: normalizedList.length - unique.size,
invalid: rows.length - normalizedList.length
};
};
const fileStats = computed(() => summarizeRows(selectedFileRawRows.value));
const manualStats = computed(() =>
summarizeRows(
manualText.value
.split(/\r?\n/)
.map((item) => item.trim())
.filter(Boolean)
)
);
const selectedTask = computed(() => tasks.value.find((item) => item.task_id === selectedTaskId.value) || tasks.value[0] || null);
const currentPhaseLabel = computed(() => selectedTask.value?.phase_label || "暂无任务");
const currentPhaseDetail = computed(() => selectedTask.value?.message || "当前还没有导入任务在执行。");
const currentTaskSummary = computed(() => {
if (!selectedTask.value) {
return "等待创建";
}
const result = selectedTask.value.result?.stats;
if (result) {
return `${selectedTask.value.source_label || "未知来源"} / 新增 ${result.added || 0} / 已存在 ${result.exists || 0} / 无效 ${result.invalid || 0}`;
}
return `${selectedTask.value.source_label || "未知来源"} / ${selectedTask.value.filename || "导入任务"} / ${statusText(selectedTask.value.status)}`;
});
const selectedTaskLogs = computed(() => {
const logs = selectedTask.value?.logs || [];
return logs.length ? logs.join("\n") : "暂无导入日志";
});
const selectedFileSummary = computed(() => {
if (!selectedName.value) {
return "";
}
if (selectedFileLineCount.value === null) {
return `已选择文件：${selectedName.value}，正在统计行数。`;
}
return `已选择文件：${selectedName.value}，预计非空行 ${selectedFileLineCount.value} 条，当前预览前 ${selectedFilePreview.value.length} 行；提交前已完成重复和非法格式的本地预检。`;
});
const draftHint = computed(() => {
if (!manualText.value.trim()) {
return "手工输入内容会自动暂存";
}
return draftSavedAt.value ? `草稿已暂存：${draftSavedAt.value}` : "手工输入内容会自动暂存";
});
const clearRefreshTimer = () => { const clearRefreshTimer = () => {
if (timer) { if (timer) {
window.clearInterval(timer); window.clearInterval(timer);
@@ -96,6 +304,28 @@ const ensureRefreshTimer = () => {
} }
}; };
const statusText = (value: string) => {
if (value === "queued") return "排队中";
if (value === "running") return "运行中";
if (value === "completed") return "已完成";
if (value === "failed") return "失败";
return value || "-";
};
const statusTagType = (value: string) => {
if (value === "running") return "warning";
if (value === "completed") return "success";
if (value === "failed") return "danger";
return "info";
};
const phaseTagType = (value: string) => {
if (value === "running") return "warning";
if (value === "completed") return "success";
if (value === "failed") return "danger";
return "info";
};
const loadSummary = async () => { const loadSummary = async () => {
const response = await importsApi.summary(); const response = await importsApi.summary();
summary.value = response.data; summary.value = response.data;
@@ -104,6 +334,11 @@ const loadSummary = async () => {
const loadTasks = async () => { const loadTasks = async () => {
const response = await importsApi.tasks(); const response = await importsApi.tasks();
tasks.value = response.data; tasks.value = response.data;
if (!selectedTaskId.value && tasks.value.length) {
selectedTaskId.value = tasks.value[0].task_id;
} else if (selectedTaskId.value && !tasks.value.some((item) => item.task_id === selectedTaskId.value) && tasks.value.length) {
selectedTaskId.value = tasks.value[0].task_id;
}
}; };
const loadAll = async (showError = true) => { const loadAll = async (showError = true) => {
@@ -122,9 +357,66 @@ const loadAll = async (showError = true) => {
} }
}; };
const handleChange = (uploadFile: any) => { const handleChange = async (uploadFile: any) => {
selectedFile.value = uploadFile.raw || null; selectedFile.value = uploadFile.raw || null;
selectedName.value = uploadFile.name || ""; selectedName.value = uploadFile.name || "";
selectedFileLineCount.value = null;
selectedFilePreview.value = [];
selectedFileRawRows.value = [];
const raw = uploadFile.raw;
if (!raw) return;
try {
const text = await raw.text();
const rows = text
.split(/\r?\n/)
.map((item: string) => item.trim())
.filter(Boolean);
selectedFileRawRows.value = rows;
selectedFileLineCount.value = rows.length;
selectedFilePreview.value = rows.slice(0, 10).map((domain: string, index: number) => ({
line: index + 1,
domain
}));
} catch {
selectedFileLineCount.value = null;
selectedFilePreview.value = [];
selectedFileRawRows.value = [];
}
};
const persistManualDraft = () => {
if (!manualText.value.trim()) {
window.localStorage.removeItem(IMPORT_DRAFT_STORAGE_KEY);
draftSavedAt.value = "";
return;
}
window.localStorage.setItem(IMPORT_DRAFT_STORAGE_KEY, manualText.value);
draftSavedAt.value = new Date().toLocaleString("zh-CN", { hour12: false });
};
const restoreManualDraft = () => {
const draft = window.localStorage.getItem(IMPORT_DRAFT_STORAGE_KEY);
if (draft) {
manualText.value = draft;
}
};
const clearManualText = () => {
manualText.value = "";
window.localStorage.removeItem(IMPORT_DRAFT_STORAGE_KEY);
draftSavedAt.value = "";
};
const downloadTemplate = () => {
const content = ["example.com", "example.net", "yourdomain.com"].join("\n");
const blob = new Blob([content], { type: "text/plain;charset=utf-8" });
const url = window.URL.createObjectURL(blob);
const link = document.createElement("a");
link.href = url;
link.download = "domain_import_sample.txt";
link.click();
window.URL.revokeObjectURL(url);
ElMessage.success("示例模板已生成");
}; };
const submitUpload = async () => { const submitUpload = async () => {
@@ -134,14 +426,21 @@ const submitUpload = async () => {
} }
const formData = new FormData(); const formData = new FormData();
formData.append("file", selectedFile.value); formData.append("file", selectedFile.value);
formData.append("source_type", String(sourceType.value));
uploading.value = true; uploading.value = true;
try { try {
const response = await importsApi.upload(formData); const response = await importsApi.upload(formData);
lastResult.value = `导入任务已创建：${response.data.filename}，状态 ${response.data.status}`; lastResult.value = `导入任务已创建：${response.data.filename}，状态 ${statusText(response.data.status)}`;
ElMessage.success("导入任务已创建"); ElMessage.success("导入任务已创建");
selectedFile.value = null; selectedFile.value = null;
selectedName.value = ""; selectedName.value = "";
selectedFileLineCount.value = null;
selectedFilePreview.value = [];
selectedFileRawRows.value = [];
await loadAll(false); await loadAll(false);
if (response.data?.task_id) {
selectedTaskId.value = response.data.task_id;
}
} catch { } catch {
ElMessage.error("创建导入任务失败"); ElMessage.error("创建导入任务失败");
} finally { } finally {
@@ -149,19 +448,66 @@ const submitUpload = async () => {
} }
}; };
const submitTextImport = async () => {
const text = manualText.value
.split(/\r?\n/)
.map((item) => item.trim())
.filter(Boolean)
.join("\n");
if (!text) {
ElMessage.warning("请先粘贴域名内容");
return;
}
submittingText.value = true;
try {
const response = await importsApi.createTextTask({
text,
filename: `manual_domains_${Date.now()}.txt`,
source_type: sourceType.value
});
lastResult.value = `文本导入任务已创建：${response.data.filename}，状态 ${statusText(response.data.status)}`;
ElMessage.success("文本导入任务已创建");
clearManualText();
await loadAll(false);
if (response.data?.task_id) {
selectedTaskId.value = response.data.task_id;
}
} catch {
ElMessage.error("创建文本导入任务失败");
} finally {
submittingText.value = false;
}
};
const retryTask = async (taskId: string) => { const retryTask = async (taskId: string) => {
try { try {
await importsApi.retry(taskId); await importsApi.retry(taskId);
ElMessage.success("导入任务已重新加入队列"); ElMessage.success("导入任务已重新加入队列");
await loadAll(false); await loadAll(false);
selectedTaskId.value = taskId;
} catch { } catch {
ElMessage.error("重试导入任务失败"); ElMessage.error("重试导入任务失败");
} }
}; };
const goDomainsBySource = (sourceTypeValue: number) => {
router.push({
path: "/domains",
query: {
source_type: String(sourceTypeValue)
}
});
};
const handleCurrentTaskChange = (row: any | null) => {
selectedTaskId.value = row?.task_id || "";
};
watch(autoRefresh, ensureRefreshTimer); watch(autoRefresh, ensureRefreshTimer);
watch(manualText, persistManualDraft);
onMounted(async () => { onMounted(async () => {
restoreManualDraft();
await loadAll(); await loadAll();
ensureRefreshTimer(); ensureRefreshTimer();
}); });
@@ -186,4 +532,79 @@ onBeforeUnmount(clearRefreshTimer);
color: #64748b; color: #64748b;
font-size: 13px; font-size: 13px;
} }
.summary-card {
border: 1px solid #dbe4ee;
border-radius: 14px;
padding: 14px 16px;
background: #f8fbff;
}
.summary-title {
color: #64748b;
font-size: 13px;
}
.summary-value {
margin: 8px 0 10px;
color: #0f172a;
font-size: 22px;
font-weight: 700;
}
.summary-note {
color: #64748b;
font-size: 12px;
line-height: 1.6;
}
.task-detail {
margin: 16px 0;
border-radius: 10px;
overflow: hidden;
border: 1px solid #1f2937;
background: #020617;
}
.task-detail-header {
display: flex;
justify-content: space-between;
gap: 12px;
align-items: center;
padding: 12px 16px;
background: #111827;
color: #e5e7eb;
}
.task-detail-title {
font-size: 15px;
font-weight: 600;
}
.task-detail-subtitle,
.task-detail-meta {
color: #94a3b8;
font-size: 12px;
}
.task-detail-meta {
display: flex;
gap: 12px;
flex-wrap: wrap;
}
.log-console {
margin: 0;
min-height: 220px;
max-height: 420px;
overflow: auto;
padding: 16px;
color: #e0e7ff;
background: #020617;
font-size: 12px;
line-height: 1.7;
font-family: Consolas, Monaco, monospace;
white-space: pre-wrap;
word-break: break-word;
}
</style> </style>

745
domain-web/src/views/juming/JumingView.vue Normal file
View File

@@ -0,0 +1,745 @@
<template>
<PageCard title="聚名采集" description="已升级为任务中心：提交后后台持续执行，支持阶段查看、日志常驻和切页回看。">
<el-alert
:title="status.cookie_ready ? `聚名登录态已就绪：${status.cookie_storage}，Cookie ${status.cookie_count || 0} 项` : '请先使用下方账号密码登录聚名；登录成功后服务器会自动保存 Cookie'"
:type="status.cookie_ready ? 'success' : 'warning'"
:closable="false"
show-icon
style="margin-bottom: 16px"
/>
<el-alert
:title="savedJumingSummary"
:type="savedJumingAccount ? 'info' : 'warning'"
:closable="false"
show-icon
style="margin-bottom: 16px"
/>
<el-alert
:title="juchaSummary"
:type="status.jucha?.cookie_ready ? 'success' : 'warning'"
:closable="false"
show-icon
style="margin-bottom: 16px"
/>
<el-form label-width="110px" class="login-form">
<el-form-item label="聚名账号">
<el-input v-model="loginForm.email" placeholder="请输入聚名账号" />
</el-form-item>
<el-form-item label="聚名密码">
<el-input v-model="loginForm.password" type="password" show-password placeholder="请输入聚名密码" />
<el-button type="primary" plain :loading="loggingIn" style="margin-left: 12px" @click="loginJuming">
账号密码登录
</el-button>
<span class="login-status" :class="{ success: status.cookie_ready, warning: !status.cookie_ready }">
{{ status.cookie_ready ? "已登录成功" : "未登录" }}
</span>
</el-form-item>
<el-form-item>
<span class="cookie-note">说明：聚名登录成功后会自动保存 Cookie，并尝试自动联名登录聚查。</span>
<el-button plain :loading="linkingJucha" style="margin-left: 12px" @click="loginJucha">重新联名登录聚查</el-button>
</el-form-item>
</el-form>
<el-form label-width="110px" class="cookie-form">
<el-form-item label="Cookie 文件">
<el-upload
:auto-upload="false"
:show-file-list="true"
:limit="1"
accept=".pkl,.pickle,.json,.txt"
:on-change="onCookieFileChange"
:on-remove="onCookieFileRemove"
>
<template #trigger>
<el-button>选择 Cookie 文件</el-button>
</template>
</el-upload>
<el-button
type="primary"
plain
:loading="uploadingCookie"
:disabled="!cookieFile"
style="margin-left: 12px"
@click="uploadCookie"
>
上传 Cookie 兜底
</el-button>
</el-form-item>
<el-form-item label="Cookie 路径">
<span class="cookie-path">{{ status.cookie_file || "-" }}</span>
</el-form-item>
<el-form-item>
<span class="cookie-note">如账号密码登录受验证码或风控影响，可临时上传已登录 Cookie 作为兜底方式。</span>
</el-form-item>
</el-form>
<el-form label-width="110px" class="juming-form">
<el-form-item label="采集类型">
<el-radio-group v-model="form.mode">
<el-radio-button label="delete_list">过期删除</el-radio-button>
<el-radio-button label="fixed_price">一口价</el-radio-button>
</el-radio-group>
</el-form-item>
<template v-if="form.mode === 'delete_list'">
<el-form-item label="起始日期">
<el-date-picker v-model="form.crawl_date" type="date" value-format="YYYY-MM-DD" />
<el-checkbox v-model="form.auto_date" style="margin-left: 12px">自动追加到今天+4天</el-checkbox>
<span class="updated-at" style="margin-left: 12px">
{{ preferencesSaving ? "偏好保存中..." : preferencesSavedAt ? `偏好已保存：${preferencesSavedAt}` : "偏好将自动保存" }}
</span>
</el-form-item>
</template>
<template v-else>
<el-form-item label="起始页码">
<el-input-number v-model="form.page_start" :min="1" />
</el-form-item>
<el-form-item label="每页数量">
<el-input-number v-model="form.page_size" :min="1" :max="1000" />
</el-form-item>
<el-form-item label="抓取页数">
<el-input-number v-model="form.page_count" :min="1" :max="20" />
</el-form-item>
</template>
<el-form-item>
<el-button type="primary" :loading="running" :disabled="!status.cookie_ready" @click="runCrawl">开始采集并入库</el-button>
<el-button plain :loading="loadingStatus" @click="refreshAll">刷新状态</el-button>
<el-switch v-model="autoRefresh" inline-prompt active-text="自动刷新" inactive-text="手动" />
<span class="updated-at">最近同步：{{ lastUpdatedAt || "暂无" }}</span>
<span v-if="!status.cookie_ready" class="crawl-tip">请先完成聚名登录或上传有效 Cookie，登录成功后此按钮会自动解锁。</span>
</el-form-item>
</el-form>
<el-alert
v-if="latestCompletedSummary"
:closable="false"
show-icon
type="success"
style="margin-bottom: 16px"
:title="latestCompletedSummary"
/>
<el-row :gutter="16" style="margin-bottom: 16px">
<el-col :xs="12" :md="6"><el-statistic title="总任务数" :value="tasks.length" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="运行中任务" :value="runningTaskCount" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="成功任务" :value="completedTaskCount" /></el-col>
<el-col :xs="12" :md="6"><el-statistic title="失败任务" :value="failedTaskCount" /></el-col>
</el-row>
<el-row :gutter="12" style="margin-bottom: 16px">
<el-col :xs="24" :md="12">
<div class="summary-card">
<div class="summary-title">当前采集阶段</div>
<div class="summary-value">{{ currentPhaseLabel }}</div>
<div class="summary-note">{{ currentPhaseDetail }}</div>
</div>
</el-col>
<el-col :xs="24" :md="12">
<div class="summary-card">
<div class="summary-title">当前任务概览</div>
<div class="summary-value">{{ currentTaskSummary }}</div>
<div class="summary-note">离开页面后回来，仍可继续查看任务进度和运行日志。</div>
</div>
</el-col>
</el-row>
<el-table :data="tasks" border highlight-current-row @current-change="handleCurrentTaskChange">
<el-table-column label="状态" min-width="100">
<template #default="{ row }">
<el-tag :type="statusTagType(row.status)" effect="dark">{{ statusText(row.status) }}</el-tag>
</template>
</el-table-column>
<el-table-column label="阶段" min-width="110">
<template #default="{ row }">
<el-tag :type="phaseTagType(row.phase_label, row.status)" plain>{{ row.phase_label || "-" }}</el-tag>
</template>
</el-table-column>
<el-table-column prop="mode" label="采集类型" min-width="110">
<template #default="{ row }">
{{ row.mode === "fixed_price" ? "一口价" : "过期删除" }}
</template>
</el-table-column>
<el-table-column prop="message" label="任务信息" min-width="280" show-overflow-tooltip />
<el-table-column prop="created_at" label="创建时间" min-width="160" />
<el-table-column prop="started_at" label="开始时间" min-width="160" />
<el-table-column prop="completed_at" label="完成时间" min-width="160" />
<el-table-column label="操作" min-width="110" fixed="right">
<template #default="{ row }">
<el-button
v-if="row.status === 'running' || row.status === 'queued'"
type="danger"
link
@click="stopTask(row.task_id)"
>
停止
</el-button>
<el-button
v-if="row.status === 'failed' || row.status === 'queued'"
type="primary"
link
@click="retryTask(row.task_id)"
>
重试
</el-button>
<span v-else>-</span>
</template>
</el-table-column>
</el-table>
<div class="task-detail">
<div class="task-detail-header">
<div>
<div class="task-detail-title">任务日志控制台</div>
<div class="task-detail-subtitle">
{{ selectedTask ? `当前查看任务：${selectedTask.task_id}` : "请选择一条任务记录查看日志" }}
</div>
</div>
<div class="task-detail-meta">
<span v-if="selectedTask">状态：{{ statusText(selectedTask.status) }}</span>
<span v-if="selectedTask?.phase_label">阶段：{{ selectedTask.phase_label }}</span>
<span v-if="selectedTask?.result">抓取：{{ selectedTask.result.domains_found || 0 }}</span>
</div>
</div>
<pre class="log-console">{{ selectedTaskLogs }}</pre>
</div>
<el-row v-if="selectedTask?.result" :gutter="16" style="margin-bottom: 16px">
<el-col :xs="12" :md="4"><el-statistic title="抓取总数" :value="selectedTask.result.domains_found || 0" /></el-col>
<el-col :xs="12" :md="4"><el-statistic title="有效域名" :value="selectedTask.result.stats?.valid || 0" /></el-col>
<el-col :xs="12" :md="4"><el-statistic title="新增入库" :value="selectedTask.result.stats?.added || 0" /></el-col>
<el-col :xs="12" :md="4"><el-statistic title="已存在" :value="selectedTask.result.stats?.exists || 0" /></el-col>
<el-col :xs="12" :md="4"><el-statistic title="无效" :value="selectedTask.result.stats?.invalid || 0" /></el-col>
<el-col :xs="12" :md="4"><el-statistic title="失败" :value="selectedTask.result.stats?.failed || 0" /></el-col>
</el-row>
<el-table v-if="selectedTimeline.length" :data="selectedTimeline" border style="margin-bottom: 16px">
<el-table-column prop="label" label="阶段" min-width="180" />
<el-table-column prop="count" label="数量" width="100" />
</el-table>
<el-table v-if="selectedSampleDomains.length" :data="selectedSampleDomains" border>
<el-table-column prop="domain" label="样本域名" min-width="260" />
</el-table>
</PageCard>
</template>
<script setup lang="ts">
import { computed, onBeforeUnmount, onMounted, reactive, ref, watch } from "vue";
import { ElMessage } from "element-plus";
import type { UploadFile } from "element-plus";
import PageCard from "@/components/PageCard.vue";
import { jumingApi, settingsApi } from "@/api/modules";
const loadingStatus = ref(false);
const running = ref(false);
const uploadingCookie = ref(false);
const loggingIn = ref(false);
const linkingJucha = ref(false);
const cookieFile = ref<File | null>(null);
const autoRefresh = ref(true);
const lastUpdatedAt = ref("");
const preferencesSavedAt = ref("");
const preferencesSaving = ref(false);
const tasks = ref<any[]>([]);
const selectedTaskId = ref("");
let timer: number | null = null;
let savePreferencesTimer: number | null = null;
let suppressPreferenceSave = false;
const status = reactive<Record<string, any>>({
cookie_ready: false,
cookie_storage: "missing",
cookie_file: "",
cookie_count: 0,
jucha: {
cookie_ready: false,
cookie_file: ""
},
linked_jucha: {
attempted: false,
ok: false,
message: ""
},
defaults: {
mode: "delete_list",
page_start: 1,
page_size: 500,
page_count: 1,
crawl_date: "",
auto_date: true
}
});
const loginForm = reactive({
email: "",
password: ""
});
const form = reactive({
mode: "delete_list",
page_start: 1,
page_size: 500,
page_count: 1,
crawl_date: "",
auto_date: true
});
const runningTaskCount = computed(() => tasks.value.filter((item) => item.status === "running" || item.status === "queued").length);
const completedTaskCount = computed(() => tasks.value.filter((item) => item.status === "completed").length);
const failedTaskCount = computed(() => tasks.value.filter((item) => item.status === "failed").length);
const selectedTask = computed(() => tasks.value.find((item) => item.task_id === selectedTaskId.value) || tasks.value[0] || null);
const currentPhaseLabel = computed(() => selectedTask.value?.phase_label || "暂无任务");
const currentPhaseDetail = computed(() => selectedTask.value?.message || "当前还没有聚名采集任务在执行。");
const currentTaskSummary = computed(() => {
if (!selectedTask.value) {
return "等待创建";
}
return `${selectedTask.value.mode === "fixed_price" ? "一口价" : "过期删除"} / ${statusText(selectedTask.value.status)}`;
});
const selectedTaskLogs = computed(() => {
const lines = selectedTask.value?.logs || [];
return lines.length ? lines.join("\n") : "暂无日志输出";
});
const selectedTimeline = computed(() => {
const payload = selectedTask.value?.result;
if (!payload) return [];
if (Array.isArray(payload.dates)) {
return payload.dates.map((item: any) => ({ label: item.date, count: item.count }));
}
if (Array.isArray(payload.pages)) {
return payload.pages.map((item: any) => ({ label: `第 ${item.page} 页`, count: item.count }));
}
return [];
});
const selectedSampleDomains = computed(() => (selectedTask.value?.result?.sample_domains || []).map((domain: string) => ({ domain })));
const latestCompletedSummary = computed(() => {
const latestCompleted = tasks.value.find((item) => item.status === "completed" && item.result?.stats);
if (!latestCompleted) return "";
const result = latestCompleted.result;
const stats = result.stats || {};
return `最近完成任务：抓取 ${result.domains_found || 0} 个域名，新增 ${stats.added || 0} 个，已存在 ${stats.exists || 0} 个，无效 ${stats.invalid || 0} 个`;
});
const maskAccount = (value: string) => {
const text = String(value || "").trim();
if (!text) return "";
const [name, domain] = text.split("@");
if (!domain) {
if (text.length <= 4) return `${text[0] || ""}***`;
return `${text.slice(0, 2)}***${text.slice(-2)}`;
}
const visible = name.length <= 2 ? `${name[0] || ""}*` : `${name.slice(0, 2)}***${name.slice(-1)}`;
return `${visible}@${domain}`;
};
const savedJumingAccount = computed(() => maskAccount(loginForm.email));
const savedJumingSummary = computed(() => {
if (!savedJumingAccount.value) {
return "当前未保存聚名账号，首次登录成功后会自动保存，后续进入页面会自动回填。";
}
return `已保存聚名账号：${savedJumingAccount.value}，运营回来时能直接确认这台服务器已录入账号。`;
});
const juchaSummary = computed(() => {
const linked = status.linked_jucha || {};
if (status.jucha?.cookie_ready) {
return `聚查登录态已就绪：${status.jucha?.cookie_file || "-"}${linked.message ? `，最近结果：${linked.message}` : ""}`;
}
return linked.message || "尚未检测到聚查登录态，聚名登录成功后可自动联名，也可手动重试。";
});
const statusText = (value: string) => {
if (value === "queued") return "排队中";
if (value === "running") return "运行中";
if (value === "completed") return "已完成";
if (value === "failed") return "失败";
if (value === "stopped") return "已停止";
return value || "-";
};
const statusTagType = (value: string) => {
if (value === "running") return "warning";
if (value === "completed") return "success";
if (value === "failed") return "danger";
if (value === "stopped") return "info";
return "";
};
const phaseTagType = (phaseLabel: string, statusValue: string) => {
if (statusValue === "failed") return "danger";
if (statusValue === "completed") return "success";
if (statusValue === "stopped") return "info";
if (["抓取中", "入库中"].includes(phaseLabel)) return "warning";
if (["启动中", "停止中", "排队中"].includes(phaseLabel)) return "";
return "info";
};
const clearRefreshTimer = () => {
if (timer) {
window.clearInterval(timer);
timer = null;
}
};
const ensureRefreshTimer = () => {
clearRefreshTimer();
if (autoRefresh.value) {
timer = window.setInterval(() => {
refreshAll(false);
}, 5000);
}
};
const applyDefaults = () => {
const defaults = status.defaults || {};
suppressPreferenceSave = true;
form.mode = defaults.mode || "delete_list";
form.page_start = defaults.page_start || 1;
form.page_size = defaults.page_size || 500;
form.page_count = defaults.page_count || 1;
form.crawl_date = defaults.crawl_date || "";
form.auto_date = defaults.auto_date ?? true;
window.setTimeout(() => {
suppressPreferenceSave = false;
}, 0);
};
const loadCredentials = async () => {
try {
const response = await settingsApi.getCredentials();
loginForm.email = response.data?.juming?.email || "";
loginForm.password = response.data?.juming?.password || "";
} catch (error: any) {
ElMessage.error(error?.message || "读取聚名账号失败");
}
};
const loadStatus = async () => {
const response = await jumingApi.status();
Object.assign(status, response.data || {});
applyDefaults();
};
const persistPreferences = async () => {
if (suppressPreferenceSave) return;
preferencesSaving.value = true;
try {
const response = await jumingApi.updatePreferences({
mode: form.mode,
page_start: form.page_start,
page_size: form.page_size,
page_count: form.page_count,
crawl_date: form.crawl_date,
auto_date: form.auto_date
});
status.defaults = response.data || status.defaults;
preferencesSavedAt.value = new Date().toLocaleString("zh-CN", { hour12: false });
} catch (error: any) {
ElMessage.error(error?.message || "保存聚名采集偏好失败");
} finally {
preferencesSaving.value = false;
}
};
const schedulePersistPreferences = () => {
if (suppressPreferenceSave) return;
if (savePreferencesTimer) {
window.clearTimeout(savePreferencesTimer);
}
savePreferencesTimer = window.setTimeout(() => {
persistPreferences();
}, 500);
};
const loadTasks = async () => {
const response = await jumingApi.tasks();
tasks.value = response.data || [];
if (!selectedTaskId.value && tasks.value.length) {
selectedTaskId.value = tasks.value[0].task_id;
} else if (selectedTaskId.value) {
const exists = tasks.value.some((item) => item.task_id === selectedTaskId.value);
if (!exists && tasks.value.length) {
selectedTaskId.value = tasks.value[0].task_id;
}
}
};
const refreshAll = async (showError = true) => {
loadingStatus.value = true;
try {
await Promise.all([loadStatus(), loadTasks()]);
lastUpdatedAt.value = new Date().toLocaleString("zh-CN", { hour12: false });
} catch (error: any) {
if (showError) {
ElMessage.error(error?.message || "刷新聚名状态失败");
}
} finally {
loadingStatus.value = false;
}
};
const runCrawl = async () => {
running.value = true;
try {
const response = await jumingApi.crawl({
mode: form.mode,
page_start: form.page_start,
page_size: form.page_size,
page_count: form.page_count,
crawl_date: form.crawl_date,
auto_date: form.auto_date
});
ElMessage.success(response.message || "聚名采集任务已创建");
await loadTasks();
if (response.data?.task_id) {
selectedTaskId.value = response.data.task_id;
}
} catch (error: any) {
ElMessage.error(error?.message || "创建聚名采集任务失败");
} finally {
running.value = false;
}
};
const onCookieFileChange = (file: UploadFile) => {
cookieFile.value = file.raw || null;
};
const onCookieFileRemove = () => {
cookieFile.value = null;
};
const uploadCookie = async () => {
if (!cookieFile.value) {
ElMessage.warning("请先选择聚名 Cookie 文件");
return;
}
uploadingCookie.value = true;
try {
const formData = new FormData();
formData.append("file", cookieFile.value, cookieFile.value.name);
const response = await jumingApi.uploadCookie(formData);
ElMessage.success(response.message || "聚名 Cookie 已上传");
cookieFile.value = null;
await refreshAll(false);
} catch (error: any) {
ElMessage.error(error?.message || "聚名 Cookie 上传失败");
} finally {
uploadingCookie.value = false;
}
};
const loginJuming = async () => {
if (!loginForm.email.trim() || !loginForm.password.trim()) {
ElMessage.warning("请输入聚名账号和密码");
return;
}
loggingIn.value = true;
try {
const response = await jumingApi.login({
email: loginForm.email.trim(),
password: loginForm.password
});
await settingsApi.updateCredentials({
juming: {
email: loginForm.email.trim(),
password: loginForm.password
}
});
ElMessage.success(response.message || "聚名登录成功");
await refreshAll(false);
} catch (error: any) {
ElMessage.error(error?.message || "聚名登录失败");
} finally {
loggingIn.value = false;
}
};
const loginJucha = async () => {
linkingJucha.value = true;
try {
const response = await jumingApi.loginJucha();
ElMessage.success(response.message || "聚查联名登录成功");
await refreshAll(false);
} catch (error: any) {
ElMessage.error(error?.message || "聚查联名登录失败");
} finally {
linkingJucha.value = false;
}
};
const retryTask = async (taskId: string) => {
try {
await jumingApi.retryTask(taskId);
ElMessage.success("聚名采集任务已重新加入队列");
await loadTasks();
selectedTaskId.value = taskId;
} catch (error: any) {
ElMessage.error(error?.message || "重试聚名采集任务失败");
}
};
const stopTask = async (taskId: string) => {
try {
await jumingApi.stopTask(taskId);
ElMessage.success("已发送停止请求");
await loadTasks();
selectedTaskId.value = taskId;
} catch (error: any) {
ElMessage.error(error?.message || "停止聚名采集任务失败");
}
};
const handleCurrentTaskChange = (row: any | null) => {
selectedTaskId.value = row?.task_id || "";
};
watch(autoRefresh, ensureRefreshTimer);
watch(
() => [form.mode, form.page_start, form.page_size, form.page_count, form.crawl_date, form.auto_date],
schedulePersistPreferences,
{ deep: false }
);
onMounted(async () => {
await Promise.all([refreshAll(), loadCredentials()]);
ensureRefreshTimer();
});
onBeforeUnmount(() => {
clearRefreshTimer();
if (savePreferencesTimer) {
window.clearTimeout(savePreferencesTimer);
savePreferencesTimer = null;
}
});
</script>
<style scoped>
.login-form {
margin-bottom: 12px;
}
.cookie-form {
margin-bottom: 12px;
}
.juming-form {
margin-bottom: 20px;
}
.cookie-path {
color: #475569;
word-break: break-all;
}
.cookie-note {
color: #64748b;
}
.login-status {
margin-left: 12px;
font-size: 13px;
font-weight: 500;
}
.login-status.success {
color: #16a34a;
}
.login-status.warning {
color: #d97706;
}
.crawl-tip {
margin-left: 12px;
color: #64748b;
font-size: 13px;
}
.updated-at {
color: #64748b;
font-size: 13px;
}
.summary-card {
border: 1px solid #dbe4ee;
border-radius: 14px;
padding: 14px 16px;
background: #f8fbff;
}
.summary-title {
color: #64748b;
font-size: 13px;
}
.summary-value {
margin: 8px 0 10px;
color: #0f172a;
font-size: 22px;
font-weight: 700;
}
.summary-note {
color: #64748b;
font-size: 12px;
line-height: 1.6;
}
.task-detail {
margin: 16px 0;
border-radius: 10px;
overflow: hidden;
border: 1px solid #1f2937;
background: #020617;
}
.task-detail-header {
display: flex;
justify-content: space-between;
gap: 12px;
align-items: center;
padding: 12px 16px;
background: #111827;
color: #e5e7eb;
}
.task-detail-title {
font-size: 15px;
font-weight: 600;
}
.task-detail-subtitle,
.task-detail-meta {
color: #94a3b8;
font-size: 12px;
}
.task-detail-meta {
display: flex;
gap: 12px;
flex-wrap: wrap;
}
.log-console {
margin: 0;
min-height: 260px;
max-height: 420px;
overflow: auto;
padding: 16px;
color: #d1fae5;
background: #020617;
font-size: 12px;
line-height: 1.7;
font-family: Consolas, Monaco, monospace;
white-space: pre-wrap;
word-break: break-word;
}
</style>

114
domain-web/src/views/sensitive-words/SensitiveWordsView.vue Normal file
View File

@@ -0,0 +1,114 @@
<template>
<PageCard title="敏感词配置" description="对齐旧桌面版主功能，支持加载、编辑、导入、导出和整库保存。">
<el-alert
:title="`当前共 ${total} 个敏感词`"
type="info"
:closable="false"
show-icon
style="margin-bottom: 16px"
/>
<el-input
v-model="text"
type="textarea"
:rows="22"
placeholder="请输入敏感词，一行一个"
style="margin-bottom: 16px"
/>
<div class="actions">
<el-button @click="triggerImport">导入</el-button>
<el-button @click="exportWords">导出</el-button>
<el-button @click="loadWords">加载</el-button>
<el-button type="primary" :loading="saving" @click="saveWords">保存</el-button>
<input ref="fileInput" type="file" accept=".txt,text/plain" class="hidden-input" @change="handleImport" />
</div>
</PageCard>
</template>
<script setup lang="ts">
import { onMounted, ref } from "vue";
import { ElMessage } from "element-plus";
import PageCard from "@/components/PageCard.vue";
import { sensitiveWordsApi } from "@/api/modules";
const text = ref("");
const total = ref(0);
const saving = ref(false);
const fileInput = ref<HTMLInputElement | null>(null);
const normalizeText = (value: string) =>
value
.split(/\r?\n/)
.map((item) => item.trim())
.filter(Boolean)
.join("\n");
const loadWords = async () => {
try {
const response = await sensitiveWordsApi.get();
text.value = response.data?.text || "";
total.value = Number(response.data?.total || 0);
} catch (error: any) {
ElMessage.error(error?.message || "加载敏感词失败");
}
};
const saveWords = async () => {
saving.value = true;
try {
const response = await sensitiveWordsApi.save({ text: normalizeText(text.value) });
text.value = response.data?.text || normalizeText(text.value);
total.value = Number(response.data?.total || 0);
ElMessage.success(response.message || "敏感词已保存");
} catch (error: any) {
ElMessage.error(error?.message || "保存敏感词失败");
} finally {
saving.value = false;
}
};
const triggerImport = () => {
fileInput.value?.click();
};
const handleImport = async (event: Event) => {
const input = event.target as HTMLInputElement;
const file = input.files?.[0];
if (!file) return;
text.value = normalizeText(await file.text());
total.value = text.value ? text.value.split("\n").length : 0;
ElMessage.success(`已导入 ${total.value} 个敏感词`);
input.value = "";
};
const exportWords = () => {
const content = normalizeText(text.value);
if (!content) {
ElMessage.warning("没有敏感词可导出");
return;
}
const blob = new Blob([`${content}\n`], { type: "text/plain;charset=utf-8" });
const url = window.URL.createObjectURL(blob);
const link = document.createElement("a");
link.href = url;
link.download = "sensitive_words.txt";
link.click();
window.URL.revokeObjectURL(url);
ElMessage.success("敏感词已导出");
};
onMounted(loadWords);
</script>
<style scoped>
.actions {
display: flex;
gap: 12px;
flex-wrap: wrap;
}
.hidden-input {
display: none;
}
</style>

249
domain-web/src/views/settings/SettingsView.vue
View File

@@ -8,8 +8,32 @@
<div class="settings-grid"> <div class="settings-grid">
<PageCard title="运行配置"> <PageCard title="运行配置">
<el-form label-position="top"> <el-form label-position="top">
<el-form-item label="检测线程数"> <el-alert
<el-input-number v-model="threadCount" :min="1" :max="64" /> :title="currentNodeCode ? `支持默认并发和节点单独覆盖；当前后台节点编码：${currentNodeCode}` : '支持默认并发和节点单独覆盖：未单独配置的机器自动走默认线程数。'"
type="info"
:closable="false"
show-icon
style="margin-bottom: 12px"
/>
<el-form-item label="默认检测线程数">
<el-input-number v-model="threadCount" :min="1" :max="256" />
</el-form-item>
<el-form-item label="节点独立线程覆盖">
<div class="node-thread-overrides">
<div v-if="!nodeThreadOverrides.length" class="node-thread-empty">
当前没有单独覆盖配置，所有节点都会走默认线程数。
</div>
<div v-for="(item, index) in nodeThreadOverrides" :key="`${item.node_code}-${index}`" class="node-thread-row">
<el-input v-model="item.node_code" placeholder="节点编码，例如 mainland-worker-01" />
<el-input-number v-model="item.thread_count" :min="1" :max="256" />
<el-button type="danger" plain @click="removeNodeThreadOverride(index)">删除</el-button>
</div>
<div class="node-thread-actions">
<el-button plain @click="addNodeThreadOverride">新增节点覆盖</el-button>
</div>
</div>
</el-form-item> </el-form-item>
<el-form-item label="Worker 运行模式"> <el-form-item label="Worker 运行模式">
@@ -26,6 +50,10 @@
<el-form-item label="API 服务名"> <el-form-item label="API 服务名">
<el-input v-model="runtimeSettings.api_service_name" /> <el-input v-model="runtimeSettings.api_service_name" />
</el-form-item> </el-form-item>
<el-form-item label="Sync Agent 服务名">
<el-input v-model="runtimeSettings.sync_agent_service_name" />
</el-form-item>
</el-form> </el-form>
</PageCard> </PageCard>
@@ -54,6 +82,47 @@
</el-form> </el-form>
</PageCard> </PageCard>
<PageCard title="第三方登录">
<el-form label-position="top">
<el-alert
:title="savedJuziseoSummary"
:type="savedJuziseoAccount ? 'info' : 'warning'"
:closable="false"
show-icon
style="margin-bottom: 12px"
/>
<el-alert
:title="juziseoStatus.cookie_ready ? `桔子SEO 登录态已就绪：${juziseoStatus.cookie_storage}，Cookie ${juziseoStatus.cookie_count || 0} 项` : '桔子SEO 尚未登录，旧桌面版已有该功能，Web 端现已补齐'"
:type="juziseoStatus.cookie_ready ? 'success' : 'warning'"
:closable="false"
show-icon
style="margin-bottom: 12px"
/>
<el-form-item label="桔子SEO账号">
<el-input v-model="juziseoForm.email" placeholder="请输入桔子SEO账号" />
</el-form-item>
<el-form-item label="桔子SEO密码">
<el-input v-model="juziseoForm.password" type="password" show-password placeholder="请输入桔子SEO密码" />
</el-form-item>
<el-form-item label="Cookie 路径">
<span class="credential-path">{{ juziseoStatus.cookie_file || "-" }}</span>
</el-form-item>
<el-form-item>
<el-button type="primary" :loading="juziseoLoggingIn" @click="loginJuziseo">登录桔子SEO</el-button>
<el-button @click="loadJuziseoStatus">刷新状态</el-button>
</el-form-item>
<div class="third-party-tip">
聚名登录入口在“聚名采集”页面；桔子SEO 登录入口已按旧桌面版补回到系统设置。
当前会同时保存账号和 Cookie，方便 Linux 长驻服务直接复用。
</div>
</el-form>
</PageCard>
<PageCard title="检测选项" class="span-2"> <PageCard title="检测选项" class="span-2">
<div class="detect-items"> <div class="detect-items">
<div v-for="(item, index) in detectItems" :key="item.key" class="detect-item"> <div v-for="(item, index) in detectItems" :key="item.key" class="detect-item">
@@ -111,7 +180,7 @@
</template> </template>
<script setup lang="ts"> <script setup lang="ts">
import { onMounted, ref } from "vue"; import { computed, onMounted, ref } from "vue";
import { ElMessage } from "element-plus"; import { ElMessage } from "element-plus";
import PageCard from "@/components/PageCard.vue"; import PageCard from "@/components/PageCard.vue";
import { settingsApi } from "@/api/modules"; import { settingsApi } from "@/api/modules";
@@ -130,6 +199,11 @@ type BackupRecord = {
backup_reason?: string; backup_reason?: string;
}; };
type NodeThreadOverrideRow = {
node_code: string;
thread_count: number;
};
const DETECT_LABELS: Record<string, string> = { const DETECT_LABELS: Record<string, string> = {
detect_register: "检查注册", detect_register: "检查注册",
detect_baidu_site: "百度 site 查询", detect_baidu_site: "百度 site 查询",
@@ -143,6 +217,8 @@ const DETECT_LABELS: Record<string, string> = {
const loading = ref(true); const loading = ref(true);
const threadCount = ref(2); const threadCount = ref(2);
const currentNodeCode = ref("");
const nodeThreadOverrides = ref<NodeThreadOverrideRow[]>([]);
const detectItems = ref<DetectItem[]>([]); const detectItems = ref<DetectItem[]>([]);
const backups = ref<BackupRecord[]>([]); const backups = ref<BackupRecord[]>([]);
const proxyConfig = ref<Record<string, any>>({ const proxyConfig = ref<Record<string, any>>({
@@ -155,9 +231,53 @@ const proxyUrlsText = ref("");
const runtimeSettings = ref({ const runtimeSettings = ref({
worker_mode: "windows-local", worker_mode: "windows-local",
worker_service_name: "domaincheck-worker", worker_service_name: "domaincheck-worker",
api_service_name: "domaincheck-api" api_service_name: "domaincheck-api",
sync_agent_service_name: "domaincheck-sync-agent"
}); });
const importInput = ref<HTMLInputElement | null>(null); const importInput = ref<HTMLInputElement | null>(null);
const juziseoLoggingIn = ref(false);
const juziseoForm = ref({
email: "",
password: ""
});
const juziseoStatus = ref({
cookie_ready: false,
cookie_storage: "missing",
cookie_file: "",
cookie_count: 0
});
const maskAccount = (value: string) => {
const text = String(value || "").trim();
if (!text) return "";
const [name, domain] = text.split("@");
if (!domain) {
if (text.length <= 4) return `${text[0] || ""}***`;
return `${text.slice(0, 2)}***${text.slice(-2)}`;
}
const visible = name.length <= 2 ? `${name[0] || ""}*` : `${name.slice(0, 2)}***${name.slice(-1)}`;
return `${visible}@${domain}`;
};
const savedJuziseoAccount = computed(() => maskAccount(juziseoForm.value.email));
const savedJuziseoSummary = computed(() => {
if (!savedJuziseoAccount.value) {
return "当前未保存桔子SEO账号，首次登录后会自动持久化到本地 JSON 与 Redis。";
}
return `已保存桔子SEO账号：${savedJuziseoAccount.value}，运营无需每次重新确认是否已录入。`;
});
const loadCredentials = async () => {
try {
const response = await settingsApi.getCredentials();
juziseoForm.value = {
email: response.data?.juziseo?.email || "",
password: response.data?.juziseo?.password || ""
};
} catch {
ElMessage.error("读取已保存账号失败");
}
};
const normalizeDetectItems = (detectOptions: Record<string, boolean | string[]>) => { const normalizeDetectItems = (detectOptions: Record<string, boolean | string[]>) => {
const order = Array.isArray(detectOptions.detect_order) ? [...(detectOptions.detect_order as string[])] : []; const order = Array.isArray(detectOptions.detect_order) ? [...(detectOptions.detect_order as string[])] : [];
@@ -186,6 +306,27 @@ const moveItem = (index: number, offset: -1 | 1) => {
detectItems.value = items; detectItems.value = items;
}; };
const normalizeNodeThreadOverrides = (payload: Record<string, number | string>) => {
nodeThreadOverrides.value = Object.entries(payload || {})
.map(([node_code, thread_count]) => ({
node_code: String(node_code || "").trim(),
thread_count: Number(thread_count || 0)
}))
.filter((item) => item.node_code && item.thread_count > 0)
.sort((a, b) => a.node_code.localeCompare(b.node_code));
};
const addNodeThreadOverride = () => {
nodeThreadOverrides.value.push({
node_code: "",
thread_count: threadCount.value || 2
});
};
const removeNodeThreadOverride = (index: number) => {
nodeThreadOverrides.value.splice(index, 1);
};
const loadBackups = async () => { const loadBackups = async () => {
try { try {
const response = await settingsApi.getSettingsBackups(); const response = await settingsApi.getSettingsBackups();
@@ -195,11 +336,27 @@ const loadBackups = async () => {
} }
}; };
const loadJuziseoStatus = async () => {
try {
const response = await settingsApi.getJuziseoStatus();
juziseoStatus.value = {
cookie_ready: Boolean(response.data?.cookie_ready),
cookie_storage: response.data?.cookie_storage || "missing",
cookie_file: response.data?.cookie_file || "",
cookie_count: Number(response.data?.cookie_count || 0)
};
} catch {
ElMessage.error("读取桔子SEO状态失败");
}
};
const loadSettings = async () => { const loadSettings = async () => {
loading.value = true; loading.value = true;
try { try {
const response = await settingsApi.getSettings(); const response = await settingsApi.getSettings();
threadCount.value = response.data.thread_count; threadCount.value = response.data.thread_count;
currentNodeCode.value = response.data.current_node_code || "";
normalizeNodeThreadOverrides(response.data.node_thread_counts || {});
normalizeDetectItems(response.data.detect_options || {}); normalizeDetectItems(response.data.detect_options || {});
proxyConfig.value = { proxyConfig.value = {
@@ -213,7 +370,8 @@ const loadSettings = async () => {
runtimeSettings.value = { runtimeSettings.value = {
worker_mode: response.data.runtime_settings?.worker_mode || "windows-local", worker_mode: response.data.runtime_settings?.worker_mode || "windows-local",
worker_service_name: response.data.runtime_settings?.worker_service_name || "domaincheck-worker", worker_service_name: response.data.runtime_settings?.worker_service_name || "domaincheck-worker",
api_service_name: response.data.runtime_settings?.api_service_name || "domaincheck-api" api_service_name: response.data.runtime_settings?.api_service_name || "domaincheck-api",
sync_agent_service_name: response.data.runtime_settings?.sync_agent_service_name || "domaincheck-sync-agent"
}; };
} catch { } catch {
ElMessage.error("读取系统设置失败"); ElMessage.error("读取系统设置失败");
@@ -235,8 +393,19 @@ const buildSettingsPayload = () => {
detectOptions[item.key] = item.enabled; detectOptions[item.key] = item.enabled;
}); });
const nodeThreadCounts = nodeThreadOverrides.value.reduce<Record<string, number>>((result, item) => {
const nodeCode = String(item.node_code || "").trim();
const threadCountValue = Number(item.thread_count || 0);
if (!nodeCode || threadCountValue <= 0) {
return result;
}
result[nodeCode] = threadCountValue;
return result;
}, {});
return { return {
thread_count: threadCount.value, thread_count: threadCount.value,
node_thread_counts: nodeThreadCounts,
detect_options: detectOptions, detect_options: detectOptions,
proxy_config: { proxy_config: {
proxy_enable: Boolean(proxyConfig.value.proxy_enable), proxy_enable: Boolean(proxyConfig.value.proxy_enable),
@@ -258,6 +427,32 @@ const saveSettings = async () => {
} }
}; };
const loginJuziseo = async () => {
if (!juziseoForm.value.email.trim() || !juziseoForm.value.password.trim()) {
ElMessage.warning("请输入桔子SEO账号和密码");
return;
}
juziseoLoggingIn.value = true;
try {
const response = await settingsApi.loginJuziseo({
email: juziseoForm.value.email.trim(),
password: juziseoForm.value.password
});
await settingsApi.updateCredentials({
juziseo: {
email: juziseoForm.value.email.trim(),
password: juziseoForm.value.password
}
});
ElMessage.success(response.message || "桔子SEO登录成功");
await loadJuziseoStatus();
} catch (error: any) {
ElMessage.error(error?.message || "桔子SEO登录失败");
} finally {
juziseoLoggingIn.value = false;
}
};
const createBackup = async () => { const createBackup = async () => {
try { try {
const response = await settingsApi.backupSettings(); const response = await settingsApi.backupSettings();
@@ -314,17 +509,53 @@ const downloadBackup = (filename: string) => {
}; };
onMounted(async () => { onMounted(async () => {
await Promise.all([loadSettings(), loadBackups()]); await Promise.all([loadSettings(), loadBackups(), loadJuziseoStatus(), loadCredentials()]);
}); });
</script> </script>
<style scoped> <style scoped>
.credential-path {
color: #475569;
word-break: break-all;
}
.third-party-tip {
color: #64748b;
line-height: 1.6;
}
.settings-grid { .settings-grid {
display: grid; display: grid;
grid-template-columns: repeat(auto-fit, minmax(320px, 1fr)); grid-template-columns: repeat(auto-fit, minmax(320px, 1fr));
gap: 16px; gap: 16px;
} }
.node-thread-overrides {
display: flex;
flex-direction: column;
gap: 10px;
}
.node-thread-row {
display: grid;
grid-template-columns: minmax(240px, 1fr) 140px auto;
gap: 10px;
align-items: center;
}
.node-thread-actions {
display: flex;
justify-content: flex-start;
}
.node-thread-empty {
padding: 10px 12px;
border: 1px dashed var(--el-border-color);
border-radius: 10px;
color: #64748b;
background: var(--el-fill-color-lighter);
}
.span-2 { .span-2 {
grid-column: 1 / -1; grid-column: 1 / -1;
} }
@@ -379,6 +610,12 @@ onMounted(async () => {
flex-wrap: wrap; flex-wrap: wrap;
} }
@media (max-width: 768px) {
.node-thread-row {
grid-template-columns: 1fr;
}
}
.backup-toolbar { .backup-toolbar {
display: flex; display: flex;
justify-content: space-between; justify-content: space-between;

1
domain-web/vite.config.ts
View File

@@ -10,6 +10,7 @@ export default defineConfig({
} }
}, },
build: { build: {
emptyOutDir: true,
chunkSizeWarningLimit: 800, chunkSizeWarningLimit: 800,
rollupOptions: { rollupOptions: {
output: { output: {

3
domainCheck/app/config.py
View File

@@ -69,6 +69,9 @@ class Config:
REDIS_PORT = int(os.getenv('REDIS_PORT', 6379)) REDIS_PORT = int(os.getenv('REDIS_PORT', 6379))
REDIS_PASSWORD = os.getenv('REDIS_PASSWORD', '') REDIS_PASSWORD = os.getenv('REDIS_PASSWORD', '')
REDIS_DB = int(os.getenv('REDIS_DB', 0)) REDIS_DB = int(os.getenv('REDIS_DB', 0))
NODE_CODE = os.getenv('NODE_CODE', 'mainland-worker-01')
NODE_REGION = os.getenv('NODE_REGION', 'mainland')
NODE_ROLE = os.getenv('NODE_ROLE', 'worker')
# 聚名网配置 # 聚名网配置
JUMING_COOKIE = os.getenv('JUMING_COOKIE', '') JUMING_COOKIE = os.getenv('JUMING_COOKIE', '')

45
domainCheck/app/detectors/wayback_detector.py
View File

@@ -221,7 +221,7 @@ class WaybackDetector(BaseDetector):
except Exception: except Exception:
pass pass
def _fetch_cdx_records(self, domain, limit=None, fast_latest=False): def _fetch_cdx_records_with_meta(self, domain, limit=None, fast_latest=False):
response = None response = None
try: try:
params = { params = {
@@ -242,7 +242,11 @@ class WaybackDetector(BaseDetector):
) )
if response.status_code != 200: if response.status_code != 200:
self._log_warning(f"获取快照记录失败: {response.status_code}") self._log_warning(f"获取快照记录失败: {response.status_code}")
return [] return {
'records': [],
'error': f"HTTP {response.status_code}",
'status_code': response.status_code,
}
records = [] records = []
seen = set() seen = set()
for raw_line in response.iter_lines(decode_unicode=True): for raw_line in response.iter_lines(decode_unicode=True):
@@ -256,10 +260,18 @@ class WaybackDetector(BaseDetector):
continue continue
seen.add(timestamp) seen.add(timestamp)
records.append({'timestamp': timestamp, 'digest': digest}) records.append({'timestamp': timestamp, 'digest': digest})
return records return {
'records': records,
'error': None,
'status_code': response.status_code,
}
except Exception as e: except Exception as e:
self._handle_exception(e, domain) self._handle_exception(e, domain)
return [] return {
'records': [],
'error': str(e),
'status_code': None,
}
finally: finally:
try: try:
if response is not None: if response is not None:
@@ -267,6 +279,9 @@ class WaybackDetector(BaseDetector):
except Exception: except Exception:
pass pass
def _fetch_cdx_records(self, domain, limit=None, fast_latest=False):
return self._fetch_cdx_records_with_meta(domain, limit=limit, fast_latest=fast_latest).get('records', [])
def get_latest_snapshot_record(self, domain): def get_latest_snapshot_record(self, domain):
records = self._fetch_cdx_records(domain, limit=-1, fast_latest=True) records = self._fetch_cdx_records(domain, limit=-1, fast_latest=True)
return records[0] if records else None return records[0] if records else None
@@ -391,7 +406,11 @@ class WaybackDetector(BaseDetector):
def scan_snapshots(self, domain, sensitive_words=None, stop_on_first_hit=True): def scan_snapshots(self, domain, sensitive_words=None, stop_on_first_hit=True):
sensitive_words = sensitive_words or config.load_sensitive_words() sensitive_words = sensitive_words or config.load_sensitive_words()
latest_record = self.get_latest_snapshot_record(domain) request_errors = []
latest_fetch = self._fetch_cdx_records_with_meta(domain, limit=-1, fast_latest=True)
latest_record = (latest_fetch.get('records') or [None])[0]
if latest_fetch.get('error'):
request_errors.append(f"latest_cdx: {latest_fetch.get('error')}")
latest_timestamp = (latest_record or {}).get('timestamp') latest_timestamp = (latest_record or {}).get('timestamp')
latest_digest = (latest_record or {}).get('digest', '') latest_digest = (latest_record or {}).get('digest', '')
matched_word = None matched_word = None
@@ -444,7 +463,19 @@ class WaybackDetector(BaseDetector):
else: else:
failed_snapshot_count += 1 failed_snapshot_count += 1
records = sorted(self.get_snapshot_records(domain), key=lambda item: item.get('timestamp', ''), reverse=True) cached_records = self._load_cached_records(domain)
if cached_records is not None:
records = cached_records
else:
records_fetch = self._fetch_cdx_records_with_meta(domain)
if records_fetch.get('error'):
request_errors.append(f"records_cdx: {records_fetch.get('error')}")
records = records_fetch.get('records') or []
if records:
self._save_cached_records(domain, records)
self._save_cached_timestamps(domain, [item['timestamp'] for item in records])
records = sorted(records, key=lambda item: item.get('timestamp', ''), reverse=True)
years = sorted({int(item['timestamp'][:4]) for item in records if len(item.get('timestamp', '')) >= 4}) years = sorted({int(item['timestamp'][:4]) for item in records if len(item.get('timestamp', '')) >= 4})
checked_snapshot_count = len(records) checked_snapshot_count = len(records)
pending_records = [] pending_records = []
@@ -534,6 +565,8 @@ class WaybackDetector(BaseDetector):
'unique_title_count': unique_title_count, 'unique_title_count': unique_title_count,
'duplicate_title_skipped': duplicate_title_skipped, 'duplicate_title_skipped': duplicate_title_skipped,
'digest_duplicate_skipped': digest_duplicate_skipped, 'digest_duplicate_skipped': digest_duplicate_skipped,
'request_error_count': len(request_errors),
'request_errors': request_errors,
'elapsed_seconds': round(time.time() - started_at, 2), 'elapsed_seconds': round(time.time() - started_at, 2),
} }

492
domainCheck/app/utils/database.py
View File

@@ -9,9 +9,11 @@
''' '''
import psycopg2 import psycopg2
from psycopg2.extras import Json
import json import json
import redis import redis
import threading import threading
import socket
from loguru import logger from loguru import logger
from app.config import config from app.config import config
from app.utils.status_codes import ( from app.utils.status_codes import (
@@ -19,6 +21,7 @@ from app.utils.status_codes import (
DETECT_STATUS_COMPLETED, DETECT_STATUS_COMPLETED,
DETECT_STATUS_FAILED, DETECT_STATUS_FAILED,
DETECT_STATUS_PENDING, DETECT_STATUS_PENDING,
DETECT_STATUS_RUNNING,
REGISTER_STATUS_AVAILABLE, REGISTER_STATUS_AVAILABLE,
REGISTER_STATUS_REGISTERED, REGISTER_STATUS_REGISTERED,
REVIEW_STATUS_PENDING, REVIEW_STATUS_PENDING,
@@ -114,7 +117,7 @@ class Database:
self.use_bloom_filter = True self.use_bloom_filter = True
else: else:
# 如果命令不存在，说明 Redis 没有加载布隆过滤器模块 # 如果命令不存在，说明 Redis 没有加载布隆过滤器模块
logger.warning(f"Redis 布隆过滤器不可用: {e}，将使用普通缓存") logger.info(f"Redis 布隆过滤器不可用: {e}，将使用普通缓存")
self.use_bloom_filter = False self.use_bloom_filter = False
except Exception as e: except Exception as e:
logger.warning(f"初始化布隆过滤器失败: {e}") logger.warning(f"初始化布隆过滤器失败: {e}")
@@ -367,9 +370,465 @@ class Database:
pass pass
return False return False
finally: finally:
# 将连接放回连接池
self.close(conn, cur) self.close(conn, cur)
def ensure_cluster_runtime_tables(self):
sql = """
CREATE TABLE IF NOT EXISTS detect_worker_nodes (
node_code VARCHAR(64) PRIMARY KEY,
region VARCHAR(32) NOT NULL DEFAULT 'unknown',
role VARCHAR(32) NOT NULL DEFAULT 'worker',
hostname VARCHAR(255) NOT NULL DEFAULT '',
ip VARCHAR(64) NOT NULL DEFAULT '',
status VARCHAR(32) NOT NULL DEFAULT 'unknown',
worker_version VARCHAR(32) NOT NULL DEFAULT '',
current_load INTEGER NOT NULL DEFAULT 0,
metadata_json JSONB,
last_heartbeat_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
update_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS detect_jobs (
id BIGSERIAL PRIMARY KEY,
job_code VARCHAR(64) NOT NULL UNIQUE,
source VARCHAR(64) NOT NULL DEFAULT 'manual',
plan_hash VARCHAR(128) NOT NULL DEFAULT '',
status VARCHAR(32) NOT NULL DEFAULT 'pending',
remark TEXT NOT NULL DEFAULT '',
created_by VARCHAR(64) NOT NULL DEFAULT '',
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
started_at TIMESTAMP,
finished_at TIMESTAMP
);
CREATE TABLE IF NOT EXISTS detect_job_items (
id BIGSERIAL PRIMARY KEY,
job_id BIGINT NOT NULL REFERENCES detect_jobs(id) ON DELETE CASCADE,
domain_id BIGINT NOT NULL,
status VARCHAR(32) NOT NULL DEFAULT 'pending',
claimed_by VARCHAR(64) NOT NULL DEFAULT '',
claim_token VARCHAR(64) NOT NULL DEFAULT '',
lease_expires_at TIMESTAMP,
attempt_count INTEGER NOT NULL DEFAULT 0,
last_error TEXT NOT NULL DEFAULT '',
result_version VARCHAR(64) NOT NULL DEFAULT '',
started_at TIMESTAMP,
finished_at TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
create_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uq_detect_job_items_job_domain UNIQUE (job_id, domain_id)
);
CREATE INDEX IF NOT EXISTS idx_detect_job_items_status_lease
ON detect_job_items(status, lease_expires_at);
CREATE TABLE IF NOT EXISTS detect_run_events (
id BIGSERIAL PRIMARY KEY,
job_id BIGINT REFERENCES detect_jobs(id) ON DELETE SET NULL,
job_item_id BIGINT REFERENCES detect_job_items(id) ON DELETE SET NULL,
node_code VARCHAR(64) NOT NULL DEFAULT '',
event_type VARCHAR(64) NOT NULL DEFAULT '',
level VARCHAR(16) NOT NULL DEFAULT 'info',
message TEXT NOT NULL DEFAULT '',
payload_json JSONB,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS detect_sync_records (
id BIGSERIAL PRIMARY KEY,
sync_type VARCHAR(32) NOT NULL DEFAULT '',
source_region VARCHAR(32) NOT NULL DEFAULT '',
target_region VARCHAR(32) NOT NULL DEFAULT '',
status VARCHAR(32) NOT NULL DEFAULT 'pending',
payload_json JSONB,
error_message TEXT NOT NULL DEFAULT '',
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
"""
return self.execute(sql)
def register_cluster_node(self, node_code, region, role, status='online', current_load=0, metadata=None):
try:
ip_addr = ''
try:
ip_addr = socket.gethostbyname(socket.gethostname())
except Exception:
pass
sql = """
INSERT INTO detect_worker_nodes (
node_code, region, role, hostname, ip, status, worker_version, current_load, metadata_json, last_heartbeat_at, update_time
) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s::jsonb, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
ON CONFLICT (node_code) DO UPDATE SET
region = EXCLUDED.region,
role = EXCLUDED.role,
hostname = EXCLUDED.hostname,
ip = EXCLUDED.ip,
status = EXCLUDED.status,
worker_version = EXCLUDED.worker_version,
current_load = EXCLUDED.current_load,
metadata_json = EXCLUDED.metadata_json,
last_heartbeat_at = CURRENT_TIMESTAMP,
update_time = CURRENT_TIMESTAMP
"""
return self.execute(
sql,
(
node_code,
region,
role,
socket.gethostname(),
ip_addr,
status,
'0.1.0',
int(current_load or 0),
json.dumps(metadata or {}, ensure_ascii=False),
),
)
except Exception as e:
logger.warning(f"注册检测节点失败: {e}")
return False
def claim_detect_job_items(self, node_code, limit=1000, lease_seconds=3600):
"""
领取一批待执行的任务项。
"""
conn = None
cur = None
claim_token = f"{node_code}-{int(threading.current_thread().ident or 0)}-{int(__import__('time').time())}"
try:
conn, cur = self.connect()
if not conn or not cur:
logger.error("领取检测任务失败: 无法获取数据库连接")
return []
cur.execute(
"""
WITH picked AS (
SELECT id
FROM detect_job_items
WHERE status IN ('pending', 'failed')
AND (lease_expires_at IS NULL OR lease_expires_at < CURRENT_TIMESTAMP)
ORDER BY create_time ASC, id ASC
FOR UPDATE SKIP LOCKED
LIMIT %s
),
updated AS (
UPDATE detect_job_items AS item
SET status = 'claimed',
claimed_by = %s,
claim_token = %s,
lease_expires_at = CURRENT_TIMESTAMP + (%s || ' seconds')::interval,
attempt_count = item.attempt_count + 1,
started_at = COALESCE(item.started_at, CURRENT_TIMESTAMP),
updated_at = CURRENT_TIMESTAMP
FROM picked
WHERE item.id = picked.id
RETURNING item.id, item.job_id, item.domain_id, item.claim_token
)
SELECT
updated.id,
updated.job_id,
updated.domain_id,
updated.claim_token,
domains.domain,
domains.source_type,
domains.register_status,
domains.detect_status,
domains.use_status,
domains.expire_date,
domains.jucha_status,
domains.juziseo_status
FROM updated
JOIN domains ON domains.id = updated.domain_id
ORDER BY updated.id ASC
""",
(limit, node_code, claim_token, max(60, int(lease_seconds or 3600))),
)
rows = cur.fetchall() or []
touched_job_ids = sorted({row[1] for row in rows})
for job_id in touched_job_ids:
cur.execute(
"""
UPDATE detect_jobs
SET status = CASE WHEN status = 'pending' THEN 'running' ELSE status END,
started_at = COALESCE(started_at, CURRENT_TIMESTAMP)
WHERE id = %s
""",
(job_id,),
)
conn.commit()
return [
{
"job_item_id": row[0],
"job_id": row[1],
"id": row[2],
"claim_token": row[3],
"domain": row[4],
"source_type": row[5],
"register_status": row[6],
"detect_status": row[7],
"use_status": row[8],
"expire_date": row[9],
"jucha_status": row[10],
"juziseo_status": row[11],
}
for row in rows
]
except Exception as e:
try:
if conn:
conn.rollback()
except Exception:
pass
logger.error(f"领取检测任务失败: {e}")
return []
finally:
self.close(conn, cur)
def recycle_expired_detect_job_items(self):
"""
回收租约过期但未完成的任务项，重新放回 pending。
"""
conn = None
cur = None
try:
conn, cur = self.connect()
if not conn or not cur:
return 0
cur.execute(
"""
WITH recycled AS (
UPDATE detect_job_items
SET status = 'pending',
claimed_by = '',
claim_token = '',
lease_expires_at = NULL,
updated_at = CURRENT_TIMESTAMP,
last_error = CASE
WHEN status = 'running' THEN 'lease expired while running'
WHEN status = 'claimed' THEN 'lease expired before running'
ELSE last_error
END
WHERE status IN ('claimed', 'running')
AND lease_expires_at IS NOT NULL
AND lease_expires_at < CURRENT_TIMESTAMP
RETURNING job_id
)
SELECT count(*), array_remove(array_agg(DISTINCT job_id), NULL)
FROM recycled
"""
)
row = cur.fetchone()
recycled_count = int((row or [0])[0] or 0)
touched_job_ids = (row or [0, []])[1] or []
for job_id in touched_job_ids:
self._refresh_detect_job_status_with_cursor(cur, job_id)
conn.commit()
return recycled_count
except Exception as e:
try:
if conn:
conn.rollback()
except Exception:
pass
logger.error(f"回收过期检测任务失败: {e}")
return 0
finally:
self.close(conn, cur)
def release_detect_job_items_for_node(self, node_code):
"""
释放指定节点遗留的 claimed/running 任务项，供节点重启后快速恢复。
"""
conn = None
cur = None
try:
conn, cur = self.connect()
if not conn or not cur:
return 0
cur.execute(
"""
WITH released AS (
UPDATE detect_job_items
SET status = 'pending',
claimed_by = '',
claim_token = '',
lease_expires_at = NULL,
updated_at = CURRENT_TIMESTAMP,
last_error = CASE
WHEN status = 'running' THEN 'released after worker restart'
WHEN status = 'claimed' THEN 'released before execution after worker restart'
ELSE last_error
END
WHERE claimed_by = %s
AND status IN ('claimed', 'running')
RETURNING job_id
)
SELECT count(*), array_remove(array_agg(DISTINCT job_id), NULL)
FROM released
""",
(node_code,),
)
row = cur.fetchone()
released_count = int((row or [0])[0] or 0)
touched_job_ids = (row or [0, []])[1] or []
for job_id in touched_job_ids:
self._refresh_detect_job_status_with_cursor(cur, job_id)
conn.commit()
return released_count
except Exception as e:
try:
if conn:
conn.rollback()
except Exception:
pass
logger.error(f"释放节点遗留任务失败: {e}")
return 0
finally:
self.close(conn, cur)
def mark_detect_job_item_running(self, job_item_id, claim_token):
return self.execute(
"""
UPDATE detect_job_items
SET status = 'running',
updated_at = CURRENT_TIMESTAMP,
lease_expires_at = CURRENT_TIMESTAMP + interval '1 hour'
WHERE id = %s AND claim_token = %s
""",
(job_item_id, claim_token),
)
def renew_detect_job_item_lease(self, job_item_id, claim_token, lease_seconds=3600):
return self.execute(
"""
UPDATE detect_job_items
SET lease_expires_at = CURRENT_TIMESTAMP + (%s || ' seconds')::interval,
updated_at = CURRENT_TIMESTAMP
WHERE id = %s AND claim_token = %s AND status IN ('claimed', 'running')
""",
(max(60, int(lease_seconds or 3600)), job_item_id, claim_token),
)
def append_detect_run_event(self, job_id, job_item_id, node_code, event_type, message, level='info', payload=None):
return self.execute(
"""
INSERT INTO detect_run_events (job_id, job_item_id, node_code, event_type, level, message, payload_json)
VALUES (%s, %s, %s, %s, %s, %s, %s)
""",
(job_id, job_item_id, node_code, event_type, level, message, Json(payload or {})),
)
def complete_detect_job_item(self, job_item_id, claim_token, final_status='completed'):
conn = None
cur = None
try:
conn, cur = self.connect()
if not conn or not cur:
return False
cur.execute(
"""
UPDATE detect_job_items
SET status = %s,
finished_at = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP,
lease_expires_at = NULL
WHERE id = %s AND claim_token = %s
RETURNING job_id
""",
(final_status, job_item_id, claim_token),
)
row = cur.fetchone()
if row:
self._refresh_detect_job_status_with_cursor(cur, row[0])
conn.commit()
return bool(row)
except Exception as e:
try:
if conn:
conn.rollback()
except Exception:
pass
logger.error(f"完成检测任务项失败: {e}")
return False
finally:
self.close(conn, cur)
def fail_detect_job_item(self, job_item_id, claim_token, reason=''):
conn = None
cur = None
try:
conn, cur = self.connect()
if not conn or not cur:
return False
cur.execute(
"""
UPDATE detect_job_items
SET status = 'failed',
last_error = %s,
finished_at = CURRENT_TIMESTAMP,
updated_at = CURRENT_TIMESTAMP,
lease_expires_at = NULL
WHERE id = %s AND claim_token = %s
RETURNING job_id
""",
(str(reason or '')[:1000], job_item_id, claim_token),
)
row = cur.fetchone()
if row:
self._refresh_detect_job_status_with_cursor(cur, row[0])
conn.commit()
return bool(row)
except Exception as e:
try:
if conn:
conn.rollback()
except Exception:
pass
logger.error(f"标记检测任务项失败: {e}")
return False
finally:
self.close(conn, cur)
def _refresh_detect_job_status_with_cursor(self, cur, job_id):
cur.execute(
"""
SELECT
count(*) FILTER (WHERE status IN ('pending', 'claimed', 'running')) AS active_count,
count(*) FILTER (WHERE status = 'failed') AS failed_count,
count(*) FILTER (WHERE status IN ('completed', 'blacklisted')) AS done_count
FROM detect_job_items
WHERE job_id = %s
""",
(job_id,),
)
row = cur.fetchone()
active_count = int(row[0] or 0)
failed_count = int(row[1] or 0)
done_count = int(row[2] or 0)
if active_count > 0:
cur.execute(
"UPDATE detect_jobs SET status = 'running', started_at = COALESCE(started_at, CURRENT_TIMESTAMP) WHERE id = %s",
(job_id,),
)
return
final_status = 'completed'
if failed_count > 0 and done_count > 0:
final_status = 'partial_failed'
elif failed_count > 0:
final_status = 'failed'
cur.execute(
"""
UPDATE detect_jobs
SET status = %s,
finished_at = CURRENT_TIMESTAMP,
started_at = COALESCE(started_at, CURRENT_TIMESTAMP)
WHERE id = %s
""",
(final_status, job_id),
)
def fetch_one(self, sql, params=None): def fetch_one(self, sql, params=None):
""" """
获取单条数据 获取单条数据
@@ -812,6 +1271,35 @@ class Database:
sql = "UPDATE domains SET detect_status = %s WHERE id = %s" sql = "UPDATE domains SET detect_status = %s WHERE id = %s"
return self.execute(sql, (status, domain_id)) return self.execute(sql, (status, domain_id))
def recycle_running_domains(self, target_status):
"""
回收异常中断后遗留的“检测中”状态。
:param target_status: 目标状态，通常使用 DETECT_STATUS_FAILED
:return: int - 影响行数
"""
conn, cur = self.connect()
if not conn or not cur:
logger.error("回收检测中状态失败: 无法获取数据库连接")
return 0
try:
cur.execute(
"UPDATE domains SET detect_status = %s WHERE detect_status = %s",
(target_status, DETECT_STATUS_RUNNING)
)
affected = cur.rowcount or 0
conn.commit()
return affected
except Exception as e:
try:
conn.rollback()
except Exception:
pass
logger.error(f"回收检测中状态失败: {e}")
return 0
finally:
self.close(conn, cur)
def update_domain_third_party_status(self, domain_id, field_name, status): def update_domain_third_party_status(self, domain_id, field_name, status):
""" """
更新第三方平台检测状态。 更新第三方平台检测状态。

2
domainCheck/detect/geetest2.py
View File

@@ -3,7 +3,6 @@ import requests, re, json, base64
import io, os, random import io, os, random
import time, cv2, json import time, cv2, json
from PIL import Image from PIL import Image
from functools import partial
from io import BytesIO from io import BytesIO
@@ -13,7 +12,6 @@ from detect.module.use_ua import randomUA
from detect.module.gap import quekou from detect.module.gap import quekou
import subprocess import subprocess
subprocess.Popen = partial(subprocess.Popen, encoding='utf-8')
import execjs import execjs

31
domainCheck/detect/jucha.py
View File

@@ -15,9 +15,9 @@ import json # JSON数据处理
import os # 操作系统接口 import os # 操作系统接口
import pickle # 序列化反序列化 import pickle # 序列化反序列化
import random # 随机数生成 import random # 随机数生成
import shutil
import subprocess # 子进程管理 import subprocess # 子进程管理
import time # 时间处理 import time # 时间处理
from functools import partial
# 导入第三方库 # 导入第三方库
import requests # HTTP请求库 import requests # HTTP请求库
@@ -33,11 +33,17 @@ from requests.cookies import RequestsCookieJar # Cookie管理
# super().__init__(*args, **kwargs) # 调用父类初始化方法 # super().__init__(*args, **kwargs) # 调用父类初始化方法
# #
# #
# subprocess.Popen = MySubprocessPopen # 替换subprocess.Popen为自定义类
os.environ["EXECJS_RUNTIME"] = "Node" # 设置JavaScript运行时环境为Node.js os.environ["EXECJS_RUNTIME"] = "Node" # 设置JavaScript运行时环境为Node.js
startupinfo = subprocess.STARTUPINFO() _ORIGINAL_POPEN = subprocess.Popen
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
subprocess.Popen = partial(subprocess.Popen, encoding='utf-8', startupinfo=startupinfo)
def create_subprocess(*args, **kwargs):
kwargs.setdefault("encoding", "utf-8")
if hasattr(subprocess, "STARTUPINFO") and hasattr(subprocess, "STARTF_USESHOWWINDOW"):
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
kwargs.setdefault("startupinfo", startupinfo)
return _ORIGINAL_POPEN(*args, **kwargs)
def resolve_node_executable() -> str: def resolve_node_executable() -> str:
@@ -46,6 +52,19 @@ def resolve_node_executable() -> str:
bundled_node = os.path.join(project_dir, "tools", "node-v20.19.4-win-x64", "node.exe") bundled_node = os.path.join(project_dir, "tools", "node-v20.19.4-win-x64", "node.exe")
if os.path.exists(bundled_node): if os.path.exists(bundled_node):
return bundled_node return bundled_node
detected = shutil.which("node")
if detected:
return detected
fallback_candidates = [
"/usr/bin/node",
"/usr/local/bin/node",
os.path.expanduser("~/.nvm/versions/node/current/bin/node"),
os.path.expanduser("~/.nvm/versions/node/v20.20.2/bin/node"),
"/root/.nvm/versions/node/v20.20.2/bin/node",
]
for candidate in fallback_candidates:
if os.path.exists(candidate):
return candidate
return "node" return "node"
def calculate_seed(t: int) -> str: # 计算验证码种子值 def calculate_seed(t: int) -> str: # 计算验证码种子值
@@ -195,7 +214,7 @@ class JC(object): # 聚查网API封装类
return False, f"缺少签名脚本: {jsFile_path}" # 返回失败 return False, f"缺少签名脚本: {jsFile_path}" # 返回失败
try: # 尝试执行Node.js脚本 try: # 尝试执行Node.js脚本
with subprocess.Popen( # 创建子进程 with create_subprocess( # 创建子进程
[resolve_node_executable(), jsFile_path], # 执行node命令 [resolve_node_executable(), jsFile_path], # 执行node命令
stdin=subprocess.PIPE, # 标准输入管道 stdin=subprocess.PIPE, # 标准输入管道
stdout=subprocess.PIPE, # 标准输出管道 stdout=subprocess.PIPE, # 标准输出管道

35
domainCheck/detect/juming.py
View File

@@ -14,9 +14,9 @@ import json # JSON处理模块
import os # 操作系统接口模块 import os # 操作系统接口模块
import pickle # 序列化模块 import pickle # 序列化模块
import random # 随机数生成模块 import random # 随机数生成模块
import shutil
import subprocess # 子进程管理模块 import subprocess # 子进程管理模块
import time # 时间处理模块 import time # 时间处理模块
from functools import partial
from typing import Optional, Tuple, Dict, List, Any # 类型提示 from typing import Optional, Tuple, Dict, List, Any # 类型提示
import requests # HTTP请求库 import requests # HTTP请求库
@@ -34,17 +34,25 @@ SLIDE_OFFSET = 290 # 滑块偏移量
SLIDE_DURATION = 611 # 滑动持续时间 SLIDE_DURATION = 611 # 滑动持续时间
class MySubprocessPopen(subprocess.Popen): # 自定义子进程类 _ORIGINAL_POPEN = subprocess.Popen
class MySubprocessPopen(_ORIGINAL_POPEN): # 自定义子进程类
def __init__(self, *args, **kwargs): # 初始化方法 def __init__(self, *args, **kwargs): # 初始化方法
kwargs['encoding'] = "UTF-8" # 设置编码为UTF-8 kwargs['encoding'] = "UTF-8" # 设置编码为UTF-8
super().__init__(*args, **kwargs) # 调用父类初始化 super().__init__(*args, **kwargs) # 调用父类初始化
subprocess.Popen = MySubprocessPopen # 替换默认的Popen类
os.environ["EXECJS_RUNTIME"] = "Node" # 设置JavaScript运行时为Node.js os.environ["EXECJS_RUNTIME"] = "Node" # 设置JavaScript运行时为Node.js
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
subprocess.Popen = partial(subprocess.Popen, encoding='utf-8', startupinfo=startupinfo) def create_subprocess(*args, **kwargs):
kwargs.setdefault("encoding", "utf-8")
if hasattr(subprocess, "STARTUPINFO") and hasattr(subprocess, "STARTF_USESHOWWINDOW"):
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
kwargs.setdefault("startupinfo", startupinfo)
return MySubprocessPopen(*args, **kwargs)
def resolve_node_executable() -> str: def resolve_node_executable() -> str:
@@ -53,6 +61,19 @@ def resolve_node_executable() -> str:
bundled_node = os.path.join(project_dir, "tools", "node-v20.19.4-win-x64", "node.exe") bundled_node = os.path.join(project_dir, "tools", "node-v20.19.4-win-x64", "node.exe")
if os.path.exists(bundled_node): if os.path.exists(bundled_node):
return bundled_node return bundled_node
detected = shutil.which("node")
if detected:
return detected
fallback_candidates = [
"/usr/bin/node",
"/usr/local/bin/node",
os.path.expanduser("~/.nvm/versions/node/current/bin/node"),
os.path.expanduser("~/.nvm/versions/node/v20.20.2/bin/node"),
"/root/.nvm/versions/node/v20.20.2/bin/node",
]
for candidate in fallback_candidates:
if os.path.exists(candidate):
return candidate
return "node" return "node"
def calculate_seed(t: int) -> str: # 计算验证码种子值 def calculate_seed(t: int) -> str: # 计算验证码种子值
@@ -230,7 +251,7 @@ class JM(object): # 聚名网API封装类
return False, f"缺少签名脚本: {jsFile_path}" # 返回失败 return False, f"缺少签名脚本: {jsFile_path}" # 返回失败
try: # 异常处理 try: # 异常处理
with subprocess.Popen( # 执行Node.js脚本 with create_subprocess( # 执行Node.js脚本
[resolve_node_executable(), jsFile_path], # 命令和参数 [resolve_node_executable(), jsFile_path], # 命令和参数
stdin=subprocess.PIPE, # 标准输入 stdin=subprocess.PIPE, # 标准输入
stdout=subprocess.PIPE, # 标准输出 stdout=subprocess.PIPE, # 标准输出

12
domainCheck/detect_options.json
View File

@@ -1,8 +1,4 @@
{ {
"detect_juziseo": false,
"detect_360_site": true,
"detect_aizhan": true,
"detect_baidu_site": true,
"detect_order": [ "detect_order": [
"detect_register", "detect_register",
"detect_baidu_site", "detect_baidu_site",
@@ -14,7 +10,11 @@
"detect_juziseo" "detect_juziseo"
], ],
"detect_register": true, "detect_register": true,
"detect_wayback": true, "detect_baidu_site": true,
"detect_360_site": true,
"detect_chinaz": true, "detect_chinaz": true,
"detect_jucha": false "detect_aizhan": true,
"detect_wayback": true,
"detect_jucha": false,
"detect_juziseo": false
} }

1138
domainCheck/detect_worker.py
View File

File diff suppressed because it is too large Load Diff

4
domainCheck/init_database.py
View File

@@ -204,6 +204,7 @@ def init_database():
baidu_site JSONB, baidu_site JSONB,
qihu360_site JSONB, qihu360_site JSONB,
google_site JSONB, google_site JSONB,
wayback_info JSONB,
chinaz_info JSONB, chinaz_info JSONB,
aizhan_info JSONB, aizhan_info JSONB,
juziseo_info JSONB, juziseo_info JSONB,
@@ -227,6 +228,8 @@ def init_database():
cur.execute("ALTER TABLE domain_detections ADD COLUMN IF NOT EXISTS is_chinese_title BOOLEAN") cur.execute("ALTER TABLE domain_detections ADD COLUMN IF NOT EXISTS is_chinese_title BOOLEAN")
# 添加backlink_count_gt_10字段 # 添加backlink_count_gt_10字段
cur.execute("ALTER TABLE domain_detections ADD COLUMN IF NOT EXISTS backlink_count_gt_10 BOOLEAN") cur.execute("ALTER TABLE domain_detections ADD COLUMN IF NOT EXISTS backlink_count_gt_10 BOOLEAN")
# 添加wayback_info字段
cur.execute("ALTER TABLE domain_detections ADD COLUMN IF NOT EXISTS wayback_info JSONB")
logger.info("为domain_detections表添加新字段成功") logger.info("为domain_detections表添加新字段成功")
except Exception as e: except Exception as e:
logger.warning(f"为domain_detections表添加新字段失败: {e}") logger.warning(f"为domain_detections表添加新字段失败: {e}")
@@ -240,6 +243,7 @@ def init_database():
cur.execute("COMMENT ON COLUMN domain_detections.baidu_site IS '百度site'") cur.execute("COMMENT ON COLUMN domain_detections.baidu_site IS '百度site'")
cur.execute("COMMENT ON COLUMN domain_detections.qihu360_site IS '360 site'") cur.execute("COMMENT ON COLUMN domain_detections.qihu360_site IS '360 site'")
cur.execute("COMMENT ON COLUMN domain_detections.google_site IS 'Google site'") cur.execute("COMMENT ON COLUMN domain_detections.google_site IS 'Google site'")
cur.execute("COMMENT ON COLUMN domain_detections.wayback_info IS '时光机检测信息'")
cur.execute("COMMENT ON COLUMN domain_detections.chinaz_info IS '站长之家信息'") cur.execute("COMMENT ON COLUMN domain_detections.chinaz_info IS '站长之家信息'")
cur.execute("COMMENT ON COLUMN domain_detections.aizhan_info IS '爱站网信息'") cur.execute("COMMENT ON COLUMN domain_detections.aizhan_info IS '爱站网信息'")
cur.execute("COMMENT ON COLUMN domain_detections.juziseo_info IS '桔子SEO信息'") cur.execute("COMMENT ON COLUMN domain_detections.juziseo_info IS '桔子SEO信息'")

2
domainCheck/proxy_config.json
View File

@@ -1,7 +1,7 @@
{ {
"proxy_url": "http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50",
"proxy_enable": true, "proxy_enable": true,
"allow_direct": true, "allow_direct": true,
"proxy_url": "http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50",
"proxy_urls": [ "proxy_urls": [
"http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50", "http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50",
"http://211.101.244.154:18008/getProxy_batch.php?group=C&count=50", "http://211.101.244.154:18008/getProxy_batch.php?group=C&count=50",

2
domainCheck/thread_count.json
View File

@@ -1,3 +1,3 @@
{ {
"thread_count": "2" "thread_count": "5"
} }