debug
This commit is contained in:
@@ -36,6 +36,10 @@
|
|||||||
|
|
||||||
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
|
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
|
||||||
|
|
||||||
|
### 9. 临时海外控制面联调清单
|
||||||
|
|
||||||
|
- `docs/20_domainCheck_临时海外控制面联调清单.md`
|
||||||
|
|
||||||
## 二、文档阅读顺序
|
## 二、文档阅读顺序
|
||||||
|
|
||||||
### 1. 先看总体方案
|
### 1. 先看总体方案
|
||||||
@@ -53,6 +57,7 @@
|
|||||||
- `docs/16_domainCheck_多机检测与跨地域部署设计.md`
|
- `docs/16_domainCheck_多机检测与跨地域部署设计.md`
|
||||||
- `docs/17_domainCheck_全流程部署实操手册.md`
|
- `docs/17_domainCheck_全流程部署实操手册.md`
|
||||||
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
|
- `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md`
|
||||||
|
- `docs/20_domainCheck_临时海外控制面联调清单.md`
|
||||||
|
|
||||||
### 3. 如果要回溯历史需求与问题
|
### 3. 如果要回溯历史需求与问题
|
||||||
|
|
||||||
|
|||||||
231
docs/20_domainCheck_临时海外控制面联调清单.md
Normal file
231
docs/20_domainCheck_临时海外控制面联调清单.md
Normal file
@@ -0,0 +1,231 @@
|
|||||||
|
# domainCheck 临时海外控制面联调清单
|
||||||
|
|
||||||
|
适用场景:
|
||||||
|
|
||||||
|
- 暂时不使用正式海外机器
|
||||||
|
- 先把大陆 `controller + worker` 对接到当前测试海外机
|
||||||
|
- 先跑通“控制面 + 同步 + 观测”全链路,再切回正式海外环境
|
||||||
|
|
||||||
|
当前临时海外控制面:
|
||||||
|
|
||||||
|
- 节点编码:`overseas-control-01`
|
||||||
|
- 节点角色:`overseas / control`
|
||||||
|
- 机器 IP:`152.53.37.118`
|
||||||
|
- 后台地址:`http://152.53.37.118/`
|
||||||
|
- API 基址:`http://152.53.37.118:8100`
|
||||||
|
|
||||||
|
## 1. 当前目标拓扑
|
||||||
|
|
||||||
|
本次联调按下面的角色分工:
|
||||||
|
|
||||||
|
- 大陆 controller
|
||||||
|
- 承担 API 控制面
|
||||||
|
- 承担 Sync Agent
|
||||||
|
- 可以兼跑本机 Worker
|
||||||
|
- 大陆 worker
|
||||||
|
- 只承担检测执行
|
||||||
|
- 临时海外控制面
|
||||||
|
- 只承担 API 控制与同步接收
|
||||||
|
- 不承载本机 Worker
|
||||||
|
- 不承载 Sync Agent
|
||||||
|
|
||||||
|
## 2. 临时海外控制面应有状态
|
||||||
|
|
||||||
|
在临时海外控制面机器上执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash domain-api/deploy/multi-region/check_temp_overseas_control.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:
|
||||||
|
|
||||||
|
- `domaincheck-api` 为 `active (running)`
|
||||||
|
- `domaincheck-worker` 为 `inactive (dead)` 或禁用
|
||||||
|
- `runtime.status.data.node`
|
||||||
|
- `code=overseas-control-01`
|
||||||
|
- `region=overseas`
|
||||||
|
- `role=control`
|
||||||
|
- `runtime.status.data.worker.expected_on_this_node=false`
|
||||||
|
- `runtime.status.data.sync_agent.expected_on_this_node=false`
|
||||||
|
- `runtime.status.data.detect.phase_label=当前节点不承载`
|
||||||
|
|
||||||
|
## 3. 大陆 controller 切到临时海外控制面
|
||||||
|
|
||||||
|
在大陆 controller 上执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -i 's#^SYNC_TARGET_API_BASE_URL=.*#SYNC_TARGET_API_BASE_URL=http://152.53.37.118:8100#' /etc/default/domaincheck-api
|
||||||
|
sed -i 's/^SYNC_PUSH_ENABLED=.*/SYNC_PUSH_ENABLED=true/' /etc/default/domaincheck-api
|
||||||
|
|
||||||
|
systemctl restart domaincheck-api
|
||||||
|
systemctl restart domaincheck-sync-agent
|
||||||
|
systemctl restart domaincheck-worker
|
||||||
|
sleep 3
|
||||||
|
```
|
||||||
|
|
||||||
|
然后检查:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s http://127.0.0.1:8100/api/v1/runtime/readiness
|
||||||
|
echo
|
||||||
|
curl -s http://127.0.0.1:8100/api/v1/runtime/sync-summary
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:
|
||||||
|
|
||||||
|
- `runtime/readiness` 至少不是大陆本机配置错误导致的 `blocking`
|
||||||
|
- `runtime/sync-summary.data.enabled=true`
|
||||||
|
- `runtime/sync-summary.data.target_api_base_url=http://152.53.37.118:8100`
|
||||||
|
|
||||||
|
## 4. 大陆 worker 检查项
|
||||||
|
|
||||||
|
在大陆 worker 上执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash domain-api/deploy/multi-region/check_mainland_worker.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:
|
||||||
|
|
||||||
|
- `NODE_CODE=mainland-worker-01`
|
||||||
|
- `NODE_REGION=mainland`
|
||||||
|
- `NODE_ROLE=worker`
|
||||||
|
- `domaincheck-worker` 为 `active (running)`
|
||||||
|
|
||||||
|
如果大陆 controller 也兼跑检测,再执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl status domaincheck-worker --no-pager -l
|
||||||
|
echo
|
||||||
|
curl -s http://127.0.0.1:8100/api/v1/runtime/status
|
||||||
|
```
|
||||||
|
|
||||||
|
预期:
|
||||||
|
|
||||||
|
- `worker.running=true`
|
||||||
|
- `worker.expected_on_this_node=true`
|
||||||
|
|
||||||
|
## 5. 最短联调命令
|
||||||
|
|
||||||
|
### 5.1 在大陆 controller 上
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash domain-api/deploy/multi-region/check_temp_link.sh http://127.0.0.1:8100 http://152.53.37.118:8100
|
||||||
|
```
|
||||||
|
|
||||||
|
关注点:
|
||||||
|
|
||||||
|
- `push_sync` 返回 `code=0` 或幂等成功
|
||||||
|
- 不再出现目标地址为空
|
||||||
|
- 不再出现 `SYNC_PUSH_ENABLED=false`
|
||||||
|
|
||||||
|
关注点:
|
||||||
|
|
||||||
|
- `cluster` 中至少能看到:
|
||||||
|
- `overseas-control-01`
|
||||||
|
- 大陆 `mainland-controller-01`
|
||||||
|
- 大陆 `mainland-worker-01`
|
||||||
|
- `sync-summary` 中能看到来自大陆的接收记录
|
||||||
|
- `readiness` 不再是“完全单节点海外态”
|
||||||
|
|
||||||
|
## 6. 后台页面应如何理解
|
||||||
|
|
||||||
|
打开:
|
||||||
|
|
||||||
|
- `http://152.53.37.118/`
|
||||||
|
|
||||||
|
在运行中心里看到下面这些内容时,属于正常:
|
||||||
|
|
||||||
|
- `本机 Worker:当前节点不承载`
|
||||||
|
- `Sync Agent:当前节点不承载`
|
||||||
|
- `当前检测阶段:当前节点不承载`
|
||||||
|
- `代理运行态:不适用`
|
||||||
|
|
||||||
|
这不是故障,而是因为当前节点就是临时海外控制面。
|
||||||
|
|
||||||
|
真正应该关注的是:
|
||||||
|
|
||||||
|
- `集群节点`
|
||||||
|
- `有效执行节点`
|
||||||
|
- `同步状态`
|
||||||
|
- `多机就绪度`
|
||||||
|
|
||||||
|
## 7. 判断“到底有几台 Worker 在工作”
|
||||||
|
|
||||||
|
如果页面上“在线 Worker 数”和“实际正在跑检测的节点数”看起来不一致,以数据库为准。
|
||||||
|
|
||||||
|
在大陆 controller 上执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash domain-api/deploy/multi-region/check_worker_participation.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
判断规则:
|
||||||
|
|
||||||
|
- `detect_worker_nodes`
|
||||||
|
- 看谁在线
|
||||||
|
- 看谁是 `worker_online=true`
|
||||||
|
- 看谁 `detect_participating=true`
|
||||||
|
- `detect_job_items`
|
||||||
|
- 看任务到底被哪台节点 `claimed_by`
|
||||||
|
- 这才是“当前真正干活的节点”
|
||||||
|
|
||||||
|
## 8. 常见误判
|
||||||
|
|
||||||
|
### 8.1 海外控制面显示“本机 Worker 不承载”
|
||||||
|
|
||||||
|
这是正常,不是故障。
|
||||||
|
|
||||||
|
### 8.2 页面显示“在线 Worker 1”,但你部署了 2 台大陆机器
|
||||||
|
|
||||||
|
先区分:
|
||||||
|
|
||||||
|
- “有效执行节点”是可承担任务的在线节点数
|
||||||
|
- “当前参与检测节点”是当前真的在领任务、跑任务的节点
|
||||||
|
|
||||||
|
如果 controller 没兼跑检测,通常只会看到独立 worker 在真正执行。
|
||||||
|
|
||||||
|
### 8.3 controller 机器也开了 worker 服务,但没有领任务
|
||||||
|
|
||||||
|
这不一定是 bug,可能只是当前调度没有分到它。应以 `detect_job_items.claimed_by` 为准,不要只看服务在线。
|
||||||
|
|
||||||
|
## 9. 联调完成后的回滚
|
||||||
|
|
||||||
|
如果你后续要切回正式海外机器,在大陆 controller 上把目标地址改回正式值即可:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed -i 's#^SYNC_TARGET_API_BASE_URL=.*#SYNC_TARGET_API_BASE_URL=http://正式海外控制面IP:8100#' /etc/default/domaincheck-api
|
||||||
|
systemctl restart domaincheck-api
|
||||||
|
systemctl restart domaincheck-sync-agent
|
||||||
|
sleep 3
|
||||||
|
```
|
||||||
|
|
||||||
|
如果临时海外控制面不再使用,可在该机器上保留 `domaincheck-api` 供后续调试,也可以停掉:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl stop domaincheck-api
|
||||||
|
```
|
||||||
|
|
||||||
|
## 10. 本次临时海外控制面结论
|
||||||
|
|
||||||
|
当前这台测试海外机已经整理为:
|
||||||
|
|
||||||
|
- 纯 `overseas-control`
|
||||||
|
- API 正常
|
||||||
|
- Worker 不承载
|
||||||
|
- Sync Agent 不承载
|
||||||
|
- 数据已清空并按首装态初始化
|
||||||
|
- 可直接作为大陆 controller 的临时同步接收端
|
||||||
|
|
||||||
|
## 11. 配套脚本
|
||||||
|
|
||||||
|
- 临时海外控制面自检:
|
||||||
|
- `domain-api/deploy/multi-region/check_temp_overseas_control.sh`
|
||||||
|
- 大陆 controller -> 临时海外控制面联调检查:
|
||||||
|
- `domain-api/deploy/multi-region/check_temp_link.sh`
|
||||||
|
- 三机拓扑总览检查:
|
||||||
|
- `domain-api/deploy/multi-region/check_temp_topology.sh`
|
||||||
|
- 当前活跃任务由谁真正执行:
|
||||||
|
- `domain-api/deploy/multi-region/check_worker_participation.sh`
|
||||||
|
- 大陆 worker 本机自检:
|
||||||
|
- `domain-api/deploy/multi-region/check_mainland_worker.sh`
|
||||||
@@ -1,3 +1,5 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
from uuid import uuid4
|
from uuid import uuid4
|
||||||
|
|
||||||
from fastapi import APIRouter
|
from fastapi import APIRouter
|
||||||
|
|||||||
@@ -28,11 +28,13 @@ def fetch_overview() -> dict:
|
|||||||
online_worker_nodes = int(cluster_summary.get("online_worker_nodes", 0) or 0)
|
online_worker_nodes = int(cluster_summary.get("online_worker_nodes", 0) or 0)
|
||||||
dedicated_online_worker_nodes = int(cluster_summary.get("dedicated_online_worker_nodes", 0) or 0)
|
dedicated_online_worker_nodes = int(cluster_summary.get("dedicated_online_worker_nodes", 0) or 0)
|
||||||
result["worker_status"] = "online" if runtime["worker"]["running"] else "offline"
|
result["worker_status"] = "online" if runtime["worker"]["running"] else "offline"
|
||||||
result["local_worker_status"] = result["worker_status"]
|
result["local_worker_status"] = "not-applicable" if not runtime["worker"].get("expected_on_this_node", True) else result["worker_status"]
|
||||||
result["cluster_worker_status"] = "online" if online_worker_nodes > 0 else "offline"
|
result["cluster_worker_status"] = "online" if online_worker_nodes > 0 else "offline"
|
||||||
result["cluster_online_worker_nodes"] = online_worker_nodes
|
result["cluster_online_worker_nodes"] = online_worker_nodes
|
||||||
result["cluster_dedicated_online_worker_nodes"] = dedicated_online_worker_nodes
|
result["cluster_dedicated_online_worker_nodes"] = dedicated_online_worker_nodes
|
||||||
result["cluster_online_control_nodes"] = int(cluster_summary.get("online_control_nodes", 0) or 0)
|
result["cluster_online_control_nodes"] = int(cluster_summary.get("online_control_nodes", 0) or 0)
|
||||||
result["api_status"] = "online"
|
result["api_status"] = "online"
|
||||||
result["worker_mode"] = runtime["worker"]["mode"]
|
result["worker_mode"] = runtime["worker"]["mode"]
|
||||||
|
result["node_region"] = runtime["node"]["region"]
|
||||||
|
result["node_role"] = runtime["node"]["role"]
|
||||||
return result
|
return result
|
||||||
|
|||||||
@@ -270,6 +270,7 @@ def get_runtime_status() -> dict:
|
|||||||
runtime_settings = get_runtime_settings()
|
runtime_settings = get_runtime_settings()
|
||||||
worker_runtime = detect_worker_runtime()
|
worker_runtime = detect_worker_runtime()
|
||||||
sync_agent_runtime = detect_sync_agent_runtime()
|
sync_agent_runtime = detect_sync_agent_runtime()
|
||||||
|
worker_expected_on_this_node = not (settings.node_region == "overseas" and settings.node_role == "control")
|
||||||
api_pid = os.getpid()
|
api_pid = os.getpid()
|
||||||
detect_snapshot = get_detect_status()
|
detect_snapshot = get_detect_status()
|
||||||
latest_run = (detect_snapshot.get("runs") or [None])[0] or {}
|
latest_run = (detect_snapshot.get("runs") or [None])[0] or {}
|
||||||
@@ -314,6 +315,35 @@ def get_runtime_status() -> dict:
|
|||||||
"queue_health": queue_health,
|
"queue_health": queue_health,
|
||||||
"capacity_plan": capacity_plan,
|
"capacity_plan": capacity_plan,
|
||||||
}
|
}
|
||||||
|
if not worker_expected_on_this_node:
|
||||||
|
detect_payload.update(
|
||||||
|
{
|
||||||
|
"phase_label": "当前节点不承载",
|
||||||
|
"phase_detail": "当前节点为海外控制面,仅承载 API 控制与同步接收,不执行本机检测任务。",
|
||||||
|
"recent_event": "",
|
||||||
|
"recent_warning": "",
|
||||||
|
"progress_percent": 0,
|
||||||
|
"active_thread_count": 0,
|
||||||
|
"max_thread_count": 0,
|
||||||
|
"available_proxy_count": 0,
|
||||||
|
"proxy_pool_count": 0,
|
||||||
|
"proxy_runtime_label": "不适用",
|
||||||
|
"proxy_runtime_detail": "当前节点不承载本机 Worker,代理、线程与检测阶段信息在此节点上不适用。",
|
||||||
|
"proxy_runtime_reason": "not_applicable",
|
||||||
|
"proxy_supplier_empty": False,
|
||||||
|
"proxy_last_refresh_status": "",
|
||||||
|
"proxy_last_refresh_time": "",
|
||||||
|
"proxy_last_refresh_source_count": 0,
|
||||||
|
"proxy_last_refresh_total_items": 0,
|
||||||
|
"proxy_last_validated_count": 0,
|
||||||
|
"proxy_last_available_count": 0,
|
||||||
|
"proxy_source_stats": [],
|
||||||
|
"dependency_alerts": [],
|
||||||
|
"active_job": None,
|
||||||
|
"runs_count": 0,
|
||||||
|
"worker_online": False,
|
||||||
|
}
|
||||||
|
)
|
||||||
detect_payload["participating_nodes"] = _build_participating_detect_nodes(
|
detect_payload["participating_nodes"] = _build_participating_detect_nodes(
|
||||||
cluster_snapshot=cluster_snapshot,
|
cluster_snapshot=cluster_snapshot,
|
||||||
detect_snapshot=detect_payload,
|
detect_snapshot=detect_payload,
|
||||||
@@ -351,6 +381,7 @@ def get_runtime_status() -> dict:
|
|||||||
"mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
|
"mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")),
|
||||||
"service_name": runtime_settings.get("worker_service_name", settings.worker_service_name),
|
"service_name": runtime_settings.get("worker_service_name", settings.worker_service_name),
|
||||||
"running": worker_runtime.get("running", False),
|
"running": worker_runtime.get("running", False),
|
||||||
|
"expected_on_this_node": worker_expected_on_this_node,
|
||||||
"process_count": worker_runtime.get("process_count", 0),
|
"process_count": worker_runtime.get("process_count", 0),
|
||||||
"latest_start_time": worker_runtime.get("latest_start_time", ""),
|
"latest_start_time": worker_runtime.get("latest_start_time", ""),
|
||||||
"message": worker_runtime.get("message", ""),
|
"message": worker_runtime.get("message", ""),
|
||||||
|
|||||||
@@ -25,6 +25,11 @@
|
|||||||
- `deploy/multi-region/bootstrap_mainland.sh`
|
- `deploy/multi-region/bootstrap_mainland.sh`
|
||||||
- `deploy/multi-region/check_cluster.sh`
|
- `deploy/multi-region/check_cluster.sh`
|
||||||
- `deploy/multi-region/check_mainland_controller.sh`
|
- `deploy/multi-region/check_mainland_controller.sh`
|
||||||
|
- `deploy/multi-region/check_mainland_worker.sh`
|
||||||
|
- `deploy/multi-region/check_worker_participation.sh`
|
||||||
|
- `deploy/multi-region/check_temp_overseas_control.sh`
|
||||||
|
- `deploy/multi-region/check_temp_link.sh`
|
||||||
|
- `deploy/multi-region/check_temp_topology.sh`
|
||||||
- `deploy/multi-region/simulate_cluster_node.py`
|
- `deploy/multi-region/simulate_cluster_node.py`
|
||||||
- `deploy/multi-region/simulate_multi_region.sh`
|
- `deploy/multi-region/simulate_multi_region.sh`
|
||||||
- `deploy/multi-region/prune_cluster_nodes.py`
|
- `deploy/multi-region/prune_cluster_nodes.py`
|
||||||
@@ -199,13 +204,48 @@ bash deploy/multi-region/check_mainland_controller.sh
|
|||||||
|
|
||||||
这条命令会直接检查:
|
这条命令会直接检查:
|
||||||
|
|
||||||
- `/etc/default/domaincheck-worker` 是否存在
|
- `/etc/default/domaincheck-api` 是否存在
|
||||||
- `NODE_ROLE=control` 是否正确
|
- `NODE_ROLE=control` 是否正确
|
||||||
- `SYNC_PUSH_ENABLED` / `SYNC_TARGET_API_BASE_URL` 是否已配置
|
- `SYNC_PUSH_ENABLED` / `SYNC_TARGET_API_BASE_URL` 是否已配置
|
||||||
|
- `domaincheck-api`
|
||||||
- `domaincheck-worker`
|
- `domaincheck-worker`
|
||||||
- `domaincheck-sync-agent`
|
- `domaincheck-sync-agent`
|
||||||
是否已启用并处于运行态
|
是否已启用并处于运行态
|
||||||
|
|
||||||
|
如果要判断“在线 Worker 有几台”和“当前真正领任务执行的是哪台”是否一致,可以执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /opt/domaincheck/domain-api
|
||||||
|
bash deploy/multi-region/check_worker_participation.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
这条命令会直接输出:
|
||||||
|
|
||||||
|
- `detect_worker_nodes` 当前在线节点概览
|
||||||
|
- 最新活跃任务的 `claimed_by / status / count`
|
||||||
|
- 一段压缩摘要 JSON
|
||||||
|
|
||||||
|
如果要检查纯大陆 worker 节点自己的配置和服务状态,可以执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /opt/domaincheck/domain-api
|
||||||
|
bash deploy/multi-region/check_mainland_worker.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
如果当前是“大陆 controller + 大陆 worker + 临时海外控制面”的联调拓扑,还可以执行:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /opt/domaincheck/domain-api
|
||||||
|
bash deploy/multi-region/check_temp_topology.sh http://127.0.0.1:8100 http://152.53.37.118:8100
|
||||||
|
```
|
||||||
|
|
||||||
|
这条命令会同时汇总:
|
||||||
|
|
||||||
|
- 大陆控制面的 `runtime/cluster`
|
||||||
|
- 大陆控制面的 `runtime/readiness`
|
||||||
|
- 最新活跃任务到底由谁 `claimed_by`
|
||||||
|
- 临时海外控制面的 `runtime/cluster`
|
||||||
|
|
||||||
如果当前还无法真正部署到大陆机器,也可以先在国外测试机上做“单机模拟多节点联调”:
|
如果当前还无法真正部署到大陆机器,也可以先在国外测试机上做“单机模拟多节点联调”:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
@@ -1,8 +1,9 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
ENV_FILE="${1:-/etc/default/domaincheck-worker}"
|
ENV_FILE="${1:-/etc/default/domaincheck-api}"
|
||||||
WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}"
|
WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}"
|
||||||
|
API_SERVICE="${API_SERVICE:-domaincheck-api}"
|
||||||
SYNC_AGENT_SERVICE="${SYNC_AGENT_SERVICE:-domaincheck-sync-agent}"
|
SYNC_AGENT_SERVICE="${SYNC_AGENT_SERVICE:-domaincheck-sync-agent}"
|
||||||
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||||
|
|
||||||
@@ -61,16 +62,22 @@ if errors:
|
|||||||
PY
|
PY
|
||||||
echo
|
echo
|
||||||
|
|
||||||
echo "[3/5] systemd status"
|
echo "[3/6] systemd status"
|
||||||
|
systemctl is-enabled "$API_SERVICE" || true
|
||||||
|
systemctl is-active "$API_SERVICE"
|
||||||
systemctl is-enabled "$WORKER_SERVICE" || true
|
systemctl is-enabled "$WORKER_SERVICE" || true
|
||||||
systemctl is-active "$WORKER_SERVICE"
|
systemctl is-active "$WORKER_SERVICE"
|
||||||
systemctl is-enabled "$SYNC_AGENT_SERVICE" || true
|
systemctl is-enabled "$SYNC_AGENT_SERVICE" || true
|
||||||
systemctl is-active "$SYNC_AGENT_SERVICE"
|
systemctl is-active "$SYNC_AGENT_SERVICE"
|
||||||
echo
|
echo
|
||||||
|
|
||||||
echo "[4/5] worker service detail"
|
echo "[4/6] api service detail"
|
||||||
|
systemctl status "$API_SERVICE" --no-pager -l | sed -n '1,25p'
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[5/6] worker service detail"
|
||||||
systemctl status "$WORKER_SERVICE" --no-pager -l | sed -n '1,25p'
|
systemctl status "$WORKER_SERVICE" --no-pager -l | sed -n '1,25p'
|
||||||
echo
|
echo
|
||||||
|
|
||||||
echo "[5/5] sync-agent detail"
|
echo "[6/6] sync-agent detail"
|
||||||
systemctl status "$SYNC_AGENT_SERVICE" --no-pager -l | sed -n '1,25p'
|
systemctl status "$SYNC_AGENT_SERVICE" --no-pager -l | sed -n '1,25p'
|
||||||
|
|||||||
70
domain-api/deploy/multi-region/check_mainland_worker.sh
Executable file
70
domain-api/deploy/multi-region/check_mainland_worker.sh
Executable file
@@ -0,0 +1,70 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ENV_FILE="${1:-/etc/default/domaincheck-worker}"
|
||||||
|
WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}"
|
||||||
|
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||||
|
|
||||||
|
echo "[1/4] environment file"
|
||||||
|
if [ ! -f "$ENV_FILE" ]; then
|
||||||
|
echo "missing env file: $ENV_FILE"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "env file: $ENV_FILE"
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[2/4] key env summary"
|
||||||
|
"${PYTHON_BIN}" - <<'PY' "$ENV_FILE"
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
env_path = Path(sys.argv[1])
|
||||||
|
values = {}
|
||||||
|
for line in env_path.read_text(encoding="utf-8").splitlines():
|
||||||
|
text = line.strip()
|
||||||
|
if not text or text.startswith("#") or "=" not in text:
|
||||||
|
continue
|
||||||
|
key, value = text.split("=", 1)
|
||||||
|
values[key.strip()] = value.strip()
|
||||||
|
|
||||||
|
required = {
|
||||||
|
"NODE_CODE": values.get("NODE_CODE", ""),
|
||||||
|
"NODE_REGION": values.get("NODE_REGION", ""),
|
||||||
|
"NODE_ROLE": values.get("NODE_ROLE", ""),
|
||||||
|
"DB_HOST": values.get("DB_HOST", ""),
|
||||||
|
"DB_PORT": values.get("DB_PORT", ""),
|
||||||
|
"REDIS_HOST": values.get("REDIS_HOST", ""),
|
||||||
|
"REDIS_PORT": values.get("REDIS_PORT", ""),
|
||||||
|
"SYNC_TARGET_API_BASE_URL": values.get("SYNC_TARGET_API_BASE_URL", ""),
|
||||||
|
}
|
||||||
|
print(json.dumps(required, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
errors = []
|
||||||
|
if required["NODE_REGION"] != "mainland":
|
||||||
|
errors.append("NODE_REGION 必须为 mainland")
|
||||||
|
if required["NODE_ROLE"] != "worker":
|
||||||
|
errors.append("NODE_ROLE 必须为 worker")
|
||||||
|
if not required["NODE_CODE"]:
|
||||||
|
errors.append("NODE_CODE 不能为空")
|
||||||
|
if not required["DB_HOST"]:
|
||||||
|
errors.append("DB_HOST 不能为空")
|
||||||
|
if not required["REDIS_HOST"]:
|
||||||
|
errors.append("REDIS_HOST 不能为空")
|
||||||
|
|
||||||
|
if errors:
|
||||||
|
print()
|
||||||
|
print("env validation failed:")
|
||||||
|
for item in errors:
|
||||||
|
print(f"- {item}")
|
||||||
|
sys.exit(1)
|
||||||
|
PY
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[3/4] systemd status"
|
||||||
|
systemctl is-enabled "$WORKER_SERVICE" || true
|
||||||
|
systemctl is-active "$WORKER_SERVICE"
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[4/4] worker service detail"
|
||||||
|
systemctl status "$WORKER_SERVICE" --no-pager -l | sed -n '1,30p'
|
||||||
80
domain-api/deploy/multi-region/check_temp_link.sh
Executable file
80
domain-api/deploy/multi-region/check_temp_link.sh
Executable file
@@ -0,0 +1,80 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
MAINLAND_BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||||
|
OVERSEAS_BASE_URL="${2:-http://152.53.37.118:8100}"
|
||||||
|
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||||
|
|
||||||
|
echo "[1/6] mainland readiness"
|
||||||
|
MAINLAND_READINESS="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/readiness")"
|
||||||
|
echo "${MAINLAND_READINESS}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[2/6] mainland sync summary"
|
||||||
|
MAINLAND_SYNC="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/sync-summary")"
|
||||||
|
echo "${MAINLAND_SYNC}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[3/6] mainland push sync"
|
||||||
|
MAINLAND_PUSH="$(curl -fsS -X POST "${MAINLAND_BASE_URL}/api/v1/runtime/actions/push_sync")"
|
||||||
|
echo "${MAINLAND_PUSH}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[4/6] overseas cluster"
|
||||||
|
OVERSEAS_CLUSTER="$(curl -fsS "${OVERSEAS_BASE_URL}/api/v1/runtime/cluster")"
|
||||||
|
echo "${OVERSEAS_CLUSTER}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[5/6] overseas sync summary"
|
||||||
|
OVERSEAS_SYNC="$(curl -fsS "${OVERSEAS_BASE_URL}/api/v1/runtime/sync-summary")"
|
||||||
|
echo "${OVERSEAS_SYNC}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[6/6] condensed summary"
|
||||||
|
"${PYTHON_BIN}" - <<'PY' "$MAINLAND_READINESS" "$MAINLAND_SYNC" "$MAINLAND_PUSH" "$OVERSEAS_CLUSTER" "$OVERSEAS_SYNC"
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
mainland_readiness = json.loads(sys.argv[1]).get("data", {})
|
||||||
|
mainland_sync = json.loads(sys.argv[2]).get("data", {})
|
||||||
|
mainland_push = json.loads(sys.argv[3])
|
||||||
|
overseas_cluster = json.loads(sys.argv[4]).get("data", {})
|
||||||
|
overseas_sync = json.loads(sys.argv[5]).get("data", {})
|
||||||
|
|
||||||
|
cluster_nodes = overseas_cluster.get("nodes") or []
|
||||||
|
cluster_summary = overseas_cluster.get("summary") or {}
|
||||||
|
type_counts = overseas_sync.get("type_counts") or {}
|
||||||
|
status_counts = overseas_sync.get("status_counts") or {}
|
||||||
|
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"mainland": {
|
||||||
|
"readiness_status": mainland_readiness.get("status", ""),
|
||||||
|
"readiness_summary": mainland_readiness.get("summary", ""),
|
||||||
|
"sync_enabled": bool(mainland_sync.get("enabled", False)),
|
||||||
|
"target_api_base_url": mainland_sync.get("target_api_base_url", ""),
|
||||||
|
"push_sync_code": mainland_push.get("code"),
|
||||||
|
"push_sync_message": mainland_push.get("message", ""),
|
||||||
|
},
|
||||||
|
"overseas": {
|
||||||
|
"nodes_total": overseas_cluster.get("nodes_total", 0),
|
||||||
|
"online_control_nodes": cluster_summary.get("online_control_nodes", 0),
|
||||||
|
"online_worker_nodes": cluster_summary.get("online_worker_nodes", 0),
|
||||||
|
"node_codes": [str(item.get("node_code") or "") for item in cluster_nodes],
|
||||||
|
"runtime_ingest": type_counts.get("runtime_ingest", 0),
|
||||||
|
"detect_result_ingest": type_counts.get("detect_result_ingest", 0),
|
||||||
|
"status_received": status_counts.get("received", 0),
|
||||||
|
"status_success": status_counts.get("success", 0),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
PY
|
||||||
75
domain-api/deploy/multi-region/check_temp_overseas_control.sh
Executable file
75
domain-api/deploy/multi-region/check_temp_overseas_control.sh
Executable file
@@ -0,0 +1,75 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||||
|
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||||
|
|
||||||
|
echo "[1/5] health"
|
||||||
|
curl -fsS "${BASE_URL}/health"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[2/5] runtime status"
|
||||||
|
STATUS_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/status")"
|
||||||
|
echo "${STATUS_JSON}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[3/5] runtime cluster"
|
||||||
|
curl -fsS "${BASE_URL}/api/v1/runtime/cluster"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[4/5] runtime readiness"
|
||||||
|
curl -fsS "${BASE_URL}/api/v1/runtime/readiness"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[5/5] condensed summary"
|
||||||
|
"${PYTHON_BIN}" - <<'PY' "$STATUS_JSON"
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
data = json.loads(sys.argv[1]).get("data", {})
|
||||||
|
node = data.get("node") or {}
|
||||||
|
worker = data.get("worker") or {}
|
||||||
|
sync_agent = data.get("sync_agent") or {}
|
||||||
|
detect = data.get("detect") or {}
|
||||||
|
cluster = data.get("cluster") or {}
|
||||||
|
summary = cluster.get("summary") or {}
|
||||||
|
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"node": {
|
||||||
|
"code": node.get("code", ""),
|
||||||
|
"region": node.get("region", ""),
|
||||||
|
"role": node.get("role", ""),
|
||||||
|
},
|
||||||
|
"worker": {
|
||||||
|
"running": bool(worker.get("running", False)),
|
||||||
|
"expected_on_this_node": bool(worker.get("expected_on_this_node", True)),
|
||||||
|
"message": worker.get("message", ""),
|
||||||
|
},
|
||||||
|
"sync_agent": {
|
||||||
|
"running": bool(sync_agent.get("running", False)),
|
||||||
|
"expected_on_this_node": bool(sync_agent.get("expected_on_this_node", False)),
|
||||||
|
"message": sync_agent.get("message", ""),
|
||||||
|
},
|
||||||
|
"detect": {
|
||||||
|
"phase_label": detect.get("phase_label", ""),
|
||||||
|
"proxy_runtime_label": detect.get("proxy_runtime_label", ""),
|
||||||
|
"active_job": (detect.get("active_job") or {}).get("job_code", ""),
|
||||||
|
},
|
||||||
|
"cluster": {
|
||||||
|
"nodes_total": cluster.get("nodes_total", 0),
|
||||||
|
"online_control_nodes": summary.get("online_control_nodes", 0),
|
||||||
|
"online_worker_nodes": summary.get("online_worker_nodes", 0),
|
||||||
|
"dedicated_online_worker_nodes": summary.get("dedicated_online_worker_nodes", 0),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
PY
|
||||||
126
domain-api/deploy/multi-region/check_temp_topology.sh
Executable file
126
domain-api/deploy/multi-region/check_temp_topology.sh
Executable file
@@ -0,0 +1,126 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
MAINLAND_BASE_URL="${1:-http://127.0.0.1:8100}"
|
||||||
|
OVERSEAS_BASE_URL="${2:-http://152.53.37.118:8100}"
|
||||||
|
PYTHON_BIN="${PYTHON_BIN:-python3}"
|
||||||
|
|
||||||
|
echo "[1/5] mainland runtime cluster"
|
||||||
|
MAINLAND_CLUSTER="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/cluster")"
|
||||||
|
echo "${MAINLAND_CLUSTER}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[2/5] mainland runtime readiness"
|
||||||
|
MAINLAND_READINESS="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/readiness")"
|
||||||
|
echo "${MAINLAND_READINESS}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[3/5] mainland worker participation"
|
||||||
|
PARTICIPATION_JSON="$(/www/server/pgsql/bin/psql -U postgres -d domain -At <<'SQL'
|
||||||
|
with latest_job as (
|
||||||
|
select id, job_code, status
|
||||||
|
from detect_jobs
|
||||||
|
where status in ('pending','running')
|
||||||
|
order by id desc
|
||||||
|
limit 1
|
||||||
|
),
|
||||||
|
node_stats as (
|
||||||
|
select
|
||||||
|
node_code,
|
||||||
|
role,
|
||||||
|
status,
|
||||||
|
coalesce(metadata_json->>'worker_online', '') as worker_online,
|
||||||
|
coalesce(metadata_json->>'detect_participating', '') as detect_participating
|
||||||
|
from detect_worker_nodes
|
||||||
|
),
|
||||||
|
item_stats as (
|
||||||
|
select
|
||||||
|
coalesce(claimed_by, '') as claimed_by,
|
||||||
|
status,
|
||||||
|
count(*) as cnt
|
||||||
|
from detect_job_items
|
||||||
|
where job_id = (select id from latest_job)
|
||||||
|
group by claimed_by, status
|
||||||
|
)
|
||||||
|
select json_build_object(
|
||||||
|
'job', (
|
||||||
|
select json_build_object(
|
||||||
|
'job_code', coalesce(job_code, ''),
|
||||||
|
'status', coalesce(status, '')
|
||||||
|
)
|
||||||
|
from latest_job
|
||||||
|
),
|
||||||
|
'online_nodes', (
|
||||||
|
select coalesce(json_agg(json_build_object(
|
||||||
|
'node_code', node_code,
|
||||||
|
'role', role,
|
||||||
|
'status', status,
|
||||||
|
'worker_online', worker_online,
|
||||||
|
'detect_participating', detect_participating
|
||||||
|
) order by node_code), '[]'::json)
|
||||||
|
from node_stats
|
||||||
|
where status in ('online', 'busy')
|
||||||
|
),
|
||||||
|
'claimed_distribution', (
|
||||||
|
select coalesce(json_agg(json_build_object(
|
||||||
|
'claimed_by', claimed_by,
|
||||||
|
'status', status,
|
||||||
|
'count', cnt
|
||||||
|
) order by claimed_by, status), '[]'::json)
|
||||||
|
from item_stats
|
||||||
|
)
|
||||||
|
);
|
||||||
|
SQL
|
||||||
|
)"
|
||||||
|
echo "${PARTICIPATION_JSON}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[4/5] overseas runtime cluster"
|
||||||
|
OVERSEAS_CLUSTER="$(curl -fsS "${OVERSEAS_BASE_URL}/api/v1/runtime/cluster")"
|
||||||
|
echo "${OVERSEAS_CLUSTER}"
|
||||||
|
echo
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[5/5] condensed summary"
|
||||||
|
"${PYTHON_BIN}" - <<'PY' "$MAINLAND_CLUSTER" "$MAINLAND_READINESS" "$PARTICIPATION_JSON" "$OVERSEAS_CLUSTER"
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
|
||||||
|
mainland_cluster = json.loads(sys.argv[1]).get("data", {})
|
||||||
|
mainland_readiness = json.loads(sys.argv[2]).get("data", {})
|
||||||
|
participation = json.loads(sys.argv[3])
|
||||||
|
overseas_cluster = json.loads(sys.argv[4]).get("data", {})
|
||||||
|
|
||||||
|
mainland_summary = mainland_cluster.get("summary") or {}
|
||||||
|
overseas_summary = overseas_cluster.get("summary") or {}
|
||||||
|
|
||||||
|
def node_codes(cluster):
|
||||||
|
return [str(item.get("node_code") or "") for item in (cluster.get("nodes") or [])]
|
||||||
|
|
||||||
|
print(
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"mainland": {
|
||||||
|
"readiness_status": mainland_readiness.get("status", ""),
|
||||||
|
"readiness_summary": mainland_readiness.get("summary", ""),
|
||||||
|
"online_worker_nodes": mainland_summary.get("online_worker_nodes", 0),
|
||||||
|
"dedicated_online_worker_nodes": mainland_summary.get("dedicated_online_worker_nodes", 0),
|
||||||
|
"online_control_nodes": mainland_summary.get("online_control_nodes", 0),
|
||||||
|
"node_codes": node_codes(mainland_cluster),
|
||||||
|
},
|
||||||
|
"participation": participation,
|
||||||
|
"overseas": {
|
||||||
|
"online_worker_nodes": overseas_summary.get("online_worker_nodes", 0),
|
||||||
|
"dedicated_online_worker_nodes": overseas_summary.get("dedicated_online_worker_nodes", 0),
|
||||||
|
"online_control_nodes": overseas_summary.get("online_control_nodes", 0),
|
||||||
|
"node_codes": node_codes(overseas_cluster),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
ensure_ascii=False,
|
||||||
|
indent=2,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
PY
|
||||||
97
domain-api/deploy/multi-region/check_worker_participation.sh
Executable file
97
domain-api/deploy/multi-region/check_worker_participation.sh
Executable file
@@ -0,0 +1,97 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
DB_NAME="${DB_NAME:-domain}"
|
||||||
|
DB_USER="${DB_USER:-postgres}"
|
||||||
|
PSQL_BIN="${PSQL_BIN:-/www/server/pgsql/bin/psql}"
|
||||||
|
|
||||||
|
echo "[1/3] cluster nodes"
|
||||||
|
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -Atc "
|
||||||
|
select
|
||||||
|
node_code || '|' ||
|
||||||
|
role || '|' ||
|
||||||
|
status || '|' ||
|
||||||
|
coalesce(current_load, 0) || '|' ||
|
||||||
|
to_char(last_heartbeat_at, 'YYYY-MM-DD HH24:MI:SS') || '|' ||
|
||||||
|
coalesce(metadata_json->>'worker_online', '') || '|' ||
|
||||||
|
coalesce(metadata_json->>'detect_participating', '')
|
||||||
|
from detect_worker_nodes
|
||||||
|
order by node_code;
|
||||||
|
"
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[2/3] active job distribution"
|
||||||
|
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -Atc "
|
||||||
|
select
|
||||||
|
coalesce(claimed_by, '') || '|' ||
|
||||||
|
status || '|' ||
|
||||||
|
count(*)
|
||||||
|
from detect_job_items
|
||||||
|
where job_id = (
|
||||||
|
select id
|
||||||
|
from detect_jobs
|
||||||
|
where status in ('pending','running')
|
||||||
|
order by id desc
|
||||||
|
limit 1
|
||||||
|
)
|
||||||
|
group by claimed_by, status
|
||||||
|
order by claimed_by, status;
|
||||||
|
"
|
||||||
|
echo
|
||||||
|
|
||||||
|
echo "[3/3] condensed summary"
|
||||||
|
"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -At <<'SQL'
|
||||||
|
with latest_job as (
|
||||||
|
select id, job_code, status
|
||||||
|
from detect_jobs
|
||||||
|
where status in ('pending','running')
|
||||||
|
order by id desc
|
||||||
|
limit 1
|
||||||
|
),
|
||||||
|
node_stats as (
|
||||||
|
select
|
||||||
|
node_code,
|
||||||
|
role,
|
||||||
|
status,
|
||||||
|
coalesce(metadata_json->>'worker_online', '') as worker_online,
|
||||||
|
coalesce(metadata_json->>'detect_participating', '') as detect_participating
|
||||||
|
from detect_worker_nodes
|
||||||
|
),
|
||||||
|
item_stats as (
|
||||||
|
select
|
||||||
|
coalesce(claimed_by, '') as claimed_by,
|
||||||
|
status,
|
||||||
|
count(*) as cnt
|
||||||
|
from detect_job_items
|
||||||
|
where job_id = (select id from latest_job)
|
||||||
|
group by claimed_by, status
|
||||||
|
)
|
||||||
|
select json_build_object(
|
||||||
|
'job', (
|
||||||
|
select json_build_object(
|
||||||
|
'job_code', coalesce(job_code, ''),
|
||||||
|
'status', coalesce(status, '')
|
||||||
|
)
|
||||||
|
from latest_job
|
||||||
|
),
|
||||||
|
'online_nodes', (
|
||||||
|
select coalesce(json_agg(json_build_object(
|
||||||
|
'node_code', node_code,
|
||||||
|
'role', role,
|
||||||
|
'status', status,
|
||||||
|
'worker_online', worker_online,
|
||||||
|
'detect_participating', detect_participating
|
||||||
|
) order by node_code), '[]'::json)
|
||||||
|
from node_stats
|
||||||
|
where status in ('online', 'busy')
|
||||||
|
),
|
||||||
|
'claimed_distribution', (
|
||||||
|
select coalesce(json_agg(json_build_object(
|
||||||
|
'claimed_by', claimed_by,
|
||||||
|
'status', status,
|
||||||
|
'count', cnt
|
||||||
|
) order by claimed_by, status), '[]'::json)
|
||||||
|
from item_stats
|
||||||
|
)
|
||||||
|
);
|
||||||
|
SQL
|
||||||
@@ -24,8 +24,8 @@
|
|||||||
<span class="status-pill" :class="runtime.apiOnline ? 'online' : 'offline'">
|
<span class="status-pill" :class="runtime.apiOnline ? 'online' : 'offline'">
|
||||||
API {{ runtime.apiOnline ? "在线" : "离线" }}
|
API {{ runtime.apiOnline ? "在线" : "离线" }}
|
||||||
</span>
|
</span>
|
||||||
<span class="status-pill" :class="runtime.localWorkerOnline ? 'online' : 'offline'">
|
<span class="status-pill" :class="runtime.localWorkerExpected ? (runtime.localWorkerOnline ? 'online' : 'offline') : 'neutral'">
|
||||||
本机 Worker {{ runtime.localWorkerOnline ? "在线" : "离线" }}
|
本机 Worker {{ runtime.localWorkerExpected ? (runtime.localWorkerOnline ? "在线" : "离线") : "不承载" }}
|
||||||
</span>
|
</span>
|
||||||
<span class="status-pill" :class="runtime.effectiveWorkerNodes > 0 ? 'online' : 'offline'">
|
<span class="status-pill" :class="runtime.effectiveWorkerNodes > 0 ? 'online' : 'offline'">
|
||||||
有效执行节点 {{ runtime.effectiveWorkerNodes }}
|
有效执行节点 {{ runtime.effectiveWorkerNodes }}
|
||||||
@@ -57,6 +57,7 @@ const authStore = useAuthStore();
|
|||||||
const runtime = ref({
|
const runtime = ref({
|
||||||
apiOnline: false,
|
apiOnline: false,
|
||||||
localWorkerOnline: false,
|
localWorkerOnline: false,
|
||||||
|
localWorkerExpected: true,
|
||||||
effectiveWorkerNodes: 0,
|
effectiveWorkerNodes: 0,
|
||||||
workerMode: "windows-local"
|
workerMode: "windows-local"
|
||||||
});
|
});
|
||||||
@@ -85,6 +86,7 @@ const loadRuntimeSummary = async () => {
|
|||||||
runtime.value = {
|
runtime.value = {
|
||||||
apiOnline: Boolean(response.data?.api?.pid),
|
apiOnline: Boolean(response.data?.api?.pid),
|
||||||
localWorkerOnline: Boolean(response.data?.worker?.running),
|
localWorkerOnline: Boolean(response.data?.worker?.running),
|
||||||
|
localWorkerExpected: Boolean(response.data?.worker?.expected_on_this_node ?? true),
|
||||||
effectiveWorkerNodes: Number(summary.online_worker_nodes || 0),
|
effectiveWorkerNodes: Number(summary.online_worker_nodes || 0),
|
||||||
workerMode: response.data?.worker?.mode || "windows-local"
|
workerMode: response.data?.worker?.mode || "windows-local"
|
||||||
};
|
};
|
||||||
@@ -92,6 +94,7 @@ const loadRuntimeSummary = async () => {
|
|||||||
runtime.value = {
|
runtime.value = {
|
||||||
apiOnline: false,
|
apiOnline: false,
|
||||||
localWorkerOnline: false,
|
localWorkerOnline: false,
|
||||||
|
localWorkerExpected: true,
|
||||||
effectiveWorkerNodes: 0,
|
effectiveWorkerNodes: 0,
|
||||||
workerMode: "unknown"
|
workerMode: "unknown"
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -80,12 +80,20 @@ const loadOverview = async (showError = true) => {
|
|||||||
loading.value = true;
|
loading.value = true;
|
||||||
try {
|
try {
|
||||||
const { data } = await dashboardApi.overview();
|
const { data } = await dashboardApi.overview();
|
||||||
|
const localWorkerValue =
|
||||||
|
data.local_worker_status === "not-applicable"
|
||||||
|
? "不承载"
|
||||||
|
: data.local_worker_status || data.worker_status;
|
||||||
|
const localWorkerNote =
|
||||||
|
data.local_worker_status === "not-applicable"
|
||||||
|
? `当前节点 ${data.node_region || "-"} / ${data.node_role || "-"},默认不承载本机 Worker`
|
||||||
|
: "当前节点本机 Worker 运行状态";
|
||||||
stats.value = [
|
stats.value = [
|
||||||
{ label: "总域名", value: String(data.domains_total), note: "数据库域名总量" },
|
{ label: "总域名", value: String(data.domains_total), note: "数据库域名总量" },
|
||||||
{ label: "待检测", value: String(data.pending_total), note: "当前待处理域名" },
|
{ label: "待检测", value: String(data.pending_total), note: "当前待处理域名" },
|
||||||
{ label: "黑名单", value: String(data.blacklist_total), note: "命中风险域名" },
|
{ label: "黑名单", value: String(data.blacklist_total), note: "命中风险域名" },
|
||||||
{ label: "API", value: data.api_status, note: "当前 API 运行状态" },
|
{ label: "API", value: data.api_status, note: "当前 API 运行状态" },
|
||||||
{ label: "本机 Worker", value: data.local_worker_status || data.worker_status, note: "当前节点本机 Worker 运行状态" },
|
{ label: "本机 Worker", value: localWorkerValue, note: localWorkerNote },
|
||||||
{
|
{
|
||||||
label: "有效执行节点",
|
label: "有效执行节点",
|
||||||
value: `${data.cluster_worker_status || "offline"} / ${data.cluster_online_worker_nodes || 0}`,
|
value: `${data.cluster_worker_status || "offline"} / ${data.cluster_online_worker_nodes || 0}`,
|
||||||
|
|||||||
@@ -24,7 +24,9 @@
|
|||||||
<div class="summary-card">
|
<div class="summary-card">
|
||||||
<div class="summary-title">本机 Worker</div>
|
<div class="summary-title">本机 Worker</div>
|
||||||
<div class="summary-value">{{ runtime.worker?.service_name || "-" }}</div>
|
<div class="summary-value">{{ runtime.worker?.service_name || "-" }}</div>
|
||||||
<el-tag :type="runtime.worker?.running ? 'success' : 'danger'">{{ runtime.worker?.running ? "本机在线" : "本机离线" }}</el-tag>
|
<el-tag :type="localWorkerTagType">
|
||||||
|
{{ localWorkerSummaryText }}
|
||||||
|
</el-tag>
|
||||||
</div>
|
</div>
|
||||||
<div class="summary-card">
|
<div class="summary-card">
|
||||||
<div class="summary-title">运行模式</div>
|
<div class="summary-title">运行模式</div>
|
||||||
@@ -118,8 +120,8 @@
|
|||||||
<span class="status-chip" :class="runtime.api?.pid ? 'ok' : 'neutral'">
|
<span class="status-chip" :class="runtime.api?.pid ? 'ok' : 'neutral'">
|
||||||
API {{ runtime.api?.pid ? "运行中" : "未知" }}
|
API {{ runtime.api?.pid ? "运行中" : "未知" }}
|
||||||
</span>
|
</span>
|
||||||
<span class="status-chip" :class="runtime.worker?.running ? 'ok' : 'danger'">
|
<span class="status-chip" :class="localWorkerChipClass">
|
||||||
本机 Worker {{ runtime.worker?.running ? "在线" : "离线" }}
|
本机 Worker {{ localWorkerStripText }}
|
||||||
</span>
|
</span>
|
||||||
<span class="status-chip" :class="(runtime.cluster?.summary?.online_worker_nodes || 0) > 0 ? 'ok' : 'danger'">
|
<span class="status-chip" :class="(runtime.cluster?.summary?.online_worker_nodes || 0) > 0 ? 'ok' : 'danger'">
|
||||||
有效执行节点 {{ runtime.cluster?.summary?.online_worker_nodes || 0 }}
|
有效执行节点 {{ runtime.cluster?.summary?.online_worker_nodes || 0 }}
|
||||||
@@ -151,10 +153,10 @@
|
|||||||
<el-descriptions-item label="API PID">{{ runtime.api?.pid || "-" }}</el-descriptions-item>
|
<el-descriptions-item label="API PID">{{ runtime.api?.pid || "-" }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="API 地址">{{ runtime.api?.health_url || "-" }}</el-descriptions-item>
|
<el-descriptions-item label="API 地址">{{ runtime.api?.health_url || "-" }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="Worker 服务">{{ runtime.worker?.service_name || "-" }}</el-descriptions-item>
|
<el-descriptions-item label="Worker 服务">{{ runtime.worker?.service_name || "-" }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="Worker 在线">{{ runtime.worker?.running ? "是" : "否" }}</el-descriptions-item>
|
<el-descriptions-item label="Worker 在线">{{ localWorkerBooleanText }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="运行模式">{{ runtime.worker?.mode || "-" }}</el-descriptions-item>
|
<el-descriptions-item label="运行模式">{{ runtime.worker?.mode || "-" }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="进程数">{{ runtime.worker?.process_count ?? "-" }}</el-descriptions-item>
|
<el-descriptions-item label="进程数">{{ runtime.worker?.process_count ?? "-" }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="运行消息">{{ runtime.worker?.message || "-" }}</el-descriptions-item>
|
<el-descriptions-item label="运行消息">{{ localWorkerMessage }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="Sync Agent 服务">{{ runtime.sync_agent?.service_name || "-" }}</el-descriptions-item>
|
<el-descriptions-item label="Sync Agent 服务">{{ runtime.sync_agent?.service_name || "-" }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="Sync Agent 在线">{{ runtime.sync_agent?.running ? "是" : "否" }}</el-descriptions-item>
|
<el-descriptions-item label="Sync Agent 在线">{{ runtime.sync_agent?.running ? "是" : "否" }}</el-descriptions-item>
|
||||||
<el-descriptions-item label="Sync Agent 启动时间">{{ runtime.sync_agent?.latest_start_time || "-" }}</el-descriptions-item>
|
<el-descriptions-item label="Sync Agent 启动时间">{{ runtime.sync_agent?.latest_start_time || "-" }}</el-descriptions-item>
|
||||||
@@ -166,8 +168,21 @@
|
|||||||
<div class="action-panel">
|
<div class="action-panel">
|
||||||
<h3>运行时动作</h3>
|
<h3>运行时动作</h3>
|
||||||
<div class="action-buttons">
|
<div class="action-buttons">
|
||||||
<el-button type="primary" :loading="actionLoading === 'start_worker'" @click="invokeAction('start_worker')">启动 Worker</el-button>
|
<el-button
|
||||||
<el-button :loading="actionLoading === 'stop_worker'" @click="invokeAction('stop_worker')">停止 Worker</el-button>
|
v-if="runtime.worker?.expected_on_this_node"
|
||||||
|
type="primary"
|
||||||
|
:loading="actionLoading === 'start_worker'"
|
||||||
|
@click="invokeAction('start_worker')"
|
||||||
|
>
|
||||||
|
启动 Worker
|
||||||
|
</el-button>
|
||||||
|
<el-button
|
||||||
|
v-if="runtime.worker?.expected_on_this_node"
|
||||||
|
:loading="actionLoading === 'stop_worker'"
|
||||||
|
@click="invokeAction('stop_worker')"
|
||||||
|
>
|
||||||
|
停止 Worker
|
||||||
|
</el-button>
|
||||||
<el-button type="warning" plain :loading="actionLoading === 'restart_api'" @click="invokeAction('restart_api')">重启 API</el-button>
|
<el-button type="warning" plain :loading="actionLoading === 'restart_api'" @click="invokeAction('restart_api')">重启 API</el-button>
|
||||||
<el-button
|
<el-button
|
||||||
v-if="runtime.sync_agent?.expected_on_this_node"
|
v-if="runtime.sync_agent?.expected_on_this_node"
|
||||||
@@ -201,7 +216,15 @@
|
|||||||
show-icon
|
show-icon
|
||||||
type="info"
|
type="info"
|
||||||
style="margin-top: 12px"
|
style="margin-top: 12px"
|
||||||
:title="`当前本机 Worker:${runtime.worker?.running ? '运行中' : '未运行'};有效执行节点:${runtime.cluster?.summary?.online_worker_nodes || 0};独立 Worker:${runtime.cluster?.summary?.dedicated_online_worker_nodes || 0};当前 API 状态:${runtime.api?.pid ? '运行中' : '未知'};Sync Agent:${runtime.sync_agent?.running ? '运行中' : '未运行'}。`"
|
:title="`当前本机 Worker:${localWorkerActionText};有效执行节点:${runtime.cluster?.summary?.online_worker_nodes || 0};独立 Worker:${runtime.cluster?.summary?.dedicated_online_worker_nodes || 0};当前 API 状态:${runtime.api?.pid ? '运行中' : '未知'};Sync Agent:${runtime.sync_agent?.running ? '运行中' : runtime.sync_agent?.expected_on_this_node ? '未运行' : '当前节点不承载'}。`"
|
||||||
|
/>
|
||||||
|
<el-alert
|
||||||
|
v-if="!runtime.worker?.expected_on_this_node"
|
||||||
|
:closable="false"
|
||||||
|
show-icon
|
||||||
|
type="info"
|
||||||
|
style="margin-top: 12px"
|
||||||
|
:title="`当前节点身份为 ${runtime.node?.region || '-'} / ${runtime.node?.role || '-'},按当前部署角色默认不承载本机 Worker;检测执行应由大陆 controller / worker 节点承担。`"
|
||||||
/>
|
/>
|
||||||
<el-alert
|
<el-alert
|
||||||
v-if="!runtime.sync_agent?.expected_on_this_node"
|
v-if="!runtime.sync_agent?.expected_on_this_node"
|
||||||
@@ -800,6 +823,45 @@ const proxyDiagnosisText = () => {
|
|||||||
return runtime.detect?.proxy_runtime_detail || "当前暂无代理运行说明";
|
return runtime.detect?.proxy_runtime_detail || "当前暂无代理运行说明";
|
||||||
};
|
};
|
||||||
|
|
||||||
|
const localWorkerExpected = computed(() => Boolean(runtime.worker?.expected_on_this_node ?? true));
|
||||||
|
|
||||||
|
const localWorkerSummaryText = computed(() => {
|
||||||
|
if (!localWorkerExpected.value) return "当前节点不承载";
|
||||||
|
return runtime.worker?.running ? "本机在线" : "本机离线";
|
||||||
|
});
|
||||||
|
|
||||||
|
const localWorkerStripText = computed(() => {
|
||||||
|
if (!localWorkerExpected.value) return "不承载";
|
||||||
|
return runtime.worker?.running ? "在线" : "离线";
|
||||||
|
});
|
||||||
|
|
||||||
|
const localWorkerBooleanText = computed(() => {
|
||||||
|
if (!localWorkerExpected.value) return "当前节点不承载";
|
||||||
|
return runtime.worker?.running ? "是" : "否";
|
||||||
|
});
|
||||||
|
|
||||||
|
const localWorkerMessage = computed(() => {
|
||||||
|
if (!localWorkerExpected.value) {
|
||||||
|
return `当前节点 ${runtime.node?.region || "-"} / ${runtime.node?.role || "-"},默认不承载本机 Worker`;
|
||||||
|
}
|
||||||
|
return runtime.worker?.message || "-";
|
||||||
|
});
|
||||||
|
|
||||||
|
const localWorkerActionText = computed(() => {
|
||||||
|
if (!localWorkerExpected.value) return "当前节点不承载";
|
||||||
|
return runtime.worker?.running ? "运行中" : "未运行";
|
||||||
|
});
|
||||||
|
|
||||||
|
const localWorkerTagType = computed(() => {
|
||||||
|
if (!localWorkerExpected.value) return "info";
|
||||||
|
return runtime.worker?.running ? "success" : "danger";
|
||||||
|
});
|
||||||
|
|
||||||
|
const localWorkerChipClass = computed(() => {
|
||||||
|
if (!localWorkerExpected.value) return "neutral";
|
||||||
|
return runtime.worker?.running ? "ok" : "danger";
|
||||||
|
});
|
||||||
|
|
||||||
const clusterIssueCount = computed(() => {
|
const clusterIssueCount = computed(() => {
|
||||||
const summary = runtime.cluster?.summary || {};
|
const summary = runtime.cluster?.summary || {};
|
||||||
return Number(summary.status_counts?.stale || 0) + Number(summary.status_counts?.offline || 0);
|
return Number(summary.status_counts?.stale || 0) + Number(summary.status_counts?.offline || 0);
|
||||||
|
|||||||
1
domainCheck/node_thread_counts.json
Normal file
1
domainCheck/node_thread_counts.json
Normal file
@@ -0,0 +1 @@
|
|||||||
|
{}
|
||||||
@@ -1,13 +1,13 @@
|
|||||||
{
|
{
|
||||||
"proxy_enable": true,
|
"proxy_enable": true,
|
||||||
"allow_direct": true,
|
"allow_direct": true,
|
||||||
"proxy_url": "http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50",
|
"proxy_url": "http://211.101.244.154:18003/getProxy_batch.php?group=B&count=50",
|
||||||
"proxy_urls": [
|
"proxy_urls": [
|
||||||
"http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50",
|
"http://211.101.244.154:18003/getProxy_batch.php?group=B&count=50",
|
||||||
"http://211.101.244.154:18008/getProxy_batch.php?group=C&count=50",
|
"http://211.101.244.154:18003/getProxy_batch.php?group=C&count=50",
|
||||||
"http://211.101.244.154:18008/getProxy_batch.php?group=D&count=50",
|
"http://211.101.244.154:18003/getProxy_batch.php?group=D&count=50",
|
||||||
"http://211.101.244.154:18008/getProxy_batch.php?group=E&count=50",
|
"http://211.101.244.154:18003/getProxy_batch.php?group=E&count=50",
|
||||||
"http://211.101.244.154:18008/getProxy_batch.php?group=F&count=50",
|
"http://211.101.244.154:18008/getProxy_batch.php?group=F&count=50",
|
||||||
"http://211.101.244.154:18008/getProxy_batch.php?group=G&count=50"
|
"http://211.101.244.154:18003/getProxy_batch.php?group=G&count=50"
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
Reference in New Issue
Block a user