diff --git a/docs/12_domainCheck_导航索引.md b/docs/12_domainCheck_导航索引.md index 9b265c8..80fc284 100644 --- a/docs/12_domainCheck_导航索引.md +++ b/docs/12_domainCheck_导航索引.md @@ -36,6 +36,10 @@ - `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md` +### 9. 临时海外控制面联调清单 + +- `docs/20_domainCheck_临时海外控制面联调清单.md` + ## 二、文档阅读顺序 ### 1. 先看总体方案 @@ -53,6 +57,7 @@ - `docs/16_domainCheck_多机检测与跨地域部署设计.md` - `docs/17_domainCheck_全流程部署实操手册.md` - `docs/18_domainCheck_CentOS9一键复制部署与更新文档.md` +- `docs/20_domainCheck_临时海外控制面联调清单.md` ### 3. 如果要回溯历史需求与问题 diff --git a/docs/20_domainCheck_临时海外控制面联调清单.md b/docs/20_domainCheck_临时海外控制面联调清单.md new file mode 100644 index 0000000..e044403 --- /dev/null +++ b/docs/20_domainCheck_临时海外控制面联调清单.md @@ -0,0 +1,231 @@ +# domainCheck 临时海外控制面联调清单 + +适用场景： + +- 暂时不使用正式海外机器 +- 先把大陆 `controller + worker` 对接到当前测试海外机 +- 先跑通“控制面 + 同步 + 观测”全链路，再切回正式海外环境 + +当前临时海外控制面： + +- 节点编码：`overseas-control-01` +- 节点角色：`overseas / control` +- 机器 IP：`152.53.37.118` +- 后台地址：`http://152.53.37.118/` +- API 基址：`http://152.53.37.118:8100` + +## 1. 当前目标拓扑 + +本次联调按下面的角色分工： + +- 大陆 controller + - 承担 API 控制面 + - 承担 Sync Agent + - 可以兼跑本机 Worker +- 大陆 worker + - 只承担检测执行 +- 临时海外控制面 + - 只承担 API 控制与同步接收 + - 不承载本机 Worker + - 不承载 Sync Agent + +## 2. 临时海外控制面应有状态 + +在临时海外控制面机器上执行： + +```bash +bash domain-api/deploy/multi-region/check_temp_overseas_control.sh +``` + +预期： + +- `domaincheck-api` 为 `active (running)` +- `domaincheck-worker` 为 `inactive (dead)` 或禁用 +- `runtime.status.data.node` + - `code=overseas-control-01` + - `region=overseas` + - `role=control` +- `runtime.status.data.worker.expected_on_this_node=false` +- `runtime.status.data.sync_agent.expected_on_this_node=false` +- `runtime.status.data.detect.phase_label=当前节点不承载` + +## 3. 大陆 controller 切到临时海外控制面 + +在大陆 controller 上执行： + +```bash +sed -i 's#^SYNC_TARGET_API_BASE_URL=.*#SYNC_TARGET_API_BASE_URL=http://152.53.37.118:8100#' /etc/default/domaincheck-api +sed -i 's/^SYNC_PUSH_ENABLED=.*/SYNC_PUSH_ENABLED=true/' /etc/default/domaincheck-api + +systemctl restart domaincheck-api +systemctl restart domaincheck-sync-agent +systemctl restart domaincheck-worker +sleep 3 +``` + +然后检查： + +```bash +curl -s http://127.0.0.1:8100/api/v1/runtime/readiness +echo +curl -s http://127.0.0.1:8100/api/v1/runtime/sync-summary +``` + +预期： + +- `runtime/readiness` 至少不是大陆本机配置错误导致的 `blocking` +- `runtime/sync-summary.data.enabled=true` +- `runtime/sync-summary.data.target_api_base_url=http://152.53.37.118:8100` + +## 4. 大陆 worker 检查项 + +在大陆 worker 上执行： + +```bash +bash domain-api/deploy/multi-region/check_mainland_worker.sh +``` + +预期： + +- `NODE_CODE=mainland-worker-01` +- `NODE_REGION=mainland` +- `NODE_ROLE=worker` +- `domaincheck-worker` 为 `active (running)` + +如果大陆 controller 也兼跑检测，再执行： + +```bash +systemctl status domaincheck-worker --no-pager -l +echo +curl -s http://127.0.0.1:8100/api/v1/runtime/status +``` + +预期： + +- `worker.running=true` +- `worker.expected_on_this_node=true` + +## 5. 最短联调命令 + +### 5.1 在大陆 controller 上 + +```bash +bash domain-api/deploy/multi-region/check_temp_link.sh http://127.0.0.1:8100 http://152.53.37.118:8100 +``` + +关注点： + +- `push_sync` 返回 `code=0` 或幂等成功 +- 不再出现目标地址为空 +- 不再出现 `SYNC_PUSH_ENABLED=false` + +关注点： + +- `cluster` 中至少能看到： + - `overseas-control-01` + - 大陆 `mainland-controller-01` + - 大陆 `mainland-worker-01` +- `sync-summary` 中能看到来自大陆的接收记录 +- `readiness` 不再是“完全单节点海外态” + +## 6. 后台页面应如何理解 + +打开： + +- `http://152.53.37.118/` + +在运行中心里看到下面这些内容时，属于正常： + +- `本机 Worker：当前节点不承载` +- `Sync Agent：当前节点不承载` +- `当前检测阶段：当前节点不承载` +- `代理运行态：不适用` + +这不是故障，而是因为当前节点就是临时海外控制面。 + +真正应该关注的是： + +- `集群节点` +- `有效执行节点` +- `同步状态` +- `多机就绪度` + +## 7. 判断“到底有几台 Worker 在工作” + +如果页面上“在线 Worker 数”和“实际正在跑检测的节点数”看起来不一致，以数据库为准。 + +在大陆 controller 上执行： + +```bash +bash domain-api/deploy/multi-region/check_worker_participation.sh +``` + +判断规则： + +- `detect_worker_nodes` + - 看谁在线 + - 看谁是 `worker_online=true` + - 看谁 `detect_participating=true` +- `detect_job_items` + - 看任务到底被哪台节点 `claimed_by` + - 这才是“当前真正干活的节点” + +## 8. 常见误判 + +### 8.1 海外控制面显示“本机 Worker 不承载” + +这是正常，不是故障。 + +### 8.2 页面显示“在线 Worker 1”，但你部署了 2 台大陆机器 + +先区分： + +- “有效执行节点”是可承担任务的在线节点数 +- “当前参与检测节点”是当前真的在领任务、跑任务的节点 + +如果 controller 没兼跑检测，通常只会看到独立 worker 在真正执行。 + +### 8.3 controller 机器也开了 worker 服务，但没有领任务 + +这不一定是 bug，可能只是当前调度没有分到它。应以 `detect_job_items.claimed_by` 为准，不要只看服务在线。 + +## 9. 联调完成后的回滚 + +如果你后续要切回正式海外机器，在大陆 controller 上把目标地址改回正式值即可： + +```bash +sed -i 's#^SYNC_TARGET_API_BASE_URL=.*#SYNC_TARGET_API_BASE_URL=http://正式海外控制面IP:8100#' /etc/default/domaincheck-api +systemctl restart domaincheck-api +systemctl restart domaincheck-sync-agent +sleep 3 +``` + +如果临时海外控制面不再使用，可在该机器上保留 `domaincheck-api` 供后续调试，也可以停掉： + +```bash +systemctl stop domaincheck-api +``` + +## 10. 本次临时海外控制面结论 + +当前这台测试海外机已经整理为： + +- 纯 `overseas-control` +- API 正常 +- Worker 不承载 +- Sync Agent 不承载 +- 数据已清空并按首装态初始化 +- 可直接作为大陆 controller 的临时同步接收端 + +## 11. 配套脚本 + +- 临时海外控制面自检： + - `domain-api/deploy/multi-region/check_temp_overseas_control.sh` +- 大陆 controller -> 临时海外控制面联调检查： + - `domain-api/deploy/multi-region/check_temp_link.sh` +- 三机拓扑总览检查： + - `domain-api/deploy/multi-region/check_temp_topology.sh` +- 当前活跃任务由谁真正执行： + - `domain-api/deploy/multi-region/check_worker_participation.sh` +- 大陆 worker 本机自检： + - `domain-api/deploy/multi-region/check_mainland_worker.sh` diff --git a/domain-api/app/api/routes/detect.py b/domain-api/app/api/routes/detect.py index d6269ac..6e260cb 100644 --- a/domain-api/app/api/routes/detect.py +++ b/domain-api/app/api/routes/detect.py @@ -1,3 +1,5 @@ +from __future__ import annotations + from uuid import uuid4 from fastapi import APIRouter diff --git a/domain-api/app/services/dashboard.py b/domain-api/app/services/dashboard.py index c0d1597..9acf6c2 100644 --- a/domain-api/app/services/dashboard.py +++ b/domain-api/app/services/dashboard.py @@ -28,11 +28,13 @@ def fetch_overview() -> dict: online_worker_nodes = int(cluster_summary.get("online_worker_nodes", 0) or 0) dedicated_online_worker_nodes = int(cluster_summary.get("dedicated_online_worker_nodes", 0) or 0) result["worker_status"] = "online" if runtime["worker"]["running"] else "offline" - result["local_worker_status"] = result["worker_status"] + result["local_worker_status"] = "not-applicable" if not runtime["worker"].get("expected_on_this_node", True) else result["worker_status"] result["cluster_worker_status"] = "online" if online_worker_nodes > 0 else "offline" result["cluster_online_worker_nodes"] = online_worker_nodes result["cluster_dedicated_online_worker_nodes"] = dedicated_online_worker_nodes result["cluster_online_control_nodes"] = int(cluster_summary.get("online_control_nodes", 0) or 0) result["api_status"] = "online" result["worker_mode"] = runtime["worker"]["mode"] + result["node_region"] = runtime["node"]["region"] + result["node_role"] = runtime["node"]["role"] return result diff --git a/domain-api/app/services/runtime_status_service.py b/domain-api/app/services/runtime_status_service.py index 0383e15..03dac4e 100644 --- a/domain-api/app/services/runtime_status_service.py +++ b/domain-api/app/services/runtime_status_service.py @@ -270,6 +270,7 @@ def get_runtime_status() -> dict: runtime_settings = get_runtime_settings() worker_runtime = detect_worker_runtime() sync_agent_runtime = detect_sync_agent_runtime() + worker_expected_on_this_node = not (settings.node_region == "overseas" and settings.node_role == "control") api_pid = os.getpid() detect_snapshot = get_detect_status() latest_run = (detect_snapshot.get("runs") or [None])[0] or {} @@ -314,6 +315,35 @@ def get_runtime_status() -> dict: "queue_health": queue_health, "capacity_plan": capacity_plan, } + if not worker_expected_on_this_node: + detect_payload.update( + { + "phase_label": "当前节点不承载", + "phase_detail": "当前节点为海外控制面，仅承载 API 控制与同步接收，不执行本机检测任务。", + "recent_event": "", + "recent_warning": "", + "progress_percent": 0, + "active_thread_count": 0, + "max_thread_count": 0, + "available_proxy_count": 0, + "proxy_pool_count": 0, + "proxy_runtime_label": "不适用", + "proxy_runtime_detail": "当前节点不承载本机 Worker，代理、线程与检测阶段信息在此节点上不适用。", + "proxy_runtime_reason": "not_applicable", + "proxy_supplier_empty": False, + "proxy_last_refresh_status": "", + "proxy_last_refresh_time": "", + "proxy_last_refresh_source_count": 0, + "proxy_last_refresh_total_items": 0, + "proxy_last_validated_count": 0, + "proxy_last_available_count": 0, + "proxy_source_stats": [], + "dependency_alerts": [], + "active_job": None, + "runs_count": 0, + "worker_online": False, + } + ) detect_payload["participating_nodes"] = _build_participating_detect_nodes( cluster_snapshot=cluster_snapshot, detect_snapshot=detect_payload, @@ -351,6 +381,7 @@ def get_runtime_status() -> dict: "mode": worker_runtime.get("mode", runtime_settings.get("worker_mode", "windows-local")), "service_name": runtime_settings.get("worker_service_name", settings.worker_service_name), "running": worker_runtime.get("running", False), + "expected_on_this_node": worker_expected_on_this_node, "process_count": worker_runtime.get("process_count", 0), "latest_start_time": worker_runtime.get("latest_start_time", ""), "message": worker_runtime.get("message", ""), diff --git a/domain-api/deploy/multi-region/README.md b/domain-api/deploy/multi-region/README.md index 614438a..4380061 100644 --- a/domain-api/deploy/multi-region/README.md +++ b/domain-api/deploy/multi-region/README.md @@ -25,6 +25,11 @@ - `deploy/multi-region/bootstrap_mainland.sh` - `deploy/multi-region/check_cluster.sh` - `deploy/multi-region/check_mainland_controller.sh` +- `deploy/multi-region/check_mainland_worker.sh` +- `deploy/multi-region/check_worker_participation.sh` +- `deploy/multi-region/check_temp_overseas_control.sh` +- `deploy/multi-region/check_temp_link.sh` +- `deploy/multi-region/check_temp_topology.sh` - `deploy/multi-region/simulate_cluster_node.py` - `deploy/multi-region/simulate_multi_region.sh` - `deploy/multi-region/prune_cluster_nodes.py` @@ -199,13 +204,48 @@ bash deploy/multi-region/check_mainland_controller.sh 这条命令会直接检查： -- `/etc/default/domaincheck-worker` 是否存在 +- `/etc/default/domaincheck-api` 是否存在 - `NODE_ROLE=control` 是否正确 - `SYNC_PUSH_ENABLED` / `SYNC_TARGET_API_BASE_URL` 是否已配置 +- `domaincheck-api` - `domaincheck-worker` - `domaincheck-sync-agent` 是否已启用并处于运行态 +如果要判断“在线 Worker 有几台”和“当前真正领任务执行的是哪台”是否一致，可以执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_worker_participation.sh +``` + +这条命令会直接输出： + +- `detect_worker_nodes` 当前在线节点概览 +- 最新活跃任务的 `claimed_by / status / count` +- 一段压缩摘要 JSON + +如果要检查纯大陆 worker 节点自己的配置和服务状态，可以执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_mainland_worker.sh +``` + +如果当前是“大陆 controller + 大陆 worker + 临时海外控制面”的联调拓扑，还可以执行： + +```bash +cd /opt/domaincheck/domain-api +bash deploy/multi-region/check_temp_topology.sh http://127.0.0.1:8100 http://152.53.37.118:8100 +``` + +这条命令会同时汇总： + +- 大陆控制面的 `runtime/cluster` +- 大陆控制面的 `runtime/readiness` +- 最新活跃任务到底由谁 `claimed_by` +- 临时海外控制面的 `runtime/cluster` + 如果当前还无法真正部署到大陆机器，也可以先在国外测试机上做“单机模拟多节点联调”： ```bash diff --git a/domain-api/deploy/multi-region/check_mainland_controller.sh b/domain-api/deploy/multi-region/check_mainland_controller.sh index e28ff7b..e16ed75 100755 --- a/domain-api/deploy/multi-region/check_mainland_controller.sh +++ b/domain-api/deploy/multi-region/check_mainland_controller.sh @@ -1,8 +1,9 @@ #!/usr/bin/env bash set -euo pipefail -ENV_FILE="${1:-/etc/default/domaincheck-worker}" +ENV_FILE="${1:-/etc/default/domaincheck-api}" WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}" +API_SERVICE="${API_SERVICE:-domaincheck-api}" SYNC_AGENT_SERVICE="${SYNC_AGENT_SERVICE:-domaincheck-sync-agent}" PYTHON_BIN="${PYTHON_BIN:-python3}" @@ -61,16 +62,22 @@ if errors: PY echo -echo "[3/5] systemd status" +echo "[3/6] systemd status" +systemctl is-enabled "$API_SERVICE" || true +systemctl is-active "$API_SERVICE" systemctl is-enabled "$WORKER_SERVICE" || true systemctl is-active "$WORKER_SERVICE" systemctl is-enabled "$SYNC_AGENT_SERVICE" || true systemctl is-active "$SYNC_AGENT_SERVICE" echo -echo "[4/5] worker service detail" +echo "[4/6] api service detail" +systemctl status "$API_SERVICE" --no-pager -l | sed -n '1,25p' +echo + +echo "[5/6] worker service detail" systemctl status "$WORKER_SERVICE" --no-pager -l | sed -n '1,25p' echo -echo "[5/5] sync-agent detail" +echo "[6/6] sync-agent detail" systemctl status "$SYNC_AGENT_SERVICE" --no-pager -l | sed -n '1,25p' diff --git a/domain-api/deploy/multi-region/check_mainland_worker.sh b/domain-api/deploy/multi-region/check_mainland_worker.sh new file mode 100755 index 0000000..1ad449c --- /dev/null +++ b/domain-api/deploy/multi-region/check_mainland_worker.sh @@ -0,0 +1,70 @@ +#!/usr/bin/env bash +set -euo pipefail + +ENV_FILE="${1:-/etc/default/domaincheck-worker}" +WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}" +PYTHON_BIN="${PYTHON_BIN:-python3}" + +echo "[1/4] environment file" +if [ ! -f "$ENV_FILE" ]; then + echo "missing env file: $ENV_FILE" + exit 1 +fi +echo "env file: $ENV_FILE" +echo + +echo "[2/4] key env summary" +"${PYTHON_BIN}" - <<'PY' "$ENV_FILE" +import json +import sys +from pathlib import Path + +env_path = Path(sys.argv[1]) +values = {} +for line in env_path.read_text(encoding="utf-8").splitlines(): + text = line.strip() + if not text or text.startswith("#") or "=" not in text: + continue + key, value = text.split("=", 1) + values[key.strip()] = value.strip() + +required = { + "NODE_CODE": values.get("NODE_CODE", ""), + "NODE_REGION": values.get("NODE_REGION", ""), + "NODE_ROLE": values.get("NODE_ROLE", ""), + "DB_HOST": values.get("DB_HOST", ""), + "DB_PORT": values.get("DB_PORT", ""), + "REDIS_HOST": values.get("REDIS_HOST", ""), + "REDIS_PORT": values.get("REDIS_PORT", ""), + "SYNC_TARGET_API_BASE_URL": values.get("SYNC_TARGET_API_BASE_URL", ""), +} +print(json.dumps(required, ensure_ascii=False, indent=2)) + +errors = [] +if required["NODE_REGION"] != "mainland": + errors.append("NODE_REGION 必须为 mainland") +if required["NODE_ROLE"] != "worker": + errors.append("NODE_ROLE 必须为 worker") +if not required["NODE_CODE"]: + errors.append("NODE_CODE 不能为空") +if not required["DB_HOST"]: + errors.append("DB_HOST 不能为空") +if not required["REDIS_HOST"]: + errors.append("REDIS_HOST 不能为空") + +if errors: + print() + print("env validation failed:") + for item in errors: + print(f"- {item}") + sys.exit(1) +PY +echo + +echo "[3/4] systemd status" +systemctl is-enabled "$WORKER_SERVICE" || true +systemctl is-active "$WORKER_SERVICE" +echo + +echo "[4/4] worker service detail" +systemctl status "$WORKER_SERVICE" --no-pager -l | sed -n '1,30p' diff --git a/domain-api/deploy/multi-region/check_temp_link.sh b/domain-api/deploy/multi-region/check_temp_link.sh new file mode 100755 index 0000000..6a8d189 --- /dev/null +++ b/domain-api/deploy/multi-region/check_temp_link.sh @@ -0,0 +1,80 @@ +#!/usr/bin/env bash +set -euo pipefail + +MAINLAND_BASE_URL="${1:-http://127.0.0.1:8100}" +OVERSEAS_BASE_URL="${2:-http://152.53.37.118:8100}" +PYTHON_BIN="${PYTHON_BIN:-python3}" + +echo "[1/6] mainland readiness" +MAINLAND_READINESS="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/readiness")" +echo "${MAINLAND_READINESS}" +echo +echo + +echo "[2/6] mainland sync summary" +MAINLAND_SYNC="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/sync-summary")" +echo "${MAINLAND_SYNC}" +echo +echo + +echo "[3/6] mainland push sync" +MAINLAND_PUSH="$(curl -fsS -X POST "${MAINLAND_BASE_URL}/api/v1/runtime/actions/push_sync")" +echo "${MAINLAND_PUSH}" +echo +echo + +echo "[4/6] overseas cluster" +OVERSEAS_CLUSTER="$(curl -fsS "${OVERSEAS_BASE_URL}/api/v1/runtime/cluster")" +echo "${OVERSEAS_CLUSTER}" +echo +echo + +echo "[5/6] overseas sync summary" +OVERSEAS_SYNC="$(curl -fsS "${OVERSEAS_BASE_URL}/api/v1/runtime/sync-summary")" +echo "${OVERSEAS_SYNC}" +echo +echo + +echo "[6/6] condensed summary" +"${PYTHON_BIN}" - <<'PY' "$MAINLAND_READINESS" "$MAINLAND_SYNC" "$MAINLAND_PUSH" "$OVERSEAS_CLUSTER" "$OVERSEAS_SYNC" +import json +import sys + +mainland_readiness = json.loads(sys.argv[1]).get("data", {}) +mainland_sync = json.loads(sys.argv[2]).get("data", {}) +mainland_push = json.loads(sys.argv[3]) +overseas_cluster = json.loads(sys.argv[4]).get("data", {}) +overseas_sync = json.loads(sys.argv[5]).get("data", {}) + +cluster_nodes = overseas_cluster.get("nodes") or [] +cluster_summary = overseas_cluster.get("summary") or {} +type_counts = overseas_sync.get("type_counts") or {} +status_counts = overseas_sync.get("status_counts") or {} + +print( + json.dumps( + { + "mainland": { + "readiness_status": mainland_readiness.get("status", ""), + "readiness_summary": mainland_readiness.get("summary", ""), + "sync_enabled": bool(mainland_sync.get("enabled", False)), + "target_api_base_url": mainland_sync.get("target_api_base_url", ""), + "push_sync_code": mainland_push.get("code"), + "push_sync_message": mainland_push.get("message", ""), + }, + "overseas": { + "nodes_total": overseas_cluster.get("nodes_total", 0), + "online_control_nodes": cluster_summary.get("online_control_nodes", 0), + "online_worker_nodes": cluster_summary.get("online_worker_nodes", 0), + "node_codes": [str(item.get("node_code") or "") for item in cluster_nodes], + "runtime_ingest": type_counts.get("runtime_ingest", 0), + "detect_result_ingest": type_counts.get("detect_result_ingest", 0), + "status_received": status_counts.get("received", 0), + "status_success": status_counts.get("success", 0), + }, + }, + ensure_ascii=False, + indent=2, + ) +) +PY diff --git a/domain-api/deploy/multi-region/check_temp_overseas_control.sh b/domain-api/deploy/multi-region/check_temp_overseas_control.sh new file mode 100755 index 0000000..66c4275 --- /dev/null +++ b/domain-api/deploy/multi-region/check_temp_overseas_control.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +set -euo pipefail + +BASE_URL="${1:-http://127.0.0.1:8100}" +PYTHON_BIN="${PYTHON_BIN:-python3}" + +echo "[1/5] health" +curl -fsS "${BASE_URL}/health" +echo +echo + +echo "[2/5] runtime status" +STATUS_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/status")" +echo "${STATUS_JSON}" +echo +echo + +echo "[3/5] runtime cluster" +curl -fsS "${BASE_URL}/api/v1/runtime/cluster" +echo +echo + +echo "[4/5] runtime readiness" +curl -fsS "${BASE_URL}/api/v1/runtime/readiness" +echo +echo + +echo "[5/5] condensed summary" +"${PYTHON_BIN}" - <<'PY' "$STATUS_JSON" +import json +import sys + +data = json.loads(sys.argv[1]).get("data", {}) +node = data.get("node") or {} +worker = data.get("worker") or {} +sync_agent = data.get("sync_agent") or {} +detect = data.get("detect") or {} +cluster = data.get("cluster") or {} +summary = cluster.get("summary") or {} + +print( + json.dumps( + { + "node": { + "code": node.get("code", ""), + "region": node.get("region", ""), + "role": node.get("role", ""), + }, + "worker": { + "running": bool(worker.get("running", False)), + "expected_on_this_node": bool(worker.get("expected_on_this_node", True)), + "message": worker.get("message", ""), + }, + "sync_agent": { + "running": bool(sync_agent.get("running", False)), + "expected_on_this_node": bool(sync_agent.get("expected_on_this_node", False)), + "message": sync_agent.get("message", ""), + }, + "detect": { + "phase_label": detect.get("phase_label", ""), + "proxy_runtime_label": detect.get("proxy_runtime_label", ""), + "active_job": (detect.get("active_job") or {}).get("job_code", ""), + }, + "cluster": { + "nodes_total": cluster.get("nodes_total", 0), + "online_control_nodes": summary.get("online_control_nodes", 0), + "online_worker_nodes": summary.get("online_worker_nodes", 0), + "dedicated_online_worker_nodes": summary.get("dedicated_online_worker_nodes", 0), + }, + }, + ensure_ascii=False, + indent=2, + ) +) +PY diff --git a/domain-api/deploy/multi-region/check_temp_topology.sh b/domain-api/deploy/multi-region/check_temp_topology.sh new file mode 100755 index 0000000..976114d --- /dev/null +++ b/domain-api/deploy/multi-region/check_temp_topology.sh @@ -0,0 +1,126 @@ +#!/usr/bin/env bash +set -euo pipefail + +MAINLAND_BASE_URL="${1:-http://127.0.0.1:8100}" +OVERSEAS_BASE_URL="${2:-http://152.53.37.118:8100}" +PYTHON_BIN="${PYTHON_BIN:-python3}" + +echo "[1/5] mainland runtime cluster" +MAINLAND_CLUSTER="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/cluster")" +echo "${MAINLAND_CLUSTER}" +echo +echo + +echo "[2/5] mainland runtime readiness" +MAINLAND_READINESS="$(curl -fsS "${MAINLAND_BASE_URL}/api/v1/runtime/readiness")" +echo "${MAINLAND_READINESS}" +echo +echo + +echo "[3/5] mainland worker participation" +PARTICIPATION_JSON="$(/www/server/pgsql/bin/psql -U postgres -d domain -At <<'SQL' +with latest_job as ( + select id, job_code, status + from detect_jobs + where status in ('pending','running') + order by id desc + limit 1 +), +node_stats as ( + select + node_code, + role, + status, + coalesce(metadata_json->>'worker_online', '') as worker_online, + coalesce(metadata_json->>'detect_participating', '') as detect_participating + from detect_worker_nodes +), +item_stats as ( + select + coalesce(claimed_by, '') as claimed_by, + status, + count(*) as cnt + from detect_job_items + where job_id = (select id from latest_job) + group by claimed_by, status +) +select json_build_object( + 'job', ( + select json_build_object( + 'job_code', coalesce(job_code, ''), + 'status', coalesce(status, '') + ) + from latest_job + ), + 'online_nodes', ( + select coalesce(json_agg(json_build_object( + 'node_code', node_code, + 'role', role, + 'status', status, + 'worker_online', worker_online, + 'detect_participating', detect_participating + ) order by node_code), '[]'::json) + from node_stats + where status in ('online', 'busy') + ), + 'claimed_distribution', ( + select coalesce(json_agg(json_build_object( + 'claimed_by', claimed_by, + 'status', status, + 'count', cnt + ) order by claimed_by, status), '[]'::json) + from item_stats + ) +); +SQL +)" +echo "${PARTICIPATION_JSON}" +echo +echo + +echo "[4/5] overseas runtime cluster" +OVERSEAS_CLUSTER="$(curl -fsS "${OVERSEAS_BASE_URL}/api/v1/runtime/cluster")" +echo "${OVERSEAS_CLUSTER}" +echo +echo + +echo "[5/5] condensed summary" +"${PYTHON_BIN}" - <<'PY' "$MAINLAND_CLUSTER" "$MAINLAND_READINESS" "$PARTICIPATION_JSON" "$OVERSEAS_CLUSTER" +import json +import sys + +mainland_cluster = json.loads(sys.argv[1]).get("data", {}) +mainland_readiness = json.loads(sys.argv[2]).get("data", {}) +participation = json.loads(sys.argv[3]) +overseas_cluster = json.loads(sys.argv[4]).get("data", {}) + +mainland_summary = mainland_cluster.get("summary") or {} +overseas_summary = overseas_cluster.get("summary") or {} + +def node_codes(cluster): + return [str(item.get("node_code") or "") for item in (cluster.get("nodes") or [])] + +print( + json.dumps( + { + "mainland": { + "readiness_status": mainland_readiness.get("status", ""), + "readiness_summary": mainland_readiness.get("summary", ""), + "online_worker_nodes": mainland_summary.get("online_worker_nodes", 0), + "dedicated_online_worker_nodes": mainland_summary.get("dedicated_online_worker_nodes", 0), + "online_control_nodes": mainland_summary.get("online_control_nodes", 0), + "node_codes": node_codes(mainland_cluster), + }, + "participation": participation, + "overseas": { + "online_worker_nodes": overseas_summary.get("online_worker_nodes", 0), + "dedicated_online_worker_nodes": overseas_summary.get("dedicated_online_worker_nodes", 0), + "online_control_nodes": overseas_summary.get("online_control_nodes", 0), + "node_codes": node_codes(overseas_cluster), + }, + }, + ensure_ascii=False, + indent=2, + ) +) +PY diff --git a/domain-api/deploy/multi-region/check_worker_participation.sh b/domain-api/deploy/multi-region/check_worker_participation.sh new file mode 100755 index 0000000..e72693c --- /dev/null +++ b/domain-api/deploy/multi-region/check_worker_participation.sh @@ -0,0 +1,97 @@ +#!/usr/bin/env bash +set -euo pipefail + +DB_NAME="${DB_NAME:-domain}" +DB_USER="${DB_USER:-postgres}" +PSQL_BIN="${PSQL_BIN:-/www/server/pgsql/bin/psql}" + +echo "[1/3] cluster nodes" +"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -Atc " +select + node_code || '|' || + role || '|' || + status || '|' || + coalesce(current_load, 0) || '|' || + to_char(last_heartbeat_at, 'YYYY-MM-DD HH24:MI:SS') || '|' || + coalesce(metadata_json->>'worker_online', '') || '|' || + coalesce(metadata_json->>'detect_participating', '') +from detect_worker_nodes +order by node_code; +" +echo + +echo "[2/3] active job distribution" +"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -Atc " +select + coalesce(claimed_by, '') || '|' || + status || '|' || + count(*) +from detect_job_items +where job_id = ( + select id + from detect_jobs + where status in ('pending','running') + order by id desc + limit 1 +) +group by claimed_by, status +order by claimed_by, status; +" +echo + +echo "[3/3] condensed summary" +"${PSQL_BIN}" -U "${DB_USER}" -d "${DB_NAME}" -At <<'SQL' +with latest_job as ( + select id, job_code, status + from detect_jobs + where status in ('pending','running') + order by id desc + limit 1 +), +node_stats as ( + select + node_code, + role, + status, + coalesce(metadata_json->>'worker_online', '') as worker_online, + coalesce(metadata_json->>'detect_participating', '') as detect_participating + from detect_worker_nodes +), +item_stats as ( + select + coalesce(claimed_by, '') as claimed_by, + status, + count(*) as cnt + from detect_job_items + where job_id = (select id from latest_job) + group by claimed_by, status +) +select json_build_object( + 'job', ( + select json_build_object( + 'job_code', coalesce(job_code, ''), + 'status', coalesce(status, '') + ) + from latest_job + ), + 'online_nodes', ( + select coalesce(json_agg(json_build_object( + 'node_code', node_code, + 'role', role, + 'status', status, + 'worker_online', worker_online, + 'detect_participating', detect_participating + ) order by node_code), '[]'::json) + from node_stats + where status in ('online', 'busy') + ), + 'claimed_distribution', ( + select coalesce(json_agg(json_build_object( + 'claimed_by', claimed_by, + 'status', status, + 'count', cnt + ) order by claimed_by, status), '[]'::json) + from item_stats + ) +); +SQL diff --git a/domain-web/src/layouts/MainLayout.vue b/domain-web/src/layouts/MainLayout.vue index f334509..4957b7e 100644 --- a/domain-web/src/layouts/MainLayout.vue +++ b/domain-web/src/layouts/MainLayout.vue @@ -24,8 +24,8 @@ API {{ runtime.apiOnline ? "在线" : "离线" }} - - 本机 Worker {{ runtime.localWorkerOnline ? "在线" : "离线" }} + + 本机 Worker {{ runtime.localWorkerExpected ? (runtime.localWorkerOnline ? "在线" : "离线") : "不承载" }} 有效执行节点 {{ runtime.effectiveWorkerNodes }} @@ -57,6 +57,7 @@ const authStore = useAuthStore(); const runtime = ref({ apiOnline: false, localWorkerOnline: false, + localWorkerExpected: true, effectiveWorkerNodes: 0, workerMode: "windows-local" }); @@ -85,6 +86,7 @@ const loadRuntimeSummary = async () => { runtime.value = { apiOnline: Boolean(response.data?.api?.pid), localWorkerOnline: Boolean(response.data?.worker?.running), + localWorkerExpected: Boolean(response.data?.worker?.expected_on_this_node ?? true), effectiveWorkerNodes: Number(summary.online_worker_nodes || 0), workerMode: response.data?.worker?.mode || "windows-local" }; @@ -92,6 +94,7 @@ const loadRuntimeSummary = async () => { runtime.value = { apiOnline: false, localWorkerOnline: false, + localWorkerExpected: true, effectiveWorkerNodes: 0, workerMode: "unknown" }; diff --git a/domain-web/src/views/dashboard/DashboardView.vue b/domain-web/src/views/dashboard/DashboardView.vue index 47daab8..bba85b5 100644 --- a/domain-web/src/views/dashboard/DashboardView.vue +++ b/domain-web/src/views/dashboard/DashboardView.vue @@ -80,12 +80,20 @@ const loadOverview = async (showError = true) => { loading.value = true; try { const { data } = await dashboardApi.overview(); + const localWorkerValue = + data.local_worker_status === "not-applicable" + ? "不承载" + : data.local_worker_status || data.worker_status; + const localWorkerNote = + data.local_worker_status === "not-applicable" + ? `当前节点 ${data.node_region || "-"} / ${data.node_role || "-"}，默认不承载本机 Worker` + : "当前节点本机 Worker 运行状态"; stats.value = [ { label: "总域名", value: String(data.domains_total), note: "数据库域名总量" }, { label: "待检测", value: String(data.pending_total), note: "当前待处理域名" }, { label: "黑名单", value: String(data.blacklist_total), note: "命中风险域名" }, { label: "API", value: data.api_status, note: "当前 API 运行状态" }, - { label: "本机 Worker", value: data.local_worker_status || data.worker_status, note: "当前节点本机 Worker 运行状态" }, + { label: "本机 Worker", value: localWorkerValue, note: localWorkerNote }, { label: "有效执行节点", value: `${data.cluster_worker_status || "offline"} / ${data.cluster_online_worker_nodes || 0}`, diff --git a/domain-web/src/views/runtime/RuntimeView.vue b/domain-web/src/views/runtime/RuntimeView.vue index fcaa287..aeb2353 100644 --- a/domain-web/src/views/runtime/RuntimeView.vue +++ b/domain-web/src/views/runtime/RuntimeView.vue @@ -24,7 +24,9 @@
本机 Worker
{{ runtime.worker?.service_name || "-" }}
- {{ runtime.worker?.running ? "本机在线" : "本机离线" }} + + {{ localWorkerSummaryText }} +
运行模式
@@ -118,8 +120,8 @@ API {{ runtime.api?.pid ? "运行中" : "未知" }} - - 本机 Worker {{ runtime.worker?.running ? "在线" : "离线" }} + + 本机 Worker {{ localWorkerStripText }} 有效执行节点 {{ runtime.cluster?.summary?.online_worker_nodes || 0 }} @@ -151,10 +153,10 @@ {{ runtime.api?.pid || "-" }} {{ runtime.api?.health_url || "-" }} {{ runtime.worker?.service_name || "-" }} - {{ runtime.worker?.running ? "是" : "否" }} + {{ localWorkerBooleanText }} {{ runtime.worker?.mode || "-" }} {{ runtime.worker?.process_count ?? "-" }} - {{ runtime.worker?.message || "-" }} + {{ localWorkerMessage }} {{ runtime.sync_agent?.service_name || "-" }} {{ runtime.sync_agent?.running ? "是" : "否" }} {{ runtime.sync_agent?.latest_start_time || "-" }} @@ -166,8 +168,21 @@

运行时动作

- 启动 Worker - 停止 Worker + + 启动 Worker + + + 停止 Worker + 重启 API + { return runtime.detect?.proxy_runtime_detail || "当前暂无代理运行说明"; }; +const localWorkerExpected = computed(() => Boolean(runtime.worker?.expected_on_this_node ?? true)); + +const localWorkerSummaryText = computed(() => { + if (!localWorkerExpected.value) return "当前节点不承载"; + return runtime.worker?.running ? "本机在线" : "本机离线"; +}); + +const localWorkerStripText = computed(() => { + if (!localWorkerExpected.value) return "不承载"; + return runtime.worker?.running ? "在线" : "离线"; +}); + +const localWorkerBooleanText = computed(() => { + if (!localWorkerExpected.value) return "当前节点不承载"; + return runtime.worker?.running ? "是" : "否"; +}); + +const localWorkerMessage = computed(() => { + if (!localWorkerExpected.value) { + return `当前节点 ${runtime.node?.region || "-"} / ${runtime.node?.role || "-"}，默认不承载本机 Worker`; + } + return runtime.worker?.message || "-"; +}); + +const localWorkerActionText = computed(() => { + if (!localWorkerExpected.value) return "当前节点不承载"; + return runtime.worker?.running ? "运行中" : "未运行"; +}); + +const localWorkerTagType = computed(() => { + if (!localWorkerExpected.value) return "info"; + return runtime.worker?.running ? "success" : "danger"; +}); + +const localWorkerChipClass = computed(() => { + if (!localWorkerExpected.value) return "neutral"; + return runtime.worker?.running ? "ok" : "danger"; +}); + const clusterIssueCount = computed(() => { const summary = runtime.cluster?.summary || {}; return Number(summary.status_counts?.stale || 0) + Number(summary.status_counts?.offline || 0); diff --git a/domainCheck/node_thread_counts.json b/domainCheck/node_thread_counts.json new file mode 100644 index 0000000..9e26dfe --- /dev/null +++ b/domainCheck/node_thread_counts.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/domainCheck/proxy_config.json b/domainCheck/proxy_config.json index f43eff8..613c29a 100644 --- a/domainCheck/proxy_config.json +++ b/domainCheck/proxy_config.json @@ -1,13 +1,13 @@ { "proxy_enable": true, "allow_direct": true, - "proxy_url": "http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50", + "proxy_url": "http://211.101.244.154:18003/getProxy_batch.php?group=B&count=50", "proxy_urls": [ - "http://211.101.244.154:18008/getProxy_batch.php?group=B&count=50", - "http://211.101.244.154:18008/getProxy_batch.php?group=C&count=50", - "http://211.101.244.154:18008/getProxy_batch.php?group=D&count=50", - "http://211.101.244.154:18008/getProxy_batch.php?group=E&count=50", + "http://211.101.244.154:18003/getProxy_batch.php?group=B&count=50", + "http://211.101.244.154:18003/getProxy_batch.php?group=C&count=50", + "http://211.101.244.154:18003/getProxy_batch.php?group=D&count=50", + "http://211.101.244.154:18003/getProxy_batch.php?group=E&count=50", "http://211.101.244.154:18008/getProxy_batch.php?group=F&count=50", - "http://211.101.244.154:18008/getProxy_batch.php?group=G&count=50" + "http://211.101.244.154:18003/getProxy_batch.php?group=G&count=50" ] } \ No newline at end of file