first

domain-api/deploy/multi-region/README.md

@@ -0,0 +1,354 @@
+# domainCheck 跨地域部署入口
+
+本文档对应当前推荐的最小可用部署形态：
+
+- 国外 `1` 台：`domain-web + domain-api + postgresql_main`
+- 大陆 `1` 台：`redis + postgresql_runtime + scheduler + detect-worker`
+
+目标：
+
+- 后台和 API 放国外
+- 检测执行放大陆
+- 后续新增大陆 Worker 时，不再改整体部署方式
+
+## 一、目录约定
+
+统一使用：
+
+```text
+/opt/domaincheck
+```
+
+仓库内部署入口：
+
+- `deploy/multi-region/bootstrap_overseas.sh`
+- `deploy/multi-region/bootstrap_mainland.sh`
+- `deploy/multi-region/check_cluster.sh`
+- `deploy/multi-region/check_mainland_controller.sh`
+- `deploy/multi-region/simulate_cluster_node.py`
+- `deploy/multi-region/simulate_multi_region.sh`
+- `deploy/multi-region/prune_cluster_nodes.py`
+- `deploy/multi-region/prune_cluster_nodes.sh`
+- `deploy/multi-region/templates/*.env.example`
+
+## 二、国外机器部署
+
+在国外机器执行：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/bootstrap_overseas.sh /opt/domaincheck
+```
+
+脚本会：
+
+- 创建运行目录
+- 检查 Python 虚拟环境
+- 安装 API 依赖
+- 安装 systemd 服务模板
+- 在 `/etc/default/domaincheck-api` 生成环境变量模板
+- 给出后续启动命令
+
+## 三、大陆机器部署
+
+在大陆机器执行：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
+```
+
+如果后面新增纯 Worker 节点：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
+```
+
+其中：
+
+- `controller`
+  表示这台机器承担 `redis + runtime-db + scheduler + worker + sync-agent`
+- `worker`
+  表示这台机器只承担 `detect-worker`
+
+脚本还会在大陆机器生成：
+
+- `/etc/default/domaincheck-worker`
+
+当前模板已内置同步相关占位配置：
+
+- `SYNC_PUSH_ENABLED`
+- `SYNC_SOURCE_REGION`
+- `SYNC_TARGET_REGION`
+- `SYNC_TARGET_API_BASE_URL`
+- `SYNC_SHARED_TOKEN`
+- `SYNC_BATCH_SIZE`
+- `SYNC_POLL_INTERVAL_SECONDS`
+
+建议把每台大陆节点自己的身份信息放在这里，而不是直接修改 service 文件正文：
+
+- `NODE_CODE`
+- `NODE_REGION=mainland`
+- `NODE_ROLE=control`
+  - 仅 `controller` 模板使用
+- `NODE_ROLE=worker`
+  - 仅纯 Worker 模板使用
+- `DB_*`
+- `REDIS_*`
+
+## 四、当前脚本定位
+
+当前入口脚本是第一版“标准化部署脚手架”，优先解决：
+
+- 目录统一
+- systemd 模板统一
+- 节点环境变量入口统一
+- 新增节点时操作步骤统一
+
+当前大陆 bootstrap 还会自动补齐一组最小 Python 依赖：
+
+- `fastapi`
+- `uvicorn`
+- `pydantic-settings`
+- `psycopg2-binary`
+- `redis`
+
+这样 `domaincheck-sync-agent` 在大陆 controller 节点上可以直接启动，不会因为 `app.sync_agent` 缺少 API 侧依赖而失败。
+
+当前脚手架还额外区分了“自动同步由谁跑”：
+
+- 海外控制面：
+  - `domain-api`
+  - 负责接收 `runtime/sync-ingest`
+- 内地 controller 节点：
+  - `domaincheck-sync-agent`
+  - 负责按轮询周期把本地最新投影推送到海外控制面
+
+当前还不会自动安装数据库主从或自动创建跨地域同步链路。
+
+原因：
+
+- 当前项目仍处于单 Worker 改造向多 Worker 任务模型过渡阶段
+- 真正的任务调度与跨地域结果同步需要后续代码配合落地
+- 但当前已经预留同步配置模板与同步记录查询接口，便于后续接入 `sync-service`
+- 当前控制面还会自动把运行态摘要写入 `detect_sync_records`
+  - `sync_type=runtime_projection`
+  - 仅在关键状态变化或达到最小采样间隔时写入
+  - 作用是先把“同步观测面”跑起来，而不是替代最终的跨地域结果同步服务
+- 当 `SYNC_PUSH_ENABLED=true` 且配置了 `SYNC_TARGET_API_BASE_URL` 后：
+  - 大陆 `domaincheck-sync-agent` 会按 `SYNC_POLL_INTERVAL_SECONDS` 自动尝试推送：
+    - 最新一条 `runtime_projection`
+    - 当前积压的 `detect_result_projection` 批次
+  - 目标控制面通过 `POST /api/v1/runtime/sync-ingest` 接收
+  - 若配置了 `SYNC_SHARED_TOKEN`，接收端会校验 `X-Domaincheck-Sync-Token`
+  - `GET /api/v1/runtime/sync-summary` 还会返回：
+    - `detect_result_batches`
+    用于直接观察最近检测任务的结果同步是否已接收、仍待推送或发生失败
+
+## 五、当前已落地能力
+
+截至 `2026-04-16`，当前代码已经具备：
+
+- 节点注册与心跳
+- 运行态表自动初始化
+- 集群节点状态接口
+- 检测任务主表与任务项表
+- 活跃任务摘要与任务详情接口
+- Worker 小批量领取、租约续租、重启释放和过期回收
+
+可验证接口：
+
+```bash
+curl http://127.0.0.1:8100/api/v1/runtime/status
+curl http://127.0.0.1:8100/api/v1/runtime/readiness
+curl http://127.0.0.1:8100/api/v1/runtime/cluster
+curl http://127.0.0.1:8100/api/v1/runtime/sync-summary
+curl http://127.0.0.1:8100/api/v1/runtime/sync-records?limit=10
+curl http://127.0.0.1:8100/api/v1/detect/job/active
+curl http://127.0.0.1:8100/api/v1/detect/jobs
+```
+
+也可以直接执行一键联调：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100
+```
+
+这条命令当前除了原始接口输出，还会额外给出一段压缩摘要，直接汇总：
+
+- `ready / attention / blocking`
+- 在线控制面 / Worker 数
+- `busy / stale / offline` 节点
+- 同步投影 / 接收计数
+- 结果批次的：
+  - `synced`
+  - `delivered`
+  - `pushing`
+  - `projected`
+- `failed`
+- `unsynced`
+
+如果要在大陆 controller 节点本机确认“这台机器本身是否已经具备 controller 身份”，还可以执行：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/check_mainland_controller.sh
+```
+
+这条命令会直接检查：
+
+- `/etc/default/domaincheck-worker` 是否存在
+- `NODE_ROLE=control` 是否正确
+- `SYNC_PUSH_ENABLED` / `SYNC_TARGET_API_BASE_URL` 是否已配置
+- `domaincheck-worker`
+- `domaincheck-sync-agent`
+  是否已启用并处于运行态
+
+如果当前还无法真正部署到大陆机器，也可以先在国外测试机上做“单机模拟多节点联调”：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/simulate_multi_region.sh http://127.0.0.1:8100
+```
+
+这条命令会临时模拟：
+
+- `mainland-controller-sim`
+- `mainland-worker-sim-01`
+
+用于提前验证：
+
+- `runtime/readiness`
+- `runtime/cluster`
+- 运行中心顶部的多机就绪度结论
+
+结束时按 `Ctrl+C`，脚本会自动清理模拟节点记录。
+
+如果测试服里已经残留了很久没心跳的旧节点记录，导致 `runtime/readiness` 一直被离线节点拖成 `attention`，可以先做清理：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30 --dry-run
+bash deploy/multi-region/prune_cluster_nodes.sh --minutes 30
+```
+
+如果只想清理某个确定已经废弃的节点：
+
+```bash
+bash deploy/multi-region/prune_cluster_nodes.sh --node-code mainland-worker-01
+```
+
+如果要判断大陆 Worker 是否已经真正接入，不要只看 `systemctl`，还要看：
+
+- `/api/v1/runtime/cluster` 中是否出现对应 `node_code`
+- `last_heartbeat_at` 是否持续刷新
+- `role` / `region` 是否符合预期
+- `metadata.job_id / metadata.cycle_token` 是否能在执行时出现
+- `summary.online_worker_nodes` 是否大于 `0`
+- `summary.status_counts.busy` 是否会在执行中增加
+
+## 六、第二台大陆 Worker 接入建议
+
+后续新增大陆 Worker 时，建议顺序为：
+
+1. 拉最新代码到新大陆机器
+2. 准备与主执行面一致的 Python 环境
+3. 配置该节点自己的：
+   - `NODE_CODE`
+   - `NODE_REGION=mainland`
+   - `NODE_ROLE=worker`
+4. 执行：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
+```
+
+5. 启动后在国外控制面检查：
+
+```bash
+curl http://127.0.0.1:8100/api/v1/runtime/cluster
+```
+
+如果接口里出现新节点，并且心跳持续更新，说明节点接入成功。
+
+如果是第二台及以上大陆 Worker，最少还要确认：
+
+- `NODE_CODE` 与其它节点不重复
+- 连接的是同一套大陆 Redis / runtime-db
+- `/etc/default/domaincheck-worker` 已按该机器单独填写
+- 若该机器不是 controller，则不要额外启 `domaincheck-sync-agent`
+
+## 七、当前推荐联调命令
+
+国外控制面建议至少保留下面这组命令：
+
+```bash
+curl http://127.0.0.1:8100/health
+curl http://127.0.0.1:8100/api/v1/runtime/preflight
+curl http://127.0.0.1:8100/api/v1/runtime/readiness
+curl http://127.0.0.1:8100/api/v1/runtime/cluster
+curl http://127.0.0.1:8100/api/v1/detect/status
+curl http://127.0.0.1:8100/api/v1/detect/job/active
+curl http://127.0.0.1:8100/api/v1/detect/jobs?limit=5
+```
+
+如需一次性确认控制面和执行面都接通，可以直接执行：
+
+```bash
+cd /opt/domaincheck/domain-api
+bash deploy/multi-region/check_cluster.sh http://127.0.0.1:8100
+```
+
+观察重点：
+
+- 控制面节点心跳应持续刷新，而不是只在 API 启动时更新一次
+- 大陆 Worker 节点在运行检测时应显示 `busy`
+- `detect/job/active` 应能看到当前任务总量、完成量、节点分布和最近事件
+- `detect/jobs` 可用于回看最近几轮任务是否正常收敛
+- 当代理池暂时为空但允许直连时，检测控制页会显示 `降级直连`
+- 当代理池为空且不允许直连时，检测会话阶段会显示 `等待代理`
+- `runtime/cluster.summary` 中应能直接看出：
+  - 当前在线控制面节点数
+  - 当前在线 Worker 节点数
+  - 当前 `busy / stale / offline` 节点清单
+- `runtime/sync-summary` 中可直接查看：
+  - 是否启用同步推送
+  - 当前配置的源地域 / 目标地域
+  - 最近同步记录和状态分布
+  - 若当前还未接入真正的 `sync-service`，也应至少能看到：
+    - `runtime_projection`
+    - `detect_result_projection`
+  - 接入自动推送后，还应能看到：
+    - `runtime_push`
+    - `runtime_ingest`
+    - `detect_result_ingest`
+
+大陆 controller 节点建议额外确认：
+
+```bash
+systemctl status domaincheck-sync-agent --no-pager -l
+```
+
+如果同步配置已填写完整，则期望：
+
+- `domaincheck-sync-agent` 为 `active (running)`
+- 海外控制面的 `runtime/sync-summary` 中能同时看到：
+  - `runtime_projection`
+  - `runtime_push`
+  - `runtime_ingest`
+
+## 八、后续演进
+
+后续会继续补：
+
+- 运行库初始化脚本
+- 节点配置模板
+- 结果同步服务
+- 多 Worker 调度服务模板
+
+配套设计文档见：
+
+- `docs/16_domainCheck_多机检测与跨地域部署设计.md`

domain-api/deploy/multi-region/bootstrap_mainland.sh

@@ -0,0 +1,88 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+BASE_DIR="${1:-/opt/domaincheck}"
+ROLE="${2:-controller}"
+API_DIR="$BASE_DIR/domain-api"
+DOMAIN_DIR="$BASE_DIR/domainCheck"
+SERVICE_NAME="domaincheck-worker"
+SYNC_AGENT_SERVICE_NAME="domaincheck-sync-agent"
+ENV_FILE="/etc/default/domaincheck-worker"
+TEMPLATE_DIR="$API_DIR/deploy/multi-region/templates"
+
+if [ "$ROLE" != "controller" ] && [ "$ROLE" != "worker" ]; then
+  echo "invalid role: $ROLE"
+  echo "usage: bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck [controller|worker]"
+  exit 1
+fi
+
+echo "[1/7] preparing directories"
+mkdir -p "$BASE_DIR" "$DOMAIN_DIR" "$API_DIR/runtime"
+
+if [ "$ROLE" = "controller" ]; then
+  mkdir -p /var/lib/domaincheck-runtime-db /var/lib/domaincheck-redis
+fi
+
+echo "[2/7] checking python environment"
+if [ ! -x "$DOMAIN_DIR/.venv/bin/python" ]; then
+  echo "missing python venv: $DOMAIN_DIR/.venv/bin/python"
+  echo "please deploy domainCheck first and create .venv there"
+  exit 1
+fi
+
+echo "[2.5/7] installing mainland runtime dependencies"
+"$DOMAIN_DIR/.venv/bin/pip" install -q fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
+
+echo "[3/7] fixing runtime permissions"
+touch "$DOMAIN_DIR/detect_worker.log"
+chmod 664 "$DOMAIN_DIR/detect_worker.log"
+
+echo "[4/7] installing worker service"
+install -m 0644 "$API_DIR/deploy/systemd/domain-worker.service" "/etc/systemd/system/${SERVICE_NAME}.service"
+if [ "$ROLE" = "controller" ]; then
+  install -m 0644 "$API_DIR/deploy/systemd/domain-sync-agent.service" "/etc/systemd/system/${SYNC_AGENT_SERVICE_NAME}.service"
+fi
+
+echo "[4.5/7] preparing environment file"
+if [ ! -f "$ENV_FILE" ]; then
+  if [ "$ROLE" = "controller" ]; then
+    TEMPLATE_FILE="$TEMPLATE_DIR/domaincheck-worker.controller.env.example"
+  else
+    TEMPLATE_FILE="$TEMPLATE_DIR/domaincheck-worker.worker.env.example"
+  fi
+  install -m 0644 "$TEMPLATE_FILE" "$ENV_FILE"
+  sed -i "s#__BASE_DIR__#${BASE_DIR}#g" "$ENV_FILE"
+  chmod 0644 "$ENV_FILE"
+  echo "created env template: $ENV_FILE"
+else
+  echo "env file already exists: $ENV_FILE"
+fi
+
+echo "[5/7] reloading systemd"
+systemctl daemon-reload
+
+if [ "$ROLE" = "controller" ]; then
+  echo "[6/7] controller notes"
+  echo "controller mode selected"
+  echo "please ensure local redis and postgresql_runtime are installed and listening on localhost"
+  echo "sync agent service: ${SYNC_AGENT_SERVICE_NAME}"
+else
+  echo "[6/7] worker node notes"
+  echo "worker mode selected"
+  echo "this node is expected to connect to mainland redis and runtime-db"
+fi
+
+echo "[7/7] summary"
+echo "mainland bootstrap completed"
+echo "role: ${ROLE}"
+echo "service: ${SERVICE_NAME}"
+echo "next commands:"
+echo "  systemctl enable ${SERVICE_NAME}"
+echo "  systemctl restart ${SERVICE_NAME}"
+echo "  systemctl status ${SERVICE_NAME} --no-pager -l"
+if [ "$ROLE" = "controller" ]; then
+  echo "  systemctl enable ${SYNC_AGENT_SERVICE_NAME}"
+  echo "  systemctl restart ${SYNC_AGENT_SERVICE_NAME}"
+  echo "  systemctl status ${SYNC_AGENT_SERVICE_NAME} --no-pager -l"
+fi
+echo "  vi ${ENV_FILE}"

domain-api/deploy/multi-region/bootstrap_overseas.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+BASE_DIR="${1:-/opt/domaincheck}"
+API_DIR="$BASE_DIR/domain-api"
+WEB_DIR="$BASE_DIR/domain-web"
+RUNTIME_DIR="$API_DIR/runtime"
+SERVICE_NAME="domaincheck-api"
+ENV_FILE="/etc/default/domaincheck-api"
+TEMPLATE_FILE="$API_DIR/deploy/multi-region/templates/domaincheck-api.env.example"
+
+echo "[1/6] preparing directories"
+mkdir -p "$BASE_DIR" "$API_DIR" "$WEB_DIR" "$RUNTIME_DIR"
+
+echo "[2/6] checking python environment"
+if [ ! -x "$BASE_DIR/domainCheck/.venv/bin/python" ]; then
+  echo "missing python venv: $BASE_DIR/domainCheck/.venv/bin/python"
+  echo "please deploy domainCheck first and create .venv there"
+  exit 1
+fi
+
+echo "[3/6] installing api dependencies"
+"$BASE_DIR/domainCheck/.venv/bin/pip" install -q fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
+
+echo "[4/6] installing systemd service"
+install -m 0644 "$API_DIR/deploy/systemd/domain-api.service" "/etc/systemd/system/${SERVICE_NAME}.service"
+
+echo "[4.5/6] preparing environment file"
+if [ ! -f "$ENV_FILE" ]; then
+  install -m 0644 "$TEMPLATE_FILE" "$ENV_FILE"
+  sed -i "s#__BASE_DIR__#${BASE_DIR}#g" "$ENV_FILE"
+  chmod 0644 "$ENV_FILE"
+  echo "created env template: $ENV_FILE"
+else
+  echo "env file already exists: $ENV_FILE"
+fi
+
+echo "[5/6] reloading systemd"
+systemctl daemon-reload
+
+echo "[6/6] summary"
+echo "overseas bootstrap completed"
+echo "service: ${SERVICE_NAME}"
+echo "next commands:"
+echo "  systemctl enable ${SERVICE_NAME}"
+echo "  systemctl restart ${SERVICE_NAME}"
+echo "  systemctl status ${SERVICE_NAME} --no-pager -l"
+echo "  vi ${ENV_FILE}"

domain-api/deploy/multi-region/check_cluster.sh

@@ -0,0 +1,89 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+BASE_URL="${1:-http://127.0.0.1:8100}"
+PYTHON_BIN="${PYTHON_BIN:-python3}"
+
+echo "[1/7] health"
+curl -fsS "${BASE_URL}/health"
+echo
+echo
+
+echo "[2/7] runtime readiness"
+curl -fsS "${BASE_URL}/api/v1/runtime/readiness"
+echo
+echo
+
+echo "[3/7] runtime cluster"
+curl -fsS "${BASE_URL}/api/v1/runtime/cluster"
+echo
+echo
+
+echo "[4/7] runtime sync summary"
+curl -fsS "${BASE_URL}/api/v1/runtime/sync-summary"
+echo
+echo
+
+echo "[5/7] detect status"
+curl -fsS "${BASE_URL}/api/v1/detect/status"
+echo
+echo
+
+echo "[6/7] active detect job"
+curl -fsS "${BASE_URL}/api/v1/detect/job/active"
+echo
+echo
+
+echo "[7/7] condensed summary"
+READINESS_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/readiness")"
+SYNC_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/sync-summary")"
+CLUSTER_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/cluster")"
+"${PYTHON_BIN}" - <<'PY' "$READINESS_JSON" "$SYNC_JSON" "$CLUSTER_JSON"
+import json
+import sys
+
+readiness = json.loads(sys.argv[1]).get("data", {})
+sync = json.loads(sys.argv[2]).get("data", {})
+cluster = json.loads(sys.argv[3]).get("data", {})
+batches = (sync.get("detect_result_batches") or {})
+states = batches.get("state_counts") or {}
+summary = cluster.get("summary") or {}
+print(
+    json.dumps(
+        {
+            "readiness": {
+                "status": readiness.get("status"),
+                "ready": bool(readiness.get("ready", False)),
+                "summary": readiness.get("summary", ""),
+            },
+            "cluster": {
+                "online_control_nodes": summary.get("online_control_nodes", 0),
+                "online_worker_nodes": summary.get("online_worker_nodes", 0),
+                "busy_nodes": summary.get("busy_nodes", []),
+                "stale_nodes": summary.get("stale_nodes", []),
+                "offline_nodes": summary.get("offline_nodes", []),
+            },
+            "sync": {
+                "enabled": bool(sync.get("enabled", False)),
+                "source_region": sync.get("source_region"),
+                "target_region": sync.get("target_region"),
+                "runtime_projection": (sync.get("type_counts") or {}).get("runtime_projection", 0),
+                "detect_result_projection": (sync.get("type_counts") or {}).get("detect_result_projection", 0),
+                "runtime_ingest": (sync.get("type_counts") or {}).get("runtime_ingest", 0),
+                "detect_result_ingest": (sync.get("type_counts") or {}).get("detect_result_ingest", 0),
+            },
+            "detect_result_batches": {
+                "jobs_total": batches.get("jobs_total", 0),
+                "synced": states.get("synced", 0),
+                "delivered": states.get("delivered", 0),
+                "pushing": states.get("pushing", 0),
+                "projected": states.get("projected", 0),
+                "failed": states.get("failed", 0),
+                "unsynced": states.get("unsynced", 0),
+            },
+        },
+        ensure_ascii=False,
+        indent=2,
+    )
+)
+PY

domain-api/deploy/multi-region/check_mainland_controller.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+ENV_FILE="${1:-/etc/default/domaincheck-worker}"
+WORKER_SERVICE="${WORKER_SERVICE:-domaincheck-worker}"
+SYNC_AGENT_SERVICE="${SYNC_AGENT_SERVICE:-domaincheck-sync-agent}"
+PYTHON_BIN="${PYTHON_BIN:-python3}"
+
+echo "[1/5] environment file"
+if [ ! -f "$ENV_FILE" ]; then
+  echo "missing env file: $ENV_FILE"
+  exit 1
+fi
+echo "env file: $ENV_FILE"
+echo
+
+echo "[2/5] key env summary"
+"${PYTHON_BIN}" - <<'PY' "$ENV_FILE"
+import json
+import sys
+from pathlib import Path
+
+env_path = Path(sys.argv[1])
+values = {}
+for line in env_path.read_text(encoding="utf-8").splitlines():
+    text = line.strip()
+    if not text or text.startswith("#") or "=" not in text:
+        continue
+    key, value = text.split("=", 1)
+    values[key.strip()] = value.strip()
+
+required = {
+    "NODE_CODE": values.get("NODE_CODE", ""),
+    "NODE_REGION": values.get("NODE_REGION", ""),
+    "NODE_ROLE": values.get("NODE_ROLE", ""),
+    "SYNC_PUSH_ENABLED": values.get("SYNC_PUSH_ENABLED", ""),
+    "SYNC_SOURCE_REGION": values.get("SYNC_SOURCE_REGION", ""),
+    "SYNC_TARGET_REGION": values.get("SYNC_TARGET_REGION", ""),
+    "SYNC_TARGET_API_BASE_URL": values.get("SYNC_TARGET_API_BASE_URL", ""),
+}
+print(json.dumps(required, ensure_ascii=False, indent=2))
+
+errors = []
+if required["NODE_REGION"] != "mainland":
+    errors.append("NODE_REGION 必须为 mainland")
+if required["NODE_ROLE"] != "control":
+    errors.append("NODE_ROLE 必须为 control")
+if str(required["SYNC_PUSH_ENABLED"]).lower() != "true":
+    errors.append("SYNC_PUSH_ENABLED 必须为 true")
+if required["SYNC_SOURCE_REGION"] != "mainland":
+    errors.append("SYNC_SOURCE_REGION 必须为 mainland")
+if not required["SYNC_TARGET_API_BASE_URL"]:
+    errors.append("必须配置 SYNC_TARGET_API_BASE_URL")
+
+if errors:
+    print()
+    print("env validation failed:")
+    for item in errors:
+        print(f"- {item}")
+    sys.exit(1)
+PY
+echo
+
+echo "[3/5] systemd status"
+systemctl is-enabled "$WORKER_SERVICE" || true
+systemctl is-active "$WORKER_SERVICE"
+systemctl is-enabled "$SYNC_AGENT_SERVICE" || true
+systemctl is-active "$SYNC_AGENT_SERVICE"
+echo
+
+echo "[4/5] worker service detail"
+systemctl status "$WORKER_SERVICE" --no-pager -l | sed -n '1,25p'
+echo
+
+echo "[5/5] sync-agent detail"
+systemctl status "$SYNC_AGENT_SERVICE" --no-pager -l | sed -n '1,25p'

domain-api/deploy/multi-region/install_mainland_controller_quick.conf.example

@@ -0,0 +1,8 @@
+GIT_REPO_URL="YOUR_GIT_REPO_URL"
+GIT_BRANCH="main"
+DB_NAME="domain"
+DB_USER="domainuser"
+DB_PASSWORD="CHANGE_ME_DB_PASSWORD"
+SYNC_SHARED_TOKEN="CHANGE_ME_SYNC_SHARED_TOKEN"
+MAINLAND_CONTROLLER_NODE_CODE="mainland-controller-01"
+TARGET_API_BASE_URL="https://api.domain.com/api/v1"

domain-api/deploy/multi-region/install_mainland_controller_quick.sh

@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+CONFIG_FILE="${SCRIPT_DIR}/install_mainland_controller_quick.conf"
+CONFIG_EXAMPLE="${SCRIPT_DIR}/install_mainland_controller_quick.conf.example"
+
+if [[ ! -f "${CONFIG_FILE}" ]]; then
+  echo "未找到配置文件：${CONFIG_FILE}"
+  echo "请先执行：cp ${CONFIG_EXAMPLE} ${CONFIG_FILE}"
+  exit 1
+fi
+
+# shellcheck disable=SC1090
+source "${CONFIG_FILE}"
+
+: "${GIT_REPO_URL:=YOUR_GIT_REPO_URL}"
+: "${GIT_BRANCH:=main}"
+: "${DB_NAME:=domain}"
+: "${DB_USER:=domainuser}"
+: "${DB_PASSWORD:=CHANGE_ME_DB_PASSWORD}"
+: "${SYNC_SHARED_TOKEN:=CHANGE_ME_SYNC_SHARED_TOKEN}"
+: "${MAINLAND_CONTROLLER_NODE_CODE:=mainland-controller-01}"
+: "${TARGET_API_BASE_URL:=https://api.domain.com/api/v1}"
+
+WORKSPACE_ROOT="/www/wwwroot"
+REPO_DIR="${WORKSPACE_ROOT}/getDomain"
+BASE_DIR="/opt/domaincheck"
+DOMAINCHECK_DIR="${BASE_DIR}/domainCheck"
+DOMAIN_API_DIR="${BASE_DIR}/domain-api"
+
+if [[ "${GIT_REPO_URL}" == "YOUR_GIT_REPO_URL" || "${DB_PASSWORD}" == "CHANGE_ME_DB_PASSWORD" || "${SYNC_SHARED_TOKEN}" == "CHANGE_ME_SYNC_SHARED_TOKEN" ]]; then
+  echo "请先修改配置文件中的必填项：GIT_REPO_URL / DB_PASSWORD / SYNC_SHARED_TOKEN"
+  exit 1
+fi
+
+mkdir -p "${WORKSPACE_ROOT}"
+cd "${WORKSPACE_ROOT}"
+if [[ ! -d "${REPO_DIR}/.git" ]]; then
+  git clone "${GIT_REPO_URL}" getDomain
+fi
+cd "${REPO_DIR}"
+git fetch --all
+git checkout "${GIT_BRANCH}"
+git pull --ff-only origin "${GIT_BRANCH}"
+
+mkdir -p "${BASE_DIR}"
+ln -sfn "${REPO_DIR}/domain-api" "${DOMAIN_API_DIR}"
+ln -sfn "${REPO_DIR}/domain-web" "${BASE_DIR}/domain-web"
+ln -sfn "${REPO_DIR}/domainCheck" "${DOMAINCHECK_DIR}"
+
+cd "${DOMAINCHECK_DIR}"
+python3.11 -m venv .venv || true
+source .venv/bin/activate
+pip install --upgrade pip
+pip install -r requirements.txt
+"${DOMAINCHECK_DIR}/.venv/bin/pip" install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
+
+cat >"${DOMAINCHECK_DIR}/.env" <<EOF
+DB_HOST=127.0.0.1
+DB_PORT=5432
+DB_DATABASE=${DB_NAME}
+DB_USER=${DB_USER}
+DB_PASSWORD=${DB_PASSWORD}
+
+REDIS_HOST=127.0.0.1
+REDIS_PORT=6379
+REDIS_PASSWORD=
+REDIS_DB=0
+EOF
+
+cd "${DOMAIN_API_DIR}"
+bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck controller
+
+cat >/etc/default/domaincheck-worker <<EOF
+WORKER_MODE=linux-systemd
+QT_QPA_PLATFORM=offscreen
+NODE_CODE=${MAINLAND_CONTROLLER_NODE_CODE}
+NODE_REGION=mainland
+NODE_ROLE=control
+
+DB_HOST=127.0.0.1
+DB_PORT=5432
+DB_DATABASE=${DB_NAME}
+DB_USER=${DB_USER}
+DB_PASSWORD=${DB_PASSWORD}
+
+REDIS_HOST=127.0.0.1
+REDIS_PORT=6379
+REDIS_PASSWORD=
+REDIS_DB=0
+
+SYNC_PUSH_ENABLED=true
+SYNC_SOURCE_REGION=mainland
+SYNC_TARGET_REGION=overseas
+SYNC_TARGET_API_BASE_URL=${TARGET_API_BASE_URL}
+SYNC_SHARED_TOKEN=${SYNC_SHARED_TOKEN}
+SYNC_BATCH_SIZE=200
+SYNC_POLL_INTERVAL_SECONDS=30
+EOF
+
+systemctl daemon-reload
+systemctl enable domaincheck-worker
+systemctl enable domaincheck-sync-agent
+systemctl restart domaincheck-worker
+systemctl restart domaincheck-sync-agent
+
+echo
+echo "国内 controller 部署完成，建议继续执行："
+echo "cd ${DOMAIN_API_DIR} && bash deploy/multi-region/check_mainland_controller.sh"

domain-api/deploy/multi-region/install_mainland_worker_quick.conf.example

@@ -0,0 +1,9 @@
+GIT_REPO_URL="YOUR_GIT_REPO_URL"
+GIT_BRANCH="main"
+DB_NAME="domain"
+DB_USER="domainuser"
+DB_PASSWORD="CHANGE_ME_DB_PASSWORD"
+SYNC_SHARED_TOKEN="CHANGE_ME_SYNC_SHARED_TOKEN"
+MAINLAND_WORKER_NODE_CODE="mainland-worker-01"
+MAINLAND_CONTROLLER_IP="127.0.0.1"
+TARGET_API_BASE_URL="https://api.domain.com/api/v1"

domain-api/deploy/multi-region/install_mainland_worker_quick.sh

@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+CONFIG_FILE="${SCRIPT_DIR}/install_mainland_worker_quick.conf"
+CONFIG_EXAMPLE="${SCRIPT_DIR}/install_mainland_worker_quick.conf.example"
+
+if [[ ! -f "${CONFIG_FILE}" ]]; then
+  echo "未找到配置文件：${CONFIG_FILE}"
+  echo "请先执行：cp ${CONFIG_EXAMPLE} ${CONFIG_FILE}"
+  exit 1
+fi
+
+# shellcheck disable=SC1090
+source "${CONFIG_FILE}"
+
+: "${GIT_REPO_URL:=YOUR_GIT_REPO_URL}"
+: "${GIT_BRANCH:=main}"
+: "${DB_NAME:=domain}"
+: "${DB_USER:=domainuser}"
+: "${DB_PASSWORD:=CHANGE_ME_DB_PASSWORD}"
+: "${SYNC_SHARED_TOKEN:=CHANGE_ME_SYNC_SHARED_TOKEN}"
+: "${MAINLAND_WORKER_NODE_CODE:=mainland-worker-01}"
+: "${MAINLAND_CONTROLLER_IP:=127.0.0.1}"
+: "${TARGET_API_BASE_URL:=https://api.domain.com/api/v1}"
+
+WORKSPACE_ROOT="/www/wwwroot"
+REPO_DIR="${WORKSPACE_ROOT}/getDomain"
+BASE_DIR="/opt/domaincheck"
+DOMAINCHECK_DIR="${BASE_DIR}/domainCheck"
+DOMAIN_API_DIR="${BASE_DIR}/domain-api"
+
+if [[ "${GIT_REPO_URL}" == "YOUR_GIT_REPO_URL" || "${DB_PASSWORD}" == "CHANGE_ME_DB_PASSWORD" || "${SYNC_SHARED_TOKEN}" == "CHANGE_ME_SYNC_SHARED_TOKEN" || "${MAINLAND_CONTROLLER_IP}" == "127.0.0.1" ]]; then
+  echo "请先修改配置文件中的必填项：GIT_REPO_URL / DB_PASSWORD / SYNC_SHARED_TOKEN / MAINLAND_CONTROLLER_IP"
+  exit 1
+fi
+
+mkdir -p "${WORKSPACE_ROOT}"
+cd "${WORKSPACE_ROOT}"
+if [[ ! -d "${REPO_DIR}/.git" ]]; then
+  git clone "${GIT_REPO_URL}" getDomain
+fi
+cd "${REPO_DIR}"
+git fetch --all
+git checkout "${GIT_BRANCH}"
+git pull --ff-only origin "${GIT_BRANCH}"
+
+mkdir -p "${BASE_DIR}"
+ln -sfn "${REPO_DIR}/domain-api" "${DOMAIN_API_DIR}"
+ln -sfn "${REPO_DIR}/domain-web" "${BASE_DIR}/domain-web"
+ln -sfn "${REPO_DIR}/domainCheck" "${DOMAINCHECK_DIR}"
+
+cd "${DOMAINCHECK_DIR}"
+python3.11 -m venv .venv || true
+source .venv/bin/activate
+pip install --upgrade pip
+pip install -r requirements.txt
+"${DOMAINCHECK_DIR}/.venv/bin/pip" install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
+
+cat >"${DOMAINCHECK_DIR}/.env" <<EOF
+DB_HOST=${MAINLAND_CONTROLLER_IP}
+DB_PORT=5432
+DB_DATABASE=${DB_NAME}
+DB_USER=${DB_USER}
+DB_PASSWORD=${DB_PASSWORD}
+
+REDIS_HOST=${MAINLAND_CONTROLLER_IP}
+REDIS_PORT=6379
+REDIS_PASSWORD=
+REDIS_DB=0
+EOF
+
+cd "${DOMAIN_API_DIR}"
+bash deploy/multi-region/bootstrap_mainland.sh /opt/domaincheck worker
+
+cat >/etc/default/domaincheck-worker <<EOF
+WORKER_MODE=linux-systemd
+QT_QPA_PLATFORM=offscreen
+NODE_CODE=${MAINLAND_WORKER_NODE_CODE}
+NODE_REGION=mainland
+NODE_ROLE=worker
+
+DB_HOST=${MAINLAND_CONTROLLER_IP}
+DB_PORT=5432
+DB_DATABASE=${DB_NAME}
+DB_USER=${DB_USER}
+DB_PASSWORD=${DB_PASSWORD}
+
+REDIS_HOST=${MAINLAND_CONTROLLER_IP}
+REDIS_PORT=6379
+REDIS_PASSWORD=
+REDIS_DB=0
+
+SYNC_PUSH_ENABLED=true
+SYNC_SOURCE_REGION=mainland
+SYNC_TARGET_REGION=overseas
+SYNC_TARGET_API_BASE_URL=${TARGET_API_BASE_URL}
+SYNC_SHARED_TOKEN=${SYNC_SHARED_TOKEN}
+SYNC_BATCH_SIZE=200
+SYNC_POLL_INTERVAL_SECONDS=30
+EOF
+
+systemctl daemon-reload
+systemctl enable domaincheck-worker
+systemctl restart domaincheck-worker
+
+echo
+echo "国内 worker 部署完成，建议继续在国外机查看："
+echo "curl https://api.domain.com/api/v1/runtime/cluster"

domain-api/deploy/multi-region/install_overseas_quick.conf.example

@@ -0,0 +1,9 @@
+GIT_REPO_URL="YOUR_GIT_REPO_URL"
+GIT_BRANCH="main"
+API_DOMAIN="api.domain.com"
+ADMIN_DOMAIN="admin.domain"
+DB_NAME="domain"
+DB_USER="domainuser"
+DB_PASSWORD="CHANGE_ME_DB_PASSWORD"
+OVERSEAS_NODE_CODE="overseas-control-01"
+NODE_VERSION="20"

domain-api/deploy/multi-region/install_overseas_quick.sh

@@ -0,0 +1,124 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+CONFIG_FILE="${SCRIPT_DIR}/install_overseas_quick.conf"
+CONFIG_EXAMPLE="${SCRIPT_DIR}/install_overseas_quick.conf.example"
+
+if [[ ! -f "${CONFIG_FILE}" ]]; then
+  echo "未找到配置文件：${CONFIG_FILE}"
+  echo "请先执行：cp ${CONFIG_EXAMPLE} ${CONFIG_FILE}"
+  exit 1
+fi
+
+# shellcheck disable=SC1090
+source "${CONFIG_FILE}"
+
+: "${GIT_REPO_URL:=YOUR_GIT_REPO_URL}"
+: "${GIT_BRANCH:=main}"
+: "${API_DOMAIN:=api.domain.com}"
+: "${ADMIN_DOMAIN:=admin.domain}"
+: "${DB_NAME:=domain}"
+: "${DB_USER:=domainuser}"
+: "${DB_PASSWORD:=CHANGE_ME_DB_PASSWORD}"
+: "${OVERSEAS_NODE_CODE:=overseas-control-01}"
+: "${NODE_VERSION:=20}"
+
+WORKSPACE_ROOT="/www/wwwroot"
+REPO_DIR="${WORKSPACE_ROOT}/getDomain"
+BASE_DIR="/opt/domaincheck"
+DOMAINCHECK_DIR="${BASE_DIR}/domainCheck"
+DOMAIN_API_DIR="${BASE_DIR}/domain-api"
+DOMAIN_WEB_DIR="${BASE_DIR}/domain-web"
+
+if [[ "${GIT_REPO_URL}" == "YOUR_GIT_REPO_URL" || "${DB_PASSWORD}" == "CHANGE_ME_DB_PASSWORD" ]]; then
+  echo "请先修改配置文件中的必填项：GIT_REPO_URL / DB_PASSWORD"
+  exit 1
+fi
+
+mkdir -p "${WORKSPACE_ROOT}"
+cd "${WORKSPACE_ROOT}"
+if [[ ! -d "${REPO_DIR}/.git" ]]; then
+  git clone "${GIT_REPO_URL}" getDomain
+fi
+cd "${REPO_DIR}"
+git fetch --all
+git checkout "${GIT_BRANCH}"
+git pull --ff-only origin "${GIT_BRANCH}"
+
+mkdir -p "${BASE_DIR}"
+ln -sfn "${REPO_DIR}/domain-api" "${DOMAIN_API_DIR}"
+ln -sfn "${REPO_DIR}/domain-web" "${DOMAIN_WEB_DIR}"
+ln -sfn "${REPO_DIR}/domainCheck" "${DOMAINCHECK_DIR}"
+
+cd "${DOMAINCHECK_DIR}"
+python3.11 -m venv .venv || true
+source .venv/bin/activate
+pip install --upgrade pip
+pip install -r requirements.txt
+"${DOMAINCHECK_DIR}/.venv/bin/pip" install fastapi uvicorn pydantic-settings psycopg2-binary redis openpyxl python-multipart
+
+cat >"${DOMAINCHECK_DIR}/.env" <<EOF
+DB_HOST=127.0.0.1
+DB_PORT=5432
+DB_DATABASE=${DB_NAME}
+DB_USER=${DB_USER}
+DB_PASSWORD=${DB_PASSWORD}
+
+REDIS_HOST=127.0.0.1
+REDIS_PORT=6379
+REDIS_PASSWORD=
+REDIS_DB=0
+EOF
+
+"${DOMAINCHECK_DIR}/.venv/bin/python" "${DOMAINCHECK_DIR}/init_database.py"
+
+cat >"${DOMAIN_WEB_DIR}/.env.production" <<EOF
+VITE_API_BASE_URL=https://${API_DOMAIN}/api/v1
+EOF
+
+export NVM_DIR="${HOME}/.nvm"
+if [[ -s "${NVM_DIR}/nvm.sh" ]]; then
+  # shellcheck disable=SC1090
+  source "${NVM_DIR}/nvm.sh"
+else
+  echo "未找到 nvm，请先安装 nvm"
+  exit 1
+fi
+nvm install "${NODE_VERSION}"
+nvm use "${NODE_VERSION}"
+
+cd "${DOMAIN_WEB_DIR}"
+npm install
+npm run build
+
+cp "${DOMAIN_API_DIR}/deploy/multi-region/templates/domaincheck-api.env.example" /etc/default/domaincheck-api
+sed -i "s#^DOMAIN_ROOT=.*#DOMAIN_ROOT=${DOMAINCHECK_DIR}#" /etc/default/domaincheck-api
+sed -i 's#^API_HOST=.*#API_HOST=0.0.0.0#' /etc/default/domaincheck-api
+sed -i 's#^API_PORT=.*#API_PORT=8100#' /etc/default/domaincheck-api
+sed -i "s#^NODE_CODE=.*#NODE_CODE=${OVERSEAS_NODE_CODE}#" /etc/default/domaincheck-api
+sed -i 's#^NODE_REGION=.*#NODE_REGION=overseas#' /etc/default/domaincheck-api
+sed -i 's#^NODE_ROLE=.*#NODE_ROLE=control#' /etc/default/domaincheck-api
+sed -i 's#^SYNC_PUSH_ENABLED=.*#SYNC_PUSH_ENABLED=false#' /etc/default/domaincheck-api
+sed -i "s#^CORS_ORIGINS=.*#CORS_ORIGINS=https://${ADMIN_DOMAIN},http://127.0.0.1:3201,http://localhost:3201#" /etc/default/domaincheck-api
+
+cp "${DOMAIN_API_DIR}/deploy/systemd/domain-api.service" /etc/systemd/system/domaincheck-api.service
+cp "${DOMAIN_API_DIR}/deploy/systemd/domain-worker.service" /etc/systemd/system/domaincheck-worker.service
+
+mkdir -p "${DOMAIN_API_DIR}/runtime"
+touch "${DOMAINCHECK_DIR}/detect_worker.log"
+chown -R www:www "${DOMAIN_API_DIR}/runtime"
+chown www:www "${DOMAINCHECK_DIR}/detect_worker.log"
+chmod 664 "${DOMAINCHECK_DIR}/detect_worker.log"
+
+systemctl daemon-reload
+systemctl enable domaincheck-api
+systemctl enable domaincheck-worker
+systemctl restart domaincheck-api
+systemctl restart domaincheck-worker
+
+echo
+echo "国外主控机部署完成，建议继续执行："
+echo "curl http://127.0.0.1:8100/health"
+echo "curl http://127.0.0.1:8100/api/v1/runtime/preflight"
+echo "curl http://127.0.0.1:8100/api/v1/runtime/readiness"

domain-api/deploy/multi-region/prune_cluster_nodes.py

@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import sys
+from datetime import datetime, timedelta
+from pathlib import Path
+
+API_ROOT = Path(__file__).resolve().parents[2]
+if str(API_ROOT) not in sys.path:
+    sys.path.insert(0, str(API_ROOT))
+
+from app.core.db import get_db
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Prune stale/offline cluster node records from detect_worker_nodes.")
+    parser.add_argument("--minutes", type=int, default=30, help="delete nodes whose heartbeat is older than this many minutes")
+    parser.add_argument("--node-code", action="append", default=[], help="delete a specific node code; can be passed multiple times")
+    parser.add_argument("--dry-run", action="store_true")
+    args = parser.parse_args()
+
+    cutoff = datetime.now() - timedelta(minutes=max(1, int(args.minutes or 30)))
+    removed: list[dict] = []
+    with get_db() as conn:
+        with conn.cursor() as cur:
+            if args.node_code:
+                cur.execute(
+                    """
+                    SELECT node_code, region, role, status, last_heartbeat_at
+                    FROM detect_worker_nodes
+                    WHERE node_code = ANY(%s)
+                    ORDER BY node_code ASC
+                    """,
+                    (list(args.node_code),),
+                )
+            else:
+                cur.execute(
+                    """
+                    SELECT node_code, region, role, status, last_heartbeat_at
+                    FROM detect_worker_nodes
+                    WHERE last_heartbeat_at < %s
+                    ORDER BY last_heartbeat_at ASC, node_code ASC
+                    """,
+                    (cutoff,),
+                )
+            rows = cur.fetchall()
+            for row in rows:
+                removed.append(
+                    {
+                        "node_code": row[0],
+                        "region": row[1],
+                        "role": row[2],
+                        "status": row[3],
+                        "last_heartbeat_at": row[4].isoformat(sep=" ", timespec="seconds") if row[4] else "",
+                    }
+                )
+            if removed and not args.dry_run:
+                cur.execute(
+                    "DELETE FROM detect_worker_nodes WHERE node_code = ANY(%s)",
+                    ([item["node_code"] for item in removed],),
+                )
+        if removed and not args.dry_run:
+            conn.commit()
+
+    if not removed:
+        print("no cluster nodes matched prune conditions")
+        return
+
+    print("matched cluster nodes:")
+    for item in removed:
+        print(
+            f"- {item['node_code']} | {item['region']} | {item['role']} | {item['status']} | {item['last_heartbeat_at']}"
+        )
+    if args.dry_run:
+        print("dry-run only, nothing deleted")
+    else:
+        print(f"deleted {len(removed)} cluster node record(s)")
+
+
+if __name__ == "__main__":
+    main()

domain-api/deploy/multi-region/prune_cluster_nodes.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}"
+
+exec "$PYTHON_BIN" "$API_DIR/deploy/multi-region/prune_cluster_nodes.py" "$@"

domain-api/deploy/multi-region/rehearse_multi_region.sh

@@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+BASE_URL="${1:-http://127.0.0.1:8100}"
+API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}"
+SIM_SCRIPT="$API_DIR/deploy/multi-region/simulate_cluster_node.py"
+PRUNE_SCRIPT="$API_DIR/deploy/multi-region/prune_cluster_nodes.sh"
+RUNTIME_DIR="$API_DIR/runtime"
+mkdir -p "$RUNTIME_DIR"
+
+CTRL_LOG="$RUNTIME_DIR/rehearse-mainland-controller.log"
+WORKER1_LOG="$RUNTIME_DIR/rehearse-mainland-worker-01.log"
+WORKER2_LOG="$RUNTIME_DIR/rehearse-mainland-worker-02.log"
+
+CTRL_NODE="mainland-controller-rehearsal"
+WORKER1_NODE="mainland-worker-rehearsal-01"
+WORKER2_NODE="mainland-worker-rehearsal-02"
+
+cleanup() {
+  set +e
+  if [ -n "${CTRL_PID:-}" ]; then kill "$CTRL_PID" >/dev/null 2>&1 || true; fi
+  if [ -n "${WORKER1_PID:-}" ]; then kill "$WORKER1_PID" >/dev/null 2>&1 || true; fi
+  if [ -n "${WORKER2_PID:-}" ]; then kill "$WORKER2_PID" >/dev/null 2>&1 || true; fi
+  bash "$PRUNE_SCRIPT" --node-code "$CTRL_NODE" --node-code "$WORKER1_NODE" --node-code "$WORKER2_NODE" >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+echo "[0/6] cleaning possible stale rehearsal nodes"
+bash "$PRUNE_SCRIPT" --node-code "$CTRL_NODE" --node-code "$WORKER1_NODE" --node-code "$WORKER2_NODE" >/dev/null 2>&1 || true
+
+echo "[1/6] start simulated mainland controller"
+"$PYTHON_BIN" "$SIM_SCRIPT" \
+  --node-code "$CTRL_NODE" \
+  --region mainland \
+  --role control \
+  --status online \
+  --detail "多机演练 controller" \
+  --interval 15 >"$CTRL_LOG" 2>&1 &
+CTRL_PID=$!
+
+echo "[2/6] start simulated mainland workers"
+"$PYTHON_BIN" "$SIM_SCRIPT" \
+  --node-code "$WORKER1_NODE" \
+  --region mainland \
+  --role worker \
+  --status busy \
+  --current-load 1 \
+  --phase "running" \
+  --detail "多机演练 worker-01" \
+  --job-id 1 \
+  --job-code "detect-rehearsal-cluster" \
+  --cycle-token "rehearsal001" \
+  --active-threads 5 \
+  --available-proxy-count 12 \
+  --interval 15 >"$WORKER1_LOG" 2>&1 &
+WORKER1_PID=$!
+
+"$PYTHON_BIN" "$SIM_SCRIPT" \
+  --node-code "$WORKER2_NODE" \
+  --region mainland \
+  --role worker \
+  --status online \
+  --current-load 0 \
+  --phase "idle" \
+  --detail "多机演练 worker-02" \
+  --job-id 0 \
+  --job-code "" \
+  --cycle-token "" \
+  --active-threads 0 \
+  --available-proxy-count 8 \
+  --interval 15 >"$WORKER2_LOG" 2>&1 &
+WORKER2_PID=$!
+
+sleep 2
+
+echo "[3/6] fetch runtime readiness"
+READINESS_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/readiness")"
+echo "$READINESS_JSON"
+echo
+
+echo "[4/6] fetch runtime cluster"
+CLUSTER_JSON="$(curl -fsS "${BASE_URL}/api/v1/runtime/cluster")"
+echo "$CLUSTER_JSON"
+echo
+
+echo "[5/6] assert rehearsal result"
+"${PYTHON_BIN}" - <<'PY' "$READINESS_JSON" "$CLUSTER_JSON" "$CTRL_NODE" "$WORKER1_NODE" "$WORKER2_NODE"
+import json
+import sys
+
+readiness = json.loads(sys.argv[1]).get("data", {})
+cluster = json.loads(sys.argv[2]).get("data", {})
+ctrl_node, worker1_node, worker2_node = sys.argv[3:6]
+
+errors = []
+if readiness.get("status") != "ready":
+    errors.append(f"readiness.status expected ready, got {readiness.get('status')!r}")
+if not readiness.get("ready", False):
+    errors.append("readiness.ready expected true")
+
+summary = cluster.get("summary") or {}
+if int(summary.get("online_control_nodes", 0) or 0) < 2:
+    errors.append(f"online_control_nodes expected >= 2, got {summary.get('online_control_nodes')}")
+if int(summary.get("online_worker_nodes", 0) or 0) < 2:
+    errors.append(f"online_worker_nodes expected >= 2, got {summary.get('online_worker_nodes')}")
+
+nodes = {str(item.get('node_code') or ''): item for item in (cluster.get("nodes") or [])}
+for node_code in (ctrl_node, worker1_node, worker2_node):
+    if node_code not in nodes:
+        errors.append(f"missing simulated node in cluster: {node_code}")
+
+if worker1_node in nodes and str(nodes[worker1_node].get("status")) != "busy":
+    errors.append(f"{worker1_node} expected busy, got {nodes[worker1_node].get('status')!r}")
+if worker2_node in nodes and str(nodes[worker2_node].get("status")) != "online":
+    errors.append(f"{worker2_node} expected online, got {nodes[worker2_node].get('status')!r}")
+
+if errors:
+    print("rehearsal failed:")
+    for item in errors:
+        print(f"- {item}")
+    raise SystemExit(1)
+
+print("rehearsal passed")
+print(json.dumps({
+    "readiness": readiness,
+    "cluster_summary": summary,
+}, ensure_ascii=False, indent=2))
+PY
+
+echo
+echo "[6/6] rehearsal completed"
+echo "controller log: $CTRL_LOG"
+echo "worker1 log: $WORKER1_LOG"
+echo "worker2 log: $WORKER2_LOG"

domain-api/deploy/multi-region/simulate_cluster_node.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+from pathlib import Path
+
+API_ROOT = Path(__file__).resolve().parents[2]
+if str(API_ROOT) not in sys.path:
+    sys.path.insert(0, str(API_ROOT))
+
+from app.core.db import get_db
+from app.services.cluster_runtime_service import ensure_runtime_schema, register_node_heartbeat
+
+
+def _metadata_from_args(args: argparse.Namespace) -> dict:
+    metadata = {
+        "service": "cluster-simulator",
+        "simulated": True,
+        "phase": args.phase or "",
+        "detail": args.detail or "",
+    }
+    if args.job_id:
+        metadata["job_id"] = args.job_id
+    if args.job_code:
+        metadata["job_code"] = args.job_code
+    if args.cycle_token:
+        metadata["cycle_token"] = args.cycle_token
+    if args.active_threads is not None:
+        metadata["active_threads"] = args.active_threads
+    if args.available_proxy_count is not None:
+        metadata["available_proxy_count"] = args.available_proxy_count
+    if args.metadata_json:
+        try:
+            custom = json.loads(args.metadata_json)
+            if isinstance(custom, dict):
+                metadata.update(custom)
+        except Exception as exc:
+            raise SystemExit(f"invalid --metadata-json: {exc}") from exc
+    return metadata
+
+
+def _delete_node(node_code: str) -> None:
+    with get_db() as conn:
+        with conn.cursor() as cur:
+            cur.execute("DELETE FROM detect_worker_nodes WHERE node_code = %s", (str(node_code),))
+        conn.commit()
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Simulate a cluster node heartbeat for multi-region rehearsal.")
+    parser.add_argument("--node-code", required=True)
+    parser.add_argument("--region", required=True)
+    parser.add_argument("--role", required=True, choices=("control", "worker"))
+    parser.add_argument("--status", default="online")
+    parser.add_argument("--current-load", type=int, default=0)
+    parser.add_argument("--phase", default="")
+    parser.add_argument("--detail", default="")
+    parser.add_argument("--job-id", type=int, default=0)
+    parser.add_argument("--job-code", default="")
+    parser.add_argument("--cycle-token", default="")
+    parser.add_argument("--active-threads", type=int, default=None)
+    parser.add_argument("--available-proxy-count", type=int, default=None)
+    parser.add_argument("--metadata-json", default="")
+    parser.add_argument("--interval", type=int, default=20)
+    parser.add_argument("--iterations", type=int, default=0, help="0 means run forever")
+    parser.add_argument("--clear", action="store_true", help="delete the node record and exit")
+    args = parser.parse_args()
+
+    ensure_runtime_schema()
+    if args.clear:
+        _delete_node(args.node_code)
+        print(f"cleared simulated node: {args.node_code}")
+        return
+
+    metadata = _metadata_from_args(args)
+    interval = max(5, int(args.interval or 20))
+    iterations = max(0, int(args.iterations or 0))
+    current = 0
+    while True:
+        current += 1
+        register_node_heartbeat(
+            node_code=args.node_code,
+            region=args.region,
+            role=args.role,
+            status=args.status,
+            current_load=max(0, int(args.current_load or 0)),
+            metadata=metadata,
+        )
+        print(
+            json.dumps(
+                {
+                    "node_code": args.node_code,
+                    "region": args.region,
+                    "role": args.role,
+                    "status": args.status,
+                    "current_load": max(0, int(args.current_load or 0)),
+                    "iteration": current,
+                    "metadata": metadata,
+                },
+                ensure_ascii=False,
+            ),
+            flush=True,
+        )
+        if iterations and current >= iterations:
+            break
+        time.sleep(interval)
+
+
+if __name__ == "__main__":
+    main()

domain-api/deploy/multi-region/simulate_multi_region.sh

@@ -0,0 +1,65 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+BASE_URL="${1:-http://127.0.0.1:8100}"
+API_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
+PYTHON_BIN="${PYTHON_BIN:-/opt/domaincheck/domainCheck/.venv/bin/python}"
+SIM_SCRIPT="$API_DIR/deploy/multi-region/simulate_cluster_node.py"
+RUNTIME_DIR="$API_DIR/runtime"
+mkdir -p "$RUNTIME_DIR"
+
+CONTROLLER_LOG="$RUNTIME_DIR/sim-mainland-controller.log"
+WORKER_LOG="$RUNTIME_DIR/sim-mainland-worker.log"
+
+cleanup() {
+  set +e
+  if [ -n "${CTRL_PID:-}" ]; then kill "$CTRL_PID" >/dev/null 2>&1 || true; fi
+  if [ -n "${WORKER_PID:-}" ]; then kill "$WORKER_PID" >/dev/null 2>&1 || true; fi
+  "$PYTHON_BIN" "$SIM_SCRIPT" --node-code mainland-controller-sim --region mainland --role control --clear >/dev/null 2>&1 || true
+  "$PYTHON_BIN" "$SIM_SCRIPT" --node-code mainland-worker-sim-01 --region mainland --role worker --clear >/dev/null 2>&1 || true
+}
+trap cleanup EXIT
+
+echo "[1/4] start simulated mainland controller"
+"$PYTHON_BIN" "$SIM_SCRIPT" \
+  --node-code mainland-controller-sim \
+  --region mainland \
+  --role control \
+  --status online \
+  --detail "模拟大陆 controller 联调节点" \
+  --interval 15 >"$CONTROLLER_LOG" 2>&1 &
+CTRL_PID=$!
+
+echo "[2/4] start simulated mainland worker"
+"$PYTHON_BIN" "$SIM_SCRIPT" \
+  --node-code mainland-worker-sim-01 \
+  --region mainland \
+  --role worker \
+  --status busy \
+  --current-load 1 \
+  --phase "running" \
+  --detail "模拟大陆 worker 执行中" \
+  --job-id 1 \
+  --job-code "detect-simulated-cluster" \
+  --cycle-token "simcycle001" \
+  --active-threads 5 \
+  --available-proxy-count 12 \
+  --interval 15 >"$WORKER_LOG" 2>&1 &
+WORKER_PID=$!
+
+sleep 2
+
+echo "[3/4] runtime readiness"
+curl -fsS "${BASE_URL}/api/v1/runtime/readiness"
+echo
+echo
+
+echo "[4/4] runtime cluster summary"
+curl -fsS "${BASE_URL}/api/v1/runtime/cluster"
+echo
+echo
+echo "simulated nodes running:"
+echo "  controller log: $CONTROLLER_LOG"
+echo "  worker log: $WORKER_LOG"
+echo "press Ctrl+C to stop simulation and clean up"
+wait

domain-api/deploy/multi-region/templates/domaincheck-api.env.example

@@ -0,0 +1,30 @@
+WORKER_MODE=linux-systemd
+API_HOST=0.0.0.0
+API_PORT=8100
+DOMAIN_ROOT=__BASE_DIR__/domainCheck
+NODE_CODE=overseas-control-01
+NODE_REGION=overseas
+NODE_ROLE=control
+CORS_ORIGINS=http://127.0.0.1:3201,http://localhost:3201
+
+# DB
+# DB_HOST=127.0.0.1
+# DB_PORT=5432
+# DB_DATABASE=domain
+# DB_USER=postgres
+# DB_PASSWORD=postgres
+
+# REDIS
+# REDIS_HOST=127.0.0.1
+# REDIS_PORT=6379
+# REDIS_PASSWORD=
+# REDIS_DB=0
+
+# SYNC
+SYNC_PUSH_ENABLED=false
+SYNC_SOURCE_REGION=overseas
+SYNC_TARGET_REGION=overseas
+SYNC_TARGET_API_BASE_URL=
+SYNC_SHARED_TOKEN=
+SYNC_BATCH_SIZE=200
+SYNC_POLL_INTERVAL_SECONDS=30

domain-api/deploy/multi-region/templates/domaincheck-worker.controller.env.example

@@ -0,0 +1,27 @@
+WORKER_MODE=linux-systemd
+QT_QPA_PLATFORM=offscreen
+NODE_CODE=mainland-controller-01
+NODE_REGION=mainland
+NODE_ROLE=control
+
+# DB
+# DB_HOST=127.0.0.1
+# DB_PORT=5432
+# DB_DATABASE=domain
+# DB_USER=postgres
+# DB_PASSWORD=postgres
+
+# REDIS
+# REDIS_HOST=127.0.0.1
+# REDIS_PORT=6379
+# REDIS_PASSWORD=
+# REDIS_DB=0
+
+# SYNC
+SYNC_PUSH_ENABLED=true
+SYNC_SOURCE_REGION=mainland
+SYNC_TARGET_REGION=overseas
+SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
+SYNC_SHARED_TOKEN=
+SYNC_BATCH_SIZE=200
+SYNC_POLL_INTERVAL_SECONDS=30

domain-api/deploy/multi-region/templates/domaincheck-worker.worker.env.example

@@ -0,0 +1,27 @@
+WORKER_MODE=linux-systemd
+QT_QPA_PLATFORM=offscreen
+NODE_CODE=mainland-worker-01
+NODE_REGION=mainland
+NODE_ROLE=worker
+
+# DB
+# DB_HOST=127.0.0.1
+# DB_PORT=5432
+# DB_DATABASE=domain
+# DB_USER=postgres
+# DB_PASSWORD=postgres
+
+# REDIS
+# REDIS_HOST=127.0.0.1
+# REDIS_PORT=6379
+# REDIS_PASSWORD=
+# REDIS_DB=0
+
+# SYNC
+SYNC_PUSH_ENABLED=true
+SYNC_SOURCE_REGION=mainland
+SYNC_TARGET_REGION=overseas
+SYNC_TARGET_API_BASE_URL=http://海外控制面IP:8100/api/v1
+SYNC_SHARED_TOKEN=
+SYNC_BATCH_SIZE=200
+SYNC_POLL_INTERVAL_SECONDS=30