703 lines
16 KiB
Markdown
703 lines
16 KiB
Markdown
# domainCheck Ops Observability Contract
|
||
|
||
## 1. 目标
|
||
|
||
这份文档用于冻结海外单脑控制面里“看现场”这一层的正式 contract。
|
||
|
||
它不负责：
|
||
|
||
- 发布决策
|
||
- Driver 推荐
|
||
- Playbook 编排
|
||
|
||
它负责回答 3 类问题：
|
||
|
||
1. 哪些节点现在真正处于执行现场
|
||
2. 这些节点最近巡检收口结果是什么
|
||
3. Node Agent 回执队列现在是否健康
|
||
|
||
也就是说，这份 contract 的职责是：
|
||
|
||
> 让页面、CLI、Codex 看到同一份现场事实，而不是各自从日志和状态字串里猜
|
||
|
||
---
|
||
|
||
## 2. 入口接口
|
||
|
||
建议把下面这些接口视为同一观察面的一组正式入口：
|
||
|
||
- `GET /api/v1/ops/overview`
|
||
- `GET /api/v1/ops/inspection-overview`
|
||
- `GET /api/v1/ops/activity-stream`
|
||
- `GET /api/v1/ops/nodes/{node_code}/handover`
|
||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/flush`
|
||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/replay`
|
||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay`
|
||
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard`
|
||
|
||
配套关系：
|
||
|
||
- `overview`
|
||
- 首页收口
|
||
- `inspection-overview`
|
||
- 节点巡检收口
|
||
- `activity-stream`
|
||
- 最近异常活动
|
||
- `handover`
|
||
- 单节点接管阻断与下一步建议
|
||
- `delivery-queue`
|
||
- Node Agent 回执健康
|
||
|
||
正式要求：
|
||
|
||
- 页面不能自己再定义第 5 套“观察口径”
|
||
- CLI 不应重新从数据库拼现场
|
||
- Codex 不应通过自然语言日志推断核心状态
|
||
|
||
---
|
||
|
||
## 3. Execution Scene Contract
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/overview`
|
||
- `overview.execution_scene`
|
||
|
||
最小字段：
|
||
|
||
```json
|
||
{
|
||
"summary": "当前存在 2 台有效执行节点，其中 1 台正在领任务，1 台在线待命。",
|
||
"dispatch_active_nodes": [
|
||
{
|
||
"node_code": "mainland-worker-01",
|
||
"region": "mainland",
|
||
"role": "worker",
|
||
"status": "busy",
|
||
"participation_state": "dispatch_active",
|
||
"participation_label": "正在执行",
|
||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||
"current_load": 9,
|
||
"items_claimed": 9,
|
||
"items_running": 5,
|
||
"items_completed_recent": 14
|
||
}
|
||
],
|
||
"recent_only_nodes": [],
|
||
"standby_nodes": [
|
||
{
|
||
"node_code": "mainland-controller-01",
|
||
"region": "mainland",
|
||
"role": "control",
|
||
"status": "online",
|
||
"participation_state": "standby",
|
||
"participation_label": "在线待命",
|
||
"participation_reason": "节点在线、有效，但当前没有领任务、执行中任务或近窗吞吐",
|
||
"current_load": 0
|
||
}
|
||
],
|
||
"load_syncing_nodes": [],
|
||
"counts": {
|
||
"dispatch_active": 1,
|
||
"recent_only": 0,
|
||
"standby": 1,
|
||
"load_syncing": 0
|
||
}
|
||
}
|
||
```
|
||
|
||
分组语义必须固定：
|
||
|
||
- `dispatch_active_nodes`
|
||
- 当前正在领任务、执行任务、或仍有实时工作负载
|
||
- `recent_only_nodes`
|
||
- 当前不再运行，但近窗刚有吞吐
|
||
- `standby_nodes`
|
||
- 在线、有效，但当前不参与
|
||
- `load_syncing_nodes`
|
||
- 节点与控制面状态尚未完全同步
|
||
|
||
正式约束：
|
||
|
||
- “在线”不等于“参与检测”
|
||
- “有效执行节点”不等于“正在跑”
|
||
- 页面展示、CLI 摘要、Codex 判断必须沿用这 4 类分组
|
||
|
||
节点参与态附加字段也应固定：
|
||
|
||
- `participation_state`
|
||
- 机器判断主键，推荐值：
|
||
- `dispatch_active`
|
||
- `recent_only`
|
||
- `standby`
|
||
- `load_syncing`
|
||
- `participation_label`
|
||
- 直接给页面、CLI、Codex 展示的人类可读标签
|
||
- `participation_reason`
|
||
- 用一句稳定摘要解释为什么节点落在当前分组
|
||
|
||
正式要求：
|
||
|
||
- 页面必须直接区分：
|
||
- 在线但未参与
|
||
- 正在执行或领任务
|
||
- 近窗刚参与但当前已收口
|
||
- 新消费方不允许再从：
|
||
- `status`
|
||
- `current_load`
|
||
- `items_running`
|
||
手工推断“这个节点到底算不算正在参与”
|
||
|
||
---
|
||
|
||
## 4. Log Sync Runtime Contract
|
||
|
||
来源：
|
||
|
||
- `overview.execution_scene.log_sync`
|
||
- 或 `overview.log_sync`
|
||
|
||
最小字段：
|
||
|
||
```json
|
||
{
|
||
"status": "partial_coverage",
|
||
"status_label": "部分覆盖",
|
||
"mode": "key",
|
||
"mode_label": "关键回传",
|
||
"enabled": true,
|
||
"source_nodes": [
|
||
"mainland-worker-01"
|
||
],
|
||
"source_node_summaries": [
|
||
{
|
||
"node_code": "mainland-worker-01",
|
||
"line_count": 42,
|
||
"key_line_count": 42,
|
||
"full_line_count": 0,
|
||
"last_at": "2026-04-18 06:40:00",
|
||
"last_line": "域名检测完成: demo.com"
|
||
}
|
||
],
|
||
"covered_participating_node_summaries": [
|
||
{
|
||
"node_code": "mainland-worker-01",
|
||
"region": "mainland",
|
||
"role": "worker",
|
||
"participation_state": "dispatch_active",
|
||
"participation_label": "正在执行",
|
||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||
"line_count": 42,
|
||
"key_line_count": 42,
|
||
"full_line_count": 0,
|
||
"last_at": "2026-04-18 06:40:00",
|
||
"last_line": "域名检测完成: demo.com"
|
||
}
|
||
],
|
||
"missing_participating_node_summaries": [
|
||
{
|
||
"node_code": "mainland-worker-02",
|
||
"region": "mainland",
|
||
"role": "worker",
|
||
"participation_state": "dispatch_active",
|
||
"participation_label": "正在执行",
|
||
"participation_reason": "当前存在已领任务和执行中工作负载",
|
||
"missing_reason": "no_sample"
|
||
}
|
||
],
|
||
"covered_participating_nodes": 1,
|
||
"participating_nodes_total": 2,
|
||
"missing_sample_node_codes": [
|
||
"mainland-worker-02"
|
||
],
|
||
"last_line_at": "2026-04-18 06:40:00",
|
||
"summary": "当前已覆盖 1/2 台参与节点，建议继续观察未回传样本节点。"
|
||
}
|
||
```
|
||
|
||
推荐状态：
|
||
|
||
- `disabled`
|
||
- `waiting_sample`
|
||
- `partial_coverage`
|
||
- `healthy`
|
||
- `full_capture`
|
||
|
||
这层是“现场可见性”最关键的辅助状态。
|
||
|
||
正式要求：
|
||
|
||
- 如果存在参与节点但日志回传还没开，优先推荐开关键回传
|
||
- 如果日志回传已开但没有样本，优先推荐抓 Worker 日志
|
||
- 页面和 Codex 都不能再自己定义另一套日志覆盖结论
|
||
|
||
节点级日志覆盖字段也应固定：
|
||
|
||
- `source_nodes`
|
||
- 为兼容旧页面保留的节点编码列表
|
||
- `source_node_summaries`
|
||
- 当前已有日志样本的节点级统计对象
|
||
- `covered_participating_node_summaries`
|
||
- 已参与且已拿到样本的节点对象
|
||
- `missing_participating_node_summaries`
|
||
- 已参与但尚未拿到样本的节点对象
|
||
|
||
`source_node_summaries[]` 至少应返回：
|
||
|
||
- `node_code`
|
||
- `line_count`
|
||
- `key_line_count`
|
||
- `full_line_count`
|
||
- `last_at`
|
||
- `last_line`
|
||
|
||
`missing_participating_node_summaries[]` 至少应返回：
|
||
|
||
- `node_code`
|
||
- `region`
|
||
- `role`
|
||
- `participation_state`
|
||
- `participation_label`
|
||
- `participation_reason`
|
||
- `missing_reason`
|
||
|
||
推荐 `missing_reason`：
|
||
|
||
- `no_sample`
|
||
- `delayed_flush`
|
||
- `filtered_out`
|
||
|
||
兼容要求：
|
||
|
||
- 旧页面仍可继续消费：
|
||
- `source_nodes`
|
||
- `missing_sample_node_codes`
|
||
- 新页面、CLI、Codex 应优先消费：
|
||
- `source_node_summaries`
|
||
- `covered_participating_node_summaries`
|
||
- `missing_participating_node_summaries`
|
||
|
||
补充节点级现场入口：
|
||
|
||
- `GET /api/v1/ops/nodes/{node_code}/scene-log`
|
||
|
||
最小字段：
|
||
|
||
```json
|
||
{
|
||
"node_code": "mainland-worker-01",
|
||
"status": "healthy",
|
||
"status_label": "关键覆盖",
|
||
"mode": "key",
|
||
"mode_label": "关键回传",
|
||
"log_sync_enabled": true,
|
||
"summary": "节点当前正在参与检测，已保留 42 条现场日志样本。",
|
||
"node": {
|
||
"region": "mainland",
|
||
"role": "worker",
|
||
"status": "busy",
|
||
"current_load": 6,
|
||
"last_heartbeat_at": "2026-04-18 06:40:00"
|
||
},
|
||
"participation": {
|
||
"detect_participating": true,
|
||
"participation_state": "claimed",
|
||
"participation_label": "已领待跑",
|
||
"participation_bucket": "dispatch_active",
|
||
"participation_reason": "已领取任务，等待线程继续执行。"
|
||
},
|
||
"source_summary": {
|
||
"line_count": 42,
|
||
"key_line_count": 42,
|
||
"full_line_count": 0,
|
||
"last_at": "2026-04-18 06:40:00",
|
||
"last_line": "[2026-04-18 06:40:00] [mainland-worker-01] 域名检测完成: demo.com"
|
||
},
|
||
"records_total": 42,
|
||
"records_visible": 80,
|
||
"records": [
|
||
{
|
||
"created_at": "2026-04-18 06:38:00",
|
||
"node_code": "mainland-worker-01",
|
||
"message": "开始检测域名: demo.com",
|
||
"mode": "key",
|
||
"line": "[2026-04-18 06:38:00] [mainland-worker-01] 开始检测域名: demo.com"
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
节点级入口的正式语义：
|
||
|
||
- 这是“看某一台节点当前现场”的正式入口，不允许页面再自己从聚合样本中二次猜测
|
||
- `status` 至少覆盖：
|
||
- `healthy`
|
||
- `full_capture`
|
||
- `historical_sample`
|
||
- `waiting_sample`
|
||
- `missing_sample`
|
||
- `disabled`
|
||
- `standby`
|
||
- `unknown`
|
||
- 当节点当前在线但未参与时，必须明确返回 `standby`
|
||
- 当节点正在参与但还没有样本时，必须明确返回 `waiting_sample`
|
||
- 当节点已有历史样本但当前不在参与面中时，必须明确返回 `historical_sample`
|
||
|
||
---
|
||
|
||
## 5. Inspection Overview Contract
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/inspection-overview`
|
||
- `overview.inspection`
|
||
|
||
最小字段：
|
||
|
||
```json
|
||
{
|
||
"status": "attention",
|
||
"status_label": "待处理",
|
||
"summary": "存在 1 台节点最近缺少 Worker 日志收口。",
|
||
"counts": {
|
||
"healthy_nodes": 1,
|
||
"attention_nodes": 1,
|
||
"problem_nodes": 0
|
||
},
|
||
"rows": [
|
||
{
|
||
"node_code": "mainland-worker-01",
|
||
"problem_kind": "missing_worker_logs",
|
||
"problem_label": "缺少 Worker 日志收口",
|
||
"problem_level": "warning",
|
||
"recommended_action_code": "open_worker_logs",
|
||
"ui_intent": {
|
||
"kind": "node_inspection_detail",
|
||
"node_code": "mainland-worker-01"
|
||
},
|
||
"latest_health_snapshot": {},
|
||
"latest_worker_logs": {},
|
||
"latest_diagnostics": {}
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
设计要求：
|
||
|
||
- 巡检不是几条离散 job 记录
|
||
- 必须按节点重新聚合
|
||
- 必须直接给出：
|
||
- `problem_kind`
|
||
- `problem_label`
|
||
- `problem_level`
|
||
- `recommended_action_code`
|
||
- `ui_intent`
|
||
|
||
这样页面、CLI、Codex 才能真正消费“节点巡检结论”，而不是再从几十条 job 记录里回推问题。
|
||
|
||
---
|
||
|
||
## 6. Inspection Row Contract
|
||
|
||
`rows[]` 中每个节点建议最少包含：
|
||
|
||
- `node_code`
|
||
- `region`
|
||
- `role`
|
||
- `effective_worker`
|
||
- `problem_kind`
|
||
- `problem_label`
|
||
- `problem_level`
|
||
- `recommended_action_code`
|
||
- `ui_intent`
|
||
- `latest_health_snapshot`
|
||
- `latest_worker_logs`
|
||
- `latest_diagnostics`
|
||
|
||
其中 3 类最近结果建议统一结构：
|
||
|
||
```json
|
||
{
|
||
"status": "success",
|
||
"status_label": "成功",
|
||
"occurred_at": "2026-04-18 06:35:00",
|
||
"job_code": "ops-20260418-001",
|
||
"summary": "最近一次 Worker 日志收集成功，共采样 120 行。"
|
||
}
|
||
```
|
||
|
||
正式边界：
|
||
|
||
- `logs.collect` 默认指 Worker 日志收口
|
||
- Node Agent 辅助排障日志不应混成同一种巡检结果
|
||
|
||
---
|
||
|
||
## 7. Delivery Queue Summary Contract
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
|
||
|
||
最小字段：
|
||
|
||
```json
|
||
{
|
||
"node": {
|
||
"node_code": "mainland-worker-01",
|
||
"title": "mainland-worker-01"
|
||
},
|
||
"summary": {
|
||
"state": "dead_letter",
|
||
"label": "死信",
|
||
"reason": "当前有 2 条记录进入死信。",
|
||
"pending_count": 3,
|
||
"dead_letter_count": 2,
|
||
"oldest_pending_at": "2026-04-18 06:20:00",
|
||
"oldest_dead_letter_at": "2026-04-18 06:18:00"
|
||
},
|
||
"record_visibility": "head_only",
|
||
"capabilities": {
|
||
"can_flush": true,
|
||
"can_replay": true,
|
||
"can_discard": true
|
||
}
|
||
}
|
||
```
|
||
|
||
推荐状态：
|
||
|
||
- `healthy`
|
||
- `retrying`
|
||
- `dead_letter`
|
||
- `stalled`
|
||
|
||
正式约束：
|
||
|
||
- 控制面要明确告诉操作者当前是 `head_only` 还是 `full_projection`
|
||
- 不允许页面假装能列出远端完整目录
|
||
- 不允许因为能看见死信就绕过 `ops job` 直接改文件
|
||
|
||
---
|
||
|
||
## 8. Delivery Queue Record Contract
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
|
||
|
||
当前阶段建议最小字段：
|
||
|
||
```json
|
||
{
|
||
"records": [
|
||
{
|
||
"record_id": 101,
|
||
"state": "dead_letter",
|
||
"state_label": "死信",
|
||
"request_id": "req-20260418-001",
|
||
"kind": "job_complete",
|
||
"created_at": "2026-04-18 06:18:00",
|
||
"updated_at": "2026-04-18 06:19:00",
|
||
"summary": "某次任务完成回执多次补发失败。",
|
||
"visibility": "head_only"
|
||
}
|
||
],
|
||
"summary": {
|
||
"returned_records": 1,
|
||
"record_visibility": "head_only"
|
||
}
|
||
}
|
||
```
|
||
|
||
当前阶段要求：
|
||
|
||
- 单条 `replay / discard` 只针对控制面可见记录
|
||
- 当前 `head_only` 不等于未来不能扩展
|
||
- 以后若进入 `full_projection`，扩的是可见范围，不是重写操作模型
|
||
|
||
---
|
||
|
||
## 9. Delivery Queue Action Contract
|
||
|
||
来源：
|
||
|
||
- `flush`
|
||
- `replay`
|
||
- `record replay`
|
||
- `record discard`
|
||
|
||
正式要求：
|
||
|
||
- 所有动作都必须落成正式 `ops job`
|
||
- 返回体要能告诉调用方：
|
||
- 是否已受理
|
||
- 生成了什么 job
|
||
- 建议跳转到哪里观察
|
||
|
||
最小返回体建议：
|
||
|
||
```json
|
||
{
|
||
"accepted": true,
|
||
"job_code": "ops-20260418-queue-001",
|
||
"ui_intent": {
|
||
"kind": "job_events",
|
||
"job_code": "ops-20260418-queue-001"
|
||
}
|
||
}
|
||
```
|
||
|
||
这里的边界必须比“临时脚本”更严格：
|
||
|
||
- `discard` 必须带原因
|
||
- `replay` 应支持限流 / limit
|
||
- `flush` 应支持显式 `requested_by`
|
||
|
||
---
|
||
|
||
## 10. Activity Stream Contract
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/activity-stream`
|
||
|
||
最小字段建议：
|
||
|
||
```json
|
||
{
|
||
"items": [
|
||
{
|
||
"kind": "ops_job",
|
||
"activity_key": "ops-job:101",
|
||
"title": "logs.collect",
|
||
"subtitle": "ops-20260418064500-a1b2c3",
|
||
"summary": "目标节点 mainland-worker-01 / 发起 web-ui / 方式 Node Agent",
|
||
"summary_text": "目标节点 mainland-worker-01 / 发起 web-ui / 方式 Node Agent",
|
||
"status": "running",
|
||
"status_label": "执行中",
|
||
"execution_mode": "remote-agent",
|
||
"execution_mode_label": "Node Agent",
|
||
"target_node_codes": [
|
||
"mainland-worker-01"
|
||
],
|
||
"occurred_at": "2026-04-18 06:45:08",
|
||
"ui_intent_kind": "job_events",
|
||
"focus_ref": {
|
||
"kind": "ops_job",
|
||
"job_id": 101,
|
||
"job_code": "ops-20260418064500-a1b2c3",
|
||
"action": "logs.collect",
|
||
"target_node_code": "mainland-worker-01"
|
||
}
|
||
}
|
||
],
|
||
"summary": {
|
||
"status": "running",
|
||
"status_label": "执行中",
|
||
"summary_text": "最近活动里存在正在推进的现场、任务、编排或回传。",
|
||
"filtered_total": 6,
|
||
"unfiltered_total": 18,
|
||
"latest_at": "2026-04-18 06:45:08"
|
||
}
|
||
}
|
||
```
|
||
|
||
正式要求：
|
||
|
||
- `activity-stream` 不是页面内部 table 数据
|
||
- 它必须是：
|
||
- 页面
|
||
- CLI
|
||
- Codex 驾驶员
|
||
- 自动驾驶编排器
|
||
共同消费的“最近活动摘要”
|
||
|
||
字段语义需要固定：
|
||
|
||
- `summary_text`
|
||
- 是直接消费的活动短摘要
|
||
- `status_label`
|
||
- 是稳定的人类可读状态标签
|
||
- `target_node_codes`
|
||
- 是活动关联的节点口径，不允许调用方再从自然语言里猜
|
||
- `ui_intent_kind`
|
||
- 是当前活动默认落点类型
|
||
- `focus_ref`
|
||
- 是跨页面 / CLI / Codex 的统一定位对象
|
||
|
||
正式边界：
|
||
|
||
- 活动流应保留老字段以兼容现有页面
|
||
- 新消费方优先用：
|
||
- `summary_text`
|
||
- `status_label`
|
||
- `focus_ref`
|
||
- 不允许新客户端再从 `title + subtitle + summary` 手工解析定位参数
|
||
|
||
## 11. 与 Driver / Playbook / Release 的边界关系
|
||
|
||
这份 contract 只负责观察面和现场治理，不负责替代其他 contract。
|
||
|
||
边界应明确为：
|
||
|
||
- `ops_driver_contract`
|
||
- 决定“现在优先做什么”
|
||
- `ops_playbook_contract`
|
||
- 决定“标准多步动作怎么编排”
|
||
- `release_hub_contract`
|
||
- 决定“版本怎么发、怎么停、怎么回滚”
|
||
- `ops_observability_contract`
|
||
- 决定“现场事实是什么、当前缺口在哪里”
|
||
|
||
这样后续系统再扩展时，不会又回到“所有东西都写在 overview 一坨大 json 里”的状态。
|
||
|
||
---
|
||
|
||
## 12. 终局验收标准
|
||
|
||
如果这份 contract 真正落地，海外单脑控制面应该做到：
|
||
|
||
### 1. 不登录大陆机器也能回答现场问题
|
||
|
||
例如：
|
||
|
||
- 谁在跑
|
||
- 谁在线但没跑
|
||
- 谁近窗刚跑过
|
||
- 谁日志没回来
|
||
- 谁队列死信了
|
||
|
||
### 2. 不复制大日志也能先做判断
|
||
|
||
先看：
|
||
|
||
- execution scene
|
||
- inspection overview
|
||
- activity stream
|
||
- delivery queue
|
||
|
||
再决定是否需要全量日志或 diagnostics bundle。
|
||
|
||
### 3. 页面、CLI、Codex 看到的是同一现场
|
||
|
||
不能出现：
|
||
|
||
- 页面说节点正在执行
|
||
- CLI 说节点待命
|
||
- Codex 说节点离线
|
||
|
||
### 4. 所有高风险动作都有正式观察落点
|
||
|
||
也就是：
|
||
|
||
- replay 后能看 job
|
||
- discard 后能看审计
|
||
- inspection 后能看节点收口
|
||
- rollout 后能看 launchpad / gate / rollout jobs
|
||
|
||
这样这套系统才能真正从“调试台”升级成“驾驶舱”。
|