Files
getDomain/docs/schemas/ops_agent_protocol.md
2026-04-18 23:52:51 +08:00

817 lines
17 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# domainCheck Ops Agent Protocol
## 1. 目标
这份文档用于冻结 `Node Agent <-> Overseas Control Plane` 的正式 contract。
适用范围:
- `POST /api/v1/ops/agent/tokens`
- `POST /api/v1/ops/agent/bootstrap-plan`
- `GET /api/v1/ops/nodes/{node_code}/handover`
- `POST /api/v1/ops/nodes/{node_code}/handover/bootstrap-plan`
- `POST /api/v1/ops/agent/register`
- `POST /api/v1/ops/agent/heartbeat`
- `POST /api/v1/ops/agent/pull`
- `POST /api/v1/ops/agent/jobs/{job_id}/start`
- `POST /api/v1/ops/agent/jobs/{job_id}/complete`
- `POST /api/v1/ops/agent/jobs/{job_id}/events`
设计原则:
- Agent 只执行声明过的结构化动作
- 控制面负责调度、门禁、编排和审计
- 所有回执都必须可重放、可去重、可死信化
---
## 2. 公共响应包裹
所有接口统一返回:
```json
{
"code": 0,
"message": "ok",
"detail_code": "",
"data": {}
}
```
约束:
- `code=0` 表示成功
- `code=1` 表示业务失败
- `detail_code` 用于结构化错误语义
- `message` 只承担可读说明,不承担机器判断主逻辑
常见 `detail_code`
- `agent_token_invalid`
- `agent_token_expired`
- `agent_token_node_mismatch`
- `ops_job_not_owned_by_agent`
- `ops_job_invalid_status_transition`
- `release_checksum_mismatch`
- `release_health_check_failed`
---
## 3. 认证与请求头
Agent 相关接口统一使用:
- `Content-Type: application/json`
- `X-Domaincheck-Agent-Token: <token>`
其中:
- `tokens`
- `bootstrap-plan`
不要求 Agent Token。
其余接口都要求 Token。
---
## 4. 公共节点载荷
`register``heartbeat` 当前共享 `_base_payload()`
最小字段:
```json
{
"node_code": "mainland-worker-02",
"region": "mainland",
"role": "worker",
"title": "mainland-worker-02",
"hostname": "host-a",
"ip": "10.0.0.12",
"agent_version": "0.1.0",
"capabilities": [
"service.start",
"service.stop",
"service.restart",
"service.status",
"health.snapshot",
"logs.collect",
"diagnostics.collect",
"deploy.release"
],
"labels": {},
"metadata": {
"service_names": {
"api": "domaincheck-api",
"worker": "domaincheck-worker",
"sync_agent": "domaincheck-sync-agent",
"node_agent": "domaincheck-node-agent"
},
"delivery_queue": {
"state": "healthy",
"label": "正常",
"reason": "当前没有待重试回执,也没有死信记录。",
"pending_count": 0,
"dead_letter_count": 0,
"last_flush_at": "",
"oldest_pending_at": "",
"oldest_pending_request_id": "",
"oldest_pending_kind": "",
"oldest_dead_letter_at": "",
"oldest_dead_letter_request_id": "",
"oldest_dead_letter_kind": ""
}
}
}
```
说明:
- `service_names` 是控制面识别节点服务拓扑的最小入口
- `delivery_queue` 是控制面识别 Agent 回执现场的最小入口
---
## 5. Bootstrap Contract
### 5.1 Issue Token
`POST /api/v1/ops/agent/tokens`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02",
"issued_by": "web-ui",
"expires_in_hours": 72,
"metadata": {
"source": "ops-center"
}
}
```
返回体关键字段:
- `token`
- `token_preview`
- `record_id`
- `node_code`
- `expires_at`
- `created_at`
### 5.2 Bootstrap Plan
`POST /api/v1/ops/agent/bootstrap-plan`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02",
"node_region": "mainland",
"node_role": "worker",
"issued_by": "web-ui",
"expires_in_hours": 72,
"control_plane_base_url": "https://ops.example.com",
"root_dir": "/opt/domaincheck",
"metadata": {
"source": "ops-center"
}
}
```
返回体关键字段:
- Token 相关字段
- `control_plane_base_url`
- `bootstrap_plan.node_code`
- `bootstrap_plan.node_region`
- `bootstrap_plan.node_role`
- `bootstrap_plan.root_dir`
- `bootstrap_plan.env_file`
- `bootstrap_plan.service_name`
- `bootstrap_plan.service_file`
- `bootstrap_plan.install_script_path`
- `bootstrap_plan.env_content`
- `bootstrap_plan.command_lines`
- `bootstrap_plan.command_block`
- `bootstrap_plan.health_checks`
- `bootstrap_plan.health_check_block`
- `bootstrap_plan.bootstrap_script_name`
- `bootstrap_plan.bootstrap_script_path`
- `bootstrap_plan.bootstrap_script_content`
- `bootstrap_plan.bootstrap_run_script_block`
约束:
- `bootstrap_plan` 是标准接入方案对象
- 页面、Codex、CLI 必须复用这一个对象
- 不允许各端再次手写 env 或 shell 逻辑
---
## 6. Runtime Contract
### 6.1 Register
`POST /api/v1/ops/agent/register`
语义:
- 校验 Token 与 `node_code`
- Upsert Agent 运行态
- 将节点推入托管目录候选态
最小成功返回字段:
- `node_code`
- `expires_at`
- `capabilities`
### 6.2 Heartbeat
`POST /api/v1/ops/agent/heartbeat`
语义:
- 刷新 `last_seen_at`
- 刷新节点身份、服务名和队列快照
- 维持控制面中的 Agent 在线态
最小成功返回字段:
- `node_code`
- `server_time`
- `expires_at`
### 6.3 Pull
`POST /api/v1/ops/agent/pull?limit=1`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02"
}
```
返回体关键字段:
```json
{
"jobs": [
{
"id": 123,
"job_code": "ops-xxx",
"action": "health.snapshot",
"target_node_code": "mainland-worker-02",
"status": "dispatching",
"execution_mode": "remote-agent",
"payload": {},
"metadata": {},
"policy": {},
"steps": []
}
],
"count": 1
}
```
约束:
- Agent 只能拿到属于自己的任务
- Agent 不负责选择任务
- 调度权始终在控制面
### 6.3.1 Agent Job Envelope
为了避免 Agent 再从自然语言动作名里“猜执行上下文”,`pull.data.jobs[]` 当前已经按统一任务包裹返回。
当前 `pull.data` 顶层也会额外带:
- `protocol_version`
- `envelope_type`
- `node_code`
- `limit`
- `count`
推荐最小结构:
```json
{
"protocol_version": "ops-agent/v1",
"envelope_type": "agent_job",
"id": 123,
"job_id": 123,
"job_code": "ops-20260418-001",
"job_type": "ops_action",
"action": "logs.collect",
"target_node_code": "mainland-worker-02",
"status": "dispatching",
"execution_mode": "remote-agent",
"step_key": "worker_logs",
"step_title": "收集 Worker 日志",
"job_ref": {
"job_id": 123,
"job_code": "ops-20260418-001",
"action": "logs.collect",
"target_node_code": "mainland-worker-02"
},
"step_ref": {
"step_id": 456,
"step_key": "worker_logs",
"step_title": "收集 Worker 日志"
},
"payload": {
"service_name": "domaincheck-worker",
"lines": 120,
"include_agent_logs": false
},
"policy": {
"timeout_seconds": 120,
"stop_on_failure": true,
"auto_approve": true
},
"metadata": {
"requested_by": "playbook:inspection.standard",
"source": "ops-center"
},
"release_context": {
"release_id": 0,
"release_version": "",
"rollout_id": 0,
"rollout_code": ""
},
"focus_ref": {
"kind": "ops_job",
"job_id": 123,
"job_code": "ops-20260418-001",
"action": "logs.collect",
"target_node_code": "mainland-worker-02"
}
}
```
正式要求:
- Agent 必须优先消费:
- `action`
- `payload`
- `policy`
- Agent 不应通过:
- `step_title`
- `summary`
- `notes`
去反推真实执行参数
- 与发布相关的任务应通过:
- `release_context`
传递 release / rollout 关联,而不是让 Agent 自己查询“当前版本”
这层的意义是:
- `ops job`
- 仍是正式执行颗粒度
- `Agent Job Envelope`
- 是 Node Agent 拿到的稳定执行视图
这样同一套 Agent 才能同时承接:
- 标准巡检
- 日志收集
- 诊断采样
- Release 部署
- Rollout 回滚
### 6.4 Start
`POST /api/v1/ops/agent/jobs/{job_id}/start`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02"
}
```
语义:
- `dispatching` 表示已派发
- `running` 表示节点已真实开工
### 6.5 Complete
`POST /api/v1/ops/agent/jobs/{job_id}/complete`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02",
"client_request_id": "complete-ops-123-1",
"status": "success",
"stdout": "",
"stderr": "",
"result": {
"summary": "ok"
},
"error_message": ""
}
```
允许状态:
- `success`
- `failed`
- `partially_succeeded`
幂等约束:
- 使用 `client_request_id`
- 控制面写入 `ops_jobs.last_agent_complete_request_id`
- 同一完成回执可安全重放,不应再次制造副作用
推荐补充字段:
- `duration_ms`
- `result.summary_text`
- `result.focus_ref`
当前成功返回里也会额外带:
- `completion_summary.job_id`
- `completion_summary.job_code`
- `completion_summary.status`
- `completion_summary.status_label`
- `completion_summary.result_summary_text`
- `completion_summary.focus_ref`
- `completion_summary.deduplicated`
这样控制面在不展开 stdout/stderr 的情况下,也能直接把:
- 任务收口摘要
- 后续跳转落点
并入 activity / inspection / rollout 观察面。
### 6.6 Events
`POST /api/v1/ops/agent/jobs/{job_id}/events`
请求体最小字段:
```json
{
"node_code": "mainland-worker-02",
"client_event_id": "event-ops-123-checksum-1",
"step_id": 0,
"event_type": "deploy_checksum_verified",
"level": "info",
"message": "checksum ok",
"payload": {
"checksum": "sha256:..."
}
}
```
去重约束:
- 使用 `client_event_id`
- 控制面通过唯一索引去重
- 同一事件可安全重放,不应重复落库
推荐补充字段:
- `occurred_at`
- `summary_text`
- `focus_ref`
当前成功返回里也会额外带:
- `event`
- 即标准化后的事件对象
- 内含 `event_key / summary_text / occurred_at / focus_ref`
这样事件流就不再只是“原始日志点”,而能直接成为:
- playbook run events
- rollout events
- activity-stream
共同消费的现场片段。
---
## 7. Agent 本地补发队列
当前主链路已经正式具备:
- `pending` 队列
- `dead-letter` 队列
- 定时 `_flush_delivery_queue()`
- 永久性错误转死信
当前重点覆盖:
- `jobs/{id}/complete`
- `jobs/{id}/events`
队列状态:
- `healthy`
- `retrying`
- `dead_letter`
语义:
- `healthy`:无积压,无死信
- `retrying`存在待补发回执Agent 会继续自动重放
- `dead_letter`:存在永久失败记录,需人工介入
控制面消费方式:
- 节点维度:
- `delivery_queue_state`
- `delivery_queue_label`
- `delivery_queue_reason`
- `delivery_queue_pending_count`
- `delivery_queue_dead_letter_count`
- 汇总维度:
- `queue_retrying_nodes`
- `queue_dead_letter_nodes`
- `queue_pending_records`
- `queue_dead_letter_records`
### 7.1 当前控制面已可见的最小现场
当前至少已经可以通过:
- `GET /api/v1/ops/nodes`
看到每台托管节点的队列现场:
```json
{
"node_code": "mainland-worker-02",
"delivery_queue_state": "dead_letter",
"delivery_queue_label": "死信 2",
"delivery_queue_reason": "当前存在 2 条死信记录,建议优先查看节点日志或诊断编排。",
"delivery_queue_pending_count": 0,
"delivery_queue_dead_letter_count": 2,
"delivery_queue_last_flush_at": "2026-04-18 06:10:00",
"delivery_queue_oldest_pending_at": "",
"delivery_queue_oldest_pending_request_id": "",
"delivery_queue_oldest_pending_kind": "",
"delivery_queue_oldest_dead_letter_at": "2026-04-18 05:59:00",
"delivery_queue_oldest_dead_letter_request_id": "complete-ops-123-1",
"delivery_queue_oldest_dead_letter_kind": "job_complete"
}
```
这已经足够让:
- 页面显示“节点存在死信”
- `overview / driver-feed / codex-brief` 产出驾驶建议
- CLI 快速判断现场是不是要先走日志或诊断
但这还不够支撑正式运维,因为它只能“看见死信”,还不能“处理死信”。
### 7.2 Delivery Queue Record 正式对象
后续控制面不应再把死信理解成一个纯计数器,而要把每条待补发 / 死信记录升级成正式对象。
推荐最小结构:
```json
{
"record_id": "dq-mainland-worker-02-20260418-001",
"node_code": "mainland-worker-02",
"state": "dead_letter",
"request_kind": "job_complete",
"client_request_id": "complete-ops-123-1",
"job_id": 123,
"job_code": "ops-20260418-001",
"target_api": "/api/v1/ops/agent/jobs/123/complete",
"detail_code": "ops_job_invalid_status_transition",
"error_message": "当前任务状态不允许 complete",
"attempt_count": 6,
"first_queued_at": "2026-04-18 05:58:00",
"last_attempt_at": "2026-04-18 05:59:00",
"next_retry_at": "",
"payload_preview": {
"status": "success"
},
"response_preview": {
"code": 1,
"detail_code": "ops_job_invalid_status_transition"
}
}
```
正式约束:
- `record_id`
- 是控制面侧唯一主键
- `client_request_id`
- 是 Agent 幂等键
- `detail_code`
- 是死信聚类主键,不能只靠 `message`
- `payload_preview / response_preview`
- 用于页面和 Codex 快速判断,不必默认展开完整原始报文
推荐状态:
- `pending`
- `retrying`
- `dead_letter`
- `replayed`
- `discarded`
### 7.3 死信操作面正式入口
为了让“看见死信”之后不用回节点本地处理,推荐把操作面固定成下面这组接口。
当前已落地的第一阶段能力:
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
当前控制面返回的记录可见性为:
- `record_visibility=head_only`
也就是:
- 当前已经能稳定查看节点级队列总览
- 也能看到 `pending / dead_letter` 的头部记录摘要
- 但还没有把节点本地全量 `pending/*.json / dead-letter/*.json` 正式同步到控制面
所以这两条 GET 入口现在属于“正式只读观察面”,而不是完整操作面。
#### a. 单节点队列总览
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue`
最小返回体:
```json
{
"node_code": "mainland-worker-02",
"summary": {
"state": "dead_letter",
"pending_count": 0,
"dead_letter_count": 2,
"last_flush_at": "2026-04-18 06:10:00"
},
"oldest_pending_record": {},
"oldest_dead_letter_record": {}
}
```
#### b. 队列记录列表
- `GET /api/v1/ops/nodes/{node_code}/delivery-queue/records`
推荐筛选参数:
- `state=pending|retrying|dead_letter|replayed|discarded`
- `request_kind=job_complete|job_event`
- `detail_code=...`
- `group_by=detail_code|request_kind|target_api`
- `limit=50`
当带 `group_by` 时,返回值应优先给出聚类摘要,而不是只返回平铺明细。
#### c. 单条死信重放
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/replay`
当前已落地,但要明确当前阶段约束:
- 控制面仍然是 `record_visibility=head_only`
- 所以单条动作目前只允许针对“当前可见头部记录”
- 真正的执行不是控制面直接改远端文件,而是创建正式 `ops job`
- 由目标节点 `Node Agent` 执行 `delivery.queue.replay`
最小请求体:
```json
{
"requested_by": "web-ui",
"reason": "确认任务状态已修复,重放这条 complete 回执"
}
```
#### d. 批量重放
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/replay`
当前已落地为正式操作面。
推荐请求体:
```json
{
"requested_by": "codex",
"selector": {
"state": "dead_letter",
"detail_code": "ops_job_invalid_status_transition"
},
"limit": 20
}
```
#### e. 丢弃死信
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/records/{record_id}/discard`
当前已落地,但与单条重放一样,仍然只允许针对当前可见头部死信记录发起单条动作。
最小请求体:
```json
{
"discarded_by": "web-ui",
"reason": "确认这条回执不再需要补发"
}
```
#### f. 主动冲刷队列
- `POST /api/v1/ops/nodes/{node_code}/delivery-queue/flush`
当前已落地为正式操作面,执行方式同样是:
- 控制面创建 `ops job`
- 目标节点 `Node Agent` 执行 `delivery.queue.flush`
语义:
- 不是“强行成功”
- 而是立即触发一次 Agent 补发冲刷
- 结果仍然回到 `pending / retrying / dead_letter / replayed`
### 7.4 为什么必须做成正式操作面
这层不是为了让页面多一个按钮,而是为了保证:
- 页面可以处理死信
- CLI 可以处理死信
- 海外 Codex 驾驶员可以处理死信
三者都不需要绕回:
- 手工 SSH 登录节点
- 手工删本地文件
- 手工重放某条 HTTP 请求
正式规则应该固定成:
- “死信查看” 走控制面对象
- “死信重放 / 丢弃 / 冲刷” 走控制面对象
- Agent 只负责执行,不负责决定如何处理死信
---
## 8. Job Status Enum
当前 Agent 主链路约束:
- `queued`
- `dispatching`
- `running`
- `success`
- `failed`
- `partially_succeeded`
语义:
- `queued`:任务已创建,尚未派发
- `dispatching`:控制面已派发,但节点尚未确认开工
- `running`:节点已真实执行
- `success`:成功完成
- `failed`:失败完成
- `partially_succeeded`:部分成功,需关注
---
## 9. 实现边界
Agent 只执行已声明动作:
- `service.start`
- `service.stop`
- `service.restart`
- `service.status`
- `health.snapshot`
- `logs.collect`
- `diagnostics.collect`
- `deploy.release`
不允许控制面长期依赖任意 shell 下发。
`shell executor` 只能作为受限兜底能力存在,并必须挂审批与审计。