741 lines
14 KiB
Markdown
741 lines
14 KiB
Markdown
# domainCheck Ops Job Contract
|
||
|
||
## 1. 目标
|
||
|
||
这份文档用于冻结 `ops job` 的正式 contract。
|
||
|
||
它是整套海外单脑控制面里最核心的执行对象。
|
||
|
||
正式定义：
|
||
|
||
> `ops job` 是一条可审计、可编排、可派发、可回执、可取消的标准运维任务记录。
|
||
|
||
它不是：
|
||
|
||
- 一条 shell 命令
|
||
- 一个页面按钮点击结果
|
||
- 一个仅供 agent 消费的内部对象
|
||
|
||
它应该同时服务于：
|
||
|
||
- 页面按钮
|
||
- CLI
|
||
- 海外 Codex 驾驶员
|
||
- Node Agent
|
||
- Release / Rollout
|
||
- Playbook Run
|
||
|
||
也就是说：
|
||
|
||
- `playbook run`
|
||
- 是编排聚合对象
|
||
- `rollout`
|
||
- 是发布推进对象
|
||
- `ops job`
|
||
- 是真正的执行颗粒度对象
|
||
|
||
---
|
||
|
||
## 2. 入口接口
|
||
|
||
当前已落地的主入口：
|
||
|
||
- `GET /api/v1/ops/jobs`
|
||
- `GET /api/v1/ops/jobs/{job_id}`
|
||
- `GET /api/v1/ops/jobs/{job_id}/events`
|
||
- `POST /api/v1/ops/jobs`
|
||
- `POST /api/v1/ops/jobs/batch`
|
||
- `POST /api/v1/ops/policy/preview`
|
||
- `POST /api/v1/ops/jobs/{job_id}/approve`
|
||
- `POST /api/v1/ops/jobs/{job_id}/cancel`
|
||
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
|
||
|
||
配套对象来源：
|
||
|
||
- `ops_jobs`
|
||
- `ops_job_steps`
|
||
- `ops_job_events`
|
||
|
||
正式约束：
|
||
|
||
- 页面、CLI、Codex 不能各自定义第 2 套任务模型
|
||
- `playbook run` 只能聚合 `ops job`
|
||
- `rollout` 只能批量生成 `ops job`
|
||
- `Node Agent` 只能拉取和回执 `ops job`
|
||
|
||
---
|
||
|
||
## 3. 当前实现里的 Job Object
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/jobs`
|
||
- `GET /api/v1/ops/jobs/{job_id}`
|
||
- 创建 / 审批 / 取消 / 派发后的返回体中的 `job`
|
||
|
||
最小字段：
|
||
|
||
```json
|
||
{
|
||
"id": 101,
|
||
"job_code": "ops-20260418064500-a1b2c3",
|
||
"action": "logs.collect",
|
||
"target_type": "node",
|
||
"target_node_code": "mainland-worker-01",
|
||
"target_node_codes": [
|
||
"mainland-worker-01"
|
||
],
|
||
"status": "queued",
|
||
"status_label": "排队中",
|
||
"execution_mode": "remote-agent",
|
||
"execution_mode_label": "Node Agent",
|
||
"requested_by": "web-ui/ops-center",
|
||
"payload": {},
|
||
"metadata": {},
|
||
"result": {},
|
||
"summary": "目标节点 mainland-worker-01 / 发起 web-ui/ops-center / 方式 Node Agent",
|
||
"summary_text": "目标节点 mainland-worker-01 / 发起 web-ui/ops-center / 方式 Node Agent",
|
||
"error_message": "",
|
||
"created_at": "2026-04-18 06:45:00",
|
||
"started_at": "",
|
||
"finished_at": "",
|
||
"updated_at": "2026-04-18 06:45:00",
|
||
"risk_level": "medium",
|
||
"approval_required": false,
|
||
"approval_status": "approved",
|
||
"approval_status_label": "已审批",
|
||
"approved_by": "",
|
||
"approved_at": "",
|
||
"blocked_reason": "",
|
||
"cancellation_reason": "",
|
||
"dispatched_at": "",
|
||
"target_selector": {},
|
||
"policy": {},
|
||
"rollout_id": 0,
|
||
"steps_total": 0,
|
||
"steps_running": 0,
|
||
"steps_success": 0,
|
||
"steps_failed": 0,
|
||
"steps_terminal": 0,
|
||
"steps_loaded": false,
|
||
"focus_ref": {
|
||
"kind": "ops_job",
|
||
"job_id": 101,
|
||
"job_code": "ops-20260418064500-a1b2c3",
|
||
"action": "logs.collect",
|
||
"target_node_code": "mainland-worker-01"
|
||
},
|
||
"steps": []
|
||
}
|
||
```
|
||
|
||
正式要求：
|
||
|
||
- `job_code`
|
||
- 是跨页面、CLI、事件流的稳定任务标识
|
||
- `action`
|
||
- 是执行语义主键，不应用 `title` 或自然语言代替
|
||
- `target_type`
|
||
- 定义任务作用范围，不应只靠是否传了 `target_node_code` 来猜
|
||
- `execution_mode`
|
||
- 是正式执行方式，不是附属备注
|
||
- `policy`
|
||
- 是创建时冻结的策略快照，不应由页面二次推导
|
||
- `status_label / approval_status_label`
|
||
- 是给页面、CLI、Codex 直接消费的人类可读标签
|
||
- `summary_text`
|
||
- 是任务摘要 contract，不要求调用方再用状态和节点自己拼一句中文
|
||
- `steps_total / steps_running / steps_success / steps_failed / steps_terminal`
|
||
- 是任务步骤聚合统计，避免前端和 CLI 再自行遍历 steps 统计
|
||
- `focus_ref`
|
||
- 是统一定位对象，后续页面、CLI、Codex 应优先消费它，而不是自己重组跳转参数
|
||
|
||
---
|
||
|
||
## 4. Job 状态机
|
||
|
||
### 4.1 当前实现中已使用的状态
|
||
|
||
当前后端已实际使用：
|
||
|
||
- `queued`
|
||
- `awaiting_approval`
|
||
- `blocked`
|
||
- `running`
|
||
- `success`
|
||
- `failed`
|
||
- `cancelled`
|
||
|
||
这些状态已经是正式 contract 的最小闭环。
|
||
|
||
### 4.2 推荐终局扩展状态
|
||
|
||
终局可继续扩展为：
|
||
|
||
- `draft`
|
||
- `approved`
|
||
- `queued`
|
||
- `dispatching`
|
||
- `running`
|
||
- `verifying`
|
||
- `partially_succeeded`
|
||
- `rollback_running`
|
||
- `rolled_back`
|
||
- `success`
|
||
- `failed`
|
||
- `cancelled`
|
||
- `timed_out`
|
||
|
||
但要明确区分：
|
||
|
||
- 当前实际 API 已稳定承诺的，是最小状态集
|
||
- 终局扩展状态不能破坏当前状态语义
|
||
|
||
---
|
||
|
||
## 5. Execution Mode Contract
|
||
|
||
当前实现里正式出现的执行方式：
|
||
|
||
- `remote-agent`
|
||
- `local-runtime`
|
||
- `control-plane`
|
||
- `ssh`
|
||
|
||
语义必须固定：
|
||
|
||
### 5.1 `remote-agent`
|
||
|
||
- 默认正式执行方式
|
||
- 由目标节点 Node Agent 拉取并执行
|
||
|
||
### 5.2 `local-runtime`
|
||
|
||
- 仅用于当前节点本机即时执行
|
||
- 主要用于本机 runtime 动作
|
||
|
||
### 5.3 `control-plane`
|
||
|
||
- 由海外控制面直接执行
|
||
- 适用于控制面内生动作
|
||
|
||
### 5.4 `ssh`
|
||
|
||
- 只作为救援和过渡执行器
|
||
- 不能成为大规模正式发布的默认方式
|
||
|
||
正式约束：
|
||
|
||
- 发布默认优先 `remote-agent`
|
||
- `ssh` 只应是兜底链路
|
||
- 页面、CLI、Codex 不应再自行创造第 5 种 execution mode
|
||
|
||
---
|
||
|
||
## 6. Target Type Contract
|
||
|
||
当前 contract 至少应允许：
|
||
|
||
- `node`
|
||
- `batch`
|
||
- `nodes`
|
||
- `rollout`
|
||
|
||
语义建议固定为：
|
||
|
||
### 6.1 `node`
|
||
|
||
- 单节点任务
|
||
|
||
### 6.2 `batch` / `nodes`
|
||
|
||
- 多节点批量任务
|
||
- 当前通常通过 `/ops/jobs/batch` 创建
|
||
|
||
### 6.3 `rollout`
|
||
|
||
- 该任务属于某轮 release rollout 推进的一部分
|
||
|
||
正式要求：
|
||
|
||
- 如果任务来自 rollout，必须能通过 `rollout_id` 回溯
|
||
- 如果任务来自 playbook，应通过 `metadata` 标出 run 归属
|
||
|
||
---
|
||
|
||
## 7. Job Step Contract
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/jobs/{job_id}`
|
||
|
||
最小字段：
|
||
|
||
```json
|
||
{
|
||
"id": 201,
|
||
"step_key": "dispatch",
|
||
"title": "调度动作",
|
||
"node_code": "mainland-worker-01",
|
||
"status": "queued",
|
||
"stdout": "",
|
||
"stderr": "",
|
||
"result": {},
|
||
"created_at": "2026-04-18 06:45:00",
|
||
"started_at": "",
|
||
"finished_at": "",
|
||
"updated_at": "2026-04-18 06:45:00"
|
||
}
|
||
```
|
||
|
||
当前实现说明：
|
||
|
||
- 现阶段每条 `ops job` 至少会先生成一个 `dispatch` step
|
||
- 以后如果引入真正 DAG / 多步骤执行，也应继续沿用 `ops_job_steps`
|
||
|
||
正式约束：
|
||
|
||
- `step_key`
|
||
- 是步骤主键，不应用标题代替
|
||
- `status`
|
||
- 必须与 job 状态联动，但不要求完全相同
|
||
- `stdout / stderr`
|
||
- 是步骤级输出，不应与整条 job 的 `result` 混成一层
|
||
|
||
---
|
||
|
||
## 8. Job Event Contract
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/jobs/{job_id}/events`
|
||
|
||
最小字段建议：
|
||
|
||
```json
|
||
{
|
||
"id": 301,
|
||
"event_key": "job-event:301",
|
||
"job_id": 101,
|
||
"step_id": 201,
|
||
"node_code": "mainland-worker-01",
|
||
"client_event_id": "evt-001",
|
||
"event_type": "job_dispatch_requested",
|
||
"level": "info",
|
||
"level_label": "信息",
|
||
"message": "任务等待 node-agent 拉取: 101",
|
||
"summary": "任务等待 node-agent 拉取: 101",
|
||
"payload": {},
|
||
"has_payload": false,
|
||
"occurred_at": "2026-04-18 06:45:03",
|
||
"created_at": "2026-04-18 06:45:03"
|
||
}
|
||
```
|
||
|
||
当前代码里事件来源包括：
|
||
|
||
- job created
|
||
- job blocked
|
||
- job awaiting approval
|
||
- job approved
|
||
- job cancelled
|
||
- job dispatch requested
|
||
- job executed locally / over ssh / on control plane
|
||
- agent start / complete / custom events
|
||
|
||
正式要求：
|
||
|
||
- `events` 是任务时间线，不是调试附属品
|
||
- 页面、CLI、Codex 都应从这里读“任务做到哪一步了”
|
||
- 不应重新从 stdout/stderr 推断关键状态
|
||
|
||
补充约束：
|
||
|
||
- `event_key`
|
||
- 是事件行的稳定前端 / CLI 标识
|
||
- `summary`
|
||
- 是给页面、CLI、Codex 直接消费的短摘要，不要求调用方再拼接 `message`
|
||
- `level_label`
|
||
- 是面向人看的等级标签
|
||
- `occurred_at`
|
||
- 是统一观察面时间字段，允许与 `created_at` 同值
|
||
|
||
### 8.1 Event List Summary Contract
|
||
|
||
当前 `GET /api/v1/ops/jobs/{job_id}/events` 建议同时返回：
|
||
|
||
```json
|
||
{
|
||
"summary": {
|
||
"job_id": 101,
|
||
"returned_total": 12,
|
||
"level_counts": {
|
||
"info": 8,
|
||
"warning": 3,
|
||
"error": 1
|
||
},
|
||
"event_type_counts": {
|
||
"job_created": 1,
|
||
"job_dispatch_requested": 1,
|
||
"executor_received": 1
|
||
},
|
||
"node_counts": {
|
||
"mainland-worker-01": 12
|
||
},
|
||
"latest_at": "2026-04-18 06:45:08",
|
||
"filters": {
|
||
"limit": 50
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
正式要求：
|
||
|
||
- 事件接口不只给一串数组
|
||
- 至少要给：
|
||
- 当前返回了多少条
|
||
- 等级分布
|
||
- 事件类型分布
|
||
- 最近时间
|
||
- 这样页面、CLI、Codex 才能先判断“有没有异常事件”再决定是否展开明细
|
||
|
||
---
|
||
|
||
## 9. List Contract 与 Compact 模式
|
||
|
||
来源：
|
||
|
||
- `GET /api/v1/ops/jobs`
|
||
|
||
当前实现特点：
|
||
|
||
- 默认 `list_ops_jobs(..., compact=True)`
|
||
- 返回 compact job
|
||
- `steps` 被置空
|
||
- `payload / metadata / result` 会被压缩
|
||
|
||
因此建议把列表 contract 明确成：
|
||
|
||
```json
|
||
{
|
||
"jobs": [
|
||
{
|
||
"id": 101,
|
||
"job_code": "ops-20260418064500-a1b2c3",
|
||
"action": "logs.collect",
|
||
"status": "queued",
|
||
"execution_mode": "remote-agent",
|
||
"requested_by": "web-ui/ops-center",
|
||
"payload": {
|
||
"tail_lines": 120
|
||
},
|
||
"metadata": {},
|
||
"result": {},
|
||
"result_summary": {},
|
||
"is_compact": true,
|
||
"steps": []
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
正式要求：
|
||
|
||
- 列表页默认只承诺 compact 视图
|
||
- 详情页再承诺 full job + steps
|
||
- 页面不应假设列表结果天然带完整 steps
|
||
|
||
---
|
||
|
||
## 10. Create Contract
|
||
|
||
来源：
|
||
|
||
- `POST /api/v1/ops/jobs`
|
||
|
||
最小请求体：
|
||
|
||
```json
|
||
{
|
||
"action": "logs.collect",
|
||
"target_type": "node",
|
||
"target_node_code": "mainland-worker-01",
|
||
"execution_mode": "remote-agent",
|
||
"requested_by": "web-ui/ops-center",
|
||
"payload": {
|
||
"tail_lines": 120
|
||
},
|
||
"metadata": {
|
||
"source": "ops-center"
|
||
},
|
||
"auto_approve": false,
|
||
"run_now": true
|
||
}
|
||
```
|
||
|
||
当前实现还支持：
|
||
|
||
- `target_selector`
|
||
- `rollout_id`
|
||
|
||
最小返回体：
|
||
|
||
```json
|
||
{
|
||
"job": {},
|
||
"executed_immediately": false
|
||
}
|
||
```
|
||
|
||
关键语义：
|
||
|
||
- `executed_immediately=true`
|
||
- 说明任务已被控制面即时执行，而不是停留在队列
|
||
- `executed_immediately=false`
|
||
- 说明任务进入了：
|
||
- `queued`
|
||
- `awaiting_approval`
|
||
- `blocked`
|
||
|
||
正式要求：
|
||
|
||
- 创建接口不能只回答“建成功了”
|
||
- 必须明确回答：
|
||
- 当前状态是什么
|
||
- 是否立即执行
|
||
- 如果没执行，是在等审批、等 agent、还是被阻断
|
||
|
||
---
|
||
|
||
## 11. Batch Create Contract
|
||
|
||
来源：
|
||
|
||
- `POST /api/v1/ops/jobs/batch`
|
||
|
||
最小请求体建议：
|
||
|
||
```json
|
||
{
|
||
"template_key": "diagnostics.collect",
|
||
"target_node_codes": [
|
||
"mainland-worker-01",
|
||
"mainland-worker-02"
|
||
],
|
||
"execution_mode": "remote-agent",
|
||
"requested_by": "web-ui/ops-center",
|
||
"payload": {},
|
||
"metadata": {},
|
||
"auto_approve": true
|
||
}
|
||
```
|
||
|
||
最小返回体建议：
|
||
|
||
```json
|
||
{
|
||
"jobs": [],
|
||
"results": []
|
||
}
|
||
```
|
||
|
||
正式要求：
|
||
|
||
- batch 只是创建方式，不是新的执行对象
|
||
- batch 最终仍要展开成多条 `ops job`
|
||
- 单条失败不能让结果完全丢失，必须保留逐项结果
|
||
|
||
---
|
||
|
||
## 12. Policy Preview Contract
|
||
|
||
来源：
|
||
|
||
- `POST /api/v1/ops/policy/preview`
|
||
- `ops_jobs.policy`
|
||
|
||
这层是创建前的正式门禁对象。
|
||
|
||
当前创建任务时，后端已先做：
|
||
|
||
- 风险等级判断
|
||
- 阻断判断
|
||
- 审批要求判断
|
||
|
||
建议最小返回字段：
|
||
|
||
```json
|
||
{
|
||
"blocked": false,
|
||
"blocking_reasons": [],
|
||
"approval_required": true,
|
||
"approval_reasons": [
|
||
"当前目标包含 control 节点"
|
||
],
|
||
"warnings": [],
|
||
"recommendations": [],
|
||
"risk_level": "high",
|
||
"target_summary": {},
|
||
"batch_plan": {},
|
||
"cluster_guardrails": {},
|
||
"operational_readiness": {}
|
||
}
|
||
```
|
||
|
||
正式要求：
|
||
|
||
- 页面、CLI、Codex 都应复用同一份 policy preview
|
||
- 不能各自再写一套“能不能执行”的判断逻辑
|
||
|
||
---
|
||
|
||
## 13. Approval / Cancel / Dispatch Contract
|
||
|
||
### 13.1 Approve
|
||
|
||
来源：
|
||
|
||
- `POST /api/v1/ops/jobs/{job_id}/approve`
|
||
|
||
最小请求体：
|
||
|
||
```json
|
||
{
|
||
"approved_by": "admin"
|
||
}
|
||
```
|
||
|
||
语义：
|
||
|
||
- 仅 `awaiting_approval` 可审批
|
||
- 审批后回到 `queued`
|
||
|
||
### 13.2 Cancel
|
||
|
||
来源：
|
||
|
||
- `POST /api/v1/ops/jobs/{job_id}/cancel`
|
||
|
||
最小请求体：
|
||
|
||
```json
|
||
{
|
||
"cancelled_by": "admin",
|
||
"reason": "本轮灰度暂停"
|
||
}
|
||
```
|
||
|
||
语义：
|
||
|
||
- `success / failed / cancelled` 不可取消
|
||
- 取消后 job 进入 `cancelled`
|
||
- steps 同步进入 `cancelled` 或保持终态
|
||
|
||
### 13.3 Dispatch
|
||
|
||
来源：
|
||
|
||
- `POST /api/v1/ops/jobs/{job_id}/dispatch`
|
||
|
||
语义：
|
||
|
||
- `queued` 才可派发
|
||
- `control-plane`
|
||
- 立即由控制面执行
|
||
- `local-runtime`
|
||
- 当前节点本机即时执行
|
||
- `ssh`
|
||
- 由 SSH 执行器执行
|
||
- `remote-agent`
|
||
- 保持 `queued`，等待对应 agent 拉取
|
||
|
||
这点必须明确：
|
||
|
||
> dispatch 不等于一定会立刻变成 running。
|
||
|
||
对于 `remote-agent`，它可能只是：
|
||
|
||
- 写入事件
|
||
- 保持排队
|
||
- 等节点稍后 pull
|
||
|
||
---
|
||
|
||
## 14. 与 Playbook / Rollout / Agent 的边界
|
||
|
||
### 14.1 与 Playbook
|
||
|
||
- `playbook run`
|
||
- 负责展开、聚合、重跑、取消整轮编排
|
||
- `ops job`
|
||
- 负责单条执行记录
|
||
|
||
### 14.2 与 Rollout
|
||
|
||
- `rollout`
|
||
- 负责发布批次推进
|
||
- `ops job`
|
||
- 负责每个节点每步的实际任务
|
||
|
||
### 14.3 与 Node Agent
|
||
|
||
- 控制面创建 `ops job`
|
||
- agent 拉取 `ops job`
|
||
- agent 回写 `ops_job_events`
|
||
- agent 推进 job / step 状态
|
||
|
||
### 14.4 与 Observability
|
||
|
||
- `ops_observability_contract`
|
||
- 看现场事实
|
||
- `ops_job_contract`
|
||
- 看执行动作本身
|
||
|
||
这两层不能混：
|
||
|
||
- 观察面告诉你“谁在跑、谁异常”
|
||
- 任务面告诉你“这一条任务如何被创建、审批、派发、执行、取消”
|
||
|
||
---
|
||
|
||
## 15. 终局约束
|
||
|
||
后续不应再回退到下面几种模式：
|
||
|
||
### 1. 页面点按钮直接跑 shell
|
||
|
||
错误。
|
||
|
||
正确方式：
|
||
|
||
- 页面创建 `ops job`
|
||
|
||
### 2. Codex 直接 SSH 改现场
|
||
|
||
错误。
|
||
|
||
正确方式：
|
||
|
||
- Codex 创建 `ops job` / `playbook run`
|
||
|
||
### 3. 发布系统不经过任务层
|
||
|
||
错误。
|
||
|
||
正确方式：
|
||
|
||
- rollout 批量生成 `ops job`
|
||
|
||
### 4. 死信治理不留痕
|
||
|
||
错误。
|
||
|
||
正确方式：
|
||
|
||
- `flush / replay / discard` 都生成正式 `ops job`
|
||
|
||
所以可以把这份文档看成整套系统的“执行订单母本”。
|
||
|
||
后续只要这份 contract 稳住，页面、CLI、Codex、Agent、Release、Playbook 就都能围绕同一个中心对象协作。
|