scobleclaw/docs/agent-judge-architecture.md

# Scobleclaw Agent-Judge Authorization Layer — Architecture

**Author:** @senior-eng, @agent-cos
**Date:** 2026-05-11 (revised 2026-05-23)
**Status:** Final — Approved Architecture
**Reference:** [scobleclaw#7](https://os.scoble.me/forgejo/marcus/scobleclaw/issues/7)
**Integration:** [kubeclaw](kubeclaw/README.md) — implements this spec as Kubernetes operator with judge sidecar

---

## 1. Principle: Actor-Judge Separation

Nate's core insight ([LLM Agents: The Security Breach Pattern Nobody's Talking About](https://youtu.be/SX1myuPEDFg)): an agent optimizing for task completion **cannot also be trusted to evaluate its own permission boundaries**. The actor (task agent) and the judge (authorization validator) must be separate model instances with divergent optimization goals.

- **Actor agents** (@engineering, @content, @market, @trust, @health, @vault, @presentations): optimize for *task success*
- **Judge agent** (@judge): optimizes for *permission boundary enforcement* — a narrow, security-hardened validator with no task-completion pressure

The judge is **not** an agent in the general sense. It has exactly one capability: reading structured proposals and returning structured verdicts. It has no write access to any surface.

## 2. Risk Classification Matrix

Each write surface is scored across four axes (1–5) and assigned an autonomy tier.

| Write Surface | Surface Area | Action Risk | Autonomy Tier | Judge Required |
|---|---|---|---|---|
| Internal memory writes (OB1, MEMORY.md) | 1 | 1 (read-only/internal) | no-judge | No |
| Vault note creation | 1 | 2 (internal write) | lightweight-approval | Pattern check only |
| Issue comments (non-PR) | 2 | 2 (internal write) | lightweight-approval | Pattern check only |
| Discord messages | 3 | 3 (external write) | full-judge-review | Yes |
| Forgejo PR creation | 4 | 4 (code mutation) | full-judge-review | Yes |
| File writes (outside workspace) | 4 | 4 (filesystem mutation) | full-judge-review | Yes |
| Exec/command on host | 5 | 5 (system control) | full-judge-review | Yes |
| Web publishing | 3 | 4 (public-facing write) | full-judge-review | Yes |
| Impersonation/financial ops | 5 | 5 (irreversible) | human-only | N/A — human gate |

**Autonomy tiers:**
- **no-judge**: Read-only ops, no proposal needed
- **lightweight-approval**: Internal writes, judge does pattern/topic whitelist check (<100ms)
- **full-judge-review**: External/mutating writes, full proposal evaluation with evidence verification
- **human-only**: Financial, impersonation, credential rotation — must route to Scott

## 3. Judge Protocol Spec

### 3.1 Proposal Format (Actor → Judge)

```json
{
  "proposal_id": "uuid",
  "agent_id": "@engineering",
  "action_type": "forgejo_pr_create",
  "target": {
    "repo": "sscoble/speckl",
    "branch": "feat/add-schema-validation",
    "base": "main"
  },
  "parameters": {
    "files_changed": ["schema/validator.ts", "schema/types.ts"],
    "estimated_diff_lines": 45
  },
  "justification": "Implementing OpenAPI schema validation per issue #42",
  "cited_evidence": [
    {"type": "issue", "url": "https://os.scoble.me/forgejo/sscoble/speckl/issues/42"},
    {"type": "task_assignment", "source": "@cos dispatch #789"}
  ],
  "scope_claim": {
    "files_only": ["src/schema/"],
    "max_lines": 80,
    "no_existing_code_deletion": true
  }
}
```

### 3.2 Verdict Format (Judge → Actor)

```json
{
  "proposal_id": "uuid",
  "verdict": "APPROVE | REJECT | CLARIFY",
  "reason": "Structured explanation in plain text",
  "constraints": {
    "scope_limit": "Must not modify files outside src/schema/",
    "require_human_review": false
  },
  "escalation": null
}
```

### 3.3 Retry/Override Logic

```
┌─────────┐    PROPOSAL     ┌─────────┐
│  ACTOR  │ ───────────────>│  JUDGE  │
└─────────┘                 └─────────┘
     ^                          │
     │        APPROVE           │──► Execute (with constraints)
     │──────────────────────────│
     │                          │
     │        CLARIFY           │──► Actor resubmits with more info (max 2 retries)
     │◄─────────────────────────│
     │                          │
     │        REJECT            │──► Logged as denied action
     │◄─────────────────────────│    Actor may NOT retry same proposal
                                  │    Actor may submit revised proposal
                                  │
                              HUMAN ESCALATION
                                  │
                              Judge can't decide → routes to Scott
                              with structured dispute (both positions)
```

- **CLARIFY**: Actor must add missing evidence or narrow scope. Max 2 clarify cycles, then auto-reject.
- **REJECT**: Hard no on *this* proposal. Actor may submit a revised proposal with different scope/justification. Repeat rejections on same target trigger an audit flag.
- **Ambiguity**: Judge rates confidence 0.0–1.0. Below 0.6 → auto-route to CLARIFY. Below 0.3 → escalate to human.

## 4. Integration Point — MCP Tool Proxy Middleware (Option B)

### Options Evaluated

| Option | Description | Pros | Cons | Verdict |
|---|---|---|---|---|
| **A** | @cos acts as dispatch + judge | Simple, no new infra | Violates actor-judge separation; @cos is an actor | ❌ |
| **B** | Middleware intercepting all MCP tool calls | Clean separation, transparent to agents | Requires MCP proxy layer | ✅ **Recommend** |
| **C** | Per-department judge agents | Granular, domain-aware | Explosion of judge instances; inconsistent rulings | ❌ |

### Recommended Architecture

Introduce a thin **judge proxy** between agent MCP tool calls and actual tool execution:

```
Agent (@engineering)
  │
  ├── mcp_call("forgejo.create_pr", params)
  │
  ▼
MCP Judge Proxy ──► Classify risk tier of tool call
  │                   If full-judge-review: build proposal → @judge → verdict
  │                   If lightweight: pattern check → approve/flag
  │                   If no-judge: pass through
  ▼
Actual Tool Execution (or rejection)
```

The proxy lives in the OpenClaw gateway layer, intercepting MCP tool invocations before they reach the tool implementation. This means:
1. **Zero agent changes** — existing agents don't need to know the judge exists
2. **Single enforcement point** — one place to configure risk tiers and judge routing
3. **Latency budget**: lightweight checks <100ms; full judge review targets <2s (single model call)
4. **Audit trail**: every proposal and verdict is logged to `~/.openclaw/judge-audit/`

### Kubernetes Integration (kubeclaw)

In the kubeclaw operator (see [kubeclaw/README.md](kubeclaw/README.md)), the judge is implemented as a **sidecar container** in each Agent Pod:

- **Actor Agent Container**: Runs the OpenClaw agent process with exactly one tool: `template_workflow(workflowTemplateRef, params)`. No raw exec, no arbitrary file writes, no network access except to judge sidecar.
- **Judge Sidecar**: Runs a separate, stronger model instance (e.g., `deepseek-v4-pro:cloud`). Evaluates every `template_workflow` call against `CompliancePolicy` CRDs.
- **Workflow Executor Sidecar**: Runs approved workflow steps with structured input/output, logs everything to audit trail.

## 5. Implementation Priority

Ordered by [risk × irreversibility]:

| Priority | Surface | Tier | Rationale |
|---|---|---|---|
| P0 | Exec/command on host | full-judge-review | Can compromise the entire system. Deny-by-default exec proxy first. |
| P0 | Forgejo PR creation | full-judge-review | Code mutation to production repos. Second-highest blast radius. |
| P1 | File writes outside workspace | full-judge-review | Filesystem mutation outside sandboxed workspace. |
| P1 | Web publishing | full-judge-review | Public-facing content; reputational and correctness risk. |
| P2 | Discord messages | full-judge-review | External communication; lower irreversibility (editable). |
| P3 | Vault/internal file writes | lightweight-approval | Internal-facing, reversible, contained blast radius. |

## 6. Metrics & Observability

| Metric | Target | Alert Threshold |
|---|---|---|
| Judge approval rate | 70-85% | <50% (judge too strict) or >95% (judge too permissive) |
| Human override rate | <5% | >10% (judge misaligned with human intent) |
| False negative rate (audited) | <2% | >5% (judge blocking legitimate work) |
| Latency: lightweight check | <100ms p95 | >250ms |
| Latency: full judge review | <2s p95 | >5s |
| Clarify cycles per approval | <1.0 avg | >2.0 (proposals poorly formed) |
| Escalations to human | <3/day | >10/day |

**Audit cadence**: Weekly review of all REJECT verdicts and all escalated proposals. Monthly spot-check of 5% of APPROVE verdicts for correctness (false-approval detection).

## 7. Implementation Sequence

### Phase 1 (Week 1): Judge Model & Protocol
- Define @judge agent config (narrow model, security-hardened prompt)
- Implement proposal/verdict JSON schemas
- Unit test judge on 50 canned proposals (balanced approve/reject/clarify)

### Phase 2 (Week 2): MCP Proxy for Exec + Forgejo
- Build MCP tool proxy in gateway layer
- Wire exec and forgejo.create_pr through judge
- Shadow mode: log verdicts but don't enforce (measure metrics)

### Phase 3 (Week 3): Enforcement + Remaining Surfaces
- Enable enforcement for exec + forgejo
- Add Discord, web publish, file write surfaces
- Human escalation path (Discord DM to Scott with structured dispute)

### Phase 4 (Week 4+): Tuning
- Review metrics, tune judge prompt for precision/recall balance
- Add lightweight-approval tier for internal writes
- Deploy kubeclaw operator with judge sidecar pattern

## 8. Design Decisions

| Decision | Resolution | Rationale |
|---|---|---|
| Judge model | `deepseek-v4-pro:cloud` | Strong reasoning model for accurate verdict evaluation; separate from actor agents' `deepseek-v4-flash:cloud` |
| Session-level approval | **Per-action, never session-scoped** | Every action gets its own verdict. Follows Zero Trust/BeyondCorp principles — continuous per-request verification. Capability-based security: unforgeable tokens per specific action, revoked after use. |
| Judge adversarial robustness | Periodic testing required | Actors may learn to write proposals that exploit judge blind spots. Schedule adversarial testing every 2 weeks after Phase 3. |
| Human escalation path | Discord DM to Scott | Structured dispute with both positions (actor's justification vs. judge's reasoning). Scott makes final call. |

## 9. References

- [scobleclaw#7](https://os.scoble.me/forgejo/marcus/scobleclaw/issues/7) — This ticket
- [kubeclaw architecture](kubeclaw/docs/architecture.md) — Kubernetes operator implementing judge sidecar
- [Agent Judge Rules](https://os.scoble.me/forgejo/marcus/scobleclaw/src/branch/main/docs/agent-judge-rules.md) — Per-action validation rules
- Nate's YouTube: [LLM Agents: The Security Breach Pattern Nobody's Talking About](https://youtu.be/SX1myuPEDFg)