scobleclaw/docs/agent-judge-architecture.md

Scobleclaw Agent-Judge Authorization Layer — Architecture

Author: @senior-eng, @agent-cos Date: 2026-05-11 (revised 2026-05-23) Status: Final — Approved Architecture Reference: scobleclaw#7 Integration: kubeclaw — implements this spec as Kubernetes operator with judge sidecar

---

1. Principle: Actor-Judge Separation

Nate's core insight (LLM Agents: The Security Breach Pattern Nobody's Talking About): an agent optimizing for task completion cannot also be trusted to evaluate its own permission boundaries. The actor (task agent) and the judge (authorization validator) must be separate model instances with divergent optimization goals.

• Actor agents (@engineering, @content, @market, @trust, @health, @vault, @presentations): optimize for task success
• Judge agent (@judge): optimizes for permission boundary enforcement — a narrow, security-hardened validator with no task-completion pressure

The judge is not an agent in the general sense. It has exactly one capability: reading structured proposals and returning structured verdicts. It has no write access to any surface.

2. Risk Classification Matrix

Each write surface is scored across four axes (1–5) and assigned an autonomy tier.

Write Surface	Surface Area	Action Risk	Autonomy Tier	Judge Required
Internal memory writes (OB1, MEMORY.md)	1	1 (read-only/internal)	no-judge	No
Vault note creation	1	2 (internal write)	lightweight-approval	Pattern check only
Issue comments (non-PR)	2	2 (internal write)	lightweight-approval	Pattern check only
Discord messages	3	3 (external write)	full-judge-review	Yes
Forgejo PR creation	4	4 (code mutation)	full-judge-review	Yes
File writes (outside workspace)	4	4 (filesystem mutation)	full-judge-review	Yes
Exec/command on host	5	5 (system control)	full-judge-review	Yes
Web publishing	3	4 (public-facing write)	full-judge-review	Yes
Impersonation/financial ops	5	5 (irreversible)	human-only	N/A — human gate

Autonomy tiers:

• no-judge: Read-only ops, no proposal needed
• lightweight-approval: Internal writes, judge does pattern/topic whitelist check (<100ms)
• full-judge-review: External/mutating writes, full proposal evaluation with evidence verification
• human-only: Financial, impersonation, credential rotation — must route to Scott

3. Judge Protocol Spec

3.1 Proposal Format (Actor → Judge)

{
  "proposal_id": "uuid",
  "agent_id": "@engineering",
  "action_type": "forgejo_pr_create",
  "target": {
    "repo": "sscoble/speckl",
    "branch": "feat/add-schema-validation",
    "base": "main"
  },
  "parameters": {
    "files_changed": ["schema/validator.ts", "schema/types.ts"],
    "estimated_diff_lines": 45
  },
  "justification": "Implementing OpenAPI schema validation per issue #42",
  "cited_evidence": [
    {"type": "issue", "url": "https://os.scoble.me/forgejo/sscoble/speckl/issues/42"},
    {"type": "task_assignment", "source": "@cos dispatch #789"}
  ],
  "scope_claim": {
    "files_only": ["src/schema/"],
    "max_lines": 80,
    "no_existing_code_deletion": true
  }
}

3.2 Verdict Format (Judge → Actor)

{
  "proposal_id": "uuid",
  "verdict": "APPROVE | REJECT | CLARIFY",
  "reason": "Structured explanation in plain text",
  "constraints": {
    "scope_limit": "Must not modify files outside src/schema/",
    "require_human_review": false
  },
  "escalation": null
}

3.3 Retry/Override Logic

┌─────────┐    PROPOSAL     ┌─────────┐
│  ACTOR  │ ───────────────>│  JUDGE  │
└─────────┘                 └─────────┘
     ^                          │
     │        APPROVE           │──► Execute (with constraints)
     │──────────────────────────│
     │                          │
     │        CLARIFY           │──► Actor resubmits with more info (max 2 retries)
     │◄─────────────────────────│
     │                          │
     │        REJECT            │──► Logged as denied action
     │◄─────────────────────────│    Actor may NOT retry same proposal
                                  │    Actor may submit revised proposal
                                  │
                              HUMAN ESCALATION
                                  │
                              Judge can't decide → routes to Scott
                              with structured dispute (both positions)

• CLARIFY: Actor must add missing evidence or narrow scope. Max 2 clarify cycles, then auto-reject.
• REJECT: Hard no on this proposal. Actor may submit a revised proposal with different scope/justification. Repeat rejections on same target trigger an audit flag.
• Ambiguity: Judge rates confidence 0.0–1.0. Below 0.6 → auto-route to CLARIFY. Below 0.3 → escalate to human.

4. Integration Point — MCP Tool Proxy Middleware (Option B)

Options Evaluated

Option	Description	Pros	Cons	Verdict
A	@cos acts as dispatch + judge	Simple, no new infra	Violates actor-judge separation; @cos is an actor	❌
B	Middleware intercepting all MCP tool calls	Clean separation, transparent to agents	Requires MCP proxy layer	✅ Recommend
C	Per-department judge agents	Granular, domain-aware	Explosion of judge instances; inconsistent rulings	❌

Recommended Architecture

Introduce a thin judge proxy between agent MCP tool calls and actual tool execution:

Agent (@engineering)
  │
  ├── mcp_call("forgejo.create_pr", params)
  │
  ▼
MCP Judge Proxy ──► Classify risk tier of tool call
  │                   If full-judge-review: build proposal → @judge → verdict
  │                   If lightweight: pattern check → approve/flag
  │                   If no-judge: pass through
  ▼
Actual Tool Execution (or rejection)

The proxy lives in the OpenClaw gateway layer, intercepting MCP tool invocations before they reach the tool implementation. This means:

1. Zero agent changes — existing agents don't need to know the judge exists
2. Single enforcement point — one place to configure risk tiers and judge routing
3. Latency budget: lightweight checks <100ms; full judge review targets <2s (single model call)
4. Audit trail: every proposal and verdict is logged to ~/.openclaw/judge-audit/

Kubernetes Integration (kubeclaw)

In the kubeclaw operator (see kubeclaw/README.md), the judge is implemented as a sidecar container in each Agent Pod:

• Actor Agent Container: Runs the OpenClaw agent process with exactly one tool: template_workflow(workflowTemplateRef, params). No raw exec, no arbitrary file writes, no network access except to judge sidecar.
• Judge Sidecar: Runs a separate, stronger model instance (e.g., deepseek-v4-pro:cloud). Evaluates every template_workflow call against CompliancePolicy CRDs.
• Workflow Executor Sidecar: Runs approved workflow steps with structured input/output, logs everything to audit trail.

5. Implementation Priority

Ordered by [risk × irreversibility]:

Priority	Surface	Tier	Rationale
P0	Exec/command on host	full-judge-review	Can compromise the entire system. Deny-by-default exec proxy first.
P0	Forgejo PR creation	full-judge-review	Code mutation to production repos. Second-highest blast radius.
P1	File writes outside workspace	full-judge-review	Filesystem mutation outside sandboxed workspace.
P1	Web publishing	full-judge-review	Public-facing content; reputational and correctness risk.
P2	Discord messages	full-judge-review	External communication; lower irreversibility (editable).
P3	Vault/internal file writes	lightweight-approval	Internal-facing, reversible, contained blast radius.

6. Metrics & Observability

Metric	Target	Alert Threshold
Judge approval rate	70-85%	<50% (judge too strict) or >95% (judge too permissive)
Human override rate	<5%	>10% (judge misaligned with human intent)
False negative rate (audited)	<2%	>5% (judge blocking legitimate work)
Latency: lightweight check	<100ms p95	>250ms
Latency: full judge review	<2s p95	>5s
Clarify cycles per approval	<1.0 avg	>2.0 (proposals poorly formed)
Escalations to human	<3/day	>10/day

Audit cadence: Weekly review of all REJECT verdicts and all escalated proposals. Monthly spot-check of 5% of APPROVE verdicts for correctness (false-approval detection).

7. Implementation Sequence

Phase 1 (Week 1): Judge Model & Protocol

• Define @judge agent config (narrow model, security-hardened prompt)
• Implement proposal/verdict JSON schemas
• Unit test judge on 50 canned proposals (balanced approve/reject/clarify)

Phase 2 (Week 2): MCP Proxy for Exec + Forgejo

• Build MCP tool proxy in gateway layer
• Wire exec and forgejo.create_pr through judge
• Shadow mode: log verdicts but don't enforce (measure metrics)

Phase 3 (Week 3): Enforcement + Remaining Surfaces

• Enable enforcement for exec + forgejo
• Add Discord, web publish, file write surfaces
• Human escalation path (Discord DM to Scott with structured dispute)

Phase 4 (Week 4+): Tuning

• Review metrics, tune judge prompt for precision/recall balance
• Add lightweight-approval tier for internal writes
• Deploy kubeclaw operator with judge sidecar pattern

8. Design Decisions

Decision	Resolution	Rationale
Judge model	deepseek-v4-pro:cloud	Strong reasoning model for accurate verdict evaluation; separate from actor agents' deepseek-v4-flash:cloud
Session-level approval	Per-action, never session-scoped	Every action gets its own verdict. Follows Zero Trust/BeyondCorp principles — continuous per-request verification. Capability-based security: unforgeable tokens per specific action, revoked after use.
Judge adversarial robustness	Periodic testing required	Actors may learn to write proposals that exploit judge blind spots. Schedule adversarial testing every 2 weeks after Phase 3.
Human escalation path	Discord DM to Scott	Structured dispute with both positions (actor's justification vs. judge's reasoning). Scott makes final call.

9. References

• scobleclaw#7 — This ticket
• kubeclaw architecture — Kubernetes operator implementing judge sidecar
• Agent Judge Rules — Per-action validation rules
• Nate's YouTube: LLM Agents: The Security Breach Pattern Nobody's Talking About