scobleclaw/kubeclaw/docs/architecture.md

# kubeclaw Architecture

> **Status:** Draft v0.1  
> **Author:** @agent-cos  
> **Date:** 2026-05-23

---

## 1. System Context

kubeclaw is a Kubernetes operator that runs Scobleclaw agents as first-class cluster workloads. It is part of the Scobleclaw ecosystem and integrates with:

- **OpenClaw Gateway** — LLM API gateway providing model access, rate limiting, and credential management
- **Agent Judge** — authorization layer that validates every action before execution
- **Scobleclaw dispatch system** — higher-level orchestration that may trigger agent deployments

## 2. Operator Pattern

kubeclaw follows standard Kubernetes operator patterns using the `controller-runtime` library:

```
WATCH: CRD events (Agent, AgentGroup, etc.)
  │
  ▼
RECONCILE: desired state → running state
  │
  ├── CREATE: Build Agent Pod spec, inject config/secrets, wire sidecars
  ├── UPDATE: Diff current vs desired → patch deployment
  └── DELETE: Tear down gracefully, flush audit logs
```

### 2.1 Reconciler Loop

Each CRD has a dedicated reconciler. The primary flow:

1. **Agent Reconciler** watches `Agent` resources
2. On create/update, it builds a `Pod` spec with:
   - Main agent container (OpenClaw agent runtime)
   - Judge sidecar (deepseek-v4-pro for authorization validation)
   - Workflow executor sidecar (runs approved template steps)
   - ConfigMap references for agent config
   - Secret references for API keys (never injected into agent context)
3. On AgentConfig reference change, it triggers re-reconciliation of all Agents using that config
4. On Provider or CompliancePolicy change, it triggers rolling update of affected agents

### 2.2 Sidecar Architecture

Each Agent Pod has three containers:

#### Actor Agent Container
- Runs the OpenClaw agent process
- Has **exactly one tool**: `template_workflow(workflowTemplateRef, params)`
- No network access except to judge sidecar (localhost) and OpenClaw gateway
- No filesystem write access outside `/workspace` (tmpfs)
- No access to K8s API

#### Judge Sidecar
- Runs a separate LLM model instance (deepseek-v4-pro or higher)
- Implements the Judge protocol from `agent-judge-architecture.md`
- Evaluates every `template_workflow` call against:
  - WorkflowTemplate spec
  - CompliancePolicy rules
  - Agent-level whitelists
- Returns APPROVE / REJECT / ESCALATE_HUMAN

#### Workflow Executor Sidecar
- Runs approved workflow steps
- Each step is a self-contained action with structured input/output
- All steps are logged to the audit trail (persistent volume + stdout)

## 3. API Design

### 3.1 CRD Group

- **Group:** `kubeclaw.nousresearch.com`
- **Version:** `v1alpha1` (initial, will graduate to v1)
- **Scope:** Mix of cluster-scoped and namespaced

### 3.2 Validation

- **CRD schema validation** — OpenAPI v3 schemas on each CRD for structural validation
- **Admission webhooks** — `ValidatingWebhookConfiguration` for semantic validation (e.g., provider must exist, channel must be reachable)
- **Status subresource** — Each CRD has a `.status` block with observed state, conditions, and last reconciliation

### 3.3 Status Reporting

All CRDs use standard Kubernetes status conditions:

```yaml
status:
  conditions:
    - type: Ready
      status: "True"
      reason: AgentPodRunning
      message: "Agent pod engineering-7d8f9c is running"
      lastTransitionTime: "2026-05-23T19:00:00Z"
    - type: JudgeConnected
      status: "True"
      reason: JudgeResponding
      message: "Judge sidecar is online and responding"
  observedGeneration: 3
  agentPodRef:
    name: engineering-7d8f9c
    namespace: agents
  lastWorkflowCount: 142
  lastJudgeVerdict: APPROVE
```

## 4. Security Architecture

### 4.1 Pod Security

- **PodSecurityContext**: Run as non-root user, read-only root filesystem
- **Seccomp profile**: Runtime default
- **AppArmor**: Kubeclaw-specific profile limiting agent to `template_workflow` only
- **NetworkPolicy**: Agents can only talk to judge sidecar (localhost) and OpenClaw gateway
- **Resource limits**: Hard CPU/memory limits per Agent spec

### 4.2 Secret Management

- API keys are standard K8s Secrets referenced by `apiKeyRef` in Provider and Channel CRDs
- Secrets are **never** injected into the agent container's environment
- Only the judge sidecar and workflow executor have access to resolved credentials
- The actor agent container never sees raw API keys

### 4.3 RBAC

```yaml
# Minimal RBAC for the controller
- apiGroups: ["kubeclaw.nousresearch.com"]
  resources: ["agents", "agentgroups", "providers", "channels", "workflowtemplates", "compliancepolicies"]
  verbs: ["get", "list", "watch", "patch"]
- apiGroups: [""]
  resources: ["pods", "configmaps", "secrets", "events"]
  verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
- apiGroups: ["networking.k8s.io"]
  resources: ["networkpolicies"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]
```

## 5. Operational Concerns

### 5.1 Observability

- **Metrics**: Standard controller-runtime metrics (ReconcileTotal, WorkQueueDepth) + custom:
  - `kubeclaw_agent_count` — number of agent deployments
  - `kubeclaw_judge_verdicts_total` — verdict counts by type (APPROVE/REJECT/ESCALATE)
  - `kubeclaw_workflow_invocations_total` — workflow calls by template
- **Logging**: Structured JSON logs, per-reconciliation trace IDs
- **Tracing**: OpenTelemetry integration for end-to-end workflow tracing

### 5.2 Reconciliation Performance

- Agent pod creation is asynchronous; controller creates Deployment, not bare Pod
- Large AgentGroups are reconciled via batch operations with worker pool
- Requeue on conflict with exponential backoff (100ms → 30s)
- Watch caching via informer with 10-minute resync period

### 5.3 Upgrade Strategy

- CRD versioning with conversion webhooks for backward compatibility
- Helm chart with `--upgrade` support
- Canary deployments for controller upgrades
- Agent pods use standard rolling update strategy

## 6. Future Enhancements

- [ ] **Horizontal Pod Autoscaling** for AgentGroups based on workflow queue depth
- [ ] **Agent-to-agent networking** via Service Mesh (Istio/Linkerd) for multi-agent coordination
- [ ] **Workflow DAG execution** — complex multi-step workflows with branching
- [ ] **Compliance attestation** — generate compliance reports for SOC2/HIPAA/GDPR audits
- [ ] **Web UI** — Dashboard for viewing agent status, workflow history, judge decisions