scobleclaw/kubeclaw/docs/architecture.md

kubeclaw Architecture

Status: Draft v0.1
Author: @agent-cos
Date: 2026-05-23

---

1. System Context

kubeclaw is a Kubernetes operator that runs Scobleclaw agents as first-class cluster workloads. It is part of the Scobleclaw ecosystem and integrates with:

• OpenClaw Gateway — LLM API gateway providing model access, rate limiting, and credential management
• Agent Judge — authorization layer that validates every action before execution
• Scobleclaw dispatch system — higher-level orchestration that may trigger agent deployments

2. Operator Pattern

kubeclaw follows standard Kubernetes operator patterns using the controller-runtime library:

WATCH: CRD events (Agent, AgentGroup, etc.)
  │
  ▼
RECONCILE: desired state → running state
  │
  ├── CREATE: Build Agent Pod spec, inject config/secrets, wire sidecars
  ├── UPDATE: Diff current vs desired → patch deployment
  └── DELETE: Tear down gracefully, flush audit logs

2.1 Reconciler Loop

Each CRD has a dedicated reconciler. The primary flow:

1. Agent Reconciler watches Agent resources
2. On create/update, it builds a Pod spec with:
   • Main agent container (OpenClaw agent runtime)
   • Judge sidecar (deepseek-v4-pro for authorization validation)
   • Workflow executor sidecar (runs approved template steps)
   • ConfigMap references for agent config
   • Secret references for API keys (never injected into agent context)
3. On AgentConfig reference change, it triggers re-reconciliation of all Agents using that config
4. On Provider or CompliancePolicy change, it triggers rolling update of affected agents

2.2 Sidecar Architecture

Each Agent Pod has three containers:

Actor Agent Container

• Runs the OpenClaw agent process
• Has exactly one tool: template_workflow(workflowTemplateRef, params)
• No network access except to judge sidecar (localhost) and OpenClaw gateway
• No filesystem write access outside /workspace (tmpfs)
• No access to K8s API

Judge Sidecar

• Runs a separate LLM model instance (deepseek-v4-pro or higher)
• Implements the Judge protocol from agent-judge-architecture.md
• Evaluates every template_workflow call against:
  • WorkflowTemplate spec
  • CompliancePolicy rules
  • Agent-level whitelists
• Returns APPROVE / REJECT / ESCALATE_HUMAN

Workflow Executor Sidecar

• Runs approved workflow steps
• Each step is a self-contained action with structured input/output
• All steps are logged to the audit trail (persistent volume + stdout)

3. API Design

3.1 CRD Group

• Group: kubeclaw.nousresearch.com
• Version: v1alpha1 (initial, will graduate to v1)
• Scope: Mix of cluster-scoped and namespaced

3.2 Validation

• CRD schema validation — OpenAPI v3 schemas on each CRD for structural validation
• Admission webhooks — ValidatingWebhookConfiguration for semantic validation (e.g., provider must exist, channel must be reachable)
• Status subresource — Each CRD has a .status block with observed state, conditions, and last reconciliation

3.3 Status Reporting

All CRDs use standard Kubernetes status conditions:

status:
  conditions:
    - type: Ready
      status: "True"
      reason: AgentPodRunning
      message: "Agent pod engineering-7d8f9c is running"
      lastTransitionTime: "2026-05-23T19:00:00Z"
    - type: JudgeConnected
      status: "True"
      reason: JudgeResponding
      message: "Judge sidecar is online and responding"
  observedGeneration: 3
  agentPodRef:
    name: engineering-7d8f9c
    namespace: agents
  lastWorkflowCount: 142
  lastJudgeVerdict: APPROVE

4. Security Architecture

4.1 Pod Security

• PodSecurityContext: Run as non-root user, read-only root filesystem
• Seccomp profile: Runtime default
• AppArmor: Kubeclaw-specific profile limiting agent to template_workflow only
• NetworkPolicy: Agents can only talk to judge sidecar (localhost) and OpenClaw gateway
• Resource limits: Hard CPU/memory limits per Agent spec

4.2 Secret Management

• API keys are standard K8s Secrets referenced by apiKeyRef in Provider and Channel CRDs
• Secrets are never injected into the agent container's environment
• Only the judge sidecar and workflow executor have access to resolved credentials
• The actor agent container never sees raw API keys

4.3 RBAC

# Minimal RBAC for the controller
- apiGroups: ["kubeclaw.nousresearch.com"]
  resources: ["agents", "agentgroups", "providers", "channels", "workflowtemplates", "compliancepolicies"]
  verbs: ["get", "list", "watch", "patch"]
- apiGroups: [""]
  resources: ["pods", "configmaps", "secrets", "events"]
  verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
- apiGroups: ["networking.k8s.io"]
  resources: ["networkpolicies"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]

5. Operational Concerns

5.1 Observability

• Metrics: Standard controller-runtime metrics (ReconcileTotal, WorkQueueDepth) + custom:
  • kubeclaw_agent_count — number of agent deployments
  • kubeclaw_judge_verdicts_total — verdict counts by type (APPROVE/REJECT/ESCALATE)
  • kubeclaw_workflow_invocations_total — workflow calls by template
• Logging: Structured JSON logs, per-reconciliation trace IDs
• Tracing: OpenTelemetry integration for end-to-end workflow tracing

5.2 Reconciliation Performance

• Agent pod creation is asynchronous; controller creates Deployment, not bare Pod
• Large AgentGroups are reconciled via batch operations with worker pool
• Requeue on conflict with exponential backoff (100ms → 30s)
• Watch caching via informer with 10-minute resync period

5.3 Upgrade Strategy

• CRD versioning with conversion webhooks for backward compatibility
• Helm chart with --upgrade support
• Canary deployments for controller upgrades
• Agent pods use standard rolling update strategy

6. Future Enhancements

• Horizontal Pod Autoscaling for AgentGroups based on workflow queue depth
• Agent-to-agent networking via Service Mesh (Istio/Linkerd) for multi-agent coordination
• Workflow DAG execution — complex multi-step workflows with branching
• Compliance attestation — generate compliance reports for SOC2/HIPAA/GDPR audits
• Web UI — Dashboard for viewing agent status, workflow history, judge decisions