sscoble/scobleclaw
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

kubeclaw: New project — OpenClaw operator for Kubernetes

Browse source 
Define kubeclaw as a Kubernetes operator that runs Scobleclaw AI agents
as first-class K8s workloads with CRDs, controller loops, and
security-hardened operator patterns.

Key deliverables in this commit:
- README.md — Project spec with CRD reference, architecture diagram,
  security model, threat model, and quick-start guide
- docs/architecture.md — Deep architecture doc covering operator pattern,
  sidecar design, API design, security architecture, and operations
- docs/development.md — Dev setup guide with prerequisites, commands,
  testing, and local dev tips

Integrates with:
- scobleclaw#7 (Agent-Judge authorization layer)
- OpenClaw gateway for LLM provider management
- Standard Kubernetes operator patterns

Closes scobleclaw#8
This commit is contained in:
agent-cos 2026-05-23 19:33:39 -07:00
parent d74bb3852a
commit 3264f9fe66
3 changed files with 801 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

478
kubeclaw/README.md Normal file
View file

@ -0,0 +1,478 @@
# kubeclaw
**OpenClaw operator for Kubernetes** — runs Scobleclaw AI agents as first-class Kubernetes workloads with CRDs, controller loops, and security-hardened operator patterns.
> **Status:** Project Kickoff — Architecture & Spec Draft
> **Author:** @agent-cos
> **Related:** scobleclaw#7 (Agent Judge architecture), scobleclaw#8 (this ticket)
---
## What It Does
Kubeclaw brings Scobleclaw's agentic engineering patterns into Kubernetes. Instead of running agents as ad-hoc processes, kubeclaw models them as declarative K8s resources that the operator reconciles into running pods with proper configuration, secrets, channels, and authorization gating.
### Key Design Decisions
1. **Security-first agent surface** — Agents get *one* tool: `template_workflow`. No raw exec, no arbitrary file writes, no open web access. Every action flows through the judge.
2. **Actor-Judge separation** — The task agent (e.g., `@engineering`) and the authorization judge (`@judge`) are separate LLM model instances. The judge validates every workflow template invocation against CompliancePolicy CRDs before execution proceeds.
3. **Declarative all the way** — Every agent, channel binding, workflow template, and compliance policy is a Kubernetes CRD. The controller reconciles desired state → running state.
4. **Pluggable providers** — LLM providers (OpenAI, Anthropic, Ollama, OpenClaw gateway) are cluster-scoped Provider CRDs with API key references.
## Architecture
```
┌─────────────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌───────────────┐ │
│ │ Agent CRD │ │ AgentGroup CRD │ │ AgentConfig │ │
│ │ (agent spec) │ │ (orchestration) │ │ (templates) │ │
│ └────────┬─────────┘ └────────┬─────────┘ └───────┬───────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ kubeclaw Controller (Reconciler) │ │
│ │ │ │
│ │ 1. Watch Agent/AgentGroup resources │ │
│ │ 2. Reconcile → create/update/delete Agent Pods │ │
│ │ 3. Inject config, secrets, channel bindings │ │
│ │ 4. Wire judge sidecar for template_workflow authorization │ │
│ └─────────────────────────┬────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ Agent Pod (3 containers) │ │
│ │ │ │
│ │ ┌─────────────────────┐ ┌─────────────────────┐ │ │
│ │ │ Actor Agent │ │ Judge Sidecar │ │ │
│ │ │ (OpenClaw agent) │──│ (deepseek-v4-pro) │ │ │
│ │ │ │ │ Validates workflow │ │ │
│ │ │ Tool: │ │ invocations against │ │ │
│ │ │ template_workflow │ │ CompliancePolicy │ │ │
│ │ └─────────────────────┘ └─────────────────────┘ │ │
│ │ │ │
│ │ ┌────────────────────────────────────────────────────────┐ │ │
│ │ │ Workflow Executor Sidecar │ │ │
│ │ │ Runs approved workflow steps, logs to audit trail │ │ │
│ │ │ Each step: structured input → structured output │ │ │
│ │ └────────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌────────────────┐ │
│ │ WorkflowTemplate │ │ CompliancePolicy │ │ Provider CRD │ │
│ │ CRD (cluster) │ │ CRD (cluster) │ │ (cluster) │ │
│ └──────────────────┘ └──────────────────┘ └────────────────┘ │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌────────────────┐ │
│ │ Channel CRD │ │ Secret (K8s) │ │ ConfigMap │ │
│ │ (namespaced) │ │ (API keys) │ │ (agent config)│ │
│ └──────────────────┘ └──────────────────┘ └────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────┘
```
## CRD Reference
### Provider (cluster-scoped)
LLM provider configuration — API endpoints, model lists, pricing.
```yaml
apiVersion: kubeclaw.nousresearch.com/v1alpha1
kind: Provider
metadata:
name: openclaw-gateway
spec:
type: openclaw # openai | anthropic | ollama | openclaw
baseURL: https://os.scoble.me/api
apiKeyRef:
name: openclaw-credentials
key: api-key
models:
- name: deepseek-v4-flash:cloud
contextWindow: 65536
costPer1KInput: 0.0005
costPer1KOutput: 0.0015
- name: deepseek-v4-pro:cloud
contextWindow: 128000
costPer1KInput: 0.001
costPer1KOutput: 0.003
defaultModel: deepseek-v4-flash:cloud
```
### Channel (namespaced)
Platform binding — Discord, Slack, Telegram, etc.
```yaml
apiVersion: kubeclaw.nousresearch.com/v1alpha1
kind: Channel
metadata:
name: engineering-discord
namespace: agents
spec:
platform: discord
botTokenRef:
name: discord-bot-tokens
key: engineering-bot
guildID: "123456789"
allowedChannels:
- "agent-logs"
- "engineering-chat"
```
### AgentConfig (namespaced)
Reusable agent configuration templates — workspace, memory, skills, sandbox, tools.
```yaml
apiVersion: kubeclaw.nousresearch.com/v1alpha1
kind: AgentConfig
metadata:
name: standard-engineering
namespace: agents
spec:
workspace:
mountPath: /workspace
size: 10Gi
memory:
enabled: true
retentionDays: 90
skills:
- id: forgejo
version: "1.0"
- id: filesystem
version: "1.0"
- id: discord
version: "1.0"
sandbox:
type: gvisor
networkPolicy: internal-only
resources:
cpu: "2"
memory: "4Gi"
gpu: 0
```
### WorkflowTemplate (cluster-scoped)
Pre-approved workflow templates — the **only** thing agents can invoke. Each template defines input parameters, execution steps, output schema, and compliance requirements.
```yaml
apiVersion: kubeclaw.nousresearch.com/v1alpha1
kind: WorkflowTemplate
metadata:
name: search-memory
spec:
description: Search agent memory for relevant context
version: "1.0"
parameters:
- name: query
type: string
required: true
description: Search query string
- name: maxResults
type: integer
default: 10
minimum: 1
maximum: 50
steps:
- id: embed-query
action: llm-embed
input: "{{ .Params.query }}"
- id: vector-search
action: vector-db-query
input: "{{ .Steps.embed-query.output }}"
limit: "{{ .Params.maxResults }}"
- id: format-results
action: llm-complete
input: "Format these memory results: {{ .Steps.vector-search.output }}"
output:
type: structured
schema:
results:
type: array
items:
type: object
properties:
id: { type: string }
content: { type: string }
timestamp: { type: string }
relevance: { type: number }
compliance:
requiredFrameworks: []
dataClassification: internal
```
### CompliancePolicy (cluster-scoped)
Regulatory and security policies the judge evaluates workflow invocations against.
```yaml
apiVersion: kubeclaw.nousresearch.com/v1alpha1
kind: CompliancePolicy
metadata:
name: soc2-baseline
spec:
framework: SOC2
version: "2025"
rules:
- id: data-access-scope
severity: critical
description: Agent must not access data outside its designated scope
eval: check_data_access_scope(request)
onViolation: REJECT
- id: pii-handling
severity: high
description: PII must not be written to external channels
eval: check_pii_in_output(request)
onViolation: REJECT
- id: audit-logging
severity: medium
description: All workflow invocations must be logged
eval: check_audit_log(request)
onViolation: ESCALATE_HUMAN
escalation:
onReject: notify_human("Compliance violation: {{ .Rule.Description }}")
onError: route_to_human("Judge error: {{ .Error }}")
```
### Agent (namespaced)
Single agent deployment — the primary CRD users create.
```yaml
apiVersion: kubeclaw.nousresearch.com/v1alpha1
kind: Agent
metadata:
name: engineering
namespace: agents
spec:
identity:
displayName: "Engineering Agent"
description: "Software engineering and architecture work"
owner: "scott"
config:
agentConfigRef: standard-engineering
overrides:
resources:
cpu: "4"
memory: "8Gi"
providers:
- ref:
name: openclaw-gateway
model: deepseek-v4-flash:cloud
channels:
- ref:
name: engineering-discord
workflowTemplates:
- ref:
name: search-memory
- ref:
name: forgejo-pr-create
- ref:
name: file-write
compliancePolicies:
- ref:
name: soc2-baseline
judge:
enabled: true
providerRef:
name: openclaw-gateway
model: deepseek-v4-pro:cloud
```
### AgentGroup (namespaced)
Multi-agent orchestration with shared configs and per-member overrides.
```yaml
apiVersion: kubeclaw.nousresearch.com/v1alpha1
kind: AgentGroup
metadata:
name: content-team
namespace: agents
spec:
sharedConfig:
agentConfigRef: standard-content
providers:
- ref:
name: openclaw-gateway
members:
- name: writer
overrides:
channels:
- ref:
name: content-discord
- name: editor
overrides:
channels:
- ref:
name: editorial-slack
- name: publisher
overrides:
channels:
- ref:
name: web-publish-discord
```
## Security Model
```
┌─────────────────────────────────────────────────────────────────────┐
│ Agent Attack Surface │
│ │
│ Available tools: [template_workflow] │
│ Not available: exec, write, web_search, sessions_*, forgejo_*, │
│ lean_experiment_*, cron, discord_send_raw │
│ │
│ Judge validates every template_workflow call against: │
│ ├─ WorkflowTemplate spec (schema, allowed steps) │
│ ├─ CompliancePolicy rules (SOC2/HIPAA/GDPR per-rule checks) │
│ ├─ Agent-level whitelists (allowed templates, data boundaries) │
│ └─ Regulatory framework system prompt │
│ │
│ Verdicts: │
│ ├─ APPROVE → workflow executor runs the steps │
│ ├─ REJECT → agent notified, human escalates if needed │
│ └─ ESCALATE_HUMAN → routed to Scott with structured dispute │
└─────────────────────────────────────────────────────────────────────┘
```
### Threat Model
| Threat | Mitigation |
|--------|-----------|
| Agent jailbreaks via prompt injection | Judge is separate model with no task-completion pressure; single-tool surface limits blast radius |
| Lateral movement between agents | Per-agent namespaces, NetworkPolicy isolation, strict Channel bindings |
| Privilege escalation via template params | WorkflowTemplate parameter validation, CompliancePolicy runtime checks |
| Data exfiltration via channels | Channel-scoped output validation, audit logging on every workflow step |
| Credential leak via log output | Secrets never injected into agent context; only referenced via `apiKeyRef` |
| Denial of service via resource exhaustion | Per-Agent resource limits, HPA on AgentGroup, OpenClaw gateway rate limiting |
## Project Structure
```
kubeclaw/
├── README.md # This file
├── config/
│ ├── crd/ # CRD YAML definitions
│ │ ├── provider.yaml
│ │ ├── channel.yaml
│ │ ├── agentconfig.yaml
│ │ ├── workflowtemplate.yaml
│ │ ├── compliancepolicy.yaml
│ │ ├── agent.yaml
│ │ └── agentgroup.yaml
│ └── samples/ # Example CRD manifests
│ ├── sample-agent.yaml
│ ├── sample-group.yaml
│ └── sample-workflow.yaml
├── charts/
│ └── kubeclaw/ # Helm chart
│ ├── Chart.yaml
│ ├── values.yaml
│ └── templates/
│ ├── crds.yaml
│ ├── controller.yaml
│ ├── rbac.yaml
│ ├── service.yaml
│ └── webhook.yaml
├── internal/
│ ├── controller/ # Reconciler logic
│ │ ├── agent_controller.go
│ │ ├── agentgroup_controller.go
│ │ └── suite_test.go
│ ├── judge/ # Judge integration
│ │ ├── client.go
│ │ ├── verifier.go
│ │ └── types.go
│ └── webhook/ # Admission webhooks
│ ├── agent_validator.go
│ └── workflow_validator.go
├── docs/
│ ├── architecture.md # Deep architecture doc
│ ├── development.md # Dev setup guide
│ └── operator-lifecycle.md # Production operations
├── main.go # Entry point
├── go.mod
├── go.sum
├── Makefile
└── Dockerfile
```
## Quick Start
```bash
# Prerequisites
# - Go 1.22+
# - kubectl
# - kind or minikube (for local dev)
# - helm
# Clone the monorepo
git clone https://os.scoble.me/forgejo/marcus/scobleclaw.git
cd scobleclaw/kubeclaw
# Install CRDs
make install
# Run controller locally (against current kubecontext)
make run
# Deploy to cluster via Helm
helm install kubeclaw ./charts/kubeclaw --namespace kubeclaw-system --create-namespace
# Create a sample agent
kubectl apply -f config/samples/sample-agent.yaml
```
## Development
See [docs/development.md](docs/development.md) for detailed setup and workflow.
### Prerequisites
- Go 1.22+
- `controller-gen` (from sigs.k8s.io/controller-tools)
- `kustomize`
- `helm`
- Access to an OpenClaw gateway or compatible LLM API
### Commands
```bash
make generate # Generate CRD deep copy and client code
make manifests # Generate CRD YAML manifests
make test # Run unit and integration tests
make run # Run controller locally
make docker-build # Build controller image
make helm-package # Package Helm chart
```
## CRD Inventory
| CRD | Scope | Purpose |
|-----|-------|---------|
| Provider | Cluster | LLM provider configs (API keys, models, pricing) |
| Channel | Namespaced | Platform channel bindings (Discord, Slack, Telegram) |
| AgentConfig | Namespaced | Reusable agent configuration templates |
| WorkflowTemplate | Cluster | Pre-approved workflow templates (only thing agents can invoke) |
| CompliancePolicy | Cluster | Security/compliance evaluation rules for judge |
| Agent | Namespaced | Single agent deployment |
| AgentGroup | Namespaced | Multi-agent orchestration with shared configs |
## Integration: Agent Judge
This project integrates directly with the [Agent Judge architecture](https://os.scoble.me/forgejo/marcus/scobleclaw/issues/7):
- The **judge sidecar** implements the Judge protocol specified in `agent-judge-architecture.md`
- Risk classification from that architecture maps to CompliancePolicy CRD rules
- The `template_workflow` tool is the enforcement point — agents can only invoke what the judge has approved
- Admission webhooks validate CRD changes at the API server level (pre-commit)
- The judge validates at runtime (post-commit, pre-execution)
## License
MIT © Scott Scoble / Greybeard Holdings, LLC

172
kubeclaw/docs/architecture.md Normal file
View file

@ -0,0 +1,172 @@
# kubeclaw Architecture
> **Status:** Draft v0.1
> **Author:** @agent-cos
> **Date:** 2026-05-23
---
## 1. System Context
kubeclaw is a Kubernetes operator that runs Scobleclaw agents as first-class cluster workloads. It is part of the Scobleclaw ecosystem and integrates with:
- **OpenClaw Gateway** — LLM API gateway providing model access, rate limiting, and credential management
- **Agent Judge** — authorization layer that validates every action before execution
- **Scobleclaw dispatch system** — higher-level orchestration that may trigger agent deployments
## 2. Operator Pattern
kubeclaw follows standard Kubernetes operator patterns using the `controller-runtime` library:
```
WATCH: CRD events (Agent, AgentGroup, etc.)
│
▼
RECONCILE: desired state → running state
│
├── CREATE: Build Agent Pod spec, inject config/secrets, wire sidecars
├── UPDATE: Diff current vs desired → patch deployment
└── DELETE: Tear down gracefully, flush audit logs
```
### 2.1 Reconciler Loop
Each CRD has a dedicated reconciler. The primary flow:
1. **Agent Reconciler** watches `Agent` resources
2. On create/update, it builds a `Pod` spec with:
- Main agent container (OpenClaw agent runtime)
- Judge sidecar (deepseek-v4-pro for authorization validation)
- Workflow executor sidecar (runs approved template steps)
- ConfigMap references for agent config
- Secret references for API keys (never injected into agent context)
3. On AgentConfig reference change, it triggers re-reconciliation of all Agents using that config
4. On Provider or CompliancePolicy change, it triggers rolling update of affected agents
### 2.2 Sidecar Architecture
Each Agent Pod has three containers:
#### Actor Agent Container
- Runs the OpenClaw agent process
- Has **exactly one tool**: `template_workflow(workflowTemplateRef, params)`
- No network access except to judge sidecar (localhost) and OpenClaw gateway
- No filesystem write access outside `/workspace` (tmpfs)
- No access to K8s API
#### Judge Sidecar
- Runs a separate LLM model instance (deepseek-v4-pro or higher)
- Implements the Judge protocol from `agent-judge-architecture.md`
- Evaluates every `template_workflow` call against:
- WorkflowTemplate spec
- CompliancePolicy rules
- Agent-level whitelists
- Returns APPROVE / REJECT / ESCALATE_HUMAN
#### Workflow Executor Sidecar
- Runs approved workflow steps
- Each step is a self-contained action with structured input/output
- All steps are logged to the audit trail (persistent volume + stdout)
## 3. API Design
### 3.1 CRD Group
- **Group:** `kubeclaw.nousresearch.com`
- **Version:** `v1alpha1` (initial, will graduate to v1)
- **Scope:** Mix of cluster-scoped and namespaced
### 3.2 Validation
- **CRD schema validation** — OpenAPI v3 schemas on each CRD for structural validation
- **Admission webhooks** — `ValidatingWebhookConfiguration` for semantic validation (e.g., provider must exist, channel must be reachable)
- **Status subresource** — Each CRD has a `.status` block with observed state, conditions, and last reconciliation
### 3.3 Status Reporting
All CRDs use standard Kubernetes status conditions:
```yaml
status:
conditions:
- type: Ready
status: "True"
reason: AgentPodRunning
message: "Agent pod engineering-7d8f9c is running"
lastTransitionTime: "2026-05-23T19:00:00Z"
- type: JudgeConnected
status: "True"
reason: JudgeResponding
message: "Judge sidecar is online and responding"
observedGeneration: 3
agentPodRef:
name: engineering-7d8f9c
namespace: agents
lastWorkflowCount: 142
lastJudgeVerdict: APPROVE
```
## 4. Security Architecture
### 4.1 Pod Security
- **PodSecurityContext**: Run as non-root user, read-only root filesystem
- **Seccomp profile**: Runtime default
- **AppArmor**: Kubeclaw-specific profile limiting agent to `template_workflow` only
- **NetworkPolicy**: Agents can only talk to judge sidecar (localhost) and OpenClaw gateway
- **Resource limits**: Hard CPU/memory limits per Agent spec
### 4.2 Secret Management
- API keys are standard K8s Secrets referenced by `apiKeyRef` in Provider and Channel CRDs
- Secrets are **never** injected into the agent container's environment
- Only the judge sidecar and workflow executor have access to resolved credentials
- The actor agent container never sees raw API keys
### 4.3 RBAC
```yaml
# Minimal RBAC for the controller
- apiGroups: ["kubeclaw.nousresearch.com"]
resources: ["agents", "agentgroups", "providers", "channels", "workflowtemplates", "compliancepolicies"]
verbs: ["get", "list", "watch", "patch"]
- apiGroups: [""]
resources: ["pods", "configmaps", "secrets", "events"]
verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["get", "list", "watch", "create", "update", "delete"]
```
## 5. Operational Concerns
### 5.1 Observability
- **Metrics**: Standard controller-runtime metrics (ReconcileTotal, WorkQueueDepth) + custom:
- `kubeclaw_agent_count` — number of agent deployments
- `kubeclaw_judge_verdicts_total` — verdict counts by type (APPROVE/REJECT/ESCALATE)
- `kubeclaw_workflow_invocations_total` — workflow calls by template
- **Logging**: Structured JSON logs, per-reconciliation trace IDs
- **Tracing**: OpenTelemetry integration for end-to-end workflow tracing
### 5.2 Reconciliation Performance
- Agent pod creation is asynchronous; controller creates Deployment, not bare Pod
- Large AgentGroups are reconciled via batch operations with worker pool
- Requeue on conflict with exponential backoff (100ms → 30s)
- Watch caching via informer with 10-minute resync period
### 5.3 Upgrade Strategy
- CRD versioning with conversion webhooks for backward compatibility
- Helm chart with `--upgrade` support
- Canary deployments for controller upgrades
- Agent pods use standard rolling update strategy
## 6. Future Enhancements
- [ ] **Horizontal Pod Autoscaling** for AgentGroups based on workflow queue depth
- [ ] **Agent-to-agent networking** via Service Mesh (Istio/Linkerd) for multi-agent coordination
- [ ] **Workflow DAG execution** — complex multi-step workflows with branching
- [ ] **Compliance attestation** — generate compliance reports for SOC2/HIPAA/GDPR audits
- [ ] **Web UI** — Dashboard for viewing agent status, workflow history, judge decisions

151
kubeclaw/docs/development.md Normal file
View file

@ -0,0 +1,151 @@
# Development Setup
> **Status:** Draft v0.1
> **Author:** @agent-cos
---
## Prerequisites
- **Go 1.22+** — `go version` should show go1.22 or later
- **kubectl** — configured with a cluster context (kind/minikube for local dev)
- **helm** — v3.x for chart deployment
- **controller-gen** — from `sigs.k8s.io/controller-tools`
- **kustomize** — built into `kubectl kustomize` or standalone v4+
- **Docker** — for building controller images
- **kind** or **minikube** — for local Kubernetes clusters
- **OpenClaw gateway** or compatible LLM API endpoint
## Quick Start
### 1. Clone the Repository
```bash
git clone https://os.scoble.me/forgejo/marcus/scobleclaw.git
cd scobleclaw/kubeclaw
```
### 2. Create a Local Cluster
```bash
kind create cluster --name kubeclaw-dev
```
Or use an existing cluster:
```bash
kubectl config get-contexts
```
### 3. Install CRDs
```bash
make install
```
This applies all CRD YAML manifests to the cluster.
### 4. Run the Controller Locally
```bash
# In one terminal:
make run
# The controller connects to the current kubecontext
# Logs show reconciliation of CRD events
```
### 5. Deploy an Agent
```bash
# Apply a Provider first (no real credentials in dev)
kubectl apply -f config/samples/sample-provider.yaml
# Apply an Agent
kubectl apply -f config/samples/sample-agent.yaml
# Watch the agent pod come up
kubectl get pods -n agents -w
```
## Project Commands
| Command | Description |
|---------|-------------|
| `make generate` | Generate CRD deep copy and client code |
| `make manifests` | Generate CRD YAML manifests from Go types |
| `make install` | Apply CRDs to cluster |
| `make uninstall` | Remove CRDs from cluster |
| `make run` | Run controller locally |
| `make test` | Run unit and integration tests |
| `make docker-build` | Build controller Docker image |
| `make docker-push` | Push controller image to registry |
| `make helm-package` | Package Helm chart |
| `make deploy` | Deploy controller via Helm |
## Testing
### Unit Tests
```bash
make test
```
Tests cover:
- Reconciler logic (mock K8s client)
- Judge verifier (mock LLM responses)
- Webhook validation
- CRD schema validation
### Integration Tests
```bash
# Requires a running cluster and controller
make test-integration
```
Uses envtest (controller-runtime's test environment) for in-process API server testing.
### End-to-End Tests
```bash
# Requires a full cluster with all components
make test-e2e
```
## Local Development Tips
### Watch Controller Logs
```bash
kubectl logs -n kubeclaw-system deployment/kubeclaw-controller -f
```
### Debug Agent Pods
```bash
# Check pod status
kubectl describe pod -n agents <agent-name>
# Check judge sidecar logs
kubectl logs -n agents <agent-pod> -c judge
# Check workflow executor logs
kubectl logs -n agents <agent-pod> -c workflow-executor
```
### Hot Reload
For rapid iteration on CRD handler logic without rebuilding Docker images:
```bash
make run # Restart on ctrl+c and re-run after code changes
```
## Architecture Notes
- The controller uses `controller-runtime`'s builder pattern: `NewControllerManagedBy(mgr).For(&Agent{}).Owns(&corev1.Pod{})`
- CRD types are in `api/v1alpha1/`
- Reconcilers are in `internal/controller/`
- The judge is implemented as a gRPC or HTTP sidecar
- Workflow templates are validated server-side before agent creation