kubeclaw: New project — OpenClaw operator for Kubernetes
Define kubeclaw as a Kubernetes operator that runs Scobleclaw AI agents as first-class K8s workloads with CRDs, controller loops, and security-hardened operator patterns. Key deliverables in this commit: - README.md — Project spec with CRD reference, architecture diagram, security model, threat model, and quick-start guide - docs/architecture.md — Deep architecture doc covering operator pattern, sidecar design, API design, security architecture, and operations - docs/development.md — Dev setup guide with prerequisites, commands, testing, and local dev tips Integrates with: - scobleclaw#7 (Agent-Judge authorization layer) - OpenClaw gateway for LLM provider management - Standard Kubernetes operator patterns Closes scobleclaw#8
This commit is contained in:
agent-cos
parent
d74bb3852a
commit
3264f9fe66
3 changed files with 801 additions and 0 deletions
172
kubeclaw/docs/architecture.md
Normal file
172
kubeclaw/docs/architecture.md
Normal file
|
|
@ -0,0 +1,172 @@
|
|||
# kubeclaw Architecture
|
||||
|
||||
> **Status:** Draft v0.1
|
||||
> **Author:** @agent-cos
|
||||
> **Date:** 2026-05-23
|
||||
|
||||
---
|
||||
|
||||
## 1. System Context
|
||||
|
||||
kubeclaw is a Kubernetes operator that runs Scobleclaw agents as first-class cluster workloads. It is part of the Scobleclaw ecosystem and integrates with:
|
||||
|
||||
- **OpenClaw Gateway** — LLM API gateway providing model access, rate limiting, and credential management
|
||||
- **Agent Judge** — authorization layer that validates every action before execution
|
||||
- **Scobleclaw dispatch system** — higher-level orchestration that may trigger agent deployments
|
||||
|
||||
## 2. Operator Pattern
|
||||
|
||||
kubeclaw follows standard Kubernetes operator patterns using the `controller-runtime` library:
|
||||
|
||||
```
|
||||
WATCH: CRD events (Agent, AgentGroup, etc.)
|
||||
│
|
||||
▼
|
||||
RECONCILE: desired state → running state
|
||||
│
|
||||
├── CREATE: Build Agent Pod spec, inject config/secrets, wire sidecars
|
||||
├── UPDATE: Diff current vs desired → patch deployment
|
||||
└── DELETE: Tear down gracefully, flush audit logs
|
||||
```
|
||||
|
||||
### 2.1 Reconciler Loop
|
||||
|
||||
Each CRD has a dedicated reconciler. The primary flow:
|
||||
|
||||
1. **Agent Reconciler** watches `Agent` resources
|
||||
2. On create/update, it builds a `Pod` spec with:
|
||||
- Main agent container (OpenClaw agent runtime)
|
||||
- Judge sidecar (deepseek-v4-pro for authorization validation)
|
||||
- Workflow executor sidecar (runs approved template steps)
|
||||
- ConfigMap references for agent config
|
||||
- Secret references for API keys (never injected into agent context)
|
||||
3. On AgentConfig reference change, it triggers re-reconciliation of all Agents using that config
|
||||
4. On Provider or CompliancePolicy change, it triggers rolling update of affected agents
|
||||
|
||||
### 2.2 Sidecar Architecture
|
||||
|
||||
Each Agent Pod has three containers:
|
||||
|
||||
#### Actor Agent Container
|
||||
- Runs the OpenClaw agent process
|
||||
- Has **exactly one tool**: `template_workflow(workflowTemplateRef, params)`
|
||||
- No network access except to judge sidecar (localhost) and OpenClaw gateway
|
||||
- No filesystem write access outside `/workspace` (tmpfs)
|
||||
- No access to K8s API
|
||||
|
||||
#### Judge Sidecar
|
||||
- Runs a separate LLM model instance (deepseek-v4-pro or higher)
|
||||
- Implements the Judge protocol from `agent-judge-architecture.md`
|
||||
- Evaluates every `template_workflow` call against:
|
||||
- WorkflowTemplate spec
|
||||
- CompliancePolicy rules
|
||||
- Agent-level whitelists
|
||||
- Returns APPROVE / REJECT / ESCALATE_HUMAN
|
||||
|
||||
#### Workflow Executor Sidecar
|
||||
- Runs approved workflow steps
|
||||
- Each step is a self-contained action with structured input/output
|
||||
- All steps are logged to the audit trail (persistent volume + stdout)
|
||||
|
||||
## 3. API Design
|
||||
|
||||
### 3.1 CRD Group
|
||||
|
||||
- **Group:** `kubeclaw.nousresearch.com`
|
||||
- **Version:** `v1alpha1` (initial, will graduate to v1)
|
||||
- **Scope:** Mix of cluster-scoped and namespaced
|
||||
|
||||
### 3.2 Validation
|
||||
|
||||
- **CRD schema validation** — OpenAPI v3 schemas on each CRD for structural validation
|
||||
- **Admission webhooks** — `ValidatingWebhookConfiguration` for semantic validation (e.g., provider must exist, channel must be reachable)
|
||||
- **Status subresource** — Each CRD has a `.status` block with observed state, conditions, and last reconciliation
|
||||
|
||||
### 3.3 Status Reporting
|
||||
|
||||
All CRDs use standard Kubernetes status conditions:
|
||||
|
||||
```yaml
|
||||
status:
|
||||
conditions:
|
||||
- type: Ready
|
||||
status: "True"
|
||||
reason: AgentPodRunning
|
||||
message: "Agent pod engineering-7d8f9c is running"
|
||||
lastTransitionTime: "2026-05-23T19:00:00Z"
|
||||
- type: JudgeConnected
|
||||
status: "True"
|
||||
reason: JudgeResponding
|
||||
message: "Judge sidecar is online and responding"
|
||||
observedGeneration: 3
|
||||
agentPodRef:
|
||||
name: engineering-7d8f9c
|
||||
namespace: agents
|
||||
lastWorkflowCount: 142
|
||||
lastJudgeVerdict: APPROVE
|
||||
```
|
||||
|
||||
## 4. Security Architecture
|
||||
|
||||
### 4.1 Pod Security
|
||||
|
||||
- **PodSecurityContext**: Run as non-root user, read-only root filesystem
|
||||
- **Seccomp profile**: Runtime default
|
||||
- **AppArmor**: Kubeclaw-specific profile limiting agent to `template_workflow` only
|
||||
- **NetworkPolicy**: Agents can only talk to judge sidecar (localhost) and OpenClaw gateway
|
||||
- **Resource limits**: Hard CPU/memory limits per Agent spec
|
||||
|
||||
### 4.2 Secret Management
|
||||
|
||||
- API keys are standard K8s Secrets referenced by `apiKeyRef` in Provider and Channel CRDs
|
||||
- Secrets are **never** injected into the agent container's environment
|
||||
- Only the judge sidecar and workflow executor have access to resolved credentials
|
||||
- The actor agent container never sees raw API keys
|
||||
|
||||
### 4.3 RBAC
|
||||
|
||||
```yaml
|
||||
# Minimal RBAC for the controller
|
||||
- apiGroups: ["kubeclaw.nousresearch.com"]
|
||||
resources: ["agents", "agentgroups", "providers", "channels", "workflowtemplates", "compliancepolicies"]
|
||||
verbs: ["get", "list", "watch", "patch"]
|
||||
- apiGroups: [""]
|
||||
resources: ["pods", "configmaps", "secrets", "events"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "delete", "patch"]
|
||||
- apiGroups: ["networking.k8s.io"]
|
||||
resources: ["networkpolicies"]
|
||||
verbs: ["get", "list", "watch", "create", "update", "delete"]
|
||||
```
|
||||
|
||||
## 5. Operational Concerns
|
||||
|
||||
### 5.1 Observability
|
||||
|
||||
- **Metrics**: Standard controller-runtime metrics (ReconcileTotal, WorkQueueDepth) + custom:
|
||||
- `kubeclaw_agent_count` — number of agent deployments
|
||||
- `kubeclaw_judge_verdicts_total` — verdict counts by type (APPROVE/REJECT/ESCALATE)
|
||||
- `kubeclaw_workflow_invocations_total` — workflow calls by template
|
||||
- **Logging**: Structured JSON logs, per-reconciliation trace IDs
|
||||
- **Tracing**: OpenTelemetry integration for end-to-end workflow tracing
|
||||
|
||||
### 5.2 Reconciliation Performance
|
||||
|
||||
- Agent pod creation is asynchronous; controller creates Deployment, not bare Pod
|
||||
- Large AgentGroups are reconciled via batch operations with worker pool
|
||||
- Requeue on conflict with exponential backoff (100ms → 30s)
|
||||
- Watch caching via informer with 10-minute resync period
|
||||
|
||||
### 5.3 Upgrade Strategy
|
||||
|
||||
- CRD versioning with conversion webhooks for backward compatibility
|
||||
- Helm chart with `--upgrade` support
|
||||
- Canary deployments for controller upgrades
|
||||
- Agent pods use standard rolling update strategy
|
||||
|
||||
## 6. Future Enhancements
|
||||
|
||||
- [ ] **Horizontal Pod Autoscaling** for AgentGroups based on workflow queue depth
|
||||
- [ ] **Agent-to-agent networking** via Service Mesh (Istio/Linkerd) for multi-agent coordination
|
||||
- [ ] **Workflow DAG execution** — complex multi-step workflows with branching
|
||||
- [ ] **Compliance attestation** — generate compliance reports for SOC2/HIPAA/GDPR audits
|
||||
- [ ] **Web UI** — Dashboard for viewing agent status, workflow history, judge decisions
|
||||
151
kubeclaw/docs/development.md
Normal file
151
kubeclaw/docs/development.md
Normal file
|
|
@ -0,0 +1,151 @@
|
|||
# Development Setup
|
||||
|
||||
> **Status:** Draft v0.1
|
||||
> **Author:** @agent-cos
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Go 1.22+** — `go version` should show go1.22 or later
|
||||
- **kubectl** — configured with a cluster context (kind/minikube for local dev)
|
||||
- **helm** — v3.x for chart deployment
|
||||
- **controller-gen** — from `sigs.k8s.io/controller-tools`
|
||||
- **kustomize** — built into `kubectl kustomize` or standalone v4+
|
||||
- **Docker** — for building controller images
|
||||
- **kind** or **minikube** — for local Kubernetes clusters
|
||||
- **OpenClaw gateway** or compatible LLM API endpoint
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. Clone the Repository
|
||||
|
||||
```bash
|
||||
git clone https://os.scoble.me/forgejo/marcus/scobleclaw.git
|
||||
cd scobleclaw/kubeclaw
|
||||
```
|
||||
|
||||
### 2. Create a Local Cluster
|
||||
|
||||
```bash
|
||||
kind create cluster --name kubeclaw-dev
|
||||
```
|
||||
|
||||
Or use an existing cluster:
|
||||
|
||||
```bash
|
||||
kubectl config get-contexts
|
||||
```
|
||||
|
||||
### 3. Install CRDs
|
||||
|
||||
```bash
|
||||
make install
|
||||
```
|
||||
|
||||
This applies all CRD YAML manifests to the cluster.
|
||||
|
||||
### 4. Run the Controller Locally
|
||||
|
||||
```bash
|
||||
# In one terminal:
|
||||
make run
|
||||
|
||||
# The controller connects to the current kubecontext
|
||||
# Logs show reconciliation of CRD events
|
||||
```
|
||||
|
||||
### 5. Deploy an Agent
|
||||
|
||||
```bash
|
||||
# Apply a Provider first (no real credentials in dev)
|
||||
kubectl apply -f config/samples/sample-provider.yaml
|
||||
|
||||
# Apply an Agent
|
||||
kubectl apply -f config/samples/sample-agent.yaml
|
||||
|
||||
# Watch the agent pod come up
|
||||
kubectl get pods -n agents -w
|
||||
```
|
||||
|
||||
## Project Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `make generate` | Generate CRD deep copy and client code |
|
||||
| `make manifests` | Generate CRD YAML manifests from Go types |
|
||||
| `make install` | Apply CRDs to cluster |
|
||||
| `make uninstall` | Remove CRDs from cluster |
|
||||
| `make run` | Run controller locally |
|
||||
| `make test` | Run unit and integration tests |
|
||||
| `make docker-build` | Build controller Docker image |
|
||||
| `make docker-push` | Push controller image to registry |
|
||||
| `make helm-package` | Package Helm chart |
|
||||
| `make deploy` | Deploy controller via Helm |
|
||||
|
||||
## Testing
|
||||
|
||||
### Unit Tests
|
||||
|
||||
```bash
|
||||
make test
|
||||
```
|
||||
|
||||
Tests cover:
|
||||
- Reconciler logic (mock K8s client)
|
||||
- Judge verifier (mock LLM responses)
|
||||
- Webhook validation
|
||||
- CRD schema validation
|
||||
|
||||
### Integration Tests
|
||||
|
||||
```bash
|
||||
# Requires a running cluster and controller
|
||||
make test-integration
|
||||
```
|
||||
|
||||
Uses envtest (controller-runtime's test environment) for in-process API server testing.
|
||||
|
||||
### End-to-End Tests
|
||||
|
||||
```bash
|
||||
# Requires a full cluster with all components
|
||||
make test-e2e
|
||||
```
|
||||
|
||||
## Local Development Tips
|
||||
|
||||
### Watch Controller Logs
|
||||
|
||||
```bash
|
||||
kubectl logs -n kubeclaw-system deployment/kubeclaw-controller -f
|
||||
```
|
||||
|
||||
### Debug Agent Pods
|
||||
|
||||
```bash
|
||||
# Check pod status
|
||||
kubectl describe pod -n agents <agent-name>
|
||||
|
||||
# Check judge sidecar logs
|
||||
kubectl logs -n agents <agent-pod> -c judge
|
||||
|
||||
# Check workflow executor logs
|
||||
kubectl logs -n agents <agent-pod> -c workflow-executor
|
||||
```
|
||||
|
||||
### Hot Reload
|
||||
|
||||
For rapid iteration on CRD handler logic without rebuilding Docker images:
|
||||
|
||||
```bash
|
||||
make run # Restart on ctrl+c and re-run after code changes
|
||||
```
|
||||
|
||||
## Architecture Notes
|
||||
|
||||
- The controller uses `controller-runtime`'s builder pattern: `NewControllerManagedBy(mgr).For(&Agent{}).Owns(&corev1.Pod{})`
|
||||
- CRD types are in `api/v1alpha1/`
|
||||
- Reconcilers are in `internal/controller/`
|
||||
- The judge is implemented as a gRPC or HTTP sidecar
|
||||
- Workflow templates are validated server-side before agent creation
|
||||
Loading…
Add table
Add a link
Reference in a new issue