The Strimzi cluster operator's admin client was timing out with
'No resolvable bootstrap urls given in bootstrap.servers' and
'KafkaQuorumCheck cannot be initialised' errors. Investigation showed:
- Operator pods scheduled to n97-node-2 (the worker) have broken outbound
network connectivity: cannot reach kube-dns (10.43.0.10:53), in-cluster
service IPs, or cross-node pod IPs.
- Operator pods scheduled to n97-mini (the control plane) have full network
connectivity and reconcile both Kafka clusters successfully.
- Fresh busybox pods in the same namespace can reach everything — the bug
is specific to operator pods (and possibly pods that started when the
operator was at full leader+controllers state).
This is the same pod-specific K3s networking bug that bit the camel-k
operator in marcus/engineering#60, but the Strimzi operator can't use
the camel-k fix (hostNetwork: true + ClusterFirstWithHostNet) because
Strimzi hard-codes its health probe port to 8080, which is already bound
on both nodes (kuma + something else on n97-mini, IPv6 wildcard on
n97-node-2).
Fix: pin the operator to n97-mini via spec.nodeName. This is a workaround
for the K3s pod-specific network bug; the right fix is in K3s itself.
When that bug is fixed upstream, the nodeName can be removed.
Changes:
- strimzi-operator-pinned.yaml: complete operator deployment manifest with
nodeName, hostAliases for the bootstrap FQDN, no UseKRaft feature gate
(0.45.2 doesn't support it), replicaCount: 1 with Recreate strategy.
- Recreate strategy is mandatory: any rolling update would still hit the
same bug because the new pod would still be scheduled to n97-node-2
during the brief window before nodeName takes effect.
Verification (2026-06-08 00:56 UTC):
- Operator pod 'strimzi-cluster-operator-9bd6cbcd6-tqgjb' on n97-mini
is the leader, 1/1 Running, 0 restarts.
- Both Kafka CRs (kafka/kafka, scobleclaw/scoble-kafka) report READY=True.
- E2E test: produced 8 messages via scoble-kafka:9092 alias, all consumed
back successfully.
Refs: marcus/engineering#62
Resolves the #63 root cause: CNPG managed.roles expects each role to reference
its own type=kubernetes.io/basic-auth Secret with literal 'username'/'password'
keys (see cloudnative-pg/internal/management/utils/secrets.go). The previous
shared 'postgres-users' Secret with role-name-suffixed keys (forgejo-password,
umami-password, scoble_audit_writer-password) was never going to reconcile, so
forgejo/umami had been silently missing from the database for the entire
lifetime of the cluster.
Changes:
- secret-{forgejo,umami,scrole}.yaml: per-role basic-auth Secrets in 'database' ns.
forgejo/umami passwords stay at the existing <change-me> placeholder. scrole
gets a new random 32-byte password.
- secret-scoble-policy-pg-creds.yaml: Opaque Secret in 'scoble-policy' ns with
username/password/jdbc-url for the audit-sink Integration to mount.
- migrate.sql: drops the previous scoble_policy_audit (decision_id/actor/action
shape) and recreates the schema.sql shape (agent_id/kamelet_ref/decision/etc.)
with the right indexes, check constraint, and scrole GRANTs.
- apply-schema-pod.yaml: one-shot Pod that runs migrate.sql using the CNPG
superuser Secret as credential source.
- schema.sql: now also documents the scrole GRANTs needed (idempotent).
- README.md: updated with apply+verify steps, #62/#64 known issues, and the
full diagnosis of #63.
End-to-end verified 2026-06-07 23:55 UTC:
- All 3 managed roles (forgejo, umami, scoble_audit_writer) exist in PG.
- scrole can SELECT/INSERT on scoble_policy_audit.
- A 'smoke-1' row is in the table; a Kafka message at scoble-kafka:9092 in the
audit-sink shape is consumable end-to-end.
- Legacy 'postgres-users' secret deleted (no longer referenced by the cluster).
Refs: marcus/engineering#43, marcus/engineering#63
- pool-manager/main.go: serve HTTP on :3990 with /healthz that does a 2s-context
pgxpool.Ping; returns 200 OK when Postgres is reachable, 503 otherwise.
Runs alongside the existing signal-blocking main loop.
- pool-manager/Dockerfile: distroless static-debian12 image exposing :3990.
- pool-manager/.gitignore: skip built pool-manager binary and *.tar.
Refs marcus/engineering#56Closesmarcus/engineering#56
- pgqueue package: Go wrapper around the SQL primitive
(enqueue, claim, heartbeat, ack, nack, janitor, listener)
- examples/agent-dispatcher: reference worker implementation
showing end-to-end flow with LISTEN/NOTIFY + heartbeats
- All 5 Go tests pass against live pool_manager DB
Reference: skills/data-stack-conventions.md
Migration: devops/migrations/2026-06-01_pgqueue_primitive.sql
- Activate / Deactivate via flox activate --detach (with graceful SIGTERM→SIGKILL)
- Non-privileged IsAlive process check with Linux /proc fallback
- Dev-mode stub when flox is missing or FLOXENV_DEV_MODE=1
- Unit and integration tests
Closes#16
Define kubeclaw as a Kubernetes operator that runs Scobleclaw AI agents
as first-class K8s workloads with CRDs, controller loops, and
security-hardened operator patterns.
Key deliverables in this commit:
- README.md — Project spec with CRD reference, architecture diagram,
security model, threat model, and quick-start guide
- docs/architecture.md — Deep architecture doc covering operator pattern,
sidecar design, API design, security architecture, and operations
- docs/development.md — Dev setup guide with prerequisites, commands,
testing, and local dev tips
Integrates with:
- scobleclaw#7 (Agent-Judge authorization layer)
- OpenClaw gateway for LLM provider management
- Standard Kubernetes operator patterns
Closes scobleclaw#8
Analyzes coach issues #24 (trust), #25 (engineering), and #26 (vault) — all
scored 5/10 for repetitive behavior patterns. Identifies common root cause:
all agents share deepseek-v4-flash:cloud model with no loop-detection
guardrails or iteration budgets.
Key findings:
- All 3 critiques are VALID — real systemic issue, not coach noise
- Root cause: single shallow model across all agents + no loop-breaker skills
- Coach uses same model to critique itself (self-referential grading)
- Three distinct failure patterns: yield loops, planning loops, cron loops
- Recommended fixes: guardrail skills, staleness detection, model tiering
Closes#9
