scobleclaw/scoble-policy/audit-sink/provisioning/README.md

scoble-kafka provisioning runbook

Date written: 2026-06-07 Owner: @agent-cos Related: marcus/engineering#43, marcus/engineering#63 (closed), marcus/engineering#62

Summary

Single-node Kafka (KRaft, no ZooKeeper) running in scobleclaw namespace. Topic scobleclaw.audit.decisions exists with 3 partitions, RF=1. End-to-end produce+consume verified from a pod in the scobleclaw namespace. scoble_audit_writer role exists in CNPG-managed Postgres, scoble_policy_audit table is the schema.sql shape (agent_id/kamelet_ref/decision/etc.).

State (as of 2026-06-07 23:55 UTC)

Resource	Status	Notes
Strimzi operator	1/1 Running	Healthy, leader-elected. Watches kafka,scobleclaw,scoble-policy.
Kafka CR scoble-kafka	NotReady	Strimzi operator's admin client can't talk to the broker on port 9091 (see #62). Broker IS responding on 9092 from in-cluster clients. Non-fatal.
KafkaNodePool dual-role	Created	1 replica, dual-role (controller+broker), 5Gi PVC on nfs-csi.
KafkaTopic scobleclaw.audit.decisions	Created manually	3 partitions, RF=1. TopicId Bn0QeTBWQQeaqVYDyusBTw. Entity operator not deployed (#62 blocks).
scoble-kafka Service (ExternalName)	Created	Aliases the bootstrap service so the audit-sink's scoble-kafka:9092 reference resolves.
scrole Secret (basic-auth)	Created	database/postgres-scobleclaw-scrole, type kubernetes.io/basic-auth.
forgejo/umami Secrets (basic-auth)	Created	database/postgres-scobleclaw-{forgejo,umami}.
scoble_audit_writer Postgres role	CREATED	Via CNPG managed.roles. Password from postgres-scobleclaw-scrole.
forgejo/umami Postgres roles	CREATED	Were missing from DB before this session — silent breakage in CNPG managed.roles controller due to wrong secret format.
scoble_policy_audit table	Created	Shape: agent_id/kamelet_ref/decision/reasons/rule_id/request_id (per schema.sql). Old shape (decision_id/actor/action/resource/verdict/metadata) was dropped.
scoble-policy-pg-creds Secret	Created	scoble-policy/scoble-policy-pg-creds, type Opaque, keys username/password/jdbc-url.
scrole GRANTs on scoble_policy_audit	Applied	SELECT, INSERT; USAGE+SELECT on the id sequence.

Files in this directory

File	Purpose
scoble-kafka.yaml	Kafka CR (KRaft single-node) + KafkaNodePool CR (dual-role, 5Gi nfs-csi PVC). Apply as a single file (multi-document YAML).
scoble-kafka-topic.yaml	KafkaTopic CR for scobleclaw.audit.decisions (3 partitions, RF=1, 7-day retention).
scoble-kafka-service.yaml	scoble-kafka Service (ExternalName) aliasing the bootstrap service.
secret-forgejo.yaml	Per-role basic-auth secret for forgejo (used by CNPG managed.roles).
secret-umami.yaml	Per-role basic-auth secret for umami.
secret-scrole.yaml	Per-role basic-auth secret for scoble_audit_writer.
secret-scoble-policy-pg-creds.yaml	Opaque secret in scoble-policy namespace with scrole credentials, for the audit-sink Integration to mount.
migrate.sql	DROP+CREATE+GRANT for the scoble_policy_audit table. Idempotent (DROPs first).
apply-schema-pod.yaml	One-shot pod that runs migrate.sql against the in-cluster CNPG using the superuser secret.
README.md	This file.
strimzi-0.45.2.tar.gz and strimzi-0.45.2/	Upstream Strimzi 0.45.2 install bundle (gitignored). Re-download from the upstream release.

Apply

export KUBECONFIG=~/.kube/config

# 1. Kafka cluster
kubectl apply -f audit-sink/provisioning/scoble-kafka.yaml

# 2. Topic (currently applied manually because entity operator is blocked by #62)
kubectl apply -f audit-sink/provisioning/scoble-kafka-topic.yaml

# 3. Service alias
kubectl apply -f audit-sink/provisioning/scoble-kafka-service.yaml

# 4. CNPG per-role basic-auth secrets
kubectl apply -f audit-sink/provisioning/secret-forgejo.yaml
kubectl apply -f audit-sink/provisioning/secret-umami.yaml
kubectl apply -f audit-sink/provisioning/secret-scrole.yaml

# 5. Cluster spec (patch existing postgres-scobleclaw Cluster to point each
# managed role at its own basic-auth secret). Already applied; included for
# re-deploy reference.
kubectl patch cluster -n database postgres-scobleclaw --type=json -p '[
  {"op":"replace","path":"/spec/managed/roles/0/passwordSecret/name","value":"postgres-scobleclaw-forgejo"},
  {"op":"replace","path":"/spec/managed/roles/1/passwordSecret/name","value":"postgres-scobleclaw-umami"},
  {"op":"replace","path":"/spec/managed/roles/2/passwordSecret/name","value":"postgres-scobleclaw-scrole"}
]'

# 6. Postgres schema (drops old shape, creates new shape, applies grants)
kubectl create configmap -n scobleclaw scrole-schema --from-file=migrate.sql=audit-sink/provisioning/migrate.sql
kubectl apply -f audit-sink/provisioning/apply-schema-pod.yaml
kubectl logs -n scobleclaw apply-schema
kubectl delete pod -n scobleclaw apply-schema --force --grace-period=0

# 7. Audit-sink creds in scoble-policy namespace
kubectl apply -f audit-sink/provisioning/secret-scoble-policy-pg-creds.yaml

Verify

# Topic exists
kubectl run -n scobleclaw verify-topic --rm --restart=Never --image=quay.io/strimzi/kafka:0.45.2-kafka-3.9.2 \
  -- /opt/kafka/bin/kafka-topics.sh --list \
  --bootstrap-server scoble-kafka:9092

# Role + table + INSERT roundtrip
kubectl run -n scobleclaw verify-pg --rm --restart=Never --image=postgres:17-alpine \
  -- sh -c "apk add --no-cache postgresql-client >/dev/null 2>&1; \
    PGPASSWORD='<the-scrole-pw>' psql \
    -h postgres-scobleclaw-rw.database.svc.cluster.local -U scoble_audit_writer -d app \
    -c \"INSERT INTO scoble_policy_audit (timestamp, agent_id, kamelet_ref, decision, reasons, request_id) VALUES (now(), 'verify', 'audit-sink', 'allow', '[]'::jsonb, 'verify-1') RETURNING id, agent_id, request_id;\""

Why the previous scoble-kafka Kafka CR was GC'd

Strimzi 0.45.2 still defaults to ZooKeeper mode. A Kafka CR with no spec.zookeeper block is invalid; the operator reconciles, marks it NotReady with InvalidResourceException: The .spec.zookeeper section of the Kafka custom resource is missing, and the next reconcile GC's the assembly (the Assembly scoble-kafka or some parts of it will be deleted by garbage collection log line). The current manifests use the proper KRaft pattern.

Why CNPG managed.roles were broken (root cause from #63)

spec.managed.roles[] references a passwordSecret per role. CNPG's GetUserPasswordFromSecret (in internal/management/utils/secrets.go) expects each secret to be type: kubernetes.io/basic-auth with username and password keys (literal). The cluster was using a single shared postgres-users Secret with role-name-suffixed keys (forgejo-password, etc.) — that doesn't match the function's expectations, and the operator kept logging username key doesn't exist inside the secret. As a result, forgejo/umami/scoble_audit_writer were silently missing from the database even though they were defined in spec.

Fix: create per-role basic-auth secrets and patch the cluster spec to point each role at its own secret.

Known issues (separate tickets)

• #62: Strimzi operator's admin client can't reach the bootstrap service on port 9091. The operator keeps rolling the broker every few minutes. Workaround: create topics manually via kafka-topics.sh from inside the broker pod. Recommended fix: move the operator into the same namespace as the Kafka cluster.
• #64: cross-namespace pod-to-pod IP routing broken (Host unreachable from kafka ns to broker pod IP 10.42.0.x). Service-virtual-IP routing works. Doesn't block the audit-sink (same namespace, goes through the service).