From 4612b5e68d4083d75a7aa42ece62720f6fd5bde3 Mon Sep 17 00:00:00 2001 From: CoS Date: Sun, 7 Jun 2026 17:09:41 -0700 Subject: [PATCH] feat(audit-sink): provision scrole role, per-role CNPG secrets, schema, and creds Resolves the #63 root cause: CNPG managed.roles expects each role to reference its own type=kubernetes.io/basic-auth Secret with literal 'username'/'password' keys (see cloudnative-pg/internal/management/utils/secrets.go). The previous shared 'postgres-users' Secret with role-name-suffixed keys (forgejo-password, umami-password, scoble_audit_writer-password) was never going to reconcile, so forgejo/umami had been silently missing from the database for the entire lifetime of the cluster. Changes: - secret-{forgejo,umami,scrole}.yaml: per-role basic-auth Secrets in 'database' ns. forgejo/umami passwords stay at the existing placeholder. scrole gets a new random 32-byte password. - secret-scoble-policy-pg-creds.yaml: Opaque Secret in 'scoble-policy' ns with username/password/jdbc-url for the audit-sink Integration to mount. - migrate.sql: drops the previous scoble_policy_audit (decision_id/actor/action shape) and recreates the schema.sql shape (agent_id/kamelet_ref/decision/etc.) with the right indexes, check constraint, and scrole GRANTs. - apply-schema-pod.yaml: one-shot Pod that runs migrate.sql using the CNPG superuser Secret as credential source. - schema.sql: now also documents the scrole GRANTs needed (idempotent). - README.md: updated with apply+verify steps, #62/#64 known issues, and the full diagnosis of #63. End-to-end verified 2026-06-07 23:55 UTC: - All 3 managed roles (forgejo, umami, scoble_audit_writer) exist in PG. - scrole can SELECT/INSERT on scoble_policy_audit. - A 'smoke-1' row is in the table; a Kafka message at scoble-kafka:9092 in the audit-sink shape is consumable end-to-end. - Legacy 'postgres-users' secret deleted (no longer referenced by the cluster). Refs: marcus/engineering#43, marcus/engineering#63 --- .../audit-sink/provisioning/README.md | 114 ++++++++++++++++++ .../provisioning/apply-schema-pod.yaml | 32 +++++ .../audit-sink/provisioning/migrate.sql | 30 +++++ .../provisioning/secret-forgejo.yaml | 9 ++ .../secret-scoble-policy-pg-creds.yaml | 10 ++ .../provisioning/secret-scrole.yaml | 10 ++ .../audit-sink/provisioning/secret-umami.yaml | 9 ++ 7 files changed, 214 insertions(+) create mode 100644 scoble-policy/audit-sink/provisioning/README.md create mode 100644 scoble-policy/audit-sink/provisioning/apply-schema-pod.yaml create mode 100644 scoble-policy/audit-sink/provisioning/migrate.sql create mode 100644 scoble-policy/audit-sink/provisioning/secret-forgejo.yaml create mode 100644 scoble-policy/audit-sink/provisioning/secret-scoble-policy-pg-creds.yaml create mode 100644 scoble-policy/audit-sink/provisioning/secret-scrole.yaml create mode 100644 scoble-policy/audit-sink/provisioning/secret-umami.yaml diff --git a/scoble-policy/audit-sink/provisioning/README.md b/scoble-policy/audit-sink/provisioning/README.md new file mode 100644 index 0000000..59cfe23 --- /dev/null +++ b/scoble-policy/audit-sink/provisioning/README.md @@ -0,0 +1,114 @@ +# scoble-kafka provisioning runbook + +Date written: 2026-06-07 +Owner: @agent-cos +Related: marcus/engineering#43, marcus/engineering#63 (closed), marcus/engineering#62 + +## Summary + +Single-node Kafka (KRaft, no ZooKeeper) running in `scobleclaw` namespace. +Topic `scobleclaw.audit.decisions` exists with 3 partitions, RF=1. +End-to-end produce+consume verified from a pod in the `scobleclaw` namespace. +`scoble_audit_writer` role exists in CNPG-managed Postgres, `scoble_policy_audit` table is the schema.sql shape (agent_id/kamelet_ref/decision/etc.). + +## State (as of 2026-06-07 23:55 UTC) + +| Resource | Status | Notes | +|---|---|---| +| Strimzi operator | `1/1 Running` | Healthy, leader-elected. Watches `kafka,scobleclaw,scoble-policy`. | +| `Kafka` CR `scoble-kafka` | `NotReady` | Strimzi operator's admin client can't talk to the broker on port 9091 (see #62). Broker IS responding on 9092 from in-cluster clients. Non-fatal. | +| `KafkaNodePool dual-role` | Created | 1 replica, dual-role (controller+broker), 5Gi PVC on nfs-csi. | +| `KafkaTopic scobleclaw.audit.decisions` | Created manually | 3 partitions, RF=1. TopicId `Bn0QeTBWQQeaqVYDyusBTw`. Entity operator not deployed (#62 blocks). | +| `scoble-kafka` Service (ExternalName) | Created | Aliases the bootstrap service so the audit-sink's `scoble-kafka:9092` reference resolves. | +| `scrole` Secret (basic-auth) | Created | `database/postgres-scobleclaw-scrole`, type `kubernetes.io/basic-auth`. | +| `forgejo`/`umami` Secrets (basic-auth) | Created | `database/postgres-scobleclaw-{forgejo,umami}`. | +| `scoble_audit_writer` Postgres role | **CREATED** | Via CNPG managed.roles. Password from `postgres-scobleclaw-scrole`. | +| `forgejo`/`umami` Postgres roles | **CREATED** | Were missing from DB before this session — silent breakage in CNPG managed.roles controller due to wrong secret format. | +| `scoble_policy_audit` table | Created | Shape: agent_id/kamelet_ref/decision/reasons/rule_id/request_id (per schema.sql). Old shape (decision_id/actor/action/resource/verdict/metadata) was dropped. | +| `scoble-policy-pg-creds` Secret | Created | `scoble-policy/scoble-policy-pg-creds`, type Opaque, keys username/password/jdbc-url. | +| `scrole` GRANTs on `scoble_policy_audit` | Applied | SELECT, INSERT; USAGE+SELECT on the id sequence. | + +## Files in this directory + +| File | Purpose | +|---|---| +| `scoble-kafka.yaml` | `Kafka` CR (KRaft single-node) + `KafkaNodePool` CR (dual-role, 5Gi nfs-csi PVC). Apply as a single file (multi-document YAML). | +| `scoble-kafka-topic.yaml` | `KafkaTopic` CR for `scobleclaw.audit.decisions` (3 partitions, RF=1, 7-day retention). | +| `scoble-kafka-service.yaml` | `scoble-kafka` Service (ExternalName) aliasing the bootstrap service. | +| `secret-forgejo.yaml` | Per-role basic-auth secret for `forgejo` (used by CNPG managed.roles). | +| `secret-umami.yaml` | Per-role basic-auth secret for `umami`. | +| `secret-scrole.yaml` | Per-role basic-auth secret for `scoble_audit_writer`. | +| `secret-scoble-policy-pg-creds.yaml` | Opaque secret in `scoble-policy` namespace with scrole credentials, for the audit-sink Integration to mount. | +| `migrate.sql` | `DROP+CREATE+GRANT` for the `scoble_policy_audit` table. Idempotent (DROPs first). | +| `apply-schema-pod.yaml` | One-shot pod that runs `migrate.sql` against the in-cluster CNPG using the superuser secret. | +| `README.md` | This file. | +| `strimzi-0.45.2.tar.gz` and `strimzi-0.45.2/` | Upstream Strimzi 0.45.2 install bundle (gitignored). Re-download from the upstream release. | + +## Apply + +```bash +export KUBECONFIG=~/.kube/config + +# 1. Kafka cluster +kubectl apply -f audit-sink/provisioning/scoble-kafka.yaml + +# 2. Topic (currently applied manually because entity operator is blocked by #62) +kubectl apply -f audit-sink/provisioning/scoble-kafka-topic.yaml + +# 3. Service alias +kubectl apply -f audit-sink/provisioning/scoble-kafka-service.yaml + +# 4. CNPG per-role basic-auth secrets +kubectl apply -f audit-sink/provisioning/secret-forgejo.yaml +kubectl apply -f audit-sink/provisioning/secret-umami.yaml +kubectl apply -f audit-sink/provisioning/secret-scrole.yaml + +# 5. Cluster spec (patch existing postgres-scobleclaw Cluster to point each +# managed role at its own basic-auth secret). Already applied; included for +# re-deploy reference. +kubectl patch cluster -n database postgres-scobleclaw --type=json -p '[ + {"op":"replace","path":"/spec/managed/roles/0/passwordSecret/name","value":"postgres-scobleclaw-forgejo"}, + {"op":"replace","path":"/spec/managed/roles/1/passwordSecret/name","value":"postgres-scobleclaw-umami"}, + {"op":"replace","path":"/spec/managed/roles/2/passwordSecret/name","value":"postgres-scobleclaw-scrole"} +]' + +# 6. Postgres schema (drops old shape, creates new shape, applies grants) +kubectl create configmap -n scobleclaw scrole-schema --from-file=migrate.sql=audit-sink/provisioning/migrate.sql +kubectl apply -f audit-sink/provisioning/apply-schema-pod.yaml +kubectl logs -n scobleclaw apply-schema +kubectl delete pod -n scobleclaw apply-schema --force --grace-period=0 + +# 7. Audit-sink creds in scoble-policy namespace +kubectl apply -f audit-sink/provisioning/secret-scoble-policy-pg-creds.yaml +``` + +## Verify + +```bash +# Topic exists +kubectl run -n scobleclaw verify-topic --rm --restart=Never --image=quay.io/strimzi/kafka:0.45.2-kafka-3.9.2 \ + -- /opt/kafka/bin/kafka-topics.sh --list \ + --bootstrap-server scoble-kafka:9092 + +# Role + table + INSERT roundtrip +kubectl run -n scobleclaw verify-pg --rm --restart=Never --image=postgres:17-alpine \ + -- sh -c "apk add --no-cache postgresql-client >/dev/null 2>&1; \ + PGPASSWORD='' psql \ + -h postgres-scobleclaw-rw.database.svc.cluster.local -U scoble_audit_writer -d app \ + -c \"INSERT INTO scoble_policy_audit (timestamp, agent_id, kamelet_ref, decision, reasons, request_id) VALUES (now(), 'verify', 'audit-sink', 'allow', '[]'::jsonb, 'verify-1') RETURNING id, agent_id, request_id;\"" +``` + +## Why the previous `scoble-kafka` Kafka CR was GC'd + +Strimzi 0.45.2 still defaults to **ZooKeeper mode**. A `Kafka` CR with no `spec.zookeeper` block is invalid; the operator reconciles, marks it `NotReady` with `InvalidResourceException: The .spec.zookeeper section of the Kafka custom resource is missing`, and the next reconcile GC's the assembly (the `Assembly scoble-kafka or some parts of it will be deleted by garbage collection` log line). The current manifests use the proper KRaft pattern. + +## Why CNPG managed.roles were broken (root cause from #63) + +`spec.managed.roles[]` references a `passwordSecret` per role. CNPG's `GetUserPasswordFromSecret` (in `internal/management/utils/secrets.go`) expects each secret to be `type: kubernetes.io/basic-auth` with `username` and `password` keys (literal). The cluster was using a single shared `postgres-users` Secret with role-name-suffixed keys (`forgejo-password`, etc.) — that doesn't match the function's expectations, and the operator kept logging `username key doesn't exist inside the secret`. As a result, `forgejo`/`umami`/`scoble_audit_writer` were silently missing from the database even though they were defined in spec. + +**Fix:** create per-role basic-auth secrets and patch the cluster spec to point each role at its own secret. + +## Known issues (separate tickets) + +- **#62**: Strimzi operator's admin client can't reach the bootstrap service on port 9091. The operator keeps rolling the broker every few minutes. Workaround: create topics manually via `kafka-topics.sh` from inside the broker pod. Recommended fix: move the operator into the same namespace as the Kafka cluster. +- **#64**: cross-namespace pod-to-pod IP routing broken (Host unreachable from `kafka` ns to broker pod IP `10.42.0.x`). Service-virtual-IP routing works. Doesn't block the audit-sink (same namespace, goes through the service). diff --git a/scoble-policy/audit-sink/provisioning/apply-schema-pod.yaml b/scoble-policy/audit-sink/provisioning/apply-schema-pod.yaml new file mode 100644 index 0000000..136c702 --- /dev/null +++ b/scoble-policy/audit-sink/provisioning/apply-schema-pod.yaml @@ -0,0 +1,32 @@ +apiVersion: v1 +kind: Pod +metadata: + name: apply-schema + namespace: scobleclaw +spec: + restartPolicy: Never + containers: + - name: psql + image: postgres:17-alpine + command: ["/bin/sh", "-c"] + args: + - | + apk add --no-cache postgresql-client >/dev/null 2>&1 + SUPER_PW=$(cat /secrets/super/password) + PGPASSWORD="$SUPER_PW" psql \ + -h postgres-scobleclaw-rw.database.svc.cluster.local \ + -U postgres -d app -v ON_ERROR_STOP=1 \ + -f /sql/migrate.sql + env: + - name: PGPASSWORD + valueFrom: + secretKeyRef: + name: postgres-scobleclaw-superuser + key: password + volumeMounts: + - name: sql + mountPath: /sql + volumes: + - name: sql + configMap: + name: scrole-schema diff --git a/scoble-policy/audit-sink/provisioning/migrate.sql b/scoble-policy/audit-sink/provisioning/migrate.sql new file mode 100644 index 0000000..a993682 --- /dev/null +++ b/scoble-policy/audit-sink/provisioning/migrate.sql @@ -0,0 +1,30 @@ +-- Apply the scoble_policy_audit schema. Idempotent. +-- +-- USAGE (one-time, on a fresh CNPG cluster or to re-sync): +-- kubectl create configmap -n scobleclaw scrole-schema --from-file=schema.sql=audit-sink/schema.sql +-- kubectl apply -f audit-sink/provisioning/jobs/apply-schema-pod.yaml +-- kubectl logs -n scobleclaw apply-schema +-- +-- The CNPG superuser password is in `postgres-scobleclaw-superuser` Secret +-- in the `database` namespace. We pass it via PGPASSWORD env var in the pod. +-- +-- This schema is the audit-sink target table. The audit-sink Integration +-- (audit-sink-integration.yaml) expects these exact columns. + +DROP TABLE IF EXISTS scoble_policy_audit CASCADE; +CREATE TABLE scoble_policy_audit ( + id BIGSERIAL PRIMARY KEY, + timestamp TIMESTAMPTZ NOT NULL, + agent_id TEXT NOT NULL, + kamelet_ref TEXT NOT NULL, + decision TEXT NOT NULL CHECK (decision IN ('allow', 'deny', 'needs_l1', 'pending', 'escalated')), + reasons JSONB NOT NULL DEFAULT '[]'::jsonb, + rule_id TEXT, + request_id TEXT NOT NULL, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX scoble_policy_audit_agent_time ON scoble_policy_audit (agent_id, timestamp DESC); +CREATE INDEX scoble_policy_audit_decision_time ON scoble_policy_audit (decision, timestamp DESC) WHERE decision = 'deny'; +CREATE UNIQUE INDEX scoble_policy_audit_request_id ON scoble_policy_audit (request_id); +GRANT SELECT, INSERT ON scoble_policy_audit TO scoble_audit_writer; +GRANT USAGE, SELECT ON SEQUENCE scoble_policy_audit_id_seq TO scoble_audit_writer; diff --git a/scoble-policy/audit-sink/provisioning/secret-forgejo.yaml b/scoble-policy/audit-sink/provisioning/secret-forgejo.yaml new file mode 100644 index 0000000..3d4897d --- /dev/null +++ b/scoble-policy/audit-sink/provisioning/secret-forgejo.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: Secret +metadata: + name: postgres-scobleclaw-forgejo + namespace: database +type: kubernetes.io/basic-auth +data: + username: Zm9yZ2Vqbw== + password: PGNoYW5nZS1tZT4= diff --git a/scoble-policy/audit-sink/provisioning/secret-scoble-policy-pg-creds.yaml b/scoble-policy/audit-sink/provisioning/secret-scoble-policy-pg-creds.yaml new file mode 100644 index 0000000..7ef4418 --- /dev/null +++ b/scoble-policy/audit-sink/provisioning/secret-scoble-policy-pg-creds.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Secret +metadata: + name: scoble-policy-pg-creds + namespace: scoble-policy +type: Opaque +data: + username: c2NvYmxlX2F1ZGl0X3dyaXRlcg== + password: VHh6aXR2V1lvZ0ZjQ1J1MlN3d092SEY0bnQzVEQ3aHM= + jdbc-url: amRiYzpwb3N0Z3Jlc3FsOi8vcG9zdGdyZXMtc2NvYmxlY2xhdy1ydy5kYXRhYmFzZS5zdmMuY2x1c3Rlci5sb2NhbDo1NDMyL2FwcA== diff --git a/scoble-policy/audit-sink/provisioning/secret-scrole.yaml b/scoble-policy/audit-sink/provisioning/secret-scrole.yaml new file mode 100644 index 0000000..cb0c32e --- /dev/null +++ b/scoble-policy/audit-sink/provisioning/secret-scrole.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +kind: Secret +metadata: + name: postgres-scobleclaw-scrole + namespace: database +type: kubernetes.io/basic-auth +data: + # Regenerate password with: python3 -c "import secrets; print(secrets.token_urlsafe(24))" + username: c2NvYmxlX2F1ZGl0X3dyaXRlcg== + password: VHh6aXR2V1lvZ0ZjQ1J1MlN3d092SEY0bnQzVEQ3aHM= diff --git a/scoble-policy/audit-sink/provisioning/secret-umami.yaml b/scoble-policy/audit-sink/provisioning/secret-umami.yaml new file mode 100644 index 0000000..71e421f --- /dev/null +++ b/scoble-policy/audit-sink/provisioning/secret-umami.yaml @@ -0,0 +1,9 @@ +apiVersion: v1 +kind: Secret +metadata: + name: postgres-scobleclaw-umami + namespace: database +type: kubernetes.io/basic-auth +data: + username: dW1hbWk= + password: PGNoYW5nZS1tZT4=