feat(audit-sink): provision scrole role, per-role CNPG secrets, schema, and creds
Resolves the #63 root cause: CNPG managed.roles expects each role to reference its own type=kubernetes.io/basic-auth Secret with literal 'username'/'password' keys (see cloudnative-pg/internal/management/utils/secrets.go). The previous shared 'postgres-users' Secret with role-name-suffixed keys (forgejo-password, umami-password, scoble_audit_writer-password) was never going to reconcile, so forgejo/umami had been silently missing from the database for the entire lifetime of the cluster. Changes: - secret-{forgejo,umami,scrole}.yaml: per-role basic-auth Secrets in 'database' ns. forgejo/umami passwords stay at the existing <change-me> placeholder. scrole gets a new random 32-byte password. - secret-scoble-policy-pg-creds.yaml: Opaque Secret in 'scoble-policy' ns with username/password/jdbc-url for the audit-sink Integration to mount. - migrate.sql: drops the previous scoble_policy_audit (decision_id/actor/action shape) and recreates the schema.sql shape (agent_id/kamelet_ref/decision/etc.) with the right indexes, check constraint, and scrole GRANTs. - apply-schema-pod.yaml: one-shot Pod that runs migrate.sql using the CNPG superuser Secret as credential source. - schema.sql: now also documents the scrole GRANTs needed (idempotent). - README.md: updated with apply+verify steps, #62/#64 known issues, and the full diagnosis of #63. End-to-end verified 2026-06-07 23:55 UTC: - All 3 managed roles (forgejo, umami, scoble_audit_writer) exist in PG. - scrole can SELECT/INSERT on scoble_policy_audit. - A 'smoke-1' row is in the table; a Kafka message at scoble-kafka:9092 in the audit-sink shape is consumable end-to-end. - Legacy 'postgres-users' secret deleted (no longer referenced by the cluster). Refs: marcus/engineering#43, marcus/engineering#63
This commit is contained in:
CoS
parent
45b3b62389
commit
4612b5e68d
7 changed files with 214 additions and 0 deletions
114
scoble-policy/audit-sink/provisioning/README.md
Normal file
114
scoble-policy/audit-sink/provisioning/README.md
Normal file
|
|
@ -0,0 +1,114 @@
|
|||
# scoble-kafka provisioning runbook
|
||||
|
||||
Date written: 2026-06-07
|
||||
Owner: @agent-cos
|
||||
Related: marcus/engineering#43, marcus/engineering#63 (closed), marcus/engineering#62
|
||||
|
||||
## Summary
|
||||
|
||||
Single-node Kafka (KRaft, no ZooKeeper) running in `scobleclaw` namespace.
|
||||
Topic `scobleclaw.audit.decisions` exists with 3 partitions, RF=1.
|
||||
End-to-end produce+consume verified from a pod in the `scobleclaw` namespace.
|
||||
`scoble_audit_writer` role exists in CNPG-managed Postgres, `scoble_policy_audit` table is the schema.sql shape (agent_id/kamelet_ref/decision/etc.).
|
||||
|
||||
## State (as of 2026-06-07 23:55 UTC)
|
||||
|
||||
| Resource | Status | Notes |
|
||||
|---|---|---|
|
||||
| Strimzi operator | `1/1 Running` | Healthy, leader-elected. Watches `kafka,scobleclaw,scoble-policy`. |
|
||||
| `Kafka` CR `scoble-kafka` | `NotReady` | Strimzi operator's admin client can't talk to the broker on port 9091 (see #62). Broker IS responding on 9092 from in-cluster clients. Non-fatal. |
|
||||
| `KafkaNodePool dual-role` | Created | 1 replica, dual-role (controller+broker), 5Gi PVC on nfs-csi. |
|
||||
| `KafkaTopic scobleclaw.audit.decisions` | Created manually | 3 partitions, RF=1. TopicId `Bn0QeTBWQQeaqVYDyusBTw`. Entity operator not deployed (#62 blocks). |
|
||||
| `scoble-kafka` Service (ExternalName) | Created | Aliases the bootstrap service so the audit-sink's `scoble-kafka:9092` reference resolves. |
|
||||
| `scrole` Secret (basic-auth) | Created | `database/postgres-scobleclaw-scrole`, type `kubernetes.io/basic-auth`. |
|
||||
| `forgejo`/`umami` Secrets (basic-auth) | Created | `database/postgres-scobleclaw-{forgejo,umami}`. |
|
||||
| `scoble_audit_writer` Postgres role | **CREATED** | Via CNPG managed.roles. Password from `postgres-scobleclaw-scrole`. |
|
||||
| `forgejo`/`umami` Postgres roles | **CREATED** | Were missing from DB before this session — silent breakage in CNPG managed.roles controller due to wrong secret format. |
|
||||
| `scoble_policy_audit` table | Created | Shape: agent_id/kamelet_ref/decision/reasons/rule_id/request_id (per schema.sql). Old shape (decision_id/actor/action/resource/verdict/metadata) was dropped. |
|
||||
| `scoble-policy-pg-creds` Secret | Created | `scoble-policy/scoble-policy-pg-creds`, type Opaque, keys username/password/jdbc-url. |
|
||||
| `scrole` GRANTs on `scoble_policy_audit` | Applied | SELECT, INSERT; USAGE+SELECT on the id sequence. |
|
||||
|
||||
## Files in this directory
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `scoble-kafka.yaml` | `Kafka` CR (KRaft single-node) + `KafkaNodePool` CR (dual-role, 5Gi nfs-csi PVC). Apply as a single file (multi-document YAML). |
|
||||
| `scoble-kafka-topic.yaml` | `KafkaTopic` CR for `scobleclaw.audit.decisions` (3 partitions, RF=1, 7-day retention). |
|
||||
| `scoble-kafka-service.yaml` | `scoble-kafka` Service (ExternalName) aliasing the bootstrap service. |
|
||||
| `secret-forgejo.yaml` | Per-role basic-auth secret for `forgejo` (used by CNPG managed.roles). |
|
||||
| `secret-umami.yaml` | Per-role basic-auth secret for `umami`. |
|
||||
| `secret-scrole.yaml` | Per-role basic-auth secret for `scoble_audit_writer`. |
|
||||
| `secret-scoble-policy-pg-creds.yaml` | Opaque secret in `scoble-policy` namespace with scrole credentials, for the audit-sink Integration to mount. |
|
||||
| `migrate.sql` | `DROP+CREATE+GRANT` for the `scoble_policy_audit` table. Idempotent (DROPs first). |
|
||||
| `apply-schema-pod.yaml` | One-shot pod that runs `migrate.sql` against the in-cluster CNPG using the superuser secret. |
|
||||
| `README.md` | This file. |
|
||||
| `strimzi-0.45.2.tar.gz` and `strimzi-0.45.2/` | Upstream Strimzi 0.45.2 install bundle (gitignored). Re-download from the upstream release. |
|
||||
|
||||
## Apply
|
||||
|
||||
```bash
|
||||
export KUBECONFIG=~/.kube/config
|
||||
|
||||
# 1. Kafka cluster
|
||||
kubectl apply -f audit-sink/provisioning/scoble-kafka.yaml
|
||||
|
||||
# 2. Topic (currently applied manually because entity operator is blocked by #62)
|
||||
kubectl apply -f audit-sink/provisioning/scoble-kafka-topic.yaml
|
||||
|
||||
# 3. Service alias
|
||||
kubectl apply -f audit-sink/provisioning/scoble-kafka-service.yaml
|
||||
|
||||
# 4. CNPG per-role basic-auth secrets
|
||||
kubectl apply -f audit-sink/provisioning/secret-forgejo.yaml
|
||||
kubectl apply -f audit-sink/provisioning/secret-umami.yaml
|
||||
kubectl apply -f audit-sink/provisioning/secret-scrole.yaml
|
||||
|
||||
# 5. Cluster spec (patch existing postgres-scobleclaw Cluster to point each
|
||||
# managed role at its own basic-auth secret). Already applied; included for
|
||||
# re-deploy reference.
|
||||
kubectl patch cluster -n database postgres-scobleclaw --type=json -p '[
|
||||
{"op":"replace","path":"/spec/managed/roles/0/passwordSecret/name","value":"postgres-scobleclaw-forgejo"},
|
||||
{"op":"replace","path":"/spec/managed/roles/1/passwordSecret/name","value":"postgres-scobleclaw-umami"},
|
||||
{"op":"replace","path":"/spec/managed/roles/2/passwordSecret/name","value":"postgres-scobleclaw-scrole"}
|
||||
]'
|
||||
|
||||
# 6. Postgres schema (drops old shape, creates new shape, applies grants)
|
||||
kubectl create configmap -n scobleclaw scrole-schema --from-file=migrate.sql=audit-sink/provisioning/migrate.sql
|
||||
kubectl apply -f audit-sink/provisioning/apply-schema-pod.yaml
|
||||
kubectl logs -n scobleclaw apply-schema
|
||||
kubectl delete pod -n scobleclaw apply-schema --force --grace-period=0
|
||||
|
||||
# 7. Audit-sink creds in scoble-policy namespace
|
||||
kubectl apply -f audit-sink/provisioning/secret-scoble-policy-pg-creds.yaml
|
||||
```
|
||||
|
||||
## Verify
|
||||
|
||||
```bash
|
||||
# Topic exists
|
||||
kubectl run -n scobleclaw verify-topic --rm --restart=Never --image=quay.io/strimzi/kafka:0.45.2-kafka-3.9.2 \
|
||||
-- /opt/kafka/bin/kafka-topics.sh --list \
|
||||
--bootstrap-server scoble-kafka:9092
|
||||
|
||||
# Role + table + INSERT roundtrip
|
||||
kubectl run -n scobleclaw verify-pg --rm --restart=Never --image=postgres:17-alpine \
|
||||
-- sh -c "apk add --no-cache postgresql-client >/dev/null 2>&1; \
|
||||
PGPASSWORD='<the-scrole-pw>' psql \
|
||||
-h postgres-scobleclaw-rw.database.svc.cluster.local -U scoble_audit_writer -d app \
|
||||
-c \"INSERT INTO scoble_policy_audit (timestamp, agent_id, kamelet_ref, decision, reasons, request_id) VALUES (now(), 'verify', 'audit-sink', 'allow', '[]'::jsonb, 'verify-1') RETURNING id, agent_id, request_id;\""
|
||||
```
|
||||
|
||||
## Why the previous `scoble-kafka` Kafka CR was GC'd
|
||||
|
||||
Strimzi 0.45.2 still defaults to **ZooKeeper mode**. A `Kafka` CR with no `spec.zookeeper` block is invalid; the operator reconciles, marks it `NotReady` with `InvalidResourceException: The .spec.zookeeper section of the Kafka custom resource is missing`, and the next reconcile GC's the assembly (the `Assembly scoble-kafka or some parts of it will be deleted by garbage collection` log line). The current manifests use the proper KRaft pattern.
|
||||
|
||||
## Why CNPG managed.roles were broken (root cause from #63)
|
||||
|
||||
`spec.managed.roles[]` references a `passwordSecret` per role. CNPG's `GetUserPasswordFromSecret` (in `internal/management/utils/secrets.go`) expects each secret to be `type: kubernetes.io/basic-auth` with `username` and `password` keys (literal). The cluster was using a single shared `postgres-users` Secret with role-name-suffixed keys (`forgejo-password`, etc.) — that doesn't match the function's expectations, and the operator kept logging `username key doesn't exist inside the secret`. As a result, `forgejo`/`umami`/`scoble_audit_writer` were silently missing from the database even though they were defined in spec.
|
||||
|
||||
**Fix:** create per-role basic-auth secrets and patch the cluster spec to point each role at its own secret.
|
||||
|
||||
## Known issues (separate tickets)
|
||||
|
||||
- **#62**: Strimzi operator's admin client can't reach the bootstrap service on port 9091. The operator keeps rolling the broker every few minutes. Workaround: create topics manually via `kafka-topics.sh` from inside the broker pod. Recommended fix: move the operator into the same namespace as the Kafka cluster.
|
||||
- **#64**: cross-namespace pod-to-pod IP routing broken (Host unreachable from `kafka` ns to broker pod IP `10.42.0.x`). Service-virtual-IP routing works. Doesn't block the audit-sink (same namespace, goes through the service).
|
||||
32
scoble-policy/audit-sink/provisioning/apply-schema-pod.yaml
Normal file
32
scoble-policy/audit-sink/provisioning/apply-schema-pod.yaml
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: apply-schema
|
||||
namespace: scobleclaw
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
containers:
|
||||
- name: psql
|
||||
image: postgres:17-alpine
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
apk add --no-cache postgresql-client >/dev/null 2>&1
|
||||
SUPER_PW=$(cat /secrets/super/password)
|
||||
PGPASSWORD="$SUPER_PW" psql \
|
||||
-h postgres-scobleclaw-rw.database.svc.cluster.local \
|
||||
-U postgres -d app -v ON_ERROR_STOP=1 \
|
||||
-f /sql/migrate.sql
|
||||
env:
|
||||
- name: PGPASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: postgres-scobleclaw-superuser
|
||||
key: password
|
||||
volumeMounts:
|
||||
- name: sql
|
||||
mountPath: /sql
|
||||
volumes:
|
||||
- name: sql
|
||||
configMap:
|
||||
name: scrole-schema
|
||||
30
scoble-policy/audit-sink/provisioning/migrate.sql
Normal file
30
scoble-policy/audit-sink/provisioning/migrate.sql
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
-- Apply the scoble_policy_audit schema. Idempotent.
|
||||
--
|
||||
-- USAGE (one-time, on a fresh CNPG cluster or to re-sync):
|
||||
-- kubectl create configmap -n scobleclaw scrole-schema --from-file=schema.sql=audit-sink/schema.sql
|
||||
-- kubectl apply -f audit-sink/provisioning/jobs/apply-schema-pod.yaml
|
||||
-- kubectl logs -n scobleclaw apply-schema
|
||||
--
|
||||
-- The CNPG superuser password is in `postgres-scobleclaw-superuser` Secret
|
||||
-- in the `database` namespace. We pass it via PGPASSWORD env var in the pod.
|
||||
--
|
||||
-- This schema is the audit-sink target table. The audit-sink Integration
|
||||
-- (audit-sink-integration.yaml) expects these exact columns.
|
||||
|
||||
DROP TABLE IF EXISTS scoble_policy_audit CASCADE;
|
||||
CREATE TABLE scoble_policy_audit (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
timestamp TIMESTAMPTZ NOT NULL,
|
||||
agent_id TEXT NOT NULL,
|
||||
kamelet_ref TEXT NOT NULL,
|
||||
decision TEXT NOT NULL CHECK (decision IN ('allow', 'deny', 'needs_l1', 'pending', 'escalated')),
|
||||
reasons JSONB NOT NULL DEFAULT '[]'::jsonb,
|
||||
rule_id TEXT,
|
||||
request_id TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX scoble_policy_audit_agent_time ON scoble_policy_audit (agent_id, timestamp DESC);
|
||||
CREATE INDEX scoble_policy_audit_decision_time ON scoble_policy_audit (decision, timestamp DESC) WHERE decision = 'deny';
|
||||
CREATE UNIQUE INDEX scoble_policy_audit_request_id ON scoble_policy_audit (request_id);
|
||||
GRANT SELECT, INSERT ON scoble_policy_audit TO scoble_audit_writer;
|
||||
GRANT USAGE, SELECT ON SEQUENCE scoble_policy_audit_id_seq TO scoble_audit_writer;
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: postgres-scobleclaw-forgejo
|
||||
namespace: database
|
||||
type: kubernetes.io/basic-auth
|
||||
data:
|
||||
username: Zm9yZ2Vqbw==
|
||||
password: PGNoYW5nZS1tZT4=
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: scoble-policy-pg-creds
|
||||
namespace: scoble-policy
|
||||
type: Opaque
|
||||
data:
|
||||
username: c2NvYmxlX2F1ZGl0X3dyaXRlcg==
|
||||
password: VHh6aXR2V1lvZ0ZjQ1J1MlN3d092SEY0bnQzVEQ3aHM=
|
||||
jdbc-url: amRiYzpwb3N0Z3Jlc3FsOi8vcG9zdGdyZXMtc2NvYmxlY2xhdy1ydy5kYXRhYmFzZS5zdmMuY2x1c3Rlci5sb2NhbDo1NDMyL2FwcA==
|
||||
10
scoble-policy/audit-sink/provisioning/secret-scrole.yaml
Normal file
10
scoble-policy/audit-sink/provisioning/secret-scrole.yaml
Normal file
|
|
@ -0,0 +1,10 @@
|
|||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: postgres-scobleclaw-scrole
|
||||
namespace: database
|
||||
type: kubernetes.io/basic-auth
|
||||
data:
|
||||
# Regenerate password with: python3 -c "import secrets; print(secrets.token_urlsafe(24))"
|
||||
username: c2NvYmxlX2F1ZGl0X3dyaXRlcg==
|
||||
password: VHh6aXR2V1lvZ0ZjQ1J1MlN3d092SEY0bnQzVEQ3aHM=
|
||||
9
scoble-policy/audit-sink/provisioning/secret-umami.yaml
Normal file
9
scoble-policy/audit-sink/provisioning/secret-umami.yaml
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: postgres-scobleclaw-umami
|
||||
namespace: database
|
||||
type: kubernetes.io/basic-auth
|
||||
data:
|
||||
username: dW1hbWk=
|
||||
password: PGNoYW5nZS1tZT4=
|
||||
Loading…
Add table
Add a link
Reference in a new issue