Phase 7: Cleanup and hardening

agent-cos commented

2026-05-04 19:40:34 +00:00

Goal

Final cleanup of bare-metal services. Harden the cluster.

Tasks

Cleanup

Remove old data directories from bare metal (after verification):
- /data/postgresql (if PostgreSQL fully migrated)
- /var/lib/forgejo (if Forgejo fully migrated)
- /var/lib/umami (if Umami fully migrated)
Archive old systemd service files
Update backup scripts to capture k8s state

Hardening

Set resource limits on all pods
Configure PodDisruptionBudgets for critical services
NetworkPolicies: restrict pod-to-pod communication
Enable k3s audit logging
Configure Velero backups (S3/Storj) for k8s resources and PVCs

Monitoring

Deploy Prometheus + Grafana on rpi4-2
Dashboard: k3s node health, Ollama utilization, Postgres metrics
Alerts: disk full, node down, OOM, high load

Documentation

Update README.md with actual Tailscale IPs
Document operational runbook (common kubectl commands)
Document disaster recovery (restore from CNPG backup)

Acceptance

All bare-metal services disabled or removed
Cluster backups configured and tested
Monitoring dashboard operational
Runbook documented

Depends On

#4, #5, #6 (all migrations complete)

## Goal Final cleanup of bare-metal services. Harden the cluster. ## Tasks ### Cleanup - [ ] Remove old data directories from bare metal (after verification): - [ ] `/data/postgresql` (if PostgreSQL fully migrated) - [ ] `/var/lib/forgejo` (if Forgejo fully migrated) - [ ] `/var/lib/umami` (if Umami fully migrated) - [ ] Archive old systemd service files - [ ] Update backup scripts to capture k8s state ### Hardening - [ ] Set resource limits on all pods - [ ] Configure PodDisruptionBudgets for critical services - [ ] NetworkPolicies: restrict pod-to-pod communication - [ ] Enable k3s audit logging - [ ] Configure Velero backups (S3/Storj) for k8s resources and PVCs ### Monitoring - [ ] Deploy Prometheus + Grafana on rpi4-2 - [ ] Dashboard: k3s node health, Ollama utilization, Postgres metrics - [ ] Alerts: disk full, node down, OOM, high load ### Documentation - [ ] Update README.md with actual Tailscale IPs - [ ] Document operational runbook (common kubectl commands) - [ ] Document disaster recovery (restore from CNPG backup) ## Acceptance - All bare-metal services disabled or removed - Cluster backups configured and tested - Monitoring dashboard operational - Runbook documented ## Depends On - #4, #5, #6 (all migrations complete)

sscoble commented

2026-05-25 02:18:27 +00:00

Owner

Triage: Still pending — blocked on all migration phases (#4, #5, #6)

Phase 7 (Cleanup + Hardening + Monitoring) depends on:

#4 ❌ (Postgres migration)
#5 ❌ (Forgejo migration)
#6 ❌ (Umami migration)

Notable: Grafana and Prometheus are already deployed in the cluster (scobleclaw-monitoring namespace, running for 10 days), so monitoring groundwork is partially laid. The rest of cleanup & hardening cannot proceed until migrations are complete. Leaving open.

## Triage: Still pending — blocked on all migration phases (#4, #5, #6) Phase 7 (Cleanup + Hardening + Monitoring) depends on: - #4 ❌ (Postgres migration) - #5 ❌ (Forgejo migration) - #6 ❌ (Umami migration) Notable: Grafana and Prometheus are already deployed in the cluster (`scobleclaw-monitoring` namespace, running for 10 days), so monitoring groundwork is partially laid. The rest of cleanup & hardening cannot proceed until migrations are complete. Leaving open.

sscoble commented

2026-05-26 00:34:12 +00:00

Owner

❌ BLOCKED

Reason: Transitively blocked. Phase 7 depends on Phases 3-6 (devops#3,4,5,6), which all depend on worker nodes (n97-2, n97-3, rpi4-1, rpi4-2) being physically powered on. Cannot proceed until nodes are reachable.

What's needed: Power on the 4 worker nodes. Once they're online and reachable via Tailscale, this ticket becomes dispatchable.

## ❌ BLOCKED **Reason:** Transitively blocked. Phase 7 depends on Phases 3-6 (devops#3,4,5,6), which all depend on worker nodes (n97-2, n97-3, rpi4-1, rpi4-2) being physically powered on. Cannot proceed until nodes are reachable. **What's needed:** Power on the 4 worker nodes. Once they're online and reachable via Tailscale, this ticket becomes dispatchable.

Rows
Columns

Phase 7: Cleanup and hardening #8