# Migration Plan: Bare Metal → k3s Cluster

## Overview

Migrate from single-node N97 (nixos) to 5-node k3s cluster with zero downtime.

## Phase 1: Foundation (Day 1)

1. **Create devops repo** ✅ (sscoble/devops)
2. **Bootstrap k3s on n97-1** — single-node control plane alongside existing services
3. **Install CNPG operator** — `kubectl apply -f cnpg-crds.yaml`
4. **Deploy local-path provisioner** — for persistent volumes on each node
5. **Install nginx ingress controller** — `helm install nginx-ingress ingress-nginx`

At this point, k3s is running but all services are still bare metal. No traffic shifts yet.

## Phase 2: Non-Critical Services (Day 2-3)

Move services that don't affect the core workflow:

1. **MeOS Dashboard → k3s** — deploy manifest, update nginx upstream
2. **MCP servers → rpi4-1** — forgejo-mcp, lean-experiment, wardrobe-ai
3. **Taint ARM nodes** — prevent x86 pods from landing on Pi 4s

## Phase 3: Data Layer (Day 4-5)

1. **Deploy CNPG Postgres cluster** — `kubectl apply -f apps/postgres/cluster.yaml`
2. **Run migration** — `./scripts/migrate-db.sh`
3. **Verify data integrity** — row counts, test queries
4. **Update Forgejo to point at CNPG** — change DATABASE_URL env var
5. **Disable bare-metal PostgreSQL** — set `services.postgresql.enable = false`

## Phase 4: Core Services (Day 6-7)

1. **Forgejo → k3s** — deploy manifest, cut over nginx upstream
2. **Umami → rpi4-2** — deploy manifest, verify analytics still collecting
3. **Homepage dashboard** — stays on n97-1 (lightweight, already working)

## Phase 5: Scale Out (Day 8+)

1. **Add n97-2 as worker** — `./scripts/bootstrap-node.sh n97-2 100.x.x.x agent`
2. **Add n97-3 as worker** — same
3. **Deploy Ollama on n97-2/n97-3** — systemd services, expose as ClusterIP
4. **Add rpi4-1/rpi4-2 as workers** — bootstrap with ARM64 support
5. **Configure node taints** — ARM workers for light work, N97s for heavy

## What Stays Bare Metal

- **OpenClaw Gateway** — manages processes, not containers. Stay outside k8s.
- **Nginx + ACME** — TLS termination on n97-1, routes to k8s Ingress
- **Ollama** — GPU/NPU passthrough is painful in k8s. Run as systemd services on workers.
- **Storj static hosting** — build + deploy pipeline stays in Nix flake
- **X11/LightDM** — only on n97-1 if you use the desktop

## Rollback Strategy

Each phase can be rolled back independently:
- **Phase 2**: Switch nginx upstreams back to bare-metal ports
- **Phase 3**: Re-enable `services.postgresql.enable = true`, restore from dump
- **Phase 4**: Same as Phase 2 — nginx upstream swap
- **Phase 5**: `kubectl drain <node> && kubectl delete node <node>`

## Post-Migration

- Add `nodeAffinity` rules for Ollama pods (n97-2, n97-3 only)
- Set up `PodDisruptionBudgets` for critical services
- Consider HA control plane (3-server etcd) if uptime matters
- Add monitoring: Prometheus + Grafana on rpi4-2
- Set up Velero for K8s backups