sscoble/devops
1
0
Fork 0
Code Issues 9 Pull requests Projects Releases Packages Wiki Activity Actions
devops/docs/migration.md
Marcus (agent-cos) 7b5ffe6b11 Initial cluster config: NixOS flake + k3s + 5 node definitions 
- Multi-host flake.nix (3x N97 x86_64 + 2x RPi4 aarch64)
- Shared modules: k3s-cluster, tailscale, meos-gateway
- Node configs: n97-1 (control plane), n97-2/n97-3 (Ollama workers), rpi4-1/rpi4-2 (ARM workers)
- K8s manifests: Forgejo, Postgres (CNPG), Umami, OpenBrain, MeOS Dashboard, MCP servers
- Bootstrap and DB migration scripts
- Migration plan doc (7-day phased rollout)
2026-05-04 12:22:09 -07:00

2.9 KiB
Raw Permalink Blame History

Migration Plan: Bare Metal → k3s Cluster

Overview

Migrate from single-node N97 (nixos) to 5-node k3s cluster with zero downtime.

Phase 1: Foundation (Day 1)

  1. Create devops repo ✅ (sscoble/devops)
  2. Bootstrap k3s on n97-1 — single-node control plane alongside existing services
  3. Install CNPG operator — kubectl apply -f cnpg-crds.yaml
  4. Deploy local-path provisioner — for persistent volumes on each node
  5. Install nginx ingress controller — helm install nginx-ingress ingress-nginx

At this point, k3s is running but all services are still bare metal. No traffic shifts yet.

Phase 2: Non-Critical Services (Day 2-3)

Move services that don't affect the core workflow:

  1. MeOS Dashboard → k3s — deploy manifest, update nginx upstream
  2. MCP servers → rpi4-1 — forgejo-mcp, lean-experiment, wardrobe-ai
  3. Taint ARM nodes — prevent x86 pods from landing on Pi 4s

Phase 3: Data Layer (Day 4-5)

  1. Deploy CNPG Postgres cluster — kubectl apply -f apps/postgres/cluster.yaml
  2. Run migration — ./scripts/migrate-db.sh
  3. Verify data integrity — row counts, test queries
  4. Update Forgejo to point at CNPG — change DATABASE_URL env var
  5. Disable bare-metal PostgreSQL — set services.postgresql.enable = false

Phase 4: Core Services (Day 6-7)

  1. Forgejo → k3s — deploy manifest, cut over nginx upstream
  2. Umami → rpi4-2 — deploy manifest, verify analytics still collecting
  3. Homepage dashboard — stays on n97-1 (lightweight, already working)

Phase 5: Scale Out (Day 8+)

  1. Add n97-2 as worker — ./scripts/bootstrap-node.sh n97-2 100.x.x.x agent
  2. Add n97-3 as worker — same
  3. Deploy Ollama on n97-2/n97-3 — systemd services, expose as ClusterIP
  4. Add rpi4-1/rpi4-2 as workers — bootstrap with ARM64 support
  5. Configure node taints — ARM workers for light work, N97s for heavy

What Stays Bare Metal

  • OpenClaw Gateway — manages processes, not containers. Stay outside k8s.
  • Nginx + ACME — TLS termination on n97-1, routes to k8s Ingress
  • Ollama — GPU/NPU passthrough is painful in k8s. Run as systemd services on workers.
  • Storj static hosting — build + deploy pipeline stays in Nix flake
  • X11/LightDM — only on n97-1 if you use the desktop

Rollback Strategy

Each phase can be rolled back independently:

  • Phase 2: Switch nginx upstreams back to bare-metal ports
  • Phase 3: Re-enable services.postgresql.enable = true, restore from dump
  • Phase 4: Same as Phase 2 — nginx upstream swap
  • Phase 5: kubectl drain <node> && kubectl delete node <node>

Post-Migration

  • Add nodeAffinity rules for Ollama pods (n97-2, n97-3 only)
  • Set up PodDisruptionBudgets for critical services
  • Consider HA control plane (3-server etcd) if uptime matters
  • Add monitoring: Prometheus + Grafana on rpi4-2
  • Set up Velero for K8s backups