Initial cluster config: NixOS flake + k3s + 5 node definitions
- Multi-host flake.nix (3x N97 x86_64 + 2x RPi4 aarch64) - Shared modules: k3s-cluster, tailscale, meos-gateway - Node configs: n97-1 (control plane), n97-2/n97-3 (Ollama workers), rpi4-1/rpi4-2 (ARM workers) - K8s manifests: Forgejo, Postgres (CNPG), Umami, OpenBrain, MeOS Dashboard, MCP servers - Bootstrap and DB migration scripts - Migration plan doc (7-day phased rollout)
This commit is contained in:
parent
791a9b6721
commit
7b5ffe6b11
21 changed files with 1184 additions and 2 deletions
69
docs/migration.md
Normal file
69
docs/migration.md
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
# Migration Plan: Bare Metal → k3s Cluster
|
||||
|
||||
## Overview
|
||||
|
||||
Migrate from single-node N97 (nixos) to 5-node k3s cluster with zero downtime.
|
||||
|
||||
## Phase 1: Foundation (Day 1)
|
||||
|
||||
1. **Create devops repo** ✅ (sscoble/devops)
|
||||
2. **Bootstrap k3s on n97-1** — single-node control plane alongside existing services
|
||||
3. **Install CNPG operator** — `kubectl apply -f cnpg-crds.yaml`
|
||||
4. **Deploy local-path provisioner** — for persistent volumes on each node
|
||||
5. **Install nginx ingress controller** — `helm install nginx-ingress ingress-nginx`
|
||||
|
||||
At this point, k3s is running but all services are still bare metal. No traffic shifts yet.
|
||||
|
||||
## Phase 2: Non-Critical Services (Day 2-3)
|
||||
|
||||
Move services that don't affect the core workflow:
|
||||
|
||||
1. **MeOS Dashboard → k3s** — deploy manifest, update nginx upstream
|
||||
2. **MCP servers → rpi4-1** — forgejo-mcp, lean-experiment, wardrobe-ai
|
||||
3. **Taint ARM nodes** — prevent x86 pods from landing on Pi 4s
|
||||
|
||||
## Phase 3: Data Layer (Day 4-5)
|
||||
|
||||
1. **Deploy CNPG Postgres cluster** — `kubectl apply -f apps/postgres/cluster.yaml`
|
||||
2. **Run migration** — `./scripts/migrate-db.sh`
|
||||
3. **Verify data integrity** — row counts, test queries
|
||||
4. **Update Forgejo to point at CNPG** — change DATABASE_URL env var
|
||||
5. **Disable bare-metal PostgreSQL** — set `services.postgresql.enable = false`
|
||||
|
||||
## Phase 4: Core Services (Day 6-7)
|
||||
|
||||
1. **Forgejo → k3s** — deploy manifest, cut over nginx upstream
|
||||
2. **Umami → rpi4-2** — deploy manifest, verify analytics still collecting
|
||||
3. **Homepage dashboard** — stays on n97-1 (lightweight, already working)
|
||||
|
||||
## Phase 5: Scale Out (Day 8+)
|
||||
|
||||
1. **Add n97-2 as worker** — `./scripts/bootstrap-node.sh n97-2 100.x.x.x agent`
|
||||
2. **Add n97-3 as worker** — same
|
||||
3. **Deploy Ollama on n97-2/n97-3** — systemd services, expose as ClusterIP
|
||||
4. **Add rpi4-1/rpi4-2 as workers** — bootstrap with ARM64 support
|
||||
5. **Configure node taints** — ARM workers for light work, N97s for heavy
|
||||
|
||||
## What Stays Bare Metal
|
||||
|
||||
- **OpenClaw Gateway** — manages processes, not containers. Stay outside k8s.
|
||||
- **Nginx + ACME** — TLS termination on n97-1, routes to k8s Ingress
|
||||
- **Ollama** — GPU/NPU passthrough is painful in k8s. Run as systemd services on workers.
|
||||
- **Storj static hosting** — build + deploy pipeline stays in Nix flake
|
||||
- **X11/LightDM** — only on n97-1 if you use the desktop
|
||||
|
||||
## Rollback Strategy
|
||||
|
||||
Each phase can be rolled back independently:
|
||||
- **Phase 2**: Switch nginx upstreams back to bare-metal ports
|
||||
- **Phase 3**: Re-enable `services.postgresql.enable = true`, restore from dump
|
||||
- **Phase 4**: Same as Phase 2 — nginx upstream swap
|
||||
- **Phase 5**: `kubectl drain <node> && kubectl delete node <node>`
|
||||
|
||||
## Post-Migration
|
||||
|
||||
- Add `nodeAffinity` rules for Ollama pods (n97-2, n97-3 only)
|
||||
- Set up `PodDisruptionBudgets` for critical services
|
||||
- Consider HA control plane (3-server etcd) if uptime matters
|
||||
- Add monitoring: Prometheus + Grafana on rpi4-2
|
||||
- Set up Velero for K8s backups
|
||||
Loading…
Add table
Add a link
Reference in a new issue