sscoble/devops
1
0
Fork 0
Code Issues 9 Pull requests Projects Releases Packages Wiki Activity Actions

Bootstrap worker nodes (n97-2, n97-3, rpi4-1, rpi4-2) #3

New issue
Open
opened 2026-05-04 19:37:03 +00:00 by agent-cos · 3 comments
agent-cos commented 2026-05-04 19:37:03 +00:00
Copy link

Goal

Join 4 worker nodes to the k3s cluster via Tailscale mesh.

Tasks per node

  1. Install NixOS on target hardware (if not already)
  2. Install and auth Tailscale: tailscale up
  3. Fill in Tailscale IP in corresponding nodes/*.nix
  4. Copy cluster token to /var/lib/k3s/cluster-token
  5. Run: ./scripts/bootstrap-node.sh <hostname> <ip> agent
  6. Verify: kubectl get nodes shows node as Ready

n97-2 (x86_64 Ollama worker)

  • NixOS installed
  • Tailscale connected
  • k3s agent joined
  • Ollama running (deepseek-v4-pro, kimi-k2.6, glm-5.1)

n97-3 (x86_64 Ollama worker)

  • NixOS installed
  • Tailscale connected
  • k3s agent joined
  • Ollama running (qwen3-coder, deepseek-v4-flash, embed models)

rpi4-1 (aarch64 light services)

  • NixOS installed (SD card or USB boot)
  • Tailscale connected
  • k3s agent joined
  • ARM taint applied

rpi4-2 (aarch64 Umami + monitoring)

  • NixOS installed
  • Tailscale connected
  • k3s agent joined
  • ARM taint applied

Acceptance

  • kubectl get nodes shows all 5 nodes Ready
  • Architecture labels correct: kubernetes.io/arch=amd64 for N97s, arm64 for Pis
  • Ollama accessible on n97-2:11434 and n97-3:11434

Depends On

  • #1 (control plane must exist to join)

Notes

  • RPi4: Use USB boot if possible (faster than SD card)
  • RPi4: USB 2.0 limits network to ~300Mbps — fine for our workloads
  • N97 nodes: Verify kvm-intel module loaded for any future VM workloads
## Goal Join 4 worker nodes to the k3s cluster via Tailscale mesh. ## Tasks per node 1. Install NixOS on target hardware (if not already) 2. Install and auth Tailscale: `tailscale up` 3. Fill in Tailscale IP in corresponding `nodes/*.nix` 4. Copy cluster token to `/var/lib/k3s/cluster-token` 5. Run: `./scripts/bootstrap-node.sh <hostname> <ip> agent` 6. Verify: `kubectl get nodes` shows node as Ready ### n97-2 (x86_64 Ollama worker) - [ ] NixOS installed - [ ] Tailscale connected - [ ] k3s agent joined - [ ] Ollama running (deepseek-v4-pro, kimi-k2.6, glm-5.1) ### n97-3 (x86_64 Ollama worker) - [ ] NixOS installed - [ ] Tailscale connected - [ ] k3s agent joined - [ ] Ollama running (qwen3-coder, deepseek-v4-flash, embed models) ### rpi4-1 (aarch64 light services) - [ ] NixOS installed (SD card or USB boot) - [ ] Tailscale connected - [ ] k3s agent joined - [ ] ARM taint applied ### rpi4-2 (aarch64 Umami + monitoring) - [ ] NixOS installed - [ ] Tailscale connected - [ ] k3s agent joined - [ ] ARM taint applied ## Acceptance - `kubectl get nodes` shows all 5 nodes Ready - Architecture labels correct: `kubernetes.io/arch=amd64` for N97s, `arm64` for Pis - Ollama accessible on n97-2:11434 and n97-3:11434 ## Depends On - #1 (control plane must exist to join) ## Notes - RPi4: Use USB boot if possible (faster than SD card) - RPi4: USB 2.0 limits network to ~300Mbps — fine for our workloads - N97 nodes: Verify `kvm-intel` module loaded for any future VM workloads
sscoble commented 2026-05-25 02:17:55 +00:00
Owner
Copy link

Triage: Still pending — no workers joined

Checked cluster state: only the control plane (n97-mini/n97-1, Tailscale 100.104.70.88) is in the cluster. None of the 4 worker nodes have been bootstrapped:

  • n97-2: ❌ Not joined. Tailscale IP unknown.
  • n97-3: ❌ Not joined. Tailscale IP unknown.
  • rpi4-1: ❌ Not joined. Tailscale IP unknown.
  • rpi4-2: ❌ Not joined. Tailscale IP unknown.

Current state: All checkboxes unchecked. This is blocked on:

  1. Issue #9 (Tailscale IPs needed for all 4 workers)
  2. Scott running NixOS install + k3s agent join on each remote node

The cluster token exists (k3s is serving), so ./scripts/bootstrap-node.sh <hostname> <ip> agent should work once the nodes are reachable over Tailscale.

## Triage: Still pending — no workers joined Checked cluster state: **only the control plane** (n97-mini/n97-1, Tailscale `100.104.70.88`) is in the cluster. None of the 4 worker nodes have been bootstrapped: - **n97-2**: ❌ Not joined. Tailscale IP unknown. - **n97-3**: ❌ Not joined. Tailscale IP unknown. - **rpi4-1**: ❌ Not joined. Tailscale IP unknown. - **rpi4-2**: ❌ Not joined. Tailscale IP unknown. **Current state:** All checkboxes unchecked. This is blocked on: 1. Issue #9 (Tailscale IPs needed for all 4 workers) 2. Scott running NixOS install + k3s agent join on each remote node The cluster token exists (k3s is serving), so `./scripts/bootstrap-node.sh <hostname> <ip> agent` should work once the nodes are reachable over Tailscale.
sscoble commented 2026-05-25 22:32:10 +00:00
Owner
Copy link

BLOCKED — This ticket requires:

  1. Worker nodes (n97-2, n97-3, rpi4-1, rpi4-2) to be reachable via Tailscale
  2. Nodes are NOT currently online (Tailscale shows only n97-mini + 2 phones active)
  3. Kubeclaw repo not found at expected location (archived)
  4. Physical access required to set up/verify worker hardware

Available from this host: n97-mini with single-node k3s (control-plane only).
Cannot bootstrap workers or migrate databases without the target nodes being online.
Scott needs to power on and verify the worker hardware before this ticket can proceed.

BLOCKED — This ticket requires: 1. Worker nodes (n97-2, n97-3, rpi4-1, rpi4-2) to be reachable via Tailscale 2. Nodes are NOT currently online (Tailscale shows only n97-mini + 2 phones active) 3. Kubeclaw repo not found at expected location (archived) 4. Physical access required to set up/verify worker hardware Available from this host: n97-mini with single-node k3s (control-plane only). Cannot bootstrap workers or migrate databases without the target nodes being online. Scott needs to power on and verify the worker hardware before this ticket can proceed.
sscoble commented 2026-05-25 23:56:07 +00:00
Owner
Copy link

❌ BLOCKED

Reason: Worker nodes offline — n97-2, n97-3, rpi4-1, rpi4-2 need physical power-on and NixOS install

This ticket was previously identified as blocked. It requires Scott's action or external infrastructure before dispatch can proceed. Once unblocked, the system will pick it up on the next dispatch cycle.

Action for Scott: n97-2, n97-3, rpi4-1, rpi4-2 need physical power-on and NixOS install

## ❌ BLOCKED **Reason:** Worker nodes offline — n97-2, n97-3, rpi4-1, rpi4-2 need physical power-on and NixOS install This ticket was previously identified as blocked. It requires Scott's action or external infrastructure before dispatch can proceed. Once unblocked, the system will pick it up on the next dispatch cycle. **Action for Scott:** n97-2, n97-3, rpi4-1, rpi4-2 need physical power-on and NixOS install
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
blocked

   
bug

   
feature

   
improvement

   
question
No labels
blocked
bug
feature
improvement
question
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: sscoble/devops#3
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?