sscoble/devops
1
0
Fork 0
Code Issues 9 Pull requests Projects Releases Packages Wiki Activity Actions

Phase 3: Migrate PostgreSQL from bare metal to CloudNative PG #4

New issue
Closed
opened 2026-05-04 19:39:50 +00:00 by agent-cos · 11 comments
agent-cos commented 2026-05-04 19:39:50 +00:00
Copy link

Status: DECOMPOSED — see children below

This ticket has been split into 3 dispatchable children matching the rollback boundary.

Children

  • devops#10 — Create CNPG cluster + verify Ready condition
  • devops#11 — Migrate data from bare-metal Postgres to CNPG cluster
  • devops#12 — Cutover: Forgejo + Umami to CNPG, disable bare-metal

Why split

10 boxes = too broad for one subagent window. Splitting by rollback boundary lets step 1+2 fail safely while bare-metal remains authoritative.

Original goal (preserved)

Move Postgres data layer to k3s CNPG operator. Enables HA later.

Original acceptance (preserved)

  • CNPG cluster healthy (1 instance for now)
  • All databases migrated (forgejo, umami, any custom)
  • Bare-metal PostgreSQL disabled on n97-1
  • Forgejo, Umami both functional via CNPG

Depends on

  • devops#1 (k3s must be up)

Close condition

Close this parent when devops#12 is verified complete and Forgejo + Umami are functional on CNPG.

## Status: DECOMPOSED — see children below This ticket has been split into 3 dispatchable children matching the rollback boundary. ## Children - **devops#10** — Create CNPG cluster + verify Ready condition - **devops#11** — Migrate data from bare-metal Postgres to CNPG cluster - **devops#12** — Cutover: Forgejo + Umami to CNPG, disable bare-metal ## Why split 10 boxes = too broad for one subagent window. Splitting by rollback boundary lets step 1+2 fail safely while bare-metal remains authoritative. ## Original goal (preserved) Move Postgres data layer to k3s CNPG operator. Enables HA later. ## Original acceptance (preserved) - CNPG cluster healthy (1 instance for now) - All databases migrated (forgejo, umami, any custom) - Bare-metal PostgreSQL disabled on n97-1 - Forgejo, Umami both functional via CNPG ## Depends on - devops#1 (k3s must be up) ## Close condition Close this parent when devops#12 is verified complete and Forgejo + Umami are functional on CNPG.
sscoble commented 2026-05-25 02:18:14 +00:00
Owner
Copy link

Triage: Still pending — blocked on workers (#3) and Tailscale IPs (#9)

Phase 3 (Postgres → CNPG) depends on:

  • #1 ✅ (control plane running)
  • #3 ❌ (no workers joined — CNPG may want to schedule on worker nodes, though single-instance could run on control-plane)

The cluster is live with system pods running (coredns, metrics-server, grafana, prometheus), so CNPG operator installation (kubectl apply --server-side -f ...) could proceed on the control plane alone if desired. However, the Postgres migration script and safe cutover likely require the full cluster topology including the PV/local-path provisioner setup.

Leaving open — pending Scott's decision on whether to start CNPG install before workers are ready.

## Triage: Still pending — blocked on workers (#3) and Tailscale IPs (#9) Phase 3 (Postgres → CNPG) depends on: - #1 ✅ (control plane running) - #3 ❌ (no workers joined — CNPG may want to schedule on worker nodes, though single-instance could run on control-plane) The cluster is live with system pods running (coredns, metrics-server, grafana, prometheus), so CNPG operator installation (`kubectl apply --server-side -f ...`) could proceed on the control plane alone if desired. However, the Postgres migration script and safe cutover likely require the full cluster topology including the PV/local-path provisioner setup. Leaving open — pending Scott's decision on whether to start CNPG install before workers are ready.
sscoble commented 2026-05-25 22:32:10 +00:00
Owner
Copy link

BLOCKED — This ticket requires:

  1. Worker nodes (n97-2, n97-3, rpi4-1, rpi4-2) to be reachable via Tailscale
  2. Nodes are NOT currently online (Tailscale shows only n97-mini + 2 phones active)
  3. Kubeclaw repo not found at expected location (archived)
  4. Physical access required to set up/verify worker hardware

Available from this host: n97-mini with single-node k3s (control-plane only).
Cannot bootstrap workers or migrate databases without the target nodes being online.
Scott needs to power on and verify the worker hardware before this ticket can proceed.

BLOCKED — This ticket requires: 1. Worker nodes (n97-2, n97-3, rpi4-1, rpi4-2) to be reachable via Tailscale 2. Nodes are NOT currently online (Tailscale shows only n97-mini + 2 phones active) 3. Kubeclaw repo not found at expected location (archived) 4. Physical access required to set up/verify worker hardware Available from this host: n97-mini with single-node k3s (control-plane only). Cannot bootstrap workers or migrate databases without the target nodes being online. Scott needs to power on and verify the worker hardware before this ticket can proceed.
sscoble commented 2026-05-25 23:56:07 +00:00
Owner
Copy link

❌ BLOCKED

Reason: Worker nodes offline — Phase 3 (Postgres migration) requires worker nodes on k3s first

This ticket was previously identified as blocked. It requires Scott's action or external infrastructure before dispatch can proceed. Once unblocked, the system will pick it up on the next dispatch cycle.

Action for Scott: Phase 3 (Postgres migration) requires worker nodes on k3s first

## ❌ BLOCKED **Reason:** Worker nodes offline — Phase 3 (Postgres migration) requires worker nodes on k3s first This ticket was previously identified as blocked. It requires Scott's action or external infrastructure before dispatch can proceed. Once unblocked, the system will pick it up on the next dispatch cycle. **Action for Scott:** Phase 3 (Postgres migration) requires worker nodes on k3s first
sscoble commented 2026-05-30 11:23:06 +00:00
Owner
Copy link

Subagent Progress Report (Cycle 4, 2026-05-30)

Status: Partially complete — reached tool-call limit before migration finish.

What was accomplished

  • Installed CNPG operator via Helm into namespace cnpg-system. Operator pod running, CRDs present.
  • Verified bare-metal PostgreSQL on n97: databases include homeschool, openbrain, pool_manager, postgres, style_prefs, umami. Users include umami, style_prefs, openbrain_mcp.
  • Confirmed Umami systemd service connects via postgresql://umami@localhost/umami?host=/run/postgresql — confirmed migration target.
  • Discovered Forgejo uses SQLite, not PostgreSQL (no services.postgresql in config, database.type = "sqlite3"). Ticket assumption that Forgejo needs PG migration needs updating.
  • Confirmed k3s node n97-mini Ready. Existing plain postgres Deployment in databases namespace must be retired.
  • Could not locate devops repo on disk or apps/postgres/cluster.yaml — these may not yet exist locally.

Next steps

  1. Create CNPG Cluster manifest and apply to database namespace.
  2. Create postgres-users Secret with credentials.
  3. Dump bare-metal Postgres (pg_dumpall) and restore into CNPG primary.
  4. Update Umami service (or k8s Deployment) to point at CNPG.
  5. Reconcile Forgejo: either switch to PostgreSQL or adjust ticket acceptance.
  6. Disable bare-metal PostgreSQL once validated.

Re-unassigned by dispatch system — infrastructure task timed out so not immediately re-dispatched. Next cycle will assess.

## Subagent Progress Report (Cycle 4, 2026-05-30) **Status:** Partially complete — reached tool-call limit before migration finish. ### What was accomplished - **Installed CNPG operator** via Helm into namespace `cnpg-system`. Operator pod running, CRDs present. - **Verified bare-metal PostgreSQL** on n97: databases include `homeschool`, `openbrain`, `pool_manager`, `postgres`, `style_prefs`, `umami`. Users include `umami`, `style_prefs`, `openbrain_mcp`. - **Confirmed Umami** systemd service connects via `postgresql://umami@localhost/umami?host=/run/postgresql` — confirmed migration target. - **Discovered Forgejo uses SQLite**, not PostgreSQL (no `services.postgresql` in config, `database.type = "sqlite3"`). Ticket assumption that Forgejo needs PG migration needs updating. - **Confirmed k3s** node `n97-mini` Ready. Existing plain `postgres` Deployment in `databases` namespace must be retired. - **Could not locate** `devops` repo on disk or `apps/postgres/cluster.yaml` — these may not yet exist locally. ### Next steps 1. Create CNPG `Cluster` manifest and apply to `database` namespace. 2. Create `postgres-users` Secret with credentials. 3. Dump bare-metal Postgres (`pg_dumpall`) and restore into CNPG primary. 4. Update Umami service (or k8s Deployment) to point at CNPG. 5. Reconcile Forgejo: either switch to PostgreSQL or adjust ticket acceptance. 6. Disable bare-metal PostgreSQL once validated. **Re-unassigned by dispatch system** — infrastructure task timed out so not immediately re-dispatched. Next cycle will assess.
sscoble commented 2026-05-30 12:07:58 +00:00
Owner
Copy link

Dispatch Cycle Update (2026-05-30 04:54 UTC)

Status: Subagent timed out after 600s without completing work.

What happened:

  • Dispatched to agent-engineering to migrate PostgreSQL from bare metal to CNPG.
  • Subagent made 26 API calls but hit the 600s time limit before producing results.
  • No partial-progress summary was returned.

Action:

  • Ticket unassigned and returned to dispatchable pool.
  • Will be re-dispatched in the next cycle.

Note to next cycle: The prior cycle found CNPG operator already installed in cnpg-system. Bare-metal PG verified with dbs: homeschool, openbrain, pool_manager, umami, style_prefs. Forgejo uses SQLite. Consider breaking this into smaller chunks (manifest creation vs migration vs NixOS rebuild).

## Dispatch Cycle Update (2026-05-30 04:54 UTC) **Status:** Subagent timed out after 600s without completing work. **What happened:** - Dispatched to agent-engineering to migrate PostgreSQL from bare metal to CNPG. - Subagent made 26 API calls but hit the 600s time limit before producing results. - No partial-progress summary was returned. **Action:** - Ticket unassigned and returned to dispatchable pool. - Will be re-dispatched in the next cycle. **Note to next cycle:** The prior cycle found CNPG operator already installed in `cnpg-system`. Bare-metal PG verified with dbs: homeschool, openbrain, pool_manager, umami, style_prefs. Forgejo uses SQLite. Consider breaking this into smaller chunks (manifest creation vs migration vs NixOS rebuild).
sscoble commented 2026-05-30 14:35:45 +00:00
Owner
Copy link

Dispatched to @agent-engineering at cycle start. WIP limit observed.

Dispatched to @agent-engineering at cycle start. WIP limit observed.
sscoble commented 2026-05-30 14:46:34 +00:00
Owner
Copy link

Dispatch cycle timeout after 600s during initial investigation. No edits applied. Will re-dispatch in a future cycle with a narrower scope.

Dispatch cycle timeout after 600s during initial investigation. No edits applied. Will re-dispatch in a future cycle with a narrower scope.
sscoble commented 2026-05-30 15:24:42 +00:00
Owner
Copy link

Dispatch cycle: ticket previously timed out twice. Per no-blind-redispatch rule, deferred until split into smaller child issues or handled directly.

Dispatch cycle: ticket previously timed out twice. Per no-blind-redispatch rule, deferred until split into smaller child issues or handled directly.
sscoble commented 2026-06-01 18:45:54 +00:00
Owner
Copy link

Subagent dispatched but timed out after 600s during CNPG Postgres migration validation and nixos-rebuild steps. Unassigned. Will retry in next cycle.

Subagent dispatched but timed out after 600s during CNPG Postgres migration validation and nixos-rebuild steps. Unassigned. Will retry in next cycle.
sscoble commented 2026-06-02 02:50:43 +00:00
Owner
Copy link

🔀 Decomposed into 3 children (2026-06-01, per Scott's "unblock what you can" directive)

This ticket had 10 checkboxes spanning cluster creation, data migration, app cutover, and nixos-rebuild. After 3 subagent timeouts, scope is too broad for one window.

Children created:

  • devops#10 — [devops#4.1] Create CNPG cluster + verify Ready condition
  • devops#11 — [devops#4.2] Migrate data from bare-metal Postgres to CNPG cluster
  • devops#12 — [devops#4.3] Cutover: Forgejo + Umami to CNPG, disable bare-metal

Why this works: Each step is independently reversible. Step 1 (cluster create) is observable. Step 2 (migrate) is testable without cutover. Step 3 (cutover) only runs after the prior two succeed — and only step 3 has irreversible consequences (disabling bare-metal PG). This separation matches the rollback boundary: if step 2 fails, bare-metal is still authoritative; if step 3 fails, you roll back via the dump file.

This parent is removed from double_timeout (scope materially changed). Note: the dispatch-timeout-patterns doctrine says infrastructure tickets should never be sub-dispatched (use terminal or manual session). The children are still dispatchable but should be handled via focused terminal sessions, not delegate_task. Close this parent when devops#12 is verified complete.

— Chief of Staff

## 🔀 Decomposed into 3 children (2026-06-01, per Scott's "unblock what you can" directive) This ticket had 10 checkboxes spanning cluster creation, data migration, app cutover, and nixos-rebuild. After 3 subagent timeouts, scope is too broad for one window. **Children created:** - **devops#10** — `[devops#4.1]` Create CNPG cluster + verify Ready condition - **devops#11** — `[devops#4.2]` Migrate data from bare-metal Postgres to CNPG cluster - **devops#12** — `[devops#4.3]` Cutover: Forgejo + Umami to CNPG, disable bare-metal **Why this works:** Each step is independently reversible. Step 1 (cluster create) is observable. Step 2 (migrate) is testable without cutover. Step 3 (cutover) only runs after the prior two succeed — and only step 3 has irreversible consequences (disabling bare-metal PG). This separation matches the rollback boundary: if step 2 fails, bare-metal is still authoritative; if step 3 fails, you roll back via the dump file. **This parent is removed from `double_timeout` (scope materially changed).** Note: the dispatch-timeout-patterns doctrine says infrastructure tickets should never be sub-dispatched (use `terminal` or manual session). The children are still dispatchable but should be handled via focused `terminal` sessions, not `delegate_task`. Close this parent when devops#12 is verified complete. — Chief of Staff
sscoble commented 2026-06-03 04:22:39 +00:00
Owner
Copy link

Housekeeping close — decomposed into children

Per CoS dispatch cycle 2026-06-03, closing this parent. All Phase 3 work now lives in the children:

  • devops#10 (id 470) — ✅ CLOSED 2026-06-03 — [devops#4.1] Create CNPG cluster + verify Ready condition
  • devops#11 (id 471) — open, labeled blocked — [devops#4.2] Migrate data from bare-metal Postgres to CNPG cluster (blocked on sscoble/devops#3 bootstrap-worker-nodes)
  • devops#12 (id 472) — open — [devops#4.3] Cutover: Forgejo + Umami to CNPG, disable bare-metal

Status: Step 1 of 3 shipped (CNPG cluster created and Ready). Steps 2 and 3 are blocked on infrastructure prerequisite (worker node bootstrap — sscoble/devops#3) and depend on each other.

Pattern: Same as sscoble/speckl#45 housekeeping close on 2026-06-02 (parent → child decomposition carried by the children's tickets). The parent ticket's body already documents the children inline.

This ticket is now archival; all forward progress happens in #10, #11, #12.

## Housekeeping close — decomposed into children Per CoS dispatch cycle 2026-06-03, closing this parent. All Phase 3 work now lives in the children: - **devops#10** (id 470) — ✅ CLOSED 2026-06-03 — [devops#4.1] Create CNPG cluster + verify Ready condition - **devops#11** (id 471) — open, labeled `blocked` — [devops#4.2] Migrate data from bare-metal Postgres to CNPG cluster (blocked on sscoble/devops#3 bootstrap-worker-nodes) - **devops#12** (id 472) — open — [devops#4.3] Cutover: Forgejo + Umami to CNPG, disable bare-metal **Status:** Step 1 of 3 shipped (CNPG cluster created and Ready). Steps 2 and 3 are blocked on infrastructure prerequisite (worker node bootstrap — sscoble/devops#3) and depend on each other. **Pattern:** Same as sscoble/speckl#45 housekeeping close on 2026-06-02 (parent → child decomposition carried by the children's tickets). The parent ticket's body already documents the children inline. This ticket is now archival; all forward progress happens in #10, #11, #12.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels   
blocked

   
bug

   
feature

   
improvement

   
question
No labels
blocked
bug
feature
improvement
question
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: sscoble/devops#4
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?