Cold-flush pipeline — hybrid timer (5min) + SIGTERM, NAS-backed #12

New issue

Closed

opened 2026-06-23 08:08:15 +00:00 by sscoble · 1 comment

sscoble commented

2026-06-23 08:08:15 +00:00

Owner

Why

D-cadence flush: batched timer (default 5 min) + on graceful shutdown. SIGTERM handler flushes state to cold Parquet before exit. Reject A (per-transition, too much I/O) and C (shutdown-only, kill -9 loses everything).

Files

api/flush.go (new — Store.Flush(), debouncer, background goroutine)
main.go (update — SIGTERM handler calls srv.Store.Flush() then exits)

Debouncer

flushCh chan struct{}, size 1, non-blocking send
Background goroutine: select { case <-time.After(s.flushInterval): s.Flush(); case <-s.flushCh: s.Flush(); case <-ctx.Done(): s.Flush(); return }
Bumps triggered by AddComment, CreatePost, PATCH — debounced to 1/sec so a burst doesn't trigger N flushes
POSTS_FLUSH_INTERVAL=0 disables the timer; flush is shutdown-only (useful for VOPR tests)

NAS path

POSTS_COLD_PATH=/mnt/nas/posts-cold (default)
Per-batch: <path>/<year>/<month>/<batch>-<unix>.parquet
Errors logged but do not panic; next flush retries
On unrecoverable NAS failure: log, alert, keep serving in memory (degrade, don't crash)

Acceptance

10k random mutations over 60s produce ≤ 12 cold flushes
SIGTERM (not SIGKILL) loses 0 mutations
kill -9 loses at most POSTS_FLUSH_INTERVAL worth
NAS failure does not crash the service; logs and alerts
POSTS_FLUSH_INTERVAL=0 mode: flush is shutdown-only, no timer goroutine

## Why D-cadence flush: batched timer (default 5 min) + on graceful shutdown. SIGTERM handler flushes state to cold Parquet before exit. Reject A (per-transition, too much I/O) and C (shutdown-only, kill -9 loses everything). ## Files - `api/flush.go` (new — `Store.Flush()`, debouncer, background goroutine) - `main.go` (update — SIGTERM handler calls `srv.Store.Flush()` then exits) ## Debouncer - `flushCh chan struct{}`, size 1, non-blocking send - Background goroutine: `select { case <-time.After(s.flushInterval): s.Flush(); case <-s.flushCh: s.Flush(); case <-ctx.Done(): s.Flush(); return }` - Bumps triggered by `AddComment`, `CreatePost`, `PATCH` — debounced to 1/sec so a burst doesn't trigger N flushes - `POSTS_FLUSH_INTERVAL=0` disables the timer; flush is shutdown-only (useful for VOPR tests) ## NAS path - `POSTS_COLD_PATH=/mnt/nas/posts-cold` (default) - Per-batch: `<path>/<year>/<month>/<batch>-<unix>.parquet` - Errors logged but do not panic; next flush retries - On unrecoverable NAS failure: log, alert, keep serving in memory (degrade, don't crash) ## Acceptance - 10k random mutations over 60s produce ≤ 12 cold flushes - SIGTERM (not SIGKILL) loses 0 mutations - `kill -9` loses at most `POSTS_FLUSH_INTERVAL` worth - NAS failure does not crash the service; logs and alerts - `POSTS_FLUSH_INTERVAL=0` mode: flush is shutdown-only, no timer goroutine

sscoble added the

labels 2026-06-23 08:08:15 +00:00

sscoble referenced this issue

2026-06-23 08:08:15 +00:00

Deploy to n97-mini: systemd unit, nginx route, port 8310/8320, K3s pod #16

agent-engineering commented

2026-07-30 21:57:49 +00:00

Closed — implementation already complete ✅

Verified at c2325: debounce burst collapse PASS, periodic timer PASS, SIGTERM flush PASS, acceptance spec PASS, empty flush skipped PASS. Files: flush.go (154 lines), flush_acceptance_test.go (207 lines). Closed by ticket dispatch system at cycle c2325.

## Closed — implementation already complete ✅ Verified at c2325: debounce burst collapse PASS, periodic timer PASS, SIGTERM flush PASS, acceptance spec PASS, empty flush skipped PASS. Files: flush.go (154 lines), flush_acceptance_test.go (207 lines). Closed by ticket dispatch system at cycle c2325.