From 967246214355009a3e3b916cba8fd62cd4a162db Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Wed, 23 Sep 2026 00:26:05 +0200 Subject: [PATCH] docs: refine Pico5 JSONL publication design --- packages/durable/docs/pico-v5-handoff.md | 34 +++++++++++++++++++++--- packages/durable/docs/pico-v5.md | 11 +++++--- 2 files changed, 38 insertions(+), 7 deletions(-) diff --git a/packages/durable/docs/pico-v5-handoff.md b/packages/durable/docs/pico-v5-handoff.md index 99de33aad..b3787b7a8 100644 --- a/packages/durable/docs/pico-v5-handoff.md +++ b/packages/durable/docs/pico-v5-handoff.md @@ -55,12 +55,38 @@ deleted-page reuse, and representative storage sizes. Implement table writes in `main.jsonl`, one document sidecar per incarnation, one sidecar per live task, and one main marker per commit. Do not add a -standalone-sidecar protocol. Serialization must also provide the storage ownership -boundary: retained indexes/materializations are detached from write arguments, -and reads never expose backend-owned cached objects. +standalone-sidecar protocol. + +Copy, rather than import, the current `ExecutionEnv`, `FileSystem`, `Shell`, Node +implementation, and their required utility files from `packages/agent/src/harness` +into `packages/durable/src/env`. Copy only the environment-related slice, not +agent skills, prompts, telemetry, or tool definitions. Extend the copied +filesystem contract with exact-byte file truncation and file flushing. Keep the +portable environment and JSONL entry points free of Node built-ins; expose Node +implementations only from `/env/node` and `/storage/jsonl/node`. Do not use the +Pico3 implementation as source material. + +Refactor the current `MemoryStorage` state machinery into a two-phase prepared +mutation: validation and detachment produce a candidate that can later be +applied without failure. Build `MemoryStorage.commit()` on that pair, and reuse +the same machinery for JSONL. JSONL must append every prepared sidecar record, +append the main marker, and only then apply the prepared in-memory mutation. +Serialization or preparation failure occurs before file I/O and does not poison +the backend. Retained indexes/materializations remain detached from write +arguments, and reads never expose backend-owned cached objects. + +JSONL creation has `fsync?: boolean`, defaulting to `false`. With `false`, append +sidecars and then the marker without an explicit flush. With `true`, append all +affected sidecars, flush each affected sidecar, and then append the main marker. +Do not explicitly flush `main.jsonl`. A main-only commit has no sidecars to +flush. Any uncertain append or flush failure poisons the open backend and +publishes no prepared in-memory mutation. Fault-test torn/short sidecar writes, failures between sidecars, every marker -boundary, unconfirmed tails, missing confirmed data, and poisoned writes. +boundary, unconfirmed tails, missing confirmed data, poisoned writes, exact-byte +tail truncation, both fsync settings and their call ordering, detached retained +state and reads, and browser-safe portable entry points. Run the complete storage +conformance suite directly and after reopen. ## 5. JSONL reclamation diff --git a/packages/durable/docs/pico-v5.md b/packages/durable/docs/pico-v5.md index 6a1abe2b4..b7388e897 100644 --- a/packages/durable/docs/pico-v5.md +++ b/packages/durable/docs/pico-v5.md @@ -2115,9 +2115,14 @@ Publication protocol: 3. Publish in memory only after the marker write succeeds. Every commit uses this protocol; there is no standalone-sidecar fast path. -Without `fsync`, it guarantees ordinary process-crash consistency, not survival -of power, host, kernel, or filesystem failure. Durable mode flushes sidecars -before the marker. +JSONL creation accepts an `fsync` option that defaults to `false`. Without +`fsync`, it guarantees ordinary process-crash consistency, not survival of +power, host, kernel, or filesystem failure. With `fsync: true`, the backend +appends all affected sidecar records, flushes each affected sidecar, and only +then appends the main marker. It does not explicitly flush `main.jsonl`; an +acknowledged tail commit may therefore still disappear, but a marker that +survives should not overtake its sidecar data. A main-only commit has no +sidecars to flush. Recovery: