mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
docs(agent): relax WP08 fork streaming requirements
This commit is contained in:
parent
d3ba857ef4
commit
d0963bf06a
1 changed files with 40 additions and 81 deletions
|
|
@ -34,15 +34,15 @@ export type ForkOptions =
|
|||
|
||||
`scope` is required. `branch` is required for branch scope. There is no default scope, no implicit `main`, and no compatibility alias.
|
||||
|
||||
**Branch scope** requires a complete configured source AgentLane: `pi.branch.tip/{branch}`, `pi.lane.config/{branch}`, and `pi.lane.state/{branch}` must all exist. A missing tip rejects (`unknown branch`); a data-only Branch (tip without config/state) rejects; a partial config/state pair, or `laneConfig`/`laneState` present without a `branchTip`, is corruption and rejects. `entryId`, when supplied, must be an entry on that Branch's current tip ancestry (inclusive); an entry elsewhere in the tree rejects. Omitted `entryId` means the current tip. `position` defaults to `"at"`; `"before"` selects the target's parent and may yield a `null` destination tip (before the root entry, or a `null` source tip with no `entryId`) — legal. The destination contains exactly one Branch with the same name, the selected tip, the copied `LaneConfiguration`, and fresh idle lane state `{ currentOperationId: null, lastOperationId: null, inbox: [] }`. No other Branch or lane exists in the destination.
|
||||
**Branch scope** requires a complete configured source AgentLane: `pi.branch.tip/{branch}`, `pi.lane.config/{branch}`, and `pi.lane.state/{branch}` must all exist. A missing tip rejects (`unknown branch`); a data-only Branch (tip without config/state) rejects. Forking does not audit unrelated lanes or malformed lane records: supported writers create lane state atomically, so branch validation checks only the selected Branch's required tip, config, and state. `entryId`, when supplied, must be an entry on that Branch's current tip ancestry (inclusive); an entry elsewhere in the tree rejects. Omitted `entryId` means the current tip. `position` defaults to `"at"`; `"before"` selects the target's parent and may yield a `null` destination tip (before the root entry, or a `null` source tip with no `entryId`) — legal. The destination contains exactly one Branch with the same name, the selected tip, the copied `LaneConfiguration`, and fresh idle lane state `{ currentOperationId: null, lastOperationId: null, inbox: [] }`. No other Branch or lane exists in the destination.
|
||||
|
||||
This deliberately makes branch forks of data-only Branches — including legacy v3 imports whose main lacked reconstructable model/thinking history — unavailable. Tree scope remains available for those sources.
|
||||
|
||||
**Tree scope** copies: every immutable entry, including entries unreachable from every current Branch tip; every Branch tip verbatim; every configured lane's config plus fresh idle lane state under the same name; data-only Branches (tip present, config/state absent) as data-only Branches. A partial config/state pair, or `laneConfig`/`laneState` without a `branchTip`, is corruption in both scopes: tree scope **rejects** it rather than dropping it.
|
||||
**Tree scope** copies: every immutable entry, including entries unreachable from every current Branch tip; every Branch tip verbatim; every configured lane's config plus fresh idle lane state under the same name; data-only Branches (tip present, config/state absent) as data-only Branches. Forking does not perform a corruption audit of lane records; supported writers create valid lane state atomically.
|
||||
|
||||
**Both scopes**: copy `pi.session.name`; copy `pi.entry.label` values only for copied entries; exclude the usage ledger, `pi.result`, all `pi.op.*`, all `pi.pending.*` (entries, tool checkpoints, assistant frames), and every trace of open-operation state. Destination usage totals start at zero; destination `messageCount` equals the number of copied message entries, exactly as current conformance already proves. Destination metadata records `parentSessionId = source.id`. Copied entries, values, and list elements preserve their original `seq`; the destination `nextSeq` high-water mark is the source's, so no sequence is reused. Transformed built-ins keep coherent seqs: the rewritten `branchTip` and each fresh idle `laneState` reuse the seq of the source's current corresponding value row, and copied config, name, labels, application values, and list elements retain theirs — destination writes stay in original seq order under an unchanged high-water mark.
|
||||
**Both scopes**: copy `pi.session.name`; copy `pi.entry.label` values only for copied entries; exclude the usage ledger, `pi.result`, all `pi.op.*`, all `pi.pending.*` (entries, tool checkpoints, assistant frames), and every trace of open-operation state. Destination usage totals start at zero; destination `messageCount` equals the number of copied message entries, exactly as current conformance already proves. Destination metadata records `parentSessionId = source.id`. Sequence preservation is not part of the fork contract: backends may allocate destination-local sequences for copied and transformed writes, provided destination sequence and list-cursor semantics remain internally valid.
|
||||
|
||||
**Application values/lists (outside the reserved `pi`/`pi.*` namespaces)**: tree scope copies every current scalar value and every surviving list element with original seqs; branch scope copies none. A `seq <= tipSeq` cutoff is forbidden as "historical reconstruction" — it is provably not one:
|
||||
**Application values/lists (outside the reserved `pi`/`pi.*` namespaces)**: tree scope copies every current scalar value and every surviving list element; branch scope copies none. A `seq <= tipSeq` cutoff is forbidden as "historical reconstruction" — it is provably not one:
|
||||
|
||||
```text
|
||||
Scalar: TX[seq 10: set my-app.state = v1] · TX[seq 12: insert e1] ·
|
||||
|
|
@ -70,29 +70,29 @@ All namespace fork knowledge lives in one core module beside `session/values.ts`
|
|||
- The exact namespace `pi` and any other `pi.*` namespace → the fork **fails**, but only when current surviving state exists at fork time (a current scalar row or a surviving list element). Historical writes later replaced or deleted are absent from current state on every backend and must not alone fail a JSONL fork — behavior is backend-equivalent. Introducing a new built-in namespace without declaring its fork semantics must break fork tests, not silently copy or drop state.
|
||||
- Neither `pi` nor `pi.*` → application: copy on tree, exclude on branch. The only built-in list namespace (`pi.pending.assistant_frame`) excludes; application lists follow the application rule.
|
||||
|
||||
The driver exposes a streaming shape — accept one committed value/list write (or current row), emit zero or more destination writes, `finish()` emits fresh idle lane states and the rewritten branch tip — holding only bounded state (lane names, destination tip). Entry-copy membership is a backend-supplied predicate so each backend uses its own index. `createForkSnapshot`, `forkSnapshotWrites`, `ForkSourceSnapshot`, `ForkDestinationSnapshot`, and the `entriesComplete` escape hatch are deleted.
|
||||
The driver exposes a streaming shape — accept one committed value/list write (or current row), emit zero or more destination writes, `finish()` emits fresh idle lane states and the rewritten branch tip. Entry-copy membership is a backend-supplied predicate so each backend uses its own index. `createForkSnapshot`, `forkSnapshotWrites`, `ForkSourceSnapshot`, `ForkDestinationSnapshot`, and the `entriesComplete` escape hatch are deleted.
|
||||
|
||||
### 1.3 Bounded-memory backend procedures
|
||||
### 1.3 Streaming backend procedures
|
||||
|
||||
Auxiliary memory during a fork must stay bounded regardless of source size. The unavoidable exceptions are Memory's destination state and the returned open destination Session (JSONL destination state after open). Source copy paths must not call array-returning full reads: no `snapshotEntriesAndValues()`, `captureForkSource()`, `readAllScalarValueRows()`, `readAllEntryRows()`, whole-file `readTextFile`, or SQLite `.all()` over unbounded row sets.
|
||||
Forks must stream source writes rather than materializing full source snapshot arrays. Source copy paths must not call array-returning full reads: no `snapshotEntriesAndValues()`, `captureForkSource()`, `readAllScalarValueRows()`, `readAllEntryRows()`, whole-file `readTextFile`, or SQLite `.all()` over unbounded row sets. JSONL may retain its structural index and legacy-v3 compaction tail in memory.
|
||||
|
||||
**Memory.** At a source `commitQueue` boundary, build the destination `InMemoryStorageState` directly by iterating source maps once through the classifier/driver. Branch scope walks `parentId` from the branch tip toward the root to compute the ancestry id set and verify `entryId` membership; copy exactly that set. No intermediate snapshot arrays; `MemoryStorage.fromSnapshot` and `captureForkSource` are removed.
|
||||
|
||||
**JSONL.** The source is read through a read-only path that captures a fixed file prefix — open a read handle, record the current length at capture time, and read only that prefix — and never writes the source: no torn-tail truncation/rewrite, no v3 normalization persistence, no `.tmp` beside the source. A torn or incomplete final line inside the captured prefix is discarded in memory. The source `nextSeq` high-water mark is derived as `max(header.nextSeq when present, highest complete captured write seq + 1)` — ordinary headers are not rewritten on every append, and a torn incomplete final write never advances it.
|
||||
**JSONL.** The source is read through a read-only path that captures the source's current `nextSeq` at the commit-queue boundary; both scans emit only complete writes below that sequence. The fork never writes the source: no torn-tail truncation/rewrite, no v3 normalization persistence, no `.tmp` beside the source. A torn or incomplete final line is discarded in memory. This relies on format-4 sources remaining append-only and not being replaced while the fork runs; compaction/replacement coordination is deferred with J1.
|
||||
|
||||
- *Both scopes* use two passes over the captured prefix with one temporary disk-backed index (deleted in `finally` on success and failure). Pass 1 streams every line and folds value/list writes into the index: per scalar address, the seq of the current surviving `set` (absent after a trailing `delete`); per list address, the set of surviving element seqs (whole-list delete clears it); plus entry `id → parentId` (parents always precede children) where branch scope needs ancestry. The fold applies the same current-state semantics replay does, without whole-state RAM.
|
||||
- Pass 2 streams the prefix again in order and emits a write only when the index proves it current and the classifier selects it: an entry write in the selected set; a scalar `set` whose seq equals the index's current-row seq for its address; a list `append` whose element seq is in the survivor set. Deletes, superseded sets, and dead application value/list history are never emitted as destination state — the destination file contains only current selected rows, already in original seq order because source lines are seq-ordered.
|
||||
- Transformed built-ins emit in place: when pass 2 reaches the seq of the source's current `branchTip` row for a kept branch it emits the (possibly rewritten) tip under that same seq, and when it reaches the seq of the current `laneState` row for a kept configured lane it emits fresh idle lane state under that same seq. No tail appends exist, so replay stays seq-monotonic and the high-water mark is untouched.
|
||||
- *Branch scope* walks tip→root through the disk index to materialize the ancestry set on disk and verify `entryId` membership before pass 2; labels emit only for member entries.
|
||||
- The destination is staged in a temp file and atomically renamed (existing `publishFileAtomically`); failure removes the temp file and the index. Legacy v3 splits by source state. Forking an **open** legacy-v3 `JsonlStorage` **rejects with a clear error** until a normal non-empty commit has upgraded and persisted its normalized format-4 ids: v3 normalization mints nondeterministic UUIDv7 tails, so an independent disk reparse cannot reproduce the ids the open Session exposes or validate a caller `entryId`; the fork must not mutate/upgrade the source itself. A **closed** legacy-v3 source uses the bounded fixed-prefix disk-backed parser/normalizer and never touches the file: tree forks succeed; branch forks may use an omitted `entryId` (the default normalized tip) when the import reconstructs a complete configured lane, while any caller-supplied process-local id from an earlier open is not stable across reparses and rejects normally; a data-only reconstructed lane rejects as everywhere. Ordinary writable `JsonlStorage.open` may retain in-memory normalization, but resident state is never a fork source.
|
||||
- There is **no resident-state path**: an open same-repository JSONL source enqueues a short boundary callback on the source `commitQueue` whose only job is to capture the fixed read-only file-handle prefix (open handle, record length), then releases the queue; the fork runs the same two-pass disk-backed procedure over that prefix while later source appends proceed. Committed writes are durable in the file before they are applied to resident state, so the prefix is authoritative at the boundary, and file order preserves seq order without resident-Map ordering ambiguity.
|
||||
- *Both scopes* use two scans up to the captured sequence boundary. Pass 1 streams every line and folds value/list writes into an in-memory structural index: per scalar address, the seq of the current surviving `set` (absent after a trailing `delete`); per list address, the set of surviving element seqs (whole-list delete clears it); plus entry `id → parentId` where branch scope needs ancestry. The fold applies the same current-state semantics replay does.
|
||||
- Pass 2 streams again and emits a write only when the index proves it current and the classifier selects it: an entry write in the selected set; a scalar `set` whose seq equals the index's current-row seq for its address; a list `append` whose element seq is in the survivor set. Deletes, superseded sets, and dead application value/list history are never emitted as destination state.
|
||||
- Transformed built-ins emit when pass 2 reaches the source's current corresponding row for a kept branch or configured lane. The destination writer may assign destination-local sequences.
|
||||
- *Branch scope* walks tip→root through the in-memory index to materialize the ancestry set and verify `entryId` membership before pass 2; labels emit only for member entries.
|
||||
- The destination is staged in a temp file and atomically renamed (existing `publishFileAtomically`); failure removes the temp file. Legacy v3 splits by source state. Forking an **open** legacy-v3 `JsonlStorage` **rejects with a clear error** until a normal non-empty commit has upgraded and persisted its normalized format-4 ids: v3 normalization mints nondeterministic UUIDv7 tails, so an independent disk reparse cannot reproduce the ids the open Session exposes or validate a caller `entryId`; the fork must not mutate/upgrade the source itself. A **closed** legacy-v3 source uses the read-only parser/normalizer and never touches the file: tree forks succeed; branch forks may use an omitted `entryId` (the default normalized tip) when the import reconstructs a complete configured lane, while any caller-supplied process-local id from an earlier open is not stable across reparses and rejects normally; a data-only reconstructed lane rejects as everywhere. The parser may retain structural metadata and compaction `retainedTail` context in memory. Ordinary writable `JsonlStorage.open` may retain in-memory normalization, but resident state is never a fork source.
|
||||
- There is **no resident-state path**: an open same-repository JSONL source enqueues a short boundary callback on the source `commitQueue` whose only job is to capture `nextSeq`, then releases the queue; the fork runs the same two-scan procedure below that boundary while later source appends proceed. Committed writes are durable in the file before they are applied to resident state, so the file is authoritative at the boundary.
|
||||
|
||||
**SQLite.** Iterator-based row transfer through a bounded-memory temporary on-disk SQLite **staging database**, used uniformly for per-file and shared-container layouts. The source reader never streams directly into the destination transaction — a WAL reader and one writer can coexist, but in shared-container mode the destination writer would hold the container's sole write lock and block the post-boundary source writer while capture is still streaming; staging into a separate file removes that coupling. Use prepared-statement iteration (`iterate`/stepwise), never source-sized `.all()` snapshot arrays:
|
||||
|
||||
- *External/closed/live-worker source:* WP07 path — `openReadOnly` on the exact canonical path, one deferred read transaction, session row and storage version validated inside it.
|
||||
- *Same-repository open source:* open the independent read-only connection **first**, then enqueue a short boundary callback on the source Storage `commitQueue` whose only job is to `BEGIN` and establish the independent reader's snapshot (issue a trivial read) before releasing the queue. Do not begin a read transaction on the source writer connection and then release its queue, and do not hold the queue for the copy duration.
|
||||
- *Stage:* while the source reader remains open, stream selected entries `ORDER BY seq` and classifier-selected current scalar/list rows (SQL-level namespace prefilters matching the classifier — enumerated built-in namespaces plus the application predicate excluding exact `pi` and `pi.%` — with each row still passing the classifier) into the temporary staging database in bounded batches. Branch scope enumerates the ancestry through the `branch_entries` segment chain up to the selected entry and answers `entryId` and label membership through that index, not an in-RAM id set. Later source commits use the original writer connection and may complete while staging streams, in **both** layouts, because stage writes target another file.
|
||||
- *Publish:* close/commit the source read transaction, then stream the stage into one destination `BEGIN IMMEDIATE` transaction — entries in seq order maintaining the destination branch index and `message_count` incrementally, then values/list elements — and delete the staging database in `finally` on success and failure. Destination `next_seq` is the source's. Shared-container destinations write only the new session's rows.
|
||||
- *Publish:* close/commit the source read transaction, then stream the stage into one destination `BEGIN IMMEDIATE` transaction — entries in source order maintaining the destination branch index and `message_count` incrementally, then values/list elements — and delete the staging database in `finally` on success and failure. The destination allocates its own valid sequence range. Shared-container destinations write only the new session's rows.
|
||||
|
||||
### 1.4 Preserved WP07 behavior
|
||||
|
||||
|
|
@ -107,7 +107,7 @@ Destination id reservation across create/open/fork/delete, no-create opens, fore
|
|||
- `ForkOptions` defaults `scope` to `"branch"` and hard-codes source/destination `main` (`fork.ts`, both backend branch readers).
|
||||
- No ancestry membership check: `selectForkContents` walks parents from any supplied `entryId` anywhere in the tree and labels the result `main`.
|
||||
- Every backend materializes the full source: `snapshotEntriesAndValues()` (Memory/JSONL), `readAllScalarValueRows` + `readAllEntryRows`/`scanBranchEntries` arrays (SQLite), all funneled through in-memory `createForkSnapshot`.
|
||||
- `createForkSnapshot` renumbers copied scalar values with fresh seqs and computes `nextSeq` as `max entry seq + 1`, losing original value seqs.
|
||||
- `createForkSnapshot` and its array snapshots force all backends through a materialized in-memory source copy.
|
||||
- A JSONL fork of a closed source uses `JsonlStorage.open()`, which rewrites the source file on a torn tail — a fork can mutate its source today.
|
||||
- Lists are never copied; application values are excluded with no declared tree policy (`values.md` explicitly defers it).
|
||||
- Namespace fork knowledge is repeated across `fork.ts`, `values.md` prose, and conformance assertions; nothing forces a new `pi.*` namespace to declare fork semantics.
|
||||
|
|
@ -118,7 +118,7 @@ Destination id reservation across create/open/fork/delete, no-create opens, fore
|
|||
1. `ForkOptions` and validation exactly as §1.1; all rejection paths create no destination file, database rows, or reserved-but-leaked ids.
|
||||
2. The closed classifier/driver of §1.2, exported from core and consumed by all three backends; unknown reserved namespaces (exact `pi` or undeclared `pi.*`) fail the fork only when current surviving state exists, identically on every backend.
|
||||
3. Streaming procedures of §1.3 on all three backends; `createForkSnapshot`/`captureForkSource`/`snapshot()` fork plumbing and their exports removed from `session/index.ts` and the sqlite-node import surface.
|
||||
4. Seq preservation (including reused seqs for rewritten tips and fresh idle lane states) and source `nextSeq` high-water on every backend (JSONL derives it per §1.3); identical logical destination state across backends for identical sources (conformance).
|
||||
4. Valid destination-local sequence allocation on every backend; identical logical destination state across backends for identical sources (conformance).
|
||||
5. JSONL source non-mutation, including torn-tail sources and legacy v3 sources.
|
||||
6. Documentation: `harness.md` §2.7 (and the §1.7 fork-related sentences), `values.md` "Forks and rewrites" and backend snapshot mentions, `post-wp05-roadmap.md` (retire the SQLite fork-cost item, add/point to this package), `packages/session-backends/sqlite-node/README.md` fork paragraphs, `packages/agent/benchmark/session/README.md` if dataset wording changes. Historical WP docs and released changelogs untouched. Format/storage versions unchanged; no migration.
|
||||
|
||||
|
|
@ -136,9 +136,9 @@ Files: `session/types.ts`, new `session/fork-policy.ts`, `session/fork.ts` (rewr
|
|||
|
||||
Files: `session/jsonl/storage.ts`, `session/jsonl/repo.ts`, `session/jsonl/types.ts`, streaming reader support in the `FileSystem` capability (`harness/types.ts`, `harness/env/*`) if required, `test/harness/jsonl-session-repo.test.ts`, `jsonl-session-repo-conformance.test.ts`, `jsonl-storage.test.ts`, `jsonl-v3-migration.test.ts`.
|
||||
|
||||
1. Read-only fixed-prefix source capture with in-memory torn-tail discard; the same capture at the source `commitQueue` boundary for open sources; no source writes on any fork path.
|
||||
2. Two-pass disk-backed current-state fold and emission for both scopes; branch ancestry membership through the disk index; in-place transformed built-ins; atomic destination publish.
|
||||
3. Open-v3 fork rejection and the closed-v3 disk-backed parser/normalizer path; update v3 fork tests per §5.
|
||||
1. Read-only sequence-boundary capture with in-memory torn-tail discard; the same capture at the source `commitQueue` boundary for open sources; no source writes on any fork path.
|
||||
2. Two-scan in-memory current-state fold and emission for both scopes; branch ancestry membership through the structural index; atomic destination publish.
|
||||
3. Open-v3 fork rejection and the closed-v3 read-only parser/normalizer path; update v3 fork tests per §5.
|
||||
|
||||
|
||||
### Slice C — SQLite streaming
|
||||
|
|
@ -159,36 +159,36 @@ Update fork option literals, add large-source fork benchmarks (tree and branch),
|
|||
|
||||
### Contract and validation
|
||||
|
||||
- branch scope rejects: unknown branch name; data-only Branch (tip only); partial config/state pair (corruption); `entryId` not on the named Branch's tip ancestry (present elsewhere in the tree); `entryId` unknown; `entryId` with a `null` tip. Each rejection leaves no destination artifact and releases its reserved id.
|
||||
- branch scope rejects: unknown branch name; data-only Branch (tip only); `entryId` not on the named Branch's tip ancestry (present elsewhere in the tree); `entryId` unknown; `entryId` with a `null` tip. Each rejection leaves no destination artifact and releases its reserved id. Fork tests do not construct or audit malformed lane records outside the selected Branch.
|
||||
- branch scope accepts: omitted `entryId` (tip), explicit tip, mid-ancestry entry, `position: "before"` at a mid entry and at the root entry (`null` destination tip), `null` source tip with no `entryId`.
|
||||
- destination shape: exactly one Branch, same name, copied config, fresh idle lane state, no other lane values, `parentSessionId` set.
|
||||
- tree scope: unreachable entries copied; every tip copied; configured lanes get config plus fresh idle state; data-only Branches stay data-only; partial pairs fault.
|
||||
- tree scope: unreachable entries copied; every tip copied; configured lanes get config plus fresh idle state; data-only Branches stay data-only. Forking does not audit malformed lane records.
|
||||
- both scopes: session name copied; labels only for copied entries (branch fork excludes labels of non-ancestry entries); `pi.result`, `pi.op.*`, `pi.pending.*` (pending entries, tool checkpoints, assistant frame lists), and usage rows absent; exact stats expectation — a fork copying N message entries reports `getStats()` = `{ messageCount: N, usage: all-zero }`, matching current conformance; the ledger-completeness invariant ("a fork's ledger starts at zero") retained.
|
||||
|
||||
### Application values and lists
|
||||
|
||||
- tree fork copies every non-`pi.*` scalar and every surviving list element with original seqs; list cursors from the source page identically in the destination; a list deleted-then-reappended in the source reproduces only survivors.
|
||||
- tree fork copies every non-`pi.*` scalar and every surviving list element; a list deleted-then-reappended in the source reproduces only survivors.
|
||||
- branch fork copies no application values/lists (pin the §1.1 scalar and list traces as regression cases: post-fork-point overwrite and delete-destroyed elements must not resurface under any implementation).
|
||||
- current surviving state in an unknown reserved namespace (exact `pi` or an undeclared `pi.*` scalar or surviving list element) fails the fork on every backend; the same namespace **set then deleted** before the fork fails no backend — including JSONL, where the dead history remains as physical lines and the disk fold must classify it as absent. Construct both cases via raw committed writes.
|
||||
|
||||
### Sequence preservation
|
||||
### Sequence allocation
|
||||
|
||||
- copied entries, values, and list elements keep source seqs; rewritten branch tips and fresh idle lane states occupy the seqs of the source's current corresponding rows; destination `nextSeq` equals the source high-water mark and the first post-fork commit allocates above it on every backend (JSONL header `nextSeq`, SQLite `next_seq`).
|
||||
- JSONL derives the high-water mark as `max(header.nextSeq when present, highest complete captured write seq + 1)`; a torn incomplete final write does not advance it.
|
||||
- sequence preservation, source high-water marks, and source list cursors are not part of the fork contract. Each destination assigns valid local sequences and list cursors; its first post-fork commit must allocate without collision.
|
||||
- JSONL captures source `nextSeq` only to establish the append-only source boundary; a torn incomplete final write does not belong to the fork.
|
||||
|
||||
### Bounded memory and source non-mutation
|
||||
### Streaming source reads and source non-mutation
|
||||
|
||||
- instrumented source readers (test decorators/spies) assert fork paths never call `snapshotEntriesAndValues`, `captureForkSource`, `readAllScalarValueRows`, `readAllEntryRows`, whole-file `readTextFile`, or source-sized `.all()` arrays on entries/values/lists — bounded iterator steps and staging-database operations are permitted; deterministic large fixtures (extend the existing benchmark dataset generators) exercise tree and branch forks over multi-thousand-entry sources.
|
||||
- instrumented source readers (test decorators/spies) assert fork paths never call `snapshotEntriesAndValues`, `captureForkSource`, `readAllScalarValueRows`, `readAllEntryRows`, whole-file `readTextFile`, or source-sized `.all()` arrays on entries/values/lists; deterministic large fixtures (extend the existing benchmark dataset generators) exercise tree and branch forks over multi-thousand-entry sources.
|
||||
- SQLite staging databases are removed on success and failure; a failed fork leaves neither a stage file nor destination rows.
|
||||
- JSONL: fork of a closed torn-tail source succeeds, the destination excludes the torn transaction wholly, and the source file bytes are unchanged (byte-compare before/after); fork of a legacy v3 source leaves the source file unchanged; no `.tmp` appears beside the source; temporary ancestry index files are removed on success and failure.
|
||||
- JSONL fixed-prefix capture: a source append completed after capture is wholly absent from the fork.
|
||||
- JSONL destination content: the destination file contains only current selected scalar rows and surviving list elements in original seq order — no delete records, superseded sets, or dead application history; the rewritten tip and fresh idle lane state sit at the source current rows' seqs.
|
||||
- legacy v3: forking an **open** v3 source rejects with the clear pre-upgrade error; after one normal non-empty commit upgrades it to format 4, a fork succeeds and preserves the persisted ids; a **closed** v3 tree fork succeeds through the disk-backed parser (instrumented: no whole-state in-memory normalization on the fork path) and leaves the source byte-identical; a closed v3 branch fork with omitted `entryId` succeeds when a complete configured lane is reconstructed, a caller-supplied id from an earlier open rejects, and a data-only reconstructed main rejects.
|
||||
- JSONL: fork of a closed torn-tail source succeeds, the destination excludes the torn transaction wholly, and the source file bytes are unchanged (byte-compare before/after); fork of a legacy v3 source leaves the source file unchanged; no `.tmp` appears beside the source.
|
||||
- JSONL sequence-boundary capture: a source append completed after capture is wholly absent from the fork.
|
||||
- JSONL destination content: the destination file contains only current selected scalar rows and surviving list elements — no delete records, superseded sets, or dead application history.
|
||||
- legacy v3: forking an **open** v3 source rejects with the clear pre-upgrade error; after one normal non-empty commit upgrades it to format 4, a fork succeeds and preserves the persisted ids; a **closed** v3 tree fork succeeds through the read-only parser and leaves the source byte-identical; a closed v3 branch fork with omitted `entryId` succeeds when a complete configured lane is reconstructed, a caller-supplied id from an earlier open rejects, and a data-only reconstructed main rejects.
|
||||
|
||||
### Coordination and ordering
|
||||
|
||||
- Memory open-source forks keep the queue-boundary conformance case: a commit admitted before the fork appears wholly; one admitted after does not.
|
||||
- JSONL open-source forks capture the fixed prefix at the `commitQueue` boundary: a commit admitted before the boundary appears wholly; a source append completed after boundary capture proceeds while the fork is still streaming and is wholly absent from that fork.
|
||||
- JSONL open-source forks capture the sequence boundary at the `commitQueue` boundary: a commit admitted before the boundary appears wholly; a source append completed after boundary capture proceeds while the fork is still streaming and is wholly absent from that fork.
|
||||
- SQLite same-repo: the independent reader opens first, the boundary callback on the source `commitQueue` establishes its snapshot and releases the queue, and a subsequent source commit on the writer connection **completes while the reader is still streaming into the stage** and is wholly absent from that fork; a later fork includes it wholly. Assert this in **both** per-file and shared-container layouts (satisfiable in shared containers because stage writes target another file), plus the WP07 live external-worker case unchanged.
|
||||
- destination-reservation races (create vs fork on one id, both orders) unchanged.
|
||||
|
||||
|
|
@ -203,7 +203,7 @@ After each slice: `npm run check`, then the modified focused tests via the repos
|
|||
Review checkpoints (delegated reviews use provider `anthropic`, model `claude-fable-5`):
|
||||
|
||||
1. After Slice A: contract, classifier closure, conformance shape.
|
||||
2. After Slice C: same-repo ordering, reader independence, bounded-memory evidence.
|
||||
2. After Slice C: same-repo ordering, reader independence, and streaming-source evidence.
|
||||
3. Final review of source, tests, docs, and exclusions.
|
||||
|
||||
## 7. Exclusions
|
||||
|
|
@ -229,59 +229,18 @@ WP08 is complete when:
|
|||
|
||||
- `ForkOptions` is exactly the §1.1 union with the stated validation, on all three backends;
|
||||
- branch forks require a complete configured source AgentLane and enforce ancestry membership; tree forks copy the complete immutable tree, every tip, configured and data-only Branches;
|
||||
- tree forks carry all current application values and surviving list elements with original seqs; branch forks carry none;
|
||||
- tree forks carry all current application values and surviving list elements; branch forks carry none;
|
||||
- one closed core classifier owns every namespace disposition; unknown reserved namespaces (exact `pi` or undeclared `pi.*`) fail forks exactly when current surviving state exists, equivalently on all backends;
|
||||
- all supported fork paths — including closed legacy v3 sources — are streaming and bounded in auxiliary memory per §1.3, proven by instrumented-reader tests over large fixtures; forking an open legacy-v3 source is explicitly unsupported (clear rejection) until an ordinary commit upgrades it, not handled by an in-memory exception;
|
||||
- JSONL forks never mutate their source, including torn-tail and legacy v3 sources, and JSONL destinations contain only current selected rows in original seq order;
|
||||
- all supported fork paths — including closed legacy v3 sources — stream source writes without materializing full source snapshot arrays, proven by instrumented-reader tests over large fixtures; JSONL may retain structural indexes and legacy compaction context in memory; forking an open legacy-v3 source is explicitly unsupported (clear rejection) until an ordinary commit upgrades it, not handled by an in-memory exception;
|
||||
- JSONL forks never mutate their source, including torn-tail and legacy v3 sources, and JSONL destinations contain only current selected rows;
|
||||
- SQLite forks stage through a temporary on-disk database (source reader → stage while open, stage → one destination `BEGIN IMMEDIATE` after reader close, stage deleted in `finally`), with the independent-reader boundary design and concurrent later writer commits proven in both layouts, preserving all WP07 behavior;
|
||||
- copied seqs and `nextSeq` high-water marks are preserved and backends produce identical logical destinations;
|
||||
- destinations allocate valid local sequences and backends produce identical logical destinations;
|
||||
- shared conformance, focused backend tests, benchmarks, `npm run check`, and `./test.sh` pass;
|
||||
- normative docs and the roadmap reflect the new contract with historical documents untouched;
|
||||
- final Fable review reports no blocker.
|
||||
|
||||
## 9. Open questions / potential spec changes
|
||||
## 9. TODO
|
||||
|
||||
These record implementation decisions and potential specification changes. The requirements above remain unchanged until explicitly revised.
|
||||
|
||||
### 9.1 Corruption checks
|
||||
|
||||
Current: reject any malformed lane. Possible relaxation: trust supported atomic state and perform no fork-time corruption audit; branch forks only require the selected tip, config, and state. This removes global lane inventory validation.
|
||||
|
||||
### 9.2 Sequence preservation
|
||||
|
||||
Current: preserve source sequences, `nextSeq`, and list cursors. Possible relaxation: keep entry sequences but assign destination-local sequences to values, list elements, and reconstructed lane rows. Revisit if preservation materially complicates streaming.
|
||||
|
||||
### 9.3 JSONL source capture boundary
|
||||
|
||||
Original requirement: capture the exact source file length at the commit-queue boundary and have both passes read only up to that length through the same fixed file handle.
|
||||
|
||||
Current implementation: capture the source's current `nextSeq` at the commit-queue boundary and have both independently opened scans stop before that sequence. This is simpler and avoids adding byte-range semantics to `FileSystem`, while still excluding later append-only commits transaction by transaction.
|
||||
|
||||
The current implementation assumes a format-4 source remains append-only and is not replaced while the fork runs. Concurrent compaction or another whole-file replacement could otherwise make the two scans observe different file incarnations. Decide whether to restore fixed-length capture, retain the sequence boundary with an explicit no-replacement invariant, or add replacement coordination when J1 compaction is implemented.
|
||||
|
||||
### 9.4 Legacy-v3 normalization memory
|
||||
|
||||
Original requirement: normalize a closed legacy-v3 source through a bounded disk-backed parser and index.
|
||||
|
||||
Current implementation does not do that. Instead, it:
|
||||
|
||||
- scans the v3 file once for structure, stable IDs, labels/configuration, usage, and current values;
|
||||
- does not keep all message payloads in memory during that scan;
|
||||
- later reopens the v3 file to stream selected normalized writes directly into the fork output;
|
||||
- no longer writes an intermediate normalized v4 temp file.
|
||||
|
||||
For compactions, the replay pass may cache needed context messages so it can build `retainedTail`. Later append-only records are excluded by the captured complete-record count.
|
||||
|
||||
The source must not be edited or replaced between passes. Header and physical-entry checks catch some changes, but this is not snapshot isolation.
|
||||
|
||||
Memory is improved versus storing all normalized writes, but can still grow large if selected compactions need many context messages. Decide whether this is acceptable, or whether WP08 still needs the originally planned disk-backed implementation.
|
||||
|
||||
### 9.5 JSONL structural-index memory
|
||||
|
||||
The current fork implementation uses in-memory structural indexes. For v4, this includes scalar/list state, entry parent links, selected ancestry, and lane inventory. For v3, this includes structural metadata, stable ID mappings, derived values, and the compaction context-message cache described in §9.4.
|
||||
|
||||
This does not meet the disk-backed indexing requirement in §1.3. Decide whether in-memory structural indexes are acceptable, or whether WP08 still requires disk-backed indexes before completion.
|
||||
|
||||
### 9.6 TODO: Cleanup after invocation cancellation
|
||||
### Cleanup after invocation cancellation
|
||||
|
||||
Cancellation-safe filesystem cleanup is deferred. Cleanup currently receives the caller's Context, so an aborted invocation can leave a staged file behind even though readers close and the destination id reservation is released. Define the cleanup Context policy before implementing this; do not import Chord's `withoutAbortSignal` directly into JSONL code as a temporary workaround.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue