docs(agent): restore non-Pico5 documentation

This commit is contained in:
Mario Zechner 2026-09-18 13:48:19 +02:00
parent cf33309117
commit b5ef419d5b
70 changed files with 39593 additions and 2 deletions

3
.gitattributes vendored
View file

@ -9,6 +9,9 @@
# Shell scripts should keep LF # Shell scripts should keep LF
*.sh text eol=lf *.sh text eol=lf
# Archived patch artifacts preserve whitespace exactly
packages/agent/docs/**/*.diff -whitespace
# Common binary assets # Common binary assets
*.png binary *.png binary
*.jpg binary *.jpg binary

View file

@ -0,0 +1,355 @@
# Assistant partial durability — implementation handoff
This document specifies durable partial assistant messages for ordinary assistant generation and deferred-response polling. It builds on:
- bound typed value/list addresses from `values.md`;
- `AssistantMessageFrame`, `AssistantMessageFrameEncoder`, and `reduceAssistantMessageFrames()` from `@earendil-works/pi-ai`;
- the assistant intent/effect/settlement state machine in `harness.md`.
The design persists compact replayable stream frames without making them operation-state authority and without storing a growing full partial message on every update.
## Goals
1. Reconstruct the latest committed partial assistant message after process loss.
2. Preserve provider stream order without repeated full-snapshot write amplification.
3. Avoid provider backpressure from storage commits.
4. Reuse pi-ai's canonical frame conversion and reduction semantics.
5. Preserve current public assistant event ordering.
6. Keep operation `effect_pending` state authoritative for recovery.
7. Delete all partial frames atomically with normal or synthetic response settlement.
## Non-goals
- Inferring provider completion from frames.
- Persisting terminal `done`/`error` events separately from response settlement.
- Exactly-once provider requests.
- Public frame cursors or a frame-persistence event.
- Generic batching, timers, coalescing, or flush APIs.
- Persisting structural summary-generation streams.
- Persisting arbitrary provider SDK events or repeated full `partial` snapshots.
## Storage
Built-in address constructor in `session/values.ts`:
```ts
export const pendingAssistantFrames = (
operationId: string,
responseEntryId: string,
) => list<AssistantMessageFrame>(
"pi.pending.assistant_frame",
`${operationId}:${responseEntryId}`,
);
```
The procedure binds one exact address for ordinary generation or a deferred poll:
```ts
const frames = pendingAssistantFrames(operationId, responseEntryId);
```
`responseEntryId` is already reserved in assistant/deferred `effect_pending` state. No frame count, cursor, or list identity is stored in operation state, and later list operations receive only `frames`—never another key.
Each list element is one `AssistantMessageFrame`. The storage transaction's global write sequence orders frames. The list is auxiliary:
- missing is valid;
- it does not prove request admission, completion, success, or failure;
- it never selects the restart point;
- base restore does not read it.
## Frame contract
Create one pi-ai encoder per provider stream and feed it every event in order:
```ts
const encoder = new AssistantMessageFrameEncoder();
const frame = encoder.encode(event);
```
`partial` is the provider's shared live response-so-far helper, not an event-time snapshot. The encoder keeps per-open-block counters and trims text/thinking delta prefixes already represented by an advanced block-start snapshot. It temporarily buffers only the raw JSON prefix needed to synchronize an already-advanced tool call, then emits one checkpoint and resumes compact deltas. It never clones the growing full message on every event.
- `start` produces an empty-content metadata frame;
- a non-terminal event produces zero or one frame;
- covered queued deltas produce no frame;
- terminal `done` and `error` produce no frame because final response settlement is separate;
- a setup `error` before `start` is valid and produces no frames;
- text/thinking/tool end frames contain the authoritative completed block value;
- completed tool-call arguments remain unvalidated against a tool schema.
Do not define a second harness frame codec or reducer. Persistence stores the exported pi-ai value directly; hydration calls `reduceAssistantMessageFrames()`.
Provider event blocks may interleave. Encoding and reduction rely on `contentIndex`, never block contiguity. Text and ordinary thinking blocks must be empty when `*_start` is published and then append only through matching deltas until end; redacted thinking may be complete at start and emit no deltas. Streaming tool calls start with empty arguments and emit their complete raw JSON through deltas; a provider that starts with complete arguments must emit a cumulative delta prefix that parses to that snapshot at an event boundary before later argument deltas.
## Fresh stream scheduling
The simplest scheduling is deliberate: one list-append transaction per converted frame, enqueued without awaiting storage in the provider loop.
For every `start` or update event:
```text
encode the event against the per-stream frame encoder
→ when a frame is returned, synchronously enqueue invocation-fenced appendList(frames, frame)
→ attach the ordinary harness-fault observer to the returned promise
→ replace process-local latestFrameWrite promise reference
→ emit and await the existing message_start/message_update event
→ consume the next provider event
```
`appendList()` is called synchronously for each returned frame, so all frame mutations enter the Session line in provider-event order. The procedure does not await each write. Every promise is immediately observed for fault propagation before the latest reference replaces its predecessor. Covered queued events allocate no durable frame or write. Encoder state is proportional to open block count plus the unsynchronized prefix of an active tool-call JSON stream; it retains no second full assistant message. Model output limits bound queued frame bytes to bounded output plus frame/transaction overhead.
The event side retains existing behavior: `AssistantStreamObserver.start/update` awaits `events.emit()` for every event. There is no separate latest event-delivery promise because no assistant event delivery remains outstanding when the provider loop advances.
When the provider stream settles:
```text
stop frame admission
→ await latestFrameWrite when present
→ run after_response
→ emit message_end
→ classify and commit final response settlement
```
The Session mutation line is FIFO, so completion of `latestFrameWrite` implies every earlier append completed. There is no array of promises, timer, batcher, active/waiting state, coalescer, or public/internal flush method.
A failed frame append faults the harness before `after_response` starts. The complete final response remains process-local and does not commit after a storage fault.
## Normal settlement
Every assistant response settlement that used a frame list deletes that exact list in the same transaction as the immutable response entry, usage, tip, and next operation state:
```text
TX[
insert response entry R,
insert usage U,
setValue(branchTip(lane), R),
deleteList(frames),
setValue(operationState(operationId), classified next state)
]
```
This applies to:
- successful assistant responses;
- provider `error`/`aborted` responses;
- valid deferred responses;
- deferred poll responses.
`after_response` may transform the final response before settlement. Frames preserve provider-stream observation, while the immutable entry remains the post-hook canonical result.
A crash after the final frame append but before settlement still restores `effect_pending`; frames do not turn a complete-looking draft into a settled response.
## Unknown-outcome generation recovery
An orphaned assistant generation `effect_pending` has no surviving provider stream. Activation:
1. constructs `frames = pendingAssistantFrames(operationId, responseEntryId)` from current typed state and reads bounded pages from that exact address;
2. reduces frame values with `reduceAssistantMessageFrames()`;
3. constructs a harness-owned synthetic assistant response under the already-reserved response ID;
4. preserves reconstructed partial content and safe message identity metadata when frames exist;
5. sets `stopReason: "error"`, zero usage, and an explicit interruption/unknown-outcome `errorMessage`/diagnostic;
6. commits the synthetic response, zero-usage row, frame-list deletion, tip, and ordinary retry/failure state atomically.
Required warning meaning:
```text
The provider request was interrupted. The preceding content is the latest
committed partial response; newer live output may be missing, and the external
request outcome is unknown.
```
If no start frame committed, the harness constructs the same synthetic error with empty content from the captured model/API identity.
The synthetic response follows ordinary assistant error classification:
- attempts remain → insert the error response and enter the ordinary retry-wait/next-attempt path;
- cap reached → insert it and terminal-fail the operation in the same settlement transaction.
Error responses remain durable transcript history but are omitted from later provider context by the existing projection rule. Partial tool calls inside the interrupted error response never execute.
Recovery does not run `after_response`: there is no trustworthy complete provider result to transform.
## Cancellation
A live cancelled provider stream settles through its ordinary final `aborted` response. All accepted frames are awaited first, and normal settlement deletes the list.
For restored cancelled assistant/deferred `effect_pending`, cancellation reconciliation:
- reduces available frames;
- constructs a synthetic `aborted` response preserving the committed partial content;
- uses zero usage and the existing reserved IDs;
- atomically inserts the response and deletes the frame list;
- starts no provider request and runs no response hook.
Cancellation wins classification even if reduced content appears complete.
## Deferred polling
A deferred poll that returns an `AssistantMessageEventStream` uses the same `pendingAssistantFrames(operationId, responseEntryId)` address constructor.
- normal pending/ready/error settlement deletes the poll's frame list;
- restored cancellation synthesizes an aborted response from frames;
- an unknown restored poll without a permit remains suspended and may expose its durable partial in snapshots;
- when a poll permit replaces the unknown poll with fresh reserved response/usage IDs, that intent transaction deletes the abandoned old frame-list address;
- the replacement poll starts a fresh list under its fresh response ID.
Frames never change poll-number rules.
## Structural generation scope
Structural summary-generation streams remain process-local. They do not emit public assistant-message lifecycle and may span multiple nested provider requests before one structural publication. Existing attempt-level retry and usage recovery remains authoritative.
Do not store their intermediate text at a `pendingAssistantFrames(...)` address in this slice. If structural partial diagnostics become a requirement, add a separate explicitly scoped consumer rather than silently reusing transcript-assistant semantics.
## Snapshots and reconnect
`LaneSnapshot.streamingMessage` means the latest observed partial assistant message, not proof that a provider stream is currently attached.
Precedence:
1. newest process-local partial while a live stream is owned;
2. otherwise, for assistant/deferred `effect_pending`, the value reduced from committed frame pages;
3. otherwise absent.
A restored lane may therefore be `suspended` with a non-undefined `streamingMessage`. The field remains outside `transcript`; only `entry_added` moves a complete response into transcript history and clears the partial.
Snapshot hydration constructs the exact bound frame address from trusted typed operation state and enforces the assistant consumer's total frame/page budget. It does not scan arbitrary lists or perform a broad semantic restore audit.
Reconnect replays no historical `message_start` or `message_update` events. The snapshot carries the durable partial. Recovery later emits the ordinary recovery-tagged synthetic message lifecycle for the response it actually settles.
## Events
A started generation retains this live event ordering:
```text
message_start
→ message_update* each listener delivery awaited by provider loop
→ await latest frame write
→ after_response
→ message_end
→ atomic response settlement + frame-list delete
→ entry_added
→ usage
```
A request setup failure may produce `error` before `start`; that path emits no `message_start` or frame and proceeds through `after_response`, `message_end`, and ordinary error settlement. Successful `done` and updates before `start` are protocol defects.
Frame commits emit ordinary storage telemetry only. There is no public frame event and no claim that a `message_update` was durable. `entry_added` remains the only proof that the final assistant entry committed.
A crash may occur after a live update event but before its asynchronously queued frame append commits. Reconnect then shows the latest committed frame prefix, which may be older than the last live event.
## Close, fault, and external finalization
Close is a controlled crash:
- no synthetic response is written;
- already-enqueued frame commits are ordinary admitted session work and may finish under the close barrier;
- process loss may discard mutations that had not committed;
- reopen restores the latest committed frame prefix under unchanged `effect_pending` state.
A frame-commit storage failure faults the harness. No later response settlement commits in that process.
Authorized external finalization deletes the operation-owned frame-list address in its terminal transaction. Every append mutation verifies current operation/response ownership on the Session line:
- append first → external terminal cleanup deletes the list;
- finalization first → stale append rejects without recreating state.
## Terminal cleanup, forks, and migrations
Normal/synthetic response settlement should already delete its exact frame address. The operation terminal transaction also defensively constructs and deletes the current operation-owned frame address when state is assistant/deferred `effect_pending`.
Idle forks never copy lists in the `pi.pending.assistant_frame` address family. Precise rewrites and migrations page frame lists and preserve element sequences when retaining them.
A migration changing `AssistantMessageFrame` shape must map every surviving element or explicitly delete the whole list and leave `effect_pending` recovery with no partial. It must never infer completion from legacy frames.
JSONL retains deleted frame bytes until snapshot compaction. Logical deletion is immediate.
## Races
| Race | Required result |
|---|---|
| frame append vs next frame | synchronous lane enqueue preserves provider-event order |
| frame append vs stream settlement | settlement awaits the latest promise; all accepted appends finish first |
| live update event vs frame commit | either may finish first; event is observation and reconnect uses only committed frames |
| append vs external finalization | append first is deleted by cleanup; finalization first fences the append |
| process loss with queued writes | only the committed prefix restores |
| final frame vs response settlement | frame commits first; settlement atomically deletes the list and inserts final entry |
| activation vs snapshot | both reduce the same committed sequence prefix; activation may then settle and clear it |
| unknown generation vs retry | synthetic partial error commits under old reserved IDs before later attempt starts |
| unknown deferred poll vs replacement | old list is deleted with fresh replacement intent; new response ID gets a new list |
## Invariants
1. Scalar assistant/deferred state is the sole restart authority.
2. One effect-pending response ID constructs exactly one assistant frame-list address.
3. Every stored element is an exported pi-ai `AssistantMessageFrame`.
4. Terminal `done`/`error` events are never stored as frames.
5. Frame order is a subsequence of provider event order; zero-frame covered events do not disturb order.
6. Awaiting the latest frame-write promise at stream settlement implies all accepted appends completed.
7. Frames never establish provider completion or suppress unknown-outcome recovery.
8. Final or synthetic response settlement atomically deletes the exact frame list.
9. A restored partial may appear in `streamingMessage` but never in `transcript` before settlement.
10. Interrupted partial tool calls never produce a tool plan because the synthetic response stops with `error`/`aborted`.
11. Structural generation never writes a `pendingAssistantFrames(...)` list in this slice.
12. Terminal cleanup, external finalization, and idle forks leave no operation-owned assistant frame list.
## Required tests
### Frame integration
- one per-stream encoder handles shared live partials and synchronous event bursts without duplicate content;
- each event appends zero or one frame, with covered queued deltas appending nothing;
- `done`/`error` append nothing, including pre-generation error;
- interleaved content indexes preserve sequence;
- provider loop does not await individual frame writes;
- frame appends enqueue synchronously before the next provider event;
- only the latest promise reference is retained;
- awaiting the latest promise implies all earlier writes completed;
- bounded output bounds queued frame memory;
- storage failure prevents `after_response` and faults the harness.
### Settlement and recovery
- every ordinary response class deletes the frame list atomically;
- crash at every frame/settlement boundary;
- no frames and partial text/thinking/tool-call frames;
- authoritative end-frame content survives;
- interrupted generation commits partial synthetic error then retries or fails at cap;
- interrupted partial tool calls never execute;
- recovery uses zero usage and existing reserved IDs;
- cancellation preserves committed partial content in synthetic aborted response;
- `after_response` never runs for restored synthetic settlement.
### Deferred, snapshots, and lifecycle
- deferred poll frame persistence and normal cleanup;
- unknown poll snapshot without permit;
- replacement intent deletes abandoned poll frames;
- live partial takes precedence over durable reduction;
- reopen exposes reduced `streamingMessage` on suspended effect-pending lane;
- no historical update event replay;
- recovery settlement clears partial on `entry_added`;
- structural generation writes no assistant frame list.
### Storage lifecycle
- Memory/JSONL/SQLite reduce identical frame sequences;
- frame list is absent after normal, synthetic, cancellation, and external terminal transitions;
- idle fork excludes frames;
- JSONL compaction removes deleted frame bytes;
- migration maps or explicitly discards every legacy frame;
- instrumentation records append/delete ordering without frame content in telemetry.
## Implementation map
Expected runtime areas:
- built-in `pendingAssistantFrames(operationId, responseEntryId)` address constructor in `session/values.ts`;
- bound value/list storage implementation from `values.md`;
- assistant execution observer and generation procedure;
- deferred polling procedure;
- activation and cancellation recovery;
- snapshot hydration;
- terminal cleanup, forks, and migrations;
- instrumented writer and backend conformance tests.
Implement bound typed value/list addresses first, then frame enqueue/settlement, then recovery/snapshots. Update `harness.md` with this complete lifecycle before implementing runtime assistant parity.

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,214 @@
# 01-delta: known defects and measured findings
Everything here was found **after** `delta.md` and the prototype implementation beside it were written. None of it is applied to the code in this directory. Production now lives in `packages/chord/src/delta/index.ts`: flush-time dirty tracking resolves D1, and production remeasurement closes D2 without a new API. See [`append-decision.md`](append-decision.md). Each item below remains the original problem, evidence, and candidate-fix record.
Every number was measured with **`node --experimental-strip-types`**, not `tsx`.
That matters: tsx transpiles through esbuild and inflated the same benchmark by
**2.6x** (183 µs vs 63 µs per write). Do not benchmark this module through a
transpiler.
---
## D1. Interleaved paths defeat coalescing (correctness-adjacent, unbounded memory)
**Severity: high.** This is the one to fix first.
### Problem
`flush()` coalesces only **adjacent** ops. A producer that alternates between two
paths therefore never coalesces anything, and the ops array grows with the number
of writes rather than with the size of the value.
This is not hypothetical. It is exactly what bash does: it updates the output text
and a byte counter on every chunk.
```ts
const next = t.state.content[0].text + chunk;
t.state.content[0].text = next.length > CAP ? next.slice(next.length - CAP) : next;
t.state.truncation.totalBytes += chunk.length; // <- the other path
```
### Evidence
50 KB window, 200-byte chunks, no flush in between:
| held-back writes | ops | bytes |
| --- | --- | --- |
| 1 | 3 | 0.3 KB |
| 100 | 201 | 26 KB |
| **1000** | **2001** | **264 KB** |
Without interleaving the same workload gives 1 op and 51 KB. The window is
bounded at 50 KB; the ops describing it are not.
### Why it matters beyond memory
Any design where the sink **holds back** — rate limiting, backpressure, a slow
consumer — depends on held-back writes costing nothing. They currently cost
everything. See `../../04-tool-output/rate-limiting.md`.
### Candidate fix
Fold at **record** time, keyed by path rather than by adjacency: one slot per
path holding at most two ops (`[s]`, `[d]`, `[t]`, `[a]`, or `[t, a]`), emitted in
first-touch order.
A prototype reached **2 ops / 51 KB** on the same workload. It is not included
here because it was not verified to the standard the rest of this unit meets.
**Four folding rules, each a specific claim about the vocabulary:**
1. A later `s` supersedes an earlier `s` on the same path — but **not** an earlier
`d`, which must survive or the key changes position on reinsertion (§5.1).
2. Consecutive `a` merge. Once the merged append is **strictly longer** than the
value, the window has been sliding and the append is the more expensive
spelling, so collapse to `s`. Strictly: while a string is still *growing*
toward its cap the append equals the value, and collapsing there would resend
the whole string every batch.
3. `t` and `a` on one path **commute** — the truncate drops from the front, the
append adds to the end — so a rolling window's `t,a,t,a,...` folds to one of
each. Without this, pairwise merging never fires at all.
4. `t` or `a` following a pending `s` changes nothing: the `s` already means "this
path becomes its current value".
**The trap that prototype hit.** Slots emit in first-touch order, which loses the
fact that a child write happened *before* a later parent write:
```
script : a={x:1}; a.b=99; a={c:2}
producer: {"a":{"c":2}}
replica : {"a":{"c":2,"b":99}} <- WRONG
```
A set or delete must invalidate every pending slot **below** its path, at record
time. The existing `verify-dead` style test passed 1190 sequences while this was
broken, because its generator only nested one level. Any test for this must
generate genuinely nested paths with parent overwrites interleaved between child
writes.
---
## D2. `overlap()` was 94.5% of prototype tracker time — closed
**Historical severity: high. Production decision:** do not add an explicit append/truncate API. The prototype profile was dominated by a slow `startsWith` path; production uses a flattened-slice comparison and measured 17.8–18.7 µs per 200 KB assistant append flush and 2.43–2.46 µs per 50 KB rolling-window flush locally. See [`append-decision.md`](append-decision.md). The original evidence follows.
### Problem
The tracker infers "you appended and evicted" by searching two 50 KB strings.
Each new value produced by `next.slice(-CAP)` is a V8 `SlicedString` — a lazy
substring — and `indexOf`/`endsWith` force it to **flatten**, copying 50 KB per
call.
### Evidence
CPU profile of a 20 000-write rolling-window loop, node, flush at the end:
```
94.5% overlap
2.7% (anon)
1.2% (garbage collector)
0.5% set
0.3% emit
0.0% fold
```
And the flattening is directly observable:
| `overlap()` on a 50 KB window | µs |
| --- | --- |
| the same two strings reused | 37 |
| **a fresh sliced string each call** | **149** |
An earlier benchmark reported 29 µs because it reused two already-flattened
strings 2000 times — a case that never occurs in the real loop. **Do not benchmark
string operations against a reused fixture.**
### Candidate fix
Do not infer what the producer already knows. `ToolOutput` performs the windowing,
so it can emit the ops directly rather than assigning a new string and having the
tracker search for the difference:
```ts
out.append(chunk); // -> ["t", path, dropped], ["a", path, chunk]
```
Two integers — chars appended, chars evicted — describe everything that happened
between any two observations of a bounded window. A prototype of this shape held
**exactly the window** in memory under every load, with no ops array, no overlap
detection and no folding rules at all.
This does **not** remove `overlap`: it stays for the general case where a producer
assigns a whole new string and the delta genuinely has to be discovered. It
removes it from the hot path.
### Do not
Do not "optimise" `overlap` itself before doing the above. Its algorithm is
already correct and bounded (`maxOverlapScan`); the cost is string flattening,
which no probe strategy avoids.
---
## D3. Nested-path proxy overhead is next, and currently invisible
**Severity: unknown — measure before acting.**
`overlap` dominates so completely that nothing else shows in a profile. But two
benchmarks disagree in a way that points at path depth:
| | µs per write |
| --- | --- |
| top-level path (`{ text }`) | 63 |
| nested path (`{ content: [{ text }] }`) | 244 |
4x for depth alone. Each nested property access wraps a child proxy and builds a
cache key by joining the path, so a read of `content[0].text` allocates on every
access.
**Re-profile only if production delta cost becomes material.** This may be the next bottleneck or it may be
nothing; the current numbers cannot distinguish them because `overlap` swamps both.
---
## D4. Things measured and found NOT worth doing
Recorded so nobody repeats the work. All were under **1.5% combined** in the
profile above.
**A path trie replacing `JSON.stringify(path)` keys.** Genuinely faster in
isolation — 6x on the `LaneSnapshot` shape, 93x at 500 distinct paths — because
stringify cost scales with path depth while a trie walk is one `Map.get` per
segment. But the whole slot layer is 0.5% of runtime. Build it only if a current production profile says so.
**Trie as storage vs trie as index.** If a trie is ever built, it should map path
to a **position in the ops array**, not hold the ops. Holding them loses first-touch
ordering and forces a sort at flush to rebuild it. Measured difference: none
(1.4x / 0.9x / 1.0x). The argument is structural, not performance.
**A `maxDepth` heuristic to skip descendant scanning.** Rejected: the flat map was
losing on the common case too, so the heuristic would have hidden a real cost
rather than removed it.
**Eager string concatenation in the `a`+`a` merge.** Real (it rebuilds a growing
string per write) but worth ~5%. Compare lengths and materialise only when
emitting.
---
## D5. Measurement errors made while producing these numbers
Listed because each one produced a confident wrong conclusion, and the same traps
are still there.
- **Benchmarking through `tsx`** — inflated everything 2.6x.
- **Reusing a flattened string** in a string benchmark — hid the real cost 4x.
- **Repetitive fixtures** (`"x".repeat(n)`) — overlap detection hits its candidate
bound and gives up, so the benchmark measures the fallback path, not the real one.
Use varied text.
- **Measuring a cache hit as if it were work** — `Markdown.render()` returns
`cachedLines` when text and width are unchanged, so a repeat-the-same-call loop
measured 0.065 ms for something that costs 0.86 ms.
- **Reasoning about complexity while the constant dominated** — twice.
- **Measuring heap instead of retained size** — a fixture generating 800 x 1 MB
strings reported 40 MB of "growth" that was its own garbage.

View file

@ -0,0 +1,39 @@
# Decision: no explicit text append/truncate API
**Status: closed.** Do not add `appendText`, explicit truncate, a text-specific dirty-node kind, or proxy-to-tracker lookup machinery.
## Why it was considered
The tracker marks dirty paths and compares them with its accepted baseline at `flush()`. For a growing or rolling string, confirming the relationship can scan the retained string even though the producer already knows it appended or evicted text.
The original benchmark made this look expensive because its append fast path used `after.startsWith(before)`. V8 walked the producer's cons string character by character. Production Chord now uses:
```ts
if (after.length > before.length && after.slice(0, before.length) === before) {
// emit append
}
```
The slice flattens once and comparison uses the native string path. Flush-time dirty tracking also removed the prototype's retained-op amplification.
## Local confirmation
Measured 2026-09-01 against `origin/dev` at `1a7bc80e7`, using `packages/chord/src/delta/index.ts` directly under Node 26.0.0 on an Apple M5 Max. Each workload used 3,000 warmups followed by 11 samples of 10,000 mutation-plus-flush iterations; a second process reproduced the result.
| Workload | Median µs/flush, run 1 | Median µs/flush, run 2 |
| --- | ---: | ---: |
| 200 KB assistant string, append 8 characters | 18.68 | 17.81 |
| 50 KB rolling window, slide 32 varied characters | 2.46 | 2.43 |
| Transcript push, one small entry | 0.74 | 0.78 |
The assistant tracker started with 200,000 varied characters, appended `" abcdef"`, and flushed each append. The rolling tracker started with 50,000 varied characters, then assigned `text.slice(32) + chunk` using a distinct 32-character `chunk:<base36 index>:durable-stream` value on every flush; every flush was asserted to emit `t` + `a`. The transcript tracker pushed `{ id: "e<index>", text: "message <index>" }` and asserted one `p` per flush. Setup and garbage collection were outside each timed loop.
At 100 assistant updates per second, the growing-string case consumes about 1.8 ms/s, approximately 0.18% of one core. The measured rolling-window cost is itself too small to justify the abandoned explicit API.
## Why the API is rejected
The prototype required a text-specific dirty-node state, proxy-to-tracker lookup, repeated-append/drop folding, and interaction rules for later whole-value replacement. It produced subtle silent failures: one implementation fell through to the ordinary differ while tests still passed, and another drifted when drops reached into earlier appends.
That complexity does not justify saving microseconds below the surrounding replication, isolation, and rendering costs. Keep ordinary string mutation and the generic Chord op vocabulary.
Reopen only when a production profile shows delta flush time is a meaningful fraction of a real workload. Re-measure the generic fast path first; a regression there is cheaper and safer to fix than adding a producer-specific API.

View file

@ -0,0 +1,826 @@
// ─────────────────────────────────────────────────────────────────────────────
// harness/delta — intent-recording change tracking over plain JSON.
//
// Depends on nothing else in the harness. Session storage, the runtime and the
// facet host consume it; keep the arrows pointing that way.
// ─────────────────────────────────────────────────────────────────────────────
export type JsonValue = null | boolean | number | string | JsonValue[] | { [k: string]: JsonValue };
export type Seg = string | number;
export type Path = readonly Seg[];
export type NonEmptyPath = readonly [Seg, ...Seg[]];
/** A path inline, or an id assigned by the encoder on second use. */
export type PathRef<P extends Path = Path> = P | number;
/**
* Tuples are the form — in memory, on the wire, on disk.
*
* `r` is the ONLY op that replaces a whole value. `s`/`d`/`a`/`t` cannot target
* the root: the type forbids it. `p` may, and only because a tracked value can
* itself be an array — but a `p` that replaces its entire target is normalised to
* `r`/`s` at record time, so a root `p` is always a partial modification.
*
* `Op` knows nothing about the path dictionary. Interning, id references and
* omitted paths live in `WireOp` and exist only between `encode` and `decode`.
*/
export type Op =
| readonly ["r", JsonValue]
| readonly ["s", NonEmptyPath, JsonValue]
| readonly ["d", NonEmptyPath]
| readonly ["a", NonEmptyPath, string]
| readonly ["t", NonEmptyPath, number]
| readonly ["p", Path, number, number, JsonValue[]];
/**
* What crosses a boundary. Adds two compressions and nothing else:
*
* ["#", id, path] defines an id, emitted on a path's SECOND use
* a numeric PathRef references a previously defined id
* a shortened tuple reuses the previous op's path; arity disambiguates
*
* ["r", value] carries no path, so it encodes to itself — which is why isBase
* works unchanged on either vocabulary.
*/
export type WireOp =
| readonly ["r", JsonValue]
| readonly ["s", PathRef<NonEmptyPath>, JsonValue] | readonly ["s", JsonValue]
| readonly ["d", PathRef<NonEmptyPath>] | readonly ["d"]
| readonly ["a", PathRef<NonEmptyPath>, string] | readonly ["a", string]
| readonly ["t", PathRef<NonEmptyPath>, number] | readonly ["t", number]
| readonly ["p", PathRef, number, number, JsonValue[]]
| readonly ["p", number, number, JsonValue[]]
| readonly ["#", number, Path];
// ─── Classification ──────────────────────────────────────────────────────────
export const isReplace = (op: Op | WireOp): boolean => op[0] === "r";
/**
* A batch begins with a replacement. Flush guarantees `r` is at index 0 or absent,
* so this is exact rather than a heuristic.
*/
export const isBase = (ops: readonly (Op | WireOp)[]): boolean =>
ops.length > 0 && ops[0]![0] === "r";
// ─── Overlap ─────────────────────────────────────────────────────────────────
/**
* Longest suffix of `a` that is a prefix of `b`. Probes with indexOf and verifies
* with endsWith, so the hot loops are native. A hand-written KMP is
* asymptotically equivalent and ~47x slower in practice.
*
* Always correct: the returned n satisfies a.endsWith(b.slice(0, n)).
*/
export function overlap(a: string, b: string, scan: number, probe = 64, maxCandidates = 8): number {
if (a.length === 0 || b.length === 0 || scan === 0) return 0;
const tail = a.length > scan ? a.slice(a.length - scan) : a;
// A probe of length h can only find overlaps of at least h — the head must
// actually occur in `a`. So try a long head first (few candidates, and it
// catches the large overlaps a rolling window produces), then fall back to one
// character, which finds any overlap at the cost of more candidates.
//
// Candidates are bounded because repetitive output — a build log, or any run of
// one character — makes a long head match at thousands of positions, each
// costing a full endsWith. Giving up returns 0, which emits a set: larger, never
// wrong. Unbounded, a 50 KB window of one repeated character took 4.7 s per
// 2000 slides against 90 ms bounded.
for (const h of [Math.min(probe, b.length), 1]) {
const head = b.slice(0, h);
let tried = 0;
for (let k = tail.indexOf(head); k !== -1; k = tail.indexOf(head, k + 1)) {
if (++tried > maxCandidates) break;
const n = tail.length - k;
if (n <= b.length && tail.endsWith(b.slice(0, n))) return n;
}
if (h === 1) break;
}
return 0;
}
// ─── Tracker ─────────────────────────────────────────────────────────────────
export interface TrackerOptions {
maxOverlapScan?: number;
coalesce?: boolean;
}
export interface Tracker<T extends object> {
/**
* Mutate this. Plain TS: assignment, push, splice, delete, all of it.
*
* Assigning to it replaces the whole value: emits `["r", next]` and discards
* ops recorded before it, which describe a value that no longer exists.
*
* `tracker.state = tracker.target` is therefore a checkpoint — it forces a base
* batch without changing anything. Discarding the pending ops is correct
* because the proxy mutates the target directly, so the value already carries
* them.
*/
state: T;
/** The untracked backing object. Reads only — writing here emits nothing. */
readonly target: T;
flush(): Op[];
/**
* Clear pending ops and make the next flush a base batch, without changing
* the value.
*
* Recovery replays from the last base batch, so a producer that wants "at most
* N batches to replay" calls this every N. The pending ops are already
* reflected in the value — the proxy mutates the target directly — so
* discarding them loses nothing.
*/
rebase(): void;
/** Discard without emitting. */
discard(): void;
readonly dirty: boolean;
}
const isObj = (v: unknown): v is object => v !== null && typeof v === "object";
const clone = <T>(v: T): T => (isObj(v) ? (structuredClone(v) as T) : v);
const INDEX = /^(?:0|[1-9]\d*)$/;
const norm = (t: object, k: string | symbol): Seg | symbol =>
typeof k === "symbol" ? k : Array.isArray(t) && INDEX.test(k) ? Number(k) : k;
const MUTATORS = new Set(["push", "pop", "shift", "unshift", "splice", "sort", "reverse", "fill", "copyWithin"]);
export function track<T extends object>(root: T, options: TrackerOptions = {}): Tracker<T> {
const scan = options.maxOverlapScan ?? 65_536;
const doCoalesce = options.coalesce ?? true;
let ops: Op[] = [];
const emit = (op: Op) => {
ops.push(op);
};
/**
* Reject reserved keys at record time, not just on apply.
*
* `JSON.parse('{"__proto__":{}}')` yields an OWN property, so a producer can
* hold state containing one without doing anything unusual. The tracker would
* then emit a path the applier refuses — state that cannot be replicated. Fail
* where the mistake is, at the write.
*/
const guard = (seg: Seg | symbol): Seg => {
if (typeof seg === "symbol") throw new UnsafePathError(String(seg));
if (typeof seg === "string" && FORBIDDEN.has(seg)) throw new UnsafePathError(seg);
return seg;
};
/**
* A splice covering the whole target is a replacement written the long way.
* Only the interception knows the pre-splice length, so this is a record-time
* rewrite, not something flush can recover.
*/
const spliceOrReplace = (path: Path, before: number, index: number, remove: number, items: JsonValue[]) => {
if (index === 0 && remove === before) {
if (path.length === 0) emit(["r", items]);
else emit(["s", path as NonEmptyPath, items]);
return;
}
emit(["p", [...path], index, remove, items]);
};
const cache = new WeakMap<object, Map<string, object>>();
const wrap = <V extends object>(obj: V, path: Path): V => {
let byPath = cache.get(obj);
if (byPath === undefined) cache.set(obj, (byPath = new Map()));
const cacheKey = path.join("\u0000");
const hit = byPath.get(cacheKey);
if (hit !== undefined) return hit as V;
const proxy = new Proxy(obj, {
get(t, k, r) {
if (Array.isArray(t) && typeof k === "string" && MUTATORS.has(k)) {
return (...args: unknown[]) => {
const before = t.length;
const result = (Array.prototype[k as "push"] as Function).apply(t, args);
const items = (xs: unknown[]) => xs.map(clone) as JsonValue[];
switch (k) {
case "push": spliceOrReplace(path, before, before, 0, items(args)); break;
case "unshift": spliceOrReplace(path, before, 0, 0, items(args)); break;
case "pop": if (before > 0) spliceOrReplace(path, before, before - 1, 1, []); break;
case "shift": if (before > 0) spliceOrReplace(path, before, 0, 1, []); break;
case "splice": {
const raw = Number(args[0] ?? 0);
const index = raw < 0 ? Math.max(0, before + raw) : Math.min(raw, before);
const remove = args.length < 2 ? before - index : Math.max(0, Math.min(Number(args[1]), before - index));
spliceOrReplace(path, before, index, remove, items(args.slice(2)));
break;
}
// sort/reverse/fill/copyWithin permute wholesale; intent is not a splice.
default:
if (path.length === 0) emit(["r", clone(t) as JsonValue]);
else emit(["s", path as NonEmptyPath, clone(t) as JsonValue]);
}
return result;
};
}
const v = Reflect.get(t, k, r);
return isObj(v) ? wrap(v, [...path, guard(norm(t, k))]) : v;
},
set(t, k, v, r) {
// `length` is a real mutation but not a document location.
if (Array.isArray(t) && k === "length") {
const before = t.length;
const next = Number(v);
if (next < before) spliceOrReplace(path, before, next, before - next, []);
else if (next > before) emit(["p", [...path], before, 0, Array(next - before).fill(null)]);
return Reflect.set(t, k, v, r);
}
const at = [...path, guard(norm(t, k))] as unknown as NonEmptyPath;
if (v === undefined) { // JSON has no undefined
emit(["d", at]);
return Reflect.deleteProperty(t, k);
}
const prev = (t as Record<string | symbol, unknown>)[k];
if (typeof prev === "string" && typeof v === "string") {
if (prev === v) return true;
if (v.length > prev.length && v.startsWith(prev)) {
emit(["a", at, v.slice(prev.length)]); // fast path, no probe
} else {
const ov = overlap(prev, v, scan);
if (ov === 0) emit(["s", at, v]);
else {
emit(["t", at, prev.length - ov]);
if (v.length > ov) emit(["a", at, v.slice(ov)]);
}
}
} else {
assertJsonValue(v);
emit(["s", at, clone(v) as JsonValue]);
}
return Reflect.set(t, k, v, r);
},
deleteProperty(t, k) {
emit(["d", [...path, guard(norm(t, k))] as unknown as NonEmptyPath]);
return Reflect.deleteProperty(t, k);
},
});
byPath.set(cacheKey, proxy);
return proxy as V;
};
let state = wrap(root, []);
// The first flush is always a base batch. A consumer starts with nothing, so
// a stream that opens with deltas has nothing to apply them to — and that
// fails at runtime, in the consumer, rather than where the mistake is.
let forceBase = true;
return {
get state() { return state; },
// A setter, so the obvious thing works. Without it `tracker.state = next`
// silently swaps the proxy for a plain object and stops tracking.
set state(next: T) {
ops = [];
root = next;
state = wrap(root, []);
forceBase = true;
},
get target() { return root; },
// Same effect as assigning the current value back, minus the re-wrap: the
// value has not changed, so the proxy cache is still valid.
rebase() { ops = []; forceBase = true; },
get dirty() { return ops.length > 0; },
discard() { ops = []; },
flush() {
if (forceBase) {
forceBase = false;
ops = [];
return [["r", clone(root) as JsonValue]];
}
if (ops.length === 0) return [];
let out = ops;
ops = [];
// Everything before a replacement is dead.
const lastReplace = out.findLastIndex(isReplace);
if (lastReplace > 0) out = out.slice(lastReplace);
if (doCoalesce) out = coalesce(dropDead(out));
return out;
},
};
}
const samePath = (a: Op, b: Op): boolean => {
const pa = pathOf(a);
const pb = pathOf(b);
if (pa === undefined || pb === undefined) return false;
return JSON.stringify(pa) === JSON.stringify(pb);
};
function pathOf(op: Op): Path | undefined {
return op[0] === "r" ? undefined : op[1];
}
/**
* Drop ops that a later op makes unreachable.
*
* Ops apply in order, so an op is dead if a later one overwrites the whole
* subtree it lives in. Only `s`, `d` and `r` do that — `a`, `t` and `p` modify
* what is already there, so they never dominate.
*
* O(n * d): walk backwards keeping a map of overwritten paths, and for each op
* test its own prefixes against that map. `d` is the path depth, which is small
* and bounded. The obvious formulation — compare each op against every
* dominator — is O(n^2 * d) and degrades on exactly the wide-flush input this
* pass exists to clean up. Measured: 12-14 microseconds per op, flat from 100 to
* 5000 ops, and 10001 redundant writes collapse to 2 emitted ops.
*
* OBJECT KEY ORDER IS NOT PRESERVED. A producer that does
* `set c; set a; delete c; set c` leaves `c` last, because deleting and
* reinserting moves it. The replica receives only the surviving `set c`, applies
* it to a base where `c` never existed, and `c` lands first. Values are
* identical — verified over 1823 randomised sequences with zero value
* mismatches — but the serialised forms can differ.
*
* This is fine because nothing compares a replica against its producer: the
* consumer holds the state and nobody diffs it back. The one visible case is a
* consumer that iterates keys for display, where two views could order rows
* differently. It requires a delete-and-recreate of the same key within one
* flush. Do not hash or content-address a replicated value.
*/
function dropDead(ops: Op[]): Op[] {
const overwritten = new Map<string, "s" | "d">();
const kept: Op[] = [];
for (let i = ops.length - 1; i >= 0; i--) {
const op = ops[i]!;
if (op[0] === "r") {
kept.push(op);
overwritten.clear();
overwritten.set("", "s"); // the root is now overwritten
continue;
}
const path = opPath(op);
if (path === undefined) { kept.push(op); continue; } // arity form, or an interned id
let dead = overwritten.has("");
let key = "";
for (let k = 0; k < path.length && !dead; k++) {
key += `\u0000${String(path[k])}`;
const kind = overwritten.get(key);
if (kind === undefined) continue;
// A `d` at exactly a later `s`'s path is NOT dead. Dropping it leaves the
// key in its original slot, while the producer's delete-then-set moved it
// to the end: same value, different serialisation.
if (k === path.length - 1 && op[0] === "d" && kind === "s") break;
dead = true;
}
if (dead) continue;
kept.push(op);
if (op[0] === "s" || op[0] === "d") overwritten.set(key, op[0]);
}
return kept.reverse();
}
/** Total, because `Op` paths are always inline. */
function opPath(op: Op): Path | undefined {
return op[0] === "r" ? undefined : op[1];
}
/** Merge adjacent ops describing the same path. Lossless. */
function coalesce(ops: Op[]): Op[] {
const out: Op[] = [];
for (const op of ops) {
const last = out[out.length - 1];
if (last !== undefined && samePath(last, op)) {
if (last[0] === "a" && op[0] === "a") {
out[out.length - 1] = ["a", last[1], `${last[2]}${op[2]}`];
continue;
}
if (last[0] === "s" && op[0] === "s") { // later write wins
out[out.length - 1] = op;
continue;
}
if (last[0] === "s" && op[0] === "a" && typeof last[2] === "string") {
out[out.length - 1] = ["s", last[1], `${last[2]}${op[2]}`];
continue;
}
}
out.push(op);
}
return out;
}
// ─── Path safety ─────────────────────────────────────────────────────────────
/**
* Segments that reach the prototype chain.
*
* `JSON.parse` is safe on its own — it makes `__proto__` an own property. What is
* not safe is `parent[key] = value`, which is exactly what an applier does, and
* paths are data: `["s", ["__proto__", "isAdmin"], true]` pollutes
* `Object.prototype` for the whole process.
*
* Ops arrive from a facet, a plugin compartment, or a tool whose details may echo
* model output, so none of it is trusted input.
*/
const FORBIDDEN = new Set(["__proto__", "constructor", "prototype"]);
export class UnsafePathError extends Error {
// Not a parameter property: Node's --experimental-strip-types rejects those,
// and these files are meant to run under it directly.
readonly segment: Seg;
constructor(segment: Seg) {
super(`unsafe path segment: ${String(segment)}`);
this.segment = segment;
this.name = "UnsafePathError";
}
}
/**
* Rejects anything JSON.stringify would not round-trip.
*
* structuredClone is NOT a JSON check: it clones a Map happily, and the op then
* carries `{}` while the producer keeps a real Map — producer and replica diverge
* silently, forever.
*/
export function assertJsonValue(value: unknown, seen = new Set<unknown>()): void {
if (value === null) return;
const t = typeof value;
if (t === "string" || t === "boolean") return;
if (t === "number") {
if (!Number.isFinite(value as number)) throw new TypeError("non-finite number is not JSON");
return;
}
if (t !== "object") throw new TypeError(`${t} is not a JsonValue`);
if (seen.has(value)) throw new TypeError("cyclic value is not JSON");
seen.add(value);
if (Array.isArray(value)) {
for (const v of value) assertJsonValue(v, seen);
} else {
const proto = Object.getPrototypeOf(value);
if (proto !== Object.prototype && proto !== null) {
throw new TypeError(`${(value as object).constructor?.name ?? "object"} is not a JsonValue`);
}
if (Object.getOwnPropertySymbols(value).length > 0) {
throw new TypeError("symbol keys are not JSON");
}
for (const v of Object.values(value as object)) assertJsonValue(v, seen);
}
seen.delete(value);
}
/**
* Verb, arity and payload shape for a **decoded** op: paths inline, no `#`, no
* short forms. `apply` uses this.
*
* Validating `Op` against the wire grammar would be laxer than the type: a
* two-element `["s", value]` would pass, and `apply` would then read the value as
* a path. Each vocabulary gets the validator that matches it.
*/
export function assertValidOp(op: unknown): void {
if (!Array.isArray(op) || op.length === 0) throw new TypeError("op is not a tuple");
switch (op[0]) {
case "r": if (op.length !== 2) throw new TypeError("r arity"); assertJsonValue(op[1]); return;
case "s": if (op.length !== 3) throw new TypeError("s arity"); assertPathArg(op[1]); assertJsonValue(op[2]); return;
case "d": if (op.length !== 2) throw new TypeError("d arity"); assertPathArg(op[1]); return;
case "a": if (op.length !== 3 || typeof op[2] !== "string") throw new TypeError("a shape"); assertPathArg(op[1]); return;
case "t": if (op.length !== 3 || !Number.isInteger(op[2])) throw new TypeError("t shape"); assertPathArg(op[1]); return;
case "p": {
if (op.length !== 5) throw new TypeError("p arity");
assertPathArg(op[1]);
if (!Number.isInteger(op[2]) || op[2] < 0) throw new TypeError("p index");
if (!Number.isInteger(op[3]) || op[3] < 0) throw new TypeError("p remove");
if (!Array.isArray(op[4])) throw new TypeError("p items");
for (const v of op[4]) assertJsonValue(v);
return;
}
// Silently skipping an unknown verb is how a newer producer's op vanishes.
default: throw new TypeError(`unknown op verb: ${String(op[0])}`);
}
}
function assertPathArg(p: unknown): void {
if (!Array.isArray(p)) throw new TypeError("path is not an array");
assertSafePath(p as Path);
}
/** The same, for the wire grammar: ids and short forms are legal here. */
export function assertValidWireOp(op: unknown): void {
if (!Array.isArray(op) || op.length === 0) throw new TypeError("op is not a tuple");
const [verb] = op as unknown[];
const okRef = (r: unknown): void => {
if (typeof r === "number") {
if (!Number.isInteger(r) || r < 0) throw new TypeError("bad path id");
return;
}
// A string is not a path. Unchecked, `"a".slice(0, -1)` is `""`, so it
// resolves to the ROOT and writes there — a path that is not a path, accepted.
if (!Array.isArray(r)) throw new TypeError("path is not an array");
assertSafePath(r as Path);
};
switch (verb) {
case "r": if (op.length !== 2) throw new TypeError("r arity"); assertJsonValue(op[1]); return;
case "s": if (op.length === 3) { okRef(op[1]); assertJsonValue(op[2]); }
else if (op.length === 2) assertJsonValue(op[1]);
else throw new TypeError("s arity"); return;
case "d": if (op.length === 2) okRef(op[1]); else if (op.length !== 1) throw new TypeError("d arity"); return;
case "a": if (op.length === 3) { okRef(op[1]); if (typeof op[2] !== "string") throw new TypeError("a value"); }
else if (op.length === 2) { if (typeof op[1] !== "string") throw new TypeError("a value"); }
else throw new TypeError("a arity"); return;
case "t": if (op.length === 3) { okRef(op[1]); if (!Number.isInteger(op[2])) throw new TypeError("t count"); }
else if (op.length === 2) { if (!Number.isInteger(op[1])) throw new TypeError("t count"); }
else throw new TypeError("t arity"); return;
case "p": {
const [i, r, items] = op.length === 5 ? [op[2], op[3], op[4]] : op.length === 4 ? [op[1], op[2], op[3]] : [];
if (items === undefined) throw new TypeError("p arity");
if (op.length === 5) okRef(op[1]);
if (!Number.isInteger(i) || (i as number) < 0) throw new TypeError("p index");
if (!Number.isInteger(r) || (r as number) < 0) throw new TypeError("p remove");
if (!Array.isArray(items)) throw new TypeError("p items");
for (const v of items) assertJsonValue(v);
return;
}
case "#": {
if (op.length !== 3 || !Number.isInteger(op[1]) || !Array.isArray(op[2])) throw new TypeError("# shape");
assertSafePath(op[2] as Path);
return;
}
// Silently skipping an unknown verb is how a newer producer's op vanishes.
default: throw new TypeError(`unknown op verb: ${String(verb)}`);
}
}
export function assertSafePath(path: Path): void {
for (const seg of path) {
if (typeof seg === "string") {
if (FORBIDDEN.has(seg)) throw new UnsafePathError(seg);
} else if (!Number.isInteger(seg) || seg < 0) {
throw new UnsafePathError(seg);
}
}
}
/**
* An index may address an existing element or append exactly one past the end.
*
* This is not an arbitrary cap — it is what keeps the value a `JsonValue`. A
* sparse array does not survive a JSON round trip: holes serialise to `null` and
* return as real properties, so `arr[7] = x` on a length-3 array already produces
* state a replica cannot match. Rejecting the write is more honest than silently
* diverging.
*
* It also removes the denial of service it would otherwise permit:
* `["s", ["xs", 4294967290], 1]` allocates a 4.29-billion-entry array from one op.
* Growth stays possible and stays proportional — the tracker already emits
* `arr.length = n` as a splice of explicit nulls, whose op size grows with the
* gap, so a large growth costs a large op rather than a small one.
*/
function assertIndexInRange(parent: readonly unknown[], index: number): void {
if (index > parent.length) throw new UnsafePathError(index);
}
// ─── Applier ─────────────────────────────────────────────────────────────────
export class PathError extends Error {
readonly path: Path | number;
constructor(path: Path | number) {
super(`unresolvable path: ${JSON.stringify(path)}`);
this.path = path;
this.name = "PathError";
}
}
/**
* Apply ops to a plain mutable value. Returns the value, because `r` replaces it
* outright and cannot be done in place.
*
* Takes decoded ops. Path ids and omitted paths are a wire concern — run
* `decode` first if the ops came from a boundary.
*/
export function apply<T>(target: T, ops: readonly Op[]): T {
let root = target as unknown as JsonValue;
for (const op of ops) {
assertValidOp(op);
if (op[0] === "r") {
// Adopted, not copied. The consumer owns the batch it was handed.
//
// Fanning one batch out to several consumers in-process therefore makes
// their replicas alias each other. That is an ownership rule, not a
// defect: copy the batch at the fan-out point, or let each consumer
// decode its own. A batch that crosses a real boundary is already
// distinct, because serialisation produces fresh objects.
root = op[1];
continue;
}
const path = op[1];
assertSafePath(path);
if (op[0] === "p") {
const target_ = path.length === 0 ? root : resolve(root, path);
if (!Array.isArray(target_)) throw new PathError(path);
target_.splice(op[2], op[3], ...op[4]);
continue;
}
// s/d/a/t can never target the root — the type forbids it.
const parent = resolve(root, path.slice(0, -1)) as Record<Seg, JsonValue>;
const key = path[path.length - 1]!;
if (Array.isArray(parent) && typeof key === "number") assertIndexInRange(parent, key);
// defineProperty rather than assignment: a setter inherited from the prototype
// chain would otherwise run on write.
const write = (value: JsonValue) => {
Object.defineProperty(parent, key, { value, writable: true, enumerable: true, configurable: true });
};
const read = (): unknown => (Object.hasOwn(parent, key) ? parent[key] : undefined);
switch (op[0]) {
case "s": write(op[2]); break;
case "d": Array.isArray(parent) ? (parent as unknown as JsonValue[]).splice(Number(key), 1) : delete parent[key]; break;
case "a": write(`${String(read() ?? "")}${op[2]}`); break;
case "t": write(String(read() ?? "").slice(op[2])); break;
}
}
return root as unknown as T;
}
function resolve(root: JsonValue, path: Path): JsonValue {
let node: JsonValue = root;
for (const seg of path) {
if (!isObj(node)) throw new PathError(path);
// Own properties only: an inherited getter must not run, and a walk must not
// escape the value into the prototype chain.
if (!Object.hasOwn(node, seg as PropertyKey)) throw new PathError(path);
node = (node as Record<Seg, JsonValue>)[seg]!;
}
if (!isObj(node)) throw new PathError(path);
return node;
}
// ─── Codec ───────────────────────────────────────────────────────────────────
//
// Path interning and arity omission, per `delta.md` §4. These live between the
// tracker and a boundary; `Op` and `apply` know nothing about them.
//
// ONE PAIR PER STREAM. The table spans an entire subscription or file: a path
// interned in batch 3 is referenced in batch 40. A second consumer that
// subscribes at batch 40 has never seen the definition, so it needs its own
// encoder — sharing one across consumers hands the late subscriber ids it cannot
// resolve. A base batch does not rescue it, because `["r", value]` carries no
// path refs and leaves the table empty.
const pathKey = (path: Path): string => path.join("\u0000");
export interface Encoder {
encode(ops: readonly Op[]): WireOp[];
}
/**
* Intern on SECOND use. A definition costs more than the path it replaces, so
* interning on first use loses on the many paths written exactly once —
* measured worse than no interning at all (`delta.md` §4.1).
*/
export function encoder(): Encoder {
const seen = new Set<string>();
const ids = new Map<string, number>();
let nextId = 0;
let previous: string | undefined; // last path in THIS batch
return {
encode(ops) {
// Arity omission is scoped to a batch. Letting it span batches would make
// a batch's first op depend on the previous batch's last one, so a reader
// that skips or reorders a batch decodes into the wrong path. Ids are the
// only cross-batch state, and the dictionary makes those explicit.
previous = undefined;
const out: WireOp[] = [];
for (const op of ops) {
if (op[0] === "r") {
out.push(op);
// A base batch is a RECOVERY POINT: a reader replays from the last one
// with a fresh decoder. So everything after it must be self-contained.
// Keeping ids across a replacement emits references to definitions the
// reader never saw — recovery fails with an unresolvable path id.
seen.clear();
ids.clear();
nextId = 0;
previous = undefined;
continue;
}
const path = op[1];
const key = pathKey(path);
// Same path as the previous op: drop the ref entirely.
if (key === previous) {
switch (op[0]) {
case "s": out.push(["s", op[2]]); break;
case "d": out.push(["d"]); break;
case "a": out.push(["a", op[2]]); break;
case "t": out.push(["t", op[2]]); break;
case "p": out.push(["p", op[2], op[3], op[4]]); break;
}
continue;
}
let ref: PathRef = path;
const existing = ids.get(key);
if (existing !== undefined) {
ref = existing;
} else if (seen.has(key)) {
const id = nextId++;
ids.set(key, id);
out.push(["#", id, path]); // second use: define, then reference
ref = id;
} else {
seen.add(key); // first use: inline
}
switch (op[0]) {
case "s": out.push(["s", ref as PathRef<NonEmptyPath>, op[2]]); break;
case "d": out.push(["d", ref as PathRef<NonEmptyPath>]); break;
case "a": out.push(["a", ref as PathRef<NonEmptyPath>, op[2]]); break;
case "t": out.push(["t", ref as PathRef<NonEmptyPath>, op[2]]); break;
case "p": out.push(["p", ref, op[2], op[3], op[4]]); break;
}
previous = key;
}
return out;
},
};
}
export interface Decoder {
decode(wire: readonly WireOp[]): Op[];
}
export function decoder(): Decoder {
const paths = new Map<number, Path>();
return {
decode(wire) {
let previous: Path | undefined; // scoped to the batch, as in encode
const out: Op[] = [];
for (const op of wire) {
assertValidWireOp(op);
if (op[0] === "#") {
assertSafePath(op[2]);
paths.set(op[1], op[2]);
continue;
}
if (op[0] === "r") {
out.push(op);
previous = undefined;
continue;
}
// Arity tells us whether a ref is present: the short forms omit it.
const short =
(op[0] === "d" && op.length === 1) ||
(op[0] !== "d" && op[0] !== "p" && op.length === 2) ||
(op[0] === "p" && op.length === 4);
let path: Path;
if (short) {
if (previous === undefined) throw new PathError([]);
path = previous;
} else {
const ref = op[1] as PathRef;
if (typeof ref === "number") {
const resolved = paths.get(ref);
if (resolved === undefined) throw new PathError(ref);
path = resolved;
} else {
path = ref;
}
previous = path;
}
switch (op[0]) {
case "s": out.push(["s", path as NonEmptyPath, (short ? op[1] : op[2]) as JsonValue]); break;
case "d": out.push(["d", path as NonEmptyPath]); break;
case "a": out.push(["a", path as NonEmptyPath, (short ? op[1] : op[2]) as string]); break;
case "t": out.push(["t", path as NonEmptyPath, (short ? op[1] : op[2]) as number]); break;
case "p": {
const [i, r, items] = short
? [op[1] as number, op[2] as number, op[3] as JsonValue[]]
: [op[2] as number, op[3] as number, op[4] as JsonValue[]];
out.push(["p", path, i, r, items]);
break;
}
}
}
return out;
},
};
}

View file

@ -0,0 +1,113 @@
/**
* Not a test — a guard against the cliffs found during implementation. Reports
* units you can reason about: cost per WRITE, and bytes on the wire relative to
* bytes consumed.
*
* npx vitest bench delta.bench.ts
*
* The headline finding: cost is per write, not per byte. A 100-byte chunk costs
* about the same as a 1 KB chunk, because the fixed cost is the window slice and
* the overlap probe. So chunk size — set by the exec env's coalescing interval,
* not by this package — decides both throughput and wire size.
*/
import { bench, describe } from "vitest";
import { decoder, encoder, track } from "./delta-impl.ts";
const WORDS = "the quick brown fox jumps over lazy dog while considering strategies".split(" ");
const text = (n: number): string => {
let s = "";
while (s.length < n) s += `${WORDS[Math.floor(Math.random() * WORDS.length)]} `;
return s.slice(0, n);
};
const CAP = 50_000;
const WRITES = 500;
const FLUSH_EVERY = 20;
/** One write = one chunk handed to the sink. Returns nothing; bench times it. */
function drive(initial: string, chunkSize: number, cap = CAP): void {
const t = track({ out: initial });
for (let i = 0; i < WRITES; i++) {
const next = t.state.out + text(chunkSize);
t.state.out = next.length > cap ? next.slice(next.length - cap) : next;
if (i % FLUSH_EVERY === FLUSH_EVERY - 1) t.flush();
}
t.flush();
}
/**
* Measured on the reference machine, 500 writes each. Reproduced here so a
* regression is legible rather than just "slower".
*
* append only, 100 B 127 µs/write 8k writes/s 101% of input on wire
* rolling, 100 B 176 µs/write 6k writes/s 135% of input on wire
* rolling, 1 KB 68 µs/write 15k writes/s 104% of input on wire
* rolling, 8 KB 366 µs/write 3k writes/s 31% of input on wire
* pathological, 100 B 307 µs/write 3k writes/s 727% of input on wire
*
* Two things to notice.
*
* 135% at 100-byte chunks: past the cap every write emits `truncate` + `append`,
* and two op envelopes exceed a 100-byte payload — we put MORE on the wire than we
* consumed. At 8 KB it is 31%. Flush granularity is therefore a size lever, not
* just an update-rate lever, and the exec env's coalescing is what keeps chunks
* off the bad end of that curve.
*
* The pathological row is bounded only because the overlap candidate scan is
* capped. Unbounded it was 4.7 s for 2000 writes against 93 ms; the cap makes it
* give up and emit a set — larger, never wrong.
*/
describe("rolling window, 50 KB cap", () => {
bench("100 B chunks", () => { drive(text(CAP), 100); });
bench("1 KB chunks", () => { drive(text(CAP), 1024); });
bench("8 KB chunks", () => { drive(text(CAP), 8192); });
bench("pathological: one repeated character", () => { drive("x".repeat(CAP), 100); });
});
describe("below the cap", () => {
// The dominant case for most tools: startsWith fast path, no probe at all,
// and one append per flush rather than a truncate/append pair.
bench("append only, never fills", () => { drive("", 100, Number.MAX_SAFE_INTEGER); });
});
// ── Codec: what path interning and arity omission actually save ────────────
//
// Reported as wire bytes relative to inline paths, on three shapes:
// - one hot path (a rolling tool-output window): omission does the work
// - a few alternating paths (lane state): interning does the work
// - many distinct paths: neither helps, and definitions cost a little
describe("codec", () => {
const streams = {
"one hot path": () => {
const t = track({ content: [{ type: "text", text: "" }] });
return Array.from({ length: 200 }, (_, i) => {
(t.state.content[0] as { text: string }).text += `line ${i} of build output\n`;
return t.flush();
});
},
"four alternating paths": () => {
const t = track({ a: "", b: "", c: "", d: "" });
return Array.from({ length: 200 }, (_, i) => {
for (const k of ["a", "b", "c", "d"] as const) t.state[k] += `${i}`;
return t.flush();
});
},
"200 distinct paths": () => {
const root: Record<string, number> = {};
for (let i = 0; i < 200; i++) root[`f${i}`] = 0;
const t = track(root);
return Array.from({ length: 5 }, (_, r) => {
for (let i = 0; i < 200; i++) t.state[`f${i}`] = r + 1;
return t.flush();
});
},
};
for (const [label, build] of Object.entries(streams)) {
bench(label, () => {
const enc = encoder();
const dec = decoder();
for (const ops of build()) dec.decode(enc.encode(ops));
});
}
});

View file

@ -0,0 +1,133 @@
/**
* Runnable illustrations of the two things that are easy to get wrong.
*
* node --experimental-strip-types delta.examples.ts
*
* Nothing here is a test — `delta.test.ts` covers the same ground with
* assertions. This exists to be read.
*/
import { apply, decoder, encoder, isBase, track, type Op } from "./delta-impl.ts";
const show = (label: string, ops: Op[]) => {
console.log(` ${label.padEnd(22)} ${ops.length} op(s), base=${isBase(ops)}`);
for (const op of ops) console.log(` ${trunc(JSON.stringify(op), 66)}`);
};
const trunc = (s: string, n: number) => (s.length <= n ? s : `${s.slice(0, n)}…`);
// ─────────────────────────────────────────────────────────────────────────
// 1. Replacing the whole value
// ─────────────────────────────────────────────────────────────────────────
console.log("\n1. REPLACING THE WHOLE VALUE\n");
const before = { user: { id: "u1", name: "ada" }, items: ["a", "b"], note: "n".repeat(120) };
const after = { user: { id: "u2", name: "bob" }, items: ["x"], note: "m".repeat(120) };
{
// The wrong way. `state` is a property of the TRACKER, not of the tracked
// object. Without a setter this swaps the proxy for a plain object and every
// later mutation is silently untracked — no error, no ops.
//
// The implementation defines `state` as a setter precisely so this works.
const t = track<Record<string, unknown>>(structuredClone(before));
t.flush(); // drain the opening base batch
t.state.items = ["changed first"]; // recorded…
t.state = structuredClone(after); // …then discarded: it described a dead value
show("assign to .state", t.flush());
t.state.user = { id: "u3" }; // still tracked afterwards
show(" and still tracked", t.flush());
}
{
// Dead-op elimination: a field written three times emits one op, and a parent
// replaced after its child emits one. Object key order is not preserved by
// this pass — see delta.md §5.1.
const t = track<Record<string, any>>(structuredClone(before));
t.flush(); // drain the opening base batch
t.state.items = ["first"];
t.state.items = ["second"];
t.state.user.name = "dropped";
t.state.user = { id: "u9" };
show("4 writes, 2 survive", t.flush());
}
// ─────────────────────────────────────────────────────────────────────────
// 2. Why checkpointing exists
// ─────────────────────────────────────────────────────────────────────────
console.log("\n2. RECOVERY LENGTH\n");
/**
* A bash-shaped workload: output arrives continuously and the sink writes
* durably on an interval. Each flush is one batch appended to a list.
* On resume the reader folds back to the last BASE batch, so that distance is
* the cost of recovery.
*/
function simulate(writes: number, checkpointEvery: number | null) {
const CAP = 50_000;
const t = track({ out: "" });
const isBaseBatch: boolean[] = [];
for (let i = 0; i < writes; i++) {
const next = `${t.state.out}[${i}] cc -c src/file${i}.c -o build/file${i}.o\n`;
t.state.out = next.length > CAP ? next.slice(next.length - CAP) : next;
if (checkpointEvery !== null && i % checkpointEvery === checkpointEvery - 1) t.rebase();
const ops = t.flush();
if (ops.length > 0) isBaseBatch.push(isBase(ops));
}
const lastBase = isBaseBatch.lastIndexOf(true);
return { written: isBaseBatch.length, replay: isBaseBatch.length - lastBase - 1 };
}
for (const [label, every] of [["never", null], ["rebase() every 50", 50]] as const) {
const r = simulate(500, every);
console.log(` ${label.padEnd(18)} ${r.written} batches written → ${r.replay} to replay on recovery`);
}
console.log(`
Nothing produces a base batch on its own: flush() emits ops, and a replacement
happens only when the producer asks for one. So a stream of appends stays a
stream of deltas indefinitely, and recovery grows without bound.
Two callers need rebase():
- pendingToolOutput, during a long-running command. Without it a ten-minute
build leaves hundreds of batches to fold on resume.
- a facet host on resubscribe. facets.md §9.2 requires the first batch of a
subscription to be a base batch, and the host must be able to produce one
on demand.
`);
// ─────────────────────────────────────────────────────────────────────────
// 3. A replica always agrees
// ─────────────────────────────────────────────────────────────────────────
console.log("3. PRODUCER AND REPLICA AGREE\n");
{
// The whole loop: tracker -> encode -> [boundary] -> decode -> apply.
const t = track<Record<string, unknown>>(structuredClone(before));
const enc = encoder();
const dec = decoder();
let replica: unknown;
const step = (label: string, mutate: () => void) => {
mutate();
const ops = dec.decode(enc.encode(t.flush()));
if (ops.length === 0) return;
// No base-batch branch: apply handles `r` by replacing, and tolerates an
// undefined target because a root op never reads it.
replica = apply(replica, ops);
const same = JSON.stringify(replica) === JSON.stringify(t.target);
console.log(` ${label.padEnd(22)} ${ops.length} op(s) ${isBase(ops) ? "BASE " : "delta"} matches: ${same}`);
};
step("opening base batch", () => {});
step("append to a string", () => { (t.state.note as string); t.state.note = `${t.state.note}!`; });
step("push", () => { (t.state.items as string[]).push("c"); });
step("nested set", () => { (t.state.user as Record<string, unknown>).name = "cy"; });
step("delete", () => { delete t.state.note; });
step("whole replacement", () => { t.state = structuredClone(after); });
step("rebase", () => { t.rebase(); });
}
console.log();

View file

@ -0,0 +1,617 @@
# Delta Tracking and the Op Vocabulary
> **Production status:** landed in `packages/chord/src/delta/index.ts`, with tests in `packages/chord/test/delta.test.ts`. Chord owns the dependency-free `Op`/`WireOp`, tracker, applier, codec, and validation boundary; Session storage, the Harness, and facets consume it. The implementation and tests beside this document are historical prototype and benchmark evidence, not production source.
>
> The landed tracker computes deltas at `flush()` from a dirty tree and baseline rather than retaining one op per mutation. That resolves FINDINGS D1. Production remeasurement also closes FINDINGS D2: the generic string path is below surrounding replication/rendering costs, so an explicit append/truncate API is rejected ([decision](append-decision.md)).
One mechanism covers assistant partials, tool output, tool details, lane state, and arbitrary facet state — on the wire and in durable storage.
```bash
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/delta.test.ts
# run from packages/chord
```
## 1. Why not an existing library
Immer, Valtio, Mutative and Colyseus all **record effects, not intent**. A string
is a leaf, so `text += chunk` is one write of the whole new string; there is no
representation in which the library could know you appended. `unshift` is the
same story — the engine shifts every index, so each is a write.
Measured, all four:
| operation | what they emit |
| --- | --- |
| `text += "x"` on 40 KB | `replace` carrying 40 KB |
| `arr.unshift(x)` on 100 items | ~100 index writes |
| `arr.pop()` | a write to a `length` path, which is not a document location |
Immer's own pitfalls page states the guarantee: patches are correct and
**explicitly not minimal**. Colyseus documents the same array weakness — removing
the first of 20 items costs 38 extra bytes.
Diffing base against result ourselves does not fix it either. A differ sees two
values, not intent. Prefix comparison catches pure growth, but a rolling window
that drops from the front *and* appends is neither a prefix nor a suffix of the
old value; measured on 5000 elements the fallback was
`{index: 0, remove: 5000, items: 5000}` — a full replace, in exactly the case the
optimisation existed for.
So the win is not a blind whole-value differ. The tracker records which paths and array operations became dirty, then compares only those subtrees against its accepted baseline at `flush()`.
## 2. Ops
**Tuples are the form. Everywhere. There is no keyed variant.**
```ts
export type Seg = string | number;
export type Path = readonly Seg[];
/** Non-empty: `s`/`d`/`a`/`t` cannot target the root. Enforced by the type. */
export type NonEmptyPath = readonly [Seg, ...Seg[]];
export type Op =
| readonly ["r", JsonValue] // replace the value
| readonly ["s", NonEmptyPath, JsonValue] // set
| readonly ["d", NonEmptyPath] // delete
| readonly ["a", NonEmptyPath, string] // append
| readonly ["t", NonEmptyPath, number] // truncate, in chars
| readonly ["p", Path, number, number, JsonValue[]] // splice: index, remove, items
```
Interning, id references and omitted paths are **not** here — they live in
`WireOp` and exist only between `encode` and `decode` (§4).
**`r` is the only op that replaces a whole value.** It is an op rather than a
frame kind for the same reason `a` and `t` exist: they are specialisations of `s`
that are smaller and state intent.
Do not encode a replacement as a set at the root path. Base-batch detection is a
**correctness boundary** — recovery stops there (§9) — so it must be a token
comparison, `ops[0]?.[0] === "r"`, and not a path inspection. A path-inspecting
predicate has to handle the two-element arity form, where `op[1]` is a value
rather than a path, and misclassifies `["s", []]`.
**Only `p` may target the root**, and only because a tracked value can itself be
an array — `entries.push(x)` on a root array is `["p", [], 3, 0, [x]]`. A `p`
covering its entire target is normalised at flush time to `r` (root) or `s`
(nested), so a root `p` is always a partial modification. The other four verbs
take a `NonEmptyPath`: `["s", [], v]` and `["d", []]` do not typecheck.
**Do not add a keyed in-memory shape with a codec at the serialization boundary.**
Two representations of one thing means two appliers, two size estimates, and a
conversion nobody needs. Ops are tuples in memory, on the wire, and on disk;
readability is what a debug formatter is for.
Six verbs. No `move`, `copy` or `test`: the first two are size optimisations for
a case we do not have, and `test` belongs to a conflict model we do not have,
since there is exactly one authoritative writer.
`chars` counts UTF-16 code units, matching `String.prototype.slice`. **Not bytes.**
Byte caps are a producer concern and never cross a boundary.
The format is JSON-Patch-*shaped*, not RFC 6902 conformant. That costs nothing:
RFC 6902 has had no successor since 2013, still has the same six ops, and nothing
anywhere adds a string splice. Immer was never conformant either.
## 3. The tracker
State is **plain TypeScript**. No containers, no handles, no schema, no
decorators. You mutate normally.
```ts
const t = track(laneView);
t.state.operation.streamingMessage.content[0].text += delta; // -> append
t.state.transcript.push(entry); // -> splice
t.state.tools[0].details.failures.push({ name, msg }); // -> splice
delete t.state.config.model; // -> delete
const ops = t.flush();
```
Three mechanisms, one per case:
**Objects** — ordinary `set` / `deleteProperty` traps.
**Arrays — mutator methods intercepted in the `get` trap.** `arr.push(x)` passes
through `get(arr, "push")` first, so the tracker returns its own function that
records `splice(len, 0, [x])` and then delegates. Intent is captured *before* the
engine performs its index writes, which is why `unshift` is one op rather than
O(n). This is the thing no other library does.
**Strings — dirty marking on the `set` trap, overlap detection at `flush()`.** Strings are primitives, so there is nothing to intercept. Given the accepted `prev` and final `next`, find the longest suffix of `prev` that is a prefix of `next`:
```
overlap === prev.length -> append(next.slice(overlap))
overlap > 0 -> truncate(prev.length - overlap) + append(rest)
overlap === 0 -> set(next)
```
Always correct, because the overlap is **verified**, not guessed. The rolling
window works:
```ts
s.out = s.out.slice(4) + "!!!";
// ["truncate", ["out"], 4]
// ["append", ["out"], "!!!"]
```
### 3.1 Overlap must not use a hand-written KMP
A KMP failure function is asymptotically right and **47x slower in practice**,
because it runs character by character in JS. Measured on 2000 window slides over
a 50 KB string: 2870 ms for KMP, 61 ms for a native probe.
The probe: `indexOf` a short head of `next` within `prev`, verify each candidate
with `endsWith`. Both are native. Verified identical to KMP across 200 window
slides.
A `startsWith` fast path runs first, so a pure append — the dominant case — skips
the probe entirely.
### 3.2 Rules the implementation must hold
**Inserted values are adopted; emitted payloads are cloned at flush.** Callers may retain read-only references but must not mutate adopted objects outside the tracker. `flush()` clones payloads and advances a cloned accepted baseline, so emitted ops never alias producer state.
**`x = undefined` normalises to `delete`.** JSON has no `undefined`, and a `set`
with a missing value is indistinguishable from a lost value after a
`JSON.stringify` round trip.
**`arr.length = n` is translated, never emitted as a path.** Shrink becomes a
splice, grow becomes a splice of nulls. `length` is a real mutation but not a
document location — this is the bug Immer ships (issue 208). Dropping it silently,
as an early prototype did, means `arr.length = 0` never reaches the replica.
**Numeric array keys are normalised to numbers.** Proxy traps deliver `"0"`;
paths carry `0`. Valtio ships the string form and it is worse for both size and
comparison.
**Proxies are cached per object and path**, so `s.a === s.a` and proxies are not
rebuilt on every access.
**Scope is `JsonValue` only, and it must be CHECKED, not assumed.**
`structuredClone` is not a JSON check — it clones `Map`, `Set`, `Date` and
`RegExp` happily, and the result then differs from what `JSON.stringify` produces,
so producer and replica diverge silently. Assert the value at record time and
reject symbol keys. See §7.4 for what leaks without it.
**Exactly two ops reach the root**, and the applier's root branch must handle
both: `r` always, and `p` when the tracked value is itself an array.
`s`/`d`/`a`/`t` cannot, by type, which makes the branch's fallthrough provably
unreachable. Handling only `r` there passes every hand-written test and fails on
the first randomised sequence that produces a root splice.
**Two writes to one address inside a transaction chain.** When several writes to
the same address are emitted in one commit, each must be recorded against the
previous one's result, not against pre-transaction state. The applier replays them
in order; anything else applies the second against a stale base. This did not fire
in the workload that surfaced it — it would have corrupted silently when it did.
### 3.2.1 Replacing the whole value
Assigning to `state` replaces it:
```ts
tracker.state = next; // emits ["r", next]; discards ops recorded before it
```
`state` **must be a setter on the tracker**, not a plain property. Without one,
`tracker.state = next` swaps the proxy for a plain object and every later
mutation is silently untracked — no error, no ops, `target` unchanged. It is also
the first thing anyone tries.
Prior ops are discarded because they describe a value that no longer exists.
Only a *full* replacement collapses to `r`. Rewrite part of the value and the ops
survive, because they are genuinely cheaper:
```
partial rewrite 2 op(s), base=false
["s",["user"],{"id":"u2","name":"bob"}]
["s",["items"],["x"]]
```
### 3.2.2 The first flush is a base batch
`track()` opens a stream, and a consumer starts with nothing, so the first flush
always emits `["r", value]` — carrying any mutations made before it:
```ts
const t = track({ x: 0, l: [] });
t.state.x = 100;
t.state.l.push("xyz");
t.flush(); // [["r", { x: 100, l: ["xyz"] }]]
t.state.x = 101;
t.flush(); // [["s", ["x"], 101]]
```
Requiring the producer to remember a `rebase()` first would fail at runtime, in
the consumer, far from the mistake.
### 3.2.3 Forcing a base batch later
```ts
tracker.rebase(); // next flush is ["r", value]; value unchanged
```
Discarding the pending ops is correct: the proxy mutates the target directly, so
the value already carries them. `tracker.state = tracker.state` has the same effect in the landed implementation, but `rebase()` states the intent directly.
**Nothing produces a base batch on its own.** `flush()` emits ops; a replacement
happens only when the producer asks for one. So a stream of appends stays a
stream of deltas indefinitely.
Recovery replays from the last base batch (§9). From `delta.examples.ts`, a
bash-shaped workload of 500 durable writes into a 50 KB rolling window:
| | batches written | to replay on recovery |
| --- | --- | --- |
| never | 500 | **499** |
| `rebase()` every 50 | 500 | 0 |
Two callers need this:
- **A durable sink.** `pendingToolOutput` accumulates one batch per checkpoint for
as long as a command runs. A `make -j8` running for ten minutes otherwise leaves
hundreds of batches to fold on resume. Checkpointing every N makes "at most N
batches to replay" a policy rather than an accident.
- **A facet host on resubscribe.** `facets.md` §9.2 requires the first batch of
every subscription to be a base batch, and a consumer that sees a gap
resubscribes to get one. The host has to be able to produce one on demand rather
than wait for one to happen by chance.
### 3.3 Known gaps
- `sort` / `reverse` / `fill` / `copyWithin` mark the array dirty and emit the resulting structural/index changes rather than preserving the producer's method intent. Add a dedicated op only if a measured workload needs one.
- A manual index-shift loop (`for (…) a[i] = a[i+1]`) can still cost O(n) sets. Correct, not minimal, and unavoidable — the producer genuinely wrote every element.
- Rolling-window string assignment still runs overlap discovery at flush, deliberately: production measurements do not justify text-specific tracker state or API surface ([decision](append-decision.md)).
### 3.4 Constraints the string algorithm must hold
Each of these is cheap to get wrong in a way that passes hand-written tests.
**A fixed-length probe cannot find an overlap shorter than the probe.** The head
must actually occur in `a`: `"abcdefgh"` -> `"defghxyz"` overlaps by 5, and a
64-character head cannot occur in an 8-character string. Try a long head first,
then fall back to a one-character head. A test using only 50 KB strings, where
overlaps are huge, will pass against a broken implementation.
**Bound the candidate scan.** Repetitive output — a build log, or a run of one
character — makes a long head match at thousands of positions, each costing a full
`endsWith`. Unbounded, 2000 slides over a 50 KB window take 4.7 s against 93 ms
bounded. Exceeding the bound returns 0, which emits a set: larger, never wrong.
**Property tests before anything depends on this.** 3000 random mutation
round-trips find the root-splice hole in §3.2 on the first run; the hand-written
cases do not. `delta.test.ts` is the suite to port.
## 4. The codec
Two vocabularies, not one.
`Op` is what the tracker produces and `apply` consumes. **Paths are always
inline.** It knows nothing about a dictionary.
`WireOp` is what crosses a boundary. It adds exactly two compressions:
```
["#", id, path] defines an id, emitted on a path's SECOND use
a numeric PathRef references a previously defined id
a shortened tuple reuses the previous op's path; arity disambiguates
```
`encoder().encode(ops): WireOp[]` and `decoder().decode(wire): Op[]` are the only places either exists. Keeping them out of `Op` means `apply` has no id resolution, no `#`
case, and no previous-path state — three branches removed from the hot path —
and dirty-tree generation always works with inline paths before encoding.
`["r", value]` carries no path, so it encodes to itself. That is why `isBase`
works unchanged on either vocabulary.
### 4.1 Two rules that are easy to get wrong
**One encoder/decoder pair per stream.** The id table spans a whole subscription
or file: a path interned in batch 3 is referenced in batch 40. A second consumer
that subscribes at batch 40 has never seen the definition, so it needs its own
encoder. Sharing one across consumers hands the late subscriber ids it cannot
resolve, and a base batch does not rescue it — `["r", value]` carries no refs and
leaves the table empty.
**Reset the table on a base batch.** A reader replays from the *last* base batch
with a fresh decoder, so everything after one must be self-contained. Carrying
ids across a replacement emits references to definitions the reader never saw:
```
4 : [["a",0,"x4"],["a",1,"y4"]]
5 BASE: [["r",{…}]]
6 : [["a",0,"x6"]] <- id 0 was defined in batch 1
-> RECOVERY FAILED: PathError - unresolvable path: 0
```
**Arity omission is scoped to a batch.** Letting it span batches makes a batch's
first op depend on the previous batch's last one, so a reader that skips or
reorders a batch decodes into the wrong path. Ids are the only cross-batch state,
and the dictionary makes those explicit.
### 4.2 Intern on second use, not first
A definition costs more than the path it replaces, so interning on first use
loses on every path written exactly once — and most are. Measured on a bulk
workload: **255.5 KB first-use versus 179.6 KB with no interning at all.**
### 4.3 Measured
Wire bytes against inline paths, round-trip verified:
| stream | inline | wire | saved |
| --- | --- | --- | --- |
| one hot path (a rolling tool-output window) | 11,290 B | 7,538 B | **33.2%** |
| four alternating paths (lane state) | 14,160 B | 11,032 B | **22.1%** |
| 200 distinct paths | 16,455 B | 16,035 B | 2.6% |
Omission does the work in the first case, interning in the second. In the third
neither helps and the definitions cost a little — which is the case second-use
interning exists to bound.
## 5. Flush emits ops, and drops the dead ones
`flush()` computes ops for the dirty paths against the last accepted baseline. It does not retain one op per mutation, so repeated and interleaved writes stay bounded by changed state rather than write count. There is no size comparison or replacement heuristic: a replacement is something the producer asks for by assigning `state` or calling `rebase()`.
An earlier design compared op bytes against the value's serialised size and
replaced when ops were larger. It was removed. Measured against emitting ops
unconditionally, on six workloads, it changed the output in two — both requiring
hundreds of distinct paths to change in a single flush. Neither the harness nor
the replication path does that: `LaneSnapshot` is folded one event at a time, and
the widest case (`run_end`) touches four fields. The rule cost a per-flush
comparison, a running size estimate, and an invalidation rule for the ops that
could not maintain it.
### 5.1 Dirty-tree collapse
The landed tracker records only which subtrees are dirty. At flush it compares each dirty subtree's accepted baseline with its final value and emits the surviving structural change. Repeated writes to one field collapse naturally; alternating writes to two fields retain two dirty paths rather than one op per mutation; a parent replacement subsumes dirty descendants where the final comparison permits it. `packages/chord/test/delta.test.ts` pins the interleaved rolling-window case at at most three ops after 1,000 alternating writes.
This replaces the prototype's backwards dead-op pass and adjacent-only coalescer. Do not port those algorithms into Chord: flush-time generation is the D1 fix.
> **Object key order is not a replicated invariant.** Values round-trip, but delete-and-reinsert activity within one flush can produce a different insertion order on a replica. Do not hash or content-address a replicated value, and sort explicitly where display order matters.
## 6. There is no frame type
The logical batch is `Op[]`; transport and durable storage carry the statefully encoded `WireOp[]`. Do not wrap either batch.
**`seq` does not belong in the payload.** It would defend against a lossy
transport we do not have: a durable list element already carries `seq` from
storage, an in-process callback cannot skip, and SSE either delivers in order or
breaks — and a break means resubscribe, which means a base batch. A second copy of
storage's `seq` could only disagree with the first. The SSE binding stamps `id:`,
which is where transport metadata belongs.
**Nothing else needs wrapping either.** A `kind` discriminator is redundant once a
replacement is an op (§2), and the address already says which value a batch
belongs to. What would remain is a struct around an array.
A producer batch is just `Op[]`, and its encoded boundary form is `WireOp[]`. A **base batch** is one whose first op is `r`;
`isBase(ops)` is `ops[0]?.[0] === "r"`, exact rather than heuristic because flush
guarantees `r` appears at index 0 or not at all (§5).
Everything a frame used to do is now done by something that already existed:
| was on the frame | now |
| --- | --- |
| `seq` | the list element's `seq`, or the SSE `id:` |
| `kind: "replace"` | the `r` op |
| which value it belongs to | the address |
| "this is a snapshot" | the `"base"` storage tag |
**Resubscription is a base batch plus buffered batches** — the same thing the lane
adapter already does for the harness: snapshot first, then whatever accumulated
while the client caught up. There is no `Last-Event-ID` and no cross-subscription
resume; that would need a retained op log, which §5 deliberately does not keep.
A consumer that sees a gap, fails to resolve a path, or connects cold takes one
path: ask for a base batch. That is why there is no `Rebase` type.
The cost, stated plainly: **a consumer cannot detect staleness from the payload
alone.** It relies on the transport reporting a break. For SSE and in-process that
is sound. If a lossy or multiplexed transport ever appears, sequencing goes on that
transport's envelope — not back into the ops.
## 7. Safety
Ops arrive from a facet, a plugin compartment, or a tool whose details may echo
model output. **None of it is trusted input**, and the boundary where untrusted
data meets trusted machinery is where this design has to hold.
The reference point is CVE-2025-55182 — RCE in React Server Components, CVSS 10.0,
exploited in the wild. Their Flight protocol is a compact tagged wire format that
reconstructs structure, so the resemblance is real. The failure was *"fails to
validate the structure correctly... treats the fake object as genuine"*: a forged
Chunk resolved as a Promise and exposed internal state containing gadgets to reach
`Function`.
**We are structurally safer, and not because of diligence:**
| | Flight | ops |
| --- | --- | --- |
| can describe runtime objects | yes — Chunks resolve as Promises | no |
| can reference code or modules | yes — client components | no |
| values | arbitrary object graphs | `JsonValue` |
| worst case from a forged payload | RCE | corrupted replica state |
Flight *must* reference code; that is its job. An op can only put a `JsonValue` at
a path, so there is no first link in a gadget chain. This is why an attacker cannot
plant a non-resolving `then` on `Object.prototype`: a data-only `then` is not
callable, and `await` resolves normally when `then` is not callable.
> **Rule: no op may name something the host resolves.** Not a mutation name, not a
> component id, not a module reference. Mutation names were removed for an
> unrelated reason (§8); this is the second and better reason never to reintroduce
> them, because it is the ingredient that made Flight exploitable.
### 7.1 Paths are the dangerous part
`JSON.parse` is safe on its own — `{"__proto__":{}}` becomes an *own* property.
The hazard is `parent[key]`, which is exactly what applying a path does.
A single key is not enough. `x["__proto__"] = v` swaps *x's own* parent, which is
local. It takes a **walk** — `x["__proto__"]["polluted"] = v` — to reach the shared
prototype, and a path is precisely a walk.
`constructor` is worse than `__proto__`, because `({}).constructor.constructor` is
`Function`. That ladder is closed here only because op values cannot be functions;
it is closed properly by rejecting the segment.
**Reserved path segments: `__proto__`, `constructor`, `prototype`.** Rejected at
record time *and* at apply time, including through an interned path id.
They are reserved as *segments*, not as values. An object with a literal
`"__proto__"` key replicates fine as a whole value; only a path walking *through*
it is refused. This is a genuine restriction on what is mutable — document it, do
not pretend it away.
Two further measures on the same hazard:
- **Write with `Object.defineProperty`**, never assignment, so an inherited setter
cannot run.
- **Resolve own properties only** (`Object.hasOwn`), so a walk cannot escape into
the prototype chain and an inherited getter cannot fire. This blocks
`constructor` a second time, since it is inherited.
What prototype pollution actually buys an attacker here is narrower than it first
appears: **it flips defaults, it does not overwrite explicit values.** An own
property shadows. So `{name:"bob", isAdmin:false}` is unaffected; an option bag read
as `opts.skipSandbox` is not. Our own `ShellExecOptions`, `ShellOutputCaptureOptions`,
`ListReadOptions` and `TrackerOptions` are exactly that shape. Note also that
`"x" in {}` and `const {x = false} = {}` both lie under pollution — which is why
the applier uses `Object.hasOwn`.
### 7.2 Array indices
An index may address an existing element or append exactly one past the end.
This is not an arbitrary cap; it is what keeps the value a `JsonValue`. A sparse
array does not survive a JSON round trip — holes serialise to `null` and return as
real properties — so `arr[7] = x` on a length-3 array already produces state a
replica cannot match. Rejecting the write is more honest than diverging.
It removes a denial of service as a side effect rather than as its purpose:
`["s",["xs",4294967290],1]` would otherwise allocate a 4.29-billion-entry array
from one op. Growth stays available and stays proportional, because
`arr.length = n` is emitted as a splice of explicit nulls whose op size grows with
the gap.
### 7.3 Validate op structure on decode
This is the RSC lesson applied to us. A decoder must not trust tuple shape.
Measured against an unvalidated applier:
| malformed op | result |
| --- | --- |
| `["p",["xs"],0,0,"not-an-array"]` | string spread into the array: `["n","o","t",…]` |
| `["s","a",9]` — path is a string | accepted; `"a".slice(0,-1)` is `""`, so it wrote at the root |
| `["ZZZ",["a"],9]` | silently ignored, replica diverges with no error |
Validate: verb is known, arity matches the verb, a path is an array of strings and
non-negative integers, `p` carries integer index and count plus an array of items,
`#` defines an array path. An unknown verb is an **error**, not a no-op — silently
skipping it is how a newer producer's op disappears and a replica drifts.
**One validator per vocabulary.** `assertValidOp` guards `apply` and rejects ids,
short forms and `#`; `assertValidWireOp` guards `decode` and permits them.
Validating a decoded op against the wire grammar is laxer than its own type: a
two-element `["s", value]` would pass, and `apply` would then read the value as a
path.
### 7.4 The tracker's cage leaks on types, not shape
A facet cannot forge an op: it mutates plain objects and the tracker builds the
tuples, so shape is well-formed by construction. Values and keys are another
matter, and `structuredClone` is **not** a JSON check.
| what a facet writes | what happens |
| --- | --- |
| a function | throws (`DataCloneError`) |
| a BigInt, a cycle | throws |
| `new Map([[1,2]])` | op carries `{}`, producer keeps a real Map — **silent divergence** |
| `new Date(0)` | op carries an ISO string, producer keeps a Date |
| `state[Symbol("s")] = 1` | emits `["s",[null],1]` — **a malformed op from our own tracker** |
So the tracker must check what it is given, at record time, where the mistake is:
- **Reject symbol keys.** They are not path segments.
- **Assert `JsonValue` on every recorded value.** Reject `Map`, `Set`, `Date`,
`RegExp`, typed arrays and class instances explicitly. The scope restriction in
§3.2 was a comment; it has to be a check.
A getter on state is safe — the trap records the computed result — and a facet
value that *looks* like an op is nested inside `["s", path, value]` and can never
be read as a top-level op, because nothing flattens.
### 7.5 Applier
```ts
export function apply<T>(target: T | undefined, ops: readonly Op[]): T;
```
Six verbs, no domain knowledge, no library, no tool code, no registry lookup, and
**no path table** — ids and omitted paths are resolved by `decode` before `apply`
sees anything (§4).
It returns the value rather than mutating in place, because `r` replaces it
outright.
Runs against a **plain mutable object the consumer owns**. It must not be pointed
at a value produced by Immer, which deep-freezes and would throw.
A page of code in any language, which is the property that keeps non-JS consumers
viable. Colyseus ships decoders for C#, Lua and Haxe on the same basis.
## 8. What this removes from the codebase
Concrete deletions, not simplifications in principle:
- **Immer**, from the harness and from the facet layer. See §1 for why a
patch-producing library cannot serve this purpose.
- **Per-type reducers.** There is one applier, `apply(target, ops)`, with no
domain knowledge and no registry. Nobody writes a fold for `ToolOutputState` or
for a plugin's shape.
- **`detailMutations` and `initialDetails`**, and with them the argument that
`details: unknown` forces special handling. A structural tracker never needs
the type.
- **`Rebase` as a reducer return value.** A fold that cannot apply leaves state
alone and the host sends a base batch.
- **Mutation names on the wire**, and therefore mutation-name version skew. Names
never cross a boundary, so adding or renaming one is not a breaking change.
And three things not to build, each of which looks reasonable until §1:
- **A blind whole-value differ.** The landed tracker compares only dirty subtrees against its accepted baseline; it does not scan unrelated state.
- **A keyed op shape plus a codec.** Tuples are the form everywhere (§2).
- **A frame wrapper.** What travels is `Op[]` (§6).
## 9. Durable form
A tracked value is stored as a **list of encoded `WireOp[]` batches**, one appended per flush. Base batches carry the storage tag `"base"`; one stateful decoder per value decodes them before `apply`.
Recovery reads backwards to the last base batch and applies forward:
```ts
readList(address, { order: "desc", stopAtTag: "base", limit: 100 })
```
`stopAtTag` is a stop condition *within a page*: if no base batch is in the page,
the consumer pages again with the cursor. The tag lives on the storage record
beside `seq`, never inside the value, so storage never parses ops. See
[scopes.md](../02-scopes/scopes.md) §11.
This is what makes "a replacement truncates recovery" (§5) real rather than
aspirational — a reader stops at the last base batch instead of replaying from
the beginning.
## 10. Open questions
- Prefix interning (a trie over path heads) if a workload emerges with many
distinct paths sharing long prefixes. Second-use interning (§4) removes the
pathological case; this would go further.
- Whether array reordering needs a dedicated op; the landed tracker currently marks the array dirty and emits the resulting structural/index changes.
- Cross-language replicas: the applier is a page of code in any language, but the
interning tables and the wire framing are not specified for a non-JS consumer.
Address interning in the JSONL log is the same trick as path interning, applied
one layer up over `namespace` + `key`. It is **not** an open question — it is
specified in [scopes.md](../02-scopes/scopes.md) §12 and is a separate dictionary from this one.

View file

@ -0,0 +1,563 @@
import { describe, expect, it } from "vitest";
import { apply, assertJsonValue, assertValidOp, assertValidWireOp, decoder, encoder, isBase, overlap, track, type Op, type WireOp } from "./delta-impl.ts";
const PAD = "p".repeat(400); // see "adaptive emission": a delta must be able to win
describe("tracker: intent", () => {
it("records an append, not a replacement", () => {
const t = track({ s: "", pad: PAD });
t.flush(); // drain the base batch
t.state.s += "ab";
t.state.s += "cd";
const ops = t.flush();
expect(ops).toEqual([["a", ["s"], "abcd"]]);
expect(apply({ s: "", pad: PAD }, ops)).toEqual({ s: "abcd", pad: PAD });
});
it("recovers truncate+append from a rolling window", () => {
// The case every effect-recording library degrades to a whole-value set.
const t = track({ s: "abcdefgh", pad: PAD });
t.flush(); // drain the base batch
t.state.s = t.state.s.slice(3) + "xyz";
const ops = t.flush();
expect(ops).toEqual([["t", ["s"], 3], ["a", ["s"], "xyz"]]);
expect(apply({ s: "abcdefgh", pad: PAD }, ops)).toEqual({ s: "defghxyz", pad: PAD });
});
it("records array intent as one splice", () => {
const t = track({ xs: [1, 2], pad: PAD });
t.flush(); // drain the base batch
t.state.xs.push(3);
expect(t.flush()).toEqual([["p", ["xs"], 2, 0, [3]]]);
});
it("normalises undefined to a delete", () => {
const t = track({ a: 1, pad: PAD });
t.flush(); // drain the base batch
(t.state as Record<string, unknown>).a = undefined;
expect(t.flush()).toEqual([["d", ["a"]]]);
});
});
describe("tracker: root ops", () => {
it("splices a value that is itself an array", () => {
const t = track([1, 2, 3, PAD]);
t.flush(); // drain the base batch
t.state.push(4);
const ops = t.flush();
expect(ops).toEqual([["p", [], 4, 0, [4]]]);
expect(apply([1, 2, 3, PAD], ops)).toEqual([1, 2, 3, PAD, 4]);
});
it("normalises a splice covering the whole root to a replacement", () => {
const t = track([1, 2, 3]);
t.flush(); // drain the base batch
t.state.splice(0, 3, 9);
const ops = t.flush();
expect(ops).toEqual([["r", [9]]]);
expect(isBase(ops)).toBe(true);
});
it("normalises a nested splice-all to a set, not a replacement", () => {
const t = track({ xs: [1, 2, 3], pad: PAD });
t.flush(); // drain the base batch
t.state.xs.splice(0, 3, 9);
expect(t.flush()).toEqual([["s", ["xs"], [9]]]);
});
it("normalises length = 0 on the root", () => {
const t = track([1, 2, 3]);
t.flush(); // drain the base batch
t.state.length = 0;
const ops = t.flush();
expect(ops).toEqual([["r", []]]);
expect(apply([1, 2, 3], ops)).toEqual([]);
});
});
describe("replacing the whole value", () => {
// The proxy cannot observe `tracker.state = next`: `state` belongs to the
// tracker, not to the tracked object, so assigning to it swaps the proxy for a
// plain object and silently stops tracking. Replacement must be a method.
it("assigning to state emits a base batch and keeps tracking", () => {
const t = track<Record<string, unknown>>({ p: 1, q: PAD });
t.flush(); // drain the base batch
t.state.p = 2;
t.state = { r: 9, s: "new" };
const ops = t.flush();
expect(ops).toEqual([["r", { r: 9, s: "new" }]]);
expect(isBase(ops)).toBe(true);
expect(apply({}, ops)).toEqual(t.target);
t.state.r = 10; // the new value is tracked
expect(t.flush()).toEqual([["s", ["r"], 10]]);
});
it("assigning to state discards ops recorded before it", () => {
// They describe a value that no longer exists.
const t = track<Record<string, unknown>>({ p: 1, q: PAD });
t.flush(); // drain the base batch
t.state.p = 2;
t.state = { z: 1 };
expect(t.flush()).toEqual([["r", { z: 1 }]]);
});
it("a partial rewrite keeps its ops", () => {
const t = track<Record<string, unknown>>({ a: "x".repeat(200), b: "y".repeat(200) });
t.flush(); // drain the base batch
t.state.a = "p";
const ops = t.flush();
expect(isBase(ops)).toBe(false);
expect(ops).toEqual([["s", ["a"], "p"]]);
});
it("rebase() forces a base batch without changing the value", () => {
// This is the checkpoint. Recovery replays from the last base batch, so a
// producer must be able to bound that rather than depend on the adaptive
// size rule, which compares each flush independently.
const t = track<Record<string, unknown>>({ p: 1, pad: PAD });
t.flush(); // drain the base batch
t.rebase();
const ops = t.flush();
expect(isBase(ops)).toBe(true);
expect(apply({}, ops)).toEqual(t.target);
});
it("rebase() applies once, not to every later flush", () => {
const t = track<Record<string, unknown>>({ p: 1, pad: PAD });
t.flush(); // drain the base batch
t.rebase();
t.flush();
t.state.p = 2;
expect(t.flush()).toEqual([["s", ["p"], 2]]);
});
});
describe("the first flush", () => {
it("is always a base batch", () => {
// A consumer starts with nothing, so a stream opening with deltas has
// nothing to apply them to. Requiring the producer to remember would fail
// at runtime, in the consumer, rather than where the mistake is.
const t = track<Record<string, unknown>>({ x: 0 });
t.state.x = 100;
const ops = t.flush();
expect(isBase(ops)).toBe(true);
expect(ops).toEqual([["r", { x: 100 }]]); // carries mutations made before it
});
it("is followed by deltas", () => {
const t = track<Record<string, unknown>>({ x: 0, pad: PAD });
t.flush();
t.state.x = 1;
expect(t.flush()).toEqual([["s", ["x"], 1]]);
});
it("emits nothing when untouched", () => {
// An unmutated tracker has nothing to say; the base batch is owed to a
// consumer, and there is no consumer until someone flushes.
const t = track<Record<string, unknown>>({ x: 0 });
expect(t.flush()).toEqual([["r", { x: 0 }]]);
expect(t.flush()).toEqual([]);
});
});
describe("apply and fan-out", () => {
it("adopts an `r` payload rather than copying it", () => {
// The consumer owns the batch it was handed. Copying every base batch
// defensively doubles the memory of the one op carrying the whole value.
//
// The rule that follows: do not hand one batch to two in-process
// consumers. This test pins the aliasing so nobody "fixes" it by adding a
// clone back.
const t = track<Record<string, unknown>>({ n: 0 });
const batch = t.flush();
const a = apply(undefined, batch) as Record<string, unknown>;
const b = apply(undefined, batch) as Record<string, unknown>;
a.n = 1;
expect(b.n).toBe(1);
});
it("does not alias the producer", () => {
const t = track<Record<string, unknown>>({ x: 0 });
const replica = apply(undefined, t.flush()) as Record<string, unknown>;
t.state.x = 999;
t.flush();
expect(replica.x).toBe(0);
});
it("folds a whole stream without a base-batch branch", () => {
// `apply` handles `r` by replacing, and tolerates an undefined target,
// so a consumer needs no isBase check and no clone of its own.
const t = track<Record<string, any>>({ x: 0, l: [] as string[] });
const enc = encoder();
const dec = decoder();
let replica: unknown;
const send = () => { replica = apply(replica, dec.decode(enc.encode(t.flush()))); };
t.state.x = 100;
t.state.l.push("xyz");
send();
t.state.x = 101;
send();
expect(replica).toEqual(t.target);
});
});
describe("dead-op elimination", () => {
// Ops apply in order, so an op is dead if a later one overwrites the whole
// subtree it lives in. Only s/d/r dominate; a/t/p modify what is there.
const run = (mutate: (s: any) => void, init: any = { a: { b: 1 }, x: 1, y: 2 }) => {
const t = track<any>(structuredClone(init));
t.flush(); // drain the base batch
mutate(t.state);
return t.flush();
};
it("collapses repeated writes to one field", () => {
expect(run((s) => { s.x = 1; s.x = 2; s.x = 3; })).toEqual([["s", ["x"], 3]]);
});
it("drops a write superseded by a later, non-adjacent one", () => {
expect(run((s) => { s.x = 10; s.y = 20; s.x = 30; }))
.toEqual([["s", ["y"], 20], ["s", ["x"], 30]]);
});
it("drops a child write when the parent is replaced after it", () => {
expect(run((s) => { s.a.b = 99; s.a = { c: 5 }; })).toEqual([["s", ["a"], { c: 5 }]]);
});
it("keeps a child write that follows the parent replacement", () => {
expect(run((s) => { s.a = { b: 1 }; s.a.b = 7; }))
.toEqual([["s", ["a"], { b: 1 }], ["s", ["a", "b"], 7]]);
});
it("collapses set-then-delete to the delete", () => {
expect(run((s) => { s.x = 5; delete s.x; })).toEqual([["d", ["x"]]]);
});
it("keeps delete-then-set, to preserve key position", () => {
// Dropping the delete would leave the key in its original slot while the
// producer moved it to the end. Same value, different serialisation.
expect(run((s) => { delete s.x; s.x = 5; })).toEqual([["d", ["x"]], ["s", ["x"], 5]]);
});
it("is linear in the number of ops", () => {
// The naive formulation compares every op against every dominator, which
// is quadratic and degrades on exactly the wide flush this pass cleans up.
const wide = (n: number) => {
const root: Record<string, number> = {};
for (let i = 0; i < n; i++) root[`f${i}`] = i;
const t = track(root);
t.flush(); // drain the base batch
const started = performance.now();
for (let i = 0; i < n; i++) t.state[`f${i}`] = i + 1;
t.flush();
return performance.now() - started;
};
wide(200); // warm
const small = Math.max(wide(250), 0.1);
const large = wide(2500);
expect(large / small).toBeLessThan(40); // linear would be ~10x
});
it("collapses a pathological redundant producer", () => {
const t = track<any>({ a: { b: 1 }, x: 1 });
t.flush(); // drain the base batch
for (let i = 0; i < 2000; i++) { t.state.x = i; t.state.a.b = i; }
t.state.a = { done: true };
expect(t.flush()).toHaveLength(2);
});
});
describe("flush", () => {
it("drops everything before a replacement", () => {
const t = track([1, 2]);
t.flush(); // drain the base batch
t.state.push(3);
t.state.splice(0, 3, 7);
expect(t.flush()).toEqual([["r", [7]]]);
});
it("keeps the prefix when the replacement is nested", () => {
// `s` on a subtree is not a root replacement, so earlier ops stay live.
const t = track({ a: 1, xs: [1, 2, 3], pad: PAD });
t.flush(); // drain the base batch
t.state.a = 2;
t.state.xs.splice(0, 3, 9);
expect(t.flush()).toEqual([["s", ["a"], 2], ["s", ["xs"], [9]]]);
});
it("rejects a constructor walk", () => {
// ({}).constructor.constructor is Function — the classic escape ladder.
expect(() => apply({}, JSON.parse('[["s",["constructor","prototype","gadget"],true]]'))).toThrow();
expect(({} as Record<string, unknown>).gadget).toBeUndefined();
});
it("rejects a forbidden path reached through an interned id", () => {
const dict = new Map<number, readonly (string | number)[]>([[0, ["__proto__", "w"]]]);
expect(() => apply({}, [["s", 0, true]], (id) => dict.get(id)!)).toThrow();
});
it("does not run an inherited setter", () => {
Object.defineProperty(Object.prototype, "trap", {
get() { throw new Error("inherited getter ran"); },
configurable: true,
});
try {
expect(() => apply({}, [["s", ["trap"], 1]])).not.toThrow();
} finally {
delete (Object.prototype as Record<string, unknown>).trap;
}
});
it("allows a reserved name as a VALUE key", () => {
// Reserved as segments, not as values.
const out = apply({}, [["s", ["a"], JSON.parse('{"__proto__":{"z":1}}')]]);
expect(({} as Record<string, unknown>).z).toBeUndefined();
expect(out).toBeDefined();
});
});
describe("safety: array indices", () => {
// An index may address an existing element or append exactly one past the end.
// Not an arbitrary cap: a sparse array does not survive a JSON round trip, so
// arr[7] = x on a length-3 array already produces unreplicable state.
it("writes an existing index", () => {
expect(apply({ xs: [1, 2, 3] }, [["s", ["xs", 1], 9]])).toEqual({ xs: [1, 9, 3] });
});
it("appends one past the end", () => {
expect(apply({ xs: [1, 2, 3] }, [["s", ["xs", 3], 9]])).toEqual({ xs: [1, 2, 3, 9] });
});
it("rejects a gap", () => {
expect(() => apply({ xs: [1, 2, 3] }, [["s", ["xs", 5], 9]])).toThrow();
});
it("rejects a huge index", () => {
// Would otherwise allocate 4.29 billion entries from one op.
expect(() => apply({ xs: [] }, [["s", ["xs", 4_294_967_290], 1]])).toThrow();
});
it("allows explicit growth with nulls", () => {
expect(apply({ xs: [1] }, [["p", ["xs"], 1, 0, [null, null, 9]]])).toEqual({ xs: [1, null, null, 9] });
});
});
describe("safety: op structure", () => {
// The CVE-2025-55182 lesson: a decoder must not trust tuple shape.
it("rejects an unknown verb rather than skipping it", () => {
// Silently skipping is how a newer producer's op vanishes and a replica drifts.
expect(() => apply({ a: 1 }, [["ZZZ", ["a"], 9] as never])).toThrow();
});
it("rejects non-array splice items", () => {
// Unvalidated, this spreads a string: ["n","o","t","-","a",…]
expect(() => apply({ xs: [1] }, [["p", ["xs"], 0, 0, "not-an-array"] as never])).toThrow();
});
it("rejects a string path", () => {
// Unvalidated, "a".slice(0,-1) is "", so it resolved to the ROOT and wrote.
expect(() => apply({ a: 1 }, [["s", "a", 9] as never])).toThrow();
});
it("rejects a non-tuple op", () => {
expect(() => apply({ a: 1 }, [{ op: "s" } as never])).toThrow();
expect(() => apply({ a: 1 }, [null as never])).toThrow();
});
it("clamps a splice remove past the end, as Array.prototype.splice does", () => {
// Deterministic and identical on both sides — not a hole.
expect(apply({ xs: [1, 2] }, [["p", ["xs"], 0, 1e9, []]])).toEqual({ xs: [] });
});
});
describe("safety: what a facet can put on the wire", () => {
// A facet cannot forge an op — it mutates plain objects and the tracker builds
// the tuples. Values and keys are the leak, and structuredClone is NOT a JSON
// check: it clones a Map happily, and the op then carries {} while the producer
// keeps a real Map.
const rejects = (name: string, mutate: (s: Record<string, unknown>) => void) =>
it(`rejects ${name}`, () => {
const t = track<Record<string, unknown>>({ pad: PAD });
t.flush(); // drain the base batch
expect(() => mutate(t.state)).toThrow();
});
rejects("a function", (s) => { s.f = () => {}; });
rejects("a Map", (s) => { s.m = new Map([[1, 2]]); });
rejects("a Date", (s) => { s.d = new Date(0); });
rejects("a BigInt", (s) => { s.b = 1n; });
rejects("a cycle", (s) => { const a: Record<string, unknown> = {}; a.self = a; s.c = a; });
rejects("a symbol key", (s) => { s[Symbol("k") as unknown as string] = 1; });
it("allows a getter on state, recording the computed result", () => {
const t = track<Record<string, unknown>>({ pad: PAD });
t.flush(); // drain the base batch
Object.defineProperty(t.target, "g", { get: () => 42, enumerable: true });
t.state.x = 1;
expect(t.flush()).toEqual([["s", ["x"], 1]]);
});
it("accepts a value that merely looks like an op", () => {
// Nested inside ["s", path, value]; nothing flattens, so it cannot be read
// as a top-level op.
const t = track<Record<string, unknown>>({ pad: PAD });
t.flush(); // drain the base batch
t.state.x = ["r", { evil: true }];
expect(t.flush()).toEqual([["s", ["x"], ["r", { evil: true }]]]);
});
});
describe("assertions", () => {
it("assertJsonValue rejects non-JSON", () => {
for (const bad of [new Map(), new Set(), new Date(), /re/, () => {}, Number.NaN, Infinity]) {
expect(() => assertJsonValue(bad)).toThrow();
}
for (const good of [null, true, 1, "s", [1, "a"], { a: { b: [null] } }]) {
expect(() => assertJsonValue(good)).not.toThrow();
}
});
it("assertValidOp accepts decoded ops and rejects wire forms", () => {
// Validating Op against the wire grammar would be laxer than the type: a
// two-element ["s", value] would pass, and apply would read the value as a
// path. Each vocabulary gets the validator that matches it.
for (const op of [
["r", { a: 1 }], ["s", ["a"], 1], ["d", ["a"]],
["a", ["a"], "x"], ["t", ["a"], 2], ["p", ["a"], 0, 0, []],
] as never[]) {
expect(() => assertValidOp(op)).not.toThrow();
}
for (const wireOnly of [["s", 1], ["d"], ["a", "x"], ["t", 2], ["p", 0, 0, []], ["#", 0, ["a"]], ["s", 0, 1]] as never[]) {
expect(() => assertValidOp(wireOnly)).toThrow();
expect(() => assertValidWireOp(wireOnly)).not.toThrow();
}
});
});
describe("codec: path interning and arity omission", () => {
// One pair per stream. The table spans a whole subscription or file, so a
// second consumer joining later needs its own encoder.
const roundTrip = (batches: Op[][]) => {
const enc = encoder();
const dec = decoder();
return batches.map((ops) => dec.decode(enc.encode(ops)));
};
it("round-trips a stream exactly", () => {
const t = track<Record<string, any>>({ a: { deep: "" }, b: { deep: "" } });
t.flush(); // drain the base batch
const batches: Op[][] = [];
for (let i = 0; i < 6; i++) {
t.state.a.deep += `x${i}`;
t.state.b.deep += `y${i}`;
batches.push(t.flush());
}
expect(roundTrip(batches)).toEqual(batches);
});
it("interns on second use, not first", () => {
const enc = encoder();
const p = ["a", "deep"] as const;
const first = enc.encode([["a", p, "1"]]);
const second = enc.encode([["a", p, "2"]]);
expect(first).toEqual([["a", p, "1"]]); // inline
expect(second).toEqual([["#", 0, p], ["a", 0, "2"]]); // define, then use
});
it("omits the path when it repeats", () => {
const enc = encoder();
const p = ["a"] as const;
expect(enc.encode([["s", p, 1], ["s", p, 2]])).toEqual([["s", p, 1], ["s", 2]]);
});
it("resets the table on a base batch, so recovery is self-contained", () => {
// A reader replays from the LAST base batch with a fresh decoder. Ids
// defined before it were never seen, so keeping them breaks recovery with
// an unresolvable path id.
const enc = encoder();
const p = ["a", "deep"] as const;
enc.encode([["a", p, "1"]]);
enc.encode([["a", p, "2"]]); // id 0 assigned
const base = enc.encode([["r", { a: { deep: "x" } }]]);
const after = enc.encode([["a", p, "3"]]);
expect(base).toEqual([["r", { a: { deep: "x" } }]]);
expect(after).toEqual([["a", p, "3"]]); // inline again
const dec = decoder();
expect(() => dec.decode(base)).not.toThrow();
expect(dec.decode(after)).toEqual([["a", p, "3"]]);
});
it("survives recovery from the last base batch", () => {
const enc = encoder();
const t = track<Record<string, any>>({ a: { deep: "" }, b: { deep: "" } });
t.flush(); // drain the base batch
const wire: WireOp[][] = [];
for (let i = 0; i < 8; i++) {
t.state.a.deep += `x${i}`;
t.state.b.deep += `y${i}`;
if (i === 5) t.rebase();
wire.push(enc.encode(t.flush()));
}
const lastBase = wire.map(isBase).lastIndexOf(true);
const dec = decoder();
let replica: unknown;
for (const w of wire.slice(lastBase)) {
const ops = dec.decode(w);
replica = replica === undefined
? structuredClone((ops[0] as ["r", unknown])[1])
: apply(replica, ops);
}
expect(replica).toEqual(t.target);
});
it("round-trips random streams", () => {
for (let round = 0; round < 300; round++) {
const t = track<Record<string, any>>({ a: { p: "", q: "" }, b: [] as unknown[], c: 0 });
t.flush(); // drain the base batch
const batches: Op[][] = [];
for (let i = 0; i < 8; i++) {
const r = Math.random();
if (r < 0.3) t.state.a.p += "x";
else if (r < 0.5) t.state.a.q += "y";
else if (r < 0.65) t.state.b.push(i);
else if (r < 0.8) t.state.c = i;
else if (r < 0.9) delete t.state.c;
else t.rebase();
const ops = t.flush();
if (ops.length > 0) batches.push(ops);
}
expect(roundTrip(batches)).toEqual(batches);
}
});
});
describe("property: random round-trip", () => {
// 3000 random sequences found the root-splice hole on the first run, before any
// hand-written case did.
const rnd = (d = 0): unknown => {
const r = Math.random();
if (d > 2 || r < 0.3) return Math.floor(Math.random() * 5);
if (r < 0.45) return ["x", "y", null, true][Math.floor(Math.random() * 4)];
if (r < 0.7) return Array.from({ length: Math.floor(Math.random() * 4) }, () => rnd(d + 1));
const o: Record<string, unknown> = {};
for (const k of ["a", "b", "c"]) if (Math.random() < 0.6) o[k] = rnd(d + 1);
return o;
};
it("a replica always matches the producer", () => {
let checked = 0;
for (let i = 0; i < 3000; i++) {
const base = rnd();
if (typeof base !== "object" || base === null) continue;
const t = track(structuredClone(base) as object);
t.flush(); // drain the base batch
const next = rnd();
if (Array.isArray(t.state) && Array.isArray(next)) {
(t.state as unknown[]).splice(0, (t.state as unknown[]).length, ...next);
} else if (!Array.isArray(t.state) && typeof next === "object" && next !== null && !Array.isArray(next)) {
const s = t.state as Record<string, unknown>;
for (const k of Object.keys(s)) if (!(k in next)) delete s[k];
for (const [k, v] of Object.entries(next)) s[k] = v;
} else continue;
expect(apply(structuredClone(base), t.flush())).toEqual(t.target);
checked++;
}
// Most random pairs are shape-incompatible and skipped; this only guards
// against the loop silently checking nothing.
expect(checked).toBeGreaterThan(200);
});
});

View file

@ -0,0 +1,285 @@
// ─────────────────────────────────────────────────────────────────────────────
// harness/delta — intent-recording change tracking over plain JSON.
//
// Depends on nothing else in the harness. Session storage, the runtime and the
// facet host consume it; keep the arrows pointing that way.
//
// State is plain TS. You mutate it normally. Ops come out carrying what you
// *did*, not what changed — `text += chunk` yields an append, not a 50 KB
// replacement; `arr.unshift(x)` yields one splice, not O(n) index writes.
// ─────────────────────────────────────────────────────────────────────────────
export type JsonValue = null | boolean | number | string | JsonValue[] | { [k: string]: JsonValue };
export type Seg = string | number;
export type Path = readonly Seg[];
/** Non-empty: `s`/`d`/`a`/`t` cannot target the root. Enforced by the type. */
export type NonEmptyPath = readonly [Seg, ...Seg[]];
/** A path inline on first use, an interned id thereafter. */
export type PathRef<P extends Path = Path> = P | number;
/**
* Tuples are the form — in memory, on the wire, on disk. There is no keyed
* variant and no codec between them.
*
* `r` is the ONLY op that replaces a whole value. It is an op rather than a
* frame kind for the same reason `a` and `t` exist: they are specialisations of
* `s` that are smaller and state intent. It also makes base detection a token
* comparison rather than a path inspection, which matters because recovery stops
* at the last base batch — a correctness boundary.
*
* Only `p` may target the root, and only because a tracked value can itself be
* an array. A `p` covering its entire target is normalised at RECORD time to `r`
* or `s`, so a root `p` is always a partial modification.
*
* The two-element forms reuse the previous op's path; arity disambiguates.
*/
export type Op =
| readonly ["r", JsonValue]
| readonly ["s", NonEmptyPath, JsonValue]
| readonly ["d", NonEmptyPath]
| readonly ["a", NonEmptyPath, string]
| readonly ["t", NonEmptyPath, number]
| readonly ["p", Path, number, number, JsonValue[]];
/**
* What crosses a boundary. Adds path interning and arity omission, and nothing
* else. `["r", value]` carries no path, so it encodes to itself — which is why
* `isBase` works on either vocabulary.
*/
export type WireOp =
| readonly ["r", JsonValue]
| readonly ["s", PathRef<NonEmptyPath>, JsonValue] | readonly ["s", JsonValue]
| readonly ["d", PathRef<NonEmptyPath>] | readonly ["d"]
| readonly ["a", PathRef<NonEmptyPath>, string] | readonly ["a", string]
| readonly ["t", PathRef<NonEmptyPath>, number] | readonly ["t", number]
| readonly ["p", PathRef, number, number, JsonValue[]]
| readonly ["p", number, number, JsonValue[]]
| readonly ["#", number, Path];
/**
* There is no frame type. What travels is `Op[]`.
*
* A **base batch** is one whose first op is `r`; the first batch of a stream is
* always one, and storage tags it "base" so recovery can stop there. Sequencing
* lives on the transport — a list element's `seq`, or an SSE `id:` — never in the
* payload. See `delta.md` §6.
*/
// ─── Safety ──────────────────────────────────────────────────────────────────
/**
* Path segments that reach the prototype chain. Rejected at record time AND at
* apply time, including through an interned id.
*
* Reserved as SEGMENTS, not as values: an object with a literal `__proto__` key
* replicates fine as a whole value; only a path walking through it is refused.
* That is a real restriction on what is mutable — see `delta.md` §7.1.
*/
export declare const RESERVED_SEGMENTS: ReadonlySet<string>;
export declare class UnsafePathError extends Error {
readonly segment: Seg;
}
/** Rejects reserved segments, symbol keys, and negative or non-integer indices. */
export declare function assertSafePath(path: Path): void;
/**
* Rejects anything `JSON.stringify` would not round-trip: Map, Set, Date, RegExp,
* typed arrays, class instances, functions, BigInt, cycles.
*
* `structuredClone` is NOT a JSON check — it clones a Map happily, and the op then
* carries `{}` while the producer keeps a real Map. Silent divergence.
*/
export declare function assertJsonValue(value: unknown): asserts value is JsonValue;
/**
* Validate an op's structure before applying it. A decoder must not trust tuple
* shape: an unvalidated applier spreads a string into an array given
* `["p",["xs"],0,0,"not-an-array"]`, and writes at the root given a string path.
*
* An unknown verb is an error, not a no-op — skipping it is how a newer producer's
* op vanishes and a replica drifts. This is the CVE-2025-55182 lesson
* (`delta.md` §7.3).
*/
export declare function assertValidOp(op: unknown): asserts op is Op;
/**
* The same, for the wire grammar, where ids and short forms are legal.
* `decode` uses this; `apply` uses `assertValidOp`. Validating a decoded op
* against the wire grammar is laxer than its type — a two-element `["s", value]`
* would pass and then be read as a path.
*/
export declare function assertValidWireOp(op: unknown): asserts op is WireOp;
// ─── Classification ─────────────────────────────────────────────────────────
//
// Ops are tuples, so classification is easy to hand-roll and easy to get wrong.
// `op[0] === "s"` is not enough — a set on any path is "s"; what makes a base is
// that it targets the ROOT. And because a tuple may omit the PathRef (arity
// disambiguates), `op[1]` is a PathRef or a value depending on length. These
// exist so nobody writes that comparison twice.
/** True for the `r` op. */
export declare function isReplace(op: Op): boolean;
/**
* True when this batch begins with `r` — a **base batch**. Exact rather than a
* heuristic, because flush guarantees `r` appears at index 0 or not at all.
* Storage tags such elements `"base"` so recovery can stop at the last one.
*/
export declare function isBase(ops: readonly (Op | WireOp)[]): boolean;
// ─── Codec ───────────────────────────────────────────────────────────────────
/**
* Path interning and arity omission. ONE PAIR PER STREAM — the id table spans a
* whole subscription or file, so a consumer joining later needs its own.
*
* The table resets on a base batch, because a reader replays from the last one
* with a fresh decoder and everything after it must be self-contained.
*/
export declare function encoder(): { encode(ops: readonly Op[]): WireOp[] };
export declare function decoder(): { decode(wire: readonly WireOp[]): Op[] };
// ─── Tracker ─────────────────────────────────────────────────────────────────
export interface TrackerOptions {
/**
* Cap on characters compared when deriving append/truncate from a string
* write. A window slide's overlap lies near the end, so a cap bounds cost
* without changing the result. Default 65536; 0 disables detection.
*/
maxOverlapScan?: number;
/** Merge compatible adjacent ops on flush. Default true. */
coalesce?: boolean;
}
export interface Tracker<T extends object> {
/**
* Mutate this. Plain TS: assignment, push, splice, delete, all of it.
*
* Assigning to it replaces the whole value: emits `["r", next]` and discards
* ops recorded before it. It must be a SETTER on the tracker — a plain
* property would be overwritten, swapping the proxy for a plain object and
* silently ending tracking.
*
* To force a base batch without changing the value, call `rebase()`.
*/
state: T;
/** The untracked backing object. Reads only — writing here emits nothing. */
readonly target: T;
/**
* Drain. Empty when nothing changed.
*
* The FIRST flush is always a base batch, carrying the value including any
* mutations made before it — a consumer starts with nothing, so a stream must
* open with something to apply deltas to.
*
* Ops that a later op makes unreachable are dropped first — a field written
* three times emits one op, and a parent replaced after its child emits one.
* Object key order is not preserved by that pass; see `delta.md` §5.
*/
flush(): Op[];
/** Discard without emitting. For a producer that just sent a base batch. */
discard(): void;
get dirty(): boolean;
}
export declare function track<T extends object>(root: T, options?: TrackerOptions): Tracker<T>;
// ─── Applier ─────────────────────────────────────────────────────────────────
/**
* Apply ops to a plain mutable object. Five verbs plus the dictionary entry, no
* domain knowledge, no library, no registry lookup.
*
* Returns the value, because a root set replaces it outright and cannot be done
* in place. Throws `PathError` when a path does not resolve, which a consumer
* treats exactly like a sequence gap: request a fresh base frame.
*
* Must not be pointed at a value produced by Immer, which deep-freezes.
*/
/**
* Fold ops into a value and return it — returned rather than mutated in place,
* because `r` replaces the value outright. Tolerates an `undefined` target,
* since a root op never reads it, so a consumer needs no base-batch branch.
*
* An `r` payload is **adopted, not copied**: the consumer owns the batch it was
* handed. Do not hand one batch to two in-process consumers, or their replicas
* will alias.
*/
export declare function apply<T>(target: T, ops: readonly Op[]): T;
export declare class PathError extends Error {
readonly path: Path;
}
// ─── Replica ─────────────────────────────────────────────────────────────────
export interface Replica<T> {
/** Current value. `undefined` until the first base frame. */
readonly value: T | undefined;
/**
* Feed one batch, with the sequence number the transport supplied. Returns
* "gap" when `seq` is not contiguous or a path failed to resolve; the caller
* then requests a fresh base batch.
*/
receive(ops: readonly Op[], seq: number): "ok" | "gap";
}
export declare function replica<T>(): Replica<T>;
// ─── Semantics ───────────────────────────────────────────────────────────────
//
// ROOT OPS
// `["s", [], v]` replaces the value; `["p", [], …]` splices it when the value
// is an array. The applier must special-case `path.length === 0` before
// resolving a parent — a prototype handling only a root set threw on the first
// randomised sequence that produced a root splice.
//
// STRINGS
// A write of `next` over `prev` derives ops from the longest suffix of `prev`
// that is a prefix of `next`:
// overlap === prev.length -> append(next.slice(overlap))
// overlap > 0 -> truncate(prev.length - overlap) + append(rest)
// overlap === 0 -> set(next)
// Always correct: the overlap is verified, not guessed. This is what makes a
// rolling window cheap — the case every effect-recording library degrades on.
//
// Use a native probe, not a hand-written KMP: `indexOf` a short head of `next`
// in `prev`, verify with `endsWith`. Measured 47x faster over 2000 slides on a
// 50 KB string, verified identical across 200.
//
// `t` counts UTF-16 code units, matching String.prototype.slice. Not bytes.
//
// ARRAYS
// Mutator methods are intercepted in the `get` trap, so intent is recorded
// before the engine performs index writes:
// push/unshift/pop/shift/splice -> one `p`
// sort/reverse/fill/copyWithin -> `s` of the whole array (rare)
// Direct index writes are honest `s` ops. `a.length = n` is translated —
// shrink to a splice, grow to a splice of nulls — never emitted as a path,
// because `length` is not a document location.
//
// TRANSACTIONS
// Several writes to one address in one commit must each be recorded against
// the previous one's result, not against pre-transaction state. The applier
// replays in order; anything else applies the second against a stale base.
//
// ALIASING
// Op payloads are structured-cloned at record time. Without this a later
// mutation retroactively changes an already-emitted op and a local replay
// aliases the source — an observed bug, not a hypothetical.
//
// UNDEFINED
// `x = undefined` normalises to `d`. JSON has no undefined, and a set with a
// missing value is indistinguishable from a lost value after a round trip.
//
// SCOPE
// JsonValue only. No Map, Set, Date, RegExp, class instances, getters, symbol
// keys, or cycles. That keeps this to five traps and one string algorithm, and
// makes structuredClone on payloads always safe.

View file

@ -0,0 +1,604 @@
# Scoped storage Step 1 — actionable implementation handoff
**Status: actionable, not implemented.**
**Prepared against:** `c4b0e35ab` (`dev`). Re-audit named source files if the implementation baseline changes materially.
This handoff implements storage scopes only. **Read this file before [`scopes.md`](scopes.md).** It supersedes conflicting details there, especially per-scope sequence spaces, operation-derived orphan detection, one-use scope IDs, raw-string scope arguments, sidecars ending in `.jsonl`, the proposed interned retirement tuple, and bundling JSONL delta/address encoding into the same change. The broader motivation and measurements in `scopes.md` remain design evidence; its signatures and implementation order are not current where this file differs.
Step 2—Chord-encoded JSONL values, address interning, compact tuple records, and their measurements—is a separate package. Do not begin Step 2 until Step 1 has been tested, reviewed, explicitly approved by the user, committed, and pushed.
## 0. Delivery protocol
This package has a mandatory user checkpoint:
1. Implement Step 1 only.
2. Run every required focused test, `npm run check`, and `./test.sh`.
3. Review the complete diff and report results. Delegated reviews, if used, run with provider `anthropic` and model `claude-fable-5`.
4. **Stop and wait for explicit user approval.**
5. After approval, commit only this package's files and push that commit.
6. Confirm the pushed commit before designing or implementing Step 2.
Do not combine the Step 1 and Step 2 commits. Do not commit or push before approval.
## 1. Mandatory reading
Read completely before editing:
1. `packages/agent/docs/harness.md` §§0.2–0.9, 1.1–1.7, 2.7–2.8, 3.7–3.8, 3.13, 4.4–4.8, 5.4, Part 7, Part 9.
2. `packages/agent/docs/values.md`.
3. `packages/agent/docs/mobile-handoff/README.md`.
4. [`scopes.md`](scopes.md), using this handoff where they conflict.
5. `../01-delta/delta.md` for vocabulary ownership only; Step 1 does not store Chord ops.
6. `../04-tool-output/harness-tools.md` §§7.1–7.7 for later-consumer context; do not implement its output redesign here.
7. `../05-assistant-output/message-update.md` §7 for later-consumer context; do not implement it here.
8. `packages/agent/src/harness/session/{types,values,commit,in-memory-storage-state,memory,session,index}.ts`.
9. Every file under `packages/agent/src/harness/session/jsonl/` and the `FileSystem` declaration/implementation in `packages/agent/src/harness/{types,env/nodejs}.ts`.
10. `packages/agent/src/harness/runtime/{progress,lane}.ts`, `runtime/drive/{terminal,response,deferred,tools,reconcile,structural,boundary}.ts`, and every caller of `operationCleanupWrites`. Inspect `tool-placement.ts` through the grep audit, but current source has no scoped delete there.
11. `packages/session-backends/sqlite-node/src/{index,sqlite/index}.ts`, `sqlite/{storage,repo}.ts`, `migrations/001_initial.sql`, and `session/{values,session-sequences}.ts`.
12. Shared storage/repository conformance and every focused test named in §9.
13. `scopes.variance.ts` for the covariant-address/invariant-write phantom mechanism only. Its current no-ID `EphemeralScope` and `retireScope(id: string)` signatures are superseded by §2 and must be updated in Slice A.
Do not use `dist/` as implementation input. Format 4 remains WIP; no R11 migration is part of this package.
## 2. Fixed scope contract
### 2.1 Runtime scope identity
A scope ID is a reusable logical lifetime name, not a globally one-use token:
```ts
export type SessionScope = { readonly kind: "session" };
export interface EphemeralScope {
readonly kind: "ephemeral";
readonly id: string;
}
export type Scope = SessionScope | EphemeralScope;
export function ephemeralScope(id: string): EphemeralScope;
```
`ephemeralScope(id)` requires a non-empty well-formed Unicode string and returns a frozen value. Equal IDs identify the same physical scope; object identity has no meaning. JSONL encodes IDs with `encodeURIComponent`. The encoded component must be at most 180 ASCII characters; reject longer IDs and lone-surrogate input. The physical filename always has a fixed `scope-` prefix, so encoded `.`/`..` values cannot become path segments. Do not directly interpolate unencoded caller input into a path.
No create/open-scope transaction exists. The first scoped value/list write creates physical state. A later `retireScope(scope)` ends everything in that scope through its assigned global sequence. A write under the same ID after retirement begins a new logical lifetime.
Harness operation scopes use `ephemeralScope(operationId)`. Generic application scopes may use stable names. Storage never parses a scope ID as an operation ID and never reads Harness operation state to decide lifetime.
### 2.2 Reuse and retirement boundary
Global sequence order defines reusable generations:
```text
seq 10 scoped write S
seq 20 scoped write S
seq 30 retireScope(S)
seq 40 scoped write S # new lifetime
```
For a given scope ID, only scoped records with `seq` greater than its latest committed retirement sequence are live. Memory and SQLite realize this by deleting current rows/maps at retirement and allowing later writes to recreate them. JSONL uses the retirement sequence as its replay boundary (§6).
Retirement is idempotent in the storage sense: retiring a scope with no current state is legal and advances the boundary. A later write still begins a new lifetime. Storage does not maintain an all-time used-ID registry or reject reuse.
A scoped write reaching the commit line after retirement is therefore a new-lifetime write. Harness invocation fencing and settlement drains must continue to ensure a late assistant/tool progress job cannot recreate operation output after terminal retirement. Add explicit tests for this boundary.
### 2.3 Explicit retirement is the sole authority
```ts
export function retireScope(scope: EphemeralScope): Write<SessionScope>;
```
Storage never infers orphanage from operation absence, namespace, current state, or a missing owner. The only valid lifetime boundary is a committed `retireScope` write. If an owner never retires its scope, that is an owner defect and the scope remains live.
Crash behavior:
- crash before the retirement transaction commits: the scope remains live and reopens;
- crash after retirement commits but before JSONL unlink: replay sees the retirement boundary, ignores pre-boundary sidecar records, and retries physical deletion;
- crash during a later reused lifetime: records after the latest retirement reopen normally.
Repository deletion separately removes every physical sidecar belonging to the deleted Session. It does not need operation semantics.
### 2.4 Address and write typing
Addresses carry a covariant scope tag; writes carry an invariant scope tag. Preserve value-type invariance independently:
```ts
interface Value<T, Sc extends Scope = SessionScope> { /* existing fields + scope */ }
interface ValueList<T, Sc extends Scope = SessionScope> { /* existing fields + scope */ }
declare function value<T>(namespace: string, key?: string): Value<T, SessionScope>;
declare function value<T>(namespace: string, key: string, scope: EphemeralScope): Value<T, EphemeralScope>;
declare function list<T>(namespace: string, key?: string): ValueList<T, SessionScope>;
declare function list<T>(namespace: string, key: string, scope: EphemeralScope): ValueList<T, EphemeralScope>;
```
A session-scoped address has no runtime scope ID. An ephemeral address carries its scope ID. Scope is part of physical identity: equal namespace/key addresses in session scope and ephemeral scope, or in two different ephemeral IDs, are distinct.
`setValue`, `deleteValue`, `appendList`, and `deleteList` preserve the address scope in `Write<Sc>`. Entry and usage writes are session-scoped. `retireScope` is session-scoped even though applying it deletes ephemeral state.
All writes in an ordinary transaction have one static scope. A session transaction may include session values, entries, usage, and one or more `retireScope` writes. It may not include a direct ephemeral set/delete/append. An ephemeral transaction may include value/list writes only. At runtime, every ephemeral write in one transaction must carry the same scope ID; two different IDs have the same TypeScript tag and require this assertion.
Make the minimum generic changes necessary to preserve this through `Write`, `CommittedWrite`, `Storage.commit`, Session mutation capabilities, `CommitDecision`, and lane commands. Follow the variance proof in `scopes.variance.ts`; do not make readers invariant or propagate unnecessary scope type parameters through read-only APIs.
### 2.5 One global sequence space
Memory, JSONL main records, JSONL sidecar records, and SQLite rows share one Session-global sequence space. Every admitted commit remains serialized and receives increasing sequences in admission order. Gaps remain legal.
- Memory continues using one `nextSeq`.
- SQLite continues allocating from `sessions.next_seq` inside the write transaction.
- JSONL main and sidecar appends share one commit queue and one resident `nextSeq`.
JSONL open computes the high-water mark from the main header and every complete record in every matching sidecar **before** deleting or ignoring retired physical files. A retired sidecar's sequences may then become gaps but can never be reused. Torn uncommitted final transactions do not advance the durable high-water mark.
No reservation record, per-scope sequence counter, or range allocator is permitted.
## 3. List tags
Step 1 adds only the mechanism required by later tracked-output recovery:
```ts
export interface ListElement<T> {
seq: number;
value: T;
tag?: string;
}
export interface ListReadOptions {
cursor?: ListCursor;
order?: "asc" | "desc";
limit?: number;
stopAtTag?: string;
}
export function appendList<T, Sc extends Scope>(
address: ValueList<T, Sc>,
element: NoInfer<T>,
tag?: string,
): ListAppendWrite<Sc>;
```
Tags are non-empty strings when present. Storage stores and returns the tag without interpreting the element.
Read semantics:
1. Apply address scope, exclusive cursor, and order.
2. Inspect at most the normalized `limit` elements.
3. Stop at and include the first element carrying `stopAtTag`.
4. Return that possibly shortened page.
`stopAtTag` never searches beyond the page limit. If absent from the page, the caller pages again from the last returned sequence. Do **not** add the draft `tag` filter; no current consumer needs it and combining filtering with the existing cursor shape is ambiguous.
SQLite may fetch the indexed `limit` rows and truncate at the first returned tag in TypeScript; storage still never parses payloads. Preserve the existing primary-key query plan with no temporary sort.
## 4. Step 1 durable payloads remain unchanged
This package changes lifetime and routing, not output representation:
- `pendingToolOutput` remains `Value<AgentToolResult<unknown>, EphemeralScope>`;
- `pendingAssistantFrames` remains `ValueList<AssistantMessageFrame, EphemeralScope>`;
- `operationToolMemo` remains a `JsonValue` scalar;
- JSONL transaction records remain readable keyed objects;
- assistant progress still appends one frame per accepted frame;
- tool checkpoints still replace whole snapshots;
- `message_update` and `tool_update` event shapes do not change;
- safe replay retains its current checkpoint behavior, including the known delete-before-replay bug owned by the later tool-output package.
Do not rename `pendingAssistantFrames`, introduce `pendingAssistantOutput`, store `WireOp[]`, add Chord trackers/codecs, redesign `AgentHarnessTool`, or change output cadence here.
## 5. Runtime ownership and cleanup
### 5.1 Addresses moved to the operation scope
Construct these addresses with `ephemeralScope(operationId)`:
- `operationToolMemo(operationId, invocationId, name)`;
- `pendingToolOutput(operationId, invocationId)`;
- `pendingAssistantFrames(operationId, responseEntryId)`.
All other current built-ins remain session-scoped. In particular, `pi.op.state`, `pi.op.meta`, `pi.op.tool_args`, `pi.op.preparation`, and `pi.pending.entry` remain in the main scope because their writes coordinate atomically with lane/operation state.
### 5.2 No cross-file cleanup transaction
Remove direct ephemeral deletes from transactions that also write session state:
- assistant response settlement;
- deferred-response supersession;
- tool outcome staging in `drive/tools.ts`;
- cancellation/recovery decisions;
- terminal per-address cleanup.
Current `drive/tool-placement.ts` has no ephemeral delete; do not invent a change there.
`operationCleanupWrites` stops scanning tool memos and tool outputs and stops constructing a state-directed assistant-frame delete. It keeps session-scoped operation/meta/args/preparation/pending-entry cleanup and adds exactly one `retireScope(ephemeralScope(operationId))` in the universal terminal suffix.
After removing the current staging/terminal scans, remove `operationToolMemoPrefix` and `pendingToolOutputPrefix`; current source has no other production consumers. Update the normative exported prefix-constructor count from five to three and update exact constructor tests rather than retaining dead inventory APIs.
An ephemeral-only delete remains legal. For example, the current safe-replay checkpoint delete remains because its transaction contains no session write; fixing its behavior belongs to the later tool-output package, not scoped storage or JSONL optimization.
### 5.3 Unreachable scoped residue
After a response settles, a deferred response is superseded, or a tool reaches `outcome_ready`, its former frames/checkpoints/memos may remain physically and logically addressable inside the still-active operation scope. Current scalar operation state no longer references them; no recovery or snapshot path scans the scope to infer authority. The terminal retirement deletes the whole scope.
Call this **unreachable scoped residue**, not orphanage. Update `harness.md` invariants that currently require an `outcome_ready` call to have no physically present memo/checkpoint. The replacement invariant is that settled child state never consumes or exposes residue and terminal retirement removes all scoped state atomically with operation completion.
Close and fault are controlled crashes and never retire the scope. Reopen restores scoped progress only through addresses derived from current authoritative operation state.
## 6. JSONL sidecars
### 6.1 Physical layout
The main session file remains unchanged except for committed scope-retirement records. Every main file owns one known sidecar directory at `${mainPath}.scopes`; this requires no dirname API or `FileSystem` extension. Create it lazily with `createDir(..., { recursive: true })`. Discover scopes with `listDir` on that exact path. A sidecar filename is exactly `scope-${encodeURIComponent(scope.id)}.scope`, subject to §2.1's 180-character encoded-component limit. The suffix does not end in `.jsonl`, and the sidecar directory itself is a directory, so repository listing never mistakes either for a Session.
A sidecar begins with this exact header:
```ts
interface JsonlScopeHeader {
v: 4;
kind: "scope_header";
sessionId: string;
scopeId: string;
storageVersion: 1;
}
```
Validate the exact session/scope identity before replay. Create the first header through a temp file inside `${mainPath}.scopes` plus atomic rename. Temporary filenames end in `.tmp` and are never discovered as sidecars. Use the existing `FileSystem` capability; do not add a dirname method, extend `JsonlStorageOptions`, or add Node-only filesystem access to agent core.
Step 1 sidecar writes retain the current object record spelling and carry/validate their scope ID in committed shapes. Each physical transaction remains one complete line or one array line. Sidecar transactions may contain only value/list writes for that exact scope.
### 6.2 Retirement record
The main log records an explicit committed write:
```ts
interface CommittedScopeRetireWrite {
kind: "scope";
op: "retire";
seq: number;
scopeId: string;
}
```
Its Step 1 JSON representation is the corresponding readable object. Do not use the draft `['!', addrId, seq]`: a scope is not an interned value/list address. Compact tuple spelling belongs to Step 2 and must be designed there.
Applying a committed retirement removes resident values/lists in that scope. After the complete main transaction is durable, JSONL synchronously serializes lifecycle cleanup on the same commit queue: close any owned sidecar resource, attempt unlink, then release the queue. Failure to unlink is non-fatal to the already-committed business transaction; the sidecar remains logically bounded by the retirement sequence and deletion is retried on reopen.
### 6.3 Replay and reusable IDs
Open performs these logical stages before admitting writes:
1. Read/repair the main file and parse complete main transactions.
2. Discover and validate matching sidecars; ignore unrelated files.
3. Parse complete sidecar transactions and repair torn final transactions.
4. Compute the global sequence high-water mark from the header and **all** complete main/sidecar records, including records later excluded by retirement.
5. Determine the latest retirement sequence per scope from main transactions.
6. Keep sidecar writes whose sequence is greater than that scope's latest retirement.
7. Merge retained main and sidecar transactions by global sequence and replay them in global order, preserving transaction boundaries and rejecting duplicate/non-monotonic sequences.
8. Advance to the precomputed high-water mark.
9. Remove sidecars with no records after their latest retirement; cleanup failure does not make retired contents live.
Main transactions have sequence gaps where sidecar transactions occurred. Sidecars have gaps where other scopes/main commits occurred. Both remain valid.
A sidecar containing records after a retirement is an active reused lifetime and must not be deleted. Serialize retirement cleanup against later writes so an old unlink cannot race and remove a newly reused sidecar.
Malformed interior records or identity mismatch in an active sidecar are storage corruption and fail open. A torn final transaction is discarded wholly. Temporary files left by interrupted atomic creation are cleanup artifacts and never become sidecars.
### 6.4 Legacy v3
Legacy v3 has no scopes. An ordinary Harness operation commits session-scoped acceptance before any progress write, so it upgrades first in the normal path. Still define and test the generic storage behavior when the first caller write is ephemeral: complete the existing v3-to-v4 main rewrite/usage adjustment under the global commit queue, then commit the scoped transaction without placing it in the main file. A crash before the sidecar append leaves no committed caller transaction and may reuse its uncommitted sequence.
### 6.5 Repository and fork behavior
- Repository `list()` ignores `${mainPath}.scopes` directories and their files through its existing non-file/`.jsonl` filtering.
- Repository `delete()` removes the main file through its current path, then removes `${mainPath}.scopes` recursively with `force: true`; a scope-directory cleanup failure rejects deletion after the main-file result is known, without inferring Harness state. Missing scope directories are legal.
- `JsonlStorage.close()` drains its admitted main/sidecar commits as today; `JsonlSessionRepo.close()` retains its current lifecycle behavior because repository ownership is a separate package. No progress scope is retired merely because a handle closes.
- Preserve the current fork allowlist exactly. Do not add generic application values or lists to either fork scope. Ensure source snapshots and backend fork readers ignore all ephemeral values/lists regardless of namespace; current destinations remain logically unchanged except that ephemeral state cannot leak.
- Fork destinations contain no sidecars or retirement state.
- WP08 may later replace fork materialization but must preserve this scope policy.
J1 compaction is excluded. When J1 eventually lands, it may omit a historical retirement record only after no physical sidecar can be made live by losing that boundary.
## 7. Memory backend
Extend `InMemoryStorageState` physical value/list identity with scope. Session scope and each ephemeral ID are distinct. Preserve one global `nextSeq` and current stats behavior.
Applying `CommittedScopeRetireWrite` deletes every scalar/list in that scope in the same synchronous application of the surrounding transaction. It does not delete entries or usage, which cannot be ephemeral. A later scoped write recreates state under the same ID.
Snapshots expose scope so fork code can reject ephemeral state, but `createForkSnapshot` keeps its current built-in allowlist and must not start copying generic session-scoped application values or lists. Instrumentation must still report exact committed write order, including retirement.
## 8. SQLite backend
Change WIP schema in place with no migration:
```sql
scalar_values(
session_id TEXT NOT NULL,
scope_id TEXT NOT NULL,
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY(session_id, scope_id, namespace, key)
) WITHOUT ROWID;
list_values(
session_id TEXT NOT NULL,
scope_id TEXT NOT NULL,
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
tag TEXT,
PRIMARY KEY(session_id, scope_id, namespace, key, seq)
) WITHOUT ROWID;
```
Use `scope_id = ''` for session scope and the exact scope ID otherwise. Every point read, prefix scan, set/delete, append/delete, snapshot, and query-plan assertion includes scope identity.
Inside the same `BEGIN IMMEDIATE` transaction, a retirement write executes:
```sql
DELETE FROM scalar_values WHERE session_id = ? AND scope_id = ?;
DELETE FROM list_values WHERE session_id = ? AND scope_id = ?;
```
It still consumes its assigned global sequence through `sessions.next_seq`, even though no retired-scope tombstone row remains. A later write under that ID recreates rows naturally. Retirement rollback must restore both scoped rows and all sibling main writes.
Fork paths copy only `scope_id = ''`. Shared-container deletion remains Session-scoped and removes all values/lists regardless of scope along with the other Session rows.
## 9. Required tests
Port tests before or with each implementation slice. Use the repository Vitest binary from the owning package. Do not run the full Vitest suite directly.
### 9.1 Type and constructor tests
- Session addresses default to `SessionScope`.
- Ephemeral overloads retain `EphemeralScope` while preserving invariant `T`.
- Address scope is covariant for reads; write scope is invariant.
- Same-scope session transactions typecheck.
- Same-scope ephemeral transactions typecheck.
- Session writes plus `retireScope` typecheck.
- Direct session + ephemeral writes in one transaction fail with `@ts-expect-error`.
- Two ephemeral IDs evade static distinction but fail the runtime same-ID assertion.
- Scope object identity is irrelevant; equal IDs access the same state.
- Empty and overlong encoded IDs reject; separators/Unicode encode without path escape.
- Session and ephemeral addresses with equal namespace/key do not alias.
### 9.2 Shared backend conformance
Run identically on Memory, JSONL, and SQLite:
- ephemeral scalar set/get/replace/delete;
- ephemeral list append/page/whole-list-delete;
- independent scopes with equal namespace/key;
- session scope independent from equal ephemeral address;
- mixed session/ephemeral transaction rejection before mutation;
- two ephemeral IDs in one transaction rejection before mutation;
- retirement deletes every value/list in the exact scope and no other scope;
- retirement atomic with session sibling writes;
- retirement of absent state is legal;
- write after retirement recreates a new lifetime;
- repeated retirement advances the boundary without making later writes disappear;
- main, scope A, scope B, retirement, and reused-A commits receive globally increasing sequences in admission order;
- retired/deleted physical elements leave legal gaps and first post-reopen commit allocates above every complete prior record;
- list tags round-trip, page limits remain bounded, and ascending/descending `stopAtTag` includes the marker;
- stats ignore scoped values/lists and retirement;
- close drains admitted scoped commits and rejects later reads/writes.
### 9.3 JSONL focused tests
- first scoped write creates exactly one valid sidecar and no main value/list record;
- several writes in one scoped transaction remain one physical sidecar line;
- main and multiple sidecars replay in global sequence order;
- header `nextSeq` and all complete sidecar records contribute to reopen high-water;
- torn sidecar final transaction is discarded wholly and repaired;
- malformed interior active sidecar and header identity mismatch fail open;
- crash before retirement leaves scoped data live;
- crash after main retirement but before unlink leaves data logically absent and cleanup retries;
- unlink failure does not reject the already-committed terminal transaction;
- reuse after retirement replays only records after the latest retirement sequence;
- multiple retirement/reuse cycles choose the latest boundary;
- retirement cleanup is serialized so a later reuse cannot be deleted by an earlier unlink;
- ignored pre-boundary records still prevent sequence reuse;
- sidecars and temp files never appear in repository listing;
- repository deletion removes all sidecars;
- forks exclude generic application-owned ephemeral values/lists;
- legacy-v3 first session write and first scoped-write paths both produce valid v4 state;
- close/reopen preserves active operation progress and does not retire it.
### 9.4 SQLite focused tests
- schema and query helpers include `scope_id`; list rows preserve nullable tags;
- one transaction atomically deletes scoped scalar/list rows and writes terminal main state;
- forced failure after scope deletion rolls back deletion and sibling writes;
- reuse after retirement recreates rows under the same scope ID;
- `sessions.next_seq` advances for retirement;
- list paging plan still uses the scoped primary key with no temp b-tree;
- both per-file and shared-container layouts isolate Session and scope IDs;
- fork and Session deletion exclude/remove scoped rows correctly.
### 9.5 Harness integration
Update existing exact-write tests rather than weakening them:
- assistant/deferred settlement drops mixed-scope frame deletes;
- tool outcome staging drops mixed-scope checkpoint/memo deletes;
- unreachable residue is never consumed by snapshot/recovery after its owning child settles;
- safe-replay's current ephemeral-only delete remains ordered and fenced;
- every terminal leaf emits one retirement in the documented cleanup position;
- close/fault emits no retirement and reopen recovers current frames/checkpoint/memos;
- cancellation and unknown-outcome recovery retire only at terminal completion;
- late queued assistant/tool progress cannot commit after terminal retirement or reopen the scope;
- terminal cleanup no longer scans memo/output prefixes;
- every focused test and fixture found by the §12 old-delete/prefix greps is updated deliberately; do not rely on a historical numeric test count;
- every terminal path reached through `operationCleanupWrites` callers in `drive/{response,reconcile,structural,boundary}.ts` is covered, including each operation family and cancellation.
Likely focused files include:
```text
packages/agent/test/harness/values.test.ts
packages/agent/test/harness/{memory,jsonl}-storage*.test.ts
packages/agent/test/harness/{memory,jsonl}-session-repo*.test.ts
packages/agent/test/harness/runtime/drive-{terminal,retry-deferred,tools,reconcile,generation}.test.ts
packages/session-backends/sqlite-node/test/{storage,storage-conformance,repo,repo-conformance}.test.ts
```
Use compiler and grep guards to find additional affected tests rather than assuming this list is exhaustive.
## 10. Implementation slices
### Slice A — contract, types, Memory reference, list tags
Primary files:
```text
CREATE packages/agent/src/harness/session/scope.ts for `SessionScope`, `EphemeralScope`, `Scope`, `ephemeralScope`, scope phantoms/helpers, and runtime scope-ID validation
MODIFY packages/agent/src/harness/session/{types,values,commit,in-memory-storage-state,memory,session,index}.ts
MODIFY packages/agent/src/harness/session/testing/{conformance/storage,instrumented-storage,storage-decorator,gating-storage}.ts
MODIFY packages/agent/test/harness/{values,memory-conformance,memory-storage,storage-backed-session}.test.ts
MODIFY packages/agent/docs/mobile-handoff/01-harness/02-scopes/scopes.variance.ts
```
1. Add scope values/address/write variance and reusable-ID semantics.
2. Add retirement committed write and one-scope transaction validation.
3. Add tag/`stopAtTag` behavior.
4. Implement Memory physical identity, retirement, global sequences, snapshots.
5. Pass type tests and Memory conformance before continuing.
### Slice B — SQLite
Primary files:
```text
MODIFY packages/session-backends/sqlite-node/src/sqlite/migrations/001_initial.sql
MODIFY packages/session-backends/sqlite-node/src/sqlite/{storage,repo}.ts
MODIFY packages/session-backends/sqlite-node/src/sqlite/session/{values,session-sequences}.ts
MODIFY packages/session-backends/sqlite-node/test/{storage,storage-conformance,repo,repo-conformance}.test.ts
```
1. Add scoped schema/query identity and tags.
2. Implement atomic `DELETE ... WHERE scope_id` retirement.
3. Preserve global sequence allocation, stats, branch indexes, shared-container behavior.
4. Exclude ephemeral rows from forks.
5. Pass SQLite focused/conformance tests before continuing.
### Slice C — JSONL sidecars
Primary files:
```text
CREATE packages/agent/src/harness/session/jsonl/scope-files.ts for fixed directory/file naming, exact header parsing/serialization, discovery, atomic first creation, torn-tail parsing/repair, and recursive cleanup helpers
MODIFY packages/agent/src/harness/session/jsonl/{types,codec,storage,repo,legacy-v3,index}.ts
MODIFY packages/agent/test/harness/jsonl-{storage,storage-conformance,session-repo,session-repo-conformance,v3-migration}.test.ts
```
1. Add sidecar naming/header/create/discovery.
2. Route scoped commits to sidecars while allocating globally.
3. Persist main retirement records and serialized unlink.
4. Implement multi-file high-water calculation and globally ordered replay with reusable boundaries.
5. Handle torn tails, corruption, v3, listing, delete, close, and forks.
6. Pass all JSONL tests before runtime migration.
### Slice D — Harness migration
Primary files:
```text
MODIFY packages/agent/src/harness/session/values.ts
MODIFY packages/agent/src/harness/runtime/{progress,lane}.ts to propagate scoped write generics through progress channels and lane commands
MODIFY packages/agent/src/harness/runtime/drive/{terminal,response,deferred,tools,reconcile,structural,boundary}.ts
INSPECT packages/agent/src/harness/runtime/drive/tool-placement.ts; current source needs no scoped cleanup change
MODIFY all focused runtime tests found by the §12 greps
```
1. Scope memo/tool/frame addresses by operation ID.
2. Remove mixed-scope deletes and cleanup scans.
3. Add terminal retirement in the universal suffix.
4. Preserve current output/replay/event behavior.
5. Prove late-write fencing and every terminal/close/recovery path.
### Slice E — documentation and final validation
Update current normative/reference docs:
```text
packages/agent/docs/harness.md
packages/agent/docs/values.md
packages/agent/docs/post-wp05-roadmap.md
packages/agent/docs/mobile-handoff/README.md
packages/agent/docs/mobile-handoff/01-harness/02-scopes/scopes.md
packages/session-backends/sqlite-node/README.md
packages/agent/src/harness/telemetry.ts and generated `packages/agent/docs/telemetry-schema.md`: add the `scope` session-write item kind without implementing spans
```
Do not rewrite historical WP00–WP07 handoffs or released changelog sections. On `dev`, do not add changelog entries under the repository's main-only rule.
Run:
```bash
npm run check
./test.sh
```
Also run every modified focused test after each slice. Record full command results for the approval checkpoint.
## 11. Exclusions
Do not include:
- Chord `Op`/`WireOp` storage integration;
- JSONL address/path interning or compact tuple records;
- pending-output address renames;
- `ToolOutput`, tool API changes, replay seeding, memo/checkpoint atomicity, progress cadence, rate limiting, exec-env changes;
- compact `message_update`, assistant incremental reducer changes, or protocol replication changes;
- J1 main-log snapshot compaction;
- WP08 fork redesign beyond excluding ephemeral state from current forks;
- repository lifecycle contract changes;
- telemetry implementation beyond keeping the declared schema accurate for the new write kind;
- migrations or compatibility for older WIP format-4 SQLite schemas;
- application-level automatic scope ownership, operation-derived orphan inference, leases, TTLs, or background scope collection.
If implementation requires an excluded item, stop and revise this handoff before expanding scope.
## 12. Grep guards
Before final review, inspect every remaining match:
```text
operationToolMemoPrefix
pendingToolOutputPrefix
deleteList(pendingAssistantFrames
deleteValue(pendingToolOutput
scopeId
retireScope
stopAtTag
```
Expected outcomes:
- no session-scoped transaction directly deletes an ephemeral address;
- no terminal path omits retirement;
- no backend point read/write omits scope identity;
- no fork copies ephemeral state;
- no JSONL sidecar is treated as a Session file;
- no per-scope sequence counter exists;
- no operation-state/orphan inference exists in storage.
## 13. Exit condition
Step 1 is implementation-complete when:
- scope/address/write typing rejects mixed-scope transactions and runtime validation rejects two ephemeral IDs in one transaction;
- IDs are reusable through latest-retirement global sequence boundaries with no historical ID registry;
- Memory, JSONL, and SQLite share one global sequence space and pass common conformance;
- SQLite retirement deletes exact scoped rows atomically with terminal state;
- JSONL writes ephemeral values/lists only to sidecars, records retirement only in the main log, replays all files in global order, preserves high-water marks, and deletes retired sidecars without operation inference;
- list tags and bounded `stopAtTag` work identically across backends;
- current tool/assistant payload and event formats are unchanged;
- every terminal path retires once, while close/fault preserves active scope recovery;
- forks and repository listing/deletion handle sidecars correctly;
- focused tests, `npm run check`, and `./test.sh` pass;
- final review has no blocker;
- the complete diff and results have been presented to the user and implementation has stopped for approval.
Only after explicit approval may Step 1 be committed and pushed. Step 2 starts only after that push is confirmed.

View file

@ -0,0 +1,574 @@
# Session Storage: Scopes
> **Status:** Design evidence. The actionable Step 1 contract is [`implementation-handoff.md`](implementation-handoff.md), which supersedes this document on sequence allocation, reusable scope IDs, retirement authority/records, sidecar layout, implementation order, and the separation of JSONL encoding into Step 2.
>
> **Scope:** `packages/agent/src/harness/session/`, plus the call sites in
> `runtime/` listed in §10.
>
> **Depends on [01-delta](../01-delta/delta.md)** for the landed Chord op vocabulary that pending state is stored in (§2, §11) and for the `WireOp` form the record encoding carries (§12). The two
> are otherwise independent and compound: encoding shrinks every write, scopes
> move one class of write out of the main log entirely.
>
> Every claim here was checked against `runtime/lane.ts`, `runtime/progress.ts`
> and `runtime/drive/*.ts`. The type-level enforcement in §6 was verified with
> `tsc --strict`, including that mixed-scope commits are the *only* thing it
> rejects.
## 1. The problem
Operation state — pending tool output, pending assistant output — matters only
while the operation runs. It is crash-recovery scaffolding, not history.
The JSONL log is append-only, so it persists anyway. Worse, deleting it *adds*
records: `deleteValue` and `deleteList` write a line each, and those lines also
persist. Today only a full snapshot rewrite reclaims the space, and no in-place
compaction exists (`jsonl/storage.ts` has `createFromSnapshot`, used for fork).
Measured on 20 operations, each 400 assistant frames plus 2 tools at 60
checkpoints of a 50 KB window:
| | size |
| --- | --- |
| total JSONL | 93.89 MB |
| settled transcript entries — the part that is actually history | **0.06 MB** |
Three orders of magnitude of the file is scaffolding for finished operations.
**Two independent fixes, and they are not alternatives:**
- **Encoding** ([delta.md](../01-delta/delta.md)): value writes carry Chord ops rather than whole values, and addresses are interned. Takes the same workload to **5.32 MB in a single file**, with no change to atomicity. The op vocabulary and flush-time tracker have landed; storage integration has not.
- **Scopes** (this document): pending state leaves the main log entirely, so it is
never history to begin with.
Do the encoding first. Scopes are worth it because 5.32 MB of dead weight per 20
operations still accumulates, and there is no compaction pass to reclaim it.
## 2. Scope on the address
Scope has two halves that must not be confused: a **type-level tag** carrying no
data, and a **runtime scope id** that names the file.
```ts
export type SessionScope = { readonly kind: "session" };
export type EphemeralScope = { readonly kind: "ephemeral" };
export type Scope = SessionScope | EphemeralScope;
// Passing a scopeId is what makes an address ephemeral.
export function value<T>(namespace: string, key: string): Value<T, SessionScope>;
export function value<T>(namespace: string, key: string, scopeId: string): Value<T, EphemeralScope>;
export function list<T>(namespace: string, key: string): ValueList<T, SessionScope>;
export function list<T>(namespace: string, key: string, scopeId: string): ValueList<T, EphemeralScope>;
/** A main-log record retiring an ephemeral scope. See §5. */
export function retireScope(id: string): Write<SessionScope>;
```
The suffix is not decoration: `Session` already names the session interface in
`harness/session/types.ts`, and reusing it produces `TS2440`/`TS2484` plus an
ambiguous re-export from `session/index.ts`.
```ts
interface Value<T, Sc extends Scope = SessionScope> extends ScopedAddress<Sc> {
readonly namespace: string;
readonly key: string;
/** Present iff Sc is Ephemeral. Routes the write and names the sidecar. */
readonly scopeId?: string;
}
```
The id is a plain runtime string, because it is an operation id — generated at
runtime and unavailable to the type system. That is precisely why §6 can
distinguish *session from ephemeral* statically but cannot distinguish *two
different ephemeral scopes*: the tag is in the type, the id is not.
```ts
// Durable lists carry encoded batches; one encoder/decoder pair belongs to each value stream.
export const pendingToolOutput = (operationId: string, invocationId: string) =>
list<WireOp[]>("pi.pending.tool_output", `${operationId}:${invocationId}`, operationId);
export const pendingAssistantOutput = (operationId: string, entryId: string) =>
list<WireOp[]>("pi.pending.assistant_output", `${operationId}:${entryId}`, operationId);
export const operationToolMemo = (operationId: string, invocationId: string, name: string) =>
value<JsonValue>("pi.op.tool_memo", `${operationId}:${invocationId}:${name}`, operationId);
// unchanged — no scopeId, so Session by inference
export const laneStateValue = (lane: string) => value<DurableLaneState>("pi.lane.state", lane);
```
**Scope is the file.** A session-scoped write goes to the main log; an
ephemeral-scoped write goes to `<session>.<scopeId>.jsonl`.
Other backends need no sidecars at all: in-memory drops a `Map` keyed by scope id,
SQLite issues one `DELETE ... WHERE scope = ?`. Scope is meaningful to them only as
a lifetime hint. It is JSONL that needs the file split, and that is where the
constraints in §4 and §5 come from.
## 3. Which addresses are scoped, and why
The boundary is drawn by **write isolation**, not by lifetime.
Scoping everything under `pi.op.*` on the grounds that it dies with the operation
looks right and is wrong. `lane.ts` shows why: **every** operation commit bundles
`operationState` with `laneState`.
```ts
// lane.ts:405/431, 659/660, 749/750, 1071/1072 — the same shape each time
writes: [
...decision.writes,
setValue(operationStateValue(operationId), decision.operationState),
setValue(laneStateValue(this.name), durableLaneState(...)),
]
```
That pair is coordination state: lane state says the operation is at step N,
operation state holds the data for step N. Splitting them across files makes a
crash produce a lane that believes it is somewhere the data does not support.
The **bulk** values behave differently. `openProgress` commits exactly one write
(`runtime/progress.ts:51`), and `pendingToolOutput` / `pendingAssistantFrames`
appear in multi-write transactions only as *deletes*.
| | addresses |
| --- | --- |
| **ephemeral (sidecar)** | `pi.pending.tool_output`, `pi.pending.assistant_output`, `pi.op.tool_memo` |
| **session (main log)** | `pi.lane.state`, `pi.op.state`, `pi.op.meta`, `pi.result`, `pi.pending.entry`, `pi.branch.tip`, `pi.op.tool_args`, `pi.op.preparation` |
`operationToolMemo` is ephemeral so that the memo-plus-checkpoint bundling in
`harness-tools.md` §7.5 is a single-file transaction. `terminal.ts:34` already
treats `operationToolMemoPrefix` and `pendingToolOutputPrefix` as the same cleanup
class, so this matches how the code already thinks about them.
The size win is unaffected by leaving `operationState` behind: effectively all of
the 93.89 MB in §1 is pending tool and assistant output.
## 4. Two files are not atomic
Serialising on the Session line gives **ordering, not atomicity**. Two `write()`
calls to two descriptors, two fsyncs, no journal spanning them. A crash between
leaves the files disagreeing, and JSONL has nothing to repair it with.
So the design does not try to make cross-file transactions work. It ensures none
exist:
> **All writes in a transaction share one scope.**
That is achievable because it is already true of the code — see §5 for the audit.
Two consequences of *not* needing cross-file commits:
- **No write ordering between files is required.** The commit-record pattern
(sidecar first, fsync, then a main-log record acknowledging it) is unnecessary if
no transaction spans files. There is nothing to acknowledge.
- **No commit marker for sidecar-only writes.** One tempting design is a small
main-log record for `openProgress`, since nothing acknowledges its write. It is
not needed: that transaction is one write in one file and already atomic. Torn
tails are handled the same way as in the main log — every line is a complete op
or a whole batch, so a reader stops at the last intact one.
What remains is a **durability** choice, not a consistency one: if the sidecar is
fsynced less aggressively, recent checkpoints can be lost. That is the same bounded
loss the checkpoint interval already accepts, but it should be a deliberate policy
rather than an accident.
## 5. Audit: does the existing code satisfy the rule?
Every commit in `lane.ts` (12 sites) and every `writes:` producer in
`runtime/drive/*.ts` (32 sites) was read.
**Result: yes, with one adjustment.** No transaction writes to two files. The
spanning cases are all *deletes* of ephemeral state bundled with main-log writes:
- `response.ts:340` — the settle commit: `insertEntry`, `insertUsage`,
`setValue(branchTip)`, plus `deleteList(pendingAssistantFrames(...))`.
- `terminal.ts:50` `operationCleanupWrites` — bundles main-log deletes with
`toolMemos` and `toolOutputs` deletes, returned into the same array as
`operationResultValue` and `laneStateValue`.
- `tools.ts:230, 257` — `deleteValue(pendingToolOutput)` with main-log writes.
- `deferred.ts:157` — `deleteList(pendingAssistantFrames)`.
**The adjustment: retirement becomes a main-log record.**
```ts
retireScope(operationId): Write<SessionScope>
```
The log is the truth; the sidecar file is a cache. Recovery reads the retire record
and ignores — then removes — the sidecar. Unlinking eagerly after commit is a pure
optimisation; losing the unlink costs disk, never correctness.
This makes all four sites above single-scope, because the individual ephemeral
deletes disappear. It also *simplifies* `operationCleanupWrites`: the `scanValues`
calls for `operationToolMemoPrefix` and `pendingToolOutputPrefix` are replaced by
one `retireScope`.
And it makes the open-time sweep principled rather than heuristic. A sidecar is
dead iff the main log retired it or its operation is absent — no inference from
operation status.
### 5.1 Two ephemeral scopes in one transaction cannot arise
A lane holds `operation: { meta, state } | null` — **singular**. Every write set
uses one `operationId`, either `drive.operationId` or the newly created one in
`startOperation`, which asserts no operation is active. Retire and start are always
separate transactions: the settle commits at `lane.ts:427` and `response.ts:339`
set lane state to `null`.
Concurrency is across **lanes**, and each lane has at most one active operation, so
two ephemeral scopes never meet in a commit. The runtime assertion in §6 is defence
against a future change, not a live gap.
## 6. Static enforcement
> **`scopes.variance.ts` ships alongside this doc** and is the executable form of
> this section: `npx tsc --noEmit --strict --lib es2023 scopes.variance.ts`.
> Silence means the rule holds. Its three `@ts-expect-error` lines fail the
> compile if they ever start type-checking, so it catches the enforcement being
> weakened as well as broken.
Two phantom types, differing only in where `Sc` appears. This distinction is
load-bearing and is the single easiest thing to get wrong here.
```ts
declare const storedScopeType: unique symbol;
/** Addresses: COVARIANT — Sc only in return position. */
export interface ScopedAddress<Sc extends Scope> {
readonly [storedScopeType]?: () => Sc;
}
/** Writes: INVARIANT — Sc in both parameter and return position. */
export interface Scoped<Sc extends Scope> {
readonly [storedScopeType]?: (scope: Sc) => Sc;
}
export interface Value<T, Sc extends Scope = SessionScope>
extends StoredAddressBase, ScopedAddress<Sc> { … }
export interface ValueSetWrite<Sc extends Scope> extends Scoped<Sc> { … }
```
**Why they differ.** Reading an address is safe at any scope — `getValue` does not
care which file a value lives in, so `Value<T, SessionScope>` must be usable where
`Value<T, Scope>` is expected. Forming a transaction is *not* safe at any scope,
because two files are not atomic, so a commit must pin down exactly one.
Enforcement therefore belongs where transactions are formed, not where addresses
are read. `setValue<T, Sc>(address: Value<T, Sc>, …): Write<Sc>` is the bridge: it
infers the scope from a covariant address and stamps it onto an invariant write.
> **Getting this wrong is expensive.** Making addresses invariant too breaks every
> reader — `getValue`, `scanValues`, `readList` and everything downstream — and
> produced a 36-error cascade across storage backends that had done nothing wrong.
> The tempting workaround, making each reader generic over `Sc` to mean "any
> scope", propagates a type parameter through the whole storage stack to express
> something variance gives for free.
Verified with `tsc --noEmit --strict`. Reads pass at both scopes:
```ts
getValue(laneState); // Value<T, SessionScope>
getValue(pendingOutput); // Value<T, EphemeralScope>
```
Single-scope commits pass:
```ts
commit([setValue(laneState, a), setValue(operationState, b)]);
commit([setValue(toolMemo, m), setValue(pendingOutput, o)]);
commit([setValue(laneState, a), retireScope("op_1")]);
```
Mixed-scope commits fail, and are the *only* things that fail:
```ts
commit([setValue(laneState, a), setValue(pendingOutput, o)]); // ERROR
commit([setValue(pendingOutput, o), retireScope("op_1")]); // ERROR
```
### 6.1 What the type system cannot catch
Two different ephemeral scopes both type as `Write<EphemeralScope>`, because the id
is a runtime string (§2). A runtime assertion at commit covers it — compare
`scopeId` across all writes — and per §5.1 it should never fire.
## 7. Recovery and sweeping
On open, enumerate sidecars alongside the main log and load them. Their records
participate in replay exactly as main-log records do.
A sidecar is removed when the main log holds a `retireScope` record for it, or when
its operation is absent — a crash between commit and unlink. Torn-tail semantics
are unchanged: each line is self-contained, so a crash mid-write loses exactly the
trailing line.
## 8. Sequence numbers
Scoped writes get their own sequence space per scope. Nothing outside a scope
orders against its contents, and keeping them separate means `nextSeq` in the main
header does not account for numbers consumed by files that will be deleted.
Shared numbering would leave large gaps after several operations settled, and the
high-water mark would need separate persistence.
## 9. Measured effect
Same workload as §1:
| | size | of today |
| --- | --- | --- |
| today, single JSONL | 93.89 MB | — |
| ops + interned addresses, single JSONL | 5.32 MB | 5.7% |
| **plus scopes — main log** | **0.06 MB** | **0.06%** |
| — sidecars, retired on settle | 5.26 MB | peak 0.26 MB per operation |
The 5.7% figure is **rate-dependent and must not be quoted alone**: it holds at
2 KB of tool output per checkpoint and inverts above the cap, where a whole-value
replacement wins.
The scope result is **not** rate-dependent. The main log holds settled entries
regardless of encoding, and that is the number that governs session file growth.
## 10. What changes in the existing code
The compiler finds these for you: once addresses carry a scope and `Write<Sc>` is
invariant, every site that mixes scopes in one transaction fails to typecheck.
This list is what a full pass turned up, so you know when you are done.
**New, in `session/values.ts` and `session/types.ts`**
`SessionScope` / `EphemeralScope` / `Scope`; the two phantoms from §6;
`Value<T, Sc>` and `ValueList<T, Sc>`; `scopeId` on the address; the `value` /
`list` overloads; `retireScope`; scope-preserving `setValue` / `deleteValue` /
`appendList` / `deleteList`; `Write<Sc>` with entries, usage and retirement
session-only by construction.
**Generified**
`CommitDecision<TResult, Sc>`, `lane.command<TResult, Sc>`, `Storage.commit<Sc>`.
Give `Sc` a default of `SessionScope`: conditional types are not inference sites,
so without a default a mixed array falls back to the constraint `Scope` and every
call site fails. `settleOperation` and `continueOperation` stay **non**-generic —
see §5.
**Committed writes**
`CommittedScopeRetireWrite`, and `scopeId` carried through the committed shapes so
`JsonlStorage` can route on it.
**Call sites that must change** — each currently mixes scopes in one transaction:
| site | what it does today | what it becomes |
| --- | --- | --- |
| `drive/terminal.ts` `operationCleanupWrites` | enumerates tool memos and tool outputs via `scanValues`, deletes each alongside session deletes | one `retireScope(operationId)`; two of its four scans disappear |
| `drive/response.ts` (settle) | `deleteList(pendingAssistantOutput)` bundled with operation state | drop it — the sidecar is discarded at retire |
| `drive/deferred.ts` (superseded response) | same | same |
| `drive/tools.ts` (tool settle) | `deleteValue(pendingToolOutput)` plus memo deletes in a session write array | drop them |
| `drive/tool-placement.ts` | `deleteValue(pendingToolOutput)` | drop it |
The pattern is the same everywhere: **intra-operation cleanup of ephemeral state
is impossible**, because those commits also write operation and lane state. The
sidecar is discarded wholesale at retire. The cost is that a multi-turn operation
holds superseded pending output until settle — bounded by turns, and never
reaching the main log.
**Storage**
Sidecar routing in `JsonlStorage.commit` (assert one scope id per transaction,
route the append), unlink on a `scope` record **after** the commit, and an
open-time sweep that loads sidecars and removes any the main log already retired.
In-memory drops a `Map` keyed by scope id; SQLite issues one
`DELETE ... WHERE scope = ?` and treats a retire record as a no-op, because it has
no sidecar and the terminal transaction deletes the scoped values explicitly.
**Expected test churn**
Nine tests assert the old cleanup write set and will fail — `drive-terminal` (3),
`drive-retry-deferred` (2), `drive-tools`, `drive-reconcile`, `drive-generation`,
and one pinning reserved namespaces. They expect six or seven individual deletes
where the new code emits fewer plus one `retireScope`. That is the change landing,
not a regression.
## 11. List tags and stop conditions
Tracked state is stored as a list of op batches (`delta.md` §9). Recovery needs "the
batches since the last base batch" without unpacking every row, which lists cannot
express today: `ListReadOptions` has `cursor`, `order` and `limit` only.
`BranchScan` already solves the same problem for entries, and its vocabulary is
the one to copy — `stopAtType` / `stopAtId` for a terminator, plain field names
for a filter.
```ts
appendList<T, Sc>(address: ValueList<T, Sc>, element: T, tag?: string): ListAppendWrite<Sc>;
export interface ListElement<T> {
seq: number;
value: T;
tag?: string;
}
export interface ListReadOptions {
cursor?: ListCursor;
order?: "asc" | "desc";
limit?: number;
/** Include elements up to and including the first carrying this tag, then stop. */
stopAtTag?: string;
/** Return only elements carrying this tag. */
tag?: string;
}
```
**`stopAtTag` is a stop condition within a page, not a guarantee.** It can only end
a page earlier than `limit` would; it never overrides it. If the tagged element is
not in the page, the consumer sees no tagged element and pages again with the
cursor — the same loop `readAssistantFrames` already runs. There is no ordering
question between the two bounds and no "not found" error.
For that loop to work, `readList` must return the tag on each element. Otherwise a
consumer cannot tell "page ended at the tag" from "page ended at the limit"
without parsing the value, which is the thing the tag exists to avoid.
**The producer sets the tag, not storage.** For a tracked value the caller
already knows, because `isBase(ops)` is a token comparison on the first op
(`delta.md` §2):
```ts
const ops = tracker.flush();
if (ops.length === 0) return;
writes: [appendList(address, enc.encode(ops), isBase(ops) ? "base" : undefined)];
```
Storage must never inspect an element to derive a tag. That is what keeps op
batches opaque to the storage layer, and it is the same property that makes
`EntryScan.type` work: the discriminant is a stored column, not something derived
from the payload.
**The tag lives on the storage record**, beside `seq` and `value`:
```jsonc
["l",7,9,[["r",{…}]],"base"] // per §12.1
```
Storage must never parse an element to evaluate a predicate. That is what keeps
ops opaque to storage and the durable path free of domain knowledge
(`harness-tools.md` §7.7). It is also why `EntryScan.type` works: the discriminant
is a stored column, not something derived from the payload.
`order` stays `"asc" | "desc"` rather than `BranchScan`'s
`"newestFirst" | "oldestFirst"`. Branch order is semantic — a walk from a tip
through a tree. List order is over `seq`. Borrowing the branch words would imply a
traversal that is not happening.
Backends: SQLite gets a `tag` column and a real predicate. JSONL holds the list in
memory already, so it scans backwards and checks a field — no worse than today.
In-memory likewise.
The primitive generalises past frames: any list wanting "the tail since the last
checkpoint" gets it.
## 12. JSONL record encoding
Two dictionaries at two layers. They are the same trick and are independent: the
**address** dictionary is over `namespace` + `key` on the record; the **path**
dictionary is over paths *inside* a value's ops (`delta.md` §4).
### 12.1 Records
```
["@", addrId, namespace, key] address definition
["v", addrId, seq, wireOps] value write — WireOp[], delta.md §4
["l", addrId, seq, element, tag?] list append — element is WireOp[]
for a tracked value
["x", addrId, seq] delete (value or list)
["!", addrId, seq] retire an ephemeral scope
```
**Record verbs and op verbs are separate vocabularies read at different levels**,
but they must not look alike. `!` rather than `r` for retire: `r` is delta's
replace op, and a reader scanning a file should not have to remember which nesting
level they are at to know what a tuple means.
Retire takes an address id like everything else, defined by an `["@", id, …]`
whose namespace is the scope. That keeps one interning rule for the whole file
rather than a special case for one record.
Entries and usage rows keep their current keyed form: they are written once, never
repeat an address, and are read by machinery that has nothing to do with ops.
An address is defined on its **second** use, matching path interning — a
definition is pure overhead for an address written once, and a session has many of
those.
### 12.2 Worked example
Four writes across two addresses, before and after.
```jsonc
// today — 1250 bytes
{"kind":"value","op":"set","seq":7,"namespace":"pi.op.state","key":"01a04cf6-…","value":{"at":"starting","control":{…},"settings":{…},"latestAssistantEntryId":null}}
{"kind":"value","op":"set","seq":8,"namespace":"pi.lane.state","key":"main","value":{"currentOperationId":"01a04cf6-…",…}}
{"kind":"value","op":"set","seq":9,"namespace":"pi.op.state","key":"01a04cf6-…","value":{"at":"checkpoint",…}}
{"kind":"value","op":"set","seq":12,"namespace":"pi.op.state","key":"01a04cf6-…","value":{"at":"assistant.ready",…}}
```
```jsonc
// with both dictionaries — 547 bytes
["@",0,"pi.op.state","01a04cf6-…"]
["v",0,7,[["r",{"at":"starting","control":{…},"settings":{…},"latestAssistantEntryId":null}]]]
["@",1,"pi.lane.state","main"]
["v",1,8,[["r",{"currentOperationId":"01a04cf6-…",…}]]]
["v",0,9,[["#",0,["at"]],["s",0,"checkpoint"]]]
["v",0,12,[["s","assistant.ready"]]]
```
| line | before | after |
| --- | --- | --- |
| address definition | – | 61 |
| op.state -> `starting` (base batch) | 353 | 256 |
| address definition | – | 31 |
| lane.state (base batch) | 181 | 114 |
| op.state -> `checkpoint` | 355 | **48** |
| op.state -> `assistant.ready` | 361 | **37** |
| **total** | **1250** | **547** (44%) |
The shape matters more than the total. Base batches barely shrink — 353 to 256 is
envelope only, since the value ships whole either way. The transitions collapse
~8x because they carry one changed field instead of the whole state including the
never-changing `settings` block.
Both dictionaries appear on line 5: `0` in `["v",0,9,…]` is an **address** id,
while `["#",0,["at"]]` defines a **path** id inside the value's ops. Same trick,
different layers, separate tables. Line 6 then drops the path entirely — arity
omission, because it targets the same path as the previous op in that batch.
**Do not quote 44% as the file-level saving.** Dictionary entries are one-time, so
a real operation's ~11 op.state writes amortise them and the ratio improves; but
transcript entries are untouched, and they were 18 KB of a 67 KB tool-heavy run.
Applied there this hits the ~27 KB of value writes and would take the file down by
roughly a fifth, not by half.
### 12.3 Reading
A reader builds both tables while replaying, so a torn tail costs exactly the
trailing lines and no header needs rewriting. A snapshot rewrite starts fresh
tables and re-emits definitions naturally.
A numeric `addrId` for which no definition has been seen is a corrupt file, not a
recoverable state — unlike a missing path id, which cannot occur because path
definitions are inside the same record as their first use.
## 13. Open questions
- File-handle pressure across concurrent lanes, each with a live sidecar. Probably
fine, unmeasured.
- fsync policy for sidecars (§4) — matched to the main log, or relaxed given the
contents are bounded-loss scaffolding.
- Whether a long-running operation should rotate its sidecar. `rebase()`
(`delta.md` §3.2.3) bounds *recovery* length but not file size, so a command
running for hours still grows its sidecar without bound.
- Whether an in-place compaction pass for the main log is worth building anyway.
`createFromSnapshot` plus atomic replace is the mechanism; scopes reduce the need
but do not remove it, since superseded session-scoped values still accumulate.

View file

@ -0,0 +1,65 @@
/**
* Executable form of scopes.md §6.
*
* npx tsc --noEmit --strict --lib es2023 scopes.variance.ts
*
* Silence means the rule holds: reads accept either scope, single-scope commits
* pass, and mixed-scope commits are the ONLY thing rejected. The three
* `@ts-expect-error` lines fail the compile if they ever start type-checking,
* so this catches the enforcement being weakened as well as broken.
*
* The variance split is the part that is easy to get backwards. Making addresses
* invariant too breaks every reader — getValue, scanValues, readList and
* everything downstream — and the tempting fix, making each reader generic over
* Sc, propagates a type parameter through the whole storage stack to express
* what variance gives for free.
*/
declare const tag: unique symbol;
type SessionScope = { readonly kind: "session" };
type EphemeralScope = { readonly kind: "ephemeral" };
type Scope = SessionScope | EphemeralScope;
/** Addresses: COVARIANT — Sc only in return position. Readers take any scope. */
interface ScopedAddress<Sc extends Scope> { readonly [tag]?: () => Sc }
/** Writes: INVARIANT — Sc in both positions. Mixed commits cannot unify. */
interface Scoped<Sc extends Scope> { readonly [tag]?: (s: Sc) => Sc }
interface Value<T, Sc extends Scope = SessionScope> extends ScopedAddress<Sc> { k: "v" }
interface ValueList<T, Sc extends Scope = SessionScope> extends ScopedAddress<Sc> { k: "l" }
interface VSet<Sc extends Scope = Scope> extends Scoped<Sc> { kind: "value"; op: "set" }
interface VDel<Sc extends Scope = Scope> extends Scoped<Sc> { kind: "value"; op: "delete" }
interface LApp<Sc extends Scope = Scope> extends Scoped<Sc> { kind: "list"; op: "append" }
interface EntryWrite { kind: "entry" }
interface RetireWrite extends Scoped<SessionScope> { kind: "scope" }
type Write<Sc extends Scope = SessionScope> =
| VSet<Sc> | VDel<Sc> | LApp<Sc>
| (Sc extends SessionScope ? EntryWrite | RetireWrite : never);
declare function setValue<T, Sc extends Scope>(a: Value<T, Sc>, v: T): VSet<Sc>;
declare function deleteValue<T, Sc extends Scope>(a: Value<T, Sc>): VDel<Sc>;
declare function appendList<T, Sc extends Scope>(a: ValueList<T, Sc>, v: T): LApp<Sc>;
declare function retireScope(id: string): RetireWrite;
declare function insertEntry(e: unknown): EntryWrite;
declare function commit<Sc extends Scope = SessionScope>(w: readonly Write<Sc>[]): void;
declare function getValue<T>(a: Value<T, Scope>): T; // reader: any scope
const laneState = null as unknown as Value<string>; // session
const pendingOut = null as unknown as ValueList<string, EphemeralScope>; // ephemeral
const toolMemo = null as unknown as Value<string, EphemeralScope>;
// READS accept either scope — that is what covariance buys.
getValue(laneState);
getValue(toolMemo);
// Single-scope commits pass.
commit([setValue(laneState, "a")]);
commit([setValue(laneState, "a"), insertEntry({}), retireScope("op_1")]);
commit([appendList(pendingOut, "x"), setValue(toolMemo, "m")]);
// @ts-expect-error mixed scopes in one transaction
commit([setValue(laneState, "a"), appendList(pendingOut, "x")]);
// @ts-expect-error a retire is session-only, so it cannot join an ephemeral commit
commit([appendList(pendingOut, "x"), retireScope("op_1")]);
// @ts-expect-error entries are session-only
commit<EphemeralScope>([insertEntry({})]);

View file

@ -0,0 +1,172 @@
import { type Static, Type } from "typebox";
import type { Context } from "../context.ts";
import type { AgentHarnessTool } from "../types.ts";
import { getOrThrow } from "../types.ts";
import { DEFAULT_MAX_BYTES, DEFAULT_MAX_LINES, formatSize, type TruncationResult } from "../utils/truncate.ts";
import type { ExecutionToolContext } from "./tool-context.ts";
const MAX_TIMEOUT_SECONDS = 2_147_483_647 / 1000;
/**
* Coalescing cadence for live updates. The env owns capping and coalescing now
* (`executionenv.md` §3), so bash no longer has an update throttle or a
* checkpoint interval of its own.
*/
const BASH_UPDATE_INTERVAL_MS = 100;
const bashSchema = Type.Object({
command: Type.String({ description: "Bash command to execute" }),
timeout: Type.Optional(Type.Number({ description: "Timeout in seconds (optional, no default timeout)" })),
});
export type BashToolInput = Static<typeof bashSchema>;
export interface BashToolDetails {
truncation?: TruncationResult;
/** Set by the exec env, on the machine the command ran on. */
spillPath?: string;
}
export interface BashExecution {
command: string;
cwd: string;
env: Record<string, string>;
inheritEnv: boolean;
}
export type BashPrepare<TContext extends ExecutionToolContext = ExecutionToolContext> = (
execution: BashExecution,
toolContext: TContext,
context: Context,
) => void | Promise<void>;
export interface BashToolOptions<TContext extends ExecutionToolContext = ExecutionToolContext> {
commandPrefix?: string;
prepare?: BashPrepare<TContext>;
}
function validateTimeout(timeout: number | undefined): void {
if (timeout === undefined) return;
if (!Number.isFinite(timeout) || timeout <= 0) {
throw new Error("Invalid timeout: must be a finite number of seconds");
}
if (timeout > MAX_TIMEOUT_SECONDS) {
throw new Error(`Invalid timeout: maximum is ${MAX_TIMEOUT_SECONDS} seconds`);
}
}
export function createBashTool<TContext extends ExecutionToolContext = ExecutionToolContext>(
options?: BashToolOptions<TContext>,
): AgentHarnessTool<TContext, typeof bashSchema, BashToolDetails | undefined> {
return {
name: "bash",
label: "bash",
description: `Execute a bash command in the current working directory. Returns stdout and stderr. Output is truncated to last ${DEFAULT_MAX_LINES} lines or ${DEFAULT_MAX_BYTES / 1024}KB (whichever is hit first). If truncated, full output is saved to a temp file. Optionally provide a timeout in seconds.`,
parameters: bashSchema,
async execute(_toolCallId, { command, timeout }, onUpdate, toolContext, _invocation, context) {
validateTimeout(timeout);
const { env } = toolContext;
const execution: BashExecution = {
command: options?.commandPrefix ? `${options.commandPrefix}\n${command}` : command,
cwd: env.cwd,
env: {},
inheritEnv: true,
};
await options?.prepare?.(execution, toolContext, context);
// The env caps, coalesces and spills at the source. bash keeps a mirror
// only to build the settled result; it never truncates and never chooses
// a spill path.
let view = "";
let truncation: TruncationResult | undefined;
let spillPath: string | undefined;
const publish = (checkpoint: boolean): void => {
onUpdate(
{
content: [{ type: "text" as const, text: view }],
details: { truncation, spillPath },
},
checkpoint ? { checkpoint: true } : undefined,
);
};
onUpdate({ content: [], details: undefined });
const result = await env.exec(
execution.command,
{
cwd: execution.cwd,
env: execution.env,
inheritEnv: execution.inheritEnv,
timeout,
capture: {
limits: { maxBytes: DEFAULT_MAX_BYTES, maxLines: DEFAULT_MAX_LINES, retain: "tail" },
intervalMs: BASH_UPDATE_INTERVAL_MS,
spill: true,
},
onUpdate: (update) => {
switch (update.kind) {
case "chunk":
view += update.text;
break;
case "snapshot":
view = update.output.text;
truncation = update.output.truncation;
spillPath = update.output.spillPath;
break;
case "counters":
truncation = update.truncation;
break;
}
// A snapshot is the durable-worthy state: it is the only kind that
// can express eviction, so it is where a crash loses least.
publish(update.kind === "snapshot");
},
},
context,
);
if (!result.ok) {
const error = result.error;
const status =
error.code === "timeout"
? `Command timed out after ${timeout} seconds`
: error.code === "aborted"
? "Command aborted"
: error.message;
throw new Error(view ? `${view}\n\n${status}` : status, { cause: error });
}
const captured = result.value.output;
view = captured.text;
truncation = captured.truncation;
spillPath = captured.spillPath;
let outputText = captured.text;
let details: BashToolDetails | undefined;
if (captured.truncation.truncated) {
details = { truncation: captured.truncation, spillPath };
const startLine = captured.truncation.totalLines - captured.truncation.outputLines + 1;
const endLine = captured.truncation.totalLines;
if (captured.truncation.lastLinePartial) {
// One line alone exceeded the byte cap. Line numbers would be
// misleading here, so report the line's own size instead — and only
// the env can measure it, since it holds the raw buffer.
const lastLineSize = formatSize(captured.lastLineBytes ?? captured.truncation.outputBytes);
outputText += `\n\n[Showing last ${formatSize(captured.truncation.outputBytes)} of line ${endLine} (line is ${lastLineSize}). Full output: ${spillPath}]`;
} else if (captured.truncation.truncatedBy === "lines") {
outputText += `\n\n[Showing lines ${startLine}-${endLine} of ${captured.truncation.totalLines}. Full output: ${spillPath}]`;
} else {
outputText += `\n\n[Showing lines ${startLine}-${endLine} of ${captured.truncation.totalLines} (${formatSize(DEFAULT_MAX_BYTES)} limit). Full output: ${spillPath}]`;
}
}
if (result.value.exitCode !== 0) {
throw new Error(
`${outputText ? `${outputText}\n\n` : ""}Command exited with code ${result.value.exitCode}`,
);
}
return { content: [{ type: "text", text: outputText || "(no output)" }], details };
},
};
}

View file

@ -0,0 +1,148 @@
# ExecutionEnv: bounded shell output
**Status:** implemented in production source. The prototype files beside this document are historical evidence; production lives in:
- `packages/agent/src/harness/utils/adaptive-publisher.ts`
- `packages/agent/src/harness/utils/output-capture.ts`
- `packages/agent/src/harness/env/nodejs.ts`
- `packages/agent/src/harness/tools/bash.ts`
The generic `ToolOutput` use of the same publisher remains in `04-tool-output`.
## 1. Problem
The old `Shell.exec()` accumulated complete strings inside `NodeExecutionEnv`:
```ts
stdout += chunk;
stderr += chunk;
```
Bash truncated only after those strings had already been built. `cat 1gb.txt` therefore materialized a gigabyte in the worker before any tool-level bound could help.
Spilling also belongs where bytes originate. If execution is remote, a spill on the worker is inaccessible to the model's `read` and `grep` tools, and creating it after transport would first send the complete gigabyte over the connection.
## 2. Boundary
`ExecutionEnv` now owns:
- a bounded head or tail view;
- complete byte and line totals;
- lazy source-local spill after the view first crosses its limits;
- persistent source-local spill with bounded write-stream backpressure;
- adaptive publication of the latest bounded state;
- a forced final publication before settlement.
It does not return or retain separate `stdout` and `stderr` values. Both pipes feed one arrival-ordered model-visible text view, matching bash and `ToolOutput`; preserving stream styling through tail eviction would require a segmented retained state that the Harness does not expose. Text is folded from updates, and `ShellExecResult` contains only exit and truncation/spill metadata.
Bash owns only command semantics and its model-visible footer. Its old rolling buffer, spill creation, 100 ms throttle, and full-output accumulation are gone. The existing two-second durable checkpoint request remains temporarily until `ToolOutput` owns durable cadence.
## 3. Contract
```ts
interface ShellOutputLimits {
maxBytes: number;
maxLines: number;
retain?: "head" | "tail";
}
interface ShellOutputCaptureOptions {
limits: ShellOutputLimits;
spill?: boolean;
}
type ShellOutputTruncation = Omit<TruncationResult, "content">;
interface ShellOutputMetadata {
truncation: ShellOutputTruncation;
spillPath?: string;
lastLineBytes?: number;
}
interface ShellOutputView extends ShellOutputMetadata {
text: string;
}
type ShellOutputUpdate =
| { kind: "replace"; output: ShellOutputView }
| { kind: "append"; text: string; metadata: ShellOutputMetadata }
| { kind: "slide"; drop: number; text: string; metadata: ShellOutputMetadata }
| { kind: "metadata"; metadata: ShellOutputMetadata };
interface ShellExecResult extends ShellOutputMetadata {
exitCode: number;
}
```
`drop` counts JavaScript string code units, matching `slice()`. Updates are ordered. A consumer applies them with `applyShellOutputUpdate()`.
A complete replacement establishes the initial state or recovers when no verified overlap exists. An append carries only a growing suffix. A slide drops a prefix and appends the new suffix. Metadata updates move totals or the spill path without resending text.
The compatibility `executeShellWithCapture()` helper still returns one bounded final view. Its `onChunk` callback receives initial, append, and slide text only; metadata and post-turnover replacements are not mislabeled as new bytes.
## 4. Adaptive publication
`AdaptivePublisher` keeps only the latest dirty state. Intermediate process writes never become a queue of output updates.
Policy is harness-global rather than per tool:
```ts
minIntervalMs = 100;
targetBytesPerSecond = 100 * 1024;
nextDelayMs = max(minIntervalMs, encodedUpdateBytes * 1000 / targetBytesPerSecond);
```
The first dirty state after idle is immediate. Writes received before the deadline collapse into the latest bounded state. One trailing timer guarantees eventual publication. Finalization bypasses the deadline once, still bounded by the retained cap.
The publisher commits its baseline before invoking the consumer. If a consumer applies an update and then throws, finalization cannot emit the same delta twice. The command fails with `callback_error`.
### Workload behavior
| workload | result |
| --- | --- |
| Finishes below cap | immediate initial state, small appends, forced final state |
| Below-cap trickle | isolated writes are immediate; sustained writes are at most 100 ms apart |
| Full-force output after cap | complete turnovers are cap-sized replacements spaced by their encoded size |
| Post-cap trickle | small verified `slide` updates remain responsive; the complete window is not resent |
| Burst then silence | one leading update and one trailing update |
| Error, timeout, or abort | latest bounded state is forced before the error settles |
At a 50 KB cap and 100 KB/s target, repeated complete turnovers settle near two updates per second. Small post-cap slides still use the 100 ms floor.
The rate bound is amortized. An immediate-after-idle update and forced terminal update may each create one cap-bounded burst.
## 5. Capture and spill
`OutputCapture` trims its decoded line-aware working buffer back to twice the byte cap when it exceeds four times the cap. Tail mode drops old text; head mode preserves the original prefix. This amortizes UTF-8 trimming instead of rescanning the retained window for every process chunk.
Spill creation is lazy. Before crossing, the source keeps at most the bounded prefix needed to create a complete archive. On crossing it:
1. pauses stdout and stderr while it creates the file inside the execution environment;
2. opens one persistent append stream with a bounded 8 MB high-water mark;
3. writes the preserved raw prefix and subsequent raw chunks in arrival order;
4. resumes immediately while the writer accepts data;
5. pauses again only when `write()` reports backpressure, then resumes on `drain`.
This avoids both an unbounded promise chain and one async file-open/append cycle per process chunk. A spill create or stream-write failure kills the child and fails execution rather than publishing lossy success.
The spill path is force-published as metadata when it becomes available. Spill writes are awaited before the final output flush.
Node streams remain raw for spill throughput and exact archival bytes. `OutputCapture` uses one streaming `TextDecoder`, so a read boundary cannot split a code point; invalid display control characters are removed only from bounded snapshots, not by scanning the complete raw stream. Line totals count a final unterminated line, and `lastLineBytes` remains exact even when one line exceeds the working buffer.
## 6. Remote execution
When worker and execution environment are colocated, updates are in-process and this publisher primarily bounds capture work. `ToolOutput` remains the downstream event/durability limiter.
When an execution environment is physically remote from its worker, the same bounded updates cross that transport. Full-force output cannot transfer the complete stream: intermediate writes collapse at the source, while the complete stream remains in the source-local spill.
Each real costly boundary gets its own publisher instance. A colocated deployment may bypass transport serialization; a custom tool that bypasses `ExecutionEnv` still passes through the future `ToolOutput` publisher.
## 7. Remaining work
- Move generic custom-tool composition, text retention, event publication, and durable cadence into `ToolOutput`.
- Replace whole `AgentToolResult` progress with Chord operations at that downstream boundary.
- Make memo and output checkpoint persistence one atomic transaction.
- Seed replay from durable output rather than deleting it.
- Put spills in an environment-owned session directory and sweep them after session lifetime plus a crash-retention floor.
- Decide explicit limits for images and structured details; text is bounded, those values are not yet.
- Define raw binary-output behavior. Current shell output remains lossy UTF-8 text.

View file

@ -0,0 +1,137 @@
diff --git a/packages/agent/src/harness/env/nodejs.ts b/packages/agent/src/harness/env/nodejs.ts
index 732dc84db..db9eb9a57 100644
--- a/packages/agent/src/harness/env/nodejs.ts
+++ b/packages/agent/src/harness/env/nodejs.ts
@@ -19,6 +19,7 @@ import { basename, isAbsolute, join, resolve } from "node:path";
import { createInterface } from "node:readline";
import { fileURLToPath } from "node:url";
import type { Context } from "../context.ts";
+import { OutputCapture } from "../utils/output-capture.ts";
import {
type ExecutionEnv,
ExecutionError,
@@ -28,7 +29,10 @@ import {
type FileKind,
ok,
type Result,
+ type ShellCapabilities,
type ShellExecOptions,
+ type ShellExecResult,
+ type ShellStream,
toError,
} from "../types.ts";
@@ -365,11 +369,15 @@ export class NodeExecutionEnv implements ExecutionEnv {
return ok(join(...parts));
}
+ capabilities(): ShellCapabilities {
+ return { capture: true, spill: true, liveUpdates: true };
+ }
+
async exec(
command: string,
options: ShellExecOptions | undefined,
context: Context,
- ): Promise<Result<{ stdout: string; stderr: string; exitCode: number }, ExecutionError>> {
+ ): Promise<Result<ShellExecResult, ExecutionError>> {
const signal = context.abortSignal;
if (signal?.aborted) return err(new ExecutionError("aborted", "aborted"));
const timeoutMsResult = resolveTimeoutMs(options?.timeout);
@@ -392,9 +400,28 @@ export class NodeExecutionEnv implements ExecutionEnv {
);
}
+ // Spill lives in the exec env, on the machine where the model's own `read`
+ // and `grep` run — the whole reason capping happens here rather than on the
+ // agent side (`executionenv.md` §1).
+ let spillHandle: string | undefined;
+ let spillChain: Promise<void> = Promise.resolve();
+ const wantSpill = options?.capture?.spill === true;
+
return await new Promise((resolvePromise) => {
- let stdout = "";
- let stderr = "";
+ const capture = new OutputCapture(options?.capture, options?.onUpdate);
+ const spillChunk = (chunk: string): Promise<void> => {
+ if (!wantSpill) return Promise.resolve();
+ spillChain = spillChain.then(async () => {
+ if (spillHandle === undefined) {
+ const created = await this.createTempFile({ prefix: "pi-output-", suffix: ".log" }, context);
+ if (!created.ok) return;
+ spillHandle = created.value;
+ capture.setSpillPath(spillHandle);
+ }
+ await this.appendFile(spillHandle, chunk, context);
+ });
+ return spillChain;
+ };
let settled = false;
let timedOut = false;
let callbackError: ExecutionError | undefined;
@@ -407,7 +434,7 @@ export class NodeExecutionEnv implements ExecutionEnv {
}
};
- const settle = (result: Result<{ stdout: string; stderr: string; exitCode: number }, ExecutionError>) => {
+ const settle = (result: Result<ShellExecResult, ExecutionError>) => {
if (timeoutId) clearTimeout(timeoutId);
if (signal) signal.removeEventListener("abort", onAbort);
if (child?.pid) this.activeChildPids.delete(child.pid);
@@ -458,31 +485,32 @@ export class NodeExecutionEnv implements ExecutionEnv {
}
}
+ // setEncoding("utf8") is what keeps a chunk boundary from splitting a
+ // codepoint; without it a multi-byte sequence straddling a read bakes in a
+ // replacement character (`executionenv.md` §8).
child.stdout?.setEncoding("utf8");
child.stderr?.setEncoding("utf8");
- child.stdout?.on("data", (chunk: string) => {
- stdout += chunk;
- try {
- options?.onStdout?.(chunk, context);
- } catch (error) {
- const cause = toError(error);
- callbackError = new ExecutionError("callback_error", cause.message, cause);
- onAbort();
- }
- });
- child.stderr?.on("data", (chunk: string) => {
- stderr += chunk;
+ const feed = (stream: ShellStream, chunk: string) => {
try {
- options?.onStderr?.(chunk, context);
+ capture.push(stream, chunk, context);
+ void spillChunk(chunk);
} catch (error) {
const cause = toError(error);
callbackError = new ExecutionError("callback_error", cause.message, cause);
onAbort();
}
- });
+ };
+ child.stdout?.on("data", (chunk: string) => feed("stdout", chunk));
+ child.stderr?.on("data", (chunk: string) => feed("stderr", chunk));
void waitForChildProcess(child).then(
- (code) => {
+ async (code) => {
+ // The spill chain is async and lazy: the file is created on the first
+ // chunk. Await it before snapshotting, or the returned CapturedOutput
+ // reports no spillPath for output that was in fact archived.
+ await spillChain;
+ capture.flush(context);
+ capture.dispose();
if (callbackError) {
settle(err(callbackError));
return;
@@ -495,7 +523,7 @@ export class NodeExecutionEnv implements ExecutionEnv {
settle(err(new ExecutionError("aborted", "aborted")));
return;
}
- settle(ok({ stdout, stderr, exitCode: code ?? 0 }));
+ settle(ok({ output: capture.snapshot(), exitCode: code ?? 0 }));
},
(error: Error) => settle(err(new ExecutionError("spawn_error", error.message, error))),
);

View file

@ -0,0 +1,149 @@
diff --git a/packages/agent/test/harness/nodejs-env.test.ts b/packages/agent/test/harness/nodejs-env.test.ts
index 361040495..ee28ef506 100644
--- a/packages/agent/test/harness/nodejs-env.test.ts
+++ b/packages/agent/test/harness/nodejs-env.test.ts
@@ -8,7 +8,6 @@ import { afterEach, describe, expect, it } from "vitest";
import { BACKGROUND_CONTEXT, withAbortSignal } from "../../src/harness/context.ts";
import { NodeExecutionEnv } from "../../src/harness/env/nodejs.ts";
import { FileError, getOrThrow } from "../../src/harness/types.ts";
-import { executeShellWithCapture } from "../../src/harness/utils/shell-output.ts";
import { createTempDir } from "./session-test-utils.ts";
const chmodRestorePaths: string[] = [];
@@ -303,7 +302,8 @@ describe("NodeExecutionEnv", () => {
BACKGROUND_CONTEXT,
),
);
- expect(result).toEqual({ stdout: `${await realpath(root)}:ok`, stderr: "", exitCode: 0 });
+ expect(result.output.text).toContain(`${await realpath(root)}:ok`.trim());
+ expect(result.exitCode).toBe(0);
});
it.each([
@@ -334,7 +334,7 @@ describe("NodeExecutionEnv", () => {
),
);
- expect(result.stdout).toBe(`${expectedSessionFile}|true|preserved`);
+ expect(result.output.text).toBe(`${expectedSessionFile}|true|preserved`);
},
);
@@ -358,7 +358,7 @@ describe("NodeExecutionEnv", () => {
),
);
- expect(result.stdout).toBe("::explicit");
+ expect(result.output.text).toBe("::explicit");
} finally {
if (previousInherited === undefined) delete process.env[inheritedKey];
else process.env[inheritedKey] = previousInherited;
@@ -396,7 +396,9 @@ describe("NodeExecutionEnv", () => {
await wslEnv.exec(`name='World'; echo "Hello, ${nameExpansion}!"`, undefined, BACKGROUND_CONTEXT),
);
- expect(result).toEqual({ stdout: "Hello, World!\n", stderr: "args:-s\n", exitCode: 0 });
+ expect(result.output.text).toContain("Hello, World!");
+ expect(result.output.text).toContain("args:-s");
+ expect(result.exitCode).toBe(0);
} finally {
process.chdir(originalCwd);
process.env.PATH = originalPath;
@@ -425,7 +427,7 @@ describe("NodeExecutionEnv", () => {
() => controller.abort(),
),
);
- expect(result.stdout).toContain("child-exiting");
+ expect(result.output.text).toContain("child-exiting");
} finally {
controller.abort();
cleanupDetachedChild(pidFile);
@@ -448,25 +450,30 @@ describe("NodeExecutionEnv", () => {
it("streams stdout and stderr chunks", async () => {
const root = createTempDir();
const env = new NodeExecutionEnv({ cwd: root });
- let stdout = "";
- let stderr = "";
+ // Streams are aggregated in arrival order now; `chunk` carries `stream` so a
+ // renderer can still colour stderr (`executionenv.md` §6).
+ let seen = "";
+ const streams: string[] = [];
const result = getOrThrow(
await env.exec(
"printf out; printf err >&2",
{
- onStdout: (chunk) => {
- stdout += chunk;
- },
- onStderr: (chunk) => {
- stderr += chunk;
+ onUpdate: (update) => {
+ if (update.kind !== "chunk") return;
+ seen += update.text;
+ streams.push(update.stream);
},
},
BACKGROUND_CONTEXT,
),
);
- expect(result).toEqual({ stdout: "out", stderr: "err", exitCode: 0 });
- expect(stdout).toBe("out");
- expect(stderr).toBe("err");
+ expect(result.exitCode).toBe(0);
+ expect(result.output.text).toContain("out");
+ expect(result.output.text).toContain("err");
+ expect(seen).toContain("out");
+ expect(seen).toContain("err");
+ expect(streams).toContain("stdout");
+ expect(streams).toContain("stderr");
});
it("reports a missing working directory before spawning", async () => {
@@ -484,7 +491,8 @@ describe("NodeExecutionEnv", () => {
const root = createTempDir();
const env = new NodeExecutionEnv({ cwd: root });
const result = getOrThrow(await env.exec("exit 7", undefined, BACKGROUND_CONTEXT));
- expect(result).toEqual({ stdout: "", stderr: "", exitCode: 7 });
+ expect(result.output.text).toContain("".trim());
+ expect(result.exitCode).toBe(7);
});
it("returns timeout errors for commands exceeding the timeout", async () => {
@@ -501,7 +509,7 @@ describe("NodeExecutionEnv", () => {
const result = await env.exec(
"printf out",
{
- onStdout: () => {
+ onUpdate: () => {
throw new Error("callback failed");
},
},
@@ -542,12 +550,23 @@ describe("NodeExecutionEnv", () => {
const root = createTempDir();
const env = new NodeExecutionEnv({ cwd: root });
const result = getOrThrow(
- await executeShellWithCapture(env, "yes line | head -n 15000", undefined, BACKGROUND_CONTEXT),
+ await env.exec(
+ "yes line | head -n 15000",
+ {
+ capture: {
+ limits: { maxBytes: 50 * 1024, maxLines: 2000, retain: "tail" },
+ spill: true,
+ },
+ },
+ BACKGROUND_CONTEXT,
+ ),
);
- expect(result.truncated).toBe(true);
- expect(result.fullOutputPath).toBeDefined();
- const fullOutput = getOrThrow(await env.readTextFile(result.fullOutputPath!, BACKGROUND_CONTEXT));
+ expect(result.output.truncation.truncated).toBe(true);
+ // The spill lives inside the exec env, on the machine the command ran on —
+ // which is where the model's own read and grep run (`executionenv.md` §1).
+ expect(result.output.spillPath).toBeDefined();
+ const fullOutput = getOrThrow(await env.readTextFile(result.output.spillPath!, BACKGROUND_CONTEXT));
expect(fullOutput.split("\n").length).toBeGreaterThan(10000);
- expect(result.output.length).toBeLessThan(fullOutput.length);
+ expect(result.output.text.length).toBeLessThan(fullOutput.length);
});
});

View file

@ -0,0 +1,194 @@
import type { Context } from "../context.ts";
import type { CaptureOptions, CapturedOutput, ShellOutputUpdate, ShellStream } from "../types.ts";
import {
DEFAULT_MAX_BYTES,
DEFAULT_MAX_LINES,
type TruncationResult,
truncateHead,
truncateTail,
utf8ByteLength,
} from "./truncate.ts";
/**
* Bounded capture for a Shell implementation.
*
* Owns the two-stage structure that `shell-output.ts` had inside `bash.ts`: a raw
* buffer trimmed to `maxBytes * 2` as a memory guard, and a retained view
* computed from it per snapshot. `truncateTail` is line-aware — it walks
* backwards line by line and takes a partial only when one line alone exceeds the
* cap — so eviction is not byte arithmetic and cannot be done incrementally.
*
* Emits the three update kinds from `executionenv.md` §4 and enforces the
* ordering invariant: once the cap is crossed, a `chunk` never follows without an
* intervening `snapshot`.
*/
export class OutputCapture {
readonly #maxBytes: number;
readonly #maxLines: number;
readonly #retain: "head" | "tail";
readonly #intervalMs: number;
readonly #onUpdate?: (update: ShellOutputUpdate, context: Context) => void;
/** Raw rolling buffer, held at twice the cap. */
#buffer = "";
/** Everything ever produced. */
#totalBytes = 0;
/** Newlines seen. A trailing partial line is counted separately — a command
* emitting one 60 KB line with no newline has produced one line, not zero. */
#newlines = 0;
#endsWithNewline = true;
/** True once the retained view has begun evicting (tail) or filled (head). */
#crossed = false;
/** Pending chunks for the current coalescing window. */
#pending: { stream: ShellStream; text: string }[] = [];
#lastFlush = 0;
#timer: ReturnType<typeof setTimeout> | undefined;
#spillPath: string | undefined;
constructor(
options: CaptureOptions | undefined,
onUpdate: ((update: ShellOutputUpdate, context: Context) => void) | undefined,
) {
this.#maxBytes = options?.limits.maxBytes ?? DEFAULT_MAX_BYTES;
this.#maxLines = options?.limits.maxLines ?? DEFAULT_MAX_LINES;
this.#retain = options?.limits.retain ?? "tail";
this.#intervalMs = options?.intervalMs ?? 0;
this.#onUpdate = onUpdate;
}
push(stream: ShellStream, text: string, context: Context): void {
if (text === "") return;
this.#totalBytes += utf8ByteLength(text);
this.#newlines += countNewlines(text);
this.#endsWithNewline = text.endsWith("\n");
this.#buffer += text;
// Memory guard. The retained view is computed from this, never equal to it.
const guard = this.#maxBytes * 2;
if (utf8ByteLength(this.#buffer) > guard) {
this.#buffer = trimToLastBytes(this.#buffer, guard);
}
this.#pending.push({ stream, text });
this.#schedule(context);
}
setSpillPath(path: string | undefined): void {
this.#spillPath = path;
}
/** The bounded view, recomputed from the raw buffer. */
snapshot(): CapturedOutput {
const truncation =
this.#retain === "head"
? truncateHead(this.#buffer, { maxBytes: this.#maxBytes, maxLines: this.#maxLines })
: truncateTail(this.#buffer, { maxBytes: this.#maxBytes, maxLines: this.#maxLines });
// The buffer already lost its front, so totals come from counters rather
// than from what survives.
const totalLines = this.#totalLines();
const truncated = this.#totalBytes > this.#maxBytes || totalLines > this.#maxLines;
return {
text: truncation.content,
truncation: {
...truncation,
lastLinePartial: truncation.lastLinePartial,
truncated,
truncatedBy: truncated ? (totalLines > this.#maxLines ? "lines" : "bytes") : null,
totalBytes: this.#totalBytes,
totalLines,
},
...(this.#spillPath === undefined ? {} : { spillPath: this.#spillPath }),
...(truncation.lastLinePartial ? { lastLineBytes: this.#lastLineBytes() } : {}),
};
}
#totalLines(): number {
return this.#newlines + (this.#endsWithNewline || this.#totalBytes === 0 ? 0 : 1);
}
/** Bytes in the final line of the raw buffer, for the single-oversized-line case. */
#lastLineBytes(): number {
const start = this.#buffer.lastIndexOf("\n");
return utf8ByteLength(start === -1 ? this.#buffer : this.#buffer.slice(start + 1));
}
#schedule(context: Context): void {
if (this.#onUpdate === undefined) return;
if (this.#intervalMs === 0) {
this.flush(context);
return;
}
if (this.#timer !== undefined) return;
const wait = Math.max(0, this.#lastFlush + this.#intervalMs - Date.now());
this.#timer = setTimeout(() => {
this.#timer = undefined;
this.flush(context);
}, wait);
}
/** Emit one update for everything buffered since the last flush. */
flush(context: Context): void {
if (this.#onUpdate === undefined) return;
this.#lastFlush = Date.now();
const pending = this.#pending;
this.#pending = [];
if (pending.length === 0) return;
const nowCrossed = this.#totalBytes > this.#maxBytes || this.#totalLines() > this.#maxLines;
if (this.#retain === "head") {
// Head never evicts. Below the cap the appended text is complete; above
// it, only totals move — resending maxBytes to change a number is the
// thing `counters` exists to avoid.
if (!nowCrossed) {
this.#onUpdate({ kind: "chunk", stream: pending[0]!.stream, text: joined(pending) }, context);
} else {
this.#onUpdate({ kind: "counters", truncation: this.snapshot().truncation }, context);
}
this.#crossed = nowCrossed;
return;
}
// Tail. Once the buffer is full every byte in evicts a byte out, so there is
// no cheap-trickle regime past the cap: chunks until full, snapshots after.
if (nowCrossed) {
this.#crossed = true;
this.#onUpdate({ kind: "snapshot", output: this.snapshot() }, context);
return;
}
if (this.#crossed) {
// Defensive: never let a chunk follow the crossing.
this.#onUpdate({ kind: "snapshot", output: this.snapshot() }, context);
return;
}
this.#onUpdate({ kind: "chunk", stream: pending[0]!.stream, text: joined(pending) }, context);
}
dispose(): void {
if (this.#timer !== undefined) clearTimeout(this.#timer);
this.#timer = undefined;
}
}
const joined = (pending: { text: string }[]): string => pending.map((p) => p.text).join("");
function countNewlines(text: string): number {
let count = 0;
for (let index = text.indexOf("\n"); index !== -1; index = text.indexOf("\n", index + 1)) count++;
return count;
}
/** Keep the last `bytes` UTF-8 bytes without splitting a codepoint. */
function trimToLastBytes(text: string, bytes: number): string {
if (utf8ByteLength(text) <= bytes) return text;
let low = 0;
let high = text.length;
while (low < high) {
const mid = Math.floor((low + high) / 2);
if (utf8ByteLength(text.slice(mid)) > bytes) low = mid + 1;
else high = mid;
}
return text.slice(low);
}
export type { TruncationResult };

View file

@ -0,0 +1,211 @@
# Worked Example: `bash` End to End
One tool through every layer. `execute` is simplified — timeout validation,
cancellation and exit-code branches are elided. Everything on the output path is
shown.
Layers: exec env → `ToolOutput` sink → harness events → durable storage → lane
state → facet → wire → consumer.
---
## 1. The tool
```ts
// packages/agent/src/harness/tools/bash.ts
export interface BashToolDetails { spillPath?: string; truncation?: ShellOutputTruncation }
export function createBashTool(): AgentHarnessTool<ExecutionToolContext, typeof bashSchema, BashToolDetails> {
return {
name: "bash",
parameters: bashSchema,
output: { retain: "tail", maxBytes: DEFAULT_MAX_BYTES, maxLines: DEFAULT_MAX_LINES },
async execute(_id, { command, timeout }, signal, out, context) {
const env = context.env;
let view: ShellOutputView | undefined;
const result = getOrThrow(await env.exec(command, {
cwd: env.cwd,
inheritEnv: true,
timeout,
capture: { limits: this.output, spill: true },
onUpdate: (u) => {
view = applyShellOutputUpdate(view, u);
if (u.kind === "append") out.write(u.text);
else out.replace(view.text);
out.details.truncation = view.truncation;
if (view.spillPath) out.details.spillPath = view.spillPath;
},
}, context));
if (result.spillPath) out.details.spillPath = result.spillPath;
if (result.truncation.truncated) out.write(`\n\n[${describe(result.truncation)}]`);
if (result.exitCode) throw new Error(`Command exited with code ${result.exitCode}`);
},
};
}
```
What is gone from today's implementation: the rolling `tailOutput` buffer,
`truncateTail`, `ensureFullOutputFile`, `createTempFile`,
`BASH_UPDATE_THROTTLE_MS`, `BASH_CHECKPOINT_INTERVAL_MS`, `updateDirty`,
`lastCheckpoint`, `scheduleOutputUpdate`, `emitOutputUpdate`,
`clearUpdateTimer`. About 40 lines of tool-local machinery, replaced by a capture
policy every tool gets.
`execute` returns `void`. `details` is one field, set once — and note the tool no
longer computes `truncation`, because the env owns the window and the tool no
longer knows what was dropped.
## 2. The exec env caps at the source
`env.exec` applies `ShellOutputLimits` where the bytes originate. For a sandbox host
that means `cat 1gb.txt` never ships 1 GB to the agent machine, and the spill
lands where the model's own `read` and `grep` run. See [`execenv.md`](../03-execenv/execenv.md).
The initial state is a bounded `replace`. Growth emits `append`; a moving tail emits `slide { drop, text }`; complete turnover falls back to a bounded `replace`; metadata moves totals and spill paths without resending text. The adaptive publisher keeps small slides responsive and spaces cap-sized turnovers according to their encoded size.
## 3. The sink
`ToolOutput` folds those into `ToolOutputState`:
```ts
{
content: [{ type: "text", text: "…the retained window…" }],
details: { spillPath: "/tmp/pi-session-x/bash-8f2.log" },
usage: undefined,
addedTools: undefined,
terminate: false,
truncation: { truncated: true, truncatedBy: "bytes", totalLines: 8123, totalBytes: 262144 },
}
```
`out.write(text)` appends to `content[0].text`; `out.replace(text)` assigns
the whole retained view when the env sends a `snapshot`, because an append cannot
express eviction. The tracker (`delta.md`) records intent either way: an append is
an `a`, and a window slide assigned wholesale becomes `t` + `a` via verified
overlap detection.
## 4. Ops
A 256 KB build over ~20 flushes, 50 KB window:
The first batch is a base batch — it begins with `r`, carrying the initial state. Everything
after is deltas. The repeated content path is interned on its second use, and
consecutive ops on it omit the id entirely:
```jsonc
[["r",{"content":[{"type":"text","text":""}],"details":{},"terminate":false,"truncation":{…}}]]
[["a",["content",0,"text"],"make: Entering directory …\n"]]
[["#",0,["content",0,"text"]],["a",0,"cc -c src/a.c …\n"]]
[["a",0,"cc -c src/b.c …\n"]]
…
[["s",["details","spillPath"],"/tmp/pi-session-x/bash-8f2.log"]]
…
[["t",0,4096],["a","cc -c src/z.c\n"],["s",["truncation","totalBytes"],262144]]
```
Details are one `s`, in one flush out of twenty. The `t` + `a` pair is the window slide. The current tracker recovers it by verified overlap; production remeasurement shows the generic path is already negligible, so no explicit append/truncate producer API is added ([decision](../01-delta/append-decision.md)).
## 5. Harness events
```ts
{ type: "tool_start", toolCallId: "call_7", toolName: "bash",
args: { command: "make -j8" } }
{ type: "tool_update", toolCallId: "call_7", ops: [ … ] }
{ type: "tool_end", toolCallId: "call_7", isError: false }
```
`tool_start` is identity only — no `caps`, no `initial`. The first batch is a base
batch, so the initial state arrives on the update channel, and the caps are
already inside it as `truncation.maxBytes` / `maxLines`.
`tool_end` carries no content and no details. Every byte already went out;
re-sending would duplicate any images.
## 6. Durable storage
`pendingToolOutput(operationId, "call_7")`, **ephemeral-scoped** to the operation,
so it lives in `<session>.op_….jsonl` and is retired on settle rather than
persisting in the main log forever. Retirement is a main-log `retireScope` record,
so it commits atomically with the settle writes; the unlink is a consequence of
replaying that record, not part of the transaction ([scopes.md §5](../02-scopes/scopes.md)).
A `list<WireOp[]>`, one encoded batch appended per flush, base batches tagged `"base"` so recovery reads backwards with `stopAtTag` and stops there ([scopes.md §11](../02-scopes/scopes.md#11-list-tags-and-stop-conditions)). The tracker emits structural ops; the producer calls `rebase()` periodically to write a capped root replacement and bound recovery replay. The durable interval is a sink policy. Shell capture controls only source-state and transport publication; memo, terminal, and recovery-base flushes are forced by `ToolOutput`.
Recovery seeds a fresh `ToolOutput` from that state — it does **not** delete it,
which is what `clearReplayCheckpoint` does today and is a bug
(`harness-tools.md` §7.4). If bash memoised "spilled to /tmp/…" and crashed, a
replay that discarded the state would create a second spill file and lose the
first.
The memo write and this checkpoint commit in the same transaction. Both are
ephemeral-scoped, so both land in the same sidecar — and the type system rejects a
commit that mixes scopes, so this cannot regress silently.
## 7. Lane state and the facet
```ts
export function reduceLaneSnapshot(view: LaneView, event: HarnessEvent): void {
switch (event.type) {
case "tool_start":
view.operation.tools.push({ id: event.toolCallId, name: event.toolName,
args: event.args, output: undefined });
return;
case "tool_update": {
const tool = view.operation.tools.find((t) => t.id === event.toolCallId);
if (tool === undefined) return; // host will send a base batch
tool.output = apply(tool.output, event.ops);
return;
}
case "tool_end": {
const i = view.operation.tools.findIndex(t => t.id === event.toolCallId);
if (i >= 0) view.operation.tools.splice(i, 1);
return;
}
}
}
```
Plain mutation, no Immer, no return value. An event for a tool the view has never
seen simply does nothing; the host sends a `replace`.
A lane facet runs this same function under the tracker, so the facet's own ops
fall out against **its** shape — which need not be `LaneView` and generally is
not. The facet never writes an op.
## 8. Wire and consumer
```jsonc
{ "seq": 0, "ops": [["r",{"transcript":[],"operation":null}]] }
{ "seq": 1, "ops": [ … ] }
```
One shape: a batch of ops. The first is a base batch — it begins with `r` —
and everything after is deltas. A gap, a reconnect, a provider
reload, or a fold that could not apply all take one path: send a fresh `replace`.
The consumer is `apply` — six verbs, no domain knowledge, no library, no tool
code, against a plain mutable object it owns.
## 9. What this run costs
256 KB of output, ~20 flushes, 50 KB window:
| | today | this design |
|---|---|---|
| durable writes | full `AgentToolResult` per checkpoint | structural ops plus explicit periodic capped base batches |
| durable location | main log, forever | sidecar, unlinked on settle |
| wire per flush | whole snapshot | one `truncate` + one `append` |
| details written | rebuilt whole, every flush | one `set`, once |
| truncation logic | in `bash.ts` | in the exec env, shared by all tools |
| spill location | `/tmp`, OS-cleaned | exec env, session-scoped |
The details row is the one worth dwelling on. Nothing about bash's details
changed — they were always small. What changed is that they stopped riding inside
a container replaced wholesale on every update, which is why no tool anywhere
currently bothers to mutate them incrementally.

View file

@ -0,0 +1,462 @@
# Tool Output and Progress
> **Scope:** harness-local. Nothing here depends on the facet system, RPC, or any
> presentation. Depends on [delta tracking](../01-delta/delta.md) (the landed Chord op vocabulary and tracker), [execution environments](../03-execenv/execenv.md) (where truncation happens), and [scoped storage](../02-scopes/scopes.md) (durability).
## 1. The problem
Three faults, one cause.
**Tools each implement their own truncation.** `bash.ts` owns a rolling buffer,
`truncateTail`, a spill file, an update throttle and a checkpoint interval.
`read.ts` has its own truncation. Every future tool producing bulk output
reimplements this, differently.
**Progress is a whole value.** `onUpdate(partialResult)` hands over a complete
`AgentToolResult` on every update, `tool_update` carries that whole result, and
`openToolProgress` persists it with `setValue`.
**`details: unknown` was believed to force this** — you cannot append to a value
whose shape you do not know. That premise is now false. A structural tracker
(`delta.md`) records ops against JSON without knowing the type, so details need
no special handling at all.
## 2. `ToolOutput`
The harness constructs a sink per invocation and passes it to `execute`. **Tools
return nothing**; the sink holds the result.
```ts
interface ToolOutput<TDetails extends JsonValue> {
/** Append to the text block. */
write(text: string): void;
/** Append an image block. Images are never windowed. */
image(image: ImageContent): void;
/** Replace the retained text wholesale. Chord still recovers a verified `t` + `a` slide when possible. */
replace(text: string): void;
/** Apply source-owned truncation totals and spill metadata without resending text. */
capture(metadata: ShellOutputMetadata): void;
/** The tool's own details object. Mutate it directly. */
readonly details: TDetails;
/** Accumulates. A subagent making several model calls adds to it. */
usage(usage: Usage): void;
/** Replaces. */
addTools(names: string[]): void;
/** Replaces — a tool that decides to terminate and then recovers can say so. */
terminate(value: boolean): void;
}
```
```ts
execute(
toolCallId: string,
params: Static<TParameters>,
signal: AbortSignal,
out: ToolOutput<TDetails>,
context: Context,
): Promise<void>;
```
`execute` returns `void`. Everything the tool produces flows through the sink —
including `usage` and `addTools`, which cannot be settle-time return values
because a replayed tool must be able to seed from durable state (§7.4).
**Failure is a thrown error.** `isError` is set by the harness when `execute`
rejects. A tool may throw anything, including errors from libraries it did not
write.
**`terminate` is orthogonal to how execution ended**, so a failing tool can ask
the loop to stop:
```ts
try { await thing(); } catch (error) { out.terminate(true); throw error; }
```
This closes a gap in the current implementation, where `executeToolCall`'s catch
hardcodes `isError: true` with no terminate, and `immediateError`'s terminate
parameter is only ever passed `true` by `applyBeforeToolDecision` for a hook
block.
### 2.1 Details are just an object
```ts
async execute(id, params, signal, out, context) {
out.details.total = 42;
out.details.passed += 1;
out.details.failures.push({ name, message });
out.details.current = undefined; // -> delete
}
```
Fully granular ops fall out, verified against the prototype:
```jsonc
["#",0,["details","passed"]]
["s",0,1]
["p",["details","failures"],0,0,[{…}]]
["d",["details","current"]]
```
No recipes, no mutation map, no `initialDetails` on `tool_start`, no Immer, and
no consumer ever runs tool code. `TDetails` remains the tool's own exported type
so a renderer can cast `call.details` to it — that is all it is for.
A survey of `packages/agent/src/harness/tools/` found **no tool mutating details
incrementally today**; only `bash.ts` writes them mid-stream and it rebuilds them
whole, because the container was replaced wholesale anyway. This design removes
that cause, and costs nothing if nobody takes it up.
**Caveat:** details are now unbounded. A tool pushing to `failures` in a loop
grows without limit, and unlike text there is no cap. Nothing shipped does this;
the door is open in a way it was not when details were a replace-only value.
### 2.2 Partial output survives failure
Today `executeToolCall` catches and returns `createErrorToolResult(message)`,
building a fresh result from the error string alone — a tool that streamed 8 KB
and then threw reports only the error.
With the sink, what was written is already the result. The harness appends the
error text as content and keeps the rest. The error text is model-visible
content, not presentation: the model needs to read why the call failed.
`abortedMessage` and `interruptedMessage` should be made consistent with this;
today they build replacement results via `syntheticMessage`, so a cancelled
long-running command loses its partial output too.
## 3. Content
Content is exactly **one text block followed by zero or more image blocks**. A
tool cannot interleave text between images — text written before and after an
image lands in the same block. Deliberate: truncation only ever touches a string,
and an image is never partially anything.
Images are **never windowed**. A byte or line cap over base64 payloads is
meaningless and `truncateTail` operates on text. `maxBytes` / `maxLines` govern
text only.
### 3.1 Retention mode
Declared on the tool definition:
```ts
output?: {
retain?: "head" | "tail"; // default "tail"
maxBytes?: number;
maxLines?: number;
}
```
**`head`** — append until the cap, then stop. Nothing is ever removed. Right for
output meaningful from the start: file reads, listings, greps.
**`tail`** — a rolling window. Right for anything whose interesting part is at
the end: builds, test runs.
Head+tail is **not** offered. `truncate.ts` exports `truncateHead` and
`truncateTail`; neither combines them and none is being added.
### 3.2 The tool does not truncate — the exec env does
For output originating in the execution environment, capping, coalescing and
spilling happen **at the source**. See [`execenv.md`](../03-execenv/execenv.md). The tool passes its
`ShellOutputLimits` into `env.exec` and pipes the resulting updates into the sink.
This is not a convenience. Reading a 1 GB file on a sandbox host must not ship
1 GB to the agent machine to be capped there, and the spill file must land where
the model's own `read` and `grep` run.
For output originating on the agent machine (subagents, in-process work),
`ToolOutput` applies the same logic locally. Same code, different location.
Note this reverses an earlier decision that there is no spill in the sink and a
temp-file path is tool-specific. The exec-env argument — model reachability
across a machine boundary — is what changed it.
## 4. Tool output state
`tool_update` carries `Op[]` (`delta.md` §6) targeting the invocation's
`ToolOutputState`:
```ts
interface ToolOutputState {
content: (TextContent | ImageContent)[];
details: JsonValue;
usage?: Usage;
addedTools?: string[];
terminate: boolean;
truncation: ShellOutputTruncation; // totals over everything ever written, without duplicate text
}
```
Ops rather than a typed variant union, for one decisive reason: **only ops give
details granularity without the harness knowing `TDetails`**. A typed union would
need per-tool recipes, which is the machinery §2.1 deletes.
Text still gets delta treatment, because the sink applies the cap *before*
mutating, so a rolling window produces `truncate` + `append` on
`content[0].text` rather than a whole-value set.
Interned ops also measure **smaller than a typed frame vocabulary** on every
workload, and 10x smaller on details, because a frame repeats `toolCallId` where
an interned op carries one integer. See `delta.md` §4.1.
There is no `drop` event. The sink knows what it evicted and expresses it as
`truncate`; a consumer needs no separate signal and derives nothing.
## 5. Harness events
```ts
| { type: "tool_start";
runId; turnId; toolCallId; toolName;
args: unknown }
| { type: "tool_update";
runId; turnId; toolCallId;
ops: Op[] } // a base batch begins with `r`
| { type: "tool_end";
runId; turnId; toolCallId;
isError: boolean }
```
**`tool_start` carries identity only.** An earlier draft added `caps` and
`initial`; both were redundant and are gone. The first batch is always a base batch
(`delta.md` §6), so the initial state arrives on the update channel — carrying it
twice means two ways to establish the base, which can disagree. And `caps` are
already inside the state: `ToolOutputState.truncation` carries `maxBytes` and
`maxLines`, which is what a renderer needs to say "50 KB limit". The draft itself
conceded a consumer "no longer has to" fold identically, since the producer's ops
encode eviction — that was the last reason for `caps` to travel, and it does not
hold.
`tool_update` carries ops. A base batch travels the same channel as a delta,
because a replacement is itself an op (`delta.md` §2) — there is no second shape. `message_update` has the identical shape
(`message-update.md` §5.1); a consumer folds tool output and assistant output with
one code path.
**`tool_end` carries no content, no details, no usage, no terminate.** Every byte
already went out. Re-sending would duplicate each base64 image at the moment
nothing new has happened.
> **The fold is the result.** Nothing a consumer folded is ever re-sent to
> confirm it.
This dissolves an earlier open question about settle-time truncation disagreeing
with the running fold. There is no separate settle-time truncation: the sink's
window *is* the truncation. Anything a tool wants to add at the end — bash's
`[Showing lines 8000-8123 of 8123]` footer — is `out.write(footer)`, one more
append.
The harness still assembles `AgentToolResult` in process for the model, and
`createToolResultMessage` still writes the settled `ToolResultMessage` to the
transcript with `content`, `details`, `usage`, `isError`.
Neither is a wire event.
## 6. Lane reduction
```ts
export function reduceLaneSnapshot(view: LaneView, event: HarnessEvent): void;
```
Plain mutation on a plain object. No `Draft`, no `produce`, no Immer, and **no
`Rebase` return value** — a fold that cannot apply an event leaves state alone,
and the host sends a `replace`. (An earlier draft had `void | Rebase`; under
Immer that throws, and without Immer there is nothing to return to. Note
`reducer.ts:4` currently defines `LaneSnapshotReduction = LaneSnapshot | { rebase: true }`,
so this is a real change to existing code, and `navigation_end` is the event that
drives it.)
The event is the only input besides the view. No registry, no `resolve`, no tool
code — so a tool rewritten between crash and resume cannot make a persisted
stream unreadable.
When the harness is wrapped by a facet, the same mutation runs under the tracker
from `delta.md` and ops fall out. The harness itself does not know this.
## 7. Durability
### 7.1 What exists today
`pendingToolOutput(operationId, invocationId)` is a
`value<AgentToolResult<unknown>>`, and — importantly — `progress.write(partial)`
fires **only** when `options?.checkpoint === true` (`drive/tools.ts:325`). Not on
every update. bash checkpoints every 2 s with a `JSON.stringify` dedupe; every
other tool never checkpoints at all.
It is also **not a progress buffer**. It is the interruption checkpoint. On
resume, if the tool is not replay-safe, `readCheckpoint` turns it into a real
`ToolResultMessage` for the model: `[...checkpoint.content, INTERRUPTION_MARKER]`
plus `details` and `usage`.
Note the implication that misled an earlier draft: because the checkpoint must
*be* the current state, it was read as needing `value` semantics. It does not. It
needs to be *derivable*, and folding encoded batches from the last base batch derives it — which is the point of tagging base batches (§7.3). `checkpoint: true`
requests a durable write, not a replacement.
`pendingAssistantFrames` is a `list<AssistantMessageFrame>` appended per frame,
`deleteList` on settle in `response.ts:345`, `deferred.ts:157`, `terminal.ts:47`.
Note `operationCleanupWrites` (`terminal.ts:26`) does four `scanValues` calls to
enumerate what to delete at settle. Under scopes, the two covering
`operationToolMemoPrefix` and `pendingToolOutputPrefix` are replaced by a single
`retireScope(operationId)`.
### 7.2 Renaming
`pendingAssistantFrames` → **`pendingAssistantOutput`**, matching
`pendingToolOutput`. Frames stop being the durable unit; both addresses now hold
tracked state written as ops or snapshots.
### 7.3 What to write, and when
Both addresses are **ephemeral-scoped** ([scopes.md](../02-scopes/scopes.md)), so they live in a
sidecar retired on settle rather than in the main log forever.
Two independent wins, in order of importance:
- **Encoding.** Ops instead of whole values, plus address interning: 93.89 MB to
5.32 MB in a single file, no change to atomicity. Do this first.
- **Scopes.** Pending state leaves the main log entirely: 5.32 MB to 0.06 MB
surviving.
**Both are `list<WireOp[]>`**, not values. The sink owns one stateful Chord encoder/decoder pair per durable value stream. One encoded batch is appended per flush, and a logical batch whose first op is `r` is tagged `"base"` on the storage record.
Recovery reads backwards with `stopAtTag: "base"` and applies forward
([delta.md §9](../01-delta/delta.md#9-durable-form), [scopes.md §11](../02-scopes/scopes.md#11-list-tags-and-stop-conditions)).
Writing one flush:
```ts
const ops = out.flush();
if (ops.length === 0) return;
const wire = enc.encode(ops);
writes: [appendList(address, wire, isBase(ops) ? "base" : undefined)];
```
`isBase` comes from Chord. It checks for the root replacement op `r`; ordinary nested sets use `s`. Classification lives beside the vocabulary so the comparison is written once.
The landed tracker emits structural ops unconditionally. There is no serialized-size comparison or adaptive replacement heuristic. A producer requests a base batch explicitly with `rebase()`; the output sink's cap bounds that replacement, while periodic rebasing bounds recovery work. Production remeasurement rejects a text-specific append/truncate API: the generic path measured 2.43–2.46 µs per 50 KB rolling-window flush locally, below surrounding costs. Keep ordinary tracked string mutation; see the [decision record](../01-delta/append-decision.md).
**Checkpointing is not deleted.** `BASH_CHECKPOINT_INTERVAL_MS` remains only as an interim compatibility mechanism. The generic sink owns durable frequency because Shell cannot price a storage write or enforce memo/output atomicity. Forced memo, terminal, and recovery-base writes bypass ordinary pacing while remaining cap-bounded.
### 7.4 Replay must seed, not discard
`clearReplayCheckpoint` currently writes `deleteValue(pendingToolOutput(...))`
before re-executing a replay-safe tool (`drive/tools.ts:257`). **This is a bug.**
Replay-safe means the tool is re-executed, but memos exist precisely so it does
*not* redo work it already did — and skipped work emits nothing. Any output for
memoised work is lost today.
The fix: seed a fresh `ToolOutput` from the durable state, then re-execute. The
tool appends to a sink that already holds what it produced before the crash.
This is also why nothing may be settle-only. `usage` and `addTools` must survive
a seed, so they flow through the sink like everything else.
### 7.5 The memo invariant
> A tool's memo write and its output checkpoint must commit in the same
> transaction.
Otherwise a tool does work, sets a memo, crashes before the next checkpoint, and
on replay skips the work while the seeded output has no record of it.
**This does not hold today.** `setMemo` (`drive/tools.ts:112`) and `openProgress`
(`runtime/progress.ts:44`) are two separate `lane.command` calls, hence two
transactions. Making it hold means bundling the checkpoint into the memo's commit:
```ts
setMemo(name, value) {
validateMemoName(name);
if (!active) return Promise.reject(ended());
return lane.command<void>((state) => {
if (!ownsEffect(state)) return { kind: "reject", error: ended() };
const memo = operationToolMemo(drive.operationId, call.resultEntryId, name);
return {
kind: "commit",
writes: [
value === undefined ? deleteValue(memo) : setValue(memo, value),
setValue(pendingToolOutput(drive.operationId, call.resultEntryId), out.snapshot()),
],
next: state,
materialize: () => undefined,
};
}, drive.context);
}
```
Both addresses are **ephemeral-scoped**, so this is a single-file transaction and
statically enforced as one ([scopes.md §3 and §6](../02-scopes/scopes.md)). `operationToolMemo` is
placed in that scope precisely for this. Ordering on the Session line would not be
enough — two file writes are not atomic. The periodic checkpoint stays as it
is — best-effort, for the interruption path; this forces one where correctness
requires it.
The invariant holds because the tool does X, writes X's output to the sink, *then*
calls `setMemo("did X")` — so the sink's state at commit time already contains X's
output.
**Ordering alone does not work**, in case it looks tempting:
- memo first, checkpoint second → crash between → replay skips X and the seeded
output lacks it → silent loss;
- checkpoint first, memo second → crash between → replay redoes X and appends
again → duplicated output.
Duplication is the less bad failure, so ordered writes are a tolerable fallback,
but neither is correct.
**Cost:** `setMemo` now writes a full capped output state rather than a small
value. Memos are rare — a handful per invocation — so this is bounded by
`memo count x cap`, not by output volume.
### 7.6 `openProgress` has a write-ordering race
`commitWrite(item)` captures `item` when `write()` is called, and the write is
fire-and-forget with only `latest` tracked. A checkpoint captured at T1 can
therefore commit *after* a memo-bundled checkpoint at T2, overwriting newer state
with older — reintroducing exactly the loss §7.5 prevents.
Fix: resolve the sink's state inside the command planner rather than at call time.
```ts
commitWrite: () => setValue(address, out.snapshot()) // evaluated under the Session line
```
`lane.command` serializes on the Session line, so checkpoint writes become
monotonic by construction. This removes the race generally, not only against
memos.
### 7.7 Nothing on the durable path runs tool code
Ops are interpreted by a six-verb applier with no domain knowledge, so a tool
rewritten between crash and resume cannot make a persisted stream unreadable.
> **The durable path uses only harness-owned reducers.**
This also rules out persisting *facet* ops. The harness has no facet state,
facets come and go, and a recovering harness must rebuild its working values with
no facet present.
## 8. Open questions
- **Property tests for the tracker** (`delta.md` §3.3). Everything here rests on
producer and replica agreeing; nothing currently proves they do.
- Coalescing window: per tick, or a byte/time threshold.
- Whether image count needs a bound. Images are unwindowed, so a tool pushing
them in a loop grows `content` without limit. Currently treated as a tool bug.
- Whether details need a bound for the same reason (§2.1).
- Whether `retain: "head"` should keep emitting counter-only updates once capped
so a renderer can report how much was suppressed. [`execenv.md`](../03-execenv/execenv.md) answers this
for exec-originated output; agent-side output needs the same answer.
- Whether a failing tool *should* be able to terminate, or whether the current
inability is deliberate — there is a reasonable argument the model should
receive the error and decide.
- **Derived values.** `arguments` parsed from accumulated JSON should not be
replicated; derive it on demand. Safe because `parseStreamingJson` is total —
four fallbacks ending in `{}`, it cannot throw — so a replica derives with no
error path and no agreement protocol. Generalises to: no derived fields in
replicated state.

View file

@ -0,0 +1,125 @@
# Bounded output publication
**Status:** the shared adaptive publisher and its `ExecutionEnv` use are implemented. Generic `ToolOutput` integration remains design work.
Depends on landed Chord delta tracking, source-bounded execution output, and scoped storage for durable batches.
## 1. Invariant
> The durable record, what the model sees, and what the UI shows are the same bounded view.
A spill file is not a second view. It is a file inside the execution environment that the model reaches through ordinary file tools.
Every uncontrolled producer boundary needs two independent bounds:
- **state size:** the latest retained text is capped;
- **publication:** both encoded bytes and event count are paced.
Delta encoding is complementary. It compresses a published change; it does not cap state or decide when publication occurs.
## 2. Boundaries
```text
remote process
-> optional ExecutionEnv publisher
-> worker ToolOutput publisher
-> events + durability + replication
```
A publisher instance is required only before a real costly boundary:
- a physically remote execution environment limits output before transport;
- `ToolOutput` limits custom tools and downstream event/storage traffic;
- a colocated environment can feed its bounded updates in process without another serialized transport;
- custom tools bypass `ExecutionEnv` but cannot bypass `ToolOutput`.
The control algorithm is shared. Payload vocabularies differ: Shell uses replace/append/slide/metadata; `ToolOutput` uses Chord operations.
## 3. Why size or cadence alone is insufficient
A fixed 50 KB snapshot is safe per event but not over time. At 100 ms it permits ten complete snapshots per second, approximately 500 KB/s plus envelopes.
A byte budget alone also permits excessive tiny events and durable transactions. A minimum interval bounds count; encoded-size debt bounds bandwidth.
The original handoff incorrectly treated `intervalMs = 100` as 100 emits/s. It is ten emits/s. The fixed interval was still non-adaptive: it delayed small trickles while allowing complete windows at the same frequency.
## 4. Landed adaptive algorithm
`packages/agent/src/harness/utils/adaptive-publisher.ts` implements:
```ts
nextDelayMs = max(globalMinEmitInterval, encodedUpdateBytes * 1000 / globalTargetBytesPerSecond);
```
Current harness-global policy:
```ts
minEmitInterval = 100 ms;
targetBytesPerSecond = 100 KB/s;
```
Behavior:
1. The first dirty state after idle publishes immediately.
2. Writes before the next deadline collapse into the latest state.
3. One trailing timer publishes held state after the deadline.
4. Completion and correctness boundaries force one bounded publication.
5. The publisher commits its baseline before consumer delivery, preventing duplicate deltas if a consumer applies and then throws.
This is an amortized token bucket with a cap-sized burst. A leading or forced terminal update may exceed the target over a short interval, but sustained encoded bytes converge to the target and sustained event count cannot exceed the minimum-interval floor except for explicit forced correctness writes.
## 5. Scenario traces
Assume a 50 KB cap, 100 KB/s target, and 100 ms floor.
### Below cap, completes
The initial state publishes immediately. Small appends publish no faster than the floor, and final dirty state is forced. Total encoded text is approximately the produced text.
### Below cap, trickling
Writes arriving more than 100 ms apart publish immediately because the preceding deadline has already passed. Faster writes collapse into one append per floor interval.
### Above cap, full force
The retained state never exceeds 50 KB. Complete window turnovers encode as bounded replacements. A roughly 50 KB update buys about 500 ms of silence, yielding approximately two updates and 100 KB per second regardless of raw producer throughput.
For shell execution, the complete stream goes to a source-local spill under backpressure rather than over the output-update channel.
### Above cap, trickling
A small tail movement encodes as truncate plus append (Chord) or `slide` (Shell). Its encoded size is small, so the 100 ms floor dominates and output remains responsive. Resending a complete 50 KB snapshot for each tiny slide would be both slower and larger.
### Burst then silence
The leading state is immediate. Held writes collapse, and one trailing timer publishes the latest residue. There is no polling timer.
### Huge single write
The producer may already have allocated its input, but the boundary retains and publishes only the configured cap. Shell spill keeps the complete source stream. Arbitrary custom-tool images and structured details still require separate limits.
## 6. Forced writes
These bypass pacing once, while remaining state-size bounded:
- command/tool completion;
- error or abort;
- a memo and output checkpoint committed atomically;
- an explicit recovery base/rebase.
A terminal flush cancels the trailing timer before `tool_end`, preventing a late update for a settled invocation.
## 7. `ToolOutput` application
The sink will own one Chord tracker and encoder per invocation. Mutations remain local while publication is blocked; flush-time dirty tracking means held writes collapse without retaining one op per write.
One sink flush feeds the live event and current model-visible state. Durable batches use the same logical flush, encoded as per-stream `WireOp[]`. Periodic `rebase()` bounds recovery replay; a memo correctness flush writes a tagged base batch in the same transaction as the memo.
Durable cadence belongs to the sink, not to Shell or bash. Bash's existing two-second checkpoint request remains only as an interim compatibility mechanism until the sink migration.
## 8. Remaining decisions
- Explicit byte/count rejection policy for images.
- Bounds for structured details; arbitrary JSON cannot be meaningfully tail-windowed.
- Whether ordinary durable writes ride every live sink publication initially or use a slower measured cadence. Memo and terminal correctness flushes are not optional.
- Exact global production values after profiling real remote transport and storage.

View file

@ -0,0 +1,227 @@
# `message_update` Write Amplification
> **Scope:** harness-local. Depends on [delta tracking](../01-delta/delta.md) for the landed Chord `Op`/`WireOp` vocabulary and [scoped storage](../02-scopes/scopes.md) for durability. Chord delta tracking has landed; scoped storage and this Harness integration have not.
## 1. The problem
```ts
{ type: "message_update", runId, message, event, frame? }
```
Three representations of the same thing travel together:
- `message` — the full `AssistantMessage`;
- `event` — an `AssistantMessageEvent`, which itself carries `partial: AssistantMessage`,
a second full copy;
- `frame` — the actual delta, optional.
Per streamed token this is roughly two complete snapshots plus a delta, so bytes grow
quadratically over a response.
The reducer does not even use the delta:
```ts
case "message_update":
if (next.operation?.id === event.runId && event.message.role === "assistant") {
next.operation.streamingMessage = event.message;
}
```
A straight assignment. So on any wire, sending the event is *worse* than sending a
snapshot — it ships two where `replace` ships one.
The wire adapter is already halfway to the fix: it drops `event` and sends `message`
plus `frame`. That is the full snapshot together with the delta that produced it.
## 2. The compact form already exists
```ts
/**
* Compact, replayable assistant-message progress. Terminal settlement is
* intentionally excluded and must be persisted separately.
*/
export type AssistantMessageFrame =
| { type: "start"; partial: AssistantMessage }
| { type: "text_start" | "text_delta" | "text_end"; contentIndex: number; ... }
| { type: "thinking_start" | "thinking_delta" | "thinking_end"; ... }
| { type: "toolcall_start" | "toolcall_checkpoint" | "toolcall_delta" | "toolcall_end"; ... };
```
`AssistantMessageFrameEncoder` produces it, `reduceAssistantMessageFrames` folds it,
and `openFrameProgress` already persists it with `appendList`. The delta format is
built, used, and durable. It simply is not what `message_update` carries.
## 3. Precedent
pi already does wire-is-delta one layer down. `PiMessagesEvent` — the serialized form
a pi-messages backend sends — has no `partial`:
```ts
| { type: "text_delta"; contentIndex: number; delta: string }
| { type: "text_end"; contentIndex: number; content: string; contentSignature?: string }
```
`pi-messages.ts` then rehydrates: it holds a local `partial`, mutates it per event
(`partial.content[i].text += event.delta`), and returns the in-process
`AssistantMessageEvent` with `partial` attached.
So the convention is established. It just stops at the provider edge instead of
continuing through the harness.
## 4. The change
```ts
| { type: "message_update"; runId: string; entryId: string; frame: AssistantMessageFrame }
```
`message` and `event` are removed; `frame` stops being optional.
The blast radius is small because `HarnessEvent` is not what most streaming consumers
read. `AgentEvent` (`agent-loop.ts`) and `AgentSessionEvent` (`agent-session.ts`) are
separate unions that happen to share the tag name and build their own
`message_update` from the pi-ai event directly. They are out of scope here.
Real consumers and producers of `HarnessEvent.message_update`:
| site | change |
| --- | --- |
| `runtime/drive/response.ts` | emit the required semantic frame; stop attaching full snapshots |
| `runtime/reducer.ts` | fold the frame instead of assigning `event.message` |
| lane/facet state adapter | run that fold under the Chord tracker and emit `Op[]`/encoded `WireOp[]` |
| `experimental/harness-wire-adapter.ts` | stop treating raw `HarnessEvent` as the final replication format |
| `harness/telemetry.ts` | name list only |
| `protocol/harness.ts` | replication carries encoded `WireOp[]`, not the raw event |
`message_end` continues to carry the settled message, because frames deliberately
exclude terminal settlement. That is once per message, not once per token.
## 5. An incremental applier is needed
`reduceAssistantMessageFrames` is a whole-stream fold over an `Iterable`. The
reducer needs a step function:
```ts
export function applyAssistantMessageFrame(
state: AssistantFrameState,
frame: AssistantMessageFrame,
): void;
```
Plain mutation on a plain object. **No `Draft`, no Immer.** An earlier draft
argued for a draft-mutating signature so it would compose inside a `produce`
recipe; that motivation is gone ([delta.md §8](../01-delta/delta.md#8-what-this-removes-from-the-codebase)). The step function is still
needed — the whole-stream version becomes a loop over it — just for the simpler
reason that the reducer folds one frame at a time.
### 5.1 pi-ai frames stay at the pi-ai boundary; Chord ops cross replication boundaries
`AssistantMessageFrame` is pi-ai's semantic delta vocabulary (`text_delta`, `text_end`, …). [Delta tracking §6](../01-delta/delta.md#6-there-is-no-frame-type) defines no second frame wrapper: in-process replication carries `Op[]`, and a wire adapter carries encoded `WireOp[]`. Pi-ai frames stop at the fold; Chord ops cross the replication boundary.
`AssistantMessageFrame` is pi-ai's own delta vocabulary and stays. What changes is
that it is no longer the durable unit or the replication unit.
The harness folds frames into `LaneView` by plain mutation. Under the Chord tracker that yields:
```json
["a",["operation","streamingMessage","content",0,"text"],"Let me "]
```
Measured, interned ops are **smaller than frames** on this workload — 13.6 KB
against 21.5 KB raw for 200 deltas — because a frame carries three keys of its own
where an interned op carries one integer. So the size argument for putting frames
on the wire is dead.
What frames keep is semantic: `text_end` carries authoritative content plus a
signature in one atomic unit, where ops would need two with a weaker contract
about their relationship. That is why they remain the *input* vocabulary and are
folded before anything crosses a boundary.
### 5.2 Reducer state must live in the reduced value
Text and thinking fold purely: `block.text += frame.delta`, and `*_end` overwrites
with authoritative content plus signature. The `ended` flag in `ReducerBlockState`
is used only for validation and can be dropped.
Tool calls cannot. `toolcall_delta` does `state.json += frame.delta`, accumulating
a **raw JSON string that is never stored on the message** — the block holds
`arguments`, the parsed value. You cannot recover the accumulator from the
snapshot, and you cannot append a delta to a parsed object.
So the accumulator has to become part of the value being folded — e.g.
`operation.frameState[contentIndex].json` on `LaneView` — leaving
`AssistantMessage` clean. Generally:
> **A replicated reducer's state must be part of the replicated value.** Anything
> held beside it diverges on any consumer that did not run the producer's fold.
The corollary is that `arguments` itself should **not** be replicated. It is
derived from `json`, and a parsed object is a fresh reference on every parse, so
storing both ships two copies of the same information. Derive it on demand.
This is safe precisely because `parseStreamingJson` is **total** — four fallbacks
ending in `{}`, it cannot throw — so a replica derives unconditionally with no
error path and no agreement protocol. There is no parse-failure state to
represent, and no block needs an error slot.
### 5.3 Parse cost
`parseStreamingJson` on a growing string once per delta is quadratic per message.
Since `arguments` is now derived rather than replicated, this cost falls on whoever reads it rather than on every consumer. Presentation can refresh derived arguments at semantic checkpoints and `toolcall_end` instead of parsing on every delta; that policy is separate from why the encoder currently emits a checkpoint (§6).
## 6. What `toolcall_checkpoint` is for
`EncoderBlockState` carries `caughtUp` and `catchupJson` because a queued provider event's shared `partial` may already be ahead of that event's delta. The checkpoint catches the semantic frame stream up to the authoritative tool-call arguments visible at block start; it is not currently a general late-subscriber protocol.
After frames fold into tracked state, a Chord root replacement (`r`) is the replication and durable-recovery resync point. `toolcall_checkpoint` remains semantic input to that fold rather than carrying transport responsibility.
## 7. Write volume for pending output
`openFrameProgress` calls `appendList(pendingAssistantFrames(...))`, so every
frame is a line. A long response is thousands of writes.
**The address is renamed `pendingAssistantOutput`** and becomes a `list<WireOp[]>`, matching `pendingToolOutput` ([tool-output handoff §7.2](../04-tool-output/harness-tools.md#72-renaming)), and it stops being a list of frames. The progress sink encodes tracked `Op[]` with one stateful encoder per response before appending each durable `WireOp[]` batch; explicit `rebase()` calls produce bounded root-replacement batches for recovery. The list lives in an **ephemeral scope** so it is unlinked on settle rather than persisting in the main log ([scoped storage](../02-scopes/scopes.md)).
The important property is that this list is **not history**: `deleteList` runs on
settle in `response.ts`, `deferred.ts`, and `terminal.ts`. It exists so a crash
mid-response can recover a partial assistant message. The settled message is persisted
separately.
That means per-frame durability buys almost nothing, and the write rate can be traded
away directly:
- **Coalesce with a bounded, non-resetting window.** The first pending frame opens the
window; later frames join it *without* extending the deadline, so a continuously
streaming response cannot postpone the first write indefinitely — the failure mode a
naive debounce has. Frames admitted during an active write form the next batch.
- **Concatenate on flush.** A run of `text_delta` frames for one `contentIndex`
collapses to a single frame with the joined text. The fold result is identical, so
this is lossless for our purposes.
A crash then loses at most one window of in-flight streaming.
### 7.1 Contrast with DeepSeek Harness
DSH cannot make this trade. Its `assistant/chunk` events are canonical log entries, so
per-token durability is mandatory and it attacks size instead:
- the same bounded non-resetting write-behind window;
- **packed rows** — runs of consecutive chunk deltas stored as
`text-chunks` / `reasoning-chunks` / `tool-call-chunks`, lossless and roughly 60%
smaller on a real session, with reading unconditional so layout never depends on the
write switch;
- checksummed zstd frames by default, with recovery from a torn final frame.
Their packing must reconstruct exact event boundaries, sequence numbers and timestamps,
because `seq = log.length` and validation requires a contiguous logical log. Ours does
not, because the frames are discarded at settle — so plain concatenation is available
to us and packing machinery is not needed.
## 8. Consequence for ad-hoc listeners
`HarnessEvent.message_update` stops being self-describing. A listener that attaches
mid-stream sees a delta against a partial it does not hold.
Any correct listener already has the base, because `watch()` installs the subscription
and captures the snapshot inside one `readLane` critical section, buffering until
`start()`. But a listener that calls `on("message_update")` directly, without a watch,
is no longer viable — worth knowing before committing.

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,162 @@
# Facet sandbox — isolated-vm
**Unmodified pi facet code running in a V8 isolate with no ambient authority.**
Full JIT, so ~1.8× native rather than the 8–17× a WASM interpreter costs.
```bash
npm install
npm run demo # a facet: contributions, components, commands, callbacks
npm run audit # escape audit — the interesting one
npm run bench # crossing cost, isolate cost, 300 components
npm test # 412 property assertions
```
Node 22+ (`--experimental-strip-types`). See `../facet-sandbox-poc` for the
QuickJS-WASM equivalent.
## The safety claim, and why it is structural
isolated-vm **can** be used unsafely, and that is what `facets.md` §14.2 objected
to. The footgun is handing the guest a live handle to a host object — via
`derefInto()`, or by returning a `Reference` from a host call — after which the
guest walks that object's prototype chain into the host realm.
This membrane makes that impossible **by construction, not by care**:
1. `encode()` is the only way a host value reaches the guest, and it is
`JSON.stringify` with a replacer. **Its output is a string.** A string cannot
carry a reference.
2. Host callables never cross. They become an integer id into `hostTable`, which
lives entirely host-side. The guest receives a **number**.
3. Exactly **one** `Reference` is ever given to the guest — `__invokeRef`, the
single call-in point.
4. `derefInto()` is used exactly once, on the guest's **own** global. No host
object is ever its argument.
The guest's entire view of the host is `{ number, string }`. There is no object
graph, so there is nothing to walk.
### Audit results (`npm run audit`)
```
Ambient authority:
require / process / fetch ["undefined","undefined","undefined"]
global names visible 64 globals
Classic escape ladders:
Function('return process')() undefined
constructor walk on host data undefined
The isolated-vm Reference footgun:
__invokeRef.deref() blocked: TypeError ("Cannot dereference
this from current isolate")
__invokeRef.copySync() blocked: TypeError
__invokeRef.getSync('constructor') blocked: TypeError
derefInto() is callable? inert (object)
invoke derefInto() result blocked: TypeError
host globals via derefInto() no host globals
Resource bounds:
spinning guest interrupted after 205ms
isolate usable afterwards true
```
One probe gave a **false positive** in a first draft: testing `f.deref === undefined`
to prove a returned host function is a Proxy rather than a Reference. The
membrane's proxy returns a proxy for *every* property, so `f.deref` is truthy.
That is not a leak — calling it routes to `hostFn["deref"]`, which does not exist
and throws host-side. The audit now calls it instead of checking for absence.
## Performance
| | isolated-vm | QuickJS | native |
| --- | --- | --- | --- |
| 300 markdown components, small | **70 ms** | 494 ms | ~40 ms |
| 300 markdown components, typical | **108 ms** | 1118 ms | 64 ms |
| 300 markdown components, large | **419 ms** | 4260 ms | ~150 ms |
| 146 KB bundle load | **35 ms** | 104 ms | — |
| membrane crossing | **4.5 µs** | 7–17 µs | — |
| per compartment | 1080 KB, 5.5 ms | **77 KB, 0.8 ms** | — |
**~1.8× native at typical size.** QuickJS is 8–17× and gets worse with input size.
Isolates cost **14× more memory than a QuickJS runtime** (1080 KB vs 77 KB) and
are slower to create. For a handful of facets that is irrelevant; for hundreds it
would not be.
`Intl.Segmenter` **exists** here, which is the single hard blocker for reusing
`packages/tui` components under QuickJS. That alone may decide it.
## Limitations
**Budgets do not nest.** A timeout bounds one `evalSync`. A guest function
re-entered *by the host* — a contributed callback, a component method — gets
`callGuestRef`'s own budget, not the outer one. Asserted explicitly in
`property-test.ts`; a runaway callback took 5005 ms under an outer budget of 50 ms.
Bounding total facet time needs separate accounting.
**Async is a settle-callback, not a native promise.** `applySync` is synchronous,
so a host Promise cannot cross. The host returns a token, the guest builds a real
Promise against it, and the host settles it by calling back in. Verified: two
sequential guest `await`s while the host event loop ticked 22 times.
**Native addon, and the ABI matrix is real.** `isolated-vm` ships prebuilds and
only falls back to `node-gyp rebuild` when none matches — which is why install
takes a second, not an hour. It is **not** building V8; V8 is already in the Node
binary. But coverage is narrow:
| version | engines | prebuilds |
| --- | --- | --- |
| **6.2.0** | `>=22.0.0` | linux-x64/arm64, darwin-arm64, win32-x64 — abi127, abi137 |
| 7.0.1 | `>=24.0.0` | — |
| 7.0.0 | `>=26.0.0` | — |
darwin-**x64** is absent everywhere. Installing `isolated-vm@7` on Node 22 falls
through to a source build and **fails without Python and a C++ toolchain** —
verified here. Note also that 6.2.0 is `latest` on npm despite 7.0.1 existing,
because it published one day later.
The project is **not abandonware** — 6.0.1 (Jul 2025) through 7.0.1 (Aug 2026),
actively released. §14.2's "maintenance mode" claim is out of date and should be
corrected. The real risk is different: adopting it couples your minimum Node
version to theirs, and they dropped Node 20 in 6.x and Node 22 in 7.x within a
year. Shipping SEA builds moves that from every user's install to your CI.
**Memory: references release, but lazily.** The `WeakRef` + `FinalizationRegistry`
interning table ports unchanged from the QuickJS membrane and works — 20 000
crossings went to **0 live refs**. But finalization is not prompt: 5 000 refs were
still live 200 ms after a forced GC. Under sustained churn refs accumulate faster
than the collector reclaims them, and a 32 MB isolate did hit its limit.
The consequence is an API rule, not a membrane fix: **do not create references in
a hot path.** `slots.claim(factory)` at construct is one reference forever; a
component method returning a fresh closure every frame is one per frame.
**Hitting `memoryLimit` kills the facet cleanly.** It throws a catchable
`"Isolate was disposed during execution due to memory limit"` and the host
survives — but the isolate is dead and unrecoverable, so teardown must tolerate
an already-disposed isolate. `dispose()` on it throws `"Isolate is already
disposed"`; the membrane guards this.
> **Do not call `isolate.getHeapStatisticsSync()` on a stressed isolate.** It
> **aborted the whole process** during testing — a hard crash, not an exception.
> A `heapMB()` helper was removed from the membrane rather than shipped.
**Still one process, one engine.** A separate isolate is a far stronger boundary
than SES, but Figma's argument for QuickJS was that a *different VM* cannot
confuse objects because the representations differ. Here the guarantee is V8's
isolate boundary plus the membrane's discipline — strong, and structurally
enforced above, but not the same category of claim.
## Files
- `src/membrane.ts` — the membrane. The safety argument is in the header comment.
- `src/facet-example.js` — unmodified facet code: `ctx.use()`, `slots.claim()`
with a closure, a class instance returned to the host, a subscribe callback.
- `src/demo-facet.ts` — loads it and exercises four boundary crossings.
- `src/escape-audit.ts` — the audit above.
- `src/property-test.ts` — 412 assertions: random `JsonValue` round-trips both
directions, identity across crossings, callables, error propagation, reference
release under GC, prototype pollution both ways, budgets, async, disposal.
- `src/bench.ts` — the numbers above.
- `src/markdown-bundle.js` — the real pi `Markdown` component, esbuild-bundled.

View file

@ -0,0 +1,13 @@
{
"name": "facet-sandbox-ivm",
"private": true,
"type": "module",
"description": "Running pi facets in an isolated-vm isolate: full JIT, no ambient authority.",
"scripts": {
"demo": "node --experimental-strip-types src/demo-facet.ts",
"audit": "node --experimental-strip-types src/escape-audit.ts",
"bench": "node --experimental-strip-types src/bench.ts",
"test": "node --experimental-strip-types --expose-gc src/property-test.ts"
},
"dependencies": { "isolated-vm": "6.2.0" }
}

View file

@ -0,0 +1,48 @@
/**
* Cost of the isolate, the membrane, and a real component at transcript scale.
* Compare against `../facet-sandbox-poc` (QuickJS) and native.
*/
import ivm from "isolated-vm";
import { readFileSync } from "node:fs";
import { createCompartment } from "./membrane.ts";
// ── membrane crossing ─────────────────────────────────────────────────────
{
const c = await createCompartment();
await c.endow("__h", { echo: (v: unknown) => v });
c.load(`globalThis.spin = function (n) { for (var i = 0; i < n; i++) __h.echo("x"); };`);
c.load(`spin(2000);`);
const N = 20000, t = Date.now();
c.load(`spin(${N});`, 60000);
console.log(`membrane crossing: ${((Date.now() - t) * 1000 / N).toFixed(1)} µs`);
c.dispose();
}
// ── isolate creation ──────────────────────────────────────────────────────
{
const before = process.memoryUsage().rss, t = Date.now();
const held = Array.from({ length: 50 }, () => {
const iso = new ivm.Isolate({ memoryLimit: 32 });
const ctx = iso.createContextSync();
ctx.evalSync(`var c = { render: function () { return ["x"]; } };`);
return iso;
});
console.log(`50 isolates: ${(((process.memoryUsage().rss - before) / 50) / 1024).toFixed(0)} KB each, ${Date.now() - t} ms total`);
for (const i of held) i.dispose();
}
// ── real Markdown at transcript scale ─────────────────────────────────────
{
const iso = new ivm.Isolate({ memoryLimit: 512 });
const ctx = iso.createContextSync();
ctx.global.setSync("global", ctx.global.derefInto());
const t = Date.now();
ctx.evalSync(readFileSync(new URL("./markdown-bundle.js", import.meta.url), "utf8"), { timeout: 60000 });
console.log(`146 KB bundle load: ${Date.now() - t} ms`);
for (const [label, pts] of [["small", 2], ["typical", 8], ["large", 35]] as [string, number][]) {
ctx.evalSync(`__bench(${pts}, 20, 50)`, { timeout: 120000 }); // warm the JIT
const ms = ctx.evalSync(`__bench(${pts}, 300, 3)`, { timeout: 300000 });
console.log(`300 markdown components, ${label.padEnd(8)} ${ms.toFixed(0)} ms per full repaint`);
}
iso.dispose();
}

View file

@ -0,0 +1,22 @@
import { readFileSync } from "node:fs";
import { createCompartment } from "./membrane.ts";
const mounted = new Map(), commands = new Map(), notifications = [], subs = [];
const Tui = {
slots: { claim: (s, f) => mounted.set(s, f) },
commands: { add: (n, h) => commands.set(n, h) },
notify: (m) => notifications.push(m),
};
const Transcript = { tail: { subscribe: (f) => subs.push(f) } };
const c = await createCompartment();
await c.endow("__host", { use: (t) => (t === "Tui" ? Tui : Transcript) });
c.load(readFileSync(new URL("./facet-example.js", import.meta.url), "utf8"));
c.load(`globalThis.__r = __facet.construct({ use: function (t) { return __host.use(t); } });`);
console.log("contributions:", { slots: [...mounted.keys()], commands: [...commands.keys()], subs: subs.length });
const comp = mounted.get("footer")({ label: "presses" });
console.log(comp.render(30).join("\n"));
comp.handleInput("abc");
console.log(comp.render(30).join("\n"));
console.log("command ->", commands.get("footer.reset")());
subs[0]({ entries: [1, 2, 3] });
console.log("notifications:", notifications);
c.dispose();

View file

@ -0,0 +1,63 @@
/**
* Escape audit.
*
* The claim the membrane makes is that the guest's entire view of the host is
* `{ number, string }` — no object graph, so nothing to walk. This tries to
* break that, including the specific `Reference` footgun that makes isolated-vm
* dangerous when used naively.
*/
import { createCompartment } from "./membrane.ts";
const c = await createCompartment();
const seen: string[] = [];
await c.endow("__h", {
sink: (v: string) => seen.push(v),
give: () => ({ nested: { a: 1 } }),
giveFn: () => () => "host function result",
});
const probe = (name: string, src: string) => {
c.load(`globalThis.__p = String(${src});`);
c.load(`__h.sink(__p);`);
const got = seen.pop();
const bad = got !== undefined && /GOT (?!undefined)|INVOKED|CALLABLE|^object$/.test(got) && !/blocked/.test(got);
console.log(` ${bad ? "!!" : "ok"} ${name.padEnd(34)} ${got}`);
return !bad;
};
console.log("Ambient authority:");
let ok = true;
ok = probe("require / process / fetch", `JSON.stringify([typeof require, typeof process, typeof fetch])`) && ok;
ok = probe("global names visible", `Object.getOwnPropertyNames(globalThis).length + " globals"`) && ok;
console.log("\nClassic escape ladders:");
ok = probe("Function('return process')()", `(function(){ try { return String((function(){}).constructor("return typeof process")()); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("constructor walk on host data", `(function(){ try { var o = __h.give(); return String(o.constructor.constructor("return typeof process")()); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
console.log("\nThe isolated-vm Reference footgun:");
ok = probe("__invokeRef.deref()", `(function(){ try { return "GOT " + typeof __invokeRef.deref(); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("__invokeRef.copySync()", `(function(){ try { return "GOT " + typeof __invokeRef.copySync(); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("__invokeRef.getSync('constructor')", `(function(){ try { return "GOT " + typeof __invokeRef.getSync("constructor"); } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("derefInto() is callable?", `(function(){ var d = __invokeRef.derefInto(); return typeof d === "function" ? "CALLABLE" : "inert (" + typeof d + ")"; })()`) && ok;
ok = probe("invoke derefInto() result", `(function(){ try { __invokeRef.derefInto()(); return "INVOKED"; } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("host globals via derefInto()", `(function(){ var d = __invokeRef.derefInto(); return String(d.process || d.global || "no host globals"); })()`) && ok;
console.log("\nWhat a host FUNCTION looks like to the guest:");
ok = probe("typeof a returned host fn", `(function(){ var f = __h.giveFn(); return typeof f; })()`) && ok;
// NOTE: `f.deref` is NOT undefined — the membrane's proxy returns a proxy for
// every property. That is not a leak: calling it routes to hostFn["deref"],
// which does not exist, and throws host-side. Testing for the property's
// absence gave a false positive in a first draft of this audit.
ok = probe("calling f.deref() reaches nothing", `(function(){ try { var f = __h.giveFn(); f.deref(); return "INVOKED"; } catch(e){ return "blocked: "+e.name; } })()`) && ok;
ok = probe("calling the host fn works", `(function(){ var f = __h.giveFn(); return String(f()); })()`) && ok;
console.log("\nResource bounds:");
const t0 = Date.now();
let interrupted = false;
try { c.load(`while(true){}`, 200); } catch { interrupted = true; }
console.log(` ${interrupted ? "ok" : "!!"} spinning guest interrupted after ${Date.now() - t0}ms`);
console.log(` ok isolate usable afterwards ${c.load(`1+1`) === 2}`);
c.dispose();
console.log(`\n${ok && interrupted ? "AUDIT PASSED" : "AUDIT FAILED"}`);
process.exit(ok && interrupted ? 0 : 1);

View file

@ -0,0 +1,42 @@
// ── REAL facet code. Nothing here knows it is sandboxed. ────────────────────
// No imports rewritten, no string marshalling, no __dispatch. It uses the
// host API exactly as facets.md §11.1 describes it.
class FooterComponent {
constructor(props) { this.label = props.label; this.count = 0; }
render(width) {
const text = ` ${this.label}: ${this.count} `;
const pad = Math.max(0, width - text.length - 2);
const bar = "\u2500".repeat(Math.max(0, width - 2));
return [`\u250c${bar}\u2510`, `\u2502${text}${" ".repeat(pad)}\u2502`, `\u2514${bar}\u2518`];
}
handleInput(data) { this.count += data.length; }
}
const facet = {
id: "@demo/footer:tui",
uses: ["Tui", "Transcript"],
construct(ctx) {
const tui = ctx.use("Tui");
const transcript = ctx.use("Transcript");
// A factory crossing guest → host, by reference.
tui.slots.claim("footer", (props) => new FooterComponent(props));
// A command handler, likewise.
tui.commands.add("footer.reset", () => {
tui.notify("footer reset");
return "ok";
});
// A subscription: host calls back into the guest.
transcript.tail.subscribe((tail) => {
tui.notify(`transcript now has ${tail.entries.length} entries`);
});
return [];
},
};
globalThis.__facet = facet;

View file

@ -0,0 +1,34 @@
import ivm from "isolated-vm";
const rss = () => process.memoryUsage().rss / 1024 / 1024;
// 1. Are two CONTEXTS in one isolate isolated from each other?
{
const iso = new ivm.Isolate({ memoryLimit: 64 });
const a = iso.createContextSync(), b = iso.createContextSync();
a.evalSync(`globalThis.secret = "from-A"`);
console.log("context B sees A's global:", b.evalSync(`typeof globalThis.secret`));
// Can a spin in A be timed out without harming B?
try { a.evalSync(`while(true){}`, { timeout: 100 }); } catch { /* expected */ }
console.log("B usable after A was interrupted:", b.evalSync(`1+1`) === 2);
// Does A's memory bomb kill B?
try { a.evalSync(`var x=[]; for(;;) x.push(new Array(50000).fill(1));`, { timeout: 15000 }); }
catch (e: any) { console.log("A bomb:", e.message.slice(0, 45)); }
try { console.log("B after A's bomb:", b.evalSync(`1+1`)); }
catch (e: any) { console.log("B after A's bomb: DEAD -", e.message.slice(0, 40)); }
try { iso.dispose(); } catch {}
}
// 2. Cost per facet, both ways.
for (const mode of ["one isolate, N contexts", "N isolates"]) {
const before = rss(), t = Date.now();
const held: any[] = [];
const shared = mode.startsWith("one") ? new ivm.Isolate({ memoryLimit: 512 }) : null;
for (let i = 0; i < 50; i++) {
const iso = shared ?? new ivm.Isolate({ memoryLimit: 32 });
const ctx = iso.createContextSync();
ctx.evalSync(`var c = { render: function () { return ["x"]; } };`);
held.push(iso);
}
console.log(`50 facets, ${mode.padEnd(24)}: ${(((rss() - before) / 50) * 1024).toFixed(0)} KB each, ${Date.now() - t} ms`);
if (shared) { try { shared.dispose(); } catch {} } else for (const i of held) { try { i.dispose(); } catch {} }
}

View file

@ -0,0 +1,27 @@
import { createCompartment } from "./membrane.ts";
const c = await createCompartment();
let seen = 0;
await c.endow("__l", { drop: (_fn: unknown) => { seen++; } });
c.load(`globalThis.churn = function (n) { for (var i = 0; i < n; i++) __l.drop(function () { return i; }); };`);
const rss = () => process.memoryUsage().rss / 1024 / 1024;
const before = rss();
c.load(`churn(20000);`, 60000);
const peak = c.debugRefCount();
const afterChurn = rss();
globalThis.gc?.(); await new Promise((r) => setTimeout(r, 100));
globalThis.gc?.(); await new Promise((r) => setTimeout(r, 100));
const settled = c.debugRefCount();
console.log(`20000 crossings: host called ${seen}x`);
console.log(` guest refs: ${peak.guestRefs} -> ${settled.guestRefs}`);
console.log(` host table: ${peak.hostTable} -> ${settled.hostTable}`);
console.log(` rss: ${before.toFixed(0)} -> ${afterChurn.toFixed(0)} -> ${rss().toFixed(0)} MB`);
// Cost of the release path: one evalSync crossing per finalized ref.
const t = Date.now();
c.load(`churn(5000);`, 60000);
globalThis.gc?.(); await new Promise((r) => setTimeout(r, 200));
console.log(` 5000 more + gc: ${Date.now() - t} ms, refs now ${c.debugRefCount().guestRefs}`);
c.dispose();

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,197 @@
import ivm from "isolated-vm";
/**
* A membrane between the host realm and an isolated-vm isolate.
*
* ── The safety argument, which is structural, not a convention ──────────────
*
* isolated-vm CAN be used unsafely. The documented footgun is handing the guest
* a live handle to a host object — via `derefInto()`, or by returning a
* `Reference` from a host call — after which the guest can walk that object's
* prototype chain into the host realm.
*
* This membrane makes that impossible by construction rather than by care:
*
* 1. `encode()` is the ONLY way a host value reaches the guest, and it is
* `JSON.stringify` with a replacer. Its output is a string. A string
* cannot carry a reference.
* 2. Host callables do not cross. They are replaced by an integer id into
* `hostTable`, which lives entirely host-side. The guest receives a number.
* 3. Exactly ONE `Reference` is ever given to the guest — `__invokeRef`, the
* single call-in point — and its `deref()` throws cross-isolate ("Cannot
* dereference this from current isolate"). Its `derefInto()` yields an
* inert `Dereference` marker: not callable, no host globals.
* 4. `derefInto()` is used exactly once, on the guest's OWN global. No host
* object is ever an argument to it.
*
* The consequence: the guest's entire view of the host is `{ number, string }`.
* There is no object graph to walk, so there is nothing to walk it into. This is
* the same property the QuickJS membrane has, and it is why the choice between
* them is about performance and packaging rather than about safety.
*
* Verified by `src/escape-audit.ts`.
*/
export async function createCompartment({ memoryLimit = 256 } = {}) {
const isolate = new ivm.Isolate({ memoryLimit });
const ctx = await isolate.createContext();
await ctx.global.set("global", ctx.global.derefInto()); // guest's own global only
const hostTable = new Map();
let nextHostId = 1;
const exposeHost = (o) => { const id = nextHostId++; hostTable.set(id, o); return id; };
const guestProxies = new Map();
const finalizer = new FinalizationRegistry((id) => {
guestProxies.delete(id);
if (!released) ctx.evalSync(`delete __refs[${JSON.stringify(String(id))}]`);
});
let released = false;
// Async: `applySync` is synchronous, so a host Promise cannot itself cross.
// Instead the host returns a token, the guest builds a real Promise against
// it, and the host settles that Promise by calling back in when it resolves.
let nextPending = 1;
function encodePending(promise) {
const token = nextPending++;
promise.then(
(v) => settle(token, true, v === undefined ? "" : encode(v)),
(e) => settle(token, false, JSON.stringify(String(e?.message ?? e))),
);
return JSON.stringify({ __pending: token });
}
function settle(token, ok, json) {
if (released) return;
ctx.evalSync(`__settle(${token}, ${ok}, ${JSON.stringify(json)})`);
}
function encode(v) {
if (v && typeof v.then === "function") return encodePending(v);
return JSON.stringify(v, (_k, val) => {
if (typeof val === "function") return { __ref: exposeHost(val) };
if (val && typeof val === "object" && !Array.isArray(val) && hasMethods(val))
return { __obj: exposeHost(val), keys: methodNames(val) };
return val;
});
}
const decode = (json) => JSON.parse(json, (_k, v) => reviveGuest(v));
function intern(id, build) {
const hit = guestProxies.get(id)?.deref();
if (hit !== undefined) return hit;
const p = build();
guestProxies.set(id, new WeakRef(p));
finalizer.register(p, id);
return p;
}
function reviveGuest(v) {
if (!v || typeof v !== "object") return v;
if (typeof v.__guestRef === "number") { const r = v.__guestRef; return intern(r, () => (...a) => callGuestRef(r, a)); }
if (typeof v.__guestObj === "number") {
const r = v.__guestObj;
return intern(r, () => { const o = {}; for (const k of v.keys) o[k] = (...a) => callGuestRef(r, [k, a]); return o; });
}
return v;
}
function callGuestRef(ref, args, timeout = 5000) {
const out = ctx.evalSync(
`__refs[${JSON.stringify(String(ref))}](${JSON.stringify(JSON.stringify(args))})`,
{ timeout },
);
return out === "" || out === undefined ? undefined : decode(out);
}
// The single host entry point. A Reference, applied synchronously with copied
// arguments — no host object graph is ever exposed.
const invoke = new ivm.Reference((idJson, pathJson, argsJson) => {
const path = JSON.parse(pathJson);
let target = hostTable.get(JSON.parse(idJson));
for (let i = 0; i < path.length - 1; i++) target = target[path[i]];
const result = path.length ? target[path[path.length - 1]](...decode(argsJson)) : target(...decode(argsJson));
return result === undefined ? "" : encode(result);
});
await ctx.global.set("__invokeRef", invoke);
await ctx.eval(`
globalThis.__refs = {}; globalThis.__nextRef = 1;
globalThis.__outgoing = new WeakMap();
globalThis.__methodNames = function (o) {
var out = [], p = o;
while (p && p !== Object.prototype) {
for (const k of Object.getOwnPropertyNames(p))
if (k !== "constructor" && typeof o[k] === "function" && out.indexOf(k) < 0) out.push(k);
p = Object.getPrototypeOf(p);
}
return out;
};
globalThis.__mkRef = function (fn) {
var prior = __outgoing.get(fn); if (prior !== undefined) return { __guestRef: prior };
var id = __nextRef++; __outgoing.set(fn, id);
__refs[String(id)] = function (j) { var r = fn.apply(null, JSON.parse(j)); return r === undefined ? "" : __encode(r); };
return { __guestRef: id };
};
globalThis.__mkObjRef = function (obj) {
var prior = __outgoing.get(obj);
if (prior !== undefined) return { __guestObj: prior, keys: __methodNames(obj) };
var id = __nextRef++; __outgoing.set(obj, id);
__refs[String(id)] = function (j) { var c = JSON.parse(j); var r = obj[c[0]].apply(obj, c[1]); return r === undefined ? "" : __encode(r); };
return { __guestObj: id, keys: __methodNames(obj) };
};
globalThis.__encode = function (v) {
return JSON.stringify(v, function (k, val) {
if (typeof val === "function") return __mkRef(val);
if (val && typeof val === "object" && !Array.isArray(val) && __methodNames(val).length) return __mkObjRef(val);
return val;
});
};
globalThis.__pending = {};
globalThis.__settle = function (token, ok, json) {
var d = __pending[token];
if (!d) return;
delete __pending[token];
var value = json === "" ? undefined : JSON.parse(json);
ok ? d.resolve(__revive(value)) : d.reject(new Error(value));
};
globalThis.__revive = function (v) {
if (v && typeof v === "object" && typeof v.__pending === "number") {
var token = v.__pending;
return new Promise(function (res, rej) { __pending[token] = { resolve: res, reject: rej }; });
}
if (v && typeof v === "object") {
if (typeof v.__ref === "number") return __wrap(v.__ref, []);
if (typeof v.__obj === "number") return __wrap(v.__obj, []);
for (var k in v) v[k] = __revive(v[k]);
}
return v;
};
globalThis.__wrap = function (id, path) {
return new Proxy(function () {}, {
get: function (t, prop) {
if (typeof prop !== "string") return undefined;
if (prop === "then" || prop === "toJSON" || prop === "inspect") return undefined;
if (prop === "length" || prop === "name") return t[prop];
return __wrap(id, path.concat([prop]));
},
apply: function (_t, _this, args) {
var out = __invokeRef.applySync(undefined, [JSON.stringify(id), JSON.stringify(path), __encode(args)]);
return out === "" ? undefined : __revive(JSON.parse(out));
},
});
};
`);
return {
async endow(name, obj) { await ctx.eval(`globalThis[${JSON.stringify(name)}] = __wrap(${exposeHost(obj)}, []);`); },
load: (src, timeout = 30000) => ctx.evalSync(src, { timeout }),
debugRefCount: () => ({ hostTable: hostTable.size, guestRefs: ctx.evalSync(`Object.keys(__refs).length`) }),
dispose() {
released = true;
// An isolate that hit its memory limit is already disposed, and calling
// dispose() again throws "Isolate is already disposed".
try { isolate.dispose(); } catch { /* already gone */ }
},
};
}
const RESERVED = new Set(["constructor"]);
const methodNames = (o) => { const out = []; for (let p = o; p && p !== Object.prototype; p = Object.getPrototypeOf(p)) for (const k of Object.getOwnPropertyNames(p)) if (!RESERVED.has(k) && typeof o[k] === "function" && !out.includes(k)) out.push(k); return out; };
const hasMethods = (o) => methodNames(o).length > 0;

View file

@ -0,0 +1,137 @@
/**
* Property tests for the membrane.
*
* Every bug found by hand while building it failed SILENTLY — wrong output, no
* throw. That is the worst possible failure mode for a boundary, and the reason
* these exist. Run with: node --experimental-strip-types --expose-gc property-test.ts
*/
import { createCompartment } from "./membrane.ts";
let pass = 0;
const failures: string[] = [];
const check = (name: string, ok: boolean, detail = "") => {
if (ok) pass++;
else failures.push(`${name}${detail ? ` — ${detail}` : ""}`);
};
/** Random JsonValue. The membrane claims to round-trip exactly these. */
function randomJson(depth = 0): unknown {
const r = Math.random();
if (depth > 3 || r < 0.25) return Math.floor(Math.random() * 1000) - 500;
if (r < 0.35) return Math.random() * 1e6 - 5e5;
if (r < 0.45) return Math.random() < 0.5;
if (r < 0.5) return null;
if (r < 0.65) {
const alphabet = ['a', 'z', '0', ' ', '"', '\\', '\n', '\u00e9', '\u4e2d', '\u{1f600}'];
let s = "";
const n = Math.floor(Math.random() * 12);
for (let i = 0; i < n; i++) s += alphabet[Math.floor(Math.random() * alphabet.length)];
return s;
}
if (r < 0.8) return Array.from({ length: Math.floor(Math.random() * 5) }, () => randomJson(depth + 1));
const o: Record<string, unknown> = {};
for (const k of ["a", "b", "c", "d", "__proto__x", "0", "length"]) {
if (Math.random() < 0.5) o[k] = randomJson(depth + 1);
}
return o;
}
const c = await createCompartment();
// ── 1. Value round-trip, both directions ──────────────────────────────────
const echoed: unknown[] = [];
await c.endow("__h", { echo: (v: unknown) => v, sink: (v: unknown) => { echoed.push(v); } });
c.load(`globalThis.roundTrip = function (v) { return __h.echo(v); };
globalThis.sendBack = function (v) { __h.sink(v); };`);
for (let i = 0; i < 400; i++) {
const value = randomJson();
const json = JSON.stringify(value);
if (json === undefined) continue;
// host -> guest -> host
c.load(`globalThis.__t = roundTrip(${json});`);
c.load(`sendBack(__t);`);
const got = echoed.pop();
check("value round-trip", JSON.stringify(got) === json, `${json} became ${JSON.stringify(got)}`);
}
// ── 2. Identity: the same guest object must map to the same host proxy ────
const seen: unknown[] = [];
await c.endow("__id", { take: (o: unknown) => { seen.push(o); } });
c.load(`globalThis.shared = { m: function () { return 1; } };
globalThis.sendShared = function () { __id.take(shared); __id.take(shared); };`);
c.load(`sendShared();`);
check("identity preserved across crossings", seen[0] === seen[1]);
c.load(`sendShared();`);
check("identity stable across separate calls", seen[0] === seen[2]);
// ── 3. Callables survive and remain callable ──────────────────────────────
let held: any;
await c.endow("__f", { keep: (fn: unknown) => { held = fn; } });
c.load(`__f.keep(function (a, b) { return a + b; });`);
check("guest function callable from host", held(2, 3) === 5, `got ${held?.(2, 3)}`);
// ── 4. Errors cross with name and message ─────────────────────────────────
c.load(`__f.keep(function () { throw new TypeError("boom"); });`);
let msg = "";
try { held(); } catch (e: any) { msg = e.message; }
check("guest throw reaches host", msg.includes("boom"), `got "${msg}"`);
// ── 5. Reference release: crossings must not leak ─────────────────────────
await c.endow("__l", { drop: (_fn: unknown) => {} });
c.load(`globalThis.churn = function () { for (var i = 0; i < 500; i++) __l.drop(function () { return i; }); };`);
c.load(`churn();`);
const beforeGc = c.debugRefCount().guestRefs;
globalThis.gc?.();
await new Promise((r) => setTimeout(r, 60));
globalThis.gc?.();
await new Promise((r) => setTimeout(r, 60));
const afterGc = c.debugRefCount().guestRefs;
check("dropped references are released", afterGc < beforeGc / 2, `${beforeGc} -> ${afterGc}`);
// ── 5b. Prototype pollution must not cross ────────────────────────────────
// JSON.parse is safe (it makes __proto__ an own property), but a membrane that
// assigns parsed keys onto a fresh object is not.
const before = ({} as any).polluted;
c.load(`__h.sink(JSON.parse('{"__proto__":{"polluted":true}}'));`);
check("guest cannot pollute host Object.prototype", ({} as any).polluted === before);
await c.endow("__p", { give: () => JSON.parse('{"__proto__":{"hostPolluted":true}}') });
await c.endow("__q", { show: (v: string) => check("host cannot pollute guest prototype", v === "undefined", v) });
c.load(`var got = __p.give(); __q.show(String(({}).hostPolluted));`);
// ── 6. Interrupt budget, and the VM survives it ───────────────────────────
// Direct: the budget on this call bounds it.
{
const t0 = Date.now();
try { c.load(`while (true) {}`, 100); } catch { /* expected */ }
check("runaway guest code is interrupted", Date.now() - t0 < 600, `took ${Date.now() - t0}ms`);
}
// Re-entrant: a guest function invoked BY THE HOST gets `callGuestRef`'s own
// budget, not the outer one. Budgets do not nest — see README "Limitations".
c.load(`globalThis.spin = function () { while (true) {} };`);
await c.endow("__s", { run: (fn: any) => { try { fn(); return "returned"; } catch { return "threw"; } } });
{
const t0 = Date.now();
try { c.load(`globalThis.__x = __s.run(spin);`, 50); } catch { /* expected */ }
const took = Date.now() - t0;
check("re-entrant call uses its OWN budget, not the outer one", took > 1000,
`took ${took}ms — if this is now <1s the nesting behaviour changed`);
}
c.load(`globalThis.__alive = 1 + 1;`);
check("VM usable after an interrupt", true);
// ── 7. Async, and clean teardown with a pending promise ───────────────────
await c.endow("__a", { never: () => new Promise(() => {}), soon: () => Promise.resolve("v") });
c.load(`globalThis.out = null; __a.soon().then(function (v) { out = v; });`);
await new Promise((r) => setTimeout(r, 30));
await c.endow("__o", { show: (v: string) => check("async host result reaches guest", v === '"v"', v) });
c.load(`__o.show(JSON.stringify(out));`);
c.load(`__a.never().then(function () {}, function () {});`); // deliberately unsettled
let disposeOk = true;
try { c.dispose(); } catch { disposeOk = false; }
check("dispose with a pending promise does not crash", disposeOk);
console.log(`\n${pass} passed, ${failures.length} failed`);
for (const f of failures.slice(0, 10)) console.log(" FAIL", f);
process.exit(failures.length === 0 ? 0 : 1);

View file

@ -0,0 +1,81 @@
# pi — design handoff
Work in numbered order. Each unit is self-contained and independently testable;
later units consume earlier ones.
```
01-harness/
01-delta/ op vocabulary, tracker, applier, codec [LANDED IN CHORD]
02-scopes/ storage scopes and list tags [STEP 1 ACTIONABLE]
03-execenv/ bounded Shell output, capture, spill [PRODUCTION CODE + TESTS]
04-tool-output/ the ToolOutput sink [SPEC ONLY]
05-assistant-output/ assistant partials, symmetric with 04 [SPEC ONLY]
02-plugins/
01-facets/ the facet system [SPEC ONLY]
02-sandbox/ isolated-vm membrane [CODE + 412 tests]
```
Base everything on a clean checkout of `origin/dev`.
## Read this first
**Three units ship working code. Four are specifications.** The table above says
which. Do not assume a doc describes something that exists.
**`01-delta/FINDINGS.md` is historical evidence, not an implementation queue.** Production lives in `packages/chord/src/delta/index.ts`: flush-time dirty tracking fixed D1, and production remeasurement closed D2. The explicit append/truncate API is rejected; see [`01-harness/01-delta/append-decision.md`](01-harness/01-delta/append-decision.md). The code beside the handoff remains prototype and benchmark evidence.
**If a doc and the code disagree, the code wins** — fix the doc and say so in the
commit.
**Port the tests before the implementation.** Each group's comment explains the
failure it guards against, and several of those failures are silent: wrong output,
no exception.
**Benchmark with `node --experimental-strip-types`, never through a transpiler.**
Measuring this module through `tsx` inflates results 2.6x. `FINDINGS.md` D5 lists
five more measurement traps, each of which produced a confident wrong conclusion.
## Unit status
| unit | ships | state |
| --- | --- | --- |
| **01-delta** | production implementation and tests in `packages/chord`; prototype evidence here | landed; D1 fixed, explicit text API rejected after production remeasurement |
| **02-scopes** | spec + [actionable Step 1 handoff](01-harness/02-scopes/implementation-handoff.md) + `scopes.variance.ts` | Step 1 scopes/list tags actionable, not implemented; JSONL Chord encoding/address interning deferred to separately approved Step 2 |
| **03-execenv** | production implementation in `packages/agent`; prototype evidence here | source-bounded adaptive output, lazy spill backpressure, and bash migration landed; bash's temporary checkpoint cadence moves to `ToolOutput` next |
| **04-tool-output** | spec + design notes | **not built.** The piece every measurement of the op encoding depends on |
| **05-assistant-output** | spec | not built. Same shape as 04; do it after |
| **02-plugins/01-facets** | spec, ~1800 lines | not built. §14 rewritten to match the sandbox PoC |
| **02-plugins/02-sandbox** | working PoC, 412 assertions | `npm install && npm run audit` |
## Suggested order
1. **`02-scopes` Step 1** — follow the [actionable implementation handoff](01-harness/02-scopes/implementation-handoff.md); stop for approval before its separate Step 2.
2. **`04-tool-output`** — reuse the landed adaptive publisher for generic tools, Chord event/durable batches, terminal flushes, and atomic memo checkpoints.
3. **05**, then **02-plugins**.
## Live bugs on `origin/dev`, independent of this design
- `drive/tools.ts:257` — `clearReplayCheckpoint` deletes `pendingToolOutput` before
re-executing a replay-safe tool. Memos exist so a replayed tool skips work, and
skipped work emits nothing, so output for memoised work is lost today. Seed from
it instead (`harness-tools.md` §7.4).
- `runtime/progress.ts:44` — `commitWrite(item)` captures the value at call time
and writes fire-and-forget, so an older checkpoint can land after a newer one.
- memo and checkpoint are two transactions (`drive/tools.ts:112` vs
`progress.ts:44`). They must be one (`harness-tools.md` §7.5).
**Expected test churn:** nine tests assert the old cleanup write set and will fail
once `retireScope` replaces the per-address deletes. That is the change landing.
## Environment
- Node 22+ for every shipped `.ts` file. They run under
`node --experimental-strip-types` with no build step and no dependencies.
- In the pi repo, tests run from the package:
`cd packages/agent && npx vitest run --config vitest.harness.config.ts`.
The root vitest config does **not** alias `@earendil-works/pi-ai`; the
per-package harness config does.
- Typecheck with `npx tsgo --noEmit` from the repo root. **Baseline is ~788
pre-existing errors**, almost all in `packages/ai/test`. Count only:
`grep "error TS" | grep -E "packages/(agent|session-backends)/src"`.
- `packages/ai` cannot be built offline — model data is fetched at build time.

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,305 @@
# Pico rendering
This document defines how a presentation renders a replicated Pico conversation. It covers local and
remote clients, initial attachment, live streaming, terminal handoff, renderer registration and stable
component identity. The harness model remains defined by `pico-v3.md`; the examples and client-facing
surface belong in `pico-usage-guide.md`.
The target is the directness of coding-agent interactive mode and `experimental/mini`, without their
hard-coded event vocabulary or duplicate hydration logic.
## 1. The presentation owns a replica, not agent state
A presentation owns one `ConversationView`. It does not read storage, parse commits or hold live task,
provider, tool or model objects.
Opening a watch atomically captures the view and registers its listener. The harness buffers changes
until delivery starts. Each committed change is folded into the view before listeners observe its
events. An in-process client uses the harness-owned view; a remote client applies the same exported
kind-free reducer before notifying its UI.
Delivery over one WebSocket or SSE connection is ordered and gap-free. If delivery fails or a bounded
queue overflows, that watch closes. Reconnection opens a new watch and receives a fresh view. There are
no cursors, epochs, acknowledgements, deduplication, replay requests or in-place resnapshot protocol.
The view is authoritative. Events identify the affected portion so a live renderer can avoid scanning
the whole view; they are not a second source of presentation state.
## 2. Local view access
The raw collections remain serializable. A local wrapper adds indexed and typed access:
```ts
interface ConversationView {
readonly conversation: Conversation;
readonly entries: readonly Entry[];
readonly tasks: readonly Task[];
readonly previews: ReadonlyMap<Id, JsonValue>;
readonly inbox: readonly Element<QueuedInput>[];
readonly values: ReadonlyMap<Address, JsonValue>;
readonly context: readonly Id[];
readonly faulted: boolean;
readonly readAt: Id;
entry(id: Id): Entry | undefined;
task(id: Id): Task | undefined;
entriesOf<K extends EntryDefinition>(kind: K): readonly EntryOf<K>[];
tasksOf<K extends TaskDefinition>(kind: K): readonly TaskOf<K>[];
preview<S extends TaskStateBase, H extends HookPoints, C extends ConfigSpec, P,
R extends TaskRoles<S>>(
kind: TaskKind<S, H, C, P, R>, taskId: Id,
): P | undefined;
}
```
`preview(kind, taskId)` returns `undefined` when the task is absent, has another kind or has no live
preview. Built-in and plugin renderers use it without casts. `task(id)` is sufficient to classify a
`task_output` event because the reducer has already folded that event before the listener runs.
A remote wire view may remain arrays and maps encoded as JSON. The wrapper builds its indexes locally;
those indexes are not another replicated representation.
## 3. Two rendering paths
Both paths use the same registered renderers and component keys. They differ only in how much of the
view they visit.
### 3.1 Hydration
Hydration runs on first attachment and after opening a replacement watch. It walks the current entries
and live tasks and upserts their blocks:
```ts
function hydrate(view: ConversationView): void {
for (const entry of view.entries) entryRenderers.render(entry, view);
for (const task of view.tasks) taskRenderers.render(task, view);
chrome.sync(view);
}
```
This is the view-driven path used by Mini. Existing blocks are reconciled by stable key, so a fresh
view need not tear down unchanged terminal components.
### 3.2 Live events
Normal streaming uses targeted event handling, as interactive mode does:
```ts
function handle(event: ConversationEvent, view: ConversationView): void {
switch (event.type) {
case "entry":
entryRenderers.render(event.entry, view);
break;
case "task_start":
case "task_update":
taskRenderers.render(event.task, view);
break;
case "task_output": {
const task = view.task(event.task);
if (task) taskRenderers.render(task, view);
break;
}
case "task_end":
taskRenderers.end(event.task, view);
break;
case "inbox":
case "value":
case "context":
chrome.sync(view, event);
break;
case "fault":
case "closed":
chrome.connection(event);
break;
}
}
```
A streamed token therefore updates one task renderer. It does not rebuild every transcript block or
run every layout. A client that does not need the optimization may ignore the event payload and call
`hydrate(view)` after every notification; it remains correct.
The renderer callbacks, not a mandatory full-tree diff, are the shared path between hydration and live
updates.
## 4. Renderer registry
Entry and task renderers are registered by kind. Registration is typed by the kind object even though
the wire and internal registry use its string token:
```ts
renderers.entry(noteKind, previous => ({
render(entry, host, view) { /* entry is NoteEntry */ },
}));
renderers.task(downloadKind, previous => ({
render(task, preview, host, view) { /* typed task and DownloadPreview */ },
end(task, host, view) { /* terminal typed task */ },
}));
```
The exact component type belongs to the presentation package, not Pico. The common contract is:
- `render` upserts components through the host using stable keys;
- `end` handles terminal live-state cleanup and does not need a retired preview;
- renderers do not directly parent components in the transcript container;
- registration may replace or wrap `previous`;
- built-ins and plugins use the same API;
- removing renderer code does not affect the replicated view.
An unregistered entry kind uses a generic renderer: model text when present, otherwise collapsed
kind/data. An unregistered task kind uses kind, status and a collapsed raw preview when present.
Unknown kinds must remain visible and must not break hydration.
The heterogeneous registry may erase renderer types internally. That cast or validation boundary is
owned once by the registry; renderer authors do not cast task state, entry data or previews.
## 5. Renderer host
The host owns component instances and placement. Renderers only describe or update blocks through it.
At minimum it maintains:
```text
entry id -> settled transcript block
live task id -> live task block or region
tool-call id -> tool component
inbox item id -> queued-input component
```
It provides keyed `get`, `upsert`, `move`/`adopt` and `remove` operations. A renderer can update an
existing component without knowing which container currently parents it. The host preserves expansion,
selection and other local UI state while adopting or moving a block.
No persistent `tool task id -> tool-call id` map is required. Every tool-task state variant carries
its call:
- on `task_output`, `view.task(event.task).state.call.id` identifies the component;
- on `task_end`, the event's terminal task carries the same call;
- tool-result entries use their indexed call key.
A renderer may cache expensive presentation-only products such as parsed Markdown. Such caches are
never authoritative and may be discarded on hydration.
## 6. Stable identity and terminal handoff
Every rendered object has a stable semantic key:
| Object | Key |
| --- | --- |
| transcript entry | entry ID |
| live generation | generation task ID |
| assistant content block | generation task ID + content index |
| tool component | tool-call ID |
| live tool preview lookup | tool task ID |
| inbox item | list-element/input ID |
| compaction, job or plugin task | task ID |
A component may change ownership without changing its semantic identity.
### Assistant handoff
A generation preview renders a partial assistant message under the generation task ID. The final
assistant entry has `byTaskId` equal to that generation task. Its renderer adopts the existing live
component, applies the immutable final message and then associates the settled transcript position
with the entry ID. It does not append a duplicate assistant component.
If no matching live component exists, as on initial attachment, the entry renderer creates the same
settled component directly.
### Tool handoff
Tool-call components are born while the assistant preview streams and are keyed immediately by the
call ID. A later tool-task start marks that same component running. Tool output updates it from the
typed preview. The terminal tool-result entry updates the same call-ID component with immutable final
output. No phase creates a second tool component.
### Other tasks
A task that has no terminal entry removes or finalizes its live block in `end`. A task kind that
requires a durable transcript representation writes that entry in the same commit in which it
settles. The view is already folded to the whole commit before listeners run, so no client-visible idle
or missing-result state exists between those events.
## 7. Task provenance required by rendering
Task creation records generic provenance just as entry creation does:
```ts
interface Task {
// ...
readonly byTaskId?: Id;
}
```
`ConversationTx` fills `byTaskId` from the invocation whose commit created the task; it is absent for
tasks created outside a task invocation. Task kinds do not write it themselves.
This replaces kind-specific provenance such as `JobState.origin`. A job created by a tool has
`byTaskId` equal to the tool task. A renderer follows that task and narrows it with `toolKind.is` to
select the originating tool renderer. Nested delegated work follows the chain as needed.
## 8. Structured previews
A preview is the complete current render model for one live task, not a text delta.
The generation preview is the full partial `AssistantMessage` assembled from
`AssistantMessageFrame`s. It retains ordered content blocks, text, thinking, signatures, partial tool
arguments, provider metadata, usage and diagnostics. `task_output.ops` only transports changes; the
renderer reads the already-updated message through `view.preview(generationKind, task.id)`.
A tool preview is the complete current `ToolOutputState`: text or structured content, plugin details,
images, usage, delegation, handoff, added tools, termination, diagnostics and truncation metadata. Tool
renderers own plugin-specific interpretation. The generic host does not flatten it to text.
Terminal rendering uses immutable entries or terminal task state because settlement retires the live
preview.
## 9. Layouts are optional presentation policy
Grouping tool calls, collapsing a run until its final answer and similar structures are useful, but a
general composable layout/reconciliation pipeline is not required by the Pico watch or renderer
contract.
A presentation may add layouts above the keyed renderer host. Layouts must preserve block keys and
must not make preview delivery depend on rebuilding the complete transcript. They should initially run
only for structural changes unless a particular layout explicitly depends on live preview content.
The first coding-agent client should reach feature parity with direct keyed placement. Add a general
layout API only after at least two concrete layouts demonstrate that a shared abstraction is smaller
than their local implementations.
## 10. Client-owned visible work
A client operation that runs over time and should survive in the transcript is an ordinary task kind,
not a presentation-only event stream. For example, `!cmd` is a `pi.user_bash` task owned by the coding
agent or bash plugin. It shares process-output helpers with jobs, publishes a typed preview, and writes
its terminal entry in the same commit as settlement.
The presentation starts it with a command and then handles its generic `task_start`, `task_output`,
`task_end` and `entry` events. Escape calls `abortTask(id)`. Plugins use exactly the same pattern for
client-initiated work.
## 11. Mapping from interactive mode
| Interactive-mode event | Pico rendering |
| --- | --- |
| `agent_start` / `turn_start` / assistant `message_start` | generation `task_start` |
| `message_update` | generation `task_output`; read typed assistant preview |
| `message_end` | assistant `entry`; adopt by `byTaskId` |
| `tool_execution_start` | tool `task_start`; identify component by call ID |
| `tool_execution_update` | tool `task_output`; read typed tool preview |
| `tool_execution_end` | tool `task_end` plus tool-result `entry` |
| `compaction_start` / `_end` | collapse `task_start` / `task_end`; summary `entry` |
| retry events | generation or collapse `task_update` |
| `agent_end` / `agent_settled` | no live foreground task after the commit |
| `queue_update` | `inbox` |
| thinking/model change | typed configuration `value` |
| custom entry | registered entry renderer or generic fallback |
| `bash_execution_update` | `pi.user_bash` `task_output` |
The generic vocabulary changes dispatch, not component behavior. Compared with interactive mode, Pico
removes the second independently authored hydration implementation. Compared with Mini, it replaces
hard-coded operation fields with typed task kinds and replaceable renderers while retaining the same
small keyed-component model.

View file

@ -0,0 +1,37 @@
# Pico implementation state
## Authority
[`pico-simple-handoff.md`](pico-simple-handoff.md) is the sole normative implementation specification.
Read it completely before implementation. Other Pico documents and prototypes are historical inputs only.
The original specification and WP1 received delegated review. Peer review then identified and resolved
TaskOutput contract inconsistencies in JSONL replay, mutation failure handling, watch reconnection,
kind replacement and generation partial state.
## Git state
- Branch: `pico`
- Branch base when created: `f3c672245`.
- WP1 implements compile-time declarations under `packages/agent/src/harness/pico/` with focused compile
coverage.
- Implement and commit one small work package at a time for user review.
## Next action
Review the WP1 commit, then begin WP2 only after explicit user approval. WP2 is the mutation algebra and
`MemoryStorage`; follow its prerequisites, acceptance criteria and one-commit boundary in the handoff.
## Adopted core
The harness owns `pending -> running -> terminal`. Task input is immutable typed JSON. A checkpoint is an
optional full replacement with `phase: string`. Execute/recover return typed terminal closures applied
atomically with outcome and scratch retirement. Cancellation is mark, revoke writes, signal, join, fresh
abort, then a restricted abort closure.
`turn: true` is a trusted replaceable task-kind capability. Background is independent because speculative
manual collapse is background plus turn. Non-turn tasks use limited transactions and `accept`/`write`;
turn tasks may append model-affecting entries directly. Every missing live kind becomes orphaned at open.
Input groups belong to built-in generation/post_tools. V1 is job-first and excludes arbitrary unfinished
promise adoption. Shared TaskOutput storage/runtime/watch foundations are settled; concrete tool/job
payloads, bounded capture/spill policy and model projection remain gated.

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,773 @@
# pico
Durable agent harness. One session file, any number of conversations, every piece of work is a task that
survives a crash, and a view a UI renders directly.
This guide shows how to use it. `pico-handoff-v2.md` is the implementation specification.
- [Mental model](#mental-model)
- [Quick start](#quick-start)
- [Opening, closing, recovery](#opening-closing-recovery)
- [Sending input](#sending-input)
- [Configuration](#configuration)
- [System prompt](#system-prompt)
- [Watching](#watching)
- [Forks, compaction, reset](#forks-compaction-reset)
- [Plugin state](#plugin-state)
- [Writing tools](#writing-tools)
- [Hooks](#hooks)
- [Writing entry kinds](#writing-entry-kinds)
- [Writing task kinds](#writing-task-kinds)
- [Subagents and jobs](#subagents-and-jobs)
- [Storage backends](#storage-backends)
## Mental model
A **session** holds **conversations**. A conversation has:
- a **transcript**: append-only immutable **entries** (user, assistant, tool result, summary, notice, ...);
- **tasks**: units of work (generate, run a tool, compact, run a process). Each survives a crash;
- **state**: keyed values and lists (model in use, plan mode, whatever a plugin needs).
The model **context** is derived from entries, not stored: the newest head entry says where context starts,
edits omit or replace earlier messages. Compaction appends a summary with a head; nothing is rewritten.
All writes go through **commits**: one callback, one atomic batch. Tasks run concurrently, commits serialize.
Every async call takes a Chord `Context` last (`ctx`). It carries cancellation and telemetry. Cancelling a
wait removes the waiter; it never cancels durable work.
## Quick start
```typescript
import { Harness, JsonlStorage, systemSections } from '@earendil-works/pi-agent/pico';
import { builtinModels } from '@earendil-works/pi-ai/providers/all';
import { BACKGROUND_CONTEXT as ctx } from '@earendil-works/chord/context';
const storage = await JsonlStorage.open('./session.jsonl', ctx);
const h = await Harness.open(storage, {
models: builtinModels(),
// no `tools`: the ordinary tool API is not settled yet (see Writing tools); this is a tool-free agent
root: { // applied once, when the session file is new
config: {
model: { provider: 'anthropic', modelId: 'claude-opus-5' },
thinkingLevel: 'high',
},
},
}, ctx);
const c = await h.root(ctx);
// System prompt: edit typed sections before each request. Stored when it changes, replayed after a restart.
// Registered on this conversation only: subagents get their own identity (see Subagents).
c.hook.systemInstructions(({ sections }) => {
sections.set(systemSections.identity, 'You are a careful engineer working in this repository.');
sections.set(systemSections.environment, { cwd: process.cwd() });
});
// Watch: `w.view` is always current; each delivery is one commit's events.
const w = await c.watch(ctx);
w.start((d) => {
if (d.type !== 'commit') return;
for (const e of d.events) {
if (e.type === 'task_output') process.stdout.write(renderStreaming(w.view.taskOutputs));
if (e.type === 'entry') console.log(`\n[${e.entry.kind} ${e.entry.id}]`);
if (e.type === 'config' && e.key === 'model') statusLine.set(e.value); // typed: ModelRef | undefined
}
});
// Send: durable, returns at once. Wait for this input's answer.
const input = await c.send({ requestId: 'cli-1', content: 'List the public API of the parser' }, ctx);
const result = await input.wait(ctx);
if (result.status === 'done' && result.answer) console.log(result.answer.message);
// Change settings any time; the next request uses them.
await c.config.model.set({ provider: 'openai', modelId: 'gpt-5.6' }, ctx);
w.unsubscribe();
await h.close(ctx);
```
Kill it mid-run and run it again: the second run recovers the in-flight generation (or job) and continues;
tools will recover the same way once their API lands.
## Opening, closing, recovery
```typescript
const h = await Harness.open(storage, options, ctx); // inert: nothing runs yet
await h.resume(ctx); // start/recover every eligible task
```
`send`, `wait`, `waitForIdle`, `collapse` and abort operations call `resume` for you. Reads, inspection and
watch do not. After a crash, `open` + `resume` is the whole recovery procedure:
| was running | on resume |
|---|---|
| generation | partial output is restored; request retried or failed within budget |
| tool with `replay: 'safe'` | runs again (once the tool API lands) |
| other tool | "interrupted" error result; the turn continues (once the tool API lands) |
| tool waiting on a child | waits again on the same child (once the mediated child API lands) |
| job | asks the process host for the run's status; if the host still knows it, continues; if not, reruns under the same key if `rerun` (possibly duplicating the effect), else `interrupted`. `nodeProcessHost()` forgets runs on restart |
```typescript
const { pending, running, orphaned, unknownEntryKinds } = await h.inspect(ctx);
```
`orphaned` are live tasks whose kind was not registered at open; they were terminalized. Register all
kinds before opening.
```typescript
await h.close(ctx); // stop, write nothing; everything resumes next open
await h.shutdown(ctx); // abort every live task cleanly, keep queued input, then close
```
Options:
| option | |
|---|---|
| `models` | pi-ai models; `generationKind.config.model` is resolved against it |
| `tools` | tool declarations the model may call; not usable yet, the execution API is gated (see Writing tools) |
| `taskKinds`, `entryKinds`, `sections` | plugin kinds and system section definitions |
| `root.config` | initial built-in config (`model`, `thinkingLevel`, `selectedTools`, ...) for a fresh session; ignored on reopen |
| `root.values`, `root.sections`, `rootValues` | advanced: arbitrary initial values / section payloads for the fresh root |
| `processHost` | how `pi.job` runs processes; `nodeProcessHost()` for local jobs (in-memory: forgets runs on restart); omitted means every job fails with reason `spawn` |
Registries supply implementations, not selections: registering `bash` does not select it. Select through
`c.config.selectedTools`. `h.tools`, `h.taskKinds`, `h.entryKinds`, `h.sections` have
`register`/`replace`/`remove` for changes after open; replacing or removing a task kind with live tasks
rejects. Hooks are registered on `h.hooks` or `c.hooks` (below), not in open options.
## Sending input
```typescript
const input = await c.send({ content: 'Inspect the parser', requestId: 'req-1' }, ctx);
input.id; // stable input id
await input.result(ctx); // queued | placed | done | unanswered, no waiting
const r = await input.wait(ctx); // done | unanswered
await input.abort(ctx); // aborted | already_placed | not_found
if (r.status === 'done') {
r.input; // UserEntry: the placed user entry
r.answer?.entry; // AssistantEntry, when an answer was owed
r.answer?.message; // the stored AssistantMessage (entry.model[0]); text, thinking, tool calls, usage
} else {
r.reason; // terminated | aborted | failed | stale
r.input; // present if the input had been placed before it went unanswered
}
```
Outcomes come with their entries attached; there is nothing to look up afterwards. `message` is the message as
stored, not the current context projection, so a later edit or compaction does not change what `wait`
returned.
`requestId` makes `send` idempotent for the session: retrying returns the original handle and writes nothing.
`h.input(requestId, ctx)` finds it later.
While the conversation is busy (a generation or tool is live), input is queued:
```typescript
await c.send({ content: 'then write tests' }, ctx); // followUp: after the answer
await c.send({ content: 'focus on the tokenizer', whenBusy: 'steer' }, ctx); // steer: at the next tool boundary
await c.send({ content: '...', whenBusy: 'reject' }, ctx); // throws ConversationBusy
```
Steer joins the current turn after the running tools finish. FollowUp starts a new turn after the final
answer. Both survive restarts. Passive content (a notice the model should see next time) uses `write`:
```typescript
await c.commit((tx) => tx.write(noticeKind, {
model: [{ role: 'user', content: 'CI finished: 3 failures', timestamp: Date.now() }],
}), ctx);
```
`write` appends immediately if idle, otherwise queues for the next boundary. It never asks for an answer.
There is no "queue this for the next run" mode. If you accumulate context that should not trigger a turn yet
(observations, partial instructions), keep it in your own value or list and fold it into a single `send` when
you do want an answer; use `write` only when a passive transcript entry is the point.
```typescript
await c.waitForIdle(ctx); // this conversation's foreground work
await h.waitForIdle(ctx); // whole session
await c.abort(ctx); // abort this conversation's foreground cancellation set (turn, foreground tasks, owned subagents); withdraws queued steer/followUp
await h.abortTask(id, ctx); // one task; queued input is kept
```
## Configuration
Built-in configuration is a set of typed properties on `c.config`, one per value, plus a batched setter:
```typescript
await c.config.model.set({ provider: 'openai', modelId: 'gpt-5.6' }, ctx);
await c.config.thinkingLevel.set('high', ctx);
await c.config.selectedTools.set(['read', 'grep'], ctx);
await c.config.followUpMode.set('all', ctx);
const model = await c.config.model.get(ctx); // current (or the definition's default)
const then = await c.config.model.get(entryId, ctx); // as of an entry (rewindable values only)
const mode = await c.config.followUpMode.get(ctx); // sticky: current only
// several at once, one commit; keys and value types are exact
await c.config.set({ model: { provider: 'openai', modelId: 'gpt-5.6' }, thinkingLevel: 'low', followUpMode: 'all' }, ctx);
const all = await c.config.get(ctx);
```
| property | type | default | rewindable |
|---|---|---|---|
| `model` | `ModelRef` (`{ provider, modelId }`) | none: generation fails `no_model` | yes |
| `thinkingLevel` | `ThinkingLevel` (`off` … `xhigh`, `max`) | `'off'` | yes |
| `selectedTools` | `string[]` | `[]` | yes |
| `profile` | `string` | `'default'` | yes |
| `retry` | pi-ai `RetryPolicy` | `{ enabled: true, maxRetries: 3, baseDelayMs: 2000, maxAgentDelayMs: 60000 }` | no |
| `steeringMode`, `followUpMode` | `'all' \| 'one-at-a-time'` | `'one-at-a-time'` | no |
| `threshold` | `number` (context tokens; 0 = never) | `0` | yes |
| `keepRecent` | `number` (tokens kept after compaction) | `20000` | yes |
Defaults live on the definitions and are what `get`, `view.config` and `config` events return when nothing is
set; `delete` restores them. The types say so: `c.config.followUpMode.get(ctx)` is `'all' | 'one-at-a-time'`,
never `undefined`; only `model` (no default) can be `undefined`.
These are the same definitions the built-in tasks read (`generationKind.config.model` and friends). Each kind
exports a typed bundle of reusable conversation-value definitions; `c.config` is assembled from all of them,
so there is no second schema, and a name clash between kinds or with `set`/`get` is a compile error. The
definitions carry no conversation ID; the handle binds them, and you can pass them anywhere a conversation
value is accepted (`c.value(generationKind.config.model)`, `root.values`, child seeds). Plugin configuration is
not in `c.config`; plugins declare their own values and use `c.value(token)`.
Rewindable means a fork at yesterday's answer gets yesterday's model. Queue modes are sticky: they describe
how you want the present handled, not history. The next generation reads current values; nothing is cached.
## System prompt
Pico sends pi-ai `{ messages }` only. Instructions and the tool loadout are system messages inside the
transcript at the position they took effect, so the model's cached prefix survives a change and a fork sees
exactly what its source saw.
Sections are typed tokens with a renderer. A `systemInstructions` hook edits a draft before each request:
```typescript
import { defineSystemSection, systemSections } from '@earendil-works/pi-agent/pico';
import { generationKind } from '@earendil-works/pi-agent/pico/kinds';
const rules = defineSystemSection<string[]>({
key: 'myplugin.rules',
render: (r) => r.map((x) => `- ${x}`).join('\n'),
});
await h.sections.register(rules, ctx);
// host, harness-wide: sets the authoritative base every time, so it runs first
h.hooks.on(generationKind, generationKind.hooks.systemInstructions, ({ sections }) => {
sections.set(systemSections.identity, 'You are a coding assistant.');
sections.set(systemSections.skills, skillsCache.current);
sections.set(rules, ['Run relevant tests.']);
});
// plugin, on this conversation: runs after the host and modifies typed payloads, never parses prose
c.hook.systemInstructions(({ sections }) => {
sections.set(rules, [...(sections.get(rules) ?? []), 'Check migration safety.']);
sections.wrap(rules, (text) => `Repository policy:\n${text}`);
});
// rare: override the loadout for this request instead of c.config.selectedTools
c.hook.systemInstructions(({ tools }) => ({ tools: tools.filter((t) => t.name !== 'bash') }));
```
Handlers run harness-wide first, then outer conversation to inner, on one draft; the plugin above therefore
sees the host's base. Registration is synchronous and process-local (it returns an unsubscribe function);
`h.sections.register` is asynchronous because the section registry is shared state. `get` returns a copy;
`set` keeps an existing key's position; `delete` is explicit (null is a value); `wrap` transforms rendered
text and is not stored. A throwing handler loses only its own edits. The tool loadout is
`c.config.selectedTools` resolved against `h.tools`; a handler that returns `tools` overrides it for that
request.
After the hooks, the draft is compared to what was last stored. No change: nothing is written. A change:
one `pi.system` entry with the changed sections' payload and rendered text, and a system message with
`toolsRemoved`/`toolsAdded`. After a compaction or reset, the next request gets a fresh complete baseline.
A plugin that disappears leaves its rendered text in place until someone deletes the section.
Because handlers rerun on every preparation, a handler that appends to what it reads accumulates unless an
earlier handler resets the base. Make the chain idempotent as a set.
## Watching
```typescript
const w = await c.watch({ values: [planMode], lists: [moves] }, ctx); // or c.watch(ctx); === h.watchConversation(c.id, ...)
w.view; // ConversationView, captured atomically with the subscription
w.start((d) => { // one delivery per commit: { type: 'commit', first, last, events } | { type: 'closed', reason }
if (d.type === 'commit') for (const e of d.events) { /* w.view is already updated */ }
});
w.unsubscribe();
```
```typescript
// abbreviated sketch; the exact readonly declarations are in the handoff, section 14
interface ConversationView {
conversation: Conversation;
entries: Entry[]; // the active transcript in order: everything from the newest head's boundary through now
context: Id[]; // entry ids the model currently sees, derived from entries
tasks: Task[]; // live tasks in this conversation
taskOutputs: { id: Id; kind: string; value: OutputState }[]; // streaming assistant text, tool progress
inbox: Element<QueuedInput>[];
config: { model?, thinkingLevel, selectedTools, profile, retry, steeringMode, followUpMode, threshold, keepRecent }; // defaults resolved: every key except model is present and never undefined
values: { address: Value; value?: JsonValue }[]; // only what you asked for in { values: [...] }
lists: { address: List; elements: Element[] }[]; // only what you asked for in { lists: [...] }
readAt: Seq;
}
```
A requested list is captured whole and then kept current by `list` events (`append`/`remove`/`clear` ops in
commit order, already folded when the listener runs). Built-in config cannot be requested as a value; it is
always in `view.config`.
`view.config` has the same properties as `c.config`, so a status line reads `view.config.model` and a settings
panel binds to it directly; nothing needs to be requested. `values` is for your own state (`planMode` above).
Built-in config changes arrive as their own event, typed by key:
```typescript
w.start((d) => {
if (d.type !== 'commit') return;
for (const e of d.events) {
if (e.type === 'config') {
// e.key narrows e.value / e.previous
if (e.key === 'model') statusLine.setModel(e.value); // ModelRef | undefined
if (e.key === 'selectedTools') toolbar.set(e.value); // readonly string[]: defaulted, never undefined
}
if (e.type === 'value') plugins.update(e.value.address, e.value.value); // your requested values
if (e.type === 'list') board.render(w.view.lists); // ops already folded
}
});
```
`c.config.set({ model, followUpMode })` yields two `config` events in the same delivery; `w.view.config` already
reflects both when the listener runs. Read the initial state from `w.view.config`, then follow events.
The listener is a wake-up: `w.view` is already updated when it runs. A renderer that only looks at the view is
correct.
`entries` is what the conversation is currently working with, in transcript order, not a fixed number of
recent entries. It is captured once and then maintained incrementally: an ordinary entry appends; an entry that
carries a `head` (summary, reset, handoff) first drops the entries before its retained boundary and then
appends itself, all within the same delivery. A summary therefore sits after the entries it retains (the model
sees it first; that is context projection, not the view). A reset or handoff points at itself, so it clears the
list and becomes the first entry. The older transcript is still in storage and readable through `h.entries({ conversationId: c.id, limit: 50 }, ctx)` (newest first, paged) and `h.getEntry`, it just is
not part of the live view. There is no window size to tune.
| you want | look at |
|---|---|
| in a turn | `view.tasks` has a `pi.generation`/`pi.tool`/`pi.post_tools` |
| streaming text | `view.taskOutputs` entry for the generation's `output.id` |
| a tool running | `pi.tool` task; its `output` for progress |
| retrying | generation `checkpoint.phase === 'retrying'` |
| current model / tools / thinking | `view.config.model`, `.selectedTools`, `.thinkingLevel`; changes: `config` event with `key`/`value`/`previous` |
| compacting | `pi.collapse` task present |
| queued input | `view.inbox` |
Overflow (listener too slow, 256 commits buffered) closes the watch with `{ type: 'closed', reason:
'overflow' }`. Open a new one; its view is the fresh truth.
Remote UIs: ship `w.view` once, then every delivery. `applyConversationCommit(view, commit)` is exported and
needs no kinds.
```typescript
const sw = await h.watchSession({ values: [sessionName] }, ctx); // conversation list, session values
```
### Plugin state in a UI
Requested Pico values and lists (as in the watch above) are for the process that owns the Harness: the plugin's
session-side facet, or a host renderer in the same process. Pico does not push plugin state to remote UIs. A
plugin that has something to show elsewhere follows the Chord facet pattern
used by `coding-agent/examples/plugins/pi-example-plugin` and the experimental `models-provider` /
`transcript-provider` services:
```typescript
// contract.ts: what the UI may see and call
export interface PlanService {
readonly state: ReplicatedState<{ active: boolean; steps: string[] }>;
toggle(input: { active: boolean }, ctx: Context): Promise<void>;
}
export const PlanService = defineService<PlanService>('myplugin.plan');
// session.ts: runs beside the Harness; Pico is the durable truth, the replicated state is a projection
import { BACKGROUND_CONTEXT } from '@earendil-works/chord/context';
export default defineFacet({
id: 'myplugin/session',
setup(env) {
const state = env.replicatedState({ active: false, steps: [] });
env.provide(PlanService, {
state,
async toggle({ active }, ctx) {
await c.value(planMode).set(active, ctx); // durable
state.state.active = active; state.publish(ctx); // live
},
});
env.onActivate(async () => { // reopen: rebuild the projection from Pico
state.state.active = (await c.value(planMode).get(BACKGROUND_CONTEXT)) ?? false;
state.publish(BACKGROUND_CONTEXT);
});
},
});
// tui.ts: never sees the Harness or addresses
export default defineFacet({
id: 'myplugin/tui',
setup(env) {
const plan = env.use(PlanService);
env.onActivate(() => env.own(plan.state.subscribe((value) => badge.set(value.active)))); // (value, context, delivery) => void
},
});
```
Built-in config is the exception and is automatic (`view.config`); everything else a UI should see is a
service DTO the plugin defines.
## Forks, compaction, reset
```typescript
const alt = await c.fork({ at: entryId }, ctx); // shares history through entryId; independent after
const alt2 = await c.fork({ at: 'start' }, ctx); // empty transcript, same session
const back = await c.fork({ at: earlier, abort: true }, ctx); // "go back": abort the source's turn first
await alt.send({ content: 'try recursive descent instead' }, ctx);
```
Forks inherit rewindable state (model, tools) and the system messages visible at the fork point. They inherit
no tasks and no queued input. Nothing in the source changes.
```typescript
const taskId = await c.collapse({ instructions: 'keep API decisions verbatim' }, ctx);
await h.getTask(taskId, ctx);
```
Manual compaction runs in the background while the model keeps working; it declines with `NothingToCollapse`
if everything already fits in `keepRecent`. Threshold compaction is created by generation automatically when the
context estimate crosses `threshold` and runs in the foreground (aborting the conversation aborts it too). Both append a `pi.summary` entry with a head; the transcript keeps everything.
```typescript
await c.reset({ handoff: 'We settled on recursive descent; continue from the lexer.' }, ctx);
await c.reset(ctx); // /clear
```
Reset appends a `pi.handoff` or `pi.reset` entry with `head: self`. If the conversation is busy it is queued for
the next boundary; either way `reset` resolves once the request is durable, and the entry shows up in the watch
when it lands.
## Plugin state
```typescript
import { conversationValue, conversationList, sessionValue } from '@earendil-works/pi-agent/pico';
type Move = { readonly x: number; readonly y: number };
const planMode = conversationValue<boolean>('plan.mode', { rewind: true });
const moves = conversationList<Move>('game.moves', { rewind: true });
const expanded = conversationValue<boolean>('ui.expanded', { rewind: false });
const name = sessionValue<string>('app.name');
await c.value(planMode).set(true, ctx); // c binds it to its own conversation
await c.value(planMode).get(ctx);
await c.value(planMode).get(entryId, ctx); // rewindable only
const id = await c.list(moves).append({ x: 1, y: 2 }, ctx);
await c.list(moves).read(entryId, ctx);
await c.list(moves).remove(id, ctx);
await h.value(name).set('parser work', ctx);
planMode.bind(otherConversationId) // explicit Value<boolean> when you need one
```
A conversation address is declared once and bound when used. It carries rewind policy and payload type.
Your definitions work everywhere: `c.value`, commits, `root.values`, child seeds, watch `values`. Pico's
built-in config tokens (`generationKind.config.model` and friends) are exported and work the same way. Pico's
internals (inbox, input results, request receipts, task output) are not addressable through any
generic handle; they surface only as `view.inbox`, `InputHandle` and `view.taskOutputs`, and you cannot define
values in those namespaces.
Anything that must land together goes in one commit (assumes `noteKind` and `reminderKind` are registered, see
below; a kind object that is not the registered one is rejected as `StaleDefinition`):
```typescript
const id = await c.commit(async (tx) => {
const current = await tx.value(planMode).get(); // reads first
tx.value(planMode).set(!current); // rewindable state before entries
const note = await tx.write(noteKind, { data: { text: 'toggled' } });
tx.task(reminderKind, { background: true, input: { about: note, at: Date.now() + 3600_000 } });
return note;
}, ctx);
```
Rules inside a commit: reads before the first write (`ReadAfterWrite` otherwise); rewindable writes before
entry appends; no awaiting anything but `tx` reads; a throw discards everything.
## Writing tools
**The ordinary tool API is not settled.** This is the one deliberate gap in Pico v1: what a tool's `execute`
receives and how it streams progress is still being decided, so nothing in this section is a compile contract.
What *is* settled:
- A tool declares `name`, `description`, `parameters` (TypeBox schema), `replay: 'safe' | 'unsafe'` (may the
call be rerun after a crash) and `output` bounds (`maxBytes`, `maxLines`, `retain`).
- A tool returns a `ToolResult`: `content` (text/images the model reads), typed `details` for UIs, `isError`,
`usage`, `diagnostics` (commentary about the call, rendered after the output), and optional `control`.
- `control`: `{ terminate: true }` ends the turn after this exchange; `{ handoff: message }` ends it and resets
context with that message (what `new_context` does); `{ addTools: [...] }` adds tools to `selectedTools` so
the very next model request (normally the continuation right after this tool round) already offers them.
- Pico enforces the output bounds, reports truncation, stores the exact model message in the result entry and
the rest in its data, and streams the tool's live progress as the tool task's output (`view.taskOutputs`).
- A throw becomes an `isError` result; it never faults the session. Tools cannot touch the transcript, the task
or the inbox, and get no commit authority.
- Not yet available: durable per-call memos, starting a background job from a tool, starting a subagent from a
tool. Requirements are fixed (keyed and idempotent across a crash, cleanup policy recorded before the tool
returns, a wait timeout never cancels the child, a job is a task and a subagent is a conversation) but the
API shape is not. Until it is, the built-in `bash` cannot background and there is no `subagent` tool.
Illustration only (the `execute` signature and the `tool` facade are the gated part):
```typescript
// ILLUSTRATIVE, not the final API
const params = Type.Object({ path: Type.String(), pattern: Type.Optional(Type.String()) });
export const countLines = {
name: 'count_lines',
description: 'Count lines in a file',
parameters: params,
replay: 'safe',
output: { maxBytes: 64_000, maxLines: 200, retain: 'head' },
async execute(args /* Static<typeof params> */, tool /* gated facade */, ctx) {
const lines = (await fs.readFile(args.path, 'utf8')).split('\n');
const matching = args.pattern ? lines.filter((l) => l.includes(args.pattern!)) : lines;
return {
content: [{ type: 'text', text: matching.join('\n') }],
details: { lines: lines.length },
diagnostics: matching.length > 200 ? [{ severity: 'warn', message: `showing 200 of ${matching.length}` }] : [],
};
},
};
```
## Hooks
Hooks belong to the task kind that runs them; each kind declares its own typed points. The built-in points are
flattened onto `c.hook`, one method each: `systemInstructions`, `beforeRequest`, `afterResponse`, `onYield`,
`beforeTool`, `afterTool`, `afterTools`, `beforeCollapse`. Each is the generic
`c.hooks.on(kind, kind.hooks.point, handler, { subtree })` with kind and point bound; `h.hooks.on(...)`
registers harness-wide. Plugin kinds use the generic form with their own point tokens. Input, output and
failure behaviour come from the point:
```typescript
import { toolKind } from '@earendil-works/pi-agent/pico/kinds'; // only for the generic form
// conversation-scoped, named
c.hook.beforeTool(async ({ call }, { conversationId }, ctx) => { // a `block` from an earlier handler already stopped the chain
if (call.name === 'write' && await c.config.profile.get(ctx) === 'plan')
return { block: { reason: 'plan mode: no edits' } };
if (call.name === 'bash' && !(await ui.approve(call.arguments, ctx)))
return { block: { reason: 'denied by user' } };
return { call: { ...call, arguments: normalize(call.arguments) } }; // may rewrite arguments
}, { subtree: true }); // shared policy: applies to subagents this conversation owns too
c.hook.afterTool(async ({ call, result }) => { metrics.record(call.name); });
c.hook.beforeRequest(async ({ request }) => ({ request: withTracing(request) }));
c.hook.onYield(async ({ answer }, { conversationId }, ctx) => {
if (await goalIncomplete(conversationId, ctx)) return { continue: 'The goal is not met yet; continue.' };
});
c.hook.beforeCollapse(async ({ reason, entries }) => {
if (reason === 'manual' && entries.length < 10) return { decline: true };
});
// harness-wide, generic
h.hooks.on(toolKind, toolKind.hooks.beforeTool, handler); // same as c.hook.beforeTool, for every conversation
h.hooks.on(myKind, myKind.hooks.myPoint, handler); // your own kinds' points, by token
```
| kind | point | returns | several handlers | on throw |
|---|---|---|---|---|
| generation | `systemInstructions` | draft edits; `{ tools? }` | all run | its edits dropped, skipped |
| generation | `beforeRequest` | `{ request? }` | chained: each output is shallow-merged into the input the next handler sees | skipped |
| generation | `onYield` | `{ continue: string }` or nothing | first non-void answer wins | skipped |
| generation | `afterResponse` | nothing | all run | reported |
| tool | `beforeTool` | `{ call? }` or `{ block }` | chained (patch); `block` stops the chain | **blocks the tool** |
| tool | `afterTool` | `{ result? }` | chained (patch) | skipped |
| post_tools | `afterTools` | nothing | all run | reported |
| collapse | `beforeCollapse` | `{ decline }` or `{ instructions?, summary? }` | first answer wins | skipped |
Hooks run outside the commit line and may take as long as they like (a human approval is a hook that waits).
They may run again after a crash; key external side effects. Cancellation always propagates. `onYield`'s
`continue` appends a user entry and starts another generation for the same input; if the user typed something
meanwhile, that wins and the yield decision is dropped.
Your own task kinds declare points the same way (see below); `h.hooks.on(myKind, myKind.hooks.myPoint, ...)` then
type-checks against your declaration.
## Writing entry kinds
```typescript
import { defineEntry, type EntryBase, type EntryData, type ModelProjection } from '@earendil-works/pi-agent/pico';
type NoteEntry = EntryBase & EntryData<{ text: string }>;
export const noteKind = defineEntry<NoteEntry>('myplugin.note');
await h.entryKinds.register(noteKind, ctx); // or pass it in Harness.open({ entryKinds: [noteKind] })
// Data-only: the model sees nothing. `write` resolves with its admission/input id, not the later entry id;
// the committed entry appears in the watch.
await c.commit((tx) => tx.write(noteKind, { data: { text: 'plan accepted' } }), ctx);
type PinnedEntry = EntryBase & EntryData<{ text: string }> & ModelProjection<UserMessage>;
export const pinnedKind = defineEntry<PinnedEntry>('myplugin.pinned');
await h.entryKinds.register(pinnedKind, ctx);
const text = 'Always run the tests before claiming a fix works.';
await c.commit((tx) => tx.write(pinnedKind, {
data: { text },
model: [{ role: 'user', content: `<pinned>${text}</pinned>`, timestamp: Date.now() }],
}), ctx);
// later, from a watch event or an entry scan:
const entry = await h.getEntry(entryId, ctx);
if (noteKind.is(entry)) console.log(entry.data.text);
```
An entry may also carry `head` (context starts here) and `edits` (omit/replace earlier messages). A missing
plugin loses typed access, not context behaviour: facets are stored.
## Writing task kinds
```typescript
import { defineTask } from '@earendil-works/pi-agent/pico';
type Input = { readonly about: Id; readonly at: number };
type Checkpoint = { readonly phase: 'firing'; readonly key: string };
export const reminderKind = defineTask<Input, Checkpoint, { fired: boolean }, { message: string }, null>()({
kind: 'myplugin.reminder',
async execute(task, rt, ctx) {
if (task.input.at > rt.now()) await rt.sleep(task.input.at, ctx);
await rt.commit((tx) => tx.checkpoint({ phase: 'firing', key: `reminder:${task.id}` }), ctx);
await notifications.fire(task.input.about, `reminder:${task.id}`, ctx);
return () => ({ status: 'completed', result: { fired: true } });
},
async recover(task, rt, ctx) {
if (!task.checkpoint) return reminderKind.execute(task, rt, ctx); // crashed before the checkpoint: nothing external happened
// the checkpoint proves we intended to fire; the call is keyed, so firing again is safe if it did not land
if (!(await notifications.check(task.checkpoint.key, ctx))) {
await notifications.fire(task.input.about, task.checkpoint.key, ctx);
}
return () => ({ status: 'completed', result: { fired: true } });
},
async abort() {
return () => null;
},
});
await h.taskKinds.register(reminderKind, ctx); // fine for a fresh session. Once tasks of this kind can exist across restarts,
// pass it in Harness.open({ taskKinds: [reminderKind] }) on every open: open orphans
// live tasks whose kind is not registered *at open*, and nothing resurrects them
const id = await c.commit((tx) => tx.task(reminderKind, { background: true, input: { about: entryId, at: Date.now() + 3600_000 } }), ctx);
const task = await h.getTask(id, ctx);
```
The contract:
1. Pico marks the task `running` before calling `execute`. Commit a checkpoint before any external effect you
would need to reconcile after a crash.
2. `execute`/`recover` do their effects, then return a closure. Pico runs the closure on the commit line; its
writes, the outcome and scratch cleanup commit atomically. Do not perform effects in the closure.
3. `recover` is called instead of `execute` when a running task is found at open. Read `task.checkpoint` and
decide.
4. On abort, the old invocation is signalled through `ctx` and its later writes rejected. `abort` then runs
as a fresh call for cleanup and returns its own closure.
5. `after: [ids]` runs the task after those are terminal (any outcome). `rt.waitForTask(id, ctx)` observes.
6. Configuration lives in scoped values, not in task input. Task input is what recovery needs.
7. A task may `tx.write(...)` a passive entry (a notice) in any commit or in its terminal closure. It lands
now if the conversation is idle, at the next boundary if busy. Tasks never append directly and never ask
for an answer; that is `send`.
A kind may declare hook points. It decides when to run them; handlers registered on `h.hooks`/`c.hooks` for
this kind and point are invoked outside the line:
```typescript
import { defineHookPoint } from '@earendil-works/pi-agent/pico';
export const deployKind = defineTask<DeployInput, DeployCheckpoint, DeployResult, DeployFailure, null>()({
kind: 'myplugin.deploy',
hooks: {
beforeDeploy: defineHookPoint<{ target: string }, { approve: false; reason: string } | { approve: true }>({
fold: 'first', onThrow: 'abort',
}),
},
async execute(task, rt, ctx) {
const decision = await rt.hooks.run(deployKind.hooks.beforeDeploy, { target: task.input.target }, ctx);
if (decision.threw || decision.output?.approve === false)
return () => ({ status: 'failed', failure: { message: 'deployment declined' } });
...
},
...
});
await h.taskKinds.register(deployKind, ctx);
h.hooks.on(deployKind, deployKind.hooks.beforeDeploy, async ({ target }, info, ctx) =>
(await ui.approve(`deploy to ${target}?`, ctx)) ? { approve: true } : { approve: false, reason: 'denied' });
```
`fold` says how several handlers combine (`collect` all outputs, `first` non-void wins, `chain` shallow-merges
each handler's patch into the input the next handler sees). `onThrow` says whether a throwing handler is skipped
or stops the point.
Kinds declaring `output` get `rt.output` (read/mutate/replace) whose deltas stream to watchers:
```typescript
// abbreviated: recover/abort omitted
type Progress = { pct: number; log: string[] };
const progressKind = defineTaskOutput<Progress>('myplugin.progress');
defineTask<{ readonly steps: number }, never, { readonly done: true }, { readonly message: string }, null, Progress>()({
kind: 'myplugin.long',
output: { kind: progressKind, initial: () => ({ pct: 0, log: [] }) },
async execute(task, rt, ctx) {
await rt.output.mutate((o) => { o.pct = 50; o.log.push('halfway'); }, ctx);
return () => ({ status: 'completed', result: { done: true } });
},
// ...
});
```
## Subagents and jobs
A subagent is a conversation owned by the task that created it. There is no subagent task and no subagent
object; the model will get one tool, the API gets a conversation handle. The tool-side operation is not yet
specified (see Writing tools). What is settled: a child never inherits the parent's `c.config` implicitly. Its
creator seeds `model`, `selectedTools`, `profile` and any other definitions it wants, plus initial system
section values, in the same commit that creates the child and accepts its first input, so the first
generation already sees them. A child created without a first input is inert; register its hooks and set its
config, then `send`. From the host a subagent is just another conversation: `h.conversation(id,
ctx)`, `h.conversations({ owner, limit: 100 }, ctx)`. Its own `systemInstructions` hooks (registered on the child
handle or inherited via `subtree`) give it its own identity. Aborting the owning tool call aborts it unless
the (gated) child API created it with `abortWithTool: false`.
A job is a process:
```typescript
// Harness.open(storage, { ..., processHost: nodeProcessHost() }, ctx) -- without a host, jobs fail with reason 'spawn'
const id = await c.commit((tx) => tx.task(jobKind, {
background: true,
input: { command: 'npm', args: ['run', 'dev'], cwd, notify: false, rerun: true },
}), ctx);
await h.abortTask(id, ctx); // SIGTERM, then SIGKILL
```
`nodeProcessHost()` keeps runs in memory only: after a restart it no longer knows them, so a job either
reruns (`rerun: true`, same key, possibly repeating the effect) or fails `interrupted`. A supervising host can
do better; that is the host's contract, not Pico's.
Its stdout/stderr tail (and how many bytes were dropped) is in its task output; `notify: true` appends a `pi.notice` when it finishes (per occurrence for a schedule) so the
model sees it on its next request (which may be the continuation of the current turn).
## Storage backends
```typescript
MemoryStorage.create() // tests
await JsonlStorage.open('./session.jsonl', ctx) // local; one file plus live sidecars
await SqliteStorage.open('./sessions.db', { session: 'parser' }, ctx) // many sessions, servers
```
Same behaviour everywhere. JSONL and SQLite take a cross-process lock; a second writer fails rather than
racing.

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,622 @@
# pico implementation plan
Clean room. Nothing from pico2 is copied; its tests are read for the bugs they caught (a failed
generation that never ended its run, refs inside task patches, a collapse deadlocking appends,
quiescence firing during a retry wait, watch dropping late deltas) so that packages 6, 9, 12 and
16 each carry a case for them.
The target is bottom-up modules with their own tests and explicit dependencies, green after every
step. The numbered groups below are a provisional coverage map, not final work-package sizes or a
completed dependency audit. After the remaining conceptual blockers are settled, run the independent
whole-document review and final work-package decomposition described below. `pico-v3.md` is the
reference design; `pico-usage-guide.md` shows the intended surface.
Next design discussions: evaluate status-indexed task authoring and returned plans separately, then
simplify the tool/sink/preview contract. The steps proposal has not been adopted. Groups 10, 14 and 15
preserve capability/test coverage, but their current API sketches are not ready-to-implement contracts.
Preview delivery coalescing is deferred. Approval/question workflows and durable answer reuse belong
to workspace/plugins; only their access to existing task identity/scratch is a Pico integration question.
## 1. Types and ids
`Id`, `EntryIdentity`, `EntryBase`, the composable `EntryData` / `ModelProjection` /
`ContextHead` / `ContextEdits` facets, `Entry`, `EntryKind`, `EntryInput`, `ContextEdit`, `Task`,
`TaskRole`, status-tagged task-state unions and their derived `Orphaned` variant, compiler-only
`TypedTask`/`TaskDefinition` witnesses preserving the complete union and literal role map, stored task roles
and `turn` flag, `Conversation`, `QueuedInput`, `InputResult`, `InboxOp`, acceptance
receipts, `Call` (an alias of Chord Context), `TaskRuntime`/`ToolRuntime`, private typed invocation
identity, `Address`/`Value`/`List`/`Scope`, `Write`/`CommitBatch`, `Page`/`Cursor` and query shapes.
Every async harness/handle/runtime method takes a required final Call; no duplicate signal options,
task-conversation facades or admission gates.
Add immutable `SystemSection<T>` definitions created by `defineSystemSection<T>`: a durable string
`key` and synchronous pure `render(value: T): string`, with JSON-representable payloads. Tokens provide
typed access without plugin casts; only their keys and payloads are serialized. Export typed built-ins
`systemSections.identity`, `.environment` and `.skills`; plugins may define/register their own.
There is no discovery/read callback on a token and no renderer callback on an entry kind. Payload
shapes are trusted after wire validation; shape changes need migration or compatible replacement.
`SystemSectionDraft` exposes typed `get(token)`, `set(token, value)`, `delete(token)` and
`wrap(token, rendered => string)`. `get` returns an owned copy; array changes use get+set, not an append
operator. Persist ordered `SectionChange` records:
```ts
type SectionChange =
| { key: string; action: "set"; value: JsonValue; rendered: string }
| { key: string; action: "remove" };
interface SystemData {
baseline?: true;
sections?: readonly SectionChange[];
}
type SystemEntry = EntryBase & EntryData<SystemData> & ModelProjection<SystemMessage> & Partial<ContextEdits>;
```
Baseline sections are a complete ordered array of set records. Null and empty string are valid
payloads; deletion is an explicit action. Definitions and wrapper closures are never stored. Tool
definitions live only in model fields, never duplicated in SystemData.
Tests: typed token/draft inference without plugin casts; owned-copy reads; JSON null versus deletion;
every entry facet and built-in combination; persisted payload+rendered text and model-only tool definitions.
Task type tests: `defineTaskKind<States>()({ ... })` infers literal roles; exact role-map keys, start-only
initial status, reserved orphaned, consistent common-field types/optionality; orphaned exposes only
common fields via `Pick<S, Exclude<keyof S, "status">>`, preserving common optional fields. Typed reads
include orphaned; kind methods exclude it. No compiler witness is stored or needed by untyped readers.
## 2. Memory storage
`Storage` and `MemoryStorage`: `commit` numbering from `lastSeq`, point reads, fork-aware
`scanEntries` / `scanTasks` / `scanConversations` with cursors, target-capped `newestHead` over stored
numeric boundaries, stored entry data/model/edits, value versions and list elements read at a
position, `ownedFrom`, the set of entry kind strings written and session-wide live-task scans.
Use existing batched `getTasks` for named owners; no header/projection APIs or inbox-specific queries.
Tests: every row of the §7.2 query table against hand-built batches; ids are `lastSeq + 1 + i`; a
head is found without a kind; entry reads/scans/head lookup return full entries after indexed
selection; live-task scans skip terminal rows (named owner reads may fetch them); `remove` and
`clear` hide by position.
## 3. The line and `Tx`
`commit(plan, call)` on a serialized line: buffered writes, ids final at call time, rewindable
conversation value/list writes after an entry throw, a throwing plan discards everything, publish
after persist; apply the entire batch to live indexes before scheduling or testing idle. Driver
callbacks, signals and task methods dispatch outside the line. Session and sticky conversation state, task
writes and conversation writes may appear anywhere. `task` / `patch` / `settle` materialize the role
from the kind's status map; status lives only in `state.status`. `patch(task, status, payload)` and
`settle(task, status, payload)` always require an explicit status and its complete payload without
status. Both replace state with `{ ...payload, status }`; no partial merge or status-free overload.
Patch targets start/inflight roles, settle targets terminal roles and retires scratch; neither exposes
orphaned. A bare id must first be read with a kind to obtain the typed witness. Materialize `turn` from the kind and
maintain the indexed `inTurn` predicate. `entry` rejects outside-turn model-visible writes while turn
tasks are live; `write` places immediately or queues at a safe boundary, returning an inputId.
`value` / `list` / `entry` / `task` / `patch` / `settle` build the batch of §7.3.
Tests: concurrent commits serialize; a rejected commit consumes no ids; each builder verb produces
the expected write; reads inside a plan see committed state only; session and sticky conversation
writes may follow and reference a new entry; rewindable value set/delete and list
append/remove/clear after an entry each reject; exact invocation-token checks run on the line;
post-mark main/scratch mutation rejects before builder; only an owned task's current invocation may
patch its state/status or settle it (host abort marks remain allowed); caller cancellation never
abandons admitted persistence; Tx/ScratchTx reads are asynchronous and builders may await them
without releasing the line; writes remain synchronous. No external effects or nested line entry in builders.
Compile-time cases: required target fields and types, no extra top-level keys on literals/variables/spreads,
no duplicated status, status/payload correlation under union arguments, no role/status widening from
inference, narrowed task retains all transition targets, typed reads cannot write orphaned, no bare-id
escape. Cover valid same-status full replacement and optional fields; document structural typing/cast
limits rather than adding deep exact-type machinery. Runtime cases: replacement drops prior-variant
fields, rejects wrong roles/reserved statuses/duplicated payload status, current terminal tasks cannot
change, and same-status replacement does not advance the epoch. Payload validation stays at wire boundaries.
## 4. Entry kinds and context
`EntryKind` as `kind` plus `is`, the registry and typed append helpers for the built-in kinds
(`user`, `assistant`, `tool_result`, `system`, `notice`, `summary`, `handoff`, `reset`). Writers
materialize optional model messages and stored controls; context = newest stored head prepended to
the fork-aware range from its numeric boundary, older heads excluded, stored edits folded in
transcript order, stored model arrays concatenated, then pi-ai tool results ordered by call index.
This package has no managed-system state fold, baseline hoisting or system-kind projection filter.
Generation preparation writes any required system changes and omission edits (group 9); the generic
projector applies stored facets only. Arbitrary plugin head writers need no system-specific behavior.
Tests: data-only and model-only entries; summary keeps the tail; handoff/reset normalize `"self"`;
repeated compaction subsumes; a stored head below the previous visible boundary is rejected; edits
omit/replace targets and persist across turns; arbitrary managed-system targets reject, while atomic
fresh-baseline supersession is permitted; context is identical with its plugin kind unregistered;
retained system deltas remain visible until a later stored omission applies; tool-result order.
## 5. Forks and historical reads
`createConversation` with `parent`, the shared prefix in fork-aware `scanEntries`, capped-source
lookup for values and lists, arbitrary transcript-entry fork points.
Tests: a fork sees the head, edits and values in force at its entry; heads/results the source adds
later are invisible; deep fork chains; successful incomplete tool exchanges project with missing
results but inherit no tasks; state committed after the entry (a model change) is not in the fork.
## 6. Task kinds and the driver
`TaskKind` with `(task, runtime, call)` methods, registry and `TaskRuntime`; one Call-final convention
for operations, existing Context-aware env/provider/hook boundaries. Driver decisions, attachments,
waiters, invocation completion and lifecycle run on the commit line. Only effects/callbacks dispatch
outside it. Seed live tasks once; committed batches update live, reverse-dependency and conversation
indexes. Resolve ownership upward only for live/owned work; evict unused ancestry. Stable attachments
are distinct from temporary waiters. No Wake, polling loop, worker pool, poison graph, gate or parked role.
One invocation per task ID, concurrent across IDs; reserve before dispatch and release only after
actual return, even if already terminal. Durable abort revokes execute/recover main and scratch writes,
then signals outside line; fresh abort invocation alone writes cancellation outcomes. Status epochs
count actual committed transitions, including same-start/end cycles. Task-contract faults fail-stop:
reject waiters, stop admission, signal/join and close; preserve durable recovery state.
Tests: one open scan, no full scan per completion; 100,000 historical children with two live tasks;
whole-batch settle/successor publication; exact epoch rules; owned-task exclusive mutation; one
invocation per ID; blocked calls do not block others; abort bypasses dependencies and never overlaps
execute; repeated mark spares running abort; abort-handler failure; cooperative signal window;
known direct/dependency self-waits, including background caller and new cycle-creating admission;
already-aborted waiter registration and every cancellation/idle race; repeated drive uses one
attachment; foreground idle while attached background work continues; reopen recovers marked work.
## 7. Scratch
Scratch batches, one task per batch, `ScratchTx`, retire on settle, sidecar-independent (memory).
Tests: a crash before settle keeps scratch; settle deletes it; writes after mark/settle/retired token
reject; new attempts clear scratch; persisted assistant frames use the pi-ai compact frame encoder,
not raw cumulative provider events; a rejected stream-frame write cancels and joins the producer
before invocation completion (iterator exit is insufficient); harness progress bridges own every promise, suppress expected
late cancellation only, report persistence errors, and drain before invocation completion. Raw task
scratch calls must be awaited/caught; no successful silent no-op.
## 8. Harness shell and handles
`Harness.open` (built-in registries, `kinds` and `replace` options, kinds-set check, `inspect`,
`drive`, `close`, `shutdown`), `ConversationHandle` (`commit`, `value` / `list`, `config` /
`settings`, `fork` with `abort`, `abort`, `hooks.on` scoped with `subtree`), `acceptance(requestId)`,
`result(inputId, call)`, `abortTask`, `conversations` with `parent` / independent filtering. Call is
required on every async public/runtime operation, including reads and lifecycle; no raw host-lifecycle
Harness exposed to tasks. No section-order option or separate ordering configuration. No agent
behaviour yet. Root creation applies explicit `rootValues` atomically and only once; reopen preserves
durable model/thinking/selectedTools. Registry contents do not imply tool selection. Children use
explicit value-inheritance policy; missing required generation configuration fails clearly.
Initial `Harness.open` accepts `sections: [...]` definitions in addition to the built-in tokens.
Mutable registry operations take a required final Call and serialize on the line:
- `h.sections.register(token, call)`, `.replace(token, call)`, `.remove(tokenOrKey, call)`.
- Parallel `h.entryKinds` and `h.taskKinds` register/replace/remove APIs use kind definitions or names.
Register rejects duplicates; replacement is explicit and must understand the stored shape. Removing
code never deletes durable entries or section state. In-flight preparation retains its captured section
registry snapshot; later preparations see newer definitions, including explicit renderer replacements.
No rendering runs during registry mutation or replay. Task-kind removal rejects while live instances
exist. Open settles missing-kind foreground tasks as `orphaned`, parks missing-kind background tasks,
and leaves terminal history untouched. Registration restores parked recovery in already attached
scopes; it neither implicitly drives other scopes nor resurrects terminal tasks.
Tests: rootValues applies atomically only to a fresh root, ignored on reopen; no automatic tool
selection from registry changes; required-config errors; open on empty vs existing storage;
initial/custom/built-in section definitions; duplicate
registration rejects; compatible explicit replacement; removal preserves durable state and stored
rendered fallback; registry changes during preparation do not alter its captured definitions, while
later preparation sees the replacement. Unregistered entry kinds are reported without history scans
and stored facets still derive context; missing foreground kinds become orphaned, missing background
kinds remain parked and reported, registration restores recovery, terminal history stays untouched;
removal with live instances rejects. Replace by name keeps `h.kinds.<name>` consistent; `settings` round-trips; scoped hooks run after
harness-wide ones, innermost last; derived Call preserves typed private identity and telemetry;
stale/foreign task token rejects; close stops admission in one nonpersistent line job, joins outside;
shutdown atomically marks live tasks only, then permits abort cleanup; queued items/results remain
unchanged, including idle inbox-only conversations; crash/reopen child cleanup marks tasks only and
preserves queues; repeated close/shutdown share completion; close interrupt rejects shutdown;
lifecycle calls with task identity reject; delete rejects outstanding terminal invocations.
## 9. Generation kind
One stable generation task carries `inputs: Id[]` and cycles pending → streaming → retry_wait /
deferred → streaming until done / failed / aborted on a faux provider; explicit terminal results for
its whole input group; captured config (model, thinking, selected tools, profile, budget);
`system_instructions`, `before_request`, `after_response`, `on_yield`; retry sleeps in execute;
recover from frames; usage recorded per attempt.
**Durable configuration and typed preparation.** Config remains ordinary scoped state, persisted
when changed with its declared rewind/sticky policy. Host files and catalogue contents are not
magically stored as config. The Call-final `system_instructions` hook receives captured config and
one shared mutable `SystemSectionDraft` as `sections`. Sections are mutated, not returned; a handler
may return `{ tools?: readonly Tool[] }` as a full desired loadout. Handlers run sequentially in
registration order, with outer scopes before inner ones.
Seed each new draft from canonical durable section payloads and rendered text. Typed `get(token)`
returns a copy; changes require `set(token, value)`. Setting an existing key retains its position;
a new key appends; `delete(token)` explicitly removes it. Array payloads use typed get+set, without
an append operator. Missing contributions or registered definitions are not deletions. An untouched
seeded section retains its stored rendered text, including after restart; an unavailable renderer
uses that fallback. Initial sections need a provided base payload or stored fallback.
Explicit set/wrap or renderer replacement recomputes through the captured registered renderer.
`wrap(token, rendered => string)` requires a registered section; wrappers are synchronous, pure and
applied in registration order after rendering its payload, not on top of already wrapped stored text.
Wrappers belong only to the current draft and reset each preparation. Host handlers refresh full
current base payloads before plugin transformations; this intentionally rebuilds wrappers. Preservation
of a missing wrapper across a refreshed base is not promised. The composed transformation CHAIN must
reach a fixed point under repeated preparation, or use an earlier authoritative base reset. Individually
idempotent handlers are insufficient; the harness does not add a generic convergence loop.
Disk discovery and caches remain private to host/hook closures or services, refreshed by their own
watcher/TTL policy. Tokens have no source-read callback and entries retain no callback state. Refresh
failure is not deletion. Under the existing skip-failed-handler policy, discard that handler's draft
mutations, including wrappers, while retaining earlier handlers' changes; never publish a partial delete.
After hooks, capture payloads and pure rendered results outside the line. No plugin renderer runs on
replay or on the commit line.
**Canonical section state, separate from model projection.** Fold fork-visible managed system data
through the target, starting at the most recent baseline, independently of model heads and baseline
supersession omissions. Those facets control model projection, not canonical section payloads. Use existing indexed
kind scans, not unrelated transcript scans or a new storage API. An optional per-handle cache retains
current section state and its prepared cursor, not full history; each fresh baseline checkpoints all
canonical sections. There is no additional sticky full-state write. Missing contributors therefore
survive restart and compaction even when their original baseline is outside the retained model range.
Compare both payload and final rendered text by key, never by parsing prose. Baseline data contains
complete ordered set records; delta data contains only set/remove changes. JSON null and empty string
are values, not removal. A payload-only change persists a metadata-only managed delta with `model: []`
when there is no independent tool change. A renderer/wrapper change with unchanged payload emits a
model update when rendered text changes. Pure reorder emits no entry or order-change delta. Render
baselines and simultaneous changes in draft order with generic initial/change/remove labels by stable
key; historical messages are never reordered. A system entry records canonical instructions PREPARED
for a request, not proof the provider received them; appending it invokes no provider.
Tool differences remain independent of section differences. Complete definitions occur only in
SystemMessage `toolsAdded`/`toolsRemoved`, never SystemData. Fold removals before additions; additions
upsert by name, including same-name definition/schema replacement without a removal for that name.
A deletion carries the previous stored full definition, not a lookup in today's catalogue.
**Epoch preparation, not head-time behavior.** Every generation/provider request preparation checks
the newest visible head and effective managed system entries. A missing current-epoch baseline
requires a fresh baseline. Arbitrary managed-system edits reject; the current baseline's supersession
omissions do not invalidate it repeatedly. Rebuilding uses canonical payloads/rendered fallback from
the independent data fold, never just the model tail.
Append stored omission edits on that same fresh baseline entry for superseded retained managed
baselines AND deltas, never arbitrary system notices. Before preparation, generic context may expose
dangling old deltas; no request bypasses preparation. Head writers stay generic. The fresh baseline
remains at its appended position after the retained tail, not in a hidden prepended slot.
Apply planned omissions before folding tool declarations from ALL remaining effective SystemMessages,
including non-managed messages. A fresh baseline adds the complete desired tool set and explicitly
removes unwanted remaining declarations; `baseline: true` is not a pi-ai tool-map reset. Never emit
removals derived solely from messages that the same preparation is about to omit.
Capture canonical section state/cursor and the section registry snapshot before hooks. After hook and
render completion outside the line, verify on the line that no concurrent managed section-state change
occurred, including metadata-only changes. If it changed, restart preparation outside the line from
fresh canonical state. A compaction-only head change does not stale the payload draft: recompute only
the model baseline/delta choice, planned omissions and effective tool differences on the line.
Atomically persist any baseline/delta (including metadata-only), inflight status and
`state.requestThrough` on the prepared generation variant. Capture the cutoff even when no system
change is needed; unprepared variants have none and deferred variants retain their request's cutoff. After
persistence, the same line operation catches the current model cache up to the cutoff and captures an
immutable array of effective entry references, including immutable replacement projections, before
releasing the line. Current caches advance independently; the invocation keeps its snapshot, not mutable
cache containers. Clone messages only for request-local mutating normalization/hooks. Cold/recovery
reads at an older cutoff derive from storage without rewinding current caches. Release snapshot and
captured registry references after invocation; no version registry.
**Request-local overrides.** Preserve arbitrary `before_request` message transformations on a
request-local copy; they cannot mutate stored entries or switch Pico out of messages-only mode.
Canonical prepared state remains the basis for later diffs. Exact transformed requests are not
reconstructible from that state unless explicitly captured; no mandatory second request ledger.
Validate returned tool calls against the actual offered definitions after transformation, while
retaining implementation availability and permission checks (group 10).
Tests:
- Typed built-in/custom tokens; owned-copy get and explicit set; shared-draft registration/inner order;
existing-key replacement retains position, new key appends, explicit delete differs from null/empty
payload, array updates use get+set, and wrappers require registered sections.
- Untouched seeds retain rendered text; explicit set/wrap and renderer replacement recompute; wrappers
compose in registration order and reset per draft. Authoritative base refresh rebuilds wrappers;
missing wrappers need not survive that refresh. Check stability of the complete transformation chain,
not merely its individual handlers, and repeated preparation with earlier authoritative base reset.
- Missing contributor/definition and failed refresh preserve stored payload/text; a skipped handler's
partial mutations cannot leak. Registry remove never deletes section state; re-registration and
compatible replacement work. No renderer runs on replay/line; no closures enter persisted records.
- Persist payload AND rendered text; changed payload/unchanged text gives metadata-only `model: []`;
unchanged payload/changed rendering gives a model update; explicit removals and reorder-only no-op.
Baseline/change rendering follows draft order. Tool changes remain independent, including tool-only
empty content, additions/removals, same-name schema changes and `addTools`; no duplication in data.
- Config change commits before the next system entry; crash after config change preserves it; crash
after system append before invocation does not duplicate unchanged canonical preparation. Canonical
recovery folds metadata-only records and never requires the original contributor/renderer.
- Independent canonical fold across fork cutoffs, model heads and supersession omissions; missing-plugin
sections survive an original baseline outside the model range and appear in the next full baseline.
Repeated compaction uses the latest canonical checkpoint without an unbounded handle history cache.
- Compaction/reset/handoff and arbitrary plugin heads; a retained old model baseline still requires a
new epoch baseline; retained tail order, baseline-time omissions and preserved notices; fork before
and after preparation uses its own canonical state/inherited config without changing the source.
- Arbitrary managed-system omit/replace edits reject; fresh-baseline supersession commits controls and
instructions together without changing the seed's canonical data. Superseded omissions do not trigger
repeated baselines. Compute tool removals after planned omissions; include remaining non-managed tool
declarations. Generic projection runs no system-kind code.
- Concurrent canonical writes during hooks/render, including metadata-only deltas, force preparation
retry outside the line; registry changes preserve the captured snapshot and affect later prepares.
Head-only changes recompute epoch/omissions without discarding the payload draft. A later head during
projection/streaming cannot change `requestThrough`, including when no system entry was appended.
- Preparation snapshots include baseline omissions; later heads, edits and cache updates cannot change
captured entries or replacement projections. Normalization/hooks mutate message clones only.
Cold/recovery derivation at an older cutoff leaves current caches unchanged; invocation completion
releases snapshot references without a version registry.
- Request-local overrides leave stored state unchanged, remain messages-only and determine actual
offered-tool validation; adapter fixtures are gated on the pi-ai prerequisite in group 20.
- In-band provider abort without a durable mark has a kind-level outcome; post-mark execute settlement
rejects and fresh abort writes optional partial, cancelled input results and known usage atomically.
No mark branch in normal settlement; missing post-cutoff usage is unknown; crash while streaming
publishes the partial; retry budget exhausted → failed with no successor.
## 10. Tools, post_tools, exchanges
**API redesign pending:** retain the capabilities and behavioral tests below; simplify the tool,
sink and preview authoring interfaces together before implementing them.
The tool kind with the sink (`ToolOutput`, `ToolOutputState`, limits enforced by the sink, `diag`,
`delegate`, `handoff`, `addTools`, `terminate`), tool-result entries with structured data plus their
materialized model message, `before_tool` (fail-closed) and `after_tool`,
replay policy on recover; post_tools with `after`, carried input groups, terminate / handoff / steer /
next generation; `accept` idle vs busy; `prompt`, `result` and request acceptance lookup. Validate
calls against the actual definitions offered by the prepared request after request-local overrides,
not merely captured selected-tool names or today's catalogue. Preserve registry and permission checks.
Tests: transformed offered-tool definitions, removed tools and same-name schema changes; parallel tools
completing in either order; sequential via `after`; an aborted generation
creates no tool tasks/results while an aborted existing tool writes its own error result;
`new_context` resets after the exchange, never inside it; `addTools` writes the rewindable loadout
before any handoff/user entry in the settlement commit and appears in the next turn's `toolsAdded`;
a throwing tool → error result, `terminate` still honoured; truncation diag from the sink; a lost
accept response is recovered through `acceptance(requestId)`; a duplicate create reports the first
receipt without comparing payloads or modes; results remain point-readable after further turns.
A missing tool implementation produces an ordinary error result through the registered tool task kind;
the generation/post_tools input owner remains present. Separately, post_tools handles orphaned tool
tasks by writing unavailable-tool results from their common fields.
Turn-task entry appends remain immediate; outside-turn model-visible writes queue while inTurn is
nonempty and land at post_tools/final boundaries. Data-only entries are never blocked.
## 11. Inbox
`pi.inbox` as a conversation sticky list whose element id is `inputId` and whose value holds mode,
user content or a write's entry draft and optional request id; append/remove/clear watch operations;
queued/placed/done/unanswered result variants; the three placement points; carried generation/post_tools
input groups; `queueInput` and `abortInput`; abort draining steer and followUp while preserving write
and nextRun. A write reaches done without an answer in its placement commit. Input-group ownership
and results are built-in harness behavior, implemented by generation/post_tools; compatible replacements
retain that contract. No generic plugin-payload inference or new input-owner protocol is needed.
Tests: idle append/remove is one commit and emits no inbox event; busy image payload is one append
operation; the modes table; steer joins at post_tools but starts a group after a final answer;
followUp starts the next group; nextRun waits for idle accept; writes are placed without joining;
several inputs resolve to one answer; cancel/land and abort/group-transfer in both orders; queued and
placed crash recovery; cancelled unplaced payload is unavailable; input queued during collapse lands.
## 12. Collapse
Manual, threshold and overflow; `before_collapse`; publish only if no newer head; the overflow
chain (generation settles → collapse → new generation carrying the attempt).
Tests: a summary lands under a running generation and later entries stay in context; a competing
head makes a summary stale while intervening edits do not; overflow retries once and no live task
ever waits on the collapse; threshold before a turn; abort of a running collapse leaves appends flowing.
A head adds no implicit system omissions: next request preparation writes them with its fresh baseline.
Test retained deltas before preparation and their stored omission afterward, without invalidating an
already-prepared request at its frozen cutoff.
## 13. Subagents
The `subagent` tool (`run`, `spawn`, `send`, `status`, `wait`, `stop`), ownership links, foreground
reach through live owners, `run`'s recover driving the child again, `spawn` initialization of config,
explicit child input results.
Tests: restart in the middle of `run`; conversation abort reaches a `run` child and spares a `spawn`
child; abortTask marks only owner and fresh cleanup cancels its child; drive waiter observes Call;
cleanup tolerates already-terminal child/job; `stop`; nested children; child's own baseline.
## 14. Jobs and the budget
**Authoring API provisional:** job-first execution remains the initial supported approach. How tool
code starts, observes and delegates work is part of the tool/sink/preview simplification; the current
helper/sink signatures are not final.
`jobKind` on `ExecutionEnv.exec` with output into its scratch, `waitForTask` / `jobOutput`, `bash`
delegating first and waiting with the budget, `notify` and the `notice` entry, the `job` tool,
schedules.
Tests: job creation atomically stores job id and cancellation policy on the tool; fresh tool abort
uses those durable references, not execute locals or catch writes; budget expiry settles delegated
and job continues; delegation requests notification in separate sticky state, not a patch to an owned
job; both notification/completion orders publish once for exited, lost and killed; schedule cycles one
stable id; foreground drive
ignores background recurrence; recover → lost or safe rerun; abort kills non-detached job.
Remaining integration design: arbitrary-promise budget adoption must explicitly transfer effect and
sink ownership before the tool invocation releases. A raced, abandoned promise is not permitted.
Preserve the capability (crash outcome lost), but do not turn arbitrary-work adoption into an
implementation package until the simpler tool contract and ownership transfer are settled.
## 15. Previews
**API redesign pending:** retain incremental live output and reconstruction from durable scratch.
The tracker/sink-facing API below is a sketch to simplify, not a separate framework to implement first.
Delivery coalescing is deferred; do not add a frame scheduler or delay scratch durability for it.
`runtime.preview` as a Chord tracker per task; the generation applies stream events to a partial
message, the tool's preview is its sink state, the job's the same; `preview.init` on attach and
reopen; flush after each scratch commit.
Tests: one token → one `a` op and nothing else; no `r` mid-stream; init after reopen yields a base
equal to the live preview; a sliding tool tail → `t` + `a`.
## 16. Watch
`ConversationView`, `ConversationEvent`, the exported kind-free `applyEvent`, `WatchHandle`
(capture on the line, bounded buffering, `resnapshot`, `unsubscribe`), the session watch with
`report` and `usage`, the usage ledger (`pi.usage` + totals).
Tests: the fold is correct (view after N events equals a fresh capture, randomized); head and edit
entries update derived context; inbox append/remove/clear operations update the view and same-commit
append/remove cancels; all events of one commit delivered together; a thin-client reducer over a
recorded stream with no kinds loaded;
lag → fault → resnapshot; `resnapshot` from inside the listener; usage totals equal the ledger
fold, failed and aborted attempts included.
## 17. JSONL storage
Append-only batches, replay on open, torn tail discarded, malformed line fails open, entry
`data`/`model`/`head`/`edits` and the kind-string set from replay. Whole value sets and intrinsic list
operations in main and scratch files; no Chord storage codec. Replay main first, then surviving live
scratch; per-file sequence gaps are valid. Recover lastSeq from maximum complete surviving batch
endpoint, including clear/remove. Retired scratch is ignored; its later settlement covers its ids.
Physically remove torn suffixes before appending; malformed complete replayed batches fail.
Tests: conformance against memory; full replacement values on disk; compact incremental frames/output
operations avoid repeated growing snapshots; main=100/live scratch=150 reopens at 150; settle=151
with failed unlink ignores retired scratch, even malformed; clear/remove high-water; torn main and
live scratch tails; nonoverlapping ranges; accepted payload plus placement has two JSONL copies,
including idle acceptance. Backend memory and disk growth claims match whole-value behavior.
## 18. SQLite storage
Tables and indexes of §7.5, nullable entry JSON columns for data/model/edits and an indexed integer
head boundary, scratch rows deleted in the settle transaction, storage version and `migrate`.
Tests: conformance three ways; removed sticky inbox elements may be physically discarded while
input results remain point-readable; a cold reopen decodes live tasks and only named required owner
records, including terminal owners; no retained terminal payload cache; a version mismatch rejects.
## 19. Race matrix and telemetry
The §10.2 matrix in both orders with faux clocks, fake processes and storage barriers; spans per
task call and per commit.
Tests: the matrix; span tree for one turn with a tool and retry; nested Call preserves active telemetry
parent and private invocation identity; drive-caller cancellation does not become the task signal;
providers/env interpret signals, hooks propagate cancellation rather than swallowing it; no callback
runs on the line; stale RPC-bound invocation rejected. Metadata transport never carries task authority.
## 20. Clients and pi-ai integration prerequisite
mini (`worker/run.ts`, `worker/lane-service.ts`, TUI `apply(view)`), the experimental agent's four
seam files (`session-worker.ts`, `agent-controller-provider.ts`, `models-provider.ts`,
`transcript-provider.ts`), real providers. The integration milestone remains: system deltas on a live
model, a retry, a spawned subagent surviving a restart, speculative compaction under a running turn.
**External prerequisite, not landed functionality.** PRs
[#9116](https://github.com/earendil-works/pi/pull/9116) and
[#9117](https://github.com/earendil-works/pi/pull/9117) are open dependencies in this design round.
Their agreed target and fixtures must be delivered and verified before integration; neither an open
PR's types nor this plan establish that the behavior already exists.
Target: pi-ai enters system-message mode when BOTH `context.systemPrompt` and `context.tools` are
undefined. Pico supplies only `{ messages }`, never parallel top-level instructions/tools. SystemMessage
carries text plus complete JSON `toolsAdded`/`toolsRemoved` definitions. Provider/model translation
belongs to pi-ai: unsupported mid-conversation changes become user messages bracketed with `<system>`
at their historical positions, and adapters derive required bulk wire tool declarations from message
history. Pico does not implement fallback or flatten changes into a rewritten top-level prompt.
Cache preservation is best-effort, not a universal prefix-cache or instruction-priority guarantee.
Required adapter fixtures: both top-level fields absent versus either supplied; ordered baseline and
section updates; empty-content tool-only changes; removals and same-name upserts; historical calls to
removed/replaced tools; historical-position fallback; compaction with retained tail, stored omissions
and an appended fresh baseline. Include messages-only preparation for summarizer/provider request
paths, not just ordinary generation. Run faux fixtures without provider credentials; live integration
waits for the dependency contract and separately authorized smoke tests.
The current sketch puts groups 1–16 before this milestone and 17–19 later. That ordering and those
sizes are provisional, to be verified and split at the final planning gate below.
## Final review and work-package gate
After the remaining conceptual blockers are settled, perform an independent whole-document audit of
spec, guide and this plan: reconcile APIs, invariants, examples, cross-references, external prerequisites
and coverage while preserving agreed features and explicitly tracking residual integration questions.
That audit is still ahead, not completed by the blocker-7 update.
Then replace the provisional groups with small, self-contained, testable, human-reviewable work
packages. Each needs explicit prerequisites, bounded scope, interfaces and acceptance tests, with a
green incremental verification step and no unresolved forward dependencies. Do not treat the current
numbering or package sizes as the final implementation decomposition.
## 21. Runtime schema bundle
`harness.schema(call)` walks the three registries and emits one document describing everything a
client can see in this session: the core protocol types (`ConversationView`, `ConversationEvent`,
`InputResult`, `QueuedInput`, `DeltaOp`), plus each registered entry kind's `data`, each task
kind's state union and preview, and each tool's parameters and details. Generated at runtime, not
at build time, because the interesting half is per installation: the plugin kinds and tools that
happen to be registered. A client fetches it once per session and can then read `Entry.data`,
`Task.state` and previews from plugins its authors never heard of.
Schemas are optional per kind. A kind that declares none is still usable; its payload is opaque
JSON to a foreign client, which is the generic-fallback case a renderer already handles. Declaring
is progressive: add a schema to the kinds you want third-party clients to understand.
Two open options for how a kind declares one, to be picked after trying the first:
- A small descriptor owned by pico (about twelve cases: scalars, literal, array, object with
optional fields, tagged union, ref, unknown) with `Static<S>` deriving the TypeScript type, plus
adapters `toJsonSchema` / `toTypeBox` / `toZod` and `fromTypeBox` / `fromZod` for authors who
already declare with a validator. Pico then depends on no validation library. Cost: hand-rolled
conditional types with worse errors than TypeBox's, and adapters must reject what the descriptor
cannot express (refinements, formats, dynamic keys) rather than silently dropping it.
- JSON Schema as the descriptor, since tool `parameters` already is one, with adapters only for
authoring convenience. One representation fewer; authors without a validator library write JSON
Schema by hand.
Schemas describe, they do not validate: stored objects are trusted (§7.3) and validation belongs at
wire boundaries, so a host that wants strict ingest validation opts in. Versioning is per kind, not
global; a kind that changes its shape bumps its own version, which is also the migration signal.
Needed when a client is not JavaScript. Not part of the gate.
Not Pico packages: approval/question policy, presentation, durable answers and their replay are
workspace/plugin responsibilities. `before_tool` may block, rewrite args or wait; existing scratch
and scoped values provide persistence. The workspace/plugin layer chooses answer keys and lifetimes;
Pico's remaining integration question is authorized hook access to task identity/scratch, not a new
memo protocol. Session migration remains outside this experimental scope.
## What comes from the lane harness, and how
Clean room means no imports from `src/harness/runtime`, `session`, `agent-harness.ts` or the
`dom`/`pico`/`pico2` spikes. Where the old code has something worth keeping, it is **copied** into
`src/harness/pico/` and owned there; pico must build with the rest of `src/harness` deleted.
Copy (under `packages/agent/src/harness/` unless noted):
| what | from | into | package |
|---|---|---|---|
| `ExecutionEnv` / `FileSystem` / `Shell` types, the Node env, capture and spill | `types.ts`, `env/`, `tools/tool-context.ts` (03-execenv) | `pico/env/` | 10, 14 |
| shell output limits, `applyShellOutputUpdate`, truncation totals | `utils/` | `pico/env/output.ts` | 10 |
| the built-in tools (`read`, `write`, `edit`, `bash`, `image`) | `tools/*.ts` | `pico/tools/`, adapted after the tool/output interface is simplified | 10, 14 |
| system prompt helpers, skills, context files, templates | `system-prompt.ts`, `skills.ts`, `prompt-templates.ts` | host handlers refresh typed base payloads; pure section renderers format them, without a source-read framework | 9, 20 |
| telemetry span helpers | `telemetry.ts` | `pico/telemetry.ts` | 19 |
Depend on, as packages (they are not the old harness):
- `@earendil-works/chord` Context types and `@earendil-works/chord/context` helpers (1, 6, 8, 19)
- `@earendil-works/chord/delta` (15, 16: preview/watch only; not storage)
- `@earendil-works/pi-ai`: `faux` provider for tests, `utils/estimate` for thresholds; SystemMessage and
messages-only adapter behavior depend on the verified PR #9116/#9117 target above (4, 9, 12, 20)
Read before writing the equivalent, then close the file:
- `runtime/drive/retry.ts`, `deferred.ts`, `response.ts`: retryable-error classification, overflow detection, deferred polling (9)
- `compaction/`, `runtime/drive/boundary.ts`: the exchange-cut rule and the summarizer prompt (12)
- `runtime/drive/recovery.ts`, `restore.ts`: the per-phase recovery case list (9, 10, 14)
- `runtime/drive/tool-placement.ts`: the result order projection must reproduce (4)
- `hooks.ts`, `docs/harness.md` §before_tool: the exact decision shape (10)
- `session/`, `test/harness/jsonl-*.test.ts`: torn-tail and malformed-line rules (17)
- `session/`, `test/harness/mutation-line.test.ts`: line discipline edge cases (3)
- `agent-harness.ts`: `LaneSnapshot` / `HarnessEventPayload`, for the `toLaneSnapshot(view)` shim only (20)
Tests to port (`packages/agent/test/harness/`): the three conformance suites as the model for 17–18;
`compaction`, `branch`, `context`, `execution-*`, `values`, `mutation-line` as the parity source for
19; `system-prompt`, `prompt-templates` for the host handler in 9; `tools`, `truncate`,
`output-capture` for 10; pico v1's 27 ported scenarios as the shortest list of behaviours to
re-prove.
Do not read: `runtime/lane.ts`, `reducer.ts`, `drive/reconcile.ts`, `structural.ts`,
`terminal.ts`, `checkpoint.ts`, and the `dom` spike. They are the operation state machines and the
reconciler this design replaces.

View file

@ -0,0 +1,473 @@
# Harness V3 hardening handoff
## Goal
Harden the implementation in `harness-v3.zip` without reverting its core design. Return a new standalone zip for review. Do not integrate it into the pi repository yet; the accepted implementation will later be incorporated under `packages/agent/src/harness/pico3/`.
Baseline archive:
- `~/Downloads/harness-v3.zip`
- SHA-256: `1c69c5c5d923fe1c80200cc5e49ddb3868158e1b6a844efa3e2f03eb24ff5be7`
- Current runtime suite: 67 passing tests
- Current kernel + kinds: approximately 3,448 lines, excluding storage backends
The current code is a good executable design spike. Preserve its legible fixed-core, document/view, and phase-map architecture. Correctness takes precedence over preserving the exact line count.
## Companion documents and supersessions
This copy in `packages/agent/docs/pico/v3/` is canonical. Two companion
documents in the same folder were written after this handoff and take
precedence where they overlap:
- `view-and-events.md`: the watch protocol, the single view document, ops by
path, named events, the commit-by-commit mapping, watchers and late joiners.
- `plugins.md`: the harness registration surface, the four durable primitives
(entry, checkpoint, slot, namespace), tool/hook/custom-kind/entry/namespace
extension points with examples, unregister/re-register/reload, and the Chord
facet layer.
Specifically superseded in this document:
| here | replaced by |
|---|---|
| Accepted direction #3 (separate `rewindable`/`sticky` view fields with separate op lists) | one flat view document with `config`, `inbox`, `turn`, `compaction`, `tasks`, `plugins` and one op list (`view-and-events.md` §2–§4); the storage split stays |
| Accepted direction #2 "live task slots ... in the view" and §13 last paragraph (`TaskRef<K>` slot typing) | slots remain, but the view publishes kind-described `tasks[id].status` via `describe()`, never raw task records (`view-and-events.md` §2.2, §6.6; `plugins.md` §4.3) |
| §9 Watch protocol: envelope shape, `Seq` ordering and `applyDelta` | `Envelope { revision, ops, events }` with a contiguous per-watch `revision` (storage `Seq` is not exposed), folded with `applyImmutable`; the active-transcript definition and capture rules in §9 here remain normative (`view-and-events.md` §3, §5, §9) |
| §9 bounded capacity / overflow-close | the raw watch invokes its listener synchronously in order; only the pre-`start` buffer is bounded (overflow closes the watch); asynchronous consumers own their queue (`view-and-events.md` §3; `plugins.md` §6.1 for the Chord bridge) |
| §13 plugin state visibility | a namespace is in the view only through its declared `view` projection; memos and slot working state are never in the view (`plugins.md` §1.5, §2, §4.5) |
| §15 `beforeTool` throw = block; hooks by name | hooks are registered with their namespace token (`h.hooks(ns, kind, handlers)`); `BeforeToolApi.waiting(ctx)` is asynchronous; memo keys are `(task, namespace, name)` (`plugins.md` §4.2) |
| §16 / view §6.2 generation failure closures | every generation failure resolves the group `unanswered/failed`, runs the final boundary and starts a successor for queued triggers (`view-and-events.md` §6.2) |
| §4.4 entries / §1 `pi.*` | `pi.*` entry names reject for non-core callers except the allow-listed `pi.notice` (`plugins.md` §4.4) |
| §13 plugin state (`plugins[pluginId]` slices, extra document shard) | `h.namespace(ns, defaults)` tokens, `tx.plugins(ns)`, `tx.emit(ns, …)`, lazy seeding; extra shards stay deferred (`plugins.md` §2, §3, §4.5) |
| §15 tool API (candidate) | `ToolApi` with `memo` (slot memoOnce), `waiting`/`memo` on `BeforeToolApi`, `waitingOn` on the tool slot (`plugins.md` §4.1–§4.2; `view-and-events.md` §2, §6.3) |
| §1 "install fixed built-ins internally" / registration at open | registration capabilities callable while open, before-`resume()` rule, `quiescent()/hold()` for idle reload and `suspend()/reopen` otherwise; no handler-withdrawal machinery in v1 (`plugins.md` §3, §5) |
| §16 `pi.plugin` handlers map | folded into custom task kinds; `pi.plugin` may stay as a convenience kind but is not the plugin mechanism |
Everything else here (authority explanation, proxy lifetime, prospective busy,
display-only assistants, read semantics, phase-map contract, session line,
waiters, storage atomicity, terminal retention, strict types, context/sections,
tool/process behaviour, known defects, test matrix, deliverable) stands.
## Accepted direction
These are decisions, not open questions:
1. **Fixed core.** `pi.generation`, `pi.tool`, `pi.post_tools`, `pi.collapse`, and the Harness methods built on them are fixed and unreplaceable. `pi.job` and `pi.plugin` are built-ins too, though they use ordinary-task authority. Plugins cannot register any `pi.*` task kind or replace a built-in.
2. **Document state.** Keep Chord-delta-backed session, conversation-rewindable, and conversation-sticky documents. Keep the core rendering fields (`inbox`, `turn`, live task slots, core config) represented in the authoritative conversation view.
3. **Rendering-oriented view.** Keep `rewindable` and `sticky` as separate top-level `ConversationView` fields with separate Chord op lists. Keep commit deltas easy to fold and render. A renderer should not reconstruct turns by correlating unrelated channels.
4. **Phase-map tasks.** Keep `initial` plus an exhaustive phase map. Do not restore a separate handwritten `execute`/`recover` pair.
5. **One implementation, few capability views.** One internal `TxImpl` is desirable. Expose narrow host/ordinary/core transaction types or façades over it rather than a large hierarchy. Runtime checks remain authoritative.
6. **Extensibility.** Preserve custom entries, custom phase tasks, tools, core hooks, system sections, typed config, jobs, owned conversations/subagents, plugin state, and compatibility with separate Chord RPC/presentation services.
7. **Single process.** Do not add cross-process locking. Document that a storage path has one owning process/session at a time. Reject a second Session over the same Storage object in-process; behavior from two processes opening the same JSONL path is unsupported.
8. **Commit-granular document history.** A fork at an entry sees the final rewindable document state of the atomic commit containing that entry. Document this explicitly. Do not reintroduce per-write `Seq` or rewindable-before-entry ordering unless a concrete failure demonstrates it is needed.
9. **Separate stable IDs and commit sequence.** Object IDs remain stable identity. `Seq` orders atomic commits and watch envelopes.
10. **No terminal pruning.** Terminal task records and outcomes remain durably readable. Terminal checkpoints may be dropped once no downstream contract relies on them; `ToolInput.offered` already removes the generation-checkpoint dependency.
## What `boundary()` and `resolveInputs()` mean
These are core authority and must not be exposed to ordinary tasks.
### `boundary(conversationId, at, headBoundary)`
`boundary()` implements inbox placement at a safe turn boundary. `headBoundary` is the current retained transcript boundary, supplied by core code so `boundary()` performs no storage scan after same-batch appends. It:
1. reads the ordered sticky inbox;
2. finds a queued self-head write (reset/handoff) and marks older steer/follow-up inputs stale;
3. selects every passive write, steer inputs according to `steeringMode`, and at a final boundary follow-ups according to `followUpMode`;
4. places selected entries strictly in global input-ID order;
5. updates their durable `Input` records;
6. removes selected/stale items from the inbox;
7. returns trigger input IDs and whether a self-head ended the old input group.
An ordinary task calling it could drain input in the middle of an assistant/tool exchange, stale requests, place a reset at an invalid point, or consume triggers without creating the required successor generation. Only fixed core turn code may call it.
### `resolveInputs(ids, resolution)`
`resolveInputs()` controls the durable lifecycle observed by `InputHandle`: done/unanswered, answer ID, reason, and detail. Generation, post-tools, failure, termination, and abort paths call it only when the corresponding transcript/task transition commits.
An ordinary task calling it could claim another input was answered by a fake entry, abort unrelated requests, or wake waiters before the transcript contains the asserted result. Keep `setInput`, `resolveInputs`, and direct input-table writes core-internal.
## Proxy lifetime requirement
`Proxy.revocable(target, handler)` returns `{ proxy, revoke }`. After `revoke()`, every operation through that proxy throws `TypeError`.
Revoking only the root document proxy is insufficient because a nested proxy can escape first:
```ts
let escaped: object;
await c.commit((tx) => {
escaped = tx.rewindable(c.id).plugins;
}, ctx);
// Revoking only the root does not necessarily revoke `escaped`.
```
Implement a transaction-scoped **revocable membrane** over the cached Chord tracker:
- every object/array returned from a document proxy is wrapped lazily;
- all wrappers share one active/revoked state;
- preserve object identity inside the transaction with a `WeakMap`;
- mutations forward to the underlying Chord proxy so its tracker still records ops;
- at transaction finish, callback failure, validation failure, or storage failure, revoke every wrapper;
- retained root or nested handles throw after the callback;
- no raw nested Chord proxy may escape through `get`, iteration, descriptors, or array access.
If a correct membrane is too invasive, add tracker-level revocation to Chord so every nested proxy checks one tracker lifetime flag. Do not rely only on documentation or TypeScript to prevent escape.
Tests must retain both a root and nested object/array, mutate them after commit/failure, assert a throw, and prove no later unrelated commit persists the attempted mutation.
## Required hardening
### 1. Fixed registry and token identity
- Install fixed built-ins internally.
- Reject duplicate task-kind names.
- Reject plugin task names beginning with `pi.`.
- A user-supplied `{ core: true }` must confer no authority.
- Use current registered token identity for typed task creation and hook registration; a stale/redeclared token rejects.
- Make hook and registry unsubscribe functions idempotent. A second call must not splice/delete an unrelated newer registration.
- Reject or define replacement semantics for duplicate tool, entry, and section names. Unregister must only remove the exact object it registered.
Tests: attempts to replace `pi.generation`, forge core authority, use a stale token, and unsubscribe twice.
### 2. Capability surfaces and scope enforcement
Keep one `TxImpl`, but expose at most these author-facing capability views:
- **HostTx:** reads; plugin/application document state; passive `write`; create ordinary tasks and conversations. No direct core task creation, entry append, boundary, or input resolution.
- **TaskTx:** reads in the task's conversation/owned subtree; own checkpoint; own typed slot/plugin state; passive write in its own conversation; create ordinary tasks and owned conversations. No admission/boundary/input authority.
- **CoreTx:** internal full authority for fixed turn machinery.
- Abort mode must have its own runtime checks even if its public type is an `Omit`/narrow view.
Runtime authorization must cover every method, not only `appendEntry` and `createTask`:
- `send`, `boundary`, `setInput`, `resolveInputs`, raw core document fields, core task creation, and core entry append are internal;
- ordinary tasks cannot mutate `sticky.inbox`, `sticky.turn`, or another task's slot even via a cast;
- a task may access only its conversation and conversations it owns;
- parent selection for an owned conversation must stay in that subtree;
- host APIs invoke fixed core capabilities through internal tokens, not user-controlled strings.
Add `kind` and an unforgeable invocation token to the internal task invoker. Every runtime commit checks that the token is still the active invocation, the task is live, and an ordinary run invocation is not marked. A captured runtime cannot write after the method returned, after terminalization, or after a durable mark.
Tests must include malicious casts so runtime checks, not only TypeScript, are exercised.
### 3. Prospective task/turn state
Fix `busy()` and boundary decisions to use the prospective state of the current atomic commit.
Concrete failing trace today:
```text
generation terminal closure
→ tx.write(notice)
→ busy() sees the same generation as live
→ notice is queued
→ generation terminalizes
→ no boundary remains to place it
```
When executing a terminal closure, the current task counts as gone. A same-batch terminal `task.patch` also counts as gone; a same-batch created live turn task counts as present. An idle terminal-closure write appends immediately.
Regression tests:
- overflow with nothing collapsible appends its notice and leaves no inbox item;
- an idle job notification appends immediately;
- a same-batch successor generation keeps the conversation busy;
- no input/write becomes stranded when the last turn task terminalizes.
### 4. Display-only assistant entries
Provider-error and aborted partial assistant messages are presentation history only:
- they never enter model context;
- they never create tool work;
- later requests do not include them.
Prefer omitting `model` and storing the display message in strict-JSON `data`, or add an equally explicit non-context projection facet. Do not make `deriveContext` infer this from an arbitrary error string.
Tests: abort/error, send another input, inspect the next provider request, and assert no aborted/error assistant message appears.
### 5. Read semantics
Tracked document reads have read-your-writes inside one transaction. Keep that advantage.
Apply one rule to every read: if buffered writes can change its answer, the transaction must either provide a complete explicit overlay or reject it. Do not silently return committed pre-transaction state.
Maintain complete tx-local overlays for direct object reads:
- `entry(id)` sees same-batch entry creation;
- `task(id)` sees same-batch task creation and every later patch;
- `conversation(id)` sees same-batch conversation creation;
- `input(id)` and request-key lookup see every same-batch input creation/replacement.
These direct reads remain valid after writes. This is required because terminal closures append their result entry and then resolve existing inputs. `putInput` must update both the ID and request-key overlays, not only `writes`/`effects`.
Scan-shaped reads have no general overlay. Reject a scan only after a buffered write that can affect that scan's domain:
- after an entry append: `newestEntry`, `scanEntries`, and `context` reject;
- after a task create/patch: task scans reject;
- analogous future table scans follow the same rule.
Use a named `ReadAfterWrite` error and poison the transaction even if the callback catches it. Refactor the known offenders:
- threshold-collapse selection reads context before appending the assistant and carries the result into the terminal closure;
- `boundary(conversationId, at, headBoundary)` never scans storage: the caller reads the newest head before its first append, or passes the materialized ID of a same-batch self-head entry it just appended;
- while placing queued writes in ID order, `boundary()` updates its local head boundary after each selected head write, so later explicit-head validation sees earlier same-batch heads;
- `send()` performs request-key lookup before writing;
- generation/post-tools compute context-dependent decisions before appending; post-tools passes a same-batch handoff ID as `headBoundary` when it just appended that self-head.
Document reads remain exempt because the tracked proxy supplies an actual overlay.
### 6. Phase-map contract
Keep the phase-map model, but encode phase role rather than relying on comments:
- a normal phase can be reached by a returned `next` transition;
- an in-flight/recovery phase is written immediately before an external effect and is entered by the scheduler only after reopen;
- the scheduler rejects a normal transition into an in-flight-only phase.
A transition builder may atomically return:
- a next checkpoint;
- a typed completion;
- `"retry"` when its snapshot became stale.
Add an explicit `Carry` facility if it removes repeated unchecked destructuring across phase checkpoints. A kind method or terminal closure that returns an invalid shape is a task-contract fault; do not invent an untyped `{ reason: "threw" }` outside the kind's declared failure type.
Retain and expand the existing recovery tests for generation, tool, collapse, job, post-tools, marked tasks, and ordinary custom tasks.
### 7. Session line, failure, and cancellation
- Every storage interaction, waiter registration, watch capture, registry mutation, reservation, and commit enters one FIFO Session line.
- **No `AsyncLocalStorage` / `node:async_hooks`, anywhere.** Nested-line detection uses Chord contexts only: the Session derives a line context for every callback it runs (`withContextValue(LINE_KEY, marker, ctx)` with a private `createContextKey`), passes it to transaction builders (`commit((tx, ctx) => …)`; task builders receive `(tx, current, ctx)`), and `commit`/`read`/`onLine` reject `NestedLineOperation` when the supplied `ctx` carries `LINE_KEY`. Inside a builder, authors use the builder's `ctx` for everything; calling the Session from inside a builder with an outer ctx is a documented, undetectable self-wait, not something the kernel prevents.
- Check caller cancellation before queuing and again before invoking a transaction callback.
- Once persistence starts, call `Storage.commit` exactly once with `withoutAbortSignal(ctx)`.
- A storage commit rejection faults the Session, closes storage with a non-cancellable context, and makes already-queued/later operations reject `Faulted`.
- Callback or validation failure before `Storage.commit` discards the transaction but does not fault the Session.
- Listener, hook-reporting, watch, and scheduler callback failures never turn a successfully persisted commit into a caller-visible commit failure.
- Do not swallow document/tool-stream persistence failures. The current `flushing.then(...).catch(() => {})` pattern must go.
- Enforce one owning Session per Storage object in-process. Successful close/fault cleanup releases it; failed close keeps ownership.
### 8. Waiters
Register waiters atomically on the Session line:
```text
enter line
read current task/input/idle state
if terminal, return immediately
otherwise install waiter
leave line
```
Cancellation only removes/rejects that waiter; it does not cancel durable work. Handle an already-aborted context. Add deterministic tests where completion is forced between the old read and registration points.
### 9. Watch protocol
Keep the rendering-friendly `ConversationView` and Chord document ops, but implement the settled watch behavior:
- capture and subscription are one line operation;
- committed envelopes are delivered in commit order with a contiguous per-watch `revision` (storage `Seq` is not exposed);
- one commit produces one envelope folded into the view before the listener runs;
- listener execution is off-line, synchronous and in order;
- listener throw or decoder failure closes only that watch and reports through `onError`; it never rejects the already-persisted writer;
- envelopes before `start()` are buffered with a bounded capacity (default 256); overflow closes the watch; no other queue exists in the raw watch;
- no cursor, replay, acknowledgement, deduplication, or resnapshot protocol; the client opens a fresh watch;
- `stop`/unsubscribe is idempotent;
- no callbacks after close/stop.
Define the initial entry capture exactly:
- Let `H` be the newest fork-visible entry carrying a head.
- If `H` exists, the active transcript is the chronological union of `H` and every fork-visible entry whose ID is at least `H.head`, without duplicating `H`. `H` remains at its actual chronological position.
- If no head exists, the active transcript is the complete fork-visible transcript in chronological order.
- Capture walks `scanEntries` pages newest-first until it passes `H.head`, using the same fork-aware ancestry walk as `deriveContext`, then reverses the collected entries. There is no public capture limit parameter and no internal fixed cap.
The captured transcript must retain all of the following:
1. fork-inherited entries when `H.head` points into a parent conversation;
2. chronological order—a summary/head entry stays at its written position and is not moved to the front as context derivation does;
3. inner or older head entries that remain inside the active range; unlike model-context derivation, the view does not drop them;
4. display-only entries, including aborted/error assistants, `pi.usage`, and model-less plugin entries;
5. every entry's edits verbatim; the view is the transcript, not the edited/model-message projection;
6. every later committed entry for the lifetime of the watch—there is no live cutoff.
When a committed entry carries a new head, `applyEnvelope` truncates the existing active transcript using this same fork-aware boundary definition and then appends the new entry at its chronological position. It must not apply context derivation's special treatment of heads. A fresh capture and a client that folded every committed envelope must produce identical `entries`.
Add a regression with a child conversation whose newest head points to an entry in its parent and with an older head entry still inside that retained range. Assert that capture includes the inherited target, preserves both heads in chronological order, and equals the result of incremental folding.
The view keeps fixed core fields for rendering. Plugin RPC/presentation DTOs may filter or project it through normal Chord services.
### 10. Storage atomicity and JSONL recovery
No cross-process lock.
MemoryStorage:
- validate the complete batch before mutating, or stage into copies and publish only after every write succeeds;
- one failed batch changes no table, document, ID high-water, or commit sequence;
- stable IDs never reuse once committed; callback-minted but uncommitted IDs may be reused only after reopen.
JSONL multi-file publication:
1. Assign one commit `Seq`.
2. Append and optionally fsync all sidecar records for that `Seq`.
3. Always append one `main.jsonl` record/commit marker last, even if there are no main-table writes. The marker lists the sidecar refs expected for that commit.
4. The main marker is the publication point.
5. Replay uses stored `Seq` values, validates monotonicity/shape, and applies sidecar records only when the corresponding main marker confirms them.
6. Unconfirmed sidecar tails are ignored or truncated.
7. Physically truncate bytes after the last newline before opening for append; otherwise a later append extends a corrupt torn line.
8. Sticky sidecar rewrite/truncate runs on the Session line and cannot race appends. Use temp file + fsync + rename as appropriate, but no directory/cross-process locking is required.
9. Carry committed ID high-water in every record needed to prevent reuse after retired/unlinked sidecars.
10. A malformed complete record fails open; a stale retired task sidecar cannot resurrect state.
Add fault-injection tests for a crash after each sidecar append and before/after the main marker. Graceful `Harness.close()` is not a storage crash test.
Durability modes and failure boundary:
- `fsync: false` covers callback failures, append failures, and process termination while the OS/filesystem remain alive. Complete published records recover; torn final lines are truncated; unconfirmed sidecar tails roll back.
- `fsync: false` does **not** promise that a returned commit survives power loss, kernel/VM-host failure, storage-cache loss, or a filesystem that loses or reorders completed writes.
- Worst case without fsync: acknowledged tail commits disappear. If sidecars survive without their marker, replay rolls them back. If a marker survives without a required sidecar, open fails rather than exposing a half-commit. If the lost commit was a pre-effect tool/job checkpoint or memo, recovery may attempt the external effect again; external idempotency is still required.
- `fsync: true` flushes each sidecar before flushing the main marker, strengthening file-data durability and ordering. It is not a complete database guarantee until parent-directory metadata for file creation, rename, and unlink is also synchronized, and it cannot strengthen a filesystem or device that provides weaker guarantees.
### 11. Terminal task retention
Remove `pruneTerminal` behavior that makes terminal tasks unreadable. `getTask`, `waitForTask`, task scans, dependencies, and typed outcomes must continue to work after arbitrary later activity and reopen.
Reduce retained size by:
- clearing terminal checkpoints when safe;
- storing compact complete terminal records or validated patches;
- unlinking live-only task sidecars after the terminal record is durably published.
Do not infer that every outcome is represented by an entry: job and plugin task results are not.
### 12. Strict durable types
Harden durable positions without rebuilding V2's entire address type system:
- `JsonValue` only; no durable `unknown`;
- `Stored<T> = JsonRepresentation<T>` for imported pi-ai values;
- stored entries/messages/tool calls/usage/deferred handles use `Stored<T>`;
- task input/checkpoint/result/failure/aborted values extend strict JSON;
- tool details, plugin inputs/results, document/plugin state, section payloads, and process state are validated/encoded before persistence;
- readonly durable snapshots at public boundaries;
- no `any`, no inline/dynamic imports, erasable TypeScript;
- use the repository's pinned `typebox` 1.x API after checking its installed types; do not use `@sinclair/typebox` 0.34;
- `ToolDeclaration.execute` receives `Static<P>` rather than `unknown` plus author casts;
- replace `never`-parameter registry erasure and unchecked casts with a small internal erased adapter whose validation boundary is explicit.
Make the standalone `tsconfig` portable: no `/home/claude` paths, exclude design-history sketches from production type checking, and include a real compile-test command. Remove unused `partial-json` if it remains unused.
### 13. Config, documents, and plugin state
Keep fixed core document fields and the flat typed `c.config` facade.
- Derive/seed defaults from registered kind definitions rather than duplicating them in `defaultRewindable`/`defaultSticky` and kind configs.
- Resetting a key is an explicit operation (`config.reset(keys)`; RPC `configReset`), never assigning `null`: storage deletes the override, the resolved view sets the declared default; only `model`, which has no default, is deleted from the view.
- Do not use `value ?? default`, because durable `null` is a legitimate value. Test property presence.
- Custom kind config defaults must be visible both through `c.config` and to the task itself before any explicit set.
- Reject config-key collisions at compile time where practical and always at installation.
For plugin state, first implement typed/namespaced slices inside the existing documents instead of recreating arbitrary value/list addresses:
```text
session.plugins[pluginId]
rewindable.plugins[pluginId]
sticky.plugins[pluginId]
```
At minimum, associate an owning namespace/type with a plugin or task-kind token so typos and cross-plugin writes are not silently accepted. If a plugin has large independent state, support a plugin-declared extra document shard using the same Chord fold semantics; do not add this complexity without a test demonstrating the cost/isolation need.
Associate a task slot type and initializer with its task-kind witness. `TaskRef<K>` should determine the slot type; remove caller-invented `slot<T>()` casts. Core turn/tool slots remain fixed and rendering-oriented.
### 14. Context, transcript, and sections
Preserve the current centralized context derivation and full system-section design:
- head/range/edit chronology;
- newest edit wins;
- tool results reordered to call order;
- missing fork-truncated results synthesized;
- messages-only requests;
- managed `pi.system` baseline/delta chronology;
- preparation snapshot/retry loop;
- historical tool additions/removals;
- section seeds before first preparation.
Fix and test:
- transcript/head invariants before persistence;
- display-only assistant exclusion;
- complete active transcript capture;
- private conversation section-seed metadata (do not expose it in public `ConversationView`);
- `SystemEntry` typing permits metadata-only `model: []` deltas;
- ancestry/subtree hook reconstruction after reopen;
- registry changes are line-serialized and preparation revision checks remain correct.
### 15. Tool and process behavior
Keep `ToolInput.offered` and the persisted final validated call. Recovery from `started` requires persisted-safe, current-safe, and current-schema-valid evidence; `beforeTool` never reruns after `started`.
The V3 tool API is a candidate, now no longer an unspecified gap, but harden it:
- `beforeTool` runs before `started`; a throw blocks;
- kernel owns bounded streaming;
- fix tail retention so a single chunk larger than `maxBytes` is sliced during `push`, counted as dropped, and never retained whole;
- enforce both byte and line bounds for streaming and returned content;
- no swallowed flush errors;
- a tool cannot mutate call identity or protected turn fields through `progress`;
- the final tool-result message is bounded and strict JSON;
- use the stored final call consistently for invocation/recovery evidence;
- preserve fixed five-second job TERM-to-KILL grace, optional `ProcessHost`, and polling capped at one second.
### 16. Known implementation defects to cover
Also fix these concrete issues found during review:
- `Harness.waitForIdle` has the same check-then-register race as input waiters.
- Reopened owned-conversation ancestry is reconstructed incorrectly by looking for a conversation whose ID equals the owner task ID.
- Listener invocation can currently throw after persistence and reject the writer.
- Scheduler drain/abort errors can become unhandled rejections.
- Ordinary kind throws currently manufacture a failure outside the declared failure type.
- `createConversation` and document access omit subtree validation.
- Hook matching by name accepts stale tokens.
- `config.get` treats stored `null` as absent.
- custom kind defaults are returned by `c.config` but are not necessarily present for task document reads.
- `SystemEntry` witness requires one model message although metadata-only deltas use none.
- JSONL replay ignores recorded sequence values.
- sticky truncate currently runs outside the line.
- use `rt.now()` rather than ambient `Date.now()` in kind-controlled durable records where determinism/testing matters.
## Required test matrix
Retain all existing behavioral and recovery tests, then add focused regressions for every numbered hardening section. At minimum, the final report must show:
1. runtime tests for memory and JSONL;
2. strict TypeScript compile tests for public capability surfaces, kind/config/hook/task-ref/slot inference, and strict durable JSON;
3. fault-injected JSONL atomicity at every publication cut;
4. escaped root and nested proxy rejection;
5. stale invocation and post-mark commit rejection;
6. malicious ordinary-task attempts against every core-only operation;
7. waiter and idle-registration race tests;
8. ordered/gap-free watch capture, listener throw, overflow, stop, and reconnect tests;
9. display-only assistant and terminal-closure write regressions;
10. terminal task lookup after many retirements and reopen;
11. real TypeBox validation tests;
12. the existing generation/tool/post-tools/collapse/job/subagent crash matrix.
Tests must not use real providers, API keys, or paid endpoints. Fake models and process hosts only.
## Deliverable
Return a new zip containing:
- the hardened standalone implementation;
- portable `package.json` and `tsconfig.json` with pinned dependencies;
- all runtime and compile tests;
- updated README and usage guide describing the fixed-core/document/phase design and explicit single-process assumption;
- a concise `CHANGES.md` mapping every item in this handoff to code/tests;
- production LOC count excluding storage backends and tests;
- exact commands and results for install, typecheck, and tests;
- unresolved questions or deliberate deviations, with no silent omissions.
Do not modify `packages/agent/src/harness/pico` or the approved V2 files. The returned zip will receive another independent review before incorporation as `packages/agent/src/harness/pico3`.

Binary file not shown.

View file

@ -0,0 +1,603 @@
# Pico3 plugins
Status: proposal for how the hardened V3 (`pico3`) harness is extended, and how
those extensions are packaged as Chord facets in the coding agent. Companion of
`view-and-events.md`. Written against `harness-v3.zip`, `pico-handoff-v2.md`
(§8–§12, §16) and `packages/agent/docs/plugins.md` (the coding-agent facet
architecture), read in full.
Read order: §1–§5 are the harness alone (no Chord); §6 is the Chord layer; §7 tests.
## 1. Principles
1. **There is no plugin object.** The harness exposes registration capabilities;
whoever composes the process (a Chord facet, a test, a script) calls them.
The coding agent's "plugin" is a package of host-specific Chord facets that
call these capabilities (§6); the harness never sees the package.
2. **One durability story.** Everything durable is one of four primitives (§2):
entry, task checkpoint, task slot, namespace slice. Tools, hooks and custom
task kinds use the same four; none has a private mechanism.
3. **Nothing durable belongs to a facet.** Chord service instances and
replicated state are live process state; a reload replaces code, never state.
4. **Core authority stays core.** Extensions cannot append entries mid-turn,
touch `turn`/`inbox`, resolve inputs, create core tasks, or write outside their
namespace or their own task's slot. Enforced at runtime on every method, by
type at the surface.
5. **An extension's presentation is only what it projects.** The view shows a
namespace's declared `view` projection (nothing by default), a kind's
`describe()` output, the core-rendered tool slot fields, and entries. Raw
namespace state (caches, keys, secrets) and slot working state (memos,
idempotency evidence) never appear in the view.
Here, “durable” means recoverable according to the selected storage mode. JSONL
with `fsync: false` covers process termination while the OS/filesystem remain
alive, but acknowledged tail commits may disappear after power, kernel, VM-host,
or storage-cache failure. `fsync: true` strengthens file-data durability; see
`hardening-handoff.md` §10 for the exact boundary and worst cases. Neither mode
makes an external effect exactly once without an external idempotency key.
## 2. Durable primitives
| primitive | lives in | lifetime | written by | forks / history | in the view |
|---|---|---|---|---|---|
| **entry** | transcript | forever, immutable | anyone via `c.write` (boundary-safe); core via `appendEntry` | inherited up to the fork point; rewind = `at` | `entries` |
| **task checkpoint** | task record | the task | its kind, whole replacement | no | via `describe()` → `turn.generation`, `compaction`, `tasks[id].status` |
| **task slot** | sticky document, `tasks[id]` (turn tasks: `turn.tools[i]`) | the task; retired atomically with its terminal record | its kind; tools running in it; hooks running in it | no | `tasks[id]`, `turn.tools[i]` |
| **namespace slice** | `plugins[ns]` in the session, rewindable or sticky document | the conversation / session | holders of the namespace token | rewindable slices are inherited and rewindable; sticky/session are present-only | `plugins[ns]` |
Rules:
- **Checkpoint** is the kind's recovery program counter, nothing else. It is not
a place for working state (that is the slot) or long-lived state (namespace).
- **Slot** is working state that should die with the task: progress, streamed
output, and **memos**. `memoOnce(slot, key, candidate)` is one operation inside
one commit: if `Object.hasOwn(slot.memos, key)` return the stored value, else
store `candidate` and return it (`??=` is wrong: `null` is a valid value). It
returns the durable winner to every caller. A memo is durable coordination and
evidence; it is **not** exactly-once execution by itself. For an external
effect, persist the idempotency key *before* the effect and replay the effect
with the same key, relying on the external side to dedupe or to answer a
result lookup; persist a result *after* an effect only when a duplicate is
otherwise harmless. Slot contents are never in the view.
- **Namespace slice** defaults are declared at registration, per document, and
seeded lazily (`plugins[ns] ??= defaults` on first access). Choose the
document by semantics: rewindable when a fork should see the value as it was
(plan mode, chosen model overrides), sticky for the present (UI preferences,
caches), session for session-wide.
- **Entries** are the only primitive the model can see (when they carry
`model`); everything else is invisible to the model by construction.
- Protected: `turn`, `inbox`, core config keys and other tasks' slots reject
writes from extensions.
What is deliberately **not** durable: Chord keyed service instances (an open
approval dialog), hook handler registrations, tool and section registrations.
All are re-established by code at startup or reload.
## 3. Registration surface
```ts
const h = await Harness.open(storage, { models, processHost?, onReport? }, ctx);
// nothing runs yet; register, then:
h.resume();
```
All registration methods return an unregister function and may be called while
the harness is open (§5 for what happens then).
```ts
h.registerTool(declaration: ToolDeclaration): () => void
h.registerSection(section: SystemSection): () => void
h.registerTaskKind(kind: Kind): () => void // ordinary kinds only; config keys checked for collisions
h.namespace<T extends NamespaceShape>(ns: string, defaults: T, opts?: { view?: (slice: T) => JsonValue }): Namespace<T>
// token with idempotent `unregister()`; unique, not `pi.*`; no `view` ⇒ nothing in the view
h.hooks(ns: Namespace, kind, handlers: Partial<HooksOf<typeof kind>>, opts?): () => void // harness-wide; bound to the namespace (§4.2)
c.hooks(ns, kind, handlers, { subtree? }): () => void // one conversation (+ owned)
defineEntry<E>(kind: string): EntryKind<E> // stateless; throws on `pi.*` (§4.4)
h.conversation(id, ctx): Promise<ConversationHandle | undefined>
h.quiescent(): boolean; h.hold(): () => void; h.suspend(ctx): Promise<void> // §5
```
Token identity: the object passed to `tx.plugins(ns)`, `tx.emit(ns, …)`,
`tx.createTask(kind, …)`, `h.hooks(ns, kind, …)` must be the currently
registered one; a redeclared or unregistered token rejects with
`StaleDefinition`. The namespace *string* is what persists (memo keys, state
slices); the token is only current runtime authority. This is what makes reload
safe: code from a retired generation cannot keep writing, and a replacement
registration of the same string finds the same state and memos.
Before `resume()`: task kinds whose tasks may exist in storage, and hook
handlers that recovery reruns (`beforeTool`, `systemInstructions`,
`beforeRequest`, `beforeCollapse`). A task whose kind is unregistered at
`resume()` is orphaned (open-time reconciliation, `pico-handoff-v2.md` §13). A
kind registered later only affects tasks created later. Tools, sections,
namespaces and observer hooks may be registered at any time.
Inside a commit, extensions get:
```ts
tx.plugins(ns): T // typed slice in the declared document(s), tracked proxy
tx.emit(ns, name: string, data: JsonValue): void // -> { type: `plugin.${ns}.${name}`, data } in this commit's envelope
tx.slot(task): Slot // own task only (kinds); tools/hooks use the api variants
memoOnce(tx.slot(task), key, candidate) // first writer wins; hasOwn semantics (§2)
tx.write(conversationId, entryKind, input): Promise<Id>
tx.createTask(kind, input, opts): TaskRef<K> // ordinary kinds
```
## 4. Extension points
Each with what it may do, where its durable state lives, and a worked example
against the raw harness. The three examples are the ones §6 turns into Chord
facets.
### 4.1 Tools
`ToolDeclaration` as in the sketch (`name`, `description`, `parameters` TypeBox,
`replay`, `output` bounds, `execute(args: Static<P>, api, ctx)`), plus the
tool-facing API:
```ts
interface ToolApi {
readonly taskId: Id; readonly conversationId: Id; readonly callId: string;
stream(chunk: string | Uint8Array): void; // kernel-bounded; becomes the result content if none returned
progress(update: (slot: ToolSlot) => void, ctx): Promise<void>; // own slot: progress, details, continuedBy
memo<T extends JsonValue>(name: string, candidate: T, ctx): Promise<T>; // memoOnce on the own slot
memo<T extends JsonValue>(name: string, ctx): Promise<T | undefined>; // read
task<K>(kind: K, input: InputOf<K>, opts, ctx): Promise<TaskRef<K>>;
conversation(spec, ctx): Promise<OwnedConversation>;
waitForTask(ref, ctx): Promise<TaskOf<K>>;
slot<T>(ref, ctx): Promise<T | undefined>; // another task's slot, read-only snapshot
}
```
Durability: the tool's **slot** (`turn.tools[i]`) for streamed output, progress
and memos; **entries** through its result. A tool has no checkpoint; it runs
inside `pi.tool`, whose `started{call}` checkpoint is the effect evidence.
How memos and `replay` interact: `beforeTool` and the `started` checkpoint
happen before `execute`; a crash after `started` re-invokes only `replay:
"safe"` tools. A safe tool that performs a keyed external effect records the
idempotency key as a memo *before* the effect and reuses it on re-invocation;
the external side dedupes or answers a result lookup for that key. Recording a
result *after* the effect (as the question tool does) is only correct when a
repeated effect is harmless, which asking a human again is. Memos are scoped to
the tool task's slot and retired with it, so they cannot leak into a later call
with the same id.
Example: a question tool that waits for a human. `ask` is whatever the host
wires in (§6.2 wires a Chord dialog service; a test wires a promise).
```ts
export function questionTool(ask: (req: QuestionRequest, api: ToolApi, ctx: Context) => Promise<QuestionResponse>): ToolDeclaration {
return {
name: "question", description: "Ask the user a question and wait for the answer.",
parameters: QuestionParamsSchema, replay: "safe",
async execute(params, api, ctx) {
let response = await api.memo<QuestionResponse>("answer", ctx); // answered before a crash?
if (response === undefined) {
const candidate = await ask(params, api, ctx); // may wait for a long time; abort unwinds through ctx
response = await api.memo("answer", candidate, ctx); // first writer wins
}
return response.outcome === "cancelled"
? { content: [{ type: "text", text: "User cancelled" }], isError: true }
: { content: [{ type: "text", text: `User answered: ${response.answer}` }], details: { ...params, answer: response.answer } };
},
};
}
```
Crash before the memo: `pi.tool` recovers `started` (safe), re-invokes, no memo,
asks again. Crash after: re-invokes, memo present, returns without asking.
Two answers racing: both call `api.memo("answer", …)`, both receive the winner.
### 4.2 Hooks
Declared per kind, typed (`GenerationHooks`, `ToolHooks`, `PostToolsHooks`,
`CollapseHooks`, plus any custom kind's `H`). Handlers receive
`(…payload, api, ctx)`. `HookApi` is identity only; each point's runner passes
a point-specific info object with exactly the capabilities that point can use:
```ts
interface HookApi { readonly kind: string; readonly taskId: Id; readonly conversationId: Id }
interface BeforeToolApi extends HookApi {
readonly callId: string;
waiting(ctx: Context): Promise<void>; // commits turn.tools[i].waitingOn = <this handler's namespace>; idempotent
memo<T extends JsonValue>(name: string, candidate: T, ctx: Context): Promise<T>; // memoOnce on the tool task's slot, keyed (task, namespace, name)
memo<T extends JsonValue>(name: string, ctx: Context): Promise<T | undefined>;
emit(name: string, data: JsonValue, ctx: Context): Promise<void>; // plugin.<namespace>.<name>
}
// Hooks are registered with their namespace token (`h.hooks(ns, kind, handlers)`), so `waiting`, `memo` and
// `emit` need no namespace argument and two plugins cannot collide on a memo name. `waitingOn` is cleared by the
// kernel atomically with `started` (handler allowed), with the synthetic result (handler blocked or threw), or in
// its own commit when the handler unwinds on abort or suspend.
interface ToolHooks {
beforeTool(call: ToolCall, api: BeforeToolApi, ctx: Context): { call?: ToolCall; block?: string } | void | Promise<…>;
afterTool(call: ToolCall, result: ToolResult, api: HookApi & { readonly callId: string }, ctx: Context): ToolResult | void | Promise<…>;
}
// generation points: HookApi & { cutoff } / { attempt } as today; collapse: HookApi. A `memo` on another
// point's hosting-task slot is added to that point's info type when a real case needs it, not by default.
```
Durability: a hook has no task of its own. State tied to the invocation it is
running in goes in the **hosting task's slot** via the point's `api.memo`,
keyed by `(hosting task, namespace string, name)` so it survives recovery and
re-registration and cannot collide with another plugin's memo (retired with
that task, no cleanup code); state that outlives tasks goes in a **namespace**.
There is no hook invocation id: nothing durable identifies "the same handler
invocation" across a crash or a reload, and the rerun must find the earlier
decision. Hooks commit through `h.conversation(api.conversationId, ctx)`
handles (never from inside a transaction builder: use the builder's own `ctx`
there, see `hardening-handoff.md` §7) and may wait as long as they like; `beforeTool` handlers rerun after a
crash or a suspend/reopen (they run before `started`), which is why their
decision must be a memo.
Example: tool approval.
```ts
export function installApproval(h: Harness, ask: (req: ApprovalRequest, ctx: Context) => Promise<ApprovalDecision>): () => void {
const ns = h.namespace<{ autoAllow: string[] }>("approval", { sticky: { autoAllow: [] } }); // long-lived: which tools skip approval; no `view` ⇒ private
return h.hooks(ns, kinds.tool, {
async beforeTool(call, api, ctx) {
const c = await h.conversation(api.conversationId, ctx);
const auto = await c.commit((tx) => tx.plugins(ns).autoAllow.includes(call.name), ctx);
if (auto) return;
let d = await api.memo<ApprovalDecision>("decision", ctx); // decided before a crash / reload? key = (task, "approval", "decision")
if (d === undefined) {
await api.waiting(ctx); // view: turn.tools[i].waitingOn = "approval"; event tool.waiting
const candidate = await ask({ conversationId: c.id, callId: call.id, taskId: api.taskId, name: call.name, args: call.arguments }, ctx);
d = await api.memo("decision", candidate, ctx); // first writer wins
}
return d.decision === "deny" ? { block: `denied${d.by ? ` by ${d.by}` : ""}` } : undefined;
},
});
}
```
No `afterTool` cleanup: the memo lives in the tool task's slot and is retired
with it. Late joiners see `waitingOn` in the snapshot. Two clients deciding at
once both get the memo winner. Abort or suspend unwinds `ask` through `ctx`;
the kernel clears `waitingOn` when the handler returns for any reason. Asking a
human twice after a crash between `waiting` and the memo is harmless, which is
why a post-effect memo is acceptable here.
### 4.3 Custom task kinds
Shape as in the sketch and the hardening handoff: `defineTask({ name, config?,
hooks?, initial, phases, abort, describe? })`, exhaustive phase map, in-flight
phases written before an effect and entered only after reopen, `next` builders
that may checkpoint, terminalize or `"retry"`.
Durability:
| need | primitive |
|---|---|
| where to resume after a crash | **checkpoint** (`tx.checkpoint(next)` / returned `next`) |
| progress, partial output, per-run memos | **slot** (`tx.slot(current)`, `memoOnce`) — shown as `tasks[id].status` after `describe()` |
| state that outlives the task, per conversation or session | **namespace** |
| facts for the transcript / model | **entries** via `tx.write` (placed at a boundary when busy) |
`describe(task): JsonValue` turns checkpoint + slot into the rendering-shaped
status the view publishes; if omitted, the view shows `{ phase }`.
Example: a background repository indexer, resumable and observable.
```ts
type IndexInput = { root: string };
type IndexCheckpoint = { phase: "scanning"; cursor: string | null } | { phase: "writing"; batchKey: string };
type IndexSlot = { files: number; lastFile?: string; memos?: Record<string, JsonValue> };
export const indexer = defineTask<IndexInput, IndexCheckpoint, { files: number }, { reason: string }, null>({
name: "repo.indexer",
config: { rewindable: { indexIgnore: [] as string[] } },
describe: (task) => ({ stage: task.checkpoint?.phase ?? "starting", files: task.slot?.files ?? 0 }),
async initial(task, rt, ctx) {
await rt.commit((tx) => { tx.slot(task).files = 0; tx.checkpoint({ phase: "scanning", cursor: null }); }, ctx);
return scan(task, null, rt, ctx);
},
phases: {
scanning: (task, rt, ctx) => scan(task, task.checkpoint.cursor, rt, ctx),
// in-flight: only entered after a crash. The batch key was persisted before the effect; the write is
// repeated with the same key and is safe only because writeBatch(key) is idempotent on the receiving side.
async writing(task, rt, ctx) {
await writeBatch(task.checkpoint.batchKey, ctx);
return { next: { phase: "scanning", cursor: task.checkpoint.batchKey } };
},
},
async abort() { return () => null; },
});
async function scan(task, cursor, rt, ctx) {
const ignore = (await rt.rewindable(task.conversationId, ctx)).indexIgnore;
const batch = await nextBatch(task.input.root, cursor, ignore, ctx);
if (batch === undefined) {
return { done: async (tx, current) => {
await tx.write(current.conversationId, entries.notice, { model: [{ role: "user", content: `Indexed ${tx.slot(current).files} files.`, timestamp: rt.now() }] });
return { status: "completed", result: { files: tx.slot(current).files } };
} };
}
await rt.commit((tx) => tx.checkpoint({ phase: "writing", batchKey: batch.key }), ctx); // idempotency key before the effect
await writeBatch(batch.key, ctx);
return { next: (tx, current) => {
const s = tx.slot(current); s.files += batch.files.length; s.lastFile = batch.files.at(-1);
return { phase: "scanning", cursor: batch.key };
} };
}
```
Registered with `h.registerTaskKind(indexer)` before `resume()`; started with
`c.commit((tx) => tx.createTask(indexer, { root }, { background: true }), ctx)`
or from a tool via `api.task(indexer, …)`. The view shows
`tasks[id] = { kind: "repo.indexer", background: true, status: { stage, files } }`;
a UI renders it from that alone.
### 4.4 Custom entries
```ts
interface CiEntry extends Entry { kind: "ci.result"; data: { run: number; ok: boolean }; model: [UserMessage] }
const ci = defineEntry<CiEntry>("ci.result");
await c.write(ci, { data: { run: 812, ok: false }, model: [{ role: "user", content: "CI run 812 failed.", timestamp: now }] }, ctx);
const last = await c.commit((tx) => tx.newestEntry(c.id, ci), ctx); // CiEntry | undefined
for (const e of view.entries) if (ci.is(e)) render(e.data.run);
```
Omit `model` for bookkeeping the model must not see. `head` and `edits` are
core-only. `defineEntry` throws on a `pi.*` name and `c.write`/`tx.write` reject
`pi.*` at runtime for non-core callers even after a cast, with one allow-listed
exception: `pi.notice` (one user message, optional `data`, no `head`/`edits`),
which hosts, jobs and tools use for notices. No entry registry or schema is
required. Placement is boundary-safe: immediately when idle, otherwise at the
next boundary (`view-and-events.md` §6.1).
### 4.5 Namespaces
```ts
const ns = h.namespace<{ planMode: { enabled: boolean }; cache: Record<string, string> }>(
"plan",
{ rewindable: { planMode: { enabled: false } }, sticky: { cache: {} } },
{ view: (s) => ({ planMode: s.planMode }) }, // only this projection reaches the view; `cache` never does
);
await c.commit((tx) => { tx.plugins(ns).planMode.enabled = true; tx.emit(ns, "toggled", { enabled: true }); }, ctx);
// view: plugins.plan.planMode; envelope: ["s", ["plugins","plan","planMode","enabled"], true] + { type: "plugin.plan.toggled", data }
env.own(ns.unregister); // idempotent; state stays, token goes stale
```
Defaults are per document; a slice may span documents (`{ rewindable: …, sticky:
… }`) and `tx.plugins(ns)` returns the merged typed object with each key routed
to its document (same mechanism as core `config`). `view` is a pure projection
the kernel re-evaluates when the slice changes; without it the namespace has no
presence in the view. Re-registering the same string with changed defaults adds
new keys lazily and leaves removed keys as inert stored data.
### 4.6 Owned conversations and jobs
Subagents (`api.conversation`) and processes (`pi.job` via `api.task(kinds.job, …)`)
are core features with their own durability; extensions use them, they do not
extend them. See `pico-handoff-v2.md` §8, §10.5 and the hardening handoff §15.
## 5. Unregister, re-register, reload
Every registration can be undone and redone while the harness is open.
| contribution | on unregister | on re-register |
|---|---|---|
| namespace (`ns.unregister()`, idempotent) | token stale; `tx.plugins`/`tx.emit`/`h.hooks(ns, …)` with it reject | new token for the same string; stored state and memos untouched; new defaults seeded lazily |
| hooks | removed from future chains; a handler already running continues to completion (it is cancelled only by abort or `suspend`) | joins the chain |
| task kind | pending tasks of the kind are not dispatched (not orphaned; orphaning is open-time only); running invocations keep the kind object they captured | pending tasks dispatch with the new implementation; config keys re-checked |
| tool / section | revision bump; in-flight invocations keep their captured declaration; preparation snapshots retry | same |
| entry kind | nothing | nothing |
Reload replaces code, never durable state. Two modes, chosen by the host:
- **Quiescent reload.** `h.quiescent()` is true when the scheduler holds **no
invocation**, running or sleeping: no phase handler, no `retrying`/`deferred`
sleep, no job poll, no waiting `beforeTool`. Live task *records* may exist
(pending, or between invocations). Then: `release = h.hold()` (defers new
dispatch and new hook chains; commits and reads continue) → deactivate old
facets (their disposals unregister) → activate new ones (re-register) →
`release()`.
- **Suspend and reopen** when not quiescent (the first safe version for active
work). `h.suspend(ctx)`: stop dispatch, cancel every in-process invocation
context, await unwind, close storage; task records stay live and nothing is
terminalized. Reopen with the new registrations and let ordinary recovery run:
pre-effect phases rerun, `started` tools replay if safe and still safe, unsafe
ones become interrupted results, other phases recover per their kind. Memos
make reruns cheap (an approval already decided is not asked again). The
injected `ProcessHost` must outlive the harness instance so running jobs
reconcile through `status(key)` instead of becoming `unknown`.
Cancelling live calls of a tool or kind that the new generation no longer
registers is not a transparent migration: it durably aborts them, cannot undo
external effects, and post-tools observes aborted results. It is an explicit
host policy (`suspend({ abortRemoved: true })`), never the default. Generation
leases (old registrations kept alive until captured invocations drain while new
work uses replacements) are a possible later refinement; they are not required
for v1.
Registration belongs in `onActivate`, not `setup`: during a shape-preserving
reload the replacement's `setup` runs before the old facet is deactivated, and
two live registrations of one namespace string would collide.
## 6. The Chord layer
The coding agent composes the harness and its extensions as Chord facets
(`packages/agent/docs/plugins.md`). Nothing in §1–§5 changes; this section only
says which facet calls what.
### 6.1 Built-in facets (plumbing)
Provided once by the coding agent, consumed by every feature:
```ts
// session worker
export const Harness = defineService<HarnessHandle>("pi.harness", { local: true }); // the in-process handle from §3; never remote
export const ConversationService = defineService<{ // keyed by conversation id; remote
view: ReplicatedState<ConversationView>; // view-and-events.md; one publish per commit, on the line
send(input: SendInput, ctx): Promise<Id>; write(entry: NewEntry, ctx): Promise<Id>;
inputAbort(id: Id, ctx): Promise<"aborted" | "already_placed" | "not_found">;
configSet(patch: Partial<Config>, ctx): Promise<void>;
abort(ctx): Promise<void>; reset(handoff: string | undefined, ctx): Promise<void>; collapse(instructions: string | undefined, ctx): Promise<Id>;
fork(at: Id | "start", spec: ForkSpec, ctx): Promise<Id>; entries(scan: EntryScan, ctx): Promise<Entry[]>;
}>("pi.conversation");
export const harnessSessionFacet = defineFacet({
id: "@pi/harness",
setup(env) {
const conversations = env.provideMany(ConversationService);
const handle = createHarnessHandle(); // wraps Harness.open/resume/hold and the registration surface
env.provide(Harness, handle);
env.onActivate(async () => {
await handle.open(storage, { models, processHost }, ctx);
handle.onConversation((c) => { // existing at open, and every fork/subagent created later
const view = env.replicatedState(c.snapshot()); // built on the line
c.attachView(view); // bounded ordered adapter: raw envelopes → view.state + publish(), off the line (below)
env.own(conversations.spawn(String(c.id), { view, send: (i, cx) => c.send(i, cx).then((r) => r.id), /* … */ }));
});
});
// resume() is called by the session worker after the facet host reports activation complete (all registrations done)
},
});
```
`ConversationView` gains `commit: { events }` so events ride inside the
replicated member; a subscriber reads `value.commit.events` after each delivery.
No revision or storage sequence is placed in the view: Chord's
`ReplicatedStateDelivery.sequence` is contiguous per member and plays that role.
Publication must not happen on the Session line. Chord's `publish()` invokes
source and subscriber listeners synchronously and a subscriber may throw or be
slow, and `subscribe()` itself publishes pending mutations. The bridge is a
bounded ordered adapter: the kernel enqueues raw envelopes from its own view
tracker; the adapter, off the line, applies one envelope's ops to
`view.state`, sets `commit.events`, and calls `publish(ctx)` in the **same
synchronous block** (no await between apply and publish). One raw envelope =
one publish = one Chord sequence. If the queue overflows or `publish` throws,
close that keyed instance and respawn it with a fresh snapshot; a persisted Pico
commit is never turned into a failure. Chord's subscription snapshot then gives
late joiners the member at its current sequence, so §9 of
`view-and-events.md` is implemented by Chord, not by us.
### 6.2 A feature: contract + session facet + presentation facet
The rule from `plugins.md`: one shared contract module, one facet per host,
connected by service IDs. The session facet calls §3; the presentation facet
observes services and the view. Facets keep **no durable state**: everything
durable is in the harness through §2, and keyed service instances are channels.
**Approval** (hook + dialog service):
```ts
// contract.ts
export interface ApprovalRequest { conversationId: Id; callId: string; taskId: Id; name: string; args: JsonValue }
export type ApprovalDecision = { decision: "allow" | "deny"; by?: string };
export interface ApprovalDialog { readonly request: ReplicatedState<ApprovalRequest>; decide(d: ApprovalDecision, ctx: Context): Promise<void> }
export const ApprovalDialogs = defineService<ApprovalDialog>("pi.approval-dialog"); // keyed `${conversationId}:${callId}`
// session.ts
export const approvalSessionFacet = defineFacet({
id: "@pi/approval",
setup(env) {
const harness = env.use(Harness);
const dialogs = env.provideMany(ApprovalDialogs);
env.onActivate(() => {
env.own(installApproval(harness, (req, ctx) => { // §4.2, unchanged
const completion = Promise.withResolvers<ApprovalDecision>();
const close = dialogs.spawn(`${req.conversationId}:${req.callId}`, {
request: env.replicatedState(req),
async decide(d, _ctx) { completion.resolve(d); },
});
return awaitAbortable(completion.promise, ctx.abortSignal).finally(close);
}));
});
},
});
// tui.ts
export const approvalTuiFacet = defineFacet({
id: "@pi/approval",
setup(env) {
const tui = env.use(Tui);
env.observe(ApprovalDialogs, async (dialog, ctx) => { // one task per open instance, incl. hydrated ones
const r = dialog.request.value!;
const modal = await tui.acquireModal(ctx.abortSignal);
try { await dialog.decide({ decision: (await modal.confirm(`Allow ${r.name}?`, pretty(r.args))) ? "allow" : "deny", by: "tui" }, ctx); }
finally { modal.close(); }
});
},
});
```
The decision memo (`api.memo("approval", …)`) is written by the hook in the
session worker, so `decide` only resolves the local promise; the first-writer
rule is applied where the durability is. A second client's `decide` resolves an
already-resolved promise: harmless.
**Question tool** (tool + dialog service): identical shape with `questionTool`
from §4.1 as the session contribution (`harness.registerTool(questionTool(ask))`)
and `ToolApi.memo` instead of `BeforeToolApi.memo`. This is the `packages/agent/docs/plugins.md`
question example with `memoOnce` replaced by the slot memo.
**Indexer** (custom kind, no service of its own):
```ts
// session.ts
export const indexerSessionFacet = defineFacet({
id: "@pi/indexer",
setup(env) {
const harness = env.use(Harness);
env.onActivate(() => { env.own(harness.registerTaskKind(indexer)); }); // before resume(): tasks may exist in storage
},
});
// tui.ts: nothing to provide; render view.tasks[id].status for kind "repo.indexer" from ConversationService.view,
// and offer a /index slash command that calls conversation.commit-equivalent RPC (or a tiny IndexerService.start method).
```
### 6.3 What the session worker does at reload
```text
if (harness.quiescent()) {
release = harness.hold()
FacetHost.reload(candidates) // old facets' env.own() disposals unregister; new onActivate re-registers
release()
} else {
await harness.suspend(ctx) // cancel invocations, close storage; task records stay live
FacetHost.reload(candidates)
await harness.reopen(ctx) // recovery runs against the new registrations
}
```
Durable state, task records and open conversations are untouched either way
(§5).
## 7. Tests the hardening must include
- Memo: two concurrent `api.memo` writers receive the same winner; `null` is a
stored value (hasOwn, not `??=`); a crash after the memo skips `ask`, before it
re-asks; the memo is gone after the tool task terminalizes; two namespaces
using the same memo name on one task do not collide.
- Hooks: `await api.waiting(ctx)` sets `waitingOn` in its own commit and it is
cleared atomically with `started`, with the synthetic result on block/throw,
and in its own commit on abort/suspend; the decision memo survives a crash
between `waiting` and `started`; memos never appear in the view.
- Custom kind: crash in the in-flight phase re-enters it and repeats the keyed
effect (test with a fake idempotent sink and a crash between effect and the
next checkpoint); `describe()` output appears in `tasks[id].status`;
unregistered kind leaves pending tasks pending and orphans only at open.
- Namespaces: writes outside `plugins[ns]` reject; stale token rejects;
`unregister` is idempotent; re-register keeps values and memos and seeds new
defaults; only the `view` projection appears in the view; rewindable slice is
inherited by a fork and readable at an entry.
- Entries: `defineEntry("pi.x")` throws; `write` of any `pi.*` except `pi.notice`
rejects at runtime after a cast; `pi.notice` with `head`/`edits` rejects.
- Reload: `quiescent()` is false while any invocation exists (including a
waiting `beforeTool` and a polling job); `hold()` defers dispatch and hook
chains but not commits; suspend/reopen with the approval facet lets a pending
call be approved through the new generation without asking twice when a
memo exists; `ProcessHost` survives suspend/reopen and a running job
reconciles.
- Chord: loopback facet tests for approval and question (session + presentation
fixtures), late-joiner hydration of an open dialog instance, and
`ConversationService.view` sequence equals commit sequence.

View file

@ -0,0 +1,525 @@
# Pico3 view and events
Status: proposal for the hardened V3 (`pico3`) watch protocol. Replaces the
`ConversationView { entries, tasks, rewindable, sticky }` shape and the per-channel
delta of the V3 sketch. Written against `harness-v3.zip` and the Codex app-server
protocol (`codex-rs/app-server-protocol/src/protocol/v2/*`), read in full.
Companion: `plugins.md` (harness extension surface, durability, Chord facets).
## 1. Principles
1. **One snapshot, then one envelope per commit.** A watch captures the view and
subscribes in one line operation; every later commit that touches the
conversation produces exactly one envelope, delivered in commit order with a
contiguous per-watch `revision`. No cursor, replay or acknowledgement: a gap
means "open a fresh watch".
2. **Ops are the truth, events are annotations.** The envelope carries Chord ops
that transform the view document, and named events the kernel attaches because
it produced those ops and knows why. Ops fully determine the replica: a client
that folds only ops is never wrong. Events annotate cause and outcome and may
carry transient facts the view deliberately does not retain; a client that
consumes only events never scans the view.
3. **Path = event.** Every field of the view has a stable path, so an op on that
path is a typed notification. Nothing in the view is ever discovered by
diffing.
4. **The view is rendering-shaped, not kernel-shaped.** No task records, no
checkpoints, no documents. In-progress work is described by the kind that owns
it, in the shape a renderer wants.
5. **Atomic by construction.** Everything a commit changes lands in one envelope:
the assistant entry, the cleared streaming slot, the resolved inputs and the
`turn.ended` event are never observable separately. This is the one property
Codex's protocol lacks (`item/completed` and `turn/completed` are separate
messages).
## 2. The view document
One JSON document per conversation. The kernel keeps a Chord tracker for it per
watched conversation, mutates it through the proxy after each commit, and
`flush()`es the ops into the envelope. Clients fold with `applyImmutable`.
```ts
interface ConversationView {
conversation: { id: Id; parent?: { conversationId: Id; at: Id }; owner?: Id };
/** Active transcript: newest head H, then every fork-visible entry with id >= H.head,
* chronological. All fork-visible entries when there is no head. See §2.1. */
entries: Entry[];
/** Resolved configuration, flat: every built-in and registered kind's declared keys,
* defaults applied. `model` is the only optional built-in. */
config: {
model?: ModelRef; thinkingLevel: ThinkingLevel; selectedTools: string[]; profile: string;
threshold: number; keepRecent: number;
retry: RetryPolicy; steeringMode: QueueMode; followUpMode: QueueMode;
[pluginKindKey: string]: JsonValue;
};
/** Queued steer / followUp / passive writes, admission order. */
inbox: QueuedInput[];
/** Present while an input group is live (from placement to its end); absent when idle. */
turn?: TurnView;
/** Present while a pi.collapse task is live. */
compaction?: { taskId: Id; reason: "threshold" | "manual" | "overflow"; stage: "summarizing" | "retrying"; attempt: number; retryAt?: number };
/** Non-turn live tasks (jobs, plugin kinds), described by their kind. Removed on terminal. */
tasks: { [taskId: string]: { kind: string; background?: true; marked?: true; status: JsonValue } };
/** Plugin presentation slices, by namespace: the namespace's declared `view` projection, never raw state. */
plugins: { [namespace: string]: JsonValue };
}
interface TurnView {
inputs: Id[]; // the input group
generation?: GenerationStatus; // present while a pi.generation is live
message?: AssistantMessage; // streaming assistant message (frames applied in place)
tools: ToolSlot[]; // tool calls of the newest assistant message in this turn, by call index
}
type GenerationStatus =
| { stage: "waiting"; on: "compaction" } // replacement generation pending behind a collapse
| { stage: "preparing" } // before `prepared`
| { stage: "requesting"; attempt: number } // request sent, no frame yet
| { stage: "streaming"; attempt: number } // first frame applied
| { stage: "retrying"; attempt: number; retryAt: number; lastError: string }
| { stage: "deferred"; attempt: number; pollAt: number };
interface ToolSlot {
callId: string; name: string; args: JsonValue;
status: "pending" | "running" | "done" | "error" | "aborted";
waitingOn?: string; // namespace of the beforeTool handler that awaited BeforeToolApi.waiting(ctx); cleared when it returns
output?: string; // bounded stream so far (api.stream)
progress?: string; details?: JsonValue;
continuedBy?: Id; // background task that took over; see tasks[continuedBy]
entry?: Id; // the pi.tool_result entry once landed
}
// Memos and other slot working state are never in the view (plugins.md §2); only the fields above are rendered.
```
### 2.1 Active transcript
- `H` = newest fork-visible entry with a `head`; `from` = `H.head`.
- `entries` = every fork-visible entry with `id >= from`, chronological. Inherited
parent entries are included when `from` points into a parent.
- Older head entries inside the range stay (a renderer may mark them). Display-only
entries (`pi.assistant` with `stopReason` `error`/`aborted`, `pi.usage`, model-less
plugin entries) stay. `edits` are carried verbatim; the view is the transcript,
not the model projection.
- A new head entry truncates: the kernel emits one splice removing everything
before `head` and one appending the entry (§4, `entries`).
- No `limit` anywhere on the public surface; the capture pages internally.
### 2.2 What is deliberately not in the view
- `Task` records, checkpoints, outcomes: available through `getTask`/`scanTasks`
for tooling and tests, never pushed.
- The two storage documents (`rewindable`, `sticky`): their split is a fork/history
property; the view flattens their keys into `config`, `inbox`, `turn`, `tasks`,
`plugins`. Which key lives in which document is declared per kind and matters
only to `stateAt(entry)` / `fork(at)`.
- Section seed metadata on conversations.
## 3. Envelope
```ts
interface Envelope {
revision: number; // contiguous per watch: snapshot.revision + 1, +2, …; storage Seq is not exposed
ops: Op[]; // Chord ops over ConversationView (§4)
events: ViewEvent[]; // named annotations for this commit (§5); may be empty
}
```
`revision` is per watch, not the storage commit sequence: a conversation watch
sees only commits that touch its conversation, so the global `Seq` would have
legitimate gaps. Storage `Seq` stays internal unless a public commit-correlation
use appears.
Delivery: folded into the kernel's view tracker on the line; the listener is
invoked **synchronously, in order, off the line**, one envelope per commit that
changed anything visible to this conversation. Envelopes arriving before
`start()` are buffered (bounded; overflow closes the watch). Asynchronous
consumers (the Chord bridge, application queues) own their own queue and
backpressure policy (`plugins.md` §6.1). A listener throw closes that watch and
reports through `onError`. `stop` is idempotent; nothing is delivered after
`stop`.
Base flush: after the kernel rebases the view tracker (reopen of a cold watch is a
fresh snapshot instead), an envelope may carry `["r", view]` as its only op.
Clients must handle it as "replace everything". Events in that envelope are
still valid.
## 4. Ops by path
Chord op vocabulary: `["r", v]` replace root, `["s", path, v]` set, `["d", path]`
delete, `["a", path, str]` string append, `["t", path, n]` string truncate,
`["p", path, index, deleteCount, items]` array splice.
| path | ops you will see | meaning |
|---|---|---|
| `["entries"]` | `["p", ["entries"], n, 0, [e…]]` | entries appended (always at the end) |
| `["entries"]` | `["p", ["entries"], 0, k, []]` then `["p", …, n, 0, [head]]` | new head: transcript truncated, head entry appended |
| `["config", key]` | `s` / `d` | config changed; `config.reset(keys)` sets the declared default (only `model`, which has none, is deleted) |
| `["inbox"]` | `p` | queued, placed (removed), aborted (removed) |
| `["turn"]` | `["s", ["turn"], {…}]` / `["d", ["turn"]]` | input group started / ended |
| `["turn","generation"]` | `s` / `d` | generation stage changes / generation ended |
| `["turn","message"]` | `s` (first frame), `a`/`s` on content paths, `d` | streaming assistant message |
| `["turn","message","content",i,"text"]` | `a` | text delta |
| `["turn","message","content",i,"arguments"]` | `s` | tool-call arguments materialized at checkpoint/end |
| `["turn","tools"]` | `s` (whole array, with the assistant entry) | tool slots created for a new assistant message |
| `["turn","tools",i,"status"]` | `s` | pending → running → done/error/aborted |
| `["turn","tools",i,"waitingOn"]` | `s` / `d` | a `beforeTool` handler (namespace = value) is waiting on a human/service; cleared when it returns |
| `["turn","tools",i,"output"]` | `s` | bounded stream flush (whole string; retention may drop the head) |
| `["turn","tools",i,"progress"]`, `[…,"details"]`, `[…,"continuedBy"]` | `s` | `api.progress` |
| `["compaction"]` | `s` / `d` | collapse started / stage changed / ended |
| `["tasks", id]` | `s` / `d` | non-turn task appeared / status changed / terminal |
| `["tasks", id, "status", …]` | any | kind-described status detail (jobs: stdout, exitCode, …) |
| `["plugins", ns, …]` | any | the namespace's `view` projection |
Rule for kernel authors: mutate the tracked view through its proxy so the
tracker records minimal ops; never rebuild subtrees. Streaming applies each
frame to `turn.message` in place (same reducer as pi-ai's frame reducer).
## 5. Events
Events are emitted by core code inside the commit that causes them
(`tx.emit(event)`, core authority). Extensions emit `{ type: "plugin.<ns>.<name>", … }`
through their namespace token from their own commits. Rule: ops fully determine
the replica; events annotate the cause and outcome of the commit and may carry
transient facts the view does not retain (`turn.ended.reason`,
`task.ended.outcome`, `compaction.failed`, `warning`). An event must never be the
only place a late joiner could learn something needed to render steady state.
```ts
type ViewEvent =
// transcript
| { type: "entry.added"; entry: Entry } // one per appended entry, in order
| { type: "head.moved"; entry: Entry } // summary / reset / handoff landed
// input group
| { type: "turn.started"; inputs: Id[] }
| { type: "turn.ended"; inputs: Id[]; status: "done"; answer: Id }
| { type: "turn.ended"; inputs: Id[]; status: "unanswered"; reason: "terminated" | "aborted" | "failed" | "stale"; detail?: string }
| { type: "input.queued"; input: Id; mode: "steer" | "followUp" | "write" }
| { type: "input.placed"; input: Id; entry: Id }
| { type: "input.aborted"; input: Id }
// generation
| { type: "generation.started"; taskId: Id; attempt: number } // request sent
| { type: "generation.retrying"; taskId: Id; attempt: number; retryAt: number; error: string }
| { type: "generation.deferred"; taskId: Id; pollAt: number }
| { type: "generation.completed"; taskId: Id; entry: Id; toolCalls: number }
| { type: "generation.failed"; taskId: Id; reason: "provider" | "overflow" | "retries_exhausted" | "no_model"; detail: string; entry?: Id }
// tools
| { type: "tool.waiting"; taskId: Id; callId: string; on: string } // a beforeTool handler is waiting; `on` is its namespace
| { type: "tool.started"; taskId: Id; callId: string; name: string }
| { type: "tool.finished"; taskId: Id; callId: string; entry: Id; isError: boolean; control?: ToolControl }
| { type: "tool.aborted"; taskId: Id; callId: string; entry: Id }
// compaction
| { type: "compaction.started"; taskId: Id; reason: "threshold" | "manual" | "overflow"; through: Id }
| { type: "compaction.retrying"; taskId: Id; attempt: number; retryAt: number; error: string }
| { type: "compaction.finished"; taskId: Id; summary: Id }
| { type: "compaction.failed"; taskId: Id; reason: "stale" | "declined" | "provider" | "retries_exhausted" | "no_model"; detail: string }
// other tasks
| { type: "task.started"; taskId: Id; kind: string; background?: true }
| { type: "task.ended"; taskId: Id; kind: string; outcome: "completed" | "failed" | "aborted" | "orphaned" }
// settings and diagnostics
| { type: "config.changed"; keys: string[] }
| { type: "warning"; source: string; message: string } // hook skipped, tool output truncated, selected tool missing, …
| { type: `plugin.${string}`; data: JsonValue };
```
`warning` has no state counterpart at all; it exists for exactly the
occurrences Codex models as `warning` / `error{willRetry}` notifications.
## 6. Commit-by-commit mapping
What each kernel moment writes, the ops it produces, and the events attached.
Ops are abbreviated; every row is one atomic envelope.
### 6.1 Admission
| moment | ops | events |
|---|---|---|
| `send`, idle | `entries +pi.user`; `turn = { inputs, tools: [] }`; (older queued items placed first: `inbox` splice, `entries +…`) | `input.placed`, `entry.added`, `turn.started` |
| `send`, busy (`steer`/`followUp`) | `inbox +item` | `input.queued` |
| `write`, idle | `entries +entry` (+ head truncation if it carries a head) | `entry.added` (+ `head.moved`) |
| `write`, busy | `inbox +item` | `input.queued{mode:"write"}` |
| `InputHandle.abort` while queued | `inbox -item` | `input.aborted` |
| `config.set` | `config.<key>` set/delete | `config.changed{keys}` |
### 6.2 Generation
| moment | ops | events |
|---|---|---|
| task created (successor/replacement) | `turn.generation = { stage: "preparing" }` (turn already present) | `task.started` is **not** emitted for turn kinds |
| `prepared` commit | optional `entries +pi.system`; `turn.tools = []`; `turn.message` deleted; `turn.generation = { stage: "requesting", attempt }` | `entry.added` (system) |
| `requesting` checkpoint | `turn.generation.attempt` | `generation.started` |
| frames (coalesced ≥256 B / 100 ms) | first: `turn.message = partial`, `turn.generation.stage = "streaming"`; then `a`/`s` on content paths | none |
| retry decision | `entries +pi.usage`; `turn.message` deleted; `turn.generation = { stage: "retrying", … }` | `entry.added`, `generation.retrying` |
| deferred | `turn.generation = { stage: "deferred", pollAt }` | `generation.deferred` |
| response with tool calls (closure) | `entries +pi.assistant`; `turn.message` deleted; `turn.generation` deleted; `turn.tools = slots(status:"pending")`; (threshold: `compaction = {…}`) | `entry.added`, `generation.completed{toolCalls:n}`, (`compaction.started`). Slots are `pending`; `tool.started` fires at each tool's `started` checkpoint (§6.3) |
| final answer, no triggers, no continuation (closure) | `entries +pi.assistant`; `turn` deleted; boundary placements (`inbox` splices, `entries +writes`) | `entry.added`, `generation.completed`, `turn.ended{done, answer}`, `input.placed`/`entry.added` for placed writes |
| final answer, `onYield` continuation | `entries +pi.assistant`, `entries +pi.user{continuation}`; `turn.generation = { stage: "preparing" }` | `entry.added`×2, `generation.completed`; **no** `turn.ended` (same group continues) |
| final answer with queued followUp/steer triggers | as "no triggers", then `entries +pi.user`, `turn = { inputs: triggers, tools: [] }`, `turn.generation = { stage:"preparing" }` | `turn.ended{done}`, `input.placed`, `entry.added`, `turn.started` |
| terminal error / aborted-by-provider (closure) | `entries +pi.assistant` (display-only; `model` absent, message in `data`); group resolved `unanswered/failed`; **final boundary runs**: placed writes, selected triggers; `turn` deleted, or replaced by a successor turn for triggers | `entry.added`, `generation.failed{provider|retries_exhausted, entry}`, `turn.ended{unanswered, failed}`, then `input.placed`/`entry.added`/`turn.started` as in "with triggers" |
| `no_model`, overflow with nothing collapsible | as above; overflow additionally `entries +pi.notice` (placed immediately: the closing task counts as gone). `no_model` is not special-cased: a successor for queued triggers fails fast and resolves its own group | `generation.failed`, `turn.ended{unanswered, failed}`, `entry.added`, … |
| overflow with collapse (closure) | `compaction = { reason: "overflow", … }`; `turn.generation = { stage: "waiting", on: "compaction" }` (replacement created `after` collapse) | `generation.failed{overflow}`, `compaction.started`; turn continues |
| abort (fresh abort closure) | optional `entries +pi.assistant` (display-only partial, only if content was streamed); `turn` deleted | (`entry.added`), `turn.ended{unanswered, aborted}` |
### 6.3 Tool
| moment | ops | events |
|---|---|---|
| `beforeTool` handler awaits `api.waiting(ctx)` (`BeforeToolApi`, `plugins.md` §4.2) | `turn.tools[i].waitingOn = ns` (its own commit) | `tool.waiting{callId, on: ns}` |
| `started` checkpoint (after `beforeTool` allowed, before invocation) | `turn.tools[i].waitingOn` deleted; `turn.tools[i].status = "running"` (one commit) | `tool.started` |
| handler blocks/throws | `waitingOn` deleted atomically with the synthetic error result below | |
| handler unwinds on abort/suspend | `waitingOn` deleted in its own commit | none |
| stream flush (≤ every 100 ms) | `turn.tools[i].output = text` | none |
| `api.progress` | `turn.tools[i].{progress,details,continuedBy}` | none |
| result (closure) | `entries +pi.tool_result`; `turn.tools[i].status = done|error`; `turn.tools[i].entry` | `entry.added`, `tool.finished{isError, control}`, (`warning` if truncated) |
| synthetic error before invocation (not offered / missing / invalid / blocked) | same as result with `status: "error"` | `entry.added`, `tool.finished{isError:true}` |
| abort closure | `entries +pi.tool_result(aborted)`; `turn.tools[i].status = "aborted"` | `entry.added`, `tool.aborted` |
| recovery: `started` + unsafe/current-unsafe/schema-invalid | as synthetic error (`interrupted`) | `tool.finished{isError:true}` |
### 6.4 Post-tools
| moment | ops | events |
|---|---|---|
| continuation (default) | boundary placements (`inbox` splices, `entries +pi.user` for steer, `+writes`); `turn.inputs += triggers`; `turn.generation = { stage: "preparing" }` | `input.placed`…, `entry.added`…; no turn event |
| synthesized missing results (orphaned/aborted tool) | `entries +pi.tool_result` per missing call | `entry.added`, `warning{source:"post_tools"}` |
| `addTools` | `config.selectedTools` | `config.changed{["selectedTools"]}` |
| `terminate` | `turn` deleted; boundary(final) placements; successor turn if triggers | `turn.ended{done, answer: assistant}`, then as §6.2 "with triggers" |
| `handoff` | `entries` truncated + `+pi.handoff` (head); `turn` deleted; placements; successor turn | `head.moved`, `entry.added`, `turn.ended{done}`, … |
| queued self-head write ended the group | `entries` truncated + `+reset/handoff`; `turn` deleted; earlier queued steer/followUp removed from `inbox` | `head.moved`, `turn.ended{unanswered, terminated}`, `input.aborted`… (stale) |
| abort closure | `turn` deleted | `turn.ended{unanswered, aborted}` |
### 6.5 Collapse
| moment | ops | events |
|---|---|---|
| task created | `compaction = { taskId, reason, stage: "summarizing", attempt: 1 }` | `compaction.started{through}` |
| retry | `compaction.stage = "retrying"`, `retryAt` | `compaction.retrying` |
| summary lands (closure) | `entries` truncated to `head` + `+pi.summary`; `compaction` deleted | `head.moved`, `entry.added`, `compaction.finished{summary}` |
| stale / declined / failed | `compaction` deleted | `compaction.failed{reason}` |
| hook-supplied summary | same as "summary lands" without a provider stage | same |
Manual collapse (`c.collapse`) is `background`; it appears in `compaction`
exactly the same way. At most one live collapse and exactly one live generation
per conversation are core invariants (rejected before persistence, never
generalized to maps in the view): a manual collapse while one is live rejects
`CollapseInProgress`; a threshold trigger while one is live is skipped;
`NothingToCollapse` rejects the call and produces no envelope. An overflow
replacement generation waits behind its collapse as `turn.generation = { stage:
"waiting", on: "compaction" }`.
### 6.6 Jobs and plugin kinds
| moment | ops | events |
|---|---|---|
| task created | `tasks[id] = { kind, background, status: describe(task) }` | `task.started` |
| checkpoint / slot update | `tasks[id].status…` (job: `stage`, `stdout`, `stderr`, `exitCode`, `occurrence`) | none |
| job notice (`notify: true`) | `entries +pi.notice` (immediately if idle, else via `inbox`) | `entry.added` or `input.queued{write}` |
| terminal | `tasks[id]` deleted | `task.ended{outcome}` |
| marked | `tasks[id].marked = true` | none |
`describe(task)` is declared on the kind next to its checkpoint union; the
kernel calls it whenever the task record changes and writes the result to
`tasks[id].status`. Turn kinds do not use it: their status lives in `turn`.
### 6.7 Conversation-level
| moment | ops | events |
|---|---|---|
| `abort(conversation)` | `inbox` steer/followUp removed; then per-task envelopes as above | `input.aborted`…, then `turn.ended{aborted}` etc. |
| fork created | (new conversation; own watch) | none on the parent |
| owned conversation created by a tool | `turn.tools[i].details` if the tool records it; nothing else on the parent | none |
## 7. Codex mapping
For anyone coming from `codex app-server`.
| Codex | pico3 |
|---|---|
| `Thread` | conversation; `thread/status/changed` ≈ `turn` present/absent (+ `tasks` non-empty for background activity) |
| `Turn { id, items, status, error }` | `turn` (live) + `turn.ended{status}`; items are the entries appended during the group |
| `turn/started`, `turn/completed` | `turn.started`, `turn.ended` (in the same envelope as the last entry, never separate) |
| `ThreadItem::UserMessage` | `pi.user` entry |
| `ThreadItem::AgentMessage` + `item/agentMessage/delta` | `turn.message` (streaming) → `pi.assistant` entry; ops on `["turn","message",…]` |
| `ThreadItem::Reasoning` + deltas | thinking blocks inside `turn.message.content` / the assistant entry |
| `ThreadItem::CommandExecution { status, aggregatedOutput, exitCode }` + `item/commandExecution/outputDelta` | `turn.tools[i]` (`status`, `output`) → `pi.tool_result` entry |
| `ThreadItem::McpToolCall`, `DynamicToolCall`, `FileChange` | same: a tool slot; the tool declaration decides `details` |
| `ThreadItem::SubAgentActivity`, `CollabAgentToolCall` | tool slot with `details`/`continuedBy`; the child is its own conversation/watch |
| `ThreadItem::ContextCompaction` | `compaction` (live) + `pi.summary` entry + `compaction.finished` |
| `error { willRetry: true }` | `generation.retrying` **and** `turn.generation.stage = "retrying"` (visible to late joiners) |
| `error { willRetry: false }` | `generation.failed`, `turn.ended{failed}` |
| `thread/tokenUsage/updated` | `usage` on the assistant entry; `pi.usage` entries for failed attempts |
| `warning`, `deprecationNotice` | `warning` event |
| `thread/queue/changed` | ops on `["inbox"]`, `input.*` events |
| `thread/settings/updated` | ops on `["config",…]`, `config.changed` |
| `thread/resume` (history + `active_turn_snapshot`) | `watch` snapshot: `entries` + `turn` captured atomically with the subscription |
| `thread/items/list`, `thread/turns/list` pagination | `h.entries(scan)` for history beyond the active transcript |
| `process/outputDelta`, `process/exited` | `tasks[id].status` for `pi.job` |
| `item/started` → `item/completed` (same id) | tool: `callId` is stable across the slot and the result entry; assistant: atomic swap, no id needed |
What pico3 has that Codex does not: atomic multi-field envelopes; retry and
compaction state visible in a snapshot; one replica protocol for state-driven and
event-driven clients.
What Codex has that pico3 does not: turn-level unified diff (`turn/diff/updated`),
plans, approvals (`waitingOnApproval`). Approvals are a `beforeTool` hook waiting
on user input; the kernel shows that as `turn.tools[i].waitingOn` and the
`tool.waiting` event (see `plugins.md` §4.2 for the approval plugin).
## 8. Consumption patterns
Event-driven (imperative UI, third-party client):
```ts
w.start(({ ops, events }) => {
view = applyImmutable(view, ops); // keep the replica current for lookups
for (const e of events) switch (e.type) {
case "entry.added": transcript.append(e.entry); break;
case "turn.started": status.set("working"); break;
case "turn.ended": status.set(e.status === "done" ? "idle" : `stopped: ${e.reason}`); break;
case "generation.retrying": status.set(`retrying at ${e.retryAt}`); break;
case "compaction.started": banner.show("compacting…"); break;
case "compaction.finished": banner.hide(); break;
case "tool.started": transcript.showRunning(e.callId); break;
case "tool.finished": transcript.settle(e.callId, e.entry); break;
}
});
```
State-driven (React/Solid): `useSelector(v => v.turn?.message)`,
`useSelector(v => v.compaction)`. `applyImmutable` preserves identity of untouched
branches, so selectors are O(1) and no traversal happens.
Path router (when neither fits): dispatch each op on its first one or two path
segments; the table in §4 is the complete list of paths.
Presentation services (Chord RPC): keep a derived tracked document in the shape
the client wants (e.g. Codex-style `turns[]`), update it from envelopes, ship its
own ops. Project image blobs out of `entries[].model` there, not in the kernel.
## 9. Watchers, late joiners, multiple clients
### 9.1 One view per conversation, many watchers
- The kernel keeps **one** tracked view document per conversation that has at
least one watcher (not one per watch). It is built on the line the first time
a watch is opened and dropped when the last watch stops.
- After every commit that touches the conversation, the kernel mutates that one
document through its proxy, `flush()`es once, attaches the commit's events, and
hands the same `Envelope` to every watcher. Fan-out cost is per conversation;
the kernel never knows how many clients exist or what they render.
- `watch(ctx)` returns `{ view, revision, start, stop }` where `view` is a
`structuredClone` of the tracker target taken **on the line**, `revision` is
this watch's starting revision, and the subscription is installed in the same
line operation. The first envelope a watcher receives is therefore exactly
`revision + 1`; there is no window in which a commit can land between snapshot
and subscription.
### 9.2 Late joiners
A client that attaches mid-flight needs no history and no events: everything
in progress is state and is in the snapshot.
| in flight at attach time | where the late joiner finds it |
|---|---|
| assistant message streaming | `turn.message` (partial), `turn.generation.stage === "streaming"` |
| provider backoff | `turn.generation = { stage: "retrying", retryAt, lastError }` |
| tool running | `turn.tools[i].status === "running"`, `output` so far |
| tool waiting for approval / a question | `turn.tools[i].waitingOn === ns` (+ the plugin's own Chord service instance, `plugins.md` §6) |
| compaction | `compaction` |
| background job | `tasks[id].status` |
| queued follow-ups | `inbox` |
This is the concrete difference from an event protocol: Codex must answer the
same question with `thread/resume` + `active_turn_snapshot()` + `activeFlags`,
and still cannot show "retrying".
### 9.3 Reconnect
A client keeps `lastRevision`. If an envelope arrives with
`revision !== lastRevision + 1` (only possible on a transport drop between a
Chord RPC service and its client; in that transport Chord's own
`ReplicatedStateDelivery.sequence` plays this role), the client discards its
replica and opens a fresh watch. Nothing is client-specific, so there is nothing
to resume, acknowledge or replay.
### 9.4 Concurrent decisions
Any action that answers something the view shows (approve a tool, abort an
input, change config) is a commit on the line. Two clients acting on the same
item are serialized: the first commit records the outcome, the second sees the
recorded outcome inside its own transaction and rejects with a typed error
("already decided", `already_placed`, …). Both then receive the same envelope
and converge. Plugin APIs must therefore do their check-and-record inside one
commit (`plugins.md` §2, `memoOnce`), never check-then-commit.
### 9.5 Owned conversations
A subagent is its own conversation with its own watch. The parent view shows
only what the parent's tool put in its slot (`details`, `continuedBy`); a UI that
wants to render the child watches the child. Presentation services may merge the
two into one derived document if a client wants a tree.
## 10. Plugins
Everything an extension can contribute (tools, hooks, task kinds, entry kinds,
namespaces), where its durable state lives, how it appears in this view, and how
it is packaged as Chord facets is specified in `plugins.md`. The view-facing
rules are:
- an extension's presentation is only what it projects: the namespace's
declared `view` projection under `plugins[ns]` (nothing by default), the kind's
`describe()` output under `tasks[id].status`, the core-rendered tool slot fields
(`status`, `waitingOn`, `output`, `progress`, `details`, `continuedBy`, `entry`),
and entries. Raw namespace state, memos and other slot working state never
appear;
- extension events are `plugin.<ns>.*`, emitted with the namespace token from the
extension's own commits; core events remain core-only;
- nothing about Chord services is visible in the view. A keyed service instance
(an open approval dialog) is a channel, not state; the corresponding state is
`waitingOn` on the slot.
## 11. Kernel implementation notes
- One tracked view document per watched conversation (§9.1); the kernel applies
each commit's effects to it (entries, config keys, inbox, turn, compaction,
tasks, plugins) through the proxy and flushes.
- Core closures record events with `tx.emit(event)`; the kernel attaches the
list to the envelope. Events are core authority; extensions emit only through
a namespace token (`plugins.md` §3).
- `turn.generation`, `compaction` and `tasks[id].status` are derived from task
records by kind-owned `describe(task)`; the raw records never leave the kernel.
- Display-only assistant entries carry the message in `data.display` and no
`model`, so context derivation needs no special case.
- Envelope assembly happens on the line; listener dispatch off the line,
synchronous and in order; a listener throw closes that watch only.
- No `AsyncLocalStorage`: line re-entry is detected through the Chord context
the Session hands to its callbacks (`hardening-handoff.md` §7).
- Chord bridge (`plugins.md` §6.1): a bounded ordered adapter applies one raw
envelope's ops to the Chord `MutableReplicatedState` and calls `publish()` in
the same synchronous block (Chord's `subscribe()` publishes pending mutations
itself, so apply and publish must not be separated by an await); overflow or
publish failure closes and respawns that instance; a persisted commit is never
failed by a listener.
- Tests: for every row in §6, assert the exact ops and events of that commit
(memory backend, fake provider), plus snapshot-equals-fold after every
envelope and a head-in-parent capture case.
## 12. Open items
- `warning` payload shape (source enum vs free string).
- Application-level backpressure for slow asynchronous consumers is TBD; the
raw watch itself is not (synchronous listener, bounded pre-start buffer).
- Whether `turn.tools` should keep finished slots of the previous assistant
message when the continuation generation prepares (current: cleared at
`prepared`; Codex keeps all items of the turn). Recommendation: keep them, keyed
by call index within `turn.exchanges[]` only if a UI needs it; otherwise
entries already hold the results.
- Whether plugin events need a schema registry for RPC clients (the plugin
token could carry TypeBox schemas per event name).
- Whether plugin-declared **extra documents** are needed for large plugin state;
start with slices in the core documents and add shards only with a
demonstrated cost.

2492
packages/agent/docs/pico2.md Normal file

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,254 @@
# Post-WP05 roadmap audit
**Audit baseline:** `5507d76ee` (`dev`, 2026-08-27).
**Status:** Planning inventory, not a behavior contract. [`harness.md`](harness.md) remains normative where it agrees with the current product boundary. Contradictions listed below must be resolved explicitly; this document does not silently choose one side.
## Scope and method
This audit covers the durable AgentHarness and its directly coupled Session backends and presentation path:
- `packages/agent/src/harness`, `packages/agent/src/search`, and their tests/docs;
- `packages/session-backends/sqlite-node`;
- the current `packages/protocol`, `packages/client`, and `packages/server` presentation path;
- `packages/coding-agent/src/experimental` and its focused tests where they host that path;
- telemetry plumbing in `packages/telemetry`, `packages/ai`, and `packages/agent`.
The inventory was checked against current source, tests, package READMEs, the complete `harness.md`, completed WP00–WP07 status, the actionable WP08 handoff, explicit stubs/TODOs/skips, and the shipped package boundaries. Historical handoffs are not treated as backlog when current source and the completed WP05 contract supersede them.
## Executive result
WP05 is complete through M10. Its remaining assistant-output work is owned by the [mobile assistant-output handoff](mobile-handoff/01-harness/05-assistant-output/message-update.md) and its numbered prerequisites. The current Harness execution graph has no unfinished runtime path: `watchSession()` is the sole `SliceNotImplemented` Harness method.
That does **not** mean the surrounding durable system is complete. The remaining audit findings are:
1. normative JSONL snapshot-compaction behavior with no implementation;
2. one required Harness method stub (`watchSession`);
3. a deliberate removal of raw `RemoteSession` that conflicts with later normative WP06/`harness.md` text;
4. a public search type skeleton that conflicts with the newer search design and has no implementation;
5. a full telemetry vocabulary whose only production span is the tool-hook span;
6. smaller repository, client-watch, query-bound, documentation, and end-to-end-test gaps.
WP07 completed SQLite host-ownership alignment and live-source fork support after the audit baseline; its historical handoff is [`work-packages/07-sqlite-host-ownership-live-forks.md`](work-packages/07-sqlite-host-ownership-live-forks.md). WP08 now owns the separate named-branch, tree-state, and bounded-memory fork redesign; its actionable handoff is [`work-packages/08-named-branch-streaming-forks.md`](work-packages/08-named-branch-streaming-forks.md).
## Required missing functionality and contract contradictions
### R12 — Session-wide Harness watch
**Evidence**
- `AgentHarness.watchSession(context)` is public in `src/harness/agent-harness.ts`.
- `Harness.watchSession()` throws `SliceNotImplemented("watchSession")` in `src/harness/runtime/harness.ts`.
- `SessionSnapshot` currently contains only `{ lanes: LaneInfo[]; faulted: boolean }`.
- Lane watch, event buffering, delivery-tail barriers, and `resnapshot()` already exist in `src/harness/events.ts` and `src/harness/runtime/lane.ts`.
**Remaining boundary**
Define one coherent capture and fold for dynamic lane inventory and fault state. Decide whether the intentionally small `SessionSnapshot` stays small or gains session metadata/stats/global configuration. Then implement snapshot-before-events, lane creation, resnapshot, listener reentrancy, close/fault behavior, and a session reducer if event-only replication is promised.
**Dependency**
Independent of the mobile assistant-output handoff and SQLite internals. It should precede any revisioned Transcript service or remote session-wide observation built on it.
### JSONL snapshot compaction
**Evidence**
`harness.md` §1.7 normatively specifies temp-file-and-rename snapshot compaction, preserved sequence high-water marks/list cursors, threshold checks on open, and reclamation after terminal/outcome deletions. `JsonlStorage` implements atomic creation, torn-tail repair, legacy-v3 rewrite, append, and fork snapshots, but no current-state snapshot rewrite or dead-byte accounting exists.
**Consequence**
Superseded `pi.op.state`, deleted pending payloads, deleted tool checkpoints, and deleted assistant-frame lists remain physical bytes indefinitely. Generic compaction and the [mobile assistant-output handoff](mobile-handoff/01-harness/05-assistant-output/message-update.md) are complementary. Compaction reclaims dead session-scoped write history after the fact; the handoff moves pending assistant/tool output into ephemeral scopes so it never becomes main-log history, and replaces full/per-frame replication with Chord op batches.
**Dependency**
The assistant-output handoff does not depend on J1: scoped storage is the intended lifetime mechanism for pending output, while J1 remains the reclamation mechanism for superseded session-scoped state. Measure both independently so their effects are not conflated.
### Remote Session contract contradiction — decision required
**Current product boundary**
Commit `f8a6e670d` deliberately deleted `RemoteSession`, its raw Session RPC protocol, and the server mutation-scope manager, replacing them with attachment-fenced routed semantic services. Current protocol/server READMEs explicitly state that real `Session` and `AgentHarness` objects remain process-local. The shipped path supports Session discovery/creation/attachment, main-lane prompt/watch, and allowlisted plugin-service calls; it does not expose `Session`, `SessionMutation`, values/lists, branches, or storage over RPC.
**Conflicting contract**
WP06, written after that deletion, requires the “current keyless RemoteSession mutation transport,” includes remote begin/read/commit/publication/end in required tests and stop conditions, and forbids its removal. `harness.md` §§2.8 and 9.1 likewise state that local and remote implementations preserve that lifecycle. No such implementation, protocol schema, client facade, server-held scope, worker adapter, or conformance test exists at the audit baseline.
**Required decision**
Choose one before scheduling implementation:
1. **Process-local Session remains intentional:** remove the false RemoteSession requirements from normative current-state docs while preserving semantic service RPC; or
2. **RemoteSession is required:** commission a dedicated package for mutation begin/read/commit/end, disconnect/timeout cleanup, publication-before-end, values/lists/branches/entries/stats, protocol validation, client/server/worker adapters, and remote conformance.
Do not count the current lane-watch compatibility RPC or plugin-service RPC as RemoteSession. Do not restore the deleted 507-line facade unchanged: it predates the keyless Session/Branch contract and relied heavily on untyped decoding.
### Telemetry contract exceeds implementation
**Evidence**
`src/harness/telemetry.ts` and generated `docs/telemetry-schema.md` declare `pi.ai.request`, operation, checkpoint, turn, step, tool, hook, sleep, event-handler, and session-write spans. Production source starts only `pi.harness.hook`, and only for registered `before_tool`/`after_tool` handlers. AI options propagate `telemetryContext`, but no provider path starts `pi.ai.request`. Server request ingress has cancellation but no trace carrier or client/server RPC spans. `TODO_CONTEXT` remains at transport/worker lifecycle and event-delivery boundaries.
**Remaining boundary**
Treat this as separate packages:
1. local Harness/Session/AI instrumentation and runtime tests;
2. RPC trace-carrier/client/server propagation;
3. an optional application-selected exporter/adapter.
First reconcile whether every declared span is still wanted. If retained, implement it; if not, remove the unsupported public schema surface and correct `harness.md`. Do not mix telemetry with Context/RPC cancellation, which already has independent request-ID signaling.
### S3 — Search
**Evidence**
`src/search/index.ts` exports a public `SessionSearchService` skeleton with `sync()`, `notify()`, and array-returning `searchEntries()`. `harness.md` §2.8 instead specifies a standalone service, separate catch-up/notify utilities, generation-aware cursors, and optional `AsyncIterable` entry search. There is no factory, sync utility, cursor store, projection, or source SQLite FTS implementation. At the audit baseline the SQLite README advertised nonexistent `createSqliteSessionSearch()` behavior; this audit corrected that README rather than treating the absent API as implemented.
**Remaining boundary**
Before implementation, replace or reconcile the draft public interface and decide metadata filtering (`cwd`), candidate restriction, or indexed metadata. Post-filtering after ranked `limit` is unsound. Then implement standalone catch-up and a separate SQLite FTS5 projection; do not add repository search methods.
### R11 — Schema migrations, activation-gated
No current format-4 migration is required. Memory is current-only; JSONL and SQLite reject unsupported storage versions; SQLite runs only idempotent `001_initial.sql`. R11 becomes required immediately before the first incompatible durable storage version/address/state change after format 4 is stabilized. It is not prerequisite work for the mobile assistant-output handoff or current WIP format replacement.
When activated, it must provide ordered transactional migrate-on-open under exclusive ownership, version-specific JSONL decoding plus post-migration compaction, and total mappings for every reachable open operation leaf and surviving value/list.
## Correctness and data-safety debt
### SQLite host ownership and live-source forks — completed by WP07
The authoritative product rule is in `plugins.md`: exactly one host-assigned process owns writable Session authority; normally it is the Session worker, while the server may temporarily own a newly created or forked destination before closing it and handing it off. Storage backends do not implement writer ownership. The server closes a worker before destructive repository administration.
SQLite now follows that rule: the writer-lease schema/module, claims, renewal timer, lease-loss path, and pre-commit callback are gone, with no replacement lock or ownership primitive. Create/open/fork/delete retain repository-local ID reservation. Metadata open and deletion use a true no-create read-write mode; listing and external fork sources use no-create read-only connections.
Same-repository forks retain the source `commitQueue` ordering seam. A source owned elsewhere, including a live worker, is read from its exact canonical container through one independent read-only deferred WAL transaction. Focused per-file and shared-container tests commit a complete later source transaction after the reader establishes its snapshot and before it closes: the first fork excludes the transaction wholly and a later fork includes it wholly.
WP07 also completed canonical `(containerPath, sessionId)` active identity, safe explicit-ID filenames, custom `databasePath` parent creation, Session-scoped shared deletion, WAL/SHM cleanup, and all-settled SQLite repository close. Writable open/delete reject foreign metadata; foreign fork sources are read only from their exact path. Equal-`createdAt` list ordering still has no deterministic tie-break and remains a later behavior-preserving cleanup.
### Repository close ownership is unspecified
`JsonlSessionRepo.close()` contains the only active Agent source TODO and closes no open Session handles. Memory still uses fail-fast `Promise.all`; SQLite now performs backend-local all-settled cleanup of currently open handles, but an already-admitted create/open/fork can still register a handle after repository close captured that set. `SessionRepo` itself declares no `close()` method and shared conformance does not define repository-to-handle ownership or draining of admitted repository operations. Resolve ownership and common cleanup in one repository-lifecycle package; do not patch one backend further without deciding the common contract.
### Client watch staleness after disconnect
`Client` clears its active watch-listener map on disconnect, but existing `LaneWatch` objects keep local `ready`/`started` state and old watch IDs. After reconnect/reattach they can call `start()` or `resnapshot()` and fail remotely instead of deterministically rejecting as stale, contrary to `packages/client/README.md`. Service-subscription objects have the sibling problem: their listeners are cleared and surviving objects become silently dead. Fix both with connection/attachment incarnation fencing and focused reconnect tests. This is independent of R12: the current client method is a compatibility main-lane watch.
### Query bounds and SQLite bind limits
- SQLite `getEntries(ids)` emits one placeholder per requested ID and can exceed the engine’s variable limit.
- Entry, usage, and branch limits use ad hoc `Math.max(0, limit)` behavior. Memory and SQLite diverge for `NaN`, infinities, fractions, and extreme values; unlike list reads, there is no shared normalization contract.
Define cross-backend query-limit semantics in agent conformance, then chunk SQLite ID lookups. This is a storage-contract hardening package, not part of WP07.
### Harness contract and conformance closure
- The public `OperationStatus` includes `"running"`, but lane inspection, snapshots, and `reduceLaneSnapshot` currently produce only `"open"` or `"aborting"`. Define and implement its producer or remove the dead variant.
- The pre-rewrite abort contract bound/published `operation_abort` before resolving the cancellation promise and signalling the live gate. Current `Lane.command()` materializes the result — resolving/signalling — before constructing and binding the event batch, although it still binds recipients before releasing the Session mutation line. Decide whether to change the implementation or retain/document the current no-interleaving order; add an explicit ordering test.
- The production gate-close contract permits only `HarnessClosed | HarnessFault`, but the private source primitive accepts any `Error` and isolated tests use that wider type. Narrow the source declaration and fixtures or explicitly retain the private widening.
- Part 9 of `harness.md` is the required conformance matrix. Existing focused tests cover the graph extensively, including cancellation reconciliation over all 13 leaves, but there is no audited one-to-one proof that every close/reopen leaf case and every race row has both deterministic orders. Audit the matrix and add only the missing cases rather than claiming blanket completion.
Keep this package separate from telemetry, RemoteSession, and the mobile assistant-output handoff; it is local contract/test closure.
### Disabled real worker persistence regression
`packages/coding-agent/test/experimental-remote-runtime.test.ts` still skips “completes and persists a prompt through the worker-owned Harness” with the obsolete note “Re-enable with runtime no-tool execution.” No-tool execution now exists. Re-enable or replace it with a deterministic faux-provider real-worker persistence test; do not use a real paid provider.
## Performance debt
### Mobile assistant-output handoff — durable and replication amplification
The user-supplied motivating mini Session outside the repository is 303,920 bytes across 569 physical lines. These external measurements are evidence, not a reproducible checked-in fixture:
- 477 assistant-frame appends totaling about 118,418 serialized write bytes;
- 12 frame-list deletes; physical lines mentioning the frame namespace total 148,214 bytes;
- about 51,568 bytes of superseded `pi.op.state` writes and 26,192 bytes of one structural preparation, showing why generic JSONL compaction and frame-specific bounding are distinct.
The authoritative design is the [mobile assistant-output handoff](mobile-handoff/01-harness/05-assistant-output/message-update.md), following the numbered `01-harness` prerequisites in [`mobile-handoff/README.md`](mobile-handoff/README.md). Chord delta tracking has landed; scoped storage, tool-output integration, and assistant-output integration have not.
Implementation must add a deterministic repository fixture and measurement script that reproduces or replaces these figures, then measure Memory logical elements, SQLite rows/pages/WAL, JSONL peak sidecar/main-log bytes, and reopen/replay time. The handoff must preserve unknown-outcome recovery, invocation fencing, non-blocking provider streaming, and settlement retirement while eliminating per-frame durable writes and quadratic `message_update` replication. Numerical budgets belong in its implementation handoff/tests, not in a competing standalone design.
### SQLite branch divergence
`createDivergentBranchForEntry()` copies every row after the newest compaction; with no compaction it copies root-through-parent. A first divergence from a long uncompacted transcript is therefore O(history) writes. This exposes an internal contradiction in `harness.md` §2.6: its opening bounded-prefix promise conflicts with its own compaction-based copy algorithm, which the implementation follows. Change both the specification and segment representation so a divergence can reference a covering segment at the parent boundary. Preserve shared-container support and add large uncompacted divergence plus chain-soundness tests/benchmarks.
### Fork contract and materialization — WP08 actionable
All current backends materialize source-sized fork snapshots. Branch scope defaults to `main` without named-Branch ancestry validation; tree scope omits application values/lists; JSONL closed-source fork replay may repair its source. WP08 replaces that contract with required named-branch or tree scope, one closed built-in-state policy, complete current application state for tree forks, and backend-specific bounded-memory copy procedures. It preserves WP07's host ownership, physical identity, same-repository ordering, and independent live-source WAL boundary.
### SQLite catalog, statements, stats, and reclamation
- Default per-session `list()` synchronously opens/configures every SQLite file serially and silently skips failures. A shared container or external catalog is the scalable deployment choice; bounded async scheduling does not make `DatabaseSync` nonblocking.
- Most hot queries prepare a fresh statement each call. Cache narrowly owned statements after measuring.
- Each usage row parses and rewrites the full aggregate JSON usage payload.
- Shared-container row deletion does not reclaim pages. Define maintenance/VACUUM policy separately; never add `VACUUM` to ordinary deletion casually.
These are measurable optimization/operations packages, not correctness fixes.
### Pending-payload amplification and mutation-line parallelism
Queued payloads deliberately write `pendingEntry → immutable entry`; measure pathological payloads before changing it. Keyed Session mutation lines remain optional and require profiling plus a fresh mutable-ownership audit. Neither is current correctness work.
## Behavior-preserving cleanup and documentation repair
Completed by this audit:
- rewrote the SQLite README’s nonexistent `SqliteSessionRepository`/search APIs, wrong `await using` and `appendMessage` example, FTS trigger/rebuild claims, and “one shared connection” claim;
- reconciled `harness.md`’s default one-file SQLite wording with the supported optional shared container;
- reconciled `harness.md` Part 8 with this audit and corrected stale drive-ownership/RPC-cancellation status in `telemetry.md`;
- corrected coding-agent settings docs: install telemetry configuration also controls selected provider attribution headers.
Remaining cleanup:
- **Harness-owned DTO boundary:** the durable Harness still imports legacy agent-loop DTOs from `src/types.ts`: `AgentMessage`, `AgentToolResult`, `AgentToolCall`, `AgentTool`, `QueueMode`, and `ThinkingLevel`. Make a later one-shot cut to independent `HarnessMessage`, `CustomHarnessMessages`, `HarnessToolResult`, `HarnessToolCall`, `HarnessQueueMode`, and `HarnessThinkingLevel` definitions; keep `AgentHarnessTool` as the executable Harness configuration type. Update Harness internals, root exports, declaration-merging tests, documentation, and experimental coding-agent consumers together. Do not alias the new message/tool DTOs back to legacy types, because that preserves the coupling this work is intended to remove.
- Preserve shared-container support; do not remove it incidentally.
- Remove or use unused `insertEntryRow()` and `insertUsageLedgerRow()`.
- Consolidate duplicated SQLite branch payload/structure scan plumbing only after correctness tests pin both paths.
- Decide whether unused `sessions.metadata` and unmeasured indexes have a future owner before schema removal; do not change schema casually.
- Consolidate `startAiSpan()`/`startHarnessSpan()` implementation only if the telemetry surface is retained.
- Mark historical durability documents as delivered where they still read as implementation queues. WP00–WP07, `runtime-simplification.md`, `values.md`’s old consumer deferrals, and external-finalization designs are not active runtime backlog.
## Optional or deferred product capabilities
These are not blockers for the durable Harness:
- standalone S3 search after its API decisions;
- Accounts removal and revisioned Transcript production;
- authenticated workspace/client authorization for the experimental local server;
- private returned references, service flow control, multi-pane presentation, plugin kernel/reload completion, and version-skew negotiation;
- administrative precise rewrite tooling;
- a partitioned Postgres backend/retention policy;
- generic remote Harness/object capabilities;
- DeltaState or delta replication before concrete snapshot pressure justifies it;
- a production telemetry exporter;
- schema migrations before an incompatible stabilized-format change activates R11.
## Recommended dependency order
The order is by data safety first, then dependencies. Independent tracks may proceed in parallel only when they do not edit the same contracts.
1. **Harness contract/conformance closure.** Resolve `OperationStatus.running`, abort signal/event binding order, gate-close typing, and the Part 9 coverage matrix.
2. **Remote Session decision (decision only).** Resolve the false normative boundary early. If process-local wins, repair the docs. If raw RemoteSession wins, later create a dedicated protocol/client/server/worker package; do not fold it into telemetry or R12.
3. **Client watch/subscription staleness** and **repository lifecycle contract.** Small independent correctness packages; complete them before expanding server/worker lifecycle semantics. The lifecycle package must also address Memory's fail-fast repository close.
4. **[Mobile Harness handoff](mobile-handoff/README.md).** Follow its numbered prerequisites through scoped storage, tool output, and assistant output; preserve all recovery boundaries and land deterministic amplification measurements.
5. **JSONL snapshot compaction.** Implement the already-normative physical reclamation path and metrics for remaining session-scoped history.
6. **R12 session-wide watch.** Complete the only Harness method stub before building revisioned Transcript/session-wide remote observation.
7. **Telemetry, if retained:** reconcile schemas, then local instrumentation, then RPC propagation, then an optional exporter. RPC propagation follows the Remote Session/product-boundary decision.
8. **WP08 — named-branch and streaming forks.** Implement the actionable handoff without reopening WP07 ownership or lifecycle decisions.
9. **SQLite branch/query performance hardening.** Keep separate from completed WP07 ownership alignment and WP08 fork semantics; require benchmarks.
10. **S3 search.** Resolve its API/filter/cursor decisions, then implement catch-up and the standalone FTS projection.
11. **R11 migrations.** Activate immediately before the first incompatible stabilized durable schema change, not earlier.
## Stop conditions for roadmap accuracy
This inventory must be updated when any of these facts changes:
- `watchSession` stops being the sole Harness `SliceNotImplemented` method;
- raw RemoteSession is either recommissioned or removed from the normative contract;
- JSONL snapshot compaction lands;
- telemetry schemas are implemented or removed;
- S3’s public API is reconciled;
- a durable format change activates R11;
- WP08 lands or its fork contract changes;
- the host-authority contract changes.

215
packages/agent/docs/rpc.md Normal file
View file

@ -0,0 +1,215 @@
# Facet Service RPC
Chord owns the application-neutral service semantics and pluggable strict-JSON connection
boundary. Pi owns the concrete wire envelope, routing, attachment state, and error adapters
described here. The current implementation treats `JsonValue` as a static contract and defers
runtime rejection of unsupported values to the concrete serializer.
> **Status:** Design specification for experimental facet-service RPC semantics.
## Role
`provide()`/`use()` and `provideMany().spawn()`/`observe()` are the facet system's hidden RPC. Facets share TypeScript service contracts, while the transport carries service/member identifiers, strict JSON values, request/subscription correlation, keyed generations, and binding control messages. Hosts construct the typed local implementation or facade; TypeScript types and arbitrary objects never cross the wire. Independently loaded processes may temporarily run different source generations, so their service contracts must remain forward compatible across the supported skew window; version negotiation remains deferred.
```text
session/server facet: provide() / provideMany().spawn()
↕ hidden service RPC
presentation/session facet: use() / observe()
```
The service system is the extension boundary. Presentation facets receive semantic services and replicated state; they never receive a raw Harness, Session, tool registry, hook registry, credential store, or storage handle.
## Non-goals
Do not serialize `Context`, `AbortSignal`, telemetry objects, callbacks, tools, hooks, functions, or arbitrary object graphs. Do not make core Harness or Session implementations aware of transport mechanics. Do not make disconnect perform durable or service-owned cancellation. Do not build per-method codecs for values already constrained to JSON.
## Service contracts and typed facades
A service token is a shared TypeScript contract and stable service ID. It is not a generated descriptor and creates no provider. Tokens are remotely publishable by default; process-local tokens declare `{ local: true }`. `provide()` adds one singleton implementation to the host graph. `provideMany()` registers one multi-instance service owner during facet setup and returns a `ServiceSpawner` whose later `spawn()` calls publish instances. The host automatically publishes every non-local provision. A token has one mode in one host service graph: mixing singleton and keyed use is an error.
The provider classifies every exposed implementation member as a method or Chord-created `ReplicatedState` and publishes that member table in subscription snapshots. The consumer obtains member names from ordinary property access—for example, a JavaScript `Proxy` receives `"state"` for `models.state` and `"refresh"` for `models.refresh(context)`. Accessed slots are validated against the provider-announced kind when the facade binds.
Local and remote `use()` both return a stable, lazy typed facade shared by consumers of that token. During synchronous facet setup the facade is disconnected, so setup can capture it but cannot invoke methods, read state, or register member subscriptions. After assembly, a local facade resolves through a direct process-local implementation slot and a remote facade binds through the host's connected services. Reloading the providing facet temporarily marks that same facade unavailable, then swaps its target; RPC singletons clear readiness and install a complete replacement snapshot on their existing subscription so captured methods and member facades address the replacement. While no provider is bound, invoking a method fails and state remains unhydrated; no call is queued merely because it was made through a proxy.
Remote methods return promises and accept and return strict JSON apart from their declared `Context`; `void` is a successful response without a result field. Private returned references are not supported. The client removes the context before transport and the receiving host constructs a fresh local context. The contract position is host-controlled and must be consistent; the examples use one required trailing `Context`. Business absence is JSON `null` or an options object, never transported `undefined`.
Use static assertions and runtime validation. Static checks constrain remote methods and replicated-state members; runtime boundaries reject unsupported members and non-JSON arguments, results, and state values. TypeScript supplies typed facades but does not authenticate a peer or create runtime metadata.
`{ local: true }` removes only remote publication and its wire-contract restrictions. Local and non-local provisions otherwise use the same dependency ledger, activation order, stable singleton slots, keyed-instance generations, observer cancellation, disposal, and provider-facet reload. Local singleton slots and the local keyed registry hold arbitrary object contracts directly; non-local provisions additionally install their implementations in the remote provider.
## Dependency ledger
Type erasure does not hide service identity: every service token retains its stable ID at runtime. Facet environments are created by the host with a non-forgeable owner identity, and their setup-time service methods append to a generation-scoped ledger:
- `provide()` records a singleton provision;
- `provideMany()` records a keyed provision;
- `use()` records a singleton requirement; and
- `observe()` records a keyed requirement.
Facets always call unqualified `env.use()` or `env.observe()`; routing is not encoded in the call. These operations return source-independent disconnected handles during setup. After setup, each connection returns its provider-generated service catalogue, and the host binds every requirement to its local provision or exactly one connected provider. The method supplies the mode and the token supplies the ID. The host therefore needs no reflection over the erased `T` and no handwritten parallel dependency list.
First acquisition or provision is permitted only during facet setup. Later commands, hooks, and activation callbacks use setup-acquired singleton facades, observer registrations, or `ServiceSpawner` capabilities. In particular, dynamic instances are spawned through the capability returned by `provideMany()`; late spawning cannot introduce a previously undeclared provision.
After setup, each host privately resolves the recorded requirements against local provisions and connection catalogues to reject missing providers, duplicate remote offers, and mode mismatches and to derive lifecycle edges. A selected-Session connection with no live attachment may provisionally accept unresolved requirements as unavailable; attachment validates them against the worker's generated catalogue and caches that catalogue for later detached generations. Connection bindings are generation-owned and include only selected requirements, so failed or retired generations release their subscriptions without disposing the underlying transport connection. This internal service graph is distinct from the module loader's source import graph; it is not a facet-authored or facet-visible plan.
## Bindings and identity
A presentation host combines services from its connected server and selected Session in one graph. All of its facets use the same unqualified environment API. A Session-service call never accepts a client-selected durable `sessionId`; the server authorizes and routes the presentation's selected Session binding to its worker.
### Server control plane
Session listing and management are ordinary server singleton services, not generic remote `Session` methods. `SessionDirectory` exposes presentation-safe session summaries as replicated state. `SessionManagement` exposes `create`, `remove`, `attach`, and `detach` methods. A TUI or web facet consumes both through its environment:
```ts
const directory = env.use(SessionDirectory);
const management = env.use(SessionManagement);
```
The server binds one service provider to each presentation connection. It derives workspace and client authority from locally authenticated connection identity, not from summary fields or method arguments. It may project directory state per client; either way, summaries never expose server-private fields such as owner IDs or working directories.
`management.attach(sessionId, context)` changes the selected-Session services in the presentation host. The server closes the presentation's previous Session-scoped requests, subscriptions, and observer tasks, binds the presentation's Session services to the worker, then hydrates their singleton state and keyed-instance directory. The server authorizes the selected Session against the connection identity. Attachment state is host control state reporting this selection and its health; it is not a directory service. `detach()` performs the same cleanup without a replacement.
The host needs a private, host-owned binding incarnation for that route. It changes when the presentation attaches, detaches, switches session, or replaces a failed worker. Its representation is deliberately unspecified. The binding prevents a delayed frame for the old selected session from being applied to the new one; it is not a facet-visible service value or a substitute for authorization.
A replicated-state source has structural identity:
```text
(provider binding, service ID, optional instance key + generation, member name)
```
There is no separately discoverable state ID. An added instance key is an application-level logical key. Its host-owned generation changes when a closed key is reused, so a stale proxy cannot call the replacement. `requestId` identifies one transport invocation for response and cancellation. A Harness/tool `invocationId` may be a useful instance key—as in the question example—but it does not replace the service, binding, or generation parts of the live address.
## Calls, context, and routing
A call carries enough control-plane information to select a provider binding, service, optional keyed instance, and member, plus a request ID, JSON arguments, and trace carrier. The server may parse those control-plane fields to route a Session call, but it does not parse facet business payloads or load facet contracts. The service endpoint validates the member and values, creates a request-local abort controller and `Context`, installs authenticated identity, and invokes the local implementation.
The client maps `context.abortSignal` to cancellation of that one request. Disconnect cancels that connection's active calls and closes its subscriptions. Neither action cancels service-owned work or writes durable Harness cancellation. Per-client request correlation reaches the worker so request IDs from different presentations cannot collide.
## Replicated state and keyed instances
`ReplicatedState` is authoritative latest-value replication, not event history, durable storage, a CRDT, or multi-writer state. A cold replica has `value === undefined`; subscribing before hydration records a listener without invoking it. Hydration installs a complete snapshot atomically before later updates are delivered, so there is no snapshot/update gap. Once hydrated, `value` is synchronous and subscribing reports the current value followed by later updates. The first snapshot callback uses a fresh hydration context; an already hydrated replica uses a fresh local delivery context rather than retaining the original write context. State values are borrowed immutable JSON and are not defensively cloned; callers must not mutate or retain them.
Remote hydration uses a fresh delivery context parented to the subscription, while updates reconstruct fresh delivery contexts from source trace metadata. Disconnect, provider withdrawal, replacement, and route switching clear readiness. Reconnect or replacement installs a complete snapshot before later updates. The transport buffers updates that race hydration and checks their sequence. Acknowledgements, flow control, and gap recovery remain separate protocol mechanics.
`observe()` is keyed-instance discovery, not a `ReplicatedState` containing proxies. It reconciles a complete initial directory with ordered additions, replacements, and removals. Each instance's initial state members hydrate before its observer task starts. Closing an instance rejects new calls, aborts only that instance's observer task, and allows admitted calls to settle. A Session facet's `env.observe()` registration aborts old tasks when that facet generation closes; the replacement generation reconciles the fresh directory.
## Private returned references
Private returned references are outside the initial service contract. Prefer keyed services for discoverable live instances. If a concrete feature requires caller-private remote identity, its reference must be passed explicitly rather than discovered by `observe()` and scoped to the recipient and provider binding.
No generic Harness projection is part of this design. Raw Harness, Session, lane, tool, hook, and storage objects remain local authority. If a future integration needs a remote callback or a general object capability, it needs a separate explicit protocol and policy; it is not an extension of service RPC.
## Context, cancellation, and telemetry
Every remote method receives a fresh local `Context`; the sender's object, signal, telemetry implementation, and arbitrary typed values never cross the wire. The client maps the call's abort signal to that request and injects a trace carrier. The endpoint constructs a request-local abort signal and telemetry parent. Cancellation is forwarded through the server to Session workers and remains isolated by client plus request ID.
The span relationship is:
```text
caller
└─ rpc.client
└─ rpc.server
└─ service implementation
```
Three cancellation domains remain separate: aborting one RPC invocation; explicitly cancelling service-owned work such as `job.cancel()`; and durable Harness cancellation such as `requestAbort()`. Transport cancellation and disconnect perform only the first. Work that outlives a call must detach into a service-owned task with its own controller and telemetry root.
## Security and lifecycle
Only loaded service tokens not marked local may be registered at the remote boundary. Only the owning `ServiceSpawner` may spawn instances. Services marked `{ local: true }` are never discoverable remotely. Local services may use unrestricted object contracts. Remote providers validate member kinds, while concrete serializers enforce JSON business values. Clients cannot forge control envelopes as ordinary values, choose instance generations, select a different Session route in a service call, or cancel another client's request.
The server authenticates connections, authorizes attachment, and reconstructs client identity in the service `Context`. Ordinary business arguments never carry authority. Credentials, prompts, completions, tool data, filesystem contents, and other sensitive values require an explicit presentation-safe contract.
Facet environments own registrations, added service instances, observations, and resources explicitly registered through `own()`. Connection bindings own their transport subscriptions and active request controllers. Already-admitted inbound calls are not attached to the providing facet lifecycle: provider withdrawal rejects new calls, but an admitted method may continue while the old facet deactivates. The provider's own Session work remains alive unless its lifecycle policy stops it.
## Tests
The facet-facing semantics are tested over loopback and framed transports:
- setup-time dependency-ledger ownership, rejection of late acquisition, local and remote `use()`, keyed-provider ownership, singleton/keyed mode validation, token-driven RPC publication, lazy member access, stable local and RPC singleton facades across provider-facet replacement, and `{ local: true }` services remaining unreachable remotely;
- strict JSON boundaries, method context reconstruction, and request cancellation isolation without serializing context values;
- server/Session facet isolation, selected-Session switching, stale-frame rejection, and worker-side per-client request correlation;
- cold and hydrated `ReplicatedState`, snapshot/update race freedom, fresh delivery contexts, and clearing/re-hydration on disconnect, reconnect, and provider replacement;
- instance directory hydration, ordered reconciliation, state hydration before observer tasks, generation-based stale rejection, and task cleanup on close or switch.
Additional tests cover authenticated attachment and identity, telemetry propagation, flow control and gap recovery, plus the question and shared-review application patterns. If private references are added, they require separate lifetime and isolation coverage.
## Open protocol mechanics
Exact service call, cancellation, subscription, snapshot/update, keyed-instance, unavailable, and replacement frames are defined in `packages/protocol/src/protocol.ts`. Provider and namespace layers own member classification, lazy facades, buffering, and sequencing.
Still open are acknowledgements, flow control, sequence-gap recovery, reference collection if references are added, protocol-version negotiation, and how a future multi-pane presentation represents more than one selected Session. Singleton provider replacement must continue to install a complete replacement snapshot on the existing subscription so method and state member slots retain identity.
## Example: directory and selected session
The directory and management services are normal server services. Their contracts carry only presentation-safe values:
```ts
interface SessionSummary {
serverId: string;
sessionId: string;
createdAt: string;
}
interface SessionDirectory {
readonly state: ReplicatedState<{ revision: number; sessions: SessionSummary[] }>;
}
interface SessionManagement {
create(options: { id?: string }, context: Context): Promise<SessionSummary>;
remove(sessionId: string, context: Context): Promise<void>;
attach(sessionId: string, context: Context): Promise<void>;
detach(context: Context): Promise<void>;
}
const SessionDirectory = defineService<SessionDirectory>("pi.session-directory");
const SessionManagement = defineService<SessionManagement>("pi.session-management");
```
A server facet derives the client from an authenticated `Context`, authorizes the requested Session, and performs the binding transition:
```ts
serverContext.provide(SessionDirectory, { state: directoryState });
serverContext.provide(SessionManagement, {
async attach(sessionId, context) {
const client = requireClientIdentity(context);
authorizeSession(client, sessionId);
await attachments.bind(client.clientId, sessionId, context);
},
async detach(context) {
await attachments.unbind(requireClientIdentity(context).clientId, context);
},
});
```
A presentation facet renders and selects Sessions:
```ts
setup(env) {
const directory = env.use(SessionDirectory);
const management = env.use(SessionManagement);
const tui = env.use(Tui);
tui.commands.register("sessions.switch", async (operation) => {
const snapshot = directory.state.value;
if (snapshot === undefined) return;
const sessionId = await tui.select(
"Sessions",
snapshot.sessions.map((session) => ({ label: session.sessionId, value: session.sessionId })),
{ signal: operation.abortSignal },
);
if (sessionId !== undefined) await management.attach(sessionId, operation);
});
}
```
Another facet in the same presentation acquires Session services through the same API:
```ts
setup(env) {
const models = env.use(Models);
// After attach() settles, `models` addresses the selected worker.
}
```
The presentation never routes `models` with a selected `sessionId`; its host routes each service token and transport retains the selected-Session binding. The server closes the prior Session binding's resources before hydrating the new one.

View file

@ -0,0 +1,391 @@
# Existing AgentHarness runtime simplification
## Scope
Rework the real implementation under `packages/agent/src/harness/runtime/` and the canonical durable types in `packages/agent/src/harness/session/types.ts`. This is not the isolated scratch spike.
The execution graph is total and public drive is enabled. Format 4 is still work in progress, so the durable type replacement required no migration or compatibility representation.
## Implementation status
Pre-M6 completion measurements:
- pre-simplification runtime at `eb1185d93`: 5,358 TypeScript lines;
- runtime after the first simplification at `417905647`: 4,667 lines;
- simplified substrate at `0e77e57d9`: 4,654 lines;
- total pre-M6 reduction: 704 lines (13.1%).
Implemented by the simplification pass:
- explicit `ContinueOperationResult<T>` with `cancel_requested` instead of implicit `undefined`;
- concrete generation, deferred-poll, and tool-call phases: prepare immutable input, publish durable intent, perform the effect, publish the durable outcome;
- shared assistant stream protocol consumption and Drive response lifecycle;
- `publishResponse` for common assistant/deferred durable publication and `publishConfigurationFailure` for their identical pre-intent failure transition;
- source-order tool publication, one-read immutable source validation, and shared tool-batch reconstruction isolated in `runtime/drive/tool-placement.ts`;
- same-call tool status defenses, unchecked `toolOperation()` casts, and their result unions removed; genuine sibling/progress/memo checks retained;
- centralized lane-storage classification without duplicate projection reads;
- checkpoint dead-check removal and exact checkpoint preservation;
- shared transcript mechanics in `runtime/transcript.ts`;
- narrower `LanePatch`, no redundant Drive promise-settlement flag, and no unused operation reject arm.
At simplification completion, the largest files were `runtime/lane.ts` (~996), `runtime/drive/tools.ts` (~663), `runtime/drive/checkpoint.ts` (~452), and `runtime/drive/response.ts` (~417).
### WP05 structural status
M6 established structural execution on the simplified substrate. Its first coherent slice added:
- shared compaction-bounded transcript/context reads and committed-entry event decoration;
- caller-owned one-provider-request seams for compaction and branch summaries while preserving retrying behavior for existing non-harness callers;
- direct procedures for every structural `at` leaf in `runtime/drive/structural.ts`;
- once-per-trigger threshold routing and atomic overflow preparation publication;
- per-request structural intent and usage settlement, attempt retry/recovery, hook decisions, compaction/navigation publication, and terminal cleanup;
- focused coverage for threshold/overflow entry, split-turn request accounting, generated and hook results, retry/cap/recovery, model absence, mid-request durable cancellation, navigation, and preparation corruption.
R1a, R2, and R3 then moved queued input to the lane, replaced hydrated family outcomes with immutable operation records, and collapsed the family cross-product to 13 neutral leaves. R1b replaced transitional checkpoint drains and marker fields with one shared atomic boundary planner, a transcript-derived threshold guard, and off-line finish-hook replanning. M7 removed drained control, added atomic drain-and-return cancellation, reconciled all 13 leaves, and installed the total direct dispatcher.
M8 added primitive/convenience lane surfaces, one ordered tagged inbox, idle ownership, snapshot/event replication, `reduceLaneSnapshot`, and remote resnapshot. M9 reconciled the normative specification. M10 forwards exactly one stable ordinary-provider identity derived from Session metadata id plus lane name; structural summaries keep fresh request identities. At M10 completion the runtime is 7,410 lines, `runtime/drive` is 4,147, `lane.ts` is 1,992, `structural.ts` is 1,221, and the new normative reducer is 193 lines. Relative to the reviewed M7 checkpoint, the public/replication surface added 1,130 runtime lines; the durable 13-leaf procedure model stayed unchanged.
Keep the visible durable procedure order:
`prepare → publish intent → perform effect → publish outcome`.
Do not introduce a generic Procedure interface, runner, scheduler, graph, callback plan, or dependency facade. Historical work-package documents remain unchanged.
## Core model
A Lane is one process-local actor over one durable lane projection.
- `Lane.state` is authoritative for tip, configuration, current operation, control, inbox IDs, and the latest operation ID.
- Every supported mutation commits through the Session mutation line and publishes the matching `Lane.state` before releasing it.
- One lane-owned Drive is the sole writer that advances an operation state.
- Same-operation callers observe the same Drive. No caller owns it.
- Invocation cancellation stops only that caller's observation after Drive installation.
- `requestAbort` is the only durable operation cancellation.
- Close seals mutation admission. It neither mutates operation state nor replaces a Drive.
- Process loss destroys all live continuations; recovery starts from the durable state after attachment.
- Session reads during execution dereference content named by `Lane.state`; they never rediscover control state.
## Durable state
`OperationState` has one discriminator, `at`, with 13 direct leaves:
- `starting`
- `checkpoint`
- `assistant.ready`
- `assistant.effect_pending`
- `assistant.retry_wait`
- `tools`
- `deferred.suspended`
- `deferred.effect_pending`
- `summary.deciding`
- `summary.ready`
- `summary.effect_pending`
- `summary.retry_wait`
- `navigation.ready_to_commit`
Every leaf carries one uniform `OperationScope`. The summary quadruple carries one `SummaryTask`; its closed `ResultBoundary` selects in-run checkpoint resumption, standalone finish, or navigation commit. The summary algorithm is derived from that boundary rather than duplicated in state. `Control` remains orthogonal. `ToolBatch` remains a child state machine because parallel tool children genuinely mutate sibling call statuses concurrently.
## State and content boundary
Procedures never read these addresses to decide execution:
- `laneState`
- `operationMeta`
- `operationState`
- `branchTip`
- `laneConfig`
- `operationResult`
They use the current `Lane.state` supplied by the mutation line.
Storage reads remain only for content and cleanup:
- prompt, assistant, deferred-source, final-assistant, and completed-tool entries;
- compaction-bounded branch context;
- `pendingEntry` payloads;
- assistant-frame lists;
- tool arguments, memos, and checkpoints;
- structural preparations;
- staged tool outcomes;
- operation-owned prefix scans required for terminal cleanup.
## Concurrency model
### Operation advancement
While a Drive procedure is alive, no supported concurrent actor can change or remove its operation state:
- inbox methods change only inbox fields;
- `requestAbort` changes only `control`;
- close prevents later mutation admission;
- another same-operation caller joins the existing Drive;
- another operation cannot be accepted while the lane is busy;
- process crash removes the continuation itself.
Consequently, ordinary procedure transitions do not recheck operation existence, operation ID, operation kind, `at`, attempt identity, or nested status. Those are procedure preconditions established by dispatch, not supported races.
The remaining real races are:
1. cancellation arriving before effect admission or before settlement;
2. inbox arrival before checkpoint routing or terminal finish;
3. parallel tool children staging and materializing sibling outcomes;
4. queued frame/checkpoint/memo writes racing effect settlement;
5. retry timers racing cancellation or close;
6. deferred permit consumption;
7. accept/claim serialization on the Session line.
### Cancellation boundaries
Cancellation is checked in three places only:
1. the drive loop before ordinary dispatch;
2. the gate immediately before an external effect starts;
3. effect settlement, which sees current `control` and commits the appropriate cancelled result.
An ordinary transition helper may centrally decline progress when current control is `cancel_requested`. Procedures do not repeat that branch manually.
### Close
Closing the harness:
1. marks harness/lane admission closed;
2. seals and drains the Session mutation line;
3. rejects client observations through a harness-close observation promise;
4. keeps every detached pass promise observed so a later rejection is never unhandled;
5. closes the Session after admitted mutations drain.
Close performs no durable write, installs no replacement Drive, and creates no ownership-loss state. A late effect may return, but its mutation is rejected with `HarnessClosed`.
Whether close also signals process-local provider/tool work is a resource-cleanup policy, not durable state-machine behavior. Do not couple it to Drive replacement or recovery.
## Small concrete mutation API
Replace repeated `LaneCommand` ceremony in procedures with two concrete Lane operations. They are not a scheduler, graph, or action interpreter.
### `continueOperation`
Used for ordinary non-terminal progress.
- Enters the Session mutation line.
- Receives the current authoritative Lane projection and operation state.
- If control is cancelled, returns explicit `cancel_requested` without invoking the semantic planner; callers cannot confuse cancellation with a planner value.
- The planner supplies procedure-specific writes, the next complete `OperationState`, materialization, and events.
- The helper appends the `operationState` write and publishes the matching process-local operation projection.
- It does not verify an expected state or `at` value.
### `settleOperation`
Used after an admitted provider/tool/structural effect and for genuine parallel child transitions.
- Enters the Session mutation line even when control is cancelled.
- Supplies current control/inbox fields and the process-local effect result to the semantic settlement planner.
- Atomically commits payload, usage, tip movement, cleanup, and the classified next state.
- Appends the canonical operation-state write and publishes the matching projection.
- When the planner returns a terminal decision, appends the immutable `operationResult`, idle `laneState` with `lastOperationId`, and the idle process-local projection.
The caller supplies the typed outcome, cleanup/publication writes, last result, and event. Terminal business decisions remain visible in the owning procedure.
## Durable procedure shape
Effectful procedures expose four concrete phases:
```text
prepare immutable inputs
→ publish durable effect intent
→ perform the external effect
→ publish one durable outcome
```
The intent phase must remain visible and precede the external effect. Otherwise a crash after the effect but before its intent would leave no recoverable unknown-outcome marker. Each procedure uses concrete functions such as `prepareGeneration`, `publishGenerationIntent`, `performGeneration`, and `publishResponse`; there is no generic Procedure abstraction.
## Drive lifecycle
Delete the installer-owned model.
- Remove `installerSignal`.
- Remove `DriveAbandoned`.
- Remove `LostOwnership` and `lost_ownership` from procedure results.
- Remove exact-object ABA fencing and `commandDriveOwned`.
- Remove `finalizedOutcome` and planned external-finalization owner retention unless the Flue investigation establishes a concrete requirement.
- Keep `activeDrive` only as the lane's install/join slot.
- Installation transfers work to the lane. Every caller, including the installer, observes completion with its own invocation Context.
- A caller signal abort before installation installs nothing. After installation it only rejects that caller's observation.
- The Drive is removed after its pass settles or reaches a durable wait. No live pass is replaced in-process.
`requestAbort` retains the two-part gate ordering:
```text
beginAbort before cancellation mutation
commit cancel_requested
signalAbort after commit
```
This prevents a new effect from entering while the durable marker is being committed.
## Checks that remain
Do not remove validation of external or referenced content:
- required entry existence and role;
- pending payload kind;
- deferred handle identity;
- provider stream protocol ordering;
- response stop-reason invariants;
- UUIDv7 follower timestamp parsing;
- configured model/tool availability;
- tool-call source index and staged result identity;
- parallel tool call status and ready-prefix placement;
- progress/memo invocation identity;
- retry timestamp and deferred permit arithmetic.
These validate data or genuine child concurrency. They are not defensive revalidation of the operation state machine.
## Completed staged implementation plan
### Stage 1 — Canonical flat durable types
Files:
- `src/harness/session/types.ts`
- compile-only consumers in `src/harness/runtime/`, restore, conformance helpers, and focused tests
- `docs/harness.md`
- `docs/work-packages/05-direct-durable-drive.md`
Actions:
- Replace nested operation state declarations with the flat `at` union.
- Preserve every durable datum without compatibility aliases.
- Update pattern matching mechanically without changing behavior.
- Keep ToolBatch/ToolCall nested.
- Update normative documentation in the same change.
Exit: `npm run check`; existing focused runtime tests pass; no `phase.kind`, generation `status`, deferred `status`, or structural decision `status` remains in canonical operation state.
### Stage 2 — Add canonical transition operations
Files:
- `src/harness/runtime/lane.ts`
- `src/harness/runtime/types.ts`
- focused Lane tests
Actions:
- Add `continueOperation` and `settleOperation`, including the terminal-decision suffix.
- Pair every durable operation-state write with process-local projection publication in one implementation.
- Centralize ordinary cancellation diversion.
- Trust the dispatcher-established current leaf; perform no expected-state check.
- Keep procedure-specific writes and event builders visible at call sites.
Exit: focused tests prove durable state and `Lane.state` remain identical after every helper commit.
### Stage 3 — Convert starting, checkpoint, and assistant
Files:
- `runtime/drive/checkpoint.ts`
- `runtime/drive/generation.ts`
- `runtime/drive/recovery.ts`
- `runtime/progress.ts`
Actions:
- Remove repeated operation/null/kind/state checks and `same*` predicates.
- Convert ordinary progress to `continueOperation`.
- Convert assistant settlement to `settleOperation`.
- Keep only checkpoint inbox/finish races and progress-channel ownership checks.
- Fold assistant recovery into the effect-pending handler if that is smaller.
Exit: no control-state storage reads; no repeated assistant state verification; focused generation tests pass.
### Stage 4 — Convert deferred and tools
Files:
- `runtime/drive/deferred.ts`
- `runtime/drive/tools.ts`
- `runtime/progress.ts`
Actions:
- Share assistant/deferred response-entry, usage, and tool-plan construction where their semantics match.
- Preserve deferred permit and handle checks.
- Remove top-level operation-state revalidation.
- Preserve per-call tool status merging, completion-order staging, source-order placement, memo fencing, and progress fencing.
- Use one tool-batch procedure for live, recovery, and cancellation modes instead of separate ownership-result paths.
Exit: tool status checks exist only for genuine sibling concurrency; focused deferred/tool/progress tests pass.
### Stage 5 — Remove ownership-loss machinery
Files:
- `runtime/types.ts`
- `runtime/lane.ts`
- `execution/effect-gate.ts`
- all existing `runtime/drive/*.ts`
- associated focused tests
Actions:
- Remove `LostOwnership`, `commandDriveOwned`, exact Drive checks, `installerSignal`, `DriveAbandoned`, and `finalizedOutcome`.
- Make all drive callers observation peers.
- Add observation-only Context cancellation to Drive completion waiting.
- Keep `activeDrive` only for install/join arbitration.
- Remove ABA/replacement tests and replace them with install/join/observation-cancellation tests.
Exit: grep finds no ownership-loss or installer-abandonment vocabulary in production runtime.
### Stage 6 — Close and fault
Files:
- `runtime/harness.ts`
- `runtime/lane.ts`
- Drive observation helper
- lifecycle tests
Actions:
- Seal mutation admission and drain admitted mutations.
- Reject client observations on close/fault without replacing a Drive or changing durable operation state.
- Observe detached pass failures.
- Verify a late effect cannot commit after close.
- Decide separately whether to signal local effects for resource cleanup.
Exit: close and process loss leave the same durable restart point; no close path writes cancellation or synthetic settlement.
### Stage 7 — Finish WP05 on the simpler substrate
Files:
- `runtime/drive/structural.ts`
- `runtime/drive/reconcile.ts`
- `runtime/drive.ts`
- `runtime/lane.ts` public surfaces
Actions:
- Implement structural generation directly over flat leaves and the transition operations.
- Implement cancellation reconciliation as one flat-state switch.
- Implement the total drive switch as one `state.at` switch.
- Wire public claim/join/observation and convenience methods only after every leaf is total.
External finalization is excluded unless the Flue investigation identifies a concrete, current caller that cannot be expressed through close, explicit abort, recovery, or offline administration.
## Validation
After every code stage:
```bash
npm run check
```
Run each modified focused test file from the package root. Do not run the full Vitest suite directly. Public drive was enabled only after the final stage made every leaf total.
Final audit:
```bash
rg 'lost_ownership|LostOwnership|DriveAbandoned|commandDriveOwned|installerSignal|finalizedOutcome' packages/agent/src/harness
rg 'operationState\(|laneState\(|branchTip\(|laneConfig\(' packages/agent/src/harness/runtime/drive
```
The second audit may match write constructors, but no reader call may use those control addresses.

View file

@ -0,0 +1,251 @@
# Invocation Context and Telemetry Design Notes
> **Status:** Design input, not a normative contract. The context primitives and required trailing `Context` parameters have landed across the harness, sessions, execution capabilities, and hosted-harness adapters. Local propagation is scaffolding rather than proof of complete telemetry semantics: most runtime spans and cross-process trace propagation remain design or implementation work. Drive ownership is now harness-owned and request-ID RPC cancellation is implemented; neither supplies distributed trace parentage. Fold accepted final behavior into `harness.md`. `telemetry-schema.md` remains the generated reference for span names and attributes.
## Goal
`Session`, `Branch`, `AgentLane`, and `AgentHarness` receive invocation-scoped control data explicitly through a required trailing `Context` parameter. The same receiver may serve concurrent local callers or RPC clients, so it cannot retain a mutable or default caller context.
The invocation context must solve two related problems without `AsyncLocalStorage`:
1. preserve correct telemetry parentage through concurrent asynchronous work;
2. carry an `AbortSignal`, when one exists, that an RPC adapter can map to request cancellation.
This work must reuse `@earendil-works/pi-telemetry`. It must not introduce another span abstraction.
## Context model
The implemented public types are:
```ts
interface ContextKey<T> {
readonly token: symbol;
readonly valueType?: (value: T) => T;
}
interface Context {
readonly abortSignal: AbortSignal | undefined;
readonly telemetryContext: TelemetryContext;
value<T>(key: ContextKey<T>): T | undefined;
toString(): string;
}
```
`valueType` is a type-only marker. Runtime lookup uses the key's symbol token. `createContextKey<T>(description)` creates and freezes a key with a unique token.
A context is immutable. Derivation creates a parent-linked, copy-on-write layer. Helper arguments put the value first and parent context last:
```ts
const requestContext = withAbortSignal(requestSignal, parentContext);
const spanContext = withTelemetryContext(span, requestContext);
const tenantContext = withContextValue(tenantKey, tenantId, spanContext);
```
The implemented behavior is:
- `BACKGROUND_CONTEXT` and `TODO_CONTEXT` are distinct empty roots whose `abortSignal` is `undefined`;
- `telemetryContext` is always available and falls back to `NOOP_TELEMETRY_CONTEXT` when no telemetry value has been installed;
- `withAbortSignal(signal, context)` preserves the supplied signal when the parent has none and otherwise combines it with the parent signal using `AbortSignal.any()`;
- `withCancel(context)` returns an independently cancellable child context and a `cancel(reason?)` function; parent cancellation still reaches the child;
- typed values use symbol identity and immutable copy-on-write layers, and a newer value for the same key shadows its parent value;
- the built-in abort-signal and telemetry keys are private; callers use the named properties instead of retrieving those values by key;
- `toString()` is diagnostic and records the root plus each layered key description.
Context values are cross-cutting request metadata, not business dependencies. Suitable typed values include request IDs, authenticated principals, tenant IDs, and diagnostic metadata. Storage, models, tools, durable state, and business payloads do not belong in the context. Contexts, signals, telemetry objects, and backend-native span objects are never durable data.
## Receiver ownership
Shared receivers retain identity and durable/process state, not invocation context:
```text
AgentHarness receiver ── no caller context
AgentLane receiver ── no caller context
Session receiver ── no caller context
Branch receiver ── no caller context
```
Every invocation supplies its own context. This prevents concurrent callers from overwriting each other's telemetry parent or cancellation signal.
A process-local object representing one ongoing invocation may retain its derived context. Examples are an active drive task or an event subscription. This is different from storing a default context on the shared harness or session receiver.
`AgentHarnessOptions.telemetryContext` has been removed. A harness-level default cannot represent two concurrent callers with different parents.
## Existing typed telemetry remains authoritative
The design retains:
- `TelemetryContext` and `TelemetrySpan`;
- callback-owned span lifetime;
- `AI_TELEMETRY_SCHEMA` and `HARNESS_TELEMETRY_SCHEMA`;
- typed span names, start attributes, completion attributes, and events;
- `startAiSpan()`, `startHarnessSpan()`, and `createTypedSpanStarter()`;
- adapter conformance behavior.
`startAiSpan()` and `startHarnessSpan()` package span derivation by giving their callbacks both the typed span and a derived invocation context:
```ts
return startHarnessSpan(
"pi.harness.run",
attributes,
async (span, runContext) => {
return runDrive(runContext);
},
context,
);
```
The helpers delegate to `context.telemetryContext.startSpan()` and install the callback-owned span in a child context with `withTelemetryContext(span, context)`. Lower work must receive that child context rather than the parent invocation context.
Do not mutate a context to install an active span. Do not use a process-global or receiver-global current span.
## Concurrent parentage
Explicit propagation supports concurrent sibling calls:
```ts
await parent.telemetryContext.startSpan({ name: "caller" }, async (callerSpan) => {
const callerContext = withTelemetryContext(callerSpan, parent);
await Promise.all([
laneA.drive(optionsA, callerContext),
laneB.drive(optionsB, callerContext),
]);
});
```
Each invocation derives its own child context. Nested work receives the child belonging to that invocation. Correct parentage does not depend on promise scheduling or ambient state.
Tests must cross concurrent branches deliberately so accidental receiver-level context is visible. A sequential parent/child test is insufficient.
## Callbacks, hooks, and events
Host-local callbacks invoked as part of an operation receive the current invocation context in their declared trailing position:
```ts
handler(event, context);
tool.execute(toolCallId, params, onUpdate, toolContext, invocation, context);
mutation(mutator, context);
```
Current propagation preserves context through callbacks and gives `before_tool` and `after_tool` handlers a child context derived from `pi.harness.hook`. Extending that span behavior to every hook type remains work; handlers without an installed hook span currently receive the operation context directly.
Within the harness process, events preserve the context that caused each event, and buffered event watchers store `{ event, context }` rather than only `event`. Starting `pi.harness.event_handler` from that event context and passing its child context to each listener remains work. Event registration itself is host-local configuration and has no operation parent.
Session mutation callbacks and commits receive the same explicit invocation context. Starting `pi.session.write` from the committing invocation and passing its child context through the storage commit remains work.
## Drive execution and joiners
Several callers may call `drive()` for the same durable operation. Arbitration decides which call installs process-local execution and which calls join it. This is a core runtime concern, not an RPC concern; concurrent local callers have the same issue.
One active execution has one telemetry parent. It cannot be reparented when another caller joins.
```text
installer caller
└─ drive.execute
└─ provider/tool work
joiner caller
└─ drive.join
```
A joiner span describes that caller's wait. It carries at least lane name, durable operation ID, and a process-local execution ID. It ends with an outcome such as `settled`, `caller_cancelled`, `execution_stopped`, or `harness_closed`.
The joiner must not overwrite the active execution context. Correlate the two spans using operation/execution attributes. Telemetry span links would model this relationship better, but the current telemetry contract has no links. Adding links is an optional telemetry-package design question, not a reason to invent multiple parents.
Distributed traces permit an execution span to outlive the installer RPC span. Parent and child spans may overlap or settle in either order once the child has started.
## Invocation cancellation versus durable cancellation
An aborted invocation signal is process-local control. It does not mean that durable cancellation was requested.
```text
context.abortSignal is present and aborts
→ stop only that caller's observation; an installed lane-owned Drive continues
→ do not write cancel_requested
→ preserve the same durable operation state
```
An undefined `abortSignal`, as exposed by both empty roots, means that the invocation has no cancellation signal.
Only `requestAbort()`/`abort()` writes durable `cancel_requested` and permits durable aborted settlement.
The runtime must track the stop cause instead of interpreting every aborted provider response as durable cancellation:
```ts
type ExecutionStopCause =
| "no_drive_waiters"
| "invocation_cancelled"
| "harness_closed"
| "durable_cancel_requested";
```
Only `durable_cancel_requested` may normalize and commit a durable aborted outcome. An invocation/disconnect abort must not produce an assistant `stopReason: "aborted"` settlement while durable control remains `running`; that path would violate the durable state machine.
Drive ownership is resolved as **harness-owned**: once installed, execution survives caller cancellation/disconnect until durable settlement or wait, explicit durable cancellation, close, fault, or process loss. Joiner signals control only their own observations. Signals from unrelated joiners must never be combined with `AbortSignal.any()` and attached directly to shared execution. One canceled joiner cannot cancel every other caller.
## RPC trace propagation
Client and server spans can belong to one distributed trace:
```text
caller
└─ rpc.client
└─ rpc.server
└─ harness/session operation
```
The client does not serialize `TelemetryContext`. It injects a backend-neutral trace carrier from the `rpc.client` span. The server extracts that carrier into a fresh local `TelemetryContext` and starts `rpc.server` from it.
A transport-facing adapter boundary is required:
```ts
interface TelemetryPropagation {
inject(context: TelemetryContext): JsonValue | undefined;
extract(carrier: JsonValue | undefined): TelemetryContext;
}
```
A production implementation may use W3C `traceparent`/`tracestate`. The current telemetry package has no carrier injection/extraction API, so the accepted design must decide whether this adapter belongs in the telemetry package, RPC infrastructure, or a backend integration package. It must still reuse the existing `TelemetryContext` span contract.
RPC cancellation and telemetry propagation are independent control-plane channels:
- trace metadata reconstructs telemetry parentage;
- request ID plus cancel/disconnect messages controls the server request signal;
- neither channel appears in serialized method arguments.
## Interface migration scaffolding
Receiver methods now use a required trailing `Context`. Concrete implementations, calls, callback adapters, and object-literal façades have been migrated rather than relying only on interface assignability.
`TODO_CONTEXT` remains a temporary migration marker, not a semantic root. Current uses cluster at unresolved transport and worker boundaries that cannot yet reconstruct a caller context, notably Pi protocol request ingress and worker RPC ingress. `BACKGROUND_CONTEXT` means intentionally start without a caller.
Continue to inventory `TODO_CONTEXT` separately. Replace each transport-boundary use only when the boundary can construct a request-local cancellation context and telemetry parent; substituting `BACKGROUND_CONTEXT` would hide unfinished propagation. Compilation still does not prove telemetry or cancellation correctness.
## Required tests for the later handoff
Current tests cover immutable typed-value layering and shadowing, distinct empty roots, parent/child abort composition, sibling cancellation isolation, and tool-hook child parentage. Remaining handoff coverage includes:
- crossed concurrent telemetry branches on one shared receiver;
- every hook type, tools, event handlers, and session writes receive the intended child context;
- buffered events retain their emitting context under delayed delivery;
- no receiver-level telemetry default;
- pre-aborted invocation starts no external effect;
- installer and joiner cancellation isolation under lane-owned execution;
- invocation abort leaves the installed Drive and durable state unchanged;
- durable abort commits the durable aborted outcome;
- close and disconnect do not masquerade as durable cancellation;
- client → server trace reconstruction;
- event delivery reconstructs source trace metadata;
- missing/malformed trace carriers degrade to no-op/root telemetry without affecting business behavior.
## Resolved migration decisions
- Receiver methods use one required trailing `Context`.
- Shared Harness, AgentLane, Session, and Branch receivers retain no default invocation context.
- `Context`, `AbortSignal`, and `TelemetryContext` objects are never serialized across RPC boundaries.
## Open decisions before the telemetry handoff
- whether telemetry links are required for joiners;
- trace-carrier adapter ownership and shape;
- which context values, if any, may cross an RPC boundary;
- exact span names/outcome attributes for RPC calls and drive join waits.

View file

@ -0,0 +1,704 @@
# Tool durability — implementation handoff
This document specifies the durable tool-call lifecycle and the minimal harness-specific progress-checkpoint extension to the current tool API. It does not otherwise finalize the harness-native public tool interface; that interface must provide the capabilities required here without exposing raw session storage.
The design has two independent additions:
1. a durable `outcome_ready` state between external-effect settlement and source-ordered conversation placement;
2. opt-in durable replacement checkpoints for complete bounded `onUpdate` snapshots.
## Problem
Parallel tool effects finish in completion order, while tool-result entries must enter the conversation in assistant source order.
Without an intermediate durable state:
```text
calls: A, B, C
B finishes
C finishes
A is still running
process crashes
```
B and C exist only in process memory because A prevented source-ordered placement. Recovery treats them as unresolved and may replay or interrupt effects that already completed.
The solution separates two orders:
1. **outcome durability:** actual completion order;
2. **entry materialization:** assistant source order.
A complete finalized result becomes durable immediately in `pi.pending.entry`; the call becomes `outcome_ready`; placement happens later when every earlier source position is complete or ready.
## Goals
1. Never rerun a tool after its complete finalized outcome is durable.
2. Durably retain out-of-order parallel outcomes without violating transcript order.
3. Preserve the existing whole-tool `replay: "safe" | "never"` contract.
4. Support invocation-scoped durable memoization for Flue-style `step.do`.
5. Preserve tool-selected bounded progress checkpoints for reconnect and unsafe interruption recovery.
6. Fence late tool writes after outcome settlement, cancellation, or external finalization.
7. Keep final tree entries canonical, complete, bounded independently of progress snapshots, and immutable.
## Non-goals
- Exactly-once arbitrary external effects.
- A nested durable state machine for each `step.do` call.
- Inferring tool completion from progress output.
- Treating partial output as the canonical final result.
- Giving tools raw `Session`, `SessionMutator`, or bound storage-address access.
- Finalizing the public harness-native tool type in this document.
## Durable identities
Each call already reserves its result entry ID before execution. Use that as the stable public invocation identity:
```text
invocationId = resultEntryId
```
It is session-unique, survives safe replay, and is distinct from the provider's batch-local `toolCallId`, which a later assistant message may reuse.
The surrounding operation state continues to provide:
- `operationId`;
- `turnId`/generation step ID;
- `sourceIndex`;
- assistant entry ID;
- captured configuration and execution mode.
## Storage
### Existing bound values
```ts
operationToolArgs(operationId, turnId, sourceIndex)
// Effective validated arguments, persisted before effect admission.
pendingEntry(resultEntryId)
// Complete finalized ToolResultMessage while outcome_ready awaits placement.
```
### Invocation memos
Define the operation-owned address constructor in `session/values.ts`:
```ts
export const operationToolMemo = (
operationId: string,
invocationId: string,
memoName: string,
) => value<JsonValue>(
"pi.op.tool_memo",
`${operationId}:${invocationId}:${memoName}`,
);
```
`memoName` must be non-empty and contain no `:`. Names may use dots or slashes for application-local grouping. `setMemo(name, undefined)` deletes the exact bound value.
`scanValues(operationToolMemoPrefix(operationId))` permits defensive operation cleanup. `scanValues(operationToolMemoPrefix(operationId, invocationId))` permits atomic invocation cleanup when one outcome becomes ready. Core cleanup uses these owner-defined prefix constructors rather than repeating raw reserved namespace/key grammar.
### Partial tool output
The durable recovery value is the latest complete bounded progress snapshot selected by the tool. That is total current state, so use one bound value address:
```ts
export const pendingToolOutput = (
operationId: string,
invocationId: string,
) => value<AgentToolResult<unknown>>(
"pi.pending.tool_output",
`${operationId}:${invocationId}`,
);
```
This is auxiliary observation data. It never proves the effect succeeded or completed. The stored value has exactly the same content/details/usage shape as the live `partialResult`; recovery does not need a tool-specific progress codec.
The tool owns snapshot bounding, checkpoint cadence, and duplicate suppression. The harness owns synchronous enqueue, promise tracking, invocation fencing, and cleanup. The first API has no generic byte cap and does not truncate or reinterpret typed tool data; in-process tools are trusted to honor the bounded-snapshot contract. There is no tool-visible `flush()` method and no durable list of progress updates.
A crash may lose live updates newer than the latest committed checkpoint. JSONL physical growth is proportional to the size and frequency of distinct requested checkpoints until compaction; Memory and SQLite retain one current value. At 50 KiB every two seconds, JSONL's uncompacted worst case is approximately 15 MiB per ten minutes of continuously changing output.
## Tool update API
Keep the existing full-snapshot update callback and add harness-specific options:
```ts
export interface AgentHarnessToolUpdateOptions {
/** Request replacement of this invocation's durable recovery checkpoint. */
checkpoint?: true;
}
export type AgentHarnessToolUpdateCallback<TDetails> = (
partialResult: AgentToolResult<TDetails>,
options?: AgentHarnessToolUpdateOptions,
) => void;
```
`AgentHarnessTool` uses this callback instead of the legacy `AgentToolUpdateCallback`. The harness always supplies it, even without a live listener, because a tool may request persistence through it. The legacy `AgentTool` and old agent loop remain unchanged because they cannot honor durable checkpoints.
Every callback invocation remains an immediate live update. The callback is synchronous and returns `void`; `checkpoint: true` additionally requests persistence of that complete snapshot. It is not a durability acknowledgement. Internally, the harness retains the latest `events.emit(tool_update)` promise so existing listener delivery completes before `after_tool`; tools do not await or receive that promise.
Every `checkpoint:true` call synchronously enqueues one scalar replacement on the Session mutation line, attaches the ordinary harness-fault observer to that promise, and replaces the process-local `latestCheckpointWrite` reference. Writes themselves are neither dropped nor coalesced, and replacing the reference never leaves an earlier rejection unobserved:
- Session mutation FIFO preserves request order;
- each mutation verifies the same call remains `effect_pending`;
- completion of the latest promise implies completion of every earlier checkpoint write;
- tool-promise settlement stops accepting updates and awaits that latest promise before `after_tool`;
- a failed checkpoint commit faults the harness under the ordinary storage-fault rule.
A tool that requests checkpoints faster than storage can queue work in memory. Under the trusted-tool contract, cadence is the tool's responsibility. The built-in bash policy bounds this queue in ordinary use.
Tools should compare against their last requested checkpoint when duplicate suppression matters. Storage does not read and deep-compare the current scalar as part of every checkpoint.
### Bash policy
The built-in bash tool keeps its current 100 ms live update cadence. It requests a checkpoint at most once every two seconds and only when the complete bounded snapshot differs from its last requested checkpoint:
```ts
const BASH_UPDATE_THROTTLE_MS = 100;
const BASH_CHECKPOINT_INTERVAL_MS = 2_000;
```
The current `ShellCaptureProgress` already supplies a bounded snapshot: the last 2,000 lines or 50 KiB, plus truncation metadata and the overflow-file path. The initial empty update is live-only. Output volume never accelerates checkpoint frequency. A short tool may settle without writing any checkpoint because its complete final result commits instead.
## Tool-call state
Extend the call union with `outcome_ready`:
```ts
type ToolCall =
| {
status: "planned";
sourceIndex: number;
resultEntryId: string;
}
| {
status: "effect_pending";
sourceIndex: number;
resultEntryId: string;
replay: "never" | "safe";
}
| {
status: "outcome_ready";
sourceIndex: number;
resultEntryId: string;
terminate: boolean;
}
| {
status: "completed";
sourceIndex: number;
resultEntryId: string;
terminate: boolean;
};
```
`outcome_ready` means:
- execution, error normalization, and `after_tool` have finished, or the harness has constructed a final synthetic result;
- the complete final `ToolResultMessage` exists at `pendingEntry(resultEntryId)`;
- invocation memos and partial-output storage are gone;
- the tool must never execute again;
- the immutable result entry may not exist yet because an earlier call has not materialized.
The exact union may later carry small settlement metadata, but it must not duplicate the finalized result payload.
## State transitions
```text
planned
├─ real effect cleared → effect_pending
└─ immediate/synthetic → outcome_ready
effect_pending
├─ live effect settles → outcome_ready
├─ safe orphan replay settles→ outcome_ready
└─ unsafe orphan synthesis → outcome_ready
outcome_ready
└─ source position eligible → completed
```
An implementation may fuse `outcome_ready → completed` into the same transaction that finalizes an immediately placeable head call, but the semantic checks and tests must still cover durable `outcome_ready` for out-of-order calls. Prefer implementing the explicit two-transaction form first.
## Fresh execution
### Clearance and intent
Unchanged effect sandwich:
```text
planned
→ prepare arguments, run before_tool, validate replacements
→ TX[
set pi.op.tool_args,
set call = effect_pending(replay)
]
→ post-commit tool_start
→ admit tool execution
```
The invocation-scoped capability becomes active only for this durable `effect_pending` call.
### Partial output
Every `onUpdate(partialResult, options)` publishes the live update through the existing event/snapshot path. When `options.checkpoint === true`, the harness additionally requests a scalar replacement:
```text
TX[
setValue(pendingToolOutput(operationId, invocationId), partialResult)
]
```
The mutation verifies the same operation, turn, source position, and invocation remain `effect_pending`. It does not rewrite `pi.op.state`. A late checkpoint after settlement returns without committing.
The tool must checkpoint bounded complete snapshots, not growing unbounded values. Bash uses the same bounded `ShellCaptureProgress` snapshot it already sends live. Clients may render and locally retain live updates newer than the durable checkpoint, but those updates are explicitly process-local.
### Finalization to `outcome_ready`
When the tool promise settles:
1. synchronously stop accepting updates and expire the invocation capability;
2. await the latest tracked `tool_update` delivery and latest checkpoint-write promise; each implies completion of its preceding queue;
3. run `after_tool` when this is a real fresh or safely replayed result and cancellation did not prevent the hook;
4. construct the complete final `ToolResultMessage`;
5. commit the result as `outcome_ready`;
6. emit and await `tool_end` from the committed staging transition.
`setMemo()` returns a promise and tools must await it; `step.do` always does. An unawaited pre-return mutation is still enqueued before staging and is deleted by staging. Calls begun after capability expiry reject. No separate invocation-write drain exists.
Transaction:
```text
TX[
setValue(
pendingEntry(resultEntryId),
{ type: "message", payload: finalizedToolResultMessage },
),
deleteValue(pendingToolOutput(operationId, invocationId)),
deleteValue(memo.address) for every memo returned before commit by
scanValues(operationToolMemoPrefix(operationId, invocationId)),
setValue(operationState(operationId), call = outcome_ready(terminate))
]
```
The transaction is the linearization point after which the invocation can never replay. Its post-commit `tool_end` is therefore durable evidence that the finalized outcome is ready; it is no longer a pre-commit effect observation.
The staged message contains the final:
- text/image content;
- provider tool-call ID and tool name;
- details;
- `isError`;
- usage snapshot, when reported;
- added tool names;
- timestamp.
`terminate` remains orchestration state because it controls the batch continuation and is copied to the immutable entry's `terminate` field at placement.
## Source-ordered materialization
After any call becomes `outcome_ready`, find the contiguous ready prefix beginning at the first non-completed source position.
Example:
```text
[completed, outcome_ready, outcome_ready, effect_pending]
└──────── ready prefix ────────┘
```
Before the placement transaction, emit and await each finalized result's `message_start` and `message_end` in source order. Materialize the prefix in one transaction when practical, then emit `entry_added` and reported usage events in the same source order:
```text
TX[
insert result entry i from pendingEntry(i),
deleteValue(pendingEntry(i)),
insert tool usage row i if reported,
insert result entry i+1 with parent = result i,
deleteValue(pendingEntry(i+1)),
insert tool usage row i+1 if reported,
setValue(branchTip(lane), newest result),
setValue(operationState(operationId),
calls i..i+1 = completed and, when complete, next checkpoint)
]
```
Each inserted entry uses its already-reserved `resultEntryId`. Writes construct the parent chain in source order inside the transaction.
Tool-reported usage remains durable in the staged message until placement. The initial implementation writes its ledger row atomically with entry materialization, matching the current entry/usage ordering and avoiding a ledger row that references an entry not yet present. No usage ID reservation is needed because a failed placement transaction writes neither row nor completed state.
When the final call materializes, the same transaction calls `scanValues(operationToolArgsPrefix(operationId, turnId))` and deletes every returned address, and transitions to the correct checkpoint:
- every result terminates → `may_finish`, no final assistant required;
- otherwise → `need_assistant(false)`.
## Parallel execution
Outcome staging follows actual completion order. Entry materialization follows source order.
```text
A, B, C start
B finishes → B outcome_ready
C finishes → C outcome_ready
A finishes → A outcome_ready
materialize A, B, C
```
Crash after B and C stage:
```text
A effect_pending
B outcome_ready
C outcome_ready
```
Recovery applies unknown-outcome policy only to A. B and C require no tool registration or hook execution to become entries.
The durable invariant changes from “completed calls form a source-ordered prefix” to:
- completed calls form a source-ordered prefix;
- after that prefix, parallel calls may be `planned`, `effect_pending`, or `outcome_ready` in any mixture;
- only source-ordered materialization extends the completed prefix.
Sequential execution constructs at most one non-planned call after the completed prefix. Committed call state is trusted on restore; the owning procedure enforces this shape while creating and consuming transitions rather than through a broad restore audit.
## Unsafe recovery with partial output
For orphaned `effect_pending` with `replay: "never"`:
1. read `pendingToolOutput(operationId, invocationId)` when present;
2. preserve its bounded content and serializable details;
3. append a mandatory human-readable interruption marker;
4. construct a harness-owned `ToolResultMessage` with `isError: true`;
5. commit it as `outcome_ready` and clean invocation state.
The marker must state that output is partial and the external outcome is unknown. `isError: true` describes the result delivered to the model; it does not assert that the external effect failed.
Example final text suffix:
```text
[Tool execution was interrupted. The preceding output is the latest durable progress snapshot; newer live output may be missing, and the external outcome is unknown.]
```
Rules:
- do not run `after_tool` for this synthetic result;
- preserve checkpoint `usage` when present, but ignore checkpoint `terminate` because progress never has final-result authority;
- set `terminate: false` and add no tools;
- an absent checkpoint value is also valid and yields only the interruption result;
- never infer completion from an apparent success line in partial output;
- cleanup of partial output and invocation memos is atomic with staging the synthetic result.
## Safe recovery
For orphaned `effect_pending` where both the stored and current declarations are `replay: "safe"`:
1. retain invocation memos;
2. atomically delete `pendingToolOutput(operationId, invocationId)`;
3. emit/reset process-local progress observation as needed;
4. rerun the tool with persisted arguments and the same `invocationId`;
5. completed `step.do` calls return their memoized values;
6. new partial output reconstructs a clean progress stream;
7. finalization follows the ordinary `outcome_ready` path.
Deleting old progress prevents duplicate chunks when replayed code emits progress again. A crash after the delete but before replay admission remains `effect_pending`; the next recovery repeats the same safe procedure.
If the current tool declaration is missing or no longer safe, use unsafe interruption recovery rather than suspending.
## Invocation memos
The harness-native tool call receives a purpose-built invocation capability conceptually equivalent to:
```ts
interface AgentHarnessToolInvocation {
readonly invocationId: string;
readonly operationId: string;
readonly turnId: string;
getMemo(key: string): Promise<JsonValue | undefined>;
setMemo(key: string, value: JsonValue | undefined): Promise<void>;
}
```
Each operation:
1. validates the memo name and checks process-local capability expiry;
2. synchronously enqueues work on the Session mutation line before returning its promise;
3. verifies the operation, turn, source position, and invocation are still the same `effect_pending` call when that job executes;
4. constructs and reads or writes only `operationToolMemo(operationId, invocationId, name)`;
5. rejects after capability expiry or durable ownership loss.
The durable check matters for authorized external finalization. In ordinary execution, a memo mutation initiated before the tool returns is FIFO-ordered before outcome staging; one initiated afterward fails the expired-capability check.
A late zombie callback can neither recreate memos after `outcome_ready` nor write into a later operation.
Memos are immediate durable replay state, not application-visible settlement state. They survive close/crash while the call remains `effect_pending` and are deleted when any real or synthetic outcome becomes ready.
Terminal cleanup defensively scans and deletes the operation-owned families:
```ts
scanValues(operationToolMemoPrefix(operationId))
scanValues(pendingToolOutputPrefix(operationId))
```
in addition to other operation-owned addresses. Each returned `StoredValue` supplies its exact bound address for `deleteValue`; no later operation receives a raw key.
## Flue-style `step.do`
Build `step.do` over invocation memos; it does not need its own harness state union:
```ts
interface ToolSteps {
do<T extends JsonValue>(
name: string,
effect: () => T | Promise<T>,
): Promise<T>;
}
```
Algorithm:
```text
validate deterministic unique name
→ getMemo("step/" + name)
→ present: return stored value
→ absent: run effect
→ setMemo("step/" + name, value)
→ await durability
→ return value
```
Crash behavior:
```text
before/during effect → effect may run on replay
effect returned, memo not committed → effect may run on replay
memo committed → replay returns memo
step A memoized, step B interrupted → rerun tool; A skips, B runs
```
This is exactly-once recorded and at-least-once executed. It does not make arbitrary external effects exactly once. An application may derive a stable external idempotency key from `(invocationId, stepName)` when the external API supports one.
Errors are not memoized. A thrown effect either contributes to the current tool result or runs again after safe whole-tool recovery.
Do not add per-step `replay: "never"` in this slice. Supporting it correctly requires a nested `planned → effect_pending → completed` state and an explicit unknown-outcome policy. Whole-tool replay policy is sufficient for the Flue use case already discussed.
Within one live execution, calling the same step name twice is an invariant error. Names must be deterministic across safe replay.
## Application persistent state
Flue-style application state is distinct from invocation memos.
Invocation memo:
```text
step completed → memo becomes visible immediately
```
Application state:
```text
tool stages state change
→ crash before outcome_ready: state must not appear committed
→ outcome_ready: state and finalized result become visible together
```
Do not implement application state by calling `Session.setValue()` directly during execution.
Two valid implementation stages:
1. Flue keeps its state externally and atomically stores application state plus a full result memo keyed by `invocationId`.
2. A later harness API accepts staged application value writes and promotes them in the `outcome_ready` transaction.
The second option is required only if Flue's `usePersistentState` moves into harness-owned session values. Its public typing and conflict semantics remain an open design item for the harness-native tool discussion.
## Cancellation
Cancellation reconciliation never replays a restored tool.
- `planned` calls receive a synthetic aborted result and become `outcome_ready`;
- a live started call may finalize its real local result under cancelled control, then become `outcome_ready` with `terminate: false`;
- restored `effect_pending` calls use an interrupted synthetic result, optionally including partial output, regardless of safe replay declaration;
- existing `outcome_ready` calls are preserved and materialized in source order;
- invocation memos and partial output are deleted with each staged cancellation outcome;
- no `before_tool` or `after_tool` starts during restored synthetic reconciliation.
The aborted terminal transaction runs only after every call outcome has materialized and accepted deferred writes have drained under the existing cancellation rules.
## Close and external finalization
Close is still a controlled crash:
- memo/checkpoint mutations already enqueued under the admission barrier may finish;
- live output newer than the latest committed tool-requested checkpoint may be lost;
- no synthetic outcome or cancellation marker is written;
- durable state remains at `effect_pending` or `outcome_ready`.
External finalization deletes operation-owned arguments, invocation memos, partial output, staged pending outcomes, and other pending entries in its terminal transaction. A live task that later tries to stage an outcome fails the ownership fence and stops through `OperationEnded`.
## Restore and consumption-time reads
Base restore constructs the trusted lane/operation projection from required owner values. It does not hydrate or semantically audit tool arguments, invocation memos, progress checkpoints, staged outcomes, completed entries, completed-prefix shape, or captured execution-mode relationships.
The procedure responsible for the current typed state performs only its exact consumption-time reads:
### `planned`
Clearance needs no auxiliary restore read. It prepares the call and writes arguments before effect admission.
### `effect_pending`
Activation reads `operationToolArgs(operationId, turnId, sourceIndex)` and optionally reads `pendingToolOutput(operationId, invocationId)`. Missing required arguments are an invariant defect at consumption. Invocation memos are read only through the scoped capability. Safe replay, unsafe interruption, and snapshots use no broad prefix scan.
### `outcome_ready`
Materialization reads `pendingEntry(resultEntryId)`. Its absence or wrong trusted message relationship is an invariant defect when materialization consumes it. No tool identity or effect recovery is needed.
### `completed`
Ordinary dispatch performs no restore-time entry audit. Context/tree reads later consume the immutable entry through their normal typed paths.
Every live mutation still verifies current operation, turn, source position, invocation, and status on the Session line. Those checks fence concurrent settlement, cancellation, and external finalization; they are not historical restore validation. Terminal prefix cleanup remains defensive and does not make orphan scans part of restore.
## Snapshots and reconnect
A reconnecting client may see:
- live process-local progress newer than the latest durable checkpoint before disconnect;
- after process replacement, only the latest committed bounded checkpoint;
- `outcome_ready` calls as settled rows in `runningTools` until source-ordered materialization;
- completed calls in the transcript.
`LaneSnapshot.operation.runningTools` is a discriminated union. An effect-pending tool has `status: "running"` and an optional `result` containing the latest complete progress snapshot, falling back to the durable checkpoint after reopen. An outcome-ready call has `status: "settled"`, its required complete final `result`, and `isError`; it remains there until its immutable result entry's `entry_added` removes the row and places the same presentation in the transcript. Planned and completed calls are omitted.
## Events and hooks
- `tool_start` begins public processing presentation for a fresh call; it is emitted from the commit that establishes effect intent or a synthetic staged outcome and does not by itself prove an external effect started. It carries effective arguments for an intended effect and source arguments for an immediate synthetic result.
- live progress events and durable progress checkpoints do not prove completion.
- The harness awaits the latest `tool_update` delivery before `after_tool`, preserving the existing listener ordering without making `onUpdate` async.
- `tool_end` carries the complete finalized result after its `outcome_ready` staging commit, in completion order. It is durable settlement evidence and does not repeat the arguments from `tool_start`.
- for a fresh blocked, invalid, truncated, or planned-cancellation synthetic outcome, the staging commit emits `tool_start` followed by `tool_end`; these paths still run no tool effect or post-effect hook. Cancellation after effect intent uses the earlier intent-bound start and a staging-bound end.
- an unsafe restored effect is already represented as running by the initial snapshot and may emit only a recovery-tagged `tool_end` when interruption synthesis stages.
- message lifecycle and `entry_added` occur when the staged result materializes, not when it first becomes `outcome_ready`; `entry_added` removes only that settled row.
- passive listeners cannot mutate invocation state reentrantly.
Instrumented-storage tests assert `intent commit → tool_start → tool_update* → outcome staging → tool_end → source-ordered placement` for execution and `outcome staging → tool_start → tool_end → source-ordered placement` for fresh synthetic results. Historical events are not replayed; a safely replayed execution emits recovery `tool_start` from its checkpoint-clear commit and `tool_end` from outcome staging.
## Races
| Race | Required result |
|---|---|
| checkpoint vs tool settlement | every accepted checkpoint was enqueued first; settlement awaits the latest promise, then staging deletes the checkpoint value; a late update is ignored |
| memo write vs `outcome_ready` | an awaited or pre-return-enqueued write precedes staging and is then deleted; a post-return call rejects; external finalization first causes the durable ownership check to reject |
| B outcome vs earlier A settlement | B stages independently; placement waits for A |
| crash after outcome staging | tool never replays; pending result later materializes |
| crash during source-prefix placement | transaction exposes either none or all of that placement prefix |
| safe replay vs old partial output | the old bound checkpoint value is deleted before replay emits new progress |
| cancellation vs real settlement | Session mutation order chooses real cancelled-control result or synthetic reconciliation; at most one outcome stages |
| terminal finalization vs late result | terminal ownership wins or outcome stages first; late task never recreates operation data |
| external finalization vs memo/checkpoint mutation | mutation first is removed by terminal cleanup; finalization first makes the mutation's durable ownership check reject |
## Invariants
1. `invocationId` equals the reserved result entry ID and is stable across safe replay.
2. A call in `outcome_ready` or `completed` never executes again.
3. Every `outcome_ready` call has exactly one complete matching `pi.pending.entry` value.
4. Completed calls form a source-ordered prefix.
5. Only source-ordered materialization extends that prefix.
6. Parallel calls after the completed prefix may mix planned, effect-pending, and outcome-ready states.
7. Invocation memos exist only while their call is `effect_pending`.
8. Partial output is auxiliary and never establishes effect completion.
9. Unsafe synthetic results explicitly state that captured output is incomplete and the external outcome is unknown.
10. Staging an outcome atomically deletes its invocation memos and partial output.
11. Materialization atomically inserts the immutable entry and deletes its staged pending value.
12. A late invocation capability cannot write after outcome settlement or operation loss.
13. `step.do` values are memoized only after their memo write commits; effects remain at-least-once.
14. Operation terminal cleanup leaves no tool args, invocation memos, partial output, or staged outcomes.
## Required tests
### State and restore
- every trusted planned/effect-pending/outcome-ready/completed projection restores without auxiliary reads;
- base restore does not audit completed-prefix or execution-mode relationships;
- effect-pending consumption reads exact required arguments and optional bounded checkpoint;
- outcome-ready consumption reads the exact staged result;
- missing required arguments or staged result fails at the consuming procedure, not base restore;
- invocation memos and partial output are absent after staging;
- snapshot/activation hydrate only the exact bounded checkpoint value when needed.
### Parallel ordering
- B and C stage while A remains pending;
- crash/reopen proves B and C never replay;
- A recovery followed by one source-ordered A/B/C placement transaction;
- mixed `[completed, outcome_ready, planned, effect_pending, outcome_ready]` state;
- immediate synthetic outcomes stage out of order;
- all-terminating batch transitions correctly after ordered placement.
### Replay and interruption
- safe replay uses persisted arguments and the same invocation ID;
- safe replay preserves step memos but clears old partial output;
- current declaration downgrade from safe to never interrupts;
- unsafe recovery with no checkpoint and with a complete bounded checkpoint;
- synthetic result is error/incomplete/unknown and never runs `after_tool`;
- cancellation never safely replays a restored call.
### Invocation memos and `step.do`
- set/get/delete memo with invocation-address scoping;
- completed step skips effect after reopen;
- crash before memo commit reruns effect;
- crash after memo commit returns memo;
- several completed steps followed by one interrupted step;
- duplicate live step names reject;
- memo write racing outcome staging;
- capability write after expiry, cancellation outcome staging, and external finalization rejects;
- terminal cleanup removes crash-leaked memos.
### Partial output
- ordinary updates stay live-only;
- `checkpoint: true` writes the complete bounded snapshot;
- tool-selected checkpoint cadence and duplicate suppression;
- bash emits live snapshots at 100 ms and requests distinct checkpoints at most every two seconds;
- every selected checkpoint enqueues one write, and awaiting the latest promise at tool settlement implies all earlier writes completed;
- a checkpoint after outcome staging cannot recreate the address's value;
- Memory, JSONL, and SQLite restore the same checkpoint value;
- JSONL growth follows checkpoint cadence rather than raw bash output volume;
- terminal compaction reclaims superseded/deleted checkpoint snapshots according to the existing dead-byte policy.
### Atomicity and instrumentation
- exact intent, synchronous update acceptance, asynchronous update-delivery, `after_tool`, outcome-ready staging, post-commit `tool_end`, source-ordered message lifecycle, and materialization order;
- outcome staging is atomic with memo/output cleanup;
- materialization is atomic with pending deletion, usage, tip, and state;
- crash at every boundary;
- no effect starts before intent;
- no effect or hook starts from `outcome_ready`;
- terminal transaction removes every operation-owned tool value.
## Implementation map
Expected runtime areas:
- operation-state types in `packages/agent/src/harness/session/types.ts`;
- trusted restore projection and exact consumption-time address reads;
- tool-batch procedure and source-ordered materialization;
- terminal cleanup and cancellation reconciliation;
- harness-specific update options, snapshots, and events;
- invocation-scoped capability implementation on the Session mutation line;
- backend conformance through the bound value/list APIs;
- instrumented-storage transaction assertions.
Concrete built-in address constructors in `session/values.ts`:
```text
operationToolMemo(operationId, invocationId, name) → value("pi.op.tool_memo", ...)
pendingToolOutput(operationId, invocationId) → value("pi.pending.tool_output", ...)
pendingEntry(resultEntryId) → value("pi.pending.entry", ...)
```
Implement `outcome_ready` and invocation memos before progress checkpoints. The state solves incorrect parallel replay by itself; checkpoints improve reconnect observation and unsafe interruption diagnostics without becoming completion authority.

View file

@ -0,0 +1,735 @@
# Typed values and lists
This document specifies the mutable storage primitive used by Session, the harness, and applications.
The public abstraction is a **bound typed address**:
- `value<T>(namespace, key?)` names one replaceable durable value;
- `list<T>(namespace, key?)` names one append-only durable list whose elements have type `T`.
The namespace/key pair is bound once when the address is constructed. Every later operation receives only that address. Application code therefore writes:
```ts
const state = value<ApplicationState>("my-app.state");
const events = list<ApplicationEvent>("my-app.events");
await session.getValue(state, context);
await session.setValue(state, nextState, context);
await session.readList(events, { limit: 100 }, context);
await session.appendList(events, event, context);
```
It does **not** repeatedly pass a second unexplained key:
```ts
// Not the API.
await session.readList(events, "another-key", { limit: 100 }, context);
```
When an application genuinely has keyed instances, it constructs the address for that instance:
```ts
const workspaceEvents = (workspaceId: string) =>
list<ApplicationEvent>("my-app.events", workspaceId);
await session.readList(workspaceEvents("pi"), { limit: 100 }, context);
```
Storage may physically index the address as `(kind, namespace, key)`, but that representation does not leak into each read or write call. Storage, Session, harness code, and applications use the same address vocabulary. There is no global value-type map, dynamic registry, token catalog, or separate application-state storage mechanism.
## Goals
1. Give one exact durable address one compile-time value type.
2. Let applications define scalar values and lists without declaration merging or editing a core type map.
3. Use the same typed addresses and operation names from Storage through Session.
4. Preserve current scalar replacement semantics.
5. Append one list element without reading or rewriting existing elements.
6. Give every list element its own existing session-global transaction `seq`, used for ordering and pagination.
7. Commit values and list elements atomically with entries and usage.
8. Produce identical logical behavior on Memory, JSONL, and SQLite.
9. Keep list reads bounded and explicit.
10. Keep scalar operation state authoritative; auxiliary lists never select a recovery state.
## Non-goals
This slice does not define:
- assistant-frame contents or reduction semantics;
- tool-progress semantics;
- runtime validation of trusted in-process values;
- per-element or per-list byte limits;
- list truncation or per-element deletion;
- a generic event log, journal, stream-resumption protocol, or operation reducer;
- globally registering address objects;
- exposing raw Session or transaction access to tools.
Consumers own address construction, content limits, cleanup points, fork policy, migration policy, and consumption-time hydration. `assistant-durability.md` defines the first list consumer.
## Bound address model
```ts
declare const storedValueType: unique symbol;
interface StoredAddressBase {
/** Stable persisted grouping name. */
readonly namespace: string;
/** Exact member inside that grouping. Empty is legal. */
readonly key: string;
readonly kind: "value" | "list";
}
export interface Value<T> extends StoredAddressBase {
readonly kind: "value";
/** Compile-time only and invariant in T. */
readonly [storedValueType]?: (value: T) => T;
}
export interface ValueList<T> extends StoredAddressBase {
readonly kind: "list";
/** T is one element, not the whole list. */
readonly [storedValueType]?: (value: T) => T;
}
export function value<T>(namespace: string, key = ""): Value<T> {
validateAddress(namespace, key);
return Object.freeze({ namespace, key, kind: "value" });
}
export function list<T>(namespace: string, key = ""): ValueList<T> {
validateAddress(namespace, key);
return Object.freeze({ namespace, key, kind: "list" });
}
```
The phantom function makes `T` invariant: an address for one type cannot silently widen to another. It has no runtime field.
Rules:
- `namespace` must be non-empty;
- namespace `pi` and every `pi.*` namespace are reserved for built-ins by contract;
- applications that construct a reserved address are defective trusted in-process code; no runtime privilege split, registry, or catalog exists;
- neither component may contain the Memory backend's internal separator (`\u0000`);
- an empty key is valid and is the natural address for one application-wide value or list;
- object identity has no durable meaning;
- separately constructed addresses with the same `(kind, namespace, key)` identify the same durable location;
- constructing the same durable location with incompatible TypeScript types is a trusted-programming defect;
- scalar and list addresses may not share the same `(namespace, key)` in one storage version; violating this is a trusted-programming defect and storage performs no cross-kind collision check;
- changing an address's namespace, key, kind, or incompatible value shape requires migration.
The two components remain separate rather than concatenated. Dynamic application keys and operation IDs therefore require no escaping convention beyond the storage separator rule.
### Exact addresses, not families
An address names one value or one list. Internal code uses small constructors when it has dynamic keys:
```ts
export const branchTip = (lane: string) =>
value<string | null>("pi.branch.tip", lane);
export const operationState = (operationId: string) =>
value<OperationState>("pi.op.state", operationId);
export const operationToolArgs = (
operationId: string,
stepId: string,
sourceIndex: number,
) => value<Record<string, JsonValue>>(
"pi.op.tool_args",
`${operationId}:${stepId}:${sourceIndex}`,
);
export const pendingAssistantFrames = (
operationId: string,
responseEntryId: string,
) => list<AssistantMessageFrame>(
"pi.pending.assistant_frame",
`${operationId}:${responseEntryId}`,
);
```
This encapsulates each key grammar at its owner. Call sites receive an already-bound typed address:
```ts
await reader.getValue(operationState(operationId));
await reader.readList(pendingAssistantFrames(operationId, responseEntryId), options);
```
### No global value map
Delete the existing global namespace-to-type maps:
```ts
interface RegisterValues { /* delete */ }
interface ListRegisterValues { /* delete */ }
type RegisterNamespace = keyof RegisterValues; // delete
```
A type belongs to an address constructor instead:
```ts
export const applicationState = value<MyApplicationState>("my-app.state");
export const applicationEvents = list<MyApplicationEvent>("my-app.events");
```
Applications should use a stable, collision-resistant namespace prefix. Namespace `pi` and the complete `pi.*` prefix are reserved for built-ins by contract; similar-looking names such as `pi2` remain legal. The same `value()` and `list()` constructors serve core and application code. Tests assert that every built-in address uses its reserved prefix. There is no runtime privilege split, registry, or catalog.
## Built-in addresses
Built-in constructors live together in `packages/agent/src/harness/session/values.ts` and are imported directly by consumers. Representative definitions:
```ts
export const branchTip = (lane: string) =>
value<string | null>("pi.branch.tip", lane);
export const laneConfig = (lane: string) =>
value<LaneConfiguration>("pi.lane.config", lane);
export const laneState = (lane: string) =>
value<LaneState>("pi.lane.state", lane);
export const operationResult = (operationId: string) =>
value<OperationResultRecord>("pi.result", operationId);
/** Used only by scanValues() to enumerate Branch names. */
export const branchTipInventoryPrefix = () =>
value<string | null>("pi.branch.tip");
export const operationMeta = (operationId: string) =>
value<OperationMeta>("pi.op.meta", operationId);
export const operationState = (operationId: string) =>
value<OperationState>("pi.op.state", operationId);
export const operationToolArgs = (operationId: string, stepId: string, sourceIndex: number) =>
value<Record<string, JsonValue>>(
"pi.op.tool_args",
`${operationId}:${stepId}:${sourceIndex}`,
);
export const operationToolMemo = (operationId: string, invocationId: string, name: string) =>
value<JsonValue>("pi.op.tool_memo", `${operationId}:${invocationId}:${name}`);
export const operationPreparation = (operationId: string, taskId: string) =>
value<DurableStructuralPreparation>(
"pi.op.preparation",
`${operationId}:${taskId}`,
);
/** Prefix addresses are exported only for namespace-scoped scanValues(). */
export const operationToolArgsPrefix = (operationId: string, stepId?: string) =>
value<Record<string, JsonValue>>(
"pi.op.tool_args",
stepId === undefined ? `${operationId}:` : `${operationId}:${stepId}:`,
);
export const operationToolMemoPrefix = (operationId: string, invocationId?: string) =>
value<JsonValue>(
"pi.op.tool_memo",
invocationId === undefined ? `${operationId}:` : `${operationId}:${invocationId}:`,
);
export const operationPreparationPrefix = (operationId: string) =>
value<DurableStructuralPreparation>("pi.op.preparation", `${operationId}:`);
export const pendingEntry = (entryId: string) =>
value<PendingEntry>("pi.pending.entry", entryId);
export const pendingToolOutput = (operationId: string, invocationId: string) =>
value<AgentToolResult<unknown>>(
"pi.pending.tool_output",
`${operationId}:${invocationId}`,
);
export const pendingAssistantFrames = (operationId: string, responseEntryId: string) =>
list<AssistantMessageFrame>(
"pi.pending.assistant_frame",
`${operationId}:${responseEntryId}`,
);
export const pendingToolOutputPrefix = (operationId: string) =>
value<AgentToolResult<unknown>>("pi.pending.tool_output", `${operationId}:`);
export const sessionName = value<string>("pi.session.name");
export const entryLabel = (entryId: string) => value<string>("pi.entry.label", entryId);
```
`OperationMeta` is immutable acceptance metadata stored at `pi.op.meta`. The process-local `Operation` projection is `{ meta: OperationMeta, state: OperationState }`, assembled from the separate metadata and state values; it is never stored at one address.
The five exported scan-prefix constructors are `branchTipInventoryPrefix`, `operationToolArgsPrefix`, `operationToolMemoPrefix`, `operationPreparationPrefix`, and `pendingToolOutputPrefix`. Their addresses are consumed only by `scanValues()`.
Applications define their own `value()` and `list()` addresses directly; there is no built-in custom application-state namespace or custom-state API. `AgentHarnessToolInvocation.getMemo()` and `setMemo()` are invocation-fenced capabilities over `operationToolMemo(...)`, not raw Session access. Invocation memos remain operation-owned and are deleted when their tool outcome becomes durable.
Tests assert that built-in constructors produce the documented kind, namespace, and key grammar. Because constructors may be dynamic, there is no runtime catalog that tries to enumerate every possible address.
## Shared read API
Storage, Session, SessionReader, and SessionMutator use the same read signatures:
```ts
export interface StoredValue<T> {
address: Value<T>;
value: T;
seq: number;
}
export interface ListElement<T> {
/** Global transaction-write sequence assigned by storage. */
seq: number;
value: T;
}
export interface ListCursor {
seq: number;
}
export interface ListReadOptions {
/** Exclusive cursor. */
cursor?: ListCursor;
/** Default: asc. */
order?: "asc" | "desc";
/** Query-page size. Default: 1,000. Values above 10,000 clamp to 10,000. */
limit?: number;
}
interface ValueReader {
getValue<T>(address: Value<T>): Promise<StoredValue<T> | undefined>;
/** Internal bounded-prefix operation. The address key is interpreted as a prefix. */
scanValues<T>(prefix: Value<T>): Promise<StoredValue<T>[]>;
readList<T>(
address: ValueList<T>,
options?: ListReadOptions,
): Promise<ListElement<T>[]>;
}
```
`scanValues(prefixAddress)` scans scalar addresses with exactly that namespace and keys beginning with the bound key, returning them in key-ascending order. Core call sites use only the exported prefix constructors above so raw namespace/key grammar stays in `session/values.ts`. Prefix addresses are passed only to `scanValues()`, never to exact get/set/delete operations. There is no unrestricted cross-namespace dump. Ordinary application reads use exact addresses.
`Session` exposes direct one-transition writes using the same addresses:
```ts
interface Session extends ValueReader {
setValue<T>(
address: Value<T>,
next: NoInfer<T>,
context: Context,
): Promise<void>;
deleteValue<T>(address: Value<T>, context: Context): Promise<void>;
appendList<T>(
address: ValueList<T>,
element: NoInfer<T>,
context: Context,
): Promise<void>;
deleteList<T>(address: ValueList<T>, context: Context): Promise<void>;
}
```
Purpose-specific helpers such as `getName()`, `setName()`, `getLabel()`, and `setLabel()` may remain thin wrappers over built-in addresses. Applications define and use their own scalar/list addresses directly.
`SessionMutator` remains a read capability plus one atomic `commit(writes)`. It does not expose direct `setValue()`/`appendList()` methods that would consume its only commit separately; callers construct a typed write array and commit it together.
## Typed transaction writes
Writes are constructed through typed helpers. Entry and usage constructors hide their storage discriminants; value/list erasure happens only after the helper has checked the address/value type relationship:
```ts
interface EntryWrite {
kind: "entry";
entry: NewEntry;
}
interface UsageWrite {
kind: "usage";
row: Omit<UsageRow, "seq">;
}
interface ValueSetWrite {
kind: "value";
op: "set";
namespace: string;
key: string;
value: unknown;
}
interface ValueDeleteWrite {
kind: "value";
op: "delete";
namespace: string;
key: string;
}
interface ListAppendWrite {
kind: "list";
op: "append";
namespace: string;
key: string;
value: unknown;
}
interface ListDeleteWrite {
kind: "list";
op: "delete";
namespace: string;
key: string;
}
export function insertEntry(entry: NewEntry): EntryWrite;
export function insertUsage(row: Omit<UsageRow, "seq">): UsageWrite;
export function setValue<T>(address: Value<T>, next: NoInfer<T>): ValueSetWrite;
export function deleteValue<T>(address: Value<T>): ValueDeleteWrite;
export function appendList<T>(address: ValueList<T>, element: NoInfer<T>): ListAppendWrite;
export function deleteList<T>(address: ValueList<T>): ListDeleteWrite;
```
`NoInfer<T>` makes the address authoritative. TypeScript must not infer a wider `T` from an incompatible write value.
`Write` includes all six helper return types. One transaction may mix every write kind atomically. Harness and application code use the helpers rather than manually constructing storage write shapes.
The direct Session methods and transaction helpers intentionally use the same operation names. One performs and commits a single Session mutation; the other constructs a write for an explicitly composed transaction.
## Scalar semantics
For one `Value<T>` address:
- `setValue` replaces the current value;
- `deleteValue` removes it;
- deleting an absent value is a no-op;
- set after delete recreates it;
- there is no retained value history;
- the current value records the `seq` of its latest set;
- a failed transaction exposes neither the scalar write nor any sibling write.
## List semantics
One append write carries one immutable element. A transaction appending several elements contains several append writes. Every write receives its existing globally increasing transaction sequence:
```text
TX[
appendList(frames, A), // seq 41
setValue(operationState, X), // seq 42
appendList(frames, B), // seq 43
]
```
Reading `frames` returns `A`, then `B`. Gaps from unrelated writes are expected. A list element's `seq` is session-global and unique to that committed write; it is an ordering/cursor identity, not an application domain ID. Applications that need domain identity include it in `T`.
Rules:
- append never reads existing elements;
- an element is immutable after commit;
- `deleteList(address)` removes every element at that exact address;
- deleting an absent list is a no-op;
- delete followed by append in one transaction creates a fresh list atomically;
- there is no per-element update, delete, insertion, or truncation;
- all validation and serialization required to admit a transaction completes before Memory state changes;
- a failed transaction exposes none of its list or non-list writes.
“Append-only” describes elements while the list exists. Whole-list deletion is lifecycle cleanup, not element mutation.
### List reads
- ascending reads return `seq > cursor.seq`;
- descending reads return `seq < cursor.seq`;
- results are ordered according to `order` before `limit` is applied;
- absent and empty both return `[]`;
- callers continue with the last returned element's `seq`;
- an empty page ends iteration;
- `limit` is only the query-page size: it must be a positive safe integer, defaults to 1,000, and values above 10,000 clamp to 10,000; it never limits total list length or bytes.
```ts
let cursor: ListCursor | undefined;
while (true) {
const page = await reader.readList(events, { cursor, order: "asc", limit: 100 });
if (page.length === 0) break;
consume(page);
cursor = { seq: page[page.length - 1]!.seq };
}
```
A cursor is a sequence filter, not a snapshot or list-incarnation token. Concurrent later appends may appear on later ascending pages. Whole-list deletion may make a cursor stale; reads simply apply its sequence comparison to currently surviving elements.
Do not add an unbounded “read the whole list” helper.
## Assistant partial frames
Assistant partial durability is the first built-in list consumer:
```ts
const frames = pendingAssistantFrames(operationId, responseEntryId);
```
`AssistantMessageFrame`, `AssistantMessageFrameEncoder`, and `reduceAssistantMessageFrames()` come from `@earendil-works/pi-ai`. Do not define a second frame codec or reducer.
For every convertible non-terminal provider event, the assistant procedure:
```text
convert event to frame
→ synchronously enqueue appendList(frames, frame) on the Session mutation line
→ attach the ordinary harness-fault observer to that returned promise
→ replace the process-local latestFrameWrite reference
→ emit and await the existing message event
→ consume the next provider event
```
The provider loop does not await storage for every frame. Synchronous enqueue preserves provider-event order. Replacing the latest-promise reference never leaves an earlier rejection unobserved because every promise receives the fault observer. Bounded output bounds queued work. On stream settlement, the procedure stops frame admission and awaits the latest append promise before `after_response`; Session mutation FIFO means that completion implies every earlier append completed. There is no timer, batcher, coalescer, or flush API.
Scalar assistant `effect_pending` remains authoritative. Each append verifies that the same operation, attempt, and response ID still own the lane when its mutation executes. Frames never prove request admission, completion, success, or failure.
Final or synthetic assistant settlement deletes the exact list atomically with its immutable response, usage, Branch tip, and next scalar state:
```text
TX[
insert final assistant entry,
insert usage,
deleteList(frames),
setValue(operationState(operationId), nextState),
]
```
`assistant-durability.md` defines frame conversion, unknown-outcome synthesis, cancellation, deferred polling, snapshots, and event ordering.
## Restore policy
Scalar operation state remains the sole restart authority:
1. construct the trusted lane/operation projection from required scalar values;
2. trust committed typed values rather than auditing every referenced payload or phase relationship;
3. when a procedure or snapshot consumes auxiliary state, derive its exact bound address from current typed scalar state;
4. hydrate only the bounded scalar values or list pages that consumer requires.
A missing auxiliary list is legal unless its consumer explicitly requires an element. List contents never prove that an external effect completed. Live mutations still verify current operation, phase, attempt, and reserved identity as concurrency fencing; that is not restore validation.
Base restore does not enumerate lists. For assistant frames, snapshot or recovery derives `pendingAssistantFrames(operationId, responseEntryId)` only while consuming a typed assistant/deferred `effect_pending` state.
Each list consumer defines:
- address grammar;
- element and total-byte bounds;
- page/hydration budget;
- cleanup transitions;
- fork and migration policy.
## Memory backend
Memory may keep separate maps for current values and list elements:
```ts
const scalarValues = new Map<string, StoredValue<unknown>>();
const listValues = new Map<string, ListElement<unknown>[]>();
function physicalKey(address: StoredAddressBase): string {
return `${address.namespace}\u0000${address.key}`;
}
```
- scalar set replaces one map value;
- scalar delete removes it;
- list append pushes the already-sequenced element;
- list delete removes the complete array;
- list read filters by exclusive cursor and slices to the validated limit;
- transaction preparation completes before entries, values, lists, usage, or stats mutate.
Storage snapshots used by JSONL/fork tooling include current scalar values and surviving list elements with original sequence numbers.
## SQLite backend
The logical schema has one current-value table and one list-element table:
```sql
CREATE TABLE scalar_values (
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY (namespace, key)
) WITHOUT ROWID;
CREATE TABLE list_values (
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY (namespace, key, seq)
) WITHOUT ROWID;
```
WP01 replaces the unfinished format-4 schema in place: edit `sqlite/migrations/001_initial.sql`, rename the physical `registers` table to `scalar_values`, add `list_values`, and keep `SQLITE_STORAGE_VERSION = 1`. There is no migration runner in this WIP implementation, and pre-WP01 SQLite files are unsupported. Do not add migration machinery in this package.
List operations:
```sql
INSERT INTO list_values(namespace, key, seq, value) VALUES (?, ?, ?, ?);
SELECT seq, value FROM list_values
WHERE namespace = ? AND key = ? AND seq > ?
ORDER BY seq ASC LIMIT ?;
SELECT seq, value FROM list_values
WHERE namespace = ? AND key = ? AND seq < ?
ORDER BY seq DESC LIMIT ?;
DELETE FROM list_values WHERE namespace = ? AND key = ?;
```
For a missing cursor, omit the sequence predicate. Every write participates in the existing `BEGIN IMMEDIATE` transaction; writable Session ownership belongs to the host lifecycle, not SQLite storage. Assert with `EXPLAIN QUERY PLAN` that paging uses the primary key and no temporary sort.
## JSONL backend
Logical records carry the bound address's physical components:
```jsonl
{"kind":"list","op":"append","seq":41,"namespace":"pi.pending.assistant_frame","key":"O:R","value":{"type":"text_delta","contentIndex":0,"delta":"hi"}}
{"kind":"list","op":"delete","seq":52,"namespace":"pi.pending.assistant_frame","key":"O:R"}
```
Scalar records use `kind:"value"` with `op:"set"|"delete"`. WP01 keeps JSONL format 4 and storage version 1 but replaces the unfinished record spelling in place; pre-WP01 format-4 files are unsupported and no legacy `kind:"register"` decoder remains.
Replay folds records into the Memory state:
- scalar set replaces the current address;
- scalar delete removes it;
- list append adds `{ seq, value }`;
- list delete removes the complete list.
A transaction remains one physical JSONL line, using an array for multiple writes. Torn-tail handling therefore remains atomic without new framing.
### Snapshot compaction
Compaction writes every surviving list element with its original `seq`, merged in sequence order with surviving entries, scalar values, and usage rows. Do not collapse one live list into a synthetic element or assign new sequence numbers; either change breaks cursors and backend equivalence.
Deleted lists produce no snapshot records. Snapshot rewrites persist `nextSeq` in the format-4 header so dropping the latest delete cannot permit sequence reuse; ordinary append-only files may omit that field and derive it from replayed writes.
## Forks and rewrites
Fork and precise-rewrite code decides policy per concrete address grammar:
- operation-owned `pi.op.*` scalar values are not copied into an idle fork;
- immutable `pi.result` operation records are not copied by forks;
- `pi.pending.entry`, `pi.pending.tool_output`, and `pi.pending.assistant_frame` values/lists are not copied;
- lane and semantic session values follow their existing scope rules;
- application-defined values/lists are not copied by the generic fork; a consuming feature must add an explicit address-specific policy before relying on copied application state.
A precise rewrite retaining list elements preserves their `seq` values unless it explicitly remaps the entire destination sequence space.
## Schema evolution
A bound address's namespace, key grammar, kind, and value type are durable schema:
- changing namespace or key grammar requires explicit address migration;
- changing scalar to list or list to scalar requires explicit migration;
- storage never infers or coerces kind from observed records;
- changing TypeScript value shape requires total value migration when old stored values are incompatible;
- a list migration pages elements in sequence order and either maps them while preserving `seq` or deletes the complete list;
- a migration must not load an unbounded logical list at once.
Adding generic list storage replaces the current WIP backend schema in place. Constructing a new application address with no persisted value requires no migration.
## Instrumentation and telemetry
The instrumented storage decorator exposes the address-based read API and records committed erased writes in exact transaction order.
Telemetry session-write item kinds distinguish scalar-value writes from list writes. Namespace/key names may be attributes when the telemetry schema permits them, but values, assistant frames, prompts, and tool output never enter telemetry.
Append-path tests prove that no `readList` call occurs before append commit. Frame-persistence promises always receive the harness fault observer, even when an earlier promise is no longer the latest settlement-order reference.
## Invariants
1. One bound address has one stable namespace/key/kind and one trusted value type in a storage version.
2. Address object identity has no durable meaning.
3. Namespace `pi` and every `pi.*` are reserved by contract; every built-in namespace starts with `pi.`, and application use is a trusted-programming defect.
4. Exactly five built-in prefix constructors encapsulate Branch inventory and operation cleanup grammar; their results are consumed only by namespace-scoped `scanValues()`.
5. Scalar and list addresses must not occupy the same physical location; this is a trusted-programming rule, not a runtime cross-kind collision check.
6. Typed reads and helper-constructed writes preserve `T`.
7. Scalar helpers reject list addresses; list helpers reject scalar addresses.
8. A Session/Storage operation never requires a second key after address construction.
9. Every list element is immutable and carries its globally unique committed write `seq`.
10. Elements at one list address are returned in sequence order on every backend.
11. Append performs no read of the target list.
12. Scalar/list writes are atomic with entries and usage in the same transaction.
13. Whole-list delete leaves no elements at that address.
14. Missing and empty lists both read as `[]`.
15. Base restore depends only on required scalar state and never enumerates auxiliary lists.
16. Auxiliary lists never establish effect completion or select a restart state.
17. JSONL compaction preserves surviving element sequences.
18. Terminal cleanup leaves no operation-owned scalar values or lists.
## Required tests
### Address typing and identity
- `value<T>()` and `list<T>()` preserve their declared `T` invariantly;
- scalar reads infer the bound address's value type;
- list reads infer its element type;
- `setValue` rejects an incompatible value at compile time;
- `appendList` rejects an incompatible element at compile time;
- scalar helpers reject list addresses and list helpers reject scalar addresses;
- independently constructed equal addresses access the same durable location;
- incompatible definitions of one physical address are documented/tested as a programming defect;
- empty keys work, while empty namespaces and separator-containing components reject;
- core and application code use the same `value()` and `list()` constructors, with no private constructor, privilege token, registry, or catalog;
- built-in address constructors produce exact `pi.branch.tip`, `pi.lane.*`, `pi.op.*`, `pi.pending.*`, `pi.session.name`, and `pi.entry.label` namespace/key/kind triples;
- every built-in namespace starts with `pi.`, while application fixtures use non-reserved namespaces;
- `branchTipInventoryPrefix()` binds the empty-key `pi.branch.tip` inventory prefix and is used only to enumerate Branches through `scanValues`;
- tool-args prefixes cover one operation and optionally one step, tool-memo prefixes cover one operation and optionally one invocation, preparation and tool-output prefixes cover exactly one operation;
- each prefix constructor result is used only by `scanValues`, and no inventory or cleanup call constructs a raw reserved namespace;
- application addresses work without declaration merging or core catalogs;
- no Storage or Session operation accepts an additional key argument.
### Scalar regression
- set/get/delete/recreate behavior is unchanged;
- replacement retains only the latest logical value and latest set `seq`;
- typed write helpers preserve mixed transaction order;
- prefix scans interpret the bound address key as a prefix and remain namespace-scoped;
- new scalar JSONL/SQLite files use only the value/list schema; pre-WP01 WIP files are explicitly unsupported.
### List conformance
Extend the shared backend conformance suite:
- append one element and page it;
- multiple appends to one address in one transaction;
- appends separated by unrelated writes preserve per-list order;
- every element receives its own global write `seq`;
- ascending and descending exclusive cursors;
- default, explicit, invalid, and capped limits;
- absent list returns `[]`;
- whole-list delete and delete of absent list;
- delete followed by append in one transaction;
- rollback when a later write is invalid;
- atomic list + entry + usage + scalar transaction;
- identical pages and cursors on Memory, JSONL, and SQLite;
- JSONL torn multi-write transaction exposes no list element;
- JSONL replay and compaction preserve cursors;
- SQLite paging uses the primary key without temporary sorting;
- append performs no list read;
- base restore constructs trusted scalar projection without list reads, followed by bounded consumption-time hydration;
- close rejects later reads and honors already-admitted commits.
### Application surface
- an application-wide scalar value requires no extra key at get/set;
- an application-wide list requires no extra key at read/append;
- an application can construct dynamic per-workspace addresses explicitly;
- Storage and Session accept the same address objects and infer the same types;
- direct Session writes serialize and commit once;
- explicit `Session.mutate()` can atomically combine typed value/list writes with entries and usage.
### Assistant-frame integration — deferred beyond WP01
- every converted non-terminal frame appends under the exact bound effect-pending response address;
- terminal `done`/`error` events append nothing;
- appends enqueue synchronously without provider backpressure;
- every frame-write promise has an observed fault path;
- only the latest promise reference is retained for settlement ordering;
- awaiting the latest promise implies every earlier append completed;
- reduced pages reconstruct the same partial message as uninterrupted streaming;
- missing list restores as no durable partial;
- final/synthetic settlement atomically deletes the frame list;
- unknown-effect recovery reads only the bounded list derived from current scalar state;
- external finalization deletes the operation-owned list;
- idle forks contain no frame list;
- backend byte growth is append-linear rather than repeated-snapshot growth.
## Implementation map
Expected primary changes:
- replace `session/registers.ts` with `packages/agent/src/harness/session/values.ts` containing addresses, constructors, typed write helpers, and built-in address constructors;
- remove `RegisterValues`, namespace unions, register token types, and raw namespace/key read signatures from `session/types.ts`;
- expose `ValueReader` through Storage, SessionReader, SessionMutator, and Session;
- expose direct application scalar/list methods on Session using bound addresses;
- update Memory state, JSONL codec/storage, snapshots, fork/rewrite code, instrumentation, and conformance suites;
- replace SQLite's unfinished initial schema in place with `scalar_values` and `list_values`; keep storage version 1 and add no migration runner;
- update telemetry schema sources and regenerate `telemetry-schema.md`; do not edit that generated file manually.
WP01 stops after generic addresses/storage and projection-only restore coverage. `assistant-durability.md` specifies the later consuming lifecycle; assistant execution, deferred polling, recovery, snapshot hydration, memo/checkpoint capabilities, and operation cleanup land only with their runtime work packages.

View file

@ -0,0 +1,83 @@
# WP00 — Runtime1 removal
## Status
Complete. No tag. Runtime2 is the sole public implementation. Do not release while its execution path is incomplete.
## Goal
Make runtime2 the sole public harness implementation, delete runtime1 and its obsolete tests, then stop before adding runtime2 behavior.
## Prerequisites
- The approved acceptance/hook redesign and durability handoffs are present in `harness.md`, `values.md`, `assistant-durability.md`, and `tool-durability.md`.
- Existing tests are evidence, not authority.
## Work, in order
1. **Reconcile the contract.** Fold the approved acceptance/hook redesign into `harness.md`, including durable `starting`, atomic hook-free acceptance, driver-owned `before_run`, `before_drive`, request-local system-prompt transformation, trusted restore, and removal of process-origin activation semantics. Audit §§0.4, 1.2, 3.1–3.6, 4.1–4.2, 4.5, 5.1–5.2, 5.5–5.6, and Parts 8–9; remove every stale `BeforeResumePrepared`, `before_resume`, `resumeData`, `systemPromptOverride`, stable-ID routing, reservation, and `fresh | continue | resume` activation reference. Delete obsolete runtime planning documents once `harness.md` and linked handoffs own the active contract.
2. **Harvest before deletion.** Inspect `agent-harness-runtime.test.ts`, `agent-harness-r2/r3/r4.test.ts`, and old `restore.test.ts`. Preserve unique scenarios in the detailed future rows or a temporary categorized inventory; explicitly discard old reservation, `before_resume`, `resumeData`, persisted hook prompt override, semantic restore audit, and pre-`outcome_ready` tool-crash behavior.
3. **Remove runtime1-only public members.** Delete `before_resume`, `BeforeResumePrepared`, `resumeData`, `systemPromptOverride`, and stable hook-ID routing. Add the approved `before_drive` and `transform_context` shapes. Update telemetry schema source and regenerate its document. Do not implement `starting` or acceptance behavior here.
4. **Switch the factory.** Add `packages/agent/src/harness/runtime2/index.ts`, point `agent-harness.ts` at it, and add a constructor-selection regression. Verify the experimental coding-agent worker still creates the harness, subscribes to events, and closes it.
5. **Delete runtime1 source and tests** listed below.
6. **Update `[Unreleased]` on `main` or a pull-request branch** for the public breaking removals and temporarily incomplete factory. Repository policy forbids changelog edits on `dev`, so WP00 records but does not perform that release-facing step here.
7. Run the retained tests and checks. Fix every failure; do not restore compatibility shims.
## Delete
```text
packages/agent/src/harness/runtime/**
packages/agent/src/harness/restore.ts
packages/agent/test/harness/agent-harness-runtime.test.ts
packages/agent/test/harness/agent-harness-r2.test.ts
packages/agent/test/harness/agent-harness-r3.test.ts
packages/agent/test/harness/agent-harness-r4.test.ts
packages/agent/test/harness/restore.test.ts
packages/agent/test/harness/scratch/r1.ts
packages/agent/test/harness/scratch/r2.ts
packages/agent/test/harness/scratch/r3.ts
packages/agent/test/harness/scratch/r4.ts
```
## Retain
- all `test/harness/runtime2/**` tests;
- Session, Branch, storage, repository, backend-conformance, and instrumentation tests;
- execution assistant/tool/primitives tests;
- config, hooks, events, telemetry, compaction, and branch-summary code/tests;
- `types.test.ts`, updated for the reduced public contract;
- `packages/agent/src/agent-loop.ts` unchanged.
Do not parameterize obsolete runtime1 suites against runtime2 and do not keep a runtime1 smoke suite.
## Acceptance
- No source import references `harness/runtime/*`.
- Public `AgentHarness.create()` selects runtime2.
- Runtime2 creation, events, inspection, close, and fault tests pass.
- These coding-agent tests pass:
- `experimental-remote-runtime.test.ts`
- `experimental-session-worker-manager.test.ts` (the upstream replacement for the removed `experimental-session-worker.test.ts`)
- `experimental-session-worker-lifecycle.test.ts`
- Every modified test passes individually.
- Agent and root TypeScript pass.
- `git diff --check` and `npm run check` pass.
## Outcome
- The accepted hook/drive contract is normative in `harness.md`; stale acceptance/resume contracts are absent.
- Runtime1 source, validating restore, obsolete suites, and R1–R4 scratch scenarios are deleted.
- `AgentHarness.create()` resolves through `runtime2/index.ts`; a constructor-selection regression proves it.
- The scenario harvest added missing tool-close, identity-preflight, recovery-ordering, turn-bracket, and telemetry cases to future rows.
- Upstream added two real remote prompt tests after this handoff was drafted. They remain present but skipped with an R2 re-enable requirement because runtime2 execution is intentionally incomplete. Worker creation, attachment, lifecycle, operation correlation, and close coverage passes.
## Non-goals
- No bound-value/list implementation.
- No implementation of `starting` or atomic acceptance.
- No drive owner, provider, retry, deferred, or tool execution.
- No runtime1 parity work, compatibility layer, archaeology tag, or release.
## Stop condition
Stop once runtime1 is absent, runtime2 is the public factory, retained coverage is green, and all checks pass. Report the deletion and harvested scenarios; do not begin another work package.

View file

@ -0,0 +1,327 @@
# WP01 — Bound values and lists
## Status
Complete. `harness.md` is normative. [`values.md`](../values.md) supplies the detailed address, backend, and conformance design.
## Goal
Replace the retained register/custom-state storage surface with bound `Value<T>` and `ValueList<T>` addresses across Session, Memory, JSONL, SQLite, instrumentation, tests, and public application access. Stop before runtime execution consumers.
## Decisions fixed for this package
1. **Core and application namespaces.** Every core address uses the exact documented `pi.*` namespace. Applications use their own non-reserved namespaces through the same public `value()` / `list()` constructors. `fact.custom` and its API are deleted, not renamed to a built-in custom namespace.
2. **Forks.** Generic forks copy only explicitly handled core addresses: Branch tip/lane configuration plus fresh lane state, session name, and labels whose targets copy. They copy no `pi.op.*`, `pi.pending.*`, list, ledger, or arbitrary application address. A later application feature must add an address-specific fork policy before relying on copied application state.
3. **Trusted kind discipline.** Using one `(namespace, key)` as both a value and a list is a trusted-programming defect. Backends do not add cross-kind collision checks, triggers, registries, or catalogs.
4. **Operation names.** Keep source `OperationMeta` for immutable acceptance metadata and `Operation` for the process-local `{ meta: OperationMeta, state: OperationState }` projection. `operationMeta(id)` binds `Value<OperationMeta>`; the composite is never persisted as one value.
5. **Query ordering and bounds.** `scanValues()` returns key-ascending results. `readList()` limits only one query page, never total list length or bytes: reject non-positive or non-safe limits, default to 1,000, and clamp larger values to 10,000.
6. **No WIP compatibility.** Replace the unfinished format-4 storage schema in place. JSONL remains format 4/storage version 1 but accepts only the new value/list records. SQLite keeps `SQLITE_STORAGE_VERSION = 1`, edits `001_initial.sql` in place, renames `registers` to `scalar_values`, and adds `list_values`. Pre-WP01 format-4 JSONL and SQLite files are unsupported. Add no migration runner or legacy decoder.
7. **Generic infrastructure only.** WP01 defines every built-in value/list constructor, including future assistant/tool addresses, but does not implement their runtime consumers.
## Public and storage contract
Add `packages/agent/src/harness/session/values.ts` with:
- invariant `Value<T>` and `ValueList<T>` address types;
- universal `value<T>(namespace, key?)` and `list<T>(namespace, key?)` constructors;
- namespace/key validation only: non-empty namespace and no `\u0000` component;
- `StoredValue<T>`, `ListElement<T>`, `ListCursor`, and `ListReadOptions`;
- typed `setValue`, `deleteValue`, `appendList`, and `deleteList` write helpers using `NoInfer<T>`;
- every exact built-in constructor and the five scan-prefix constructors from `values.md`.
Replace the old API throughout:
```ts
getRegister(namespace, key) -> getValue(address)
listRegisters(namespace, prefix) -> scanValues(prefixAddress)
register set/delete writes -> typed value helpers
```
Storage and the historical Session reader, mutator, tree-view, and repository surfaces expose the same bound-address reads:
```ts
getValue<T>(address: Value<T>): Promise<StoredValue<T> | undefined>;
scanValues<T>(prefix: Value<T>): Promise<StoredValue<T>[]>;
readList<T>(address: ValueList<T>, options?: ListReadOptions): Promise<ListElement<T>[]>;
```
The historical tree-view and Session surfaces additionally expose one-commit direct writes:
```ts
setValue<T>(address: Value<T>, next: NoInfer<T>): Promise<void>;
deleteValue<T>(address: Value<T>): Promise<void>;
appendList<T>(address: ValueList<T>, element: NoInfer<T>): Promise<void>;
deleteList<T>(address: ValueList<T>): Promise<void>;
```
`SessionMutator` retains one explicit `commit(writes)` and does not gain direct committing methods. Each write array composes helper-constructed value/list writes with entries and usage.
Keep `getName` / `setName` and `getLabel` / `setLabel` as wrappers over `sessionName` and `entryLabel(id)`. Delete `getCustomFact` / `setCustomFact`. Rename the public passive metadata event from `fact_update` to the `value_update` shape already specified by `harness.md`; it covers only session-name and entry-label wrappers, not arbitrary application writes.
## Files
### Add
- `packages/agent/src/harness/session/values.ts`
- `packages/agent/test/harness/values.test.ts`
- `packages/session-backends/sqlite-node/src/sqlite/session/values.ts`
### Delete or rename
- delete `packages/session-backends/sqlite-node/src/sqlite/session/registers.ts` after moving its scalar behavior to `values.ts`;
- remove all register/global-map/custom-fact declarations from `packages/agent/src/harness/session/types.ts`.
### Agent source
- `packages/agent/src/harness/agent-harness.ts`
- `packages/agent/src/harness/session/types.ts`
- `packages/agent/src/harness/session/commit.ts`
- `packages/agent/src/harness/session/storage-state.ts`
- `packages/agent/src/harness/session/memory.ts`
- `packages/agent/src/harness/session/session.ts`
- `packages/agent/src/harness/session/fork.ts`
- `packages/agent/src/harness/session/index.ts`
- `packages/agent/src/harness/session/jsonl/storage.ts`
- `packages/agent/src/harness/session/jsonl/repo.ts`
- `packages/agent/src/harness/session/testing/storage-decorator.ts`
- `packages/agent/src/harness/session/testing/instrumented-storage.ts`
- `packages/agent/src/harness/session/testing/types.ts`
- `packages/agent/src/harness/session/testing/conformance/storage.ts`
- `packages/agent/src/harness/session/testing/conformance/session-repo.ts`
- `packages/agent/src/harness/session/testing/benchmark/storage.ts`
- `packages/agent/src/harness/session/testing/benchmark/session-repo.ts`
- `packages/agent/src/harness/session/testing/index.ts`
- `packages/agent/src/harness/runtime2/restore.ts`
- `packages/agent/src/harness/runtime2/harness.ts`
- `packages/agent/src/harness/runtime2/lane.ts`
- `packages/agent/src/harness/telemetry.ts`
- `packages/agent/src/index.ts` and `packages/agent/src/node.ts` only as needed to verify the new public exports; do not add a second export path.
### Agent tests and generated documentation
- `packages/agent/test/harness/memory-storage.test.ts`
- `packages/agent/test/harness/memory-conformance.test.ts`
- `packages/agent/test/harness/memory-session-repo.test.ts`
- `packages/agent/test/harness/jsonl-storage.test.ts`
- `packages/agent/test/harness/jsonl-storage-conformance.test.ts`
- `packages/agent/test/harness/jsonl-session-repo.test.ts`
- `packages/agent/test/harness/jsonl-session-repo-conformance.test.ts`
- `packages/agent/test/harness/storage-backed-session.test.ts`
- `packages/agent/test/harness/session-tree.test.ts`
- `packages/agent/test/harness/session-create-lane.test.ts`
- `packages/agent/test/harness/instrumented-storage.test.ts`
- `packages/agent/test/harness/types.test.ts`
- `packages/agent/test/harness/telemetry.test.ts`
- `packages/agent/test/harness/runtime2/harness.test.ts`
- `packages/agent/test/harness/runtime2/lane.test.ts`
- `packages/agent/test/harness/runtime2/restore.test.ts`
- regenerated `packages/agent/docs/telemetry-schema.md`
### SQLite backend
- `packages/session-backends/sqlite-node/src/sqlite/migrations/001_initial.sql`
- `packages/session-backends/sqlite-node/src/sqlite/repo.ts`
- `packages/session-backends/sqlite-node/src/sqlite/session.ts`
- `packages/session-backends/sqlite-node/src/sqlite/storage.ts`
- `packages/session-backends/sqlite-node/src/sqlite/index.ts` if needed for the renamed module
- `packages/session-backends/sqlite-node/test/storage.test.ts`
- `packages/session-backends/sqlite-node/test/storage-conformance.test.ts`
- `packages/session-backends/sqlite-node/test/repo.test.ts`
- `packages/session-backends/sqlite-node/test/adapter.test.ts`
- `packages/session-backends/sqlite-node/test/sql.test.ts`
### Coding-agent consumer
- `packages/coding-agent/test/experimental-session-support.ts`
- verify `packages/coding-agent/test/experimental-remote-runtime.test.ts`
- verify `packages/coding-agent/test/experimental-server-replacement.test.ts`
If the final old-API grep identifies another retained source/test call site, it belongs to WP01; do not add a compatibility shim to avoid touching it.
## Work, in order
1. **Add the address vocabulary.** Implement `values.ts`, export it through the existing session/root path, and add focused compile-time/runtime address tests. Include exact built-in namespace/key/kind tests and prefix-constructor tests before migrating callers.
2. **Cut the shared API once.** Replace register types and writes in `types.ts`/`commit.ts`; split `StorageState` into current scalar values and surviving list elements; implement Memory reads/writes, ordered prefix scans, paged list reads, transaction validation/application, snapshots, and direct Session methods. Remove custom-fact APIs and migrate name/label wrappers.
3. **Migrate JSONL and generic fork/snapshot code.** Encode only `kind:"value"` and `kind:"list"`; replay set/delete/append/delete; preserve transaction-line torn-tail atomicity; serialize surviving list elements with original `seq` merged in global sequence order; preserve the sequence high-water mark. This extends existing snapshot serialization only—do not add a new compaction trigger or precise-rewrite feature. Forks copy the fixed core set from Decisions item 2, re-sequence destination scalar values after copied entries as today, and copy no lists.
4. **Migrate instrumentation, conformance, and benchmarks.** The storage decorator exposes all three reads; instrumented storage records erased value/list writes in exact order without content telemetry. Extend shared conformance before backend-specific assertions.
5. **Migrate runtime2 shell call sites.** Replace lane/harness raw writes with built-in constructors/helpers. `restore.ts` uses `scanValues(branchTipInventoryPrefix())` plus exact `getValue` lookups and performs no `readList()` call. Do not add acceptance, drive, hydration, or cleanup behavior.
6. **Replace the SQLite WIP schema and adapter.** Edit `001_initial.sql` in place, implement scalar operations and indexed list append/delete/paging in `session/values.ts`, keep every write inside the existing `BEGIN IMMEDIATE` writer-lease transaction, update both fork snapshot paths, and retain all entry/usage/branch/lease behavior from current `dev`.
7. **Migrate public events, tests, and the coding-agent helper.** Remove old type assertions and raw namespaces. Change `fact_update` to `value_update`. Keep the two remote prompt tests skipped for the existing WP00 reason; WP01 must not alter runtime execution.
8. **Update telemetry and documentation.** Change `pi.session.write` item kinds from `register` to `value` and `list`, regenerate `telemetry-schema.md`, run the old-API sweeps, and record any branch-policy-deferred changelog requirement. Do not edit a changelog on `gramps` unless it becomes a pull-request branch or the user requests it.
## Backend requirements
### Memory
- prepare and validate the complete transaction before mutating entries, values, lists, usage, or stats;
- current scalar replacement stores only the latest value and set `seq`;
- list append performs no list read and stores each global write `seq`;
- list delete removes the whole exact key;
- snapshots include current scalar values and surviving list elements.
### JSONL
- keep format 4/storage version 1 with no legacy register decode;
- one single-write object or multi-write array remains one atomic line;
- replay produces the same logical state as Memory;
- torn final lines expose no partial transaction;
- snapshot serialization preserves surviving list-element sequences and the next-sequence high-water mark.
### SQLite
Use:
```sql
CREATE TABLE scalar_values (
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY (namespace, key)
) WITHOUT ROWID;
CREATE TABLE list_values (
namespace TEXT NOT NULL,
key TEXT NOT NULL,
seq INTEGER NOT NULL,
value TEXT NOT NULL,
PRIMARY KEY (namespace, key, seq)
) WITHOUT ROWID;
```
Ascending and descending list queries use the primary key with an exclusive sequence predicate and `LIMIT`. Add `EXPLAIN QUERY PLAN` assertions proving no table scan or temporary ordering b-tree. Do not change storage version, add migrations, or weaken the current lease/fence/fork behavior.
## Required coverage
### Address and type tests
- invariant address typing and inferred scalar/list result types;
- `NoInfer` rejects incompatible set/append values;
- scalar helpers reject list addresses and list helpers reject scalar addresses;
- independently constructed equal addresses resolve the same location;
- empty key works; empty namespace and `\u0000` components reject;
- exact built-in namespaces/key grammars and exactly five prefix constructors;
- application-wide and dynamic non-reserved addresses require no second operation-time key;
- no registry, catalog, privilege constructor, global value map, or runtime `pi.*` gate.
### Shared scalar/list conformance
- scalar set/get/delete/delete-absent/recreate and latest set `seq`;
- namespace-scoped, key-ascending prefix scans;
- append one and several elements, including several appends in one transaction;
- appends separated by unrelated writes preserve per-list order and global element sequences;
- ascending/descending exclusive cursors;
- default, explicit, invalid, and clamped query-page limits;
- absent list, whole-list delete, delete-absent, and delete-then-append;
- atomic entry + usage + value + list transactions;
- rollback when any sibling write is invalid;
- no list read on append;
- close rejects later reads while already-admitted commits drain.
Do not add a value/list collision test: cross-kind misuse is intentionally an unenforced trusted-programming defect.
### Backend and repository tests
- identical Memory/JSONL/SQLite pages and cursors;
- JSONL single/multi-write replay, torn-tail behavior, and sequence-preserving snapshot output;
- SQLite query plans and writer-lease transaction behavior;
- branch/tree forks copy session name, eligible labels, and lane configuration/Branch tip with fresh lane state;
- forks exclude operation/pending values, all lists, application addresses, last results, queues, and ledger rows;
- repository parent metadata, entry IDs, stats, branch indexes, v3 normalization, UUIDv7/follower IDs, and current SQLite lease/fork scenarios remain unchanged;
- runtime2 restore enumerates lanes through the one prefix constructor and reads no list.
## Deferred consumers
The following `values.md` requirements are explicitly not WP01 coverage:
- assistant frame conversion, append scheduling, settlement, recovery, cancellation, snapshot hydration, and byte-growth tests (R2/R3/R6/R12);
- invocation `getMemo` / `setMemo`, tool-output checkpoint writes, outcome cleanup, and prefix-driven operation cleanup (R4/R6);
- any consumption-time list hydration beyond proving base restore reads no list;
- runtime acceptance, driving, provider/tool effects, or operation-state redesign.
WP01 still exports `pendingAssistantFrames`, `operationToolMemo`, `pendingToolOutput`, and every cleanup prefix so later packages do not redesign storage.
## Removal checks
These must have zero matches in retained source/tests, excluding immutable released changelog history and archived prose:
```bash
rg -n 'getRegister\(|listRegisters\(|RegisterValues|RegisterNamespace|RegisterSetWrite|\bRegister<' \
packages/agent packages/session-backends/sqlite-node packages/coding-agent \
--glob '!**/dist/**' --glob '!**/CHANGELOG.md' --glob '!**/docs/**'
rg -n 'kind: "register"|getCustomFact\(|setCustomFact\(|fact\.(name|label|custom)|fact_update' \
packages/agent packages/session-backends/sqlite-node packages/coding-agent \
--glob '!**/dist/**' --glob '!**/CHANGELOG.md' --glob '!**/docs/**'
rg -n '"(branch\.tip|lane\.(config|state|lastResult)|op\.(meta|state|tool_args|preparation)|pending\.entry)"' \
packages/agent packages/session-backends/sqlite-node packages/coding-agent \
--glob '!**/dist/**' --glob '!**/CHANGELOG.md' --glob '!**/docs/**'
rg -n '\bregisters\b' packages/agent/src/harness/session packages/session-backends/sqlite-node/src \
--glob '*.ts' --glob '*.sql'
rg -n '"register"' \
packages/agent/src/harness/telemetry.ts \
packages/agent/test/harness/telemetry.test.ts \
packages/agent/docs/telemetry-schema.md
```
Do not treat unrelated model/provider/hook registration terminology as storage API residue.
## Validation
Run each created or modified test file directly and iterate until green. At minimum:
```bash
# From packages/agent
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
test/harness/values.test.ts \
test/harness/memory-storage.test.ts \
test/harness/memory-conformance.test.ts \
test/harness/memory-session-repo.test.ts \
test/harness/jsonl-storage.test.ts \
test/harness/jsonl-storage-conformance.test.ts \
test/harness/jsonl-session-repo.test.ts \
test/harness/jsonl-session-repo-conformance.test.ts \
test/harness/storage-backed-session.test.ts \
test/harness/session-tree.test.ts \
test/harness/session-create-lane.test.ts \
test/harness/instrumented-storage.test.ts \
test/harness/types.test.ts \
test/harness/telemetry.test.ts \
test/harness/runtime2/harness.test.ts \
test/harness/runtime2/lane.test.ts \
test/harness/runtime2/restore.test.ts
# From packages/session-backends/sqlite-node
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
test/adapter.test.ts \
test/repo.test.ts \
test/sql.test.ts \
test/storage.test.ts \
test/storage-conformance.test.ts
# From packages/coding-agent
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
test/experimental-remote-runtime.test.ts \
test/experimental-server-replacement.test.ts
```
Then run from the repository root:
```bash
cd packages/agent && npm run check:telemetry-docs
cd "$(git rev-parse --show-toplevel)"
node_modules/.bin/tsgo --noEmit -p packages/agent/tsconfig.build.json
node_modules/.bin/tsgo --noEmit
git diff --check
npm run check
./test.sh
```
Never run unrestricted Vitest, `npm test`, paid-provider tests, or `npm run build`.
## Stop condition
Stop when every retained backend and Session surface uses bound values/lists; all core addresses use the exact `pi.*` grammar; arbitrary application addresses work but generic forks exclude them; old register/fact/custom-state APIs and physical names are absent; base restore performs no list read; schema/compatibility decisions above are implemented; focused, conformance, TypeScript, telemetry-doc, diff, and repository checks pass. Report the final schema and fork behavior. Do not begin runtime acceptance, assistant/tool consumers, or any later work package.

View file

@ -0,0 +1,542 @@
# WP02 — Atomic acceptance and coherent lane observation
## Status
Complete. Phase A established minimal attachment, open-operation inventory, Session-line inspection/watch capture, commit-continuation recipient binding, and in-band identity-failure vocabulary. Phase B landed in `beac75ecc` with focused, monorepo-check, full-suite, and final Fable review passing.
Implementation also updated downstream protocol/coding-agent wire projections required by the changed public contract. No execution owner, provider/tool effect, timer, retry, polling, cancellation procedure, manual action, or terminal settlement was added.
## Goal
Deliver two effect-free boundaries:
```text
idle lane
→ atomic prompt/skill/template acceptance
→ durable open operation in payload-free starting
open or running lane
→ Session-line watch capture
→ complete snapshot plus gap-free subsequent events
```
After `AgentHarness.create(options, context)` succeeds:
- every configured lane has a complete small process-local projection;
- `open` inventories every durable current operation without predicting model/tool availability;
- attachment starts no hook, provider, tool, timer, breakpoint, drive owner, or application callback;
- `inspectExecution(context)` observes the small projection and local owner on the Session line;
- `watch(context)` registers buffering, clones live presentation, and performs bounded snapshot reads in one no-write lane job;
- every committing lane job publishes memory and synchronously binds its event batch in the commit continuation;
- the mutation does not await listener delivery, while the public operation does.
WP02 does not implement drive, provider generation, hooks, tools, retries, deferred polling, cancellation procedures, manual action execution, or terminal settlement.
## Decisions
### 1. Attachment restores projection, not presentation
Passing the open Session to `AgentHarness.create()` transfers orchestration ownership until create rejects or the harness closes. Direct `Session.mutate`, `Session.createLane`, reserved-address writes, and second-harness construction are prohibited during that interval, so lane inventory cannot race an out-of-band lane creation.
Attachment reads only:
- `branchTip`, `laneConfig`, `laneState`, optional `laneLastResult`;
- `operationMeta` and `operationState` for a current operation.
It validates required existence, operation id/lane ownership, and intent/state kind compatibility. Projection corruption faults `create()`.
Attachment does not read transcript, queues, pending writes, drained payloads, deferred sources, frames, tool calls, arguments, checkpoints, preparations, memos, or staged outcomes. Those references are checked by `watch()` or drive when consumed. Missing or contradictory required payload data is terminal storage corruption and faults that consumer. Optional frame/checkpoint absence is legal.
### 2. Watch owns detailed snapshot reads
One no-write Session mutation job defines the watch boundary:
```text
enter after all earlier lane jobs
→ synchronously register buffering watcher
→ synchronously clone live presentation state
→ perform bounded durable reads while later lane jobs are excluded
→ assemble snapshot
→ release Session line
→ return handle
```
There is no special first-watch cache. The same path handles immediate post-attachment watch, reconnect, and watch during live execution.
The bounded read set is:
- one compaction-bounded branch scan from the current tip;
- exact `pendingEntry(id)` reads for next-run, steer, follow-up, writes, and abort drains;
- exact deferred source entry when represented;
- tools-phase assistant entry and exact args for represented `effect_pending` calls, using `batch.turnId` as the args step id;
- bounded exact assistant-frame pages and exact optional tool checkpoints.
`ToolCall.sourceIndex` is the index in the assistant message's full content array, not a filtered tool-call ordinal. A represented call must index a tool-call block.
### 3. Recipient binding belongs in the commit continuation
A successful committing lane job performs:
```text
commit
→ publish small owned projection
→ synchronously bind recipients and append the complete `{ event, context }` batch
→ return from the mutation without awaiting delivery
→ await delivery before the public operation resolves
```
WP02 initially implemented this with `enqueue()` plus caller-operated `start()`. WP04 replaces that gate with one immediate `emitBatch()` call in the same commit-observation continuation. Recipient binding and delivery awaiting remain unchanged. A listener or watcher registered after `emitBatch()` cannot receive that historical event.
The only watcher/publication orders are:
```text
watcher first
→ snapshot-before + complete buffered event batch
publication/`emitBatch` first
→ snapshot-after + no old event
```
A live provider/tool presentation update follows the same synchronous publish-plus-`emitBatch` discipline. Frame/checkpoint commits are lane jobs and queue behind watch capture.
### 4. Inspection is a no-write Session-line observation
`inspectExecution(context)` observes:
- lane and tip;
- configured model identity as unresolved `{ provider, modelId }` strings;
- current operation id/kind/start time;
- process status `running`, `open`, or durable `aborting`;
- captured model identity when the current durable phase contains one;
- optional latest result.
It does not resolve model/tool registries and does not read transcript or presentation payloads.
### 5. Missing implementations are in-band outcomes
There is no `blocked`, missing-identity suspension, predictive classifier, or acceptance registry preflight.
At the actual execution boundary:
- unavailable captured model or configured active-tool definition before provider intent becomes a non-retryable configuration failure;
- pre-intent configuration failure reserves no response/usage ids and fabricates no assistant response or usage row;
- restored `effect_pending` settles uncertainty under its existing reserved ids before any later configuration failure;
- unavailable deferred model durably abandons redemption through configuration failure; R7 implements and tests restored `effect_pending` abandonment, including deletion of its exact old assistant-frame list while dropping the reserved response/usage strings without fabricating settlement;
- missing requested tool stages a direct `isError` `ToolResultMessage` and continues;
- missing/no-longer-safe replay implementation synthesizes interruption instead of waiting.
Synthetic harness tool results omit `details`. A tool owns the type of its details contract; the harness must not invent `{}` or a diagnostic object. `isError` and human-readable content carry the tool-level diagnosis. Run-level configuration failure remains machine-readable through `OperationError` and `laneLastResult`.
Stable configuration error codes:
- `model_unavailable`, details `{ provider, modelId }`;
- `configured_tools_unavailable`, details `{ tools: string[] }`.
`failure_drain` gains `{ kind: "configuration" }` provenance. Actual transitions land with their owning execution packages; WP02 lands the normative/source vocabulary only.
### 6. Acceptance is independent of process registries
Acceptance validates durable caller input and lane state, not current model/tool registrations. This avoids a time-of-check/time-of-use check and permits acceptance in one process followed by execution in another.
A misconfigured convenience prompt eventually returns a durable failed run from drive. Explicit hosted acceptance remains durable even before an execution worker loads implementations.
### 7. Invocation Context remains explicit
Every current public harness/lane operation receives trailing `context: Context`. Acceptance, attachment, watch capture, Session reads/commit, faults, and event publication preserve it. Shared harness/lane/Session receivers retain no caller Context. Context and its signal/telemetry values are never durable business data.
Buffered events retain the exact emitting Context. Invocation cancellation remains distinct from durable `requestAbort()`.
## End-state public contract
```ts
export interface ModelIdentity {
provider: string;
modelId: string;
}
export type OperationStatus = "running" | "open" | "aborting";
export interface OpenOperation {
lane: string;
operationId: string;
kind: "run" | "compaction" | "navigation";
startedAt: number;
aborting?: true;
}
export interface CurrentOperationInfo {
id: string;
kind: "run" | "compaction" | "navigation";
startedAt: number;
status: OperationStatus;
capturedModel?: ModelIdentity;
}
export interface LaneExecutionInfo {
lane: string;
tipId: string | null;
configuredModel: ModelIdentity;
current: CurrentOperationInfo | null;
lastResult?: LaneLastResult;
}
export interface LaneInfo {
name: string;
tipId: string | null;
operation: CurrentOperationInfo | null;
}
export interface AgentHarnessConstructor {
create<TContext extends object | undefined = object | undefined>(
options: AgentHarnessOptions<TContext>,
context: Context,
): Promise<{ harness: AgentHarness<TContext>; open: OpenOperation[] }>;
}
```
Rules:
- `open` has exactly one item per durable current operation and omits idle lanes;
- `aborting:true` comes only from durable `cancel_requested`;
- `open` is inventory, not scheduling or identity advice;
- normal applications establish watch and call `resume(context)`;
- hosted schedulers retain expected-id `drive` fencing;
- configured/captured identity fields are durable strings and may not resolve.
### Outcomes
Delete `MissingIdentitySuspension`, `MissingIdentities`, missing-identity drive waiting, and missing-identity suspension events.
Keep deferred suspension as provider semantics:
```ts
{ kind: "suspended"; reason: "deferred"; ... }
```
WP05 removes the withdrawn action outcome before execution is enabled. Convenience operation outcomes remain operation-tagged branches of `ResumeOutcome`.
### Snapshot
```ts
export interface LaneSnapshot {
lane: string;
transcript: Entry[];
tipId: string | null;
lastResult?: LaneLastResult;
operation: null | {
id: string;
kind: "run" | "compaction" | "navigation";
startedAt: number;
status: OperationStatus;
action?: ActionInfo;
retry?: { attempt: number; maxAttempts: number; nextAttemptAt: number };
deferred?: { handle: DeferredHandle; poll: number };
drained?: { steer: QueuedItem[]; followUp: QueuedItem[] };
streamingMessage?: AssistantMessage;
runningTools: {
toolCallId: string;
toolName: string;
args: unknown;
partialResult?: AgentToolResult<unknown>;
}[];
};
queues: { steer: QueuedItem[]; followUp: QueuedItem[]; nextRun: QueuedItem[] };
pendingWrites: {
entryId: string;
type: EntryType;
customType?: string;
message?: AgentMessage;
data?: JsonValue;
}[];
faulted: boolean;
}
```
Configuration is not duplicated in snapshots. `inspectExecution()` exposes configured/captured model identities; getters expose current configuration.
## Atomic run acceptance
WP02 implements `accept()` for prompt, skill, and prompt-template requests. Compaction/navigation acceptance remains with their execution packages.
State-independent normalization occurs before `Lane.command(plan, context)`:
- prompt strings/images;
- supplied message or message array;
- explicit skill formatting;
- prompt-template formatting;
- pending assistant rejection;
- unknown skill/template errors;
- supplied or minted operation and prompt-entry ids.
Public prompt convenience overloads remain exactly `[text, images | undefined, context]` and `[messageOrMessages, context]`, but convenience implementations remain `SliceNotImplemented` until R2.
Inside one lane command:
1. reject busy;
2. capture current `pendingNextRun` ids;
3. read and validate captured pending messages;
4. reject zero placed messages;
5. parent captured next-run entries before request prompt entries;
6. commit exactly once;
7. publish the small owned projection;
8. synchronously call `emitBatch` with the acceptance event batch and accepting Context;
9. return from the mutation callback without awaiting delivery;
10. await event delivery before `accept` resolves.
Exact writes:
```text
insert captured nextRun message entries
insert request prompt entries
delete captured pendingEntry values
set branchTip
set operationMeta
set operationState(run starting)
set laneState(current operation, pendingNextRun=[])
```
Exact event order:
```text
run_start
for each placed message:
message_start
message_end
entry_added
queue_update if nextRun was captured
```
Acceptance starts no drive or effect and writes no Context.
## Phase A — normative rewrite
Update `harness.md` before runtime source:
- replace eager attachment hydration with minimal projection restore;
- replace predictive status/classifier with open inventory;
- specify configured/captured identity inspection without resolution;
- move detailed reads to ad-hoc Session-line watch capture;
- require recipient binding in the exact commit-observation continuation;
- remove identity preflight/suspension/error/event types;
- specify in-band model/tool unavailability and configuration provenance;
- require direct detail-free synthetic tool-result messages;
- update invariants, races, roadmap, glossary, and Appendix C.
Review stop:
1. `git diff --check`;
2. fresh Terra contradiction/source-feasibility audit;
3. full-context Fable review against complete docs and source;
4. resolve all findings and repeat until no findings;
5. obtain user approval before runtime source.
## Phase B — implementation
### Public and durable types
Modify `agent-harness.ts`:
- remove `SuspendedOperation`, `MissingIdentityInfo`, `MissingIdentities`, and missing-identity outcome/event branches;
- add `ModelIdentity`, `OperationStatus`, `OpenOperation`, and corrected inspection/snapshot types;
- change create result from `suspended` to `open`;
- land operation-tagged action-required outcome families;
- correct `executeAction`/`runToCompletion` signatures;
- retain trailing Context everywhere.
Modify session types:
- add `failure_drain` configuration provenance;
- add the already-normative `ToolCall.outcome_ready` vocabulary without producers;
- document `sourceIndex` as full assistant-content index;
- add callback-scoped `scanBranch` to `SessionReader`; inheritance by `SessionMutator` and `Session` is intentional;
- implement it in `StorageBackedSession` and its mutator and `MemorySessionFacade`; Session mutation authority remains process-local and has no remote Session facade.
### Event publication
WP02 initially added synchronous recipient binding with a reserved delivery gate. WP04 supersedes only that mechanism:
- `HarnessEventBus.emitBatch()` snapshots ordinary and watcher recipients synchronously;
- delivery uses only that bound list;
- watcher buffers retain `{ event, context }`;
- a watcher registered after `emitBatch` receives nothing from that event.
`LaneCommand` commit decisions retain a synchronous post-commit event batch. After commit succeeds, `Lane.command` publishes `next`, calls `emitBatch` as its final mutation action, and carries the delivery promise outside `Session.mutate` before awaiting it. Every existing event-producing commit uses this path, including direct idle/pending appends, lane configuration setters, and session-name/entry-label setters.
WP04 also moves harness lane publication into `Session.createLane`'s committed-publication callback. Session commits, Harness publishes `lanesByName` and calls `emitBatch(lane_created)` in that continuation, and Session awaits the retained delivery promise after releasing the line.
Do not execute listeners on the line.
### Attachment
Keep `restore.ts` projection-only. Validate lane/operation ownership and kind compatibility. Remove `describeSuspension` and all payload hydration from `createAgentHarness`. Construct lanes and return `open` inventory without resolving registries.
### Inspection
Implement `inspectExecution(context)` as a no-write `Lane.command` observation. Derive captured model identity from the current durable phase. Read no storage payloads and resolve no registry identities.
### Watch
Implement `watch(context)` as one no-write lane job:
- register watcher and clone live presentation synchronously;
- perform the bounded read matrix above through callback-scoped readers;
- assemble isolated snapshot payloads;
- fault and unsubscribe on required corruption;
- retain legal optional absence;
- release the line before returning.
A focused internal snapshot helper is allowed. Do not create a persistent hydrated presentation cache or generic reducer.
### Out of scope
`drive`, `resume`, prompt convenience, compaction/navigation acceptance, abort/queues, `executeAction`, and `runToCompletion` remain `SliceNotImplemented` where not already implemented.
Provider/tool/configuration-failure transitions are specified now but implemented by R2/R3/R4/R7/R8. WP02 adds no effect, active operation, timer, hook, provider request, tool execution, retry, deferred fetch, cancellation reconciliation, or terminal transaction.
## Required tests
### Public types
- `SuspendedOperation`, `MissingIdentities`, and missing-identity status/outcomes/events are absent;
- open/current/status unions narrow exhaustively;
- configured and captured model identities are unresolved strings;
- action-required outcome families and method signatures match;
- prompt overload tuples remain exact;
- `AgentHarnessOptions` has no receiver telemetry default.
### Acceptance
- text-only, images-only, text-plus-images, supplied arrays;
- pending assistant rejection;
- skills/templates and unknown resources;
- empty input writes nothing unless captured nextRun supplies input;
- no identity registry preflight;
- supplied/minted operation ids;
- exact writes, parent chain, metadata, starting state, settings, commit materialization;
- pending-next-run capture/deletion;
- busy run/structural operation;
- one concurrent accept winner;
- commit failure and close races;
- exact event order and object-identical accepting Context;
- no hook, provider, tool, timer, drive-owner, or option callback invocation; passive event-listener delivery remains required.
### Attachment and inspection
- idle omitted and every open operation inventoried;
- durable cancellation sets only `aborting:true`;
- no transcript/pending/frame/tool reads at create;
- configured and captured model identities may differ and remain visible when unresolved;
- inspection runs on the Session line and performs no payload reads;
- projection corruption faults create;
- no option callback/effect starts.
### Event publication and watch
- recipient set binds at `emitBatch`: a watcher registered after publication but before delivery receives nothing;
- state/event batch publication happens before Session-line release;
- direct append, lane configuration, session-name/entry-label, acceptance, and lane-creation commits all use that publication path;
- `value_update` and `lane_created` bind recipients in their commit continuation, so later listeners receive neither historical event;
- watcher-first gives snapshot-before plus complete events;
- publication-first gives snapshot-after without old events;
- live update during awaited capture appears only as a buffered event after the cloned live snapshot;
- frame/checkpoint mutations queue behind capture;
- first watch and reconnect use the same path;
- exact transcript, queues, writes, drain, deferred, frame, tool args/checkpoint fields;
- required payload corruption faults watch and removes watcher;
- absent frames/checkpoints omit optional partials;
- pre-registration lifecycle is not replayed;
- buffered events retain object-identical emitting Context;
- payload mutation cannot affect later state/listeners;
- close/fault lifecycle matches the event bus.
### In-band identity vocabulary
Type/direct-state tests prove:
- acceptance has no `MissingIdentities` path;
- configuration failure provenance and stable error codes are representable;
- deferred configuration abandonment remains assigned to R7 rather than adding an execution transition here;
- missing-tool synthetic `ToolResultMessage` may omit `details`;
- sourceIndex uses full assistant-content indexing;
- outcome-ready calls are not rendered as running.
No execution transition is added in WP02.
## Files
### Add
- `packages/agent/test/harness/runtime2/accept.test.ts`
- focused watch tests if existing files become oversized.
### Modify
- `packages/agent/docs/harness.md`
- `packages/agent/docs/work-packages/02-atomic-run-acceptance.md`
- `packages/agent/src/harness/agent-harness.ts`
- `packages/agent/src/harness/events.ts`
- `packages/agent/src/harness/session/types.ts`
- `packages/agent/src/harness/session/session.ts`
- `packages/agent/src/harness/session/memory.ts`
- `packages/agent/src/harness/session/remote.ts`
- `packages/server/src/remote-session-manager.ts`
- `packages/agent/src/harness/runtime2/harness.ts`
- `packages/agent/src/harness/runtime2/lane.ts`
- `packages/agent/src/harness/runtime2/restore.ts`
- `packages/agent/src/harness/runtime2/types.ts`
- `packages/agent/test/harness/runtime2/harness.test.ts`
- `packages/agent/test/harness/runtime2/lane.test.ts`
- `packages/agent/test/harness/runtime2/restore.test.ts`
- `packages/agent/test/harness/types.test.ts`
- `packages/agent/test/harness/storage-backed-session.test.ts`
- `packages/agent/test/harness/memory-session-repo.test.ts`
- `packages/server/test/conformance.test.ts`
- event/session test files required by recipient binding and callback-scoped branch reads.
No backend schema, telemetry schema, coding-agent, or changelog change is expected. Stop for boundary review if one becomes necessary. On `dev`, defer changelog entries.
## Validation
After Phase A:
```bash
git diff --check -- \
packages/agent/docs/harness.md \
packages/agent/docs/work-packages/02-atomic-run-acceptance.md
```
After Phase B:
```bash
cd packages/agent
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
test/harness/runtime2/accept.test.ts \
test/harness/runtime2/harness.test.ts \
test/harness/runtime2/lane.test.ts \
test/harness/runtime2/restore.test.ts \
test/harness/types.test.ts
cd "$(git rev-parse --show-toplevel)"
git diff --check
npm run check
./test.sh
```
Report runtime2 source line counts. The synchronized pre-WP02 runtime2 baseline is 967 lines; treat growth above 1,900 source lines as a design review trigger, not a target.
## Stop condition
Stop when:
- acceptance commits exactly once into payload-free `starting` without registry preflight;
- attachment returns minimal complete projections and open inventory;
- inspection is a coherent Session-line no-write observation;
- watch captures detailed state ad hoc on the Session line;
- state publication and `emitBatch` recipient binding occur in the commit continuation while the mutation never awaits delivery;
- snapshots and buffered events have no gap or duplicate;
- required payload corruption faults its consumer;
- no execution effect or owner is introduced;
- focused tests, `npm run check`, and full tests pass;
- final Fable review has no findings.
Do not begin the first real drive package.

View file

@ -0,0 +1,183 @@
# WP03 — Remove drive deadlines
## Status
Complete. `DriveOptions.deadline` and the `DriveOutcome` `yielded` branch are removed from public types, active documentation, future package requirements, invariants, races, and tests. Obsolete `runtime2.md` is deleted. Focused tests, `npm run check`, and `./test.sh` pass.
WP02 is complete at `beac75ecc`. Preserve unrelated concurrent source work, especially current `packages/agent/src/harness/runtime2/lane.ts` changes, JSONL/fork work, plugins, RPC, and experimental directories.
## Problem
`DriveOptions.deadline` and the corresponding `DriveOutcome { kind: "yielded" }` do not provide a correctness boundary.
A deadline is checked only before starting another transition or effect. An admitted provider/tool/hook may run beyond it and the host may terminate the process anyway:
```text
check deadline
→ admit provider or tool
→ host limit expires while the effect is running
→ process dies with durable effect_pending
```
Unknown-outcome recovery remains mandatory. The deadline therefore does not prevent process loss, bound admitted work, make effects exactly once, or simplify recovery. Flue-style tool memoization depends on stable invocation ids and durable memos, not drive deadlines.
Deadline handling instead adds wall-clock policy to the durable core:
- a `yielded` public outcome unrelated to durable state;
- safe-boundary checks before hooks/effects/transitions;
- deadline-versus-retry-timer arbitration;
- deadline-versus-effect-admission races;
- convenience loops and event-bracket behavior for yields.
The host already owns scheduling and termination. Process loss is a controlled crash boundary recovered from durable operation state.
## Decision
Remove drive deadlines completely:
```ts
interface DriveOptions {
operationId: string;
waitForRetry?: boolean;
pollDeferred?: boolean;
}
type DriveOutcome =
| { kind: "settled"; operationId: string; outcome: TerminalOperationOutcome }
| { kind: "waiting"; operationId: string; reason: "retry"; notBefore: number }
| { kind: "waiting"; operationId: string; reason: "deferred"; deferred: DeferredHandle }
| { kind: "action_required"; operationId: string; action: ActionInfo };
```
There is no deprecated alias, ignored `deadline` field, compatibility overload, alternate timestamp option, or replacement pause flag in WP03.
The next drive package uses direct durable transitions. Deterministic tests gate commits and control hooks, providers, tools, and timers without adding production execution barriers.
## Host behavior
Hosts own execution budgets:
```text
invoke drive
→ terminal or durable waiting result: schedule normally
→ planned shutdown: stop routing/releasing work and close session processes
→ forced termination: replacement attaches and recovers durable open operations
```
A process-per-session host may stop routing work and close before exit, and use process termination as the hard fence for non-cooperative providers, tools, hooks, storage, or event listeners. This operational policy does not require a wall-clock field in `DriveOptions`.
WP03 does not add `stopAfterCheckpoint`, `pause`, `quiesce`, or in-process crash simulation. Those ideas remain outside the direct durable-drive design. In-process process-loss simulation is not a public core primitive: an old continuation requires fencing, for which process isolation is the reliable mechanism.
## Work
### Public types
In `packages/agent/src/harness/agent-harness.ts`:
- delete `DriveOptions.deadline`;
- delete the `DriveOutcome` `yielded` branch;
- preserve expected-id fencing, retry waiting, deferred waiting, and manual action branches unchanged.
No current runtime2 drive implementation exists, so this package adds no execution behavior or owner.
### Normative documentation
Update `packages/agent/docs/harness.md` completely:
- remove safe-yield scheduling language from non-goals/orientation;
- remove only the deadline half of cancellation/deadline prerequisite wording in §3.6, §5.6's `before_drive` row, and invariant 22; the cancellation prerequisite remains;
- remove deadline checks and yielded returns from the drive-pass pseudocode;
- remove deadline policy from pass joining, retry waiting, convenience composition, recovery, and public method prose;
- remove `deadline` from `DriveOptions` and `yielded` from `DriveOutcome`;
- remove deadline-specific event/turn requirements;
- remove deadline language from `before_drive` hook timing;
- remove deadline/yield requirements from future work rows;
- replace invariant 25 with the explicit no-wall-clock-policy invariant while retaining the rule that an admitted effect settles normally or is recovered after task loss;
- remove deadline races from the race catalog;
- update the drive-pass glossary.
Do not alter generic RPC timeout/deadline documentation or unrelated process/model-catalog timeouts. Those are invocation/transport policies, not `AgentLane.drive`.
Delete obsolete `packages/agent/docs/runtime2.md`; `harness.md` plus linked handoffs are the sole implementation plan and history needed for active work.
Mark WP02 complete in its handoff and Part 8. Add WP03 as the concrete cleanup package. Leave the former R2/R3 drive rows as historical future candidates, minus deadline/yield requirements, until the reviewed direct-drive handoff replaces them.
### Delete
- `packages/agent/docs/runtime2.md`
### Type tests
Extend `packages/agent/test/harness/types.test.ts`:
- assert `keyof DriveOptions` is exactly `"operationId" | "waitForRetry" | "pollDeferred"`;
- assert `DriveOutcome["kind"]` excludes `"yielded"`;
- add `@ts-expect-error` coverage proving callers cannot supply `deadline`;
- retain existing drive/result signatures.
Run the focused type test after editing it.
### Downstream compatibility
Search protocol, server, coding-agent, examples, and tests for structural mirrors or exhaustive `DriveOutcome` switches. Update only compile-time/type compatibility forced by the public removal. Coding-agent experimental worker/remote-runtime behavior and tests are out of scope.
The current protocol harness schemas expose prompt/run/watch DTOs, not `DriveOptions` or `DriveOutcome`; no protocol edit is expected unless the final search proves otherwise.
## Non-goals
WP03 does not implement or redesign:
- `drive`, `resume`, prompt conveniences, operation ownership, or latest-result lookup;
- manual actions or automatic barrier release;
- provider/tool execution, recovery, retry timers, deferred polling, abort, or terminal settlement;
- checkpoint pause/quiesce;
- close admission changes;
- worker/RPC cancellation or experimental remote prompting;
- storage/schema/migration behavior.
## Required checks
```bash
# No drive deadline/yield contract remains in active harness docs or source.
rg -n 'deadline|yield' \
packages/agent/src/harness \
packages/agent/test/harness \
packages/agent/docs/harness.md \
packages/agent/docs/work-packages
cd packages/agent
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run \
test/harness/types.test.ts
cd "$(git rev-parse --show-toplevel)"
git diff --check
npm run check
./test.sh
```
The removal grep may still match this handoff's historical problem explanation. Every remaining match must be reviewed; no active API, normative behavior, future acceptance criterion, or test expectation may retain the removed contract.
## Review
Before implementation:
1. Fable reviews this handoff against complete docs/source.
2. As explicitly requested by the user, `openai-codex/gpt-5.6-sol` reviews it with thinking level high.
3. Resolve all findings and repeat until no findings.
After implementation, repeat both reviews over the final documentation/type diff.
## Stop condition
Stop when:
- `DriveOptions` has no wall-clock budget;
- `DriveOutcome` has no `yielded` branch;
- active normative docs contain no deadline/yield behavior;
- future drive rows contain no hidden deadline requirements;
- type tests prove removal;
- unrelated timeout/deadline APIs remain untouched;
- focused tests, `npm run check`, and `./test.sh` pass;
- final Fable and Codex reviews have no findings.
Do not begin the direct durable-drive implementation in this package.

View file

@ -0,0 +1,337 @@
# WP04 — Mutation publication and event delivery
## Status
Complete. Phase A and the final implementation rereview passed Fable with no findings. Focused agent/server/SQLite tests, `npm run build`, `npm run check`, and `./test.sh` pass. `packages/agent/docs/harness.md` remains normative.
> Historical note: WP06 later replaced the lane-creation API and keyed line described in this completed handoff with atomic `AgentHarness.lane()` acquisition on one Session line. The event-publication guarantees remain current.
WP02 established atomic acceptance, recipient binding, and coherent lane watches. WP03 removed drive deadlines. WP04 removes the caller-operated event-delivery gate without weakening those guarantees and makes the historical `Session.createLane()` own lane creation end to end. The direct durable-drive package follows as WP05.
## Problem
A committing lane job currently uses a two-part event API:
```text
inside Session.mutate:
commit
publish process-local state
delivery = events.enqueue(batch, context)
outside Session.mutate:
await delivery.start()
```
The split preserves the correct boundary, but it is a footgun: calling `emit()` too early, calling `start()` too early, or dropping `start()` can violate observation semantics or stall the global event tail. The historical `Harness.createLane()` repeats the choreography manually.
Lane creation has a second one-off boundary. The historical `Session.createLane()` owns validation and the durable transaction, but Harness cannot publish its process-local `Lane` and bind `lane_created` recipients from that same commit continuation. Harness therefore opens `Session.mutate()` itself and calls exported `createLaneWithMutator()`.
WP04 removes both caller-operated seams while preserving current direct-listener and hook barriers.
## Required semantics
### Direct events remain awaited observations
Direct `events.on()` listeners remain passive but causally ordered:
```text
hook or preparation
→ commit
→ publish process-local state
→ bind and append event batch
→ release lane mutation line
→ await direct listeners
→ resolve operation
→ later hook or transition
```
Passive means a listener cannot transform the in-flight operation and listener failures are isolated as `handler_error`. It does not mean fire-and-forget. An extension may update process-local state in an event listener and inspect it from a later hook. Awaiting also supplies producer backpressure.
A direct listener must not call a state-mutating harness API: an emitted mutation would queue a later event behind the event currently awaiting that listener. Read-only lane calls remain legal.
Deliberate exceptions remain unchanged:
- watchers and RPC/watch consumers use their own buffered FIFO and are not awaited by operations;
- fault publication is fire-and-forget, and close never waits for listener completion;
- high-frequency tool updates enqueue every event and retain only the latest delivery promise; tool settlement awaits that promise before `after_tool` and outcome publication, which drains all earlier updates through global FIFO without per-update backpressure.
### Commit and recipient binding remain one continuation
Every event-producing committing lane job performs, in the exact continuation that observes successful commit:
```text
commit succeeds
→ publish complete process-local state
→ synchronously call emitBatch(batch, context)
- clone payloads
- bind current ordinary listeners and watchers
- append the complete batch to the global delivery tail
→ return from the mutation callback
```
There is no scheduler-owned after-release publication phase. Moving recipient binding into a later promise continuation creates a gap in which another task may observe committed state and register a listener before the old event binds recipients.
`emitBatch()` starts asynchronous delivery immediately and returns its completion promise. The mutation callback never awaits that promise. Listener code may begin after publication/binding but before the mutation line technically releases; a reentrant lane read queues behind the current job. The command carries the promise out of `Session.mutate()` and awaits it before resolving publicly.
This preserves the two legal watcher races:
```text
watcher registration first
→ snapshot-before + complete buffered batch
commit publication first
→ snapshot-after + no old event
```
### Ordering
- One non-empty `emitBatch()` call publishes one contiguous batch.
- Batches enter the existing global event tail in `emitBatch()` invocation order, including across lanes.
- Events within a batch retain source order.
- Direct listeners run serially in registration order.
- Same-lane committing jobs bind batches in lane mutation order.
- A lane procedure awaits each command's delivery before invoking its next hook or transition.
- Hooks on unrelated lanes may overlap; there is no total cross-lane hook order.
- Context remains the exact emitting invocation Context and is always the final parameter.
## Event bus contract
Replace the public/internal delivery split with:
```ts
class HarnessEventBus implements Events {
emit(event: HarnessEvent, context: Context): Promise<void>;
emitBatch(events: readonly HarnessEvent[], context: Context): Promise<void>;
}
```
`emitBatch()`:
1. returns an already-resolved promise for an empty batch or a closed bus;
2. synchronously structured-clones every payload and binds its current recipients;
3. appends one contiguous delivery to the existing global tail;
4. delivers cloned payloads serially with the emitting Context;
5. isolates listener failures and publishes non-recursive `handler_error` as today;
6. returns a promise that resolves after eligible direct listeners settle and never rejects because a listener failed.
Synchronous publication defects such as an uncloneable internal payload still throw in the caller's commit continuation and follow the existing harness-fault path.
Delete:
- `HarnessEventDelivery`;
- `HarnessEventBus.enqueue()`;
- caller-operated `start()`;
- delivery gates and `pendingStarts`;
- close-time forced gate release.
`close(error)` seals listener/watch registration and future publication immediately. Already appended batches retain their bound recipients and drain through the existing tail; listener completion still does not block Harness close.
## Lane command integration
The commit branch of `Lane.command()` returns a plain internal outcome containing the caller result and optional delivery promise:
```ts
const events = decision.events?.(commit) ?? [];
const delivery = events.length === 0
? undefined
: this.onEvent(events, context);
return {
kind: "return",
result,
...(delivery === undefined ? {} : { delivery }),
};
```
`onEvent` returns `Promise<void>` and delegates to `emitBatch()`. It is called only after commit, complete process-local state publication, and synchronous result materialization, as the final action before returning the mutation outcome.
After `Session.mutate()` returns:
```ts
if (outcome.kind === "reject") throw outcome.error;
await outcome.delivery;
return outcome.result;
```
Expected no-commit rejections publish no events. Commit/materialization/publication errors retain the existing harness-fault semantics.
## Session lane creation contract
The historical `Session.createLane()` owns validation, commit, and the synchronous committed-publication callback. Context remains last:
```ts
createLane(
name: string,
at: string | null,
configuration: LaneConfiguration,
onCommitted: ((context: Context) => void | Promise<void>) | undefined,
context: Context,
): Promise<SessionTree>;
```
The implementation performs:
```text
enter the prospective lane's mutation line
→ validate name, absence, complete lane shape, and target
→ commit lane configuration + leaf + idle lane state
→ synchronously invoke onCommitted(context) in that same commit continuation
→ retain its returned promise inside a non-thenable outcome object
→ return from the mutation callback and release the line
→ await the retained promise
→ return Session.view(name)
```
The callback receives neither `SessionTree` nor `SessionMutator`. It is only the process-local publication point. Its synchronous prefix must complete the publication needed before another same-lane job can run. Ordinary Session callers pass `undefined`.
If validation or commit fails, the callback is not invoked. If the callback throws after commit or its retained promise rejects after line release, the durable lane remains and the caller rejects; Harness converts that committed-publication defect to its existing fault path. A promise returned by the Harness callback is event delivery, whose listener failures are isolated by the bus.
The current lane validation/transaction implementation becomes private to Session. Delete exported `createLaneWithMutator()` and its direct tests.
Harness preconstructs a detached `Lane`, then calls Session:
```ts
const lane = this.buildLane(name, state);
await this.session.createLane(
name,
at,
this.seed,
(context) => {
if (this.closedError !== undefined) lane.seal(this.closedError);
this.lanesByName.set(name, lane);
return this.events.emitBatch(
[{ type: "lane_created", lane: name, at }],
context,
);
},
context,
);
return Result.ok(lane);
```
The callback's synchronous prefix publishes `lanesByName` and binds `lane_created` before the first creation job releases its line. A queued duplicate therefore cannot report `LaneExists` before the winner is visible through `harness.lane(name)`.
If close or fault wins while commit is admitted, the callback publishes the new Lane sealed. Emission on the already-closed bus is a resolved no-op, matching the existing admitted-creation race. The successful admitted creation still returns its sealed Lane.
## Scope
### Source
Modify:
- `packages/agent/src/harness/events.ts`;
- `packages/agent/src/harness/runtime2/lane.ts`;
- `packages/agent/src/harness/runtime2/harness.ts`;
- `packages/agent/src/harness/session/types.ts`;
- local Session implementations and tests required by the historical `createLane` signature;
- direct event primitive, acceptance, watch, lane, and harness tests.
Remote/experimental runtime behavior is not a design constraint for WP04. Session mutation authority remains process-local; do not add remote Session callback transport, protocol machinery, compatibility abstractions, or boundary tests.
### Documentation
Update normative `harness.md`:
- replace enqueue/start language with synchronous `emitBatch` binding and post-mutation awaiting;
- state accurately that listener execution may begin after publication/binding but before technical line release;
- retain awaited direct-listener, event/hook, watcher, Context, close, and ordering semantics;
- replace shared exported mutator-procedure lane creation with the historical `Session.createLane(onCommitted, context)`;
- update invariants, races, tests, glossary, and Part 8;
- link WP04 and move direct durable drive to WP05.
Update the historical WP02 handoff only where it points forward or claims the old mechanism remains current. Do not rewrite its completed-package record as though WP04 behavior landed in WP02.
## Non-goals
- fire-and-forget direct events or a public flush API;
- sequence/watermark delivery redesign;
- per-extension event queues;
- changing hook aggregation or event/hook causal barriers;
- changing watcher/RPC buffering;
- changing tool-update settlement barriers;
- making event-listener mutation safe;
- drive, breakpoints, providers, tools, recovery, retries, polling, abort, or terminal settlement;
- generic Session post-commit/after-release task APIs;
- remote callback execution or remote-runtime redesign.
## Required tests
### Event primitives
- `emitBatch` binds ordinary listeners and watchers synchronously;
- a listener registered after `emitBatch` but before delayed delivery receives nothing;
- a complete batch is contiguous and preserves event order;
- concurrent batch publication preserves invocation order;
- listener payload mutation remains isolated;
- listener rejection emits one non-recursive `handler_error` and does not reject delivery;
- empty and post-close batches resolve without delivery;
- already appended batches drain after close;
- no gate/start tests remain.
### Lane commands and watch
- direct listeners may perform reentrant read-only lane inspection without deadlock;
- a command does not resolve until its direct listeners finish;
- committed memory is visible to listeners;
- commit failure publishes nothing;
- a late listener that observes durable state receives no historical event;
- watcher-first yields snapshot-before plus the complete buffered batch;
- publication-first yields snapshot-after without replay;
- source Context identity survives delayed and buffered delivery.
### Lane creation
- Session callback runs exactly once after successful commit and never on validation/commit failure;
- callback synchronous publication occurs before a queued duplicate reports `LaneExists`;
- Harness Lane and durable configuration are visible to `lane_created` listeners;
- Harness waits for asynchronous `lane_created` listeners before resolving;
- admitted creation racing close publishes a sealed Lane and does not require event delivery;
- callback throw and retained-promise rejection after commit follow the documented committed-publication failure path;
- ordinary Session creation with `undefined` returns its view;
- no exported `createLaneWithMutator` remains.
### Ordering barriers
- acceptance events finish before `accept()` resolves;
- command resolution cannot overtake its awaited direct event delivery; WP05 tests the first subsequent procedure hook;
- event batches are globally ordered across concurrent lane publication;
- tool-update latest-delivery settlement behavior remains unchanged where implemented.
## Validation
After documentation:
```bash
git diff --check -- \
packages/agent/docs/harness.md \
packages/agent/docs/work-packages/02-atomic-run-acceptance.md \
packages/agent/docs/work-packages/04-mutation-publication.md \
packages/agent/docs/work-packages/05-direct-durable-drive.md
```
After implementation, run every modified focused test, then:
```bash
git diff --check
npm run check
./test.sh
```
Review the final implementation with Fable before declaring WP04 complete. Do not commit without explicit user approval.
## Stop condition
WP04 is complete when:
- event publication has one `emitBatch()` operation and no caller-operated gate;
- recipient binding remains in the exact commit-observation continuation;
- direct event delivery remains globally FIFO and awaited before public operation resolution;
- existing event/hook causal barriers remain intact;
- lane watches retain exactly the two coherent race outcomes;
- Session owns lane creation and invokes Harness publication before releasing the creation job;
- `createLaneWithMutator()` is gone;
- Context is trailing and source-identical throughout;
- focused tests, `npm run check`, and `./test.sh` pass;
- final Fable review reports no findings.

View file

@ -0,0 +1,606 @@
# WP05 — Direct durable drive
**Status: WP05 complete through M10: lane-owned inbox, immutable result records, 13 family-neutral leaves, atomic boundary planning, total cancellation/dispatch, public and replicated lane surfaces, documentation reconciliation, and lane-safe provider cache identity. Remaining assistant-output work is owned by the [mobile assistant-output handoff](../mobile-handoff/01-harness/05-assistant-output/message-update.md), outside the public-drive gate.**
WP06's Session/Branch/Lane separation is part of the foundation. Public drive is enabled; `watchSession` is the sole deferred Harness method.
Format 4 remains work in progress. Every durable type replacement in this package requires no migration and no compatibility decoder for the pre-redesign shapes.
The standalone-compaction inbox/promotion design previously in this document is **withdrawn** (§5). R1–R3 and M7/M8 replace it. M9 reconciled `docs/harness.md`, which is normative on its own again.
## 0. Mandatory reading
Read completely before implementation work:
1. `packages/agent/docs/harness.md` (normative; reconciled by M9)
2. `packages/agent/docs/runtime-simplification.md` (its 22-leaf list is replaced by R3)
3. `packages/agent/src/harness/session/types.ts`
4. `packages/agent/src/harness/session/values.ts`
5. `packages/agent/src/harness/runtime/lane.ts`
6. `packages/agent/src/harness/runtime/types.ts`
7. every existing `packages/agent/src/harness/runtime/drive/*.ts`
8. `packages/agent/src/harness/runtime/progress.ts`
9. `packages/agent/src/harness/runtime/restore.ts`
10. `packages/agent/src/harness/execution/{effect-gate,assistant,tools}.ts`
11. relevant focused runtime tests
Do not inspect Git history or removed runtime implementations. The current source and these documents are the only implementation inputs.
## 1. Runtime model (target)
### Lane authority
`Lane.state` is authoritative while a Harness owns its Session. It contains all orchestration state required to dispatch:
- tip;
- lane configuration;
- the lane inbox: tagged queued-input ids (R1);
- last operation id (R2);
- operation metadata;
- flat operation state, including control.
Every supported mutation commits on the one Session mutation line and publishes the matching `Lane.state` before releasing the line. Drive procedures never reread `laneState`, `operationMeta`, `operationState`, `branchTip`, `laneConfig`, or `operationResult` from storage.
`SessionReader` is used only to dereference content named by in-memory state and to enumerate operation-owned cleanup addresses:
- tree entries and branch context;
- pending entry payloads;
- assistant frame lists;
- tool arguments, checkpoints, and memos;
- structural preparations;
- staged tool outcomes;
- cleanup-prefix scans.
### One lane-owned inbox (R1)
Queued input is lane-owned, never operation-owned. `LaneState` (durable and process-local) carries one ordered inbox of `{ entryId, kind }` items with `kind: "steer" | "followUp" | "nextRun" | "write"`. Payload staging is unchanged: enqueue mints the entry id and writes `pendingEntry(id)`; the inbox holds ids only. Item lifecycle is admission → consumption | cancellation; terminal cleanup never touches queue payloads.
Tags are consumption-eligibility markers, not ownership. Enqueue always succeeds — during runs, structural operations, cancellation, and idle. Every drain point selects eligible items per tag, applying queue modes during selection, but places all selected items in the inbox's single global admission order. Tag grouping must never reorder user input. At acceptance, request prompt entries follow the selected inbox items because the request is the newest admission.
| Drain point | Eligibility and decision order |
| --- | --- |
| acceptance (idle lane) | write + nextRun (all), steer (`steeringMode`: all or oldest), and followUp (`followUpMode`) are eligible. Place selected items in global admission order, then request prompt entries. The acceptance transaction places them and removes their ids; `starting` drains nothing. followUp is eligible because an idle lane vacuously satisfies its "after current work" condition. |
| turn-end boundary pass (run) | write + steer are eligible before threshold/continuation planning and are placed in global admission order. followUp becomes eligible only at `may_finish`, before `before_run_end` and finish. nextRun is never eligible mid-run and never blocks finish. |
| idle direct append | queued write items are placed in admission order, then the new entry, in one commit |
| abort (M7) | steer + followUp are removed from the inbox, payload values deleted, and payloads returned; nextRun and write items stay |
**One decision, at most one commit (R1b).** A boundary pass may enter the mutation line several times for bounded reads and for `before_run_end` mediation, but it performs **at most one commit**, and that commit always lands in a state that neither drains nor rechecks (`assistant.ready`, `summary.deciding`, placed entries + `assistant.ready`, or a terminal transaction). No boundary decision ever commits back into `checkpoint`. A crash before the commit re-runs the whole decision with nothing consumed; a crash after it lands past the decision point. Consequently `skipInboxOnce` and `thresholdCheckedTriggerEntryId` are deleted: drains cannot re-fire because their target states do not drain, and the threshold marker is replaced by a guard derived from the branch itself — threshold fires only when `shouldCompact` holds **and** the branch's newest compaction entry is older than the trigger entry, so a committed threshold compaction is its own durable marker. Mode remainders stay queued and are consumed at later boundaries, giving the per-turn steer cadence and per-run-end followUp cadence.
There is no terminal drain. `cancelQueued` triage is unchanged and single-location. Queued input survives operation termination (including abort, for nextRun/write) until consumed or cancelled; this is a deliberate product decision.
### Flat state (R3)
`OperationState` has one `at` discriminator with **13** direct leaves:
- `starting`
- `checkpoint`
- `assistant.ready`
- `assistant.effect_pending`
- `assistant.retry_wait`
- `tools`
- `deferred.suspended`
- `deferred.effect_pending`
- `summary.deciding`
- `summary.ready`
- `summary.effect_pending`
- `summary.retry_wait`
- `navigation.ready_to_commit`
Every leaf carries one uniform scope `{ control, settings, latestAssistantEntryId }`. There are no `run.`/`compaction.`/`navigation.` leaf prefixes, no per-family scope types, and no intersection zoo. The four `summary.*` leaves carry a `SummaryTask` whose `boundary` datum (a closed three-arm union, §8) decides what happens at the result boundary. `starting` and `navigation.ready_to_commit` are intent-locked entry leaves. `ToolBatch`/`ToolCall` remain the nested child collection state machine. `Control` stays orthogonal and loses its drained fields when M7 installs drain-and-return abort.
### Durable operation results (R2)
Every terminal transaction writes one small immutable result record at `operationResult(operationId)` (namespace `pi.result`, operation id key). The record is lane-lived; keeping it outside `pi.op.*` keeps that namespace's "deleted no later than the terminal transaction" grammar total and exception-free. `LaneState.lastOperationId` points at the newest record. `laneLastResult` (value, address constructor, and type union) is deleted.
- `drive(id)` becomes total for settled operations: current id → install/join; record exists → return the record itself; neither → `OperationMismatch`.
- Recovery and attachment never read result records; they are observation only.
- Terminal-control invariant: a terminal transaction that executes under durable `cancel_requested` always records `status: "aborted"`; equivalently, any other status implies running terminal control. (Every non-aborted terminal path goes through `continueOperation`, which diverts under cancellation, and the mutation line serializes the marker against the terminal commit.)
- There is no listing, filtering, pagination, or retention API for records — and no hydration layer. `getResult(operationId)` and the drive settled arm are the entire read surface; both return the record with no entry dereference, so observation can never fault on missing entries.
### One lane-owned Drive
Unchanged: one process-local Drive pass per lane; first matching caller installs, later matching callers observe the same completion; no caller owns it after installation; invocation cancellation rejects only that caller's observation; an installed Drive is never abandoned or replaced in-process; `requestAbort(operationId)` is the only durable operation cancellation; a stale operation id returns `OperationMismatch` and affects nothing. `Drive.context` removes the installing invocation signal.
### Close
Unchanged: close is not abort. It seals mutation admission, rejects local observations with `HarnessClosed`, observes detached pass failures, drains mutations admitted before the seal, and closes the Session. It writes no cancellation marker or synthetic terminal state and never replaces a Drive.
### No external finalization
Unchanged: no live-process external-finalization path, `OperationEnded`, `finalizedOutcome`, ownership-loss result, or exact-Drive ABA fence.
## 2. Procedure shape
Live procedures are ordinary straight-line async functions:
```text
prepare
→ commit intent
→ admit and await effect
→ commit settlement
```
A procedure is the sole writer that changes its top-level `at` leaf. `requestAbort` changes only `control` (and the lane inbox); inbox admission changes only the lane inbox. Operation state has exactly two supported writers: the procedure and `requestAbort`.
The Lane supplies `continueOperation` and `settleOperation` as today. A terminal decision appends the universal suffix:
1. procedure-specific publication and cleanup writes;
2. `setValue(operationResult(operationId), record)`;
3. idle `laneState` with `currentOperationId: null`, `lastOperationId: operationId`, and the preserved inbox;
4. idle process-local projection;
5. terminal result and event materialization.
The state argument is a type capability established by dispatcher control flow, never compared with current state at runtime.
## 3. Remaining concurrency checks
Keep checks only where another supported writer can change the relevant fact:
1. `requestAbort` versus effect admission and settlement;
2. lane-inbox arrival before a drain point, a summary result boundary, or terminal finish;
3. parallel tool-call statuses and source-ready placement;
4. queued frame/checkpoint writes versus settlement;
5. invocation memo/checkpoint calls versus effect completion;
6. retry timers versus cancellation/close;
7. deferred permit consumption;
8. accept/claim serialization;
9. external/provider/content validation.
Do not reintroduce operation identity, expected-`at`, exact-Drive, or ownership-loss checks into ordinary transitions.
`requestAbort` keeps the two-step gate order: `beginAbort` before the cancellation mutation, commit `cancel_requested`, `signalAbort` after the commit.
## 4. Completed milestones
### M0 — Withdrawn execution-step controls
Complete. Breakpoints, manual drive, and drive deadlines are absent.
### M1–M2 — Foundations and terminal mechanics
Complete. Includes the split effect gate, deterministic gated storage, authoritative process-local Lane projection, commit-result usage totals, progress channels, terminal cleanup, and immutable operation-result observation.
### M3 — Assistant generation and recovery
Complete. Includes `run.starting → run.checkpoint`, assistant ready/intent/effect/settlement, frame persistence and cleanup, configuration failure, unknown-outcome assistant recovery, and response/usage/state atomicity.
### M4 — Retry and deferred
Complete. Includes durable retry waits, local wait policy, one deferred poll permit per pass, unknown-poll replacement under fresh ids, and event-stream-preserving `Models.streamDeferred`.
### M5 — Durable tools
Complete. Includes planned/effect-pending/outcome-ready/completed calls, safe replay, unsafe interruption, invocation memos, bounded checkpoints, completion-order staging, source-order placement, sequential and parallel modes, and tool-reported usage.
### Pre-M6 simplification
Complete. Canonical flat `at` types, `continueOperation`/`settleOperation`, removal of the installer-owned Drive model, and conversion of M3–M5 procedures.
### M6 structural foundation
Committed (`9b23c6583`): `runtime/drive/structural.ts`, `test/harness/runtime/drive-structural.test.ts`, the one-provider-request seams in the compaction modules, threshold routing, overflow preparation, per-request structural intent/usage settlement, hook decisions, structural retry/recovery, and unsummarized navigation commit. Its family cross-product layer (triplicated quadruples, three-armed publishers) is rebuilt by R3; its effect seam, preparation plumbing, threshold/overflow logic, and navigation commit are retained.
## 5. Withdrawn: standalone-compaction inbox and run-continuation promotion
The design that gave `compaction.*` leaves a `RunScope` inbox and a one-way cross-family move into `run.*` at the result boundary is withdrawn. Its problems: a relaxed intent/state invariant with bespoke restore rules, `compact()` resolving with a run outcome that does not fit `CompactionResult`, suppressed `run_start` producing unbalanced event brackets, fused-consumption rules to avoid `cancelQueued` stranding, and promotion-specific arms in reconciliation, cleanup, watch, and admission.
Replacement, spread across this package:
- queued input during any operation is lane-owned (R1) — no operation inbox exists to promote;
- continuation after standalone compaction/navigation is a **second ordinary run operation** with a fresh id, composed in the convenience layer (M8);
- each operation delivers exactly one result; both results are returned by the convenience call and both are durable records (R2).
No code implements promotion. M9 removed the withdrawn design from `harness.md`; this section remains only as the historical decision record.
## 6. R1 — Lane-owned tagged inbox
### Goal
Move all queued input to one lane-owned tagged inbox first (R1a, implemented), then — after R3 — replace the transitional stepwise checkpoint drains with one shared atomic boundary planner (R1b). `Control.drained*` remains until M7 introduces drain-and-return abort; it is not part of inbox ownership.
### R1a — Ownership relocation
- `src/harness/session/types.ts` — durable `LaneState { currentOperationId, inbox }` with `InboxItem { entryId, kind }` replaces `pendingNextRun`; delete `Inbox` and `RunScope.inbox`. Keep `Control.drained*`, `skipInboxOnce`, and `thresholdCheckedTriggerEntryId` temporarily.
- `src/harness/runtime/types.ts` — process-local `LaneState.inbox` replaces `pendingNextRun`; `LanePatch.inbox` pairs durable lane-state replacement with process-local publication.
- `src/harness/runtime/lane.ts` — acceptance selects per tag and mode but places selected items in global admission order, then request prompt entries; it deletes selected pending values and removes their ids atomically with the `LaneBusy` check. A lone queued write never validates acceptance. `append` during any operation enqueues a write-tagged lane item without touching operation state; idle `append` places queued writes first, then the new entry, in one commit. `watch` derives queues and pending writes from the lane inbox.
- `src/harness/runtime/drive/checkpoint.ts` — transitional stepwise drains read and replace the lane inbox rather than operation state. `skipInboxOnce` and `thresholdCheckedTriggerEntryId` continue to protect those temporary multi-commit boundaries.
- `src/harness/runtime/drive/terminal.ts` — operation cleanup never deletes lane-inbox payloads. Staged `outcome_ready` results, drained abort payloads, and other operation-owned addresses are unchanged.
- `src/harness/runtime/drive/{generation,response,deferred,tools,structural}.ts` — `runScopeOf` and scope copies drop `inbox`; ordinary operation-state transitions cannot clobber concurrent input.
- `src/harness/runtime/restore.ts`, fork/legacy normalization, and focused tests use the tagged lane inbox.
### R1b — Shared atomic boundary planner
**Lands after R3**, because pre-R3 the structural result boundary is spread across the triplicated publishers: fusing the planner into three arms and then collapsing them in R3 would be double work, while post-R3 the single `ResultBoundary` switch is exactly one planner call site. The two `CheckpointData` patch fields ride through R3's mechanical rename as inert baggage.
**Core invariant: no boundary decision ever commits back into `checkpoint`.** `checkpoint` is the durable "boundary pending" resting leaf, entered only from settlements, run-start consumption, and deferred redemption. Every exit of its pass is `assistant.ready` (with any placed entries), `summary.deciding`, a placed follow-up + `assistant.ready`, or the terminal transaction.
One shared boundary planner with a process-local `threshold: "check" | "skip"` parameter has exactly two callers:
- the `checkpoint` pass (`threshold: "check"`): select write + steer (modes, global admission order) → threshold guard → continuation → followUp (at `may_finish`) → `before_run_end` → finish; one commit;
- the structural result boundary for `resume_checkpoint` (`threshold: "skip"`): the **same planner runs inside the publication commit** — compaction entry (on success) first, then selected write/steer items, then the routed continuation, all in one transaction. Any placed conversational item routes to `assistant.ready` with that item as trigger, overriding a `may_finish` resume continuation (steer semantics). With nothing queued: `need_assistant` resumes commit `assistant.ready` directly; `may_finish` resumes commit `checkpoint{may_finish}` as the resting leaf and the live pass continues into the finish phases.
**Threshold without the durable marker.** The guard is derived from the branch: threshold fires only when `shouldCompact` holds and the newest compaction entry is older than the trigger entry. A committed threshold compaction is therefore its own marker — any crash re-entry sees the newer compaction and skips. A **declined** threshold compaction publishes no entry and needs no marker: decline never lands on `checkpoint`. The decline decision stays process-local at `summary.deciding` until its single continuation commit (`assistant.ready`, follow-up + ready, or the terminal after `before_run_end`); a crash in that window restores `deciding` and re-runs `before_compaction` under its documented repetition contract. A live decline/recheck loop is structurally impossible because no decline path re-enters the threshold-checking pass.
**`before_run_end` without hooks on the mutation line.** The finish arm is the one multi-phase case: (1) on-line verdict, no commit — reached only at `may_finish` with no eligible write/steer/followUp; (2) hook runs off the line; (3) on-line **replan from current state** — if the inbox gained eligible items or control changed, the stale hook result is dropped and the planner takes the new decision; otherwise the single commit is the follow-up injection (entry + `assistant.ready`) or the terminal transaction. Stale-dropping is replanning, not a version check.
Concrete control trace (post-R3 names; threshold compaction with `may_finish` resume and a steer arriving mid-compaction):
```text
TX1 settlement: response n7 + usage + tip, S=checkpoint{may_finish, trigger n7}
pass (check): shouldCompact && newestCompaction < n7 → prepare
TX2: preparation, S=summary.deciding{boundary: resume_checkpoint{may_finish, n7}}
TX3 (any time): steer s1 → pendingEntry(s1), laneState.inbox += s1
TX4..n: deciding → ready → effect_pending → nested request/usage settlements
TX-pub (skip): insert compaction c1, tip=c1, insert s1 entry (parent c1),
delete pendingEntry(s1), laneState.inbox -= s1,
S=assistant.ready{trigger s1} ← one commit, checkpoint never re-entered
…turn answers s1; next settlement → checkpoint{may_finish, trigger n9}
pass (check): newestCompaction c1 > … tokens small → no threshold; inbox empty; verdict finish
→ before_run_end off-line → replan: still empty, still running
TX-final: record + cleanup + laneState{current: null, last: op}
```
Crash between TX4..n and TX-pub → structural recovery (attempt unknown, per existing rules); crash during the hook window → `checkpoint{may_finish}` restored, pass re-runs (threshold skipped by the recency guard), hook re-runs per contract.
Slices, in order (the order is load-bearing — deleting the marker while decline still re-entered `checkpoint` would live-loop):
1. **R1b-1**: fuse the future shared planner into the structural `resume_checkpoint` boundary (`threshold: "skip"`); structural success/decline route directly except for a `may_finish` resting checkpoint, which remains safe for either outcome while the durable marker still exists. Keep the route inline until R1b-2 creates its second caller. Both patch fields are still written and checked by the old checkpoint pass — harmless.
2. **R1b-2**: convert the `checkpoint` pass to single-commit exits with the recency guard, implement the three-phase finish arm, delete `skipInboxOnce` and `thresholdCheckedTriggerEntryId`.
### Rules
- The §1 eligibility table, global placement order, and one-decision-one-commit target are normative.
- Queue modes apply at selection time. Remainders preserve global admission order and are consumed only at later eligible boundaries.
- nextRun items are never consumed mid-run and never block run finish.
- Deferred-suspension enqueue always admits; consumption happens at the post-redemption boundary.
### Focused validation
Enqueue during idle/run/structural/cancelled states always admits; acceptance preserves global order with modes applied and remainders retained; `one-at-a-time` cadence gives each remaining steer its own later boundary; followUp is captured at idle acceptance and only at `may_finish` during a run; a boundary crash before its commit consumes nothing; threshold is checked at most once per boundary by construction; idle append places queued writes first; items survive every terminal status and process loss; `cancelQueued` works at every boundary; watch groups by tag without changing order; no terminal transaction deletes a lane-inbox payload; a lone queued write never validates acceptance.
## 7. R2 — Neutral operation outcome and durable result records
### Goal
Replace the three per-family outcome unions and `laneLastResult` with one immutable per-operation result record. The record **is** the public settled outcome; there is no separate outcome type, no embedded entries, and no hydration.
### Types
```ts
type TerminalStatus = "completed" | "declined" | "aborted" | "failed";
/** Stored at operationResult(operationId) — namespace "pi.result" — by the terminal transaction. Immutable, lane-lived. */
interface OperationResultRecord {
operationId: string;
kind: "run" | "compaction" | "navigation"; // meta.intent.kind; matches OperationAdmission.kind
status: TerminalStatus;
error?: OperationError; // status "failed"
fromTipId: string | null; // meta.sourceTipId — start of the transcript segment
tipId: string | null; // lane tip at terminal — end of the segment
startedAt: number; // Unix ms, from meta
endedAt: number; // Unix ms, Date.now() at terminal planning
}
/** Convenience-only suspension observation for prompt()/resume() (M8). Never stored. */
interface SuspendedRun { operationId: string; status: "suspended"; deferred: DeferredHandle }
```
The record is a disposition plus a pointer to the transcript segment `(fromTipId, tipId]`. It never lists intermediate work (compactions, turns, tools) and never embeds entries. There is **no `tipEntry` and no `OperationOutcome` union**: a caller that wants the payload dereferences `tipId` with the already-public `getEntry`/`findEntry` (for runs, the final message was additionally delivered by `message_end`/`entry_added`). This buys three things: terminal planners perform zero outcome reads, observation cannot fault on a missing entry (a `getEntries` throw inside hydration was a harness-fault path for a read-only convenience), and the future protocol never serializes `Entry` inside results — a result frame is eight flat fields. `SuspendedRun` is the one non-terminal observation, exists only on convenience returns, and carries nothing derivable elsewhere.
### Files
- `src/harness/session/types.ts` — add the record; delete the `LaneLastResult` union; `LaneState` gains `lastOperationId: string | null`; delete `failure_drain`, `RunFailureDrainOperation`, and `FailureProvenance`.
- `src/harness/session/values.ts` — `operationResult(operationId) = value<OperationResultRecord>("pi.result", operationId)`; delete `laneLastResult`. The `pi.op.*` lifetime grammar stays total: no operation-lived namespace survives its terminal transaction.
- `src/harness/agent-harness.ts` — delete `RunOutcome`, `CompactionOutcome`, `NavigationOutcome`, `OptionalFinalAssistant`, `TerminalOperationOutcome`, `ResumeOutcome`, and the transitional `OperationOutcome` union; `DriveOutcome.settled` carries `OperationResultRecord` directly (no duplicate `operationId` field); result aliases become
`RunResult = Result<OperationResultRecord | SuspendedRun, …>`,
`CompactionResult = Result<{ compaction: OperationResultRecord; run?: OperationResultRecord | SuspendedRun }, …>`,
`NavigationResult = Result<{ navigation: OperationResultRecord; run?: OperationResultRecord | SuspendedRun }, …>`,
`ResumeResult = Result<OperationResultRecord | SuspendedRun, …>`;
`QueueResult` loses `NoActiveRun` and absorbs `NextRunResult`. `SuspendedRun` is declared here but constructed only by M8 convenience paths.
- End-event payloads, one shape per event type: `run_end` and `navigation_end` carry `{ runId, status, error?, fromTipId, tipId }`. `compaction_end` is a segment event, not a terminal event — it also closes in-run threshold/overflow brackets — and always carries `{ runId, reason, status: "completed" | "declined" | "failed" | "aborted", error?, entryId? }`, where `entryId` names the compaction entry on success. Embedded `entry`/`summaryEntry`/final-assistant fields are dropped everywhere (`entry_added`/`message_end` already delivered payloads).
- `src/harness/runtime/types.ts` — `FinishDecision` carries the record instead of `lastResult`.
- `src/harness/runtime/lane.ts` — terminal suffix per §2; `getLastResult` is replaced by `getResult(operationId, context): Promise<OperationResultRecord | undefined>` — one `getValue`, nothing else; `inspectExecution` reports `lastOperationId` only.
- `src/harness/runtime/drive/terminal.ts` — `hydrateTerminalOutcome`/`hydrateOperationOutcome` are deleted outright; the file keeps only `operationCleanupWrites` and the `operationResultRecord` constructor. Finish decisions materialize `{ kind: "settled", outcome: record }` from the record they already built — no reader dereference inside any terminal planner.
- `src/harness/runtime/drive/{checkpoint,response,structural,recovery,deferred,tools}.ts` — every finish decision constructs the record; `runCompletion`, per-family last-result construction, and final-assistant plumbing in results are deleted. Every former `failure_drain` producer instead commits terminal failure directly: response publication and cleanup stay in the same transaction, in-run structural failure closes `compaction_end` then `run_end`, and queued lane input remains untouched for a later ordinary run. (`latestAssistantEntryId` stays in scope for cancellation classification and checkpoint logic; it just no longer feeds results.)
- `src/harness/runtime/restore.ts` — restore `lastOperationId`; attachment never reads records.
- Focused tests.
### Rules
- Records are written exactly once, by the terminal transaction, and never deleted, updated, or read by recovery.
- Terminal-control invariant (§1): a terminal transaction under durable `cancel_requested` records `aborted`; no path commits any other status under cancellation. M7 adds the enforcing test; M9 adds the Part 9 invariant.
- Forks exclude `pi.result` values.
- `drive(id)` arms: current → install/join; record → returned directly; neither → `OperationMismatch`.
### Focused validation
Record written for every terminal path of every operation kind and status; `drive(id)` returns records for arbitrarily old ids across reopen; `getResult` is a plain value read returning the record or `undefined`; no terminal planner or observation path reads entries for the result; `lastOperationId` restore; suspension never stored; fork exclusion; segment pointers correct (`fromTipId` = pre-acceptance tip, including navigation); `compaction_end` closes both standalone and in-run brackets, including `aborted` via reconciliation; each former failure-drain producer finishes in one transaction and preserves every lane-inbox item for a later run.
## 8. R3 — Family-neutral leaves and structural rebuild
### Goal
Collapse 22 leaves to the 13 in §1; rebuild `structural.ts`'s state-shape layer on one summary quadruple with an explicit result-boundary datum.
### Types
```ts
interface OperationScope { control: Control; settings: RunSettings; latestAssistantEntryId: string | null }
type ResultBoundary = // closed; do not extend
| { kind: "resume_checkpoint"; resumeAfter: CheckpointData } // in-run threshold/overflow
| { kind: "finish" } // standalone compaction
| { kind: "commit_navigation"; targetId: string; label?: string };
interface SummaryTask {
taskId: string;
reason?: "manual" | "threshold" | "overflow";
customInstructions?: string;
boundary: ResultBoundary;
}
```
The summary algorithm kind is **derived from the boundary** (`resume_checkpoint`/`finish` → compaction; `commit_navigation` → branch summary); it is not stored, so no contradictory kind/boundary combination is representable. `summary.ready/effect_pending/retry_wait` additionally carry the generation snapshot (result entry id, configuration, stream options, retry policy, attempt counters); the snapshot drops the `taskId`/`kind`/`reason` fields it previously duplicated. Canonical declarations land in `session/types.ts`. Uniform scope means compaction/navigation acceptance captures `settings` exactly like run acceptance; `latestAssistantEntryId` stays null for structural intents.
### Reachability (restore check; replaces the intent-prefix check)
| Intent | Admissible leaves |
| --- | --- |
| run | `starting`, `checkpoint`, `assistant.*`, `tools`, `deferred.*`, `summary.*` with boundary `resume_checkpoint` |
| compaction | `summary.*` with boundary `finish` |
| navigation | `navigation.ready_to_commit`; `summary.*` with boundary `commit_navigation` |
Forbidden and unreachable by construction (assert in tests, never implement): any edge into `starting` or `navigation.ready_to_commit` other than acceptance; `summary.*` → `tools`/`assistant.*` directly; `navigation.ready_to_commit` → `summary.*`; terminal → anything.
### Result-boundary rule
Every summary boundary (hook decline, hook-supplied result, generated success, terminal generation failure, model unavailability) plans on the mutation line and switches on `boundary` in one visible place:
- `resume_checkpoint` → publish result and restore the marked checkpoint on success or threshold decline; overflow decline or structural failure closes the compaction bracket and terminal-fails the run in the same commit;
- `finish` → publish the compaction entry and terminal-complete (or terminal declined/failed) in one commit;
- `commit_navigation` → the single move/summary/label/terminal commit (or terminal declined/failed).
A `cancel_requested` boundary defers to reconciliation and never takes its continuation.
### Files
- `src/harness/session/types.ts` — the 13-leaf union, `OperationScope`, `SummaryTask`, `ResultBoundary`; delete `RunScope`, `CompactionScope`, `NavigationScope`, `NavigationSummaryScope`, `RunCompactionScope`, `StructuralTask`, per-family `Extract` aliases, `isRunOperationState`.
- `src/harness/runtime/drive/structural.ts` — rebuild: keep `durableCompactionPreparation`/readers, the nested request seam, `performStructuralAttempt`, `runCompactionThreshold`, `prepareOverflowCompaction`, `commitNavigation`; delete the seven union aliases, the `startsWith` guards, and the three-armed `effectPendingFromReady`/`retryWaitFromEffect`/`readyFromRetryWait`/`publishStructuralReady`/`publishStructuralDecline`/`publishStructuralFailure`/`publishCompactionResult`-vs-`publishNavigationSummary` split; one quadruple of converters plus one boundary switch replace them.
- `src/harness/runtime/drive/{checkpoint,generation,response,recovery,deferred,tools,tool-placement,terminal}.ts` — leaf literal renames (`run.checkpoint` → `checkpoint`, …); `response.ts` overflow arm constructs `summary.deciding` with `resume_checkpoint`; no other behavior change.
- `src/harness/runtime/progress.ts` — frame/checkpoint fences currently match the leaf literals `"run.assistant.effect_pending"`, `"run.deferred.effect_pending"`, and `"run.tools"`; retarget them to the neutral leaves.
- `src/harness/runtime/restore.ts` — reachability predicate above.
- `src/harness/runtime/lane.ts` — `capturedModel` and `watch` switch over the neutral leaves; acceptance writes neutral initial leaves.
- `docs/runtime-simplification.md` — replace the 22-leaf list and status.
- `test/harness/runtime/drive-structural.test.ts` and other focused tests — retarget; behavioral assertions survive.
### Focused validation
Everything the M6 foundation covered, re-expressed over neutral leaves, plus: reachability accept/reject matrix (including corrupted boundary/intent combinations faulting restore); each boundary arm × {success, hook result, decline, failure, model absence} × {running, cancelled}; a post-terminal leak scan asserting every `pi.op.*` address is gone for every leaf while the `pi.result` record is present; crash/reopen at every leaf under every intent that can reach it.
## 9. M7 — Cancellation reconciliation and total switch
### Goal
**Implemented and reviewed.** Make the internal graph total without public wiring.
Create `src/harness/runtime/drive/reconcile.ts`, `src/harness/runtime/drive.ts`, and focused reconciliation and switch tests.
### `requestOperationAbort`
Package-private expected-id primitive:
1. reject with `OperationMismatch` when the expected id is not the current durable operation, including when that id already has a settled result record;
2. synchronously `beginAbort` on a matching live Drive;
3. one commit on the Session line: `control = cancel_requested` (with no drained fields) **plus** removal of steer/followUp-tagged ids from the lane inbox and deletion of their `pendingEntry` values; payloads are read in the same mutation and returned; nextRun/write items stay;
4. `signalAbort` after the commit;
5. after a newly requested abort commits, publish `{ type: "operation_abort", operationId, steer, followUp }` and return once cancellation is durable. This family-neutral event replaces `run_abort`/`runId`; the lane snapshot already identifies the operation kind. Repeat calls against the same current cancelled operation publish nothing, drain nothing further, and report `newlyRequested: false`.
With no Drive but a matching current durable operation, it commits the same marker and starts no pass. There is no `control.drained*`; the drained payloads exist only in the returned result (accepted loss: a crash inside the abort window loses their content).
### Reconciliation
Checked before `before_drive` and ordinary dispatch; a single switch over the 13 leaves; never starts new ordinary work. It must handle:
- assistant and deferred effects with live or reconstructed results;
- planned/effect-pending/outcome-ready tool calls;
- structural process-local results (discarded unless already atomically published);
- cancelled summary boundaries finishing `aborted` without taking their `ResultBoundary` continuation;
- retry waits, checkpoints, and suspended deferred work;
- best-effort deferred-provider cancellation using the Drive's close-only signal, never its already-triggered operation-abort gate;
- the aborted terminal transaction (result record `status: "aborted"`);
- the terminal-control invariant test: no terminal path commits a non-aborted status under cancelled control (this keeps §10's continuation rule decidable from the record alone).
Queued lane-inbox items are **not** applied or deleted by reconciliation; nextRun/write items simply remain queued.
Each Drive owns a private close controller and exposes its signal to mandatory cleanup. `closeGate()` aborts that controller on harness close or fault; operation abort does not. Deferred-provider cancellation uses this close-only signal, so the cleanup request can start after durable operation cancellation but cannot outlive harness shutdown. Do not add a harness-global cancellation controller.
### Total switch
`runtime/drive.ts` owns one direct `state.at` switch over 13 leaves. It imports only complete procedure modules. No graph table, action interpreter, ownership-loss arm, external-finalization arm, or storage-state reload. A `continue` result must correspond to a replaced `Lane.state` projection **or** to currently observable `cancel_requested` control, which the switch routes to reconciliation before ordinary dispatch; any other unchanged continuation is an invariant defect.
Public methods remain guarded through M7.
## 10. M8 — Public surfaces
**Implemented.** Execution guards were removed only after every leaf and reconciliation path became total.
Order:
1. admit compaction/navigation requests (acceptance captures `settings`; writes the R3 entry leaves);
2. implement `drive` install/join/record lookup (three arms, §7);
3. expose `requestAbort`;
4. add convenience compositions;
5. add queues/configuration/usage/idle surfaces;
6. retain `watchSession` as the sole `SliceNotImplemented` method.
### Queues
`steer`/`followUp`/`nextRun` are tag sugar over one enqueue and always admit. `queue_update` and `LaneSnapshot.queues` expose the same tagged ordered inbox, including pending writes; clients group by tag without reordering it.
### Client replication surface
Findings from a working RPC-shaped client replica (mini TUI exercise) fold in here. The goal is one testable contract: a remote client renders exclusively from a replicated `LaneSnapshot` and the event stream, with no side-channel getters.
- **Normative reducer.** Export `reduceLaneSnapshot(snapshot, event): LaneSnapshot | { rebase: true }` from `packages/agent`. It is the single supported event fold: in-run `compaction_start`/`compaction_end` are segments and must not clear `operation` (the fold knows the open operation's kind, so no event field is needed); `run_suspend` keeps `operation` non-null with `deferred` set; only `run_end`, `navigation_end`, and `compaction_end` under a compaction-kind operation are operation-terminal. `navigation_end` returns `{ rebase: true }` because the moved tip may lie outside the replica. Conformance assertion: for every non-navigation flow, folding a `watch()` snapshot over its own event stream equals a later `watch()` snapshot — this equivalence test is what keeps the event vocabulary complete.
- **Snapshot completeness.** `LaneSnapshot` gains `configuration: LaneConfiguration`, `lastResult?: OperationResultRecord` (replacing the bare `lastOperationId` field), a real `faulted` flag (the current hardcoded `false` is a defect), and session `stats: SessionStats` as the usage baseline (usage events already carry `totals`; the snapshot supplies the value before the first event).
- **Replicable configuration events.** Every `config_update` variant whose value is data carries `value`/`previous` (`streamOptions`, `retryPolicy`, `compactionSettings`, `steeringMode`, `followUpMode`, plus the existing lane variants); `tools`/`resources` remain notification-only because registries are code — clients re-fetch names.
- **Identity-based `setModel`.** `setModel` accepts `ModelIdentity` (`{ provider, modelId }`); a live `Model` object is a process-local registry concern, and an unregistered identity fails in-band at generation exactly like any registry absence.
- **Re-basing.** `WatchHandle.resnapshot(context): Promise<LaneSnapshot>` captures a fresh snapshot on the mutation line, serialized against the same stream — the recovery path after `{ rebase: true }` with no teardown/re-subscribe choreography.
- **Start-event timestamps.** `run_start`/`compaction_start`/`navigation_start` carry `startedAt` (operation starts: `meta.startedAt`; in-run segment starts: the commit timestamp), so folds never invent times.
### Convenience compositions and continuation
- `prompt`/`skill`/`promptFromTemplate`: accept + drive; return `Result<OperationResultRecord | SuspendedRun, …>` — the record for terminal outcomes, `SuspendedRun` when the run defers.
- `compact`/`navigateTree`: accept A + drive A. If A settled with status `completed`, `declined`, or `failed` and eligible steer/followUp/nextRun items exist, accept a continuation run B with the **public empty-prompt request** (`{ kind: "prompt", prompt: "" }`): acceptance places no request messages and is legal exactly when its capture places at least one conversational item; `OperationMeta.intent` is an ordinary run intent with empty `promptEntryIds`. Then drive B and return `{ compaction|navigation: A, run?: B }`. Never after `aborted`: abort already drained steer/followUp, and the terminal-control invariant (§7) makes `status ∈ {completed, declined, failed}` imply the operation was never durably cancelled, so the rule is decidable from the record alone. B is an ordinary run — it emits `run_start`, runs `before_run` (with `prompt: []`, the existing captured-only acceptance shape), and owns the full run graph with no continuation-aware special case.
- Continuation races: a competing accept that wins the idle window captures the queued input into its own run; the continuation accept then fails empty (`InvalidMessage`) or `LaneBusy`, and the convenience returns A without `run`. Both histories are valid. A crash between A's terminal and B's acceptance leaves the items queued; no auto-start on reopen.
- `resume`: inspect + drive current id, one deferred-poll permit; returns `OperationResultRecord | SuspendedRun`.
- `abort`: inspect + `requestAbort` + ensure a reconciliation pass; returns the drained steer/followUp payloads.
- `getResult(operationId)`: public.
- Equivalence: every convenience call ≡ its primitive composition using only public request kinds (`compact()` with continuation ≡ `accept(A); drive(A); accept({ kind: "prompt", prompt: "" }); drive(B)`), byte-identical writes and events — externally reproducible by any scheduler.
### Focused validation
One install and same-id joins; stale-id isolation; record lookup for old ids; caller cancellation before/after installation; close/fault during cooperative and non-cooperative effects; accept/drive vs convenience equivalence including continuation via the public empty-prompt request; steer-only, followUp-only, and nextRun-only continuation each start run B; both continuation race orders; abort drain-and-return including the both-orders abort/settlement race; enqueue during every state through public surfaces; full crash matrix across all 13 leaves; no unhandled detached rejection; no `SliceNotImplemented` except `watchSession`; reducer fold-equivalence across every non-navigation flow (including in-run compaction segments, suspend/resume, retry waits, and reattach mid-stream); `navigation_end` fold returns rebase and `resnapshot` restores equivalence; snapshot `configuration`/`lastResult`/`faulted`/`stats` replicate through events alone.
## 11. M9 — Documentation reconciliation
**Implemented.** `docs/harness.md` is reconciled to the runtime and normative on its own. Updated sections include:
- §1.3 address table and lifetime grammar: new lane-lived `pi.result` namespace, `laneLastResult` removed, `LaneState.inbox`/`lastOperationId`; `pi.op.*` stays strictly operation-lived;
- §1.7 JSONL/SQLite examples referencing `pi.lane.lastResult`; state the bounded growth tradeoff — one small immutable record per operation, retained forever, carried through JSONL snapshot compaction;
- §2.9 precise rewrite: decide and document the policy for records whose `tipId` the rewrite removes (retain-dangling or delete);
- §3.1–§3.2 state shapes: 13 neutral leaves, `OperationScope`, `SummaryTask`/`ResultBoundary` (kind derived from boundary), `Control` without drained fields, `CheckpointData` without `skipInboxOnce`/`thresholdCheckedTriggerEntryId`;
- §3.5 graph (one summary quadruple, boundary arms);
- §3.6 acceptance: per-tag mode-respecting selection, global admission-order placement, prompt entries last, idle followUp eligibility, and empty-prompt continuation acceptance;
- §3.9/§3.10 summary machinery expressed once over the boundary datum;
- §3.11 complete rewrite: one lane inbox, the selection table, one-decision-one-commit boundaries, silent deferral of late steer (an unconsumed steer becomes future-run input rather than an error), and unbounded write pendency during structural operations with the `waitForIdle`-then-append escape hatch;
- §3.12 checkpoint procedure: the one-commit boundary decision replaces the stepwise algorithm; failures terminalize while queued lane input remains for a later ordinary run;
- §3.13 terminal transactions: result records, universal suffix, observation contract (`drive(id)` total, `getResult`);
- §4.6 abort: drain-and-return, no drained control, and the client-visible consequence — drained steer/followUp payloads exist only in the returned result, so a crash or lost response in that window loses their content permanently;
- §5.1 lane surface and all result types; §5.5 `queue_update` and end-event payloads (`compaction_end` as a segment event with `aborted`);
- Part 9: invariants 12–16, 21, 26 (result-record lifetime, observation via records, continuation equivalence), the new terminal-control invariant (a non-aborted terminal status implies running terminal control), and the race catalog rows touching drained items, `cancelQueued`, and continuation;
- Appendix A glossary (Inbox, Result record, Continuation run, Boundary pass; remove Drained);
- §5.5 event taxonomy: state explicitly which events are operation-terminal (`run_end`, `navigation_end`, compaction-kind `compaction_end`) versus segment brackets (in-run `compaction_start`/`compaction_end`) versus non-terminal lifecycle (`run_suspend` leaves the operation durably open), and name `reduceLaneSnapshot` as the normative fold;
- §2.5 branch-scan sharp edge: `stopAtType` applies after ordering, so `oldestFirst` + `stopAtType: "compaction"` returns the oldest segment; document the canonical context read as `newestFirst` + reverse;
- serving-layer boundary statement: branch-relative reads/appends remain on `AgentLane`, while whole-tree browsing, forks, label inventory, and session listing live beside it in Session/repository services composed by an RPC facade.
Also update `docs/runtime-simplification.md` status and remove every remaining promotion reference in the repository's docs. Grep gate:
```bash
rg -i 'promotion|drainedSteer|drainedFollowUp|laneLastResult|lastResult|skipInboxOnce|thresholdChecked' packages/agent/src packages/agent/docs
```
(matches must be zero in `src`; docs may retain historical notes only inside completed-milestone records.)
## 12. M10 — Provider KV-cache identity review
### Goal
**Implemented and reviewed.** Core Session identity alone is not a valid provider cache lineage: several lanes may issue concurrent requests over divergent transcripts. Ordinary assistant requests therefore derive identity from Session metadata id plus lane name; structural requests remain isolated.
### Review scope
- Inventory every harness and pi-ai path that sets, preserves, derives, or consumes `SimpleStreamOptions.sessionId`, including prompt-cache keys, affinity headers, WebSocket/session-resource caches, deferred requests, and structural summary requests. Ordinary harness generation now forwards the derived lane identity; legacy `src/agent.ts` keeps its independent conversation id; structural summary requests mint fresh ids.
- Define a provider-request cache lineage distinct from the durable Session id and operation id. Concurrent lanes must never share one lineage merely because they belong to the same Session.
- Derive ordinary assistant lineage as `Session metadata id + ":" + lane name`; do not store another durable identifier. It remains stable for the lane's lifetime and differs across lanes in one Session.
- Preserve same-lane reuse across ordinary assistant turns and retries. Compaction, navigation, branch replacement, and model changes may miss an old prefix, but cannot incorrectly reuse it; do not add rotation machinery.
- Keep structural summary requests isolated: `cacheRetention: "none"` and fresh request identity per nested request remain the baseline unless the review proves a safe, useful alternative.
- Distinguish cache identity from observability correlation. Session, lane, operation, task, and request ids remain available to telemetry without being reused blindly as provider affinity/cache keys.
- Review provider-specific semantics rather than assuming `sessionId` means only KV caching; some adapters also use it for headers, WebSocket reuse, fallback state, or resource cleanup.
### Focused validation
- Two active lanes in one Session issue concurrent, divergent prompts and receive distinct provider cache/affinity identities.
- One lane keeps a stable lineage across append-only assistant/tool turns where reuse is valid.
- Context-prefix discontinuities safely miss old cached prefixes without lineage rotation.
- Assistant retries retain the lane identity; deferred handle polling needs no cache identity.
- Structural split-turn requests remain isolated from transcript assistant caching.
- Faux-provider tests assert stable same-lane and distinct cross-lane request identities.
### Exit condition
Ordinary assistant requests use the derived lane identity, structural requests keep fresh identities with `cacheRetention: "none"`, and deferred handle polling sends no cache identity. The policy is documented in `docs/harness.md` and independently reviewed before WP05 completion.
## 13. Mobile assistant-output handoff
**Tracked follow-up; not part of the M8/M10 public-drive gate.** A trivial mini coding-agent Session produced an approximately 300 KB JSONL file because live assistant streaming persists many `pi.pending.assistant_frame` list appends. Logical frame cleanup does not reclaim bytes already appended to the JSONL history, so short conversations can have disproportionate durable storage and replay cost.
The authoritative follow-up is the [mobile assistant-output handoff](../mobile-handoff/01-harness/05-assistant-output/message-update.md) and its numbered prerequisites in the [mobile handoff README](../mobile-handoff/README.md). It addresses the complete path rather than only batching frames: Chord op tracking, scoped pending-output durability, tool/assistant output reduction, and `message_update` replication amplification. Preserve the existing crash contract: an admitted assistant effect remains reconstructible, progress observation remains useful, and settlement retires operation-owned pending state.
Exit checks:
- representative short streamed responses do not create hundreds of kilobytes of durable frame history;
- crash/reopen at every assistant and deferred effect boundary reconstructs the same pending message;
- frame/checkpoint writers remain fenced to the owning durable effect;
- backend conformance covers both reconstruction and bounded write amplification.
## 14. Module boundaries
```text
session/** durable storage; imports no runtime module
execution/** neutral provider/tool/gate mechanics
runtime/types.ts LaneState, Drive, command decisions
runtime/progress.ts frame/tool progress channels
runtime/drive/*.ts direct procedures
runtime/drive.ts total flat switch, created only in M7
runtime/lane.ts Lane actor and public surfaces
runtime/harness.ts Harness lifecycle and Lane composition
```
Procedure modules import concrete `Lane<TContext>` type-only. `TContext` remains `object | undefined` invariant. No `any`, `Lane<any>`, `as unknown as`, `@ts-expect-error`, inline imports, parameter properties, enums, or other non-erasable TypeScript syntax.
## 15. Exclusions
Do not introduce:
- generic scheduler, graph, action interpreter, or effect-plan DSL — `ResultBoundary` stays a closed three-arm data union switched in one visible place;
- per-family outcome unions, `OptionalFinalAssistant`, or embedded result entries in stored records;
- result-record listing, filtering, pagination, status queries, or retention machinery;
- operation-owned queues, drained-control fields, or terminal queue cleanup;
- automatic continuation anywhere below the convenience layer (never inside `drive`);
- a second mutation line or transaction framework;
- expected-`at` runtime checks for the sole top-level writer;
- process-local Drive replacement or caller-owned lifetime;
- external finalization;
- storage rereads of authoritative control state;
- read caches, read budgets, or generic `getValues` batching;
- compatibility aliases for any pre-redesign durable shape;
- a `phase`/segment discriminator field on structural events — the reducer derives segment-vs-terminal from the open operation's kind;
- whole-tree, fork, label-inventory, or repository methods on `AgentLane`; branch-relative reads/appends remain part of the lane facade, while broader administration lives beside it.
Procedure-specific writes, effect admission, settlement classification, and event construction remain visible at their call sites.
## 16. Validation and reviews
After every code stage:
```bash
npm run check
```
Run each modified focused test file from the package root. Do not invoke the full Vitest suite directly. Run `./test.sh` only for final package validation or when explicitly requested.
Review checkpoints are mandatory at the end of R3, R1b, M7, M10, and final completion. Delegated reviews use provider `anthropic` and model `claude-fable-5`.
Final greps:
```bash
rg 'lost_ownership|LostOwnership|DriveAbandoned|commandDriveOwned|installerSignal|finalizedOutcome|OperationEnded' \
packages/agent/src/harness packages/agent/test/harness
rg 'drainedSteer|drainedFollowUp|laneLastResult|RunOutcome|CompactionOutcome|NavigationOutcome|TerminalOperationOutcome|skipInboxOnce|thresholdCheckedTriggerEntryId' \
packages/agent/src/harness
rg 'SliceNotImplemented' packages/agent/src/harness/runtime
rg ': any\b|<any>|as unknown as|@ts-expect-error' packages/agent/src/harness/runtime
```
Final exit conditions:
- every flat leaf is driveable and reconcilable;
- public primitive/convenience behavior is equivalent, including continuation;
- every focused test and backend conformance path passes;
- public drive exposes no partial graph;
- `watchSession` is the only deferred public method;
- `drive(id)` answers every settled operation id in the session;
- harness.md is self-consistent with the implementation (M9);
- provider cache/affinity identity is lane-safe across concurrent and context-reset histories (M10);
- independent final review reports no blocker.

View file

@ -0,0 +1,734 @@
# WP06 — Session, Branch, Lane separation
**Status: implemented before WP05 M4.** WP05 M3 remains complete. Retry/deferred work resumes on the composition, mutation, and ownership boundaries landed here.
This package replaces the mixed `SessionTree`/implicit-main inheritance design with four explicit concepts:
```text
Session global durable data + one mutation line
Branch one path through the entry tree, with a movable tip
AgentLane Branch data surface + agent operations/configuration
AgentHarness manager of AgentLanes; never a lane itself
```
---
## 0. Mandatory reading
Read completely before editing:
1. `packages/agent/docs/harness.md`.
2. `packages/agent/docs/work-packages/05-direct-durable-drive.md`.
3. `packages/agent/src/harness/session/types.ts`.
4. `packages/agent/src/harness/session/session.ts`.
5. `packages/agent/src/harness/session/memory.ts`.
6. `packages/agent/src/harness/session/jsonl/repo.ts` and `jsonl/storage.ts`.
7. `packages/session-backends/sqlite-node/src/sqlite/session.ts`, `storage.ts`, and repo implementation.
8. `packages/agent/src/harness/runtime/lane.ts`, `harness.ts`, `restore.ts`, and `types.ts`.
9. `packages/agent/src/harness/agent-harness.ts`.
10. `packages/agent/src/harness/session/fork.ts` and repository conformance tests.
11. Every test named in §8 before modifying it.
Do not inspect deleted runtime implementations or Git history. Current source, `harness.md`, WP05, and this package are the only sources of truth.
---
## 1. Problem
### 1.1 `SessionTree` mixes unrelated ownership
`SessionTree` currently contains:
```ts
interface SessionTree {
// Lane/path data.
getLeafId(...): Promise<string | null>;
findEntriesOnBranch(...): Promise<Entry[]>;
findEntryOnBranch(...): Promise<Entry | undefined>;
appendMessage(...): Promise<string>;
appendCustomEntry(...): Promise<string>;
// Session-global data that is not lane- or branch-owned.
getEntry(...): Promise<Entry | undefined>;
getStats(...): Promise<SessionStats>;
findEntries(...): Promise<Entry[]>;
findEntry(...): Promise<Entry | undefined>;
getValue(...): Promise<StoredValue<unknown> | undefined>;
setValue(...): Promise<void>;
readList(...): Promise<ListElement<unknown>[]>;
appendList(...): Promise<void>;
getName(...): Promise<string | undefined>;
setName(...): Promise<void>;
getLabel(...): Promise<string | undefined>;
setLabel(...): Promise<void>;
}
```
A selected tree view changes only which mutation queue a global write enters. It does not change the durable address. Two views can therefore read the same global value under different lane lines and commit a lost update.
### 1.2 `Session` silently means `main`
`Session extends SessionTree`. Its inherited branch methods and high-level writes silently delegate to the `main` lane. `session.setValue(...)` is actually `setValueForLane("main", ...)`. A fresh repository session creates a partial implicit main lane before any harness exists.
### 1.3 `AgentHarness` silently means `main`
`AgentHarness extends AgentLane`, and the runtime `Harness extends Lane`. The manager object is inserted into its own lane map as `main`. Calls such as `harness.prompt(...)`, `harness.watch(...)`, or `harness.getModel(...)` silently target main.
### 1.4 Per-lane mutation queues solve the wrong problem
Storage already serializes atomic commits and assigns one session-wide `seq` per write. Per-lane mutation queues allow useful preparation overlap, but this runtime does not need that complexity now: all harness mutation callbacks perform bounded storage reads, prepare one write set, commit at most once, publish process-local state, and return. Providers, tools, hooks, timers, and asynchronous event delivery are outside mutation callbacks.
The approved first implementation uses one Session mutation line. Keyed lines may be added later if profiling proves the global line is a bottleneck; that future change requires a mutable-ownership audit but no public API redesign.
---
## 2. Approved terminology and ownership
### Session
Owns:
- session metadata;
- global entry and usage queries;
- application values and lists;
- session name and entry labels;
- Branch discovery/creation;
- one process-local mutation line for every supported mutation;
- one open storage/backend lifecycle.
A Session does not implement Branch and has no implicit branch.
### Branch
Data-only capability describing one named path through the immutable entry tree.
Owns only:
- its current tip;
- branch-relative entry queries;
- direct extension of its tip with message or custom entries.
A Branch has no model configuration, queues, operation state, drive, hooks, or agent policy.
### AgentLane
One Branch plus agent configuration and operations. It exposes Branch methods directly rather than exposing a nested `Branch`, tree, store, view, or access object.
When idle, `AgentLane.appendMessage` / `appendCustomEntry` extend the tip directly. During an active run, they retain the existing deferred-write semantics: reserve the entry id, persist `pendingEntry(id)`, and enqueue the id in the operation inbox for checkpoint placement. A raw Branch append is always a direct data append; raw Branch mutation while a Harness owns the corresponding lane is a trusted-programming defect.
### AgentHarness
Owns global registries/configuration, hooks, events, lifecycle, and a map of AgentLanes. It is not an AgentLane and exposes no implicit-main operation methods.
---
## 3. Target public types
### 3.1 Session reader and mutation
```ts
export interface SessionReader {
getEntries(ids: string[], context: Context): Promise<Map<string, Entry>>;
getValue<T>(
address: Value<T>,
context: Context,
): Promise<StoredValue<T> | undefined>;
scanValues<T>(prefix: Value<T>, context: Context): Promise<StoredValue<T>[]>;
readList<T>(
address: ValueList<T>,
options: ListReadOptions | undefined,
context: Context,
): Promise<ListElement<T>[]>;
scanBranch(query: StorageBranchScan, context: Context): Promise<Entry[]>;
}
export interface SessionMutation extends SessionReader {
/** Exactly zero or one attempt. A second call rejects, including after failure. */
commit(writes: Write[], context: Context): Promise<CommitResult>;
/** Wait for any admitted commit, invalidate the capability, and release the Session line. */
end(context: Context): Promise<void>;
}
export type SessionMutator = Omit<SessionMutation, "end">;
export type SessionMutationCallback<TResult> = (
mutator: SessionMutator,
context: Context,
) => TResult | Promise<TResult>;
```
`beginMutation()`/`SessionMutation.end()` remain the transportable scope used by the current remote Session protocol. They are keyless and carry no lane field. Normal local/harness code uses `mutate()`; no caller selects a mutation key.
### 3.2 Branch
```ts
export interface Branch {
readonly name: string;
getTipId(context: Context): Promise<string | null>;
findEntries(
query: BranchScan | undefined,
context: Context,
): Promise<Entry[]>;
findEntry(
query: BranchScan | undefined,
context: Context,
): Promise<Entry | undefined>;
appendMessage(message: AgentMessage, context: Context): Promise<string>;
appendCustomEntry(
customType: string,
data: JsonValue | undefined,
context: Context,
): Promise<string>;
}
```
Because the receiver is already a Branch, the public names are `findEntries` and `findEntry`, not `findEntriesOnBranch` and `findEntryOnBranch`.
### 3.3 Session
```ts
export interface Session<
TMetadata extends SessionMetadata = SessionMetadata,
> extends SessionReader {
readonly metadata: TMetadata;
readonly idGenerator: IdGenerator;
// Direct reads. No mutation-line acquisition.
getEntry(id: string, context: Context): Promise<Entry | undefined>;
getStats(context: Context): Promise<SessionStats>;
findEntries(
query: EntryQuery | undefined,
context: Context,
): Promise<Entry[]>;
findEntry(
query: EntryQuery | undefined,
context: Context,
): Promise<Entry | undefined>;
getName(context: Context): Promise<string | undefined>;
getLabel(targetId: string, context: Context): Promise<string | undefined>;
// Existing Branch acquisition performs durable I/O and therefore receives Context.
branch(name: string, context: Context): Promise<Branch | undefined>;
createBranch(
name: string,
at: string | null,
context: Context,
): Promise<Branch>;
// Transportable explicit scope; RemoteSession maps begin/read/commit/end over RPC.
beginMutation(context: Context): Promise<SessionMutation>;
// Trusted sharp edge. The callback holds the sole Session mutation line.
mutate<TResult>(
mutation: SessionMutationCallback<TResult>,
context: Context,
): Promise<TResult>;
// One-write conveniences implemented through mutate().
setValue<T>(
address: Value<T>,
next: NoInfer<T>,
context: Context,
): Promise<void>;
deleteValue<T>(address: Value<T>, context: Context): Promise<void>;
appendList<T>(
address: ValueList<T>,
element: NoInfer<T>,
context: Context,
): Promise<void>;
deleteList<T>(address: ValueList<T>, context: Context): Promise<void>;
setName(name: string | undefined, context: Context): Promise<void>;
setLabel(
targetId: string,
label: string | undefined,
context: Context,
): Promise<void>;
close(context: Context): Promise<void>;
}
```
`mutate()` remains public. Plugins are trusted not to retain the mutator, invoke nested public writers, perform effects, or hold the line across unbounded work. Misuse may block every mutation in that Session and is a plugin defect. `beginMutation()` exists for transport/lifecycle integration rather than ordinary plugin work; every direct caller must call `end()` in `finally`.
### 3.4 AgentLane
`AgentLane` keeps its operation/configuration/observation methods and directly adds the Branch surface:
```ts
export interface AgentLane {
readonly name: string;
getTipId(context: Context): Promise<string | null>;
findEntries(
query: BranchScan | undefined,
context: Context,
): Promise<Entry[]>;
findEntry(
query: BranchScan | undefined,
context: Context,
): Promise<Entry | undefined>;
appendMessage(message: AgentMessage, context: Context): Promise<string>;
appendCustomEntry(
customType: string,
data: JsonValue | undefined,
context: Context,
): Promise<string>;
getLastResult(context: Context): Promise<LaneLastResult | undefined>;
accept(
request: OperationRequest,
context: Context,
): Promise<OperationAdmissionResult>;
drive(options: DriveOptions, context: Context): Promise<DriveResult>;
requestAbort(
operationId: string,
context: Context,
): Promise<AbortRequestResult>;
inspectExecution(context: Context): Promise<LaneExecutionInfo>;
// Existing convenience, queue, configuration, idle, and watch methods remain.
}
```
Delete `AgentLane.sessionTree`.
### 3.5 AgentHarness
```ts
export interface AcquireLaneOptions {
/** Used only when the AgentLane does not exist. Defaults to null. */
createAt?: string | null;
}
export interface AgentHarness<
TContext extends object | undefined = object | undefined,
> {
lane(name: string, context: Context): Promise<AgentLane>;
lane(
name: string,
options: AcquireLaneOptions,
context: Context,
): Promise<AgentLane>;
lanes(context: Context): Promise<LaneInfo[]>;
// Session-global metadata wrappers preserve existing value_update events.
getName(context: Context): Promise<string | undefined>;
setName(name: string | undefined, context: Context): Promise<void>;
getLabel(targetId: string, context: Context): Promise<string | undefined>;
setLabel(
targetId: string,
label: string | undefined,
context: Context,
): Promise<void>;
// Existing global tools/resources/options/settings/hooks/events/watchSession/close surface.
}
```
`AgentHarness` does not extend `AgentLane`. Delete `createLane`; `lane()` is atomic get-or-create. Existing lanes ignore `createAt`. A missing lane uses `createAt ?? null`. Concurrent acquisitions return the same published AgentLane. Invalid names and unknown non-null targets reject with the existing tagged errors; close/fault reject with their existing lifecycle errors.
A fresh Session and fresh Harness contain no implicit main Branch or AgentLane. `await harness.lane("main", context)` creates main completely. `lanes()` may return `[]`.
---
## 4. Mutation and read semantics
### 4.1 One Session mutation line
Replace `LaneMutationLine` with a single `MutationLine`:
```ts
export class MutationLine {
private tail: Promise<void> = Promise.resolve();
private sealedError: Error | undefined;
run<TResult>(operation: () => TResult | Promise<TResult>): Promise<TResult>;
seal(error: Error): Promise<void>;
}
```
`StorageBackedSession.beginMutation()` acquires that line and returns one explicit keyless capability; only `end()` releases it. `mutate()` is the callback convenience built from begin/end and always ends in `finally`. The callback may read, prepare, commit once, publish process-local state, synchronously bind event recipients, and return. `close()` seals admission and waits for every acquired scope to end before closing Storage.
High-level Session writes, Branch creation/appends, every `Lane.command`, progress writes, restore snapshots requiring coherence, and Harness lane acquisition all call the same `Session.mutate()`.
### 4.2 Reads bypass the line
Ordinary Session and Branch reads call Storage directly. They observe the latest fully applied atomic storage commit at the time each read executes:
- a queued/planning/unapplied mutation is invisible;
- no partial commit is visible;
- once Storage commit resolves, direct reads may observe it even while the mutation callback is still publishing process-local state;
- multiple reads outside one `mutate()` are not a snapshot;
- coherent read-decide-write/CAS uses `Session.mutate()`.
### 4.3 Effects stay outside
A mutation callback must not perform or await:
- providers or deferred fetch/cancel;
- tools;
- hooks;
- timers;
- asynchronous event delivery;
- idle callbacks or Drive completion;
- nested Session/Branch/AgentLane mutators.
The callback may synchronously call `emitBatch` after publication to bind recipients and retain its delivery promise. The public operation awaits delivery after `mutate()` returns.
### 4.4 Storage remains independently atomic
Storage retains its one-session commit serializer and assigns one global `seq` per write. The Session mutation line protects read-decide-commit-publication procedures; the Storage queue protects atomic transaction application, sequence assignment, fork snapshots, and backend callers. Do not merge the two abstractions.
---
## 5. Branch and lane durable shape
### 5.1 No implicit main
Repository `create()` writes only session metadata/header/catalog state. It writes no branch tip, lane configuration, or lane state. Remove main seeding from Memory and JSONL creation and from SQLite initialization.
Legacy coding-agent v3 normalization may still produce a main Branch because the imported transcript has one selected path.
### 5.2 Branch completeness
A Branch exists exactly when its required tip value exists. `createBranch(name, at, context)` validates name, absence, and non-null target, then writes the tip in one mutation. It writes no model configuration or operation state.
Use Branch terminology in source. This package renames the typed constructor and public concepts to `branchTip`/`tipId`. The persisted namespace and durable field spelling decision must be consistent across all backends and docs:
- use `pi.branch.tip` and rename format-4 fields from leaf to tip rather than retain misleading new-code aliases;
- format 4 and the new harness are WIP, so replace their schema and field names in place: no storage-version bump, migration, compatibility decoder, or old-format rejection path;
- legacy coding-agent v3 import remains supported: it maps its selected main leaf to a main Branch tip and walks that selected physical ancestry to reconstruct the nearest `model_change`, `thinking_level_change`, and `active_tools_change` independently; unsupported nearest values do not fall back to older history;
- when the importer can reconstruct a total configuration, it writes ordinary `laneConfig("main")` plus fresh idle `laneState("main")` before returning the Session; missing active-tools history normalizes to `[]` because v3 did not persist the initial tool inventory;
- if required model or thinking configuration is missing or unsupported, the importer leaves a data-only main Branch rather than persisting partial compatibility state;
- update the legacy active-tools record type to read its encoded `activeToolNames` array;
- update protocol schemas and experimental adapters for public `tipId` fields in the same package.
Inventory uses `branchTipInventoryPrefix()`. Legacy import emits only ordinary Branch/Lane values; there is no temporary compatibility address or attachment-time migration.
### 5.3 AgentLane completeness
An AgentLane adds total `laneConfig`, `laneState`, optional `laneLastResult`, and optional current operation values to an existing Branch.
`harness.lane(name, options?, context)` executes one Session mutation and handles exactly these cases:
| Durable state | Result |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Branch absent; lane values absent | validate `createAt`, commit Branch tip + the immutable `AgentHarnessOptions` seed config + idle lane state, publish one AgentLane and `lane_created { at: createAt ?? null }` |
| Branch present; lane config/state absent and no last result | commit the immutable seed config + idle lane state at the existing tip, publish one AgentLane and `lane_created { at: existingTip }` |
| Branch and complete lane values present | return the restored/published AgentLane; no commit/event |
| Any partial or contradictory combination | fault as storage corruption |
The committing callback publishes the new Branch/AgentLane into process-local maps and synchronously calls `emitBatch(lane_created, context)` before returning from `Session.mutate()`. Event delivery is awaited after line release.
`AgentHarness.create()` restores every complete durable AgentLane and open operation, but creates nothing and requires no main. A data-only Branch remains a Branch until `harness.lane(name, ...)` attaches agent state.
### 5.4 Append behavior
`Branch.appendMessage` / `appendCustomEntry` always commit an immutable entry parented to the current tip and move the tip in the same transaction.
`AgentLane.appendMessage` / `appendCustomEntry` retain harness semantics:
- idle: append and move tip immediately;
- active run: stage `pendingEntry(id)` and enqueue `inbox.writes`;
- active structural operation: retain the existing wait/re-evaluate contract when that surface lands;
- pending assistant messages reject before commit.
Both return the entry id reserved before their mutation.
---
## 6. Runtime composition
### 6.1 Harness
Replace inheritance with composition:
```ts
export class Harness<
TContext extends object | undefined,
> implements AgentHarness<TContext> {
readonly session: Session;
readonly models: Models;
readonly hooks: HookRegistry;
readonly events: HarnessEventBus;
readonly lanesByName = new Map<string, Lane<TContext>>();
// global config/lifecycle fields
}
```
The constructor builds every restored Lane as an ordinary object. It never calls `super("main", ...)` and never inserts `this` into `lanesByName`.
Fault and close iterate ordinary Lane objects. Global getters/setters live only on Harness. Coding-agent experimental services and workers must acquire/cache `main` explicitly before invoking lane operations.
### 6.2 Lane
`Lane.command` and `commandDriveOwned` call keyless `session.mutate(plan, context)`. The live `Lane.state` remains the authoritative process projection. Exact Drive fencing remains adjacent to commit admission.
Lane directly implements Branch query/append names. It may hold a package-private Branch implementation for direct data reads, but no nested Branch is exposed publicly.
### 6.3 Restore
Restore no longer enters a named mutation line. `AgentHarness.create()` owns the Session attachment interval and performs one bounded keyless `Session.mutate()` callback to inventory/restore every complete AgentLane before publishing the Harness; it commits only when an intentional attachment normalization is required. Coherent live watch/inspection likewise uses `Session.mutate()` as a no-commit callback.
Restore inventories complete AgentLanes separately from data-only Branches. Missing main is legal.
---
## 7. Repository, backend, and fork requirements
### Memory/JSONL/SQLite
All Session facades:
- remove lane arguments and lane fields from begin/mutate capabilities;
- retain explicit `beginMutation(context)` / `SessionMutation.end(context)` forwarding for local lifecycle and remote transport;
- keep an explicit scope admitted until `end()` and keep callback `mutate()` admitted through its implicit finally/end;
- expose Branch acquisition/creation;
- keep direct reads outside the line;
- keep Storage commit sequencing unchanged.
SQLite and future SQL backends still serialize sequence-allocating commits in Storage. The Session mutation line intentionally serializes complete callbacks only within one open Session owner; separate sessions remain concurrent.
### Forks
Preserve one coherent source Storage snapshot. Update terms from lane leaf to Branch tip.
- branch-scope fork creates destination Branch `main` at the copied path;
- if no explicit source entry is provided, source `main` must exist or fork rejects;
- branch scope copies source main's configuration and writes idle lane state **together iff** source main is a complete configured AgentLane; an unconfigured/data-only source main produces only a data-only destination main Branch;
- tree scope copies every Branch tip; each complete configured source AgentLane copies its configuration plus fresh idle lane state together, while each data-only Branch remains data-only;
- operation values, pending values/lists, last results, and usage rows remain excluded;
- destination sessions do not gain an unrelated implicit main;
- retain the explicit begin/commit/end fork-ordering seam: code that must admit a commit before starting a fork snapshot calls keyless `beginMutation()`, invokes `commit()`, starts the repository snapshot only after commit admission, and calls `end()` in `finally`; Storage queues the source snapshot against commits to choose one coherent boundary.
---
## 8. Implementation phases and file manifest
Public drive remains disabled. Complete this package before resuming WP05 M4.
### Phase A — Session mutation and Branch
**Rename**
- `src/harness/session/lane-mutations.ts` → `mutation-line.ts`.
**Modify**
- `src/harness/session/types.ts` — `Branch`, keyless `Session.mutate` and keyless begin/end transport scope, no `SessionTree`, Session-global methods.
- `src/harness/session/session.ts` — one line, Branch implementation, no implicit-main delegates, direct Branch append; rename lane-creation validation errors to Branch terminology.
- Memory/JSONL/SQLite format-4 schema and codecs — replace WIP leaf/lane spellings in place; do not add a version gate or migration.
- `src/harness/session/memory.ts`.
- `src/harness/session/jsonl/repo.ts`, `jsonl/legacy-v3.ts`, and related open/create facade files.
- `packages/session-backends/sqlite-node/src/sqlite/session.ts` and repo creation.
- `src/harness/session/fork.ts`.
- `src/harness/session/index.ts` and package exports.
- storage/repository benchmarks and conformance call sites.
**Rename tests**
- `test/harness/session-tree.test.ts` → `branch.test.ts`.
- `test/harness/session-create-lane.test.ts` → `session-create-branch.test.ts`.
**Modify tests**
- `test/harness/storage-backed-session.test.ts`.
- `test/harness/memory-session-repo.test.ts`.
- `test/harness/jsonl-session-repo.test.ts`.
- `test/harness/memory-conformance.test.ts`.
- `src/harness/session/testing/conformance/session-repo.ts`.
- SQLite repo/storage tests.
- compaction/branch-summarization type fixtures.
### Phase B — Harness/Lane composition
**Modify**
- `src/harness/agent-harness.ts`.
- `src/harness/runtime/harness.ts`.
- `src/harness/runtime/lane.ts`.
- `src/harness/runtime/restore.ts`.
- `src/harness/runtime/types.ts` where `leafId` becomes `tipId`.
- `src/harness/session/values.ts` and durable state types for Branch tip naming.
- all runtime drive modules already present (`checkpoint`, `generation`, `recovery`, `terminal`, `progress`) only where names/signatures change.
- `src/harness/compaction/branch-summarization.ts`.
- `packages/protocol/src/harness.ts`.
- `packages/coding-agent/src/experimental/services/agent-controller-provider.ts`.
- `packages/coding-agent/src/experimental/services/models-provider.ts`.
- `packages/coding-agent/src/experimental/session-worker.ts`.
- experimental harness wire/session worker tests.
**Modify focused tests**
- every `test/harness/runtime/*.test.ts` helper/call site using keyed mutate, implicit Harness-as-main, `sessionTree`, `leafId`, or `createLane`;
- `test/harness/types.test.ts`;
- `test/harness/branch-summarization.test.ts`;
- protocol and coding-agent experimental tests.
Phases A and B are one atomic landing. Removing `SessionTree`, keyed mutate, and Harness inheritance cannot compile as separately committed compatibility phases, and this package intentionally adds no temporary aliases.
### Phase C — Normative documentation
Update `packages/agent/docs/harness.md` completely and consistently:
- orientation/system model and worked examples;
- bound Branch tip addresses;
- Branch, Session metadata, queries, forks, and repository boundary;
- operation metadata/result/snapshot `tipId` terminology;
- one Session mutation line in Parts 3–5;
- attachment with no required main;
- public Branch, AgentLane, AgentHarness, and Session surfaces;
- event/watcher ordering under the Session line;
- remove the old second harness-settings-line lock-order narrative: harness-global settings and every durable lane mutation now serialize through the sole Session line when a coherent snapshot is required; pure synchronous registry reads remain direct;
- work-package table, invariants, races, backend conformance, and glossary.
Update WP05 before M4:
- replace `SessionTree`/lane-line/inheritance assumptions;
- replace `leafId` source examples where the new type names require it;
- keep all M0–M3 historical behavior and M4–M8 durable requirements;
- state that WP06 is the foundation between M3 and M4.
Update every current supporting document whose public names or ordering statements change:
- `docs/assistant-durability.md` and `docs/tool-durability.md` — Session-line FIFO and `branchTip` terminology;
- `docs/values.md` — Session-global value/list surface, Branch tip addresses, and no `SessionTree`;
- `docs/telemetry.md` — receiver inventory and Session mutation spans;
- `docs/plugins.md` — remove `sessionTree`; the plugin's AgentLane directly supplies Branch methods, while a scoped Session-data facet supplies global value/list/name/label/query methods and excludes raw `mutate`, `idGenerator`, close, and backend authority;
- `docs/extensions/pi-extensions-v2.md` and `docs/extensions/pi-server-artifact/index.md` where examples/types use the changed surfaces;
- completed WP00–WP04 only where a forward-looking/current-state statement would otherwise claim the removed API still exists.
Retain the current remote Session mutation contract and update it from a named lane line to the sole Session line: worker `RemoteSession.mutate()` performs keyless begin RPC → local callback with remote reads/one remote commit → local post-commit publication → end RPC. The server holds the Session line through commit and publication until end acknowledgment. Disconnect/timeout terminates the scope under the existing hosting policy. Update the current remote protocol/vertical-slice documentation and every implementation/test present on dev; do not delete or defer this behavior.
Do not rewrite released changelog sections. Add no changelog entry on a non-main/non-PR development branch.
---
## 9. Required tests
### Session mutation line
- concurrent `mutate()` callbacks serialize globally, including callbacks invoked from different AgentLanes;
- a second callback does not enter until the first callback returns after commit/publication;
- direct reads do not wait for a callback that has not committed;
- direct reads observe a fully landed commit even while that mutation callback remains open after commit;
- no direct read observes a partial multi-write commit;
- two read-modify-write counter mutations produce `1`, then `2`;
- separate direct `getValue` + `setValue` calls remain deliberately non-atomic;
- zero-commit callback is legal;
- keyless `beginMutation()` excludes every other mutation until `end()`, commit does not release it, end-without-commit is legal, repeated end is idempotent, and close waits for end;
- RemoteSession begin/read/commit/publication/end preserves that same scope;
- second commit attempt rejects, including after a failed first attempt;
- nested public write from a mutation callback is documented as invalid and deterministically blocks/rejects according to the chosen guard;
- close-first rejects mutation; mutation-first completes and close waits; a trusted callback that never returns can block close;
- Storage commit `seq` and stats behavior is unchanged.
### Session and Branch
- fresh Session has zero Branches and no main tip value;
- no storage-version bump, migration, compatibility decoder, or rejection path is added for the replaced WIP format-4 schema;
- legacy coding-agent v3 import reconstructs ordinary total main-lane configuration plus idle state before returning when valid model/thinking history exists; otherwise it returns a data-only main Branch;
- legacy config tests cover complete and incomplete histories, invalid legacy fields, and branched histories where only changes on the selected main path apply;
- `branch(name, context)` returns undefined for absence and receives the exact Context on reads;
- `createBranch` validates name, target, and duplicate atomically;
- two concurrent creates have one winner;
- Branch queries default to its tip and preserve scan/filter/cursor behavior;
- direct Branch message/custom append extends its tip atomically;
- custom entry with absent data remains valid;
- pending assistant append rejects;
- Session global values/lists/name/labels/global queries no longer depend on a branch;
- all pre-close Branch objects reject after close;
- branch-scope fork copies config + idle state together iff source main is configured; tree scope does the same independently per configured lane; data-only Branches remain data-only;
- explicit keyless begin/commit/end preserves commit-before-fork-snapshot ordering and captures one Storage-serialized snapshot boundary.
### AgentHarness and AgentLane
- Harness has no AgentLane methods at type or runtime;
- Session has no Branch methods at type or runtime;
- AgentLane has Branch methods directly and no `sessionTree`/nested Branch property;
- fresh Harness `lanes()` is empty;
- `lane("main")` creates complete main atomically;
- missing named lane defaults to null tip; `createAt` anchors creation;
- existing data-only Branch becomes one complete AgentLane without moving its tip;
- existing complete AgentLane acquisition commits nothing and emits no creation event;
- concurrent acquisitions return the same object and emit exactly one `lane_created`;
- creation commit publishes the Lane and binds recipients before releasing `Session.mutate`;
- invalid name/unknown anchor commit nothing;
- restored complete lanes/open operations are inventoried without creating main;
- partial Branch/config/lane-state combinations fault;
- Harness global metadata wrappers retain commit → publication → `value_update` delivery;
- AgentLane idle append moves the tip; active-run append stages one deferred write and preserves authoritative state;
- close/fault seal every ordinary Lane without relying on Harness inheritance.
### Regression
- M2 exact-Drive ABA fence still checks immediately before commit admission;
- M3 generation intent/frame/settlement writes are byte-identical apart from renamed durable/public fields;
- frame FIFO remains correct under the one Session line;
- watch has only snapshot-first or publication-first outcomes;
- usage totals remain commit-boundary exact across lanes;
- Context remains trailing and source-identical throughout;
- Memory, JSONL, and SQLite repository/storage conformance pass.
---
## 10. Exclusions
Do not add:
- keyed mutation lines, arbitrary lock names, resource locks, multi-lock ordering, versions, or optimistic retries;
- a scheduler, transaction framework, action interpreter, or generic post-commit task system;
- a nested Branch/tree/store/access property on AgentLane;
- compatibility aliases for `SessionTree`, `view`, implicit Session main methods, Harness lane methods, or `createLane`;
- keyed/named begin/end RPC scopes or removal of the current keyless RemoteSession mutation transport;
- effects inside `Session.mutate()`;
- a second Storage commit/sequence mechanism;
- automatic work start from `AgentHarness.create()` or `harness.lane()`;
- public drive before WP05 M8.
Do not modify provider/tool behavior, durable execution phases, retry/deferred policy, or assistant-frame semantics beyond signature/name propagation required by this package.
---
## 11. Validation
Run every modified focused test, then:
```bash
git diff --check
npm run check
./test.sh
rg -n "SessionTree|sessionTree|\.view\(|LaneMutationLine|extends AgentLane|extends Lane" \
packages/agent/src packages/agent/test packages/agent/docs \
packages/session-backends packages/protocol packages/coding-agent/src/experimental \
packages/coding-agent/test/experimental*
rg -n "beginMutation\([^)]*,|mutate\([^)]*,[^)]*," \
packages/agent/src packages/agent/test packages/session-backends packages/coding-agent/src/experimental
rg -n "session\.mutate\([^)]*\"|\.mutate\(\s*[a-zA-Z_$][a-zA-Z0-9_$]*\s*," \
packages/agent/src packages/agent/test packages/session-backends
```
Expected first grep matches only this package's problem statement, explicitly historical work-package API descriptions, negative type assertions, and unrelated coding-agent tree-widget names such as `SessionTreeNode`; no current harness `SessionTree` concept remains. The keyed-mutate grep must be empty after manually checking false positives.
Review the complete implementation and normative doc update with Fable before committing. Do not commit without explicit user approval.
---
## 12. Stop condition
WP06 is complete when:
- Session has one keyless mutation line; callback `mutate()` and explicit begin/commit/end remote transport share it, and neither accepts a lane key;
- direct reads bypass that line and expose only fully applied Storage commits;
- `SessionTree` and implicit-main Session behavior are gone;
- Branch is the data-only path/tip abstraction;
- AgentLane exposes Branch methods directly and retains operation-aware append semantics;
- AgentHarness is composition-only, has no AgentLane methods, and atomically gets/creates lanes;
- fresh Session/Harness require no main;
- all process-local publication and event-binding boundaries remain correct;
- all three backends and focused race tests pass;
- `harness.md` and WP05 describe the new model consistently;
- final Fable review reports no findings;
- WP05 M4 may resume.

View file

@ -0,0 +1,372 @@
# WP07 — SQLite host ownership and live-source forks
**Status: implemented.**
The delivered backend has no writer lease or replacement ownership primitive. It provides no-create read-write/read-only opens, queued same-repository snapshots, independent read-only WAL snapshots for live external sources, canonical physical identity, path-safe IDs, repository-local deletion reservation, and all-settled close. The tests cover both per-file and shared-container layouts, including a writer commit completed after a read snapshot boundary but before that reader closes.
This package aligns `packages/session-backends/sqlite-node` with the product ownership model: the server owns Session records and worker lifecycle, and exactly one host-assigned process owns writable Session authority at a time. Normally that process is the Session worker. The server may temporarily own a newly created or forked destination, but it closes that Session before handing its metadata to a worker.
Storage does not implement writer ownership. Remove the SQLite writer lease; do not repair or replace it.
A server-side fork is intentionally different from a second writer: it may open a live worker-owned source concurrently for one coherent read-only snapshot while the worker keeps committing. Shared SQLite containers remain supported.
## 0. Mandatory reading
Read completely before editing:
1. `packages/agent/docs/plugins.md` ownership, replacement, and removal sections.
2. `packages/server/README.md` and relevant Session routing/removal source and tests.
3. `packages/agent/docs/harness.md` §§0.6, 1.4–1.7, 2.7–2.8, 4.3, and Part 9.
4. `packages/agent/docs/post-wp05-roadmap.md`.
5. completed WP06 §7 and repository/fork conformance.
6. every source file under `packages/session-backends/sqlite-node/src`.
7. every test and benchmark under `packages/session-backends/sqlite-node`.
8. `packages/session-backends/sqlite-node/README.md` and `CHANGELOG.md`.
Do not use stale files under `dist/` as implementation input. Completed WP01/WP06 documents are historical; do not rewrite them to hide the earlier lease implementation.
## 1. Fixed architecture
### 1.1 Writable authority belongs to the host
Exactly one host-assigned process owns a writable Session. The Session worker is the normal owner. Worker replacement closes the old owner before the new worker opens the Session. Server management serializes creation, forking, removal, and attachment lifecycle around that ownership transfer.
Memory, JSONL, and SQLite do not detect a second process opening the same Session for writes. Bypassing the server/worker lifecycle is a trusted-host defect, not a storage race to repair. A repository still rejects duplicate writable handles it owns in one process.
Do not add a storage lease, filesystem lock, fencing token, heartbeat, timeout-based takeover, deletion tombstone, quarantine protocol, or generic lock manager.
### 1.2 Read-only fork access may overlap the worker
The server owns repository administration and may fork a source while its Session worker continues writing it. The source side of that fork:
- opens the exact source container without creating it;
- is read-only;
- uses one deferred `BEGIN` transaction;
- reads the version-gated Session row, scalar values, and selected entries/branch index in that transaction;
- never upgrades to a write or claims writable authority;
- closes after `COMMIT`/`ROLLBACK`.
SQLite WAL permits later worker commits while the read transaction remains open. The fork sees each source commit entirely before or entirely after its snapshot boundary, never a mixture.
### 1.3 Same-repository fork ordering remains distinct
A source already open in the same repository uses its active `SqliteStorage.snapshot()` path. That snapshot is queued on the source `commitQueue`, preserving WP06's admitted-commit ordering seam and the existing conformance case.
Do not replace this path with an independent connection. Instead, bind active storage lookup to exact physical identity plus Session ID so metadata for another container cannot select it accidentally.
### 1.4 Destination ownership does not overlap
`SessionRepo.create()` and `fork()` continue returning an open Session. A server may temporarily own that new destination, capture its metadata, and close it before launching a worker. This is a valid ownership transfer, not a reason to redesign `SessionRepo` into a record-only API.
## 2. Problems in current source
### 2.1 SQLite duplicates host ownership
Current source contains:
- `writer_lease` schema state;
- claim, renew, and release helpers;
- lease claims in create/open/fork;
- an idle renewal timer and lease-loss path in `SqliteOpenSession`;
- a pre-commit renewal callback in `SqliteStorage`;
- lease-based deletion checks.
This is a second, incomplete ownership system. Its pre-commit renewal is not atomic with the following data transaction, but the correct fix is deletion of the ownership mechanism, not transaction-local fencing.
### 2.2 Non-creation access can create files
The database factory exposes only `open(path)`, which creates a missing SQLite file. Metadata open, listing probes, deletion, and fork-source reads must not turn a removed path into an empty database.
Fork-source reads also configure `PRAGMA journal_mode = WAL`, which is a write-oriented setup step and must not run on a read-only connection.
### 2.3 Delete does not reserve its local critical section
`delete()` checks `pendingIds` but does not reserve the ID. A same-repository create/open/fork destination can enter while asynchronous destructive work is in progress. Host lifecycle owns cross-process ordering; the repository still must serialize its own local operations.
### 2.4 Physical identity, paths, and close need correction
- `openStorages` is keyed only by Session ID, so the same ID at another physical path may select the wrong active source.
- `create()` makes `options.directory`, not the parent of an explicit `databasePath`.
- Per-session filenames interpolate arbitrary caller IDs directly; `/`, `\`, `..`, `%`, and platform separators must not escape `directory`.
- `repo.close()` uses fail-fast `Promise.all`, so it may return before every open Session has attempted to drain and close.
Deterministic list ordering, bind-variable limits, branch-copy cost, fork scalar filtering, prepared statements, and VACUUM policy remain separate.
## 3. Required result
### 3.1 Remove storage-layer writer ownership
Delete all runtime lease behavior:
- delete `src/sqlite/session/writer-lease.ts`;
- remove `writer_lease` from WIP `001_initial.sql`;
- remove lease deletion from `deleteSessionRows()`;
- remove claim/renew/release code from `SqliteSessionRepo`;
- remove `beforeCommit` from `SqliteStorage`;
- remove renewal/release options, timer, and `leaseError` from `SqliteOpenSession`;
- remove lease-specific tests and replace them with host-authority and live-fork coverage.
Keep:
- the Storage `commitQueue`;
- one `BEGIN IMMEDIATE` transaction per commit;
- in-transaction `next_seq` allocation;
- entry/usage uniqueness and parent triggers;
- Session mutation admission and close draining;
- process-local duplicate-open rejection.
Format 4 remains WIP. Remove the table from new schema in place; an old file containing an unused `writer_lease` table remains readable and the table and stale rows are ignored forever. Post-WP07 code does not delete them because doing so serves no runtime behavior. A pre-WP07 binary cannot open a new post-WP07 database without that table; backward compatibility for this WIP format is not required. Add no migration, compatibility path, or storage-version bump.
### 3.2 Add explicit database open modes
Extend `SqliteDatabaseFactory` with narrow operations:
- `open(path)` — intentional creation or create-if-missing;
- `openExisting(path)` — read-write open that fails if the file does not exist;
- `openReadOnly(path)` — read-only open that fails if the file does not exist.
The Node adapter uses `DatabaseSync(path, { readOnly: true })` for read-only access. Implement and test an actual no-create read-write mode for `openExisting`; do not rely on `access()` followed by a create-capable open.
Split connection setup:
- writable connections establish WAL mode and `busy_timeout`;
- read-only connections set only read-safe options such as `busy_timeout` and never attempt to change journal mode.
Use no-create modes for metadata open, listing probes, deletion, and fork-source reads.
### 3.3 Preserve both fork-source paths
**Source open in this repository:** retain `SqliteStorage.snapshot()` and queue it after prior admitted commits. Replace the ID-only active map key with canonical `(containerPath, sessionId)` identity and use the same helper for publish, lookup, and removal.
**Source not open in this repository:** this includes a closed source and a source currently owned by a worker in another process. Open the exact source through `openReadOnly`, then capture it in one deferred read transaction. Validate the Session row and storage version inside that transaction. Do not consult destination reservations, claim source ownership, or block the worker's later commits.
The destination remains a normal writable create/fork transaction after source capture. In shared-container mode the source worker and destination transaction may use the same file; SQLite serializes destination writes while preserving Session row isolation.
### 3.4 Make deletion locally exclusive
The host must close the Session worker before calling `repo.delete()`. Direct cross-process deletion of a live Session is unsupported.
Within one `SqliteSessionRepo`, deletion must reserve the Session ID from entry through completion and release it in `finally`:
- an already open or reserved Session rejects deletion;
- create/open/fork destination for that ID rejects while deletion runs;
- shared-container deletion removes only the target Session's rows in one `BEGIN IMMEDIATE` transaction on one connection;
- per-file deletion removes the database and its WAL/SHM sidecars after a no-create open/existence check;
- a missing Session rejects without creating a file.
Do not add a lease check, tombstone, quarantine rename, or stale-deleter protocol. Cross-process removal ordering is the server's responsibility.
### 3.5 Bind metadata to physical identity and make paths safe
- Canonical identity is `(canonical container path, sessionId)`.
- In per-file mode metadata must identify the repository-affine encoded path for its durable ID.
- In shared-container mode metadata must identify the configured canonical container and Session ID.
- A foreign or mismatched path must never alias a local active source. Pin in focused tests and the SQLite README whether foreign source metadata is rejected or read only from its exact path; do not silently substitute a local storage by ID.
- Create `dirname(databasePath)` when `databasePath` is configured.
- Encode arbitrary explicit IDs into safe per-session filenames without changing the durable ID. The encoding must prevent path escape and round-trip through metadata/list/open/fork.
- Two different Session IDs in one shared container remain independently addressable.
### 3.6 Drain all repository-owned closes
`SqliteSessionRepo.close(context)` must:
1. seal repository admission once;
2. start close on every currently open Session;
3. wait for every close to settle;
4. resolve when all succeed;
5. otherwise reject only after all cleanup attempts, returning the one error or an `AggregateError` containing all failures;
6. return the same promise on repeated close.
This is backend-local resource cleanup. Do not change the shared `SessionRepo` interface or JSONL lifecycle in this package.
## 4. Implementation slices
### Slice A — remove writer leases
Files:
- delete `src/sqlite/session/writer-lease.ts`;
- `src/sqlite/migrations/001_initial.sql`;
- `src/sqlite/session/session-row.ts`;
- `src/sqlite/storage.ts`;
- `src/sqlite/session.ts`;
- `src/sqlite/repo.ts`;
- lease-focused repository tests.
Tasks:
1. Remove schema/runtime lease state and timer behavior.
2. Preserve commit serialization, one write transaction, mutation draining, and process-local duplicate-open behavior.
3. Prove ordinary commits now use one write transaction rather than renewal plus write.
### Slice B — no-create opens and deletion reservation
Files:
- `src/index.ts`;
- `src/sqlite/types.ts`;
- `src/sqlite/repo.ts`;
- focused adapter/repository tests.
Tasks:
1. Add `openExisting` and `openReadOnly` with tested no-create behavior.
2. Separate writable and read-only connection configuration.
3. Reserve deletion locally for its full critical section.
4. Delete one shared-container Session in one transaction; preserve unrelated Sessions.
### Slice C — live-source read-only forks and identity
Files:
- `src/sqlite/repo.ts`;
- repository/conformance tests.
Tasks:
1. Key active storage by canonical container plus Session ID; keep same-repository queue ordering.
2. Use one independent read-only deferred transaction for non-open/live-worker sources.
3. Validate source metadata/version inside the snapshot.
4. Preserve shared-container destination behavior.
### Slice D — paths and close draining
Files:
- `src/sqlite/repo.ts`;
- focused repository/conformance tests.
Tasks:
1. Create the actual custom container parent.
2. Encode arbitrary IDs safely.
3. Reject or exactly handle foreign metadata without active-source aliasing.
4. Make repository close all-settled and error-complete.
### Slice E — documentation
Files:
- `packages/agent/docs/harness.md`;
- `packages/agent/docs/post-wp05-roadmap.md`;
- `packages/agent/docs/values.md`;
- `packages/session-backends/sqlite-node/README.md`;
- changelog only under normal branch rules.
Document host-owned writable authority, the two fork-source paths, no-create opens, local deletion reservation, and absence of storage-layer ownership.
## 5. Required tests
Use real independent `node:sqlite` connections. Test-only wrappers may expose deterministic transaction boundaries; production code gets no sleeps or race flags.
### Lease removal
- new schema has no `writer_lease` table;
- create/open/fork/commit/close perform no lease reads or writes and start no renewal timer;
- same-repository duplicate writable open still rejects through process-local reservation;
- ordinary commit remains one `BEGIN IMMEDIATE` transaction.
### Live fork source
For both per-file and shared-container layouts:
- a server repository forks a source held open by a separate repository/connection representing its worker;
- source capture uses a distinct read-only connection and claims no writable ownership;
- a source commit completed before the snapshot boundary appears wholly in the fork;
- a commit after the read snapshot is established can complete before the reader closes and is wholly absent from that fork;
- no fork contains an entry without the same commit's Branch tip/value/stats changes;
- a later fork includes the later commit;
- same-repository admitted-commit fork conformance remains unchanged.
### Deletion
- open/reserved Session first → same-repository delete rejects;
- delete reservation first → same-repository create/open/fork destination for that ID rejects;
- shared deletion removes only the target rows;
- per-file deletion removes database/WAL/SHM files;
- missing-path open/list/fork/delete does not create an empty database;
- tests state the host precondition: worker close precedes deletion; no cross-process bypass safety is promised.
### Identity and paths
- same Session ID at two physical paths cannot cross-select active source storage;
- two Session IDs in one shared container remain independent;
- `databasePath` succeeds when its parent does not exist;
- explicit IDs containing `../`, `/`, `\`, `%`, dots, and Unicode stay inside `directory` and preserve the metadata ID;
- metadata returned by create/list/open/fork names the actual container.
### Close
- one Session close failure does not prevent every other Session cleanup attempt;
- multiple failures are reported after all settle;
- repeated repository close returns the same promise;
- every successfully closed Session releases its connection.
### Regression
- existing storage and repository conformance remains semantically unchanged;
- fork destination reservation and same-repository source ordering remain intact;
- fork snapshots exclude operation/pending/result/usage/application state exactly as before;
- shared-container create/list/open/fork/delete remains supported;
- every write transaction still uses `BEGIN IMMEDIATE`;
- no migration, storage-version bump, compatibility layer, or replacement ownership primitive appears.
## 6. Validation and review
After each code slice:
```bash
npm run check
```
Run each modified focused test from `packages/session-backends/sqlite-node` with the repository Vitest binary. Final validation:
```bash
./test.sh
```
Review checkpoints:
1. Fable after Slice A: verify no storage ownership mechanism remains and close draining is intact.
2. Fable after Slice C: verify same-repository ordering and live-worker read-only overlap both hold.
3. Final Fable review of source, tests, docs, and exclusions.
Delegated reviews use provider `anthropic` and model `claude-fable-5`.
## 7. Exclusions
Do not include:
- any storage lease, lock, fence, heartbeat, takeover, tombstone, or quarantine;
- a record-only `SessionRepo` redesign or compatibility facade;
- server/router/worker-manager redesign; record any backend-independent lifecycle race separately;
- SQLite branch-segment redesign or uncompacted-divergence optimization;
- fork scalar filtering/indexing;
- `getEntries` bind-limit chunking or general query-limit normalization;
- statement caches or stats aggregation optimization;
- catalog redesign, async database replacement, or VACUUM policy;
- search/FTS;
- R11 migration machinery or a storage-version bump;
- [mobile assistant-output handoff](../mobile-handoff/01-harness/05-assistant-output/message-update.md) changes or JSONL compaction;
- repository-wide `SessionRepo.close()` contract changes;
- removal of shared-container support;
- transaction DSLs, schedulers, generic lock managers, or compatibility layers.
If implementation requires an excluded item, stop and revise the handoff rather than expanding silently.
## 8. Exit condition
WP07 is complete when:
- SQLite contains no active or schema-defined writer lease and implements no replacement ownership mechanism;
- host ownership is the documented single-writer authority;
- same-repository active-source forks retain their queued commit boundary;
- live worker-owned sources fork through an independent read-only snapshot while later WAL commits continue;
- deletion reserves its same-repository critical section and assumes worker-first host removal;
- non-creation paths cannot create empty databases;
- active source identity includes physical container plus Session ID;
- explicit IDs cannot escape the directory and custom database parents are created;
- repository close waits for every cleanup attempt;
- shared-container mode remains fully covered;
- focused tests, `npm run check`, and `./test.sh` pass;
- final Fable review reports no blocker.

View file

@ -0,0 +1,246 @@
# WP08 — Named-branch and tree forks with streaming copies
**Status: in progress — implementing Slice C.**
This package replaces the fork contract: `ForkOptions` gains a mandatory scope and a mandatory named source branch, branch forks validate a complete configured source AgentLane and ancestry membership, tree forks copy the complete immutable tree plus current application values/lists, and all three backends replace materialized source snapshot arrays with bounded-memory streaming copies. A JSONL fork never repairs or mutates its source. One closed core classifier owns every namespace's fork disposition.
WP07 is a hard dependency and is preserved unchanged: no-create database modes, canonical `(containerPath, sessionId)` identity, independent read-only WAL reader for external/live-worker sources, repository-local deletion reservation, and all-settled close. This package supersedes only the roadmap's "SQLite fork cost" performance item.
## 0. Mandatory reading
Read completely before editing:
1. `packages/agent/docs/harness.md` §§0.6, 1.3–1.7, 2.3, 2.7–2.9, Part 9 (invariants 3–5, 13, 16; ledger completeness).
2. `packages/agent/docs/values.md`, especially "Forks and rewrites" and the backend sections.
3. `packages/agent/docs/post-wp05-roadmap.md` (SQLite fork cost, repository lifecycle context).
4. Completed WP06 §7 and WP07 (historical; do not edit them).
5. `packages/agent/src/harness/session/fork.ts`, `types.ts` (`ForkOptions`, `SessionRepo`), `values.ts`, `in-memory-storage-state.ts`, `memory.ts`, `session/index.ts`.
6. `packages/agent/src/harness/session/jsonl/repo.ts`, `jsonl/storage.ts`, `jsonl/codec.ts`, `jsonl/legacy-v3.ts`, `jsonl/types.ts`.
7. `packages/session-backends/sqlite-node/src/sqlite/repo.ts`, `storage.ts`, `session/values.ts`, `session/entries.ts`, `session/branch-entries.ts`, `types.ts`.
8. `packages/agent/src/harness/session/testing/conformance/session-repo.ts` and every test named in §5.
9. `packages/agent/src/harness/session/testing/benchmark/session-repo.ts` and both `session-repo.bench.ts` files.
Do not use `dist/` output as implementation input. WP00–WP07 documents and released changelog sections are immutable in this package.
## 1. Fixed architecture
### 1.1 Public contract
```ts
export type ForkOptions =
| { scope: "branch"; branch: string; entryId?: string; position?: "before" | "at"; id?: string }
| { scope: "tree"; id?: string };
```
`scope` is required. `branch` is required for branch scope. There is no default scope, no implicit `main`, and no compatibility alias.
**Branch scope** requires a complete configured source AgentLane: `pi.branch.tip/{branch}`, `pi.lane.config/{branch}`, and `pi.lane.state/{branch}` must all exist. A missing tip rejects (`unknown branch`); a data-only Branch (tip without config/state) rejects. Forking does not audit unrelated lanes or malformed lane records: supported writers create lane state atomically, so branch validation checks only the selected Branch's required tip, config, and state. `entryId`, when supplied, must be an entry on that Branch's current tip ancestry (inclusive); an entry elsewhere in the tree rejects. Omitted `entryId` means the current tip. `position` defaults to `"at"`; `"before"` selects the target's parent and may yield a `null` destination tip (before the root entry, or a `null` source tip with no `entryId`) — legal. The destination contains exactly one Branch with the same name, the selected tip, the copied `LaneConfiguration`, and fresh idle lane state `{ currentOperationId: null, lastOperationId: null, inbox: [] }`. No other Branch or lane exists in the destination.
This deliberately makes branch forks of data-only Branches — including legacy v3 imports whose main lacked reconstructable model/thinking history — unavailable. Tree scope remains available for those sources.
**Tree scope** copies: every immutable entry, including entries unreachable from every current Branch tip; every Branch tip verbatim; every configured lane's config plus fresh idle lane state under the same name; data-only Branches (tip present, config/state absent) as data-only Branches. Forking does not perform a corruption audit of lane records; supported writers create valid lane state atomically.
**Both scopes**: copy `pi.session.name`; copy `pi.entry.label` values only for copied entries; exclude the usage ledger, `pi.result`, all `pi.op.*`, all `pi.pending.*` (entries, tool checkpoints, assistant frames), and every trace of open-operation state. Destination usage totals start at zero; destination `messageCount` equals the number of copied message entries, exactly as current conformance already proves. Destination metadata records `parentSessionId = source.id`. Sequence preservation is not part of the fork contract: backends may allocate destination-local sequences for copied and transformed writes, provided destination sequence and list-cursor semantics remain internally valid.
**Application values/lists (outside the reserved `pi`/`pi.*` namespaces)**: tree scope copies every current scalar value and every surviving list element; branch scope copies none. A `seq <= tipSeq` cutoff is forbidden as "historical reconstruction" — it is provably not one:
```text
Scalar: TX[seq 10: set my-app.state = v1] · TX[seq 12: insert e1] ·
TX[seq 50: set my-app.state = v2] (v2 describes work after e1)
Branch fork at e1, cutoff seq <= 12: only v2 exists (v1 was replaced, no
history is retained); 50 > 12 excludes it → state absent, though the app
demonstrably had state v1 at the fork point. The cutoff cannot recover v1.
List: append seq 5 · append seq 20 · deleteList seq 40 · append seq 60
Cutoff seq <= 30: elements 5 and 20 no longer exist (whole-list delete
destroyed them); only 60 survives and is excluded → empty list, though the
list held {5, 20} at seq-30 time.
```
Any cutoff filters *survivors*, silently mixing rewound intent with post-deletion reality. Branch scope therefore copies nothing application-owned, matching its fresh idle lane state and zero ledger; applications own their own re-derivation.
### 1.2 One closed fork classifier
All namespace fork knowledge lives in one core module beside `session/values.ts` and `session/fork.ts` (e.g. `session/fork-policy.ts`). It is a closed switch, not a registry, plugin policy, or DSL:
- `pi.op.*`, `pi.pending.*`, `pi.result` → exclude, both scopes.
- `pi.session.name` → copy.
- `pi.entry.label` → copy iff the keyed entry is copied.
- `pi.branch.tip`, `pi.lane.config`, `pi.lane.state` → structured lane actions; the scope-specific rules (branch keeps only the named lane and rewrites its tip; lane state is always replaced with fresh idle state; tree keeps all) live in one shared driver consumed by all backends.
- The exact namespace `pi` and any other `pi.*` namespace → the fork **fails**, but only when current surviving state exists at fork time (a current scalar row or a surviving list element). Historical writes later replaced or deleted are absent from current state on every backend and must not alone fail a JSONL fork — behavior is backend-equivalent. Introducing a new built-in namespace without declaring its fork semantics must break fork tests, not silently copy or drop state.
- Neither `pi` nor `pi.*` → application: copy on tree, exclude on branch. The only built-in list namespace (`pi.pending.assistant_frame`) excludes; application lists follow the application rule.
The driver exposes a streaming shape — accept one committed value/list write (or current row), emit zero or more destination writes, `finish()` emits fresh idle lane states and the rewritten branch tip. Entry-copy membership is a backend-supplied predicate so each backend uses its own index. `createForkSnapshot`, `forkSnapshotWrites`, `ForkSourceSnapshot`, `ForkDestinationSnapshot`, and the `entriesComplete` escape hatch are deleted.
### 1.3 Streaming backend procedures
Forks must stream source writes rather than materializing full source snapshot arrays. Source copy paths must not call array-returning full reads: no `snapshotEntriesAndValues()`, `captureForkSource()`, `readAllScalarValueRows()`, `readAllEntryRows()`, whole-file `readTextFile`, or SQLite `.all()` over unbounded row sets. JSONL may retain its structural index and legacy-v3 compaction tail in memory.
**Memory.** At a source `commitQueue` boundary, build the destination `InMemoryStorageState` directly by iterating source maps once through the classifier/driver. Branch scope walks `parentId` from the branch tip toward the root to compute the ancestry id set and verify `entryId` membership; copy exactly that set. No intermediate snapshot arrays; `MemoryStorage.fromSnapshot` and `captureForkSource` are removed.
**JSONL.** The source is read through a read-only path that captures the source's current `nextSeq` at the commit-queue boundary; both scans emit only complete writes below that sequence. The fork never writes the source: no torn-tail truncation/rewrite, no v3 normalization persistence, no `.tmp` beside the source. A torn or incomplete final line is discarded in memory. This relies on format-4 sources remaining append-only and not being replaced while the fork runs; compaction/replacement coordination is deferred with J1.
- *Both scopes* use two scans up to the captured sequence boundary. Pass 1 streams every line and folds value/list writes into an in-memory structural index: per scalar address, the seq of the current surviving `set` (absent after a trailing `delete`); per list address, the set of surviving element seqs (whole-list delete clears it); plus entry `id → parentId` where branch scope needs ancestry. The fold applies the same current-state semantics replay does.
- Pass 2 streams again and emits a write only when the index proves it current and the classifier selects it: an entry write in the selected set; a scalar `set` whose seq equals the index's current-row seq for its address; a list `append` whose element seq is in the survivor set. Deletes, superseded sets, and dead application value/list history are never emitted as destination state.
- Transformed built-ins emit when pass 2 reaches the source's current corresponding row for a kept branch or configured lane. The destination writer may assign destination-local sequences.
- *Branch scope* walks tip→root through the in-memory index to materialize the ancestry set and verify `entryId` membership before pass 2; labels emit only for member entries.
- The destination is staged in a temp file and atomically renamed (existing `publishFileAtomically`); failure removes the temp file. Legacy v3 splits by source state. Forking an **open** legacy-v3 `JsonlStorage` **rejects with a clear error** until a normal non-empty commit has upgraded and persisted its normalized format-4 ids: v3 normalization mints nondeterministic UUIDv7 tails, so an independent disk reparse cannot reproduce the ids the open Session exposes or validate a caller `entryId`; the fork must not mutate/upgrade the source itself. A **closed** legacy-v3 source uses the read-only parser/normalizer and never touches the file: tree forks succeed; branch forks may use an omitted `entryId` (the default normalized tip) when the import reconstructs a complete configured lane, while any caller-supplied process-local id from an earlier open is not stable across reparses and rejects normally; a data-only reconstructed lane rejects as everywhere. The parser may retain structural metadata and compaction `retainedTail` context in memory. Ordinary writable `JsonlStorage.open` may retain in-memory normalization, but resident state is never a fork source.
- There is **no resident-state path**: an open same-repository JSONL source enqueues a short boundary callback on the source `commitQueue` whose only job is to capture `nextSeq`, then releases the queue; the fork runs the same two-scan procedure below that boundary while later source appends proceed. Committed writes are durable in the file before they are applied to resident state, so the file is authoritative at the boundary.
**SQLite.** Iterator-based row transfer through a bounded-memory temporary on-disk SQLite **staging database**, used uniformly for per-file and shared-container layouts. The source reader never streams directly into the destination transaction — a WAL reader and one writer can coexist, but in shared-container mode the destination writer would hold the container's sole write lock and block the post-boundary source writer while capture is still streaming; staging into a separate file removes that coupling. Use prepared-statement iteration (`iterate`/stepwise), never source-sized `.all()` snapshot arrays:
- *External/closed/live-worker source:* WP07 path — `openReadOnly` on the exact canonical path, one deferred read transaction, session row and storage version validated inside it.
- *Same-repository open source:* open the independent read-only connection **first**, then enqueue a short boundary callback on the source Storage `commitQueue` whose only job is to `BEGIN` and establish the independent reader's snapshot (issue a trivial read) before releasing the queue. Do not begin a read transaction on the source writer connection and then release its queue, and do not hold the queue for the copy duration.
- *Stage:* while the source reader remains open, stream selected entries `ORDER BY seq` and classifier-selected current scalar/list rows (SQL-level namespace prefilters matching the classifier — enumerated built-in namespaces plus the application predicate excluding exact `pi` and `pi.%` — with each row still passing the classifier) into the temporary staging database in bounded batches. Branch scope enumerates the ancestry through the `branch_entries` segment chain up to the selected entry and answers `entryId` and label membership through that index, not an in-RAM id set. Later source commits use the original writer connection and may complete while staging streams, in **both** layouts, because stage writes target another file.
- *Publish:* close/commit the source read transaction, then stream the stage into one destination `BEGIN IMMEDIATE` transaction — entries in source order maintaining the destination branch index and `message_count` incrementally, then values/list elements — and delete the staging database in `finally` on success and failure. The destination allocates its own valid sequence range. Shared-container destinations write only the new session's rows.
### 1.4 Preserved WP07 behavior
Destination id reservation across create/open/fork/delete, no-create opens, foreign-metadata rejection, per-file and shared-container layouts, WAL commit-boundary wholeness (a source commit is wholly inside or wholly outside one fork), and all-settled repository close are unchanged.
### 1.5 Coding-agent status (record only)
`/fork`, `/clone`, and `--fork` run entirely on the legacy `SessionManager` (`createBranchedSession`, `forkFrom`) and are not migrated here. Future mapping when coding-agent adopts `SessionRepo`: `/fork` → `{ scope: "branch", branch: "main", entryId, position: "before" }`; `/clone` → `{ scope: "branch", branch: "main" }`; `--fork` → `{ scope: "tree" }`.
## 2. Problems in current source
- `ForkOptions` defaults `scope` to `"branch"` and hard-codes source/destination `main` (`fork.ts`, both backend branch readers).
- No ancestry membership check: `selectForkContents` walks parents from any supplied `entryId` anywhere in the tree and labels the result `main`.
- Every backend materializes the full source: `snapshotEntriesAndValues()` (Memory/JSONL), `readAllScalarValueRows` + `readAllEntryRows`/`scanBranchEntries` arrays (SQLite), all funneled through in-memory `createForkSnapshot`.
- `createForkSnapshot` and its array snapshots force all backends through a materialized in-memory source copy.
- A JSONL fork of a closed source uses `JsonlStorage.open()`, which rewrites the source file on a torn tail — a fork can mutate its source today.
- Lists are never copied; application values are excluded with no declared tree policy (`values.md` explicitly defers it).
- Namespace fork knowledge is repeated across `fork.ts`, `values.md` prose, and conformance assertions; nothing forces a new `pi.*` namespace to declare fork semantics.
- `entriesComplete?: false` exists only to let SQLite branch snapshots skip tip validation for other branches.
## 3. Required result
1. `ForkOptions` and validation exactly as §1.1; all rejection paths create no destination file, database rows, or reserved-but-leaked ids.
2. The closed classifier/driver of §1.2, exported from core and consumed by all three backends; unknown reserved namespaces (exact `pi` or undeclared `pi.*`) fail the fork only when current surviving state exists, identically on every backend.
3. Streaming procedures of §1.3 on all three backends; `createForkSnapshot`/`captureForkSource`/`snapshot()` fork plumbing and their exports removed from `session/index.ts` and the sqlite-node import surface.
4. Valid destination-local sequence allocation on every backend; identical logical destination state across backends for identical sources (conformance).
5. JSONL source non-mutation, including torn-tail sources and legacy v3 sources.
6. Documentation: `harness.md` §2.7 (and the §1.7 fork-related sentences), `values.md` "Forks and rewrites" and backend snapshot mentions, `post-wp05-roadmap.md` (retire the SQLite fork-cost item, add/point to this package), `packages/session-backends/sqlite-node/README.md` fork paragraphs, `packages/agent/benchmark/session/README.md` if dataset wording changes. Historical WP docs and released changelogs untouched. Format/storage versions unchanged; no migration.
## 4. Implementation slices
### Slice A — contract and classifier (core, Memory reference)
Files: `session/types.ts`, new `session/fork-policy.ts`, `session/fork.ts` (rewritten or deleted), `session/values.ts` (only if new helpers are needed), `session/index.ts`, `session/memory.ts`, `session/in-memory-storage-state.ts`, conformance `testing/conformance/session-repo.ts`, `test/harness/memory-conformance.test.ts`, `test/harness/memory-session-repo.test.ts`.
1. Replace `ForkOptions`; implement validation and the classifier/driver.
2. Rewrite Memory fork as direct destination construction at a commit-queue boundary.
3. Rewrite the shared fork conformance for the new contract (§5) and pass it on Memory.
### Slice B — JSONL streaming
Files: `session/jsonl/storage.ts`, `session/jsonl/repo.ts`, `session/jsonl/types.ts`, streaming reader support in the `FileSystem` capability (`harness/types.ts`, `harness/env/*`) if required, `test/harness/jsonl-session-repo.test.ts`, `jsonl-session-repo-conformance.test.ts`, `jsonl-storage.test.ts`, `jsonl-v3-migration.test.ts`.
1. Read-only sequence-boundary capture with in-memory torn-tail discard; the same capture at the source `commitQueue` boundary for open sources; no source writes on any fork path.
2. Two-scan in-memory current-state fold and emission for both scopes; branch ancestry membership through the structural index; atomic destination publish.
3. Open-v3 fork rejection and the closed-v3 read-only parser/normalizer path; update v3 fork tests per §5.
### Slice C — SQLite streaming
Files: `sqlite/repo.ts`, `sqlite/storage.ts`, `sqlite/session/values.ts`, `sqlite/session/entries.ts`, `sqlite/session/branch-entries.ts`, `sqlite/types.ts`, `test/repo.test.ts`, `test/repo-conformance.test.ts`.
1. Replace `SqliteStorage.snapshot()`/array snapshot helpers with the independent-reader-plus-boundary-callback design and iterator transfer into the temporary staging database, then stage-to-destination publication with `finally` cleanup.
2. Branch ancestry/membership/labels through the branch index; classifier-matching SQL prefilters.
3. Preserve WP07 identity, reservation, no-create, and close coverage; both layouts.
### Slice D — benchmarks and documentation
Files: `testing/benchmark/session-repo.ts`, both `session-repo.bench.ts` files, `benchmark/session/README.md`, `harness.md`, `values.md`, `post-wp05-roadmap.md`, sqlite-node `README.md`, changelogs only under normal branch rules.
Update fork option literals, add large-source fork benchmarks (tree and branch), and land the §3.6 documentation set.
## 5. Required tests
### Contract and validation
- branch scope rejects: unknown branch name; data-only Branch (tip only); `entryId` not on the named Branch's tip ancestry (present elsewhere in the tree); `entryId` unknown; `entryId` with a `null` tip. Each rejection leaves no destination artifact and releases its reserved id. Fork tests do not construct or audit malformed lane records outside the selected Branch.
- branch scope accepts: omitted `entryId` (tip), explicit tip, mid-ancestry entry, `position: "before"` at a mid entry and at the root entry (`null` destination tip), `null` source tip with no `entryId`.
- destination shape: exactly one Branch, same name, copied config, fresh idle lane state, no other lane values, `parentSessionId` set.
- tree scope: unreachable entries copied; every tip copied; configured lanes get config plus fresh idle state; data-only Branches stay data-only. Forking does not audit malformed lane records.
- both scopes: session name copied; labels only for copied entries (branch fork excludes labels of non-ancestry entries); `pi.result`, `pi.op.*`, `pi.pending.*` (pending entries, tool checkpoints, assistant frame lists), and usage rows absent; exact stats expectation — a fork copying N message entries reports `getStats()` = `{ messageCount: N, usage: all-zero }`, matching current conformance; the ledger-completeness invariant ("a fork's ledger starts at zero") retained.
### Application values and lists
- tree fork copies every non-`pi.*` scalar and every surviving list element; a list deleted-then-reappended in the source reproduces only survivors.
- branch fork copies no application values/lists (pin the §1.1 scalar and list traces as regression cases: post-fork-point overwrite and delete-destroyed elements must not resurface under any implementation).
- current surviving state in an unknown reserved namespace (exact `pi` or an undeclared `pi.*` scalar or surviving list element) fails the fork on every backend; the same namespace **set then deleted** before the fork fails no backend — including JSONL, where the dead history remains as physical lines and the disk fold must classify it as absent. Construct both cases via raw committed writes.
### Sequence allocation
- sequence preservation, source high-water marks, and source list cursors are not part of the fork contract. Each destination assigns valid local sequences and list cursors; its first post-fork commit must allocate without collision.
- JSONL captures source `nextSeq` only to establish the append-only source boundary; a torn incomplete final write does not belong to the fork.
### Streaming source reads and source non-mutation
- instrumented source readers (test decorators/spies) assert fork paths never call `snapshotEntriesAndValues`, `captureForkSource`, `readAllScalarValueRows`, `readAllEntryRows`, whole-file `readTextFile`, or source-sized `.all()` arrays on entries/values/lists; deterministic large fixtures (extend the existing benchmark dataset generators) exercise tree and branch forks over multi-thousand-entry sources.
- SQLite staging databases are removed on success and failure; a failed fork leaves neither a stage file nor destination rows.
- JSONL: fork of a closed torn-tail source succeeds, the destination excludes the torn transaction wholly, and the source file bytes are unchanged (byte-compare before/after); fork of a legacy v3 source leaves the source file unchanged; no `.tmp` appears beside the source.
- JSONL sequence-boundary capture: a source append completed after capture is wholly absent from the fork.
- JSONL destination content: the destination file contains only current selected scalar rows and surviving list elements — no delete records, superseded sets, or dead application history.
- legacy v3: forking an **open** v3 source rejects with the clear pre-upgrade error; after one normal non-empty commit upgrades it to format 4, a fork succeeds and preserves the persisted ids; a **closed** v3 tree fork succeeds through the read-only parser and leaves the source byte-identical; a closed v3 branch fork with omitted `entryId` succeeds when a complete configured lane is reconstructed, a caller-supplied id from an earlier open rejects, and a data-only reconstructed main rejects.
### Coordination and ordering
- Memory open-source forks keep the queue-boundary conformance case: a commit admitted before the fork appears wholly; one admitted after does not.
- JSONL open-source forks capture the sequence boundary at the `commitQueue` boundary: a commit admitted before the boundary appears wholly; a source append completed after boundary capture proceeds while the fork is still streaming and is wholly absent from that fork.
- SQLite same-repo: the independent reader opens first, the boundary callback on the source `commitQueue` establishes its snapshot and releases the queue, and a subsequent source commit on the writer connection **completes while the reader is still streaming into the stage** and is wholly absent from that fork; a later fork includes it wholly. Assert this in **both** per-file and shared-container layouts (satisfiable in shared containers because stage writes target another file), plus the WP07 live external-worker case unchanged.
- destination-reservation races (create vs fork on one id, both orders) unchanged.
### Backend equivalence
- one shared conformance source (entries, unreachable entries, multiple lanes, data-only Branch, labels, app scalars/lists with deletes, open operation with pending/frame/checkpoint state, usage rows) forks to identical logical destination state on Memory, JSONL, and SQLite for both scopes.
## 6. Validation and review
After each slice: `npm run check`, then the modified focused tests via the repository Vitest binary from the owning package; `./test.sh` before final review. Grep guards: no remaining `createForkSnapshot|captureForkSource|ForkSourceSnapshot|entriesComplete` outside historical docs and this handoff; no **production fork source path** referencing `"main"` as a literal (this document and the recorded coding-agent mapping in §1.5 intentionally mention main).
Review checkpoints (delegated reviews use provider `anthropic`, model `claude-fable-5`):
1. After Slice A: contract, classifier closure, conformance shape.
2. After Slice C: same-repo ordering, reader independence, and streaming-source evidence.
3. Final review of source, tests, docs, and exclusions.
## 7. Exclusions
Do not include:
- compatibility aliases, a default scope, or implicit `main`;
- any `seq <= tip` cutoff or other historical-state reconstruction;
- a fork-policy registry, plugin hook, or DSL; per-address application opt-ins;
- copying usage rows, `pi.result`, or any open-operation state under any option;
- coding-agent `/fork`, `/clone`, `--fork`, or RPC migration;
- storage-version bumps, migrations, or format changes (formats stay WIP-in-place);
- J1 snapshot compaction (the streaming reader may be built shareable, nothing more);
- SQLite branch-segment redesign or the uncompacted-divergence fix;
- `SessionRepo` interface changes beyond `ForkOptions` (repository `close()` belongs to the lifecycle package);
- precise-rewrite tooling; search; edits to WP00–WP07 documents or released changelog sections.
If implementation requires an excluded item, stop and revise the handoff.
## 8. Exit condition
WP08 is complete when:
- `ForkOptions` is exactly the §1.1 union with the stated validation, on all three backends;
- branch forks require a complete configured source AgentLane and enforce ancestry membership; tree forks copy the complete immutable tree, every tip, configured and data-only Branches;
- tree forks carry all current application values and surviving list elements; branch forks carry none;
- one closed core classifier owns every namespace disposition; unknown reserved namespaces (exact `pi` or undeclared `pi.*`) fail forks exactly when current surviving state exists, equivalently on all backends;
- all supported fork paths — including closed legacy v3 sources — stream source writes without materializing full source snapshot arrays, proven by instrumented-reader tests over large fixtures; JSONL may retain structural indexes and legacy compaction context in memory; forking an open legacy-v3 source is explicitly unsupported (clear rejection) until an ordinary commit upgrades it, not handled by an in-memory exception;
- JSONL forks never mutate their source, including torn-tail and legacy v3 sources, and JSONL destinations contain only current selected rows;
- SQLite forks stage through a temporary on-disk database (source reader → stage while open, stage → one destination `BEGIN IMMEDIATE` after reader close, stage deleted in `finally`), with the independent-reader boundary design and concurrent later writer commits proven in both layouts, preserving all WP07 behavior;
- destinations allocate valid local sequences and backends produce identical logical destinations;
- shared conformance, focused backend tests, benchmarks, `npm run check`, and `./test.sh` pass;
- normative docs and the roadmap reflect the new contract with historical documents untouched;
- final Fable review reports no blocker.
## 9. TODO
### Cleanup after invocation cancellation
Cancellation-safe filesystem cleanup is deferred. Cleanup currently receives the caller's Context, so an aborted invocation can leave a staged file behind even though readers close and the destination id reservation is released. Define the cleanup Context policy before implementing this; do not import Chord's `withoutAbortSignal` directly into JSONL code as a temporary workaround.

View file

@ -0,0 +1,517 @@
# Work package 09 — LaneSnapshot settled-but-unplaced tools
## Status and baseline
- Repository: `earendil-works/pi`
- Branch at handoff creation: `dev`
- Baseline commit: `d14d6b22327d545d6a253f932165b63e48d7f9c8`
- The user reported the worktree clean immediately before this handoff.
- This document began as an implementation handoff after conversation compaction and now records the implemented design. It is not itself the normative harness specification; `packages/agent/docs/harness.md` remains normative.
## Goal
Keep every started or settled-but-unplaced tool call visible in `LaneSnapshot.operation.runningTools` until its immutable `toolResult` entry is placed in the transcript.
The intended projection is:
```text
planned → not yet represented in runningTools
effect_pending → runningTools(status: "running")
outcome_ready → runningTools(status: "settled")
completed → transcript toolResult entry
```
For each call after it becomes presentation-active, `runningTools` and placed transcript entries must not have a gap or overlap. Placement is a source-prefix flush, not an all-tools barrier. Remove a call from `runningTools` on its own `entry_added`, never on `turn_end`.
## Original bug
A call disappeared between real-effect completion and source-ordered tree placement:
1. `packages/agent/src/harness/runtime/reducer.ts`: `tool_end` spliced the call out of `runningTools`.
2. `packages/agent/src/harness/runtime/lane.ts`: `captureLaneSnapshot()`, `case "tools"`, projected only `effect_pending` calls and skipped `outcome_ready` calls.
3. The finalized result was already durable at `pendingEntry(resultEntryId)`, but a fresh/reconnected snapshot could not display it.
4. It reappeared only after `entry_added` placed the immutable `toolResult` entry.
For a parallel batch `[A, B, C]`, if B settles while A remains pending, B may stay `outcome_ready` until A is ready. If A is already placed, B can place without waiting for C. Therefore clearing everything at `turn_end` is wrong: early-placed results would temporarily exist in both `transcript` and `runningTools`.
## Confirmed current architecture
Mini does **not** replicate structural object deltas.
- `packages/coding-agent/src/experimental/mini/worker/lane-service.ts` sends one initial full snapshot and then forwards individual `HarnessEvent` objects.
- `packages/coding-agent/src/experimental/mini/tui/session.ts` folds those events through `reduceLaneSnapshot()`.
- Reconnect/rebase fetches another complete snapshot.
- `tool_update` currently carries a complete replacement progress result, not a nested diff.
Implemented durable tool flow in `packages/agent/src/harness/runtime/drive/tools.ts`:
```text
prepare
→ before_tool
→ intent commit (effect_pending + effective args), then tool_start
→ execute/update/checkpoint
→ after_tool
→ finalize
→ publishToolOutcome staging commit (pendingEntry + outcome_ready), then tool_end
→ materializeReady prefix placement
→ entry_added
```
All real, immediate synthetic, cancellation, and recovery outcomes converge through `publishToolOutcome()`.
`Lane.settleOperation()` supports commit-bound events. It commits, publishes process-local state, constructs the event batch, and the public operation awaits delivery. In parallel execution, `materializeReady()` is scheduled only after the outcome-completion promise resolves. Consequently, `tool_end` is delivered after staging and before placement.
## Agreed event contract change
Do **not** add `tool_result_ready` or `tool_outcome_ready`.
Instead, redefine harness `tool_start`/`tool_end` as tool-call processing/result lifecycle events rather than exclusively real external-effect lifecycle events.
### Fresh executed call
```text
intent commit
→ tool_start
→ tool_update*
→ execute/finalize
→ TX[pendingEntry + outcome_ready + cleanup]
→ tool_end
→ source-ordered placement
→ entry_added
```
### Fresh synthetic call
```text
TX[pendingEntry + outcome_ready]
→ tool_start
→ tool_end
→ source-ordered placement
→ entry_added
```
The staging transaction's post-commit event batch contains `tool_start` followed by `tool_end`, so a watcher cannot observe either lifecycle event without the authoritative staged state.
Fresh synthetic calls include:
- unknown tool;
- argument preparation or validation failure;
- `before_tool` denial or invalid replacement arguments;
- genuine assistant `length`/truncated call handling;
- cancellation while still `planned` or after intent but before effect admission.
### Recovery
Historical lifecycle events are not replayed.
- A restored `effect_pending` call is already represented by the initial snapshot.
- Safe replay emits recovery-tagged `tool_start` from the checkpoint-clear commit and `tool_end` from the later outcome-staging commit.
- Unsafe interruption synthesis may emit a recovery-tagged `tool_end` without a newly emitted `tool_start`; the initial snapshot supplied the running row.
- A call already restored as `outcome_ready` appears as settled in the initial snapshot and needs no replayed end event before placement.
### Meaning of `tool_end`
After this change, `tool_end` means:
> The complete final tool result is durably staged and the call is `outcome_ready`.
It becomes the authoritative reducer transition from running to settled. It must be emitted **after** the staging commit, not before it.
The old distinction between actually executed and synthetic results was encoded by omitting lifecycle events. There is no in-repo runtime consumer requiring that distinction. If preserving it is desired, discuss adding an explicit field such as `execution: "executed" | "synthetic"`; this field was discussed but **not agreed**, so do not add it silently.
## LaneSnapshot type
Change `LaneSnapshot.operation.runningTools` in `packages/agent/src/harness/agent-harness.ts` to use one `result` field for both progress and final output. Do not retain a separate `partialResult` field in the snapshot.
Prefer a discriminated union so invalid combinations are unrepresentable:
```ts
type SnapshotTool =
| {
status: "running";
toolCallId: string;
toolName: string;
args: unknown;
result?: AgentToolResult<unknown>; // latest complete progress snapshot
}
| {
status: "settled";
toolCallId: string;
toolName: string;
args: unknown;
result: AgentToolResult<unknown>; // complete finalized result
isError: boolean;
};
```
The user explicitly agreed to the discriminated union.
`tool_update.partialResult` may remain named `partialResult` in the event API; the reducer assigns it to the snapshot row's unified `result` field.
The current mini transport sends semantic events, not structural deltas, so `tool_end` still carries the complete final result even if it equals the latest update. Unifying the snapshot field is still the correct state model.
## Exact reducer behavior
File: `packages/agent/src/harness/runtime/reducer.ts`
### `tool_start`
- Resolve the operation with `matchingOperation(snapshot, event.runId)`.
- Upsert by `toolCallId`; do not blindly push.
- Set `status: "running"`, `toolName`, `args`.
- Clear stale settled-only fields if replacing an existing row.
- Preserve no stale final result. A newly started/replayed call may receive later `tool_update` values.
Upsert is required because a watch may capture durable `effect_pending` state before the buffered `tool_start` event is delivered.
### `tool_update`
- Use `matchingOperation(snapshot, event.runId)`, not `snapshot.operation` directly.
- Find the matching row.
- For a running row, replace `result` with `event.partialResult`.
- Ignore wrong-operation or missing rows.
### `tool_end`
- Resolve with `matchingOperation(snapshot, event.runId)`.
- Find the existing row by batch-local `toolCallId`; only one tool batch is presentation-active at a time.
- Replace it with `status: "settled"`, preserving its arguments and using `event.result` and `event.isError`.
- This naturally removes the provisional interpretation of the old `result`; there is no separate `partialResult` to delete.
- The finalized result remains displayed until placement.
`tool_end` does not carry arguments and cannot create a row. Fresh synthetic `tool_start` and `tool_end` are emitted together after the staging commit, eliminating the old capture/event gap. Unsafe recovery relies on the initial snapshot's running row.
### `entry_added`
If `event.entry` is a message whose role is `toolResult`, remove the matching batch-local `toolCallId` from `snapshot.operation?.runningTools`, then apply the transcript update. Harness events are serialized, trusted, emitted exactly once, and not historically replayed, so neither duplicate-entry handling nor cross-batch identity is needed.
Do not clear tool rows on `turn_end`.
## Exact authoritative capture behavior
File: `packages/agent/src/harness/runtime/lane.ts`, `captureLaneSnapshot()`, `case "tools"`.
The assistant entry is already loaded once. For each batch call:
### `planned`
Skip. It has not become presentation-active yet.
### `completed`
Skip. Its `toolResult` entry must already be in the captured transcript.
### `effect_pending`
- Validate that `assistant.message.content[sourceIndex]` is the matching `toolCall` block.
- Read `operationToolArgs(operationId, turnId, sourceIndex)`; it is required for effect-pending calls.
- Read optional `pendingToolOutput(operationId, resultEntryId)`.
- Project:
```ts
{
status: "running",
toolCallId: block.id,
toolName: block.name,
args: persistedArgs,
...(checkpoint === undefined ? {} : { result: checkpoint })
}
```
### `outcome_ready`
- Validate the source tool-call block.
- Read `pendingEntry(call.resultEntryId)`.
- Require a message payload with role `toolResult`.
- Validate staged `toolCallId` and `toolName` against the source block.
- Read `operationToolArgs(...)` when present.
- Use `persistedArgs ?? block.arguments`. Immediate synthetic calls may never have written `operationToolArgs`, and this absence is legal only for the outcome-ready projection.
- Reconstruct the canonical `AgentToolResult` from the staged `ToolResultMessage` and the durable call termination flag as needed.
- Project `status: "settled"`, `result`, and `isError`.
The event and capture representations must normalize the final result identically so folding through `tool_end` equals a later authoritative snapshot. Pay attention to optional `details`, `usage`, and `terminate`; do not rely on incidental object-property presence differences.
A missing or mismatched staged result for `outcome_ready` is presentation corruption and must fault snapshot capture.
## Runtime event production changes
Primary file: `packages/agent/src/harness/runtime/drive/tools.ts`
Related helpers: `packages/agent/src/harness/execution/tools.ts` and `packages/agent/src/harness/runtime/drive/tool-placement.ts`.
### Internal outcome shape
Current:
```ts
type ToolOutcome = { message: ToolResultMessage<unknown>; terminate: boolean };
```
Extend/refactor it so post-commit event production has the complete canonical final result and `isError`, without lossy reconstruction. It must retain enough data for:
- staged `ToolResultMessage`;
- `tool_end.result`;
- `tool_end.isError`;
- durable/effective `terminate` after cancellation normalization.
Synthetic helpers currently return `ToolResultMessage` directly. Refactor carefully so synthetic outcomes also carry the canonical result data. Do not invent `details` in the transcript: existing unknown/invalid synthetic results deliberately omit message details.
### Commit-bound `tool_start`
For fresh execution, `publishToolIntent()` attaches `tool_start` to the commit that persists effective arguments and changes the call to `effect_pending`. The public operation awaits delivery before admitting `executeToolCall()`, preserving `tool_start → tool_update*` without requiring each update callback to await delivery.
For a fresh synthetic call that never writes effect intent, `publishToolOutcome()` attaches `tool_start` before `tool_end` in the outcome-staging commit's event batch. It reports the source block arguments.
For safe recovery, the checkpoint-clear commit emits recovery-tagged `tool_start` using persisted effective arguments. Do not emit a fresh start for an already-restored unsafe `effect_pending` call; its initial snapshot is the baseline.
### Post-commit `tool_end`
Remove the current pre-staging `tool_end` emission from `performToolInvocation()`.
`publishToolOutcome()` attaches `tool_end` to the same staging command's `events` callback. The event carries:
- `runId`, `turnId`, `toolCallId`, `toolName`;
- canonical final `result`;
- `isError`;
- cancellation-normalized durable `terminate`;
- `recovery: true` where applicable.
Arguments belong to `tool_start` and are not repeated on `tool_end`. Event data describes the state actually committed, especially cancellation forcing `terminate: false`.
Because `Lane.command()` awaits retained event delivery and `runParallel()` schedules materialization from the outcome-completion promise, the required order is:
```text
staging commit
→ tool_end delivery
→ source-ready message lifecycle
→ placement commit
→ entry_added
```
### Call sites to audit
Every `publishToolOutcome()` call must supply the source tool call and recovery context correctly:
- immediate outcome in `startToolInvocation()`;
- cancellation after intent but before execution;
- normal `performToolInvocation()` completion;
- safe replay completion;
- unsafe recovery interruption;
- sequential cancellation of `planned`;
- sequential cancellation of `effect_pending`.
Also audit the synchronous `AbortRequested` path inside `performToolInvocation()`: a call with durable intent must still have coherent start/end presentation even if effect admission fails immediately.
## Mini and other presentation consumers
### Mini
File: `packages/coding-agent/src/experimental/mini/tui/view.ts`, `MiniTui.apply()`.
For each `runningTools` row:
```text
status running:
markExecutionStarted()
if result exists: updateResult({...result, isError:false}, true)
status settled:
do not call markExecutionStarted()
updateResult({...result, isError}, false)
```
The final result remains visible while awaiting placement. After `entry_added`, transcript synchronization supplies the immutable `ToolResultMessage` and the row is no longer in `runningTools`.
### Other shared consumer
Apply equivalent handling in:
- `packages/coding-agent/src/experimental/client-tui-chat.ts`
Read `packages/coding-agent/src/modes/interactive/components/tool-execution.ts` before editing to confirm `updateResult(result, isPartial)` semantics.
## Tests
### Reducer tests
File: `packages/agent/test/harness/runtime/reducer.test.ts`
Add a parallel batch event-fold test with calls 0, 1, 2:
1. Establish all three as running.
2. Call 2 settles before call 0.
3. Call 0 settles and is placed while call 1 still runs.
4. Call 1 settles.
5. Placement flushes calls 1 and 2 in source order.
6. After every settlement and placement event, assert each presentation-active call appears in exactly one of:
- `operation.runningTools`; or
- transcript `toolResult` entries.
7. Assert call 2 remains present with `status:"settled"` and its final result while blocked by earlier calls.
8. Assert each `entry_added` removes only its matching active row.
Add focused coverage for:
- `tool_update` from a stale/wrong `runId` does not mutate the current operation;
- `tool_start` upserts rather than duplicates a row captured from durable intent;
- `tool_end` settles the existing batch-local row.
### Capture/watch tests
File: `packages/agent/test/harness/runtime/watch.test.ts`
Construct a durable tools state containing:
- `planned` (omitted);
- `effect_pending` with checkpoint (`status:"running"`, checkpoint exposed as `result`);
- `effect_pending` without checkpoint;
- real `outcome_ready` with persisted effective args;
- synthetic `outcome_ready` without `operationToolArgs`, falling back to source block arguments;
- completed call represented only by a transcript entry where practical.
Assert staged settled content, `isError`, arguments, and absence of duplicates. Add missing/mismatched `pendingEntry` corruption assertions.
### Runtime tool tests
File: `packages/agent/test/harness/runtime/drive-tools.test.ts`
Update/add ordering assertions proving:
- real: `intent commit < tool_start < tool_update* < staging commit < tool_end < entry_added`;
- immediate synthetic: `staging commit < tool_start < tool_end < entry_added`, no tool effect and no `after_tool`;
- planned cancellation gets coherent start/end;
- unsafe recovery uses initial snapshot plus recovery-tagged end without replaying effects;
- B can emit post-commit end and remain settled while A blocks placement;
- source-order placement remains unchanged;
- a crash after staging cannot replay the call.
The baseline normative tests/documentation required `tool_end` before staging; the implementation reverses those expectations deliberately.
### Type/event catalog tests
Audit:
- `packages/agent/test/harness/types.test.ts`
- `packages/agent/src/harness/telemetry.ts`
No new event name is added. `tool_end` omits arguments and its semantics change.
### Mini regression
After unit tests, run the real mini abort smoke test used previously:
- execute exactly `sleep 20`;
- send Escape after about two seconds;
- assert `Command aborted` and elapsed time appear;
- assert raw ANSI contains `toolErrorBg` (`48;2;60;40;40`);
- verify the durable session contains an `isError:true` tool result and operation status `aborted`.
Also exercise a parallel batch where a later tool finishes first and verify its final result remains visible until in-order placement.
## Documentation changes
Read both documents completely before editing:
- `packages/agent/docs/harness.md` (normative)
- `packages/agent/docs/tool-durability.md`
The implementation updates baseline statements that required:
- `tool_end` before staging;
- `tool_start`/`tool_end` only for real effects;
- synthetic outcomes emitting no lifecycle;
- `outcome_ready` being omitted from `runningTools`;
- snapshot field `partialResult`.
Important known locations from the baseline:
- `harness.md` §3.8 around lines 784–796;
- `harness.md` §5.4 `LaneSnapshot` around lines 1101–1134;
- `harness.md` §5.5 events around lines 1140–1158;
- `harness.md` tool phases around lines 1214–1224;
- `harness.md` conformance requirements around lines 1386–1400;
- `tool-durability.md` finalization around lines 255–280;
- `tool-durability.md` snapshots/events around lines 568–584;
- `tool-durability.md` test requirements around lines 671–679.
The revised docs must state:
- `tool_end` is post-staging durability evidence for a final result;
- fresh synthetic outcomes receive start/end lifecycle;
- recovery events are not historically replayed;
- `outcome_ready` remains projected as settled until placement;
- `entry_added` moves settled presentation into transcript;
- the unified snapshot `result` is provisional when running and final when settled.
Do not modify the separate `response.ts`/`tool-placement.ts` recovery `turn_end` discrepancy unless separately requested.
## Files to reread completely after compaction
Core/specification:
1. `packages/agent/docs/harness.md`
2. `packages/agent/docs/tool-durability.md`
3. `packages/agent/src/harness/agent-harness.ts`
4. `packages/agent/src/harness/runtime/reducer.ts`
5. `packages/agent/src/harness/runtime/lane.ts`
6. `packages/agent/src/harness/runtime/drive/tools.ts`
7. `packages/agent/src/harness/runtime/drive/tool-placement.ts`
8. `packages/agent/src/harness/execution/tools.ts`
9. `packages/agent/src/harness/runtime/types.ts`
10. `packages/agent/src/harness/session/types.ts`
11. `packages/agent/src/harness/events.ts`
12. `packages/agent/src/harness/telemetry.ts`
Tests:
13. `packages/agent/test/harness/runtime/reducer.test.ts`
14. `packages/agent/test/harness/runtime/watch.test.ts`
15. `packages/agent/test/harness/runtime/drive-tools.test.ts`
16. `packages/agent/test/harness/types.test.ts`
17. Relevant test helpers imported by those files.
Mini/presentation:
18. `packages/coding-agent/src/experimental/mini/tui/session.ts`
19. `packages/coding-agent/src/experimental/mini/worker/lane-service.ts`
20. `packages/coding-agent/src/experimental/mini/shared/protocol.ts`
21. `packages/coding-agent/src/experimental/mini/tui/view.ts`
22. `packages/coding-agent/src/experimental/client-tui-chat.ts`
23. `packages/coding-agent/src/modes/interactive/components/tool-execution.ts`
Before editing, run `git status --short` and inspect current diffs because other Pi sessions may share the worktree.
## Validation commands
From repository root, after changes:
```bash
cd packages/agent
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/harness/runtime/reducer.test.ts
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/harness/runtime/watch.test.ts
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/harness/runtime/drive-tools.test.ts
node "$(git rev-parse --show-toplevel)/node_modules/vitest/dist/cli.js" --run test/harness/types.test.ts
cd "$(git rev-parse --show-toplevel)"
npm run check
```
Do not run `npm test`, the full Vitest suite, or `npm run build` unless requested.
If a delegated review is used, repository policy requires:
```text
--provider anthropic --model claude-fable-5
```
Keep extensions enabled.
## Non-goals
- No generic structural-delta transport for mini.
- No optimization to avoid the final result crossing the wire once in `tool_end` and again in `entry_added`.
- No change to source-prefix placement semantics.
- No clearing on `turn_end`.
- No change to tool effect replay/durability rules.
- No change to `after_tool`: it still runs only for actual fresh/safely replayed effects under its existing cancellation contract.
- No work on the separate recovery `turn_end` discrepancy between `response.ts` and `tool-placement.ts`.
- No backward-compatibility layer unless the user explicitly asks for one.
- Do not commit unless the user asks.

View file

@ -111,7 +111,7 @@ export function generateTelemetryDocs(outputPath: string, check: boolean): void
if (process.argv[1] && pathToFileURL(resolve(process.argv[1])).href === import.meta.url) { if (process.argv[1] && pathToFileURL(resolve(process.argv[1])).href === import.meta.url) {
generateTelemetryDocs( generateTelemetryDocs(
resolve(import.meta.dirname, "../telemetry-schema.md"), resolve(import.meta.dirname, "../docs/telemetry-schema.md"),
process.argv.includes("--check"), process.argv.includes("--check"),
); );
} }

View file

@ -34,7 +34,7 @@ describe("agent telemetry schemas", () => {
"pi.harness.event_handler", "pi.harness.event_handler",
"pi.session.write", "pi.session.write",
]); ]);
const actual = readFileSync(resolve(import.meta.dirname, "../../telemetry-schema.md"), "utf8"); const actual = readFileSync(resolve(import.meta.dirname, "../../docs/telemetry-schema.md"), "utf8");
expect(actual).toBe(renderAgentTelemetrySchemaMarkdown()); expect(actual).toBe(renderAgentTelemetrySchemaMarkdown());
}); });