From 34bad4471ae9c4a937c256d288d1f1c0c1ee06e5 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 17 Aug 2026 02:27:01 +0200 Subject: [PATCH] docs(agent): consolidate harness runtime plan --- packages/agent/docs/assistant-durability.md | 348 +++++ packages/agent/docs/harness.md | 1132 ++++++++++------- packages/agent/docs/registers.md | 820 ++++++------ packages/agent/docs/runtime2.md | 792 ++++++++++++ packages/agent/docs/tool-durability.md | 280 ++-- .../docs/work-packages/00-runtime1-removal.md | 75 ++ .../work-packages/01-bound-values-lists.md | 57 + 7 files changed, 2532 insertions(+), 972 deletions(-) create mode 100644 packages/agent/docs/assistant-durability.md create mode 100644 packages/agent/docs/runtime2.md create mode 100644 packages/agent/docs/work-packages/00-runtime1-removal.md create mode 100644 packages/agent/docs/work-packages/01-bound-values-lists.md diff --git a/packages/agent/docs/assistant-durability.md b/packages/agent/docs/assistant-durability.md new file mode 100644 index 000000000..beec9029e --- /dev/null +++ b/packages/agent/docs/assistant-durability.md @@ -0,0 +1,348 @@ +# Assistant partial durability — implementation handoff + +This document specifies durable partial assistant messages for ordinary assistant generation and deferred-response polling. It builds on: + +- bound typed value/list addresses from `registers.md`; +- `AssistantMessageFrame`, `assistantMessageEventToFrame()`, and `reduceAssistantMessageFrames()` from `@earendil-works/pi-ai`; +- the assistant intent/effect/settlement state machine in `harness.md`. + +The design persists compact replayable stream frames without making them operation-state authority and without storing a growing full partial message on every update. + +## Goals + +1. Reconstruct the latest committed partial assistant message after process loss. +2. Preserve provider stream order without repeated full-snapshot write amplification. +3. Avoid provider backpressure from storage commits. +4. Reuse pi-ai's canonical frame conversion and reduction semantics. +5. Preserve current public assistant event ordering. +6. Keep operation `effect_pending` state authoritative for recovery. +7. Delete all partial frames atomically with normal or synthetic response settlement. + +## Non-goals + +- Inferring provider completion from frames. +- Persisting terminal `done`/`error` events separately from response settlement. +- Exactly-once provider requests. +- Public frame cursors or a frame-persistence event. +- Generic batching, timers, coalescing, or flush APIs. +- Persisting structural summary-generation streams. +- Persisting arbitrary provider SDK events or repeated full `partial` snapshots. + +## Storage + +Built-in address constructor in `session/values.ts`: + +```ts +export const pendingAssistantFrames = ( + operationId: string, + responseEntryId: string, +) => list( + "pi.pending.assistant_frame", + `${operationId}:${responseEntryId}`, +); +``` + +The procedure binds one exact address for ordinary generation or a deferred poll: + +```ts +const frames = pendingAssistantFrames(operationId, responseEntryId); +``` + +`responseEntryId` is already reserved in assistant/deferred `effect_pending` state. No frame count, cursor, or list identity is stored in operation state, and later list operations receive only `frames`—never another key. + +Each list element is one `AssistantMessageFrame`. The storage transaction's global write sequence orders frames. The list is auxiliary: + +- missing is valid; +- it does not prove request admission, completion, success, or failure; +- it never selects the restart point; +- base restore does not read it. + +## Frame contract + +For each pi-ai stream event: + +```ts +const frame = assistantMessageEventToFrame(event); +``` + +- `start` and non-terminal block events produce a frame; +- terminal `done` and `error` return `undefined` because final response settlement is separate; +- later frames omit repeated growing `partial` snapshots; +- text/thinking/tool end frames may contain one authoritative completed block value because some providers, including OpenAI Responses, can supply or correct content only at the end; +- unfinished tool-call JSON is reconstructed by pi-ai's reducer; +- completed tool-call arguments are parsed but remain unvalidated against a tool schema. + +Do not define a second harness frame codec or reducer. Persistence stores the exported pi-ai value directly; hydration calls `reduceAssistantMessageFrames()`. + +Provider event blocks may interleave. Storage and reduction rely on `contentIndex`, never block contiguity. + +## Fresh stream scheduling + +The simplest scheduling is deliberate: one list-append transaction per converted frame, enqueued without awaiting storage in the provider loop. + +For every `start` or update event: + +```text +convert event to frame +→ synchronously enqueue invocation-fenced appendList(frames, frame) +→ attach the ordinary harness-fault observer to the returned promise +→ replace process-local latestFrameWrite promise reference +→ emit and await the existing message_start/message_update event +→ consume the next provider event +``` + +`appendList()` is called synchronously, so all frame mutations enter the lane line in provider-event order. The procedure does not await each write. Every promise is immediately observed for fault propagation before the latest reference replaces its predecessor. Model output limits bound the number and total size of queued frames; a response may queue at most its bounded output plus constant frame/transaction overhead. + +The event side retains existing behavior: `AssistantStreamObserver.start/update` awaits `events.emit()` for every event. There is no separate latest event-delivery promise because no assistant event delivery remains outstanding when the provider loop advances. + +When the provider stream settles: + +```text +stop frame admission +→ await latestFrameWrite when present +→ run after_response +→ emit message_end +→ classify and commit final response settlement +``` + +The lane mutation line is FIFO, so completion of `latestFrameWrite` implies every earlier append completed. There is no array of promises, timer, batcher, active/waiting state, coalescer, or public/internal flush method. + +A failed frame append faults the harness before `after_response` starts. The complete final response remains process-local and does not commit after a storage fault. + +## Normal settlement + +Every assistant response settlement that used a frame list deletes that exact list in the same transaction as the immutable response entry, usage, leaf, and next operation state: + +```text +TX[ + insert response entry R, + insert usage U, + setValue(laneLeaf(lane), R), + deleteList(frames), + setValue(operationState(operationId), classified next state) +] +``` + +This applies to: + +- successful assistant responses; +- provider `error`/`aborted` responses; +- valid deferred responses; +- deferred poll responses. + +`after_response` may transform the final response before settlement. Frames preserve provider-stream observation, while the immutable entry remains the post-hook canonical result. + +A crash after the final frame append but before settlement still restores `effect_pending`; frames do not turn a complete-looking draft into a settled response. + +## Unknown-outcome generation recovery + +An orphaned assistant generation `effect_pending` has no surviving provider stream. Activation: + +1. constructs `frames = pendingAssistantFrames(operationId, responseEntryId)` from current typed state and reads bounded pages from that exact address; +2. reduces frame values with `reduceAssistantMessageFrames()`; +3. constructs a harness-owned synthetic assistant response under the already-reserved response ID; +4. preserves reconstructed partial content and safe message identity metadata when frames exist; +5. sets `stopReason: "error"`, zero usage, and an explicit interruption/unknown-outcome `errorMessage`/diagnostic; +6. commits the synthetic response, zero-usage row, frame-list deletion, leaf, and ordinary retry/failure state atomically. + +Required warning meaning: + +```text +The provider request was interrupted. The preceding content is the latest +committed partial response; newer live output may be missing, and the external +request outcome is unknown. +``` + +If no start frame committed, the harness constructs the same synthetic error with empty content from the captured model/API identity. + +The synthetic response follows ordinary assistant error classification: + +- attempts remain → insert the error response and enter the ordinary retry-wait/next-attempt path; +- cap reached → insert it and enter `failure_drain`. + +Error responses remain durable transcript history but are omitted from later provider context by the existing projection rule. Partial tool calls inside the interrupted error response never execute. + +Recovery does not run `after_response`: there is no trustworthy complete provider result to transform. + +## Cancellation + +A live cancelled provider stream settles through its ordinary final `aborted` response. All accepted frames are awaited first, and normal settlement deletes the list. + +For restored cancelled assistant/deferred `effect_pending`, cancellation reconciliation: + +- reduces available frames; +- constructs a synthetic `aborted` response preserving the committed partial content; +- uses zero usage and the existing reserved IDs; +- atomically inserts the response and deletes the frame list; +- starts no provider request and runs no response hook. + +Cancellation wins classification even if reduced content appears complete. + +## Deferred polling + +A deferred poll that returns an `AssistantMessageEventStream` uses the same `pendingAssistantFrames(operationId, responseEntryId)` address constructor. + +- normal pending/ready/error settlement deletes the poll's frame list; +- restored cancellation synthesizes an aborted response from frames; +- an unknown restored poll without a permit remains suspended and may expose its durable partial in snapshots; +- when a poll permit replaces the unknown poll with fresh reserved response/usage IDs, that intent transaction deletes the abandoned old frame-list address; +- the replacement poll starts a fresh list under its fresh response ID. + +Frames never change deferred-handle equality or poll-number rules. + +## Structural generation scope + +Structural summary-generation streams remain process-local. They do not emit public assistant-message lifecycle and may span multiple nested provider requests before one structural publication. Existing attempt-level retry and usage recovery remains authoritative. + +Do not store their intermediate text at a `pendingAssistantFrames(...)` address in this slice. If structural partial diagnostics become a requirement, add a separate explicitly scoped consumer rather than silently reusing transcript-assistant semantics. + +## Snapshots and reconnect + +`LaneSnapshot.streamingMessage` means the latest observed partial assistant message, not proof that a provider stream is currently attached. + +Precedence: + +1. newest process-local partial while a live stream is owned; +2. otherwise, for assistant/deferred `effect_pending`, the value reduced from committed frame pages; +3. otherwise absent. + +A restored lane may therefore be `suspended` with a non-undefined `streamingMessage`. The field remains outside `transcript`; only `entry_added` moves a complete response into transcript history and clears the partial. + +Snapshot hydration constructs the exact bound frame address from trusted typed operation state and enforces the assistant consumer's total frame/page budget. It does not scan arbitrary lists or perform a broad semantic restore audit. + +Reconnect replays no historical `message_start` or `message_update` events. The snapshot carries the durable partial. Recovery later emits the ordinary recovery-tagged synthetic message lifecycle for the response it actually settles. + +## Events + +Live event ordering remains: + +```text +message_start +→ message_update* each listener delivery awaited by provider loop +→ await latest frame write +→ after_response +→ message_end +→ atomic response settlement + frame-list delete +→ entry_added +→ usage +``` + +Frame commits emit ordinary storage telemetry only. There is no public frame event and no claim that a `message_update` was durable. `entry_added` remains the only proof that the final assistant entry committed. + +A crash may occur after a live update event but before its asynchronously queued frame append commits. Reconnect then shows the latest committed frame prefix, which may be older than the last live event. + +## Close, fault, and external finalization + +Close is a controlled crash: + +- no synthetic response is written; +- already-enqueued frame commits are ordinary admitted session work and may finish under the close barrier; +- process loss may discard mutations that had not committed; +- reopen restores the latest committed frame prefix under unchanged `effect_pending` state. + +A frame-commit storage failure faults the harness. No later response settlement commits in that process. + +Authorized external finalization deletes the operation-owned frame-list address in its terminal transaction. Every append mutation verifies current operation/response ownership on the lane line: + +- append first → external terminal cleanup deletes the list; +- finalization first → stale append rejects without recreating state. + +## Terminal cleanup, forks, and migrations + +Normal/synthetic response settlement should already delete its exact frame address. The operation terminal transaction also defensively constructs and deletes the current operation-owned frame address when state is assistant/deferred `effect_pending`. + +Idle forks never copy lists in the `pi.pending.assistant_frame` address family. Precise rewrites and migrations page frame lists and preserve element sequences when retaining them. + +A migration changing `AssistantMessageFrame` shape must map every surviving element or explicitly delete the whole list and leave `effect_pending` recovery with no partial. It must never infer completion from legacy frames. + +JSONL retains deleted frame bytes until snapshot compaction. Logical deletion is immediate. + +## Races + +| Race | Required result | +|---|---| +| frame append vs next frame | synchronous lane enqueue preserves provider-event order | +| frame append vs stream settlement | settlement awaits the latest promise; all accepted appends finish first | +| live update event vs frame commit | either may finish first; event is observation and reconnect uses only committed frames | +| append vs external finalization | append first is deleted by cleanup; finalization first fences the append | +| process loss with queued writes | only the committed prefix restores | +| final frame vs response settlement | frame commits first; settlement atomically deletes the list and inserts final entry | +| activation vs snapshot | both reduce the same committed sequence prefix; activation may then settle and clear it | +| unknown generation vs retry | synthetic partial error commits under old reserved IDs before later attempt starts | +| unknown deferred poll vs replacement | old list is deleted with fresh replacement intent; new response ID gets a new list | + +## Invariants + +1. Scalar assistant/deferred state is the sole restart authority. +2. One effect-pending response ID constructs exactly one assistant frame-list address. +3. Every stored element is an exported pi-ai `AssistantMessageFrame`. +4. Terminal `done`/`error` events are never stored as frames. +5. Frame order equals provider event order. +6. Awaiting the latest frame-write promise at stream settlement implies all accepted appends completed. +7. Frames never establish provider completion or suppress unknown-outcome recovery. +8. Final or synthetic response settlement atomically deletes the exact frame list. +9. A restored partial may appear in `streamingMessage` but never in `transcript` before settlement. +10. Interrupted partial tool calls never produce a tool plan because the synthetic response stops with `error`/`aborted`. +11. Structural generation never writes a `pendingAssistantFrames(...)` list in this slice. +12. Terminal cleanup, external finalization, and idle forks leave no operation-owned assistant frame list. + +## Required tests + +### Frame integration + +- each convertible pi-ai event appends exactly one frame; +- `done`/`error` append nothing; +- interleaved content indexes preserve sequence; +- provider loop does not await individual frame writes; +- frame appends enqueue synchronously before the next provider event; +- only the latest promise reference is retained; +- awaiting the latest promise implies all earlier writes completed; +- bounded output bounds queued frame memory; +- storage failure prevents `after_response` and faults the harness. + +### Settlement and recovery + +- every ordinary response class deletes the frame list atomically; +- crash at every frame/settlement boundary; +- no frames and partial text/thinking/tool-call frames; +- authoritative end-frame content survives; +- interrupted generation commits partial synthetic error then retries or fails at cap; +- interrupted partial tool calls never execute; +- recovery uses zero usage and existing reserved IDs; +- cancellation preserves committed partial content in synthetic aborted response; +- `after_response` never runs for restored synthetic settlement. + +### Deferred, snapshots, and lifecycle + +- deferred poll frame persistence and normal cleanup; +- unknown poll snapshot without permit; +- replacement intent deletes abandoned poll frames; +- live partial takes precedence over durable reduction; +- reopen exposes reduced `streamingMessage` on suspended effect-pending lane; +- no historical update event replay; +- recovery settlement clears partial on `entry_added`; +- structural generation writes no assistant frame list. + +### Storage lifecycle + +- Memory/JSONL/SQLite reduce identical frame sequences; +- frame list is absent after normal, synthetic, cancellation, and external terminal transitions; +- idle fork excludes frames; +- JSONL compaction removes deleted frame bytes; +- migration maps or explicitly discards every legacy frame; +- instrumentation records append/delete ordering without frame content in telemetry. + +## Implementation map + +Expected runtime areas: + +- built-in `pendingAssistantFrames(operationId, responseEntryId)` address constructor in `session/values.ts`; +- bound value/list storage implementation from `registers.md`; +- assistant execution observer and generation procedure; +- deferred polling procedure; +- activation and cancellation recovery; +- snapshot hydration; +- terminal cleanup, forks, and migrations; +- instrumented writer and backend conformance tests. + +Implement bound typed value/list addresses first, then frame enqueue/settlement, then recovery/snapshots. Update `harness.md` with this complete lifecycle before implementing runtime2 assistant parity. diff --git a/packages/agent/docs/harness.md b/packages/agent/docs/harness.md index dc887555e..c6389f1ea 100644 --- a/packages/agent/docs/harness.md +++ b/packages/agent/docs/harness.md @@ -12,17 +12,17 @@ - [Part 1 — Storage](#part-1--storage) - [1.1 The model](#11-the-model) - [1.2 Identity](#12-identity) - - [1.3 Register namespaces](#13-register-namespaces) + - [1.3 Bound values and lists](#13-bound-values-and-lists) - [1.4 Transactions](#14-transactions) - [1.5 Queries](#15-queries) - [1.6 Usage ledger](#16-usage-ledger) - [1.7 Backends](#17-backends) - - [1.8 Why write-once plus registers](#18-why-write-once-plus-registers) + - [1.8 Why write-once plus values and lists](#18-why-write-once-plus-values-and-lists) - [Part 2 — The conversation tree](#part-2--the-conversation-tree) - [2.1 Entries](#21-entries) - [2.2 Placement](#22-placement) - [2.3 Lanes](#23-lanes) - - [2.4 Facts](#24-facts) + - [2.4 Session metadata and application values](#24-session-metadata-and-application-values) - [2.5 Branch queries and context](#25-branch-queries-and-context) - [2.6 The branch index](#26-the-branch-index) - [2.7 Forks](#27-forks) @@ -31,7 +31,7 @@ - [Part 3 — The operation state machine](#part-3--the-operation-state-machine) - [3.1 Operations](#31-operations) - [3.2 Operation state — the durable restart point](#32-operation-state--the-durable-restart-point) - - [3.3 Lane state and current-state validity](#33-lane-state-and-current-state-validity) + - [3.3 Lane state and the restore projection](#33-lane-state-and-the-restore-projection) - [3.4 The atomic transition rule](#34-the-atomic-transition-rule) - [3.5 The graph](#35-the-graph) - [3.6 Acceptance](#36-acceptance) @@ -82,6 +82,8 @@ A durable runtime for agent conversations. It persists conversation and operation state so interrupted work can resume without repeating settled effects. +> **Pending WP00 reconciliation:** the acceptance, activation, and hook text that still mentions pre-acceptance `before_run`, `before_resume`, `resumeData`, hook-produced `systemPromptOverride`, stable hook-ID routing, or process-origin activation categories is superseded by `runtime2.md`'s “Superseding hook and execution design.” [WP00](work-packages/00-runtime1-removal.md) folds that approved design into this document before deleting runtime1. All other sections remain normative. + ## 0.2 System model ### Session @@ -95,7 +97,7 @@ A session groups related work and has four parts: └── e ── f ``` -- **Facts.** Mutable, namespaced key-value state. Built-ins include the session name and entry labels; applications may store custom facts. +- **Values and lists.** Mutable state at bound typed addresses. Built-ins include the session name and entry labels; applications define their own collision-resistant `value()` and `list()` addresses directly. - **Lanes.** Named cursors into the tree. Every session has `main`. A lane owns its leaf, model configuration, queues, and at most one operation. Additional lanes support Slack threads, subagents, and other parallel work over shared history. - **Usage ledger.** Append-only token and cost events for the session. @@ -107,7 +109,7 @@ An **operation** is one accepted unit of lane work: a run, compaction, or naviga ### Storage -Below the session and harness, `Storage` exposes atomic transactions and queries over three durable forms: immutable entries, mutable registers, and append-only usage rows. Registers form a mutable, namespaced key-value store. Facts live there; internal harness namespaces durably store pending content and lane and operation state needed for crash recovery. In particular, `op.meta` is written once with an operation's metadata, while `op.state` is replaced after each transition with its complete current state. A tool may also request occasional durable replacement checkpoints of its bounded live progress; those checkpoints remain auxiliary and never prove effect completion. The terminal transaction deletes operation-owned registers and writes `lane.lastResult`. No partial transaction is visible. +Below the session and harness, `Storage` exposes atomic transactions and queries over three durable forms: immutable entries, mutable bound values/lists, and append-only usage rows. `value(namespace, key?)` binds one replaceable current value; `list(namespace, key?)` binds one list of immutable appended elements deleted only as a whole address (§1.3). Session metadata, application state, and harness state use the same address mechanism. In particular, `pi.op.meta` is written once with an operation's metadata, while `pi.op.state` is replaced after each transition with its complete current state. A tool may also request occasional durable replacement checkpoints of its bounded live progress; those checkpoints remain auxiliary and never prove effect completion. The terminal transaction deletes operation-owned values/lists and writes `pi.lane.lastResult`. No partial transaction is visible. ## 0.3 The three stores @@ -117,15 +119,16 @@ Everything in Parts 1–5 follows from these. ```text entries the conversation tree — write-once, append-only -registers current mutable state — namespaced typed cells, overwrite or delete +values/lists current mutable state — typed replaceable values + and append-only lists (append or whole-list delete) usage ledger cost history — append-only rows ``` -*Every payload is in an entry, a register, or the ledger; there is no third place.* An entry is the complete conversation record — placement and payload in one row. A register holds its current typed value directly; overwriting discards the old value, and deletion removes the key. Complete content that durably exists before tree placement—queued input, deferred writes, finalized out-of-order tool results—waits in `pending.entry` and becomes an entry in the transaction that places it. Auxiliary tool progress may occupy `pending.tool_output` only while its effect is uncertain. Per-backend projections — branch index, full-text search, stats — are rebuildable from the three stores and carry no authority. +*Every payload is in an entry, a bound value/list, or the ledger; there is no third place.* An entry is the complete conversation record—placement and payload in one row. A `Value` holds its current value directly; replacement discards the old value, and deletion removes it. A `ValueList` holds immutable elements ordered by their write sequence; appending never reads or rewrites earlier elements, and deletion removes the whole list. Complete content that durably exists before tree placement—queued input, deferred writes, finalized out-of-order tool results—waits in `pi.pending.entry` and becomes an entry in the transaction that places it. Auxiliary tool progress may occupy `pi.pending.tool_output` only while its effect is uncertain; streamed assistant frames occupy the `pi.pending.assistant_frame` list only while their response is effect-pending (§3.7). Per-backend projections — branch index, full-text search, stats — are rebuildable from the three stores and carry no authority. -**2. Atomic transactions.** A transaction is a set of entry inserts, usage inserts, and register writes (set or delete), committed all-or-none with strictly increasing sequence numbers. There is no crash state inside a transaction. This is the only write primitive. +**2. Atomic transactions.** A transaction is a set of entry inserts, usage inserts, value writes (set or delete), and list writes (append or whole-list delete), committed all-or-none with strictly increasing sequence numbers. There is no crash state inside a transaction. This is the only write primitive. -**3. The durable restart point.** After every durable transition, the harness overwrites one register — `op.state/{operationId}` — with the *complete* current state of the operation. While a process-local operation task is alive, ordinary JavaScript control flow may continue across that state; after the task is lost, recovery reads the register and starts at the procedure responsible for it. Recovery never replays a journal or infers position from what is missing. The state is *total* — it never depends on a previous state. Small captured values (configuration, stream options, retry policy) are inline; large stable payloads live in sibling `op.*` registers or are named by id. When the operation ends, the terminal transaction deletes its registers: a finished session holds exactly the conversation, the ledger, and a handful of lane and fact registers. There is no dead state to collect. +**3. The durable restart point.** After every durable transition, the harness replaces `operationState(operationId)` with the *complete* current state of the operation. While a process-local operation task is alive, ordinary JavaScript control flow may continue across that state; after the task is lost, recovery reads that value and starts at the procedure responsible for it. Recovery never replays a journal or infers position from what is missing. The state is *total* — it never depends on a previous state. Small captured values (configuration, stream options, retry policy) are inline; large stable payloads live at sibling operation-owned addresses or are named by id. When the operation ends, the terminal transaction deletes those values/lists: a finished session holds exactly the conversation, the ledger, and a handful of lane and semantic session values. There is no dead state to collect. **4. The effect sandwich.** Provider requests and real tool calls are wrapped in two commits: @@ -149,37 +152,40 @@ lane.prompt("what changed in auth last week?") What happens, in order: -1. **Acceptance.** The harness validates, runs the `before_run` hook, and commits one transaction: the user-message entry, the operation's `op.meta` register, and its first `op.state` — *"I am at a checkpoint, and I need an assistant response."* +1. **Acceptance.** The harness validates, runs the `before_run` hook, and commits one transaction: the user-message entry, the operation's metadata value and its first state value — *"I am at a checkpoint, and I need an assistant response."* 2. **Intent.** After an internal ready-state commit, it commits the request intent: *"I am about to make a provider request. The response will be entry `0195c8d1-53a0-7c44-…` and the usage row will be `0195c8d1-53a0-7d18-…`."* Both ids are minted now; nothing has been sent yet. -3. **The request.** Streaming happens. This is the only part that is not durable. -4. **Settlement.** One transaction commits the response entry, its usage row, and the next state: *"the response has tool calls; here is the batch plan, with result ids already assigned."* +3. **The request.** Streaming happens. Each streamed event converts to a compact frame that is appended, without blocking the stream, to the response's `pi.pending.assistant_frame` list (§3.7). The request's *outcome* is the only part that is not durable — frames never prove completion. +4. **Settlement.** One transaction commits the response entry, its usage row, and the next state, and deletes the response's frame list: *"the response has tool calls; here is the batch plan, with result ids already assigned."* 5. Tool calls follow intent → effect → outcome settlement. A finalized outcome becomes durable immediately, then materializes as a tree entry in assistant source order. This distinction matters only when parallel calls finish out of order. -6. When the model stops without tool calls, a terminal transaction deletes the operation's registers, records the outcome in `lane.lastResult`, and leaves the lane idle. +6. When the model stops without tool calls, a terminal transaction deletes the operation's values/lists, records the outcome in `pi.lane.lastResult`, and leaves the lane idle. As a trace (ids abbreviated; every `TX[...]` is one atomic commit, in normative write order): ```text -TX[ insert entry n1 (user msg), upsert lane.leaf = n1, - upsert op.meta/O, upsert op.state/O = checkpoint, - upsert lane.state = { currentOperationId: O } ] -TX[ upsert op.state/O = assistant ready (config snapshot) ] -TX[ upsert op.state/O = effect_pending (reserves response n2, usage u1) ] +TX[ insert entry n1 (user msg), upsert pi.lane.leaf = n1, + upsert pi.op.meta/O, upsert pi.op.state/O = checkpoint, + upsert pi.lane.state = { currentOperationId: O } ] +TX[ upsert pi.op.state/O = assistant ready (config snapshot) ] +TX[ upsert pi.op.state/O = effect_pending (reserves response n2, usage u1) ] … provider streams … ← the uncertain window -TX[ insert entry n2, upsert lane.leaf = n2, insert usage u1, - upsert op.state/O = tools (result id n3 reserved) ] -TX[ upsert op.tool_args/O:s1:0, upsert op.state/O = call 0 effect_pending ] -… tool runs; selected bounded updates may replace pending.tool_output/O:n3 … -TX[ upsert pending.entry/n3 = finalized tool result, - delete pending.tool_output/O:n3, upsert op.state/O = call 0 outcome_ready ] -TX[ insert entry n3, delete pending.entry/n3, upsert lane.leaf = n3, - upsert op.state/O = checkpoint ] +TX[ append pi.pending.assistant_frame/O:n2 += frame ] ← one per streamed event, + enqueued without awaiting +TX[ insert entry n2, upsert pi.lane.leaf = n2, insert usage u1, + delete list pi.pending.assistant_frame/O:n2, + upsert pi.op.state/O = tools (result id n3 reserved) ] +TX[ upsert pi.op.tool_args/O:s1:0, upsert pi.op.state/O = call 0 effect_pending ] +… tool runs; selected bounded updates may replace pi.pending.tool_output/O:n3 … +TX[ upsert pi.pending.entry/n3 = finalized tool result, + delete pi.pending.tool_output/O:n3, upsert pi.op.state/O = call 0 outcome_ready ] +TX[ insert entry n3, delete pi.pending.entry/n3, upsert pi.lane.leaf = n3, + upsert pi.op.state/O = checkpoint ] … second turn: ready · intent · stream · settle (n4, u2) … -TX[ delete op.meta/O, op.state/O, op.tool_args/O:*, - upsert lane.lastResult = { O, completed, n4 }, - upsert lane.state = { currentOperationId: null } ] +TX[ delete pi.op.meta/O, pi.op.state/O, pi.op.tool_args/O:*, + upsert pi.lane.lastResult = { O, completed, n4 }, + upsert pi.lane.state = { currentOperationId: null } ] ``` -Kill the process between any two of those transactions and restart. The harness reads the lane's registers, sees exactly which of those sentences was the last one committed, and continues. If it died in step 3, it knows a request may have been billed and may or may not have produced output — that is the one genuinely uncertain window in the whole system, and there is a stated policy for it. +Kill the process between any two of those transactions and restart. The harness reads the lane's required values, sees exactly which of those sentences was the last one committed, and continues. If it died in step 3, it knows a request may have been billed and may or may not have produced output — that is the one genuinely uncertain window in the whole system, and there is a stated policy for it. The committed frame prefix preserves the latest durable partial for that policy's synthetic settlement and for reconnect display; it never proves how the request ended. Meanwhile a second thread in the same channel is running its own lane, over the same 400 entries of shared history, with no coordination between them. @@ -192,23 +198,23 @@ lane.prompt("delete the stale migrations and run the test suite") The model returns two tool calls. The harness commits the batch plan, then commits `call 0 is about to execute, with these exact arguments, and it declares itself unsafe to replay`. The tool starts deleting files. It emits bounded progress snapshots every 100 ms for live UI observation and requests a durable checkpoint every two seconds. The process is killed after one checkpoint commits. ```text -TX[ insert entry n2 (assistant, 2 calls), insert usage u1, upsert lane.leaf = n2, - upsert op.state/O = tools (result ids n3, n4 reserved) ] -TX[ upsert op.tool_args/O:s1:0, upsert op.state/O = call 0 effect_pending, +TX[ insert entry n2 (assistant, 2 calls), insert usage u1, upsert pi.lane.leaf = n2, + upsert pi.op.state/O = tools (result ids n3, n4 reserved) ] +TX[ upsert pi.op.tool_args/O:s1:0, upsert pi.op.state/O = call 0 effect_pending, replay: "never" ] … tool deletes files; live updates u1 … u19 … -TX[ upsert pending.tool_output/O:n3 = bounded update u1 ] +TX[ upsert pi.pending.tool_output/O:n3 = bounded update u1 ] … live updates u2 … u19 … ← CRASH ``` -On restart the harness performs the bounded restore reads from §4.4; `op.state` is the decisive restart point and says `calls[0].status = "effect_pending", replay = "never"`. It does not re-run the deletion. When activation reconciles the orphan, it uses the latest durable checkpoint when present, appends an explicit warning that newer live output may be missing and the external outcome is unknown, and stages the synthetic error result under the reserved result id: +On restart the harness performs the bounded restore reads from §4.4; `pi.op.state` is the decisive restart point and says `calls[0].status = "effect_pending", replay = "never"`. It does not re-run the deletion. When activation reconciles the orphan, it uses the latest durable checkpoint when present, appends an explicit warning that newer live output may be missing and the external outcome is unknown, and stages the synthetic error result under the reserved result id: ```text -TX[ upsert pending.entry/n3 = synthetic interrupted result containing u1, - delete pending.tool_output/O:n3, - upsert op.state/O = call 0 outcome_ready ] -TX[ insert entry n3, delete pending.entry/n3, upsert lane.leaf = n3, - upsert op.state/O = call 0 completed ] +TX[ upsert pi.pending.entry/n3 = synthetic interrupted result containing u1, + delete pi.pending.tool_output/O:n3, + upsert pi.op.state/O = call 0 outcome_ready ] +TX[ insert entry n3, delete pi.pending.entry/n3, upsert pi.lane.leaf = n3, + upsert pi.op.state/O = call 0 completed ] ``` The conversation stays coherent — every tool call has a result — and nothing ran twice. If no checkpoint committed before the crash, the result contains only the interruption warning. @@ -218,18 +224,18 @@ Had the tool declared `replay: "safe"` (a read, a query), the harness would have ## 0.6 Non-goals - **Exactly-once external effects.** See above. Hooks with their own side effects must be idempotent, keyed by operation id. -- **Provider stream resumption.** Partial streams are process-local, never persisted. A settled response is persisted *completely* before anything classifies it. +- **Provider stream resumption.** The harness never reattaches to a provider stream. Committed assistant frames (§3.7) preserve the latest durable partial for recovery and reconnect display, but a settled response is persisted *completely* before anything classifies it. - **Multiple writers.** One process per session. The serving layer routes accordingly, and the SQLite backend enforces it with a fenced lease (§1.7). Lanes cover the workload that looks like multi-writer. - **Work scheduling.** The harness never creates platform alarms, scans repositories for abandoned sessions, leases hosted submissions, or promises an HTTP receipt. It reports durable waits and safe yields through `drive`; the serving layer decides when and where to call it again. - **Replication.** A session lives in one place. -- **Durable write history.** Registers hold only current values: an overwritten register is gone, and no API or table exposes write history. Order-of-write assertions in tests use an instrumented storage decorator around `commit()` (Part 9); production auditing belongs to the telemetry layer (§5.8). -- **Deletion as a runtime feature.** Entries and usage rows are never deleted: compaction changes provider context, not storage, and terminal cleanup deletes only registers. Note that `retainedTail` copies old messages forward into newer compaction entries and summaries derive from old content, so compaction is not erasure either. Compliance-grade "erase this" is the administrative precise rewrite (§2.9), the sole sanctioned exception. +- **Durable write history.** Replaceable values retain only current state; lists retain elements only until whole-list deletion. No API or table exposes replaced values or deleted elements. Order-of-write assertions in tests use an instrumented storage decorator around `commit()` (Part 9); production auditing belongs to the telemetry layer (§5.8). +- **Deletion as a runtime feature.** Entries and usage rows are never deleted: compaction changes provider context, not storage, and terminal cleanup deletes only values/lists. Note that `retainedTail` copies old messages forward into newer compaction entries and summaries derive from old content, so compaction is not erasure either. Compliance-grade "erase this" is the administrative precise rewrite (§2.9), the sole sanctioned exception. ## 0.7 Notation and source types -- `TX[ a, b, c ]` — one atomic commit containing writes `a`, `b`, `c` in that order. The write vocabulary is `insert entry`, `insert usage`, `upsert namespace/key = value`, and `delete namespace/key`. +- `TX[ a, b, c ]` — one atomic commit containing writes `a`, `b`, `c` in that order. The write vocabulary is `insert entry`, `insert usage`, `setValue(address, value)`, `deleteValue(address)`, `appendList(address, element)`, and `deleteList(address)`. Internal transaction traces may abbreviate a bound address as its persisted `namespace/key`; this is never an API signature or a second key argument (§1.3). - Ids are UUIDv7s (§1.2). Examples abbreviate them: short tags — `e_*` entry ids, `u_*` usage ids, `op_*` operation ids — stand in for full ids where the time prefix is irrelevant; where the prefix matters, examples show it (`0195c8d1-4a2e-7b31-…`). -- `S(next)` — overwrite the `op.state/{operationId}` register with the next total operation state. `L(next)` — the same for `lane.state/{lane}`. +- `S(next)` — overwrite the `operationState(operationId)` value with the next total operation state. `L(next)` — the same for `laneState(lane)`. - **must / must not** are normative. Everything else is explanation. Source type provenance: @@ -237,6 +243,7 @@ Source type provenance: - `AgentMessage`, `AgentTool`, `AgentToolResult`, `QueueMode`, and `ThinkingLevel`: `packages/agent/src/types.ts`. - `Skill`, `PromptTemplate`, `AgentHarnessResources` (`Resources` below), `AgentHarnessTool`, `AgentHarnessToolInvocation`, `AgentHarnessToolUpdateCallback`, `AgentHarnessToolUpdateOptions`, `AgentHarnessStreamOptions`, and `AgentHarnessStreamOptionsPatch`: `packages/agent/src/harness/types.ts`. - `Model`, `Models`, `Usage`, `RetryPolicy`, `StopReason`, `AssistantMessage`, `ImageContent`, provider messages, stream options, and deferred handles: `packages/ai`. +- `AssistantMessageFrame`, `assistantMessageEventToFrame`, and `reduceAssistantMessageFrames`: `packages/ai` (`@earendil-works/pi-ai`), `src/utils/assistant-message-frame.ts`. The harness defines no second frame codec or reducer. - `CompactionSettings`, `CompactionPreparation`, `CompactResult`, `BranchPreparation`, and `BranchSummaryResult`: `packages/agent/src/harness/compaction/`. Existing preparation and split-turn algorithms remain the implementation starting point unless this document explicitly changes them. - `TelemetryContext` and typed schema helpers: `packages/telemetry`; the agent-owned schemas remain in `packages/agent/src/harness/telemetry.ts`. @@ -262,7 +269,7 @@ Runtime schema validation belongs at untrusted wire boundaries, before remote in # Part 1 — Storage -Storage knows nothing about agents, lanes, or conversations. It stores entries and usage rows, updates registers, and answers a small fixed set of queries. Parts 2–4 are built entirely on this. +Storage knows nothing about agents, lanes, or conversations. It stores entries and usage rows, updates bound values/lists, and answers a small fixed set of queries. Parts 2–4 are built entirely on this. ## 1.1 The model @@ -284,13 +291,34 @@ interface EntryBase { type EntryType = "message" | "compaction" | "branch_summary" | "custom"; -/** The only mutable store. A namespaced key holding its current typed value - directly. Overwrite replaces the value; delete removes the key. */ -interface Register { - namespace: N; - key: string; - value: RegisterValues[N]; - seq: number; // seq of the write that last set this register +/** The only mutable store, addressed by bound typed addresses (§1.3). */ +declare const storedValueType: unique symbol; + +interface StoredAddressBase { + readonly namespace: string; + readonly key: string; + readonly kind: "value" | "list"; +} +interface Value extends StoredAddressBase { + readonly kind: "value"; + readonly [storedValueType]?: (value: T) => T; +} +interface ValueList extends StoredAddressBase { + readonly kind: "list"; + readonly [storedValueType]?: (value: T) => T; +} + +function value(namespace: string, key = ""): Value; +function list(namespace: string, key = ""): ValueList; + +interface StoredValue { + address: Value; + value: T; + seq: number; // seq of the write that last set this value +} +interface ListElement { + seq: number; // global write seq assigned to the append + value: T; } /** Append-only cost ledger row. Never modified, never deleted (§1.6). */ @@ -314,33 +342,120 @@ Minting rules: 2. **Tool-result ids inherit their assistant id's timestamp** (`idGenerator.next(timestampMs?)`, fresh random tail), so a call-and-results group is time-cohesive under id order even across a midnight boundary. 3. Synthetic settlements write under already-reserved ids (§4.5) — no special case. -**Opaque payloads** — custom entry `data`, `details`, `fact.custom` values, message text, hook `resumeData` — may embed entry ids. The harness never tracks those references and they may go stale; copy content, don't reference it. +**Opaque payloads** — custom entry `data`, application-defined values, `details`, message text, and hook `resumeData` — may embed entry ids. The harness never tracks those references and they may go stale; copy content, don't reference it. **Absolutes.** Within a session, entries and usage rows are never deleted — the precise rewrite (§2.9) is the sole exception. A missing parent is always corruption. -## 1.3 Register namespaces +## 1.3 Bound values and lists + +The public storage abstraction is a **bound typed address**. `value(namespace, key?)` names one replaceable durable value. `list(namespace, key?)` names one append-only durable list whose elements have type `T`. Namespace and key are bound once; every later read or write receives only the address. There is no global value-type map, token catalog, declaration merging, or separate application-state storage mechanism. ```ts -interface RegisterValues { - "lane.leaf": string | null; // entry id; null = lane at the root - "lane.config": LaneConfiguration; // §2.3 - "lane.state": LaneState; // §3.3 - "lane.lastResult": LaneLastResult; // §3.13 - "op.meta": Operation; // §3.1 - "op.state": OperationState; // §3.2 — durable restart point - "op.tool_args": Record; // effective tool arguments (§3.8) - "op.tool_fact": JsonValue; // invocation-scoped durable memo (§3.8) - "op.preparation": DurableStructuralPreparation; // §3.9 - "pending.entry": PendingEntry; // unplaced complete content (§2.2) - "pending.tool_output": AgentToolResult; // bounded progress checkpoint (§3.8) - "fact.name": string; - "fact.label": string; - "fact.custom": JsonValue; // JSON null is a legal value -} -type RegisterNamespace = keyof RegisterValues; +const state = value("my-app.state"); +const events = list("my-app.events"); +await session.getValue(state); +await session.setValue(state, nextState); +await session.readList(events, { limit: 100 }); +await session.appendList(events, event); +``` + +Dynamic locations use small owner-defined address constructors. Core and applications use the same universal `value()` and `list()` constructors. Namespace `pi` and every `pi.*` namespace are reserved for built-ins by contract; application misuse is a trusted-programming defect, and constructors perform no namespace-ownership check: + +```ts +// packages/agent/src/harness/session/values.ts +export const laneLeaf = (lane: string) => + value("pi.lane.leaf", lane); +export const laneConfig = (lane: string) => + value("pi.lane.config", lane); +export const laneState = (lane: string) => + value("pi.lane.state", lane); +export const laneLastResult = (lane: string) => + value("pi.lane.lastResult", lane); +/** Used only by scanValues() to enumerate configured lane names. */ +export const laneLeafInventoryPrefix = () => + value("pi.lane.leaf"); + +export const operationMeta = (operationId: string) => + value("pi.op.meta", operationId); +export const operationState = (operationId: string) => + value("pi.op.state", operationId); +export const operationToolArgs = ( + operationId: string, + stepId: string, + sourceIndex: number, +) => value>( + "pi.op.tool_args", + `${operationId}:${stepId}:${sourceIndex}`, +); +export const operationToolMemo = ( + operationId: string, + invocationId: string, + name: string, +) => value("pi.op.tool_memo", `${operationId}:${invocationId}:${name}`); +export const operationPreparation = (operationId: string, taskId: string) => + value( + "pi.op.preparation", + `${operationId}:${taskId}`, + ); + +/** Prefix addresses are valid only as scanValues() inputs. */ +export const operationToolArgsPrefix = (operationId: string, stepId?: string) => + value>( + "pi.op.tool_args", + stepId === undefined ? `${operationId}:` : `${operationId}:${stepId}:`, + ); +export const operationToolMemoPrefix = (operationId: string, invocationId?: string) => + value( + "pi.op.tool_memo", + invocationId === undefined ? `${operationId}:` : `${operationId}:${invocationId}:`, + ); +export const operationPreparationPrefix = (operationId: string) => + value("pi.op.preparation", `${operationId}:`); + +export const pendingEntry = (entryId: string) => + value("pi.pending.entry", entryId); +export const pendingToolOutput = (operationId: string, invocationId: string) => + value>( + "pi.pending.tool_output", + `${operationId}:${invocationId}`, + ); +export const pendingAssistantFrames = ( + operationId: string, + responseEntryId: string, +) => list( + "pi.pending.assistant_frame", + `${operationId}:${responseEntryId}`, +); +export const pendingToolOutputPrefix = (operationId: string) => + value>( + "pi.pending.tool_output", + `${operationId}:`, + ); + +export const sessionName = value("pi.session.name"); +export const entryLabel = (entryId: string) => + value("pi.entry.label", entryId); +// Applications define their own value()/list() addresses directly. +``` + +The complete rules are: + +- `namespace` must be non-empty; neither component may contain `\u0000`; +- namespace `pi` and every `pi.*` namespace are reserved for built-ins by contract; application use is a trusted-programming defect; +- core and applications use one `value()` and `list()` constructor path with no runtime address registry or catalog; +- every built-in namespace starts with `pi.`, enforced by exact constructor tests rather than runtime privilege checks; +- an empty key is legal and naturally addresses one session-wide value or list; +- object identity has no durable meaning; equal `(kind, namespace, key)` triples name the same location; +- constructing one location with incompatible TypeScript types is a trusted-programming defect; +- value and list addresses may not share one `(namespace, key)` in a storage version; +- changing namespace, key grammar, kind, or incompatible value shape requires migration (§7.4); +- built-in constructors live in `session/values.ts` and are imported directly; there is no runtime catalog or dependency-injection bundle; +- later operations never accept another key after address construction. + +```ts /** Unplaced content: current mutable state until the placement transaction - writes the complete entry and deletes this register (§2.2). */ + writes the complete entry and deletes this value (§2.2). */ type PendingEntry = | { type: "message"; payload: AgentMessage } | { type: "custom"; customType: string; payload?: JsonValue }; @@ -358,67 +473,77 @@ type DurableStructuralPreparation = fileOps: DurableFileOperations; totalTokens: number }; ``` -| Namespace | Key | Value | Meaning | -|---|---|---|---| -| `lane.leaf` | lane name | entry id or `null` | where this lane appends next | -| `lane.config` | lane name | `LaneConfiguration` | total lane configuration | -| `lane.state` | lane name | `LaneState` (§3.3) | `currentOperationId`, `pendingNextRun` | -| `lane.lastResult` | lane name | `LaneLastResult` (§3.13) | terminal outcome of the lane's most recent operation | -| `op.meta` | operation id | `Operation` (§3.1) | acceptance data; written once, never overwritten | -| `op.state` | operation id | `OperationState` (§3.2) | total operation state — the **durable restart point** | -| `op.tool_args` | `{opId}:{stepId}:{sourceIndex}` | effective arguments | written once at tool clearance (§3.8) | -| `op.tool_fact` | `{opId}:{invocationId}:{factName}` | `JsonValue` | invocation-scoped durable memo; only while the call is effect-pending (§3.8) | -| `op.preparation` | `{opId}:{taskId}` | `DurableStructuralPreparation` | written once before the decision hook (§3.9) | -| `pending.entry` | reserved entry id | `PendingEntry` | complete content awaiting placement, including outcome-ready tool results (§2.2, §3.8) | -| `pending.tool_output` | `{opId}:{invocationId}` | `AgentToolResult` | latest tool-requested bounded progress checkpoint (§3.8) | -| `fact.name` | `""` | string | session name | -| `fact.label` | entry id | string | entry label | -| `fact.custom` | application key | `JsonValue` | application state | +| Address constructor | Kind | Persisted namespace and key | Value | Meaning | +|---|---|---|---|---| +| `laneLeaf(lane)` | value | `pi.lane.leaf`, lane | entry id or `null` | where this lane appends next | +| `laneConfig(lane)` | value | `pi.lane.config`, lane | `LaneConfiguration` | total lane configuration | +| `laneState(lane)` | value | `pi.lane.state`, lane | `LaneState` (§3.3) | `currentOperationId`, `pendingNextRun` | +| `laneLastResult(lane)` | value | `pi.lane.lastResult`, lane | `LaneLastResult` (§3.13) | latest terminal outcome | +| `operationMeta(opId)` | value | `pi.op.meta`, operation id | `Operation` (§3.1) | acceptance data; written once | +| `operationState(opId)` | value | `pi.op.state`, operation id | `OperationState` (§3.2) | total durable restart point | +| `operationToolArgs(opId, stepId, sourceIndex)` | value | `pi.op.tool_args`, `{opId}:{stepId}:{sourceIndex}` | effective arguments | written once at clearance | +| `operationToolMemo(opId, invocationId, name)` | value | `pi.op.tool_memo`, `{opId}:{invocationId}:{name}` | `JsonValue` | invocation-scoped durable memo | +| `operationPreparation(opId, taskId)` | value | `pi.op.preparation`, `{opId}:{taskId}` | `DurableStructuralPreparation` | structural preparation | +| `pendingEntry(entryId)` | value | `pi.pending.entry`, reserved entry id | `PendingEntry` | complete content awaiting placement | +| `pendingToolOutput(opId, invocationId)` | value | `pi.pending.tool_output`, `{opId}:{invocationId}` | `AgentToolResult` | latest bounded progress checkpoint | +| `pendingAssistantFrames(opId, responseEntryId)` | list | `pi.pending.assistant_frame`, `{opId}:{responseEntryId}` | `AssistantMessageFrame` elements | committed stream-frame prefix | +| `sessionName` | value | `pi.session.name`, empty key | string | session name | +| `entryLabel(entryId)` | value | `pi.entry.label`, entry id | string | entry label | -That is the complete set. Two lifetimes are visible in the key shape: +The five exported scan-prefix constructors—`laneLeafInventoryPrefix`, `operationToolArgsPrefix`, `operationToolMemoPrefix`, `operationPreparationPrefix`, and `pendingToolOutputPrefix`—encapsulate lane inventory and operation-cleanup grammar. Their results are valid only as namespace-scoped `scanValues()` inputs, never exact get/set/delete addresses. + +These lifetimes are reflected in the address grammars: ```text -lane.* fact.* session-lived; facts are deleted only by explicit application action -op.* operation-lived; deleted no later than the terminal transaction (§3.13) -pending.entry lives until its content is placed, cancelled, or terminally discarded -pending.tool_output exists only while its call is effect-pending +pi.lane.* pi.session.* pi.entry.* session-lived semantic values +pi.op.* operation-lived; deleted no later than the terminal transaction (§3.13) +pi.pending.entry lives until placement, cancellation, or terminal cleanup +pi.pending.tool_output exists only while its invocation is effect-pending +pi.pending.assistant_frame exists only while its response is effect-pending ``` -- `op.meta` and `op.preparation` keys are written exactly once; `op.tool_args` keys are written once per call. `op.tool_fact` keys are invocation-scoped current values used for durable memoization and die when the invocation reaches `outcome_ready`. Every `op.*` key is deleted no later than the terminal transaction. -- Operation-owned `pending.entry` registers still unconsumed at the end include remaining inbox/drained items and any staged tool outcomes; the terminal transaction deletes them. A pending item's register normally dies in its placement transaction. Lane-owned `pendingNextRun` values outlive operations and die only when consumed or cancelled (§3.11). -- `pending.tool_output` is optional auxiliary state. A tool selects bounded complete updates with `checkpoint:true`; the harness replaces the scalar under invocation fencing. Outcome settlement deletes it atomically. Safe replay deletes it before re-execution; unsafe recovery may consume it into an interrupted result. -- `lane.lastResult` is written only by terminal transactions and overwritten by the next one on its lane — one bounded register per lane, forever. Recovery never reads it; it exists so an application that accepted an operation, crashed, and reopened can still learn its outcome (§3.13). -- Deleting a fact removes its register. Storing JSON `null` in `fact.custom` is a different, legal state; there are no tombstones. -- Cancellations leave no trace: `cancelQueued` triages as pending → `cancelled`, entry exists → `already_consumed`, else → `not_found` (§3.11). A client retrying a lost cancel treats `not_found` as success. +- `pi.op.meta` and `pi.op.preparation` values are written exactly once; `pi.op.tool_args` once per call. Invocation memos die when the invocation reaches `outcome_ready`. Every `pi.op.*` value is deleted no later than the terminal transaction. +- Operation-owned pending entries still unconsumed at the end include inbox/drained items and staged tool outcomes. Lane-owned `pendingNextRun` values outlive operations and die only when consumed or cancelled (§3.11). +- Tool output is optional auxiliary state. Outcome staging deletes it atomically; safe replay deletes it before re-execution; unsafe recovery may consume it into an interrupted result. +- Assistant frames are auxiliary list elements ordered by global write `seq`. A missing list is valid. Frames never prove request admission, completion, or failure and never select a restart point. Response settlement deletes the exact bound list atomically (§3.7). +- `pi.lane.lastResult` is overwritten only by terminal transactions. Recovery never reads it. +- Deleting a bound value removes it; JSON `null` remains distinct from absence when the address type permits it. ## 1.4 Transactions ```ts -/** Mapped discriminated union: the namespace forces the value type. */ -type RegisterSetWrite = { - [N in RegisterNamespace]: { kind: "register"; op: "set"; namespace: N; - key: string; value: RegisterValues[N] } -}[RegisterNamespace]; - +/** Erased storage representations. Constructed only through helpers that + check the bound address/value relationship before erasure. */ type Write = | { kind: "entry"; entry: Omit } | { kind: "usage"; row: Omit } - | RegisterSetWrite - | { kind: "register"; op: "delete"; namespace: RegisterNamespace; key: string }; + | { kind: "value"; op: "set"; namespace: string; key: string; value: unknown } + | { kind: "value"; op: "delete"; namespace: string; key: string } + | { kind: "list"; op: "append"; namespace: string; key: string; value: unknown } + | { kind: "list"; op: "delete"; namespace: string; key: string }; + +function setValue(address: Value, next: NoInfer): Write; +function deleteValue(address: Value): Write; +function appendList(address: ValueList, element: NoInfer): Write; +function deleteList(address: ValueList): Write; interface Transaction { writes: Write[] } interface CommitResult { firstSeq: number; seqs: number[]; timestamp: number } ``` +The raw write shapes are storage internals; harness and application code use the helpers. Value helpers cannot target list addresses and list helpers cannot target value addresses. `NoInfer` makes the address authoritative instead of widening `T` from an incompatible write value. + Rules: 1. A transaction commits **all-or-none**. There is no observable state in which some of its writes exist and others do not. -2. Writes receive **strictly increasing** `seq` values in the order given; gaps are legal, within and between transactions. `seq` is monotonic session-wide across all lanes and all write kinds. A register `set` stamps the register with its assigned `seq`. -3. Within a transaction, writes apply in order: an entry may name a parent created earlier in the same transaction; a register value may reference entry or usage ids created earlier in the same transaction. A placement transaction inserts the complete entry and deletes its `pending.entry` register together (§2.2) — there is never a moment where both exist. +2. Writes receive **strictly increasing** `seq` values in the order given; gaps are legal, within and between transactions. `seq` is monotonic session-wide across all lanes and all write kinds. A value `set` stamps the stored value with its assigned `seq`. +3. Within a transaction, writes apply in order: an entry may name a parent created earlier in the same transaction; a stored value may reference entry or usage ids created earlier in the same transaction. A placement transaction inserts the complete entry and deletes its `pendingEntry(id)` value together (§2.2) — there is never a moment where both exist. 4. Entry and usage ids share one session-wide id namespace. Writing either kind under any existing id is **corruption**, not an update. -5. A register `set` with the same `(namespace, key)` replaces the current value; `delete` removes the key; a later `set` recreates it. No history is retained. A `delete` naming an absent key is a no-op, so public deletions such as clearing an unset label stay legal. -6. Transactions on one session are **serialized**. There is one writer and one queue. +5. A scalar `set` with the same `(namespace, key)` replaces the current value; `delete` removes the key; a later `set` recreates it. No history is retained. A `delete` naming an absent key is a no-op, so public deletions such as clearing an unset label stay legal. +6. One list `append` carries one element and never reads existing elements. An element is immutable after commit and ordered by its assigned write `seq`; gaps from unrelated writes are irrelevant. There is no per-element update, delete, insertion, or truncation. +7. A list `delete` removes every element under `(namespace, key)`; deleting an absent list is a no-op; `delete` followed by `append` in one transaction atomically creates a fresh list. “Append-only” describes elements while the key exists — whole-key deletion is lifecycle cleanup, not element mutation. +8. Transactions on one session are **serialized**. There is one writer and one queue. Session passes typed transactions directly to storage without a codec, runtime shape validation, or defensive cloning. A failed admitted commit **faults the harness**: all effects stop, all calls reject, and the process must be restarted. A partially applied transaction is not tolerated. @@ -432,12 +557,11 @@ interface Storage { getEntries(ids: string[]): Promise>; - getRegister(namespace: N, key: string): - Promise | undefined>; - /** keyPrefix is ideally an indexed prefix listing over (namespace, key); - terminal cleanup's op.* prefix scans use it (§3.13). */ - listRegisters(namespace: N, keyPrefix?: string): - Promise[]>; + getValue(address: Value): Promise | undefined>; + /** Internal namespace-scoped prefix scan. The bound address key is the prefix. */ + scanValues(prefix: Value): Promise[]>; + readList(address: ValueList, options?: ListReadOptions): + Promise[]>; scanBranch(q: BranchScan): Promise; // §2.5 scanBranchStructure(q: BranchScan): Promise; @@ -461,17 +585,26 @@ interface UsageScan { fromSeq?: number; toSeq?: number; order?: "asc" | "desc"; limit?: number; } + +interface ListCursor { seq: number } +interface ListReadOptions { + cursor?: ListCursor; // exclusive + order?: "asc" | "desc"; // default "asc" + limit?: number; // positive safe integer; default 1,000, cap 10,000 +} ``` -There is deliberately no cross-namespace register scan and no durable write log. Restore, facts, forks, and execution follow exact ids and keys; entry inventory uses `scanEntries`; ledger reads use `scanUsage`; totals use the stats projection (§1.6); test-order assertions wrap `commit()` with the instrumented-storage decorator (Part 9); production auditing belongs to telemetry (§5.8). +List read semantics: ascending reads return `seq > cursor.seq`, descending reads `seq < cursor.seq`; results are ordered before `limit` applies; absent and empty keys both return `[]`; callers continue with the last returned element's `seq`, and an empty page ends iteration. A cursor is a sequence filter, not a snapshot or key-incarnation token: concurrent later appends may appear on later ascending pages, and after a whole-key delete a read simply applies the comparison to surviving elements. There is deliberately no unbounded “read the whole list” helper. -Recovery and execution reads must be index-driven and bounded. They may not infer state from an absent value, and there is no register history to fold. Exact dereference is allowed: one current state may name a bounded set of entries and registers, fetched in one batch without order-dependent reduction. Public inventory and debugging APIs may intentionally read more than a hot path; their `limit`/pagination behavior is explicit at the `SessionTree` layer. +`scanValues(prefixAddress)` is namespace-scoped and interprets the bound key as a prefix. Core inventory/cleanup uses only `laneLeafInventoryPrefix`, `operationToolArgsPrefix`, `operationToolMemoPrefix`, `operationPreparationPrefix`, and `pendingToolOutputPrefix`; core call sites do not repeat raw reserved namespace/key grammar. Ordinary reads use exact addresses. There is no cross-namespace value dump or durable write log. Entry inventory uses `scanEntries`; ledger reads use `scanUsage`; totals use the stats projection (§1.6); test-order assertions wrap `commit()` with the instrumented-storage decorator (Part 9). + +Recovery and execution reads must be index-driven and bounded. They may not infer state from an absent value, and there is no value history to fold. Exact dereference is allowed: current typed state may name a bounded set of entries and bound values. An exact list address derived from current state may be read in bounded sequence pages and reduced by its consumer—assistant frames use pi-ai's `reduceAssistantMessageFrames` (§3.7). Base restore never reads lists (§4.4). Public inventory and debugging APIs expose explicit limits/pagination at the `SessionTree` layer. `close()` is idempotent. It seals admission, rejects later reads/commits on that instance, drains commits admitted before the seal, then releases resources and the writer claim. Durable data is reopened through the repository. ## 1.6 Usage ledger -Every settled provider attempt writes one `UsageRow` — successful, failed, retried, and synthetic attempts alike, including attempts whose operation later aborts. Settlement transactions write the response entry and its usage row together (§3.7); synthetic settlements write zero usage under the reserved usage id. Rows are append-only: terminal cleanup deletes an operation's registers but never its ledger rows, so billing survives everything that can happen to orchestration state. +Every settled provider attempt writes one `UsageRow` — successful, failed, retried, and synthetic attempts alike, including attempts whose operation later aborts. Settlement transactions write the response entry and its usage row together (§3.7); synthetic settlements write zero usage under the reserved usage id. Rows are append-only: terminal cleanup deletes an operation's values/lists but never its ledger rows, so billing survives everything that can happen to orchestration state. ```jsonc { "id": "u_7", "seq": 815, "entryId": "e_51", "adjustment": false, @@ -491,11 +624,12 @@ Three encodings of one model ship now — Memory, JSONL, SQLite — and all thre ```ts entries: Map -registers: Map // key: `${namespace}\u0000${key}` +scalarValues: Map> // physical key: namespace + separator + key +listValues: Map[]> // same physical key shape usage: Map ``` -One queue serializes commits. A commit checks storage invariants, assigns sequence numbers and the transaction timestamp, then applies the writes synchronously. A register delete is a map delete. Reads are map lookups; `scanBranch` walks `parentId` and filters in RAM. Memory retains and returns typed values directly without defensive cloning. There is no log: Memory holds exactly the live state and nothing else. +One queue serializes commits. A commit checks storage invariants, assigns sequence numbers and the transaction timestamp, then applies the writes synchronously; all validation and serialization needed to admit a transaction completes before any map mutates. A scalar-value delete is a map delete; a list append pushes the already-sequenced element; a whole-key list delete removes the array; a list read filters by the exclusive cursor and slices to the validated limit. Reads are map lookups; `scanBranch` walks `parentId` and filters in RAM. Memory retains and returns typed values directly without defensive cloning. There is no log: Memory holds exactly the live state and nothing else. ### JSONL @@ -504,30 +638,40 @@ The file is not the state; it is the **replay recipe** for the Memory maps above ```jsonl {"v":4,"kind":"header","id":"s_1","storageVersion":1,"createdAt":1700000000000,"cwd":"..."} [{"kind":"entry","seq":101,"timestamp":1700000000000,"id":"e_50","parentId":"e_41","type":"message","message":{"role":"user","content":[...]}}, - {"kind":"register","op":"set","seq":102,"namespace":"op.meta","key":"op_9","value":{...}}, - {"kind":"register","op":"set","seq":103,"namespace":"op.state","key":"op_9","value":{...}}, - {"kind":"register","op":"set","seq":104,"namespace":"lane.leaf","key":"main","value":"e_50"}, - {"kind":"register","op":"set","seq":105,"namespace":"lane.state","key":"main","value":{...}}] -{"kind":"usage","seq":110,"id":"u_7","entryId":"e_51","adjustment":false,"usage":{...}} -{"kind":"register","op":"delete","seq":131,"namespace":"op.state","key":"op_9"} + {"kind":"value","op":"set","seq":102,"namespace":"pi.op.meta","key":"op_9","value":{...}}, + {"kind":"value","op":"set","seq":103,"namespace":"pi.op.state","key":"op_9","value":{...}}, + {"kind":"value","op":"set","seq":104,"namespace":"pi.lane.leaf","key":"main","value":"e_50"}, + {"kind":"value","op":"set","seq":105,"namespace":"pi.lane.state","key":"main","value":{...}}] +{"kind":"value","op":"set","seq":106,"namespace":"pi.op.state","key":"op_9","value":{"phase":"assistant-ready",...}} +{"kind":"value","op":"set","seq":107,"namespace":"pi.op.state","key":"op_9","value":{"phase":"assistant-effect-pending","responseEntryId":"e_51","usageId":"u_7",...}} +{"kind":"list","op":"append","seq":108,"namespace":"pi.pending.assistant_frame","key":"op_9:e_51","value":{"type":"text_delta","contentIndex":0,"delta":"hi"}} +[{"kind":"entry","seq":109,"timestamp":1700000000100,"id":"e_51","parentId":"e_50","type":"message","message":{"role":"assistant","content":[...]}}, + {"kind":"value","op":"set","seq":110,"namespace":"pi.lane.leaf","key":"main","value":"e_51"}, + {"kind":"usage","seq":111,"id":"u_7","entryId":"e_51","adjustment":false,"usage":{...}}, + {"kind":"list","op":"delete","seq":112,"namespace":"pi.pending.assistant_frame","key":"op_9:e_51"}, + {"kind":"value","op":"set","seq":113,"namespace":"pi.op.state","key":"op_9","value":{...}}] +[{"kind":"value","op":"delete","seq":131,"namespace":"pi.op.meta","key":"op_9"}, + {"kind":"value","op":"delete","seq":132,"namespace":"pi.op.state","key":"op_9"}, + {"kind":"value","op":"set","seq":133,"namespace":"pi.lane.lastResult","key":"main","value":{...}}, + {"kind":"value","op":"set","seq":134,"namespace":"pi.lane.state","key":"main","value":{"currentOperationId":null,"pendingNextRun":[]}}] ``` - This is format 4. The incompatible format-4 code currently in the source tree is unfinished and is replaced in place; no migration for it is required. Coding-agent format 3 remains supported (Appendix B). -- Open replays lines in order into the Memory maps: entries and usage rows accumulate; a later register `set` overwrites the key, `delete` removes it. That is *decoding*, not recovery logic. Open verifies persisted sequence monotonicity — strictly increasing, gaps legal (§1.4) — and timestamps, and never regenerates committed timestamps. All queries then run in RAM. +- Open replays lines in order into the Memory maps: entries and usage rows accumulate; a later scalar `set` overwrites the key, `delete` removes it; a list `append` adds `{ seq, value }` under its key and a list `delete` removes the whole key. That is *decoding*, not recovery logic. Open verifies persisted sequence monotonicity — strictly increasing, gaps legal (§1.4) — and timestamps, and never regenerates committed timestamps. All queries then run in RAM. - **A torn final line is discarded whole**, including every element of an array, and is truncated before new writes are admitted. This is what makes "no crash prefix inside a transaction" true here. - A malformed *interior* line or invalid transaction framing is corruption. Superseded old-shape register lines from before a schema migration replay as keyed raw JSON so the migration can convert the current value (Part 7); compaction retires the old bytes. Externally edited shape-invalid data is unsupported rather than runtime-validated on read. - Durability is process-crash level: a resolved `commit()` survives process death. No fsync promise. -- Optional: retain `(offset, length)` per entry and load payloads lazily, keeping only structure and registers resident. Do this only if profiling demands it. +- Optional: retain `(offset, length)` per entry and load payloads lazily, keeping only structure and current values/lists resident. Do this only if profiling demands it. -**Snapshot compaction.** In SQLite a register `set` is an in-place upsert — a 30-turn run leaves one `op.state` row and then zero. In JSONL every `set` appends, so the same run appends ~10 full `op.state` lines, all dead the moment the terminal `delete` line lands: the file grows with *write history* even though the logical state does not. The fix is rewriting the file as `header + current entries + current registers + usage rows`, via temp file + atomic rename; surviving lines keep their original `seq` values, and the gaps the dropped lines leave are legal (§1.4), so compaction needs no renumbering machinery. For a four-entry run: +**Snapshot compaction.** In SQLite a value `set` is an in-place upsert—a 30-turn run leaves one `pi.op.state` row and then zero. In JSONL every `set` appends, so the same run appends ~10 full `pi.op.state` lines, all dead when the terminal `delete` lands: the file grows with *write history* even though logical state does not. The fix is rewriting the file as `header + current entries + current values + surviving list elements + usage rows`, via temp file + atomic rename; surviving lines keep their original `seq` values, and the gaps the dropped lines leave are legal (§1.4), so compaction needs no renumbering machinery. Each surviving list element is rewritten as an append record carrying its original `seq`, merged in sequence order — never collapsed into one synthetic append and never renumbered, so list cursors survive compaction. Deleted lists produce no snapshot records; the sequence high-water mark is preserved through the existing mechanism so dropping a trailing delete line cannot permit sequence reuse. For a four-entry run: ```text -before compaction: ~10 transaction lines, ~27 writes — op.state revisions, +before compaction: ~10 transaction lines, ~27 writes — pi.op.state revisions, tool args, pending payloads, all dead since the terminal line -after compaction: header + 4 entry lines + 2 usage lines + 4 lane register lines +after compaction: header + 4 entry lines + 2 usage lines + 4 lane value lines ``` -When to compact: on open when the dead-bytes ratio crosses a threshold; after a terminal or outcome-staging deletion when tool checkpoint rewrites pushed the file across that threshold; always after a schema migration (Part 7). Between compactions, normal operation is append-only and O(1) per commit. Deleted pending payloads, superseded state revisions, and superseded `pending.tool_output` checkpoints **linger as bytes** until compaction — logical deletion is immediate, physical deletion is deferred. Tool authors therefore own bounded checkpoint values, cadence, and duplicate suppression; bash emits live updates at 100 ms but requests distinct durable checkpoints at most every two seconds. At 50 KiB per checkpoint, continuously changing output can add approximately 15 MiB per ten minutes before compaction. A deployment that needs prompt physical removal of sensitive cancelled content compacts eagerly at terminal boundaries. +When to compact: on open when the dead-bytes ratio crosses a threshold; after a terminal or outcome-staging deletion when tool checkpoint rewrites or deleted frame lists pushed the file across that threshold; always after a schema migration (Part 7). Between compactions, normal operation is append-only and O(1) per commit. Deleted pending payloads, superseded state revisions, superseded `pendingToolOutput(operationId, invocationId)` checkpoints, and deleted `pi.pending.assistant_frame` lists **linger as bytes** until compaction — logical deletion is immediate, physical deletion is deferred. Tool authors therefore own bounded checkpoint values, cadence, and duplicate suppression; bash emits live updates at 100 ms but requests distinct durable checkpoints at most every two seconds. At 50 KiB per checkpoint, continuously changing output can add approximately 15 MiB per ten minutes before compaction. Assistant frame lists grow append-linear with bounded model output — one compact frame per stream event, never a repeated growing snapshot. A deployment that needs prompt physical removal of sensitive cancelled content compacts eagerly at terminal boundaries. ### SQLite @@ -542,14 +686,18 @@ entries(id TEXT PRIMARY KEY, parent_id TEXT, seq INTEGER, type TEXT, CREATE INDEX ix_entry_parent ON entries(parent_id); CREATE INDEX ix_entry_seq ON entries(seq, type); -registers(namespace TEXT, key TEXT, seq INTEGER, value TEXT, - PRIMARY KEY (namespace, key)); +scalar_values(namespace TEXT, key TEXT, seq INTEGER, value TEXT, + PRIMARY KEY (namespace, key)); + +list_values(namespace TEXT NOT NULL, key TEXT NOT NULL, seq INTEGER NOT NULL, + value TEXT NOT NULL, + PRIMARY KEY (namespace, key, seq)) WITHOUT ROWID; usage_ledger(id TEXT PRIMARY KEY, seq INTEGER, entry_id TEXT, adjustment INTEGER, usage TEXT, details TEXT) WITHOUT ROWID; CREATE INDEX ix_usage_seq ON usage_ledger(seq); --- Private branch index (§2.6). Not registers; no equivalent in the other backends. +-- Private branch index (§2.6). Not values/lists; no equivalent in other backends. branch_entries(branch_id TEXT, entry_id TEXT, entry_seq INTEGER, entry_type TEXT, PRIMARY KEY (branch_id, entry_id)) WITHOUT ROWID; -- Ordered scans. entry_seq must follow branch_id directly or ORDER BY needs a @@ -568,7 +716,9 @@ session(created_at, parent_session_id, storage_version, metadata, writer_lease(owner_id TEXT, fence INTEGER, expires_at_ms INTEGER); ``` -One `commit()` is one SQL transaction: insert entries, insert ledger rows, upsert or delete registers, maintain the branch index, and bump the session stats: maintained projections in `session`: `message_count` for message entries and `usage_payload` for aggregate usage (cached/uncached tokens and cost). Never an UPDATE or DELETE on an entry or ledger row; mutability is confined to registers, the branch index (`branch_meta` tips and bases), stats, sequences, the session catalog row, and leases. +These are logical table names. An implementation may retain an already-applied physical table name while exposing the bound-address API; do not rewrite applied migrations solely for terminology. + +One `commit()` is one SQL transaction: insert entries, insert ledger rows, replace or delete scalar values, insert or whole-list-delete list elements, maintain the branch index, and bump session stats (`message_count` and aggregate `usage_payload`). Never update or delete an entry or ledger row; mutability is confined to values/lists, the branch index, stats, sequences, the session catalog row, and leases. List paging is `SELECT seq, value FROM list_values WHERE namespace = ? AND key = ? AND seq > ? ORDER BY seq ASC LIMIT ?` (descending symmetric; omit the predicate without a cursor); assert with `EXPLAIN QUERY PLAN` that it uses the primary key with no temporary sort. **Every transaction must open with `BEGIN IMMEDIATE`.** A deferred `BEGIN` that reads before it writes takes a read snapshot and must later upgrade to the write @@ -624,14 +774,14 @@ regression. Because the file is the session, the precise rewrite (§2.9) and forks are file operations: build a fresh database (`VACUUM INTO` or row copy over one read snapshot) and, for the rewrite, atomically swap it over the old path — the same shape JSONL uses. -## 1.8 Why write-once plus registers +## 1.8 Why write-once plus values and lists -- **Recovery is a read.** Five register point-lookups per lane, then exact-id dereference (§4.4). No reducer exists to have a bug. +- **Recovery is a read.** Five value point-lookups per lane (§4.4); procedures dereference exact ids and keys as they consume them. The only reducer on any durable path is pi-ai's frame reducer, applied to one bounded list a consumer explicitly names (§3.7). - **Crash states are enumerable.** Between transactions, never inside one. -- **Cleanup is deletion, not collection.** A 30-turn run overwrites one `op.state` register ~30 times and then deletes it. What remains is exactly the conversation, the ledger, and a handful of lane and fact registers — no dead state values, no history rows, nothing to garbage-collect. (JSONL defers *physical* reclamation to snapshot compaction; the logical state is identical.) -- **No repair-by-rewrite.** Recovery appends entries and overwrites only the registers it owns, with the same transitions normal execution would commit; interrupt it and rerun it and you get the same result. +- **Cleanup is deletion, not collection.** A 30-turn run replaces one `operationState(operationId)` value ~30 times and then deletes it. What remains is exactly the conversation, the ledger, and a handful of lane and semantic session values — no dead state values, no history rows, nothing to garbage-collect. (JSONL defers *physical* reclamation to snapshot compaction; the logical state is identical.) +- **No repair-by-rewrite.** Recovery appends entries and replaces only the values it owns, with the same transitions normal execution would commit; interrupt it and rerun it and you get the same result. - **Concurrency is trivial.** Readers never see partial state; there is nothing to lock. -- **Deliberate staging writes.** Queued content is serialized into `pending.entry` at enqueue and again into its immutable entry at placement. Finalized tool outcomes also stage in `pending.entry` before source-ordered materialization, which prevents a completed parallel effect from replaying after a crash. Assistant settlements remain born placed. In every case, staging has one owner and dies atomically with placement or cleanup. +- **Deliberate staging writes.** Queued content is serialized into `pi.pending.entry` at enqueue and again into its immutable entry at placement. Finalized tool outcomes also stage in `pi.pending.entry` before source-ordered materialization, which prevents a completed parallel effect from replaying after a crash. Assistant settlements remain born placed; their streamed frames append once each to an operation-owned list and die atomically with settlement (§3.7). In every case, staging has one owner and dies atomically with placement or cleanup. --- @@ -671,7 +821,7 @@ Rules: The tree's central rule: -> An **entry** is created, complete, when placement happens. Content that is durable *before* placement is current mutable state and waits in a `pending.entry` register; the placement transaction writes the entry and deletes the register. Neither is ever modified after that. +> An **entry** is created, complete, when placement happens. Content that is durable *before* placement is current mutable state and waits in a `pendingEntry(id)` value; the placement transaction writes the entry and deletes the pending value. Neither is ever modified after that. The cases are mechanical: @@ -679,53 +829,53 @@ The cases are mechanical: ``` TX[ insert e_a4 = { parent: e_q1, type: "message", message: }, - upsert lane.leaf/main = "e_a4" ] + upsert pi.lane.leaf/main = "e_a4" ] ``` -**Content first, placement later — queued input.** `steer`, `followUp`, `nextRun`, and deferred tree writes mint the entry id at enqueue and use it as the register key. Queue state references content by that id. Two transactions may be far apart: +**Content first, placement later — queued input.** `steer`, `followUp`, `nextRun`, and deferred tree writes mint the entry id at enqueue and construct `pendingEntry(id)`. Queue state references content by that id. Two transactions may be far apart: ``` -t0 TX[ upsert pending.entry/e_q1 = { type: "message", payload: <200KB message> }, +t0 TX[ upsert pi.pending.entry/e_q1 = { type: "message", payload: <200KB message> }, S(next){ ...inbox.steer += "e_q1" } ] -t1 TX[ insert e_q1 = { parent: e_a3, type: "message", message: }, - delete pending.entry/e_q1, - upsert lane.leaf/main = "e_q1", +t1 TX[ insert e_q1 = { parent: e_a3, type: "message", message: }, + delete pi.pending.entry/e_q1, + upsert pi.lane.leaf/main = "e_q1", S(next){ ...inbox.steer -= "e_q1" } ] ``` -Crash before `t1`: the item is still queued. Crash after: it is placed and the register is gone. Until placement or cancellation, exactly one of register and entry exists. Cancellation deletes the register and the content never enters the tree (§3.11). +Crash before `t1`: the item is still queued. Crash after: it is placed and the pending value is gone. Until placement or cancellation, exactly one of pending value and entry exists. Cancellation deletes the value and the content never enters the tree (§3.11). -**Content first, placement later — finalized parallel tool outcomes.** A tool result id begins as a plain reserved string in `op.state`. When execution and `after_tool` finish, the complete final `ToolResultMessage` is staged in `pending.entry/{resultEntryId}` and the call becomes `outcome_ready`. It enters the tree only when every earlier source position is ready: +**Content first, placement later — finalized parallel tool outcomes.** A tool result id begins as a plain reserved string in `pi.op.state`. When execution and `after_tool` finish, the complete final `ToolResultMessage` is staged in `pendingEntry(resultEntryId)` and the call becomes `outcome_ready`. It enters the tree only when every earlier source position is ready: ``` -t0 TX[ upsert pending.entry/e_r2 = finalized result, +t0 TX[ upsert pi.pending.entry/e_r2 = finalized result, S(call 1 = outcome_ready) ] t1 TX[ insert e_r2 after the earlier result, - delete pending.entry/e_r2, + delete pi.pending.entry/e_r2, S(call 1 = completed) ] ``` This lets effects settle in completion order while entries materialize in assistant source order. Crash before `t0` leaves an uncertain effect; crash after `t0` never re-executes it; crash after `t1` sees the immutable entry. -**Id reserved before content exists.** Assistant response, tool-result, and usage ids are minted as strings in operation state. Assistant settlement places its result directly. Tool settlement may pass through `pending.entry` as described above. +**Id reserved before content exists.** Assistant response, tool-result, and usage ids are minted as strings in operation state. Assistant settlement places its result directly; during the effect window the reserved response id also keys the auxiliary `pi.pending.assistant_frame` list, which settlement deletes (§3.7). Tool settlement may pass through `pi.pending.entry` as described above. Consequences to rely on: -- A queued or outcome-ready item is invisible to tree queries but visible through its owning state and `pending.entry` payload. -- Queue placement/cancellation and outcome-ready materialization delete `pending.entry` atomically with their state change. -- A reserved tool-result id moves through `string only → pending.entry → immutable entry`; no two representations coexist at a commit boundary. +- A queued or outcome-ready item is invisible to tree queries but visible through its owning state and `pendingEntry(id)` payload. +- Queue placement/cancellation and outcome-ready materialization delete `pi.pending.entry` atomically with their state change. +- A reserved tool-result id moves through `string only → pi.pending.entry → immutable entry`; no two representations coexist at a commit boundary. - Queued input still pays the deliberate content double-write (§1.8). Finalized tool outcomes also stage once before placement when source ordering requires it; this additional write is what prevents completed parallel effects from replaying after a crash. ## 2.3 Lanes -A configured lane is three registers — plus `lane.lastResult` once its first operation has ended (§3.13). Fresh or normalized-v3 `main` may temporarily lack `lane.config` until first harness attachment: +A configured lane is three values — plus `pi.lane.lastResult` once its first operation has ended (§3.13). Fresh or normalized-v3 `main` may temporarily lack `pi.lane.config` until first harness attachment: ``` -lane.leaf/{name} = entry id or null -lane.config/{name} = LaneConfiguration // absent only for unconfigured main -lane.state/{name} = LaneState +pi.lane.leaf/{name} = entry id or null +pi.lane.config/{name} = LaneConfiguration // absent only for unconfigured main +pi.lane.state/{name} = LaneState ``` ```ts @@ -737,14 +887,14 @@ interface LaneConfiguration { ``` - A lane's leaf moves in exactly two ways: the lane appends an entry (leaf becomes that entry), or the lane navigates (leaf jumps to an existing entry). -- `LaneConfiguration` is **total**. A setter overwrites the whole register; it is never a patch and never a tree entry. -- `Session.createLane(name, at, configuration)` is the primitive that creates a lane. It internally opens a mutation on the new lane, validates the permanent lane name, rejects an existing lane or unknown non-null anchor, and consumes that mutator's sole commit to write all three registers atomically. The supplied configuration is a total seed; it is not inferred from the anchor or another lane. The method returns the new lane's `SessionTree` view. +- `LaneConfiguration` is **total**. A setter replaces the whole value; it is never a patch and never a tree entry. +- `Session.createLane(name, at, configuration)` is the primitive that creates a lane. It internally opens a mutation on the new lane, validates the permanent lane name, rejects an existing lane or unknown non-null anchor, and consumes that mutator's sole commit to write all three values atomically. The supplied configuration is a total seed; it is not inferred from the anchor or another lane. The method returns the new lane's `SessionTree` view. - Creating a lane copies no tree content, no history, and no configuration from its anchor: ``` -TX[ upsert lane.config/{name} = , - upsert lane.leaf/{name} = anchorEntryId, - upsert lane.state/{name} = { currentOperationId: null, pendingNextRun: [] } ] +TX[ upsert pi.lane.config/{name} = , + upsert pi.lane.leaf/{name} = anchorEntryId, + upsert pi.lane.state/{name} = { currentOperationId: null, pendingNextRun: [] } ] ``` - `AgentHarness.createLane()` invokes this primitive with the harness's immutable captured seed, then publishes the harness event. `Session.createLane()` owns its mutation-line admission; callers must not wrap it in another mutation. Session-level callers use the same primitive directly when they need tree and lane management without runtime orchestration. @@ -752,17 +902,25 @@ TX[ upsert lane.config/{name} = , - `main` exists in every session. - Two lanes at the same leaf simply diverge on their next append. -## 2.4 Facts +## 2.4 Session metadata and application values -Session-scoped, latest-wins, not part of the tree. +Session name and entry labels are latest-wins values outside the tree: -``` -fact.name/"" = string -fact.label/{entryId} = string -fact.custom/{key} = JsonValue +```ts +sessionName // value("pi.session.name") +entryLabel(entryId) // value("pi.entry.label", entryId) ``` -Setting a fact to `undefined` deletes its register — real deletion, not a tombstone; deleting an unset fact is a no-op (§1.4). JSON `null` is a legitimate custom value, stored directly, and is distinguishable from deletion because the register itself exists or does not. The built-in and custom namespaces never overlap. Fact writes commit immediately and never move a leaf. +`getName`/`setName` and `getLabel`/`setLabel` are purpose-specific wrappers over those addresses. Passing `undefined` to a setter deletes the value; deleting an absent value is a no-op (§1.4). These writes commit immediately and never move a leaf. + +Applications define stable, collision-resistant addresses directly: + +```ts +const applicationState = value("my-app.state"); +const applicationEvents = list("my-app.events"); +``` + +There is no built-in application namespace or separate application-state API. Application values and lists use the same `SessionTree` address methods as harness storage and must define their own fork and migration policy. ## 2.5 Branch queries and context @@ -826,7 +984,7 @@ Tests assert these invariants and the required query plans. No wall-clock thresh ## 2.7 Forks -A fork is a repository operation over one coherent source-session snapshot. It copies selected entries, latest facts, lane leaves, and total configuration; it never copies `op.*`, `pending.entry`, `pending.tool_output`, or `lane.lastResult` registers or ledger rows — destination lanes start with a fresh empty `LaneState`. +A fork is a repository operation over one coherent source-session snapshot. It copies selected entries, latest semantic values, lane leaves, and total configuration; it never copies operation/pending values, assistant frame lists, or ledger rows — destination lanes start with a fresh empty `LaneState`. Application-defined values and lists require an explicit fork policy when their consuming feature is added; a precise rewrite (§2.9) that retains a list element preserves its sequence cursor unless it explicitly remaps the whole destination sequence space. ```ts type ForkOptions = @@ -836,8 +994,8 @@ type ForkOptions = - Memory and JSONL obtain the snapshot as one job on the source storage queue. SQLite uses one read transaction. - Branch scope copies one path and creates only destination `main`. Tree scope copies the whole tree and every lane leaf/configuration. -- The destination is idle and its token/cost ledger starts at zero. Entry-local display usage remains on copied entries. -- Facts follow the selected scope: name/custom facts always copy; labels copy only when their target copies unless tree scope copies all targets. +- The destination is idle and its usage/cost ledger starts at zero. Entry-local display usage remains on copied entries. +- Semantic values follow their declared scope: the session name copies; labels copy only when their target copies unless tree scope copies all targets; application-defined values/lists follow their explicit policy. - Any message may be the fork point. Request construction heals orphaned tool calls. - Copied entries keep their ids. - `id` optionally supplies the destination session id. @@ -874,10 +1032,10 @@ interface SessionRepo>; - getRegister(namespace: N, key: string): - Promise | undefined>; - listRegisters(namespace: N, keyPrefix?: string): - Promise[]>; + getValue(address: Value): Promise | undefined>; + scanValues(prefix: Value): Promise[]>; + readList(address: ValueList, options?: ListReadOptions): + Promise[]>; } /** Callback-scoped write capability bound to one lane. */ @@ -909,7 +1067,7 @@ interface Session `Session.mutate` is the only raw session write surface. It queues the complete callback on the named lane's mutation line and supplies a lane-bound `SessionMutator`. The mutator is valid only until that callback returns or throws. Reads do not consume it. Its first `commit()` attempt consumes its commit capability before awaiting storage, so a second attempt rejects even when the first failed. The callback may return without committing. It must not retain the mutator, invoke another mutating `Session`/`SessionTree` method, wait for lane idleness, or perform provider, tool, hook, or timer effects; those operations either reacquire the same line or hold it across work that must remain concurrent. Separate durable transitions use separate `mutate` calls. -The high-level mutating methods on `SessionTree` and `Session` — fact setters, appends, and `createLane` — acquire the appropriate mutation line themselves and consume one private mutator commit. Callers invoke them directly rather than wrapping them in `mutate`. Reads remain available directly on `Session`, `SessionTree`, and `SessionMutator`. +The high-level mutating methods on `SessionTree` and `Session` — value setters, appends, and `createLane` — acquire the appropriate mutation line themselves and consume one private mutator commit. Callers invoke them directly rather than wrapping them in `mutate`. Reads remain available directly on `Session`, `SessionTree`, and `SessionMutator`. Declaration-merged custom `AgentMessage` variants are trusted in-process types like other internal values; extensions that violate them are defective. A new repository session creates `main` with null leaf and an empty `LaneState`, but no configuration; first harness attachment writes its seed configuration. Every additional lane is created through `Session.createLane()` with a total configuration. @@ -1041,7 +1199,7 @@ await search.searchSessions({ text: "auth", where: { cwd: "/repo" } }); ## 2.9 The precise rewrite -Entries and usage rows are never deleted (§1.2). The sole sanctioned exception is the **precise rewrite**: an administrative repository operation that copies the retained set — entries, usage rows, facts, lane registers — into a fresh session store over a coherent snapshot, exactly as a fork does (§2.8), then atomically swaps it for the old store. Its keep-predicate can express what no runtime mechanism may: compliance-grade erasure (including content copied forward into `retainedTail`s and summaries), pruning abandoned branches, and re-minting legacy-format ids (Appendix B). It is tooling above the harness — no harness surface exposes it, and no core rule depends on it. +Entries and usage rows are never deleted (§1.2). The sole sanctioned exception is the **precise rewrite**: an administrative repository operation that copies the retained set — entries, usage rows, semantic values, and lane values — into a fresh session store over a coherent snapshot, exactly as a fork does (§2.8), then atomically swaps it for the old store. Its keep-predicate can express what no runtime mechanism may: compliance-grade erasure (including content copied forward into `retainedTail`s and summaries), pruning abandoned branches, and re-minting legacy-format ids (Appendix B). It is tooling above the harness — no harness surface exposes it, and no core rule depends on it. # Part 3 — The operation state machine @@ -1062,13 +1220,13 @@ interface Operation { } ``` -Acceptance data lives in the `op.meta/{operationId}` register: written once at acceptance, never overwritten, and deleted by the terminal transaction (§3.13). `sourceLeafId` is the lane's leaf *before* the operation; entries the operation itself appends come after it. `promptEntryIds` name the caller's normalized prompt entries, born placed in the acceptance transaction (§3.6). The id is either supplied to `accept` or minted before pre-acceptance work; it correlates a hosted submission with current operation and `lane.lastResult`, but the harness retains no forever-idempotency index after a later terminal result overwrites that register. +Acceptance data lives in the `operationMeta(operationId)` value: written once at acceptance, never overwritten, and deleted by the terminal transaction (§3.13). `sourceLeafId` is the lane's leaf *before* the operation; entries the operation itself appends come after it. `promptEntryIds` name the caller's normalized prompt entries, born placed in the acceptance transaction (§3.6). The id is either supplied to `accept` or minted before pre-acceptance work; it correlates a hosted submission with current operation and `pi.lane.lastResult`, but the harness retains no forever-idempotency index after a later terminal result replaces that value. ## 3.2 Operation state — the durable restart point -`op.state/{operationId}` holds one total `OperationState` directly. Every durable transition overwrites the whole register; the terminal transaction deletes it (§3.13). There is no finished member of the union — an ended operation has no state at all, and its outcome lives in `lane.lastResult`. +`operationState(operationId)` holds one total `OperationState` directly. Every durable transition replaces the whole value; the terminal transaction deletes it (§3.13). There is no finished member of the union — an ended operation has no state at all, and its outcome lives in `pi.lane.lastResult`. -The register is authoritative after process loss, but it is not the finer instruction pointer of a live async procedure. For example, a fresh assistant procedure commits `effect_pending` and retains its JavaScript continuation through the request and settlement. If that continuation is lost, the same durable state becomes the activation point for unknown-outcome recovery (§4.1, §4.5). +That value is authoritative after process loss, but it is not the finer instruction pointer of a live async procedure. For example, a fresh assistant procedure commits `effect_pending` and retains its JavaScript continuation through the request and settlement. If that continuation is lost, the same durable state becomes the activation point for unknown-outcome recovery (§4.1, §4.5). ```ts type OperationState = RunState | CompactionState | NavigationState; @@ -1076,7 +1234,7 @@ type OperationState = RunState | CompactionState | NavigationState; type Control = | { status: "running" } | { status: "cancel_requested"; requestedAt: number; - /** Drained queue ids. Their pending.entry registers survive the drain + /** Drained queue ids. Their `pendingEntry(id)` values survive the drain and are deleted only by the terminal transaction (§3.11, §3.13). */ drainedSteer: string[]; drainedFollowUp: string[] }; @@ -1124,7 +1282,7 @@ type Continuation = interface Inbox { /** Reserved entry ids. Payloads — and, for writes, the entry type and - customType — live in each id's pending.entry register (§1.3, §2.2). */ + customType — live at `pendingEntry(id)` (§1.3, §2.2). */ steer: string[]; followUp: string[]; writes: string[]; @@ -1133,11 +1291,11 @@ interface Inbox { interface OperationError { code: string; message: string; details?: JsonValue } ``` -A queue item is one entry id; everything else about it — payload, write type, `customType` — is dereferenced from its `pending.entry` register. +A queue item is one entry id; everything else about it — payload, write type, `customType` — is dereferenced from its `pendingEntry(id)` value. `latestAssistantEntryId` updates in the same settlement transaction as every assistant generation or deferred-fetch response. It lets finish and resume construct results/events without a branch scan. A tool batch retains its producing turn id while tool work remains active. -Any transition that appends conversational input or tool results and requires another assistant writes a checkpoint with `need_assistant(false)` and the appended entry as `triggerEntryId`. A `may_finish` checkpoint sets `triggerEntryId` to the entry that caused the boundary: the settled response for a `stop`/genuine-`length` settlement (§3.7), the newest result entry for an all-terminating tool batch (§3.8) — so threshold dedup (§3.12) and restore validation (§3.3) always name an existing entry. An unprojected custom write preserves the current checkpoint, including trigger and overflow flag. Entering threshold compaction first copies the checkpoint to `resumeAfter` with `thresholdCheckedTriggerEntryId = triggerEntryId`; decline, empty preparation, success, and crash therefore cannot recheck the same boundary. +Any transition that appends conversational input or tool results and requires another assistant writes a checkpoint with `need_assistant(false)` and the appended entry as `triggerEntryId`. A `may_finish` checkpoint sets `triggerEntryId` to the entry that caused the boundary: the settled response for a `stop`/genuine-`length` settlement (§3.7), the newest result entry for an all-terminating tool batch (§3.8) — so threshold dedup (§3.12) and every later consumer always name an existing entry. An unprojected custom write preserves the current checkpoint, including trigger and overflow flag. Entering threshold compaction first copies the checkpoint to `resumeAfter` with `thresholdCheckedTriggerEntryId = triggerEntryId`; decline, empty preparation, success, and crash therefore cannot recheck the same boundary. ### Generation @@ -1166,7 +1324,7 @@ type Generation = notBefore: number; errorMessage: string }; ``` -The context snapshots configuration, stream options, and retry policy **inline**; `LaneConfiguration` is small. Recovery can therefore report exactly what is missing without resolving anything (§4.4). For each attempt, `before_request` runs from generation `ready` (an elapsed retry wait first returns to `ready`). Its curated patch is composed with the context's captured base stream options, then `intendedOutputLimit` and `contextWindow` are calculated and persisted in the `effect_pending` intent before dispatch. A pre-intent crash may rerun the hook. Harness-owned `before_payload`/`after_response` callbacks are mounted only after intent and cannot be replaced through stream options. +The context snapshots configuration, stream options, and retry policy **inline**; `LaneConfiguration` is small. Recovery can therefore report exactly what is missing without resolving anything (§4.4). No frame count, cursor, or list identity is stored in the state: the procedure constructs `pendingAssistantFrames(operationId, responseEntryId)` from the effect-pending state (§3.7). For each attempt, `before_request` runs from generation `ready` (an elapsed retry wait first returns to `ready`). Its curated patch is composed with the context's captured base stream options, then `intendedOutputLimit` and `contextWindow` are calculated and persisted in the `effect_pending` intent before dispatch. A pre-intent crash may rerun the hook. Harness-owned `before_payload`/`after_response` callbacks are mounted only after intent and cannot be replaced through stream options. ### Tool batch @@ -1190,9 +1348,9 @@ type ToolCall = terminate: boolean }; ``` -The source call comes from `assistantEntryId` plus `sourceIndex`; large effective arguments live once in the `op.tool_args/{operationId}:{stepId}:{sourceIndex}` register — the producing generation's `stepId` disambiguates batches across turns — written at clearance (§3.8) and located by that deterministic key. Persist them unconditionally because `prepareArguments`, not only `before_tool`, may change them. +The source call comes from `assistantEntryId` plus `sourceIndex`; large effective arguments live once in the `operationToolArgs(operationId, stepId, sourceIndex)` value — the producing generation's `stepId` disambiguates batches across turns — written at clearance (§3.8) and located by that deterministic key. Persist them unconditionally because `prepareArguments`, not only `before_tool`, may change them. -`outcome_ready` means execution and post-effect hooks are finished and the complete finalized result is durable in `pending.entry/{resultEntryId}`, but an earlier source call prevents tree placement. Such a call never executes again. Completed calls form a source-ordered prefix. In parallel mode, the suffix may mix `planned`, `effect_pending`, and `outcome_ready`; source-ordered materialization alone extends the completed prefix. In sequential mode, the suffix contains at most one `effect_pending` or `outcome_ready` call before planned calls. +`outcome_ready` means execution and post-effect hooks are finished and the complete finalized result is durable in `pendingEntry(resultEntryId)`, but an earlier source call prevents tree placement. Such a call never executes again. Completed calls form a source-ordered prefix. In parallel mode, the suffix may mix `planned`, `effect_pending`, and `outcome_ready`; source-ordered materialization alone extends the completed prefix. In sequential mode, the suffix contains at most one `effect_pending` or `outcome_ready` call before planned calls. ### Deferred @@ -1207,19 +1365,21 @@ type Deferred = One `drive({ pollDeferred: true })` performs at most one `fetchDeferred(handle, { wait: 0 })`; `resume()` is the convenience call that supplies that permit. Suspended `poll` is the number of completed polls; a fresh intent uses `poll + 1`, and that 1-based value is `before_request.attempt` and the poll turn-id suffix. A poll starts from the original generation's copied base stream options, forces `deferred:false`, runs `before_request`, mounts `before_payload`/`after_response`, then commits its fresh intent and dispatches like assistant generation. Current global stream settings do not affect it. There is no polling retry cap, backoff, or internal loop. A `deferred` response has a valid handle only when `message.deferred` exists, its id is non-empty, and its `{ provider, modelId, api }` equals the captured request model identity/API. An invalid initial handle is normalized to a durable `error` response and response-provenance `failure_drain`. A later pending response must carry a handle completely equal to its source handle and becomes the next source; a mismatch is normalized the same way. Response, usage, `latestAssistantEntryId`, and failure state commit atomically. +A deferred poll that returns an event stream persists frames exactly like assistant generation, at `pendingAssistantFrames(operationId, responseEntryId)` for the poll's reserved response id (§3.7). Every row below that inserts a response entry additionally deletes that response's frame list in the same transaction. Frames never change deferred-handle equality or poll-number rules. + The complete transition table — every row is one `commit()`; classification order (§3.7) applies to every poll settlement, cancellation first: | From | Trigger | Transaction | To | |---|---|---|---| | assistant `effect_pending` | settlement classifies `deferred` with a valid handle | §3.7's deferred row | suspended, `poll: 0`, `sourceEntryId: R` | | suspended, poll *k* | a drive with `pollDeferred: true`: the poll's `before_request` settlement commits its intent, consuming the pass's single poll permit | mint fresh R′ and U′, then `TX[ S(deferred{effect_pending, poll k+1, responseEntryId R′, usageId U′}) ]` | effect_pending, poll *k*+1 | -| effect_pending, poll *k*+1 | fetch returns **pending** with a completely equal handle | `TX[ insert response entry R′, upsert lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, deferred{suspended, sourceEntryId R′, poll k+1}) ]` — the pending response becomes the next source and the operation re-suspends; no second poll this invocation | suspended, poll *k*+1 | -| effect_pending | fetch returns **pending** with a mismatched handle | normalize to a durable `error` response explaining the mismatch: `TX[ insert normalized response R′, upsert lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, failure_drain{error, provenance:response R′}) ]` | failure_drain | -| effect_pending | fetch returns **ready** with tool calls | `TX[ insert response R′, upsert lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, tools{plan with reserved result ids}) ]` — result ids minted as followers of R′ (§1.2) | tools | -| effect_pending | fetch returns **ready** without tool calls | `TX[ insert response R′, upsert lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, checkpoint{may_finish, includeFinalAssistant:true}) ]` | checkpoint | -| effect_pending | fetch settles as a provider `error` | `TX[ insert response R′, upsert lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, failure_drain{error, provenance:response R′}) ]` — polls have no retry path | failure_drain | -| effect_pending, restored, running control | crash left the poll's outcome unknown; the next drive with a poll permit replaces it | mint fresh R″/U″ and commit a fresh intent at the **same** poll number — an unknown-outcome poll never completed, so `poll` does not increment; the old reserved id strings are abandoned, never materialized | effect_pending, poll *k*+1 | -| effect_pending, cancelled control | reconciliation, live or restored (§4.5, §4.6) | synthetic settlement under the **existing** reserved ids: `TX[ insert synthetic aborted response R′, upsert lane.leaf = R′, insert zero usage U′, S(latestAssistantEntryId=R′, cancelled checkpoint{may_finish}) ]` | cancelled checkpoint → aborted finish | +| effect_pending, poll *k*+1 | fetch returns **pending** with a completely equal handle | `TX[ insert response entry R′, upsert pi.lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, deferred{suspended, sourceEntryId R′, poll k+1}) ]` — the pending response becomes the next source and the operation re-suspends; no second poll this invocation | suspended, poll *k*+1 | +| effect_pending | fetch returns **pending** with a mismatched handle | normalize to a durable `error` response explaining the mismatch: `TX[ insert normalized response R′, upsert pi.lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, failure_drain{error, provenance:response R′}) ]` | failure_drain | +| effect_pending | fetch returns **ready** with tool calls | `TX[ insert response R′, upsert pi.lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, tools{plan with reserved result ids}) ]` — result ids minted as followers of R′ (§1.2) | tools | +| effect_pending | fetch returns **ready** without tool calls | `TX[ insert response R′, upsert pi.lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, checkpoint{may_finish, includeFinalAssistant:true}) ]` | checkpoint | +| effect_pending | fetch settles as a provider `error` | `TX[ insert response R′, upsert pi.lane.leaf = R′, insert usage U′, S(latestAssistantEntryId=R′, failure_drain{error, provenance:response R′}) ]` — polls have no retry path | failure_drain | +| effect_pending, restored, running control | crash left the poll's outcome unknown; the next drive with a poll permit replaces it | mint fresh R″/U″ and commit a fresh intent at the **same** poll number — an unknown-outcome poll never completed, so `poll` does not increment; the old reserved id strings are abandoned, never materialized, and the intent transaction deletes the abandoned old frame list; the replacement poll starts a fresh list under its fresh response id | effect_pending, poll *k*+1 | +| effect_pending, cancelled control | reconciliation, live or restored (§4.5, §4.6) | synthetic settlement under the **existing** reserved ids, preserving partial content reduced from committed frames: `TX[ insert synthetic aborted response R′, upsert pi.lane.leaf = R′, insert zero usage U′, delete frame list O:R′, S(latestAssistantEntryId=R′, cancelled checkpoint{may_finish}) ]` | cancelled checkpoint → aborted finish | | suspended, cancelled control | reconciliation | no fetch starts; best-effort `cancel_deferred` targets the newest source (§4.6), and the operation finishes through the aborted terminal transaction | terminal | ### Structural work @@ -1264,36 +1424,31 @@ type NavigationState = phase: { kind: "summary"; structural: StructuralDecision } }; ``` -Structural preparation is built from the reserved source leaf and settings snapshot, normalized (`Set` file-operation fields become sorted arrays), and written once to the `op.preparation/{operationId}:{taskId}` register before the decision hook, in the same transaction as the `deciding` state (§3.9). State carries only `taskId`; the deterministic key locates the register, and hooks/generators hydrate arrays back to the source preparation types. Reopen never rebuilds it from current settings, so the provider sees the same summary input the hook approved. +Structural preparation is built from the reserved source leaf and settings snapshot, normalized (`Set` file-operation fields become sorted arrays), and written once to the `operationPreparation(operationId, taskId)` value before the decision hook, in the same transaction as the `deciding` state (§3.9). State carries only `taskId`; the deterministic constructor locates the value, and hooks/generators hydrate arrays back to the source preparation types. Reopen never rebuilds it from current settings, so the provider sees the same summary input the hook approved. One structural attempt may make one or two sequential provider requests using the existing compaction implementation. Its request callback first commits `request:{index,usageId}`, reaches the provider-request breakpoint, synchronously checks `EffectGate.assertOpen()` immediately before invoking that request, then atomically writes usage and clears/advances the request field. Intermediate content remains process-local; any activated orphaned `effect_pending` attempt is treated as wholly uncertain and starts a later attempt under the captured policy rather than continuing request two. A durable `generating` decision prevents its decision hook from rerunning. -## 3.3 Lane state and current-state validity +## 3.3 Lane state and the restore projection ```ts interface LaneState { currentOperationId: string | null; - /** Reserved entry ids; payloads in pending.entry registers (§2.2). */ + /** Reserved entry ids; payloads at `pendingEntry(id)` addresses (§2.2). */ pendingNextRun: string[]; } ``` -Restore checks only semantic consistency of the current lane and operation registers and the entries/registers they directly name; there is no history to audit and none exists. Required checks: +Restore trusts committed internal values (§0.8) and checks only that the values required to construct the lane projection exist: `laneState(lane)` holds a `LaneState`, and when it names operation O, `operationMeta(O)` and `operationState(O)` exist. Nothing else is audited at restore — no referenced-entry existence, pending-payload presence, tool-call shape, or state-discriminant relationships. -- `lane.state/{lane}` holds a `LaneState`; when it names operation O, `op.meta/O` holds an `Operation` for that lane, and `op.state/O` holds an `OperationState` compatible with O's intent kind; -- every entry id the current state or `op.meta` names — trigger, latest assistant, batch assistant, deferred source, completed results, prompt entries, a non-null `sourceLeafId`, a navigation intent's non-null `targetId`, the lane leaf — resolves to an existing entry of the expected type; -- every id in `inbox.*`, `control.drained*`, and `pendingNextRun` has its `pending.entry` register; `pendingNextRun`, steer, follow-up, and drained queue items are messages; every effect-pending call has its `op.tool_args` register; every outcome-ready call has a matching tool-result `pending.entry`; every structural decision has its `op.preparation` register; -- tool source indices are complete, ordered, unique, in range, and use unique result ids; completed result entries and outcome-ready pending values match their source calls; completed calls form a source-ordered prefix, while only the parallel suffix may freely mix planned, effect-pending, and outcome-ready states; -- a cancelled run has empty steer/follow-up inboxes, structural operations have no drained run queues, and a current operation may reference an `aborted` assistant response only under cancelled control; -- cancellation, navigation source/target, and structural-source combinations satisfy the state discriminants. +Everything the state names — trigger and result entries, pending payloads, tool arguments, preparations, staged outcomes, optional checkpoints, frame lists — is dereferenced and checked by the procedure that consumes it, at the moment it is consumed. A required value found missing or mismatched there is an invariant defect raised at consumption, not a restore failure. Live operation/phase/effect-identity checks on the lane mutation line remain required when asynchronous work settles; they are concurrency fences (§3.4, §4.3), not restore validation. -`lane.lastResult` is never a recovery input (§3.13). The bounded checks above establish relationships that types alone cannot express, such as referenced-row existence, owner identity, intent/state compatibility, and reserved-id consistency. They do not revalidate object shapes already guaranteed by internal types. +`pi.lane.lastResult` is never a recovery input (§3.13). ## 3.4 The atomic transition rule -> Compute the next total state in memory, then atomically commit every entry insert, usage insert, and register write that makes that state true. +> Compute the next total state in memory, then atomically commit every entry insert, usage insert, and value/list write that makes that state true. -A transition rereads the latest operation and lane registers inside the lane mutation line, verifies the semantic state it extends, and changes only the fields it owns. In particular, settlement preserves newer inbox/control fields, and the terminal transaction clears `currentOperationId` while preserving concurrently accepted `pendingNextRun`. An unexpected non-terminal phase or effect identity is an invariant defect, not ordinary stale-state replanning; absent operation registers mean a terminal transaction already won (§4.3, §4.9). Every edge below is exactly one `commit()`. +A transition rereads the latest operation and lane values inside the lane mutation line, verifies the semantic state it extends, and changes only the fields it owns. In particular, settlement preserves newer inbox/control fields, and the terminal transaction clears `currentOperationId` while preserving concurrently accepted `pendingNextRun`. An unexpected non-terminal phase or effect identity is an invariant defect, not ordinary stale-state replanning; absent operation values mean a terminal transaction already won (§4.3, §4.9). Every edge below is exactly one `commit()`. ## 3.5 The graph @@ -1334,7 +1489,7 @@ stateDiagram-v2 terminal --> [*] ``` -`terminal` is not a state. It is the terminal transaction (§3.13): after it commits, the operation has no `op.state` register at all. +`terminal` is not a state. It is the terminal transaction (§3.13): after it commits, the operation has no `operationState(operationId)` value at all. Standalone operations: @@ -1356,12 +1511,12 @@ A declined summarized navigation moves nothing: the leaf stays at the source, an | From | Request | Transaction | |---|---|---| -| idle lane | prompt, skill, or prompt-template request after expansion and `before_run` | `TX[ insert entries for captured nextRun items (payloads from their pending.entry registers) and the new messages (caller prompt, hook injections) in order, delete the captured pending.entry registers, upsert lane.leaf = newest entry, upsert op.meta/O, S(run{captured settings, checkpoint need_assistant(false), trigger = newest entry, skipInboxOnce, empty inbox}), L({currentOperationId: O, captured ids removed from pendingNextRun}) ]` | -| reserved idle lane | compaction request with non-empty preparation | `TX[ upsert op.preparation/O:{taskId} = P, upsert op.meta/O, S(compaction{deciding, taskId}), L({currentOperationId: O}) ]` | -| idle lane | unsummarized navigation request after validation | `TX[ upsert op.meta/O, S(navigation{ready_to_commit}), L ]` | -| reserved idle lane | summarized navigation request with preparation | `TX[ upsert op.preparation/O:{taskId} = P, upsert op.meta/O, S(navigation{summary.deciding, taskId}), L ]` | +| idle lane | prompt, skill, or prompt-template request after expansion and `before_run` | `TX[ insert entries for captured nextRun items (payloads from their pending values) and the new messages (caller prompt, hook injections) in order, delete the captured pending values, upsert pi.lane.leaf = newest entry, upsert pi.op.meta/O, S(run{captured settings, checkpoint need_assistant(false), trigger = newest entry, skipInboxOnce, empty inbox}), L({currentOperationId: O, captured ids removed from pendingNextRun}) ]` | +| reserved idle lane | compaction request with non-empty preparation | `TX[ upsert pi.op.preparation/O:{taskId} = P, upsert pi.op.meta/O, S(compaction{deciding, taskId}), L({currentOperationId: O}) ]` | +| idle lane | unsummarized navigation request after validation | `TX[ upsert pi.op.meta/O, S(navigation{ready_to_commit}), L ]` | +| reserved idle lane | summarized navigation request with preparation | `TX[ upsert pi.op.preparation/O:{taskId} = P, upsert pi.op.meta/O, S(navigation{summary.deciding, taskId}), L ]` | -Captured `nextRun` items already have their payloads in `pending.entry` registers; acceptance inserts their entries from those payloads, deletes the registers, and removes the ids from `pendingNextRun` — the placement half of queued-content staging (§1.8). A late-captured item keeps its enqueue-minted id (§1.2). +Captured `nextRun` items already have their payloads in `pendingEntry(id)` values; acceptance inserts their entries from those payloads, deletes the values, and removes the ids from `pendingNextRun` — the placement half of queued-content staging (§1.8). A late-captured item keeps its enqueue-minted id (§1.2). Compaction first chooses its supplied-or-minted operation id and takes a process-local lane admission reservation, then reads preparation. Summarized navigation uses the same reservation while collecting/building branch preparation; unsummarized navigation needs none because validation and acceptance share one lane-line job. While reserved, competing operation requests receive `LaneBusy` naming that provisional id/kind and idle tree writes wait; `nextRun` and configuration changes may still commit because they do not move the leaf. Empty compaction preparation releases the reservation and returns `NothingToCompact` with no operation write. Non-empty preparation is accepted only against the unchanged reserved source leaf. Process death drops the reservation and leaves the lane idle. @@ -1375,17 +1530,41 @@ Pre-acceptance rejections write **nothing**: `LaneBusy`, `NothingToCompact`, `In |---|---|---|---| | checkpoint `need_assistant` | ordinary procedure | snapshot current lane config, stream options, and normalized retry policy inline into the context in `TX[ S(assistant{ready, nextAttempt:1}) ]` | ready | | assistant `ready` | `before_request` aggregate completes | mint R and U, then `TX[ S(assistant{effect_pending, attempt=nextAttempt, responseEntryId R, usageId U, intendedOutputLimit, contextWindow}) ]` | effect_pending | -| effect_pending | settles with tool calls | `TX[ insert response entry R, upsert lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, tools{plan with reserved result ids}) ]` | tools | -| effect_pending | retryable error, attempts remain | `TX[ insert response entry R, upsert lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, assistant{retry_wait, nextAttempt k+1, notBefore}) ]` | retry_wait | -| effect_pending | first overflow, preparation non-empty | `TX[ insert response entry R **normalized to error**, upsert lane.leaf = R, insert usage U, upsert op.preparation/O:{taskId} = P, S(latestAssistantEntryId=R, compaction{reason:overflow, structural:{deciding, taskId}, resumeAfter:{checkpoint, prior trigger, need_assistant(true)}}) ]` | compaction | -| effect_pending | first overflow, preparation empty | `TX[ insert normalized response entry R, upsert lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, failure_drain{error, provenance:response R}) ]` | failure_drain | -| effect_pending | `stopReason: "deferred"` | `TX[ insert response entry R, upsert lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, deferred{suspended, sourceEntryId R, poll 0, configuration/options copied}) ]` | deferred | -| effect_pending | `stop` or genuine `length` | `TX[ insert response entry R, upsert lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, checkpoint{may_finish, includeFinalAssistant:true}) ]` | checkpoint | -| effect_pending | terminal error, retries exhausted, or 2nd overflow | `TX[ insert response entry R, upsert lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, failure_drain{error, provenance:response R}) ]` | failure_drain | +| effect_pending | settles with tool calls | `TX[ insert response entry R, upsert pi.lane.leaf = R, insert usage U, delete frame list O:R, S(latestAssistantEntryId=R, tools{plan with reserved result ids}) ]` | tools | +| effect_pending | retryable error, attempts remain | `TX[ insert response entry R, upsert pi.lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, assistant{retry_wait, nextAttempt k+1, notBefore}) ]` | retry_wait | +| effect_pending | first overflow, preparation non-empty | `TX[ insert response entry R **normalized to error**, upsert pi.lane.leaf = R, insert usage U, upsert pi.op.preparation/O:{taskId} = P, S(latestAssistantEntryId=R, compaction{reason:overflow, structural:{deciding, taskId}, resumeAfter:{checkpoint, prior trigger, need_assistant(true)}}) ]` | compaction | +| effect_pending | first overflow, preparation empty | `TX[ insert normalized response entry R, upsert pi.lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, failure_drain{error, provenance:response R}) ]` | failure_drain | +| effect_pending | `stopReason: "deferred"` | `TX[ insert response entry R, upsert pi.lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, deferred{suspended, sourceEntryId R, poll 0, configuration/options copied}) ]` | deferred | +| effect_pending | `stop` or genuine `length` | `TX[ insert response entry R, upsert pi.lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, checkpoint{may_finish, includeFinalAssistant:true}) ]` | checkpoint | +| effect_pending | terminal error, retries exhausted, or 2nd overflow | `TX[ insert response entry R, upsert pi.lane.leaf = R, insert usage U, S(latestAssistantEntryId=R, failure_drain{error, provenance:response R}) ]` | failure_drain | | retry_wait | gated timer reaches `notBefore` | `TX[ S(assistant{ready, nextAttempt:k+1}) ]` | ready | +Every settlement row above — tool plan, retry, overflow, deferred, checkpoint, and failure alike — additionally executes `deleteList(pendingAssistantFrames(O, R))`; the first row shows it explicitly and the rest abbreviate. Settlement and frame cleanup are one atomic commit. + **There is never a durable "response without usage" or "response and usage without a decision."** All three land together or none do. `R` and `U` are minted at intent and exist only as strings in the state until settlement inserts the complete rows (§2.2). A settlement that plans tools mints each `resultEntryId` as a follower of `R`, inheriting its 48-bit timestamp (§1.2), so the assistant and its results form one id-cohesive group by construction. +### Streamed frame persistence + +During the effect window, the assistant procedure converts every provider stream event with pi-ai's `assistantMessageEventToFrame`. Terminal `done`/`error` events convert to no frame — final settlement is separate and terminal events are never stored. The converter snapshots the initial partial once in the `start` frame, strips repeated growing `partial` snapshots from later frames, and lets end frames carry one authoritative completed block value where a provider protocol supplies or corrects content only at the end. Interleaved blocks rely on `contentIndex`, never contiguity. Hydration calls `reduceAssistantMessageFrames`; the harness defines no second frame codec or reducer. + +For every convertible event, the procedure: + +```text +convert event to frame +→ construct frames = pendingAssistantFrames(O, R) +→ synchronously enqueue one invocation-fenced appendList(frames, frame) +→ attach the ordinary harness-fault observer to the returned promise +→ replace only the process-local latest frame-write promise reference +→ emit and await the existing message_start/message_update event delivery +→ consume the next provider event +``` + +The provider loop never awaits storage per frame; existing assistant event delivery remains awaited per event. Because `appendList` is enqueued synchronously, lane-line FIFO preserves provider-event order, and the model's bounded output bounds queued frame memory — a response queues at most its bounded output plus constant frame/transaction overhead. Replacing the latest promise reference never leaves an earlier rejection unobserved: every promise carries the fault observer before it is superseded. + +At stream settlement, the procedure stops frame admission and awaits the latest frame-write promise before `after_response`; lane FIFO means its completion implies completion of every earlier append. There is **no** timer, batcher, coalescer, active/waiting state, or flush API — public or internal. A failed frame append faults the harness before `after_response` starts; the complete final response stays process-local and never commits after a storage fault. + +Each append mutation verifies that the same operation, attempt, and reserved response id still own the lane when it executes; a terminal transaction that won first fences a stale append without recreating operation state (§4.9). Frames are auxiliary: a missing list is valid, frames never prove request admission or completion, never select a restart point, and a crash after the final frame append but before settlement still restores `effect_pending` — a complete-looking draft is not a settled response. Structural summary streams persist no frames (§3.9). There is no frame-persistence event, and reconnect replays no historical updates (§5.4, §5.5). + ### Classification order Pure, computed in memory before the settlement transaction. First match wins. @@ -1422,17 +1601,17 @@ Tool execution separates effect completion from source-ordered tree placement: | From | Trigger | Transaction | To | |---|---|---|---| -| call *i* `planned` | clearance passed (`before_tool`, lookup, arg validation) | `TX[ upsert op.tool_args/O:{stepId}:{i} = effective args, S(call i = effect_pending, replay) ]` | dispatch | -| call *i* `effect_pending` | tool calls `onUpdate(partial, { checkpoint:true })` | `TX[ upsert pending.tool_output/O:{resultEntryId} = partial ]` after invocation fencing; state unchanged | `effect_pending` | -| call *i* `effect_pending` | effect settled, update delivery awaited, `after_tool` applied, and `tool_end` emitted | `TX[ upsert pending.entry/{resultEntryId} = finalized result, delete pending.tool_output/O:{resultEntryId}, delete op.tool_fact/O:{resultEntryId}:*, S(call i = outcome_ready, terminate) ]` | `outcome_ready` | -| call *i* `planned` | unknown tool / invalid args / `before_tool` blocks or throws / control cancelled | stage the complete synthetic result in `pending.entry` and write `outcome_ready` without effect intent | `outcome_ready` | -| source-ready prefix | first non-completed calls are `outcome_ready` | `TX[ insert result entries in source order, delete their pending.entry values, insert reported usage, upsert lane.leaf, S(calls = completed / next checkpoint) ]` | `completed` or checkpoint | +| call *i* `planned` | clearance passed (`before_tool`, lookup, arg validation) | `TX[ upsert pi.op.tool_args/O:{stepId}:{i} = effective args, S(call i = effect_pending, replay) ]` | dispatch | +| call *i* `effect_pending` | tool calls `onUpdate(partial, { checkpoint:true })` | `TX[ upsert pi.pending.tool_output/O:{resultEntryId} = partial ]` after invocation fencing; state unchanged | `effect_pending` | +| call *i* `effect_pending` | effect settled; latest update delivery and latest checkpoint write awaited; `after_tool` applied; `tool_end` emitted | `TX[ upsert pi.pending.entry/{resultEntryId} = finalized result, delete pi.pending.tool_output/O:{resultEntryId}, delete pi.op.tool_memo/O:{resultEntryId}:*, S(call i = outcome_ready, terminate) ]` | `outcome_ready` | +| call *i* `planned` | unknown tool / invalid args / `before_tool` blocks or throws / control cancelled | stage the complete synthetic result in `pi.pending.entry` and write `outcome_ready` without effect intent | `outcome_ready` | +| source-ready prefix | first non-completed calls are `outcome_ready` | `TX[ insert result entries in source order, delete their pi.pending.entry values, insert reported usage, upsert pi.lane.leaf, S(calls = completed / next checkpoint) ]` | `completed` or checkpoint | -Every `onUpdate` remains a process-local `tool_update` observation. The synchronous callback calls `events.emit(tool_update)` and retains the latest delivery promise internally; tools neither receive nor await that promise. `checkpoint:true` additionally requests replacement of the invocation's bounded durable progress snapshot. Tools own bounding, cadence, and duplicate suppression; this first API imposes no generic byte cap or truncation on trusted in-process values. Checkpoint requests are drop-coalesced to one active commit and one latest waiting value: the synchronous request path immediately enqueues an active lane-line mutation or replaces the in-memory waiting value. When the tool promise settles, the harness stops accepting updates, closes checkpoint admission, and drops the waiting snapshot. An active checkpoint is already ordered before outcome staging on the lane mutation line; staging later deletes it. A failed active commit follows the ordinary storage-fault path and prevents staging. Before `after_tool`, the procedure awaits the latest update-delivery promise, which implies delivery of every earlier update. The built-in bash tool emits bounded live snapshots every 100 ms and requests a distinct checkpoint at most every two seconds. +Every `onUpdate` remains a process-local `tool_update` observation. The synchronous callback calls `events.emit(tool_update)` and retains the latest delivery promise internally; tools neither receive nor await that promise. `checkpoint:true` additionally requests replacement of the invocation's bounded durable progress snapshot: every such call synchronously enqueues one invocation-fenced scalar replacement on the lane mutation line, attaches the ordinary harness-fault observer to the returned promise, and replaces only the process-local latest checkpoint-write promise reference. No checkpoint write is dropped or coalesced; lane-line FIFO preserves request order, and each mutation verifies that the same call remains `effect_pending` when it executes. The tool alone controls cadence and duplicate suppression — a tool that requests checkpoints faster than storage can commit queues work in memory under the trusted-tool contract. Tools own bounding; this first API imposes no generic byte cap or truncation on trusted in-process values. When the tool promise settles, the harness stops accepting updates and closes checkpoint admission; a late request returns without committing. Before `after_tool`, the procedure awaits the latest update-delivery promise **and** the latest checkpoint-write promise — each implies completion of everything earlier in its queue. Checkpoint writes are ordered before outcome staging on the lane mutation line, and staging deletes the scalar; a failed checkpoint commit follows the ordinary storage-fault path and prevents staging. The built-in bash tool emits bounded live snapshots every 100 ms and requests a checkpoint at most every two seconds, only when the bounded snapshot differs from its last requested checkpoint. Outcome staging is the point after which the tool can never replay. After `after_tool`, the procedure constructs the complete canonical final result—bounded independently of progress snapshots—emits and awaits `tool_end` for a real execution, then stages its `ToolResultMessage`; the state carries only `terminate` and the reserved id. Tool-reported usage remains inside that staged message until materialization, where its ledger row commits atomically with the immutable entry. Added tool names likewise become active from the materialized transcript point, never from invisible staging. -After any outcome stages, the procedure materializes the contiguous `outcome_ready` prefix beginning at the first non-completed source position. Several results may enter the tree in one transaction, each parented to the previous inserted result. When the final call materializes, that same state write deletes the batch's `op.tool_args/{O}:{stepId}:*` keys and chooses: +After any outcome stages, the procedure materializes the contiguous `outcome_ready` prefix beginning at the first non-completed source position. Several results may enter the tree in one transaction, each parented to the previous inserted result. When the final call materializes, that same transaction deletes the addresses returned by `scanValues(operationToolArgsPrefix(O, stepId))` and chooses: - **every** completed call set `terminate: true` → `checkpoint{may_finish, includeFinalAssistant: false}`; - otherwise → `checkpoint{need_assistant(overflowRecoveryUsed: false)}`. @@ -1446,7 +1625,7 @@ Modes: Blocked and invalid calls skip intent/execution but still stage a synthetic outcome. A crash before staging reruns ordinary clearance, including `before_tool` under its replay contract. A crash after staging never reruns the hook or tool. -Calls are tracked internally by `sourceIndex`. Hooks and events see provider `toolCallId` and tool name. `AgentHarnessToolInvocation.invocationId` equals the reserved `resultEntryId`, remains stable across safe replay, and scopes durable facts under `op.tool_fact/O:{invocationId}:*`. Fact names must be non-empty and contain no `:`; `setFact(name, undefined)` deletes. Fact operations synchronously enqueue on the lane mutation line before returning their promises, and tools must await writes. Each job verifies the same effect-pending invocation when it executes, protecting against authorized external finalization. A pre-return write is FIFO-ordered before staging and then deleted by staging; a post-return call rejects after capability expiry. No separate write drain exists. Flue-style `step.do(name, effect)` awaits these operations: a committed value returns on replay, while a crash before its fact commit may rerun the effect. There is no nested per-step replay state and no exactly-once external-effect promise. +Calls are tracked internally by `sourceIndex`. Hooks and events see provider `toolCallId` and tool name. `AgentHarnessToolInvocation.invocationId` equals the reserved `resultEntryId`, remains stable across safe replay, and scopes durable memos under `operationToolMemo(O, invocationId, name)` addresses. Memo names must be non-empty and contain no `:`; `setMemo(name, undefined)` deletes. Memo operations synchronously enqueue on the lane mutation line before returning their promises, and tools must await writes. Each job verifies the same effect-pending invocation when it executes, protecting against authorized external finalization. A pre-return write is FIFO-ordered before staging and then deleted by staging; a post-return call rejects after capability expiry. No separate write drain exists. Flue-style `step.do(name, effect)` awaits these operations: a committed value returns on replay, while a crash before its memo commit may rerun the effect. There is no nested per-step replay state and no exactly-once external-effect promise. ## 3.9 Summary generation — compaction and navigation summaries @@ -1463,21 +1642,21 @@ Both operations generate a summary through the same `deciding → generating → | `threshold` | context-size check at a checkpoint | back to the stored `resumeAfter` | | `overflow` | a request that did not fit | `failure_drain` | -"Auto compaction" is the in-run row: `threshold` and `overflow`. Non-empty preparation and the transition into `deciding` commit together (`upsert op.preparation/O:{taskId}` plus the structural state and, for threshold, marked `resumeAfter`). Preparation returning `undefined` never creates `StructuralDecision`: threshold atomically marks the checkpoint checked and continues; overflow atomically enters response-provenance `failure_drain` using the normalized overflow response. Neither path emits structural lifecycle. Empty standalone preparation is rejected before acceptance. +"Auto compaction" is the in-run row: `threshold` and `overflow`. Non-empty preparation and the transition into `deciding` commit together (`upsert pi.op.preparation/O:{taskId}` plus the structural state and, for threshold, marked `resumeAfter`). Preparation returning `undefined` never creates `StructuralDecision`: threshold atomically marks the checkpoint checked and continues; overflow atomically enters response-provenance `failure_drain` using the normalized overflow response. Neither path emits structural lifecycle. Empty standalone preparation is rejected before acceptance. | From | Trigger | Transaction | |---|---|---| | deciding | hook declines | standalone: the terminal transaction (§3.13) with outcome `declined` · threshold: `TX[ S(restore marked resumeAfter) ]` · overflow: `TX[ S(failure_drain{error, provenance:structural taskId}) ]` | -| deciding | hook supplies compaction | standalone: `TX[ insert hook usage row?, insert compaction entry, upsert lane.leaf, terminal writes (§3.13) ]`; in-run: same result-publication writes plus `S(resumeAfter)` | +| deciding | hook supplies compaction | standalone: `TX[ insert hook usage row?, insert compaction entry, upsert pi.lane.leaf, terminal writes (§3.13) ]`; in-run: same result-publication writes plus `S(resumeAfter)` | | deciding | hook supplies navigation summary | use §3.10's final transaction with the hook usage/result | | deciding | hook selects generation | conditionally snapshot current config/policy inline in `TX[ S(generating{ready}) ]` — **the decision hook will never run again** | | generating ready / retry elapsed | ordinary procedure | `TX[ S(effect_pending, attempt k) ]` | | generating effect_pending | one nested request returns | `TX[ insert usage row under request.usageId, S(effect_pending, request cleared, usageIds += id) ]`; commit another request intent before request two | | generating effect_pending | retryable attempt outcome | usage is already durable; `TX[ S(retry_wait) ]` | | generating effect_pending | terminal or attempts exhausted | standalone: the terminal transaction (§3.13) with outcome `failed` · in-run: `TX[ S(failure_drain{provenance:structural taskId}) ]` | -| generating effect_pending | compaction succeeded | standalone: `TX[ insert result entry, upsert lane.leaf, terminal writes (§3.13) ]`; in-run: result-publication writes plus `S(resumeAfter)` | +| generating effect_pending | compaction succeeded | standalone: `TX[ insert result entry, upsert pi.lane.leaf, terminal writes (§3.13) ]`; in-run: result-publication writes plus `S(resumeAfter)` | -Structural provider streams are internal: they emit **no** public assistant-message lifecycle. The existing summary generator is retained, but its one/two request callback uses the nested request intent/effect/usage boundaries from §3.2 and §4.2. Intermediate content is not persisted; a crash before the final transaction makes the whole attempt unknown, and a later numbered attempt starts only under the captured retry policy. Failed-attempt usage stays in the ledger regardless — terminal cleanup deletes registers, never ledger rows (§1.6). +Structural provider streams are internal: they emit **no** public assistant-message lifecycle and persist **no** `pendingAssistantFrames(...)` list — a future structural-diagnostics need would add a separate explicitly scoped consumer rather than silently reusing transcript-assistant semantics. The existing summary generator is retained, but its one/two request callback uses the nested request intent/effect/usage boundaries from §3.2 and §4.2. Intermediate content is not persisted; a crash before the final transaction makes the whole attempt unknown, and a later numbered attempt starts only under the captured retry policy. Failed-attempt usage stays in the ledger regardless — terminal cleanup deletes values/lists, never ledger rows (§1.6). ### Worked example — overflow @@ -1493,8 +1672,8 @@ Structural provider streams are internal: they emit **no** public assistant-mess ``` TX[ insert e_41 = { …assistant response, stopReason: "error", errorMessage: "context window exceeded: …" }, - upsert lane.leaf/main = "e_41", insert usage u_41, - upsert op.preparation/op_9:t_1 = , + upsert pi.lane.leaf/main = "e_41", insert usage u_41, + upsert pi.op.preparation/op_9:t_1 = , S(compaction{ reason: overflow, structural: { deciding, taskId: "t_1" }, resumeAfter: { checkpoint, triggerEntryId: "e_40", @@ -1527,43 +1706,43 @@ Unsummarized and summarized both finish in **one** transaction — navigation's ``` TX[ insert hook-reported usage row (only for a hook-supplied summary), - upsert lane.leaf = target, + upsert pi.lane.leaf = target, insert summary entry with its display usage snapshot (when summarize; parent is the target; fromId = the operation's sourceLeafId — the pre-navigation source leaf), - upsert lane.leaf = summary entry (when summarize), - upsert fact.label (when a label is present), - delete the operation's op.* registers, - upsert lane.lastResult = { kind: "navigation", outcome: "completed", + upsert pi.lane.leaf = summary entry (when summarize), + setValue(entryLabel(targetId), label) (when a label is present), + delete the operation-owned values/lists, + upsert pi.lane.lastResult = { kind: "navigation", outcome: "completed", oldLeafId: sourceLeafId, leafId, summaryEntryId?: generatedSummaryId }, L({ currentOperationId: null }) ] ``` -Writes apply in order inside the transaction. Generated provider usage was already written per request in §3.9 and is not written again here; the summary payload only snapshots its producing attempt's usage. The summary entry explicitly names the target as parent, and the following register write makes that summary the completed lane leaf. A crash sees either an untouched navigation still at its source, or a fully completed one. **No prepared-summary state and no post-move recovery state exist.** Abort before this transaction ends in an aborted terminal transaction with no entry appended; abort after it means the operation completed. +Writes apply in order inside the transaction. Generated provider usage was already written per request in §3.9 and is not written again here; the summary payload only snapshots its producing attempt's usage. The summary entry explicitly names the target as parent, and the following value write makes that summary the completed lane leaf. A crash sees either an untouched navigation still at its source, or a fully completed one. **No prepared-summary state and no post-move recovery state exist.** Abort before this transaction ends in an aborted terminal transaction with no entry appended; abort after it means the operation completed. ## 3.11 Inbox, queues, deferred writes -Every queued admission mints the item's entry id (§1.2) and writes its payload once into `pending.entry/{id}`; queue lists carry only the id. +Every queued admission mints the item's entry id (§1.2) and writes its payload once into `pendingEntry(id)`; queue lists carry only the id. | Public input | Admitted when | Transaction | |---|---|---| -| `nextRun(msg)` | any state, including idle | `TX[ upsert pending.entry/{id} = payload, L(pendingNextRun += id) ]` — never starts a run | -| `steer(msg)` | open run with running control — including deferred suspension; under `cancel_requested` → `NoActiveRun` | `TX[ upsert pending.entry/{id} = payload, S(inbox.steer += id) ]` | -| `followUp(msg)` | open run with running control — including deferred suspension; under `cancel_requested` → `NoActiveRun` | `TX[ upsert pending.entry/{id} = payload, S(inbox.followUp += id) ]` | -| tree write, run active | including suspended and cancelling | `TX[ upsert pending.entry/{id} = payload, S(inbox.writes += id) ]` — survives abort | -| tree write, lane idle | idle | `TX[ insert entry, upsert lane.leaf ]` | +| `nextRun(msg)` | any state, including idle | `TX[ upsert pi.pending.entry/{id} = payload, L(pendingNextRun += id) ]` — never starts a run | +| `steer(msg)` | open run with running control — including deferred suspension; under `cancel_requested` → `NoActiveRun` | `TX[ upsert pi.pending.entry/{id} = payload, S(inbox.steer += id) ]` | +| `followUp(msg)` | open run with running control — including deferred suspension; under `cancel_requested` → `NoActiveRun` | `TX[ upsert pi.pending.entry/{id} = payload, S(inbox.followUp += id) ]` | +| tree write, run active | including suspended and cancelling | `TX[ upsert pi.pending.entry/{id} = payload, S(inbox.writes += id) ]` — survives abort | +| tree write, lane idle | idle | `TX[ insert entry, upsert pi.lane.leaf ]` | | tree write, structural op open | — | wait for the operation to end, then re-evaluate | -| `cancelQueued(id)` | item still pending | `TX[ S or L with the id removed, delete pending.entry/{id} ]` | -| checkpoint consumes input | eligible | `TX[ insert entries from the register payloads, delete their pending.entry registers, upsert lane.leaf, S(ids removed, continuation → need_assistant(false), triggerEntryId = newest entry, skipInboxOnce = true) ]` | -| first `abort()` | run active | `TX[ S(control = cancel_requested, requestedAt, drainedSteer, drainedFollowUp, steer/followUp emptied) ]` — drained pending.entry registers are **not** deleted | +| `cancelQueued(id)` | item still pending | `TX[ S or L with the id removed, delete pi.pending.entry/{id} ]` | +| checkpoint consumes input | eligible | `TX[ insert entries from pending values, delete those values, upsert pi.lane.leaf, S(ids removed, continuation → need_assistant(false), triggerEntryId = newest entry, skipInboxOnce = true) ]` | +| first `abort()` | run active | `TX[ S(control = cancel_requested, requestedAt, drainedSteer, drainedFollowUp, steer/followUp emptied) ]` — drained pending values are **not** deleted | | finish | inbox empty, no required continuation | the terminal transaction (§3.13) | -`cancelQueued` triage, in order: the id is still pending in a queue list → remove it and delete its `pending.entry` register in one transaction; the content is gone, never having touched the tree, and the call returns `cancelled`. An entry under that id exists → `already_consumed`. Neither → `not_found` — previously cancelled, cleared by abort, or never existed. A client retrying a lost cancel treats `not_found` as success. There are no disposition registers, and nothing here is ever a recovery input. +`cancelQueued` triage, in order: the id is still pending in a queue list → remove it and delete its `pendingEntry(id)` value in one transaction; the content is gone, never having touched the tree, and the call returns `cancelled`. An entry under that id exists → `already_consumed`. Neither → `not_found` — previously cancelled, cleared by abort, or never existed. A client retrying a lost cancel treats `not_found` as success. There are no disposition values, and nothing here is ever a recovery input. -The first `abort()` moves steer/follow-up ids into `control.drainedSteer`/`control.drainedFollowUp` but deletes none of their `pending.entry` registers: `AbortResult` and a post-crash `SuspendedOperation.aborting` dereference the drained payloads from those registers. They die in the terminal transaction (§3.13), never earlier. Deferred writes stay in `inbox.writes` and are applied during reconciliation. +The first `abort()` moves steer/follow-up ids into `control.drainedSteer`/`control.drainedFollowUp` but deletes none of their `pendingEntry(id)` values: `AbortResult` and a post-crash `SuspendedOperation.aborting` dereference the drained payloads from those values. They die in the terminal transaction (§3.13), never earlier. Deferred writes stay in `inbox.writes` and are applied during reconciliation. -Because acceptance, cancellation, consumption, abort, and finish all serialize on the lane mutation line, every race has exactly two possible histories, and **no item can be both pending and applied** in durable state: at every commit boundary a queued id has its register (pending or drained), its entry (consumed), or neither (cancelled) — never both. +Because acceptance, cancellation, consumption, abort, and finish all serialize on the lane mutation line, every race has exactly two possible histories, and **no item can be both pending and applied** in durable state: at every commit boundary a queued id has its pending value (pending or drained), its entry (consumed), or neither (cancelled) — never both. ## 3.12 The checkpoint procedure @@ -1579,15 +1758,15 @@ Order matters. At each queue drain point, `"all"` consumes every currently eligi Consumed steer/follow-up and projecting message writes enter `need_assistant(false)`, set `triggerEntryId` to the newest appended entry, and set `skipInboxOnce`. Tool results do the same unless every result terminates. An unprojected custom write is appended and removed from the inbox but preserves the prior continuation, failure provenance, and overflow flag. Under cancelled control, every deferred write is appended and removed without changing phase/continuation or starting work; reconciliation ends in an aborted terminal transaction after writes drain. -`before_run_end` may return a follow-up. It commits **only** if control is still running and the operation is still at the same finish boundary; otherwise the stale hook result is dropped. The follow-up is born placed — its entry and the `need_assistant` state commit together, with no pending register. +`before_run_end` may return a follow-up. It commits **only** if control is still running and the operation is still at the same finish boundary; otherwise the stale hook result is dropped. The follow-up is born placed — its entry and the `need_assistant` state commit together, with no pending value. `failure_drain` applies accepted writes, then eligible steer and follow-up input in the same order. Projecting user-context input atomically enters `checkpoint{need_assistant(false)}` and clears the failure. Unprojected custom writes do not. With no such input, it finishes failed without `before_run_end` or another provider request. ## 3.13 Terminal transactions -There is no finished state. An operation ends by ceasing to exist: one **terminal transaction** deletes every register the operation owns, records the outcome in `lane.lastResult`, and clears the lane's `currentOperationId`. After it commits, the operation's only durable footprint is the conversation entries and ledger rows it produced. +There is no finished state. An operation ends by ceasing to exist: one **terminal transaction** deletes every value/list the operation owns, records the outcome in `pi.lane.lastResult`, and clears the lane's `currentOperationId`. After it commits, the operation's only durable footprint is the conversation entries and ledger rows it produced. -The result is computed in memory, pre-commit, from the final operation state — the same value the caller's promise resolves with. What lands durably is its register form: +The result is computed in memory, pre-commit, from the final operation state — the same value the caller's promise resolves with. What lands durably is its stored-value form: ```ts type FailedLaneLastResult = @@ -1619,7 +1798,7 @@ type LaneLastResult = summaryEntryId?: string })); ``` -A normal run finish copies `RunState.latestAssistantEntryId` and records `runCompletion: "assistant"` when `may_finish.includeFinalAssistant` is true. An all-terminating tool batch records `runCompletion: "terminated_tools"` and omits the final assistant. Failed and aborted run outcomes include the newest settled assistant when non-null and omit the field otherwise. Structural operations omit `runCompletion` and the final assistant. Navigation records its source leaf and, when it publishes one, the summary entry id, so a later matching `drive` can hydrate the complete public result after `op.meta` has been deleted. Only terminal transitions construct a `LaneLastResult`. +A normal run finish copies `RunState.latestAssistantEntryId` and records `runCompletion: "assistant"` when `may_finish.includeFinalAssistant` is true. An all-terminating tool batch records `runCompletion: "terminated_tools"` and omits the final assistant. Failed and aborted run outcomes include the newest settled assistant when non-null and omit the field otherwise. Structural operations omit `runCompletion` and the final assistant. Navigation records its source leaf and, when it publishes one, the summary entry id, so a later matching `drive` can hydrate the complete public result after `pi.op.meta` has been deleted. Only terminal transitions construct a `LaneLastResult`. Every terminal transaction, for every operation kind and outcome, has one shape: @@ -1627,40 +1806,46 @@ Every terminal transaction, for every operation kind and outcome, has one shape: TX[ , - delete op.meta/{O}, - delete op.state/{O}, - delete op.tool_args/{O}:* defensive prefix scan — listRegisters with - keyPrefix (§1.5); batch completion already - deletes these atomically (§3.8), - delete op.tool_fact/{O}:* defensive cleanup of invocation memos, - delete op.preparation/{O}:* prefix scan; in-run compactions leave their - preparation after resume, - delete pending.tool_output/{O}:* defensive cleanup of progress checkpoints, - delete pending.entry/{id} for every operation-owned pending id, - upsert lane.lastResult/{lane} = , + delete pi.op.meta/{O}, + delete pi.op.state/{O}, + delete addresses from scanValues(operationToolArgsPrefix(O)) + defensively; batch completion already deletes + these atomically (§3.8), + delete addresses from scanValues(operationToolMemoPrefix(O)) + defensive cleanup of invocation memos, + delete addresses from scanValues(operationPreparationPrefix(O)) + in-run compactions leave preparation after resume, + delete addresses from scanValues(pendingToolOutputPrefix(O)) + defensive cleanup of progress checkpoints, + delete list pi.pending.assistant_frame/{O}:{R} + when current state is assistant/deferred + effect_pending with response id R; response + settlement already deleted its exact key, + delete pi.pending.entry/{id} for every operation-owned pending id, + upsert pi.lane.lastResult/{lane} = , L({ currentOperationId: null }) ] ``` -Operation-owned pending ids are the remaining `inbox.steer ∪ inbox.followUp ∪ inbox.writes`, `control.drainedSteer ∪ control.drainedFollowUp`, and every staged `outcome_ready` result not already materialized. **Never `lane.state.pendingNextRun`**: those registers are lane-owned, outlive operations, and die only when consumed or cancelled. Ledger rows are never deleted (§1.6). The `L` write rereads the latest `LaneState` on the lane mutation line and clears only `currentOperationId`, preserving concurrently accepted `pendingNextRun` (§3.4). +Operation-owned pending ids are the remaining `inbox.steer ∪ inbox.followUp ∪ inbox.writes`, `control.drainedSteer ∪ control.drainedFollowUp`, and every staged `outcome_ready` result not already materialized. **Never `pi.lane.state.pendingNextRun`**: those values are lane-owned, outlive operations, and die only when consumed or cancelled. Ledger rows are never deleted (§1.6). The `L` write rereads the latest `LaneState` on the lane mutation line and clears only `currentOperationId`, preserving concurrently accepted `pendingNextRun` (§3.4). For the completed run of §0.4's shape — prompt `e_50`, tool call `e_51`/`e_52`, final answer `e_53`: ``` -TX[ delete op.meta/op_9, - delete op.state/op_9, - delete op.tool_args/op_9:s_1:0, ← usually already gone at batch completion - upsert lane.lastResult/main = { operationId: "op_9", kind: "run", +TX[ delete pi.op.meta/op_9, + delete pi.op.state/op_9, + delete pi.op.tool_args/op_9:s_1:0, ← usually already gone at batch completion + upsert pi.lane.lastResult/main = { operationId: "op_9", kind: "run", outcome: "completed", leafId: "e_53", finalAssistantEntryId: "e_53", runCompletion: "assistant" }, - upsert lane.state/main = { currentOperationId: null, pendingNextRun: [] } ] + upsert pi.lane.state/main = { currentOperationId: null, pendingNextRun: [] } ] ``` -After it, the session holds exactly the conversation entries, the ledger rows, and the lane's registers (`lane.leaf`, `lane.config`, `lane.state`, `lane.lastResult`). The run's `op.state` revisions, tool arguments, invocation facts, tool progress checkpoints, and pending payloads are gone logically. JSONL reclaims their superseded bytes through snapshot compaction (§1.7). +After it, the session holds exactly the conversation entries, the ledger rows, and the lane's values (`laneLeaf`, `laneConfig`, `laneState`, `laneLastResult`). The run's `pi.op.state` revisions, tool arguments, invocation memos, tool progress checkpoints, assistant frame lists, and pending payloads are gone logically. JSONL reclaims their superseded bytes through snapshot compaction (§1.7). -**The observation contract.** A terminal outcome is observable once through the live caller's promise (and the corresponding `run_end`/`compaction_end`/`navigation_end` event), which carries the full in-memory result, and thereafter through `lane.lastResult` until the next terminal transaction on the same lane overwrites it. `lane.lastResult` is written only by terminal transactions — one bounded register per lane, forever. Recovery never reads it: restore treats a lane with `currentOperationId: null` as idle regardless of the register's content. It exists so an application that accepted an operation, lost its process, and reopened can still answer "what happened to `op_9`?" — including outcomes the tree alone cannot reconstruct: a structural failure's error, `declined`, and the `aborted`-versus-`completed` ambiguity of a leaf that moved. +**The observation contract.** A terminal outcome is observable once through the live caller's promise (and the corresponding `run_end`/`compaction_end`/`navigation_end` event), which carries the full in-memory result, and thereafter through `pi.lane.lastResult` until the next terminal transaction on the same lane overwrites it. `pi.lane.lastResult` is written only by terminal transactions — one bounded value per lane, forever. Recovery never reads it: restore treats a lane with `currentOperationId: null` as idle regardless of that value's content. It exists so an application that accepted an operation, lost its process, and reopened can still answer "what happened to `op_9`?" — including outcomes the tree alone cannot reconstruct: a structural failure's error, `declined`, and the `aborted`-versus-`completed` ambiguity of a leaf that moved. -The invariant this section carries (restated in Part 9): operation-owned `op.*`, `pending.entry`, and `pending.tool_output` registers may exist only while their operation is open, because the terminal transaction deletes them atomically with clearing `currentOperationId`. There is no partial-cleanup state to observe or repair. +The invariant this section carries (restated in Part 9): operation-owned values and assistant-frame lists may exist only while their operation is open, because the terminal transaction deletes them atomically with clearing `currentOperationId`. There is no partial-cleanup state to observe or repair. # Part 4 — Execution, recovery, abort, close @@ -1692,7 +1877,7 @@ The lane implementation also has one breakpoint barrier used by `peekAction()`/` This is not another state machine. `completion` is the ordinary promise returned by the async operation procedure. `effectGate` owns only abort-versus-start arbitration and the operation's cooperative signal (§4.2). Current state, retry data, responses awaiting settlement, structural intermediate content, and parallel tool promises remain ordinary procedure-local variables. Streaming drafts and running-tool display values are ordinary lane fields used only for snapshots. -`op.state/{operationId}` is the operation's **durable restart point**, not the live JavaScript instruction pointer. While a task is alive, its call stack is the finer program counter: +`pi.op.state/{operationId}` is the operation's **durable restart point**, not the live JavaScript instruction pointer. While a task is alive, its call stack is the finer program counter: ```text live task: @@ -1760,9 +1945,9 @@ Normal execution handles checkpoints, ready/retry states, planned or locally own ### Task installation, joining, and removal -`accept` commits the first durable state and returns without installing `ActiveOperation`. A durably accepted but locally unowned operation is ordinary. `drive({ operationId })` owns activation. On the lane mutation line it reads current ownership and `lane.lastResult`, then applies exactly one rule: +`accept` commits the first durable state and returns without installing `ActiveOperation`. A durably accepted but locally unowned operation is ordinary. `drive({ operationId })` owns activation. On the lane mutation line it reads current ownership and `pi.lane.lastResult`, then applies exactly one rule: -- current operation absent and `lane.lastResult.operationId === operationId`: return its already-settled outcome; +- current operation absent and `pi.lane.lastResult.operationId === operationId`: return its already-settled outcome; - current operation absent with another/no last id, or current operation has another id: return `OperationMismatch` without starting work; - the same operation already has `ActiveOperation`: join that pass's `completion`; - the same operation has no task: install one deferred completion promise and fresh gate before releasing the lane line, then start the pass after `Session.mutate` returns. @@ -1825,7 +2010,7 @@ breakpoint → operation code ``` -Public lane calls—acceptance, `steer`, `followUp`, `nextRun`, `cancelQueued`, `requestAbort`/`abort`, configuration and fact setters, and tree writes—do not wait at this barrier. They remain able to race a parked drive pass through `Session.mutate`. In manual mode a registered `before_run` hook parks before acceptance; without one, acceptance commits immediately and the first operation breakpoint follows when `drive` claims it. +Public lane calls—acceptance, `steer`, `followUp`, `nextRun`, `cancelQueued`, `requestAbort`/`abort`, configuration and value setters, and tree writes—do not wait at this barrier. They remain able to race a parked drive pass through `Session.mutate`. In manual mode a registered `before_run` hook parks before acceptance; without one, acceptance commits immediately and the first operation breakpoint follows when `drive` claims it. Abort rejects an ordinary parked breakpoint with `AbortRequested` instead of requiring another `executeAction()`. The signal carries the abort-mutation promise; the task waits for it, reloads durable ownership/control, and either enters cancellation reconciliation or observes that the operation already ended, without executing the ordinary statement after the interrupted barrier. Reconciliation's own commits and cleanup effects still park at their documented breakpoints. Close/fault rejects a parked barrier without executing the following code. An operation performs zero procedure writes and starts zero hooks/providers/tools/timers while parked. @@ -1915,7 +2100,7 @@ Request preparation, identity preflight, context transformation, and option comp No other code calls `assertOpen()`. In particular, it is not used for: -- `Session.mutate` or any public queue/configuration/fact/tree mutation; +- `Session.mutate` or any public queue/configuration/value/tree mutation; - pure classification, transaction construction, or synthetic settlement; - `prepareArguments`, `systemPrompt`, `toolContext`, `toProviderMessages`, or entry projectors; - awaiting a provider/tool promise that was already admitted; @@ -1941,7 +2126,7 @@ abort first → task waits for the abort commit and reconciles ``` -This gate is not durable state, a mutex, a scheduler, or a second mutation line. If the process dies before the cancellation commit, its closed gate disappears and no cancellation exists. Recovery trusts only `op.state.control`. +This gate is not durable state, a mutex, a scheduler, or a second mutation line. If the process dies before the cancellation commit, its closed gate disappears and no cancellation exists. Recovery trusts only `pi.op.state.control`. Every procedure boundary emits its existing telemetry span around the ordinary code following the breakpoint. Breakpoints themselves do not replace telemetry. @@ -1949,21 +2134,21 @@ Every procedure boundary emits its existing telemetry span around the ordinary c Every state-dependent mutation on a lane is linearized by `Session.mutate`: read latest state, decide, commit at most once, and complete the process-local update before the next mutation starts. Provider, tool, hook, and timer work never occupies the line. The callback-scoped `SessionMutator` enforces the one-commit limit; a high-level `SessionTree` write uses the same mechanism internally. -An operation transition rereads the latest `lane.state`, `op.state`, leaf, and any dependent registers inside that mutation. It verifies the expected semantic restart point or pending effect identity, preserves fields owned by concurrent public calls, and commits the complete next state. A mismatch is not an ordinary boolean "replan" result: +An operation transition rereads the latest `pi.lane.state`, `pi.op.state`, leaf, and any dependent values inside that mutation. It verifies the expected semantic restart point or pending effect identity, preserves fields owned by concurrent public calls, and commits the complete next state. A mismatch is not an ordinary boolean "replan" result: - `cancel_requested` routes through the task's cancellation control flow; -- absent operation registers mean its terminal transaction already committed (§4.9); +- absent operation values mean its terminal transaction already committed (§4.9); - any other unexpected phase/identity is an invariant defect. Settlement therefore cannot overwrite steer/write acceptance or cancellation with a stale snapshot. It rereads the latest state, verifies that the same request or tool call remains pending, and merges the output while preserving newer inbox/control fields. -What serializes on the lane line: operation acceptance, drive installation/join/removal, execution inspection, queue enqueue/cancel/consumption, deferred writes, cancellation requests, tool progress checkpoints, invocation fact access, outcome staging, source-ordered outcome materialization, lane-configuration setters, finish, and lane creation. All durable races consequently have the orders listed in Part 9. The effect gate is needed only because synchronous external-call admission is not a durable mutation. +What serializes on the lane line: operation acceptance, drive installation/join/removal, execution inspection, queue enqueue/cancel/consumption, deferred writes, cancellation requests, assistant frame appends, tool progress checkpoints, invocation memo access, outcome staging, source-ordered outcome materialization, lane-configuration setters, finish, and lane creation. All durable races consequently have the orders listed in Part 9. The effect gate is needed only because synchronous external-call admission is not a durable mutation. Harness-global settings use a second process-local mutation line with a monotonically increasing revision. Run acceptance snapshots compaction, queue modes, and tool execution into `RunState.settings`. A generation or structural-generation start snapshots only current stream options and retry policy, then reads lane configuration on its `Session.mutate` job and commits those values with the ready state. Global setters take only the settings line; lane setters take only the lane line. Snapshot procedures take the settings line before the lane line, and no code acquires them in reverse order. This gives setter-first or step-start-first ordering without holding either line across external work. ## 4.4 Restore -Recovery is point lookups against registers. No history, no folding, no journal replay, no tree walk. Per lane: +Recovery is point lookups against bound values. No history, no folding, no journal replay, no tree walk, no reference audit. Session restore first calls `scanValues(laneLeafInventoryPrefix())`; the returned keys are the trusted configured-lane inventory, and `main` is required. It then restores each listed lane exactly once. Per lane: ```ts interface CurrentOperation { @@ -1977,25 +2162,25 @@ interface CurrentOperation { async function restore(lane: string): Promise< { kind: "idle"; lane: string } | { kind: "suspended"; current: CurrentOperation } > { - const config = await storage.getRegister("lane.config", lane); - const state = await storage.getRegister("lane.state", lane); - const leaf = await storage.getRegister("lane.leaf", lane); + const config = await storage.getValue(laneConfig(lane)); + const state = await storage.getValue(laneState(lane)); + const leaf = await storage.getValue(laneLeaf(lane)); + + // Projection existence only (§3.3). Committed internal values are trusted; + // nothing they reference is dereferenced here. + if (!config || !state || !leaf) { + throw new Error(`Invariant defect: incomplete lane projection for ${lane}`); + } const opId = state.value.currentOperationId; - const meta = opId ? await storage.getRegister("op.meta", opId) : undefined; - const opState = opId ? await storage.getRegister("op.state", opId) : undefined; - - // Idle lanes are validated too: leaf existence and every pendingNextRun - // id's pending.entry register (§3.3). Only operation checks depend on opId. - const entryIds = directEntryIds(opState?.value, meta?.value, state.value, leaf.value); - const registerKeys = directRegisterKeys(opState?.value, state.value); - const [entries, registers] = await Promise.all([ - storage.getEntries(entryIds), getRegisters(registerKeys), - ]); - validateCurrent({ config, state, leaf, meta, opState }, entries, registers); - if (!opId) return { kind: "idle", lane }; + const meta = await storage.getValue(operationMeta(opId)); + const opState = await storage.getValue(operationState(opId)); + if (!meta || !opState) { + throw new Error(`Invariant defect: incomplete operation projection for ${opId}`); + } + return { kind: "suspended", current: { operation: meta.value, state: opState.value, @@ -2006,16 +2191,11 @@ async function restore(lane: string): Promise< } ``` -Five register point-lookups: three lane registers, then—only when an operation is open—`op.meta` and `op.state`. The restart point contains or directly names everything activation needs. +Five value point-lookups: three lane values, then—only when an operation is open—operation metadata and state. The restart point contains or directly names everything activation needs. -**Bounded hydration and semantic checks.** From the loaded state, collect what it names directly and fetch it in one batch: +**Trusted projection, deferred dereference.** Committed internal values are trusted (§0.8, §3.3). Restore hydrates nothing beyond the five values above and audits no relationships. Every id the state names — trigger and result entries, prompt entries, pending payloads, tool arguments, preparations, staged outcomes, optional tool-output checkpoints, frame lists — is dereferenced by the procedure that consumes it, through the bounded exact address that procedure constructs; a required value found missing or mismatched there is an invariant defect raised at consumption. Absent auxiliary values — a `pendingToolOutput(operationId, invocationId)` checkpoint or a `pendingAssistantFrames(operationId, responseEntryId)` list — are always legal and never invalidate an effect-pending state. Invocation memos are read only by resumed application code and defensively cleaned by invocation/operation prefix. Configuration, stream options, and retry policy need no recovery lookup when captured inline. -- **entries:** `triggerEntryId`, `latestAssistantEntryId`, `batch.assistantEntryId`, deferred `sourceEntryId`, completed `resultEntryId`s, the lane leaf, and from `op.meta`—`promptEntryIds`, a non-null `sourceLeafId`, and a navigation intent's non-null `targetId`; -- **registers:** `op.tool_args/…` and optional `pending.tool_output/…` for effect-pending calls, required `pending.entry/…` for outcome-ready calls, `op.preparation/…` for structural work, and `pending.entry/…` for every `inbox.*`, `control.drained*`, and `pendingNextRun` id. - -Then run §3.3's bounded semantic checks over exactly that set. Every required named object exists and has the expected identity and relationship; an absent tool-output checkpoint is legal and never invalidates an effect-pending call. Completed tool calls form a source-ordered prefix. In sequential mode the suffix has at most one effect-pending or outcome-ready call before planned calls; in parallel mode it may mix planned, effect-pending, and outcome-ready calls. Restore does not resolve current tool code merely to audit historical choices. A tool result progresses through reserved string, outcome-ready `pending.entry`, then immutable entry; restore rejects collisions between those representations. Invocation facts are not scanned during base restore—they are read only by resumed application code and defensively cleaned by invocation/operation prefix. Configuration, stream options, and retry policy need no recovery lookup when captured inline. - -Restore never reads register history, folds anything, scans tables, builds provider context, audits completed operations, or infers a transition from an absent planned value. It installs no `ActiveOperation` and invokes no breakpoint. The application or serving layer explicitly activates an open operation with `drive({ operationId })`; `resume()` is the convenience composition that inspects and drives the current operation with one deferred-poll permit. Cancellation is durably admitted through `requestAbort(operationId)` or its `abort()` convenience wrapper. +Restore never reads value history, folds anything, scans tables, reads lists, builds provider context, audits completed operations, or infers a transition from an absent planned value. It installs no `ActiveOperation` and invokes no breakpoint. The application or serving layer explicitly activates an open operation with `drive({ operationId })`; `resume()` is the convenience composition that inspects and drives the current operation with one deferred-poll permit. Cancellation is durably admitted through `requestAbort(operationId)` or its `abort()` convenience wrapper. Harness creation does not write a durable `suspended` state. An open operation remains at its committed phase with control `running` or `cancel_requested`. `suspended` is the public observation that no process-local `drive()` call currently owns that operation. A matching `drive({ operationId })` installs the local owner and dispatches from the committed phase; process loss removes the owner without changing durable control or phase. @@ -2024,9 +2204,9 @@ Harness creation does not write a durable `suspended` state. An open operation r The process died after committing an assistant intent: ```text -lane.state/main -> { currentOperationId: "op_9" } -op.meta/op_9 -> { intent: run, sourceLeafId: "e_41" } -op.state/op_9 -> { phase: assistant effect_pending, attempt: 1, +pi.lane.state/main -> { currentOperationId: "op_9" } +pi.op.meta/op_9 -> { intent: run, sourceLeafId: "e_41" } +pi.op.state/op_9 -> { phase: assistant effect_pending, attempt: 1, responseEntryId: "e_51", usageId: "u_7", context: { configuration: { model: {...}, ... }, retryPolicy: { maxAttempts: 3, ... } } } @@ -2060,10 +2240,10 @@ The resume prelude first reloads durable control. If cancellation is requested, | Orphaned restart point | Activation recovery | |---|---| -| assistant generation `effect_pending` | If the captured retry policy permits, commit a later numbered `ready` attempt. Otherwise commit a synthetic error response and zero usage under the reserved ids, then continue through failure drain. | +| assistant generation `effect_pending` | Read bounded pages from `pendingAssistantFrames(O, R)` and reduce them with `reduceAssistantMessageFrames`. Commit, under the reserved ids, a synthetic zero-usage `error` response carrying the reconstructed partial content — empty, from the captured model/API identity, when no start frame committed — with an explicit warning that the request was interrupted, the preceding content is the latest committed partial, newer live output may be missing, and the external outcome is unknown. The same transaction deletes the frame list. The committed error then follows ordinary classification: attempts remaining → retry_wait and a later numbered attempt under fresh ids; cap reached → failure drain. Partial tool calls inside it never execute, and `after_response` never runs — there is no trustworthy complete provider result to transform. | | structural generation `effect_pending` | Treat the entire attempt as uncertain, including any completed first split-turn request whose intermediate text was process-local. Advance to a later `ready` attempt under the captured policy or fail at the cap. Already committed request-usage rows remain in the ledger. | -| tool call `effect_pending` | If both stored and current declarations say `safe`, delete any old progress checkpoint and re-execute persisted arguments with the same invocation facts/id. Otherwise preserve checkpoint content/details/usage when present, ignore its added-tool/termination hints, append an explicit latest-durable/newer-live-may-be-missing/unknown-outcome warning, and stage a non-terminating synthetic error without `after_tool`. | -| deferred poll `effect_pending` | The activation's single explicit poll permit replaces the unknown poll with fresh response/usage ids at the same poll number and fetches once. Without a permit it remains suspended. There is no cap. | +| tool call `effect_pending` | If both stored and current declarations say `safe`, delete any old progress checkpoint and re-execute persisted arguments with the same invocation memos/id. Otherwise preserve checkpoint content/details/usage when present, ignore its added-tool/termination hints, append an explicit latest-durable/newer-live-may-be-missing/unknown-outcome warning, and stage a non-terminating synthetic error without `after_tool`. | +| deferred poll `effect_pending` | The activation's single explicit poll permit replaces the unknown poll with fresh response/usage ids at the same poll number and fetches once; the replacement intent transaction deletes the abandoned old frame list (§3.2). Without a permit it remains suspended and may expose its durable partial in snapshots. There is no cap. | After this prelude removes or takes live ownership of every orphaned pending effect, the same ordinary procedures used by fresh execution continue. Calls already in `outcome_ready` require no identity or effect recovery; ordinary source-order materialization places their staged results. Recovery is not a second end-to-end driver. @@ -2076,7 +2256,7 @@ Atomic transactions have no internal prefix, so every repeat-sensitive effect st | during/after effect, before settlement | `effect_pending` | same unknown-outcome policy | | after settlement commit | output + usage + next state | continue; never re-settle | -Queue application and final structural commits remain atomic as specified in Part 3. A crash before one sees the prior complete state; a crash after one sees the next complete state. A crash after durable abort activates cancellation reconciliation. A crash after terminal cleanup sees an idle lane and `lane.lastResult`. +Queue application and final structural commits remain atomic as specified in Part 3. A crash before one sees the prior complete state; a crash after one sees the next complete state. A crash after durable abort activates cancellation reconciliation. A crash after terminal cleanup sees an idle lane and `pi.lane.lastResult`. Retry waits are ordinary restartable states with two caller policies: @@ -2111,7 +2291,7 @@ For a live operation, the first matching `requestAbort()` performs this order: 3. `Session.mutate` rereads the latest operation and either commits `cancel_requested` or observes that the expected id no longer owns the lane; 4. for runs, atomically move current steer/follow-up ids to `control.drained*` - and empty those inbox lists; their `pending.entry` registers remain; + and empty those inbox lists; their `pendingEntry(id)` values remain; 5. after a cancellation commit, resolve the promise and pull the `EffectGate`'s `AbortController`; after an ownership mismatch, resolve it and close with `OperationEnded`; 6. return `AbortRequestResult` once cancellation is durable, or `OperationMismatch` @@ -2126,7 +2306,7 @@ The effect gate orders only whether external work started. The lane mutation lin - settlement first → its normal next state commits, then abort marks that state cancelled; - terminal first → abort returns `NoActiveOperation`. -With queued steer/follow-up, successful abort removes their ids from the inbox, records them in `control.drained*`, and returns their dereferenced payloads. They never enter the tree during reconciliation. Their `pending.entry` registers survive so repeated abort and post-crash reporting can read them, then the terminal transaction deletes them. +With queued steer/follow-up, successful abort removes their ids from the inbox, records them in `control.drained*`, and returns their dereferenced payloads. They never enter the tree during reconciliation. Their `pendingEntry(id)` values survive so repeated abort and post-crash reporting can read them, then the terminal transaction deletes them. ### No live task @@ -2136,15 +2316,15 @@ After reopen, safe yield, or suspension there may be no gate to close or work to Cancellation reconciliation is separate from ordinary dispatch. It never starts a new ordinary provider request, tool, decision hook, or retry. It handles the current durable phase plus any local results still owned by the task: -- an intended assistant/fetch that really started settles under its reserved ids as `aborted`, retaining reported usage; -- an intended assistant/fetch with no live result settles synthetically as `aborted` with zero usage; +- an intended assistant/fetch that really started settles under its reserved ids as `aborted`, retaining reported usage; all accepted frame appends are awaited first, and normal settlement deletes the frame list; +- an intended assistant/fetch with no live result settles synthetically as `aborted` with zero usage, preserving the partial content reduced from committed frames; the settlement deletes the frame list, starts no provider request, and runs no response hook — cancellation wins classification even if the reduced content appears complete; - planned, unstarted tools stage aborted synthetic outcomes; - restored effect-pending tools stage interrupted outcomes, including the latest durable progress checkpoint when present, and are never replayed; - live started tools keep their raw/finalized result with `terminate:false`, regardless of whether the abort marker or outcome-staging commit wins the lane line; - already outcome-ready tools keep their staged complete results and materialize in source order; - structural work not already atomically published is discarded; - accepted deferred writes are still appended in order; -- operation-owned pending and drained registers are deleted by the aborted terminal transaction. +- operation-owned pending and drained values are deleted by the aborted terminal transaction. A scope holding a response or parallel tool promises performs the relevant local settlement before handing off to the shared remainder. This is why those values stay procedure-local rather than moving into a generic runtime object. @@ -2169,7 +2349,7 @@ seal harness admission → close storage and release the writer lease ``` -A harness-wide admission barrier orders close against every operation and public commit. A commit admitted first may finish and close waits for it; close first prevents admission. A provider/tool result produced after sealing cannot commit. A tool checkpoint already admitted may drain; a merely requested/waiting checkpoint may be lost. Durable state therefore stops at the last committed restart point, often `effect_pending` with zero or one bounded checkpoint, exactly as after process death. +A harness-wide admission barrier orders close against every operation and public commit. A commit admitted first may finish and close waits for it; close first prevents admission. A provider/tool result produced after sealing cannot commit. Tool checkpoint replacements and assistant frame appends already enqueued are ordinary admitted session work and may drain under the barrier; a request after sealing cannot commit, and live output newer than the last committed checkpoint or frame may be lost. Durable state therefore stops at the last committed restart point, often `effect_pending` with a committed frame prefix or bounded checkpoint, exactly as after process death. No synthetic response or cancellation marker is written. Reopening reports the open operation. A later matching `drive()`—direct or through `resume()`—applies activation recovery. No close-specific recovery path exists, and the durable aborted-implies-cancelled invariant remains true because close admits no locally aborted settlement under running control. @@ -2181,9 +2361,9 @@ Close rejects active drive and convenience-operation promises with `HarnessClose ## 4.9 External finalization -An authorized administrative operation may commit the terminal transaction while a task still exists. It uses the same lane mutation line. This specification adds no public `forceFinalize()` method: authorization, SLA policy, and initiation belong to repository or hosting administration, while the terminal-transaction helper and live-task notification remain package-internal. The next task mutation rereads current registers and finds that its operation is no longer current; the mutation raises the internal `OperationEnded` signal rather than returning a normal stale boolean. +An authorized administrative operation may commit the terminal transaction while a task still exists. It uses the same lane mutation line. This specification adds no public `forceFinalize()` method: authorization, SLA policy, and initiation belong to repository or hosting administration, while the terminal-transaction helper and live-task notification remain package-internal. The next task mutation rereads current values and finds that its operation is no longer current; the mutation raises the internal `OperationEnded` signal rather than returning a normal stale boolean. -The task then stops, closes its effect gate, discards process-local results without writing, emits end observation, and resolves from the matching `lane.lastResult` written by the terminal transaction. It never recreates operation registers or commits a competing terminal result. An unowned operation needs no task to stop; a later matching `drive()` returns the settled result from `lane.lastResult`, while `resume()` sees an idle lane and returns `NothingToResume`. +The task then stops, closes its effect gate, discards process-local results without writing, emits end observation, and resolves from the matching `pi.lane.lastResult` written by the terminal transaction. It never recreates operation values/lists or commits a competing terminal result. Scalar and list invocation writes are fenced the same way: invocation memo writes, tool checkpoint replacements, and assistant frame appends each verify durable ownership when their lane job executes, so a write enqueued before finalization is deleted by the terminal cleanup (which removes all operation-owned values/lists) and a write after it rejects without recreating state. An unowned operation needs no task to stop; a later matching `drive()` returns the settled result from `pi.lane.lastResult`, while `resume()` sees an idle lane and returns `NothingToResume`. At most one terminal transaction commits because every candidate rereads ownership on the lane line before writing. An in-process finalizer publishes the terminal events, records the matching result on the `ActiveOperation`, closes any parked breakpoint and its gate with `OperationEnded` immediately after the terminal commit, and pulls the controller so cooperative provider/tool work stops. The task then resolves without publishing a duplicate terminal event once its current awaited work returns; its outer `finally` removes the record. A new operation therefore waits/gets `LaneBusy` during that cleanup even though durable state is already idle. A replacement-process finalizer first acquires ownership after close/crash, so no live task exists to notify. @@ -2296,7 +2476,7 @@ interface WatchHandle { snapshot: T; start(listener: EventListener): void; un The convenience surface adds only caller-lifetime policy: `prompt`/`skill`/`compact`/`navigateTree` call `accept` then drive the returned id with local retry waiting; `resume` inspects and drives the current id with one deferred-poll permit; `abort` requests cancellation and ensures reconciliation is driven. A serving layer may instead store `OperationAdmission`, return to its caller, schedule from `waiting.notBefore`, and invoke the same primitives later. -`getLastResult()` is the post-crash reconciliation path: an application that accepted an operation, lost its process, and reopened reads the `lane.lastResult` register for the outcome its promise never delivered (§3.13). `inspectExecution()` reads current ownership and that latest result coherently, which lets a scheduler reconcile its own durable submission without an accept/settle race. `lane.lastResult` is still overwritten by the next terminal operation; permanent per-submission receipts remain a serving-layer concern. It is also how a caller learns the outcome of an operation finalized externally (§4.9). +`getLastResult()` is the post-crash reconciliation path: an application that accepted an operation, lost its process, and reopened reads `laneLastResult(lane)` for the outcome its promise never delivered (§3.13). `inspectExecution()` reads current ownership and that latest result coherently, which lets a scheduler reconcile its own durable submission without an accept/settle race. `pi.lane.lastResult` is still overwritten by the next terminal operation; permanent per-submission receipts remain a serving-layer concern. It is also how a caller learns the outcome of an operation finalized externally (§4.9). `waitForIdle()` registers on the lane mutation line and resolves when all earlier admitted lane jobs have settled, `currentOperationId` is null, and no process-local operation/admission reservation is held. Later operations may start immediately after it resolves. Multiple waiters resolve together; close/fault rejects pending waiters. @@ -2414,7 +2594,7 @@ class HarnessFault extends Error { class HarnessClosed extends Error {} ``` -`cancelQueued` has no unknown-item error: an id that is neither pending nor materialized returns `not_found` (§3.11) — previously cancelled, cleared by abort, or never existed — and a client retrying a lost cancel treats it as success. `AbortRequestResult` and `AbortResult` dereference steer/follow-up payloads from the drained items' surviving `pending.entry` registers (§4.6). `requestAbort` is idempotent for the same open operation and reports whether it wrote the marker; its expected-id fence fails rather than cancelling another operation. `recordUsage` mints its ledger row id at commit (§1.6) and returns it. +`cancelQueued` has no unknown-item error: an id that is neither pending nor materialized returns `not_found` (§3.11) — previously cancelled, cleared by abort, or never existed — and a client retrying a lost cancel treats it as success. `AbortRequestResult` and `AbortResult` dereference steer/follow-up payloads from the drained items' surviving `pendingEntry(id)` values (§4.6). `requestAbort` is idempotent for the same open operation and reports whether it wrote the marker; its expected-id fence fails rather than cancelling another operation. `recordUsage` mints its ledger row id at commit (§1.6) and returns it. `runId` is the operation's durable `operationId`; the public name remains for compatibility. `HarnessFault` and `HarnessClosed` reject promises; they are not tagged expected errors and not members of these unions. @@ -2470,8 +2650,8 @@ type SuspendedOperation = { kind: "run" | "compaction" | "navigation"; startedAt: number; prompt?: AgentMessage[]; - /** Payloads dereferenced from the drained items' surviving pending.entry - registers (§4.6). */ + /** Payloads dereferenced from the drained items' surviving + `pendingEntry(id)` values (§4.6). */ aborting?: { steer: AgentMessage[]; followUp: AgentMessage[] }; } & ( | { reason: "deferred"; deferred: DeferredHandle; missing?: never } @@ -2494,8 +2674,8 @@ interface AgentHarnessToolInvocation { turnId: string; /** Invocation-scoped durable values. `undefined` deletes. */ - getFact(key: string): Promise; - setFact(key: string, value: JsonValue | undefined): Promise; + getMemo(key: string): Promise; + setMemo(key: string, value: JsonValue | undefined): Promise; } interface AgentHarnessToolUpdateOptions { @@ -2557,13 +2737,13 @@ type EntryProjector = (entry: CustomEntry) => AgentMessage[] | undefined | Promise; ``` -`create()` copies the three seed fields into one immutable `LaneConfiguration`, storing the model as `{ provider, modelId }`. Before restore, it commits that seed as the first `lane.config` for a fresh or normalized-v3 `main`. Existing lanes use only their current config; the seed never overrides them. A configuration-less lane in a format-4 session is corrupt. +`create()` copies the three seed fields into one immutable `LaneConfiguration`, storing the model as `{ provider, modelId }`. Before restore, it commits that seed as the first `pi.lane.config` for a fresh or normalized-v3 `main`. Existing lanes use only their current config; the seed never overrides them. A configuration-less lane in a format-4 session is corrupt. -`createLane(name, at)` calls `Session.createLane(name, at, capturedSeed)` on the lane mutation line, regardless of later changes to existing lanes. It maps session validation failures to `LaneExists`, `InvalidLane`, or `UnknownTarget`, and emits `lane_created` only after the session transaction commits. Setters replace only their lane's register value. Reopen options can seed new lanes but cannot alter existing ones without a setter. Applications opt into deferred generation through `setStreamOptions({ deferred: ... })` or initial `streamOptions`; `before_request` may patch the same curated field per attempt. +`createLane(name, at)` calls `Session.createLane(name, at, capturedSeed)` on the lane mutation line, regardless of later changes to existing lanes. It maps session validation failures to `LaneExists`, `InvalidLane`, or `UnknownTarget`, and emits `lane_created` only after the session transaction commits. Setters replace only their lane's bound value. Reopen options can seed new lanes but cannot alter existing ones without a setter. Applications opt into deferred generation through `setStreamOptions({ deferred: ... })` or initial `streamOptions`; `before_request` may patch the same curated field per attempt. Initial, replacement, and hook-patched stream options are trusted typed internal values. Patch deletion semantics are applied before publication; extensions that return values outside the declared types are defective rather than runtime-validated by the harness. -`systemPrompt`, `toolContext`, `toProviderMessages`, and `entryProjectors` are deterministic/idempotent computation callbacks and may repeat after a crash; effectful interception belongs in hooks. `before_run` receives one preview evaluation of `systemPrompt`. A hook override is fixed in `Operation`; without one, the callback is evaluated again per provider request. `toolContext` is resolved once per live batch; each bound tool call receives its stable `AgentHarnessToolInvocation` and a required synchronous update callback even when no live listener exists. A `replay:"safe"` tool may implement Flue-style durable memoization over `getFact`/`setFact`; committed values survive replay until the call reaches `outcome_ready`. Tools must await fact writes. These methods are invocation-scoped capabilities, not raw register access. +`systemPrompt`, `toolContext`, `toProviderMessages`, and `entryProjectors` are deterministic/idempotent computation callbacks and may repeat after a crash; effectful interception belongs in hooks. `before_run` receives one preview evaluation of `systemPrompt`. A hook override is fixed in `Operation`; without one, the callback is evaluated again per provider request. `toolContext` is resolved once per live batch; each bound tool call receives its stable `AgentHarnessToolInvocation` and a required synchronous update callback even when no live listener exists. A `replay:"safe"` tool may implement Flue-style durable memoization over `getMemo`/`setMemo`; committed values survive replay until the call reaches `outcome_ready`. Tools must await memo writes. These methods are invocation-scoped capabilities, not raw Session access. ## 5.3 SessionTree @@ -2573,16 +2753,22 @@ interface SessionTree { getEntry(id: string): Promise; getStats(): Promise; - // Global facts. Latest wins; not branch-scoped. undefined deletes the - // register; JSON null is a legitimate custom value. Custom keys cannot - // collide with name or labels. + // Bound application values and lists. The address already contains its key. + getValue(address: Value): Promise | undefined>; + scanValues(prefix: Value): Promise[]>; + readList(address: ValueList, options?: ListReadOptions): + Promise[]>; + setValue(address: Value, next: NoInfer): Promise; + deleteValue(address: Value): Promise; + appendList(address: ValueList, element: NoInfer): Promise; + deleteList(address: ValueList): Promise; + + // Purpose-specific wrappers for session metadata. Latest wins and is not + // branch-scoped. undefined deletes the bound value. getName(): Promise; setName(name: string | undefined): Promise; getLabel(targetId: string): Promise; setLabel(targetId: string, label: string | undefined): Promise; - getCustomFact(key: string): Promise; - setCustomFact(key: string, value: JsonValue | undefined): Promise; - /** Session-wide, all branches, sequence order. */ findEntries(query?: EntryQuery): Promise; findEntry(query?: EntryQuery): Promise; @@ -2632,7 +2818,7 @@ interface LaneSnapshot { status: "running" | "suspended" | "aborting"; startedAt: number; suspended?: SuspendedOperation; - streamingMessage?: AssistantMessage; // message_start until entry commit + streamingMessage?: AssistantMessage; // live partial, or restored frame reduction (§3.7) runningTools: { toolCallId: string; toolName: string; args: unknown; /** Latest process-local update, or restored durable checkpoint. */ partialResult?: AgentToolResult }[]; @@ -2651,7 +2837,7 @@ interface SessionSnapshot { } ``` -`operation.status` derives from durable state plus process-local drive ownership: `suspended` whenever an open operation has no `ActiveOperation` (including accepted-but-undriven, hosted retry wait, safe yield, deferred, restored, or missing-identity state); `aborting` when `control.status === "cancel_requested"`; otherwise `running`. `suspended` details are present when the existing `SuspendedOperation` vocabulary can describe the reason; a transient accepted/yielded state needs no invented durable descriptor. The missing-identity marker stores the exact `SuspendedOperation`, survives until a successful drive/resume attempt or cancellation in this process, and is reconstructed as `reason:"crash"` after reopen. It changes snapshots but never durable recovery state. `queues` and `pendingWrites` derive from `inbox` and `pendingNextRun`, with content dereferenced from each id's `pending.entry` register; abort-drained items are exposed only through `AbortResult` and `SuspendedOperation.aborting`, never as still-queued. `streamingMessage` is process-local. A running tool's `partialResult` prefers its latest process-local update and falls back to the optional `pending.tool_output` checkpoint after reopen. Outcome-ready calls are settled, not running, and do not appear in `runningTools` while awaiting source-ordered placement. +`operation.status` derives from durable state plus process-local drive ownership: `suspended` whenever an open operation has no `ActiveOperation` (including accepted-but-undriven, hosted retry wait, safe yield, deferred, restored, or missing-identity state); `aborting` when `control.status === "cancel_requested"`; otherwise `running`. `suspended` details are present when the existing `SuspendedOperation` vocabulary can describe the reason; a transient accepted/yielded state needs no invented durable descriptor. The missing-identity marker stores the exact `SuspendedOperation`, survives until a successful drive/resume attempt or cancellation in this process, and is reconstructed as `reason:"crash"` after reopen. It changes snapshots but never durable recovery state. `queues` and `pendingWrites` derive from `inbox` and `pendingNextRun`, with content dereferenced from each id's `pendingEntry(id)` value; abort-drained items are exposed only through `AbortResult` and `SuspendedOperation.aborting`, never as still-queued. `streamingMessage` means the latest observed partial assistant message, not proof that a stream is attached. Precedence: the newest process-local partial while a live stream is owned; otherwise, for assistant/deferred `effect_pending`, the value reduced from committed `pendingAssistantFrames(operationId, responseEntryId)` pages; otherwise absent. A restored lane may therefore be `suspended` with a non-undefined `streamingMessage`. Snapshot hydration constructs the exact bound frame address from current typed state and stays within the assistant consumer's page budget; it never scans arbitrary lists. A running tool's `partialResult` prefers its latest process-local update and falls back to the optional `pendingToolOutput(operationId, invocationId)` checkpoint after reopen. Outcome-ready calls are settled, not running, and do not appear in `runningTools` while awaiting source-ordered placement. Rules: @@ -2659,14 +2845,14 @@ Rules: - `streamingMessage` is not part of `transcript`. `message_end` replaces it with the final post-hook value but does not clear it; the matching `entry_added` confirms the append, adds the entry to `transcript`, and clears the draft. - Direct messages and finalized tool results use the same immediate `message_start` → `message_end` lifecycle and enter `transcript` only on `entry_added`. They never populate `streamingMessage`. - An `aborting` snapshot reports only state that actually exists. It never synthesizes a streaming assistant message. -- Reconnect means a new `watch()`. Process loss discards assistant drafts and tool updates newer than the latest committed checkpoint. A restored effect-pending tool may expose that checkpoint as `partialResult`; after unsafe activation the same checkpoint is incorporated into the interrupted result entry. Every transcript entry remains complete. +- Reconnect means a new `watch()`. Process loss discards live output newer than the latest committed assistant frame or tool checkpoint. A restored effect-pending generation exposes its reduced frames as `streamingMessage`; a restored effect-pending tool may expose its checkpoint as `partialResult`; after unknown-outcome or unsafe activation the same values are incorporated into the synthetic interrupted response or result entry. Reconnect replays no historical `message_start`/`message_update` or `tool_update` events — the snapshot carries the durable partial, and recovery later emits its own recovery-tagged lifecycle for what it actually settles. Every transcript entry remains complete. - A lane watcher receives events whose `lane` matches, plus events with no lane. The harness-global `usage` event is the explicit exception: it carries its originating lane but reaches every watcher, because its totals are session-wide. ## 5.5 Events One flat stream. `events.on(type, listener)` matches across the harness; lane watchers filter as above. Events are **passive**: listeners cannot transform the in-flight procedure, payloads are isolated from procedure state and from other listeners, and a throw produces `handler_error` plus telemetry without affecting execution. Listeners run in registration order and delivery awaits each one. A listener may read harness state and perform external observational effects, but must not call state-mutating harness methods; interception belongs in hooks, and new operations belong to an external caller after listener delivery. This avoids reentrant mutation and event-delivery deadlocks. -Durable-fact events fire **after** commit — `entry_added` means queryable. Multi-write events wait for full success, then follow mutation order. `run_start`, standalone `compaction_start`, and `navigation_start` publish after their acceptance transaction and before `accept` resolves; no post-acceptance work has started yet. Explicit accept+drive and their convenience composition publish the same lifecycle once. Process-local lifecycle events need not be durable: `message_end` precedes the entry insert. +Durable tree and session-metadata events fire **after** commit — `entry_added` means queryable. Multi-write events wait for full success, then follow mutation order. `run_start`, standalone `compaction_start`, and `navigation_start` publish after their acceptance transaction and before `accept` resolves; no post-acceptance work has started yet. Explicit accept+drive and their convenience composition publish the same lifecycle once. Process-local lifecycle events need not be durable: `message_end` precedes the entry insert. ```ts type HarnessEventPayload = @@ -2713,15 +2899,14 @@ type HarnessEventPayload = | { type: "tool_end"; runId: string; turnId: string; toolCallId: string; toolName: string; result: AgentToolResult; isError: boolean; terminate: boolean } - // Tree, queues, facts + // Tree, queues, session metadata | { type: "entry_added"; entry: Entry } | { type: "write_pending"; runId: string; entryId: string; entryType: EntryType } | { type: "queue_update"; steer: QueuedItem[]; followUp: QueuedItem[]; nextRun: QueuedItem[] } - | ({ type: "fact_update" } & ( - | { fact: "name"; name: string | undefined } - | { fact: "label"; targetId: string; label: string | undefined } - | { fact: "custom"; key: string; value: JsonValue | undefined })) + | ({ type: "value_update" } & ( + | { value: "session_name"; name: string | undefined } + | { value: "entry_label"; targetId: string; label: string | undefined })) // Configuration | ({ type: "config_update" } & ( @@ -2749,7 +2934,7 @@ type HarnessEventPayload = | { type: "usage"; lane: string; row: UsageRow; totals: Usage }; type SpecialEventPayload = Extract; + { type: "fault" | "value_update" | "usage" | "config_update" | "handler_error" }>; type LaneEventPayload = Exclude; type ConfigEventPayload = Extract; type LaneConfigEventPayload = Extract & + | (Extract & { lane?: never; recovery?: never }) | (Extract & { recovery?: never }) | (GlobalConfigEventPayload & { lane?: never; recovery?: never }) @@ -2781,19 +2966,21 @@ interface Events { } ``` -`lane` is required on run/turn/retry/message/tool, entry/write/queue, lane model/thinking/active-tool configuration, structural, and lane-created events. It is absent on facts, faults, and harness-global configuration. `handler_error` follows the failed handler's scope. `usage` is the global-delivery exception: base `lane` is absent, while its payload carries the origin lane and the complete ledger row, including its durable `seq` (§1.6). `recovery: true` appears on process-local lifecycle re-emitted by resume activation (`drive` directly or through `resume()`), never on events for already-existing durable entries. Cross-lane events are process ordered, not globally sequence ordered. A totals consumer keeps the greatest usage `row.seq` it has applied, preventing a late older event from regressing totals. +`lane` is required on run/turn/retry/message/tool, entry/write/queue, lane model/thinking/active-tool configuration, structural, and lane-created events. It is absent on session-metadata value updates, faults, and harness-global configuration. `handler_error` follows the failed handler's scope. `usage` is the global-delivery exception: base `lane` is absent, while its payload carries the origin lane and the complete ledger row, including its durable `seq` (§1.6). `recovery: true` appears on process-local lifecycle re-emitted by resume activation (`drive` directly or through `resume()`), never on events for already-existing durable entries. Cross-lane events are process ordered, not globally sequence ordered. A totals consumer keeps the greatest usage `row.seq` it has applied, preventing a late older event from regressing totals. Ordering for a streamed assistant response, asserted exactly by the conformance tests: ``` -message_start → message_update* → after_response hook → message_end (final value, -optional reserved id) → atomic response + usage + classified-state commit -→ entry_added → usage +message_start → message_update* (each delivery awaited; its converted frame +append is enqueued without awaiting storage) → stop frame admission and await +the latest frame write → after_response hook → message_end (final value, +optional reserved id) → atomic response + usage + frame-list delete + +classified-state commit → entry_added → usage ``` -Only `entry_added` proves message durability. Classification is computed before the transaction and becomes durable with it; it is not a separate event. Abort and overflow classification may normalize the committed response after `message_end`, so `entry_added` is authoritative for those two cases. A synthetic settlement performs no provider effect, update, or response hook: `message_start → message_end → atomic commit → entry_added → usage`. +Frame commits emit ordinary storage telemetry only: there is no frame-persistence event, and no event claims a `message_update` was durable. A crash may follow a live update whose queued frame append never committed; reconnect then shows the older committed frame prefix. Only `entry_added` proves message durability. Classification is computed before the transaction and becomes durable with it; it is not a separate event. Abort and overflow classification may normalize the committed response after `message_end`, so `entry_added` is authoritative for those two cases. A synthetic settlement performs no provider effect, update, or response hook: `message_start → message_end → atomic commit → entry_added → usage`. -`tool_update` is always an immediate process-local event, including when its originating callback also requests `checkpoint:true`; no second public event announces checkpoint commit. The synchronous callback retains the latest asynchronous delivery promise, and tool-promise settlement awaits it before `after_tool`, preserving the existing listener order without backpressuring each update. Reconnect snapshots reveal only the latest committed checkpoint. For a real effect, `tool_end` follows finalization but precedes the `outcome_ready` staging commit, in completion order; it is observation, not proof of durability. Finalized result-message lifecycle and `entry_added` occur later in source order when ready values materialize. Synthetic blocked and unsafe-recovery outcomes emit no tool-effect lifecycle. A crash after `tool_end` but before staging may recover or safely replay the still-uncertain call. Historical events are not replayed, though a safely replayed execution emits its own recovery-tagged lifecycle. +`tool_update` is always an immediate process-local event, including when its originating callback also requests `checkpoint:true`; no second public event announces checkpoint commit, and no checkpoint write is dropped or coalesced (§3.8). The synchronous callback retains the latest asynchronous delivery promise, and tool-promise settlement awaits it — together with the latest checkpoint-write promise — before `after_tool`, preserving the existing listener order without backpressuring each update. Reconnect snapshots reveal only the latest committed checkpoint. For a real effect, `tool_end` follows finalization but precedes the `outcome_ready` staging commit, in completion order; it is observation, not proof of durability. Finalized result-message lifecycle and `entry_added` occur later in source order when ready values materialize. Synthetic blocked and unsafe-recovery outcomes emit no tool-effect lifecycle. A crash after `tool_end` but before staging may recover or safely replay the still-uncertain call. Historical events are not replayed, though a safely replayed execution emits its own recovery-tagged lifecycle. Nesting: @@ -2929,7 +3116,7 @@ One accepted-operation hook invocation reaches one breakpoint, calls `EffectGate | `transform_context` | per request, `AgentMessage` level, before `toProviderMessages` | `{ messages }` | `{ messages }` | | `before_request` | per request, provider-neutral options | `{ model, step, attempt, streamOptions }` | `{ streamOptions? }` | | `before_payload` | per request, provider-specific wire payload | `{ model, payload }` | `{ payload }` | -| `after_response` | per response, after streaming settles, before `message_end` and the commit | `{ status, headers, message }` | `{ message? }` (must keep role) | +| `after_response` | per response, after streaming settles and the latest frame write completes (§3.7), before `message_end` and the commit | `{ status, headers, message }` | `{ message? }` (must keep role) | | `before_tool` | after validation, before execution | `{ toolCallId, toolName, args }` | `{ args?, block?: { reason: string; terminate?: boolean } }` | | `after_tool` | after execution, before outcome staging; patch semantics | `{ toolCallId, toolName, args, content, details, isError, usage? }` | `{ content?, details?, isError?, usage?, terminate? }` | | `before_compaction` | in `deciding` | `{ reason, preparation, customInstructions? }` | `{ decline?, compaction? }` | @@ -2984,9 +3171,10 @@ interface HarnessAssistantStreamConfig { /** Adapter for the before_payload hook; undefined keeps the payload. */ beforePayload?: (payload: unknown, model: Model) => unknown | undefined | Promise; - /** Adapter for after_response; runs after settlement and before observer.end. */ - afterResponse?: (message: SettledAssistantMessage, - metadata: AssistantResponseMetadata) => + /** Always-installed harness adapter. It drains accepted frame writes, then + runs the optional after_response hook pipeline, before observer.end. */ + afterResponse: (message: SettledAssistantMessage, + metadata: AssistantResponseMetadata) => Promise; /** Models-backed adapter. It resolves the captured identity, synchronously checks EffectGate, then immediately invokes Models (§4.2). */ @@ -3019,7 +3207,7 @@ transformContext → return the final settled message ``` -It never mutates `messages`. If the stream terminates without a start event, it emits `observer.start` for the final message before `observer.end`, matching the harness lifecycle contract. If abort interrupts the parked `afterResponse` adapter, the block awaits the carried abort-mutation promise, skips that hook, emits `observer.end` with the raw settled message, and returns it so the caller can commit it under the now-current cancellation control. `beforePayload` maps to pi-ai's payload callback. Response metadata capture maps to pi-ai's `onResponse`; it is distinct from `afterResponse`, because `onResponse` runs before the response body is consumed while the harness hook transforms the settled assistant message afterward. The harness exposes neither callback through `AgentHarnessStreamOptions`. +It never mutates `messages`. The harness-supplied observer converts each streamed event with `assistantMessageEventToFrame` and synchronously enqueues its invocation-fenced frame append without awaiting storage (§3.7); `afterResponse` is always installed, even with no hook listeners, because it first stops frame admission and awaits the latest frame-write promise before invoking the optional `after_response` pipeline. If the stream terminates without a start event, it emits `observer.start` for the final message before `observer.end`, matching the harness lifecycle contract; this synthesized lifecycle start occurs after stream consumption and is not converted into or appended as a durable frame. If abort interrupts the parked `afterResponse` adapter, the block awaits the carried abort-mutation promise, skips that hook, emits `observer.end` with the raw settled message, and returns it so the caller can commit it under the now-current cancellation control. `beforePayload` maps to pi-ai's payload callback. Response metadata capture maps to pi-ai's `onResponse`; it is distinct from `afterResponse`, because `onResponse` runs before the response body is consumed while the harness hook transforms the settled assistant message afterward. The harness exposes neither callback through `AgentHarnessStreamOptions`. The request function, not this block, owns registry dispatch, auth, and operation admission: @@ -3104,20 +3292,20 @@ planned call → prepareToolCall lookup · prepareArguments · initial validation → before_tool hook effect → applyBeforeToolDecision block or validate replacement arguments -→ commit op.tool_args + effect_pending intent +→ commit pi.op.tool_args + effect_pending intent → executeToolCall effect + live updates + checkpoint requests -→ stop updates; expire fact capability; close checkpoint admission; drop waiting checkpoint -→ await latest tool_update delivery +→ stop updates; expire memo capability; close checkpoint admission +→ await latest tool_update delivery and latest checkpoint write → after_tool hook effect → finalizeToolCall → emit and await tool_end -→ commit pending.entry + outcome_ready + invocation cleanup +→ commit pi.pending.entry + outcome_ready + invocation cleanup → materialize source-ready outcomes as entries + usage ``` -Unknown tools, `prepareArguments` failures, invalid initial/replacement arguments, and blocked calls produce `ImmediateToolOutcome` and skip intent/execution before staging a synthetic `outcome_ready` result. `AgentTool.prepareArguments` remains deterministic/idempotent computation and may repeat before intent; effectful policy belongs in `before_tool`. At `tool.execute` operation admission, `executeToolCall` calls `EffectGate.assertOpen()` immediately before `tool.execute` and passes `EffectGate.signal`; it converts expected tool throws to an error result, stops accepting updates when the tool promise settles, and emits raw tool-effect telemetry. Every update emits live observation; `checkpoint:true` additionally enters the invocation-fenced drop-coalescing path for `pending.tool_output`. The procedure retains only the latest event-delivery promise because the event bus serializes delivery, closes checkpoint admission at tool-promise settlement, and drops the waiting checkpoint. `finalizeToolCall` applies the documented field-by-field patch semantics before `tool_end` and outcome staging. +Unknown tools, `prepareArguments` failures, invalid initial/replacement arguments, and blocked calls produce `ImmediateToolOutcome` and skip intent/execution before staging a synthetic `outcome_ready` result. `AgentTool.prepareArguments` remains deterministic/idempotent computation and may repeat before intent; effectful policy belongs in `before_tool`. At `tool.execute` operation admission, `executeToolCall` calls `EffectGate.assertOpen()` immediately before `tool.execute` and passes `EffectGate.signal`; it converts expected tool throws to an error result, stops accepting updates when the tool promise settles, and emits raw tool-effect telemetry. Every update emits live observation; `checkpoint:true` additionally enqueues one invocation-fenced `pi.pending.tool_output` replacement and replaces only the latest checkpoint-write promise reference (§3.8). The procedure retains only the latest event-delivery promise because the event bus serializes delivery, and only the latest checkpoint-write promise because the lane line is FIFO; it closes checkpoint admission at tool-promise settlement and awaits both before `after_tool`. `finalizeToolCall` applies the documented field-by-field patch semantics before `tool_end` and outcome staging. -Before starting any call in a live batch, the tool-batch procedure resolves `toolContext` once and binds the complete captured active-tool set into ordinary `AgentTool` adapters retained in a procedure-local snapshot. Each adapter also binds `AgentHarnessToolInvocation { invocationId: resultEntryId, operationId, turnId, getFact, setFact }` and the harness-specific update callback. A missing captured active tool therefore suspends before any call starts; provider calls to names outside that active snapshot still become ordinary unknown-tool results. Every call in the batch observes the same application context and its own stable invocation identity. Safe replay creates a new code/context snapshot but passes the same invocation id/facts after deleting the stale progress checkpoint. `AgentTool.replay` defaults to `"never"` when omitted. +Before starting any call in a live batch, the tool-batch procedure resolves `toolContext` once and binds the complete captured active-tool set into ordinary `AgentTool` adapters retained in a procedure-local snapshot. Each adapter also binds `AgentHarnessToolInvocation { invocationId: resultEntryId, operationId, turnId, getMemo, setMemo }` and the harness-specific update callback. A missing captured active tool therefore suspends before any call starts; provider calls to names outside that active snapshot still become ordinary unknown-tool results. Every call in the batch observes the same application context and its own stable invocation identity. Safe replay creates a new code/context snapshot but passes the same invocation id and memos after deleting the stale progress checkpoint. `AgentTool.replay` defaults to `"never"` when omitted. There is deliberately no harness `executeToolBatch`. In parallel mode the direct procedure makes one source-ordered start pass. Each position either starts a real promise or retains an immediate outcome until it can be staged. Effects/finalization then settle independently: each complete real or synthetic result commits `outcome_ready` immediately in completion order. A separate lane-line job materializes the contiguous ready prefix in source order. Durably, completed calls form a prefix while the remaining suffix may mix `planned`, `effect_pending`, and `outcome_ready`; for example `[effect_pending, outcome_ready, effect_pending]` is valid after the completed prefix. A crash discards only unstaged process-local outcomes. Recovery safely replays or interrupts orphaned effects, materializes already-ready outcomes without resolving tool code, and reruns ordinary clearance for planned positions. The same procedure owns cancellation and durable batch completion. Genuine-`length` calls bypass effects but stage their specified synthetic outcomes (§3.7). @@ -3142,7 +3330,7 @@ pi.ai.request Operation, step, tool, hook, event, and write parents follow the actual async procedure nesting. Sleep spans permit run, compaction, navigation, turn, and checkpoint parents. `stepId`/`taskId` correlate retries and recovery. Every provider request/fetch/cancel uses `pi.ai.request`; each real or safely replayed phase-two tool effect uses one tool span. -Every storage transaction uses one `pi.session.write`. Its start attributes include `pi.session.item_count` and `pi.session.item_kinds` (`entry`, `usage`, `register`). A calling procedure may supply its lane/operation ids; storage never infers them from payloads. End attributes include first and last committed sequence. Tool checkpoint and invocation-fact commits are ordinary register writes under this span and emit no additional tool-effect span; snapshot content never enters telemetry. No span is emitted when a mutation returns without committing. Synthetic settlements and blocked/invalid tools emit no provider/tool-effect span. +Every storage transaction uses one `pi.session.write`. Its start attributes include `pi.session.item_count` and `pi.session.item_kinds` (`entry`, `usage`, `value`, `list`); list appends/deletes are never reported as value replacements. A calling procedure may supply its lane/operation ids; storage never infers them from payloads. End attributes include first and last committed sequence. Tool-checkpoint, invocation-memo, and assistant-frame commits are ordinary value/list writes under this span and emit no additional tool- or provider-effect span; address namespaces may be attributes, but snapshot and frame content never enters telemetry. No span is emitted when a mutation returns without committing. Synthetic settlements and blocked/invalid tools emit no provider/tool-effect span. Telemetry attributes may contain declared ids, names, counts, durations, statuses, and usage. They must never contain prompts, completions, tool arguments/results, file contents, provider payloads, headers, handles, or credentials. Events and hooks may contain such content. The existing generated schema document and adapter/runtime conformance tests remain authoritative; implementation slices extend instrumentation only through those schemas. @@ -3150,11 +3338,11 @@ Telemetry attributes may contain declared ids, names, counts, durations, statuse **This part is informative.** Nothing in it binds the shipping backends: Memory, JSONL, and SQLite never partition and never delete entries or usage rows (§1.2), and no core rule references this part for its correctness. It exists to show that the identity choices in §1.2 are sufficient for the one backend that would eventually retire old data — a possible Postgres deployment with TTL retention. It is a bridge we cross when we get there; this sketch is the current best guess, not a contract. -- **The id is the partition key.** UUIDv7 sorts bytewise in time order, so the bulk tables — entries, usage ledger — use `PARTITION BY RANGE (id)` on the uuid id column, with period-boundary UUIDs (zeroed tails) as bounds. No partition column exists anywhere; §1.2's time prefix is the whole mechanism. Registers, `branch_meta`, stats, leases, and sessions stay in a hot unpartitioned catalog. `branch_entries` partitions by `entry_id` with the same bounds, so dropping a period cleans the branch index for free; `branch_meta` stays hot, and base pointers dangling into a dropped period are trimmed lazily on first access. -- **Pre-pass repair.** Before a period P is dropped, an online repairer makes live state stop referencing it: reparent edges crossing into P onto the nearest retained ancestor, found by an indexed uuid-range query; null any dormant `lane.leaf` decoding into P via a register-seq CAS; force-expire open operations still referencing P register-only — the terminal transaction of §3.13 writing `lane.lastResult`, no synthetic entries, with any live task stopping through external finalization (§4.9); delete `fact.label` registers whose keys decode into P with one uuid-range delete. -- **The commit barrier.** Repair races ordinary commits, so the final step is atomic against all of them: `BEGIN; LOCK entries, registers IN ACCESS EXCLUSIVE MODE; ; ALTER TABLE … DETACH PARTITION p; COMMIT;` — plain `DETACH`, not `CONCURRENTLY`, precisely because it is transactional under the lock; the `DROP TABLE` happens later, unhurried. The barrier makes repair-plus-detach one linearization point: every commit sees either the fully attached period or a fully repaired store without it. +- **The id is the partition key.** UUIDv7 sorts bytewise in time order, so the bulk tables — entries, usage ledger — use `PARTITION BY RANGE (id)` on the uuid id column, with period-boundary UUIDs (zeroed tails) as bounds. No partition column exists anywhere; §1.2's time prefix is the whole mechanism. Scalar values, `branch_meta`, stats, leases, and sessions stay in a hot unpartitioned catalog. `branch_entries` partitions by `entry_id` with the same bounds, so dropping a period cleans the branch index for free; `branch_meta` stays hot, and base pointers dangling into a dropped period are trimmed lazily on first access. +- **Pre-pass repair.** Before a period P is dropped, an online repairer makes live state stop referencing it: reparent edges crossing into P onto the nearest retained ancestor, found by an indexed uuid-range query; null any dormant `pi.lane.leaf` decoding into P via a value-seq CAS; force-expire open operations still referencing P through values only — the terminal transaction of §3.13 writing `pi.lane.lastResult`, no synthetic entries, with any live task stopping through external finalization (§4.9); delete `pi.entry.label` values whose keys decode into P with one uuid-range delete. +- **The commit barrier.** Repair races ordinary commits, so the final step is atomic against all of them: `BEGIN; LOCK entries, scalar_values IN ACCESS EXCLUSIVE MODE; ; ALTER TABLE … DETACH PARTITION p; COMMIT;` — plain `DETACH`, not `CONCURRENTLY`, precisely because it is transactional under the lock; the `DROP TABLE` happens later, unhurried. The barrier makes repair-plus-detach one linearization point: every commit sees either the fully attached period or a fully repaired store without it. - **The default partition.** A `DEFAULT` partition absorbs stray inserts whose ids predate every attached partition — an ancient `pendingNextRun` item consumed years after its mint still places under its reserved id and simply lands there. Nothing errors and nothing is lost; the default partition stays small and is never dropped. -- **Register access under an external repairer.** A backend that admits an external repairer must perform register reads and CAS checks inside the commit transaction itself, so a repairer holding the barrier cannot interleave between a harness's read and its dependent write. The shipping backends need no such rule: single-writer sessions have no external repairer. +- **Value access under an external repairer.** A backend that admits an external repairer must perform value reads and CAS checks inside the commit transaction itself, so a repairer holding the barrier cannot interleave between a harness's read and its dependent write. The shipping backends need no such rule: single-writer sessions have no external repairer. Everything else a real deployment would need — retention policy, per-session versus per-deployment periods, operational partition-count limits — is deliberately unspecified until the backend is real. @@ -3172,23 +3360,24 @@ Migration cost is proportional to what must be converted, and this design keeps what exists at upgrade time migration burden ──────────────────────────── ──────────────── entries, usage rows (years) cannot rewrite — must stay read-compatible -lane/fact registers (a few per lane) trivial: a for-loop at open -op.* registers only for OPEN operations — usually zero -pending.entry registers queued items plus staged tool outcomes -pending.tool_output registers optional open-call checkpoints only +lane/semantic values (a few per lane) trivial: a for-loop at open +pi.op.* values only for OPEN operations — usually zero +pi.pending.entry values queued items plus staged tool outcomes +pi.pending.tool_output values optional open-call checkpoints only +pi.pending.assistant_frame lists open-response frames only — usually zero ``` -Because no history is retained, the entire mutable surface is a few dozen current registers — which is what makes migrate-on-open tractable at all. And the fenced single-writer lease (§1.7) means the opening process owns the session exclusively — migration has no concurrency story to solve. +Because no history is retained, the entire mutable surface is a few dozen current values/lists — which is what makes migrate-on-open tractable at all. And the fenced single-writer lease (§1.7) means the opening process owns the session exclusively — migration has no concurrency story to solve. ## 7.3 The mechanism: storage version plus migrate-on-open -One session-level `storageVersion` lives in the catalog or header (§1.7, §2.8). A version number is preferable to versioned namespace suffixes (`lane.state.v2`): one number to check, chained `v1→v2→v3` migrations, no probing of historical namespace names, and register keys stay stable for point lookups. +One session-level `storageVersion` lives in the catalog or header (§1.7, §2.8). A version number is preferable to versioned namespace suffixes (`pi.lane.state.v2`): one number to check, chained `v1→v2→v3` migrations, no probing of historical namespace names, and address components stay stable for point lookups. ```text open session: version == current → proceed version < current → run migrations in order, each one transaction: - convert lane/fact/pending register values + convert lane/semantic/pending values handle open operations (§7.4) bump the version version > current → refuse to open (older binary, newer session) @@ -3202,68 +3391,85 @@ Legacy coding-agent format 3 predates `storageVersion` entirely; it normalizes t ## 7.4 Migrations are total -Register conversion is a field mapping; a state-machine shape change is more. If the next version removes `failure_drain`, or restructures the tool-batch lifecycle, an old `op.state` sitting mid-phase has no field-by-field equivalent in the new machine. The rule: **migrations are total.** A vN→vN+1 migration translates every register value — lane/fact registers, `pending.entry`, optional `pending.tool_output`, invocation facts, and open operations' `op.meta`/`op.state` included. In particular, the migration adding `outcome_ready` must distinguish staged finalized tool results from still-uncertain effects. The author of a state-machine change writes the mapping for every reachable old state in the same change. A state with no natural successor maps to an explicit safe choice; there is no force-settle path or silent partial escape hatch. +Value conversion is a field mapping; a state-machine shape change is more. If the next version removes `failure_drain`, or restructures the tool-batch lifecycle, an old `pi.op.state` sitting mid-phase has no field-by-field equivalent in the new machine. The rule: **migrations are total.** A vN→vN+1 migration translates every stored value/list — lane/semantic values, `pi.pending.entry`, optional `pi.pending.tool_output`, invocation memos, and open operations' `pi.op.meta`/`pi.op.state` included. In particular, the migration adding `outcome_ready` must distinguish staged finalized tool results from still-uncertain effects. The author of a state-machine change writes the mapping for every reachable old state in the same change. A state with no natural successor maps to an explicit safe choice; there is no force-settle path or silent partial escape hatch. -This is tractable for the same reason migrate-on-open is tractable at all (§7.2): the entire mutable surface is a few dozen current registers, and migration runs at open under the writer lease, so it sees **quiescent** registers — no operation task is running, no effect is in flight, and every `op.state` is exactly the total state some transaction committed. A migration is a pure function over a small, fully enumerable, fully typed set of values. +This is tractable for the same reason migrate-on-open is tractable at all (§7.2): the entire mutable surface is a few dozen current values/lists, and migration runs at open under the writer lease, so it sees **quiescent** state — no operation task is running, no effect is in flight, and every `pi.op.state` is exactly the total state some transaction committed. A migration is a pure function over a small, fully enumerable, fully typed set of values. + +Address and list rules (§1.3, §1.4) extend the same discipline: + +- A bound address's namespace, key grammar, and kind are static for one storage version. Changing any component is an explicit address migration; changing value↔list is an explicit migration; storage never infers or coerces kind. Changing the TypeScript value shape requires a total value migration when old stored values are incompatible. Adding a new address with no stored value rewrites nothing. +- A list migration pages current elements in sequence order and either maps values while preserving each element's `seq` or deletes the whole key; it never loads an unbounded logical list at once. +- A migration changing `AssistantMessageFrame` shape must map every surviving element or explicitly delete the whole list, leaving `effect_pending` recovery with no partial. It must never infer completion from legacy frames. ## 7.5 The three strata, restated as policy ```text -entries + usage the stability budget goes HERE. Payloads are provider-shaped - messages plus three simple structural types; changes must be - read-compatible forever, because years of entries cannot - be rewritten at open time — the precise rewrite (§2.9) - exists, but it is administrative, not an open-time step. Custom - entry payloads are the application's contract. +entries + usage the stability budget goes HERE. Payloads are provider-shaped + messages plus three simple structural types; changes must be + read-compatible forever, because years of entries cannot + be rewritten at open time — the precise rewrite (§2.9) + exists, but it is administrative, not an open-time step. Custom + entry payloads are the application's contract. -lane / fact migrate on open, mechanically. A few registers per lane, -registers cheap forever. +lane/session values migrate on open, mechanically. A few values per lane, + cheap forever. -op.* / pending.* ephemeral by construction and few in number. Every - state-machine change ships the total register mapping for - its own states (§7.4). This is where the machine is allowed - to churn between versions, because the mapping cost is - bounded by open operations — usually zero. +pi.op.* / pi.pending.* ephemeral by construction and few in number. Every + state-machine change ships the total value/list mapping for + its own states (§7.4). This is where the machine is allowed + to churn between versions, because the mapping cost is + bounded by open operations — usually zero. ``` The design conclusion: the volatile part of the system — orchestration — was made ephemeral, and the durable part — the conversation — was made structurally boring. Schema evolution is exactly as hard as the boring part, which is the best available outcome. -# Part 8 — Build order +# Part 8 — Work packages -One shared slice lands the complete type surface; everything after it splits into two independent tracks. **Track S** (storage, search, dev TUI) parallelizes across owners — its slices depend only on slices 1–2 and never on each other. **Track R** (runtime) is sequential, runs entirely against the Memory backend, and never waits on Track S. The tracks cannot block each other. +This part is a rolling plan for the repository as it exists now, not a replay of its historical build order. `harness.md` remains the normative behavior contract; a work-package handoff defines one executable implementation boundary. -Each slice implements its named behavior end to end and adds focused tests for its normal path, every state it introduces, every owned crash boundary, and both orders of owned races. State-specific restore checks, identity preflight, hooks, events, and breakpoint descriptors land with the slice that first creates or executes that state; earlier slices do not implement generic future-state machinery. The complete public types may precede runtime behavior, but unimplemented methods remain direct stubs rather than speculative shared execution paths. Passing those tests and `npm run check` is each slice's acceptance criterion. If implementation exposes a design contradiction, missing transition, or materially simpler design, stop and send it for review — do not silently improvise a new durable contract inside a slice. +## Work-package workflow + +1. Keep a future package's detailed row here until that package is actionable. +2. Reassess the row using evidence from completed packages; boundaries may change. +3. Move exact files, tests, ordering, exclusions, and stop conditions into one handoff. +4. Move any newly discovered normative behavior into Parts 0–7 or Part 9. +5. Only then replace the detailed row with a concise overview link. + +Do not scaffold handoffs for distant work. This intentionally leaves Part 8 hybrid: linked rows for concrete packages and detailed rows for future candidates. `runtime2.md` is implementation history/status, not a competing package plan. + +Every package implements its named concern end to end and tests its normal path, introduced states, owned crash boundaries, and both orders of owned races. Consumption-time dereference checks, identity preflight, hooks, events, and breakpoint descriptors land with the package that first needs them; earlier packages do not build generic future machinery. If implementation exposes a contradiction or materially simpler boundary, stop for review rather than silently extending the package. + +## Concrete packages + +| ID | Outcome | Dependency | Handoff | +|---|---|---|---| +| WP00 | Reconcile acceptance/hooks, harvest runtime1 scenarios, switch the public factory, and delete runtime1 without adding behavior. | Approved redesign and durability handoffs are present. | [Runtime1 removal](work-packages/00-runtime1-removal.md) | +| WP01 | Replace retained register/custom-state APIs with bound values/lists across Session, Memory, JSONL, SQLite, instrumentation, conformance, and public application access. | WP00 | [Bound values and lists](work-packages/01-bound-values-lists.md) | + +The old Types, Session/Memory, JSONL, and SQLite rows described work already present in the repository. Their remaining cross-backend delta is WP01; they are not separate future packages. + +## Future package candidates + +These rows preserve requirements, not frozen package boundaries. Create their handoffs only when they become actionable. | # | Slice | Implement | Required focused tests | |---|---|---|---| -| 1 | **Types** (complete through the original runtime surface) | The shared storage/session/state surface: `Entry`/`Register`/`UsageRow` and `RegisterValues` including the full Part 3 state tree, `Write`/`Transaction`/`Storage`/`SessionReader`/`SessionMutator`/`Session`/`SessionTree`/`SessionRepo`, scans, the id-generator, the search interfaces from §2.8, `storageVersion`, and the original Part 5 surface. R1 extends the runtime-facing types with `OperationRequest`, `accept`/`drive`/`requestAbort`/`inspectExecution`, `OperationMismatch`, and `AgentHarnessToolInvocation`; no storage migration is involved. | Type-level only; no behavior. | -| 2 | **Session layer, Memory, conformance** | Entry materialization with inline payloads, atomic configured lane creation, lane/config/state registers, facts, branch/global queries, context projection, `SessionTree`/views, UUIDv7 generator with follower minting, stats projection, the Memory backend with repository lifecycle/forks and the `storageVersion` gate at open, the backend conformance suite, and the instrumented-storage decorator (Part 9). Internal values pass directly between Session and Storage without codecs, cloning, or runtime shape validation. | Rollback, sequence order, duplicate ids, register set/delete/recreate, delete-of-absent-key no-op, atomic lane creation and duplicate/name/anchor rejection, fact deletion vs JSON `null`, stats-equals-ledger, follower minting, placement, divergence, filters/cursors/stops, custom entries with and without data, context projection, fork before first attachment, configured fork snapshots/facts/zero ledger, close. | -| S1 | **JSONL** | Format 4: single-item/array transaction lines, register set/delete replay, header `storageVersion`, torn-tail handling, snapshot compaction (GC keep-predicate), the file-based repository, format-3 read normalization and first-write temp/rename conversion with id re-minting (Appendix B). Replace the unfinished current v4 without migration. | Backend conformance, corrupt interior/final lines, whole-array tear, compaction logical-equivalence, every format-3 rule including id re-minting and reference remapping, resolved/unresolved parent paths, aggregate imported usage adjustment. | -| S2 | **SQLite** | One database file per session: entries/registers/usage-ledger tables, one-row session/lease rows, transactions, `storageVersion`, the file-based repository, segmented branch cache, `VACUUM INTO`-based rewrite/fork, and explicit repair. No values table, no `slot_history`, no `getLog`, no search projection, no migration. | Shared conformance, `BEGIN IMMEDIATE`, fencing, query plans, segment-chain soundness, register upsert/delete, forks/stats/repair. | | S3 | **Search** | The standalone `SessionSearchService` plus repo catch-up utilities (§2.8): core entry search as `SessionSearch`, session-level ranked results, optional `searchEntries?: SessionSearch`, `remove()`, the `SessionSearchSyncTarget` cursor/index-batch contract, sync enumeration and catch-up outside the service contract, debounced notify as a utility, `(sessionId, storeGeneration)` cursor keys, and the reference SQLite FTS5 implementation working over any backend's repository through the sync utility. | Cursor catch-up from empty against existing sessions, idempotent re-index after crash mid-batch, notify-utility/sweep equivalence, sessions-vs-entries queries and ranking, removal and reconciliation, shared-index multi-process discipline. | -| S4 | **Dev TUI and Client** | A minimal `AgentClient` over one lane — `LaneSnapshot` plus `watch()` events, `prompt`/`steer`/`followUp`/`abort`/`resume`/`cancelQueued`, `lane.lastResult` read — and a throwaway alt-screen TUI on `packages/tui`: transcript from snapshot and events, input box, status/queue display, abort key. Built first against a scripted fake client on the slice-1 types; binds to the real harness as Track R lands. Not final. | Compiles; fake-client smoke test. No durability obligations. | -| R1 | **Runtime shell and execution primitives** | Extend the public types with `OperationRequest`, admission/drive/cancellation/inspection results, `OperationMismatch`, `AgentHarnessToolInvocation`, and harness-specific tool-update checkpoint options. Implement lane/settings mutation lines; one drive-pass `ActiveOperation` slot per lane; expected-id fencing, same-operation drive joining, coherent `inspectExecution`, safe deadline-yield plumbing, `ActionInfo` breakpoint barrier, `EffectGate` and its internal abort control flow, base current-state validation, hook/event primitives, restore inventory, registry lookup plumbing, and fault/close plumbing. Also implement the independently testable behavior-only blocks under `packages/agent/src/harness/execution/` from §5.7; `agent-loop.ts` remains unchanged. | Type surface clean; coherent inspection; expected-id mismatch/latest-result lookup; one same-id pass installed/joined; stale-id drive isolation; safe yield; breakpoint and effect-gate coverage; task lifecycle; hook aggregation; ordered passive event buffering; restore without history reads; base lane validation; direct assistant/tool block tests; old `AgentToolUpdateCallback` remains unchanged while the harness callback carries `checkpoint:true`. | -| R2 | **Minimal no-tool run** | Compose R1's assistant block with generic prompt `accept`, optional caller-supplied operation id, prompt expansion, `before_run`, the acceptance transaction without task installation, pending-capture placement, fresh `drive`, and `prompt` as `accept`+`drive`; add run/checkpoint/generation semantic restore checks, captured request options/thinking and model preflight, payload/response hooks, one generation intent/effect/settlement, usage, terminal cleanup plus `lane.lastResult`, results, basic events/telemetry, and stable breakpoint descriptors. | Acceptance returns before any drive work; crash between accept and drive resumes the initial state; prompt convenience and explicit accept+drive produce byte-identical writes/events; supplied operation id reaches `op.meta`/events/result; successful run with final assistant fields; caller rejection and adapter-reported failures; exact ordering; terminal cleanup; same-id drive join during the accept/drive gap; close at every boundary; invalid run/checkpoint/generation references. | -| R3 | **Generation recovery and retry** | Activation-only unknown-effect recovery, synthetic cap settlement, ordinary stop/error/deferred classification, provider-compliant `aborted`, failure-drain foundation, and both retry-wait policies: explicit drive returns `waiting/notBefore`, while convenience drive may own the abort-aware timer. Define stable breakpoint descriptors for retry timers and recovery commits. Overflow classification remains explicitly unimplemented until R9. | Every generation state before/after reopen; accept/close/reopen/drive; caps/backoff; no timer under `waitForRetry:false`; due and future `notBefore`; deadline before retry; convenience/explicit-drive durable equivalence; timer abort-first/start-first; stable descriptors; stop/error/aborted/deferred classification; missing identities. | -| R4 | **Tools** | Compose R1's tool phases with bound context, invocation-scoped facts, `op.tool_args`, scalar `pending.tool_output` checkpoints, `outcome_ready` staging in `pending.entry`, completion-order outcome durability, source-ordered materialization, activation recovery/replay, sequential/parallel promises, blocked/genuine-length outcomes, hooks/events/usage, and stable descriptors. The built-in bash tool emits live updates at 100 ms and requests distinct checkpoints at most every two seconds. | Stable invocation id/facts across safe replay; every planned/effect-pending/outcome-ready/completed state; parallel out-of-order staging without replay; synchronous update callback with delivery awaited before `after_tool`; `tool_end` before staging; checkpoint one-active/one-waiting coalescing with waiting-drop at tool settlement and late-write fencing; JSONL checkpoint growth/compaction; safe replay deletes old checkpoint; unsafe recovery with/without checkpoint; source-ordered materialization; termination and abort-ready states. | -| R5 | **Inbox, configuration, and writes** | `nextRun`/steer/follow-up via `pending.entry` registers, pending-payload and queue semantic restore checks, `cancelQueued` triage (`not_found`), durable drain markers, checkpoint consumption with register deletion, immediate total config setters, deferred tree writes, adjustments, and stable breakpoint descriptors for the checkpoint boundaries introduced here. | Capture/cancel/consume races, repeated cancellation answering `not_found`, invalid queue payload kinds and missing pending registers, one-at-a-time crash after one drain, register/entry exclusivity at every boundary, custom-write continuation, config-step race, writes surviving reopen. | -| R6 | **Abort, close, and failure drain** | Add expected-id `requestAbort` as commit/signalling only. Compose `abort()` from inspection, marker, and same-id drive. Implement orthogonal control, per-phase reconciliation including planned/effect-pending/outcome-ready tools, best-effort deferred cancellation, waiters/run-when-idle, controlled-crash close, complete terminal deletion of facts/checkpoints/staged outcomes, external-finalization stop, and stable descriptors. | Cancellation with/without live pass; idempotency/fencing; abort at every tool state; restored interruption incorporates latest checkpoint without replay; outcome-ready values survive and materialize; checkpoint/fact cleanup; parked barriers; deferred cancellation; close races; external finalization; failure drain. | -| R7 | **Deferred provider redemption** | One poll per drive pass carrying `pollDeferred:true`; `resume()` composes inspection and that drive option. Add deferred-state semantic restore checks and provider identity preflight. Preserve copied configuration/options, per-poll hooks, exact source lineage/equality, fresh intent after unknown poll, mismatch-to-error, ready tools, advancement of R6 cancellation, and stable poll descriptors. | `pollDeferred:false` returns waiting without work; one permit performs at most one poll; invalid/missing/mismatched durable handles; repeated pending, ready/error/aborted/mismatch, crash positions, convenience/primitive equivalence, no cap/backoff/loop, newest-handle cancellation. | -| R8 | **Manual compaction** | Add compaction `OperationRequest` acceptance without task installation and compose `compact()` from accept+drive. Implement reserved-lane preparation, `op.preparation/{opId}:{taskId}`, total structural state and its semantic restore checks, structural model preflight, hook/generated sources, nested request intents/usage, retained tail, retry/recovery/abort, and stable structural descriptors. | Explicit accept+drive versus convenience equivalence; empty/reservation race; accept/close/reopen/drive; invalid task/preparation/source relationships; hook decline/result; crash after request one of split-turn generation; every state/crash; no public summary-stream messages. | +| S4 | **Dev TUI and Client** | A minimal `AgentClient` over one lane — `LaneSnapshot` plus `watch()` events, `prompt`/`steer`/`followUp`/`abort`/`resume`/`cancelQueued`, `pi.lane.lastResult` read — and a throwaway alt-screen TUI on `packages/tui`: transcript from snapshot and events, input box, status/queue display, abort key. Built first against a scripted fake client on the shared types; binds to the real harness as runtime packages land. Not final. | Compiles; fake-client smoke test. No durability obligations. | +| R1 | **Runtime shell and execution primitives** | Extend the public types with `OperationRequest`, admission/drive/cancellation/inspection results, `OperationMismatch`, `AgentHarnessToolInvocation`, and harness-specific tool-update checkpoint options. Implement lane/settings mutation lines; one drive-pass `ActiveOperation` slot per lane; expected-id fencing, same-operation drive joining, coherent `inspectExecution`, safe deadline-yield plumbing, `ActionInfo` breakpoint barrier, `EffectGate` and its internal abort control flow, projection-existence restore checks, hook/event primitives, restore inventory, registry lookup plumbing, and fault/close plumbing. Also implement the independently testable behavior-only blocks under `packages/agent/src/harness/execution/` from §5.7; `agent-loop.ts` remains unchanged. | Type surface clean; coherent inspection; expected-id mismatch/latest-result lookup; one same-id pass installed/joined; stale-id drive isolation; safe yield; breakpoint and effect-gate coverage; task lifecycle; hook aggregation; ordered passive event buffering; restore without history reads, list reads, or reference audits; direct assistant/tool block tests; old `AgentToolUpdateCallback` remains unchanged while the harness callback carries `checkpoint:true`. | +| R2 | **Minimal no-tool run** | Compose R1's assistant block with generic prompt `accept`, optional caller-supplied operation id, prompt expansion, `before_run`, the acceptance transaction without task installation, pending-capture placement, fresh `drive`, and `prompt` as `accept`+`drive`; add projection-only restore with consumption-time dereference, captured request options/thinking and model preflight, payload/response hooks, one generation intent/effect/settlement with per-event frame appends (synchronous enqueue, latest-promise settlement fencing) and atomic frame-list deletion, usage, terminal cleanup plus `pi.lane.lastResult`, results, basic events/telemetry, and stable breakpoint descriptors. | Acceptance returns before any drive work; crash between accept and drive resumes the initial state; prompt convenience and explicit accept+drive produce byte-identical writes/events; supplied operation id reaches `pi.op.meta`/events/result; successful run with final assistant fields; one frame append per convertible event with no per-frame storage await and only the latest promise retained; settlement awaits the latest frame write and deletes the list atomically; frame-append storage failure faults before `after_response`; caller rejection and adapter-reported failures; exact ordering; terminal cleanup; same-id drive join during the accept/drive gap; close at every boundary; invalid references rejected at consumption. | +| R3 | **Generation recovery and retry** | Activation-only unknown-effect recovery that reduces committed frames into the synthetic zero-usage partial error under the reserved ids and then classifies ordinarily (retry or failure drain), ordinary stop/error/deferred classification, provider-compliant `aborted`, failure-drain foundation, and both retry-wait policies: explicit drive returns `waiting/notBefore`, while convenience drive may own the abort-aware timer. Define stable breakpoint descriptors for retry timers and recovery commits. Overflow classification remains explicitly unimplemented until R9. | Every generation state before/after reopen; accept/close/reopen/drive; recovery with no frames, partial frames, and authoritative end-frame content; synthetic settlement always commits under reserved ids with the interruption warning and deletes the frame list; interrupted partial tool calls never execute; `after_response` never runs for synthetic settlement; caps/backoff; no timer under `waitForRetry:false`; due and future `notBefore`; deadline before retry; convenience/explicit-drive durable equivalence; timer abort-first/start-first; stable descriptors; stop/error/aborted/deferred classification; missing identities. | +| R4 | **Tools** | Compose R1's tool phases with bound context, invocation-scoped memos, `pi.op.tool_args`, scalar `pendingToolOutput(operationId, invocationId)` checkpoints, `outcome_ready` staging in `pi.pending.entry`, completion-order outcome durability, source-ordered materialization, activation recovery/replay, sequential/parallel promises, blocked/genuine-length outcomes, hooks/events/usage, and stable descriptors. The built-in bash tool emits live updates at 100 ms and requests distinct checkpoints at most every two seconds. | Stable invocation id/memos across safe replay; every planned/effect-pending/outcome-ready/completed state; parallel out-of-order staging without replay; synchronous update callback with the latest delivery and latest checkpoint write awaited before `after_tool`; `tool_end` before staging; every requested checkpoint enqueued synchronously with only the latest promise reference retained and no write dropped or coalesced; tool-selected cadence/duplicate suppression; late-write fencing after settlement; JSONL checkpoint growth/compaction; safe replay deletes old checkpoint; unsafe recovery with/without checkpoint; source-ordered materialization; termination and abort-ready states. | +| R5 | **Inbox, configuration, and writes** | `nextRun`/steer/follow-up via `pendingEntry(id)` values, pending-payload consumption-time dereference, `cancelQueued` triage (`not_found`), durable drain markers, checkpoint consumption with pending-value deletion, immediate total config setters, deferred tree writes, adjustments, and stable breakpoint descriptors for the checkpoint boundaries introduced here. | Capture/cancel/consume races, repeated cancellation answering `not_found`, invalid queue payload kinds and missing pending values rejected at consumption, one-at-a-time crash after one drain, pending-value/entry exclusivity at every boundary, custom-write continuation, config-step race, writes surviving reopen. | +| R6 | **Abort, close, and failure drain** | Add expected-id `requestAbort` as commit/signalling only. Compose `abort()` from inspection, marker, and same-id drive. Implement orthogonal control, per-phase reconciliation including planned/effect-pending/outcome-ready tools, best-effort deferred cancellation, waiters/run-when-idle, controlled-crash close, complete terminal deletion of memos/checkpoints/staged outcomes, external-finalization stop, and stable descriptors. | Cancellation with/without live pass; idempotency/fencing; abort at every tool state; restored interruption incorporates latest checkpoint without replay; restored cancellation synthesizes aborted responses preserving frame-reduced partial content and deleting the list; outcome-ready values survive and materialize; checkpoint/memo/frame cleanup; parked barriers; deferred cancellation; close races; external finalization fences late scalar/list writes; failure drain. | +| R7 | **Deferred provider redemption** | One poll per drive pass carrying `pollDeferred:true`; `resume()` composes inspection and that drive option. Add deferred consumption-time dereference and provider identity preflight. Preserve copied configuration/options, per-poll hooks, exact source lineage/equality, poll frame persistence with fresh intent after unknown poll deleting the abandoned frame list, mismatch-to-error, ready tools, advancement of R6 cancellation, and stable poll descriptors. | `pollDeferred:false` returns waiting without work; one permit performs at most one poll; invalid/missing/mismatched durable handles; repeated pending, ready/error/aborted/mismatch, crash positions, poll frame cleanup on every settlement and replacement, unknown-poll snapshot without permit, convenience/primitive equivalence, no cap/backoff/loop, newest-handle cancellation. | +| R8 | **Manual compaction** | Add compaction `OperationRequest` acceptance without task installation and compose `compact()` from accept+drive. Implement reserved-lane preparation, `pi.op.preparation/{opId}:{taskId}`, total structural state with consumption-time dereference, structural model preflight, hook/generated sources, nested request intents/usage, retained tail, retry/recovery/abort, and stable structural descriptors. | Explicit accept+drive versus convenience equivalence; empty/reservation race; accept/close/reopen/drive; invalid task/preparation/source relationships; hook decline/result; crash after request one of split-turn generation; every state/crash; no public summary-stream messages and no structural frame persistence. | | R9 | **Threshold and overflow compaction** | In-run structural decision, durable once-per-trigger threshold marker, continuation preservation, all overflow predicates, atomic response/preparation publication, specified normalization/projection, one overflow recovery flag, bounded second failure, and stable breakpoint descriptors for added boundaries. | Threshold decline/empty across reopen, all overflow classifier/preparation inputs, no overflow tool plan, genuine length, crash/reopen at every transition. | -| R10 | **Navigation** | Add navigation `OperationRequest` acceptance without task installation and compose `navigateTree()` from accept+drive. Implement navigation source/target/summary semantic restore checks, validation, summarized decision/generation, one final transaction combining move/summary/leaf/label with terminal writes, summary-only navigation hook, and stable descriptors. | Explicit accept+drive versus convenience equivalence; root/current/unknown rejection; invalid durable source/target/summary combinations; accept/close/reopen/drive; summarized/unsummarized paths; final leaf at summary; abort race; exact atomic publication including cleanup. | -| R11 | **Schema version and migrations** | Chained migrate-on-open under the writer lease, with total mappings for open `op.state` including `outcome_ready`, staged `pending.entry` results, invocation facts, and optional progress checkpoints; JSONL lenient old-shape replay, mandatory post-migration compaction, refuse-newer. | Version gate; chained crash-idempotent migration; planned/effect-pending/outcome-ready tool states mapped safely; staged result/checkpoint/fact mappings; resumed operation correctness; old bytes retired. | -| R12 | **Surface completion** | Complete snapshots/watch including restored tool checkpoints and outcome-ready visibility, event catalog/order/filtering, telemetry, public exports, backend parity, hosted-control/convenience documentation, and remove scaffold code including the S4 fake client. | Snapshot/event gap; attach during active effect with live/checkpoint divergence, restored effect-pending, outcome-ready, waiting/yielded/terminal states; no replay of old tool lifecycle; stale-wake fencing; sensitive-event/content-free telemetry; full race/crash matrix on all backends. | +| R10 | **Navigation** | Add navigation `OperationRequest` acceptance without task installation and compose `navigateTree()` from accept+drive. Implement navigation source/target/summary consumption-time dereference, validation, summarized decision/generation, one final transaction combining move/summary/leaf/label with terminal writes, summary-only navigation hook, and stable descriptors. | Explicit accept+drive versus convenience equivalence; root/current/unknown rejection; invalid durable source/target/summary combinations; accept/close/reopen/drive; summarized/unsummarized paths; final leaf at summary; abort race; exact atomic publication including cleanup. | +| R11 | **Schema version and migrations** | Chained migrate-on-open under the writer lease, with total mappings for open `pi.op.state` including `outcome_ready`, staged `pi.pending.entry` results, invocation memos, optional progress checkpoints, and assistant frame lists (paged in sequence order, mapped preserving `seq` or deleted whole-key); address namespace/key/kind migrations; JSONL lenient old-shape replay, mandatory post-migration compaction, refuse-newer. | Version gate; chained crash-idempotent migration; planned/effect-pending/outcome-ready tool states mapped safely; staged result/checkpoint/memo mappings; every legacy frame mapped or the whole list explicitly discarded with no completion inference; resumed operation correctness; old bytes retired. | +| R12 | **Surface completion** | Complete snapshots/watch including restored tool checkpoints, restored `streamingMessage` reduced from frames, and outcome-ready visibility, event catalog/order/filtering, telemetry, public exports, backend parity, hosted-control/convenience documentation, and remove scaffold code including the S4 fake client. | Snapshot/event gap; attach during active effect with live/durable divergence for both frames and checkpoints; restored effect-pending exposing reduced `streamingMessage`; live partial precedence over durable reduction; outcome-ready, waiting/yielded/terminal states; no replay of old message or tool lifecycle; stale-wake fencing; sensitive-event/content-free telemetry; full race/crash matrix on all backends. | | P1 | **Protocol schemas** | Future work after the internal harness is complete: define shared TypeBox schemas for serializable pi-ai and harness protocol data, derive the corresponding TypeScript types from those schemas, and reuse them across client/server protocol boundaries. Validation runs only on untrusted wire input and never inside Session, Storage, operation procedures, or in-process extensions. | Schema/type parity, accepted and rejected client/server payloads, protocol round trips, and no validators or schema construction on internal storage paths. | -Existing source guidance: - -- `packages/agent/src/harness/**` and all of its tests are **deletable outright** in slice 1 — no obligation to adapt anything. Salvaging pieces (the compaction preparation/split-turn algorithms for R8–R9 and session fragments) is optional and never required. -- `packages/agent/src/agent-loop.ts` is an independent compatibility implementation. Do not modify or refactor it for the harness. R1 creates the harness-side replacement for its provider/tool mechanics under `packages/agent/src/harness/execution/` (§5.7); later runtime slices compose those primitives. -- `packages/session-backends/sqlite-node`: S2 may keep the working transaction and lease primitives or start clean. -- Telemetry contracts (`packages/telemetry`, the agent-owned schemas) remain authoritative. -- Existing tests are evidence, not authority. Keep those that assert unchanged behavior; delete the rest with the code they tested. - # Part 9 — Invariants and tests ## 9.1 Invariants @@ -3272,40 +3478,43 @@ Storage: 1. Entries and usage rows are **write-once** and share one session-wide id namespace. Writing either kind under any existing id is corruption. 2. Transactions are all-or-none, with strictly increasing `seq` in write order; gaps are legal. `seq` is monotonic session-wide. -3. Registers are the only mutable state. A register delete removes the key; there are no tombstones, and JSON `null` is a legal value only where a namespace's type permits it. -4. **Every payload lives in exactly one place**: an entry, a register, or the ledger. There is no third place data can hide. -5. No read on a hot path may fold history or infer state from an absent value — no history exists to fold. Execution, recovery, and branch hot paths must be index-driven; inventory and debugging APIs page through indexes. +3. Bound values and lists are the only mutable state. `setValue` replaces the current value and `deleteValue` removes it; `appendList` adds one immutable element and `deleteList` removes every element at the exact address. There are no tombstones or per-element mutations, and JSON `null` is legal only where an address's type permits it. +4. **Every payload lives in exactly one place**: an entry, a bound value/list, or the ledger. There is no third place data can hide. +5. No read on a hot path may fold history or infer state from an absent value — no scalar history exists to fold. Execution, recovery, and branch hot paths must be index-driven; inventory and debugging APIs page through indexes. Bounded paged reads of an exact list address derived from current typed state are the one sanctioned ordered read; their contents are auxiliary and never restart authority. Every bound address has one stable namespace, key, kind, and trusted value type per storage version; value helpers cannot target list addresses or vice versa. Namespace `pi` and every `pi.*` namespace are reserved by contract; every built-in namespace starts with `pi.`, and application use is a trusted-programming defect. Core and applications use the same constructors with no privilege split. Exactly five core prefix constructors encapsulate lane inventory and operation-cleanup grammar and are consumed only by `scanValues`. Tree: 6. An entry's parent chain never changes. Branches share prefixes; nothing is copied. 7. Entries are trusted typed internal values. Only a custom entry may omit payload data; external shape corruption is unsupported rather than revalidated on internal reads. -8. Configuration and orchestration never enter the tree. Deleting every operation-owned `op.*`, `pending.entry`, and `pending.tool_output` register must leave a complete, valid conversation and ledger. +8. Configuration and orchestration never enter the tree. Deleting every operation-owned value and list must leave a complete, valid conversation and ledger. 9. A lane's leaf moves only by append or navigation. 10. A branch segment chain, followed to its end, yields the full root path (§2.6). 11. A missing parent is corruption — always (§1.2). Operations: -12. `lane.state/{lane}` confers lane ownership, and `op.state/{operationId}` confers operation-state ownership. An open lane names operation O, `op.meta/O` holds that lane's compatible `Operation`, and `op.state/O` holds an `OperationState` compatible with O's intent kind; state values carry no duplicate owner metadata. -13. Operation-owned `op.*`, `pending.entry`, and `pending.tool_output` registers may exist only while their operation is open: the terminal transaction deletes them atomically with clearing `currentOperationId` (§3.13). Lane-owned `pendingNextRun` registers are never deleted by it. -14. Acceptance must observe `currentOperationId === null`, commits no `ActiveOperation`, and returns before any post-acceptance hook/provider/tool/timer work begins. A supplied operation id obeys §1.2 and is the exact id written to `op.meta`, events, and `lane.lastResult`. -15. A reserved id may exist only with the content its intent named. Queued-content ids begin in `pending.entry`; settlement-family ids begin as strings in `op.state`. A tool-result id may then move through `string only → outcome-ready pending.entry → immutable entry`; no two representations coexist at a commit boundary (§2.2). -16. Only terminal transitions construct a `LaneLastResult`. A terminal outcome is observable once through the live promise and thereafter through `lane.lastResult` until the next terminal transaction on that lane; recovery never reads it. +12. `laneState(lane)` confers lane ownership, and `operationState(operationId)` confers operation-state ownership. An open lane names operation O, `operationMeta(O)` holds that lane's compatible `Operation`, and `operationState(O)` holds an `OperationState` compatible with O's intent kind; state values carry no duplicate owner metadata. +13. Operation-owned values and lists may exist only while their operation is open: the terminal transaction deletes them atomically with clearing `currentOperationId` (§3.13). Lane-owned `pendingNextRun` values are never deleted by it. +14. Acceptance must observe `currentOperationId === null`, commits no `ActiveOperation`, and returns before any post-acceptance hook/provider/tool/timer work begins. A supplied operation id obeys §1.2 and is the exact id written to `pi.op.meta`, events, and `pi.lane.lastResult`. +15. A reserved id may exist only with the content its intent named. Queued-content ids begin in `pi.pending.entry`; settlement-family ids begin as strings in `pi.op.state`. A tool-result id may then move through `string only → outcome-ready pi.pending.entry → immutable entry`; no two representations coexist at a commit boundary (§2.2). An effect-pending response id may additionally key its auxiliary frame list (§3.7); frames are observation, not a content representation, and die with settlement. +16. Only terminal transitions construct a `LaneLastResult`. A terminal outcome is observable once through the live promise and thereafter through `pi.lane.lastResult` until the next terminal transaction on that lane; recovery never reads it. 17. At most one operation is open per lane. Two is corruption. 18. `overflowRecoveryUsed` is `true` only after overflow compaction. A transition that adds projecting conversational input or tool results and requires an assistant writes `false`; an unprojected custom write preserves it. 19. **The settlement transaction that commits a response with `stopReason: "aborted"` must, in that same transaction, write an operation state with `control.status === "cancel_requested"`.** The invariant is scoped to the committing transaction — later terminal cleanup or forks may remove the state without violating it. Providers must comply with the harness-owned signal contract; violation is corruption. -20. Current-state semantic checks (§3.3) run on every latest lane/operation state before execution — idle lanes included (§4.4). `lane.lastResult` never determines an open operation's next procedure. -21. At most one terminal transaction ever commits per operation. A task mutation that finds its operation's registers absent raises internal `OperationEnded`; the task stops without writing and resolves from `lane.lastResult` (§4.9). +20. Restore checks only projection-value existence (§3.3, §4.4); committed internal values are trusted, and every other named value is dereferenced and checked by the procedure that consumes it. Live phase/effect-identity checks on the lane line remain concurrency fences. `pi.lane.lastResult` never determines an open operation's next procedure. +21. At most one terminal transaction ever commits per operation. A task mutation that finds its operation values absent raises internal `OperationEnded`; the task stops without writing and resolves from `pi.lane.lastResult` (§4.9). 22. At most one `ActiveOperation` drive pass exists per lane. Acceptance and taskless `requestAbort` never install one. A matching `drive` installs it before releasing the lane mutation line; another matching drive joins that pass, and a stale id starts nothing. Every pass's outer `finally` removes it in a no-write lane job only after final local observation; external finalization retains and signals it until that path runs. 23. The §4.2 `assertOpen()` catalog is complete. Every listed hook/provider/tool/timer integration checks immediately before invocation with no yield between them; no unlisted code checks the operation gate. Preparation stays before the check, and admitted asynchronous provider setup/delegation are continuations of the same operation owning `EffectGate.signal`. 24. `drive` and `requestAbort` are fenced by expected operation id. They may affect only that current operation; `drive` may also return its matching latest terminal result. A stale wake for A cannot drive or cancel B. 25. `drive` returns `yielded` only at an effect-free safe boundary. It starts no following transition, hook, provider, tool, or timer after deciding to yield. An already-admitted effect may outlive the caller's deadline and must settle normally or be recovered after task loss. 26. Convenience operations and their explicit primitive compositions produce the same durable writes, events, results, and recovery behavior. Their only difference is process-local waiting/scheduling policy. -27. Each logical tool call's public `invocationId` is its reserved `resultEntryId`: unique within the session and unchanged across safe replay. Tools must await invocation-fact writes. Such writes synchronously enqueue, verify effect-pending ownership on the lane line, and are deleted with outcome staging. +27. Each logical tool call's public `invocationId` is its reserved `resultEntryId`: unique within the session and unchanged across safe replay. Tools must await invocation-memo writes. Such writes synchronously enqueue, verify effect-pending ownership on the lane line, and are deleted with outcome staging. 28. Completed tool calls form a source-ordered prefix. A sequential suffix permits at most one effect-pending or outcome-ready call before planned calls; a parallel suffix may mix `planned`, `effect_pending`, and `outcome_ready`. Completion-order outcome staging never extends the prefix; source-ordered materialization does. -29. Every outcome-ready call has exactly one matching finalized `pending.entry`, no immutable result entry, no invocation facts, and no tool-output checkpoint. Outcome-ready and completed calls never execute again. -30. A tool progress checkpoint is an optional bounded complete `AgentToolResult` snapshot, selected with `checkpoint:true`. It never proves completion. At most one checkpoint commit and one latest waiting request exist per invocation; tool-promise settlement drops the waiting request, while staging or terminal cleanup deletes an active value and prevents late recreation. +29. Every outcome-ready call has exactly one matching finalized `pi.pending.entry`, no immutable result entry, no invocation memos, and no tool-output checkpoint. Outcome-ready and completed calls never execute again. +30. A tool progress checkpoint is an optional bounded complete `AgentToolResult` snapshot, selected with `checkpoint:true`. It never proves completion. Every selected checkpoint synchronously enqueues one invocation-fenced scalar replacement; no write is dropped or coalesced, only the latest write promise reference is retained, and awaiting it implies completion of every earlier write. Staging or terminal cleanup deletes the value and fences late recreation. +31. Scalar assistant/deferred state is the sole restart authority for streamed partials. One effect-pending response id constructs exactly one `pendingAssistantFrames(operationId, responseEntryId)` address; every element is an exported pi-ai `AssistantMessageFrame`; frame order equals provider event order; terminal `done`/`error` events are never stored; frames never establish provider completion or suppress unknown-outcome recovery. +32. Every final or synthetic response settlement — normal, recovery, cancellation, or external finalization — atomically deletes its exact frame list. Idle forks contain no frame lists. A restored partial may appear in `streamingMessage` but never in `transcript` before settlement. +33. The provider loop never awaits storage per frame; frame appends are enqueued synchronously in provider-event order, and awaiting the latest frame-write promise at stream settlement implies every accepted append completed. ## 9.2 Race catalog @@ -3319,8 +3528,11 @@ Each durable mutation race has exactly two durable histories. Process-local owne | stale `drive(A)`/`requestAbort(A)` vs current B | expected-id mismatch; B is untouched | | `requestAbort` vs response settlement | marker first → normalized `aborted`; response first → stop reason preserved | | `abort` vs started tool outcome staging | abort first → real result stages under cancelled control; outcome first → finalized result is preserved and later materializes | -| checkpoint vs tool settlement | active checkpoint was enqueued first and is later deleted by staging; waiting checkpoint is dropped when the tool promise settles; a late update is ignored | -| external finalization vs fact/checkpoint mutation | mutation first → terminal cleanup deletes it; finalization first → durable ownership check rejects the mutation | +| checkpoint vs tool settlement | every accepted checkpoint was enqueued before settlement closed admission; settlement awaits the latest write, then staging deletes the scalar; a late update is fenced without committing | +| assistant frame append vs response settlement | settlement awaits the latest frame write, then its transaction deletes the list; a crash between leaves the committed frame prefix under `effect_pending` | +| assistant frame append vs external finalization | append first → terminal cleanup deletes the list; finalization first → the fenced append rejects without recreating state | +| live update event vs its queued frame/checkpoint commit | either finishes first; events are observation, and reconnect uses only committed frames/checkpoints | +| external finalization vs memo/checkpoint mutation | mutation first → terminal cleanup deletes it; finalization first → durable ownership check rejects the mutation | | later tool B settles vs earlier tool A | B stages outcome-ready immediately; tree placement waits for A | | `abort` vs `before_run_end` follow-up | follow-up dropped; or committed and the run continues | | `cancelQueued` vs checkpoint consumption | `cancelled`; or `already_consumed` | @@ -3337,19 +3549,19 @@ Each durable mutation race has exactly two durable histories. Process-local owne ## 9.3 Test tiers -**Tier A — state and drive.** For every state in Part 3, construct it durably, close, reopen, drive its expected operation id, and assert the next breakpoint, wait, yield, or terminal result. Coverage must include: restore with no branch walk and no re-resolution of captured step configuration; assistant intent with no settlement, below and at the retry cap; settlement followed by each classification branch; every settled stop reason surviving except the two deliberate normalizations; a self-contained deferred step with copied configuration, consecutive polls, repeated equal-handle pending responses, ready and terminal responses, and handle-mismatch normalization into durable failure; every tool state including planned, effect-pending safe/unsafe with and without a checkpoint, outcome-ready, and completed; parallel B/C outcomes staged before A and never replayed after reopen; source-ordered materialization; invocation facts surviving safe replay and dying at outcome staging; a batch where every call terminates; genuine-length synthetic outcomes; every overflow crash position; every navigation state; abort at every position; missing identities; every terminal transaction proving deletion of tool args, facts, checkpoints, staged outcomes, and pending payloads; `lane.lastResult`; preserved `pendingNextRun`; representation exclusivity for every reserved id; and every half-completed recovery prefix. +**Tier A — state and drive.** For every state in Part 3, construct it durably, close, reopen, drive its expected operation id, and assert the next breakpoint, wait, yield, or terminal result. Coverage must include: projection-only restore with no branch walk, no list reads, no reference audit, and no re-resolution of captured step configuration — missing referenced values are detected at consumption; assistant intent with no settlement — with no frames, partial frames, and authoritative end-frame content — recovered into synthetic partial settlements below and at the retry cap; settlement followed by each classification branch; every settled stop reason surviving except the two deliberate normalizations; a self-contained deferred step with copied configuration, consecutive polls, repeated equal-handle pending responses, ready and terminal responses, and handle-mismatch normalization into durable failure; every tool state including planned, effect-pending safe/unsafe with and without a checkpoint, outcome-ready, and completed; parallel B/C outcomes staged before A and never replayed after reopen; source-ordered materialization; invocation memos surviving safe replay and dying at outcome staging; a batch where every call terminates; genuine-length synthetic outcomes; every overflow crash position; every navigation state; abort at every position; missing identities; every terminal transaction proving deletion of tool args, memos, checkpoints, frame lists, staged outcomes, and pending payloads; `pi.lane.lastResult`; preserved `pendingNextRun`; representation exclusivity for every reserved id; and every half-completed recovery prefix. For each recovery prefix: close, reopen, drive, and compare against uninterrupted recovery. Invoking recovery twice from the initial prefix is **not** sufficient. Every operation kind also covers accept → close before first drive → reopen → drive. -One corruption assertion constructs an `aborted` response with running control directly and requires load rejection. Provider conformance separately proves implementations emit `aborted` only for the supplied signal. +One corruption assertion constructs an `aborted` response with running control directly and requires the consuming transition to reject it as an invariant defect. Provider conformance separately proves implementations emit `aborted` only for the supplied signal. -**Tier B — writer conformance.** Run the public harness against the instrumented-storage decorator: a spy wrapping `Storage.commit()` that records every transaction's writes in order. Assert exact write order and content against the Part 3 transaction tables and §5.5 ordering. Faux provider/tool/hook spies interleave starts/events with commits. This tier catches: effects before intent; failure to await the latest update delivery before `after_tool`; `tool_end` emitted after rather than before outcome staging; missing response/usage settlement; checkpoint writes after ownership loss; outcomes not staged before replay becomes impossible; out-of-order tree placement; result ids reserved late; invocation facts or staged/checkpoint registers leaked by outcome/terminal cleanup. +**Tier B — writer conformance.** Run the public harness against the instrumented-storage decorator: a spy wrapping `Storage.commit()` that records every transaction's writes in order. Assert exact write order and content against the Part 3 transaction tables and §5.5 ordering. Faux provider/tool/hook spies interleave starts/events with commits. This tier catches: effects before intent; failure to await the latest update delivery or latest checkpoint write before `after_tool`; a provider loop awaiting storage per frame; frame appends out of provider-event order or persisted for `done`/`error`; a settlement transaction missing its frame-list delete; `tool_end` emitted after rather than before outcome staging; missing response/usage settlement; checkpoint or frame writes after ownership loss; outcomes not staged before replay becomes impossible; out-of-order tree placement; result ids reserved late; invocation memos or staged/checkpoint/frame values leaked by outcome/terminal cleanup. **Tier C — deterministic interleavings.** Every race in §9.2, both orders, using the manual breakpoint barrier. Every introduced breakpoint kind and JSON-safe details shape is asserted explicitly. **Cross-cutting:** -- **Backend conformance.** One suite, three backends, identical results — including checkpoint scalar set/replace/delete and torn-transaction handling. Memory/SQLite retain one current checkpoint; JSONL may retain superseded bytes physically but compaction produces identical logical state. Internal values are not cloned or shape-validated. Write-order assertions use the instrumented decorator, never a durable log. +- **Backend conformance.** One suite, three backends, identical results — including checkpoint scalar set/replace/delete, list append/page/whole-key-delete with identical sequence cursors and reduced frame sequences, and torn-transaction handling exposing no list element. Memory/SQLite retain one current checkpoint; JSONL may retain superseded bytes physically but compaction produces identical logical state, including preserved list cursors. Internal values are not cloned or shape-validated. Write-order assertions use the instrumented decorator, never a durable log. - **Drive equivalence.** The same scenario in automatic and manual breakpoint mode must produce byte-identical durable state. Convenience calls and explicit `accept`/`drive`/`requestAbort` compositions must likewise produce byte-identical durable state and equivalent events/results. - **Breakpoint catalog.** Each runtime slice specifies and tests the stable `ActionInfo.kind` values and JSON-safe details for the boundaries it introduces. A parked operation performs no following procedure work; abort rejects interruptible ordinary barriers into reconciliation but leaves cancellation barriers parked; close/fault rejects every barrier. - **Effect-start gate.** Cover every item in §4.2's complete catalog and assert that no other path calls `assertOpen()`. At each listed integration, force both orders of abort versus operation admission: the check and invocation are adjacent, abort-first invokes nothing, and admission-first gives the complete operation `EffectGate.signal`. Provider tests assert that request preparation precedes the check and that the same signal reaches Models auth/lazy/provider work. Hook tests treat each aggregate pipeline as one admitted unit. A cancelled activation must enter reconciliation without invoking `before_resume`. @@ -3366,17 +3578,18 @@ One corruption assertion constructs an `aborted` response with running control d | Term | Meaning | |---|---| | **Entry** | Write-once conversation record: placement and payload in one row. Its id is the public entry id. | -| **Register** | Namespaced mutable cell holding its current typed value directly. Overwrite replaces; delete removes the key. | +| **Value address** | Bound `Value` naming one replaceable current value by stable namespace and key. | +| **List address** | Bound `ValueList` naming one append-only list; each element has its own global write sequence and whole-list deletion is lifecycle cleanup. | | **Usage row** | Append-only cost ledger row. Never modified, never deleted. | -| **Pending entry** | Complete unplaced content in `pending.entry`, including queued input and finalized outcome-ready tool results, until placement/cancellation/cleanup. | -| **Session** | One conversation: tree, facts, ledger, lanes. | +| **Pending entry** | Complete unplaced content in `pi.pending.entry`, including queued input and finalized outcome-ready tool results, until placement/cancellation/cleanup. | +| **Session** | One conversation: tree, values/lists, ledger, lanes. | | **Lane** | Named cursor into the tree with its own config, queues, and one operation. | | **Operation** | One accepted unit of work: run, compaction, or navigation. | | **Acceptance** | The atomic creation of one durable operation by `accept`; it starts no driver. | | **Drive pass** | One process-local ownership span created or joined by `drive`, ending at settlement, a durable wait, or safe yield. | | **Effect** | Anything not pure computation: commit, provider request, tool, hook, timer. | | **Repeat-sensitive effect** | One whose repetition is observable outside the harness. | -| **Operation state** | The complete durable state of one open operation — the `op.state` register and restart point after task loss. | +| **Operation state** | The complete durable state of one open operation — the `operationState(operationId)` value and restart point after task loss. | | **Active operation** | The one process-local async task for the lane's current drive pass, with its completion promise and `EffectGate`. | | **Breakpoint** | A manual-drive barrier described by `ActionInfo`; it parks before ordinary code but does not execute that code. | | **Effect gate** | Process-local synchronous arbitration of ordinary operation admission against a cancellation request, paired with the operation's cooperative signal. | @@ -3386,12 +3599,13 @@ One corruption assertion constructs an `aborted` response with running control d | **Control** | Orthogonal cancellation flag: `running` or `cancel_requested`. | | **Checkpoint** | The state between turns where queues, writes, and finishing are decided. | | **Continuation** | Durable answer to "does this run still owe an assistant turn?" | -| **Tool checkpoint** | Optional bounded complete live-update snapshot selected by a tool and stored as the current `pending.tool_output` value; auxiliary, never completion authority. | -| **Outcome ready** | Tool call whose complete finalized result is durable in `pending.entry` and will never execute again, but awaits source-ordered tree placement. | -| **Invocation fact** | Tool-invocation-scoped durable current value used for replay-safe memoization; valid only while the call is effect-pending. | -| **Terminal transaction** | The commit that deletes an operation's registers, writes `lane.lastResult`, and clears `currentOperationId`. | +| **Tool checkpoint** | Optional bounded complete live-update snapshot selected by a tool and stored as the current `pi.pending.tool_output` value; auxiliary, never completion authority. | +| **Assistant frame** | Compact replayable pi-ai stream frame appended to the `pi.pending.assistant_frame` list while its response is effect-pending; reduced by `reduceAssistantMessageFrames`; auxiliary, never completion authority. | +| **Outcome ready** | Tool call whose complete finalized result is durable in `pi.pending.entry` and will never execute again, but awaits source-ordered tree placement. | +| **Invocation memo** | Tool-invocation-scoped durable current value used for replay-safe memoization; valid only while the call is effect-pending. | +| **Terminal transaction** | The commit that deletes an operation's values/lists, writes `pi.lane.lastResult`, and clears `currentOperationId`. | | **Segment** | A branch-index range that references an older branch instead of copying it. | -| **External finalization** | A terminal transaction committed from outside the live task; the task detects absent registers, stops without writing, and resolves from `lane.lastResult` (§4.9). | +| **External finalization** | A terminal transaction committed from outside the live task; the task detects absent operation values, stops without writing, and resolves from `pi.lane.lastResult` (§4.9). | | **Precise rewrite** | The administrative copy-retained-and-swap rebuild of a session store — the sole sanctioned path that removes entries or usage rows (§2.9). | # Appendix B — Coding-agent v3-format compatibility @@ -3399,7 +3613,7 @@ One corruption assertion constructs an `aborted` response with running control d "v3" in this appendix names the legacy coding-agent JSONL session format, not this document. Old coding-agent v3 JSONL files must open unchanged and restore idle. Normalization on load: - `custom_message` becomes a custom agent message. -- `label` and `session_info` become facts (latest by file position wins) and leave the tree. A label targets its nearest retained parent. +- `label` and `session_info` become session-name/entry-label values (latest by file position wins) and leave the tree. A label targets its nearest retained parent. - Legacy `model_change`, `thinking_level_change`, and `active_tools_change` nodes disappear. They do **not** initialize or alter `LaneConfiguration`; a normalized `main` uses the immutable options seed. - Each retained child of a discarded node is reparented to its nearest retained ancestor. - `main`'s leaf is the final physical node resolved through discarded nodes to its nearest retained ancestor. diff --git a/packages/agent/docs/registers.md b/packages/agent/docs/registers.md index 4222a6f32..43e8654b4 100644 --- a/packages/agent/docs/registers.md +++ b/packages/agent/docs/registers.md @@ -1,224 +1,260 @@ -# Typed scalar and list registers — implementation handoff +# Typed values and lists — implementation handoff -This document specifies the register storage primitive used by the harness. +This document specifies the mutable storage primitive used by Session, the harness, and applications. -Registers are identified by ordinary typed token values rather than a global `RegisterValues` or `ListRegisterValues` namespace map: +The public abstraction is a **bound typed address**: -- **scalar register:** one current value per `(token, key)`; set replaces it; -- **list register:** immutable values appended under `(token, key)` and read in bounded sequence pages. +- `value(namespace, key?)` names one replaceable durable value; +- `list(namespace, key?)` names one append-only durable list whose elements have type `T`. -Built-in tokens live together in `packages/agent/src/harness/session/registers.ts` and are imported directly by their consumers. There is no `HarnessRegisters` bundle, runtime catalog, declaration merging, or raw register access from tools. +The namespace/key pair is bound once when the address is constructed. Every later operation receives only that address. Application code therefore writes: + +```ts +const state = value("my-app.state"); +const events = list("my-app.events"); + +await session.getValue(state); +await session.setValue(state, nextState); +await session.readList(events, { limit: 100 }); +await session.appendList(events, event); +``` + +It does **not** repeatedly pass a second unexplained key: + +```ts +// Not the API. +await session.readList(events, "another-key", { limit: 100 }); +``` + +When an application genuinely has keyed instances, it constructs the address for that instance: + +```ts +const workspaceEvents = (workspaceId: string) => + list("my-app.events", workspaceId); + +await session.readList(workspaceEvents("pi"), { limit: 100 }); +``` + +Storage may physically index the address as `(kind, namespace, key)`, but that representation does not leak into each read or write call. Storage, Session, harness code, and applications use the same address vocabulary. There is no global value-type map, dynamic registry, token catalog, or separate application-state storage mechanism. ## Goals -1. Keep persisted register IDs stable while moving their value types onto tokens. -2. Allow built-in and future application register definitions without editing a global type map. -3. Preserve type inference through reads, writes, scans, and list appends. -4. Preserve existing scalar replacement semantics. -5. Append a list element without reading or rewriting existing elements. -6. Use the existing transaction sequence as list order and cursor. -7. Produce identical logical behavior on Memory, JSONL, and SQLite. -8. Keep list reads bounded and explicit. -9. Commit scalar/list writes atomically with entries and usage. +1. Give one exact durable address one compile-time value type. +2. Let applications define scalar values and lists without declaration merging or editing a core type map. +3. Use the same typed addresses and operation names from Storage through Session. +4. Preserve current scalar replacement semantics. +5. Append one list element without reading or rewriting existing elements. +6. Give every list element its own existing session-global transaction `seq`, used for ordering and pagination. +7. Commit values and list elements atomically with entries and usage. +8. Produce identical logical behavior on Memory, JSONL, and SQLite. +9. Keep list reads bounded and explicit. +10. Keep scalar operation state authoritative; auxiliary lists never select a recovery state. ## Non-goals This slice does not define: - assistant-frame contents or reduction semantics; -- tool progress semantics; -- per-token runtime value validation; -- per-element or per-key byte limits; +- tool-progress semantics; +- runtime validation of trusted in-process values; +- per-element or per-list byte limits; - list truncation or per-element deletion; -- a generic event log or operation-state reducer; -- dynamic registration, token catalogs, or dependency-injected built-in token bundles. +- a generic event log, journal, stream-resumption protocol, or operation reducer; +- globally registering address objects; +- exposing raw Session or transaction access to tools. -Consumers own key grammar, content limits, cleanup points, and restore hydration policy. Values remain trusted typed in-process data under the validation boundary in `harness.md`. +Consumers own address construction, content limits, cleanup points, fork policy, migration policy, and consumption-time hydration. `assistant-durability.md` defines the first list consumer. -## Token model - -A token carries its register kind, stable persisted ID, and compile-time value type: +## Bound address model ```ts -declare const registerValueType: unique symbol; +declare const storedValueType: unique symbol; -interface RegisterTokenBase { - readonly id: string; - readonly kind: "scalar" | "list"; +interface StoredAddressBase { + /** Stable persisted grouping name. */ + readonly namespace: string; + /** Exact member inside that grouping. Empty is legal. */ + readonly key: string; + readonly kind: "value" | "list"; } -export interface ScalarRegisterToken extends RegisterTokenBase { - readonly kind: "scalar"; - /** Compile-time only; makes the value type part of the token. */ - readonly [registerValueType]?: (value: T) => T; +export interface Value extends StoredAddressBase { + readonly kind: "value"; + /** Compile-time only and invariant in T. */ + readonly [storedValueType]?: (value: T) => T; } -export interface ListRegisterToken extends RegisterTokenBase { +export interface ValueList extends StoredAddressBase { readonly kind: "list"; - /** Compile-time only; T is one element, not the whole list. */ - readonly [registerValueType]?: (value: T) => T; + /** T is one element, not the whole list. */ + readonly [storedValueType]?: (value: T) => T; } -export function scalarRegister(id: string): ScalarRegisterToken { - validateRegisterId(id); - return Object.freeze({ id, kind: "scalar" }); +export function value(namespace: string, key = ""): Value { + validateAddress(namespace, key); + return Object.freeze({ namespace, key, kind: "value" }); } -export function listRegister(id: string): ListRegisterToken { - validateRegisterId(id); - return Object.freeze({ id, kind: "list" }); +export function list(namespace: string, key = ""): ValueList { + validateAddress(namespace, key); + return Object.freeze({ namespace, key, kind: "list" }); } ``` -The phantom function keeps the type invariant: a token for one value type cannot silently widen to a token for another. It has no runtime field. +The phantom function makes `T` invariant: an address for one type cannot silently widen to another. It has no runtime field. -A register ID must be non-empty and must not contain the internal Memory-map separator (`\u0000`). It is serialized into JSONL and SQLite and therefore remains stable across reopen. Changing an ID or changing its scalar/list kind requires a storage migration. +Rules: -Token object identity carries no durability meaning. Storage addresses a register by `(kind, token.id, key)`. +- `namespace` must be non-empty; +- namespace `pi` and every `pi.*` namespace are reserved for built-ins by contract; +- applications that construct a reserved address are defective trusted in-process code; no runtime privilege split, registry, or catalog exists; +- neither component may contain the Memory backend's internal separator (`\u0000`); +- an empty key is valid and is the natural address for one application-wide value or list; +- object identity has no durable meaning; +- separately constructed addresses with the same `(kind, namespace, key)` identify the same durable location; +- constructing the same durable location with incompatible TypeScript types is a trusted-programming defect; +- scalar and list addresses may not share the same `(namespace, key)` in one storage version; +- changing an address's namespace, key, kind, or incompatible value shape requires migration. + +The two components remain separate rather than concatenated. Dynamic application keys and operation IDs therefore require no escaping convention beyond the storage separator rule. + +### Exact addresses, not families + +An address names one value or one list. Internal code uses small constructors when it has dynamic keys: + +```ts +export const laneLeaf = (lane: string) => + value("pi.lane.leaf", lane); + +export const operationState = (operationId: string) => + value("pi.op.state", operationId); + +export const operationToolArgs = ( + operationId: string, + stepId: string, + sourceIndex: number, +) => value>( + "pi.op.tool_args", + `${operationId}:${stepId}:${sourceIndex}`, +); + +export const pendingAssistantFrames = ( + operationId: string, + responseEntryId: string, +) => list( + "pi.pending.assistant_frame", + `${operationId}:${responseEntryId}`, +); +``` + +This encapsulates each key grammar at its owner. Call sites receive an already-bound typed address: + +```ts +await reader.getValue(operationState(operationId)); +await reader.readList(pendingAssistantFrames(operationId, responseEntryId), options); +``` ### No global value map -Delete these shapes: +Delete the existing global namespace-to-type maps: ```ts -interface RegisterValues { /* namespace → value */ } -interface ListRegisterValues { /* namespace → element */ } - -type RegisterNamespace = keyof RegisterValues; -type ListRegisterNamespace = keyof ListRegisterValues; +interface RegisterValues { /* delete */ } +interface ListRegisterValues { /* delete */ } +type RegisterNamespace = keyof RegisterValues; // delete ``` -A type belongs to its token regardless of where the token is defined: +A type belongs to an address constructor instead: ```ts -export const laneLeaf = scalarRegister("lane.leaf"); -export const applicationState = scalarRegister("my-app.state"); +export const applicationState = value("my-app.state"); +export const applicationEvents = list("my-app.events"); ``` -The storage API does not need to know either type centrally. +Applications should use a stable, collision-resistant namespace prefix. Namespace `pi` and the complete `pi.*` prefix are reserved for built-ins by contract; similar-looking names such as `pi2` remain legal. The same `value()` and `list()` constructors serve core and application code. Tests assert that every built-in address uses its reserved prefix. There is no runtime privilege split, registry, or catalog. -## Built-in tokens +## Built-in addresses -Define built-ins directly in `packages/agent/src/harness/session/registers.ts`: +Built-in constructors live together in `packages/agent/src/harness/session/values.ts` and are imported directly by consumers. Representative definitions: ```ts -export const laneLeaf = scalarRegister("lane.leaf"); -export const laneConfig = scalarRegister("lane.config"); -export const laneState = scalarRegister("lane.state"); -export const laneLastResult = scalarRegister("lane.lastResult"); +export const laneLeaf = (lane: string) => value("pi.lane.leaf", lane); +export const laneConfig = (lane: string) => value("pi.lane.config", lane); +export const laneState = (lane: string) => value("pi.lane.state", lane); +export const laneLastResult = (lane: string) => + value("pi.lane.lastResult", lane); -export const operationMeta = scalarRegister("op.meta"); -export const operationState = scalarRegister("op.state"); -export const operationToolArgs = - scalarRegister>("op.tool_args"); -export const operationToolFact = scalarRegister("op.tool_fact"); -export const operationPreparation = - scalarRegister("op.preparation"); +/** Used only by scanValues() to enumerate configured lane names. */ +export const laneLeafInventoryPrefix = () => value("pi.lane.leaf"); -export const pendingEntry = scalarRegister("pending.entry"); -export const pendingToolOutput = - scalarRegister>("pending.tool_output"); +export const operationMeta = (operationId: string) => + value("pi.op.meta", operationId); +export const operationState = (operationId: string) => + value("pi.op.state", operationId); +export const operationToolArgs = (operationId: string, stepId: string, sourceIndex: number) => + value>( + "pi.op.tool_args", + `${operationId}:${stepId}:${sourceIndex}`, + ); +export const operationToolMemo = (operationId: string, invocationId: string, name: string) => + value("pi.op.tool_memo", `${operationId}:${invocationId}:${name}`); +export const operationPreparation = (operationId: string, taskId: string) => + value( + "pi.op.preparation", + `${operationId}:${taskId}`, + ); -/** Compact frames defined by the assistant-partial durability slice. */ -export const pendingAssistantFrames = - listRegister("pending.assistant_frame"); +/** Prefix addresses are exported only for namespace-scoped scanValues(). */ +export const operationToolArgsPrefix = (operationId: string, stepId?: string) => + value>( + "pi.op.tool_args", + stepId === undefined ? `${operationId}:` : `${operationId}:${stepId}:`, + ); +export const operationToolMemoPrefix = (operationId: string, invocationId?: string) => + value( + "pi.op.tool_memo", + invocationId === undefined ? `${operationId}:` : `${operationId}:${invocationId}:`, + ); +export const operationPreparationPrefix = (operationId: string) => + value("pi.op.preparation", `${operationId}:`); -export const factName = scalarRegister("fact.name"); -export const factLabel = scalarRegister("fact.label"); -export const factCustom = scalarRegister("fact.custom"); +export const pendingEntry = (entryId: string) => + value("pi.pending.entry", entryId); +export const pendingToolOutput = (operationId: string, invocationId: string) => + value>( + "pi.pending.tool_output", + `${operationId}:${invocationId}`, + ); +export const pendingAssistantFrames = (operationId: string, responseEntryId: string) => + list( + "pi.pending.assistant_frame", + `${operationId}:${responseEntryId}`, + ); +export const pendingToolOutputPrefix = (operationId: string) => + value>("pi.pending.tool_output", `${operationId}:`); + +export const sessionName = value("pi.session.name"); +export const entryLabel = (entryId: string) => value("pi.entry.label", entryId); ``` -These are normal constants, not a configuration object. Harness code imports the token it uses: +The five exported scan-prefix constructors are `laneLeafInventoryPrefix`, `operationToolArgsPrefix`, `operationToolMemoPrefix`, `operationPreparationPrefix`, and `pendingToolOutputPrefix`. Their addresses are consumed only by `scanValues()`. + +Applications define their own `value()` and `list()` addresses directly; there is no built-in custom application-state namespace or custom-state API. `AgentHarnessToolInvocation.getMemo()` and `setMemo()` are invocation-fenced capabilities over `operationToolMemo(...)`, not raw Session access. Invocation memos remain operation-owned and are deleted when their tool outcome becomes durable. + +Tests assert that built-in constructors produce the documented kind, namespace, and key grammar. Because constructors may be dynamic, there is no runtime catalog that tries to enumerate every possible address. + +## Shared read API + +Storage, Session, SessionReader, and SessionMutator use the same read signatures: ```ts -await reader.getRegister(operationState, operationId); -await reader.readList(pendingAssistantFrames, frameKey, options); -``` - -A future application token may be defined in the application's own `registers.ts`. Persisted IDs must be globally unique within a session. Defining two tokens with the same `(kind, id)` but different TypeScript value types is a trusted-programming defect. Defining scalar and list tokens with the same ID is also a defect. - -Tests assert that all built-in exported tokens have unique IDs and fixed kinds. - -## Typed scalar API - -```ts -export interface ScalarRegisterValue { - key: string; +export interface StoredValue { + address: Value; value: T; seq: number; } -interface SessionReader { - getRegister( - register: ScalarRegisterToken, - key: string, - ): Promise | undefined>; - - listRegisters( - register: ScalarRegisterToken, - keyPrefix?: string, - ): Promise[]>; -} -``` - -`listRegisters` lists scalar keys under one token; it is unrelated to list-register element reads. It retains its existing bounded-use contract: harness callers use exact operation-owned prefixes, and no public API exposes an unrestricted session dump. - -Register writes are created through typed helpers. The stored transaction representation is erased only after the helper checks the token/value relationship: - -```ts -interface RegisterSetWrite { - kind: "register"; - op: "set"; - namespace: string; - key: string; - value: unknown; -} - -interface RegisterDeleteWrite { - kind: "register"; - op: "delete"; - namespace: string; - key: string; -} - -export function setRegister( - register: ScalarRegisterToken, - key: string, - value: NoInfer, -): RegisterSetWrite { - return { - kind: "register", - op: "set", - namespace: register.id, - key, - value, - }; -} - -export function deleteRegister( - register: ScalarRegisterToken, - key: string, -): RegisterDeleteWrite { - if (register.kind !== "scalar") throw new Error("Expected scalar register"); - return { - kind: "register", - op: "delete", - namespace: register.id, - key, - }; -} -``` - -`NoInfer` makes the token authoritative. TypeScript must not widen `T` from an incompatible value argument. - -The raw write interfaces are storage internals. Harness and application code use `setRegister()` and `deleteRegister()` rather than manually constructing erased writes. - -## Typed list API - -A list token's `T` is one immutable element: - -```ts -export interface ListRegisterElement { +export interface ListElement { /** Global transaction-write sequence assigned by storage. */ seq: number; value: T; @@ -231,97 +267,129 @@ export interface ListCursor { export interface ListReadOptions { /** Exclusive cursor. */ cursor?: ListCursor; - /** Default: `asc`. */ + /** Default: asc. */ order?: "asc" | "desc"; /** Default: 1,000. Maximum: 10,000. */ limit?: number; } -interface SessionReader { +interface ValueReader { + getValue(address: Value): Promise | undefined>; + + /** Internal bounded-prefix operation. The address key is interpreted as a prefix. */ + scanValues(prefix: Value): Promise[]>; + readList( - register: ListRegisterToken, - key: string, + address: ValueList, options?: ListReadOptions, - ): Promise[]>; + ): Promise[]>; } ``` -List writes also use typed helpers: +`scanValues(prefixAddress)` scans scalar addresses with exactly that namespace and keys beginning with the bound key. Core call sites use only the exported prefix constructors above so raw namespace/key grammar stays in `session/values.ts`. Prefix addresses are passed only to `scanValues()`, never to exact get/set/delete operations. There is no unrestricted cross-namespace dump. Ordinary application reads use exact addresses. + +`SessionTree` exposes direct one-transition writes using the same addresses: ```ts -interface ListRegisterAppendWrite { - kind: "list-register"; +interface SessionTree extends ValueReader { + setValue(address: Value, next: NoInfer): Promise; + deleteValue(address: Value): Promise; + appendList(address: ValueList, element: NoInfer): Promise; + deleteList(address: ValueList): Promise; +} +``` + +Purpose-specific helpers such as `getName()`, `setName()`, `getLabel()`, and `setLabel()` may remain thin wrappers over built-in addresses. Applications define and use their own scalar/list addresses directly. + +`SessionMutator` remains a read capability plus one atomic `commit()`. It does not expose direct `setValue()`/`appendList()` methods that would consume its only transaction separately; callers construct typed writes and include them in that commit. + +## Typed transaction writes + +Writes are constructed from bound addresses. Erasure happens only after the helper has checked the address/value type relationship: + +```ts +interface ValueSetWrite { + kind: "value"; + op: "set"; + namespace: string; + key: string; + value: unknown; +} + +interface ValueDeleteWrite { + kind: "value"; + op: "delete"; + namespace: string; + key: string; +} + +interface ListAppendWrite { + kind: "list"; op: "append"; namespace: string; key: string; value: unknown; } -interface ListRegisterDeleteWrite { - kind: "list-register"; +interface ListDeleteWrite { + kind: "list"; op: "delete"; namespace: string; key: string; } -export function appendList( - register: ListRegisterToken, - key: string, - value: NoInfer, -): ListRegisterAppendWrite { - return { - kind: "list-register", - op: "append", - namespace: register.id, - key, - value, - }; -} - -export function deleteList( - register: ListRegisterToken, - key: string, -): ListRegisterDeleteWrite { - if (register.kind !== "list") throw new Error("Expected list register"); - return { - kind: "list-register", - op: "delete", - namespace: register.id, - key, - }; -} +export function setValue(address: Value, next: NoInfer): ValueSetWrite; +export function deleteValue(address: Value): ValueDeleteWrite; +export function appendList(address: ValueList, element: NoInfer): ListAppendWrite; +export function deleteList(address: ValueList): ListDeleteWrite; ``` -`Write` adds the four erased scalar/list write representations alongside entries and usage. A transaction may mix every write kind atomically. +`NoInfer` makes the address authoritative. TypeScript must not infer a wider `T` from an incompatible write value. -## List write semantics +`Write` includes these four erased representations alongside entry and usage writes. One transaction may mix every write kind atomically. Harness and application code use the helpers rather than manually constructing erased writes. -One append write carries one element. A transaction appending several elements contains several append writes. Every write receives its existing globally increasing transaction sequence: +The direct Session methods and transaction helpers intentionally use the same operation names. One performs and commits a single Session mutation; the other constructs a write for an explicitly composed transaction. + +## Scalar semantics + +For one `Value` address: + +- `setValue` replaces the current value; +- `deleteValue` removes it; +- deleting an absent value is a no-op; +- set after delete recreates it; +- there is no retained value history; +- the current value records the `seq` of its latest set; +- a failed transaction exposes neither the scalar write nor any sibling write. + +## List semantics + +One append write carries one immutable element. A transaction appending several elements contains several append writes. Every write receives its existing globally increasing transaction sequence: ```text TX[ - append pendingAssistantFrames/O:R = A, // seq 41 - set operationState/O = X, // seq 42 - append pendingAssistantFrames/O:R = B // seq 43 + appendList(frames, A), // seq 41 + setValue(operationState, X), // seq 42 + appendList(frames, B), // seq 43 ] ``` -Elements under list key `O:R` are ordered `A, B` by sequence. Gaps caused by unrelated writes are irrelevant. +Reading `frames` returns `A`, then `B`. Gaps from unrelated writes are expected. A list element's `seq` is session-global and unique to that committed write; it is an ordering/cursor identity, not an application domain ID. Applications that need domain identity include it in `T`. Rules: - append never reads existing elements; - an element is immutable after commit; -- whole-key delete removes every element under `(token.id, key)`; -- delete of an absent list is a no-op; +- `deleteList(address)` removes every element at that exact address; +- deleting an absent list is a no-op; - delete followed by append in one transaction creates a fresh list atomically; - there is no per-element update, delete, insertion, or truncation; -- all validation and serialization needed to admit a transaction completes before Memory state changes; +- all validation and serialization required to admit a transaction completes before Memory state changes; - a failed transaction exposes none of its list or non-list writes. -“Append-only” describes elements while the key exists. Whole-key deletion is lifecycle cleanup, not element mutation. +“Append-only” describes elements while the list exists. Whole-list deletion is lifecycle cleanup, not element mutation. -## List read semantics +### List reads - ascending reads return `seq > cursor.seq`; - descending reads return `seq < cursor.seq`; @@ -334,110 +402,112 @@ Rules: ```ts let cursor: ListCursor | undefined; while (true) { - const page = await reader.readList(pendingAssistantFrames, frameKey, { - cursor, - order: "asc", - limit: 100, - }); + const page = await reader.readList(events, { cursor, order: "asc", limit: 100 }); if (page.length === 0) break; consume(page); cursor = { seq: page[page.length - 1]!.seq }; } ``` -A cursor is a sequence filter, not a snapshot or key-incarnation token. Concurrent later appends may appear on later ascending pages. A whole-key delete may make a cursor stale; reads simply apply its sequence comparison to the currently surviving elements. +A cursor is a sequence filter, not a snapshot or list-incarnation token. Concurrent later appends may appear on later ascending pages. Whole-list deletion may make a cursor stale; reads simply apply its sequence comparison to currently surviving elements. Do not add an unbounded “read the whole list” helper. ## Assistant partial frames -Assistant partial durability is the first list consumer: +Assistant partial durability is the first built-in list consumer: -```text -register: pendingAssistantFrames -key: {operationId}:{responseEntryId} -value: one bounded DurableAssistantFrame +```ts +const frames = pendingAssistantFrames(operationId, responseEntryId); ``` -The frame type is finalized with the assistant-stream design, not by storage. It must be a compact incremental frame. Do not append raw `AssistantMessageEvent`: update events contain a growing `partial` message and would reintroduce write amplification inside the list elements. +`AssistantMessageFrame`, `assistantMessageEventToFrame()`, and `reduceAssistantMessageFrames()` come from `@earendil-works/pi-ai`. Do not define a second frame codec or reducer. -The assistant procedure may batch several frame appends into one transaction, but each frame remains an independently sequenced list element. Its scalar `effect_pending` state remains authoritative. +For every convertible non-terminal provider event, the assistant procedure: -On restore: +```text +convert event to frame +→ synchronously enqueue appendList(frames, frame) on the lane mutation line +→ attach the ordinary harness-fault observer to that returned promise +→ replace the process-local latestFrameWrite reference +→ emit and await the existing message event +→ consume the next provider event +``` -1. restore and validate scalar lane/operation state; -2. derive the exact frame key from the valid assistant effect-pending state; -3. read bounded pages from `pendingAssistantFrames`; -4. reduce frames into the latest durable partial assistant message. +The provider loop does not await storage for every frame. Synchronous enqueue preserves provider-event order. Replacing the latest-promise reference never leaves an earlier rejection unobserved because every promise receives the fault observer. Bounded output bounds queued work. On stream settlement, the procedure stops frame admission and awaits the latest append promise before `after_response`; lane-line FIFO means that completion implies every earlier append completed. There is no timer, batcher, coalescer, or flush API. -A missing frame list is valid and means no partial frame committed. Frames never prove provider completion and never select a restart state. +Scalar assistant `effect_pending` remains authoritative. Each append verifies that the same operation, attempt, and response ID still own the lane when its mutation executes. Frames never prove request admission, completion, success, or failure. -Final assistant settlement atomically commits the complete immutable assistant entry and deletes the whole frame-list key: +Final or synthetic assistant settlement deletes the exact list atomically with its immutable response, usage, leaf, and next scalar state: ```text TX[ insert final assistant entry, insert usage, - deleteList(pendingAssistantFrames, O:R), - set operationState/O = next state + deleteList(frames), + setValue(operationState(operationId), nextState), ] ``` -Unknown-effect recovery may reduce the latest durable frames into its synthetic interrupted assistant response, but that policy belongs in the assistant durability specification. +`assistant-durability.md` defines frame conversion, unknown-outcome synthesis, cancellation, deferred polling, snapshots, and event ordering. ## Restore policy -Scalar state remains authoritative for every list consumer: +Scalar operation state remains the sole restart authority: -1. restore scalar lane and operation registers; -2. perform current-state semantic validation; -3. derive auxiliary list keys from valid scalar state; -4. optionally hydrate bounded list pages for snapshots, diagnostics, or activation. +1. construct the trusted lane/operation projection from required scalar values; +2. trust committed typed values rather than auditing every referenced payload or phase relationship; +3. when a procedure or snapshot consumes auxiliary state, derive its exact bound address from current typed scalar state; +4. hydrate only the bounded scalar values or list pages that consumer requires. -A missing list never invalidates an otherwise valid operation. List contents never prove that an external effect completed. +A missing auxiliary list is legal unless its consumer explicitly requires an element. List contents never prove that an external effect completed. Live mutations still verify current operation, phase, attempt, and reserved identity as concurrency fencing; that is not restore validation. -Each concrete list consumer defines: +Base restore does not enumerate lists. For assistant frames, snapshot or recovery derives `pendingAssistantFrames(operationId, responseEntryId)` only while consuming a typed assistant/deferred `effect_pending` state. -- key grammar; -- element and total byte bounds; +Each list consumer defines: + +- address grammar; +- element and total-byte bounds; - page/hydration budget; - cleanup transitions; - fork and migration policy. ## Memory backend -Keep scalar registers in the existing map, now addressed by token ID: +Memory may keep separate maps for current values and list elements: ```ts -const scalars = new Map>(); -// map key: `${register.id}\u0000${key}` +const scalarValues = new Map>(); +const listValues = new Map[]>(); + +function physicalKey(address: StoredAddressBase): string { + return `${address.namespace}\u0000${address.key}`; +} ``` -Add list storage: - -```ts -type StoredListElement = { seq: number; value: unknown }; -const lists = new Map(); -// map key: `${register.id}\u0000${key}` -``` - -- scalar set replaces the map value; +- scalar set replaces one map value; - scalar delete removes it; - list append pushes the already-sequenced element; - list delete removes the complete array; - list read filters by exclusive cursor and slices to the validated limit; -- transaction preparation completes before entries, scalar registers, lists, usage, or stats mutate. +- transaction preparation completes before entries, values, lists, usage, or stats mutate. Storage snapshots used by JSONL/fork tooling include current scalar values and surviving list elements with original sequence numbers. ## SQLite backend -The existing scalar register table continues to store `token.id` in its namespace column. No scalar schema change is required. - -Add a schema migration for lists rather than modifying an applied migration: +The logical schema has one current-value table and one list-element table: ```sql -CREATE TABLE list_registers ( +CREATE TABLE scalar_values ( + namespace TEXT NOT NULL, + key TEXT NOT NULL, + seq INTEGER NOT NULL, + value TEXT NOT NULL, + PRIMARY KEY (namespace, key) +) WITHOUT ROWID; + +CREATE TABLE list_values ( namespace TEXT NOT NULL, key TEXT NOT NULL, seq INTEGER NOT NULL, @@ -446,182 +516,194 @@ CREATE TABLE list_registers ( ) WITHOUT ROWID; ``` -Operations: +An implementation may retain an already-applied physical table name while exposing this logical API; do not rewrite an applied migration solely for terminology. Adding list storage uses a new migration when required by the backend's existing schema history. + +List operations: ```sql -INSERT INTO list_registers(namespace, key, seq, value) -VALUES (?, ?, ?, ?); +INSERT INTO list_values(namespace, key, seq, value) VALUES (?, ?, ?, ?); -SELECT seq, value -FROM list_registers +SELECT seq, value FROM list_values WHERE namespace = ? AND key = ? AND seq > ? -ORDER BY seq ASC -LIMIT ?; +ORDER BY seq ASC LIMIT ?; -SELECT seq, value -FROM list_registers +SELECT seq, value FROM list_values WHERE namespace = ? AND key = ? AND seq < ? -ORDER BY seq DESC -LIMIT ?; +ORDER BY seq DESC LIMIT ?; -DELETE FROM list_registers -WHERE namespace = ? AND key = ?; +DELETE FROM list_values WHERE namespace = ? AND key = ?; ``` -For a missing cursor, omit the sequence predicate. All writes run in the existing `BEGIN IMMEDIATE` transaction under the writer lease. No marker table, counter row, connection, or new lock order is needed. - -Assert with `EXPLAIN QUERY PLAN` that paging uses the primary key and no temporary sort. +For a missing cursor, omit the sequence predicate. Every write participates in the existing `BEGIN IMMEDIATE` transaction under the writer lease. Assert with `EXPLAIN QUERY PLAN` that paging uses the primary key and no temporary sort. ## JSONL backend -Scalar records retain their current persisted namespace string. Typed tokens are an API change, not a JSONL format rewrite. - -List records use the token ID as `namespace`: +Logical records carry the bound address's physical components: ```jsonl -{"kind":"list-register","op":"append","seq":41,"namespace":"pending.assistant_frame","key":"O:R","value":{"type":"text_delta","contentIndex":0,"delta":"hi"}} -{"kind":"list-register","op":"delete","seq":52,"namespace":"pending.assistant_frame","key":"O:R"} +{"kind":"list","op":"append","seq":41,"namespace":"pi.pending.assistant_frame","key":"O:R","value":{"type":"text_delta","contentIndex":0,"delta":"hi"}} +{"kind":"list","op":"delete","seq":52,"namespace":"pi.pending.assistant_frame","key":"O:R"} ``` -Replay folds records into the Memory list map: +Scalar records use `kind:"value"` with `op:"set"|"delete"`. An implementation may retain an old physical record spelling while decoding it into the new logical API when compatibility requires it. -- append adds `{ seq, value }`; -- delete removes the complete list key. +Replay folds records into the Memory state: + +- scalar set replaces the current address; +- scalar delete removes it; +- list append adds `{ seq, value }`; +- list delete removes the complete list. A transaction remains one physical JSONL line, using an array for multiple writes. Torn-tail handling therefore remains atomic without new framing. ### Snapshot compaction -Compaction preserves list cursors. Write every surviving element as an append record carrying its original `seq`, merged in sequence order with surviving entries, scalar registers, and usage rows. +Compaction writes every surviving list element with its original `seq`, merged in sequence order with surviving entries, scalar values, and usage rows. Do not collapse one live list into a synthetic element or assign new sequence numbers; either change breaks cursors and backend equivalence. -Do not collapse a live list into one synthetic append or assign new sequence numbers. Either changes cursors and backend ordering. - -Deleted lists produce no snapshot records. Preserve the storage sequence high-water mark through the existing mechanism so dropping the latest delete cannot permit sequence reuse. +Deleted lists produce no snapshot records. Preserve the sequence high-water mark so dropping the latest delete cannot permit sequence reuse. ## Forks and rewrites -Fork and precise-rewrite code decides policy per concrete token, using token identity in source code and the stable ID in storage. +Fork and precise-rewrite code decides policy per concrete address grammar: -Operation-owned tokens are not copied into an idle fork: +- operation-owned `pi.op.*` scalar values are not copied into an idle fork; +- `pi.pending.entry`, `pi.pending.tool_output`, and `pi.pending.assistant_frame` values/lists are not copied; +- lane and semantic session values follow their existing scope rules; +- application-defined values/lists require an explicit policy when their consuming feature is added. -- `operationMeta` and other `op.*` scalar tokens; -- `pendingEntry` and `pendingToolOutput`; -- `pendingAssistantFrames`. - -Application-defined persistent tokens may define a different policy when their consuming feature is added. - -A precise rewrite retaining a list element preserves its sequence cursor unless the rewrite explicitly remaps the entire destination sequence space. +A precise rewrite retaining list elements preserves their `seq` values unless it explicitly remaps the entire destination sequence space. ## Schema evolution -A token's persisted ID and kind are static for one storage version. +A bound address's namespace, key grammar, kind, and value type are durable schema: -- changing an ID requires an explicit old-ID to new-ID migration; -- changing scalar to list or list to scalar requires an explicit migration; -- storage never infers or coerces a kind change from observed records; -- changing only the TypeScript value type requires a total value migration when old stored values are not already valid under the new type. +- changing namespace or key grammar requires explicit address migration; +- changing scalar to list or list to scalar requires explicit migration; +- storage never infers or coerces kind from observed records; +- changing TypeScript value shape requires total value migration when old stored values are incompatible; +- a list migration pages elements in sequence order and either maps them while preserving `seq` or deletes the complete list; +- a migration must not load an unbounded logical list at once. -A list migration pages current elements in sequence order and either maps values while preserving `seq` or deletes the whole key/token. It must not load an unbounded logical list at once. +Adding generic list storage is a backend schema change. Constructing a new application address with no persisted value requires no migration. -Adding the generic SQLite list table is a storage schema migration. Adding a new token with no stored values does not rewrite existing sessions. +## Instrumentation and telemetry -## Instrumentation +The instrumented storage decorator exposes the address-based read API and records committed erased writes in exact transaction order. -The instrumented storage decorator exposes token-based scalar/list reads and records committed erased writes in exact transaction order. +Telemetry session-write item kinds distinguish scalar-value writes from list writes. Namespace/key names may be attributes when the telemetry schema permits them, but values, assistant frames, prompts, and tool output never enter telemetry. -Telemetry session-write item kinds include `list-register` explicitly; list appends/deletes are not reported as scalar writes. Token IDs may be attributes, but values and assistant-frame contents never enter telemetry. - -Append-path tests prove no `readList` call occurred before commit. +Append-path tests prove that no `readList` call occurs before append commit. Frame-persistence promises always receive the harness fault observer, even when an earlier promise is no longer the latest settlement-order reference. ## Invariants -1. Every token has one stable persisted ID and one static kind in a storage version. -2. Built-in token IDs are unique. -3. Token-based reads and helper-constructed writes preserve the token's value type. -4. A scalar helper cannot target a list token; a list helper cannot target a scalar token. -5. Every list element is immutable and addressed by its globally unique write `seq`. -6. Elements under one list key are returned in sequence order on every backend. -7. Append performs no read of the target list. -8. Scalar/list writes are atomic with entries and usage in the same transaction. -9. Whole-key list delete leaves no elements under that key. -10. Missing and empty lists both read as `[]`. -11. Scalar restore and validation never depend on list contents. -12. JSONL compaction preserves surviving element sequences. -13. Deleting operation-owned scalar and list state at terminal cleanup leaves no recoverable partial assistant content. +1. One bound address has one stable namespace/key/kind and one trusted value type in a storage version. +2. Address object identity has no durable meaning. +3. Namespace `pi` and every `pi.*` are reserved by contract; every built-in namespace starts with `pi.`, and application use is a trusted-programming defect. +4. Exactly five built-in prefix constructors encapsulate lane inventory and operation cleanup grammar; their results are consumed only by namespace-scoped `scanValues()`. +5. Scalar and list addresses cannot occupy the same physical location. +6. Typed reads and helper-constructed writes preserve `T`. +7. Scalar helpers reject list addresses; list helpers reject scalar addresses. +8. A Session/Storage operation never requires a second key after address construction. +9. Every list element is immutable and carries its globally unique committed write `seq`. +10. Elements at one list address are returned in sequence order on every backend. +11. Append performs no read of the target list. +12. Scalar/list writes are atomic with entries and usage in the same transaction. +13. Whole-list delete leaves no elements at that address. +14. Missing and empty lists both read as `[]`. +15. Base restore depends only on required scalar state and never enumerates auxiliary lists. +16. Auxiliary lists never establish effect completion or select a restart state. +17. JSONL compaction preserves surviving element sequences. +18. Terminal cleanup leaves no operation-owned scalar values or lists. ## Required tests -### Token typing and identity +### Address typing and identity -- scalar reads infer the token's value type; -- list reads infer the token's element type; -- `setRegister` rejects an incompatible value at compile time; +- `value()` and `list()` preserve their declared `T` invariantly; +- scalar reads infer the bound address's value type; +- list reads infer its element type; +- `setValue` rejects an incompatible value at compile time; - `appendList` rejects an incompatible element at compile time; -- scalar helpers reject list tokens and list helpers reject scalar tokens; -- independently defined application tokens work without declaration merging; -- built-in IDs are non-empty, separator-free, unique, and have fixed kinds; -- token IDs round-trip through Memory, JSONL, and SQLite; -- two differently typed definitions using one persisted ID are documented/tested as a programming defect rather than supported aliasing. +- scalar helpers reject list addresses and list helpers reject scalar addresses; +- independently constructed equal addresses access the same durable location; +- incompatible definitions of one physical address are documented/tested as a programming defect; +- empty keys work, while empty namespaces and separator-containing components reject; +- core and application code use the same `value()` and `list()` constructors, with no private constructor, privilege token, registry, or catalog; +- built-in address constructors produce exact `pi.lane.*`, `pi.op.*`, `pi.pending.*`, `pi.session.name`, and `pi.entry.label` namespace/key/kind triples; +- every built-in namespace starts with `pi.`, while application fixtures use non-reserved namespaces; +- `laneLeafInventoryPrefix()` binds the empty-key `pi.lane.leaf` inventory prefix and is used only to enumerate lanes through `scanValues`; +- tool-args prefixes cover one operation and optionally one step, tool-memo prefixes cover one operation and optionally one invocation, preparation and tool-output prefixes cover exactly one operation; +- each prefix constructor result is used only by `scanValues`, and no inventory or cleanup call constructs a raw reserved namespace; +- application addresses work without declaration merging or core catalogs; +- no Storage or Session operation accepts an additional key argument. ### Scalar regression -- existing scalar set/get/list/delete/recreate behavior is unchanged; -- replacement retains only the latest logical value; -- token-based helpers preserve mixed transaction write order; -- existing scalar JSONL and SQLite files open without a format rewrite. +- set/get/delete/recreate behavior is unchanged; +- replacement retains only the latest logical value and latest set `seq`; +- typed write helpers preserve mixed transaction order; +- prefix scans interpret the bound address key as a prefix and remain namespace-scoped; +- existing scalar JSONL/SQLite data opens through the compatibility policy chosen by those backends. ### List conformance Extend the shared backend conformance suite: - append one element and page it; -- multiple appends to one key in one transaction; -- appends separated by unrelated writes preserve per-key order; +- multiple appends to one address in one transaction; +- appends separated by unrelated writes preserve per-list order; +- every element receives its own global write `seq`; - ascending and descending exclusive cursors; - default, explicit, invalid, and capped limits; -- absent key returns `[]`; -- whole-key delete and delete of absent key; +- absent list returns `[]`; +- whole-list delete and delete of absent list; - delete followed by append in one transaction; - rollback when a later write is invalid; - atomic list + entry + usage + scalar transaction; -- identical pages and sequence cursors on Memory, JSONL, and SQLite; +- identical pages and cursors on Memory, JSONL, and SQLite; - JSONL torn multi-write transaction exposes no list element; -- JSONL replay and snapshot compaction preserve cursors; -- SQLite paging query plans use the primary key; -- append path performs no list read; -- restore validation succeeds without reading lists, followed by bounded hydration; -- close rejects later list reads and honors already-admitted list commits. +- JSONL replay and compaction preserve cursors; +- SQLite paging uses the primary key without temporary sorting; +- append performs no list read; +- base restore constructs trusted scalar projection without list reads, followed by bounded consumption-time hydration; +- close rejects later reads and honors already-admitted commits. + +### Application surface + +- an application-wide scalar value requires no extra key at get/set; +- an application-wide list requires no extra key at read/append; +- an application can construct dynamic per-workspace addresses explicitly; +- Storage and Session accept the same address objects and infer the same types; +- direct Session writes serialize and commit once; +- explicit `Session.mutate()` can atomically combine typed value/list writes with entries and usage. ### Assistant-frame integration -- compact frames append under the exact effect-pending response key; -- raw events with growing `partial` snapshots are never persisted; -- frame pages reduce to the same partial message as uninterrupted streaming; -- missing/empty frame list restores as no durable partial; -- final assistant settlement atomically deletes the frame list; -- unknown-effect recovery reads only the bounded list named by scalar state; -- external finalization deletes the operation-owned frame list; -- idle forks contain no assistant-frame list; -- backend byte-growth tests show append-linear rather than repeated-snapshot growth. +- every converted non-terminal frame appends under the exact bound effect-pending response address; +- terminal `done`/`error` events append nothing; +- appends enqueue synchronously without provider backpressure; +- every frame-write promise has an observed fault path; +- only the latest promise reference is retained for settlement ordering; +- awaiting the latest promise implies every earlier append completed; +- reduced pages reconstruct the same partial message as uninterrupted streaming; +- missing list restores as no durable partial; +- final/synthetic settlement atomically deletes the frame list; +- unknown-effect recovery reads only the bounded list derived from current scalar state; +- external finalization deletes the operation-owned list; +- idle forks contain no frame list; +- backend byte growth is append-linear rather than repeated-snapshot growth. ## Implementation map -Primary files expected to change: +Expected primary changes: -- new `packages/agent/src/harness/session/registers.ts` for token types, factories, helpers, and built-in constants; -- `packages/agent/src/harness/session/types.ts` to remove global value maps and expose token-based reader methods; -- `packages/agent/src/harness/session/storage-state.ts`; -- `packages/agent/src/harness/session/memory.ts`; -- `packages/agent/src/harness/session/session.ts`; -- `packages/agent/src/harness/session/jsonl/codec.ts`; -- `packages/agent/src/harness/session/jsonl/storage.ts`; -- `packages/agent/src/harness/session/testing/storage-decorator.ts`; -- `packages/agent/src/harness/session/testing/instrumented-storage.ts`; -- `packages/agent/src/harness/session/testing/conformance/storage.ts`; -- `packages/session-backends/sqlite-node/src/sqlite/migrations.ts`; -- a new SQLite migration for `list_registers`; -- `packages/session-backends/sqlite-node/src/sqlite/session.ts`; -- `packages/session-backends/sqlite-node/src/sqlite/storage.ts`; -- focused SQLite list paging and query-plan tests. +- replace `session/registers.ts` with `packages/agent/src/harness/session/values.ts` containing addresses, constructors, typed write helpers, and built-in address constructors; +- remove `RegisterValues`, namespace unions, register token types, and raw namespace/key read signatures from `session/types.ts`; +- expose `ValueReader` through Storage, SessionReader, SessionMutator, Session, and SessionTree; +- add direct application scalar/list methods to SessionTree using bound addresses; +- update Memory state, JSONL codec/storage, snapshots, fork/rewrite code, instrumentation, and conformance suites; +- add SQLite list storage and address-based adapters without rewriting applied migrations solely for terminology; +- update assistant execution, deferred polling, recovery, snapshots, and cleanup to use `pendingAssistantFrames(operationId, responseEntryId)`; +- update telemetry schema sources and regenerate `telemetry-schema.md`; do not edit that generated file manually. -After implementation, update `harness.md` to use imported tokens rather than namespace-string/type-map examples and add the concrete assistant-frame lifecycle. +`assistant-durability.md` specifies the consuming lifecycle. `harness.md` must replace raw namespace/key examples and global register maps with this bound-address model before implementation. diff --git a/packages/agent/docs/runtime2.md b/packages/agent/docs/runtime2.md new file mode 100644 index 000000000..c89bec766 --- /dev/null +++ b/packages/agent/docs/runtime2.md @@ -0,0 +1,792 @@ +# AgentHarness `runtime2/` implementation status + +## Goal + +Build a second AgentHarness runtime from the bottom up under: + +```text +packages/agent/src/harness/runtime2/ +``` + +It must eventually export: + +```ts +export async function createAgentHarness< + TContext extends object | undefined = object | undefined, +>( + options: AgentHarnessOptions, +): Promise<{ + harness: AgentHarness; + suspended: SuspendedOperation[]; +}>; +``` + +Runtime2 replaces runtime1 through [WP00](work-packages/00-runtime1-removal.md), before further execution behavior is added. [WP01](work-packages/01-bound-values-lists.md) then converts the retained Session/storage surface across all existing backends. Do not release while the public runtime2 execution path is incomplete. + +`harness.md` Part 8 and concrete `work-packages/*.md` files are the only active package plan. The R2.x sections below record historical checkpoints and design evidence; they are not executable work packages and must not override a handoff. + +## Current baseline + +Current pushed `dev` head: + +```text +03f576f54 feat(agent): inspect restored runtime2 operations +├── 5030a5f53 feat(agent): add runtime2 harness config +├── d5d656d4e feat(agent): add runtime2 lane creation +├── 0697589ac fix(agent): tighten runtime2 harness boundaries +└── earlier runtime2 restore/facade/close commits +``` + +Runtime2 currently consists only of concrete `Harness` and `Lane` implementations plus restore/types. There is no separate runtime/controller abstraction. The oversized R2.4 acceptance/drive attempt was deliberately reverted in full; no TypeScript or test changes from that attempt remain. The focused runtime2 suite is back to the committed 44 tests across: + +- `runtime2/restore.test.ts` +- `runtime2/lane.test.ts` +- `runtime2/harness.test.ts` + +Current invariants: + +- one restore constructs authoritative owned `LaneState`; +- all lane-control writes route through `Lane.command()` and `Session.mutate()`; +- command planners make state-dependent decisions under the lane line and return explicit `commit`, `return`, or `reject` outcomes; +- commit commands publish memory after commit and synchronously materialize from `CommitResult`; +- committed values and owned state are immutable snapshots updated only by replacement; +- providers, tools, hooks, timers, events, and task waits stay outside the lane line; +- the Lane-backed `sessionTree` routes leaf/branch reads and idle/run appends through owned state; +- close/fault seal lanes before queued work can proceed, while admitted successful commits still publish; +- harness creation never writes durable suspension; `suspension` is process-local inspection metadata and clears when committed ownership moves to another operation or idle; +- durable control `running` means open/permitted; public `running` will mean a real process-local drive owns an actual continuation; +- do not install a process owner, join promise, or `running` status around an effect-free classification shell; ownership lands with the first real drive procedure; +- missing identity payloads are `{ tools: string[]; model?: string }`; one lane configuration can reference only one model; +- runtime1 remains production-selected only until WP00 switches the factory and deletes it. + +Latest committed validation before the reverted experiment: 44 focused runtime2 tests, 156 focused tests, full TypeScript, `git diff --check`, and `npm run check` passed. Revalidate after implementing the redesigned contract below. + +## Slice completeness rule + +A slice must close the entire named concern at parity, not only its immediately callable methods. Before editing, inventory all related constructor inputs, owned state, defaults, validation, public accessors, events, lifecycle behavior, recovery implications, and tests. Do not defer adjacent behavior merely because no current procedure consumes it yet. If scope must be deferred, name that boundary before implementation and get explicit agreement. + +## Reading policy + +`packages/agent/docs/harness.md` is the normative source. Until their designs are folded into it, `registers.md`, `tool-durability.md`, and `assistant-durability.md` are binding handoffs for bound typed value/list addresses, tool outcomes/checkpoints/memos, and assistant frames. Runtime1 is implementation inspiration only: do not preserve its abstractions, internal bookkeeping, or accidental behavior unless the normative contract independently requires them. For each slice, read its listed sections completely. If any handoff or `harness.md` changed since the previous slice, inspect that diff and reread affected sections. + +Read a source or test file completely before editing that file. For dependencies that will not be edited, begin with the relevant exported types or functions and expand only when a concrete question requires it. + +Do **not** pre-read later runtime phases. In particular, R2.0 must not load assistant procedures, tool procedures, or the complete R2–R4 behavioral suites. The purpose of the slices is to keep both implementation and reasoning local. + +This plan is implementation guidance, not source authority. `harness.md` remains normative; correct this file whenever it disagrees with committed behavior or that contract. + +### R2.0 — Restore one lane + +Read: + +- `harness.md`: §§0.2–0.3, 0.7–0.8, 1.3–1.4, 3.1–3.3, and 4.4; +- the lane/configuration/operation definitions in `packages/agent/src/harness/session/types.ts` and built-in address constructors in `session/values.ts`; +- repository/session helpers needed by the focused restore test; +- only the exported shape of the existing `restoreLane()` when comparing terminology. + +Do not read runtime1 procedures or R2–R4 behavior tests. Runtime2 intentionally trusts committed internal values and does not copy runtime1's semantic restore audit. + +### R2.1 — Restore a configured session + +Read: + +- `harness.md`: §§2.3, 2.8, and the lane inventory portions of 4.4; +- completed `runtime2/types.ts`, `runtime2/lane.ts`, and `runtime2/restore.ts`; +- address-based `Session.getValue()`, `Session.scanValues()`, `Session.readList()`, and `Session.mutate()` signatures in `session/types.ts`; +- focused session tests only when a repository helper is needed. + +Do not read runtime1 initialization, provider, assistant, tool, or full harness-facade implementations. + +### R2.1a — Lane owner and transition kernel + +Read: + +- `harness.md`: §§1.4, 3.4, 4.3, 4.7–4.8, and invariants 2, 21–23; +- completed runtime2 restore/session-loader files; +- `packages/agent/src/harness/session/lane-mutations.ts`; +- `SessionMutator`, `Transaction`, and `CommitResult` definitions in `session/types.ts`; +- focused mutation-line and instrumented-storage tests only when needed for test helpers. + +Do not read provider, assistant, tool, or full harness-facade implementations. + +### R2.2 — Main seeding and runtime shell + +Read: + +- `harness.md`: §§2.3, 2.8, 4.4–4.5, and 5.2; +- completed runtime2 restore/session-loader files; +- main-lane creation and configuration-seeding portions of `session/session.ts`; +- only constructor/seeding cases in `agent-harness-runtime.test.ts`. + +Do not read live assistant or tool procedures, and do not reintroduce runtime1's semantic restore audit. + +### R2.3 — In-memory reads, config, lanes, and close + +Read: + +- `harness.md`: §§2.3–2.4, 4.3, 4.7–4.8, 5.1–5.5; +- config, lane registry, facade delegation, close, and fault portions of `runtime/agent-harness-runtime.ts`; +- query/configuration/public-session-view portions of `runtime/lane-runtime.ts`; +- `packages/agent/src/harness/events.ts` and `hooks.ts` only for surfaces this slice instantiates; +- corresponding shell/configuration/close tests in `agent-harness-runtime.test.ts` and session tests. + +Do not read operation execution procedures. + +### R2.4 — Contract reconciliation and atomic run acceptance + +Read completely before editing: + +- this document's “Superseding hook and execution design” and R2.4 section; +- the current diff of `harness.md`, then §§3.1–3.6, 4.1–4.4, 5.1–5.3, 5.5–5.6, build rows R1–R2, and matching invariants/races; these sections still contain superseded hook semantics and must be reconciled first; +- current `runtime2/harness.ts`, `lane.ts`, `types.ts`, `restore.ts`, and all focused runtime2 tests; +- public request/result/event/hook types in `agent-harness.ts`; +- neutral `hooks.ts`, `skills.ts`, `prompt-templates.ts`, bound value/list addresses and transaction helpers, and pending-entry reads; +- runtime1 acceptance only to inventory production/public-contract consequences, never as architecture; +- acceptance and restore cases in `agent-harness-r2.test.ts` and `restore.test.ts`. + +Do not read provider/tool procedures or implement ownership in a no-work shell. Neutral harness imports remain allowed; `runtime2/` must never import `harness/runtime/*`. + +### R2.5 — Minimal no-tool run parity + +Read: + +- `harness.md`: §§3.2, 3.7, 3.12–3.13, 4.1–4.2, 5.5–5.8; +- `registers.md` assistant-frame consumer and `assistant-durability.md` fresh-stream/settlement sections; +- pi-ai's exported assistant-message frame converter/reducer; +- `execution/assistant.ts`; +- checkpoint, ordinary-ready/intent/settlement, and terminal paths in: + - `runtime/run-driver.ts`; + - `runtime/checkpoint-procedure.ts`; + - `runtime/assistant-procedure.ts`; + - `runtime/transitions.ts`; +- `execution-assistant.test.ts`; +- `agent-harness-r2.test.ts`; +- only R2-relevant restore cases in `restore.test.ts`. + +Do not read R3 recovery/retry or R4 tool cases except where they share a type already established in R2.0. + +### R2.6 — Generation recovery and retry parity + +Read: + +- `harness.md`: generation/deferred portions of §§3.2, 3.7, 4.4–4.6, and R3 rows in Parts 8–9; +- `assistant-durability.md` recovery, cancellation, deferred, snapshot, and race sections; +- assistant recovery, retry-wait, missing-identity, and deferred-suspension paths in `runtime/assistant-procedure.ts` and `runtime/run-driver.ts`; +- retry/deferred classifiers in `runtime/transitions.ts`; +- `agent-harness-r3.test.ts`; +- R3-relevant cases in `restore.test.ts`. + +Do not read tool-batch implementation or R4 tests. + +### R2.7 — Tool parity + +Read: + +- `harness.md`: §§3.8, 4.5–4.6 tool portions, 5.6–5.8 tool portions, and the R4 rows in Parts 8–9; +- `tool-durability.md` completely; +- `execution/tools.ts`; +- `runtime/tool-batch-procedure.ts`; +- the tool transition helpers used by that procedure; +- `execution-tools.test.ts`; +- `agent-harness-r4.test.ts`; +- tool-batch cases in `restore.test.ts`. + +Read assistant code only in the slice where assistant settlement creates a tool plan. + +### Expanding a slice's reads + +If implementation exposes a concrete dependency not listed above: + +1. state the specific unresolved question; +2. locate the smallest defining source or test with `rg`; +3. read that file or relevant section; +4. do not recursively read its entire dependency graph; +5. add it to this plan's slice list if future agents will need it. + +## Value/list and durability prerequisites + +Before further assistant/tool runtime slices, implement `registers.md` repo-wide: + +- replace global namespace/value maps and register tokens with `value(namespace, key?)` and `list(namespace, key?)`; +- put direct built-in address constructors in `session/values.ts`, using the universal `value()`/`list()` constructors and the `pi.*` namespaces/key grammars documented by `registers.md`; +- expose `getValue`/`setValue`/`deleteValue`/`scanValues` and `readList`/`appendList`/`deleteList` without a second key argument; core lane inventory and cleanup use only the five exported `*Prefix` constructors as `scanValues` inputs; +- add backend-conformant append/read/whole-list-delete storage; +- add `pendingAssistantFrames(operationId, responseEntryId)` as a `ValueList` constructor. + +Namespace `pi` and every `pi.*` namespace are reserved for built-ins by contract. Core and applications use the same `value()`/`list()` constructors; application misuse is a trusted-programming defect, not a runtime rejection. Runtime2 imports built-in addresses directly; do not add a private constructor, privilege token, bundle, catalog, runtime registry, `ValuePrefix` type, or runtime-specific storage abstraction. Every call receives an already-bound address. Stable physical namespace/key pairs preserve existing durable data; the new list table/records are a storage schema addition. + +R2.5 owns fresh assistant frame append and cleanup. R2.6 owns frame-based generation/deferred recovery and snapshots. R2.7 owns `outcome_ready`, invocation memos, and tool checkpoint values exactly as specified by `tool-durability.md`. + +## Non-goals until parity + +Do not implement or mix in: + +- R5 inbox/configuration/write behavior (`pendingNextRun` placement during R2.4 acceptance is the explicit exception; enqueue/cancel/consume behavior remains R5); +- checkpoint hook; +- caller-reserved queue IDs; +- explicit owner-requested failure; +- application-state semantics beyond migration to application-defined bound addresses; +- durable state-shape normalization unrelated to the specified tool lifecycle; +- summary/navigation slices beyond currently implemented R1–R4 behavior; +- structural-summary partial-frame persistence; +- compatibility shims between runtime1 and runtime2 internals. + +## Core principle + +Each lane has one in-memory owner while the harness instance exists. + +```text +create harness +→ restore each lane once +→ lane owns current leaf/config/queues/operation/last-result in memory +→ all lane-affecting commands serialize through that owner +→ compute private next state + exact transaction +→ commit +→ replace in-memory state +→ publish observations +``` + +External effects never hold the lane command line: + +```text +lane command: commit intent and publish pending state +→ provider/tool/hook/timer effect outside lane owner +→ lane command: settle against latest in-memory state +``` + +A crash discards memory. The next `createAgentHarness()` restores the committed state again. + +### Commit ordering + +Never mutate the owned state before storage commit succeeds. + +Correct: + +```text +current S0 +→ private candidate S1 +→ commit durable writes for S1 +→ lane.current = S1 +``` + +Incorrect: + +```text +lane.current = S1 +→ commit +→ attempt rollback on failure +``` + +The lane command remains serialized through commit and the memory replacement. A queued command sees either S0 before the transition or S1 after it, never an uncommitted state. + +## Target directory rule + +Runtime2 source must never import `../runtime/*.ts`. Copy the small primitive it needs into `runtime2/`, or move genuinely shared code to a neutral harness module. Runtime2 owns `Config`; do not reintroduce `RuntimeSettings`, a Runtime controller, or settings-line terminology. + +Create files only as a slice needs them. Do not scaffold the final directory with empty modules. + +Likely eventual shape—not a mandate: + +```text +runtime2/ + index.ts eventual stable runtime2 export + types.ts lane/config types + state.ts exhaustive refinements/invariants + lane.ts owner + commit-then-publish kernel + restore.ts one-time durable restore into lane memory + harness.ts facade, registries, config, close + operation.ts drive ownership/fencing + run.ts direct phase dispatch + assistant.ts intent/effect/settlement/recovery + tools.ts batch intent/effect/outcome/recovery + checkpoint.ts checkpoint/terminal procedure +``` + +Merge modules when that is easier to read. Do not create helper files solely to reduce line counts. + +## Public/test switching strategy + +Runtime2 currently exports `createAgentHarness()` from `runtime2/harness.ts`. Add `runtime2/index.ts` only when constructor aliasing/test-selection work begins, then make it the stable runtime2 export. Until then, unimplemented runtime2 methods use the local `SliceNotImplemented` frontier from `runtime2/types.ts`. + +Do not change the production import in: + +```text +packages/agent/src/harness/agent-harness.ts +``` + +until parity. + +### Preferred test selection + +Add a runtime2 Vitest config or test alias that redirects only: + +```text +./runtime/agent-harness-runtime.ts +``` + +as imported by `agent-harness.ts`, to: + +```text +./runtime2/harness.ts +``` + +This allows existing tests to continue importing public `AgentHarness` unchanged. First prove the alias with a tiny constructor-selection test; do not trust an unverified Vite alias. + +If exact aliasing is unreliable, use a test-only constructor: + +```ts +import type { AgentHarnessConstructor } from "../../src/harness/agent-harness.ts"; +import { createAgentHarness } from "../../src/harness/runtime2/harness.ts"; + +const Runtime2AgentHarness: AgentHarnessConstructor = { + create: createAgentHarness, +}; +``` + +Then parameterize only the scenarios runtime2 currently supports. Do not add a mutable production-global runtime selector. + +When constructor aliasing begins, add `runtime2/index.ts`, point tests at that stable export, and verify selection with a tiny constructor test. At final parity, switch the one production import to `runtime2/index.ts`, run every existing test unchanged, then remove runtime1 in a separate reviewed change. + +## Size and readability gates + +Measure after every slice: + +```bash +find packages/agent/src/harness/runtime2 -name '*.ts' -print0 | xargs -0 wc -l +``` + +Targets are diagnostic, not an excuse for compressed code: + +- lane transition kernel: target ≤ 250 lines; +- restore projection: target ≤ 250 lines; +- facade/settings/close at R1 parity: target ≤ 450 lines; +- no procedure file above 700 lines without explicit review; +- R2 minimal no-tool parity: target total ≤ 2,000 lines; +- R3 recovery/retry parity: target total ≤ 2,800 lines; +- R4 tool parity: target total ≤ 3,400 lines. + +Stop and review if: + +- runtime2 exceeds a gate by more than 20%; +- `restoreLane()` appears in an ordinary live transition; +- the same ownership/phase check appears in three procedures; +- a generic context object accumulates unrelated runtime methods; +- a module cannot explain its protocol in one sentence; +- code size is moving toward runtime1 without additional behavior. + +Do not reduce lines by hiding transitions behind a generic reducer or callback framework. + +# Historical runtime2 checkpoints + +These sections explain how the current runtime2 shell was produced and preserve later design discoveries. They are not the active sequence. Execute only a concrete work-package handoff linked from `harness.md` Part 8. + +## R2.0 — Restore one lane + +### Goal + +Load the current committed control values for one configured lane into a small process-local `Lane` without starting work or interpreting the state. + +### Files + +Create only: + +```text +runtime2/types.ts +runtime2/lane.ts +runtime2/restore.ts +packages/agent/test/harness/runtime2/restore.test.ts +``` + +### Contract + +`LaneState` is the direct in-memory projection of: + +```ts +laneLeaf(lane) +laneConfig(lane) +laneState(lane) +laneLastResult(lane) // when present +operationMeta(operationId) // when currentOperationId is non-null +operationState(operationId) // when currentOperationId is non-null +``` + +`Lane` owns its name and one `LaneState`. It does not implement `AgentLane` yet. + +Restore trusts committed internal values. It checks only that values required to construct the projection exist. It does not: + +- validate referenced entries or semantic phase relationships; +- hydrate entries, pending payloads, preparations, or tool arguments; +- infer state from the transcript; +- enumerate lanes or seed main configuration; +- create hooks, events, effects, timers, drives, or public facade stubs. + +Exact durable payload reads remain available to the later procedure that needs them. Live operation/phase/effect checks remain required when asynchronous work settles; they are concurrency fences, not restore validation. + +### Tests + +- idle configured lane; +- optional latest result; +- open operation metadata and state; +- no interpretation of referenced payload ids; +- missing required lane value; +- missing current operation metadata/state. + +### Stop condition + +Report source and test line counts, focused test result, TypeScript result, and whether any additional runtime2 abstraction was required. Do not add a lane mutation kernel or session-wide loader yet. + +## R2.1 — Restore a configured session + +### Goal + +Load every configured lane in one existing session into memory without starting work. + +### Contract + +Add one small session loader: + +```ts +restoreSession(session: Session): Promise> +``` + +It calls `scanValues(laneLeafInventoryPrefix())` for the trusted lane inventory, requires `main`, and calls runtime2's one-lane restore exactly once for each listed lane under that lane's session mutation line. It does not seed configuration, scan other namespaces for orphaned values/lists, validate semantic state, or construct a public harness. + +### Required tests + +- configured main-only session; +- multiple configured lanes; +- open operations remain loaded and inert; +- missing `main` rejects; +- no hooks, providers, tools, timers, or writes; +- each lane is restored exactly once. + +### Stop condition + +Report code size and focused tests. Do not add main seeding, a public factory, or lane mutation behavior. + +## R2.1a — Lane owner and transition kernel + +### Goal + +Prove commit-then-publish on the real session mutation line without AgentHarness, providers, tools, hooks, events, or lifecycle machinery. + +### Kernel contract + +`Lane.transition()` accepts a synchronous planner. Inside `Session.mutate()` it: + +1. plans a transaction, candidate state, and result from current memory; +2. commits the transaction; +3. replaces lane memory only after commit succeeds; +4. returns the planned result before releasing the lane line. + +External async effects cannot be passed through the planner. Storage-assigned entry metadata and observation publication are added only when a concrete transition requires them. + +### Required tests + +1. Memory remains unchanged while commit is pending. +2. Commit failure leaves memory and storage unchanged. +3. A queued transition is not planned until the prior commit publishes memory, then sees that memory. + +### Stop condition + +Report line count and focused tests. Do not add lifecycle faulting, close, observations, or execution behavior. + +## R2.2 — Main seeding and runtime shell + +### Goal + +Seed an unconfigured `main`, call `restoreSession()`, and place the resulting lane map under one runtime2 harness shell without starting work. + +### Behavior + +- validate only constructor options needed to build the immutable lane seed; +- commit the seed when `main` has no configuration; +- call runtime2's configured-session loader; +- create no drive task/provider/tool/hook/timer; +- expose the exact `createAgentHarness()` signature only when the shell can satisfy it without casts or a speculative abstraction framework. + +### Required tests + +- fresh session/main seeding; +- configured lanes are never overwritten by the seed; +- idle and open multi-lane restore; +- last result loaded once; +- no effects during create; +- no full restore after construction. + +### Stop condition + +The runtime2 shell owns the restored lane map. Do not implement operation acceptance or execution. + +## R2.3 — In-memory reads, config, lanes, and close + +### Goal + +Make non-execution surfaces use owned memory. + +### Completed + +- `Harness` and `Lane` are the concrete `AgentHarness` and `AgentLane` implementations; there is no separate runtime/controller layer. +- `Lane.command()` is the central effect-free serialized primitive with explicit `commit`, `return`, and `reject` outcomes. +- Commit commands support bounded reads, one durable commit, memory publication, and synchronous `CommitResult` materialization. +- Lane configuration getters/setters derive queued updates from current owned memory. +- Lane lookup/listing and leaf/last-result reads use owned memory. +- The `sessionTree` facade routes leaf/branch reads and idle/run appends through `Lane`; application values and lists remain Session-owned. +- Close and fault synchronously seal every lane; close drains admitted session work without durable cancellation. +- `Harness.createLane()` calls `Session.createLane()` directly with the captured seed, publishes a known idle candidate after commit, maps expected Session errors, and emits `lane_created` after map publication. +- Harness-global config is one replace-only process-local `Config` snapshot. It captures mutable registries/policies plus immutable execution inputs (`systemPrompt`, `toolContext`, provider conversion, projectors, telemetry, drive mode, and tool-execution mode). Setters validate, publish synchronously, then emit `config_update`; no second mutation line exists. +- Shared config defaults, validators, and missing-identity classification live in `packages/agent/src/harness/config.ts` and are used by both implementations. +- Restored suspension descriptors hydrate run prompts and deferred handles once, classify captured missing identities, and stay owned by their lanes. +- `Lane.inspectExecution()` and `Harness.lanes()` read the same owned state without storage access. +- Application-defined values and lists pass directly through the Session facade while lanes are idle or active. +- Focused tests cover ordering, commit failure, fault publication, close, no-commit promise boxing, expected rejection, appends, restore, lane creation, config, inspection, suspension classification, and application values and lists. + +### Remaining R2.3 work + +None. Watch surfaces have no runtime1 implementation and remain a later separate concern. + +### Session boundary decision + +Harness-owned lane/tree/operation writes route through `Lane.command()`. Application value/list reads and writes remain direct Session operations because they do not mutate lane control state. + +## Superseding hook and execution design + +The earlier R2.4 plan above was wrong. It put `before_run` inside acceptance, which forced an admission reservation, waiting appends, provisional ownership, two acceptance commands, and close/fault races before any durable operation existed. The rejected implementation then rebuilt runtime1's context/procedure shape through `OperationHost`, `OperationLocalState`, arbitration unions, and a fake `ActiveDrive` around a command that had no work. All of that code was reverted. + +### Contract decisions + +1. **`accept()` is one atomic lane command.** It normalizes and validates state-independent input outside the line, then under `Lane.command()` checks `state.operation`, captures current `pendingNextRun`, preflights identities, and commits caller/captured entries, leaf, `pi.op.meta`, `pi.op.state`, and `pi.lane.state`. There is no hook, provider, timer, process reservation, or process owner in acceptance. Concurrent accepts serialize naturally; the loser sees the committed operation and returns `LaneBusy`. An empty caller request with no captured entry still rejects; a later hook cannot rescue it. +2. **A run is accepted in durable `{ kind: "starting" }` phase.** This payload-free phase is the only applied-once marker for initial durable extension context. It replaces acceptance-time hook output and avoids another stored value/flag. +3. **`before_run` moves to the driver and returns only durable messages.** On the first real drive of `starting`, after ownership and deadline/cancellation checks, run the hook outside the line. Its consuming command appends returned messages and commits `starting → checkpoint` atomically. Crash before that commit may rerun the hook; crash after it cannot. Hook invocation is at-least-once, while its core-interpreted durable output is applied once. Injected messages become a stable conversation prefix and preserve provider KV-cache ordering. +4. **Add `before_drive` as a pass prerequisite.** It runs once per newly installed real drive owner before recovery or ordinary effects; same-operation joiners do not rerun it. It runs again after yield, retry wait, deferred/missing-identity suspension, and process restart. Under `cancel_requested`, reconciliation wins and ordinary prerequisites do not run. A prerequisite failure rejects that drive pass without faulting the harness or writing durable progress; close/abort must interrupt or win through the normal gate. +5. **Delete `before_resume`.** Extensions should not depend on process-origin categories. Durable phase decides dispatch/recovery. There is no public or internal `fresh | continue | restore` taxonomy. An unowned `effect_pending` phase is orphaned by construction once real owners span every admitted effect. +6. **Delete `resumeData` and stable hook-ID routing.** Extension-private durable state belongs in extension-owned bound value/list addresses or audited custom entries keyed by lane/operation ID. Such writes are separate transactions and may orphan/replay; extensions own idempotency and cleanup, matching the exactly-once non-goal. Do not add transactional hook value writes until extension namespace, lane/session scope, version, and fork policy are designed explicitly. +7. **Delete hook-produced durable `systemPromptOverride`.** Remove it from `before_run`, `BeforeResumePrepared`, and `OperationMeta`. Widen request-time `transform_context` to receive/return `{ messages, systemPrompt }`. Repeating the same effective system prompt preserves its cache key; changing it or earlier messages invalidates the prefix by design. Use `before_run` for durable message injection, not per-request `transform_context`. +8. **Install a process owner only around real asynchronous work.** `before_drive`/`before_run` are real work, so the slice that implements them may install one simple Lane-owned work promise before invoking them. The model is direct: `accept()` checks durable `state.operation`; `drive()` joins the matching promise or installs one promise and starts actual work after the line. Do not add a controller, host adapter, local-state wrapper, action interpreter, or fake frontier pass. + +### Hook API target + +```ts +before_run: { + event: { prompt: AgentMessage[]; resources: Resources }; + result: { messages?: AgentMessage[] } | undefined; +}; +before_drive: { + event: { operation: "run" | "compaction" | "navigation" }; + result: void; +}; +transform_context: { + event: { messages: AgentMessage[]; systemPrompt: string }; + result: { messages?: AgentMessage[]; systemPrompt?: string } | undefined; +}; +``` + +`HookInvocation` continues to add `lane` and `runId`. `before_run` is an at-least-once durable-settlement hook. `before_drive` is a fail-closed pass prerequisite. `transform_context` is request-local and replayed per provider request. Stable registration IDs are optional metadata only, not a durability protocol. + +### Durable transitions + +```text +idle accept +→ TX[ captured nextRun entries, caller entries, pending deletes, + pi.lane.leaf, pi.op.meta(run prompt ids only), + pi.op.state(run starting, running control, captured run settings, empty inbox), + pi.lane.state(currentOperationId, captured ids removed) ] + +starting + running control +→ before_drive +→ before_run +→ TX[ injected message entries, pi.lane.leaf when injected, + pi.op.state(checkpoint need_assistant(false), trigger=newest entry, + skipInboxOnce=true, preserve concurrent inbox/control) ] + +starting + cancel_requested +→ cancellation reconciliation and aborted terminal transaction; + invoke neither before_drive nor before_run +``` + +Restoring `starting` reports the open run with caller prompt only and starts nothing. Public appends after acceptance already see an active run and enter its inbox; there is no pre-acceptance waiting window. + +### Request reconstruction and cache discipline + +DeepSeek Harness's useful precedent is that model-visible injected context becomes durable before dispatch, dynamic context snapshots are appended only when changed, and effective request headers can reconstruct requests. Pi should not copy Cordis or event sourcing, but R2.5 must decide before implementation whether to persist the effective model identity, stream options, system prompt, transformed message ordering, and tool schema identity before provider dispatch, or explicitly require deterministic/idempotent request transforms and accept weaker reconstruction. Do not casually use `transform_context` for timestamps, workspace notices, policy state, or other durable run context; place those through `before_run`. + +### Broader hook audit to resolve before later slices + +- Keep `before_request` for provider-neutral request options. +- Reconsider `before_payload`: provider-specific wire mutation likely belongs in provider-adapter middleware. +- Split tool policy from mutation: a monotonic allow/deny/ask guard cannot be overridden by listener order; an around-tool seam owns timeout/retry/tracing/cache/signal lifetime; a result transform precedes commit; a passive final-result event follows normalization and commit. +- Reconsider `before_tool.args`: rewritten effective arguments currently disappear when operation-owned values are cleaned while the assistant entry retains original arguments, weakening auditability. Either remove rewriting or preserve effective arguments durably. +- Prefer one configured compaction service and one branch-summary service over first-result-wins provider hooks; retain separate veto policy only if needed. +- Replace latest-follow-up-wins `before_run_end` with an explicit `before_finish` continuation contract if multiple handlers can contribute. +- Extension registrations eventually need scoped install/dispose and namespaced values/lists, but do not import a DI/plugin framework. + +### Exactly-once statement + +No external hook is globally exactly once. Core guarantees only that a hook's interpreted output commits before dependent durable progress. A crash before the consuming transaction may repeat the hook. External side effects require extension-owned idempotency keyed by stable operation/invocation IDs. `before_drive` is intentionally repeated per owned pass. + +## R2.4 — Contract reconciliation and atomic run acceptance + +### Required preparation + +The current `harness.md`, shared `HookMap`, `OperationMeta`, runtime1 behavior, and parity tests still describe the superseded pre-acceptance hook/resumeData/systemPromptOverride contract. Reconcile the normative document and public type strategy before implementation. This is a real contract change. Because runtime1 remains production-selected and shares the public types, explicitly decide whether to update runtime1 to the new contract, isolate the experimental runtime2 contract temporarily, or move the production switch earlier. Do not silently preserve compatibility shims. + +### Goal + +Implement the smallest complete concern: the revised public/durable contract plus one-command prompt/skill/template acceptance into `RunPhase.starting`. Do not install drive ownership unless this same slice also implements real `before_drive`/`before_run` work and the `starting → checkpoint` transition. + +### Required behavior + +- prompt, skill, and prompt-template normalization; +- write-nothing `Closed`, `LaneBusy`, missing-identity, pending-assistant, unknown-resource, and empty-input results; +- captured `pendingNextRun` placement/deletion before caller entries; +- exact acceptance transaction above; +- owned memory publication after commit; +- `run_start`, message lifecycle, `entry_added`, and trailing `queue_update` after publication; +- accepted-undriven and accept→reopen `starting` inspection/restore; +- no hook, breakpoint, process reservation, process owner, provider, tool, timer, or runtime1 import. + +### Required tests + +- exact transaction and durable-owned equality; +- concurrent accepts serialize and only one commits; +- append-vs-accept has only the natural two orders: append first becomes an earlier tree entry; accept first routes the append into the run inbox; +- config-vs-accept snapshots one complete order; +- all expected rejections write nothing; +- commit failure leaves memory idle and faults; +- admitted commit wins close and publishes; +- close-first returns `Closed`; +- pendingNextRun ordering and events; +- reopen restores `starting` with caller prompt and starts no work. + +Stop after this concern unless the user explicitly approves combining it with the real starting-phase drive slice. + +## R2.5 — Minimal no-tool run parity + +### Goal + +Implement the first real drive owner together with all work it owns; never land an owner-only shell. + +Implement direct flow: + +```text +drive arbitration installs one Lane-owned promise under the lane line +→ same-id drives join; stale ids fence +→ cancellation/deadline precheck +→ before_drive prerequisite outside the line +→ starting: before_run outside the line +→ commit injected messages + starting → checkpoint +→ generation.ready commit +→ request-local context/system-prompt transform +→ before_request/input preparation +→ effect_pending intent commit +→ provider effect; synchronously enqueue each converted assistant frame without awaiting storage +→ await latest frame-write promise before after_response +→ assistant settlement commit atomically deleting the frame list +→ finish terminal commit +→ remove owner only after final publication +``` + +The owner is one ordinary Lane field, not a runtime/controller/context abstraction. Close/fault rejects it; effects and hooks remain outside `Lane.command()`. Reuse execution-only assistant mechanics where useful, never runtime1 orchestration. Before provider work, settle the request-reconstruction decision described above. + +### Required tests + +Run the complete R2 suite against runtime2, including exact writes, events, telemetry, breakpoints, close boundaries, convenience/primitive equivalence, one frame append per convertible event, no per-frame storage await, latest-promise settlement fencing, and atomic frame-list deletion. + +### Size review + +Target total runtime2 implementation ≤ 2,000 lines. If larger, stop before R3 and identify why rather than introducing context/framework abstractions. + +## R2.6 — Generation recovery and retry parity + +### Goal + +Pass committed R3 behavior. + +Implement: + +- orphaned assistant `effect_pending` recovery when no live owner exists, reducing committed frames into the synthetic partial error before ordinary retry/failure classification; +- repeated `before_drive` prerequisites on every newly installed pass; +- synthetic cap settlement; +- retry wait/wake/deadline policy; +- deferred suspension/poll frame persistence, replacement cleanup, and snapshot foundation; +- missing-identity suspension; +- activation-only unknown-outcome policy. + +Recovery reads only the already-restored in-memory state. A same-process safe yield/hosted wake must not pretend a crash occurred. + +Run the complete R3 suite and scratch example against runtime2. + +### Size review + +Target total ≤ 2,200 lines. + +## R2.7 — Tool parity + +### Goal + +Pass the complete target R4 contract, including `tool-durability.md`. + +Implement: + +- complete durable tool plans and stable invocation ID = result entry ID; +- bound typed tool-argument, invocation-memo, progress-checkpoint, and staged-result addresses; +- planned → effect_pending → outcome_ready → completed; +- invocation-scoped `getMemo`/`setMemo` and Flue-style memoization support; +- sequential and parallel execution with completion-order outcome staging; +- source-ordered result materialization; +- immediate/synthetic outcomes; +- safe replay and unsafe interruption using the latest checkpoint value; +- every selected checkpoint synchronously enqueued, latest promise retained, and latest awaited before `after_tool`; +- hooks/events/usage/telemetry/breakpoints; +- termination and cancellation handling at every tool state. + +Parallel effects remain ordinary local promises. Intent, checkpoint, outcome-staging, and materialization commands queue through the lane owner. Current call state is read from memory, never via full restore. Do not reintroduce one-active/one-waiting checkpoint coalescing. + +Run the complete R4 suite and scratch example against runtime2. + +### Size review + +Target total ≤ 3,000 lines; stretch ≤ 2,500. If runtime2 approaches runtime1 size, do not continue to parity cleanup without architectural review. + +## Superseded parity/switch roadmap + +The former R2.8 parity-first switch plan was superseded by the approved deletion strategy. Execute [WP00 — Runtime1 removal](work-packages/00-runtime1-removal.md), then [WP01 — Bound values and lists](work-packages/01-bound-values-lists.md). Those handoffs are the sole source for deletion lists, retained tests, validation, non-goals, and stop conditions. Do not recreate a dual-runtime parity gate or legacy smoke suite. + +# Comparison discipline + +The target contract and transaction tables in `harness.md`, plus the scoped redesign in this document until it is folded back, are the authority. Compare runtime2 behavior against those documents: + +- transaction write arrays and durable value/entry state; +- events and ordering; +- telemetry spans/attributes; +- breakpoints/details; +- public results/errors and suspended descriptors; +- storage read counts and source line counts. + +Do not require parity with runtime1 on redesigned acceptance, hooks, FSM, restore validation, bound value/list storage, assistant partial durability, or tool durability. Until deletion, compare it only for scenarios whose documented contract is explicitly unchanged. After deletion, use the harvested scenario inventory and ordinary git history for archaeology; do not recreate a dual-runtime harness. + +# Coding rules + +- No `any`. +- No dynamic/inline imports. +- Erasable TypeScript syntax only. +- Keep phase/ID checks beside the transition they protect. +- No generic reducer/action framework. +- No external effect inside the lane command line. +- No mutation of published memory before commit. +- No full restore in live procedures. +- Inline one-use helpers. +- Check external API types from source/node_modules. +- Do not modify runtime1 except as required by WP00 deletion; never change the target contract to preserve it. +- Do not touch unrelated coding-agent worktree files. +- Do not commit unless the user asks. + +# Validation + +Use the active work-package handoff; it owns the exact test list. Every code package also runs agent/root TypeScript, `git diff --check`, and `npm run check`. Never run unrestricted Vitest, `npm test`, or paid-provider tests. + +# Next assignment only + +Execute [WP00 — Runtime1 removal](work-packages/00-runtime1-removal.md) and stop at its boundary. Then execute [WP01 — Bound values and lists](work-packages/01-bound-values-lists.md) as a separate package. Do not resume the historical R2.4–R2.8 sequence or begin provider execution without a new concrete handoff linked from `harness.md` Part 8. + +The completed Fable and DeepSeek reviews are summarized in this document; no subagent session or temporary review checkout is expected to survive compaction. diff --git a/packages/agent/docs/tool-durability.md b/packages/agent/docs/tool-durability.md index 199416ec1..cd1d1bfee 100644 --- a/packages/agent/docs/tool-durability.md +++ b/packages/agent/docs/tool-durability.md @@ -28,7 +28,7 @@ The solution separates two orders: 1. **outcome durability:** actual completion order; 2. **entry materialization:** assistant source order. -A complete finalized result becomes durable immediately in `pending.entry`; the call becomes `outcome_ready`; placement happens later when every earlier source position is complete or ready. +A complete finalized result becomes durable immediately in `pi.pending.entry`; the call becomes `outcome_ready`; placement happens later when every earlier source position is complete or ready. ## Goals @@ -46,7 +46,7 @@ A complete finalized result becomes durable immediately in `pending.entry`; the - A nested durable state machine for each `step.do` call. - Inferring tool completion from progress output. - Treating partial output as the canonical final result. -- Giving tools raw `Session`, `SessionMutator`, scalar-register, or list-register access. +- Giving tools raw `Session`, `SessionMutator`, or bound storage-address access. - Finalizing the public harness-native tool type in this document. ## Durable identities @@ -69,55 +69,52 @@ The surrounding operation state continues to provide: ## Storage -### Existing scalar namespaces - -```text -op.tool_args/{operationId}:{turnId}:{sourceIndex} - Effective validated arguments, persisted before effect admission. - -pending.entry/{resultEntryId} - Complete finalized ToolResultMessage while outcome_ready awaits placement. -``` - -### Invocation facts - -Add an operation-owned scalar namespace: +### Existing bound values ```ts -interface RegisterValues { - "op.tool_fact": JsonValue; -} +operationToolArgs(operationId, turnId, sourceIndex) +// Effective validated arguments, persisted before effect admission. + +pendingEntry(resultEntryId) +// Complete finalized ToolResultMessage while outcome_ready awaits placement. ``` -Physical key: +### Invocation memos -```text -{operationId}:{invocationId}:{factName} +Define the operation-owned address constructor in `session/values.ts`: + +```ts +export const operationToolMemo = ( + operationId: string, + invocationId: string, + memoName: string, +) => value( + "pi.op.tool_memo", + `${operationId}:${invocationId}:${memoName}`, +); ``` -`factName` must be non-empty and contain no `:`. Names may use dots or slashes for application-local grouping. `setFact(name, undefined)` deletes the physical register. +`memoName` must be non-empty and contain no `:`. Names may use dots or slashes for application-local grouping. `setMemo(name, undefined)` deletes the exact bound value. -The operation prefix permits defensive terminal cleanup. The invocation prefix permits atomic cleanup when one outcome becomes ready. +`scanValues(operationToolMemoPrefix(operationId))` permits defensive operation cleanup. `scanValues(operationToolMemoPrefix(operationId, invocationId))` permits atomic invocation cleanup when one outcome becomes ready. Core cleanup uses these owner-defined prefix constructors rather than repeating raw reserved namespace/key grammar. ### Partial tool output -The durable recovery value is the latest complete bounded progress snapshot selected by the tool. That is total current state, so use a scalar namespace: +The durable recovery value is the latest complete bounded progress snapshot selected by the tool. That is total current state, so use one bound value address: ```ts -interface RegisterValues { - "pending.tool_output": AgentToolResult; -} -``` - -Physical key: - -```text -{operationId}:{invocationId} +export const pendingToolOutput = ( + operationId: string, + invocationId: string, +) => value>( + "pi.pending.tool_output", + `${operationId}:${invocationId}`, +); ``` This is auxiliary observation data. It never proves the effect succeeded or completed. The stored value has exactly the same content/details/usage shape as the live `partialResult`; recovery does not need a tool-specific progress codec. -The tool owns snapshot bounding, cadence, and duplicate suppression. The harness owns persistence, drop-coalescing, invocation fencing, and cleanup. The first API has no generic byte cap and does not truncate or reinterpret typed tool data; in-process tools are trusted to honor the bounded-snapshot contract. There is no tool-visible `flush()` method and no durable list of progress updates. +The tool owns snapshot bounding, checkpoint cadence, and duplicate suppression. The harness owns synchronous enqueue, promise tracking, invocation fencing, and cleanup. The first API has no generic byte cap and does not truncate or reinterpret typed tool data; in-process tools are trusted to honor the bounded-snapshot contract. There is no tool-visible `flush()` method and no durable list of progress updates. A crash may lose live updates newer than the latest committed checkpoint. JSONL physical growth is proportional to the size and frequency of distinct requested checkpoints until compaction; Memory and SQLite retain one current value. At 50 KiB every two seconds, JSONL's uncompacted worst case is approximately 15 MiB per ten minutes of continuously changing output. @@ -141,14 +138,15 @@ export type AgentHarnessToolUpdateCallback = ( Every callback invocation remains an immediate live update. The callback is synchronous and returns `void`; `checkpoint: true` additionally requests persistence of that complete snapshot. It is not a durability acknowledgement. Internally, the harness retains the latest `events.emit(tool_update)` promise so existing listener delivery completes before `after_tool`; tools do not await or receive that promise. -Checkpoint requests are drop-coalesced. The synchronous request path either immediately enqueues the active lane-line mutation or replaces the in-memory waiting value: +Every `checkpoint:true` call synchronously enqueues one scalar replacement on the lane mutation line, attaches the ordinary harness-fault observer to that promise, and replaces the process-local `latestCheckpointWrite` reference. Writes themselves are neither dropped nor coalesced, and replacing the reference never leaves an earlier rejection unobserved: -- at most one checkpoint commit is active per invocation; -- at most one newer requested snapshot waits; -- a newer requested snapshot replaces the waiting one; -- an admitted commit verifies the same call remains `effect_pending`; -- tool-promise settlement closes checkpoint admission and drops the waiting, not-yet-enqueued snapshot; an active checkpoint is already ordered on the lane mutation line before outcome staging; -- a failed admitted commit faults the harness under the ordinary storage-fault rule. +- lane-line FIFO preserves request order; +- each mutation verifies the same call remains `effect_pending`; +- completion of the latest promise implies completion of every earlier checkpoint write; +- tool-promise settlement stops accepting updates and awaits that latest promise before `after_tool`; +- a failed checkpoint commit faults the harness under the ordinary storage-fault rule. + +A tool that requests checkpoints faster than storage can queue work in memory. Under the trusted-tool contract, cadence is the tool's responsibility. The built-in bash policy bounds this queue in ordinary use. Tools should compare against their last requested checkpoint when duplicate suppression matters. Storage does not read and deep-compare the current scalar as part of every checkpoint. @@ -197,8 +195,8 @@ type ToolCall = `outcome_ready` means: - execution, error normalization, and `after_tool` have finished, or the harness has constructed a final synthetic result; -- the complete final `ToolResultMessage` exists in `pending.entry/{resultEntryId}`; -- invocation facts and partial-output storage are gone; +- the complete final `ToolResultMessage` exists at `pendingEntry(resultEntryId)`; +- invocation memos and partial-output storage are gone; - the tool must never execute again; - the immutable result entry may not exist yet because an earlier call has not materialized. @@ -232,7 +230,7 @@ Unchanged effect sandwich: planned → prepare arguments, run before_tool, validate replacements → TX[ - set op.tool_args, + set pi.op.tool_args, set call = effect_pending(replay) ] → admit tool execution @@ -246,11 +244,11 @@ Every `onUpdate(partialResult, options)` publishes the live update through the e ```text TX[ - set pending.tool_output/{operationId}:{invocationId} = partialResult + setValue(pendingToolOutput(operationId, invocationId), partialResult) ] ``` -The mutation verifies the same operation, turn, source position, and invocation remain `effect_pending`. It does not rewrite `op.state`. A late checkpoint after settlement returns without committing. +The mutation verifies the same operation, turn, source position, and invocation remain `effect_pending`. It does not rewrite `pi.op.state`. A late checkpoint after settlement returns without committing. The tool must checkpoint bounded complete snapshots, not growing unbounded values. Bash uses the same bounded `ShellCaptureProgress` snapshot it already sends live. Clients may render and locally retain live updates newer than the durable checkpoint, but those updates are explicitly process-local. @@ -258,25 +256,27 @@ The tool must checkpoint bounded complete snapshots, not growing unbounded value When the tool promise settles: -1. synchronously stop accepting updates, expire the invocation capability, and close checkpoint admission; -2. drop the waiting checkpoint; any active checkpoint or pre-return fact mutation is already ahead of staging on the lane mutation line; -3. await the latest tracked `tool_update` delivery, which implies delivery of every earlier update; -4. run `after_tool` when this is a real fresh or safely replayed result and cancellation did not prevent the hook; -5. construct the complete final `ToolResultMessage`; -6. emit and await `tool_end` for a real execution; -7. commit the result as `outcome_ready`. +1. synchronously stop accepting updates and expire the invocation capability; +2. await the latest tracked `tool_update` delivery and latest checkpoint-write promise; each implies completion of its preceding queue; +3. run `after_tool` when this is a real fresh or safely replayed result and cancellation did not prevent the hook; +4. construct the complete final `ToolResultMessage`; +5. emit and await `tool_end` for a real execution; +6. commit the result as `outcome_ready`. -`setFact()` returns a promise and tools must await it; `step.do` always does. An unawaited pre-return mutation is still enqueued before staging and is deleted by staging. Calls begun after capability expiry reject. No separate invocation-write drain exists. +`setMemo()` returns a promise and tools must await it; `step.do` always does. An unawaited pre-return mutation is still enqueued before staging and is deleted by staging. Calls begun after capability expiry reject. No separate invocation-write drain exists. Transaction: ```text TX[ - set pending.entry/{resultEntryId} - = { type: "message", payload: finalizedToolResultMessage }, - delete pending.tool_output/{operationId}:{invocationId}, - delete op.tool_fact/{operationId}:{invocationId}:*, - set op.state call = outcome_ready(terminate) + setValue( + pendingEntry(resultEntryId), + { type: "message", payload: finalizedToolResultMessage }, + ), + deleteValue(pendingToolOutput(operationId, invocationId)), + deleteValue(memo.address) for every memo returned before commit by + scanValues(operationToolMemoPrefix(operationId, invocationId)), + setValue(operationState(operationId), call = outcome_ready(terminate)) ] ``` @@ -309,17 +309,17 @@ Before the placement transaction, emit and await each finalized result's `messag ```text TX[ - insert result entry i from pending.entry/i, - delete pending.entry/i, + insert result entry i from pendingEntry(i), + deleteValue(pendingEntry(i)), insert tool usage row i if reported, insert result entry i+1 with parent = result i, - delete pending.entry/i+1, + deleteValue(pendingEntry(i+1)), insert tool usage row i+1 if reported, - set lane.leaf = newest result, - set calls i..i+1 = completed, - set next checkpoint in the same state write when the whole batch completed + setValue(laneLeaf(lane), newest result), + setValue(operationState(operationId), + calls i..i+1 = completed and, when complete, next checkpoint) ] ``` @@ -327,7 +327,7 @@ Each inserted entry uses its already-reserved `resultEntryId`. Writes construct Tool-reported usage remains durable in the staged message until placement. The initial implementation writes its ledger row atomically with entry materialization, matching the current entry/usage ordering and avoiding a ledger row that references an entry not yet present. No usage ID reservation is needed because a failed placement transaction writes neither row nor completed state. -When the final call materializes, the same transaction deletes the batch's `op.tool_args` keys and transitions to the correct checkpoint: +When the final call materializes, the same transaction calls `scanValues(operationToolArgsPrefix(operationId, turnId))` and deletes every returned address, and transitions to the correct checkpoint: - every result terminates → `may_finish`, no final assistant required; - otherwise → `need_assistant(false)`. @@ -360,13 +360,13 @@ The durable invariant changes from “completed calls form a source-ordered pref - after that prefix, parallel calls may be `planned`, `effect_pending`, or `outcome_ready` in any mixture; - only source-ordered materialization extends the completed prefix. -Sequential execution normally has at most one non-planned call after the completed prefix, but restore should validate this from the captured execution mode as it does today. +Sequential execution constructs at most one non-planned call after the completed prefix. Committed call state is trusted on restore; the owning procedure enforces this shape while creating and consuming transitions rather than through a broad restore audit. ## Unsafe recovery with partial output For orphaned `effect_pending` with `replay: "never"`: -1. read `pending.tool_output/{operationId}:{invocationId}` when present; +1. read `pendingToolOutput(operationId, invocationId)` when present; 2. preserve its bounded content and serializable details; 3. append a mandatory human-readable interruption marker; 4. construct a harness-owned `ToolResultMessage` with `isError: true`; @@ -385,16 +385,16 @@ Rules: - do not run `after_tool` for this synthetic result; - preserve checkpoint `usage` when present, but ignore checkpoint `addedToolNames` and `terminate` because progress never has final-result authority; - set `terminate: false` and add no tools; -- no partial register is also valid and yields only the interruption result; +- an absent checkpoint value is also valid and yields only the interruption result; - never infer completion from an apparent success line in partial output; -- cleanup of partial output and invocation facts is atomic with staging the synthetic result. +- cleanup of partial output and invocation memos is atomic with staging the synthetic result. ## Safe recovery For orphaned `effect_pending` where both the stored and current declarations are `replay: "safe"`: -1. retain invocation facts; -2. atomically delete the previous `pending.tool_output` key; +1. retain invocation memos; +2. atomically delete `pendingToolOutput(operationId, invocationId)`; 3. emit/reset process-local progress observation as needed; 4. rerun the tool with persisted arguments and the same `invocationId`; 5. completed `step.do` calls return their memoized values; @@ -405,7 +405,7 @@ Deleting old progress prevents duplicate chunks when replayed code emits progres If the current tool declaration is missing, suspend before replay. If it is present but no longer safe, use unsafe interruption recovery. -## Invocation facts +## Invocation memos The harness-native tool call receives a purpose-built invocation capability conceptually equivalent to: @@ -415,37 +415,37 @@ interface AgentHarnessToolInvocation { readonly operationId: string; readonly turnId: string; - getFact(key: string): Promise; - setFact(key: string, value: JsonValue | undefined): Promise; + getMemo(key: string): Promise; + setMemo(key: string, value: JsonValue | undefined): Promise; } ``` Each operation: -1. validates the fact name and checks process-local capability expiry; +1. validates the memo name and checks process-local capability expiry; 2. synchronously enqueues work on the lane mutation line before returning its promise; 3. verifies the operation, turn, source position, and invocation are still the same `effect_pending` call when that job executes; -4. reads or writes only the invocation's physical prefix; +4. constructs and reads or writes only `operationToolMemo(operationId, invocationId, name)`; 5. rejects after capability expiry or durable ownership loss. -The durable check matters for authorized external finalization. In ordinary execution, a fact mutation initiated before the tool returns is FIFO-ordered before outcome staging; one initiated afterward fails the expired-capability check. +The durable check matters for authorized external finalization. In ordinary execution, a memo mutation initiated before the tool returns is FIFO-ordered before outcome staging; one initiated afterward fails the expired-capability check. -A late zombie callback can neither recreate facts after `outcome_ready` nor write into a later operation. +A late zombie callback can neither recreate memos after `outcome_ready` nor write into a later operation. -Facts are immediate durable memo state, not application-visible settlement state. They survive close/crash while the call remains `effect_pending` and are deleted when any real or synthetic outcome becomes ready. +Memos are immediate durable replay state, not application-visible settlement state. They survive close/crash while the call remains `effect_pending` and are deleted when any real or synthetic outcome becomes ready. -Terminal cleanup defensively deletes: +Terminal cleanup defensively scans and deletes the operation-owned families: -```text -op.tool_fact/{operationId}:* -pending.tool_output/{operationId}:* +```ts +scanValues(operationToolMemoPrefix(operationId)) +scanValues(pendingToolOutputPrefix(operationId)) ``` -in addition to existing operation-owned registers. +in addition to other operation-owned addresses. Each returned `StoredValue` supplies its exact bound address for `deleteValue`; no later operation receives a raw key. ## Flue-style `step.do` -Build `step.do` over invocation facts; it does not need its own harness state union: +Build `step.do` over invocation memos; it does not need its own harness state union: ```ts interface ToolSteps { @@ -460,10 +460,10 @@ Algorithm: ```text validate deterministic unique name -→ getFact("step/" + name) +→ getMemo("step/" + name) → present: return stored value → absent: run effect -→ setFact("step/" + name, value) +→ setMemo("step/" + name, value) → await durability → return value ``` @@ -487,9 +487,9 @@ Within one live execution, calling the same step name twice is an invariant erro ## Application persistent state -Flue-style application state is distinct from invocation facts. +Flue-style application state is distinct from invocation memos. -Invocation fact: +Invocation memo: ```text step completed → memo becomes visible immediately @@ -503,14 +503,14 @@ tool stages state change → outcome_ready: state and finalized result become visible together ``` -Do not implement application state by calling public `setCustomFact()` directly during execution. +Do not implement application state by calling `Session.setValue()` directly during execution. Two valid implementation stages: 1. Flue keeps its state externally and atomically stores application state plus a full result memo keyed by `invocationId`. -2. A later harness API accepts staged application fact writes and promotes them in the `outcome_ready` transaction. +2. A later harness API accepts staged application value writes and promotes them in the `outcome_ready` transaction. -The second option is required only if Flue's `usePersistentState` moves into harness-owned session facts. Its public typing and conflict semantics remain an open design item for the harness-native tool discussion. +The second option is required only if Flue's `usePersistentState` moves into harness-owned session values. Its public typing and conflict semantics remain an open design item for the harness-native tool discussion. ## Cancellation @@ -520,7 +520,7 @@ Cancellation reconciliation never replays a restored tool. - a live started call may finalize its real local result under cancelled control, then become `outcome_ready` with `terminate: false`; - restored `effect_pending` calls use an interrupted synthetic result, optionally including partial output, regardless of safe replay declaration; - existing `outcome_ready` calls are preserved and materialized in source order; -- invocation facts and partial output are deleted with each staged cancellation outcome; +- invocation memos and partial output are deleted with each staged cancellation outcome; - no `before_tool` or `after_tool` starts during restored synthetic reconciliation. The aborted terminal transaction runs only after every call outcome has materialized and accepted deferred writes have drained under the existing cancellation rules. @@ -529,45 +529,36 @@ The aborted terminal transaction runs only after every call outcome has material Close is still a controlled crash: -- fact/checkpoint mutations already enqueued under the admission barrier may finish; +- memo/checkpoint mutations already enqueued under the admission barrier may finish; - live output newer than the latest committed tool-requested checkpoint may be lost; - no synthetic outcome or cancellation marker is written; - durable state remains at `effect_pending` or `outcome_ready`. -External finalization deletes operation-owned arguments, invocation facts, partial output, staged pending outcomes, and other pending entries in its terminal transaction. A live task that later tries to stage an outcome fails the ownership fence and stops through `OperationEnded`. +External finalization deletes operation-owned arguments, invocation memos, partial output, staged pending outcomes, and other pending entries in its terminal transaction. A live task that later tries to stage an outcome fails the ownership fence and stops through `OperationEnded`. -## Restore and validation +## Restore and consumption-time reads -Scalar restore remains authoritative. Extend direct hydration and semantic checks: +Base restore constructs the trusted lane/operation projection from required owner values. It does not hydrate or semantically audit tool arguments, invocation memos, progress checkpoints, staged outcomes, completed entries, completed-prefix shape, or captured execution-mode relationships. + +The procedure responsible for the current typed state performs only its exact consumption-time reads: ### `planned` -- result ID is reserved but has no entry or `pending.entry`; -- no `op.tool_args` is required; -- no partial output or invocation facts are required. +Clearance needs no auxiliary restore read. It prepares the call and writes arguments before effect admission. ### `effect_pending` -- matching `op.tool_args` exists; -- result entry and `pending.entry/{resultEntryId}` do not exist; -- invocation facts may exist but are not required for base validation; -- a partial-output scalar may exist but is auxiliary and must satisfy the trusted tool's bounded-snapshot contract. +Activation reads `operationToolArgs(operationId, turnId, sourceIndex)` and optionally reads `pendingToolOutput(operationId, invocationId)`. Missing required arguments are an invariant defect at consumption. Invocation memos are read only through the scoped capability. Safe replay, unsafe interruption, and snapshots use no broad prefix scan. ### `outcome_ready` -- `pending.entry/{resultEntryId}` exists; -- it contains a `ToolResultMessage` matching the source call's provider call ID and tool name; -- immutable result entry does not yet exist; -- invocation facts and partial output should be absent after the atomic staging transaction; -- no tool identity is needed to materialize it. +Materialization reads `pendingEntry(resultEntryId)`. Its absence or wrong trusted message relationship is an invariant defect when materialization consumes it. No tool identity or effect recovery is needed. ### `completed` -- immutable result entry exists and matches the source call; -- no staged pending entry exists; -- no invocation facts or partial output exist. +Ordinary dispatch performs no restore-time entry audit. Context/tree reads later consume the immutable entry through their normal typed paths. -Restore validates the completed-prefix and execution-mode shape described above. Partial output does not participate in semantic validation. Activation or snapshot hydration may read the exact scalar afterward. +Every live mutation still verifies current operation, turn, source position, invocation, and status on the lane line. Those checks fence concurrent settlement, cancellation, and external finalization; they are not historical restore validation. Terminal prefix cleanup remains defensive and does not make orphan scans part of restore. ## Snapshots and reconnect @@ -596,43 +587,44 @@ Instrumented-storage tests assert this ordering. A crash after `tool_end` but be | Race | Required result | |---|---| -| checkpoint vs tool settlement | an active checkpoint was enqueued first and staging later deletes it; the waiting checkpoint is dropped when the tool promise settles; a late update is ignored | -| fact write vs `outcome_ready` | an awaited or pre-return-enqueued write precedes staging and is then deleted; a post-return call rejects; external finalization first causes the durable ownership check to reject | +| checkpoint vs tool settlement | every accepted checkpoint was enqueued first; settlement awaits the latest promise, then staging deletes the checkpoint value; a late update is ignored | +| memo write vs `outcome_ready` | an awaited or pre-return-enqueued write precedes staging and is then deleted; a post-return call rejects; external finalization first causes the durable ownership check to reject | | B outcome vs earlier A settlement | B stages independently; placement waits for A | | crash after outcome staging | tool never replays; pending result later materializes | | crash during source-prefix placement | transaction exposes either none or all of that placement prefix | -| safe replay vs old partial output | old key is deleted before replay emits new progress | +| safe replay vs old partial output | the old bound checkpoint value is deleted before replay emits new progress | | cancellation vs real settlement | lane-line order chooses real cancelled-control result or synthetic reconciliation; at most one outcome stages | | terminal finalization vs late result | terminal ownership wins or outcome stages first; late task never recreates operation data | -| external finalization vs fact/checkpoint mutation | mutation first is removed by terminal cleanup; finalization first makes the mutation's durable ownership check reject | +| external finalization vs memo/checkpoint mutation | mutation first is removed by terminal cleanup; finalization first makes the mutation's durable ownership check reject | ## Invariants 1. `invocationId` equals the reserved result entry ID and is stable across safe replay. 2. A call in `outcome_ready` or `completed` never executes again. -3. Every `outcome_ready` call has exactly one complete matching `pending.entry` value. +3. Every `outcome_ready` call has exactly one complete matching `pi.pending.entry` value. 4. Completed calls form a source-ordered prefix. 5. Only source-ordered materialization extends that prefix. 6. Parallel calls after the completed prefix may mix planned, effect-pending, and outcome-ready states. -7. Invocation facts exist only while their call is `effect_pending`. +7. Invocation memos exist only while their call is `effect_pending`. 8. Partial output is auxiliary and never establishes effect completion. 9. Unsafe synthetic results explicitly state that captured output is incomplete and the external outcome is unknown. -10. Staging an outcome atomically deletes its invocation facts and partial output. +10. Staging an outcome atomically deletes its invocation memos and partial output. 11. Materialization atomically inserts the immutable entry and deletes its staged pending value. 12. A late invocation capability cannot write after outcome settlement or operation loss. -13. `step.do` values are memoized only after their fact write commits; effects remain at-least-once. -14. Operation terminal cleanup leaves no tool args, invocation facts, partial output, or staged outcomes. +13. `step.do` values are memoized only after their memo write commits; effects remain at-least-once. +14. Operation terminal cleanup leaves no tool args, invocation memos, partial output, or staged outcomes. ## Required tests ### State and restore -- every planned/effect-pending/outcome-ready/completed state; -- missing or mismatched pending result for `outcome_ready`; -- pending result incorrectly coexisting with completed entry; -- invocation facts and partial output absent after staging; -- restore validates without reading partial output; -- activation then hydrates the exact bounded scalar when needed. +- every trusted planned/effect-pending/outcome-ready/completed projection restores without auxiliary reads; +- base restore does not audit completed-prefix or execution-mode relationships; +- effect-pending consumption reads exact required arguments and optional bounded checkpoint; +- outcome-ready consumption reads the exact staged result; +- missing required arguments or staged result fails at the consuming procedure, not base restore; +- invocation memos and partial output are absent after staging; +- snapshot/activation hydrate only the exact bounded checkpoint value when needed. ### Parallel ordering @@ -646,23 +638,23 @@ Instrumented-storage tests assert this ordering. A crash after `tool_end` but be ### Replay and interruption - safe replay uses persisted arguments and the same invocation ID; -- safe replay preserves step facts but clears old partial output; +- safe replay preserves step memos but clears old partial output; - current declaration downgrade from safe to never interrupts; - unsafe recovery with no checkpoint and with a complete bounded checkpoint; - synthetic result is error/incomplete/unknown and never runs `after_tool`; - cancellation never safely replays a restored call. -### Invocation facts and `step.do` +### Invocation memos and `step.do` -- set/get/delete fact with physical invocation scoping; +- set/get/delete memo with invocation-address scoping; - completed step skips effect after reopen; - crash before memo commit reruns effect; - crash after memo commit returns memo; - several completed steps followed by one interrupted step; - duplicate live step names reject; -- fact write racing outcome staging; +- memo write racing outcome staging; - capability write after expiry, cancellation outcome staging, and external finalization rejects; -- terminal cleanup removes crash-leaked facts. +- terminal cleanup removes crash-leaked memos. ### Partial output @@ -670,8 +662,8 @@ Instrumented-storage tests assert this ordering. A crash after `tool_end` but be - `checkpoint: true` writes the complete bounded snapshot; - tool-selected checkpoint cadence and duplicate suppression; - bash emits live snapshots at 100 ms and requests distinct checkpoints at most every two seconds; -- drop-coalescing permits at most one active and one waiting checkpoint, and tool-promise settlement drops the waiting value; -- a checkpoint after outcome staging cannot recreate the key; +- every selected checkpoint enqueues one write, and awaiting the latest promise at tool settlement implies all earlier writes completed; +- a checkpoint after outcome staging cannot recreate the address's value; - Memory, JSONL, and SQLite restore the same checkpoint value; - JSONL growth follows checkpoint cadence rather than raw bash output volume; - terminal compaction reclaims superseded/deleted checkpoint snapshots according to the existing dead-byte policy. @@ -679,7 +671,7 @@ Instrumented-storage tests assert this ordering. A crash after `tool_end` but be ### Atomicity and instrumentation - exact intent, synchronous update acceptance, asynchronous update-delivery, `after_tool`, `tool_end`, outcome-ready, source-ordered message lifecycle, and materialization order; -- outcome staging is atomic with fact/output cleanup; +- outcome staging is atomic with memo/output cleanup; - materialization is atomic with pending deletion, usage, leaf, and state; - crash at every boundary; - no effect starts before intent; @@ -691,20 +683,20 @@ Instrumented-storage tests assert this ordering. A crash after `tool_end` but be Expected runtime areas: - operation-state types in `packages/agent/src/harness/session/types.ts`; -- restore key collection and semantic validation; +- trusted restore projection and exact consumption-time address reads; - tool-batch procedure and source-ordered materialization; - terminal cleanup and cancellation reconciliation; - harness-specific update options, snapshots, and events; - invocation-scoped capability implementation on the lane mutation line; -- backend conformance through the already-implemented scalar/list register APIs; +- backend conformance through the bound value/list APIs; - instrumented-storage transaction assertions. -Concrete namespace additions: +Concrete built-in address constructors in `session/values.ts`: ```text -scalar: op.tool_fact -scalar: pending.tool_output -existing scalar staging: pending.entry +operationToolMemo(operationId, invocationId, name) → value("pi.op.tool_memo", ...) +pendingToolOutput(operationId, invocationId) → value("pi.pending.tool_output", ...) +pendingEntry(resultEntryId) → value("pi.pending.entry", ...) ``` -Implement `outcome_ready` and invocation facts before progress checkpoints. The state solves incorrect parallel replay by itself; checkpoints improve reconnect observation and unsafe interruption diagnostics without becoming completion authority. +Implement `outcome_ready` and invocation memos before progress checkpoints. The state solves incorrect parallel replay by itself; checkpoints improve reconnect observation and unsafe interruption diagnostics without becoming completion authority. diff --git a/packages/agent/docs/work-packages/00-runtime1-removal.md b/packages/agent/docs/work-packages/00-runtime1-removal.md new file mode 100644 index 000000000..784de56df --- /dev/null +++ b/packages/agent/docs/work-packages/00-runtime1-removal.md @@ -0,0 +1,75 @@ +# WP00 — Runtime1 removal + +## Status + +Approved; implementation not started. No tag. Do not release while the public runtime2 execution path is incomplete. + +## Goal + +Make runtime2 the sole public harness implementation, delete runtime1 and its obsolete tests, then stop before adding runtime2 behavior. + +## Prerequisites + +- The approved acceptance/hook redesign and durability handoffs are present in `runtime2.md`, `registers.md`, `assistant-durability.md`, and `tool-durability.md`. +- Existing tests are evidence, not authority. + +## Work, in order + +1. **Reconcile the contract.** Fold the approved acceptance/hook redesign into `harness.md`, including durable `starting`, atomic hook-free acceptance, driver-owned `before_run`, `before_drive`, request-local system-prompt transformation, trusted restore, and removal of process-origin activation semantics. Audit §§0.4, 1.2, 3.1–3.6, 4.1–4.2, 4.5, 5.1–5.2, 5.5–5.6, and Parts 8–9; remove every stale `BeforeResumePrepared`, `before_resume`, `resumeData`, `systemPromptOverride`, stable-ID routing, reservation, and `fresh | continue | resume` activation reference. Then remove the temporary supersession notice and duplicated redesign from `runtime2.md`. +2. **Harvest before deletion.** Inspect `agent-harness-runtime.test.ts`, `agent-harness-r2/r3/r4.test.ts`, and old `restore.test.ts`. Preserve unique scenarios in the detailed future rows or a temporary categorized inventory; explicitly discard old reservation, `before_resume`, `resumeData`, persisted hook prompt override, semantic restore audit, and pre-`outcome_ready` tool-crash behavior. +3. **Remove runtime1-only public members.** Delete `before_resume`, `BeforeResumePrepared`, `resumeData`, `systemPromptOverride`, and stable hook-ID routing. Add the approved `before_drive` and `transform_context` shapes. Update telemetry schema source and regenerate its document. Do not implement `starting` or acceptance behavior here. +4. **Switch the factory.** Add `packages/agent/src/harness/runtime2/index.ts`, point `agent-harness.ts` at it, and add a constructor-selection regression. Verify the experimental coding-agent worker still creates the harness, subscribes to events, and closes it. +5. **Delete runtime1 source and tests** listed below. +6. **Update `[Unreleased]`** for the public breaking removals and temporarily incomplete factory. +7. Run the retained tests and checks. Fix every failure; do not restore compatibility shims. + +## Delete + +```text +packages/agent/src/harness/runtime/** +packages/agent/src/harness/restore.ts +packages/agent/test/harness/agent-harness-runtime.test.ts +packages/agent/test/harness/agent-harness-r2.test.ts +packages/agent/test/harness/agent-harness-r3.test.ts +packages/agent/test/harness/agent-harness-r4.test.ts +packages/agent/test/harness/restore.test.ts +packages/agent/test/harness/scratch/r1.ts +packages/agent/test/harness/scratch/r2.ts +packages/agent/test/harness/scratch/r3.ts +packages/agent/test/harness/scratch/r4.ts +``` + +## Retain + +- all `test/harness/runtime2/**` tests; +- Session, SessionTree, storage, repository, backend-conformance, and instrumentation tests; +- execution assistant/tool/primitives tests; +- config, hooks, events, telemetry, compaction, and branch-summary code/tests; +- `types.test.ts`, updated for the reduced public contract; +- `packages/agent/src/agent-loop.ts` unchanged. + +Do not parameterize obsolete runtime1 suites against runtime2 and do not keep a runtime1 smoke suite. + +## Acceptance + +- No source import references `harness/runtime/*`. +- Public `AgentHarness.create()` selects runtime2. +- Runtime2 creation, events, inspection, close, and fault tests pass. +- These coding-agent tests pass: + - `experimental-remote-runtime.test.ts` + - `experimental-session-worker.test.ts` + - `experimental-session-worker-lifecycle.test.ts` +- Every modified test passes individually. +- Agent and root TypeScript pass. +- `git diff --check` and `npm run check` pass. + +## Non-goals + +- No bound-value/list implementation. +- No implementation of `starting` or atomic acceptance. +- No drive owner, provider, retry, deferred, or tool execution. +- No runtime1 parity work, compatibility layer, archaeology tag, or release. + +## Stop condition + +Stop once runtime1 is absent, runtime2 is the public factory, retained coverage is green, and all checks pass. Report the deletion and harvested scenarios; do not begin another work package. diff --git a/packages/agent/docs/work-packages/01-bound-values-lists.md b/packages/agent/docs/work-packages/01-bound-values-lists.md new file mode 100644 index 000000000..2d52dd67a --- /dev/null +++ b/packages/agent/docs/work-packages/01-bound-values-lists.md @@ -0,0 +1,57 @@ +# WP01 — Bound values and lists + +## Status + +Ready after WP00. The normative design is [`registers.md`](../registers.md). + +## Goal + +Replace the retained register/custom-state API with bound `Value` and `ValueList` addresses across every existing backend and application surface, then stop before runtime execution work. + +## Scope + +1. Add `session/values.ts` with universal `value()` / `list()`, typed write helpers, all `pi.*` built-in address/prefix constructors, and no registry, catalog, privilege path, or global type map. +2. Replace `RegisterValues`, namespace unions, `Register`, `getRegister`, `listRegisters`, raw register writes, custom-state APIs, and old persisted semantic names. +3. Expose the same bound-address reads through Storage, SessionReader, SessionMutator, Session, and SessionTree; add direct Session scalar/list writes and retain one-commit `Session.mutate()` composition. +4. Implement scalar replacement and list append/page/whole-list-delete in Memory, JSONL, and SQLite, including replay, compaction, snapshots, forks/rewrites, instrumentation, and shared conformance. +5. Use `pi.session.name`, `pi.entry.label`, and the complete built-in namespace/key grammar from `registers.md`. Applications define their own addresses directly. +6. Update telemetry schema source for value/list write kinds and regenerate `telemetry-schema.md`. +7. Update all retained source/tests and public exports; no old API shim remains. + +## Preserve + +- transaction atomicity and global write `seq` ordering; +- JSONL torn-tail and compaction behavior, preserving list-element sequences; +- SQLite `BEGIN IMMEDIATE`, writer fencing, and indexed list paging without temporary sorting; +- repository/fork semantics, branch indexes, stats, UUIDv7/follower IDs, v3 normalization, and existing entry/usage behavior; +- projection-only restore: no list read until bounded consumption-time hydration; +- trusted in-process values with no codecs, cloning, or runtime shape validation. + +## Required coverage + +- type inference and incompatible scalar/list write rejection; +- exact `pi.*` built-in addresses and five scan-prefix constructors; +- application-wide and dynamically keyed addresses with no second operation-time key; +- scalar set/get/delete/recreate and namespace-scoped prefix scans; +- list append, global element sequences, asc/desc cursors, limits, absent reads, whole delete, and delete-then-append; +- atomic entry + usage + scalar + list transactions and rollback; +- identical Memory/JSONL/SQLite behavior; +- JSONL replay, torn transactions, and cursor-preserving compaction; +- SQLite query plans; +- forks excluding operation-owned values/lists while preserving semantic session values; +- close rejecting later reads while admitted commits drain; +- zero old register or custom-state API references outside immutable release history. + +## Non-goals + +- No runtime acceptance, drive, provider, assistant-frame consumer, tool memo/checkpoint consumer, or operation-state redesign. +- No generic event log, list truncation, per-element deletion, registry, catalog, or compatibility facade. +- Do not rewrite applied backend migrations solely for terminology; follow the compatibility policy in `registers.md`. + +## Validation + +Run every modified test individually, the shared backend conformance suites, SQLite package tests, agent/root TypeScript, `git diff --check`, and `npm run check`. + +## Stop condition + +Stop when every existing backend and retained Session surface uses bound values/lists, old APIs are absent, and all checks pass. Report schema/compatibility choices; do not begin runtime execution work.