diff --git a/packages/agent/docs/pico-v3.md b/packages/agent/docs/pico-v3.md new file mode 100644 index 000000000..1189fd3ec --- /dev/null +++ b/packages/agent/docs/pico-v3.md @@ -0,0 +1,2113 @@ +# pico v3 + +Design under discussion. Code and traces illustrate the proposed behavior; they are not existing +package exports. + +## 1. Goals and assumptions + +### 1.1 Small concepts, replaceable behavior + +Run agents using conversations, immutable entries, durable tasks, and scoped values/lists. +Built-in agent behavior is composed from tasks, not embedded in a fixed scheduler state machine. + +```text + ┌─ tool A ─┐ +generation ──────────┼─ tool B ─┼─ post_tools ─ generation + └─ tool C ─┘ + +replace a task definition + → change that behavior + → keep the scheduler and storage +``` + +The scheduler understands task lifecycle, dependencies, timing and cancellation. It does not +understand prompts, tool arguments or summaries. Storage understands stored objects and atomic +mutations, not task behavior. + +### 1.2 One writer per session + +Concurrent execution is allowed. Concurrent independent writers to the same session are not. + +```text +caller ───────┐ +tool ─────────┼─ session owner ─ serialized commands ─ storage +background ──┘ + +one command: + read committed state + construct mutations and assign numeric IDs + commit the complete batch + publish committed changes +``` + +All mutations, including working-state writes, pass through that owner. IDs are unique within the +session and are persisted unchanged. Uncommitted IDs cannot escape to callers or external effects. + +Different sessions may have different owners and share a SQLite database. SQLite still serializes +its write transactions. After a crash, a replacement owner resumes from durable state; it must not +run alongside the previous owner. + +### 1.3 Durable work, explicit recovery + +Durable storage preserves work that has been accepted, including work that has not started yet. +Opening a session starts no task effects. + +```text +commit pending task + → crash + → open: task is still pending + → drive: execute it + +commit external-effect intent + → perform external effect + → commit outcome and required successor tasks together +``` + +A crash during an external effect leaves uncertainty, not proof that nothing happened. Each task +definition must decide how to recover: retry safely, adopt existing work, or report interruption. +Durability does not promise exactly-once external effects. + +```text +close → stop local execution; leave unfinished durable tasks resumable +abort → durably cancel selected work; drive its cancellation to settlement +``` + +### 1.4 Long-lived sessions without residency machinery + +Old transcript entries and terminal tasks remain queryable. The harness should not retain them +merely because it encountered them earlier. + +```text +check a dependency: + read the named task + inspect its terminal status + return + // no harness-owned reference needs to survive the check +``` + +Active executions naturally retain their inputs and working data. Other query results are ordinary +local variables. No pin/unpin API, residency manager, or sweep that maintains a second resident +model. Routine execution should depend on current work and context, not scan accumulated history. + +Backend memory is a separate choice: + +| Backend | Storage-owned memory | Persistence | +|---|---|---| +| Memory | All stored data and indexes | None; useful for tests | +| JSONL | All stored data and indexes, loaded on open | Main file plus working sidecars | +| SQLite | Read results and database caches; no required full-history JS copy | Database | + +JSONL loading everything is intentional, not a violation of the working-set goal. Choose SQLite +when minimizing resident process memory matters. No fixed session-length limit is imposed; storage +capacity, numeric sequence limits and the amount of simultaneously live work remain real limits. + +### 1.5 Efficiency must be demonstrated + +```text +same faux-provider workload on each harness/backend + → CPU time + → total process memory + → actual disk bytes, including working data and auxiliary files +``` + +Prefer small atomic writes, indexed queries and batched reads. Do not turn a point lookup into a +history scan or reload every object to answer a small question. Measure hot paths before adding +caches or specialized machinery. A faster spike is evidence to investigate, not proof that its +interfaces or correctness shortcuts belong in this design. + +## 2. Conversations + +A conversation has a linear transcript of immutable entries and a context list selecting entries +for the model. Tasks and state are associated with it, but do not implicitly become model input. + +### 2.1 Entries and the transcript + +Minimal shapes; `JsonValue` means strict JSON. Historical-position and ownership metadata are +introduced separately. + +```ts +interface Entry { + readonly id: number; + readonly conversationId: number; + readonly kind: string; + readonly content: JsonValue; +} + +interface Conversation { + readonly id: number; + readonly context: readonly number[]; +} +``` + +These are read snapshots. Changing context produces a new conversation snapshot; an existing +entry's content never changes. + +```text +conversation A + +entry ID kind content +10 user "Inspect the parser" +20 assistant "The parser has two problems ..." +30 user "Explain the first one" +40 assistant "The first problem is ..." + +transcript: [10, 20, 30, 40] +``` + +- New entries append; existing entries are never patched, reordered or renumbered. +- IDs increase in transcript order, but need not be consecutive numbers. +- A note can belong to the transcript without belonging to model context. + +### 2.2 Context selects; projection converts + +```text +transcript context IDs provider messages +what happened what is selected what the model receives + +[10, 20, 30, 40] ──────────> [10, 20, 30, 40] ──────> user, assistant, user, assistant + projection +``` + +The context stores IDs, not copied messages or a second transcript. + +```text +entryById = readEntries(conversation.context) // batched read +messages = project selected entries in context order +normalize tool-result groups into their call order +``` + +An entry's kind defines its message contribution. A summary can project as a user message; a +selected custom entry can project nothing. Projection reads content and returns messages. It +writes no entries, state or tasks. + +Tool completion order need not equal call order: + +```text +assistant calls: [A, B] +transcript: assistant → result B → result A +model messages: assistant → result A → result B +``` + +The transcript remains unchanged by that normalization. + +### 2.3 Three context operations + +```text +appendContext(ids) + extend with newly appended selected entries, in transcript order + +replaceContextPrefix(through, replacement) + replace the prefix ending at through; preserve the existing suffix + +resetContext(ids) + replace the whole context with an empty or bootstrap context +``` + +Appending selected content and appending its context ID happen in the same commit. Context-only +changes do not create generation tasks. + +**Compaction:** append a summary entry, then replace a prefix in the same commit. + +```text +before: + transcript [10, 20, 30, 40] + context [10, 20, 30, 40] + +commit: + append summary 50 describing the prefix through 20 + replaceContextPrefix(through=20, replacement=50) + +after: + transcript [10, 20, 30, 40, 50] + context [50, 30, 40] +``` + +Summary 50 is newer than entries 30 and 40, but appears before them in model context. The entries +it summarizes remain in the transcript. Repeated compaction summarizes the previous summary too, +not an ever-growing chain of summary heads. + +**Reset:** discard the selected context, not the transcript. + +```text +commit: + append handoff entry 70 + resetContext([70]) + +after: + transcript [10, 20, 30, 40, 50, 70] + context [70] + +resetContext([]) + → empty model context + → same transcript +``` + +Baseline constraints: + +```text +context shape: optional summary/handoff head + selected entries in transcript order +IDs: existing entries in this conversation or its inherited prefix; no duplicates +replacement: a contiguous PREFIX of the current context, not a consecutive numeric ID range +exchange: replacement/reset must not split an assistant's tool-call/result exchange +concurrency: no replacement/reset during a provider request or unresolved tool exchange +``` + +A summary prepared against an old prefix may still replace it after later appends. A competing +replacement/reset invalidates that prepared prefix. The commit must check this before publishing. + +These operations are not an arbitrary list editor: no middle deletion or unrestricted reordering. +Reset accepts an empty context or a bootstrap head, not an arbitrary permutation of old entries. +Reset alone neither cancels existing tasks nor requests a new model response. + +### 2.4 Forks share history, not future changes + +A historical fork creates a new conversation at a selected historical position. Its transcript +shares the source prefix; new entries belong to the new conversation. + +```text +A transcript: 10 ─ 20 ─ 30 ─ 40 ─ 50 ─ 70 + │ + └────────────── 90 ─ 100 B + +B transcript: [10, 20, 90, 100] + shared local +``` + +Entries 10 and 20 retain their original IDs and owning conversation. There are no copied entry +rows for them. Each conversation still presents a linear transcript. + +The fork inherits the context that existed at its selected position, not today's context truncated +by entry ID: + +```text +P = historical position after the first exchange + +A context at P: [10, 20] +A context now: [70] + +create B from A at P: + B context starts as [10, 20] + NOT [] obtained by filtering today's [70] +``` + +B has its own context list. Later appends, compaction and resets in A cannot change B. Live tasks +are not inherited; continuing B requires explicit new work. + +Here P is a historical position already selected by the caller. §3 defines how selecting a +transcript entry resolves to that position, including the entry-versus-commit distinction. State +initialization and historical inheritance are defined in §4. + +### 2.5 Fork history and task ownership are different relationships + +```text +historical relationship: + conversation A ── fork at P ──> conversation B + +execution ownership: + conversation A ── task T ── owns ──> conversation C +``` + +B is an independent fork. C is an owned child, such as a subagent conversation. A child can start +with fresh or inherited context; that choice is separate from which state gets initialized. + +```text +historical source → where inherited content/state came from +owning task → which execution owns this conversation +``` + +A historical relationship alone does not pull B into A's drive or cancellation scope. Ownership +is explicit; scheduling and cancellation follow the task relationships defined in §§5–6. + +## 3. Identity, commits and historical positions + +One session-global sequence orders mutations. A creation mutation's sequence is the new object's +numeric ID. A complete commit, not an individual mutation, is a historical snapshot boundary. + +### 3.1 Allocate while constructing the command + +```text +last committed sequence: 99 + +build command: + 100 append user entry → entry ID 100 + 101 append context ID 100 + 102 set planMode = true + 103 create generation task → task ID 103 + +persist ONE batch [100–103] +publish the committed changes +return entry ID 100 to caller +``` + +The command can reference IDs it just created. It cannot call external services with them or +return them to callers before persistence succeeds. Storage preserves the supplied numbers. + +```text +command A constructs [100–103] ─ commits ─ command B starts at 104 +command A rejects ─ no write ─ command B starts at 100 +``` + +Numbers are positive safe integers; exhaustion rejects. There are no UUID columns on ordinary +entries, conversations or tasks, and no separately reserved ID ranges. + +### 3.2 Commit boundaries are part of the read result + +Extend the entry snapshot from §2 with its creation commit's end: + +```ts +interface Entry { + readonly commitEnd: number; +} +``` + +The end is known once construction finishes. The caller does not need to enumerate the other +mutations in the commit. + +```text +getEntry(100) + → { id:100, conversationId:1, commitEnd:103, ... } + +getConversation(1, asOf=103) + → context as of that boundary + +getValue(conversation=1, planMode, asOf=103) + → true +``` + +```text +Memory entry metadata contains commitEnd=103 +JSONL commit envelope contains [100,103]; replay derives the entry metadata +SQLite entry row contains commit_end=103, inserted in the same SQL transaction +``` + +Task creation at 103 belongs to the source's durable work. It does not mean a fork inherits that +task. Context and rewindable state are historical; task execution is not rewound. + +### 3.3 Proposed fork contract: after the containing commit + +```text +fork(A, atEntry=100) + → resolve entry 100 to commitEnd 103 + → fork A's transcript/context/rewindable state through 103 + → store source A and boundary 103 + → inherit no live tasks +``` + +This deliberately means **after the commit containing the selected entry**, not an intermediate +state after only mutation 100. No second filter truncates context to IDs <= 100. + +```text +commit [200–203]: + 200 append entry X + 201 append entry Y + 202 append context IDs [200,201] + 203 set value V + +fork at X ─┐ + ├─ same snapshot: includes X, Y and V +fork at Y ─┘ +``` + +This is a proposed semantic choice to review, not merely an indexing trick. A UI must not imply +that X and Y select different snapshots. Strict entry-prefix forks would need a different rule +for associating context/state changes with entries; they must not silently mix a truncated +transcript with later state from the same commit. + +A caller can also select a complete main-commit boundary directly: + +```text +commit [204]: set V = 2 // no transcript entry +fork(A, asOf=204) // selects this state-only change +``` + +Entry and explicit-boundary targets use the same historical query path. A position inside a +commit rejects. A fork requiring usable model context must also select a complete tool exchange; +reject an incomplete exchange rather than silently moving to a different historical position. + +### 3.4 Sequence gaps are normal + +Working writes use the same allocator but do not become retained conversation history. + +```text +100–103 main commit +104–109 working progress +110–112 main settlement, including retirement of that working scope + +retained main history: 100,101,102,103,110,111,112 +next sequence: 113 +``` + +Never use an array offset as the durable identity. A deleted working file does not free its IDs. +Working commits are not fork targets. The backend preserves the global high-water mark even when +working contents are retired; main historical boundaries remain independently identifiable. + +### 3.5 Failure before or after admission + +```text +construction/validation rejection: + no durable writes; no IDs consumed; session remains usable + +persistence outcome uncertain: + stop using this session handle + reopen storage + recover the last complete durable batch and its high-water mark + continue from recovered state, not the old process's guess +``` + +No observer may see half a commit. No effect may treat an uncertain write failure as permission +to retry the same external action with newly allocated IDs. + +### 3.6 Caller-provided identities use ordinary state + +External request keys are optional session-scoped values, not alternate internal IDs. + +```text +accept(input, requestId="client-42"), ONE command: + read session value (pi.request-input, "client-42") + if present: resolve the existing acceptance + otherwise: + create accepted input and its required task/queue changes + set session value (pi.request-input, "client-42") = input ID + +commit succeeds, reply is lost + → caller retries "client-42" + → same accepted input ID, no second input +``` + +Lookup and insertion happen inside the serialized command. Core owns these mappings; normal +callers cannot overwrite them. Conflicting key reuse must reject rather than silently accept a +different request; the exact input-equivalence check belongs with acceptance in §8. Mapping a key +to an input does not answer which eventual result belongs to that input. + +## 4. State + +State is addressed independently of transcript content. Its mutations share the commit sequence, +so a historical boundary selects context and rewindable state together. + +### 4.1 Scope and rewind behavior + +| Address scope | Rewind behavior | Lifetime | +|---|---|---| +| Session | Sticky; one current value/list | Session | +| Conversation | Rewindable or sticky | Conversation and required fork history | +| Named working scope | Never rewound or inherited | Until explicit retirement | + +```ts +const sessionName = sessionValue("pi.session.name"); +const plan = conversationValue("plugin.plan", { rewind: true }); +const expanded = conversationValue("ui.expanded", { rewind: false }); +const moves = conversationList("game.moves", { rewind: true }); + +await conversation.setValue(plan, true); +const enabled = await conversation.getValue(plan) ?? false; +await conversation.appendList(moves, move); +``` + +Addresses bind namespace, optional key, scope and rewind policy. Payload types come from the +address; writes must not infer a different type to accommodate the supplied value. + +```ts +// Value and ValueList denote typed addresses, not wrappers around current data. +interface StateWrites { + setValue(address: Value, value: NoInfer): Promise; + appendList(address: ValueList, value: NoInfer): Promise; +} +``` + +The final interfaces also distinguish address scopes. A conversation address cannot be used as a +session address. Constructors require no global registration; defaults belong at the read site. +Storage persists scope/rewind information rather than depending on token objects surviving restart. +All persisted payloads are strict JSON; callers cannot mutate stored data through returned aliases. + +### 4.2 Small mutation vocabulary + +```text +value: set(value), delete() +list: append(value), clear() +read: getValue(address), readList(address, cursor, limit) +``` + +An append gets a numeric element ID from its mutation sequence. Baseline lists preserve append +order; they do not support replacing individual elements, inserting in the middle or moving them. +Structured editing uses delta values, as illustrated in §4.5. + +```text +rewindable list: + 40 append A + 50 append B + 60 clear + 70 append C + +read through 50 → [A, B] +read through 60 → [] +read through 70 → [C] +``` + +Clear hides older elements at and after its position; it does not erase history required by earlier +forks. Deleting a rewindable scalar similarly records absence. JSON null remains a stored value. + +Session/sticky state exposes only its current contents. JSONL may still contain previous mutations; +the harness does not interpret them as rewindable history. + +### 4.3 Initialization is explicit; a historical fork preserves history + +Creating a new subagent conversation chooses initial state independently of context: + +```ts +await conversation.spawn({ + prompt: "Inspect the parser", + context: "inherit", + values: { + inherit: [model, thinkingLevel], + set: [setValue(activeTools, ["read", "grep"])], + }, +}); +``` + +```text +ONE creation command: + read selected current source values + create child conversation + copy selected present values, then apply overrides + append prompt and create initial generation + +unselected plan mode → absent in child +inherited context → does not imply inherited plugin state +``` + +This initialization copies values, not a live link to parent state. Subsequent reads have no +unselected parent fallback. An absent selected value stays absent unless overridden. The baseline +selection is scalar-only; lists require an explicit initialization policy, not an accidental copy +of an unbounded log. + +A historical fork has a different purpose: + +```text +fork A at P: + rewindable conversation values/lists → inherit all visible history through P + sticky conversation state → copy only explicitly selected current values/overrides + session state → same session-wide state, not copied + working state → never inherited +``` + +The proposed sticky default is no inheritance; callers opt in at creation. Sticky values selected +for copying come from the current source, not P: sticky state is outside the historical snapshot +promise. This is a separate choice from the fork's automatic rewindable inheritance. + +### 4.4 Rewindable lookup follows the fork boundary + +Only local changes need new records. A historical fork does not copy every scalar version or list +prefix into its own storage rows. + +```text +A: plan=false at 40; plan=true at 80 +B: fork A through 60 + +read B.plan: + local version? no + follow source A, capped at 60 + newest version <= 60 is false + +B deletes plan: + local deletion found → absent; do not fall back to A + +B sets plan=true: + local set found → true +``` + +At every ancestor, retain the tighter cutoff: + +```text +readValue(C, address, cutoff): + local = newest local version at or before cutoff + if local is a set: return its value + if local is a deletion: return absent + if C has no historical-state source: return absent + return readValue(C.source, address, min(cutoff, C.sourceBoundary)) +``` + +Lists combine inherited visible ranges and local appends, honoring clear operations and the same +ancestor cutoffs. A list cursor identifies an element position, not an array offset. Fixed asOf +plus cursor/limit gives stable pages while later writes continue. + +Memory/JSONL use indexed arrays; SQLite uses indexed version/range queries. Cost may include fork +depth, but not a scan of unrelated session history. An independently initialized child terminates +fallback; a later historical fork of that child inherits its local history normally. + +### 4.5 Structured state uses deltas and checkpoints + +```text +checkpoint value: { through: 500, state: ... } +delta list: ... 501, 507, 510, 518 ... + +hydrate at P=510: + read checkpoint visible at P + read visible deltas after checkpoint.through through P + apply deltas to privately owned state +``` + +```text +update, ONE command: + read/derive from committed state + append delta batch + optionally set checkpoint including that batch +commit +publish the updated state +``` + +The consumer owns the delta vocabulary, reducer and checkpoint cadence. Chord's delta machinery is +a candidate, not a storage dependency decided here. Storage does not execute plugin reducers. +Checkpointing bounds replay only when its cadence is maintained; it does not authorize deleting +older history while arbitrary historical forks remain supported. A live consumer can retain its +current state; it need not hydrate on every edit. + +### 4.6 Working state is ordinary state with a named lifetime + +```ts +const work = workingScope(String(taskId)); +const frames = list("pi.pending.frames", { scope: work }); +const checkpoint = value("pi.pending.checkpoint", { scope: work }); +``` + +```text +working-only commit: + append frame + set checkpoint + +main settlement commit: + append immutable result assembled from working data + settle task + retireWorkingScope(work) +``` + +Working values/lists survive restart until explicit retirement. They are not transcript entries, +rewindable state, or a separate scratch-storage API. Keys distinguish attempts so recovery cannot +mistake an earlier request's frames for the current request's output. + +```text +allowed commit: main writes, including retirement of working scopes +allowed commit: value/list writes in ONE working scope +rejected: direct main + working writes, or writes in two working scopes +``` + +This lets JSONL persist each batch to one file. Main retirement is authoritative; unlink happens +afterward. SQLite can delete working rows inside the main settlement transaction. + +```text +crash before settlement → task and working data remain recoverable +crash after settlement → result and retirement both exist; leftover file is invisible +close → no retirement +``` + +Before settlement, stop admitting progress from that execution and drain already admitted writes. +Late callbacks must be rejected, not allowed to recreate retired task data. Generic scope names +can be reused deliberately; retirement ends the old contents, not every future use of the name. +Storage never guesses retirement from a missing task, task kind or owner status. + +## 5. Tasks + +A task is durable work associated with one conversation. Its definition supplies behavior; its +stored record supplies the inputs, progress and eventual outcome needed across process lifetimes. + +### 5.1 Task records and lifecycle + +```ts +type TaskRole = "ready" | "inflight" | "waiting" | "terminal"; + +interface Task { + readonly id: number; + readonly conversationId: number; + readonly kind: string; + readonly status: string; + readonly role: TaskRole; + readonly state: State; + readonly after: readonly number[]; + readonly foreground: boolean; + readonly spawnedBy?: number; + readonly ownedConversationId?: number; +} +``` + +Timing and cancellation metadata are added in §6. State is validated JSON, not a promise, closure +or process handle. External handles must have a durable representation that recovery understands. + +| Role | Meaning | Action when covered by a drive and not executing locally | +|---|---|---| +| ready | Can execute once dependencies and timing/permissions allow | Execute | +| inflight | External-effect intent was committed; outcome may be unknown | Recover | +| waiting | A child or external observer supplies the next transition | Restore observation when needed | +| terminal | Immutable outcome | None | + +A definition maps its statuses to these generic roles: + +```text +generation: + pending, retry_wait, deferred → ready + streaming, polling → inflight + done, failed, aborted → terminal + +job: + planned → ready + spawning → inflight + running → waiting + exited, killed, lost → terminal +``` + +The harness derives role from status when validating a write. Storage persists/indexes that generic +metadata so querying live work does not decode every historical task's state or execute kind code. +Role is not independently editable. Terminal records accept no further patches. + +### 5.2 Definitions are replaceable units of behavior + +```ts +interface TaskDefinition { + readonly kind: string; + readonly initialStatus: string; + readonly roles: Readonly>; + readonly transitions: Readonly>; + validateState(value: unknown): State; + execute(task: Task, ctx: TaskExecution): Promise; + recover(task: Task, ctx: TaskExecution): Promise; + abort(task: Task, ctx: TaskExecution): Promise; +} +``` + +TaskExecution supplies the execution signal, working scope and serialized command access. Its +concrete surface belongs in §§6 and 9; it is not another durable object. + +```text +registry: + "generation" → custom generation definition + "tool" → built-in tool definition + "post_tools" → custom exchange policy + "job" → application process integration +``` + +The scheduler does not change when a definition changes. A replacement must understand the +persisted statuses/state of its live tasks; incompatible durable formats need an explicit migration. +Unknown live kinds or invalid live statuses reject restoration before execution begins. + +A task handles ordinary provider/tool failures according to its policy. Storage failures and +invariant violations are not ordinary tool errors: they fault the session instead of being +swallowed and retried indefinitely. + +### 5.3 Dependencies mean terminal, not successful + +These traces assume T has a ready-role status and its timing/permissions allow execution. + +```text +T.after = [A, B] + +A running, B done → T blocked +A failed, B done → T eligible +A aborted, B done → T eligible unless T itself was cancelled +``` + +T interprets the outcomes it requires. The scheduler does not propagate success/failure policy +along edges. Referenced tasks must exist, and dependency edits must preserve an acyclic graph. +Dependencies stay within one execution-ownership tree; independent forks do not become runnable +merely because another tree refers to them. + +```text +spawnedBy = who created this task // provenance +after = which tasks must settle // execution dependency +``` + +These are different. A generation can create a collapse task and then depend on it; that provenance +link does not create a dependency cycle. + +### 5.4 A generation publishes tools and their join atomically + +```text +G returns assistant with calls [A, B] + +ONE settlement command: + append assistant entry + create tool A + create tool B + create P = post_tools, after:[A,B] + settle G + +A settles its own result ─┐ + ├─ P becomes eligible ─ { settle P; create G2 } +B settles its own result ─┘ +``` + +Each terminal command also records known usage and retires its working scope when present. A tool +settles only itself: no sibling inspection, queue draining or next-generation creation. P owns the +exchange decision and remains foreground even after both tools have settled. + +```text +parallel: + A; B; C; P.after=[A,B,C] + +sequential: + A; B.after=[A]; C.after=[B]; P.after=[A,B,C] +``` + +Same tools, same join, different dependencies. No tool-specific scheduler lock is required. + +```text +crash before G settlement → G remains live; normal recovery applies +crash after G settlement → assistant, tools and P all exist +crash after last tool → P already exists; no tail scan or successor repair +``` + +A final-answer generation either creates its explicit continuation in its terminal command or +ends the chain. An idle user-shaped context does not create work by inference. + +### 5.5 A durable wait is not a task-lifetime promise + +```text +DO NOT: + create child task + await child task's entire lifetime inside this effect + +DO: + commit dependency and continuation state + return from this effect + execute the continuation when its dependencies become terminal +``` + +A foreground tool delegating to a subagent can reuse its own task: + +```text +tool execute: + ONE command: + create subagent owner S + set tool status=finishing, after=[S] + return + +S starts child conversation, then returns in waiting status +child final settlement also settles S +existing tool.finishing becomes eligible + → read S's result + → run result hooks + → commit own tool result and terminal status +``` + +Recovery uses stored tasks/dependencies, not reconstructed chains of waitFor promises. Joining this +process's current execution of the SAME task during cancellation is different; §6 defines that +ordering so effect and abort handlers cannot write concurrently. + +### 5.6 Foreground work and detached background work + +Foreground is chosen when work is created, not inferred from a tool name or the transcript tail. + +```text +local foreground roots: + generation, its tools, post_tools, automatic collapse + +required foreground work: + those roots + unfinished dependencies + + required foreground work in their owned conversations + +detached background work: + other live tasks, such as background jobs, schedules and background subagents +``` + +A task needed by a foreground dependency is part of that required work even if it was not created +as a local foreground root. To detach a launch, return its ID instead of creating such a dependency. +Detailed cancellation selection is defined in §6. + +```text +foreground launch: + { create S; tool finishing after:[S] } + parent exchange cannot finish until S supplies a result + +background launch: + { create S; tool result contains S.id; settle tool } + parent exchange can finish while S continues +``` + +Normal conversation abort targets foreground/required work, not detached background work. The +public API also needs cancellation of a selected background operation and all-background shutdown; +these select cancellation scopes, not arbitrary edits to task status. + +### 5.7 Ownership links a task to a child conversation + +```text +conversation A + task S ── owns ──> conversation C + generation, tools, post_tools, ... +``` + +```ts +interface Conversation { + readonly ownerTaskId?: number; +} +``` + +S.ownedConversationId and C.ownerTaskId are reciprocal and commit together. Fork provenance does +not establish ownership. Child completion settles its existing owner in the same command when +appropriate; it does not depend on a later notification manufacturing missing work. + +An ordinary fork has no owner and no copied tasks. A task can create a child, initialize its chosen +state/context, append its prompt and create its first generation atomically. Starting any of those +effects waits until that creation command is durable. + +### 5.8 What every definition must specify + +```text +creation inputs/settings captured; initial status; foreground/dependencies +execution durable intent before external work; bounded working-state writes +parking persisted continuation/dependency/observer state; effect returns +settlement output + usage + terminal state + successors + working retirement +recovery retry/adopt/interrupted policy for every nonterminal status +cancellation stop/join own execution; required result/cleanup; no normal successors +``` + +Normal successor-creating commands re-check current task status and cancellation before committing. +Task-specific cancellation must preserve required result records, such as an error result for an +unexecuted tool call. No definition may depend on the scheduler knowing its domain semantics. + +## 6. Scheduling and cancellation + +The scheduler executes durable task records. A drive call selects where execution is allowed; +it does not create work or become a second owner of tasks already executing. + +### 6.1 Generic scheduling metadata + +```ts +interface Task { + readonly notBefore?: number; // earliest execution time + readonly requiredPermit?: string; // optional host permission, e.g. deferred polling + readonly abortRequested?: true; // durable cancellation membership +} + +interface DriveOptions { + readonly permits?: readonly string[]; + readonly signal?: AbortSignal; // cancels this caller's wait, not durable tasks +} +``` + +The scheduler tests a permit name, not a provider's polling semantics. A retry stores a timestamp +on the existing task, not a promise that must survive restart. + +```text +ready to execute = ready role + + every after dependency terminal + + notBefore reached + + requiredPermit supplied by a covering drive + + not cancelled or already executing +``` + +### 6.2 Drive scopes and outcomes + +```text +conversation.drive() → that conversation and descendants reached through task ownership +harness.drive() → all independent conversation trees + +fork provenance → never expands a drive scope +``` + +Overlapping callers share one scheduler. Permissions apply only inside the caller's scope; a +polling permit for one tree must not authorize unrelated trees. + +| Outcome | Conversation drive | Session drive | +|---|---|---| +| idle | No required foreground work or selected cancellation remains in its tree | No live tasks anywhere | +| suspended | Work remains but needs an outside event/permission | Same | +| closed | Host closed before the requested condition | Same | + +A future timer is local progress: wait for it rather than report suspension. An installed human +or process observer can leave the drive suspended. The observer may commit its task's outcome; +subsequent task execution requires a covering drive again. + +Foreground completion is rooted at the requested conversation: foreground tasks inside a detached +child do not make its parent busy. Driving that child directly does wait for its own foreground. +A conversation drive services background tasks while active, but does not wait solely for detached +work. Hosts supervising background work drive the session and resume after relevant external events. + +### 6.3 One execution claim per task + +```ts +interface ExecutionClaim { + readonly controller: AbortController; + readonly promise: Promise; +} +const executing = new Map(); +``` + +This map owns actual executions, not a mirror of stored tasks. Waiting observers and their re-arm +markers exist only for live waiting phases; discard them on phase exit, settlement or close. +Determine scope membership from live task queries and ownership ancestor point reads, not by +walking every completed child conversation ever created. Temporary ancestor lookups can be shared +within a pass without becoming a permanent resident model. + +```text +scheduler pass: + read live tasks in covered scopes, using bounded indexed queries + signal EVERY affected executing cancellation target + reconcile cancellation targets under exclusive claims + + for each unmarked, unclaimed task: + ready → test dependencies/time/permit; claim and execute + inflight → claim and recover + waiting → if not armed for this waiting phase: claim and recover to re-arm + + derive caller outcomes from committed state + await commit | execution completion | next timer | outside notification +``` + +Claim before launching. Do not await an entire effect inside the scan before considering the next +task. Initial execution can install its observer; recovery must not install a duplicate. A waiting +owner whose child supplies completion needs no subscription and can simply mark re-arming complete. + +Effects and observers re-check current task status, cancellation and their execution validity on +the command line before writing. Events wake checks; they are not durable continuation messages. +An unchanged ready task must not be relaunched forever after an unexpected exception: report it +and fault the handle. Ordinary retry belongs in a committed task transition. + +### 6.4 Cancellation selects work, then drives cleanup + +```text +conversation.abort(): + select current required foreground work (§5.6) + +conversation.abortBackground(taskId): + require a detached background root in this conversation + select it, unfinished dependencies, and all live work in their owned subtrees + +conversation.abortBackground(): + apply that selection to detached background work in this ownership tree +``` + +The background handle is a task ID, but this is scope cancellation, not permission to terminalize +an arbitrary node or skip its cleanup. A task currently required by foreground work is not detached; +use conversation abort for that chain. Further dependency-sharing policies remain a review point. + +```text +ONE cancellation command: + capture selected task IDs; mark every nonterminal target abortRequested + for foreground abort, cancel currently queued steer/followUp in affected conversations + keep context-only writes and nextRun input + +repeated abort while the same foreground cancellation is outstanding: + join existing cancellation; do not drain newly arrived input again +``` + +The marks survive restart. Do not shrink selection as earlier tasks settle. New foreground input +queues while cancellation is outstanding; normal successors and owned-child creation cannot escape +a marked task. Background cancellation also prevents new work beneath its marked owner. + +### 6.5 Join an execution before invoking its abort handler + +```text +for each marked task: + keep/acquire ONE exclusive execution claim + signal and await its current local execution, if any + report its rejection; storage faults forbid further writes + re-read committed task + if already terminal: release claim + otherwise: invoke definition.abort under the SAME claim + release only after cleanup returns +``` + +Signal all affected executions before waiting on a slow one. Unstarted tasks can abort without +waiting for their normal after dependencies. An uncooperative effect can delay its own cleanup; +a timeout must not silently admit a second writer for that task. + +A marked child-conversation owner settles only after its selected child work is terminal. The +scheduler waits for that condition, not an effect-held child-lifetime promise. Child completion +checks the owner's cancellation mark before publishing success. Direct foreground abort of an +owned child must also complete its owner as cancelled. That child drive may finish cancellation +of its marked owning task after selected child work is terminal, even though the owner record is +in the parent. It must not start other parent effects; parent-side continuation requires a covering +drive of the parent. + +```text +post_tools settlement wins: + { settle P; create G2 } → abort selects G2 + +abort wins: + { mark P } → P.abort settles without G2 +``` + +Task-specific error/result records still belong to task handlers. Cancellation does not synthesize +provider messages or interpret tool arguments in the scheduler. + +### 6.6 Open, close, shutdown and deletion + +```text +open: + open storage; validate live task definitions + expose inspect(), state and transcript queries + start no provider, tool or process effects + +close: + seal public/effect admission + signal local executions; stop observers/progress admission + wait for cooperative executions and admitted persistence + release storage; write no cancellation or terminal outcomes + +harness.shutdown: + seal public admission but allow internal cancellation commands + mark ALL live session work, including background and owned descendants + cancel all queued inputs, including write/nextRun; preserve their immutable history + drive selected cancellation to settlement + close +``` + +Close itself adds no outcomes; commands already admitted before its barrier can still complete, +including terminal settlements. It interrupts local execution, not necessarily an external process +that can be adopted later. Shutdown uses each kind's abort policy to stop external work where supported. A crash during +shutdown leaves durable cancellation marks for the next owner to reconcile. + +Observer-wait cancellation only removes that caller's drive interest; it never signals shared +executions as if the conversation were aborted. Close remains cooperative and may block on a +broken external integration. Bounded-close/forced-process policies need an explicit host contract. + +Deleting a conversation is a separate atomic mutation: reject while its ownership subtree has +live work. Once deleted, reject new work there. Preserve immutable history and source metadata +needed by independent forks; logical deletion is not permission to erase their inherited data. + +## 7. Storage + +One storage interface answers entity/state queries and commits atomic mutation batches. There is +no public journal reader, separate read-index interface, scratch store or payload-address system. + +### 7.1 Store the information queries need + +Complete the conversation snapshot with its historical source and context-replacement revision: + +```ts +interface Conversation { + readonly source?: { readonly conversationId: number; readonly asOf: number }; + readonly inheritState: boolean; + readonly prefixRevision: number; +} + +interface Entry { + readonly metadata?: JsonObject; // small preview/attribution/exchange fields + readonly key?: string; // optional indexed content key, not a state address + readonly byTaskId?: number; +} +type EntryHeader = Omit; +``` + +Source links cap inherited transcript access. Historical forks also set inheritState=true; +independently initialized children set it false even when inheriting context. Owner links remain +separate. Source and ownership identity are not arbitrary mutable configuration. + +Current context is materialized. Its historical edits are retained. Prefix replacement/reset bumps +prefixRevision; append does not. Task role, conversation ID and scheduling fields are queryable +without decoding kind-specific state. Entry metadata permits exchange checks without loading large +message/image content. + +### 7.2 Queries, motivated by callers + +| Caller | Required read | Must not do | +|---|---|---| +| Scheduler | Live tasks in a scope; dependency IDs in one batch | Load terminal task history | +| Generation | Current context IDs, then those entries in a batch | Reconstruct context by scanning transcript | +| UI | Transcript page before/after an ID, with a limit | Load the whole conversation | +| Validation | Named task/conversation records and entry headers | Load unrelated message content | +| Fork | Entry commit boundary; historical context/state | Truncate today's context | +| State consumer | Latest value/version; bounded list range | Replay unrelated journal records | +| Background status | Task and named result entry | Keep an ancient task resident forever | +| Reopen on SQLite | Live tasks and required conversations | Replay all historical task transitions | + +Filters apply before limits. Latest matching record means an indexed descending query with limit=1, +not loading a collection and taking its last item. Conversation batch reads matter just as entry +and task batch reads do. + +```ts +interface CursorPage { + readonly after?: number; + readonly before?: number; + readonly order?: "asc" | "desc"; + readonly limit: number; +} +interface Page { + readonly items: readonly T[]; + readonly asOf: number; + readonly next?: number; +} +interface ConversationQuery extends CursorPage { + readonly sourceConversationId?: number; + readonly ownerTaskId?: number; +} +interface EntryQuery extends CursorPage { + readonly conversationId: number; + readonly kind?: string; + readonly key?: string; + readonly asOf?: number; +} +interface TaskQuery extends CursorPage { + readonly conversationIds?: readonly number[]; + readonly live?: boolean; + readonly role?: TaskRole; + readonly kind?: string; + readonly foreground?: boolean; + readonly abortRequested?: boolean; +} +interface ListQuery extends CursorPage { readonly asOf?: number } +interface StoredValue { readonly seq: number; readonly value: T } +interface ListElement { readonly id: number; readonly value: T } +interface CommitBoundary { readonly first: number; readonly last: number } +``` + +Transcript queries return the effective inherited prefix plus local entries; each entry retains +its original conversationId. Tasks are never inherited by that query convention. Historical state +uses §4's capped source traversal. A page cutoff is not a generic timestamp for every data class: +sticky state and task scans expose current state, not historical snapshots. Transcript/rewindable +pages carry a main boundary; current task/sticky/working pages carry a global observation cutoff, +which is not a fork target. Historical reads on sticky/working addresses reject. Coherent current +multi-page reads use the command line; a cursor alone does not freeze mutable sticky contents. + +### 7.3 One interface, no second transaction layer + +```ts +interface Storage { + readonly head: number; // global allocator high-water mark + readonly mainHead: number; // last main commit, not another allocator + commit(batch: CommitBatch): Promise; + + getConversations(ids: readonly number[], asOf?: number): Promise>; + scanConversations(query: ConversationQuery): Promise>; + getEntries(ids: readonly number[]): Promise>; + getEntries(ids: readonly number[], options: { content: false }): Promise>; + scanEntries(query: EntryQuery): Promise>; + getTasks(ids: readonly number[]): Promise>; + scanTasks(query: TaskQuery): Promise>; + + getValue(address: Value, asOf?: number): Promise | undefined>; + scanValues(query: ValueQuery): Promise; + readList(address: ValueList, query: ListQuery): Promise>>; + close(): Promise; +} +``` + +ValueQuery is a bounded scope/namespace scan ordered by address key; its cursor is an address key, +not a numeric entity cursor. ValuePage returns addresses with their current visible values. No +arbitrary predicates, plugin reducers, user comparators or general SQL expressions cross this API. +State tokens arrive with their conversation/working identity bound; scope-specific receiver shapes +are shown in §9. + +Each logical read, including ancestor traversal, observes one committed state. The session command +line supplies coherent multi-read operations and read-modify-write; no second public storage +transaction callback is needed. Historical context reads affect context, not current sticky state. +Tasks remain current even when their originating entries are old. + +### 7.4 Atomic mutations + +```text +main mutation vocabulary: + createConversation, deleteConversation + appendEntry, createTask, patchTask + appendContext, replaceContextPrefix, resetContext + setValue, deleteValue, appendList, clearList + retireWorkingScope + +working mutation vocabulary: + setValue, deleteValue, appendList, clearList +``` + +```ts +interface CommitBatch { + readonly scope: "main" | { readonly working: string }; + readonly writes: readonly Write[]; // tagged mutations above; each has its final seq +} +``` + +```text +harness command: + read committed state; construct mutations and IDs + validate against committed state + earlier writes in this batch + no mutations → return builder result without a storage commit + otherwise persist one batch + notify observers; resolve caller + +storage commit: + check scope, supplied sequences and structural constraints + prepare only touched data/index changes + persist atomically + publish new query-visible state and heads together +``` + +Task definitions validate status transitions/state; the harness enforces ownership, dependency, +context and admission rules. Storage stores the validated generic records without running kinds. +Rejected construction changes nothing. An admitted failure faults the handle (§3.5). Cancelling a +caller after persistence admission must not abandon the transaction or expose half a batch. + +### 7.5 Memory and JSONL: the same queries, with optional files + +```text +current maps: conversations, tasks, scoped values/lists +ordered data: entries per conversation, state versions, context edits +lookup maps: IDs, addresses, live-task membership, source/owner relationships +``` + +Memory applies prepared changes without exposing intermediate state. Indexes may reference the +same immutable payload objects; they need not duplicate content. JSONL uses that representation +and loads it fully on open. + +```json +{"first":100,"last":103,"writes":[ + {"type":"appendEntry","conversationId":1,"kind":"user","content":"Inspect"}, + {"type":"appendContext","conversationId":1,"ids":[100]}, + {"type":"setValue","conversationId":1,"namespace":"plugin.plan","value":true}, + {"type":"createTask","conversationId":1,"kind":"generation","status":"pending"} +]} +``` + +Shown across lines for reading; the file contains one complete line per batch. This is illustrative +encoding, omitting address/task fields. A compact encoding may derive each stored seq from first+i; +logical writes already carried their final IDs before encoding. + +```text +JSONL commit → encode whole batch → write all bytes + newline → publish in-memory changes +JSONL open → validate complete batches → rebuild maps/indexes → expose queries +``` + +Do not publish in-memory changes before the write completes. Handle short writes. Discard only an +unterminated final batch; malformed complete lines fail open rather than silently deleting history. + +```text +session.jsonl main batches +session.scopes/ working batches for one named scope +``` + +On open, compute global high-water from all complete main/working batches before cleanup. Main +retirement hides that scope's records through its retirement seq. Later deliberate reuse survives; +serialize old unlink before reuse so delayed cleanup cannot delete a new lifetime. Only explicit +retirement authorizes cleanup. Process-crash durability is required; fsync/power-loss policy is an +explicit backend option, not a claim that ordinary writes are immune to power loss. + +### 7.6 SQLite: queryable rows, not mandatory raw-journal replay + +```text +conversations / current_context_elements +entries +tasks // current records, including terminal outcomes +current_values / value_versions +list_elements / historical_clear_markers +context_edits +commit_boundaries / session_metadata +``` + +Every shared-container key starts with session ID. Suggested indexed suffixes: + +```text +entries (conversation, id), (conversation, kind, key, id) +tasks (role, id), (conversation, role, id) +values (scope, namespace, key) +value_versions (conversation, namespace, key, seq) +list_elements (scope, namespace, key, id) +context_edits (conversation, seq) +conversations (sourceConversationId), (ownerTaskId) +``` + +Live scans also need ordered partial indexes, for example (session_id, id) WHERE role != 'terminal', +so scanning across nonterminal roles does not sort all task history. Additional combinations follow +actual query plans, not an index for every possible filter. + +```sql +-- One local step in a rewindable scalar lookup. +SELECT seq, deleted, value +FROM value_versions +WHERE session_id = :session AND conversation_id = :conversation + AND namespace = :namespace AND key = :key AND seq <= :cutoff +ORDER BY seq DESC LIMIT 1; +``` + +```text +BEGIN + validate global head + insert entry 100 with commit_end=103 + append context element 100; retain context edit 101 + insert rewindable value version 102 + insert current task 103 with indexed role + record main boundary 103; advance global head +COMMIT +``` + +No full transcript/context rewrite per append. Current context can use ordered element rows; +prefix replacement removes the selected prefix and inserts its head without rewriting the retained +suffix. A historical fork stores a source link and its initial context, not copies of source entry +or value/list inventories. Sticky updates overwrite current contents; working retirement deletes +working rows inside the settlement transaction. Retain only the histories the API promises. + +### 7.7 Historical context is not today's context minus later IDs + +```text +contextAsOf(C, P): + start from latest reset <= P, otherwise C's initial context + page later context edits through P in order + append → extend + replace → replace named prefix, keeping the existing suffix + reset → replace whole list +``` + +A replacement is not a full checkpoint: it says nothing about the retained suffix's earlier +appends. The baseline replay may span from creation/reset; it is correct but not necessarily cheap +for ancient forks. Indexed context checkpoints can optimize it later if measurements justify them. +Current generation requests use the materialized context instead of this historical path. + +## 8. Built-in task flows + +These are task definitions and boundary policies, not scheduler special cases. Braces below group +one main commit. Every attempt outcome includes its reported usage; terminal outcomes retire their +working data. Hooks and external calls run outside the command line. + +### 8.1 Acceptance, placement and result ownership + +Proposed input representation: use an immutable admission entry as the input identity. Idle input +can be a selected user entry directly; busy input is unselected until a later placement entry. + +```text +idle accept A: + { user A with inline message; select A; G.inputs=[A]; result[A]=running } + +busy accept B: + { unselected input B with inline message; enqueue B; result[B]=queued } + +later consume B: + { user placement U referring to B; select U; dequeue B; + create G2.inputs=[B]; result[B]=running } +``` + +The placement is new, so context remains in transcript order. Its message is resolved from B's +immutable content, not copied into another stored payload. Projection batch-loads such references; +placement references point directly to admission entries, never chains of placements. UI rendering +can distinguish arrival from placement rather than displaying the same message twice. + +Queues use ordinary sticky values under conversation namespaces, one address per queued input. +Separate namespaces identify queue modes; fixed-width decimal input-ID keys preserve numeric +admission order in bounded key scans. Dequeue deletes the current queue value; it does not require +rewriting a growing inbox array or retaining consumed queue rows. +Session-scoped input-result values preserve each acceptance's outcome independently of that queue. + +```text +G1.inputs=[A] +B queued while G1 runs +G1 produces final answer R + +{ result[A]=done(R); settle G1; + place B; dequeue B; G2.inputs=[B]; result[B]=running } + +result(B) is NOT R just because R was appended after B's admission +``` + +At a tool boundary, newly consumed steering joins the current pending input group. A yield hook's +continuation also retains that group. Only a completed answer/stopping boundary resolves it. +Follow-ups consumed after an answer start a new group. Several inputs can legitimately share one +answer. Carry input IDs through generation/post_tools state, not through a new durable run object. + +```ts +type InputResult = + | { readonly status: "queued" | "running" } + | { readonly status: "done" | "placed"; readonly entryId: number } + | { readonly status: "cancelled" | "failed" | "stopped"; readonly reason?: string; readonly entryId?: number }; +``` + +Placed is the terminal receipt for a context-only write, not an assistant answer. Cancelling a +queued input records cancelled; cancelling an active chain resolves its pending inputs in the +terminal boundary command. Retrying/deferred tasks leave them running. Session-wide request keys +resolve the original admission; reusing a key for another conversation or different normalized +semantic input rejects. Harness-generated metadata is not part of input equivalence. + +This representation/result contract is proposed for review. It avoids chronological attribution, +a new input table, and duplicate message payloads; it does add placement records for queued input. + +### 8.2 Queue boundaries + +| Input mode | When it can be placed | Requests generation? | +|---|---|---| +| write | Next safe context boundary | No | +| steer | Next tool or final-answer boundary; explicit idle acceptance | Yes | +| followUp | Normal final-answer boundary; explicit idle acceptance | Yes | +| nextRun | Next explicit idle acceptance only | Yes | + +```text +idle acceptance: + place eligible queued items, then new input + create ONE generation with all selected trigger IDs + +post_tools: + place safe writes and selected steering + continue current input group unless cancelled/terminated + +final-answer generation: + if yield hook continues: retain current input group + otherwise resolve current group with this answer + eligible steer/followUp → place them and create next group/generation + +failure/termination: + resolve current group; do not consume triggers with no successor + +foreground abort: + cancel queued steer/followUp; preserve write/nextRun +``` + +Steering/follow-up modes select all eligible items or one oldest item of each mode. Preserve +admission order among selected items. Safe write placement never becomes a hidden run request. +Selection, placement, dequeue, result-state changes and required generation creation share a commit. +Queued input takes precedence over an old yield decision; stale hook preparation is discarded. + +### 8.3 Generation + +```text +create: + capture model, thinking, tools, provider options and retry policy + persist inputs and pending status + +pending/retry execution: + check current status/cancellation and request/exchange exclusion + completed captured collapse → interpret its outcome once; do not recreate it on every retry + live collapse → attach dependency and return + automatic collapse needed → create C and G.after=[C] together; return + capture main historical context boundary; commit streaming intent + project context; apply before_request; re-check execution validity + call provider; persist frames under this attempt's working keys + +outcome: + calls → { assistant; tool tasks; post_tools with inputs; settle G } + final answer → { assistant; resolve/continue inputs; settle G; required successor } + deferred → { usage; same G.deferred(handle), requiredPermit="deferred-poll" } + retryable → { usage; same G.retry_wait, attempt+1, notBefore } + overflow → { usage; same G waiting on one captured collapse retry } + failure → { partial/error if needed; resolve inputs failed; settle G } +``` + +A generation task exists throughout preflight/collapse/retry. Retries do not create a replacement +with a fresh budget. Polling records intent before fetching the stored handle; another deferred +response updates the same task. Transitions clear permit/timing metadata that no longer applies; +a normal retry must not accidentally retain a deferred-poll requirement. Poll permits and retry +timestamps remain generic scheduler data. +Known usage commits even for failed, deferred or discarded attempts; no report means unknown cost, +not zero and not an exactly-once billing guarantee. + +```text +recover polling → deferred; retain handle and budget +recover streaming → committed partial: publish interrupted outcome + required call errors + no partial: retry within budget or fail +abort → drain frames; best-effort provider cancellation; required partial/results; + resolve pending inputs cancelled; no tools or normal successor +``` + +The interrupted-stream policy and provider-specific retry/overflow classification need parity +review. Do not catch persistence/invariant failures as provider errors. A never-started generation +has no fabricated assistant response. Stop reasons containing unusable/truncated calls still need +error results without invoking those tools. + +### 8.4 Tool execution and post_tools + +```text +tool execute: + validate offered-tool authorization and arguments + before_tool → allow | block | hold + allow → commit running intent with effective args and replay policy + invoke tool with execution signal, identity and bounded sink + block/truncated/missing → prepare own error result without external invocation + run after_tool outside line + { own result; usage; active-tool additions; control intent; settle tool } +``` + +Registry presence is not authorization: the generation must have offered the tool. Persist the +actual arguments/replay decision before invocation. Ordinary throws produce error results, not +cancellation. Capture terminate/handoff intent in the outcome; the tool does not reset context or +inspect siblings. Foreground delegation uses the finishing phase in §5.5. + +```text +recover → replay only when captured AND current tool policy permit it; + otherwise interrupted result from checkpoint, no uncertain control side effects +abort → join own execution; drain working writes; own aborted error result +``` + +```text +post_tools, ONE command: + require all calls terminal; re-check its cancellation + read outcomes and choose exchange policy + stopped → resolve current inputs stopped/cancelled; no successor + continue → place eligible writes/steering; apply valid handoff; create G2 with inputs + settle post_tools + +post_tools.abort: + { resolve its pending input group cancelled; settle post_tools } + no context edit, trigger consumption or successor +``` + +A new_context tool returns handoff intent. The join appends/selects that handoff and resets context +only after the exchange is complete. Ordinary errors can still be followed by generation. The draft +uses any terminate request to stop; conflicting handoffs reject the handoff rather than choosing +completion order. These policies are reviewable independently of the join mechanism. + +### 8.5 Collapse, including speculative completion + +```text +create C: + capture complete prefix, through, prefixRevision, historical context and model/settings + allow one live collapse per conversation; generation can depend on that existing C + +execute: + commit summarizing intent → call summarizer + persist complete candidate, including reported usage, in working scope + { usage; candidate reference/phase identifying that the attempt outcome was recorded } + editable context + matching prefix → { summary; replace prefix; settle C } + stale prefix/cancelled → terminal outcome without context publication + context busy → park C waiting for an editable-context predicate +``` + +The parked task restores an observer that checks committed context/request state, then makes C +ready to publish. Events only wake that check. Re-check prefix and editability in the publishing +command; do not record usage twice. + +**Do not wait for a generation's terminal status to make context editable:** a retry can keep the +same generation alive while it waits for C. That would form a deadlock. The context predicate, +not the generation's lifetime, controls speculative publication. + +```text +G retry_wait after:[C] → no provider request in flight +C sees editable context → publish and settle +G becomes eligible → retry with new context +``` + +Recovery reuses a durable complete candidate or retries within budget. Abort records known usage +and settles without replacement. C never creates G. Failed/declined overflow compaction ends the +bounded overflow retry; threshold/manual failure can leave G using unchanged context. + +### 8.6 Subagent and approval + +```text +subagent owner S: + { child with owner=S; selected initial state; prompt; first generation; S waiting } + return + +child terminal foreground boundary: + { its output/input outcomes; terminal child task; settle S with result reference } +``` + +For inherited subagent context, the proposed default is the last complete-exchange boundary before +the launching exchange, not its unresolved calls. Selected initial values still come from the +creation command. This is initialization, not a historical fork claiming one state/context snapshot. +Recovery drives the existing child tasks; it does not recreate the child or await a child-lifetime +promise. Cancellation completion follows §6.5 and must not report success for a marked owner. + +```text +hold tool: + { create approval A; tool waiting after:[A] } + +grant: + { A granted; matching tool ready with approved arguments } + +deny: + { A denied; matching tool finishing with denial outcome } + tool runs its result hooks, writes own result and settles +``` + +Approval recovery restores the human decision surface; it never grants automatically. Cancellation +marks the approval and its required tool independently. Grant/deny versus abort is serialized; +late decisions cannot restart terminal work. + +### 8.7 Jobs and schedules + +```text +job: + { spawning intent with durable adoption/log identity } + spawn or reconcile existing process + { running with verified process identity } + install output/exit observers; return + +exit: + drain output → { immutable result; usage if reported; terminal job } +``` + +Recovery adopts the same process, or records lost if its outcome cannot be recovered. PID alone is +not identity. Abort stops the process through its host service, drains output and records killed. +Close detaches local observation; an adoptable external job can survive it. Process launch/exit +reconciliation and log retention are explicit host capabilities, not storage guesses. + +A recurring schedule needs a stable cancellation handle, not a trail of successor IDs: + +```text +schedule S ready at notBefore: + { create action J; S collecting after:[J] } + return + +J terminal → S collecting: + { record outcome reference; + S scheduled with next notBefore and after:[], or terminal if complete } +``` + +The action can be a job or subagent owner. S remains the background root; cancellation includes +its unfinished dependency J and owned work. Recurrence and missed-run policy are persisted; +positive intervals and bounded catch-up prevent a restart from launching an unbounded backlog. +Recovery uses the stored timestamp/dependency; it must not create a second J while collecting. +Abort settles S without recurrence. Captured cancellation marks keep its action/owned work selected +even if S settles first; S does not hold an abort promise across J's lifetime. + +### 8.8 Hooks are preparation or observation, not another scheduler + +| Point | Decision | +|---|---| +| before_request | Transform the prepared request | +| after_response | Observe provider outcome | +| before_tool | Allow, block with result, or hold for approval | +| after_tool | Process/observe this tool's result | +| on_yield | Stop or request explicit continuation | +| before_collapse | Allow with instructions or decline | + +Run hooks outside the command line, then revalidate their decisions before committing. Hooks may +repeat after a crash; external hook side effects need their own idempotence. Commit listeners, +unlike hooks, must not await another command on the same serialized line. + +Hook ordering, transformation rights and exception policy need an explicit compatibility decision; +the task chain does not settle those questions. Cancellation suppresses yield/normal successor +creation even when a hook was already preparing a continuation. + +## 9. Public API and observation + +The earlier shapes describe stored snapshots and behavior. Handles issue commands; they are not +stored conversation objects. These sketches assemble those surfaces without requiring a second +state model or exposing backend layout. + +### 9.1 Conversation and session handles + +```ts +interface Accepted { readonly inputId: number } +type HistoryTarget = { readonly atEntry: number } | { readonly asOf: number }; +type DriveOutcome = "idle" | "suspended" | "closed"; + +interface ConversationHandle { + readonly id: number; + snapshot(): Promise; + accept(input: AgentInput, options?: { requestId?: string }): Promise; + result(inputId: number): Promise; + drive(options?: DriveOptions): Promise; + prompt(input: AgentInput, options?: { requestId?: string }): Promise<{ inputId: number; result: InputResult }>; + + appendMessage(message: AgentMessage): Promise; // context only; may queue + steer(input: AgentInput): Promise; + followUp(input: AgentInput): Promise; + nextRun(input: AgentInput): Promise; + cancelQueued(inputId: number): Promise<"cancelled" | "already_consumed" | "not_found">; + + abort(): Promise; // durable intent; drive performs cleanup + abortBackground(taskId?: number): Promise; + collapse(options?: { instructions?: string }): Promise; + resetContext(ids: readonly number[]): Promise; + fork(target: HistoryTarget, options?: { sticky?: Initialization }): Promise; + rewind(target: HistoryTarget, options?: { keepRunning?: boolean; sticky?: Initialization }): Promise; + spawn(options: SpawnOptions): Promise; // subagent owner ID; no effect starts here + + getValue(address: ConversationValue, asOf?: number): Promise; + setValue(address: ConversationValue, value: NoInfer): Promise; + deleteValue(address: ConversationValue): Promise; + appendList(address: ConversationList, value: NoInfer): Promise; + clearList(address: ConversationList): Promise; + readList(address: ConversationList, query: ListQuery): Promise>>; + command(build: (tx: ConversationCommand) => T | Promise): Promise; +} + +interface Harness { + root(): Promise; + conversation(id: number): Promise; + conversations(query: ConversationQuery): Promise>; + inspect(): Promise<{ ready: readonly Task[]; inflight: readonly Task[]; waiting: readonly Task[] }>; + drive(options?: DriveOptions): Promise; + entry(id: number): Promise; + entries(conversationId: number, query: CursorPage): Promise>; + task(id: number): Promise; + getValue(address: SessionValue): Promise; + setValue(address: SessionValue, value: NoInfer): Promise; + deleteValue(address: SessionValue): Promise; + appendList(address: SessionList, value: NoInfer): Promise; + clearList(address: SessionList): Promise; + readList(address: SessionList, query: ListQuery): Promise>>; + command(build: (tx: Command) => T | Promise): Promise; + command(scope: WorkingScope, build: (tx: WorkingCommand) => T | Promise): Promise; + watch(conversationId: number, options: { limit: number }, listener: (event: WatchEvent) => void): Promise<() => void>; + decideApproval(taskId: number, decision: "grant" | "deny"): Promise; + deleteConversation(id: number): Promise; + shutdown(): Promise; + close(): Promise; +} +``` + +AgentInput is string/user-message/message-batch input normalized at acceptance. AgentMessage and +provider model/usage types reuse the existing agent/pi-ai boundary. Initialization is the typed +scalar selection/override shape in §4.3; SpawnOptions adds prompt and fresh/inherit context. +These are proposed signatures, not new copies of provider message types. Direct spawn creates a +detached owner; foreground tool delegation makes that work required through its dependency. + +Prompt accepts, drives while observing that input, and returns its result when terminal or progress +suspends/closes. It never substitutes another input's answer. Rewind normally requests/drains source +foreground cancellation before forking; keepRunning leaves source work alone. Validate the target +before cancellation so an invalid history request does not abort otherwise valid work. + +### 9.2 Commands return results only after commit + +Command exposes the mutations from §7.4 and read-only storage queries. Creation/list-append methods +return final numeric IDs immediately inside the builder; the outer command resolves only after +persistence and publication. ConversationCommand binds the conversation automatically. WorkingCommand +exposes only value/list writes, checked against one exact scope ID. + +```ts +await conversation.command(async tx => { + const current = await tx.getValue(plan) ?? false; + tx.setValue(plan, !current); +}); + +const entryId = await harness.command(tx => { + const id = tx.appendEntry(conversationId, "note", content); + tx.setValue(labelAddress(id), "checkpoint"); + return id; +}); +// entryId can escape here, not during the builder +``` + +Reads see pre-command committed state; validation applies buffered mutations in order. Buffer +construction does not perform external effects or publish plugin state. A throwing builder discards +its mutations. Runtime checks supplement token typing for strict JSON, address policy and exact +working-scope identity; changing an established address's rewind policy is not an implicit migration. + +### 9.3 Task and tool integration + +```ts +interface TaskExecution { + readonly signal: AbortSignal; + readonly workingScope: WorkingScope; + readonly services: TaskServices; + command(build: (tx: Command) => T | Promise): Promise; + command(scope: WorkingScope, build: (tx: WorkingCommand) => T | Promise): Promise; + observe( + subscribe: (notify: (event: Event) => void) => () => void, + reconcile: (event: Event | undefined, ctx: TaskExecution) => Promise, + ): void; +} + +interface Tool { + readonly name: string; + readonly description: string; + readonly parameters: JsonObject; // provider-compatible JSON Schema + readonly replay?: "safe" | "never"; + execute(args: JsonObject, ctx: ToolExecution): Promise; +} +``` + +TaskServices supplies models, registered tools, hooks and host resources. ToolExecution supplies +invocation/call identity, the signal and bounded output/checkpoint/memo access using ordinary working +values/lists. The tool schema validates arguments before invocation. ToolDelegation identifies a +subagent/job launch and foreground/background choice; it does not contain an in-process promise. + +```text +tool output surface: + write text / add image → durable bounded progress; capped final result + set details / report usage → final result/attempt metadata + addTools / terminate / handoff → control intent interpreted at settlement/boundary + checkpoint / typed memo → ordinary working values, retired with that execution +``` + +Output backpressure must be explicit: await admitted progress or use a bounded buffer drained before +settlement. Delta frames/checkpoints must not silently rewrite an entire growing assistant message +for every token. Caps, image handling and durable log references are part of tool policy. + +Observe installs a live-phase subscription, not another executor. Each reconciliation receives a +fresh execution context/signal under the same per-task claim used by effects/recovery/abort. Register +before an initial reconciliation with event=undefined, so an already-satisfied condition is not lost. +Events arriving during a claim queue/coalesce within a bounded policy. Phase exit/cancellation seals +and drains admitted callbacks. A stale callback cannot write through an expired context. Waiting +recovery restores missing subscriptions, not duplicates. Teardown and settlement are distinct. + +### 9.4 Watch committed state without gaps + +```text +ONE serialized step: + capture main/global observation boundaries + read conversation, bounded transcript page, live task metadata, + queued-input state and requested reduced working previews + register listener + +then: + deliver relevant main commits as whole batches + deliver committed working previews and transient notifications separately +``` + +```text +base { conversation, entries, referencedInputs, tasks, queued, previews, asOf } +commit { firstSeq, lastSeq, changes:[...] } +preview { taskId, kind, value } +signal { kind:"fault" | "handler_error" | "idle", ... } +``` + +This union is WatchEvent. Entry pages use immutable IDs and a cutoff for deduplication. Commit +filtering can omit unrelated changes but must preserve batch boundaries; clients publish after +reducing the complete delivered batch. A terminal task and its successor must not appear as an +observable idle gap. + +Batch-load admission payloads referenced by visible placement entries into referencedInputs; they +need not appear as extra transcript rows. Arbitrary plugin state is hydrated by its explicit state +consumer, not automatically by a transcript watch. A commit listener cannot await a new command on +the same line; schedule it after delivery. Listener exceptions do not undo commits. Slow-consumer +buffering is bounded; a disconnected/lagging client can rebind from a fresh base. + +If a Chord adapter needs consecutive revisions, those belong to its live binding: + +```text +main commits 104, 117, 130 → binding updates 1, 2, 3 +rebind/rewind → fresh base, update counter starts again +``` + +Journal gaps are not transport gaps. Do not persist another state-sequence allocator. The adapter +publishes only after durable commit and rejects events from retired bindings; its concrete codec +and private-draft integration remain to be chosen. + +### 9.5 End-to-end usage + +```ts +const h = await Harness.open(storage, hostOptions); +const c = await h.root(); +const off = await h.watch(c.id, { limit: 100 }, render); + +const accepted = await c.accept("Inspect the parser", { requestId: "web-42" }); +const outcome = await c.drive(); +const result = await c.result(accepted.inputId); +if (result?.status === "done") show(await h.entry(result.entryId)); +// suspended: wait for the needed event/permission, then drive again +``` + +```ts +// Cancellation is durable intent; keep or start a drive for cleanup. +const driving = c.drive(); +await c.abort(); +await driving; +await c.drive(); // also covers a previously idle/suspended drive + +const child = await c.spawn({ + prompt: "Inspect only tests", + context: "fresh", + values: { inherit: [model], set: [setValue(activeTools, ["read"])] }, +}); +await h.drive(); // supervise detached work too; may suspend on outside events +console.log(await h.task(child)); + +const alternate = await c.fork({ atEntry: completedAnswerId }); +await alternate.accept("Try a different implementation"); +await alternate.drive(); // source and unrelated forks remain parked + +off(); +await h.close(); +``` + +Harness.open receives storage, task definitions, tools/model services, hooks/resources and initial +root values. It creates root plus initial values only for empty storage. Reopening existing storage +creates no new work. Host invocation cancellation/telemetry should be threaded through these calls, +not replaced with a separate tracing or cancellation framework. + +## 10. Validation, performance and review decisions + +This is a design, not a claim of implementation or benchmark parity. Tests should prove behavior +from committed state, then benchmarks should measure equivalent work under the chosen guarantees. + +### 10.1 Backend conformance from the same mutation stream + +```text +validated batches ─┬─ reference replay used only by tests + ├─ Memory + ├─ JSONL → close → reopen + └─ SQLite → close → reopen + +compare: + current objects and generic live-task metadata + transcript pages and inherited prefixes + historical contexts, scalar tombstones, list clears/ranges + input/request mappings and result outcomes + visible working lifetimes and next allocated sequence +``` + +Reference replay is a test oracle, not a required public storage API. Use randomized small histories +and explicit deep-fork cases. Include value-only commits, multi-entry commits and independent child +initialization so message-only tests cannot accidentally pass a broken history implementation. + +```text +fork before/after a value delete → correct absence/fallback +fork before/after list clear → correct inherited ranges +fork before a later summary → earlier context, not today's filtered context +nested source cutoffs → no later ancestor version leaks +sticky state → unchanged by historical reads; explicit copy on creation +``` + +### 10.2 Failure and concurrency matrix + +| Scenario | Required observation | +|---|---| +| Persistence paused after construction | Queries/watchers still see the old complete state | +| Commit durable but caller reply lost | Reopen sees the whole batch; keyed retry reuses the input | +| Main + working writes in one batch | Reject without partial mutation | +| Torn final JSONL batch | Discard only that incomplete tail | +| Malformed complete JSONL batch | Fail open, never truncate valid later history silently | +| Retirement committed, unlink fails | Old working contents remain invisible | +| Reused scope, delayed old cleanup | New lifetime survives | +| Effect returns while abort starts | One claim covers effect join and abort reconciliation | +| Parallel tools settle in either order | Existing join becomes ready; exactly one continuation | +| Abort versus join/final settlement | No unmarked successor escapes the selected cancellation | +| Child finishes during owner cancellation | No success outcome overwrites cancellation | +| Observer notification during close | Drain/reject by admission order; no late terminal overwrite | +| Two overlapping drives | One execution per task; permissions stay scoped | +| One drive caller cancels its wait | Other callers and durable task intent are unaffected | +| Queued B precedes A's answer in transcript | B remains pending until its own group completes | +| Provider retry while collapse waits to publish | No generation/collapse dependency deadlock | +| Watch registration races a commit | Base includes it or the stream delivers it, never neither | + +Use faux providers, fake processes/clocks and storage barriers. No real providers or paid tokens. +Exercise both durable orders, not a test that happens to win the same race every time. Validate +that failed/no-op commands consume no IDs and that old task IDs remain queryable after compaction. + +### 10.3 Performance measurements + +```text +workloads: + many short turns, with/without tools + frequent compaction plus cold reopen + parallel/sequential tools and held approvals + large streaming output/checkpoints + deep forks and repeated historical state reads + many completed child conversations, few live tasks + many genuinely live background tasks + queued input, cancellation and result lookup + +measure separately: + CPU user/system time; wall latency distributions + process RSS, heap, external/array-buffer memory; peaks and post-GC retained size + actual main/sidecar/database/WAL/auxiliary-file bytes + query count, rows decoded, logical mutations and physical bytes written +``` + +Same provider messages, tool output, frame cadence, compaction boundaries, hooks, durability mode +and observation workload on both implementations. Warm-up and measurement instrumentation must be +comparable. Include fault handling costs required by the final contract, not just a happy-path spike. + +Memory/JSONL history growth is expected. SQLite should avoid full-history JS copies. In every case, +separate backend-owned data, live execution data and faux-provider/test bookkeeping before claiming +anything about the harness working set. Do not hide backend memory when reporting total process use. + +The exported pico2 spike reports 9.7 s versus 35.9 s for the lane harness on 2,000 faux turns with +two tools per turn. That motivates investigation, not a validated result for this design. Its +four-way test measures wall time and serialized main-mutation character counts; pico2 scratch and +usage writes bypass that counter. The export has no SQLite implementation. Its projection cache +also means the timing difference cannot be attributed solely to storage interfaces. + +The useful lessons are cheap current/live queries, batched ID reads, small mutations and avoiding +history scans on routine execution—not adopting its types, residency machinery or unsafe shortcuts. + +### 10.4 Decisions to review before implementation + +| Decision | Current proposal / remaining work | +|---|---| +| Historical position | Entry selects its whole commit; reject incomplete exchange targets (§3.3) | +| Sticky conversation inheritance | Copy only explicitly selected current scalars; lists need a deliberate policy (§4.3) | +| State/list vocabulary | Set/delete and append/clear; structured edits as deltas, not arbitrary mutable-list operations | +| Input identity and results | Admission/placement entries plus ordinary result values and input groups (§8.1); finalize normalization/queue policies | +| Background dependency sharing | Required dependencies join foreground cancellation; review interaction with previously detached work | +| Recovery and shutdown | Cooperative close; explicit external adoption/idempotence and bounded host policies | +| Compaction | Correct historical replay first; review checkpoints, token budgeting and oversized/split turns | +| Hooks and tool policy | Confirm transformation/error rules, effective arguments, termination aggregation and conflicting handoffs | +| Observation/replication | Finalize bounded buffering, requested-state hydration and Chord adapter/codec boundaries | +| Backend details | Concrete schema/query plans, snapshot ownership and power-loss policy; no inferred crash guarantees | +| Old-harness capabilities | Explicit parity decisions for provider options/cache behavior, telemetry, usage adjustments, navigation/import and host context | + +Do not silently remove an existing capability because the smaller core does not mention it. Decide +whether it belongs in a task, a host integration, or the public contract before implementing that +part. The next step is feedback on this design, not copying the exported spike into the repository. diff --git a/packages/agent/docs/pico/pico-usage-guide.md b/packages/agent/docs/pico/pico-usage-guide.md new file mode 100644 index 000000000..118f0f41b --- /dev/null +++ b/packages/agent/docs/pico/pico-usage-guide.md @@ -0,0 +1,1034 @@ +# pico + +Durable agent harness for pi: one session file, any number of conversations, every piece of work +recorded as a task that survives a crash, and a view any UI can render. + +**Note**: this guide is about using the harness. `pico-v3.md` is the design and the reference for +why things are the way they are. + +## Table of Contents + +- [The Mental Model](#the-mental-model) +- [Installation](#installation) +- [Quick Start](#quick-start) +- [Sessions and Conversations](#sessions-and-conversations) + - [Opening](#opening) + - [What Happens on Reopen](#what-happens-on-reopen) + - [Closing](#closing) +- [Configuration](#configuration) + - [Settings Declared by Kinds](#settings-declared-by-kinds) + - [Reading and Writing](#reading-and-writing) + - [What Is Rewindable](#what-is-rewindable) +- [System Prompt and Tool Loadout](#system-prompt-and-tool-loadout) + - [How It Works](#how-it-works) + - [Answering the Hook](#answering-the-hook) + - [Changing the Loadout](#changing-the-loadout) + - [Sections from Plugins](#sections-from-plugins) + - [Subagents Have Their Own](#subagents-have-their-own) + - [Compaction, Forks and Restarts](#compaction-forks-and-restarts) +- [Prompting](#prompting) + - [prompt, accept, drive](#prompt-accept-drive) + - [Input While Busy](#input-while-busy) + - [Aborting](#aborting) +- [Watching](#watching) + - [The View](#the-view) + - [Events](#events) + - [Rendering](#rendering) + - [Remote Clients](#remote-clients) + - [Session Watch](#session-watch) +- [Forks](#forks) +- [Compaction and Reset](#compaction-and-reset) +- [Subagents](#subagents) +- [Jobs and Schedules](#jobs-and-schedules) +- [Plugin State](#plugin-state) + - [Values and Lists](#values-and-lists) + - [Atomic Commits](#atomic-commits) +- [Writing Tools](#writing-tools) + - [The Sink](#the-sink) + - [Diagnostics](#diagnostics) + - [Long-Running Tools](#long-running-tools) +- [Hooks](#hooks) +- [Writing Kinds](#writing-kinds) + - [Entry Kinds](#entry-kinds) + - [Task Kinds](#task-kinds) +- [Recovery](#recovery) +- [Storage Backends](#storage-backends) + +## The Mental Model + +A **session** is one storage file (or one row set in SQLite). It holds **conversations**. A +conversation has three things: + +- a **transcript**: an append-only list of immutable **entries**. User messages, assistant messages, + tool results, summaries, system instructions, and anything a plugin wants to record. Entries are + never edited or reordered. +- **tasks**: the units of work. Generating a response is a task, running a tool is a task, so is + compacting, running a background process, or anything a plugin wants done. A task has a status + that changes as it runs and is written to storage at every step, which is what makes a crash + recoverable. +- **state**: keyed **values** and **lists**, for the model in use, plan mode, a game board, whatever + a plugin needs to remember. + +What the model sees, the **context**, is not stored; it is derived from the transcript: the newest +"head" entry (a summary, a handoff, a reset) and everything after it. Compaction is appending a +head. That is why forks, compaction and reset never edit anything. + +Every id is a session sequence number, minted when the write is built and never changed: + +```typescript +type Id = number; +``` + +All writes happen through **commits**: a closure that runs on the session's single write line, +where everything inside it lands together or not at all. Tasks run concurrently; writes never do. + +Nothing runs by itself. Opening a session starts nothing. **Driving** a conversation is what makes +its tasks execute, and a `prompt` is just accept + drive + read the answer. A UI **watches** a +conversation and gets a **view** it can render from directly. + +## Installation + +```bash +npm install @earendil-works/pi-agent +``` + +## Quick Start + +```typescript +import { Harness, JsonlStorage } from '@earendil-works/pi-agent'; +import { readTool, writeTool, bashTool } from '@earendil-works/pi-agent/tools'; +import { builtinModels } from '@earendil-works/pi-ai/providers/all'; + +// One file per session. Reopening the same file resumes it. +const storage = await JsonlStorage.open('./session.jsonl'); + +// The built-in kinds (generation, tool, post_tools, collapse, job; the entry kinds) and the +// subagent and job tools are registered by open. You add models and the tools you want. Nothing runs yet. +const h = await Harness.open(storage, { + models: builtinModels(), + tools: [readTool, writeTool, bashTool], +}); +const { generation: generationKind } = h.kinds; // the registered kinds, for config and hooks + +const c = await h.root(); + +// Config is declared by the kinds that read it. The generation kind declares model, thinking and +// selected tools; `settings` is shorthand for its config. +await c.settings.set({ + model: { provider: 'anthropic', modelId: 'claude-opus-5' }, + thinking: 'high', + selectedTools: ['read', 'write', 'bash'], +}); + +// The system prompt is not stored as config. Each turn the harness asks this hook what the +// instructions should be right now, compares with what the transcript says the model was told, +// and writes only the difference. See "System Prompt and Tool Loadout". +c.hooks.on(generationKind, 'system_instructions', async ({ config }) => ({ + sections: { + identity: 'You are a careful engineer working in this repository.', + cwd: `Working directory: ${process.cwd()}`, + }, + tools: h.tools.select(config.selectedTools), // complete tool definitions +}), { subtree: true }); + +// Watch the conversation. `view` is a plain object a UI renders from; attach whenever you like, +// the view is complete as of the moment you attach and events follow from there. +const w = await h.watch(c.id, { tail: 100 }); +w.start(event => { + if (event.type === 'task_output') { + // the generation's preview is the partial assistant message; a tool's is its output so far + const preview = w.view.previews.get(event.task); + process.stdout.write(renderPreview(preview)); + } + if (event.type === 'entry') console.log(`\n[entry ${event.entry.id} ${event.entry.kind}]`); +}); + +// accept the input, drive the conversation until it is idle, return the answer to that input +const answer = await c.prompt('Inspect the parser and list the public API'); +console.log(answer?.content); + +w.unsubscribe(); +await h.close(); // cancels nothing durable; reopening the file continues exactly here +``` + +Run it, kill it in the middle, run it again: the second run recovers whatever was in flight +(publishing a partial answer, rerunning or reporting an interrupted tool) and continues. That is the +whole point. + +Snippets below assume `h` and `c` set up like this. + +## Sessions and Conversations + +### Opening + +`Harness.open(storage, options)` takes the storage backend and everything that defines behaviour: + +| option | what it is | +|---|---| +| `models` | a pi-ai `Models` collection | +| `tools` | what the model may call, besides the built-in `subagent` and `job` tools; a `ToolRegistry` or an array | +| `kinds` | plugin entry and task kinds, added to the built-ins | +| `replace` | a built-in kind swapped by name (`{ generation: myGenerationKind }`); it must keep the statuses and hook names | +| `rootValues` | initial values for the root conversation on a fresh session | + +The built-in kinds are registered by `open` itself, because `accept`, `prompt`, `steer` and +`collapse` cannot work without them. Refer to whatever is registered through `h.kinds`: + +```typescript +h.kinds.generation // config: model, thinking, selectedTools, ...; hooks: system_instructions, before_request, on_yield +h.kinds.tool // hooks: before_tool, after_tool +h.kinds.collapse // hooks: before_collapse +h.kinds.job +``` + +A replacement is usually a wrapper that delegates to the original for everything it doesn't +change: + +```typescript +import { generationKind } from '@earendil-works/pi-agent/kinds'; +const h = await Harness.open(storage, { models, tools, replace: { + generation: { ...generationKind, async execute(task, ctx) { await audit(task); return generationKind.execute(task, ctx); } }, +}}); +``` + +Open checks that every kind found in storage is registered (without scanning the transcript: the +set of kinds ever written is kept) and reports live work, but starts nothing: + +```typescript +const { start, inflight } = await h.inspect(); +// start: tasks that never began or must begin again (a planned tool, a retry, a scheduled job) +// inflight: tasks the last process was running when it stopped; recover() will handle them +``` + +### What Happens on Reopen + +Nothing, until something drives a conversation. `h.drive()` recovers and continues everything; +`c.drive()` does it for one conversation and the conversations its tasks own. A UI usually attaches +a watch to what it shows and drives that. + +```typescript +const h = await Harness.open(storage, opts); +void h.drive(); // resume everything in the background +``` + +### Closing + +```typescript +await h.close(); // cancel in-process work, write nothing; everything resumes on the next open +await h.shutdown(); // mark all live work aborted, drain queues, wait for cleanup, then close +``` + +`close` is the normal exit: a crash and a close leave storage in the same state. + +## Configuration + +### Settings Declared by Kinds + +There is no settings object. A task kind declares the values it reads, typed, and that declaration +is the only place the address is spelled: + +```typescript +generationKind.config +// { +// model: conversationValue('pi.model', { rewind: true }), +// thinking: conversationValue('pi.thinking', { rewind: true }), +// selectedTools: conversationValue('pi.tools.selected', { rewind: true }), +// profile: conversationValue('pi.prompt.profile', { rewind: true }), +// budgetMs: conversationValue('pi.tool.budget', { rewind: false }), +// } +``` + +A UI can list what a conversation is configured with by walking the registered kinds' `config`; +a plugin declares its own values the same way (see [Task Kinds](#task-kinds)). + +### Reading and Writing + +```typescript +// all of a kind's values, one batched read +const { model, thinking } = await c.config(generationKind).get(); + +// some of them, one commit +await c.config(generationKind).set({ thinking: 'low' }); + +// `settings` is config(generationKind), because that is what every UI touches +await c.settings.set({ model: { provider: 'openai', modelId: 'gpt-5.6' } }); + +// one value, one point read, by its declared address +const tools = await c.value(generationKind.config.selectedTools).get(); +``` + +A change is a normal commit: it appears in the watch stream as a `value` event, and the next +generation picks it up. Nothing is cached in memory that could disagree with storage. + +### What Is Rewindable + +A value is either **rewindable** (its history is kept, a fork at an entry sees the value in force +there) or **sticky** (current only; the UI's present, not the conversation's history). Model, +thinking and selected tools are rewindable: forking at yesterday's answer gets yesterday's model. +Something like `ui.expanded` is sticky. + +Children created by `spawn` don't inherit history; they are initialized explicitly (see +[Subagents](#subagents)). + +## System Prompt and Tool Loadout + +### How It Works + +pi-ai models the system prompt as messages in the transcript: the first rendered prompt of a +session is the provider's stable, cached baseline, and later changes are appended as +`SystemMessage`s that say what changed (and carry `toolsAdded`/`toolsRemoved` with complete tool +definitions). pico is built on that, and keeps two things apart: + +- **What you want right now.** Config (model, thinking, selected tools, a profile) plus whatever + the host has: cwd, skills, context files, the tool catalogue. The host answers a hook with it + each turn. None of it is stored. +- **What the model has been told.** `system` entries in the transcript: the baseline that opened the + current context and one entry per change delivered since. Immutable facts about requests that + happened, so they fork and compact like every other entry. + +At the top of every turn the generation renders the first, folds the second out of the context, +and appends the difference, if any, before it projects the request. The host keeps rendering "the +whole prompt" the way it always did; the harness turns that into append-only deltas. + +### Answering the Hook + +The hook is `system_instructions` on the generation kind. It returns keyed **sections** (rendered +in the order given) and the **tool definitions** the model may call. Register it on the conversation +with `subtree: true` so subagents inherit it unless they register their own: + +```typescript +c.hooks.on(generationKind, 'system_instructions', async ({ conversationId, config }) => ({ + sections: { + identity: IDENTITY, + cwd: `Working directory: ${cwd}`, + context_files: await renderContextFiles(cwd), // AGENTS.md and friends + skills: skills.render(), + }, + tools: h.tools.select(config.selectedTools), // complete definitions, from the catalogue +}), { subtree: true }); +``` + +After the first turn the transcript holds what was sent: + +```text +10 user "Inspect the parser" +11 system { baseline, sections: [identity, cwd, context_files, skills], tools: [read, write, bash] } +12 assistant ... +``` + +The tool definitions are stored in full, not as names. Rendering a persisted message may depend +only on what is stored on it, so a later change to a tool's description is a change the model gets +told about, and an old transcript renders the way it was sent. + +### Changing the Loadout + +Every change is either a config write or something the host renders differently. Nothing is +written at that moment; the next generation notices and appends one entry: + +```typescript +await c.settings.set({ selectedTools: ['read', 'write', 'bash', 'grep'] }); +// next turn → 20 system { toolsAdded: [grep] } + +await c.settings.set({ selectedTools: ['read'] }); +// next turn → 30 system { toolsRemoved: [write, bash, grep] } + +// the user installs a skill; skills.render() changes +// next turn → 40 system { sections: [skills] } "the skills section now reads: ..." + +h.tools.replace('mcp', newDefinitions); // an MCP server restarted with a changed schema +// next turn → 50 system { toolsAdded: [the changed definition] } same name, new definition = a change +``` + +The provider's cached prefix through the previous message is untouched by any of these. On models +with native mid-conversation system messages the entry goes out as one; elsewhere pi-ai renders it +as a tagged user turn. + +### Sections from Plugins + +A plugin contributes sections from its own state. Handlers for the same point merge by key; +`undefined` removes a key. Register harness-wide when the section applies to every conversation: + +```typescript +const planMode = conversationValue('plan.mode', { rewind: true }); + +h.hooks.on(generationKind, 'system_instructions', async ({ conversationId }) => ({ + sections: { + plan_mode: (await h.conversation(conversationId).value(planMode).get()) ? PLAN_GUIDANCE : undefined, + }, +})); + +await c.value(planMode).set(true); +// next turn → 60 system { sections: [plan_mode] } "the plan_mode guidance now applies: ..." +await c.value(planMode).set(false); +// next turn → 70 system { sections: [plan_mode removed] } +``` + +Conversation-scoped handlers run after harness-wide ones, innermost conversation last, so a child's +sections win by key. + +A tool can change the loadout from inside a call. It goes through the same path: post_tools updates +the config, the next generation emits the delta. + +```typescript +// inside a discovery tool +out.addTools(['calculator']); +// post_tools: selectedTools += calculator +// next turn → 80 system { toolsAdded: [calculator] } +``` + +### Subagents Have Their Own + +A subagent is a different agent: different instructions, different loadout, often a different +model. `spawn` sets its config, and either the parent's `subtree` handler answers (branching on +`config.profile`) or the child registers its own: + +```typescript +const childId = await c.spawn({ + prompt: 'Audit the tests', + values: { + inherit: [generationKind.config.model], + set: [ + [generationKind.config.selectedTools, ['read', 'grep']], + [generationKind.config.profile, 'auditor'], + ], + }, +}); + +const child = await h.conversation(childId); +child.hooks.on(generationKind, 'system_instructions', async ({ config }) => ({ + sections: { identity: AUDITOR_IDENTITY, cwd: `Working directory: ${cwd}` }, + tools: h.tools.select(config.selectedTools), +})); +// child's first turn → its own baseline: two sections, two tools +``` + +### Compaction, Forks and Restarts + +Nothing to do in any of them. + +- **Compaction / handoff / reset** append a head; the context after it has no baseline, so the next + turn writes a fresh one from the current answer to the hook. The summary never has to describe + the prompt, and the provider's prefix was changing at the head anyway. An older delta that a + compaction kept in its tail is subsumed by the new baseline and dropped at projection. +- **A fork** carries the `system` entries in its shared prefix and its own (rewindable) config; its + first turn diffs the two: + + ```typescript + const b = await c.fork(12); // config as of 12: tools [read, write, bash] + await b.settings.set({ selectedTools: ['read'] }); + await b.prompt('...'); // b's turn: toolsRemoved [write, bash]; the prefix's 11 is still the baseline + ``` + +- **A restart** with a changed host (new skill, different cwd) emits "these sections now apply" + and nothing else; a host that comes back the same emits nothing. + +## Prompting + +### prompt, accept, drive + +`prompt` is three things: accept the input, drive the conversation until it is idle, read the +answer to that input. + +```typescript +const answer = await c.prompt('Inspect the parser'); +// AssistantEntry | undefined (the run ended without an answer: aborted, failed) +``` + +The pieces are available separately, and they compose across retries. `accept` is idempotent by +`requestId`, and `answerTo` finds the answer to an input however far the conversation has moved +since, so a client that lost the reply can ask again: + +```typescript +const { entryId } = await c.accept('Inspect the parser', { requestId: 'req-42' }); +const outcome = await c.drive(); // 'idle' | 'closed' +const answer = await c.answerTo(entryId); // last assistant entry after it, before the next user entry +``` + +`drive` resolves when the conversation's foreground is idle: no generation, tool or automatic +collapse is live. Background work (a spawned subagent, a job) keeps running and doesn't keep +`drive` waiting. `h.drive()` waits for everything. + +### Input While Busy + +While a turn runs, new input is queued rather than refused. Which mode decides where it lands and +whether it asks for another turn: + +```typescript +await c.steer('Focus on the tokenizer first'); // next tool boundary or final answer; requests a turn +await c.followUp('Then write the tests'); // after the final answer; requests a turn +await c.nextRun('Remind me to commit'); // only at the next idle accept +await c.write('user stepped away'); // next boundary; no turn requested +``` + +The queue is visible in the view (`view.inbox`) and any item can be withdrawn until it lands: + +```typescript +await c.cancelQueued(itemId); // 'cancelled' | 'not_found' +``` + +### Aborting + +```typescript +await c.abort(); // every live foreground task of this conversation, and of conversations they own +await h.abortTask(id); // one background task: a job, a schedule +``` + +An abort is a durable mark on the task, then cleanup: the generation publishes what it had, tools +write error results for their calls, a subagent tool aborts its child. If the process dies between +the mark and the cleanup, the next one finishes it. Queued `steer` and `followUp` are dropped by +`abort`; `write` and `nextRun` stay. + +## Watching + +### The View + +A UI never reads storage or parses commits. It watches a conversation and gets a **view**: a plain +JSON object the harness keeps current, plus typed events that say what changed. + +```typescript +const w = await h.watch(c.id, { tail: 100, values: [myPlugin.config.mode] }); +w.view // ConversationView, captured atomically with the subscription +w.start(listener); +w.resnapshot(); // fresh capture, same subscription (if the client fell behind) +w.unsubscribe(); +``` + +```typescript +interface ConversationView { + conversation: Conversation; + entries: Entry[]; // the last `tail` entries; page older ones with h.entries(id, { before }) + context: Id[]; // what the model currently sees, as entry ids + tasks: Task[]; // live tasks, typed by kind + inbox: Element[]; // queued input + values: Map; // every value the registered kinds declare, plus the ones you asked for + previews: Map; // per live task: what it is producing right now + faulted: boolean; + readAt: Id; +} +``` + +`previews` is where streaming lives. Each task kind defines what its preview is: the generation's +is the partial `AssistantMessage`, a tool's is its `ToolOutputState` so far, a job's is the same +shape for its process output. + +### Events + +```typescript +type ConversationEvent = + | { type: 'entry'; entry: Entry } + | { type: 'task_start'; task: Task } + | { type: 'task_update'; task: Task; previous: Task } + | { type: 'task_end'; task: Task } + | { type: 'task_output'; task: Id; ops: DeltaOp[] } // already applied to view.previews + | { type: 'value'; addr: Address; value: JsonValue | undefined } + | { type: 'inbox'; items: Element[] } + | { type: 'context'; ids: Id[] } // a head landed + | { type: 'fault'; error: unknown } + | { type: 'closed' }; +``` + +The view is authoritative and the event is a wake-up: when the listener runs, `w.view` has already +been folded. A renderer may ignore the event payload entirely and be correct. Because every piece +of work is a task of a known kind, four task events cover what used to need a name per case: + +| you want to know | look at | +|---|---| +| a turn started / ended | `task_start` / `task_end` where `generationKind.is(task)` | +| a retry is scheduled | `task_update`, status `retry_wait`, `state.attempt`, `state.notBefore` | +| the response is streaming | `task_output` on the generation; `view.previews.get(task)` | +| a tool is running / its output | `task_start` / `task_output` / `task_end` on the tool task | +| compaction started / ended | `task_start` / `task_end` where `collapseKind.is(task)` | +| the model changed | `value` with `addr === generationKind.config.model` | +| queued input changed | `inbox` | + +Events of one commit arrive together, in order, so a finished task and its successor never show +as an idle gap. + +### Rendering + +A renderer is a function of the view, diffing against what it last drew. Settled entries are keyed +by id and only appended; live things are keyed by task id, and a tool block keeps its key when the +task settles and its result entry appears, so nothing is torn down and rebuilt: + +```typescript +function render(view: ConversationView, event?: ConversationEvent) { + transcript.sync(view.entries); + + const gen = view.tasks.find(t => generationKind.is(t)); + streaming.set(gen ? view.previews.get(gen.id) as AssistantMessage : undefined); + status.set( + gen?.status === 'retry_wait' ? `retrying (${gen.state.attempt}/${gen.state.maxAttempts})` : + gen?.status === 'deferred' ? 'waiting for provider' : + view.tasks.some(collapseKind.is) ? 'compacting…' : undefined); + + for (const t of view.tasks.filter(toolKind.is)) + toolBlocks.upsert(t.id, { call: t.state.call, phase: t.status, output: view.previews.get(t.id) as ToolOutputState }); + + for (const t of view.tasks.filter(jobKind.is)) + jobBlocks.upsert(t.id, { tool: t.state.origin?.tool ?? 'job', output: view.previews.get(t.id) as ToolOutputState }); + + queue.set(view.inbox.map(i => i.value)); + working.set(view.tasks.some(t => !t.background)); + statusLine.set({ model: view.values.get(generationKind.config.model) }); +} +``` + +Tool components are registered once per tool name and fed one shape, `ToolOutputState`, whether it +comes from a live tool task's preview, a settled `tool_result` entry, or a job the tool started +(`state.origin.tool` says which component). + +### Remote Clients + +The view is plain JSON and every event is proportional to its change, so a process without a +harness (mini's TUI, a phone) runs the same fold on the same events. `applyEvent(view, event)` is +exported and needs no kinds; preview ops are Chord delta ops, which the client applies with the +same module. + +```typescript +// worker (has the harness) // ui process +const w = await h.watch(c.id, { tail: 100 }); on('view', m => { view = m.view; render(view); }); +send({ type: 'view', view: w.view }); +w.start(e => send({ type: 'event', event: e })); on('event', m => { applyEvent(view, m.event); render(view, m.event); }); +``` + +### Session Watch + +What isn't one conversation's: the conversation list, session values, usage totals, faults, and +reports (a hook threw, a task kind misbehaved and was stopped). + +```typescript +const sw = await h.watch(); +sw.view.conversations; sw.view.values; sw.view.faulted; +sw.start(e => { + if (e.type === 'usage') status.setCost(e.totals); + if (e.type === 'report') log.warn(e.kind, e.task, e.error); + if (e.type === 'conversation') tree.refresh(); +}); +``` + +## Forks + +A fork is a new conversation whose transcript starts as a shared prefix of the source. Nothing is +copied and nothing in the source is deleted. It carries the context as it was at that entry, the +rewindable values in force there, and the prompt-as-sent, so its first turn is exact. + +```typescript +const alt = await c.fork(answer.id); // the source keeps running, untouched +await alt.prompt('Try a different implementation'); + +const back = await c.fork(earlier.id, { abort: true }); // "go back": aborts the source's foreground first +``` + +Forking at an assistant entry whose tool calls aren't answered yet is refused (the fork would end +mid-exchange); fork at the last result instead. Which conversation a UI treats as "current" is the +UI's business; the harness only has conversations. + +```typescript +const all = await h.conversations(); +const independent = all.items.filter(x => x.owner === undefined); // root and forks: their own drive scopes +const children = await h.conversations({ parent: c.id }); // forks and owned children of c +``` + +## Compaction and Reset + +Compaction appends a summary entry that replaces a prefix; the summary is a head, so the context +becomes the summary plus what came after the replaced prefix. It runs as a background task and may +run while the model keeps working: whatever lands meanwhile has higher ids than the summary's +reach and stays in context. + +```typescript +const collapseId = await c.collapse(); +const collapseId = await c.collapse({ instructions: 'keep the API decisions verbatim' }); +``` + +Threshold and overflow compaction happen inside the generation; nothing to call. Reset starts the +context over, with or without a handoff message: + +```typescript +await c.reset('Continue from here: we settled on a recursive-descent parser.'); // handoff +await c.reset(); // /clear +``` + +The transcript keeps everything either way; only the context changes. + +## Subagents + +A subagent is a conversation. There is no separate object to talk to: the model gets one tool with +a `command` argument, and the API gets a conversation handle. + +```typescript +// the model calls: +subagent({ command: 'run', prompt: 'Audit the tests', context: 'fresh', tools: ['read', 'grep'] }) // waits for the answer +subagent({ command: 'spawn', prompt: 'Profile the build' }) // returns the child's id +subagent({ command: 'send', id: 88, text: 'also check CI' }) +subagent({ command: 'status', id: 88 }) +subagent({ command: 'wait', id: 88 }) +subagent({ command: 'stop', id: 88 }) +``` + +```typescript +// the API +const childId = await c.spawn({ prompt: 'Profile the build', context: 'fresh', + values: { inherit: [generationKind.config.model] } }); +const child = await h.conversation(childId); +await child.accept('also check CI'); +await child.drive(); // or let h.drive() / the parent's drive carry it +await child.abort(); +``` + +`run` keeps the calling tool inflight while it drives the child, so the child is part of the +parent's foreground: aborting the parent reaches it. `spawn` settles the tool at once; the child is +detached, runs while the parent goes on, and only `child.abort()` (or the `stop` command) ends it. +Either way the child is driven by whatever drives the parent's tree, and it survives a restart like +any conversation. + +## Jobs and Schedules + +A job is a background task that runs a process: durable, recoverable, killable, with its output +streamed into its preview. The model normally gets one from `bash` (asked to background, or run +past its budget) and controls it through the `job` tool: + +```typescript +job({ command: 'wait', id: 91, budgetMs: 30_000 }) +job({ command: 'status', id: 91 }) +job({ command: 'stop', id: 91 }) +job({ command: 'list' }) +``` + +From the API a job is a task; a schedule is a job with `every`: + +```typescript +const dev = await c.commit(tx => tx.task(jobKind, { background: true, + state: { cmd: ['npm', 'run', 'dev'], cwd } })); + +const nightly = await c.commit(tx => tx.task(jobKind, { background: true, + state: { cmd: ['npm', 'test'], cwd, every: 24 * 3600_000, notBefore: tonightAt(2) } })); + +await h.abortTask(nightly); // ends the schedule wherever it is +``` + +A schedule is one task looping `planned → running → planned`; there is no trail of run ids to +chase. A job whose starting call returned early appends a `notice` entry when it finishes, so the +model learns of it on its next turn without polling. Its output stays in the task (preview while +live, terminal state after), not in the transcript. + +## Plugin State + +### Values and Lists + +Declare an address once; read and write through handles or inside commits. The address carries the +scope (session or conversation), the rewind policy and the payload type. + +```typescript +const planMode = conversationValue('plan.mode', { rewind: true }); +const moves = conversationList('game.moves', { rewind: true }); +const expanded = conversationValue('ui.expanded', { rewind: false }); +const name = sessionValue('pi.session.name'); + +await c.value(planMode).set(true); +const on = await c.value(planMode).get(); +const then = await c.value(planMode).get(entryId); // as of an entry: rewindable only + +const elementId = await c.list(moves).append({ x: 1, y: 2 }); +const page = await c.list(moves).read({ limit: 50 }); +await c.list(moves).remove(elementId); +await c.list(moves).clear(); + +await h.value(name).set('parser work'); +``` + +Rewindable state is what makes plugins survive forks: a fork before the move doesn't see the move, +a fork before plan mode was turned on doesn't have it on. Nothing to register, nothing to re-derive. + +### Atomic Commits + +Anything that must land together goes in one commit: a closure on the session's write line. Reads +inside it see committed state; ids are final when returned; a throw discards everything. + +```typescript +const entry = await c.commit(tx => { + tx.value(planMode).set(false); // state first + const id = tx.entry(myPlugin.noteKind, { text: 'plan accepted' }); // then entries + tx.task(myPlugin.reminderKind, { background: true, // tasks anywhere + state: { about: id, at: Date.now() + 3600_000 } }); + return id; +}); +``` + +One rule: in a commit, state writes come before entries. A fork at an entry is "everything up to +that id", and that is what makes the state a command wrote alongside its entry part of the fork, +and state committed later not. + +## Writing Tools + +### The Sink + +A tool's `execute` returns nothing. Everything it produces goes through a sink, so output streams +to the UI as it happens, is bounded once in one place, and settles into a `ToolOutputState` that +the transcript, the model and every renderer share. Failure is a throw; the harness sets `isError`. + +```typescript +import { Type, type Tool } from '@earendil-works/pi-agent'; + +export const countLinesTool: Tool<{ i: string; path: string; pattern?: string }, { lines: number; matching: number }> = { + name: 'count_lines', + description: 'Count lines in a file, optionally only those matching a pattern', + parameters: Type.Object({ + i: Type.String({ description: 'What you are trying to find out' }), // intent: streams first, shows in the UI + path: Type.String(), + pattern: Type.Optional(Type.String()), + }), + output: { maxBytes: 64_000, maxLines: 200, retain: 'head' }, // the sink enforces this + replay: 'safe', // read-only: may be rerun after a crash + + async execute(toolCallId, params, signal, out, ctx) { + const lines = getOrThrow(await ctx.env.readTextLines(params.path, {}, { signal })); // ExecutionEnv: FileSystem & Shell + const re = params.pattern ? new RegExp(params.pattern) : undefined; + let matching = 0; + for (const [i, line] of lines.entries()) { + if (re && !re.test(line)) continue; + matching++; + out.write(`${i + 1}: ${line}\n`); // bounded by `output`; the sink truncates and diags + } + if (matching > 200) out.diag('warn', `showing 200 of ${matching} matching lines`, 'cap'); + out.details.lines = lines.length; // typed, for UIs + out.details.matching = matching; + }, +}; +``` + +The sink: + +| call | effect | +|---|---| +| `write(text)` / `replace(text)` / `image(img)` | the content the model reads | +| `details` | the tool's typed object for UIs; mutate it | +| `usage(u)` | accumulates | +| `addTools(names)` | change the loadout from the next turn on | +| `terminate(true)` | stop the turn after this exchange, whether or not the call failed | +| `handoff(message)` | ask for a context reset after the exchange (what `new_context` does) | +| `delegate(jobId)` | this call's work continues as that job | +| `diag(severity, message, code?)` | commentary about the call, kept out of the data | + +A tool never touches the transcript, its siblings or the queue. What it may do is create things in +its own conversation through `ctx.conversation.commit(...)`: a job, a child conversation. + +### Diagnostics + +Anything that is *about* the call rather than its output goes through `diag`: truncation, a spilled +file, a corrected path, "the file changed on disk since you read it". The harness emits the ones it +owns (the sink itself reports truncation and spill); a tool adds only what it alone knows. +Projection puts the data first and the commentary after it, delimited, so the model can treat tool +output as data, and a UI renders diagnostics as callouts by severity: + +```text +...last matching line + +[warn] stopped at 500 matches + +``` + +### Long-Running Tools + +A call may not block a turn forever. Tools that run processes create a job first and wait on it +with the budget from config; if the budget runs out, the call settles with what it has and the work +goes on: + +```typescript +async execute(toolCallId, params, signal, out, ctx) { + const job = await ctx.conversation.commit(tx => tx.task(jobKind, { background: true, + state: { cmd: params.cmd, cwd: params.cwd ?? ctx.env.cwd, + origin: { tool: 'bash', task: ctx.taskId, callId: toolCallId } } })); // so UIs render it as bash + + if (params.background) { out.delegate(job); out.write(`started job ${job}`); return; } + + let done: boolean; + try { done = await ctx.waitForTask(job, { budgetMs: ctx.budgetMs, signal }); } + catch (e) { if (isAbortError(e)) await ctx.harness.abortTask(job); throw e; } // user aborted: kill what we started + + const o = await ctx.jobOutput(job); // the job's ToolOutputState, live or settled + out.replace(o.text); out.capture(o.truncation); + if (done) { out.details.exitCode = o.exitCode; if (o.exitCode) out.diag('warn', `exit code ${o.exitCode}`, 'exit'); } + else { out.delegate(job); out.diag('info', `still running as job ${job}; use job wait / status / stop`, 'budget'); + await ctx.conversation.commit(tx => tx.patch(job, { state: { notify: true } })); } +} +``` + +The job owns its output from the first byte; the tool only ever copies a snapshot. A UI shows the +job's output live either way, rendered with `bash`'s component. + +## Hooks + +Hooks belong to the kind that runs them. A kind declares its points and their types; you register +handlers per kind and point, harness-wide or scoped to a conversation. Handlers run outside the +write line and their decisions are re-validated inside a commit, so they may take as long as they +like (a human approval is a hook that waits). + +```typescript +// policy: everywhere +h.hooks.on(toolKind, 'before_tool', async ({ toolName, args, conversationId }) => { + if (toolName === 'write' && await h.conversation(conversationId).value(planMode).get()) + return { block: { reason: 'plan mode: no edits' } }; + if (toolName === 'bash' && !(await ui.approve(args))) + return { block: { reason: 'denied by user' } }; + return { args }; // may rewrite arguments +}); + +h.hooks.on(toolKind, 'after_tool', async ({ toolCallId, output }) => { metrics.record(output.usage); }); + +h.hooks.on(generationKind, 'before_request', async ({ request }) => ({ request: withTracing(request) })); + +h.hooks.on(generationKind, 'on_yield', async ({ answer, conversationId }) => { + if (await goalIncomplete(conversationId)) return { continue: 'The goal is not met yet; continue.' }; +}); + +h.hooks.on(collapseKind, 'before_collapse', async ({ reason, context }) => { + if (reason === 'manual' && context.length < 10) return { decline: true }; +}); + +// instructions: per conversation (see System Prompt and Tool Loadout) +c.hooks.on(generationKind, 'system_instructions', handler, { subtree: true }); +``` + +Points and their fail behaviour: + +| kind | point | returns | on throw | +|---|---|---|---| +| generation | `system_instructions` | sections + tools | reported, skipped | +| generation | `before_request` | a transformed request | reported, skipped | +| generation | `after_response` | nothing | reported | +| generation | `on_yield` | `{ continue?: string }` | reported, skipped | +| tool | `before_tool` | `{ args? }` or `{ block }` | **blocks the tool** | +| tool | `after_tool` | nothing | reported | +| collapse | `before_collapse` | `{ decline? \| instructions? \| summary? }` | reported, skipped | + +A handler may run again after a crash (the task it belongs to is re-executed), so its external side +effects need their own idempotence. + +## Writing Kinds + +Kinds are how you extend the harness with new behaviour rather than new state. An entry kind says +what a kind of entry means to the model; a task kind says how a kind of work runs. + +### Entry Kinds + +```typescript +interface NoteContent { text: string; pinned?: boolean } + +// the model never sees this kind: no `project` +export const noteKind = defineEntryKind({ kind: 'myplugin.note' }); + +// the model sees this one +export const pinnedKind = defineEntryKind({ + kind: 'myplugin.pinned', + project: e => [{ role: 'user', content: `${e.content.text}`, timestamp: 0 }], +}); + +// a head kind: a context that starts here and keeps only what follows (a window, no summary) +export const windowKind = defineEntryKind<{ through: Id }>({ + kind: 'myplugin.window', + head: content => content.through, // entries after this id are in context; the window entry itself projects nothing +}); +``` + +`head` runs once, at append, and the number it returns is stored and indexed, which is how the +context is found without scanning. A head kind that wants to reshape what follows (drop old tool +results, shorten one) adds `compose(head, tail)`, which runs in memory on the already-bounded +range. + +Reads are typed by the kind, or untyped and narrowed: + +```typescript +const note = await h.getEntry(noteKind, id); // Entry | undefined (also undefined for another kind) +const any = await h.getEntry(id); +if (noteKind.is(any)) any.content.pinned; +``` + +### Task Kinds + +A task kind declares its statuses and their roles, its config, its hooks, and three functions. +Here is a reminder that fires once: + +```typescript +interface ReminderState { about: Id; at: number; fired?: boolean } + +export const reminderKind = defineTaskKind({ + kind: 'myplugin.reminder', + initialStatus: 'scheduled', + roles: { scheduled: 'start', firing: 'inflight', done: 'terminal', aborted: 'terminal' }, + config: { intervalMs: conversationValue('myplugin.reminder.interval', { rewind: false }) }, + hooks: { before_fire: { failClosed: false } }, + + async execute(task, ctx) { + if (task.state.at > Date.now()) await ctx.sleep(task.state.at); // throws on abort + await ctx.commit(tx => tx.patch(task.id, { status: 'firing' })); // record intent before any effect + const { skip } = await ctx.hooks(reminderKind).run('before_fire', { about: task.state.about }); + await ctx.commit(tx => { + if (!skip && !tx.getTask(task.id)!.abort) // a marked task creates no successor + tx.entry(noticeKind, { text: `Reminder: see entry ${task.state.about}` }); + tx.settle(task.id, 'done', { ...task.state, fired: !skip }); + }); + }, + + async recover(task, ctx) { return this.execute(task, ctx); }, // safe to redo + async abort(task, ctx) { await ctx.commit(tx => tx.settle(task.id, 'aborted', task.state)); }, + + // no preview: nothing to show while sleeping +}); +``` + +The rules an execution follows, and the driver enforces: + +1. Commit an inflight status before any external effect. A crash before that re-runs `execute`; a + crash after goes through `recover`. +2. You may block on the world: a provider stream, a process, a child conversation, a sleep. The + driver runs executions concurrently; a blocked one holds up nothing. +3. Change your status or settle before returning. Returning unchanged is reported as a bug. +4. Never wait on another task. Depend on it at creation (`after`), or let it be a conversation you + drive. + +A kind with a `preview` decides what a UI sees while the task runs: `preview.init(scratch)` builds +it once on attach or reopen, and after that the kind mutates `ctx.preview.state` in place (the +generation applies stream events to a partial message; a tool's preview is its sink). The harness +flushes the Chord delta tracker after each scratch commit into `task_output` ops, so a token costs +one append op, not a diff of the whole object. + +## Recovery + +There is nothing to write. After a crash: + +```typescript +const h = await Harness.open(storage, opts); +await h.drive(); +``` + +| what was running | what happens | +|---|---| +| a generation, streaming | its frames were in scratch: the partial is published, then retry within budget or fail | +| a generation, between retries | it sleeps out the remaining backoff and tries again | +| a tool with `replay: 'safe'` | runs again | +| any other tool | an "interrupted" error result; the turn continues | +| a `run` subagent tool | finds its child and drives it again | +| a job | reruns if it said so, otherwise records `lost` | +| a scheduled job | sleeps until its time | + +Successors are written in the same commit as the settlement that decides them, so a crash never +leaves "the tools finished but nobody started the next turn": post_tools already exists and is +startable. + +## Storage Backends + +| backend | loads | good for | +|---|---|---| +| `MemoryStorage` | everything | tests, ephemeral sessions | +| `JsonlStorage` | everything, from one append-only file plus a scratch file per live task | local sessions; the default | +| `SqliteStorage` | only what queries return | long sessions, many sessions in one file, servers | + +All three answer the same queries with the same results; the conformance suite replays one +mutation stream into each and compares. JSONL encodes large values as Chord deltas so a plugin's +structured state stays linear in the file; SQLite stores values whole so a point read is never a +replay. + +```typescript +const storage = await SqliteStorage.open('./sessions.db', { session: 'parser-work' }); +``` diff --git a/packages/agent/docs/pico/pico-v3.md b/packages/agent/docs/pico/pico-v3.md new file mode 100644 index 000000000..d19c32d6d --- /dev/null +++ b/packages/agent/docs/pico/pico-v3.md @@ -0,0 +1,1760 @@ +# pico v3 + +A design for the agent harness: how a session is stored, how work is scheduled, and how the +built-in agent behaviour is composed from that. Code in this document illustrates the model; it is +not a package export. + +## 1. Goals + +**Few concepts, replaceable behaviour.** A session is conversations, immutable entries, durable +tasks, and scoped state. Agent behaviour (generate, run tools, compact, delegate) is written as +task kinds. The scheduler understands task lifecycle, dependencies and cancellation, nothing +about prompts or summaries. Storage understands objects and atomic batches, nothing about tasks. +Replace a task kind and you change the behaviour; the scheduler and storage stay. + +**One writer per session.** Execution is concurrent; writing is not. Every mutation, including +working-state writes, goes through one owner that runs commits one at a time. A commit reads +committed state, constructs its writes and their ids, commits them as one batch, then publishes. +Different sessions can have different owners and share a SQLite file. After a crash a new owner +resumes from durable state; it must never run alongside the old one. + +**Durable work, explicit recovery.** Accepted work survives a crash, including work that hasn't +started. Opening a session starts nothing. A task records its intent before doing anything external +and commits the outcome together with whatever should happen next. A crash in the middle leaves +uncertainty, not proof that nothing happened; each task kind decides how to recover: retry, +adopt, or report interruption. Nothing promises exactly-once external effects. + +**Long sessions without a residency layer.** Old entries and finished tasks are queryable but never +retained because they were once seen. Routine execution reads the current context and the live +tasks; it never scans accumulated history. What memory a backend uses is a backend choice: memory +and JSONL keep everything loaded; SQLite keeps only what queries return. + +**Measure.** Same workload on each harness and backend: CPU, process memory, bytes on disk +including sidecars. Small writes, indexed reads, batched lookups. A faster prototype is a reason to +look, not proof that its shortcuts belong here. + +## 2. Conversations + +A session holds conversations. A conversation has three things: + +- a **transcript**: an append-only list of immutable entries (messages, tool results, summaries, + plugin-defined entries), which is what happened; +- **tasks**: the durable units of work that act on it, such as a generation or a tool call, each + with a status that changes as it runs (§5); +- **state**: keyed values and lists that plugins and the harness read and write, such as the model + in use or a plan-mode flag (§4). + +What the model sees, the context, is derived from the transcript by a fixed rule. Tasks and state +are never model input by themselves. + +### 2.1 Entries and their kinds + +```ts +type Id = number; // a session sequence number; one alias so the representation can change + +interface Entry { + readonly id: Id; // its sequence number + readonly conversationId: Id; + readonly kind: string; + readonly content: Content; + readonly key?: string; // optional indexed key, e.g. the tool call a result answers + readonly head?: Id; // set at append, indexed: entries after this id are in context + readonly byTaskId?: Id; // which task wrote it +} +``` + +Entries append. They are never patched, reordered or renumbered. Ids increase in transcript order +but need not be consecutive. + +Kinds are code, not data: the built-ins are registered by `Harness.open`, plugins add theirs +next to the task kinds (§5.2) and tools (§9.5). Storage sees only the `kind` string on each entry and never runs kind code; the scan +and projection look the string up in the registry. + +```ts +interface EntryKind { + readonly kind: string; + project?(entry: Entry): readonly Message[]; // absent: the model never sees this kind + head?(content: Content, id: Id): Id; // evaluated at append + compose?(head: Entry, tail: readonly Entry[]): readonly Entry[]; // in memory, on the range + is(entry: Entry): entry is Entry; +} +``` + +`project` turns an entry into the messages the model receives. A kind without it is invisible to the +model: it is in the transcript and never in the context. + +`head` lets a kind start a context. It runs once, when the entry is appended, and returns one +number, stored on the entry and indexed: the context is this entry plus the entries after that +number. That is the only thing storage knows about heads, and it is enough to find the newest one +and bound the range with an index. `summary` returns the last entry it replaces; `handoff` and `reset` return the entry's own id, so +nothing before it is included; a plugin can define its own. A new head's number must be at least +the previous head's id: heads replace, they never stack, so a context contains exactly one head +and what follows it, and a head that wants to keep something from its predecessor copies it into +its own content. + +A head replaces the prefix before its number with whatever its content projects; `compose`, if the +kind has one, then reshapes the entries after it, in the harness, on the already-bounded range: +drop old tool results, shorten one, keep only the last N exchanges. Every compaction scheme is one +of those two things or both: a summary is content; a prune is `compose`; a prune after a summary +is a head that copies the summary text into its content and prunes the tail. Nothing is copied +that `compose` can drop in place, and storage knows only the number. + +The built-ins are `user`, `assistant`, `tool_result`, `system`, `notice`, `summary`, `handoff` and +`reset`. +All have `project` except `reset`, which starts an empty context; `handoff` starts a context with +its own message; `system` projects to pi-ai's `SystemMessage`, each at its position in the +transcript, so the prompt in force at any entry is on record and a provider's cached prefix is +never rewritten: + +```ts +interface SystemContent { + readonly baseline?: true; // opens an epoch (§8.2) + readonly sections?: readonly { key: string; text: string }[]; // in render order; baseline: all; delta: the changed ones + readonly toolsAdded?: readonly Tool[]; // complete definitions as sent, never names resolved later + readonly toolsRemoved?: readonly Tool[]; +} +``` + +Definitions are stored, not looked up, and sections are stored in the order they were rendered, +because rendering a persisted message must depend only on what is stored on it; a changed +description, schema or section order is therefore a delta like any other. `project` turns the +entry into pi-ai's `SystemMessage` at request time: a baseline's sections into the top-level prompt +(or one system message where there is no baseline slot), a delta's into "the `plan_mode` guidance +now applies: …" with `toolsAdded`/`toolsRemoved` attached. Plugin-defined kinds usually have no `project`. The +kind is the type witness for reads and writes, so nothing is cast and nothing is validated: +stored objects are trusted (§7.3), and a kind that changes its shape ships a migration. + +```ts +tx.entry(toolResultKind, conversationId, content); // content: ToolResultContent +const e = await harness.getEntry(summaryKind, id); // Entry +const any = await harness.getEntry(id); // Entry +if (summaryKind.is(any)) any.content.through; // narrowed +``` + +The kind is optional on reads: without it you get the untyped entry, and `kind.is(entry)` narrows +it. A typed read of an entry of another kind returns `undefined`, the same as a missing entry; +batched reads omit it. + +Open checks the set of entry kinds ever written against the registry: on SQLite that is a `kinds` +table with one row per distinct kind, filled by `INSERT OR IGNORE` in the commit that first uses a +kind; on JSONL and memory it falls out of the replay open does anyway. No scan of entries; an +unknown kind rejects before anything runs. + +### 2.2 Context is derived, not stored + +The context is the newest head and the entries after its number, projected: + +```text +head = newest entry of the conversation with head set (one indexed query) +range = no head → all entries; otherwise the head, plus entries with id > head.head +context = the head, then the entries after its number, oldest first (one range query) + then the head kind's compose over that tail, if it has one; + entries whose kind has no project contribute nothing +``` + +No context list is stored, so nothing can drift from the transcript, and repeated compaction +subsumes by construction: an older head lies inside the newer one's range. Projection then converts +the entries to provider messages and normalizes each tool exchange so results follow their calls in +call order, whatever order they landed in. + +The cost is the size of the context, never the length of the session: the newest head is one point +lookup on the head index, and the range is one bounded read. The harness caches the current context +of each open conversation and maintains it incrementally (an append pushes, a new head recomputes); +the cache is a convenience, the transcript is the truth. + +### 2.3 Compaction and reset + +**Compaction** appends one summary entry; its kind sets `head` to the last entry it replaces: + +```text +transcript [10 user, 20 asst, 30 user, 40 asst] context [10, 20, 30, 40] +append 50 summary, head = 20 +transcript [10, 20, 30, 40, 50] context [50, 30, 40] +``` + +The summary is newer than 30 and 40 but projects before them. Rules: the replaced prefix must end on a complete exchange, lie inside the current context, include +the previous head if there is one (§2.1), and not separate a `system` entry from the exchange it +governs; a summary prepared against an old context may +still land later, since anything appended meanwhile +has a higher id than its head number; a competing head invalidates it, and the commit checks that +(§8.4). + +**Reset** appends a `handoff` entry (with a message the model sees) or a `reset` entry (nothing), +either with `head` pointing at itself. The +transcript keeps everything. Reset by itself neither cancels tasks nor requests a response. + +### 2.4 Forks share history, not future changes + +A fork is a new conversation whose transcript starts as a prefix of its source, shared by +reference: entries 10 and 20 below keep their ids and their owning conversation; nothing is copied. + +```text +A: 10 ─ 20 ─ 30 ─ 40 ─ 50 ─ 70 + └──────────── 90 ─ 100 B, forked at 20 +B transcript: [10, 20, 90, 100] +``` + +Because context is derived from the transcript, B's context is what A's context was at the fork +point: the newest head among `[10, 20, ...]` is the one that existed then, and none that A added +later can be seen. Later appends, compaction and resets in A cannot affect B. Live tasks are not inherited; +continuing B is new work. + +### 2.5 History and ownership are different relationships + +```ts +interface Conversation { + readonly id: Id; + readonly parent?: { readonly conversationId: Id; readonly at: Id }; // forked from, at entry + readonly owner?: Id; // task that created it +} +``` + +```text +history: A ── fork at P ──> B B: parent = A at P, no owner; independent +ownership: A ── task T ── owns ──> C C: owner = T; a child, e.g. a subagent's conversation +``` + +A fork never enters its source's drive or cancellation scope. An owned conversation does, through +the task that owns it. A child can start with fresh or inherited context; that is separate from +which state it is initialized with (§4.3). + +## 3. Identity and commits + +### 3.1 Ids are minted while building the commit + +One session-wide sequence orders every mutation. A creation's sequence number is the object's id. + +```ts +// last committed: 99 +const entryId = await conv.commit(tx => { + tx.value(planMode).set(true); // 100 + const id = tx.entry(userKind, message); // 101 + tx.task(generationKind, { state: { input: id } }); // 102 + return id; +}); +// one batch [100–102] persisted and published; entryId === 101 here +``` + +A commit can reference ids it just minted. It cannot hand them to an external service or return +them to a caller before the batch is durable. A rejected commit consumes nothing: the next one +starts at 100 again. Numbers are positive safe integers; no uuids, no reserved ranges. + +### 3.2 State before entries + +A commit can write state and an entry together (a tool that turns on plan mode and writes its +result), and a fork at that entry must include the state. It must not include state committed later: a `/model` change made after an answer belongs to a fork +made after it, not to a fork at the answer. So the boundary is exactly the entry's commit, and the +way to know it without a field, a stamp or a lookup is write order: **in one commit, state writes +come before entries.** Sequence order is call order, so the state is numbered below the +entries, and a fork at entry X is "everything ≤ X" with nothing attached: + +```ts +// tool settlement, one commit +await ctx.commit(tx => { + tx.value(planMode).set(true); // 300 + const result = tx.entry(toolResultKind, content); // 301 + tx.settle(task, { result }); // 302 +}); + +await conv.value(model).set("claude-opus-5"); // 303, a later commit + +const b = await conv.fork(301); // has the result and plan mode; does not have the new model +``` + +The builder throws on a state write after an entry. It's a programming error, not a runtime +condition: state never needs the id of an entry from its own commit, and a violation shows up in +the first test. Tasks go anywhere; they are not inherited by forks and usually come after the +entries they name. Historical reads (§4) take an entry as their position; "before the first entry" +is the empty position. An assistant entry with tool calls is half an exchange (its results come +after it), so a fork there would end with unanswered calls and is refused; fork at the last result +instead. + +### 3.3 Failure before and after admission + +Validation failure: no writes, no ids consumed, the session keeps working. Uncertain persistence: +stop using this handle, reopen, recover the last complete batch and its `lastSeq`, continue +from that. No observer ever sees half a commit. An effect must not treat an uncertain write as +permission to redo the same external action with new ids. + +### 3.4 Request keys are ordinary state + +A caller's request id is a session value: `accept(input, requestId)` looks the key up inside the +commit; if present it returns the existing acceptance, otherwise it creates the input and records +the key in the same batch. A lost reply plus a retry yields the same accepted id. Reusing a key for +different input rejects. + +## 4. State + +State is addressed independently of the transcript. Its writes share the sequence, so one +position (an entry, §3.2) selects context and rewindable state together. + +### 4.1 Scopes + +| Scope | Rewind | Lifetime | +|---|---|---| +| Session | Sticky: one current value | The session | +| Conversation | Rewindable or sticky, chosen per address | The conversation and its fork history | +| Scratch | Never rewound, never inherited | One live execution, until settlement | + +```ts +const sessionName = sessionValue("pi.session.name"); +const plan = conversationValue("plugin.plan", { rewind: true }); +const expanded = conversationValue("ui.expanded", { rewind: false }); +const moves = conversationList("game.moves", { rewind: true }); +``` + +```ts +type Scope = { session: true } | { conversation: number } | { scratch: number /* task id */ }; +interface Address { readonly scope: Scope; readonly namespace: string; readonly key?: string; readonly rewind: boolean } +interface Value extends Address { readonly __value?: T } // phantom: the payload type +interface List extends Address { readonly __list?: T } +``` + +An address binds namespace, key, scope and rewind policy, and carries its payload type. The +constructors above make `Value`/`List` for a scope; a conversation handle binds +`conversation: id` itself, so plugin code writes `conversationValue("plugin.plan")` and never the id. +Values are strict JSON; reads return copies. A conversation address cannot be used as a session address. +Storage persists scope and policy; nothing depends on the address object surviving a restart. + +### 4.2 Vocabulary + +```text +value: tx.value(addr).get(), .set(v), .delete() +list: tx.list(addr).append(v) → element id, .remove(id), .clear(), .read(cursor, limit) +``` + +A rewindable history reads as of an entry (§3.2): a list appended at 40 and 50, cleared at 60 and +appended at 70 reads `[A, B]` at entry 55, `[]` at 65, `[C]` at 75. A remove hides one element the +same way a clear hides all of them. Clear hides; it doesn't erase +what an earlier fork needs. Deleting a rewindable value records absence. Sticky state exposes only +its current contents. + +### 4.3 Initialization is explicit; forks inherit + +A new child conversation (a subagent) chooses its initial state independently of its context: + +```ts +spawn({ prompt, context: "inherit", + values: { inherit: [generationKind.config.model, generationKind.config.thinking], + set: [[generationKind.config.selectedTools, ["read", "grep"]]] } }) +``` + +One commit creates the child, copies the selected current values, applies overrides, appends the +user prompt, creates the first generation; that generation writes the child's first `system` +entry from `system_instructions` (§8.2). Unselected values are absent in the child. Lists are not +copied unless a policy says so. + +A historical fork inherits differently: all rewindable history as of the fork position, no sticky +state unless selected, the same session state, no scratch. + +### 4.4 Capped-source lookup + +A fork stores no copies. Reading a rewindable address in fork B, forked from A at an entry: + +```text +read(B, addr, at): // at = an entry id + local version ≤ at? set → value; delete → absent + no parent? absent + read(B.parent.conversationId, addr, min(at, B.parent.at)) +``` + +Lists combine inherited ranges with local appends the same way, honouring clears at every level. +Cost is fork depth, never unrelated history. A child initialized by copying (§4.3) has an owner but no `parent`, so lookup stops there. + +### 4.5 Structured plugin state + +Plugins with structured state build it on values and lists: a checkpoint value `{ through, state }` +plus a delta list; hydrate by reading the checkpoint visible at a position and the deltas after it. +The plugin owns its delta vocabulary and reducer; storage runs nothing. + +### 4.6 Scratch + +Scratch is a task's working state while it runs: streamed frames, a checkpoint, a memo. One scope +per task, any number of addresses in it, durable across a restart, gone when the task settles. + +```ts +const frames = scratchList("frames"); +const checkpoint = scratchValue("checkpoint"); + +// a new attempt starts clean; then each frame is one small scratch commit +await ctx.scratch(sc => { sc.list(frames).clear(); }); +for await (const frame of stream) { + await ctx.scratch(sc => { sc.list(frames).append(frame); }); +} + +// after a restart, recover reads what made it to disk +const partial = await ctx.scratch(sc => sc.list(frames).read()); + +// settlement: the result becomes an entry; settle retires the scope in the same commit +await ctx.commit(tx => { + const id = tx.entry(assistantKind, assemble(partial)); + tx.settle(task, { result: id }); +}); +``` + +Three rules: + +- A scratch commit writes one task's scratch and nothing else; a main commit never writes scratch. + On JSONL that makes every commit one file (§7.4). +- `settle` deletes the task's scratch in the same commit. Crash before settle: task and scratch are + both there for recovery. After: both gone. +- A task stops writing scratch before it settles. A frame that arrives late can't recreate what + settle deleted. + +A retry is the same task (§5.3), so an attempt clears the scratch before it starts streaming; +what is on disk is always the current attempt's. Scratch is never inherited, never rewound, never in the transcript; a task that wants to keep +something writes an entry or a value at settlement. + +## 5. Tasks + +A task is durable work belonging to one conversation. Its kind supplies behaviour; its record +supplies the inputs, progress and outcome that must survive a process. + +### 5.1 The record + +```ts +type TaskRole = "start" | "inflight" | "terminal"; + +interface Task { + readonly id: Id; + readonly conversationId: Id; + readonly kind: string; + readonly status: string; // kind-specific + readonly state: State; // JSON + readonly after: readonly Id[]; // dependencies, fixed at creation + readonly background?: true; // fixed at creation; absent = foreground (§5.6) + readonly owns?: readonly Id[]; // conversations this task created (§5.7) + readonly abort?: true; // durable cancellation mark (§6.3) +} +``` + +A kind maps its statuses to three roles: + +```text +start the driver may call execute generation: pending, retry_wait, deferred +inflight intent was committed; call recover generation: streaming; job: running +terminal done; never patched again generation: done, failed, aborted +``` + +Storage indexes the role so "live tasks in this scope" is one query that decodes no state. Role is +derived from status on every write; it is not editable on its own. + +### 5.2 Kinds + +```ts +interface TaskKind { + readonly kind: string; + readonly initialStatus: string; + readonly roles: Readonly>; + readonly hooks?: HookSpecs; // the points this kind runs (§8.7) + readonly config?: Config; // the values this kind reads (below) + preview?: { // what UIs see while the task runs (§9.4) + init(scratch: ScratchReader): Preview; // built once, on attach or reopen, from durable scratch + }; // afterwards the kind mutates ctx.preview.state in place + execute(task: Task, ctx: TaskContext): Promise; + recover(task: Task, ctx: TaskContext): Promise; + abort(task: Task, ctx: TaskContext): Promise; +} +``` + +A kind that reads config declares it, so the values it depends on are on the kind and nowhere +else, typed, and usable by callers that initialize a conversation: + +```ts +type ConfigSpec = Record>; + +const generationKind = defineTaskKind({ + kind: "generation", + config: { + model: conversationValue("pi.model", { rewind: true }), + thinking: conversationValue("pi.thinking", { rewind: true }), + selectedTools: conversationValue("pi.tools.selected", { rewind: true }), + profile: conversationValue("pi.prompt.profile", { rewind: true }), + budgetMs: conversationValue("pi.tool.budget", { rewind: false }), + }, + ... +}); + +await c.config(generationKind).set({ model: "claude-opus-5" }); // or c.settings.set(...) for the generation +const cfg = await ctx.config(generationKind); // { model, thinking, ... } typed, read on the line +await c.spawn({ prompt, values: { inherit: [generationKind.config.model], + set: [[generationKind.config.selectedTools, ["read", "grep"]]] } }); +``` + +`ctx.config(kind)` reads every declared value in one batched read. A UI can list what a +conversation is configured with by walking the registered kinds' `config`. Values a plugin defines +for itself are declared on the plugin's own kind the same way. + +`TaskContext` is what an execution gets from the driver: + +```ts +interface TaskContext { + readonly signal: AbortSignal; // fresh per call; fires on abort mark or close + commit(plan: (tx: ConversationTx) => T): Promise; // bound to the task's conversation (§9.2) + scratch(plan: (sc: ScratchTx) => T): Promise; // bound to the task's scratch (§4.6) + readonly preview: Tracker; // Chord delta tracker; mutate .state, the harness flushes after each scratch commit + sleep(untilMs: number): Promise; // throws AbortError on abort + config(kind: TaskKind): Promise>; // declared values, batched + conversation(id: Id): ConversationHandle; // for driving a child (§8.5) + readonly models: ModelRegistry; // provider streams + readonly tools: ToolRegistry; // tool implementations, by name + hooks(kind: TaskKind): HookRunner; // run this kind's points (§8.7) + readonly env: ExecutionEnv; // FileSystem & Shell: exec, files, temp dirs +} +``` + +The registry maps kind names to kinds; a replacement must understand the persisted statuses and +state of its live tasks. Open reads the live tasks anyway (`inspect()`), so an unknown live kind +rejects before anything runs; terminal tasks of a forgotten kind are only a problem when read typed. + +Typed access uses the kind as the witness, on the harness and handles outside a commit and on the +`Tx` inside one: + +```ts +getTask(kind: TaskKind, id: Id): Task | undefined; +getTasks(kind: TaskKind, ids: readonly Id[]): ReadonlyMap>; +``` + +The getter returns `Task` without casts, or `undefined` when the task is missing or of another +kind; without the kind it returns the untyped task, and `kind.is(task)` narrows. + +### 5.3 What an execution may do + +Four rules, and they are the whole contract: + +1. Commit an inflight status before any external effect. A crash before that point re-runs + execute from the top; a crash after it goes through recover. +2. You may block on the world: a provider stream, a process, a human inside a hook, a conversation + you created. The driver runs executions concurrently; a blocked one holds up nothing. +3. Change your status or settle before returning. An execute that returns with its task unchanged + is a bug; the driver reports it and does not call it again. +4. Never wait on another task. If you need work done first, depend on it at creation (`after`). + If you spawn work you need the result of, it isn't another task: it's a conversation you drive + (§8.5), or a process you wait for. + +Timing is the task's business: a retry stores `notBefore` in its state and sleeps in its own +execute (`ctx.sleep`, which throws on abort). Not everything on the signal throws: pi-ai's provider stream +completes with `stopReason: "aborted"`, and the generation settles from that in-band (§8.2). A recurring job sets its next `notBefore` and +returns in a start status. The scheduler has no timers. + +### 5.4 Dependencies mean terminal, not successful + +`after` lists tasks that must be terminal before this one starts. Terminal, not done: a task whose +dependency failed or was aborted still starts and decides for itself what that means. Dependencies +are set at creation, reference existing tasks in the same ownership tree, form no cycles, and are +never edited. A task that creates a successor and depends on it in the same commit is fine; a live +task adding a dependency on something it just spawned is not, and that is what rule 4 forbids. + +### 5.5 post_tools joins an exchange + +The generation's settlement publishes the exchange atomically: + +```ts +await ctx.commit(tx => { + const assistant = tx.entry(assistantKind, message); // calls [A, B] + const tools = message.calls.map(call => + tx.task(toolKind, { state: { call, assistant } })); + tx.task(postToolsKind, { after: tools, state: { assistant, tools } }); + tx.settle(task, { assistant }); +}); +``` + +Each tool executes and settles itself: hooks, execution, result entry, terminal status. It does not +look at siblings, queues or context. post_tools starts when both are terminal, reads them with the +typed getter, and decides what happens next: + +```ts +async execute(task, ctx) { + await ctx.commit(tx => { + const tools = tx.getTasks(toolKind, task.state.tools); // Map> + const outcomes = [...tools.values()].map(t => t.state.output); // ToolOutputState + if (outcomes.some(o => o.terminate) || tx.getTask(task.id)!.abort) + return tx.settle(task, { stopped: true }); // outcome lands, no successor + const handoff = outcomes.find(o => o.handoff); + if (handoff) tx.entry(handoffKind, handoff.handoff); + const added = outcomes.flatMap(o => o.addedTools ?? []); + if (added.length) tx.value(generationKind.config.selectedTools).set([...current, ...added]); // next generation emits toolsAdded + landSteering(tx, task.conversationId); // §8.1 + tx.task(generationKind, { state: nextGeneration(task) }); + tx.settle(task, {}); + }); +} +``` + +Sequential tools are the same mechanism with `B after: [A]`. + +```text +crash before generation settles → generation live; recover +crash after it settles → assistant, tools and post_tools exist; nothing to repair +crash after the last tool → post_tools exists and is startable; nothing to infer +``` + +### 5.6 Foreground and background + +A task is foreground unless created with `background: true`; the flag is fixed at creation. Generations, tools, post_tools and automatic collapses are +foreground; jobs, schedules, spawned subagents and manual collapses are not. The +foreground set of a conversation is its live foreground tasks, plus, through any live foreground +task that owns a conversation, that conversation's foreground set. Abort and "is this conversation +busy" are defined on that set. Detached work is cancelled by its own handle. + +### 5.7 Ownership + +A task that creates a conversation adds it to `owns`; the conversation records `owner`. Both are +written in the creation commit. A task may own several (a fan-out tool driving three children); +a conversation has one owner. Ownership defines drive scope and cancellation reach. Fork +provenance is a different link (§2.5) and creates no ownership. + +## 6. Scheduling and cancellation + +### 6.1 The driver + +The driver keeps one in-memory structure: `owned`, the tasks whose execute, recover or abort this +process is currently running. Everything else it needs is on disk. + +```ts +class Driver { + private owned = new Map }>(); + private waiters: Array<{ scope: Scope; resolve: (o: "idle" | "closed") => void }> = []; + private wake = new Wake(); + private closed = false; + + constructor(private storage: Storage, private kinds: Map) { void this.loop(); } + + kick() { this.wake.set(); } // called by the harness after any commit that + // created or patched a task + + // attach a scope; resolves when its foreground is idle. The loop keeps serving it afterwards. + drive(scope: Scope): Promise<"idle" | "closed"> { + return new Promise(resolve => { this.waiters.push({ scope, resolve }); this.wake.set(); }); + } + + close() { this.closed = true; for (const o of this.owned.values()) o.controller.abort(); this.wake.set(); } + + private scopes = new Set(); // every scope ever attached, until close + + private async loop() { + while (!this.closed) { + for (const w of this.waiters) this.scopes.add(w.scope); + const convs = new Set(); + for (const scope of this.scopes) for (const c of await scope.conversations()) convs.add(c); + const live = await this.storage.scanTasks({ conversationIds: [...convs], live: true }); + + for (const task of live) { + if (this.owned.has(task.id)) continue; + const kind = this.kinds.get(task.kind)!; + if (task.abort) this.run(task, kind, "abort"); + else if (kind.roles[task.status] === "start") { + if (await this.allTerminal(task.after)) this.run(task, kind, "execute"); + } + else if (kind.roles[task.status] === "inflight") this.run(task, kind, "recover"); + } + + this.waiters = this.waiters.filter(w => !(w.scope.isIdle(live) && (w.resolve("idle"), true))); + await this.wake.wait(); + } + for (const w of this.waiters) w.resolve("closed"); + } + + private run(task: Task, kind: TaskKind, what: "execute" | "recover" | "abort") { + const controller = new AbortController(); + const ctx = this.contextFor(task, controller.signal); + const done = (async () => { + try { + if (what === "abort") { + const current = this.owned.get(task.id); // already running here? + if (current) { current.controller.abort(); await current.done; } + const fresh = await this.storage.getTask(task.id); + if (fresh && this.roleOf(fresh) !== "terminal") await kind.abort(fresh, ctx); + } else { + await kind[what](task, ctx); + const after = await this.storage.getTask(task.id); + if (after && after.status === task.status && !after.abort) { + this.report(new Error(`${task.kind}.${what} returned without changing status`)); + this.poison(task.id); + } + } + } catch (e) { + if (!isAbortError(e)) { this.report(e); this.poison(task.id); } // report → session watch event (§9.4) + } finally { + this.owned.delete(task.id); + this.wake.set(); + } + })(); + this.owned.set(task.id, { controller, done }); + } + + private allTerminal(ids: readonly number[]) { /* batched getTasks; every role terminal */ } + private roleOf(t: Task) { return this.kinds.get(t.kind)!.roles[t.status]; } + private poison(id: Id) { /* remember id; skip it in later passes */ } +} +``` + +`Wake` is a promise that resolves when set and is replaced after each wait; nothing platform-specific: + +```ts +class Wake { + private resolve = () => {}; + private p = new Promise(r => (this.resolve = r)); + set() { this.resolve(); } + wait() { const p = this.p; this.p = new Promise(r => (this.resolve = r)); return p; } +} +``` + +`AbortController` and `AbortSignal` are web standards; the driver runs anywhere JS does. + +A `drive()` call attaches its scope and resolves when the scope's foreground is idle; the loop +doesn't stop there. A spawned child in the scope keeps being served after its parent's `drive()` +has returned, so its chain never stalls, and a fork that nobody attached is never touched. + +Calls are started, not awaited: a task enters `owned` when its call starts and leaves when it +returns, and the loop goes on to the next task. A blocked execute (a stream, a process, a child +conversation, a sleep) holds up nothing. A pass runs after any owned call returns and after any +commit that created or patched a task; between passes the driver waits. It decides nothing else; +whatever should happen next was written by a settlement. + +Two guards: an `AbortError` from a task the driver cancelled is the normal unwind, any other +rejection is reported and the task is poisoned (not re-run in this process); and an execute or +recover that returns with its task's status unchanged is reported and poisoned too, so a buggy +kind can't spin. + +Each call gets a fresh controller. The abort mark on disk is the intent; the controller is how the +intent reaches an execution in this process. The abort path joins the running call before running +the kind's `abort`, so effect and abort never write the same task concurrently, and re-reads the +task first because the effect may have settled in-band (a stream that ended with `aborted`). If +nobody owns a marked task, the mark alone is enough: the next process to open sees it. + +### 6.2 Scope and outcome + +A scope is the set of conversations a `drive()` call attaches to the loop, and the rule for when +that call resolves: + +```ts +interface Scope { + conversations(): Promise; // which conversations' tasks this drive runs + isIdle(live: readonly Task[]): boolean; // when this drive call returns "idle" +} + +// conversation.drive(): the conversation and every conversation owned from it, recursively, +// whether the owning task is still live or not (a spawned child keeps running here) +const conversationScope = (root: Id): Scope => ({ + async conversations() { + const out = [root]; + for (const c of out) + out.push(...(await storage.scanConversations({ ownedFrom: c })).items.map(x => x.id)); + return out; + }, + // idle when the root has no live foreground task. The foreground set (§5.6) reaches into owned + // conversations only through live foreground tasks, so if the root has none, the set is empty; + // a child owned by a settled spawn is driven here but never keeps us here. + isIdle: live => !live.some(t => t.conversationId === root && !t.background), +}); + +// harness.drive(): every conversation, done when nothing at all is live +const sessionScope: Scope = { + conversations: async () => (await storage.scanConversations({})).items.map(c => c.id), + isIdle: live => live.length === 0, +}; +``` + +`ownedFrom: c` is "conversations whose `owner` task belongs to c", one indexed query. + +Forks are not reached through ownership, so nothing drives them by accident; each is its own +tree. A running background job keeps `harness.drive()` unresolved, which is what "running" means. Both +return `closed` if the harness closes first. There is no "suspended": a task waiting on the world +is an owned execution, and a host that doesn't want some work to proceed in this process doesn't +drive that scope. + +### 6.3 Cancellation is a mark, then cleanup + +```text +abortTask(id) set the mark on one task; reject if terminal +abort(conv) one commit: mark every task in the conversation's foreground set; + cancel queued steer/followUp input; keep write and nextRun input +``` + +Marks survive restart and are never shrunk as tasks settle. A settlement re-reads its task on the +line and writes its outcome regardless, but creates successors only if the mark is absent; in the +built-ins that check sits in the one helper that decides what happens next, and the harness +enforces it anyway: a commit issued by a marked task that creates a task or conversation fails, +and the kind is reported as buggy. An execution that ignores its signal delays its own cleanup (§6.1 joins it first), and a timeout +must not admit a second writer. `abort` must settle the task and write whatever records the exchange needs: +a tool's error result, a generation's partial and the error results for calls that never ran. + +Either order of a race is correct: if post_tools settles first, the generation it created is marked +by the same abort commit; if the mark lands first, post_tools' abort settles it without a +successor. Background work: `abortTask(job)`, or `abort(childConversation)` for a detached child. + +### 6.4 Open, close, shutdown, delete + +**Open** opens storage, checks the set of entry kinds written (§2.1) and the live tasks' kinds +against the registry, exposes `inspect()` (live tasks by role) and queries, +starts nothing. **Close** stops admitting commits, cancels owned executions, waits for them and for +admitted persistence, releases storage; it writes no outcomes, so unfinished work resumes on the +next open. **Shutdown** marks all live work, drains queued input, drives cancellation to settlement, +then closes. **Delete** of a conversation rejects while its ownership subtree has live work, then +rejects new work; entries an independent fork inherits are never erased. + +### 6.5 Telemetry, storage version, kill boundary + +Telemetry is one span per task call (execute, recover, abort) from the driver and one per commit +from the line, tagged with task id, kind, conversation and outcome; provider streams and tool +executions nest under the task span. The host supplies the tracer; nothing is recorded in storage. + +Session metadata carries a storage version, checked at open; a mismatch rejects with the versions, +and a backend exposes `migrate(from, to)` to be run before open. Durable state is provider +messages, small task states and values, so migrations are rare and mechanical. + +An execution that ignores its signal delays its own abort indefinitely, because the driver joins it +before running the kind's `abort` (§6.1). First release: everything runs in-process and +cooperatively, as in the lane harness. Later, plugin code moves into an isolate with an API +membrane (mobile-handoff `02-plugins`), which becomes the kill boundary for it: dispose the +isolate, the effect rejects, the abort path settles from the host side. Nothing in this design +changes for that; the types are already JSON across the boundary and the sink and `ExecutionEnv` +are already the objects that would be proxied. + +## 7. Storage + +One interface: point reads, batched reads, bounded indexed scans, one atomic commit. No journal +reader, no second transaction layer, no residency API. + +### 7.1 What is stored + +Conversations (`parent` for forks, `owner` for owned children), entries, tasks +(with indexed role and scheduling fields), values and their rewindable versions, list elements and +clear markers, scratch scopes. Context is not stored (§2.2). + +### 7.2 Queries, from the callers' side + +| Caller | Reads | Never | +|---|---|---| +| Driver | live tasks in a scope; their `after` ids in one batch | terminal history | +| Context | newest head by index; the range after it | a scan of the transcript | +| post_tools | tool tasks by id | sibling scans | +| UI | a transcript page before/after an id, with a limit | the whole conversation | +| Validation | named task/conversation records, entry headers | unrelated content | +| Fork | everything ≤ the entry id | today's state filtered | +| State consumer | latest version by index; a bounded list range | a replay of unrelated records | +| Reopen (SQLite) | live tasks and the conversations they need | every historical transition | + +"Latest" is an indexed descending query with limit 1, never load-and-take-last. Filters apply +before limits. + +```ts +interface Page { readonly items: readonly T[]; readonly next?: Id; readonly readAt: Id } + +interface Cursor { readonly after?: Id; readonly before?: Id; readonly limit: number } +interface ConversationQuery extends Cursor { readonly parent?: Id; readonly ownedFrom?: Id } +interface EntryQuery extends Cursor { readonly conversationId: Id; readonly kind?: string; readonly key?: string } +interface TaskQuery extends Cursor { readonly conversationIds: readonly Id[]; readonly live?: boolean; + readonly role?: TaskRole; readonly kind?: string; readonly abort?: boolean } +interface ListQuery extends Cursor { readonly at?: Id } // at = an entry id (§3.2) +interface ValueQuery { readonly scope: Scope; readonly namespace: string; readonly after?: string; readonly limit: number } // keys, ordered + +type EntryHeader = Omit; +interface Version { readonly seq: Id; readonly value: T } +interface Element { readonly id: Id; readonly value: T } + +interface Storage { + readonly lastSeq: Id; // last committed sequence + commit(batch: CommitBatch): Promise<{ first: number; last: number }>; + + getConversations(ids: readonly Id[]): Promise>; + scanConversations(q: ConversationQuery): Promise>; + + getEntries(ids: readonly Id[]): Promise>; + getEntryHeaders(ids: readonly Id[]): Promise>; + scanEntries(q: EntryQuery): Promise>; + newestHead(conversationId: Id): Promise; // §2.2 + + getTasks(ids: readonly Id[]): Promise>; + scanTasks(q: TaskQuery): Promise>; + + getValue(addr: Value, at?: Id): Promise | undefined>; // at = an entry id + scanValues(q: ValueQuery): Promise }>>; + readList(addr: List, q: ListQuery): Promise>>; + + close(): Promise; +} +``` + +`Value`, `List`, `Address` and `Scope` (session / conversation / scratch) are the address types +of §4.1; +`Conversation`, `Entry`, `Task` and `TaskRole` are the records of §§2 and 5; `CommitBatch` is §7.3. + +Pages carry a cursor and the sequence they were read at. Transcript reads return the inherited +prefix plus local entries; tasks are never inherited. Historical reads on sticky or scratch +addresses reject. Coherent multi-read operations run on the line; a cursor alone doesn't +freeze mutable state. + +### 7.3 Batches + +```ts +type StateWrite = + | { type: "value.set"; addr: Address; value: JsonValue } + | { type: "value.delete"; addr: Address } + | { type: "list.append"; addr: Address; value: JsonValue } + | { type: "list.remove"; addr: Address; element: number } + | { type: "list.clear"; addr: Address }; + +type MainWrite = + | StateWrite + | { type: "createConversation"; conversation: Conversation } + | { type: "deleteConversation"; id: number } + | { type: "entry"; entry: Entry } + | { type: "task"; task: Task } + | { type: "patch"; id: number; status?: string; state?: JsonValue; abort?: true } + | { type: "settle"; id: number; status: string; state: JsonValue }; // also retires scratch + +type CommitBatch = + | { readonly kind: "main"; readonly writes: readonly MainWrite[] } + | { readonly kind: "scratch"; readonly task: number; readonly writes: readonly StateWrite[] }; +``` + +A scratch batch can only carry state writes, by type; `ScratchTx` (§4.6) only exposes those. Write +*i* gets sequence `lastSeq + 1 + i`. + +The Tx read `lastSeq` when it started and numbered its writes +from there; the line guarantees nothing landed in between, so storage numbering from its own +`lastSeq + 1` yields the same ids. Records that carry an id (`entry`, `task`, a list element) let +storage check that for free. The harness validates against +committed state plus earlier writes in the batch (kinds validate status transitions; the harness +enforces ownership, dependencies, exchange rules and admission; nobody validates payload shapes: +stored objects are trusted, and schema validation belongs at wire boundaries); storage checks scope, +sequences and structure, prepares only touched data, persists atomically, publishes. A commit with +no writes writes nothing. + +### 7.4 Memory and JSONL + +The memory backend holds current maps (conversations, tasks, values), ordered entries per +conversation, version histories, and lookup indexes (ids, addresses, live-task membership, parent +and owner links). Indexes reference the same immutable payloads. JSONL is the memory backend plus +one line per batch: + +```json +{"first":100,"writes":[ + {"type":"value.set","conversationId":1,"namespace":"plugin.plan","value":true}, + {"type":"entry","conversationId":1,"kind":"user","content":"Inspect"}, + {"type":"task","conversationId":1,"kind":"generation","status":"pending"}]} +``` + +Write all bytes, then publish in-memory changes, never before. On open: validate complete batches, +rebuild maps and `lastSeq`, discard only an unterminated final line, fail on a malformed complete one. Durability +against process crash is required; fsync policy is a backend option. + +**Values as deltas.** A JSONL file is one ordered stream replayed from the start, which is what +Chord's delta module (`track` / `flush` / `apply`, with a path-compressing encoder per stream) is +built for. A `value.set` is written as the full value when it is the address's first version or +small, and otherwise as the ops against the previous version; replay applies either. That keeps a +large, frequently updated value (a plugin's structured state) linear in the file instead of +rewriting it on every set. Lists don't need it: elements are immutable and `remove`/`clear` are +already one op. A line encoded as ops is only meaningful after the lines before it, which is fine +for a log and is why this stays out of SQLite. + +**Scratch** goes to `session.scratch/`, one file per scope, the same delta encoding, so a +streamed assistant text or a growing tool output is an append per flush rather than a rewrite. +Retirement in the main file is authoritative; the unlink happens afterwards, and a delayed unlink +must never remove a deliberately reused scope. + +### 7.5 SQLite + +Tables: conversations, entries, tasks (current records including terminal), values and +value_versions, list_elements and clear_markers, scratch, commit_boundaries, kinds (one row per +distinct entry kind), session_metadata (storage version, §6.5). Keys start with the session id. Indexes follow the +query table: entries by (conversation, id), (conversation, kind, key, id) and a partial index +(conversation, id) where head is set; tasks by (role, id) and (conversation, role, id) with a +partial index on non-terminal roles; value_versions by (conversation, namespace, key, seq); +conversations by parent and by owner. One commit is one transaction: number the writes from `lastSeq + 1`, +insert rows, record the boundary, advance `lastSeq`. No full rewrite of anything per append. + +Values are stored whole, current and historical: a point read must not become a replay chain, and +disk is cheaper than that. Scratch is rows in the `scratch` table keyed by (task, address), written +in the task's own small transactions, and retired by `DELETE ... WHERE task = ?` inside the +settlement transaction: atomic with the result and the terminal status, nothing to unlink. Reopen +reads live tasks and what they reference, not history. + +## 8. Built-in flows + +Task kinds, not scheduler special cases. Braces group one commit. Every +attempt records its usage; every settlement retires its scratch. Hooks and external calls run +outside the line, and their decisions are re-validated inside it. + +### 8.1 Accepting input + +A conversation is busy when its foreground set is not empty. + +```text +idle accept: { append user entry; create generation, foreground } +busy accept: { append { message, mode } to the inbox } +``` + +The inbox is a conversation-scoped sticky list, `pi.inbox`. A queued item is the message itself; +nothing is written to the transcript until it lands. Three places dequeue: post_tools after a tool +exchange, the generation when it settles a final answer, and `accept` when the conversation is +idle. Each reads the list, takes the items its mode table allows, appends a `user` entry per item, +removes them, and creates the next generation, all in one commit. `cancelQueued(elementId)` is a +remove. Nothing else exists. + +| Mode | Lands at | Requests a generation | +|---|---|---| +| write | post_tools or a final answer | no | +| steer | post_tools or a final answer | yes | +| followUp | a final answer | yes | +| nextRun | an idle accept only | yes | + +Landing selects either everything eligible or the oldest item per mode, by configuration, in +admission order. Selection, landing, and the generation it triggers share one commit. Foreground +abort cancels queued steer and followUp; write and nextRun stay. + +### 8.2 Generation + +```text +create: capture the kind's declared config (model, thinking, selected tools, profile; §5.2, + rewindable values) and provider options and retry policy, so a retry uses what the + attempt used +execute: if a collapse is needed by threshold: { create collapse C; create generation G' after:[C]; settle } + desired = system_instructions hook, per conversation: keyed sections (the host's identity, + cwd, skills, context files; a plugin's plan_mode) merged across handlers and + rendered in declared order, plus tool definitions selected from the catalogue; + none of the inputs is stored + sent = the newest baseline `system` entry in the context plus the deltas after it; + deltas before it (kept by a compaction) are subsumed and ignored, by the fold + and by projection alike + { if no sent: system entry with the baseline (text + full tool definitions); + else if sent ≠ desired: system entry with the diff, per section ("the plan_mode guidance + now applies: …", "the cwd section changed: …") and toolsAdded / toolsRemoved as full + definitions (a changed schema counts); status streaming } + project the context; before_request; re-check status and mark + stream; frames to scratch +outcome: + calls { assistant; tools; post_tools; settle done } + final { assistant; resolve or continue (§8.3); settle done } + deferred { status deferred, handle } → execute again: poll with sleeps until final + retryable { status retry_wait, attempt+1, notBefore } → execute again: sleep, then stream + overflow { settle failed(overflow); create collapse C; C's settlement creates G' } + failure { partial and error results if any; settle failed } + aborted the stream ends with stopReason "aborted": { partial; error results for its calls; settle aborted } +``` + +**Prompt and tools.** Two things the lane harness fuses are kept apart. What the client wants is +config: model, thinking level and selected tools are rewindable conversation values (a fork at an +entry gets the ones in force there; a child is initialized explicitly); the catalogue and the +prompt inputs are the host's and are asked for each turn through `system_instructions`. What the model has been told is the `system` entries in the +transcript: the baseline that opened the current epoch and one entry per change delivered since. +The generation diffs the two at the top of every turn and writes only the delta. A head (summary, +handoff, reset) starts a new epoch: the context has no baseline after it, so the next generation +writes a fresh one, and any older delta a compaction kept is subsumed by it and dropped at projection; a fork carries +the sent-state in its prefix and diffs against its own config; a restart with a changed host +emits "these sections now apply" and nothing else. Projection puts the epoch's baseline in the +provider's baseline slot, later entries as system messages at their positions, and the folded tool +set as the callable list, or folds everything into one top-level prompt on providers without +native system messages. + +One task lives through retries and deferral; retries share a budget carried in state. Overflow +does not keep the generation alive: it settles and hands the retry to the collapse's settlement, +carrying the attempt count, so nothing live ever waits on the collapse. + +```text +recover streaming: frames in scratch → publish the partial and error results for its calls; settle + no frames → retry within budget, else fail +recover deferred: execute again; the handle is in state +abort: only reached if the effect didn't settle in-band (e.g. it was between attempts): + partial from scratch if any; error results; settle aborted +``` + +Usage is recorded for failed, deferred and discarded attempts too; a missing report is unknown cost, +not zero. The record is a session list, `pi.usage`, one element per attempt (conversation, task, +model, tokens, cost), appended in the settlement commit together with an update of the session +value `pi.usage.totals`, so stats are a point read and never a fold. Tools and jobs append the same +way through the sink's `usage`. Persistence and invariant failures are not provider errors: they fault the session. + +### 8.3 Tools and post_tools + +```text +tool execute: + check the call was offered by its generation; validate arguments + before_tool → allow(args) | block(reason, terminate?) (a human approval waits inside the hook) + block / invalid / unknown tool → own error result, no invocation + { status running; effective args; replay policy } + invoke the tool with the signal and the output sink (§9.3); the sink's ops go to scratch + after_tool outside the line + { result entry from the folded sink; usage; settle done with result id and ToolOutputState } +``` + +The tool never touches context, siblings or queues. A throwing `before_tool` blocks the tool. An +ordinary tool throw is an error result, not a cancellation. + +```text +recover: replay only if the stored policy allows; else an interrupted result from the checkpoint +abort: join own execution; drain scratch; own aborted error result; settle +``` + +post_tools is the code in §5.5: read the tool tasks, stop on terminate, write the handoff if one +was requested, land steering, create the next generation, settle. Its abort settles it with no +successor. + +**Blocking budget.** A tool call may not block a turn indefinitely. Tools that run processes or +child conversations create a job (or a conversation) first and wait on it with the budget; if the +budget runs out, the call settles now with what it has, `out.delegate(job)` and a diag, and the +work continues as the job: + +```ts +const job = await ctx.conversation.commit(tx => tx.task(jobKind, { background: true, + state: { cmd, cwd, origin: { tool: "bash", task: ctx.taskId, callId: toolCallId }, notify: false } })); +const done = await ctx.waitForTask(job, { budgetMs: ctx.budgetMs, signal }); // false when the budget ran out +const o = await ctx.jobOutput(job); // ToolOutputState from the job's preview or terminal state +out.replace(o.text); out.capture(o.truncation); +if (!done) { out.delegate(job); out.diag("info", `still running as job ${job}; use job wait / status / stop`, "budget"); + await ctx.conversation.commit(tx => tx.patch(job, { state: { notify: true } })); } +``` + +The budget is the generation's config (`budgetMs`, default a few minutes); the tool kind passes it +as `ctx.budgetMs`. `waitForTask` throws on abort like `sleep`; a tool that started a job for a +non-backgrounded call aborts it then (`abortTask`). A tool that awaits arbitrary work instead of +delegating is raced against the budget by the tool kind, which on expiry creates a job that adopts +the promise and retargets the sink into the job's scratch; that path is best-effort across a crash +(the job recovers as `lost`). + +`new_context` is a tool that calls `out.handoff(message)`; the reset happens here, after the exchange +is complete, never inside the tool, because a head landing mid-exchange would cut it in half. Conflicting handoffs reject rather than pick an order. + +A final answer's continuation is decided in the generation's settlement: `on_yield` may request an +explicit continuation (a user entry and a generation); eligible followUp input lands and starts the +next turn; an owned conversation's final answer settles nothing by itself, its owner is a tool +driving it (§8.5). An idle conversation whose tail is a user entry creates no work by inference. + +### 8.4 Collapse + +```text +create: capture the prefix to replace (ending on a complete exchange), the newest head id, + the context, settings; one live collapse per conversation +execute: { status summarizing } + before_collapse (may decline or supply the summary); call the summarizer; candidate to scratch + { if no head newer than the captured one: append summary (head = last replaced entry); settle done + else: settle failed(stale) } +``` + +Entries that landed while it ran have ids above the summary's head number, so they stay in context +without anyone doing anything; a summary may land while a generation streams, since the stream +already projected its context and the replaced prefix is old. Automatic collapses (threshold, +overflow) are foreground; manual ones are background. A collapse never creates a generation except +through the settlement chain in §8.2. Abort records usage and settles without publishing. Other +compaction schemes (windowing, pruning, replacing single entries) are other head kinds (§2.1): +content for what is replaced, `compose` for what is kept. + +### 8.5 Subagents + +A subagent is a conversation. There is no subagent task, and there is one tool, `subagent`, whose +argument is a command: + +```ts +type SubagentCommand = + | { command: "run"; prompt: string; context?: "fresh" | "inherit"; tools?: string[] } // foreground + | { command: "spawn"; prompt: string; context?: "fresh" | "inherit"; tools?: string[] } // background + | { command: "send"; id: Id; text: string } + | { command: "status"; id: Id } + | { command: "wait"; id: Id } + | { command: "stop"; id: Id }; +``` + +```text +run: { create child conversation (this task owns it); initial values; prompt; first generation } + await child.drive() // blocked on the world; the tool stays inflight + child.answerTo(prompt entry); { result entry; settle } + recover: the child exists; drive it again + abort: the mark on this tool marks the child's foreground set through ownership; error result + +spawn: the same creation commit; settle at once with the child's id +send: child.accept(text) → queued if the child is busy +status: the child's tail and live tasks +wait: await child.drive(); child.answerTo(the last sent prompt); settle (recover: drive again) +stop: child.abort() +``` + +`id` is whatever `spawn` returned, passed back by the model; the tool resolves it with +`harness.conversation(id)`. No map anywhere, nothing to rebuild after a restart. Between `spawn` and +`wait`, whatever drives the parent's tree drives the child. A spawned child is owned by a task that +is already terminal, so it is not in the parent's foreground set and aborting the parent leaves it +alone; `stop` is the way to cancel it. Inherited context for a child is the parent's context at the +last complete exchange before the launching one. + +### 8.6 Jobs and schedules + +A job is a background task that runs a process. Its state: + +```ts +interface JobState { + cmd: string[]; cwd: string; limits?: ShellOutputLimits; + origin?: { tool: string; task: Id; callId: string }; // the call that started it, so UIs render it with that tool's component + notify?: boolean; // the call returned early: append a notice on completion + every?: number; notBefore?: number; rerun?: "safe"; // recurrence and recovery policy + startedAt?: number; output?: ToolOutputState; exitCode?: number; +} +``` + +Starting one is creating the task; a tool does it in its execute (§8.3), a UI in a commit: + +```ts +const jobId = await conv.commit(tx => + tx.task(jobKind, { background: true, state: { cmd: ["npm", "test"], cwd, every: 6 * 3600_000 } })); +``` + +```ts +const jobKind: TaskKind = { + kind: "job", + initialStatus: "planned", + roles: { planned: "start", running: "inflight", exited: "terminal", killed: "terminal", lost: "terminal" }, + + async execute(task, ctx) { + if (task.state.notBefore && task.state.notBefore > Date.now()) await ctx.sleep(task.state.notBefore); + await ctx.commit(tx => tx.patch(task.id, { status: "running", state: { ...task.state, startedAt: Date.now() } })); + const output = scratchValue("output"); + const result = await ctx.env.exec(task.state.cmd, { // blocked on the world + cwd: task.state.cwd, + capture: { limits: { maxBytes: 64_000, maxLines: 2000, retain: "tail" }, spill: true }, + onUpdate: update => void ctx.scratch(sc => sc.value(output).set(applyShellOutputUpdate(sc.value(output).get(), update))), + }, { signal: ctx.signal }); + await this.finish(task, ctx, result); + }, + + async recover(task, ctx) { // the process died with the old owner; nothing to adopt + if (task.state.rerun === "safe") return this.execute(task, ctx); + await ctx.commit(tx => tx.settle(task.id, "lost", task.state)); + }, + + async abort(task, ctx) { // only reached if exec didn't return on the signal + await ctx.commit(tx => tx.settle(task.id, "killed", task.state)); + }, + + async finish(task, ctx, result) { + const output = await ctx.scratch(sc => sc.value(scratchValue("output")).get()); + await ctx.commit(tx => { + if (ctx.signal.aborted) return tx.settle(task.id, "killed", { ...task.state, output }); + const r = getOrThrow(result); + if (task.state.notify) // the call that started it returned early (§9.3): tell the model + tx.entry(noticeKind, { text: `job ${task.id} (${task.state.origin?.tool}: ${task.state.cmd.join(" ")}) finished, exit ${r.exitCode}` }); + if (task.state.every) // recurring: same task, next time + tx.patch(task.id, { status: "planned", state: { ...task.state, notBefore: Date.now() + task.state.every } }); + else + tx.settle(task.id, "exited", { ...task.state, output, exitCode: r.exitCode }); + }); + }, +}; +``` + +`ctx.env` is the harness's `ExecutionEnv` (`FileSystem & Shell`); `exec` runs the command, captures +output through `onUpdate` (here into scratch, so a watcher can show it live) and returns when it +exits or the signal fires. `running` is inflight, so a restart goes through `recover`; since the +shell doesn't hand out an adoptable process, recover reruns if the job declared that safe and +otherwise settles `lost`. A recurring job is the same task looping `planned → running → planned`; +`abortTask(id)` ends it wherever it is. Output stays in scratch while it runs and in the terminal state +after, where `jobOutput`, the `job` tool and UIs read it; there is no result entry, since a call that +waited already carries the output. A job whose call returned early appends a `notice` entry (a kind +that projects to a `SystemMessage`) when it finishes, so the next generation learns of it without +polling. Interval and catch-up policy are in state, +so a restart never launches a backlog. + +### 8.7 Hooks + +Hooks belong to kinds. A kind declares its points and their types; plugins register handlers per +kind and point; the harness only routes: + +```ts +type HookPoints = Record; + +interface GenerationHooks extends HookPoints { + system_instructions: { input: { conversationId: number; config: GenerationConfig }; // per conversation, not per harness + output: { sections: Record; tools: readonly Tool[] } }; + // sections merge by key across handlers (a plugin adds plan_mode; undefined removes); + // the generation renders them in the host's declared order; tools are definitions + before_request: { input: { request: ProviderRequest }; output: { request?: ProviderRequest } }; + after_response: { input: { response: AssistantMessage; usage: Usage }; output: void }; + on_yield: { input: { answer: AssistantEntry }; output: { continue?: string } }; +} +interface ToolHooks extends HookPoints { + before_tool: { input: { toolCallId: string; toolName: string; args: JsonObject }; + output: { args?: JsonObject; block?: { reason: string; terminate?: boolean } } }; + after_tool: { input: { toolCallId: string; output: ToolOutputState }; output: void }; +} +interface CollapseHooks extends HookPoints { + before_collapse: { input: { reason: "manual" | "threshold" | "overflow"; context: readonly Entry[] }; + output: { decline?: boolean; instructions?: string; summary?: string } }; +} + +const generationKind: TaskKind = { ..., hooks: { before_request: {}, after_response: {}, on_yield: { failClosed: false } } }; +const toolKind: TaskKind = { ..., hooks: { before_tool: { failClosed: true }, after_tool: {} } }; + +// a plugin registers a handler +harness.hooks.on(toolKind, "before_tool", async ({ toolName, args }) => { + if (toolName === "bash" && !(await ui.approve(args))) return { block: { reason: "denied" } }; // may wait for a human +}); + +// the kind runs it +const decision = await ctx.hooks(toolKind).run("before_tool", { toolCallId, toolName, args }); +``` + +A handler can be scoped to a conversation instead of the whole harness; for `system_instructions` +that is the normal case, since a subagent is a different agent: + +```ts +root.hooks.on(generationKind, "system_instructions", hostInstructions, { subtree: true }); // root and what it owns +child.hooks.on(generationKind, "system_instructions", investigatorInstructions); // this child only; overrides for it +``` + +Scoped handlers run after harness-wide ones for the same point, innermost conversation last, so a +child's sections win by key. Registration is process state, not durable; a presentation registers +when it attaches. + +Handlers run in registration order, each seeing the merged output so far; the kind decides what +to do with the result. A handler runs outside the line and its decision is applied inside a commit +that re-checks the task's status and mark. Handlers may run again after a crash, so their external +side effects need their own idempotence. `failClosed` says what a throw means: for `before_tool` +a throw blocks the tool; for the others a throw is reported and skipped. `before_tool` may wait as +long as it likes (a human approval is a hook that waits); the task is still `planned` meanwhile +and a crash simply asks again. A plugin kind declares its own points the same way, and a plugin +that replaces a built-in kind keeps its hook names so existing handlers keep working. Commit +listeners are not hooks: they observe, never decide, and never await a commit on the same line. + +## 9. API + +### 9.1 Handles + +```ts +interface ConversationHandle { + readonly id: Id; + snapshot(): Promise; + accept(input, options?: { requestId? }): Promise<{ entryId: number }>; + prompt(input, options?): Promise; // accept + drive + answerTo(entryId) + answerTo(entryId: Id): Promise; // last assistant entry after it and before the next user entry + drive(options?: { signal? }): Promise<"idle" | "closed">; + steer(input); followUp(input); nextRun(input); write(message); + cancelQueued(elementId): Promise<"cancelled" | "not_found">; + abort(): Promise; + collapse(options?: { instructions? }): Promise; // collapse task id + reset(handoff?): Promise; + fork(at: Id | "start", options?: { abort?: boolean; values? }): Promise; // at an entry id + spawn(options): Promise; // child conversation id + value(addr) / list(addr) + config(kind: TaskKind): { get(): Promise>; set(partial: Partial>): Promise }; + readonly hooks: { on(kind, point, handler, o?: { subtree?: boolean }): () => void }; // scoped to this conversation (§8.7) + readonly settings: ConfigHandle; // sugar: config(generationKind) + commit(plan: (tx: ConversationTx) => T): Promise; +} + +interface Harness { + root(): Promise; + conversation(id): Promise; + conversations(q?): Promise>; // independent ones: owner === undefined + inspect(): Promise<{ start: Task[]; inflight: Task[] }>; + drive(options?): Promise<"idle" | "closed">; + getEntry(id); getEntry(kind, id); getEntries(ids); getEntries(kind, ids); entries(conversationId, page); + getTask(id); getTask(kind, id); getTasks(ids); getTasks(kind, ids); + abortTask(id): Promise; + value(addr) / list(addr) (session scope) + commit(plan: (tx: Tx) => T): Promise; + watch(conversationId, { tail: number; values?: Address[]; raw?: boolean }): Promise>; // §9.4 + watch(): Promise>; + deleteConversation(id); shutdown(); close(); +} +``` + +`drive`'s signal cancels the caller's wait, not the work. `fork({ abort: true })` marks the source's +foreground set in the same commit that creates the fork, for "go back to that point"; which +conversation a UI treats as current is the UI's business. `prompt` returns the answer to this input, or +`undefined` if the run ended without one. `answerTo(entryId)` is the read behind it: the last +assistant entry after the input and before the next user entry (or the tail), one bounded range +query; so a retried `accept` with the same `requestId` followed by `drive` and `answerTo` yields +the same answer however far the conversation has moved since. Nothing else maps inputs to results. + +### 9.2 Commits + +```ts +const entryId = await harness.commit(tx => { + tx.value(planMode).set(true); // state first (§3.2) + const id = tx.entry(noteKind, conversationId, content); // final id, inside the closure + tx.task(reminderKind, { background: true, state: { about: id } }); + return id; // any value; resolved after commit +}); +``` + +```ts +interface Tx { + // reads (committed state) + getEntry(id) / getEntry(kind, id) / getEntries(...) / getTask(id) / getTask(kind, id) / getTasks(...) + value(addr: Value): { get(at?): T | undefined; set(v: T): void; delete(): void }; + list(addr: List): { append(v: T): Id; remove(id: Id): void; clear(): void; read(q): Page }; + // writes; state before entries (§3.2) + entry(kind: EntryKind, conversationId: Id, content: C): Id; + task(kind: TaskKind, spec: { state: S; after?: Id[]; background?: true; owns?: Id[] }): Id; + patch(task: Task | Id, changes: { status?: string; state?: Partial }): void; + settle(task: Task | Id, outcome: Partial & { status?: string }): void; // terminal; retires scratch + createConversation(spec): Id; deleteConversation(id: Id): void; +} +``` + +The implementation is small enough to show. It numbers as it goes, keeps the one ordering rule, and +hands storage a batch; storage numbers from its own `lastSeq` and gets the same ids (§7.3). + +```ts +class Tx { + private writes: MainWrite[] = []; + private seq: number; + private sawEntry = false; + constructor(private storage: Storage, private kinds: Kinds) { this.seq = storage.lastSeq; } + + private push(w: MainWrite): number { this.writes.push(w); return ++this.seq; } + private state(w: StateWrite): void { + if (this.sawEntry) throw new Error("state writes must come before entries (§3.2)"); + this.push(w); + } + + value(addr: Value) { + return { + get: (at?: number) => this.storage.getValue(addr, at).then(v => v?.value), // committed state + set: (value: T) => this.state({ type: "value.set", addr, value }), + delete: () => this.state({ type: "value.delete", addr }), + }; + } + list(addr: List) { + return { + append: (value: T) => { this.state({ type: "list.append", addr, value }); return this.seq; }, + remove: (element: number) => this.state({ type: "list.remove", addr, element }), + clear: () => this.state({ type: "list.clear", addr }), + read: (q: ListQuery) => this.storage.readList(addr, q), + }; + } + + entry(kind: EntryKind, conversationId: number, content: C): number { + this.sawEntry = true; + const id = this.seq + 1; + const head = kind.head?.(content, id); + this.push({ type: "entry", entry: { id, conversationId, kind: kind.kind, content, head } }); + return id; + } + task(kind: TaskKind, spec: { conversationId: number; state: S; after?: number[]; background?: true; owns?: number[] }): number { + const id = this.seq + 1; + this.push({ type: "task", task: { id, kind: kind.kind, status: kind.initialStatus, ...spec } }); + return id; + } + patch(id: number, changes: { status?: string; state?: JsonValue; abort?: true }) { this.push({ type: "patch", id, ...changes }); } + settle(id: number, status: string, state: JsonValue) { this.push({ type: "settle", id, status, state }); } + + getTask = this.storage.getTask; getTasks = this.storage.getTasks; // reads: committed state + getEntry = this.storage.getEntry; getEntries = this.storage.getEntries; + + batch(): CommitBatch { return { kind: "main", writes: this.writes }; } +} + +// the line: one commit at a time +async commit(plan: (tx: Tx) => T): Promise { + return this.line.run(async () => { + const tx = new Tx(this.storage, this.kinds); + const result = await plan(tx); // throws → nothing written + const batch = tx.batch(); + if (batch.writes.length === 0) return result; + this.validate(batch); // ownership, dependencies, exchange rules, marks + await this.storage.commit(batch); + this.publish(batch); // watchers + if (batch.writes.some(w => w.type === "task" || w.type === "patch" || w.type === "settle")) this.driver.kick(); + return result; + }); +} +``` + +`ConversationTx` is the same with `conversationId` bound. `ScratchTx` is the `value`/`list` half +with the batch tagged `scratch`. The closure runs on the line: reads see committed state, writes are buffered and validated +in order, ids are final when returned, and a throw discards everything. The outer promise resolves +with the closure's value after persistence and publication, which is when ids may escape. +`ConversationHandle.commit` binds the conversation; `TaskContext.commit` is the same for task +code. + +### 9.3 Task and tool integration + +```ts +interface Tool { + readonly name: string; readonly description: string; readonly parameters: JsonSchema; + readonly output?: ShellOutputLimits; // retained window for text output + readonly replay?: "safe" | "never"; + execute(toolCallId: string, params: TParams, signal: AbortSignal, + out: ToolOutput, ctx: ToolContext): Promise; +} + +// the sink; everything a tool produces goes through it (mobile-handoff/01-harness/04-tool-output) +interface ToolOutput { + write(text: string): void; // append to the text block + image(image: ImageContent): void; // images are never windowed + replace(text: string): void; // replace the retained text wholesale + capture(metadata: ShellOutputMetadata): void; // truncation totals / spill path, no text resent + readonly details: TDetails; // the tool's own details object; mutate it + usage(usage: Usage): void; // accumulates + addTools(names: string[]): void; // replaces; post_tools updates selected tools (§5.5) + terminate(value: boolean): void; // replaces; orthogonal to how execution ended + handoff(message: string): void; // replaces; new_context asks for a reset (§8.3) + delegate(job: Id): void; // this call's work continues as that job (§8.6); generic, so UIs can follow it + diag(severity: "info" | "warn" | "error", message: string, code?: string): void; // commentary, see below +} + +interface ToolContext { + readonly taskId: Id; // the tool task; identity for origin/for links + readonly budgetMs: number; // blocking budget (§8.3) + readonly env: ExecutionEnv; // FileSystem & Shell + readonly conversation: ConversationHandle; // the tool's own conversation (for subagents, §8.5) + readonly harness: Harness; // conversation(id), abortTask, for spawned children + waitForTask(id: Id, o: { budgetMs?: number; signal: AbortSignal }): Promise; // true = terminal + jobOutput(id: Id): Promise; // preview or terminal state +} + +// what the sink folds to; recorded on the tool task and the tool_result entry at settlement +interface ToolOutputState { + content: (TextContent | ImageContent)[]; + details: JsonValue; + usage?: Usage; + addedTools?: string[]; + terminate: boolean; + handoff?: string; + delegated?: Id; + diags: readonly { severity: "info" | "warn" | "error"; message: string; code?: string }[]; + truncation: ShellOutputTruncation; // totals over everything ever written +} +``` + +`execute` returns nothing and reports failure by throwing; the harness sets `isError` when it +rejects, and a tool may throw anything. `terminate` is orthogonal to how execution ended, so a +failing tool can still stop the turn (`out.terminate(true); throw error`). Usage and `addTools` go +through the sink rather than a return value so that a replayed tool can seed from durable state. +The sink's text ops (`write`, `replace`, and the `slide` the env emits for a moving tail) are what +scratch and the watch stream carry, so a UI shows output live without the tool doing anything. + +**Diagnostics are a channel, not text.** Commentary about a call (it was truncated, the output +spilled to a file, the path was resolved differently, the file changed on disk since it was read, +the search stopped at 500 matches) goes through `out.diag`, never into the text the model reads as +the tool's data. The harness emits the ones it owns: the sink calls `diag` itself when it truncates +or spills, the path resolver when it corrects a path, the blocking budget when a call continues as +a job. A tool adds only what it alone knows. Projection renders the data first and the diagnostics +after it, delimited: + +```text +…last line of the file + +[warn] output truncated: 2,000 of 51,204 lines shown; full output at /tmp/pi/out-4421.log + +``` + +so the model can parse tool output as data, a plugin can post-process it without stripping notices +it doesn't know, and a UI renders diagnostics as callouts by severity. `isError` says whether the +call failed; a `warn` diag does not change it. The transcript records exactly the commentary the +model saw, because it is in the entry's `ToolOutputState`. + +### 9.4 Watch + +The transport is the commit stream, gap-free and replayable. The interface is typed events derived +from it by the harness, and a view the harness keeps current, so a client renders and never parses +commits: + +```ts +interface WatchHandle { + readonly view: View; // captured on the line at watch(); folded before each event is delivered + start(listener: (event: Event) => void): void; // delivers everything since the capture, then live + resnapshot(): Promise; // fresh capture, same subscription (when lagging) + unsubscribe(): void; +} + +interface ConversationView { + readonly conversation: Conversation; + readonly entries: readonly Entry[]; // the last `tail` entries; older ones via entries(id, { before }) + readonly context: readonly Id[]; // the derived context, as ids + readonly tasks: readonly Task[]; // live tasks, typed by kind (retry attempt, deferred handle, ToolOutputState ... in state) + readonly inbox: readonly Element[]; + readonly values: ReadonlyMap; // every value the registered kinds declare in config, plus any asked for + readonly previews: ReadonlyMap; // per live task: the kind's tracked preview (§5.2) + readonly faulted: boolean; + readonly readAt: Id; +} + +type ConversationEvent = + | { type: "entry"; entry: Entry } + | { type: "task_start"; task: Task } + | { type: "task_update"; task: Task; previous: Task } + | { type: "task_end"; task: Task } + | { type: "task_output"; task: Id; ops: readonly DeltaOp[] } // ops on the task's preview; applied to view.previews + | { type: "value"; addr: Address; value: JsonValue | undefined } + | { type: "inbox"; items: readonly Element[] } + | { type: "context"; ids: readonly Id[] } // a head landed + | { type: "fault"; error: unknown } | { type: "closed" }; + +const w = await h.watch(c.id, { tail: 100, values: [myPlugin.config.mode] }); +render(w.view); +w.start(event => render(w.view, event)); // w.view is already folded when the listener runs +``` + +The view is authoritative and the events are wake-ups: a renderer is `apply(view)`, diffing against +what it last drew (entries keyed by id, tool components keyed by task id), and it may ignore an +event's payload and still be correct. The payload is for skipping work on a hot path (`task_output`: +redraw one component from its preview) and for logs and tests. + +Because every piece of work is a task of a known kind, the four task events carry what the lane +harness needed a name per case for, and cover plugin kinds without anyone adding one: + +| lane event | v3 | +|---|---| +| run_start / run_end | task_start / task_end where `generationKind.is(task)` | +| retry_scheduled, run_suspend | task_update with status `retry_wait` (attempt, notBefore) / `deferred` (handle) in state | +| message_update | task_output on the generation; `view.previews` holds the partial assistant message | +| tool_start / tool_update / tool_end | task_start / task_output / task_end on the tool task; `ToolOutputState` in state | +| compaction_start / compaction_end | task_start / task_end where `collapseKind.is(task)`; reason and summary id in state | +| navigation_start / end | a conversation created (session watch); with `abort`, task_end aborted on the source | +| config_update, value_update | value | +| queue_update | inbox | +| entry_added, message_end | entry | + +Events are delivered per commit, in commit order, all events of one commit before any of the +next, so a terminal task and its successor never show as an idle gap. `watch` captures the view +and registers the subscription in one serialized step on the line, so nothing falls between them. +Task output travels as delta ops only (§7.4); the folded preview lives in the view, never in the event. `resnapshot` marks a +barrier on the delivery tail while the line holds the new capture, so calling it from inside the +listener neither deadlocks nor refolds stale state. Buffering is bounded; a lagging client gets a +`fault` and calls `resnapshot`. A listener never awaits a commit on the same line. + +Events are proportional to their change, and the view fold is a generic reducer, +`applyEvent(view, event)`, exported by the harness package and needing no kinds: append the entry, +upsert the task, apply the ops to `previews[task]`, set the value. The harness maintains `w.view` +with it, and a UI in another process runs the same reducer on the same events; nothing is +re-diffed for the wire: + +```ts +// worker // ui process +const w = await h.watch(c.id, { tail: 100 }); on("view", m => { view = m.view; ui.apply(view); }); +send({ type: "view", view: w.view }); +w.start(e => send({ type: "event", event: e })); on("event", m => { applyEvent(view, m.event); ui.apply(view, m.event); }); +``` + +The replication contract, so a reducer in another language can't drift: events of one commit +travel as one message; an `Address` serializes to one canonical string key; and preview ops are +Chord delta ops (`packages/chord/src/delta`, mobile-handoff `01-delta`): `r` replace, `s` set, `d` +delete, `a` append, `t` truncate, `p` splice, with the wire codec's interning optional for thin +clients. That applier is already the mobile port; the reducer adds its ten cases on top and the +harness package's implementation is the reference, tested by porting `delta.test.ts` first. + +`task_output.ops` are ops on the task's *preview*, which is a Chord delta tracker the kind mutates +in place (`ctx.preview.state`): the generation applies each stream event to a tracked partial +message, the tool's preview is the sink's tracked state, a job's is the same. The harness flushes +the tracker after each scratch commit; a streamed token is one `["a", path, delta]`, computed from +dirty paths and a memcmp, never a recomputed object (a reassigned preview would flush as a full +replace). On attach or reopen the harness builds the preview once with `preview.init(scratch)` +and the first flush is the base. Fan-out to several in-process watchers uses `applyImmutable` or +copies the batch, since `apply` adopts `r` payloads. + +The lane-shaped +snapshot (`operation`, `runningTools`, `retry`, `deferred`, `streamingMessage`) is a pure function +of `view.tasks` and `view.previews`; a renderer that wants it computes it. `h.watch(c.id, { raw: +true })` delivers the commits themselves, for a receiver that has a harness and runs the fold. + +There is a session-level watch for what isn't one conversation's: + +```ts +interface SessionView { + readonly conversations: readonly Conversation[]; + readonly values: ReadonlyMap; // session-scoped: pi.usage.totals, pi.session.name, ... + readonly faulted: boolean; + readonly readAt: Id; +} +type SessionEvent = + | { type: "conversation"; conversation: Conversation; change: "created" | "deleted" } + | { type: "value"; addr: Address; value: JsonValue | undefined } + | { type: "usage"; row: UsageRow; totals: Usage } + | { type: "report"; task?: Id; kind?: string; error: unknown } // hook errors, poisoned tasks + | { type: "fault"; error: unknown } | { type: "closed" }; +harness.watch(): Promise>; +``` + +### 9.5 End to end + +```ts +const h = await Harness.open(storage, options); +const c = await h.root(); +const w = await h.watch(c.id, { tail: 100 }); render(w.view); w.start(e => render(w.view, e)); + +const answer = await c.prompt("Inspect the parser"); + +const driving = c.drive(); await c.abort(); await driving; // durable intent, then cleanup + +const child = await c.spawn({ prompt: "Inspect only tests", context: "fresh", + values: { inherit: [model] } }); +await h.drive(); // drives the child too +console.log(await h.conversation(child)); + +const alt = await c.fork({ atEntry: answer.id }); +await alt.prompt("Try a different implementation"); // source stays parked + +w.unsubscribe(); await h.close(); +``` + +```ts +const h = await Harness.open(storage, { + models, + tools: [readTool, writeTool, bashTool, ...pluginTools], // subagent and job tools are built in + kinds: { entry: pluginEntryKinds, task: pluginTaskKinds }, // added to the built-ins + replace: { generation: myGenerationKind }, // swap a built-in by name; same statuses, hook names + rootValues, +}); +h.kinds.generation; h.kinds.tool; h.kinds.postTools; h.kinds.collapse; h.kinds.job; // whatever is registered under the name +``` + +The built-in entry kinds (`user`, `assistant`, `tool_result`, `system`, `notice`, `summary`, +`handoff`, `reset`) and task kinds (`generation`, `tool`, `post_tools`, `collapse`, `job`) are +registered by `open` itself, because `accept`, `prompt`, `steer` and `collapse` cannot work without +them. A replacement is registered under the built-in's name and must understand its persisted +statuses and keep its hook names, so existing handlers keep working; a wrapper that delegates to +the original is the usual shape. `h.kinds.` is how handles and plugins refer to whatever is +currently registered, so `c.settings` is the config of the generation kind in use, not of a +specific import. + +`Harness.open` takes storage, models, tools, additional kinds, replacements and initial root values; it creates +the root only for empty storage and never creates work on reopen. + +## 10. Validation + +### 10.1 Backend conformance + +One validated mutation stream replayed into memory, JSONL (close, reopen) and SQLite (close, +reopen), compared on: current objects and live-task metadata, transcript pages and inherited +prefixes, derived contexts at several boundaries, value tombstones and list clears, request keys, +scratch lifetimes, next allocated sequence. Randomized small histories plus explicit deep-fork +cases: fork before and after a delete, before and after a clear, before a later summary, nested +source cutoffs, sticky state untouched by historical reads. + +### 10.2 Failure and concurrency + +| Scenario | Required | +|---|---| +| Persistence paused after construction | readers see the old complete state | +| Commit durable, reply lost | reopen sees the batch; keyed retry returns the same id | +| Main and scratch writes in one batch | reject, nothing written | +| Torn final JSONL line | only that line discarded | +| Malformed complete line | open fails; no silent truncation | +| Retirement committed, unlink fails | old scratch invisible | +| Effect returns as abort starts | one owner; join then abort | +| Parallel tools settle in either order | post_tools starts once | +| Abort vs post_tools settlement | no unmarked successor | +| Child finishes while its owning tool is marked | tool settles aborted, not done | +| Two overlapping drives | one execution per task | +| A drive caller cancels its wait | other callers and tasks unaffected | +| Summary lands after a reset | rejected as stale | +| Cancel vs land of the same inbox item | one wins on the line; the loser reports it | +| Overflow retry while collapse runs | no live task waits on the collapse | +| Watch registration races a commit | base includes it or the stream delivers it | + +Faux providers, fake processes and clocks, storage barriers; both orders of every race; failed +commits consume no ids; old task ids remain queryable after compaction. + +### 10.3 Performance + +Workloads: many short turns with and without tools; frequent compaction plus cold reopen; parallel +and sequential tools; large streaming output; deep forks with repeated historical reads; many +finished child conversations with few live tasks; many live background tasks; queued input and +cancellation. Measure CPU, wall latency, RSS and heap, bytes on disk including scratch and WAL, +query counts and rows decoded. Report backend-owned memory separately from live execution data; +JSONL growth with history is expected, SQLite must not keep a full-history copy. + +The pico2 prototype's 9.7 s against 35.9 s on 2,000 faux turns is a reason to measure this design +the same way, not a result for it. Its lessons are the ones above: cheap live queries, batched +reads, small writes, no history scans on the hot path. diff --git a/packages/agent/docs/pico/pico-work.md b/packages/agent/docs/pico/pico-work.md new file mode 100644 index 000000000..e664c60a1 --- /dev/null +++ b/packages/agent/docs/pico/pico-work.md @@ -0,0 +1,240 @@ +# pico implementation plan + +Clean room. Nothing from pico2 is copied; its tests are read for the bugs they caught (a failed +generation that never ended its run, refs inside task patches, a collapse deadlocking appends, +quiescence firing during a retry wait, watch dropping late deltas) so that packages 6, 9, 12 and +16 each carry a case for them. + +Bottom up. Each package is a module with its own tests and no forward references; the suite is +green after every one. The reference for every detail is `pico-v3.md`; `pico-usage-guide.md` +shows the surface each package has to end up with. + +## 1. Types and ids + +`Id`, `Entry`, `Task`, `TaskRole`, `Conversation`, `Address`/`Value`/`List`/`Scope`, +`Write`/`CommitBatch`, `Page`/`Cursor` and the query shapes (§2, §4.1, §5.1, §7.2–7.3). No code. + +Test: it compiles; a fixture file with one instance of every record shape. + +## 2. Memory storage + +`Storage` and `MemoryStorage`: `commit` numbering from `lastSeq`, point reads, `scanEntries` / +`scanTasks` / `scanConversations` with cursors, `newestHead`, value versions and list elements read +at a position, `ownedFrom`, the set of entry kinds written. + +Tests: every row of the §7.2 query table against hand-built batches; ids are `lastSeq + 1 + i`; a +live-task scan decodes no terminal rows; `remove` and `clear` hide by position. + +## 3. The line and `Tx` + +`commit(plan)` on a serialized line: buffered writes, ids final at call time, state-before-entries +throws, a throwing plan discards everything, publish after persist, `kick` when a batch touched a +task. `value` / `list` / `entry` / `task` / `patch` / `settle` build the batch of §7.3. + +Tests: concurrent commits serialize; a rejected commit consumes no ids; each builder verb produces +the expected write; reads inside a plan see committed state only. + +## 4. Entry kinds and context + +`EntryKind`, the registry, the built-in kinds (`user`, `assistant`, `tool_result`, `system`, +`notice`, `summary`, `handoff`, `reset`) with `project` and `head`; context = newest head + range; +`compose`; projection to pi-ai messages with tool results ordered by call index; the `system` +kind's epoch fold (newest baseline plus the deltas after it, older deltas dropped) and its +projection into the baseline slot / `SystemMessage`s. + +Tests: summary keeps the tail; handoff and reset; repeated compaction subsumes; a head number below +the previous head's id is rejected; the fold on a context that kept an old delta; tool-result order. + +## 5. Forks and historical reads + +`createConversation` with `parent`, the shared prefix in `scanEntries`, capped-source lookup for +values and lists, refusal of a fork inside an unresolved exchange. + +Tests: a fork sees the head and the values in force at its entry; heads the source adds later are +invisible; deep fork chains; state committed after the entry (a model change) is not in the fork. + +## 6. Task kinds and the driver + +`TaskKind`, the registry, `TaskContext` (`commit`, `scratch`, `sleep`, `signal`, `config`, +`hooks`), the driver of §6.1 (`owned`, `Wake`, attached scopes, `drive` as a waiter, poison guards, +abort join and re-read), scopes of §6.2. Test kinds only: a counter, a blocker, a kind that +returns without changing status, a kind that ignores its signal. + +Tests: roles drive execute / recover; a blocked execute holds nothing up; the spin guard; `after` +gates a start; `drive` resolves on foreground idle while a background task keeps running and the +loop keeps serving it; abort marks in both race orders; reopen recovers inflight; a mark written +by one process is applied by the next. + +## 7. Scratch + +Scratch batches, one task per batch, `ScratchTx`, retire on settle, sidecar-independent (memory). + +Tests: a crash before settle keeps scratch; settle deletes it; a write after settle is rejected; a +new attempt clears its list. + +## 8. Harness shell and handles + +`Harness.open` (built-in registries, `kinds` and `replace` options, kinds-set check, `inspect`, +`drive`, `close`, `shutdown`), `ConversationHandle` (`commit`, `value` / `list`, `config` / +`settings`, `fork` with `abort`, `abort`, `hooks.on` scoped with `subtree`), `abortTask`, +`conversations` with `parent` / independent filtering. No agent behaviour yet. + +Tests: open on empty vs existing storage; an unknown kind rejects without scanning entries; +replace by name keeps `h.kinds.` consistent; `settings` round-trips; scoped hooks run after +harness-wide ones, innermost last. + +## 9. Generation kind + +Statuses pending → streaming → done / failed / retry_wait / deferred / aborted on a faux provider; +config capture (model, thinking, selected tools, profile, budget); `system_instructions` with +sections merged across handlers and the diff writing `system` entries; `before_request`, +`after_response`, `on_yield`; retry sleeps in execute; recover from frames; usage recorded per +attempt. + +Tests: the six system-entry traces from the guide (tool added, removed, host section changed, MCP +schema changed, plugin section on and off, `addTools`); fresh baseline after a head; a fork diffs +against its own config; a restart emits only the changed section; in-band `aborted` stop reason; +crash while streaming publishes the partial; retry budget exhausted → failed with no successor. + +## 10. Tools, post_tools, exchanges + +The tool kind with the sink (`ToolOutput`, `ToolOutputState`, limits enforced by the sink, `diag`, +`delegate`, `handoff`, `addTools`, `terminate`), `before_tool` (fail-closed) and `after_tool`, +replay policy on recover; post_tools with `after`, terminate / handoff / steer / next generation; +`accept` idle vs busy; `prompt` and `answerTo`. + +Tests: parallel tools completing in either order; sequential via `after`; error results for every +call on abort; `new_context` resets after the exchange, never inside it; `addTools` → the next +turn's `toolsAdded`; a throwing tool → error result, `terminate` still honoured; truncation diag +from the sink; `accept` with the same `requestId` twice → same entry, `answerTo` finds it after +further turns. + +## 11. Inbox + +`pi.inbox` as a conversation list, the modes, the three dequeue points, `cancelQueued`, abort +draining steer and followUp. + +Tests: the modes table; cancel vs land in both orders; input queued during a collapse lands. + +## 12. Collapse + +Manual, threshold and overflow; `before_collapse`; publish only if no newer head; the overflow +chain (generation settles → collapse → new generation carrying the attempt). + +Tests: a summary lands under a running generation and later entries stay in context; a stale +summary fails; overflow retries once and no live task ever waits on the collapse; threshold before +a turn; abort of a running collapse leaves appends flowing. + +## 13. Subagents + +The `subagent` tool (`run`, `spawn`, `send`, `status`, `wait`, `stop`), ownership links, foreground +reach through live owners, `run`'s recover driving the child again, `spawn` initialization of +config, `answerTo` on the child. + +Tests: restart in the middle of `run`; parent abort reaches a `run` child and spares a `spawn` +child; `stop`; nested children; the child's first turn writes its own baseline. + +## 14. Jobs and the budget + +`jobKind` on `ExecutionEnv.exec` with output into its scratch, `waitForTask` / `jobOutput`, `bash` +delegating first and waiting with the budget, `notify` and the `notice` entry, the `job` tool, +schedules. + +Tests: budget expiry settles the call with `delegated` and the job continues; the notice appears +on completion; a schedule loops and `abortTask` ends it; recover → `lost` or rerun by policy; a +user abort during the wait kills a non-backgrounded job. + +## 15. Previews + +`ctx.preview` as a Chord tracker per task; the generation applies stream events to a partial +message, the tool's preview is its sink state, the job's the same; `preview.init` on attach and +reopen; flush after each scratch commit. + +Tests: one token → one `a` op and nothing else; no `r` mid-stream; init after reopen yields a base +equal to the live preview; a sliding tool tail → `t` + `a`. + +## 16. Watch + +`ConversationView`, `ConversationEvent`, the exported kind-free `applyEvent`, `WatchHandle` +(capture on the line, bounded buffering, `resnapshot`, `unsubscribe`), the session watch with +`report` and `usage`, the usage ledger (`pi.usage` + totals). + +Tests: the fold is correct (view after N events equals a fresh capture, randomized); all events of +one commit delivered together; a thin-client reducer over a recorded stream with no kinds loaded; +lag → fault → resnapshot; `resnapshot` from inside the listener; usage totals equal the ledger +fold, failed and aborted attempts included. + +## 17. JSONL storage + +Append-only batches, replay on open, torn tail discarded, malformed line fails open, the kinds set +from replay, values as Chord deltas (full when first or small, ops otherwise), scratch sidecars +retired after the main-file settle. + +Tests: conformance against memory (one mutation stream, identical query results); linear file +growth under repeated sets of a large value; crash between the settle write and the sidecar unlink. + +## 18. SQLite storage + +Tables and indexes of §7.5, scratch rows deleted in the settle transaction, storage version and +`migrate`. + +Tests: conformance three ways; a cold reopen decodes only live rows (count them); a version +mismatch rejects. + +## 19. Race matrix and telemetry + +The §10.2 matrix in both orders with faux clocks, fake processes and storage barriers; spans per +task call and per commit. + +Tests: the matrix; span tree for one turn with a tool and a retry. + +## 20. Clients + +mini (`worker/run.ts`, `worker/lane-service.ts`, TUI `apply(view)`), the experimental agent's four +seam files (`session-worker.ts`, `agent-controller-provider.ts`, `models-provider.ts`, +`transcript-provider.ts`), real providers. This is the gate: system deltas on a live model, a +retry, a spawned subagent surviving a restart, speculative compaction under a running turn. + +Packages 1–16 are what the gate needs; 17–19 can land after it. + +## What comes from the lane harness, and how + +Clean room means no imports from `src/harness/runtime`, `session`, `agent-harness.ts` or the +`dom`/`pico`/`pico2` spikes. Where the old code has something worth keeping, it is **copied** into +`src/harness/pico/` and owned there; pico must build with the rest of `src/harness` deleted. + +Copy (under `packages/agent/src/harness/` unless noted): + +| what | from | into | package | +|---|---|---|---| +| `ExecutionEnv` / `FileSystem` / `Shell` types, the Node env, capture and spill | `types.ts`, `env/`, `tools/tool-context.ts` (03-execenv) | `pico/env/` | 10, 14 | +| shell output limits, `applyShellOutputUpdate`, truncation totals | `utils/` | `pico/env/output.ts` | 10 | +| the built-in tools (`read`, `write`, `edit`, `bash`, `image`) | `tools/*.ts` | `pico/tools/`, rewritten to the sink signature | 10, 14 | +| system prompt builder, skills, context files, templates | `system-prompt.ts`, `skills.ts`, `prompt-templates.ts`; the keyed-sections builder in `coding-agent` | the host's `system_instructions` handler, not the harness | 9, 20 | +| telemetry span helpers | `telemetry.ts` | `pico/telemetry.ts` | 19 | + +Depend on, as packages (they are not the old harness): + +- `@earendil-works/chord/delta` (15, 16, 17) +- `@earendil-works/pi-ai`: `faux` provider for tests, `utils/estimate` for thresholds, `SystemMessage` (4, 9, 12) + +Read before writing the equivalent, then close the file: + +- `runtime/drive/retry.ts`, `deferred.ts`, `response.ts`: retryable-error classification, overflow detection, deferred polling (9) +- `compaction/`, `runtime/drive/boundary.ts`: the exchange-cut rule and the summarizer prompt (12) +- `runtime/drive/recovery.ts`, `restore.ts`: the per-phase recovery case list (9, 10, 14) +- `runtime/drive/tool-placement.ts`: the result order projection must reproduce (4) +- `hooks.ts`, `docs/harness.md` §before_tool: the exact decision shape (10) +- `session/`, `test/harness/jsonl-*.test.ts`: torn-tail and malformed-line rules (17) +- `session/`, `test/harness/mutation-line.test.ts`: line discipline edge cases (3) +- `agent-harness.ts`: `LaneSnapshot` / `HarnessEventPayload`, for the `toLaneSnapshot(view)` shim only (20) + +Tests to port (`packages/agent/test/harness/`): the three conformance suites as the model for 17–18; +`compaction`, `branch`, `context`, `execution-*`, `values`, `mutation-line` as the parity source for +19; `system-prompt`, `prompt-templates` for the host handler in 9; `tools`, `truncate`, +`output-capture` for 10; pico v1's 27 ported scenarios as the shortest list of behaviours to +re-prove. + +Do not read: `runtime/lane.ts`, `reducer.ts`, `drive/reconcile.ts`, `structural.ts`, +`terminal.ts`, `checkpoint.ts`, and the `dom` spike. They are the operation state machines and the +reconciler this design replaces. diff --git a/packages/agent/docs/pico2.md b/packages/agent/docs/pico2.md index 7e2132dff..1e1188d5a 100644 --- a/packages/agent/docs/pico2.md +++ b/packages/agent/docs/pico2.md @@ -1,38 +1,46 @@ # pico v2 -A harness for running agents. This document is the design: read it and build it. +A harness for running agents. This document is the design under review, not an implementation +claim. Decisions still needed before implementation are listed in §25. Assumed knowledge: pi-ai (`Models`, `streamSimple`, tool definitions, thinking levels, deferred handles, `AssistantMessage`/`UserMessage`/`ToolResultMessage`). Everything else is defined here. -Code is TypeScript; a snippet is a shape unless it says otherwise. Normative words: MUST, MUST NOT, -SHOULD, MAY. +Interfaces and examples are proposed TypeScript shapes, not existing package exports or a claim of +implementation completeness. JsonValue/JsonObject mean strict JSON. AssistantMessage, UserMessage, +AssistantMessageFrame, DeferredHandle, Usage and Models reuse pi-ai; AgentMessage/ThinkingLevel and +Context reuse the existing agent harness. Host capabilities and explicitly open contracts are not +silently implemented by these sketches. Normative words: MUST, MUST NOT, SHOULD, MAY. History: `pico.md` (v1) modelled a session as one resident tree of nodes; `pico-v2-tree-draft.md` tried to page that tree; `pico-v2-folds-draft.md` replaced the tree with a transcript and used "folds" to bound the model's context. This version keeps v1's driver, kinds, scratch, line, hooks, -vars and watch, and replaces both the tree and the folds with three separately-lived things and an +typed values and watch, and replaces both the tree and the folds with three separately-lived things and an explicit context list. --- ## 0. In one page -A **session** is one **journal** (an append-only log of small records) and a set of -**conversations**. A conversation is three things: +A **session** stores conversations, entries, tasks, values and lists. Its committed writes share +one ordered sequence; the backend decides physical representation. JSONL keeps main history and +separate named working scopes (§4). A conversation is three things: 1. a **transcript**: an append-only list of immutable **entries** (a user message, an assistant message, a tool result, a summary): what happened; 2. **tasks**: small mutable records with a status and code (a generation in progress, a tool executing, a background job, a subagent, an approval waiting): what the driver runs; -3. **keyed values**: named settings and plugin state written as immutable entries at transcript - positions: what forks and rewinds read as of a position. +3. **typed values and lists**: settings and plugin state with immutable versions at journal + positions: what forks and rewinds read as of a committed boundary. + +Session-scoped values hold metadata such as entry labels and request identities. They do not rewind. +Conversation values and lists follow history, independently of transcript selection (§5). The model does not read the transcript. It reads the conversation's **context**: a short list of entry ids, edited by three ops (`append`, `replace`, `reset`), that always has the shape "one head entry (a summary or a handoff), then selected entries in transcript order". -A **kind** defines what an entry, a task or a value is and, for tasks, the code that runs, -recovers and aborts it. A **read model** (tables for SQLite; rebuilt in memory for JSONL) answers +A **kind** defines what an entry or task is and, for tasks, the code that runs, +recovers and aborts it. Values and lists use typed bound addresses, not kind registration. A **read model** (tables for SQLite; rebuilt in memory for JSONL) answers "what is live", "what is this task now", "the newest value of key K before position P" and pages the transcript, without replaying the journal. Memory holds each live conversation's context entries and its live tasks; everything else is read on demand. A **subagent** is a task that @@ -44,22 +52,28 @@ as they commit. ## 1. Definitions -- **seq**: the position of a record in the journal; a positive integer assigned at append. Every - entry, task and conversation is identified by the seq of the journal record that created - it. There are no other ids. A caller-supplied idempotency id is a keyed attribute. +- **seq**: the position of a record in the journal; a session-global positive safe integer minted + while building a command on the line (§7). Every entry, task and conversation is identified by + the seq of the journal record that created it. Storage preserves these seqs; it does not assign + or remap them. Internal identities need no UUIDs or separately allocated ids. An optional external + request identity maps to the accepted input's numeric id through a session value (§18.1). - **conversation**: an identity (its creation seq) with a mutable **conversation state** (§4.1), a - transcript, work, keyed values, and a context list. The **root** is the conversation with no parent. + transcript, work, typed values/lists, and a context list. The **root** is the conversation with no parent. - **entry**: an immutable transcript element (§2). Written once; never patched; never reordered. - **task**: a mutable element with a lifecycle (§3): a generation, a tool execution, a job, a subagent, a collapse, an approval. Patched by `set`. -- **keyed value**: an entry with a `key`; read as of a position (§5). +- **value**: a scalar at a typed address; session-scoped (latest only) or conversation-scoped + (versioned, with historical reads). A conversation **list** stores immutable sequenced elements (§5). +- **history boundary**: the last seq of a complete commit, never a position inside one. - **transcript order**: the order of a conversation's entries by id (seq). There is no other order. - **context**: the ordered list of entry ids the model reads (§6). Lives in the conversation state. - **live**: a task whose status role is not `terminal` (§3.2). A conversation is live if it has live tasks, a live owner task, or an attached watcher/handle. - **the line**: the session's single command queue (§7). -- **scratch**: durable working storage per live task, off-journal (§3.5). +- **scratch**: ordinary values/lists in a named working scope, durable until explicit retirement (§3.5). - **projection**: the messages the model receives, built from the context list (§6.4). -- **canEditContext** (§8.1): no generation in flight and no unresolved foreground calls; the precondition for context edits, forks, and `next()`. +- **canEditContext** (§8.1): no generation request in flight and no unresolved foreground calls. + A safe context edit is not a request to generate. +- **generation task**: the durable obligation to generate; never inferred from the context tail. --- @@ -71,14 +85,15 @@ as they commit. interface Entry { id: number; // seq of its journal record; also its transcript order conv: number; // owning conversation - kind: string; // "user" | "assistant" | "tool_result" | "summary" | "value" | plugin content kinds + kind: string; // "user" | "assistant" | "tool_result" | "summary" | plugin content kinds by?: number; // the task that wrote it, if any callIndex?: number; // tool results: index of the call within the assistant message they answer through?: number; // summaries: the last entry the summary replaced (§6.2) - key?: string; // keyed values (§5) + key?: string; // optional indexed content key; not a typed value address meta: JsonObject; // small resident fields: timestamp, preview, sizes + payload?: JsonValue; + commitEnd: number; // containing MAIN commit; supports entry-based historical selection } -// the payload (the message, the summary text, the value) is in the journal record at `id` ``` Entries are immutable and the transcript is append-only: no journal record targets an existing @@ -92,7 +107,7 @@ defineContent({ kind, meta: M, payload: P, keyed?: true, context: "select" | "no { project(entry, ctx): Promise }) ``` - `context`: whether inserting an entry of this kind also appends its id to the conversation's - context list (§6.1). Core: `user`, `assistant`, `tool_result` → `select`; `summary`, `value`, + context list (§6.1). Core: `user`, `assistant`, `tool_result` → `select`; `summary`, `custom` → `none` (a summary enters the context only through the `ctx_replace` issued by the command that creates it; a reset's bootstrap enters through `ctx_reset`). There is no per-entry override. - `project`: what a selected entry contributes to the model's messages. May read the payload @@ -100,91 +115,759 @@ defineContent({ kind, meta: M, payload: P, keyed?: true, context: "select" | "no --- -## 3. Tasks +## 3. Tasks, scheduling and cancellation -### 3.1 Task +### 3.1 Task record ```ts -interface Task { - id: number; // seq of its ins_task - conv: number; // owning conversation - kind: string; // "generation" | "tool" | "job" | "subagent" | "collapse" | "approval" | plugin work kinds - at: number; // transcript position (entry id) it started from - for?: number; // the task that spawned it (a tool → its generation) - status: string; - state: JsonObject; // small resident fields: attempt, model, call args, pid, produced, calls, … - owns?: number; // subagent: the conversation it owns +type TaskRole = "start" | "inflight" | "waiting" | "terminal"; +interface Task { + id: number; // creation-record seq + conv: number; + kind: string; + at?: number; // originating entry, including an inherited entry + for?: number; // spawning task; provenance, not an execution dependency + after: number[]; // all must be terminal before normal effect starts + status: S; + state: D; // schema-validated JSON; no promises, closures or mutable aliases + owns?: number; // child conversation, reciprocal with conversation.owner + abort?: true; // internal durable cancellation target; not a public task-cancel API } ``` -### 3.2 Statuses and roles +`after` expresses a durable wait. It means **settled**, not succeeded. There is no automatic +failure propagation along edges, no per-edge cancellation policy, and no kind-level concurrency +flag. Ordinary errors are outcomes. Concrete coordinators interpret the outcomes they need. -| role | meaning | driver action when the task is not owned | +```text +parallel tools: T1, T2, T3; P.after = [T1,T2,T3] +sequential tools: T1; T2.after = [T1]; T3.after = [T2]; P.after = [T1,T2,T3] +collapse: G.after = [C] +foreground child: tool.finishing.after = [subagent] +``` + +Dependencies must exist, remain in the same ownership tree, and form an acyclic graph. Check +creation and dependency edits against earlier records in the command. `for` is not a dependency +edge: G may spawn C and then wait for C without creating a cycle. Independent forks never become +implicitly runnable because another scope refers to them. + +### 3.2 Roles and lifecycle + +| Role | Meaning | Scheduler action when unowned | |---|---|---| -| `start` | not begun, or must begin again | run `effect` | -| `inflight` | someone was acting; unowned means interrupted | run `recover` | -| `waiting` | waiting on the outside world (a human, a process, a child conversation, time) | nothing (one `recover` per process, so a kind can re-arm) | -| `terminal` | finished | nothing | +| `start` | Work or a local continuation is ready once dependencies/time permit | `effect` | +| `inflight` | External work was begun; an unowned task may have been interrupted | `recover` | +| `waiting` | An external observer or owned child will supply the next transition | Re-arm once after open | +| `terminal` | Immutable outcome | None | -`transitions` lists the allowed next statuses per non-terminal status. A `start` status MAY carry -`state.notBefore`; the driver arms a timer. +A kind declares its statuses and allowed transitions. Status roles are interpreted by the +harness, not by storage. Every nonterminal status must have a recovery/cancellation story. -### 3.3 Task kinds - -```ts -defineTask({ kind, state: S, statuses: St, roles, transitions, initial, - concurrency: "parallel" | "exclusive", ownsConversation?: true }, - { effect, recover, abort, next? }) +```text +external work: commit inflight intent → perform effect → commit outcome +local work: read/prepare → commit outcome +park: commit new status/dependencies → RETURN +settle: output + usage + terminal status + required successors, ONE COMMAND ``` -- `effect(task, ctx)`: MUST write an inflight status before any external action; MUST settle with - one command carrying everything that belongs together (produced entries, spawned tasks, context - appends). A task's result is an entry (a result message, a summary, a job's tail as a `value`). Only the external call is caught; storage/invariant failures propagate. -- `recover(task, ctx)`: the task is inflight and unowned. Idempotent; decides from scratch and the - kind's replay rule. -- `abort(task, ctx)`: MUST reach a terminal status and release external resources. An aborted - generation with calls MUST write error `tool_result` entries for them (§6.3), so an exchange is - always complete. -- `next?(conv, ctx)`: for kinds with `ownsConversation` (§8.2). -- `concurrency` per kind per conversation. Core: generation, collapse exclusive; tool parallel - (`toolExecution: "sequential"` makes it exclusive); job, subagent, approval parallel. -Core statuses (role in parentheses): -`generation`: pending(start) streaming(inflight) retry_wait(start) deferred(start) polling(inflight) done aborted failed; -`tool`: planned(start) truncated(start) running(inflight) waiting(waiting) done aborted; -`collapse`: pending(start) summarizing(inflight) retry_wait(start) done failed; -`job`: planned(start) spawning(inflight) running(waiting) exited killed lost; -`subagent`: planned(start) running(waiting) done aborted failed; -`approval`: pending(start) waiting(waiting) granted denied cancelled. +A task MUST NOT await another task's lifetime inside `effect`, `recover` or `abort`. It either +settles and leaves successors, or records a dependency/continuation and returns. A control task +such as `post_tools` needs no fake inflight write when its entire action is a command. -### 3.4 Context handed to kinds +### 3.3 Kind and execution interfaces + +These are proposed interface shapes, not existing exported APIs. Schemas validate strict JSON; +concrete status/state fields are specified in §3.9–§3.15. ```ts -interface Ctx { - signal: AbortSignal; - conv: ConversationView; // §4.2, all reads async - set(id, patch: Partial & { status?: St }): Promise; - settle(id, patch: Partial & { status: St }): Promise; - commit(plan: (view) => Op[] | undefined): Promise; // several ops, one command - ops: { entry(kind, meta, payload?, opts?: { by?, callIndex?, through?, key? }): Op; - task(kind, state, opts?: { for?, at? }): Op; value(key, value): Op; - set(id, patch): Op; settle(id, patch): Op; - ctxAppend(ids: Ref[]): Op; ctxReplace(through: number, withRef: Ref, expectPrefixVersion: number): Op; ctxReset(ids: Ref[]): Op; - ref(opIndex: number): Ref }; // refer to an entry created earlier in the same command - scratch(id): Scratch; payload(id): Promise; - sleep(ms): Promise; emit(delta): Promise; - hooks; vars; models; tools; resources; retry; collapse; +interface JsonSchema { + parse(value: unknown): T; +} +interface TaskDefinition { + kind: string; + state: JsonSchema; + initial: S; + roles: Record; + transitions: Record; + effect(task: Readonly>, ctx: TaskContext): Promise; + recover(task: Readonly>, ctx: TaskContext): Promise; + abort(task: Readonly>, ctx: TaskContext): Promise; +} +interface TaskPatch { + status?: S; + state?: Partial; + after?: number[]; +} +interface TaskContext { + readonly signal: AbortSignal; + readonly conv: ConversationView; + readonly services: TaskServices; + readonly working: WorkingScope; // ordinary values/lists, §5.6 + set(patch: TaskPatch): Promise; // this task only + settle(status: S, state?: Partial): Promise; + commit(build: (tx: Command) => void | Promise): Promise; + commit(scope: WorkingScope, build: (tx: WorkingCommand) => void | Promise): Promise; + getValue(address: Value): Promise; + readList(address: ValueList, options: ListReadOptions): Promise[]>; + payload(entryId: number): Promise; + emit(kind: string, payload: JsonValue): Promise; + sleep(ms: number): Promise; // this effect's timer, interrupted by signal } ``` -Writes happen only through `ctx`, and every write is a command on the line (§7). Ids are seqs; a -plan refers to records it creates in the same command by op index. -### 3.5 Scratch +`TaskServices` is the typed host service bundle (§12), not another durable object. Provider/tool +exceptions are classified by their kinds. Storage and invariant failures escape and fault the +session; a broad catch around effect plus settlement must not convert them into tool errors. +An unexpected kind exception is reported and faults the session rather than endlessly relaunching +unchanged start-role work. Faulted handles cannot continue cancellation writes. -Per live task: an append-only log plus a small key/value map, on disk beside the journal -(JSONL: one file per task; SQLite: a table), never in the journal, never replayed. The kind -declares `reduce(entries) → value` (a partial assistant message from frames; a tool's output so -far; a job's tail). Read by `recover`, `abort` and watch. Retired by settlement; orphans retired on -open. Appends are announced to watchers as ephemeral deltas (§9). +### 3.4 Foreground, ownership and busy state + +```text +local foreground: + generation, or task whose for names a generation + // tools, post_tools, and automatic collapse are foreground + +required foreground work: + local foreground plus its unfinished after dependencies + plus foreground work in conversations owned by those required tasks + +background: + other work; e.g. a detached job/subagent, standalone approval, manual collapse + +busy(conv): + local foreground remains OR cancellation targets remain in its required ownership subtree +``` + +`post_tools` is foreground even after every tool has settled. This prevents acceptance between +tool completion and continuation from accidentally creating a second generation. Background +work does not request generation. Automatic collapse uses `for: G`; a standalone manual collapse +does not. A waiting foreground tool keeps its required subagent/approval inside cancellation. + +```text +conversation.parent = provenance and historical inheritance +conversation.owner = subagent task that owns this conversation, if any +scope(conv) = conv + descendants reached through task.owns, not parent links +``` + +Root and every ownerless fork are independent drive scopes. A foreground wait on a subagent is +represented by the tool's dependency, not inferred from its parent's transcript or tool name. + +### 3.5 Scratch and publication + +Scratch is a use of ordinary values/lists in a named working scope (§5.6), not a storage subsystem. +Kinds reduce bounded list pages into partial output. Task outcomes reference immutable entries; +control tasks may have no output entry. A task uses `workingScope(String(task.id))` by convention. Keys distinguish attempt/poll identities; +recovery reads only the current request's keys. Superseded attempt data may remain unreachable +until retirement, but cannot be mistaken for the active attempt's frames/candidate. + +```text +progress: working-scope-only commit → publish preview +settlement: stop producer/sink admission → drain admitted progress writes + main commit: output + terminal task + retireScope(working) +cleanup: backend may remove physical working files AFTER that commit +``` + +Retirement is logically atomic with settlement. Failed unlink is harmless: committed retirement +makes the old contents invisible. Storage never infers retirement from owner absence/status. Close +and fault retire nothing. Invocation fencing rejects late writes; generic scopes allow reuse. + +### 3.6 Scheduler and process-local ownership + +One scheduler per session services the union of active drive scopes. Multiple callers may overlap; +they never install competing executors. Removing a caller does not cancel durable work. + +```ts +interface OwnedRun { + controller: AbortController; + promise: Promise; // current effect/recovery and any joined cancellation cleanup +} +const running = new Map(); +const recovered = new Set(); // inherited waiting tasks already re-armed in this process +const openedAt = committedHead; +``` + +These structures are process-local. The map is the owned set; no duplicate owned bitmap, durable +lease, or promise registry is needed. Each retry/phase gets a fresh controller. The scheduler owns +any promise replacement/cleanup; effect completion cannot delete a newer ownership claim. + +```text +pass: + scope = union of active drive scopes + for marked tasks in scope: + signal EVERY owned affected execution before waiting on any one + arrange exclusive cancellation reconciliation (§3.7) + for unmarked, unowned live tasks in scope: + start: + if all after tasks terminal, notBefore elapsed, required poll permit present: + claim synchronously; launch effect; do not await its whole lifetime in this pass + inflight: + claim synchronously; launch recover + waiting: + if task.id <= openedAt and not recovered: + mark recovered before launch; claim; launch recover to re-arm and return + resolve drive observations from committed state + wait for commit | effect completion | timer | external observer notification +``` + +External observers for jobs/approvals are disposable subscriptions, not promises held by a task +waiting for another task. Each observer writes through the line and checks the current task's +status/cancellation before publishing. The host retires observers when their task ends/closes. + +The scheduler reads task metadata, not transcript payloads or sibling tool outcomes. Terminal +dependency states use indexed point reads and may be cached while referenced by live tasks; +never repeatedly replay history. No global reverse-dependency index is required initially. +No conversation `next()`, tail inference, or open-time successor repair exists. + +### 3.7 Conversation cancellation + +Public cancellation is `conv.abort()`. **There is no public `abortTask(id)`.** A task's internal +`abort` bit records membership in a conversation cancellation; it is not a second user-facing +cancellation mechanism. `cancelQueued(id)` only withdraws unconsumed input. + +```text +abort(conv), ONE COMMAND: + if this conversation already has outstanding cancellation targets: join; do not drain again + select current foreground + required dependency/owned-child closure + mark every selected nonterminal task abort = true + drain currently queued steer/followUp in affected foreground conversations + preserve write/nextRun; return drained input identities/content + +normal admission/settlement: + marked work cannot start an external action or create normal successors + input arriving during cancellation queues; do not consume it during cleanup + new foreground tasks cannot bypass outstanding cancellation in that conversation +``` + +Selection is durable in the task marks. Do not recompute a smaller target set when an earlier +parent becomes terminal. Detached background work is not selected by ordinary foreground abort. +Deletion/explicit whole-conversation shutdown selects all live tasks in the ownership subtree; +this is still a conversation operation, not arbitrary graph-node cancellation. + +```text +reconcile a marked task: + retain ONE ownership claim through the entire sequence + signal its controller if an effect/recovery is owned here + await that task's own execution returning, whether fulfilled or rejected + report rejection; storage/invariant faults still prevent further writes + re-read task; if already terminal, do nothing + otherwise run kind.abort under the SAME exclusive claim + release claim only when that invocation finishes +``` + +That await is allowed: it joins this process's current execution of the SAME task. It prevents +its effect and abort handler from writing concurrently. It is not a wait for dependency tasks. +An uncooperative external effect can delay cancellation; no timeout may silently permit a second +writer. Driver passes can still signal/cancel other independent tasks. + +Never-started sequential tools can abort immediately, regardless of their `after` dependencies. +Each writes its required error result. `post_tools.abort` creates no generation and performs no +unsafe context edit; queued writes may remain queued. For a marked conversation owner, defer abort invocation until marked owned descendants are terminal +(§3.13); no effect promise waits for that condition. Normal child completion checks the owner's +marker before settling it, so cancellation cannot become success. + +```text +abort before P settlement: P is marked → no G2 +P settlement before abort: P + G2 commit together → abort selects G2 +crash after marking: open reports marked tasks → next covered drive reconciles them +crash after effect returned: marked inflight task remains → abort reconciliation resumes +``` + +The host is the sole live writer. Another process may reopen after handoff/crash; concurrent +independent writers are not supported. A remote client sends cancellation through the owning host. + +### 3.8 Open, drive, suspension and close + +```text +open: + hydrate storage/residency; validate kinds/statuses/dependencies + start nothing; write no repair generation; expose inspect() +inspect(): session-wide { start, inflight, waiting }, including cancellation marks +conv.drive(): run only its ownership tree +harness.drive(): run every independent tree in the session +``` + +A task in `start` can still be blocked by a dependency, retry time or deferred-poll permission. +`inspect()` reports durable roles, not a promise that every start-role task is runnable now. + +```text +conv.drive → idle: no foreground or outstanding cancellation in its ownership scope +harness.drive → idle: no live tasks anywhere +suspended: work remains, but progress requires an outside event/permission +closed: harness closed before requested condition +``` + +While a conversation drive is active it also services background work in that scope, but it does +not keep waiting solely for detached background tasks after foreground quiescence. A host wanting +to supervise all background work keeps a session drive active. Inherited waiting-task recovery is +once per session process, not once per caller or scope. + +`close()` seals commands and scratch admission, signals owned executions, detaches observers, +and drains admitted persistence. It writes no cancellation marks and no task settlements. Effects +must release resources/cooperate with close; the current cooperative-wait policy and a possible +bounded-close alternative remain explicit in §25. A caller cancelling its wait is not `abort()`. + +### 3.9 Generation + +**Responsibility:** one provider generation, including its retries/deferred polls. It publishes a +response and either tool work plus a join, or the normal final-answer boundary. It never awaits tools. + +```ts +type GenerationStatus = "pending" | "streaming" | "retry_wait" | "deferred" | "polling" + | "done" | "failed" | "aborted"; +interface GenerationState { + settings: GenerationSettings; // captured when this generation is created + attempt: number; // starts at 1 + notBefore?: number; + requestAsOf?: number; // committed context boundary used by current request + collapse?: { id: number; reason: "threshold" | "overflow" | "manual" }; + overflowRetried?: true; + deferred?: DeferredHandle; + poll?: number; + produced?: number; // immutable assistant entry + calls?: number[]; + postTools?: number; + error?: string; +} +``` + +Roles: pending/retry_wait/deferred = start; streaming/polling = inflight; others terminal. +Transitions: pending/retry_wait → streaming or failed/aborted; streaming → retry_wait/deferred or +terminal; deferred → polling or aborted; polling → deferred/retry_wait or terminal. Adding a +collapse dependency can preserve the current start status. Terminal states accept no patches. + +```text +creation: idle acceptance, a final-answer continuation, or post_tools settlement + capture model/thinking/tools/tool order/options/retry policy from current settings + task pending, attempt = 1 + +pending/retry_wait effect: + if captured collapse dependency exists: + it is terminal (scheduler readiness) + overflow collapse unsuccessful → fail this generation; no replacement + threshold/manual failure → proceed using unchanged context + else if automatic compaction is indicated: + decide before_collapse outside line + revalidate context and cancellation inside line + if approved: + C = create collapse { for: G, captured prefix/model } + set G { after: [C], collapse: { id:C, reason:threshold } } + RETURN; no promise waits for C + capture conversation's MAIN commitEnd as context boundary; commit streaming intent + build projection + request; run before_request outside line + re-check cancellation/close before provider call + stream provider; scratch receives frames + classify outcome below +``` + +A live manual collapse is attached as a dependency before admitting a new request, rather than +creating a competing automatic collapse. A generation always exists while automatic compaction +runs. There is no state where consumed input is represented only by a background collapse. + +```text +provider outcome, in precedence order: + close → leave task recoverable; no terminal write + marked cancellation → aborted partial + matching error results; no successors + deferred → reported usage + set deferred(handle, poll, notBefore) + overflow, not already retried: + decide collapse; commit usage + existing G.retry_wait + C + G.after=[C] + declined/impossible → usage + failed G + other error → reported usage + bounded retry_wait, else failed G + successful response with calls → settleWithCalls + successful response without calls → settleFinal +``` + +```text +settleWithCalls, ONE COMMAND: + A = assistant entry with call list + for each call in source order: + Ti = tool { for:G, at:A, callIndex, captured active-tool authorization } + parallel: Ti.after = [] + sequential: Ti.after = [previous tool] except first + P = post_tools { for:G, at:A, after: all Ti } + usage(G, reportedUsage) + settle G { done, produced:A, calls:all Ti, postTools:P } +``` + +The response, all tools, and exactly one P commit together. Stop reasons other than toolUse with +calls create truncated tool tasks: they produce error results but do not execute external tools. +Projection later sorts results by callIndex, regardless of their transcript completion order. + +```text +settleFinal: + prepare on_yield outside line only if eligible queued input is absent + ONE COMMAND, checking latest cancellation and queue state: + A = assistant entry; usage; settle G + select safe writes and eligible steering/followUp (§18) + if continuing: + place selected items in admission order; dequeue + place hook continuation if applicable + create next generation with freshly captured settings + else: + place safe writes only + finishOwnedConversation(tx, conv, { outcome:done, result:A }) +``` + +Stale hook decisions are discarded/recomputed before committing. A new queued trigger takes +precedence over an old yield decision. `finishOwnedConversation` is ordinary settlement code +(§3.13), not a later scheduler pass. Terminal failure uses the same owner notification with a +failed outcome and no successor. Input-result attribution remains separately open (§18.2). + +**Deferred:** admitted only with `pollDeferred`. Commit polling intent and increment poll before +fetching; still-deferred/fetch-error outcomes return to deferred with backoff on the same task. +Abort attempts provider cancellation best-effort, then completes locally. No new generation is +created for a deferred poll or retry. Retry delays use the captured bounded exponential policy. + +**Recovery:** polling becomes deferred; streaming with committed frames publishes an interrupted +aborted partial and matching error results; streaming without frames retries within budget or +fails. This preserves the current draft's policy; old-lane interrupted-stream parity is pending +(§25). Recovery must never resurrect a marked generation or create a fresh retry budget. + +**Abort:** stop/drain frames, cancel deferred handle if present, write required partial/error +entries and known usage, settle aborted. A never-started generation has no fabricated assistant +response. No tools, join, yield continuation or provider retry follows cancellation. + +### 3.10 Tool execution + +**Responsibility:** one call. No sibling reads, last-sibling checks, queue draining, context reset, +or next-generation creation. The same handler serves parallel and sequential execution. + +```ts +type ToolStatus = "planned" | "truncated" | "running" | "waiting" | "finishing" | "done" | "aborted"; +interface ToolState { + callIndex: number; + callId: string; // provider call identity; task.id is harness invocation identity + name: string; + args: JsonObject; + allowed: boolean; // captured from generation's active-tool names + replay: "safe" | "never"; + approved?: true; + approval?: number; + delegated?: number; // a subagent/job whose outcome supplies this tool result + produced?: number; + terminate?: boolean; + handoff?: string; // requested reset; interpreted by post_tools only + error?: string; +} +``` + +Roles: planned/truncated/finishing = start; running = inflight; waiting = waiting; done/aborted = +terminal. Planned can become waiting/running/done/aborted; running can become finishing/done/aborted; +waiting can become planned/done/aborted; finishing becomes done/aborted. Safe recovery can return +running to planned while preserving effective arguments/memos. Tool and sink interfaces: §16. + +```text +planned effect: + unavailable/inactive tool → settle own error result + validate arguments against tool schema + before_tool outside line, unless already approved: + block → settle own error result + hold → create approval + set this tool waiting, ONE COMMAND; RETURN + commit running intent with effective args and replay declaration + execute with own signal, invocation identity, scratch/sink and application context + drain output; run after_tool outside line + ONE COMMAND: + re-check cancellation + append own result; reported usage; active-tools additions + settle this tool, retaining terminate/handoff control metadata + +truncated effect: + settle own "not executed" result; never invoke tool +``` + +A definition merely present in the registry is insufficient authorization: the generation must +have offered it. Arguments and replay declarations are captured before the external invocation. +An ordinary tool throw becomes `isError`; storage errors do not. Unknown addTools names remain +dropped under the existing draft policy. Final text is capped; truncation is explicit (§16). + +**Delegation without awaiting another task:** a harness-native tool can return a durable wait action. + +```text +ONE COMMAND: + S = create subagent/job with for = this tool + set tool { finishing, delegated:S, after:[S] } +RETURN from current execution + +S eventually terminal → scheduler can run this SAME tool's finishing phase +finishing effect: + load S's committed result + apply after_tool outside line + commit this tool's own result + terminal outcome +``` + +This reuses the tool task rather than inventing an operation wrapper. The extra phase is needed +when async after_tool processing must occur outside the child settlement command. No in-process +waitFor(S) exists, either initially or on recovery. Background delegation instead settles the tool +with the task id immediately; it has no dependency on the detached work. + +**Recovery:** replay only when both captured and current declarations permit safe replay and no +cancellation won. Preserve task id, effective arguments and memos. Otherwise settle an interrupted +error result from the last checkpoint, without applying uncertain addTools/terminate/handoff effects. +Waiting holds are restored as holds; finishing dependencies survive without re-arming a promise. + +**Abort:** after joining its own execution, drain scratch and settle an error tool result with +aborted metadata. Unstarted calls also get results, completing the assistant exchange. No dependency +must finish first. This handler does not inspect siblings or invoke post_tools itself. + +### 3.11 Post-tools + +**Responsibility:** join one generation's tool results and decide what follows. This is a real +exchange coordinator, not a conversation-level progression loop. + +```ts +type PostToolsStatus = "pending" | "done" | "aborted"; +interface PostToolsState { + generation: number; + produced?: number; // optional terminal result reference, never payload +} +``` + +Created by generation settlement, `for:G`, `after:all calls`. Pending = start; done/aborted = +terminal. It has no external effect and no inflight status. Its effect is one serialized command: + +```text +P.effect, ONE COMMAND: + require all after tasks terminal and P still pending + read completed call outcomes // ONLY this coordinator reads siblings + if marked cancellation: settle P aborted; RETURN + select safe writes, mode-selected steering, and any valid handoff + place selected content in admission order; dequeue consumed items + apply handoff reset after full exchange completion, if requested + if exchange was aborted or termination policy says stop: + do not consume generation-triggering input + settle P done + finishOwnedConversation(tx, conv, terminal outcome) + else: + G2 = create generation with current model/thinking/tools/options + settle P done +``` + +Selection precedes mutation: on a stopping path only safe writes are consumed. Follow-ups wait +for a normal final answer, not this tool boundary. A handoff is an explicit trigger, not tail +inference. Compaction belongs to the already-created G2; it can attach a collapse dependency. + +Current draft termination policy is any completed call requesting terminate, with all sibling +results retained. Its difference from the old lane's all-results policy remains listed in §25. +An ordinary error result is not cancellation. Conflicting handoffs require a defined selection +policy before implementation; do not silently pick completion order (§25). + +**Recovery:** there is no unknown external action. If the commit did not land, P remains pending; +if it landed, P is terminal and G2 already exists. **Abort:** settle P aborted, create nothing, +perform no context edit. Remaining writes stay queued; other marked tasks finish independently. + +```text +T2 settles → P blocked +T3 settles → P blocked +T1 settles → P eligible, already durable +P settles + G2 created → one commit +``` + +No tool knew which one finished last. One additional task/terminal record per exchange buys a +single owner for the exchange policy and removes orchestration from every tool implementation. + +### 3.12 Collapse + +**Responsibility:** summarize a captured prefix and publish a validated replacement. It never +creates a generation. A pending generation already represents automatic continuation. + +```ts +type CollapseStatus = "pending" | "summarizing" | "retry_wait" | "publishing" | "done" | "failed" | "aborted"; +interface CollapseState { + through: number; + prefixVersion: number; + contextAsOf: number; + settings: GenerationSettings; + instructions?: string; + attempt: number; + notBefore?: number; + produced?: number; + error?: string; +} +``` + +Pending/retry_wait/publishing = start; summarizing = inflight; others terminal. Pending/retry_wait +→ summarizing; summarizing → publishing/retry_wait/done/failed/aborted; publishing → done/failed/ +aborted. A publishing task can replace its after list to wait for a current exchange boundary. + +```text +creation: + capture complete-exchange cut, prefixVersion, context boundary and summary settings + one live collapse per conversation; reuse existing task rather than race duplicate insertion + +effect: + commit summarizing intent + project captured prefix, including prior summary; call summarizer + complete candidate → persist in task scratch + ONE COMMAND: + usage + outcome + cancelled/stale prefix → terminal outcome, no context publication + context editable → summary entry + ctx_replace + settle done + otherwise → set publishing with after = current request/exchange coordinator + RETURN + +publishing effect, ONE COMMAND: + re-check cancellation, prefixVersion and canEditContext + if another exchange is now open: update after to its generation/post_tools; RETURN + otherwise publish candidate + ctx_replace + settle done +``` + +The durable publishing phase is needed only when a speculative summary finishes during a busy +exchange. It waits through dependencies, never an effect-held promise. Reported provider usage +commits once with the attempt outcome, not again when publishing. Scratch remains until terminal +settlement. Prefix changes reject publication; appends beyond through survive unchanged. + +**Recovery:** reuse a complete persisted candidate when available; otherwise retry interrupted +summarization within budget or fail. Publishing reads its candidate and revalidates. **Abort:** +stop summarizer, drain scratch, record known uncommitted usage and settle aborted, without summary +publication. Automatic G waits on this same task and interprets its terminal outcome (§3.9). +Token-budget/split-turn parity remains an explicit compaction-policy decision (§25). + +### 3.13 Subagent + +**Responsibility:** own one child conversation and expose its terminal foreground result. + +```ts +type SubagentStatus = "planned" | "running" | "done" | "failed" | "aborted"; +interface SubagentState { + prompt: AgentInput; + context: "fresh" | "inherit"; + values: { + inherit: { namespace: string; key: string }[]; + set: { namespace: string; key: string; value: JsonValue }[]; + }; // serialize selections; resolve them in child creation command + produced?: number; // may reference result in owned conversation + error?: string; +} +``` + +Planned = start; running = waiting; others terminal. Planned → running/failed/aborted; running → +terminal. `owns` identifies the child. Scalar initialization is precisely defined in §8.3. + +```text +planned effect, ONE COMMAND: + create child { parent history, owner:S, inheritValues:false } + write selected initial values + overrides + append prompt + create child's initial generation + set S running { owns:child } +RETURN +``` + +No external action requires a fake inflight status here. The child's final-generation or stopping +post_tools settlement performs `finishOwnedConversation` in the SAME command: + +```text +finishOwnedConversation(tx, child, outcome): + if child.owner is absent: return + S = read owner; require S owns child and is still running + settle S with outcome/result entry reference + // A foreground launching tool already exists in finishing, after:[S]. + // It is now eligible; no lost event or new task is required. +``` + +**Recovery:** a planned owner creates child once; a running owner starts nothing and returns. +Its child tasks carry all durable continuation. No waitFor or child-completion subscription needs +rebuilding. Completed result payload is not copied merely to settle the owner. + +**Abort:** conversation cancellation already marked required child work in the same marking +command. Scheduler defers owner abort invocation until its marked owned descendants are terminal; +it does not hold an effect promise while they finish. Then settle S aborted. If a child is aborted +directly, its live owner is included for cancellation completion; a parent-side finishing tool +can subsequently report the child's aborted outcome when its drive scope runs. Background child +work is untouched by unrelated parent foreground cancellation. + +### 3.14 Job + +**Responsibility:** run/adopt a process with durable output, without keeping a foreground tool +alive unless that tool explicitly depends on it. + +```ts +type JobStatus = "planned" | "spawning" | "running" | "exited" | "killed" | "lost"; +interface JobState { + command: string; + logPath: string; + pid?: number; + processIdentity?: string; // host-specific adoption identity, not pid alone + notBefore?: number; + produced?: number; + exitCode?: number; +} +``` + +Planned = start; spawning = inflight; running = waiting; others terminal. Planned → spawning/killed; +spawning → running/lost/killed; running → exited/lost/killed. + +```text +effect: + commit spawning with durable log/adoption location + spawn/adopt through host process service + commit running with process identity + install exit/output observers; RETURN + +exit observer: + drain bounded output/checkpoints + commit immutable job-result entry + usage if reported + settle exited + a dependent tool.finishing becomes eligible through after +``` + +**Recovery:** spawning reconciles the uncertain launch through the host adoption identity; never +blindly spawn a second process. Running recovery, once after open, reattaches output/exit observers +if the same process is alive, otherwise records lost. A pid by itself is not proof of identity. +Exact durable adoption/exit-status capability is a host contract, pending finalization (§25). + +**Abort:** when included by required foreground cancellation or whole-conversation shutdown, +stop the process through its service, drain output, write result and settle killed. A detached job +survives ordinary parent foreground abort and harness close. Explicit background shutdown/deletion +is conversation-scoped. Domain-specific process control is not a generic task-cancel endpoint. + +A schedule uses planned/notBefore and, for repeated execution, a successor job created atomically +with the preceding outcome. The schedule definition must supply a finite/bounded recurrence policy; +there is no hidden recurring-work scheduler or task held across another job's lifetime. + +### 3.15 Approval + +**Responsibility:** hold a durable human decision. The waiting tool, not a promise, blocks its +exchange and sequential followers. + +```ts +type ApprovalStatus = "pending" | "waiting" | "granted" | "denied" | "cancelled"; +interface ApprovalState { + tool?: number; + prompt: string; + produced?: number; + decision?: JsonValue; +} +``` + +Pending = start; waiting = waiting; others terminal. Pending → waiting/cancelled; waiting → +granted/denied/cancelled. + +```text +before_tool hold, ONE COMMAND: + A = approval { for:tool, tool:tool.id, prompt } + set tool waiting { approval:A, after:[A] } + +approval.effect: + commit waiting; publish human prompt through observation; RETURN + +approve(A), ONE COMMAND: + verify waiting and not marked + record decision + settle A granted + set matching waiting tool planned { approved:true } + +deny(A), ONE COMMAND: + record decision + settle A denied + write matching tool's own blocked result + settle tool done +``` + +Decision commands identify the existing approval but are not cancellation APIs. If asynchronous +result hooks are required on denial, ready the existing tool's finishing phase with the decision +instead of invoking hooks on the line; that phase settles only that tool. This is the same durable +continuation pattern as subagent results. + +**Recovery:** re-announce/reattach the human decision surface once; do not grant or rerun the tool. +**Abort:** record cancelled approval; the independently marked foreground tool writes its aborted +result. Denial/grant versus cancellation is decided on the line. Late/repeated decisions return the +existing outcome or a rejection; they never restart terminal work. --- @@ -193,71 +876,188 @@ open. Appends are announced to watchers as ephemeral deltas (§9). ### 4.1 Journal records ``` -ins_conv { parent?: {conv, at, asOf}, config, context: number[] } // at: parent entry id; asOf: parent seq -set_conv { id, patch } // inbox, abort, config, label +ins_conv { parent?: {conv, at?, asOf, inheritValues}, owner?, context: number[] } +set_conv { id, patch } // inbox, label, queue modes del_conv { id } ins_entry { conv, kind, by?, callIndex?, through?, key?, meta, payload? } -ins_task { conv, kind, at, for?, state } +ins_task { conv, kind, at?, for?, after, state } set { id, patch } // tasks only settle { id, patch (status terminal) } // tasks only; a task's result is an entry ctx_append { conv, ids } ctx_replace { conv, through, with, expectPrefixVersion } ctx_reset { conv, ids } +value_set { conv?, workingScope?, namespace, key, value } +value_delete { conv?, workingScope?, namespace, key } // conversation deletion = tombstone +list_append { conv?, workingScope?, namespace, key, value, tag? } +list_delete { workingScope?, namespace, key } // session/working lists only +scope_retire { workingScope } // MAIN-scope write +// usage(task, conv, usage) appends to the main session list pi.usage; no special storage verb ``` -JSONL: one line per **commit** (every record of one command; atomic; a torn tail is truncated on -open). SQLite: one row per journal record, one transaction per commit. Payloads live only in the -journal, addressed by seq. Nothing is rewritten. +`parent.asOf` is a committed parent-history boundary; optional `at` records an entry-based selection. +Historical forks set `inheritValues: true`. Subagents set it to false and store selected initial +values as local writes (§8.3). Parent lineage and context inheritance do not imply value inheritance. +Value/list records are journal records, not transcript entries, and never implicitly enter context. + +The command supplies final numeric seqs (§7); storage never remaps them. All scopes share one +session-global sequence space. Each admitted batch is contiguous; retained history and individual +physical files may have gaps after working writes/retirement. Backend encoding is private. JSONL +uses one complete line per commit in exactly one file; SQLite uses one transaction. Main-scope +history is retained for historical reads; obsolete working data need not be retained. **Conversation state** (derived; in memory; in a table for SQLite): ```ts interface ConversationState { id: number; - parent?: { conv: number; at: number; asOf: number }; - config: { model, thinkingLevel, seedTools }; // sticky vars (§5) - inbox: InboxItem[]; // steer | followUp | nextRun, each with its message entry id - abort?: true; + commitEnd: number; // last MAIN write affecting this conversation + parent?: { conv: number; at?: number; asOf: number; inheritValues: boolean }; + // model, thinkingLevel and activeTools are conversation values (§5), not fields here + inbox: InboxItem[]; // write | steer | followUp | nextRun; each names stored content + owner?: number; // owning subagent task, absent on independent forks label?: string; context: number[]; // §6 prefixVersion: number; // bumped by ctx_replace / ctx_reset only - steeringMode?, followUpMode?; + steeringMode: "all" | "one-at-a-time"; + followUpMode: "all" | "one-at-a-time"; } ``` -### 4.2 Read model +### 4.2 Stored data and indexes -Maintained by the append's transaction (SQLite) or rebuilt by one header-only scan of the journal -on open (JSONL; cost proportional to the file, accepted). Tables and the queries they serve: +Storage exposes conversations, entries (including conv), tasks, values and lists. Scratch and usage +are uses of values/lists. It does not expose a parallel ReadIndex or a raw journal/payload API. +`Entry` includes its payload and the end seq of its creation commit, so entry-based forks need no +separate commit-boundary lookup. Conversation historical reads include their context list. +```text +current: conversations, tasks, session values/lists, working values/lists +history: entries, conversation versions/context edits, conversation value/list versions +indexes: by conversation/id; live tasks; owner/parent; value/list address + seq ``` -entries(id, conv, kind, key, by, through, meta) index (conv, id); (conv, kind, key, id) -tasks(id, conv, kind, at, for, status, role, state, owns, last_seq) index (role) where role <> 'terminal'; (conv, kind); (owns) -conversations(id, parent_conv, parent_at, parent_asof, state) index (parent_conv) -ConversationView (async, bounded): - live() → tasks where role <> 'terminal' (session-wide) - entry(id) / payload(id) / task(id) / conversation(id) - entries(conv, { before?: id, after?: id, limit }) → a transcript page in transcript order - lastEntry(conv) / lastEntryWhere(conv, pred, { limit }) - findValue(conv, key, { before: id, asOf?: seq }) → newest keyed value before a position (and seq) - find(conv, kind, key?, { limit }) - tasks(conv, { kind?, live?: true }) - conversations({ parent?: conv }) +Historical scalar/list lookup follows §5.3–§5.4. Index and payload layouts are backend choices; +these queries must use bounded indexed reads, not full-history scans. Main-history indexes may +reference retained records; never retain deleted working payloads just to preserve a dense array. +Shared SQLite containers prefix every key with session id. No closure table or journal prev pointers. + +### 4.3 Storage interface and backends + +Follow the existing lane Storage: one atomic commit, entity reads/scans, values/lists. No separate +scratch interface, raw-record reader, or public index layer. `Context` is the existing harness +invocation/cancellation context. Types below are proposed interfaces, not current package exports. + +```ts +interface CommitResult { firstSeq: number; lastSeq: number } +interface HistoricalRead { asOf: number } // complete MAIN commit boundary; 0 = empty +interface Scan { + after?: number; before?: number; // exclusive id/seq bounds + order?: "asc" | "desc"; limit: number; // default asc; 1..10_000 +} +interface ConversationScan extends Scan { parent?: number; owner?: number } +interface EntryScan extends Scan { conv?: number; kind?: string; key?: string; asOf?: number } +interface TaskScan extends Scan { conv?: number; kind?: string; live?: boolean } +interface ListReadOptions extends Scan { asOf?: number; stopAtTag?: string } +interface ListElement { seq: number; value: T; tag?: string } +interface StoredValue { value: T; seq: number } + +interface Storage { + readonly head: number; // highest committed GLOBAL seq, including working writes + commit(writes: Write[], context: Context): Promise; + getConversations(ids: number[], context: Context, at?: HistoricalRead): Promise>; + scanConversations(query: ConversationScan, context: Context): Promise; + getEntries(ids: number[], context: Context): Promise>; + scanEntries(query: EntryScan, context: Context): Promise; + getTasks(ids: number[], context: Context): Promise>; + scanTasks(query: TaskScan, context: Context): Promise; + getValue(address: Value, context: Context, at?: HistoricalRead): Promise | undefined>; + scanValues(prefix: Value, context: Context): Promise[]>; + readList(address: ValueList, options: ListReadOptions, context: Context): Promise[]>; + close(context: Context): Promise; +} ``` -The read model is the **backend's**: SQLite keeps these tables on disk; memory and JSONL keep them -in memory and rebuild them by the open scan. The harness asks it questions (`ReadIndex`) and never -depends on how it is kept. The journal is the authority; if they disagree the read model is rebuilt. -### 4.3 Backends +Storage addresses include their bound conversation or working-scope identity (§5.6). Historical +reads apply to main conversation data only; session/working values expose current state. Current +multi-query snapshots use the session line. Every single logical query, including ancestor walks, +observes one committed state. Reads return immutable snapshots; callers cannot mutate owned data. +Filters precede limits. `stopAtTag` includes its matching element and never searches beyond limit. -- **SQLite**: journal + read-model tables in one file, one transaction per commit (WAL, - `synchronous = NORMAL`). Open = three queries. For long-lived sessions. -- **JSONL**: the journal (one line per commit) plus scratch sidecars and the usage ledger, nothing - else on disk; no index files. In memory it is the memory backend: records and read model, rebuilt - by one scan on open (torn last line truncated). For portable, inspectable, medium sessions; its - memory grows with the journal, by decision. -- **Memory**: tests. -Conformance: read model equals replay at every commit; pages/find/findValue equal a scan; scratch -survival and retirement; torn-tail tolerance. +```text +commit: + validate one write scope: main OR one exact working id + require final supplied seqs == head+1, head+2, ...; reject unsafe integers + validate/prepare touched rows; persist one atomic batch + publish rows/indexes/head together; only then resolve + no callbacks, task handlers or plugin reducers inside storage + +scope retirement in a MAIN commit: + invalidate working data through the retirement seq atomically with sibling main writes + physical cleanup may follow; cleanup failure does not undo the committed outcome + +failure: + pre-admission rejection → no changes or consumed seqs + admitted persistence failure → fault; outcome may already be durable; reopen, never blind retry +close: + seal admission; drain admitted persistence; release resources; retire no scopes +``` + +**Memory:** maps for current entities/scoped values/lists, ordered references for historical data. +Prepare only touched rows; publish synchronously without awaits/callbacks. Retirement drops current +working maps and releases obsolete payloads. Main history remains available. Binary-search ordered +value/list references; ancestor traversal costs fork depth, not unrelated session history. + +**SQLite:** one transaction updates entities, historical versions, global head and any retirement. +Working value/list rows are ordinary scoped rows, not a journal of every old checkpoint. + +```text +BEGIN IMMEDIATE + validate supplied seqs against stored head + apply entity/value/list writes and required history/index changes + retirement: DELETE working scalar/list rows WHERE session_id=? AND scope_id=? + advance global head +COMMIT +``` + +Use `(session, conv, address, seq)` historical indexes, `(session, workingScope, address, seq)` +working-list indexes, entity primary keys, live-task and conversation-owner indexes. Read compound +queries in one synchronous SQL read transaction; no awaits while holding it. WAL with +`synchronous=NORMAL` is the baseline. Rebuildable indexes are private, not another API contract. + +**JSONL:** adopt the lane scoped-storage protocol described in +[`implementation-handoff.md`](mobile-handoff/01-harness/02-scopes/implementation-handoff.md). +That document is design evidence, not a claim that scoped storage is implemented in this tree. + +```text +mainPath main entity/history commits + scope_retire records +mainPath.scopes/scope-.scope one named working scope, value/list commits only + +commit: + one session queue and global counter for BOTH kinds of file + encode complete batch; writeAll to exactly ONE file, handling short writes + publish in-memory state only after complete newline reached OS + main retirement committed → serialize attempted unlink before later scope reuse + failed unlink → old data still logically retired; retry cleanup on reopen + +open: + validate main/sidecar identity and complete transactions + compute high-water mark from header + ALL complete main/sidecar records BEFORE cleanup + find latest main retirement seq per working id + retain only scoped writes newer than that retirement + replay retained transactions in global order; preserve boundaries; reject duplicate/overlapping seqs + discard only unterminated final transactions; malformed complete data fails open + remove files containing no writes newer than their retirement +``` + +One batch has consecutive seqs, but files/history have legal gaps. Never use array position as a +seq. No per-scope counter, range reservation, cross-file acknowledgement, owner-derived sweeping, +or scope-creation transaction. First scoped write creates the sidecar atomically with a validated +header; names are encoded, fixed-prefix components, never raw paths. Follow the linked handoff's +Unicode/length validation and `.scope` layout so repository listing cannot mistake them for sessions. + +Scopes are reusable. Retirement ends writes through its global seq, not every future use of the +name. A delayed unlink must never delete a later lifetime. Main compaction must retain retirement +boundaries while old physical files could otherwise become live. Repository deletion removes the +whole scope directory. Close/fault never retire. Main and sidecar complete writes provide process- +crash durability; power-loss durability/fsync is a separate explicit policy, not a cross-file protocol. ### 4.4 Memory @@ -272,28 +1072,245 @@ does not depend on how the backend indexes. After each commit the harness keeps some context list names and the tasks that are live or a newest generation, and drops the rest; a commit that puts an old entry on a list (a fork) loads it first. Objects are values (a write yields a new object). All reads are async; a hit is a microtask, a -miss goes to the read model or the journal. Residency is proportional to (live conversations × -context size) + live tasks, independent of the journal's length on every backend; the process's -total memory additionally includes the backend's read model where the backend keeps it in memory -(memory, JSONL) and nothing else with SQLite. +miss uses the ordinary Storage entity/value/list reads. Execution residency is proportional to (live conversations × +context size) + live tasks, plus the currently needed settings. Active plugin replicas additionally +hold their current state and tracking data; they do not retain every historical state revision. +The process's total memory also includes retained main history/indexes on memory/JSONL, live working +scope contents, bounded query/cache state on SQLite, and active dependency metadata. A bounded execution working set is not a claim of bounded +total process memory on memory/JSONL. --- -## 5. Keyed values (vars) and plugin state +## 5. Typed values, lists and plugin state -`declareVar(name, { rewind: boolean, inherit: boolean })`. -- `rewind: true`: written as a `value` entry (`key = name`, one bounded immutable value per write); - read by `findValue(conv, name, { before })`: one index read. A fork reads as of its `{at, asOf}`. - Examples: `plan.mode`; `activeTools` (written by the tool kind when a result adds tools). -- `rewind: false`: a field of `config`; latest wins; forks copy it. Examples: `model`, - `thinkingLevel`; `seedTools` seeds `activeTools`. -- `inherit`: default for seeding a child; the creating kind MAY override. +### 5.1 Bound addresses -Plugin state has two shapes and no third: keyed value entries (the default: a board is -`board.state` written whole on each change, found by `find(conv, kind, key)` without any lifecycle) -or a task for an actual lifecycle. A sequence (a list) is repeated entries under one key; -reading it as of a position is `find(conv, kind, key, { before, limit })`. Replicated plugin state -(§9.1) is a view over such entries. Scratch is for un-journaled live preview only. +Keep the old harness's typed bound-address approach: namespace and optional key are bound once; +subsequent operations take that address, not another string key. Scope and kind are part of its +TypeScript type. No registration, declaration merging, runtime token catalog, or per-write rewind +flag is required. `declareVar` is replaced by these constructors: + +```ts +const model = conversationValue("pi.model"); +const thinkingLevel = conversationValue("pi.thinking"); +const activeTools = conversationValue("pi.active-tools"); +const planMode = conversationValue("my-plugin.plan-mode"); +const moves = conversationList("my-game.moves"); + +const sessionName = sessionValue("pi.session.name"); +const entryLabel = (id: number) => sessionValue("pi.entry.label", String(id)); +const requestInput = (requestId: string) => sessionValue("pi.request-input", requestId); + +await conv.setValue(planMode, true); +const enabled = await conv.getValue(planMode) ?? false; +await conv.deleteValue(planMode); +await conv.appendList(moves, move); +await harness.setValue(entryLabel(42), "before migration"); +``` + +Namespace must be non-empty; namespace/key cannot contain NUL; the empty key is legal. Equal +(scope, kind, namespace, key) tuples name the same address within the receiver's scope. `pi` and +`pi.*` are reserved for core. Constructing the same address with incompatible value types is a +programming defect, not a reason for a runtime registry. Value types are determined by the address +(`NoInfer` on writes); wrong scope, wrong kind and wrong value type are compile-time errors. +Defaults belong in caller code, not token definitions. Stored payloads are JSON values. + +### 5.2 Scope and atomic writes + +- **Session values:** latest value only through the public API; setting replaces and deletion + removes. Session name, entry labels and request identities do not change on rewind or fork. + Their journal records remain durable; the current-value index needs only the latest set's seq. +- **Session lists:** current append/page/delete semantics; no rewind or fork inheritance. Usage + is a main session list. Working lists use the same operations with explicit lifetime (§5.6). +- **Conversation values:** each set appends an immutable version at its journal seq. Deletion + appends a tombstone that masks both earlier local and inherited versions. JSON `null` is a value, + not deletion. Historical reads select a complete commit boundary. +- **Conversation lists:** each append writes one immutable sequenced element. No per-element + update, insertion or deletion. Forks share prefixes; they do not truncate parent lists (§5.4). + +Model identity, thinking level and active tool names are ordinary rewindable conversation values, +not sticky configuration fields. Root creation writes their initial values in its creation command; +subagent initialization supplies its selected settings (§8.3). Generations capture the effective +settings they use; later setting changes do not rewrite an already-created generation. + +Convenience setters issue one command each. Read-modify-write uses the serialized command line: + +```ts +await conv.command(async (tx) => { + const todos = await tx.getValue(todoBoard) ?? []; + tx.setValue(todoBoard, [...todos, newTodo]); +}); +``` + +Reads use the pre-command committed view (§7); buffered writes apply in order at commit. A bound +conversation command supplies its conversation id for conversation-address operations. The +session-level command uses an explicit conversation id, allowing input creation, request mapping, +child creation and initial values to commit atomically. Value/list writes never select transcript +content or trigger generation by themselves. + +### 5.3 Historical scalar lookup + +Only local versions are stored. To read address K in conversation C through boundary U: + +1. Seek the newest local version of K with `seq <= U`. +2. A set returns its value; a tombstone returns absent and MUST NOT fall back to the parent. +3. If no local version exists, stop if C has no parent or `parent.inheritValues` is false. +4. Otherwise continue in the parent with `U = min(U, parent.asOf)`. + +For example, A writes false at 40 and true at 80; B forks A through commit 60. B reads false. +Parent writes after 60 never leak into B, including through further nested forks. A local deletion +in B masks false; a later local set recreates the value only in B's history. + +**Memory/JSONL:** map conversation and address to an ordered array of version references; binary +search the cutoff. References point to journal records rather than copying payloads. Parent lookup +is a map lookup. Worst case is one binary search per ancestor, not a scan of journal history. + +**SQLite:** use the `value_versions` primary key (§4.2), one indexed seek per ancestor: + +```sql +SELECT seq, deleted +FROM value_versions +WHERE conv = ? AND namespace = ? AND key = ? AND seq <= ? +ORDER BY seq DESC +LIMIT 1; +``` + +Fetch the winning version's payload by its indexed reference (or directly from that version row). Parent metadata is a point read and may be +cached for active conversations. No copied inventory, ancestor-closure table or recursive journal +scan is required. The baseline cost depends on fork depth; it is not constant-time. Query-plan +tests must show indexed searches without a temporary ordering b-tree. + +### 5.4 Historical lists and paging + +```ts +const page = await conv.readList(moves, { after: cursor, asOf, limit: 100 }); +// [{ seq, value }, ...], oldest first; continue with the last returned seq +``` + +`after` is exclusive; `asOf` is an inclusive committed boundary. Omit `asOf` for the latest boundary +at that read; pass a fixed boundary across pages for a stable historical read. Limits are positive, +bounded, and required in this API sketch. A cursor is a sequence filter for the same conversation +and address, not a replication revision or a transferable replica binding. + +Traverse the same parent chain, carrying the minimum cutoff at every boundary, and read ranges +oldest ancestor first. A historical fork sees its ancestor's visible prefix followed by its own +appends. Subagents stop value/list traversal at their initialization boundary (§8.3). Descendants' +local writes follow their inherited prefixes in journal order, so concatenating these ranges needs +no global sort or copying. Skip ranges outside the requested cursor/cutoff and stop when the page +limit is filled. + +Memory/JSONL binary-search each local element array. SQLite uses `list_elements` with equality on +conversation/address, `seq > after AND seq <= cutoff`, `ORDER BY seq ASC LIMIT remaining`, then +fetches only those element payloads. An ordered-index outer loop plus addressed payload lookups +must not become a journal scan or temporary sort. Baseline traversal has fork-depth overhead plus +indexed reads of the requested elements, not work proportional to unrelated history. + +### 5.5 Checkpoints and replicated state + +Plugins own their delta shapes, reducers, validation and checkpoint cadence. A checkpoint is an +ordinary rewindable value; it contains the state and the last included list-element seq: + +```ts +const checkpoint = conversationValue<{ through: number; state: CanvasState }>("my-canvas.checkpoint"); +const deltas = conversationList("my-canvas.deltas"); +``` + +Hydration reads the checkpoint visible at the target, then pages visible deltas after `through`. +Checkpoint and delta writes that belong together share a command. An active wrapper retains the +current state so ordinary updates do not replay history on every write. Reopen and rewind reconstruct +from the selected checkpoint and suffix, without inverse operations. Checkpoint cadence bounds the +replay suffix only when maintained across the inherited history, not merely counted anew per child. +Checkpoints do not authorize deleting older deltas while arbitrary rewind is promised. + +Replication is a layer over these values/lists, not another durable store (§9.1). A task is needed +only for an actual lifecycle; plugin state alone does not create one. + +### 5.6 Named working scopes + +History scope (session/conversation) and working lifetime are distinct concepts. Working addresses +are session-owned, durable until explicitly retired, and never inherit/rewind. Use a named group, +not a boolean retention hint, so related checkpoints/memos can commit and retire together. + +```ts +interface MainScope { readonly kind: "main" } +interface WorkingScope { readonly kind: "working"; readonly id: string } +type WriteScope = MainScope | WorkingScope; +declare function workingScope(id: string): WorkingScope; + +declare const valueType: unique symbol; +declare const addressScope: unique symbol; +declare const writeScope: unique symbol; +interface Value { + readonly kind: "value"; + readonly scope: "session" | "bound-conversation" | "working"; + readonly namespace: string; readonly key: string; + readonly conv?: number; readonly workingScope?: string; + readonly [valueType]?: (value: T) => T; // invariant value type + readonly [addressScope]?: () => Sc; // covariant scope for reads +} +interface ValueList extends Omit, "kind"> { + readonly kind: "list"; +} +// Write is the §4.1 tagged union, with final seq and this invariant phantom: +interface ScopedWrite { + readonly seq: number; + readonly [writeScope]?: (scope: Sc) => Sc; +} +type SessionValue = Value & { readonly scope: "session" }; +type SessionList = ValueList & { readonly scope: "session" }; +interface ScalarSelection { + readonly scope: "conversation"; readonly kind: "value"; + readonly namespace: string; readonly key: string; +} +interface ConversationValue extends ScalarSelection { + readonly [valueType]?: (value: T) => T; +} +interface ConversationList extends Omit, "kind"> { readonly kind: "list" } +interface InitialValueWrite { address: ScalarSelection; value: JsonValue } +declare function conversationValue(namespace: string, key?: string): ConversationValue; +declare function conversationList(namespace: string, key?: string): ConversationList; +declare function sessionValue(namespace: string, key?: string): SessionValue; +declare function sessionList(namespace: string, key?: string): SessionList; +declare function setValue(address: ConversationValue, value: NoInfer): InitialValueWrite; +declare function value(namespace: string, key?: string): SessionValue; +declare function value(namespace: string, key: string, scope: WorkingScope): Value; +declare function list(namespace: string, key?: string): SessionList; +declare function list(namespace: string, key: string, scope: WorkingScope): ValueList; +``` + +`sessionValue`/`sessionList` are explicitly named main-session constructors. Conversation tokens +bind a conversation at the harness boundary; the backend receives a numeric conv on the address. +Only main addresses may carry conv. Equal scope id + namespace/key + kind means equal address; +object identity does not matter. Retention information is stored, not reconstructed from tokens. + +```ts +const work = workingScope(String(taskId)); +const frames = list("pi.pending.frames", "", work); +const memo = value("pi.pending.memo", "step", work); + +await harness.command(work, tx => { + tx.appendList(frames, frame); + tx.setValue(memo, checkpoint); +}); + +// After sealing/draining that invocation's progress: +await harness.command(tx => { + const resultId = tx.entry(convId, "tool_result", meta, result); + tx.settle(taskId, { status: "done", state: { produced: resultId } }); + tx.retireScope(work); // MAIN write, not a direct working delete +}); +``` + +All writes in a command are main OR one working id. Type invariance rejects main+working mixes; +a runtime check rejects two different working ids (their static tag is identical). Main commands +may retire multiple working scopes. Reads may use either lifetime; only writes are restricted. +Never mix a direct working append/set/delete with an entity settlement, even on SQLite. + +Retirement is the sole storage authority: no task-name parsing, missing-owner heuristic, TTL or +implicit cleanup at close. Reusing a scope after retirement is legal. Harness invocation fencing +must prevent a late progress job from accidentally becoming that new lifetime. State coordinating +a task transition remains main-scoped; only independently committed working data goes in the scope. +A typed address alone does not make a cross-scope atomic update possible. --- @@ -307,8 +1324,8 @@ reading it as of a position is `find(conv, kind, key, { before, limit })`. Repli - `ctx_replace { through, with, expectPrefixVersion }`: remove the prefix up to and including entry `through`, put entry `with` (a summary) at the head. Rejected if `prefixVersion` moved. - `ctx_reset { ids }`: replace the whole list with a bootstrap: `[]` (a user's `/clear`: waits for - input), a `user` entry written by the model's `new_context` tool (the handoff: the tail is a user - message, so generation continues), or a `summary`-kind note (text without a trigger). + input), a `user` handoff requested by `new_context` (post_tools resets and creates its generation), + or a `summary`-kind note. Reset alone never requests generation. Both `replace` and `reset` bump `prefixVersion`; `append` does not. Consequently the list always has the shape **[head?, then selected entries in transcript order]**, and a summarization that started @@ -321,19 +1338,23 @@ The context as of an earlier commit is reconstructed by replaying `ctx_*` record ### 6.2 Summaries and handoffs A `summary` entry is appended by the `collapse` task and carries `through`. A handoff is a `user` -entry appended by the model's `new_context` tool and made the sole context by `ctx_reset`. Both +entry appended by post_tools for the model's `new_context` call and selected by `ctx_reset`. Both are ordinary transcript entries at their append position; their place *in the context* is the head. UIs draw a summary immediately after its `through` (so retained turns follow it) and a handoff at its own position (§9.2). ### 6.3 Boundaries -Context edits (`replace`, `reset`), forks and requests happen only when the conversation is -quiescent (§8.1), so a request never contains an assistant message whose tool results are still -pending. Settlements complete exchanges: an aborted or failed generation writes error results for -any calls it produced; a deferred generation has produced nothing yet. Timing alone does not make a -cut valid: `through` may not split an exchange (§6.1 validation), so a replacement never leaves a -tool result whose call was summarized away. A reset drops whole exchanges by construction. +```text +provider request: generation task exists; no unresolved foreground calls +replace/reset: canEditContext (§8.1), checked at committing boundary +historical fork: safe source boundary; subagent capture is separate (§8.3) +queued write: safe message boundary; never splice into an open tool exchange +``` + +Pending/retry-waiting generation may permit edits, never duplicate generation. Settlements complete +exchanges, including required error results. `through` must not split an exchange; reset drops +whole exchanges. Empty/note-only resets create no work. ### 6.4 Projection @@ -345,53 +1366,167 @@ Nothing else. Projections never write. ### 6.5 Collapse policy -`next()` starts a `collapse` task when the last generation's usage exceeds `threshold × contextWindow` -and more than `keepRecent` turns are on the list; also on provider overflow (the generation parks -in `retry_wait`; the collapse runs first; the generation retries once); also manually. The decision -(`before_collapse` hooks) is taken before the task is inserted; declined publishes nothing. The -task captures `through` (the cut, on a user boundary before the kept tail) and `prefixVersion`, -summarizes the projection of the prefix (which contains the previous summary, so summaries subsume), -and settles with `ins_entry summary {through} + ctx_replace`. Repeated compaction never accumulates -summaries. +```text +generation.pending, before provider intent: + if captured collapseId exists: observe that task; do not recreate it + else if usage > threshold × contextWindow and more than keepRecent turns remain: + decide before_collapse outside line; revalidate context inside line + if approved: commit collapse task + generation.after=[collapse] and captured reason + RETURN; generation stays pending and scheduler waits for its dependency + build request from resulting context + commit streaming intent → call provider + +provider overflow: + decide collapse outside line; revalidate before outcome commit + commit attempt usage + generation.retry_wait + collapse task/id together + existing generation waits for that collapse, then retries once + declined/impossible/failed overflow collapse → fail existing generation; no replacement + +manual collapse: create collapse task only; does not request generation +collapse success: commit summary entry + ctx_replace + usage + terminal collapse +``` + +The id is a specific task dependency, not a progression flag. Driver can run collapse while its +generation waits; reopen observes the same task. Retry timers never bypass unfinished dependencies. +Threshold decline proceeds with existing generation, not another conversation check. Threshold +collapse failure proceeds with unchanged context; overflow collapse failure ends the generation. +Summary captures `through` and `prefixVersion`, includes the previous summary, and subsumes it. ### 6.6 Context as of a commit -The context as of a committed boundary `asOf` is reconstructed from the conversation's `ctx_*` -records: find the newest `ctx_replace` or `ctx_reset` with `seq ≤ asOf`; the list is its head -(`with` / `ids`) followed by every `ctx_append` id greater than `through` (for a replace) with -`seq ≤ asOf`, minus nothing (later replaces are by definition after `asOf`). Replay is therefore -bounded by compaction cadence, not by the session, and needs no checkpoint. The read model indexes -`ctx_*` by `(conv, seq)`; only `fork` (§8.4) and replicated-state hydration (§9.1) use this. -`asOf` MUST be the seq of a commit's last record; a seq inside a multi-record commit is rejected, so -a state that was never externally visible cannot be selected. +```text +contextAsOf(conv, asOf): + require complete commit boundary, with conv already created + base = newest ctx_reset <= asOf, otherwise conv's ins_conv + context = copy(base.ids or base.context) // inherited initial context included + page context_ops in (base.seq, asOf], oldest first: + ctx_append: append ids + ctx_replace: replace prefix through named entry; retain EXISTING suffix + ctx_reset: replace entire list + return context +``` + +Replacement is not a full checkpoint: `[u1,a1,u2,a2] → [summary,u2,a2]` retains u2/a2 although their +appends preceded replacement. Indexed baseline replay excludes unrelated history but may span the +conversation since creation/reset. Compaction-bounded reconstruction remains an optimization to +review. Current context is a materialized point read; plugin hydration uses value/list indexes. --- ## 7. The line -One command queue per session. A command is `plan(view) → ops | undefined`: the plan reads (async -allowed), returns ops; the line loads the rows the ops target, validates (§7.2), appends the ops as -one commit, updates the read model, updates memory (§4.4), delivers to listeners, resolves the -caller. Nothing else writes. +One command queue per session. `session.command(build)` (exposed to task handlers as +`ctx.commit(build)`) runs the callback with a command-local buffer and the previous command's +committed view. Reads may be async; no other command runs until this one finishes. + +The buffer starts its sequence counter at the last committed seq + 1. Adding a record mints its +seq immediately and advances that counter. Creation methods return the seq as the new object's id; +later records use it directly. These methods only build the command: they perform no storage write +and publish no state. + +```ts +interface ValueReader { + getValue(address: Value): Promise; + readList(address: ValueList, options: ListReadOptions): Promise[]>; +} +interface ValueCommand extends ValueReader { + setValue(address: Value, value: NoInfer): number; + deleteValue(address: Value): void; + appendList(address: ValueList, value: NoInfer, tag?: string): number; + deleteList(address: ValueList): void; // reject historical conversation-list deletion +} +interface WorkingCommand extends ValueCommand {} +interface EntryOptions { by?: number; callIndex?: number; through?: number; key?: string } +interface TaskOptions { for?: number; at?: number; after?: number[] } +interface Command extends ValueCommand { + readonly view: ConversationView; // committed pre-command reader + conversation(options: Pick): number; + setConversation(id: number, patch: Partial): void; + deleteConversation(id: number): void; + entry(conv: number, kind: string, meta: JsonObject, payload?: JsonValue, opts?: EntryOptions): number; + task(conv: number, kind: string, state: JsonObject, opts?: TaskOptions): number; + set(id: number, patch: TaskPatch): void; + settle(id: number, patch: { status: string; state?: JsonObject }): void; + ctxReplace(conv: number, through: number, withId: number, expectPrefixVersion: number): void; + ctxReset(conv: number, ids: number[]): void; + retireScope(scope: WorkingScope): number; // always a MAIN write + usage(taskId: number, conv: number, usage: Usage): number; // session-list append convenience + bindValue(conv: number, address: ConversationValue): Value; + bindList(conv: number, address: ConversationList): ValueList; +} +``` + +Bound `conv.command` supplies conversation-token reads/writes without repeating conv. Main command +reads can inspect working values; its writes cannot directly change them. WorkingCommand exposes +only value/list writes, and runtime validation pins all writes to the supplied working id. + +```ts +await ctx.commit((tx) => { + const assistantId = tx.entry(convId, "assistant", meta, { message }, { by: generationId }); + const calls = message.content.filter(block => block.type === "toolCall"); + const toolIds: number[] = []; + for (const [callIndex, call] of calls.entries()) { + toolIds.push(tx.task(convId, "tool", { + status: "planned", callIndex, callId: call.id, name: call.name, + args: call.arguments, allowed: activeToolNames.includes(call.name), replay: "never", + }, { + for: generationId, at: assistantId, + after: sequential && toolIds.length ? [toolIds[toolIds.length - 1]!] : [], + })); + } + const postTools = tx.task(convId, "post_tools", { status: "pending", generation: generationId }, { + for: generationId, at: assistantId, after: toolIds, + }); + tx.usage(generationId, convId, message.usage); + tx.settle(generationId, { status: "done", state: { produced: assistantId, calls: toolIds, postTools } }); + tx.retireScope(workingScope(String(generationId))); +}); +``` + +`entry` for a selected kind also buffers its `ctx_append` immediately, with its own seq (§6.1). +Callers neither append it themselves nor calculate offsets around it. All methods use the same +counter, including patches and context operations; ids require no `{ref}` placeholders, reserved +ranges, deep traversal, or fix-up pass. + +Normal callback return commits the buffer; returning without adding records is a no-op. A throw +or validation rejection discards the whole buffer without advancing the committed sequence. Minted +ids MUST NOT be published or used for external effects before the command succeeds. The command +object is valid only during its callback; it cannot be retained to add records later. + +After the callback returns, the line loads the rows the records target, validates them in order +against committed state plus earlier records in the buffer (§7.2), and appends the whole buffer as +one atomic commit. Storage verifies the supplied seqs are contiguous and start immediately after +its global committed head; it never renumbers them. Storage publishes its read model as part of commit +(§4.3), not via a second harness write. The line then updates execution residency (§4.4), delivers +to listeners, and resolves `true` (`false` for a no-op). A storage failure faults +the session (§7.3); reopening derives the next seq from the last complete durable commit. Sequence +exhaustion rejects rather than allocating an unsafe integer. Nothing else writes the journal. ### 7.1 Guarantees -1. No two commands interleave; a plan sees the previous command's state. +1. No two commands interleave; a callback sees the previous command's committed state. 2. A caller resolves after delivery; a listener sees the state its records describe; a listener MUST NOT await a command on the same session inside its callback; a throwing listener is reported as `handler_error` and does not affect the commit. -3. A settlement retires its record's scratch. +3. Terminal settlement seals/drains working writes, then commits outcome + scope retirement together. + Physical unlink follows; no task-status-based orphan inference exists (§3.5). +4. A completed task and its successors commit together; no later driver/event pass must create them. ### 7.2 Validation (rejects the whole commit) - `ins_entry`: the conversation exists; `through` (if any) is an entry of it. - `ctx_replace` / `ctx_reset`: the resulting list (§6.1) has the head+ordered-suffix shape; `through` does not split an exchange; `expectPrefixVersion` matches. -- `ins_task`: the conversation exists; `at` is one of its entries; `for` is a task of it; - the status is `initial` or a `start` status. +- `ins_task`: conversation exists; optional at belongs to it or its inherited prefix; for is valid; + initial status is allowed; after targets exist in the same ownership tree and are acyclic. +- Every command writes main OR one working id. Main retirement is the sole cross-lifetime action; + it retires logical scope contents without requiring a second-file transaction. - `set`/`settle`: target exists; keys are the kind's; a status change is an allowed transition; `settle` reaches a terminal status; a terminal task accepts no patch. - `ctx_*`: every id exists in this conversation or its inherited prefix. -- `del_conv`: every live task in the conversation (transitively through owned conversations) was - aborted in the same command. +- Conversation value/list writes: the target conversation exists, including when created earlier + in the same command. Parent history boundaries are complete commits; a historical fork's + inheritance link and a subagent's independent initialization survive reopen unchanged. +- `del_conv`: ownership subtree has no live tasks; cancellation was driven before deletion. + Retain immutable history required by independent forks; reject new work in deleted conversations. ### 7.3 Faults A failure after a plan's decision faults the session: later commands reject; the process should @@ -399,100 +1534,111 @@ exit and reopen. Rejections (§19) are not faults. --- -## 8. Driver and conversations +## 8. Conversations and admission -### 8.1 Loop +### 8.1 Context editing is not scheduling +Scheduler, drive scopes and cancellation are defined only in §3.6–§3.8. + +```text +canAcceptNow = no foreground work and no outstanding cancellation (§3.4) +canEditContext = no streaming/polling request and no unresolved call in the current exchange ``` -tick: - for each live task not owned: - role start → if concurrency allows in its conversation: own, run effect - role inflight → own, run recover - role waiting → nothing (once per process: recover, to re-arm) - for each live conversation with an abort marker: abort its live tasks (transitively), clear the marker - for each live conversation with canEditContext (§8.1): kind(conv).next(conv) - if nothing owned: no live tasks → "idle"; only waiting → "suspended" - await any completion | timer | commit | signal (events only wake the loop; every pass re-reads state) + +Pending post_tools blocks acceptance, even after all calls settled. Pending/retrying/deferred +generation can permit safe edits but never another generation. A held call blocks replacement/reset. +Background jobs do not block edits. Context-only writes never become generation requests on open. + +### 8.2 Acceptance and the chain + +```text +idle accept, ONE MAIN COMMAND: + place eligible queued content + input in selected admission order; dequeue + capture settings; create pending generation + write optional external request mapping + +busy/cancelling accept, ONE MAIN COMMAND: + store unselected input + queue followUp + write optional external request mapping ``` -The live set is in memory; the tick scans nothing. One loop per session; `conv.drive()` is a join -handle that resolves when that conversation is quiescent (the driver emits `quiescent(conv)` after a -`next()` that changed nothing) or when the loop returns. -Three facts about a conversation, each derivable from the live set and the conversation state, -decide scheduling; nothing is derived from "the last transcript entry": -- `canEditContext`: no `generation` of this conversation is in flight (`streaming`, `polling`) and - every task spawned by the newest generation (`state.calls`) is terminal. Context edits - (`replace`, `reset`), forks and rewinds require it. A generation in `pending`, `retry_wait` or - `deferred` does not block it (the overflow path edits the context while its generation waits). -- `hasPendingGeneration`: a `generation` of this conversation is live in any status (`pending`, - `streaming`, `retry_wait`, `deferred`, `polling`). While true, `next()` never creates another - generation, whatever the context's tail says (a deferred generation with `pollDeferred: false` - simply waits; the drive returns `suspended`). -- `needsGeneration`: `canEditContext && !hasPendingGeneration` and the conversation has an - **unconsumed continuation**: the context's tail is a user message or a tool result of a - generation without open calls, **and the run has not ended**. The run has ended when the newest - generation answers the current tail and settled `failed` or `aborted`, or one of its tool results - carries `terminate` or `aborted` (settlements write those on the entries: an assistant entry - carries `calls`, a tool result carries `answers`, `terminate`, `aborted`). Nothing stores "run - ended": a new user entry moves the tail past that generation, which is what a later `accept`, - steer, follow-up or `on_yield` continuation does. So a final failure does not re-trigger with a - fresh retry budget, and `terminate` ends the run even though the tail is a tool result. -Background tasks (jobs, older approvals) and keyed-value writes affect none of the three. - -### 8.2 `next(conv)` - -Reference behaviour for `session`, inherited by `subagent` and `fork`; called when -`canEditContext`; reads the tail of the **context list**, never the transcript: -1. abort marker → return. -2. steers in the inbox → insert as user entries (mode `all` | `one-at-a-time`); return. -3. `needsGeneration` → collapse decision (§6.5); else insert a `generation` - `{ at: , model, thinking, tools }` with tools from - `findValue(conv, "activeTools")` or `seedTools`. -4. the last selected entry is an assistant message of a settled generation without calls → - follow-ups in the inbox → insert as user entries; else `on_yield` hooks: `continue(message)` - inserts a user entry; else quiescent: a `subagent` settles its task with the last assistant - message as `result`; the root does nothing. -5. after a reset the tail is whatever the bootstrap was: a user handoff continues; an empty list - waits for the next `accept`. -6. `nextRun` items are consumed by the next `accept`. +Acceptance starts no provider effect. The chain is generation → tools + post_tools → next generation +(§3.9–§3.11). Each tool settles only itself. The join exists before any call starts; the last tool +merely makes it eligible. Post_tools and final-answer generation own exchange/turn-end decisions. ### 8.3 Subagents and jobs -A subagent is a `subagent` task with `owns`: its effect creates the conversation (`ins_conv` with -`parent: {conv, at, asOf}`, config seeded per var flags, an empty or inherited context per -`context: "fresh" | "inherit"`) and inserts the prompt as the child's first user entry; it sits in -`running` while the child works; the child's `next()` settles it with `result`. A tool that starts -one either waits for the task to be terminal (foreground) and writes a result entry, or settles -at once with the task id (background). `abort` on a conversation reaches owned conversations. +A subagent is a `subagent` task with `owns`. Its creation is **initialization**, not historical +value inheritance: the creating tool chooses scalar addresses to copy and typed overrides. Context +inheritance is independent. Public API sketch: -A job: effect `set spawning` → spawn or adopt a process whose output goes to a durable log file -owned by the job (`state.logPath`) → `set running {pid}`; output is tailed into scratch; exit → -`settle exited {code}` with the tail as payload. `recover`: pid alive → adopt by `logPath`; else -`settle lost`. `abort`: kill. Scheduled: the effect ends with `set { status: planned, notBefore }`; -a kind MAY model each run as its own task `for: job`. +```ts +await conv.spawn({ + prompt: "inspect the parser", + context: "inherit", + values: { + inherit: [model, thinkingLevel], + set: [setValue(activeTools, ["read", "grep"])], + }, +}); +``` + +`setValue(address, value)` here constructs a typed initial write; it does not commit independently. +There is no universal `inherit` flag on a token: one tool may need a value another must exclude. +Common selections are ordinary arrays of typed addresses, not a registry. Selection is scalar-only; +canvas/game lists are not implicitly copied into subagents. + +Inside one serialized creation command, capture the parent boundary, read selected effective +values, create the child with `owner` and `parent: {conv, at?, asOf, inheritValues: false}`, write the selected +present values and overrides, insert the prompt AND its initial generation task, and mark the +owner task `running` with `owns`. +An absent selected value stays absent unless overridden. Copy values, not mutable object aliases. +Only the resulting local values are stored, not token objects or a durable selection policy. +The child cannot drive before this command commits. It has no parent value/list fallback: plan +mode and plugin state not selected are absent even when its context was inherited. Model, thinking +level and tool names become ordinary rewindable local values. A later historical fork of the child +inherits these values normally and stops at the child's independent initialization boundary. + +The child gets an empty or inherited context per `context: "fresh" | "inherit"`. Initialization may +happen while the parent runs; an inherited context must end at a complete exchange and does not edit +the parent's context. The exact context-capture policy for an executing parent remains under review +(§25). The child finishing command settles its owner (§3.13). A foreground tool is already parked +in finishing with after:[owner]; it becomes eligible without a waitFor promise. Background launch +settles the tool with the id immediately. Job behavior is defined in §3.14. ### 8.4 Forks and rewind -A fork is a conversation with `parent: {conv, at, asOf}` and `context: "inherit"`. `asOf` is the -commit boundary that **contains** entry `at` (the seq of the last record of the commit that -inserted it), so everything that commit did is visible as a unit: a tool settlement that appended -a result and wrote `activeTools` in one command is inherited whole. At creation the fork copies the -parent's context as of `asOf` (§6.6), restricted to ids `≤ at` and trimmed to the last complete -exchange, and reads keyed values with `{ asOf }` (position is implied: a value written by that -commit or earlier is visible, a later one is not). Its projection is its own context list (the -copied head + its own appends); nothing about the parent's later edits reaches it. `rewind(conv, at)` -= abort every task of `conv` with `at > at` (default; `keepRunning` opts out), then fork. -`/tree` = `conversations({ parent })`. +A historical fork records `parent: {conv, at?, asOf, inheritValues: true}`. It inherits **all** +conversation values and lists visible at that boundary; no per-token filter applies. Fork creation +copies neither value inventories nor list prefixes. Local writes override inherited scalars, and +local list appends follow inherited prefixes (§5.3–§5.4). Session values remain session-wide. + +A target may be `{ at: entryId }` or `{ asOf: commitEndSeq }`. Entry selection resolves to the commit +boundary that **contains** the entry, so a tool result and active-tools update in the same command +are inherited together. An explicit commit target supports moves or canvas changes that contain +no message entry. A seq inside a commit is rejected, not rounded to a partially visible state. + +With inherited context, copy the parent's context as of the selected boundary (§6.6); for an +entry selection, also restrict to the selected entry position as in the original fork contract. +Trim to the last complete exchange. The fork evolves independently of later parent edits. +Historical forks inherit context/values, not live tasks. Continuing requires explicit generation-task +creation or acceptance; driving copied user-ended context must not infer work. + +Unless keepRunning, `rewind` aborts/drains current conversation foreground work before creating +the fork. It does not selectively cancel graph nodes by at. keepRunning leaves the source chain +untouched. Entry/commit targets select inherited history, not cancellation membership. `/tree` +lists by parent and distinguishes independent forks by absent owner. ### 8.5 Open, resume, close, delete - `open`: conversation states, live tasks, the entries on live conversations' context lists, scratch of live tasks. Start nothing; report live tasks by role. -- `resume(opts)`: the loop. `pollDeferred`. -- `abort(conv)`: `set_conv {abort}`, drain steer/followUp (keep nextRun); the driver aborts live - tasks transitively and clears the marker. A no-op when nothing is live. -- `close()`: stop admitting commands; cancel in-process signals; live tasks stay live; jobs keep running. -- `delete(conv)`: report live tasks; on confirm, abort them, `del_conv`. +- `drive(opts)`: schedule the selected ownership scope; session drive covers all trees (§3.8). +- `abort(conv)`: mark foreground targets and drain eligible queues once (§3.7). +- `close()`: seal admission, signal executions, drain persistence; retire/settle nothing. +- `shutdown(conv)`: conversation cancellation including background work. +- `delete(conv)`: report work; on confirm, shutdown/drain ownership subtree, then del_conv. + Retain history needed by independent forks; physical repository deletion is separate. --- @@ -509,30 +1655,50 @@ step, so there is no gap. Base: tasks: [ live tasks ], scratch: { [taskId]: reduced } } ``` -Then, in order, each with its seq: every journal record concerning the conversation (`ins_entry` -with payload, `ins_task`, `set`, `settle`, `set_conv`, `ctx_*`; owned conversations if requested), +Then deliver relevant main writes in commit batches, retaining each write's seq (`ins_entry`, +`ins_task`, `set`, `settle`, `set_conv`, context/value/list edits and scope retirement; owned +conversations if requested). Usage is a main-list append. Working progress is delivered as a preview, and ephemeral deltas `{ frame | progress | kind-emitted, taskId }`, plus `quiescent` and `fault`. Older pages: `entries(conv, { before, limit })`, each with its own `asOf`; the client ignores stream records with `seq ≤ asOf` for entries it paged in. Remote clients RPC into the host, so the same one-step base applies to them. -**Replicated plugin state** (`ReplicatedState`) is a view over keyed entries with an adapter -contract: -- **Two sequences.** Hydration carries `{ journalThrough, stateSequence, ops }`: the journal - watermark (for catch-up and dedupe) and the state stream's own consecutive sequence (deltas since - the base), because Chord's replica requires consecutive updates and a journal filtered by key has - gaps. Each delta entry carries its `stateSequence`. -- **Read-modify-write on the line.** `state.update(fn)` is `line.commit(view => { const draft = - hydrate(view); fn(draft); return encode(draft) })`: the current value is read and the ops are - derived *inside* the planner, so two concurrent updates serialize instead of both computing from - the same base. Publication to the local replica happens on delivery; a failed commit publishes - nothing. Preparation done outside the line (a summarizer) is revalidated inside the planner. -- **Binding.** A replica is bound to one conversation and one history. Switching to a fork or a - rewound history is the same one-step operation as `watch` (§9.1): on the line, drop the old - binding, read the hydration, register for the new binding, with no commit in between. Deliveries - still arriving for the old binding are rejected by binding id. `stateSequence` is monotonic - within a binding across periodic bases; it restarts only with an explicit hydration. -- State values are byte-capped (one bounded immutable value per write); base cadence bounds hydration length. +**Replicated plugin state** is a wrapper over conversation checkpoints and delta lists (§5.5), +exposing Chord's `ReplicatedState` read surface. Persistence and conversation identity remain +Pi/application concerns, not Chord concepts. + +- **No durable replication revisions.** Storage records only journal seqs. At fresh hydration, + the adapter establishes stream revision 0, then numbers emitted update batches 1, 2, 3, …. + Journal updates at 104 and 117 can therefore be stream revisions 1 and 2. Chord's consecutive + revision check remains unchanged; no stored `stateSequence` or second allocation index exists. + The adapter retains the journal cutoff for catch-up independently of the live stream counter. +- **Commit before publication.** Read the current committed state and derive/validate changes on + the line; buffer delta/checkpoint writes; commit; then publish the corresponding state before + the next command runs. Concurrent updates cannot compute from the same stale base. A failed + command publishes nothing and its private draft must not contaminate a subsequent update. + Retain the current committed state while active; hydration is not repeated per write. +- **Hydration and binding.** Read the visible checkpoint and delta suffix, capture the journal + boundary and register update capture in one line step. On rewind, reconstruct the selected + history, invalidate the old binding, hydrate the new one and restart its live counter. Reject + deliveries from retired bindings. A periodic checkpoint does not by itself restart an existing + replication stream or require a client-visible update when the state is unchanged. +- **Actual Chord boundary.** `replicatedState(initial)` exposes tracked `.state`, published `.value` + and synchronous `.publish(context)`; its public subscription delivers complete values. Its + decoded-operation subscription and producer registration are private. Moreover, subscribing to + a producer or its service can flush pending mutations. Therefore mutating an exposed producer + and then awaiting storage is NOT a safe persistence adapter. Draft tracking must remain private + until commit; the exact supported Chord adapter seam is still to be designed (§25). +- **Delta ownership and encoding.** Chord Delta publicly supplies `track`, `apply`, `applyImmutable`, + `encoder` and `decoder`. Decoded `Op[]` carry complete paths; compressed `WireOp[]` may depend on + prior path definitions. A complete replacement resets those dictionaries. Arbitrary wire batches + cannot be independently replayed or spliced across histories without their matching codec state. + Durable encoding and codec handling at checkpoints/forks remain an explicit design choice (§25), + not something list storage infers. Mutable replay must own its payloads; it must not mutate the + authoritative in-memory journal. Immutable application shares unchanged subtrees but copies + changed-path containers, whose cost can depend on container size. + +The wrapper owns checkpoint cadence and state/batch bounds. Rewind reconstructs; it does not send +inverse operations. Historical data cannot be pruned merely because a newer checkpoint exists. ### 9.2 Rendering @@ -558,73 +1724,251 @@ Points and decisions: `before_request` (edit), `after_response` (observe), `befo --- -## 11. Reference behaviours (informative) +## 11. Concrete task index -- **generation.effect**: `set streaming`; projection (§6.4) + `before_request`; if aborted return; - stream (frames → scratch, frame deltas); settle: overflow → `set retry_wait{overflow}` + collapse - decision; deferred → `set deferred{handle}`; error → `retry_wait`/`failed`; else one command: - `ins_entry assistant {by}` (+ `ctx_append`) + `settle done {produced, calls}` + one `ins_task tool - {for, at: produced, callIndex, args}` per call (`truncated` if the stop reason is not toolUse). -- **generation.recover**: polling → deferred; frames → settle aborted with the partial and error - results for its calls; none → retry or fail. -- **tool.effect**: truncated/missing → error result; `before_tool` unless `approved`: block → error - result, hold → `set waiting`; `set running`; execute with the `ToolSink` (scratch; caps; details, - usage, addTools, terminate, progress, memo); settle: `ins_entry tool_result {by, callIndex}` - (+ `ctx_append`) + `settle done|aborted` (+ `value activeTools` when tools were added). -- **tool.recover**: replay-safe → run again; else an interrupted result with the last checkpoint. -- **collapse.effect**: capture `through` and `prefixVersion`; `set summarizing`; summarize the - projection of the prefix; settle: `ins_entry summary {through}` + `ctx_replace {through, with, - expectPrefixVersion}` + `settle done` in one command; a version mismatch fails the record. -- **reset by a tool** (`new_context`): the tool's settlement is one command: its result entry - (transcript only), `settle done`, `ins_entry user {by}`, `ctx_reset [handoff]`. The - `canEditContext` precondition is evaluated against the state *after* the settlement in the same - command, so the tool's own task does not block it; if sibling calls of the same generation are - still unresolved, the tool instead records `state.pendingReset = { handoff }` on the - conversation and settles; `next()` applies the reset in the command that runs after the last - sibling settles. A user's `/clear` is `ctx_reset []` and requires `canEditContext`. +§3 is the authoritative task contract; this index introduces no second lifecycle definition. + +| Task | Responsibility | Durable continuation | +|---|---|---| +| generation (§3.9) | provider response/retry/poll | tools + post_tools, or final boundary | +| tool (§3.10) | one call and own result | no sibling/turn orchestration | +| post_tools (§3.11) | join calls, apply exchange policy | next generation or finish | +| collapse (§3.12) | summarize and safely replace prefix | existing generation becomes eligible | +| subagent (§3.13) | own child conversation | child terminal command settles owner | +| job (§3.14) | spawn/adopt process, observe exit | dependent tool or scheduled job | +| approval (§3.15) | human decision | resume/finish existing tool | + +`new_context` is a tool outcome, not another task kind: the tool stores handoff intent; post_tools +appends the handoff, resets context and creates the next generation atomically. --- -## 12. Public API +## 12. Public API and usage + +### 12.1 Shared types and host services ```ts -Harness.open(storage, { kinds, hooks, models, tools, resources, retry, collapse, toolExecution }) // starts nothing -harness.root() / conversation(id) / conversations() / inspect() -harness.watch(conv, { limit }, listener) / entries(conv, opts) / entry(id) / payload(id) / task(id) / find(conv, kind, key?) -harness.close() - -conv.accept(input, { id? }) → { entryId } -conv.drive({ pollDeferred? }) → "idle" | "suspended" -conv.prompt(input) → AssistantMessage | undefined -conv.result(entryId) → pending | settled { entryId, message } -conv.steer / followUp / nextRun / cancelQueued / abort / waitForIdle / runWhenIdle -conv.collapse({ instructions? }) / resetContext(bootstrapIds) / fork({ at, context, label?, summary? }) / rewind(at, { keepRunning? }) -conv.insert(kind, meta, payload?) / conv.task(kind, state) / conv.set(taskId, patch) -conv.setConfig(partial) / conv.state -harness.delete(conv, { confirm }) -vars.declare / get / set; hooks.on(point, handler, { priority }) +type AgentInput = string | UserMessage | readonly AgentMessage[]; +interface ModelIdentity { provider: string; modelId: string } +interface RetryPolicy { maxAttempts: number; baseDelayMs: number } +interface CollapsePolicy { threshold: number; keepRecent: number } +interface GenerationSettings { + model: ModelIdentity; + thinkingLevel: ThinkingLevel; + activeToolNames: string[]; + toolOrder: "parallel" | "sequential"; // creates after edges; not a scheduler lock + streamOptions: JsonObject; + retry: RetryPolicy; +} +interface DriveOptions { pollDeferred?: boolean } +type DriveOutcome = "idle" | "suspended" | "closed"; +interface Inspection { start: Task[]; inflight: Task[]; waiting: Task[] } +interface Accepted { entryId: number } +interface AbortReport { drained: InboxItem[] } +type HistoryTarget = { at: number } | { asOf: number }; +interface TaskServices { + models: Models; + tools: ReadonlyMap; + hooks: Hooks; // typed points in §20 + resources: ReadonlyMap; // host capabilities; checked by consuming kind + retry: RetryPolicy; + collapse: CollapsePolicy; +} +interface SpawnOptions { + prompt: AgentInput; + context: "fresh" | "inherit"; + values: { inherit: readonly ScalarSelection[]; set: readonly InitialValueWrite[] }; +} +interface RegisteredKind { readonly kind: string } // opaque result of validated registration +declare function defineTask(definition: TaskDefinition): RegisteredKind; +interface HarnessOptions extends TaskServices { + initial: GenerationSettings; + kinds: readonly RegisteredKind[]; + systemPrompt: string | (() => string | Promise); +} ``` +`RegisteredKind` is the validated/erased registration returned by defineTask/defineContent; each +constructor preserves its schema's state/status typing. This is not an arbitrary object registry +for values. Provider option patching and process-adoption capabilities remain host seams (§25). + +### 12.2 Harness and conversation + +```ts +interface ConversationView { + state(): Promise; + entry(id: number): Promise; + task(id: number): Promise; + entries(query: Omit): Promise; + tasks(query: Omit): Promise; + getValue(address: ConversationValue, at?: HistoricalRead): Promise; + readList(address: ConversationList, options: ListReadOptions): Promise[]>; +} +interface ConversationCommand { + readonly view: ConversationView; + getValue(address: ConversationValue): Promise; + setValue(address: ConversationValue, value: NoInfer): number; + deleteValue(address: ConversationValue): void; + appendList(address: ConversationList, value: NoInfer, tag?: string): number; + readList(address: ConversationList, options: ListReadOptions): Promise[]>; + entry(kind: string, meta: JsonObject, payload?: JsonValue, options?: EntryOptions): number; + task(kind: string, state: JsonObject, options?: TaskOptions): number; + set(id: number, patch: TaskPatch): void; +} +interface Conversation extends ConversationView { + readonly id: number; + accept(input: AgentInput, options?: { requestId?: string }): Promise; + drive(options?: DriveOptions): Promise; + prompt(input: AgentInput, options?: { requestId?: string }): Promise; + result(inputId: number): Promise; // contract still open in §18.2 + appendMessage(message: AgentMessage): Promise; // context only, deferred if unsafe + steer(input: AgentInput): Promise; + followUp(input: AgentInput): Promise; + nextRun(input: AgentInput): Promise; + cancelQueued(id: number): Promise<"cancelled" | "already_consumed" | "not_found">; + abort(): Promise; // commit intent; drive performs cancellation + shutdown(): Promise; // mark + drive whole ownership subtree, including background + waitForIdle(): Promise; // observe only; starts no work + runWhenIdle(callback: () => Promise): Promise; // host-side admission; semantics to finalize (§25) + collapse(options?: { instructions?: string }): Promise; // create task, not provider effect + resetContext(ids: number[]): Promise; + fork(options: HistoryTarget & { context: "fresh" | "inherit"; label?: string; summary?: string }): Promise; + rewind(target: HistoryTarget, options?: { keepRunning?: boolean }): Promise; + spawn(options: SpawnOptions): Promise; // create owner task; return immediately + setValue(address: ConversationValue, value: NoInfer): Promise; + deleteValue(address: ConversationValue): Promise; + appendList(address: ConversationList, value: NoInfer, tag?: string): Promise; + command(build: (tx: ConversationCommand) => void | Promise): Promise; +} +interface Harness extends ValueReader { + root(): Promise; + conversation(id: number): Promise; + conversations(query: ConversationScan): Promise; + inspect(): Inspection; + drive(options?: DriveOptions): Promise; + command(build: (tx: Command) => void | Promise): Promise; + command(scope: WorkingScope, build: (tx: WorkingCommand) => void | Promise): Promise; + setValue(address: Value, value: NoInfer): Promise; + deleteValue(address: Value): Promise; + watch(conv: number, options: { limit: number }, listener: (event: WatchEvent) => void): Promise<() => void>; + decideApproval(id: number, decision: "grant" | "deny", details?: JsonValue): Promise; + delete(conv: number, options: { confirm: boolean }): Promise; + close(): Promise; +} +declare const Harness: { + open(storage: Storage, options: HarnessOptions): Promise; +}; +``` + +Invocation Context is supplied by the host binding; Storage receives it explicitly. Public async +wait cancellation must remain observer-only; it never silently calls abort. Expected-input fencing +and exact observer Context overloads remain open (§25). `InputResult` needs the explicit attribution +contract in §18.2, not chronological answer guessing. WatchEvent is the wire union in §21. + +Conversation tokens carry scope/kind/value type but no conversation id until bound. SpawnOptions +contains prompt, fresh/inherit context, scalar address selections and typed initial writes (§8.3). +Heterogeneous selections erase only after each write has passed its address's NoInfer check. + +### 12.3 Accept, inspect, drive and cancel + +```ts +const h = await Harness.open(storage, options); // no effects +const root = await h.root(); +const accepted = await root.accept("Inspect the parser", { requestId: "web-request-42" }); +console.log(h.inspect()); // includes pending generation +const outcome = await root.drive(); +if (outcome === "idle") console.log(await root.result(accepted.entryId)); + +// Another caller can cancel the conversation while drive is active: +const driving = root.drive(); +const cancelled = await root.abort(); // durable marks; no per-task cancel +await driving; // provided this scope remains driven +console.log(cancelled.drained); + +// After a restart, inspect for display, then drive; do not iterate tasks to resume each one. +console.log(h.inspect()); +await h.drive({ pollDeferred: true }); // all independent trees +``` + +### 12.4 Queueing and independent scopes + +```ts +const first = await root.accept("Analyze A"); +await root.accept("Then analyze B"); // first generation exists: queues followUp +await root.steer("Use read-only tools"); +await root.nextRun("Apply this on the next explicit idle acceptance"); +await root.drive(); + +await root.appendMessage({ role: "user", content: "Context only", timestamp: Date.now() }); +await root.drive(); // creates nothing merely from this message + +const fork = await root.fork({ at: first.entryId, context: "inherit" }); +await fork.accept("Try a different approach"); +await fork.drive(); // parent and sibling forks remain parked +``` + +### 12.5 Values, child initialization and working state + +```ts +const model = conversationValue("pi.model"); +const thinking = conversationValue("pi.thinking"); +const tools = conversationValue("pi.active-tools"); +const plan = conversationValue("my-plugin.plan"); +await root.setValue(plan, true); +await root.spawn({ + prompt: "Inspect only the parser", + context: "inherit", + values: { inherit: [model, thinking], set: [setValue(tools, ["read", "grep"])] }, +}); +await root.drive(); + +// Working value/list writes use the SAME command path, pinned to one named lifetime. +const work = workingScope("upload-preview"); +const chunks = list("my-plugin.chunks", "", work); +await h.command(work, tx => { tx.appendList(chunks, "first chunk"); }); +await h.command(tx => { tx.retireScope(work); }); +await h.close(); +``` + +The examples are API illustrations, not benchmark results or executable integration tests. + --- ## 13. Invariants and tests 1. The read model equals a replay at every commit, on every backend. -2. Every effect's first write is an inflight status; every settlement is one command. +2. Every external effect is preceded by committed inflight intent; settlement and successors are one command. 3. Every non-terminal status has a driver action; an unknown status fails at open. -4. After `abort(conv)` and one drive, no live task remains under `conv` (transitively); scratch retired. +4. After abort and a completed cancellation drive, all selected foreground/required tasks are terminal + and their working scopes retired. Detached background tasks survive; shutdown includes them. 5. Delivery order and the listener rule (§7.1). 6. Projections never write; the driver never reads a payload. 7. Entries are never patched or reordered; a terminal task accepts no patch. 8. The context list is always `[head?, then entries in transcript order]`; every id on it exists; `through` never splits an exchange; `ctx_replace` with a stale `prefixVersion` rejects; a summarization survives appends and is rejected by a competing replace/reset; the context as of any commit boundary is reconstructible by replaying `ctx_*` records from the newest reset. -9. **Residency**: after open, memory holds live conversation states, live tasks, their scratch, and the entries on live conversations' context lists; nothing else. -10. **Bound**: resident entries = Σ context lists; resident tasks = |live| + one newest generation per conversation; independent of journal length on every backend. Measured at ≥100k turns with a cold reopen at every sample (on memory/JSONL the process heap also carries the backend's read model, by decision). +9. **Residency**: execution holds live conversation states, live tasks, their scratch, context entries and needed settings. Backend history and active plugin replicas are separate, explicitly counted costs (§4.4). +10. **Bound**: resident entries = union of context-list ids; resident tasks = |live| plus newest generations not already live; independent of journal length on every backend. Verify at ≥100k turns with cold reopens; this is a required measurement, not a validated result. Count full journal/index memory on memory/JSONL and active plugin tracking state separately. 11. A fork's context never changes when its parent summarizes or resets later. 12. A task older than any summary survives a cold open, is recovered, settles, and is readable by id. 13. Watch has no gap: base and subscription are one line step; a paged-in entry never regresses a streamed update (`asOf`). +14. Session metadata survives navigation unchanged; historical scalar reads respect every ancestor cutoff and deletion tombstone. +15. Historical forks copy no value/list prefixes; subagents store only selected initial scalar values and overrides, with no parent fallback. Both behaviors survive reopen. +16. List pagination and checkpoint replay produce the same visible history across nested forks as a reference traversal; no page leaks sibling or later-parent writes. +17. Input acceptance and external identity mapping are one atomic commit; a lost response cannot leave just one of them durable. +18. Replication publishes only committed state. Journal gaps require no durable replication counter; fresh bindings hydrate before consecutive emitted updates. +19. Acceptance/settlement/explicit caller commands create generation tasks, never driver tail inference. +20. Generation creates its tools and exactly one post_tools atomically. Tools settle only themselves; + post_tools remains foreground and creates exactly one successor with its own settlement. +21. Context-only writes request no generation, before or after reopen; pending/retrying/deferred work prevents duplicate foreground admission. +22. Reported usage commits with its attempt outcome, including retries and stale collapse publication. +23. Main/working mixed writes reject before mutation. Retirement commits with task outcome; physical + unlink failure cannot resurrect retired data. Late progress cannot reopen a finished task's scope. +24. Overlapping drive scopes share ownership; effect/recovery and abort never write concurrently. +25. Open starts nothing, invents no successors, and restores only explicitly live working lifetimes. Scenarios: the v1 suite re-expressed; the walkthroughs in §15; the adversarial list: rewind after -mutating an old keyed value; a plugin writing thousands of unselected values (memory flat); compact +replacing a conversation value; a plugin writing thousands of versions (execution residency flat, +backend history counted separately); nested forks before/after deletes and checkpoints; selective +subagent settings without inherited plan mode; compact repeatedly (one head, never a chain); an ancient job waiting across many summaries; fork before a later parent summary; a reset while a summarization runs (summary rejected); page while updates arrive (`asOf`); render a summary with a retained tail at `limit: 3` (§15.6). @@ -639,102 +1983,109 @@ arrive (`asOf`); render a summary with a retained tail at `limit: 3` (§15.6). | in-flight state | one 13-leaf op record per lane | status on nodes | status on tasks; one live set | | model context | scan back to the compaction entry | collapse node / folds | explicit list: head + suffix; three ops | | memory | window per lane | whole tree (v1) | context entries + live tasks | -| history | branch index, `getEntry` | resident tree | transcript pages by id, `findValue`, `find` | +| history | branch index, `getEntry` | resident tree | transcript pages, typed historical value/list reads | | subagents | child lanes | nodes | tasks owning conversations | -| plugin state | closures + custom entries | nodes | keyed value entries | -| rewind | move a cursor | fork node | fork with `{at, asOf}`, copied context, tasks aborted by `at` | +| plugin state | session values/lists + custom entries | nodes | typed session values and rewindable conversation values/lists | +| rewind | move a cursor | fork node | historical fork; optionally cancel source conversation foreground | | storage | entries + branch index | ops + checkpoints | journal + read model; JSONL rebuilds by scan, SQLite keeps tables | --- ## 15. Walkthroughs -Notation: `E` entry, `T` task, `set`/`settle` patches, `ctx` context ops; seq = id; live set after -each step; `C.context` shown when it changes. `at` on a task = the entry it started from; `by` on an -entry = the task that wrote it; `for`/`calls` are the explicit dependencies. +Task names below stand for numeric creation seqs. Braces group ONE atomic main commit. Progress +commits go only to the named working scope; terminal commits include its retirement. -### 15.1 Prompt → generation with a tool call → settle +### 15.1 One tool +```text +accept: { user U; create G1 } +request: { G1 streaming } → provider +response: { assistant A; create T; create P after:[T]; settle G1; retire G1 scope } +execution: { T running } → tool +result: { tool result R; settle T; retire T scope } // P remains live, no idle gap +coordination: { settle P; create G2 } +answer: { assistant B; settle G2; retire G2 scope } // no continuation: idle ``` - 1 E user C "add auth" live {} ctx [1] - 2 T gen C {pending, at:1} {2} - 3 set 2 {streaming} (frames → scratch(2); frame deltas) - 4 E assistant C msg(text + call echo) by:2 ctx [1,4] ┐ one commit - 5 settle 2 {done, produced:4, calls:[6]} {6} │ - 6 T tool C {planned, for:2, at:4, callId:x, name:echo} ┘ - 7 set 6 {running} (output → scratch(6); progress deltas) - 8 E tool_result C result by:6 callIndex:0 ctx [1,4,8] ┐ - 9 settle 6 {done, produced:8} {} ┘ -``` -Tick: C's newest generation (2) is terminal and its call (6) is terminal → quiescent → `next`: -last entry is a tool result of a generation without open calls → generation: -``` -10 T gen C {pending, at:8} … 12 E assistant C "done" by:10 ctx [1,4,8,12] 13 settle 10 {done, produced:12, calls:[]} -``` -`next`: settled generation without calls, no follow-ups, `on_yield` passes → `quiescent(C)`; -`prompt()` resolves; `result(1)` = entry 12. Projection for 10 was `[1, 4, 8]` by list; the live -set never exceeded one id; a crash after 7 reopens with `{6}` live and recovers it. -### 15.2 Parallel calls, out of order, one blocked +T knows nothing about P or other tools. P exists before T starts, so a crash after T's result +requires no repair: P is already ready. Crash during T executes its normal recovery policy. -``` - 5 settle 2 {done, produced:4, calls:[6,7,8]} 6,7,8 T tool {planned, for:2} {6,7,8} - 9 E tool_result "write is not allowed in plan mode" by:8 callIndex:2 10 settle 8 {done, blocked} {6,7} (before_tool blocked it) -11 set 7 {running} 12 set 6 {running} -13 E tool_result by:6 callIndex:0 14 settle 6 {done} {7} -15 E tool_result by:7 callIndex:1 16 settle 7 {done} {} -``` -`C.context = [1, 4, 9, 13, 15]` (append order); projection orders the three results by `callIndex` -within message 4: 13, 15, 9. Quiescent after 16 → generation. Sequential mode: `tool` exclusive. +### 15.2 Parallel and sequential calls -### 15.3 Held call, released from a UI, crash in between +```text +parallel response: + { create T1,T2,T3; create P after:[T1,T2,T3]; settle G1 } +completion order: + { R2; settle T2 } → P blocked + { R3; settle T3 } → P blocked + { R1; settle T1 } → P ready + { settle P; create G2 } +sequential response: + { T1; T2 after:[T1]; T3 after:[T2]; P after:[T1,T2,T3]; settle G1 } +execution: + T1 terminal → T2 ready → T2 terminal → T3 ready → T3 terminal → P ready ``` - 6 T tool C {planned, for:2, name:deploy} 7 set 6 {waiting} {6} (before_tool → hold) -``` -Not quiescent (call 6 is not terminal) → no generation. Every watcher shows a waiting tool with its -call. Crash; reopen: live `{6}`, waiting → reported, not started. UI: `set 6 {planned, approved:true}` -→ effect skips the gate → runs → `E tool_result` + `settle done`. Abort instead: the driver aborts -6 → error result + `settle aborted`; the exchange is complete; the turn ends (an aborted call means -no new generation). -### 15.4 Subagent, foreground and background +Both use the same tool and join handlers. The transcript preserves completion order; model +projection emits results in call order. A blocked tool produces an ordinary error result and +settles. Only P interprets completed exchange outcomes and decides whether generation continues. -``` - 6 T tool C {planned, for:2, name:subagent, args:{prompt}} 7 set 6 {running} {6} - 8 T subagent C {planned, for:6, at:4, prompt, context:"fresh"} {6,8} - 9 ins_conv S {parent:{conv:C, at:4, asOf:8}, config, context:[]} -10 set 8 {running, owns:S} -11 E user S "explore" by:8 S.context [11] -``` -S is live (its owner 8 is live) and quiescent with a user entry last → `next(S)` inserts generation -12 … which proceeds as §15.1 inside S: -``` -… 20 settle 12 {done, produced:19, calls:[]} S.context [11, 19] -21 settle 8 {done, result: } {6} -``` -Foreground: the tool effect was awaiting "8 terminal" → `22 E tool_result C "child said: …" by:6` + -`23 settle 6 {done}` → `C.context [1, 4, 22]`. Background: the tool settles right after 10 with -"started subagent 8"; 8 stays live on its own; a later `subagent_status` tool reads task 8 by id or -S's transcript. C's projection never contains S's transcript. `watch(S)` shows S. Abort S only: -`set_conv S {abort}` → 8 settles aborted → 6 (foreground) writes an error result. Crash while S's -generation streams: reopen → `{6, 8, 12}` live → 12 recovers (partial or retry), 6 re-awaits 8. -Nested: S's tool can start T; `conversations({parent})` lists C, S, T. +### 15.3 Hold, restart and conversation cancellation -### 15.5 Background job across a restart, consumed later +```text +T1 held: { create approval A; T1 waiting after:[A] } +T2 after:[T1] and P remain blocked +open: inspect reports all tasks; starts nothing +approve: { A granted; T1 planned approved:true } → covered drive runs T1 +instead, conv.abort(): + { mark T1,T2,T3,P and required approval A; drain steer/followUp once } + signal owned effects; join each task's own execution before abort handler + each tool writes its error result; unstarted tools need not await predecessors + P aborts without generation; every terminal task retires its working scope ``` - 6 T tool C {planned, for:2, name:bash, args:{cmd, background:true}} {6} - 7 T job C {planned, at:4, cmd, for:6} {6,7} - 8 set 7 {spawning} 9 set 7 {running, pid:4242, logPath} (log tail → scratch(7); progress deltas) -10 E tool_result C "started job 7" by:6 11 settle 6 {done} {7} C.context [1,4,10] + +Only conversation cancellation is public. Deleting a queue item is separate from cancelling live +work. Detached background tasks survive foreground cancellation; shutdown/delete includes them. + +### 15.4 Foreground/background subagent + +```text +foreground tool T delegates: + { create owner S; T finishing after:[S] } // effect returns +S starts: + { create child C owner:S; initial values; prompt; Gchild; S running owns:C } +child finishes: + { final assistant B; settle Gchild; settle S result:B } // existing T is now eligible +T finishes: + { parent tool result from B; settle T } // parent P waits for all calls + +background tool T delegates: + { create S; tool result containing S.id; settle T } // no dependency on S ``` -Generation 2's turn is quiescent (its call is done); job 7 is live but not a dependency → `next` -proceeds. Twenty turns and one summary later (`ctx_replace` moved the head), job 7 is untouched: -tasks are never on the list. Crash; reopen: `{7}` running, unowned → `job.recover`: pid alive → -adopt by `logPath`; dead → `settle lost {tail}`. Exit: `95 settle 7 {exited, code:0} payload{tail}`. -Turn 30: `bash_result 7` reads task 7 by id (one row) and its payload; writes a result entry. Its -`at: 4` was folded long ago; nobody needed it. + +No effect holds a promise across child lifetime. Recovery drives existing child tasks and stored +dependencies. Child completion does not execute parent work outside the caller's drive scope; +it only settles its owner as part of that completion. Parent tool finishing runs when covered. +Forks have no owner and are never pulled into a parent's scope merely through provenance. + +### 15.5 Background job across restart + +```text +{ create job J; tool result J.id; settle launching tool } +{ J spawning, durable adoption/log location } → spawn/adopt +{ J running, verified process identity } → install output/exit observers, return +parent P creates next generation without waiting for J + +open → inspect J running; covered drive re-arms observer once +exit → { immutable result E; settle J exited produced:E; retire J scope } +later tool → indexed read J and E, even if J.at is long outside current context +``` + +The external process log and bounded working checkpoints have explicit host lifetime rules. +Neither context compaction nor ordinary foreground cancellation makes a detached job disappear. ### 15.6 Collapse, speculative, subsuming, and the rendering at `limit: 3` @@ -759,14 +2110,14 @@ transcript order as stored; only the summary's display position is relocated. ### 15.7 Reset (Codex-style) -The model calls `new_context` with a handoff note. If it is the generation's only unresolved call, -its settlement carries the reset (the precondition is checked against the post-settlement state); -otherwise the reset is recorded as pending and applied when the last sibling settles: +The new_context tool settles its own result with handoff intent. It never checks siblings or edits +context. Once all calls settle, post_tools applies the handoff: +```text +{ append handoff H; ctx_reset [H]; create G2; settle post_tools } ``` -40 E user C "" by: 41 ctx_reset {ids:[40]} ctx [40]; prefixVersion 2 - (the tail is a user message → the next generation projects [40]; a user /clear resets to [] and waits) -``` -Transcript unchanged; jobs unchanged; keyed values unchanged. The next generation projects `[40]` +This is explicit continuation. `/clear` alone creates no task; changing context while a generation +is already pending does not implicitly cancel that task. +Transcript unchanged; jobs unchanged; conversation values/lists unchanged. The next generation projects `[40]` plus subsequent appends. A watcher draws the handoff at its position and dims everything above it. ### 15.8 Fork and rewind @@ -776,47 +2127,68 @@ Fork at `a1` (entry 2) after §15.6 published: the fork copies C's context restr applicable (it was published after the fork point and the fork's `asOf` is earlier), so the fork reads C's context **as of `asOf`** = `[1, 2]`, trims to the last complete exchange, and starts from `[1, 2]`. Later parent summaries never reach it (invariant 11). `rewind(C, at: 2)` while tool 7 runs -under generation 2: abort work with `at > 2` (tool 7 → error result, exchange complete), then fork. -Keyed values in the fork: `findValue(F, key, { asOf })`, where `asOf` is the commit that inserted entry 2. +under generation 2: cancel/drain the source conversation's foreground chain, then fork. +Values/lists in the fork follow the parent link through the commit containing entry 2; later parent +versions are invisible. A separate `{ asOf }` target can select a value-only or delta-only commit. ### 15.9 Plugin state and replicated state -Plan mode on at turn 12: `E value C key:"plan.mode" {on:true, file}` (unselected). `before_request` -at turn 20: `findValue(C, "plan.mode", { before })` → strips write tools. Fork at turn 10: same read -with `before: 10` → off. A todo board: `E value key:"todo.board" {items}` on each change; found by -`find(C, "value", "todo.board")`; no lifecycle, no residency. A canvas: `E value key:"canvas.base"` -at creation and every N strokes; `E value key:"canvas.delta"` (chord ops) per stroke batch; a new -client hydrates with the newest base before `asOf` plus the deltas after it, then applies stream -records; rewind to P reads the same with `before: P`. Un-journaled preview (cursors) is scratch on -the canvas's live task, if it has one. +Plan mode enabled at commit 80: `value_set C my-plugin.plan-mode=true`. A historical fork through +commit 60 sees the preceding value (or the caller's default if absent); parent changes after 60 +are invisible. Deleting plan mode in the fork masks the inherited value without altering C. +A subagent can copy C's model/thinking level but omit plan mode and override active tools; all +initial settings commit with child creation (§8.3). + +A canvas stores a checkpoint value at creation and periodically, plus one list append per delta +batch. Hydration finds the checkpoint visible through the selected commit and replays only its +visible suffix, including inherited ranges. A fork appends locally without copying parent strokes. +An entry label or external request mapping remains a session value and does not rewind. + +If committed deltas have journal seqs 104 and 117, a freshly hydrated replication binding emits +revisions 1 and 2. Neither revision is stored. Rewind replaces the binding and hydrates the selected +state afresh. Un-journaled preview (cursors) is scratch on the canvas's live task, if it has one. ### 15.10 Open, in general -Read conversation states; read live tasks (`role <> 'terminal'`); for each live conversation read -the entries on its context list (one batched read); read scratch of live tasks. Then `resume()`. -Cost is independent of the journal's length; JSONL pays a header scan first. +Read live tasks (`terminal = false`) and their conversation states; for each live conversation read +the entries on its context list (one batched read); read named working values/lists. Then drive the desired scope. +Execution-state reads use indexes rather than replaying the journal. JSONL first imports the +complete journal; active plugin state hydrates separately from checkpoints and bounded delta pages. --- -# Part II — the complete contract +# Part II — interface details and validation -Parts I (§0–§15) defined the model. This part specifies every surface an implementation needs to get -the edge cases right. Templates and skills are a layer above the harness and are not specified here. +The task definitions are in §3. The following sections supplement them without defining another +lifecycle. Open design questions are explicit in §25. Templates and skills remain a layer above. ## 16. Tools ```ts interface Tool { - name: string; description: string; parameters: JsonSchema; - replay?: "safe" | "never"; // default "never": an interrupted execution is not re-run - output?: { retain?: "head" | "tail"; maxBytes?: number }; // default tail, 64 KiB - execute(args: JsonObject, signal: AbortSignal, out: ToolSink, resources: Resources): Promise; + name: string; description: string; + parameters: JsonObject; // provider-compatible JSON Schema + replay?: "safe" | "never"; // default never + output?: { retain?: "head" | "tail"; maxBytes?: number }; // default tail, 64 KiB + execute(args: JsonObject, ctx: ToolExecutionContext): Promise; } +interface ToolExecutionContext { + signal: AbortSignal; + invocationId: number; callId: string; + conv: ConversationView; + out: ToolSink; + resources: ReadonlyMap; +} +type ToolDelegation = + | { kind: "subagent"; options: SpawnOptions; background?: boolean } + | { kind: "job"; options: { command: string; notBefore?: number }; background?: boolean }; interface ToolSink { write(text: string): void; // appended to scratch; capped per `output`; truncation is marked in the result details(json: JsonValue): void; // result details, replaced on repeat - usage(u: Usage): void; // recorded to the usage ledger at settlement - addTools(names: string[]): void; // becomes a `value activeTools` entry at settlement; unknown names are dropped + image(image: ImageContent): void; // image block in final tool result + handoff(text: string): void; // intent for post_tools, never edits context here + usage(u: Usage): void; // journaled atomically with attempt outcome + addTools(names: string[]): void; // updates the activeTools conversation value at settlement; unknown names are dropped terminate(v: boolean): void; // the run stops after this exchange; the assistant's other calls still settle progress(partial: JsonObject): Promise; // durable checkpoint in scratch; a `progress` delta to watchers; reported by recovery memo: { get(key): Promise; set(key, v): Promise }; // durable per-execution memos, retired at settlement @@ -829,65 +2201,97 @@ bytes }` with the capped text (`[truncated: N bytes total]` appended when capped images. A call for a tool not in `tools` settles with `not found`; a `truncated` call (stop reason other than toolUse) settles with `not executed` and never runs. `addTools` names not present in `tools` are ignored. `terminate` ends the run after the current exchange: no new generation, the -turn is quiescent, `drive` returns `idle`. Concurrency: `tool` is parallel per conversation unless -`toolExecution: "sequential"`, in which case calls run in `callIndex` order and a `hold` blocks the -ones after it. +turn is quiescent, `drive` returns `idle`. Calls are parallel unless +the captured `toolOrder` is sequential: generation creates after edges in callIndex order, and a +held predecessor remains nonterminal. The scheduler has no tool-specific concurrency lock. -## 17. Generation +## 17. Provider request boundary -**Request**: `systemPrompt` (option, static or `() => string`) · messages = projection (§6.4) · -tools = the definitions of `activeTools ∩ tools` · `streamOptions` (option, merged with the call's) -· `reasoning` from `thinkingLevel` (`off` → none). `before_request` hooks run after `set streaming` -and before `streamSimple`; a hook that throws fails the generation (`failed`, `errorMessage`); the -abort marker is re-checked after the hooks and before the provider call. +Generation lifecycle, classifications, retry, polling, recovery and cancellation are defined once, +in §3.9. Request construction projects the captured context (§6.4), resolves captured model/tools, +resolves the system prompt and applies before_request. Tools must still enforce captured active-tool +authorization, not merely registry presence. Capture the MAIN context boundary before request intent; +working progress seqs are not navigation targets. Provider payload/cache/option parity remains in §25. -**Settlement classification**, in order: aborted (signal) → `aborted` with the partial from frames -and an error result per call; `stopReason: "deferred"` → `deferred {handle, poll: 0, notBefore}`; -`stopReason: "error"` and overflow (provider message matches the overflow patterns, and this attempt -is not already an overflow retry) → `retry_wait {overflow, notBefore: 0}` and a collapse decision; -if the collapse is declined or nothing can be folded → `failed`; `stopReason: "error"` otherwise → -`retry_wait {attempt+1, notBefore: now + base × 2^(attempt−1)}` while `attempt < maxAttempts`, else -`failed`; anything else → `done`: one command with the assistant entry (+ `ctx_append`), `settle -{produced, calls, usage, stop}`, one `tool` record per call (`truncated` if the stop reason is not -toolUse). Usage is recorded to the ledger keyed by the generation id. +## 18. Queues and context-only writes -**Deferred**: `deferred` is a `start` status honoured only when `pollDeferred` is true; the effect -`set polling {poll+1}` → `fetchDeferred(handle)` → final message → the same classification; still -deferred → `set deferred {notBefore: now + pollAfterMs}`; fetch error → `deferred` with backoff. -`cancelDeferred(handle)` is attempted on abort (best effort). `pollDeferred: false` leaves such -generations alone and `drive` returns `suspended`. +```ts +interface InboxItem { + id: number; + kind: "write" | "steer" | "followUp" | "nextRun"; + entry: number; // durable unselected content, not yet placed in context +} +``` -**Retry** applies to generation and collapse: `attempt` starts at 1; `retry.maxAttempts` bounds it; -`retry.baseDelayMs` is the base of the backoff; `retry_wait` is a `start` status with `notBefore`; -an abort during `retry_wait` settles `aborted` immediately (the timer is not waited out). +```text +API idle busy +accept(input) place + create generation enqueue followUp +appendMessage / selected place safely; no generation enqueue write + context-only insertion +steer / followUp / nextRun enqueue its tag enqueue its tag -**Recovery**: `streaming` unowned → frames present → `aborted` with the partial and error results -for any calls in it; none → `retry_wait {attempt+1}` or `failed`. `polling` unowned → `deferred`. +boundary eligible items +idle acceptance writes + nextRun + mode-selected steer/followUp, then accepted input +post_tools writes + mode-selected steer; successor unless stopped +normal final answer writes; mode-selected steer/followUp if continuing +idle context-only append earlier writes, then new write; no generation +abort drain steer/followUp; retain write/nextRun +``` -## 18. Queues +Preserve admission order within selected items. `steeringMode`/`followUpMode` default to `all`; +`one-at-a-time` selects the oldest item of that tag, leaving others for later boundaries. `write` +placement never requests generation, even when it leaves a user-shaped context tail. `nextRun` +is consumed only by explicit idle acceptance. On failure/termination, place safe writes but do +not consume input requiring a successor that will not be created. -Inbox items: `{ id, kind: "steer" | "followUp" | "nextRun", entry }` where `entry` is the message -stored as an unselected `value`-like entry (`key: "inbox"`) so the text is journaled once. -- `accept(input)` on an idle conversation (`canEditContext && !hasPendingGeneration` and no - unanswered selected user entry; background tasks do not count): any `nextRun` items are inserted - as user entries first, then the prompt; the run starts. -- `accept` on a busy conversation queues a `followUp`. `steer(text)` queues a steer; `followUp` - and `nextRun` queue their kinds. All return the item id. -- `steeringMode` / `followUpMode` (`all` | `one-at-a-time`, per conversation, default `all`): - `next()` drains steers before a generation, follow-ups when the turn would otherwise end; in - `one-at-a-time` only the oldest is served per boundary. -- `cancelQueued(id)` → `cancelled` | `already_consumed` | `not_found`. -- `abort` drains steer and followUp items and returns them to the caller; `nextRun` items survive. -- `result(entryId)` = the first assistant entry after the input entry whose generation settled with - no calls; `pending` while the conversation has live tasks or the input is still queued; - `unknown` for an id that is neither. +```text +enqueue: unselected content + inbox item in one command +consume: selected placement + dequeue + required generation in one command +cancelQueued: queued → cancelled; already placed → already_consumed; absent → not_found +``` + +Representation linking unselected content to placement/cancellation history remains open with +result attribution (§18.2). Admission position is not necessarily model-context position; do not +silently duplicate payloads or assume old entry ids can be appended out of transcript order. +The four queue behaviors and atomic consumption do not depend on that representation choice. + +### 18.1 Caller-known acceptance identity + +Ordinary `accept(input)` returns its numeric input id. An optional caller-supplied `requestId` +exists for recovery when acceptance commits but its response is lost. It is an opaque external key, +not an internal entity id, and need not be a UUIDv7. Only keyed acceptances pay its storage cost. + +Inside the serialized acceptance command, look up `requestInput(requestId)` (§5.1). If absent, +create the accepted input and set that session value to its id in the same commit. If present, +resolve the existing acceptance rather than inserting another input. There is no crash window +between two records in one atomic command and no separate request-specific storage abstraction. + +Memory/JSONL maintain the current keyed map from journal replay; SQLite uses the session-values +primary key to resolve the request key, then fetches its numeric input id from the journal payload. +No UUID field or secondary UUID index is needed on every entry, task or conversation. The mapping +is session-wide, survives rewind and is not copied as child conversation state. Core owns these +mappings; accepting a request must not overwrite an existing identity association. + +Exact behavior for reusing a key with different input or a different conversation must be settled +before implementation; it must not silently accept a new request or return another conversation's +answer. Lookup itself is simply `harness.getValue(requestInput(requestId))` followed by input/result +lookup. A convenience result overload is not required for this mechanism. + +### 18.2 Pending result attribution contract + +The original rule, "first final assistant after the input", is invalid. If B is queued while A +runs, A's final answer can appear after B's acceptance entry without answering B. An explicit durable +input-to-completion association is needed, including queued consumption, cancellation, failure, +and multiple inputs consumed together. Its representation and exact result outcomes remain open; +request identity lookup does not solve this second association. Do not implement chronological +result inference as a substitute. ## 19. Outcomes, reports, errors -- `drive(opts)` returns `idle` (no live tasks), `suspended` (only waiting tasks, or deferred with - `pollDeferred: false`), `closed`. Waiting for an outcome is a state predicate re-checked after - every commit and driver pass; events are wake-ups, never the source of truth. `conv.drive()` joins the session loop and resolves on `quiescent(conv)` or when the loop - returns; two callers joining the same conversation await the same condition. +- Drive outcomes and ownership scope are defined in §3.8. Conversation idle concerns foreground; + session idle means no live tasks. Suspended includes dependency-blocked work whose prerequisite + awaits an outside event. Observe committed predicates after commits/passes; events only wake. + Callers share execution, not authority to cancel it. - `inspect()` → `{ start: Task[], inflight: Task[], waiting: Task[] }` from the live set; `open()` reports the same and starts nothing. - Errors: `Closed` (command after `close`), `Faulted` (command after a fault; `.reason`), @@ -898,17 +2302,39 @@ stored as an unselected `value`-like entry (`key: "inbox"`) so the text is journ ## 20. Hooks, precisely +```ts +interface HookContract { input: I; decision: D } +interface ProviderRequest { + systemPrompt: string; messages: AgentMessage[]; tools: Tool[]; streamOptions: JsonObject; +} +interface HookContracts { + before_request: HookContract<{ conv: ConversationView; generation: Task; request: ProviderRequest }, ProviderRequest>; + after_response: HookContract<{ generation: Task; message: AssistantMessage }, void>; + before_tool: HookContract<{ call: Task; args: JsonObject }, { kind: "allow" } | { kind: "block"; text: string } | { kind: "hold" }>; + after_tool: HookContract<{ call: Task; result: JsonValue }, void>; + on_yield: HookContract<{ conv: ConversationView; generation: Task; message: AssistantMessage }, { kind: "pass" } | { kind: "continue"; message: AgentMessage }>; + before_collapse: HookContract<{ conv: ConversationView; through: number; prefixVersion: number }, { kind: "decline" } | { kind: "allow"; meta: JsonObject }>; +} +interface Hooks { + on

( + point: P, + handler: (input: HookContracts[P]["input"]) => HookContracts[P]["decision"] | Promise, + options?: { priority?: number }, + ): () => void; +} +``` + | point | input | decision | combination | |---|---|---|---| | `before_request` | `{ conv, generation, request: { systemPrompt, messages, tools, streamOptions } }` | `request` | applied in order | | `after_response` | `{ generation, message }` | — | all run | | `before_tool` | `{ call: Task, args }` | `allow` \| `block(text)` \| `hold` | first block/hold wins; later hooks do not run | | `after_tool` | `{ call, result }` | — | all run | -| `on_yield` | `{ conv, turn: Entry }` | `pass` \| `continue(message)` | first continue wins | +| `on_yield` | `{ conv, generation, message }` (completed final response candidate) | `pass` \| `continue(message)` | first continue wins | | `before_collapse` | `{ conv, through, prefixVersion }` | `decline` \| `{ meta }` | first decline wins; metas merged | A hook that throws: `before_request` fails the generation; `before_tool` blocks with the error text; `before_collapse` declines; `on_yield` passes; observers are reported as `handler_error` and ignored. Hooks are called by effects, outside the line, so a hook -MAY `await ctx.commit(...)` (append entries or keyed values, insert work); those commands interleave +MAY `await ctx.commit(...)` (append entries, write typed values/lists, insert work); those commands interleave with other actors under §24. A `before_request` hook affects *this* request only through the request it returns; an entry it appends is projected from the next generation on. Only listeners (watch/`on`) are forbidden from issuing commands inside their callback. @@ -917,26 +2343,41 @@ request it returns; an entry it appends is projected from the next generation on ``` base { type: "base", asOf, conversation: ConversationState, entries: Entry&{payload}[], tasks: Task[], scratch: { [taskId]: JsonValue } } -record { type: "record", seq, record: ins_entry&{payload} | ins_task | set | settle | set_conv | ctx_append | ctx_replace | ctx_reset | ins_conv | del_conv } +commit { type: "commit", firstSeq, lastSeq, writes: [ { seq, change } ] } // relevant MAIN writes, §4.1 delta { type: "delta", taskId, kind: "frame" | "progress" | string, payload } signal { type: "quiescent", conv } | { type: "fault", message } | { type: "handler_error", taskId?, message } page entries(conv, { before | after, limit }) → { asOf, entries: Entry&{payload}[] } ``` -Base and subscription are one line step; every record after the base has `seq > asOf`; a page has -its own `asOf` and records with `seq ≤ asOf` are ignored for its entries. `del_conv` on the watched -conversation ends the watch. A `ctx_replace`/`ctx_reset` record is the client's cue to dim or drop. +Base and subscription are one line step. Commit batches retain boundaries even when filtering +out unrelated writes; clients reduce a whole batch before publishing. Thus tool/result retirement +or post_tools/successor creation cannot become a partial visible transition. Preview/signal fields +are not durable work requests. Pages carry their own asOf for deduplication. del_conv ends the watch; +context replacement/reset cues model-context rendering. WatchEvent is this base/commit/delta/signal union. ## 22. Usage ledger and telemetry -**Usage**: one row per settlement that carried usage (`generation`, `collapse`, `tool` via the -sink): `{ taskId, conv, seq, usage }`; append-only; queried by conversation or session; never -rewritten; summed by clients. Not in the journal. +**Usage** is journaled with the corresponding attempt outcome, not independently written to a ledger. + +```text +provider completes: commit reported usage + outcome/retry/deferred transition together +collapse loses prefix race: commit reported usage + failed collapse, without summary/context edit +tool completes: commit usage + own result + own settlement + working-scope retirement +post_tools completes: commit exchange policy + successor generation, if continuing +identity: usage record seq, NOT taskId; a task can have several attempts +query: read the main session usage list; any aggregate/filter indexes are backend implementation details +fork: costs remain attributed to original conversations; no copied billing rows +cleanup: task completion/scratch retirement never removes usage +``` + +An interrupted request with no provider report has unknown cost; no exactly-once billing claim. +Reject stale/repeated outcomes instead of appending duplicate usage. Usage is an ordinary main +session list; no separate ledger authority/file or redundant full usage payload in task patches. **Telemetry** uses pi's callback `TelemetryContext` (typed schemas, no second contract). Spans and their parents follow the procedure nesting: ``` harness.open · harness.drive (per pass) - conversation.next + acceptance / task settlement (explicit successor creation) task.effect / task.recover / task.abort (attributes: kind, taskId, status from→to, attempt) ai.request (model, usage, stop reason, durations; never prompts or completions) tool.execute (name, callIndex, bytes, isError; never args or results) @@ -952,36 +2393,100 @@ only that caller's observation. ## 23. Storage conformance Each backend passes: -1. `append` is atomic per commit and ordered; seqs are dense and monotone. +1. commit preserves command-minted global seqs and atomic batches. Each admitted batch starts at + global head+1 and is contiguous; main files, working files and retained history have legal gaps. 2. Read model equals a replay of the journal after every commit (entries, tasks, conversations, context lists, `prefixVersion`). -3. `entries(conv, {before|after, limit})` equals a scan; ranks are correct; `findValue`/`find` - equal a scan. -4. Scratch: append/read/set/get per task id; survives a process crash; retired by settlement; - orphans removed on open; a crash between settlement and retirement is tolerated. +3. Transcript pages and content queries equal a scan; typed value reads and list pages equal + reference history traversal, including nested fork cutoffs and deletion tombstones. +4. Working values/lists use ordinary reads/writes, survive restart, and disappear only through + explicit retirement. Result + task settlement + retirement survive or disappear together. + Crash before physical unlink leaves logically retired files, not recoverable task state. 5. JSONL: one line per commit; a torn final line is truncated on open; rebuild-by-scan yields the same read model as SQLite's tables for the same journal. -6. Usage rows survive and are queryable. -7. Open on an empty store creates the root conversation once; open on a non-empty store writes nothing. +6. Usage survives atomically with its outcome, including multiple attempts under one task id. +7. Storage open creates no conversation/tasks. Harness open on empty storage creates root + initial + values in one command; nonempty open starts no work and writes no journal records. +8. Command construction: creation methods return their final numeric ids before append; subsequent + records reference them directly, including automatic context appends. No-op and rejected commands + consume no seqs; concurrent commands allocate disjoint consecutive ranges; reopen continues after + the last complete commit. Payloads such as `{ ref: 0 }` round-trip unchanged. +9. SQLite value/list queries use ordered indexes without journal scans or temporary sorts; + sparse keys and deep fork chains do not require reading unrelated history. Fork creation adds + no inherited value/list rows. Subagent creation stores only selected present values and overrides. +10. Reopen preserves metadata, historical versions, list cursors and subagent initialization. + Checkpoint payloads in the memory journal remain unchanged after mutable replay by a wrapper. +11. Acceptance and request mapping survive or disappear together; a retry after a lost response + does not create a second input. Conflicting reuse tests follow the policy still pending in §18.1. +12. Paused persistence exposes old complete state, never proposed indexes. Short/failed JSONL writes + and crash after newline/SQL commit but before publication reopen to one complete prefix. + Malformed complete JSONL lines fail open; only unterminated final bytes are discarded. +13. Close drains admitted main/working writes and retires nothing. Reused scope ids retain only + records newer than latest retirement. Old cleanup cannot delete a reused lifetime. +14. Crash around acceptance, tool and post_tools/final settlement: join always exists; no consumed + trigger without generation, no duplicate successor on reopen; usage agrees with outcomes. +15. Mixed main/working and two-working-id commits reject. Memory, JSONL and SQLite agree on logical + retirement. High-water computation includes retired physical records before cleanup. +16. Late progress is rejected by invocation fencing, not by a global ban on scope-id reuse. ## 24. Race catalog (each has exactly two durable orders; tests assert both) | race | orders | |---|---| | `accept` vs `accept` on an idle conversation | first is the prompt; second queues as followUp | -| `accept` vs process loss before `drive` | entry absent → caller retries by id (idempotent); present → `drive` runs it | +| keyed `accept` vs process loss before response/`drive` | input and mapping both absent → retry may accept; both present → resolve existing input, never duplicate it | | `drive` vs `drive` | both join one pass | -| `abort` vs generation settlement | marker first → `aborted` with partial and error results; settlement first → `done`; a later abort is a no-op (stale) | +| `abort` vs generation settlement | marker first → aborted, no successor; settlement first → output and successors commit, then abort reaches any still-live successor | | `abort` vs tool outcome | marker first → the effect's signal fires and it settles `aborted`; outcome first → `done`, result preserved | | `abort` vs `retry_wait` | settles `aborted` without waiting the timer | -| `abort` vs collapse settlement | marker first → collapse `failed`, list unchanged; settlement first → list replaced | +| `abort` vs collapse settlement | marker first → collapse aborted, list unchanged; settlement first → list replaced | | `abort` vs `on_yield` continuation | marker first → the continuation entry is not inserted; entry first → the run continues and abort settles it | -| `cancelQueued` vs boundary consumption | `cancelled` | `already_consumed` | -| `setConfig` vs generation start | old config or new; the generation records what it used | +| `cancelQueued` vs boundary consumption | cancellation first → cancelled; consumption first → already_consumed | +| conversation setting write vs generation creation | old value or new; the generation captures what it uses | | `nextRun` vs `accept` | captured by this prompt or stays for the next | | collapse settlement vs `ctx_reset` | reset first → `version_mismatch`, collapse `failed`; settlement first → the reset applies to the new list | | collapse settlement vs appends | appends survive either order | -| frame/progress write vs settlement | settlement awaits pending scratch writes, then retires scratch; a crash between leaves orphans for open | +| frame/progress write vs settlement | seal/drain first; outcome + retirement commit together; failed unlink leaves logically dead data | | watcher registration vs commit | one line step: old base + all later records, or new base | | `close` vs settlement | settlement committed before close, or the task stays inflight and recovers next open | | fork vs parent summary | the fork copied its list as of `asOf`; the summary is not applied to it | +| subagent initialization vs parent setting write | selected old values or selected new values at one captured boundary; no mixed initialization | +| plugin update vs plugin update | second command derives its update from the first committed state | +| replica binding vs committed delta | hydration includes it, or the new stream emits it after hydration; never a gap | +| parallel tool outcomes | either order settles only those tools; existing post_tools becomes ready when all dependencies are terminal | +| steering vs generation creation | included with creation, or queued for next boundary | +| context-only write vs drive/reopen | context may change; neither order creates generation | +| settlement vs uncertain persistence outcome | output/usage/successors/retirement all absent or all durable on reopen; old handle faults | +| post_tools settlement vs conversation abort | marked join creates nothing, or successor commits first and becomes a cancellation target | +| scope retirement vs reuse | later writes survive; older unlink is serialized before new lifetime | +| working progress vs retirement | admitted old writes drain before retirement; stale producer cannot recreate task data | +| overlapping drive callers vs cancellation | one shared ownership claim covers effect join and abort handler | + +## 25. Remaining design decisions + +These are not settled by the value/list and identity decisions above: + +- **Result attribution:** durable input-to-completion association and conflicting request-key reuse + (§18). Caller-known identity must not be confused with result ownership. +- **Historical context optimization:** §6.6 defines correct creation/reset-based replay; faster + compaction-bounded reconstruction still needs review. Replacement alone is not a checkpoint. +- **Subagent context:** safe inherited-context capture while parent tool exchange is open; caller + selections are copied at child creation, not implicitly at address construction. +- **Host and background lifecycle:** durable process adoption/exit status, bounded schedules, and + whether close retains cooperative waiting or gains a bounded host policy. Finalize runWhenIdle + admission/observer cancellation without introducing a second execution owner. +- **Queue/response policy:** exact input normalization, steering precedence/mode capture, conflicting + handoffs, retained failed/deferred attempt messages and projection. Keep representation and policy + separate from the settled chain mechanism. +- **Old-harness parity:** intentionally decide differences in tool termination aggregation, hook + transformations/error handling, interrupted-stream recovery, retryable/overflow classification, + deferred poll permits/external retry scheduling, token-budget/split-turn compaction, summarized + navigation/import, usage adjustments/totals and complete watch hydration. These are not silently + removed by rewriting tasks. Expected-input cancellation/result fencing remains tied to attribution. +- **Chord adapter:** choose a supported producer/operation seam and durable delta encoding, including + checkpoint/fork codec state. Do not mutate an exposed producer while a commit is pending. Stream + revisions remain process-local emission bookkeeping, regardless of this choice. + +Implementation, stronger regression coverage, and equivalent faux-provider benchmarks follow design +review. CPU time, total process memory, and actual disk bytes (including scratch and usage) must be +measured separately; the spike's existing figures are not validated comparisons for this design.