From ce5ec9ca355a852fb1f25c088cf409df47a95213 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Wed, 9 Sep 2026 20:40:04 +0200 Subject: [PATCH] docs(agent): persist pico entry projections --- packages/agent/docs/pico/pico-usage-guide.md | 143 ++++---- packages/agent/docs/pico/pico-v3.md | 332 +++++++++++-------- packages/agent/docs/pico/pico-work.md | 57 ++-- 3 files changed, 309 insertions(+), 223 deletions(-) diff --git a/packages/agent/docs/pico/pico-usage-guide.md b/packages/agent/docs/pico/pico-usage-guide.md index 0c6f9c24d..708e260b6 100644 --- a/packages/agent/docs/pico/pico-usage-guide.md +++ b/packages/agent/docs/pico/pico-usage-guide.md @@ -69,10 +69,10 @@ conversation has three things: - **state**: keyed **values** and **lists**, for the model in use, plan mode, a game board, whatever a plugin needs to remember. -What the model sees, the **context**, is not stored. The newest **head** supplies a replacement -(summary, handoff or nothing) and the first retained transcript entry. The harness prepends that -head, reads forward from the retained entry, applies immutable **edit** entries that omit or replace -individual projections, then projects messages. Compaction appends a head; tool-result pruning +What the model sees, the **context**, is not stored as a list. Each entry may store `model` +messages, a **head** boundary, and **edits** that omit or replace earlier model messages. The harness +prepends the newest head, reads forward from its stored boundary, folds retained edits, then performs +request-local tool and provider normalization. Compaction appends a head; tool-result pruning appends an edit. Forks, compaction and reset never mutate old entries. Every id is a session sequence number, minted when the write is built and never changed: @@ -147,7 +147,7 @@ w.start(event => { // accept the input, drive the conversation until it is idle, return the answer to that input const answer = await c.prompt('Inspect the parser and list the public API'); -console.log(answer?.content); +console.log(answer?.model[0]?.content); w.unsubscribe(); await h.close(); // cancels nothing durable; reopening the file continues exactly here @@ -193,8 +193,10 @@ const h = await Harness.open(storage, { models, tools, replace: { }}); ``` -Open checks that every kind found in storage is registered (without scanning the transcript: the -set of kinds ever written is kept) and reports live work, but starts nothing: +Open checks the recorded kind strings without scanning the transcript and reports live work, but +starts nothing. An unregistered historical entry kind is reported, not rejected: its stored +`model`, `head` and `edits` still build context, while its typed data and custom renderer are +unavailable. A live task kind is still required. ```typescript const { start, inflight } = await h.inspect(); @@ -284,13 +286,14 @@ definitions). pico is built on that, and keeps two things apart: - **What you want right now.** Config (model, thinking, selected tools, a profile) plus whatever the host has: cwd, skills, context files, the tool catalogue. The host answers a hook with it each turn. None of it is stored. -- **What the model has been told.** `system` entries in the transcript: the baseline that opened the - current context and one entry per change delivered since. Immutable facts about requests that - happened, so they fork and compact like every other entry. +- **What the model has been told.** `system` entries in the transcript: structured `data` for the + baseline or delta and the exact materialized `SystemMessage` in `model`. These are immutable facts + about requests that happened, so they fork and compact like every other entry. -At the top of every turn the generation renders the first, folds the second out of the context, -and appends the difference, if any, before it projects the request. The host keeps rendering "the -whole prompt" the way it always did; the harness turns that into append-only deltas. +At the top of every turn the generation renders the first, folds the second out of the context, and +appends the difference, if any, with both its structured data and model message before it projects +the request. The host keeps rendering "the whole prompt" the way it always did; the harness turns +that into append-only deltas. ### Answering the Hook @@ -318,9 +321,9 @@ After the first turn the transcript holds what was sent: 12 assistant ... ``` -The tool definitions are stored in full, not as names. Rendering a persisted message may depend -only on what is stored on it, so a later change to a tool's description is a change the model gets -told about, and an old transcript renders the way it was sent. +The tool definitions are stored in full, not as names, and the model message is materialized in the +same append. A later change to a tool's description is therefore a change the model gets told about, +and an old transcript is sent the way it was originally recorded without running entry-kind code. ### Changing the Loadout @@ -436,7 +439,7 @@ const answer = await c.prompt('Inspect the parser'); // AssistantEntry | undefined (the run ended without an answer: aborted, failed) ``` -The pieces are available separately, and they compose across retries. `accept` is idempotent by +The pieces are available separately, and they work across retries. `accept` is idempotent by `requestId`, and `answerTo` finds the answer to an input however far the conversation has moved since, so a client that lost the reply can ask again: @@ -622,7 +625,7 @@ await alt.prompt('Try a different implementation'); const back = await c.fork(earlier.id, { abort: true }); // "go back": aborts the source's foreground first ``` -Any content entry is a valid fork point, including an assistant with unanswered tool calls or one +Any transcript entry is a valid fork point, including an assistant with unanswered tool calls or one of several results. The request projection supplies missing results for a successful incomplete exchange without inheriting or executing the source tasks. Which conversation a UI treats as "current" is the UI's business; the harness only has conversations. @@ -635,10 +638,11 @@ const children = await h.conversations({ parent: c.id }); // forks and ## Compaction and Reset -Compaction appends a summary head whose content names the first retained entry. The context becomes -the summary followed by the transcript from that entry. It runs as a background task and may run -while the model keeps working: ordinary entries landing meanwhile remain after the prepared retained -boundary. Only a competing head makes the summary stale; edit entries do not. +Compaction appends a summary entry with its model message and first retained entry id stored on the +entry. The context becomes the summary followed by the transcript from that boundary. It runs as a +background task and may run while the model keeps working: ordinary entries landing meanwhile +remain after the prepared boundary. Only a competing head makes the summary stale; edit entries do +not. ```typescript const collapseId = await c.collapse(); @@ -752,7 +756,7 @@ inside it see committed state; ids are final when returned; a throw discards eve ```typescript const entry = await c.commit(tx => { tx.value(planMode).set(false); // rewindable state first - const id = tx.entry(myPlugin.noteKind, { text: 'plan accepted' }); + const id = tx.entry(myPlugin.noteKind, { data: { text: 'plan accepted' } }); tx.value(expanded).set(true); // sticky state may follow tx.task(myPlugin.reminderKind, { background: true, // tasks anywhere state: { about: id, at: Date.now() + 3600_000 } }); @@ -825,8 +829,9 @@ its own conversation through `ctx.conversation.commit(...)`: a job, a child conv Anything that is *about* the call rather than its output goes through `diag`: truncation, a spilled file, a corrected path, "the file changed on disk since you read it". The harness emits the ones it owns (the sink itself reports truncation and spill); a tool adds only what it alone knows. -Projection puts the data first and the commentary after it, delimited, so the model can treat tool -output as data, and a UI renders diagnostics as callouts by severity: +Tool settlement puts the output first and the commentary after it and stores that exact message in +entry `model`. Non-message details, usage, control flags, diagnostics and truncation metadata become +entry `data`. A UI combines the two and renders diagnostics as callouts by severity: ```text ...last matching line @@ -914,56 +919,78 @@ effects need their own idempotence. ## Writing Kinds -Kinds are how you extend the harness with new behaviour rather than new state. An entry kind says -what a kind of entry means to the model; a task kind says how a kind of work runs. +Kinds are how you extend the harness with new behaviour rather than new state. An entry kind is a +typed name for an immutable transcript shape; a task kind says how a kind of work runs. ### Entry Kinds +Entry facets combine. `data` is optional kind-specific JSON for logic and custom UI rendering; +`model` is an optional stored `Message[]`; `head` and `edits` are optional stored context controls. +No facet is derived while reading. + ```typescript -interface NoteContent { text: string; pinned?: boolean } +interface NoteData { text: string; pinned?: boolean } +type NoteEntry = EntryBase & EntryData; -// the model never sees this kind: no `project` -export const noteKind = defineEntryKind({ kind: 'myplugin.note' }); +export const noteKind = defineEntryKind('myplugin.note'); -// the model sees this one -export const pinnedKind = defineEntryKind({ - kind: 'myplugin.pinned', - project: e => [{ role: 'user', content: `${e.content.text}`, timestamp: 0 }], +await c.commit(tx => tx.entry(noteKind, { + data: { text: 'Parser plan accepted', pinned: false }, +})); // transcript/UI only; the model sees nothing +``` + +A plugin that wants both typed data and a model message keeps the append-time conversion in an +ordinary helper: + +```typescript +type PinnedEntry = EntryBase & EntryData & ModelProjection; +export const pinnedKind = defineEntryKind('myplugin.pinned'); + +function appendPinned(tx: ConversationTx, data: NoteData, timestamp: number): Id { + return tx.entry(pinnedKind, { + data, + model: [{ role: 'user', content: `${data.text}`, timestamp }], + }); +} +``` + +Heads and edits are supplied the same way: + +```typescript +type WindowEntry = EntryBase & EntryData<{ retainFrom: Id }> & ContextHead; +export const windowKind = defineEntryKind('myplugin.window'); + +tx.entry(windowKind, { + data: { retainFrom }, + head: retainFrom, // first retained entry, inclusive }); -// a head kind: prepend this entry, then retain the transcript from retainFrom -export const windowKind = defineEntryKind<{ retainFrom: Id }>({ - kind: 'myplugin.window', - head: entry => entry.content.retainFrom, -}); +type ToolResultEditEntry = EntryBase & ContextEdits; +export const toolResultEditKind = defineEntryKind('myplugin.tool_result_edit'); -type ToolResultEdit = { target: Id; messages?: Message[] }; -export const toolResultEditKind = defineEntryKind({ - kind: 'myplugin.tool_result_edit', - edit: entry => entry.content.messages === undefined - ? [{ target: entry.content.target, action: 'omit' }] - : [{ target: entry.content.target, action: 'replace', messages: entry.content.messages }], +tx.entry(toolResultEditKind, { + edits: replacement === undefined + ? [{ target, action: 'omit' }] + : [{ target, action: 'replace', messages: replacement }], }); ``` -The presence of `head` makes writes of that kind persist `Entry.head = true`, which supports an -indexed newest-head lookup. The function itself returns the first retained entry id. Commit -validation requires that value not move before the previous visible head's returned value. The -latest head is prepended and older heads are excluded from its retained range. - -`edit` interprets immutable entry content as projection operations. Edits are applied in transcript -order; the newest edit for a target wins. They omit or replace individual retained projections -without changing the target entry. There is no dynamic `compose`: append another head or edit when -context should change. +`head: 'self'` in an append draft stores the new entry id, which is how reset and handoff discard +everything earlier. Commit validation requires a stored head boundary not to move before the +previous visible boundary. Edits apply in transcript order; the newest edit per target wins. Append +another head or edit when context should change; no entry-kind callback runs during context reads. Reads are typed by the kind, or untyped and narrowed: ```typescript -const note = await h.getEntry(noteKind, id); // Entry | undefined (also undefined for another kind) -const any = await h.getEntry(id); -if (noteKind.is(any)) any.content.pinned; +const note = await h.getEntry(noteKind, id); // NoteEntry | undefined (also undefined for another kind) +const any = await h.getEntry(id); // Entry | undefined +if (noteKind.is(any)) any.data.pinned; ``` +A missing plugin removes that narrowing and its custom renderer, but not context behavior: the +entry's model messages, head and edits are stored independently of the kind. + ### Task Kinds A task kind declares its statuses and their roles, its config, its hooks, and three functions. @@ -985,7 +1012,7 @@ export const reminderKind = defineTaskKind({ const { skip } = await ctx.hooks(reminderKind).run('before_fire', { about: task.state.about }); await ctx.commit(tx => { if (!skip && !tx.getTask(task.id)!.abort) // a marked task creates no successor - tx.entry(noticeKind, { text: `Reminder: see entry ${task.state.about}` }); + tx.entry(noticeKind, { model: [noticeMessage(`Reminder: see entry ${task.state.about}`)] }); tx.settle(task.id, 'done', { ...task.state, fired: !skip }); }); }, diff --git a/packages/agent/docs/pico/pico-v3.md b/packages/agent/docs/pico/pico-v3.md index d1702b66b..7b6ad4b0c 100644 --- a/packages/agent/docs/pico/pico-v3.md +++ b/packages/agent/docs/pico/pico-v3.md @@ -52,134 +52,157 @@ are never model input by themselves. ```ts type Id = number; // a session sequence number; one alias so the representation can change -interface Entry { - readonly id: Id; // its sequence number - readonly conversationId: Id; - readonly kind: string; - readonly content: Content; - readonly key?: string; // optional indexed key, e.g. the tool call a result answers - readonly head?: true; // this kind is a context head; indexed - readonly byTaskId?: Id; // which task wrote it -} -``` - -Entries append. They are never patched, reordered or renumbered. Ids increase in transcript order -but need not be consecutive. - -Kinds are code, not data: the built-ins are registered by `Harness.open`, plugins add theirs -next to the task kinds (§5.2) and tools (§9.5). Storage sees only the `kind` string on each entry and never runs kind code; the scan -and projection look the string up in the registry. - -```ts type ContextEdit = | { readonly target: Id; readonly action: "omit" } | { readonly target: Id; readonly action: "replace"; readonly messages: readonly Message[] }; -interface EntryKind { +interface EntryIdentity { + readonly id: Id; // its sequence number + readonly conversationId: Id; readonly kind: string; - project?(entry: Entry): readonly Message[]; // absent: the model never sees this kind - head?(entry: Entry): Id; // first retained entry, inclusive - edit?(entry: Entry): readonly ContextEdit[]; - is(entry: Entry): entry is Entry; + readonly byTaskId?: Id; // which task wrote it +} + +interface EntryBase extends EntryIdentity { + readonly key?: string; // optional indexed key, e.g. the tool call a result answers +} + +interface EntryData { + readonly data: Data; // kind-specific durable data; never model input by itself +} + +interface ModelProjection { + readonly model: readonly Model[]; // provider-neutral messages materialized at append +} + +interface ContextHead { + readonly head: Id; // first retained transcript entry, inclusive +} + +interface ContextEdits { + readonly edits: readonly ContextEdit[]; +} + +type Entry = EntryBase & Partial< + EntryData & ModelProjection & ContextHead & ContextEdits +>; + +type UserEntry = EntryBase & ModelProjection; +type AssistantEntry = EntryBase & ModelProjection; + +interface SummaryData { readonly summarizedThrough: Id } +type SummaryEntry = EntryBase & EntryData & ModelProjection & ContextHead; +``` + +The facets combine. A user entry may be only `EntryBase & ModelProjection`; a tool +result may also carry non-message `EntryData` for its renderer; a summary has data, +a model projection and a head; a reset needs only a head. Entries append and are never patched, +reordered or renumbered. Ids increase in transcript order but need not be consecutive. + +`model` is the exact provider-neutral message projection chosen by the writer. It is not the final +provider request: context selection, edit folding, tool-exchange normalization and provider +conversion still happen per request. `data` is optional arbitrary kind-specific JSON for typed +logic and rendering. Nothing implicitly derives one from the other, so storing both never requires +duplicating a message. + +An entry kind is only a name and type witness: + +```ts +interface EntryKind { + readonly kind: string; + is(entry: Entry): entry is E; } ``` -`project` turns an entry into the messages the model receives. A kind without it is invisible to the -model: it is in the transcript and never in the context. +A plugin normally keeps append-time construction in a typed helper. It may compute `model`, `head` +and `edits` from its data before calling `Tx.entry`, but no such computation is registered on the +kind or run while reading. Asynchronous inputs are prepared before entering the commit line. -A kind with `head` defines a context head. `Tx.entry` stores only `head: true`, so storage can index -heads without understanding the kind. When deriving context, the harness calls `head(entry)` to get -the first retained entry id, inclusive. A summary returns the first entry it did not summarize; -`handoff` and `reset` return their own id, so nothing earlier is retained. A plugin can define its -own head kind. - -For every new head: +A stored `head` is the inclusive first retained entry id. Presence of the column marks and indexes +the entry as a head. `Tx.entry` accepts `"self"` for reset and handoff drafts and stores the minted +entry id. Every new head must satisfy: ```text -newKind.head(newHead) >= oldKind.head(previousVisibleHead) +newHead.head >= previousVisibleHead.head ``` -The commit line loads and interprets both entries and enforces this. Plugin code supplies the -candidate but cannot move context backwards. Older heads are controls, not retained context entries: -the newest head replaces them and carries any predecessor summary information it still needs. +The commit line validates this directly from stored entries. A head cannot move context backwards. +Older heads are controls, not retained context entries: the newest head replaces them and carries +any predecessor summary information it still needs. -A kind with `edit` defines a context edit. Its immutable entry content is interpreted as standard -operations that omit or replace the projection of earlier visible entries. Edits are folded in -transcript order; the newest applicable edit for a target wins. The target entry itself is never -changed, and an edit whose target is outside the selected range is a no-op. Retained edit entries -are applied again on later turns. Tool-result pruning and shortening are edits. Dynamic policies -such as "always keep only the last N exchanges" are unsupported; code appends a new head or edit -when it wants context to change. There is no `compose` callback. +Stored `edits` omit or replace the model projection of earlier visible entries. Edits are folded in +transcript order; the newest applicable edit for a target wins. The target entry is never changed, +and an edit whose target is outside the selected range is a no-op. Retained edit entries apply again +on later turns. Tool-result pruning and shortening are edits. Dynamic policies such as "always keep +only the last N exchanges" are unsupported; code appends a new head or edit when context should +change. There is no `compose` callback or read-time entry-kind behavior. The built-ins are `user`, `assistant`, `tool_result`, `system`, `notice`, `summary`, `handoff` and -`reset`. -All have `project` except `reset`, which starts an empty context; `handoff` starts a context with -its own message; `system` projects to pi-ai's `SystemMessage`, each at its position in the -transcript, so the prompt in force at any entry is on record and a provider's cached prefix is -never rewritten: +`reset`. Their writers materialize model messages when they append. `reset` stores no model; +`handoff` stores its message and starts a context; `system` stores the exact pi-ai `SystemMessage` +at its transcript position, so the prompt in force at any entry is on record and a provider's +cached prefix is never rewritten: ```ts -interface SystemContent { +interface SystemData { readonly baseline?: true; // opens an epoch (§8.2) readonly sections?: readonly { key: string; text: string }[]; // in render order; baseline: all; delta: the changed ones readonly toolsAdded?: readonly Tool[]; // complete definitions as sent, never names resolved later readonly toolsRemoved?: readonly Tool[]; } + +type SystemEntry = EntryBase & EntryData & ModelProjection; ``` -Definitions are stored, not looked up, and sections are stored in the order they were rendered, -because rendering a persisted message must depend only on what is stored on it; a changed -description, schema or section order is therefore a delta like any other. `project` turns the -entry into pi-ai's `SystemMessage` at request time: a baseline's sections into the top-level prompt -(or one system message where there is no baseline slot), a delta's into "the `plan_mode` guidance -now applies: …" with `toolsAdded`/`toolsRemoved` attached. Plugin-defined kinds usually have no `project`. The -kind is the type witness for reads and writes, so nothing is cast and nothing is validated: -stored objects are trusted (§7.3), and a kind that changes its shape ships a migration. +Definitions and sections are stored in `data` for the next generation's structured comparison and +in the materialized message in `model`. A changed description, schema or section order is therefore +a delta like any other. Plugin-defined entries omit `model` when the model should not see them. +The kind types reads and writes; stored objects are trusted after wire validation (§7.3), and a kind +that changes its data shape ships a migration. ```ts -tx.entry(toolResultKind, conversationId, content); // content: ToolResultContent -const e = await harness.getEntry(summaryKind, id); // Entry -const any = await harness.getEntry(id); // Entry -if (summaryKind.is(any)) any.content.through; // narrowed +type ToolResultEntry = EntryBase & EntryData & ModelProjection; +const toolResultKind = defineEntryKind("tool_result"); + +tx.entry(toolResultKind, conversationId, { data: resultData, model: [message], key: callId }); +const e = await harness.getEntry(summaryKind, id); // SummaryEntry | undefined +const any = await harness.getEntry(id); // Entry | undefined +if (summaryKind.is(any)) any.data.summarizedThrough; // narrowed ``` -The kind is optional on reads: without it you get the untyped entry, and `kind.is(entry)` narrows -it. A typed read of an entry of another kind returns `undefined`, the same as a missing entry; -batched reads omit it. - -Open checks the set of entry kinds ever written against the registry: on SQLite that is a `kinds` -table with one row per distinct kind, filled by `INSERT OR IGNORE` in the commit that first uses a -kind; on JSONL and memory it falls out of the replay open does anyway. No scan of entries; an -unknown kind rejects before anything runs. +Without a kind, a read returns the untyped entry. A typed read of another kind returns `undefined`, +the same as a missing entry; batched reads omit it. The session records the distinct kind strings +without scanning entries. Open reports an unregistered entry kind but does not reject: stored +`model`, `head` and `edits` preserve context, while only kind-specific typing and rendering are +unavailable. ### 2.2 Context is derived, not stored The generation derives context from the transcript: ```text -H = newest head visible at the request/fork target (one indexed query) -from = no H → transcript start; otherwise kind(H).head(H) +H = newest entry with stored `head` visible at the request/fork target +from = no H → transcript start; otherwise H.head range = fork-aware transcript entries from `from` through target, inclusive -controls= fold edit entries in transcript order +controls= fold stored `edits` in transcript order context = no H → eligible range entries H exists → H, then eligible range entries excluding every head omitted targets disappear; replaced targets keep their id and use replacement messages -messages= project through each entry's kind; normalize tool exchanges for the selected model +messages= concatenate stored `model` arrays; normalize tool exchanges for the selected model ``` -Edit entries are controls and do not project unless their kind separately defines `project`. -Entries whose kind has neither control behavior nor `project` contribute nothing. Existing tool -results are placed with their calls in call order. For a successful assistant cut before all its -results, request-local pi-ai transformation supplies missing results; results beyond the fork cutoff -and source tasks are never inherited. Error/aborted assistant outputs are excluded from subsequent -requests and never create tool tasks/results. +An entry without `model` contributes no messages. An edit entry may independently have a model +projection. Existing tool results are placed with their calls in call order. For a successful +assistant cut before all its results, request-local pi-ai transformation supplies missing results; +results beyond the fork cutoff and source tasks are never inherited. Error/aborted assistant +outputs are excluded from subsequent requests and never create tool tasks/results. No context list is stored, so nothing can drift from the transcript. A conversation handle keeps an optional process-local cache of the current derived entries, candidate membership and winning edits. -`contextEntries(through)` loads it once, catches it up with entries after its cursor, slices it when -a new head arrives and folds new edits. Generation tasks receive that handle; `ConversationView` -exposes the resulting context ids. A historical target older than the cache is derived separately -and does not rewind it. Each request receives an immutable snapshot. +`contextEntries(through)` loads it once, catches it up with entries after its cursor, slices it from +the stored boundary when a new head arrives and folds stored edits. Generation tasks receive that +handle; `ConversationView` exposes the resulting context ids. A historical target older than the +cache is derived separately and does not rewind it. Each request receives an immutable snapshot. On a cold handle, cost is the fork-aware candidate range plus edit folding and fork depth, not necessarily the final projected-message count. Memory and JSONL answer from their in-memory indexes; @@ -188,12 +211,11 @@ convenience owned by the handle and may be garbage-collected with it; the transc ### 2.3 Compaction and reset -**Compaction** appends one summary head. Its typed content names the first retained entry, and its -kind returns that id: +**Compaction** appends one summary with a stored model projection and head boundary: ```text transcript [10 user, 20 asst, 30 user, 40 asst] context [10, 20, 30, 40] -append 50 summary { retainFrom: 30 }, head:true +append 50 summary { model:[summary], head:30 } scan [30 user, 40 asst, 50 summary] context [50 summary, 30 user, 40 asst] ``` @@ -205,9 +227,9 @@ may still land later: ordinary entries appended while it runs are at or after it boundary and remain in the range. Only a competing head invalidates it; intervening context edits do not (§8.4). -**Reset** appends a `handoff` head (with a message the model sees) or a `reset` head (nothing). -Their `head(entry)` returns `entry.id`. The transcript keeps everything. Reset by itself neither -cancels tasks nor requests a response. +**Reset** appends a `handoff` head with `model` or a `reset` head without it. Both pass +`head: "self"` to `Tx.entry`, which stores the new entry id as the boundary. The transcript keeps +everything. Reset by itself neither cancels tasks nor requests a response. ### 2.4 Forks share history, not future changes @@ -254,7 +276,7 @@ One session-wide sequence orders every mutation. A creation's sequence number is // last committed: 99 const entryId = await conv.commit(tx => { tx.value(planMode).set(true); // 100 - const id = tx.entry(userKind, message); // 101 + const id = tx.entry(userKind, { model: [message] }); // 101 tx.task(generationKind, { state: { input: id } }); // 102 return id; }); @@ -280,7 +302,7 @@ history through X: // tool settlement, one commit await ctx.commit(tx => { tx.value(planMode).set(true); // 300 - const result = tx.entry(toolResultKind, content); // 301 + const result = tx.entry(toolResultKind, { data: resultData, model: [message] }); // 301 tx.settle(task, { result }); // 302 }); @@ -294,7 +316,7 @@ before persistence and consumes no ids. Session state and sticky conversation st anywhere because forks do not reconstruct their history, so they may reference ids minted earlier in the same commit. Tasks also go anywhere; they are not inherited by forks and usually come after the entries they name. Historical reads (§4) take an entry as their position; "before the first -entry" is the empty position. Any content entry is a valid fork point, including an assistant call +entry" is the empty position. Any transcript entry is a valid fork point, including an assistant call or one of several tool results. Projection repairs the resulting successful incomplete exchange for the request only; it never inherits or executes the source tasks. @@ -423,7 +445,7 @@ const partial = await ctx.scratch(sc => sc.list(frames).read()); // settlement: the result becomes an entry; settle retires the scope in the same commit await ctx.commit(tx => { - const id = tx.entry(assistantKind, assemble(partial)); + const id = tx.entry(assistantKind, { model: [assemble(partial)] }); tx.settle(task, { result: id }); }); ``` @@ -587,7 +609,7 @@ The generation's settlement publishes the exchange atomically: ```ts await ctx.commit(tx => { - const assistant = tx.entry(assistantKind, message); // calls [A, B] + const assistant = tx.entry(assistantKind, { model: [message] }); // calls [A, B] const tools = message.calls.map(call => tx.task(toolKind, { state: { call, assistant } })); tx.task(postToolsKind, { after: tools, state: { assistant, tools } }); @@ -610,7 +632,9 @@ async execute(task, ctx) { const selectedTools = added.length ? [...current, ...added] : current; if (added.length) tx.value(generationKind.config.selectedTools).set(selectedTools); const handoff = outcomes.find(o => o.handoff); - if (handoff) tx.entry(handoffKind, handoff.handoff); + if (handoff) tx.entry(handoffKind, { + data: { text: handoff.handoff }, model: [handoffMessage(handoff.handoff)], head: "self", + }); landSteering(tx, task.conversationId); // §8.1 tx.task(generationKind, { state: nextGeneration(task, { selectedTools }) }); tx.settle(task, {}); @@ -824,9 +848,10 @@ successor. Background work: `abortTask(job)`, or `abort(childConversation)` for ### 6.4 Open, close, shutdown, delete -**Open** opens storage, checks the set of entry kinds written (§2.1) and the live tasks' kinds -against the registry, exposes `inspect()` (live tasks by role) and queries, -starts nothing. **Close** stops admitting commits, cancels owned executions, waits for them and for +**Open** opens storage, reports unregistered entry kind strings (§2.1), checks live task kinds +against the registry, exposes `inspect()` (live tasks by role) and queries, and starts nothing. +Unknown entries remain fully usable for context from their stored facets; an unknown live task kind +still rejects. **Close** stops admitting commits, cancels owned executions, waits for them and for admitted persistence, releases storage; it writes no outcomes, so unfinished work resumes on the next open. **Shutdown** marks all live work, drains queued input, drives cancellation to settlement, then closes. **Delete** of a conversation rejects while its ownership subtree has live work, then @@ -889,7 +914,7 @@ interface TaskQuery extends Cursor { readonly conversationIds: readonly interface ListQuery extends Cursor { readonly at?: Id } // at = an entry id (§3.2) interface ValueQuery { readonly scope: Scope; readonly namespace: string; readonly after?: string; readonly limit: number } // keys, ordered -type EntryHeader = Omit; +type EntryHeader = Omit; // context fields without arbitrary plugin data interface Version { readonly seq: Id; readonly value: T } interface Element { readonly id: Id; readonly value: T } @@ -924,7 +949,7 @@ Pages carry a cursor and the sequence they were read at. Transcript reads are lo reads: they combine each fork's capped source prefix with local entries, carrying every ancestor cutoff recursively. `from` and `through` are inclusive entry positions in that logical transcript; source entries after a fork point never appear. `newestHead(conversationId, at)` applies the same -cutoffs and returns the newest visible `head:true` entry at or before `at`. Rewindable conversation +cutoffs and returns the newest visible entry with `head` at or before `at`. Rewindable conversation state follows the same ancestor cutoffs. Session state stays current; sticky conversation state is copied from current state only when the fork explicitly selects it. Tasks are never inherited. Historical reads on sticky or scratch addresses reject. Coherent multi-read operations run on the @@ -963,11 +988,11 @@ from there; the line guarantees nothing landed in between, so storage numbering storage check that for free. The harness validates against committed state plus earlier writes in the batch (kinds validate status transitions; the harness enforces ownership, dependencies, exchange rules and admission; nobody validates payload shapes: -stored objects are trusted, and schema validation belongs at wire boundaries). For a `head:true` -entry, validation calls its kind's `head(entry)`, requires the returned id to be visible, and requires -it to be at or after the previous fork-visible head kind's returned id. Edit targets must be earlier -visible entries. Storage checks scope, sequences and structure, prepares only touched data, persists -atomically, publishes. A commit with no writes writes nothing. +stored objects are trusted, and schema validation belongs at wire boundaries). For an entry with +`head`, validation requires the stored id to be visible and at or after the previous fork-visible +head's stored boundary. Stored edit targets must be earlier visible entries. No entry-kind code runs. +Storage checks scope, sequences and structure, prepares only touched data, persists atomically, +publishes. A commit with no writes writes nothing. ### 7.4 Memory and JSONL @@ -979,7 +1004,7 @@ one line per batch: ```json {"first":100,"writes":[ {"type":"value.set","conversationId":1,"namespace":"plugin.plan","value":true}, - {"type":"entry","conversationId":1,"kind":"user","content":"Inspect"}, + {"type":"entry","conversationId":1,"kind":"user","model":[{"role":"user","content":"Inspect","timestamp":0}]}, {"type":"task","conversationId":1,"kind":"generation","status":"pending"}]} ``` @@ -1005,12 +1030,15 @@ must never remove a deliberately reused scope. Tables: conversations, entries, tasks (current records including terminal), values and value_versions, list_elements and clear_markers, scratch, commit_boundaries, kinds (one row per -distinct entry kind), session_metadata (storage version, §6.5). Keys start with the session id. Indexes follow the -query table: entries by (conversation, id), (conversation, kind, key, id) and a partial index -(conversation, id) where head is set; tasks by (role, id) and (conversation, role, id) with a +distinct entry kind), session_metadata (storage version, §6.5). An entry row has nullable JSON +columns for `data`, `model` and `edits`, plus a nullable integer `head`; storage understands those +core fields but never a plugin's data shape. Keys start with the session id. Indexes follow the query +table: entries by (conversation, id), (conversation, kind, key, id) and a partial index +(conversation, id) where head is not null; tasks by (role, id) and (conversation, role, id) with a partial index on non-terminal roles; value_versions by (conversation, namespace, key, seq); -conversations by parent and by owner. One commit is one transaction: number the writes from `lastSeq + 1`, -insert rows, record the boundary, advance `lastSeq`. No full rewrite of anything per append. +conversations by parent and by owner. One commit is one transaction: number the writes from +`lastSeq + 1`, insert rows, record the boundary, advance `lastSeq`. No full rewrite of anything per +append. Values are stored whole, current and historical: a point read must not become a replay chain, and disk is cheaper than that. Scratch is rows in the `scratch` table keyed by (task, address), written @@ -1064,7 +1092,7 @@ execute: if a collapse is needed by threshold: { create collapse C; create g none of the inputs is stored sent = the newest baseline `system` entry in the context plus the deltas after it; deltas before it (kept by a compaction) are subsumed and ignored, by the fold - and by projection alike + and by request projection alike { if no sent: system entry with the baseline (text + full tool definitions); else if sent ≠ desired: system entry with the diff, per section ("the plan_mode guidance now applies: …", "the cwd section changed: …") and toolsAdded / toolsRemoved as full @@ -1091,10 +1119,10 @@ The generation diffs the two at the top of every turn and writes only the delta. handoff, reset) starts a new epoch: the context has no baseline after it, so the next generation writes a fresh one, and any older delta a compaction kept is subsumed by it and dropped at projection; a fork carries the sent-state in its prefix and diffs against its own config; a restart with a changed host -emits "these sections now apply" and nothing else. Projection puts the epoch's baseline in the -provider's baseline slot, later entries as system messages at their positions, and the folded tool -set as the callable list, or folds everything into one top-level prompt on providers without -native system messages. +emits "these sections now apply" and nothing else. Request projection puts the epoch's stored +baseline message in the provider's baseline slot, later stored system messages at their positions, +and the folded tool set as the callable list, or folds +everything into one top-level prompt on providers without native system messages. One task lives through retries and deferral; retries share a budget carried in state. Overflow does not keep the generation alive: it settles and hands the retry to the collapse's settlement, @@ -1177,7 +1205,7 @@ create: capture the prefix to replace (ending on a complete exchange), its fir execute: { status summarizing } before_collapse (may decline or supply the summary); call the summarizer; candidate to scratch { if no head newer than the captured one: - append summary { retainFrom:firstRetained }, whose kind marks it head:true; settle done + append summary { model:[summary], head:firstRetained }; settle done else: settle failed(stale) } ``` @@ -1187,8 +1215,8 @@ but do not invalidate or recompute the summary. A summary may land while a gener since the stream already projected its context and the replaced prefix is old. Automatic collapses (threshold, overflow) are foreground; manual ones are background. A collapse never creates a generation except through the settlement chain in §8.2. Abort records usage and settles without -publishing. Prefix pruning/windowing uses another head kind; pruning or replacing individual -retained entries uses edit entries (§2.1). Dynamic `compose` policies are unsupported. +publishing. Prefix pruning/windowing appends another entry with a stored head; pruning or replacing +individual retained entries uses edit entries (§2.1). Dynamic `compose` policies are unsupported. ### 8.5 Subagents @@ -1280,7 +1308,9 @@ const jobKind: TaskKind = { if (ctx.signal.aborted) return tx.settle(task.id, "killed", { ...task.state, output }); const r = getOrThrow(result); if (task.state.notify) // the call that started it returned early (§9.3): tell the model - tx.entry(noticeKind, { text: `job ${task.id} (${task.state.origin?.tool}: ${task.state.cmd.join(" ")}) finished, exit ${r.exitCode}` }); + tx.entry(noticeKind, { + model: [noticeMessage(`job ${task.id} (${task.state.origin?.tool}: ${task.state.cmd.join(" ")}) finished, exit ${r.exitCode}`)], + }); if (task.state.every) // recurring: same task, next time tx.patch(task.id, { status: "planned", state: { ...task.state, notBefore: Date.now() + task.state.every } }); else @@ -1297,8 +1327,8 @@ shell doesn't hand out an adoptable process, recover reruns if the job declared otherwise settles `lost`. A recurring job is the same task looping `planned → running → planned`; `abortTask(id)` ends it wherever it is. Output stays in scratch while it runs and in the terminal state after, where `jobOutput`, the `job` tool and UIs read it; there is no result entry, since a call that -waited already carries the output. A job whose call returned early appends a `notice` entry (a kind -that projects to a `SystemMessage`) when it finishes, so the next generation learns of it without +waited already carries the output. A job whose call returned early appends a `notice` entry with a stored `SystemMessage` when it +finishes, so the next generation learns of it without polling. Interval and catch-up policy are in state, so a restart never launches a backlog. @@ -1419,20 +1449,35 @@ the same answer however far the conversation has moved since. Nothing else maps ```ts const entryId = await harness.commit(tx => { tx.value(planMode).set(true); // state first (§3.2) - const id = tx.entry(noteKind, conversationId, content); // final id, inside the closure + const id = tx.entry(noteKind, conversationId, { + data: { text: "plan accepted" }, + model: [{ role: "user", content: "plan accepted", timestamp: 0 }], + }); // final id, inside the closure tx.task(reminderKind, { background: true, state: { about: id } }); - return id; // any value; resolved after commit + return id; // any value; resolved after commit }); ``` ```ts +type EntryInput = + Omit & + (E extends ContextHead ? { readonly head: Id | "self" } : { readonly head?: never }); + +interface EntryDraft { + readonly key?: string; + readonly data?: JsonValue; + readonly model?: readonly Message[]; + readonly head?: Id | "self"; + readonly edits?: readonly ContextEdit[]; +} + interface Tx { // reads (committed state) getEntry(id) / getEntry(kind, id) / getEntries(...) / getTask(id) / getTask(kind, id) / getTasks(...) value(addr: Value): { get(at?): T | undefined; set(v: T): void; delete(): void }; list(addr: List): { append(v: T): Id; remove(id: Id): void; clear(): void; read(q): Page }; // writes; rewindable conversation state before entries (§3.2) - entry(kind: EntryKind, conversationId: Id, content: C): Id; + entry(kind: EntryKind, conversationId: Id, input: EntryInput): Id; task(kind: TaskKind, spec: { state: S; after?: Id[]; background?: true; owns?: Id[] }): Id; patch(task: Task | Id, changes: { status?: string; state?: Partial }): void; settle(task: Task | Id, outcome: Partial & { status?: string }): void; // terminal; retires scratch @@ -1473,12 +1518,13 @@ class Tx { }; } - entry(kind: EntryKind, conversationId: number, content: C): number { + entry(kind: EntryKind, conversationId: number, draft: EntryDraft): number { this.sawEntry = true; const id = this.seq + 1; - const entry: Entry = { - id, conversationId, kind: kind.kind, content, - ...(kind.head === undefined ? {} : { head: true }), + const { head, ...fields } = draft; + const entry: Entry = { + id, conversationId, kind: kind.kind, ...fields, + ...(head === "self" ? { head: id } : head === undefined ? {} : { head }), }; this.push({ type: "entry", entry }); return id; @@ -1556,7 +1602,7 @@ interface ToolContext { jobOutput(id: Id): Promise; // preview or terminal state } -// what the sink folds to; recorded on the tool task and the tool_result entry at settlement +// what the live sink folds to; recorded on the tool task and used to build the result entry interface ToolOutputState { content: (TextContent | ImageContent)[]; details: JsonValue; @@ -1568,6 +1614,8 @@ interface ToolOutputState { diags: readonly { severity: "info" | "warn" | "error"; message: string; code?: string }[]; truncation: ShellOutputTruncation; // totals over everything ever written } + +type ToolResultData = Omit; ``` `execute` returns nothing and reports failure by throwing; the harness sets `isError` when it @@ -1582,8 +1630,8 @@ spilled to a file, the path was resolved differently, the file changed on disk s the search stopped at 500 matches) goes through `out.diag`, never into the text the model reads as the tool's data. The harness emits the ones it owns: the sink calls `diag` itself when it truncates or spills, the path resolver when it corrects a path, the blocking budget when a call continues as -a job. A tool adds only what it alone knows. Projection renders the data first and the diagnostics -after it, delimited: +a job. A tool adds only what it alone knows. Tool settlement renders the data first and the +diagnostics after it, delimited, and stores that message on the result entry: ```text …last line of the file @@ -1594,8 +1642,10 @@ after it, delimited: so the model can parse tool output as data, a plugin can post-process it without stripping notices it doesn't know, and a UI renders diagnostics as callouts by severity. `isError` says whether the -call failed; a `warn` diag does not change it. The transcript records exactly the commentary the -model saw, because it is in the entry's `ToolOutputState`. +call failed; a `warn` diag does not change it. Tool settlement stores the exact rendered result as +entry `model`; the non-message fields of `ToolOutputState` become `ToolResultData` for typed logic +and rendering. The transcript therefore records exactly the commentary the model saw without +copying the message into `data`. ### 9.4 Watch @@ -1754,12 +1804,12 @@ h.kinds.generation; h.kinds.tool; h.kinds.postTools; h.kinds.collapse; h.kin The built-in entry kinds (`user`, `assistant`, `tool_result`, `system`, `notice`, `summary`, `handoff`, `reset`) and task kinds (`generation`, `tool`, `post_tools`, `collapse`, `job`) are -registered by `open` itself, because `accept`, `prompt`, `steer` and `collapse` cannot work without -them. A replacement is registered under the built-in's name and must understand its persisted -statuses and keep its hook names, so existing handlers keep working; a wrapper that delegates to -the original is the usual shape. `h.kinds.` is how handles and plugins refer to whatever is -currently registered, so `c.settings` is the config of the generation kind in use, not of a -specific import. +registered by `open` itself, because `accept`, `prompt`, `steer` and `collapse` need them to write +and type new records. Reading context does not need entry kinds. A task-kind replacement is +registered under the built-in's name and must understand its persisted statuses and keep its hook +names, so existing handlers keep working; a wrapper that delegates to the original is the usual +shape. `h.kinds.` is how handles and plugins refer to whatever is currently registered, so +`c.settings` is the config of the generation kind in use, not of a specific import. `Harness.open` takes storage, models, tools, additional kinds, replacements and initial root values; it creates the root only for empty storage and never creates work on reopen. diff --git a/packages/agent/docs/pico/pico-work.md b/packages/agent/docs/pico/pico-work.md index 70dcb8071..a484bba53 100644 --- a/packages/agent/docs/pico/pico-work.md +++ b/packages/agent/docs/pico/pico-work.md @@ -11,19 +11,23 @@ shows the surface each package has to end up with. ## 1. Types and ids -`Id`, `Entry`, `EntryKind`, `ContextEdit`, `Task`, `TaskRole`, `Conversation`, -`Address`/`Value`/`List`/`Scope`, `Write`/`CommitBatch`, `Page`/`Cursor` and the query shapes -(§2, §4.1, §5.1, §7.2–7.3). No code. +`Id`, `EntryIdentity`, `EntryBase`, the composable `EntryData` / `ModelProjection` / +`ContextHead` / `ContextEdits` facets, `Entry`, `EntryKind`, `EntryInput`, `ContextEdit`, `Task`, +`TaskRole`, `Conversation`, `Address`/`Value`/`List`/`Scope`, `Write`/`CommitBatch`, `Page`/`Cursor` +and the query shapes (§2, §4.1, §5.1, §7.2–7.3). No code. -Test: it compiles; a fixture file with one instance of every record shape. +Test: it compiles; fixtures cover entries with no data, no model, every individual facet and the +built-in facet combinations. ## 2. Memory storage `Storage` and `MemoryStorage`: `commit` numbering from `lastSeq`, point reads, fork-aware -`scanEntries` / `scanTasks` / `scanConversations` with cursors, target-capped `newestHead`, value -versions and list elements read at a position, `ownedFrom`, the set of entry kinds written. +`scanEntries` / `scanTasks` / `scanConversations` with cursors, target-capped `newestHead` over stored +numeric boundaries, stored entry data/model/edits, value versions and list elements read at a +position, `ownedFrom`, the set of entry kind strings written. Tests: every row of the §7.2 query table against hand-built batches; ids are `lastSeq + 1 + i`; a +head is found without a kind; entry headers omit arbitrary data but retain model/head/edits; a live-task scan decodes no terminal rows; `remove` and `clear` hide by position. ## 3. The line and `Tx` @@ -41,21 +45,23 @@ append/remove/clear after an entry each reject. ## 4. Entry kinds and context -`EntryKind`, the registry, the built-in kinds (`user`, `assistant`, `tool_result`, `system`, -`notice`, `summary`, `handoff`, `reset`) with `project`, `head` and `edit`; context = newest head -prepended to the fork-aware range from its returned id, older heads excluded, edits folded in -transcript order; projection to pi-ai messages with tool results ordered by call index; the `system` -kind's epoch fold (newest baseline plus the deltas after it, older deltas dropped) and its projection -into the baseline slot / `SystemMessage`s. No `compose`. +`EntryKind` as `kind` plus `is`, the registry and typed append helpers for the built-in kinds +(`user`, `assistant`, `tool_result`, `system`, `notice`, `summary`, `handoff`, `reset`). Writers +materialize optional model messages and stored controls; context = newest stored head prepended to +the fork-aware range from its numeric boundary, older heads excluded, stored edits folded in +transcript order, stored model arrays concatenated, then pi-ai tool results ordered by call index. +The `system` data fold selects the newest baseline plus later deltas and places their stored messages +in the baseline slot / `SystemMessage`s. There is no read-time entry-kind behavior. -Tests: summary keeps the tail; handoff and reset; repeated compaction subsumes; a returned head id -below the previous visible head kind's returned id is rejected; edits omit/replace targets and persist -across turns; the fold on a context that kept an old delta; tool-result order. +Tests: data-only and model-only entries; summary keeps the tail; handoff/reset normalize `"self"`; +repeated compaction subsumes; a stored head below the previous visible boundary is rejected; edits +omit/replace targets and persist across turns; context is identical with its plugin kind unregistered; +the fold on a context that kept an old delta; tool-result order. ## 5. Forks and historical reads `createConversation` with `parent`, the shared prefix in fork-aware `scanEntries`, capped-source -lookup for values and lists, arbitrary content-entry fork points. +lookup for values and lists, arbitrary transcript-entry fork points. Tests: a fork sees the head, edits and values in force at its entry; heads/results the source adds later are invisible; deep fork chains; successful incomplete tool exchanges project with missing @@ -87,7 +93,8 @@ new attempt clears its list. `settings`, `fork` with `abort`, `abort`, `hooks.on` scoped with `subtree`), `abortTask`, `conversations` with `parent` / independent filtering. No agent behaviour yet. -Tests: open on empty vs existing storage; an unknown kind rejects without scanning entries; +Tests: open on empty vs existing storage; an unregistered entry kind is reported without scanning +entries and its stored model/head/edits still derive context; an unknown live task kind rejects; replace by name keeps `h.kinds.` consistent; `settings` round-trips; scoped hooks run after harness-wide ones, innermost last. @@ -95,7 +102,8 @@ harness-wide ones, innermost last. Statuses pending → streaming → done / failed / retry_wait / deferred / aborted on a faux provider; config capture (model, thinking, selected tools, profile, budget); `system_instructions` with -sections merged across handlers and the diff writing `system` entries; `before_request`, +sections merged across handlers and the diff writing `system` data plus its materialized model +message; `before_request`, `after_response`, `on_yield`; retry sleeps in execute; recover from frames; usage recorded per attempt. @@ -107,7 +115,8 @@ crash while streaming publishes the partial; retry budget exhausted → failed w ## 10. Tools, post_tools, exchanges The tool kind with the sink (`ToolOutput`, `ToolOutputState`, limits enforced by the sink, `diag`, -`delegate`, `handoff`, `addTools`, `terminate`), `before_tool` (fail-closed) and `after_tool`, +`delegate`, `handoff`, `addTools`, `terminate`), tool-result entries with structured data plus their +materialized model message, `before_tool` (fail-closed) and `after_tool`, replay policy on recover; post_tools with `after`, terminate / handoff / steer / next generation; `accept` idle vs busy; `prompt` and `answerTo`. @@ -177,17 +186,17 @@ fold, failed and aborted attempts included. ## 17. JSONL storage -Append-only batches, replay on open, torn tail discarded, malformed line fails open, the kinds set -from replay, values as Chord deltas (full when first or small, ops otherwise), scratch sidecars -retired after the main-file settle. +Append-only batches, replay on open, torn tail discarded, malformed line fails open, entry +`data`/`model`/`head`/`edits` and the kind-string set from replay, values as Chord deltas (full when +first or small, ops otherwise), scratch sidecars retired after the main-file settle. Tests: conformance against memory (one mutation stream, identical query results); linear file growth under repeated sets of a large value; crash between the settle write and the sidecar unlink. ## 18. SQLite storage -Tables and indexes of §7.5, scratch rows deleted in the settle transaction, storage version and -`migrate`. +Tables and indexes of §7.5, nullable entry JSON columns for data/model/edits and an indexed integer +head boundary, scratch rows deleted in the settle transaction, storage version and `migrate`. Tests: conformance three ways; a cold reopen decodes only live rows (count them); a version mismatch rejects.