From 09b031600d8e911eeea0b115555f11d3f3466093 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sun, 13 Sep 2026 22:49:14 +0200 Subject: [PATCH] docs(agent): pico3 review decisions: revision, visibility, memos, waiting, reload - Envelope carries a contiguous per-watch revision; storage Seq stays internal; Chord bridge publishes off the line through a bounded ordered adapter - Namespaces reach the view only through a declared view projection; memos and slot working state are never in the view - memoOnce uses property presence; memos are coordination/evidence, not exactly-once; idempotency key before the effect - Hooks bound to their namespace token; BeforeToolApi.waiting(ctx) is async; memo keys are (task, namespace, name) - Reload: quiescent hold/reload/release, otherwise suspend/reopen; no handler withdrawal machinery in v1; Namespace.unregister - pi.* entry names reject except pi.notice; config reset is an operation; generation failures run the final boundary and start successors --- .../agent/docs/pico/v3/hardening-handoff.md | 18 +- packages/agent/docs/pico/v3/plugins.md | 252 +++++++++++------- .../agent/docs/pico/v3/view-and-events.md | 122 ++++++--- 3 files changed, 251 insertions(+), 141 deletions(-) diff --git a/packages/agent/docs/pico/v3/hardening-handoff.md b/packages/agent/docs/pico/v3/hardening-handoff.md index f0abeff31..83b01eec7 100644 --- a/packages/agent/docs/pico/v3/hardening-handoff.md +++ b/packages/agent/docs/pico/v3/hardening-handoff.md @@ -32,11 +32,15 @@ Specifically superseded in this document: |---|---| | Accepted direction #3 (separate `rewindable`/`sticky` view fields with separate op lists) | one flat view document with `config`, `inbox`, `turn`, `compaction`, `tasks`, `plugins` and one op list (`view-and-events.md` §2–§4); the storage split stays | | Accepted direction #2 "live task slots ... in the view" and §13 last paragraph (`TaskRef` slot typing) | slots remain, but the view publishes kind-described `tasks[id].status` via `describe()`, never raw task records (`view-and-events.md` §2.2, §6.6; `plugins.md` §4.3) | -| §9 Watch protocol: envelope shape and `applyDelta` | `Envelope { seq, ops, events }` folded with `applyImmutable`; the active-transcript definition and capture rules in §9 here remain normative (`view-and-events.md` §3, §5, §9) | -| §9 bounded capacity / overflow-close | retained for in-process watches; with Chord `ReplicatedState` as the transport (`plugins.md` §6.1) Chord owns sequencing and gap detection | +| §9 Watch protocol: envelope shape, `Seq` ordering and `applyDelta` | `Envelope { revision, ops, events }` with a contiguous per-watch `revision` (storage `Seq` is not exposed), folded with `applyImmutable`; the active-transcript definition and capture rules in §9 here remain normative (`view-and-events.md` §3, §5, §9) | +| §9 bounded capacity / overflow-close | the raw watch invokes its listener synchronously in order; only the pre-`start` buffer is bounded (overflow closes the watch); asynchronous consumers own their queue (`view-and-events.md` §3; `plugins.md` §6.1 for the Chord bridge) | +| §13 plugin state visibility | a namespace is in the view only through its declared `view` projection; memos and slot working state are never in the view (`plugins.md` §1.5, §2, §4.5) | +| §15 `beforeTool` throw = block; hooks by name | hooks are registered with their namespace token (`h.hooks(ns, kind, handlers)`); `BeforeToolApi.waiting(ctx)` is asynchronous; memo keys are `(task, namespace, name)` (`plugins.md` §4.2) | +| §16 / view §6.2 generation failure closures | every generation failure resolves the group `unanswered/failed`, runs the final boundary and starts a successor for queued triggers (`view-and-events.md` §6.2) | +| §4.4 entries / §1 `pi.*` | `pi.*` entry names reject for non-core callers except the allow-listed `pi.notice` (`plugins.md` §4.4) | | §13 plugin state (`plugins[pluginId]` slices, extra document shard) | `h.namespace(ns, defaults)` tokens, `tx.plugins(ns)`, `tx.emit(ns, …)`, lazy seeding; extra shards stay deferred (`plugins.md` §2, §3, §4.5) | | §15 tool API (candidate) | `ToolApi` with `memo` (slot memoOnce), `waiting`/`memo` on `BeforeToolApi`, `waitingOn` on the tool slot (`plugins.md` §4.1–§4.2; `view-and-events.md` §2, §6.3) | -| §1 "install fixed built-ins internally" / registration at open | registration capabilities callable while open, before-`resume()` rule, `hold()/release()`, handler withdrawal and chain rerun (`plugins.md` §3, §5) | +| §1 "install fixed built-ins internally" / registration at open | registration capabilities callable while open, before-`resume()` rule, `quiescent()/hold()` for idle reload and `suspend()/reopen` otherwise; no handler-withdrawal machinery in v1 (`plugins.md` §3, §5) | | §16 `pi.plugin` handlers map | folded into custom task kinds; `pi.plugin` may stay as a convenience kind but is not the plugin mechanism | Everything else here (authority explanation, proxy lifetime, prospective busy, @@ -262,11 +266,11 @@ Cancellation only removes/rejects that waiter; it does not cancel durable work. Keep the rendering-friendly `ConversationView` and Chord document ops, but implement the settled watch behavior: - capture and subscription are one line operation; -- committed envelopes are delivered in commit-`Seq` order; +- committed envelopes are delivered in commit order with a contiguous per-watch `revision` (storage `Seq` is not exposed); - one commit produces one envelope folded into the view before the listener runs; -- listener execution is off-line; +- listener execution is off-line, synchronous and in order; - listener throw or decoder failure closes only that watch and reports through `onError`; it never rejects the already-persisted writer; -- bounded capacity (default 256); overflow closes the watch; +- envelopes before `start()` are buffered with a bounded capacity (default 256); overflow closes the watch; no other queue exists in the raw watch; - no cursor, replay, acknowledgement, deduplication, or resnapshot protocol; the client opens a fresh watch; - `stop`/unsubscribe is idempotent; - no callbacks after close/stop. @@ -352,7 +356,7 @@ Make the standalone `tsconfig` portable: no `/home/claude` paths, exclude design Keep fixed core document fields and the flat typed `c.config` facade. - Derive/seed defaults from registered kind definitions rather than duplicating them in `defaultRewindable`/`defaultSticky` and kind configs. -- Deleting/unsetting a key restores its declared default; only model remains optional among built-ins. +- Resetting a key is an explicit operation (`config.reset(keys)`; RPC `configReset`), never assigning `null`: storage deletes the override, the resolved view sets the declared default; only `model`, which has no default, is deleted from the view. - Do not use `value ?? default`, because durable `null` is a legitimate value. Test property presence. - Custom kind config defaults must be visible both through `c.config` and to the task itself before any explicit set. - Reject config-key collisions at compile time where practical and always at installation. diff --git a/packages/agent/docs/pico/v3/plugins.md b/packages/agent/docs/pico/v3/plugins.md index 1b47cc613..51078af43 100644 --- a/packages/agent/docs/pico/v3/plugins.md +++ b/packages/agent/docs/pico/v3/plugins.md @@ -23,8 +23,11 @@ Read order: §1–§5 are the harness alone (no Chord); §6 is the Chord layer; touch `turn`/`inbox`, resolve inputs, create core tasks, or write outside their namespace or their own task's slot. Enforced at runtime on every method, by type at the surface. -5. **Everything an extension does is visible through the view** (`plugins[ns]`, - `tasks[id]`, `turn.tools[i]`, entries) and nowhere else. +5. **An extension's presentation is only what it projects.** The view shows a + namespace's declared `view` projection (nothing by default), a kind's + `describe()` output, the core-rendered tool slot fields, and entries. Raw + namespace state (caches, keys, secrets) and slot working state (memos, + idempotency evidence) never appear in the view. ## 2. Durable primitives @@ -40,10 +43,15 @@ Rules: - **Checkpoint** is the kind's recovery program counter, nothing else. It is not a place for working state (that is the slot) or long-lived state (namespace). - **Slot** is working state that should die with the task: progress, streamed - output, and **memos**. `memoOnce(slot, name, candidate)` is one operation: - `slot.memos[name] ??= candidate` inside one commit; it returns the durable - winner to every caller. It is the pico3 equivalent of the lane harness's - invocation memo and of Codex's item completion record. + output, and **memos**. `memoOnce(slot, key, candidate)` is one operation inside + one commit: if `Object.hasOwn(slot.memos, key)` return the stored value, else + store `candidate` and return it (`??=` is wrong: `null` is a valid value). It + returns the durable winner to every caller. A memo is durable coordination and + evidence; it is **not** exactly-once execution by itself. For an external + effect, persist the idempotency key *before* the effect and replay the effect + with the same key, relying on the external side to dedupe or to answer a + result lookup; persist a result *after* an effect only when a duplicate is + otherwise harmless. Slot contents are never in the view. - **Namespace slice** defaults are declared at registration, per document, and seeded lazily (`plugins[ns] ??= defaults` on first access). Choose the document by semantics: rewindable when a fork should see the value as it was @@ -73,18 +81,22 @@ the harness is open (§5 for what happens then). h.registerTool(declaration: ToolDeclaration): () => void h.registerSection(section: SystemSection): () => void h.registerTaskKind(kind: Kind): () => void // ordinary kinds only; config keys checked for collisions -h.namespace(ns: string, defaults: T): Namespace // token; unique, not `pi.*` -h.hooks(kind, handlers: Partial>, opts?): () => void // harness-wide -c.hooks(kind, handlers, { subtree? }): () => void // one conversation (+ owned) -defineEntry(kind: string): EntryKind // stateless; no registration +h.namespace(ns: string, defaults: T, opts?: { view?: (slice: T) => JsonValue }): Namespace + // token with idempotent `unregister()`; unique, not `pi.*`; no `view` ⇒ nothing in the view +h.hooks(ns: Namespace, kind, handlers: Partial>, opts?): () => void // harness-wide; bound to the namespace (§4.2) +c.hooks(ns, kind, handlers, { subtree? }): () => void // one conversation (+ owned) +defineEntry(kind: string): EntryKind // stateless; throws on `pi.*` (§4.4) h.conversation(id, ctx): Promise -h.hold(): () => void // §5 +h.quiescent(): boolean; h.hold(): () => void; h.suspend(ctx): Promise // §5 ``` Token identity: the object passed to `tx.plugins(ns)`, `tx.emit(ns, …)`, -`tx.createTask(kind, …)`, `h.hooks(kind, …)` must be the currently registered -one; a redeclared or unregistered token rejects with `StaleDefinition`. This is -what makes reload safe: code from a retired generation cannot keep writing. +`tx.createTask(kind, …)`, `h.hooks(ns, kind, …)` must be the currently +registered one; a redeclared or unregistered token rejects with +`StaleDefinition`. The namespace *string* is what persists (memo keys, state +slices); the token is only current runtime authority. This is what makes reload +safe: code from a retired generation cannot keep writing, and a replacement +registration of the same string finds the same state and memos. Before `resume()`: task kinds whose tasks may exist in storage, and hook handlers that recovery reruns (`beforeTool`, `systemInstructions`, @@ -98,8 +110,8 @@ Inside a commit, extensions get: ```ts tx.plugins(ns): T // typed slice in the declared document(s), tracked proxy tx.emit(ns, name: string, data: JsonValue): void // -> { type: `plugin.${ns}.${name}`, data } in this commit's envelope -tx.slot(task): Slot // own task only (kinds); tools/hooks use api/info variants -memoOnce(tx.slot(task), name, candidate) // first writer wins +tx.slot(task): Slot // own task only (kinds); tools/hooks use the api variants +memoOnce(tx.slot(task), key, candidate) // first writer wins; hasOwn semantics (§2) tx.write(conversationId, entryKind, input): Promise tx.createTask(kind, input, opts): TaskRef // ordinary kinds ``` @@ -136,10 +148,13 @@ inside `pi.tool`, whose `started{call}` checkpoint is the effect evidence. How memos and `replay` interact: `beforeTool` and the `started` checkpoint happen before `execute`; a crash after `started` re-invokes only `replay: -"safe"` tools. A safe tool that performed a keyed external effect records the -key or result as a memo *before* the effect and reads it first on re-invocation, -so the effect is not repeated. The memo is retired with the task, so it cannot -leak into a later call with the same id. +"safe"` tools. A safe tool that performs a keyed external effect records the +idempotency key as a memo *before* the effect and reuses it on re-invocation; +the external side dedupes or answers a result lookup for that key. Recording a +result *after* the effect (as the question tool does) is only correct when a +repeated effect is harmless, which asking a human again is. Memos are scoped to +the tool task's slot and retired with it, so they cannot leak into a later call +with the same id. Example: a question tool that waits for a human. `ask` is whatever the host wires in (§6.2 wires a Chord dialog service; a test wires a promise). @@ -179,10 +194,15 @@ interface HookApi { readonly kind: string; readonly taskId: Id; readonly convers interface BeforeToolApi extends HookApi { readonly callId: string; - waiting(on: string): void; // kernel sets turn.tools[i].waitingOn = on until the handler returns (any way) - memo(name: string, candidate: T, ctx: Context): Promise; // memoOnce on the tool task's slot + waiting(ctx: Context): Promise; // commits turn.tools[i].waitingOn = ; idempotent + memo(name: string, candidate: T, ctx: Context): Promise; // memoOnce on the tool task's slot, keyed (task, namespace, name) memo(name: string, ctx: Context): Promise; + emit(name: string, data: JsonValue, ctx: Context): Promise; // plugin.. } +// Hooks are registered with their namespace token (`h.hooks(ns, kind, handlers)`), so `waiting`, `memo` and +// `emit` need no namespace argument and two plugins cannot collide on a memo name. `waitingOn` is cleared by the +// kernel atomically with `started` (handler allowed), with the synthetic result (handler blocked or threw), or in +// its own commit when the handler unwinds on abort or suspend. interface ToolHooks { beforeTool(call: ToolCall, api: BeforeToolApi, ctx: Context): { call?: ToolCall; block?: string } | void | Promise<…>; @@ -193,30 +213,33 @@ interface ToolHooks { ``` Durability: a hook has no task of its own. State tied to the invocation it is -running in goes in the **hosting task's slot** via the point's `api.memo` -(retired with that task, no cleanup code); state that outlives tasks goes in a -**namespace**. -Hooks commit through `h.conversation(api.conversationId, ctx)` handles and may -wait as long as they like; `beforeTool` handlers rerun after a crash (they run -before `started`) and after withdrawal (§5), which is why their decision must be -a memo. +running in goes in the **hosting task's slot** via the point's `api.memo`, +keyed by `(hosting task, namespace string, name)` so it survives recovery and +re-registration and cannot collide with another plugin's memo (retired with +that task, no cleanup code); state that outlives tasks goes in a **namespace**. +There is no hook invocation id: nothing durable identifies "the same handler +invocation" across a crash or a reload, and the rerun must find the earlier +decision. Hooks commit through `h.conversation(api.conversationId, ctx)` +handles and may wait as long as they like; `beforeTool` handlers rerun after a +crash or a suspend/reopen (they run before `started`), which is why their +decision must be a memo. Example: tool approval. ```ts export function installApproval(h: Harness, ask: (req: ApprovalRequest, ctx: Context) => Promise): () => void { - const policy = h.namespace<{ autoAllow: string[] }>("approval", { sticky: { autoAllow: [] } }); // long-lived: which tools skip approval + const ns = h.namespace<{ autoAllow: string[] }>("approval", { sticky: { autoAllow: [] } }); // long-lived: which tools skip approval; no `view` ⇒ private - return h.hooks(kinds.tool, { + return h.hooks(ns, kinds.tool, { async beforeTool(call, api, ctx) { const c = await h.conversation(api.conversationId, ctx); - const auto = await c.commit((tx) => tx.plugins(policy).autoAllow.includes(call.name), ctx); + const auto = await c.commit((tx) => tx.plugins(ns).autoAllow.includes(call.name), ctx); if (auto) return; - let d = await api.memo("approval", ctx); // decided before a crash / reload? + let d = await api.memo("decision", ctx); // decided before a crash / reload? key = (task, "approval", "decision") if (d === undefined) { - api.waiting("approval"); // view: turn.tools[i].waitingOn = "approval"; event tool.waiting + await api.waiting(ctx); // view: turn.tools[i].waitingOn = "approval"; event tool.waiting const candidate = await ask({ conversationId: c.id, callId: call.id, taskId: api.taskId, name: call.name, args: call.arguments }, ctx); - d = await api.memo("approval", candidate, ctx); // first writer wins + d = await api.memo("decision", candidate, ctx); // first writer wins } return d.decision === "deny" ? { block: `denied${d.by ? ` by ${d.by}` : ""}` } : undefined; }, @@ -226,8 +249,10 @@ export function installApproval(h: Harness, ask: (req: ApprovalRequest, ctx: Con No `afterTool` cleanup: the memo lives in the tool task's slot and is retired with it. Late joiners see `waitingOn` in the snapshot. Two clients deciding at -once both get the memo winner. Abort unwinds `ask` through `ctx`; the kernel -clears `waitingOn` when the handler returns for any reason. +once both get the memo winner. Abort or suspend unwinds `ask` through `ctx`; +the kernel clears `waitingOn` when the handler returns for any reason. Asking a +human twice after a crash between `waiting` and the memo is harmless, which is +why a post-effect memo is acceptable here. ### 4.3 Custom task kinds @@ -266,10 +291,10 @@ export const indexer = defineTask scan(task, task.checkpoint.cursor, rt, ctx), - // in-flight: only entered after a crash. The batch key is the memo that tells us whether the write landed. + // in-flight: only entered after a crash. The batch key was persisted before the effect; the write is + // repeated with the same key and is safe only because writeBatch(key) is idempotent on the receiving side. async writing(task, rt, ctx) { - const done = await rt.commit((tx) => tx.slot(task).memos?.[task.checkpoint.batchKey], ctx); - if (done === undefined) await writeBatch(task.checkpoint.batchKey, ctx); // idempotent by key + await writeBatch(task.checkpoint.batchKey, ctx); return { next: { phase: "scanning", cursor: task.checkpoint.batchKey } }; }, }, @@ -285,11 +310,10 @@ async function scan(task, cursor, rt, ctx) { return { status: "completed", result: { files: tx.slot(current).files } }; } }; } - await rt.commit((tx) => tx.checkpoint({ phase: "writing", batchKey: batch.key }), ctx); // before the effect + await rt.commit((tx) => tx.checkpoint({ phase: "writing", batchKey: batch.key }), ctx); // idempotency key before the effect await writeBatch(batch.key, ctx); return { next: (tx, current) => { const s = tx.slot(current); s.files += batch.files.length; s.lastFile = batch.files.at(-1); - (s.memos ??= {})[batch.key] = true; // effect evidence return { phase: "scanning", cursor: batch.key }; } }; } @@ -313,21 +337,32 @@ for (const e of view.entries) if (ci.is(e)) render(e.data.run); ``` Omit `model` for bookkeeping the model must not see. `head` and `edits` are -core-only. Placement is boundary-safe: immediately when idle, otherwise at the +core-only. `defineEntry` throws on a `pi.*` name and `c.write`/`tx.write` reject +`pi.*` at runtime for non-core callers even after a cast, with one allow-listed +exception: `pi.notice` (one user message, optional `data`, no `head`/`edits`), +which hosts, jobs and tools use for notices. No entry registry or schema is +required. Placement is boundary-safe: immediately when idle, otherwise at the next boundary (`view-and-events.md` §6.1). ### 4.5 Namespaces ```ts -const ns = h.namespace<{ planMode: { enabled: boolean } }>("plan", { rewindable: { planMode: { enabled: false } } }); +const ns = h.namespace<{ planMode: { enabled: boolean }; cache: Record }>( + "plan", + { rewindable: { planMode: { enabled: false } }, sticky: { cache: {} } }, + { view: (s) => ({ planMode: s.planMode }) }, // only this projection reaches the view; `cache` never does +); await c.commit((tx) => { tx.plugins(ns).planMode.enabled = true; tx.emit(ns, "toggled", { enabled: true }); }, ctx); // view: plugins.plan.planMode; envelope: ["s", ["plugins","plan","planMode","enabled"], true] + { type: "plugin.plan.toggled", data } +env.own(ns.unregister); // idempotent; state stays, token goes stale ``` Defaults are per document; a slice may span documents (`{ rewindable: …, sticky: … }`) and `tx.plugins(ns)` returns the merged typed object with each key routed -to its document (same mechanism as core `config`). Re-registering with changed -defaults adds new keys lazily and leaves removed keys as inert stored data. +to its document (same mechanism as core `config`). `view` is a pure projection +the kernel re-evaluates when the slice changes; without it the namespace has no +presence in the view. Re-registering the same string with changed defaults adds +new keys lazily and leaves removed keys as inert stored data. ### 4.6 Owned conversations and jobs @@ -337,36 +372,46 @@ extend them. See `pico-handoff-v2.md` §8, §10.5 and the hardening handoff §15 ## 5. Unregister, re-register, reload -Every registration can be undone and redone while the harness is open. What -happens to in-flight work: +Every registration can be undone and redone while the harness is open. | contribution | on unregister | on re-register | |---|---|---| -| namespace | token stale; `tx.plugins`/`tx.emit` with it reject | new token for the same `ns`; stored state untouched; new defaults seeded lazily | -| hooks | removed from chains; a **running** handler is cancelled through its registration ctx and the chain re-runs from the start with current handlers (`beforeTool`: safe, it precedes `started`; observers: skipped) | joins the chain | +| namespace (`ns.unregister()`, idempotent) | token stale; `tx.plugins`/`tx.emit`/`h.hooks(ns, …)` with it reject | new token for the same string; stored state and memos untouched; new defaults seeded lazily | +| hooks | removed from future chains; a handler already running continues to completion (it is cancelled only by abort or `suspend`) | joins the chain | | task kind | pending tasks of the kind are not dispatched (not orphaned; orphaning is open-time only); running invocations keep the kind object they captured | pending tasks dispatch with the new implementation; config keys re-checked | | tool / section | revision bump; in-flight invocations keep their captured declaration; preparation snapshots retry | same | | entry kind | nothing | nothing | -Handler withdrawal is what makes a reload of the approval feature safe: the old -handler's `ask` promise can never settle once its dialog is gone, so the kernel -cancels it and re-runs `beforeTool`; the new handler asks again through the new -dialog, or finds the memo if a decision had already landed. +Reload replaces code, never durable state. Two modes, chosen by the host: -`h.hold(): () => void` defers *starting* new hook chains and new task dispatch -until released; commits, streaming and running invocations continue. The -reload sequence a host must follow: +- **Quiescent reload.** `h.quiescent()` is true when the scheduler holds **no + invocation**, running or sleeping: no phase handler, no `retrying`/`deferred` + sleep, no job poll, no waiting `beforeTool`. Live task *records* may exist + (pending, or between invocations). Then: `release = h.hold()` (defers new + dispatch and new hook chains; commits and reads continue) → deactivate old + facets (their disposals unregister) → activate new ones (re-register) → + `release()`. +- **Suspend and reopen** when not quiescent (the first safe version for active + work). `h.suspend(ctx)`: stop dispatch, cancel every in-process invocation + context, await unwind, close storage; task records stay live and nothing is + terminalized. Reopen with the new registrations and let ordinary recovery run: + pre-effect phases rerun, `started` tools replay if safe and still safe, unsafe + ones become interrupted results, other phases recover per their kind. Memos + make reruns cheap (an approval already decided is not asked again). The + injected `ProcessHost` must outlive the harness instance so running jobs + reconcile through `status(key)` instead of becoming `unknown`. -```text -release = h.hold() - deactivate old facets → their env.own() disposals unregister - activate new facets → onActivate re-registers (same ns / kind names, new tokens) -release() -``` +Cancelling live calls of a tool or kind that the new generation no longer +registers is not a transparent migration: it durably aborts them, cannot undo +external effects, and post-tools observes aborted results. It is an explicit +host policy (`suspend({ abortRemoved: true })`), never the default. Generation +leases (old registrations kept alive until captured invocations drain while new +work uses replacements) are a possible later refinement; they are not required +for v1. -Registration therefore belongs in `onActivate`, not `setup`: during a -shape-preserving reload the replacement's `setup` runs before the old facet is -deactivated, and two live registrations of one namespace would collide. +Registration belongs in `onActivate`, not `setup`: during a shape-preserving +reload the replacement's `setup` runs before the old facet is deactivated, and +two live registrations of one namespace string would collide. ## 6. The Chord layer @@ -400,7 +445,7 @@ export const harnessSessionFacet = defineFacet({ await handle.open(storage, { models, processHost }, ctx); handle.onConversation((c) => { // existing at open, and every fork/subagent created later const view = env.replicatedState(c.snapshot()); // built on the line - c.attachView(view); // kernel mutates view.state and calls view.publish(ctx) on the line after each commit + c.attachView(view); // bounded ordered adapter: raw envelopes → view.state + publish(), off the line (below) env.own(conversations.spawn(String(c.id), { view, send: (i, cx) => c.send(i, cx).then((r) => r.id), /* … */ })); }); }); @@ -409,11 +454,23 @@ export const harnessSessionFacet = defineFacet({ }); ``` -`ConversationView` gains `commit: { seq, events }` so events ride inside the -replicated member (one Chord sequence = one commit); a subscriber reads -`value.commit.events` after each delivery. Chord's subscription snapshot gives -late joiners the member at its current sequence, so §9 of `view-and-events.md` -is implemented by Chord, not by us. +`ConversationView` gains `commit: { events }` so events ride inside the +replicated member; a subscriber reads `value.commit.events` after each delivery. +No revision or storage sequence is placed in the view: Chord's +`ReplicatedStateDelivery.sequence` is contiguous per member and plays that role. + +Publication must not happen on the Session line. Chord's `publish()` invokes +source and subscriber listeners synchronously and a subscriber may throw or be +slow, and `subscribe()` itself publishes pending mutations. The bridge is a +bounded ordered adapter: the kernel enqueues raw envelopes from its own view +tracker; the adapter, off the line, applies one envelope's ops to +`view.state`, sets `commit.events`, and calls `publish(ctx)` in the **same +synchronous block** (no await between apply and publish). One raw envelope = +one publish = one Chord sequence. If the queue overflows or `publish` throws, +close that keyed instance and respawn it with a fresh snapshot; a persisted Pico +commit is never turned into a failure. Chord's subscription snapshot then gives +late joiners the member at its current sequence, so §9 of +`view-and-events.md` is implemented by Chord, not by us. ### 6.2 A feature: contract + session facet + presentation facet @@ -493,31 +550,46 @@ export const indexerSessionFacet = defineFacet({ ### 6.3 What the session worker does at reload ```text -release = harness.hold() -FacetHost.reload(candidates) // old facets' env.own() disposals unregister; new onActivate re-registers -release() +if (harness.quiescent()) { + release = harness.hold() + FacetHost.reload(candidates) // old facets' env.own() disposals unregister; new onActivate re-registers + release() +} else { + await harness.suspend(ctx) // cancel invocations, close storage; task records stay live + FacetHost.reload(candidates) + await harness.reopen(ctx) // recovery runs against the new registrations +} ``` -Nothing else. Durable state, running tasks and open conversations are -untouched; withdrawn hook handlers are re-run per §5. +Durable state, task records and open conversations are untouched either way +(§5). ## 7. Tests the hardening must include -- Memo: two concurrent `api.memo` writers receive the same winner; a crash after - the memo skips `ask`; before it re-asks; the memo is gone after the tool task - terminalizes. -- Hooks: `BeforeToolApi.waiting` sets/clears `waitingOn` (also on throw and abort); - withdrawal cancels a waiting `beforeTool` and re-runs the chain; the decision - memo survives a crash between `waiting` and `started`. -- Custom kind: crash in the in-flight phase re-enters it; batch memo prevents a - duplicate write; `describe()` output appears in `tasks[id].status`; +- Memo: two concurrent `api.memo` writers receive the same winner; `null` is a + stored value (hasOwn, not `??=`); a crash after the memo skips `ask`, before it + re-asks; the memo is gone after the tool task terminalizes; two namespaces + using the same memo name on one task do not collide. +- Hooks: `await api.waiting(ctx)` sets `waitingOn` in its own commit and it is + cleared atomically with `started`, with the synthetic result on block/throw, + and in its own commit on abort/suspend; the decision memo survives a crash + between `waiting` and `started`; memos never appear in the view. +- Custom kind: crash in the in-flight phase re-enters it and repeats the keyed + effect (test with a fake idempotent sink and a crash between effect and the + next checkpoint); `describe()` output appears in `tasks[id].status`; unregistered kind leaves pending tasks pending and orphans only at open. - Namespaces: writes outside `plugins[ns]` reject; stale token rejects; - re-register keeps values and seeds new defaults; rewindable slice is inherited - by a fork and readable at an entry. -- Reload: `hold()` defers dispatch and hook chains but not commits; the - hold/reload/release sequence with the approval facet lets a pending call be - approved through the new generation. + `unregister` is idempotent; re-register keeps values and memos and seeds new + defaults; only the `view` projection appears in the view; rewindable slice is + inherited by a fork and readable at an entry. +- Entries: `defineEntry("pi.x")` throws; `write` of any `pi.*` except `pi.notice` + rejects at runtime after a cast; `pi.notice` with `head`/`edits` rejects. +- Reload: `quiescent()` is false while any invocation exists (including a + waiting `beforeTool` and a polling job); `hold()` defers dispatch and hook + chains but not commits; suspend/reopen with the approval facet lets a pending + call be approved through the new generation without asking twice when a + memo exists; `ProcessHost` survives suspend/reopen and a running job + reconciles. - Chord: loopback facet tests for approval and question (session + presentation fixtures), late-joiner hydration of an open dialog instance, and `ConversationService.view` sequence equals commit sequence. diff --git a/packages/agent/docs/pico/v3/view-and-events.md b/packages/agent/docs/pico/v3/view-and-events.md index 972e45219..1f37b04b2 100644 --- a/packages/agent/docs/pico/v3/view-and-events.md +++ b/packages/agent/docs/pico/v3/view-and-events.md @@ -10,12 +10,15 @@ Companion: `plugins.md` (harness extension surface, durability, Chord facets). 1. **One snapshot, then one envelope per commit.** A watch captures the view and subscribes in one line operation; every later commit that touches the - conversation produces exactly one envelope, delivered in commit-`Seq` order. - No cursor, replay or acknowledgement: a gap means "open a fresh watch". + conversation produces exactly one envelope, delivered in commit order with a + contiguous per-watch `revision`. No cursor, replay or acknowledgement: a gap + means "open a fresh watch". 2. **Ops are the truth, events are annotations.** The envelope carries Chord ops that transform the view document, and named events the kernel attaches because - it produced those ops and knows why. A client that folds only ops is never - wrong. A client that consumes only events never scans the view. + it produced those ops and knows why. Ops fully determine the replica: a client + that folds only ops is never wrong. Events annotate cause and outcome and may + carry transient facts the view deliberately does not retain; a client that + consumes only events never scans the view. 3. **Path = event.** Every field of the view has a stable path, so an op on that path is a typed notification. Nothing in the view is ever discovered by diffing. @@ -63,7 +66,7 @@ interface ConversationView { /** Non-turn live tasks (jobs, plugin kinds), described by their kind. Removed on terminal. */ tasks: { [taskId: string]: { kind: string; background?: true; marked?: true; status: JsonValue } }; - /** Plugin state slices, by namespace. */ + /** Plugin presentation slices, by namespace: the namespace's declared `view` projection, never raw state. */ plugins: { [namespace: string]: JsonValue }; } @@ -75,6 +78,7 @@ interface TurnView { } type GenerationStatus = + | { stage: "waiting"; on: "compaction" } // replacement generation pending behind a collapse | { stage: "preparing" } // before `prepared` | { stage: "requesting"; attempt: number } // request sent, no frame yet | { stage: "streaming"; attempt: number } // first frame applied @@ -84,13 +88,13 @@ type GenerationStatus = interface ToolSlot { callId: string; name: string; args: JsonValue; status: "pending" | "running" | "done" | "error" | "aborted"; - waitingOn?: string; // set by the kernel while a beforeTool handler called BeforeToolApi.waiting(on); cleared when it returns + waitingOn?: string; // namespace of the beforeTool handler that awaited BeforeToolApi.waiting(ctx); cleared when it returns output?: string; // bounded stream so far (api.stream) progress?: string; details?: JsonValue; continuedBy?: Id; // background task that took over; see tasks[continuedBy] entry?: Id; // the pi.tool_result entry once landed - memos?: { [name: string]: JsonValue };// tool/hook memos (plugins.md §2); retired with the task } +// Memos and other slot working state are never in the view (plugins.md §2); only the fields above are rendered. ``` ### 2.1 Active transcript @@ -120,16 +124,25 @@ interface ToolSlot { ```ts interface Envelope { - seq: Seq; // commit sequence; strictly increasing per watch + revision: number; // contiguous per watch: snapshot.revision + 1, +2, …; storage Seq is not exposed ops: Op[]; // Chord ops over ConversationView (§4) events: ViewEvent[]; // named annotations for this commit (§5); may be empty } ``` -Delivery: folded into the kernel's view tracker on the line, listener called off -the line, one envelope per commit that changed anything visible to this -conversation. A listener throw closes that watch and reports through `onError`. -`stop` is idempotent; nothing is delivered after `stop`. +`revision` is per watch, not the storage commit sequence: a conversation watch +sees only commits that touch its conversation, so the global `Seq` would have +legitimate gaps. Storage `Seq` stays internal unless a public commit-correlation +use appears. + +Delivery: folded into the kernel's view tracker on the line; the listener is +invoked **synchronously, in order, off the line**, one envelope per commit that +changed anything visible to this conversation. Envelopes arriving before +`start()` are buffered (bounded; overflow closes the watch). Asynchronous +consumers (the Chord bridge, application queues) own their own queue and +backpressure policy (`plugins.md` §6.1). A listener throw closes that watch and +reports through `onError`. `stop` is idempotent; nothing is delivered after +`stop`. Base flush: after the kernel rebases the view tracker (reopen of a cold watch is a fresh snapshot instead), an envelope may carry `["r", view]` as its only op. @@ -146,7 +159,7 @@ delete, `["a", path, str]` string append, `["t", path, n]` string truncate, |---|---|---| | `["entries"]` | `["p", ["entries"], n, 0, [e…]]` | entries appended (always at the end) | | `["entries"]` | `["p", ["entries"], 0, k, []]` then `["p", …, n, 0, [head]]` | new head: transcript truncated, head entry appended | -| `["config", key]` | `s` / `d` | config changed / reset to default | +| `["config", key]` | `s` / `d` | config changed; `config.reset(keys)` sets the declared default (only `model`, which has none, is deleted) | | `["inbox"]` | `p` | queued, placed (removed), aborted (removed) | | `["turn"]` | `["s", ["turn"], {…}]` / `["d", ["turn"]]` | input group started / ended | | `["turn","generation"]` | `s` / `d` | generation stage changes / generation ended | @@ -155,14 +168,13 @@ delete, `["a", path, str]` string append, `["t", path, n]` string truncate, | `["turn","message","content",i,"arguments"]` | `s` | tool-call arguments materialized at checkpoint/end | | `["turn","tools"]` | `s` (whole array, with the assistant entry) | tool slots created for a new assistant message | | `["turn","tools",i,"status"]` | `s` | pending → running → done/error/aborted | -| `["turn","tools",i,"waitingOn"]` | `s` / `d` | a `beforeTool` handler is waiting on a human/service (approval, question); cleared when it returns | -| `["turn","tools",i,"memos",name]` | `s` | a memo was recorded (`api.memo` / hook memo) | +| `["turn","tools",i,"waitingOn"]` | `s` / `d` | a `beforeTool` handler (namespace = value) is waiting on a human/service; cleared when it returns | | `["turn","tools",i,"output"]` | `s` | bounded stream flush (whole string; retention may drop the head) | | `["turn","tools",i,"progress"]`, `[…,"details"]`, `[…,"continuedBy"]` | `s` | `api.progress` | | `["compaction"]` | `s` / `d` | collapse started / stage changed / ended | | `["tasks", id]` | `s` / `d` | non-turn task appeared / status changed / terminal | | `["tasks", id, "status", …]` | any | kind-described status detail (jobs: stdout, exitCode, …) | -| `["plugins", ns, …]` | any | plugin state | +| `["plugins", ns, …]` | any | the namespace's `view` projection | Rule for kernel authors: mutate the tracked view through its proxy so the tracker records minimal ops; never rebuild subtrees. Streaming applies each @@ -171,9 +183,12 @@ frame to `turn.message` in place (same reducer as pi-ai's frame reducer). ## 5. Events Events are emitted by core code inside the commit that causes them -(`tx.emit(event)`, core authority). Plugin kinds may emit -`{ type: "plugin..", … }` from their own commits. Every event is -reconstructible from the same envelope's ops; the reverse is not required. +(`tx.emit(event)`, core authority). Extensions emit `{ type: "plugin..", … }` +through their namespace token from their own commits. Rule: ops fully determine +the replica; events annotate the cause and outcome of the commit and may carry +transient facts the view does not retain (`turn.ended.reason`, +`task.ended.outcome`, `compaction.failed`, `warning`). An event must never be the +only place a late joiner could learn something needed to render steady state. ```ts type ViewEvent = @@ -194,7 +209,7 @@ type ViewEvent = | { type: "generation.completed"; taskId: Id; entry: Id; toolCalls: number } | { type: "generation.failed"; taskId: Id; reason: "provider" | "overflow" | "retries_exhausted" | "no_model"; detail: string; entry?: Id } // tools - | { type: "tool.waiting"; taskId: Id; callId: string; on: string } // a beforeTool handler is waiting (approval, question, …) + | { type: "tool.waiting"; taskId: Id; callId: string; on: string } // a beforeTool handler is waiting; `on` is its namespace | { type: "tool.started"; taskId: Id; callId: string; name: string } | { type: "tool.finished"; taskId: Id; callId: string; entry: Id; isError: boolean; control?: ToolControl } | { type: "tool.aborted"; taskId: Id; callId: string; entry: Id } @@ -212,7 +227,7 @@ type ViewEvent = | { type: `plugin.${string}`; data: JsonValue }; ``` -`warning` is the one event with no state counterpart; it exists for exactly the +`warning` has no state counterpart at all; it exists for exactly the occurrences Codex models as `warning` / `error{willRetry}` notifications. ## 6. Commit-by-commit mapping @@ -245,18 +260,19 @@ Ops are abbreviated; every row is one atomic envelope. | final answer, no triggers, no continuation (closure) | `entries +pi.assistant`; `turn` deleted; boundary placements (`inbox` splices, `entries +writes`) | `entry.added`, `generation.completed`, `turn.ended{done, answer}`, `input.placed`/`entry.added` for placed writes | | final answer, `onYield` continuation | `entries +pi.assistant`, `entries +pi.user{continuation}`; `turn.generation = { stage: "preparing" }` | `entry.added`×2, `generation.completed`; **no** `turn.ended` (same group continues) | | final answer with queued followUp/steer triggers | as "no triggers", then `entries +pi.user`, `turn = { inputs: triggers, tools: [] }`, `turn.generation = { stage:"preparing" }` | `turn.ended{done}`, `input.placed`, `entry.added`, `turn.started` | -| terminal error / aborted-by-provider (closure) | `entries +pi.assistant` (display-only; `model` absent, message in `data`); `turn` deleted | `entry.added`, `generation.failed{provider|retries_exhausted, entry}`, `turn.ended{unanswered, failed}` | -| `no_model`, overflow with nothing collapsible | `turn` deleted; overflow: `entries +pi.notice` (placed immediately: the closing task counts as gone) | `generation.failed`, `turn.ended{unanswered, failed}`, `entry.added` | -| overflow with collapse (closure) | `compaction = { reason: "overflow", … }`; `turn.generation = { stage: "preparing" }` (replacement created `after` collapse) | `generation.failed{overflow}`, `compaction.started`; turn continues | +| terminal error / aborted-by-provider (closure) | `entries +pi.assistant` (display-only; `model` absent, message in `data`); group resolved `unanswered/failed`; **final boundary runs**: placed writes, selected triggers; `turn` deleted, or replaced by a successor turn for triggers | `entry.added`, `generation.failed{provider|retries_exhausted, entry}`, `turn.ended{unanswered, failed}`, then `input.placed`/`entry.added`/`turn.started` as in "with triggers" | +| `no_model`, overflow with nothing collapsible | as above; overflow additionally `entries +pi.notice` (placed immediately: the closing task counts as gone). `no_model` is not special-cased: a successor for queued triggers fails fast and resolves its own group | `generation.failed`, `turn.ended{unanswered, failed}`, `entry.added`, … | +| overflow with collapse (closure) | `compaction = { reason: "overflow", … }`; `turn.generation = { stage: "waiting", on: "compaction" }` (replacement created `after` collapse) | `generation.failed{overflow}`, `compaction.started`; turn continues | | abort (fresh abort closure) | optional `entries +pi.assistant` (display-only partial, only if content was streamed); `turn` deleted | (`entry.added`), `turn.ended{unanswered, aborted}` | ### 6.3 Tool | moment | ops | events | |---|---|---| -| `beforeTool` handler calls `api.waiting(on)` (`BeforeToolApi`, `plugins.md` §4.2) | `turn.tools[i].waitingOn = on` | `tool.waiting{callId, on}` | -| handler returns / is withdrawn | `turn.tools[i].waitingOn` deleted | none | -| `started` checkpoint (after `beforeTool`, before invocation) | `turn.tools[i].status = "running"` | `tool.started` | +| `beforeTool` handler awaits `api.waiting(ctx)` (`BeforeToolApi`, `plugins.md` §4.2) | `turn.tools[i].waitingOn = ns` (its own commit) | `tool.waiting{callId, on: ns}` | +| `started` checkpoint (after `beforeTool` allowed, before invocation) | `turn.tools[i].waitingOn` deleted; `turn.tools[i].status = "running"` (one commit) | `tool.started` | +| handler blocks/throws | `waitingOn` deleted atomically with the synthetic error result below | | +| handler unwinds on abort/suspend | `waitingOn` deleted in its own commit | none | | stream flush (≤ every 100 ms) | `turn.tools[i].output = text` | none | | `api.progress` | `turn.tools[i].{progress,details,continuedBy}` | none | | result (closure) | `entries +pi.tool_result`; `turn.tools[i].status = done|error`; `turn.tools[i].entry` | `entry.added`, `tool.finished{isError, control}`, (`warning` if truncated) | @@ -287,8 +303,13 @@ Ops are abbreviated; every row is one atomic envelope. | hook-supplied summary | same as "summary lands" without a provider stage | same | Manual collapse (`c.collapse`) is `background`; it appears in `compaction` -exactly the same way. `NothingToCollapse` rejects the call and produces no -envelope. +exactly the same way. At most one live collapse and exactly one live generation +per conversation are core invariants (rejected before persistence, never +generalized to maps in the view): a manual collapse while one is live rejects +`CollapseInProgress`; a threshold trigger while one is live is skipped; +`NothingToCollapse` rejects the call and produces no envelope. An overflow +replacement generation waits behind its collapse as `turn.generation = { stage: +"waiting", on: "compaction" }`. ### 6.6 Jobs and plugin kinds @@ -390,11 +411,11 @@ own ops. Project image blobs out of `entries[].model` there, not in the kernel. document through its proxy, `flush()`es once, attaches the commit's events, and hands the same `Envelope` to every watcher. Fan-out cost is per conversation; the kernel never knows how many clients exist or what they render. -- `watch(ctx)` returns `{ view, start, stop }` where `view` is a - `structuredClone` of the tracker target taken **on the line**, tagged with the - current commit `seq`, and the subscription is installed in the same line - operation. The first envelope a watcher receives is therefore exactly - `view.seq + 1`; there is no window in which a commit can land between snapshot +- `watch(ctx)` returns `{ view, revision, start, stop }` where `view` is a + `structuredClone` of the tracker target taken **on the line**, `revision` is + this watch's starting revision, and the subscription is installed in the same + line operation. The first envelope a watcher receives is therefore exactly + `revision + 1`; there is no window in which a commit can land between snapshot and subscription. ### 9.2 Late joiners @@ -407,7 +428,7 @@ in progress is state and is in the snapshot. | assistant message streaming | `turn.message` (partial), `turn.generation.stage === "streaming"` | | provider backoff | `turn.generation = { stage: "retrying", retryAt, lastError }` | | tool running | `turn.tools[i].status === "running"`, `output` so far | -| tool waiting for approval / a question | `turn.tools[i].waitingOn` (+ the plugin's own Chord service instance, `plugins.md` §6) | +| tool waiting for approval / a question | `turn.tools[i].waitingOn === ns` (+ the plugin's own Chord service instance, `plugins.md` §6) | | compaction | `compaction` | | background job | `tasks[id].status` | | queued follow-ups | `inbox` | @@ -418,10 +439,12 @@ and still cannot show "retrying". ### 9.3 Reconnect -A client keeps `lastSeq`. If an envelope arrives with `seq !== lastSeq + 1` -(only possible on a transport drop between a Chord RPC service and its client), -the client discards its replica and opens a fresh watch. Nothing is -client-specific, so there is nothing to resume, acknowledge or replay. +A client keeps `lastRevision`. If an envelope arrives with +`revision !== lastRevision + 1` (only possible on a transport drop between a +Chord RPC service and its client; in that transport Chord's own +`ReplicatedStateDelivery.sequence` plays this role), the client discards its +replica and opens a fresh watch. Nothing is client-specific, so there is nothing +to resume, acknowledge or replay. ### 9.4 Concurrent decisions @@ -447,9 +470,12 @@ namespaces), where its durable state lives, how it appears in this view, and how it is packaged as Chord facets is specified in `plugins.md`. The view-facing rules are: -- extension state appears only as `plugins[ns]` (namespace slices), - `tasks[id].status` (kind-described), `turn.tools[i].{waitingOn,memos}` and - `tasks[id].memos` (task slots), and entries; +- an extension's presentation is only what it projects: the namespace's + declared `view` projection under `plugins[ns]` (nothing by default), the kind's + `describe()` output under `tasks[id].status`, the core-rendered tool slot fields + (`status`, `waitingOn`, `output`, `progress`, `details`, `continuedBy`, `entry`), + and entries. Raw namespace state, memos and other slot working state never + appear; - extension events are `plugin..*`, emitted with the namespace token from the extension's own commits; core events remain core-only; - nothing about Chord services is visible in the view. A keyed service instance @@ -468,8 +494,14 @@ rules are: records by kind-owned `describe(task)`; the raw records never leave the kernel. - Display-only assistant entries carry the message in `data.display` and no `model`, so context derivation needs no special case. -- Envelope assembly happens on the line; listener dispatch off the line; a - listener throw closes that watch only. +- Envelope assembly happens on the line; listener dispatch off the line, + synchronous and in order; a listener throw closes that watch only. +- Chord bridge (`plugins.md` §6.1): a bounded ordered adapter applies one raw + envelope's ops to the Chord `MutableReplicatedState` and calls `publish()` in + the same synchronous block (Chord's `subscribe()` publishes pending mutations + itself, so apply and publish must not be separated by an await); overflow or + publish failure closes and respawns that instance; a persisted commit is never + failed by a listener. - Tests: for every row in §6, assert the exact ops and events of that commit (memory backend, fake provider), plus snapshot-equals-fold after every envelope and a head-in-parent capture case. @@ -477,6 +509,8 @@ rules are: ## 12. Open items - `warning` payload shape (source enum vs free string). +- Application-level backpressure for slow asynchronous consumers is TBD; the + raw watch itself is not (synchronous listener, bounded pre-start buffer). - Whether `turn.tools` should keep finished slots of the previous assistant message when the continuation generation prepares (current: cleared at `prepared`; Codex keeps all items of the turn). Recommendation: keep them, keyed