StoreStorage (Durable's storage on Chord v2's store, @earendil-works/pi-durable/storage/store) passes
Durable's storage conformance suite and runs Durable's harness tests (npm run test:store). The store
gets owner names and claims, one id allocator, Durable's Storage semantics for refused steps, history
none/latest/rewindable with reads at a point, archive revisions and applyBatch. self.follow follows
Durable's synchronous observers as Chord sources. Persisted jobs: records and state in the owner's
store, restarts as Durable's tasks, abort stored, deferred state. Chord's ./next export returns for
the adapter surface.
GenerationHooks.afterRun asks, after a run answered, for durable tasks
(runEndTask(task, input)). The answer commits and the run's inputs settle
done first; the generation then creates the tasks as its children in a
finish phase and waits for them in drain, keeping pi.live.run (run.afterRun)
and the conversation busy, so idle waits include them, Stop aborts them and
a crash resumes them. Queued input is placed when they are terminal.
pi.live.run.first and runFailure.first name where a run's history starts,
also for a continue after a failure.
closes#11141
Conversation.submit(tx, draft) and ConversationHandle.submit(tx, draft) run
the same admission inside the caller's commit and return the submission ID,
so a host's own checks and writes and the admission are stored together or
not at all. Admission must come before the commit's first table write, once
per commit, and throws ReadAfterWrite otherwise on every path. A first
argument that is not a draft is taken as a transaction; one of another
Session, or a wrapped one, is refused. The handle form also rejects once its
invocation is aborted. The pi.durable.submit span covers the admission only.
closes#11149
An input with whenIdle: "reject" is refused with the new ConversationIdle
when the conversation has no run, before any write, so it never starts a
run. With whenBusy: "steer" it steers the current run or is refused.
Request-ID deduplication runs first.
Chord's shared watch: listeners take a tagged call, { kind: "value", value, ops, retired } or
{ kind: "failed", error }; a recoverable failure keeps the watch open and the next value arrives as
[["r", value]]; WatchEnd is tagged by kind, and an aborted context ends a watch "aborted".
Durable: observers take the watch's shapes (advance(call, ctx), end(end)); onReport(report) with
report.kind and report.error, session_failed is storage_failed; subscribeCommits is observeCommits;
SessionEnd and ReportFrom are tagged by kind; FileChange is tagged and file watchers stop with
stop(ctx); CompactionFailure and CompactionStatus name their trigger (pi.live version 2 with a
migrate); the task_outdated report uses the scheduler's reasons; TaskInspection's ready is pending;
DocumentCommitChange is tagged. Stored formats are unchanged.
A generation whose request or poll fetch was in flight at a host death counts
one failed attempt of the retry policy (backoff, the partial kept, the same
deferred handle polled again); without retries left the run ends with
runFailure interrupted. restartLimit ends every other task visibly instead of
parking it (tool error result, compactionFailure, runFailure restart_limit,
faulted extension tasks) and reports task_interrupted; retryTask, task_parked
and the restart_limit blocked reason are removed.
closes#11139
ConversationView.entries is a window over the fork-aware history with before;
conversation.watch(ctx, { from, min, max }) and observeConversation take the
window. ConversationTranscript merges windows into a client's transcript with
gaps, loadEarlier, fillGap and resume. pi.live.runFailure says why a run ended
without an answer. watchEvents and its event types are removed; the changelog
has the migration guide.
closes#11327
Durable publishes a stored commit inside Storage.commit's adopt: the
telemetry events, its own observers (document watches, conversation
views, task graph, scheduler, waits), and a dispatch of host
subscribeCommits listeners one microtask later, before every wait and
abort the commit triggers. A commit the storage rejects after adoption is
published before the Session fails; a throwing telemetry adapter is
reported as host_error "telemetry" instead of rejecting a durable commit.
Chord's watch listeners take one change object, { value, ops, retired }
({ value } for native watches); retirement is a flag, not an end.
Durable's document watches follow their address, and a task document's
watch ends after its retirement call. New synchronous observers for
adapters: observeDoc, observeConversation, observeTaskGraph; the watches
are built on them. The env file watcher's change type is renamed
FileChange.
A commit may be marked deferrable ({ persistence: { kind: "deferred", withinMs? } }) on Session, Conversation,
TaskRuntime and a tool's api.commit. When every write changes an existing conversation document with history
"latest" and fork "initial", it is adopted in memory and shown to watches at once, and stored with the next commit
that reads or writes the document, before the document leaves memory, at close, or within withinMs (default
settings.progress.persistWithinMs, 1000 ms). The generation partial throttle and the tool output throttle use it.
subscribeCommits() gets these commits only once stored, with exactly what was stored; internal observers get
memory-only publications. Spec §1/§13 changed, new §4.1. 32 streams over SQLite: storage transactions 562/s -> 99/s,
disk 14.3 -> 4.9 MB/s, CPU 29% -> 22% of one core. A crash loses up to about 1 s of streamed content.
- chord/delta: PendingOps (push, size, take -> PendingTake): raw pending ops, r drops everything
before it, a leaf index merges appends and primitive sets per path (cleared by any other op),
past 10,000 ops only "replace with the current value"; one batch is kept by reference and
received ops are never mutated
- chord: Watch queues frames in PendingOps while its listener is busy or not started and delivers
one call with the newest value, the folded ops (or [["r", newest]] after an overflow) and the
newest frame's context; replaces the 100-frame collapse. NativeWatch is unchanged and takes an
optional replace()
- durable: watchEvents keeps every event batch in order (NativeWatch with a snapshot replace);
document, view and task-graph watches fold; spec, README, example 04 and tests updated
- tests: fold fuzzers (tracker, appendText, splices/reorders, root arrays, diffRevisions) with
codec round trips, adversarial cases, cap, ownership, cost and GC guards
HarnessOptions.readOnly opens a viewer of a session another process drives, such as a synced copy on a device that only shows it. Open leaves running tasks as committed, commits with writes reject with ReadOnlySession before they reach Storage, and resume() and every call that would run tasks reject with it.
Breaking changes:
- Durable Storage.commit(writes, adopt, context): storages call adopt(seq) once,
synchronously, when the batch is durable and before their next commit
- Chord traceIn(options, fn, context): context last
- WatchHandle and WatchEnd move to Chord (Durable re-exports them); end reasons
ended (was session_closed) and failed (was session_failed); retirement is an
explicit source event, and an ended watch drops its references into the Session
- Removed Session.documentState(), Conversation.viewState(), Harness.taskGraph()
and DocumentState; use watchDoc(), watch(), watchTaskGraph()
Every built-in storage implements ReplayableStorage: head(), replay() at an explicit sequence, and export() of the current state as a log that replays to the same reads. SQLite schema version 3 adds sequence indexes so export streams. @earendil-works/pi-durable/sync adds ReplicatingStorage, which appends each committed batch to an outbox with fork copies resolved, compactRecords(), and versioned JSON log lines.
- ClientAttribution is { name, url, categories? } with required name and url, so
OpenRouter always gets an HTTP-Referer. identity.ts maps the fields to each
provider's headers; NVIDIA gets the name with its first letter upper-cased.
attribution: null turns it off. PI_ATTRIBUTION is pi's, including the
cli-agent category; agent-core, durable, and the coding agent use it.
- Categories are a string array, joined for OpenRouter.
- withClientIdentity() is a typed wrapper instead of a Proxy and keeps a
MutableModels mutable. Durable no longer caches wrappers in a WeakMap.
- The Codex originator stays pi; login agentName no longer follows attribution.
- Models and the coding agent's ModelRuntime add x-opencode-session for custom
providers pointing at OpenCode, replacing the coding agent's transformHeaders.
pi-ai gets a ClientIdentity request option and createModels() default that choose the User-Agent (detailed with version, short, or custom) and provider attribution headers for OpenRouter, NVIDIA NIM, and OpenCode. Attribution is off by default in pi-ai and on by default in pi-agent-core and pi-durable (HarnessOptions.identity). pi-coding-agent sets its identity from the telemetry setting instead of adding headers itself.
Breaking change: defineDoc and defineDocFamily now require an explicit checkpointWhen predicate; no default is supplied. Set policies on built-in documents, examples, tests, and triage state, and document the required API.
- pi-ai: AI_TELEMETRY_SCHEMA; pi.ai.request and pi.ai.poll spans around Models requests via a lazyStream observer
(no provider changes), first_chunk and block events (never per delta), usage/cost end attributes, provider_error
status with a bounded message, aborted stays ok; duration, first-chunk, token and cost metrics; option renamed
telemetryContext -> telemetry (a parent span)
- pi-telemetry: TelemetryContext carries readonly metrics and propagation capabilities on every span
- chord/context: Context.telemetry is always a TelemetrySpan (NOOP default); withTelemetry takes spans
- durable: DURABLE_TELEMETRY_SCHEMA; pi.durable.submit (inputs start their own trace linked to the caller) and
pi.durable.phase spans for every task phase; stored carriers on submissions and tasks (TaskOptions.trace, links);
entry, submission, checkpoint, replay and report events after commit; resumed tasks stay in their trace; phase
duration, active task and report metrics; failed phases carry fixed categories with bounded messages
onReport info is tagged by type (session_failed, task_faulted, task_parked,
migration_failed, task_outdated, task_error, extension_error, host_error,
listener_error). Task reports carry a ReportTask; restart-limit and outdated
definition reasons are typed fields instead of error.cause strings; host_error
names the failing HarnessOptions callback.
- A close writes the tasks it stopped back as pending, so only a crash counts a restart.
- An abort mark starts the count over; an abort handler cut off restartLimit times ends the task faulted.
- Only a changed state resets the count; raised and lowered limits reach the scheduler and the graph.
- wake() stays active while a reconcile pass is scheduled or underway; failed migrations recompute it.
- Reports name the extension whose code failed (hooks, wrappers, sections, tools, a task's installer),
runtime.report() takes a source, and the parked report carries blocked: restart_limit.
- The task graph keeps only what blocked reasons read, and shows abandoned tasks waiting for a definition.
- Open counts restarts per task; at settings.restartLimit (default 5) a task is parked as blocked
(restart_limit) until Harness.retryTask().
- onReport(error, info) with ReportInfo: session, task (faulted), extension, listener, host, timer, wake.
Task faults are reported.
- settings.afterFailedRun: keep, continue, or withdraw the inputs queued behind a failed run.
- TaskGraphNode.blocked follows registry changes and failed migrations.
- README: fallback alarm before Harness.open in the Durable Object example.
The README example and the test worker arm the recovery alarm before resume() in the constructor and in
alarm(). Timers should not fire early, and callers recheck; nativeTimer no longer needs a non-null assertion.
Due sleeps are picked up at every scheduling pass, so a pass that drops the timer cannot strand them.
Code mode drops a global's result after the script finished instead of reading an ended invocation's clock.
A foreground timer the host declines falls back to setTimeout; the default timer never fires early; Progress
rechecks its clock; an undelivered onWake is offered again. Spec, README, and tests follow.
Runs the storage conformance suite, the schema migration, an alarm-woken sleep across an eviction, and a
turn recovered through the recovery alarm in workerd via @cloudflare/vitest-plugin (npm run test:workers).
fix(durable): decline background timers in Cloudflare Workers. With nodejs_compat, timers have an unref(),
so the idle-memory timers were armed and a pending one kept a Durable Object from hibernating or being evicted.
Tasks end a phase asleep with runtime.sleepUntil() or { status: "sleeping", until, checkpoint }:
no invocation is held, the deadline survives restarts, and the task resumes at its checkpoint once due.
Generation retry/poll and compaction retry sleep this way. Harness.wake() and HarnessOptions.onWake let
a host such as a Durable Object wake the process itself. HarnessOptions.setTimer arms every Harness
timer, and chord Progress paces on an injectable clock. SQLite migrates to schema version 2 to admit
the sleeping status.
An environment may list the files, directories and symbolic links below a
directory: relative / paths, depth first in name order, without .git and
what git ignores in work trees, bounded with a truncated flag. Callers fall
back to their own walk when it is missing or fails with not_supported.
The env conformance suite checks it with the new listFiles option.
Chord: README rewritten for the new runtime; notes.md and requests.md removed, their open limits and
decisions folded into spec.md section 12; guide and spec status lines updated. Durable: drops the
pico-v5 handoff and delta findings, renames the Pico spec, and uses current Chord terms. Delta README,
issue categories, and PI_RADIUS_GATEWAY updated.
- ToolRegistration.deferred: not offered to the model until modelTools names it
- modelTools gains { remove?, add? }; control.addTools writes names to add, so
loaded tools follow the others in load order and a load only appends
- Agent.deferred lists deferred tools not loaded yet
- @earendil-works/pi-durable/tool-search: the coding agent's tool_search (BM25)
- the codemode section leaves deferred tools out
- structuredOutput is always the tool's own value; a schema tool's error text
without one becomes a tool_error diagnostic on nested results
- code mode: a schema tool's error result with its value resolves (bash nonzero
exit), schema-less tools resolve with their text, as in the coding agent
- codemode prompt section with tool declarations by namespace within an inline
budget; searchTools/describeTool/describeNamespace from the start data
- result layout of the coding agent: header, text markers, console block,
truncation within max_output_tokens and the tool's output limits, spill and
image files in a recorded output directory (a rerun overwrites them)
- models.*: catalog globals; classify/generateImages as hidden internal tools
(nested calls with usage), at most four at once
- Lark grammar of the script source as constrainedSampling
A host that closes its CodemodeSandbox before the Harness, as on a graceful
shutdown, made the running code mode call store "Script error (aborted):
Sandbox closed" as its result, so the script never resumed. A run the sandbox
aborts without the call asking for it now stores nothing and ends with the
call's invocation; the next process replays the script.
A codemode tool whose scripts call the conversation's tools as nested calls and
survive a restart. The script runs deterministically (pi-codemode start()); each
result is journaled before it is delivered. A rerun starts the script again from
the stored start data, delivers the journaled results in order (nested calls
reattach by key c<id>), checks every recorded call count, holds calls the
journal lacks until replay ends and checks them against earlier attempts' keys,
then continues live. A call launches only once the count of the drain that made
it is committed, so a rerun can prove which calls are new. Divergence, a changed
module version or sandbox fingerprint, and a passed deadline end the call with a
stored error; a decided outcome is returned without a VM. store()/load() keep
values per conversation, rewindable and forked as of the fork's entry.
pi-codemode: CodemodeRun.finished, set when the outcome is decided.
Programs calling a tool with structuredOutputSchema got isError and no text when
its error result omitted structuredOutput; they now get its bounded output,
collapsed like a schema-less tool's, so a code mode script sees the error text.
A thrown error's message stays in its tool_error diagnostic.
A generation waiting on a blocking threshold compaction sent its request
whatever the outcome. When summarization failed, the context was still above
the blocking threshold: a session at 996K of a 1M window got an empty length
answer on every turn and nothing showed the error. The generation now fails
with model_error (Compaction failed: <error>) when its compaction failed,
faulted, was orphaned, or was aborted directly; a compaction that found
nothing to compact or was declined still lets the request go ahead.
pi.live.compactionFailure keeps the newest failure (task, reason, error)
until a compaction produces a summary. The agent events' snapshot carries it,
and compaction_end carries error for a failed compaction.
- Input submissions take entry: { kind, data } (InputEntry): placed as that
kind with that data and the content as user message, instead of pi.user.
Kinds must be non-empty strings outside pi.*. Queued inputs carry it in
their pi.inbox item.
- Writes with only kind and data (no model, head, edits) are appended and
settled done at once, busy or not; only a queued reset holds them back.
- Experimental hosts render such inputs as user messages and title by them.
closes#10407
createNodePhotonImages() now runs Photon in a worker thread, so preparing a large photo (a second or two) no longer blocks the event loop: in the test a 6-megapixel image blocked the loop over 200 ms inline and under 100 ms with the worker. The worker keeps the process alive only while it prepares an image, exits after 30 s without images to free Photon's WebAssembly memory, and a worker that cannot start falls back to the calling thread. Options: { worker: false, idleMs }.
The Session kept every document it loaded or created in memory for the life of the process: every conversation and
session document ever touched, built-in and third-party. Now a cached document that no watch or document state
observes leaves the cache once it was not read or written for `documentRetentionMs` (HarnessSettings and
createSession options; default ten minutes, 0 drops it right after use, Infinity keeps documents; anything that is not
a number of milliseconds means the default), and loads from Storage again on its next access. An observer pins the
incarnation it observes and releases it when that incarnation retires.
Eviction runs on the mutation line, after every commit outcome (published, no-write, rejected, failed settlement) and
from one unreferenced timer for the next expiry, so a transaction never loses a tracker it holds. Where timers cannot
be unreferenced only commits evict. Cache bookkeeping is contained: a throwing clock or setting never changes an
operation's outcome. A hot document moves in the eviction order at most once per granule (a tenth of the retention,
up to 1 s), keeping the fan-out bench at HEAD's speed.
`npm test` runs the durable suite a second time with every unattached document evicted right after use.
5,000 conversations kept 26,000 documents and about 3.8 KB each; with a 50 ms retention, after waiting, none
(bench:document-cache).
NestedCallsDoc (one index per caller, a key per nested call, copied on every admission) and NestedResultDoc are
replaced by NestedCallDoc (`pi.tool.nested-call`): a task-scoped family of the caller keyed by the call's key,
holding { taskId, result? }. Admission creates the member; the nested call's terminal commit adds its result;
executeTool() reads it by key. No commit rewrites another call's document.
TaskRuntime.ownedTasks(context) returns copies of the live tasks a task owns directly, whichever invocation
created them, from the scheduler's index. A tool call settling aborts the live nested calls among them, which
the removed index used to list. A task memo marks a tool that made a nested call in any attempt, so settling a
leaf still skips scanning pi.live.
8000 parallel nested calls: 9.2 s -> 2.7 s, linear (bench:scheduler-fanout).
