StoreStorage (Durable's storage on Chord v2's store, @earendil-works/pi-durable/storage/store) passes
Durable's storage conformance suite and runs Durable's harness tests (npm run test:store). The store
gets owner names and claims, one id allocator, Durable's Storage semantics for refused steps, history
none/latest/rewindable with reads at a point, archive revisions and applyBatch. self.follow follows
Durable's synchronous observers as Chord sources. Persisted jobs: records and state in the owner's
store, restarts as Durable's tasks, abort stored, deferred state. Chord's ./next export returns for
the adapter surface.
ConversationView.entries is a window over the fork-aware history with before;
conversation.watch(ctx, { from, min, max }) and observeConversation take the
window. ConversationTranscript merges windows into a client's transcript with
gaps, loadEarlier, fillGap and resume. pi.live.runFailure says why a run ended
without an answer. watchEvents and its event types are removed; the changelog
has the migration guide.
closes#11327
Every built-in storage implements ReplayableStorage: head(), replay() at an explicit sequence, and export() of the current state as a log that replays to the same reads. SQLite schema version 3 adds sequence indexes so export streams. @earendil-works/pi-durable/sync adds ReplicatingStorage, which appends each committed batch to an outbox with fork copies resolved, compactRecords(), and versioned JSON log lines.
Runs the storage conformance suite, the schema migration, an alarm-woken sleep across an eviction, and a
turn recovered through the recovery alarm in workerd via @cloudflare/vitest-plugin (npm run test:workers).
fix(durable): decline background timers in Cloudflare Workers. With nodejs_compat, timers have an unref(),
so the idle-memory timers were armed and a pending one kept a Durable Object from hibernating or being evicted.
- ToolRegistration.deferred: not offered to the model until modelTools names it
- modelTools gains { remove?, add? }; control.addTools writes names to add, so
loaded tools follow the others in load order and a load only appends
- Agent.deferred lists deferred tools not loaded yet
- @earendil-works/pi-durable/tool-search: the coding agent's tool_search (BM25)
- the codemode section leaves deferred tools out
details.calls (CodemodeToolDetails): each tool call the script made, in call
order, with status ok/error/cancelled, duration, and cut arguments and error;
calls still running or not yet launched when the script ended are cancelled.
Written once with the result: live progress is pi.live.nestedTools.
bench:codemode compares a code mode script with a plain tool making the same
nested calls: 1,000 parallel calls 0.82 vs 0.78 ms/call, sequential 1.52 vs
1.10 ms/call (two more commits per call: the journal entry and the count).
A codemode tool whose scripts call the conversation's tools as nested calls and
survive a restart. The script runs deterministically (pi-codemode start()); each
result is journaled before it is delivered. A rerun starts the script again from
the stored start data, delivers the journaled results in order (nested calls
reattach by key c<id>), checks every recorded call count, holds calls the
journal lacks until replay ends and checks them against earlier attempts' keys,
then continues live. A call launches only once the count of the drain that made
it is committed, so a rerun can prove which calls are new. Divergence, a changed
module version or sandbox fingerprint, and a passed deadline end the call with a
stored error; a decided outcome is returned without a VM. store()/load() keep
values per conversation, rewindable and forked as of the fork's entry.
pi-codemode: CodemodeRun.finished, set when the outcome is decided.
The delta tracker planned every structural array change with a walk over all
base elements; inserts and removals that keep the remaining elements in order
now get the same plan from the changed ranges alone. applyImmutable built an
appended array as a copy that the insert then grew, a second copy; an append now
builds it with one concat. Ops, values, immutability and errors are unchanged.
32,000 one-item appends, one commit each: 11.3 s -> 1.4 s
(npm run bench:document-append in packages/durable).
The Session kept every document it loaded or created in memory for the life of the process: every conversation and
session document ever touched, built-in and third-party. Now a cached document that no watch or document state
observes leaves the cache once it was not read or written for `documentRetentionMs` (HarnessSettings and
createSession options; default ten minutes, 0 drops it right after use, Infinity keeps documents; anything that is not
a number of milliseconds means the default), and loads from Storage again on its next access. An observer pins the
incarnation it observes and releases it when that incarnation retires.
Eviction runs on the mutation line, after every commit outcome (published, no-write, rejected, failed settlement) and
from one unreferenced timer for the next expiry, so a transaction never loses a tracker it holds. Where timers cannot
be unreferenced only commits evict. Cache bookkeeping is contained: a throwing clock or setting never changes an
operation's outcome. A hot document moves in the eviction order at most once per granule (a tenth of the retention,
up to 1 s), keeping the fan-out bench at HEAD's speed.
`npm test` runs the durable suite a second time with every unattached document evicted right after use.
5,000 conversations kept 26,000 documents and about 3.8 KB each; with a 50 ms retention, after waiting, none
(bench:document-cache).
The scheduler kept every conversation edge it saw and, for conversation owners, chain loads, and owners of known
tasks, the ended tasks above them, for the life of the process. Now an owner edge or an ended task's ownership is kept
only while something is listed below it in #below: only the chains above live work are known, whole. Unneeded nodes
are queued and swept at the end of each publication, or by a sweep job on the Session line after operations that
publish nothing (rejected commits, reads), never during an operation, so a commit that creates work below a chain its
own callback loaded keeps it. Later work in an old conversation loads its chain from storage again.
Removes #conversationOwners, #pinned, #knownChildren, and #forget. A queued input triggers the withdrawal scan only
while some task has cancellation intent, and chains are loaded only for queued inputs.
10,000 ended subagent cycles under a live parent: 10,000 settled tasks and 10,001 edges before, 0 and 1 now
(bench:scheduler-memory); a later message into an old subagent conversation reads 7 records instead of 5.
Every reconcile pass walked up from every live task to the root to find cancelling owners, and owned live work,
idle scopes, chain loads, reservation, finalization, and failFast checks each scanned all live tasks. A chain of n
held tasks cost about n^3 steps: 500 took 12 s, 20,000 never finished.
The ownership semantics are unchanged; in-memory indexes, rebuilt at open, hold each pass's candidates:
- #below: the ownership tree downward around live work; owned live work, idle scopes, and conversation aborts walk
down from one node. Ended tasks stay while walks pass through them (conversation owners, chain loads, owners of
known tasks), so loaded chains stay loaded.
- #intent: cascades walk down from tasks with cancellation intent, not entering another such task, so the nearest
cancelling owner still decides the reason; upward checks return at once when it is empty.
- #unloaded: only tasks whose chain is not loaded are loaded, and count as inside every scope until then.
- #finalizeChecks: finalize is a worklist in rounds from the nearest live holder of ended work.
- #runnable and #waiters: reservation and failFast candidates.
No walk recurses.
Chain of 500 held tasks: 12.3 s -> 50 ms; 20,000: 1.5 s. 8000 parallel nested calls: 18.8 s -> 9.2 s
(bench:scheduler-fanout); the rest is document copying of the nested-call index.
read returns an image file as one image block (programs get an ImageContent) plus an info diagnostic,
instead of an unsupported_image error. Limits are the conversation model's inputLimits.images.resize,
by default 2000x2000 pixels and 4.5 MB of base64.
- Without a processor: PNG, JPEG, GIF, and WebP within the limits pass through undecoded; the byte limit
is checked before reading the file, dimensions from the header. Larger images and BMP are error results.
- With createCodingTools({ images }): the processor orients by EXIF, converts BMP to PNG, and shrinks to
fit. ImageProcessor.prepare is async.
- @earendil-works/pi-durable/images: createPhotonImages(wasm) on @silvia-odwyer/photon 0.3.3 (web build);
/images/node and /images/cloudflare load the wasm. The root and ./tools entries never reach it (enforced
in check-entry-graphs).
- An image must be read whole: one that changes while read is read again, unlike a growing text file.
- A model without image input gets a diagnostic that it sees a placeholder.
Nested tool calls: ToolExecutionApi.executeTool() runs another tool as its own pi.tool task owned by the caller,
keyed for replay-safe reattachment. Results for programs live in the caller's NestedResultDoc until it settles;
pi.live.nestedTools shows each call with the arguments it runs with and a summary. TaskOptions.abandonOnRestart
aborts work its creator awaited only in memory before it runs again after a restart.
Structured output: structuredOutputSchema and structuredOutput, validated; a tool without a schema gives programs its
bounded output (one text as a string, one image as itself, else the list). ToolExecutionResult.content is now output.
callers and modelTools decide who may call a tool.
Storage errors: any error a Storage method throws fails the Session, which closes itself; later calls and waits reject
with SessionFailed, watches end with session_failed, and Session.closed settles. Nothing retries in the Harness.
StorageRejected is removed; StorageRequestError marks invalid reads that fail only their call. Listener and onReport
errors are contained and reported.
Also updates scripts/issues/triage.ts and the vacation experiment for the content -> output rename.
runtime.context() keeps each conversation's last read range and derived
view. A later read by any of the conversation's tasks with the same head
marker scans and derives only the entries committed since, so a
generation no longer rereads and re-derives the whole transcript for
each model request. Busy conversations keep the range; idle ones for
settings.contextRetentionMs (default ten minutes), checked on task
changes and by one unreferenced timer where the runtime supports it.
Kept entries are frozen; each read returns its own view arrays.
Add openDurableObjectSqliteStorage(ctx.storage) in
@earendil-works/pi-durable/storage/sqlite/cloudflare: SQLite storage on
a SQLite-backed Cloudflare Durable Object, without per-row copies.
Rename docs/pico-v5.md to docs/spec.md and link the design documents on GitHub
instead of shipping docs/. Add a default condition to every export so require()
of the ESM package works. Reset the changelog to the initial release.
Generation offers active registered tools through positional system entries and
hands a tool-calling answer to parallel or sequential pi.tool tasks and a
pi.post-tools task that applies addTools/terminate and continues the run. Tool
tasks validate, run beforeTool/afterTool, record intent with the replay policy,
publish bounded, sanitized, adaptively throttled progress in pi.live.tools, and
write results with a rendered diagnostics block. Hook dispatch with conversation
and subtree scopes, prepareArguments, HarnessOptions.env, and read/bash/edit/write
tools in pi-durable/tools. The env shell now streams raw output and spills past
thresholds; its bounded capture and publisher are removed.
Addresses GHSA-82fw-gwwq-j7x9 (vitest mocker path traversal) and
GHSA-3wwx-pv8p-q78v (undici 6.x permessage-deflate DoS). Both are
dev/example-only; the shipped coding-agent shrinkwrap is unchanged.
Replace the tsgo native preview with typescript@7.0.2 and port the check scripts to the TS 7 API. Target ES2024, drop useDefineForClassFields: false and unused decorator options, and enable verbatimModuleSyntax. Replace tsx with node plus the source resolver hook, passed to --import as a file URL.
closes#9965
