The pure swap is gone: a reload restarts every live object of a changed plugin
(documents and tables kept). Calls running at the commit and every call they cause
(the ctx call chain; an old flag on call from trusted links) run on the open old
attempt, which seals once its work drained and its successor's recorded requests are
ready, bounded by reloadDrainTimeout (5 s); then the successor's factory runs. New work
waits at the gate; same-host consumers see the draining object live; a reload cycle
fails at once with unavailable/reload_cycle. Example 64, design section 21 rewritten.
Adds experimental session sync (`@earendil-works/pi-durable/sync`).
Devices upload and pull a session's commits through a remote, reading
them from the Chord store's outbox and applying them with `applyBatch`.
If two devices continue the same session, the one that uploads second
forks to a new session.
I took out the extra sequence indexes from the earlier version of this
branch, since the Chord outbox already persists every commit, so this no
longer touches storage.
---------
Co-authored-by: Armin Ronacher <armin.ronacher@active-4.com>
Before a provider sends or forwards a call's result or error, it sends the origin's
waiting state, past the window if it is full, so a resolved call's changes are
applied on the page. TuiScope.optimistic shows a value while calls are pending; the
guide shows useOptimistic over a Chord reference; example 63.
buildWorkerBase bundles with esbuild code splitting into one private directory: the entry
and its static chunks load at each fork, a lazily imported module and the packages it
imports only when imported. @earendil-works/chord/workers/child exports the worker's side
(workerMain, processLock) without esbuild.
createWorkers bundles the worker entry once (esbuild, Node builtins and installed
packages external) and forks every worker from that snapshot, so an edit of the
app's source while the server runs reaches workers only through a restart. The kit
watches the bundle's inputs as base code. p2 creates its session workers with
SessionWorkers.create and keeps the worker base's inputs as a dev restart baseline.
Commits queued behind a running commit step share the next step's backend
transaction: each keeps its own step, seq, outbox commit and adoption in
queue order, the metadata is written once. A refusal rolls the group back
and its admitted commits run again alone; series, exports, replays and empty
commits end a group; backend failures while closing are latched; any throw
after COMMIT stops the store with memory at the durable group.
A hold unbinding with loading (a restart, a parked swap, a live failure retrying) keeps its calls in flight; each
gets one answer from what it waits on (the instance it runs on, or the gate). Calls sent by the old method names
before the consumer learns of the loading wait at the gate too. failed and removal still reject.
The server drops a page's own chord.* attributes and writes chord.host and the principal hook's result itself; tests
for the hook returning nothing, the NOOP adapter and calls over a link.
The four telemetry gaps (rows on commits, chord.host, microsecond times, a read-only store) and: pages record and
send span batches over their link to the server's ingest object (checked, capped, rate-limited, stamped browser and
with the link's principal); workers send theirs too, so the server is the store's only writer; principalAttributes
lets an app choose what a span records about its principal. The tutorial shows a real page's click traced to the
worker, with the trace viewer.
Chord's telemetry schema; spans for calls, methods, relays, holds, activations, natives, loads, reloads, resolves,
revocations, routines, jobs and HTTP requests; trace carriers on hold and call (bounded, printable ASCII) continued by
relays and workers; events and host-wide metrics; the app kit passes an adapter to its hosts. renderTrace escapes
control characters. Example 62 prints a trace across three hosts.
A call arriving while an attempt activates waits for its outcome (bounded by readyTimeout) instead of failing
not_ready while its client sees live. Fates start in the reverse of the order objects went live, so a pure
dependent of a swapping provider swaps instead of parking.
defineMessages with typed t, t.message/t.render for messages as data, the Translations registry with fallbacks and
conflicts, a locale preference per principal with the device as default and a live switch, useMessages and
scope.messages, formatTime/formatNumber. The tutorial gains a chapter on languages.
self.query(table, q, limit) runs before the object goes live and again in the commit step of every commit that
touched the table, publishing only the diff; a reload swaps the object. The tutorial and 59-chat use it.
The heartbeat no longer counts time the process did not run (a laptop's sleep, a blocked event loop) as silence: after such a pause each open link pings at once and closes only if its peer stays silent for close - ping.
A running worker's lost link is now a stop instead of a failure: worker_link_lost is reported and the held route starts a new worker. A second lost link within a minute of the last recovery still fails the address.
A worker killed by a signal left its lock's socket file, and every later start of its address failed closed on it, retry too. The workers component now removes a lock socket file nobody accepts on before each start and after a child's exit by a signal, in the address's queue. processLock also treats ENOTSOCK as stale instead of retrying until its context ended.
- get, hold and registry on a host that has not loaded throw not_ready at once.
- A registry with contributions and no provider is reported (registry_undeclared) at publication.
- A request waiting unbound is reported once after 5 s (unbound_long; one clock per host), and keeps waiting.
- An invalid plugin name makes load and reload throw before any change.
- A transaction cancelled by the end of its context is reported (transaction_cancelled) besides rejecting.
- Docs: who a reference acts as, no per-link call limit, await what you start in a root or a handler.
src/next is src (the main export), src/next-node is src/node with short subpaths (/http, /workers, /plugins,
/plugins/runtime, /app, /surfaces, /web, /react, /tui, /tui/launcher); the foundations get /json and /watch;
Chord v1 moves to src/old (/old, /old/node) until p2 runs on v2; tests and docs move likewise (test/, docs/,
test/old, docs/old). Durable and env import the foundations from their subpaths.
- Web: base bundles built from the app's entry at start (content-hashed, served
forever), plugin scripts that register themselves and one combined file,
WebScripts/TuiScripts showing the current set, startPage with the connection
loop, in-place plugin reloads and a reload on a new base bundle.
- React bindings: HostProvider, useHold, useWatch, useRegistry, useRefState.
- Terminal: startTerminal, a launcher that signs in with a token, a token file or
device pairing and always runs the app's terminal code in a sandboxed child
process (pipes, empty environment, --permission granting nothing, no SQLite,
a builtin allowlist and a frozen process inside); output filter, confirmed
copy/open/attach actions; startTerminalRuntime and TuiScope inside.
- Themes: the Themes registry, ThemeChoice per principal, CSS variables on the
page, terminal colours by role; CSS modules in plugin builds.
- App kit: startServer gains web and tui; startApp renamed startServer,
exportAddress/importAddress renamed exportData/importData and export no longer
stops the worker; ephemeral sessions end with their client's last link.
- Examples 56-web, 57-tui, 58-themes, 59-chat.
chord/next/app: startApp and startWorker with the fixed app layout, worker kinds,
AppInfo and WorkerInfo; sessions (hashed tokens, rolling cookies, bearer tokens),
the one-time owner link, the owner token file, device pairing; Files per host
with ordered chunked uploads; export and import per address; plugin roots,
installs and applyChanges; optional users, invite and ephemeral login facets.
HTTP: deny by default (public routes and sockets), remoteAddress,
readSignedCookie. Examples 53-55.
Plugins import the app's versioned API (createPlugins api option: unlisted
versions, other kinds' modules, mixed versions and direct Chord imports refused),
each other by relative path within a root or by package name across
repositories (npm or git pins), with per-kind files checked by name.
chord.plugins lists a package's plugins. Required roots, single-file plugins,
installPlugin from git or npm, testPlugin with stub facets.
plugin-authoring.md for authors and agents. Examples 49-52.
A relayed hold released (or its principal refused or revoked) while the route resolved made #resolve throw before adding the connection, so the spawned host's link was never read and its own upstream references failed unavailable/ready_timeout after readyTimeout. The connection is now added right after resolve; the checks only fail the relay.
Schema helpers are a thin layer over pi-schema's builders with the same JSON,
so fingerprints and stored data are unchanged (proven by schema-switch tests and
a store written by the TypeBox code). Checks through pi-schema; error messages
are pi-schema's; format is an annotation. v1 contracts take any JSON Schema.
map(bytes()) works.
chord/next/http: a Web fetch handler built from the registry Http (routes, wraps,
sockets, hosts, relays, CSP sources) and Authenticators, responses as descriptions,
a security baseline on by default, Chord's own Node listener as a host-lifetime
object, nodeRequest for apps mounting it, links and plugin sockets.
chord/next/workers: a forked worker per address, started on demand, stopped when
idle, never restarted by itself, plugins built by the server and shipped.
Plugins: kinds with platform and host packages, createPluginRuntime, scripts(kind),
rollout results per host, imports of other plugins by name. Kernel: RouteHandle
(retry, suspend), no route backoff, anonymous host.root, origins only from relays.
ws is an optional peer dependency.
Plugin discovery, esbuild builds (optional peer dependency), vm evaluation with
pluginFile and mapStack, dependency installs, a watcher with dirty marks and
rollout across hosts; host.load and host.reload per plugin with the closure rule;
examples 35-39 as folders with real plugin directories.
A Session no longer holds its line while a commit is stored. Commits that
arrive while one storage call is in flight go out together in the next call,
each with its own seq and outcome; a job runs until it touches a key an
unpublished commit wrote, then waits for that commit's publication. Outcomes
are delivered in line order; publication is in line order, one commit per
macrotask. The scheduler's decisions run as regions that record the keys they
read and rerun after a wait, with side effects only after a clean check.
Storage.commit(batches, receive, context) replaces commit(writes, adopt,
context): prefix semantics (Durable's own checks refuse a batch and keep the
ones before it; any other failure rejects the call), receipts per stored
batch, optional publishing() scope. Memory, SQLite, JSONL, IndexedDB and
StoreStorage implement it natively; StoreStorage through the new
ChordStore.commitSeries (one store transaction per call), replacing
commitInBatch.
A normal run makes no storage reads: bounded record cache, conversation
tails, task-document index, kept context ranges. A tool slot's running status
and a generation's attempt are stored with the next commit (12 instead of 14
commits per run).
One Harness with 128 conversations over storage with a 4 ms round trip: about
5 -> 440 agent runs per second. Design: packages/durable/docs/group-commit.md.
Registries: shared tokens (defineRegistry), one provider per host (host.serve(R, { compose, accepts })),
contributions from facets and host-lifetime objects, plain values or followed Chord watchables, a table
sorted by plugin, lazy compose in the batch that changed the table, failures left out and reported with
the last good value kept; self.registry as a declared dependency, host.registry outside objects, older
readers mapped by accepts[old].value; keyedEntries and orderedEntries. Native sources: self.writable,
registry values and ref.state. self.retryOn wakes failed objects on registry values. host.root gives code
outside objects a call scope sealed at settle, abort and host.close.
Durable: registryReader(value, context); a Session close ends synchronous observers failed
("session closed"), watches still end ended.
StoreStorage (Durable's storage on Chord v2's store, @earendil-works/pi-durable/storage/store) passes
Durable's storage conformance suite and runs Durable's harness tests (npm run test:store). The store
gets owner names and claims, one id allocator, Durable's Storage semantics for refused steps, history
none/latest/rewindable with reads at a point, archive revisions and applyBatch. self.follow follows
Durable's synchronous observers as Chord sources. Persisted jobs: records and state in the owner's
store, restarts as Durable's tasks, abort stored, deferred state. Chord's ./next export returns for
the adapter surface.
The next object runtime moves from the private repository into pi: src/next (contracts, hosts and
objects, documents, derives, links and relays, routines and jobs, the store) with its tests and
examples in test/next, src/next-node (Node compression), and its design in docs/next (design.md, the
decision log decisions.md, the slice plan slices.md, slice specs, books and benchmarks). AGENTS.md
gets its code style. Not exported: the API changes with every slice until it has an entry point.
Chord and Durable change a document one transaction at a time, so a tracker registered a WeakRef per
change for competitors that never existed on their hot path. The WeakRef targets survived scavenges
and V8 could pretenure every overlay context into old space (47.5 MB promoted against 2 MB in a
tracker loop). The tracker now keeps its open change in a plain field and moves it into the weak
registry only when a second change opens; stale semantics are unchanged. A draft dropped without
prepare or abort stays referenced until the tracker's next change or adopt (at most one per tracker).
`stored` and `rawNodes` are created on first use.
Chord's shared watch: listeners take a tagged call, { kind: "value", value, ops, retired } or
{ kind: "failed", error }; a recoverable failure keeps the watch open and the next value arrives as
[["r", value]]; WatchEnd is tagged by kind, and an aborted context ends a watch "aborted".
Durable: observers take the watch's shapes (advance(call, ctx), end(end)); onReport(report) with
report.kind and report.error, session_failed is storage_failed; subscribeCommits is observeCommits;
SessionEnd and ReportFrom are tagged by kind; FileChange is tagged and file watchers stop with
stop(ctx); CompactionFailure and CompactionStatus name their trigger (pi.live version 2 with a
migrate); the task_outdated report uses the scheduler's reasons; TaskInspection's ready is pending;
DocumentCommitChange is tagged. Stored formats are unchanged.
The Node driver takes the count from node:sqlite; the Durable Object driver reads
SELECT changes() after the statement, since the cursor's rowsWritten also counts
index writes (verified in workerd). run is unchanged and costs nothing more.
Durable publishes a stored commit inside Storage.commit's adopt: the
telemetry events, its own observers (document watches, conversation
views, task graph, scheduler, waits), and a dispatch of host
subscribeCommits listeners one microtask later, before every wait and
abort the commit triggers. A commit the storage rejects after adoption is
published before the Session fails; a throwing telemetry adapter is
reported as host_error "telemetry" instead of rejecting a durable commit.
Chord's watch listeners take one change object, { value, ops, retired }
({ value } for native watches); retirement is a flag, not an end.
Durable's document watches follow their address, and a task document's
watch ends after its retirement call. New synchronous observers for
adapters: observeDoc, observeConversation, observeTaskGraph; the watches
are built on them. The env file watcher's change type is renamed
FileChange.
Tracker revisions, applyImmutable() and applyImmutableBatches() copy each container on a changed path with
spread (Object.assign() onto a null-prototype object) instead of one Object.defineProperty() per key: same own
data properties, prototype and own __proto__ member, about 25 times faster per container. The copy-count test
counts key listings through a proxy instead of spying on Object.keys.
Adds a tracker GC regression test: 10,000 adopted changes retain only the document, not a context registration
per change. Follow-up to #9265.
- chord/delta: PendingOps (push, size, take -> PendingTake): raw pending ops, r drops everything
before it, a leaf index merges appends and primitive sets per path (cleared by any other op),
past 10,000 ops only "replace with the current value"; one batch is kept by reference and
received ops are never mutated
- chord: Watch queues frames in PendingOps while its listener is busy or not started and delivers
one call with the newest value, the folded ops (or [["r", newest]] after an overflow) and the
newest frame's context; replaces the 100-frame collapse. NativeWatch is unchanged and takes an
optional replace()
- durable: watchEvents keeps every event batch in order (NativeWatch with a snapshot replace);
document, view and task-graph watches fold; spec, README, example 04 and tests updated
- tests: fold fuzzers (tracker, appendText, splices/reorders, root arrays, diffRevisions) with
codec round trips, adversarial cases, cap, ownership, cost and GC guards
Follow-up to #9265.
- chord: Delta appendText(draft, key, text) / StringKeys<T> record an append op without the
O(length) prefix compare that a plain `draft[key] += text` costs in prepare()
- durable: partial commits append text and thinking deltas with appendText (assignJson plan,
checked by length), and apply tool-call argument changes from their own StreamingJsonParser
of the raw toolcall_delta text instead of reassigning the arguments snapshot
- durable "." entry-graph budget 73 -> 76 for the parser it now imports
- ai: StreamingJsonParser / ToolCallArgumentsStream read each argument delta once (tolerant,
immutable snapshots, linear cost); every provider, pi-messages and the faux provider use it
and keep stream state (partialJson, partialArgs, index, customInput, streamIndex, Bedrock
reasoning bytes) off the content blocks; grammar tool input appends per delta
- agent: streamProxy uses the same parser
- chord: Delta append ops own their text (ownedSlice), so ops and replicas no longer keep the
whole growing string alive
- durable, ai, mcp: bounded error excerpts and summaries own their characters
- coding-agent: tool calls render streamed arguments once per frame, not once per delta
Breaking changes:
- Durable Storage.commit(writes, adopt, context): storages call adopt(seq) once,
synchronously, when the batch is durable and before their next commit
- Chord traceIn(options, fn, context): context last
- WatchHandle and WatchEnd move to Chord (Durable re-exports them); end reasons
ended (was session_closed) and failed (was session_failed); retirement is an
explicit source event, and an ended watch drops its references into the Session
- Removed Session.documentState(), Conversation.viewState(), Harness.taskGraph()
and DocumentState; use watchDoc(), watch(), watchTaskGraph()
Reverts 48c8f3228 and 4759f09fb. Session sync replicates pi-durable commits instead of SQLite pages, so NodeSqliteDatabase.exclusive(), the replicated connection wrapper, the WAL reader, page checksums, the segment container, start-marker keys and the zstd page compressor have no user.
NodeSqliteDatabase.exclusive() runs an operation alone on the connection, outside a transaction, and rolls back a transaction the operation left open.
The internal replicated connection wrapper opens a database file with automatic checkpoints off, reports every operation, and runs a hook before SQLite checkpoints and deletes the WAL on close. It is not exported from the package yet.
- pi-telemetry: StoredSpan/StoredTrace/TelemetryStore (findTraces with name, attribute, error, time-window filters and
cursor paging; getTrace; findMetrics), trace ids carried in carriers, renderTrace in the /render subpath; the in-memory
adapter implements the store; the adapter contract trusts typed inputs (malformed-input conformance cases removed)
- chord: @earendil-works/chord/telemetry-sqlite over chord/sqlite: settled spans with inline events and links, metrics
aggregated per series per 10 s, 1 s batched flush with retention, bounded queue (dropped and counted), store queries
