sscoble/pi-mono
1
0
Fork 0
mirror of https://github.com/badlogic/pi-mono.git synced 2026-11-05 23:37:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

feat(chord): reload restarts every affected object; old work finishes on old code

Browse source 
The pure swap is gone: a reload restarts every live object of a changed plugin
(documents and tables kept). Calls running at the commit and every call they cause
(the ctx call chain; an old flag on call from trusted links) run on the open old
attempt, which seals once its work drained and its successor's recorded requests are
ready, bounded by reloadDrainTimeout (5 s); then the successor's factory runs. New work
waits at the gate; same-host consumers see the draining object live; a reload cycle
fails at once with unavailable/reload_cycle. Example 64, design section 21 rewritten.
This commit is contained in:
Mario Zechner 2026-11-05 12:15:27 +00:00
parent 5a31a6d5f4
commit a5d209afd0
70 changed files with 2975 additions and 1146 deletions
Download patch file Download diff file Expand all files Collapse all files

3
packages/chord/CHANGELOG.md
View file

@ -8,6 +8,7 @@
- Chord v1 (`@earendil-works/chord/old`, `@earendil-works/chord/old/node`) removed, with its tests and docs (`src/old`, `test/old`, `docs/old`). Apps still on v1 pin `@earendil-works/chord@1.5.0`.
- Chord v2's `startWorker` no longer takes `telemetry` (it was added unreleased in slice 9): with a telemetry store on the server, the kit's workers send their spans and metrics to it, and no worker opens the store's file. `startServer`'s `telemetry` is a store with `ingest` (`createSqliteTelemetry`'s). Migration: drop `telemetry` from `startWorker` and the store's path from the worker's environment.
- Chord v2's `createWorkers` returns a promise and rejects when the worker base does not build (decision "Workers start from the base code the server booted with"): it bundles the worker kind's `entry` and the base code it imports once with esbuild (`--conditions` from `execArgv`; Node builtins and packages resolved under `node_modules` external, by their files; workspace packages linked into `node_modules` inlined; `import.meta.url` and `__dirname` naming each source file; split at each dynamic `import()` esbuild can follow, so a lazily imported module and the packages it imports load only when the worker imports it; inline source maps, forked with `--enable-source-maps`) into a private temporary directory removed at `close`, and forks every worker from that snapshot: an edit of the app's source while the server runs (an agent half way through a change) no longer reaches workers forked later, which kept failing to start until the edit was fixed; a server restart applies it. `workers.inputs` lists the files the base was built from; the app kit's `startServer` builds each worker kind's base before loading (a broken one rejects the start, everything closed) and marks `base` when one of those files changes (it marked only the entry).
- Chord v2's `host.reload` restarts every live object of a changed plugin; the pure swap in place is removed (decision "Reload restarts every affected object; old work finishes on old code"). `ReloadReport.objects` no longer has the kind `swapped`: a live object is `restarted`, or `parked { cause }` when its successor's recorded requests find a failure. Old work (the calls running at the commit and every call they cause through their `ctx`, and calls marked `old` over a trusted link) runs on the object's open old attempt; new calls wait at the gate for the new code. The old attempt is sealed once that work drained and its successor's requests are ready, bounded by the new host option `reloadDrainTimeout` (default 5 s, at most `closeTimeout`; the app kit's `timeouts.reloadDrainTimeout` reaches the server and every worker): a method still running then is cut `released` and reported `drain_timeout`. A wait of old work that its own old attempt blocks fails at once with `unavailable/reload_cycle`; a getter of old work for a version the new code dropped binds the open old attempt or fails at once. `call` gains `old` (binary flag `0x20`, the text codec's seventh element; a relay drops it from untrusted links; `PROTOCOL_VERSION` stays 1); `waiting` causes keep the reason `unavailable/version`. Consequences: a reference shows `loading` briefly after every reload of its object (it stayed bound across a swap); a value read right after `host.reload` resolves may still be the old code's; an in-memory job run ends `aborted { reload }` at every reload of its object (it finished on its old code across a swap); routines restart when the successor goes live. Migration: drop `swapped` from code reading the report; open a root inside a method with `withoutAbortSignal(ctx)`, not a fresh context, so its calls are old work; move work longer than `reloadDrainTimeout` into jobs.
### Added
@ -21,7 +22,7 @@
- Chord v2's root exports `invalid(reason, message, details?)` and the new `forbidden(message, details?)`, the `ChordError`s a plugin throws for arguments or params it refuses (`invalid/args`, `invalid/params`) and for a principal it refuses (`forbidden`); an app's API re-exports them, and the caller gets the same `{ code, reason }` as from Chord's own checks.
- Chord v2's React bindings gain `usePageState(key, initial)`, `useState`'s pair with the value kept by the page's host under `key` instead of the component: a component of a reloaded web plugin is a new function and remounts (its `useState` starts over), and page state is what survives (a half-typed message, a scroll position); any value, never on the wire, gone with the page. `TuiScope` gains `pageState(key, initial)` on the same per-host store, its changes calling the scope's `changed`.
- Chord v2 reports mistakes that waited silently through `onError` (decision "Mistakes that wait are reported at once"): `unbound_long { waitingFor, error, since }`, a request waiting unbound (a type or registry not served here, not at its version, a registry whose first `compose` threw) once after 5 s, on one timer per host (it keeps waiting until its `readyTimeout`); `registry_undeclared { registry, plugins }`, a registry with contributions and no provider at a publication (`load`, a reload's commit) or when a host-lifetime object's contributions enter, naming the contributing plugins, again only once they change; `transaction_cancelled { object, error }`, a document change or transaction that did not commit because the context it was started with ended first (a root whose `fn` returned, a caller that cancelled, its object's end), besides rejecting.
- Chord v2 gains live queries: `self.query(table, q, limit)` in the factory of a table's owner is a watchable of the query's rows (at most `limit`, in its order), usable as a state member and a derive source. Chord runs it before the object goes live and again inside the commit step of every commit of the object that writes the table, publishing only the diff in that commit's batch; a commit writing no table with a live query runs nothing (`chord.query.runs` counts the runs). Only the object's own tables; the limit is required (`invalid/args` at declaration). A run that fails (a row the code cannot read) is a failed member, reported once as the new `HostError` `query_failed { object, table, error }`, and heals at the next good run. A live query is no native, so an object showing table rows live swaps on reload instead of restarting, as reading them in `self.open` made it. Each touching commit runs the query once (100 rows: about 60 µs, 80 µs watched): not for tables written hundreds of times a second.
- Chord v2 gains live queries: `self.query(table, q, limit)` in the factory of a table's owner is a watchable of the query's rows (at most `limit`, in its order), usable as a state member and a derive source. Chord runs it before the object goes live and again inside the commit step of every commit of the object that writes the table, publishing only the diff in that commit's batch; a commit writing no table with a live query runs nothing (`chord.query.runs` counts the runs). Only the object's own tables; the limit is required (`invalid/args` at declaration). A run that fails (a row the code cannot read) is a failed member, reported once as the new `HostError` `query_failed { object, table, error }`, and heals at the next good run. Each touching commit runs the query once (100 rows: about 60 µs, 80 µs watched): not for tables written hundreds of times a second.
- Chord v2 gains messages and languages in the app kit (design §23): `defineMessages(namespace, { en: {...}, de: {...} })` returns a typed `t(key, ...args)` (the first catalog types the others: a missing key, an unknown one or a wrong argument is a type error; values are strings or functions), `t.message(key, ...args)` builds a message as data (`MessageData`, `{ namespace, key, params }`, for state members typed with `MessageSchema`), `t.render(message)` renders one (a foreign namespace or an unknown key from a link shows the key), `t.translations` lists its catalogs, and `MessagesOf<typeof t>` types a translation. The `Translations` registry takes a list of `{ namespace, locale, messages, override? }` per contributor (one catalog per namespace and locale: the first by plugin name wins, reported; `override: true` wins); a missing key falls back down the locale's chain (`pt-BR`, `pt`) to the first catalog, reported once. `LocaleChoice` keeps one locale per principal on the server (the theme choice's store, now shared as one preference); a client follows it with `followActiveLanguage`, defaulting to `navigator.languages` on a page and the system locale in the terminal (`startTerminal`'s `locales`), matched against the catalogs, switched live. `useMessages(t)` (`/react`) and `TuiScope.messages(t)` (`/tui`) render in the host's language and re-render or redraw on a change; `startPage` sets `lang` and `dir` on the page (`followLanguage`, `/web`); `formatTime` and `formatNumber` use `Intl` in the language being rendered. The kit serves `Translations` on the server, workers, pages and terminals and `LocaleChoice` on the server; client errors gain `{ kind: "messages", report }`.
- Chord v2 gains telemetry across hosts (slice 9): `CHORD_TELEMETRY_SCHEMA` (root export) types every span, event and metric Chord records. With a host adapter (`createHost({ telemetry })`), calls (`chord.call`), methods (`chord.method`, local and remote), relays (`chord.relay`), holds (`chord.hold`), activations (`chord.activate`, ending `live`, `failed`, `parked` or `ended`, with `chord.retry` and `chord.park` events), natives' opens and closes, `load`, `reload` (a `chord.reload.fate` event per object), resolves (a worker start included), revocations, routine calls, job phases (`intent`, `run`, `finish`: a persisted run resumed in a new attempt or process stays under its stored carrier) and HTTP requests (`chord.http`: method, route pattern, status, never the path) are spans; failures carry a fixed category and the message cut at 500 characters, a cancel is no error; commits are `chord.commit { documents, ops, rows, duration }` events of the span they ran under. `hold` and `call` carry the caller's trace carrier (`trace`, at most 8 keys, keys and values of at most 256 characters of printable ASCII, else the message fails the message schema and the link closes; the binary codec's kind byte flag `0x40`), relays forward their own span's, and the provider runs the method under it (an absent or unknown carrier: the adapter's root); a reference's carrier rides on its first hold only, a re-hold after a reconnect being a fresh root linked to it. A hold's `type` and its address's `kind` are bounded at 256 characters by the message schema, and span attributes a peer chose at 256 characters. `ctx.trace(name, attributes, fn)` runs `fn` in a child span with a child ctx. Host-wide metrics: `chord.link.bytes`, `chord.link.state_messages`, `chord.commit.duration`, `chord.calls.in_flight`, `chord.holds.active`, `chord.links.open`, `chord.park.duration`, `chord.backpressure.wait`. Without an adapter nothing is started, built or sent: a call over a link costs what it did before. `createHttp`, `startServer` and `startWorker` take a `telemetry` adapter (`createHttp` continues a W3C `traceparent` only from a request a contributed relay verified: an app's proxy must strip it from browsers). A host with an adapter counts the cost counters of its links' and root scopes' work on it also when the app passed a context without a span. The SQLite telemetry store keeps spans of one microsecond in the order they started.

213
packages/chord/docs/design.md
View file

@ -279,9 +279,9 @@ the app exits with its own code for its supervisor or reopens.
| `readyTimeout` | an activation attempt's own steps (its wait for the previous instance's teardown included; paused while the attempt is parked on its dependencies, slice 7a); a getter's whole wait; how long a call made while its reference is loading waits (the park deadline) | 60 s | per `get`/`hold`/`registry`: `{ readyTimeout }` |
The park deadline's per-type override is not built (only if a real case needs it). Timeouts of later slices belong to
their mechanisms: the routine handover uses `closeTimeout`, a job `finish` backs off 1 s to 60 s, provider-side
`linger` is per type; a reload has no install timeout (it takes the app's context, slice 7b); a per-type drain
override only if a case needs it (section 26). A route's `resolve` is bounded by the app.
their mechanisms: a restart's old attempt is bounded by `reloadDrainTimeout` (default 5 s, at most `closeTimeout`;
section 21), a job `finish` backs off 1 s to 60 s, provider-side `linger` is per type; a reload has no install timeout
(it takes the app's context, slice 7b); a per-type drain override only if a case needs it (section 26). A route's `resolve` is bounded by the app.
- **Incarnations** are minted by Chord (uuidv7, code copied from pi-ai, no dependency), never passed in, and carried
in the link handshake (`hello`): the other side sees a restart as a new incarnation of the same identity;
@ -395,7 +395,7 @@ host.route(Session, {
its consumer acknowledges it before it sends the call that answer frees a slot for, so running calls and messages
past the window stay within N per origin together. Nothing extra is sent when nothing waits (the common path: a
free window sent the commit's `state` already). Bindings waiting for a slot are not sent past it: their `bound`
carries the latest state when it goes (a hold rebinding after a reload's swap shows its old values until then).
carries the latest state when it goes (a hold rebinding after a restart's old attempt ended shows its old values until then).
Not covered: changes committed after the answer (a method that returns before its effect lands) or on another host
that does not relay the answer.
Not built, by decision: a byte window, credit messages, snapshot admission or debt, an adapter `backlog()` API,
@ -677,8 +677,7 @@ host.serve(Notes, ({ session }, principal, self) =>
a job or routine start); commits are checked when they enter the store's commit step (sealed: rejected `released`,
nothing written) and at adopt (sealed during the SQL: durable, adopted into the object's documents, its staged job
changes adopted into the object too: decision "Second spec review: the majority" (3)); at the seal a transaction not yet in the step is aborted and the line released, a started step
finishes before teardown completes; cleanup (references, presence, activity counts) still runs. A pure swap is not a
replacement.
finishes before teardown completes; cleanup (references, presence, activity counts) still runs.
- **Ending** (slice 1, 2): new calls are rejected; every caller of a running call gets `released` at once; the
handler keeps running in the background under an aborted context and its `self.tx` throws `released` (a running
method can change nothing after the end). Each call runs under a context aborted by its caller and by the object's
@ -789,9 +788,9 @@ host.serve(NoteStore, ({ session }, self) => {
routines and jobs.
- Handles stay valid for their attempt's life (owner, 2026-10-26: documents and the line belong to the object; each
attempt's handles carry its seal token and definition version); nothing is "read at use" or re-fetched.
- On a reload the factory runs again: in staging against a recording `self` (declarations only), then for real; pure
objects swap in place (new methods and derives, same documents), native objects restart after their drain,
objects of a removed type end and their holders wait (section 21).
- On a reload the factory runs again: in staging against a recording `self` (declarations only), then for real; every
live object restarts (same documents; its running calls and the calls they cause finish on the old attempt, the new
factory runs once that drained), objects of a removed type end and their holders wait (section 21).
### Pattern: shared store, per-principal views
@ -848,8 +847,8 @@ location are dropped.
reload, retry, restart; a new object after its type was removed and served again; another variant), the reference
moves its subscriptions to the new one and sends one frame, a diff from the last value (consumers' derives recompute,
a failed one heals; a member the new variant lacks fails `invalid/member`). While it points at nothing, its members
keep their last values (a pointer, no copy): while loading (a restart, a retry, a swap whose new instance is not live
yet) its watches get no call (`ref.state` says `loading`); while failed (a live failure, its version refused, its type
keep their last values (a pointer, no copy): while loading (a restart's successor not live yet, a retry) its watches
get no call (`ref.state` says `loading`); while failed (a live failure, its version refused, its type
removed, a failed dependency) they get one `failed` call with the cause's error and stay open (decision "Failures are
watch calls"); a reference that never had a value reads `not_ready` while loading and its failure's error while failed, and its
watches wait for its first values. A member the new variant lacks is not a failed reference: it fails `invalid/member`
@ -865,8 +864,8 @@ location are dropped.
the re-exposing object's reference member, followed the same way. Over a link the same rule: the replica keeps its
last values readable while its hold is failed. Nothing outlives its object instance and nothing is handed between
objects.
- **References across a reload** (slice 7b §6): a reference is never ended by a code change. Its object swapped in
place or restarted: it follows the new instance (above; a hold over a link gets a fresh `bound`, §22). Its version no longer accepted, or a member of it refused (`contract_mismatch`): it leaves the object and waits
- **References across a reload** (slice 7b §6): a reference is never ended by a code change. Its object restarted: it
follows the old instance while that serves its version, then the new instance (above; a hold over a link gets a fresh `bound`, §22). Its version no longer accepted, or a member of it refused (`contract_mismatch`): it leaves the object and waits
unbound (`failed { waiting }`) with its last values; the object and its other holders stay. Its type removed: it
waits unbound `not_served` with its last values; a publication that serves the name again binds it to the next
object (a new object: its in-memory documents start over from `initial()`, persisted ones load again). On a removal
@ -936,7 +935,10 @@ parented under it; a throw marks the span failed (`trace_failed`) and is rethrow
`fn` gets `ctx` itself (slice 9; round-3 cut C11 kept, as this section already stated it); in a job's `run` also `jobId`
and `state` (slice 5). Pass `ctx` to everything you wait
on. Code outside objects gets a `ctx` from `host.root` (slice 8). Durable's `Context` is the same module
(`../context`); bridging is explicit. No code versions, no request ids.
(`../context`); bridging is explicit. No code versions, no request ids. A method's `ctx` is also a node of its **call
chain** (the attempt the method runs on, then the chain of the call that made it): during a reload it tells old work
from new (section 21), so a root opened inside a method takes the method's `ctx` (`withoutAbortSignal(ctx)`), never a
fresh context.
**Await what you start in a root or a handler** (decision "Mistakes that wait are reported at once" (e), owner,
2026-11-03; footgun 10 of the p2 port). What is started with a context ends with it: a root's scope seals when `fn`
@ -1296,7 +1298,7 @@ commits are the exact invalidation: a commit not writing the table runs nothing)
(params decide it); **a limit is required** (`invalid/args` at declaration; the store's query language keeps the limit
an argument of the runner, as `page` has it). A run that fails (a row the code cannot read) is a failed member, as a
throwing derive (one failure call, `query_failed` once per episode, `[["r", rows]]` at the next good run); the object
stays live. It is no native, so an object with live queries swaps on reload (section 21). Cost: one query per
stays live. It is no native: a restart's successor runs its live queries again at its activation (section 21). Cost: one query per
touching commit, bounded by the limit (100 rows: about 60 µs unwatched, 80 µs watched, `benchmarks.md`); **there may
be dragons** for tables written hundreds of times a second. Incremental maintenance of the result (row-level changes
kept in memory, a re-run only when a removal empties the window or a bulk write hides which rows changed) is left for
@ -1457,7 +1459,7 @@ self.onLive(async (ctx) => { for await (const f of watcher.value.files(ctx.abort
has finished or after `closeTimeout` (30 s at most, Q7: a longer script is aborted); Chord then opens the new one
immediately. Code mode's sandbox does
this; its tool entry looks up the current sandbox at `execute()`.
- **Swap through a registry:** the resource is a registry contribution; readers always find the current one, and a
- **Replace through a registry:** the resource is a registry contribution; readers always find the current one, and a
reader that must wait for one uses a cancellable "wait until present" with the park deadline.
---
@ -1487,12 +1489,11 @@ self.onLive(async (ctx) => { // set up, wait
and stop when it ends (their `ctx` is aborted, then
the drain waits for running calls, bounded by `closeTimeout`). Each call gets its own `ctx` (the object's principal
for a remote type, the host's for a local one; the host's incarnation; `ctx.get` released when the call settles).
- **On reload** the old routines' `ctx` is aborted; the new routines start once all old routine calls of the object
have finished (bounded; a stuck call is reported and the new routines start anyway): a handover per object, so
routines need no names. Example it prevents: an old handler midway through `restart(W1)` and a new one seeing W1
still pending and restarting it again. The bound is `closeTimeout` (owner, 2026-10-24); calls do not wait for the
handover, only the new routines do (slice 7b). A stuck old call past the handover is reported `drain_timeout` and
the whole old attempt is sealed then (audit A12: one seal token per attempt; its method calls had the same bound).
- **On reload** the old routines' `ctx` is aborted at the restart; their calls drain with the old attempt, and the
new routines start when the successor goes live, after the old attempt's seal: routines need no names. Example it
prevents: an old handler midway through `restart(W1)` and a new one seeing W1 still pending and restarting it again.
The bound is the reload drain bound (`reloadDrainTimeout`); a call still running then is reported `drain_timeout`
and the whole old attempt is sealed (audit A12: one seal token per attempt).
- **`on(source, fn)`:** any watchable (a document, a derive, a reference member, a registry value, `ref.state`). The
first call gets the current value (`[["r", value]]`, W2b), then one call per change; calls never overlap (S8);
changes made during a call arrive folded in the next (newest value, folded ops); a running call is not aborted by a
@ -1507,8 +1508,8 @@ self.onLive(async (ctx) => { // set up, wait
job for long work. Chord keeps no state for routines across reloads or restarts; what must survive is a document of
the object (e.g. a `lastCleanup` time, which also survives a crash).
- **Helpers:** `sleep(ms, ctx)` (rejects when `ctx` is aborted, leaves no timer or listener) and `untilAborted(ctx)`.
Routines and handlers wait only through calls that take `ctx`; otherwise Chord cannot stop them, and the handover
and the drain report them stuck.
Routines and handlers wait only through calls that take `ctx`; otherwise Chord cannot stop them, and the drain
reports them stuck.
- **Errors:** `onError` gets `routine_failed { object, routine, error }`, `routine` a `Position` (`{ kind: "on" |
"onLive", position }`; natives use the same type); never uncaught.
- **Keep-alive:** routines do not count in `host.activity` (an `onLive` runs forever).
@ -1633,12 +1634,10 @@ host.serve(ChatStore, ({ conversation }, self) => { // local: no princi
forwards and its result forwards (`{ 1: { args: (old) => current, result: (old) => current } }`; a producer's
direction, unlike a contract's). Its records keep every run's version: a new attempt's `self.job(Def, impl)` at
another version takes the records over (the same version with another fingerprint, over args, result and state, is
refused at declaration, as documents). A run started on an older version and still running finishes on its old
code across a pure swap (its invocation keeps the old implementation); `finish` runs on the new code with the
outcome's args and result mapped forwards (once per run, never per publication); `running` shows its args mapped; a
run not invoked yet runs on the new code with its args mapped. Its state document keeps its version while the old
code writes it and migrates through its own `migrate` once its run settled or moved to the new code (the outcome
`finish` gets). Without an adapter, for a run of a newer version than the code (a rollback), or for a job the new
refused at declaration, as documents). A reload restarts the job's object: a running in-memory run ends `aborted
{ reload }` at the old attempt's seal (a persisted one is stopped and resumed per `resume`, R3); `finish` runs on the
new code with the outcome's args and result mapped forwards (once per run, never per publication); a run not invoked
yet runs on the new code with its args mapped, and its state document migrates through its own `migrate`. Without an adapter, for a run of a newer version than the code (a rollback), or for a job the new
code no longer declares, the run is `blocked { version }` (kept, not invoked) until a reload brings matching code;
blocked runs (`finish_failed` ones too) are retried on every reload of their object's plugin; a persisted run of
another version or fingerprint stays blocked with its record as stored until the reload that brings its version
@ -1899,9 +1898,9 @@ agent asks everyone present. Its store must live in an object (it predates "nati
**complete new plugin set** after `load`: a plugin in it and not loaded is added; one loaded with another facet value is
changed (Chord compares no code); one passed as the very same facet object is unchanged and untouched (owner,
2026-10-29: base code passes its core facet value every time); one loaded and not in it is removed. It resolves with a
report once the commit happened and every object took its fate (swaps and retries had their first dependency check;
it does not wait for objects to go live); a plugin whose new code fails keeps its old code and is listed in the
report's `failed` (per plugin, below).
report once the commit happened and every object took its fate (a restart reports what its successor's recorded
requests found at once, a retry its new attempt's first dependency check; it does not wait for objects to go live); a
plugin whose new code fails keeps its old code and is listed in the report's `failed` (per plugin, below).
Reloads run one after another per host; before `load` resolved it throws, and so does a plugin set with an invalid
name (before anything changes); after `close` it rejects `released/closed`;
`context` bounds it (decision 4: no install watchdog): its abort before the commit rejects `aborted` and changes
@ -1914,8 +1913,8 @@ type ReloadReport = {
readonly plugins: { readonly added: readonly string[]; readonly changed: readonly string[]; readonly removed: readonly string[] };
readonly failed: readonly { readonly plugin: string; readonly error: unknown }[]; // new code failed: old code kept, an added one left out
readonly objects: readonly ({ readonly object: string } & (
| { readonly kind: "swapped" } | { readonly kind: "restarted" } | { readonly kind: "ended" }
| { readonly kind: "parked"; readonly cause: FailureCause } | { readonly kind: "retried" }))[];
| { readonly kind: "restarted" } | { readonly kind: "ended" } | { readonly kind: "retried" }
| { readonly kind: "parked"; readonly cause: FailureCause }))[];
};
```
@ -1948,8 +1947,8 @@ type ReloadReport = {
`linger` and provider, `bindProvider(contract, options.accepts)`; registry providers; facet contributions by
plugin), then, for
every live, activating, parked or failed object of a changed or removed plugin whose type is still served, its new
factory with the object's params (and principal) against a recording `self`: it records what the factory opens,
the documents, tables and jobs it declares with their versions, its contributions and requests, runs the checks an
factory with the object's params (and principal) against a recording `self`: it records the documents, tables
and jobs it declares with their versions, its contributions and requests, runs the checks an
attempt makes without effects (declared twice; persisted data outside a stable identity; contributions outside a
host-lifetime object; a document or job at an older version, or at its version with another schema), and hands out
inert handles. Staging is synchronous, so no request or arriving hold meets it half done (audit A1: model §1's
@ -1961,56 +1960,73 @@ type ReloadReport = {
After the batch, each object takes its fate, host-lifetime objects of new types start (no barrier), and the
publication wakes the requests indexed under the names the reload serves (`UnboundIndex.wake(names)`, per name),
refused holds of those names re-run their start.
- **Fates** (decided in staging; objects of plugins not in the reload are untouched):
- **Fates** (decided in staging; objects of plugins not in the reload are untouched; decision "Reload restarts every
affected object; old work finishes on old code", which superseded the pure swap):
| The object | Fate |
| --- | --- |
| its type is removed | **ended**: as an end (natives close `end`, callers of running calls rejected `released`, handlers drained, in-memory documents gone, persisted ones kept); its holders' requests are not ended: they wait unbound `not_served` (§7) |
| failed, parked, activating or waiting for its backoff | **retried** on the new code (`ref.retry()`'s rule, whatever broke it); an activating attempt is sealed and its natives close `reload` |
| live, neither its live attempt nor the new factory opens natives, and no table of the new factory is newer than the store's | **swapped** in place (below); live queries are no natives (section 11): an object showing table rows through `self.query` swaps, one reading them in `self.open` restarts |
| live, otherwise | **restarted**: new calls park (it shows `loading`), its contributions leave, running calls drain (bounded by `closeTimeout`; a call outliving the drain is cut off by the seal fence), the attempt is sealed (in-memory job runs end `aborted { reload }`, persisted ones are stopped and resumed, R3), natives close `reload`, the new attempt activates after that teardown (documents stay; eager table migrations run as at activation) |
| any of the above whose new code waits for a dependency | **parked** (decision 4): no overlap of old and new code; reported with the cause |
| live (natives or not) | **restarted** (below): its old attempt serves old work until that drained and its successor's requests are ready, bounded by the reload drain bound; then it is sealed (in-memory job runs end `aborted { reload }`, persisted ones are stopped and resumed, R3), natives close `reload`, and the successor runs its factory and activates (documents stay; eager table migrations run as at activation) |
| a restart whose successor's recorded requests find a failure, or a retry whose new attempt waits for a dependency | **parked** (decision 4): reported with the cause; the successor activates once the dependency is there |
A restarted object gets a new instance id (an old activation chain never matches it); a swapped one keeps it; a
retried one keeps it as any retry does. Cycles are checked again for the new attempts; `parked_long` is unchanged.
- **Fates start dependents first** (decision "Swaps: dependents first; calls during a swap never see `not_ready`"):
the removals first, then the other fates in the reverse of the order the objects went live (a dependent goes live
after its providers). Example: `todo.store` and its view `todo.todos(groceries)`, both pure, in one plugin. Started
provider first, the view's new attempt would bind the store while its swap shows `loading` and park (its
contributions leaving, its holds over a link unbinding) until the store's new instance is live. Started first, it
binds the store while that is still live, passes its dependency check and reads the store's old instance until
the new one is live (one frame), its calls waiting at the store's gate: both are reported `swapped`, nothing parks.
This is the path a provider-only reload under a live dependent already takes. A removed provider ends first, so a
dependent whose new code still needs it parks (decision 4); a provider takes its new served type with its own fate,
so a dependent needing a version only the new code serves waits unbound until then and parks on the provider's swap
(never reading that version off the old instance). A pure dependent still parks only then, or when its provider
went live after it (a provider that failed live and came back), or when its new code needs something that is not
live (a new or missing dependency).
- **The pure swap** (decision "The seal fence": not a replacement while its calls run): a new attempt of the same
object (its documents, line and job records) activates at once; calls arriving meanwhile wait at
the gate (bounded by `readyTimeout`, `unavailable/park_timeout`). The gate holds a call while any attempt of the
object activates (a swap, parked or not, a restart after its drain), until the object is live, failed or ended;
a parked attempt showing `failed { waiting }` rejects it with that cause. A call never gets `not_ready` there: a
hold over a link stays bound while the old instance still changes, so its consumer sees `live` and sends. When the
hold then leaves the old instance (a parked swap's retirement), it tells the consumer `loading` and keeps the calls
still in flight: they go on waiting at the gate (§22, decision "Calls in flight survive a `loading` unbind"). The
new attempt's requests are made and bound
before the old attempt's are released (a request equal to an old one binds to the same target with no gap, M§3.9).
Running calls finish on the old attempt with its handles valid, except R15: an old call touching a document whose
version rose throws once (`released/version`), the document intact, the call not resent; the same for a table whose
version rose (old table handles' version fence: old code never issues SQL for a dropped column). Once the swap
ended (the new attempt live, parked or failed) and the old attempt's calls settled (bounded by `closeTimeout`,
`drain_timeout`), the old attempt is sealed and its references released; its job invocations finish on their old
code. Retiring attempts are bounded by `closeTimeout` each; a later swap never waits for them (several may retire
at once after quick reloads), while a restart, an end and the host's close do (an end rejects their callers at
once). The references move to the new instance's members by name (one frame from the last value; a member the new
variant lacks fails `invalid/member`; §7); exposed derives are matched by their member names, internal ones are
rebuilt; in-memory documents migrate through `migrate` when the new code declares a higher version; the
contributions are replaced in one batch; `self.writable` starts at its new initial value (decision 2); `retryOn`
sources are re-declared; the new attempt's live queries run at its activation (the old attempt's are re-run by the
commits of its remaining calls until it is sealed, then dropped). **Routines hand over per object:** the old attempt's routines are aborted; the new ones
start once the old routine calls finished, bounded by `closeTimeout` (a stuck one is reported `drain_timeout` and
the whole old attempt is sealed then, audit A12: one seal token per attempt); calls never wait.
A restarted object gets a new instance id (an old activation chain never matches it); a retried one keeps it as
any retry does. Cycles are checked again for the new attempts; `parked_long` is unchanged. Live queries are no
natives (section 11): the successor's run at its activation.
- **Fates start in the plan's order**, removals first; the restarts' readiness and successors start once every fate
started, so the requests they bind meet the new served types only.
- **Old work** (decision "Reload restarts every affected object; old work finishes on old code", rule 2): the calls
running on an object at the commit and every call they cause. On one host they are found through `ctx`: a method's
`ctx` is a node of its call chain (the attempt it runs on, then the chain of the call that made it), so a call,
`ctx.get` or root scope given the method's `ctx` (`withoutAbortSignal(ctx)` for a root) is old work while any
attempt in its chain drains for a restart. The host counts its draining attempts and the old calls it runs for
other hosts, so outside reloads no chain is walked. Across hosts a consumer marks such a call `old` (§22); a
provider honours the mark only from a trusted link and runs the call as old work, counted while it runs; a relay
drops it from a link that is not trusted (a page cannot claim old work).
- **Old work runs on old attempts:** a call of old work on an object whose old attempt is open runs there, with its
handles valid; a getter of old work binds that attempt for any version it serves. A getter for a version the new
code dropped binds the open old attempt or fails at once (`contract_mismatch`, never a wait for a code update). An
object that did not exist starts on the new code; its requests bind open old attempts as any request does (rule 5
below). **New work** (anything else) waits at the gate and runs on the new code (bounded by `readyTimeout`,
`unavailable/park_timeout`); a parked successor showing `failed { waiting }` rejects it with that cause. A call
never gets `not_ready` there: a hold over a link stays bound while the old attempt is open, so its consumer sees
`live` and sends; when the hold then leaves the old instance, it tells the consumer `loading` and keeps the calls
still in flight: they go on waiting at the gate (§22, decision "Calls in flight survive a `loading` unbind").
- **The seal** (rule 3): the old attempt is sealed once its own calls and the old work routed to it drained (a call
routed after the drain began joins it) and its **readiness** holds: the requests its successor's factory made in
staging, bound once every fate started, are each live, ended, or failed for a reason other than another host's
version skew (`contract_mismatch` or `unavailable/version` from a remote provider, kept on `waiting` causes along a
chain; it heals when that host's reload lands). Registry requests are not waited for (the old attempt's
contributions keep them). All of it is bounded by the **reload drain bound** `reloadDrainTimeout` (a host option,
default 5 s, at most `closeTimeout`; the app kit passes `timeouts.reloadDrainTimeout` to the server and every
worker): a method still running then is cut `released` and reported `drain_timeout`; long work belongs in jobs.
Only after the seal does the successor's factory run, so no new declaration meets old work: old work writes the
documents and tables at their old versions, and the successor migrates what it wrote. The readiness bindings and
the old attempt's references are released only once the successor's own requests bound (a dependency held only by
them never ends, M§3.9); presence lists (§19) never show readiness bindings (they are no consumers of their own).
A failed request is shown by the successor while it lasts (new calls get the cause, consumers of the old attempt
see the failure) and no longer once it recovered; from the old attempt's end on, the successor's own dependency
checks decide what it shows. The version fences stay for calls cut by the seal (R15: a document or table whose version
rose throws `released/version` there).
- **Consumers meanwhile** (rule 5): a same-host consumer whose version the open old attempt binds sees the object
`live` and follows the old instance (its new code may read the provider's old state for up to the drain); a
reference or hold of a version only the new code serves shows `loading` until the successor is live. At the seal
the old attempt stops being the instance its holders bind (`loading`; a hold waiting for a window slot unbinds);
once its admitted commits published it stops changing and the references take their last values from it. When the
successor goes live they move to its members by name (one frame from the last value; a member the new variant
lacks fails `invalid/member`; §7); internal derives are rebuilt; in-memory documents migrate through `migrate` when
the new code declares a higher version; `self.writable` starts at its new initial value (decision 2); `retryOn`
sources are re-declared.
- **Contributions:** an old attempt with natives withdraws its contributions at the seal (F-b); one without keeps
them until its successor is live, which replaces them in one batch, or drops them when the successor fails or parks
(F-c). **Routines** stop at the restart; their calls drain with the old attempt's; the successor's start when it
goes live.
- **Reload cycles** (rule 6): a wait of old work on an object whose activation waits, through its parked
dependencies or the gates its opener waits at, on the old attempt that very work runs on fails at once with
`unavailable/reload_cycle`, naming the chain (`app.f(fresh) → app.p`: old work creating an object whose new code
needs the raised provider). Checked when the wait starts and whenever an attempt parks or an opener begins waiting
at a gate. What leaves Chord (roots opened without the method's `ctx`, Durable tasks, an app's locks) and cycles
across hosts end at the reload drain bound.
- **Natives are not kept across reloads by name:** natives have no line, transactions or schema, so old and new code
would interleave unchecked. In practice a restart costs little (the Durable harness is base code, terminal PTYs live
in their own process, model runtimes, sandboxes and the search index rebuild in milliseconds); reloading the plugin
@ -2056,6 +2072,13 @@ type ReloadReport = {
another host sees follows §7: kept and freshly bound while its version is accepted; `failed
{ waiting }` (`version`, `not_served`) otherwise, bound again by the install that serves it; a consumer whose own
code moves to a new version releases its old request and makes a new one.
- **A contract change between hosts in one rollout leaves a gap** (decision "The rollout order stays; no `loading`
grace"): a consumer on an earlier step whose provider on a later step moves to a new version waits for the
provider's step (server first: server consumers of worker types). A restarting consumer's readiness waits out the
other host's version refusal, so its old attempt keeps serving meanwhile, up to the reload drain bound; past it, or
for a consumer created meanwhile, the reference waits `failed { waiting }` with its last values. `accepts` cannot
close the gap within one rollout (the old provider never serves the new version). The recipe: two applies, the
provider first (serving the new version and accepting the old one), then its consumers.
- **The app's loader keeps its own duties:** the desired code set, a digest over the facet bundles' content hashes (an
unchanged server restart reinstalls nothing), serialized installs, a host that starts or reconnects installing the
current set before anything is routed to it, a rejected own install failing its own requests with the reason.
@ -2069,8 +2092,8 @@ worker (the reload suite), not only by unit tests. "The page" is any client over
| Change rolled out | What stays on screen | Calls | State |
| --- | --- | --- | --- |
| a plugin whose new build, evaluation or staging fails | everything; the plugin keeps its old code, the error is reported to the app (a page's own evaluation failure is reported in that page) | unaffected | unaffected |
| pure code (no natives) on the server or a worker | the view stays bound; one fresh snapshot once the new code is live; values change only where the new code computes differently | a running call finishes on the old code; a call arriving meanwhile waits and runs on the new code; never `not_ready` | documents and tables unchanged; `self.writable` values start over |
| code holding natives (restart) | `live` with the last values (and what the draining old code still commits) while the old code drains, up to `closeTimeout`; then `loading` briefly; then a fresh snapshot | running calls get up to `closeTimeout` on the old code; calls arriving meanwhile wait (the provider host's `readyTimeout`) and run on the new code | documents and tables unchanged; natives reopened; `self.writable` values start over |
| pure code (no natives) on the server or a worker (a restart) | `live` with the last values (and what the old code still commits) while the old code drains, up to `reloadDrainTimeout`; then `loading` briefly; then a fresh snapshot from the new code | running calls, and the calls they cause on any host, finish on the old code (up to `reloadDrainTimeout`, then `released`); calls arriving meanwhile wait (the provider host's `readyTimeout`) and run on the new code; never `not_ready` | documents and tables unchanged; `self.writable` values start over; in-memory job runs end `aborted { reload }`, their `finish` on the new code |
| code holding natives (a restart) | as pure code; the old code's natives close (`reload`) at the seal, the new code's open before the fresh snapshot | as pure code | as pure code; natives reopened |
| a contract change the new code still `accepts` | the old page keeps working through the mapping | answered through the mapping | unchanged |
| a contract change not accepted, or a type removed | the reference waits with its last values (`failed { waiting }`) until the page's code updates or the type returns | calls fail at once with the cause (`contract_mismatch`, `unavailable` not served; §7) | unchanged |
| a web or terminal plugin | the other plugins untouched, scroll kept; the reloaded plugin's components remount | the reloaded plugin's calls as above | component state starts over; `usePageState`/`scope.pageState` survive |
@ -2172,7 +2195,7 @@ validation, no clone of snapshots or ops.
| `revoke` | parent to child (route link) | the revocation value (4b) |
| `release` | to provider | `origin?`, `hold` |
| `retry` | to provider | `origin?`, `hold`, `hops` (the retry's hop budget, 1-16: 16 from the first host, one less for each host it passes on from, none sent at 0; audit A5); relays forward it unchanged |
| `call` | to provider | `origin?`, `call`, `hold`, `method` (index), `args?` (part; absent: no arguments), `segments?` |
| `call` | to provider | `origin?`, `call`, `hold`, `method` (index), `args?` (part; absent: no arguments), `segments?`, `old?` (`true`: old work of a reload, §21) |
| `abort` | to provider | `origin?`, `call` (named `cancel` until 2026-10-27) |
| `ack` | to provider | `origin?`; one per applied `bound`/`state` message |
| `closed` | relay to provider | `origin`: the downstream origin ended |
@ -2186,7 +2209,7 @@ validation, no clone of snapshots or ops.
| `ping`, `pong` | both | the heartbeat |
A `bound` on a hold that is bound already replaces its replicas in one batch, another variant allowed (re-holds,
slice 7b's restarted objects and its swaps to another variant).
slice 7b's restarted objects, another variant included).
- **Codecs are pluggable per link** (both ends use the same): `binaryCodec` (default: a kind byte with a flag for a
following origin varint, varints, raw incarnations and fingerprints, length-prefixed strings and parts; about 23 B
for a streamed token) and `textCodec` (Delta-style JSON tuples without field names, a message with an origin wrapped
@ -2260,6 +2283,12 @@ validation, no clone of snapshots or ops.
the text codec writes it as the last element (a hold's after `limit`, absent ones before it `null`; a call's after its
payload, whose two fields are then positional, `null` when absent). Relays read it from the envelope and forward
their own span's carrier (a relay recording nothing forwards it as it came). `PROTOCOL_VERSION` stays 1.
- **Old work across hosts** (decision "Reload restarts every affected object; old work finishes on old code"): a
consumer marks a `call` `old: true` when its caller is old work of a reload on its host (§21); the provider honours
the mark only from a trusted link (it runs the call on open old attempts, as old work of its own host while it
runs) and ignores it otherwise; a relay drops it from a downstream link that is not trusted. The binary codec flags
it in the call's kind byte (`0x20`; on any other kind that bit makes an unknown kind); the text codec writes it as a
call's seventh element, after the trace carrier (`null` when absent). `PROTOCOL_VERSION` stays 1.
- **Adapted bindings get diffs** (slice 7b; decision "References are the only indirection" (5)): a consumer bound
through `accepts` attaches to the object instance's shared mapping for its version: per member one derive of the
instance's member through `(current) => old` (a changed member), of its whole state (a removed member), or the
@ -2273,16 +2302,14 @@ validation, no clone of snapshots or ops.
failure (its calls waiting for a slot reject with it; the provider rejects its calls in flight once); the provider
detaches the hold's sources, drops its unsent pending ops and a queued `bound`, and keeps the ack slots of `bound`
and `state` messages already sent until their `ack` or the link's loss. A reload sends it when the object's type is
removed (`not_served`), its version is no longer accepted (`version`), or a swap parks on a failed dependency; a
restart, a parked swap still loading or a live failure already retrying sends `loading`; both are followed by a fresh
`bound` when the object is live again (applied as a re-binding). A swap in place keeps the hold bound and sends it a
fresh `bound` once the new instance is live (audit A9, 2026-10-30: one rebind path; a tab's watchers may get a frame
for an unchanged member).
removed (`not_served`), its version is no longer accepted (`version`), or a restart's successor parks on a failed
dependency; a restart (once its old attempt no longer serves the hold's version) or a live failure already retrying
sends `loading`; both are followed by a fresh `bound` when the object is live again (applied as a re-binding; a
tab's watchers may get a frame for an unchanged member).
- **A `loading` unbind keeps the calls in flight** (decision "Calls in flight survive a `loading` unbind"): the
provider unbinds the hold as above but rejects none of its calls, and the consumer keeps them (only its calls still
waiting for a slot reject `unavailable`). Each kept call gets exactly one answer from what it already waits on: the
instance it runs on (its result within `closeTimeout`, else `released` at the seal; a hold binding afresh after a swap
in place may unbind while its live instance still runs calls) or the gate (the new instance's result, the parked
instance it runs on (its result within the drain, else `released` at the seal) or the gate (the new instance's result, the parked
cause, `unavailable` after `readyTimeout`, the object's end); an `abort`, a `release`, `gone` or the link's close
ends it as any call. Until its fresh `bound` is sent the hold takes calls by the names it was bound with (sent before
its consumer learnt of the `loading`): they wait at the gate too, never `not_ready`; after a `failed` it takes none.
@ -2687,7 +2714,7 @@ this document):
| 11 | where a flow lives | 20 | when login is ported |
| 14 | project plugins | 21 | own pass over `plugin-loading.md` |
| 16 | a Durable Object profile | 24 | when a DO target becomes concrete |
| 24 | a per-type drain override: a host option or fixed (no install watchdog: `reload` takes the app's context, decision "Slice 7b's open points" (4); the routine handover uses `closeTimeout`; a failing job `finish` backs off 1-60 s and blocks after five failures, decided) | 3 | if a real case needs it |
| 24 | a per-type drain override: a host option or fixed (no install watchdog: `reload` takes the app's context, decision "Slice 7b's open points" (4); a restart's old attempt is bounded by `reloadDrainTimeout`; a failing job `finish` backs off 1-60 s and blocks after five failures, decided) | 3 | if a real case needs it |
| 28 | Astra's circuit breaker (a hard transport-buffer cap plus a stall deadline); the heartbeat is built, the cap was never decided | 3 | if a real case needs it |
Apps (p3), from the functionality map, decided per domain when p3 is designed:

17
packages/chord/docs/guide-plugins-hosts/book.md
View file

@ -45,8 +45,8 @@ This book builds the app in that order. Each chapter adds one concept and shows
server, workers and tabs load the scripts for their kind.
6. **Plugins meet through types and registries.** One plugin uses another's objects through its contract, or
contributes to a registry another defines. Nothing else crosses between plugins.
7. **An edit is a rollout.** Change a file, build, roll out: the server, then every worker, swap the plugin's objects in
place; a broken edit keeps the old code running and reports why.
7. **An edit is a rollout.** Change a file, build, roll out: the server, then every worker, restart the plugin's
objects (running work finishes on the old code); a broken edit keeps the old code running and reports why.
:::
:::flow wide animate | How it all fits: plugins on disk are built once on the server; each host evaluates the facets of its kind; tabs and the TUI are clients that reach workers' objects through the server.
@ -225,9 +225,9 @@ self.onLive(async (ctx) => { // a routine start
});
```
Natives are why some objects restart when their plugin changes: an open connection cannot be moved to new code, so
Chord closes it and opens it again. Objects without natives are swapped in place. A first read of a table is no
native: declare it as a live query, not with `self.open`, or the object restarts at every change of its plugin.
Every live object restarts when its plugin changes; natives close (`reload`) and open again with the new code, since
an open connection cannot be moved to new code. A first read of a table needs no native: declare it as a live query,
not with `self.open`.
**Await what you start in a root or a handler.** What is started with a context ends with it. Code outside objects
gets its `ctx` from `host.root(fn, ctx)`, and that scope seals when `fn` returns; a handler's context ends when its
@ -698,11 +698,12 @@ then gives every object of the changed plugin its fate:
| The object | What happens |
| --- | --- |
| holds no natives (open files, sockets), the new code neither | **swapped** in place: new methods, same documents; running calls finish on the old code |
| holds natives | **restarted**: calls wait, running ones finish, natives close and reopen with the new code |
| live | **restarted**: running calls, and the calls they make with their `ctx`, finish on the old code; new calls wait for the new code; same documents; natives close and reopen |
| not live (failed, parked) | **retried** on the new code |
| its type was removed | **ended**; its users wait until something serves it again |
The tab's reference follows the new object and gets one frame with the difference. Nothing reconnects. If the edit
The tab's reference stays live while the old code drains, shows `loading` for a moment, then follows the new object
with one frame of difference. Nothing reconnects. If the edit
is broken (a syntax error, a factory that throws), that plugin keeps its old code on that host and the report says why;
every other plugin of the rollout applies.

21
packages/chord/docs/guide-plugins-hosts/index.html
View file

File diff suppressed because one or more lines are too long

12
packages/chord/docs/plugin-authoring.md
View file

@ -82,8 +82,8 @@ host.serve(Store, (_params, self) => { // a local type with empt
(`invalid/args` without one).
- One query per writing commit, at most `limit` rows (for 100 rows about 60 µs more per commit, 80 µs when watched;
`benchmarks.md`): there may be dragons for a table written hundreds of times a second. Keep a document for those.
- Never read a table in `self.open` to fill a window: that makes a native, and the object restarts at every reload of
its plugin instead of swapping in place. A live query is no native.
- Never read a table in `self.open` to fill a window: a live query reruns by itself after every commit and at a
reload's restart; a read in `self.open` is a one-time snapshot.
## What a plugin may import
@ -158,6 +158,14 @@ The app's own contracts for third parties come with its API (`@acme/app/v3` expo
changes shape, bump its `version`; the provider serves older consumers through `accepts`, else they wait "contract
mismatch" (visible, never wrong data).
## Reloads
A reload of the plugin restarts each of its live objects. The calls running at that moment, and every call they make
with their `ctx`, finish on the old code; new calls wait for the new code; documents and tables stay. Two rules keep
that working: a root opened inside a method takes the method's `ctx` (`host.root(fn, withoutAbortSignal(ctx))`), so its
calls count as the method's; and work longer than a few seconds is a job, since a call still running at
`reloadDrainTimeout` (5 s) is cut `released`.
## How the app builds it
The server builds each facet with one esbuild call from source: TypeScript and TSX (JSX with the automatic React

14
packages/chord/docs/slices.md
View file

@ -51,6 +51,10 @@ slice spec's section citations are for orientation, not a reading limit (owner,
reloads, resolves, revocations, routines, job phases and HTTP requests, carriers on `hold` and `call`, `ctx.trace`,
commit events and the host-wide metrics, `telemetry` options of `createHttp`, `startServer` and `startWorker`;
example 62; `test/bench/telemetry.ts` (with R24's container width).
- **Reload restarts every affected object** (built 2026-11-05, decision "Reload restarts every affected object; old
work finishes on old code"): the pure swap removed; old work through the call chain (`host/old-work.ts`) and
`call.old`, a restart's readiness (`host/readiness.ts`), reload cycles (`host/reload-cycle.ts`),
`reloadDrainTimeout`; example 64.
- **Chord's shared code** (upstream pi): `src/watch.ts` (`Watch`, `NativeWatch`, `WatchHandle`, `WatchEnd`,
`NativeWatchable`), Delta (`track`, `diffRevisions`, `applyImmutable`, encoders, `PendingOps`, `appendText`),
context, JSON, output, SQLite drivers.
@ -74,7 +78,7 @@ telemetry last, since it instruments the app kit too; 2026-11-01: the plugin mod
| 6b | **Durable and jobs on the store** | persisted jobs, Durable's `StoreStorage`, the Chord source over Durable's views (`self.follow`) |
| 8 | **Registries and `host.root`** | registries, `self.writable`, `retryOn`, Durable's `registryReader`, `host.root` (`chord(fn, context)` and the harness object are p3's base code) |
| 7a | **The dependency model** | requests that outlive providers, parked attempts without timers, wakes, the publication gate, getters, registries as dependencies, `parked_long` (`slice-7a.md`, `dependencies/model.md`) |
| 7b | **Reload in the kernel** | `host.reload`: staging, commit, rollback; swaps, restarts, removal and re-serving; holds and `accepts` across a reload; jobs and registries across a reload; per-name wakes (`slice-7b/` brief) |
| 7b | **Reload in the kernel** | `host.reload`: staging, commit, rollback; restarts (swaps until 2026-11-05), removal and re-serving; holds and `accepts` across a reload; jobs and registries across a reload; per-name wakes (`slice-7b/` brief) |
| 7c | **Plugins** | `chord/plugins` (discovery, `buildFacet`, `evaluateFacet`, `pluginFile`, `installDependencies`, the watcher, `rollout`, `mapStack`; no dev/prod split; the kernel's `load`/`reload` per plugin; `slice-7c.md`) |
| 10 | **HTTP and workers** | `chord/http` (server, `Http` registry, responses as descriptions, WebSocket entry) and `chord/workers` (a worker per address, IPC, idle stop) |
| 12 | **The plugin model** | imports (relative within a repository, package names pinned across), versioned app API with Chord re-exported, per-kind API modules and contract files with the build's import check, single-file plugins, git installs, `testPlugin`, a required root (`slice-12.md`) |
@ -147,7 +151,7 @@ pattern, no code. Decided cuts are noted where they apply.
| 12 | `self.derive`, failures per member, lazy when unwatched, `mapShared` | 1 |
| 12 | container-width benchmark (R24) | 9 |
| 13 | `self.open`, close reasons, order, restart patterns (R16) | 2 (restarts on reload: 7b) |
| 14 | `self.on` (W2b), `self.onLive`, `sleep`, `untilAborted` (C4 decided); the handover per object on reload | 5 (handover: 7b) |
| 14 | `self.on` (W2b), `self.onLive`, `sleep`, `untilAborted` (C4 decided); routines across a restart | 5 (reload: 7b) |
| 15 | `defineJob`, `self.job`, intent/run/finish, state documents, dedupe (Q9), `busy`, `running`, blocked `finish` (C3 decided); persistence, resume, restart limit; `accepts` in use | 5 (persistence: 6b; `accepts` and reload: 7b) |
| 16 | `defineFacet<P>`, facet as declarations only (documents and routines removed in 6a) | 1, 6a |
| 17 | registries: `defineRegistry`, contributions, `compose`, versions (Q8), use-site guards | 8 |
@ -156,7 +160,7 @@ pattern, no code. Decided cuts are noted where they apply.
| 18 | platform bindings as core objects | doc (8) |
| 19 | `host.references` (presence) | 4b |
| 20 | flows kept by their owner, the `steps` helper, the Questions service | doc (5); deferred decisions #9, #11 |
| 21 | `host.reload`: staging, atomic commit, pure swaps, native restarts, removal and re-serving | 7b |
| 21 | `host.reload`: staging, atomic commit, restarts with old work on old code, removal and re-serving | 7b, restart-only 2026-11-05 |
| 21 | loading plugins (loader D), dev loop; the rollout across hosts is the app's (decision "Slice 7b's open points" (3)) | 7c |
| 22 | message format, header schema, unknown ids, one message per commit per link, consumers apply ops, plugin errors | 3a |
| 22 | relays, size limits per part, compression at the origin, adapted bindings get diffs | 4a (relays), 4b (limits, compression), 7b |
@ -186,8 +190,8 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over.
- **Bump `PROTOCOL_VERSION`** for any message or reason change once p3 has shipped (slice 3b added `retry`, `loading`
and `failed`'s `retryIn`; slice 4a parts, the list `bound`, `hello`'s `calls`, `ack`, `ping`, `pong`, `closed`,
`origin` and the hold's `address`, `principal` and `incarnation`; all without a bump: nothing is deployed). (Owner,
slice 3b.)
`origin` and the hold's `address`, `principal` and `incarnation`; the reload build `call`'s `old` and the
`unavailable` reasons `version` and `reload_cycle`; all without a bump: nothing is deployed). (Owner, slice 3b.)
- **The park deadline's per-type override** is not built: a parked call waits the reference's `readyTimeout`. Build
it only if a real case needs it. (Slice 3b spec.)
- No CBOR; revisit only if benchmarks show encoding as a bottleneck (slice 3a's and 4a's do not). (L2 decisions.)

31
packages/chord/docs/tutorial/book.md
View file

@ -1343,7 +1343,7 @@ console.log(" ada adds ' ':", refused);
<!-- output: main -->
```text
9. lists changed, rolled out to: server, list, web, tui
on list|groceries: todo.store swapped, todo.todos(groceries) swapped
on list|groceries: todo.store restarted, todo.todos(groceries) restarted
ada adds ' tea ': [ ] tea (ada)
ada adds ' ': { code: 'invalid', reason: 'args' }
```
@ -1355,26 +1355,25 @@ and rolled it out host by host. On each host Chord stages the new code (it runs
| The object | Fate |
| --- | --- |
| opens no natives, and neither does its new factory | **swapped** in place: new methods, the same documents; running calls finish on the old code |
| opens natives | **restarted**: calls wait, running ones finish, natives close and open again with the new code |
| live | **restarted**: running calls finish on the old code, new calls wait for the new code, the same documents and tables |
| not live (failed, parked, activating) | **retried** on the new code |
| its new code waits for a dependency | **parked** until the dependency is live |
| its type was removed | **ended**; its holders wait until something serves it again |
The store opens no natives, and neither does its new factory: its live query is not a native, Chord runs it. So the
store was **swapped**, not restarted. The sequence on the list's worker:
Both the store and the view were live, so both **restarted**. The sequence on the list's worker:
1. The store's new attempt starts beside the old one: the same object, instance, table and line. Calls arriving now
wait at a gate; calls already running finish on the old code.
2. The new attempt runs its live query (the same read as at the first activation) and goes live; the gate opens,
and references move to its members by name, each with one frame from its last value.
3. The old attempt is sealed once its calls settled; its query leaves with it, so later commits run only the new one.
1. The old attempts stay open: calls already running finish on the old code, and so does every call they make with
their `ctx` (**old work**: the view's running `add` reaches the store's old attempt, never waiting for the new
one). Calls arriving now are new work: they wait at a gate.
2. Each old attempt is sealed once its calls drained and the requests its new factory made in staging are ready (the
view's new code needs the store, whose old attempt still serves it). That takes as long as the running calls, at
most `reloadDrainTimeout` (5 s); a call still running then is cut.
3. The new attempt runs its factory, its live query runs again (the same read as at the first activation), and it
goes live: the gate opens, and references move to its members by name, each with one frame from its last value.
The views were swapped too, and before the store: Chord starts the fates dependents first, so each view's new
attempt found the store still live, read its old values until step 2 and moved to its new members with one frame;
its calls to the store waited at the gate. Had the store gone first, the views would have found it between steps 1
and 2 and **parked** for that moment. Ada's reference never showed `loading`, and her next add was trimmed.
Had the store kept its window as a one-time read in `self.open`, it would count as holding a native and **restart**
at every reload of its plugin: running calls drained, natives closed and opened again, a new instance.
Ada's reference stayed `live` while the old code drained, showed `loading` for a moment, then the new code's values;
her next add was trimmed. Long work does not belong in a method: a call longer than the drain bound is cut at a
reload, so it goes into a job.
Ada's blank todo is now empty after trimming, so the store refused it with
`invalid("args", …)` (chapter II): her client got `{ code: "invalid", reason: "args" }`, Chord's own error, which

35
packages/chord/docs/tutorial/index.html
View file

@ -1185,7 +1185,7 @@ console.<span class="tk-f">log</span>(<span class="tk-s">" ada adds ' tea ':
console.<span class="tk-f">log</span>(<span class="tk-s">" ada adds ' ':"</span>, refused);</code></pre>
<!-- output: main -->
<pre class="plain"><code>9. lists changed, rolled out to: server, list, web, tui
on list|groceries: todo.store swapped, todo.todos(groceries) swapped
on list|groceries: todo.store restarted, todo.todos(groceries) restarted
ada adds ' tea ': [ ] tea (ada)
ada adds ' ': { code: 'invalid', reason: 'args' }</code></pre>
<p>The watcher marked <code>@todo/lists</code> dirty when the file changed; <code>applyChanges</code> rebuilt it (one esbuild call per facet)
@ -1200,12 +1200,12 @@ and rolled it out host by host. On each host Chord stages the new code (it runs
</tr>
</thead>
<tbody><tr>
<td>opens no natives, and neither does its new factory</td>
<td><strong>swapped</strong> in place: new methods, the same documents; running calls finish on the old code</td>
<td>live</td>
<td><strong>restarted</strong>: running calls finish on the old code, new calls wait for the new code, the same documents and tables</td>
</tr>
<tr>
<td>opens natives</td>
<td><strong>restarted</strong>: calls wait, running ones finish, natives close and open again with the new code</td>
<td>not live (failed, parked, activating)</td>
<td><strong>retried</strong> on the new code</td>
</tr>
<tr>
<td>its new code waits for a dependency</td>
@ -1216,21 +1216,20 @@ and rolled it out host by host. On each host Chord stages the new code (it runs
<td><strong>ended</strong>; its holders wait until something serves it again</td>
</tr>
</tbody></table>
<p>The store opens no natives, and neither does its new factory: its live query is not a native, Chord runs it. So the
store was <strong>swapped</strong>, not restarted. The sequence on the list&#39;s worker:</p>
<p>Both the store and the view were live, so both <strong>restarted</strong>. The sequence on the list&#39;s worker:</p>
<ol>
<li>The store&#39;s new attempt starts beside the old one: the same object, instance, table and line. Calls arriving now
wait at a gate; calls already running finish on the old code.</li>
<li>The new attempt runs its live query (the same read as at the first activation) and goes live; the gate opens,
and references move to its members by name, each with one frame from its last value.</li>
<li>The old attempt is sealed once its calls settled; its query leaves with it, so later commits run only the new one.</li>
<li>The old attempts stay open: calls already running finish on the old code, and so does every call they make with
their <code>ctx</code> (<strong>old work</strong>: the view&#39;s running <code>add</code> reaches the store&#39;s old attempt, never waiting for the new
one). Calls arriving now are new work: they wait at a gate.</li>
<li>Each old attempt is sealed once its calls drained and the requests its new factory made in staging are ready (the
view&#39;s new code needs the store, whose old attempt still serves it). That takes as long as the running calls, at
most <code>reloadDrainTimeout</code> (5 s); a call still running then is cut.</li>
<li>The new attempt runs its factory, its live query runs again (the same read as at the first activation), and it
goes live: the gate opens, and references move to its members by name, each with one frame from its last value.</li>
</ol>
<p>The views were swapped too, and before the store: Chord starts the fates dependents first, so each view&#39;s new
attempt found the store still live, read its old values until step 2 and moved to its new members with one frame;
its calls to the store waited at the gate. Had the store gone first, the views would have found it between steps 1
and 2 and <strong>parked</strong> for that moment. Ada&#39;s reference never showed <code>loading</code>, and her next add was trimmed.
Had the store kept its window as a one-time read in <code>self.open</code>, it would count as holding a native and <strong>restart</strong>
at every reload of its plugin: running calls drained, natives closed and opened again, a new instance.</p>
<p>Ada&#39;s reference stayed <code>live</code> while the old code drained, showed <code>loading</code> for a moment, then the new code&#39;s values;
her next add was trimmed. Long work does not belong in a method: a call longer than the drain bound is cut at a
reload, so it goes into a job.</p>
<p>Ada&#39;s blank todo is now empty after trimming, so the store refused it with
<code>invalid(&quot;args&quot;, …)</code> (chapter II): her client got <code>{ code: &quot;invalid&quot;, reason: &quot;args&quot; }</code>, Chord&#39;s own error, which
crossed the worker&#39;s link and the server&#39;s as it was. Had the edit broken the plugin, that plugin alone would have kept its old code

5
packages/chord/src/data/errors.ts
View file

@ -18,8 +18,9 @@ export const CHORD_ERROR_REASONS = {
// `discarded`: a transaction cut at the seal.
released: ["ended", "closed", "settled", "version", "discarded"],
too_large: ["part"],
// `undeclared`: a registry no plugin serves on this host.
unavailable: ["locked", "park_timeout", "busy", "storage_failed", "undeclared"],
// `undeclared`: a registry no plugin serves on this host; `version`: waiting for a dependency refused for its version
// (it heals with another host's reload); `reload_cycle`: old work waiting on its own drain during a reload.
unavailable: ["locked", "park_timeout", "busy", "storage_failed", "undeclared", "version", "reload_cycle"],
aborted: [],
} as const;

7
packages/chord/src/data/failure.ts
View file

@ -52,7 +52,12 @@ export function waitingFor(blocker: string, error: ChordError): FailureCause {
if (label.length + root.length > room) label = clipped(label, room - root.length);
}
const message = `${PREFIX}${label}${SEPARATOR}${root}`;
return { kind: "waiting", error: new ChordError({ code: "unavailable", message }, { cause: error }) };
// A version refusal, here or along the chain, keeps its reason: it heals with another host's reload.
const version = error.code === "contract_mismatch" || (error.code === "unavailable" && error.reason === "version");
const init = version
? ({ code: "unavailable", reason: "version", message } as const)
: ({ code: "unavailable", message } as const);
return { kind: "waiting", error: new ChordError(init, { cause: error }) };
}
/** `text` within `length` characters, clipped with "…" when longer. */

25
packages/chord/src/host/README.md
View file

@ -124,7 +124,7 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded:
```
- `createHost<P, R>({ address?, principal?, principalKey?, telemetry?, onError, closeTimeout?, readyTimeout?,
heartbeat?, revokedBy?, compression? })`; `R` is the app's revocation type (JSON). `address` (`Session({ id: "S1" })`, see `contracts/`'s `hostKind`) makes the host addressable: it serves only
reloadDrainTimeout?, heartbeat?, revokedBy?, compression? })`; `R` is the app's revocation type (JSON). `address` (`Session({ id: "S1" })`, see `contracts/`'s `hostKind`) makes the host addressable: it serves only
objects placed there; without one it is a client. A host with a `principal` acts as it for its own requests; a host
without one (a client) has none. A host serves remote types only with both a principal and an address (`serve` of a
remote type throws at `load` otherwise).
@ -132,7 +132,9 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded:
to its part of an identity (L1's canonical JSON by default; the same function on every host). `closeTimeout` (30 s)
bounds each native's close and the drain of an ending object's calls; `readyTimeout` (60 s) bounds every activation
attempt (its wait for the previous instance's teardown included; paused while it is parked on its dependencies) and
is the default deadline of a `get` and of `host.registry`. `heartbeat: { ping?, close? }`: a link silent for `ping` ms (15 s)
is the default deadline of a `get` and of `host.registry`. `reloadDrainTimeout` (5 s, at most `closeTimeout`) bounds
a restart's old attempt at a reload: its old work and its successor's readiness (`host.reload` below).
`heartbeat: { ping?, close? }`: a link silent for `ping` ms (15 s)
sends a `ping`, one silent for `close` ms (30 s), its handshake included, is closed (its references go `loading`);
one interval timer per host (`min(ping, close) / 10`), running only while it has links. A handler that blocks the
event loop longer than `close` gets its links closed by its peers; its own late tick counts the time it did not run
@ -195,7 +197,7 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded:
(`object`, `member?`, `error`: a derive's compute threw, once per failure episode, watched or not; `member` when it is
an exposed state member), `query_failed` (`object`, `table`, `error`: a live query's run in a commit threw, once per
failure episode; the query is a failed member until a good run), `close_failed` (`object`, `native` position,
`error`: a close that threw while nobody waited), `close_timeout` (`object`, `native`: a close still running at its `closeTimeout`, a possible leak), `drain_timeout` (`object`: calls still running at `closeTimeout`; their natives close anyway),
`error`: a close that threw while nobody waited), `close_timeout` (`object`, `native`: a close still running at its `closeTimeout`, a possible leak), `drain_timeout` (`object`: calls still running at `closeTimeout`, or at `reloadDrainTimeout` for a reload's restart; their natives close anyway),
`listener_error` (a throwing watch listener), `delivery_failed` (a frame delivery or a link's flush that threw) and
`protocol_error` (`error`: what closed a link, see `link/`) and `resolve_failed` (`address`: its key, `error`: a
route's `resolve` rejected; the address's holders fail `waiting` with it), `routine_failed` (`object`, `routine`: its position,
@ -273,7 +275,18 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded:
because their new code failed before (a build, an evaluation: the plugins component): the same for the closure rule.
The report lists them all in `failed`; the rest commits. `ctx`'s abort before the commit rejects `aborted` and
changes nothing; after it, it ends only the wait for the report: `aborted` with `details.committed`
(`CommittedReload`: `added`, `changed`, `removed`, `failed` by plugin name), the commit standing.
(`CommittedReload`: `added`, `changed`, `removed`, `failed` by plugin name), the commit standing. The report's
`objects` give each affected object's fate: `ended` (its type removed; holders wait `not_served`), `retried` (it was
not live), `restarted`, or `parked { cause }` (a restart whose successor's recorded requests found a failure, a
retry whose new attempt waits for a dependency). Every live object restarts (decision "Reload restarts every
affected object; old work finishes on old code"): **old work** (the calls running at the commit and every call they
cause through their `ctx`, on this host or, marked `old`, over a trusted link) runs on the open old attempt; a getter
of old work for a version the new code dropped binds that attempt or fails at once; new calls wait at the gate for
the new code. The old attempt is sealed once that work drained and its successor's requests (as staging recorded
them) are ready, bounded by `reloadDrainTimeout` (a method still running then is cut `released`); then its natives
close (`reload`) and the successor runs its factory. Consumers whose version the old attempt binds see the object
`live` meanwhile. A wait of old work that its own old attempt blocks fails at once with `unavailable/reload_cycle`.
A root opened inside a method takes the method's `ctx` (`withoutAbortSignal(ctx)`), so its calls are old work too.
- `host.hold(Type, params, ctx, { readyTimeout? })`: a reference at once (`Held<C>`: its `variant` is `undefined` until
its first binding), which follows its object until released and never rejects: `loading`
while the object activates, retries or is parked on dependencies under way, while there is no parent link or it is
@ -378,7 +391,7 @@ server.accept(webSocketLink(ws), { principal, limits: { part: 16 * MiB, decoded:
at declaration (`invalid/args`) without a positive integer `limit`, for a query of another table, or a table this
factory did not declare. A run that fails (a row its code cannot read) makes it a failed member, as a throwing
derive: one failure call, `query_failed` once, `[["r", rows]]` at the next good run; the object stays live. It is
no native: an object with live queries swaps on reload.
no native: a reload's restart runs the successor's live queries again at its activation.
- A table or document the code cannot read (a newer version, a history edited in place, a failed `rows` step or
`migrate`) fails the object `broken`; a live object whose read fails so fails while live (`object_failed`), keeping
its documents for the next attempt.
@ -542,7 +555,7 @@ alive:
- **Drain in the background:** `close` detaches the old resource and returns at once; the resource disposes of itself
when its work has finished, or after the native's `closeTimeout` at most (code mode's sandbox). Chord then opens the
new one immediately; callers look the current resource up when they start (`watcher.value` at each use).
- **Swap through a registry** (slice 8): the resource is a registry contribution; readers always find the current one,
- **Replace through a registry** (slice 8): the resource is a registry contribution; readers always find the current one,
and a reader that must wait for one waits, cancellably, until one is present.
A native that follows a relocation (design §6) is not restarted at all: the native is a stable manager opened once; a

116
packages/chord/src/host/attempt.ts
View file

@ -1,12 +1,12 @@
// Attempts (§6, §13, §14, §15): one activation of an object, a fresh instance each time. It is the `self` its factory
// declares with (handles to the object's documents and jobs, under its seal token); it runs the activation steps,
// each failing with its own kind; it takes the live object's calls and starts its routines and jobs; and it ends with
// its seal (slice 6a §1.3), the drain and the closes of its natives. A reload's swap hands it over (slice 7b §5): its
// routines stop and hand over to the new attempt's, its jobs to the new attempt's, and it is sealed once its calls
// settled.
// its seal (slice 6a §1.3), the drain and the closes of its natives. A reload's restart drains it (decision "Reload
// restarts every affected object; old work finishes on old code"): its routines stop, it serves old work under its
// own contract, and it is sealed once that work settled.
import type { TelemetryContext, TelemetrySpan } from "@earendil-works/pi-telemetry";
import { awaitWithContext, type Context, withAbortSignal, withTelemetry } from "../context/index.ts";
import { awaitWithContext, type Context, withAbortSignal, withContextValue, withTelemetry } from "../context/index.ts";
import type { Contract, Implementation, ParamsOf } from "../contracts/index.ts";
import { contractData, type ShapeOf } from "../contracts/internal.ts";
import { ChordError, type FailureCause, type Principal } from "../data/index.ts";
@ -60,6 +60,7 @@ import {
type NativeSlot,
} from "./native.ts";
import type { ChordObject, Served } from "./object.ts";
import { OPENING } from "./old-work.ts";
import type { Failed, RefState } from "./ref-state.ts";
import type { Binding, CheckedParams, Reference, RetryWalk } from "./reference.ts";
import { type RetrySource, retrySource } from "./retry-on.ts";
@ -191,11 +192,11 @@ type Work = Map<Promise<unknown>, (() => void) | undefined>;
type RoutineRun = { readonly abort: AbortController; readonly calls: Work };
/**
* What a reload did to an attempt: `started` (a reload started it), `handedOver` (a swap: its jobs are the new
* attempt's, its invocations finish on their own), `callsDrained` (a swap or restart waited for its calls: its end's
* drain does not again), `routinesStopped` (its routines stopped, and do not start late after a handover).
* What a reload did to an attempt: `started` (a reload started it), `callsDrained` (a restart drains its calls: old
* work runs on it until its seal, and its end's drain does not wait for them again), `routinesStopped` (its routines
* stopped, and do not start late).
*/
type Reloaded = { started: boolean; handedOver: boolean; callsDrained: boolean; routinesStopped: boolean };
type Reloaded = { started: boolean; callsDrained: boolean; routinesStopped: boolean };
/** A first dependency check that did not park: the shared answer. */
const NOT_PARKED = { done: true, cause: undefined } as const;
@ -317,8 +318,8 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
*/
readonly #running: Work = new Map();
/**
* Its running job invocations, as `#running`: the drain at its end waits for them; a swap does not (they finish on
* their old code, slice 7b §8). Made by the first.
* Its running job invocations, as `#running`: the drain at its end waits for them (a restart's drain does not: they
* are cut at its seal). Made by the first.
*/
#runs: Work | undefined;
/** What activation returns once the attempt was abandoned: a timeout, a cycle through its object, or its end. */
@ -334,8 +335,8 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
/** Its registry reads, contributions, wake signals and writables, made by the first: most attempts have none. */
#registryDeclarations: RegistryDeclarations | undefined;
/**
* Its first dependency check, for a reload (its report, a swap's handover): waited for by someone, or done with why
* it parked (`undefined`: it did not). Made by the first who asks or by the check.
* Its first dependency check, for a reload (its report, its predecessor's references): waited for by someone, or
* done with why it parked (`undefined`: it did not). Made by the first who asks or by the check.
*/
#check:
| { readonly done: false; readonly waiting: PromiseWithResolvers<FailureCause | undefined> }
@ -348,7 +349,8 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
this.#object = object;
this.#served = object.served;
this.#context = context;
this.#opening = withAbortSignal(this.#abort.signal, context);
// Its opener's calls are known as its object's: their gate waits are the object's edges for reload cycles.
this.#opening = withContextValue(OPENING, object, withAbortSignal(this.#abort.signal, context));
this.#trace = trace;
}
@ -691,12 +693,8 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
this.abandon({ kind: "failed", cause: { kind: "transient", error } });
}
/**
* The object went live: its jobs take starts and its routines start, in this turn; both stop when it is sealed.
* After a swap (`previous`, the attempt it replaced) its routines start once the previous attempt's routine calls
* finished (the handover per object, §14), calls never wait for it.
*/
live(previous?: Attempt): void {
/** The object went live: its jobs take starts and its routines start, in this turn; both stop when it is sealed. */
live(): void {
this.#isLive = true;
const early = this.#endedEarly;
if (early !== undefined) {
@ -704,9 +702,7 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
return;
}
for (const job of this.#jobs.values()) job.live();
if (this.#routines.empty) return;
if (previous === undefined) this.#startRoutines();
else void previous.handOverRoutines().then(() => this.#startRoutines());
if (!this.#routines.empty) this.#startRoutines();
}
/** Starts its routines, unless it was sealed meanwhile. */
@ -730,17 +726,7 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
this.#routineRun?.abort.abort(reason);
}
/**
* A reload swaps this live attempt out (slice 7b §5): its routines stop, its jobs are handed over (their running
* invocations finish on this code), it takes no new starts; its calls run on until `drainCalls` settled.
*/
handOver(): void {
this.#reload().handedOver = true;
this.stopRoutines();
for (const job of this.#jobs.values()) job.handOver();
}
/** Its routines stop (their ctx aborted); their running calls are waited for by the drain or the handover. */
/** Its routines stop (their ctx aborted); their running calls are waited for by the drain. */
stopRoutines(): void {
this.#reload().routinesStopped = true;
this.#stopRoutines(
@ -749,32 +735,40 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
}
/**
* The new attempt's routines may start: once this attempt's routine calls finished, bounded by `closeTimeout` (a
* stuck one is reported `drain_timeout`; the whole attempt is sealed by then: the swap's retirement drains its
* calls and routine calls on the same bound from the swap's start, then seals it, so a stuck call's later commit
* is refused `released`).
* A restart drains it: from now on it serves old work (calls caused by calls on old attempts, routed by its object)
* until its seal, counted as the host's old work.
*/
async handOverRoutines(): Promise<void> {
const { host, label } = this.#object;
const calls = this.#routineRun?.calls ?? NO_WORK;
if ((await drained(calls.keys(), calls.size, host.timeouts.close)) === "timeout") {
host.report({ kind: "drain_timeout", object: label });
beginDrain(): void {
this.#reload().callsDrained = true;
this.#object.host.oldWork.open();
}
/** Whether a restart drains it and it is not sealed yet: old work runs on it. */
get draining(): boolean {
return this.#reloaded?.callsDrained === true && this.#abandoned === undefined;
}
/**
* Waits for its running calls and routine calls (not its job invocations), bounded by `closeTimeout`
* (`drain_timeout` reported): a reload's swap or restart lets them finish before the seal. Its end's drain then
* waits for them no more.
* Waits until no call or routine call runs on it, routed old work included (it joins while the drain runs), or until
* `due` (`Date.now()` time; `drain_timeout` reported). Its end's drain then waits for them no more.
*/
async drainCalls(): Promise<void> {
async drainCalls(due: number): Promise<void> {
const { host, label } = this.#object;
for (;;) {
const calls = [...this.#running.keys(), ...(this.#routineRun?.calls ?? NO_WORK).keys()];
this.#reload().callsDrained = true;
if ((await drained(calls, calls.length, host.timeouts.close)) === "timeout") {
if (calls.length === 0) return;
const left = due - Date.now();
if (left <= 0 || (await drained(calls, calls.length, left)) === "timeout") {
host.report({ kind: "drain_timeout", object: label });
return;
}
}
}
/** The type it was made with: its contract and provider (old work and its consumers use it by that contract). */
get served(): Served {
return this.#served;
}
/** A reload started it: once live, its object retries its blocked job runs (slice 7b §8). */
startedByReload(): void {
@ -788,7 +782,7 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
/** What a reload did to it, made by the first such step. */
#reload(): Reloaded {
this.#reloaded ??= { started: false, handedOver: false, callsDrained: false, routinesStopped: false };
this.#reloaded ??= { started: false, callsDrained: false, routinesStopped: false };
return this.#reloaded;
}
@ -797,14 +791,14 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
return this.#jobs.has(name);
}
/** Whether its factory opens natives (`self.open`): a reload restarts it instead of swapping it. */
/** Whether its factory opens natives (`self.open`): a restart's contributions then leave at its seal (F-b). */
get opensNatives(): boolean {
return this.#natives.length > 0;
}
/**
* Its first dependency check: why it parked (`parkedCause`), `undefined` when it went on (its dependencies live) or
* its activation ended before (a reload's report and a swap's handover wait for it).
* its activation ended before (its predecessor's references are released once it is done).
*/
get checked(): Promise<FailureCause | undefined> {
const check = this.#check;
@ -867,7 +861,8 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
let handler: Method | undefined;
try {
context.abortSignal?.throwIfAborted();
const { served } = this.#object;
// Its own served type: an old attempt answers old work by the contract it was made with.
const served = this.#served;
const own = this.#members!.methods.get(method);
if (own === undefined && version === served.contract.version) throw this.#noMethod(method);
checked = checkArgs(served.provider, method, args, version, context);
@ -889,6 +884,7 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
span === undefined ? callContext : withTelemetry(span.span, callContext),
acting,
host,
this,
);
// Admitted to the drain before the handler runs: the handler may end its own object before it returns.
const running = Promise.withResolvers<unknown>();
@ -1035,6 +1031,8 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
* stay readable.
*/
seal(error: ChordError, runs: CutReason = "restart"): void {
// A drained old attempt stops counting as old work at its seal (once: an abandoned attempt is not draining).
if (this.draining) this.#object.host.oldWork.close();
this.abandon({ kind: "ended", error });
const { documents } = this.#object;
documents.fence(this.#token);
@ -1049,12 +1047,11 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
const { host, label } = this.#object;
if ((await this.#drained()) === "timeout") host.report({ kind: "drain_timeout", object: label });
// The close writer: the persisted runs the seal stopped, and the held state changes (slice 6b §4.2).
const handedOver = this.#reloaded?.handedOver === true;
if (!handedOver) for (const job of this.#jobs.values()) await job.writeAtClose(this.#context);
for (const job of this.#jobs.values()) await job.writeAtClose(this.#context);
// Work stuck past the drain keeps running, but no longer counts as a call or holds its `ctx.get` references.
cleanUp(this.#running);
cleanUp(this.#routineRun?.calls);
if (!handedOver) cleanUp(this.#runs);
cleanUp(this.#runs);
const failures: CloseFailed[] = [];
for (const native of [...this.#natives].reverse()) {
const failure = await native.close(reason);
@ -1064,14 +1061,13 @@ export class Attempt implements StoreSelf, KeyDeclarer, RegistryWaiter {
}
/**
* The drain at its end, bounded by `closeTimeout`: its running calls (unless a reload's `drainCalls` waited for them
* already), routine calls and job invocations (unless a swap handed its jobs over: they finish on their own).
* The drain at its end, bounded by `closeTimeout`: its running calls (unless a restart's `drainCalls` waited for them
* already), routine calls and job invocations.
*/
#drained(): Promise<"settled" | "timeout"> {
const { close } = this.#object.host.timeouts;
const reloaded = this.#reloaded;
const running = reloaded?.callsDrained === true ? NO_WORK : this.#running;
const runs = reloaded?.handedOver === true ? undefined : this.#runs;
const running = this.#reloaded?.callsDrained === true ? NO_WORK : this.#running;
const runs = this.#runs;
const routineCalls = this.#routineRun?.calls;
// Most attempts have only calls: no list is made for them.
if (routineCalls === undefined && runs === undefined) return drained(running.keys(), running.size, close);

32
packages/chord/src/host/ctx.ts
View file

@ -1,6 +1,6 @@
// The `ctx` a method handler gets (§8): its call's context, who called, the calling host's incarnation, call-scoped
// references and child spans (`ctx.trace`, slice 9); and the abort signal of a call's context, linked to long-lived
// sources until it settles.
// references and child spans (`ctx.trace`, slice 9), its node of the call chain (old work during a reload); and the
// abort signal of a call's context, linked to long-lived sources until it settles.
import type { SpanAttributes, TelemetrySpan } from "@earendil-works/pi-telemetry";
import { type Context, type ContextKey, withAbortSignal, withoutAbortSignal, withTelemetry } from "../context/index.ts";
@ -8,6 +8,7 @@ import type { Contract, ParamsOf } from "../contracts/index.ts";
import { ChordError, type Principal } from "../data/index.ts";
import { ended, untraced } from "../telemetry.ts";
import type { AnyHost, GetOptions } from "./host.ts";
import { CALL_CHAIN, type CallNode, type OldWorkSource } from "./old-work.ts";
import type { Binding, Reference } from "./reference.ts";
/** A handler's context. */
@ -33,22 +34,28 @@ export interface Ctx<P extends Principal = Principal> extends Context {
/** Who makes a call: the principal it acts as and its host's incarnation. */
export type Caller = { readonly principal: Principal | undefined; readonly incarnation: string };
/** The ctx of one handler run (or root scope): its references are released by `settle` when it settled. */
export class CallContext implements Ctx {
/**
* The ctx of one handler run (or root scope): its references are released by `settle` when it settled. A method
* call's ctx is its own node of the call chain (`on`: the attempt it runs on), so the chain costs no allocation.
*/
export class CallContext implements Ctx, CallNode {
readonly principal: Principal | undefined;
readonly incarnation: string;
readonly #context: Context;
readonly #host: AnyHost;
/** The attempt a method call runs on; `undefined` for a root scope, a routine or a job run (no chain node). */
readonly #on: OldWorkSource | undefined;
/** The references taken with `get`; `undefined` once the handler settled. */
#references: Binding[] | undefined = [];
/** Aborted when the handler settles: a `get` still waiting gives up and releases its reference. By the first. */
#settling: AbortController | undefined;
constructor(context: Context, caller: Caller, host: AnyHost) {
constructor(context: Context, caller: Caller, host: AnyHost, on?: OldWorkSource) {
this.#context = context;
this.principal = caller.principal;
this.incarnation = caller.incarnation;
this.#host = host;
this.#on = on;
}
get abortSignal(): AbortSignal | undefined {
@ -60,9 +67,21 @@ export class CallContext implements Ctx {
}
value<T>(key: ContextKey<T>): T | undefined {
// A method call's ctx is its chain node; the key's type is `CallNode`, which this is.
if (key.token === CALL_CHAIN.token && this.#on !== undefined) return this as unknown as T;
return this.#context.value(key);
}
/** The attempt its handler runs on (only read through `CALL_CHAIN`, which a method call's ctx answers). */
get on(): OldWorkSource {
return this.#on!;
}
/** The chain of the call that made this one. */
up(): CallNode | undefined {
return this.#context.value(CALL_CHAIN);
}
toString(): string {
return `${this.#context}.Ctx`;
}
@ -79,7 +98,8 @@ export class CallContext implements Ctx {
// Made for the caller: its principal and its origin's incarnation travel with the reference's holds.
const caller = { principal: this.principal, incarnation: this.incarnation };
this.#settling ??= new AbortController();
const scoped = withLinkedAbortSignal(this.#settling.signal, this.#context);
// Under this ctx itself: the getter belongs to the call's chain (old work during a reload).
const scoped = withLinkedAbortSignal(this.#settling.signal, this);
let binding: Binding;
try {
binding = await this.#host.acquire(Type, params, caller, scoped.context, options);

161
packages/chord/src/host/host.ts
View file

@ -1,8 +1,8 @@
// Hosts (§3): `createHost`, loading facets and the start barrier, links to children, the parent (connected again after
// a drop) and routes, the origins it relays holds for, getting and holding references as a principal, the heartbeat
// of every link, revocation (§4), presence (§19) and keep-alive counts, registries and root scopes (slice 8), reloads
// (slice 7b: staging, the commit, each object's fate, the report), closing; one batch per host, the backoff map and the
// endings new instances wait for.
// (staging, the commit, each object's fate, the report; old work and its reload cycles), closing; one batch per host,
// the backoff map and the endings new instances wait for.
import {
NOOP_TELEMETRY_CONTEXT,
@ -69,8 +69,10 @@ import type { Facet } from "./facet.ts";
import { contributionOf } from "./members.ts";
import type { CloseFailed } from "./native.ts";
import { ChordObject, type Holder, type ReloadFate, type Served } from "./object.ts";
import { chainAttempts, OldWork } from "./old-work.ts";
import { type Presence, References } from "./presence.ts";
import { Binding, type CheckedParams, type Held, type Reference, type Target } from "./reference.ts";
import { ReloadCycles } from "./reload-cycle.ts";
import { RevocationKeys, type RevokedBy } from "./revocation.ts";
import { type RootOptions, RootScopes } from "./root.ts";
import { Route, type RouteHandle, type RouteOptions } from "./route.ts";
@ -119,6 +121,12 @@ export type HostOptions<P extends Principal = Principal, R extends JsonValue = J
readonly closeTimeout?: number;
/** How long an activation attempt, and a `get`, may take until the object is live; 60 s by default. */
readonly readyTimeout?: number;
/**
* How long a reload's restart keeps its old attempt open for old work (its calls, routed calls, its successor's
* readiness) before it is sealed and anything still running is cut off (`released`); 5 s by default, at most
* `closeTimeout`. Ends in seconds the waits Chord cannot see (a root opened without the method's `ctx`).
*/
readonly reloadDrainTimeout?: number;
/**
* The heartbeat of every link: a link silent for `ping` ms (15 s by default) sends a `ping`; one silent for `close`
* ms (30 s by default), its handshake included, is closed.
@ -277,14 +285,14 @@ export interface Host<P extends Principal = Principal, R extends JsonValue = Jso
* and so does a changed plugin whose recorded requests would not bind because of it (the closure rule); both are
* reported (`failed`); `options.failed` names plugins the caller passes as their loaded facet because their new
* code failed before (a build, an evaluation): the same for the closure rule and the report. Then it commits in one
* batch (types, registries,
* contributions; a publication waking the requests for the names it serves) and every object of a changed or
* removed plugin takes its fate: swapped in place, restarted after its drain, retried, or ended (its type
* removed). Resolves with the report once every fate started (swaps and retries checked their dependencies; it
* does not wait for objects to go live). Reloads run one after another. Throws before `load` resolved and for a
* name that is no npm name (before anything changes); rejects `released/closed` after `close`; `context`'s abort
* before the commit rejects `aborted` and changes nothing, after it only ends the wait (`aborted`, its
* `details.committed` naming what the commit applied: `CommittedReload`).
* batch (types, registries, contributions; a publication waking the requests for the names it serves) and every
* object of a changed or removed plugin takes its fate: restarted (its old attempt finishing old work first),
* retried, or ended (its type removed). Resolves with the report once every fate started (retries checked their
* dependencies; a restart reports what its readiness found at once; it does not wait for objects to go live).
* Reloads run one after another. Throws before `load` resolved and for a name that is no npm name (before anything
* changes); rejects `released/closed` after `close`; `context`'s abort before the commit rejects `aborted` and
* changes nothing, after it only ends the wait (`aborted`, its `details.committed` naming what the commit applied:
* `CommittedReload`).
*/
reload(facets: readonly PluginFacet<P>[], context: Context, options?: ReloadOptions): Promise<ReloadReport>;
/**
@ -402,12 +410,11 @@ export type ReloadReport = {
};
/**
* One object's fate in a reload: `swapped` (pure, in place), `restarted` (natives: drained, closed with reason
* `reload`, activated again), `ended` (its type is gone), `parked` (its new code waits for a dependency, `cause`), or
* `retried` (it was failed, parked or activating: a new attempt runs on the new code).
* One object's fate in a reload: `restarted` (its old attempt drained, sealed and closed with reason `reload`, a new
* attempt on the new code), `ended` (its type is gone), `parked` (its new code waits for a failed dependency,
* `cause`), or `retried` (it was failed, parked or activating: a new attempt runs on the new code).
*/
export type ReloadedObject = { readonly object: string } & (
| { readonly kind: "swapped" }
| { readonly kind: "restarted" }
| { readonly kind: "ended" }
| { readonly kind: "parked"; readonly cause: FailureCause }
@ -454,15 +461,28 @@ type StagingPass<P extends Principal> =
/** A recorded new factory of an object staging planned a fate for: whose new code it is, and what it declared. */
type RecordedFactory = { readonly plugin: string; readonly recorded: Recorded };
/** An affected object's fate, as staging decided it. */
/** An affected object's fate, as staging decided it, with its new factory's requests as staging recorded them. */
type PlannedFate =
| { readonly kind: "end"; readonly object: ChordObject }
| { readonly kind: ReloadFate; readonly object: ChordObject; readonly served: Served };
| {
readonly kind: ReloadFate;
readonly object: ChordObject;
readonly served: Served;
readonly requests: readonly Request[];
};
/** A fate started by a reload's commit, for its report: a swap's or retry's new attempt is checked first. */
/**
* A fate started by a reload's commit, for its report: a retry's new attempt is checked first, a restart reports what
* its readiness found.
*/
type StartedFate =
| { readonly object: string; readonly kind: "ended" }
| { readonly object: string; readonly kind: "swapped" | "restarted" | "retried"; readonly attempt: Attempt };
| {
readonly object: string;
readonly kind: "restarted" | "retried";
readonly target: ChordObject;
readonly attempt: Attempt;
};
/** Creates a host. */
export function createHost<P extends Principal = Principal, R extends JsonValue = JsonValue>(
@ -483,7 +503,11 @@ const NO_ATTRIBUTES: Readonly<SpanAttributes> = Object.freeze({});
/** The host: its served types, its objects by identity, its batch, backoff and endings. */
export class ChordHost<P extends Principal = Principal, R extends JsonValue = JsonValue> implements Host<P, R> {
readonly batch: Batch;
readonly timeouts: { readonly close: number; readonly ready: number };
readonly timeouts: { readonly close: number; readonly ready: number; readonly reload: number };
/** Its old work during reloads (whether a context runs under it): see `old-work.ts`. */
readonly oldWork = new OldWork();
/** The waits of old work on objects that are not live, cut at once when they cycle through their own drain. */
readonly reloadCycles = new ReloadCycles();
readonly backoff = new Backoff((identity) => this.#due(identity));
readonly address: Address | undefined;
readonly principal: P | undefined;
@ -566,7 +590,9 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
this.#activity = new Activity(this.batch);
// `revokedBy` is called with this host's principals only, the `P`s its links stamp.
this.#revocationKeys = new RevocationKeys(options.revokedBy as RevokedBy<Principal, JsonValue> | undefined);
this.timeouts = { close: options.closeTimeout ?? 30_000, ready: options.readyTimeout ?? 60_000 };
const close = options.closeTimeout ?? 30_000;
const reload = Math.min(close, options.reloadDrainTimeout ?? 5_000);
this.timeouts = { close, ready: options.readyTimeout ?? 60_000, reload };
this.#heartbeat = { ping: options.heartbeat?.ping ?? 15_000, close: options.heartbeat?.close ?? 30_000 };
this.#destination = new Destination(this.address, this.#routes, this.#parent);
// Chord's own: `openStore` made it.
@ -792,7 +818,7 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
try {
const declared = recordFactory(object, served, context);
recorded.push({ plugin: served.plugin, recorded: declared });
fates.push({ kind: object.reloadFate(declared), object, served });
fates.push({ kind: object.reloadFate(), object, served, requests: declared.requests });
} catch (error) {
if (!failing.has(served.plugin)) failing.set(served.plugin, error);
}
@ -900,25 +926,22 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
}
/**
* Starts `fates` (in the order of `#objects`: providers before their dependents, `wentLive`) and returns them in
* that order, for the report. The removals first, then the others dependents first (decision "Swaps: dependents
* first"): a dependent's new attempt binds its provider while that is still live, so it does not park on the
* provider's own swap or restart, and reads the old instance until the new one is live. A removed provider is gone
* before, so a dependent whose new code still needs it parks (decision 4). A provider takes its new served type with
* its own fate: a dependent of a version only that serves waits unbound until then (`ChordObject.hold`), then binds
* the swapping provider and parks.
* Starts `fates` and returns them in their order, for the report: the removals first (a dependent whose new code
* still needs a removed provider then parks, decision 4), then the restarts and retries; then the restarts'
* readiness and successors, once every object has its new served type (their requests bind against the new code).
*/
#startFates(fates: readonly PlannedFate[]): StartedFate[] {
const started = new Map<PlannedFate, StartedFate>();
for (const fate of fates) if (fate.kind === "end") started.set(fate, this.#startFate(fate));
for (const fate of fates.toReversed()) if (fate.kind !== "end") started.set(fate, this.#startFate(fate));
for (const fate of fates) if (fate.kind !== "end") started.set(fate, this.#startFate(fate));
for (const fate of fates) if (fate.kind === "restart") fate.object.startSuccessor();
// Every fate started above.
return fates.map((fate) => started.get(fate)!);
}
/**
* Starts one object's fate after the commit (slice 7b §4): a removed type's object ends, its holders waiting
* `not_served`; any other swaps, restarts or retries on its new served type.
* Starts one object's fate after the commit: a removed type's object ends, its holders waiting `not_served`; any
* other restarts or retries on its new served type.
*/
#startFate(fate: PlannedFate): StartedFate {
const { object } = fate;
@ -930,14 +953,14 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
object.remove({ kind: "waiting", error });
return { object: object.label, kind: "ended" };
}
const attempt = object.reload(fate.served, fate.kind);
return { object: object.label, kind: STARTED[fate.kind], attempt };
const attempt = object.reload(fate.served, fate.kind, fate.requests);
return { object: object.label, kind: fate.kind === "restart" ? "restarted" : "retried", target: object, attempt };
}
/**
* The report, once every swap and retry had its first dependency check (a parked one reports why); restarts are
* reported at once (their check follows the drain). `context`'s abort ends only this wait (`aborted`, its `details`
* naming what the commit applied: `committed`).
* The report, once every retry had its first dependency check (a parked one reports why); a restart reports at once
* what its readiness found (its successor's own check follows the old attempt's end). `context`'s abort ends only
* this wait (`aborted`, its `details` naming what the commit applied: `committed`).
*/
async #report(
started: readonly StartedFate[],
@ -945,8 +968,8 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
context: Context,
): Promise<ReloadedObject[]> {
const checks = started.map(async (fate): Promise<ReloadedObject> => {
if (fate.kind === "ended" || fate.kind === "restarted") return { object: fate.object, kind: fate.kind };
const cause = await fate.attempt.checked;
if (fate.kind === "ended") return { object: fate.object, kind: fate.kind };
const cause = await this.#checked(fate);
return cause === undefined
? { object: fate.object, kind: fate.kind }
: { object: fate.object, kind: "parked", cause };
@ -962,6 +985,18 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
}
}
/**
* Why a restarted or retried object's new attempt waits, for the report: what a restart's readiness found, at once
* (its new attempt's own check follows the old attempt's end), also for a retry of an attempt that still waits for
* an open old attempt; otherwise its new attempt's first dependency check.
*/
#checked(fate: Extract<StartedFate, { readonly kind: "restarted" | "retried" }>): Promise<FailureCause | undefined> {
const { target, attempt } = fate;
if (fate.kind === "restarted" || target.openOldAttempt !== undefined)
return Promise.resolve(target.readinessCause);
return attempt.checked;
}
/** This host as staging sees it: whom it acts as, its address and its presence lists. */
#stagingHost(): StagingHost {
// Setups get presence lists of this host's `P`; staging passes them on as they are.
@ -1079,7 +1114,7 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
references = new References(this.batch);
for (const object of this.#objects.values()) {
if (object.served.contract.name !== Type.name) continue;
for (const holder of object.holders) references.record(object, holder);
for (const holder of object.holders) if (holder.listed) references.record(object, holder);
}
this.#references.set(Type.name, references);
}
@ -1243,7 +1278,10 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
const waiting = withLinkedAbortSignal(expired.signal, context);
const deadline = setTimeout(() => expired.abort(binding.lastFailure ?? notLive(binding.label, timeout)), timeout);
try {
await binding.live(waiting.context);
// Old work's getter whose target's activation waits on that work's own drain fails at once (rule 6).
const target = binding.target;
const live = binding.live(waiting.context);
await (target instanceof ChordObject ? this.waitAsOldWork(live, target, Type, context) : live);
return binding;
} catch (error) {
binding.release();
@ -1254,14 +1292,25 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
}
}
/**
* `waiting` (a getter's or a gate's wait on `target`, for `contract`'s version) as `context` makes it: as old work it
* is cut at once with `unavailable/reload_cycle` when it cycles through that work's own drain (`ReloadCycles`);
* otherwise as it is.
*/
waitAsOldWork<T>(waiting: Promise<T>, target: ChordObject, contract: Contract, context: Context): Promise<T> {
if (!this.oldWork.of(context)) return waiting;
const chain = chainAttempts(context);
return chain.size === 0 ? waiting : this.reloadCycles.wait(waiting, target, contract, chain);
}
/** Adds `delta` to the keep-alive count of `kind` (`host.activity`), where the work happens. */
count(kind: keyof ActivityCounts, delta: number): void {
this.#activity.count(kind, delta);
}
/** `holder` references `object`: listed in its type's presence, if someone asked for it. */
/** `holder` references `object`: listed in its type's presence, if someone asked for it and it is listed. */
referenced(object: ChordObject, holder: Holder, context: Context): void {
this.#references.get(object.served.contract.name)?.add(object, holder, context);
if (holder.listed) this.#references.get(object.served.contract.name)?.add(object, holder, context);
}
/** `holders` of `object` ended: gone from its type's presence. */
@ -1318,6 +1367,14 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
unboundCause(served, Type.kind, Type.name, Type.version, Type.fingerprint) ??
this.#misplaced(Type, served!, params.params);
if (unbound !== undefined) {
// Old work never waits for a code update (rule 2): a version the new code dropped binds the object while its
// open old attempt still serves it, and fails at once otherwise.
if (unbound.error.code === "contract_mismatch" && this.oldWork.of(context)) {
const object = this.#objects.get(this.#identity(served!, params, principal, context));
if (object === undefined || !object.oldServes(Type.version, Type.fingerprint)) throw unbound.error;
binding.bind(object);
return object;
}
this.#wait(binding, Type.name, unbound, () => this.#place(binding, Type, params, context));
return undefined;
}
@ -1564,6 +1621,14 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
return made;
}
/** The identity of `served`'s object for `params` and `principal` (a remote type's includes its principal key). */
#identity(served: Served, params: CheckedParams, principal: Principal | undefined, context: Context): string {
if (served.kind === "local") return identityKey(served.contract, params.key);
// A host serving remote types has a principal, which its references act as unless a link stamped its own (only
// `P`s).
return identityKey(served.contract, params.key, principalKey(principal as P, context, this.#principalKey));
}
/**
* The object for `params` (checked by L1) and, for a remote type, `principal` (its key is part of the identity),
* created if there is none; its first attempt starts at once, or when its identity's backoff window ends. A new one
@ -1580,13 +1645,8 @@ export class ChordHost<P extends Principal = Principal, R extends JsonValue = Js
chain: readonly string[] = NO_CHAIN,
dependent?: ChordObject,
): ChordObject {
const { key, params: checked } = params;
const identity =
served.kind === "local"
? identityKey(served.contract, key)
: // A host serving remote types has a principal, which its references act as unless a
// link stamped its own (only `P`s).
identityKey(served.contract, key, principalKey(principal as P, context, this.#principalKey));
const { params: checked, key } = params;
const identity = this.#identity(served, params, principal, context);
let object = this.#objects.get(identity);
if (object === undefined) {
// Shared by every getter: one getter's cancellation does not cancel it.
@ -1766,9 +1826,6 @@ async function untilUsable(value: RegistryValueSource, usable: () => boolean, co
}
}
/** What the report calls a fate. */
const STARTED = { swap: "swapped", restart: "restarted", retry: "retried" } as const;
/** The `aborted` error of a reload whose context aborted before its commit: nothing changed. */
function reloadAborted(): ChordError {
return new ChordError({ code: "aborted", message: "reload: aborted" });

7
packages/chord/src/host/internal.ts
View file

@ -1,11 +1,12 @@
// Hosts for Chord's layers: what the remote sides of links use of a host, its objects and references; what jobs use of
// a handler's ctx, the backoff and ids.
// Hosts for Chord's layers: what the remote sides of links use of a host, its objects and references (and old work
// arriving from another host); what jobs use of a handler's ctx, the backoff and ids.
export type { PinnedKey } from "./attempt.ts";
export type { Attempt, PinnedKey } from "./attempt.ts";
export { nextDelay } from "./backoff.ts";
export { CallContext, type Caller, SpanCtx } from "./ctx.ts";
export type { AnyHost, ChordHost, Serving } from "./host.ts";
export type { ChordObject, Holder, Removal } from "./object.ts";
export { asRemoteOldWork } from "./old-work.ts";
export { LIVE, LOADING, type RefState } from "./ref-state.ts";
export {
type Attachable,

447
packages/chord/src/host/object.ts
View file

@ -5,8 +5,9 @@
// (the start barrier, decision (a)), a live object's failure (slice 6a §1.4), the waits and cycle checks of activations
// (Q5, its instance id and the activation chain it was created for), lingering after its last reference (§3, D8), the
// job runs that keep it alive (§15), its contributions to registries while live and its wake signals (`retryOn`, slice
// 8), its ending, revocation included (§4, R11), and what a reload does to it (slice 7b §4): a pure swap in place, a
// native restart after the drain, a retry on the new code, or its end by its type's removal, after which its
// 8), its ending, revocation included (§4, R11), and what a reload does to it (decision "Reload restarts every affected
// object; old work finishes on old code"): a restart, whose old attempt serves old work until it drained and its
// successor's requests are ready, a retry on the new code, or its end by its type's removal, after which its
// references wait with their last values and follow the next object of the identity.
import type { TelemetryContext } from "@earendil-works/pi-telemetry";
@ -30,10 +31,12 @@ import type { Caller } from "./ctx.ts";
import type { Dependency } from "./dependencies.ts";
import type { AnyHost } from "./host.ts";
import type { CloseFailed, NativeCloseReason } from "./native.ts";
import { OPENING } from "./old-work.ts";
import { Readiness } from "./readiness.ts";
import { type Ended, type Failed, LIVE, LOADING, type RefState } from "./ref-state.ts";
import { type CheckedParams, type Followed, type RetryWalk, retryWalk, type Target } from "./reference.ts";
import { type RetrySource, RetryWakes } from "./retry-on.ts";
import { migratesTables, type Recorded } from "./staging.ts";
import type { Request } from "./staging.ts";
import { uuidv7 } from "./uuid.ts";
/** A `local` contract, as a served type has it. */
@ -77,6 +80,8 @@ export interface Holder {
readonly caller: Caller;
/** The consumer's contract version and fingerprint (its request's): the provider binds it or it waits unbound. */
readonly contract: { readonly version: number; readonly fingerprint: string };
/** Whether presence lists show it (§19): not a restart's readiness binding, which is no consumer of its own. */
readonly listed: boolean;
/**
* Inside a batch, before `moved`: what it follows of the object changed (`ChordObject.followed`): a reference moves its
* members' subscriptions to the new instance, or keeps their last values; a hold over a link has nothing to do here.
@ -107,15 +112,8 @@ export type Removal = {
readonly context: Context;
};
/** What a reload does to a live, activating, parked or failed object whose type is still served (slice 7b §4). */
export type ReloadFate = "swap" | "restart" | "retry";
/**
* A pure swap in progress (slice 7b §5): the attempt it replaces (its calls run on, then it is sealed), whether the new
* attempt parked (holders then see what it shows, and the old one's contributions left the tables), and the end of
* the swap (its new attempt live, parked or failed, or the object ended), which the old attempt's retirement waits for.
*/
type Swap = { readonly old: Attempt; parked: boolean; readonly done: PromiseWithResolvers<void> };
/** What a reload does to an object whose type is still served: a live one restarts, any other is retried. */
export type ReloadFate = "restart" | "retry";
/**
* Where an object stands. `waiting`: for its identity's backoff window before its first attempt. `activating` has what
@ -128,8 +126,6 @@ type ObjectState =
readonly kind: "activating";
readonly attempt: Attempt;
readonly shown: RefState;
/** Set while a reload's pure swap replaces the live attempt. */
readonly swap?: Swap;
}
| { readonly kind: "live"; readonly attempt: Attempt }
| { readonly kind: "failed"; readonly failure: Failed }
@ -169,8 +165,10 @@ const NO_CONTRIBUTIONS: readonly RegistryContribution[] = [];
* or when the host closes.
*/
export class ChordObject implements Target {
/** An object on this host (a reference's target that is not held over a link). */
readonly kind = "local";
readonly host: AnyHost;
/** Its served type: replaced by a reload that swaps, restarts or retries it. */
/** Its served type: replaced by a reload that restarts or retries it. */
served: Served;
readonly params: ParamsOf<Contract>;
/** Its params' identity key (empty for empty params). */
@ -218,23 +216,39 @@ export class ChordObject implements Target {
/** Its running and finishing job runs: they keep it alive like references (D4), without being references. */
#runs = 0;
/**
* The instance its references follow (`followed`): the live one, the old one while a swap brings its successor in
* place, or the last one that went live while it still changes after a failure or a restart (references take their
* last values from it when it stops); `undefined` before the first went live and once it stopped changing.
* The instance its references follow (`followed`): the live one, or the last one that went live while it still
* changes after a failure or during a restart (references take their last values from it when it stops);
* `undefined` before the first went live and once it stopped changing.
*/
#exposed: Attempt | undefined;
/**
* Whether references still follow `#exposed`, which may still change: while it is live, drains (a restart), retires
* (a swap's old instance until its successor is live or it retired), or failed live until the commits admitted
* before it published (every member, its own and re-exposed ones: audit A10); not once it stopped changing.
* Whether references still follow `#exposed`, which may still change: while it is live, while a restart keeps it
* open for old work, or failed live until the commits admitted before it published (every member, its own and
* re-exposed ones: audit A10); not once it stopped changing.
*/
#following = false;
/** The one timer storing its jobs' held state changes by their earliest deadline (D8), while one is held. */
#flush: { readonly deadline: number; readonly timer: ReturnType<typeof setTimeout> } | undefined;
/** Its wake signals (`retryOn`), the last attempt's declaration; made by the first. */
#wakes: RetryWakes | undefined;
/** Its swapped-out attempts while they retire (bounded by `closeTimeout`); made by the first swap. */
#retiring: Map<Attempt, Promise<void>> | undefined;
/**
* A restart's old attempt while it is open (until it stopped changing): old work runs on it, and consumers whose
* version it binds see it live (decision "Reload restarts every affected object", rules 2 and 5).
*/
#old: Attempt | undefined;
/**
* A restart's old attempt without natives whose contributions stay in the tables until its successor is live (F-c),
* replaced there in one batch; they leave if the successor fails, parks on a failure or the object ends.
*/
#kept: Attempt | undefined;
/** A restart's readiness, until its successor's first dependency check. */
#readiness: Readiness | undefined;
/** A restart's successor that shows its readiness's failures, until the old attempt ended. */
#showsReadiness: Attempt | undefined;
/** Starts a restart's readiness and successor, once every fate of its reload started (`startSuccessor`). */
#startSuccessor: (() => void) | undefined;
/** The objects whose gate its activation's opener waits at now: its wait edges for reload cycles. */
#gateWaits: Set<ChordObject> | undefined;
/** Its live queries by table, across its attempts; made by the first table it declares. */
#liveQueries: LiveQueries | undefined;
/** Resolves at its next change of state, for calls waiting at the gate; made by the first one. */
@ -284,9 +298,39 @@ export class ChordObject implements Target {
return state.kind === "activating" ? state.shown : LOADING;
}
/** Whether its state members have values: once live (a local object stays live until it ends). */
/**
* Whether its state members have values: once live (a local object stays live until it ends), and while a
* restart's old attempt is open (its consumers check their version through `stateFor`).
*/
get hydrated(): boolean {
return this.#state.kind === "live";
return this.#state.kind === "live" || this.openOldAttempt !== undefined;
}
/**
* The state a same-host consumer of `version` and `fingerprint` sees (rule 5): `live` while a restart's open old
* attempt binds that version and its successor is not parked on a failure; otherwise `state`.
*/
stateFor(version: number, fingerprint: string): RefState {
const state = this.#state;
if (state.kind === "activating" && state.shown.kind !== "failed" && this.oldServes(version, fingerprint)) {
return LIVE;
}
return this.state;
}
/** Whether a restart's open old attempt binds `version` and `fingerprint`: its consumers of that version may use it. */
oldServes(version: number, fingerprint: string): boolean {
const old = this.openOldAttempt;
return old !== undefined && binds(old.served.provider, version, fingerprint);
}
/**
* A restart's old attempt while it is open (draining, not sealed yet): old work runs on it, and consumers whose
* version it binds see it as the live instance; `undefined` otherwise.
*/
get openOldAttempt(): Attempt | undefined {
const old = this.#old;
return old?.draining ? old : undefined;
}
/** Resolves at its first outcome (the start barrier waits for it); never rejects. */
@ -344,9 +388,12 @@ export class ChordObject implements Target {
return this.#holders;
}
/** The variant a live remote object's factory chose; `undefined` for a local object and before it is live. */
/**
* The variant a live remote object's factory chose (a restart's open old attempt's meanwhile); `undefined` for a
* local object and before it is live.
*/
get variant(): string | undefined {
return this.#state.kind === "live" ? this.#state.attempt.variant : undefined;
return this.#state.kind === "live" ? this.#state.attempt.variant : this.openOldAttempt?.variant;
}
/** The live object's state members, as Chord follows them. */
@ -401,7 +448,8 @@ export class ChordObject implements Target {
/**
* Calls a method of the live object for `caller`, bound at `version` (an older one maps its arguments and result,
* Q10), its `chord.method` span under `parent` (the caller's span on this host, a call's carrier over a link). While
* an attempt activates (a reload's swap or restart), the call waits for its outcome (the gate).
* an attempt activates (a reload's restart, a retry), new work waits for its outcome (the gate); old work runs on a
* restart's open old attempt.
*/
async call(
method: string,
@ -411,17 +459,17 @@ export class ChordObject implements Target {
version: number,
parent?: TelemetryContext,
): Promise<unknown> {
if (this.#state.kind === "activating") await this.#gate(context);
return this.#live().call(method, args, context, caller, version, parent ?? context.telemetry);
if (this.#state.kind === "activating") {
// Old work runs on the open old attempt (rule 2): work begun before the reload finishes on old code, and a
// restart's drain never waits on itself. The old attempt checks the call's version against its own contract.
const old = this.openOldAttempt;
if (old !== undefined && this.host.oldWork.of(context)) {
return old.call(method, args, context, caller, version, parent ?? context.telemetry);
}
/**
* Whether a reload's pure swap runs in place: the new attempt neither parked nor failed yet. Holds stay bound
* meanwhile (their calls wait at the gate) and get a fresh `bound` once it is live.
*/
get swapsInPlace(): boolean {
const state = this.#state;
return state.kind === "activating" && state.swap !== undefined && !state.swap.parked;
// Old work here found no open old attempt (else it was routed above): any version cycles through the gate.
await this.host.waitAsOldWork(this.#openerGate(context), this, this.served.contract, context);
}
return this.#live().call(method, args, context, caller, version, parent ?? context.telemetry);
}
/** Whether a consumer of `contract`'s version and fingerprint binds to this object's provider now (R6, Q10). */
@ -437,7 +485,7 @@ export class ChordObject implements Target {
/**
* What a reference of `version` follows for state member `member` now (decision "References are the only
* indirection"): the shown instance's member (its mapping at an older version) while that instance is live, while a
* swap brings its successor in place, and after a live failure until its admitted commits published; else nothing:
* restart keeps it open, and after a live failure until its admitted commits published; else nothing:
* `failed` with the cause's error, or `loading`, with that member to take a last value from.
*/
followed(member: string, version: number): Followed {
@ -449,7 +497,7 @@ export class ChordObject implements Target {
shown = exposed.memberAt(member, version);
} catch (error) {
const failure = asChordError(error, this.label);
// The shown instance does not serve this version (a swap's old one, while a reload brings it): nothing yet.
// The shown instance does not serve this version (a restart's old one, while a reload brings it): nothing yet.
if (failure.code !== "contract_mismatch") lacking = failure;
}
}
@ -477,9 +525,12 @@ export class ChordObject implements Target {
return this.#following ? this.#exposed : undefined;
}
/** The live instance, which references pin their keys on (a new one: they pin again); `undefined` when not live. */
/**
* The live instance, which references pin their keys on (a new one: they pin again), or a restart's open old
* attempt; `undefined` otherwise.
*/
get instance(): Attempt | undefined {
return this.#state.kind === "live" ? this.#state.attempt : undefined;
return this.#state.kind === "live" ? this.#state.attempt : this.openOldAttempt;
}
/** The live instance's state member `member` as a consumer of `version` follows it (a hold over a link). */
@ -492,12 +543,43 @@ export class ChordObject implements Target {
return this.#live().keyMapping(member, version);
}
/**
* The gate, with an activation's opener waiting here recorded as an edge of its object (a wait of old work on that
* object then sees it), the waits of old work checked again when one starts.
*/
async #openerGate(context: Context): Promise<void> {
const opener = context.value(OPENING);
if (opener === undefined) return this.#gate(context);
opener.#gateWaits ??= new Set();
const waits = opener.#gateWaits;
waits.add(this);
this.host.reloadCycles.recheck();
try {
await this.#gate(context);
} finally {
waits.delete(this);
}
}
/**
* What its activation waits for now (reload cycles, rule 6): the objects its parked attempt's dependencies are bound
* to, and those whose gate its opener waits at.
*/
waitEdges(): ChordObject[] {
const edges = [...(this.#gateWaits ?? [])];
for (const dependency of this.#waitingFor) {
const target = dependency.kind === "reference" ? dependency.binding.target : undefined;
if (target instanceof ChordObject) edges.push(target);
}
return edges;
}
/**
* Waits while an attempt activates until the object is live, failed or ended (`#live` then calls or throws);
* rejects with the cause once the attempt parks on a failed dependency (`failed { waiting }`), and with
* `unavailable/park_timeout` after `readyTimeout`, as a call parked on a reference. A hold over a link stays bound
* while the old instance still changes (a swap, a restart's drain), so its consumer sees `live` and sends calls
* that arrive here: they wait for the new attempt, never `not_ready` (decision "Swaps: dependents first").
* while a restart's old attempt is open, so its consumer sees `live` and sends calls that arrive here: new work
* waits for the new attempt, never `not_ready`.
*/
async #gate(context: Context): Promise<void> {
const { ready } = this.host.timeouts;
@ -724,16 +806,18 @@ export class ChordObject implements Target {
parked(attempt: Attempt, shown: RefState, firstPass: boolean): void {
const state = this.#state;
if (state.kind !== "activating" || state.attempt !== attempt) return;
const { swap } = state;
// A swap's new attempt parks (decision 4): no overlap of old and new code, holders see what it shows.
const parks = swap !== undefined && !swap.parked;
if (parks) this.#swapParked(swap);
if (parks || !sameState(state.shown, shown)) {
this.#state = { kind: "activating", attempt, shown, ...(swap === undefined ? {} : { swap }) };
if (shown.kind === "failed") this.#lastFailure = shown.cause.error;
if (!sameState(state.shown, shown)) {
this.#state = { kind: "activating", attempt, shown };
if (shown.kind === "failed") {
this.#lastFailure = shown.cause.error;
// A restart's successor parked on a failure: the old attempt's kept contributions leave (F-c).
this.#dropKept();
}
this.#notify();
}
if (firstPass) this.#settleFirstOutcome();
// A park may close a cycle with a wait of old work (rule 6).
this.host.reloadCycles.recheck();
}
/** Its attempt `attempt` left parking for its own steps: it shows `loading` until its outcome. */
@ -743,11 +827,12 @@ export class ChordObject implements Target {
if (!sameState(state.shown, LOADING)) this.parked(attempt, LOADING, false);
}
/** A swap's new attempt parked: the old attempt's contributions leave the tables (it is not live any more). */
#swapParked(swap: Swap): void {
swap.parked = true;
this.#withdraw(swap.old);
swap.done.resolve();
/** The contributions a restart kept for its successor leave the tables (F-c): it will not take them over now. */
#dropKept(): void {
const kept = this.#kept;
if (kept === undefined) return;
this.#kept = undefined;
this.#withdraw(kept);
}
/**
@ -779,119 +864,136 @@ export class ChordObject implements Target {
if (teardown.kind !== "none") this.host.ending(this.identity, this.#reported(this.#tearDown(teardown, END)));
}
/**
* What a reload with `recorded` (the new factory's declarations) does to this object, whose type is still served:
* a live object swaps in place when neither its live attempt nor the new factory opens natives and no table of the
* new factory is newer than the store's, else it restarts; any other is retried.
*/
reloadFate(recorded: Recorded): ReloadFate {
const state = this.#state;
if (state.kind !== "live") return "retry";
if (state.attempt.opensNatives || recorded.natives > 0 || migratesTables(this, recorded)) return "restart";
return "swap";
/** What a reload does to this object, whose type is still served: a live object restarts, any other is retried. */
reloadFate(): ReloadFate {
return this.#state.kind === "live" ? "restart" : "retry";
}
/**
* Takes `fate` with its new served type (after the reload's commit); returns the new attempt, whose first
* dependency check the reload's report waits for.
* Takes `fate` with its new served type (after the reload's commit) and, for a restart, its new factory's requests as
* staging recorded them; returns the new attempt, whose first dependency check the reload's report waits for (a
* retry's).
*/
reload(served: Served, fate: ReloadFate): Attempt {
reload(served: Served, fate: ReloadFate, requests: readonly Request[]): Attempt {
// Its members per version stay: the previous version's are mapped once the new attempt goes live.
this.served = served;
const attempt = fate === "swap" ? this.#swap() : fate === "restart" ? this.#restart() : this.#retryReloaded();
const attempt = fate === "restart" ? this.#restart(requests) : this.#retryReloaded();
attempt.startedByReload();
return attempt;
}
/**
* A pure swap (slice 7b §5): a new attempt of the same object (its documents, line and job records, its instance id),
* its activation at once; the live attempt keeps its running calls, its routines aborted, its jobs handed over,
* and is sealed once its calls settled and the new attempt's requests bound (`#retire`). Calls arriving meanwhile
* wait at the gate.
* A restart (decision "Reload restarts every affected object; old work finishes on old code"): its routines stop,
* new calls wait at the gate, old work runs on the old attempt, and consumers whose version it binds see it live; its
* contributions stay (until its seal with natives, F-b; until its successor is live without, F-c). It is sealed once
* its successor's recorded requests are ready and its calls, routed old work included, drained, bounded by the
* reload drain bound; then its natives close with reason `reload` and the successor, a new instance (D4), runs its
* factory. Readiness and successor start once every fate of the reload started (`startSuccessor`).
*/
#swap(): Attempt {
// A swap starts from a live object (`reloadFate`).
#restart(requests: readonly Request[]): Attempt {
// A restart starts from a live object (`reloadFate`).
const old = (this.#state as Extract<ObjectState, { readonly kind: "live" }>).attempt;
this.#old = old;
old.beginDrain();
old.stopRoutines();
if (!old.opensNatives) this.#kept = old;
this.#instance = uuidv7();
const attempt = this.#replace();
const shows = (cause: FailureCause | undefined): void => this.#readinessShown(attempt, cause);
const readiness = new Readiness(this.host, this.#caller(), requests, shows, this.#context);
this.#readiness = readiness;
const due = Date.now() + this.host.timeouts.reload;
const ended = this.#endOld(old, readiness, due);
this.host.ending(this.identity, this.#reported(ended));
const ending = this.host.endingOf(this.identity);
const attempt = this.#newAttempt();
const swap: Swap = { old, parked: false, done: Promise.withResolvers() };
this.#state = { kind: "activating", attempt, shown: LOADING, swap };
old.handOver();
this.#retiringAdd(old, this.#retire(swap));
this.#notify();
void this.#run(attempt, ending);
this.#startSuccessor = () => this.#startRestarted(attempt, old, readiness, ended, ending);
return attempt;
}
/**
* Keeps `retire` (a swapped-out attempt's) until it settled: a restart, an end and the host's close wait for it; a
* later swap does not (it holds no natives, its documents are the object's: Fable, Opus F1), so a second reload is
* never held up by the first one's old calls.
* Ends a restart's old attempt: once its successor's requests are ready and its calls drained (routed old work joins
* while the drain runs), by `due`; sealed (in-memory job runs end `aborted { reload }`), its references take their
* last values once its admitted commits published (a store write is not a running call's promise), its natives
* close with reason `reload`.
*/
#retiringAdd(old: Attempt, retire: Promise<void>): void {
this.#retiring ??= new Map();
const retiring = this.#retiring;
retiring.set(old, retire);
void retire.finally(() => {
retiring.delete(old);
});
}
/** Its swapped-out attempts still retiring, as one wait; `undefined` when none is. */
#retiringNow(): Promise<void> | undefined {
const retiring = this.#retiring;
if (retiring === undefined || retiring.size === 0) return undefined;
return Promise.all(retiring.values()).then(() => {});
}
/**
* Seals a swapped attempt once its running calls (and routine calls) settled, bounded by `closeTimeout`, and the swap
* ended (the new attempt live, parked, failed: its requests bound, so no request it shares with the old one ends in
* between, M§3.9, and its holders compared the old members with the new); a parked swap's references take their last
* values from it once the commits it admitted before the seal published; then its references are released. Its job
* invocations run on, on their old code.
*/
async #retire({ old, done }: Swap): Promise<void> {
await Promise.all([old.drainCalls(), done.promise]);
old.seal(this.#reloaded());
// A parked swap's old instance stops changing once its admitted commits published (its successor, live, took over
// its references before).
await this.documents.settled();
this.#stopFollowing(old);
const failures = await old.close(END);
old.detach();
old.releaseReferences();
for (const failure of failures) this.host.report(failure);
}
/**
* A native restart (decision "A native restart drains first"): new calls park (it shows `loading`), its
* contributions leave, the live attempt's running calls drain (bounded by `closeTimeout`), then it is sealed (its
* in-memory job runs end `aborted { reload }`), its references take their last values once its admitted commits
* published, its natives close with reason `reload`; the new attempt, a new instance (D4), activates after that
* teardown.
*/
#restart(): Attempt {
const old = (this.#state as Extract<ObjectState, { readonly kind: "live" }>).attempt;
this.#withdraw(old);
old.stopRoutines();
this.#instance = uuidv7();
const teardown = (async () => {
await Promise.all([old.drainCalls(), this.#retiringNow()]);
async #endOld(old: Attempt, readiness: Readiness, due: number): Promise<CloseFailed[]> {
await readiness.ready(due);
await old.drainCalls(due);
// With natives its contributions leave at the seal (F-b); without, they stay for its successor (F-c), unless a
// failure dropped them already.
if (old.opensNatives) this.#withdraw(old);
old.seal(this.#reloaded(), "reload");
// Drained and sealed: its references take their last values once the commits it admitted before published
// (a store write is not a running call's promise; Sol).
// Sealed, it is no longer the instance its holders bind (`instance`, `stateFor`): a hold waiting for a window
// slot unbinds now, before an `ack` could bind it to nothing; bound holds and references follow it on until it
// stopped changing.
this.#notify();
await this.documents.settled();
this.#stopFollowing(old);
const failures = await old.close(RELOAD);
old.detach();
old.releaseReferences();
return failures;
})();
this.host.ending(this.identity, this.#reported(teardown));
const attempt = this.#replace();
void this.#run(attempt, this.host.endingOf(this.identity));
return attempt;
}
/** Starts a restart's readiness and its successor, once every fate of the reload started (its served types are new). */
startSuccessor(): void {
const start = this.#startSuccessor;
this.#startSuccessor = undefined;
start?.();
}
/**
* Binds the readiness, whose failures the successor shows until the old attempt ended (`#readinessShown`). The
* successor runs after `ending`; once the old attempt ended (`ended`) and the successor's own requests bound, the old
* attempt's references and the readiness bindings are released (F-a: a dependency held only by them never ends; a
* successor a later reload replaces never releases them early).
*/
#startRestarted(
attempt: Attempt,
old: Attempt,
readiness: Readiness,
ended: Promise<unknown>,
ending: Promise<void> | undefined,
): void {
this.#showsReadiness = attempt;
// Before the successor's own steps (`ending` settles after `ended`): its own checks show what it waits for then.
void ended.then(() => {
if (this.#showsReadiness === attempt) this.#showsReadiness = undefined;
});
readiness.start();
void this.#run(attempt, ending);
void Promise.all([ended, attempt.checked]).then(() => {
old.releaseReferences();
readiness.release();
if (this.#readiness === readiness) this.#readiness = undefined;
});
}
/** What a restart's readiness finds now: a failed request the successor waits for, if any (the report's). */
get readinessCause(): FailureCause | undefined {
return this.#readiness?.cause;
}
/**
* A restart's readiness found `cause` (a failed request), or none any more: while the old attempt has not ended, the
* successor `attempt` shows it (new calls get the cause, consumers see the failure; old work still runs on the old
* attempt), and shows `loading` again once the request recovered. Its kept contributions stay: only the successor's
* own failure or park drops them (F-c).
*/
#readinessShown(attempt: Attempt, cause: FailureCause | undefined): void {
const state = this.#state;
if (this.#showsReadiness !== attempt || state.kind !== "activating" || state.attempt !== attempt) return;
const shown: RefState = cause === undefined ? LOADING : { kind: "failed", cause, retryAt: undefined };
if (sameState(state.shown, shown)) return;
this.#state = { kind: "activating", attempt, shown };
if (cause !== undefined) this.#lastFailure = cause.error;
this.#notify();
this.host.reloadCycles.recheck();
}
/** Whom its own requests act as: the host for a local object, its principal for a remote one. */
#caller(): Caller {
const { host, served, principal } = this;
return { principal: served.kind === "remote" ? principal : host.principal, incarnation: host.incarnation };
}
/**
@ -903,9 +1005,7 @@ export class ChordObject implements Target {
#retryReloaded(): Attempt {
const state = this.#state;
if (state.kind === "activating") {
const { attempt: old, swap } = state;
if (swap !== undefined && !swap.parked) this.#swapParked(swap);
swap?.done.resolve();
const { attempt: old } = state;
const parked = old.isParked;
// A parked attempt's successor waits for no teardown of it (its references are released after it bound).
const ending = this.host.endingOf(this.identity);
@ -977,7 +1077,6 @@ export class ChordObject implements Target {
#wentLive(attempt: Attempt): void {
if (this.#wakes !== undefined) this.#wakes.during = undefined;
const swap = this.#swapOf(attempt);
// Before the new instance is shown: holders of a version its provider does not bind leave with their last values,
// which their references take from the instance shown so far.
const unbound = this.#unbindUnfit(attempt);
@ -985,8 +1084,9 @@ export class ChordObject implements Target {
this.#exposed = attempt;
this.#following = true;
const { contributions } = attempt;
// The old attempt's contributions leave in the batch the new ones enter (slice 7b §5), so no reader sees a gap.
const leaving = swap !== undefined && !swap.parked ? swap.old.contributions : NO_CONTRIBUTIONS;
// A restart's kept contributions leave in the batch the new ones enter (F-c), so no reader sees a gap.
const leaving = this.#kept?.contributions ?? NO_CONTRIBUTIONS;
this.#kept = undefined;
// In the registries' tables before anyone sees it live (a getter released now reads them recomposed).
if (contributions.length > 0 || leaving.length > 0) {
this.host.batch.within(() => {
@ -1001,9 +1101,7 @@ export class ChordObject implements Target {
// Its references move to the new instance (one frame each); holds get a fresh `bound`.
this.#notify();
this.#settleFirstOutcome();
swap?.done.resolve();
// A swap's routines start once the old attempt's routine calls finished (the handover, §14).
attempt.live(swap?.old);
attempt.live();
this.#takeJobs(attempt);
// Its last holders may have left for their version: it ends as at their release.
if (unbound) this.#idle();
@ -1073,21 +1171,13 @@ export class ChordObject implements Target {
holder.removed({ cause, host: this.host, params: this.checkedParams, context: this.#context });
}
/** The swap `attempt` (still the current one) ends: `undefined` when it is no swap's new attempt. */
#swapOf(attempt: Attempt): Swap | undefined {
const state = this.#state;
return state.kind === "activating" && state.attempt === attempt ? state.swap : undefined;
}
/**
* Records a failed attempt: reported, torn down in the background (reason `end`), its references released once its
* natives closed, its holders told. Transient: a backoff window, whose end is the failure's `retryAt`; broken: no
* automatic retry.
*/
#failed(attempt: Attempt, cause: FailureCause): void {
const swap = this.#swapOf(attempt);
if (swap !== undefined && !swap.parked) this.#swapParked(swap);
swap?.done.resolve();
this.#dropKept();
const wokenBy = this.#wakes?.during === attempt;
if (this.#wakes !== undefined) this.#wakes.during = undefined;
if (cause.kind === "transient") this.host.backoff.failed(this.identity);
@ -1150,11 +1240,12 @@ export class ChordObject implements Target {
}
/**
* `attempt`, the instance shown, stopped changing (a live failure's admitted commits published, a restart's drain, a
* swapped or parked old instance retiring): its references take their last values from it, holds leave it, and the
* object lets it go (a reference made later never attached to it: it takes nothing from it).
* `attempt`, the instance shown, stopped changing (a live failure's admitted commits published, a restart's old
* attempt sealed): its references take their last values from it, holds leave it, and the object lets it go (a
* reference made later never attached to it: it takes nothing from it).
*/
#stopFollowing(attempt: Attempt): void {
if (this.#old === attempt) this.#old = undefined;
if (this.#exposed !== attempt || !this.#following) return;
this.#following = false;
this.#notify();
@ -1191,12 +1282,15 @@ export class ChordObject implements Target {
const { error } = ending;
const state = this.#state;
if (state.kind === "ended") return { kind: "none" };
const endings = this.host.endingOf(this.identity);
// Its ending rejects the callers of its swapped-out attempts' calls too, at once.
if (this.#retiring !== undefined) for (const old of this.#retiring.keys()) old.seal(error);
const retiring = this.#retiringNow();
// Its swapped-out attempts end before its teardown completes (they share its documents).
const previous = retiring === undefined ? endings : Promise.all([endings, retiring]).then(() => {});
// A restart's teardown (its old attempt's end) finishes before this one (they share its documents).
const previous = this.host.endingOf(this.identity);
// Its ending rejects the callers of a restart's old attempt too, at once; that attempt's end closes it.
this.#old?.seal(error);
this.#dropKept();
this.#readiness?.release();
this.#readiness = undefined;
this.#showsReadiness = undefined;
this.#startSuccessor = undefined;
this.#state = { kind: "ended", ended: { kind: "ended", error } };
this.#wakeGate();
for (const records of this.#jobs.values()) records.end(error);
@ -1224,11 +1318,6 @@ export class ChordObject implements Target {
else holder.removed(removal);
}
this.#exposed = undefined;
if (state.kind === "activating" && state.swap !== undefined) {
if (!state.swap.parked) this.#withdraw(state.swap.old);
state.swap.old.seal(error);
state.swap.done.resolve();
}
// A failed attempt is closed (or closing) since it failed.
if (state.kind === "waiting" || state.kind === "failed") return { kind: "end", previous };
if (state.kind === "live") this.#withdraw(state.attempt);
@ -1302,12 +1391,14 @@ export class ChordObject implements Target {
}
/**
* The live attempt; `not_ready` before the object went live and while it activates again, the cause while failed,
* `released` once it ended.
* The live attempt, or a restart's open old attempt (its consumers saw it live); `not_ready` before the object went
* live and while it activates again, the cause while failed, `released` once it ended.
*/
#live(): Attempt {
const state = this.#state;
if (state.kind === "live") return state.attempt;
const old = this.openOldAttempt;
if (old !== undefined && state.kind === "activating") return old;
if (state.kind === "ended") throw this.ended();
if (state.kind === "failed") throw state.failure.cause.error;
throw new ChordError({ code: "not_ready", message: `${this.label}: not live yet` });

69
packages/chord/src/host/old-work.ts Normal file
View file

@ -0,0 +1,69 @@
// Old work during a reload (decision "Reload restarts every affected object; old work finishes on old code"): the calls
// running on a restart's open old attempt and every call they cause. A call is old work when its call chain (the method
// ctxs it runs under, found through its context) holds an attempt that a restart drains, or when it arrived as old work
// from another host (`call.old` on a trusted link). The host counts its open old attempts and the old calls it runs
// for other hosts, so outside reloads no chain is walked.
import { type Context, type ContextKey, createContextKey, withContextValue } from "../context/index.ts";
import type { ChordObject } from "./object.ts";
/** One link of a call chain: the attempt a method call runs on, and the chain of the call that made it (`up`). */
export type CallNode = {
/** The attempt the call runs on: old work while that attempt drains for a restart. */
readonly on: OldWorkSource;
up(): CallNode | undefined;
};
/** What a call chain's node runs on: an attempt, or another host's old work. */
export type OldWorkSource = { readonly draining: boolean };
/** The key under which a context carries its call chain (a method ctx is its own node: no allocation per call). */
export const CALL_CHAIN: ContextKey<CallNode> = createContextKey<CallNode>("chord.callChain");
/**
* The key under which an attempt's activation context carries its object: a call its opener makes is known as that
* object's (its gate waits are the object's wait edges for reload cycles).
*/
export const OPENING: ContextKey<ChordObject> = createContextKey<ChordObject>("chord.opening");
/** The host's old work: whether a context runs under it, counted so that the check is free outside reloads. */
export class OldWork {
/** The open old attempts of restarts and the old calls from other hosts running here now. */
#count = 0;
/** An old attempt began draining, or an old call from another host began running here. */
open(): void {
this.#count++;
}
/** The counterpart of `open`. */
close(): void {
this.#count--;
}
/** Whether `context` runs under old work (walked only while this host has any). */
of(context: Context): boolean {
if (this.#count === 0) return false;
for (let node = context.value(CALL_CHAIN); node !== undefined; node = node.up()) {
if (node.on.draining) return true;
}
return false;
}
}
/** The chain node of a call that arrived as old work from another host: always old, with nothing above it. */
const REMOTE_OLD_WORK: CallNode = Object.freeze({ on: Object.freeze({ draining: true }), up: () => undefined });
/** `context` marked as old work that arrived from another host (`call.old` on a trusted link). */
export function asRemoteOldWork(context: Context): Context {
return withContextValue(CALL_CHAIN, REMOTE_OLD_WORK, context);
}
/** The attempts of `context`'s call chain, for the reload cycle check (another host's old work has none here). */
export function chainAttempts(context: Context): ReadonlySet<OldWorkSource> {
const attempts = new Set<OldWorkSource>();
for (let node = context.value(CALL_CHAIN); node !== undefined; node = node.up()) {
if (node !== REMOTE_OLD_WORK) attempts.add(node.on);
}
return attempts;
}

148
packages/chord/src/host/readiness.ts Normal file
View file

@ -0,0 +1,148 @@
// A restart's readiness (decision "Reload restarts every affected object; old work finishes on old code", rule 3):
// the requests its new factory made in staging, bound once every fate of the reload started (every served type is the
// new one then), and held until the successor's own requests bound. The old attempt is sealed only once they are
// ready, so old work that reaches the object meanwhile still finds its old attempt open.
import { type Context, withAbortSignal } from "../context/index.ts";
import type { FailureCause } from "../data/index.ts";
import { asChordError, checkParams, waitingFor } from "../data/internal.ts";
import type { Caller } from "./ctx.ts";
import type { AnyHost } from "./host.ts";
import type { Binding } from "./reference.ts";
import type { Request } from "./staging.ts";
/**
* One restart's readiness: ready once each recorded type request is live (an open old instance that binds its version
* counts) or failed for a reason other than another host's version skew (which heals by itself), or at its bound.
* Registry requests are served by facets in the commit and kept by the old attempt's contributions until its seal: it
* does not wait for them. Its failure follows the requests: `shows` gets each change of it.
*/
export class Readiness {
readonly #host: AnyHost;
readonly #caller: Caller;
readonly #requests: readonly Request[];
readonly #shows: (cause: FailureCause | undefined) => void;
readonly #context: Context;
readonly #bindings: Binding[] = [];
readonly #started = Promise.withResolvers<void>();
/** Ends the waits on its bindings at its release. */
readonly #abort = new AbortController();
/** A request refused for good at the start (an authority, a host address): it heals with no change of a binding. */
#refused: FailureCause | undefined;
/** The failed request's cause `shows` got last (`from`), as the successor waits for it (`waiting`). */
#failed: { readonly from: FailureCause; readonly waiting: FailureCause } | undefined;
constructor(
host: AnyHost,
caller: Caller,
requests: readonly Request[],
shows: (cause: FailureCause | undefined) => void,
context: Context,
) {
this.#host = host;
this.#caller = caller;
this.#requests = requests;
this.#shows = shows;
this.#context = context;
}
/** What the successor waits for now: a request refused for good, or `waiting` for the first failed one. */
get cause(): FailureCause | undefined {
return this.#refused ?? this.#failed?.waiting;
}
/**
* Binds the recorded type requests as the object's own would be (once every fate of the reload started), kept out
* of presence lists (they are no consumers of their own), and follows them: `shows` gets each change of `cause`.
*/
start(): void {
const following = withAbortSignal(this.#abort.signal, this.#context);
for (const request of this.#requests) {
if (request.kind !== "type") continue;
const { contract, params } = request;
const binding = this.#host.reference(contract, undefined, this.#caller, {});
binding.unlist();
this.#bindings.push(binding);
try {
this.#host.bind(binding, contract, checkParams(contract, params, this.#context), this.#context);
} catch (error) {
// Refused for good (an authority, a host address): the successor's factory meets the same refusal.
this.#refused ??= { kind: "broken", error: asChordError(error, contract.name) };
continue;
}
// Checked at every change of the binding until the release (never done): the failure shown follows it.
binding
.until(() => {
this.#changed();
return false;
}, following)
.catch(() => {});
}
this.#changed();
this.#started.resolve();
}
/** Resolves once every request is ready, or at `due` (`Date.now()` time), whichever is first. */
async ready(due: number): Promise<void> {
const timer = Promise.withResolvers<void>();
const timeout = setTimeout(timer.resolve, Math.max(0, due - Date.now()));
try {
await Promise.race([timer.promise, this.#allReady()]);
} finally {
clearTimeout(timeout);
}
}
/** The successor's own requests bound: its bindings are let go (a dependency held only by them never ends). */
release(): void {
this.#abort.abort();
for (const binding of this.#bindings.splice(0)) binding.release();
}
/** A binding changed: `shows` gets the failure if it is another one now. */
#changed(): void {
const before = this.cause;
const failed = this.#firstFailed();
if (failed?.cause !== this.#failed?.from) {
this.#failed = failed && { from: failed.cause, waiting: waitingFor(failed.label, failed.cause.error) };
}
const cause = this.cause;
if (cause !== before) this.#shows(cause);
}
/** The first request failed for a reason other than another host's version skew, with its cause. */
#firstFailed(): { readonly label: string; readonly cause: FailureCause } | undefined {
for (const binding of this.#bindings) {
const state = binding.refState;
if (state.kind === "failed" && !versionSkew(binding)) return { label: binding.label, cause: state.cause };
}
return undefined;
}
async #allReady(): Promise<void> {
await this.#started.promise;
const context = withAbortSignal(this.#abort.signal, this.#context);
// A wait ends with the binding (released) or the abort: either way nothing waits any more.
const waits = this.#bindings.map((binding) => binding.until(() => isReady(binding), context).catch(() => {}));
await Promise.all(waits);
}
}
/** Whether a request is ready: live, ended, or failed for a reason other than a version skew. */
function isReady(binding: Binding): boolean {
const state = binding.refState;
if (state.kind === "failed") return !versionSkew(binding);
return state.kind !== "loading";
}
/**
* Whether `binding` failed for another host's version refusal (its own, or a dependency's along the chain: `waiting`
* causes keep the reason): that heals when the other host's reload lands, so the old attempt stays open meanwhile,
* bounded as always. A refusal on this host heals by no other reload: it is no skew.
*/
function versionSkew(binding: Binding): boolean {
const state = binding.refState;
if (state.kind !== "failed" || binding.target?.kind !== "remote") return false;
const { error } = state.cause;
return error.code === "contract_mismatch" || (error.code === "unavailable" && error.reason === "version");
}

2
packages/chord/src/host/reference-keys.ts
View file

@ -1,7 +1,7 @@
// The keys a reference holds (slice 6a §12; decision "References are the only indirection" (4)): its key watches and
// its factory's pins (`at(key)`) of its object's keyed members, one source per member and key shared by them, pinned on
// the object's live instance (loaded, mapped to the reference's version). While the object still follows the instance a
// key is pinned on (a swap's old instance, a restart's drain, a live failure's admitted commits) the key follows it on,
// key is pinned on (a restart's open old attempt, a live failure's admitted commits) the key follows it on,
// as members do. When the reference moves to a new instance or a new object it pins the key there again, keeping the
// last value until that pin loads, then one frame; while it points at nothing the key keeps its last value (one
// `failed` call when its target failed, none while it loads).

42
packages/chord/src/host/reference.ts
View file

@ -69,10 +69,17 @@ type Lifecycle = {
* or a route address (`RemoteObject`). It tells its holders of each change of `state`.
*/
export interface Target {
/** Where it is: an object on this host, or one held over a link (another host decides what it serves). */
readonly kind: "local" | "remote";
/** The object in errors: its type, and its params key when it has params. */
readonly label: string;
/** Its state as its references show it. */
readonly state: RefState;
/**
* Its state as a reference of `version` and `fingerprint` sees it: on this host `live` also while a restart's open
* old attempt binds that version (decision "Reload restarts every affected object", rule 5); `state` over a link.
*/
stateFor(version: number, fingerprint: string): RefState;
/** Whether its state members have values: live once, for a remote object bound once. */
readonly hydrated: boolean;
/** The variant bound last; `undefined` for a local object and before the first binding. */
@ -248,6 +255,8 @@ export class Binding implements Holder, UnboundRequest {
#heldKeys: ReferenceKeys | undefined;
/** The instance it attached to last: it takes last values only from that one (never one it never followed). */
#attached: Attachable | undefined;
/** Whether presence lists show it: every reference but a restart's readiness bindings (`unlist`). */
#listed = true;
constructor(
contract: Contract,
@ -302,7 +311,11 @@ export class Binding implements Holder, UnboundRequest {
/** The reference's state. */
get refState(): RefState {
const state = this.#state;
if (state.kind === "bound") return state.target.state;
if (state.kind === "bound") {
// A bound reference keeps its contract until it is released.
const { version, fingerprint } = this.#contract!;
return state.target.stateFor(version, fingerprint);
}
if (state.kind === "waiting") return state.failure ?? LOADING;
if (state.kind === "refused") return state.failure;
return state.kind === "unbound" ? LOADING : state.state;
@ -456,6 +469,27 @@ export class Binding implements Holder, UnboundRequest {
this.dependent?.dependencyMoved();
}
/** Whether presence lists show it (§19). */
get listed(): boolean {
return this.#listed;
}
/** Keeps it out of presence lists, before it binds: a restart's readiness binding is no consumer of its own. */
unlist(): void {
this.#listed = false;
}
/** Whether its members have values now: bound to an object that shows it live and has them. */
#hydratedNow(): boolean {
const state = this.#state;
return state.kind === "bound" && state.target.hydrated && this.refState.kind === "live";
}
/** Resolves once `done(state)` holds, checked now and at every change; rejects at its end or `context`'s abort. */
until(done: (state: RefState) => boolean, context: Context): Promise<void> {
return this.#until((state) => (done(state) ? DONE : WAIT), context);
}
/** Resolves when the reference is live; rejects with a broken failure's error, or its end. Getters wait for it. */
live(context: Context): Promise<void> {
return this.#until((state) => {
@ -530,7 +564,7 @@ export class Binding implements Holder, UnboundRequest {
const state = this.#state;
switch (state.kind) {
case "bound": {
const current = state.target.state;
const current = this.refState;
if (current.kind === "failed") throw current.cause.error;
return state.target;
}
@ -568,13 +602,13 @@ export class Binding implements Holder, UnboundRequest {
* through (Opus G3).
*/
async firstValues(context: Context): Promise<void> {
if (this.#state.kind === "bound" && this.#state.target.hydrated) return;
if (this.#hydratedNow()) return;
await this.#until((state) => (state.kind === "live" ? DONE : WAIT), context);
}
/** Resolves once the object's members have values; rejects when the reference fails or is released first. */
async hydrated(context: Context): Promise<void> {
if (this.#state.kind === "bound" && this.#state.target.hydrated) return;
if (this.#hydratedNow()) return;
await this.#until((state) => {
if (state.kind === "live") return DONE;
if (state.kind === "failed") return { kind: "failed", error: state.cause.error };

97
packages/chord/src/host/reload-cycle.ts Normal file
View file

@ -0,0 +1,97 @@
// Reload cycles (decision "Reload restarts every affected object; old work finishes on old code", rule 6): a wait of
// old work on an object whose activation waits, through its parked dependencies or the gates its opener waits at, on
// the old attempt that very work runs on. Nothing would end it but the reload drain bound, so it fails at once with
// `unavailable/reload_cycle`, naming the chain. Checked when the wait starts and again whenever an attempt parks or an
// opener starts waiting at a gate (a cycle may close after the wait began).
import type { Contract } from "../contracts/index.ts";
import { ChordError } from "../data/index.ts";
import type { ChordObject } from "./object.ts";
import type { OldWorkSource } from "./old-work.ts";
/** One wait of old work: on what, for which version, under which attempts (its call chain's), and how it is cut. */
type OldWait = {
readonly target: ChordObject;
readonly contract: Contract;
readonly chain: ReadonlySet<OldWorkSource>;
readonly cut: (error: ChordError) => void;
};
/** A host's waits of old work on objects that are not live, while they last. */
export class ReloadCycles {
readonly #waits = new Set<OldWait>();
/**
* `waiting` (a getter's or a gate's wait on `target`, for `contract`'s version) as old work under `chain` makes it:
* rejected at once with `unavailable/reload_cycle` if it closes a cycle through that work's own drain, now or once a
* later park closes it.
*/
wait<T>(
waiting: Promise<T>,
target: ChordObject,
contract: Contract,
chain: ReadonlySet<OldWorkSource>,
): Promise<T> {
const { promise, resolve, reject } = Promise.withResolvers<T>();
const wait: OldWait = {
target,
contract,
chain,
cut: (error) => {
this.#waits.delete(wait);
reject(error);
},
};
this.#waits.add(wait);
waiting.then(
(value) => {
this.#waits.delete(wait);
resolve(value);
},
(error: unknown) => {
this.#waits.delete(wait);
reject(error);
},
);
this.#check(wait);
return promise;
}
/** An attempt parked, or an opener began waiting at a gate: every wait is checked again. */
recheck(): void {
if (this.#waits.size === 0) return;
for (const wait of [...this.#waits]) this.#check(wait);
}
#check(wait: OldWait): void {
if (!this.#waits.has(wait)) return;
const cycle = cycleFrom(wait.target, wait.contract, wait.chain);
if (cycle === undefined) return;
const chain = cycle.join(" → ");
const message = `${wait.target.label}: work its reload drains waits for it, and it waits for that drain (${chain})`;
wait.cut(new ChordError({ code: "unavailable", reason: "reload_cycle", message }));
}
}
/**
* The labels of a chain of waits from `start` to an object whose open old attempt is in `chain`; `undefined` when
* there is none. The start itself counts only through its edges: if its open old attempt serves `contract`'s version,
* the old work is routed there (or sees it live) and does not wait at all.
*/
function cycleFrom(start: ChordObject, contract: Contract, chain: ReadonlySet<OldWorkSource>): string[] | undefined {
if (start.oldServes(contract.version, contract.fingerprint)) return undefined;
const visited = new Set<ChordObject>([start]);
const walk = (object: ChordObject): string[] | undefined => {
for (const next of object.waitEdges()) {
if (visited.has(next)) continue;
visited.add(next);
const old = next.openOldAttempt;
if (old !== undefined && chain.has(old)) return [next.label];
const rest = walk(next);
if (rest !== undefined) return [next.label, ...rest];
}
return undefined;
};
const rest = walk(start);
return rest === undefined ? undefined : [start.label, ...rest];
}

24
packages/chord/src/host/staging.ts
View file

@ -2,8 +2,9 @@
// its own (the types it serves with their providers, the registries it serves, its facet contributions) with `load`'s
// checks, merged into the staged table beside the live one, where a type or registry another plugin serves already
// fails the plugin; and an object's new factory run against a recording `self`, which keeps what the factory declares
// (natives, documents, tables, jobs, contributions, requests with their contract versions) and activates, binds and
// opens nothing, so a reload decides each object's fate, and which plugins keep their old code, before its commit.
// (documents, tables, jobs, contributions, requests with their contract versions) and activates, binds and opens
// nothing, so a reload decides which plugins keep their old code before its commit, and a restart knows the requests
// its successor will make (its readiness).
// `load` stages its facets the same way and publishes them.
import type { Context } from "../context/index.ts";
@ -172,8 +173,6 @@ function stageEntry(
/** What a recorded factory declared: what a reload's fate and checks need, nothing activated. */
export type Recorded = {
/** How many natives it opens (`self.open`). */
readonly natives: number;
readonly documents: readonly AnyDefinition[];
readonly tables: readonly TableDefinition[];
readonly jobs: readonly JobDefinition<unknown, unknown, unknown>[];
@ -232,7 +231,6 @@ class RecordingSelf {
readonly #object: ChordObject;
readonly #served: Served;
readonly #context: Context;
#natives = 0;
readonly #documents = new Map<string, AnyDefinition>();
readonly #tables = new Map<string, TableDefinition>();
readonly #jobs = new Map<string, JobDefinition<unknown, unknown, unknown>>();
@ -247,7 +245,6 @@ class RecordingSelf {
get recorded(): Recorded {
return {
natives: this.#natives,
documents: [...this.#documents.values()],
tables: [...this.#tables.values()],
jobs: [...this.#jobs.values()],
@ -331,7 +328,6 @@ class RecordingSelf {
open(): never {
this.#assertDeclaring("open");
this.#natives++;
return INERT;
}
@ -387,17 +383,3 @@ class RecordingSelf {
if (!this.declaring) throw new Error(`${this.#object.label}: self.${declaration} is valid only in the factory`);
}
}
/**
* Whether `recorded`'s tables are at a version newer than `object`'s store has them: their eager migration runs at an
* activation, so the object restarts instead of swapping (§4).
*/
export function migratesTables(object: ChordObject, recorded: Recorded): boolean {
const { store } = object.host;
if (store === undefined) return false;
const owner = object.served.contract.name;
return recorded.tables.some((table) => {
const stored = store.table(tableSqlName(owner, table.name));
return stored !== undefined && stored.version < table.version;
});
}

29
packages/chord/src/jobs/job.ts
View file

@ -93,7 +93,7 @@ export type JobOwner = {
* when it settles or when the attempt ends with it still running.
*/
admit(running: Promise<unknown>, cleanup?: () => void): void;
/** Adds a run's invocation to the attempt's drain at its end (a swap does not wait for it), as `admit`. */
/** Adds a run's invocation to the attempt's drain at its end, as `admit`. */
admitRun(running: Promise<unknown>, cleanup: () => void): void;
};
@ -122,11 +122,8 @@ export class Job implements JobHandle<JsonValue, unknown, JsonValue | undefined>
readonly #definition: JobDefinition<unknown, unknown, unknown>;
readonly #implementation: AnyJobImplementation;
readonly #owner: JobOwner;
/**
* `declared` until the attempt is live (starts are refused), `handed` once a reload's swap gave its records to the
* new attempt's job (starts refused, its invocations finish here), `sealed` once it is sealed.
*/
#state: "declared" | "live" | "handed" | "sealed" = "declared";
/** `declared` until the attempt is live (starts are refused), `sealed` once it is sealed. */
#state: "declared" | "live" | "sealed" = "declared";
/** Runs whose `run` runs on this attempt. */
readonly #invocations = new Set<JobRun>();
/** The seal's error and cut reason, once sealed: a reservation committing after it cuts its run off with them. */
@ -207,7 +204,7 @@ export class Job implements JobHandle<JsonValue, unknown, JsonValue | undefined>
if (tx.documents !== this.#owner.documents) {
throw new Error(`${this.#owner.documents.label}: a transaction starts only its own object's jobs`);
}
if (this.#state === "sealed" || this.#state === "handed") {
if (this.#state === "sealed") {
throw new ChordError({
code: "released",
reason: "ended",
@ -227,28 +224,14 @@ export class Job implements JobHandle<JsonValue, unknown, JsonValue | undefined>
*/
seal(error: ChordError, reason: CutReason): void {
if (this.#state === "sealed") return;
const handed = this.#state === "handed";
this.#state = "sealed";
this.#sealed = { error, reason };
this.#records.detach(this, this.#finishing);
this.#finishing.clear();
// A handed-over job's invocations finish on its code; their outcomes go to the new attempt's job.
if (!handed) for (const run of this.#invocations) this.#records.cut(run, error, reason);
for (const run of this.#invocations) this.#records.cut(run, error, reason);
this.#invocations.clear();
}
/**
* A reload swapped its attempt out (slice 7b §8): starts are refused, the records go to the new attempt's job once it
* is live, the runs it invokes finish on this code (their outcome is finished by the new code), a finish running here
* completes here.
*/
handOver(): void {
if (this.#state !== "live" && this.#state !== "declared") return;
this.#state = "handed";
// Runs needing a live attempt meanwhile wait for the new one.
this.#records.detach(this, []);
}
/**
* Runs `run` for a committed run (with its signal aborted already after an early abort); a persisted run's
* reservation commits first (`running`: a crash from then on counts, D6). The reservation, the invocation and the
@ -307,7 +290,7 @@ export class Job implements JobHandle<JsonValue, unknown, JsonValue | undefined>
this.#invocations.add(run);
const running = (async () => {
try {
// Its own version's args: a handed-over attempt's late start runs its own code (Sol, reservation).
// Its own version's args: a run started by old work during a restart runs on the old code (its definition).
return await this.#implementation.run(this.#records.argsOf(run, this.#definition), ctx);
} finally {
// The fence closes as `run` settles: a change made later is refused.

4
packages/chord/src/link/README.md
View file

@ -77,7 +77,9 @@ tab.connect(webSocketLink(new WebSocket(url)), { principal: { kind: "server" } }
`hold { hold, type, version, fingerprint, params, address, principal?, incarnation? }` (`address`: `{ kind, params }`
the caller's `hostAddress` computed, it routes the hold; the provider places it by its params; `principal` and `incarnation`: whom it acts for, read only
on a parent link; `chain?`: the instance ids of the activations waiting for the hold, at most 32, read only on a
trusted link; `limit?`: the decoded limit per part of the link a relay admitted it on, read only on a parent link), `release`, `call { call, hold, method, args?, segments? }`, `abort`, `bound { holds: [{ hold,
trusted link; `limit?`: the decoded limit per part of the link a relay admitted it on, read only on a parent link), `release`, `call { call, hold, method, args?, segments?, old? }` (`old: true`: old work of a reload on the
caller's host, honoured only from a trusted link and dropped by a relay from one that is not; binary: `0x20` in the
call's kind byte, text: the seventh element, after the trace carrier), `abort`, `bound { holds: [{ hold,
variant, members: [{ member, value }], failed }] }` (one message may bind several holds: first bindings, re-holds),
`state { holds: [{ hold, members: [{ member, ops }], failed }] }`, `ack {}` (one per applied `bound` or `state` of
its origin), `closed { origin }` (a relay's downstream origin ended: its holds and calls go), `ping {}`, `pong {}`,

11
packages/chord/src/link/binary.ts
View file

@ -49,6 +49,9 @@ const ORIGIN = 0x80;
/** The kind byte's flag for a trace carrier (a `hold`'s or `call`'s; the message schema refuses it elsewhere). */
const TRACE = 0x40;
/** The kind byte's flag for a `call` that is old work of a reload (`call.old`); another kind with it is unknown. */
const OLD = 0x20;
/** A payload's flags: present, and carrying byte segments. */
const PRESENT = 1;
@ -73,8 +76,9 @@ export const binaryCodec: Codec = {
const writer = new Writer();
const origin = "origin" in message ? message.origin : undefined;
const trace = message.kind === "hold" || message.kind === "call" ? message.trace : undefined;
const old = message.kind === "call" && message.old === true ? OLD : 0;
writer.byte(
KIND_BYTES.get(message.kind)! | (origin === undefined ? 0 : ORIGIN) | (trace === undefined ? 0 : TRACE),
KIND_BYTES.get(message.kind)! | (origin === undefined ? 0 : ORIGIN) | (trace === undefined ? 0 : TRACE) | old,
);
if (origin !== undefined) writer.varint(origin);
if (trace !== undefined) writer.json(trace);
@ -84,8 +88,8 @@ export const binaryCodec: Codec = {
decode(bytes) {
const reader = new Reader(bytes);
const byte = reader.byte();
const kind = KINDS[byte & ~(ORIGIN | TRACE)];
if (kind === undefined) throw new Error("binary: an unknown kind");
const kind = KINDS[byte & ~(ORIGIN | TRACE | OLD)];
if (kind === undefined || (byte & OLD && kind !== "call")) throw new Error("binary: an unknown kind");
const origin = byte & ORIGIN ? reader.varint() : undefined;
const trace = byte & TRACE ? reader.json() : undefined;
const message = decodeFields(kind, reader);
@ -93,6 +97,7 @@ export const binaryCodec: Codec = {
// A fresh object; the message schema refuses an origin or a trace on a kind without one, and checks the trace.
if (origin !== undefined) (message as { origin?: number }).origin = origin;
if (trace !== undefined) (message as { trace?: unknown }).trace = trace;
if (byte & OLD) (message as { old?: true }).old = true;
return message;
},
};

2
packages/chord/src/link/message.ts
View file

@ -114,6 +114,8 @@ const MESSAGE = union([
args: opt(part),
segments,
trace,
// Old work of a reload (decision "Reload restarts every affected object"): honoured only from a trusted link.
old: opt(lit(true)),
}),
obj({ kind: lit("abort"), origin: opt(origin), call: id }),
// Asks the provider for an attempt of a failed hold's object now (`ref.retry()`); `hops` is its budget: the host

21
packages/chord/src/link/text.ts
View file

@ -5,7 +5,7 @@
// one), their lengths the tuple's last field. A message with an origin is wrapped: `["@", origin, tuple]`. Every part
// is tagged with its codec, so no JSON value is reserved: `[0, json]`, or `[1, base64]` for a part its origin deflated
// (the text codec itself never compresses). A `hold`'s or `call`'s trace carrier is an optional element after its
// other fields (slice 9).
// other fields (slice 9); a `call`'s old-work flag (`old`) follows it as `true` (the carrier then `null` when absent).
import {
type BoundEntry,
@ -86,12 +86,14 @@ function ownTuple(message: Message): string {
return json(["R", message.hold]);
case "call": {
const head = json(["c", message.call, message.hold, message.method]);
const { args, segments, trace } = message;
if (trace === undefined) return withPayload(head, args, segments);
// With a trace the payload's fields are positional (null when absent), the carrier last.
const { args, segments, trace, old } = message;
if (trace === undefined && old !== true) return withPayload(head, args, segments);
// With a trace or the old-work flag the payload's fields are positional (null when absent), the carrier next
// (null when absent before the flag), the flag last.
const lengths = segments === undefined ? "null" : json(segments.map((segment) => segment.byteLength));
const part = args === undefined ? "null" : partText(args);
return `${head.slice(0, -1)},${part},${lengths},${json(trace)}]`;
const tail = old === true ? `${json(trace ?? null)},true` : json(trace);
return `${head.slice(0, -1)},${part},${lengths},${tail}]`;
}
case "abort":
return json(["x", message.call]);
@ -214,7 +216,7 @@ function fromFields(fields: unknown[], rest: Uint8Array): Message {
arity(1);
return { kind: "release", hold: field(0) };
case "c": {
arity(3, 4, 5, 6);
arity(3, 4, 5, 6, 7);
const call = {
kind: "call",
call: field<number>(0),
@ -222,9 +224,12 @@ function fromFields(fields: unknown[], rest: Uint8Array): Message {
method: field<number>(2),
} as const;
if (f.length < 6) return { ...call, ...payload("args", f.slice(3), rest) };
// With a trace: the payload's fields positional, null when absent.
// With a trace or the old-work flag: the payload's fields positional, null when absent.
const fields = f[3] === null ? [] : f[4] === null ? [f[3]] : [f[3], f[4]];
return { ...call, ...payload("args", fields, rest), trace: f[5] as Record<string, string> };
// The message schema checks the carrier and the flag (`true` only).
const trace = f[5] === null ? {} : { trace: f[5] as Record<string, string> };
const old = f.length === 7 ? { old: f[6] as true } : {};
return { ...call, ...payload("args", fields, rest), ...trace, ...old };
}
case "x":
arity(1);

16
packages/chord/src/node/app/app.ts
View file

@ -144,7 +144,10 @@ export type AppOptions<P extends Principal> = {
* each worker kind's.
*/
readonly base?: readonly string[];
/** The `closeTimeout` and `readyTimeout` of every host (the server's and each worker's): 30 s and 60 s by default. */
/**
* The `closeTimeout`, `readyTimeout` and `reloadDrainTimeout` of every host (the server's and each worker's): 30 s,
* 60 s and 5 s by default.
*/
readonly timeouts?: AppTimeouts;
readonly onError: (error: AppError) => void;
} & (OwnerPrincipal extends P ? unknown : { readonly "the app's principal type must have { kind: 'owner' }": never });
@ -164,8 +167,15 @@ export type AppLimits = {
readonly importEntries?: number;
};
/** The hosts' timeouts in milliseconds (`HostOptions.closeTimeout`, `HostOptions.readyTimeout`). */
export type AppTimeouts = { readonly closeTimeout?: number; readonly readyTimeout?: number };
/**
* The hosts' timeouts in milliseconds (`HostOptions.closeTimeout`, `HostOptions.readyTimeout`,
* `HostOptions.reloadDrainTimeout`).
*/
export type AppTimeouts = {
readonly closeTimeout?: number;
readonly readyTimeout?: number;
readonly reloadDrainTimeout?: number;
};
/** The kit's web surface options (`webSurface`'s, its kind `web` and plugins the kit's). */
export type AppWeb = Pick<WebSurfaceOptions, "entry" | "page" | "paths" | "public">;

6
packages/chord/src/remote/README.md
View file

@ -97,9 +97,9 @@ within each link's limits, compression at the origin, activation chains and revo
waiting), each new cause of an object parked on a failed dependency too, the current one at once when the object is
failed already; the next attempt (or the parked object's next `loading`) is announced with `loading`; the hold stays,
so a `retry` (from any holder) renews even a broken object for every holder (a parked object passes it on, one walk
of this host with one hop less of the message's budget). After a swap in place it gets a fresh `bound` once the new
instance is live, answering calls by the names it was bound with until that is sent (the window may hold it). An
object leaving `live` unbinds it with `failed` (its calls in flight rejected) or `loading`: then its calls in flight
of this host with one hop less of the message's budget). Across a reload's restart it stays bound while the old
attempt serves its version; a fresh `bound` follows once the new instance is live, the hold answering calls by the
names it was bound with until that is sent (the window may hold it). An object leaving `live` unbinds it with `failed` (its calls in flight rejected) or `loading`: then its calls in flight
stay, each answered by the instance it runs on or by the object's gate, and calls by the names it was bound with
still arrive and wait at the gate until its fresh `bound` (decision "Calls in flight survive a `loading` unbind"). Live,
it binds through the window (`boundEntry`: the variant and each member's value as a part, names as indexes into the

4
packages/chord/src/remote/consumer.ts
View file

@ -41,6 +41,8 @@ type OutgoingCall = {
readonly segments: readonly Uint8Array[] | undefined;
/** The caller's trace carrier (its `chord.call` span's), when its adapter propagates. */
readonly trace: TelemetryCarrier | undefined;
/** Whether the caller is old work of a reload here (`call.old`: the provider routes it to open old attempts). */
readonly old: boolean;
readonly resolve: (value: unknown) => void;
readonly reject: (error: unknown) => void;
id: number | undefined;
@ -109,6 +111,7 @@ export class ConsumerSide {
args: part,
segments: payload?.segments,
trace: carrierOf(context.telemetry),
old: link.host.oldWork.of(context),
resolve: (value) => {
done();
resolve(value);
@ -203,6 +206,7 @@ export class ConsumerSide {
...(args === undefined ? {} : { args }),
...(segments === undefined ? {} : { segments }),
trace,
...(call.old ? { old: true as const } : {}),
});
}

93
packages/chord/src/remote/hold.ts
View file

@ -2,11 +2,11 @@
// unbound index until then), why it is unbound (refused: kept in the index while it waits for a name, dependency
// model §1), the object it keeps, the notices of its object's failures and new attempts, the snapshot and subscription
// taken in one synchronous step when a slot of the window is free, and the ops of its members per batch (sent at
// once, or pending while the window is full), each member with its own Delta encoder. Across a reload (slice 7b §6) a
// bound hold is kept: a swap in place sends it a fresh `bound` once the new instance is live (it answers calls
// meanwhile), an object leaving `live` unbinds it (`loading` or `failed`, then a fresh `bound`; a `loading` one keeps
// its calls in flight and takes more by its old names), a version no longer bound or a removed type leaves it waiting
// unbound. A hold of an older version follows its object's adapted members (Q10).
// once, or pending while the window is full), each member with its own Delta encoder. Across a reload a bound hold is
// kept while its object shows it live (a restart's open old attempt that binds its version, decision "Reload restarts
// every affected object"), then unbound (`loading` or `failed`, then a fresh `bound`; a `loading` one keeps its calls
// in flight and takes more by its old names); a version no longer bound or a removed type leaves it waiting unbound. A
// hold of an older version follows its object's adapted members (Q10).
import type { TelemetryContext } from "@earendil-works/pi-telemetry";
import type { Context } from "../context/index.ts";
@ -15,9 +15,11 @@ import { ChordError, type FailureCause, isChordError, type Principal } from "../
import { asChordError } from "../data/internal.ts";
import { type Encoder, encoder, type Op, PendingOps } from "../delta/index.ts";
import {
type Attempt,
type Caller,
type ChordObject,
type Holder,
type RefState,
type Removal,
retryWalk,
type Serving,
@ -70,9 +72,9 @@ type Followed = {
* object is live (the activation, parking and retries; `told`: the failure its consumer was told, so the next attempt
* is announced with `loading` and a call that crossed the failure on the wire gets it); `binding` while its object is live and its `bound` waits for a slot;
* `bound` with the object's variant and the members it follows. A `pending` hold unbound with `loading` and the
* `binding` after it, or a `binding` after a swap in place, keep the names it was bound with, `previous`: its consumer's
* calls sent by those names (before it learnt of the `loading`, or still bound) wait at its object's gate and are
* answered by name, design §22; only contract data, so the old instance and its sources go meanwhile.
* `binding` after it keep the names it was bound with, `previous`: its consumer's calls sent by those names (before it
* learnt of the `loading`) wait at its object's gate and are answered by name, design §22; only contract data, so the
* old instance and its sources go meanwhile.
*/
type HoldState =
| { readonly kind: "unplaced"; readonly request: HoldRequest }
@ -86,7 +88,7 @@ type HoldState =
readonly names: VariantNames;
/** The members it follows, in the order they were subscribed. */
readonly followed: Followed[];
/** The instance it was bound to: a swap brings another, which it binds to afresh. */
/** The instance it was bound to: it follows it on while it still changes (a restart's open old attempt). */
readonly instance: unknown;
/**
* A rebinding's names before it (`until`: its `bound`, this host's own message number): calls its consumer sent
@ -126,6 +128,8 @@ export class Hold implements Holder, UnboundRequest {
readonly id: number;
/** A hold is declared by no object on this host. */
readonly dependent = undefined;
/** Presence lists show every hold. */
readonly listed = true;
/** Whom the hold acts for: its calls run as it, with its origin's incarnation. */
readonly caller: Caller;
/** The limits per part of its params and its calls' arguments: its link's, the decoded one smaller when stamped (Q2). */
@ -161,8 +165,8 @@ export class Hold implements Holder, UnboundRequest {
}
/**
* Whether calls on it reach its object: bound, or by the names it was bound with while it rebinds (a swap in place, a
* `loading` unbind: its fresh `bound` not sent yet).
* Whether calls on it reach its object: bound, or by the names it was bound with while it rebinds (after a `loading`
* unbind: its fresh `bound` not sent yet).
*/
get callable(): boolean {
return this.#names() !== undefined;
@ -272,30 +276,34 @@ export class Hold implements Holder, UnboundRequest {
}
/**
* The object's state changed, or the hold took it: a pending hold binds once it is live (when a slot is free),
* sends each failure, and announces the attempt after a failure with `loading`. A bound hold (or one whose `bound`
* waits for a slot) whose object leaves `live` goes back to pending (slice 7b §6), except while a reload's swap runs
* in place; a bound one whose object went live with another attempt (a swap) gets a fresh `bound` (audit A9).
* The object's state changed, or the hold took it: a pending hold binds once its object shows it live (when a slot
* is free), sends each failure, and announces the attempt after a failure with `loading`. A bound hold (or one whose
* `bound` waits for a slot) whose object no longer shows it live goes back to pending (slice 7b §6).
*/
moved(): void {
const current = this.#state;
const object = this.#object;
if (object === undefined) return;
if (current.kind === "bound" || current.kind === "binding") {
const { kind } = object.state;
if (kind === "live") {
if (current.kind === "bound" && object.instance !== current.instance) this.#rebind(current);
return;
}
// Its instance still changes (a swap's old one, a restart's drain, a live failure's admitted commits): it follows
// it on until it stopped (slice 7b, decision "References are the only indirection").
const shown = this.#shown(object);
if (shown.kind === "live") return;
// Its instance still changes (a live failure's admitted commits, a restart's old attempt draining): it follows it
// on until it stopped (decision "References are the only indirection").
if (current.kind === "bound" && object.attachable === current.instance) return;
if (current.kind === "binding" && kind === "loading" && object.swapsInPlace) return;
this.#unbind(object.state.kind === "failed" ? object.state.cause.error : undefined);
this.#unbind(shown.kind === "failed" ? shown.cause.error : undefined);
}
this.#pending(object);
}
/**
* What its object shows to its version and fingerprint: `live` also while a restart's open old attempt binds them
* (decision "Reload restarts every affected object", rule 5).
*/
#shown(object: ChordObject): RefState {
const { version, fingerprint } = this.contract;
return object.stateFor(version, fingerprint);
}
/** Member `followed` failed with `error`: its failure waits to be sent, replacing one still waiting. */
#memberFailed(followed: Followed, error: unknown): void {
followed.pending.take();
@ -312,7 +320,7 @@ export class Hold implements Holder, UnboundRequest {
}
/**
* Its object left `live` (a restart, a parked swap, a live failure): no longer subscribed or waiting for a slot,
* Its object no longer shows it live (a restart, a live failure): no longer subscribed or waiting for a slot,
* pending again. With `error` (a failure, sent by `#pending`, which a consumer applies to a bound hold as a rebinding
* too, model §5) its calls in flight are rejected. Without (`loading`, which tells the consumer it rebinds) they stay,
* and calls by the names it was bound with keep arriving (sent before its consumer learnt of the `loading`): each gets
@ -340,21 +348,6 @@ export class Hold implements Holder, UnboundRequest {
this.#provider.send({ kind: "loading", hold: this.id });
}
/**
* A swap of its object went live in place (slice 7b §5, audit A9): no longer subscribed to the old instance, it gets
* a fresh `bound` of the new one's members (as a restart, a variant or a version change does), answering calls by
* the names its consumer has until it applied that.
*/
#rebind(state: Bound): void {
// The names its consumer has: the old binding's, or older ones while it has not applied that `bound` yet.
const { variant, names } = this.#names()!;
unfollow(state);
// Its ops still waiting for a slot are the old instance's: the `bound` replaces them.
this.#provider.unbind(this);
this.#state = { kind: "binding", previous: { variant, names } };
this.#provider.bind(this, this.#provider.link.context);
}
/**
* A `retry` message with `hops` left of its budget: the object is asked for an attempt now (it ignores this unless
* failed; parked, it passes it on, to other hosts with one hop less); a refused hold is started again (a route's
@ -374,7 +367,7 @@ export class Hold implements Holder, UnboundRequest {
#pending(object: ChordObject): void {
const current = this.#state;
if (current.kind !== "pending") return;
const state = object.state;
const state = this.#shown(object);
if (state.kind === "live") {
// Calls by the names it was bound with are still answered until the fresh `bound` is sent.
this.#state = current.previous === undefined ? BINDING : { kind: "binding", previous: current.previous };
@ -424,10 +417,11 @@ export class Hold implements Holder, UnboundRequest {
*/
boundEntry(context: Context): BoundEntry {
const object = this.#object!;
const { contract } = object.served;
// A live remote object has its variant.
const variant = object.variant!;
const state = this.#bindTo(object);
// It binds the instance its object shows it (`#shown` live): the live one, or a restart's open old attempt; the
// object tells its holds at that attempt's seal (`moved`), so a hold waiting for a slot never binds it later.
const instance = object.instance!;
const { contract } = instance.served;
const state = this.#bindTo(instance);
const members: BoundEntry["members"][number][] = [];
const failed: FailedMember[] = [];
const { version } = this.contract;
@ -458,7 +452,7 @@ export class Hold implements Holder, UnboundRequest {
this.#failed = [];
this.#retired = NO_RETIRED;
chordMetrics(context.telemetry.metrics).counter("chord.link.encodes").add(1);
return { hold: this.id, variant: variantOrder(contract).indexOf(variant), members, failed };
return { hold: this.id, variant: variantOrder(contract).indexOf(instance.variant!), members, failed };
}
/**
@ -536,15 +530,16 @@ export class Hold implements Holder, UnboundRequest {
* The binding: its variant and names at the hold's version, with no member followed yet, and the names its consumer
* still has (a rebinding's) until it acknowledged this `bound`, the next of this host's own messages.
*/
#bindTo(object: ChordObject): Bound {
const variant = variantsAt(object.served.contract, this.contract.version).get(object.variant!)!;
#bindTo(instance: Attempt): Bound {
// A live remote object's instance has its variant.
const variant = variantsAt(instance.served.contract, this.contract.version).get(instance.variant!)!;
const previous = this.#names();
const state: Bound = {
kind: "bound",
variant,
names: variantNames(variant),
followed: [],
instance: object.instance,
instance,
...(previous === undefined
? {}
: { previous: { variant: previous.variant, names: previous.names, until: this.#provider.nextOwn } }),

29
packages/chord/src/remote/provider.ts
View file

@ -7,7 +7,7 @@
import type { TelemetryContext } from "@earendil-works/pi-telemetry";
import { type Context, withAbortSignal } from "../context/index.ts";
import { ChordError, isChordError, type Principal } from "../data/index.ts";
import type { Caller } from "../host/internal.ts";
import { asRemoteOldWork, type Caller } from "../host/internal.ts";
import {
type BoundEntry,
type ConsumerMessage,
@ -393,7 +393,7 @@ export class ProviderSide {
* A call: one on a hold this side does not know is ignored (Q1), more than this side's N at once and compressed
* arguments from a child (clients never compress) are protocol errors. On a relayed hold it is forwarded, its
* arguments checked against this link's part limit first (the bytes are here). On a served hold that is not callable
* (not bound yet, nor rebinding by the names its consumer still has after a swap in place or a `loading`) it is
* (not bound yet, nor rebinding by the names its consumer still has after a `loading`) it is
* refused `not_ready`; its method is an index into the bound variant (one outside it is a protocol error) and its
* arguments are opened and parsed here within the hold's limits (past one the call fails `too_large/part`; a
* malformed part is a protocol error), absent ones `null`. Arguments this host must inflate asynchronously are
@ -441,12 +441,25 @@ export class ProviderSide {
this.#calls.set(message.call, call);
// The caller's span over the link: its carrier with this host's adapter (absent or unknown: the adapter's root).
const parent = parentOf(message.trace, this.link.host.telemetry);
this.#run(message.call, call, method.name, args, parent);
// Old work of another host's reload (`call.old`), honoured only from a trusted link (decision "Reload restarts
// every affected object"): it runs on open old attempts here, and counts as this host's old work while it runs.
if (message.old === true && this.link.trusted) {
const { oldWork } = this.link.host;
oldWork.open();
const close = (): void => oldWork.close();
this.#run(message.call, call, method.name, args, parent, asRemoteOldWork(this.link.context)).then(
close,
close,
);
return;
}
this.#run(message.call, call, method.name, args, parent, this.link.context);
}
/**
* Runs a call and sends its result (absent when `null`) or error, unless it was aborted, released or gone
* meanwhile. A result that cannot be encoded is sent as its error.
* Runs a call under `scope` (the link's context, marked when it is another host's old work) and sends its result
* (absent when `null`) or error, unless it was aborted, released or gone meanwhile. A result that cannot be encoded
* is sent as its error. Returns the handler's settling.
*/
#run(
id: number,
@ -454,8 +467,9 @@ export class ProviderSide {
method: string,
args: unknown,
parent: TelemetryContext,
): void {
const context = withAbortSignal(call.abort.signal, this.link.context);
scope: Context,
): Promise<unknown> {
const context = withAbortSignal(call.abort.signal, scope);
let running: Promise<unknown>;
try {
running = call.hold.call(method, args, parent, context);
@ -477,6 +491,7 @@ export class ProviderSide {
},
(error: unknown) => this.reply(id, { kind: "error", call: id, error: wireError(error) }),
);
return running;
}
/** Answers call `id` with `error` before it was admitted: it took no slot. */

12
packages/chord/src/remote/relay.ts
View file

@ -169,9 +169,11 @@ export class RelayedHold {
this.#downstream.reply(message.call, { kind: "error", call: message.call, error: wireError(error) });
return undefined;
}
// A downstream link that is not trusted (a page) cannot claim old work: its flag is dropped here.
const forwarded = message.old === true && !this.#downstream.link.trusted ? unmarked(message) : message;
const adapter = upstream.link.host.telemetry;
if (untraced(adapter)) {
upstream.link.send({ ...message, origin: upstream.origin });
upstream.link.send({ ...forwarded, origin: upstream.origin });
return undefined;
}
const span = new OpenSpan(
@ -180,7 +182,7 @@ export class RelayedHold {
this.#attributes(`#${message.method}`),
);
const trace = carrierOf(span.span) ?? message.trace;
upstream.link.send({ ...message, origin: upstream.origin, ...(trace === undefined ? {} : { trace }) });
upstream.link.send({ ...forwarded, origin: upstream.origin, ...(trace === undefined ? {} : { trace }) });
return span;
}
@ -238,3 +240,9 @@ export function relayEnded(
span.span.setAttributes({ outcome });
span.end();
}
/** `message` without its old-work flag (`call.old`), as a relay forwards it from a link that is not trusted. */
function unmarked(message: Extract<Message, { readonly kind: "call" }>): Extract<Message, { readonly kind: "call" }> {
const { old: _old, ...rest } = message;
return rest;
}

5
packages/chord/src/remote/remote-object.ts
View file

@ -126,6 +126,11 @@ export class RemoteObject implements Target {
return this.#state;
}
/** Over a link the provider decides what each version sees: its state as it is. */
stateFor(): RefState {
return this.#state;
}
/** Its `chord.hold` span until its first `bound` or failure (a route's resolve links to it); none when not recorded. */
get span(): TelemetrySpan | undefined {
return this.#holding?.span;

6
packages/chord/src/store/table-handle.ts
View file

@ -132,9 +132,9 @@ export class TableHandle implements Table<Columns, readonly string[]> {
}
/**
* The version fence of old table handles (R15, slice 7b §5): once the store's table is at a newer version than this
* handle's definition (a reload's new code migrated it), old code reads and writes it no more (`released/version`),
* so it never issues SQL for a dropped column.
* The version fence of old table handles (R15): once the store's table is at a newer version than this handle's
* definition (a reload's new code migrated it), old code still running (a call cut off at its restart's seal) reads
* and writes it no more (`released/version`), so it never issues SQL for a dropped column.
*/
assertVersion(): void {
const stored = this.store.tableVersion(this.sql.table);

2
packages/chord/src/telemetry.ts
View file

@ -101,7 +101,7 @@ const fate = {
fate: {
type: "string",
required: true,
values: ["swapped", "restarted", "ended", "parked", "retried"],
values: ["restarted", "ended", "parked", "retried"],
description: "What the object did.",
},
},

21
packages/chord/test/bench/reload.ts
View file

@ -1,7 +1,8 @@
// Benchmark of `host.reload` (slice 7b §11): one plugin of 20 reloaded on a host with 1,000 live objects, as a pure
// swap and as a native restart. Per case: the reload's time until it resolves (staging, commit, every fate started and
// every swap checked), the time until every object of the plugin is live again, and the heap retained after 20
// reloads against after the first (old attempts and facets collected). Medians over the reloads after a warm-up.
// Benchmark of `host.reload` (slice 7b §11; decision "Reload restarts every affected object"): one plugin of 20
// reloaded on a host with 1,000 live objects, every one of its objects restarted, pure code and code holding a native.
// Per case: the reload's time until it resolves (staging, commit, every fate started), the time until every object of
// the plugin is live again, and the heap retained after 20 reloads against after the first (old attempts, their
// readiness and facets collected). Medians over the reloads after a warm-up.
// Run by hand from packages/chord (not in CI), pinned for comparisons:
// taskset -c 6 node --conditions=source --experimental-strip-types --expose-gc test/bench/reload.ts
import { performance } from "node:perf_hooks";
@ -68,8 +69,8 @@ function median(values: readonly number[]): number {
}
/**
* One case: `perPlugin(index)` objects per plugin (1,000 in all), plugin 0 reloaded `RELOADS` times; `native` makes
* plugin 0 a native object (restarts) instead of a pure one (swaps).
* One case: `counts[index]` objects per plugin (1,000 in all), plugin 0 reloaded `RELOADS` times; `native` makes
* plugin 0's objects open a native.
*/
async function run(name: string, counts: readonly number[], native: boolean): Promise<void> {
const host: Host = createHost({ onError: (error) => console.log("onError", error.kind) });
@ -112,7 +113,7 @@ async function run(name: string, counts: readonly number[], native: boolean): Pr
const spread = Array.from({ length: PLUGINS }, () => OBJECTS / PLUGINS);
const concentrated = [OBJECTS, ...Array.from({ length: PLUGINS - 1 }, () => 0)];
await run("swap, 50 of 1,000 objects in the reloaded plugin", spread, false);
await run("swap, all 1,000 objects in the reloaded plugin", concentrated, false);
await run("restart, 50 of 1,000 objects in the reloaded plugin", spread, true);
await run("restart, all 1,000 objects in the reloaded plugin", concentrated, true);
await run("pure, 50 of 1,000 objects in the reloaded plugin", spread, false);
await run("pure, all 1,000 objects in the reloaded plugin", concentrated, false);
await run("native, 50 of 1,000 objects in the reloaded plugin", spread, true);
await run("native, all 1,000 objects in the reloaded plugin", concentrated, true);

12
packages/chord/test/data/errors.test.ts
View file

@ -26,7 +26,14 @@ describe("ChordError", () => {
if (error.code === "invalid") expectTypeOf(error.reason).toEqualTypeOf<ChordErrorReason<"invalid">>();
if (error.code === "unavailable")
expectTypeOf(error.reason).toEqualTypeOf<
"locked" | "park_timeout" | "busy" | "storage_failed" | "undeclared" | undefined
| "locked"
| "park_timeout"
| "busy"
| "storage_failed"
| "undeclared"
| "version"
| "reload_cycle"
| undefined
>();
if (error.code === "gone") expectTypeOf(error.reason).toEqualTypeOf<undefined>();
expect(error).toBeInstanceOf(Error);
@ -75,6 +82,7 @@ describe("ChordError", () => {
describe("protocol table", () => {
it("is pinned together with PROTOCOL_VERSION", () => {
// PROTOCOL_VERSION stays 1 until p3 ships (design §22): the table changes with it unbumped until then.
const pinned = {
version: 1,
reasons: {
@ -85,7 +93,7 @@ describe("protocol table", () => {
not_ready: [],
released: ["ended", "closed", "settled", "version", "discarded"],
too_large: ["part"],
unavailable: ["locked", "park_timeout", "busy", "storage_failed", "undeclared"],
unavailable: ["locked", "park_timeout", "busy", "storage_failed", "undeclared", "version", "reload_cycle"],
aborted: [],
},
};

47
packages/chord/test/examples/33-reload.ts
View file

@ -1,9 +1,10 @@
// Reload in the kernel (slice 7b): `host.reload(facets, ctx)` switches a running host to a new plugin set. Staging
// runs every setup and every affected object's new factory against a recording `self` (nothing activates; a throw
// rejects the reload and nothing changes), the commit swaps the served types, registries and contributions at once,
// then every object of a changed or removed plugin takes its fate. Here: a pure notes store is swapped in place while
// a call runs on its old code, a search index with a native restarts after its running call drained, and a removed
// plugin's object ends while its holder waits for it to come back.
// rejects the reload and nothing changes), the commit replaces the served types, registries and contributions at
// once, then every object of a changed or removed plugin takes its fate: a live one restarts (its running calls finish
// on the old code, new calls wait for the new code), a removed plugin's one ends. Here: a pure notes store restarts
// while a call runs on its old code, a search index with a native restarts after its running call drained, and a
// removed plugin's object ends while its holder waits for it to come back.
// Run from packages/chord:
// node --conditions=source --experimental-strip-types test/examples/33-reload.ts
import { BACKGROUND_CONTEXT, type Context } from "../../src/context/index.ts";
@ -14,7 +15,7 @@ import { constant } from "../../src/state/index.ts";
const context = BACKGROUND_CONTEXT;
/** The notes store: a list, its count, and `add` (which can be told to wait, to show a call across a swap). */
/** The notes store: a list, its count, and `add` (which can be told to wait, to show a call across a restart). */
const Notes = local("example.notes", {
version: 1,
params: {},
@ -104,10 +105,11 @@ const notes = await host.get(Notes, {}, context);
await notes.add({ text: "first" }, context);
console.log("1. loaded:", notes.list.value, "banner", notes.banner.value);
// 2. A pure swap while a call runs: the running `add` finishes on the old code, the next call runs on the new one.
// 2. A restart of pure code while a call runs: the running `add` finishes on the old code; the next call waits at the
// gate until the old attempt drained, then runs on the new code. Documents stay.
slowAdd = Promise.withResolvers();
const running = notes.add({ text: "second" }, context);
const swap = await host.reload(
const restarted = await host.reload(
[
{ name: "@example/notes", facet: notesV2 },
{ name: "@example/search", facet: search },
@ -115,15 +117,11 @@ const swap = await host.reload(
],
context,
);
console.log("2. reload:", JSON.stringify(swap));
console.log("2. reload:", JSON.stringify(restarted));
const next = notes.add({ text: "third" }, context);
slowAdd.resolve();
slowAdd = undefined;
console.log(
" the old call returned",
await running,
"; the new code adds",
await notes.add({ text: "third" }, context),
);
console.log(" the old call returned", await running, "; the waiting call, on the new code,", await next);
console.log(" list:", notes.list.value, "banner", notes.banner.value, "(same documents, new code)");
// 3. A restart: the search object holds a native. Its running call drains first, then the native closes (reload).
@ -170,16 +168,25 @@ await host.reload(
await new Promise((resolve) => setTimeout(resolve, 0));
console.log("5. served again: clock holder", shown(clock.state.value), clock.zone.value);
// 6. A staging error rejects the reload and changes nothing.
// 6. A staging error fails that plugin only: it keeps its old code, reported; nothing else changes.
const broken = defineFacet((h) =>
h.serve(Notes, () => {
throw new Error("a typo in the new factory");
}),
);
try {
await host.reload([{ name: "@example/notes", facet: broken }], context);
} catch (error) {
console.log("6. rejected:", (error as Error).message, "- banner still", notes.banner.value);
}
const failed = await host.reload(
[
{ name: "@example/notes", facet: broken },
{ name: "@example/search", facet: searchV2 },
{ name: "@example/clock", facet: clockPlugin },
],
context,
);
console.log(
"6. failed:",
failed.failed.map(({ plugin, error }) => `${plugin}: ${(error as Error).message}`),
"- banner still",
notes.banner.value,
);
await host.close(context);

28
packages/chord/test/examples/36-plugin-failures/main.ts
View file

@ -3,17 +3,16 @@
// is listed in the report's `failed`: a syntax error (its build failed), a module that throws while it is evaluated
// (`mapStack` names the plugin's file and line), a factory that throws while the host stages it. The closure rule: a
// provider and its consumer edited together, the provider broken, the consumer kept on its old code too (its new code
// needs the provider's new version) and still working; the next good edit applies both. Last, a reload whose context
// aborts after its commit: the commit stands, and the component keeps what was committed.
// needs the provider's new version) and still working; the next good edit applies both. Last, a restart whose new code
// needs an object that takes a while to open: the old code keeps serving until it is open.
// Run from packages/chord:
// node --conditions=source --experimental-strip-types test/examples/36-plugin-failures/main.ts
import { cp, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { BACKGROUND_CONTEXT, withAbortSignal } from "../../../src/context/index.ts";
import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts";
import { int, local, str } from "../../../src/contracts/index.ts";
import { isChordError } from "../../../src/data/index.ts";
import { createHost, type ReloadReport } from "../../../src/host/index.ts";
import * as chord from "../../../src/index.ts";
import { createPlugins, mapStack } from "../../../src/node/plugins/index.ts";
@ -125,30 +124,21 @@ console.log(" running:", values());
await until(() => summary.text.value.startsWith("price v2"));
console.log(" then:", values());
// 4. A reload whose context aborts after its commit, while it waits for its report: here the app shuts down the moment
// it sees the status object leave `live` (its new code waits for a new `Cache`, which takes 200 ms to open). The
// reload rejects `aborted`, but its commit stands: `details.committed` says what it applied, and the component's
// loaded set is what was committed (a later reload does not roll the cache plugin back).
// 4. A restart whose new code needs a new `Cache` (a native that takes 200 ms to open): the reload resolves at once;
// the status reference stays live on the old code until that `Cache` is open, then shows `loading` briefly and
// the new code's values.
console.log("4. status:", status.text.value);
await edit("cache/server.ts", 'const generation = "v1"', 'const generation = "v2"');
const cache = await buildDirty(["@example/cache"]);
const abort = new AbortController();
const watchStatus = await status.state.watch(context);
watchStatus.start(async (call) => {
if (call.kind !== "value") return;
console.log(" the status reference:", call.value.kind);
if (call.value.kind !== "live") abort.abort(new Error("the app shuts down"));
if (call.kind === "value") console.log(" the status reference:", call.value.kind, "|", status.text.value);
});
try {
await plugins.reload(host, cache, withAbortSignal(abort.signal, context));
} catch (error) {
if (!isChordError(error, "aborted")) throw error;
console.log(" rejected:", error.code, error.message, JSON.stringify(error.details));
}
print(await plugins.reload(host, await buildDirty(["@example/cache"]), context));
await until(() => status.text.value === "cache v2");
console.log(" status:", status.text.value);
await edit("greeter/label.ts", '"hi"', '"hey"');
print(await plugins.reload(host, await buildDirty(["@example/greeter"]), context));
await until(() => greeting.text.value === "hey");
console.log(" after another reload:", status.text.value, "| greeting:", greeting.text.value);
await watchStatus.stop();

4
packages/chord/test/examples/36-plugin-failures/plugins/cache/server.ts vendored
View file

@ -1,6 +1,6 @@
// The cache plugin's server facet: `Cache` opens a store per generation, which takes a while (a native); `Status` reads
// the cache of the plugin's generation. A reload to a new generation swaps `Status` in place; its new code needs a new
// `Cache` object, and its first dependency check waits until that one is open: the reload's report waits for it too.
// the cache of the plugin's generation. A reload to a new generation restarts `Status`; its new code needs a new
// `Cache` object, and its old attempt stays open until that one is open.
import { constant, defineFacet, local, str } from "@earendil-works/chord";

5
packages/chord/test/examples/51-single-file-plugin/main.ts
View file

@ -75,10 +75,11 @@ await rm(join(root, "hello.web.ts"));
await until(() => plugins.state.value.set && plugins.state.value.dirty.includes("hello"));
console.log("3. marks:", JSON.stringify(plugins.state.value));
// 3. A build discovers again; the reload changes `hello` in place (the same plugin, its object swapped), a holder
// sees the new text.
// 3. A build discovers again; the reload changes `hello` (the same plugin, its object restarted), a holder sees the
// new text once the new attempt is live.
const built = await plugins.build(plugins.state.value.dirty, context);
const report = await plugins.reload(host, built, context);
await until(() => hello.text.value === "hello from a directory");
console.log("4. reload:", JSON.stringify(report.plugins), "| hello:", hello.text.value);
console.log(" hello is now:", plugins.discovery.plugins.find(({ name }) => name === "hello")?.layout);

7
packages/chord/test/examples/61-tutorial/main.ts
View file

@ -211,10 +211,9 @@ if (terminal !== undefined) {
// 9. An edit applied in place: the lists plugin's worker facet trims a todo's text, so a blank one is empty and the
// store refuses it with Chord's `invalid/args`, which reaches Ada as it would a refused argument. The
// watcher marks the plugin dirty; applying the changes rolls it out to the server, the list's worker, the page and
// the terminal. On the worker each object of the plugin takes its fate, dependents first: each view swaps in place
// while the store is still live, then the store, which opens no native, swaps in place too (its live query runs
// again); the views read the store's old values until its new attempt is live, and their calls wait for it. Ada's
// reference follows them and never shows `loading`.
// the terminal. On the worker each object of the plugin restarts: running calls finish on the old code, new calls
// wait for the new code, documents and tables stay (the store's live query runs again on its new attempt). Ada's
// reference stays live while the old code drains, then shows `loading` briefly and the new code's snapshot.
// The edit: `add` in the `Todos` view, before and after.
const BEFORE = "add: (text: string, ctx: Ctx) => store.add({ text, by }, ctx),";
const AFTER = "add: (text: string, ctx: Ctx) => store.add({ text: text.trim(), by }, ctx),";

136
packages/chord/test/examples/64-old-work.ts Normal file
View file

@ -0,0 +1,136 @@
// Old work finishes on old code (decision "Reload restarts every affected object; old work finishes on old code"). A
// reload restarts every live object of a changed plugin. The calls running at the commit, and every call they cause
// (found through the `ctx` they pass on, and over trusted links), are old work: they run on the old attempts, which
// stay open until that work drained and their successors' requests are ready. New calls wait for the new code. Here:
// a provider's running call reaches its dependent while both restart, once with the provider's contract raised; a wait
// that would cycle through its own drain fails at once; and a call still running at `reloadDrainTimeout` is cut.
// Run from packages/chord:
// node --conditions=source --experimental-strip-types test/examples/64-old-work.ts
import { BACKGROUND_CONTEXT } from "../../src/context/index.ts";
import { evolve, local, method, str } from "../../src/contracts/index.ts";
import type { ChordError } from "../../src/data/index.ts";
import { type Ctx, createHost, defineFacet } from "../../src/host/index.ts";
import { constant } from "../../src/state/index.ts";
const context = BACKGROUND_CONTEXT;
/** A provider: `work` reaches back into the plugin once `gate` opens; `x` answers its code. */
const Provider = local("example.provider", {
version: 1,
params: {},
members: { work: method(undefined, str()), x: method(undefined, str()) },
});
/** The provider raised by a reload: a new member, version 1 still accepted. */
const ProviderV2 = evolve(Provider, (e) => e.addState("extra", str()));
/** A dependent: it holds the provider, and `ping` calls the provider's `x` back. */
const Dependent = local("example.dependent", { version: 1, params: {}, members: { ping: method(undefined, str()) } });
let gate = Promise.withResolvers<void>();
/** The plugin at `code`; `raised` serves `ProviderV2` (the dependent's code then needs it). */
function plugin(code: string, raised = false) {
const P = raised ? ProviderV2 : Provider;
return defineFacet((host) => {
host.serve(
P as typeof Provider,
() => ({
...(raised ? { extra: constant("x") } : {}),
work: async (ctx: Ctx) => {
await gate.promise;
const dependent = await ctx.get(Dependent, {});
return `${code} work -> ${await dependent.ping(ctx)}`;
},
x: async () => `${code} x`,
}),
raised ? { accepts: { 1: {} } } : {},
);
host.serve(Dependent, (_, self) => {
const provider = self.get(P as typeof Provider, {});
return { ping: async (ctx: Ctx) => `${code} ping -> ${await provider.x(ctx)}` };
});
});
}
/** Starts `work`, reloads to `next` while it waits at its gate, then opens the gate: prints how `work` ended. */
async function workAcross(label: string, host: ReturnType<typeof createHost>, next: ReturnType<typeof plugin>) {
gate = Promise.withResolvers();
const provider = await host.get(Provider, {}, context);
await host.get(Dependent, {}, context);
const working = provider
.work(context)
.catch((error: ChordError) => `${error.code}/${error.reason}: ${error.message}`);
await new Promise((resolve) => setTimeout(resolve, 0));
const report = await host.reload([{ name: "@example/app", facet: next }], context);
const started = Date.now();
gate.resolve();
console.log(`${label}\n reload: ${report.objects.map(({ object, kind }) => `${object} ${kind}`).join(", ")}`);
console.log(` work: ${await working} (${Date.now() - started < 1000 ? "at once" : "late"})`);
console.log(` a new call: ${await provider.x(context)}`);
}
// 1. Both objects restart while the provider's `work` runs. Its `ctx.get` of the dependent reaches the dependent's new
// attempt, which binds the provider's open old attempt; the back-call `x` is old work too, so it runs on the
// provider's old code. No call waits for the drain it is part of.
const host = createHost({ onError: () => {} });
await host.load([{ name: "@example/app", facet: plugin("v1") }], context);
await workAcross("1. both restart", host, plugin("v2"));
// 2. The reload raises the provider's version and the dependent's new code needs it. The dependent's successor waits
// for the provider's (which waits for `work`): its old attempt stays open, and `work` (started on v2, the code now
// replaced) runs on the old code there.
await workAcross("2. the provider's version raised", host, plugin("v3", true));
await host.close(context);
// 3. Old work that would wait for its own drain: `work` creates an object whose new code needs the raised provider,
// whose new attempt waits for `work` to end. It fails at once with `unavailable/reload_cycle`, naming the chain,
// instead of waiting for the reload drain bound.
// With params: an object of no params is a host-lifetime one, opened at the load.
const Fresh = local("example.fresh", {
version: 1,
params: { id: str() },
members: { ping: method(undefined, str()) },
});
const cycling = (code: string, raised: boolean) =>
defineFacet((h) => {
const P = raised ? ProviderV2 : Provider;
h.serve(
P as typeof Provider,
() => ({
...(raised ? { extra: constant("x") } : {}),
work: async (ctx: Ctx) => {
await gate.promise;
return (await ctx.get(Fresh, { id: "new" })).ping(ctx);
},
x: async () => code,
}),
raised ? { accepts: { 1: {} } } : {},
);
h.serve(Dependent, () => ({ ping: async () => code }));
h.serve(Fresh, (_, self) => {
if (raised) self.get(ProviderV2, {});
return { ping: async () => `${code} fresh` };
});
});
const cycle = createHost({ onError: () => {} });
await cycle.load([{ name: "@example/app", facet: cycling("v1", false) }], context);
await workAcross("3. a wait through its own drain", cycle, cycling("v2", true));
await cycle.close(context);
// 4. A call still running at `reloadDrainTimeout` (default 5 s, at most `closeTimeout`) is cut `released`: long work
// belongs in jobs. Here the bound is 200 ms and `work` never opens its gate.
const bounded = createHost({ reloadDrainTimeout: 200, onError: () => {} });
await bounded.load([{ name: "@example/app", facet: plugin("v1") }], context);
const provider = await bounded.get(Provider, {}, context);
gate = Promise.withResolvers();
const stuck = provider.work(context).catch((error: ChordError) => `${error.code}: ${error.message}`);
await new Promise((resolve) => setTimeout(resolve, 0));
const cutAt = Date.now();
await bounded.reload([{ name: "@example/app", facet: plugin("v2") }], context);
console.log(
`4. the reload drain bound\n work: ${await stuck} (after ${Math.round((Date.now() - cutAt) / 100) * 100} ms)`,
);
console.log(` a new call: ${await provider.x(context)}`);
gate.resolve();
await bounded.close(context);

14
packages/chord/test/host/adapted.test.ts
View file

@ -867,7 +867,7 @@ describe("reload: an older consumer's reads while its type is removed (fix round
});
});
describe("reload: an older consumer's first read during a parked swap (fix round 2)", () => {
describe("reload: an older consumer's first read during a parked restart (fix round 2)", () => {
it("maps with the provider whose attempt is shown until the new one goes live (Sol)", async () => {
const V1 = local("app.lazy", { version: 1, params: {}, members: { x: int() } });
const V2 = evolve(V1, (e) => e.state("x", obj({ n: int() })));
@ -878,7 +878,7 @@ describe("reload: an older consumer's first read during a parked swap (fix round
context,
);
const old = await host.get(V1, {}, context);
// Nothing read before the swap: the first read comes while the new attempt parks on a missing dependency.
// Nothing read before the reload: the first read comes while the new attempt parks on a missing dependency.
const v2 = defineFacet((h) =>
h.serve(
V2,
@ -1172,8 +1172,8 @@ describe("reload: stopping a keyed watch drops its pin and last value while its
});
});
describe("reload: a reference that never had values reads none during a parked swap (fix round 4)", () => {
it("has a newer reference read nothing during a parked swap, its watch waiting, then the new instance's values (Sol)", async () => {
describe("reload: a reference that never had values reads none during a parked restart (fix round 4)", () => {
it("has a newer reference read nothing during a parked restart, its watch waiting, then the new instance's values (Sol)", async () => {
const N1 = local("app.unshown", { version: 1, params: {}, members: { x: int() } });
const N2 = evolve(N1, (e) => e.state("x", obj({ n: int() })));
const Missing = local("app.unshown.missing", { version: 1, params: {}, members: {} });
@ -1196,8 +1196,8 @@ describe("reload: a reference that never had values reads none during a parked s
await host.reload([{ name: "@app/unshown", facet: v2 }], context);
const newer = host.hold(N2, {}, context);
expect(newer.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } });
// The swap parks: the shown values are v1's; a v2 reference never had values, never v1's shape: its read throws
// the parked swap's failure.
// The restart parks: the shown values are v1's; a v2 reference never had values, never v1's shape: its read throws
// the parked restart's failure.
expect(old.x.value).toBe(1);
expect(() => newer.x.value).toThrow(expect.objectContaining({ code: "unavailable" }));
// A watch waits for its first values through the failure (Opus G3).
@ -1238,7 +1238,7 @@ describe("reload: a reference reads its last value while its version's return pa
context,
);
expect(ref.x.value).toEqual({ n: 7 });
// v2 is served again but its swap parks: the sources taken back are v2's own, shown before.
// v2 is served again but its restart parks: the sources taken back are v2's own, shown before.
const parked = defineFacet((h) =>
h.serve(R2, (_, self) => {
self.get(Missing, {});

34
packages/chord/test/host/ctx.test.ts
View file

@ -5,6 +5,7 @@ import { int, local, method, remote, str } from "../../src/contracts/index.ts";
import { withLinkedAbortSignal } from "../../src/host/ctx.ts";
import type { ChordHost } from "../../src/host/host.ts";
import type { Ctx, Host, Reference } from "../../src/host/index.ts";
import { CALL_CHAIN } from "../../src/host/old-work.ts";
import { constant } from "../../src/state/index.ts";
import {
flushed,
@ -155,6 +156,39 @@ describe("ctx", () => {
});
});
describe('the call chain (decision "Reload restarts every affected object": old work)', () => {
it("a method's ctx is its own chain node, above it the chain of the call that made it; a root's has none", async () => {
const Inner = local("app.inner", { version: 1, params: {}, members: { run: method() } });
const Outer = local("app.outer", { version: 1, params: {}, members: { run: method() } });
const seen: { inner?: Ctx; outer?: Ctx; root?: Ctx } = {};
const { host } = await loaded((h) => {
h.serve(Inner, () => ({
run: async (ctx) => {
seen.inner = ctx;
return null;
},
}));
h.serve(Outer, () => ({
run: async (ctx) => {
seen.outer = ctx;
await (await ctx.get(Inner, {})).run(ctx);
return null;
},
}));
});
await host.root(async (root) => {
seen.root = root;
await (await root.get(Outer, {})).run(root);
}, context);
const inner = seen.inner!.value(CALL_CHAIN);
expect(inner).toBe(seen.inner);
expect(inner!.up()).toBe(seen.outer);
expect(inner!.up()!.up()).toBeUndefined();
expect(seen.root!.value(CALL_CHAIN)).toBeUndefined();
await host.close(context);
});
});
describe("local handlers act as the host (§4, audit)", () => {
it("a local object called from a remote handler's ctx.get runs with this host's principal and incarnation", async () => {
const Store = local("app.local-store", { version: 1, params: {}, members: { who: method(undefined, str()) } });

5
packages/chord/test/host/host.test.ts
View file

@ -1456,6 +1456,7 @@ describe("reload: the API (slice 7b §1)", () => {
code: "aborted",
details: { committed: { added: ["@app/new"], changed: ["@app/info"], removed: ["@app/gone"], failed: [] } },
});
await flushed();
expect(info.title.value).toBe("b");
await host.close(context);
});
@ -1492,6 +1493,8 @@ describe("reload: the API (slice 7b §1)", () => {
expect(report.failed.map(({ plugin, error }) => [plugin, (error as Error).message])).toEqual([
["@app/other", 'plugin "@app/other": app.info is served twice (plugin "@app/info")'],
]);
// Restarted: the new attempt goes live once the old one ended.
await flushed();
expect(info.title.value).toBe("b");
expect(info.state.value).toEqual({ kind: "live" });
await host.close(context);
@ -1812,7 +1815,7 @@ describe("the host's spans (slice 9)", () => {
fates: 1,
});
expect(reload.events).toEqual([
{ name: "chord.reload.fate", attributes: { object: "app.info", fate: "swapped" } },
{ name: "chord.reload.fate", attributes: { object: "app.info", fate: "restarted" } },
]);
});

713
packages/chord/test/host/object.test.ts
View file

@ -1,27 +1,30 @@
import { describe, expect, it, vi } from "vitest";
import { type Context, BACKGROUND_CONTEXT as context } from "../../src/context/index.ts";
import { type Context, BACKGROUND_CONTEXT as context, withoutAbortSignal } from "../../src/context/index.ts";
import { arr, evolve, int, local, method, obj, remote, str } from "../../src/contracts/index.ts";
import { broken, ChordError } from "../../src/data/index.ts";
import { defineState, type SingletonHandle } from "../../src/documents/index.ts";
import { Attempt } from "../../src/host/attempt.ts";
import {
type Ctx,
createHost,
defineFacet,
type FacetHost,
type FacetSelf,
type Host,
type HostError,
type PluginFacet,
type Reference,
type Self,
type StoreSelf,
untilAborted,
} from "../../src/host/index.ts";
import { ChordObject } from "../../src/host/object.ts";
import { defineJob } from "../../src/jobs/index.ts";
import { memoryLinkPair } from "../../src/link/index.ts";
import { jsonPart, type Message } from "../../src/link/internal.ts";
import { defineRegistry } from "../../src/registries/index.ts";
import { constant, type Watchable } from "../../src/state/index.ts";
import { query } from "../../src/store/query.ts";
import { defineTable } from "../../src/store/table.ts";
import {
changesOf,
delivered,
@ -32,7 +35,6 @@ import {
flushed,
holdNextCommit,
instancesOf,
linked,
loaded,
memoryStore,
metered,
@ -47,7 +49,6 @@ import {
StoreRestartable,
statesOf,
storeRestartable,
type TestPrincipal,
testCodec,
transported,
valuesOf,
@ -1234,14 +1235,14 @@ function servePure(h: FacetHost, label: string, by: number, gates: Promise<void>
});
}
describe("reload: fates (slice 7b §4)", () => {
it("swaps a pure object in place: same documents and instance id, new code for new calls", async () => {
describe('reload: fates (decision "Reload restarts every affected object")', () => {
it("restarts a live object: its documents kept, new code for new calls", async () => {
const { host } = await loaded((h) => servePure(h, "v1", 1));
const pure = await host.get(Pure, {}, context);
await pure.bump(context);
const objects = instancesOf(ChordObject, "reloadFate");
const report = await reloadApp(host, (h) => servePure(h, "v2", 10));
expect(report.objects).toEqual([{ object: "app.pure", kind: "swapped" }]);
expect(report.objects).toEqual([{ object: "app.pure", kind: "restarted" }]);
expect(objects).toHaveLength(1);
await flushed();
expect(pure.state.value).toEqual({ kind: "live" });
@ -1250,7 +1251,7 @@ describe("reload: fates (slice 7b §4)", () => {
await host.close(context);
});
it("restarts an object with natives: the drain first, its natives closed with reason reload, a new instance", async () => {
it("shows the old attempt live while it drains, closes its natives with reason reload, then opens the new ones", async () => {
const closes: string[] = [];
const serve = (h: FacetHost, label: string, gates: Promise<void>[] = []) =>
h.serve(Pure, (_, self) => {
@ -1279,14 +1280,15 @@ describe("reload: fates (slice 7b §4)", () => {
await flushed();
const report = await reloadApp(host, (h) => serve(h, "v2"));
expect(report.objects).toEqual([{ object: "app.pure", kind: "restarted" }]);
expect(pure.state.value).toEqual({ kind: "loading" });
// A new call parks until the new attempt is live.
const parked = pure.bump(context);
// The old attempt binds this reference's version: it sees it live while it drains (rule 5).
expect(pure.state.value).toEqual({ kind: "live" });
// A new call waits at the gate until the new attempt is live.
const waiting = pure.bump(context);
await flushed();
expect(closes).toEqual([]);
gate.resolve();
expect(await running).toBe(1);
expect(await parked).toBe(2);
expect(await waiting).toBe(2);
expect(closes).toEqual(["bump done on v1", "v1 reload", "bump done on v2"]);
expect(pure.label.value).toBe("v2");
await host.close(context);
@ -1375,7 +1377,7 @@ describe("reload: fates (slice 7b §4)", () => {
await host.close(context);
});
it("parks a swap whose new factory needs a missing dependency; it goes live when the dependency is served", async () => {
it("reports a restart parked whose new factory needs a missing dependency; it goes live once that is served", async () => {
const Dep = local("app.dep", { version: 1, params: {}, members: { name: str() } });
const { host } = await loaded((h) => servePure(h, "v1", 1));
const pure = await host.get(Pure, {}, context);
@ -1412,235 +1414,67 @@ describe("reload: fates (slice 7b §4)", () => {
});
});
describe('reload: fates start dependents first (decision "Swaps: dependents first")', () => {
/** A provider: its name and `echo`, which answers it. */
const Base = local("app.base", { version: 1, params: {}, members: { name: str(), echo: method(undefined, str()) } });
/** `Base` v2: a member added, so a consumer of v2 binds only the new code. */
const BaseV2 = evolve(Base, (e) => e.addState("extra", str()));
/** A dependent with empty params (it contributes): its label over the provider's name, `ask` through `echo`. */
const User = local("app.user", { version: 1, params: {}, members: { label: str(), ask: method(undefined, str()) } });
/** A remote dependent held over a link. */
const Viewer = remote("app.viewer", {
version: 1,
params: {},
hostAddress: () => Server(),
variants: { v: { state: { label: str() }, methods: { ask: method(undefined, str()) } } },
});
const Names = defineRegistry<string, string[]>("app.user-names", { version: 1 });
/** A host loaded with `facets`, its background errors collected. */
async function loadedFacets(facets: readonly PluginFacet[]) {
const errors: HostError[] = [];
const host = createHost({ onError: (error) => errors.push(error) });
await host.load(facets, context);
return { host, errors };
}
/** Serves `Base` as code `code`; its new code opens a native that waits for `opened` when given (a restart). */
function serveBase(h: FacetHost, code: string, opened?: Promise<void>): void {
h.serve(Base, (_, self) => {
if (opened !== undefined)
describe("reload: a restart's readiness while the old attempt drains", () => {
it("shows a failed request while it lasts: a request that recovers shows the object live again, new calls wait", async () => {
const Flaky = local("app.flaky", { version: 1, params: { id: int() }, members: { name: str() } });
let failing = true;
const flaky = defineFacet((h) =>
h.serve(Flaky, (_, self) => {
self.open(
async () => await opened,
async () => {
if (failing) throw new Error("flaky is down");
return null;
},
() => {},
);
return { name: constant(code), echo: async () => code };
});
}
/** A dependent's members as code `code` over `base`: its label over the provider's name, `ask` through `echo`. */
function over(own: Self, code: string, base: Reference<typeof Base>) {
return {
label: own.derive([base.name], ([name]) => `${code} ${name}`),
ask: async (ctx: Context) => `${code}>${await base.echo(ctx)}`,
};
}
/** Serves `User` as code `code` over `Base`. */
function serveUser(h: FacetHost, code: string): void {
h.serve(User, (_, own) => over(own, code, own.get(Base, {})));
}
it("a pure dependent and its provider in one plugin both swap: no parking, its contribution replaced in one batch", async () => {
const composed: string[][] = [];
const registry = defineFacet((h) =>
h.serve(Names, {
compose: (table) => {
const names = table.map(({ value }) => value);
composed.push(names);
return names;
},
return { name: constant("flaky") };
}),
);
const app = (code: string) =>
defineFacet((h) => {
serveBase(h, code);
h.serve(User, (_, own) => {
(own as unknown as StoreSelf).contribute(Names, code);
return over(own, code, own.get(Base, {}));
});
});
const { host, errors } = await loadedFacets([
{ name: "@test/app", facet: app("v1") },
{ name: "@test/names", facet: registry },
]);
const user = await host.get(User, {}, context);
const names = await host.registry(Names, context);
expect(names.value).toEqual(["v1"]);
const frames: string[][] = [];
(await names.watch(context)).start(async (call) => {
if (call.kind === "value") frames.push(call.value);
});
composed.length = 0;
const report = await host.reload(
const gate = Promise.withResolvers<void>();
const gates = [gate.promise];
const host = createHost({ onError: () => {} });
await host.load(
[
{ name: "@test/app", facet: app("v2") },
{ name: "@test/names", facet: registry },
{ name: "@test/app", facet: defineFacet((h) => servePure(h, "v1", 1, gates)) },
{ name: "@test/flaky", facet: flaky },
],
context,
);
// The report keeps the order of the objects: the provider first.
expect(report.objects).toEqual([
{ object: "app.base", kind: "swapped" },
{ object: "app.user", kind: "swapped" },
]);
const pure = await host.get(Pure, {}, context);
const down = host.hold(Flaky, { id: 1 }, context);
await vi.waitFor(() => expect(down.state.value.kind).toBe("failed"));
const running = pure.bump(context);
await flushed();
// The old contribution left in the batch the new one entered: one composition, never a table without it.
expect(names.value).toEqual(["v2"]);
expect(composed).toEqual([["v2"]]);
await delivered();
expect(frames).toEqual([["v2"]]);
expect(user.label.value).toBe("v2 v2");
expect(await user.ask(context)).toBe("v2>v2");
expect(failures(errors)).toEqual([]);
await host.close(context);
const needsFlaky = (h: FacetHost) =>
h.serve(Pure, (_, self) => {
self.get(Flaky, { id: 1 });
return { n: constant(0), label: constant("v2"), bump: async () => 2 };
});
it("a remote dependent's holds never see loading; its calls reach the provider's new code", async () => {
const serve = (h: FacetHost<TestPrincipal>, code: string) => {
serveBase(h as FacetHost, code);
h.serve(Viewer, (_, _principal, own) => ({ variant: "v", members: over(own, code, own.get(Base, {})) }));
};
const { server, client } = await linked((h) => serve(h, "v1"));
const viewer = await client.get(Viewer, {}, context);
const states = statesOf(await viewer.state.watch(context));
const report = await server.reload(
[{ name: "@test/app", facet: defineFacet<TestPrincipal>((h) => serve(h, "v2")) }],
await host.reload(
[
{ name: "@test/app", facet: defineFacet(needsFlaky) },
{ name: "@test/flaky", facet: flaky },
],
context,
);
expect(report.objects).toEqual([
{ object: "app.base", kind: "swapped" },
{ object: "app.viewer", kind: "swapped" },
]);
expect(await viewer.ask(context)).toBe("v2>v2");
await transported();
expect(viewer.label.value).toBe("v2 v2");
expect(states).toEqual([]);
await client.close(context);
await server.close(context);
});
it("a dependent whose new code needs the provider's new version parks, then goes live: never on the old instance", async () => {
const app = (code: string, next: boolean) =>
defineFacet((h) => {
if (next) {
h.serve(BaseV2, () => ({ name: constant(code), extra: constant("extra"), echo: async () => code }));
} else serveBase(h, code);
h.serve(User, (_, own) => {
if (next) {
const base = own.get(BaseV2, {});
return {
label: own.derive([base.name, base.extra], ([name, extra]) => `${code} ${name} ${extra}`),
ask: async (ctx: Context) => `${code}>${await base.echo(ctx)}`,
};
}
return over(own, code, own.get(Base, {}));
});
});
const { host, errors } = await loadedFacets([{ name: "@test/app", facet: app("v1", false) }]);
const user = await host.get(User, {}, context);
const report = await host.reload([{ name: "@test/app", facet: app("v2", true) }], context);
// Started first, the dependent's v2 request does not bind the provider still on v1 (`ChordObject.hold`): it
// waits unbound, then binds the provider's swap and parks until its new instance is live.
expect(report.objects).toEqual([
{ object: "app.base", kind: "swapped" },
{
object: "app.user",
kind: "parked",
cause: {
kind: "waiting",
error: expect.objectContaining({ message: expect.stringContaining("app.base v2 is not served here") }),
},
},
]);
await flushed();
expect(user.state.value).toEqual({ kind: "live" });
expect(user.label.value).toBe("v2 v2 extra");
expect(await user.ask(context)).toBe("v2>v2");
expect(errors.filter(({ kind }) => kind === "activation_failed" || kind === "object_failed")).toEqual([]);
await host.close(context);
});
it("a pure dependent of a provider that restarts swaps; its calls park until the provider's new instance is live", async () => {
const opened = Promise.withResolvers<void>();
const app = (code: string, natives: boolean) =>
defineFacet((h) => {
serveBase(h, code, natives ? opened.promise : undefined);
serveUser(h, code);
});
const { host } = await loadedFacets([{ name: "@test/app", facet: app("v1", false) }]);
const user = await host.get(User, {}, context);
const report = await host.reload([{ name: "@test/app", facet: app("v2", true) }], context);
expect(report.objects).toEqual([
{ object: "app.base", kind: "restarted" },
{ object: "app.user", kind: "swapped" },
]);
await flushed();
// Live on its new code over the provider's last values while the provider restarts.
expect(user.state.value).toEqual({ kind: "live" });
expect(user.label.value).toBe("v2 v1");
const asked = user.ask(context);
await flushed();
opened.resolve();
expect(await asked).toBe("v2>v2");
await flushed();
expect(user.label.value).toBe("v2 v2");
await host.close(context);
});
it("a provider that went live after its dependent (a retry after a failure) swaps first: the dependent parks and recovers", async () => {
const selves: Self[] = [];
const app = (code: string) =>
defineFacet((h) => {
h.serve(Base, (_, own) => {
selves.push(own);
return { name: constant(code), echo: async () => code };
});
serveUser(h, code);
});
const { host } = await loadedFacets([{ name: "@test/app", facet: app("v1") }]);
const user = await host.get(User, {}, context);
const base = await host.get(Base, {}, context);
failLive(attempted, selves.at(-1)!, new ChordError({ code: "unavailable", message: "boom" }));
await flushed();
base.retry();
await flushed();
expect(base.state.value).toEqual({ kind: "live" });
const report = await host.reload([{ name: "@test/app", facet: app("v2") }], context);
// The objects in the order they went live: the dependent first, so the provider's swap starts before it.
expect(report.objects).toEqual([
{ object: "app.user", kind: "parked", cause: expect.objectContaining({ kind: "waiting" }) },
{ object: "app.base", kind: "swapped" },
]);
await flushed();
expect(user.state.value).toEqual({ kind: "live" });
expect(user.label.value).toBe("v2 v2");
expect(await user.ask(context)).toBe("v2>v2");
// The successor's request failed: holders see it while it lasts (old work still runs on the old attempt).
expect(pure.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } });
failing = false;
down.retry();
await vi.waitFor(() => expect(down.state.value.kind).toBe("live"));
// Recovered during the drain: the old attempt is shown again, and a new call waits for the new code.
expect(pure.state.value).toEqual({ kind: "live" });
const next = pure.bump(context);
gate.resolve();
expect(await running).toBe(1);
expect(await next).toBe(2);
expect(pure.label.value).toBe("v2");
down.release();
await host.close(context);
});
});
describe("reload: the pure swap (slice 7b §5)", () => {
describe("reload: a restart's old attempt (rules 1-4)", () => {
it("lets a running call finish on the old attempt's handles; a new call waits at the gate for the new code", async () => {
const gate = Promise.withResolvers<void>();
const { host } = await loaded((h) => servePure(h, "v1", 1, [gate.promise]));
@ -1648,14 +1482,14 @@ describe("reload: the pure swap (slice 7b §5)", () => {
const old = pure.bump(context);
await flushed();
await reloadApp(host, (h) => servePure(h, "v2", 10));
// The old call commits through its old handle after the swap.
// The old call commits through its old handle after the commit.
gate.resolve();
expect(await old).toBe(1);
expect(await pure.bump(context)).toBe(11);
await host.close(context);
});
it("an old call touching a document whose version rose throws once (R15); the document is migrated", async () => {
it("old work writes a document whose version the reload raises; the new code migrates what it wrote", async () => {
const V1 = defineState("doc", COUNTER);
const V2 = defineState("doc", {
schema: obj({ n: int(), unit: str() }),
@ -1686,10 +1520,13 @@ describe("reload: the pure swap (slice 7b §5)", () => {
await flushed();
await reloadApp(host, (h) => serve(h, V2, []));
await flushed();
expect(pure.label.value).toBe('{"n":1,"unit":"pcs"}');
// The new factory has not run: the old attempt's documents are as the old code declared them.
expect(pure.label.value).toBe('{"n":1}');
gate.resolve();
expect(await rejection(old)).toMatchObject({ code: "released", reason: "version" });
expect(await pure.bump(context)).toBe(2);
expect(await old).toBe(2);
await flushed();
expect(pure.label.value).toBe('{"n":2,"unit":"pcs"}');
expect(await pure.bump(context)).toBe(3);
await host.close(context);
});
@ -1765,7 +1602,7 @@ describe("reload: the pure swap (slice 7b §5)", () => {
await host.close(context);
});
it("hands its routines over: the old ones aborted, the new ones start after the old calls finished", async () => {
it("stops its routines at once; their calls drain before the new ones start", async () => {
const events: string[] = [];
const release = Promise.withResolvers<void>();
const serve = (h: FacetHost, label: string) =>
@ -1836,10 +1673,10 @@ describe("reload: the pure swap (slice 7b §5)", () => {
});
});
describe("reload: a stuck routine call past the handover", () => {
describe("reload: a stuck routine call past the drain", () => {
fakeTimers();
it("an old routine call stuck past the handover: drain_timeout, the old attempt sealed, its commit refused (audit A12)", async () => {
it("an old routine call stuck past the reload drain bound: drain_timeout, the old attempt sealed, its commit refused", async () => {
const stuck = Promise.withResolvers<void>();
let late: Promise<unknown> | undefined;
const errors: HostError[] = [];
@ -1871,10 +1708,404 @@ describe("reload: a stuck routine call past the handover", () => {
});
});
/** The provider of the old-work shapes: `work` reaches back into the plugin after its gate; `x` answers at once. */
const Provider = local("app.provider", {
version: 1,
params: {},
members: { work: method(undefined, str()), x: method(undefined, str()) },
});
/** `Provider` raised by a reload: a new member, `accepts` keeping version 1. */
const ProviderV2 = evolve(Provider, (e) => e.addState("extra", str()));
/** The dependent of the old-work shapes: it holds the provider; `ping` may call it back. */
const Dependent = local("app.dependent", {
version: 1,
params: {},
members: { ping: method(undefined, str()), slow: method(undefined, str()) },
});
/** `Dependent` raised by a reload: a new member, `accepts` keeping version 1. */
const DependentV2 = evolve(Dependent, (e) => e.addState("extra", str()));
/** A third object the shapes create or chain through: it holds the provider. */
const Third = local("app.third", { version: 1, params: { id: str() }, members: { ping: method(undefined, str()) } });
/** How a shape's plugin behaves: what `work` does after its gate, what `ping` does, which contracts the reload raises. */
type Shape = {
readonly work: (ctx: Ctx, code: string) => Promise<string>;
readonly ping?: (provider: Reference<typeof Provider>, ctx: Ctx, code: string) => Promise<string>;
readonly slow?: (provider: Reference<typeof Provider>, ctx: Ctx, code: string) => Promise<string>;
readonly raise?: "provider" | "both";
};
/** The shape's plugin at `code` (`v1`, then `v2` for the reload); `work` and `slow` first await `gates.shift()`. */
function shapePlugin(code: string, shape: Shape, gates: Promise<void>[]) {
const raised = code !== "v1";
const P = raised && shape.raise !== undefined ? ProviderV2 : Provider;
const D = raised && shape.raise === "both" ? DependentV2 : Dependent;
const extra = P === ProviderV2 ? { extra: constant("x") } : {};
return defineFacet((h) => {
h.serve(
P as typeof Provider,
() => ({
...extra,
work: async (ctx: Ctx) => {
await gates.shift();
return shape.work(ctx, code);
},
x: async () => `${code} x`,
}),
P === ProviderV2 ? { accepts: { 1: {} } } : {},
);
h.serve(
D as typeof Dependent,
(_, self) => {
const provider = self.get(P as typeof Provider, {});
return {
...(D === DependentV2 ? { extra: constant("x") } : {}),
ping: async (ctx: Ctx) => (shape.ping === undefined ? `${code} ping` : shape.ping(provider, ctx, code)),
slow: async (ctx: Ctx) => {
await gates.shift();
return shape.slow === undefined ? `${code} slow` : shape.slow(provider, ctx, code);
},
};
},
D === DependentV2 ? { accepts: { 1: {} } } : {},
);
h.serve(Third, (_, self) => {
self.get(P as typeof Provider, {});
return { ping: async () => `${code} third` };
});
});
}
/**
* Loads the shape, starts the provider's `work` (held at a gate), reloads to `v2` and opens the gate: resolves with
* `work`'s answer (or its error's message) and how long it took after the gate opened.
*/
async function runShape(shape: Shape, options: { readonly before?: (host: Host) => Promise<void> } = {}) {
const gates: Promise<void>[] = [];
const gate = Promise.withResolvers<void>();
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: shapePlugin("v1", shape, gates) }], context);
const provider = await host.get(Provider, {}, context);
await host.get(Dependent, {}, context);
await options.before?.(host);
gates.push(gate.promise);
const work = provider.work(context).catch((error: Error) => `rejected: ${error.message}`);
await flushed();
await host.reload([{ name: "@test/app", facet: shapePlugin("v2", shape, gates) }], context);
const opened = performance.now();
gate.resolve();
const answer = await work;
const took = performance.now() - opened;
await host.close(context);
return { answer, took };
}
describe('reload: old work finishes on old code (decision "Reload restarts every affected object")', () => {
it("a provider's running call reaches its dependent through ctx.get while both restart: no stall", async () => {
const { answer, took } = await runShape({
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Dependent, {})).ping(ctx)}`,
});
// The dependent's new attempt binds the provider's open old attempt (rule 5) and goes live.
expect(answer).toBe("v1 work -> v2 ping");
expect(took).toBeLessThan(1_000);
});
it("a chain of three: the work reaches an object whose new attempt needs the dependent, which needs the provider", async () => {
const { answer } = await runShape({
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Third, { id: "a" })).ping(ctx)}`,
});
expect(answer).toBe("v1 work -> v2 third");
});
it("re-entry: work → dependent → the provider again runs on the provider's old attempt", async () => {
const { answer } = await runShape({
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Dependent, {})).ping(ctx)}`,
ping: async (provider, ctx, code) => `${code} ping -> ${await provider.x(ctx)}`,
});
expect(answer).toBe("v1 work -> v2 ping -> v1 x");
});
it("self re-entry: work → ctx.get of its own object runs on its old attempt", async () => {
const { answer } = await runShape({
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Provider, {})).x(ctx)}`,
});
expect(answer).toBe("v1 work -> v1 x");
});
it("mutual drains: the provider's call reaches the dependent while the dependent's call reaches the provider", async () => {
const gates: Promise<void>[] = [];
const shape: Shape = {
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Dependent, {})).ping(ctx)}`,
slow: async (provider, ctx, code) => `${code} slow -> ${await provider.x(ctx)}`,
};
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: shapePlugin("v1", shape, gates) }], context);
const provider = await host.get(Provider, {}, context);
const dependent = await host.get(Dependent, {}, context);
const first = Promise.withResolvers<void>();
const second = Promise.withResolvers<void>();
gates.push(first.promise, second.promise);
const work = provider.work(context);
const slow = dependent.slow(context);
await flushed();
await host.reload([{ name: "@test/app", facet: shapePlugin("v2", shape, gates) }], context);
second.resolve();
first.resolve();
expect(await slow).toBe("v1 slow -> v1 x");
expect(await work).toMatch(/^v1 work -> v[12] ping$/);
await host.close(context);
});
it("the reload raises the provider's version and the dependent's new code needs it: the work runs on the dependent's old attempt", async () => {
const { answer, took } = await runShape({
raise: "provider",
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Dependent, {})).ping(ctx)}`,
});
// The dependent's successor waits for the provider's: its old attempt stays open for old work (rule 3).
expect(answer).toBe("v1 work -> v1 ping");
expect(took).toBeLessThan(1_000);
});
it("both versions raised (with accepts): old work's getter at the old version binds the dependent's open old attempt", async () => {
const { answer } = await runShape({
raise: "both",
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Dependent, {})).ping(ctx)}`,
});
expect(answer).toBe("v1 work -> v1 ping");
});
it("old work asking for a version the new code dropped, with no old attempt serving it, fails at once", async () => {
const Dropped = local("app.dropped", { version: 1, params: {}, members: { ping: method(undefined, str()) } });
const Raised = local("app.dropped", {
version: 2,
params: {},
members: { ping: method(undefined, str()), n: int() },
});
const app = (code: string, gate: Promise<void>) =>
defineFacet((h) => {
h.serve(Provider, () => ({
work: async (ctx: Ctx) => {
await gate;
return (await ctx.get(Dropped, {})).ping(ctx);
},
x: async () => code,
}));
if (code === "v1") h.serve(Dropped, () => ({ ping: async () => "v1 ping" }));
else h.serve(Raised, () => ({ ping: async () => "v2 ping", n: constant(0) }));
});
const gate = Promise.withResolvers<void>();
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: app("v1", gate.promise) }], context);
const provider = await host.get(Provider, {}, context);
const work = rejection(provider.work(context));
await flushed();
await host.reload([{ name: "@test/app", facet: app("v2", gate.promise) }], context);
const opened = performance.now();
gate.resolve();
// No object of the dropped version is open: old work never waits for a code update (rule 2).
expect(await work).toMatchObject({ code: "contract_mismatch" });
expect(performance.now() - opened).toBeLessThan(1_000);
await host.close(context);
});
it("old work creating an object that did not exist: it starts on the new code and binds the provider's open old attempt", async () => {
const { answer } = await runShape({
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Third, { id: "fresh" })).ping(ctx)}`,
});
expect(answer).toBe("v1 work -> v2 third");
});
it("a root given the method's ctx (without its signal) is old work; one without it is new work", async () => {
let host: Host | undefined;
const viaRoot = (scope: (ctx: Ctx) => Context): Shape => ({
work: async (ctx, code) => {
const answer = await host!.root(async (root) => (await root.get(Provider, {})).x(root), scope(ctx));
return `${code} work -> ${answer}`;
},
});
const run = async (shape: Shape) => {
const gates: Promise<void>[] = [];
const gate = Promise.withResolvers<void>();
host = createHost({ reloadDrainTimeout: 300, onError: () => {} });
await host.load([{ name: "@test/app", facet: shapePlugin("v1", shape, gates) }], context);
const provider = await host.get(Provider, {}, context);
gates.push(gate.promise);
const work = provider.work(context).catch((error: Error) => `rejected: ${error.message}`);
await flushed();
await host.reload([{ name: "@test/app", facet: shapePlugin("v2", shape, gates) }], context);
gate.resolve();
const answer = await work;
await host.close(context);
return answer;
};
expect(await run(viaRoot((ctx) => withoutAbortSignal(ctx)))).toBe("v1 work -> v1 x");
// Without the ctx the root's call is new work: it waits for the provider's new attempt, which waits for this
// call's drain, until the reload drain bound cuts the call.
expect(await run(viaRoot(() => context))).toMatch(/^rejected: .*replaced by a reload/);
});
it("a call routed to an old attempt after its drain began joins the drain: it is not cut at the seal", async () => {
const gates: Promise<void>[] = [];
const long = Promise.withResolvers<void>();
const shape: Shape = {
raise: "provider",
work: async (ctx, code) => `${code} work -> ${await (await ctx.get(Dependent, {})).slow(ctx)}`,
};
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: shapePlugin("v1", shape, gates) }], context);
const provider = await host.get(Provider, {}, context);
await host.get(Dependent, {}, context);
const gate = Promise.withResolvers<void>();
gates.push(gate.promise, long.promise);
const work = provider.work(context);
await flushed();
await host.reload([{ name: "@test/app", facet: shapePlugin("v2", shape, gates) }], context);
gate.resolve();
// The dependent's old attempt runs the routed `slow` while its successor waits for the provider's.
await new Promise((resolve) => setTimeout(resolve, 50));
long.resolve();
expect(await work).toBe("v1 work -> v1 slow");
await host.close(context);
});
it("old work starts a job, writes documents and tables and reads a live query on the old code; the new factory runs after", async () => {
const store = await memoryStore();
const opened: string[] = [];
const latch = Promise.withResolvers<void>();
const Notes1 = defineState("notes", {
schema: obj({ list: arr(str()) }),
version: 1,
initial: () => ({ list: [] }),
persist: true,
checkpointWhen: () => true,
});
const Notes2 = defineState("notes", {
schema: obj({ list: arr(str()), count: int() }),
version: 2,
initial: () => ({ list: [], count: 0 }),
migrate: (old) => ({ list: old.list as string[], count: (old.list as string[]).length }),
persist: true,
checkpointWhen: () => true,
});
const Job1 = defineJob("app.job", { version: 1, args: obj({ text: str() }), result: str() });
const Job2 = defineJob("app.job", {
version: 2,
args: obj({ text: str(), priority: int() }),
result: str(),
accepts: { 1: { args: (old: { text: string }) => ({ ...old, priority: 0 }), result: (result) => result } },
});
const Items = defineTable("items", { version: 1, columns: { id: int(), text: str() }, key: ["id"] });
const Store = local("app.store", {
version: 1,
params: {},
members: {
rows: arr(obj({ id: int(), text: str() })),
work: method(undefined, str()),
read: method(undefined, str()),
},
});
const v1 = defineFacet((h) =>
h.serve(Store, (_, self) => {
self.open(
async () => void opened.push("v1"),
() => {},
);
const notes = self.state(Notes1);
const job = self.job(Job1, { run: async ({ text }) => `v1 ran ${text}` });
const items = self.table(Items);
const rows = self.query(
items,
query(Items).orderBy((item) => item.id, "desc"),
10,
);
return {
rows,
work: async (ctx: Ctx) => {
await latch.promise;
await notes.change((draft) => void draft.list.push("old"), ctx);
await self.tx(async (tx) => tx.put(items, { id: 1, text: "old" }), ctx);
const outcome = await (await job.start({ text: "a" }, ctx)).outcome(ctx);
const result = outcome.kind === "done" ? outcome.result : outcome.error.kind;
return `v1: ${notes.value.list.join(",")} | ${JSON.stringify(rows.value)} | ${result}`;
},
read: async () => `v1: ${JSON.stringify(notes.value)}`,
};
}),
);
const v2 = defineFacet((h) =>
h.serve(Store, (_, self) => {
self.open(
async () => void opened.push("v2"),
() => {},
);
const notes = self.state(Notes2);
self.job(Job2, { run: async ({ text, priority }) => `v2 ran ${text} ${priority}` });
const items = self.table(Items);
return {
rows: self.query(
items,
query(Items).orderBy((item) => item.id, "desc"),
10,
),
work: async () => "v2",
read: async () => `v2: ${JSON.stringify(notes.value)}`,
};
}),
);
const host = createHost({ store, onError: () => {} });
await host.load([{ name: "@test/app", facet: v1 }], context);
const reference = await host.get(Store, {}, context);
const working = reference.work(context);
await flushed();
await host.reload([{ name: "@test/app", facet: v2 }], context);
await flushed();
expect(opened).toEqual(["v1"]);
latch.resolve();
expect(await working).toBe('v1: old | [{"id":1,"text":"old"}] | v1 ran a');
expect(await reference.read(context)).toBe('v2: {"list":["old"],"count":1}');
expect(opened).toEqual(["v1", "v2"]);
await host.close(context);
await store.close(context);
});
});
describe("reload: the reload drain bound", () => {
fakeTimers();
it("cuts a call still running at reloadDrainTimeout (released), reports drain_timeout, then the new code serves", async () => {
const stuck = Promise.withResolvers<void>();
const { host, errors } = await loaded((h) => servePure(h, "v1", 1, [stuck.promise]), {
reloadDrainTimeout: 2_000,
});
const pure = await host.get(Pure, {}, context);
const old = rejection(pure.bump(context));
await vi.advanceTimersByTimeAsync(0);
await reloadApp(host, (h) => servePure(h, "v2", 10));
await vi.advanceTimersByTimeAsync(1_999);
expect(errors.map(({ kind }) => kind)).not.toContain("drain_timeout");
await vi.advanceTimersByTimeAsync(1);
expect(errors.map(({ kind }) => kind)).toContain("drain_timeout");
expect(await old).toMatchObject({ code: "released" });
await vi.advanceTimersByTimeAsync(0);
expect(pure.label.value).toBe("v2");
stuck.resolve();
await host.close(context);
});
it("is at most closeTimeout", () => {
const host = createHost({ closeTimeout: 1_000, reloadDrainTimeout: 5_000, onError: () => {} });
expect((host as unknown as { readonly timeouts: { readonly reload: number } }).timeouts.reload).toBe(1_000);
});
});
describe("reload: GC", () => {
const attempts = instancesOf(Attempt, "activate");
it("collects a swapped-out attempt and its closures once its calls settled", async () => {
it("collects a restart's old attempt and its closures once it ended", async () => {
const gate = Promise.withResolvers<void>();
const { host } = await loaded((h) => servePure(h, "v1", 1, [gate.promise]));
const pure = await host.get(Pure, {}, context);
@ -2014,8 +2245,8 @@ describe("reload: a parked attempt's handover (fix round 1)", () => {
});
});
describe("reload: swaps one after another (fix round 1)", () => {
it("a second swap does not wait for the first swap's old call; that call still finishes on its old code", async () => {
describe("reload: two reloads in a row", () => {
it("the second does not wait for the first's old call, which finishes on its old code; the last code serves", async () => {
const gate = Promise.withResolvers<void>();
const { host, errors } = await loaded((h) => servePure(h, "v1", 1, [gate.promise]), { closeTimeout: 60_000 });
const pure = await host.get(Pure, {}, context);
@ -2027,19 +2258,21 @@ describe("reload: swaps one after another (fix round 1)", () => {
resolved = true;
});
await flushed();
// Not held up by v1's call (Fable, Opus F1): resolved within a few turns, the gate open for new calls.
// Its report does not wait for the old attempt's drain: resolved within a few turns.
expect(resolved).toBe(true);
await second;
expect(pure.label.value).toBe("v3");
// The old attempt still runs v1's call: the reference sees it live meanwhile.
expect(pure.label.value).toBe("v1");
gate.resolve();
expect(await old).toBe(1);
expect(await pure.bump(context)).toBe(101);
expect(pure.label.value).toBe("v3");
expect(errors).toEqual([]);
await host.close(context);
});
});
describe("reload: a reference detaching from a parked swap's old instance (Opus G1)", () => {
describe("reload: a reference detaching from a parked restart's old attempt (Opus G1)", () => {
it("follows the old instance while its call drains, then keeps its last value with the failure once", async () => {
const Missing = local("app.missing", { version: 1, params: {}, members: {} });
const gate = Promise.withResolvers<void>();
@ -2057,7 +2290,7 @@ describe("reload: a reference detaching from a parked swap's old instance (Opus
const report = await host.reload([{ name: "@test/app", facet: parked }], context);
expect(report.objects[0]).toMatchObject({ kind: "parked" });
expect(pure.state.value).toMatchObject({ kind: "failed", cause: { kind: "waiting" } });
// The old instance's call still commits: the reference follows it until it retires.
// The old attempt's call still commits: the reference follows it until it ended.
gate.resolve();
expect(await bumping).toBe(1);
await delivered();

80
packages/chord/test/host/old-work.test.ts Normal file
View file

@ -0,0 +1,80 @@
// Tests of old work's bookkeeping (decision "Reload restarts every affected object; old work finishes on old code"):
// the call chain walk, its guard outside reloads, another host's old work, and the chain's attempts for reload cycles.
// The behaviour of old work in a reload is tested with the object (object.test.ts) and over links (provider.test.ts).
import { describe, expect, it } from "vitest";
import { type Context, BACKGROUND_CONTEXT as context, withContextValue } from "../../src/context/index.ts";
import {
asRemoteOldWork,
CALL_CHAIN,
type CallNode,
chainAttempts,
OldWork,
type OldWorkSource,
} from "../../src/host/old-work.ts";
/** A chain of nodes on `sources`, innermost first, and a count of the `up` steps taken through it. */
function chain(...sources: OldWorkSource[]): { readonly context: Context; readonly steps: () => number } {
let steps = 0;
let node: CallNode | undefined;
for (const on of sources.toReversed()) {
const up = node;
node = {
on,
up: () => {
steps++;
return up;
},
};
}
return { context: node === undefined ? context : withContextValue(CALL_CHAIN, node, context), steps: () => steps };
}
describe("OldWork.of: whether a context runs under old work", () => {
it("finds a draining attempt anywhere up the call chain", () => {
const oldWork = new OldWork();
oldWork.open();
expect(oldWork.of(chain({ draining: false }, { draining: false }, { draining: true }).context)).toBe(true);
expect(oldWork.of(chain({ draining: false }, { draining: false }).context)).toBe(false);
expect(oldWork.of(context)).toBe(false);
});
it("walks no chain while the host has no old work open (no cost outside reloads)", () => {
const oldWork = new OldWork();
const calls = chain({ draining: false }, { draining: true });
expect(oldWork.of(calls.context)).toBe(false);
expect(calls.steps()).toBe(0);
oldWork.open();
expect(oldWork.of(calls.context)).toBe(true);
expect(calls.steps()).toBe(1);
oldWork.close();
expect(oldWork.of(calls.context)).toBe(false);
expect(calls.steps()).toBe(1);
});
});
describe("asRemoteOldWork: a call that arrived as old work from another host", () => {
it("is old work while counted, whatever chain it had", () => {
const oldWork = new OldWork();
const marked = asRemoteOldWork(chain({ draining: false }).context);
expect(oldWork.of(marked)).toBe(false);
oldWork.open();
expect(oldWork.of(marked)).toBe(true);
});
it("has no attempt of this host in its chain (a reload cycle cannot pass through another host's drain)", () => {
const local = { draining: true };
expect([...chainAttempts(asRemoteOldWork(context))]).toEqual([]);
const above = chain(local).context;
expect([...chainAttempts(above)]).toEqual([local]);
});
});
describe("chainAttempts: the attempts a call chain runs on", () => {
it("collects each attempt once, draining or not", () => {
const a = { draining: true };
const b = { draining: false };
expect([...chainAttempts(chain(a, b, a).context)]).toEqual([a, b]);
expect([...chainAttempts(context)]).toEqual([]);
});
});

246
packages/chord/test/host/readiness.test.ts Normal file
View file

@ -0,0 +1,246 @@
// Tests of a restart's readiness (decision "Reload restarts every affected object; old work finishes on old code",
// rule 3): the recorded requests bound as the successor's would be, when they count as ready, the bound, the cause a
// reload reports and its changes, presence, and the release once the successor's own requests bound.
import { describe, expect, it } from "vitest";
import { BACKGROUND_CONTEXT as context } from "../../src/context/index.ts";
import { evolve, int, local, remote, str } from "../../src/contracts/index.ts";
import type { FailureCause } from "../../src/data/index.ts";
import { createHost, defineFacet, type FacetHost } from "../../src/host/index.ts";
import type { ChordHost } from "../../src/host/internal.ts";
import { Readiness } from "../../src/host/readiness.ts";
import type { Request } from "../../src/host/staging.ts";
import { defineRegistry } from "../../src/registries/index.ts";
import { constant } from "../../src/state/index.ts";
import { expectCollected, flushed, instancesOf, linked, Server, type TestPrincipal } from "../support.ts";
const Slow = local("app.slow", { version: 1, params: { id: int() }, members: { n: int() } });
/** Every readiness started while this file runs, weakly. */
const started = instancesOf(Readiness, "start");
const Missing = local("app.missing", { version: 1, params: {}, members: { n: int() } });
// As staging records it (a request names the registry whatever its types).
const Tags = defineRegistry<unknown, unknown>("app.tags", { version: 1 });
// With params: an object of no params is a host-lifetime one, opened at the load and never ended for lack of holders.
const slowRequest: Request = { kind: "type", contract: Slow, params: { id: 1 } };
/** A host serving `Slow`, whose native opens once `opened` resolves and logs its close to `closed`. */
async function slowHost(opened: Promise<void>, closed: string[] = []) {
const host = createHost({ onError: () => {} });
const facet = defineFacet((h) =>
h.serve(Slow, (_, self) => {
self.open(
async () => {
await opened;
return null;
},
(_native, reason) => void closed.push(reason.kind),
);
return { n: constant(1) };
}),
);
await host.load([{ name: "@test/slow", facet }], context);
return host as ChordHost;
}
/** A readiness of `requests` on `host`, as the host's own caller; each failure it shows is pushed to `shown`. */
function readinessOf(host: ChordHost, requests: readonly Request[], shown: (string | undefined)[] = []): Readiness {
const caller = { principal: host.principal, incarnation: host.incarnation };
const shows = (cause: FailureCause | undefined): void => void shown.push(cause?.error.message);
return new Readiness(host, caller, requests, shows, context);
}
/** Whether `promise` settled after a few turns. */
async function settled(promise: Promise<unknown>): Promise<boolean> {
let done = false;
void promise.then(() => {
done = true;
});
await new Promise((resolve) => setTimeout(resolve, 10));
return done;
}
describe("Readiness: when the recorded requests are ready", () => {
it("waits for a request still loading and resolves once it is live", async () => {
const opened = Promise.withResolvers<void>();
const host = await slowHost(opened.promise);
const readiness = readinessOf(host, [slowRequest]);
const ready = readiness.ready(Date.now() + 10_000);
readiness.start();
expect(await settled(ready)).toBe(false);
opened.resolve();
await ready;
expect(readiness.cause).toBeUndefined();
readiness.release();
await host.close(context);
});
it("waits for nothing before it started: the requests bind only once every fate started", async () => {
const host = await slowHost(Promise.resolve());
const readiness = readinessOf(host, [slowRequest]);
const ready = readiness.ready(Date.now() + 10_000);
expect(await settled(ready)).toBe(false);
readiness.start();
await ready;
readiness.release();
await host.close(context);
});
it("resolves at its bound when a request never gets ready", async () => {
const host = await slowHost(new Promise(() => {}));
const readiness = readinessOf(host, [slowRequest]);
readiness.start();
const started = Date.now();
await readiness.ready(started + 50);
expect(Date.now() - started).toBeGreaterThanOrEqual(40);
readiness.release();
await host.close(context);
});
it("counts a request failed on this host as ready; the reload reports it as what the successor waits for", async () => {
const host = await slowHost(Promise.resolve());
const readiness = readinessOf(host, [{ kind: "type", contract: Missing, params: {} }]);
readiness.start();
await readiness.ready(Date.now() + 10_000);
expect(readiness.cause).toMatchObject({
kind: "waiting",
error: { code: "unavailable", message: "waiting for app.missing: app.missing is not served here" },
});
readiness.release();
await host.close(context);
});
it("waits, up to its bound, for a request another host refuses for its version: that host's reload heals it", async () => {
const Shown = remote("app.shown", {
version: 1,
params: {},
hostAddress: () => Server(),
variants: { main: { state: { text: str() } } },
});
const ShownV2 = evolve(Shown, (e) => e.addState("main", "n", int()));
const serve = (h: FacetHost<TestPrincipal>, raised: boolean) =>
raised
? h.serve(ShownV2, () => ({ variant: "main", members: { text: constant("v2"), n: constant(2) } }))
: h.serve(Shown, () => ({ variant: "main", members: { text: constant("v1") } }));
const { server, client } = await linked((h) => serve(h, false));
const readiness = readinessOf(client as ChordHost, [{ kind: "type", contract: ShownV2, params: {} }]);
readiness.start();
const ready = readiness.ready(Date.now() + 10_000);
expect(await settled(ready)).toBe(false);
expect(readiness.cause).toBeUndefined();
await server.reload([{ name: "@test/app", facet: defineFacet<TestPrincipal>((h) => serve(h, true)) }], context);
await ready;
readiness.release();
await client.close(context);
await server.close(context);
});
it("does not wait for registry requests: the old attempt's contributions keep them until its seal", async () => {
const host = await slowHost(new Promise(() => {}));
const readiness = readinessOf(host, [{ kind: "registry", registry: Tags }]);
readiness.start();
await readiness.ready(Date.now() + 10_000);
expect(readiness.cause).toBeUndefined();
readiness.release();
await host.close(context);
});
});
describe("Readiness: its failure follows the requests", () => {
it("shows a request's failure, and none once the request recovered", async () => {
const Flaky = local("app.flaky", { version: 1, params: { id: int() }, members: { n: int() } });
let failing = true;
const host = createHost({ onError: () => {} }) as ChordHost;
const facet = defineFacet((h) =>
h.serve(Flaky, (_, self) => {
self.open(
async () => {
if (failing) throw new Error("not yet");
return null;
},
() => {},
);
return { n: constant(1) };
}),
);
await host.load([{ name: "@test/flaky", facet }], context);
const held = host.hold(Flaky, { id: 1 }, context);
await held.state.watch(context);
await new Promise((resolve) => setTimeout(resolve, 10));
expect(held.state.value.kind).toBe("failed");
const shown: (string | undefined)[] = [];
const readiness = readinessOf(host, [{ kind: "type", contract: Flaky, params: { id: 1 } }], shown);
readiness.start();
expect(shown).toEqual([expect.stringContaining("not yet")]);
failing = false;
held.retry();
await new Promise((resolve) => setTimeout(resolve, 10));
expect(shown.at(-1)).toBeUndefined();
expect(readiness.cause).toBeUndefined();
readiness.release();
held.release();
await host.close(context);
});
});
describe("Readiness: presence", () => {
it("keeps its bindings out of presence lists: they are no consumers of their own", async () => {
const host = await slowHost(Promise.resolve());
const listed = host.references(Slow);
const readiness = readinessOf(host, [slowRequest]);
readiness.start();
await readiness.ready(Date.now() + 10_000);
expect(listed.value).toEqual([]);
const held = host.hold(Slow, { id: 1 }, context);
await flushed();
expect(listed.value).toHaveLength(1);
readiness.release();
held.release();
await host.close(context);
});
});
describe("Readiness.release: the successor's own requests bound", () => {
it("lets go of its bindings: an object held only by them ends", async () => {
const closed: string[] = [];
const host = await slowHost(Promise.resolve(), closed);
const readiness = readinessOf(host, [slowRequest]);
readiness.start();
await readiness.ready(Date.now() + 10_000);
expect(closed).toEqual([]);
readiness.release();
await new Promise((resolve) => setTimeout(resolve, 10));
expect(closed).toEqual(["end"]);
await host.close(context);
});
it("ends a wait still running: ready resolves at once", async () => {
const host = await slowHost(new Promise(() => {}));
const readiness = readinessOf(host, [slowRequest]);
readiness.start();
const ready = readiness.ready(Date.now() + 10_000);
readiness.release();
expect(await settled(ready)).toBe(true);
await host.close(context);
});
});
describe("Readiness: memory", () => {
it("is collected with its bindings once the restart's successor is live", async () => {
const Dependent = local("app.dependent", { version: 1, params: {}, members: { n: int() } });
const host = createHost({ onError: () => {} });
const facet = (n: number) =>
defineFacet((h) => {
h.serve(Slow, () => ({ n: constant(1) }));
h.serve(Dependent, (_, self) => ({ n: self.derive([self.get(Slow, { id: 1 }).n], ([m]) => m + n) }));
});
await host.load([{ name: "@test/app", facet: facet(1) }], context);
const dependent = await host.get(Dependent, {}, context);
const from = started.length;
await host.reload([{ name: "@test/app", facet: facet(2) }], context);
await flushed();
expect(dependent.n.value).toBe(3);
expect(started.length).toBeGreaterThan(from);
await expectCollected(...started.slice(from));
await host.close(context);
});
});

2
packages/chord/test/host/reference-keys.test.ts
View file

@ -677,7 +677,7 @@ describe("keys across a restart and a failed re-pin (Astra's coverage note)", ()
persistence.read = read;
throw new Error("disk");
};
// A swap in place: the new instance re-exposes the cold provider's notes; the key's pin there fails to load.
// A restart: the new instance re-exposes the cold provider's notes; the key's pin there fails to load.
await host.reload(
[
{ name: "@test/providers", facet: providers },

225
packages/chord/test/host/reload-cycle.test.ts Normal file
View file

@ -0,0 +1,225 @@
// Tests of reload cycles (decision "Reload restarts every affected object; old work finishes on old code", rule 6): a
// wait of old work on an object whose activation waits on the old attempt that very work runs on fails at once with
// `unavailable/reload_cycle`, naming the chain, whether the cycle is closed when the wait starts or closes later; and
// that neither the waits nor an opener's gate waits keep anything once they ended.
import { describe, expect, it, vi } from "vitest";
import { BACKGROUND_CONTEXT as context } from "../../src/context/index.ts";
import { int, local, method, str } from "../../src/contracts/index.ts";
import type { ChordError } from "../../src/data/index.ts";
import { createHost, defineFacet, type Self } from "../../src/host/index.ts";
import { ChordObject } from "../../src/host/object.ts";
import { constant } from "../../src/state/index.ts";
import { expectCollected, flushed, instancesOf, rejection } from "../support.ts";
const Provider = local("app.p", {
version: 1,
params: {},
members: { work: method(undefined, str()), x: method(undefined, str()) },
});
const Fresh = local("app.f", { version: 1, params: { id: str() }, members: { ping: method(undefined, str()) } });
describe("a wait of old work that cycles through its own drain", () => {
it("fails at once when old work creates an object whose new code needs the provider's raised contract", async () => {
const Raised = local("app.p", {
version: 2,
params: {},
members: { work: method(undefined, str()), x: method(undefined, str()), n: int() },
});
const gate = Promise.withResolvers<void>();
const facet = (code: "v1" | "v2") =>
defineFacet((h) => {
if (code === "v1")
h.serve(Provider, () => ({
work: async (ctx) => {
await gate.promise;
return (await ctx.get(Fresh, { id: "fresh" })).ping(ctx);
},
x: async () => "v1 x",
}));
else h.serve(Raised, () => ({ n: constant(0), work: async () => "v2", x: async () => "v2 x" }));
h.serve(Fresh, (_, self) => {
// v2 needs the raised provider: its new attempt, which waits for this very work's drain.
if (code === "v1") self.get(Provider, {});
else self.get(Raised, {});
return { ping: async () => `${code} f` };
});
});
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: facet("v1") }], context);
const provider = await host.get(Provider, {}, context);
const working = rejection(provider.work(context));
await flushed();
await host.reload([{ name: "@test/app", facet: facet("v2") }], context);
const opened = performance.now();
gate.resolve();
const error = (await working) as ChordError;
expect(error).toMatchObject({ code: "unavailable", reason: "reload_cycle" });
expect(error.message).toBe(
"app.f(fresh): work its reload drains waits for it, and it waits for that drain (app.f(fresh) → app.p)",
);
// The reload drain bound (5 s) never came into it.
expect(performance.now() - opened).toBeLessThan(1_000);
await host.close(context);
});
it("fails once a later wait closes the cycle: the new attempt's opener calls the provider old work waits on", async () => {
const Dependent = local("app.d", { version: 1, params: {}, members: { ping: method(undefined, str()) } });
const gate = Promise.withResolvers<void>();
const facet = (code: "v1" | "v2") =>
defineFacet((h) => {
h.serve(Provider, () => ({
work: async (ctx) => {
await gate.promise;
return (await ctx.get(Dependent, {})).ping(ctx);
},
x: async () => `${code} x`,
}));
h.serve(Dependent, (_, self) => {
const provider = self.get(Provider, {});
// The new code's opener calls the provider: new work, which waits at its gate for its new attempt.
if (code === "v2")
self.open(
async (opening) => provider.x(opening),
() => {},
);
return { ping: async () => `${code} ping` };
});
});
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: facet("v1") }], context);
const provider = await host.get(Provider, {}, context);
await host.get(Dependent, {}, context);
const working = rejection(provider.work(context));
await flushed();
await host.reload([{ name: "@test/app", facet: facet("v2") }], context);
const opened = performance.now();
gate.resolve();
const error = (await working) as ChordError;
expect(error).toMatchObject({ code: "unavailable", reason: "reload_cycle" });
expect(error.message).toContain("(app.d → app.p)");
expect(performance.now() - opened).toBeLessThan(1_000);
await host.close(context);
});
it("does not fail a wait whose target waits on another object's drain only (no cycle through this work)", async () => {
const Dependent = local("app.d", { version: 1, params: {}, members: { ping: method(undefined, str()) } });
const gate = Promise.withResolvers<void>();
const facet = (code: "v1" | "v2") =>
defineFacet((h) => {
h.serve(Provider, () => ({
work: async (ctx) => {
await gate.promise;
return (await ctx.get(Dependent, {})).ping(ctx);
},
x: async () => `${code} x`,
}));
h.serve(Dependent, (_, self) => {
self.get(Provider, {});
return { ping: async () => `${code} ping` };
});
});
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: facet("v1") }], context);
const provider = await host.get(Provider, {}, context);
const working = provider.work(context);
await flushed();
await host.reload([{ name: "@test/app", facet: facet("v2") }], context);
gate.resolve();
// The dependent's new attempt binds the provider's open old attempt: the work's wait ends, never cut.
expect(await working).toBe("v2 ping");
await host.close(context);
});
});
describe("memory", () => {
const reloaded = instancesOf(ChordObject, "reload");
it("a cut wait keeps neither its target nor the old attempt it ran on", async () => {
const Raised = local("app.p", {
version: 2,
params: {},
members: { work: method(undefined, str()), x: method(undefined, str()), n: int() },
});
const gate = Promise.withResolvers<void>();
const selves: WeakRef<Self>[] = [];
const facet = (code: "v1" | "v2") =>
defineFacet((h) => {
if (code === "v1")
h.serve(Provider, (_, self) => {
selves.push(new WeakRef(self));
return {
work: async (ctx) => {
await gate.promise;
return (await ctx.get(Fresh, { id: "fresh" })).ping(ctx);
},
x: async () => "v1 x",
};
});
else h.serve(Raised, () => ({ n: constant(0), work: async () => "v2", x: async () => "v2 x" }));
h.serve(Fresh, (_, self) => {
selves.push(new WeakRef(self));
self.get(code === "v1" ? Provider : Raised, {});
return { ping: async () => `${code} f` };
});
});
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: facet("v1") }], context);
const provider = await host.get(Provider, {}, context);
const working = rejection(provider.work(context));
await flushed();
await host.reload([{ name: "@test/app", facet: facet("v2") }], context);
gate.resolve();
expect(await working).toMatchObject({ reason: "reload_cycle" });
await flushed();
// The old provider and the fresh object's attempts, the wait's chain and target.
await expectCollected(...selves);
provider.release();
await host.close(context);
});
it("an opener's gate wait leaves no wait edge once it ended, nor keeps the old attempt", async () => {
const Dependent = local("app.d", { version: 1, params: {}, members: { ping: method(undefined, str()) } });
const gate = Promise.withResolvers<void>();
const selves: WeakRef<Self>[] = [];
const facet = (code: "v1" | "v2") =>
defineFacet((h) => {
h.serve(Provider, (_, self) => {
if (code === "v1") selves.push(new WeakRef(self));
return {
work: async () => {
await gate.promise;
return `${code} work`;
},
x: async () => `${code} x`,
};
});
h.serve(Dependent, (_, self) => {
const provider = self.get(Provider, {});
// The new code's opener calls the provider: it waits at its gate while the provider restarts.
if (code === "v2")
self.open(
async (opening) => provider.x(opening),
() => {},
);
return { ping: async () => `${code} ping` };
});
});
const host = createHost({ onError: () => {} });
await host.load([{ name: "@test/app", facet: facet("v1") }], context);
const provider = await host.get(Provider, {}, context);
const dependent = await host.get(Dependent, {}, context);
const working = provider.work(context);
await flushed();
await host.reload([{ name: "@test/app", facet: facet("v2") }], context);
gate.resolve();
expect(await working).toBe("v1 work");
await vi.waitFor(async () => expect(await dependent.ping(context)).toBe("v2 ping"));
const object = reloaded.map((ref) => ref.deref()).find((each) => each?.label === "app.d");
expect(object?.waitEdges()).toEqual([]);
await expectCollected(...selves);
provider.release();
dependent.release();
await host.close(context);
});
});

18
packages/chord/test/host/staging.test.ts
View file

@ -17,6 +17,7 @@ import {
import { defineJob } from "../../src/jobs/index.ts";
import { defineRegistry } from "../../src/registries/index.ts";
import { constant } from "../../src/state/index.ts";
import { tableSqlName } from "../../src/store/sql.ts";
import { defineTable } from "../../src/store/table.ts";
import { evolveTable } from "../../src/store/table-evolve.ts";
import { flushed, memoryStore, Server } from "../support.ts";
@ -133,8 +134,10 @@ describe("staging: recorded declarations decide the fate", () => {
self.state(defineState("fresh", { schema: obj({ n: int() }), version: 1, initial: () => ({ n: 0 }) }));
});
const report = await host.reload([{ name: "@test/app", facet }], context);
// It opens natives now: a restart, whose activation opened its native once.
expect(report.objects).toEqual([{ object: "app.store", kind: "restarted" }]);
// Its readiness binds the recorded request: `Dep` is missing, so the restart is reported parked.
expect(report.objects).toEqual([
{ object: "app.store", kind: "parked", cause: expect.objectContaining({ kind: "waiting" }) },
]);
await flushed();
// Neither staging nor the restart opened it: the new attempt parks on the missing `Dep` first.
expect(opened).toEqual([]);
@ -142,7 +145,7 @@ describe("staging: recorded declarations decide the fate", () => {
await host.close(context);
});
it("a table of the new factory at a newer version than the store's restarts the object (its eager migration)", async () => {
it("a table of the new factory at a newer version migrates at the restart's new attempt", async () => {
const store = await memoryStore();
const V1 = defineTable("items", { version: 1, columns: { id: str(), name: str() }, key: ["id"] });
const V2 = evolveTable(V1, (t) => t.rename("name", "title"));
@ -153,12 +156,15 @@ describe("staging: recorded declarations decide the fate", () => {
[{ name: "@test/app", facet: serveStore("v2", (self) => self.table(V1)) }],
context,
);
expect(same.objects).toEqual([{ object: "app.store", kind: "swapped" }]);
expect(same.objects).toEqual([{ object: "app.store", kind: "restarted" }]);
await flushed();
const newer = await host.reload(
[{ name: "@test/app", facet: serveStore("v3", (self) => self.table(V2)) }],
context,
);
expect(newer.objects).toEqual([{ object: "app.store", kind: "restarted" }]);
await flushed();
expect(store.table(tableSqlName("app.store", "items"))?.version).toBe(2);
await host.close(context);
await store.close(context);
});
@ -268,7 +274,7 @@ describe("staging per plugin (slice 7c §4)", () => {
);
expect(report.plugins).toEqual({ added: [], changed: ["@test/store"], removed: [] });
expect(failedOf(report)).toEqual([["@test/notes", 'plugin "@test/notes": setup threw']]);
expect(report.objects).toEqual([{ object: "app.store", kind: "swapped" }]);
expect(report.objects).toEqual([{ object: "app.store", kind: "restarted" }]);
await flushed();
expect([store.label.value, counted.count.value, counted.state.value]).toEqual(["v2", 1, { kind: "live" }]);
// Fixed later, it applies as any change does.
@ -481,7 +487,7 @@ describe("staging per plugin (slice 7c §4)", () => {
);
expect([report.plugins.changed, report.failed]).toEqual([["@test/notes", "@test/view"], []]);
expect(report.objects).toEqual([
{ object: "app.notes", kind: "swapped" },
{ object: "app.notes", kind: "restarted" },
expect.objectContaining({ object: "app.view", kind: "parked" }),
]);
await flushed();

26
packages/chord/test/jobs/records.test.ts
View file

@ -271,8 +271,8 @@ function runner(
);
}
describe("jobs across a reload (slice 7b §8)", () => {
it("an older run finishes on its old code across a swap; its finish runs on the new code, mapped forwards once", async () => {
describe('jobs across a reload (slice 7b §8; decision "Reload restarts every affected object")', () => {
it("an older in-memory run is cut at the restart's seal; its finish runs on the new code, mapped forwards once", async () => {
const finished: Finished[] = [];
const gate = Promise.withResolvers<void>();
const host = createHost({ onError: () => {} });
@ -282,17 +282,15 @@ describe("jobs across a reload (slice 7b §8)", () => {
await flushed();
await host.reload([{ name: "@test/app", facet: runner("v2", WorkV2, finished) }], context);
await flushed();
// Listed with its args mapped forwards.
expect(jobs.running.value).toMatchObject([{ version: 1, args: { text: "a", priority: 0 }, phase: "running" }]);
gate.resolve();
await flushed();
expect(finished).toEqual([
{
code: "v2",
outcome: expect.objectContaining({
kind: "done",
kind: "failed",
args: { text: "a", priority: 0 },
result: { text: "a by v1" },
error: { kind: "aborted", reason: "reload" },
}),
},
]);
@ -315,7 +313,10 @@ describe("jobs across a reload (slice 7b §8)", () => {
await host.reload([{ name: "@test/app", facet: runner("v1b", Work, finished) }], context);
await flushed();
expect(finished).toEqual([
{ code: "v1b", outcome: expect.objectContaining({ kind: "done", result: "a by v1" }) },
{
code: "v1b",
outcome: expect.objectContaining({ kind: "failed", error: { kind: "aborted", reason: "reload" } }),
},
]);
expect(jobs.running.value).toEqual([]);
await host.close(context);
@ -357,12 +358,17 @@ describe("jobs across a reload (slice 7b §8)", () => {
expect(finished).toEqual([]);
await host.reload([{ name: "@test/app", facet: runner("v3", Work, finished) }], context);
await flushed();
expect(finished).toEqual([{ code: "v3", outcome: expect.objectContaining({ kind: "done", result: "a by v1" }) }]);
expect(finished).toEqual([
{
code: "v3",
outcome: expect.objectContaining({ kind: "failed", error: { kind: "aborted", reason: "reload" } }),
},
]);
await host.close(context);
});
});
describe("a reservation released after a swap (final review)", () => {
describe("a reservation released during a restart (final review)", () => {
it("starts the old code's run with its own args, not the new version's mapping (Sol)", async () => {
const Owner = local("app.reserve-owner", { version: 1, params: {}, members: {} });
const V1 = defineJob("app.reserve", {
@ -401,7 +407,7 @@ describe("a reservation released after a swap (final review)", () => {
],
context,
);
// The run's reservation (the second commit) waits on `gate` until the swap to V2 happened.
// The run's reservation (the second commit) waits on `gate` until the reload to V2 committed.
const entered = Promise.withResolvers<void>();
const gate = Promise.withResolvers<void>();
const commit = store.commit.bind(store);

26
packages/chord/test/link/message.test.ts
View file

@ -201,6 +201,8 @@ const EVERY_KIND: Message[] = [
},
{ kind: "call", call: 7, hold: 0, method: 1, args: '{"text":"x"}', trace: { a: "b" } },
{ kind: "call", origin: 2, call: 8, hold: 3, method: 0, trace: {} },
{ kind: "call", call: 9, hold: 0, method: 1, args: '{"text":"x"}', old: true },
{ kind: "call", origin: 2, call: 10, hold: 3, method: 0, trace: { a: "b" }, old: true },
];
/** A carrier at the bounds: 8 keys, keys and values of 256 characters. */
@ -306,7 +308,7 @@ forEachCodec(() => {
const trailing = new Uint8Array([...release, ...(codec === binaryCodec ? [0] : [0x20, 0x31])]);
expect(() => codec.decode(new Uint8Array())).toThrow();
expect(() =>
codec.decode(codec === binaryCodec ? new Uint8Array([99]) : new TextEncoder().encode('["?"]')),
codec.decode(codec === binaryCodec ? new Uint8Array([31]) : new TextEncoder().encode('["?"]')),
).toThrow("unknown kind");
expect(() => codec.decode(trailing)).toThrow();
});
@ -496,6 +498,15 @@ describe("the binary codec", () => {
expect(plainCall[0]).toBe(3);
});
it("flags a call that is old work of a reload in the kind byte (0x20); another kind with that flag is unknown", () => {
const call = binaryCodec.encode({ kind: "call", call: 0, hold: 0, method: 0, old: true });
expect(call[0]).toBe(3 | 0x20);
expect(binaryCodec.decode(call)).toEqual({ kind: "call", call: 0, hold: 0, method: 0, old: true });
const release = binaryCodec.encode({ kind: "release", hold: 1 });
release[0]! |= 0x20;
expect(() => binaryCodec.decode(release)).toThrow("unknown kind");
});
it("refuses unknown hold flags", () => {
const hold = binaryCodec.encode({
kind: "hold",
@ -626,6 +637,19 @@ describe("the text codec", () => {
expect(text({ ...call, args: "[1]" })).toBe('["c",0,1,2,[0,[1]]]');
});
it("writes a call's old-work flag after its carrier (null when absent), its payload's fields positional then", () => {
const call = { kind: "call", call: 0, hold: 1, method: 2 } as const;
expect(text({ ...call, old: true })).toBe('["c",0,1,2,null,null,null,true]');
expect(text({ ...call, args: "[1]", trace: { a: "b" }, old: true })).toBe(
'["c",0,1,2,[0,[1]],null,{"a":"b"},true]',
);
});
it("leaves the old-work flag's value to the message schema: only true passes", () => {
const decoded = textCodec.decode(new TextEncoder().encode('["c",0,1,2,null,null,null,1]'));
expect(() => checkMessage(decoded, context)).toThrow("message schema");
});
it("puts byte segments after the tuple and a newline, their lengths the tuple's last field", () => {
const bytes = textCodec.encode({
kind: "result",

5
packages/chord/test/node/app/app.test.ts
View file

@ -11,7 +11,7 @@ import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts";
import { hostKind, local, remote, str, unknown } from "../../../src/contracts/index.ts";
import { addressParam, HostData } from "../../../src/node/app/data.ts";
import { AppInfo } from "../../../src/node/app/index.ts";
import { changesOf, rejection } from "../../support.ts";
import { changesOf, flushed, rejection } from "../../support.ts";
import { send } from "../http/support.ts";
import { gitRepository, writePlugin } from "../plugins/support.ts";
import { ECHO_SERVER, NOTES_PLUGIN, Notes, Project, tempDir, testApp, WORKER_ENTRY } from "./support.ts";
@ -186,6 +186,8 @@ describe("workers", () => {
const result = await app.applyChanges(context);
expect(result.stopped).toBeUndefined();
expect(result.steps.map(({ name }) => name)).toEqual(["server"]);
// Restarted: the new attempt goes live once the old one ended.
await flushed();
expect(echo.text.value).toBe("changed");
expect(app.plugins.state.value.dirty).toEqual([]);
await app.close(context);
@ -235,6 +237,7 @@ describe("applying changes one after another", () => {
await first;
await second;
expect(events).toEqual(["build 1", "rollout", "build 2", "rollout"]);
await flushed();
expect(echo.text.value).toBe("v3");
} finally {
echo.release();

18
packages/chord/test/node/fixtures/reload/app.ts
View file

@ -28,7 +28,7 @@ export type ReloadApp = Awaited<ReturnType<typeof testApp>>;
* The hosts' timeouts of the suite's apps (the server's and the workers'): a call outliving a restart's drain is cut
* off after 2 s, a call waits at most 5 s for its object to go live.
*/
export const TIMEOUTS = { closeTimeout: 2000, readyTimeout: 5000 } as const;
export const TIMEOUTS = { closeTimeout: 2000, readyTimeout: 5000, reloadDrainTimeout: 1000 } as const;
/**
* Starts the suite's app with `plugins` (directory name to files; each plugin is `@test/<dir>`) and project workers;
@ -203,7 +203,7 @@ export class ReloadPage {
/**
* Signs in with a new owner browser session (the owner link is single use: a second page needs its own session)
* and waits as `signIn` does.
* and waits as `signIn` does, and until each view (each holding the counter) registered its reference.
*/
async signInAgain(app: ReloadApp, views: readonly string[]): Promise<void> {
const sessions = await app.app.host.get(Sessions, {}, context);
@ -218,7 +218,19 @@ export class ReloadPage {
await this.browser.call("Network.setCookie", { name: pair!.slice(0, at), value: pair!.slice(at + 1), url });
await this.browser.open(url);
await this.browser.wait('globalThis.page?.connection === "connected"');
for (const view of views) await this.browser.wait(`document.querySelector('[data-view="${view}"]')`);
await this.#viewsShown(views);
}
/**
* Waits until each view is on the page and has registered its reference: a component tracks it in an effect one
* render after `useHold` returned it, so a call made right after the view appeared could find none (a test calling
* at once needs this).
*/
async #viewsShown(views: readonly string[]): Promise<void> {
for (const view of views) {
await this.browser.wait(`document.querySelector('[data-view="${view}"]')`);
await this.browser.wait(`globalThis.reload.refs[${JSON.stringify(view)}] !== undefined`);
}
}
/** The text of `selector` within view `view`, `null` when absent. */

21
packages/chord/test/node/plugins/plugins.test.ts
View file

@ -20,7 +20,16 @@ import {
type RolloutStep,
} from "../../../src/node/plugins/index.ts";
import { constant } from "../../../src/state/index.ts";
import { changesOf, delivered, expectCollected, rejection, settledHeap, transported, valuesOf } from "../../support.ts";
import {
changesOf,
delivered,
expectCollected,
flushed,
rejection,
settledHeap,
transported,
valuesOf,
} from "../../support.ts";
import { gitRepository, HOST_PACKAGES, kindsOf, tempRoot, writeFiles, writePackage, writePlugin } from "./support.ts";
/** The fixtures' contracts, as the `contracts` fixture declares them. */
@ -127,7 +136,7 @@ describe("per plugin at reload", () => {
expect(built.facets.map(({ plugin, facet }) => [plugin, facet])).toEqual([["@test/b", "worker"]]);
const report = await plugins.reload(host, built, context);
expect(report.plugins).toEqual({ added: [], changed: ["@test/b"], removed: [] });
expect(report.objects).toEqual([{ object: "test.b", kind: "swapped" }]);
expect(report.objects).toEqual([{ object: "test.b", kind: "restarted" }]);
expect([await label("a"), await label("b"), await label("c")]).toEqual(["a1", "b2", "c1"]);
// Built again unchanged: the same digest, nothing reloaded.
const again = await plugins.reload(host, await plugins.build(["@test/b"], context), context);
@ -597,8 +606,8 @@ describe("end to end: a server and a worker", () => {
expect(
rolled.steps.map(({ name, hosts }) => [name, reportOf(hosts).plugins.changed, reportOf(hosts).objects]),
).toEqual([
["server", ["@test/notes"], [{ object: "test.index", kind: "swapped" }]],
["worker", ["@test/notes"], [{ object: "test.notes", kind: "swapped" }]],
["server", ["@test/notes"], [{ object: "test.index", kind: "restarted" }]],
["worker", ["@test/notes"], [{ object: "test.notes", kind: "restarted" }]],
]);
await transported();
await delivered();
@ -689,9 +698,12 @@ describe("fix round 1", () => {
details: { committed: { added: [], changed: ["@test/a"], removed: [], failed: [] } },
});
hook.mockRestore();
// Restarted: the new attempt goes live once the old one ended.
await flushed();
expect(item.label.value).toBe("a2");
await itemPlugin(root, "b", "b2");
await plugins.reload(host, await plugins.build(["@test/b"], context), context);
await flushed();
expect(item.label.value).toBe("a2");
await host.close(context);
plugins.close();
@ -731,6 +743,7 @@ describe("fix round 1", () => {
await vi.waitFor(() => expect(plugins.state.value).toMatchObject({ dirty: ["@test/a"] }), { timeout: 5_000 });
const built = await plugins.build(["@test/a"], context);
expect((await plugins.reload(host, structuredClone(built), context)).plugins.changed).toEqual(["@test/a"]);
await flushed();
expect(item.label.value).toBe("a2");
// The marks clear only through the build object itself (a copy carries no snapshot).
const result = await plugins.rollout([plugins.step("worker", host)], structuredClone(built), context);

3
packages/chord/test/node/reload/contracts.test.ts
View file

@ -64,7 +64,8 @@ describe.skipIf(CHROMIUM === undefined)("a reload changing a contract", () => {
expect(await page.result(await page.call("alpha", "add", 1))).toEqual(ok("v2:3"));
await page.waitText("alpha", ".label", "v2:3");
expect(await page.result(await page.call("alpha", "code"))).toEqual(ok("v2"));
expect((await page.states("alpha")).slice(settled)).toEqual([]);
// The counter restarted: `loading` once its old attempt ended, then the new code's fresh snapshot.
expect((await page.states("alpha")).slice(settled)).toEqual(["loading", "live"]);
expect(await page.text("alpha", ".count")).toBe("3");
expect(await page.result(await page.call("alpha", "visits"))).toEqual(ok(2));
expect(await page.value("alpha", ".draft")).toBe("half-typed");

16
packages/chord/test/node/reload/jobs.test.ts
View file

@ -1,7 +1,7 @@
// Reload suite, a case the table leaves out (the audit's "jobs running across a reload"; design §15 "Versions"): a job
// run started from the page, running on a worker while its object reloads. A pure swap: the run finishes on its old
// code and `finish` runs on the new code. A native restart: the in-memory run ends `aborted { reload }` and `finish`
// runs on the new code. The page watches the finished runs' entries.
// run started from the page, running on a worker while its object restarts (with or without natives: every reload
// restarts). The in-memory run ends `aborted { reload }` at the old attempt's seal and `finish` runs on the new code.
// The page watches the finished runs' entries.
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts";
@ -30,11 +30,13 @@ describe.skipIf(CHROMIUM === undefined)("a job running across a reload", () => {
});
it.each([
["a pure swap", false, "swapped", "v2-finish(done:run-v1:a)"],
["a native restart", true, "restarted", "v2-finish(aborted:reload)"],
["pure code", false],
["code holding natives", true],
] as const)(
"across %s",
async (_change, native, fate, entry) => {
"across a restart of %s",
async (_change, native) => {
const fate = "restarted";
const entry = "v2-finish(aborted:reload)";
const app = await startReloadApp({ latch: LATCH_PLUGIN, tasks: tasksPlugin("v1", native) });
const hand = await latches(app, "worker");
try {

49
packages/chord/test/node/reload/pure.test.ts
View file

@ -1,8 +1,8 @@
// Reload suite, row "pure code (no natives) on the server or a worker" (design §21 "What a user sees during a reload"):
// the view stays bound, one fresh snapshot once the new code is live, values change only where the new code computes
// differently; a running call finishes on the old code, a call arriving meanwhile waits and runs on the new code, never
// `not_ready`; documents and tables unchanged. Through a real page, for a counter on a project's worker and on the
// server.
// a restart, as for code holding natives. While the old code drains its running call the view stays `live` with its
// last values; a call arriving meanwhile waits at the gate and runs on the new code, never `not_ready`; then `loading`
// briefly and a fresh snapshot from the new code; documents and tables unchanged. Through a real page, for a counter
// on a project's worker and on the server.
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { BACKGROUND_CONTEXT as context } from "../../../src/context/index.ts";
@ -32,7 +32,7 @@ describe.skipIf(CHROMIUM === undefined)("a reload of pure code", () => {
});
describe.each<Placement>(["worker", "server"])("on the %s", (placement) => {
it("keeps the view bound, finishes a running call on the old code and runs new calls on the new code", async () => {
it("stays live while a running call finishes on the old code, then shows the new code's snapshot", async () => {
const app = await startReloadApp({
latch: LATCH_PLUGIN,
counter: counterPlugin({ code: "v1", placement }),
@ -65,26 +65,32 @@ describe.skipIf(CHROMIUM === undefined)("a reload of pure code", () => {
expect(rolloutFailures(result)).toEqual([]);
const step = placement === "worker" ? "project/project|p1" : "server/server";
expect(rolloutFates(result)).toEqual([
`${step}: reload.store swapped`,
`${step}: reload.counter(p1) swapped`,
`${step}: reload.store restarted`,
`${step}: reload.counter(p1) restarted`,
]);
// Arriving after the commit, while the old call still runs: the new code answers.
expect(await page.result(await page.call("alpha", "add", 1))).toEqual(ok("v2:3"));
expect(await page.peek(running)).toEqual({ status: "pending" });
// Arriving after the commit, while the old call still runs: it waits at the gate; the view stays live
// with the last values.
const arriving = await page.call("alpha", "add", 1);
await new Promise((resolve) => setTimeout(resolve, 300));
expect(await page.peek(arriving)).toEqual({ status: "pending" });
expect((await page.states("alpha")).slice(settled)).toEqual([]);
expect(await page.text("alpha", ".label")).toBe("v1:2");
await hand.open("running");
expect(await page.result(running)).toEqual(ok("v1:running"));
expect(await page.result(arriving)).toEqual(ok("v2:3"));
for (const index of during) {
const answer = await page.result(index);
expect([ok("v1"), ok("v2")]).toContainEqual(answer);
}
await page.waitText("alpha", ".label", "v2:3");
// Bound throughout: the reference never left `live`, its label went from the old value to the new.
expect((await page.states("alpha")).slice(settled - 1)).toEqual(["live"]);
await page.waitText("alpha", ".state", "live");
// `loading` once the old attempt ended, then the new code's fresh snapshot; never a failure.
expect((await page.states("alpha")).slice(settled)).toEqual(["loading", "live"]);
const frames = await page.frames("alpha");
expect(frames.filter((frame) => typeof frame === "string" && frame.startsWith("failed"))).toEqual([]);
expect(frames.slice(frames.indexOf("v1:2"))).toEqual(["v1:2", "v2:2", "v2:3"]);
expect(frames.at(-1)).toBe("v2:3");
// Documents and tables unchanged: the count went on from 2, the table kept the old code's rows.
expect(await page.text("alpha", ".count")).toBe("3");
expect(await page.result(await page.call("alpha", "visits"))).toEqual(ok(2));
@ -97,12 +103,9 @@ describe.skipIf(CHROMIUM === undefined)("a reload of pure code", () => {
}, 120_000);
});
// The audit's F1: every `self.get` of a type on another host opens a new hold, which starts `loading`, so the new
// attempt of a pure object over a worker object parks instead of swapping in place.
// Expected to fail until the open design point is settled: "a parked swap keeps the old instance serving" (under review).
it.fails(
"keeps the view of a pure server object over a worker object bound when only the server object reloads",
async () => {
// The audit's F1, as decided (decision "Reload restarts every affected object"): the server object restarts; its
// successor's hold of the worker object starts `loading`, so the view shows `loading` once, then the new code.
it("shows loading once, then the new code, for a pure server object over a worker object when only it reloads", async () => {
const app = await startReloadApp({
latch: LATCH_PLUGIN,
counter: counterPlugin({ code: "v1", placement: "worker" }),
@ -122,12 +125,10 @@ describe.skipIf(CHROMIUM === undefined)("a reload of pure code", () => {
expect(rolloutFailures(result)).toEqual([]);
await page.waitText("alpha", ".label", "s2(0)");
expect(await page.result(await page.call("alpha", "code"))).toEqual(ok("s2"));
expect((await page.states("alpha")).slice(settled)).toEqual([]);
expect(rolloutFates(result)).toEqual(["server/server: reload.summary(p1) swapped"]);
expect((await page.states("alpha")).slice(settled)).toEqual(["loading", "live"]);
expect(rolloutFates(result)).toEqual(["server/server: reload.summary(p1) restarted"]);
} finally {
await app.app.close(context);
}
},
120_000,
);
}, 120_000);
});

8
packages/chord/test/node/reload/restart.test.ts
View file

@ -55,7 +55,7 @@ describe.skipIf(CHROMIUM === undefined)("a reload restarting an object with nati
expect(rolloutFailures(result)).toEqual([]);
const step = placement === "worker" ? "project/project|p1" : "server/server";
expect(rolloutFates(result)).toEqual([
`${step}: reload.store swapped`,
`${step}: reload.store restarted`,
`${step}: reload.counter(p1) restarted`,
]);
@ -96,7 +96,7 @@ describe.skipIf(CHROMIUM === undefined)("a reload restarting an object with nati
});
it.each<Placement>(["worker", "server"])(
"cuts a running call off after closeTimeout on the %s, then runs the waiting call on the new code",
"cuts a running call off at the reload drain bound on the %s, then runs the waiting call on the new code",
async (placement) => {
const app = await startReloadApp({
latch: LATCH_PLUGIN,
@ -113,7 +113,9 @@ describe.skipIf(CHROMIUM === undefined)("a reload restarting an object with nati
await editAndApply(app, "counter", counterPlugin({ code: "v2", placement, native: true }));
const arriving = await page.call("alpha", "code");
const cut = await page.result(stuck);
expect(Date.now() - started).toBeGreaterThanOrEqual(TIMEOUTS.closeTimeout - 100);
// The kit passes `reloadDrainTimeout` to the server and the workers: the cut comes before `closeTimeout`.
expect(Date.now() - started).toBeGreaterThanOrEqual(TIMEOUTS.reloadDrainTimeout - 100);
expect(Date.now() - started).toBeLessThan(TIMEOUTS.closeTimeout);
expect(cut).toMatchObject({ status: "error", code: "released" });
expect(await page.result(arriving)).toEqual(ok("v2"));
await page.waitText("alpha", ".state", "live");

12
packages/chord/test/node/reload/rollout.test.ts
View file

@ -59,8 +59,8 @@ describe.skipIf(CHROMIUM === undefined)("a rollout across hosts", () => {
["web", ["web"]],
]);
expect(rolloutFates(result)).toEqual([
"project/project|p1: reload.store swapped",
"project/project|p1: reload.counter(p1) swapped",
"project/project|p1: reload.store restarted",
"project/project|p1: reload.counter(p1) restarted",
]);
// Project p2 had no worker during the rollout: its first hold starts one on the current set.
await page.hold("p2", "alpha", 10_000, "p2");
@ -110,7 +110,10 @@ describe.skipIf(CHROMIUM === undefined)("a rollout across hosts", () => {
expect(rolloutFailures(await second)).toEqual([]);
await page.waitText("alpha", ".label", "v3:2");
expect(await page.result(await page.call("alpha", "add", 1))).toEqual(ok("v3:3"));
expect((await page.states("alpha")).slice(settled)).toEqual([]);
// Each restart shows `loading` once its old attempt ended, never a failure.
const states = (await page.states("alpha")).slice(settled);
expect(states.filter((state) => state.startsWith("failed"))).toEqual([]);
expect(states.at(-1)).toBe("live");
expect(app.app.plugins.state.value.dirty).toEqual([]);
} finally {
await app.app.close(context);
@ -148,7 +151,8 @@ describe.skipIf(CHROMIUM === undefined)("a rollout across hosts", () => {
// A case the table leaves out (the audit's "a contract change between hosts, consumer first"): the counter's
// contract moves to v2 without `accepts`, and the server's summary, which holds it, changes with it (it copies the
// contract file). The server step runs first: the summary's new request meets the old worker serving v1.
// Expected to fail until the open design point is settled: "a parked swap keeps the old instance serving" (under review).
// The documented gap (decision "The rollout order stays; no `loading` grace", design §21 "Rollout across hosts"): a
// consumer on an earlier step waits for its provider's later step; the recipe is two applies, the provider first.
it.fails(
"keeps the view of a server consumer bound while its worker provider moves to a new version with it",
async () => {

2
packages/chord/test/node/workers/control.test.ts
View file

@ -28,7 +28,7 @@ describe("errors", () => {
plugins: { added: ["@a/x"], changed: [], removed: [] },
failed: [{ plugin: "@a/y", error: new Error("broken") }],
objects: [
{ object: "a.one", kind: "swapped" },
{ object: "a.one", kind: "restarted" },
{
object: "a.two",
kind: "parked",

145
packages/chord/test/remote/consumer.test.ts
View file

@ -1,13 +1,14 @@
import { describe, expect, it, vi } from "vitest";
import { BACKGROUND_CONTEXT as context, withCancel } from "../../src/context/index.ts";
import { int, local, map, method, remote, str } from "../../src/contracts/index.ts";
import { evolve, int, local, map, method, remote, str } from "../../src/contracts/index.ts";
import { PluginError } from "../../src/data/index.ts";
import { createHost, defineFacet, type HostError } from "../../src/host/index.ts";
import { type Ctx, createHost, defineFacet, type FacetHost, type HostError } from "../../src/host/index.ts";
import { resolveWatchable } from "../../src/host/reference.ts";
import { memoryLinkPair } from "../../src/link/index.ts";
import { DeflatedPart, jsonPart, type Part, wireError } from "../../src/link/internal.ts";
import { nodeCompression } from "../../src/node/compression.ts";
import { RemoteObject } from "../../src/remote/remote-object.ts";
import { constant } from "../../src/state/index.ts";
import {
addNote,
bindNext,
@ -19,13 +20,17 @@ import {
linked,
metered,
Notes,
PlacedNotes,
RawPeer,
rawParent,
rawProvider,
rejection,
relayed,
SERVER_ADDRESS,
Server,
Session,
serveNotes,
type TestPrincipal,
testCodec,
transported,
} from "../support.ts";
@ -720,3 +725,139 @@ forEachCodec(() => {
});
});
});
describe('old work across hosts (decision "Reload restarts every affected object", `call.old`)', () => {
type Side = "server" | "worker";
const at = { server: () => Server(), worker: () => Session({ id: "S1" }) } as const;
/**
* A provider on `side` whose `work` (after its gate) calls a dependent on the other host, which holds the provider
* and calls its `x` back over the link: both reload at once while `work` runs. `work` runs as the worker's principal:
* the server's link to the worker is trusted, so the dependent's hold of the provider names it either way, and the
* back-call reaches the very object running `work`. When `raised`, the reload raises the provider's version (no
* `accepts`) and the dependent's new code needs it; `dependentFirst` reloads the dependent's host first and the
* provider's 50 ms later.
*/
async function backCall(side: Side, options: { readonly raised?: boolean; readonly dependentFirst?: boolean } = {}) {
const { raised = false, dependentFirst = false } = options;
const other: Side = side === "server" ? "worker" : "server";
const Provider = remote(`app.provider-${side}`, {
version: 1,
params: {},
hostAddress: at[side],
variants: { v: { state: {}, methods: { work: method(undefined, str()), x: method(undefined, str()) } } },
});
const ProviderV2 = evolve(Provider, (e) => e.addState("v", "more", str()));
const Dependent = remote(`app.dependent-${other}`, {
version: 1,
params: {},
hostAddress: at[other],
variants: { v: { state: {}, methods: { ping: method(undefined, str()) } } },
});
const gates: Promise<void>[] = [];
const serve = (h: FacetHost<TestPrincipal>, onSide: Side, code: string) => {
const P = raised && code === "v2" ? ProviderV2 : Provider;
const more = P === ProviderV2 ? { more: constant("m") } : {};
if (onSide === side)
h.serve(P as typeof Provider, () => ({
variant: "v",
members: {
...more,
work: async (ctx: Ctx) => {
await gates.shift();
return `${code} work -> ${await (await ctx.get(Dependent, {})).ping(ctx)}`;
},
x: async () => `${code} x`,
},
}));
else
h.serve(Dependent, (_, _principal, self) => {
const provider = self.get(P as typeof Provider, {});
return {
variant: "v",
members: { ping: async (ctx: Ctx) => `${code} ping -> ${await provider.x(ctx)}` },
};
});
};
const workerFacet = (code: string) =>
defineFacet<TestPrincipal>((h) => {
serveNotes(h, PlacedNotes);
serve(h, "worker", code);
});
const { tab, server, workers } = await relayed({
trusted: true,
server: (h) => serve(h, "server", "v1"),
worker: (h) => serve(h, "worker", "v1"),
});
// The worker exists once a hold reached it.
await tab.get(PlacedNotes, { session: "S1" }, context);
const worker = workers.get("session|S1")!;
const host = side === "server" ? server : worker;
const principal: TestPrincipal = { kind: "worker", session: "S1" };
const work = () => host.root(async (ctx) => (await ctx.get(Provider, {})).work(ctx), context, { principal });
// A root keeps the dependent referenced (else it ends after each run, and the work would create it afresh).
const held = Promise.withResolvers<void>();
const kept = host.root(
async (ctx) => {
await ctx.get(Dependent, {});
await held.promise;
},
context,
{ principal },
);
expect(await work()).toBe("v1 work -> v1 ping -> v1 x");
const gate = Promise.withResolvers<void>();
gates.push(gate.promise);
const running = work();
await transported();
const reloadServer = () =>
server.reload(
[{ name: "@test/server", facet: defineFacet<TestPrincipal>((h) => serve(h, "server", "v2")) }],
context,
);
const reloadWorker = () => worker.reload([{ name: "@test/worker", facet: workerFacet("v2") }], context);
const [first, second] = side === "server" ? [reloadWorker, reloadServer] : [reloadServer, reloadWorker];
const reports = dependentFirst
? [await first(), await new Promise((resolve) => setTimeout(resolve, 50)).then(second)]
: await Promise.all([first(), second()]);
expect(reports.flatMap(({ failed }) => failed)).toEqual([]);
const opened = performance.now();
gate.resolve();
const answer = await running;
const took = performance.now() - opened;
held.resolve();
await kept;
await server.close(context);
return { answer, took };
}
it("a back-call from the dependent on the worker runs on the server provider's open old attempt", async () => {
const { answer, took } = await backCall("server");
// Without the flag, `x` would wait for the provider's new attempt, which waits for this very call's drain.
expect(answer).toBe("v1 work -> v2 ping -> v1 x");
expect(took).toBeLessThan(1_000);
});
it("a back-call from the dependent on the server runs on the worker provider's open old attempt", async () => {
const { answer, took } = await backCall("worker");
expect(answer).toBe("v1 work -> v2 ping -> v1 x");
expect(took).toBeLessThan(1_000);
});
it("the provider's version raised, which the dependent's new code needs: the work runs on the dependent's old attempt", async () => {
for (const side of ["server", "worker"] as const) {
const { answer, took } = await backCall(side, { raised: true });
// The dependent's successor waits for the provider's: its old attempt stays open for the work.
expect(answer).toBe("v1 work -> v1 ping -> v1 x");
expect(took).toBeLessThan(1_000);
}
});
it("the same with the dependent's host reloaded first: its readiness waits out the other host's version refusal", async () => {
for (const side of ["server", "worker"] as const) {
const { answer, took } = await backCall(side, { raised: true, dependentFirst: true });
expect(answer).toBe("v1 work -> v1 ping -> v1 x");
expect(took).toBeLessThan(1_000);
}
});
});

268
packages/chord/test/remote/hold.test.ts
View file

@ -8,6 +8,7 @@ import { defineState } from "../../src/documents/index.ts";
import { type Ctx, createHost, defineFacet, type FacetHost, type Host, type Self } from "../../src/host/index.ts";
import type { ChordHost } from "../../src/host/internal.ts";
import { ChordObject } from "../../src/host/object.ts";
import { defineJob } from "../../src/jobs/index.ts";
import { type Link, memoryLinkPair } from "../../src/link/index.ts";
import { binaryCodec, jsonPart, type Message } from "../../src/link/internal.ts";
import { Hold } from "../../src/remote/hold.ts";
@ -20,10 +21,8 @@ import {
fakeTimers,
flushed,
forEachCodec,
holdNextCommit,
instancesOf,
linked,
memoryStore,
PlacedNotes,
partValue,
RawPeer,
@ -975,23 +974,6 @@ function reloadServer(server: Host<TestPrincipal>, setup: (host: FacetHost<TestP
}
describe("holds across a reload (slice 7b §6)", () => {
it("a swapped object keeps its holds: a fresh bound once the new instance is live, never loading (audit A9)", async () => {
const { server, client } = await linked((h) => serveShown(h, "v1"));
const shown = await client.get(Shown, {}, context);
await shown.bump(context);
const states = statesOf(await shown.state.watch(context));
const label = changesOf(await shown.label.watch(context));
const n = changesOf(await shown.n.watch(context));
await reloadServer(server, (h) => serveShown(h, "v2"));
await transported();
expect(label).toEqual([{ kind: "value", value: "v2", ops: [["r", "v2"]], retired: false }]);
// The fresh bound carries every member: an unchanged one may show one frame (nothing lost or stale).
expect(valuesOf(n)).toEqual([1]);
expect(states).toEqual([]);
expect(await shown.bump(context)).toBe(2);
await server.close(context);
});
it("a restarted object's holds see loading, then a fresh bound", async () => {
const { server, client } = await linked((h) => serveShown(h, "v1", true));
const shown = await client.get(Shown, {}, context);
@ -1086,7 +1068,7 @@ describe("holds across a reload (slice 7b §6)", () => {
await server.close(context);
});
it("a swap sends each bound hold one fresh bound with the new instance's values, no state (audit A9)", async () => {
it("a restart sends each bound hold loading, then one fresh bound with the new instance's values", async () => {
const server = createHost<TestPrincipal>({ address: Server(), principal: { kind: "server" }, onError: () => {} });
await server.load(
[{ name: "@test/app", facet: defineFacet<TestPrincipal>((h) => serveShown(h, "v1")) }],
@ -1104,14 +1086,14 @@ describe("holds across a reload (slice 7b §6)", () => {
peer.received.length = 0;
await reloadServer(server, (h) => serveShown(h, "v2"));
await transported();
expect(new Set(peer.kinds())).toEqual(new Set(["bound"]));
expect(new Set(peer.kinds())).toEqual(new Set(["loading", "bound"]));
const entries = peer.received.flatMap((message) => (message.kind === "bound" ? message.holds : []));
expect(entries.map(({ hold }) => hold).sort()).toEqual([0, 1]);
for (const entry of entries) expect(SHOWN.named(entry.members)).toEqual({ label: "v2", n: 0 });
await server.close(context);
});
it("a swap during a full window: calls are answered by the old names until the fresh bound goes (audit A9)", async () => {
it("reloads during a full window: calls are answered by the old names until the fresh bound goes (audit A9)", async () => {
const Two = remote("app.two-variants", {
version: 1,
params: {},
@ -1159,7 +1141,7 @@ describe("holds across a reload (slice 7b §6)", () => {
);
await server.reload([{ name: "@test/app", facet: changed }], context);
await peer.next("bound");
// Swaps in place while the window is full: the fresh bound waits for a slot, the hold answers by its old names.
// Restarts while the window is full: the fresh bound waits for a slot, the hold answers by its old names.
await server.reload([{ name: "@test/app", facet: serveTwo("a", "v2") }], context);
await transported();
peer.send({ kind: "call", call: 0, hold: 0, method: A.methods.bump! });
@ -1179,7 +1161,7 @@ describe("holds across a reload (slice 7b §6)", () => {
await server.close(context);
});
it("collects the swapped-out instance while the hold's fresh bound waits for a slot (Astra, audit A9 GC)", async () => {
it("collects the old instance while the hold's fresh bound waits for a slot (Astra, audit A9 GC)", async () => {
const Type = remote("astra.audit-gc", {
version: 1,
params: {},
@ -1218,6 +1200,112 @@ describe("holds across a reload (slice 7b §6)", () => {
await server.close(context);
});
it("a hold waiting for a window slot whose slot frees between the old attempt's seal and its end binds safely", async () => {
const Window = remote("app.window", {
version: 1,
params: {},
hostAddress: () => Server(),
variants: {
main: {
state: { n: int() },
methods: { bump: method(undefined, int()), slow: method(undefined, str()), start: method() },
},
},
});
const Wait = defineJob("app.window-wait", { version: 1, args: obj({}), result: str() });
const gate = Promise.withResolvers<void>();
/** Called once the old attempt's seal cut the job's run: the test frees the window's slots then. */
let atSeal = (): void => {};
const serveWindow = (code: string) =>
defineFacet<TestPrincipal>((h) =>
h.serve(Window, (_, _principal, self) => {
self.open(
async () => code,
() => {},
);
const doc = self.state(N);
const job = self.job(Wait, {
run: (_args, ctx) =>
new Promise<string>((resolve) =>
ctx.abortSignal?.addEventListener("abort", () => {
atSeal();
resolve("cut");
}),
),
});
return {
variant: "main",
members: {
n: self.derive([doc], ([d]) => d.n),
bump: async (ctx: Context) => {
await doc.change((draft) => {
draft.n += 1;
}, ctx);
return doc.value.n;
},
start: async (ctx: Context) => {
await job.start({}, ctx);
return null;
},
slow: async () => {
await gate.promise;
return `${code} slow`;
},
},
};
}),
);
const errors: string[] = [];
const server = createHost<TestPrincipal>({
address: Server(),
principal: { kind: "server" },
onError: (error) => errors.push(error.kind),
});
await server.load([{ name: "@test/app", facet: serveWindow("v1") }], context);
const [a, b] = memoryLinkPair();
server.accept(a, { principal: { kind: "owner" }, codec: testCodec() }, context);
const peer = new RawPeer(b, { ack: false });
const names = wireNames(Window, "main");
await peer.next("hello");
hold(peer, 0, Window);
await peer.next("bound");
peer.ack();
peer.send({ kind: "call", call: 0, hold: 0, method: names.methods.start! });
await peer.next("result");
// Two messages unacknowledged: the window (two) is full.
peer.send({ kind: "call", call: 1, hold: 0, method: names.methods.bump! });
await peer.next("state");
peer.send({ kind: "call", call: 2, hold: 0, method: names.methods.bump! });
await peer.next("result");
await peer.next("result");
await transported();
// A running call keeps the old attempt open; the acks come right at its seal (the job's run is cut there).
peer.send({ kind: "call", call: 3, hold: 0, method: names.methods.slow! });
await transported();
atSeal = () => {
peer.ack();
peer.ack();
};
await server.reload([{ name: "@test/app", facet: serveWindow("v2") }], context);
// A second hold during the drain: it binds the old attempt shown live, but waits for a slot.
hold(peer, 1, Window);
await transported();
gate.resolve();
expect(partValue((await peer.next("result")).value!)).toBe("v1 slow");
await transported();
await transported();
expect(errors).not.toContain("protocol_error");
expect(peer.closed).toBe(false);
// Both holds end on the new code.
for (let rounds = 0; rounds < 10 && !peer.received.some((m) => m.kind === "bound"); rounds++) await transported();
peer.ack();
peer.ack();
await transported();
const bound = peer.received.filter((message) => message.kind === "bound").flatMap(({ holds }) => holds);
expect(new Set(bound.map(({ hold }) => hold))).toEqual(new Set([0, 1]));
await server.close(context);
});
it("through a relay: a worker's reload reaches the tab as it reaches a direct consumer", async () => {
const { tab, workers, server } = await relayed();
const notes = await tab.get(PlacedNotes, { session: "S1" }, context);
@ -1237,7 +1325,7 @@ describe("holds across a reload (slice 7b §6)", () => {
});
});
describe("holds across more than one swap (fix round 1)", () => {
describe("holds across more than one reload (fix round 1)", () => {
it("keeps a real subscriber when a changed constant later becomes dynamic", async () => {
type P = { kind: "server" } | { kind: "tab" };
const Server = hostKind("sol-server");
@ -1328,7 +1416,7 @@ describe("a hold after its object's live failure (final review)", () => {
});
});
describe("a hold across a swap whose mapping fails (final review)", () => {
describe("a hold across a reload whose mapping fails (final review)", () => {
it("gets the member's failure in its fresh bound and heals with the new source (Sol, audit A9)", async () => {
const W1 = remote("app.swap-failed", {
version: 1,
@ -1374,7 +1462,7 @@ describe("a hold across a swap whose mapping fails (final review)", () => {
});
});
describe('calls over a link while a swap parks (decision "Swaps: dependents first")', () => {
describe('calls over a link while a restart waits (decision "Reload restarts every affected object")', () => {
/** A provider on the server: its name, its keys (`entries`) and `set`, which writes key `x`. */
const Provider = local("app.gate-provider", {
version: 1,
@ -1459,7 +1547,7 @@ describe('calls over a link while a swap parks (decision "Swaps: dependents firs
});
}
it("a call arriving after the swap parked, while an old call drains, waits for the new attempt (never not_ready)", async () => {
it("a call arriving while the old attempt drains and the successor's request opens waits for the new code (never not_ready)", async () => {
const gate = Promise.withResolvers<void>();
const opened = Promise.withResolvers<void>();
const { server, client } = await linked((h) => {
@ -1482,12 +1570,12 @@ describe('calls over a link while a swap parks (decision "Swaps: dependents firs
],
context,
);
// The new code of `Gated` waits for `Slow`, a new type still opening: its swap parks.
// The new code of `Gated` needs `Slow`, a new type still opening: its old attempt stays open meanwhile.
expect(report.objects).toEqual([
{ object: "app.gate-provider", kind: "swapped" },
{ object: "app.gated", kind: "parked", cause: expect.objectContaining({ kind: "waiting" }) },
{ object: "app.gate-provider", kind: "restarted" },
{ object: "app.gated", kind: "restarted" },
]);
// The old call still runs, so the hold stays bound: the client sees live and sends.
// The old attempt still runs its call, so the hold stays bound: the client sees live and sends new work.
const outcome = gated.bump(context).then(
(value) => ({ value }),
(error: ChordError) => ({ code: error.code }),
@ -1495,101 +1583,14 @@ describe('calls over a link while a swap parks (decision "Swaps: dependents firs
await transported();
expect(gated.state.value).toEqual({ kind: "live" });
opened.resolve();
expect(await outcome).toEqual({ value: 2 });
gate.resolve();
expect(await old).toBe(1);
expect(await outcome).toEqual({ value: 2 });
await transported();
expect(gated.label.value).toBe("v2 p2");
await client.close(context);
await server.close(context);
});
it("a call already waiting at the gate when the swap parks survives its hold's loading: answered by the new code", async () => {
const store = await memoryStore();
const server = createHost<TestPrincipal>({
address: Server(),
principal: { kind: "server" },
store,
onError: () => {},
});
const client = createHost<TestPrincipal>({ onError: () => {} });
const providerV1 = defineFacet<TestPrincipal>((h) => serveProvider(h, "p1", { persisted: true }));
await server.load(
[
{ name: "@test/provider", facet: providerV1 },
{ name: "@test/app", facet: defineFacet<TestPrincipal>((h) => serveGated(h, 1, {})) },
],
context,
);
await client.load([], context);
const [up, down] = memoryLinkPair();
server.accept(up, { principal: { kind: "owner" }, codec: testCodec() }, context);
client.connect(down, { principal: { kind: "server" }, codec: testCodec() }, context);
try {
const gated = await client.get(Gated, {}, context);
const provider = await server.get(Provider, {}, context);
// The provider's line is held by a commit: a key loads only after it.
const held = holdNextCommit(store);
const setting = provider.set(1, context);
await held.entered;
// `Gated`'s new code pins the provider's key `k`: its swap stays in place while the key waits for the line.
const appV2 = defineFacet<TestPrincipal>((h) => serveGated(h, 2, { pin: true }));
const report = await server.reload(
[
{ name: "@test/provider", facet: providerV1 },
{ name: "@test/app", facet: appV2 },
],
context,
);
expect(report.objects).toEqual([{ object: "app.gated", kind: "swapped" }]);
const states = statesOf(await gated.state.watch(context));
const outcome = gated.bump(context).then(
(value) => ({ value }),
(error: ChordError) => ({ code: error.code, message: error.message }),
);
await transported();
// The provider swaps too while the key waits; its new code needs \`Slow\`, still opening. Once the key loaded,
// the swap of \`Gated\` checks again, finds the provider loading and parks, with the call waiting at the gate.
const opened = Promise.withResolvers<void>();
const reloading = server.reload(
[
{
name: "@test/provider",
facet: defineFacet<TestPrincipal>((h) => {
serveSlow(h, opened.promise);
serveProvider(h, "p2", { persisted: true, slow: true });
}),
},
{ name: "@test/app", facet: appV2 },
],
context,
);
held.release();
await setting;
expect((await reloading).objects).toEqual([
{ object: "app.gate-provider", kind: "parked", cause: expect.objectContaining({ kind: "waiting" }) },
]);
await transported();
// Parked: its hold left the old instance and tells the client \`loading\`; the call that waited at the gate stays
// (decision "Calls in flight survive a \`loading\` unbind"), unanswered until the new attempt's outcome.
expect(states).toEqual(["loading"]);
let answered = false;
void outcome.then(() => {
answered = true;
});
await transported();
expect(answered).toBe(false);
opened.resolve();
expect(await outcome).toEqual({ value: 2 });
await transported();
expect(states).toEqual(["loading", "live"]);
expect(await gated.bump(context)).toBe(2);
expect(gated.label.value).toBe("v2 0");
} finally {
await client.close(context);
await server.close(context);
}
});
});
// A call sent after the hold's `loading` by the names it was bound with (as a consumer sends it before it learns of the
@ -1867,33 +1868,6 @@ describe('calls in flight across a `loading` unbind (decision "Calls in flight s
await server.close(context);
});
it("binding with the previous names, then a restart: the call running on the swapped-in instance drains to its result", async () => {
const gate = Promise.withResolvers<void>();
const opened = Promise.withResolvers<void>();
const { server, peer, reload } = await keptServer({ n: 1 }, { ack: false });
// A second `bound` in flight fills the window (two); the next swap in place leaves its fresh `bound` waiting.
await reload({ n: 2 });
await peer.next("bound");
await reload({ n: 3, gate: gate.promise });
await transported();
expect(peer.kinds()).toEqual([]);
call(peer, 0, "bump");
await transported();
// A restart unbinds the binding hold at once: the call runs on the live (unsealed) instance and stays.
await reload({ n: 4, native: true, opened: opened.promise });
await peer.next("loading");
await transported();
expect(answersTo(peer, 0)).toEqual([]);
gate.resolve();
const result = await peer.next("result");
expect([result.call, partValue(result.value!)]).toEqual([0, 3]);
opened.resolve();
peer.ack();
peer.ack();
await peer.next("bound");
await server.close(context);
});
it("a revocation racing the rebind ends the kept call with the link: no result", async () => {
const ran: string[] = [];
const { server, peer, reload } = await keptServer({ n: 1, native: true, ran });

65
packages/chord/test/remote/provider.test.ts
View file

@ -1166,3 +1166,68 @@ describe("telemetry at the provider (slice 9)", () => {
expect(telemetry.getMetrics().filter(({ name }) => name === "chord.backpressure.wait")).toHaveLength(1);
});
});
describe('old work from another host (decision "Reload restarts every affected object", `call.old`)', () => {
/** A remote type whose `slow` waits for `gate` (the call a restart drains) and whose `ping` answers its code. */
const Coded = remote("app.coded", {
version: 1,
params: {},
hostAddress: () => Server(),
variants: { v: { state: {}, methods: { slow: method(undefined, int()), ping: method(undefined, int()) } } },
});
const CODED = wireNames(Coded, "v");
/** Serves `Coded` as code `code`, `slow` first awaiting `gate`. */
const serveCoded = (h: FacetHost<TestPrincipal>, code: number, gate?: Promise<void>) =>
h.serve(Coded, () => ({
variant: "v",
members: {
slow: async () => {
await gate;
return code;
},
ping: async () => code,
},
}));
/**
* Holds `Coded` on a server restarting it while its call `slow` drains, then sends `ping` (marked old when `old`) on
* a link trusted when `trusted`: resolves with the code that answered it and whether that came before the drain.
*/
async function pingDuringDrain(old: boolean, trusted: boolean) {
const gate = Promise.withResolvers<void>();
const { server, peer } = await rawLinked((h) => serveCoded(h, 1, gate.promise), { trusted });
await peer.next("hello");
hold(peer, 0, Coded, {});
await peer.next("bound");
call(peer, 0, 0, CODED.methods.slow!);
await transported();
await server.reload([{ name: "@test/app", facet: defineFacet<TestPrincipal>((h) => serveCoded(h, 2)) }], context);
peer.send({
kind: "call",
call: 1,
hold: 0,
method: CODED.methods.ping!,
...(old ? { old: true as const } : {}),
});
await transported();
const early = peer.received.some((message) => message.kind === "result" && message.call === 1);
gate.resolve();
const first = await peer.next("result");
const result = first.call === 1 ? first : await peer.next("result");
await server.close(context);
return { code: partValue(result.value!), early };
}
it("runs a marked call from a trusted link on the open old attempt: answered by the old code, before the drain", async () => {
expect(await pingDuringDrain(true, true)).toEqual({ code: 1, early: true });
});
it("ignores the mark from a link that is not trusted: the call is new work and waits for the new code", async () => {
expect(await pingDuringDrain(true, false)).toEqual({ code: 2, early: false });
});
it("runs an unmarked call from a trusted link as new work", async () => {
expect(await pingDuringDrain(false, true)).toEqual({ code: 2, early: false });
});
});

26
packages/chord/test/remote/relay.test.ts
View file

@ -64,10 +64,14 @@ async function rawRelay(options: { readonly principal?: TestPrincipal; readonly
return { ...relay, workers, raw: tab };
}
/** A raw tab on a new link the server accepts as `principal` (owner by default). */
function rawTab(server: Awaited<ReturnType<typeof relayed>>["server"], principal?: TestPrincipal): RawPeer {
/** A raw tab on a new link the server accepts as `principal` (owner by default), trusted when `trusted`. */
function rawTab(
server: Awaited<ReturnType<typeof relayed>>["server"],
principal?: TestPrincipal,
trusted = false,
): RawPeer {
const [a, b] = memoryLinkPair();
server.accept(a, { principal: principal ?? { kind: "owner" }, codec: testCodec() }, context);
server.accept(a, { principal: principal ?? { kind: "owner" }, codec: testCodec(), trusted }, context);
return new RawPeer(b);
}
@ -150,6 +154,22 @@ forEachCodec(() => {
expect(await worker.next("retry")).toEqual({ kind: "retry", origin: 1, hold: 7, hops: 5 });
});
it("forward a call's old-work flag from a trusted downstream link only (a page cannot claim old work)", async () => {
const { raw, server, workers } = await rawRelay();
holdPlaced(raw, 0);
const worker = await workerAt(workers);
bindPlaced(worker, await worker.next("hold"));
await raw.next("bound");
raw.send({ kind: "call", call: 0, hold: 0, method: wait, old: true });
expect((await worker.next("call")).old).toBeUndefined();
const trusted = rawTab(server, undefined, true);
holdPlaced(trusted, 0);
bindPlaced(worker, await worker.next("hold"));
await trusted.next("bound");
trusted.send({ kind: "call", call: 0, hold: 0, method: wait, old: true });
expect(await worker.next("call")).toMatchObject({ call: 0, origin: 2, old: true });
});
it("give each downstream link its own origin, never reused", async () => {
const { raw, server, workers } = await rawRelay();
holdPlaced(raw, 0);

8
packages/chord/test/store/live-query.test.ts
View file

@ -1,6 +1,6 @@
// Tests of live queries (decision "Live queries over an object's own tables"): `self.query(table, q, limit)` runs
// before its object goes live, again in the commit step of every commit of the object that writes its table, and
// publishes the diff in that commit's batch; a commit writing another table runs none; a reload swaps the object;
// publishes the diff in that commit's batch; a commit writing another table runs none; a reload restarts the object;
// the limit is required; a failing run is a failed member; an ended attempt's query is collected; a refresh applied
// again adopts only its last runs.
@ -221,12 +221,12 @@ describe("publication", () => {
});
describe("reload", () => {
it("swaps the object (no restart): the new attempt's query runs at its activation and takes over", async () => {
it("restarts the object: the new attempt's query runs at its activation and takes over", async () => {
const store = await memoryStore();
const { board: b, host, context: counting, counts } = await board(store);
for (const id of [1, 2, 3]) await b.put({ id, text: `t${id}` }, context);
const report = await host.reload([{ name: "@test/app", facet: defineFacet((h) => serveBoard(h, 2)) }], context);
expect(report.objects).toEqual([{ object: "app.board", kind: "swapped" }]);
expect(report.objects).toEqual([{ object: "app.board", kind: "restarted" }]);
await flushed();
expect(ids(b.items.value)).toEqual([3, 2]);
const runs = counts()["chord.query.runs"] ?? 0;
@ -324,7 +324,7 @@ describe("failure", () => {
});
describe("memory", () => {
it("collects a swapped-out attempt's query: no commit re-runs it", async () => {
it("collects a restarted attempt's query: no commit re-runs it", async () => {
const store = await memoryStore();
const { host } = await board(store);
const from = queries.length - 1;

12
packages/chord/test/support.ts
View file

@ -241,12 +241,14 @@ export async function loaded(
readonly context?: Context;
readonly closeTimeout?: number;
readonly readyTimeout?: number;
readonly reloadDrainTimeout?: number;
readonly store?: Store;
} = {},
): Promise<{ readonly host: Host; readonly errors: HostError[]; readonly failed: readonly PluginFailure[] }> {
const errors: HostError[] = [];
const { telemetry, closeTimeout, readyTimeout, store } = options;
const host = createHost({ telemetry, closeTimeout, readyTimeout, store, onError: (error) => errors.push(error) });
const { telemetry, closeTimeout, readyTimeout, reloadDrainTimeout, store } = options;
const onError = (error: HostError) => errors.push(error);
const host = createHost({ telemetry, closeTimeout, readyTimeout, reloadDrainTimeout, store, onError });
const facets = [{ name: "@test/app", facet: defineFacet(setup) }];
const { failed } = await host.load(facets, options.context ?? BACKGROUND_CONTEXT);
return { host, errors, failed };
@ -544,7 +546,7 @@ export function wireNames(Type: Contract, variant: string) {
};
}
/** A server host as in `linked` and a raw peer on a link it accepted as `child`, with `limits`. */
/** A server host as in `linked` and a raw peer on a link it accepted as `child` (trusted when `trusted`), with `limits`. */
export async function rawLinked(
setup: (host: FacetHost<TestPrincipal>) => void,
options: {
@ -556,6 +558,8 @@ export async function rawLinked(
/** What the raw peer is to the host: a child (`accept`, the default) or its parent (`connect`). */
readonly role?: "child" | "parent";
readonly telemetry?: TelemetryContext;
/** Whether the host trusts the raw peer's link (its messages are not checked; `call.old` is honoured). */
readonly trusted?: boolean;
} = {},
): Promise<{ readonly server: Host<TestPrincipal>; readonly peer: RawPeer; readonly errors: HostError[] }> {
const errors: HostError[] = [];
@ -570,7 +574,7 @@ export async function rawLinked(
await server.load([{ name: "@test/app", facet: defineFacet<TestPrincipal>(setup) }], BACKGROUND_CONTEXT);
const [a, b] = memoryLinkPair();
const child = (options.child ?? { kind: "owner" }) as TestPrincipal;
const link = { principal: child, codec: testCodec(), limits: options.limits };
const link = { principal: child, codec: testCodec(), limits: options.limits, trusted: options.trusted };
if (options.role === "parent") server.connect(a, link, BACKGROUND_CONTEXT);
else server.accept(a, link, BACKGROUND_CONTEXT);
return { server, peer: new RawPeer(b, { hello: options.hello }), errors };