From 61f73f2a808e9cabb2b568170f771a56373a8824 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Sat, 31 Oct 2026 07:56:27 +0000 Subject: [PATCH] docs(chord): slice 10 spec review round 2 folded (Q1-Q6), ready to build --- packages/chord/docs/next/decisions.md | 21 +++ packages/chord/docs/next/design.md | 36 ++-- packages/chord/docs/next/slice-10.md | 257 ++++++++++++++++---------- 3 files changed, 207 insertions(+), 107 deletions(-) diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index de24791fc..e28177bbb 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1967,6 +1967,27 @@ validation and copies the v2 kernel added around Delta. (R6, "b, common enough") `bytes` may omit `size`: chunked transfer encoding, no ranges (no `Accept-Ranges`), `read` called with growing offsets until it returns an empty chunk (a zip built on the fly, a growing log, an export paged from a worker). + Spec review round 2 (Fable, Opus, Astra, Sol): (Q1, recommended) holders of a failed worker get `failed { waiting }` + with the stored error as cause and stay on the address's path: a later link of the address (Retry, a rollout) binds + them without a re-hold; a `get` waits its `readyTimeout` meanwhile. + (Q2, "a") A rollout that reaches workers clears the failures from before it, then calls `RouteHandle.retry` for the + cleared addresses that have holders (they start with the new code and are listed in the step); an unheld one starts + at its next hold; a worker that fails during this rollout stays failed (no loop: one start per held failed address per + rollout a person triggers). + (Q3, "a"; refines R1) A `stop`'s suspension lasts while the address has holders; they see `failed { waiting }` + ("suspended"); with the last holder the record goes, and a later hold starts the worker on demand (as after an idle + stop); whether an archived session may be opened is the app's (`authorize`, its views). Memory stays bounded. + (Q4, "a") The start timer (60 s) covers fork, lock and connect (`started`) only; after `started` a slow `load` is + objects showing `loading`; `ready` sets `state` to `running` and lets idle stops begin; a `load` that never ends + shows `starting` until `stop` (no kill, no retry). + (Q5, "a"; refines R3) A worker's idleness is `references`, `jobs`, `lingering` and the app's `busy`, not `calls`: a + remote call runs on a hold (counted), local calls come from routines (not counted, §14), jobs, Durable tasks (`busy`) + and handlers under a hold; no message per local call, no timer. The first report goes with `ready` (`ready { idle }`), + so a worker idle from its start is stopped too. + (Q6, owner: "if sockets are a contribution, why keep them when the contribution is removed?": design §17's rule, an + entry lives as long as its contributor and consumers keep nothing beyond a use) A plugin socket is closed (1012, + "service restart") when the contribution that handed it out leaves the table (a reload changing or removing it); a + `/chord` link or socket whose `Host` is no longer allowed closes (1008) when the allowed hosts change. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/design.md b/packages/chord/docs/next/design.md index f0ff3136b..7b1e3cded 100644 --- a/packages/chord/docs/next/design.md +++ b/packages/chord/docs/next/design.md @@ -314,7 +314,8 @@ host.route(Session, { (R1) stops resolving an address (its holders stay `loading`) until `retry`. A network route that wants to reconnect keeps its own policy in its `resolve`. The workers component answers a failed address's `resolve` with its stored failure at once: workers never restart by themselves (§23). -- Chord never closes a route link itself except at `host.close`; stopping idle workers is the app's (`host.activity`). +- Chord closes a route link itself only at `host.close`, on the heartbeat and on a protocol error; stopping idle + workers is the app's (`host.activity`, the workers component, §23). - Work arriving on a route's links runs under the context given to `host.route(Kind, options, context)` (slice 4b). ### Links @@ -2110,8 +2111,10 @@ public API; nothing in the kernel depends on them. ### HTTP (`@earendil-works/chord/next/http`) -`httpServer(host, { listen, authenticate, headers?, links? }, ctx)` listens first, holds every request and upgrade -until `load` resolved, then serves. `authenticate(request, ctx) => P | undefined` is the app's one hook (D3: who a +`httpServer(host, { listen, allowedHosts?, authenticate, headers?, links?, closeTimeout?, onError }, ctx)` listens +first; requests and upgrades wait only for the router's publication (`host.registry(Http)`, resolved once), never for +the start barrier. `Host` (loopback or allowed: the option plus contributed `hosts`) and an upgrade's `Origin` are +checked before anything else (R4); connections through a host no longer allowed close. `authenticate(request, ctx) => P | undefined` is the app's one hook (D3: who a request is, and which credential wins, is app policy; no Chord registry of resolvers); `headers` are added to every response, also those no route makes; `links.path` (`/chord`) upgrades to `host.accept(webSocketLink(ws), { principal, limits })`, a socket without a principal refused. `ws` is an optional peer dependency (D7). @@ -2128,7 +2131,7 @@ router as it was when it started. ```ts type HttpResponse = { readonly headers?: Readonly> } & ( // D5 | { kind: "json"; value: JsonValue; status?: number } | { kind: "html"; body: string; status?: number } - | { kind: "bytes"; mime: string; size: number; read(offset, length, ctx): Promise; download?: string; ranges?: boolean } + | { kind: "bytes"; mime: string; size?: number; read(offset, length, ctx): Promise; download?: string; ranges?: boolean } // no size: chunked (R6) | { kind: "redirect"; to: string; status?: 302 | 303 | 307 } | { kind: "error"; status: number; message?: string }); ``` @@ -2149,19 +2152,24 @@ Wraps (D2) run within a phase in contributor order (no numeric priorities: plugi order; Fastify's lifecycle phases and encapsulation are the model). Wraps may add a security boundary on top of a route's checks, never instead of them. `sockets` (D6): `{ "/terminal": (socket, req, ctx) => … }`, a `ws` WebSocket after the origin check and `authenticate` (no principal: 403 before plugin code), for a plugin's own protocol; wraps do -not apply. +not apply; closed (1012) when the contribution that handed them out leaves the table (§17). ### Workers (`@earendil-works/chord/next/workers`) -`createWorkers(host, Kind, { entry, start, plugins, idleMs?, onOutput? }, ctx)` routes `Kind` itself (a forked worker per -address, IPC, a start message with the app's start data and the worker kind's built scripts, `ready` within 60 s), a -`state` watchable per address and `stop(address)`; `workerMain(fn)` is the child's side. One live worker per address -(a kernel lock taken by the child before it opens its store). Keep-alive (D9, R3): the child reports when `host.activity` -and the app's `busy` counts flip between all zero and not; the server stops a worker idle (all zero) for `idleMs` (5 min). Workers start on demand and -never restart by themselves (R2): a crash or a failed start leaves the address failed with its error until -`workers.retry(address)` or a rollout reaching workers; `workers.stop` suspends the address (R1). Workers never build (D11): the server's plugins -component builds every plugin; workers evaluate shipped scripts, and `workers.step(name)` ships a rollout's scripts to -every running worker (D12: one report per host). +`createWorkers(host, Kind, { kind, entry, start, plugins, lock, idleMs?, onOutput?, onError }, ctx)` routes `Kind` +itself (a forked worker per address over IPC, the link made at the fork, a start message with the app's start data and +the worker kind's complete scripts; the 60 s start timer covers fork, lock and connect only, Q4), with a `state` +watchable per address, `stop`, `retry` and `close`; `workerMain(fn, { hostPackages, redact, onError })` is the child's +side. One live worker per address (a kernel lock the child takes, waiting for a draining old worker without polling). +Keep-alive (D9, R3, Q5): the child reports when `host.activity`'s references, jobs and lingering objects and the app's +`busy` counts flip between all zero and not (local calls do not count; the first state with `ready`); the server stops +a worker idle (all zero) for `idleMs` (5 min). Workers start on demand and never restart by themselves (R2): a crash, +a `fatal`, a link lost while its child lives (killed after the grace) or a failed start leaves the address failed with +its error; its holders see `failed { waiting }` and stay on the path (Q1); `workers.retry(address)` or a rollout +reaching workers clears it and starts the held ones (Q2). `workers.stop` suspends the address while it has holders +(R1, Q3). Workers never build (D11): the server's plugins component builds every plugin; workers evaluate shipped +complete sets (`createPluginRuntime`, no esbuild), and `workers.step(name)` ships a rollout's set to every worker (D12: +one result per host; a worker stopped cleanly during it is listed `stopped`). ### Channels diff --git a/packages/chord/docs/next/slice-10.md b/packages/chord/docs/next/slice-10.md index d0d63866c..4865ccad1 100644 --- a/packages/chord/docs/next/slice-10.md +++ b/packages/chord/docs/next/slice-10.md @@ -1,7 +1,8 @@ # Slice 10: HTTP and workers (spec) -Status: **ready for the owner's approval** (owner, 2026-10-30: D1-D12, D10b and the spec review's R1-R6 in the decision -"Slice 10's points"; spec reviews by Fable and Opus, `/tmp/s10-spec-review-{fable,opus}.md`, folded in). +Status: **ready for the owner's approval** (owner, 2026-10-30: D1-D12, D10b, round 1's R1-R6 and round 2's Q1-Q6 in the +decision "Slice 10's points"; spec reviews round 1 by Fable and Opus, `/tmp/s10-spec-review-{fable,opus}.md`, round 2 +by Fable, Opus, Astra and Sol, `/tmp/s10-spec-review2-*.md`, folded in). Slice 10 builds two Node components of the app kit (decision "Slice 7b's open points" (6)): `chord/next/http` (the server's HTTP and WebSocket entry, routes contributed by plugins through a registry, responses as descriptions) and `chord/next/workers` (a worker process per host address, started on demand, stopped when idle, its plugins shipped from @@ -44,23 +45,30 @@ const http = await httpServer(server, { allowedHosts: () => app.allowedHosts(), // besides loopback and contributed `hosts` (R4) authenticate: (request, ctx) => app.principalOf(request, ctx), // D3: the app's policy, per request and upgrade headers: (request, principal) => app.securityHeaders(request, principal), // every response, also a 404 or a 403 - links: { path: "/chord", limits: { part: 16 * MiB, decoded: 64 * MiB }, maxPayload: 64 * MiB }, + links: { path: "/chord", options: (request, principal) => ({ limits: linkLimits(principal), maxPayload: 64 * MiB }) }, + closeTimeout: 30_000, // `close`'s bound on running requests onError: (error) => log(error), // `HttpError`, below }, ctx); // the component's facet goes into the plugins component's core, so every reload passes it unchanged (7c) await plugins.load(server, { kind: "server", core: [{ name: "@app/core", facet: core }, { name: "chord.http", facet: http.facet }] }, ctx); … -await http.close(ctx); // stops accepting, closes sockets, waits for running requests (bounded by `closeTimeout`) +await http.close(ctx); // first (below): no new requests, sockets closed, running requests waited for up to `closeTimeout` +await workers.close(ctx); +await server.close(ctx); ``` - **Order:** `httpServer` listens at once (plugins see the real port; a page reconnecting while plugins load is not - lost). A request or upgrade waits in `host.registry(Http, ctx)` (the publication and the router's first `compose`; - 503 after `readyTimeout`); nothing waits for the start barrier (a route's `ctx.get` waits for its object, and `/chord` - holds are placed by the publication gate). -- **Checks before anything else** (R4), in order: the `Host` header is loopback (`127.0.0.1:`, `localhost:`) - or allowed (`allowedHosts()` plus every contribution's `hosts`), else 403; an upgrade's `Origin`, when sent, equals - the request's origin, else 403; `request.origin` is built only from an accepted `Host` (`http://` on loopback, - `https://` for an allowed host). Then `authenticate`. + lost). The component resolves `host.registry(Http, ctx)` once (the publication and the router's first `compose`); + requests and upgrades arriving before it wait for it (503 after `readyTimeout`), later ones read the router's `.value` + per request (no getter, timer or listener per request). Nothing waits for the start barrier: a route's `ctx.get` + waits for its object, and `/chord` holds are placed by the publication gate. The plugin name `chord.http` is the + component's. +- **Checks before anything else** (R4), once per request and per upgrade at its start, against the router of that + moment: the `Host` header is loopback (`127.0.0.1:`, `localhost:`, `[::1]:`, without a port when it + is 80) or allowed (`allowedHosts()` plus every contribution's `hosts`), else 403; an upgrade's `Origin`, when sent, + equals the request's origin, else 403; `request.origin` is built only from an accepted `Host` (`http://` on loopback, + `https://` for an allowed host). Then `authenticate`. A hook that throws (`allowedHosts`, `authenticate`, `headers`, + `links.options`) is a 500, or a refused upgrade (403), reported `hook_failed { hook, error }`. - **`Http`** (D1): the registry `"chord.http"` (version 1), defined and served by the component's own facet; its `compose` builds the router from every contribution `{ routes?, wraps?, sockets?, hosts? }`; a contribution may be a writable (a plugin that adds and removes its tunnel host at runtime). Each request uses the router as it was when it @@ -84,9 +92,10 @@ export default defineFacet((host, self) => { }); ``` -- **A request's scope:** one `host.root` scope with the request's principal (`ctx.principal`, `undefined` when - `authenticate` found none) spans matching, wraps, the route and the whole write of its response; its signal aborts - when the client disconnects. A socket handler's scope lasts until the socket closes. +- **A request's scope:** one `host.root` scope with the request's principal spans matching, wraps, the route and the + whole write of its response; its signal aborts when the client disconnects. `ctx.principal` is `undefined` when + `authenticate` found none: the kernel's `host.root(fn, ctx, { principal: undefined })` runs anonymous (§6), never as + the server. A socket handler's scope lasts until the socket closes. - **`HttpRequest`:** `method`, `path`, `params`, `query` (`URLSearchParams`), `headers` (lowercased, read-only), `origin`, `json(limit?)`, `body(limit?)` (D4). `json` parses at most `limit` bytes (1 MiB by default) into a `JsonValue`; `body` is an async iterable of `Uint8Array` chunks, failing past `limit`. Both throw an `HttpFailure` @@ -126,15 +135,23 @@ type HttpResponse = { readonly headers?: Readonly> } & ( and closes it when the request's scope seals, so an unwritten response (`HEAD`, a 416, a disconnect, a wrap that replaced it) opens nothing or leaks nothing. Without `mime` it picks from a small fixed table by extension (text, images, audio, video), else `application/octet-stream`; active types (HTML, SVG) only when the route names them. -- **Sockets** (D6): `sockets: { "/terminal": (socket, request, ctx) => … }`, patterns as routes. After the checks and - `authenticate` (no principal: 403 before plugin code), the component upgrades and hands the handler a `ws` - WebSocket with an `error` listener already attached (an oversized frame closes it, never the server); per contribution - options `maxPayload` (1 MiB by default) and no `perMessageDeflate`. Wraps do not apply. -- **The links endpoint:** `links.path` upgrades to `host.accept(webSocketLink(socket), { principal, limits }, ctx)`, - with a per-socket output cap (p2's `limitedSocket`: a client that stops reading past 4 MiB unsent is cut off) and the - transport's `maxPayload`. +- **Sockets** (D6, Q6): `sockets: { "/terminal": (socket, request, ctx) => … }`, patterns as routes. After the checks + and `authenticate` (no principal: 403 before plugin code), the component upgrades and hands the handler a `ws` + WebSocket with an `error` listener already attached (an oversized frame closes it, never the server); per + contribution options `maxPayload` (1 MiB by default) and no `perMessageDeflate`. Wraps do not apply. When the + contribution that handed it out leaves the table (a reload changing or removing it), the component closes its + sockets (1012, "service restart"): old code does not keep serving them (design §17). +- **The links endpoint:** `links.path` (bootstrap) upgrades to `host.accept(webSocketLink(socket), { principal, + limits }, ctx)`, its `limits` and `maxPayload` from `links.options(request, principal)` per socket (a socket keeps + what it got); a per-socket output cap (p2's `limitedSocket`: a client that stops reading past 4 MiB unsent is cut off). +- **Allowed hosts change** (Q6): when the allowed set changes, links and sockets accepted through a `Host` no longer + allowed close (1008, "policy"). +- **`close(ctx)`:** new requests and upgrades are refused (503), plugin sockets and links closed (1001), running + requests waited for up to `closeTimeout`, then their scopes sealed. Run it before `server.close`, whose seal would + otherwise cut running requests mid-write. - **Errors** (`onError`, `HttpError`): `route_failed { pattern, plugin, error }`, `wrap_failed { name, plugin, error }`, - `socket_failed { pattern, plugin, error }`, `response_failed { pattern, error }` (a short `read`, a write error). + `socket_failed { pattern, plugin, error }`, `response_failed { pattern, error }` (a short `read`, a write error), + `hook_failed { hook, error }`. - `ws` is an optional peer dependency (D7), imported only here. ## 3. `chord/next/workers` @@ -142,8 +159,9 @@ type HttpResponse = { readonly headers?: Readonly> } & ( ```ts // server.ts const workers = createWorkers(server, Session, { + kind: "worker", // the facet kind it ships entry: "./worker.ts", execArgv: ["--import", resolver], env: { … }, // bootstrap - authorize: (principal, address) => app.mayReach(principal, address), // the route's `authorize` + authorize: (principal, address) => app.mayReach(principal, address), // the route's `authorize` (synchronous) principal: (address) => ({ kind: "worker", session: address.params.id }), start: (address, ctx) => app.startData(address, ctx), // JSON, before each start lock: (address) => `app:${realpath(dataDir)}:worker:${address.key}`, // one live worker per address @@ -153,57 +171,87 @@ const workers = createWorkers(server, Session, { onError: (error) => log(error), // `WorkersError`, below }, ctx); workers.state // JSON watchable (R5) -await workers.stop(Session({ id }), ctx); // archive: suspended, then stopped; a held reference just waits +await workers.stop(Session({ id }), ctx); // archive: suspended while held (Q3), then stopped await workers.retry(Session({ id }), ctx); // the Retry button: a failed or suspended address resolves again -await workers.close(ctx); // after `server.close`: no new starts, every child stopped +await workers.close(ctx); // no new starts, every child stopped ``` ```ts // worker.ts -await workerMain(async ({ address, start, parent, plugins }, ctx) => { +await workerMain(async ({ address, start, parent, plugins, onClose }, ctx) => { const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(join(start.dir, "session.sqlite"))), ctx); + onClose((ctx) => store.close(ctx)); // closers run in reverse order const worker = createHost

({ address, principal: { kind: "worker", session: start.id }, store, onError }); - worker.connect(parent, { principal: { kind: "server" }, trusted: true }, ctx); // W6: connect, then load + onClose((ctx) => worker.close(ctx)); + worker.connect(parent, { principal: { kind: "server" }, trusted: true }, ctx); // `started` goes out now: connect + // before anything slow (a store opens in ms) await plugins.load(worker, { kind: "worker", core: [{ name: "@app/core", facet: coreWorker(start) }] }, ctx); - return { host: worker, busy: harness.activeTasks, close: (ctx) => store.close(ctx) }; -}, { onError: (error) => log(redact(error)) }); + return { busy: harness.activeTasks }; // extra keep-alive counts +}, { + hostPackages: { "@earendil-works/chord/next": chordNext, "@app/core": appCore }, // the child's own instances + redact: (error) => ({ message: redactSecrets(String(error)), ...codeOf(error) }), // what crosses as `fatal` + onError: (error) => log(error), +}); ``` - **The route:** `createWorkers` calls `host.route(Kind, { authorize, resolve }, ctx)` itself and keeps the - `RouteHandle` (`retry`, `suspend`: D10b, R1). -- **A start** (`resolve(address)`): one queue per address orders starts, stops and exits (a start waits for the - previous child's exit; a stop during a start waits for it, then stops; an aborted start kills its child). A start - calls `start(address, ctx)` (a throw rejects the start), forks `entry` (`serialization: "advanced"`, a detached - process group, stdout and stderr piped: the stderr tail kept, mirrored to the server's when there is no `onOutput`), - makes `ipcLink(child)` at once (so the child's `hello` and early holds are queued, never lost), and sends the start - message `{ address, start, scripts, lock }` (`scripts`: the worker kind's complete current scripts, below). The child - takes the lock (waiting for a draining old worker to let it go, within the start timeout), connects and sends - `started`; `resolve` then returns `{ link, principal: principal(address), trusted: true }`, so the handshake and the - worker's server traffic run during its `load`. `ready` (after the app's function returned) ends the start timer (60 - s from the fork; past it the tree is killed and the start fails). -- **Workers never restart by themselves** (R2): a crash (an exit nobody asked for) or a failed start (a `start` that - threw, a lock not taken in time, `fatal`, the start timer) leaves the address `failed` with its error - (`worker_failed` reported once); `resolve` of a failed address rejects at once with that error, so its holders and - new holds see `failed`, and nothing forks. `workers.retry(address)` (`RouteHandle.retry`) or a rollout that reaches - workers clears it. An idle stop or `stop` is no failure: the next hold starts it on demand. -- **Keep-alive** (D9, R3): the child watches `host.activity` and the app's `busy` (a JSON watchable of numbers) and - sends `{ idle: true }` when every count becomes zero, `{ idle: false }` when one becomes nonzero; a rollout applying in - the child counts as busy. The server stops a worker idle since `idleMs()` (5 min by default; checked every second), - never while it starts. A stop (idle, `stop`, `close`): `shutdown`; the child closes its host, runs the app's `close`, - exits; it forces its own exit after 3 s; the server kills the tree after 5 s. + `RouteHandle` (`retry`, `suspend`: D10b, R1). `authorize` is synchronous (the kernel's): a plugin backing it keeps an + index. +- **Per address, events and operations:** a child's events (its exit, its link's close, `started`, `ready`, `fatal`) + are observed as they happen and recorded at once on that child (a late event of an old child never changes a new + one); the operations (start, stop) run in one queue per address: a start waits for the previous child's exit, a + stop during a start waits for it then stops, and a start queued behind an exit checks the address's state after it + (a crash recorded meanwhile makes it reject with that failure). Stops record their intent before `shutdown`, so the + exit they cause is no crash. +- **A start** (`resolve(address)`): calls `start(address, ctx)` (abortable; a throw rejects the start), checks that the + component is not closing, forks `entry` (`serialization: "advanced"`, a detached process group, stdout and stderr + piped: the stderr tail kept, mirrored to the server's when there is no `onOutput`), makes `ipcLink(child)` at once + (so the child's `hello` and early holds are queued, never lost), and sends the start message `{ address, start, + scripts, lock }` (`scripts`: the worker kind's complete current set, read when the message is sent). The child takes + the lock (when an old worker still holds it, it waits by connecting to the lock's socket and trying again when that + connection closes: no polling), and sends `started` when the app's host sets its listener on `parent` (`parent` is + a thin wrapper of the IPC link, so `connect` is the signal: the app connects before anything slow). `resolve` then + returns `{ link, principal: principal(address), trusted: true }`, and the handshake and the worker's server traffic + run during its `load`. The start timer (60 s from the fork) covers fork, lock and connect only (Q4): past it the tree is + killed and the start fails. `ready` (after the app's function returned, carrying the idle state) sets `state` to + `running` and lets idle stops begin; a `load` that never ends shows `starting` until `stop`. +- **Workers never restart by themselves** (R2): a crash (an exit nobody asked for), a `fatal`, a link lost while its + child lives, or a failed start (`start` threw, the start timer, a lock not taken in time) leaves the address `failed` + with its error (`worker_failed` reported once). A link lost while its child lives (the heartbeat after a blocked + event loop, a protocol error) starts the stop: the grace (5 s), then the tree is killed; its exit is the crash ("link + lost"). `resolve` of a failed address rejects at once with that error; its holders and new holds see `failed { + waiting }` with it as cause (Q1) and stay on the address's path, nothing forks. `workers.retry(address)` + (`RouteHandle.retry`) or a rollout reaching workers clears it (Q2, §4). An idle stop is no failure: the next hold + starts the worker on demand. +- **Keep-alive** (D9, R3, Q5): the child's idle state is all of `host.activity`'s `references`, `jobs` and `lingering` + and the app's `busy` (a JSON watchable of numbers) at zero, not `calls` (a remote call runs on a hold; local calls + come from routines, jobs and tasks counted elsewhere). `ready { idle }` carries the first state, then `{ idle }` on + each flip; a rollout applying in the child counts as busy. The server stops a worker idle since `idleMs()` (5 min by + default; checked every second), never before `ready`. A stop (idle, `stop`, `close`): `shutdown`; the child aborts its + `ctx`, runs the closers registered with `onClose` in reverse order and exits, forcing its own exit after 3 s; the + server kills the tree after 5 s. - **`state`** (R5): per address key `{ status: "starting" | "running" | "stopping", pid, startedAt }` or `{ status: "failed", error: { message, code?, reason? }, at, stderrTail? }`; no entry when stopped or never started. -- **`stop(address, ctx)`:** `RouteHandle.suspend(address)` (holders stay `loading`, nothing resolves it), then the - stop; `retry(address)` lifts the suspension. **`close(ctx)`:** new starts reject `unavailable/closed`, every child is - stopped (grace, then kill), the timers cleared; `createWorkers`'s `ctx` aborting closes it too. -- **The child** (`workerMain(fn, { onError })`): reads the start message, takes the lock, makes `ipcLink(process)`, - calls `fn({ address, start, parent, plugins }, ctx)`; `started` once connected, `ready` once `fn` returned; reports - `idle` flips; handles `shutdown`, the parent's disconnect, `SIGTERM` and `SIGINT` alike (close the host, the app's - `close`, exit); an uncaught error is sent as `fatal { message, code?, reason? }` through `onError` (the app redacts), - then the child exits. +- **`stop(address, ctx)`** (R1, Q3): `RouteHandle.suspend(address)`, then the stop. The suspension lasts while the + address has holders (they see `failed { waiting }`, "suspended"); with the last holder it goes, and a later hold + starts the worker on demand; `retry(address)` lifts it at once. Whether an archived session may be opened is the + app's (`authorize`, its views). +- **`close(ctx)`:** cancels active and queued starts first (a `start(address)` waiting on the app's barrier is + aborted; nothing forks after it), then stops every child (grace, then kill), clears the timers; new starts reject + `released/closed`; the caller's cancellation ends only its own wait. `createWorkers`'s `ctx` aborting (the host + closing aborts the route's context) closes it too, and from then on every exit is a stop, never a crash (no + `worker_failed` at shutdown). +- **The child** (`workerMain(fn, { hostPackages, redact, onError })`): reads the start message, takes the lock, makes + `ipcLink(process)` and its wrapper `parent`, creates its plugin runtime with `hostPackages` (the child's own module + instances; functions cannot cross IPC), calls `fn({ address, start, parent, plugins, onClose }, ctx)`. `shutdown`, + the parent's disconnect, `SIGTERM` and `SIGINT` alike abort `ctx` and run the closers registered so far, also while + `fn` still runs (its startup is cancelled through `ctx`). An uncaught error, or `fn` throwing, sends `fatal` with + `redact(error)` (never the original message) and exits; after `started` that is a crash like any other. - **Control messages** (JSON beside the link's `Uint8Array` frames; the IPC link skips them): parent to child `start`, - `shutdown`, `reload { scripts }`; child to parent `started`, `ready`, `idle { idle }`, `fatal { error }`, `reloaded - { report } | { error }`. Errors cross as `{ message, code?, reason? }` (a structured clone keeps no `ChordError`). + `shutdown`, `reload { scripts }`; child to parent `started`, `ready { idle }`, `idle { idle }`, `fatal { error }`, + `reloaded { report } | { error }`. One `reload` is outstanding per child at a time: a reload whose caller aborted + keeps the child's slot until its `reloaded` or the child's exit, and the next waits for it. Errors cross as `{ + message, code?, reason? }`. - **Errors** (`onError`, `WorkersError`): `worker_failed { address, error, stderrTail? }`, `worker_exited { address, code, signal }` (a clean exit after a stop is not reported). @@ -213,8 +261,9 @@ Workers never build. The plugins component (7c) gets `scripts(kind, ctx)`: the c discovered plugin's facet of it, built on first use, then updated by every rollout that ships that kind), each with its plugin record (`name`, `dir`, `chord.external`). The component's per-host loading (its loaded map, `#prepare`, `adopt`, the queue per host) moves into a file without esbuild or the watcher, `createPluginRuntime({ hostPackages, onError })` -with `load(host, { kind, core, scripts }, ctx)` and `reload(host, shipped, ctx)`; `createPlugins` uses it, and the -child's `plugins` is one (`import` of `chord/next/plugins/runtime` loads no esbuild). +with `load(host, { kind, core, scripts }, ctx)` and `reload(host, scripts, ctx)` (a complete set: removed = loaded − +shipped; same digests unchanged); `createPlugins` uses it, and the child's `plugins` is one (`import` of +`chord/next/plugins/runtime` loads no esbuild). ## 4. The rollout across server and workers @@ -222,17 +271,25 @@ child's `plugins` is one (`import` of `chord/next/plugins/runtime` loads no esbu const built = await plugins.build(plugins.state.value.dirty, ctx); const result = await plugins.rollout([plugins.step("server", server), workers.step("workers")], built, ctx); // result.steps: [{ name: "server", hosts: [{ host: "server", report }] }, -// { name: "workers", hosts: [{ host: "session|S1", report }, { host: "session|S2", error }] }] +// { name: "workers", hosts: [{ host: "session|S1", report }, { host: "session|S2", stopped: true }, +// { host: "session|S3", error }] }] ``` -`RolloutStep.reload(built, ctx)` returns `{ hosts: readonly ({ host: string; report: ReloadReport } | { host: string; -error: { message, code?, reason? } })[] }` (D12; 7c's in-process step returns a list of one, its `host` the name given to -`step`). `workers.step(name)` first replaces the workers' current set (`plugins.scripts`), then ships the change to -every worker whose start message or last reload carried an older set, starting ones included (a child queues a reload -that arrives before its `ready` until its load is done); workers reload in parallel; one that closed or failed during -the step is listed with its error and does not stop the rollout, and every plugin of its kind in the build counts as -failed there (stays dirty). A rollout that reaches workers clears every failed address (R2). A plugin's dirty mark -clears only when it applied on every host of every step. +`RolloutStep.reload(built, ctx)` returns `{ hosts: readonly ({ host; report } | { host; stopped: true } | { host; error +})[] }` (D12; 7c's in-process step returns a list of one, its `host` the name given to `step`). `workers.step(name)`: + +1. replaces the workers' current set (`plugins.scripts(kind)` updated from `built`); +2. clears the failures recorded before this rollout and calls `RouteHandle.retry` for those addresses that have holders + (Q2): they start with the new set and are listed; an unheld one starts at its next hold; +3. ships the complete set to every running or starting worker whose start message or last reload carried an older one + (a child queues a reload that arrives before its `ready` until its load is done); workers reload in parallel. + +A worker's entry settles on its `reloaded` or on its exit or link loss, whichever comes first. A worker stopped cleanly +during the step (idle, `stop`) is listed `stopped` and dirties nothing (its next start gets the current set); one that +crashed or failed to apply is listed with its error and does not stop the rollout, and every plugin of its kind in the +build counts as failed there (stays dirty); a worker failing during this rollout stays failed. A plugin's dirty mark +clears only when it applied on every host of every step, and a kind built in the rollout that no step shipped keeps its +plugins dirty. ## 5. Not in slice 10 @@ -243,34 +300,48 @@ channel component (D6); telemetry spans for requests (slice 9). ## 6. Kernel changes -- `host.route(Kind, options, ctx)` returns a `RouteHandle { retry(address), suspend(address) }` (D10b, R1). -- A route has no backoff timer (R2): a rejected `resolve` or a link closed at once fails the route's holders with its - error (`failed`, not `loading`); the route resolves again only for a new hold, on `retry`, or once after the - address's link dropped while holds exist; a suspended address resolves nothing until `retry`. -- Design §3's paragraph on a `resolve` waiting for the worker's `load` (a deadlock) is updated: 7a's barrier ends its - first pass on a wait on another host, and `resolve` returns the link before `load` anyway. +- `host.route(Kind, options, ctx)` returns a `RouteHandle { retry(address), suspend(address) }` (D10b, R1, Q3). +- A route has no backoff timer (R2): a rejected `resolve`, or a link that closes before its handshake completed, fails + the route's holders `failed { waiting }` with its error (they stay on the address's path, Q1) and resolves nothing + more; the route resolves again only for a new hold, on `retry`, or once after a link that had completed its handshake + dropped while holds exist; a suspended address resolves nothing while it has holders and forgets the suspension with + the last one. +- `host.root(fn, ctx, { principal: undefined })` runs anonymous (`ctx.principal` undefined); only an omitted + `principal` takes the host's (a public-API change of `RootOptions`: `"principal" in options`). +- Design §3: the paragraph on a `resolve` waiting for the worker's `load` is updated (done), and "Chord never closes a + route link itself except at `host.close`" becomes "except at `host.close`, the heartbeat and protocol errors". ## 7. Tests, examples, cost - One test file per source file. HTTP: listening before the publication (a request waits, then is served; 503 past - `readyTimeout`); `Host` and `Origin` checks, allowed hosts from the option and from a writable contribution changing - at runtime; authentication per request and upgrade; patterns, decoding, precedence, `HEAD`, 404; conflicts reported; - wraps (phases, order, scopes, a missing route, a throwing wrap); every response kind (ranges 206/416, multi-range, - `HEAD`, download names, chunked without `size`, a short `read`, a disconnect stopping `read`, header merging); - `json`/`body` limits and their statuses, an unread body; a throwing route; the links endpoint (`host.accept`, a - refused socket, the output cap); plugin sockets (an oversized frame closes only that socket); a reload swapping a - route while a request runs on the old router; `fileResponse` closing its handle on every path; `close`. Workers: a - start on the first hold; the link made at the fork (a load longer than the heartbeat's 30 s binds); the lock (a - draining old worker waited for); the start timer; `start` throwing; a crash and a failed start staying failed (no - fork for a new hold), `retry` and a rollout clearing it; idle stop and a hold keeping it; `busy`; the `idle` flips (no - message per call); `stop` with holders (suspended: no restart); stop during start and start during stop; `close`; - the rollout step (a worker starting during it gets the new set; a failed worker listed); the child's shutdown, - disconnect and signals. Kernel: `RouteHandle.retry` and `suspend`; no re-resolve without a hold, a retry or one drop. + `readyTimeout`); the router resolved once; `Host` (loopback forms, allowed from the option and from a writable + contribution changing at runtime) and `Origin` checks; anonymous requests (`ctx.principal` undefined on a server with + a principal); authentication per request and upgrade; throwing hooks; patterns, decoding, precedence, `HEAD`, 404; + conflicts reported; wraps (phases, order, scopes, a missing route, a throwing wrap); every response kind (ranges + 206/416, multi-range, `HEAD`, download names, chunked without `size`, a short `read`, a disconnect stopping `read`, + header merging); `json`/`body` limits and their statuses, an unread body; a throwing route; the links endpoint + (`host.accept`, options per socket, a refused socket, the output cap); plugin sockets (an oversized frame closes only + that socket; a reload removing the contribution closes its sockets 1012); a removed allowed host closing its + connections; a reload swapping a route while a request runs on the old router; `fileResponse` closing its handle on + every path; `close` (refusals, waiting, then sealing). Workers: a start on the first hold; the link made at the fork + (a load longer than the heartbeat's 30 s binds); `started` on `connect`; the start timer covering only the start (a + slow `load` is not killed); the lock (an old worker's lock waited for without polling); `start` throwing; a crash, + a `fatal`, a link lost with a live child (killed after the grace), and a failed start each staying failed (no fork for + a new hold; holders `failed { waiting }`), `retry` and a rollout clearing them and rebinding the holders; idle stop, + a worker idle from its start, a hold keeping it; `busy`; 1,000 sequential local calls sending no message; + `stop` with holders (suspended, then forgotten with the last holder); stop during start and start during stop; a + crash while a resolve is queued; `close` (cancelling a start waiting in `start(address)`, no `worker_failed` at + shutdown); the rollout step (complete sets, a worker starting during it, a clean stop listed `stopped`, a crash during + it, an aborted reload's late reply, a kind no step shipped staying dirty); the child's shutdown during its startup, + disconnect and signals; `redact` (a secret in the original message reaches neither `fatal` nor `state`). Kernel: + `RouteHandle.retry` and `suspend`; a link closing before its handshake (no re-resolve loop); no re-resolve without a + hold, a retry or one drop of a healthy link; `host.root` with an explicit `undefined` principal. - GC: requests (their scopes, bodies, file handles) and stopped workers' state collected; no listener left on a closed - socket; 1,000 addresses started and stopped leave no `state` entries. + socket; 1,000 addresses started and stopped leave no `state` entries, no per-address queue records and no + suspensions. - Examples (each a folder, as 7c's): `40-http` (routes, a wrap, a download with ranges, a socket, an allowed host added at runtime), `41-workers` (a server routing a session kind to forked workers, idle stop, a crash staying failed - until Retry, a rollout reaching the workers). + until Retry, a rollout reaching the workers and restarting a held failed one). - Benchmark: a route's overhead per request against a plain Node handler (bar: within 10% of a keep-alive round trip; Opus measured 2.5 µs of 54 µs); `bytes` throughput for a 100 MiB file against `pipeline`; a worker's cold start - (fork to `ready`) with shipped scripts; memory per idle worker on the server. + (fork to `started` and to `ready`) with shipped scripts; memory per idle worker on the server.