mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
docs(chord): slice 10 spec review round 2 folded (Q1-Q6), ready to build
This commit is contained in:
Mario Zechner
parent
35dfa75737
commit
61f73f2a80
3 changed files with 207 additions and 107 deletions
|
|
@ -1967,6 +1967,27 @@ validation and copies the v2 kernel added around Delta.
|
|||
(R6, "b, common enough") `bytes` may omit `size`: chunked transfer encoding, no ranges (no `Accept-Ranges`), `read`
|
||||
called with growing offsets until it returns an empty chunk (a zip built on the fly, a growing log, an export paged
|
||||
from a worker).
|
||||
Spec review round 2 (Fable, Opus, Astra, Sol): (Q1, recommended) holders of a failed worker get `failed { waiting }`
|
||||
with the stored error as cause and stay on the address's path: a later link of the address (Retry, a rollout) binds
|
||||
them without a re-hold; a `get` waits its `readyTimeout` meanwhile.
|
||||
(Q2, "a") A rollout that reaches workers clears the failures from before it, then calls `RouteHandle.retry` for the
|
||||
cleared addresses that have holders (they start with the new code and are listed in the step); an unheld one starts
|
||||
at its next hold; a worker that fails during this rollout stays failed (no loop: one start per held failed address per
|
||||
rollout a person triggers).
|
||||
(Q3, "a"; refines R1) A `stop`'s suspension lasts while the address has holders; they see `failed { waiting }`
|
||||
("suspended"); with the last holder the record goes, and a later hold starts the worker on demand (as after an idle
|
||||
stop); whether an archived session may be opened is the app's (`authorize`, its views). Memory stays bounded.
|
||||
(Q4, "a") The start timer (60 s) covers fork, lock and connect (`started`) only; after `started` a slow `load` is
|
||||
objects showing `loading`; `ready` sets `state` to `running` and lets idle stops begin; a `load` that never ends
|
||||
shows `starting` until `stop` (no kill, no retry).
|
||||
(Q5, "a"; refines R3) A worker's idleness is `references`, `jobs`, `lingering` and the app's `busy`, not `calls`: a
|
||||
remote call runs on a hold (counted), local calls come from routines (not counted, §14), jobs, Durable tasks (`busy`)
|
||||
and handlers under a hold; no message per local call, no timer. The first report goes with `ready` (`ready { idle }`),
|
||||
so a worker idle from its start is stopped too.
|
||||
(Q6, owner: "if sockets are a contribution, why keep them when the contribution is removed?": design §17's rule, an
|
||||
entry lives as long as its contributor and consumers keep nothing beyond a use) A plugin socket is closed (1012,
|
||||
"service restart") when the contribution that handed it out leaves the table (a reload changing or removing it); a
|
||||
`/chord` link or socket whose `Host` is no longer allowed closes (1008) when the allowed hosts change.
|
||||
- **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):**
|
||||
a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into
|
||||
each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a
|
||||
|
|
|
|||
|
|
@ -314,7 +314,8 @@ host.route(Session, {
|
|||
(R1) stops resolving an address (its holders stay `loading`) until `retry`. A network route that wants to reconnect
|
||||
keeps its own policy in its `resolve`. The workers component answers a failed address's `resolve` with its stored
|
||||
failure at once: workers never restart by themselves (§23).
|
||||
- Chord never closes a route link itself except at `host.close`; stopping idle workers is the app's (`host.activity`).
|
||||
- Chord closes a route link itself only at `host.close`, on the heartbeat and on a protocol error; stopping idle
|
||||
workers is the app's (`host.activity`, the workers component, §23).
|
||||
- Work arriving on a route's links runs under the context given to `host.route(Kind, options, context)` (slice 4b).
|
||||
|
||||
### Links
|
||||
|
|
@ -2110,8 +2111,10 @@ public API; nothing in the kernel depends on them.
|
|||
|
||||
### HTTP (`@earendil-works/chord/next/http`)
|
||||
|
||||
`httpServer(host, { listen, authenticate, headers?, links? }, ctx)` listens first, holds every request and upgrade
|
||||
until `load` resolved, then serves. `authenticate(request, ctx) => P | undefined` is the app's one hook (D3: who a
|
||||
`httpServer(host, { listen, allowedHosts?, authenticate, headers?, links?, closeTimeout?, onError }, ctx)` listens
|
||||
first; requests and upgrades wait only for the router's publication (`host.registry(Http)`, resolved once), never for
|
||||
the start barrier. `Host` (loopback or allowed: the option plus contributed `hosts`) and an upgrade's `Origin` are
|
||||
checked before anything else (R4); connections through a host no longer allowed close. `authenticate(request, ctx) => P | undefined` is the app's one hook (D3: who a
|
||||
request is, and which credential wins, is app policy; no Chord registry of resolvers); `headers` are added to every
|
||||
response, also those no route makes; `links.path` (`/chord`) upgrades to `host.accept(webSocketLink(ws), { principal,
|
||||
limits })`, a socket without a principal refused. `ws` is an optional peer dependency (D7).
|
||||
|
|
@ -2128,7 +2131,7 @@ router as it was when it started.
|
|||
```ts
|
||||
type HttpResponse = { readonly headers?: Readonly<Record<string, string>> } & ( // D5
|
||||
| { kind: "json"; value: JsonValue; status?: number } | { kind: "html"; body: string; status?: number }
|
||||
| { kind: "bytes"; mime: string; size: number; read(offset, length, ctx): Promise<Uint8Array>; download?: string; ranges?: boolean }
|
||||
| { kind: "bytes"; mime: string; size?: number; read(offset, length, ctx): Promise<Uint8Array>; download?: string; ranges?: boolean } // no size: chunked (R6)
|
||||
| { kind: "redirect"; to: string; status?: 302 | 303 | 307 } | { kind: "error"; status: number; message?: string });
|
||||
```
|
||||
|
||||
|
|
@ -2149,19 +2152,24 @@ Wraps (D2) run within a phase in contributor order (no numeric priorities: plugi
|
|||
order; Fastify's lifecycle phases and encapsulation are the model). Wraps may add a security boundary on top of a
|
||||
route's checks, never instead of them. `sockets` (D6): `{ "/terminal": (socket, req, ctx) => … }`, a `ws` WebSocket
|
||||
after the origin check and `authenticate` (no principal: 403 before plugin code), for a plugin's own protocol; wraps do
|
||||
not apply.
|
||||
not apply; closed (1012) when the contribution that handed them out leaves the table (§17).
|
||||
|
||||
### Workers (`@earendil-works/chord/next/workers`)
|
||||
|
||||
`createWorkers(host, Kind, { entry, start, plugins, idleMs?, onOutput? }, ctx)` routes `Kind` itself (a forked worker per
|
||||
address, IPC, a start message with the app's start data and the worker kind's built scripts, `ready` within 60 s), a
|
||||
`state` watchable per address and `stop(address)`; `workerMain(fn)` is the child's side. One live worker per address
|
||||
(a kernel lock taken by the child before it opens its store). Keep-alive (D9, R3): the child reports when `host.activity`
|
||||
and the app's `busy` counts flip between all zero and not; the server stops a worker idle (all zero) for `idleMs` (5 min). Workers start on demand and
|
||||
never restart by themselves (R2): a crash or a failed start leaves the address failed with its error until
|
||||
`workers.retry(address)` or a rollout reaching workers; `workers.stop` suspends the address (R1). Workers never build (D11): the server's plugins
|
||||
component builds every plugin; workers evaluate shipped scripts, and `workers.step(name)` ships a rollout's scripts to
|
||||
every running worker (D12: one report per host).
|
||||
`createWorkers(host, Kind, { kind, entry, start, plugins, lock, idleMs?, onOutput?, onError }, ctx)` routes `Kind`
|
||||
itself (a forked worker per address over IPC, the link made at the fork, a start message with the app's start data and
|
||||
the worker kind's complete scripts; the 60 s start timer covers fork, lock and connect only, Q4), with a `state`
|
||||
watchable per address, `stop`, `retry` and `close`; `workerMain(fn, { hostPackages, redact, onError })` is the child's
|
||||
side. One live worker per address (a kernel lock the child takes, waiting for a draining old worker without polling).
|
||||
Keep-alive (D9, R3, Q5): the child reports when `host.activity`'s references, jobs and lingering objects and the app's
|
||||
`busy` counts flip between all zero and not (local calls do not count; the first state with `ready`); the server stops
|
||||
a worker idle (all zero) for `idleMs` (5 min). Workers start on demand and never restart by themselves (R2): a crash,
|
||||
a `fatal`, a link lost while its child lives (killed after the grace) or a failed start leaves the address failed with
|
||||
its error; its holders see `failed { waiting }` and stay on the path (Q1); `workers.retry(address)` or a rollout
|
||||
reaching workers clears it and starts the held ones (Q2). `workers.stop` suspends the address while it has holders
|
||||
(R1, Q3). Workers never build (D11): the server's plugins component builds every plugin; workers evaluate shipped
|
||||
complete sets (`createPluginRuntime`, no esbuild), and `workers.step(name)` ships a rollout's set to every worker (D12:
|
||||
one result per host; a worker stopped cleanly during it is listed `stopped`).
|
||||
|
||||
### Channels
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,8 @@
|
|||
# Slice 10: HTTP and workers (spec)
|
||||
|
||||
Status: **ready for the owner's approval** (owner, 2026-10-30: D1-D12, D10b and the spec review's R1-R6 in the decision
|
||||
"Slice 10's points"; spec reviews by Fable and Opus, `/tmp/s10-spec-review-{fable,opus}.md`, folded in).
|
||||
Status: **ready for the owner's approval** (owner, 2026-10-30: D1-D12, D10b, round 1's R1-R6 and round 2's Q1-Q6 in the
|
||||
decision "Slice 10's points"; spec reviews round 1 by Fable and Opus, `/tmp/s10-spec-review-{fable,opus}.md`, round 2
|
||||
by Fable, Opus, Astra and Sol, `/tmp/s10-spec-review2-*.md`, folded in).
|
||||
Slice 10 builds two Node components of the app kit (decision "Slice 7b's open points" (6)): `chord/next/http` (the
|
||||
server's HTTP and WebSocket entry, routes contributed by plugins through a registry, responses as descriptions) and
|
||||
`chord/next/workers` (a worker process per host address, started on demand, stopped when idle, its plugins shipped from
|
||||
|
|
@ -44,23 +45,30 @@ const http = await httpServer(server, {
|
|||
allowedHosts: () => app.allowedHosts(), // besides loopback and contributed `hosts` (R4)
|
||||
authenticate: (request, ctx) => app.principalOf(request, ctx), // D3: the app's policy, per request and upgrade
|
||||
headers: (request, principal) => app.securityHeaders(request, principal), // every response, also a 404 or a 403
|
||||
links: { path: "/chord", limits: { part: 16 * MiB, decoded: 64 * MiB }, maxPayload: 64 * MiB },
|
||||
links: { path: "/chord", options: (request, principal) => ({ limits: linkLimits(principal), maxPayload: 64 * MiB }) },
|
||||
closeTimeout: 30_000, // `close`'s bound on running requests
|
||||
onError: (error) => log(error), // `HttpError`, below
|
||||
}, ctx);
|
||||
// the component's facet goes into the plugins component's core, so every reload passes it unchanged (7c)
|
||||
await plugins.load(server, { kind: "server", core: [{ name: "@app/core", facet: core }, { name: "chord.http", facet: http.facet }] }, ctx);
|
||||
…
|
||||
await http.close(ctx); // stops accepting, closes sockets, waits for running requests (bounded by `closeTimeout`)
|
||||
await http.close(ctx); // first (below): no new requests, sockets closed, running requests waited for up to `closeTimeout`
|
||||
await workers.close(ctx);
|
||||
await server.close(ctx);
|
||||
```
|
||||
|
||||
- **Order:** `httpServer` listens at once (plugins see the real port; a page reconnecting while plugins load is not
|
||||
lost). A request or upgrade waits in `host.registry(Http, ctx)` (the publication and the router's first `compose`;
|
||||
503 after `readyTimeout`); nothing waits for the start barrier (a route's `ctx.get` waits for its object, and `/chord`
|
||||
holds are placed by the publication gate).
|
||||
- **Checks before anything else** (R4), in order: the `Host` header is loopback (`127.0.0.1:<port>`, `localhost:<port>`)
|
||||
or allowed (`allowedHosts()` plus every contribution's `hosts`), else 403; an upgrade's `Origin`, when sent, equals
|
||||
the request's origin, else 403; `request.origin` is built only from an accepted `Host` (`http://` on loopback,
|
||||
`https://` for an allowed host). Then `authenticate`.
|
||||
lost). The component resolves `host.registry(Http, ctx)` once (the publication and the router's first `compose`);
|
||||
requests and upgrades arriving before it wait for it (503 after `readyTimeout`), later ones read the router's `.value`
|
||||
per request (no getter, timer or listener per request). Nothing waits for the start barrier: a route's `ctx.get`
|
||||
waits for its object, and `/chord` holds are placed by the publication gate. The plugin name `chord.http` is the
|
||||
component's.
|
||||
- **Checks before anything else** (R4), once per request and per upgrade at its start, against the router of that
|
||||
moment: the `Host` header is loopback (`127.0.0.1:<port>`, `localhost:<port>`, `[::1]:<port>`, without a port when it
|
||||
is 80) or allowed (`allowedHosts()` plus every contribution's `hosts`), else 403; an upgrade's `Origin`, when sent,
|
||||
equals the request's origin, else 403; `request.origin` is built only from an accepted `Host` (`http://` on loopback,
|
||||
`https://` for an allowed host). Then `authenticate`. A hook that throws (`allowedHosts`, `authenticate`, `headers`,
|
||||
`links.options`) is a 500, or a refused upgrade (403), reported `hook_failed { hook, error }`.
|
||||
- **`Http`** (D1): the registry `"chord.http"` (version 1), defined and served by the component's own facet; its
|
||||
`compose` builds the router from every contribution `{ routes?, wraps?, sockets?, hosts? }`; a contribution may be a
|
||||
writable (a plugin that adds and removes its tunnel host at runtime). Each request uses the router as it was when it
|
||||
|
|
@ -84,9 +92,10 @@ export default defineFacet((host, self) => {
|
|||
});
|
||||
```
|
||||
|
||||
- **A request's scope:** one `host.root` scope with the request's principal (`ctx.principal`, `undefined` when
|
||||
`authenticate` found none) spans matching, wraps, the route and the whole write of its response; its signal aborts
|
||||
when the client disconnects. A socket handler's scope lasts until the socket closes.
|
||||
- **A request's scope:** one `host.root` scope with the request's principal spans matching, wraps, the route and the
|
||||
whole write of its response; its signal aborts when the client disconnects. `ctx.principal` is `undefined` when
|
||||
`authenticate` found none: the kernel's `host.root(fn, ctx, { principal: undefined })` runs anonymous (§6), never as
|
||||
the server. A socket handler's scope lasts until the socket closes.
|
||||
- **`HttpRequest`:** `method`, `path`, `params`, `query` (`URLSearchParams`), `headers` (lowercased, read-only),
|
||||
`origin`, `json(limit?)`, `body(limit?)` (D4). `json` parses at most `limit` bytes (1 MiB by default) into a
|
||||
`JsonValue`; `body` is an async iterable of `Uint8Array` chunks, failing past `limit`. Both throw an `HttpFailure`
|
||||
|
|
@ -126,15 +135,23 @@ type HttpResponse = { readonly headers?: Readonly<Record<string, string>> } & (
|
|||
and closes it when the request's scope seals, so an unwritten response (`HEAD`, a 416, a disconnect, a wrap that
|
||||
replaced it) opens nothing or leaks nothing. Without `mime` it picks from a small fixed table by extension (text,
|
||||
images, audio, video), else `application/octet-stream`; active types (HTML, SVG) only when the route names them.
|
||||
- **Sockets** (D6): `sockets: { "/terminal": (socket, request, ctx) => … }`, patterns as routes. After the checks and
|
||||
`authenticate` (no principal: 403 before plugin code), the component upgrades and hands the handler a `ws`
|
||||
WebSocket with an `error` listener already attached (an oversized frame closes it, never the server); per contribution
|
||||
options `maxPayload` (1 MiB by default) and no `perMessageDeflate`. Wraps do not apply.
|
||||
- **The links endpoint:** `links.path` upgrades to `host.accept(webSocketLink(socket), { principal, limits }, ctx)`,
|
||||
with a per-socket output cap (p2's `limitedSocket`: a client that stops reading past 4 MiB unsent is cut off) and the
|
||||
transport's `maxPayload`.
|
||||
- **Sockets** (D6, Q6): `sockets: { "/terminal": (socket, request, ctx) => … }`, patterns as routes. After the checks
|
||||
and `authenticate` (no principal: 403 before plugin code), the component upgrades and hands the handler a `ws`
|
||||
WebSocket with an `error` listener already attached (an oversized frame closes it, never the server); per
|
||||
contribution options `maxPayload` (1 MiB by default) and no `perMessageDeflate`. Wraps do not apply. When the
|
||||
contribution that handed it out leaves the table (a reload changing or removing it), the component closes its
|
||||
sockets (1012, "service restart"): old code does not keep serving them (design §17).
|
||||
- **The links endpoint:** `links.path` (bootstrap) upgrades to `host.accept(webSocketLink(socket), { principal,
|
||||
limits }, ctx)`, its `limits` and `maxPayload` from `links.options(request, principal)` per socket (a socket keeps
|
||||
what it got); a per-socket output cap (p2's `limitedSocket`: a client that stops reading past 4 MiB unsent is cut off).
|
||||
- **Allowed hosts change** (Q6): when the allowed set changes, links and sockets accepted through a `Host` no longer
|
||||
allowed close (1008, "policy").
|
||||
- **`close(ctx)`:** new requests and upgrades are refused (503), plugin sockets and links closed (1001), running
|
||||
requests waited for up to `closeTimeout`, then their scopes sealed. Run it before `server.close`, whose seal would
|
||||
otherwise cut running requests mid-write.
|
||||
- **Errors** (`onError`, `HttpError`): `route_failed { pattern, plugin, error }`, `wrap_failed { name, plugin, error }`,
|
||||
`socket_failed { pattern, plugin, error }`, `response_failed { pattern, error }` (a short `read`, a write error).
|
||||
`socket_failed { pattern, plugin, error }`, `response_failed { pattern, error }` (a short `read`, a write error),
|
||||
`hook_failed { hook, error }`.
|
||||
- `ws` is an optional peer dependency (D7), imported only here.
|
||||
|
||||
## 3. `chord/next/workers`
|
||||
|
|
@ -142,8 +159,9 @@ type HttpResponse = { readonly headers?: Readonly<Record<string, string>> } & (
|
|||
```ts
|
||||
// server.ts
|
||||
const workers = createWorkers(server, Session, {
|
||||
kind: "worker", // the facet kind it ships
|
||||
entry: "./worker.ts", execArgv: ["--import", resolver], env: { … }, // bootstrap
|
||||
authorize: (principal, address) => app.mayReach(principal, address), // the route's `authorize`
|
||||
authorize: (principal, address) => app.mayReach(principal, address), // the route's `authorize` (synchronous)
|
||||
principal: (address) => ({ kind: "worker", session: address.params.id }),
|
||||
start: (address, ctx) => app.startData(address, ctx), // JSON, before each start
|
||||
lock: (address) => `app:${realpath(dataDir)}:worker:${address.key}`, // one live worker per address
|
||||
|
|
@ -153,57 +171,87 @@ const workers = createWorkers(server, Session, {
|
|||
onError: (error) => log(error), // `WorkersError`, below
|
||||
}, ctx);
|
||||
workers.state // JSON watchable (R5)
|
||||
await workers.stop(Session({ id }), ctx); // archive: suspended, then stopped; a held reference just waits
|
||||
await workers.stop(Session({ id }), ctx); // archive: suspended while held (Q3), then stopped
|
||||
await workers.retry(Session({ id }), ctx); // the Retry button: a failed or suspended address resolves again
|
||||
await workers.close(ctx); // after `server.close`: no new starts, every child stopped
|
||||
await workers.close(ctx); // no new starts, every child stopped
|
||||
```
|
||||
|
||||
```ts
|
||||
// worker.ts
|
||||
await workerMain(async ({ address, start, parent, plugins }, ctx) => {
|
||||
await workerMain(async ({ address, start, parent, plugins, onClose }, ctx) => {
|
||||
const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(join(start.dir, "session.sqlite"))), ctx);
|
||||
onClose((ctx) => store.close(ctx)); // closers run in reverse order
|
||||
const worker = createHost<P>({ address, principal: { kind: "worker", session: start.id }, store, onError });
|
||||
worker.connect(parent, { principal: { kind: "server" }, trusted: true }, ctx); // W6: connect, then load
|
||||
onClose((ctx) => worker.close(ctx));
|
||||
worker.connect(parent, { principal: { kind: "server" }, trusted: true }, ctx); // `started` goes out now: connect
|
||||
// before anything slow (a store opens in ms)
|
||||
await plugins.load(worker, { kind: "worker", core: [{ name: "@app/core", facet: coreWorker(start) }] }, ctx);
|
||||
return { host: worker, busy: harness.activeTasks, close: (ctx) => store.close(ctx) };
|
||||
}, { onError: (error) => log(redact(error)) });
|
||||
return { busy: harness.activeTasks }; // extra keep-alive counts
|
||||
}, {
|
||||
hostPackages: { "@earendil-works/chord/next": chordNext, "@app/core": appCore }, // the child's own instances
|
||||
redact: (error) => ({ message: redactSecrets(String(error)), ...codeOf(error) }), // what crosses as `fatal`
|
||||
onError: (error) => log(error),
|
||||
});
|
||||
```
|
||||
|
||||
- **The route:** `createWorkers` calls `host.route(Kind, { authorize, resolve }, ctx)` itself and keeps the
|
||||
`RouteHandle` (`retry`, `suspend`: D10b, R1).
|
||||
- **A start** (`resolve(address)`): one queue per address orders starts, stops and exits (a start waits for the
|
||||
previous child's exit; a stop during a start waits for it, then stops; an aborted start kills its child). A start
|
||||
calls `start(address, ctx)` (a throw rejects the start), forks `entry` (`serialization: "advanced"`, a detached
|
||||
process group, stdout and stderr piped: the stderr tail kept, mirrored to the server's when there is no `onOutput`),
|
||||
makes `ipcLink(child)` at once (so the child's `hello` and early holds are queued, never lost), and sends the start
|
||||
message `{ address, start, scripts, lock }` (`scripts`: the worker kind's complete current scripts, below). The child
|
||||
takes the lock (waiting for a draining old worker to let it go, within the start timeout), connects and sends
|
||||
`started`; `resolve` then returns `{ link, principal: principal(address), trusted: true }`, so the handshake and the
|
||||
worker's server traffic run during its `load`. `ready` (after the app's function returned) ends the start timer (60
|
||||
s from the fork; past it the tree is killed and the start fails).
|
||||
- **Workers never restart by themselves** (R2): a crash (an exit nobody asked for) or a failed start (a `start` that
|
||||
threw, a lock not taken in time, `fatal`, the start timer) leaves the address `failed` with its error
|
||||
(`worker_failed` reported once); `resolve` of a failed address rejects at once with that error, so its holders and
|
||||
new holds see `failed`, and nothing forks. `workers.retry(address)` (`RouteHandle.retry`) or a rollout that reaches
|
||||
workers clears it. An idle stop or `stop` is no failure: the next hold starts it on demand.
|
||||
- **Keep-alive** (D9, R3): the child watches `host.activity` and the app's `busy` (a JSON watchable of numbers) and
|
||||
sends `{ idle: true }` when every count becomes zero, `{ idle: false }` when one becomes nonzero; a rollout applying in
|
||||
the child counts as busy. The server stops a worker idle since `idleMs()` (5 min by default; checked every second),
|
||||
never while it starts. A stop (idle, `stop`, `close`): `shutdown`; the child closes its host, runs the app's `close`,
|
||||
exits; it forces its own exit after 3 s; the server kills the tree after 5 s.
|
||||
`RouteHandle` (`retry`, `suspend`: D10b, R1). `authorize` is synchronous (the kernel's): a plugin backing it keeps an
|
||||
index.
|
||||
- **Per address, events and operations:** a child's events (its exit, its link's close, `started`, `ready`, `fatal`)
|
||||
are observed as they happen and recorded at once on that child (a late event of an old child never changes a new
|
||||
one); the operations (start, stop) run in one queue per address: a start waits for the previous child's exit, a
|
||||
stop during a start waits for it then stops, and a start queued behind an exit checks the address's state after it
|
||||
(a crash recorded meanwhile makes it reject with that failure). Stops record their intent before `shutdown`, so the
|
||||
exit they cause is no crash.
|
||||
- **A start** (`resolve(address)`): calls `start(address, ctx)` (abortable; a throw rejects the start), checks that the
|
||||
component is not closing, forks `entry` (`serialization: "advanced"`, a detached process group, stdout and stderr
|
||||
piped: the stderr tail kept, mirrored to the server's when there is no `onOutput`), makes `ipcLink(child)` at once
|
||||
(so the child's `hello` and early holds are queued, never lost), and sends the start message `{ address, start,
|
||||
scripts, lock }` (`scripts`: the worker kind's complete current set, read when the message is sent). The child takes
|
||||
the lock (when an old worker still holds it, it waits by connecting to the lock's socket and trying again when that
|
||||
connection closes: no polling), and sends `started` when the app's host sets its listener on `parent` (`parent` is
|
||||
a thin wrapper of the IPC link, so `connect` is the signal: the app connects before anything slow). `resolve` then
|
||||
returns `{ link, principal: principal(address), trusted: true }`, and the handshake and the worker's server traffic
|
||||
run during its `load`. The start timer (60 s from the fork) covers fork, lock and connect only (Q4): past it the tree is
|
||||
killed and the start fails. `ready` (after the app's function returned, carrying the idle state) sets `state` to
|
||||
`running` and lets idle stops begin; a `load` that never ends shows `starting` until `stop`.
|
||||
- **Workers never restart by themselves** (R2): a crash (an exit nobody asked for), a `fatal`, a link lost while its
|
||||
child lives, or a failed start (`start` threw, the start timer, a lock not taken in time) leaves the address `failed`
|
||||
with its error (`worker_failed` reported once). A link lost while its child lives (the heartbeat after a blocked
|
||||
event loop, a protocol error) starts the stop: the grace (5 s), then the tree is killed; its exit is the crash ("link
|
||||
lost"). `resolve` of a failed address rejects at once with that error; its holders and new holds see `failed {
|
||||
waiting }` with it as cause (Q1) and stay on the address's path, nothing forks. `workers.retry(address)`
|
||||
(`RouteHandle.retry`) or a rollout reaching workers clears it (Q2, §4). An idle stop is no failure: the next hold
|
||||
starts the worker on demand.
|
||||
- **Keep-alive** (D9, R3, Q5): the child's idle state is all of `host.activity`'s `references`, `jobs` and `lingering`
|
||||
and the app's `busy` (a JSON watchable of numbers) at zero, not `calls` (a remote call runs on a hold; local calls
|
||||
come from routines, jobs and tasks counted elsewhere). `ready { idle }` carries the first state, then `{ idle }` on
|
||||
each flip; a rollout applying in the child counts as busy. The server stops a worker idle since `idleMs()` (5 min by
|
||||
default; checked every second), never before `ready`. A stop (idle, `stop`, `close`): `shutdown`; the child aborts its
|
||||
`ctx`, runs the closers registered with `onClose` in reverse order and exits, forcing its own exit after 3 s; the
|
||||
server kills the tree after 5 s.
|
||||
- **`state`** (R5): per address key `{ status: "starting" | "running" | "stopping", pid, startedAt }` or `{ status:
|
||||
"failed", error: { message, code?, reason? }, at, stderrTail? }`; no entry when stopped or never started.
|
||||
- **`stop(address, ctx)`:** `RouteHandle.suspend(address)` (holders stay `loading`, nothing resolves it), then the
|
||||
stop; `retry(address)` lifts the suspension. **`close(ctx)`:** new starts reject `unavailable/closed`, every child is
|
||||
stopped (grace, then kill), the timers cleared; `createWorkers`'s `ctx` aborting closes it too.
|
||||
- **The child** (`workerMain(fn, { onError })`): reads the start message, takes the lock, makes `ipcLink(process)`,
|
||||
calls `fn({ address, start, parent, plugins }, ctx)`; `started` once connected, `ready` once `fn` returned; reports
|
||||
`idle` flips; handles `shutdown`, the parent's disconnect, `SIGTERM` and `SIGINT` alike (close the host, the app's
|
||||
`close`, exit); an uncaught error is sent as `fatal { message, code?, reason? }` through `onError` (the app redacts),
|
||||
then the child exits.
|
||||
- **`stop(address, ctx)`** (R1, Q3): `RouteHandle.suspend(address)`, then the stop. The suspension lasts while the
|
||||
address has holders (they see `failed { waiting }`, "suspended"); with the last holder it goes, and a later hold
|
||||
starts the worker on demand; `retry(address)` lifts it at once. Whether an archived session may be opened is the
|
||||
app's (`authorize`, its views).
|
||||
- **`close(ctx)`:** cancels active and queued starts first (a `start(address)` waiting on the app's barrier is
|
||||
aborted; nothing forks after it), then stops every child (grace, then kill), clears the timers; new starts reject
|
||||
`released/closed`; the caller's cancellation ends only its own wait. `createWorkers`'s `ctx` aborting (the host
|
||||
closing aborts the route's context) closes it too, and from then on every exit is a stop, never a crash (no
|
||||
`worker_failed` at shutdown).
|
||||
- **The child** (`workerMain(fn, { hostPackages, redact, onError })`): reads the start message, takes the lock, makes
|
||||
`ipcLink(process)` and its wrapper `parent`, creates its plugin runtime with `hostPackages` (the child's own module
|
||||
instances; functions cannot cross IPC), calls `fn({ address, start, parent, plugins, onClose }, ctx)`. `shutdown`,
|
||||
the parent's disconnect, `SIGTERM` and `SIGINT` alike abort `ctx` and run the closers registered so far, also while
|
||||
`fn` still runs (its startup is cancelled through `ctx`). An uncaught error, or `fn` throwing, sends `fatal` with
|
||||
`redact(error)` (never the original message) and exits; after `started` that is a crash like any other.
|
||||
- **Control messages** (JSON beside the link's `Uint8Array` frames; the IPC link skips them): parent to child `start`,
|
||||
`shutdown`, `reload { scripts }`; child to parent `started`, `ready`, `idle { idle }`, `fatal { error }`, `reloaded
|
||||
{ report } | { error }`. Errors cross as `{ message, code?, reason? }` (a structured clone keeps no `ChordError`).
|
||||
`shutdown`, `reload { scripts }`; child to parent `started`, `ready { idle }`, `idle { idle }`, `fatal { error }`,
|
||||
`reloaded { report } | { error }`. One `reload` is outstanding per child at a time: a reload whose caller aborted
|
||||
keeps the child's slot until its `reloaded` or the child's exit, and the next waits for it. Errors cross as `{
|
||||
message, code?, reason? }`.
|
||||
- **Errors** (`onError`, `WorkersError`): `worker_failed { address, error, stderrTail? }`, `worker_exited { address,
|
||||
code, signal }` (a clean exit after a stop is not reported).
|
||||
|
||||
|
|
@ -213,8 +261,9 @@ Workers never build. The plugins component (7c) gets `scripts(kind, ctx)`: the c
|
|||
discovered plugin's facet of it, built on first use, then updated by every rollout that ships that kind), each with its
|
||||
plugin record (`name`, `dir`, `chord.external`). The component's per-host loading (its loaded map, `#prepare`, `adopt`,
|
||||
the queue per host) moves into a file without esbuild or the watcher, `createPluginRuntime({ hostPackages, onError })`
|
||||
with `load(host, { kind, core, scripts }, ctx)` and `reload(host, shipped, ctx)`; `createPlugins` uses it, and the
|
||||
child's `plugins` is one (`import` of `chord/next/plugins/runtime` loads no esbuild).
|
||||
with `load(host, { kind, core, scripts }, ctx)` and `reload(host, scripts, ctx)` (a complete set: removed = loaded −
|
||||
shipped; same digests unchanged); `createPlugins` uses it, and the child's `plugins` is one (`import` of
|
||||
`chord/next/plugins/runtime` loads no esbuild).
|
||||
|
||||
## 4. The rollout across server and workers
|
||||
|
||||
|
|
@ -222,17 +271,25 @@ child's `plugins` is one (`import` of `chord/next/plugins/runtime` loads no esbu
|
|||
const built = await plugins.build(plugins.state.value.dirty, ctx);
|
||||
const result = await plugins.rollout([plugins.step("server", server), workers.step("workers")], built, ctx);
|
||||
// result.steps: [{ name: "server", hosts: [{ host: "server", report }] },
|
||||
// { name: "workers", hosts: [{ host: "session|S1", report }, { host: "session|S2", error }] }]
|
||||
// { name: "workers", hosts: [{ host: "session|S1", report }, { host: "session|S2", stopped: true },
|
||||
// { host: "session|S3", error }] }]
|
||||
```
|
||||
|
||||
`RolloutStep.reload(built, ctx)` returns `{ hosts: readonly ({ host: string; report: ReloadReport } | { host: string;
|
||||
error: { message, code?, reason? } })[] }` (D12; 7c's in-process step returns a list of one, its `host` the name given to
|
||||
`step`). `workers.step(name)` first replaces the workers' current set (`plugins.scripts`), then ships the change to
|
||||
every worker whose start message or last reload carried an older set, starting ones included (a child queues a reload
|
||||
that arrives before its `ready` until its load is done); workers reload in parallel; one that closed or failed during
|
||||
the step is listed with its error and does not stop the rollout, and every plugin of its kind in the build counts as
|
||||
failed there (stays dirty). A rollout that reaches workers clears every failed address (R2). A plugin's dirty mark
|
||||
clears only when it applied on every host of every step.
|
||||
`RolloutStep.reload(built, ctx)` returns `{ hosts: readonly ({ host; report } | { host; stopped: true } | { host; error
|
||||
})[] }` (D12; 7c's in-process step returns a list of one, its `host` the name given to `step`). `workers.step(name)`:
|
||||
|
||||
1. replaces the workers' current set (`plugins.scripts(kind)` updated from `built`);
|
||||
2. clears the failures recorded before this rollout and calls `RouteHandle.retry` for those addresses that have holders
|
||||
(Q2): they start with the new set and are listed; an unheld one starts at its next hold;
|
||||
3. ships the complete set to every running or starting worker whose start message or last reload carried an older one
|
||||
(a child queues a reload that arrives before its `ready` until its load is done); workers reload in parallel.
|
||||
|
||||
A worker's entry settles on its `reloaded` or on its exit or link loss, whichever comes first. A worker stopped cleanly
|
||||
during the step (idle, `stop`) is listed `stopped` and dirties nothing (its next start gets the current set); one that
|
||||
crashed or failed to apply is listed with its error and does not stop the rollout, and every plugin of its kind in the
|
||||
build counts as failed there (stays dirty); a worker failing during this rollout stays failed. A plugin's dirty mark
|
||||
clears only when it applied on every host of every step, and a kind built in the rollout that no step shipped keeps its
|
||||
plugins dirty.
|
||||
|
||||
## 5. Not in slice 10
|
||||
|
||||
|
|
@ -243,34 +300,48 @@ channel component (D6); telemetry spans for requests (slice 9).
|
|||
|
||||
## 6. Kernel changes
|
||||
|
||||
- `host.route(Kind, options, ctx)` returns a `RouteHandle { retry(address), suspend(address) }` (D10b, R1).
|
||||
- A route has no backoff timer (R2): a rejected `resolve` or a link closed at once fails the route's holders with its
|
||||
error (`failed`, not `loading`); the route resolves again only for a new hold, on `retry`, or once after the
|
||||
address's link dropped while holds exist; a suspended address resolves nothing until `retry`.
|
||||
- Design §3's paragraph on a `resolve` waiting for the worker's `load` (a deadlock) is updated: 7a's barrier ends its
|
||||
first pass on a wait on another host, and `resolve` returns the link before `load` anyway.
|
||||
- `host.route(Kind, options, ctx)` returns a `RouteHandle { retry(address), suspend(address) }` (D10b, R1, Q3).
|
||||
- A route has no backoff timer (R2): a rejected `resolve`, or a link that closes before its handshake completed, fails
|
||||
the route's holders `failed { waiting }` with its error (they stay on the address's path, Q1) and resolves nothing
|
||||
more; the route resolves again only for a new hold, on `retry`, or once after a link that had completed its handshake
|
||||
dropped while holds exist; a suspended address resolves nothing while it has holders and forgets the suspension with
|
||||
the last one.
|
||||
- `host.root(fn, ctx, { principal: undefined })` runs anonymous (`ctx.principal` undefined); only an omitted
|
||||
`principal` takes the host's (a public-API change of `RootOptions`: `"principal" in options`).
|
||||
- Design §3: the paragraph on a `resolve` waiting for the worker's `load` is updated (done), and "Chord never closes a
|
||||
route link itself except at `host.close`" becomes "except at `host.close`, the heartbeat and protocol errors".
|
||||
|
||||
## 7. Tests, examples, cost
|
||||
|
||||
- One test file per source file. HTTP: listening before the publication (a request waits, then is served; 503 past
|
||||
`readyTimeout`); `Host` and `Origin` checks, allowed hosts from the option and from a writable contribution changing
|
||||
at runtime; authentication per request and upgrade; patterns, decoding, precedence, `HEAD`, 404; conflicts reported;
|
||||
wraps (phases, order, scopes, a missing route, a throwing wrap); every response kind (ranges 206/416, multi-range,
|
||||
`HEAD`, download names, chunked without `size`, a short `read`, a disconnect stopping `read`, header merging);
|
||||
`json`/`body` limits and their statuses, an unread body; a throwing route; the links endpoint (`host.accept`, a
|
||||
refused socket, the output cap); plugin sockets (an oversized frame closes only that socket); a reload swapping a
|
||||
route while a request runs on the old router; `fileResponse` closing its handle on every path; `close`. Workers: a
|
||||
start on the first hold; the link made at the fork (a load longer than the heartbeat's 30 s binds); the lock (a
|
||||
draining old worker waited for); the start timer; `start` throwing; a crash and a failed start staying failed (no
|
||||
fork for a new hold), `retry` and a rollout clearing it; idle stop and a hold keeping it; `busy`; the `idle` flips (no
|
||||
message per call); `stop` with holders (suspended: no restart); stop during start and start during stop; `close`;
|
||||
the rollout step (a worker starting during it gets the new set; a failed worker listed); the child's shutdown,
|
||||
disconnect and signals. Kernel: `RouteHandle.retry` and `suspend`; no re-resolve without a hold, a retry or one drop.
|
||||
`readyTimeout`); the router resolved once; `Host` (loopback forms, allowed from the option and from a writable
|
||||
contribution changing at runtime) and `Origin` checks; anonymous requests (`ctx.principal` undefined on a server with
|
||||
a principal); authentication per request and upgrade; throwing hooks; patterns, decoding, precedence, `HEAD`, 404;
|
||||
conflicts reported; wraps (phases, order, scopes, a missing route, a throwing wrap); every response kind (ranges
|
||||
206/416, multi-range, `HEAD`, download names, chunked without `size`, a short `read`, a disconnect stopping `read`,
|
||||
header merging); `json`/`body` limits and their statuses, an unread body; a throwing route; the links endpoint
|
||||
(`host.accept`, options per socket, a refused socket, the output cap); plugin sockets (an oversized frame closes only
|
||||
that socket; a reload removing the contribution closes its sockets 1012); a removed allowed host closing its
|
||||
connections; a reload swapping a route while a request runs on the old router; `fileResponse` closing its handle on
|
||||
every path; `close` (refusals, waiting, then sealing). Workers: a start on the first hold; the link made at the fork
|
||||
(a load longer than the heartbeat's 30 s binds); `started` on `connect`; the start timer covering only the start (a
|
||||
slow `load` is not killed); the lock (an old worker's lock waited for without polling); `start` throwing; a crash,
|
||||
a `fatal`, a link lost with a live child (killed after the grace), and a failed start each staying failed (no fork for
|
||||
a new hold; holders `failed { waiting }`), `retry` and a rollout clearing them and rebinding the holders; idle stop,
|
||||
a worker idle from its start, a hold keeping it; `busy`; 1,000 sequential local calls sending no message;
|
||||
`stop` with holders (suspended, then forgotten with the last holder); stop during start and start during stop; a
|
||||
crash while a resolve is queued; `close` (cancelling a start waiting in `start(address)`, no `worker_failed` at
|
||||
shutdown); the rollout step (complete sets, a worker starting during it, a clean stop listed `stopped`, a crash during
|
||||
it, an aborted reload's late reply, a kind no step shipped staying dirty); the child's shutdown during its startup,
|
||||
disconnect and signals; `redact` (a secret in the original message reaches neither `fatal` nor `state`). Kernel:
|
||||
`RouteHandle.retry` and `suspend`; a link closing before its handshake (no re-resolve loop); no re-resolve without a
|
||||
hold, a retry or one drop of a healthy link; `host.root` with an explicit `undefined` principal.
|
||||
- GC: requests (their scopes, bodies, file handles) and stopped workers' state collected; no listener left on a closed
|
||||
socket; 1,000 addresses started and stopped leave no `state` entries.
|
||||
socket; 1,000 addresses started and stopped leave no `state` entries, no per-address queue records and no
|
||||
suspensions.
|
||||
- Examples (each a folder, as 7c's): `40-http` (routes, a wrap, a download with ranges, a socket, an allowed host
|
||||
added at runtime), `41-workers` (a server routing a session kind to forked workers, idle stop, a crash staying failed
|
||||
until Retry, a rollout reaching the workers).
|
||||
until Retry, a rollout reaching the workers and restarting a held failed one).
|
||||
- Benchmark: a route's overhead per request against a plain Node handler (bar: within 10% of a keep-alive round
|
||||
trip; Opus measured 2.5 µs of 54 µs); `bytes` throughput for a 100 MiB file against `pipeline`; a worker's cold start
|
||||
(fork to `ready`) with shipped scripts; memory per idle worker on the server.
|
||||
(fork to `started` and to `ready`) with shipped scripts; memory per idle worker on the server.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue