From f8497996fec24134f159f9e046849427e1175a30 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Fri, 30 Oct 2026 21:00:20 +0000 Subject: [PATCH] docs(chord): slice 10 spec draft, HTTP points D1-D8 --- packages/chord/docs/next/decisions.md | 30 ++++ packages/chord/docs/next/slice-10.md | 231 ++++++++++++++++++++++++++ 2 files changed, 261 insertions(+) create mode 100644 packages/chord/docs/next/slice-10.md diff --git a/packages/chord/docs/next/decisions.md b/packages/chord/docs/next/decisions.md index 6e2b982b6..e6034d216 100644 --- a/packages/chord/docs/next/decisions.md +++ b/packages/chord/docs/next/decisions.md @@ -1877,6 +1877,36 @@ validation and copies the v2 kernel added around Delta. reload, swapping the committed plugin back silently. Rejected: resolving with the report after the commit despite the abort (it would change the rule that an abort after the commit rejects). Examples for 7c (owner): full coverage, each example a folder with the host code and real plugin directories (35-39). +- **Slice 10's points (owner, 2026-10-30; `slice-10.md`):** (D1, "a") the HTTP component defines the registry `Http` + (`"chord.http"`) and serves it through a facet of its own (`http.facet`, plugin name `chord.http`) that the app loads + with its plugins; its `compose` builds the router; the server reads it with `host.registry(Http, ctx)` per request. + (D2, "a") Wraps are built in slice 10 as designed (decision "HTTP routes and wraps", design §23): a contribution's + `wraps` run around routes in the component's phases (`observe`, `headers`, `transform`, outermost first), within a + phase in contributor order, by default around their own plugin's routes (`scope: "own"`; `"all"` explicit; `route` + narrows to one, a missing one a reported conflict). The server's `headers` option stays for every response, also + those no route makes (a 404, a refused request). + (D3, "a") Authentication is one app function, `authenticate(request, ctx) => P | undefined`, called once per request + and per upgrade; no Chord registry of resolvers (p3 composes its own from sharing, devices and the relay, with its + own precedence rules). + (D4, "k": a) Request bodies: `request.json(limit?)` (parsed JSON, 1 MiB by default, malformed: 400) and + `request.body(limit?)` (an async iterable of byte chunks, for uploads); nothing is read until a route asks; past its + limit the read stops and the request is answered 413; a disconnect aborts `ctx` and ends the iteration. + (D5, "a") Responses are descriptions: `json`, `html` (a full document; replaces the design's `page { title, body }`), + `bytes` (`mime`, `size`, `read(offset, length, ctx)`, `download?`, `ranges?`; 206/416, `HEAD`, safe + `Content-Disposition`, 64 KiB reads with backpressure, stopped on disconnect), `redirect`, `error`; `headers` on every + kind (cookies, a preview's own CSP); `fileResponse(path, options)` over a file; a throwing route is a 500 without its + message. + (D6, "a") Plugin WebSockets are `sockets` in the `Http` contribution: `{ "/terminal": (socket, request, ctx) => … }`; + the component checks the origin and runs `authenticate` (no principal: refused, 403, before plugin code), upgrades + and hands the plugin a `ws` WebSocket for its own protocol; patterns as routes; wraps do not apply; the links path is + the component's (`host.accept`). The design's channel component (acknowledged byte channels) waits for a second user + and would be built on top of it. + (D7, "a") `ws` is an optional peer dependency of Chord (pinned exactly), imported only by `next-node/http`, as + esbuild for plugins. + (D8, "a") Route patterns: exact segments, `:name` (one non-empty segment, percent-decoded, `params.name`) and a + trailing `/*` (the rest, possibly empty, `params["*"]`); the most specific match wins, segment by segment from the + left (exact before `:name` before `*`); one pattern from two contributions is a reported conflict, the first by + plugin name wins; `HEAD` falls back to the `GET` route. - **A plugin's npm dependencies are bundled; external only by opt-in (owner, 2026-10-29: "explicit opt in + (a)"):** a plugin's `dependencies` are installed into its own `node_modules` (`npm install --ignore-scripts`) and bundled into each facet that imports them (host packages, its `peerDependencies`, stay external and are passed in). On Node a diff --git a/packages/chord/docs/next/slice-10.md b/packages/chord/docs/next/slice-10.md new file mode 100644 index 000000000..ce1f56e13 --- /dev/null +++ b/packages/chord/docs/next/slice-10.md @@ -0,0 +1,231 @@ +# Slice 10: HTTP and workers (spec) + +Status: **draft** (owner, 2026-10-30: "do it"; the points marked **Decision needed** go to the owner one at a time). +Slice 10 builds two Node components of the app kit (decision "Slice 7b's open points" (6)): `chord/next/http` (the +server's HTTP and WebSocket entry, routes contributed by plugins through a registry, responses as descriptions) and +`chord/next/workers` (a worker process per host address, started on demand, stopped when idle, its plugins shipped from +the server). Both live in `packages/chord/src/next-node/` (Node-only); the kernel stays portable. The web page host and +the TUI are slice 11, telemetry slice 9. + +Read in full before writing code: this spec; `decisions.md`; `design.md` (§3 hosts, routes, `host.activity`, §17, +§18, §21 "Loading plugins and the reload loop", §23 in full); `slices.md`; `slice-7c.md` and +`src/next-node/plugins/README.md`; the 7b brief `slice-7b/book.md` (chapters "Assets and Files", "HTTP", "A Mini p2 +from Chord"); p2's `src/hosts/server.ts`, `src/runtime/server.ts`, `src/hosts/worker.ts`, `src/hosts/socket-limits.ts`, +`src/hosts/principals.ts`, `plugins/files/server.ts`, `plugins/terminals/server.ts`, `plugins/radius/server.ts`; Chord +v1's `src/manager.ts` (`processLock`, `forkDriver`, `createHostManager`, `childControl`). + +## 1. What p2 has, and what moves into Chord + +p2's server host (`src/hosts/server.ts`, 742 lines; `src/runtime/server.ts`, 720) does, besides p2's own policy: + +| p2 today | Moves to | +| --- | --- | +| a Node `http` server on loopback, requests parked until `load` resolved, upgrades captured before listening | `http` | +| `Routes` (`"METHOD /path"`, `/prefix/*`), handlers writing Node's `ServerResponse` themselves (ranges, download headers, `HEAD` by hand in `files`) | `http`: routes returning descriptions | +| `Upgrades` (a WebSocket by path for terminals and voice), `/chord` (`acceptWebSockets` with per-socket limits) | `http` | +| `Authenticators`, `RequestResolvers` (cookies, devices, the Radius relay), loopback and allowed hosts, Fetch Metadata, CSP and the loopback cookie | the app (p3's base code), through `http`'s one `authenticate` hook (D3) | +| `SessionHosts`: a worker per session, `createHostManager` + `forkDriver`, a kernel lock per session, idle stop from reported counts, crash notices, ready timeout | `workers` | +| relocation and pause of a worker, the catalog, `WorkerInitValues` | the app (built on `workers`' `stop` and start data) | +| `hosts/worker.ts`: the child's control channel, lock, start data, connect, load, report counts, shutdown | `workers`' child side | +| the dev reload across server and workers, worker plugin lists | `plugins` (7c) plus `workers`' rollout step (§4) | + +## 2. `chord/next/http` + +```ts +const http = await httpServer(server, { + listen: { host: "127.0.0.1", port: 8788 }, + authenticate: (request, ctx) => app.principalOf(request, ctx), // the app's policy: cookies, loopback, relays + headers: (request) => app.securityHeaders(request), // every response's headers (CSP, nosniff, framing) + links: { path: "/chord", limits: { part: 16 * MiB, decoded: 64 * MiB } }, // host.accept for every socket there +}, ctx); +await server.load([{ name: "@app/core", facet: core }, { name: "chord.http", facet: http.facet }, ...plugins], ctx); +… +await http.close(ctx); +``` + +- **Order:** the server listens first (plugins see the real port; a page reconnecting while plugins load is not lost), + holds every request and upgrade until `load` resolved, then serves. +- **`Http`**, a registry the component defines and serves through its facet (D1): plugins contribute routes and + sockets; the composed router is followed, each request uses the router as it was when it started. + +```ts +// plugins/notes/server.ts +export default defineFacet((host, self) => { + self.contribute(Http, { + routes: { + "GET /notes/:id/export": async (request, ctx) => { + if (ctx.principal?.kind !== "owner") return { kind: "error", status: 403 }; + const notes = await ctx.get(NoteStore, {}); + return { kind: "json", value: await notes.export({ id: request.params.id }, ctx) }; + }, + "GET /notes/template.md": async (_request, ctx) => fileResponse(pluginFile("template.md"), { download: "template.md" }), + }, + sockets: { "/terminal": (socket, request, ctx) => terminals.attach(socket, ctx) }, // D6 + }); +}); +``` + +- **A route** is `(request: HttpRequest, ctx: Ctx) => Promise`, run in a `host.root` scope with the + request's principal (`ctx.principal`; `undefined` when `authenticate` found none) and an abort signal that fires when + the client disconnects. `HttpRequest`: `method`, `path`, `params` (from the pattern), `query` + (`URLSearchParams`), `headers` (lowercased, read-only), `origin` (scheme, host, port), and the body (D4). +- **Patterns** (D8): `"METHOD /a/b"`, `:name` segments (`params.name`), a trailing `/*` (`params["*"]`, the rest); + the most specific match wins, segment by segment from the left (exact before `:name` before `*`); two contributions with + one pattern are a conflict reported through `compose` (`registry_conflict`), the first by plugin name wins. `HEAD` + falls back to the `GET` route. +- **Responses are descriptions** (D5): the component writes status, headers and body, so every plugin gets ranges, + `HEAD`, download headers and abort on disconnect from one place: + +```ts +type HttpResponse = { readonly headers?: Readonly> } & ( + | { readonly kind: "json"; readonly value: JsonValue; readonly status?: number } + | { readonly kind: "html"; readonly body: string; readonly status?: number } // its CSP via `headers` + | { readonly kind: "bytes"; readonly mime: string; readonly size: number; + readonly read: (offset: number, length: number, ctx: Ctx) => Promise; + readonly download?: string; // Content-Disposition: attachment with this file name, made safe + readonly ranges?: boolean } // 206/416 and Accept-Ranges + | { readonly kind: "redirect"; readonly to: string; readonly status?: 302 | 303 | 307 } + | { readonly kind: "error"; readonly status: number; readonly message?: string }); +``` + + `bytes` is read in chunks (64 KiB) with the socket's backpressure, stopped when the client disconnects; a route + that throws is a 500 (reported, never its message); one that returns after a disconnect writes nothing. +- **`fileResponse(path, { download?, ranges?, mime? })`**: a `bytes` response over a file (its size from `stat`, `read` through a file + handle opened once and closed when the response ends), for plugin files and downloads. +- **The links endpoint** (`links.path`): an upgrade there is authenticated, then `host.accept(webSocketLink(ws), { + principal, limits })`; a socket without a principal is refused (403). Per-socket output cap as p2's + `limitedSocket` (a client that stops reading is cut off). +- **Wraps** (D2): middleware around routes, below. **Not built:** request resolvers as a registry (D3). + +> **Decided D1 (owner, 2026-10-30: "a"):** the component exports the token `Http` (`"chord.http"`) and a facet the +> app loads with its plugins (`http.facet`, plugin name `chord.http`), which serves the registry (`compose`: the +> router); the server reads it with `host.registry(Http, ctx)`. + +> **Decided D2 (owner, 2026-10-30: "a"):** wraps are built as designed (§23): `wraps: [{ name, phase, scope?, route?, +> wrap(request, next, ctx) }]` in the component's phases `observe`, `headers`, `transform` (outermost first), within a +> phase in contributor order (plugin name, then type), `scope: "own"` by default (around the contributing plugin's own +> routes), `"all"` explicit, `route` narrowing to one pattern (a missing one is a reported conflict). A wrap adds a +> boundary on top of a route's own checks, never instead of them. A wrap that throws is a 500 (reported, naming its +> plugin). The `headers` option stays for every response, also those no route makes (a 404, a refused request). + +> **Decided D3 (owner, 2026-10-30: "a"):** one app function `authenticate(request, ctx) => P | undefined`, called +> once per request and per upgrade; no Chord registry of resolvers: who a request is, and which credential wins, is the +> app's policy (p3 composes it from its own registry). + +> **Decided D4 (owner, 2026-10-30: "k"):** `request.json(limit?)` (parsed JSON, at most 1 MiB by default, malformed: +> 400) and `request.body(limit?)` (an async iterable of `Uint8Array` chunks, for uploads streamed to a file); nothing is +> read before the route asks; past its limit the read stops and the request is answered 413; a client that disconnects +> aborts `ctx`, which ends the iteration. + +> **Decided D5 (owner, 2026-10-30: "a"):** the five kinds above, `headers` on every kind (cookies, a preview's own +> CSP), `html` a full document instead of the design's `page { title, body }`. + +> **Decided D6 (owner, 2026-10-30: "a"):** `sockets: { "/terminal": (socket, request, ctx) => … }`: after the +> origin check and `authenticate` (a socket without a principal is refused, 403, before the plugin sees it), the +> component upgrades and hands the plugin a `ws` WebSocket (its own protocol and flow control, as p2's terminals and +> voice); the plugin authorizes further; patterns as routes; wraps do not apply. The design's channel component waits +> for a second user and would be built on top of this. + +> **Decided D7 (owner, 2026-10-30: "a"):** `ws` is an optional peer dependency of Chord (pinned exactly, the version +> its tests use), imported only by `next-node/http`, as esbuild. + +> **Decided D8 (owner, 2026-10-30: "a"):** `:name` matches one non-empty segment (percent-decoded), a trailing `/*` +> the rest (possibly empty); the most specific match wins, segment by segment from the left (exact before `:name` +> before `*`). + +## 3. `chord/next/workers` + +```ts +// server.ts +const workers = createWorkers({ + entry: "./worker.ts", // the child's entry module + start: (address, ctx) => app.startData(address, ctx), // the app's start data (JSON), before each start + plugins, // the plugins component: the worker kind's scripts + idleMs: 300_000, + onOutput: (address, stream, chunk) => logs.append(address.key, stream, chunk), +}, ctx); +server.route(Session, { authorize: app.authorizeSession, resolve: workers.resolve }, ctx); +workers.state // JSON watchable: { [address key]: { status, pid?, startedAt?, error? } } +await workers.stop(Session({ id }), ctx); // archive; the next hold starts it again +``` + +```ts +// worker.ts +await workerMain(async ({ address, start, parent, plugins }, ctx) => { + const worker = createHost

({ address, principal: { kind: "worker", session: start.id }, store, onError }); + worker.connect(parent, { principal: { kind: "server" }, trusted: true }, ctx); // W6: connect, then load + await plugins.load(worker, { kind: "worker", core: [{ name: "@app/core", facet: coreWorker(start) }] }, ctx); + return { host: worker, busy: harness.activeTasks }; // extra keep-alive counts (Durable tasks) +}); +``` + +- **`resolve(address, ctx)`** (a route's `resolve`): the address's running worker's link, or starts one: forks `entry` + (`serialization: "advanced"`, detached process group, killed as a tree), sends the start message (`address`, the + app's start data, the worker kind's built scripts), waits for `ready` within `readyTimeout` (60 s), and returns + `{ link: ipcLink(child), principal, trusted: true }`. One start at a time per address; a failed start rejects (the + route's backoff and `resolve_failed` are the kernel's: D10). +- **One live worker per address** across processes (a server restart while an old worker drains): the child takes a + kernel lock named after the app's data and the address before it opens its store (v1's `processLock`, ported); a + second fails its start `unavailable/locked`. +- **Keep-alive** (D9): the child reports `host.activity` (`references`, `calls`, `jobs`, `lingering`) and the app's + `busy` counts whenever they change; the server stops a worker whose counts were all zero for `idleMs`, and never one + that is starting. A stop sends `shutdown`, the child closes its host (`host.close`) and exits; after a grace period + (5 s) the process tree is killed. +- **Exits:** a worker that exits unasked (a crash, an OOM) is reported (`worker_exited { address, code, signal, stderr + tail }`) and its link closes; its holders go `loading` and the route resolves again while holds exist (the kernel's + backoff). `state` shows the last error until the next start succeeds. +- **The child** (`workerMain`): reads the start message, takes the lock, makes `ipcLink(process)`, calls the app's + function, sends `ready` once it returned (after `load`), reports counts, handles `shutdown` and the parent's + disconnect (close the host, exit), and reports an uncaught error as `fatal` (redacted by the app's `onError`). +- **Plugins in workers** (D11): workers never run esbuild. The start message carries the worker kind's current scripts + (`Built`, plain data, 7c); the child's `plugins` is the evaluating half of the plugins component (no build, no + watcher, no esbuild import): `load` evaluates them, `reload` evaluates new ones and calls `host.reload` per plugin, + as on the server. + +## 4. The rollout across server and workers + +```ts +const built = await plugins.build(plugins.state.value.dirty, ctx); +const result = await plugins.rollout([plugins.step("server", server), workers.step("workers")], built, ctx); +``` + +`workers.step(name)` ships `built` (its worker kind's facets and plugin records) to every running worker over its +control channel; each evaluates and reloads; the step resolves with one report per worker (`RolloutStep.reload` +returns `{ hosts: [{ host, report }] }`, D12). A worker starting during or after it gets the current scripts in its +start message. A worker that fails to apply (closed, aborted) is listed and does not stop the rollout; a plugin kept old +on any worker stays dirty (7c's rule). + +> **Decision needed D9: keep-alive inputs.** Chord's `host.activity` plus app counts (`busy`: a JSON watchable of +> numbers, e.g. Durable's active tasks) reported to the server; `idleMs` default 5 min; checked every second. +> Recommended (p2's rule, with Chord's counts instead of p2's own). + +> **Decision needed D10: failed starts.** p2 keeps a failed start sticky until the owner presses Retry; with Chord's +> route backoff a failed start is retried after 1 s, doubling to 60 s, while holds exist. Recommended: the kernel's +> backoff (no sticky error); `state` shows the error, and `workers.retry(address)` skips the backoff (a Retry button). + +> **Decision needed D11: scripts shipped to workers.** As above, against each worker building its own plugins (240 ms +> of esbuild per cold start, and a second build per worker for one edit). Recommended. + +> **Decision needed D12: a step over many hosts.** `RolloutStep.reload` returns `{ hosts: [{ host, report }] }` instead +> of one `ReloadReport` (7c's in-process step returns one host). Recommended. + +## 5. Not in slice 10 + +The web page host, the browser's evaluation of scripts and serving them (slice 11); the TUI (slice 11); relocation and +pause of workers (the app's); request resolvers and authenticators as registries (the app's); chunked +transfers as a Chord helper (an owner pattern, design §23, unchanged); telemetry spans for requests (slice 9). + +## 6. Tests, examples, cost + +- One test file per source file. HTTP: listen-before-load holding requests and upgrades; authentication per request and + upgrade; patterns and precedence; conflicts reported; wraps (phases, order, scopes, a missing route, a throwing wrap); every response kind (ranges 206/416, `HEAD`, download names + made safe, a disconnect stopping `read`); bodies past their limit; a throwing route (500, reported); the links + endpoint (`host.accept`, a refused socket, the output cap); plugin sockets; a reload swapping a route while a request + runs on the old router; close. Workers: a start on the first hold, the lock, `ready` timeout, idle stop and a hold + keeping it, `busy` counts, a crash (reported, re-resolved while held), `stop`, `retry`, the rollout step (a worker + starting during it gets the new scripts), the child's shutdown and disconnect. +- GC: requests (their root scopes and bodies) and stopped workers' state collected; no listener left on a closed socket. +- Examples (each a folder, as 7c's): `40-http` (routes, a download with ranges, a socket), `41-workers` (a server + routing a session kind to forked workers, idle stop, a crash and recovery, a rollout reaching the workers). +- Benchmark: a route's overhead per request against a plain Node handler; a worker's cold start (fork to `ready`) with + shipped scripts against building in the child.