mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
feat(chord): split the worker base at dynamic imports; workers/child entry
buildWorkerBase bundles with esbuild code splitting into one private directory: the entry and its static chunks load at each fork, a lazily imported module and the packages it imports only when imported. @earendil-works/chord/workers/child exports the worker's side (workerMain, processLock) without esbuild.
This commit is contained in:
Mario Zechner
parent
da95cd79c6
commit
7ac1a9e3a0
16 changed files with 203 additions and 65 deletions
|
|
@ -7,11 +7,11 @@
|
|||
- Chord v2 is the package's root: `@earendil-works/chord` exports v2 (was `@earendil-works/chord/next`). Chord v1 moved unchanged to `@earendil-works/chord/old` (was the root) and `@earendil-works/chord/old/node` (was `/node`); it stays until p2 runs on v2. The foundations v1's root re-exported have their own subpaths: `@earendil-works/chord/json` (`JsonValue`, `JsonRepresentation`, `copyJson`, `isJsonValue`) and `@earendil-works/chord/watch` (`Watch`, `NativeWatch` and the watch types), beside `/context` and `/delta`. Migration: imports of `@earendil-works/chord/next` drop `/next`; v1's imports of `@earendil-works/chord` become `@earendil-works/chord/old`, and `/node` becomes `/old/node`; imports of `Context`, `Draft`, `JsonValue`, `copyJson` or the watch types from the root move to `/context`, `/delta`, `/json` and `/watch`. In the repository, v2's `src/next` is now `src`, `src/next-node` is `src/node`, `src/next-web`, `src/next-react` and `src/next-tui` are `src/web`, `src/react` and `src/tui`, v1 is in `src/old`, and tests and docs moved likewise (`test/next` to `test`, `docs/next` to `docs`, v1's to `test/old` and `docs/old`).
|
||||
- Chord v1 (`@earendil-works/chord/old`, `@earendil-works/chord/old/node`) removed, with its tests and docs (`src/old`, `test/old`, `docs/old`). Apps still on v1 pin `@earendil-works/chord@1.5.0`.
|
||||
- Chord v2's `startWorker` no longer takes `telemetry` (it was added unreleased in slice 9): with a telemetry store on the server, the kit's workers send their spans and metrics to it, and no worker opens the store's file. `startServer`'s `telemetry` is a store with `ingest` (`createSqliteTelemetry`'s). Migration: drop `telemetry` from `startWorker` and the store's path from the worker's environment.
|
||||
- Chord v2's `createWorkers` returns a promise and rejects when the worker base does not build (decision "Workers start from the base code the server booted with"): it bundles the worker kind's `entry` and the base code it imports once with esbuild (`--conditions` from `execArgv`; Node builtins and packages resolved under `node_modules` external, by their files; workspace packages linked into `node_modules` inlined; `import.meta.url` and `__dirname` naming each source file; an inline source map, forked with `--enable-source-maps`) into a private temporary directory removed at `close`, and forks every worker from that snapshot: an edit of the app's source while the server runs (an agent half way through a change) no longer reaches workers forked later, which kept failing to start until the edit was fixed; a server restart applies it. `workers.inputs` lists the files the base was built from; the app kit's `startServer` builds each worker kind's base before loading (a broken one rejects the start, everything closed) and marks `base` when one of those files changes (it marked only the entry).
|
||||
- Chord v2's `createWorkers` returns a promise and rejects when the worker base does not build (decision "Workers start from the base code the server booted with"): it bundles the worker kind's `entry` and the base code it imports once with esbuild (`--conditions` from `execArgv`; Node builtins and packages resolved under `node_modules` external, by their files; workspace packages linked into `node_modules` inlined; `import.meta.url` and `__dirname` naming each source file; split at each dynamic `import()` esbuild can follow, so a lazily imported module and the packages it imports load only when the worker imports it; inline source maps, forked with `--enable-source-maps`) into a private temporary directory removed at `close`, and forks every worker from that snapshot: an edit of the app's source while the server runs (an agent half way through a change) no longer reaches workers forked later, which kept failing to start until the edit was fixed; a server restart applies it. `workers.inputs` lists the files the base was built from; the app kit's `startServer` builds each worker kind's base before loading (a broken one rejects the start, everything closed) and marks `base` when one of those files changes (it marked only the entry).
|
||||
|
||||
### Added
|
||||
|
||||
- Chord v2's Node components and client surfaces are exported: `@earendil-works/chord/http`, `/workers`, `/plugins`, `/plugins/runtime` and `/app`; `/surfaces` (the web and terminal surfaces' server side), `/web`, `/react`, `/tui` and `/tui/launcher`.
|
||||
- Chord v2's Node components and client surfaces are exported: `@earendil-works/chord/http`, `/workers`, `/workers/child` (the worker's side, `workerMain` and `processLock`, without esbuild), `/plugins`, `/plugins/runtime` and `/app`; `/surfaces` (the web and terminal surfaces' server side), `/web`, `/react`, `/tui` and `/tui/launcher`.
|
||||
- Chord v2's plugins component gains the plugin model (slice 12): `createPlugins({ api: { name, versions } })` checks every import of the app's versioned API (`@acme/app/v3`, per-kind modules such as `@acme/app/v3/web`): an unlisted version, another kind's module, two versions in one facet and a direct import of Chord are build errors naming the file and line. Roots may be `{ path, required }`: a required root's plugin failing at `load` rejects it with a `PluginError` after the load settled. Files named `<name>.<kind>.<ext>` (`contract.web.tsx`) are of that kind, refused in facets of another kind with the chain of imports named. Single-file plugins (`hello.server.ts` at a root's top, grouped by name, host packages only). `installPlugin({ git, ref } | { npm, version }, root, ctx)` installs plugins from git (a shallow clone of a branch, tag or commit; a repository of several plugins too) or npm; a package lists the plugin folders it provides in its `package.json` (`"chord": { "plugins": ["./notes", "plugins/*", "!plugins/draft"] }`), used by discovery and `installPlugin` (without it: one plugin when the package's top has facet files, else its subfolders with a `package.json`). `testPlugin(dir, { kind, modules, stubs, principal }, ctx)` builds a plugin as production does and loads it on an in-memory host beside stub facets, collecting its contributions to registries nobody serves and listing the contracts it waits for. Agents' guide: `docs/plugin-authoring.md`.
|
||||
- Chord v2 gains the app kit (slice 13, `@earendil-works/chord/app`): `startServer<P>({ name, dataDir?, root?, listen?, api, kinds, principal, workers?, files?, install?, onError }, ctx)` lays out the data directory (`server/store.sqlite`, `workers/<kind>/<address>/`, `plugins/`, `logs/`, `owner-token`; one server per data directory), creates the server's store and host, HTTP, one workers component per worker kind and the plugins component over the app's required root and the user's root, and loads the server's plugins with the kit's objects; `startWorker({ api, kinds, busy?, onError })` is a worker's entry (its directory and store, the link to the server first, its plugins). The start-data objects `AppInfo` and `WorkerInfo` serve process values to plugins. Sessions (`Sessions`: `issue`, `verify`, `revoke`, `list`; tokens kept as SHA-256 hashes, compared in constant time; persisted ones end after 400 days unused and their browser cookie is renewed daily while used, ephemeral ones live in memory, at most `limits.ephemeralSessions`) authenticate a signed `HttpOnly`, `SameSite=Strict` cookie or `Authorization: Bearer`; a one-time owner link per start (`app.ownerLink()`, the token in the URL's fragment, single use, 10 minutes), an owner token file (mode 0600, written atomically) and device pairing (`POST /_chord/pair`, owner approval, a poll returning the token once; bounded in all and per client address) sign the owner in. The data directory, its stores and secret files are made private (0700, 0600) at every start; the home directory, a file system's root and another user's directory are refused. `Files` per host: small and chunked writes at a stored offset, ranged reads, a watchable index, caps, at most `limits.uploads` uploads under way. `app.exportData(address, path, ctx)` and `app.importData(archive, address, ctx)` move the data a worker created (its store and files) as a tar without stopping anything: a running worker writes its own store at one consistent point, else the server reads it under the worker's process lock (import refusing an existing address, another app's archive, every path leaving the target, and archives past `limits.importBytes` or `limits.importEntries`). `app.applyChanges(ctx)` and `app.plugins.install(source, ctx)`. Optional plugins as exported facets: `defineUsers(profile)`, `inviteFacet()`, `ephemeralLoginFacet()`.
|
||||
- Chord v2 gains web and terminal surfaces (slice 11): `webSurface` and `tuiSurface` (`@earendil-works/chord/surfaces`) build a client kind's base bundle from the app's entry at start (content-hashed, served forever) and publish its plugins' facets as scripts that register themselves (`chordFacet("<plugin>", "<digest>", function (require, module, exports) {…})`) and one combined file, with `WebScripts` and `TuiScripts` showing the current set (a rollout step publishes without waiting for clients) and the page served through the app's own template (`PageAssets`, `pageScripts`). `startPage` (`@earendil-works/chord/web`) runs a page's client host: the connection loop (`keepConnected`: 250 ms doubling to 5 s, wakes, silent sockets, `canRetry`), plugins applied in place on a new set (a failed one keeps its old facet), a reload when the server's base bundle changed, and the active theme as CSS variables. React bindings (`@earendil-works/chord/react`): `HostProvider`, `useHold`, `useWatch`, `useRegistry`, `useRefState` (React is an optional peer dependency). The terminal: `startTerminal` (`@earendil-works/chord/tui/launcher`) signs in with a token, a token file or device pairing and runs the app's terminal code always in a sandbox (a child process on pipes with Node's permission model: no file system, terminal, processes, workers, network or SQLite; Node 26.5.1 or later), filtering its output, relaying its link and confirming its actions (copy, open a URL, attach a file the user names); `startTerminalRuntime` and `TuiScope` (`@earendil-works/chord/tui`) inside it. Themes: the `Themes` registry, `ThemeChoice` (one dark and one light pick per principal), `followActiveTheme`, `themeVariables`, `terminalColor`. `startServer` gains `web` and `tui`. Plugin builds scope `*.module.css` files per plugin and file. Ephemeral sessions end with their client's last link (after a minute). The HTTP component's links endpoint gains an `opened` hook.
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ The design, the rule book for all of this, is [docs/design.md](docs/design.md).
|
|||
| `@earendil-works/chord` | Contracts (`local`, `remote`, `method`, the schema builders), errors (`ChordError`, `isChordError`), hosts and facets (`createHost`, `defineFacet`), documents (`defineState`), derives and natives (`constant`, `mapShared`), routines and jobs (`defineJob`), links (`memoryLinkPair`, `ipcLink`, `webSocketLink`), registries (`defineRegistry`), the store (`openStore`, `defineTable`, queries, export and import), and the client surfaces' shared part (`Themes`, `ThemeChoice`, `WebScripts`, `TuiScripts`, messages and languages: `defineMessages`, `Translations`, `LocaleChoice`, `formatTime`, `formatNumber`). | Node and browsers |
|
||||
| `@earendil-works/chord/http` | HTTP: a Web-standard `fetch` handler (`createHttp`) built from the routes, wraps and sockets plugins contribute to the registry `Http`, authenticators (`Authenticators`), responses as descriptions, a security baseline on by default, Chord's Node listener and `nodeRequest`, the links endpoint. | Node |
|
||||
| `@earendil-works/chord/workers` | Workers: `createWorkers` runs a forked worker per address of a host kind, started on demand and stopped when idle; `workerMain` is a worker's entry; `processLock`. | Node |
|
||||
| `@earendil-works/chord/workers/child` | The worker's side alone: `workerMain` and `processLock`, without esbuild (a worker entry imports this). | Node |
|
||||
| `@earendil-works/chord/plugins` | Plugins: discovery, builds with esbuild (`buildFacet`), evaluation, the watcher, rollouts, `createPlugins`, `installPlugin`, `testPlugin`, `pluginFile`. | Node |
|
||||
| `@earendil-works/chord/plugins/runtime` | `createPluginRuntime`: loads built plugins without esbuild (a worker's side). | Node |
|
||||
| `@earendil-works/chord/app` | The app kit: `startServer`, `startWorker`, the start-data objects, sessions and the owner's sign-in, device pairing, files, export and import per address, optional plugins (users, invites, ephemeral login). | Node |
|
||||
|
|
@ -36,7 +37,7 @@ The design, the rule book for all of this, is [docs/design.md](docs/design.md).
|
|||
| `@earendil-works/chord/sqlite/node` | `NodeSqliteDatabase`, `openNodeSqliteDatabase`, and `NodeSqliteDatabaseOptions`. | Node |
|
||||
| `@earendil-works/chord/sqlite/cloudflare` | `DurableObjectSqliteDatabase` and the structural `DurableObjectSqliteStorage` binding type. | Cloudflare Durable Objects |
|
||||
|
||||
`esbuild` (for `/plugins` and `/surfaces`), `ws` (for `/http` and `/tui/launcher`) and `react` (for `/react`) are optional peer dependencies, imported only there; `/app` uses all but `react`.
|
||||
`esbuild` (for `/plugins`, `/surfaces` and `/workers`), `ws` (for `/http` and `/tui/launcher`) and `react` (for `/react`) are optional peer dependencies, imported only there; `/app` uses all but `react`.
|
||||
|
||||
## Quick start
|
||||
|
||||
|
|
|
|||
|
|
@ -2397,13 +2397,20 @@ draining old holder without polling).
|
|||
installed; they change only through an install, which needs a restart anyway). Resolution as the child's Node does it
|
||||
(its `execArgv` `--conditions`, `main` only, `tsconfig.json` paths as esbuild reads them); every imported module runs
|
||||
(`sideEffects: false` drops nothing); `import.meta.url`, `.dirname`, `.filename`, `__dirname` and `__filename` name
|
||||
each module's source file. One ES module with an inline source map, forked with `--enable-source-maps` before the
|
||||
app's `execArgv` (stack traces name the source files); `execArgv` and `env` apply to every fork as before (modules an
|
||||
`--import` names load from disk at each fork). The file lives in a private temporary directory (0700, written again
|
||||
into a new one when a cleaner removed it), removed at `close`. Plugins are unaffected: workers evaluate the server's
|
||||
built scripts. Cost: a build at start (the kit's test worker 55-100 ms, p2's session worker 215-250 ms, 4.2 MB of
|
||||
code); a fork then loads one file (the kit's test worker 60 ms instead of 360 ms from source, p2's 350 ms instead of
|
||||
760 ms).
|
||||
each module's source file. Split at each dynamic `import()` with a literal specifier: the entry and the chunks it
|
||||
imports statically load at each fork, a lazily imported module and the packages it imports only when the worker
|
||||
first imports it (an app's lazily loaded providers stay lazy; a module still runs after its imports, but modules
|
||||
that do not import each other may run in another order than unbundled); a variable-specifier import resolves at run
|
||||
time against its module's source file, outside the snapshot. ES modules with inline source maps, forked with
|
||||
`--enable-source-maps` before the app's `execArgv` (stack traces name the source files); `execArgv` and `env` apply to
|
||||
every fork as before (modules an `--import` names load from disk at each fork). The files live in one private
|
||||
temporary directory (0700, written again into a new one when a cleaner removed one of them), removed at `close`. The
|
||||
worker's side alone is `@earendil-works/chord/workers/child` (`workerMain`, `processLock`): a worker entry importing
|
||||
it loads no esbuild. Plugins are unaffected: workers evaluate the server's built scripts. Cost: a build at start
|
||||
(the kit's test worker 55-100 ms, p2's session worker 215-250 ms, 4.2 MB of code); a fork then loads the entry and
|
||||
its static chunks (the kit's test worker 60 ms instead of 360 ms from source, p2's 350 ms instead of 760 ms; on a
|
||||
phone, splitting and the child entry took p2's fork from 690 to 301 loaded modules, no provider SDK or esbuild among them, about 300 ms
|
||||
less to `workerMain`).
|
||||
- **Keep-alive** (D9, R3, Q5): the child reports when `host.activity`'s references, jobs and lingering objects and the
|
||||
app's `busy` counts flip between all zero and not (local calls do not count; the first state with `ready`, which
|
||||
also carries the load report); the server stops a worker idle for `idleMs` (5 min).
|
||||
|
|
|
|||
|
|
@ -71,6 +71,11 @@
|
|||
"types": "./dist/node/workers/index.d.ts",
|
||||
"import": "./dist/node/workers/index.js"
|
||||
},
|
||||
"./workers/child": {
|
||||
"source": "./src/node/workers/child.ts",
|
||||
"types": "./dist/node/workers/child.d.ts",
|
||||
"import": "./dist/node/workers/child.js"
|
||||
},
|
||||
"./plugins": {
|
||||
"source": "./src/node/plugins/index.ts",
|
||||
"types": "./dist/node/plugins/index.d.ts",
|
||||
|
|
|
|||
|
|
@ -15,7 +15,7 @@ import { openStore, sqliteBackend } from "../../store/index.ts";
|
|||
import { translationsFacet } from "../../surfaces/index.ts";
|
||||
import { sendWorkerTelemetry, TelemetryRecorder } from "../../telemetry-ingest/index.ts";
|
||||
import type { Watchable } from "../../watch.ts";
|
||||
import { workerMain } from "../workers/index.ts";
|
||||
import { workerMain } from "../workers/child.ts";
|
||||
import type { AppApi, AppKind, KitStart } from "./app.ts";
|
||||
import { hostDataFacet } from "./data.ts";
|
||||
import { filesFacet } from "./files.ts";
|
||||
|
|
|
|||
|
|
@ -3,7 +3,9 @@
|
|||
The workers component (slice 10 §3, §4, design §23, decision "Slice 10's points"): a forked worker process per address
|
||||
of one host kind, started on demand by the route it owns, stopped when idle, never restarted by itself; forked from the
|
||||
kind's worker base, bundled when the component is created; its plugins are built by the server and shipped.
|
||||
`workerMain` is the worker's side. Node-only. Exported as `@earendil-works/chord/workers`.
|
||||
`workerMain` is the worker's side. Node-only. Exported as `@earendil-works/chord/workers`; the worker's side alone
|
||||
(`workerMain`, `processLock`) as `@earendil-works/chord/workers/child`, which loads no esbuild: a worker entry imports
|
||||
that.
|
||||
|
||||
## For apps (`index.ts`)
|
||||
|
||||
|
|
@ -27,7 +29,7 @@ await workers.retry(Session({ id }), ctx); // a failed or suspended address
|
|||
await plugins.rollout([plugins.step("server", server), workers.step("workers")], built, ctx);
|
||||
await workers.close(ctx);
|
||||
|
||||
// worker.ts
|
||||
// worker.ts (imports `workerMain` from "@earendil-works/chord/workers/child")
|
||||
workerMain(async ({ address, start, parent, plugins, onClose }, ctx) => {
|
||||
const host = createHost({ address, principal: { kind: "worker" }, store, onError });
|
||||
onClose((ctx) => host.close(ctx)); // closers run in reverse order
|
||||
|
|
@ -49,10 +51,18 @@ workerMain(async ({ address, start, parent, plugins, onClose }, ctx) => {
|
|||
builtins and every package resolved under `node_modules` (native addons, packages reading their own files). Imports
|
||||
resolve as the child's Node would (`--conditions` from `execArgv`, `main` only, `tsconfig.json` paths); every
|
||||
imported module runs (`sideEffects: false` drops none); `import.meta.url`, `.dirname`, `.filename`, `__dirname` and
|
||||
`__filename` name each module's source file. One ES module with an inline source map, forked with
|
||||
`--enable-source-maps` ahead of the app's `execArgv` (stack traces name the source files). Its file is in a private
|
||||
temporary directory, written again into a new one when a cleaner removed it, removed at `close`. `inputs` lists the
|
||||
files it was built from: the kit marks `base` when one changes.
|
||||
`__filename` name each module's source file. Split at each dynamic `import()` esbuild can follow (a literal
|
||||
specifier): the entry module and the chunks it imports statically load at each fork; a lazily imported module, with
|
||||
the packages it imports, is a chunk Node loads when the worker first imports it (pi-ai's provider APIs and their
|
||||
SDKs). A module the entry's graph shares with a lazy chunk moves into a shared chunk, which runs before the
|
||||
importing chunk's own modules: a module still runs after its imports, but two modules that do not import each other
|
||||
may run in another order than unbundled (esbuild's splitting; base code must not rely on sibling order). A variable
|
||||
specifier stays an import Node resolves at run time, against the module's source file (its
|
||||
`import.meta.url`): outside the snapshot, so base code that must be in it imports with a literal specifier. ES modules
|
||||
with inline source maps, forked with `--enable-source-maps` ahead of the app's `execArgv` (stack traces name the
|
||||
source files). The files are in one private temporary directory, written again into a new one when a cleaner removed
|
||||
one of them (checked at each fork; a chunk removed while a worker runs fails that worker's import of it), removed at
|
||||
`close`. `inputs` lists the files it was built from: the kit marks `base` when one changes.
|
||||
- **Per address:** a child's events (`started`, `ready`, `idle`, `fatal`, its exit, its link closed by this side) are
|
||||
recorded on that child as they happen (a late event of an old child changes nothing); starts and stops run one after
|
||||
another. A start waits for the previous child's exit and rejects with a crash recorded meanwhile; a stop records its
|
||||
|
|
@ -112,6 +122,7 @@ stays failed, Retry brings it back), `43-rollout` (an edit rolled out to the ser
|
|||
|
||||
## Inside the directory
|
||||
|
||||
`base.ts` (the worker base: the entry bundled once), `control.ts` (control messages and errors as they cross),
|
||||
`base.ts` (the worker base: the entry bundled once), `child.ts` (the worker's side as its own entry point),
|
||||
`control.ts` (control messages and errors as they cross),
|
||||
`lock.ts` (process locks), `process.ts` (one forked worker as the server sees it), `workers.ts` (the component: the
|
||||
route, per-address records, idle stops, the rollout step, the state), `main.ts` (`workerMain`).
|
||||
|
|
|
|||
|
|
@ -4,13 +4,15 @@
|
|||
// server runs reaches workers only through a restart of the server. Packages installed under `node_modules` (native
|
||||
// addons, packages reading their own files) and Node builtins stay external: Node loads them at each fork from where
|
||||
// the build resolved them. Everything else, the app's files and workspace packages linked into `node_modules`, is
|
||||
// inlined. The bundle is one ES module with an inline source map in a private temporary directory.
|
||||
// inlined. The bundle is split at each dynamic `import()` esbuild can follow: the entry module and the chunks it imports
|
||||
// statically load at each fork, a lazily imported module (pi-ai's provider APIs and the packages they import) only when
|
||||
// the worker imports it. Every file is an ES module with an inline source map, in one private temporary directory.
|
||||
|
||||
import { existsSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { isBuiltin } from "node:module";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, extname, join, parse, resolve, sep } from "node:path";
|
||||
import { basename, dirname, extname, join, parse, resolve, sep } from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { build, type Loader, type Plugin } from "esbuild";
|
||||
import { metafileInputs } from "../surfaces/bundle.ts";
|
||||
|
|
@ -19,7 +21,10 @@ import { metafileInputs } from "../surfaces/bundle.ts";
|
|||
export interface WorkerBase {
|
||||
/** The files esbuild read for it (absolute): the app's base code a change of which outdates it. */
|
||||
readonly inputs: readonly string[];
|
||||
/** The bundle's path, written again into a new private directory when a temporary-file cleaner removed it. */
|
||||
/**
|
||||
* The entry module's path, its chunks beside it; all written again into a new private directory when a
|
||||
* temporary-file cleaner removed one of them.
|
||||
*/
|
||||
file(): string;
|
||||
/** Removes the bundle's directory. */
|
||||
remove(): void;
|
||||
|
|
@ -33,6 +38,9 @@ const MODULE_URL = "__chordModuleUrl";
|
|||
const MODULE_DIRNAME = "__chordModuleDirname";
|
||||
const MODULE_FILENAME = "__chordModuleFilename";
|
||||
|
||||
/** The entry module's name in the bundle's directory (`worker.mjs`); chunks are named `<name>-<hash>.mjs`. */
|
||||
const ENTRY_NAME = "worker";
|
||||
|
||||
/**
|
||||
* Bundles `entry` for workers forked with `execArgv` (its `--conditions` resolve imports as the child's Node would).
|
||||
* Rejects with esbuild's failure: the server does not start with a broken worker base.
|
||||
|
|
@ -41,8 +49,16 @@ export async function buildWorkerBase(entry: string, execArgv: readonly string[]
|
|||
// Built as if at the file system's root: the source map's paths relative to it, made absolute by `sourceRoot`.
|
||||
const root = parse(resolve(entry)).root;
|
||||
const result = await build({
|
||||
entryPoints: [entry],
|
||||
outfile: join(root, "worker.mjs"),
|
||||
entryPoints: [{ in: entry, out: ENTRY_NAME }],
|
||||
// One flat directory: every chunk imports the others as `./<name>`, and the source maps' paths stay relative to
|
||||
// the root.
|
||||
outdir: root,
|
||||
entryNames: "[name]",
|
||||
chunkNames: "[name]-[hash]",
|
||||
outExtension: { ".js": ".mjs" },
|
||||
// A dynamic `import()` esbuild can follow becomes a chunk Node loads only when the worker imports it, with the
|
||||
// packages it imports: inlined into one module, every lazily imported module would load and run at each fork.
|
||||
splitting: true,
|
||||
write: false,
|
||||
bundle: true,
|
||||
format: "esm",
|
||||
|
|
@ -58,7 +74,8 @@ export async function buildWorkerBase(entry: string, execArgv: readonly string[]
|
|||
sourcemap: "inline",
|
||||
sourcesContent: false,
|
||||
sourceRoot: pathToFileURL(root).href,
|
||||
// A CommonJS file of the base code requires through this; external packages by their absolute paths.
|
||||
// A CommonJS file of the base code requires through this (each chunk has its own); external packages by their
|
||||
// absolute paths.
|
||||
banner: {
|
||||
js: 'import { createRequire as __chordCreateRequire } from "node:module"; const require = __chordCreateRequire(import.meta.url);',
|
||||
},
|
||||
|
|
@ -73,9 +90,8 @@ export async function buildWorkerBase(entry: string, execArgv: readonly string[]
|
|||
logLevel: "silent",
|
||||
metafile: true,
|
||||
});
|
||||
// One entry, one output file.
|
||||
const [output] = result.outputFiles;
|
||||
return new BundleFile(output!.contents, metafileInputs(result.metafile));
|
||||
const files = result.outputFiles.map((output) => ({ name: basename(output.path), contents: output.contents }));
|
||||
return new BundleFiles(files, metafileInputs(result.metafile));
|
||||
}
|
||||
|
||||
/** The conditions of `--conditions=<name>`, `--conditions <name>`, `-C <name>` and `-C=<name>` in `execArgv`. */
|
||||
|
|
@ -160,37 +176,42 @@ function moduleLocations(): Plugin {
|
|||
};
|
||||
}
|
||||
|
||||
/** The bundle as a file: written into a private temporary directory, again into a new one when it is gone. */
|
||||
class BundleFile implements WorkerBase {
|
||||
readonly inputs: readonly string[];
|
||||
readonly #contents: Uint8Array;
|
||||
#dir = "";
|
||||
#path = "";
|
||||
/** One output file of the bundle: its name in the bundle's directory and its contents. */
|
||||
interface BundleFile {
|
||||
readonly name: string;
|
||||
readonly contents: Uint8Array;
|
||||
}
|
||||
|
||||
constructor(contents: Uint8Array, inputs: readonly string[]) {
|
||||
this.#contents = contents;
|
||||
/** The bundle as files: written into a private temporary directory, again into a new one when one of them is gone. */
|
||||
class BundleFiles implements WorkerBase {
|
||||
readonly inputs: readonly string[];
|
||||
readonly #files: readonly BundleFile[];
|
||||
#dir = "";
|
||||
|
||||
constructor(files: readonly BundleFile[], inputs: readonly string[]) {
|
||||
this.#files = files;
|
||||
this.inputs = inputs;
|
||||
this.#write();
|
||||
}
|
||||
|
||||
file(): string {
|
||||
// A long-running server's temporary directory may be cleaned (systemd-tmpfiles, macOS): a fork needs the file. A
|
||||
// new directory, never the old name: another user may have made that one since.
|
||||
if (!existsSync(this.#path)) {
|
||||
// A long-running server's temporary directory may be cleaned (systemd-tmpfiles, macOS), a chunk no worker
|
||||
// imported for a while first: a fork needs all of them. A new directory, never the old name: another user may
|
||||
// have made that one since.
|
||||
if (!this.#files.every((file) => existsSync(join(this.#dir, file.name)))) {
|
||||
this.remove();
|
||||
this.#write();
|
||||
}
|
||||
return this.#path;
|
||||
return join(this.#dir, `${ENTRY_NAME}.mjs`);
|
||||
}
|
||||
|
||||
remove(): void {
|
||||
rmSync(this.#dir, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
/** Writes the bundle into a new private directory (mode 0700, the file 0600). */
|
||||
/** Writes the bundle's files into a new private directory (mode 0700, each file 0600). */
|
||||
#write(): void {
|
||||
this.#dir = mkdtempSync(join(tmpdir(), "chord-worker-"));
|
||||
this.#path = join(this.#dir, "worker.mjs");
|
||||
writeFileSync(this.#path, this.#contents, { mode: 0o600 });
|
||||
for (const file of this.#files) writeFileSync(join(this.#dir, file.name), file.contents, { mode: 0o600 });
|
||||
}
|
||||
}
|
||||
|
|
|
|||
13
packages/chord/src/node/workers/child.ts
Normal file
13
packages/chord/src/node/workers/child.ts
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
// The worker process's side of the workers component (`@earendil-works/chord/workers/child`): `workerMain` and
|
||||
// `processLock` without the server's side, whose worker base bundles with esbuild. A worker entry imports this, so a
|
||||
// forked worker never loads esbuild.
|
||||
|
||||
export type { CrossedError } from "./control.ts";
|
||||
export { type ProcessLock, processLock } from "./lock.ts";
|
||||
export {
|
||||
type WorkerMainOptions,
|
||||
type WorkerPlugins,
|
||||
type WorkerResult,
|
||||
type WorkerStart,
|
||||
workerMain,
|
||||
} from "./main.ts";
|
||||
|
|
@ -1,16 +1,8 @@
|
|||
// Chord v2's workers component (`@earendil-works/chord/workers`, slice 10 §3, §4): a forked worker process per
|
||||
// address of a host kind, started on demand, stopped when idle, never restarted by itself, its plugins shipped from the
|
||||
// server; `workerMain` is the worker's side. Node-only.
|
||||
// server; `workerMain` is the worker's side (also `@earendil-works/chord/workers/child`, without esbuild). Node-only.
|
||||
|
||||
export type { CrossedError } from "./control.ts";
|
||||
export { type ProcessLock, processLock } from "./lock.ts";
|
||||
export {
|
||||
type WorkerMainOptions,
|
||||
type WorkerPlugins,
|
||||
type WorkerResult,
|
||||
type WorkerStart,
|
||||
workerMain,
|
||||
} from "./main.ts";
|
||||
export * from "./child.ts";
|
||||
export {
|
||||
createWorkers,
|
||||
type WorkerState,
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts";
|
||||
import { createHost } from "../../../src/host/index.ts";
|
||||
import * as chord from "../../../src/index.ts";
|
||||
import { workerMain } from "../../../src/node/workers/index.ts";
|
||||
import { workerMain } from "../../../src/node/workers/child.ts";
|
||||
|
||||
workerMain(
|
||||
async ({ address, parent, plugins, onClose }, context) => {
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts";
|
||||
import { createHost } from "../../../src/host/index.ts";
|
||||
import * as chord from "../../../src/index.ts";
|
||||
import { workerMain } from "../../../src/node/workers/index.ts";
|
||||
import { workerMain } from "../../../src/node/workers/child.ts";
|
||||
|
||||
workerMain(
|
||||
async ({ address, parent, plugins, onClose }, context) => {
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@
|
|||
import { BACKGROUND_CONTEXT } from "../../../src/context/index.ts";
|
||||
import { createHost } from "../../../src/host/index.ts";
|
||||
import * as chord from "../../../src/index.ts";
|
||||
import { workerMain } from "../../../src/node/workers/index.ts";
|
||||
import { workerMain } from "../../../src/node/workers/child.ts";
|
||||
|
||||
workerMain(
|
||||
async ({ address, parent, plugins, onClose }, context) => {
|
||||
|
|
|
|||
|
|
@ -1,11 +1,12 @@
|
|||
// Tests of the worker base (decision "Workers start from the base code the server booted with"): the entry and its base
|
||||
// code bundled once (an edit after the build runs nowhere, every imported module runs), installed packages external and
|
||||
// Node builtins kept, a workspace package linked into `node_modules` inlined, the child's `--conditions`, module
|
||||
// locations and stack traces naming the source files, a broken entry rejected, and the file in a private directory
|
||||
// restored when a cleaner removed it.
|
||||
// code bundled once (an edit after the build runs nowhere, every imported module runs), split at dynamic imports (a
|
||||
// lazily imported module and the packages it imports load only when imported, still from the snapshot), installed
|
||||
// packages external and Node builtins kept, a workspace package linked into `node_modules` inlined, the child's
|
||||
// `--conditions`, module locations and stack traces naming the source files, a broken entry rejected, and the files in a
|
||||
// private directory restored when a cleaner removed one.
|
||||
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import { existsSync, statSync } from "node:fs";
|
||||
import { existsSync, readdirSync, statSync } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
|
|
@ -65,6 +66,35 @@ describe("the snapshot", () => {
|
|||
expect(run(await built(join(dir, "entry.ts")))).toBe("effect\nentry");
|
||||
});
|
||||
|
||||
it("splits at a dynamic import: the module and the package it imports load only when imported, from the snapshot", async () => {
|
||||
const dir = await project({
|
||||
"entry.ts": [
|
||||
'import { shared } from "./shared.ts";',
|
||||
'console.log("entry", shared);',
|
||||
'const lazy = await import("./lazy.ts");',
|
||||
"console.log(lazy.lazy);",
|
||||
].join("\n"),
|
||||
"shared.ts": 'export const shared = "shared";\n',
|
||||
"lazy.ts": [
|
||||
'import { shared } from "./shared.ts";',
|
||||
'import { where } from "installed";',
|
||||
'console.log("lazy runs");',
|
||||
'export const lazy = ["lazy", shared, where].join(" ");',
|
||||
].join("\n"),
|
||||
"node_modules/installed/package.json": '{ "name": "installed", "type": "module", "exports": "./index.js" }',
|
||||
"node_modules/installed/index.js": 'console.log("installed runs");\nexport const where = "installed";\n',
|
||||
});
|
||||
const base = await built(join(dir, "entry.ts"));
|
||||
await writeFile(join(dir, "lazy.ts"), 'export const lazy = "edited";\n');
|
||||
expect(run(base)).toBe("entry shared\ninstalled runs\nlazy runs\nlazy shared installed");
|
||||
expect([...base.inputs].sort()).toEqual([join(dir, "entry.ts"), join(dir, "lazy.ts"), join(dir, "shared.ts")]);
|
||||
// The entry imports neither the lazy module's chunk nor the installed package statically: a fork loads neither.
|
||||
const entry = await readFile(base.file(), "utf8");
|
||||
expect(entry).not.toContain("installed/index.js");
|
||||
expect(entry).toMatch(/await import\("\.\/lazy-[A-Z0-9]+\.mjs"\)/);
|
||||
expect(readdirSync(dirname(base.file())).length).toBeGreaterThan(1);
|
||||
});
|
||||
|
||||
it("a broken entry rejects with esbuild's failure", async () => {
|
||||
const dir = await project({ "entry.ts": 'import { label } from "./missing.ts";\nconsole.log(label);\n' });
|
||||
await expect(buildWorkerBase(join(dir, "entry.ts"), [])).rejects.toThrow(/Could not resolve "\.\/missing\.ts"/);
|
||||
|
|
@ -150,16 +180,25 @@ describe("module locations and stack traces", () => {
|
|||
});
|
||||
});
|
||||
|
||||
it("maps stack traces to the source files and lines with --enable-source-maps", async () => {
|
||||
it("maps stack traces to the source files and lines with --enable-source-maps, in the entry and in a chunk", async () => {
|
||||
const dir = await project({
|
||||
"entry.ts": 'import { fail } from "./fail.ts";\nfail();\n',
|
||||
"lazy-entry.ts": 'const { failLater } = await import("./lazy.ts");\nfailLater();\n',
|
||||
"lazy.ts": 'import { fail } from "./fail.ts";\n\nexport function failLater(): void {\n\tfail();\n}\n',
|
||||
"fail.ts":
|
||||
"const at: string = import.meta.url;\n\nexport function fail(): void {\n\tthrow new Error(at);\n}\n",
|
||||
});
|
||||
const base = await built(join(dir, "entry.ts"));
|
||||
const { stderr } = spawnSync(process.execPath, ["--enable-source-maps", base.file()], { encoding: "utf8" });
|
||||
expect(stderr).toContain(`${join(dir, "fail.ts")}:4:`);
|
||||
expect(stderr).toContain(`${join(dir, "entry.ts")}:2:`);
|
||||
const stderrOf = async (entry: string): Promise<string> => {
|
||||
const base = await built(join(dir, entry));
|
||||
return spawnSync(process.execPath, ["--enable-source-maps", base.file()], { encoding: "utf8" }).stderr;
|
||||
};
|
||||
const direct = await stderrOf("entry.ts");
|
||||
expect(direct).toContain(`${join(dir, "fail.ts")}:4:`);
|
||||
expect(direct).toContain(`${join(dir, "entry.ts")}:2:`);
|
||||
const lazy = await stderrOf("lazy-entry.ts");
|
||||
expect(lazy).toContain(`${join(dir, "fail.ts")}:4:`);
|
||||
expect(lazy).toContain(`${join(dir, "lazy.ts")}:4:`);
|
||||
expect(lazy).toContain(`${join(dir, "lazy-entry.ts")}:2:`);
|
||||
});
|
||||
});
|
||||
|
||||
|
|
@ -176,4 +215,21 @@ describe("the bundle's file", () => {
|
|||
base.remove();
|
||||
expect(existsSync(dirname(second))).toBe(false);
|
||||
});
|
||||
|
||||
it("written again when a cleaner removed only a chunk: a fork may import every chunk", async () => {
|
||||
const dir = await project({
|
||||
"entry.ts": 'const { lazy } = await import("./lazy.ts");\nconsole.log(lazy);\n',
|
||||
"lazy.ts": 'export const lazy = "lazy";\n',
|
||||
});
|
||||
const base = await built(join(dir, "entry.ts"));
|
||||
const first = base.file();
|
||||
const chunks = readdirSync(dirname(first)).filter((name) => name !== "worker.mjs");
|
||||
expect(chunks).toHaveLength(1);
|
||||
expect(statSync(join(dirname(first), chunks[0]!)).mode & 0o777).toBe(0o600);
|
||||
await rm(join(dirname(first), chunks[0]!));
|
||||
const second = base.file();
|
||||
expect(dirname(second)).not.toBe(dirname(first));
|
||||
expect(existsSync(dirname(first))).toBe(false);
|
||||
expect(run(base)).toBe("lazy");
|
||||
});
|
||||
});
|
||||
|
|
|
|||
28
packages/chord/test/node/workers/child.test.ts
Normal file
28
packages/chord/test/node/workers/child.test.ts
Normal file
|
|
@ -0,0 +1,28 @@
|
|||
// Tests of the worker's side as its own entry point (`@earendil-works/chord/workers/child`): importing it loads no
|
||||
// esbuild, so a forked worker does not pay for the server's worker base build.
|
||||
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
describe("the child entry's imports", () => {
|
||||
it("loads no esbuild: a worker importing workerMain and processLock never imports it", () => {
|
||||
const child = fileURLToPath(new URL("../../../src/node/workers/child.ts", import.meta.url));
|
||||
// A hook refusing esbuild, then the child entry imported.
|
||||
const script = [
|
||||
'import { registerHooks } from "node:module";',
|
||||
"registerHooks({ resolve(specifier, context, next) {",
|
||||
' if (specifier === "esbuild") throw new Error("esbuild imported");',
|
||||
" return next(specifier, context);",
|
||||
"} });",
|
||||
`const { processLock, workerMain } = await import(${JSON.stringify(child)});`,
|
||||
"console.log(typeof processLock, typeof workerMain);",
|
||||
].join("\n");
|
||||
const output = execFileSync(
|
||||
process.execPath,
|
||||
["--experimental-strip-types", "--no-warnings", "--input-type=module", "--eval", script],
|
||||
{ encoding: "utf8" },
|
||||
);
|
||||
expect(output.trim()).toBe("function function");
|
||||
});
|
||||
});
|
||||
|
|
@ -43,6 +43,9 @@
|
|||
"@earendil-works/chord/workers": [
|
||||
"../../../packages/chord/src/node/workers/index.ts"
|
||||
],
|
||||
"@earendil-works/chord/workers/child": [
|
||||
"../../../packages/chord/src/node/workers/child.ts"
|
||||
],
|
||||
"@earendil-works/chord/plugins": [
|
||||
"../../../packages/chord/src/node/plugins/index.ts"
|
||||
],
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@
|
|||
"@earendil-works/chord/telemetry-sqlite": ["./packages/chord/src/telemetry-sqlite/index.ts"],
|
||||
"@earendil-works/chord/http": ["./packages/chord/src/node/http/index.ts"],
|
||||
"@earendil-works/chord/workers": ["./packages/chord/src/node/workers/index.ts"],
|
||||
"@earendil-works/chord/workers/child": ["./packages/chord/src/node/workers/child.ts"],
|
||||
"@earendil-works/chord/plugins": ["./packages/chord/src/node/plugins/index.ts"],
|
||||
"@earendil-works/chord/plugins/runtime": ["./packages/chord/src/node/plugins/runtime.ts"],
|
||||
"@earendil-works/chord/app": ["./packages/chord/src/node/app/index.ts"],
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue