sscoble/pi-mono
1
0
Fork 0
mirror of https://github.com/badlogic/pi-mono.git synced 2026-11-05 23:37:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

feat(chord): replace the facet and service runtime with the chord2 host runtime

Browse source 
Moves chord2 (contracts, objects, cells, roots, plugins, hot reload, memory/IPC/WebSocket links)
from p2 into packages/chord, with its tests (now on vitest) and docs. The root export is the new
runtime plus the shared base (Context, JsonValue, JsonRepresentation, Draft, copyJson, isJsonValue,
replicatedState and its types). ./node now holds the host manager, child control, process locks,
and nodeLoader. ./context and ./delta are unchanged; ./bundler is removed.

replicatedState stays a standalone module with the same exports. It no longer exposes the exact-op
subscription the removed service provider used, so listener failures of mutable states are always
reported asynchronously. The package depends on typebox instead of esbuild. The wire format stays JSON.
This commit is contained in:
Mario Zechner 2026-10-12 10:29:06 +00:00
parent 7472cda0d6
commit 0104fef1e9
92 changed files with 15191 additions and 8974 deletions
Download patch file Download diff file Expand all files Collapse all files

302
packages/chord/README.md
View file

@ -1,113 +1,95 @@
# @earendil-works/chord
Chord is an application-composition runtime for systems assembled from
plugins/extensions. It provides facets, services, replicated state, and a
pluggable remote-service boundary. It is developed as a standalone package in
the Pi monorepo, but it is not a Pi package: it does not depend on any other Pi
workspace package and can be used by unrelated applications.
Chord builds applications out of **objects** that live in several processes (a server, workers, browser tabs) and can be **hot reloaded** while they run.
## What Chord is for
- An **object** is state plus code at an address. It is created on demand and lives while someone holds it.
- Everyone gets a **handle**, never the object. Handles work the same locally and across processes, and survive reloads.
- Every object knows its **principal**: an owner and a guest get different objects, with different methods.
- **State** is observable JSON. Remote viewers get a snapshot followed by deltas.
- **Hot reload** swaps code while it runs. Work in flight finishes on the code it started with, state is kept, and incompatible changes migrate at a safe point.
A single application feature may need to run in several environments: for
example, an agent worker, a terminal UI, and a remote WebUI. Chord provides the
generic machinery to write such extensions in a way that is both delightful for
humans as well as agents.
Contracts are runtime values described with [TypeBox](https://github.com/sinclairzx81/typebox) schemas. Hosts exchange JSON frames over links: in memory, a Node IPC channel, or a WebSocket.
The design has a few connected pieces:
## Entry points
- **Plugins** are synchronous setup units that declare the services they provide
and require. After every plugin has declared its shape, a host validates the
complete dependency graph, binds services, activates providers before consumers,
and disposes resources in reverse dependency order. These units are called
*facets*.
| Import | Contents | Runs in |
|--------|----------|---------|
| `@earendil-works/chord` | Contracts (`local`, `remote`, `method`, `opaque`), declarations (`cell`, `derivation`, `listener`, `resource`), hosts (`createHost`, `defineFacet`, `defineRegistry`), links (`memoryLinkPair`, `ipcLink`, `webSocketLink`, `acceptWebSockets`), errors (`ChordError`, `isChordError`, `forbidden`, `invalid`). Also the shared base: `Context`, `JsonValue`, `JsonRepresentation`, `Draft`, `copyJson`, `isJsonValue`, and `replicatedState` with its types. | Node and browsers |
| `@earendil-works/chord/node` | Child host processes: `createHostManager`, `forkDriver`, `childControl`, `processLock`; the facet module loader `nodeLoader`. | Node |
| `@earendil-works/chord/context` | Explicit cancellation and invocation values: `BACKGROUND_CONTEXT`, `withCancel`, `withAbortSignal`, `withoutAbortSignal`, `withContextValue`, `awaitWithContext`. | Node and browsers |
| `@earendil-works/chord/delta` | The transactional JSON tracker, structural diff, immutable apply, and the wire encoder/decoder for operation batches. | Node and browsers |
- **Facets** are parts of a plugin. Each facet is bundled up separately and runs
in the process or environment where it's supposed to run. You can use facets
to split a plugin into separate pieces that need to be loaded into different
processes and environments (think backend, browser, TUI etc.)
## Quick start
- **Services** are typed, stable tokens with either one provider (**singleton**)
or dynamic keyed instances (**keyed**). A service can be process-local, with
an unrestricted JavaScript contract, or remotely exposable. Consumers retain a
stable facade while a provider disconnects or is replaced.
```ts
import { Type } from "typebox";
import { cell, createHost, method, remote } from "@earendil-works/chord";
- **Replicated state** exposes authoritative state to local and remote
connected consumers. Producers publish atomic overlay transactions with
`change(context, callback)`; consumers receive complete immutable values. Draft
proxies exist only during the callback and become unusable afterward. Preparation
materializes a structurally shared immutable candidate and one exact decoded
operation batch, while each remote client/state stream owns independent path-codec
state. Replicas become unready on disconnect or replacement until rehydrated.
// contract.ts: shared by provider and consumers
export const Counter = remote("demo.counter", {
version: 1,
input: Type.Object({ name: Type.String() }),
variants: {
default: {
state: { count: Type.Number() },
methods: { increment: method(Type.Object({ by: Type.Number() }), Type.Null()) },
},
},
});
- **Delta tracking** records and coalesces operations over tracked plain JSON.
It preserves common string and array operations, supports durable base
batches, and validates untrusted operations as they are applied. Batches
guarantee convergence but are not canonical or necessarily minimal.
const host = createHost({ name: "server", principalKey: (p) => p.kind });
- **Remote service sources** advertise services available outside a facet host
and open bindings for the services its facets require. Bindings carry logical
calls and subscriptions through an application-supplied adapter. Chord
requires strict-JSON arguments, results, snapshots, updates, and catalogues,
but does not prescribe framing, routing, transport, or an application wire
envelope. `JsonRepresentation<T>` derives a wire-safe type for application data
with unknown payloads, while `isJsonValue()` validates received values at an
adapter boundary. Symmetric RPC peers are planned as one optional
implementation of this boundary.
host.serve(Counter, ({ name }, _principal, self) => {
const state = self.state({ count: cell(Type.Number(), 0) });
return {
kind: "default",
count: state.count,
increment: async ({ by }, ctx) => {
state.count.change(ctx, (n) => n + by);
return null;
},
};
});
- **Context** Chord provides a Go-like context system for cancellation and
invocation-scoped application values. Applications can carry permissions or
telemetry through those values without Chord depending on either.
await host.root(async (ctx) => {
const clicks = ctx.get(Counter, { name: "clicks" }); // a handle; held until this root ends
clicks.count.subscribe((n) => console.log("count", n));
await clicks.increment({ by: 2 }, ctx); // logs 0, then 2
});
```
The current runtime exports service tokens, singleton and keyed providers,
remote bindings, replicated state, facet hosts, and facet loaders from
`@earendil-works/chord`. Import public types and general runtime APIs from the
package root. Context constants and functions live in
`@earendil-works/chord/context` because their generic names should not pollute
the root API.
Chord-owned identifiers use the `chord.*` namespace and its reserved service
prefix is `$chord.*`.
## Documentation
## Remote service adapters
- [Tutorial](docs/tutorial.md): a shared todo list, built up section by section. Every example runs as `test/tutorial.test.ts`.
- [Usage guide](docs/guide.md): API reference for contracts, objects, state, roots, lifetime, plugins, hot reload, hosts, and links.
- [Specification](docs/spec.md): compatibility, lifetime, and reload semantics.
- [Implementation notes](docs/notes.md) and [requests](docs/requests.md): decisions, review findings, and resolved requests.
- [Delta guide](src/delta/README.md): ownership, lifecycle, operation, and replica contracts of `@earendil-works/chord/delta`.
Chord owns its transport-independent service wire grammar. Consumer adapters
use `createServiceCatalogueCall()`, `createServiceSubscribeCall()`, and
`createServiceUnsubscribeCall()` for `$chord.service` control calls.
`createRemoteServiceEndpoint()` handles those calls for one provider consumer,
including subscription activation and cleanup. `parseServiceCall()`,
`parseServiceCatalogue()`, and the decoded/wire snapshot and update parsers
validate Chord semantics after an adapter has established a strict-JSON
boundary. `RemoteServiceErrorCode` and `REMOTE_SERVICE_ERROR_CODES` define the
service errors that may cross that boundary.
## Replicated state
Replicated state operations use one `createServiceStateEncoder()` at the
provider side and one `createServiceStateDecoder()` at the consumer side for
each subscription. Those registries create an independent Delta path dictionary
for every instance/member state and reset it on replacement, unavailability,
close, or fresh hydration. Applications may place these values inside any
routing, request, response, or event envelope; Chord does not prescribe that
outer protocol.
`replicatedState` is a standalone observable immutable JSON value, independent of hosts. Durable uses it for document, view, and task-graph states.
Provider subscriptions atomically capture a snapshot and buffer subsequent
updates until activation. Activation and reentrant publication use the same FIFO.
The provider retains at most 100 pending updates per subscription, excluding the
running delivery. Adding update 101 replaces the entire pending queue with
`{ type: "reset", snapshot }`: a current full subscription snapshot whose state
members contain `[["r", value]]` and their new sequence baselines. It also captures
current instance membership, so discarded spawn/close/replacement events cannot
leave stale instances behind. Live keyed generations retain their existing handles.
The reset carries the context of the publication that triggered overflow.
```ts
import { replicatedState } from "@earendil-works/chord";
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
Consumers and codecs must handle this explicit reset before accepting subsequent
ordinary deltas. A root operation in an ordinary `state` update does **not** permit
a sequence gap. Resets restart the subscription's path dictionaries, and later
deltas must be contiguous from each reset baseline. Transport adapters must
preserve snapshot/reset/update order; they must not drop encoded delta batches or
assume their own asynchronous queues are bounded by the provider's queue.
const status = replicatedState({ output: "", count: 0 }); // takes ownership of the initial value
status.subscribe((value, context, delivery) => console.log(delivery.kind, delivery.sequence, value.count));
status.change(BACKGROUND_CONTEXT, (draft) => {
draft.output += "done\n";
draft.count += 1;
});
```
`replicatedState(source, { onError })` instead attaches a publication-only state to an authoritative revision source (`ReplicatedStateSource`): it captures a snapshot, then publishes every committed frame in cursor order.
A successful `change()` publishes exactly one atomic revision. If its callback throws, the value and sequence stay unchanged. Draft handles become unusable when the callback returns. Values placed through drafts are copied; `replicatedState(initial)` and `replace(context, value)` take ownership of alias-free strict-JSON roots without copying them, so callers must not mutate them afterwards. Published values are not frozen.
Each subscription serializes its callbacks: hydration settles before updates start. At most 100 deliveries wait behind a running callback; on overflow only the newest pending value is kept, so update sequences may skip. Listener failures go to `onError` (attached states) or are thrown in a microtask (mutable states), and never stop other subscriptions. Unsubscribing discards pending work without aborting the running callback.
## Tracking JSON deltas
Import the standalone transactional tracker from `@earendil-works/chord/delta`:
```ts
import { applyImmutable, track } from "@earendil-works/chord/delta";
@ -121,144 +103,4 @@ tracker.adopt(prepared);
const replica = applyImmutable(prepared.base, prepared.ops);
```
`tracker.value` is always the latest adopted immutable revision. Preparation does
not change authority; adoption validates the preparation and swaps the root pointer.
Assigned containers are copied by value and unchanged subtrees may be shared between
revisions.
The tracker uses trusted immutable ownership rather than defensive copying or
freezing. `track(initial)`, `prepareReplace(value)`, `replicatedState(initial)`, and
`replace(context, value)` take ownership of alias-free strict-JSON roots without
walking them. Callers must not mutate transferred or published data. Values placed
through drafts are already copied, so that walk also rejects non-strict JSON before
the draft changes. Published values are not frozen. In-process loopback consumers
may share their containers with the provider; mutating a consumed value violates
the contract and can corrupt authority. Clone or serialize at any mutable trust
boundary.
Replicated state provides the same model through a callback:
```ts
const initial = { output: "", count: 0 };
const status = env.replicatedState(initial); // transfers ownership of initial
status.change(context, (draft) => {
draft.output += "done\n";
draft.count += 1;
});
```
A successful `change()` publishes exactly one atomic revision. If its callback
throws, the original value and sequence remain unchanged. Draft handles become
unusable when the callback returns. Chord emits string append and front-truncate
operations, array splices and permutations, sets, and deletes; large edit sets may
fold into a complete replacement. Remote connection plumbing encodes each batch
independently for every client/state pairing.
### Public state subscriptions
`state.subscribe(async (value, context, delivery) => { ... })` serializes callbacks
independently for each subscription. Hydration starts immediately when a value is
available, and its returned promise must settle before updates start. Synchronous
callbacks still run synchronously. Read the captured `value` inside an asynchronous
callback: `state.value` may already refer to a later revision.
Each subscription retains at most 100 pending complete-value deliveries, excluding
the running callback. On overflow, only the newest pending value, context, and
delivery metadata are retained; a not-yet-started initial hydration is preserved.
Thus public delivery sequences may skip. This is a frame-count policy, not a byte
limit. Internal exact-operation subscriptions remain synchronous and receive every
revision, independently of slow public callbacks.
The returned unsubscribe function immediately discards pending work and prevents
new callbacks. It neither aborts nor waits for the running callback. Synchronous
throws and promise rejections are observed, reported, and do not stop other
subscriptions or subsequent callbacks. Attached sources and remote bindings use
their `onError` handler; local mutable states (and attached sources without a
handler) report failures by throwing in a microtask. Unsubscribing does not hide a
later rejection from the running callback.
The standalone [Delta guide](src/delta/README.md) defines the complete ownership,
lifecycle, operation, and replica contracts.
## Bundling and loading facets
`@earendil-works/chord/bundler` uses esbuild to turn ESM or TypeScript application
entries into independent, content-addressed CommonJS files. The package-level API
reads plugin identity and build configuration from `package.json`, then applies
facet path conventions supplied by the host application:
```json
{
"name": "@example/my-plugin",
"version": "1.0.0",
"type": "module",
"peerDependencies": {
"@earendil-works/chord": "^0.84.4"
},
"chord": {
"facets": {
"worker": "./src/custom-worker.ts",
"presentation": false
}
}
}
```
```ts
import { bundleFacetPackage } from "@earendil-works/chord/bundler";
await bundleFacetPackage({
packagePath: "/path/to/my-plugin",
outdir: "/application-owned/plugin-builds/my-plugin",
defaultFacets: {
worker: "src/worker.ts",
presentation: "src/presentation.ts",
},
});
```
Existing conventional files become entries unless `chord.facets` overrides or
disables them. Peer dependencies are externalized and resolved against the host
when loading. Chord never installs dependencies or runs package lifecycle
scripts. `bundleFacets()` remains available as the lower-level API for callers
that already have explicit plugin identity and entry mappings.
The output directory contains one `.cjs` file per entry plus
`chord-facets.json`. Load one application-selected entry through the Node-only
loader:
```ts
import { createFacetBundleLoader } from "@earendil-works/chord/node";
const loader = createFacetBundleLoader({
manifestPath: "/application-owned/plugin-builds/my-plugin/chord-facets.json",
entry: "worker",
resolveExternal: (specifier) => import.meta.resolve(specifier),
});
const loaded = await loader.load();
```
Each `load()` verifies SHA-256 integrity and compiles the CommonJS body directly
with `node:vm` instead of putting the plugin into Node's CommonJS or ESM module
cache. Externals are resolved by the host and loaded through a restricted
`require`; esbuild lowers dynamic imports so they use the same path. Disposing a
retired generation releases the loader's facet references, making its compiled
code eligible for garbage collection once plugin-owned resources are also gone.
For transport to another Node host, `readFacetBundleArtifact()` packages one
verified manifest entry with its source, and `createFacetBundleArtifactLoader()`
materializes fresh temporary generations while resolving externals against the
receiving host.
To reload, load a candidate, pass its facets to `FacetHost.reload()`, dispose the
candidate on failure, and dispose the retired `LoadedFacets` only after a
successful cutover. The host activates and validates the candidate while the
currently active providers remain routed, then replaces each singleton directly
without an unavailable interval. Stable service handles therefore do not become
disconnected during an ordinary reload. Keyed instances
remain incarnation-specific and replacements receive fresh generations. The
bundler writes a complete temporary directory before replacing the previous
output, so loaders do not observe partially built generations.
See [PLANNING.md](PLANNING.md) for the broader RPC and generation-loading
architecture.
`tracker.value` is always the latest adopted immutable revision. Preparation does not change authority; adoption validates the preparation and swaps the root pointer. Assigned containers are copied by value and unchanged subtrees may be shared between revisions.