sscoble/pi-mono
1
0
Fork 0
mirror of https://github.com/badlogic/pi-mono.git synced 2026-11-05 23:37:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

docs(chord): slice 6b ready to build; claims per table and namespace

Browse source
This commit is contained in:
Mario Zechner 2026-10-28 15:09:35 +00:00
parent 7f3cfb060d
commit b2fcfee083
4 changed files with 70 additions and 51 deletions
Download patch file Download diff file Expand all files Collapse all files

9
packages/chord/docs/next/decisions.md
View file

@ -1717,8 +1717,8 @@ validation and copies the v2 kernel added around Delta.
declares persisted data (`self.table`, persisted `self.state`, persisted `self.job`); `StoreStorage.open(store, owner,
ctx)` claims `owner`; a second claim of a name is refused (`invalid/args`): a second harness, or a plugin type that
meets the harness's name. `StoreStorage` declares its own tables (`entries`, `tasks`, …) through the store, the path
`self.table` takes, named `"<owner>:<table>"`, and writes its documents under `owner` in the engine's namespace
`durable`. In p3 the harness object passes its own type (`StoreStorage.open(store, Harness.name, ctx)`; base code gives
`self.table` takes, named `"<owner>:<table>"`, and writes its documents under `owner` in the engine's namespaces
(Durable's scopes; corrected 2026-10-27, it was "`durable`"). In p3 the harness object passes its own type (`StoreStorage.open(store, Harness.name, ctx)`; base code gives
the store to the core facet); Durable's conformance suite passes a test name. The owner name plays the role of
`SqliteStorage`'s `prefix`, checked by the store. Objects never see the store: their API writes to the host's store
under their type; a host without a store fails an object declaring persisted data (`broken`, 6a). Rejected: an owner
@ -1731,6 +1731,11 @@ validation and copies the v2 kernel added around Delta.
failed: the object that opened Durable fails `transient`, its retry reopens its natives, Durable included, and attaches
again, as Durable's own recovery is to reopen from Storage) or `ended` (the observed owner ended normally, a task
document's task: final, the member keeps its last value, its watches end `ended`).
- **Slice 6b: claims per table and per document namespace (owner, 2026-10-27: "agree"; refines "owner names and
claims"; Fable's second review N2):** what can collide is claimed once per store: a table by its SQL name, documents by
`(owner, namespace)`; so the object that opens Durable can also declare its own persisted data under the same owner
name, and two harnesses can share a store under two owner names (owner: "seems like they could"); a second
`StoreStorage` on the same owner is refused by its tables. "One harness per store" is no rule, only how p3 uses it.
- **The performance and memory bar, and converging reviews (owner, 2026-10-27: "yes"):** supersedes "as good as
HEAD in everything" for Chord v2 slices and pi changes. CPU: each hot path (calls, changes, watches, links) within
about 5% of HEAD, measured in bursts with alternating processes pinned to one CPU; worse is fixed or goes to the

10
packages/chord/docs/next/design.md
View file

@ -215,7 +215,7 @@ and closed by the app with `store.close(ctx)` after `host.close`; the host never
background, as a tagged `HostError`: `listener_error` (a watch listener threw), `delivery_failed` (a frame delivery
threw), `activation_failed { object, failure }`, `close_failed { object, native, error }`, `close_timeout { object,
native }`, `drain_timeout { object }`, `protocol_error { error }`, `resolve_failed { address, error }`; later routine
errors (slice 5), `derive_failed { object, member?, error }` (a derive's compute threw, once per failure episode, slice 6a), `object_failed { object, failure }` (a live object failing, slice 6a; `failure` where the value is a `FailureCause`, `error` otherwise, 2026-10-27) and `storage_failed { error }`
errors (slice 5), `derive_failed { object, member?, error }` (a derive's compute threw, once per failure episode, slice 6a), `object_failed { object, failure }` (a live object failing, slice 6a, `broken`; `transient` when a `self.follow` source failed, slice 6b R4; `failure` where the value is a `FailureCause`, `error` otherwise, 2026-10-27), `job_interrupted { object, job, id, restarts }` (a persisted run ended at the restart limit, slice 6b D6) and `storage_failed { error }`
(slice 6a), which every app must handle: Chord never ends the process,
the app exits with its own code for its supervisor or reopens.
@ -1024,8 +1024,12 @@ host.serve(DraftStore, (_, self) => { // local, empty params: may
- **Statements per commit (owner, 2026-10-26):** the store keeps `lastSeq` (the seq of its last commit) in memory (read at open,
persisted with each commit, S9) and folds the documents engine's checks into the writes (unique keys, a partial
unique index on the live address, conditional `UPDATE … WHERE id = ? AND version = ?`): a delta commit is about three
statements (revision, the record's counted conditional `UPDATE`, `lastSeq`), plus one `INSERT` with an outbox. Durable's ids in the store: `StoreStorage` keeps a `record_ids` table (one
namespace across Durable's five tables), its counter at open `coalesce(max(id), 1) + 1`.
statements (revision, the record's counted conditional `UPDATE`, `lastSeq`), plus one `INSERT` with an outbox. Durable's ids in the store (slice 6b D3): one id allocator per store
for every id (Durable's `mintId` and Chord's document ids; it starts at 2, saturates at `Number.MAX_SAFE_INTEGER`, each
commit persists it, outbox commits and archives carry it as `nextId`); `StoreStorage`'s `record_ids` table only checks
an id's record type. Every table and document namespace in the store is claimed once (slice 6b "owner names and
claims"): a table by its SQL name `"<owner>:<table>"`, documents by `(owner, namespace)`; an object claims under its
type, `StoreStorage.open(store, owner, ctx)` under `owner` (two harnesses on one store under two names are fine).
- **Tables in a transaction (owner, 2026-10-26):** the table handle first, keys as objects with exactly the key
columns, reads async, writes synchronous and buffered like a draft's ops (checked and staged, no I/O; written in the
commit step or dropped on failure); no write shorthands on the handle:

70
packages/chord/docs/next/slice-6b.md
View file

@ -1,8 +1,8 @@
# Slice 6b: Durable and jobs on the store (spec)
Status: **revised after the first spec review** (2026-10-27): D1-D9 and R1-R4 decided by the owner (decision log,
entries "Slice 6b …" and "owner names and claims"); the four reviews (`/tmp/s6b-review-{astra,fable,opus,sol}.md`)
kept every decision and found the gaps this revision closes. Written by the redesign session after reading, in full,
Status: **ready to build** (2026-10-27): D1-D9, R1-R4 and the claims decided by the owner (decision log, entries
"Slice 6b …" and "owner names and claims"); two spec reviews (`/tmp/s6b-review{,2}-{astra,fable,opus,sol}.md`)
kept every decision; this version closes their findings. Written by the redesign session after reading, in full,
Durable's `types.ts`, its SQLite storage, schema and migrations, the storage conformance suite and benchmark,
`session/session.ts`, `session/transaction.ts`, `session/forks.ts`, the scheduler's restarts, `writeAtClose` and `abort`,
and Chord v2's store, engine, applier, commit step and jobs. Durable's memory, JSONL and IndexedDB storages stay as they
@ -53,10 +53,14 @@ Every item below changes 6a's code; nothing is deployed, so the schema and the a
- Tables are `"<owner>:<table>"` (`"notes.store:edits"`), document owners `"<owner>"`; no host kind. The applier and
import check two name parts.
- Every owner name is claimed once per store (in memory, for the store's lifetime): an object claims its type when it
first declares persisted data; `StoreStorage.open` claims its name; a second claim is refused `invalid/args`; a
claim is released by its owner's end (`StoreStorage.close`, the object's end), so a new Session can reopen over the
same store.
- **Claims** (owner, 2026-10-27, decision "owner names and claims", refined): what can collide is claimed once per
store, in memory: a table by its SQL name, documents by `(owner, namespace)`. An object claims its tables and
`(type, "chord")` (plus `chord.jobs` and `chord.job-states` with persisted jobs) when it declares them;
`StoreStorage.open` claims its five tables and its document namespaces (Durable's scopes contain a `:`, Chord's
namespaces never do). A second claim is refused `invalid/args`; a claim is released by its owner's end
(`StoreStorage.close` after its calls underway drained, the object's end), so a new Session can reopen over the same
store. So the object that opens Durable can declare its own persisted data under the same owner name, and two
harnesses can share a store under two owner names.
### 1.2 One id allocator (D3)
@ -67,8 +71,10 @@ Every item below changes 6a's code; nothing is deployed, so the schema and the a
- It starts at 2 (id 1 is Durable's root conversation). It saturates at `Number.MAX_SAFE_INTEGER`: an explicit write of
that id is accepted, the persisted value stays `MAX_SAFE_INTEGER`, every later `mintId` rejects "ID space is
exhausted", and a Chord creation at the saturated allocator refuses its own commit (1.4).
- Outbox commits carry the allocator's value after them (`{ seq, nextId, writes }`); the archive manifest's
`nextDocumentId` is `nextId`; import and batch apply restore it, so a replica mints above every id it holds.
- Each commit takes the allocator's value once, after its writes applied, for the outbox commit (`{ seq, nextId,
writes }`) and the metadata row (mints meanwhile only raise it further); an export reads it once with `lastSeq`; the
archive manifest's `nextDocumentId` is `nextId`; import and batch apply restore it, so a replica mints above every id
it holds.
### 1.3 The commit step
@ -93,13 +99,14 @@ Every item below changes 6a's code; nothing is deployed, so the schema and the a
outcome: the store stops (`storage_failed`). The Cloudflare facade is checked to keep the same identity; a test
injects failures at `BEGIN`, in the body, at `COMMIT` and at `ROLLBACK` on both drivers' facades.
- **The refusal's error reaches the owner unchanged** (Durable's suite matches its messages); the store classifies it
for Chord's owners: an `int` `CHECK` failure (SQLite's extended code `SQLITE_CONSTRAINT_CHECK`) is `invalid/args` and
for Chord's owners: an `int` `CHECK` failure (SQLite's extended code `SQLITE_CONSTRAINT_CHECK` on Node; the error
message on the Durable Object facade, which has no extended codes) is `invalid/args` and
fails only that transaction; every other refusal of a Chord commit (the engine's counted `UPDATE`/`DELETE`, the
live-address index, a primary key, the saturated allocator, a backend error in a statement) fails the object live
(`object_failed`, `broken`: memory and disk disagree).
- **6a's owners:** a refused `self.tx` rejects with the error above and the object fails as above; a refused
migration commit (a chunk, (A) or (Z)) fails the activation `broken`, the table stays `migrating`, committed chunks
stay; a refused job start rejects the start; a refused outcome or finish commit fails the object (records and
migration commit (a chunk, (A) or (Z)) is rolled back whole, fails the activation `broken`, the table stays
`migrating`, earlier committed chunks stay; a refused job start rejects the start; a refused outcome or finish commit fails the object (records and
memory disagree); a refused commit that carried held changes (D8) fails the object.
- **Reads:** a read's callback returns data, never throws for a request: an adapter's request checks (an unknown
conversation, a cursor, history a document does not keep) happen after the guarded read returned, from tagged
@ -108,9 +115,9 @@ Every item below changes 6a's code; nothing is deployed, so the schema and the a
### 1.5 The documents engine
- **Addresses:** `(owner, namespace, kind, key?)`: `owner` the claimed name; `namespace` the adapter's partition
(Chord: `chord` for documents, `chord.jobs` and `chord.job-states` for persisted runs; Durable: its scope,
`session`, `conversation:<id>`, `task:<id>`); `kind` and `key` as the adapter writes them (Durable JSON-encodes both,
- **Addresses:** `(owner, namespace, kind, key?)`: `owner` the owner name; `namespace` the adapter's partition, claimed
with the owner (Chord: `chord` for documents, `chord.jobs` and `chord.job-states` for persisted runs; Durable: the
document's scope, `session`, `conversation:<id>`, `task:<id>`, which Durable's reads by scope need); `kind` and `key` as the adapter writes them (Durable JSON-encodes both,
as its indexed strings). One live record per address (the partial unique index).
- **`history` per record (R1):** `none` (Chord: older revisions dropped at a new base; row and revisions deleted at
retirement), `latest` (Durable's session, task and latest conversation documents: older revisions dropped; the row
@ -130,12 +137,13 @@ Every item below changes 6a's code; nothing is deployed, so the schema and the a
- **Format 1 gains** (nothing deployed): `document.create` carries `history`, `retiredAt?` and `meta`; a
`document.revision { id, seq, kind, version, content }` part per kept revision, in seq order after its create; for
`none` and `latest` records only the current base and its deltas are exported (as 6a), for `rewindable` every
revision, retired records included. Import checks: revisions in seq order, a base first, `seq <= lastSeq`, lifetimes
live `none` and `latest` records only the current base and its deltas are exported (as 6a); a retired `latest` record
is exported as its record alone (no base: its content was dropped at retirement); for `rewindable` every revision,
retired records included. Import checks: revisions in seq order, a base first, `seq <= lastSeq`, lifetimes
at one address never overlapping (a second live record is still refused), versions per revision; the `end` count
includes revisions.
- **Batch apply** (`applyBatch(store, commits, ctx)`, internal): only on a store with no attached host (no owner's
memory to update); `commits[0].seq` follows `lastSeq` (S5); one backend transaction for the batch, every write applied
- **Batch apply** (`applyBatch(store, commits, ctx)`, internal): refused while any owner holds a claim on the store
(an object or a `StoreStorage`: its memory would go stale) and run through the store's commit queue; `commits[0].seq` follows `lastSeq` (S5); one backend transaction for the batch, every write applied
with its commit's seq, `lastSeq` and `nextId` set from the last commit; the applier handles every `StoreWrite` kind
(`alter`, `dropIndex`, `update`/`updateWhere`/`delete`/`deleteWhere` with their recorded counts and versions,
`document.change`, `document.retire`, `document.revision`), checked against the store's records (a trust boundary);
@ -154,10 +162,11 @@ every commit, Durable's reopen harness), and Durable's Session and harness test
`StoreStorage.open(store, owner, ctx)`: claims `owner` (1.1); declares its tables through the store's table
preparation (the path `self.table` takes: created at the newest version or migrated); raises the allocator to
`max(record_ids.id) + 1` (one index read; covers stores built by import or batch apply); attaches to the store's host
`max(record_ids.id) + 1`, saturating at `Number.MAX_SAFE_INTEGER` (one index read; covers stores built by import or
batch apply); attaches to the store's host
batch if there is one. Chord exports for it: `StoreOwnerTables` (declare and prepare tables under an owner),
staged table writes and engine writes, the step reader, the read transaction, `mintId`, the commit with a fence. Its
`close(context)` releases the claim and refuses every later call ("StoreStorage is closed"); the store stays open (the
`close(context)` refuses new calls ("StoreStorage is closed"), waits for its calls underway, then releases its claims; the store stays open (the
app owns it).
### 2.2 Tables
@ -183,8 +192,11 @@ adopt: (seq) => adopt(seq) }` (adopted in the host's batch, 1.3). Inside the ste
another record type, a conversation, entry or document written twice: Durable's messages), the per-document actions
of `prepareDocumentActions`/`checkDocumentActions` (one content command per document, a delta on its version, one live
incarnation per address: "already has a current incarnation"), then the table `put`s (`record_ids` only for ids not yet
claimed by the same type), then per document: its creation (explicit id) or content, then its retirement; a
create-plus-retire is an empty lifetime (`createdAt = retiredAt = seq`). Copies (`document.copy`) are resolved before
claimed by the same type), then the documents in two passes: first every document that already exists (its content,
then its retirement), which frees the addresses a creation of this commit takes, then the creations and copies (each
with its retirement, a create-plus-retire being an empty lifetime, `createdAt = retiredAt = seq`). Durable's
"reconstructs rewindable documents" case (`create secondId` at address A, `retire firstId` at A, `change firstId`)
needs this order. Copies (`document.copy`) are resolved before
the step (S11): the source read at its point, checked against the copy's record ("changed in the copy batch",
kind, key, history, fork), entered as a creation carrying the base. A refused check rejects with Durable's message (a
refusal, 1.4): the store goes on, Durable's Session fails (`GuardedStorage`).
@ -219,7 +231,8 @@ Durable from other objects is slice 8's (the harness in a registry entry, carry-
the source in that batch; a call outside any batch (Durable's deferred commits, `#hold`, and the task graph's
`blocked` refresh, even on the store; any Durable storage that is not the store) is queued and one batch on the next
microtask applies everything queued. A derive over two views of one commit sees both changes in one computation.
- **Ends (R4):** `failed` fails the object `transient` (its retry reopens Durable and attaches again); `ended` is final:
- **Ends (R4):** `failed` fails the live object `transient` (reported once as `object_failed` with that failure, its
identity's backoff, several sources ending at once fail it once; its retry reopens Durable and attaches again); `ended` is final:
a pending retirement call reaches the source's watches first (Durable's `endAfterDelivery`), then they end `ended`;
the member keeps its last value; a new `watch` on an ended follow source opens at that value and ends `ended` at once.
@ -258,16 +271,19 @@ rebuilt as the in-memory one.
| `finish` fails | record `finishFailures + 1`; at 5 `blocked: "finish_failed"` | retried after the backoff, a blocked run on reload |
| `finish` succeeds | record deleted, state retired (Q9), with what `finish` wrote | the run settles |
| clean stop (host close, a native restart, R3's seal) | the stopped runs written `stopped`, held state flushed, by the close writer | |
| activation (once per object, not per retry) | each `running` record: `restarts + 1` (or, at 5, `failed { restart_limit }` and `finishing`, `job_interrupted` reported); `stopped` and `pending`: back to `pending` | maps, keys, active count, `running` rebuilt before anything runs; `rerun` invokes, `abandon` records `failed { aborted, restart }`; an abort-marked run finishes `aborted` without `run`; `finishing` resumes `finish`; `blocked` stays |
| activation (records loaded once per object; each live attempt resumes what waits) | each `running` record: `restarts + 1` (or, at 5, `failed { restart_limit }` and `finishing`, `job_interrupted` reported); `stopped` and `pending`: back to `pending` | maps, keys, active count, `running` rebuilt before anything runs; `rerun` invokes, `abandon` records `failed { aborted, restart }`; an abort-marked run finishes `aborted` without `run`; `finishing` resumes `finish`; `blocked` stays; a record of another job version, or of the same version with another fingerprint, is `blocked: "version"` |
- **A retry after a seal (R3):** the runs the seal stopped are resumed by the next live attempt only once their old
invocation settled (bounded by `closeTimeout`, as the drain; an invocation outliving it is fenced, its run resumes
then), so a `run` that ignores its signal never runs twice at once.
- **The close writer:** at the object's end, after the drain and before the documents drop, one commit past the seal
(Durable's `writeAtClose`): the runs stopped by the end written `stopped`, held state flushed; a run whose `run`
outlived the drain is fenced and written `stopped` too; blocked persisted runs keep their records (dropped only by
`abort`). In-memory runs keep 6a's rules.
- **Pinning:** records and state documents of unsettled runs stay in memory (never evicted): `ctx.state.value` and
`running` read them synchronously.
- **Fingerprint:** the job's fingerprint (args, result, state) is stored; a record of the job's version with another
fingerprint is `blocked: "version"` (resolved by slice 7).
- **Fingerprint:** the job's fingerprint (args, result, state) is stored; a mismatch is `blocked: "version"` (resolved
by slice 7). A run ending at the limit reports `job_interrupted { object, job, id, restarts }` (a new `HostError`).
### 4.3 Deferred state (D8)

32
packages/chord/docs/next/slices.md
View file

@ -163,25 +163,16 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over.
### Slice 6b: Durable and jobs on the store (from slice 5 and L3a-d)
- **Persisted jobs** (from slice 5): `persist: { resume: "rerun" | "abandon" }` built (record and state document in
the owner's store; only in `local` objects with empty params, P3, enforced at declaration); outcomes and abort
requests persisted (a recorded outcome resumes only `finish`); the restart limit (five crash interruptions,
persisted, clean closes not counted; whether a state change resets the count; what a clean close records); deferred
persistence for state documents (flushed before the outcome is recorded); the `finish` failure count persisted; the
job fingerprint stored with every record; the run's start trace carrier (§29).
- **`StoreStorage`**: Durable's `Storage` on the store, passing Durable's storage conformance suite (memory and across
reopen); ids from a `record_ids` table (S10, W13); fork copies resolved before the step (S11); Durable's `adopt` run
inside the host's batch.
- **`self.follow`** (was `durableView`, decision "Slice 6b D9"): a Chord source over Durable's synchronous observers (`observeConversation`, `observeTaskGraph`,
`observeDoc`; decision "Durable's synchronous observers"), marked inside Chord's batch in the store and grouped into
one batch per Durable commit on the next microtask outside it; `self.follow` (working name) and the Durable harness
carried in a registry entry. The task graph's `blocked` refresh arrives outside any commit.
- **The applier for batch apply** (slice 6a builder, 2026-10-27): 6a's applier (`store/applier.ts`) applies the
writes an archive holds (`table`, `put`, `index`, `document.create`), checked against the store's records; the
replica's batch apply extends it to every `StoreWrite` kind (`alter`, `dropIndex`, `update`/`updateWhere` with their
recorded counts and `v`, `delete`/`deleteWhere`, `document.change`, `document.retire`), one code path with import.
- Not carry-over any more: Durable's synchronous view-change hook is the upstream "Durable observers" change
(publication at adoption, `observeXXX`), which lands before 6a.
Specified in `slice-6b.md` (decided 2026-10-27, D1-D9, R1-R4, "owner names and claims"); each entry below is taken there.
- **Persisted jobs** (from slice 5): `persist: { resume }`, records and state documents in the owner's store (D5), the
restart limit (D6), `abort` stored (D7), deferred state (D8), the seal as a clean stop (R3), the `finish` failure
count persisted, the job fingerprint and the start trace carrier stored.
- **`StoreStorage`** (D1-D4, R1, R2, claims): Durable's `Storage` on the store, passing its conformance suite; one id
allocator (D3); Durable's checks in the step; fork copies resolved before the step (S11); `adopt` in the host's batch.
- **`self.follow`** (D9, R4): a Chord source over Durable's synchronous observers. Reaching Durable from other objects
(the harness in a registry entry) moves to slice 8.
- **The applier for batch apply**: every `StoreWrite` kind, `applyBatch`, one code path with import.
### Slice 7: reload and rollout
@ -233,6 +224,9 @@ deleted (the slice's spec and code are the record). Moved here from `carry-over.
### Slice 8: registries and `host.root`
- **Durable's views from other objects** (from slice 6b): the harness carried in a registry entry, so an object other
than the one that opened Durable can `self.follow` its views.
- **`host.root(fn, context, { principal? })`**: a call scope for code outside objects, built with its first users
(Durable hooks, `chord(fn, context)`); decide what an aborted, uncooperative `fn` means for its references (match
the handler and `CallContext` rules). (Slice 3b, owner.)