sscoble/pi-mono
1
0
Fork 0
mirror of https://github.com/badlogic/pi-mono.git synced 2026-11-05 23:37:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

feat(durable): group commit and batch Storage.commit

Browse source 
A Session no longer holds its line while a commit is stored. Commits that
arrive while one storage call is in flight go out together in the next call,
each with its own seq and outcome; a job runs until it touches a key an
unpublished commit wrote, then waits for that commit's publication. Outcomes
are delivered in line order; publication is in line order, one commit per
macrotask. The scheduler's decisions run as regions that record the keys they
read and rerun after a wait, with side effects only after a clean check.

Storage.commit(batches, receive, context) replaces commit(writes, adopt,
context): prefix semantics (Durable's own checks refuse a batch and keep the
ones before it; any other failure rejects the call), receipts per stored
batch, optional publishing() scope. Memory, SQLite, JSONL, IndexedDB and
StoreStorage implement it natively; StoreStorage through the new
ChordStore.commitSeries (one store transaction per call), replacing
commitInBatch.

A normal run makes no storage reads: bounded record cache, conversation
tails, task-document index, kept context ranges. A tool slot's running status
and a generation's attempt are stored with the next commit (12 instead of 14
commits per run).

One Harness with 128 conversations over storage with a 4 ms round trip: about
5 -> 440 agent runs per second. Design: packages/durable/docs/group-commit.md.
This commit is contained in:
Mario Zechner 2026-10-30 10:40:53 +00:00
parent 908aa7b74b
commit e5c6182bab
81 changed files with 8626 additions and 1475 deletions
Download patch file Download diff file Expand all files Collapse all files

49
packages/durable/CHANGELOG.md
View file

@ -5,7 +5,41 @@
### Breaking Changes
- `SqliteExecutor` (re-exported from Chord) gains `runCounted(sql, ...params)`, resolving with the rows the statement changed; custom `SqliteDatabase` implementations add it.
- `Storage.commit(writes, adopt, context)` takes an `adopt(seq)` callback, the Session's in-memory adoption and publication of the batch: a storage calls it exactly once, synchronously, once the batch is durable and before its next commit starts, then resolves with `seq`, and never for a commit that fails before the batch is durable. Memory, SQLite, JSONL, and IndexedDB storages do; custom storages and wrappers must call or forward it, and the storage conformance suite checks it. A commit that rejects after its adoption is published and fails the Session without rolling the adopted changes back.
- `Storage.commit()` stores a group of commits in one call: `commit(batches, receive, context)` resolves with a `CommitResult` (`{ stored, refused? }`) and replaces `commit(writes, adopt, context)`. Each batch is one atomic commit with its own seq, checked against the state the batches before it left. The first batch Durable's own checks refuse (global IDs, document actions, copy sources) is not stored, nor anything after it; the batches before it are, and the call resolves with their number in `stored` and the refusal in `refused`. Any other failure (a driver or I/O error, a failed `COMMIT`) rejects the whole call as an unknown outcome and stores nothing the backend could roll back. `receive(index, seq)` is called once per stored batch, in order, synchronously, once they are durable and before the next call starts; receipts made before a rejection stand. Every batch has at least one write. The optional `publishing(publish, context)` runs each commit's publication inside the storage's own scope (`StoreStorage`: the attached host's batch). Memory, SQLite, JSONL, IndexedDB and `StoreStorage` store a call in one transaction or append, and the storage conformance suite checks the contract. Migration for a custom Storage:
```ts
// Before
async commit(writes, adopt, context) {
const seq = await this.transaction((tx) => {
check(tx, writes); // throws when Durable's rules refuse the commit
return apply(tx, writes);
});
adopt(seq);
return seq;
}
// After
async commit(batches, receive, context) {
let refused: { error: unknown } | undefined;
const seqs = await this.transaction((tx) => {
const stored: Seq[] = [];
for (const writes of batches) {
try {
check(tx, writes); // only Durable's own checks; reads the earlier batches' writes
} catch (error) {
refused = { error };
break; // keep the batches before it
}
stored.push(apply(tx, writes)); // a throw rejects the call and rolls all of it back
}
return stored;
});
seqs.forEach((seq, index) => receive(index, seq));
return refused === undefined ? { stored: seqs.length } : { stored: seqs.length, refused: refused.error };
}
```
A database error during the checks is not a refusal: it must reject the call. A wrapper forwards `commit()` and `publishing()`. Code that called `storage.commit(writes, adopt, context)` directly, such as a test seeding a storage, calls `commitWrites(storage, writes, context)` from `@earendil-works/pi-durable/testing` (one batch; resolves with its seq, throws the refusal).
- `Storage.mintId()`'s contract is narrowed to IDs greater than every ID minted since the storage was opened and every ID stored, also across reopen. The bundled storages already behave so.
- Watches use Chord's `WatchHandle` and `WatchEnd` (Durable re-exports both, with `WatchChange` and `NativeWatchChange`). End reasons are `stopped`, `cancelled`, `ended` (was `session_closed`), `listener_error`, and `failed` (was `session_failed`); `retired` is gone (see below). An ended watch drops its listener and every reference into its Session, so a stopped watch that is still referenced keeps only its last value.
- Watch listeners take one change object before the context: `watch.start(async ({ value, ops, retired }, context) => …)` for document, conversation view (`Conversation.watch()`), and task graph watches. Migration: `start(async (value, ops, ctx) => …)` becomes `start(async ({ value, ops }, ctx) => …)`.
@ -102,9 +136,9 @@
### Added
- `StoreStorage` (`@earendil-works/pi-durable/storage/store`): Durable's `Storage` on Chord v2's store. `StoreStorage.open(store, owner, context)` keeps a Session's records as declared tables of the store under `owner` and its documents in the store's documents engine, takes its ids from the store's one allocator, and commits each batch as one write of the store's commit step, adopted in the batch of the host the store serves, so a Chord object and a Durable Session commit through one queue and one outbox. It passes the storage conformance suite; `close()` leaves the store open (the app owns it).
- `StoreStorage` (`@earendil-works/pi-durable/storage/store`): Durable's `Storage` on Chord v2's store. `StoreStorage.open(store, owner, context)` keeps a Session's records as declared tables of the store under `owner` and its documents in the store's documents engine, takes its ids from the store's one allocator, and stores each `commit()` call in one store transaction (`ChordStore.commitSeries`), one seq and outbox row per batch, publishing each commit in the batch of the host the store serves, so a Chord object and a Durable Session commit through one queue and one outbox. It passes the storage conformance suite; `close()` leaves the store open (the app owns it).
- Deferred commits: `Session.commit()`, `Conversation.commit()`, `TaskRuntime.commit()`, and a tool's `api.commit()` take `{ persistence: { kind: "deferred", withinMs? } }`. A commit that changes only existing conversation documents with `history: "latest"` and `fork: "initial"` is then adopted in memory and shown to watches at once, and stored with the next commit that reads or writes the document, before the document leaves memory, at close, or within `withinMs` (default `settings.progress.persistWithinMs`, or `createSession(storage, { persistWithinMs })`); any other write makes it immediate. A crash loses what is held. See spec §4.1.
- Synchronous observers for adapters, such as a Chord source over a Durable view: `Session.observeDoc(token, ...scope, observer, context)`, `Harness.observeConversation(id, observer, context)`, and `Harness.observeTaskGraph(observer, context)` attach an `Observer<T>` (`advance(change, context)`, `end(failure)`) atomically on the Session line, like a watch, and return an `Observation<T>` (`value`, `detach()`; `observeDoc` gives `undefined` for an absent document). `advance` gets every change unfolded as memory adopts it, for a stored commit inside Storage's `adopt` and for a deferred one before it is stored, so it must only record it: no awaits, no Session calls, no throws. A document observer gets one call per change at its address (two for a commit that retires and recreates it), the task graph also a call outside any commit when a registry change or a failed migration changes a `blocked` reason, and a change can arrive before the attaching promise resolves. The attaching context governs only the attachment; a closed Session or an invalid argument rejects the promise. One that throws is reported as `listener_error`, detached, and ended with the error. `end` comes once, at Session close or failure, or after a task document's retirement with its task. Document, view, and task graph watches are built on them (spec §9.1).
- Synchronous observers for adapters, such as a Chord source over a Durable view: `Session.observeDoc(token, ...scope, observer, context)`, `Harness.observeConversation(id, observer, context)`, and `Harness.observeTaskGraph(observer, context)` attach an `Observer<T>` (`advance(change, context)`, `end(failure)`) atomically on the Session line, like a watch, and return an `Observation<T>` (`value`, `detach()`; `observeDoc` gives `undefined` for an absent document). `advance` gets every change unfolded as memory adopts it, for a stored commit when it is published after its storage call settled, and for a deferred one when memory adopts it, before it is stored, so it must only record it: no awaits, no Session calls, no throws. A document observer gets one call per change at its address (two for a commit that retires and recreates it), the task graph also a call outside any commit when a registry change or a failed migration changes a `blocked` reason, and a change can arrive before the attaching promise resolves. The attaching context governs only the attachment; a closed Session or an invalid argument rejects the promise. One that throws is reported as `listener_error`, detached, and ended with the error. `end` comes once, at Session close or failure, or after a task document's retirement with its task. Document, view, and task graph watches are built on them (spec §9.1).
- `ConversationTranscript` keeps what a client has of one conversation over any `ConversationView`: `merge(view)` returns the view with `items`, the client's whole transcript (entries, a `gap` where a later window did not continue it, and an `earlier` marker until the start is loaded), instead of the window; `loadEarlier(context)` and `fillGap(gap, context)` page entries through a loader such as `Conversation.entries()`, and `resume` is the window that continues it after a drop: `conversation.watch(context, transcript.resume)`. A merge costs O(items) and returns the same `items` array when it adds nothing; concurrent loads of the same page are shared ([#11327](https://github.com/earendil-works/pi/issues/11327)).
- `pi.live.runFailure` (`RunFailure`, `RunFailureReason`) records why the newest run ended without an answer, `model_error`, `no_model`, `aborted`, `reset`, `faulted`, or `orphaned`, with the run's task and inputs and a `detail` where one exists (the error text, the missing model, the fault, the orphan reason). The commit that ends the run sets it; a later commit that starts a run removes it ([#11327](https://github.com/earendil-works/pi/issues/11327)).
- `RunFailureReason` gains `interrupted` (the host stopped while the model was answering, with no retry left) and `restart_limit` (the run's task was cut off `settings.restartLimit` times in a row) ([#11139](https://github.com/earendil-works/pi/issues/11139)).
@ -122,8 +156,13 @@
### Changed
- Document, conversation view, and task graph watches use Chord's watch handles (`Watch`).
- A stored commit is published when Storage adopts it, inside `Storage.commit()`: the commit's telemetry events and Durable's own observers (document watches, conversation views, the task graph, the scheduler, and waits) run inside `adopt`, so two views never show one commit apart. `subscribeCommits()` listeners run one microtask later, outside the storage's synchronous step, before every wait the commit resolves and every abort signal it fires; they have no order relative to watch listeners and `onWake`. With a storage that works after adopting (JSONL's sidecar reclaim), all of these run before `Storage.commit()` resolves.
- `subscribeCommits()` listeners are read when a commit's dispatch runs: a listener added between adoption and the dispatch gets the commit only when one was registered at adoption; one added after the dispatch ran misses it.
- Group commit: a Session no longer holds its line while a commit is stored. Commits that arrive while one storage call is in flight go out together in the next call, each with its own seq and its own outcome, so the conversations of one Session share storage round trips instead of queueing behind each other's. A job runs while earlier commits are being stored until it touches a key one of them wrote (a record, a document, a conversation's entries, a task's membership or cancellation state); then it waits until that commit is stored and published. Outcomes are delivered in line order, and every job after a commit that fails the Session gets `SessionFailed`. One Harness serving 128 conversations over storage with a 4 ms round trip went from about 5 to about 440 agent runs per second (median run 25 s to 0.28 s); with local SQLite or the Chord store, throughput at 128 conversations rises by about 30%, while with in-memory storage and no latency each commit costs somewhat more CPU than before (up to 25% fewer runs per second). At most 256 jobs wait for their outcome. See `docs/group-commit.md` and spec §4.
- A stored commit is published after its storage call settled, in line order, the commits of one call a macrotask apart (inside the storage's `publishing()` scope if it has one): the read caches and memory adopt it, then its telemetry events and Durable's own observers (document watches, conversation views, the task graph, the scheduler, and waits) run, so two views never show one commit apart. `observeCommits()` observers run one microtask later, before every wait the commit resolves and every abort signal it fires; they have no order relative to watch listeners and `onWake`. A call that settles after something else failed the Session publishes nothing; what it stored is durable and appears after reopen.
- A normal agent run makes no storage reads: the Session keeps recently read conversation, task and submission records (bounded by count, and by live work for tasks), the newest entry of each conversation, and the task documents of live tasks, and the scheduler keeps the context range of a running conversation. Read results (`tx.task()`, `tx.conversation()`, `getTask()`, `runtime.entry()`, published entries) are frozen objects shared with these caches; mutating one throws.
- A tool slot's `running` status and a nested call's final arguments, and a generation's attempt, are stored with the next commit, like streamed progress: a run with one tool call stores 12 commits instead of 14. After a crash a slot may show `pending` while its task's checkpoint is at execution; the rerun sets it again.
- A wait whose commit is still being stored when the Harness closes ends with the closed error; the commit is still stored and published before close completes.
- Cancellation is checked when a job starts, which can be earlier than before: an invocation-bound submit queued behind an abort mark that is not stored yet lands, and the abort's cascade then aborts its run.
- `subscribeCommits()` listeners are read when a commit's dispatch runs: a listener added between publication and the dispatch gets the commit only when one was registered at adoption; one added after the dispatch ran misses it.
- The abort signal of a newly marked running task fires after `subscribeCommits()` listeners instead of before them.
- A throwing Durable commit observer fails the Session after `Storage.commit()` returned, after the other observers and the host listeners got the commit; the commit still stands. When Storage also rejects that commit after adopting it, the storage error fails the Session and the observer's error is reported as `listener_error`.

2
packages/durable/README.md
View file

@ -889,7 +889,7 @@ await database.transaction(async (transaction) => {
});
```
A custom backend's `commit(writes, adopt, context)` calls `adopt(seq)` once the batch is durable, synchronously and before its next commit starts, and then resolves with `seq`; a commit that fails before the batch is durable never calls it. `adopt` is the Session's in-memory adoption and publication of the batch, so memory and every view never show a partial log: inside it the Session's own observers run (views, task graph, scheduler, waits), and `subscribeCommits()` listeners, watch listeners and the code the commit wakes follow in microtasks, so at the backend's next `await` if it still works after `adopt` (as JSONL reclaims sidecars), before its `commit` resolves. `adopt` calls no Storage method and never throws; once it ran, the commit is published even if the backend then rejects, and the Session then fails. Custom backends can run the shared conformance suite, which checks this too, with any Vitest- or Jest-compatible runner:
A custom backend's `commit(batches, receive, context)` stores `batches` in order, each as one atomic commit with its own seq, checked against the state the earlier ones left, in as few round trips as it can. It stops at the first batch its own checks refuse (global IDs, document actions, copy resolution): the batches before it stay stored, it and the later ones are not. Once they are durable it calls `receive(index, seq)` per stored batch, in order and synchronously, then resolves with `{ stored, refused? }`; it rejects for an unknown outcome (connection lost, a failed statement or `COMMIT`). `receive` is a durability receipt: the Session publishes each batch later, one at a time, so memory and every view never show a partial log. `receive` calls no Storage method and never throws. Custom backends can run the shared conformance suite, which checks this too, with any Vitest- or Jest-compatible runner:
```typescript
import { registerStorageConformance } from "@earendil-works/pi-durable/testing";

296
packages/durable/docs/group-commit.md Normal file
View file

@ -0,0 +1,296 @@
# Group commit for the Durable Session
Status: design, revision 2 (after three reviews). Branch `durable-perf` (worktree `~/pi-perf` on slayer), on top of the
uncommitted prerequisites (record cache, conversation tails, kept context ranges advanced at publish, task-document
index, deferred tool-slot `running` and generation attempt). Read `docs/spec.md` §1, §4, §4.1, §5, §8, §10.1 alongside.
## 1. Problem
A Session has one mutation line (`SessionImpl.#tail`). Each commit runs its callback, settles, calls `Storage.commit()`,
and adopts while holding the line, so every storage round trip of every conversation is serialized. After the
prerequisites the line makes no storage reads in a normal run; a run that calls one tool stores 12 commits. Measured on
slayer (one Harness, faux model, MemoryStorage with a simulated round trip per call, 32 concurrent conversations):
26.8 runs/s at 4 ms, 84 at 1 ms, 895 at 0 ms (CPU, one core). Throughput is fixed by the line.
Group commit sends the commits that accumulate while one write is in flight as one storage call. Modelled with result
fences (each job on a run's critical path waits about one round trip): ~250 runs/s at 4 ms with 32 conversations,
rising toward the CPU ceiling with more conversations. A conflict probe (`/tmp/conflict`) measured, per run, 0 writing
jobs that would wait for one tool call or a streamed answer, 0.13 for three parallel tools.
## 2. Semantics
### Unchanged
- Every logical commit keeps its own `seq`; history (`snapshotAsOf`, fork `asOf` copies, `createdAt`/`retiredAt`)
keeps per-commit boundaries.
- A commit's promise resolves only once it is durable and published; outcomes are delivered in line order (§5.1).
- Publication is in line order, deferred commits included (§5.2): telemetry events, Session listeners, the
`observeCommits()` dispatch, waits and aborts per commit; host observers of commit A run, and the continuations of
public waits A resolves start, before commit B is published (§5.3).
- No callback observes unstored state (§4). Memory adopts a commit at its publication.
- A commit adopted by Storage and then rejected stays published and rejects, as today (§3.4).
- Storage failure is fatal; callers are never told something storage does not hold.
### Changed (accepted)
1. A callback can run while an earlier commit's storage call is in flight. It runs until its first touch of a key an
unpublished commit wrote, waits until that commit's storage call settled and it is published, then continues. Code
before that touch runs early, assignments to caller variables included; acting on such values before the commit
resolves was already unsafe (a callback's own writes can fail to store today).
2. Cancellation is checked when a job starts; a job can start earlier.
3. If a batch fails fatally, later callbacks may already have run; their outcomes are `SessionFailed` (§5.1).
4. On a path that fails the Session, which commits survive differs: a stored prefix survives. Every caller's outcome
matches storage. On a bug path (adoption or an internal listener of part i throws) parts after i can be durable but
unpublished; their commits reject with `SessionFailed`; reopening recovers them. Likewise, once another error has
failed the Session, a call that settles afterwards, and the parts of a call not yet published when it fails, are
durable but unpublished and reject with `SessionFailed` (the first error); an `observeCommits()` consumer, such as a
replica, misses them until reopen.
5. Read results (`tx.task()`, `tx.conversation()`, `getTask()`, `runtime.entry()`, entries in published commits) are
frozen shared objects; mutating one throws (documented on `Tx`).
6. Prerequisite presentation changes (spec §8.2, §8.4 updated): a tool slot's `running` and a nested slot's final
arguments, and the generation's attempt, are stored like progress; after a crash a slot may show `pending` while
the task's checkpoint is `execute` (a safe rerun sets it again).
7. A wait whose commit is in flight when close begins ends with the closed error; the commit is still stored and
published by the drain (today this already happens when close lands before adopt).
8. Spec text to update: invariant 7, §4 (host listeners run while the line is held), §4.1 (holds leave at the last step
before `Storage.commit()`: now at logical admission), §10.1 (per-part outcomes), the `observeCommits` doc, the
`SessionImpl` class doc, the scheduler class invariant (decisions stay serialized; their storage overlaps).
## 3. Storage
### 3.1 `commit`
```ts
/**
* Persist `batches` in order, each as one ordinary atomic commit with its own seq, checked against the state the
* earlier ones left, in as few round trips as the backend can. Stops at the first batch Durable's own checks refuse:
* the batches before it are durable, it and the ones after are not. Calls `receive(index, seq)` once per durable batch,
* in order, synchronously, after all of them are durable and before the next commit starts. Resolves with the number of
* durable batches and the refusal, if any. Rejects for an unknown outcome (connection lost, statement or commit
* failed); `receive` calls made before the rejection still stand.
*/
commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
context: Context,
): Promise<{ readonly stored: number; readonly refused?: unknown }>;
```
`receive` is a durability receipt, not publication: the Session publishes later (§5.3). It is Storage's only write
method (it replaced the earlier one-batch `commit(writes, adopt, context)`, and took its name): every backend implements
it natively. Tests and suites that write one batch directly use `commitWrites()` (`@earendil-works/pi-durable/testing`),
which throws the refusal.
Example: `[A, B, C, D]`, C refused. A and B are durable and published; C rejects with the refusal; D rejects with
`SessionFailed`; the Session fails. This is today's outcome (A and B stored, C fails, D never reaches storage).
### 3.2 Backends
A refusal is one of Durable's own checks (`checkGlobalIds`, `checkDocumentActions`, copy resolution), run against the
transaction's state before that batch writes anything. Any other error is an unknown outcome. No savepoints (Durable
Objects' SQL API does not allow them).
- SQLite: one transaction; per batch: read `next_seq`, run the checks, apply, advance; on a refusal stop and commit the
prefix. `prepareDocumentActions` and `candidateNextId` per batch.
- MemoryStorage: per batch `prepareCommit()` (checks and detaches, changing nothing) then apply; stop at the first
refusal (prefix semantics allow it).
- JSONL: check batch by batch against the in-memory state, append the prefix's commit markers in one write; a torn append
recovers a shorter complete prefix, which prefix semantics allow.
- IndexedDB: one `readwrite` transaction per call (metadata read once, written once at the end). Per batch every read
and check runs first (`checkGlobalIds`, `checkDocumentActions`, then `resolveCopies`, which reads each copy's source
against the state the earlier batches left), and only then its writes. A refusal stops issuing writes; the transaction
commits the prefix. An error of IndexedDB itself (a `DOMException`, or a failed request or transaction without an
error, also as a check's cause) aborts the transaction: the call rejects with nothing stored.
- StoreStorage (p3's primary storage): one series of the Chord store's commit queue per call
(`ChordStore.commitSeries`): one backend transaction for all batches, one seq and one outbox commit per batch, the
store's metadata (`last_seq`, `next_id`) written once per call. A batch's `DurableCommit.check(step)` (ids,
documents, copies through the step's reader, so it sees the earlier batches; the same reads a single commit makes)
runs before its writes; a throw refuses it and the series commits the prefix, unless one of the check's statements
failed (the store watches the check's executor), which rolls the series back and rejects, as does a throw of the
writes; a failed `COMMIT` is an unknown outcome that stops the store. Receipts run after `COMMIT`, before the next
step of the queue; the Session then publishes each commit in its own host batch (`publishing`). Memory per call is
the call's staged commits and one receipt entry per batch; the catalog is copied only if a batch changes a table
record.
A Storage may also implement `publishing?(publish: () => void, context: Context): void`: the Session calls it for each
part's publication and the Storage runs `publish` inside its own scope. StoreStorage implements it with its attached
host's `batch.within(...)`, so Chord sources over Durable's views stay marked in one batch per commit.
### 3.3 Wrappers
- `CachedStorage` applies write-through at publication (§5.3), not at the receipt. Its read rule becomes per key: a read
records `version(key)` at its start and keeps its result only if no commit published since wrote that key. Versions
exist only for keys with a read in flight (registered at read start, removed at read end), so the map is bounded by
reads in flight.
- `GuardedStorage.commit` guards the call but does not fail the Session itself on rejection; the writer does
(§3.4), after publishing the received parts.
### 3.4 Outcomes per logical commit
The writer keeps a ledger per call: which parts were received.
- Received: published (§5.3), then resolved through the fences.
- `commit` resolved with a refusal at part k: parts before k as received; part k rejects with the refusal; parts
after k and every queued job: `SessionFailed`; the Session fails after the received parts are published.
- `commit` rejected: received parts are published first, then reject with the error (adopted-then-rejected, as
today); unreceived parts reject with the error; the Session fails.
- Adoption or an internal listener of part i throws: part i as today (the Session fails after its storage call);
parts after i are not published and reject with `SessionFailed`.
## 4. The line
### 4.1 Mechanics
- The line runs one job at a time; callbacks never interleave with each other's synchronous code.
- A job with writes, after its callback and settlement succeed, takes the held changes it will store (`takeHeld`) and
joins the pending batch (logical admission). Its outcome is parked (§5.1).
- The writer runs off the line: it sends the pending batch with `commit` when no call is in flight and the previous
call's parts are published. A job waiting on a gate never blocks the writer.
- Backpressure: at most `MAX_OUTSTANDING` (initially 256) admitted jobs whose outcome is not delivered, writing or not;
beyond it the line waits.
- Unpublished write set: per key, the newest unpublished writer. Registered at logical admission (immediate commits) or
at callback end (deferred commits, which stay in it until their publication); removed at publication only if the key
still maps to that writer.
- Touch: every read or write a job makes checks its key. On a hit the job awaits the gate of the newest writer of that
key, then proceeds. A gate opens when that commit is published and its storage call settled (holds released).
### 4.2 Keys
| Key | Read by | Written by |
|---|---|---|
| `task:<id>` | `tx.task`, committed-task reads, every task record a scheduler region reads (decided task, wait members and their `on`, cycle walks) | task create or write |
| `owned:<node>` | every node a walk over live children expands (`#hasOwnedLive`, `#ownedLive`, `#walkDown`, `#waitCycle`, `ownedTasks`, `#idle` in a line job) | task creation and a task becoming terminal: every node from the task's parent up to and including the nearest live task or the root conversation (the nodes `#relist` lists or unlists), chain loaded at admission if needed; conversation creation under a task: that task's chain |
| `roots` | global idle in a line job (`#idle(undefined)`) | every task creation whose chain reaches an ownerless conversation (directly or through terminal tasks and conversations), and every task becoming terminal there (conservative: whether the root is listed can change under pending commits) |
| `intent` | every cancellation check (`#belowCancelled`, `#cancellingOwner`, including the empty-intent fast path, `#inScope`, `#nearestLiveAbove`) | abort marks (set, upgraded, cleared), a non-completed held or terminal outcome, restart marks |
| `conv:<id>` | `tx.conversation`, `#requireConversation`, edge loads | conversation creation |
| `entry:<id>` | `tx.entry(id)`, `storage.entry(conv, id)` | entry append |
| `entry-append:<conv>` | `latestEntryId`, `latestHeadMarker`, entry scans, context bounds | entry append (blind) |
| `sub:<id>` | `tx.submission`, settlement/placement resolution | submission create or write |
| `req:<conv>:<requestId>` | `submissionByRequest` | submission creation with a request ID |
| `doc:<address>` | `tx.doc`, `retireDoc`, cold loads, `conversationDocumentOnLine`, fork copies of the source | document create/change/retire/copy at the address; held changes the commit takes |
| `docscope:<scope>` | `scanDocuments`, task-document index reads | document creation/retirement in the scope (blind) |
| `*` | scans that cannot be keyed (`scanTasks`, `scanSubmissions` by status, `scanConversations`); `unloadDocuments`, close, version reload | conflicts with every unpublished write |
Blind writes conflict with reads of the key, not with each other (appends with distinct minted IDs, creations in one
scope). Created records' keys are ordinary keys: a minted ID can leak through a caller variable before publication, and a
read of it waits.
### 4.3 Liveness-only reads (not keyed)
A stale read that only delays work needs no key: a reservation pass not seeing a task a pending commit made runnable;
a waiter's check whether its dependencies are still live; the projection of immutable ownership fields (`owner`,
`conversationId`, `background`) during chain loading (not the rest of `#loadChain`, which installs settled nodes and
null edges after the check); a reservation pass skipping a task with an invocation without reading its record.
Cancellation decisions and terminalization are never liveness-only.
### 4.4 Line reads outside `tx`
`readOnLine(job)` becomes `readOnLine((storage) => job)`: the job gets a gated Storage view whose reads record keys and
wait on gates. All current `readOnLine` call sites (harness.ts, scheduler.ts runtime reads, submissions.ts, task-graph.ts,
view.ts, `captureContextBounds`) use it; the Session's line helpers (`latestEntryId`, `latestHeadMarker`,
`conversationDocumentOnLine`, cold loads) are gated too. Off-line reads keep using the ungated Storage.
## 5. Outcomes and publication
### 5.1 Outcome fences
Every job's outcome (commits with and without writes, `readOnLine` jobs, deferred commits, callback errors) is
delivered in line order. A Session-fatal failure makes every later outcome `SessionFailed`; a nonfatal rejection (a
callback's own error, cancellation, validation, `ReadOnlySession`) affects only its job. Fences hold outcomes, not the
line. Exemption: a scheduler-internal job without writes whose outcome goes only to scheduler code and whose touch set
is clean (no hit) delivers at once; it is equivalent to having run before every unpublished commit. A job with writes
(a reservation, a step that faults, a reconcile pass) is never exempt: its outcome waits for its own publication, so no
invocation starts on a reservation that is not durable.
### 5.2 Deferred commits
A deferred commit's callback runs and settles like any job; its memory adoption and publication are fenced like an
immediate commit's publication: they happen when every earlier unpublished commit is published. Its keys stay in the
unpublished write set until then. Holds are taken at logical admission, so a commit never stores a hold adopted after
it in line order.
### 5.3 Publishing a call's parts
After `commit` settles (or the received prefix is known, §3.4): publish part 0 (cache write-through, conversation
tails, adoption, Session listeners, host dispatch, waits), then a macrotask hop (`setImmediate` where available, else a
`MessageChannel` message), then part 1, and so on. The hop lets public wait continuations (`Submission.wait()`,
`waitForTask()`, `commit()`) of part i start before part i+1 is published. Tests use the public APIs.
### 5.4 Close, unload, holds, eviction
- Close: seal admission; run close listeners; signal every invocation (scheduler); let admitted callbacks finish,
gates included; drain pending and in-flight calls and their publication; join invocations and run `writeAtClose()`
through the writer; store remaining holds through the writer; end observers; close Storage.
- `unloadDocuments()`, version reloads (also nested inside a callback), hold timer stores: `*` barriers through the
writer; holds captured after the barrier.
- Documents with unpublished changes stay pinned.
## 6. Scheduler
Scheduler state changes at publication, as today. Inside a callback, decisions read it synchronously. Each decision is
restructured as a pure plan:
1. A decision region runs with a synchronous "recording" flag set; helpers record keys (§4.2) only while it is set
(the same helpers run at publication inside `#observe`, unrecorded).
2. The region reads, records, and computes a plan or an error, staging nothing and mutating no scheduler state.
3. If a recorded key has an unpublished writer: discard the plan (or error), await the gate, reload scopes
(`#loadScopes`), rebuild the overlay (`overlayOf`, `#links`), and run the region again. A waiting job holds the line,
so unpublished writes only shrink; re-runs converge.
4. After a clean check: apply the plan: stage writes, mutate scheduler sets, create or end invocations, run `migrate`,
take registry snapshots, report. None of these runs inside a region, so none repeats.
Sites to restructure (hardest first): `#reserve` (runnable iteration, abandoned marks, cut-off faults, `#resolve` with
`migrate` and reports, invocation creation); `#step`/`#decide` (registry refresh, `task_outdated` reports,
cancellation, `#end`); `#finalize` (rounds, overlay mutation, awaited `#settleOutcome` between rounds: one region per
round); `#validateWait`/`#waitCycle` (async member loads, cycle errors gated like plans); `abort`; `#commitState`
terminal (chain loads per overlay task, walk); `#reconcile` (intent walks, marks; it captures and clears work lists
before its commit: move the take inside, after the check). `abortConversation` and cascades scan with `*`.
Sweep: `#sweep` and `#scheduleSweep` do not drop nodes loaded by a job whose commit is unpublished; chain loads record
the nodes they install for their job, released at that commit's publication or failure.
Later, the ownership index can move into the Session behind `tx` reads with the same keys.
## 7. Bounds
- Outstanding jobs: at most `MAX_OUTSTANDING`, a count, not bytes: pending writes are bounded by what one commit can
hold (tool output is bounded by output limits), times the count.
- Unpublished write set: one entry per key per newest writer, removed at publication.
- Cache versions: only keys with a read in flight.
- Fences, gates, ledgers: settled at publication or Session failure; nothing outlives close.
- Existing: record cache (conversations 4096, submissions 256; tasks: every live one, the ended ones a waiting or just-resumed task names, plus 256 ended), conversation tails (4096 IDs, 256 markers),
task-document index (live tasks and their documents), kept context ranges (busy or within retention; an idle range
past its expiry is dropped when an entry would extend it; expiry fixed at idle start, the timer uses the current
setting: document and test), document cache (retention).
## 8. Tests
- `commit` per backend (conformance for every backend): prefix stored, refusal index, seq per batch, copies
against earlier batches' state, JSONL torn append; IndexedDB: a copy refused mid-call keeps the prefix, an IndexedDB
error in a check or a write rejects with nothing stored; StoreStorage: one backend transaction per call; Chord's
`commitSeries`: refusal, a failed check statement and a throwing write roll back, a throwing receipt stops the store.
- Gating per key class, with deterministic interleavings: records; ancestor chain membership (live P → terminal B →
terminal A → empty C, creation in C vs P terminal); global idle with empty roots; pending abort mark vs child step
(empty-intent fast path); concurrent opposite waits; leaked entry ID read by `tx.entry`; request IDs; absent
addresses; held documents (hold created after a reader prepared); `*` barriers; `readOnLine` read after an
un-awaited commit.
- Outcome fences: no-write, read, deferred, callback errors in line order; nonfatal rejections do not poison later jobs;
fatal failure poisons later outcomes.
- Deferred publication behind a pending immediate commit, and when that commit fails.
- Failures per part: refusal, rejection after receipts (received parts published, then rejected), adoption failure.
- Publication order through public waits: A's `Submission.wait()`/`waitForTask()` continuation sees B unpublished.
- Close during in-flight calls with an invocation waiting on its signal; unload, version reload, hold timer during
flight; backpressure on all outstanding jobs.
- Scheduler retries: `migrate`, registry snapshots, reports and cleanup run once across re-runs; sweep vs chain loads.
- Randomized interleavings: random commits over overlapping keys, random latency and refusals; storage equals the stored
commits replayed in seq order, no callback observed unpublished state (instrumented oracle), outcomes match storage.
- Prerequisites: tool crash at both sides of the intent commit (sibling flushing presentation first); kept range
installed after a publication between return and install; live retention change; cache eviction and close; tails
eviction, fork markers.
- Bounds: outstanding, write sets, versions, fences return to zero when idle.
- Existing suites: durable (both retention modes), workers.
## 9. Benchmark
Conc probe at 0/1/4/20 ms with 1/8/32/64/128 conversations, one tool, three parallel tools, streaming; dpetrou's repro
as is and with warm documents. Report throughput, p50/p95 run latency, batch sizes, gates per run, fence delay.

300
packages/durable/docs/spec.md
View file

@ -65,11 +65,15 @@ The core rule is:
Required invariants:
1. One Session commit is atomic across all record and document writes.
2. A stored commit is published once Storage made it durable, when Storage
adopts it (section 4), also when `Storage.commit()` then rejects; a commit
that fails before it is durable publishes nothing. A deferred commit
(section 4.1) is published to watches and the conversation view when memory
adopts it, and to `observeCommits()` once it is stored.
2. A stored commit is published once Storage made it durable, after the
storage call that stored it settled, in line order (section 4), also when
that call then rejects; a commit that fails before it is durable publishes
nothing. Once the Session has failed, nothing more is published: a commit
whose call settles after another error failed the Session, or whose turn to
be published comes after that, is durable but unpublished, and reopening
recovers it (section 10.1). A deferred commit (section 4.1) is published to watches and the
conversation view when memory adopts it, once the commits before it are
published, and to `observeCommits()` once it is stored.
3. All visible progress is durable, except what a deferred commit holds: what
watches see can be ahead of Storage by up to that commit's window
(`withinMs`), and a crash loses it. A commit is deferred only when its caller
@ -81,15 +85,16 @@ Required invariants:
6. Document drafts are fully revoked when their transaction callback settles:
the Session synchronously prepares or aborts every open change at that point.
Values assigned into a draft are copied by value and must be strict JSON.
7. The mutation line remains held through storage settlement and committed-state
adoption. Commit/close observers run synchronously only to capture immutable
state; document-state and watch user callbacks run later, as microtasks the
line never waits for, and never inside Storage's synchronous `adopt`. With a
storage that works after `adopt` (JSONL), they may run while the line is
still held by the commit that published.
7. The mutation line runs one commit callback at a time; their storage
overlaps (group commit, section 4). A callback never observes what an
unpublished commit wrote: at its first read or write of such a key it waits
until that commit is stored and published. Commits are published in line
order, and every outcome is delivered in line order. Commit/close observers
run synchronously only to capture immutable state; document-state and watch
user callbacks run later, as microtasks the line never waits for.
8. A storage error is fatal to the open Session (section 10.1), which must be
reopened. A commit it fails publishes nothing, unless Storage had already
made it durable and adopted it. Preparation and checkpoint failures occur before
made it durable (a durability receipt). Preparation and checkpoint failures occur before
storage admission and roll back normally.
## 2. Core records
@ -1607,24 +1612,25 @@ callback fails with no pending acquisition
callback succeeds with no pending acquisition
prepare every open change -> immutable next revision + self-contained Chord Op[]
Session evaluates each required/ordinary document write exactly once
Storage.commit persists the atomic batch while the Session line remains held
commit deferred and holdable (section 4.1)
adopt every prepared change by pointer swap; queue its ops per document, unstored
watches and view mounts capture it synchronously; observeCommits() does not
release the line; invoke listeners later
otherwise, right before Storage admission
join the writer's queue unstored; release the line
once the commits before it are published: adopt every prepared change by pointer swap;
queue its ops per document; watches and view mounts capture it synchronously;
observeCommits() does not; invoke listeners later
otherwise, at logical admission (section 4)
take the held ops of every document the commit acquired and put them in front of its own
storage makes it durable and calls adopt(seq), inside Storage.commit() (section 4)
adopt every prepared change by pointer swap; record its telemetry events
join the writer's pending batch; release the line
the writer's storage call makes it durable (a receipt) and settles
after the commits before it: apply it to the read caches, adopt every prepared change by
pointer swap; record its telemetry events
queue one microtask that calls observeCommits() observers with what was stored
the Session's own observers, including watches and view mounts (section 9.3), capture it
synchronously; the waits they resolve and the aborts they queue run after that microtask
storage resolves
fail the Session if an own observer threw (the commit stands); release the line
storage rejects after adopt(seq)
the commit stays adopted and published; the storage error fails the Session (section 10.1),
fail the Session if an own observer threw (the commit stands, the rest of the call is not published)
the call rejects after it stored the commit
the commit is published; the storage error rejects it and fails the Session (section 10.1),
and an own observer's throw is reported
storage fails before adopt(seq)
the commit is not stored (refused, rejected, or after a failure)
abort every prepared change, fail the Session (section 10.1), and publish nothing
```
@ -1834,65 +1840,92 @@ granule, a tenth of the retention up to one second. Task documents also retire
with their task (section 5.5). `createSession(storage, { documentRetentionMs, setTimer })`
sets them for a plain Session; the Harness reads its settings at each use.
A Session commit callback may be asynchronous. It owns the Session mutation
line through callback execution, preparation, storage settlement, committed
baseline adoption, and publication. Adoption runs inside
`Storage.commit()`: the Session passes an `adopt(seq)` callback, which the
storage calls synchronously once the batch is durable and before its next
commit (section 10), so memory always shows a complete prefix of the log, plus
what deferred commits hold (section 4.1). External model, process, tool,
network, and human effects run outside the line.
A Session commit callback may be asynchronous. The Session mutation line runs
one job at a time: a commit's callback and its settlement, or a read on the
line. A commit with writes then joins the pending batch (logical admission),
and the line goes on with the next job. Storage overlaps the line (group
commit): a writer off the line sends every commit admitted while no storage
call is in flight as one `Storage.commit()` call (section 10), each with
its own `seq`, and publishes them one by one, in line order. Memory
always shows a complete prefix of the log, plus what deferred commits hold
(section 4.1). External model, process, tool, network, and human effects run
outside the line.
The commit is published when it is adopted, so every view changes in the same
step as memory. Inside `adopt`, after the trackers, come the commit's telemetry
No job observes what an unpublished commit wrote. Every read or write of a job
touches a key: a record (task, conversation, entry, submission, a
conversation-scoped request ID), a document address or incarnation, a scope's
documents, a conversation's newest entries, or, in the scheduler, the live work
below an ownership node, the ownerless roots, and the cancellation intent set;
a scan that cannot be keyed touches every key. While an admitted commit that
writes the key is unpublished, the job waits until it is stored and published,
then goes on; a job waiting holds the line, so the unpublished writes only
shrink. Code before a job's first such touch runs early: a callback can run
while an earlier commit is being stored, and cancellation is checked when the
job starts. The scheduler decides in regions that record the keys they read and
run again after such a wait (section 5.4). A commit's outcome is delivered once
it is published, and every job's outcome (commits with and without writes,
reads, deferred commits, callback errors) in line order; a scheduler commit
that writes nothing and waited at no gate delivers to the scheduler at once. A failure that fails
the Session makes every later outcome `SessionFailed`; a callback's own error,
a cancellation, or a refused read-only commit affects only its job. At most 256
admitted jobs wait for their outcome; beyond that the line waits. The bound is a
count, not bytes: what waits is bounded by what one commit can hold (tool output
by its output limits), times the count.
A commit is published after the storage call that stored it settled, so every
view changes in the same step as memory. A Storage with `publishing()` runs each
commit's publication inside a scope of its own: the store backend runs it in its
attached host's batch, so Chord sources over Durable's views are marked in one
batch per commit. Its publication applies it to the Session's read caches and
adopts it in memory; then come the commit's telemetry
events (`pi.durable.entry`, `pi.durable.submission`), then the Session's own
observers, synchronously: document watches, conversation view mounts, the task
graph, the scheduler, and submission waits, with the observers adapters attach
(`observeDoc()`, `observeConversation()`, `observeTaskGraph()`, section 9.1). They
advance watches and resolve waits, but run no task code: the abort signal of a task the commit newly marks
follows in a microtask. Host `observeCommits()` observers run one microtask
after `adopt`, outside the storage's synchronous step, before every wait the
commit resolves and every abort it queues: those continuations are queued by
the own observers, after the dispatch. There is no order between host listeners
and watch listeners, nor `onWake`: they are independent observers of the
commit, and a watch delivery already queued for an earlier frame may deliver
this commit first. The host listeners run before the Session sees
`Storage.commit()` resolve, while the line is still held. A storage that works
after `adopt` before it resolves, as JSONL does when it reclaims sidecars, runs
what the commit wakes during that work too; its next commit still waits.
after the publication, before every wait the commit resolves and every abort it
queues: those continuations are queued by the own observers, after the
dispatch. There is no order between host listeners and watch listeners, nor
`onWake`: they are independent observers of the commit, and a watch delivery
already queued for an earlier frame may deliver this commit first. Two commits
of one storage call are published a macrotask apart, so the host listeners of
the first, and the continuations of the waits it resolves (`commit()`,
`Submission.wait()`, `waitForTask()`), run before the second is published.
```text
adopt(41): trackers, telemetry events, own observers (views advance, waits resolve)
commit([.., 41, 42]) settles
publish 41: caches, trackers, telemetry events, own observers (views advance, waits resolve)
microtask: observeCommits() observers
microtasks after it: wait continuations, abort listeners
microtasks after it: wait continuations, session.commit() resolves, abort listeners
microtasks, in no order relative to the above: watch listeners, onWake
Storage.commit() resolves; session.commit() resolves
macrotask: publish 42
```
`adopt` never throws. A throwing telemetry adapter is reported (`host_error`,
Publication never throws. A throwing telemetry adapter is reported (`host_error`,
callback `telemetry`; a plain Session drops it, as it drops every report) and
the commit resolves; a host listener's throw is reported (section 10.1), and an
adapter's observer that throws is also detached and ended (section 9.1). The
first throw of an own observer is kept, the other observers and the host
listeners still get the commit, and it fails the Session once
`Storage.commit()` returned; the commit stands and resolves. A throw while
memory adopts the batch is different: nothing is published, the Session fails
once `Storage.commit()` returned, and the commit rejects with that error. A
commit that Storage rejects after its adoption is durable: it stays adopted and
published, waits it resolved stay resolved, the commit rejects with the storage
error, and the Session fails with it; an own observer's throw in that commit is
then reported as a `listener_error`.
listeners still get the commit, and it fails the Session after the host
listeners got it; the commit stands and resolves, and the commits after it in
the same call are not published (section 10.1). A throw while memory adopts
the commit is different: nothing more of it is published, the Session fails,
and the commit rejects with that error. A commit Storage stored whose call then
rejects is durable: it is published, waits it resolved stay resolved, the
commit rejects with the storage error, and the Session fails with it; an own
observer's throw in that commit is then reported as a `listener_error`.
`observeCommits(observer)` observes stored commits in one synchronous
`observer.advance(publication, context)` call per observer, on the line, each with exactly what that storage
transaction stored (section 4.1). It attaches synchronously and returns a `CommitObservation` with an idempotent
`detach()`; unlike the other `observeXXX()` it has no `value`: commits are events, not a state, and the Session does
not know Storage's newest `seq` before its own first commit. Adoption
not know Storage's newest `seq` before its own first commit. A publication
queues the one dispatch of a commit only when an observer is attached; the
dispatch calls the observers attached when it runs. So an observer attached
between `adopt` and the dispatch gets the commit only when some observer was
attached at `adopt`; one detached in between does not get it; one attached after
the dispatch ran misses it. `observer.end(end)` comes once with a `WatchEnd`: `ended` when the Session has closed,
between the publication and the dispatch gets the commit only when some
observer was attached at the publication; one detached in between does not get
it; one attached after the dispatch ran misses it. `observer.end(end)` comes once with a `WatchEnd`: `ended` when the Session has closed,
after its last publication, or `failed` with the error that failed it; never after `detach()`. An observer whose
`advance` throws, or returns a promise that rejects, is reported (section 10.1), detached, and ended
`listener_error`; the others still get the commit. `subscribeClose()` observes close synchronously when it
@ -1916,9 +1949,15 @@ await session.commit(async tx => {
```
Mutation admission occurs on the Session line before a commit callback starts. Closing seals mutation admission and task
reservation. Already-admitted commits settle before storage closes. Once a
commit is admitted, caller cancellation does not interrupt storage settlement or
undo the commit. Cancelling a close wait does not reopen admission.
reservation, runs the close listeners, and signals every task invocation; the
callbacks already admitted finish, waits at their touches included, every
pending and in-flight storage call settles and is published, the writes at
close and the held changes go through the same writer, and then storage closes.
Once a commit is admitted, caller cancellation does not interrupt storage
settlement or undo the commit. Cancelling a close wait does not reopen admission.
`unloadDocuments()`, a reload for another definition version, and the store at a
held document's deadline wait until every admitted commit is published, then
store what is held at that point through the writer.
Table rules:
@ -1991,9 +2030,12 @@ What is held is stored by the first of:
another definition version, which reloads the document from Storage;
retirement is case 1.
Held operations leave their queue only at the last step before
`Storage.commit()`, after every fallible step: a callback, settlement, or
checkpoint failure of a normal commit keeps them. A store fails like any
Held operations leave their queue only at the commit's logical admission
(section 4), after every fallible step: a callback, settlement, or checkpoint
failure of a normal commit keeps them, and a commit never stores a hold adopted
after it in line order. A held commit is adopted in memory and published when
the commits before it are published, and what it writes stays unpublished until
then: a commit that reads the document waits for it. A store fails like any
storage write: it fails the Session (section 10.1). Past Chord's
`MAX_PENDING_OPS` (10,000) held operations, or when a held commit's
`checkpointWhen` selected a base, the store writes a base of the current value.
@ -2002,7 +2044,7 @@ A failed Session stores nothing; after a reopen nothing is held.
Publication: the Session's own observers see each held commit once, when memory
adopts it, and of a normal commit that stores held changes only its own
operations. `observeCommits()` sees every storage transaction with exactly
what it stored, published when Storage adopts it (section 4): nothing for a
what it stored, published once stored (section 4): nothing for a
held commit, and the stored operations, held first, for a commit or a deadline
that stores them. A document the commit retires is published as its retirement
only: the held operations and the commit's own change stored before the
@ -2385,7 +2427,16 @@ or its `orphaned` settlement when its definition is unavailable.
### 5.4 Scheduler, abort, and ownership
The scheduler serially reserves eligible tasks, then runs handlers off the
Session line. One in-memory `TaskInvocation` contains mode, abort controller,
Session line. Its decisions stay serialized on the line while their storage
overlaps (section 4): each reads the scheduler's memory, which changes as
commits are published, in a region that records what it reads (a task's
record, the live work below an ownership node, the ownerless roots, the
cancellation intent set) and stages nothing; when a commit that writes one of
those is unpublished, the region waits for it and runs again, so `migrate`,
registry snapshots, reports, and cleanup run once, after a decision stands.
Every commit writes the memberships its task changes alter: a task created or
ended changes the live work of every node from its parent up to the nearest
live task or the root conversation, and the chain is loaded at its admission. One in-memory `TaskInvocation` contains mode, abort controller,
and completion promise.
Abort protocol:
@ -4638,9 +4689,10 @@ each commit writes at most one window, and the throttle bounds the rate.
Generation creates `tools` with one `pending` slot per call in the commit that
appends the tool-calling answer; a call its request did not offer is flagged
`unoffered`. A tool task sets its slot `running` in its intent commit,
publishes throttled output and details into it with deferred commits (section
4.1), and in its terminal commit sets
`unoffered`. A tool task sets its slot `running` (and a nested slot's final
arguments) with a deferred commit (section 4.1) right before its intent commit,
publishes throttled output and details into it with deferred commits, and in its
terminal commit sets
`done` and `entry` and removes `output`, `droppedBytes`, `droppedLines`,
`details`, and `diagnostics`, which the result entry now carries (section 7.3).
The generation's `tools` phase removes `tools`. Slot updates apply only while a slot with the task's
@ -4823,7 +4875,9 @@ The run's inputs live in `pi.live.run`, not in the task input.
`prepare`. `request` converts a leftover partial (below) before it resolves
the model.
- `request` first converts a committed partial left in `pi.live` by an
interrupted attempt into an aborted `pi.assistant` entry. When the host died
interrupted attempt into an aborted `pi.assistant` entry, and sets
`pi.live.generation` to the attempt with a deferred commit (section 4.1), which
the conversion makes immediate. When the host died
while this request was in flight (the record's `restarts`, section 5.4), the
attempt failed: that commit also either sleeps at `retry` until
`now + retryDelayMs(policy, attempt)`, with `pi.live.generation.retry` saying
@ -4993,8 +5047,10 @@ Below, `{ entryId }` is a model-issued call's result.
phase's agent `tools` (section 7.3), validates
the arguments with pi-ai
`validateToolArguments()`, runs the `beforeTool` chain, and validates again
(section 7.3). One commit then records intent: it moves to `execute` with the
final arguments and the tool's replay policy and sets the slot `running`. The
(section 7.3). A deferred commit sets the slot `running`; one commit then
records intent: it moves to `execute` with the final arguments and the tool's
replay policy. After a crash the slot may still show `pending` while the task's
checkpoint is `execute`; a safe rerun sets it `running` again. The
same handler executes the tool, runs the `afterTool` chain, and commits the
result; it builds `api.env` with `HarnessOptions.env` right before executing
(section 2.2). A tool that does not resolve, invalid arguments, or a block
@ -5422,20 +5478,20 @@ interface Harness {
Attaching works like acquiring a watch, atomically on the Session line: the
observation's `value` includes every commit before it, and `advance` gets every
later change that a watch would get, unfolded, as memory adopts it: for a
stored commit inside Storage's `adopt` (section 4), for a deferred one when
stored commit at its publication (section 4), for a deferred one when
memory adopts it, before it is stored (section 4.1). A commit gives a view or
the graph at most one call, and a document observer one call per change at its
address: a commit that retires the document and creates it again gives two, the
retirement and then the new incarnation's `[["r", value]]`. The task graph also
advances without a commit, when a registry change or a failed migration changes
a task's `blocked` reason (section 9.4): that call runs on the Session line,
outside any `adopt`, with a context of its own. A document observer follows its
outside any commit's publication, with a context of its own. A document observer follows its
address as a document watch does, and `observeDoc()` gives `undefined` for an
absent one. `end` comes once: when the Session closes (`failed` with the error
"session closed": an adapter following the Session, such as Chord's `self.follow`,
fails and follows the next Session; watches end `ended`) or fails (`failed` with
the error), or, for a task document, right after the retirement change of the
commit that makes its task terminal, inside that commit's `adopt` (`ended`).
commit that makes its task terminal, at that commit's publication (`ended`).
`detach()` is idempotent, and nothing follows it, not even the rest of the
commit underway; a detached observer is referenced by nothing of the Session.
`context` governs only the attachment: cancelling it while attaching rejects
@ -5617,7 +5673,7 @@ The Harness keeps at most one mount per conversation. The first `watch()` or
observer builds its documents on the Session line, and each attachment reads
its window there with bounded scans; the mount is dropped when its last
observer detaches. A mount derives each next revision from the Session's
commits as memory adopts them, for a stored commit inside Storage's `adopt`
commits as memory adopts them, for a stored commit at its publication
(section 4) and for a held one when memory adopts it (section 4.1), like
`watchDoc()`: it shows committed state, including deferred commits before they
are stored (section 4.1). One publication that touches the conversation yields
@ -5787,11 +5843,11 @@ The Harness keeps at most one graph mount. The first `watchTaskGraph()` builds i
live tasks and the conversations they own, one owner scan per live task, and
holds the line while it does; the mount is dropped when its last observer
detaches. It advances from the Session's stored commits as memory adopts them,
inside Storage's `adopt` (section 4), like the conversation view: a commit that
at their publication (section 4), like the conversation view: a commit that
touches a conversation view and the graph changes both in that one step. A
`blocked` reason that changes without a commit (a registry change, a failed
migration) is derived again in a job on the Session line and published there,
outside any `adopt` and any commit, with the background context. One
outside any commit and its publication, with the background context. One
publication that changes a node yields one Chord batch
that sets or deletes `tasks[id]`, or sets a node's `conversations`. A
publication that changes no node creates no revision. Task and conversation
@ -5913,12 +5969,27 @@ type StorageWrite =
*/
interface Storage {
/**
* Persist the batch atomically. Once it is durable, call `adopt(seq)` exactly once, synchronously and before this
* storage's next commit starts, then resolve with `seq`; a commit that fails before the batch is durable never calls
* it. `adopt` adopts and publishes the batch: it runs the Session's own observers (section 4), which call no Storage
* method, and it never throws.
* Persist `batches` in order, each as one ordinary atomic commit with its own seq, checked against the state the
* earlier ones left, in as few round trips as the backend can. Every batch has at least one write: a commit without
* writes never reaches Storage, so the Session never sends an empty batch. Stops at the first batch Durable's own
* checks refuse (global IDs, document actions, copy resolution): the batches before it are durable, it and the ones
* after are not. Calls `receive(index, seq)` once per durable batch, in order, synchronously, after all of them are
* durable and before the next commit starts, and resolves with the number of durable batches and the refusal, if
* any. Rejects for an unknown outcome (connection lost, statement or commit failed); `receive` calls made before the
* rejection still stand. A receipt is not publication: the Session publishes later; it calls no Storage method and
* never throws.
*/
commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq>;
commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
context: Context,
): Promise<{ readonly stored: number; readonly refused?: unknown }>;
/**
* Optional. Run `publish`, the Session's publication of one commit (cache write-through, adoption, observers),
* inside this storage's own scope, such as a store host's batch. Call it exactly once, now or soon after; the
* Session publishes the next commit only once it ran.
*/
publishing?(publish: () => void, context: Context): void;
/**
* Allocate from the one global numeric namespace, above every ID minted since this storage was opened and every ID
* stored, also across reopen: scans order by ID as creation order, and the Harness relies on it. The generic brand
@ -5949,6 +6020,19 @@ interface Storage {
}
```
Every backend implements `commit()` natively. It checks each batch (global
IDs, document actions, copy resolution) before that batch writes anything, so
a refusal leaves the earlier batches stored and nothing of its own; no
savepoints are needed, which Durable Objects' SQL does not allow. SQLite runs
one transaction per call. IndexedDB runs one `readwrite` transaction per call:
on a refusal it stops issuing writes and the transaction commits the prefix;
an error of IndexedDB itself aborts it, and the call rejects with nothing
stored. The store backend stores a call as one series of the store's commit
queue (`ChordStore.commitSeries`): one backend transaction, one seq and one
outbox commit per batch, the store's metadata written once. JSONL appends every batch's sidecar records and then
all their markers in one write (a torn append recovers a shorter complete
prefix), and memory applies them one by one.
Every error a method throws is final for the Session (section 10.1), so a
backend retries its own transient failures. A read throws `StorageRequestError`
for an invalid request, with no durable effect; from `commit()` it is fatal like any
@ -6060,12 +6144,35 @@ exempt: once a batch is admitted, whether it committed is unknown.
A commit callback that catches a failed read cannot commit afterwards: the Session
checks its health after the callback. Adopting a committed batch in memory that
fails also fails the Session, since memory would be behind storage; so does a
`Storage.commit()` that resolves without adopting, or rejects after it adopted.
A batch Storage adopted is durable, so it is not rolled back: it was published
at adoption (section 4), `observeCommits()` observers get it and the waits it
settles resolve; then the commit rejects with the storage error and the Session
fails as above, with the storage error. An own observer that threw on that
commit is reported as a `listener_error`.
storage call that resolves without a receipt for a batch it reports stored.
Each commit of one storage call (group commit, section 4) gets its own outcome,
in line order:
- A commit Storage stored (a receipt) is published, then resolves.
- A call that stops at a refusal of commit `k`: the commits before `k` are
published and resolve; `k` rejects with the refusal; the commits after `k`,
and every commit admitted later, reject with `SessionFailed`, unpublished;
the Session fails with the refusal once the commits before `k` are
published.
- A call that rejects: the commits it stored are durable, so they are not
rolled back: they are published, `observeCommits()` observers get them and
the waits they settle resolve, and then they reject with the storage error,
as do the commits it did not store; the Session fails with that error. An own
observer that threw on such a commit is reported as a `listener_error`.
- A commit whose adoption or own observer throws: it rejects with the adoption
error, or stands and resolves for an observer's throw, and the Session fails;
the commits after it in the call are durable but not published, and reject
with `SessionFailed`. Reopening recovers them.
- A call that settles after another error already failed the Session, or a
commit of a call whose publication comes after another error failed it: what
it stored is durable but not published, and rejects with `SessionFailed`
whose cause is the first error, whatever the call returned. Reopening
recovers it; an `observeCommits()` consumer, such as a replica, misses it
until then.
A job whose outcome comes after the one a failure is gets `SessionFailed`, even
when its callback ran before the failure; a job before it keeps its own outcome.
**The Session's own bookkeeping fails it too.** A throw in a commit the
scheduler makes for itself (reconciliation, reservation, a fault or terminal
@ -6075,9 +6182,9 @@ storage failure, a host callback such as a `RegistryReader` whose `snapshot()`
throws, or a bug, and running the commit again fixes none of them. A throw in a commit
listener of the Session's own components (scheduler, submissions, views, task
graph, document observers) fails it as well, since their memory would fall
behind storage; the commit itself stands. Those listeners run inside Storage's
`adopt` (section 4), so the Session fails once `Storage.commit()` returned,
after the other observers and the `observeCommits()` observers got the commit.
behind storage; the commit itself stands. Those listeners run at the commit's
publication (section 4), so the Session fails after the other observers and the
`observeCommits()` observers got the commit.
**User code fails only its unit.** A task phase that throws ends that task
`faulted`; a throwing `migrate` blocks it as `migration_failed`; a tool that
@ -6144,7 +6251,8 @@ after a committed base or retirement.
### 11.2 SQLite
One SQL transaction is one Session commit. SQLite stores:
One SQL transaction is one storage call: the batches of a `commit()` call,
each checked before it writes. SQLite stores:
- conversation, entry, task, and submission records;
- document records;
@ -6182,7 +6290,9 @@ Publication protocol:
2. Append one complete main marker listing those records.
3. Publish in memory only after the marker write succeeds.
Every commit uses this protocol; there is no standalone-sidecar fast path.
Every commit uses this protocol; there is no standalone-sidecar fast path. A
`commit()` call appends every batch's sidecar records first, then all its
markers in one write of `main.jsonl`; reads wait while it appends.
JSONL creation accepts an `fsync` option that defaults to `false`. Without
`fsync`, it guarantees ordinary process-crash consistency, not survival of
power, host, kernel, or filesystem failure. With `fsync: true`, the backend

94
packages/durable/src/harness/context.ts
View file

@ -17,27 +17,31 @@ export type ContextBounds = {
};
/**
* Capture the bounds of the current context, or of the context cut off at the visible entry `at`, with two O(1)
* reads. Run this on the Session line; entries at or below the tail are immutable, so `deriveContext()` can then scan
* them off the line.
* Capture the bounds of the current context, or of the context cut off at the visible entry `at`. The newest entry
* and head marker are known from the Session's commits, so the current bounds usually cost no Storage read. Run this
* on the Session line; entries at or below the tail are immutable, so `deriveContext()` can then scan them off the
* line.
*/
export async function captureContextBounds(
session: SessionImpl,
storage: Storage,
conversationId: ConversationId,
context: Context,
at?: EntryId,
/** Set when `at` is known to be visible, as an entry of a range of the conversation is. */
visible = false,
): Promise<ContextBounds | undefined> {
let tail: EntryId | undefined;
if (at === undefined) {
tail = (await storage.scanEntries({ conversationId }, 1, undefined, context)).items[0]?.id;
tail = await session.latestEntryId(conversationId, context);
if (tail === undefined) return undefined;
} else {
if ((await storage.entry(conversationId, at, context)) === undefined) {
if (!visible && (await storage.entry(conversationId, at, context)) === undefined) {
throw new Error(`Entry ${at} is not visible from conversation ${conversationId}`);
}
tail = at;
}
return { head: await storage.findLatestHeadMarker(conversationId, tail, context), tail };
return { head: await session.latestHeadMarker(conversationId, tail, context), tail };
}
/**
@ -60,6 +64,11 @@ export type ContextRange = {
readonly open: readonly Message[];
/** The baseline's system message; see `findBaseline()`. */
readonly baseline: Baseline;
/**
* Whether the range reached the conversation's newest entry when it was read, so the entries of later commits extend
* it (`advanceRange()`); a range cut off at an earlier entry is not.
*/
readonly current: boolean;
};
/** A baseline's system message, `null` when the range cannot have one, or `undefined` while a later entry may be one. */
@ -68,7 +77,9 @@ type Baseline = Message | null | undefined;
/**
* `readContext()` that reuses `previous`, an earlier range of the same conversation: with the same head marker, only
* entries after its tail are scanned. Returns the view and the range to pass to the next read. A range outlives the
* read, so its entries are frozen like `MemoryStorage` records, and the returned view has its own arrays.
* read, so its entries are frozen like `MemoryStorage` records, and the returned view has its own arrays. A caller that
* keeps the range for `advanceRange()` checks `current` again, synchronously where it installs it: a commit can be
* published between this function's return and the caller's continuation.
*/
export async function readContextFrom(
session: SessionImpl,
@ -78,23 +89,33 @@ export async function readContextFrom(
at: EntryId | undefined,
previous: ContextRange | undefined,
): Promise<{ readonly view: ContextView; readonly range: ContextRange | undefined }> {
const bounds = await session.readOnLine(() => captureContextBounds(storage, conversationId, context, at));
const visible = at !== undefined && previous !== undefined && rangeEntry(previous, at) !== undefined;
const bounds = await session.readOnLine((line) =>
captureContextBounds(session, line, conversationId, context, at, visible),
);
if (bounds === undefined) return { view: emptyView(), range: undefined };
freezeJson(bounds.head);
const current = at === undefined;
let range: ContextRange;
if (previous === undefined || previous.bounds.head?.id !== bounds.head?.id) {
range = deriveRange(bounds, freezeJson(await scanRange(storage, conversationId, rangeQuery(bounds), context)));
const entries = freezeJson(await scanRange(storage, conversationId, rangeQuery(bounds), context));
range = deriveRange(bounds, entries, current);
} else if (bounds.tail === previous.bounds.tail) {
range = previous;
} else if (bounds.tail < previous.bounds.tail) {
range = deriveRange(
bounds,
previous.entries.filter((entry) => entry.id <= bounds.tail),
false,
);
} else {
const minEntryId = idFromNumber<EntryId>(previous.bounds.tail + 1);
const added = await scanRange(storage, conversationId, { minEntryId, maxEntryId: bounds.tail }, context);
range = extendRange(previous, bounds, freezeJson(added));
range = extendRange(previous, bounds, freezeJson(added), current);
}
// Entries committed while the scan ran off the line are not in the range: it is current only if none was.
if (range.current && session.knownLatestEntryId(conversationId) !== range.bounds.tail) {
range = { ...range, current: false };
}
const { view } = range;
return {
@ -116,7 +137,7 @@ export async function readContext(
context: Context,
at?: EntryId,
): Promise<ContextView> {
const bounds = await session.readOnLine(() => captureContextBounds(storage, conversationId, context, at));
const bounds = await session.readOnLine((line) => captureContextBounds(session, line, conversationId, context, at));
return deriveContext(storage, conversationId, bounds, context);
}
@ -133,15 +154,49 @@ export async function deriveContext(
context: Context,
): Promise<ContextView> {
if (bounds === undefined) return emptyView();
return deriveRange(bounds, await scanRange(storage, conversationId, rangeQuery(bounds), context)).view;
return deriveRange(bounds, await scanRange(storage, conversationId, rangeQuery(bounds), context), false).view;
}
function emptyView(): ContextView {
return { head: undefined, entries: [], contributions: [], messages: [] };
}
/** Entry `id` of `range`, by binary search: a range holds its entries in ID order, every one visible from its conversation. */
export function rangeEntry(range: ContextRange, id: EntryId): EntryRecord | undefined {
const entries = range.entries;
let low = 0;
let high = entries.length - 1;
while (low <= high) {
const middle = (low + high) >>> 1;
const entry = entries[middle]!;
if (entry.id === id) return entry;
if (entry.id < id) low = middle + 1;
else high = middle - 1;
}
return undefined;
}
/**
* `range` with the entries one commit appended to its conversation, in ID order, without reading Storage: the
* conversation's own new entries are its newest, and a new head marker's head never lies before the range's start.
* A range that is not `current` stays as it is; the next read extends it from Storage. `undefined` drops the range.
*/
export function advanceRange(range: ContextRange, appended: readonly EntryRecord[]): ContextRange | undefined {
if (!range.current || appended.length === 0) return range;
const added = freezeJson(appended);
let head = range.bounds.head;
for (const entry of added) if (entry.head !== undefined) head = entry as typeof head;
const bounds = { head, tail: added.at(-1)!.id };
if (head?.id === range.bounds.head?.id) return extendRange(range, bounds, added, true);
const start = head!.head;
// Heads only move forward: compaction keeps a later entry, a reset starts at itself, an older head is stale.
if (range.bounds.head !== undefined && start < range.bounds.head.head) return undefined;
const entries = [...range.entries, ...added].filter((entry) => entry.id >= start);
return deriveRange(bounds, entries, true);
}
/** Derive the context view from the entries scanned within `bounds`. */
function deriveRange(bounds: ContextBounds, entries: readonly EntryRecord[]): ContextRange {
function deriveRange(bounds: ContextBounds, entries: readonly EntryRecord[], current: boolean): ContextRange {
const edits = new Map<EntryId, ContextEdit>();
// Edits of every entry in the range count, including older head markers that `selectActive()` drops.
for (const entry of entries) for (const edit of entry.edits ?? []) edits.set(edit.target, edit);
@ -151,17 +206,22 @@ function deriveRange(bounds: ContextBounds, entries: readonly EntryRecord[]): Co
const baseline = findBaseline(bounds.head, active);
const messages = leadWithSystem([...settled, ...orderToolResults(open)], baseline);
const view = { head: bounds.head, entries: active, contributions, messages };
return { bounds, entries, view, edited: new Set(edits.keys()), settled, open, baseline };
return { bounds, entries, view, edited: new Set(edits.keys()), settled, open, baseline, current };
}
/**
* `previous` extended by the entries `added` after its tail under the same head marker: only their contributions and
* the open messages are derived. An added edit can change an earlier entry, so it derives the whole range again.
*/
function extendRange(previous: ContextRange, bounds: ContextBounds, added: readonly EntryRecord[]): ContextRange {
function extendRange(
previous: ContextRange,
bounds: ContextBounds,
added: readonly EntryRecord[],
current: boolean,
): ContextRange {
const entries = [...previous.entries, ...added];
if (added.some((entry) => entry.edits !== undefined || entry.head !== undefined || previous.edited.has(entry.id))) {
return deriveRange(bounds, entries);
return deriveRange(bounds, entries, current);
}
const contributions = added.map((entry) => contribute(entry, undefined));
const { settled, open } = settle(previous.settled, [...previous.open, ...contributions.flat()]);
@ -172,7 +232,7 @@ function extendRange(previous: ContextRange, bounds: ContextBounds, added: reado
contributions: [...previous.view.contributions, ...contributions],
messages: leadWithSystem([...settled, ...orderToolResults(open)], baseline),
};
return { bounds, entries, view, edited: previous.edited, settled, open, baseline };
return { bounds, entries, view, edited: previous.edited, settled, open, baseline, current };
}
/**

20
packages/durable/src/harness/generation.ts
View file

@ -242,7 +242,7 @@ export const GenerationTask = defineTask<GenerationInput, GenerationCheckpoint,
}
await runtime.commit(async (tx) => {
// The context ends in a message the model answers, so the conversation has an entry.
let cutoff = (await tx.scanEntries({ conversationId }, 1)).items[0]!.id;
let cutoff = (await tx.latestEntryId(conversationId))!;
for (const entry of entries) cutoff = (await tx.appendEntry(SystemEntry, conversationId, entry)).id;
// Checked in this commit, so a compaction admitted during preparation counts.
if (threshold === "background" && (await tx.doc(LiveDoc, conversationId)).compactions === undefined) {
@ -266,12 +266,18 @@ export const GenerationTask = defineTask<GenerationInput, GenerationCheckpoint,
if (task.restarts !== undefined) return interrupted(runtime, task.state.checkpoint, context);
const { attempt, compacted, model: ref, thinkingLevel, streamOptions, cutoff } = task.state.checkpoint;
const conversationId = runtime.conversationId;
await runtime.commit(async (tx) => {
const live = await tx.doc(LiveDoc, conversationId);
await convertPartial(tx, live, conversationId);
live.generation = { attempt };
return undefined;
}, context);
// The attempt shown is presentation, stored like progress (`PROGRESS_COMMIT`); converting a partial left by an
// interrupted attempt appends an entry, which makes the commit immediate.
await runtime.commit(
async (tx) => {
const live = await tx.doc(LiveDoc, conversationId);
await convertPartial(tx, live, conversationId);
live.generation = { attempt };
return undefined;
},
context,
PROGRESS_COMMIT,
);
const model = runtime.models.getModel(ref.provider, ref.modelId);
if (model === undefined) return failNoModel(runtime, ref, context);
const view = await runtime.context(conversationId, context, { at: cutoff });

27
packages/durable/src/harness/harness.ts
View file

@ -161,7 +161,7 @@ class ConversationImpl<Tool extends ToolRegistration> implements Conversation {
context: Context,
): Promise<Page<EntryRecord, Cursor>> {
const scan = conversationEntries(this.id, query);
return this.#host.harness.readOnLine(() => this.#host.storage.scanEntries(scan, limit, cursor, context));
return this.#host.harness.readOnLine((storage) => storage.scanEntries(scan, limit, cursor, context));
}
fork(at: EntryId, options: ConversationCreateOptions, context: Context): Promise<Conversation> {
@ -249,14 +249,14 @@ class HarnessImpl<Tool extends ToolRegistration> extends SessionImpl implements
settleOutcome: settleSchedulerOutcome,
withdrawInputs: withdrawQueuedInputs,
conversation: async (id, binding, callContext) => {
const record = await this.readOnLine(() => storage.conversation(id, callContext));
const record = await this.readOnLine((line) => line.conversation(id, callContext));
return record === undefined ? undefined : boundConversation(id, binding, this.#submissions, this.#tasks);
},
context: withoutAbortSignal(context),
readOnly: options.readOnly === true,
});
this.#submissions = new Submissions(this, storage, now, settings, () => this.#tasks.resume(), options.telemetry);
this.#taskGraph = new TaskGraphView(this, storage, {
this.#submissions = new Submissions(this, now, settings, () => this.#tasks.resume(), options.telemetry);
this.#taskGraph = new TaskGraphView(this, {
blocked: (record) => this.#tasks.blockedReason(record),
subscribeBlocked: (listener) => this.#tasks.subscribeBlocked(listener),
});
@ -265,7 +265,7 @@ class HarnessImpl<Tool extends ToolRegistration> extends SessionImpl implements
storage,
tasks: this.#tasks,
submissions: this.#submissions,
views: new ConversationViews(this, storage),
views: new ConversationViews(this),
now,
create: (target, createOptions, context) => this.#create(target, createOptions, context),
};
@ -308,7 +308,7 @@ class HarnessImpl<Tool extends ToolRegistration> extends SessionImpl implements
}
getTask<R>(id: TaskId<R>, context: Context): Promise<TaskRecord<JsonValue, JsonValue, R> | undefined> {
return this.readOnLine(() => this.storage.task(id, context)) as Promise<
return this.readOnLine((storage) => storage.task(id, context)) as Promise<
TaskRecord<JsonValue, JsonValue, R> | undefined
>;
}
@ -319,10 +319,10 @@ class HarnessImpl<Tool extends ToolRegistration> extends SessionImpl implements
}
inspect(context: Context): Promise<HarnessInspection> {
return this.readOnLine(async () => {
return this.readOnLine(async (storage) => {
const { scheduling, tasks } = await this.#tasks.inspect(this.#options.registry.snapshot());
const scan = (status: "queued" | "placed") =>
scanAll((cursor) => this.storage.scanSubmissions({ status }, SCAN_PAGE_SIZE, cursor, context));
scanAll((cursor) => storage.scanSubmissions({ status }, SCAN_PAGE_SIZE, cursor, context));
const submissions = [...(await scan("queued")), ...(await scan("placed"))].sort((a, b) => a.id - b.id);
return { scheduling, tasks, submissions };
});
@ -356,8 +356,8 @@ class HarnessImpl<Tool extends ToolRegistration> extends SessionImpl implements
/** Sum every conversation's committed `pi.usage`. Each document is read at its own point; totals only grow. */
async usage(context: Context): Promise<UsageState> {
const conversations = await this.readOnLine(() =>
scanAll((cursor) => this.storage.scanConversations({}, SCAN_PAGE_SIZE, cursor, context)),
const conversations = await this.readOnLine((storage) =>
scanAll((cursor) => storage.scanConversations({}, SCAN_PAGE_SIZE, cursor, context)),
);
const total = UsageDoc.definition.initial();
for (const { id } of conversations) {
@ -395,7 +395,7 @@ class HarnessImpl<Tool extends ToolRegistration> extends SessionImpl implements
async conversation(id: ConversationId, context: Context): Promise<Conversation | undefined> {
this.assertUsable();
const record = await this.readOnLine(() => this.storage.conversation(id, context));
const record = await this.readOnLine((storage) => storage.conversation(id, context));
return record === undefined ? undefined : new ConversationImpl(record.id, this.#host);
}
@ -407,6 +407,11 @@ class HarnessImpl<Tool extends ToolRegistration> extends SessionImpl implements
this.#report(error, info, context);
}
/** The ownership memberships a commit's task changes write, from the scheduler's index. */
protected override membershipKeys(tx: Transaction): readonly string[] | Promise<readonly string[]> {
return this.#tasks.membershipKeys(tx);
}
/** Signal and join task invocations after the close listeners and before Storage closes; writes no task outcome. */
protected override beforeClose(): Promise<void> {
return this.#tasks.join();

912
packages/durable/src/harness/scheduler.ts
View file

File diff suppressed because it is too large Load diff

13
packages/durable/src/harness/submissions.ts
View file

@ -8,7 +8,7 @@ import {
} from "@earendil-works/pi-telemetry";
import { ConversationBusy, ConversationIdle, ReadAfterWrite } from "../errors.ts";
import type { ObservedCommit, SessionImpl } from "../session/session.ts";
import type { ConversationId, JsonObject, Storage, SubmissionId, SubmissionRecord, Tx } from "../types.ts";
import type { ConversationId, JsonObject, SubmissionId, SubmissionRecord, Tx } from "../types.ts";
import { startRun } from "./generation.ts";
import {
appendInput,
@ -34,7 +34,6 @@ type Admission = { readonly id: SubmissionId; readonly status: "queued" | "place
export class Submissions {
readonly #telemetry: TelemetryContext;
readonly #session: SessionImpl;
readonly #storage: Storage;
readonly #now: () => number;
/** Read at each admission, on the Session line. */
readonly #queueModes: () => QueueModes;
@ -45,7 +44,6 @@ export class Submissions {
constructor(
session: SessionImpl,
storage: Storage,
now: () => number,
queueModes: () => QueueModes,
resume: () => void,
@ -53,7 +51,6 @@ export class Submissions {
) {
this.#telemetry = telemetry;
this.#session = session;
this.#storage = storage;
this.#now = now;
this.#queueModes = queueModes;
this.#resume = resume;
@ -123,12 +120,12 @@ export class Submissions {
/** Handle for an existing submission, or `undefined`. */
async get(id: SubmissionId, context: Context): Promise<Submission | undefined> {
const record = await this.#session.readOnLine(() => this.#storage.submission(id, context));
const record = await this.#session.readOnLine((storage) => storage.submission(id, context));
return record === undefined ? undefined : new SubmissionHandle(record.id, this);
}
async status(id: SubmissionId, context: Context): Promise<SubmissionRecord> {
const record = await this.#session.readOnLine(() => this.#storage.submission(id, context));
const record = await this.#session.readOnLine((storage) => storage.submission(id, context));
if (record === undefined) throw new Error(`Submission ${id} does not exist`);
return record;
}
@ -136,8 +133,8 @@ export class Submissions {
async wait(id: SubmissionId, context: Context): Promise<SettledSubmissionRecord> {
this.#resume();
// Check and register on the line so no settling publication falls between them.
const found = await this.#session.readOnLine(async () => {
const record = await this.#storage.submission(id, context);
const found = await this.#session.readOnLine(async (storage) => {
const record = await storage.submission(id, context);
if (record === undefined) throw new Error(`Submission ${id} does not exist`);
if (isSettled(record)) return { promise: Promise.resolve(record) };
// Close rejects registered waiters synchronously and may begin during the read.

20
packages/durable/src/harness/task-graph.ts
View file

@ -87,14 +87,12 @@ const SCAN_PAGE_SIZE = 256;
*/
export class TaskGraphView {
readonly #session: SessionImpl;
readonly #storage: Storage;
readonly #blocking: TaskGraphBlocking;
#mount: Mount | undefined;
#closed = false;
constructor(session: SessionImpl, storage: Storage, blocking: TaskGraphBlocking) {
constructor(session: SessionImpl, blocking: TaskGraphBlocking) {
this.#session = session;
this.#storage = storage;
this.#blocking = blocking;
session.followCommits((publication, context) => {
// A commit held in memory changes documents only, which the graph does not show.
@ -153,8 +151,11 @@ export class TaskGraphView {
create: (value: TaskGraph, release: () => void) => O,
context: Context,
): Promise<{ observer: O; detach: () => void }> {
return this.#session.readOnLine(async () => {
const mount = this.#mount ?? { ...(await this.#build(context)), observers: new Set<Observer<TaskGraph>>() };
return this.#session.readOnLine(async (storage) => {
const mount = this.#mount ?? {
...(await this.#build(storage, context)),
observers: new Set<Observer<TaskGraph>>(),
};
const detach = (): void => {
mount.observers.delete(observer);
if (mount.observers.size === 0 && this.#mount === mount) this.#mount = undefined;
@ -190,19 +191,18 @@ export class TaskGraphView {
.catch(() => {});
}
async #build(context: Context): Promise<Omit<Mount, "observers">> {
/** Read the graph on the line, through the line's `storage`. */
async #build(storage: Storage, context: Context): Promise<Omit<Mount, "observers">> {
const tasks: Record<string, TaskGraphNode> = {};
const live = new Map<string, BlockedSubject>();
const records: AnyTaskRecord[] = [];
for (const status of LIVE_STATUSES) {
records.push(
...(await scanAll((cursor) => this.#storage.scanTasks({ status }, SCAN_PAGE_SIZE, cursor, context))),
);
records.push(...(await scanAll((cursor) => storage.scanTasks({ status }, SCAN_PAGE_SIZE, cursor, context))));
}
records.sort((a, b) => a.id - b.id);
for (const record of records) {
const owned = await scanAll((cursor) =>
this.#storage.scanConversations({ ownerTaskId: record.id }, SCAN_PAGE_SIZE, cursor, context),
storage.scanConversations({ ownerTaskId: record.id }, SCAN_PAGE_SIZE, cursor, context),
);
const conversations = owned.map((conversation) => conversation.id).sort((a, b) => a - b);
const subject = blockedSubject(record);

31
packages/durable/src/harness/tool.ts
View file

@ -177,13 +177,21 @@ export const ToolTask = defineTask<ToolTaskInput, ToolTaskCheckpoint, ToolTaskRe
if ("error" in validated) return reject(runtime, task.input, call, tool, invalid(validated.error), context);
// Strict JSON: repair may set optional properties to undefined, which the slot and checkpoint cannot hold.
const final = copyJson(validated.args, { omitUndefinedProperties: true }) as JsonObject;
await runtime.commit(async (tx) => {
const slot = toolSlot(await tx.doc(LiveDoc, runtime.conversationId), runtime.taskId);
if (slot !== undefined) slot.status = "running";
// A nested slot shows the arguments the call runs with, which repair, hooks, and coercion may have changed.
// Compared with the input, which the slot got at admission: reading the draft would track every leaf.
const changed = task.input.kind === "nested" && !jsonEqual(task.input.call.arguments as JsonValue, final);
if (changed && slot !== undefined && "parentTaskId" in slot) slot.arguments = final;
// The slot's status and arguments are presentation, stored like progress (`PROGRESS_COMMIT`); the intent is
// durable before the tool runs. Parallel calls of one round then share no immediate write to `pi.live`.
await runtime.commit(
async (tx) => {
const slot = toolSlot(await tx.doc(LiveDoc, runtime.conversationId), runtime.taskId);
if (slot !== undefined) slot.status = "running";
// A nested slot shows the arguments the call runs with, which repair, hooks, and coercion may have changed.
// Assigned whole: an equal value records no change, and a rerun corrects what an earlier attempt showed.
if (slot !== undefined && "parentTaskId" in slot) slot.arguments = final;
return undefined;
},
context,
PROGRESS_COMMIT,
);
await runtime.commit(() => {
const intent = { phase: "execute", arguments: final, replay: tool.replay ?? "unsafe" } as const;
return { status: "running", checkpoint: intent };
}, context);
@ -197,10 +205,15 @@ export const ToolTask = defineTask<ToolTaskInput, ToolTaskCheckpoint, ToolTaskRe
if (cutOffTooOften(task, runtime)) return endCutOff(runtime, task, call, context);
const tool = await resolveTool(runtime, task.input, call.name, context);
if (replay === "safe" && tool?.replay === "safe") {
// The rerun reports from scratch; clear what the interrupted attempt published.
// The rerun reports from scratch: clear what the interrupted attempt published, and show the slot running with
// the arguments it runs with again, as the deferred commit that showed them may have been lost with the crash.
await runtime.commit(async (tx) => {
const slot = toolSlot(await tx.doc(LiveDoc, runtime.conversationId), runtime.taskId);
if (slot !== undefined) clearProgress(slot);
if (slot !== undefined) {
clearProgress(slot);
slot.status = "running";
if ("parentTaskId" in slot) slot.arguments = args;
}
return undefined;
}, context);
return run(runtime, task.input, call, tool, args, context);

15
packages/durable/src/harness/view.ts
View file

@ -145,13 +145,11 @@ function resolveWindow(window: ConversationWindow | undefined): WindowBounds {
*/
export class ConversationViews {
readonly #session: SessionImpl;
readonly #storage: Storage;
readonly #mounts = new Map<ConversationId, Mount>();
#closed = false;
constructor(session: SessionImpl, storage: Storage) {
constructor(session: SessionImpl) {
this.#session = session;
this.#storage = storage;
VIEWS.set(session, this);
session.followCommits((publication, context) => {
for (const [id, mount] of this.#mounts)
@ -236,9 +234,9 @@ export class ConversationViews {
context: Context,
): Promise<{ observer: O; detach: () => void }> {
const bounds = resolveWindow(window);
return this.#session.readOnLine(async () => {
const mount = this.#mounts.get(id) ?? (await this.#build(id, context));
const { entries, before } = await loadWindow(this.#storage, id, bounds, context);
return this.#session.readOnLine(async (storage) => {
const mount = this.#mounts.get(id) ?? (await this.#build(storage, id, context));
const { entries, before } = await loadWindow(storage, id, bounds, context);
const value = view(mount.conversation, entries, before, mount.docs);
let windowed: Windowed | undefined;
const detach = (): void => {
@ -256,8 +254,9 @@ export class ConversationViews {
});
}
async #build(id: ConversationId, context: Context): Promise<Mount> {
const conversation = await this.#storage.conversation(id, context);
/** Hydrate a mount on the line, through the line's `storage`. */
async #build(storage: Storage, id: ConversationId, context: Context): Promise<Mount> {
const conversation = await storage.conversation(id, context);
if (conversation === undefined) throw new Error(`Conversation ${id} does not exist`);
const docs: Record<string, JsonObject> = {};
const incarnations = new Map<string, { id: DocumentId; version: number }>();

1
packages/durable/src/index.ts
View file

@ -147,6 +147,7 @@ export type {
CommitOptions,
CommitPersistence,
CommitPublication,
CommitResult,
CommonDocDefinition,
ContextEdit,
ConversationDocFamilyToken,

378
packages/durable/src/session/cached-storage.ts Normal file
View file

@ -0,0 +1,378 @@
// Write-through cache of hot Storage record reads for the Session that owns the Storage.
import type { Context, JsonValue } from "@earendil-works/chord";
import type {
CommitResult,
ConversationId,
ConversationQuery,
ConversationRecord,
Cursor,
DocumentAddress,
DocumentId,
DocumentPoint,
DocumentQuery,
DocumentRecord,
EntryId,
EntryQuery,
EntryRecord,
Id,
Page,
Seq,
Storage,
StorageWrite,
StoredDocument,
SubmissionId,
SubmissionQuery,
SubmissionRecord,
TaskId,
TaskQuery,
TaskRecord,
} from "../types.ts";
type AnyTaskRecord = TaskRecord<JsonValue, JsonValue, JsonValue>;
type StoredEntry = { readonly entry: EntryRecord; readonly commitSeq: Seq };
/**
* Storage that serves hot record reads from memory: conversations, tasks, and written submissions, each table bounded
* by a count, the least recently used record leaving first, except live tasks, which are kept until they end, and the
* ended tasks a task waits on, kept until it moved on from its wait: a task's record is read again at each of its state
* changes, a waiter reads the outcomes of what it waited on once it resumes (`runtime.outcomes()`), and both are
* bounded by live work, which the scheduler keeps in memory too. Without that, a burst of more tasks than the count
* would make each of those reads a Storage read on the line. Every commit goes through it, and one process owns a
* Storage, so it applies each batch as it is published and never goes stale: `commit()`'s batches when the Session publishes
* them (`published()`), not at their durability receipts. A read
* result is kept only when no commit published since the read started wrote its record, as the read may not observe
* that commit; versions are kept only for records with a read in flight. Cached values are frozen and shared. Entries,
* scans, and documents pass through: the Session keeps documents and the newest entry per conversation itself, and
* context ranges hold the entries a run reads.
*
* Not for a read-only Session: another process writes its Storage.
*/
export class CachedStorage implements Storage {
readonly #storage: Storage;
readonly #conversations = new Lru<ConversationId, ConversationRecord>(4096);
/** Live (non-terminal) task records, until their task ends. */
readonly #liveTasks = new Map<TaskId, AnyTaskRecord>();
/**
* Tasks some task waits on, or waited on until its wait ended and it has not moved on since: how many such waits name
* each, and its ended record once it ended.
*/
readonly #awaited = new Map<TaskId, { count: number; record?: AnyTaskRecord }>();
/** Per task that waits, or left its wait and has not changed since: what it waited on, and whether it left. */
readonly #waits = new Map<TaskId, { readonly on: readonly TaskId[]; left: boolean }>();
// Ended task records carry inputs and outcomes, such as a tool call's arguments, so fewer are kept.
readonly #tasks = new Lru<TaskId, AnyTaskRecord>(256);
readonly #submissions = new Lru<SubmissionId, SubmissionRecord>(256);
/** Per record with a read in flight: the reads, and how many published commits wrote it since the first began. */
readonly #reads = new Map<Id<string>, { count: number; version: number }>();
constructor(storage: Storage) {
this.#storage = storage;
}
/** The backend's own scope for a publication (`Storage.publishing`), when it has one. */
get publishing(): Storage["publishing"] {
return scopeOf(this.#storage);
}
/** Pass the batches through; the Session applies each one with `published()` when it publishes it. */
commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
context: Context,
): Promise<CommitResult> {
return this.#storage.commit(batches, receive, context);
}
/**
* Apply a batch `commit()` stored, when the Session publishes it. Its records are frozen in place and kept as
* they are, no copies: the Session's components that follow the publication, such as the scheduler's live tasks,
* hold the same objects, so a live task's record is in memory once.
*/
published(writes: readonly StorageWrite[]): void {
for (const write of writes) {
switch (write.type) {
case "conversation":
this.#wrote(write.value.id);
this.#conversations.set(write.value.id, freeze(write.value));
break;
case "task":
this.#wrote(write.value.id);
this.#followWait(write.value);
this.#keepTask(freeze(write.value));
break;
case "submission":
this.#wrote(write.value.id);
this.#submissions.set(write.value.id, freeze(write.value));
break;
}
}
}
mintId<I extends Id<string>>(): Promise<I> {
return this.#storage.mintId<I>();
}
conversation(id: ConversationId, context: Context): Promise<ConversationRecord | undefined> {
const cached = this.#conversations.get(id);
if (cached !== undefined) return Promise.resolve(cached);
return this.#keep(
id,
() => this.#storage.conversation(id, context),
(record) => this.#conversations.set(id, record),
);
}
scanConversations(
query: ConversationQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<ConversationRecord, Cursor>> {
return this.#storage.scanConversations(query, limit, cursor, context);
}
entry(id: EntryId, context: Context): Promise<StoredEntry | undefined>;
entry(conversationId: ConversationId, id: EntryId, context: Context): Promise<StoredEntry | undefined>;
entry(
first: EntryId | ConversationId,
second: EntryId | Context,
third?: Context,
): Promise<StoredEntry | undefined> {
if (third === undefined) return this.#storage.entry(first as EntryId, second as Context);
return this.#storage.entry(first as ConversationId, second as EntryId, third);
}
findLatestHeadMarker(
conversationId: ConversationId,
atOrBeforeEntryId: EntryId | undefined,
context: Context,
): Promise<(EntryRecord & { readonly head: EntryId }) | undefined> {
return this.#storage.findLatestHeadMarker(conversationId, atOrBeforeEntryId, context);
}
scanEntries(
query: EntryQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<EntryRecord, Cursor>> {
return this.#storage.scanEntries(query, limit, cursor, context);
}
task(id: TaskId, context: Context): Promise<AnyTaskRecord | undefined> {
const cached = this.#liveTasks.get(id) ?? this.#awaited.get(id)?.record ?? this.#tasks.get(id);
if (cached !== undefined) return Promise.resolve(cached);
return this.#keep(
id,
() => this.#storage.task(id, context),
(record) => this.#keepTask(record),
);
}
scanTasks(
query: TaskQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<AnyTaskRecord, Cursor>> {
return this.#storage.scanTasks(query, limit, cursor, context);
}
/** Cached as written only: this process writes every submission it settles, and a read one is usually settled. */
submission(id: SubmissionId, context: Context): Promise<SubmissionRecord | undefined> {
const cached = this.#submissions.get(id);
return cached === undefined ? this.#storage.submission(id, context) : Promise.resolve(cached);
}
scanSubmissions(
query: SubmissionQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<SubmissionRecord, Cursor>> {
return this.#storage.scanSubmissions(query, limit, cursor, context);
}
submissionByRequest(
conversationId: ConversationId,
requestId: string,
context: Context,
): Promise<SubmissionRecord | undefined> {
return this.#storage.submissionByRequest(conversationId, requestId, context);
}
findDocument(address: DocumentAddress, at: DocumentPoint, context: Context): Promise<DocumentRecord | undefined> {
return this.#storage.findDocument(address, at, context);
}
document(id: DocumentId, at: DocumentPoint, context: Context): Promise<StoredDocument | undefined> {
return this.#storage.document(id, at, context);
}
scanDocuments(
query: DocumentQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<DocumentRecord, Cursor>> {
return this.#storage.scanDocuments(query, limit, cursor, context);
}
close(context: Context): Promise<void> {
return this.#storage.close(context);
}
/** Records kept per table, for tests that bound memory. */
get sizes(): {
readonly conversations: number;
readonly tasks: number;
readonly liveTasks: number;
readonly awaited: number;
readonly waits: number;
readonly submissions: number;
} {
return {
conversations: this.#conversations.size,
tasks: this.#tasks.size,
liveTasks: this.#liveTasks.size,
awaited: this.#awaited.size,
waits: this.#waits.size,
submissions: this.#submissions.size,
};
}
/** Keep a frozen task record: a live one until its task ends, an ended one in the bounded table. */
#keepTask(record: AnyTaskRecord): void {
if (record.state.status === "terminal") {
this.#liveTasks.delete(record.id);
const awaited = this.#awaited.get(record.id);
if (awaited !== undefined) awaited.record = record;
else this.#tasks.set(record.id, record);
return;
}
this.#tasks.delete(record.id);
this.#liveTasks.set(record.id, record);
}
/**
* Follow a task's waits from its written record: a wait holds the ended records of its `on` until the task, after
* leaving the wait, changes again (or ends), as the phase that resumes reads their outcomes before its next commit.
*/
#followWait(record: AnyTaskRecord): void {
const wait = this.#waits.get(record.id);
if (record.state.status === "waiting") {
if (wait !== undefined) this.#release(wait.on);
const on = record.state.on;
this.#waits.set(record.id, { on, left: false });
for (const id of on) {
let awaited = this.#awaited.get(id);
if (awaited === undefined) {
// One that ended already moves over from the bounded table.
const ended = this.#tasks.get(id);
this.#tasks.delete(id);
awaited = ended === undefined ? { count: 0 } : { count: 0, record: ended };
this.#awaited.set(id, awaited);
}
awaited.count++;
}
return;
}
if (wait === undefined) return;
if (!wait.left && record.state.status !== "terminal") {
wait.left = true;
return;
}
this.#waits.delete(record.id);
this.#release(wait.on);
}
/** A wait on `on` is over: what no other wait names goes back to the bounded table. */
#release(on: readonly TaskId[]): void {
for (const id of on) {
const awaited = this.#awaited.get(id)!;
if (--awaited.count > 0) continue;
this.#awaited.delete(id);
if (awaited.record !== undefined) this.#tasks.set(id, awaited.record);
}
}
/** A published commit wrote record `id`: a read of it in flight may have missed that. */
#wrote(id: Id<string>): void {
const reads = this.#reads.get(id);
if (reads !== undefined) reads.version++;
}
/** Resolve with a read's result, frozen; keep a found record unless a commit published since wrote it. */
async #keep<T extends object>(
id: Id<string>,
read: () => Promise<T | undefined>,
keep: (value: T) => void,
): Promise<T | undefined> {
let reads = this.#reads.get(id);
if (reads === undefined) {
reads = { count: 0, version: 0 };
this.#reads.set(id, reads);
}
reads.count++;
const version = reads.version;
let value: T | undefined;
try {
value = await read();
} finally {
if (--reads.count === 0) this.#reads.delete(id);
}
if (value === undefined) return undefined;
freeze(value);
if (reads.version === version) keep(value);
return value;
}
/** Records with a read in flight, for tests that bound memory. */
get readsInFlight(): number {
return this.#reads.size;
}
}
/** `storage`'s publication scope (`Storage.publishing`), for a wrapper to expose as its own; none when it has none. */
export function scopeOf(storage: Storage): Storage["publishing"] {
if (storage.publishing === undefined) return undefined;
return (publish, context) => storage.publishing!(publish, context);
}
/** A Map bounded to `capacity` entries that drops the least recently used one. */
class Lru<K, V> {
readonly #map = new Map<K, V>();
readonly #capacity: number;
constructor(capacity: number) {
this.#capacity = capacity;
}
get(key: K): V | undefined {
const value = this.#map.get(key);
if (value === undefined) return undefined;
this.#map.delete(key);
this.#map.set(key, value);
return value;
}
get size(): number {
return this.#map.size;
}
delete(key: K): void {
this.#map.delete(key);
}
set(key: K, value: V): void {
this.#map.delete(key);
this.#map.set(key, value);
if (this.#map.size > this.#capacity) this.#map.delete(this.#map.keys().next().value!);
}
}
function freeze<T>(value: T): T {
if (value === null || typeof value !== "object" || Object.isFrozen(value)) return value;
for (const child of Object.values(value)) freeze(child);
return Object.freeze(value);
}

115
packages/durable/src/session/conversation-tails.ts Normal file
View file

@ -0,0 +1,115 @@
// The newest entry and newest head marker of each recently used conversation, kept current from the Session's commits.
import type { ConversationId, EntryId, EntryRecord, StorageWrite } from "../types.ts";
/** An entry that carries a `head`: the marker of a context range. */
export type HeadMarker = EntryRecord & { readonly head: EntryId };
/** Conversations whose newest entry ID is kept: IDs only, so many. */
const LATEST_CAPACITY = 4096;
/** Conversations whose newest head marker is kept: whole entries (a compaction summary is one), so fewer. */
const MARKER_CAPACITY = 256;
/**
* The newest entry ID and newest head marker per conversation, for the Session that owns its Storage: every entry is
* written by one of its commits, so `apply` keeps both current, and a value read from Storage stays valid until the
* next entry of that conversation. Read only on the Session line after touching the conversation's appends, so no
* commit that appends to it is unpublished and a value read from Storage can be kept. The newest entry visible from a conversation is its own newest, or for a fork without entries
* of its own the parent's entry at the fork point: IDs grow, so a conversation's own new entry is always the newest.
* Each map is bounded by a count, the least recently used conversation leaving first. Kept markers are frozen: entry
* records are immutable (`Tx.appendEntry()`).
*/
export class ConversationTails {
/** Newest entry ID per conversation, `null` for none. */
readonly #latest = new Map<ConversationId, EntryId | null>();
/** Newest head marker per conversation, `null` for none. */
readonly #markers = new Map<ConversationId, HeadMarker | null>();
/** The newest entry ID visible from `conversationId`, read with `read` when not known. */
async latest(
conversationId: ConversationId,
read: () => Promise<EntryId | undefined>,
): Promise<EntryId | undefined> {
const known = use(this.#latest, conversationId);
if (known !== undefined) return known ?? undefined;
const latest = await read();
put(this.#latest, conversationId, latest ?? null, LATEST_CAPACITY);
return latest;
}
/**
* The newest visible head marker at or before `atOrBefore` (any when `undefined`), read with `read` when not known.
* The newest marker answers every cutoff at or after it, and no marker answers every cutoff.
*/
async marker(
conversationId: ConversationId,
atOrBefore: EntryId | undefined,
read: () => Promise<HeadMarker | undefined>,
): Promise<HeadMarker | undefined> {
const marker = use(this.#markers, conversationId);
if (marker === null) return undefined;
if (marker !== undefined && (atOrBefore === undefined || atOrBefore >= marker.id)) return marker;
const latest = this.#latest.get(conversationId);
const found = await read();
// Only a read that no newer entry can change: no cutoff, or one at or after the newest entry.
if (atOrBefore === undefined || (latest !== undefined && (latest ?? 0) <= atOrBefore)) {
put(this.#markers, conversationId, found === undefined ? null : freeze(found), MARKER_CAPACITY);
}
return found;
}
/** The newest entry ID of `conversationId` if known, without reading: `null` for none, `undefined` when unknown. */
peek(conversationId: ConversationId): EntryId | null | undefined {
return this.#latest.get(conversationId);
}
/** At a commit's publication: apply a durable batch. */
apply(writes: readonly StorageWrite[]): void {
for (const write of writes) {
if (write.type === "conversation") {
// A new root conversation sees no entries; a fork sees its parent's up to the fork point.
if (write.value.parent !== undefined) continue;
put(this.#latest, write.value.id, null, LATEST_CAPACITY);
put(this.#markers, write.value.id, null, MARKER_CAPACITY);
} else if (write.type === "entry") {
const entry = write.value;
put(this.#latest, entry.conversationId, entry.id, LATEST_CAPACITY);
if (entry.head !== undefined) {
put(this.#markers, entry.conversationId, freeze(entry as HeadMarker), MARKER_CAPACITY);
}
}
}
}
/** How many conversations have their newest entry kept, for tests that bound memory. */
get size(): number {
return this.#latest.size;
}
/** How many conversations have their newest head marker kept, for tests that bound memory. */
get markers(): number {
return this.#markers.size;
}
}
/** `map`'s value for `key`, made the most recently used. */
function use<K, V>(map: Map<K, V>, key: K): V | undefined {
const value = map.get(key);
if (value === undefined) return undefined;
map.delete(key);
map.set(key, value);
return value;
}
/** Set `key` as the most recently used, dropping the least recently used beyond `capacity`. */
function put<K, V>(map: Map<K, V>, key: K, value: V, capacity: number): void {
map.delete(key);
map.set(key, value);
if (map.size > capacity) map.delete(map.keys().next().value!);
}
function freeze<T>(value: T): T {
if (value === null || typeof value !== "object" || Object.isFrozen(value)) return value;
for (const child of Object.values(value)) freeze(child);
return Object.freeze(value);
}

274
packages/durable/src/session/keys.ts Normal file
View file

@ -0,0 +1,274 @@
// The keys of the Session line: what a commit writes and a job reads, so a read of an unpublished write waits for it.
import type { Context, JsonValue } from "@earendil-works/chord";
import { addressId } from "../documents.ts";
import type {
CommitResult,
ConversationId,
ConversationQuery,
ConversationRecord,
Cursor,
DocumentAddress,
DocumentId,
DocumentPoint,
DocumentQuery,
DocumentRecord,
EntryId,
EntryQuery,
EntryRecord,
Id,
Page,
Seq,
Storage,
StorageWrite,
StoredDocument,
SubmissionId,
SubmissionQuery,
SubmissionRecord,
TaskId,
TaskQuery,
TaskRecord,
} from "../types.ts";
type StoredEntry = { readonly entry: EntryRecord; readonly commitSeq: Seq };
/** A task record. */
export function taskKey(id: TaskId): string {
return `task:${id}`;
}
/** A conversation record. */
export function conversationKey(id: ConversationId): string {
return `conv:${id}`;
}
/** An entry record. */
export function entryKey(id: EntryId): string {
return `entry:${id}`;
}
/** The newest entries of a conversation: appended blind, read by scans, the newest entry, and head markers. */
export function appendKey(id: ConversationId): string {
return `entry-append:${id}`;
}
/** A submission record. */
export function submissionKey(id: SubmissionId): string {
return `sub:${id}`;
}
/** The submission a conversation's request ID names. */
export function requestKey(conversationId: ConversationId, requestId: string): string {
return `req:${conversationId}:${requestId}`;
}
/** A document address (`addressId()`): its current incarnation, value, and held changes. */
export function documentKey(addressId: string): string {
return `doc:${addressId}`;
}
/** A document incarnation, for reads by its ID. */
export function incarnationKey(id: DocumentId): string {
return `docid:${id}`;
}
/** The incarnations alive in a scope: created and retired blind, read by scans and the task-document index. */
export function scopeKey(scope: DocumentRecord["scope"]): string {
switch (scope.kind) {
case "session":
return "docscope:session";
case "conversation":
return `docscope:c${scope.conversationId}`;
case "task":
return `docscope:t${scope.taskId}`;
}
}
/**
* A node of the scheduler's ownership tree (a task ID, or `c<conversation ID>`): the live work listed below it. Written
* by task creation and a task becoming terminal for every node from the task's parent up to and including the nearest
* live task or the root conversation; read by every walk that expands the node.
*/
export function ownedKey(node: TaskId | string): string {
return `owned:${node}`;
}
/** The ownerless conversations with live work below them, which global idle walks from. */
export const ROOTS_KEY = "roots";
/** The live tasks with cancellation intent: abort marks, and held outcomes other than `completed`. */
export const INTENT_KEY = "intent";
/** Every unpublished write: what a read that cannot be keyed, such as a scan, touches. */
export const ALL_KEYS = "*";
/** The keys of the table records in `writes`; document writes are keyed by the transaction, which knows addresses. */
export function tableKeys(writes: readonly StorageWrite[], keys: string[]): void {
for (const write of writes) {
switch (write.type) {
case "conversation":
keys.push(conversationKey(write.value.id));
break;
case "entry":
keys.push(entryKey(write.value.id), appendKey(write.value.conversationId));
break;
case "task":
keys.push(taskKey(write.value.id));
break;
case "submission":
keys.push(submissionKey(write.value.id));
if (write.value.requestId !== undefined) {
keys.push(requestKey(write.value.conversationId, write.value.requestId));
}
break;
}
}
}
/**
* The Storage a job on the line reads through: every read first touches its key (`touch`), which waits while the key
* has an unpublished writer, so the read sees what the line admitted before it. Scans that cannot be keyed touch
* `ALL_KEYS`. Commits and `close()` pass through; the line never commits through it.
*/
export class GatedStorage implements Storage {
readonly #storage: Storage;
readonly #touch: (key: string) => Promise<void> | undefined;
constructor(storage: Storage, touch: (key: string) => Promise<void> | undefined) {
this.#storage = storage;
this.#touch = touch;
}
commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
context: Context,
): Promise<CommitResult> {
return this.#storage.commit(batches, receive, context);
}
mintId<I extends Id<string>>(): Promise<I> {
return this.#storage.mintId<I>();
}
conversation(id: ConversationId, context: Context): Promise<ConversationRecord | undefined> {
return this.#after(conversationKey(id), () => this.#storage.conversation(id, context));
}
scanConversations(
query: ConversationQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<ConversationRecord, Cursor>> {
return this.#after(ALL_KEYS, () => this.#storage.scanConversations(query, limit, cursor, context));
}
entry(id: EntryId, context: Context): Promise<StoredEntry | undefined>;
entry(conversationId: ConversationId, id: EntryId, context: Context): Promise<StoredEntry | undefined>;
entry(
first: EntryId | ConversationId,
second: EntryId | Context,
third?: Context,
): Promise<StoredEntry | undefined> {
if (third === undefined) {
const id = first as EntryId;
return this.#after(entryKey(id), () => this.#storage.entry(id, second as Context));
}
const conversationId = first as ConversationId;
const id = second as EntryId;
// Visibility follows the conversation's immutable ancestry; the conversation may be unpublished itself.
return this.#after(conversationKey(conversationId), () =>
this.#after(entryKey(id), () => this.#storage.entry(conversationId, id, third)),
);
}
findLatestHeadMarker(
conversationId: ConversationId,
atOrBeforeEntryId: EntryId | undefined,
context: Context,
): Promise<(EntryRecord & { readonly head: EntryId }) | undefined> {
return this.#afterEntries(conversationId, () =>
this.#storage.findLatestHeadMarker(conversationId, atOrBeforeEntryId, context),
);
}
scanEntries(
query: EntryQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<EntryRecord, Cursor>> {
return this.#afterEntries(query.conversationId, () => this.#storage.scanEntries(query, limit, cursor, context));
}
task(id: TaskId, context: Context): Promise<TaskRecord<JsonValue, JsonValue, JsonValue> | undefined> {
return this.#after(taskKey(id), () => this.#storage.task(id, context));
}
scanTasks(
query: TaskQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<TaskRecord<JsonValue, JsonValue, JsonValue>, Cursor>> {
return this.#after(ALL_KEYS, () => this.#storage.scanTasks(query, limit, cursor, context));
}
submission(id: SubmissionId, context: Context): Promise<SubmissionRecord | undefined> {
return this.#after(submissionKey(id), () => this.#storage.submission(id, context));
}
scanSubmissions(
query: SubmissionQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<SubmissionRecord, Cursor>> {
return this.#after(ALL_KEYS, () => this.#storage.scanSubmissions(query, limit, cursor, context));
}
submissionByRequest(
conversationId: ConversationId,
requestId: string,
context: Context,
): Promise<SubmissionRecord | undefined> {
return this.#after(requestKey(conversationId, requestId), () =>
this.#storage.submissionByRequest(conversationId, requestId, context),
);
}
findDocument(address: DocumentAddress, at: DocumentPoint, context: Context): Promise<DocumentRecord | undefined> {
return this.#after(documentKey(addressId(address)), () => this.#storage.findDocument(address, at, context));
}
document(id: DocumentId, at: DocumentPoint, context: Context): Promise<StoredDocument | undefined> {
return this.#after(incarnationKey(id), () => this.#storage.document(id, at, context));
}
scanDocuments(
query: DocumentQuery,
limit: number,
cursor: Cursor | undefined,
context: Context,
): Promise<Page<DocumentRecord, Cursor>> {
return this.#after(scopeKey(query.scope), () => this.#storage.scanDocuments(query, limit, cursor, context));
}
close(context: Context): Promise<void> {
return this.#storage.close(context);
}
/**
* Run a read of a conversation's entries once neither its appends nor its creation has an unpublished writer: Storage
* refuses the read of a conversation it does not have yet.
*/
#afterEntries<T>(conversationId: ConversationId, read: () => Promise<T>): Promise<T> {
return this.#after(conversationKey(conversationId), () => this.#after(appendKey(conversationId), read));
}
/** Run `read` once `key` has no unpublished writer. */
#after<T>(key: string, read: () => Promise<T>): Promise<T> {
const gate = this.#touch(key);
return gate === undefined ? read() : gate.then(read);
}
}

679
packages/durable/src/session/session.ts
View file

File diff suppressed because it is too large Load diff

67
packages/durable/src/session/transaction.ts
View file

@ -57,6 +57,7 @@ import type {
} from "../types.ts";
import { ROOT_CONVERSATION_ID } from "../types.ts";
import { prepareForkDocumentCopies } from "./forks.ts";
import { documentKey, incarnationKey, scopeKey, tableKeys } from "./keys.ts";
type AnyTaskRecord = TaskRecord<JsonValue, JsonValue, JsonValue>;
@ -119,7 +120,10 @@ export type AdoptedDocuments = {
/** Session services used by a transaction while it holds the mutation line. */
export interface TransactionHost {
/** Reads wait while their key has an unpublished writer (`GatedStorage`). */
readonly storage: Storage;
/** Wait while `key` has an unpublished writer; `undefined` when it has none. */
touch(key: string): Promise<void> | undefined;
/** Wall clock for task lifecycle times. */
now(): number;
/** Return the cached current incarnation without loading. */
@ -137,6 +141,17 @@ export interface TransactionHost {
evict(addressId: string, recordId: DocumentId): void;
/** Stage writes that belong to every newly created or forked conversation, in its creating transaction. */
conversationCreated(tx: Transaction, record: ConversationRecord): Promise<void>;
/** The newest entry ID visible from a conversation, known from the Session's commits. */
latestEntryId(conversationId: ConversationId, context: Context): Promise<EntryId | undefined>;
/**
* The current documents of a live task when the Session knows all of them (it created the task), else `undefined`.
*/
taskDocuments(taskId: TaskId): Iterable<DocumentRecord> | undefined;
/** The newest visible head marker of a conversation, known from the Session's commits. */
latestHeadMarker(
conversationId: ConversationId,
context: Context,
): Promise<(EntryRecord & { readonly head: EntryId }) | undefined>;
}
/** Committed and candidate state for one task touched by this transaction. */
@ -304,9 +319,11 @@ export class Transaction implements Tx {
}
latestHeadMarker(conversationId: ConversationId) {
return this.#read("latestHeadMarker", () =>
this.#host.storage.findLatestHeadMarker(conversationId, undefined, this.#context),
);
return this.#read("latestHeadMarker", () => this.#host.latestHeadMarker(conversationId, this.#context));
}
latestEntryId(conversationId: ConversationId): Promise<EntryId | undefined> {
return this.#read("latestEntryId", () => this.#host.latestEntryId(conversationId, this.#context));
}
scanTasks(query: TaskQuery, limit: number, cursor?: Cursor) {
@ -548,6 +565,13 @@ export class Transaction implements Tx {
return records;
}
/** Internal: IDs of the tasks this transaction creates. */
createdTaskIds(): TaskId[] {
const ids: TaskId[] = [];
for (const [id, task] of this.#tasksById) if (task.write?.kind === "create") ids.push(id);
return ids;
}
/** Internal: conversations this transaction created or forked so far. */
stagedConversations(): ConversationRecord[] {
const records: ConversationRecord[] = [];
@ -717,6 +741,8 @@ export class Transaction implements Tx {
}
async #findRetirement(entry: DocumentEntry): Promise<void> {
const gate = this.#host.touch(documentKey(entry.addressId));
if (gate !== undefined) await gate;
const record =
this.#host.cached(entry.addressId)?.record ??
(await this.#host.storage.findDocument(entry.address, "current", this.#context));
@ -850,6 +876,22 @@ export class Transaction implements Tx {
return taken;
}
/**
* Internal: the keys this commit writes, once settled (and after `takeHeld()`), for the Session's unpublished write
* set: its table records, and the address and incarnation of every document it creates, changes, or retires, with
* the scope of every incarnation it creates or retires.
*/
writeKeys(): string[] {
const keys: string[] = [];
tableKeys(this.#writes, keys);
for (const plan of this.#plans) {
if (plan.content === undefined && !plan.retire) continue;
keys.push(documentKey(plan.addressId), incarnationKey(plan.record.id));
if (plan.retire || !("createdAt" in plan.record)) keys.push(scopeKey(plan.record.scope));
}
return keys;
}
/** Stage a content write for an incarnation, before its retirement when it retires. */
#insertBeforeRetirement(write: Extract<StorageWrite, { readonly type: "document.change" }>): void {
const at = this.#writes.findIndex(
@ -944,8 +986,21 @@ export class Transaction implements Tx {
plan.retire = true;
retiring.add(plan.record.id);
}
const retire = (record: DocumentRecord): void => {
if (retiring.has(record.id)) return;
plans.push({ addressId: addressId(record), record, retire: true });
retiring.add(record.id);
};
for (const taskId of terminalTaskIds) {
if (this.#tasksById.get(taskId)?.write?.kind === "create") continue;
// The Session knows the documents of the tasks it created; any other task's are scanned.
const gate = this.#host.touch(scopeKey({ kind: "task", taskId }));
if (gate !== undefined) await gate;
const known = this.#host.taskDocuments(taskId);
if (known !== undefined) {
for (const record of known) retire(record);
continue;
}
let cursor: Cursor | undefined;
do {
const page = await storage.scanDocuments(
@ -954,11 +1009,7 @@ export class Transaction implements Tx {
cursor,
this.#context,
);
for (const record of page.items) {
if (retiring.has(record.id)) continue;
plans.push({ addressId: addressId(record), record, retire: true });
retiring.add(record.id);
}
for (const record of page.items) retire(record);
cursor = page.next;
} while (cursor !== undefined);
}

361
packages/durable/src/session/writer.ts Normal file
View file

@ -0,0 +1,361 @@
// The Session's writer: commits the mutation line admitted, stored in batches off the line and published in line order.
import type { Context } from "@earendil-works/chord";
import { withoutAbortSignal } from "@earendil-works/chord/context";
import { SessionFailed } from "../errors.ts";
import type { CommitResult, Seq, StorageWrite } from "../types.ts";
/** A part's own failure at publication: memory could not adopt it, or one of the Session's own observers threw. */
export type PublishFailure = { readonly kind: "adoption" | "listener"; readonly error: unknown };
/** One logical commit the line admitted: stored (`writes`, never empty), or held in memory only (no `writes`). */
export type PartSpec = {
/** Line index of the job that admitted it: a failure it causes makes only later jobs' outcomes `SessionFailed`. */
readonly index: number;
readonly writes: readonly StorageWrite[] | undefined;
/** Keys it writes: a read of one waits for its publication. */
readonly keys: readonly string[];
readonly context: Context;
/** Publish it in memory, in line order, once it is durable (with its seq) or, held, once the parts before it are. */
readonly publish: (seq: Seq | undefined) => PublishFailure | undefined;
/** It will never be published: abort what it prepared. */
readonly discard: () => void;
};
/** An admitted part and its outcome. */
export type Part = PartSpec & {
/**
* Resolves once it is published; rejects once it never will be, and also after publication when its storage call
* rejected or the Session failed meanwhile. Always handled, so a part nobody awaits rejects silently.
*/
readonly outcome: Promise<void>;
};
type Admitted = Part & {
readonly settle: { resolve(): void; reject(error: unknown): void };
/** Set once its outcome is settled. */
done: boolean;
/** Opens once it is published or never will be; never rejects. */
readonly gate: PromiseWithResolvers<void>;
};
/** How a part that will not resolve settles: with an error of its own, which may be any value, or `SessionFailed`. */
type FailedOutcome = { readonly kind: "own"; readonly error: unknown } | { readonly kind: "failed" };
const SESSION_FAILED: FailedOutcome = { kind: "failed" };
/** What the writer needs of its Session. */
export type WriterHost = {
commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
context: Context,
): Promise<CommitResult>;
/** Apply a stored batch to the Session's read caches, right before it is published. */
stored(writes: readonly StorageWrite[]): void;
/**
* Run one part's publication inside the Storage's own scope (`Storage.publishing`); `undefined` when it ran at once,
* else settles once it ran.
*/
publishing(publish: () => void, context: Context): Promise<void> | undefined;
/** The Session's failure, if it failed. */
failure(): { readonly error: unknown } | undefined;
/**
* Fail the Session; `at` is the line index of the first job whose outcome the failure is, and `own` the jobs after it
* whose outcome is still their own error (the other parts of a call that rejected), not `SessionFailed`.
*/
fail(error: unknown, at: number, own: readonly number[]): void;
/** Report an own observer's throw that cannot fail the Session, as it fails with a storage error. */
report(error: unknown, context: Context): void;
};
/**
* The writer of one Session. The line admits each commit as a part once its callback and settlement succeeded; the
* writer sends every part admitted while no call is in flight as one `commit()` call, off the line, and publishes
* the parts of a call one by one, in line order, with a macrotask between two, so the continuations of the waits one
* resolves start before the next is published. Held commits are parts without writes, published when the parts before
* them are.
*
* Each part registers the keys it writes; until it is published, it is the newest unpublished writer of each, and a
* job that touches one waits for its gate. A key leaves when its newest writer is published, so the map holds one entry
* per key with an unpublished writer.
*/
export class Writer {
readonly #host: WriterHost;
/** Admitted parts not yet sent, in line order. */
#queue: Admitted[] = [];
#flight = false;
/** Per key, the newest unpublished part that writes it. */
readonly #unpublished = new Map<string, Admitted>();
/** The newest admitted part that is not yet published, whose gate opens after every earlier one's. */
#newest: Admitted | undefined;
constructor(host: WriterHost) {
this.#host = host;
}
/** Admit a part: register its keys, queue it, and send it at once unless a call is in flight. */
admit(spec: PartSpec): Part {
const outcome = Promise.withResolvers<void>();
outcome.promise.catch(() => {});
const part: Admitted = {
...spec,
outcome: outcome.promise,
settle: {
resolve: () => {
part.done = true;
outcome.resolve();
},
reject: (error) => {
part.done = true;
outcome.reject(error);
},
},
done: false,
gate: Promise.withResolvers<void>(),
};
for (const key of spec.keys) this.#unpublished.set(key, part);
this.#newest = part;
this.#queue.push(part);
this.#pump();
return part;
}
/** The gate of the newest unpublished writer of `key`, or `undefined` when it has none. */
gate(key: string): Promise<void> | undefined {
return this.#unpublished.get(key)?.gate.promise;
}
/** Opens once every part admitted so far is published or never will be. */
drained(): Promise<void> | undefined {
return this.#newest?.gate.promise;
}
/** Keys with an unpublished writer, for tests that bound memory. */
get unpublished(): number {
return this.#unpublished.size;
}
#pump(): void {
if (this.#flight || this.#queue.length === 0) return;
this.#flight = true;
const parts = this.#queue;
this.#queue = [];
void this.#flush(parts);
}
/**
* Send and publish `parts`, then what was admitted meanwhile. Never rejects: a throw the writer did not expect fails
* the Session and settles the parts (`#abandon()`), so no outcome or gate is stranded.
*/
async #flush(parts: readonly Admitted[]): Promise<void> {
try {
await this.#send(parts);
} catch (error) {
this.#abandon(parts, error);
}
this.#flight = false;
this.#pump();
}
/** Store `parts` in one call, then publish them in order; settles every one of them. */
async #send(parts: readonly Admitted[]): Promise<void> {
const stored = parts.filter((part) => part.writes !== undefined);
const seqs: (Seq | undefined)[] = [];
let result: CommitResult | undefined;
let rejected: { readonly error: unknown } | undefined;
const failed = this.#host.failure();
if (failed !== undefined) {
for (const part of parts) this.#unpublish(part, new SessionFailed(failed.error));
return;
}
if (stored.length > 0) {
try {
result = await this.#host.commit(
stored.map((part) => part.writes!),
(index, seq) => {
seqs[index] = seq;
},
withoutAbortSignal(stored[0]!.context),
);
} catch (error) {
rejected = { error };
}
}
// A call that settles once another failure failed the Session reports `SessionFailed` with that first error,
// whatever it returned (spec §10.1), and publishes nothing.
const failedMeanwhile = this.#host.failure();
if (failedMeanwhile !== undefined) {
for (const part of parts) this.#unpublish(part, new SessionFailed(failedMeanwhile.error));
return;
}
await this.#publish(parts, seqs, result, rejected);
}
/**
* Publish the received parts and held ones in order, a macrotask apart, and settle every part (group commit §3.4). A
* refusal, a rejection, or a part whose adoption or own observer throws fails the Session once the parts before it
* are published and the host observers got the last one; the parts after it are not published, and reject with
* `SessionFailed`. What the failure decides (the outcomes, and the gates of the parts it concerns) is settled only
* once the Session failed, so no job waiting at such a gate, and no later outcome, goes on as if it had not.
*/
async #publish(
parts: readonly Admitted[],
seqs: readonly (Seq | undefined)[],
result: CommitResult | undefined,
rejected: { readonly error: unknown } | undefined,
): Promise<void> {
let stop: { readonly error: unknown; readonly at: number; readonly external?: true } | undefined;
/**
* Settled once the Session failed: each part, whether it was published, and its outcome: an error of its own (any
* value, `undefined` included), or `SessionFailed`.
*/
const failing: { readonly part: Admitted; readonly outcome: FailedOutcome; readonly published: boolean }[] = [];
let storedIndex = 0;
let published = 0;
for (const part of parts) {
const index = part.writes === undefined ? undefined : storedIndex++;
if (stop !== undefined) {
// A part after a rejected call's last receipt has the call's error; any other part after a stop was not sent,
// nor any part once something else failed the Session.
const outcome: FailedOutcome =
rejected !== undefined && index !== undefined && stop.external !== true
? { kind: "own", error: rejected.error }
: SESSION_FAILED;
failing.push({ part, outcome, published: false });
continue;
}
const seq = index === undefined ? undefined : seqs[index];
if (index !== undefined && seq === undefined) {
const error =
rejected !== undefined
? rejected.error
: index === result?.stored && "refused" in result
? result.refused
: new Error("Storage committed without adopting the batch");
failing.push({ part, outcome: { kind: "own", error }, published: false });
stop = { error, at: part.index };
continue;
}
// A macrotask between two parts: what the first resolves runs before the second is published.
if (published++ > 0) await hop();
let failure: PublishFailure | undefined;
// Failed by something else meanwhile, during the hop or before a storage's scope runs the publication: nothing
// more of the call is published. Checked right before it would be, after every wait.
let external: { readonly error: unknown } | undefined;
const publication = this.#host.publishing(() => {
external = this.#host.failure();
if (external !== undefined) return;
// Its keys leave first, so what its publication schedules sees it published; its waiters wake after it.
this.#unregister(part);
// A throw anywhere in it, the read caches included, leaves memory behind what is stored: like an adoption's.
try {
if (part.writes !== undefined) this.#host.stored(part.writes);
failure = part.publish(seq);
} catch (error) {
failure = { kind: "adoption", error };
}
}, part.context);
if (publication !== undefined) await publication;
if (external !== undefined) {
stop = { error: external.error, at: part.index, external: true };
failing.push({ part, outcome: SESSION_FAILED, published: false });
continue;
}
if (failure?.kind === "adoption") {
failing.push({ part, outcome: { kind: "own", error: failure.error }, published: true });
stop = { error: failure.error, at: part.index };
continue;
}
// A part published while its call rejected is durable but rejects (spec §10.1); an own observer's throw is then
// reported, as the Session fails with the storage error.
if (rejected !== undefined) {
if (failure !== undefined) this.#host.report(failure.error, part.context);
failing.push({ part, outcome: { kind: "own", error: rejected.error }, published: true });
continue;
}
// An own observer's throw: the commit stands; the Session fails after it.
if (failure !== undefined) stop = { error: failure.error, at: part.index };
const failedMeanwhile = this.#host.failure();
if (failedMeanwhile !== undefined && failure === undefined) {
part.settle.reject(new SessionFailed(failedMeanwhile.error));
} else part.settle.resolve();
this.#open(part);
}
if (rejected !== undefined || stop !== undefined) {
// After the host dispatch of the last part published, which is one microtask away.
await Promise.resolve();
// Every part of the call that rejected keeps the storage error; the first of them is the failure's boundary.
const own = failing.flatMap(({ part, outcome }) => (outcome.kind === "own" ? [part.index] : []));
if (rejected !== undefined) this.#host.fail(rejected.error, own[0]!, own);
else this.#host.fail(stop!.error, stop!.at, own);
}
for (const { part, outcome, published: stored } of failing) {
const error = outcome.kind === "own" ? outcome.error : this.#failed(stop!.error);
if (stored) {
part.settle.reject(error);
this.#open(part);
} else this.#unpublish(part, error);
}
// Admitted while this call ran, after the failure: never sent.
const failure = this.#host.failure();
if (failure !== undefined) {
const queued = this.#queue;
this.#queue = [];
for (const part of queued) this.#unpublish(part, new SessionFailed(failure.error));
}
}
/**
* A throw the writer itself did not expect while sending or publishing `parts` (a bug): fail the Session at the first
* of them and settle every one not settled yet, so no outcome, gate, or close waits on it forever.
*/
#abandon(parts: readonly Admitted[], error: unknown): void {
this.#host.fail(error, parts[0]!.index, []);
for (const part of parts) {
if (!part.done) this.#unpublish(part, this.#failed(error));
this.#unregister(part);
this.#open(part);
}
}
/** `SessionFailed` with the Session's failure, or with `error` until the Session records it. */
#failed(error: unknown): SessionFailed {
const failure = this.#host.failure();
return new SessionFailed(failure !== undefined ? failure.error : error);
}
/** Settle a part that will never be published: discard it, reject it with `error`, and open its gate. */
#unpublish(part: Admitted, error: unknown): void {
part.discard();
part.settle.reject(error);
this.#unregister(part);
this.#open(part);
}
/** Drop the keys a part is still the newest writer of: it is published, or never will be. */
#unregister(part: Admitted): void {
for (const key of part.keys) if (this.#unpublished.get(key) === part) this.#unpublished.delete(key);
}
/** Open a part's gate: the jobs waiting for it go on. The newest part's opens `drained()` too. */
#open(part: Admitted): void {
part.gate.resolve();
if (this.#newest === part) this.#newest = undefined;
}
}
/**
* Yield to the event loop once: `setImmediate` where there is one, otherwise a `MessageChannel` message (Workers,
* browsers). Microtasks queued before it, such as wait continuations, run first.
*/
function hop(): Promise<void> {
if (typeof setImmediate === "function") return new Promise((resolve) => setImmediate(resolve));
const channel = new MessageChannel();
return new Promise((resolve) => {
channel.port1.onmessage = () => {
channel.port1.close();
resolve();
};
channel.port2.postMessage(undefined);
});
}

157
packages/durable/src/storage/indexeddb.ts
View file

@ -5,6 +5,7 @@ import { apply, type Op } from "@earendil-works/chord/delta";
import { StorageRequestError } from "../errors.ts";
import { idFromNumber, seqFromNumber } from "../ids.ts";
import type {
CommitResult,
ConversationId,
ConversationQuery,
ConversationRecord,
@ -138,12 +139,33 @@ type IndexedDbStorageOptions = {
readonly indexedDB?: IDBFactory;
};
/** A failure IndexedDB reported without an error of its own: the database's, like every `DOMException` it throws. */
class IndexedDbFailure extends Error {}
/**
* Whether `error`, or an error it was caused by (a copy's rejection wraps its cause), is IndexedDB's own: a
* `DOMException`, or a failure it reported without one. Any other error of a batch's checks is one of Durable's.
*/
function fromIndexedDb(error: unknown): boolean {
const seen = new Set<unknown>();
for (let at: unknown = error; at !== null && typeof at === "object" && !seen.has(at); ) {
if (at instanceof DOMException || at instanceof IndexedDbFailure) return true;
seen.add(at);
at = (at as { cause?: unknown }).cause;
}
return false;
}
const requestResult = <T>(request: IDBRequest<T>): Promise<T> =>
new Promise<T>((resolve, reject) => {
request.addEventListener("success", () => resolve(request.result), { once: true });
request.addEventListener("error", () => reject(request.error ?? new Error("IndexedDB request failed")), {
once: true,
});
request.addEventListener(
"error",
() => reject(request.error ?? new IndexedDbFailure("IndexedDB request failed")),
{
once: true,
},
);
});
const transactionResult = (transaction: IDBTransaction): Promise<void> =>
@ -151,12 +173,12 @@ const transactionResult = (transaction: IDBTransaction): Promise<void> =>
transaction.addEventListener("complete", () => resolve(), { once: true });
transaction.addEventListener(
"abort",
() => reject(transaction.error ?? new Error("IndexedDB transaction was aborted")),
() => reject(transaction.error ?? new IndexedDbFailure("IndexedDB transaction was aborted")),
{ once: true },
);
transaction.addEventListener(
"error",
() => reject(transaction.error ?? new Error("IndexedDB transaction failed")),
() => reject(transaction.error ?? new IndexedDbFailure("IndexedDB transaction failed")),
{ once: true },
);
});
@ -176,9 +198,13 @@ const cursorValues = <T>(
new Promise<T[]>((resolve, reject) => {
const values: T[] = [];
const request = source.openCursor(range, direction);
request.addEventListener("error", () => reject(request.error ?? new Error("IndexedDB cursor failed")), {
once: true,
});
request.addEventListener(
"error",
() => reject(request.error ?? new IndexedDbFailure("IndexedDB cursor failed")),
{
once: true,
},
);
request.addEventListener("success", () => {
const cursor = request.result;
if (cursor === null) {
@ -203,9 +229,13 @@ const firstCursorValue = <T>(
): Promise<T | undefined> =>
new Promise<T | undefined>((resolve, reject) => {
const request = source.openCursor(range, direction);
request.addEventListener("error", () => reject(request.error ?? new Error("IndexedDB cursor failed")), {
once: true,
});
request.addEventListener(
"error",
() => reject(request.error ?? new IndexedDbFailure("IndexedDB cursor failed")),
{
once: true,
},
);
request.addEventListener("success", () => {
const cursor = request.result;
if (cursor === null) {
@ -417,29 +447,54 @@ export class IndexedDbStorage implements Storage {
await requestResult(factory.deleteDatabase(name));
}
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, _context: Context): Promise<Seq> {
/**
* Commit `batches` in one IndexedDB transaction, each with its own seq. Each batch does every read and check, copy
* resolution included, against the state the earlier ones left before it issues its first write; the first one
* refused ends the call, and the transaction commits the ones before it. An error of IndexedDB itself aborts the
* transaction: the call rejects with nothing stored.
*/
async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
_context: Context,
): Promise<CommitResult> {
this.assertOpen();
const documentActions = this.prepareDocumentActions(writes);
const seq = await this.transaction(ALL_STORES, "readwrite", async (transaction) => {
const seqs: Seq[] = [];
let refused: { readonly error: unknown } | undefined;
await this.transaction(ALL_STORES, "readwrite", async (transaction) => {
const metaStore = transaction.objectStore(STORES.meta);
const metadata = await requestResult(metaStore.get(META_KEY) as IDBRequest<MetaRow>);
if (metadata === undefined) throw new Error("Durable IndexedDB metadata is missing");
const seq = seqFromNumber(metadata.nextSeq);
await this.checkGlobalIds(transaction, writes);
await this.checkDocumentActions(transaction, documentActions);
for (const write of writes) await this.applyTableWrite(transaction, write, seq);
await this.applyDocumentActions(transaction, documentActions, seq);
let next = metadata.nextSeq;
let nextId = metadata.nextId;
for (const write of writes) {
const id = writeId(write);
if (id !== undefined) nextId = Math.max(nextId, id + 1);
for (const writes of batches) {
let documentActions: Map<DocumentId, DocumentAction>;
try {
documentActions = this.prepareDocumentActions(writes);
await this.checkGlobalIds(transaction, writes);
await this.checkDocumentActions(transaction, documentActions);
await this.resolveCopies(transaction, documentActions);
} catch (error) {
if (fromIndexedDb(error)) throw error;
refused = { error };
break;
}
const seq = seqFromNumber(next++);
for (const write of writes) await this.applyTableWrite(transaction, write, seq);
await this.applyDocumentActions(transaction, documentActions, seq);
for (const write of writes) {
const id = writeId(write);
if (id !== undefined) nextId = Math.max(nextId, id + 1);
}
seqs.push(seq);
}
if (seqs.length > 0) {
await requestResult(metaStore.put({ key: META_KEY, nextId, nextSeq: next } satisfies MetaRow));
}
await requestResult(metaStore.put({ key: META_KEY, nextId, nextSeq: seq + 1 } satisfies MetaRow));
return seq;
});
// The transaction completed: the batch is durable.
adopt(seq);
return seq;
// The transaction completed: the stored batches are durable.
for (const [index, seq] of seqs.entries()) receive(index, seq);
return refused === undefined ? { stored: seqs.length } : { stored: seqs.length, refused: refused.error };
}
async mintId<I extends Id<string>>(): Promise<I> {
@ -1124,6 +1179,31 @@ export class IndexedDbStorage implements Storage {
}
}
/** Resolve each copy to a base of its source's state before this batch, which may not change the source. */
private async resolveCopies(transaction: IDBTransaction, actions: Map<DocumentId, DocumentAction>): Promise<void> {
for (const [id, action] of actions) {
if (action.copy === undefined) continue;
try {
const stored = await this.materializeDocument(transaction, action.copy.id, action.copy.at);
if (stored === undefined) throw new Error(`Fork source document ${action.copy.id} cannot be read`);
const create = action.create!;
if (
stored.record.scope.kind !== "conversation" ||
create.scope.kind !== "conversation" ||
stored.record.kind !== create.kind ||
stored.record.key !== create.key ||
stored.record.history !== create.history ||
stored.record.fork !== create.fork
) {
throw new Error(`Fork source document ${action.copy.id} does not match the copied record`);
}
action.content = { kind: "base", version: stored.version, value: stored.value };
} catch (error) {
throw new Error(`Document copy ${id} was rejected`, { cause: error });
}
}
}
private async applyDocumentActions(
transaction: IDBTransaction,
actions: ReadonlyMap<DocumentId, DocumentAction>,
@ -1132,27 +1212,8 @@ export class IndexedDbStorage implements Storage {
const documents = transaction.objectStore(STORES.documents);
const revisions = transaction.objectStore(STORES.documentRevisions);
for (const [id, action] of actions) {
let content = action.content;
if (action.copy !== undefined) {
try {
const stored = await this.materializeDocument(transaction, action.copy.id, action.copy.at);
if (stored === undefined) throw new Error(`Fork source document ${action.copy.id} cannot be read`);
const create = action.create!;
if (
stored.record.scope.kind !== "conversation" ||
create.scope.kind !== "conversation" ||
stored.record.kind !== create.kind ||
stored.record.key !== create.key ||
stored.record.history !== create.history ||
stored.record.fork !== create.fork
) {
throw new Error(`Fork source document ${action.copy.id} does not match the copied record`);
}
content = { kind: "base", version: stored.version, value: stored.value };
} catch (error) {
throw new Error(`Document copy ${id} was rejected`, { cause: error });
}
}
// Copies are resolved to their content by `resolveCopies()`.
const content = action.content;
let record: DocumentRecord;
if (action.create !== undefined) {
record = {

113
packages/durable/src/storage/jsonl/storage.ts
View file

@ -3,6 +3,7 @@ import type { FileError, FileSystem } from "../../env/index.ts";
import { StorageRequestError } from "../../errors.ts";
import { idFromNumber, seqFromNumber } from "../../ids.ts";
import type {
CommitResult,
ConversationId,
ConversationQuery,
Cursor,
@ -249,6 +250,8 @@ export class JsonlStorage implements Storage {
private readonly liveTaskSidecars = new Set<TaskId>();
private closed = false;
private poisonError: JsonlStoragePoisonedError | undefined;
/** Settles once the batch being appended is; its batches are in memory already (`commit()`). */
private writing: Promise<void> | undefined;
private constructor(fs: FileSystem, directory: string, mainPath: string, options: JsonlStorageOptions) {
this.fs = fs;
@ -275,19 +278,74 @@ export class JsonlStorage implements Storage {
return storage;
}
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq> {
/**
* Commit `batches` with one append per affected file: every batch's sidecar records first, then their markers in one
* write of `main.jsonl`. Each batch is checked against the state the earlier ones left, so memory applies each one
* as it passes, and reads wait until the appends are done; the first batch refused ends the batch. A torn append of
* the markers recovers a shorter complete prefix; any failure once a batch applied poisons the storage.
*/
async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
context: Context,
): Promise<CommitResult> {
this.assertUsable();
const prepared = this.memory.prepareCommit(writes);
const encoded = this.encodeCommit(prepared.seq, prepared.writes);
const reclamations = this.planReclamations(prepared.writes, encoded);
const seqs: Seq[] = [];
const encoded: EncodedCommit[] = [];
const reclamations = new Map<string, string>();
let refused: { readonly error: unknown } | undefined;
const writing = Promise.withResolvers<void>();
this.writing = writing.promise;
try {
for (const writes of batches) {
let prepared: ReturnType<MemoryStorage["prepareCommit"]>;
try {
prepared = this.memory.prepareCommit(writes);
} catch (error) {
refused = { error };
break;
}
const commit = this.encodeCommit(prepared.seq, prepared.writes);
// A sidecar an earlier batch replaces keeps what the later ones append, unless they replace it again.
for (const [file, content] of commit.sidecars) {
const replacement = reclamations.get(file);
if (replacement !== undefined) reclamations.set(file, replacement + content);
}
for (const [file, content] of this.planReclamations(prepared.writes, commit))
reclamations.set(file, content);
seqs.push(prepared.apply());
this.adoptSidecarState(prepared.writes);
encoded.push(commit);
}
if (encoded.length > 0) await this.append(encoded, context);
} catch (error) {
// Memory runs ahead of the file: once a batch applied, any failure poisons the storage before the reads
// waiting on the append go on, so none of them returns what is not stored.
if (seqs.length === 0) throw error;
throw this.poison(error instanceof Error ? error : new Error("JSONL commit failed", { cause: error }));
} finally {
this.writing = undefined;
writing.resolve();
}
// The markers are appended: the batches are durable. Reclaiming sidecars comes after the receipts.
for (const [index, seq] of seqs.entries()) receive(index, seq);
await this.reclaimSidecars(reclamations, context);
return refused === undefined ? { stored: seqs.length } : { stored: seqs.length, refused: refused.error };
}
/** Append the sidecar records of `commits`, flushed with `fsync`, then their markers in one write of `main.jsonl`. */
private async append(commits: readonly EncodedCommit[], context: Context): Promise<void> {
const contents = new Map<string, string>();
for (const commit of commits) {
for (const [file, content] of commit.sidecars) contents.set(file, (contents.get(file) ?? "") + content);
}
const sidecars = await Promise.all(
[...encoded.sidecars].map(async ([file, content]) => ({
[...contents].map(async ([file, content]) => ({
file,
content,
path: await this.resolveFile(file, context),
})),
);
for (const sidecar of sidecars) {
const result = await this.fs.appendFile(sidecar.path, sidecar.content, context);
if (!result.ok) throw this.poison(errorFromFile(`append to ${sidecar.file}`, result.error));
@ -298,14 +356,9 @@ export class JsonlStorage implements Storage {
if (!result.ok) throw this.poison(errorFromFile(`flush of ${sidecar.file}`, result.error));
}
}
const marker = await this.fs.appendFile(this.mainPath, encoded.marker, context);
const markers = commits.length === 1 ? commits[0]!.marker : commits.map((commit) => commit.marker).join("");
const marker = await this.fs.appendFile(this.mainPath, markers, context);
if (!marker.ok) throw this.poison(errorFromFile(`append to ${MAIN_FILE}`, marker.error));
const seq = prepared.apply();
this.adoptSidecarState(prepared.writes);
// The marker is appended: the batch is durable. Reclaiming sidecars comes after adoption.
adopt(seq);
await this.reclaimSidecars(reclamations, context);
return seq;
}
async mintId<I extends Id<string>>(): Promise<I> {
@ -313,20 +366,20 @@ export class JsonlStorage implements Storage {
}
async conversation(id: ConversationId, context: Context) {
return this.store.conversation(id, context);
return (await this.readable()).conversation(id, context);
}
async scanConversations(query: ConversationQuery, limit: number, cursor: Cursor | undefined, context: Context) {
return this.store.scanConversations(query, limit, cursor, context);
return (await this.readable()).scanConversations(query, limit, cursor, context);
}
entry(id: EntryId, context: Context): ReturnType<Storage["entry"]>;
entry(conversationId: ConversationId, id: EntryId, context: Context): ReturnType<Storage["entry"]>;
async entry(idOrConversationId: EntryId | ConversationId, idOrContext: EntryId | Context, context?: Context) {
if (context === undefined)
return this.store.entry(idFromNumber<EntryId>(idOrConversationId), idOrContext as Context);
return (await this.readable()).entry(idFromNumber<EntryId>(idOrConversationId), idOrContext as Context);
if (typeof idOrContext !== "number") throw new StorageRequestError("Storage.entry() requires an entry ID");
return this.store.entry(
return (await this.readable()).entry(
idFromNumber<ConversationId>(idOrConversationId),
idFromNumber<EntryId>(idOrContext),
context,
@ -338,43 +391,43 @@ export class JsonlStorage implements Storage {
atOrBeforeEntryId: EntryId | undefined,
context: Context,
) {
return this.store.findLatestHeadMarker(conversationId, atOrBeforeEntryId, context);
return (await this.readable()).findLatestHeadMarker(conversationId, atOrBeforeEntryId, context);
}
async scanEntries(query: EntryQuery, limit: number, cursor: Cursor | undefined, context: Context) {
return this.store.scanEntries(query, limit, cursor, context);
return (await this.readable()).scanEntries(query, limit, cursor, context);
}
async task(id: TaskId, context: Context) {
return this.store.task(id, context);
return (await this.readable()).task(id, context);
}
async scanTasks(query: TaskQuery, limit: number, cursor: Cursor | undefined, context: Context) {
return this.store.scanTasks(query, limit, cursor, context);
return (await this.readable()).scanTasks(query, limit, cursor, context);
}
async submission(id: SubmissionId, context: Context) {
return this.store.submission(id, context);
return (await this.readable()).submission(id, context);
}
async scanSubmissions(query: SubmissionQuery, limit: number, cursor: Cursor | undefined, context: Context) {
return this.store.scanSubmissions(query, limit, cursor, context);
return (await this.readable()).scanSubmissions(query, limit, cursor, context);
}
async submissionByRequest(conversationId: ConversationId, requestId: string, context: Context) {
return this.store.submissionByRequest(conversationId, requestId, context);
return (await this.readable()).submissionByRequest(conversationId, requestId, context);
}
async findDocument(address: DocumentAddress, at: DocumentPoint, context: Context) {
return this.store.findDocument(address, at, context);
return (await this.readable()).findDocument(address, at, context);
}
async document(id: DocumentId, at: DocumentPoint, context: Context) {
return this.store.document(id, at, context);
return (await this.readable()).document(id, at, context);
}
async scanDocuments(query: DocumentQuery, limit: number, cursor: Cursor | undefined, context: Context) {
return this.store.scanDocuments(query, limit, cursor, context);
return (await this.readable()).scanDocuments(query, limit, cursor, context);
}
async close(context: Context): Promise<void> {
@ -836,6 +889,12 @@ export class JsonlStorage implements Storage {
return this.memory;
}
/** The state to read once no batch is being appended: memory runs ahead of the file while one is. */
private async readable(): Promise<MemoryStorage> {
while (this.writing !== undefined) await this.writing;
return this.store;
}
private poison(cause: Error): JsonlStoragePoisonedError {
this.poisonError ??= new JsonlStoragePoisonedError(cause);
return this.poisonError;

34
packages/durable/src/storage/memory.ts
View file

@ -3,6 +3,7 @@ import { applyImmutableBatches, type Op } from "@earendil-works/chord/delta";
import { StorageRequestError } from "../errors.ts";
import { idFromNumber, seqFromNumber } from "../ids.ts";
import type {
CommitResult,
ConversationId,
ConversationQuery,
ConversationRecord,
@ -260,10 +261,35 @@ export class MemoryStorage implements Storage {
private nextSeq = 1;
private closed = false;
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, _context: Context): Promise<Seq> {
const seq = this.prepareCommit(writes).apply();
adopt(seq);
return seq;
/**
* Apply `batches` one by one, each checked against the state the earlier ones left (`prepareCommit()`, which only
* checks and detaches, so a batch it refuses changed nothing); stops at the first one refused. Memory is durable at
* once; the receipts follow the last batch.
*/
async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
_context: Context,
): Promise<CommitResult> {
this.assertOpen();
const seqs: Seq[] = [];
let refused: { readonly error: unknown } | undefined;
try {
for (const writes of batches) {
let prepared: PreparedMemoryCommit;
try {
prepared = this.prepareCommit(writes);
} catch (error) {
refused = { error };
break;
}
seqs.push(prepared.apply());
}
} finally {
// Also when a batch's apply throws: the batches applied before it are stored, and their receipts stand.
for (const [index, seq] of seqs.entries()) receive(index, seq);
}
return refused === undefined ? { stored: seqs.length } : { stored: seqs.length, refused: refused.error };
}
/** Validate and detach one commit without changing observable state. */

176
packages/durable/src/storage/sqlite/storage.ts
View file

@ -4,6 +4,7 @@ import type { SqliteDatabase, SqliteExecutor, SqliteValue } from "@earendil-work
import { StorageRequestError } from "../../errors.ts";
import { idFromNumber, seqFromNumber } from "../../ids.ts";
import type {
CommitResult,
ConversationId,
ConversationQuery,
ConversationRecord,
@ -148,6 +149,31 @@ const writeId = (write: StorageWrite): Id<string> | undefined => {
}
};
/** Whether a database call of the checks failed, whatever it threw or rejected with. */
type CheckCalls = { failed: boolean };
/**
* `executor`, setting `calls.failed` when one of its calls throws, synchronously (a synchronous driver) or by
* rejecting, with any value: then a failing check is the database's, not one of Durable's own.
*/
function watchCalls(executor: SqliteExecutor, calls: CheckCalls): SqliteExecutor {
const watched = async <T>(call: () => Promise<T>): Promise<T> => {
try {
return await call();
} catch (error) {
calls.failed = true;
throw error;
}
};
return {
exec: (sql) => watched(() => executor.exec(sql)),
run: (sql, ...params) => watched(() => executor.run(sql, ...params)),
runCounted: (sql, ...params) => watched(() => executor.runCounted(sql, ...params)),
get: (sql, ...params) => watched(() => executor.get(sql, ...params)),
all: (sql, ...params) => watched(() => executor.all(sql, ...params)),
};
}
/** Portable SQLite implementation of the Pico storage contract. */
export class SqliteStorage implements Storage {
private readonly db: SqliteDatabase;
@ -184,31 +210,89 @@ export class SqliteStorage implements Storage {
}
}
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, _context: Context): Promise<Seq> {
/**
* Commit `batches` in one SQL transaction, each with its own seq. Each batch is checked before it writes anything,
* against the state the earlier ones left, and the first one refused ends the batch: the transaction commits the ones
* before it. No savepoints, which Durable Objects do not allow: the checks run before the writes instead.
*/
async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
_context: Context,
): Promise<CommitResult> {
this.assertOpen();
const documentActions = this.prepareDocumentActions(writes);
const candidateNextId = this.candidateNextId(writes);
const seq = await this.db.transaction(async (transaction) => {
const metadata = await transaction.get<MetadataRow>(
`SELECT next_id, next_seq FROM ${this.schema.tables.durableMetadata} WHERE singleton = 1`,
);
if (metadata === undefined) throw new Error("Durable SQLite metadata is missing");
const committedSeq = seqFromNumber(metadata.next_seq);
await this.checkGlobalIds(transaction, writes);
await this.checkDocumentActions(transaction, documentActions);
for (const write of writes) await this.applyTableWrite(transaction, write, committedSeq);
await this.applyDocumentActions(transaction, documentActions, committedSeq);
await transaction.run(
`UPDATE ${this.schema.tables.durableMetadata} SET next_id = ?, next_seq = ? WHERE singleton = 1`,
String(Math.max(Number(metadata.next_id), candidateNextId)),
committedSeq + 1,
);
return committedSeq;
const seqs: Seq[] = [];
let refused: { readonly error: unknown } | undefined;
let candidateNextId = this.nextId;
await this.db.transaction(async (transaction) => {
const metadata = await this.readMetadata(transaction);
let next = metadata.next_seq;
// Whether a database call of a batch's checks failed: then the failure is an unknown outcome, not a refusal.
const calls: CheckCalls = { failed: false };
const checked = watchCalls(transaction, calls);
for (const writes of batches) {
let documentActions: Map<DocumentId, DocumentAction>;
// Only Durable's own checks refuse a batch, before it wrote anything: the ones before it stand. A failure of
// the database rejects the call, and rolls back the batches before it with the transaction.
calls.failed = false;
try {
documentActions = this.prepareDocumentActions(writes);
await this.checkBatch(checked, writes, documentActions);
} catch (error) {
if (calls.failed) throw error;
refused = { error };
break;
}
const seq = seqFromNumber(next++);
await this.applyBatch(transaction, writes, documentActions, seq);
seqs.push(seq);
candidateNextId = Math.max(candidateNextId, this.candidateNextId(writes));
}
if (seqs.length > 0) {
await this.writeMetadata(transaction, Math.max(Number(metadata.next_id), candidateNextId), next);
}
});
this.nextId = Math.max(this.nextId, candidateNextId);
// `COMMIT` returned: the batch is durable.
adopt(seq);
return seq;
// `COMMIT` returned: the stored batches are durable.
for (const [index, seq] of seqs.entries()) receive(index, seq);
return refused === undefined ? { stored: seqs.length } : { stored: seqs.length, refused: refused.error };
}
private async readMetadata(executor: SqliteExecutor): Promise<MetadataRow> {
const metadata = await executor.get<MetadataRow>(
`SELECT next_id, next_seq FROM ${this.schema.tables.durableMetadata} WHERE singleton = 1`,
);
if (metadata === undefined) throw new Error("Durable SQLite metadata is missing");
return metadata;
}
private async writeMetadata(executor: SqliteExecutor, nextId: number, nextSeq: number): Promise<void> {
await executor.run(
`UPDATE ${this.schema.tables.durableMetadata} SET next_id = ?, next_seq = ? WHERE singleton = 1`,
String(nextId),
nextSeq,
);
}
/** Durable's own checks of one batch, copy resolution included, before it writes anything. */
private async checkBatch(
executor: SqliteExecutor,
writes: readonly StorageWrite[],
documentActions: Map<DocumentId, DocumentAction>,
): Promise<void> {
await this.checkGlobalIds(executor, writes);
await this.checkDocumentActions(executor, documentActions);
await this.resolveCopies(executor, documentActions);
}
private async applyBatch(
executor: SqliteExecutor,
writes: readonly StorageWrite[],
documentActions: ReadonlyMap<DocumentId, DocumentAction>,
seq: Seq,
): Promise<void> {
for (const write of writes) await this.applyTableWrite(executor, write, seq);
await this.applyDocumentActions(executor, documentActions, seq);
}
async mintId<I extends Id<string>>(): Promise<I> {
@ -863,33 +947,39 @@ export class SqliteStorage implements Storage {
);
}
/** Resolve each copy to a base of its source's state before this batch, which may not change the source. */
private async resolveCopies(executor: SqliteExecutor, actions: Map<DocumentId, DocumentAction>): Promise<void> {
for (const [id, action] of actions) {
if (action.copy === undefined) continue;
try {
const stored = await this.materializeDocument(executor, action.copy.id, action.copy.at);
if (stored === undefined) throw new Error(`Fork source document ${action.copy.id} cannot be read`);
const create = action.create!;
if (
stored.record.scope.kind !== "conversation" ||
create.scope.kind !== "conversation" ||
stored.record.kind !== create.kind ||
stored.record.key !== create.key ||
stored.record.history !== create.history ||
stored.record.fork !== create.fork
) {
throw new Error(`Fork source document ${action.copy.id} does not match the copied record`);
}
action.content = { kind: "base", version: stored.version, value: stored.value };
} catch (error) {
throw new Error(`Document copy ${id} was rejected`, { cause: error });
}
}
}
private async applyDocumentActions(
executor: SqliteExecutor,
actions: ReadonlyMap<DocumentId, DocumentAction>,
seq: Seq,
): Promise<void> {
for (const [id, action] of actions) {
let content = action.content;
if (action.copy !== undefined) {
try {
const stored = await this.materializeDocument(executor, action.copy.id, action.copy.at);
if (stored === undefined) throw new Error(`Fork source document ${action.copy.id} cannot be read`);
const create = action.create!;
if (
stored.record.scope.kind !== "conversation" ||
create.scope.kind !== "conversation" ||
stored.record.kind !== create.kind ||
stored.record.key !== create.key ||
stored.record.history !== create.history ||
stored.record.fork !== create.fork
) {
throw new Error(`Fork source document ${action.copy.id} does not match the copied record`);
}
content = { kind: "base", version: stored.version, value: stored.value };
} catch (error) {
throw new Error(`Document copy ${id} was rejected`, { cause: error });
}
}
// Copies are resolved to their content by `resolveCopies()`.
const content = action.content;
let record: DocumentRecord;
if (action.create !== undefined) {
record = {

58
packages/durable/src/storage/store/commit.ts
View file

@ -1,8 +1,9 @@
// One Durable commit as one composite write of the store's commit step (slice 6b §2.3): staged before the queue (rows
// encoded, contents serialized, explicit ids raising the store's allocator, fork copies resolved from their sources),
// then inside the step what `SqliteStorage` does inside its transaction, with the step's reader: Durable's checks of
// ids and documents (its messages), the table puts, and the documents in two passes, existing ones first (their content,
// then their retirement, which frees the addresses this commit's creations take), then the creations and copies.
// One Durable commit as one commit of the store's series (slice 6b §2.3): staged before the queue (rows encoded,
// contents serialized, explicit ids raising the store's allocator), then inside the series' transaction what
// `SqliteStorage` does inside its own, with the step's reader: Durable's checks of ids and documents and the copies
// resolved from their sources (its messages, a refusal), then its one composite write: the table puts, and the
// documents in two passes, existing ones first (their content, then their retirement, which frees the addresses this
// commit's creations take), then the creations and copies.
import type { Context, JsonValue } from "@earendil-works/chord";
import {
@ -14,22 +15,23 @@ import {
type OwnerTable,
readDocument,
retireRecord,
type SeriesCommit,
type StagedWrite,
type Step,
type StoreWrite,
writeContent,
} from "@earendil-works/chord/next";
import { StorageRequestError } from "../../errors.ts";
import type {
DocumentContent,
DocumentCopySource,
DocumentCreate,
DocumentId,
DocumentRecord,
Id,
JsonObject,
StorageWrite,
} from "../../types.ts";
import { addressOf, durableRecord, engineRecord } from "./documents.ts";
import { addressOf, durableRecord, engineRecord, readAt } from "./documents.ts";
import { conversationRow, entryRow, submissionRow, taskRow } from "./tables.ts";
/** Durable's record types, as `record_ids` names them. */
@ -66,14 +68,10 @@ type DocumentAction = {
retire: boolean;
};
/** Reads a copy's source at its point: its stored document, `undefined` when it cannot be read there. */
export type CopySource = (
source: DocumentCopySource,
) => Promise<{ readonly record: DocumentRecord; readonly version: number; readonly value: JsonObject } | undefined>;
/** A Durable commit staged for the store's step. */
export class DurableCommit implements StagedWrite {
/** A Durable commit staged for the store's series: its checks, then itself as its one write. */
export class DurableCommit implements SeriesCommit, StagedWrite {
readonly order = "write";
readonly writes: readonly StagedWrite[];
readonly #owner: string;
readonly #tables: CommitTables;
readonly #context: Context;
@ -81,6 +79,10 @@ export class DurableCommit implements StagedWrite {
readonly #claims: { readonly id: number; readonly type: RecordType }[] = [];
readonly #rows: RowWrite[] = [];
readonly #actions = new Map<DocumentId, DocumentAction>();
/** What `check` found for `apply`, in the same step: the stored types of the claimed ids, the changed documents. */
#checked:
| { readonly claimed: Map<number, RecordType>; readonly existing: Map<DocumentId, EngineRecord> }
| undefined;
/**
* Stages `writes` (rule 8: rows and contents encoded here, before the queue); `raise` makes room above each explicit
@ -97,19 +99,35 @@ export class DurableCommit implements StagedWrite {
this.#tables = tables;
this.#context = context;
for (const write of writes) this.#stage(write, raise);
this.writes = [this];
}
/**
* Resolves the copies before the step (S11): each source read at its point and checked against the copy's record,
* entered as a creation carrying the source's value as its base.
* Durable's checks against the state the series' earlier commits left, writing nothing: the ids, the documents and
* the copies; a throw refuses the commit.
*/
async resolveCopies(read: CopySource): Promise<void> {
async check(step: Step): Promise<void> {
const claimed = await this.#checkIds(step);
const existing = await this.#checkDocuments(step);
await this.#resolveCopies(step);
this.#checked = { claimed, existing };
}
/**
* Resolves the copies (S11): each source read at its point and checked against the copy's record, entered as a
* creation carrying the source's value as its base.
*/
async #resolveCopies(step: Step): Promise<void> {
for (const [id, action] of this.#actions) {
const source = action.copy;
if (source === undefined) continue;
if (this.#actions.has(source.id)) throw new Error(`Document copy ${id} source is changed in the copy batch`);
try {
const stored = await read(source);
const read = await readAt(step, source.id, source.at, this.#owner);
if (read.kind === "current_only") {
throw new StorageRequestError(`Document ${read.id} does not retain historical content`);
}
const stored = read.document;
if (stored === undefined) throw new Error(`Fork source document ${source.id} cannot be read`);
const create = action.create!;
if (
@ -129,10 +147,10 @@ export class DurableCommit implements StagedWrite {
}
}
/** The commit's writes, after its `check` in the same step. */
async apply(step: Step): Promise<readonly StoreWrite[]> {
const recorded: StoreWrite[] = [];
const claimed = await this.#checkIds(step);
const existing = await this.#checkDocuments(step);
const { claimed, existing } = this.#checked!;
for (const write of this.#rows) {
if (claimed.get(write.id) === undefined) recorded.push(await this.#claim(step, write.id, write.type));
claimed.set(write.id, write.type);

56
packages/durable/src/storage/store/storage.ts
View file

@ -8,6 +8,7 @@ import {
type Claimant,
findDocumentEntry,
listDocumentEntries,
type SeriesCommit,
type Store,
StoreOwnerTables,
} from "@earendil-works/chord/next";
@ -15,12 +16,12 @@ import type { SqliteExecutor } from "@earendil-works/chord/sqlite";
import { StorageRequestError } from "../../errors.ts";
import { idFromNumber, seqFromNumber } from "../../ids.ts";
import type {
CommitResult,
ConversationId,
ConversationQuery,
ConversationRecord,
Cursor,
DocumentAddress,
DocumentCopySource,
DocumentId,
DocumentPoint,
DocumentQuery,
@ -84,8 +85,8 @@ function cursorId(cursor: Cursor | undefined): number | undefined {
/**
* Durable's `Storage` on Chord's store (slice 6b D1, D2, §2): its records as declared tables under an owner name, its
* documents in the store's documents engine, its ids from the store's one allocator, its commits as one composite
* write of the store's commit step, adopted in the host's batch. Several may share a store under distinct owner names
* documents in the store's documents engine, its ids from the store's one allocator, each commit call as one series
* of the store's commit queue (one backend transaction, one seq per batch), published in the host's batch. Several may share a store under distinct owner names
* (claims); the app owns the store, and `close` leaves it open.
*/
export class StoreStorage implements Storage {
@ -144,17 +145,43 @@ export class StoreStorage implements Storage {
}
}
commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq> {
return this.#call(async () => {
/**
* Commit `batches` as one series of the store (`ChordStore.commitSeries`): one backend transaction, one seq per
* batch, each batch's checks (at staging, then against the state the earlier ones left) before its writes.
*/
commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
context: Context,
): Promise<CommitResult> {
return this.#call(() => {
const store = this.#store;
const commit = new DurableCommit(writes, this.#owner, this.#tables, (id) => store.ids.raise(id), context);
await commit.resolveCopies((source) => this.#copySource(source, context));
// An empty commit stages nothing: it takes no seq, adopted at `lastSeq`.
const staged = writes.length === 0 ? [] : [commit];
return seqFromNumber(await store.commitInBatch(staged, (seq) => adopt(seqFromNumber(seq)), context));
const raise = (id: number): void => store.ids.raise(id);
const commits: SeriesCommit[] = [];
for (const writes of batches) {
try {
commits.push(new DurableCommit(writes, this.#owner, this.#tables, raise, context));
} catch (error) {
// Staging runs Durable's checks of the batch itself: a refusal in its turn, the batches before it stored.
commits.push({ check: () => Promise.reject(error), writes: [] });
break;
}
}
return store.commitSeries(commits, (index, seq) => receive(index, seqFromNumber(seq)), context);
});
}
/**
* Run a commit's publication in the attached host's batch (`batch.within`: it joins a running batch, opens one, or
* runs next while one sends): Chord sources over Durable's views are marked in one batch per commit. Plainly without
* a host.
*/
publishing(publish: () => void, context: Context): void {
const batch = this.#store.batch;
if (batch === undefined) publish();
else batch.within(publish, context);
}
mintId<I extends Id<string>>(): Promise<I> {
return this.#call(async () => idFromNumber<I>(this.#store.mintId()));
}
@ -525,15 +552,6 @@ export class StoreStorage implements Storage {
return page(values, limit, start.order);
});
}
/** A copy's source read at its point, before the step (S11); a request error is thrown here, outside the read. */
async #copySource(source: DocumentCopySource, context: Context) {
const read = await this.#store.readTransaction(
(executor) => readAt(executor, source.id, source.at, this.#owner),
context,
);
return documentOf(read);
}
}
/** A read that found nothing. */

18
packages/durable/src/testing/commit-writes.ts Normal file
View file

@ -0,0 +1,18 @@
// One batch through `Storage.commit()`, for tests and suites that seed or write a storage directly.
import type { Context } from "@earendil-works/chord";
import type { Seq, Storage, StorageWrite } from "../types.ts";
/** Commits `writes` as one batch and resolves with its seq; rejects with the refusal if the storage refused it. */
export async function commitWrites(storage: Storage, writes: readonly StorageWrite[], context: Context): Promise<Seq> {
let stored: Seq | undefined;
const result = await storage.commit(
[writes],
(_index, seq) => {
stored = seq;
},
context,
);
if (result.stored === 0) throw result.refused;
return stored!;
}

1
packages/durable/src/testing/index.ts
View file

@ -1,4 +1,5 @@
export { createExpectAssertions, type ExpectLike } from "./assertions.ts";
export { commitWrites } from "./commit-writes.ts";
export { createEnvConformance } from "./env-conformance.ts";
export { registerEnvConformance, registerStorageConformance, type StorageConformanceRunner } from "./runner.ts";
export {

58
packages/durable/src/testing/storage-benchmark.ts
View file

@ -13,9 +13,7 @@ import type {
TaskRecord,
} from "../types.ts";
import { ROOT_CONVERSATION_ID } from "../types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
import { commitWrites } from "./commit-writes.ts";
export type StorageBenchmarkScale = {
readonly name: string;
@ -92,7 +90,7 @@ export async function seedStorageBenchmark(
storage: Storage,
scale: StorageBenchmarkScale = TIMING_SCALE,
): Promise<StorageBenchmarkDataset> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, BACKGROUND_CONTEXT);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT);
let firstEntryId: EntryId | undefined;
for (let start = 0; start < scale.entryCount; start += BATCH_SIZE) {
@ -111,7 +109,7 @@ export async function seedStorageBenchmark(
},
});
}
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
await commitWrites(storage, writes, BACKGROUND_CONTEXT);
}
for (let start = 0; start < scale.taskCount; start += BATCH_SIZE) {
@ -119,7 +117,7 @@ export async function seedStorageBenchmark(
for (let index = start; index < Math.min(start + BATCH_SIZE, scale.taskCount); index++) {
writes.push({ type: "task", value: task(await storage.mintId<TaskId<JsonValue>>(), index) });
}
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
await commitWrites(storage, writes, BACKGROUND_CONTEXT);
}
let exactDocumentId: DocumentId | undefined;
@ -134,7 +132,7 @@ export async function seedStorageBenchmark(
content: { kind: "base", version: 1, value: { index, text: "x".repeat(128) } },
});
}
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
await commitWrites(storage, writes, BACKGROUND_CONTEXT);
}
const replayEntries = await Promise.all(
@ -143,7 +141,8 @@ export async function seedStorageBenchmark(
return { tail, id };
}),
);
await storage.commit(
await commitWrites(
storage,
replayEntries.map(({ tail, id }) => ({
type: "document.create" as const,
record: {
@ -156,11 +155,11 @@ export async function seedStorageBenchmark(
},
content: { kind: "base" as const, version: 1, value: { count: 0, text: "x".repeat(64) } },
})),
adopt,
BACKGROUND_CONTEXT,
);
for (let count = 1; count <= REPLAY_TAILS.at(-1)!; count++) {
await storage.commit(
await commitWrites(
storage,
replayEntries
.filter(({ tail }) => count <= tail)
.map(({ id }) => ({
@ -168,7 +167,6 @@ export async function seedStorageBenchmark(
id,
content: { kind: "delta" as const, version: 1, ops: [["s", ["count"], count] as const] },
})),
adopt,
BACKGROUND_CONTEXT,
);
}
@ -181,7 +179,8 @@ export async function seedStorageBenchmark(
history: "rewindable",
fork: "asOf",
} satisfies DocumentCreate;
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -189,12 +188,12 @@ export async function seedStorageBenchmark(
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
BACKGROUND_CONTEXT,
);
let ancientAt: Seq | undefined;
for (let count = 1; count <= HISTORY_SEGMENT_LENGTH; count++) {
ancientAt = await storage.commit(
ancientAt = await commitWrites(
storage,
[
{
type: "document.change",
@ -202,11 +201,11 @@ export async function seedStorageBenchmark(
content: { kind: "delta", version: 1, ops: [["s", ["count"], count]] },
},
],
adopt,
BACKGROUND_CONTEXT,
);
}
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -214,13 +213,13 @@ export async function seedStorageBenchmark(
content: { kind: "base", version: 1, value: { count: HISTORY_SEGMENT_LENGTH } },
},
],
adopt,
BACKGROUND_CONTEXT,
);
if (ancientAt === undefined) throw new Error("Benchmark history seed produced no commits");
let recentAt = ancientAt;
for (let count = HISTORY_SEGMENT_LENGTH + 1; count <= HISTORY_SEGMENT_LENGTH * 2; count++) {
recentAt = await storage.commit(
recentAt = await commitWrites(
storage,
[
{
type: "document.change",
@ -228,7 +227,6 @@ export async function seedStorageBenchmark(
content: { kind: "delta", version: 1, ops: [["s", ["count"], count]] },
},
],
adopt,
BACKGROUND_CONTEXT,
);
}
@ -239,18 +237,19 @@ export async function seedStorageBenchmark(
let deepestConversationId = ROOT_CONVERSATION_ID;
for (let depth = 0; depth < FORK_DEPTH; depth++) {
const conversationId = await storage.mintId<ConversationId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "conversation",
value: { id: conversationId, parent: { conversationId: parentConversationId, at: parentAt } },
},
],
adopt,
BACKGROUND_CONTEXT,
);
const ids = await Promise.all(Array.from({ length: ENTRIES_PER_FORK }, () => storage.mintId<EntryId>()));
await storage.commit(
await commitWrites(
storage,
ids.map((id, index) => ({
type: "entry" as const,
value: {
@ -260,7 +259,6 @@ export async function seedStorageBenchmark(
data: { depth, index },
},
})),
adopt,
BACKGROUND_CONTEXT,
);
parentConversationId = conversationId;
@ -400,8 +398,9 @@ export type StorageWriteBenchmark = {
/** Seed the common state expected by every write benchmark sample. */
export async function seedStorageWriteBenchmark(storage: Storage): Promise<void> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, BACKGROUND_CONTEXT);
await storage.commit(
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT);
await commitWrites(
storage,
await Promise.all(
Array.from({ length: 100 }, async (_, index) => ({
type: "entry" as const,
@ -413,7 +412,6 @@ export async function seedStorageWriteBenchmark(storage: Storage): Promise<void>
},
})),
),
adopt,
BACKGROUND_CONTEXT,
);
}
@ -424,7 +422,8 @@ export const STORAGE_WRITE_BENCHMARKS: readonly StorageWriteBenchmark[] = [
expected: 1,
async run(storage) {
const id = await storage.mintId<EntryId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "entry",
@ -436,7 +435,6 @@ export const STORAGE_WRITE_BENCHMARKS: readonly StorageWriteBenchmark[] = [
},
},
],
adopt,
BACKGROUND_CONTEXT,
);
return 1;
@ -460,7 +458,7 @@ export const STORAGE_WRITE_BENCHMARKS: readonly StorageWriteBenchmark[] = [
}),
),
);
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
await commitWrites(storage, writes, BACKGROUND_CONTEXT);
return writes.length;
},
},
@ -495,7 +493,7 @@ export const STORAGE_WRITE_BENCHMARKS: readonly StorageWriteBenchmark[] = [
content: { kind: "base", version: 1, value: { entryId, taskId } },
},
];
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
await commitWrites(storage, writes, BACKGROUND_CONTEXT);
return writes.length;
},
},

574
packages/durable/src/testing/storage-conformance.ts
View file

File diff suppressed because it is too large Load diff

58
packages/durable/src/types.ts
View file

@ -981,7 +981,8 @@ export type CommitPublication = {
/**
* Transaction surface of one Session commit callback.
* Table reads and creation results are trusted immutable values and may be shared with internal commit state.
* Table reads and creation results are trusted immutable values and may be shared with internal commit state; the
* Session may freeze them (its caches and context ranges keep them), so mutating one can throw.
*/
export interface Tx {
conversation(id: ConversationId): Promise<ConversationRecord | undefined>;
@ -997,6 +998,8 @@ export interface Tx {
scanEntries(query: EntryQuery, limit: number, cursor?: Cursor): Promise<Page<EntryRecord, Cursor>>;
/** Newest visible entry of the conversation that carries a `head`. */
latestHeadMarker(conversationId: ConversationId): Promise<(EntryRecord & { readonly head: EntryId }) | undefined>;
/** ID of the newest entry visible from the conversation; `undefined` when it has none. */
latestEntryId(conversationId: ConversationId): Promise<EntryId | undefined>;
scanTasks(
query: TaskQuery,
limit: number,
@ -1093,10 +1096,11 @@ export type DocumentWatch<T extends JsonObject> = WatchHandle<Readonly<T>>;
/**
* A synchronous observer of committed Durable state, for adapters such as a Chord source (`observeDoc()`,
* `Harness.observeConversation()`, `Harness.observeTaskGraph()`), with the shapes of Chord's watch: `advance` gets a
* `WatchCall`, `end` a `WatchEnd`. `advance` gets every change, unfolded, as memory adopts it: for a stored commit inside
* Storage's `adopt`, for a deferred one before it is stored. A view or the graph gets at most one call per commit, a
* document observer one per change at its address (two for a commit that retires and recreates it); the task graph also
* advances outside any commit and `adopt`, on the Session line, when a registry change or a failed migration changes a
* `WatchCall`, `end` a `WatchEnd`. `advance` gets every change, unfolded, as memory adopts it: for a stored commit at its
* publication once Storage stored it, for a deferred one before it is stored. A view or the graph gets at most one call
* per commit, a document observer one per change at its address (two for a commit that retires and recreates it); the
* task graph also advances outside any commit's publication, on the Session line, when a registry change or a failed
* migration changes a
* `blocked` reason. Durable's failures are final, so every call is a `value` call; a failure ends the observer. Calls
* may come before the attaching promise resolves; their ops apply from `Observation.value`. `advance` must only record
* the call, synchronously and quickly, never await, call the Session, or throw. A throw is reported as
@ -1188,7 +1192,10 @@ export type CommitOptions = { readonly persistence?: CommitPersistence };
export interface Session extends DocumentObserver {
/**
* Run one atomic transaction on the Session mutation line. Inside the callback read only through `tx`: `snapshot()`, `watchDoc()`, `getTask()`, and the other Session and runtime reads may queue on the line the callback holds, and then never resolve.
* `options.persistence` may defer storing it (`CommitPersistence`).
* `options.persistence` may defer storing it (`CommitPersistence`). The callback may run while an earlier commit is
* being stored: at its first read or write of what an unpublished commit wrote, it waits until that commit is stored
* and published; code before that runs early, so act on what it computes only once the commit resolves. Resolves
* once the commit is stored and published, after every earlier commit's outcome.
*/
commit<T>(change: (tx: Tx) => T | Promise<T>, context: Context, options?: CommitOptions): Promise<T>;
/** Seal admission, settle admitted commits, then close storage. */
@ -1243,10 +1250,10 @@ export interface Session extends DocumentObserver {
): Promise<Observation<Readonly<T>> | undefined>;
/**
* Observe stored commits, each with exactly what it stored, except that a document the commit retires is published
* as its retirement only; a commit held in memory is published when it is stored (`CommitPublication`). Storage's
* adoption of the commit queues one dispatch, which runs a microtask later, before every wait the commit resolves
* and every abort it queues, and calls the observers attached by then; a commit adopted while none was attached has
* no dispatch. There is no order relative to watch listeners. `advance` runs synchronously and must not block or
* as its retirement only; a commit held in memory is published when it is stored (`CommitPublication`). Commits are
* published in line order once Storage stored them, a macrotask apart; a publication queues one dispatch, which runs
* a microtask later, before every wait the commit resolves and every abort it queues, and calls the observers
* attached by then; a commit published while none was attached has no dispatch. There is no order relative to watch listeners. `advance` runs synchronously and must not block or
* call Session APIs; a throw is reported as `listener_error`, detaches the observer, and ends it `listener_error`.
* `end` is called once, when the Session closes (`ended`, after its last publication) or fails (`failed`), never
* after `detach()`. Attaches synchronously; the observation has no `value`: commits are events, not a state, and the
@ -1303,6 +1310,9 @@ export interface Session extends DocumentObserver {
): Promise<Readonly<T> | undefined>;
}
/** What `Storage.commit()` stored: the number of durable batches, and why the next one was refused, if one was. */
export type CommitResult = { readonly stored: number; readonly refused?: unknown };
/**
* Atomic persistence boundary for Session records.
*
@ -1318,13 +1328,29 @@ export interface Session extends DocumentObserver {
*/
export interface Storage {
/**
* Atomically persist one batch and return its sequence. Once the batch is durable, call `adopt(seq)` exactly once,
* synchronously and before this storage's next commit starts, then resolve with `seq`; a commit that fails before it
* is durable never calls `adopt`. `adopt` is the caller's in-memory adoption and publication of the batch, so memory
* and the Session's views always show a complete prefix of the log; it runs the Session's own observers, which call
* no Storage method, and never throws. Once resolved, later reads through this storage observe the batch.
* Persist `batches` in order, each as one ordinary atomic commit with its own seq, checked against the state the
* earlier ones left, in as few round trips as the backend can. Every batch has at least one write: a commit without
* writes never reaches Storage, so the Session never sends an empty batch. Stops at the first batch Durable's own checks refuse
* (global IDs, document actions, copy resolution): the batches before it are durable, it and the ones after are not.
* Calls `receive(index, seq)` once per durable batch, in order, synchronously, after all of them are durable and
* before the next commit starts. Resolves with the number of durable batches and the refusal, if any. Rejects for an
* unknown outcome (connection lost, statement or commit failed); `receive` calls made before the rejection still
* stand. `receive` is a durability receipt, not publication: the Session publishes later. Once it resolved, later
* reads through this storage observe the stored batches.
*/
commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq>;
commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
context: Context,
): Promise<CommitResult>;
/**
* Optional. Run `publish`, the Session's publication of one stored or held commit (its cache write-through,
* adoption, and observers), inside this storage's own scope, such as a store's attached host batch, so what the
* publication marks lands together. Call `publish` exactly once, now or soon after; the Session publishes the next
* commit only once it ran. Without it, the Session publishes directly.
*/
publishing?(publish: () => void, context: Context): void;
/**
* Return a fresh branded candidate from the Session-global numeric ID namespace, greater than every ID minted since

3
packages/durable/test/harness-blocked.test.ts
View file

@ -12,6 +12,7 @@ import {
} from "@earendil-works/pi-durable";
import { afterEach, describe, expect, it } from "vitest";
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { context, flush } from "./session-support.ts";
import { completed, deferred, eventually, openTasks } from "./task-support.ts";
@ -104,7 +105,7 @@ describe("recovery at open", () => {
...rest,
state: { status: "completing", outcome: { status: "completed", result: null } },
} as const;
await raw.commit([{ type: "task", value: completing }], () => {}, context);
await commitWrites(raw, [{ type: "task", value: completing }], context);
await raw.close(context);
// That pass's commit fails, as on a platform reset.

24
packages/durable/test/harness-cancellation-barrier.test.ts
View file

@ -1,5 +1,6 @@
import type { Context } from "@earendil-works/chord";
import {
type CommitResult,
createSession,
defineTask,
MemoryStorage,
@ -8,6 +9,7 @@ import {
type TaskId,
} from "@earendil-works/pi-durable";
import { describe, expect, it } from "vitest";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { context, flush } from "./session-support.ts";
import { aborted, deferred, eventually, openTasks } from "./task-support.ts";
@ -26,19 +28,25 @@ const Dormant = defineTask<null, { phase: "work" }, null>({
},
});
/** Holds every commit that abort-marks `target`, while set, until released. */
/** Holds every commit call that abort-marks `target`, while set, until released. */
class HoldMarks extends MemoryStorage {
target: TaskId | undefined;
readonly held = deferred();
readonly release = deferred();
override async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context) {
override async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
commitContext: Context,
): Promise<CommitResult> {
if (
writes.some((write) => write.type === "task" && write.value.id === this.target && write.value.abortRequested)
batches
.flat()
.some((write) => write.type === "task" && write.value.id === this.target && write.value.abortRequested)
) {
this.held.resolve();
await this.release.promise;
}
return super.commit(writes, adopt, commitContext);
return super.commit(batches, receive, commitContext);
}
}
@ -127,7 +135,8 @@ describe("work below a cancelled owner", () => {
policy: "failFast",
} as const;
const failure = { status: "failed", error: { message: "failed" } } as const;
await storage.commit(
await commitWrites(
storage,
[
{ type: "task", value: { ...owner, abortRequested: true, abortReason: "restart", state: waiting } },
{
@ -135,7 +144,6 @@ describe("work below a cancelled owner", () => {
value: { ...failedRecord, state: { status: "terminal", outcome: failure } } as typeof failedRecord,
},
],
() => {},
context,
);
// The child's definition is missing: a restart mark would keep it waiting, a request orphans it.
@ -217,9 +225,9 @@ describe("work below a cancelled owner", () => {
);
const record = (await storage.task(holder, context))!;
const completing = { status: "completing", outcome: { status: "completed", result: null } } as const;
await storage.commit(
await commitWrites(
storage,
[{ type: "task", value: { ...record, state: completing } as typeof record }],
() => {},
context,
);
const { harness } = await openTasks(storage, [Dormant]);

47
packages/durable/test/harness-context-range.test.ts Normal file
View file

@ -0,0 +1,47 @@
import { describe, expect, it } from "vitest";
import { advanceRange, readContext, readContextFrom } from "../src/harness/context.ts";
import { SessionImpl } from "../src/session/session.ts";
import { MemoryStorage } from "../src/storage/memory.ts";
import { context } from "./session-support.ts";
describe("kept context ranges", () => {
it("is not advanced when an entry was committed while its scan ran off the line", async () => {
const entered = Promise.withResolvers<void>();
const release = Promise.withResolvers<void>();
class PausedStorage extends MemoryStorage {
pause = false;
override async scanEntries(...args: Parameters<MemoryStorage["scanEntries"]>) {
const page = await super.scanEntries(...args);
if (this.pause) {
this.pause = false;
entered.resolve();
await release.promise;
}
return page;
}
}
const storage = new PausedStorage();
const session = new SessionImpl(storage);
const conversation = await session.commit(
(tx) => tx.createConversation({ ownership: { kind: "ownerless" } }),
context,
);
await session.commit((tx) => tx.appendEntry(conversation.id, { kind: "first" }), context);
storage.pause = true;
const read = readContextFrom(session, storage, conversation.id, context, undefined, undefined);
await entered.promise;
await session.commit((tx) => tx.appendEntry(conversation.id, { kind: "missed" }), context);
release.resolve();
const { range } = await read;
expect(range?.current).toBe(false);
const last = await session.commit((tx) => tx.appendEntry(conversation.id, { kind: "last" }), context);
// Not current: left as it is, so the next read extends it from Storage instead of skipping "missed".
expect(advanceRange(range!, [last])).toBe(range);
const again = await readContextFrom(session, storage, conversation.id, context, undefined, range);
const fresh = await readContext(session, storage, conversation.id, context);
expect(again.view.entries.map((entry) => entry.kind)).toEqual(["first", "missed", "last"]);
expect(again.view).toEqual(fresh);
expect(again.range?.current).toBe(true);
await session.close(context);
});
});

28
packages/durable/test/harness-context.test.ts
View file

@ -444,18 +444,18 @@ describe("conversation context", () => {
const initial = await read();
expect(initial.view).toEqual(await root.context(taskContext));
// Three new entries: the bounds probe reads one row, the extension the three new ones.
// Three new entries: the kept range took them as they were committed, so the read scans nothing.
await write({ kind: "message", model: [user("new")] });
await write({ kind: "note", data: { text: "display only" } });
await write({ kind: "message", model: [assistant("answer")] });
const extended = await read();
expect(extended.rows).toBe(4);
expect(extended.rows).toBe(0);
expect(extended.view).toEqual(await root.context(taskContext));
// A newer edit of an older entry applies to the extended range.
await write({ kind: "edit", edits: [{ target: first.id, action: "omit" }] });
const edited = await read();
expect(edited.rows).toBe(2);
expect(edited.rows).toBe(0);
expect(edited.view).toEqual(await root.context(taskContext));
expect(edited.view.messages.map(describeMessage)).not.toContain("user:first");
@ -559,9 +559,9 @@ describe("conversation context", () => {
await probe("idle");
clock += 600_000;
await probe("expired");
// Bounds probes read one row each. The child scans its own new entry; the waiting parent's later invocation and
// the probe within the retention period scan nothing new. After it, the probe reads the whole transcript again.
expect(rows).toEqual({ parent: 22, child: 2, after: 1, idle: 1, expired: 23 });
// The Session knows the bounds, and the kept range takes each committed entry, so only a first read scans: the
// parent's, and the probe's after the retention period, which reads the whole transcript again.
expect(rows).toEqual({ parent: 21, child: 0, after: 0, idle: 0, expired: 22 });
await harness.close(context);
});
@ -732,8 +732,8 @@ describe("conversation context", () => {
expect((await read(root)).messages[0]).toBe("user:first");
await append(root.id, { kind: "pi.system", model: [system({ plan: "Plan." })] });
let read2 = await read(root);
// One bounds probe and the added entry: the range was extended, not derived again.
expect(read2.rows).toBe(2);
// Nothing: the kept range took the added entry as it was committed.
expect(read2.rows).toBe(0);
expect(read2.messages.slice(0, 3)).toEqual(["system:plan", "user:first", "assistant:old 0"]);
// A system message the cached range already has keeps a later first `pi.system` entry in place.
@ -743,7 +743,7 @@ describe("conversation context", () => {
expect((await read(fork)).messages).toEqual(["user:first", "assistant:answer", "system:note"]);
await append(fork.id, { kind: "pi.system", model: [system({ note: null })] });
read2 = await read(fork);
expect(read2.rows).toBe(2);
expect(read2.rows).toBe(0);
expect(read2.messages).toEqual(["user:first", "assistant:answer", "system:note", "system:note"]);
// An omitted system message still rules out a later baseline; the extended read keeps that.
@ -761,7 +761,7 @@ describe("conversation context", () => {
expect((await read(omitted)).messages).toEqual(["user:first", "assistant:answer"]);
await append(omitted.id, { kind: "pi.system", model: [system({ legacy: null })] });
read2 = await read(omitted);
expect(read2.rows).toBe(2);
expect(read2.rows).toBe(0);
expect(read2.messages).toEqual(["user:first", "assistant:answer", "system:legacy"]);
// A first `pi.system` entry without a system message leaves no baseline for a later one to take.
@ -771,7 +771,7 @@ describe("conversation context", () => {
expect((await read(empty)).messages).toEqual(["user:first", "assistant:answer"]);
await append(empty.id, { kind: "pi.system", model: [system({ s: "Delta." })] });
read2 = await read(empty);
expect(read2.rows).toBe(2);
expect(read2.rows).toBe(0);
expect(read2.messages).toEqual(["user:first", "assistant:answer", "system:s"]);
await runtime.commit(() => DONE, taskContext);
},
@ -907,7 +907,7 @@ describe("conversation context", () => {
await late;
probeStart.resolve();
expect((await harness.waitForTask(probeId, context)).state.outcome).toEqual(DONE.outcome);
expect(rows).toEqual({ probe: 22 });
expect(rows).toEqual({ probe: 21 });
await harness.close(context);
});
@ -934,7 +934,7 @@ describe("conversation context", () => {
vi.advanceTimersByTime(1_000);
expect(vi.getTimerCount()).toBe(idle);
await probe("second");
expect(rows).toEqual({ first: 22, second: 22 });
expect(rows).toEqual({ first: 21, second: 21 });
await harness.close(context);
});
@ -954,7 +954,7 @@ describe("conversation context", () => {
expect((await harness.waitForTask(id, context)).state.outcome).toEqual(DONE.outcome);
await root.waitForIdle(context);
}
expect(rows).toEqual({ first: 22, second: 22 });
expect(rows).toEqual({ first: 21, second: 21 });
await harness.close(context);
});
});

3
packages/durable/test/harness-deferred-progress.test.ts
View file

@ -161,7 +161,8 @@ describe("generation partials", () => {
const submission = await opened.root.submit({ type: "input", content: "hi" }, context);
await waitFor(() => (opened.memory.at(-1)?.length ?? 0) > 40);
await crash(opened.harness);
expect(await storedLive(storage, opened.root.id)).toMatchObject({ generation: { attempt: 1 } });
// The attempt is presentation, deferred like the partials: none of it was stored.
expect((await storedLive(storage, opened.root.id))?.generation).toBeUndefined();
expect(storedText(await storedLive(storage, opened.root.id))).toBeUndefined();
setup.faux.setResponses([fauxAssistantMessage("answer")]);

2
packages/durable/test/harness-generation.test.ts
View file

@ -119,6 +119,8 @@ describe("generation", () => {
it("stores partials as deltas and a complete base once nothing is in flight", async () => {
const setup = chatSetup({ tokensPerSecond: 200, tokenSize: { min: 1, max: 1 } });
setup.faux.setResponses([fauxAssistantMessage("w".repeat(200))]);
// Store deferred partials well within the stream, so some reach Storage before the answer.
setup.settings.progress = { persistWithinMs: 100 };
const storage = new ControlledStorage();
const { harness, root } = await openChat(storage, setup);
const record = await storage.findDocument(

796
packages/durable/test/harness-group-commit.test.ts Normal file
View file

@ -0,0 +1,796 @@
// Group commit and the scheduler (docs/group-commit.md §6): decisions read the scheduler's memory in regions that
// wait for the unpublished commits whose keys they read, memberships written at admission, outcomes, and close.
import type { JsonValue } from "@earendil-works/chord";
import {
type ConversationId,
createRegistry,
defineExtension,
defineTask,
type Harness,
type TaskId,
type TaskRecord,
type TaskRuntime,
} from "@earendil-works/pi-durable";
import { afterEach, describe, expect, it } from "vitest";
import { SCHEDULER_INDEX_SIZES } from "../src/harness/harness.ts";
import { ownedKey, ROOTS_KEY } from "../src/session/keys.ts";
import type { SessionImpl } from "../src/session/session.ts";
import type { Transaction } from "../src/session/transaction.ts";
import { ControlledStorage, context, flush, followCommits } from "./session-support.ts";
import { aborted, type Deferred, deferred, eventually, openTasks, settled } from "./task-support.ts";
const gates = new Map<string, Deferred<void>>();
function gate(name: string): Deferred<void> {
let found = gates.get(name);
if (found === undefined) {
found = deferred<void>();
gates.set(name, found);
}
return found;
}
/** How often each named phase ran. */
const runs = new Map<string, number>();
function ran(name: string): number {
return runs.get(name) ?? 0;
}
function count(name: string): void {
runs.set(name, ran(name) + 1);
}
afterEach(() => {
gates.clear();
runs.clear();
});
const COMPLETED = { status: "terminal", outcome: { status: "completed", result: null } } as const;
function completedWith<R>(result: R) {
return { status: "terminal", outcome: { status: "completed", result } } as const;
}
/** Holds until its named gate opens, then completes; its abort handler ends it `aborted`. */
const Hold = defineTask<{ name: string }, { phase: "hold" }, null>({
name: "gc.hold",
version: 1,
initial: () => ({ phase: "hold" }),
phases: {
hold: async (task, runtime, ctx) => {
count(task.input.name);
await Promise.race([gate(task.input.name).promise, aborted(runtime.signal)]);
await runtime.commit(() => COMPLETED, ctx);
},
},
abort: async (_task, runtime, ctx) =>
runtime.commit(() => ({ status: "terminal", outcome: { status: "aborted" } }), ctx),
});
/** Commits its next phase, returns once `<name>.return` opens; the next phase counts its run and completes. */
const TwoPhases = defineTask<{ name: string }, { phase: "one" } | { phase: "two" }, null>({
name: "gc.two-phases",
version: 1,
initial: () => ({ phase: "one" }),
phases: {
one: async (task, runtime, ctx) => {
await runtime.commit(() => ({ status: "running", checkpoint: { phase: "two" } }), ctx);
await gate(`${task.input.name}.return`).promise;
},
two: async (task, runtime, ctx) => {
count(`${task.input.name}.two`);
await runtime.commit(() => COMPLETED, ctx);
},
},
abort: async (_task, runtime, ctx) =>
runtime.commit(() => ({ status: "terminal", outcome: { status: "aborted" } }), ctx),
});
/** Once its gate opens, waits on the task in its input; records why a refused wait was refused, then completes. */
const Waits = defineTask<{ name: string; on?: TaskId }, { phase: "wait" } | { phase: "done" }, null>({
name: "gc.waits",
version: 1,
initial: () => ({ phase: "wait" }),
phases: {
wait: async (task, runtime, ctx) => {
await gate(task.input.name).promise;
const on = await runtime.memo<number>("on", ctx);
try {
await runtime.commit(
() => ({ status: "waiting", checkpoint: { phase: "done" }, on: [on as TaskId], policy: "allSettled" }),
ctx,
);
} catch (error) {
refusals.set(task.input.name, error);
await runtime.commit(() => COMPLETED, ctx);
}
},
done: async (_task, runtime, ctx) => runtime.commit(() => COMPLETED, ctx),
},
abort: async (_task, runtime, ctx) =>
runtime.commit(() => ({ status: "terminal", outcome: { status: "aborted" } }), ctx),
});
const refusals = new Map<string, unknown>();
async function status(harness: Harness, id: TaskId): Promise<string | undefined> {
return (await harness.getTask(id, context))?.state.status;
}
/** Stage a task record as is, the way only the scheduler and recovery write them. */
function setTask(tx: unknown, record: TaskRecord<JsonValue, JsonValue, JsonValue>): void {
(tx as Transaction).setTask(record);
}
describe("group commit: scheduler regions", () => {
it("holds a live owner whose end races a creation below its ended tasks: live P, ended B and A, empty C", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Hold]);
const root = await harness.root(context);
harness.resume();
const tree = await root.commit(async (tx) => {
const p = await tx.createTask(Hold, { name: "p" }, { ownership: { kind: "conversation" } });
const b = await tx.createTask(Hold, { name: "b" }, { ownership: { kind: "task", taskId: p } });
const a = await tx.createTask(Hold, { name: "a" }, { ownership: { kind: "task", taskId: b } });
const c = await tx.createConversation({ ownership: { kind: "task", taskId: a } });
return { p, b, a, c: c.id };
}, context);
gate("a").resolve();
await harness.waitForTask(tree.a, context);
gate("b").resolve();
await harness.waitForTask(tree.b, context);
// The creation of D in C is admitted and in flight when P commits its end.
const held = storage.holdCommits();
const c = (await harness.conversation(tree.c, context))!;
const creating = c.commit(
(tx) => tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } }),
context,
);
await held.entered;
// P's decision reads what the creation writes, so it waits for its publication, holding the line.
gate("p").resolve();
await flush();
held.release();
const d = await creating;
await eventually(async () => (await status(harness, tree.p)) === "completing");
gate("d").resolve();
await harness.waitForTask(d, context);
expect((await harness.waitForTask(tree.p, context)).state.outcome.status).toBe("completed");
await harness.close(context);
});
it("holds an owner whose end follows a creation below its child's pending end: live P, ending B, conversation C", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Hold]);
const root = await harness.root(context);
harness.resume();
const tree = await root.commit(async (tx) => {
const p = await tx.createTask(Hold, { name: "p" }, { ownership: { kind: "conversation" } });
const b = await tx.createTask(Hold, { name: "b" }, { ownership: { kind: "task", taskId: p } });
const c = await tx.createConversation({ ownership: { kind: "task", taskId: b } });
return { p, b, c: c.id };
}, context);
const c = (await harness.conversation(tree.c, context))!;
await eventually(async () => (await status(harness, tree.b)) === "running");
// A call in flight, so what follows is admitted and stays unpublished, in this order.
const held = storage.holdCommits();
const blocker = root.commit((tx) => tx.appendEntry(root.id, { kind: "note" }), context);
await held.entered;
// B ends; then D is created in C while B still shows live; then P ends.
gate("b").resolve();
await flush();
const creating = c.commit(
(tx) => tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } }),
context,
);
await flush();
gate("p").resolve();
await flush();
held.release();
await blocker;
const d = await creating;
// In line order D's creation precedes P's decision: P holds while D is live.
await eventually(async () => (await status(harness, tree.p)) === "completing");
expect(await status(harness, d)).toBe("running");
gate("d").resolve();
expect((await harness.waitForTask(tree.p, context)).state.outcome.status).toBe("completed");
await harness.close(context);
});
it("ends a child before its next phase when its owner's abort mark is unpublished at its step", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Hold, TwoPhases]);
const root = await harness.root(context);
harness.resume();
const { owner, child } = await root.commit(async (tx) => {
const owner = await tx.createTask(Hold, { name: "o" }, { ownership: { kind: "conversation" } });
const child = await tx.createTask(TwoPhases, { name: "t" }, { ownership: { kind: "task", taskId: owner } });
return { owner, child };
}, context);
await eventually(async () => {
const record = await harness.getTask(child, context);
return record?.state.status === "running" && (record.state.checkpoint as { phase: string }).phase === "two";
});
const held = storage.holdCommits();
const marking = harness.abortTask(owner, context);
await held.entered;
// The child's phase returns: its step reads the intent set, which the unpublished mark changes.
gate("t.return").resolve();
await flush();
expect(ran("t.two")).toBe(0);
held.release();
expect(await marking).toBe("marked");
expect((await harness.waitForTask(child, context)).state.outcome.status).toBe("aborted");
expect((await harness.waitForTask(owner, context)).state.outcome.status).toBe("aborted");
expect(ran("t.two")).toBe(0);
await harness.close(context);
});
it("refuses the second of two opposite waits admitted behind each other", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Waits]);
const root = await harness.root(context);
const { x, y } = await root.commit(async (tx) => {
const x = await tx.createTask(Waits, { name: "x" }, { ownership: { kind: "conversation" } });
const y = await tx.createTask(Waits, { name: "y" }, { ownership: { kind: "conversation" } });
return { x, y };
}, context);
// Each waits on the other, named in a memo.
await root.commit(async (tx) => {
// Every table read before the first write.
const records = [(await tx.task(x))!, (await tx.task(y))!];
setTask(tx, { ...records[0]!, memos: { on: y } } as TaskRecord<JsonValue, JsonValue, JsonValue>);
setTask(tx, { ...records[1]!, memos: { on: x } } as TaskRecord<JsonValue, JsonValue, JsonValue>);
}, context);
harness.resume();
await eventually(
async () => (await status(harness, x)) === "running" && (await status(harness, y)) === "running",
);
const held = storage.holdCommits();
gate("x").resolve();
await held.entered;
gate("y").resolve();
await flush();
held.release();
await harness.waitForTask(y, context);
expect(String(refusals.get("y"))).toContain("wait cycle");
expect(refusals.has("x")).toBe(false);
expect((await harness.waitForTask(x, context)).state.outcome.status).toBe("completed");
await harness.close(context);
});
it("starts no invocation on a reservation before the reservation is published", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Hold]);
const root = await harness.root(context);
const id = await root.commit(
(tx) => tx.createTask(Hold, { name: "r" }, { ownership: { kind: "conversation" } }),
context,
);
// The reservation's write is in flight: a scheduler commit with writes is never delivered before it is published.
const held = storage.holdCommits();
harness.resume();
await held.entered;
await flush();
expect(ran("r")).toBe(0);
held.release();
await eventually(() => ran("r") === 1);
expect(await status(harness, id)).toBe("running");
gate("r").resolve();
await harness.waitForTask(id, context);
await harness.close(context);
});
it("runs a migration once although its reservation waits for an unpublished write of the task", async () => {
const storage = new ControlledStorage();
let migrations = 0;
const V2 = defineTask<{ name: string }, { phase: "hold" }, null>({
name: "gc.migrated",
version: 2,
initial: () => ({ phase: "hold" }),
migrate: (input, checkpoint) => {
migrations++;
return { input: input as { name: string }, checkpoint: checkpoint as { phase: "hold" } };
},
phases: { hold: async (_task, runtime, ctx) => runtime.commit(() => COMPLETED, ctx) },
abort: async () => {},
});
const registry = createRegistry();
registry.install(defineExtension({ name: "v2", tasks: [V2] }));
const { harness } = await openTasks(storage, [], { registry });
const root = await harness.root(context);
const id = await root.commit(
(tx) => tx.createTask(V2, { name: "m" }, { ownership: { kind: "conversation" } }),
context,
);
// Stored at version 1, and written again by a commit that is in flight when the reservation reads it.
const held = storage.holdCommits();
const writing = root.commit(async (tx) => {
const record = (await tx.task(id))!;
setTask(tx, { ...record, version: 1, memos: { touched: true } } as TaskRecord<
JsonValue,
JsonValue,
JsonValue
>);
}, context);
await held.entered;
harness.resume();
await flush();
expect(migrations).toBe(0);
held.release();
await writing;
expect((await harness.waitForTask(id, context)).state.outcome.status).toBe("completed");
expect(migrations).toBe(1);
await harness.close(context);
});
it("reports a failed migration once although its reservation waits for an unpublished write of the task", async () => {
const storage = new ControlledStorage();
let migrations = 0;
const V2 = defineTask<{ name: string }, { phase: "hold" }, null>({
name: "gc.unmigratable",
version: 2,
initial: () => ({ phase: "hold" }),
migrate: () => {
migrations++;
throw new Error("cannot migrate");
},
phases: { hold: async (_task, runtime, ctx) => runtime.commit(() => COMPLETED, ctx) },
abort: async () => {},
});
const registry = createRegistry();
registry.install(defineExtension({ name: "v2", tasks: [V2] }));
const { harness, infos } = await openTasks(storage, [], { registry });
const root = await harness.root(context);
const id = await root.commit(
(tx) => tx.createTask(V2, { name: "m" }, { ownership: { kind: "conversation" } }),
context,
);
const held = storage.holdCommits();
const writing = root.commit(async (tx) => {
const record = (await tx.task(id))!;
setTask(tx, { ...record, version: 1 } as TaskRecord<JsonValue, JsonValue, JsonValue>);
}, context);
await held.entered;
harness.resume();
await flush();
held.release();
await writing;
await eventually(() => migrations > 0);
await flush();
expect(migrations).toBe(1);
expect(infos.filter((info) => info.kind === "migration_failed")).toHaveLength(1);
await harness.close(context);
});
it("makes a reader of the roots wait for a creation under a root a pending end would unlist", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Hold]);
const root = await harness.root(context);
harness.resume();
// Root R is listed for its last live task, T.
const last = await root.commit(
(tx) => tx.createTask(Hold, { name: "t" }, { ownership: { kind: "conversation" } }),
context,
);
await eventually(async () => (await status(harness, last)) === "running");
const published = new Set<TaskId>();
followCommits(harness, (publication) => {
if (publication.kind !== "stored") return;
for (const change of publication.changes) if (change.type === "task") published.add(change.value.id);
});
// T's end is in flight; a creation under R is admitted after it, while R still shows listed.
const held = storage.holdCommits();
gate("t").resolve();
await held.entered;
let created: TaskId | undefined;
const creating = root.commit(async (tx) => {
created = await tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } });
return created;
}, context);
await flush();
// A reader of the roots on the line (global idle's key) waits until the creation is published too.
const session = harness as unknown as SessionImpl;
const reading = session.readOnLine(async () => {
const gate = session.touch(ROOTS_KEY);
if (gate !== undefined) await gate;
return created !== undefined && published.has(created);
});
held.release();
await creating;
expect(await reading).toBe(true);
gate("d").resolve();
await harness.waitForIdle(context);
await harness.close(context);
});
it("walks a creation's memberships past an ancestor whose pending end is unpublished", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Hold]);
const root = await harness.root(context);
harness.resume();
// Live P, in the root conversation, owns the empty conversation C.
const { p, c } = await root.commit(async (tx) => {
const p = await tx.createTask(Hold, { name: "p" }, { ownership: { kind: "conversation" } });
const c = await tx.createConversation({ ownership: { kind: "task", taskId: p } });
return { p, c: c.id };
}, context);
await eventually(async () => (await status(harness, p)) === "running");
const published = new Set<TaskId>();
followCommits(harness, (publication) => {
if (publication.kind !== "stored") return;
for (const change of publication.changes) if (change.type === "task") published.add(change.value.id);
});
// P's end is in flight; a creation in C is admitted after it. Its chain must not stop at P, which shows live.
const conversation = (await harness.conversation(c, context))!;
const held = storage.holdCommits();
gate("p").resolve();
await held.entered;
let created: TaskId | undefined;
const creating = conversation.commit(async (tx) => {
created = await tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } });
return created;
}, context);
await flush();
const session = harness as unknown as SessionImpl;
const reading = session.readOnLine(async () => {
const gate = session.touch(ROOTS_KEY);
if (gate !== undefined) await gate;
return created !== undefined && published.has(created);
});
held.release();
await creating;
expect(await reading).toBe(true);
gate("d").resolve();
await harness.waitForIdle(context);
await harness.close(context);
});
it("reserves the live work of the owner above an ancestor whose end is pending, for a creation below it", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Hold]);
const root = await harness.root(context);
harness.resume();
// Live G owns P, which owns the empty conversation C.
const { g, p, c } = await root.commit(async (tx) => {
const g = await tx.createTask(Hold, { name: "g" }, { ownership: { kind: "conversation" } });
const p = await tx.createTask(Hold, { name: "p" }, { ownership: { kind: "task", taskId: g } });
const c = await tx.createConversation({ ownership: { kind: "task", taskId: p } });
return { g, p, c: c.id };
}, context);
await eventually(async () => (await status(harness, p)) === "running");
const published = new Set<TaskId>();
followCommits(harness, (publication) => {
if (publication.kind !== "stored") return;
for (const change of publication.changes) if (change.type === "task") published.add(change.value.id);
});
// P's end is in flight; a creation in C is admitted after it, while P still shows live.
const conversation = (await harness.conversation(c, context))!;
const held = storage.holdCommits();
gate("p").resolve();
await held.entered;
let created: TaskId | undefined;
const creating = conversation.commit(async (tx) => {
created = await tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } });
return created;
}, context);
await flush();
// A reader of G's live work (as G's end decision is) waits until the creation is published too.
const session = harness as unknown as SessionImpl;
const reading = session.readOnLine(async () => {
const gate = session.touch(ownedKey(g));
if (gate !== undefined) await gate;
return created !== undefined && published.has(created);
});
held.release();
await creating;
expect(await reading).toBe(true);
gate("d").resolve();
gate("g").resolve();
await harness.waitForTask(g, context);
await harness.close(context);
});
it("keeps the nodes a chain load installed for an unpublished creation through the sweep", async () => {
const storage = new ControlledStorage();
let opened = await openTasks(storage, [Hold]);
const root = await opened.harness.root(context);
opened.harness.resume();
const tree = await root.commit(async (tx) => {
const b = await tx.createTask(Hold, { name: "b" }, { ownership: { kind: "conversation" } });
const c = await tx.createConversation({ ownership: { kind: "task", taskId: b } });
return { b, c: c.id };
}, context);
gate("b").resolve();
await opened.harness.waitForTask(tree.b, context);
await opened.harness.close(context);
// After reopen, B and C's edge are known only once a chain load reads them.
opened = await openTasks(storage.reopen(), [Hold]);
const harness = opened.harness;
const sizes = () =>
(harness as unknown as Record<symbol, { settled: number; unloaded: number; pins: number }>)[
SCHEDULER_INDEX_SIZES
]!;
expect(sizes().settled).toBe(0);
const c = (await harness.conversation(tree.c, context))!;
const held = storage.holdCommits();
const creating = c.commit(
(tx) => tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } }),
context,
);
await held.entered;
// The admission loaded B; the sweep that load scheduled has run on the line since, and kept it.
await harness.waitForIdle(context);
await flush();
expect(sizes().settled).toBe(1);
expect(sizes().pins).toBeGreaterThan(0);
held.release();
await creating;
// Published below a known chain: nothing is left to load, and no pin is left.
expect(sizes().unloaded).toBe(0);
expect(sizes().pins).toBe(0);
expect(await settled(harness.waitForIdle(context))).toBe(false);
harness.resume();
gate("d").resolve();
await harness.waitForIdle(context);
await harness.close(context);
});
});
describe("group commit: records a decision copies", () => {
type Probe = { readonly name: string };
/** Runtimes of running probes, by name, so a test commits for them. */
const runtimes = new Map<string, TaskRuntime<Probe, { phase: "hold"; n: number }, null, object>>();
const untilAborted = (signal: AbortSignal) =>
new Promise<void>((resolve) => signal.addEventListener("abort", () => resolve(), { once: true }));
/** Holds until aborted, its runtime exposed; fails once its gate opens. Its abort handler holds too. */
const Probe = defineTask<Probe, { phase: "hold"; n: number }, null>({
name: "gc.probe",
version: 1,
initial: () => ({ phase: "hold", n: 0 }),
phases: {
hold: async (task, runtime, ctx) => {
runtimes.set(task.input.name, runtime);
const failing = gate(task.input.name).promise.then(() => "fail" as const);
if ((await Promise.race([failing, untilAborted(runtime.signal)])) !== "fail") return;
await runtime.commit(
() => ({ status: "terminal", outcome: { status: "failed", error: { message: "gate" } } }),
ctx,
);
},
},
abort: async (_task, runtime) => untilAborted(runtime.signal),
});
/** Creates probes `x` and `y` and waits on them `failFast`. */
const Waiter = defineTask<null, { phase: "spawn" } | { phase: "done" }, null>({
name: "gc.fail-fast",
version: 1,
initial: () => ({ phase: "spawn" }),
phases: {
spawn: async (task, runtime, ctx) =>
runtime.commit(async (tx) => {
const on: TaskId[] = [];
for (const name of ["x", "y"]) {
on.push(await tx.createTask(Probe, { name }, { ownership: { kind: "task", taskId: task.id } }));
}
return { status: "waiting", checkpoint: { phase: "done" }, on, policy: "failFast" };
}, ctx),
done: async (_task, runtime, ctx) => runtime.commit(() => COMPLETED, ctx),
},
abort: async (_task, runtime) => untilAborted(runtime.signal),
});
afterEach(() => {
runtimes.clear();
});
/** Advance probe `name` to `n = 1` while the mark-writing commit is in flight, and run a reconcile pass meanwhile. */
async function raceMark(storage: ControlledStorage, harness: Harness, name: string, mark: () => Promise<unknown>) {
const heldMark = storage.holdCommits();
const marking = mark();
await heldMark.entered;
let prepared = false;
const advancing = runtimes.get(name)!.commit(() => {
prepared = true;
return { status: "running", checkpoint: { phase: "hold", n: 1 } };
}, context);
await eventually(() => prepared);
await flush();
// Global idle schedules a reconcile pass, which reaches the line while both are unpublished.
harness.waitForIdle(context).catch(() => {});
await flush();
const heldAdvance = storage.holdCommits();
heldMark.release();
await heldAdvance.entered;
await flush();
heldAdvance.release();
await Promise.all([marking, advancing]);
await flush();
}
it("marks a child in a cascade from its current record, not one its pending commit replaces", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Probe]);
const root = await harness.root(context);
const { parent, child } = await root.commit(async (tx) => {
const parent = await tx.createTask(Probe, { name: "p" }, { ownership: { kind: "conversation" } });
const child = await tx.createTask(Probe, { name: "c" }, { ownership: { kind: "task", taskId: parent } });
return { parent, child };
}, context);
harness.resume();
await eventually(() => runtimes.has("c"));
await flush();
await raceMark(storage, harness, "c", () => harness.abortTask(parent, context));
await eventually(async () => (await harness.getTask(child, context))?.abortRequested === true);
expect((await harness.getTask(child, context))?.state).toMatchObject({ checkpoint: { n: 1 } });
await harness.close(context);
});
it("marks the other tasks of a failFast wait from their current records", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Probe, Waiter]);
const root = await harness.root(context);
await root.commit((tx) => tx.createTask(Waiter, null, { ownership: { kind: "conversation" } }), context);
harness.resume();
await eventually(() => runtimes.has("x") && runtimes.has("y"));
await flush();
const y = runtimes.get("y")!.taskId;
// X fails while its commit is in flight; the pass that marks Y runs while Y's own commit is pending.
await raceMark(storage, harness, "y", async () => {
gate("x").resolve();
});
await eventually(async () => (await harness.getTask(y, context))?.abortRequested === true);
expect((await harness.getTask(y, context))?.state).toMatchObject({ checkpoint: { n: 1 } });
await harness.close(context);
});
});
describe("group commit: reads on the line", () => {
it("reads no task record from Storage for more live and awaited tasks than the record cache's count", async () => {
class Counting extends ControlledStorage {
taskReads = 0;
override task(...args: Parameters<ControlledStorage["task"]>) {
this.taskReads++;
return super.task(...args);
}
}
const Child = defineTask<null, { phase: "end" }, number>({
name: "gc.child",
version: 1,
initial: () => ({ phase: "end" }),
phases: { end: async (task, runtime, ctx) => runtime.commit(() => completedWith(task.id), ctx) },
abort: async () => {},
});
const Parent = defineTask<null, { phase: "spawn" } | { phase: "read"; on: TaskId[] }, number>({
name: "gc.parent",
version: 1,
initial: () => ({ phase: "spawn" }),
phases: {
spawn: async (task, runtime, ctx) =>
runtime.commit(async (tx) => {
const on: TaskId[] = [];
for (let n = 0; n < 300; n++) {
on.push(await tx.createTask(Child, null, { ownership: { kind: "task", taskId: task.id } }));
}
return { status: "waiting", checkpoint: { phase: "read", on }, on, policy: "allSettled" };
}, ctx),
read: async (task, runtime, ctx) => {
const outcomes = await runtime.outcomes(task.state.checkpoint.on, ctx);
const sum = outcomes.reduce((total, outcome) => total + (outcome.status === "completed" ? 1 : 0), 0);
await runtime.commit(() => completedWith(sum), ctx);
},
},
abort: async () => {},
});
const storage = new Counting();
const { harness } = await openTasks(storage, [Parent, Child]);
const root = await harness.root(context);
const id = await root.commit(
(tx) => tx.createTask(Parent, null, { ownership: { kind: "conversation" } }),
context,
);
storage.taskReads = 0;
harness.resume();
expect((await harness.waitForTask(id, context)).state.outcome).toEqual({ status: "completed", result: 300 });
// Each change of a live task, and the parent's read of the outcomes it waited on, finds the record in memory.
expect(storage.taskReads).toBe(0);
await harness.close(context);
});
});
describe("group commit: outcomes and publication through the Harness", () => {
it("finds the submission an unpublished commit admitted with the same request ID", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, []);
const root = await harness.root(context);
const draft = { type: "write", entry: { kind: "note", data: {} }, requestId: "r1" } as const;
const held = storage.holdCommits();
const first = root.submit(draft, context);
await held.entered;
const second = root.submit(draft, context);
await flush();
held.release();
expect((await second).id).toBe((await first).id);
await harness.close(context);
});
it("runs a wait continuation of one commit before the next commit of the same call is published", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, [Hold]);
const root = await harness.root(context);
const other = await harness.createConversation({ ownership: { kind: "ownerless" } }, context);
const one = await root.commit(
(tx) => tx.createTask(Hold, { name: "one" }, { ownership: { kind: "conversation" } }),
context,
);
const two = await other.commit(
(tx) => tx.createTask(Hold, { name: "two" }, { ownership: { kind: "conversation" } }),
context,
);
harness.resume();
await eventually(
async () => (await status(harness, one)) === "running" && (await status(harness, two)) === "running",
);
const ended = new Set<TaskId>();
followCommits(harness, (publication) => {
if (publication.kind !== "stored") return;
for (const change of publication.changes) {
if (change.type === "task" && change.value.state.status === "terminal") ended.add(change.value.id);
}
});
// Both end in one call, behind a commit in flight.
const held = storage.holdCommits();
const blocker = root.commit((tx) => tx.appendEntry(root.id, { kind: "note" }), context);
await held.entered;
let seen: boolean | undefined;
const waited = harness.waitForTask(one, context).then(() => {
seen = ended.has(two);
});
gate("one").resolve();
await flush();
gate("two").resolve();
await flush();
held.release();
await blocker;
await waited;
await harness.waitForTask(two, context);
expect(seen).toBe(false);
await harness.close(context);
});
it("signals an invocation when it closes during a call in flight, and stores that call before Storage closes", async () => {
const storage = new ControlledStorage();
const signalled = deferred<void>();
const Listens = defineTask<null, { phase: "wait" }, null>({
name: "gc.listens",
version: 1,
initial: () => ({ phase: "wait" }),
phases: {
wait: async (_task, runtime) => {
await aborted(runtime.signal).catch(() => signalled.resolve());
},
},
abort: async () => {},
});
const { harness } = await openTasks(storage, [Listens]);
const root = await harness.root(context);
const id = await root.commit(
(tx) => tx.createTask(Listens, null, { ownership: { kind: "conversation" } }),
context,
);
harness.resume();
await eventually(async () => (await status(harness, id)) === "running");
const held = storage.holdCommits();
const entry = root.commit((tx) => tx.appendEntry(root.id, { kind: "note" }), context);
await held.entered;
const closing = harness.close(context);
await signalled.promise;
expect(await settled(closing)).toBe(false);
held.release();
await entry;
await closing;
const reopened = await openTasks(storage.reopen(), [Listens]);
const conversation = (await reopened.harness.conversation(root.id as ConversationId, context))!;
expect((await conversation.entries({}, 10, undefined, context)).items.map((item) => item.kind)).toContain("note");
// Close wrote the task it stopped back as pending.
expect(await status(reopened.harness, id)).toBe("pending");
await reopened.harness.close(context);
});
});

31
packages/durable/test/harness-inbox.test.ts
View file

@ -14,7 +14,6 @@ import {
} from "@earendil-works/pi-ai";
import {
type Conversation,
defineDoc,
defineTool,
type EntryRecord,
GenerationTask,
@ -191,22 +190,20 @@ describe("inbox", () => {
await first.reached;
const f1 = await root.submit({ type: "input", content: "f1" }, context);
const f2 = await root.submit({ type: "input", content: "f2" }, context);
// Occupy the line, let the answer queue its boundary commit behind it, then change the mode.
const held = storage.holdCommits();
const Marker = defineDoc<{ n: number }>({
checkpointWhen: () => false,
kind: "test.marker",
version: 1,
scope: "session",
initial: () => ({ n: 0 }),
});
const occupying = root.commit(async (tx) => void (await tx.doc(Marker)).n++, context);
await held.entered;
// Occupy the line with a commit whose callback waits, let the answer queue its boundary commit behind it, then
// change the mode.
const entered = deferred();
const release = deferred();
const occupying = root.commit(async () => {
entered.resolve();
await release.promise;
}, context);
await entered.promise;
first.release();
await yielded.promise;
await flush();
setup.settings.followUpMode = "all";
held.release();
release.resolve();
await occupying;
const settled = await f2.wait(context);
expect(await status(f1)).toEqual({ ...settled, id: f1.id, entry: expect.any(Number) });
@ -898,17 +895,17 @@ describe("inbox", () => {
let inboxId: number | undefined;
class RecordingStorage extends MemoryStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
adopt: Parameters<MemoryStorage["commit"]>[1],
batches: Parameters<MemoryStorage["commit"]>[0],
receive: Parameters<MemoryStorage["commit"]>[1],
callContext: Parameters<MemoryStorage["commit"]>[2],
) {
for (const write of writes) {
for (const write of batches.flat()) {
if (write.type !== "document.change" || write.id !== inboxId) continue;
const content = write.content;
const empty = content.kind === "base" && (content.value as InboxState).items.length === 0;
written.push({ kind: content.kind, empty });
}
return super.commit(writes, adopt, callContext);
return super.commit(batches, receive, callContext);
}
}
const setup = chatSetup();

5
packages/durable/test/harness-lifecycle.test.ts
View file

@ -20,6 +20,7 @@ import {
import { describe, expect, it } from "vitest";
import type { SessionImpl } from "../src/session/session.ts";
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { chatSetup, openChat } from "./chat-support.ts";
import { addHooks, addTask, addTool } from "./harness-support.ts";
import { ControlledStorage, context, flush } from "./session-support.ts";
@ -73,12 +74,12 @@ async function seedRunningTask(storage: Storage): Promise<void> {
const id = await storage.mintId<TaskId<JsonValue>>();
const state = { status: "running", checkpoint: { phase: "run" } } as const;
const task = { id, conversationId, kind: "test.seeded", version: 1, input: null, background: false, state };
await storage.commit(
await commitWrites(
storage,
[
{ type: "conversation", value: { id: conversationId } },
{ type: "task", value: { ...task, abortRequested: false } },
],
() => {},
context,
);
}

61
packages/durable/test/harness-live-deltas.test.ts
View file

@ -138,36 +138,32 @@ describe("pi.live deltas", () => {
{ taskId: firstGeneration, inputs: [expect.any(Number)], turns: 0, first: expect.any(Number) },
],
],
// request
[["s", ["generation"], { attempt: 1 }]],
// the generation starts its tool round and keeps the run, counting the round as one turn (#11043)
expect.arrayContaining([
["d", ["generation"]],
// the generation starts its tool round and keeps the run, counting the round as one turn (#11043); the request's
// attempt is deferred, so it is stored here, its held operation first
[
["s", ["generation"], { attempt: 1 }],
["s", ["tools"], [{ callId: "c1", name: "noop", taskId: tool, status: "pending" }]],
["d", ["generation"]],
["s", ["run", "turns"], 1],
]),
// intent
[["s", ["tools", 0, "status"], "running"]],
// result
expect.arrayContaining([
],
// result: the running status is deferred like progress, so it is stored with the result, folded into its write
[
["s", ["tools", 0, "status"], "done"],
["s", ["tools", 0, "entry"], result],
]),
],
// the generation's tools phase hands the run to the next generation
expect.arrayContaining([
["d", ["tools"]],
["s", ["run", "taskId"], secondGeneration],
]),
[["s", ["generation"], { attempt: 1 }]],
// the answer ends the run
expect.arrayContaining([
// the answer ends the run, with the second request's deferred attempt first
[
["s", ["generation"], { attempt: 1 }],
["d", ["run"]],
["d", ["generation"]],
]),
],
]);
expect(commits[2]).toHaveLength(3);
expect(commits[4]).toHaveLength(2);
expect(commits[5]).toHaveLength(2);
expect(commits[3]).toHaveLength(2);
await harness.close(context);
});
@ -292,17 +288,26 @@ describe("pi.live deltas", () => {
let liveId: number | undefined;
class RecordingStorage extends MemoryStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
adopt: Parameters<MemoryStorage["commit"]>[1],
batches: Parameters<MemoryStorage["commit"]>[0],
receive: Parameters<MemoryStorage["commit"]>[1],
context: Parameters<MemoryStorage["commit"]>[2],
) {
let kind: "base" | "delta" | undefined;
for (const write of writes) {
if (write.type === "document.change" && write.id === liveId) kind = write.content.kind;
}
const seq = await super.commit(writes, adopt, context);
if (kind !== undefined) written.set(seq, kind);
return seq;
const kinds = batches.map((writes) => {
let kind: "base" | "delta" | undefined;
for (const write of writes) {
if (write.type === "document.change" && write.id === liveId) kind = write.content.kind;
}
return kind;
});
return super.commit(
batches,
(index, seq) => {
const kind = kinds[index];
if (kind !== undefined) written.set(seq, kind);
receive(index, seq);
},
context,
);
}
}
const setup = chatSetup();
@ -329,6 +334,8 @@ describe("pi.live deltas", () => {
fauxAssistantMessage([fauxText("done")]),
]);
setup.settings.toolExecution = "sequential";
// Store deferred progress within each tool's run, so running slots reach Storage as deltas.
setup.settings.progress = { persistWithinMs: 50 };
const { harness, root } = await openChat(new RecordingStorage(), setup);
const values = new Map<number, LiveState>();
harness.observeCommits({

11
packages/durable/test/harness-nested-tools.test.ts
View file

@ -35,6 +35,7 @@ import {
} from "@earendil-works/pi-durable";
import { afterEach, describe, expect, it } from "vitest";
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { allEntries, type ChatSetup, chatSetup, openChat } from "./chat-support.ts";
import { addHooks, addTool } from "./harness-support.ts";
import { endedAt, nestedResultOf, nestedResults, slotsOf, startOrder, viewHistory } from "./live-support.ts";
@ -117,11 +118,11 @@ describe("nested tool calls", () => {
// Per document written, the keys of the nested calls it holds.
const written: { kind: string; key: string | undefined }[] = [];
const commit = storage.commit.bind(storage);
storage.commit = (writes, adopt, commitContext) => {
for (const write of writes) {
storage.commit = (batches, receive, commitContext) => {
for (const write of batches.flat()) {
if (write.type === "document.create") written.push({ kind: write.record.kind, key: write.record.key });
}
return commit(writes, adopt, commitContext);
return commit(batches, receive, commitContext);
};
const { harness, root } = await openChat(storage, setup);
await (await root.submit({ type: "input", content: "go" }, context)).wait(context);
@ -1280,9 +1281,9 @@ describe("tool task version 1 and 2 records", () => {
const { kind: _kind, ...version1 } = record.input as Extract<ToolTaskInput, { kind: "model" }>;
const input = version === 1 ? version1 : record.input;
const state = { ...record.state, checkpoint } as typeof record.state;
await storage.commit(
await commitWrites(
storage,
[{ type: "task", value: { ...record, version, input, state } as typeof record }],
() => {},
context,
);
await storage.close(context);

13
packages/durable/test/harness-observers.test.ts
View file

@ -95,7 +95,7 @@ const collect = (() => {
})();
describe("observeConversation and observeTaskGraph", () => {
it("start at the current value and advance inside adopt with the same change a watch gets", async () => {
it("start at the current value and advance at the publication with the same change a watch gets", async () => {
const storage = new ControlledStorage();
const { harness } = await openTasks(storage, []);
const root = await harness.root(context);
@ -113,7 +113,8 @@ describe("observeConversation and observeTaskGraph", () => {
const entry = await root.commit((tx) => tx.appendEntry(root.id, { kind: "note", data: {} }), context);
expect(view.calls).toHaveLength(1);
const [call] = view.calls as [{ advance: WatchValue<ConversationView>; adopting: boolean }];
expect(call.adopting).toBe(true);
// Published once Storage stored it, outside its call.
expect(call.adopting).toBe(false);
expect(call.advance.retired).toBe(false);
expect(call.advance.value.entries.at(-1)?.id).toBe(entry.id);
await flush();
@ -238,7 +239,7 @@ describe("observeConversation and observeTaskGraph", () => {
ops: [["s", ["tasks", key], expect.objectContaining({ blocked: "missing_task" })]],
retired: false,
},
adopting: true,
adopting: false,
},
]);
registry.install(defineExtension({ name: "late", tasks: [Missing] }));
@ -304,7 +305,7 @@ describe("task documents", () => {
await harness.close(context);
});
it("an observer gets the retirement, then its end, inside the adopt of the commit that ends the task", async () => {
it("an observer gets the retirement, then its end, at the publication of the commit that ends the task", async () => {
const { harness, storage, id, finish } = await openWriter();
const progress = recorder<Readonly<{ lines: string[] }>>(storage);
let endedAdopting: boolean | undefined;
@ -319,10 +320,10 @@ describe("task documents", () => {
finish();
await harness.waitForTask(id, context);
expect(progress.calls).toEqual([
{ advance: { kind: "value", value: { lines: ["started"] }, ops: [], retired: true }, adopting: true },
{ advance: { kind: "value", value: { lines: ["started"] }, ops: [], retired: true }, adopting: false },
{ end: { kind: "ended" } },
]);
expect(endedAdopting).toBe(true);
expect(endedAdopting).toBe(false);
await harness.close(context);
expect(progress.calls).toHaveLength(2);
});

92
packages/durable/test/harness-ownership.test.ts
View file

@ -16,6 +16,7 @@ import {
LiveDoc,
MemoryStorage,
type Storage,
type StorageWrite,
type Submission,
type TaskId,
} from "@earendil-works/pi-durable";
@ -23,7 +24,7 @@ import { afterEach, describe, expect, it } from "vitest";
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { chatSetup, openChat } from "./chat-support.ts";
import { addTask, addTool } from "./harness-support.ts";
import { ControlledStorage, context } from "./session-support.ts";
import { ControlledStorage, context, refusingAt } from "./session-support.ts";
import { aborted, type Deferred, deferred, openTasks, settled } from "./task-support.ts";
/** Outcome a held task commits once its gate opens. */
@ -937,18 +938,25 @@ describe("ownership", () => {
await opened.harness.abortTask(tree.owner, context);
await opened.harness.close(context);
const storage = await openNodeSqliteStorage(path);
const storage: Storage = await openNodeSqliteStorage(path);
let rejectMark: TaskId | undefined;
const commit = storage.commit.bind(storage);
storage.commit = async (writes, adopt, ctx) => {
if (
writes.some((write) => write.type === "task" && write.value.id === rejectMark && write.value.abortRequested)
) {
rejectMark = undefined;
throw new Error("disk gone");
}
return commit(writes, adopt, ctx);
};
// The batch that marks `rejectMark` is refused.
storage.commit = (batches, receive, ctx) =>
refusingAt(
commit,
batches,
receive,
ctx,
(writes) => {
const marks = writes.some(
(write) => write.type === "task" && write.value.id === rejectMark && write.value.abortRequested,
);
if (marks) rejectMark = undefined;
return marks;
},
new Error("disk gone"),
);
opened = await openHarness(storage);
const child = (await opened.harness.conversation(tree.child, context))!;
const late = await child.commit(async (tx) => {
@ -971,19 +979,19 @@ describe("ownership", () => {
const failure = new Error("disk gone");
let failMark: TaskId | undefined;
class Failing extends ControlledStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
adopt: Parameters<MemoryStorage["commit"]>[1],
override commit(
batches: Parameters<MemoryStorage["commit"]>[0],
receive: Parameters<MemoryStorage["commit"]>[1],
ctx: Parameters<MemoryStorage["commit"]>[2],
) {
const marks = writes.some(
(write) => write.type === "task" && write.value.id === failMark && write.value.abortRequested,
);
if (marks) {
failMark = undefined;
throw failure;
}
return super.commit(writes, adopt, ctx);
const refuses = (writes: readonly StorageWrite[]): boolean => {
const marks = writes.some(
(write) => write.type === "task" && write.value.id === failMark && write.value.abortRequested,
);
if (marks) failMark = undefined;
return marks;
};
return refusingAt((...args) => super.commit(...args), batches, receive, ctx, refuses, failure);
}
}
const storage = new Failing();
@ -1056,7 +1064,7 @@ describe("owned conversations from tools and supervisors", () => {
await harness.close(context);
});
it("rejects a handle operation queued on the line when the invocation ends before it runs", async () => {
it("runs a handle operation queued behind the abort mark before the mark's publication, and the cascade aborts it", async () => {
const setup = chatSetup();
const ready = deferred<{ child: ConversationId; taskId: TaskId }>();
const go = deferred();
@ -1087,9 +1095,29 @@ describe("owned conversations from tools and supervisors", () => {
setup.faux.setResponses([
fauxAssistantMessage([fauxToolCall("delegate", {}, { id: "c1" })], { stopReason: "toolUse" }),
]);
const { harness, root } = await openChat(new MemoryStorage(), setup);
// Holds the storage call that writes the abort mark, so the mark is published only once the test lets it.
let holdMark: TaskId | undefined;
const stored = deferred();
class HoldMark extends MemoryStorage {
override async commit(
batches: Parameters<MemoryStorage["commit"]>[0],
receive: Parameters<MemoryStorage["commit"]>[1],
ctx: Parameters<MemoryStorage["commit"]>[2],
) {
const marks = batches
.flat()
.some((write) => write.type === "task" && write.value.id === holdMark && write.value.abortRequested);
if (marks) {
holdMark = undefined;
await stored.promise;
}
return super.commit(batches, receive, ctx);
}
}
const { harness, root } = await openChat(new HoldMark(), setup);
await root.submit({ type: "input", content: "go" }, context);
const { child, taskId } = await ready.promise;
holdMark = taskId;
// Hold the Session line, queue the abort mark behind it, then let the tool queue its submit.
const release = deferred();
const holding = root.commit(async () => {
@ -1100,10 +1128,22 @@ describe("owned conversations from tools and supervisors", () => {
const { submitting } = await queued.promise;
release.resolve();
await holding;
// The submit starts while the mark's storage call is in flight.
await new Promise((resolve) => setTimeout(resolve, 0));
stored.resolve();
await aborting;
await expect(submitting).rejects.toThrow();
// Cancellation is checked when a job starts: the submit touches nothing the mark writes, so it starts before the
// mark is published and lands. The mark's cascade then aborts the run it started in the owned conversation.
const submission = (await submitting) as Submission;
const childConversation = (await harness.conversation(child, context))!;
expect((await childConversation.entries({}, 10, undefined, context)).items).toEqual([]);
await childConversation.waitForIdle(context);
expect(await (await harness.submission(submission.id, context))!.status(context)).toMatchObject({
status: "unanswered",
reason: "aborted",
});
expect((await childConversation.entries({}, 10, undefined, context)).items.map((entry) => entry.kind)).toEqual([
"pi.user",
]);
expect((await harness.inspect(context)).submissions.filter((record) => record.conversationId === child)).toEqual(
[],
);

8
packages/durable/test/harness-restart-failure.test.ts
View file

@ -29,6 +29,7 @@ import {
} from "@earendil-works/pi-durable";
import { afterEach, describe, expect, it } from "vitest";
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { allEntries, type ChatSetup, chatSetup, openChat, textOf, unanswered, waitFor } from "./chat-support.ts";
import { addSection, addTool } from "./harness-support.ts";
import { context, followCommits } from "./session-support.ts";
@ -650,7 +651,7 @@ describe("crash edges (#11139)", () => {
// As if the charge had been cut off twice before: the open after this crash counts 3, past the limit of 2.
const raw = await openNodeSqliteStorage(path);
const record = await raw.task(taskId, context);
await raw.commit([{ type: "task", value: { ...record!, restarts: 2 } }], () => {}, context);
await commitWrites(raw, [{ type: "task", value: { ...record!, restarts: 2 } }], context);
await raw.close(context);
const setup = chatSetup();
setup.settings.restartLimit = 2;
@ -770,8 +771,9 @@ describe("the restart limit on the normal path (#11139)", () => {
for (let round = 0; round < rounds; round++) {
await (await root.submit({ type: "input", content: `round ${round}` }, context)).wait(context);
}
// One generation round with one tool call, as before the restart limit applied to every kind (#11327: 49).
expect(resolutions / rounds).toBe(49);
// One generation round with one tool call, no more often than before the restart limit applied to every kind
// (#11327: 49; group commit coalesces some reconcile passes).
expect(resolutions / rounds).toBeLessThanOrEqual(49);
await harness.close(context);
});
});

32
packages/durable/test/harness-scheduler-indexes.test.ts
View file

@ -3,7 +3,14 @@
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createSession, defineTask, type Harness, type TaskId, type TaskRuntime } from "@earendil-works/pi-durable";
import {
createSession,
defineTask,
type Harness,
type Storage,
type TaskId,
type TaskRuntime,
} from "@earendil-works/pi-durable";
import { afterEach, describe, expect, it } from "vitest";
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { context } from "./session-support.ts";
@ -104,16 +111,16 @@ describe("scheduler indexes", () => {
}, context);
await session.close(context);
// Record every mark of the leaf, in commit order.
const storage = await openNodeSqliteStorage(path);
const storage: Storage = await openNodeSqliteStorage(path);
const reasons: (string | undefined)[] = [];
const commit = storage.commit.bind(storage);
storage.commit = (writes, adopt, commitContext) => {
for (const write of writes) {
storage.commit = (batches, receive, commitContext) => {
for (const write of batches.flat()) {
if (write.type === "task" && write.value.id === ids.leaf && write.value.abortRequested) {
reasons.push(write.value.abortReason);
}
}
return commit(writes, adopt, commitContext);
return commit(batches, receive, commitContext);
};
const { harness } = await openTasks(storage, [Hold]);
harness.resume();
@ -123,17 +130,20 @@ describe("scheduler indexes", () => {
});
it("finalizes a chain of held owners in the commit that frees the last one", async () => {
const storage = await openNodeSqliteStorage(await sqlitePath());
const storage: Storage = await openNodeSqliteStorage(await sqlitePath());
// The commit, by count, that wrote each terminal record.
const endedIn = new Map<TaskId, number>();
let commits = 0;
const commit = storage.commit.bind(storage);
storage.commit = (writes, adopt, commitContext) => {
commits++;
for (const write of writes) {
if (write.type === "task" && write.value.state.status === "terminal") endedIn.set(write.value.id, commits);
storage.commit = (batches, receive, commitContext) => {
for (const writes of batches) {
commits++;
for (const write of writes) {
if (write.type === "task" && write.value.state.status === "terminal")
endedIn.set(write.value.id, commits);
}
}
return commit(writes, adopt, commitContext);
return commit(batches, receive, commitContext);
};
const opened = await openTasks(storage, [Hold]);
const harness = opened.harness;

52
packages/durable/test/harness-scheduler-retention.test.ts
View file

@ -125,21 +125,6 @@ async function sqlitePath(): Promise<string> {
}
/** Count the task and conversation reads a storage serves. */
function countReads(storage: Storage): { tasks: number; conversations: number } {
const reads = { tasks: 0, conversations: 0 };
const task = storage.task.bind(storage);
const conversation = storage.conversation.bind(storage);
storage.task = (id, readContext) => {
reads.tasks++;
return task(id, readContext);
};
storage.conversation = (id, readContext) => {
reads.conversations++;
return conversation(id, readContext);
};
return reads;
}
async function open(storage?: Storage) {
const opened = await openTasks(storage ?? (await openNodeSqliteStorage(await sqlitePath())), [
Hold,
@ -191,9 +176,7 @@ describe("scheduler retention", () => {
});
it("holds a live owner for work later created in a forgotten conversation, loading its chain again", async () => {
const storage = await openNodeSqliteStorage(await sqlitePath());
const reads = countReads(storage);
const { harness, root } = await open(storage);
const { harness, root } = await open();
// Live P -> ended B -> ended A -> conversation C, empty: forgotten.
const tree = await root.commit(async (tx) => {
const p = await tx.createTask(Hold, { name: "p" }, { ownership: { kind: "conversation" } });
@ -206,15 +189,14 @@ describe("scheduler retention", () => {
gate("b").resolve();
await harness.waitForTask(tree.b, context);
const before = sizes(harness);
const loaded = reads.tasks + reads.conversations;
const c = (await harness.conversation(tree.c, context))!;
const d = await c.commit(
(tx) => tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } }),
context,
);
gate("p").resolve();
// P holds for D: the scheduler loaded the chain from C up to P again.
await eventually(async () => (await status(harness, tree.p)).status === "completing");
expect(reads.tasks + reads.conversations).toBeGreaterThan(loaded);
gate("d").resolve();
expect((await harness.waitForTask(d, context)).state.outcome.status).toBe("completed");
expect((await harness.waitForTask(tree.p, context)).state.outcome.status).toBe("completed");
@ -307,7 +289,7 @@ describe("scheduler retention", () => {
await harness.close(context);
});
it("keeps a scope busy while the chain of new work below a forgotten conversation loads", async () => {
it("loads the chain of new work below a forgotten conversation before admitting it, and keeps the scope busy", async () => {
const path = await sqlitePath();
const storage = await openNodeSqliteStorage(path);
const { harness, root } = await open(storage);
@ -318,28 +300,36 @@ describe("scheduler retention", () => {
}, context);
gate("b").resolve();
await harness.waitForTask(tree.b, context);
// Hold the chain load's read of C: the second read once holding, after the creating commit's own.
const read = storage.conversation.bind(storage);
await harness.close(context);
// Reopen, so that B is in no cache: the chain load reads it from Storage. Hold that read.
const reopened = await openNodeSqliteStorage(path);
const read = reopened.task.bind(reopened);
const held = deferred<void>();
const reached = deferred<void>();
let holding = false;
let readsOfC = 0;
storage.conversation = async (id, readContext) => {
if (holding && id === tree.c && ++readsOfC === 2) {
reopened.task = async (id, readContext) => {
if (holding && id === tree.b) {
reached.resolve();
await held.promise;
}
return read(id, readContext);
};
const c = (await harness.conversation(tree.c, context))!;
const second = (await open(reopened)).harness;
const c = (await second.conversation(tree.c, context))!;
holding = true;
await c.commit((tx) => tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } }), context);
const committing = c.commit(
(tx) => tx.createTask(Hold, { name: "d" }, { ownership: { kind: "conversation" } }),
context,
);
// Its admission loads the chain, for the memberships it writes: the commit waits for the read.
await reached.promise;
expect(await settled(harness.waitForIdle(context))).toBe(false);
expect(await settled(committing)).toBe(false);
held.resolve();
await committing;
expect(await settled(second.waitForIdle(context))).toBe(false);
gate("d").resolve();
await harness.waitForIdle(context);
await harness.close(context);
await second.waitForIdle(context);
await second.close(context);
});
it("keeps a chain when one commit ends its last task and creates new work below it", async () => {

2
packages/durable/test/harness-storage-failure.test.ts
View file

@ -419,6 +419,8 @@ describe("a failed Storage call", () => {
}
}
const { harness, root } = await started(new CancelledRead());
// An entry to scan: the Session's cache answers the bounds of the context, not its entries.
await root.commit((tx) => tx.appendEntry(root.id, { kind: "note" }), context);
const cancelling: Context = { ...context, abortSignal: controller.signal };
await expect(root.context(cancelling)).rejects.toBe(cancelled);
await expect(root.commit((tx) => tx.appendEntry(root.id, { kind: "note" }), context)).resolves.toBeDefined();

49
packages/durable/test/harness-structured.test.ts
View file

@ -26,6 +26,7 @@ import {
type NextTaskState,
type Registry,
type Storage,
type StorageWrite,
type TaskGraph,
type TaskId,
type TaskOptions,
@ -39,7 +40,7 @@ import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { allEntries, chatSetup, openChat, waitFor } from "./chat-support.ts";
import { addHooks, addTask, addTool } from "./harness-support.ts";
import { generations, graphHistory, slotsOf, viewHistory } from "./live-support.ts";
import { ControlledStorage, context } from "./session-support.ts";
import { ControlledStorage, context, refusingAt } from "./session-support.ts";
import { aborted, type Deferred, deferred, openTasks, settled } from "./task-support.ts";
// ─── A scriptable task ──────────────────────────────────────────────────────
@ -1361,19 +1362,20 @@ describe("tool rounds", () => {
let reject: TaskId | undefined;
const failure = new Error("disk gone");
class Failing extends ControlledStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
adopt: Parameters<MemoryStorage["commit"]>[1],
override commit(
batches: Parameters<MemoryStorage["commit"]>[0],
receive: Parameters<MemoryStorage["commit"]>[1],
ctx: Parameters<MemoryStorage["commit"]>[2],
) {
const finalizes = writes.some(
(write) => write.type === "task" && write.value.id === reject && write.value.state.status === "terminal",
);
if (finalizes) {
reject = undefined;
throw failure;
}
return super.commit(writes, adopt, ctx);
const refuses = (writes: readonly StorageWrite[]): boolean => {
const finalizes = writes.some(
(write) =>
write.type === "task" && write.value.id === reject && write.value.state.status === "terminal",
);
if (finalizes) reject = undefined;
return finalizes;
};
return refusingAt((...args) => super.commit(...args), batches, receive, ctx, refuses, failure);
}
}
const storage = new Failing();
@ -1647,19 +1649,20 @@ describe("conversation abort and boundaries", () => {
let reject: TaskId | undefined;
const failure = new Error("disk gone");
class Failing extends ControlledStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
adopt: Parameters<MemoryStorage["commit"]>[1],
override commit(
batches: Parameters<MemoryStorage["commit"]>[0],
receive: Parameters<MemoryStorage["commit"]>[1],
ctx: Parameters<MemoryStorage["commit"]>[2],
) {
const finalizes = writes.some(
(write) => write.type === "task" && write.value.id === reject && write.value.state.status === "terminal",
);
if (finalizes) {
reject = undefined;
throw failure;
}
return super.commit(writes, adopt, ctx);
const refuses = (writes: readonly StorageWrite[]): boolean => {
const finalizes = writes.some(
(write) =>
write.type === "task" && write.value.id === reject && write.value.state.status === "terminal",
);
if (finalizes) reject = undefined;
return finalizes;
};
return refusingAt((...args) => super.commit(...args), batches, receive, ctx, refuses, failure);
}
}
const storage = new Failing();

11
packages/durable/test/harness-submissions.test.ts
View file

@ -163,9 +163,14 @@ describe("submissions", () => {
}
}
const setup = chatSetup();
setup.faux.setResponses([unanswered().step]);
const { harness, root } = await openChat(new HeldReads(), setup);
const submission = await root.submit({ type: "input", content: "hi" }, context);
setup.faux.setResponses([unanswered().step, unanswered().step]);
const storage = new HeldReads();
const first = await openChat(storage, setup);
const { id } = await first.root.submit({ type: "input", content: "hi" }, context);
await first.harness.close(context);
// Reopen: the Session caches the submissions it writes, so only a reopened one reads it from Storage.
const { harness } = await openChat(storage.reopen(), setup);
const submission = (await harness.submission(id, context))!;
hold = true;
const waiting = submission.wait(context);
await entered.promise;

6
packages/durable/test/harness-task-graph.test.ts
View file

@ -365,7 +365,7 @@ describe("task graph view", () => {
await harness.close(context);
});
it("advances with the conversation view inside the one storage commit that changes both", async () => {
it("advances with the conversation view at the publication of the one commit that changes both", async () => {
const storage = new ControlledStorage();
const { Parent, Child } = family(gatesNever());
const { harness } = await openTasks(storage, [Parent, Child]);
@ -383,8 +383,8 @@ describe("task graph view", () => {
await tx.appendEntry(root.id, { kind: "note" });
await tx.createTask(Parent, null, { ownership: { kind: "conversation" } });
}, context);
// Both inside the one adopt, in no promised order.
expect([...seen].sort()).toEqual(["graph adopting=true", "view adopting=true"]);
// Both at the one publication, after Storage stored it, in no promised order.
expect([...seen].sort()).toEqual(["graph adopting=false", "view adopting=false"]);
view.detach();
graph.detach();
await harness.close(context);

74
packages/durable/test/harness-tasks.test.ts
View file

@ -20,6 +20,7 @@ import {
} from "@earendil-works/pi-durable";
import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry";
import { describe, expect, it } from "vitest";
import type { SessionImpl } from "../src/session/session.ts";
import { addHooks, addTask, addTool, user } from "./harness-support.ts";
import { ControlledStorage, context, flush, followCommits, LateResolvingStorage, onValue } from "./session-support.ts";
import { aborted, abortedWith, completed, deferred, eventually, openTasks, settled } from "./task-support.ts";
@ -85,6 +86,20 @@ function withSignal(signal: AbortSignal): Context {
return withAbortSignal(signal, context);
}
/** Hold the Session line with a commit whose callback waits: the jobs queued after it start once it is released. */
function holdLine(harness: { commit(change: () => Promise<void>, context: Context): Promise<void> }): {
readonly entered: Promise<void>;
release(): void;
} {
const entered = deferred();
const release = deferred();
void harness.commit(async () => {
entered.resolve();
await release.promise;
}, context);
return { entered: entered.promise, release: () => release.resolve() };
}
describe("task phases", () => {
it("continues one invocation through checkpoint progress and completes with a typed result", async () => {
const seen: number[] = [];
@ -235,7 +250,15 @@ describe("task phases", () => {
},
abort: async () => {},
});
const { harness, root } = await openRoot([Writer, Child]);
class CountingStorage extends MemoryStorage {
documentScans = 0;
override scanDocuments(...args: Parameters<MemoryStorage["scanDocuments"]>) {
this.documentScans++;
return super.scanDocuments(...args);
}
}
const storage = new CountingStorage();
const { harness, root } = await openRoot([Writer, Child], { storage });
const conversation = await harness.createConversation({ ownership: { kind: "ownerless" } }, context);
const id = await conversation.commit(
(tx) => tx.createTask(Writer, null, { ownership: { kind: "conversation" } }),
@ -252,6 +275,10 @@ describe("task phases", () => {
expect(entry?.conversationId).toBe(conversation.id);
expect(progressSeen).toEqual(["wrote"]);
expect(await harness.snapshot(Progress, id, context)).toBeUndefined();
// The Session indexed the documents of the tasks it created, so no terminal commit scanned for them, and it dropped
// each task's entry at its end.
expect(storage.documentScans).toBe(0);
expect((harness as unknown as SessionImpl).documentCacheSizes.taskDocuments).toBe(0);
expect(await harness.getTask(id, context)).toEqual(receipt);
expect((await harness.waitForTask(childId!, context)).conversationId).toBe(conversation.id);
expect(root.id).not.toBe(conversation.id);
@ -1208,13 +1235,11 @@ describe("task close", () => {
it("writes no fault when close seals while the step after a failed phase is queued", async () => {
const storage = new ControlledStorage();
let held: ReturnType<ControlledStorage["holdCommits"]> | undefined;
let held: ReturnType<typeof holdLine> | undefined;
let harnessRef: Awaited<ReturnType<typeof openRoot>>["harness"] | undefined;
const Throws = oneStep("test.close-fault", async (_task, runtime) => {
held = storage.holdCommits();
void harnessRef!.commit(async (tx) => {
await tx.appendEntry(runtime.conversationId, { kind: "blocker" });
}, context);
const Throws = oneStep("test.close-fault", async () => {
// The step after this phase queues behind a job that holds the line.
held = holdLine(harnessRef!);
throw new Error("would fault");
});
const { harness, root } = await openRoot([Throws], { storage });
@ -1269,14 +1294,11 @@ describe("task close", () => {
it("rejects a runtime commit that was queued on the line when close sealed it", async () => {
const storage = new ControlledStorage();
let queued: Promise<unknown> | undefined;
let held: ReturnType<ControlledStorage["holdCommits"]> | undefined;
let held: ReturnType<typeof holdLine> | undefined;
let harnessRef: Awaited<ReturnType<typeof openRoot>>["harness"] | undefined;
const proceed = deferred();
const Queued = oneStep("test.close-queued-commit", async (_task, runtime) => {
held = storage.holdCommits();
void harnessRef!.commit(async (tx) => {
await tx.appendEntry(runtime.conversationId, { kind: "blocker" });
}, context);
held = holdLine(harnessRef!);
queued = runtime.commit(() => completed(null), context).catch((error: unknown) => error);
// Ignores the close signal, so the invocation is still alive when the queued commit reaches the line.
await proceed.promise;
@ -1459,7 +1481,7 @@ describe("publication at adoption", () => {
});
}
it("resolves a waitForIdle() called after adoption, before the storage resolved, after host listeners", async () => {
it("resolves a waitForIdle() called while the storage stores the commit at its publication, after host listeners", async () => {
const storage = new LateResolvingStorage();
const gate = deferred();
const Gated = gated("test.gap-idle", gate.promise);
@ -1484,7 +1506,7 @@ describe("publication at adoption", () => {
gate.resolve();
await gapped.promise;
await idle;
expect(order).toEqual(["host", "idle committing=true"]);
expect(order).toEqual(["host", "idle committing=false"]);
await harness.close(context);
});
@ -1545,7 +1567,7 @@ describe("publication at adoption", () => {
await closing;
});
it("keeps a task wait resolved when the Harness closes after adoption, before the storage resolved", async () => {
it("ends a task wait when the Harness closes while the terminal commit is stored, and still stores it", async () => {
const storage = new LateResolvingStorage();
const gate = deferred();
const Gated = gated("test.close-in-gap", gate.promise);
@ -1567,8 +1589,12 @@ describe("publication at adoption", () => {
gate.resolve();
await gapped.promise;
await closing;
expect(await waited).toEqual({ status: "completed", result: null });
// Close ends the waits before it drains the commit in flight, which is published and stored all the same.
expect(await waited).toMatchObject({ message: expect.stringContaining("closed") });
expect(await harness.closed).toEqual({ kind: "closed" });
const reopened = await openRoot([Gated], { storage: storage.reopen() });
expect((await reopened.harness.getTask(id, context))?.state.status).toBe("terminal");
await reopened.harness.close(context);
});
it("fails the Session with the storage error when it rejects after adopting, and reports an observer's throw", async () => {
@ -1627,14 +1653,22 @@ describe("publication at adoption", () => {
await harness.close(context);
});
it("calls a host listener added on a Harness between adopt and the dispatch only when one was registered at adopt", async () => {
it("calls a host listener added on a Harness during a publication only when one was registered at it", async () => {
const storage = new ControlledStorage();
const { harness, root } = await openRoot([], { storage });
const note = () => root.commit((tx) => tx.appendEntry(root.id, { kind: "note", data: {} }), context);
const late: number[] = [];
let stopLate = { detach: (): void => {} };
// No host listener at adopt, although the Harness's own observers run: no dispatch, so the late one misses it.
storage.afterAdopt = () => {
// Runs once, during the next publication, after its dispatch was queued or not.
let during: (() => void) | undefined;
followCommits(harness, () => {
const run = during;
during = undefined;
run?.();
});
// No host listener at the publication, although the Harness's own observers run: no dispatch, so the late one
// misses it.
during = () => {
stopLate = harness.observeCommits({
advance: (publication) => {
late.push(publication.seq);
@ -1651,7 +1685,7 @@ describe("publication at adoption", () => {
// One registered at adopt: the dispatch reads the listeners when it runs, the late one included.
harness.observeCommits({ advance: () => {}, end: () => {} });
const second: number[] = [];
storage.afterAdopt = () => {
during = () => {
harness.observeCommits({
advance: (publication) => {
second.push(publication.seq);

3
packages/durable/test/harness-tool-hooks.test.ts
View file

@ -31,6 +31,7 @@ import {
} from "@earendil-works/pi-durable";
import { afterEach, describe, expect, it } from "vitest";
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { allEntries, type ChatSetup, chatSetup, openChat, waitFor } from "./chat-support.ts";
import { addHooks, addTool } from "./harness-support.ts";
import { context } from "./session-support.ts";
@ -763,7 +764,7 @@ describe("calls to tools the request did not offer", () => {
if (version === 1) {
const storage = await openNodeSqliteStorage(path);
const record = (await storage.task(generation, context))!;
await storage.commit([{ type: "task", value: { ...record, version: 1 } }], () => {}, context);
await commitWrites(storage, [{ type: "task", value: { ...record, version: 1 } }], context);
await storage.close(context);
}

64
packages/durable/test/harness-tools-recovery.test.ts
View file

@ -133,6 +133,70 @@ describe("tool recovery", () => {
await opened.harness.close(context);
});
it("reruns a safe tool whose running slot a crash after its intent lost, or a sibling stored, and shows it running", async () => {
for (const sibling of [false, true]) {
const path = await sqlitePath();
const setup = chatSetup();
// The running slot is presentation, held in memory; no deadline stores it before the crash.
setup.settings.progress = { persistWithinMs: Number.POSITIVE_INFINITY };
const started = deferred();
let runs = 0;
let reopened: Harness | undefined;
const seen: (string | undefined)[] = [];
addTool(
setup.registry,
tool(
"work",
async (_args, api, callContext) => {
runs++;
if (runs === 1) {
started.resolve();
await aborted(callContext.abortSignal!);
}
const slots = (await reopened!.snapshot(LiveDoc, 1 as never, callContext))?.tools ?? [];
seen.push(slots.find((slot) => slot.taskId === api.taskId)?.status);
return {};
},
{ replay: "safe" },
),
);
// A sibling that ends once `work` started: its result commit reads `pi.live`, which stores what is held there.
addTool(
setup.registry,
tool("quick", async () => {
await started.promise;
return {};
}),
);
const calls = [fauxToolCall("work", {}, { id: "w" })];
if (sibling) calls.push(fauxToolCall("quick", {}, { id: "q" }));
setup.faux.setResponses([fauxAssistantMessage(calls, { stopReason: "toolUse" }), DONE]);
const crashed = await open(path, setup);
const id = (await crashed.root.submit({ type: "input", content: "go" }, context)).id;
await started.promise;
const slot = async (harness: Harness, name: string) =>
(await harness.snapshot(LiveDoc, crashed.root.id, context))?.tools?.find((each) => each.name === name);
const live = (harness: Harness) => slot(harness, "work");
const workId = (await live(crashed.harness))!.taskId!;
if (sibling) await waitFor(async () => (await slot(crashed.harness, "quick"))?.status === "done");
// The process dies here: another one opens the same database, without the held presentation.
const next = await openChat(await openNodeSqliteStorage(path), setup);
reopened = next.harness;
expect((await next.harness.getTask(workId, context))?.state).toMatchObject({
checkpoint: { phase: "execute" },
});
expect((await live(next.harness))?.status).toBe(sibling ? "running" : "pending");
next.harness.resume();
expect((await (await next.harness.submission(id, context))!.wait(context)).status).toBe("done");
// The rerun shows the slot running again.
expect(seen).toEqual(["running"]);
const [result] = results(await allEntries(next.root)).filter((message) => message.toolCallId === "w");
expect(result!.isError).toBe(false);
await next.harness.close(context);
await crashed.harness.close(context);
}
});
it("reruns a tool only when both the stored and the current replay policy are safe", async () => {
const cases: [stored: "safe" | "unsafe", current: "safe" | "unsafe", reruns: boolean][] = [
["safe", "safe", true],

3
packages/durable/test/harness-view.test.ts
View file

@ -142,7 +142,8 @@ describe("conversation view", () => {
await submission.wait(context);
await harness.waitForIdle(context);
await drained();
expect(frames).toHaveLength(touching.count);
// One frame per stored commit, and one for the attempt, a deferred commit stored with the answer.
expect(frames).toHaveLength(touching.count + 1);
const last = replay(initial, frames);
expect(last).toEqual(await committed(harness, root, initial.conversation, last.entries.length));
expect(frames[0]!.ops).toContainEqual(["p", ["entries"], 0, 0, [expect.objectContaining({ kind: "pi.user" })]]);

211
packages/durable/test/indexeddb-storage.test.ts
View file

@ -3,11 +3,13 @@
import type { Context, JsonValue } from "@earendil-works/chord";
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
import { registerStorageConformance } from "@earendil-works/pi-durable/testing";
import { IDBKeyRange as FakeIDBKeyRange, IDBFactory } from "fake-indexeddb";
import { describe, expect, it } from "vitest";
import { IDBKeyRange as FakeIDBKeyRange, IDBObjectStore as FakeIDBObjectStore, IDBFactory } from "fake-indexeddb";
import { describe, expect, it, vi } from "vitest";
import { idFromNumber } from "../src/ids.ts";
import { IndexedDbStorage } from "../src/storage/indexeddb.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import type {
CommitResult,
ConversationId,
DocumentCreate,
DocumentId,
@ -22,9 +24,6 @@ import type {
} from "../src/types.ts";
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
Object.defineProperty(globalThis, "IDBKeyRange", { configurable: true, value: FakeIDBKeyRange });
const context = BACKGROUND_CONTEXT;
@ -42,10 +41,14 @@ class ReopeningStorage implements Storage {
this.indexedDB = indexedDB;
}
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context): Promise<Seq> {
if (this.closed) return this.current.commit(writes, adopt, commitContext);
async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
commitContext: Context,
): Promise<CommitResult> {
if (this.closed) return this.current.commit(batches, receive, commitContext);
try {
return await this.current.commit(writes, adopt, commitContext);
return await this.current.commit(batches, receive, commitContext);
} finally {
await this.current.close(context);
this.current = await IndexedDbStorage.open(this.name, { indexedDB: this.indexedDB });
@ -135,7 +138,7 @@ function pendingTask(id: TaskId<JsonValue>, conversationId = ROOT_CONVERSATION_I
}
async function createRoot(storage: Storage): Promise<void> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
}
it("persists minted IDs across reopen", async () => {
@ -164,15 +167,16 @@ it("rolls back table writes when a later document copy fails", async () => {
history: "rewindable",
fork: "asOf",
} satisfies DocumentCreate;
await storage.commit(
await commitWrites(
storage,
[{ type: "document.create", record: source, content: { kind: "base", version: 1, value: { count: 1 } } }],
adopt,
context,
);
const taskId = await storage.mintId<TaskId<JsonValue>>();
const copyId = await storage.mintId<DocumentId>();
await expect(
storage.commit(
commitWrites(
storage,
[
{ type: "task", value: pendingTask(taskId) },
{
@ -181,7 +185,6 @@ it("rolls back table writes when a later document copy fails", async () => {
source: { id: sourceId, at: "current" },
},
],
adopt,
context,
),
).rejects.toThrow("was rejected");
@ -199,7 +202,8 @@ it("scans entries and head markers through fork ancestry", async () => {
await createRoot(storage);
const rootFirst = await storage.mintId<EntryId>();
const rootMarker = await storage.mintId<EntryId>();
await storage.commit(
await commitWrites(
storage,
[
{ type: "entry", value: { id: rootFirst, conversationId: ROOT_CONVERSATION_ID, kind: "first" } },
{
@ -212,24 +216,23 @@ it("scans entries and head markers through fork ancestry", async () => {
},
},
],
adopt,
context,
);
const childId = await storage.mintId<ConversationId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "conversation",
value: { id: childId, parent: { conversationId: ROOT_CONVERSATION_ID, at: rootMarker } },
},
],
adopt,
context,
);
const childEntry = await storage.mintId<EntryId>();
await storage.commit(
await commitWrites(
storage,
[{ type: "entry", value: { id: childEntry, conversationId: childId, kind: "child" } }],
adopt,
context,
);
expect(
@ -279,7 +282,7 @@ it("keeps one readwrite transaction active through many mixed writes", async ()
},
);
}
await storage.commit(writes, adopt, context);
await commitWrites(storage, writes, context);
expect((await storage.entry(lastEntryId!, context))?.entry.data).toEqual({ index: 39 });
expect(await storage.task(lastTaskId!, context)).toEqual(pendingTask(lastTaskId!));
expect((await storage.submission(lastSubmissionId!, context))?.requestId).toBe("bulk-39");
@ -288,3 +291,171 @@ it("keeps one readwrite transaction active through many mixed writes", async ()
await storage.close(context);
}
});
describe("commit", () => {
/** An entry of the root conversation. */
const entryWrite = (id: EntryId): StorageWrite => ({
type: "entry",
value: { id, conversationId: ROOT_CONVERSATION_ID, kind: "batch" },
});
const sourceRecord = (id: DocumentId, conversationId: ConversationId = ROOT_CONVERSATION_ID): DocumentCreate => ({
id,
kind: "source",
scope: { kind: "conversation", conversationId },
history: "rewindable",
fork: "asOf",
});
/** Opens a storage with its root conversation, runs `test` on it, closes it. */
async function withStorage(name: string, test: (storage: IndexedDbStorage) => Promise<void>): Promise<void> {
const storage = await IndexedDbStorage.open(name, { indexedDB: new IDBFactory() });
try {
await createRoot(storage);
await test(storage);
} finally {
await storage.close(context);
}
}
it("stores the batches before a refused one, and the storage goes on", () =>
withStorage("batch-prefix", async (storage) => {
const first = await storage.mintId<EntryId>();
const after = await storage.mintId<EntryId>();
const receipts: [number, Seq][] = [];
const result = await storage.commit(
[[entryWrite(first)], [entryWrite(first)], [entryWrite(after)]],
(index, seq) => receipts.push([index, seq]),
context,
);
expect(result.stored).toBe(1);
expect(String(result.refused)).toBe(`Error: ID ${first} already belongs to entry`);
expect(receipts.map(([index]) => index)).toEqual([0]);
expect((await storage.entry(first, context))?.commitSeq).toBe(receipts[0]![1]);
expect(await storage.entry(after, context)).toBeUndefined();
// The prefix advanced the metadata: the next commit takes the next seq.
const next = await commitWrites(storage, [entryWrite(after)], context);
expect(next).toBe(receipts[0]![1] + 1);
}));
it("resolves a copy against the state an earlier batch of the call left", () =>
withStorage("batch-copy", async (storage) => {
const sourceId = await storage.mintId<DocumentId>();
const childId = await storage.mintId<ConversationId>();
const copyId = await storage.mintId<DocumentId>();
const result = await storage.commit(
[
[
{
type: "document.create",
record: sourceRecord(sourceId),
content: { kind: "base", version: 1, value: { count: 1 } },
},
],
[
{
type: "document.change",
id: sourceId,
content: { kind: "delta", version: 1, ops: [["s", ["count"], 2]] },
},
],
[
{ type: "conversation", value: { id: childId } },
{
type: "document.copy",
record: sourceRecord(copyId, childId),
source: { id: sourceId, at: "current" },
},
],
],
() => {},
context,
);
expect(result).toEqual({ stored: 3 });
expect((await storage.document(copyId, "current", context))?.value).toEqual({ count: 2 });
}));
it("refuses a batch whose copy fails before it writes anything, keeping the batches before it", () =>
withStorage("batch-copy-refused", async (storage) => {
const sourceId = await storage.mintId<DocumentId>();
await commitWrites(
storage,
[
{
type: "document.create",
record: sourceRecord(sourceId),
content: { kind: "base", version: 1, value: { count: 1 } },
},
],
context,
);
const first = await storage.mintId<EntryId>();
const taskId = await storage.mintId<TaskId<JsonValue>>();
const copyId = await storage.mintId<DocumentId>();
const result = await storage.commit(
[
[entryWrite(first)],
// The task comes before the copy: the refusal must leave it unwritten.
[
{ type: "task", value: pendingTask(taskId) },
{
type: "document.copy",
record: { ...sourceRecord(copyId), kind: "mismatch" },
source: { id: sourceId, at: "current" },
},
],
],
() => {},
context,
);
expect(result.stored).toBe(1);
expect(String(result.refused)).toContain(`Document copy ${copyId} was rejected`);
expect(await storage.entry(first, context)).toBeDefined();
expect(await storage.task(taskId, context)).toBeUndefined();
expect(await storage.document(copyId, "current", context)).toBeUndefined();
}));
it("rejects with nothing stored when IndexedDB fails during a batch's checks", () =>
withStorage("batch-check-failure", async (storage) => {
const first = await storage.mintId<EntryId>();
const second = await storage.mintId<EntryId>();
const get = FakeIDBObjectStore.prototype.get;
const failure = new DOMException("disk gone", "UnknownError");
const spy = vi.spyOn(FakeIDBObjectStore.prototype, "get").mockImplementation(function (
this: InstanceType<typeof FakeIDBObjectStore>,
key: Parameters<typeof get>[0],
) {
if (this.name === "record_ids" && key === second) throw failure;
return get.call(this, key);
});
const receipts: number[] = [];
try {
await expect(
storage.commit([[entryWrite(first)], [entryWrite(second)]], (index) => receipts.push(index), context),
).rejects.toBe(failure);
} finally {
spy.mockRestore();
}
expect(receipts).toEqual([]);
expect(await storage.entry(first, context)).toBeUndefined();
await commitWrites(storage, [entryWrite(first)], context);
expect(await storage.entry(first, context)).toBeDefined();
}));
it("rejects with nothing stored when IndexedDB fails during a batch's writes", () =>
withStorage("batch-write-failure", async (storage) => {
const first = await storage.mintId<EntryId>();
const second = await storage.mintId<EntryId>();
// A function cannot be stored: the put throws IndexedDB's `DataCloneError`.
const unstorable = {
type: "entry",
value: { id: second, conversationId: ROOT_CONVERSATION_ID, kind: "batch", data: { f: () => {} } },
} as unknown as StorageWrite;
const receipts: number[] = [];
await expect(
storage.commit([[entryWrite(first)], [unstorable]], (index) => receipts.push(index), context),
).rejects.toMatchObject({ name: "DataCloneError" });
expect(receipts).toEqual([]);
expect(await storage.entry(first, context)).toBeUndefined();
expect(await storage.entry(second, context)).toBeUndefined();
}));
});

287
packages/durable/test/jsonl-storage.test.ts
View file

@ -10,7 +10,9 @@ import { NodeExecutionEnv } from "../src/env/node.ts";
import { idFromNumber } from "../src/ids.ts";
import { JsonlStorage } from "../src/storage/jsonl/index.ts";
import { openNodeJsonlStorage } from "../src/storage/jsonl/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import type {
CommitResult,
ConversationId,
DocumentCreate,
DocumentId,
@ -24,9 +26,6 @@ import type {
} from "../src/types.ts";
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
const context = BACKGROUND_CONTEXT;
type StoredTask = TaskRecord<JsonValue, JsonValue, JsonValue>;
const tempDirectories = new Set<string>();
@ -69,10 +68,14 @@ class ReopeningStorage implements Storage {
this.directory = directory;
}
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context): Promise<Seq> {
if (this.closed) return this.current.commit(writes, adopt, commitContext);
async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
commitContext: Context,
): Promise<CommitResult> {
if (this.closed) return this.current.commit(batches, receive, commitContext);
try {
return await this.current.commit(writes, adopt, commitContext);
return await this.current.commit(batches, receive, commitContext);
} finally {
await this.current.close(context);
this.current = await JsonlStorage.open(this.directory, new NodeExecutionEnv({ cwd: this.directory }), context);
@ -163,7 +166,7 @@ function terminalTask(id: TaskId<JsonValue>): StoredTask {
}
async function createRoot(storage: Storage): Promise<void> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
}
function sessionDocument(id: DocumentId, kind = "test.document"): DocumentCreate {
@ -332,8 +335,9 @@ describe("Pico JsonlStorage publication and recovery", () => {
await createRoot(storage);
const taskId = await storage.mintId<TaskId<JsonValue>>();
const documentId = await storage.mintId<DocumentId>();
await storage.commit([{ type: "task", value: pendingTask(taskId) }], adopt, context);
await storage.commit(
await commitWrites(storage, [{ type: "task", value: pendingTask(taskId) }], context);
await commitWrites(
storage,
[
{
type: "document.create",
@ -341,15 +345,15 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[{ type: "document.change", id: documentId, content: { kind: "delta", version: 1, ops: [] } }],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -357,14 +361,13 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 1 } },
},
],
adopt,
context,
);
expect(await readLines(join(directory, `task-${taskId}.jsonl`))).toHaveLength(1);
expect(await readLines(join(directory, `doc-${documentId}.jsonl`))).toHaveLength(1);
await storage.commit([{ type: "document.retire", id: documentId }], adopt, context);
await storage.commit([{ type: "task", value: terminalTask(taskId) }], adopt, context);
await commitWrites(storage, [{ type: "document.retire", id: documentId }], context);
await commitWrites(storage, [{ type: "task", value: terminalTask(taskId) }], context);
expect(await readLines(join(directory, "main.jsonl"))).toHaveLength(7);
expect(await fileExists(join(directory, `task-${taskId}.jsonl`))).toBe(false);
@ -380,12 +383,12 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory);
await createRoot(storage);
const taskId = await storage.mintId<TaskId<JsonValue>>();
await storage.commit(
await commitWrites(
storage,
[
{ type: "task", value: pendingTask(taskId, "first") },
{ type: "task", value: pendingTask(taskId, "second") },
],
adopt,
context,
);
expect(await readLines(join(directory, `task-${taskId}.jsonl`))).toHaveLength(2);
@ -402,12 +405,12 @@ describe("Pico JsonlStorage publication and recovery", () => {
const taskId = await storage.mintId<TaskId<JsonValue>>();
env.fail({ operation: "append", call: 1, mode: "short" });
await expect(
storage.commit(
commitWrites(
storage,
[
{ type: "task", value: pendingTask(taskId, "first") },
{ type: "task", value: pendingTask(taskId, `second-${"x".repeat(512)}`) },
],
adopt,
context,
),
).rejects.toThrow("poisoned");
@ -420,6 +423,46 @@ describe("Pico JsonlStorage publication and recovery", () => {
expect((await stat(join(directory, `task-${taskId}.jsonl`))).size).toBe(0);
});
it("poisons the storage when an append fails after its batches applied, before a waiting read goes on (review)", async () => {
const directory = await tempDirectory();
const gate = Promise.withResolvers<void>();
class FailingJoin extends NodeExecutionEnv {
fail = false;
override async joinPath(parts: string[], callContext: Context): Promise<Result<string, FileError>> {
if (!this.fail) return super.joinPath(parts, callContext);
await gate.promise;
return err(new FileError("unknown", "injected path resolution failure"));
}
}
const fs = new FailingJoin({ cwd: directory });
const storage = await JsonlStorage.open(directory, fs, context);
fs.fail = true;
const conversationId = idFromNumber<ConversationId>(100);
const committing = storage.commit(
[
[{ type: "conversation", value: { id: conversationId } }],
[
{
type: "document.create",
record: { id: idFromNumber<DocumentId>(101), kind: "state", scope: { kind: "session" } },
content: { kind: "base", version: 1, value: {} },
},
],
],
() => {},
context,
);
// Started while the batches are in memory and the append waits.
const read = storage.conversation(conversationId, context);
gate.resolve();
await expect(committing).rejects.toThrow("poisoned");
await expect(read).rejects.toThrow("poisoned");
await storage.close(context);
const reopened = await openStorage(directory);
expect(await reopened.conversation(conversationId, context)).toBeUndefined();
await reopened.close(context);
});
it("serializes the complete candidate before I/O and leaves preparation failures usable", async () => {
const directory = await tempDirectory();
const env = new InstrumentedEnv({ cwd: directory });
@ -428,7 +471,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
env.clear();
const id = await storage.mintId<EntryId>();
await expect(
storage.commit(
commitWrites(
storage,
[
{
type: "entry",
@ -440,16 +484,15 @@ describe("Pico JsonlStorage publication and recovery", () => {
},
},
],
adopt,
context,
),
).rejects.toThrow();
expect(env.operations).toEqual([]);
expect(await storage.entry(id, context)).toBeUndefined();
expect(
await storage.commit(
await commitWrites(
storage,
[{ type: "entry", value: { id, conversationId: ROOT_CONVERSATION_ID, kind: "good" } }],
adopt,
context,
),
).toBe(2);
@ -474,7 +517,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const secondId = await storage.mintId<DocumentId>();
env.fail(failure);
await expect(
storage.commit(
commitWrites(
storage,
[
{
type: "document.create",
@ -487,7 +531,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { text: "β" } },
},
],
adopt,
context,
),
).rejects.toThrow("poisoned");
@ -519,7 +562,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const secondId = await storage.mintId<DocumentId>();
env.fail(failure);
await expect(
storage.commit(
commitWrites(
storage,
[
{
type: "document.create",
@ -532,7 +576,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
),
).rejects.toThrow("poisoned");
@ -551,7 +594,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const firstId = await storage.mintId<DocumentId>();
const secondId = await storage.mintId<DocumentId>();
env.clear();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -564,7 +608,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
);
const expected = [
@ -576,7 +619,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
expect(env.operations).toEqual(expected);
env.clear();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -584,7 +628,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { checkpoint: true } },
},
],
adopt,
context,
);
expect(env.operations).toEqual([
@ -598,7 +641,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
]);
env.clear();
await storage.commit(
await commitWrites(
storage,
[
{
type: "entry",
@ -609,15 +653,14 @@ describe("Pico JsonlStorage publication and recovery", () => {
},
},
],
adopt,
context,
);
expect(env.operations).toEqual(["append:main.jsonl"]);
const taskId = await storage.mintId<TaskId<JsonValue>>();
await storage.commit([{ type: "task", value: pendingTask(taskId) }], adopt, context);
await commitWrites(storage, [{ type: "task", value: pendingTask(taskId) }], context);
env.clear();
await storage.commit([{ type: "task", value: terminalTask(taskId) }], adopt, context);
await commitWrites(storage, [{ type: "task", value: terminalTask(taskId) }], context);
expect(env.operations).toEqual([
"append:main.jsonl",
...(fsync ? ["flush:main.jsonl"] : []),
@ -639,7 +682,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory, env);
await createRoot(storage);
const id = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -647,10 +691,10 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -658,13 +702,13 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 1]] },
},
],
adopt,
context,
);
env.fail(failure);
await expect(
storage.commit(
commitWrites(
storage,
[
{
type: "document.change",
@ -672,7 +716,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
),
).resolves.toBe(4);
@ -694,7 +737,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
await createRoot(storage);
const taskId = await storage.mintId<TaskId<JsonValue>>();
const id = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{ type: "task", value: pendingTask(taskId) },
{
@ -703,11 +747,10 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 1 } },
},
],
adopt,
context,
);
env.fail({ operation: "remove", call: 1, mode });
await expect(storage.commit([{ type: "document.retire", id }], adopt, context)).resolves.toBe(3);
await expect(commitWrites(storage, [{ type: "document.retire", id }], context)).resolves.toBe(3);
expect(await storage.document(id, "current", context)).toBeUndefined();
await storage.close(context);
@ -726,7 +769,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory, env, { fsync: true });
await createRoot(storage);
const id = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -734,12 +778,12 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
env.fail({ operation: "flush", call: 2, mode });
await expect(
storage.commit(
commitWrites(
storage,
[
{
type: "document.change",
@ -747,7 +791,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
),
).resolves.toBe(3);
@ -782,7 +825,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory, env, { fsync: true });
await createRoot(storage);
const id = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -790,12 +834,12 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
env.fail({ operation: "flush", call: 3, mode });
await expect(
storage.commit(
commitWrites(
storage,
[
{
type: "document.change",
@ -803,12 +847,12 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
),
).resolves.toBe(3);
env.clear();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -816,7 +860,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 3]] },
},
],
adopt,
context,
);
await storage.close(context);
@ -834,9 +877,9 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory, env);
await createRoot(storage);
const id = await storage.mintId<TaskId<JsonValue>>();
await storage.commit([{ type: "task", value: pendingTask(id) }], adopt, context);
await commitWrites(storage, [{ type: "task", value: pendingTask(id) }], context);
env.fail({ operation: "remove", call: 1, mode });
await expect(storage.commit([{ type: "task", value: terminalTask(id) }], adopt, context)).resolves.toBe(3);
await expect(commitWrites(storage, [{ type: "task", value: terminalTask(id) }], context)).resolves.toBe(3);
expect(await storage.task(id, context)).toEqual(terminalTask(id));
await storage.close(context);
@ -852,7 +895,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory);
await createRoot(storage);
const id = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -860,10 +904,10 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -871,10 +915,10 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 10 } },
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -882,7 +926,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 11]] },
},
],
adopt,
context,
);
expect(await readLines(join(directory, `doc-${id}.jsonl`))).toHaveLength(2);
@ -902,12 +945,13 @@ describe("Pico JsonlStorage publication and recovery", () => {
history: "rewindable" as const,
fork: "asOf" as const,
};
const createdAt = await storage.commit(
const createdAt = await commitWrites(
storage,
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { count: 0 } } }],
adopt,
context,
);
const changedAt = await storage.commit(
const changedAt = await commitWrites(
storage,
[
{
type: "document.change",
@ -915,10 +959,10 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 1]] },
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -926,10 +970,9 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
);
await storage.commit([{ type: "document.retire", id }], adopt, context);
await commitWrites(storage, [{ type: "document.retire", id }], context);
expect(await readLines(join(directory, `doc-${id}.jsonl`))).toHaveLength(3);
const reopened = await openStorage(directory);
@ -947,7 +990,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const sessionId = await storage.mintId<DocumentId>();
const latestId = await storage.mintId<DocumentId>();
const taskDocumentId = await storage.mintId<DocumentId>();
const createdAt = await storage.commit(
const createdAt = await commitWrites(
storage,
[
{ type: "task", value: pendingTask(taskId) },
{
@ -972,17 +1016,16 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
);
const retiredAt = await storage.commit(
const retiredAt = await commitWrites(
storage,
[
{ type: "document.retire", id: sessionId },
{ type: "document.retire", id: latestId },
{ type: "document.retire", id: taskDocumentId },
{ type: "task", value: terminalTask(taskId) },
],
adopt,
context,
);
for (const file of [
@ -1012,7 +1055,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory);
await createRoot(storage);
const documentId = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -1020,7 +1064,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { text: "kept" } },
},
],
adopt,
context,
);
const sidecarPath = join(directory, `doc-${documentId}.jsonl`);
@ -1035,9 +1078,9 @@ describe("Pico JsonlStorage publication and recovery", () => {
expect((await stat(sidecarPath)).size).toBe(sidecarSize);
expect((await stat(mainPath)).size).toBe(mainSize);
expect(
await reopened.commit(
await commitWrites(
reopened,
[{ type: "document.change", id: documentId, content: { kind: "delta", version: 1, ops: [] } }],
adopt,
context,
),
).toBe(3);
@ -1048,8 +1091,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory);
await createRoot(storage);
const taskId = await storage.mintId<TaskId<JsonValue>>();
await storage.commit([{ type: "task", value: pendingTask(taskId) }], adopt, context);
await storage.commit([{ type: "task", value: terminalTask(taskId) }], adopt, context);
await commitWrites(storage, [{ type: "task", value: pendingTask(taskId) }], context);
await commitWrites(storage, [{ type: "task", value: terminalTask(taskId) }], context);
const sidecarPath = join(directory, `task-${taskId}.jsonl`);
expect(await fileExists(sidecarPath)).toBe(false);
await writeFile(
@ -1067,7 +1110,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const reopened = await openStorage(directory);
expect(await fileExists(sidecarPath)).toBe(false);
expect(await reopened.task(taskId, context)).toEqual(terminalTask(taskId));
expect(await reopened.commit([], adopt, context)).toBe(4);
const next = await reopened.mintId<ConversationId>();
expect(await commitWrites(reopened, [{ type: "conversation", value: { id: next } }], context)).toBe(4);
});
it("fails open when confirmed sidecar data is missing", async () => {
@ -1075,7 +1119,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory);
await createRoot(storage);
const documentId = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -1083,7 +1128,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
);
await writeFile(join(directory, `doc-${documentId}.jsonl`), "");
@ -1097,7 +1141,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory);
await createRoot(storage);
const documentId = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -1105,10 +1150,10 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -1116,7 +1161,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 1]] },
},
],
adopt,
context,
);
const path = join(directory, `doc-${documentId}.jsonl`);
@ -1151,7 +1195,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory);
await createRoot(storage);
const documentId = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -1159,7 +1204,6 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
);
const path = join(directory, `doc-${documentId}.jsonl`);
@ -1192,3 +1236,68 @@ describe("Pico JsonlStorage publication and recovery", () => {
).rejects.toThrow("Malformed complete doc-99.jsonl");
});
});
describe("JsonlStorage batches", () => {
it("appends a batch's markers in one write and recovers a shorter prefix from a torn one", async () => {
const directory = await tempDirectory();
const storage = await openStorage(directory);
await createRoot(storage);
const documentId = await storage.mintId<DocumentId>();
await commitWrites(
storage,
[
{
type: "document.create",
record: sessionDocument(documentId),
content: { kind: "base", version: 1, value: { n: 0 } },
},
],
context,
);
const mainPath = join(directory, "main.jsonl");
const before = (await readFile(mainPath, "utf8")).length;
const entries = [
await storage.mintId<EntryId>(),
await storage.mintId<EntryId>(),
await storage.mintId<EntryId>(),
];
const batches: StorageWrite[][] = entries.map((id, index) => [
{ type: "entry", value: { id, conversationId: ROOT_CONVERSATION_ID, kind: "note" } },
{
type: "document.change",
id: documentId,
content: { kind: "delta", version: 1, ops: [["s", ["n"], index + 1]] },
},
]);
// Reads wait for the append: a read started while it runs sees every batch durable.
const result = storage.commit(batches, () => {}, context);
const read = storage.entry(entries[2]!, context);
expect(await result).toEqual({ stored: 3 });
expect(await read).toBeDefined();
const main = await readFile(mainPath, "utf8");
const markers = main
.slice(before)
.split("\n")
.filter((line) => line.length > 0);
expect(markers).toHaveLength(3);
await storage.close(context);
openStorages.delete(storage);
// Torn inside the last marker: the two complete ones survive, and the third's sidecar record is unconfirmed.
const tornAt = main.length - Math.floor(markers[2]!.length / 2);
await writeFile(mainPath, main.slice(0, tornAt));
const reopened = await openStorage(directory);
expect(await reopened.entry(entries[0]!, context)).toBeDefined();
expect(await reopened.entry(entries[1]!, context)).toBeDefined();
expect(await reopened.entry(entries[2]!, context)).toBeUndefined();
expect((await reopened.document(documentId, "current", context))?.value).toEqual({ n: 2 });
const seq = await commitWrites(
reopened,
[{ type: "document.change", id: documentId, content: { kind: "delta", version: 1, ops: [["s", ["n"], 9]] } }],
context,
);
expect(seq).toBe(5);
const again = await openStorage(directory);
expect((await again.document(documentId, "current", context))?.value).toEqual({ n: 9 });
});
});

43
packages/durable/test/memory-storage.test.ts
View file

@ -3,13 +3,14 @@ import { registerStorageConformance } from "@earendil-works/pi-durable/testing";
import { describe, expect, it } from "vitest";
import { idFromNumber, seqFromNumber } from "../src/ids.ts";
import { MemoryStorage } from "../src/storage/memory.ts";
import { type EntryId, ROOT_CONVERSATION_ID } from "../src/types.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { type EntryId, ROOT_CONVERSATION_ID, type Seq, type StorageWrite } from "../src/types.ts";
registerStorageConformance({ describe, expect, it }, "MemoryStorage", (use) => use(new MemoryStorage()));
it("does not expose retained state through a prepared commit", async () => {
const storage = new MemoryStorage();
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], () => {}, BACKGROUND_CONTEXT);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT);
const entryId = idFromNumber<EntryId>(2);
const prepared = storage.prepareCommit([
{
@ -30,3 +31,41 @@ it("does not expose retained state through a prepared commit", async () => {
expect(prepared.apply()).toBe(seqFromNumber(2));
expect((await storage.entry(entryId, BACKGROUND_CONTEXT))?.entry.data).toEqual({ nested: [1] });
});
it("delivers the receipts of the batches applied before one whose apply throws, then rejects", async () => {
const failure = new Error("apply failed");
class FailingApply extends MemoryStorage {
failAt: Seq | undefined;
override prepareCommit(writes: readonly StorageWrite[], seq?: Seq) {
const prepared = super.prepareCommit(writes, seq);
if (prepared.seq !== this.failAt) return prepared;
return {
...prepared,
apply: (): Seq => {
throw failure;
},
};
}
}
const storage = new FailingApply();
const root = await commitWrites(
storage,
[{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }],
BACKGROUND_CONTEXT,
);
const [first, second] = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
storage.failAt = seqFromNumber(root + 2);
const receipts: [number, Seq][] = [];
await expect(
storage.commit(
[first, second].map((id) => [
{ type: "entry", value: { id, conversationId: ROOT_CONVERSATION_ID, kind: "m" } },
]),
(index, seq) => receipts.push([index, seq]),
BACKGROUND_CONTEXT,
),
).rejects.toBe(failure);
expect(receipts).toEqual([[0, root + 1]]);
expect((await storage.entry(first, BACKGROUND_CONTEXT))?.commitSeq).toBe(root + 1);
expect(await storage.entry(second, BACKGROUND_CONTEXT)).toBeUndefined();
});

8
packages/durable/test/nested-images-bench.ts
View file

@ -189,12 +189,12 @@ async function runScenario(scenario: Scenario): Promise<Result> {
let commits = 0;
const written = noWrites();
const commit = storage.commit.bind(storage);
storage.commit = (writes, adopt, commitContext) => {
commits++;
storage.commit = (batches, receive, commitContext) => {
commits += batches.length;
// SQLite commits synchronously, which can starve the timer; sample here too.
sample();
for (const write of writes) written[write.type] += JSON.stringify(write).length;
return commit(writes, adopt, commitContext);
for (const write of batches.flat()) written[write.type] += JSON.stringify(write).length;
return commit(batches, receive, commitContext);
};
const harness = await Harness.open(storage, { models, registry }, context);
const root = await harness.root(context, { agent: { model: { provider: "faux", modelId: "faux-1" } } });

8
packages/durable/test/nested-tools-bench.ts
View file

@ -133,10 +133,10 @@ async function runScenario(scenario: Scenario): Promise<{ perRun: Run[]; closedB
let commits = 0;
const written = noWrites();
const commit = storage.commit.bind(storage);
storage.commit = (writes, adopt, commitContext) => {
commits++;
for (const write of writes) written[write.type] += JSON.stringify(write).length;
return commit(writes, adopt, commitContext);
storage.commit = (batches, receive, commitContext) => {
commits += batches.length;
for (const write of batches.flat()) written[write.type] += JSON.stringify(write).length;
return commit(batches, receive, commitContext);
};
const harness = await Harness.open(storage, { models, registry }, context);
const root = await harness.root(context, { agent: { model: { provider: "faux", modelId: "faux-1" } } });

6
packages/durable/test/scheduler-fanout-bench.ts
View file

@ -41,9 +41,9 @@ async function run(calls: number): Promise<{ ms: number; commits: number }> {
const storage = new MemoryStorage();
let commits = 0;
const commit = storage.commit.bind(storage);
storage.commit = (writes, adopt, commitContext) => {
commits++;
return commit(writes, adopt, commitContext);
storage.commit = (batches, receive, commitContext) => {
commits += batches.length;
return commit(batches, receive, commitContext);
};
const harness = await Harness.open(storage, { models, registry }, context);
const root = await harness.root(context, { agent: { model: { provider: "faux", modelId: "faux-1" } } });

68
packages/durable/test/session-adoption.test.ts
View file

@ -4,6 +4,7 @@ import { basename, join } from "node:path";
import type { Context } from "@earendil-works/chord";
import {
type CommitPublication,
type CommitResult,
defineDoc,
MemoryStorage,
type Seq,
@ -41,22 +42,26 @@ class FaultyStorage extends ControlledStorage {
fault: "reject_after_adopt" | "skip_adopt" | undefined;
readonly order: string[] = [];
override async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, callContext: Context) {
override async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
callContext: Context,
): Promise<CommitResult> {
const fault = this.fault;
this.fault = undefined;
if (fault === "skip_adopt") return super.commit(writes, () => {}, callContext);
const seq = await super.commit(
writes,
(committed) => {
if (fault === "skip_adopt") return super.commit(batches, () => {}, callContext);
const result = await super.commit(
batches,
(index, committed) => {
this.order.push("adopt");
adopt(committed);
receive(index, committed);
this.order.push("adopted");
},
callContext,
);
this.order.push("resolve");
if (fault === "reject_after_adopt") throw new Error("reclaim failed");
return seq;
return result;
}
}
@ -75,8 +80,8 @@ afterEach(() => {
vi.restoreAllMocks();
});
describe("Session adoption inside Storage.commit()", () => {
it("adopts synchronously inside the commit, before Storage resolves", async () => {
describe("Session adoption after Storage stored a commit", () => {
it("takes the durability receipt inside the commit, and adopts once Storage resolved", async () => {
const { storage, session } = open();
const adopt = vi.spyOn(Transaction.prototype, "adopt");
await bump(session);
@ -128,8 +133,8 @@ describe("Session adoption inside Storage.commit()", () => {
});
});
describe("publication at adoption", () => {
it("runs its own observers inside adopt and host listeners one microtask later, outside it, before Storage resolves", async () => {
describe("publication after storage", () => {
it("runs its own observers once Storage resolved, and host listeners one microtask later", async () => {
const { storage, session } = open();
session.followCommits(() => {
storage.order.push(`observed adopting=${storage.adopting}`);
@ -141,12 +146,12 @@ describe("publication at adoption", () => {
end: () => {},
});
await bump(session);
expect(storage.order).toEqual(["adopt", "observed adopting=true", "adopted", "host adopting=false", "resolve"]);
expect(storage.order).toEqual(["adopt", "adopted", "resolve", "observed adopting=false", "host adopting=false"]);
await session.close(context);
});
it("calls the host listeners registered when its dispatch runs, and none for a commit adopted without one", async () => {
const { storage, session } = open();
it("calls the host listeners registered when its dispatch runs, and none for a commit published without one", async () => {
const { session } = open();
const calls: string[] = [];
const removed = session.observeCommits({
advance: () => {
@ -160,7 +165,11 @@ describe("publication at adoption", () => {
},
end: () => {},
});
storage.afterAdopt = () => {
// During the publication, after the dispatch was queued.
let changed = false;
session.followCommits(() => {
if (changed) return;
changed = true;
removed.detach();
session.observeCommits({
advance: () => {
@ -168,20 +177,23 @@ describe("publication at adoption", () => {
},
end: () => {},
});
};
});
await bump(session);
expect(calls).toEqual(["kept", "added"]);
// A Session with no host listener at adoption queues no dispatch, so one added before Storage resolves misses it.
// A Session with no host listener at the publication queues no dispatch, so one added during it misses it.
const bare = open();
const late: Seq[] = [];
bare.storage.afterAdopt = () => {
let added = false;
bare.session.followCommits(() => {
if (added) return;
added = true;
bare.session.observeCommits({
advance: (publication) => {
late.push(publication.seq);
},
end: () => {},
});
};
});
await bump(bare.session);
expect(late).toEqual([]);
await bump(bare.session);
@ -210,14 +222,14 @@ describe("publication at adoption", () => {
storage.order.push("close");
});
await expect(bump(session)).resolves.toBeUndefined();
// `adopted` and `resolve`: adopt returned to Storage without throwing, and the Session failed only after that.
// Published once Storage resolved; the Session failed only after the hosts got the commit.
expect(storage.order).toEqual([
"adopt",
"adopted",
"resolve",
"observed",
"observed after",
"adopted",
"host failed=false",
"resolve",
"close",
]);
expect(await session.closed).toEqual({ kind: "failed", error: bug });
@ -243,7 +255,7 @@ describe("publication at adoption", () => {
storage.order.length = 0;
storage.fault = "reject_after_adopt";
await expect(bump(session)).rejects.toThrow("reclaim failed");
expect(storage.order).toEqual(["adopt", "observed", "adopted", "host", "resolve"]);
expect(storage.order).toEqual(["adopt", "adopted", "resolve", "observed", "host"]);
expect(stored).toHaveLength(1);
expect(await session.closed).toMatchObject({ kind: "failed" });
const reopened = new SessionImpl(storage.reopen());
@ -283,7 +295,7 @@ describe("publication at adoption", () => {
await session.commit(async (tx) => {
(await tx.doc(NoteDoc, conversationId)).b = 2;
}, context);
expect(storage.order).toEqual(["observed", "adopt", "observed", "adopted", "host", "resolve"]);
expect(storage.order).toEqual(["observed", "adopt", "adopted", "resolve", "observed", "host"]);
expect(documentValues(observed[1]!).map((change) => change.ops)).toEqual([[["s", ["b"], 2]]]);
expect(stored).toHaveLength(1);
expect(documentValues(stored[0]!).map((change) => change.ops)).toEqual([
@ -296,7 +308,7 @@ describe("publication at adoption", () => {
await session.close(context);
});
it("publishes a store of held changes to host listeners one microtask after its adopt", async () => {
it("publishes a store of held changes to host listeners once Storage resolved", async () => {
const { storage, session } = open();
const conversationId = await createConversation(session);
await session.commit(async (tx) => {
@ -317,11 +329,11 @@ describe("publication at adoption", () => {
);
storage.order.length = 0;
await session.unloadDocuments();
expect(storage.order).toEqual(["adopt", "adopted", "host adopting=false", "resolve"]);
expect(storage.order).toEqual(["adopt", "adopted", "resolve", "host adopting=false"]);
await session.close(context);
});
it("publishes a JSONL commit before the storage reclaims its sidecar and resolves", async () => {
it("publishes a JSONL commit once the storage reclaimed its sidecar and resolved", async () => {
/** Node environment that records each sidecar rename, the last step of a reclaim. */
class RenameLog extends NodeExecutionEnv {
readonly order: string[] = [];
@ -360,7 +372,7 @@ describe("publication at adoption", () => {
});
await count();
env.order.push("resolved");
expect(env.order).toEqual(["host", expect.stringMatching(/^rename doc-\d+\.jsonl$/), "resolved"]);
expect(env.order).toEqual([expect.stringMatching(/^rename doc-\d+\.jsonl$/), "host", "resolved"]);
} finally {
await session.close(context);
await rm(directory, { recursive: true, force: true });

183
packages/durable/test/session-cached-storage.test.ts Normal file
View file

@ -0,0 +1,183 @@
import type { JsonValue } from "@earendil-works/chord";
import { describe, expect, it } from "vitest";
import { idFromNumber } from "../src/ids.ts";
import { CachedStorage } from "../src/session/cached-storage.ts";
import { MemoryStorage } from "../src/storage/memory.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import type { ConversationId, StorageWrite, TaskId, TaskRecord } from "../src/types.ts";
import { context } from "./session-support.ts";
function task(id: number, status: "pending" | "running" = "pending"): TaskRecord<JsonValue, JsonValue, JsonValue> {
return {
id: idFromNumber<TaskId<JsonValue>>(id),
conversationId: idFromNumber<ConversationId>(1),
kind: "test.task",
version: 1,
input: null,
background: false,
abortRequested: false,
state: { status, checkpoint: { phase: "run" } },
};
}
class CountingStorage extends MemoryStorage {
taskReads = 0;
override task(...args: Parameters<MemoryStorage["task"]>) {
this.taskReads++;
return super.task(...args);
}
}
/** Stores `writes` as one batch, then publishes it, as the Session does. */
async function commit(storage: CachedStorage, writes: StorageWrite[]): Promise<void> {
await commitWrites(storage, writes, context);
storage.published(writes);
}
describe("cached storage", () => {
it("serves a written record without a read and follows later writes", async () => {
const backend = new CountingStorage();
const storage = new CachedStorage(backend);
await commit(storage, [{ type: "conversation", value: { id: idFromNumber<ConversationId>(1) } }]);
await commit(storage, [{ type: "task", value: task(2) }]);
expect((await storage.task(idFromNumber<TaskId>(2), context))?.state.status).toBe("pending");
await commit(storage, [{ type: "task", value: task(2, "running") }]);
expect((await storage.task(idFromNumber<TaskId>(2), context))?.state.status).toBe("running");
expect(backend.taskReads).toBe(0);
});
it("keeps a read only when no commit overlapped it", async () => {
const backend = new CountingStorage();
await commitWrites(
backend,
[
{ type: "conversation", value: { id: idFromNumber<ConversationId>(1) } },
{ type: "task", value: task(2) },
],
context,
);
const storage = new CachedStorage(backend);
const overlapped = storage.task(idFromNumber<TaskId>(2), context);
const written = commit(storage, [{ type: "task", value: task(2, "running") }]);
await Promise.all([overlapped, written]);
expect((await storage.task(idFromNumber<TaskId>(2), context))?.state.status).toBe("running");
await storage.task(idFromNumber<TaskId>(2), context);
expect(backend.taskReads).toBe(1);
});
it("keeps a bounded number of ended tasks, and every live one until it ends", async () => {
const backend = new CountingStorage();
const storage = new CachedStorage(backend);
await commit(storage, [{ type: "conversation", value: { id: idFromNumber<ConversationId>(1) } }]);
for (let id = 2; id < 600; id++) await commit(storage, [{ type: "task", value: task(id) }]);
// More live tasks than the ended table holds: each one's next change finds it without a read.
expect(storage.sizes).toMatchObject({ tasks: 0, liveTasks: 598 });
for (let id = 2; id < 600; id++) await storage.task(idFromNumber<TaskId>(id), context);
expect(backend.taskReads).toBe(0);
for (let id = 2; id < 600; id++) {
const ended = { ...task(id), state: { status: "terminal", outcome: { status: "completed", result: null } } };
await commit(storage, [{ type: "task", value: ended as TaskRecord<JsonValue, JsonValue, JsonValue> }]);
}
expect(storage.sizes).toMatchObject({ tasks: 256, liveTasks: 0 });
});
it("applies a batch stored with commit when it is published, not at its receipt", async () => {
const backend = new CountingStorage();
const storage = new CachedStorage(backend);
await commit(storage, [
{ type: "conversation", value: { id: idFromNumber<ConversationId>(1) } },
{ type: "task", value: task(2) },
]);
const batch: StorageWrite[] = [{ type: "task", value: task(2, "running") }];
const receipts: number[] = [];
expect(await storage.commit([batch], (index) => receipts.push(index), context)).toEqual({ stored: 1 });
expect(receipts).toEqual([0]);
// Stored, not yet published: the cache still shows what was published.
expect((await storage.task(idFromNumber<TaskId>(2), context))?.state.status).toBe("pending");
storage.published(batch);
expect((await storage.task(idFromNumber<TaskId>(2), context))?.state.status).toBe("running");
expect(backend.taskReads).toBe(0);
});
it("drops a read a publication of its record overlapped, and keeps versions only for reads in flight", async () => {
const backend = new CountingStorage();
await commitWrites(
backend,
[
{ type: "conversation", value: { id: idFromNumber<ConversationId>(1) } },
{ type: "task", value: task(2) },
{ type: "task", value: task(3) },
],
context,
);
const storage = new CachedStorage(backend);
const overlapped = storage.task(idFromNumber<TaskId>(2), context);
const untouched = storage.task(idFromNumber<TaskId>(3), context);
expect(storage.readsInFlight).toBe(2);
// A publication of another record does not drop the read of task 3.
storage.published([{ type: "task", value: task(2, "running") }]);
await Promise.all([overlapped, untouched]);
expect(storage.readsInFlight).toBe(0);
expect((await storage.task(idFromNumber<TaskId>(2), context))?.state.status).toBe("running");
expect((await storage.task(idFromNumber<TaskId>(3), context))?.state.status).toBe("pending");
expect(backend.taskReads).toBe(2);
});
it("keeps the ended tasks a task waits on until it moved on from its wait", async () => {
const backend = new CountingStorage();
const storage = new CachedStorage(backend);
await commit(storage, [{ type: "conversation", value: { id: idFromNumber<ConversationId>(1) } }]);
const ended = (id: number) =>
({
...task(id),
state: { status: "terminal", outcome: { status: "completed", result: id } },
}) as TaskRecord<JsonValue, JsonValue, JsonValue>;
const waiting = (state: TaskRecord<JsonValue, JsonValue, JsonValue>["state"]) =>
({ ...task(2), state }) as TaskRecord<JsonValue, JsonValue, JsonValue>;
const on = [3, 4].map((id) => idFromNumber<TaskId>(id));
await commit(storage, [
{
type: "task",
value: waiting({ status: "waiting", checkpoint: { phase: "read" }, on, policy: "allSettled" }),
},
]);
await commit(storage, [
{ type: "task", value: ended(3) },
{ type: "task", value: ended(4) },
]);
// A burst of other ended tasks fills the bounded table.
for (let id = 10; id < 300; id++) await commit(storage, [{ type: "task", value: ended(id) }]);
expect(storage.sizes).toMatchObject({ awaited: 2, waits: 1 });
// Resumed: still held for the phase that reads the outcomes.
await commit(storage, [{ type: "task", value: waiting({ status: "running", checkpoint: { phase: "read" } }) }]);
for (const id of on) expect((await storage.task(id, context))?.state.status).toBe("terminal");
expect(backend.taskReads).toBe(0);
expect(storage.sizes).toMatchObject({ awaited: 2, waits: 1 });
// Its next change ends the hold: the records go back to the bounded table, and no wait is left.
await commit(storage, [{ type: "task", value: waiting({ status: "running", checkpoint: { phase: "next" } }) }]);
expect(storage.sizes).toMatchObject({ awaited: 0, waits: 0, tasks: 256 });
// A wait that ends with its task leaves nothing either.
await commit(storage, [
{
type: "task",
value: waiting({ status: "waiting", checkpoint: { phase: "read" }, on, policy: "allSettled" }),
},
]);
expect(storage.sizes).toMatchObject({ awaited: 2, waits: 1 });
await commit(storage, [{ type: "task", value: ended(2) }]);
expect(storage.sizes).toMatchObject({ awaited: 0, waits: 0 });
});
it("keeps the written record itself, frozen: no copy", async () => {
const storage = new CachedStorage(new MemoryStorage());
const conversation = { id: idFromNumber<ConversationId>(1) };
const record = task(2);
storage.published([
{ type: "conversation", value: conversation },
{ type: "task", value: record },
]);
expect(await storage.task(idFromNumber<TaskId>(2), context)).toBe(record);
expect(await storage.conversation(idFromNumber<ConversationId>(1), context)).toBe(conversation);
expect(Object.isFrozen(record.state)).toBe(true);
});
});

81
packages/durable/test/session-conversation-tails.test.ts Normal file
View file

@ -0,0 +1,81 @@
import { describe, expect, it } from "vitest";
import { idFromNumber } from "../src/ids.ts";
import { ConversationTails, type HeadMarker } from "../src/session/conversation-tails.ts";
import type { ConversationId, EntryId, EntryRecord, StorageWrite } from "../src/types.ts";
const conversation = (id: number) => idFromNumber<ConversationId>(id);
const entryId = (id: number) => idFromNumber<EntryId>(id);
function entry(id: number, conversationId: number, head?: number): StorageWrite {
const value: EntryRecord = {
id: entryId(id),
conversationId: conversation(conversationId),
kind: "note",
...(head === undefined ? {} : { head: entryId(head) }),
};
return { type: "entry", value };
}
const never = (): Promise<never> => Promise.reject(new Error("read"));
describe("conversation tails", () => {
it("reads once, then follows the commits", async () => {
const tails = new ConversationTails();
let reads = 0;
const read = async () => {
reads++;
return entryId(5);
};
expect(await tails.latest(conversation(1), read)).toBe(5);
expect(await tails.latest(conversation(1), read)).toBe(5);
expect(reads).toBe(1);
tails.apply([entry(9, 1)]);
expect(await tails.latest(conversation(1), never)).toBe(9);
});
it("knows a new root conversation is empty, and a fork only once read or written", async () => {
const tails = new ConversationTails();
tails.apply([
{ type: "conversation", value: { id: conversation(2) } },
{
type: "conversation",
value: { id: conversation(3), parent: { conversationId: conversation(2), at: entryId(4) } },
},
]);
expect(await tails.latest(conversation(2), never)).toBeUndefined();
expect(await tails.marker(conversation(2), undefined, never)).toBeUndefined();
expect(await tails.latest(conversation(3), async () => entryId(4))).toBe(4);
});
it("answers a cutoff from the newest marker only at or after it", async () => {
const tails = new ConversationTails();
tails.apply([{ type: "conversation", value: { id: conversation(1) } }, entry(10, 1), entry(11, 1, 10)]);
const marker = (await tails.marker(conversation(1), undefined, never)) as HeadMarker;
expect(marker.id).toBe(11);
expect((await tails.marker(conversation(1), entryId(12), never))?.id).toBe(11);
// An older cutoff may have an older marker: read, and not kept.
let reads = 0;
const older = async () => {
reads++;
return undefined;
};
expect(await tails.marker(conversation(1), entryId(10), older)).toBeUndefined();
expect(await tails.marker(conversation(1), entryId(10), older)).toBeUndefined();
expect(reads).toBe(2);
expect((await tails.marker(conversation(1), undefined, never))?.id).toBe(11);
});
it("keeps a bounded number of conversations, fewer with markers", () => {
const tails = new ConversationTails();
for (let id = 1; id <= 5000; id++) tails.apply([entry(2 * id, id), entry(2 * id + 1, id, 2 * id)]);
expect(tails.size).toBe(4096);
expect(tails.markers).toBe(256);
});
it("freezes the markers it keeps", () => {
const tails = new ConversationTails();
const write = entry(2, 1, 1);
tails.apply([write]);
expect(Object.isFrozen((write as { value: object }).value)).toBe(true);
});
});

8
packages/durable/test/session-document-retention.test.ts
View file

@ -1,6 +1,5 @@
// The Session's document cache is bounded by use: unattached documents leave it after the retention period and load
// from Storage again; documents with a watch or a document state never leave it.
import type { Context } from "@earendil-works/chord";
import { fauxAssistantMessage, fauxText, fauxToolCall, Type } from "@earendil-works/pi-ai";
import {
type ConversationId,
@ -10,7 +9,6 @@ import {
type HarnessSettings,
type LiveState,
MemoryStorage,
type Seq,
type StorageWrite,
} from "@earendil-works/pi-durable";
import { describe, expect, it, onTestFinished } from "vitest";
@ -218,9 +216,9 @@ describe("document cache retention", () => {
const conversationId = await createConversation(session);
const writes: StorageWrite[][] = [];
const commit = storage.commit.bind(storage);
storage.commit = (batch: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context) => {
writes.push([...batch]);
return commit(batch, adopt, commitContext);
storage.commit = (batches, receive, commitContext) => {
for (const batch of batches) writes.push([...batch]);
return commit(batches, receive, commitContext);
};
for (let n = 0; n < 5; n++) {
await bump(session, conversationId);

1017
packages/durable/test/session-group-commit.test.ts Normal file
View file

File diff suppressed because it is too large Load diff

8
packages/durable/test/session-observers.test.ts
View file

@ -84,7 +84,7 @@ describe("observeDoc", () => {
await session.close(context);
});
it("starts at the committed value and advances inside adopt with each stored commit's ops", async () => {
it("starts at the committed value and advances at the publication of each stored commit with its ops", async () => {
const { storage, session } = open();
await add(session);
const { calls, observer } = recorder(storage);
@ -95,7 +95,7 @@ describe("observeDoc", () => {
expect(calls).toEqual([
{
advance: { kind: "value", value: { count: 3 }, ops: [["s", ["count"], 3]], retired: false },
adopting: true,
adopting: false,
},
]);
observation.detach();
@ -146,10 +146,10 @@ describe("observeDoc", () => {
(await tx.doc(ConversationCounter, conversationId)).count = 9;
}, context);
expect(calls).toEqual([
{ advance: { kind: "value", value: observation.value, ops: [], retired: true }, adopting: true },
{ advance: { kind: "value", value: observation.value, ops: [], retired: true }, adopting: false },
{
advance: { kind: "value", value: { count: 9 }, ops: [["r", { count: 9 }]], retired: false },
adopting: true,
adopting: false,
},
]);
expect((calls[0] as { advance: Extract<WatchCall<Counter>, { kind: "value" }> }).advance.value).toBe(

63
packages/durable/test/session-support.ts
View file

@ -3,6 +3,7 @@ import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
import {
type CommitChange,
type CommitPublication,
type CommitResult,
type ConversationId,
type DocumentAddress,
type DocumentCommitChange,
@ -13,6 +14,7 @@ import {
MemoryStorage,
type Seq,
type Session,
type Storage,
type StorageWrite,
} from "@earendil-works/pi-durable";
import { type ObservedCommit, SessionImpl } from "../src/session/session.ts";
@ -48,6 +50,24 @@ export type Gate = {
release(): void;
};
/**
* Commits `batches` through `commit` up to the first one `refuses` matches, which it refuses with `error`: what a
* storage does whose own check refuses that batch.
*/
export async function refusingAt(
commit: Storage["commit"],
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
commitContext: Context,
refuses: (writes: readonly StorageWrite[]) => boolean,
error: unknown,
): Promise<CommitResult> {
const at = batches.findIndex(refuses);
if (at < 0) return commit(batches, receive, commitContext);
const result = await commit(batches.slice(0, at), receive, commitContext);
return result.stored < at ? result : { stored: at, refused: error };
}
/** Memory storage with observable commits, held calls, and injected commit failures. */
export class ControlledStorage extends MemoryStorage {
/** Exact borrowed batches admitted by Session. */
@ -56,9 +76,9 @@ export class ControlledStorage extends MemoryStorage {
readonly commits: (readonly StorageWrite[])[] = [];
mintCount = 0;
documentReadCount = 0;
/** Set while `adopt` runs, the storage's synchronous step: code that sees it set runs inside `Storage.commit()`'s adopt. */
/** Set while a receipt runs, the storage's synchronous step: code that sees it set runs inside `commit()`'s `receive`. */
adopting = false;
/** Runs once, in the next commit's synchronous step right after `adopt` returned. */
/** Runs once, in the next commit's synchronous step right after its first receipt returned. */
afterAdopt: (() => void) | undefined;
#commitGate: { gate: Deferred; entered: Deferred } | undefined;
#findGate: { gate: Deferred; entered: Deferred } | undefined;
@ -97,27 +117,33 @@ export class ControlledStorage extends MemoryStorage {
held.gate.resolve();
}
/**
* Records and holds the call, then fails its first batch if a failure is set (a refusal: nothing stored), or commits
* it in memory, `adopting` set while each receipt runs and `afterAdopt` run after the first one.
*/
override async commit(
writes: readonly StorageWrite[],
adopt: (seq: Seq) => void,
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
commitContext: Context,
): Promise<Seq> {
this.admittedCommits.push(writes);
this.commits.push(structuredClone(writes));
): Promise<CommitResult> {
const held = this.#commitGate;
if (held !== undefined) {
held.entered.resolve();
await held.gate.promise;
}
const failure = this.#commitFailure;
for (const writes of failure === undefined ? batches : batches.slice(0, 1)) {
this.admittedCommits.push(writes);
this.commits.push(structuredClone(writes));
}
if (failure !== undefined) {
this.#commitFailure = undefined;
throw failure;
return { stored: 0, refused: failure };
}
const step = (seq: Seq): void => {
const step = (index: number, seq: Seq): void => {
this.adopting = true;
try {
adopt(seq);
receive(index, seq);
} finally {
this.adopting = false;
}
@ -125,7 +151,7 @@ export class ControlledStorage extends MemoryStorage {
this.afterAdopt = undefined;
after?.();
};
return super.commit(writes, step, commitContext);
return super.commit(batches, step, commitContext);
}
override mintId<I extends Id<string>>(): Promise<I> {
@ -164,21 +190,26 @@ export class LateResolvingStorage extends ControlledStorage {
gap: ((writes: readonly StorageWrite[]) => boolean) | undefined;
rejectAfterAdopt: ((writes: readonly StorageWrite[]) => boolean) | undefined;
override async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context) {
override async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
commitContext: Context,
): Promise<CommitResult> {
this.committing = true;
try {
const seq = await super.commit(writes, adopt, commitContext);
const result = await super.commit(batches, receive, commitContext);
const stored = batches.slice(0, result.stored);
await flush();
if (this.gap?.(writes) === true) {
if (stored.some((writes) => this.gap?.(writes) === true)) {
this.gap = undefined;
// What the gap started runs before the commit resolves.
await flush();
}
if (this.rejectAfterAdopt?.(writes) === true) {
if (stored.some((writes) => this.rejectAfterAdopt?.(writes) === true)) {
this.rejectAfterAdopt = undefined;
throw new Error("reclaim failed");
}
return seq;
return result;
} finally {
this.committing = false;
}

9
packages/durable/test/sqlite-cloudflare.test.ts
View file

@ -5,6 +5,7 @@ import { describe, expect, it } from "vitest";
import { durableObjectStorage } from "../../chord/test/sqlite/helpers.ts";
import { idFromNumber } from "../src/ids.ts";
import { DurableObjectSqliteDatabase, openDurableObjectSqliteStorage } from "../src/storage/sqlite/cloudflare.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { type EntryId, ROOT_CONVERSATION_ID } from "../src/types.ts";
registerStorageConformance({ describe, expect, it }, "SqliteStorage on a prefixed Durable Object", async (use) =>
@ -48,20 +49,20 @@ describe("Durable Object SQLite adapter", () => {
]);
const entryId = idFromNumber<EntryId>(2);
await Promise.all([
first.commit(
commitWrites(
first,
[
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
{ type: "entry", value: { id: entryId, conversationId: ROOT_CONVERSATION_ID, kind: "first" } },
],
() => {},
BACKGROUND_CONTEXT,
),
second.commit(
commitWrites(
second,
[
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
{ type: "entry", value: { id: entryId, conversationId: ROOT_CONVERSATION_ID, kind: "second" } },
],
() => {},
BACKGROUND_CONTEXT,
),
]);

40
packages/durable/test/sqlite-facade.test.ts
View file

@ -11,11 +11,9 @@ import { idFromNumber } from "../src/ids.ts";
import type { SqliteDatabase, SqliteExecutor, SqliteValue } from "../src/storage/sqlite/index.ts";
import { SqliteStorage } from "../src/storage/sqlite/index.ts";
import { NodeSqliteDatabase, openNodeSqliteDatabase } from "../src/storage/sqlite/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { type DocumentId, type EntryId, ROOT_CONVERSATION_ID } from "../src/types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
type SettlementMode = "immediate" | "delay" | "reject";
class ControlledSettlementDatabase implements SqliteDatabase {
@ -101,29 +99,29 @@ describe("portable SQLite facade settlement", () => {
it("prepares each storage statement once per connection and reuses it across transactions", async () => {
const connection = new PrepareCountingDatabaseSync(":memory:");
const storage = await SqliteStorage.open(new NodeSqliteDatabase(connection));
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, BACKGROUND_CONTEXT);
await storage.commit(
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT);
await commitWrites(
storage,
Array.from({ length: 100 }, (_, index) => ({
type: "entry" as const,
value: { id: idFromNumber<EntryId>(index + 2), conversationId: ROOT_CONVERSATION_ID, kind: "cached" },
})),
adopt,
BACKGROUND_CONTEXT,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "entry",
value: { id: idFromNumber<EntryId>(102), conversationId: ROOT_CONVERSATION_ID, kind: "cached-again" },
},
],
adopt,
BACKGROUND_CONTEXT,
);
expect((await storage.entry(idFromNumber<EntryId>(2), BACKGROUND_CONTEXT))?.entry.kind).toBe("cached");
expect((await storage.entry(idFromNumber<EntryId>(102), BACKGROUND_CONTEXT))?.entry.kind).toBe("cached-again");
await expect(
storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, BACKGROUND_CONTEXT),
commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT),
).rejects.toThrow("ID 1 already belongs to conversation");
expect((await storage.entry(idFromNumber<EntryId>(2), BACKGROUND_CONTEXT))?.entry.kind).toBe("cached");
expect(connection.repeatedPrepares()).toEqual([]);
@ -133,9 +131,9 @@ describe("portable SQLite facade settlement", () => {
it("reads storage commits in call order", async () => {
const database = await openNodeSqliteDatabase(":memory:");
const storage = await SqliteStorage.open(database);
const commit = storage.commit(
const commit = commitWrites(
storage,
[{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }],
adopt,
BACKGROUND_CONTEXT,
);
const read = storage.conversation(ROOT_CONVERSATION_ID, BACKGROUND_CONTEXT);
@ -147,12 +145,12 @@ describe("portable SQLite facade settlement", () => {
it("lets admitted multi-query reads finish before storage closes", async () => {
const storage = await SqliteStorage.open(await openNodeSqliteDatabase(":memory:"));
const entryId = idFromNumber<EntryId>(2);
await storage.commit(
await commitWrites(
storage,
[
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
{ type: "entry", value: { id: entryId, conversationId: ROOT_CONVERSATION_ID, kind: "probe" } },
],
adopt,
BACKGROUND_CONTEXT,
);
const scan = storage.scanEntries({ conversationId: ROOT_CONVERSATION_ID }, 10, undefined, BACKGROUND_CONTEXT);
@ -184,14 +182,14 @@ describe("portable SQLite facade settlement", () => {
const storage = await opening;
database.controlNextSettlement("delay");
const committing = storage.commit(
const committing = commitWrites(
storage,
[
{
type: "entry",
value: { id: idFromNumber<EntryId>(100), conversationId: ROOT_CONVERSATION_ID, kind: "settled" },
},
],
adopt,
BACKGROUND_CONTEXT,
);
let committed = false;
@ -206,14 +204,14 @@ describe("portable SQLite facade settlement", () => {
expect(await storage.mintId<EntryId>()).toBe(101);
database.controlNextSettlement("reject");
const rejected = storage.commit(
const rejected = commitWrites(
storage,
[
{
type: "entry",
value: { id: idFromNumber<EntryId>(200), conversationId: ROOT_CONVERSATION_ID, kind: "rejected" },
},
],
adopt,
BACKGROUND_CONTEXT,
);
expect(await storage.mintId<EntryId>()).toBe(102);
@ -229,7 +227,8 @@ describe("portable SQLite facade settlement", () => {
// Each yield count starts the commit at a different point of the read's record and revision queries.
for (let yields = 0; yields < 16; yields++) {
const storage = await SqliteStorage.open(await openNodeSqliteDatabase(":memory:"));
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -237,14 +236,13 @@ describe("portable SQLite facade settlement", () => {
content: { kind: "base", version: 1, value: { value: 1 } },
},
],
adopt,
BACKGROUND_CONTEXT,
);
const read = storage.document(id, "current", BACKGROUND_CONTEXT);
for (let index = 0; index < yields; index++) await Promise.resolve();
const replace = storage.commit(
const replace = commitWrites(
storage,
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { value: 2 } } }],
adopt,
BACKGROUND_CONTEXT,
);
const [stored] = await Promise.all([read, replace]);

5
packages/durable/test/sqlite-migrations.test.ts
View file

@ -11,6 +11,7 @@ import {
type SqliteMigration,
} from "../src/storage/sqlite/index.ts";
import { openNodeSqliteDatabase, openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import { type EntryId, ROOT_CONVERSATION_ID } from "../src/types.ts";
const directories = new Set<string>();
@ -115,7 +116,8 @@ describe("durable SQLite migrations", () => {
it("rolls a failed migration back and preserves stored data for a successful retry", async () => {
const path = await databasePath();
const storage = await openNodeSqliteStorage(path);
await storage.commit(
await commitWrites(
storage,
[
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
{
@ -128,7 +130,6 @@ describe("durable SQLite migrations", () => {
},
},
],
() => {},
BACKGROUND_CONTEXT,
);
await storage.close(BACKGROUND_CONTEXT);

241
packages/durable/test/sqlite-storage.test.ts
View file

@ -4,12 +4,19 @@ import { join } from "node:path";
import { DatabaseSync } from "node:sqlite";
import type { Context, JsonValue } from "@earendil-works/chord";
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
import type { SqliteDatabase } from "@earendil-works/chord/sqlite";
import { registerStorageConformance } from "@earendil-works/pi-durable/testing";
import { afterEach, describe, expect, it } from "vitest";
import { idFromNumber } from "../src/ids.ts";
import type { SqliteStorage } from "../src/storage/sqlite/index.ts";
import { type NodeSqliteStorageOptions, openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
import { type SqliteStorage, SqliteStorage as SqliteStorageClass } from "../src/storage/sqlite/index.ts";
import {
type NodeSqliteStorageOptions,
openNodeSqliteDatabase,
openNodeSqliteStorage,
} from "../src/storage/sqlite/node.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import type {
CommitResult,
ConversationId,
DocumentCreate,
DocumentId,
@ -24,9 +31,6 @@ import type {
} from "../src/types.ts";
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
const context = BACKGROUND_CONTEXT;
const openStorages = new Set<SqliteStorage>();
const tempDirectories = new Set<string>();
@ -60,13 +64,13 @@ class ReopeningStorage implements Storage {
}
async commit(
writes: readonly StorageWrite[],
adopt: Parameters<Storage["commit"]>[1],
commitContext: Parameters<Storage["commit"]>[2],
): Promise<Seq> {
if (this.closed) return this.current.commit(writes, adopt, commitContext);
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
commitContext: Context,
): Promise<CommitResult> {
if (this.closed) return this.current.commit(batches, receive, commitContext);
try {
return await this.current.commit(writes, adopt, commitContext);
return await this.current.commit(batches, receive, commitContext);
} finally {
await this.current.close(context);
this.current = await openNodeSqliteStorage(this.path, this.options);
@ -140,7 +144,7 @@ function entry(id: EntryId, conversationId: ConversationId, data?: JsonValue): E
}
async function createRoot(storage: Storage): Promise<Seq> {
return storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
return commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
}
function pendingTask(id: TaskId<JsonValue>): TaskRecord<JsonValue, JsonValue, JsonValue> {
@ -179,7 +183,7 @@ describe("Pico SqliteStorage", () => {
expect(await createRoot(storage)).toBe(1);
const entryId = await storage.mintId<EntryId>();
expect(
await storage.commit([{ type: "entry", value: entry(entryId, ROOT_CONVERSATION_ID) }], adopt, context),
await commitWrites(storage, [{ type: "entry", value: entry(entryId, ROOT_CONVERSATION_ID) }], context),
).toBe(2);
await storage.close(context);
openStorages.delete(storage);
@ -192,9 +196,9 @@ describe("Pico SqliteStorage", () => {
});
expect(await reopened.mintId<EntryId>()).toBe(entryId + 1);
expect(
await reopened.commit(
await commitWrites(
reopened,
[{ type: "task", value: pendingTask(await reopened.mintId<TaskId<JsonValue>>()) }],
adopt,
context,
),
).toBe(3);
@ -257,7 +261,8 @@ describe("Pico SqliteStorage", () => {
const { storage, path } = await createSqliteStorage();
await createRoot(storage);
const id = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -265,7 +270,6 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { retained: true } },
},
],
adopt,
context,
);
const database = new DatabaseSync(path);
@ -283,7 +287,8 @@ describe("Pico SqliteStorage", () => {
const { storage, path } = await createSqliteStorage();
await createRoot(storage);
const id = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -291,10 +296,10 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { nested: { value: 1 }, rows: [] } },
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -306,10 +311,10 @@ describe("Pico SqliteStorage", () => {
},
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -325,10 +330,10 @@ describe("Pico SqliteStorage", () => {
},
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -336,7 +341,6 @@ describe("Pico SqliteStorage", () => {
content: { kind: "delta", version: 1, ops: [["s", ["nested", "value"], 4]] },
},
],
adopt,
context,
);
@ -367,7 +371,8 @@ describe("Pico SqliteStorage", () => {
const circular: { self?: unknown } = {};
circular.self = circular;
await expect(
storage.commit(
commitWrites(
storage,
[
{ type: "entry", value: entry(transientId, ROOT_CONVERSATION_ID) },
{
@ -375,7 +380,6 @@ describe("Pico SqliteStorage", () => {
value: { ...pendingTask(transientTaskId), input: circular as unknown as JsonValue },
},
],
adopt,
context,
),
).rejects.toThrow("circular structure");
@ -383,7 +387,7 @@ describe("Pico SqliteStorage", () => {
expect(await storage.task(transientTaskId, context)).toBeUndefined();
const committedId = await storage.mintId<EntryId>();
expect(
await storage.commit([{ type: "entry", value: entry(committedId, ROOT_CONVERSATION_ID) }], adopt, context),
await commitWrites(storage, [{ type: "entry", value: entry(committedId, ROOT_CONVERSATION_ID) }], context),
).toBe(2);
const db = new DatabaseSync(path, { readOnly: true });
@ -406,15 +410,16 @@ describe("Pico SqliteStorage", () => {
history: "rewindable",
fork: "asOf",
} satisfies DocumentCreate;
const createdAt = await storage.commit(
const createdAt = await commitWrites(
storage,
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { count: 0 } } }],
adopt,
context,
);
let ancientAt = createdAt;
let recentAt = createdAt;
for (let count = 1; count <= 40; count++) {
recentAt = await storage.commit(
recentAt = await commitWrites(
storage,
[
{
type: "document.change",
@ -425,7 +430,6 @@ describe("Pico SqliteStorage", () => {
: { kind: "delta", version: 1, ops: [["s", ["count"], count]] },
},
],
adopt,
context,
);
if (count === 5) ancientAt = recentAt;
@ -500,7 +504,8 @@ describe("Pico SqliteStorage", () => {
const { storage, path } = await createSqliteStorage();
await createRoot(storage);
const id = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -508,13 +513,12 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
for (let count = 1; count <= 10; count++) {
await storage.commit(
await commitWrites(
storage,
[{ type: "document.change", id, content: { kind: "delta", version: 1, ops: [["s", ["count"], count]] } }],
adopt,
context,
);
}
@ -530,13 +534,14 @@ describe("Pico SqliteStorage", () => {
} finally {
revisions.close();
}
await storage.commit(
await commitWrites(
storage,
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { count: 11 } } }],
adopt,
context,
);
expect(revisionCount(path, id)).toBe(1);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -544,11 +549,10 @@ describe("Pico SqliteStorage", () => {
content: { kind: "delta", version: 1, ops: [["r", { count: 12 }]] },
},
],
adopt,
context,
);
expect(revisionCount(path, id)).toBe(2);
await storage.commit([{ type: "document.retire", id }], adopt, context);
await commitWrites(storage, [{ type: "document.retire", id }], context);
expect(revisionCount(path, id)).toBe(0);
});
@ -556,7 +560,8 @@ describe("Pico SqliteStorage", () => {
const { storage, path } = await createSqliteStorage({ walAutoCheckpointPages: 1 });
await createRoot(storage);
for (let index = 0; index < 20; index++) {
await storage.commit(
await commitWrites(
storage,
[
{
type: "entry",
@ -566,7 +571,6 @@ describe("Pico SqliteStorage", () => {
}),
},
],
adopt,
context,
);
}
@ -588,7 +592,8 @@ describe("Pico SqliteStorage", () => {
await createRoot(storage);
const id = await storage.mintId<DocumentId>();
const large = "x".repeat(512 * 1024);
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -596,12 +601,11 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { text: large } },
},
],
adopt,
context,
);
await storage.commit(
await commitWrites(
storage,
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { text: "small" } } }],
adopt,
context,
);
const before = new DatabaseSync(path, { readOnly: true });
@ -614,9 +618,9 @@ describe("Pico SqliteStorage", () => {
before.close();
}
expect(freeAfterDelete).toBeGreaterThan(0);
await storage.commit(
await commitWrites(
storage,
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { text: large } } }],
adopt,
context,
);
const after = new DatabaseSync(path, { readOnly: true });
@ -634,7 +638,8 @@ describe("Pico SqliteStorage", () => {
const { storage, path } = await createSqliteStorage();
await createRoot(storage);
for (let index = 0; index < 100; index++) {
await storage.commit(
await commitWrites(
storage,
[
{
type: "entry",
@ -644,12 +649,12 @@ describe("Pico SqliteStorage", () => {
}),
},
],
adopt,
context,
);
}
const documentId = await storage.mintId<DocumentId>();
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -663,11 +668,11 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
for (let count = 1; count <= 100; count++) {
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.change",
@ -675,7 +680,6 @@ describe("Pico SqliteStorage", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], count]] },
},
],
adopt,
context,
);
}
@ -684,3 +688,130 @@ describe("Pico SqliteStorage", () => {
expect((await stat(path)).size).toBeLessThan(1024 * 1024);
});
});
describe("SqliteStorage batches", () => {
for (const mode of ["undefined rejection", "synchronous throw"] as const) {
it(`rejects the whole call, the batches before it included, when a later batch's check read fails with a ${mode}`, async () => {
// Sol's: the first batch passes its checks and is written; the second batch's check read fails.
const directory = await mkdtemp(join(tmpdir(), "pi-durable-sqlite-"));
tempDirectories.add(directory);
const database = await openNodeSqliteDatabase(join(directory, "storage.sqlite"));
let failCheck = false;
let reads = 0;
const faulty: SqliteDatabase = {
exec: (sql) => database.exec(sql),
run: (sql, ...params) => database.run(sql, ...params),
runCounted: (sql, ...params) => database.runCounted(sql, ...params),
get: (sql, ...params) => database.get(sql, ...params),
all: (sql, ...params) => database.all(sql, ...params),
close: () => database.close(),
transaction: (callback) =>
database.transaction((transaction) =>
callback({
exec: (sql) => transaction.exec(sql),
run: (sql, ...params) => transaction.run(sql, ...params),
runCounted: (sql, ...params) => transaction.runCounted(sql, ...params),
all: (sql, ...params) => transaction.all(sql, ...params),
get: (sql, ...params) => {
if (failCheck && sql.includes("record_type") && ++reads === 2) {
if (mode === "synchronous throw") throw new Error("sync driver failure");
return Promise.reject(undefined);
}
return transaction.get(sql, ...params);
},
}),
),
};
const storage = await SqliteStorageClass.open(faulty);
openStorages.add(storage);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
const ids = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
const batches: StorageWrite[][] = ids.map((id) => [
{ type: "entry", value: { id, conversationId: ROOT_CONVERSATION_ID, kind: "note" } },
]);
failCheck = true;
const receipts: number[] = [];
const outcome = await storage
.commit(batches, (index) => receipts.push(index), context)
.then(
() => "resolved",
() => "rejected",
);
failCheck = false;
expect(outcome).toBe("rejected");
expect(receipts).toEqual([]);
expect(await storage.entry(ids[0]!, context)).toBeUndefined();
});
}
/** How a check's read of record types fails in the database. */
const failures: Record<string, () => Promise<never>> = {
"an Error rejection": () => Promise.reject(new Error("disk I/O error")),
"a synchronous throw": () => {
throw new Error("disk I/O error");
},
"a primitive rejection": () => Promise.reject("disk I/O error"),
"an undefined rejection": () => Promise.reject(undefined),
};
for (const [name, fail] of Object.entries(failures)) {
it(`rejects a batch call whose check read fails in the database with ${name}, and refuses only for its own checks`, async () => {
const directory = await mkdtemp(join(tmpdir(), "pi-durable-sqlite-"));
tempDirectories.add(directory);
const database = await openNodeSqliteDatabase(join(directory, "storage.sqlite"));
// A driver failure in a check's read, armed for the next batch call.
let failCheck = false;
const faulty: SqliteDatabase = {
exec: (sql) => database.exec(sql),
run: (sql, ...params) => database.run(sql, ...params),
runCounted: (sql, ...params) => database.runCounted(sql, ...params),
get: (sql, ...params) => database.get(sql, ...params),
all: (sql, ...params) => database.all(sql, ...params),
close: () => database.close(),
transaction: (callback) =>
database.transaction((transaction) =>
callback({
exec: (sql) => transaction.exec(sql),
run: (sql, ...params) => transaction.run(sql, ...params),
runCounted: (sql, ...params) => transaction.runCounted(sql, ...params),
all: (sql, ...params) => transaction.all(sql, ...params),
get: (sql, ...params) => {
if (failCheck && sql.includes("record_type")) return fail();
return transaction.get(sql, ...params);
},
}),
),
};
const storage = await SqliteStorageClass.open(faulty);
openStorages.add(storage);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
const first = await storage.mintId<EntryId>();
const second = await storage.mintId<EntryId>();
const batches: StorageWrite[][] = [first, second].map((id) => [
{ type: "entry", value: { id, conversationId: ROOT_CONVERSATION_ID, kind: "note" } },
]);
failCheck = true;
const receipts: number[] = [];
const outcome = await storage
.commit(batches, (index) => receipts.push(index), context)
.then(
(result) => ({ kind: "resolved", result }),
() => ({ kind: "rejected" }),
);
failCheck = false;
// Unknown outcome, rolled back whole: nothing received, nothing stored.
expect(outcome).toEqual({ kind: "rejected" });
expect(receipts).toEqual([]);
expect(await storage.entry(first, context)).toBeUndefined();
// Durable's own check still refuses, keeping the batches before it.
const refused = await storage.commit(
[batches[0]!, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }]],
(index) => receipts.push(index),
context,
);
expect(refused.stored).toBe(1);
expect(String(refused.refused)).toContain("already belongs to conversation");
expect(receipts).toEqual([0]);
expect(await storage.entry(first, context)).toBeDefined();
});
}
});

769
packages/durable/test/store-storage-commit.test.ts Normal file
View file

@ -0,0 +1,769 @@
// StoreStorage's commit path (`Storage.commit` over `ChordStore.commitSeries`), p3's primary storage: per call one
// backend transaction with one seq per batch; Durable's checks of a batch refuse it in its turn, the batches before it
// stored; a failed statement rolls the call back and rejects, the store going on; a failed `COMMIT` stops the store;
// receipts after `COMMIT`; each commit published in a host batch of its own; other users of the store wait their turn.
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { setFlagsFromString } from "node:v8";
import { runInNewContext } from "node:vm";
import { BACKGROUND_CONTEXT, withTelemetry } from "@earendil-works/chord/context";
import { type ChordStore, createHost, openStore, sqliteBackend } from "@earendil-works/chord/next";
import type { SqliteDatabase, SqliteExecutor } from "@earendil-works/chord/sqlite";
import { openNodeSqliteDatabase } from "@earendil-works/chord/sqlite/node";
import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry";
import { afterEach, describe, expect, it } from "vitest";
import { idFromNumber } from "../src/ids.ts";
import { MemoryStorage } from "../src/index.ts";
import { SessionImpl } from "../src/session/session.ts";
import { StoreStorage } from "../src/storage/store/index.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import type {
CommitResult,
ConversationId,
DocumentCreate,
DocumentId,
EntryId,
Seq,
Storage,
StorageWrite,
} from "../src/types.ts";
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
import { openStoreStorage } from "./store-storage-support.ts";
const context = BACKGROUND_CONTEXT;
let directory: string | undefined;
afterEach(async () => {
if (directory !== undefined) await rm(directory, { recursive: true, force: true });
directory = undefined;
});
/** A fresh SQLite file in this test's directory. */
async function storePath(): Promise<string> {
directory ??= await mkdtemp(join(tmpdir(), "pi-durable-store-commit-"));
return join(directory, `store-${Math.random().toString(36).slice(2)}.sqlite`);
}
/**
* What the next backend transaction does besides its work: `commit` fails it after its work (a failed `COMMIT`, an
* unknown outcome); `fail` fails a statement it matches (by its SQL and its index, from 1), as a backend error; `hold`
* holds a statement until the promise it returns settles.
*/
type Faults = {
commit: boolean;
fail: ((sql: string, index: number) => boolean) | undefined;
hold: ((sql: string, index: number) => Promise<void> | undefined) | undefined;
};
/**
* A storage with its root conversation, over a store with an outbox on the SQLite file at `path`, whose backend counts
* its transactions and the statements of the last one, and takes `faults` for its next transaction.
*/
async function counted(path = ":memory:") {
const db = await openNodeSqliteDatabase(path);
const counts = { transactions: 0, statements: 0 };
const faults: Faults = { commit: false, fail: undefined, hold: undefined };
const backend: SqliteDatabase = {
exec: (sql) => db.exec(sql),
run: (sql, ...params) => db.run(sql, ...params),
runCounted: (sql, ...params) => db.runCounted(sql, ...params),
get: (sql, ...params) => db.get(sql, ...params),
all: (sql, ...params) => db.all(sql, ...params),
close: () => db.close(),
transaction: (work) => {
counts.transactions++;
counts.statements = 0;
const { commit, fail, hold } = faults;
faults.commit = false;
faults.fail = undefined;
faults.hold = undefined;
return db.transaction(async (executor) => {
const call = async <T>(sql: string, run: () => Promise<T>): Promise<T> => {
const index = ++counts.statements;
await hold?.(sql, index);
if (fail?.(sql, index) === true) throw new Error(`injected: statement ${index} failed`);
return run();
};
const result = await work({
exec: (sql) => call(sql, () => executor.exec(sql)),
run: (sql, ...params) => call(sql, () => executor.run(sql, ...params)),
runCounted: (sql, ...params) => call(sql, () => executor.runCounted(sql, ...params)),
get: (sql, ...params) => call(sql, () => executor.get(sql, ...params)),
all: (sql, ...params) => call(sql, () => executor.all(sql, ...params)),
} satisfies SqliteExecutor);
if (commit) throw new Error("COMMIT failed");
return result;
});
},
};
const store = (await openStore(sqliteBackend(backend), context, { outbox: true })) as ChordStore;
const storage = await StoreStorage.open(store, "harness", context);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
return { store, storage, counts, faults };
}
/** The same storage opened again on the file at `path`, as another process would. */
async function reopen(path: string) {
const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context)) as ChordStore;
return { store, storage: await StoreStorage.open(store, "harness", context) };
}
const entryWrite = (id: EntryId, kind = "m"): StorageWrite => ({
type: "entry",
value: { id, conversationId: ROOT_CONVERSATION_ID, kind },
});
/** One macrotask: what can run before it has run. */
const macrotask = (): Promise<void> => new Promise((resolve) => setTimeout(resolve, 0));
/** The store's outbox commits, by seq. */
async function outboxSeqs(store: ChordStore): Promise<number[]> {
const rows = await store.read((executor) => executor.all<{ seq: number }>("SELECT seq FROM chord_outbox"), context);
return rows.map((row) => row.seq);
}
describe("one transaction per call", () => {
it("stores the batches in one backend transaction, one seq and one outbox commit each, receipts in order", async () => {
const { store, storage, counts } = await counted();
const ids = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
const before = counts.transactions;
const last = store.lastSeq;
const receipts: [number, Seq][] = [];
const result = await storage.commit(
ids.map((id) => [entryWrite(id)]),
(index, seq) => receipts.push([index, seq]),
context,
);
expect(result).toEqual({ stored: 3 });
expect(counts.transactions - before).toBe(1);
expect(receipts).toEqual([
[0, last + 1],
[1, last + 2],
[2, last + 3],
]);
for (const [index, id] of ids.entries()) {
expect((await storage.entry(id, context))?.commitSeq).toBe(last + 1 + index);
}
const outbox = await store.read(
(executor) => executor.all<{ seq: number }>("SELECT seq FROM chord_outbox WHERE seq > ?", last),
context,
);
expect(outbox.map((row) => row.seq)).toEqual([last + 1, last + 2, last + 3]);
await store.close(context);
});
it("writes the metadata once per call: three statements per entry append, plus one (cost guard)", async () => {
const { store, storage, counts } = await counted();
const telemetry = new InMemoryTelemetryContext();
const span = await telemetry.startSpan({ name: "test" }, (each) => each);
const metered = withTelemetry(span, context);
const statements = (): number =>
telemetry
.getMetrics()
.filter(({ name }) => name === "chord.store.statements")
.reduce((total, { value }) => total + value, 0);
// Alone, an append is its id check, its row and claim, its outbox commit, and the metadata.
await commitWrites(storage, [entryWrite(await storage.mintId<EntryId>())], metered);
expect(statements()).toBe(5);
const ids = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
const before = counts.transactions;
await storage.commit(
ids.map((id) => [entryWrite(id)]),
() => {},
metered,
);
expect([counts.transactions - before, statements() - 5]).toEqual([1, 3 * 4 + 1]);
await store.close(context);
});
});
describe("refusals: the batches before the refused one stand", () => {
it("stores the batches before a refused one with Durable's message, none after it, and goes on", async () => {
const { store, storage } = await counted();
const first = await storage.mintId<EntryId>();
const after = await storage.mintId<EntryId>();
const last = store.lastSeq;
const receipts: number[] = [];
const result = await storage.commit(
[[entryWrite(first)], [entryWrite(first, "again")], [entryWrite(after)]],
(index) => receipts.push(index),
context,
);
expect(result.stored).toBe(1);
expect(String(result.refused)).toBe(`Error: ID ${first} already belongs to entry`);
expect(receipts).toEqual([0]);
expect((await storage.entry(first, context))?.entry.kind).toBe("m");
expect(await storage.entry(after, context)).toBeUndefined();
expect(store.lastSeq).toBe(last + 1);
expect(await commitWrites(storage, [entryWrite(after)], context)).toBe(last + 2);
await store.close(context);
});
it("refuses a batch its staging fails in that batch's turn, keeping the batches before it (review)", async () => {
const { store, storage } = await counted();
const entry = await storage.mintId<EntryId>();
const document = await storage.mintId<DocumentId>();
const receipts: number[] = [];
const result = await storage.commit(
[
[entryWrite(entry)],
// Staging refuses it: one document retired twice.
[
{ type: "document.retire", id: document },
{ type: "document.retire", id: document },
],
],
(index) => receipts.push(index),
context,
);
expect(result.stored).toBe(1);
expect(String(result.refused)).toBe(`Error: Document ${document} is retired more than once`);
expect(receipts).toEqual([0]);
expect(await storage.entry(entry, context)).toBeDefined();
await store.close(context);
});
it.each([
[
"a duplicate retirement",
[
{ type: "document.retire", id: 101 },
{ type: "document.retire", id: 101 },
],
],
[
"a duplicate content command",
[
{ type: "document.change", id: 101, content: { kind: "base", version: 1, value: {} } },
{ type: "document.change", id: 101, content: { kind: "base", version: 1, value: {} } },
],
],
] as const)(
"refuses a later batch with %s as Memory does, keeping the prefix (review)",
async (_name, malformed) => {
const batches: StorageWrite[][] = [
[{ type: "conversation", value: { id: idFromNumber<ConversationId>(100) } }],
malformed.map((write) => ({ ...write, id: idFromNumber<DocumentId>(write.id) })) as StorageWrite[],
];
const outcomes = [];
for (const storage of [new MemoryStorage(), await openStoreStorage(":memory:")]) {
const receipts: number[] = [];
const result = await storage.commit(batches, (index) => receipts.push(index), context);
const conversation = await storage.conversation(idFromNumber<ConversationId>(100), context);
outcomes.push({ stored: result.stored, refused: String(result.refused), receipts, conversation });
await storage.close(context);
}
expect(outcomes[0]).toMatchObject({ stored: 1, receipts: [0], conversation: { id: 100 } });
expect(outcomes[1]).toEqual(outcomes[0]);
},
);
it.each([
[
"a change of a document an earlier batch retired",
(id: DocumentId): StorageWrite[][] => [
[{ type: "document.retire", id }],
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { n: 2 } } }],
],
"is retired",
],
[
"a creation at the address an earlier batch took",
(id: DocumentId): StorageWrite[][] => [
[
{
type: "document.create",
record: { id: idFromNumber<DocumentId>(id + 1), kind: "n", key: "taken", scope: { kind: "session" } },
content: { kind: "base", version: 1, value: { n: 2 } },
},
],
[
{
type: "document.create",
record: { id: idFromNumber<DocumentId>(id + 2), kind: "n", key: "taken", scope: { kind: "session" } },
content: { kind: "base", version: 1, value: { n: 3 } },
},
],
],
"already has a current incarnation",
],
] as const)("refuses %s against the state the batches before it left", async (_name, calls, message) => {
const { store, storage } = await counted();
const document = idFromNumber<DocumentId>(200);
await commitWrites(
storage,
[
{
type: "document.create",
record: { id: document, kind: "n", scope: { kind: "session" } },
content: { kind: "base", version: 1, value: { n: 1 } },
},
],
context,
);
const last = store.lastSeq;
const receipts: number[] = [];
const result = await storage.commit(calls(document), (index) => receipts.push(index), context);
expect(result.stored).toBe(1);
expect(String(result.refused)).toContain(message);
expect([receipts, store.lastSeq]).toEqual([[0], last + 1]);
expect((await outboxSeqs(store)).at(-1)).toBe(last + 1);
await store.close(context);
});
it("refuses a batch whose rows staging cannot encode, keeping the batches before it", async () => {
const { store, storage } = await counted();
const entry = await storage.mintId<EntryId>();
const receipts: number[] = [];
// No integer: the Session never sends one (Storage trusts it); the store's columns refuse it at staging.
const result = await storage.commit(
[[entryWrite(entry)], [entryWrite(idFromNumber<EntryId>(1.5))]],
(index) => receipts.push(index),
context,
);
expect(result.stored).toBe(1);
expect(String(result.refused)).toContain('column "id": 1.5 is not a int value');
expect(receipts).toEqual([0]);
expect(await storage.entry(entry, context)).toBeDefined();
await store.close(context);
});
it("checks a batch against the batches before it in the call: a copy reads their state", async () => {
const { store, storage } = await counted();
const source = await storage.mintId<DocumentId>();
const child = await storage.mintId<ConversationId>();
const copy = await storage.mintId<DocumentId>();
const record = (id: DocumentId, conversationId: ConversationId) =>
({
id,
kind: "c",
scope: { kind: "conversation", conversationId },
history: "rewindable",
fork: "asOf",
}) as const;
const result = await storage.commit(
[
[
{
type: "document.create",
record: record(source, ROOT_CONVERSATION_ID),
content: { kind: "base", version: 1, value: { n: 1 } },
},
],
[{ type: "document.change", id: source, content: { kind: "delta", version: 1, ops: [["s", ["n"], 2]] } }],
[
{ type: "conversation", value: { id: child } },
{ type: "document.copy", record: record(copy, child), source: { id: source, at: "current" } },
],
],
() => {},
context,
);
expect(result).toEqual({ stored: 3 });
expect((await storage.document(copy, "current", context))?.value).toEqual({ n: 2 });
await store.close(context);
});
});
describe("failures: a rollback rejects, an unknown outcome stops the store", () => {
it("rejects with nothing stored when a batch's check fails in the backend, and the store goes on", async () => {
const { store, storage, faults } = await counted();
const [first, second] = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
const last = store.lastSeq;
// The second batch's id check: its first read of the record ids.
let claimReads = 0;
faults.fail = (sql) => sql.includes("record_ids") && ++claimReads === 2;
const receipts: number[] = [];
await expect(
storage.commit([[entryWrite(first)], [entryWrite(second)]], (index) => receipts.push(index), context),
).rejects.toThrow("injected");
expect([receipts, store.lastSeq]).toEqual([[], last]);
expect(await storage.entry(first, context)).toBeUndefined();
expect(await commitWrites(storage, [entryWrite(first)], context)).toBe(last + 1);
await store.close(context);
});
it("a failed COMMIT is an unknown outcome: the call rejects storage_failed, nothing is received or stored", async () => {
const path = await storePath();
const { store, storage, faults } = await counted(path);
const ids = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
const last = store.lastSeq;
faults.commit = true;
const receipts: number[] = [];
await expect(
storage.commit(
ids.map((id) => [entryWrite(id)]),
(index) => receipts.push(index),
context,
),
).rejects.toMatchObject({ code: "unavailable", reason: "storage_failed" });
expect(receipts).toEqual([]);
await expect(storage.entry(ids[0]!, context)).rejects.toMatchObject({ reason: "storage_failed" });
await store.close(context);
const reopened = await reopen(path);
expect(await reopened.storage.entry(ids[0]!, context)).toBeUndefined();
expect(await commitWrites(reopened.storage, [entryWrite(ids[1]!)], context)).toBe(last + 1);
await reopened.store.close(context);
});
/** The record the audit's source document and its copy share. */
const record = (id: number, conversationId: number): DocumentCreate => ({
id: idFromNumber<DocumentId>(id),
kind: "audit",
scope: { kind: "conversation", conversationId: idFromNumber<ConversationId>(conversationId) },
history: "rewindable",
fork: "asOf",
});
/** The source document, stored before the call. */
const SEED: StorageWrite[] = [
{
type: "document.create",
record: record(100, ROOT_CONVERSATION_ID),
content: { kind: "base", version: 1, value: { n: 0 } },
},
];
/** A delta of the source document setting `n`. */
const delta = (n: number): StorageWrite => ({
type: "document.change",
id: idFromNumber<DocumentId>(100),
content: { kind: "delta", version: 1, ops: [["s", ["n"], n]] },
});
/**
* Three batches: an entry and a delta; a conversation, a copy of the source as the first batch left it and an
* entry; then an entry and a delta, or, `refused`, an entry whose id the first batch took.
*/
const batches = (third: "valid" | "refused"): StorageWrite[][] => [
[entryWrite(idFromNumber<EntryId>(101), "b1"), delta(1)],
[
{ type: "conversation", value: { id: idFromNumber<ConversationId>(105) } },
{
type: "document.copy",
record: record(102, 105),
source: { id: idFromNumber<DocumentId>(100), at: "current" },
},
entryWrite(idFromNumber<EntryId>(103), "b2"),
],
third === "valid"
? [entryWrite(idFromNumber<EntryId>(104), "b3"), delta(2)]
: [entryWrite(idFromNumber<EntryId>(101), "again")],
];
/** What a storage holds of the audit's records. */
async function stateOf(storage: Storage) {
const entries: (string | null)[] = [];
for (const id of [101, 103, 104])
entries.push((await storage.entry(idFromNumber<EntryId>(id), context))?.entry.kind ?? null);
return {
entries,
source: (await storage.document(idFromNumber<DocumentId>(100), "current", context))?.value ?? null,
copy: (await storage.document(idFromNumber<DocumentId>(102), "current", context))?.value ?? null,
child: (await storage.conversation(idFromNumber<ConversationId>(105), context)) ?? null,
};
}
/** What a call returned and what it left: its result, its receipts' seqs after the store's last one, the state. */
async function outcomeOf(storage: Storage, third: "valid" | "refused", last: number) {
const seqs: number[] = [];
const result: CommitResult = await storage.commit(
batches(third),
(_index, seq) => seqs.push(seq - last),
context,
);
return { stored: result.stored, refused: String(result.refused), seqs, state: await stateOf(storage) };
}
/** Every error in `error`'s cause chain, as text. */
function chain(error: unknown): string[] {
const messages: string[] = [];
for (let at = error; at instanceof Error; at = at.cause) messages.push(at.message);
return messages;
}
it.each(["valid", "refused"] as const)(
"rolls a 3-batch call back at a failure of any of its statements, as at a failed COMMIT, and reopens before it (third %s)",
async (third) => {
// What MemoryStorage makes of the call: the outcome every retry must reach.
const memory = new MemoryStorage();
await commitWrites(memory, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
await commitWrites(memory, SEED, context);
const expected = await outcomeOf(memory, third, 2);
expect(expected).toMatchObject(
third === "valid" ? { stored: 3, seqs: [1, 2, 3] } : { stored: 2, seqs: [1, 2] },
);
// The statements of the call's transaction (a read transaction after it would count its own).
const probe = await counted();
await commitWrites(probe.storage, SEED, context);
const result = await probe.storage.commit(batches(third), () => {}, context);
const total = probe.counts.statements;
// Its cost: no read beyond a batch's checks, the metadata once (statement counts per batch in the cost guard).
expect(total).toBe(third === "valid" ? 29 : 23);
expect(result.stored).toBe(expected.stored);
expect(await stateOf(probe.storage)).toEqual(expected.state);
await probe.store.close(context);
// Each statement in turn, then the COMMIT.
for (let at = 1; at <= total + 1; at++) {
const path = await storePath();
const opened = await counted(path);
await commitWrites(opened.storage, SEED, context);
const last = opened.store.lastSeq;
const outbox = await outboxSeqs(opened.store);
const before = await stateOf(opened.storage);
if (at <= total) opened.faults.fail = (_sql, index) => index === at;
else opened.faults.commit = true;
const receipts: number[] = [];
const error = await opened.storage
.commit(batches(third), (each) => receipts.push(each), context)
.then(
() => undefined,
(rejection: unknown) => rejection,
);
expect(receipts).toEqual([]);
expect(opened.store.lastSeq).toBe(last);
if (at <= total) {
// A rollback: the call rejects with its failure, the store goes on.
expect(chain(error)).toContain(`injected: statement ${at} failed`);
expect(await stateOf(opened.storage)).toEqual(before);
} else {
expect(error).toMatchObject({ code: "unavailable", reason: "storage_failed" });
}
await opened.store.close(context);
// On disk: as before the call, the seq, the outbox, the records and the allocator.
const again = await reopen(path);
expect(again.store.lastSeq).toBe(last);
expect(await outboxSeqs(again.store)).toEqual(outbox);
expect(await stateOf(again.storage)).toEqual(before);
expect(await again.storage.mintId<EntryId>()).toBeGreaterThan(100);
// A retry reaches MemoryStorage's outcome.
expect(await outcomeOf(again.storage, third, last)).toEqual(expected);
expect(await outboxSeqs(again.store)).toEqual([...outbox, ...expected.seqs.map((seq) => last + seq)]);
await again.store.close(context);
}
},
);
it("a call on a closed storage, or on a stopped store, rejects and stores nothing", async () => {
const { store, storage, faults } = await counted();
const entry = await storage.mintId<EntryId>();
faults.commit = true;
await expect(commitWrites(storage, [entryWrite(entry)], context)).rejects.toMatchObject({
reason: "storage_failed",
});
await expect(storage.commit([[entryWrite(entry)]], () => {}, context)).rejects.toMatchObject({
reason: "storage_failed",
});
await store.close(context);
const other = await counted();
await other.storage.close(context);
await expect(other.storage.commit([[entryWrite(entry)]], () => {}, context)).rejects.toThrow("closed");
await other.store.close(context);
});
it("a call queued when the store closes rejects released, and stores nothing; the running one is stored", async () => {
const path = await storePath();
const { store, storage, faults } = await counted(path);
const [first, second] = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
const entered = Promise.withResolvers<void>();
const release = Promise.withResolvers<void>();
faults.hold = (_sql, index) => {
if (index !== 1) return undefined;
entered.resolve();
return release.promise;
};
const running = storage.commit([[entryWrite(first)]], () => {}, context);
await entered.promise;
const queued = storage.commit([[entryWrite(second)]], () => {}, context);
const closing = store.close(context);
release.resolve();
expect(await running).toEqual({ stored: 1 });
await expect(queued).rejects.toMatchObject({ code: "released" });
await closing;
const again = await reopen(path);
expect(await again.storage.entry(first, context)).toBeDefined();
expect(await again.storage.entry(second, context)).toBeUndefined();
await again.store.close(context);
});
});
describe("durability", () => {
it("keeps every batch of a stored call across a crash: reopened, the seqs, rows, documents and ids are there", async () => {
const path = await storePath();
const { store, storage } = await counted(path);
const entry = await storage.mintId<EntryId>();
const document = await storage.mintId<DocumentId>();
const record = {
id: document,
kind: "n",
scope: { kind: "conversation", conversationId: ROOT_CONVERSATION_ID },
history: "rewindable",
fork: "asOf",
} as const;
const seqs: Seq[] = [];
await storage.commit(
[
[entryWrite(entry)],
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { n: 1 } } }],
[{ type: "document.change", id: document, content: { kind: "delta", version: 1, ops: [["s", ["n"], 2]] } }],
],
(_index, seq) => seqs.push(seq),
context,
);
const minted = await storage.mintId<EntryId>();
// A crash: the store is never closed; another process opens the file.
const { store: other, storage: reopened } = await reopen(path);
expect((await reopened.entry(entry, context))?.commitSeq).toBe(seqs[0]);
expect((await reopened.document(document, seqs[1]!, context))?.value).toEqual({ n: 1 });
expect((await reopened.document(document, "current", context))?.value).toEqual({ n: 2 });
expect(await reopened.mintId<EntryId>()).toBeGreaterThan(document);
expect(await commitWrites(reopened, [entryWrite(minted)], context)).toBe(seqs[2]! + 1);
await other.close(context);
await store.close(context);
});
});
describe("publication", () => {
it("publishes each commit of a call in a host batch of its own, after the call's one transaction", async () => {
const { store, storage, counts } = await counted();
const host = createHost({ store, onError: () => {} });
const session = new SessionImpl(storage);
const events: string[] = [];
session.followCommits((commit) => {
if (commit.kind !== "stored") return;
const batch = store.batch!;
events.push(`publish ${commit.seq} running=${batch.running}`);
batch.flushing(() => events.push(`flush ${commit.seq}`));
});
// Hold the first call, so the commits after it are sent together in the next one.
const calls: number[] = [];
const gate = Promise.withResolvers<void>();
const commit = storage.commit.bind(storage);
storage.commit = async (batches, receive, ctx) => {
calls.push(batches.length);
if (calls.length === 1) await gate.promise;
return commit(batches, receive, ctx);
};
const first = session.commit((tx) => tx.appendEntry(ROOT_CONVERSATION_ID, { kind: "a" }), context);
await new Promise((resolve) => setTimeout(resolve, 0));
const rest = [1, 2, 3].map((n) =>
session.commit((tx) => tx.appendEntry(ROOT_CONVERSATION_ID, { kind: `b${n}` }), context),
);
await new Promise((resolve) => setTimeout(resolve, 0));
const before = counts.transactions;
gate.resolve();
await Promise.all([first, ...rest]);
expect(calls).toEqual([1, 3]);
expect(counts.transactions - before).toBe(2);
const seq = store.lastSeq;
expect(events).toEqual(
[seq - 3, seq - 2, seq - 1, seq].flatMap((each) => [`publish ${each} running=true`, `flush ${each}`]),
);
await session.close(context);
await host.close(context);
await store.close(context);
});
});
describe("other users of the store", () => {
it("another owner's call waits for a series in its transaction; the series' receipts come first", async () => {
const { store, storage, faults } = await counted();
const other = await StoreStorage.open(store, "other", context);
const [first, second] = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
const entered = Promise.withResolvers<void>();
const release = Promise.withResolvers<void>();
faults.hold = (_sql, index) => {
if (index !== 2) return undefined;
entered.resolve();
return release.promise;
};
const events: string[] = [];
const series = storage.commit(
[[entryWrite(first)], [entryWrite(second)]],
(index, seq) => events.push(`harness ${index} ${seq}`),
context,
);
await entered.promise;
const waiting = other.commit(
[[{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }]],
(index, seq) => events.push(`other ${index} ${seq}`),
context,
);
await macrotask();
expect(events).toEqual([]);
const last = store.lastSeq;
release.resolve();
await Promise.all([series, waiting]);
expect(events).toEqual([`harness 0 ${last + 1}`, `harness 1 ${last + 2}`, `other 0 ${last + 3}`]);
expect((await outboxSeqs(store)).slice(-3)).toEqual([last + 1, last + 2, last + 3]);
await other.close(context);
await store.close(context);
});
it("a read during a series sees all of it or none of it", async () => {
const { store, storage, faults } = await counted();
const [first, second] = [await storage.mintId<EntryId>(), await storage.mintId<EntryId>()];
const entered = Promise.withResolvers<void>();
const release = Promise.withResolvers<void>();
// Held after the first batch's writes, in the second batch's check.
let claimReads = 0;
faults.hold = (sql) => {
if (!sql.includes("record_ids") || ++claimReads !== 2) return undefined;
entered.resolve();
return release.promise;
};
const series = storage.commit([[entryWrite(first)], [entryWrite(second)]], () => {}, context);
await entered.promise;
const read = Promise.all([storage.entry(first, context), storage.entry(second, context)]);
await macrotask();
release.resolve();
await series;
const [one, two] = await read;
expect([one === undefined, two === undefined]).toEqual([false, false]);
await store.close(context);
});
it("an export holds the queue: a series waits for it and is not in its archive", async () => {
const { store, storage } = await counted();
const entry = await storage.mintId<EntryId>();
const last = store.lastSeq;
const parts = store.export(context);
const manifest = await parts.next();
expect(manifest.value).toMatchObject({ kind: "manifest", lastSeq: last });
let settled = false;
const series = storage
.commit([[entryWrite(entry)]], () => {}, context)
.finally(() => {
settled = true;
});
await macrotask();
expect(settled).toBe(false);
await parts.return?.();
expect(await series).toEqual({ stored: 1 });
expect(store.lastSeq).toBe(last + 1);
await store.close(context);
});
});
describe("GC", () => {
it("keeps nothing of a settled call: its batches are collected", async () => {
setFlagsFromString("--expose-gc");
const gc = runInNewContext("gc") as () => void;
const { store, storage } = await counted();
const sent = async (): Promise<WeakRef<object>> => {
const batches = [[entryWrite(await storage.mintId<EntryId>())]];
await storage.commit(batches, () => {}, context);
return new WeakRef(batches);
};
const ref = await sent();
for (let round = 0; round < 10 && ref.deref() !== undefined; round++) {
await new Promise((resolve) => setImmediate(resolve));
gc();
}
expect(ref.deref()).toBeUndefined();
await store.close(context);
});
});

121
packages/durable/test/store-storage.test.ts
View file

@ -32,14 +32,24 @@ import { describe, expect, it } from "vitest";
import { StorageRequestError } from "../src/errors.ts";
import { idFromNumber } from "../src/ids.ts";
import { type CompactionResult, createRegistry, Harness, LiveDoc, type Tx } from "../src/index.ts";
import { SessionImpl } from "../src/session/session.ts";
import { StoreStorage } from "../src/storage/store/index.ts";
import type { ConversationId, DocumentId, EntryId, Id, Seq, Storage, StorageWrite, TaskId } from "../src/types.ts";
import { commitWrites } from "../src/testing/commit-writes.ts";
import type {
CommitResult,
ConversationId,
DocumentId,
EntryId,
Id,
Seq,
Storage,
StorageWrite,
TaskId,
} from "../src/types.ts";
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
import { openStoreStorage } from "./store-storage-support.ts";
const context = BACKGROUND_CONTEXT;
const adopt = (): void => {};
/** A storage that closes and reopens its store file after every commit (Durable's reopen harness). */
class ReopeningStorage implements Storage {
#current: Storage;
@ -51,10 +61,14 @@ class ReopeningStorage implements Storage {
this.#path = path;
}
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context): Promise<Seq> {
if (this.#closed) return this.#current.commit(writes, adopt, commitContext);
async commit(
batches: readonly (readonly StorageWrite[])[],
receive: (index: number, seq: Seq) => void,
commitContext: Context,
): Promise<CommitResult> {
if (this.#closed) return this.#current.commit(batches, receive, commitContext);
try {
return await this.#current.commit(writes, adopt, commitContext);
return await this.#current.commit(batches, receive, commitContext);
} finally {
await this.#current.close(context);
this.#current = await openStoreStorage(this.#path);
@ -127,7 +141,7 @@ describe("claims and close", () => {
reason: "args",
});
const other = await StoreStorage.open(store, "other", context);
await first.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
await commitWrites(first, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
expect(await other.conversation(ROOT_CONVERSATION_ID, context)).toBeUndefined();
await first.close(context);
await expect(first.conversation(ROOT_CONVERSATION_ID, context)).rejects.toThrow("StoreStorage is closed");
@ -139,7 +153,7 @@ describe("claims and close", () => {
it("waits at close for its calls underway, then releases its claims", async () => {
const store = await memoryStore();
const storage = await StoreStorage.open(store, "harness", context);
const commit = storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
const commit = commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
const closing = storage.close(context);
await commit;
await closing;
@ -153,14 +167,14 @@ describe("the store's semantics (D4)", () => {
const store = await memoryStore();
const storage = await StoreStorage.open(store, "harness", context);
const root = { type: "conversation", value: { id: ROOT_CONVERSATION_ID } } as const;
await storage.commit([root], adopt, context);
await expect(storage.commit([root], adopt, context)).rejects.toThrow("ID 1 already belongs to conversation");
await commitWrites(storage, [root], context);
await expect(commitWrites(storage, [root], context)).rejects.toThrow("ID 1 already belongs to conversation");
const entry = {
id: idFromNumber<EntryId>(await storage.mintId()),
conversationId: ROOT_CONVERSATION_ID,
kind: "m",
};
await storage.commit([{ type: "entry", value: entry }], adopt, context);
await commitWrites(storage, [{ type: "entry", value: entry }], context);
expect((await storage.entry(entry.id, context))?.entry).toEqual(entry);
await store.close(context);
});
@ -169,12 +183,12 @@ describe("the store's semantics (D4)", () => {
const store = await memoryStore();
const storage = await StoreStorage.open(store, "harness", context);
const entry = { id: idFromNumber<EntryId>(500), conversationId: ROOT_CONVERSATION_ID, kind: "m" };
await storage.commit(
await commitWrites(
storage,
[
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
{ type: "entry", value: entry },
],
adopt,
context,
);
expect(await storage.mintId()).toBe(501);
@ -198,7 +212,7 @@ async function recorded() {
outbox: true,
})) as ChordStore;
const storage = await StoreStorage.open(store, "harness", context);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
return { store, storage };
}
@ -211,7 +225,8 @@ async function history(storage: Storage): Promise<{ readonly rewindable: Documen
const seqs: Seq[] = [];
const base = (n: number) => ({ kind: "base", version: 1, value: { n } }) as const;
seqs.push(
await storage.commit(
await commitWrites(
storage,
[
{
type: "document.create",
@ -224,26 +239,25 @@ async function history(storage: Storage): Promise<{ readonly rewindable: Documen
content: base(0),
},
],
adopt,
context,
),
);
seqs.push(
await storage.commit(
await commitWrites(
storage,
[{ type: "document.change", id: rewindable, content: { kind: "delta", version: 1, ops: [["s", ["n"], 1]] } }],
adopt,
context,
),
);
seqs.push(
await storage.commit(
await commitWrites(
storage,
[
{ type: "document.change", id: latest, content: { kind: "delta", version: 1, ops: [["s", ["n"], 2]] } },
{ type: "document.retire", id: latest },
{ type: "document.create", record: { id: empty, kind: "e", scope: { kind: "session" } }, content: base(3) },
{ type: "document.retire", id: empty },
],
adopt,
context,
),
);
@ -323,11 +337,11 @@ describe("head markers (the entry_heads table)", () => {
(await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context, { outbox: true })) as ChordStore;
let store = await open();
let storage = await StoreStorage.open(store, "harness", context);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
const entry = async (head?: EntryId): Promise<EntryId> => {
const id = await storage.mintId<EntryId>();
const value = { id, conversationId: ROOT_CONVERSATION_ID, kind: "k", data: {}, ...(head ? { head } : {}) };
await storage.commit([{ type: "entry", value }], adopt, context);
await commitWrites(storage, [{ type: "entry", value }], context);
return id;
};
const first = await entry();
@ -372,7 +386,30 @@ describe("reads (slice 6b §1.4, §2.4)", () => {
});
describe("commits (slice 6b §1.3, §2.5)", () => {
it("adopt in the attached host's batch, and count four statements for an entry append and for a delta", async () => {
it("publishes each commit of a Session inside the attached host's batch, and plainly without a host", async () => {
const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context)) as ChordStore;
const storage = await StoreStorage.open(store, "harness", context);
const plain: boolean[] = [];
storage.publishing(() => plain.push(store.batch === undefined), context);
expect(plain).toEqual([true]);
const host = createHost({ store, onError: () => {} });
const session = new SessionImpl(storage);
const running: boolean[] = [];
session.followCommits(() => {
running.push(store.batch!.running);
});
await session.commitWith((tx) => tx.createRootConversation(), context);
await Promise.all([
session.commit((tx) => tx.appendEntry(ROOT_CONVERSATION_ID, { kind: "a" }), context),
session.commit((tx) => tx.appendEntry(ROOT_CONVERSATION_ID, { kind: "b" }), context),
]);
expect(running).toEqual([true, true, true]);
await session.close(context);
await host.close(context);
await store.close(context);
});
it("counts four statements for an entry append and for a delta, with a host attached", async () => {
const telemetry = new InMemoryTelemetryContext();
const span = await telemetry.startSpan({ name: "test" }, (each) => each);
const metered = withTelemetry(span, context);
@ -384,34 +421,26 @@ describe("commits (slice 6b §1.3, §2.5)", () => {
const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context)) as ChordStore;
const host = createHost({ store, onError: () => {} });
const storage = await StoreStorage.open(store, "harness", context);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
let running: boolean | undefined;
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
const entry = {
id: idFromNumber<EntryId>(await storage.mintId()),
conversationId: ROOT_CONVERSATION_ID,
kind: "m",
};
const before = counts();
await storage.commit(
[{ type: "entry", value: entry }],
() => {
running = store.batch!.running;
},
metered,
);
await storage.commit([[{ type: "entry", value: entry }]], () => {}, metered);
const appended = counts() - before;
expect(running).toBe(true);
const id = await storage.mintId<DocumentId>();
const record = { id, kind: "s", scope: { kind: "session" } } as const;
await storage.commit(
await commitWrites(
storage,
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { n: 0 } } }],
adopt,
context,
);
const changed = counts();
await storage.commit(
await commitWrites(
storage,
[{ type: "document.change", id, content: { kind: "delta", version: 1, ops: [["s", ["n"], 1]] } }],
adopt,
metered,
);
expect([appended, counts() - changed]).toEqual([4, 4]);
@ -539,10 +568,11 @@ describe("Durable's views as Chord sources on the store (slice 6b §3)", () => {
describe("review regressions", () => {
it('a head-marker entry whose kind is "refused" is returned, not taken for a request error', async () => {
const storage = await openStoreStorage(":memory:");
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
const first = await storage.mintId<EntryId>();
const marker = await storage.mintId<EntryId>();
await storage.commit(
await commitWrites(
storage,
[
{ type: "entry", value: { id: first, conversationId: ROOT_CONVERSATION_ID, kind: "message" } },
{
@ -550,7 +580,6 @@ describe("review regressions", () => {
value: { id: marker, conversationId: ROOT_CONVERSATION_ID, kind: "refused", head: first },
},
],
adopt,
context,
);
expect((await storage.findLatestHeadMarker(ROOT_CONVERSATION_ID, undefined, context))?.id).toBe(marker);
@ -558,16 +587,6 @@ describe("review regressions", () => {
expect(page.items.map(({ kind }) => kind)).toEqual(["refused", "message"]);
await storage.close(context);
});
it("an empty commit takes no seq", async () => {
const { store, storage } = await recorded();
const before = store.lastSeq;
const adopted: Seq[] = [];
expect(await storage.commit([], (seq) => adopted.push(seq), context)).toBe(before);
expect(store.lastSeq).toBe(before);
expect(adopted).toEqual([before]);
await store.close(context);
});
});
describe("GC", () => {
@ -577,7 +596,7 @@ describe("GC", () => {
const store = await memoryStore();
const opened = async (): Promise<WeakRef<StoreStorage>> => {
const storage = await StoreStorage.open(store, "harness", context);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
await commitWrites(storage, [{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
await storage.close(context);
return new WeakRef(storage);
};

2
packages/durable/test/tool-output-bench.ts
View file

@ -261,7 +261,7 @@ async function openStorage(backend: Backend, path: string): Promise<Storage> {
return openNodeJsonlStorage(path, context);
}
/** Measure every Storage commit. */
/** Measure every Storage commit call. */
function timed(storage: Storage, latencies: number[]): Storage {
return new Proxy(storage, {
get(target, property) {