sscoble/pi-mono
1
0
Fork 0
mirror of https://github.com/badlogic/pi-mono.git synced 2026-11-05 23:37:25 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

feat: Durable storage adopt callback inside the commit, watch types in Chord, explicit watch retirement

Browse source 
Breaking changes:
- Durable Storage.commit(writes, adopt, context): storages call adopt(seq) once,
  synchronously, when the batch is durable and before their next commit
- Chord traceIn(options, fn, context): context last
- WatchHandle and WatchEnd move to Chord (Durable re-exports them); end reasons
  ended (was session_closed) and failed (was session_failed); retirement is an
  explicit source event, and an ended watch drops its references into the Session
- Removed Session.documentState(), Conversation.viewState(), Harness.taskGraph()
  and DocumentState; use watchDoc(), watch(), watchTaskGraph()
This commit is contained in:
Mario Zechner 2026-10-22 10:52:50 +00:00
parent 3bc6eebacf
commit 92ab536024
61 changed files with 1025 additions and 1065 deletions
Download patch file Download diff file Expand all files Collapse all files

7
packages/durable/CHANGELOG.md
View file

@ -2,6 +2,13 @@
## [Unreleased]
### Breaking Changes
- `Storage.commit(writes, adopt, context)` takes an `adopt(seq)` callback, the Session's in-memory adoption of the batch: a storage calls it exactly once, synchronously, once the batch is durable and before its next commit starts, then resolves with `seq`, and never for a commit that fails before the batch is durable. Memory, SQLite, JSONL, and IndexedDB storages do; custom storages and wrappers must call or forward it, and the storage conformance suite checks it. A commit that rejects after its adoption fails the Session without rolling the adopted changes back.
- `Storage.mintId()`'s contract is narrowed to IDs greater than every ID minted since the storage was opened and every ID stored, also across reopen. The bundled storages already behave so.
- Watches use Chord's `WatchHandle` and `WatchEnd` (Durable re-exports both). End reasons are `stopped`, `cancelled`, `retired`, `ended` (was `session_closed`), `listener_error`, and `failed` (was `session_failed`). Retirement is an explicit event of the watched document incarnation (last frame `[["r", null]]`, then `retired`), never inferred from a `null` value. An ended watch drops its listener and every reference into its Session, so a stopped watch that is still referenced keeps only its last value.
- Removed the Chord state getters `Session.documentState()`, `Conversation.viewState()`, and `Harness.taskGraph()`, and the `DocumentState` type; use `watchDoc()`, `Conversation.watch()`, and `Harness.watchTaskGraph()`. Durable no longer uses Chord's `replicatedState()`.
## [1.4.0] - 2026-10-21
### Breaking Changes

33
packages/durable/README.md
View file

@ -473,30 +473,23 @@ Work that already started keeps the code it took: a running tool call finishes u
## Watching a Conversation
Everything a UI needs is committed state. `viewState()` returns the conversation's structural view as a read-only Chord state, updated after every commit that touches it:
```typescript
const view = await root.viewState(context);
view.subscribe((value) => {
// value.entries: the active transcript
// value.docs["pi.live"]: the running generation (streamed partial, retry, deferred) and tool calls (output, details)
// value.docs["pi.inbox"], value.docs["pi.usage"], value.docs["pi.agent"], value.docs["pi.provider"]
render(value);
});
// later: view.dispose();
```
`watch()` delivers the same view with the exact Chord operations of each commit, one callback at a time:
Everything a UI needs is committed state. `watch()` returns a watch of the conversation's structural view: its value at attachment, then each commit that touches it, with the exact Chord operations of that commit, one callback at a time:
```typescript
const watch = await root.watch(context);
render(watch.value); // the state at attachment
watch.start(async (value, ops) => {
// value.entries: the active transcript
// value.docs["pi.live"]: the running generation (streamed partial, retry, deferred) and tool calls (output, details)
// value.docs["pi.inbox"], value.docs["pi.usage"], value.docs["pi.agent"], value.docs["pi.provider"]
render(value);
await send(ops); // for example to a remote client that applies them
});
// later: await watch.stop();
```
Watches are Chord's `WatchHandle` (exported by `@earendil-works/chord`; Durable re-exports it). A watch ends once, and `closed` (or `stop()`) gives the reason: `stopped`, `cancelled` (its acquisition context was cancelled), `retired` (a document watch whose incarnation retired, after it delivered `[["r", null]]`), `ended` (the Harness closed), `listener_error`, or `failed` (the Harness failed). An ended watch keeps its last value and no longer references the Harness.
A slow watch keeps at most 100 undelivered frames. After that, the pending frames are replaced by one frame holding the whole newest view. A client that joins late or reconnects starts from the current view; nothing is replayed.
Partial answers and tool output are committed at most every 100 ms by default, so a crash loses at most that window. `settings.progress` changes the intervals; a host whose storage is remote can commit less often, for example `{ partialIntervalMs: 500, outputIntervalMs: 500 }`.
@ -709,11 +702,11 @@ decide: async (task, runtime, context) => {
## Task Graph
`harness.taskGraph(context)` shows every live task of the Session as one Chord state, for a task panel or debugging. Each node has its owner edge (`owner` task, or none for a task its conversation owns), its status, whether it is `background` or abort-marked, and the conversations it owns. `harness.watchTaskGraph(context)` delivers the same value as a watch, like a conversation's `watch()`.
`harness.watchTaskGraph(context)` watches every live task of the Session as one value, for a task panel or debugging, like a conversation's `watch()`. Each node has its owner edge (`owner` task, or none for a task its conversation owns), its status, whether it is `background` or abort-marked, and the conversations it owns.
```typescript
const graph = await harness.taskGraph(context);
graph.subscribe((value) => {
const graph = await harness.watchTaskGraph(context);
graph.start(async (value) => {
for (const node of Object.values(value.tasks)) {
const status = node.state.status === "waiting" ? `waiting on ${node.state.on.join(", ")}` : node.state.status;
console.log(`${node.id} ${node.kind} ${status}`, node.owner ?? `conversation ${node.conversationId}`);
@ -746,7 +739,7 @@ await root.commit(async (tx) => {
console.log(await harness.snapshot(Todos, root.id, context));
```
`harness.watchDoc()` and `harness.documentState()` observe one document like the view above. `HarnessOptions.conversationCreated(tx, conversation)` runs in every commit that creates or forks a conversation, including a tool's raw `tx.createConversation()`, so every conversation gets your documents; `init` in `createConversation()`, `fork()`, and `root()` writes per-call data in the same commit. An extension's tools, sections, and hooks read their own documents through `api` or `input.read`, and treat an absent one as its default ([`11-extension-state.ts`](test/examples/11-extension-state.ts)).
`harness.watchDoc()` watches one document like the view above. `HarnessOptions.conversationCreated(tx, conversation)` runs in every commit that creates or forks a conversation, including a tool's raw `tx.createConversation()`, so every conversation gets your documents; `init` in `createConversation()`, `fork()`, and `root()` writes per-call data in the same commit. An extension's tools, sections, and hooks read their own documents through `api` or `input.read`, and treat an absent one as its default ([`11-extension-state.ts`](test/examples/11-extension-state.ts)).
## Usage and Cost
@ -798,7 +791,7 @@ await database.transaction(async (transaction) => {
});
```
Custom backends can run the shared conformance suite with any Vitest- or Jest-compatible runner:
A custom backend's `commit(writes, adopt, context)` calls `adopt(seq)` once the batch is durable, synchronously and before its next commit starts, and then resolves with `seq`; a commit that fails before the batch is durable never calls it. `adopt` is the Session's in-memory adoption of the batch, so memory never shows a partial log. Custom backends can run the shared conformance suite, which checks this too, with any Vitest- or Jest-compatible runner:
```typescript
import { registerStorageConformance } from "@earendil-works/pi-durable/testing";
@ -820,7 +813,7 @@ The package root loads TypeBox, because the tool task validates arguments with p
Three rules decide what an error does:
1. **A storage error ends the Harness.** The call that hit it rejects with the error. Everything after it, and every pending wait, rejects with `SessionFailed` (its `cause` is that error); watches and event streams end with `session_failed`; running tasks get their abort signal; `onReport` gets the error once. The Harness then closes itself. Nothing is retried; a custom storage retries its own transient failures. Reopen to recover: commits are atomic, so the store is consistent, and resubmitting with the same `requestId` finds a submission that was already admitted.
1. **A storage error ends the Harness.** The call that hit it rejects with the error. Everything after it, and every pending wait, rejects with `SessionFailed` (its `cause` is that error); watches and event streams end with `failed`; running tasks get their abort signal; `onReport` gets the error once. The Harness then closes itself. Nothing is retried; a custom storage retries its own transient failures. Reopen to recover: commits are atomic, so the store is consistent, and resubmitting with the same `requestId` finds a submission that was already admitted.
2. **An invalid read fails only that call:** a storage read throws `StorageRequestError` for a bad cursor, an unknown conversation, or history a document does not keep. A read you cancelled fails only itself too. The exemption covers reads only: any error from `commit()` or `mintId()`, `StorageRequestError` included, ends the Harness as in rule 1.
3. **Your code fails only its own unit:** a throwing task phase faults that task, a throwing tool gets an error result, a throwing hook is reported and skipped (`beforeTool` blocks the call), and a throwing commit callback rolls back that commit. Listeners that throw, or return a rejected promise, are reported and the others still run. The exception is a host `RegistryReader` whose `snapshot()` throws: the Harness cannot schedule without it, so it fails the Harness like a storage error.

207
packages/durable/docs/spec.md
View file

@ -5,12 +5,12 @@ It uses existing package types as follows:
```ts
import type {
AttachedReplicatedState,
Context,
Draft,
JsonRepresentation,
JsonValue,
ReplicatedState,
WatchEnd,
WatchHandle,
} from "@earendil-works/chord";
import type { Op } from "@earendil-works/chord/delta";
import type {
@ -456,7 +456,7 @@ type HarnessSettings = {
readonly followUpMode?: QueueMode;
/** How long an idle conversation keeps its last context read in memory; busy ones always keep it. */
readonly contextRetentionMs?: number;
/** How long a loaded document no watch or document state observes stays in memory after its last use (section 4). */
/** How long a loaded document no watch observes stays in memory after its last use (section 4). */
readonly documentRetentionMs?: number;
/** What a failed run does with the inputs queued behind it (section 6). */
readonly afterFailedRun?: "keep" | "continue" | "withdraw";
@ -668,7 +668,7 @@ interface Conversation {
/** `{ background: true }` also aborts background work under the conversation (section 5.4). */
abort(context: Context, options?: { readonly background?: boolean }): Promise<void>;
waitForIdle(context: Context): Promise<void>;
viewState(context: Context): Promise<AttachedReplicatedState<ConversationView>>;
/** The structural view (section 9.3) as a watch. */
watch(context: Context): Promise<ConversationWatch>;
}
@ -699,8 +699,7 @@ interface Harness extends Session {
waitForIdle(context: Context): Promise<void>;
/** Session total of every conversation's `pi.usage` (section 8.6). */
usage(context: Context): Promise<UsageState>;
/** Live tasks as a structural view (section 9.5). */
taskGraph(context: Context): Promise<AttachedReplicatedState<TaskGraph>>;
/** Live tasks as a structural view (section 9.5), as a watch. */
watchTaskGraph(context: Context): Promise<TaskGraphWatch>;
}
@ -1035,17 +1034,15 @@ one to find out. Blocked reasons are derived, never stored. Queued and placed
submissions complete the view. Finished
tasks, settled submissions, transcripts, and documents use their own APIs.
`Conversation.viewState()` returns its current structural mount as a disposable
read-only Chord state. `Conversation.watch()` atomically captures that immutable
revision and registers for later exact complete-commit frames. Its `WatchHandle`
uses the same serialized asynchronous, bounded-buffer contract as `watchDoc()`
in section 9.2. Neither carries semantic events or owns a second persistence
authority.
`Conversation.watch()` atomically captures the immutable revision of its
current structural mount and registers for later exact complete-commit frames.
Its `WatchHandle` uses the same serialized asynchronous, bounded-buffer contract
as `watchDoc()` in section 9.2. It carries no semantic events and owns no second
persistence authority.
`close()` seals mutation admission and task reservation, signals invocations,
and ends every document state, view state, and watch at that seal: a commit
that settles during close publishes nothing to them, and a state keeps its last
value. Outside the Session line it lets already-admitted storage commits settle,
and ends every watch at that seal (`ended`): a commit that settles during close
publishes nothing to them, and a watch keeps its last value. Outside the Session line it lets already-admitted storage commits settle,
joins task, tool, and hook invocations, including code that ignores its signal,
then closes Storage. Once `close()` resolves, no invocation code of this Harness
runs and no handler holds Storage, so a new Harness may open the same Storage.
@ -1297,13 +1294,6 @@ interface Session extends DocumentObserver {
snapshotAsOf<T extends JsonObject>(token: RewindableConversationDocToken<T>, conversationId: ConversationId, at: EntryId, context: Context): Promise<Readonly<T> | undefined>;
snapshotAsOf<T extends JsonObject, I extends JsonValue>(token: RewindableConversationDocFamilyToken<T, I>, conversationId: ConversationId, key: string, at: EntryId, context: Context): Promise<Readonly<T> | undefined>;
documentState<T extends JsonObject>(token: SessionDocToken<T>, context: Context): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject>(token: ConversationDocToken<T>, conversationId: ConversationId, context: Context): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject>(token: TaskDocToken<T>, taskId: TaskId, context: Context): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(token: SessionDocFamilyToken<T, I>, key: string, context: Context): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(token: ConversationDocFamilyToken<T, I>, conversationId: ConversationId, key: string, context: Context): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(token: TaskDocFamilyToken<T, I>, taskId: TaskId, key: string, context: Context): Promise<DocumentState<T> | undefined>;
}
interface Tx {
@ -1388,7 +1378,7 @@ registered and ordinary documents are not scanned at open.
- The first acquisition of one logical address is memoized before awaiting. Later
acquisitions in that transaction return the same draft; for a missing family,
the first call's detached seed wins and later seeds are ignored.
- `snapshot()`, `snapshotAsOf()`, `documentState()`, and `watchDoc()` never create
- `snapshot()`, `snapshotAsOf()`, and `watchDoc()` never create
or persist migration. They return `undefined` when the requested incarnation is
absent and migrate a reconstructed value only in memory.
- Task-scoped `tx.doc()` validates against the transaction's latest candidate task
@ -1405,7 +1395,7 @@ registered and ordinary documents are not scanned at open.
- `snapshot()` returns the current shareable immutable revision. Mutation of it
or any retained descendant is unsupported; callers that need mutable ownership
must copy it first.
- `documentState()` returns a disposable read-only Chord state bound to one committed incarnation.
- `watchDoc()` returns a watch bound to one committed incarnation (section 9.2).
- `tx.doc()` returns one revocable Astra overlay `Draft<T>` for the transaction.
- Historical reads never create documents in the past.
@ -1601,8 +1591,8 @@ no migrate -> reject older stored version
```
`migrate()` is pure and returns a complete current-version value. Migration is
access-driven: `tx.doc()`, `snapshot()`, `snapshotAsOf()`, `documentState()`, and
`watchDoc()` reconstruct and migrate through the token supplied to that call.
access-driven: `tx.doc()`, `snapshot()`, `snapshotAsOf()`, and `watchDoc()`
reconstruct and migrate through the token supplied to that call.
Harness open does not sweep ordinary documents.
- Read-only access migrates only in memory and never writes. It may cache a
@ -1666,8 +1656,8 @@ inside the creating transaction lazily reads the detached source, migrates when
required, and replaces the copy with one ordinary child create containing the
final prepared value. Definition-free copies publish explicit `document.copy`
metadata rather than a value. That metadata announces Storage-backed initial
state and is never interpreted as a document value. Document states, watches,
and mounted views hydrate by capturing their baseline and subscription
state and is never interpreted as a document value. Watches and mounted views
hydrate by capturing their baseline and subscription
atomically on the Session line: a later commit already present becomes the
baseline, while one committed after registration is delivered. Publications
and watches are convergence mechanisms, not audit streams. A mounted aggregate
@ -1678,7 +1668,7 @@ Session-line operation so it never exposes a mixture from one commit.
The Session keeps the documents it loads or creates in memory, as Chord
trackers, so reads and changes need no Storage access. The cache is bounded by
use, not by history: a document no watch or document state observes leaves it
use, not by history: a document no watch observes leaves it
once it was not read or written for `documentRetentionMs` (default ten
minutes; `0` drops it right after use), and the next access loads it from
Storage. An observed document never leaves. Eviction runs on the mutation line,
@ -1696,8 +1686,14 @@ sets them for a plain Session; the Harness reads its settings at each use.
A Session commit callback may be asynchronous. It owns the Session mutation
line through callback execution, preparation, storage settlement, committed
baseline adoption, and publication enqueue. Commit observers run on the
line; document-state and watch user callbacks run later.
baseline adoption, and publication enqueue. Adoption runs inside
`Storage.commit()`: the Session passes an `adopt(seq)` callback, which the
storage calls synchronously once the batch is durable and before its next
commit (section 10), so memory always shows a complete prefix of the log. The
callback keeps its own error: a failed adoption fails the Session once
`Storage.commit()` returns. A commit that rejects after its adoption fails the
Session too, and its adopted changes stay adopted. Commit observers run on the
line; watch listeners run later.
External model, process, tool, network, and human effects run outside it.
`subscribeCommits()` observes complete immutable publications synchronously on
the line after adoption. `subscribeClose()` observes close synchronously when it
@ -1706,8 +1702,8 @@ Harness ends waits, watches, and streams there, and signals task invocations
after every close listener has run, so what a task holds ends with the Session's
reason rather than as cancelled. Both return idempotent
disposers; their listeners must not block or call Session APIs, and one that
throws is reported without affecting the others (section 10.1). Document-state
subscribers and watch listeners still run later, off the line.
throws is reported without affecting the others (section 10.1). Watch listeners
still run later, off the line.
```ts
await session.commit(async tx => {
@ -4610,53 +4606,18 @@ aborted directly (section 8.3).
## 9. Document observation and Chord
### 9.1 Document state
### 9.1 Watches
Chord's existing replicated-state layer exposes this source-adoption contract:
Durable exposes committed state only through watches: Chord's `WatchHandle`
and its end type `WatchEnd`, exported by `@earendil-works/chord` (Durable
re-exports both):
```ts
interface ReplicatedStateSourceFrame<T> {
readonly cursor: number;
readonly value: T;
readonly ops: readonly Op[];
readonly context: Context;
}
interface ReplicatedStateSourceAttachment<T> {
/** Fixed immutable snapshot captured at the atomic attachment boundary. */
readonly snapshot: { readonly value: T; readonly cursor: number };
/** Install the sole listener and synchronously drain every buffered frame. */
activate(listener: (frame: ReplicatedStateSourceFrame<T>) => void): void;
dispose(): void;
}
interface ReplicatedStateSource<T> {
/** Atomically capture a snapshot and begin buffering every later commit. */
attach(): ReplicatedStateSourceAttachment<T>;
}
interface ReplicatedStateSourceOptions {
readonly onError?: (error: Error) => void;
}
interface AttachedReplicatedState<T> extends ReplicatedState<T> {
readonly value: T;
dispose(): void;
}
function replicatedState<T>(
source: ReplicatedStateSource<T>,
options?: ReplicatedStateSourceOptions,
): AttachedReplicatedState<T>;
type DocumentState<T extends JsonObject> =
AttachedReplicatedState<Readonly<T> | null>;
type WatchEnd =
| { readonly reason: "stopped" | "cancelled" | "session_closed" | "retired" }
| { readonly reason: "stopped" | "cancelled" | "retired" | "ended" }
| { readonly reason: "listener_error"; readonly error: Error }
/** The Session failed; `error` is what failed it, usually a storage error (section 10.1). */
| { readonly reason: "session_failed"; readonly error: unknown };
/** The source failed; for Durable, what failed the Session, usually a storage error (section 10.1). */
| { readonly reason: "failed"; readonly error: unknown };
interface WatchHandle<T> {
/** Acquisition revision before start; latest delivered immutable revision afterward. */
@ -4681,30 +4642,32 @@ interface DocumentObserver {
}
```
`documentState()` attaches Durable's committed document stream to Chord before
returning. The returned state is already hydrated with the exact shareable
immutable revision, has no mutation methods, and publishes every later exact
committed revision and operation batch without another tracker, value copy, or
re-diff. Disposing it unregisters only that observation; Durable remains the sole
document mutator.
A watch ends once, with the first reason that applies: `stopped` by `stop()`;
`cancelled` when its acquisition `Context` is cancelled; `retired` when its
document incarnation retired; `ended` when the Session closed; `listener_error`
when its listener threw or rejected; `failed` when the Session failed.
State and watch acquisition never create; absent lookup returns `undefined`.
Successful acquisition binds one concrete incarnation. Retirement publishes a
JSON `null` root replacement and ends that incarnation's stream. If the state
remains exposed, consumers see `null`, never stale state. A later recreation
requires acquiring a new state or watch.
Watch acquisition never creates; absent lookup returns `undefined`. Successful
acquisition binds one concrete incarnation. Retirement is an explicit event of
that incarnation, never inferred from a `null` value: it delivers a JSON `null`
root replacement `[["r", null]]` as the last frame, then ends the watch as
`retired`. A later recreation requires acquiring a new watch.
Session close ends every state at its seal: the state keeps its last value and
Session close ends every watch at its seal: the watch keeps its last value and
receives no frame from a commit that settles during close.
Each document state assigns its own in-memory contiguous Chord delivery sequence.
Durable does not persist or expose that sequence through `WatchHandle`. States and watches retain immutable revisions independently of the loaded tracker
cache. Reopen creates a new state lifetime and hydration.
Termination drops every reference the watch holds into its source: its listener,
its subscriptions and the closures that detach, report, and replace, and its
cancellation hook. A stopped watch that is still referenced keeps only its last
value alive, never the Session or the listener; a callback already running keeps
what it needs until it settles. Watches retain immutable revisions independently
of the loaded tracker cache. Reopen creates a new watch lifetime.
Internally, state attachment uses Chord's synchronous atomic snapshot-and-register
boundary: the snapshot includes every commit before attachment, and every later
exact frame is buffered until activation. This prevents a snapshot from being
paired with operations based on a newer unseen revision.
Acquisition captures the value and registers for later commits atomically, on
the Session line: the acquisition value includes every commit before
acquisition, and every later exact frame is buffered until `start()`. This
prevents a value from being paired with operations based on a newer unseen
revision.
### 9.2 `watchDoc`
@ -4785,8 +4748,8 @@ type ConversationView = {
These kinds and their fields are public protocol. Third-party documents are
exposed through their own Chord objects, not mounted.
The Harness keeps at most one mount per conversation. The first `viewState()`,
`watch()`, or event attachment builds its revision on the Session line from the
The Harness keeps at most one mount per conversation. The first `watch()` or
event attachment builds its revision on the Session line from the
committed active transcript and documents; the mount is dropped when its last
observer detaches. A mount derives each next revision from the Session's
`subscribeCommits()` publications, like `watchDoc()`: every publication is
@ -4813,9 +4776,8 @@ redundant nonempty batch remains a real publication. Mounted document revisions
are structurally shared under the trusted immutability contract. The mount
performs no semantic projection and owns no persistence.
`Conversation.viewState()` exposes the mount as a disposable read-only Chord
state. `Conversation.watch()` exposes the same mount through the serialized
exact-frame watch of section 9.2, with bounded pending frames and full-value
`Conversation.watch()` exposes the mount through the serialized exact-frame
watch of section 9.2, with bounded pending frames and full-value
overflow replacements.
### 9.4 Agent events (experimental)
@ -5030,7 +4992,6 @@ type TaskGraph = {
type TaskGraphWatch = WatchHandle<TaskGraph>;
interface Harness {
taskGraph(context: Context): Promise<AttachedReplicatedState<TaskGraph>>;
watchTaskGraph(context: Context): Promise<TaskGraphWatch>;
}
```
@ -5055,8 +5016,7 @@ node that lists it; a later task there is a top-level node of that conversation.
A UI that places it under its parent reads the conversation's owner edge from
`ConversationRecord.owner`, which `ConversationView.conversation` also carries.
The Harness keeps at most one graph mount. The first `taskGraph()` or
`watchTaskGraph()` builds its revision on the Session line from the committed
The Harness keeps at most one graph mount. The first `watchTaskGraph()` builds its revision on the Session line from the committed
live tasks and the conversations they own, one owner scan per live task, and
holds the line while it does; the mount is dropped when its last observer
detaches. It advances from `subscribeCommits()` publications like the
@ -5065,14 +5025,13 @@ that sets or deletes `tasks[id]`, or sets a node's `conversations`. A
publication that changes no node creates no revision. Task and conversation
records are immutable values; a node shares nothing with them.
`taskGraph()` returns the mount as a disposable read-only Chord state.
`watchTaskGraph()` returns it through the serialized exact-frame watch of
section 9.2. Both read only: neither enables scheduling. Close ends them at its
seal like every other state and watch (section 2.2).
`watchTaskGraph()` returns the mount through the serialized exact-frame watch
of section 9.2. It reads only: it never enables scheduling. Close ends it at its
seal like every other watch (section 2.2).
```ts
const graph = await harness.taskGraph(context);
graph.subscribe((value) => {
const graph = await harness.watchTaskGraph(context);
graph.start(async (value) => {
for (const node of Object.values(value.tasks)) {
const where = node.owner === undefined ? `conversation ${node.conversationId}` : `task ${node.owner}`;
render(`${node.id} ${node.kind} ${node.state.status} under ${where}`);
@ -5179,10 +5138,16 @@ type StorageWrite =
* gaps. Once commit() resolves, later reads through that Storage observe it.
*/
interface Storage {
commit(writes: readonly StorageWrite[], context: Context): Promise<Seq>;
/**
* Allocate from the one global numeric namespace, above every ID minted or stored before, also across reopen: scans
* order by ID as creation order, and the Harness relies on it. The generic brand is compile-time only.
* Persist the batch atomically. Once it is durable, call `adopt(seq)` exactly once, synchronously and before this
* storage's next commit starts, then resolve with `seq`; a commit that fails before the batch is durable never calls
* it. `adopt` must not throw.
*/
commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq>;
/**
* Allocate from the one global numeric namespace, above every ID minted since this storage was opened and every ID
* stored, also across reopen: scans order by ID as creation order, and the Harness relies on it. The generic brand
* is compile-time only.
*/
mintId<I extends Id<string>>(): Promise<I>;
@ -5296,8 +5261,8 @@ may not have committed, so a host that resubmits after reopening reuses its
and every wait that starts later rejects with `SessionFailed`, whose `cause` is
the first error; so does every Storage call still underway when it returns,
whatever it returned.
- Watches and event streams end with `{ reason: "session_failed", error }`;
attached replicated states stop updating and keep their last value.
- Watches and event streams end with `{ reason: "failed", error }` and keep
their last value.
- Running task invocations get their abort signal. `HarnessOptions.onReport`
gets the error once, after the Session is sealed, so a report handler that
calls back finds it failed.
@ -5309,7 +5274,7 @@ may not have committed, so a host that resubmits after reopening reuses its
ignores its abort signal keeps the backend open and `closed` pending.
- A failing `Storage.close()` fails the Session too, unless it already failed;
the `close()` call rejects with it, and `closed` settles `failed`. Waits and
watches had already ended as for a close (`session_closed`).
watches had already ended as for a close (`ended`).
The exceptions fail only their call, and apply to reads alone: a
`StorageRequestError`, which a Storage throws for an invalid read, with no
@ -5320,7 +5285,9 @@ exempt: once a batch is admitted, whether it committed is unknown.
A commit callback that catches a failed read cannot commit afterwards: the Session
checks its health after the callback. Adopting a committed batch in memory that
fails also fails the Session, since memory would be behind storage.
fails also fails the Session, since memory would be behind storage; so does a
`Storage.commit()` that resolves without adopting, or rejects after it adopted
(the adopted changes are not rolled back).
**The Session's own bookkeeping fails it too.** A throw in a commit the
scheduler makes for itself (reconciliation, reservation, a fault or terminal
@ -5339,8 +5306,8 @@ throw are reported and skipped, except `beforeTool`, whose throw blocks the call
a commit callback, `conversationCreated`, `init`, or a document's `initial`,
`migrate`, or `checkpointWhen` that throws rolls its commit back and rejects it.
Callbacks that must not fail their caller are contained: a host
`subscribeCommits()` or `subscribeClose()` listener, a watch listener, a
replicated-state subscriber, and a view observer that throws, or returns a
`subscribeCommits()` or `subscribeClose()` listener, a watch listener, and a
view observer that throws, or returns a
promise that rejects, are reported, and the others still run; a watch whose
listener throws ends with `listener_error`. A throwing or rejecting `onReport` is
dropped, since nothing is left to report it to. A throwing `now()` is reported
@ -5521,13 +5488,13 @@ These are contracts, not invitations to add defensive machinery:
- **Long transactions:** an async commit callback holds the Session mutation
line. Never await models, tools, processes, network calls, humans, a nested
Session commit, or a Session waiter inside it. Use methods on the current `Tx`.
- **Reads inside a commit:** `snapshot()`, `documentState()`, `watchDoc()`,
- **Reads inside a commit:** `snapshot()`, `watchDoc()`,
`getTask()`, and the other Session, Harness, and runtime reads may queue on the
mutation line, for example to load a document not in memory. Inside a commit
callback, which holds that line, such a read never resolves, and the line stays
held. Only a cache hit returns, so the hang is intermittent. Read through `tx`.
- **Explicit creation:** only typed `tx.doc()` creates an absent document. Snapshot,
state, and watch lookup return `undefined` instead.
- **Explicit creation:** only typed `tx.doc()` creates an absent document. Snapshot
and watch lookup return `undefined` instead.
- **Family initialization:** the first acquisition of an absent family address
selects its seed. Existing instances and later calls ignore seeds; a seed is
neither identity nor an update.

5
packages/durable/src/harness/events.ts
View file

@ -1,4 +1,4 @@
import type { Context, JsonValue } from "@earendil-works/chord";
import type { Context, JsonValue, WatchEnd } from "@earendil-works/chord";
import type { Op, Path } from "@earendil-works/chord/delta";
import type { AssistantMessage, Message, Usage } from "@earendil-works/pi-ai";
import { CommittedWatch } from "../session/observation.ts";
@ -11,7 +11,6 @@ import type {
SubmissionId,
SubmissionRecord,
TaskId,
WatchEnd,
} from "../types.ts";
import { AgentDoc } from "./agent.ts";
import type { InboxItem, InboxState } from "./inbox.ts";
@ -193,7 +192,7 @@ export async function watchEvents(
const events = translate(conversationId, before, after, ops, publication, held);
if (events.length > 0) watch.advance(events, [], commitContext);
},
closeSession: (failure) => watch.closeSession(failure),
end: (failure) => watch.end(failure),
};
},
context,

14
packages/durable/src/harness/harness.ts
View file

@ -1,4 +1,4 @@
import type { AttachedReplicatedState, Context, JsonValue } from "@earendil-works/chord";
import type { Context, JsonValue } from "@earendil-works/chord";
import { withAbortSignal, withoutAbortSignal } from "@earendil-works/chord/context";
import type { Models } from "@earendil-works/pi-ai";
import {
@ -45,7 +45,7 @@ import { ProviderDoc } from "./provider.ts";
import { BUILTIN_TASKS } from "./registry.ts";
import { type InvocationBinding, TaskScheduler } from "./scheduler.ts";
import { Submissions } from "./submissions.ts";
import { type TaskGraph, TaskGraphView, type TaskGraphWatch } from "./task-graph.ts";
import { TaskGraphView, type TaskGraphWatch } from "./task-graph.ts";
import { DURABLE_TELEMETRY_SCHEMA } from "./telemetry.ts";
import type {
Agent,
@ -71,7 +71,7 @@ import type {
} from "./types.ts";
import { addUsageState, UsageDoc, type UsageState } from "./usage.ts";
import { scanAll } from "./util.ts";
import { type ConversationView, ConversationViews } from "./view.ts";
import { ConversationViews } from "./view.ts";
const SCAN_PAGE_SIZE = 256;
@ -175,10 +175,6 @@ class ConversationImpl<Tool extends ToolRegistration> implements Conversation {
return this.#host.tasks.waitForIdle(this.id, context);
}
viewState(context: Context): Promise<AttachedReplicatedState<ConversationView>> {
return this.#host.views.state(this.id, context);
}
watch(context: Context): Promise<ConversationWatch> {
return this.#host.views.watch(this.id, context);
}
@ -364,10 +360,6 @@ class HarnessImpl<Tool extends ToolRegistration> extends SessionImpl implements
return total;
}
taskGraph(context: Context): Promise<AttachedReplicatedState<TaskGraph>> {
return this.#taskGraph.state(context);
}
watchTaskGraph(context: Context): Promise<TaskGraphWatch> {
return this.#taskGraph.watch(context);
}

23
packages/durable/src/harness/task-graph.ts
View file

@ -1,7 +1,7 @@
import { type AttachedReplicatedState, type Context, type JsonValue, replicatedState } from "@earendil-works/chord";
import type { Context, JsonValue, WatchHandle } from "@earendil-works/chord";
import { BACKGROUND_CONTEXT, withoutAbortSignal } from "@earendil-works/chord/context";
import { applyImmutable, type Op } from "@earendil-works/chord/delta";
import { CommittedStateSource, CommittedWatch } from "../session/observation.ts";
import { CommittedWatch } from "../session/observation.ts";
import { contained, type SessionImpl } from "../session/session.ts";
import type {
CommitPublication,
@ -11,7 +11,6 @@ import type {
TaskId,
TaskOutcome,
TaskRecord,
WatchHandle,
} from "../types.ts";
import { type BlockedReason, type BlockedSubject, blockedSubject } from "./scheduler.ts";
import { closedError, scanAll } from "./util.ts";
@ -57,7 +56,7 @@ export type TaskGraph = {
export type TaskGraphWatch = WatchHandle<TaskGraph>;
type AnyTaskRecord = TaskRecord<JsonValue, JsonValue, JsonValue>;
type Observer = CommittedStateSource<TaskGraph> | CommittedWatch<TaskGraph>;
type Observer = CommittedWatch<TaskGraph>;
type Mount = {
value: TaskGraph;
readonly observers: Set<Observer>;
@ -98,7 +97,7 @@ export class TaskGraphView {
this.#closed = true;
for (const observer of [...(this.#mount?.observers ?? [])]) {
contained(
() => observer.closeSession(this.#session.failure),
() => observer.end(this.#session.failure),
(error) => session.report(error),
);
}
@ -106,20 +105,6 @@ export class TaskGraphView {
});
}
/** A disposable read-only Chord state of the graph. */
async state(context: Context): Promise<AttachedReplicatedState<TaskGraph>> {
const { observer, detach } = await this.#attach(
(value, release) => new CommittedStateSource<TaskGraph>(value, release),
context,
);
try {
return replicatedState(observer, { onError: (error) => this.#session.report(error) });
} catch (error) {
detach();
throw error;
}
}
/** A serialized exact-frame watch of the graph; cancelling `context` stops it. */
async watch(context: Context): Promise<TaskGraphWatch> {
const { observer } = await this.#attach(

18
packages/durable/src/harness/types.ts
View file

@ -1,4 +1,4 @@
import type { AttachedReplicatedState, Context, JsonValue } from "@earendil-works/chord";
import type { Context, JsonValue, WatchHandle } from "@earendil-works/chord";
import type {
AssistantMessage,
CacheRetention,
@ -41,9 +41,8 @@ import type {
TaskRecord,
TaskState,
Tx,
WatchHandle,
} from "../types.ts";
import type { TaskGraph, TaskGraphWatch } from "./task-graph.ts";
import type { TaskGraphWatch } from "./task-graph.ts";
import type { UsageState } from "./usage.ts";
import type { ConversationView } from "./view.ts";
@ -558,7 +557,7 @@ export type HarnessSettings = {
*/
readonly contextRetentionMs?: number;
/**
* How long a loaded document that no watch or document state observes stays in memory after its last read or write;
* How long a loaded document that no watch observes stays in memory after its last read or write;
* the next access loads it from Storage. Observed documents always stay. Default ten minutes; `0` drops one right
* after use; `Infinity` keeps them. Anything else that is not a number of milliseconds means the default.
*/
@ -771,9 +770,7 @@ export interface Conversation {
* owned, transitively, by its non-background tasks.
*/
waitForIdle(context: Context): Promise<void>;
/** The structural view (spec §9.3) as a disposable read-only Chord state. */
viewState(context: Context): Promise<AttachedReplicatedState<ConversationView>>;
/** The structural view as a serialized exact-frame watch with bounded pending frames. */
/** The structural view (spec §9.3) as a serialized exact-frame watch with bounded pending frames. */
watch(context: Context): Promise<ConversationWatch>;
}
@ -830,9 +827,10 @@ export interface Harness extends Session {
waitForIdle(context: Context): Promise<void>;
/** Session total: every conversation's `pi.usage` summed. */
usage(context: Context): Promise<UsageState>;
/** Every live task with its owner edge, status, and owned conversations (spec §9.5), as a disposable Chord state. */
taskGraph(context: Context): Promise<AttachedReplicatedState<TaskGraph>>;
/** The task graph as a serialized exact-frame watch with bounded pending frames. */
/**
* Every live task with its owner edge, status, and owned conversations (spec §9.5), as a serialized exact-frame
* watch with bounded pending frames.
*/
watchTaskGraph(context: Context): Promise<TaskGraphWatch>;
}

25
packages/durable/src/harness/view.ts
View file

@ -1,7 +1,7 @@
import { type AttachedReplicatedState, type Context, type JsonValue, replicatedState } from "@earendil-works/chord";
import type { Context, JsonValue, WatchHandle } from "@earendil-works/chord";
import { withoutAbortSignal } from "@earendil-works/chord/context";
import { applyImmutable, type NonEmptyPath, type Op, type Path } from "@earendil-works/chord/delta";
import { CommittedStateSource, CommittedWatch } from "../session/observation.ts";
import { CommittedWatch } from "../session/observation.ts";
import { contained, type SessionImpl } from "../session/session.ts";
import type {
CommitPublication,
@ -12,7 +12,6 @@ import type {
EntryRecord,
JsonObject,
Storage,
WatchHandle,
} from "../types.ts";
import { AgentDoc } from "./agent.ts";
import { activeEntries, captureContextBounds } from "./context.ts";
@ -42,7 +41,8 @@ export type ViewObserver = {
publication: CommitPublication,
context: Context,
): void;
closeSession(failure: { readonly error: unknown } | undefined): void;
/** The Session closed, or failed with `failure`. */
end(failure: { readonly error: unknown } | undefined): void;
};
const MOUNTED = [
@ -94,7 +94,7 @@ export class ConversationViews {
for (const mount of this.#mounts.values())
for (const observer of [...mount.observers]) {
contained(
() => observer.closeSession(this.#session.failure),
() => observer.end(this.#session.failure),
(error) => this.report(error),
);
}
@ -107,21 +107,6 @@ export class ConversationViews {
this.#session.report(error);
}
/** A disposable read-only Chord state of the view. */
async state(id: ConversationId, context: Context): Promise<AttachedReplicatedState<ConversationView>> {
const { observer, detach } = await this.attach(
id,
(value, release) => new CommittedStateSource<ConversationView>(value, release),
context,
);
try {
return replicatedState(observer, { onError: (error) => this.#session.report(error) });
} catch (error) {
detach();
throw error;
}
}
/** A serialized exact-frame watch of the view; cancelling `context` stops it. */
async watch(id: ConversationId, context: Context): Promise<WatchHandle<ConversationView>> {
const { observer } = await this.attach(

4
packages/durable/src/index.ts
View file

@ -1,3 +1,4 @@
export type { WatchEnd, WatchHandle } from "@earendil-works/chord";
export { defineDoc, defineDocFamily } from "./documents.ts";
export {
AssistantEntry,
@ -155,7 +156,6 @@ export type {
DocumentReader,
DocumentRecord,
DocumentSemantics,
DocumentState,
DocumentWatch,
Entry,
EntryDraft,
@ -210,7 +210,5 @@ export type {
Tx,
TypedEntry,
TypedEntryDraft,
WatchEnd,
WatchHandle,
} from "./types.ts";
export { ROOT_CONVERSATION_ID } from "./types.ts";

202
packages/durable/src/session/observation.ts
View file

@ -1,167 +1,41 @@
import type {
Context,
JsonValue,
ReplicatedStateSource,
ReplicatedStateSourceAttachment,
ReplicatedStateSourceFrame,
} from "@earendil-works/chord";
import type { Context, JsonValue, WatchEnd, WatchHandle } from "@earendil-works/chord";
import { withoutAbortSignal } from "@earendil-works/chord/context";
import type { Op } from "@earendil-works/chord/delta";
import { errorMessage } from "../errors.ts";
import type { JsonObject, WatchEnd, WatchHandle } from "../types.ts";
import type { JsonObject } from "../types.ts";
export type ObservedDocumentValue = Readonly<JsonObject> | null;
/** Maximum exact committed frames retained behind one unavailable watch listener. */
const MAX_PENDING_WATCH_FRAMES = 100;
/** Canonical terminal update for a retired document incarnation. */
export const RETIREMENT_OPERATIONS: readonly Op[] = [["r", null]];
/**
* Session-to-Chord bridge owned one-to-one by one attached state: a document, or a conversation view. A `null` value
* retires it. @internal
*/
export class CommittedStateSource<T = ObservedDocumentValue> implements ReplicatedStateSource<T> {
readonly #attachments = new Set<SessionSourceAttachment<T>>();
#release: (() => void) | undefined;
/** Released on disposal. */
#value: T | undefined;
#cursor = 0;
#retired = false;
#closed = false;
constructor(value: T, release: () => void) {
this.#value = value;
this.#release = release;
}
attach(): ReplicatedStateSourceAttachment<T> {
if (this.#closed) throw new Error("State source is closed");
const attachment = new SessionSourceAttachment({ value: this.#value as T, cursor: this.#cursor }, () => {
this.#attachments.delete(attachment);
if (this.#attachments.size === 0) this.#finishDisposal();
});
this.#attachments.add(attachment);
return attachment;
}
advance(value: T, ops: readonly Op[], context: Context): void {
if (this.#closed || this.#retired) return;
this.#value = value;
this.#cursor += 1;
if (value === null) this.#retired = true;
const frame: ReplicatedStateSourceFrame<T> = {
cursor: this.#cursor,
value,
ops,
context,
};
for (const attachment of [...this.#attachments]) attachment.publish(frame);
}
/** Stop publishing; the attached states keep their last value. A failure ends them the same way. */
closeSession(): void {
if (this.#closed) return;
for (const attachment of [...this.#attachments]) attachment.dispose();
this.#finishDisposal();
}
#finishDisposal(): void {
if (this.#closed) return;
this.#closed = true;
this.#value = undefined;
const release = this.#release;
this.#release = undefined;
release?.();
}
}
class SessionSourceAttachment<T> implements ReplicatedStateSourceAttachment<T> {
readonly snapshot: { readonly value: T; readonly cursor: number };
#release: (() => void) | undefined;
readonly #frames: ReplicatedStateSourceFrame<T>[] = [];
#listener: ((frame: ReplicatedStateSourceFrame<T>) => void) | undefined;
#activated = false;
#disposed = false;
#scheduled = false;
#delivering = false;
constructor(snapshot: { readonly value: T; readonly cursor: number }, release: () => void) {
this.snapshot = snapshot;
this.#release = release;
}
activate(listener: (frame: ReplicatedStateSourceFrame<T>) => void): void {
if (this.#activated) throw new Error("State attachment is already active");
if (this.#disposed) throw new Error("State attachment is disposed");
this.#activated = true;
this.#listener = listener;
this.#drain();
}
publish(frame: ReplicatedStateSourceFrame<T>): void {
if (this.#disposed) return;
this.#frames.push(frame);
if (!this.#activated || this.#delivering || this.#scheduled) return;
this.#scheduled = true;
queueMicrotask(() => {
this.#scheduled = false;
if (this.#disposed) return;
try {
this.#drain();
} catch {
// Chord's frame listener contains source-contract failures. Isolate an
// unexpected direct listener failure to this attachment as well.
this.dispose();
}
});
}
dispose(): void {
if (this.#disposed) return;
this.#disposed = true;
this.#frames.length = 0;
this.#listener = undefined;
const release = this.#release;
this.#release = undefined;
release?.();
}
#drain(): void {
const listener = this.#listener;
if (listener === undefined || this.#delivering || this.#disposed) return;
this.#delivering = true;
try {
while (!this.#disposed) {
const frame = this.#frames.shift();
if (frame === undefined) break;
listener(frame);
}
} finally {
this.#delivering = false;
}
}
}
/** Canonical terminal frame of a retired document incarnation. */
const RETIREMENT_OPERATIONS: readonly Op[] = [["r", null]];
type WatchFrame<T> = {
readonly value: T;
readonly ops: readonly Op[];
readonly context: Context;
/** The retirement frame: the watch ends `retired` once it is delivered. */
readonly retires?: true;
};
type WatchListener<T> = (value: T, ops: readonly Op[], context: Context) => Promise<void>;
/**
* Serialized exact-frame watch bound to one document incarnation or conversation view. A `null` value retires it.
* Serialized exact-frame watch bound to one document incarnation or conversation view. Its source ends it explicitly:
* `retire()` for a retired document, `end()` for a closed or failed Session. Termination drops every reference the
* watch holds into its source, so a stopped watch that is still referenced keeps nothing but its last value alive.
* @internal
*/
export class CommittedWatch<T = ObservedDocumentValue> implements WatchHandle<T> {
readonly #detach: () => void;
readonly #report: (error: unknown) => void;
readonly #replace: (() => T) | undefined;
#detach: (() => void) | undefined;
#report: ((error: unknown) => void) | undefined;
#replace: (() => T) | undefined;
readonly #closed = Promise.withResolvers<WatchEnd>();
readonly #pending: WatchFrame<T>[] = [];
#value: T;
#listener: ((value: T, ops: readonly Op[], context: Context) => Promise<void>) | undefined;
#listener: WatchListener<T> | undefined;
#started = false;
#scheduled = false;
#running = false;
@ -190,7 +64,7 @@ export class CommittedWatch<T = ObservedDocumentValue> implements WatchHandle<T>
return this.#closed.promise;
}
start(listener: (value: T, ops: readonly Op[], context: Context) => Promise<void>): void {
start(listener: WatchListener<T>): void {
if (this.#started) throw new Error("Watch is already started");
if (this.#end !== undefined) throw new Error("Watch is stopped");
this.#started = true;
@ -216,16 +90,13 @@ export class CommittedWatch<T = ObservedDocumentValue> implements WatchHandle<T>
this.#terminate({ reason: "cancelled" });
}
/** End with `session_closed`, or with `session_failed` and the error when the Session failed. */
closeSession(failure?: { readonly error: unknown }): void {
this.#terminate(
failure === undefined ? { reason: "session_closed" } : { reason: "session_failed", error: failure.error },
);
/** The source closed: end with `ended`, or with `failed` and the error when the Session failed. */
end(failure?: { readonly error: unknown }): void {
this.#terminate(failure === undefined ? { reason: "ended" } : { reason: "failed", error: failure.error });
}
advance(value: T, ops: readonly Op[], context: Context): void {
if (this.#end !== undefined || this.#retired) return;
if (value === null) this.#retired = true;
if (this.#pending.length >= MAX_PENDING_WATCH_FRAMES) {
this.#pending.length = 0;
const replacement = this.#replace?.() ?? value;
@ -236,6 +107,18 @@ export class CommittedWatch<T = ObservedDocumentValue> implements WatchHandle<T>
if (this.#started) this.#schedule();
}
/**
* The document incarnation retired: queue the last frame, `[["r", null]]` with value `null`, after which the watch
* ends `retired`. An overflowing backlog is dropped, since the frame replaces the root anyway.
*/
retire(this: CommittedWatch<ObservedDocumentValue>, context: Context): void {
if (this.#end !== undefined || this.#retired) return;
this.#retired = true;
if (this.#pending.length >= MAX_PENDING_WATCH_FRAMES) this.#pending.length = 0;
this.#pending.push({ value: null, ops: RETIREMENT_OPERATIONS, context, retires: true });
if (this.#started) this.#schedule();
}
#schedule(): void {
if (this.#scheduled || this.#running || this.#end !== undefined) return;
this.#scheduled = true;
@ -251,19 +134,20 @@ export class CommittedWatch<T = ObservedDocumentValue> implements WatchHandle<T>
try {
while (this.#end === undefined) {
const frame = this.#pending.shift();
if (frame === undefined) break;
const listener = this.#listener;
if (frame === undefined || listener === undefined) break;
this.#value = frame.value;
const deliveryContext = withoutAbortSignal(frame.context);
try {
await this.#listener!(frame.value, frame.ops, deliveryContext);
await listener(frame.value, frame.ops, deliveryContext);
} catch (error) {
if (this.#end === undefined) {
this.#report(error);
this.#report?.(error);
this.#terminate({ reason: "listener_error", error: toError(error) });
}
break;
}
if (frame.value === null) {
if (frame.retires === true) {
this.#terminate({ reason: "retired" });
break;
}
@ -274,15 +158,25 @@ export class CommittedWatch<T = ObservedDocumentValue> implements WatchHandle<T>
}
}
/** End the watch, once: detach, drop pending frames, and settle `closed`. A callback already running keeps running. */
/**
* End the watch, once: detach, drop pending frames and every reference into the source, and settle `closed`. A
* callback already running keeps running.
*/
#terminate(end: WatchEnd): void {
if (this.#end !== undefined) return;
this.#end = end;
this.#detach();
const detach = this.#detach;
this.#detach = undefined;
this.#report = undefined;
this.#replace = undefined;
this.#listener = undefined;
this.#pending.length = 0;
if (this.#cancellationListener !== undefined) {
this.#cancellationSignal?.removeEventListener("abort", this.#cancellationListener);
}
this.#cancellationSignal = undefined;
this.#cancellationListener = undefined;
detach?.();
this.#closed.resolve(end);
}
}

148
packages/durable/src/session/session.ts
View file

@ -1,4 +1,4 @@
import { type Context, type JsonValue, replicatedState } from "@earendil-works/chord";
import type { Context, JsonValue } from "@earendil-works/chord";
import { awaitWithContext, BACKGROUND_CONTEXT, withoutAbortSignal } from "@earendil-works/chord/context";
import { type Op, track } from "@earendil-works/chord/delta";
import {
@ -27,7 +27,6 @@ import type {
DocumentPoint,
DocumentQuery,
DocumentRecord,
DocumentState,
DocumentWatch,
EntryId,
EntryQuery,
@ -57,12 +56,7 @@ import type {
TaskRecord,
Tx,
} from "../types.ts";
import {
CommittedStateSource,
CommittedWatch,
type ObservedDocumentValue,
RETIREMENT_OPERATIONS,
} from "./observation.ts";
import { CommittedWatch, type ObservedDocumentValue } from "./observation.ts";
import { type LoadedDocument, Transaction, type TransactionHost, type TransactionScope } from "./transaction.ts";
/** The default source of a report: a listener, which never fails what ran it. */
@ -105,7 +99,7 @@ export function documentRetention(ms: number | undefined): number {
/**
* Session kernel: one mutation line, the loaded document tracker cache, and committed publication.
*
* The cache is bounded by use: a document with no watch or document state attached leaves it once it was not read or
* The cache is bounded by use: a document with no watch attached leaves it once it was not read or
* written for the retention period, and the next access loads it from Storage. Eviction runs as a job on the mutation
* line, so no transaction ever holds a tracker that leaves.
*
@ -322,64 +316,6 @@ export class SessionImpl implements Session {
return loaded.tracker.value;
}
documentState<T extends JsonObject>(
token: SessionDocToken<T>,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject>(
token: ConversationDocToken<T>,
conversationId: ConversationId,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject>(
token: TaskDocToken<T>,
taskId: TaskId,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(
token: SessionDocFamilyToken<T, I>,
key: string,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(
token: ConversationDocFamilyToken<T, I>,
conversationId: ConversationId,
key: string,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(
token: TaskDocFamilyToken<T, I>,
taskId: TaskId,
key: string,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState(token: AnyDocToken, ...args: readonly unknown[]): Promise<DocumentState<JsonObject> | undefined> {
try {
this.assertUsable();
const definition = token.definition;
const resolved = resolveAddress(definition, args);
const context = args[resolved.nextArgument] as Context;
return this.#enqueue(async () => {
this.#assertHealthy();
const loaded = await this.#loadDocument(definition, resolved.id, resolved.address, context);
if (loaded === undefined) return undefined;
const { observer: source, detach } = this.#attachDocument(
definition,
loaded,
(value, release) => new CommittedStateSource<ObservedDocumentValue>(value, release),
);
try {
return replicatedState(source, { onError: (error) => this.report(error) }) as DocumentState<JsonObject>;
} catch (error) {
detach();
throw error;
}
});
} catch (error) {
return Promise.reject(error);
}
}
watchDoc<T extends JsonObject>(token: SessionDocToken<T>, context: Context): Promise<DocumentWatch<T> | undefined>;
watchDoc<T extends JsonObject>(
token: ConversationDocToken<T>,
@ -426,12 +362,7 @@ export class SessionImpl implements Session {
const loaded = await this.#loadDocument(definition, resolved.id, resolved.address, context);
if (cancelled) throw cancellationError(signal!);
if (loaded === undefined) return undefined;
return this.#attachDocument(
definition,
loaded,
(value, release) =>
new CommittedWatch<ObservedDocumentValue>(value, release, (error) => this.report(error)),
).observer;
return this.#attachDocument(definition, loaded);
});
if (watch === undefined) return undefined;
if (cancelled) {
@ -560,7 +491,7 @@ export class SessionImpl implements Session {
return this.#enqueue(async () => {
if (this.#failure !== undefined) return;
const batch = writes();
if (batch.length > 0) await this.#storage.commit(batch, BACKGROUND_CONTEXT);
if (batch.length > 0) await this.#storage.commit(batch, () => {}, BACKGROUND_CONTEXT);
});
}
@ -744,23 +675,36 @@ export class SessionImpl implements Session {
this.#sweepDocuments();
return result;
}
// Storage adopts the batch synchronously once it is durable, before its next commit; a throw here would reach
// Storage, so the callback keeps its own error.
let adoption: { readonly documents: DocumentCommitChange[] } | { readonly error: unknown } | undefined;
const adopt = (committed: Seq): void => {
try {
adoption = { documents: tx.adopt(committed) };
} catch (error) {
adoption = { error };
}
};
let seq: Seq;
try {
// Once admitted, caller cancellation does not interrupt Storage settlement.
seq = await this.#storage.commit(writes, withoutAbortSignal(context));
seq = await this.#storage.commit(writes, adopt, withoutAbortSignal(context));
} catch (error) {
// The guard has failed the Session.
tx.discard();
// The guard has failed the Session. A rejection after adoption leaves nothing staged to discard.
if (adoption === undefined) tx.discard();
throw error;
}
let documents: DocumentCommitChange[];
try {
documents = tx.adopt(seq);
} catch (error) {
// Storage already committed; a failed adoption leaves memory behind durable state.
if (adoption === undefined) {
const error = new Error("Storage committed without adopting the batch");
this.fail(error);
throw error;
}
if ("error" in adoption) {
// Storage already committed; a failed adoption leaves memory behind durable state.
this.fail(adoption.error);
throw adoption.error;
}
const documents = adoption.documents;
for (const document of documents) this.#touchDocument(addressId(document.record));
this.#publish(seq, writes, documents, context);
this.#sweepDocuments();
@ -818,14 +762,13 @@ export class SessionImpl implements Session {
}
/**
* Attach an observer to one committed incarnation: check the definition, then forward this incarnation's committed
* changes and close. `detach` removes both subscriptions.
* Attach a watch to one committed incarnation: check the definition, then forward this incarnation's committed
* changes, its retirement, and close. The watch's detach removes both subscriptions.
*/
#attachDocument<O extends CommittedStateSource | CommittedWatch>(
#attachDocument(
definition: AnyDocToken["definition"],
loaded: LoadedDocument,
create: (value: JsonObject, detach: () => void) => O,
): { observer: O; detach: () => void } {
): CommittedWatch<ObservedDocumentValue> {
checkRecordScope(definition, loaded.record);
checkRecordVersion(definition, loaded.record, loaded.storedVersion);
let unsubscribeCommit = (): void => {};
@ -837,29 +780,27 @@ export class SessionImpl implements Session {
unsubscribeClose();
unpin();
};
let observer: O;
try {
observer = create(loaded.tracker.value, detach);
} catch (error) {
unpin();
throw error;
}
const observer = new CommittedWatch<ObservedDocumentValue>(loaded.tracker.value, detach, (error) =>
this.report(error),
);
const observed = { version: loaded.valueVersion };
unsubscribeCommit = this.observeCommits((publication, context) => {
for (const change of publication.changes) {
if (change.type !== "document" || change.record.id !== loaded.record.id) continue;
// A document state's frames carry no caller cancellation; a watch observes its own cancellation.
const frameContext = observer instanceof CommittedStateSource ? withoutAbortSignal(context) : context;
if (change.value === null) {
// A retired incarnation never changes again: its watch delivers the retirement but pins nothing.
unpin();
observer.retire(context);
continue;
}
const ops = observedOperations(observed, change);
// A migration-only base changes nothing for an observer of the new version.
if (ops.length === 0) continue;
// A retired incarnation never changes again: its observers keep their last value but pin nothing.
if (change.value === null) unpin();
observer.advance(change.value, ops, frameContext);
observer.advance(change.value, ops, context);
}
});
unsubscribeClose = this.subscribeClose(() => observer.closeSession(this.#failure));
return { observer, detach };
unsubscribeClose = this.subscribeClose(() => observer.end(this.#failure));
return observer;
}
async #loadDocument(
@ -935,7 +876,6 @@ function observedOperations(
observed: { version: number },
change: Extract<DocumentCommitChange, { readonly type: "document" }>,
): readonly Op[] {
if (change.value === null) return RETIREMENT_OPERATIONS;
if (change.version === observed.version) return change.ops;
observed.version = change.version!;
return [["r", change.value]];
@ -971,8 +911,8 @@ class GuardedStorage implements Storage {
}
/** Never exempt, whatever it throws: once admitted, whether a failed batch committed is unknown. */
commit(writes: readonly StorageWrite[], context: Context): Promise<Seq> {
return this.#call(undefined, () => this.#storage.commit(writes, context));
commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq> {
return this.#call(undefined, () => this.#storage.commit(writes, adopt, context));
}
mintId<I extends Id<string>>(): Promise<I> {

7
packages/durable/src/storage/indexeddb.ts
View file

@ -417,10 +417,10 @@ export class IndexedDbStorage implements Storage {
await requestResult(factory.deleteDatabase(name));
}
async commit(writes: readonly StorageWrite[], _context: Context): Promise<Seq> {
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, _context: Context): Promise<Seq> {
this.assertOpen();
const documentActions = this.prepareDocumentActions(writes);
return this.transaction(ALL_STORES, "readwrite", async (transaction) => {
const seq = await this.transaction(ALL_STORES, "readwrite", async (transaction) => {
const metaStore = transaction.objectStore(STORES.meta);
const metadata = await requestResult(metaStore.get(META_KEY) as IDBRequest<MetaRow>);
if (metadata === undefined) throw new Error("Durable IndexedDB metadata is missing");
@ -437,6 +437,9 @@ export class IndexedDbStorage implements Storage {
await requestResult(metaStore.put({ key: META_KEY, nextId, nextSeq: seq + 1 } satisfies MetaRow));
return seq;
});
// The transaction completed: the batch is durable.
adopt(seq);
return seq;
}
async mintId<I extends Id<string>>(): Promise<I> {

4
packages/durable/src/storage/jsonl/storage.ts
View file

@ -275,7 +275,7 @@ export class JsonlStorage implements Storage {
return storage;
}
async commit(writes: readonly StorageWrite[], context: Context): Promise<Seq> {
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq> {
this.assertUsable();
const prepared = this.memory.prepareCommit(writes);
const encoded = this.encodeCommit(prepared.seq, prepared.writes);
@ -302,6 +302,8 @@ export class JsonlStorage implements Storage {
if (!marker.ok) throw this.poison(errorFromFile(`append to ${MAIN_FILE}`, marker.error));
const seq = prepared.apply();
this.adoptSidecarState(prepared.writes);
// The marker is appended: the batch is durable. Reclaiming sidecars comes after adoption.
adopt(seq);
await this.reclaimSidecars(reclamations, context);
return seq;
}

6
packages/durable/src/storage/memory.ts
View file

@ -260,8 +260,10 @@ export class MemoryStorage implements Storage {
private nextSeq = 1;
private closed = false;
async commit(writes: readonly StorageWrite[], _context: Context): Promise<Seq> {
return this.prepareCommit(writes).apply();
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, _context: Context): Promise<Seq> {
const seq = this.prepareCommit(writes).apply();
adopt(seq);
return seq;
}
/** Validate and detach one commit without changing observable state. */

4
packages/durable/src/storage/sqlite/storage.ts
View file

@ -184,7 +184,7 @@ export class SqliteStorage implements Storage {
}
}
async commit(writes: readonly StorageWrite[], _context: Context): Promise<Seq> {
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, _context: Context): Promise<Seq> {
this.assertOpen();
const documentActions = this.prepareDocumentActions(writes);
const candidateNextId = this.candidateNextId(writes);
@ -206,6 +206,8 @@ export class SqliteStorage implements Storage {
return committedSeq;
});
this.nextId = Math.max(this.nextId, candidateNextId);
// `COMMIT` returned: the batch is durable.
adopt(seq);
return seq;
}

27
packages/durable/src/testing/storage-benchmark.ts
View file

@ -14,6 +14,9 @@ import type {
} from "../types.ts";
import { ROOT_CONVERSATION_ID } from "../types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
export type StorageBenchmarkScale = {
readonly name: string;
readonly entryCount: number;
@ -89,7 +92,7 @@ export async function seedStorageBenchmark(
storage: Storage,
scale: StorageBenchmarkScale = TIMING_SCALE,
): Promise<StorageBenchmarkDataset> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, BACKGROUND_CONTEXT);
let firstEntryId: EntryId | undefined;
for (let start = 0; start < scale.entryCount; start += BATCH_SIZE) {
@ -108,7 +111,7 @@ export async function seedStorageBenchmark(
},
});
}
await storage.commit(writes, BACKGROUND_CONTEXT);
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
}
for (let start = 0; start < scale.taskCount; start += BATCH_SIZE) {
@ -116,7 +119,7 @@ export async function seedStorageBenchmark(
for (let index = start; index < Math.min(start + BATCH_SIZE, scale.taskCount); index++) {
writes.push({ type: "task", value: task(await storage.mintId<TaskId<JsonValue>>(), index) });
}
await storage.commit(writes, BACKGROUND_CONTEXT);
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
}
let exactDocumentId: DocumentId | undefined;
@ -131,7 +134,7 @@ export async function seedStorageBenchmark(
content: { kind: "base", version: 1, value: { index, text: "x".repeat(128) } },
});
}
await storage.commit(writes, BACKGROUND_CONTEXT);
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
}
const replayEntries = await Promise.all(
@ -153,6 +156,7 @@ export async function seedStorageBenchmark(
},
content: { kind: "base" as const, version: 1, value: { count: 0, text: "x".repeat(64) } },
})),
adopt,
BACKGROUND_CONTEXT,
);
for (let count = 1; count <= REPLAY_TAILS.at(-1)!; count++) {
@ -164,6 +168,7 @@ export async function seedStorageBenchmark(
id,
content: { kind: "delta" as const, version: 1, ops: [["s", ["count"], count] as const] },
})),
adopt,
BACKGROUND_CONTEXT,
);
}
@ -184,6 +189,7 @@ export async function seedStorageBenchmark(
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
BACKGROUND_CONTEXT,
);
let ancientAt: Seq | undefined;
@ -196,6 +202,7 @@ export async function seedStorageBenchmark(
content: { kind: "delta", version: 1, ops: [["s", ["count"], count]] },
},
],
adopt,
BACKGROUND_CONTEXT,
);
}
@ -207,6 +214,7 @@ export async function seedStorageBenchmark(
content: { kind: "base", version: 1, value: { count: HISTORY_SEGMENT_LENGTH } },
},
],
adopt,
BACKGROUND_CONTEXT,
);
if (ancientAt === undefined) throw new Error("Benchmark history seed produced no commits");
@ -220,6 +228,7 @@ export async function seedStorageBenchmark(
content: { kind: "delta", version: 1, ops: [["s", ["count"], count]] },
},
],
adopt,
BACKGROUND_CONTEXT,
);
}
@ -237,6 +246,7 @@ export async function seedStorageBenchmark(
value: { id: conversationId, parent: { conversationId: parentConversationId, at: parentAt } },
},
],
adopt,
BACKGROUND_CONTEXT,
);
const ids = await Promise.all(Array.from({ length: ENTRIES_PER_FORK }, () => storage.mintId<EntryId>()));
@ -250,6 +260,7 @@ export async function seedStorageBenchmark(
data: { depth, index },
},
})),
adopt,
BACKGROUND_CONTEXT,
);
parentConversationId = conversationId;
@ -389,7 +400,7 @@ export type StorageWriteBenchmark = {
/** Seed the common state expected by every write benchmark sample. */
export async function seedStorageWriteBenchmark(storage: Storage): Promise<void> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, BACKGROUND_CONTEXT);
await storage.commit(
await Promise.all(
Array.from({ length: 100 }, async (_, index) => ({
@ -402,6 +413,7 @@ export async function seedStorageWriteBenchmark(storage: Storage): Promise<void>
},
})),
),
adopt,
BACKGROUND_CONTEXT,
);
}
@ -424,6 +436,7 @@ export const STORAGE_WRITE_BENCHMARKS: readonly StorageWriteBenchmark[] = [
},
},
],
adopt,
BACKGROUND_CONTEXT,
);
return 1;
@ -447,7 +460,7 @@ export const STORAGE_WRITE_BENCHMARKS: readonly StorageWriteBenchmark[] = [
}),
),
);
await storage.commit(writes, BACKGROUND_CONTEXT);
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
return writes.length;
},
},
@ -482,7 +495,7 @@ export const STORAGE_WRITE_BENCHMARKS: readonly StorageWriteBenchmark[] = [
content: { kind: "base", version: 1, value: { entryId, taskId } },
},
];
await storage.commit(writes, BACKGROUND_CONTEXT);
await storage.commit(writes, adopt, BACKGROUND_CONTEXT);
return writes.length;
},
},

167
packages/durable/src/testing/storage-conformance.ts
View file

@ -13,6 +13,7 @@ import {
type Page,
ROOT_CONVERSATION_ID,
type ScanOrder,
type Seq,
type Storage,
type StorageWrite,
type SubmissionId,
@ -23,11 +24,14 @@ import {
} from "../types.ts";
import type { StorageConformanceAssertions, StorageConformanceCase, StorageConformanceOptions } from "./types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
const context = BACKGROUND_CONTEXT;
type StoredTask = TaskRecord<JsonValue, JsonValue, JsonValue>;
async function createRoot(storage: Storage): Promise<ConversationId> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
return ROOT_CONVERSATION_ID;
}
@ -87,8 +91,43 @@ function assertionFacade(assertions: StorageConformanceAssertions) {
});
}
/**
* `storage` with every commit checked against the adoption contract: `adopt` runs exactly once, with the committed
* sequence, before the commit resolves (so before the next commit starts), and never for a commit that rejects. The
* cases fail commits only before they are durable.
*/
function checkAdoption(storage: Storage, assertions: StorageConformanceAssertions): Storage {
const commit: Storage["commit"] = async (writes, adopt, commitContext) => {
const adopted: Seq[] = [];
let settled = false;
const recording = (seq: Seq): void => {
assertions.ok(!settled, "Storage.commit() adopted after it settled");
adopted.push(seq);
adopt(seq);
};
let seq: Seq;
try {
seq = await storage.commit(writes, recording, commitContext);
} catch (error) {
settled = true;
assertions.deepEqual(adopted, []);
throw error;
}
settled = true;
assertions.deepEqual(adopted, [seq]);
return seq;
};
return new Proxy(storage, {
get(target, property) {
if (property === "commit") return commit;
const value: unknown = Reflect.get(target, property, target);
return typeof value === "function" ? value.bind(target) : value;
},
});
}
function createCase(options: StorageConformanceOptions, name: string, test: ConformanceTest): StorageConformanceCase {
return { name, run: () => options.withStorage(test) };
return { name, run: () => options.withStorage((storage) => test(checkAdoption(storage, options.assertions))) };
}
/** Creates runner-independent cases. `withStorage` must call and await its callback exactly once per case. */
@ -100,10 +139,51 @@ export function createStorageConformance(options: StorageConformanceOptions): re
await expect(createRoot(storage)).resolves.toBe(ROOT_CONVERSATION_ID);
expect(await storage.conversation(ROOT_CONVERSATION_ID, context)).toEqual({ id: ROOT_CONVERSATION_ID });
await expect(
storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context),
storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context),
).rejects.toThrow(`ID ${ROOT_CONVERSATION_ID} already belongs to conversation`);
}),
createCase(options, "adopts a durable commit once before resolving, and never a failed one", async (storage) => {
const rootId = await createRoot(storage);
const events: string[] = [];
const entryId = await storage.mintId<EntryId>();
let read: ReturnType<Storage["entry"]> | undefined;
const seq = await storage
.commit(
[{ type: "entry", value: entry(entryId, rootId) }],
(adopted) => {
events.push(`adopt ${adopted}`);
// The batch is durable when it is adopted: a read started now observes it.
read = storage.entry(entryId, context);
},
context,
)
.then((committed) => {
events.push(`resolve ${committed}`);
return committed;
});
expect(events).toEqual([`adopt ${seq}`, `resolve ${seq}`]);
expect((await read)?.commitSeq).toBe(seq);
events.length = 0;
await expect(
storage.commit(
[{ type: "conversation", value: { id: rootId } }],
(adopted) => events.push(`adopt ${adopted}`),
context,
),
).rejects.toThrow(`ID ${rootId} already belongs to conversation`);
expect(events).toEqual([]);
const next = await storage.commit(
[{ type: "entry", value: entry(await storage.mintId<EntryId>(), rootId) }],
(adopted) => events.push(`adopt ${adopted}`),
context,
);
expect(events).toEqual([`adopt ${next}`]);
expect(next).toBeGreaterThan(seq);
}),
createCase(
options,
"commits mixed table writes atomically and rolls all of them back on failure",
@ -127,6 +207,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "task", value: task },
{ type: "submission", value: input },
],
adopt,
context,
);
@ -151,6 +232,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "entry", value: entry(transientEntryId, rootId, "assistant") },
{ type: "conversation", value: { id: rootId } },
],
adopt,
context,
),
).rejects.toThrow(`ID ${rootId} already belongs to conversation`);
@ -160,6 +242,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
expect(await storage.entry(transientEntryId, context)).toBeUndefined();
const afterRollbackSeq = await storage.commit(
[{ type: "entry", value: entry(await storage.mintId<EntryId>(), rootId, "after-rollback") }],
adopt,
context,
);
expect(afterRollbackSeq).toBeGreaterThan(initialSeq);
@ -193,6 +276,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "task", value: storedTask },
{ type: "submission", value: storedInput },
],
adopt,
context,
);
@ -229,7 +313,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
const data = JSON.parse(
'{"__proto__":{"polluted":false},"constructor":{"label":"stored"},"toString":"value"}',
) as Record<string, JsonValue>;
await storage.commit([{ type: "entry", value: entry(entryId, rootId, "note", { data }) }], context);
await storage.commit([{ type: "entry", value: entry(entryId, rootId, "note", { data }) }], adopt, context);
(Reflect.get(data, "__proto__") as Record<string, JsonValue>).polluted = true;
(Reflect.get(data, "constructor") as Record<string, JsonValue>).label = "mutated";
@ -261,6 +345,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
}),
},
],
adopt,
context,
);
@ -281,13 +366,14 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "entry", value: entry(middleId, rootId) },
{ type: "entry", value: entry(newestId, rootId) },
],
adopt,
context,
);
const first = await storage.scanEntries({ conversationId: rootId }, 2, undefined, context);
expect(first.items.map(({ id }) => id)).toEqual([newestId, middleId]);
const appendedId = await storage.mintId<EntryId>();
await storage.commit([{ type: "entry", value: entry(appendedId, rootId) }], context);
await storage.commit([{ type: "entry", value: entry(appendedId, rootId) }], adopt, context);
const second = await storage.scanEntries({ conversationId: rootId }, 2, first.next, context);
expect(second.items.map(({ id }) => id)).toEqual([oldestId]);
expect(second.next).toBeUndefined();
@ -302,6 +388,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "conversation", value: { id: thirdId } },
{ type: "conversation", value: { id: secondId } },
],
adopt,
context,
);
@ -338,6 +425,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
value: { id: thirdId, owner: { conversationId: otherOwnerId, taskId: firstTaskId } },
},
],
adopt,
context,
);
@ -378,6 +466,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
{ type: "entry", value: entry(rootExcludedSameCommit, rootId) },
],
adopt,
context,
);
const childId = await storage.mintId<ConversationId>();
@ -388,6 +477,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
value: { id: childId, parent: { conversationId: rootId, at: rootForkPoint } },
},
],
adopt,
context,
);
const childForkPoint = await storage.mintId<EntryId>();
@ -397,10 +487,11 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "entry", value: entry(childForkPoint, childId, "note") },
{ type: "entry", value: entry(childExcluded, childId) },
],
adopt,
context,
);
const rootExcludedLater = await storage.mintId<EntryId>();
await storage.commit([{ type: "entry", value: entry(rootExcludedLater, rootId) }], context);
await storage.commit([{ type: "entry", value: entry(rootExcludedLater, rootId) }], adopt, context);
const grandchildId = await storage.mintId<ConversationId>();
await storage.commit(
[
@ -409,6 +500,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
value: { id: grandchildId, parent: { conversationId: childId, at: childForkPoint } },
},
],
adopt,
context,
);
const grandchildHead = await storage.mintId<EntryId>();
@ -421,10 +513,11 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
{ type: "entry", value: entry(grandchildTail, grandchildId) },
],
adopt,
context,
);
const childExcludedLater = await storage.mintId<EntryId>();
await storage.commit([{ type: "entry", value: entry(childExcludedLater, childId) }], context);
await storage.commit([{ type: "entry", value: entry(childExcludedLater, childId) }], adopt, context);
const first = await storage.scanEntries({ conversationId: grandchildId }, 2, undefined, context);
expect(first.items.map(({ id }) => id)).toEqual([grandchildTail, grandchildHead]);
@ -544,6 +637,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
value: { id: submissionId, conversationId: rootId, type: "input", status: "queued" },
},
],
adopt,
context,
);
conversationIds.push(conversationId);
@ -592,6 +686,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "task", value: second },
{ type: "task", value: third },
],
adopt,
context,
);
@ -600,7 +695,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
state: { status: "running", checkpoint: { phase: "effect", attempt: 1 } },
abortRequested: true,
};
await storage.commit([{ type: "task", value: running }], context);
await storage.commit([{ type: "task", value: running }], adopt, context);
expect(await storage.task(firstId, context)).toEqual(running);
const terminal: StoredTask = {
id: firstId,
@ -612,7 +707,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
background: false,
abortRequested: true,
};
await storage.commit([{ type: "task", value: terminal }], context);
await storage.commit([{ type: "task", value: terminal }], adopt, context);
expect(await storage.task(firstId, context)).toEqual(terminal);
const pendingPage = await storage.scanTasks({ status: "pending" }, 1, undefined, context);
@ -648,7 +743,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
state: { status: "completing", outcome: { status: "failed", error: { message: "held" } } },
};
const writes = [owner, waiting, completing].map((value) => ({ type: "task", value }) as const);
await storage.commit(writes, context);
await storage.commit(writes, adopt, context);
expect(await storage.task(waitingId, context)).toEqual(waiting);
expect(await storage.task(completingId, context)).toEqual(completing);
const scan = async (status: StoredTask["state"]["status"]) =>
@ -660,7 +755,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
...completing,
state: { status: "terminal", outcome: completing.state.outcome! },
};
await storage.commit([{ type: "task", value: terminal }], context);
await storage.commit([{ type: "task", value: terminal }], adopt, context);
expect(await scan("completing")).toEqual([]);
expect(await scan("terminal")).toEqual([terminal]);
}),
@ -673,13 +768,13 @@ export function createStorageConformance(options: StorageConformanceOptions): re
state: { status: "sleeping", checkpoint: { phase: "next" }, until: 61_000 },
memos: { kept: true },
};
await storage.commit([{ type: "task", value: sleeping }], context);
await storage.commit([{ type: "task", value: sleeping }], adopt, context);
expect(await storage.task(sleepingId, context)).toEqual(sleeping);
const scan = async (status: StoredTask["state"]["status"]) =>
(await storage.scanTasks({ status }, 10, undefined, context)).items;
expect(await scan("sleeping")).toEqual([sleeping]);
const running: StoredTask = { ...sleeping, state: { status: "running", checkpoint: { phase: "next" } } };
await storage.commit([{ type: "task", value: running }], context);
await storage.commit([{ type: "task", value: running }], adopt, context);
expect(await scan("sleeping")).toEqual([]);
expect(await scan("running")).toEqual([running]);
}),
@ -690,7 +785,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
async (storage) => {
const rootId = await createRoot(storage);
const secondConversationId = await storage.mintId<ConversationId>();
await storage.commit([{ type: "conversation", value: { id: secondConversationId } }], context);
await storage.commit([{ type: "conversation", value: { id: secondConversationId } }], adopt, context);
const firstId = await storage.mintId<SubmissionId>();
const secondId = await storage.mintId<SubmissionId>();
const otherConversationId = await storage.mintId<SubmissionId>();
@ -721,6 +816,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "submission", value: second },
{ type: "submission", value: otherConversation },
],
adopt,
context,
);
expect(await storage.submissionByRequest(rootId, "same", context)).toEqual(first);
@ -731,7 +827,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
status: "placed",
entry: await storage.mintId<EntryId>(),
};
await storage.commit([{ type: "submission", value: placedSecond }], context);
await storage.commit([{ type: "submission", value: placedSecond }], adopt, context);
expect(await storage.submission(secondId, context)).toEqual(placedSecond);
expect(await storage.submissionByRequest(rootId, "other", context)).toEqual(placedSecond);
@ -783,6 +879,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "submission", value: queuedDone },
{ type: "submission", value: queuedFailed },
],
adopt,
context,
);
@ -802,6 +899,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "submission", value: done },
{ type: "submission", value: unanswered },
],
adopt,
context,
);
expect(await storage.submission(doneId, context)).toEqual(done);
@ -829,6 +927,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "base", version: 1, value: initial },
},
],
adopt,
context,
);
const appended = ["b"];
@ -838,6 +937,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
];
const changedAt = await storage.commit(
[{ type: "document.change", id: firstId, content: { kind: "delta", version: 1, ops } }],
adopt,
context,
);
@ -865,6 +965,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "base", version: 2, value: { items: ["checkpoint"], nested: { count: 3 } } },
},
],
adopt,
context,
);
const replacedAt = await storage.commit(
@ -879,6 +980,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
},
],
adopt,
context,
);
expect(await storage.document(firstId, changedAt, context)).toMatchObject({
@ -911,6 +1013,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "delta", version: 2, ops: [["s", ["retiring"], true]] },
},
],
adopt,
context,
);
const address = { kind: firstRecord.kind, scope: firstRecord.scope };
@ -949,6 +1052,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
};
const createdAt = await storage.commit(
[{ type: "document.create", record, content: { kind: "base", version: 1, value: initial } }],
adopt,
context,
);
const beforeReplacement = structuredClone(initial);
@ -972,6 +1076,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
},
],
adopt,
context,
);
}
@ -989,6 +1094,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "delta", version: 1, ops: [["r", replacement]] },
},
],
adopt,
context,
);
replacement.rows[0]!.value = -999;
@ -1013,6 +1119,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
},
],
adopt,
context,
);
}
@ -1038,6 +1145,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "conversation", value: { id: childId } },
{ type: "conversation", value: { id: secondChildId } },
],
adopt,
context,
);
const sourceId = await storage.mintId<DocumentId>();
@ -1056,6 +1164,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "base", version: 2, value: { count: 1, rows: [{ value: "base" }] } },
},
],
adopt,
context,
);
await storage.commit(
@ -1073,6 +1182,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
},
],
adopt,
context,
);
const historicalCopyId = await storage.mintId<DocumentId>();
@ -1104,6 +1214,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
{ type: "document.retire", id: retiredCopyId },
],
adopt,
context,
);
expect(await storage.document(historicalCopyId, "current", context)).toMatchObject({
@ -1125,6 +1236,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
{ type: "document.retire", id: sourceId },
],
adopt,
context,
);
expect((await storage.document(currentCopyId, "current", context))?.value).toEqual({
@ -1149,6 +1261,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "base", version: 4, value: { retained: "copy" } },
},
],
adopt,
context,
);
await storage.commit(
@ -1163,6 +1276,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
source: { id: latestSourceId, at: "current" },
},
],
adopt,
context,
);
await storage.commit(
@ -1174,6 +1288,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
{ type: "document.retire", id: latestSourceId },
],
adopt,
context,
);
expect(await storage.document(latestCopyId, "current", context)).toMatchObject({
@ -1193,6 +1308,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
{ type: "document.retire", id: currentCopyId },
],
adopt,
context,
);
} catch (error) {
@ -1217,6 +1333,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
source: { id: currentCopyId, at: "current" },
},
],
adopt,
context,
);
} catch (error) {
@ -1240,14 +1357,17 @@ export function createStorageConformance(options: StorageConformanceOptions): re
} satisfies DocumentCreate;
await storage.commit(
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { count: 1 } } }],
adopt,
context,
);
await storage.commit(
[{ type: "document.change", id, content: { kind: "delta", version: 1, ops: [["s", ["count"], 2]] } }],
adopt,
context,
);
const migratedAt = await storage.commit(
[{ type: "document.change", id, content: { kind: "base", version: 2, value: { count: 3 } } }],
adopt,
context,
);
expect(await storage.document(id, "current", context)).toMatchObject({ version: 2, value: { count: 3 } });
@ -1258,11 +1378,12 @@ export function createStorageConformance(options: StorageConformanceOptions): re
await expect(
storage.commit(
[{ type: "document.change", id, content: { kind: "delta", version: 1, ops: [["s", ["count"], 4]] } }],
adopt,
context,
),
).rejects.toThrow("version transition requires a base");
expect((await storage.document(id, "current", context))?.value).toEqual({ count: 3 });
await storage.commit([{ type: "document.retire", id }], context);
await storage.commit([{ type: "document.retire", id }], adopt, context);
expect(await storage.document(id, "current", context)).toBeUndefined();
},
),
@ -1340,6 +1461,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "base", version: 1, value: { owner: "other" } },
},
],
adopt,
context,
);
@ -1420,6 +1542,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
} satisfies DocumentCreate;
await storage.commit(
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { value: 1 } } }],
adopt,
context,
);
await expect(
@ -1432,6 +1555,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
{ type: "document.change", id: firstId, content: { kind: "delta", version: 1, ops: [] } },
],
adopt,
context,
),
).rejects.toThrow("already has a current incarnation");
@ -1455,6 +1579,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
},
{ type: "document.retire", id: emptyId },
],
adopt,
context,
);
expect(await storage.document(emptyId, "current", context)).toBeUndefined();
@ -1500,6 +1625,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
{ type: "submission", value: submission },
{ type: "document.create", record, content: { kind: "base", version: 1, value: { count: 1 } } },
],
adopt,
context,
);
@ -1523,6 +1649,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
),
).rejects.toThrow("already has a current incarnation");
@ -1543,6 +1670,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "delta", version: 1, ops: [["s", ["count"], 3]] },
},
],
adopt,
context,
);
expect(afterRollbackSeq).toBeGreaterThan(baselineSeq);
@ -1606,6 +1734,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
content: { kind: "base", version: 1, value: { identity: "second key" } },
},
],
adopt,
context,
);
@ -1654,17 +1783,19 @@ export function createStorageConformance(options: StorageConformanceOptions): re
createCase(options, "keeps one global record ID namespace and rejects exhausted ID minting", async (storage) => {
const rootId = await createRoot(storage);
const explicitEntryId = idFromNumber<EntryId>(100);
await storage.commit([{ type: "entry", value: entry(explicitEntryId, rootId) }], context);
await storage.commit([{ type: "entry", value: entry(explicitEntryId, rootId) }], adopt, context);
expect(await storage.mintId<EntryId>()).toBe(101);
await expect(
storage.commit(
[{ type: "task", value: pendingTask(idFromNumber<TaskId<JsonValue>>(explicitEntryId), rootId) }],
adopt,
context,
),
).rejects.toThrow(`ID ${explicitEntryId} already belongs to entry`);
await storage.commit(
[{ type: "entry", value: entry(idFromNumber<EntryId>(Number.MAX_SAFE_INTEGER), rootId, "last-id") }],
adopt,
context,
);
await expect(storage.mintId<EntryId>()).rejects.toThrow("ID space is exhausted");
@ -1675,7 +1806,7 @@ export function createStorageConformance(options: StorageConformanceOptions): re
await createRoot(storage);
await storage.close(context);
await expect(storage.conversation(ROOT_CONVERSATION_ID, context)).rejects.toThrow("closed");
await expect(storage.commit([] satisfies StorageWrite[], context)).rejects.toThrow("closed");
await expect(storage.commit([] satisfies StorageWrite[], adopt, context)).rejects.toThrow("closed");
await expect(storage.mintId<ConversationId>()).rejects.toThrow("closed");
}),
];

78
packages/durable/src/types.ts
View file

@ -1,4 +1,4 @@
import type { AttachedReplicatedState, Context, Draft, JsonValue } from "@earendil-works/chord";
import type { Context, Draft, JsonValue, WatchHandle } from "@earendil-works/chord";
import type { Op } from "@earendil-works/chord/delta";
import type { Message, Models } from "@earendil-works/pi-ai";
import type { TelemetryCarrier } from "@earendil-works/pi-telemetry";
@ -267,7 +267,7 @@ export interface TaskRuntime<I, S, R, H extends object> extends DocumentObserver
* Commit on the Session line after rereading the task. Rejects when the task is terminal, the invocation ended, the
* Harness is closing, or, in a run invocation, the task carries an abort mark. A returned state replaces the task's
* state in the same commit; returning nothing leaves it unchanged. `tx.createTask()` defaults to the task's
* conversation. Inside the callback read only through `tx`: `snapshot()`, `documentState()`, `watchDoc()`, `getTask()`, and the other Session and runtime reads may queue on the line the callback holds, and then never resolve.
* conversation. Inside the callback read only through `tx`: `snapshot()`, `watchDoc()`, `getTask()`, and the other Session and runtime reads may queue on the line the callback holds, and then never resolve.
*/
commit(
change: (
@ -882,7 +882,7 @@ export type DocumentCommitChange =
readonly ops: readonly Op[];
}
| {
/** Definition-free child initialization; consumers hydrate through state or watch acquisition. */
/** Definition-free child initialization; consumers hydrate through watch acquisition. */
readonly type: "document.copy";
readonly record: DocumentRecord;
readonly conversationId: ConversationId;
@ -1004,28 +1004,10 @@ export interface Tx {
): Promise<void>;
}
/** Disposable, read-only Chord state bound to one committed document incarnation. */
export type DocumentState<T extends JsonObject> = AttachedReplicatedState<Readonly<T> | null>;
/** Terminal result of one document watch. */
export type WatchEnd =
| { readonly reason: "stopped" | "cancelled" | "session_closed" | "retired" }
| { readonly reason: "listener_error"; readonly error: Error }
/** The Session failed; `error` is what failed it, usually a storage error (see `SessionFailed`). */
| { readonly reason: "session_failed"; readonly error: unknown };
/** Serialized exact-frame observation of an immutable value with bounded pending delivery. */
export interface WatchHandle<T> {
/** Acquisition revision before start; latest delivered immutable revision afterward. */
readonly value: T;
/** Install the sole asynchronous listener. Never invokes it inline. */
start(listener: (value: T, ops: readonly Op[], context: Context) => Promise<void>): void;
/** Idempotently stop future callbacks and return this watch's terminal result. */
stop(): Promise<WatchEnd>;
/** Settle when the watch terminates; an already-running callback remains caller-owned. */
readonly closed: Promise<WatchEnd>;
}
/**
* Watch of one committed document incarnation. A retirement delivers `[["r", null]]` with value `null`, then ends the
* watch as `retired`; the Session's close ends it as `ended`, its failure as `failed` with the error that failed it.
*/
export type DocumentWatch<T extends JsonObject> = WatchHandle<Readonly<T> | null>;
/** Committed document reads. */
@ -1069,7 +1051,7 @@ export type SessionEnd = { readonly reason: "closed" } | { readonly reason: "fai
/** Owner of one mutation line, its records, and its tracked documents. */
export interface Session extends DocumentObserver {
/**
* Run one atomic transaction on the Session mutation line. Inside the callback read only through `tx`: `snapshot()`, `documentState()`, `watchDoc()`, `getTask()`, and the other Session and runtime reads may queue on the line the callback holds, and then never resolve.
* Run one atomic transaction on the Session mutation line. Inside the callback read only through `tx`: `snapshot()`, `watchDoc()`, `getTask()`, and the other Session and runtime reads may queue on the line the callback holds, and then never resolve.
*/
commit<T>(change: (tx: Tx) => T | Promise<T>, context: Context): Promise<T>;
/** Seal admission, settle admitted commits, then close storage. */
@ -1116,38 +1098,6 @@ export interface Session extends DocumentObserver {
context: Context,
): Promise<Readonly<T> | undefined>;
documentState<T extends JsonObject>(
token: SessionDocToken<T>,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject>(
token: ConversationDocToken<T>,
conversationId: ConversationId,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject>(
token: TaskDocToken<T>,
taskId: TaskId,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(
token: SessionDocFamilyToken<T, I>,
key: string,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(
token: ConversationDocFamilyToken<T, I>,
conversationId: ConversationId,
key: string,
context: Context,
): Promise<DocumentState<T> | undefined>;
documentState<T extends JsonObject, I extends JsonValue>(
token: TaskDocFamilyToken<T, I>,
taskId: TaskId,
key: string,
context: Context,
): Promise<DocumentState<T> | undefined>;
snapshotAsOf<T extends JsonObject>(
token: RewindableConversationDocToken<T>,
conversationId: ConversationId,
@ -1178,13 +1128,17 @@ export interface Session extends DocumentObserver {
*/
export interface Storage {
/**
* Atomically persist one batch and return its sequence. Once resolved, later reads through this storage observe it.
* Atomically persist one batch and return its sequence. Once the batch is durable, call `adopt(seq)` exactly once,
* synchronously and before this storage's next commit starts, then resolve with `seq`; a commit that fails before it
* is durable never calls `adopt`. `adopt` is the caller's in-memory adoption of the batch, so memory always shows a
* complete prefix of the log; it must not throw. Once resolved, later reads through this storage observe the batch.
*/
commit(writes: readonly StorageWrite[], context: Context): Promise<Seq>;
commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq>;
/**
* Return a fresh branded candidate from the Session-global numeric ID namespace, greater than every ID minted or stored
* before, also across reopen: scans order by ID as creation order, and the Harness relies on it.
* Return a fresh branded candidate from the Session-global numeric ID namespace, greater than every ID minted since
* this storage was opened and every ID stored, also across reopen: scans order by ID as creation order, and the
* Harness relies on it.
*/
mintId<I extends Id<string>>(): Promise<I>;

38
packages/durable/test/examples/04-chord-state.ts
View file

@ -1,38 +0,0 @@
// Expose a document through Chord.
// Run from packages/durable:
// node --conditions=source --experimental-strip-types test/examples/04-chord-state.ts
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
import { createSession, defineDoc, MemoryStorage } from "../../src/index.ts";
const context = BACKGROUND_CONTEXT;
const session = createSession(new MemoryStorage());
const Notes = defineDoc<{ text: string }>({
checkpointWhen: () => false,
kind: "example.notes",
version: 1,
scope: "conversation",
history: "rewindable",
fork: "asOf",
initial: () => ({ text: "" }),
});
const chat = await session.commit(async (tx) => {
const conversation = await tx.createConversation({ ownership: { kind: "ownerless" } });
(await tx.doc(Notes, conversation.id)).text = "first";
return conversation;
}, context);
// documentState() never creates a document. It returns a hydrated read-only
// Chord state bound to the current concrete incarnation.
const notesState = await session.documentState(Notes, chat.id, context);
if (notesState === undefined) throw new Error("notes are absent");
const stopNotes = notesState.subscribe((value, _deliveryContext, delivery) => {
console.log("Chord notes:", delivery.kind, delivery.sequence, value);
});
await session.commit(async (tx) => {
(await tx.doc(Notes, chat.id)).text = "published through Chord";
}, context);
stopNotes();
notesState.dispose();
await session.close(context);

6
packages/durable/test/examples/21-late-join.ts
View file

@ -56,14 +56,14 @@ const submission = await root.submit({ type: "input", content: "Count to ten, th
await pause(500);
// Structural client: the view holds the committed transcript and documents, including the running tool's output.
const view = await root.viewState(context);
const view = await root.watch(context);
const live = view.value.docs["pi.live"] as LiveState;
console.log(
"view entries:",
view.value.entries.map((entry) => entry.kind),
);
console.log("view tool slot:", live.tools?.[0]?.status, JSON.stringify(live.tools?.[0]?.output));
view.subscribe((value) => {
view.start(async (value) => {
const slot = (value.docs["pi.live"] as LiveState).tools?.[0];
if (slot?.status === "running") console.log("view output now:", JSON.stringify(slot.output));
});
@ -96,5 +96,5 @@ await submission.wait(context);
await harness.waitForIdle(context);
await pause(0);
await stream.stop();
view.dispose();
await view.stop();
await harness.close(context);

4
packages/durable/test/examples/24-child-tasks.ts
View file

@ -128,9 +128,9 @@ console.log("The customer cancels:");
id = await checkout(root, ["visa-5", "visa-6", "visa-7", "visa-8"]);
harness.resume();
await new Promise((resolve) => setTimeout(resolve, 20));
const graph = await harness.taskGraph(context);
const graph = await harness.watchTaskGraph(context);
printGraph(graph.value);
graph.dispose();
await graph.stop();
await harness.abortTask(id, context);
console.log(` checkout: ${(await harness.waitForTask(id, context)).state.outcome.status}`);

32
packages/durable/test/harness-blocked.test.ts
View file

@ -104,7 +104,7 @@ describe("recovery at open", () => {
...rest,
state: { status: "completing", outcome: { status: "completed", result: null } },
} as const;
await raw.commit([{ type: "task", value: completing }], context);
await raw.commit([{ type: "task", value: completing }], () => {}, context);
await raw.close(context);
// That pass's commit fails, as on a platform reset.
@ -158,7 +158,9 @@ describe("tasks that cannot run", () => {
kind: "blocked",
reason: "restart_limit",
});
const graph = await harness.taskGraph(context);
const graph = await harness.watchTaskGraph(context);
// A started watch keeps its value current.
graph.start(async () => {});
expect(graph.value.tasks[String(id)]?.blocked).toBe("restart_limit");
// Parked work needs no process: nothing is active.
await eventually(() => !harness.wake().active);
@ -172,7 +174,7 @@ describe("tasks that cannot run", () => {
await harness.retryTask(id, context);
expect((await harness.waitForTask(id, context)).state.outcome.status).toBe("completed");
expect(state.started).toBe(4);
graph.dispose();
await graph.stop();
await harness.close(context);
});
@ -282,7 +284,9 @@ describe("tasks that cannot run", () => {
await eventually(() => state.started === 1);
opened.kill(); // a crash: a close would not count
opened = await openWith(path, registry, settings);
const graph = await opened.harness.taskGraph(context);
const graph = await opened.harness.watchTaskGraph(context);
// A started watch keeps its value current.
graph.start(async () => {});
opened.harness.resume();
await eventually(() => graph.value.tasks[String(id)]?.blocked === "restart_limit");
// Let a scheduling pass leave the parked task out.
@ -293,7 +297,7 @@ describe("tasks that cannot run", () => {
opened.harness.resume();
expect((await opened.harness.waitForTask(id, context)).state.outcome.status).toBe("completed");
expect(state.started).toBe(2);
graph.dispose();
await graph.stop();
await opened.harness.close(context);
});
@ -437,13 +441,15 @@ describe("tasks that cannot run", () => {
const second = createRegistry();
second.install(defineExtension({ name: "v2", tasks: [V2] }));
opened = await openWith(path, second);
const graph = await opened.harness.taskGraph(context);
const graph = await opened.harness.watchTaskGraph(context);
// A started watch keeps its value current.
graph.start(async () => {});
opened.harness.resume();
await eventually(() => graph.value.tasks[String(id)]?.blocked === "migration_failed");
await eventually(() => !opened.harness.wake().active);
expect(opened.harness.wake()).toEqual({ active: false });
await eventually(() => opened.wakes.at(-1)?.active === false);
graph.dispose();
await graph.stop();
await opened.harness.close(context);
});
@ -466,11 +472,13 @@ describe("tasks that cannot run", () => {
await opened.harness.close(context);
// The next open marks it `restart`; without its definition, it waits for one to clean up.
opened = await openWith(path, createRegistry());
const graph = await opened.harness.taskGraph(context);
const graph = await opened.harness.watchTaskGraph(context);
// A started watch keeps its value current.
graph.start(async () => {});
opened.harness.resume();
await eventually(async () => (await opened.harness.getTask(id, context))?.abortReason === "restart");
await eventually(() => graph.value.tasks[String(id)]?.blocked === "missing_task");
graph.dispose();
await graph.stop();
await opened.harness.close(context);
});
@ -485,7 +493,9 @@ describe("tasks that cannot run", () => {
const registry = createRegistry();
const path = await sqlitePath();
const { harness, wakes } = await openWith(path, registry);
const graph = await harness.taskGraph(context);
const graph = await harness.watchTaskGraph(context);
// A started watch keeps its value current.
graph.start(async () => {});
const root = await harness.root(context);
const id = await root.commit(
(tx) => tx.createTask(Missing, null, { ownership: { kind: "conversation" } }),
@ -499,7 +509,7 @@ describe("tasks that cannot run", () => {
await eventually(() => graph.value.tasks[String(id)]?.blocked === undefined);
await eventually(() => wakes.some((wake) => wake.active));
expect((await harness.waitForTask(id, context)).state.outcome.status).toBe("completed");
graph.dispose();
await graph.stop();
await harness.close(context);
});
});

20
packages/durable/test/harness-cancellation-barrier.test.ts
View file

@ -1,5 +1,12 @@
import type { Context } from "@earendil-works/chord";
import { createSession, defineTask, MemoryStorage, type StorageWrite, type TaskId } from "@earendil-works/pi-durable";
import {
createSession,
defineTask,
MemoryStorage,
type Seq,
type StorageWrite,
type TaskId,
} from "@earendil-works/pi-durable";
import { describe, expect, it } from "vitest";
import { context, flush } from "./session-support.ts";
import { aborted, deferred, eventually, openTasks } from "./task-support.ts";
@ -24,14 +31,14 @@ class HoldMarks extends MemoryStorage {
target: TaskId | undefined;
readonly held = deferred();
readonly release = deferred();
override async commit(writes: readonly StorageWrite[], commitContext: Context) {
override async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context) {
if (
writes.some((write) => write.type === "task" && write.value.id === this.target && write.value.abortRequested)
) {
this.held.resolve();
await this.release.promise;
}
return super.commit(writes, commitContext);
return super.commit(writes, adopt, commitContext);
}
}
@ -128,6 +135,7 @@ describe("work below a cancelled owner", () => {
value: { ...failedRecord, state: { status: "terminal", outcome: failure } } as typeof failedRecord,
},
],
() => {},
context,
);
// The child's definition is missing: a restart mark would keep it waiting, a request orphans it.
@ -209,7 +217,11 @@ describe("work below a cancelled owner", () => {
);
const record = (await storage.task(holder, context))!;
const completing = { status: "completing", outcome: { status: "completed", result: null } } as const;
await storage.commit([{ type: "task", value: { ...record, state: completing } as typeof record }], context);
await storage.commit(
[{ type: "task", value: { ...record, state: completing } as typeof record }],
() => {},
context,
);
const { harness } = await openTasks(storage, [Dormant]);
try {
harness.resume();

15
packages/durable/test/harness-compaction.test.ts
View file

@ -1449,9 +1449,9 @@ describe("compaction estimates and interactions", () => {
// The parent is untouched.
expect((await kinds(chat.root)).includes("pi.compaction")).toBe(false);
// The fork's view keeps the parent entries the summary kept.
const state = await fork.viewState(context);
expect(state.value.entries).toEqual(view.entries);
state.dispose();
const watch = await fork.watch(context);
expect(watch.value.entries).toEqual(view.entries);
await watch.stop();
await chat.harness.close(context);
});
@ -1896,12 +1896,13 @@ describe("compaction and the inbox", () => {
it("keeps the kept entries mounted in the conversation view", async () => {
const chat = await open();
await history(chat);
const state = await chat.root.viewState(context);
const watch = await chat.root.watch(context);
watch.start(async () => {});
chat.faux.summaries.push(summary());
await result(chat, await chat.root.compact(undefined, context));
await waitFor(() => state.value.entries[0]?.kind === "pi.compaction");
expect(state.value.entries).toEqual((await chat.root.context(context)).entries);
state.dispose();
await waitFor(() => watch.value.entries[0]?.kind === "pi.compaction");
expect(watch.value.entries).toEqual((await chat.root.context(context)).entries);
await watch.stop();
await chat.harness.close(context);
});
});

6
packages/durable/test/harness-conversations.test.ts
View file

@ -469,9 +469,9 @@ describe("Harness lifecycle", () => {
}, context);
expect(await harness.snapshot(NoteDoc, root.id, context)).toEqual({ text: "second" });
expect(await harness.snapshotAsOf(NoteDoc, root.id, entry.id, context)).toEqual({ text: "first" });
const state = await harness.documentState(NoteDoc, root.id, context);
expect(state?.value).toEqual({ text: "second" });
state?.dispose();
const watch = await harness.watchDoc(NoteDoc, root.id, context);
expect(watch?.value).toEqual({ text: "second" });
await watch?.stop();
await harness.close(context);
});
});

2
packages/durable/test/harness-events.test.ts
View file

@ -593,7 +593,7 @@ describe("agent events", () => {
const { harness, root } = await openChat(new MemoryStorage(), chatSetup());
const stream: AgentEventStream = await watchEvents(harness, root.id, context);
await harness.close(context);
expect(await stream.closed).toEqual({ reason: "session_closed" });
expect(await stream.closed).toEqual({ reason: "ended" });
});
it("starts a message at the first committed partial and ends it with the converted entry on abort", async () => {

5
packages/durable/test/harness-inbox.test.ts
View file

@ -731,7 +731,8 @@ describe("inbox", () => {
class RecordingStorage extends MemoryStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
callContext: Parameters<MemoryStorage["commit"]>[1],
adopt: Parameters<MemoryStorage["commit"]>[1],
callContext: Parameters<MemoryStorage["commit"]>[2],
) {
for (const write of writes) {
if (write.type !== "document.change" || write.id !== inboxId) continue;
@ -739,7 +740,7 @@ describe("inbox", () => {
const empty = content.kind === "base" && (content.value as InboxState).items.length === 0;
written.push({ kind: content.kind, empty });
}
return super.commit(writes, callContext);
return super.commit(writes, adopt, callContext);
}
}
const setup = chatSetup();

54
packages/durable/test/harness-lifecycle.test.ts
View file

@ -78,6 +78,7 @@ async function seedRunningTask(storage: Storage): Promise<void> {
{ type: "conversation", value: { id: conversationId } },
{ type: "task", value: { ...task, abortRequested: false } },
],
() => {},
context,
);
}
@ -274,7 +275,7 @@ describe("Harness close", () => {
await harness.close(context);
});
it("publishes no frame to states and watches from a commit that settles during close", async () => {
it("publishes no frame to watches from a commit that settles during close", async () => {
const Notes = defineDoc<{ text: string }>({
checkpointWhen: () => false,
kind: "test.close-frames",
@ -288,27 +289,15 @@ describe("Harness close", () => {
await harness.commit(async (tx) => {
(await tx.doc(Notes)).text = "before";
}, context);
const docState = (await harness.documentState(Notes, context))!;
const docWatch = (await harness.watchDoc(Notes, context))!;
const viewState = await root.viewState(context);
const viewWatch = await root.watch(context);
const graphState = await harness.taskGraph(context);
const graphWatch = await harness.watchTaskGraph(context);
const frames: string[] = [];
graphState.subscribe((_value, _context, delivery) => {
if (delivery.kind === "update") frames.push("graphState");
});
graphWatch.start(async () => void frames.push("graphWatch"));
docState.subscribe((_value, _context, delivery) => {
if (delivery.kind === "update") frames.push("docState");
});
viewState.subscribe((_value, _context, delivery) => {
if (delivery.kind === "update") frames.push("viewState");
});
docWatch.start(async () => void frames.push("docWatch"));
viewWatch.start(async () => void frames.push("viewWatch"));
const docValue = docState.value;
const viewValue = viewState.value;
const docValue = docWatch.value;
const viewValue = viewWatch.value;
const held = storage.holdCommits();
const committing = harness.commit(async (tx) => {
@ -326,12 +315,12 @@ describe("Harness close", () => {
await closing;
await flush();
expect(frames).toEqual([]);
expect(docState.value).toBe(docValue);
expect(viewState.value).toBe(viewValue);
expect(graphState.value).toEqual({ tasks: {} });
expect(await graphWatch.closed).toEqual({ reason: "session_closed" });
expect(await docWatch.closed).toEqual({ reason: "session_closed" });
expect(await viewWatch.closed).toEqual({ reason: "session_closed" });
expect(docWatch.value).toBe(docValue);
expect(viewWatch.value).toBe(viewValue);
expect(graphWatch.value).toEqual({ tasks: {} });
expect(await graphWatch.closed).toEqual({ reason: "ended" });
expect(await docWatch.closed).toEqual({ reason: "ended" });
expect(await viewWatch.closed).toEqual({ reason: "ended" });
// The commit itself settled.
expect(storage.commits.at(-1)).toContainEqual(expect.objectContaining({ type: "entry" }));
});
@ -385,12 +374,12 @@ describe("Harness close", () => {
find.release();
await expect(viewWatch).rejects.toThrow("cancelled");
// No observer kept the mount: the next observer builds a new one, and the one after that another.
const first = await root.viewState(context);
const first = await root.watch(context);
const firstValue = first.value;
first.dispose();
const second = await root.viewState(context);
await first.stop();
const second = await root.watch(context);
expect(second.value).not.toBe(firstValue);
second.dispose();
await second.stop();
await harness.close(context);
});
@ -418,9 +407,7 @@ describe("Harness close", () => {
reset: () => root.reset(undefined, context),
abort: () => root.abort(context),
conversationIdle: () => root.waitForIdle(context),
viewState: () => root.viewState(context),
watch: () => root.watch(context),
taskGraph: () => harness.taskGraph(context),
watchTaskGraph: () => harness.watchTaskGraph(context),
status: () => submission.status(context),
wait: () => submission.wait(context),
@ -488,13 +475,10 @@ describe("Harness close", () => {
snapshot: settle(harness.snapshot(Notes, context)),
settledWait: settle(write.wait(context)),
absentWatch: settle(harness.watchDoc(Absent, context)),
absentState: settle(harness.documentState(Absent, context)),
waitForTask: settle(harness.waitForTask(taskId, context)),
documentState: settle(harness.documentState(Notes, context)),
watchDoc: settle(harness.watchDoc(Notes, context)),
viewState: settle(root.viewState(context)),
watch: settle(root.watch(context)),
taskGraph: settle(harness.taskGraph(context)),
watchTaskGraph: settle(harness.watchTaskGraph(context)),
};
const closing = harness.close(context);
held.release();
@ -506,13 +490,10 @@ describe("Harness close", () => {
snapshot: "resolved",
settledWait: "resolved",
absentWatch: "undefined",
absentState: "undefined",
waitForTask: "closed",
documentState: "closed",
watchDoc: "closed",
viewState: "closed",
watch: "closed",
taskGraph: "closed",
watchTaskGraph: "closed",
});
await closing;
});
@ -551,16 +532,13 @@ describe("scheduling on a paused Harness", () => {
await harness.usage(context);
await harness.conversation(root.id, context);
await harness.snapshot(Notes, context);
(await harness.documentState(Notes, context))!.dispose();
await (await harness.watchDoc(Notes, context))!.stop();
const submission = (await harness.submission(submissionId, context))!;
await submission.status(context);
await root.agent(context);
await root.context(context);
await root.entries({}, 10, undefined, context);
(await root.viewState(context)).dispose();
await (await root.watch(context)).stop();
(await harness.taskGraph(context)).dispose();
await (await harness.watchTaskGraph(context)).stop();
await flush();
await flush();

5
packages/durable/test/harness-live-deltas.test.ts
View file

@ -283,13 +283,14 @@ describe("pi.live deltas", () => {
class RecordingStorage extends MemoryStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
context: Parameters<MemoryStorage["commit"]>[1],
adopt: Parameters<MemoryStorage["commit"]>[1],
context: Parameters<MemoryStorage["commit"]>[2],
) {
let kind: "base" | "delta" | undefined;
for (const write of writes) {
if (write.type === "document.change" && write.id === liveId) kind = write.content.kind;
}
const seq = await super.commit(writes, context);
const seq = await super.commit(writes, adopt, context);
if (kind !== undefined) written.set(seq, kind);
return seq;
}

5
packages/durable/test/harness-nested-tools.test.ts
View file

@ -115,11 +115,11 @@ describe("nested tool calls", () => {
// Per document written, the keys of the nested calls it holds.
const written: { kind: string; key: string | undefined }[] = [];
const commit = storage.commit.bind(storage);
storage.commit = (writes, commitContext) => {
storage.commit = (writes, adopt, commitContext) => {
for (const write of writes) {
if (write.type === "document.create") written.push({ kind: write.record.kind, key: write.record.key });
}
return commit(writes, commitContext);
return commit(writes, adopt, commitContext);
};
const { harness, root } = await openChat(storage, setup);
await (await root.submit({ type: "input", content: "go" }, context)).wait(context);
@ -1299,6 +1299,7 @@ describe("tool task version 1 records", () => {
const state = { ...record.state, checkpoint } as typeof record.state;
await storage.commit(
[{ type: "task", value: { ...record, version: 1, input, state } as typeof record }],
() => {},
context,
);
await storage.close(context);

9
packages/durable/test/harness-ownership.test.ts
View file

@ -940,14 +940,14 @@ describe("ownership", () => {
const storage = await openNodeSqliteStorage(path);
let rejectMark: TaskId | undefined;
const commit = storage.commit.bind(storage);
storage.commit = async (writes, ctx) => {
storage.commit = async (writes, adopt, ctx) => {
if (
writes.some((write) => write.type === "task" && write.value.id === rejectMark && write.value.abortRequested)
) {
rejectMark = undefined;
throw new Error("disk gone");
}
return commit(writes, ctx);
return commit(writes, adopt, ctx);
};
opened = await openHarness(storage);
const child = (await opened.harness.conversation(tree.child, context))!;
@ -973,7 +973,8 @@ describe("ownership", () => {
class Failing extends ControlledStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
ctx: Parameters<MemoryStorage["commit"]>[1],
adopt: Parameters<MemoryStorage["commit"]>[1],
ctx: Parameters<MemoryStorage["commit"]>[2],
) {
const marks = writes.some(
(write) => write.type === "task" && write.value.id === failMark && write.value.abortRequested,
@ -982,7 +983,7 @@ describe("ownership", () => {
failMark = undefined;
throw failure;
}
return super.commit(writes, ctx);
return super.commit(writes, adopt, ctx);
}
}
const storage = new Failing();

8
packages/durable/test/harness-scheduler-indexes.test.ts
View file

@ -107,13 +107,13 @@ describe("scheduler indexes", () => {
const storage = await openNodeSqliteStorage(path);
const reasons: (string | undefined)[] = [];
const commit = storage.commit.bind(storage);
storage.commit = (writes, commitContext) => {
storage.commit = (writes, adopt, commitContext) => {
for (const write of writes) {
if (write.type === "task" && write.value.id === ids.leaf && write.value.abortRequested) {
reasons.push(write.value.abortReason);
}
}
return commit(writes, commitContext);
return commit(writes, adopt, commitContext);
};
const { harness } = await openTasks(storage, [Hold]);
harness.resume();
@ -128,12 +128,12 @@ describe("scheduler indexes", () => {
const endedIn = new Map<TaskId, number>();
let commits = 0;
const commit = storage.commit.bind(storage);
storage.commit = (writes, commitContext) => {
storage.commit = (writes, adopt, commitContext) => {
commits++;
for (const write of writes) {
if (write.type === "task" && write.value.state.status === "terminal") endedIn.set(write.value.id, commits);
}
return commit(writes, commitContext);
return commit(writes, adopt, commitContext);
};
const opened = await openTasks(storage, [Hold]);
const harness = opened.harness;

6
packages/durable/test/harness-sleep.test.ts
View file

@ -94,7 +94,9 @@ describe("sleeping tasks", () => {
now: () => clock,
onWake: (wake) => wakes.push(wake),
});
const graph = await harness.taskGraph(context);
const graph = await harness.watchTaskGraph(context);
// A started watch keeps its value current.
graph.start(async () => {});
const id = await startNap(harness, Nap);
harness.resume();
await eventually(async () => (await stateOf(harness, id))?.status === "sleeping");
@ -133,7 +135,7 @@ describe("sleeping tasks", () => {
for (let index = 1; index < wakes.length; index++) expect(wakes[index]).not.toEqual(wakes[index - 1]);
expect(wakes).toContainEqual({ active: true });
expect(wakes).toContainEqual({ active: false, at: 2_000 });
graph.dispose();
await graph.stop();
await harness.close(context);
});

12
packages/durable/test/harness-storage-failure.test.ts
View file

@ -106,8 +106,8 @@ describe("a failed Storage call", () => {
const failed = { name: "SessionFailed", cause: failure };
await expect(task).rejects.toMatchObject(failed);
await expect(idle).rejects.toMatchObject(failed);
expect(await watch.closed).toEqual({ reason: "session_failed", error: failure });
expect(await stream.closed).toEqual({ reason: "session_failed", error: failure });
expect(await watch.closed).toEqual({ reason: "failed", error: failure });
expect(await stream.closed).toEqual({ reason: "failed", error: failure });
// Every later call, and a wait that starts now.
await expect(harness.getTask(id, context)).rejects.toMatchObject(failed);
await expect(harness.waitForIdle(context)).rejects.toMatchObject(failed);
@ -229,7 +229,7 @@ describe("a failed Storage call", () => {
await harness.close(context);
});
it("ends what a task holds with session_failed, not as cancelled by its signal", async () => {
it("ends what a task holds as failed, not as cancelled by its signal", async () => {
const ends: unknown[] = [];
const watching = deferred();
const Watcher = defineTask<null, { phase: "work" }, null>({
@ -260,7 +260,7 @@ describe("a failed Storage call", () => {
storage.failNextCommit(failure);
await expect(root.commit((tx) => tx.appendEntry(root.id, { kind: "note" }), context)).rejects.toBe(failure);
await opened.harness.closed;
expect(ends).toEqual([{ reason: "session_failed", error: failure }, "SessionFailed"]);
expect(ends).toEqual([{ reason: "failed", error: failure }, "SessionFailed"]);
await opened.harness.close(context);
});
@ -491,7 +491,7 @@ describe("listener and report errors", () => {
publication: () => {
throw failure;
},
closeSession: () => {},
end: () => {},
}),
context,
);
@ -504,7 +504,7 @@ describe("listener and report errors", () => {
while (seen.length === 0) await new Promise((resolve) => setTimeout(resolve, 1));
expect(reports).toEqual([failure]);
await harness.close(context);
expect(await watch.closed).toEqual({ reason: "session_closed" });
expect(await watch.closed).toEqual({ reason: "ended" });
});
it("contains a fulfilled thenable that calls only its fulfillment handler", async () => {

10
packages/durable/test/harness-structured.test.ts
View file

@ -1355,7 +1355,8 @@ describe("tool rounds", () => {
class Failing extends ControlledStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
ctx: Parameters<MemoryStorage["commit"]>[1],
adopt: Parameters<MemoryStorage["commit"]>[1],
ctx: Parameters<MemoryStorage["commit"]>[2],
) {
const finalizes = writes.some(
(write) => write.type === "task" && write.value.id === reject && write.value.state.status === "terminal",
@ -1364,7 +1365,7 @@ describe("tool rounds", () => {
reject = undefined;
throw failure;
}
return super.commit(writes, ctx);
return super.commit(writes, adopt, ctx);
}
}
const storage = new Failing();
@ -1632,7 +1633,8 @@ describe("conversation abort and boundaries", () => {
class Failing extends ControlledStorage {
override async commit(
writes: Parameters<MemoryStorage["commit"]>[0],
ctx: Parameters<MemoryStorage["commit"]>[1],
adopt: Parameters<MemoryStorage["commit"]>[1],
ctx: Parameters<MemoryStorage["commit"]>[2],
) {
const finalizes = writes.some(
(write) => write.type === "task" && write.value.id === reject && write.value.state.status === "terminal",
@ -1641,7 +1643,7 @@ describe("conversation abort and boundaries", () => {
reject = undefined;
throw failure;
}
return super.commit(writes, ctx);
return super.commit(writes, adopt, ctx);
}
}
const storage = new Failing();

45
packages/durable/test/harness-task-graph.test.ts
View file

@ -89,9 +89,9 @@ describe("task graph view", () => {
const root = await harness.root(context);
const seen: TaskGraph[] = [];
const observe = async () => {
const opened = await harness.taskGraph(context);
opened.subscribe((value, _context, delivery) => {
if (delivery.kind === "update") seen.push(value);
const opened = await harness.watchTaskGraph(context);
opened.start(async (value) => {
seen.push(value);
});
return opened;
};
@ -100,7 +100,7 @@ describe("task graph view", () => {
/** The advanced value equals a fresh build from Storage once the last observer left. */
const rebuild = async () => {
const advanced = graph.value;
graph.dispose();
await graph.stop();
graph = await observe();
expect(graph.value).not.toBe(advanced);
expect(graph.value).toEqual(advanced);
@ -161,7 +161,7 @@ describe("task graph view", () => {
for (let index = 1; index < seen.length; index++) expect(seen[index]).not.toEqual(seen[index - 1]);
// The commit that created the two conversations published one revision setting them.
expect(seen.some((value) => value.tasks[String(parent)]?.conversations.length === 2)).toBe(true);
graph.dispose();
await graph.stop();
await harness.close(context);
});
@ -195,7 +195,7 @@ describe("task graph view", () => {
await eventually(async () =>
(await first.harness.inspect(context)).tasks.every((task) => task.state.kind === "running"),
);
const running = await first.harness.taskGraph(context);
const running = await first.harness.watchTaskGraph(context);
expect(statuses(running.value)).toEqual({
[`test.graph-work#${foreground}`]: "running",
[`test.graph-work#${background}`]: "running",
@ -263,7 +263,8 @@ describe("task graph view", () => {
});
const { harness } = await openTasks(new MemoryStorage(), [Spawner]);
const root = await harness.root(context);
const graph = await harness.taskGraph(context);
const graph = await harness.watchTaskGraph(context);
graph.start(async () => {});
const id = await root.commit(async (tx) => {
const entry = await tx.appendEntry(root.id, { kind: "note" });
return tx.createTask(Spawner, { at: entry.id }, { ownership: { kind: "conversation" } });
@ -272,10 +273,10 @@ describe("task graph view", () => {
await eventually(() => (graph.value.tasks[String(id)]?.conversations.length ?? 0) === 2);
const advanced = graph.value;
expect(advanced.tasks[String(id)]!.conversations).toEqual([...created].sort((a, b) => a - b));
graph.dispose();
const rebuilt = await harness.taskGraph(context);
await graph.stop();
const rebuilt = await harness.watchTaskGraph(context);
expect(rebuilt.value).toEqual(advanced);
rebuilt.dispose();
await rebuilt.stop();
gate.resolve();
await harness.waitForTask(id, context);
await harness.close(context);
@ -299,13 +300,13 @@ describe("task graph view", () => {
await blocking;
await expect(cancelled).rejects.toThrow("cancelled");
// No observer kept the mount: each new observer builds a new revision.
const first = await harness.taskGraph(context);
const first = await harness.watchTaskGraph(context);
const value = first.value;
first.dispose();
const second = await harness.taskGraph(context);
await first.stop();
const second = await harness.watchTaskGraph(context);
expect(second.value).not.toBe(value);
expect(second.value).toEqual(value);
second.dispose();
await second.stop();
await harness.close(context);
});
@ -329,12 +330,12 @@ describe("task graph view", () => {
const { harness } = await openTasks(new MemoryStorage(), [Memo]);
const root = await harness.root(context);
const id = await root.commit((tx) => tx.createTask(Memo, null, { ownership: { kind: "conversation" } }), context);
const first = await harness.taskGraph(context);
const second = await harness.taskGraph(context);
const first = await harness.watchTaskGraph(context);
const second = await harness.watchTaskGraph(context);
expect(second.value).toBe(first.value);
const updates: string[] = [];
first.subscribe((value, _context, delivery) => {
if (delivery.kind === "update") updates.push(value.tasks[String(id)]?.state.status ?? "gone");
first.start(async (value) => {
updates.push(value.tasks[String(id)]?.state.status ?? "gone");
});
harness.resume();
await reached.promise;
@ -346,13 +347,13 @@ describe("task graph view", () => {
await flush();
expect(updates).toEqual(["running", "gone"]);
const last = first.value;
first.dispose();
second.dispose();
await first.stop();
await second.stop();
// No observer is left, so the mount was dropped: a new observer builds a new revision.
const rebuilt = await harness.taskGraph(context);
const rebuilt = await harness.watchTaskGraph(context);
expect(rebuilt.value).toEqual(last);
expect(rebuilt.value).not.toBe(last);
rebuilt.dispose();
await rebuilt.stop();
await harness.close(context);
});
});

2
packages/durable/test/harness-tasks.test.ts
View file

@ -1198,7 +1198,7 @@ describe("task close", () => {
await reached.promise;
await harness.close(context);
await expect(fromListener).rejects.toThrow("closed");
expect(watchEnd).toEqual({ reason: "session_closed" });
expect(watchEnd).toEqual({ reason: "ended" });
});
it("writes no fault when close seals while the step after a failed phase is queued", async () => {

30
packages/durable/test/harness-view.test.ts
View file

@ -34,9 +34,9 @@ async function record(conversation: Conversation) {
/** A freshly built view of `conversation`, for comparing with an advanced one. */
async function fresh(conversation: Conversation): Promise<ConversationView> {
const state = await conversation.viewState(context);
const value = state.value;
state.dispose();
const watch = await conversation.watch(context);
const value = watch.value;
await watch.stop();
return value;
}
@ -246,20 +246,20 @@ describe("conversation view", () => {
await harness.close(context);
});
it("keeps states and watches of one conversation independent and remounts after the last one detaches", async () => {
it("keeps watches of one conversation independent and remounts after the last one detaches", async () => {
const { harness, root } = await openChat(new MemoryStorage(), chatSetup());
const state = await root.viewState(context);
const kept = await record(root);
const { frames, stop } = await record(root);
await root.commit((tx) => tx.appendEntry(root.id, { kind: "one" }), context);
await drained();
expect(state.value.entries.map((entry) => entry.kind)).toEqual(["one"]);
expect(kept.frames.at(-1)!.value.entries.map((entry) => entry.kind)).toEqual(["one"]);
await stop();
expect(frames).toHaveLength(1);
await root.commit((tx) => tx.appendEntry(root.id, { kind: "two" }), context);
await drained();
expect(state.value.entries.map((entry) => entry.kind)).toEqual(["one", "two"]);
const last = state.value;
state.dispose();
const last = kept.frames.at(-1)!.value;
expect(last.entries.map((entry) => entry.kind)).toEqual(["one", "two"]);
await kept.stop();
// No observer is left, so the mount was dropped: a new observer builds a new revision from committed state.
const rebuilt = await fresh(root);
expect(rebuilt).toEqual(last);
@ -267,15 +267,13 @@ describe("conversation view", () => {
await harness.close(context);
});
it("ends states and watches at close and rejects later acquisition", async () => {
it("ends watches at close and rejects later acquisition", async () => {
const { harness, root } = await openChat(new MemoryStorage(), chatSetup());
const watch = await root.watch(context);
const state = await root.viewState(context);
await harness.close(context);
expect(await watch.closed).toEqual({ reason: "session_closed" });
expect(state.value.entries).toEqual([]);
expect(await watch.closed).toEqual({ reason: "ended" });
expect(watch.value.entries).toEqual([]);
await expect(root.watch(context)).rejects.toThrow();
await expect(root.viewState(context)).rejects.toThrow();
});
it("rejects an acquisition cancelled or closed while it waits for the Session line", async () => {
@ -308,9 +306,9 @@ describe("conversation view", () => {
const { harness, root } = await openChat(new MemoryStorage(), chatSetup());
const failing = await root.watch(context);
const { frames, stop } = await record(root);
const shared = await root.viewState(context);
const shared = await root.watch(context);
expect(failing.value).toBe(shared.value);
shared.dispose();
await shared.stop();
failing.start(async () => {
throw new Error("listener failed");
});

18
packages/durable/test/indexeddb-storage.test.ts
View file

@ -22,6 +22,9 @@ import type {
} from "../src/types.ts";
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
Object.defineProperty(globalThis, "IDBKeyRange", { configurable: true, value: FakeIDBKeyRange });
const context = BACKGROUND_CONTEXT;
@ -39,10 +42,10 @@ class ReopeningStorage implements Storage {
this.indexedDB = indexedDB;
}
async commit(writes: readonly StorageWrite[], commitContext: Context): Promise<Seq> {
if (this.closed) return this.current.commit(writes, commitContext);
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context): Promise<Seq> {
if (this.closed) return this.current.commit(writes, adopt, commitContext);
try {
return await this.current.commit(writes, commitContext);
return await this.current.commit(writes, adopt, commitContext);
} finally {
await this.current.close(context);
this.current = await IndexedDbStorage.open(this.name, { indexedDB: this.indexedDB });
@ -132,7 +135,7 @@ function pendingTask(id: TaskId<JsonValue>, conversationId = ROOT_CONVERSATION_I
}
async function createRoot(storage: Storage): Promise<void> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
}
it("persists minted IDs across reopen", async () => {
@ -163,6 +166,7 @@ it("rolls back table writes when a later document copy fails", async () => {
} satisfies DocumentCreate;
await storage.commit(
[{ type: "document.create", record: source, content: { kind: "base", version: 1, value: { count: 1 } } }],
adopt,
context,
);
const taskId = await storage.mintId<TaskId<JsonValue>>();
@ -177,6 +181,7 @@ it("rolls back table writes when a later document copy fails", async () => {
source: { id: sourceId, at: "current" },
},
],
adopt,
context,
),
).rejects.toThrow("was rejected");
@ -207,6 +212,7 @@ it("scans entries and head markers through fork ancestry", async () => {
},
},
],
adopt,
context,
);
const childId = await storage.mintId<ConversationId>();
@ -217,11 +223,13 @@ it("scans entries and head markers through fork ancestry", async () => {
value: { id: childId, parent: { conversationId: ROOT_CONVERSATION_ID, at: rootMarker } },
},
],
adopt,
context,
);
const childEntry = await storage.mintId<EntryId>();
await storage.commit(
[{ type: "entry", value: { id: childEntry, conversationId: childId, kind: "child" } }],
adopt,
context,
);
expect(
@ -271,7 +279,7 @@ it("keeps one readwrite transaction active through many mixed writes", async ()
},
);
}
await storage.commit(writes, context);
await storage.commit(writes, adopt, context);
expect((await storage.entry(lastEntryId!, context))?.entry.data).toEqual({ index: 39 });
expect(await storage.task(lastTaskId!, context)).toEqual(pendingTask(lastTaskId!));
expect((await storage.submission(lastSubmissionId!, context))?.requestId).toBe("bulk-39");

70
packages/durable/test/jsonl-storage.test.ts
View file

@ -24,6 +24,9 @@ import type {
} from "../src/types.ts";
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
const context = BACKGROUND_CONTEXT;
type StoredTask = TaskRecord<JsonValue, JsonValue, JsonValue>;
const tempDirectories = new Set<string>();
@ -66,10 +69,10 @@ class ReopeningStorage implements Storage {
this.directory = directory;
}
async commit(writes: readonly StorageWrite[], commitContext: Context): Promise<Seq> {
if (this.closed) return this.current.commit(writes, commitContext);
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context): Promise<Seq> {
if (this.closed) return this.current.commit(writes, adopt, commitContext);
try {
return await this.current.commit(writes, commitContext);
return await this.current.commit(writes, adopt, commitContext);
} finally {
await this.current.close(context);
this.current = await JsonlStorage.open(this.directory, new NodeExecutionEnv({ cwd: this.directory }), context);
@ -160,7 +163,7 @@ function terminalTask(id: TaskId<JsonValue>): StoredTask {
}
async function createRoot(storage: Storage): Promise<void> {
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
}
function sessionDocument(id: DocumentId, kind = "test.document"): DocumentCreate {
@ -329,7 +332,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
await createRoot(storage);
const taskId = await storage.mintId<TaskId<JsonValue>>();
const documentId = await storage.mintId<DocumentId>();
await storage.commit([{ type: "task", value: pendingTask(taskId) }], context);
await storage.commit([{ type: "task", value: pendingTask(taskId) }], adopt, context);
await storage.commit(
[
{
@ -338,10 +341,12 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
await storage.commit(
[{ type: "document.change", id: documentId, content: { kind: "delta", version: 1, ops: [] } }],
adopt,
context,
);
await storage.commit(
@ -352,13 +357,14 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 1 } },
},
],
adopt,
context,
);
expect(await readLines(join(directory, `task-${taskId}.jsonl`))).toHaveLength(1);
expect(await readLines(join(directory, `doc-${documentId}.jsonl`))).toHaveLength(1);
await storage.commit([{ type: "document.retire", id: documentId }], context);
await storage.commit([{ type: "task", value: terminalTask(taskId) }], context);
await storage.commit([{ type: "document.retire", id: documentId }], adopt, context);
await storage.commit([{ type: "task", value: terminalTask(taskId) }], adopt, context);
expect(await readLines(join(directory, "main.jsonl"))).toHaveLength(7);
expect(await fileExists(join(directory, `task-${taskId}.jsonl`))).toBe(false);
@ -379,6 +385,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
{ type: "task", value: pendingTask(taskId, "first") },
{ type: "task", value: pendingTask(taskId, "second") },
],
adopt,
context,
);
expect(await readLines(join(directory, `task-${taskId}.jsonl`))).toHaveLength(2);
@ -400,6 +407,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
{ type: "task", value: pendingTask(taskId, "first") },
{ type: "task", value: pendingTask(taskId, `second-${"x".repeat(512)}`) },
],
adopt,
context,
),
).rejects.toThrow("poisoned");
@ -432,6 +440,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
},
},
],
adopt,
context,
),
).rejects.toThrow();
@ -440,6 +449,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
expect(
await storage.commit(
[{ type: "entry", value: { id, conversationId: ROOT_CONVERSATION_ID, kind: "good" } }],
adopt,
context,
),
).toBe(2);
@ -477,6 +487,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { text: "β" } },
},
],
adopt,
context,
),
).rejects.toThrow("poisoned");
@ -521,6 +532,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
),
).rejects.toThrow("poisoned");
@ -552,6 +564,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
);
const expected = [
@ -571,6 +584,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { checkpoint: true } },
},
],
adopt,
context,
);
expect(env.operations).toEqual([
@ -595,14 +609,15 @@ describe("Pico JsonlStorage publication and recovery", () => {
},
},
],
adopt,
context,
);
expect(env.operations).toEqual(["append:main.jsonl"]);
const taskId = await storage.mintId<TaskId<JsonValue>>();
await storage.commit([{ type: "task", value: pendingTask(taskId) }], context);
await storage.commit([{ type: "task", value: pendingTask(taskId) }], adopt, context);
env.clear();
await storage.commit([{ type: "task", value: terminalTask(taskId) }], context);
await storage.commit([{ type: "task", value: terminalTask(taskId) }], adopt, context);
expect(env.operations).toEqual([
"append:main.jsonl",
...(fsync ? ["flush:main.jsonl"] : []),
@ -632,6 +647,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
await storage.commit(
@ -642,6 +658,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 1]] },
},
],
adopt,
context,
);
@ -655,6 +672,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
),
).resolves.toBe(4);
@ -685,10 +703,11 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 1 } },
},
],
adopt,
context,
);
env.fail({ operation: "remove", call: 1, mode });
await expect(storage.commit([{ type: "document.retire", id }], context)).resolves.toBe(3);
await expect(storage.commit([{ type: "document.retire", id }], adopt, context)).resolves.toBe(3);
expect(await storage.document(id, "current", context)).toBeUndefined();
await storage.close(context);
@ -715,6 +734,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
env.fail({ operation: "flush", call: 2, mode });
@ -727,6 +747,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
),
).resolves.toBe(3);
@ -769,6 +790,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
env.fail({ operation: "flush", call: 3, mode });
@ -781,6 +803,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
),
).resolves.toBe(3);
@ -793,6 +816,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 3]] },
},
],
adopt,
context,
);
await storage.close(context);
@ -810,9 +834,9 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory, env);
await createRoot(storage);
const id = await storage.mintId<TaskId<JsonValue>>();
await storage.commit([{ type: "task", value: pendingTask(id) }], context);
await storage.commit([{ type: "task", value: pendingTask(id) }], adopt, context);
env.fail({ operation: "remove", call: 1, mode });
await expect(storage.commit([{ type: "task", value: terminalTask(id) }], context)).resolves.toBe(3);
await expect(storage.commit([{ type: "task", value: terminalTask(id) }], adopt, context)).resolves.toBe(3);
expect(await storage.task(id, context)).toEqual(terminalTask(id));
await storage.close(context);
@ -836,6 +860,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
await storage.commit(
@ -846,6 +871,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 10 } },
},
],
adopt,
context,
);
await storage.commit(
@ -856,6 +882,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 11]] },
},
],
adopt,
context,
);
expect(await readLines(join(directory, `doc-${id}.jsonl`))).toHaveLength(2);
@ -877,6 +904,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
};
const createdAt = await storage.commit(
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { count: 0 } } }],
adopt,
context,
);
const changedAt = await storage.commit(
@ -887,6 +915,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 1]] },
},
],
adopt,
context,
);
await storage.commit(
@ -897,9 +926,10 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 2 } },
},
],
adopt,
context,
);
await storage.commit([{ type: "document.retire", id }], context);
await storage.commit([{ type: "document.retire", id }], adopt, context);
expect(await readLines(join(directory, `doc-${id}.jsonl`))).toHaveLength(3);
const reopened = await openStorage(directory);
@ -942,6 +972,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
);
const retiredAt = await storage.commit(
@ -951,6 +982,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
{ type: "document.retire", id: taskDocumentId },
{ type: "task", value: terminalTask(taskId) },
],
adopt,
context,
);
for (const file of [
@ -988,6 +1020,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { text: "kept" } },
},
],
adopt,
context,
);
const sidecarPath = join(directory, `doc-${documentId}.jsonl`);
@ -1004,6 +1037,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
expect(
await reopened.commit(
[{ type: "document.change", id: documentId, content: { kind: "delta", version: 1, ops: [] } }],
adopt,
context,
),
).toBe(3);
@ -1014,8 +1048,8 @@ describe("Pico JsonlStorage publication and recovery", () => {
const storage = await openStorage(directory);
await createRoot(storage);
const taskId = await storage.mintId<TaskId<JsonValue>>();
await storage.commit([{ type: "task", value: pendingTask(taskId) }], context);
await storage.commit([{ type: "task", value: terminalTask(taskId) }], context);
await storage.commit([{ type: "task", value: pendingTask(taskId) }], adopt, context);
await storage.commit([{ type: "task", value: terminalTask(taskId) }], adopt, context);
const sidecarPath = join(directory, `task-${taskId}.jsonl`);
expect(await fileExists(sidecarPath)).toBe(false);
await writeFile(
@ -1033,7 +1067,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
const reopened = await openStorage(directory);
expect(await fileExists(sidecarPath)).toBe(false);
expect(await reopened.task(taskId, context)).toEqual(terminalTask(taskId));
expect(await reopened.commit([], context)).toBe(4);
expect(await reopened.commit([], adopt, context)).toBe(4);
});
it("fails open when confirmed sidecar data is missing", async () => {
@ -1049,6 +1083,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
);
await writeFile(join(directory, `doc-${documentId}.jsonl`), "");
@ -1070,6 +1105,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
await storage.commit(
@ -1080,6 +1116,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], 1]] },
},
],
adopt,
context,
);
const path = join(directory, `doc-${documentId}.jsonl`);
@ -1122,6 +1159,7 @@ describe("Pico JsonlStorage publication and recovery", () => {
content: { kind: "base", version: 1, value: {} },
},
],
adopt,
context,
);
const path = join(directory, `doc-${documentId}.jsonl`);

2
packages/durable/test/memory-storage.test.ts
View file

@ -9,7 +9,7 @@ registerStorageConformance({ describe, expect, it }, "MemoryStorage", (use) => u
it("does not expose retained state through a prepared commit", async () => {
const storage = new MemoryStorage();
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], () => {}, BACKGROUND_CONTEXT);
const entryId = idFromNumber<EntryId>(2);
const prepared = storage.prepareCommit([
{

4
packages/durable/test/nested-images-bench.ts
View file

@ -189,12 +189,12 @@ async function runScenario(scenario: Scenario): Promise<Result> {
let commits = 0;
const written = noWrites();
const commit = storage.commit.bind(storage);
storage.commit = (writes, commitContext) => {
storage.commit = (writes, adopt, commitContext) => {
commits++;
// SQLite commits synchronously, which can starve the timer; sample here too.
sample();
for (const write of writes) written[write.type] += JSON.stringify(write).length;
return commit(writes, commitContext);
return commit(writes, adopt, commitContext);
};
const harness = await Harness.open(storage, { models, registry }, context);
const root = await harness.root(context, { agent: { model: { provider: "faux", modelId: "faux-1" } } });

4
packages/durable/test/nested-tools-bench.ts
View file

@ -133,10 +133,10 @@ async function runScenario(scenario: Scenario): Promise<{ perRun: Run[]; closedB
let commits = 0;
const written = noWrites();
const commit = storage.commit.bind(storage);
storage.commit = (writes, commitContext) => {
storage.commit = (writes, adopt, commitContext) => {
commits++;
for (const write of writes) written[write.type] += JSON.stringify(write).length;
return commit(writes, commitContext);
return commit(writes, adopt, commitContext);
};
const harness = await Harness.open(storage, { models, registry }, context);
const root = await harness.root(context, { agent: { model: { provider: "faux", modelId: "faux-1" } } });

4
packages/durable/test/scheduler-fanout-bench.ts
View file

@ -41,9 +41,9 @@ async function run(calls: number): Promise<{ ms: number; commits: number }> {
const storage = new MemoryStorage();
let commits = 0;
const commit = storage.commit.bind(storage);
storage.commit = (writes, commitContext) => {
storage.commit = (writes, adopt, commitContext) => {
commits++;
return commit(writes, commitContext);
return commit(writes, adopt, commitContext);
};
const harness = await Harness.open(storage, { models, registry }, context);
const root = await harness.root(context, { agent: { model: { provider: "faux", modelId: "faux-1" } } });

106
packages/durable/test/session-adoption.test.ts Normal file
View file

@ -0,0 +1,106 @@
import type { Context } from "@earendil-works/chord";
import { defineDoc, MemoryStorage, type Seq, type StorageWrite } from "@earendil-works/pi-durable";
import { afterEach, describe, expect, it, vi } from "vitest";
import { SessionImpl } from "../src/session/session.ts";
import { Transaction } from "../src/session/transaction.ts";
import { context } from "./session-support.ts";
const CounterDoc = defineDoc<{ count: number }>({
checkpointWhen: () => false,
kind: "adoption.counter",
version: 1,
scope: "session",
initial: () => ({ count: 0 }),
});
/** Memory storage whose next commit, once durable, misbehaves as `fault` says. */
class FaultyStorage extends MemoryStorage {
fault: "reject_after_adopt" | "skip_adopt" | undefined;
readonly order: string[] = [];
override async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, callContext: Context) {
const fault = this.fault;
this.fault = undefined;
if (fault === "skip_adopt") return super.commit(writes, () => {}, callContext);
const seq = await super.commit(
writes,
(committed) => {
this.order.push("adopt");
adopt(committed);
this.order.push("adopted");
},
callContext,
);
this.order.push("resolve");
if (fault === "reject_after_adopt") throw new Error("reclaim failed");
return seq;
}
}
function open() {
const storage = new FaultyStorage();
const session = new SessionImpl(storage);
return { storage, session };
}
const bump = (session: SessionImpl) =>
session.commit(async (tx) => {
(await tx.doc(CounterDoc)).count++;
}, context);
afterEach(() => {
vi.restoreAllMocks();
});
describe("Session adoption inside Storage.commit()", () => {
it("adopts synchronously inside the commit, before Storage resolves", async () => {
const { storage, session } = open();
const adopt = vi.spyOn(Transaction.prototype, "adopt");
await bump(session);
expect(storage.order).toEqual(["adopt", "adopted", "resolve"]);
expect(adopt).toHaveBeenCalledTimes(1);
expect(await session.snapshot(CounterDoc, context)).toEqual({ count: 1 });
await session.close(context);
});
it("fails the Session without discarding when Storage rejects after adoption", async () => {
const { storage, session } = open();
await bump(session);
const discard = vi.spyOn(Transaction.prototype, "discard");
storage.fault = "reject_after_adopt";
await expect(bump(session)).rejects.toThrow("reclaim failed");
expect(discard).not.toHaveBeenCalled();
expect(await session.closed).toMatchObject({ reason: "failed", error: new Error("reclaim failed") });
});
it("discards the staged changes when Storage rejects before adoption", async () => {
const storage = new MemoryStorage();
const session = new SessionImpl(storage);
await bump(session);
const discard = vi.spyOn(Transaction.prototype, "discard");
vi.spyOn(storage, "commit").mockRejectedValueOnce(new Error("disk full"));
await expect(bump(session)).rejects.toThrow("disk full");
expect(discard).toHaveBeenCalledTimes(1);
expect(await session.closed).toMatchObject({ reason: "failed" });
});
it("fails the Session with its own adoption error, which never reaches Storage", async () => {
const { storage, session } = open();
await bump(session);
const failure = new Error("adoption failed");
vi.spyOn(Transaction.prototype, "adopt").mockImplementationOnce(() => {
throw failure;
});
await expect(bump(session)).rejects.toBe(failure);
// Storage saw a normal adoption and resolved.
expect(storage.order.slice(-3)).toEqual(["adopt", "adopted", "resolve"]);
expect(await session.closed).toEqual({ reason: "failed", error: failure });
});
it("fails the Session when Storage resolves without adopting", async () => {
const { storage, session } = open();
storage.fault = "skip_adopt";
await expect(bump(session)).rejects.toThrow("without adopting");
expect(await session.closed).toMatchObject({ reason: "failed" });
});
});

3
packages/durable/test/session-checkpoints-migrations.test.ts
View file

@ -833,7 +833,6 @@ describe("Session tracker cache across definition versions", () => {
it("sends observers of an older shape a root replacement after a newer token writes", async () => {
const { session } = openTestSession();
await session.commit((tx) => tx.doc(V1Doc), context);
const state = (await session.documentState(V1Doc, context))!;
const watch = (await session.watchDoc(V1Doc, context))!;
const frames: (readonly unknown[])[] = [];
const delivered = new Promise<void>((resolve) => {
@ -846,10 +845,8 @@ describe("Session tracker cache across definition versions", () => {
(await tx.doc(V2Doc)).names.push("second");
}, context);
await delivered;
expect(state.value).toEqual({ names: ["first", "second"] });
expect(watch.value).toEqual({ names: ["first", "second"] });
expect(frames).toEqual([[["r", { names: ["first", "second"] }]]]);
state.dispose();
await watch.stop();
});

8
packages/durable/test/session-definitions.test.ts
View file

@ -2,7 +2,6 @@ import type { Draft } from "@earendil-works/chord";
import {
type ConversationId,
createSession,
type DocumentState,
type DocumentWatch,
defineDoc,
defineDocFamily,
@ -160,7 +159,6 @@ describe("document definitions", () => {
const Note = defineEntry<{ text: string }>("t.note");
const Marker = defineEntry("t.marker");
type Rewound = Promise<Readonly<State> | undefined>;
type StateOf = Promise<DocumentState<State> | undefined>;
type WatchOf = Promise<DocumentWatch<State> | undefined>;
const check = async (session: Session, tx: Tx, conversationId: ConversationId, taskId: TaskId, at: EntryId) => {
expectTypeOf(session.snapshotAsOf(RewindableDoc, conversationId, at, context)).toEqualTypeOf<Rewound>();
@ -174,12 +172,6 @@ describe("document definitions", () => {
// @ts-expect-error task documents keep no history
void session.snapshotAsOf(TaskDoc, taskId, at, context);
expectTypeOf(session.documentState(SessionDoc, context)).toEqualTypeOf<StateOf>();
expectTypeOf(session.documentState(LatestDoc, conversationId, context)).toEqualTypeOf<StateOf>();
expectTypeOf(session.documentState(TaskDoc, taskId, context)).toEqualTypeOf<StateOf>();
expectTypeOf(session.documentState(SessionFamily, "k", context)).toEqualTypeOf<StateOf>();
expectTypeOf(session.documentState(ConversationFamily, conversationId, "k", context)).toEqualTypeOf<StateOf>();
expectTypeOf(session.documentState(TaskFamily, taskId, "k", context)).toEqualTypeOf<StateOf>();
expectTypeOf(session.watchDoc(SessionDoc, context)).toEqualTypeOf<WatchOf>();
expectTypeOf(session.watchDoc(LatestDoc, conversationId, context)).toEqualTypeOf<WatchOf>();
expectTypeOf(session.watchDoc(TaskDoc, taskId, context)).toEqualTypeOf<WatchOf>();

39
packages/durable/test/session-document-retention.test.ts
View file

@ -10,6 +10,7 @@ import {
defineTool,
type HarnessSettings,
MemoryStorage,
type Seq,
type StorageWrite,
watchEvents,
} from "@earendil-works/pi-durable";
@ -146,12 +147,13 @@ describe("document cache retention", () => {
await session.close(context);
});
it("never evicts a document a watch or a document state observes, and evicts it once the last detaches", async () => {
it("never evicts a document a watch observes, and evicts it once the last watch detaches", async () => {
const { session, storage, clock, sizes } = open(1_000);
const conversationId = await createConversation(session);
await bump(session, conversationId);
const watch = (await session.watchDoc(CounterDoc, conversationId, context))!;
const state = (await session.documentState(CounterDoc, conversationId, context))!;
const kept = (await session.watchDoc(CounterDoc, conversationId, context))!;
kept.start(async () => {});
const seen: number[] = [];
watch.start(async (value) => {
seen.push(value?.count ?? -1);
@ -160,8 +162,7 @@ describe("document cache retention", () => {
await bump(session, conversationId);
await settleLine(session);
// Still cached: a read long after its retention passed loads nothing from Storage.
expect(state.value?.count).toBe(2);
await eventually(() => seen.includes(2));
await eventually(() => seen.includes(2) && kept.value?.count === 2);
const reads = storage.documentReadCount;
clock.now += 1_000_000;
await session.commit(async (tx) => {
@ -174,8 +175,8 @@ describe("document cache retention", () => {
clock.now += 1_000_000;
await bump(session, conversationId);
await settleLine(session);
expect(state.value?.count).toBe(3);
state.dispose();
await eventually(() => kept.value?.count === 3);
await kept.stop();
clock.now += 1_000_000;
await session.commit(async () => {}, context);
await session.commit(async (tx) => {
@ -215,9 +216,9 @@ describe("document cache retention", () => {
const conversationId = await createConversation(session);
const writes: StorageWrite[][] = [];
const commit = storage.commit.bind(storage);
storage.commit = (batch: readonly StorageWrite[], commitContext: Context) => {
storage.commit = (batch: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context) => {
writes.push([...batch]);
return commit(batch, commitContext);
return commit(batch, adopt, commitContext);
};
for (let n = 0; n < 5; n++) {
await bump(session, conversationId);
@ -353,16 +354,18 @@ describe("document cache retention", () => {
const { session, sizes } = open(0);
const conversationId = await createConversation(session);
await bump(session, conversationId);
const state = (await session.documentState(CounterDoc, conversationId, context))!;
const watch = (await session.watchDoc(CounterDoc, conversationId, context))!;
watch.start(async () => {});
await session.commit((tx) => tx.retireDoc(CounterDoc, conversationId), context);
expect(state.value).toBeNull();
expect(await watch.closed).toEqual({ reason: "retired" });
expect(watch.value).toBeNull();
await bump(session, conversationId, 5);
await eventually(async () => {
await settleLine(session);
return sizes().cached === 0;
});
expect(await session.snapshot(CounterDoc, conversationId, context)).toEqual({ count: 5 });
state.dispose();
await watch.stop();
await session.close(context);
});
@ -375,7 +378,7 @@ describe("document cache retention", () => {
return session.documentCacheSizes.cached === 0;
});
const gate = storage.holdFindDocument();
const attaching = session.documentState(CounterDoc, conversationId, context);
const attaching = session.watchDoc(CounterDoc, conversationId, context);
await gate.entered;
const closing = session.close(context);
gate.release();
@ -384,14 +387,14 @@ describe("document cache retention", () => {
expect(session.documentCacheSizes).toMatchObject({ cached: 0, idle: 0, attached: 0 });
});
it("evicts on its timer without a commit, and tolerates a detach called twice", async () => {
it("evicts on its timer without a commit, and tolerates a stop called twice", async () => {
const storage = new ControlledStorage();
const session = new SessionImpl(storage, Date.now, () => 20);
const conversationId = await createConversation(session);
await bump(session, conversationId);
const state = (await session.documentState(CounterDoc, conversationId, context))!;
state.dispose();
state.dispose();
const watch = (await session.watchDoc(CounterDoc, conversationId, context))!;
await watch.stop();
await watch.stop();
const until = Date.now() + 2_000;
while (session.documentCacheSizes.cached > 0 && Date.now() < until)
await new Promise((resolve) => setTimeout(resolve, 10));
@ -561,11 +564,11 @@ describe("document cache retention", () => {
});
const { session, sizes } = open(0);
await session.commit((tx) => tx.doc(V1).then(() => undefined), context);
const state = (await session.documentState(V1, context))!;
const watch = (await session.watchDoc(V1, context))!;
expect(await session.snapshot(V2, context)).toEqual({ count: 3, v2: true });
await settleLine(session);
expect(sizes().cached).toBe(1);
state.dispose();
await watch.stop();
await eventually(async () => {
await settleLine(session);
return sizes().cached === 0;

226
packages/durable/test/session-states.test.ts
View file

@ -1,226 +0,0 @@
import { defineDoc, defineDocFamily } from "@earendil-works/pi-durable";
import { describe, expect, it } from "vitest";
import { context, createConversation, documentChanges, flush, openTestSession } from "./session-support.ts";
type State = { value: number; retained: { label: string } };
const StateDoc = defineDoc<State>({
checkpointWhen: () => false,
kind: "state.state",
version: 1,
scope: "session",
initial: () => ({ value: 0, retained: { label: "stable" } }),
});
const FamilyDoc = defineDocFamily<{ value: number }, string>({
checkpointWhen: () => false,
kind: "state.family",
version: 1,
family: true,
scope: "session",
initial: (seed) => ({ value: seed.length }),
});
async function createState(): Promise<ReturnType<typeof openTestSession>> {
const harness = openTestSession();
await harness.session.commit((tx) => tx.doc(StateDoc).then(() => undefined), context);
await flush();
return harness;
}
describe("Session document states", () => {
it("never creates an absent document", async () => {
const { session, storage } = openTestSession();
const commits = storage.commits.length;
expect(await session.documentState(StateDoc, context)).toBeUndefined();
expect(await session.documentState(FamilyDoc, "missing", context)).toBeUndefined();
expect(storage.commits).toHaveLength(commits);
expect(storage.mintCount).toBe(0);
});
it("returns an immediately hydrated read-only state with contiguous Chord deliveries", async () => {
const { session } = await createState();
const baseline = (await session.snapshot(StateDoc, context))!;
const state = (await session.documentState(StateDoc, context))!;
const deliveries: Array<{ value: Readonly<State> | null; sequence: number }> = [];
state.subscribe((value, _context, delivery) => deliveries.push({ value, sequence: delivery.sequence }));
expect(state.value).toBe(baseline);
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 1;
}, context);
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 2;
}, context);
await flush();
expect(deliveries.map(({ sequence }) => sequence)).toEqual([0, 1, 2]);
expect(deliveries.map(({ value }) => value?.value)).toEqual([0, 1, 2]);
expect(state.value).toBe(await session.snapshot(StateDoc, context));
state.dispose();
});
it("creates independent disposable states for one incarnation", async () => {
const { session } = await createState();
const first = (await session.documentState(StateDoc, context))!;
const second = (await session.documentState(StateDoc, context))!;
expect(second).not.toBe(first);
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 1;
}, context);
await flush();
expect(first.value?.value).toBe(1);
expect(second.value?.value).toBe(1);
first.dispose();
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 2;
}, context);
await flush();
expect(first.value?.value).toBe(1);
expect(second.value?.value).toBe(2);
second.dispose();
});
it("shares exact committed value references with Chord", async () => {
const { session, publications } = await createState();
const state = (await session.documentState(StateDoc, context))!;
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 4;
}, context);
await flush();
const published = documentChanges(publications.at(-1)!)[0]!;
expect(state.value).toBe(published.value);
expect(state.value?.retained).toBe((await session.snapshot(StateDoc, context))!.retained);
state.dispose();
});
it("captures a late baseline without redelivering an already covered commit", async () => {
const { session } = await createState();
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 1;
}, context);
const state = (await session.documentState(StateDoc, context))!;
const deliveries: number[] = [];
state.subscribe((_value, _context, delivery) => deliveries.push(delivery.sequence));
await flush();
expect(state.value?.value).toBe(1);
expect(deliveries).toEqual([0]);
state.dispose();
});
it("publishes null retirement and never follows a replacement incarnation", async () => {
const { session } = await createState();
const oldState = (await session.documentState(StateDoc, context))!;
await session.commit(async (tx) => {
await tx.retireDoc(StateDoc);
(await tx.doc(StateDoc)).value = 10;
}, context);
await flush();
expect(oldState.value).toBeNull();
const replacement = (await session.documentState(StateDoc, context))!;
expect(replacement.value?.value).toBe(10);
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 11;
}, context);
await flush();
expect(oldState.value).toBeNull();
expect(replacement.value?.value).toBe(11);
oldState.dispose();
replacement.dispose();
});
it("cold-loads a definition-free fork copy", async () => {
const Copied = defineDoc<{ value: number }>({
checkpointWhen: () => false,
kind: "state.copied",
version: 1,
scope: "conversation",
history: "latest",
fork: "current",
initial: () => ({ value: 0 }),
});
const { session, storage } = openTestSession();
const parentId = await createConversation(session);
const entry = await session.commit(async (tx) => {
const created = await tx.appendEntry(parentId, { kind: "point" });
(await tx.doc(Copied, parentId)).value = 7;
return created;
}, context);
const childId = await session.commit(
async (tx) => (await tx.forkConversation(parentId, entry.id, { ownership: { kind: "ownerless" } })).id,
context,
);
const reads = storage.documentReadCount;
const state = (await session.documentState(Copied, childId, context))!;
expect(state.value).toEqual({ value: 7 });
expect(storage.documentReadCount).toBeGreaterThan(reads);
state.dispose();
});
it("hydrates a migrated tracker without writing and skips an equal version-base update", async () => {
const Old = defineDoc<{ value: number }>({
checkpointWhen: () => false,
kind: "state.migration",
version: 1,
scope: "session",
initial: () => ({ value: 3 }),
});
const Current = defineDoc<{ value: number; migrated: boolean }>({
checkpointWhen: () => false,
kind: "state.migration",
version: 2,
scope: "session",
initial: () => ({ value: 0, migrated: false }),
migrate: (value) => ({ value: value.value as number, migrated: true }),
});
const { session, storage } = openTestSession();
await session.commit((tx) => tx.doc(Old).then(() => undefined), context);
await session.unloadDocuments();
const commits = storage.commits.length;
const state = (await session.documentState(Current, context))!;
expect(state.value).toEqual({ value: 3, migrated: true });
expect(storage.commits).toHaveLength(commits);
const baseline = state.value;
await session.commit((tx) => tx.doc(Current).then(() => undefined), context);
await flush();
expect(storage.commits).toHaveLength(commits + 1);
expect(state.value).toBe(baseline);
await session.commit(async (tx) => {
(await tx.doc(Current)).value = 4;
}, context);
await flush();
expect(state.value).toEqual({ value: 4, migrated: true });
state.dispose();
});
it("continues from exact committed values after the tracker cache unloads", async () => {
const { session, storage } = await createState();
const state = (await session.documentState(StateDoc, context))!;
const baseline = state.value;
const reads = storage.documentReadCount;
await session.unloadDocuments();
const reloaded = await session.snapshot(StateDoc, context);
expect(reloaded).toEqual(baseline);
expect(reloaded).not.toBe(baseline);
expect(storage.documentReadCount).toBeGreaterThan(reads);
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 6;
}, context);
await flush();
expect(state.value?.value).toBe(6);
state.dispose();
});
it("exposes trusted shared immutable values without freezing", async () => {
const { session } = await createState();
const snapshot = (await session.snapshot(StateDoc, context))!;
const state = (await session.documentState(StateDoc, context))!;
expect(state.value).toBe(snapshot);
expect(Object.isFrozen(state.value)).toBe(false);
expect(Object.isFrozen(state.value!.retained)).toBe(false);
state.dispose();
});
});

8
packages/durable/test/session-support.ts
View file

@ -78,7 +78,11 @@ export class ControlledStorage extends MemoryStorage {
held.gate.resolve();
}
override async commit(writes: readonly StorageWrite[], commitContext: Context): Promise<Seq> {
override async commit(
writes: readonly StorageWrite[],
adopt: (seq: Seq) => void,
commitContext: Context,
): Promise<Seq> {
this.admittedCommits.push(writes);
this.commits.push(structuredClone(writes));
const held = this.#commitGate;
@ -91,7 +95,7 @@ export class ControlledStorage extends MemoryStorage {
this.#commitFailure = undefined;
throw failure;
}
return super.commit(writes, commitContext);
return super.commit(writes, adopt, commitContext);
}
override mintId<I extends Id<string>>(): Promise<I> {

84
packages/durable/test/session-watches.test.ts
View file

@ -1,4 +1,6 @@
import type { Context } from "@earendil-works/chord";
import { setFlagsFromString } from "node:v8";
import { runInNewContext } from "node:vm";
import type { Context, JsonValue } from "@earendil-works/chord";
import {
BACKGROUND_CONTEXT,
createContextKey,
@ -7,8 +9,10 @@ import {
withContextValue,
} from "@earendil-works/chord/context";
import { applyImmutable } from "@earendil-works/chord/delta";
import { defineDoc } from "@earendil-works/pi-durable";
import { type DocumentWatch, defineDoc, MemoryStorage } from "@earendil-works/pi-durable";
import { describe, expect, it } from "vitest";
import { CommittedWatch } from "../src/session/observation.ts";
import { SessionImpl } from "../src/session/session.ts";
import { context, documentChanges, flush, openTestSession } from "./session-support.ts";
type State = { value: number; items: string[]; retained: { label: string } };
@ -308,7 +312,7 @@ describe("Session document watches", () => {
const baseline = watch.value;
await session.commit((tx) => tx.retireDoc(StateDoc), context);
await session.close(context);
expect(await watch.closed).toEqual({ reason: "session_closed" });
expect(await watch.closed).toEqual({ reason: "ended" });
expect(watch.value).toBe(baseline);
expect(() => watch.start(async () => undefined)).toThrow("stopped");
});
@ -330,7 +334,7 @@ describe("Session document watches", () => {
await entered.promise;
await session.commit((tx) => tx.retireDoc(StateDoc), context);
await session.close(context);
expect(await watch.closed).toEqual({ reason: "session_closed" });
expect(await watch.closed).toEqual({ reason: "ended" });
release.resolve();
await flush();
expect(values).toEqual([1]);
@ -403,7 +407,7 @@ describe("Session document watches", () => {
await entered.promise;
await session.close(context);
expect(deliveryContext!.abortSignal).toBeUndefined();
expect(await watch.closed).toEqual({ reason: "session_closed" });
expect(await watch.closed).toEqual({ reason: "ended" });
release.resolve();
});
@ -503,4 +507,74 @@ describe("Session document watches", () => {
expect(watch.value?.value).toBe(7);
await watch.stop();
});
it("ends retired only on its source's retirement, never on a null value", async () => {
const watch = new CommittedWatch<JsonValue>(
1,
() => {},
() => {},
);
const frames: Array<{ value: JsonValue; ops: unknown }> = [];
watch.start(async (value, ops) => {
frames.push({ value, ops });
});
let end: unknown;
void watch.closed.then((settled) => {
end = settled;
});
watch.advance(null, [["r", null]], context);
watch.advance(2, [["r", 2]], context);
await flush();
expect(frames).toEqual([
{ value: null, ops: [["r", null]] },
{ value: 2, ops: [["r", 2]] },
]);
expect(end).toBeUndefined();
expect(await watch.stop()).toEqual({ reason: "stopped" });
});
it("retains neither its Session nor its listener once stopped, while still referenced", async () => {
setFlagsFromString("--expose-gc");
const gc = runInNewContext("gc") as () => void;
const collect = async (): Promise<void> => {
for (let round = 0; round < 5; round++) {
await new Promise((resolve) => setTimeout(resolve, 0));
gc();
}
};
/** A watch of a Session nothing else references; `stop` says whether to stop it before returning. */
const acquire = async (stop: boolean) => {
// No eviction timer, so nothing but the watch can keep the Session alive.
const session = new SessionImpl(
new MemoryStorage(),
Date.now,
() => Number.POSITIVE_INFINITY,
() => () => {},
);
await session.commit((tx) => tx.doc(StateDoc).then(() => undefined), context);
const watch: DocumentWatch<State> = (await session.watchDoc(StateDoc, context))!;
const listener = async (): Promise<void> => {};
watch.start(listener);
await session.commit(async (tx) => {
(await tx.doc(StateDoc)).value = 1;
}, context);
await flush();
if (stop) await watch.stop();
return { watch, session: new WeakRef(session), listener: new WeakRef(listener) };
};
// Control: a running watch keeps its Session, so the collection below is real.
const running = await acquire(false);
await collect();
expect(running.session.deref()).toBeDefined();
await running.watch.stop();
const stopped = await acquire(true);
await collect();
expect(stopped.session.deref()).toBeUndefined();
expect(stopped.listener.deref()).toBeUndefined();
// What it delivered last stays readable.
expect(stopped.watch.value?.value).toBe(1);
expect(await stopped.watch.closed).toEqual({ reason: "stopped" });
});
});

2
packages/durable/test/sqlite-cloudflare.test.ts
View file

@ -53,6 +53,7 @@ describe("Durable Object SQLite adapter", () => {
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
{ type: "entry", value: { id: entryId, conversationId: ROOT_CONVERSATION_ID, kind: "first" } },
],
() => {},
BACKGROUND_CONTEXT,
),
second.commit(
@ -60,6 +61,7 @@ describe("Durable Object SQLite adapter", () => {
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
{ type: "entry", value: { id: entryId, conversationId: ROOT_CONVERSATION_ID, kind: "second" } },
],
() => {},
BACKGROUND_CONTEXT,
),
]);

15
packages/durable/test/sqlite-facade.test.ts
View file

@ -13,6 +13,9 @@ import { SqliteStorage } from "../src/storage/sqlite/index.ts";
import { NodeSqliteDatabase, openNodeSqliteDatabase } from "../src/storage/sqlite/node.ts";
import { type DocumentId, type EntryId, ROOT_CONVERSATION_ID } from "../src/types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
type SettlementMode = "immediate" | "delay" | "reject";
class ControlledSettlementDatabase implements SqliteDatabase {
@ -94,12 +97,13 @@ describe("portable SQLite facade settlement", () => {
it("prepares each storage statement once per connection and reuses it across transactions", async () => {
const connection = new PrepareCountingDatabaseSync(":memory:");
const storage = await SqliteStorage.open(new NodeSqliteDatabase(connection));
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT);
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, BACKGROUND_CONTEXT);
await storage.commit(
Array.from({ length: 100 }, (_, index) => ({
type: "entry" as const,
value: { id: idFromNumber<EntryId>(index + 2), conversationId: ROOT_CONVERSATION_ID, kind: "cached" },
})),
adopt,
BACKGROUND_CONTEXT,
);
await storage.commit(
@ -109,12 +113,13 @@ describe("portable SQLite facade settlement", () => {
value: { id: idFromNumber<EntryId>(102), conversationId: ROOT_CONVERSATION_ID, kind: "cached-again" },
},
],
adopt,
BACKGROUND_CONTEXT,
);
expect((await storage.entry(idFromNumber<EntryId>(2), BACKGROUND_CONTEXT))?.entry.kind).toBe("cached");
expect((await storage.entry(idFromNumber<EntryId>(102), BACKGROUND_CONTEXT))?.entry.kind).toBe("cached-again");
await expect(
storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], BACKGROUND_CONTEXT),
storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, BACKGROUND_CONTEXT),
).rejects.toThrow("ID 1 already belongs to conversation");
expect((await storage.entry(idFromNumber<EntryId>(2), BACKGROUND_CONTEXT))?.entry.kind).toBe("cached");
expect(connection.repeatedPrepares()).toEqual([]);
@ -126,6 +131,7 @@ describe("portable SQLite facade settlement", () => {
const storage = await SqliteStorage.open(database);
const commit = storage.commit(
[{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }],
adopt,
BACKGROUND_CONTEXT,
);
const read = storage.conversation(ROOT_CONVERSATION_ID, BACKGROUND_CONTEXT);
@ -142,6 +148,7 @@ describe("portable SQLite facade settlement", () => {
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
{ type: "entry", value: { id: entryId, conversationId: ROOT_CONVERSATION_ID, kind: "probe" } },
],
adopt,
BACKGROUND_CONTEXT,
);
const scan = storage.scanEntries({ conversationId: ROOT_CONVERSATION_ID }, 10, undefined, BACKGROUND_CONTEXT);
@ -180,6 +187,7 @@ describe("portable SQLite facade settlement", () => {
value: { id: idFromNumber<EntryId>(100), conversationId: ROOT_CONVERSATION_ID, kind: "settled" },
},
],
adopt,
BACKGROUND_CONTEXT,
);
let committed = false;
@ -201,6 +209,7 @@ describe("portable SQLite facade settlement", () => {
value: { id: idFromNumber<EntryId>(200), conversationId: ROOT_CONVERSATION_ID, kind: "rejected" },
},
],
adopt,
BACKGROUND_CONTEXT,
);
expect(await storage.mintId<EntryId>()).toBe(102);
@ -224,12 +233,14 @@ describe("portable SQLite facade settlement", () => {
content: { kind: "base", version: 1, value: { value: 1 } },
},
],
adopt,
BACKGROUND_CONTEXT,
);
const read = storage.document(id, "current", BACKGROUND_CONTEXT);
for (let index = 0; index < yields; index++) await Promise.resolve();
const replace = storage.commit(
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { value: 2 } } }],
adopt,
BACKGROUND_CONTEXT,
);
const [stored] = await Promise.all([read, replace]);

1
packages/durable/test/sqlite-migrations.test.ts
View file

@ -128,6 +128,7 @@ describe("durable SQLite migrations", () => {
},
},
],
() => {},
BACKGROUND_CONTEXT,
);
await storage.close(BACKGROUND_CONTEXT);

47
packages/durable/test/sqlite-storage.test.ts
View file

@ -24,6 +24,9 @@ import type {
} from "../src/types.ts";
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
/** Storage-level commits here have no in-memory owner to adopt them. */
const adopt = (): void => {};
const context = BACKGROUND_CONTEXT;
const openStorages = new Set<SqliteStorage>();
const tempDirectories = new Set<string>();
@ -56,10 +59,14 @@ class ReopeningStorage implements Storage {
this.options = options;
}
async commit(writes: readonly StorageWrite[], commitContext: Parameters<Storage["commit"]>[1]): Promise<Seq> {
if (this.closed) return this.current.commit(writes, commitContext);
async commit(
writes: readonly StorageWrite[],
adopt: Parameters<Storage["commit"]>[1],
commitContext: Parameters<Storage["commit"]>[2],
): Promise<Seq> {
if (this.closed) return this.current.commit(writes, adopt, commitContext);
try {
return await this.current.commit(writes, commitContext);
return await this.current.commit(writes, adopt, commitContext);
} finally {
await this.current.close(context);
this.current = await openNodeSqliteStorage(this.path, this.options);
@ -133,7 +140,7 @@ function entry(id: EntryId, conversationId: ConversationId, data?: JsonValue): E
}
async function createRoot(storage: Storage): Promise<Seq> {
return storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], context);
return storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
}
function pendingTask(id: TaskId<JsonValue>): TaskRecord<JsonValue, JsonValue, JsonValue> {
@ -171,7 +178,9 @@ describe("Pico SqliteStorage", () => {
const { storage, path } = await createSqliteStorage();
expect(await createRoot(storage)).toBe(1);
const entryId = await storage.mintId<EntryId>();
expect(await storage.commit([{ type: "entry", value: entry(entryId, ROOT_CONVERSATION_ID) }], context)).toBe(2);
expect(
await storage.commit([{ type: "entry", value: entry(entryId, ROOT_CONVERSATION_ID) }], adopt, context),
).toBe(2);
await storage.close(context);
openStorages.delete(storage);
@ -185,6 +194,7 @@ describe("Pico SqliteStorage", () => {
expect(
await reopened.commit(
[{ type: "task", value: pendingTask(await reopened.mintId<TaskId<JsonValue>>()) }],
adopt,
context,
),
).toBe(3);
@ -255,6 +265,7 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { retained: true } },
},
],
adopt,
context,
);
const database = new DatabaseSync(path);
@ -280,6 +291,7 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { nested: { value: 1 }, rows: [] } },
},
],
adopt,
context,
);
await storage.commit(
@ -294,6 +306,7 @@ describe("Pico SqliteStorage", () => {
},
},
],
adopt,
context,
);
await storage.commit(
@ -312,6 +325,7 @@ describe("Pico SqliteStorage", () => {
},
},
],
adopt,
context,
);
await storage.commit(
@ -322,6 +336,7 @@ describe("Pico SqliteStorage", () => {
content: { kind: "delta", version: 1, ops: [["s", ["nested", "value"], 4]] },
},
],
adopt,
context,
);
@ -360,15 +375,16 @@ describe("Pico SqliteStorage", () => {
value: { ...pendingTask(transientTaskId), input: circular as unknown as JsonValue },
},
],
adopt,
context,
),
).rejects.toThrow("circular structure");
expect(await storage.entry(transientId, context)).toBeUndefined();
expect(await storage.task(transientTaskId, context)).toBeUndefined();
const committedId = await storage.mintId<EntryId>();
expect(await storage.commit([{ type: "entry", value: entry(committedId, ROOT_CONVERSATION_ID) }], context)).toBe(
2,
);
expect(
await storage.commit([{ type: "entry", value: entry(committedId, ROOT_CONVERSATION_ID) }], adopt, context),
).toBe(2);
const db = new DatabaseSync(path, { readOnly: true });
try {
@ -392,6 +408,7 @@ describe("Pico SqliteStorage", () => {
} satisfies DocumentCreate;
const createdAt = await storage.commit(
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { count: 0 } } }],
adopt,
context,
);
let ancientAt = createdAt;
@ -408,6 +425,7 @@ describe("Pico SqliteStorage", () => {
: { kind: "delta", version: 1, ops: [["s", ["count"], count]] },
},
],
adopt,
context,
);
if (count === 5) ancientAt = recentAt;
@ -490,11 +508,13 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
for (let count = 1; count <= 10; count++) {
await storage.commit(
[{ type: "document.change", id, content: { kind: "delta", version: 1, ops: [["s", ["count"], count]] } }],
adopt,
context,
);
}
@ -512,6 +532,7 @@ describe("Pico SqliteStorage", () => {
}
await storage.commit(
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { count: 11 } } }],
adopt,
context,
);
expect(revisionCount(path, id)).toBe(1);
@ -523,10 +544,11 @@ describe("Pico SqliteStorage", () => {
content: { kind: "delta", version: 1, ops: [["r", { count: 12 }]] },
},
],
adopt,
context,
);
expect(revisionCount(path, id)).toBe(2);
await storage.commit([{ type: "document.retire", id }], context);
await storage.commit([{ type: "document.retire", id }], adopt, context);
expect(revisionCount(path, id)).toBe(0);
});
@ -544,6 +566,7 @@ describe("Pico SqliteStorage", () => {
}),
},
],
adopt,
context,
);
}
@ -573,10 +596,12 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { text: large } },
},
],
adopt,
context,
);
await storage.commit(
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { text: "small" } } }],
adopt,
context,
);
const before = new DatabaseSync(path, { readOnly: true });
@ -591,6 +616,7 @@ describe("Pico SqliteStorage", () => {
expect(freeAfterDelete).toBeGreaterThan(0);
await storage.commit(
[{ type: "document.change", id, content: { kind: "base", version: 1, value: { text: large } } }],
adopt,
context,
);
const after = new DatabaseSync(path, { readOnly: true });
@ -618,6 +644,7 @@ describe("Pico SqliteStorage", () => {
}),
},
],
adopt,
context,
);
}
@ -636,6 +663,7 @@ describe("Pico SqliteStorage", () => {
content: { kind: "base", version: 1, value: { count: 0 } },
},
],
adopt,
context,
);
for (let count = 1; count <= 100; count++) {
@ -647,6 +675,7 @@ describe("Pico SqliteStorage", () => {
content: { kind: "delta", version: 1, ops: [["s", ["count"], count]] },
},
],
adopt,
context,
);
}