mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
feat(chord,durable): Chord v2 slice 6b, Durable and jobs on the store
StoreStorage (Durable's storage on Chord v2's store, @earendil-works/pi-durable/storage/store) passes Durable's storage conformance suite and runs Durable's harness tests (npm run test:store). The store gets owner names and claims, one id allocator, Durable's Storage semantics for refused steps, history none/latest/rewindable with reads at a point, archive revisions and applyBatch. self.follow follows Durable's synchronous observers as Chord sources. Persisted jobs: records and state in the owner's store, restarts as Durable's tasks, abort stored, deferred state. Chord's ./next export returns for the adapter surface.
This commit is contained in:
Mario Zechner
parent
b5fc6dc84f
commit
14b4c56e57
106 changed files with 8714 additions and 871 deletions
|
|
@ -100,6 +100,7 @@
|
|||
|
||||
### Added
|
||||
|
||||
- `StoreStorage` (`@earendil-works/pi-durable/storage/store`): Durable's `Storage` on Chord v2's store. `StoreStorage.open(store, owner, context)` keeps a Session's records as declared tables of the store under `owner` and its documents in the store's documents engine, takes its ids from the store's one allocator, and commits each batch as one write of the store's commit step, adopted in the batch of the host the store serves, so a Chord object and a Durable Session commit through one queue and one outbox. It passes the storage conformance suite; `close()` leaves the store open (the app owns it).
|
||||
- Deferred commits: `Session.commit()`, `Conversation.commit()`, `TaskRuntime.commit()`, and a tool's `api.commit()` take `{ persistence: { kind: "deferred", withinMs? } }`. A commit that changes only existing conversation documents with `history: "latest"` and `fork: "initial"` is then adopted in memory and shown to watches at once, and stored with the next commit that reads or writes the document, before the document leaves memory, at close, or within `withinMs` (default `settings.progress.persistWithinMs`, or `createSession(storage, { persistWithinMs })`); any other write makes it immediate. A crash loses what is held. See spec §4.1.
|
||||
- Synchronous observers for adapters, such as a Chord source over a Durable view: `Session.observeDoc(token, ...scope, observer, context)`, `Harness.observeConversation(id, observer, context)`, and `Harness.observeTaskGraph(observer, context)` attach an `Observer<T>` (`advance(change, context)`, `end(failure)`) atomically on the Session line, like a watch, and return an `Observation<T>` (`value`, `detach()`; `observeDoc` gives `undefined` for an absent document). `advance` gets every change unfolded as memory adopts it, for a stored commit inside Storage's `adopt` and for a deferred one before it is stored, so it must only record it: no awaits, no Session calls, no throws. A document observer gets one call per change at its address (two for a commit that retires and recreates it), the task graph also a call outside any commit when a registry change or a failed migration changes a `blocked` reason, and a change can arrive before the attaching promise resolves. The attaching context governs only the attachment; a closed Session or an invalid argument rejects the promise. One that throws is reported as `listener_error`, detached, and ended with the error. `end` comes once, at Session close or failure, or after a task document's retirement with its task. Document, view, and task graph watches are built on them (spec §9.1).
|
||||
- `ConversationTranscript` keeps what a client has of one conversation over any `ConversationView`: `merge(view)` returns the view with `items`, the client's whole transcript (entries, a `gap` where a later window did not continue it, and an `earlier` marker until the start is loaded), instead of the window; `loadEarlier(context)` and `fillGap(gap, context)` page entries through a loader such as `Conversation.entries()`, and `resume` is the window that continues it after a drop: `conversation.watch(context, transcript.resume)`. A merge costs O(items) and returns the same `items` array when it adds nothing; concurrent loads of the same page are shared ([#11327](https://github.com/earendil-works/pi/issues/11327)).
|
||||
|
|
|
|||
|
|
@ -956,6 +956,8 @@ node --conditions=source --experimental-strip-types test/examples/13-chat.ts
|
|||
| [32-tests-after-edits](test/examples/32-tests-after-edits.ts) | An `afterRun` hook that runs the tests after every run that edited files, a fix run when they fail, and input waiting for them |
|
||||
| [33-turn-limit-and-continue](test/examples/33-turn-limit-and-continue.ts) | A per-conversation turn limit, `runFailure`, and continuing without a user message |
|
||||
| [34-submit-in-commit](test/examples/34-submit-in-commit.ts) | Admitting a visitor's steer inside the host's own commit with `submit(tx, draft)` and `whenIdle: "reject"` |
|
||||
| [35-durable-on-the-store](test/examples/35-durable-on-the-store.ts) | A Harness on `StoreStorage` in a Chord host's store, next to a Chord object's own table: one commit queue, one id allocator, one export |
|
||||
| [36-durable-views](test/examples/36-durable-views.ts) | A Chord object following a Durable view with `self.follow`: its frames and a member reading the followed value |
|
||||
| [00](test/examples/00-conversation.ts)–[12](test/examples/12-recovery.ts) | The layers underneath: sessions, documents, forks, watches, the Harness, agent configuration, reload, extension state, tasks, recovery |
|
||||
|
||||
Examples that call OpenAI need `OPENAI_API_KEY`; most use the faux provider otherwise.
|
||||
|
|
|
|||
|
|
@ -84,6 +84,12 @@
|
|||
"import": "./dist/storage/jsonl/node.js",
|
||||
"default": "./dist/storage/jsonl/node.js"
|
||||
},
|
||||
"./storage/store": {
|
||||
"source": "./src/storage/store/index.ts",
|
||||
"types": "./dist/storage/store/index.d.ts",
|
||||
"import": "./dist/storage/store/index.js",
|
||||
"default": "./dist/storage/store/index.js"
|
||||
},
|
||||
"./storage/sqlite": {
|
||||
"source": "./src/storage/sqlite/index.ts",
|
||||
"types": "./dist/storage/sqlite/index.d.ts",
|
||||
|
|
@ -119,7 +125,8 @@
|
|||
"scripts": {
|
||||
"clean": "shx rm -rf dist",
|
||||
"build": "tsc -p tsconfig.build.json",
|
||||
"test": "vitest --run && PI_DURABLE_DOCUMENT_RETENTION_MS=0 vitest --run && npm run test:workers",
|
||||
"test": "vitest --run && PI_DURABLE_DOCUMENT_RETENTION_MS=0 vitest --run && npm run test:store && npm run test:workers",
|
||||
"test:store": "vitest --run --config vitest.store.config.ts",
|
||||
"test:workers": "vitest --run --config test-workers/vitest.config.ts",
|
||||
"bench:storage": "vitest bench --config vitest.benchmark.config.ts",
|
||||
"bench:storage:memory": "node --conditions=source --experimental-strip-types test/storage-memory.ts",
|
||||
|
|
|
|||
327
packages/durable/src/storage/store/commit.ts
Normal file
327
packages/durable/src/storage/store/commit.ts
Normal file
|
|
@ -0,0 +1,327 @@
|
|||
// One Durable commit as one composite write of the store's commit step (slice 6b §2.3): staged before the queue (rows
|
||||
// encoded, contents serialized, explicit ids raising the store's allocator, fork copies resolved from their sources),
|
||||
// then inside the step what `SqliteStorage` does inside its transaction, with the step's reader: Durable's checks of
|
||||
// ids and documents (its messages), the table puts, and the documents in two passes, existing ones first (their content,
|
||||
// then their retirement, which frees the addresses this commit's creations take), then the creations and copies.
|
||||
|
||||
import type { Context, JsonValue } from "@earendil-works/chord";
|
||||
import {
|
||||
type EncodedRow,
|
||||
type DocumentContent as EngineContent,
|
||||
type DocumentRecord as EngineRecord,
|
||||
findDocument,
|
||||
insertDocument,
|
||||
type OwnerTable,
|
||||
readDocument,
|
||||
retireRecord,
|
||||
type StagedWrite,
|
||||
type Step,
|
||||
type StoreWrite,
|
||||
writeContent,
|
||||
} from "@earendil-works/chord/next";
|
||||
import type {
|
||||
DocumentContent,
|
||||
DocumentCopySource,
|
||||
DocumentCreate,
|
||||
DocumentId,
|
||||
DocumentRecord,
|
||||
Id,
|
||||
JsonObject,
|
||||
StorageWrite,
|
||||
} from "../../types.ts";
|
||||
import { addressOf, durableRecord, engineRecord } from "./documents.ts";
|
||||
import { conversationRow, entryRow, submissionRow, taskRow } from "./tables.ts";
|
||||
|
||||
/** Durable's record types, as `record_ids` names them. */
|
||||
export type RecordType = "conversation" | "entry" | "task" | "submission" | "document";
|
||||
|
||||
/** The storage's tables, as a commit writes them. */
|
||||
export type CommitTables = {
|
||||
readonly recordIds: OwnerTable;
|
||||
readonly conversations: OwnerTable;
|
||||
readonly entries: OwnerTable;
|
||||
readonly entryHeads: OwnerTable;
|
||||
readonly tasks: OwnerTable;
|
||||
readonly submissions: OwnerTable;
|
||||
};
|
||||
|
||||
/** A table write of the commit: its record's id and type, its table and its row encoded at staging. */
|
||||
type RowWrite = {
|
||||
readonly id: number;
|
||||
readonly type: Exclude<RecordType, "document">;
|
||||
readonly table: OwnerTable;
|
||||
readonly row: EncodedRow;
|
||||
};
|
||||
|
||||
/** A base as the commit stores it: its version and value, and the value serialized. */
|
||||
type StagedBase = { readonly version: number; readonly value: JsonObject; readonly text: string };
|
||||
|
||||
/** What one commit does with one document, as `SqliteStorage`'s action map has it. */
|
||||
type DocumentAction = {
|
||||
create?: DocumentCreate;
|
||||
copy?: DocumentCopySource;
|
||||
/** A creation's base: the write's own, or a copy's once resolved. */
|
||||
base?: StagedBase;
|
||||
content?: { readonly content: DocumentContent; readonly text: string };
|
||||
retire: boolean;
|
||||
};
|
||||
|
||||
/** Reads a copy's source at its point: its stored document, `undefined` when it cannot be read there. */
|
||||
export type CopySource = (
|
||||
source: DocumentCopySource,
|
||||
) => Promise<{ readonly record: DocumentRecord; readonly version: number; readonly value: JsonObject } | undefined>;
|
||||
|
||||
/** A Durable commit staged for the store's step. */
|
||||
export class DurableCommit implements StagedWrite {
|
||||
readonly order = "write";
|
||||
readonly #owner: string;
|
||||
readonly #tables: CommitTables;
|
||||
readonly #context: Context;
|
||||
/** Every id a write claims, in write order (documents created or copied included), for the id checks. */
|
||||
readonly #claims: { readonly id: number; readonly type: RecordType }[] = [];
|
||||
readonly #rows: RowWrite[] = [];
|
||||
readonly #actions = new Map<DocumentId, DocumentAction>();
|
||||
|
||||
/**
|
||||
* Stages `writes` (rule 8: rows and contents encoded here, before the queue); `raise` makes room above each explicit
|
||||
* id. A malformed batch (two content commands for one document, a second retirement) throws Durable's message.
|
||||
*/
|
||||
constructor(
|
||||
writes: readonly StorageWrite[],
|
||||
owner: string,
|
||||
tables: CommitTables,
|
||||
raise: (id: number) => void,
|
||||
context: Context,
|
||||
) {
|
||||
this.#owner = owner;
|
||||
this.#tables = tables;
|
||||
this.#context = context;
|
||||
for (const write of writes) this.#stage(write, raise);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the copies before the step (S11): each source read at its point and checked against the copy's record,
|
||||
* entered as a creation carrying the source's value as its base.
|
||||
*/
|
||||
async resolveCopies(read: CopySource): Promise<void> {
|
||||
for (const [id, action] of this.#actions) {
|
||||
const source = action.copy;
|
||||
if (source === undefined) continue;
|
||||
if (this.#actions.has(source.id)) throw new Error(`Document copy ${id} source is changed in the copy batch`);
|
||||
try {
|
||||
const stored = await read(source);
|
||||
if (stored === undefined) throw new Error(`Fork source document ${source.id} cannot be read`);
|
||||
const create = action.create!;
|
||||
if (
|
||||
stored.record.scope.kind !== "conversation" ||
|
||||
create.scope.kind !== "conversation" ||
|
||||
stored.record.kind !== create.kind ||
|
||||
stored.record.key !== create.key ||
|
||||
stored.record.history !== create.history ||
|
||||
stored.record.fork !== create.fork
|
||||
) {
|
||||
throw new Error(`Fork source document ${source.id} does not match the copied record`);
|
||||
}
|
||||
action.base = { version: stored.version, value: stored.value, text: JSON.stringify(stored.value) };
|
||||
} catch (error) {
|
||||
throw new Error(`Document copy ${id} was rejected`, { cause: error });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async apply(step: Step): Promise<readonly StoreWrite[]> {
|
||||
const recorded: StoreWrite[] = [];
|
||||
const claimed = await this.#checkIds(step);
|
||||
const existing = await this.#checkDocuments(step);
|
||||
for (const write of this.#rows) {
|
||||
if (claimed.get(write.id) === undefined) recorded.push(await this.#claim(step, write.id, write.type));
|
||||
claimed.set(write.id, write.type);
|
||||
const stamp = write.table === this.#tables.entries ? "commit_seq" : undefined;
|
||||
recorded.push(await write.table.put(step, write.row, stamp));
|
||||
}
|
||||
// Existing documents first: a retirement frees its address before a creation of this commit takes it.
|
||||
for (const [id, record] of existing) recorded.push(...(await this.#change(step, record, this.#actions.get(id)!)));
|
||||
for (const [id, action] of this.#actions) {
|
||||
if (action.create === undefined) continue;
|
||||
recorded.push(await this.#claim(step, id, "document"));
|
||||
recorded.push(await this.#create(step, action.create, action.base!, action.retire));
|
||||
}
|
||||
return recorded;
|
||||
}
|
||||
|
||||
/** Stages one write: a table row encoded, a document command into its action. */
|
||||
#stage(write: StorageWrite, raise: (id: number) => void): void {
|
||||
const tables = this.#tables;
|
||||
switch (write.type) {
|
||||
case "conversation":
|
||||
this.#row(write.value.id, write.type, tables.conversations, conversationRow(write.value), raise);
|
||||
return;
|
||||
case "entry": {
|
||||
const { id, conversationId, head } = write.value;
|
||||
this.#row(id, write.type, tables.entries, entryRow(write.value), raise);
|
||||
// Its claim is the entry's: `apply` claims an id once.
|
||||
if (head !== undefined) {
|
||||
const row = tables.entryHeads.encode({ id, conversation_id: conversationId });
|
||||
this.#rows.push({ id, type: write.type, table: tables.entryHeads, row });
|
||||
}
|
||||
return;
|
||||
}
|
||||
case "task":
|
||||
this.#row(write.value.id, write.type, tables.tasks, taskRow(write.value), raise);
|
||||
return;
|
||||
case "submission":
|
||||
this.#row(write.value.id, write.type, tables.submissions, submissionRow(write.value), raise);
|
||||
return;
|
||||
case "document.create":
|
||||
case "document.copy": {
|
||||
const { id } = write.record;
|
||||
const action = this.#action(id);
|
||||
if (action.create !== undefined || action.content !== undefined || action.copy !== undefined) {
|
||||
throw new Error(`Document ${id} has more than one content command`);
|
||||
}
|
||||
action.create = write.record;
|
||||
if (write.type === "document.copy") action.copy = write.source;
|
||||
else action.base = { ...write.content, text: JSON.stringify(write.content.value) };
|
||||
this.#claims.push({ id, type: "document" });
|
||||
raise(id);
|
||||
return;
|
||||
}
|
||||
case "document.change": {
|
||||
const action = this.#action(write.id);
|
||||
// A creation's base is its content command (`SqliteStorage` counts it so too), in either order.
|
||||
if (action.create !== undefined || action.content !== undefined || action.copy !== undefined) {
|
||||
throw new Error(`Document ${write.id} has more than one content command`);
|
||||
}
|
||||
const { content } = write;
|
||||
action.content = { content, text: JSON.stringify(content.kind === "base" ? content.value : content.ops) };
|
||||
return;
|
||||
}
|
||||
case "document.retire": {
|
||||
const action = this.#action(write.id);
|
||||
if (action.retire) throw new Error(`Document ${write.id} is retired more than once`);
|
||||
action.retire = true;
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#row(id: Id<string>, type: RowWrite["type"], table: OwnerTable, row: object, raise: (id: number) => void): void {
|
||||
this.#claims.push({ id, type });
|
||||
this.#rows.push({ id, type, table, row: table.encode(row) });
|
||||
raise(id);
|
||||
}
|
||||
|
||||
#action(id: DocumentId): DocumentAction {
|
||||
let action = this.#actions.get(id);
|
||||
if (action === undefined) {
|
||||
action = { retire: false };
|
||||
this.#actions.set(id, action);
|
||||
}
|
||||
return action;
|
||||
}
|
||||
|
||||
/**
|
||||
* Durable's global id checks (`SqliteStorage.checkGlobalIds`): a conversation, entry or document id claimed by
|
||||
* anything, or written twice; a task or submission id claimed by another type. Returns the stored types.
|
||||
*/
|
||||
async #checkIds(step: Step): Promise<Map<number, RecordType>> {
|
||||
const stored = new Map<number, RecordType | undefined>();
|
||||
const claimed = new Map<number, RecordType>();
|
||||
for (const { id, type } of this.#claims) {
|
||||
let existing = stored.get(id);
|
||||
if (!stored.has(id)) {
|
||||
const row = await this.#tables.recordIds.get(step, [id], this.#context);
|
||||
existing = row?.record_type as RecordType | undefined;
|
||||
stored.set(id, existing);
|
||||
}
|
||||
const earlier = claimed.get(id);
|
||||
if (type === "conversation" || type === "entry" || type === "document") {
|
||||
if (existing !== undefined) throw new Error(`ID ${id} already belongs to ${existing}`);
|
||||
if (earlier !== undefined) throw new Error(`ID ${id} is written more than once`);
|
||||
} else {
|
||||
if (existing !== undefined && existing !== type) throw new Error(`ID ${id} already belongs to ${existing}`);
|
||||
if (earlier !== undefined && earlier !== type) throw new Error(`ID ${id} is written as two record types`);
|
||||
}
|
||||
claimed.set(id, type);
|
||||
}
|
||||
const types = new Map<number, RecordType>();
|
||||
for (const [id, type] of stored) if (type !== undefined) types.set(id, type);
|
||||
return types;
|
||||
}
|
||||
|
||||
/**
|
||||
* Durable's document checks (`SqliteStorage.checkDocumentActions`): a change or retirement of an unknown or retired
|
||||
* document, a delta on another version, more than one live incarnation per address after the commit. Returns the
|
||||
* existing records the commit changes or retires, in action order.
|
||||
*/
|
||||
async #checkDocuments(step: Step): Promise<Map<DocumentId, EngineRecord>> {
|
||||
const existing = new Map<DocumentId, EngineRecord>();
|
||||
for (const [id, action] of this.#actions) {
|
||||
if (action.create !== undefined) continue;
|
||||
const stored = await readDocument(step, id);
|
||||
const record = stored === undefined ? undefined : durableRecord(stored, this.#owner);
|
||||
if (record === undefined) throw new Error(`Unknown document: ${id}`);
|
||||
if (record.retiredAt !== undefined) throw new Error(`Document ${id} is retired`);
|
||||
const content = action.content?.content;
|
||||
if (content?.kind === "delta" && content.version !== stored!.version) {
|
||||
throw new Error(`Document ${id} version transition requires a base`);
|
||||
}
|
||||
existing.set(id, stored!);
|
||||
}
|
||||
/** Per address a live creation takes: its live incarnations after the commit so far. */
|
||||
const live = new Map<string, number>();
|
||||
for (const action of this.#actions.values()) {
|
||||
if (action.create === undefined || action.retire) continue;
|
||||
const address = addressOf(action.create, this.#owner);
|
||||
const key = JSON.stringify([address.namespace, address.kind, address.key]);
|
||||
let count = live.get(key);
|
||||
if (count === undefined) {
|
||||
// The incarnation live there now counts unless this commit retires it.
|
||||
const current = await findDocument(step, address);
|
||||
count = current === undefined || this.#actions.get(current.id as DocumentId)?.retire === true ? 0 : 1;
|
||||
}
|
||||
if (++count > 1) throw new Error("Document address already has a current incarnation");
|
||||
live.set(key, count);
|
||||
}
|
||||
return existing;
|
||||
}
|
||||
|
||||
/** Puts `id`'s `record_ids` row: its record type. */
|
||||
#claim(step: Step, id: number, type: RecordType): Promise<StoreWrite> {
|
||||
const table = this.#tables.recordIds;
|
||||
return table.put(step, table.encode({ id, record_type: type }));
|
||||
}
|
||||
|
||||
/** An existing document's content, then its retirement, as their writes. */
|
||||
async #change(step: Step, record: EngineRecord, action: DocumentAction): Promise<StoreWrite[]> {
|
||||
const writes: StoreWrite[] = [];
|
||||
let { revision } = record;
|
||||
if (action.content !== undefined) {
|
||||
const { content, text } = action.content;
|
||||
const stored = engineContent(content);
|
||||
await writeContent(step, record.id, record.history, revision, stored, text);
|
||||
writes.push({ kind: "document.change", id: record.id, revision, content: stored });
|
||||
revision++;
|
||||
}
|
||||
if (action.retire) {
|
||||
await retireRecord(step, record);
|
||||
writes.push({ kind: "document.retire", id: record.id, revision });
|
||||
}
|
||||
return writes;
|
||||
}
|
||||
|
||||
/** A creation or a resolved copy (retired at once for a create-plus-retire, an empty lifetime), as its write. */
|
||||
async #create(step: Step, create: DocumentCreate, base: StagedBase, retired: boolean): Promise<StoreWrite> {
|
||||
const record = engineRecord(create, this.#owner, base.version, step.seq, retired);
|
||||
await insertDocument(step, record, base.text);
|
||||
const value = base.value as JsonValue;
|
||||
return { kind: "document.create", record, base: { version: base.version, fingerprint: "", value } };
|
||||
}
|
||||
}
|
||||
|
||||
/** A Durable content as the engine stores it: its documents have no schema fingerprint. */
|
||||
function engineContent(content: DocumentContent): EngineContent {
|
||||
return content.kind === "base"
|
||||
? { kind: "base", version: content.version, fingerprint: "", value: content.value }
|
||||
: { kind: "delta", version: content.version, ops: content.ops };
|
||||
}
|
||||
143
packages/durable/src/storage/store/documents.ts
Normal file
143
packages/durable/src/storage/store/documents.ts
Normal file
|
|
@ -0,0 +1,143 @@
|
|||
// Durable's documents in the store's documents engine (slice 6b §1.5, R1): a Durable address as an engine address
|
||||
// under the storage's owner name (its scope as the namespace, its kind and key JSON-encoded as Durable's indexed
|
||||
// strings), a Durable record as an engine record and back (the Durable record in `meta`), and a document read at a
|
||||
// point.
|
||||
|
||||
import {
|
||||
type DocumentEntry,
|
||||
type DocumentHistory,
|
||||
type DocumentAddress as EngineAddress,
|
||||
type DocumentPoint as EnginePoint,
|
||||
type DocumentRecord as EngineRecord,
|
||||
loadContent,
|
||||
readDocumentEntry,
|
||||
} from "@earendil-works/chord/next";
|
||||
import type { SqliteExecutor } from "@earendil-works/chord/sqlite";
|
||||
import { seqFromNumber } from "../../ids.ts";
|
||||
import type {
|
||||
DocumentAddress,
|
||||
DocumentCreate,
|
||||
DocumentId,
|
||||
DocumentPoint,
|
||||
DocumentRecord,
|
||||
JsonObject,
|
||||
StoredDocument,
|
||||
} from "../../types.ts";
|
||||
import { encodeIndexed } from "./tables.ts";
|
||||
|
||||
/** The namespaces a storage claims under its owner: Durable's scopes (Chord's namespaces never contain a `:`). */
|
||||
export const SCOPE_NAMESPACES = ["session", "conversation:", "task:"] as const;
|
||||
|
||||
/** The engine namespace of a Durable scope: `session`, `conversation:<id>`, `task:<id>`. */
|
||||
export function namespaceOf(scope: DocumentRecord["scope"]): string {
|
||||
switch (scope.kind) {
|
||||
case "session":
|
||||
return "session";
|
||||
case "conversation":
|
||||
return `conversation:${scope.conversationId}`;
|
||||
case "task":
|
||||
return `task:${scope.taskId}`;
|
||||
}
|
||||
}
|
||||
|
||||
/** The engine address of a Durable address under `owner`. */
|
||||
export function addressOf(address: DocumentAddress | DocumentCreate, owner: string): EngineAddress {
|
||||
const at = { owner, namespace: namespaceOf(address.scope), kind: encodeIndexed(address.kind) };
|
||||
return address.key === undefined ? at : { ...at, key: encodeIndexed(address.key) };
|
||||
}
|
||||
|
||||
/** What a record of `scope` keeps: session and task documents and latest conversation ones their latest state. */
|
||||
export function historyOf(record: DocumentCreate | DocumentRecord): DocumentHistory {
|
||||
return record.scope.kind === "conversation" && record.history === "rewindable" ? "rewindable" : "latest";
|
||||
}
|
||||
|
||||
/**
|
||||
* The engine record of a Durable creation at `seq` (retired at once for an empty lifetime): the Durable record itself,
|
||||
* its creation seq included, in `meta`, so a read of a live record parses one JSON text from one column, as
|
||||
* `SqliteStorage`'s does (a retirement later sets only the engine's `retired_at`).
|
||||
*/
|
||||
export function engineRecord(
|
||||
create: DocumentCreate,
|
||||
owner: string,
|
||||
version: number,
|
||||
seq: number,
|
||||
retired: boolean,
|
||||
): EngineRecord {
|
||||
return {
|
||||
id: create.id,
|
||||
...addressOf(create, owner),
|
||||
history: historyOf(create),
|
||||
createdAt: seq,
|
||||
...(retired ? { retiredAt: seq } : {}),
|
||||
meta: JSON.stringify({ ...create, createdAt: seq }),
|
||||
revision: 1,
|
||||
version,
|
||||
fingerprint: "",
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The Durable record of an engine record of this storage's owner and scopes, `undefined` for anyone else's (a Chord
|
||||
* document, another owner's): its id is unknown to this storage.
|
||||
*/
|
||||
export function durableRecord(record: EngineRecord, owner: string): DocumentRecord | undefined {
|
||||
const { meta } = record;
|
||||
if (meta === undefined || !ownedHere(record, owner)) return undefined;
|
||||
return recordOfEntry({ meta, retired_at: record.retiredAt ?? null });
|
||||
}
|
||||
|
||||
/** The Durable record of an engine entry read at one of this storage's addresses or scopes (the record in `meta`). */
|
||||
export function recordOfEntry(entry: DocumentEntry): DocumentRecord {
|
||||
// Written by `engineRecord`, parsed fresh: a retirement since is set on it in place.
|
||||
const record = JSON.parse(entry.meta!) as { -readonly [K in keyof DocumentRecord]: DocumentRecord[K] };
|
||||
if (entry.retired_at !== undefined && entry.retired_at !== null) record.retiredAt = seqFromNumber(entry.retired_at);
|
||||
return record as DocumentRecord;
|
||||
}
|
||||
|
||||
/** Whether a record is this storage's: `owner`'s, in one of Durable's scopes (not a Chord document of the owner). */
|
||||
function ownedHere(record: { readonly owner: string; readonly namespace: string }, owner: string): boolean {
|
||||
const { namespace } = record;
|
||||
if (record.owner !== owner) return false;
|
||||
return namespace === "session" || namespace.startsWith("conversation:") || namespace.startsWith("task:");
|
||||
}
|
||||
|
||||
/** Whether a document is alive at `at` (the half-open lifetime). */
|
||||
export function aliveAt(record: DocumentRecord, at: DocumentPoint): boolean {
|
||||
if (at === "current") return record.retiredAt === undefined;
|
||||
return record.createdAt <= at && (record.retiredAt === undefined || at < record.retiredAt);
|
||||
}
|
||||
|
||||
/**
|
||||
* A document read at a point, as a tagged result: request errors are told after the guarded read (slice 6b §1.4),
|
||||
* never thrown inside it.
|
||||
*/
|
||||
export type DocumentRead =
|
||||
| { readonly kind: "found"; readonly document: StoredDocument | undefined }
|
||||
| { readonly kind: "current_only"; readonly id: DocumentId };
|
||||
|
||||
/**
|
||||
* Document `id` of `owner` at `at`: absent (unknown, someone else's, not alive at `at`), a record whose history does
|
||||
* not reach back to `at`, or its value from the base at or below `at` and the deltas up to it.
|
||||
*/
|
||||
export async function readAt(
|
||||
executor: Pick<SqliteExecutor, "get" | "all">,
|
||||
id: DocumentId,
|
||||
at: DocumentPoint,
|
||||
owner: string,
|
||||
): Promise<DocumentRead> {
|
||||
const entry = await readDocumentEntry(executor, id);
|
||||
if (entry === undefined || entry.meta === null || !ownedHere(entry, owner)) {
|
||||
return { kind: "found", document: undefined };
|
||||
}
|
||||
const record = recordOfEntry(entry);
|
||||
if (at !== "current" && entry.history !== "rewindable") return { kind: "current_only", id };
|
||||
if (!aliveAt(record, at)) return { kind: "found", document: undefined };
|
||||
const point: EnginePoint = at;
|
||||
const loaded = await loadContent(executor, id, point);
|
||||
// The engine stores what Durable's commits gave it: JSON objects.
|
||||
const value = loaded.value as JsonObject;
|
||||
return {
|
||||
kind: "found",
|
||||
document: { record, version: loaded.version, value, deltasSinceBase: loaded.deltasSinceBase },
|
||||
};
|
||||
}
|
||||
1
packages/durable/src/storage/store/index.ts
Normal file
1
packages/durable/src/storage/store/index.ts
Normal file
|
|
@ -0,0 +1 @@
|
|||
export { StoreStorage } from "./storage.ts";
|
||||
137
packages/durable/src/storage/store/queries.ts
Normal file
137
packages/durable/src/storage/store/queries.ts
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
// StoreStorage's reads as the store's queries (slice 6b D2, §2.4), compiled once from the store's query vocabulary
|
||||
// into SQL over the declared tables: no SQL is rendered per call, and a read selects only the record cell (by id, an
|
||||
// entry's commit seq too; each column returned costs) and parses one JSON text per record, as `SqliteStorage`'s do.
|
||||
|
||||
import type { Context } from "@earendil-works/chord";
|
||||
import type { Order, OwnerTable, WhereNode } from "@earendil-works/chord/next";
|
||||
import type { SqliteExecutor, SqliteValue } from "@earendil-works/chord/sqlite";
|
||||
import type { CommitTables } from "./commit.ts";
|
||||
|
||||
/**
|
||||
* A stored row as a query selects it: the record cell, its version (selected only once the column has versions past
|
||||
* 1), and an entry's commit seq.
|
||||
*/
|
||||
type RecordRow = { readonly record: string; readonly record$v?: number; readonly commit_seq?: number };
|
||||
|
||||
/** One condition of a read: a column, how it compares, its value; a call site keeps one order of them. */
|
||||
export type Condition = readonly [string, "eq" | "gt" | "lt" | "gte" | "lte", SqliteValue];
|
||||
|
||||
const ASCENDING: Order = { columns: ["id"], direction: "asc" };
|
||||
const DESCENDING: Order = { columns: ["id"], direction: "desc" };
|
||||
|
||||
/** The columns of a read of ids only. */
|
||||
const ID = ["id"];
|
||||
|
||||
/** A placeholder value: a query's shape binds its values at each call. */
|
||||
const ANY = 0;
|
||||
|
||||
/**
|
||||
* One table's compiled reads: by id (with the `byIdExtra` columns), and by shape (conditions in binding order, an
|
||||
* order; the record only), each shape compiled at its first use.
|
||||
*/
|
||||
export class TableQueries {
|
||||
readonly #table: OwnerTable;
|
||||
/** The record column's read of a cell at a version past 1; `undefined` while the column is at version 1. */
|
||||
readonly #versioned: ((text: string, version: number, context: Context) => unknown) | undefined;
|
||||
readonly #byId: string;
|
||||
readonly #shapes = new Map<string, string>();
|
||||
readonly #columns: readonly string[];
|
||||
|
||||
constructor(table: OwnerTable, byIdExtra: readonly string[] = []) {
|
||||
this.#table = table;
|
||||
const versioned = table.jsonVersion("record") > 1;
|
||||
this.#versioned = versioned ? table.reader("record") : undefined;
|
||||
this.#columns = versioned ? ["record", "record$v"] : ["record"];
|
||||
this.#byId = table.prepareGet([...this.#columns, ...byIdExtra]);
|
||||
}
|
||||
|
||||
/** The row with `id`, `undefined` when absent. */
|
||||
byId(executor: Pick<SqliteExecutor, "get">, id: number): Promise<RecordRow | undefined> {
|
||||
return executor.get<RecordRow>(this.#byId, id);
|
||||
}
|
||||
|
||||
/** Up to `limit` rows whose `conditions` hold, by id in `direction`. */
|
||||
select(
|
||||
executor: Pick<SqliteExecutor, "all">,
|
||||
conditions: readonly Condition[],
|
||||
direction: "ascending" | "descending",
|
||||
limit: number,
|
||||
): Promise<RecordRow[]> {
|
||||
const sql = this.#shape(conditions, direction, "many");
|
||||
return executor.all<RecordRow>(sql, ...conditions.map(([, , value]) => value), limit);
|
||||
}
|
||||
|
||||
/** The first row whose `conditions` hold, by id in `direction`; `undefined` when none does. */
|
||||
first(
|
||||
executor: Pick<SqliteExecutor, "get">,
|
||||
conditions: readonly Condition[],
|
||||
direction: "ascending" | "descending",
|
||||
): Promise<RecordRow | undefined> {
|
||||
const sql = this.#shape(conditions, direction, "first");
|
||||
return executor.get<RecordRow>(sql, ...conditions.map(([, , value]) => value));
|
||||
}
|
||||
|
||||
/** A row's record, read as its column's (version checked, migrated on access). */
|
||||
record<T>(row: RecordRow, context: Context): T {
|
||||
const versioned = this.#versioned;
|
||||
// The cell holds a record of this table the storage wrote; at version 1 it is that JSON (`OwnerTable.reader`).
|
||||
return (versioned === undefined ? JSON.parse(row.record) : versioned(row.record, row.record$v!, context)) as T;
|
||||
}
|
||||
|
||||
/** Appends the records of `rows` to `into`, in order, each read as `record` reads it (one check per call). */
|
||||
records<T>(rows: readonly RecordRow[], into: T[], context: Context): void {
|
||||
const versioned = this.#versioned;
|
||||
if (versioned === undefined) {
|
||||
for (const row of rows) into.push(JSON.parse(row.record) as T);
|
||||
return;
|
||||
}
|
||||
for (const row of rows) into.push(versioned(row.record, row.record$v!, context) as T);
|
||||
}
|
||||
|
||||
/** The compiled read of a shape: its conditions' columns and comparisons, the direction, one row or many. */
|
||||
#shape(conditions: readonly Condition[], direction: "ascending" | "descending", rows: "first" | "many"): string {
|
||||
let key = `${rows} ${direction}`;
|
||||
for (const [column, condition] of conditions) key += ` ${column} ${condition}`;
|
||||
let sql = this.#shapes.get(key);
|
||||
if (sql !== undefined) return sql;
|
||||
// One comparison per condition: the vocabulary's node of that name, its value a placeholder.
|
||||
const nodes = conditions.map(([column, condition]) => ({ [condition]: [column, ANY] }) as unknown as WhereNode);
|
||||
const where: WhereNode | undefined = nodes.length === 0 ? undefined : { and: nodes };
|
||||
const order = direction === "ascending" ? ASCENDING : DESCENDING;
|
||||
sql =
|
||||
rows === "first"
|
||||
? this.#table.prepareFirst(this.#columns, where, order)
|
||||
: this.#table.prepare(this.#columns, where, order);
|
||||
this.#shapes.set(key, sql);
|
||||
return sql;
|
||||
}
|
||||
}
|
||||
|
||||
/** The compiled reads of the storage's tables. */
|
||||
export type StorageQueries = {
|
||||
readonly conversations: TableQueries;
|
||||
readonly entries: TableQueries;
|
||||
/** The id of a conversation's newest entry with a head marker: any, or at or before an id (bound second). */
|
||||
readonly latestHead: { readonly any: string; readonly atOrBefore: string };
|
||||
readonly tasks: TableQueries;
|
||||
readonly submissions: TableQueries;
|
||||
};
|
||||
|
||||
/** The reads of `tables`. */
|
||||
export function storageQueries(tables: CommitTables): StorageQueries {
|
||||
const heads = tables.entryHeads;
|
||||
return {
|
||||
conversations: new TableQueries(tables.conversations),
|
||||
entries: new TableQueries(tables.entries, ["commit_seq"]),
|
||||
latestHead: {
|
||||
any: heads.prepareFirst(ID, { eq: ["conversation_id", ANY] }, DESCENDING),
|
||||
atOrBefore: heads.prepareFirst(
|
||||
ID,
|
||||
{ and: [{ eq: ["conversation_id", ANY] }, { lte: ["id", ANY] }] },
|
||||
DESCENDING,
|
||||
),
|
||||
},
|
||||
tasks: new TableQueries(tables.tasks),
|
||||
submissions: new TableQueries(tables.submissions),
|
||||
};
|
||||
}
|
||||
557
packages/durable/src/storage/store/storage.ts
Normal file
557
packages/durable/src/storage/store/storage.ts
Normal file
|
|
@ -0,0 +1,557 @@
|
|||
// Durable's `Storage` on Chord's store (slice 6b D1, D2, §2): opening under an owner name (claims, tables, the
|
||||
// allocator), commits as one composite write of the store's commit step, and the reads, each one guarded read of the
|
||||
// store (several queries at one state in a read transaction), its request errors in a tagged result told after it.
|
||||
|
||||
import type { Context, JsonValue } from "@earendil-works/chord";
|
||||
import {
|
||||
type ChordStore,
|
||||
type Claimant,
|
||||
findDocumentEntry,
|
||||
listDocumentEntries,
|
||||
type Store,
|
||||
StoreOwnerTables,
|
||||
} from "@earendil-works/chord/next";
|
||||
import type { SqliteExecutor } from "@earendil-works/chord/sqlite";
|
||||
import { StorageRequestError } from "../../errors.ts";
|
||||
import { idFromNumber, seqFromNumber } from "../../ids.ts";
|
||||
import type {
|
||||
ConversationId,
|
||||
ConversationQuery,
|
||||
ConversationRecord,
|
||||
Cursor,
|
||||
DocumentAddress,
|
||||
DocumentCopySource,
|
||||
DocumentId,
|
||||
DocumentPoint,
|
||||
DocumentQuery,
|
||||
DocumentRecord,
|
||||
EntryId,
|
||||
EntryQuery,
|
||||
EntryRecord,
|
||||
Id,
|
||||
Page,
|
||||
ScanOrder,
|
||||
Seq,
|
||||
Storage,
|
||||
StorageWrite,
|
||||
StoredDocument,
|
||||
SubmissionId,
|
||||
SubmissionQuery,
|
||||
SubmissionRecord,
|
||||
TaskId,
|
||||
TaskQuery,
|
||||
TaskRecord,
|
||||
} from "../../types.ts";
|
||||
import { nextCursor, type ScanStart, scanStart } from "../scan.ts";
|
||||
import { type CommitTables, DurableCommit } from "./commit.ts";
|
||||
import { addressOf, type DocumentRead, namespaceOf, readAt, recordOfEntry, SCOPE_NAMESPACES } from "./documents.ts";
|
||||
import { type Condition, type StorageQueries, storageQueries, type TableQueries } from "./queries.ts";
|
||||
import { Conversations, Entries, EntryHeads, encodeIndexed, RecordIds, Submissions, Tasks } from "./tables.ts";
|
||||
|
||||
type StoredTask = TaskRecord<JsonValue, JsonValue, JsonValue>;
|
||||
type StoredEntry = { readonly entry: EntryRecord; readonly commitSeq: Seq };
|
||||
|
||||
/**
|
||||
* A guarded read's result: its data, or its request error, which is thrown after the guarded read (slice 6b §1.4),
|
||||
* never inside it.
|
||||
*/
|
||||
type Read<T> = { readonly kind: "found"; readonly value: T } | { readonly kind: "refused"; readonly message: string };
|
||||
|
||||
/** The id condition of a table scan from `start`. */
|
||||
function scanFrom(start: ScanStart): Condition {
|
||||
return start.order === "ascending"
|
||||
? ["id", "gt", start.after ?? -1]
|
||||
: ["id", "lt", start.after ?? Number.MAX_SAFE_INTEGER];
|
||||
}
|
||||
|
||||
function page<T extends { readonly id: Id<string> }>(
|
||||
values: readonly T[],
|
||||
limit: number,
|
||||
order: ScanOrder,
|
||||
): Page<T, Cursor> {
|
||||
const items = values.slice(0, limit);
|
||||
if (values.length <= limit) return { items };
|
||||
return { items, next: nextCursor(items.at(-1)!.id, order) };
|
||||
}
|
||||
|
||||
function cursorId(cursor: Cursor | undefined): number | undefined {
|
||||
const after = cursor?.after;
|
||||
if (after === undefined) return undefined;
|
||||
if (typeof after !== "number" || !Number.isSafeInteger(after))
|
||||
throw new StorageRequestError("Invalid storage cursor");
|
||||
return after;
|
||||
}
|
||||
|
||||
/**
|
||||
* Durable's `Storage` on Chord's store (slice 6b D1, D2, §2): its records as declared tables under an owner name, its
|
||||
* documents in the store's documents engine, its ids from the store's one allocator, its commits as one composite
|
||||
* write of the store's commit step, adopted in the host's batch. Several may share a store under distinct owner names
|
||||
* (claims); the app owns the store, and `close` leaves it open.
|
||||
*/
|
||||
export class StoreStorage implements Storage {
|
||||
readonly #store: ChordStore;
|
||||
readonly #owner: string;
|
||||
readonly #claimant: Claimant;
|
||||
readonly #tables: CommitTables;
|
||||
readonly #queries: StorageQueries;
|
||||
#closed = false;
|
||||
#closing: Promise<void> | undefined;
|
||||
/** Calls underway, which `close` waits for before it releases the claims (`#drained` once they reach 0). */
|
||||
#underway = 0;
|
||||
#drained: (() => void) | undefined;
|
||||
|
||||
private constructor(store: ChordStore, owner: string, claimant: Claimant, tables: CommitTables) {
|
||||
this.#store = store;
|
||||
this.#owner = owner;
|
||||
this.#claimant = claimant;
|
||||
this.#tables = tables;
|
||||
this.#queries = storageQueries(tables);
|
||||
}
|
||||
|
||||
/**
|
||||
* Opens Durable's storage in `store` under `owner` (a lowercase `[a-z0-9@/._~-]` name): claims its tables and its
|
||||
* document namespaces (a second claim is refused `invalid/args`), creates or migrates its tables, and raises the
|
||||
* store's allocator above every id it holds (one index read). A failure releases what it claimed.
|
||||
*/
|
||||
static async open(store: Store, owner: string, context: Context): Promise<StoreStorage> {
|
||||
// Chord's own: `openStore` or `importStore` made it.
|
||||
const chord = store as ChordStore;
|
||||
const claimant: Claimant = { label: `StoreStorage(${owner})` };
|
||||
try {
|
||||
const declared = new StoreOwnerTables(chord, owner, claimant);
|
||||
const tables: CommitTables = {
|
||||
recordIds: declared.declare(RecordIds),
|
||||
conversations: declared.declare(Conversations),
|
||||
entries: declared.declare(Entries),
|
||||
entryHeads: declared.declare(EntryHeads),
|
||||
tasks: declared.declare(Tasks),
|
||||
submissions: declared.declare(Submissions),
|
||||
};
|
||||
for (const namespace of SCOPE_NAMESPACES) chord.claims.documents(owner, namespace, claimant);
|
||||
await declared.prepare(context);
|
||||
const highest = await chord.read(
|
||||
(executor) =>
|
||||
executor.get<{ readonly id: number | null }>(
|
||||
`SELECT max("id") AS id FROM "${tables.recordIds.sql.table}"`,
|
||||
),
|
||||
context,
|
||||
);
|
||||
if (highest?.id != null) chord.ids.raise(highest.id);
|
||||
return new StoreStorage(chord, owner, claimant, tables);
|
||||
} catch (error) {
|
||||
chord.claims.release(claimant);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, context: Context): Promise<Seq> {
|
||||
return this.#call(async () => {
|
||||
const store = this.#store;
|
||||
const commit = new DurableCommit(writes, this.#owner, this.#tables, (id) => store.ids.raise(id), context);
|
||||
await commit.resolveCopies((source) => this.#copySource(source, context));
|
||||
// An empty commit stages nothing: it takes no seq, adopted at `lastSeq`.
|
||||
const staged = writes.length === 0 ? [] : [commit];
|
||||
return seqFromNumber(await store.commitInBatch(staged, (seq) => adopt(seqFromNumber(seq)), context));
|
||||
});
|
||||
}
|
||||
|
||||
mintId<I extends Id<string>>(): Promise<I> {
|
||||
return this.#call(async () => idFromNumber<I>(this.#store.mintId()));
|
||||
}
|
||||
|
||||
conversation(id: ConversationId, context: Context): Promise<ConversationRecord | undefined> {
|
||||
return this.#read((executor) => this.#conversation(executor, id, context), context);
|
||||
}
|
||||
|
||||
scanConversations(
|
||||
query: ConversationQuery,
|
||||
limit: number,
|
||||
cursor: Cursor | undefined,
|
||||
context: Context,
|
||||
): Promise<Page<ConversationRecord, Cursor>> {
|
||||
return this.#reading(() => {
|
||||
const start = scanStart(query.order, cursor, "ascending");
|
||||
const conditions: Condition[] = [scanFrom(start)];
|
||||
if (query.ownerConversationId !== undefined) {
|
||||
conditions.push(["owner_conversation_id", "eq", query.ownerConversationId]);
|
||||
}
|
||||
if (query.ownerTaskId !== undefined) conditions.push(["owner_task_id", "eq", query.ownerTaskId]);
|
||||
return this.#scan<ConversationRecord>(this.#queries.conversations, conditions, start, limit, context);
|
||||
});
|
||||
}
|
||||
|
||||
entry(id: EntryId, context: Context): Promise<StoredEntry | undefined>;
|
||||
entry(conversationId: ConversationId, id: EntryId, context: Context): Promise<StoredEntry | undefined>;
|
||||
entry(
|
||||
idOrConversationId: EntryId | ConversationId,
|
||||
idOrContext: EntryId | Context,
|
||||
maybeContext?: Context,
|
||||
): Promise<StoredEntry | undefined> {
|
||||
if (maybeContext === undefined) {
|
||||
const context = idOrContext as Context;
|
||||
return this.#read((executor) => this.#entry(executor, idOrConversationId as EntryId, context), context);
|
||||
}
|
||||
const context = maybeContext;
|
||||
const id = typeof idOrContext === "number" ? idFromNumber<EntryId>(idOrContext) : undefined;
|
||||
return this.#reading(() => {
|
||||
if (id === undefined) throw new StorageRequestError("Storage.entry() requires an entry ID");
|
||||
const conversationId = idOrConversationId as ConversationId;
|
||||
return this.#store
|
||||
.read((executor) => this.#visibleEntry(executor, conversationId, id, context), context)
|
||||
.then(told);
|
||||
});
|
||||
}
|
||||
|
||||
findLatestHeadMarker(
|
||||
conversationId: ConversationId,
|
||||
atOrBeforeEntryId: EntryId | undefined,
|
||||
context: Context,
|
||||
): Promise<(EntryRecord & { readonly head: EntryId }) | undefined> {
|
||||
return this.#reading(() =>
|
||||
this.#store
|
||||
.read((executor) => this.#latestHeadMarker(executor, conversationId, atOrBeforeEntryId, context), context)
|
||||
.then(told),
|
||||
);
|
||||
}
|
||||
|
||||
scanEntries(
|
||||
query: EntryQuery,
|
||||
limit: number,
|
||||
cursor: Cursor | undefined,
|
||||
context: Context,
|
||||
): Promise<Page<EntryRecord, Cursor>> {
|
||||
return this.#reading(() => {
|
||||
const { order, after } = scanStart(query.order, cursor, "descending");
|
||||
const read =
|
||||
order === "ascending"
|
||||
? (executor: SqliteExecutor) => this.#entriesAscending(executor, query, limit, after, context)
|
||||
: (executor: SqliteExecutor) => this.#entriesDescending(executor, query, limit, after, context);
|
||||
return this.#store.read(read, context).then(told);
|
||||
});
|
||||
}
|
||||
|
||||
task(id: TaskId, context: Context): Promise<StoredTask | undefined> {
|
||||
const tasks = this.#queries.tasks;
|
||||
return this.#read(async (executor) => {
|
||||
const row = await tasks.byId(executor, id);
|
||||
return row === undefined ? undefined : tasks.record<StoredTask>(row, context);
|
||||
}, context);
|
||||
}
|
||||
|
||||
scanTasks(
|
||||
query: TaskQuery,
|
||||
limit: number,
|
||||
cursor: Cursor | undefined,
|
||||
context: Context,
|
||||
): Promise<Page<StoredTask, Cursor>> {
|
||||
return this.#reading(() => {
|
||||
const start = scanStart(query.order, cursor, "ascending");
|
||||
const conditions: Condition[] = [scanFrom(start)];
|
||||
if (query.conversationId !== undefined) conditions.push(["conversation_id", "eq", query.conversationId]);
|
||||
if (query.kind !== undefined) conditions.push(["kind", "eq", encodeIndexed(query.kind)]);
|
||||
if (query.status !== undefined) conditions.push(["status", "eq", query.status]);
|
||||
if (query.abortRequested !== undefined)
|
||||
conditions.push(["abort_requested", "eq", query.abortRequested ? 1 : 0]);
|
||||
if (query.background !== undefined) conditions.push(["background", "eq", query.background ? 1 : 0]);
|
||||
return this.#scan<StoredTask>(this.#queries.tasks, conditions, start, limit, context);
|
||||
});
|
||||
}
|
||||
|
||||
submission(id: SubmissionId, context: Context): Promise<SubmissionRecord | undefined> {
|
||||
const submissions = this.#queries.submissions;
|
||||
return this.#read(async (executor) => {
|
||||
const row = await submissions.byId(executor, id);
|
||||
return row === undefined ? undefined : submissions.record<SubmissionRecord>(row, context);
|
||||
}, context);
|
||||
}
|
||||
|
||||
scanSubmissions(
|
||||
query: SubmissionQuery,
|
||||
limit: number,
|
||||
cursor: Cursor | undefined,
|
||||
context: Context,
|
||||
): Promise<Page<SubmissionRecord, Cursor>> {
|
||||
return this.#reading(() => {
|
||||
const start = scanStart(query.order, cursor, "ascending");
|
||||
const conditions: Condition[] = [scanFrom(start)];
|
||||
if (query.conversationId !== undefined) conditions.push(["conversation_id", "eq", query.conversationId]);
|
||||
if (query.status !== undefined) conditions.push(["status", "eq", query.status]);
|
||||
return this.#scan<SubmissionRecord>(this.#queries.submissions, conditions, start, limit, context);
|
||||
});
|
||||
}
|
||||
|
||||
submissionByRequest(
|
||||
conversationId: ConversationId,
|
||||
requestId: string,
|
||||
context: Context,
|
||||
): Promise<SubmissionRecord | undefined> {
|
||||
const submissions = this.#queries.submissions;
|
||||
const conditions: Condition[] = [
|
||||
["conversation_id", "eq", conversationId],
|
||||
["request_id", "eq", encodeIndexed(requestId)],
|
||||
];
|
||||
return this.#read(async (executor) => {
|
||||
const row = await submissions.first(executor, conditions, "ascending");
|
||||
return row === undefined ? undefined : submissions.record<SubmissionRecord>(row, context);
|
||||
}, context);
|
||||
}
|
||||
|
||||
findDocument(address: DocumentAddress, at: DocumentPoint, context: Context): Promise<DocumentRecord | undefined> {
|
||||
return this.#read(async (executor) => {
|
||||
const entry = await findDocumentEntry(executor, addressOf(address, this.#owner), at);
|
||||
return entry === undefined ? undefined : recordOfEntry(entry);
|
||||
}, context);
|
||||
}
|
||||
|
||||
document(id: DocumentId, at: DocumentPoint, context: Context): Promise<StoredDocument | undefined> {
|
||||
// The record and its revisions must come from one committed state (S4).
|
||||
return this.#reading(() =>
|
||||
this.#store.readTransaction((executor) => readAt(executor, id, at, this.#owner), context).then(documentOf),
|
||||
);
|
||||
}
|
||||
|
||||
scanDocuments(
|
||||
query: DocumentQuery,
|
||||
limit: number,
|
||||
cursor: Cursor | undefined,
|
||||
context: Context,
|
||||
): Promise<Page<DocumentRecord, Cursor>> {
|
||||
return this.#reading(() => {
|
||||
const after = cursorId(cursor) ?? -1;
|
||||
const scope = {
|
||||
owner: this.#owner,
|
||||
namespace: namespaceOf(query.scope),
|
||||
...(query.kind === undefined ? {} : { kind: encodeIndexed(query.kind) }),
|
||||
};
|
||||
return this.#store
|
||||
.read((executor) => listDocumentEntries(executor, scope, query.at, after, limit + 1), context)
|
||||
.then((found) => page(found.map(recordOfEntry), limit, "ascending"));
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Refuses new calls, waits for its commits underway, then releases its claims; the store stays open (the app's),
|
||||
* and finishes the reads underway, which change nothing.
|
||||
*/
|
||||
close(_context: Context): Promise<void> {
|
||||
if (this.#closing === undefined) {
|
||||
this.#closed = true;
|
||||
this.#closing = (async () => {
|
||||
if (this.#underway > 0) {
|
||||
await new Promise<void>((resolve) => {
|
||||
this.#drained = resolve;
|
||||
});
|
||||
}
|
||||
this.#store.claims.release(this.#claimant);
|
||||
})();
|
||||
}
|
||||
return this.#closing;
|
||||
}
|
||||
|
||||
/** Runs one commit or mint of this storage: refused once it closed, counted until it settles (`close` waits). */
|
||||
async #call<T>(run: () => Promise<T>): Promise<T> {
|
||||
if (this.#closed) throw new Error("StoreStorage is closed");
|
||||
this.#underway++;
|
||||
try {
|
||||
return await run();
|
||||
} finally {
|
||||
if (--this.#underway === 0) this.#drained?.();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A call that is one guarded read of the store: refused once closed; not waited for by `close` (a read changes
|
||||
* nothing, and the store, which stays open, finishes it).
|
||||
*/
|
||||
#read<T>(read: (executor: SqliteExecutor) => Promise<T>, context: Context): Promise<T> {
|
||||
return this.#reading(() => this.#store.read(read, context));
|
||||
}
|
||||
|
||||
/**
|
||||
* A read call: `read` prepares its guarded read (a request error thrown then, before it) and returns it. Refused
|
||||
* once closed; one promise per call besides the guard's (no async wrapper).
|
||||
*/
|
||||
#reading<T>(read: () => Promise<T>): Promise<T> {
|
||||
try {
|
||||
if (this.#closed) throw new Error("StoreStorage is closed");
|
||||
return read();
|
||||
} catch (error) {
|
||||
return Promise.reject(error);
|
||||
}
|
||||
}
|
||||
|
||||
async #conversation(
|
||||
executor: SqliteExecutor,
|
||||
id: ConversationId,
|
||||
context: Context,
|
||||
): Promise<ConversationRecord | undefined> {
|
||||
const conversations = this.#queries.conversations;
|
||||
const row = await conversations.byId(executor, id);
|
||||
return row === undefined ? undefined : conversations.record<ConversationRecord>(row, context);
|
||||
}
|
||||
|
||||
#entry(executor: SqliteExecutor, id: EntryId, context: Context): Promise<StoredEntry | undefined> {
|
||||
const entries = this.#queries.entries;
|
||||
return entries.byId(executor, id).then((row) => {
|
||||
if (row === undefined) return undefined;
|
||||
return { entry: entries.record<EntryRecord>(row, context), commitSeq: seqFromNumber(row.commit_seq!) };
|
||||
});
|
||||
}
|
||||
|
||||
/** Entry `id` when visible through `conversationId`'s fork chain (`SqliteStorage.readEntry`). */
|
||||
async #visibleEntry(
|
||||
executor: SqliteExecutor,
|
||||
conversationId: ConversationId,
|
||||
id: EntryId,
|
||||
context: Context,
|
||||
): Promise<Read<StoredEntry | undefined>> {
|
||||
let conversation = await this.#conversation(executor, conversationId, context);
|
||||
if (conversation === undefined) return unknownConversation(conversationId);
|
||||
const stored = await this.#entry(executor, id, context);
|
||||
if (stored === undefined) return NOTHING_FOUND;
|
||||
let upper = Number.POSITIVE_INFINITY;
|
||||
while (conversation.id !== stored.entry.conversationId) {
|
||||
if (conversation.parent === undefined) return NOTHING_FOUND;
|
||||
upper = Math.min(upper, conversation.parent.at);
|
||||
conversation = (await this.#conversation(executor, conversation.parent.conversationId, context))!;
|
||||
}
|
||||
return stored.entry.id > upper ? NOTHING_FOUND : { kind: "found", value: stored };
|
||||
}
|
||||
|
||||
async #latestHeadMarker(
|
||||
executor: SqliteExecutor,
|
||||
conversationId: ConversationId,
|
||||
atOrBeforeEntryId: EntryId | undefined,
|
||||
context: Context,
|
||||
): Promise<Read<(EntryRecord & { readonly head: EntryId }) | undefined>> {
|
||||
let conversation = await this.#conversation(executor, conversationId, context);
|
||||
if (conversation === undefined) return unknownConversation(conversationId);
|
||||
let upper: number | undefined = atOrBeforeEntryId;
|
||||
while (true) {
|
||||
const { latestHead } = this.#queries;
|
||||
const head =
|
||||
upper === undefined
|
||||
? await executor.get<{ readonly id: number }>(latestHead.any, conversation.id)
|
||||
: await executor.get<{ readonly id: number }>(latestHead.atOrBefore, conversation.id, upper);
|
||||
if (head !== undefined) {
|
||||
// Written with its entry, in the same commit.
|
||||
const row = (await this.#queries.entries.byId(executor, head.id))!;
|
||||
const value = this.#queries.entries.record<EntryRecord & { readonly head: EntryId }>(row, context);
|
||||
return { kind: "found", value };
|
||||
}
|
||||
if (conversation.parent === undefined) return NOTHING_FOUND;
|
||||
upper = upper === undefined ? conversation.parent.at : Math.min(upper, conversation.parent.at);
|
||||
conversation = (await this.#conversation(executor, conversation.parent.conversationId, context))!;
|
||||
}
|
||||
}
|
||||
|
||||
/** Newest first: each conversation of the fork chain up to its fork point (`SqliteStorage.readEntries`). */
|
||||
async #entriesDescending(
|
||||
executor: SqliteExecutor,
|
||||
query: EntryQuery,
|
||||
limit: number,
|
||||
after: number | undefined,
|
||||
context: Context,
|
||||
): Promise<Read<Page<EntryRecord, Cursor>>> {
|
||||
let conversation = await this.#conversation(executor, query.conversationId, context);
|
||||
if (conversation === undefined) return unknownConversation(query.conversationId);
|
||||
let upper: number | undefined = query.maxEntryId;
|
||||
if (after !== undefined) upper = Math.min(upper ?? Number.MAX_SAFE_INTEGER, after - 1);
|
||||
const values: EntryRecord[] = [];
|
||||
while (true) {
|
||||
const range = this.#range(conversation.id, query.minEntryId, upper);
|
||||
const rows = await this.#queries.entries.select(executor, range, "descending", limit + 1 - values.length);
|
||||
this.#queries.entries.records(rows, values, context);
|
||||
if (values.length > limit || conversation.parent === undefined) break;
|
||||
upper = upper === undefined ? conversation.parent.at : Math.min(upper, conversation.parent.at);
|
||||
if (query.minEntryId !== undefined && upper < query.minEntryId) break;
|
||||
conversation = (await this.#conversation(executor, conversation.parent.conversationId, context))!;
|
||||
}
|
||||
return { kind: "found", value: page(values, limit, "descending") };
|
||||
}
|
||||
|
||||
/** Oldest first: the fork chain's segments from the root forward, each up to its fork point. */
|
||||
async #entriesAscending(
|
||||
executor: SqliteExecutor,
|
||||
query: EntryQuery,
|
||||
limit: number,
|
||||
after: number | undefined,
|
||||
context: Context,
|
||||
): Promise<Read<Page<EntryRecord, Cursor>>> {
|
||||
const segments: { readonly conversationId: ConversationId; readonly upper: number | undefined }[] = [];
|
||||
let conversation = await this.#conversation(executor, query.conversationId, context);
|
||||
if (conversation === undefined) return unknownConversation(query.conversationId);
|
||||
let upper: number | undefined = query.maxEntryId;
|
||||
while (true) {
|
||||
segments.push({ conversationId: conversation.id, upper });
|
||||
if (conversation.parent === undefined) break;
|
||||
upper = upper === undefined ? conversation.parent.at : Math.min(upper, conversation.parent.at);
|
||||
if (query.minEntryId !== undefined && upper < query.minEntryId) break;
|
||||
conversation = (await this.#conversation(executor, conversation.parent.conversationId, context))!;
|
||||
}
|
||||
let lower: number | undefined = query.minEntryId;
|
||||
if (after !== undefined) lower = Math.max(lower ?? after + 1, after + 1);
|
||||
const values: EntryRecord[] = [];
|
||||
for (const segment of segments.reverse()) {
|
||||
const range = this.#range(segment.conversationId, lower, segment.upper);
|
||||
const rows = await this.#queries.entries.select(executor, range, "ascending", limit + 1 - values.length);
|
||||
this.#queries.entries.records(rows, values, context);
|
||||
if (values.length > limit) break;
|
||||
}
|
||||
return { kind: "found", value: page(values, limit, "ascending") };
|
||||
}
|
||||
|
||||
/** One conversation's entries with ids from `lower` to `upper` (inclusive, each optional). */
|
||||
#range(conversationId: ConversationId, lower: number | undefined, upper: number | undefined): Condition[] {
|
||||
const conditions: Condition[] = [["conversation_id", "eq", conversationId]];
|
||||
if (lower !== undefined) conditions.push(["id", "gte", lower]);
|
||||
if (upper !== undefined) conditions.push(["id", "lte", upper]);
|
||||
return conditions;
|
||||
}
|
||||
|
||||
/** A page of a table scan: up to `limit` records of `queries`' table whose `conditions` hold, from `start`. */
|
||||
#scan<T extends { readonly id: Id<string> }>(
|
||||
queries: TableQueries,
|
||||
conditions: readonly Condition[],
|
||||
start: ScanStart,
|
||||
limit: number,
|
||||
context: Context,
|
||||
): Promise<Page<T, Cursor>> {
|
||||
return this.#store
|
||||
.read((executor) => queries.select(executor, conditions, start.order, limit + 1), context)
|
||||
.then((rows) => {
|
||||
const values: T[] = [];
|
||||
queries.records(rows, values, context);
|
||||
return page(values, limit, start.order);
|
||||
});
|
||||
}
|
||||
|
||||
/** A copy's source read at its point, before the step (S11); a request error is thrown here, outside the read. */
|
||||
async #copySource(source: DocumentCopySource, context: Context) {
|
||||
const read = await this.#store.readTransaction(
|
||||
(executor) => readAt(executor, source.id, source.at, this.#owner),
|
||||
context,
|
||||
);
|
||||
return documentOf(read);
|
||||
}
|
||||
}
|
||||
|
||||
/** A read that found nothing. */
|
||||
const NOTHING_FOUND: Read<undefined> = { kind: "found", value: undefined };
|
||||
|
||||
function unknownConversation(id: ConversationId): Read<never> {
|
||||
return { kind: "refused", message: `Unknown conversation: ${id}` };
|
||||
}
|
||||
|
||||
/** A read's data, its request error thrown now that the guarded read returned (slice 6b §1.4). */
|
||||
function told<T>(read: Read<T>): T {
|
||||
if (read.kind === "refused") throw new StorageRequestError(read.message);
|
||||
return read.value;
|
||||
}
|
||||
|
||||
/** A document read's result, a record that keeps no history at the point thrown as Durable's request error. */
|
||||
function documentOf(read: DocumentRead): StoredDocument | undefined {
|
||||
if (read.kind === "current_only")
|
||||
throw new StorageRequestError(`Document ${read.id} does not retain historical content`);
|
||||
return read.document;
|
||||
}
|
||||
122
packages/durable/src/storage/store/tables.ts
Normal file
122
packages/durable/src/storage/store/tables.ts
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
// Durable's records as declared tables of the store (slice 6b D2, §2.2): every row carries its record's indexed columns
|
||||
// and the whole record as one `json(...)` cell. Indexed strings that must stay lossless (`kind`, `request_id`) hold
|
||||
// Durable's JSON encoding; index columns are never `opt`, so absent values take sentinels no encoded value can be
|
||||
// (`0` for an absent owner, which is never an id; `""` for an absent request id: an encoded string starts with `"`).
|
||||
|
||||
import type { JsonValue } from "@earendil-works/chord";
|
||||
import { defineTable, int, json, str, unknown } from "@earendil-works/chord/next";
|
||||
import type { ConversationRecord, EntryRecord, SubmissionRecord, TaskRecord } from "../../types.ts";
|
||||
|
||||
/** A record as one `json(...)` cell. */
|
||||
const record = () => json(unknown(), { version: 1 });
|
||||
|
||||
/** Which record type owns an id: Durable's one namespace across its five record types. */
|
||||
export const RecordIds = defineTable("record_ids", {
|
||||
version: 1,
|
||||
columns: { id: int(), record_type: str() },
|
||||
key: ["id"],
|
||||
});
|
||||
|
||||
export const Conversations = defineTable("conversations", {
|
||||
version: 1,
|
||||
columns: { id: int(), owner_conversation_id: int(), owner_task_id: int(), record: record() },
|
||||
key: ["id"],
|
||||
indexes: { by_owner_conversation: ["owner_conversation_id"], by_owner_task: ["owner_task_id"] },
|
||||
});
|
||||
|
||||
export const Entries = defineTable("entries", {
|
||||
version: 1,
|
||||
columns: { id: int(), conversation_id: int(), commit_seq: int(), record: record() },
|
||||
key: ["id"],
|
||||
indexes: { by_conversation: ["conversation_id"] },
|
||||
});
|
||||
|
||||
/**
|
||||
* The entries with a head marker, by conversation: what `SqliteStorage`'s partial index on `head IS NOT NULL` is,
|
||||
* as rows written only with such an entry, so a commit of an entry without one updates no second index.
|
||||
*/
|
||||
export const EntryHeads = defineTable("entry_heads", {
|
||||
version: 1,
|
||||
columns: { id: int(), conversation_id: int() },
|
||||
key: ["id"],
|
||||
indexes: { by_conversation: ["conversation_id"] },
|
||||
});
|
||||
|
||||
export const Tasks = defineTable("tasks", {
|
||||
version: 1,
|
||||
columns: {
|
||||
id: int(),
|
||||
conversation_id: int(),
|
||||
kind: str(),
|
||||
status: str(),
|
||||
abort_requested: int(),
|
||||
background: int(),
|
||||
record: record(),
|
||||
},
|
||||
key: ["id"],
|
||||
indexes: {
|
||||
by_status: ["status"],
|
||||
by_conversation: ["conversation_id"],
|
||||
by_kind: ["kind"],
|
||||
by_abort_requested: ["abort_requested"],
|
||||
by_background: ["background"],
|
||||
},
|
||||
});
|
||||
|
||||
export const Submissions = defineTable("submissions", {
|
||||
version: 1,
|
||||
columns: { id: int(), conversation_id: int(), request_id: str(), status: str(), record: record() },
|
||||
key: ["id"],
|
||||
indexes: {
|
||||
by_request: ["conversation_id", "request_id"],
|
||||
by_conversation: ["conversation_id"],
|
||||
by_status: ["status"],
|
||||
},
|
||||
});
|
||||
|
||||
/** Durable's JSON encoding of an indexed string: lossless whatever the string (lone surrogates, NUL). */
|
||||
export const encodeIndexed = (value: string): string => JSON.stringify(value);
|
||||
|
||||
/** The row of a conversation record. */
|
||||
export function conversationRow(value: ConversationRecord): object {
|
||||
return {
|
||||
id: value.id,
|
||||
owner_conversation_id: value.owner?.conversationId ?? 0,
|
||||
owner_task_id: value.owner?.taskId ?? 0,
|
||||
record: value,
|
||||
};
|
||||
}
|
||||
|
||||
/** The row of an entry record; `commit_seq` is stamped by the step. */
|
||||
export function entryRow(value: EntryRecord): object {
|
||||
return {
|
||||
id: value.id,
|
||||
conversation_id: value.conversationId,
|
||||
commit_seq: 0,
|
||||
record: value,
|
||||
};
|
||||
}
|
||||
|
||||
/** The row of a task record. */
|
||||
export function taskRow(value: TaskRecord<JsonValue, JsonValue, JsonValue>): object {
|
||||
return {
|
||||
id: value.id,
|
||||
conversation_id: value.conversationId,
|
||||
kind: encodeIndexed(value.kind),
|
||||
status: value.state.status,
|
||||
abort_requested: value.abortRequested ? 1 : 0,
|
||||
background: value.background ? 1 : 0,
|
||||
record: value,
|
||||
};
|
||||
}
|
||||
|
||||
/** The row of a submission record. */
|
||||
export function submissionRow(value: SubmissionRecord): object {
|
||||
return {
|
||||
id: value.id,
|
||||
conversation_id: value.conversationId,
|
||||
request_id: value.requestId === undefined ? "" : encodeIndexed(value.requestId),
|
||||
status: value.status,
|
||||
record: value,
|
||||
};
|
||||
}
|
||||
|
|
@ -1646,6 +1646,30 @@ export function createStorageConformance(options: StorageConformanceOptions): re
|
|||
},
|
||||
),
|
||||
|
||||
createCase(
|
||||
options,
|
||||
"refuses a creation and a change of one document in one commit, in either order",
|
||||
async (storage) => {
|
||||
await createRoot(storage);
|
||||
for (const order of ["create first", "change first"] as const) {
|
||||
const id = await storage.mintId<DocumentId>();
|
||||
const create: StorageWrite = {
|
||||
type: "document.create",
|
||||
record: { id, kind: "twice", key: order, scope: { kind: "session" } },
|
||||
content: { kind: "base", version: 1, value: { x: 1 } },
|
||||
};
|
||||
const change: StorageWrite = {
|
||||
type: "document.change",
|
||||
id,
|
||||
content: { kind: "delta", version: 1, ops: [["s", ["x"], 2]] },
|
||||
};
|
||||
const writes = order === "create first" ? [create, change] : [change, create];
|
||||
await expect(storage.commit(writes, adopt, context)).rejects.toThrow("more than one content command");
|
||||
expect(await storage.document(id, "current", context)).toBeUndefined();
|
||||
}
|
||||
},
|
||||
),
|
||||
|
||||
createCase(
|
||||
options,
|
||||
"rolls back record tables and secondary indexes when a document command fails",
|
||||
|
|
|
|||
|
|
@ -3,7 +3,8 @@
|
|||
*
|
||||
* node --conditions=source --experimental-strip-types --expose-gc test/conversation-view-bench.ts
|
||||
*
|
||||
* Scenarios, each in its own child process so heap numbers are its own, on SQLite so history stays out of the heap:
|
||||
* Scenarios, each in its own child process so heap numbers are its own, on SQLite so history stays out of the heap
|
||||
* (`SqliteStorage`; with `PI_DURABLE_BENCH_STORAGE=store`, Durable's storage on Chord's store, `StoreStorage`):
|
||||
* - `start`: acquire a watch of a conversation of N entries (the mount is built every time; the watch is stopped).
|
||||
* - `append`: K started watches; commit entries one per commit, or in bursts of 50 per commit, and wait for delivery.
|
||||
* - `resume`: a watch with `from` 20 entries back, as a client that reconnects after a short drop.
|
||||
|
|
@ -22,6 +23,7 @@ import { BACKGROUND_CONTEXT as context } from "@earendil-works/chord/context";
|
|||
import { createModels } from "@earendil-works/pi-ai/models";
|
||||
import { type Conversation, createRegistry, type EntryId, Harness, LiveDoc, type Storage } from "../src/index.ts";
|
||||
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
|
||||
import { openStoreStorage } from "./store-storage-support.ts";
|
||||
|
||||
type Scenario =
|
||||
| { kind: "start"; entries: number }
|
||||
|
|
@ -114,7 +116,11 @@ async function run(scenario: Scenario): Promise<Record<string, number>> {
|
|||
const directory = mkdtempSync(join(tmpdir(), "pi-durable-view-bench-"));
|
||||
let storage: Storage | undefined;
|
||||
try {
|
||||
storage = await openNodeSqliteStorage(join(directory, "session.sqlite"));
|
||||
const path = join(directory, "session.sqlite");
|
||||
storage =
|
||||
process.env.PI_DURABLE_BENCH_STORAGE === "store"
|
||||
? await openStoreStorage(path)
|
||||
: await openNodeSqliteStorage(path);
|
||||
const models = createModels();
|
||||
const harness = await Harness.open(storage, { models, registry: createRegistry() }, context);
|
||||
const root = await harness.root(context);
|
||||
|
|
|
|||
108
packages/durable/test/examples/35-durable-on-the-store.ts
Normal file
108
packages/durable/test/examples/35-durable-on-the-store.ts
Normal file
|
|
@ -0,0 +1,108 @@
|
|||
// Durable on the store: a session's worker keeps everything in one store. Durable's Session (a Harness) opens
|
||||
// `StoreStorage` over the host's store under its object's type name, next to a Chord object's own table and documents;
|
||||
// both commit through the store's one commit queue (one seq each), one id allocator numbers them all, and one export
|
||||
// carries both.
|
||||
// Run from packages/durable:
|
||||
// node --conditions=source --experimental-strip-types test/examples/35-durable-on-the-store.ts
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
|
||||
import {
|
||||
createHost,
|
||||
defineFacet,
|
||||
defineTable,
|
||||
local,
|
||||
method,
|
||||
openStore,
|
||||
sqliteBackend,
|
||||
str,
|
||||
} from "@earendil-works/chord/next";
|
||||
import { openNodeSqliteDatabase } from "@earendil-works/chord/sqlite/node";
|
||||
import { createModels } from "@earendil-works/pi-ai";
|
||||
import { createRegistry, Harness, ROOT_CONVERSATION_ID } from "../../src/index.ts";
|
||||
import { StoreStorage } from "../../src/storage/store/index.ts";
|
||||
|
||||
const context = BACKGROUND_CONTEXT;
|
||||
|
||||
const Notes = defineTable("notes", { version: 1, columns: { id: str(), text: str() }, key: ["id"] });
|
||||
|
||||
// The harness object: local with empty params, so it lives as long as the host and may persist (P3).
|
||||
const Chat = local("example.chat", {
|
||||
version: 1,
|
||||
params: {},
|
||||
members: { note: method(str()), say: method(str()) },
|
||||
});
|
||||
|
||||
const facet = defineFacet((host) => {
|
||||
host.serve(Chat, (_, self) => {
|
||||
// Its own table, under its type name ...
|
||||
const notes = self.table(Notes);
|
||||
// ... and Durable's records under the same owner name: claims per table and per document namespace.
|
||||
const harness = self.open(
|
||||
async (open) => {
|
||||
const opened = await Harness.open(
|
||||
await StoreStorage.open(store, Chat.name, open),
|
||||
{ models: createModels(), registry: createRegistry() },
|
||||
open,
|
||||
);
|
||||
await opened.root(open);
|
||||
return opened;
|
||||
},
|
||||
(h, _reason, close) => h.close(close),
|
||||
);
|
||||
return {
|
||||
note: async (text, ctx) => {
|
||||
await self.tx(async (tx) => tx.put(notes, { id: `n${text.length}`, text }), ctx);
|
||||
return null;
|
||||
},
|
||||
say: async (text, ctx) => {
|
||||
const root = await harness.value.conversation(ROOT_CONVERSATION_ID, ctx);
|
||||
await root!.submit({ type: "write", entry: { kind: "example.said", data: { text } } }, ctx);
|
||||
return null;
|
||||
},
|
||||
};
|
||||
});
|
||||
});
|
||||
|
||||
const directory = await mkdtemp(join(tmpdir(), "chord-example-26-"));
|
||||
const path = join(directory, "session.sqlite");
|
||||
// 1. The worker's start code opens the store and gives it to its host.
|
||||
const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context);
|
||||
const host = createHost({ store, onError: (error) => console.log("onError", error.kind) });
|
||||
await host.load([{ name: "@example/core", facet }], context);
|
||||
const chat = await host.get(Chat, {}, context);
|
||||
|
||||
// 2. Chord's commit and Durable's commit, each one seq of the same store.
|
||||
const before = store.lastSeq;
|
||||
await chat.note("hello", context);
|
||||
await chat.say("hello, Durable", context);
|
||||
console.log("seqs taken:", store.lastSeq - before);
|
||||
|
||||
// 3. One archive holds both: Chord's table, Durable's tables and documents, all under the owner name.
|
||||
const owners = new Map<string, number>();
|
||||
for await (const part of store.export(context)) {
|
||||
const owner =
|
||||
part.kind === "table" || part.kind === "put"
|
||||
? part.table.split(":")[0]
|
||||
: part.kind === "document.create"
|
||||
? part.record.owner
|
||||
: undefined;
|
||||
if (owner !== undefined) owners.set(`${owner} ${part.kind}`, (owners.get(`${owner} ${part.kind}`) ?? 0) + 1);
|
||||
if (part.kind === "manifest") console.log("manifest:", { lastSeq: part.lastSeq, nextId: part.nextId });
|
||||
}
|
||||
console.log("parts by owner and kind:", Object.fromEntries(owners));
|
||||
|
||||
// 4. The app closes the host, then the store; a reopen finds both.
|
||||
await host.close(context);
|
||||
await store.close(context);
|
||||
const reopened = await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context);
|
||||
const storage = await StoreStorage.open(reopened, Chat.name, context);
|
||||
const page = await storage.scanEntries({ conversationId: ROOT_CONVERSATION_ID }, 10, undefined, context);
|
||||
console.log(
|
||||
"Durable entries after reopen:",
|
||||
page.items.map((entry) => entry.kind),
|
||||
);
|
||||
await storage.close(context);
|
||||
await reopened.close(context);
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
91
packages/durable/test/examples/36-durable-views.ts
Normal file
91
packages/durable/test/examples/36-durable-views.ts
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
// Durable's views as Chord sources: `self.follow(attach)` turns Durable's synchronous observers (the conversation view,
|
||||
// the task graph) into watchables, members and derive sources of the object that opened Durable. With Durable on the
|
||||
// host's store, Durable adopts its commits inside the store's commit step, in the host's batch: a derive over two views
|
||||
// of one commit computes once, and a watcher gets one frame per commit.
|
||||
// Run from packages/durable:
|
||||
// node --conditions=source --experimental-strip-types test/examples/36-durable-views.ts
|
||||
|
||||
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
|
||||
import {
|
||||
arr,
|
||||
createHost,
|
||||
defineFacet,
|
||||
int,
|
||||
local,
|
||||
method,
|
||||
obj,
|
||||
openStore,
|
||||
sqliteBackend,
|
||||
str,
|
||||
} from "@earendil-works/chord/next";
|
||||
import { openNodeSqliteDatabase } from "@earendil-works/chord/sqlite/node";
|
||||
import { createModels } from "@earendil-works/pi-ai";
|
||||
import { createRegistry, Harness, ROOT_CONVERSATION_ID } from "../../src/index.ts";
|
||||
import { StoreStorage } from "../../src/storage/store/index.ts";
|
||||
|
||||
const context = BACKGROUND_CONTEXT;
|
||||
|
||||
const Chat = local("example.chat", {
|
||||
version: 1,
|
||||
params: {},
|
||||
members: {
|
||||
// What a UI shows: the kinds of the transcript's entries and the number of tasks, from two views.
|
||||
summary: obj({ kinds: arr(str()), tasks: int() }),
|
||||
write: method(str()),
|
||||
},
|
||||
});
|
||||
|
||||
const facet = defineFacet((host) => {
|
||||
host.serve(Chat, (_, self) => {
|
||||
const harness = self.open(
|
||||
async (open) => {
|
||||
const opened = await Harness.open(
|
||||
await StoreStorage.open(store, Chat.name, open),
|
||||
{ models: createModels(), registry: createRegistry() },
|
||||
open,
|
||||
);
|
||||
await opened.root(open);
|
||||
return opened;
|
||||
},
|
||||
(h, _reason, close) => h.close(close),
|
||||
);
|
||||
// 1. Follow two of Durable's views; they attach once the natives opened.
|
||||
const view = self.follow((observer, ctx) =>
|
||||
harness.value.observeConversation(ROOT_CONVERSATION_ID, observer, ctx, { min: 50 }),
|
||||
);
|
||||
const graph = self.follow((observer, ctx) => harness.value.observeTaskGraph(observer, ctx));
|
||||
// 2. A derive over both: one compute per Durable commit.
|
||||
const summary = self.derive([view, graph], ([v, g]) => ({
|
||||
kinds: v.entries.map((entry) => entry.kind),
|
||||
tasks: Object.keys(g.tasks).length,
|
||||
}));
|
||||
return {
|
||||
summary,
|
||||
write: async (kind, ctx) => {
|
||||
const root = await harness.value.conversation(ROOT_CONVERSATION_ID, ctx);
|
||||
await root!.submit({ type: "write", entry: { kind } }, ctx);
|
||||
return null;
|
||||
},
|
||||
};
|
||||
});
|
||||
});
|
||||
|
||||
const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context);
|
||||
const host = createHost({ store, onError: (error) => console.log("onError", error.kind) });
|
||||
await host.load([{ name: "@example/core", facet }], context);
|
||||
const chat = await host.get(Chat, {}, context);
|
||||
|
||||
// 3. Watch the member: one frame per Durable commit, with its ops.
|
||||
const watch = await chat.summary.watch(context);
|
||||
watch.start(async (call) => {
|
||||
if (call.kind === "value") console.log("frame:", call.value.kinds, JSON.stringify(call.ops));
|
||||
});
|
||||
await chat.write("example.first", context);
|
||||
await chat.write("example.second", context);
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
console.log("member value:", chat.summary.value);
|
||||
|
||||
// 4. Closing the host detaches the observers; the store is the app's.
|
||||
await watch.stop();
|
||||
await host.close(context);
|
||||
await store.close(context);
|
||||
|
|
@ -17,8 +17,9 @@ import { openNodeJsonlStorage } from "../src/storage/jsonl/node.ts";
|
|||
import { MemoryStorage } from "../src/storage/memory.ts";
|
||||
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
|
||||
import type { Storage } from "../src/types.ts";
|
||||
import { openStoreStorage } from "./store-storage-support.ts";
|
||||
|
||||
const STORAGE_BENCHMARK_BACKENDS = ["memory", "sqlite", "jsonl"] as const;
|
||||
const STORAGE_BENCHMARK_BACKENDS = ["memory", "sqlite", "store", "jsonl"] as const;
|
||||
type StorageBenchmarkBackend = (typeof STORAGE_BENCHMARK_BACKENDS)[number];
|
||||
|
||||
type MemorySnapshot = {
|
||||
|
|
@ -125,11 +126,13 @@ async function runWorker(backend: StorageBenchmarkBackend, scale: StorageBenchma
|
|||
storage = new MemoryStorage();
|
||||
} else {
|
||||
directory = await mkdtemp(join(tmpdir(), `pi-durable-${backend}-memory-`));
|
||||
path = join(directory, backend === "sqlite" ? "storage.sqlite" : "storage");
|
||||
path = join(directory, backend === "jsonl" ? "storage" : "storage.sqlite");
|
||||
storage =
|
||||
backend === "sqlite"
|
||||
? await openNodeSqliteStorage(path)
|
||||
: await openNodeJsonlStorage(path, BACKGROUND_CONTEXT);
|
||||
: backend === "store"
|
||||
? await openStoreStorage(path)
|
||||
: await openNodeJsonlStorage(path, BACKGROUND_CONTEXT);
|
||||
}
|
||||
|
||||
try {
|
||||
|
|
@ -151,7 +154,7 @@ async function runWorker(backend: StorageBenchmarkBackend, scale: StorageBenchma
|
|||
collectGarbage();
|
||||
const postRead = snapshot();
|
||||
let disk: DiskFootprint | undefined;
|
||||
if (path !== undefined && backend === "sqlite") {
|
||||
if (path !== undefined && backend !== "jsonl") {
|
||||
disk = {
|
||||
...sqliteMetrics(path),
|
||||
mainBytes: await fileSize(path),
|
||||
|
|
|
|||
|
|
@ -14,8 +14,9 @@ import { openNodeJsonlStorage } from "../src/storage/jsonl/node.ts";
|
|||
import { MemoryStorage } from "../src/storage/memory.ts";
|
||||
import { openNodeSqliteStorage } from "../src/storage/sqlite/node.ts";
|
||||
import type { EntryId, Storage } from "../src/types.ts";
|
||||
import { openStoreStorage } from "./store-storage-support.ts";
|
||||
|
||||
const STORAGE_BENCHMARK_BACKENDS = ["memory", "sqlite", "jsonl"] as const;
|
||||
const STORAGE_BENCHMARK_BACKENDS = ["memory", "sqlite", "store", "jsonl"] as const;
|
||||
type StorageBenchmarkBackend = (typeof STORAGE_BENCHMARK_BACKENDS)[number];
|
||||
|
||||
const READ_OPTIONS = { time: 300, iterations: 10, warmupTime: 75, warmupIterations: 3 } as const;
|
||||
|
|
@ -38,9 +39,10 @@ async function createFixture(backend: StorageBenchmarkBackend): Promise<Fixture>
|
|||
}
|
||||
const directory = await mkdtemp(join(tmpdir(), "pi-durable-benchmark-"));
|
||||
directories.push(directory);
|
||||
if (backend === "sqlite") {
|
||||
if (backend === "sqlite" || backend === "store") {
|
||||
const path = join(directory, "storage.sqlite");
|
||||
const fixture = { backend, storage: await openNodeSqliteStorage(path) } satisfies Fixture;
|
||||
const storage = backend === "sqlite" ? await openNodeSqliteStorage(path) : await openStoreStorage(path);
|
||||
const fixture = { backend, storage } satisfies Fixture;
|
||||
fixtures.push(fixture);
|
||||
return fixture;
|
||||
}
|
||||
|
|
@ -123,11 +125,12 @@ type ReopenFixture = {
|
|||
};
|
||||
|
||||
async function openPersistentStorage(backend: PersistentBackend, path: string): Promise<Storage> {
|
||||
if (backend === "store") return openStoreStorage(path);
|
||||
return backend === "sqlite" ? openNodeSqliteStorage(path) : openNodeJsonlStorage(path, BACKGROUND_CONTEXT);
|
||||
}
|
||||
|
||||
async function copyPersistentStorage(backend: PersistentBackend, source: string, destination: string): Promise<void> {
|
||||
if (backend === "sqlite") await copyFile(source, destination);
|
||||
if (backend !== "jsonl") await copyFile(source, destination);
|
||||
else await cp(source, destination, { recursive: true });
|
||||
}
|
||||
|
||||
|
|
@ -136,7 +139,7 @@ for (const backend of STORAGE_BENCHMARK_BACKENDS) {
|
|||
if (backend === "memory") continue;
|
||||
const seedDirectory = await mkdtemp(join(tmpdir(), `pi-durable-${backend}-reopen-benchmark-`));
|
||||
directories.push(seedDirectory);
|
||||
const seedPath = join(seedDirectory, backend === "sqlite" ? "storage.sqlite" : "storage");
|
||||
const seedPath = join(seedDirectory, backend === "jsonl" ? "storage" : "storage.sqlite");
|
||||
const seed = await openPersistentStorage(backend, seedPath);
|
||||
const dataset = await seedStorageBenchmark(seed);
|
||||
await seed.close(BACKGROUND_CONTEXT);
|
||||
|
|
@ -144,7 +147,7 @@ for (const backend of STORAGE_BENCHMARK_BACKENDS) {
|
|||
Array.from({ length: REOPEN_OPTIONS.iterations + REOPEN_OPTIONS.warmupIterations }, async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), `pi-durable-${backend}-reopen-sample-`));
|
||||
directories.push(directory);
|
||||
const path = join(directory, backend === "sqlite" ? "storage.sqlite" : "storage");
|
||||
const path = join(directory, backend === "jsonl" ? "storage" : "storage.sqlite");
|
||||
await copyPersistentStorage(backend, seedPath, path);
|
||||
return path;
|
||||
}),
|
||||
|
|
|
|||
23
packages/durable/test/store-storage-node.ts
Normal file
23
packages/durable/test/store-storage-node.ts
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
// What `vitest.store.config.ts` puts in the place of `../src/storage/sqlite/node.ts`: the same exports, with
|
||||
// `openNodeSqliteStorage(path)` opening Durable's storage on Chord's store over that file (`StoreStorage`), so the
|
||||
// harness test files that take a SQLite storage run unchanged on the store.
|
||||
|
||||
import type { NodeSqliteDatabaseOptions } from "@earendil-works/chord/sqlite/node";
|
||||
import type { SqliteStorageOptions } from "../src/storage/sqlite/index.ts";
|
||||
import type { Storage } from "../src/types.ts";
|
||||
import { openStoreStorage } from "./store-storage-support.ts";
|
||||
|
||||
export {
|
||||
NodeSqliteDatabase,
|
||||
type NodeSqliteDatabaseOptions,
|
||||
openNodeSqliteDatabase,
|
||||
} from "@earendil-works/chord/sqlite/node";
|
||||
|
||||
/** Node SQLite connection settings plus durable storage options, as the original takes them. */
|
||||
export type NodeSqliteStorageOptions = NodeSqliteDatabaseOptions & SqliteStorageOptions;
|
||||
|
||||
/** A `StoreStorage` on a store over the file at `path`; a table prefix becomes nothing (the owner name separates). */
|
||||
export function openNodeSqliteStorage(path: string, options: NodeSqliteStorageOptions = {}): Promise<Storage> {
|
||||
const { prefix: _prefix, ...database } = options;
|
||||
return openStoreStorage(path, database);
|
||||
}
|
||||
46
packages/durable/test/store-storage-support.ts
Normal file
46
packages/durable/test/store-storage-support.ts
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
// StoreStorage over a SQLite file for tests: a store opened on the file, Durable's storage in it under a test owner
|
||||
// name, and a close that closes the store too (the app's part: in tests the storage is the store's only owner).
|
||||
|
||||
import { BACKGROUND_CONTEXT } from "@earendil-works/chord/context";
|
||||
import { openStore, sqliteBackend } from "@earendil-works/chord/next";
|
||||
import { type NodeSqliteDatabaseOptions, openNodeSqliteDatabase } from "@earendil-works/chord/sqlite/node";
|
||||
import { StoreStorage } from "../src/storage/store/index.ts";
|
||||
import type { Storage } from "../src/types.ts";
|
||||
|
||||
/** The owner name the test storages use. */
|
||||
export const TEST_OWNER = "durable.test";
|
||||
|
||||
/** A `StoreStorage` on a store over the SQLite file at `path` (`:memory:` too) whose `close` closes the store too. */
|
||||
export async function openStoreStorage(path: string, options: NodeSqliteDatabaseOptions = {}): Promise<Storage> {
|
||||
const store = await openStore(sqliteBackend(await openNodeSqliteDatabase(path, options)), BACKGROUND_CONTEXT);
|
||||
let storage: StoreStorage;
|
||||
try {
|
||||
storage = await StoreStorage.open(store, TEST_OWNER, BACKGROUND_CONTEXT);
|
||||
} catch (error) {
|
||||
await store.close(BACKGROUND_CONTEXT);
|
||||
throw error;
|
||||
}
|
||||
let closing: Promise<void> | undefined;
|
||||
const close = (context: Parameters<Storage["close"]>[0]): Promise<void> => {
|
||||
closing ??= storage.close(context).then(() => store.close(context));
|
||||
return closing;
|
||||
};
|
||||
// StoreStorage keeps its state in private fields: its methods run on the storage itself, bound once.
|
||||
const bound = new Map<PropertyKey, unknown>([["close", close]]);
|
||||
return new Proxy(storage, {
|
||||
get(target, property) {
|
||||
let value = bound.get(property);
|
||||
if (value !== undefined) return value;
|
||||
value = Reflect.get(target, property, target);
|
||||
if (typeof value !== "function") return value;
|
||||
value = value.bind(target);
|
||||
bound.set(property, value);
|
||||
return value;
|
||||
},
|
||||
// A test replacing a method (`storage.commit = …`) gets its own on the next read.
|
||||
set(target, property, value) {
|
||||
bound.delete(property);
|
||||
return Reflect.set(target, property, value, target);
|
||||
},
|
||||
});
|
||||
}
|
||||
592
packages/durable/test/store-storage.test.ts
Normal file
592
packages/durable/test/store-storage.test.ts
Normal file
|
|
@ -0,0 +1,592 @@
|
|||
// StoreStorage (slice 6b §2): Durable's storage conformance suite on SQLite `:memory:` and across reopen, then what is
|
||||
// its own: claims and close, owners sharing a store, the store's semantics for a refused commit, the allocator.
|
||||
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { setFlagsFromString } from "node:v8";
|
||||
import { runInNewContext } from "node:vm";
|
||||
import type { Context } from "@earendil-works/chord";
|
||||
import { BACKGROUND_CONTEXT, withTelemetry } from "@earendil-works/chord/context";
|
||||
import {
|
||||
applyBatch,
|
||||
type ChordStore,
|
||||
createHost,
|
||||
defineFacet,
|
||||
defineState,
|
||||
importStore,
|
||||
int,
|
||||
local,
|
||||
method,
|
||||
obj,
|
||||
openStore,
|
||||
type Store,
|
||||
sqliteBackend,
|
||||
str,
|
||||
} from "@earendil-works/chord/next";
|
||||
import { openNodeSqliteDatabase } from "@earendil-works/chord/sqlite/node";
|
||||
import { createModels } from "@earendil-works/pi-ai";
|
||||
import { registerStorageConformance } from "@earendil-works/pi-durable/testing";
|
||||
import { InMemoryTelemetryContext } from "@earendil-works/pi-telemetry";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { StorageRequestError } from "../src/errors.ts";
|
||||
import { idFromNumber } from "../src/ids.ts";
|
||||
import { type CompactionResult, createRegistry, Harness, LiveDoc, type Tx } from "../src/index.ts";
|
||||
import { StoreStorage } from "../src/storage/store/index.ts";
|
||||
import type { ConversationId, DocumentId, EntryId, Id, Seq, Storage, StorageWrite, TaskId } from "../src/types.ts";
|
||||
import { ROOT_CONVERSATION_ID } from "../src/types.ts";
|
||||
import { openStoreStorage } from "./store-storage-support.ts";
|
||||
|
||||
const context = BACKGROUND_CONTEXT;
|
||||
const adopt = (): void => {};
|
||||
|
||||
/** A storage that closes and reopens its store file after every commit (Durable's reopen harness). */
|
||||
class ReopeningStorage implements Storage {
|
||||
#current: Storage;
|
||||
readonly #path: string;
|
||||
#closed = false;
|
||||
|
||||
constructor(current: Storage, path: string) {
|
||||
this.#current = current;
|
||||
this.#path = path;
|
||||
}
|
||||
|
||||
async commit(writes: readonly StorageWrite[], adopt: (seq: Seq) => void, commitContext: Context): Promise<Seq> {
|
||||
if (this.#closed) return this.#current.commit(writes, adopt, commitContext);
|
||||
try {
|
||||
return await this.#current.commit(writes, adopt, commitContext);
|
||||
} finally {
|
||||
await this.#current.close(context);
|
||||
this.#current = await openStoreStorage(this.#path);
|
||||
}
|
||||
}
|
||||
|
||||
mintId<I extends Id<string>>(): Promise<I> {
|
||||
return this.#current.mintId<I>();
|
||||
}
|
||||
conversation: Storage["conversation"] = (id, c) => this.#current.conversation(id, c);
|
||||
scanConversations: Storage["scanConversations"] = (q, l, cursor, c) =>
|
||||
this.#current.scanConversations(q, l, cursor, c);
|
||||
entry(id: EntryId, c: Context): ReturnType<Storage["entry"]>;
|
||||
entry(conversationId: ConversationId, id: EntryId, c: Context): ReturnType<Storage["entry"]>;
|
||||
entry(first: EntryId | ConversationId, second: EntryId | Context, third?: Context) {
|
||||
if (third === undefined) return this.#current.entry(idFromNumber<EntryId>(first), second as Context);
|
||||
return this.#current.entry(idFromNumber<ConversationId>(first), second as EntryId, third);
|
||||
}
|
||||
findLatestHeadMarker: Storage["findLatestHeadMarker"] = (id, at, c) => this.#current.findLatestHeadMarker(id, at, c);
|
||||
scanEntries: Storage["scanEntries"] = (q, l, cursor, c) => this.#current.scanEntries(q, l, cursor, c);
|
||||
task: Storage["task"] = (id, c) => this.#current.task(id, c);
|
||||
scanTasks: Storage["scanTasks"] = (q, l, cursor, c) => this.#current.scanTasks(q, l, cursor, c);
|
||||
submission: Storage["submission"] = (id, c) => this.#current.submission(id, c);
|
||||
scanSubmissions: Storage["scanSubmissions"] = (q, l, cursor, c) => this.#current.scanSubmissions(q, l, cursor, c);
|
||||
submissionByRequest: Storage["submissionByRequest"] = (id, request, c) =>
|
||||
this.#current.submissionByRequest(id, request, c);
|
||||
findDocument: Storage["findDocument"] = (address, at, c) => this.#current.findDocument(address, at, c);
|
||||
document: Storage["document"] = (id, at, c) => this.#current.document(id, at, c);
|
||||
scanDocuments: Storage["scanDocuments"] = (q, l, cursor, c) => this.#current.scanDocuments(q, l, cursor, c);
|
||||
|
||||
async close(closeContext: Context): Promise<void> {
|
||||
if (this.#closed) return;
|
||||
this.#closed = true;
|
||||
await this.#current.close(closeContext);
|
||||
}
|
||||
}
|
||||
|
||||
registerStorageConformance({ describe, expect, it }, "StoreStorage", async (use) => {
|
||||
const storage = await openStoreStorage(":memory:");
|
||||
try {
|
||||
await use(storage);
|
||||
} finally {
|
||||
await storage.close(context);
|
||||
}
|
||||
});
|
||||
|
||||
registerStorageConformance({ describe, expect, it }, "StoreStorage across reopen", async (use) => {
|
||||
const directory = await mkdtemp(join(tmpdir(), "pi-durable-store-conformance-"));
|
||||
const path = join(directory, "store.sqlite");
|
||||
const storage = new ReopeningStorage(await openStoreStorage(path), path);
|
||||
try {
|
||||
await use(storage);
|
||||
} finally {
|
||||
await storage.close(context);
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
/** A store on `:memory:`. */
|
||||
async function memoryStore() {
|
||||
return openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context);
|
||||
}
|
||||
|
||||
describe("claims and close", () => {
|
||||
it("refuse a second StoreStorage on one owner until the first closed; two owners share a store", async () => {
|
||||
const store = await memoryStore();
|
||||
const first = await StoreStorage.open(store, "harness", context);
|
||||
await expect(StoreStorage.open(store, "harness", context)).rejects.toMatchObject({
|
||||
code: "invalid",
|
||||
reason: "args",
|
||||
});
|
||||
const other = await StoreStorage.open(store, "other", context);
|
||||
await first.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
|
||||
expect(await other.conversation(ROOT_CONVERSATION_ID, context)).toBeUndefined();
|
||||
await first.close(context);
|
||||
await expect(first.conversation(ROOT_CONVERSATION_ID, context)).rejects.toThrow("StoreStorage is closed");
|
||||
const reopened = await StoreStorage.open(store, "harness", context);
|
||||
expect(await reopened.conversation(ROOT_CONVERSATION_ID, context)).toEqual({ id: ROOT_CONVERSATION_ID });
|
||||
await store.close(context);
|
||||
});
|
||||
|
||||
it("waits at close for its calls underway, then releases its claims", async () => {
|
||||
const store = await memoryStore();
|
||||
const storage = await StoreStorage.open(store, "harness", context);
|
||||
const commit = storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
|
||||
const closing = storage.close(context);
|
||||
await commit;
|
||||
await closing;
|
||||
await StoreStorage.open(store, "harness", context);
|
||||
await store.close(context);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the store's semantics (D4)", () => {
|
||||
it("a refused Durable commit rejects with Durable's message and keeps the store usable", async () => {
|
||||
const store = await memoryStore();
|
||||
const storage = await StoreStorage.open(store, "harness", context);
|
||||
const root = { type: "conversation", value: { id: ROOT_CONVERSATION_ID } } as const;
|
||||
await storage.commit([root], adopt, context);
|
||||
await expect(storage.commit([root], adopt, context)).rejects.toThrow("ID 1 already belongs to conversation");
|
||||
const entry = {
|
||||
id: idFromNumber<EntryId>(await storage.mintId()),
|
||||
conversationId: ROOT_CONVERSATION_ID,
|
||||
kind: "m",
|
||||
};
|
||||
await storage.commit([{ type: "entry", value: entry }], adopt, context);
|
||||
expect((await storage.entry(entry.id, context))?.entry).toEqual(entry);
|
||||
await store.close(context);
|
||||
});
|
||||
|
||||
it("raises the allocator above every id it holds when it opens (a store built by import)", async () => {
|
||||
const store = await memoryStore();
|
||||
const storage = await StoreStorage.open(store, "harness", context);
|
||||
const entry = { id: idFromNumber<EntryId>(500), conversationId: ROOT_CONVERSATION_ID, kind: "m" };
|
||||
await storage.commit(
|
||||
[
|
||||
{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } },
|
||||
{ type: "entry", value: entry },
|
||||
],
|
||||
adopt,
|
||||
context,
|
||||
);
|
||||
expect(await storage.mintId()).toBe(501);
|
||||
await store.close(context);
|
||||
});
|
||||
});
|
||||
|
||||
const MemoDoc = defineState("memo", {
|
||||
schema: obj({ m: int() }),
|
||||
version: 1,
|
||||
initial: () => ({ m: 1 }),
|
||||
persist: true,
|
||||
checkpointWhen: () => true,
|
||||
});
|
||||
|
||||
const Memo = local("app.memo", { version: 1, params: {}, members: { memo: obj({ m: int() }), set: method() } });
|
||||
|
||||
/** A store on `:memory:` with an outbox from its first commit, and Durable's storage in it. */
|
||||
async function recorded() {
|
||||
const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context, {
|
||||
outbox: true,
|
||||
})) as ChordStore;
|
||||
const storage = await StoreStorage.open(store, "harness", context);
|
||||
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
|
||||
return { store, storage };
|
||||
}
|
||||
|
||||
/** Rewindable history, a retired latest document, a create-plus-retire and a change-plus-retire. */
|
||||
async function history(storage: Storage): Promise<{ readonly rewindable: DocumentId; readonly seqs: Seq[] }> {
|
||||
const scope = { kind: "conversation", conversationId: ROOT_CONVERSATION_ID } as const;
|
||||
const rewindable = await storage.mintId<DocumentId>();
|
||||
const latest = await storage.mintId<DocumentId>();
|
||||
const empty = await storage.mintId<DocumentId>();
|
||||
const seqs: Seq[] = [];
|
||||
const base = (n: number) => ({ kind: "base", version: 1, value: { n } }) as const;
|
||||
seqs.push(
|
||||
await storage.commit(
|
||||
[
|
||||
{
|
||||
type: "document.create",
|
||||
record: { id: rewindable, kind: "r", scope, history: "rewindable", fork: "asOf" },
|
||||
content: base(0),
|
||||
},
|
||||
{
|
||||
type: "document.create",
|
||||
record: { id: latest, kind: "l", scope, history: "latest", fork: "current" },
|
||||
content: base(0),
|
||||
},
|
||||
],
|
||||
adopt,
|
||||
context,
|
||||
),
|
||||
);
|
||||
seqs.push(
|
||||
await storage.commit(
|
||||
[{ type: "document.change", id: rewindable, content: { kind: "delta", version: 1, ops: [["s", ["n"], 1]] } }],
|
||||
adopt,
|
||||
context,
|
||||
),
|
||||
);
|
||||
seqs.push(
|
||||
await storage.commit(
|
||||
[
|
||||
{ type: "document.change", id: latest, content: { kind: "delta", version: 1, ops: [["s", ["n"], 2]] } },
|
||||
{ type: "document.retire", id: latest },
|
||||
{ type: "document.create", record: { id: empty, kind: "e", scope: { kind: "session" } }, content: base(3) },
|
||||
{ type: "document.retire", id: empty },
|
||||
],
|
||||
adopt,
|
||||
context,
|
||||
),
|
||||
);
|
||||
return { rewindable, seqs };
|
||||
}
|
||||
|
||||
describe("archives and batches of Durable data", () => {
|
||||
it("export and import keep rewindable history, retired records and Chord's own data: reads at a point agree", async () => {
|
||||
const { store, storage } = await recorded();
|
||||
const { rewindable, seqs } = await history(storage);
|
||||
// Chord's own data in the same store: an object's persisted document.
|
||||
const host = createHost({ store, onError: () => {} });
|
||||
const facet = defineFacet((h) =>
|
||||
h.serve(Memo, (_, self) => {
|
||||
const memo = self.state(MemoDoc);
|
||||
return {
|
||||
memo,
|
||||
set: async (ctx) => {
|
||||
await memo.change((draft) => {
|
||||
draft.m = 2;
|
||||
}, ctx);
|
||||
return null;
|
||||
},
|
||||
};
|
||||
}),
|
||||
);
|
||||
await host.load([{ name: "app", facet }], context);
|
||||
await (await host.get(Memo, {}, context)).set(context);
|
||||
await host.close(context);
|
||||
const parts = [];
|
||||
for await (const part of store.export(context)) parts.push(part);
|
||||
expect(parts.filter((part) => part.kind === "document.create").map(({ record }) => record.owner)).toContain(
|
||||
"app.memo",
|
||||
);
|
||||
const copy = await importStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), parts, context);
|
||||
const imported = await StoreStorage.open(copy, "harness", context);
|
||||
for (const at of [seqs[0]!, seqs[1]!, "current" as const]) {
|
||||
expect(await imported.document(rewindable, at, context)).toEqual(
|
||||
await storage.document(rewindable, at, context),
|
||||
);
|
||||
}
|
||||
const scope = { kind: "conversation", conversationId: ROOT_CONVERSATION_ID } as const;
|
||||
const at = seqs[1]!;
|
||||
expect(await imported.scanDocuments({ scope, at }, 10, undefined, context)).toEqual(
|
||||
await storage.scanDocuments({ scope, at }, 10, undefined, context),
|
||||
);
|
||||
const again = [];
|
||||
for await (const part of copy.export(context)) again.push(part);
|
||||
expect(again).toEqual(parts);
|
||||
expect(await imported.mintId()).toBe(await storage.mintId());
|
||||
await copy.close(context);
|
||||
await store.close(context);
|
||||
});
|
||||
|
||||
it("an outbox of Durable commits replayed into an empty store gives the same archive (create-plus-retire included)", async () => {
|
||||
const { store, storage } = await recorded();
|
||||
await history(storage);
|
||||
const commits = await (store as ChordStore).outbox!.read(0, 1_000, context);
|
||||
const replica = (await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context)) as ChordStore;
|
||||
await applyBatch(replica, commits, context);
|
||||
const exported = async (from: Store) => {
|
||||
const parts = [];
|
||||
for await (const part of from.export(context)) parts.push(part);
|
||||
return parts;
|
||||
};
|
||||
expect(await exported(replica)).toEqual(await exported(store));
|
||||
await replica.close(context);
|
||||
await store.close(context);
|
||||
});
|
||||
});
|
||||
|
||||
describe("head markers (the entry_heads table)", () => {
|
||||
it("a head entry is found after reopen, in an imported archive and in a replayed outbox", async () => {
|
||||
const directory = await mkdtemp(join(tmpdir(), "pi-durable-store-heads-"));
|
||||
const path = join(directory, "store.sqlite");
|
||||
const open = async () =>
|
||||
(await openStore(sqliteBackend(await openNodeSqliteDatabase(path)), context, { outbox: true })) as ChordStore;
|
||||
let store = await open();
|
||||
let storage = await StoreStorage.open(store, "harness", context);
|
||||
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
|
||||
const entry = async (head?: EntryId): Promise<EntryId> => {
|
||||
const id = await storage.mintId<EntryId>();
|
||||
const value = { id, conversationId: ROOT_CONVERSATION_ID, kind: "k", data: {}, ...(head ? { head } : {}) };
|
||||
await storage.commit([{ type: "entry", value }], adopt, context);
|
||||
return id;
|
||||
};
|
||||
const first = await entry();
|
||||
const marker = await entry(first);
|
||||
await entry();
|
||||
/** The newest head marker, and the one at or before the first entry (none). */
|
||||
const heads = async (of: Storage) => [
|
||||
(await of.findLatestHeadMarker(ROOT_CONVERSATION_ID, undefined, context))?.id,
|
||||
(await of.findLatestHeadMarker(ROOT_CONVERSATION_ID, first, context))?.id,
|
||||
];
|
||||
expect(await heads(storage)).toEqual([marker, undefined]);
|
||||
await storage.close(context);
|
||||
await store.close(context);
|
||||
store = await open();
|
||||
storage = await StoreStorage.open(store, "harness", context);
|
||||
expect(await heads(storage)).toEqual([marker, undefined]);
|
||||
const parts = [];
|
||||
for await (const part of store.export(context)) parts.push(part);
|
||||
expect(parts.filter((part) => part.kind === "put" && part.table === "harness:entry_heads")).toEqual([
|
||||
expect.objectContaining({ row: { id: marker, conversation_id: ROOT_CONVERSATION_ID } }),
|
||||
]);
|
||||
const copy = await importStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), parts, context);
|
||||
expect(await heads(await StoreStorage.open(copy, "harness", context))).toEqual([marker, undefined]);
|
||||
const replica = (await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context)) as ChordStore;
|
||||
await applyBatch(replica, await store.outbox!.read(0, 1_000, context), context);
|
||||
expect(await heads(await StoreStorage.open(replica, "harness", context))).toEqual([marker, undefined]);
|
||||
for (const each of [copy, replica, store]) await each.close(context);
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
});
|
||||
});
|
||||
|
||||
describe("reads (slice 6b §1.4, §2.4)", () => {
|
||||
it("a request error inside a multi-query read rejects that read only; the store goes on", async () => {
|
||||
const { store, storage } = await recorded();
|
||||
await expect(
|
||||
storage.scanEntries({ conversationId: idFromNumber<ConversationId>(999) }, 10, undefined, context),
|
||||
).rejects.toBeInstanceOf(StorageRequestError);
|
||||
await expect(storage.document(idFromNumber<DocumentId>(999), "current", context)).resolves.toBeUndefined();
|
||||
expect(await storage.conversation(ROOT_CONVERSATION_ID, context)).toEqual({ id: ROOT_CONVERSATION_ID });
|
||||
await store.close(context);
|
||||
});
|
||||
});
|
||||
|
||||
describe("commits (slice 6b §1.3, §2.5)", () => {
|
||||
it("adopt in the attached host's batch, and count four statements for an entry append and for a delta", async () => {
|
||||
const telemetry = new InMemoryTelemetryContext();
|
||||
const span = await telemetry.startSpan({ name: "test" }, (each) => each);
|
||||
const metered = withTelemetry(span, context);
|
||||
const counts = (): number =>
|
||||
telemetry
|
||||
.getMetrics()
|
||||
.filter(({ name }) => name === "chord.store.statements")
|
||||
.reduce((total, { value }) => total + value, 0);
|
||||
const store = (await openStore(sqliteBackend(await openNodeSqliteDatabase(":memory:")), context)) as ChordStore;
|
||||
const host = createHost({ store, onError: () => {} });
|
||||
const storage = await StoreStorage.open(store, "harness", context);
|
||||
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
|
||||
let running: boolean | undefined;
|
||||
const entry = {
|
||||
id: idFromNumber<EntryId>(await storage.mintId()),
|
||||
conversationId: ROOT_CONVERSATION_ID,
|
||||
kind: "m",
|
||||
};
|
||||
const before = counts();
|
||||
await storage.commit(
|
||||
[{ type: "entry", value: entry }],
|
||||
() => {
|
||||
running = store.batch!.running;
|
||||
},
|
||||
metered,
|
||||
);
|
||||
const appended = counts() - before;
|
||||
expect(running).toBe(true);
|
||||
const id = await storage.mintId<DocumentId>();
|
||||
const record = { id, kind: "s", scope: { kind: "session" } } as const;
|
||||
await storage.commit(
|
||||
[{ type: "document.create", record, content: { kind: "base", version: 1, value: { n: 0 } } }],
|
||||
adopt,
|
||||
context,
|
||||
);
|
||||
const changed = counts();
|
||||
await storage.commit(
|
||||
[{ type: "document.change", id, content: { kind: "delta", version: 1, ops: [["s", ["n"], 1]] } }],
|
||||
adopt,
|
||||
metered,
|
||||
);
|
||||
expect([appended, counts() - changed]).toEqual([4, 4]);
|
||||
await host.close(context);
|
||||
await store.close(context);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Durable's views as Chord sources on the store (slice 6b §3)", () => {
|
||||
const Chat = local("app.chat", {
|
||||
version: 1,
|
||||
params: {},
|
||||
members: { entries: int(), note: method(obj({ kind: str() })) },
|
||||
});
|
||||
|
||||
it("a derive over the conversation view and the task graph: one frame per Durable commit, both views in one batch", async () => {
|
||||
const store = await memoryStore();
|
||||
const host = createHost({ store, onError: () => {} });
|
||||
let computes = 0;
|
||||
const facet = defineFacet((h) =>
|
||||
h.serve(Chat, (_, self) => {
|
||||
const harness = self.open(
|
||||
async (open) => {
|
||||
const opened = await Harness.open(
|
||||
await StoreStorage.open(store, Chat.name, open),
|
||||
{ models: createModels(), registry: createRegistry() },
|
||||
open,
|
||||
);
|
||||
await opened.root(open);
|
||||
return opened;
|
||||
},
|
||||
(h, _reason, close) => h.close(close),
|
||||
);
|
||||
const view = self.follow((observer, ctx) =>
|
||||
harness.value.observeConversation(ROOT_CONVERSATION_ID, observer, ctx),
|
||||
);
|
||||
const graph = self.follow((observer, ctx) => harness.value.observeTaskGraph(observer, ctx));
|
||||
const entries = self.derive([view, graph], ([v]) => {
|
||||
computes++;
|
||||
return v.entries.length;
|
||||
});
|
||||
return {
|
||||
entries,
|
||||
note: async ({ kind }, ctx) => {
|
||||
const root = await harness.value.conversation(ROOT_CONVERSATION_ID, ctx);
|
||||
await root!.submit({ type: "write", entry: { kind } }, ctx);
|
||||
return null;
|
||||
},
|
||||
};
|
||||
}),
|
||||
);
|
||||
await host.load([{ name: "app", facet }], context);
|
||||
const chat = await host.get(Chat, {}, context);
|
||||
const calls: unknown[] = [];
|
||||
const watch = await chat.entries.watch(context);
|
||||
watch.start(async (call) => void calls.push(call.kind === "value" ? call.value : call.kind));
|
||||
const before = computes;
|
||||
await chat.note({ kind: "test.note" }, context);
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
expect(chat.entries.value).toBeGreaterThan(0);
|
||||
expect(calls).toEqual([chat.entries.value]);
|
||||
expect(computes - before).toBe(1);
|
||||
await host.close(context);
|
||||
await store.close(context);
|
||||
});
|
||||
|
||||
it("a deferred Durable commit, then an immediate one, reach the followed document in order", async () => {
|
||||
const store = await memoryStore();
|
||||
const host = createHost({ store, onError: () => {} });
|
||||
const Live = local("app.live", {
|
||||
version: 1,
|
||||
params: {},
|
||||
members: { n: int(), bump: method() },
|
||||
});
|
||||
/** A commit setting the root's `LiveDoc` failure note to `value`. */
|
||||
const set = (value: number) => async (tx: Tx) => {
|
||||
(await tx.doc(LiveDoc, ROOT_CONVERSATION_ID)).compactionFailure = {
|
||||
taskId: idFromNumber<TaskId<CompactionResult>>(1),
|
||||
trigger: "manual",
|
||||
error: String(value),
|
||||
};
|
||||
};
|
||||
const facet = defineFacet((h) =>
|
||||
h.serve(Live, (_, self) => {
|
||||
const harness = self.open(
|
||||
async (open) => {
|
||||
const opened = await Harness.open(
|
||||
await StoreStorage.open(store, Live.name, open),
|
||||
{ models: createModels(), registry: createRegistry() },
|
||||
open,
|
||||
);
|
||||
await (await opened.root(open)).commit(set(0), open);
|
||||
return opened;
|
||||
},
|
||||
(h, _reason, close) => h.close(close),
|
||||
);
|
||||
const live = self.follow((observer, ctx) =>
|
||||
harness.value.observeDoc(LiveDoc, ROOT_CONVERSATION_ID, observer, ctx),
|
||||
);
|
||||
return {
|
||||
n: self.derive([live], ([value]) => Number(value.compactionFailure?.error ?? -1)),
|
||||
bump: async (ctx) => {
|
||||
const root = (await harness.value.conversation(ROOT_CONVERSATION_ID, ctx))!;
|
||||
await root.commit(set(1), ctx, { persistence: { kind: "deferred", withinMs: 10_000 } });
|
||||
await root.commit(set(2), ctx);
|
||||
return null;
|
||||
},
|
||||
};
|
||||
}),
|
||||
);
|
||||
await host.load([{ name: "app", facet }], context);
|
||||
const live = await host.get(Live, {}, context);
|
||||
const seen: unknown[] = [];
|
||||
const watch = await live.n.watch(context);
|
||||
watch.start(async (call) => void seen.push(call.kind === "value" ? call.value : call.kind));
|
||||
await live.bump(context);
|
||||
await new Promise((resolve) => setTimeout(resolve, 20));
|
||||
expect(live.n.value).toBe(2);
|
||||
expect(seen.at(-1)).toBe(2);
|
||||
await host.close(context);
|
||||
await store.close(context);
|
||||
});
|
||||
});
|
||||
|
||||
describe("review regressions", () => {
|
||||
it('a head-marker entry whose kind is "refused" is returned, not taken for a request error', async () => {
|
||||
const storage = await openStoreStorage(":memory:");
|
||||
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
|
||||
const first = await storage.mintId<EntryId>();
|
||||
const marker = await storage.mintId<EntryId>();
|
||||
await storage.commit(
|
||||
[
|
||||
{ type: "entry", value: { id: first, conversationId: ROOT_CONVERSATION_ID, kind: "message" } },
|
||||
{
|
||||
type: "entry",
|
||||
value: { id: marker, conversationId: ROOT_CONVERSATION_ID, kind: "refused", head: first },
|
||||
},
|
||||
],
|
||||
adopt,
|
||||
context,
|
||||
);
|
||||
expect((await storage.findLatestHeadMarker(ROOT_CONVERSATION_ID, undefined, context))?.id).toBe(marker);
|
||||
const page = await storage.scanEntries({ conversationId: ROOT_CONVERSATION_ID }, 10, undefined, context);
|
||||
expect(page.items.map(({ kind }) => kind)).toEqual(["refused", "message"]);
|
||||
await storage.close(context);
|
||||
});
|
||||
|
||||
it("an empty commit takes no seq", async () => {
|
||||
const { store, storage } = await recorded();
|
||||
const before = store.lastSeq;
|
||||
const adopted: Seq[] = [];
|
||||
expect(await storage.commit([], (seq) => adopted.push(seq), context)).toBe(before);
|
||||
expect(store.lastSeq).toBe(before);
|
||||
expect(adopted).toEqual([before]);
|
||||
await store.close(context);
|
||||
});
|
||||
});
|
||||
|
||||
describe("GC", () => {
|
||||
it("a closed StoreStorage is collected while its store stays open", async () => {
|
||||
setFlagsFromString("--expose-gc");
|
||||
const gc = runInNewContext("gc") as () => void;
|
||||
const store = await memoryStore();
|
||||
const opened = async (): Promise<WeakRef<StoreStorage>> => {
|
||||
const storage = await StoreStorage.open(store, "harness", context);
|
||||
await storage.commit([{ type: "conversation", value: { id: ROOT_CONVERSATION_ID } }], adopt, context);
|
||||
await storage.close(context);
|
||||
return new WeakRef(storage);
|
||||
};
|
||||
const ref = await opened();
|
||||
for (let round = 0; round < 10 && ref.deref() !== undefined; round++) {
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
gc();
|
||||
}
|
||||
expect(ref.deref()).toBeUndefined();
|
||||
await store.close(context);
|
||||
});
|
||||
});
|
||||
47
packages/durable/vitest.store.config.ts
Normal file
47
packages/durable/vitest.store.config.ts
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
// The harness test files that take a SQLite storage, run on Durable's storage on Chord's store (`StoreStorage`, slice
|
||||
// 6b): `../src/storage/sqlite/node.ts`'s opener is replaced by `test/store-storage-node.ts`; the files are unchanged.
|
||||
// Four tests edit or read `SqliteStorage`'s own tables with raw SQL (`tasks`, `durable_metadata`) and are left out.
|
||||
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { defineConfig, mergeConfig } from "vitest/config";
|
||||
import base from "./vitest.config.ts";
|
||||
|
||||
const storeOpener = fileURLToPath(new URL("./test/store-storage-node.ts", import.meta.url));
|
||||
|
||||
export default mergeConfig(
|
||||
base,
|
||||
defineConfig({
|
||||
test: {
|
||||
testNamePattern:
|
||||
/^(?!.*(resumes a retry backoff after reopen|sleeps again for a retry stored as a running phase|polls a checkpoint stored without a thinking level|refuses to create the root of an empty storage))/,
|
||||
include: [
|
||||
"test/harness-after-run.test.ts",
|
||||
"test/harness-blocked.test.ts",
|
||||
"test/harness-compaction.test.ts",
|
||||
"test/harness-continue.test.ts",
|
||||
"test/harness-conversations.test.ts",
|
||||
"test/harness-generation-recovery.test.ts",
|
||||
"test/harness-inbox.test.ts",
|
||||
"test/harness-input-entries.test.ts",
|
||||
"test/harness-lifecycle.test.ts",
|
||||
"test/harness-nested-tools-restart.test.ts",
|
||||
"test/harness-nested-tools.test.ts",
|
||||
"test/harness-ownership.test.ts",
|
||||
"test/harness-read-only.test.ts",
|
||||
"test/harness-restart-failure.test.ts",
|
||||
"test/harness-scheduler-indexes.test.ts",
|
||||
"test/harness-scheduler-retention.test.ts",
|
||||
"test/harness-sleep.test.ts",
|
||||
"test/harness-structured.test.ts",
|
||||
"test/harness-submissions.test.ts",
|
||||
"test/harness-task-graph.test.ts",
|
||||
"test/harness-tasks-recovery.test.ts",
|
||||
"test/harness-tool-hooks.test.ts",
|
||||
"test/harness-tools-recovery.test.ts",
|
||||
"test/harness-turn-limit.test.ts",
|
||||
"test/harness-view.test.ts",
|
||||
],
|
||||
},
|
||||
resolve: { alias: [{ find: /^\.\.\/src\/storage\/sqlite\/node\.ts$/, replacement: storeOpener }] },
|
||||
}),
|
||||
);
|
||||
Loading…
Add table
Add a link
Reference in a new issue