diff --git a/packages/agent/docs/pico-v5-handoff.md b/packages/agent/docs/pico-v5-handoff.md new file mode 100644 index 000000000..93372b466 --- /dev/null +++ b/packages/agent/docs/pico-v5-handoff.md @@ -0,0 +1,555 @@ +# Pico5 handoff and decision register + +> Working coordination document. `pico-v5.md` is the compact design. This file +> records accepted choices, unresolved decisions, examples, and layer sign-offs so +> work survives conversation compaction. No unresolved item may be decided silently +> during implementation. + +## Current status + +- `packages/agent/src/harness/pico/` and its tests were removed. +- The untracked `pico4` source/tests were removed. +- `pico3` remains as a reference, not an implementation base. +- Storage benchmark artifacts are in `/tmp/pi-pico5-storage-bench/`. +- Astra approved starting the memory + SQLite storage/conformance slice after the + storage decisions below are signed off. + +## Accepted direction + +1. Entries are immutable and paged; the Session is not one materialized DOM. +2. Authors mutate ordinary JSON documents; Chord produces nested/list/string ops. +3. History policy is per document: `latest` or `rewindable`. +4. Built-ins use three conversation documents: + - `rewindable`: historical configuration; + - `sticky`: present state such as queues and retry policy; + - `live`: high-churn turn, generation, tool, and compaction presentation state. +5. Typed document definitions and dynamic families are required. Family instances + may be task-owned and retire automatically. +6. Task recovery state is a complete replacement, separate from presentation and + application documents. +7. There is no visible-but-undurable publication. Every Chord-visible update follows + a successful durable commit. +8. Interrupted assistant/tool partials are converted into durable aborted/interrupted + entries by their recovering task kind, then removed from `live`. +9. A built-in needs cross-task document observation, so `watchDoc` is initial scope. +10. UI status is state, not an event protocol. There is no public event channel + initially; clients observe document paths. A bounded `live` notice field may be + designed with the built-in view if concrete transient facts require it. +11. Third-party documents initially use their own Chord services. Automatic mounting + into the agent view is deferred. +12. SQLite is the long-running/cloud backend. JSONL remains local and readable. +13. Implementation proceeds layer by layer and stops for user sign-off after each. + +## Decisions required before storage implementation + +### S1. Document identity and families — accepted + +A definition carries type, logical ID, scope, history, initializer, and optional +family ownership rules: + +```ts +const live = defineDoc({ + id: "pi.live", + scope: "conversation", + history: "latest", + initial: () => ({ tools: [] }), +}); + +const jobOutput = defineDocFamily({ + id: "pi.job", + scope: "conversation", + history: "latest", + owner: "task", + initial: input => ({ command: input.command, stdout: "" }), +}); +``` + +A persisted instance has a unique, never-reused storage identity plus definition ID, +scope owner, family instance ID, creation sequence, history policy, and retirement +sequence. Reusing one logical family key creates a new incarnation. + +### S2. Historical document membership — accepted + +A rewindable document retired today still exists in an as-of query from yesterday. +A latest document can be deleted after retirement. + +Example: + +```text +10 create rewindable review A +20 append entry E +30 retire A +40 fork at E +``` + +The fork enumerates A at commit 20 and copies its as-of value into a new child-owned +instance. Current enumeration at commit 40 omits A. + +### S3. Storage ownership and copying — proposed recommendation + +Simplest safe contract: + +- Commit inputs are borrowed until `Storage.commit()` settles. +- A backend retaining anything after settlement detaches it first. +- Memory uses a small recursive `copyJson`; JSONL/SQLite detach through serialization. +- Every storage read returns a fresh owned JSON copy. Mutating it cannot corrupt the + backend or another caller. Public types remain readonly, but correctness does not + rely on TypeScript at runtime. +- Large strings are immutable and may share their string value during an in-memory + copy; object/array containers never share mutably. + +Example failure this prevents: + +```ts +const task = { phase: "running" }; +await storage.commit([{ type: "task", task }]); +task.phase = "done"; // must not mutate MemoryStorage behind its back +``` + +Decision: approve this contract, or replace it with explicit ownership transfer and +state which caller may still use document ops for Chord publication. + +### S4. Table reads and writes inside `Tx` — proposed recommendation + +Adopt the other design's simple rule: + +```text +table reads first -> first table write -> later table reads throw ReadAfterWrite +document reads/mutations always read their transaction draft +``` + +Example: + +```ts +const inputs = await tx.inputs(ids); // read +const head = await tx.newestEntry(c); // read +const entry = tx.appendEntry(c, assistant); // first table write +tx.doc(live, c).message = undefined; // document draft is allowed +// await tx.entry(other) now throws ReadAfterWrite +``` + +Benefit: no shadow indexes for newly created entries/tasks/inputs. Cost: helpers must +load everything before their first table write. Validate all built-in settlement +traces before accepting. + +Decision: adopt this rule, or support read-your-writes for tables and pay for overlays. + +### S5. Escaped document draft — open + +A callback can retain a document proxy: + +```ts +let escaped: LiveState; +await session.commit(tx => { escaped = tx.doc(live, c); }); +escaped.message = partial; +``` + +Options: + +- **Detached stale draft:** mutation succeeds locally but has no effect. Simplest, + but silently hides a bug. +- **Revoked draft:** every later read/write throws. Clearest, but requires tracker + proxies to share an active guard or another membrane. +- **Debug assertion:** stale mutation is detected at the next transaction. Cheaper, + but delayed. + +Recommendation: first determine whether Chord tracker proxies can cheaply share one +revocation guard. Do not let an escaped proxy mutate committed state or contaminate a +later commit. + +### S6. Base policy — partly accepted + +Every document may receive bases. Storage gets both incremental ops and a borrowed +candidate value, serializing only one: + +```ts +persist(shouldBase(id, ops) ? [["r", value]] : ops); +``` + +- `latest`: a committed base permits reclaiming older records. +- `rewindable`: bases bound as-of replay; historical segments remain. +- Trigger on both encoded bytes and record count. + +Open values: + +- byte threshold; +- record-count threshold; +- whether document definitions may request `rebaseWhenInitial`. + +`pi.live` benefits from `rebaseWhenInitial`: when a turn ends at `{ tools: [] }`, a +tiny base allows immediate reclamation of streamed output. The definition/document +manager can recognize its initial state; storage still owns physical reclamation. + +### S7. Task rows — accepted + +Task records are full replacements. SQLite uses one live row and `DELETE` on terminal. +JSONL uses one task sidecar and unlinks it only after terminal publication. + +The runtime envelope owns kind/input/dependencies/memos/effect IDs. Kind-defined +recovery state cannot overwrite memos. A terminal transaction transfers any needed +outcome and retires every task-owned document family instance atomically. + +This atomic retirement removes the need for an application-level orphan sweep: + +- crash before terminal: task and owned docs remain live and recover; +- crash after terminal marker: logical retirement is known, and storage may clean + leftover files during open. + +### S8. SQLite representation — accepted direction + +Use one SQL transaction per Session commit: + +```text +conversations / entries / inputs +document metadata +document base+delta records indexed by document and commit +a current live-task row +``` + +Do not translate Chord ops into SQLite JSON functions initially. The benchmark found +full JSONB replacement slower than TEXT and generic SQL patching slower with more WAL +writes. Current/as-of reads select a base and indexed tail; they do not scan unrelated +documents. + +Still choose schema details, WAL checkpoint cadence, and `synchronous` default during +the SQLite layer. + +### S9. JSONL physical layout — current lean: uniform sidecars + +Simplest layout from the other design: + +```text +main.jsonl tables + one publication marker per commit +doc-.jsonl one file per document instance +task-.jsonl one file per live task +``` + +Every sidecar record is prepared. Sidecars append first; one main marker is always +last. This deliberately permits records such as: + +```json +{"seq":14,"refs":["task-9"]} +{"seq":15,"refs":["task-9"]} +{"seq":16,"refs":["pi.live-1"]} +``` + +The markers are cosmetic overhead but make every commit use one protocol. They can be +compact; fields such as `writes:[]` and repeated `maxId` are unnecessary if high-water +is represented elsewhere. Without `fsync`, the second append is unlikely to be the +performance bottleneck; the benchmark's no-fsync sidecar backend still handled about +9.5k–30k commits/s depending on workload while opening/closing each sidecar. + +Alternative: standalone one-file commits avoid markers but introduce two record modes, +distributed commit discovery, and more recovery cases. + +Decision: approve uniform marker publication or select the standalone optimization. + +### S10. JSONL process-crash contract — accepted direction + +No `fsync` by default. Awaited/full sidecar writes precede the awaited/full main +marker. For an ordinary process crash with the OS/filesystem alive: + +- torn final lines are removed; +- sidecar records without a marker are uncommitted and removed; +- a complete marker publishes the transaction. + +Power loss, host/kernel failure, or weak filesystem ordering may lose acknowledged +commits. Optional durable mode may fsync sidecars before the marker. + +Any uncertain write error poisons the open backend until reopen. Reclamation occurs +only after its base/retirement commits and uses temp + rename; cached descriptors are +then reopened. + +### S11. Missing confirmed sidecar — open + +Example: + +```text +sidecar record 20 was reported written +main marker 20 survives +sidecar record 20 is absent after machine failure +``` + +This should not arise from the stated ordinary process-crash contract. Options: + +- **Fail open:** report corruption and preserve all bytes for diagnosis/repair. + Simplest and never invents state. +- **Roll back globally:** truncate every file to the latest complete global prefix. + Recovers availability but may discard acknowledged later commits and must interact + correctly with reclaimed prefixes and standalone records if those exist. +- **Best-effort skip:** invalid because it exposes a partial atomic commit. + +Recommendation: fail open initially. Add a separate repair tool only after physical +fault tests establish a safe prefix algorithm. + +### S12. Main JSONL growth — accepted limitation + +Document/task sidecars reclaim high-churn data; `main.jsonl` still grows by entries, +lifecycle records, and commit markers. JSONL may scan it on open. It is the local +backend. Years-long products use SQLite. Do not add global JSONL compaction initially. + +### S13. Storage query surface — needs exact signatures + +The first storage contract must support: + +- create/current/retired conversation records; +- paged immutable entries and entry -> commit mapping; +- current task lookup/scan; +- current and as-of document enumeration; +- current and as-of document value; +- family-instance enumeration by definition/scope/owner; +- one atomic mixed commit; +- close/reopen. + +As-of results distinguish not-yet-created, retired after target, and unavailable +history. Concrete TypeScript signatures are the first layer's first sign-off. + +## Decisions required before document/task implementation + +### D1. Three built-in documents — accepted + +```ts +rewindable // model/tool/config choices inherited by historical fork +sticky // inbox, queue modes, retry policy +live // current message/tools/generation/collapse and maybe notices +``` + +`live` is durable because recovery converts its partial output into transcript entries. +Its high churn is independently reclaimable when it returns to initial state. + +### D2. Document adapter to Chord — accepted direction + +A thin adapter such as `replicatedState.own(documentSource)` exposes committed value +and ops. It owns no second tracker. Each document and mounted view has a contiguous +publication revision separate from sparse global Session commit sequence. + +Hydration captures value/revision and subscribes atomically. Listener errors after +persistence are isolated. Current Chord needs a small public source adapter; it does +not need harness knowledge. + +### D3. Cross-task `watchDoc` — accepted, exact scope open + +Required semantics: + +```ts +const watch = await api.watchDoc(jobOutput, job.id, context); +consume(watch.value); // snapshot first +watch.start((ops, value) => consume(value)); +``` + +- capture + subscription are one Session-line operation; +- callbacks run off the line and cannot re-enter a commit synchronously; +- watch is volatile and invocation-owned; +- recovery re-enters the handler, captures a fresh snapshot, and subscribes again; +- retirement appears as document absence; +- `stop()` is idempotent. + +Open scope: same conversation only, or any conversation owned by the observing task's +subtree. Initial built-in use must provide the concrete required edge. + +### D4. Reopen conversion — accepted + +Generation recovery reads `live.message`, appends an assistant entry with +`stopReason: "aborted"`, clears the preview, and retries/fails atomically. Tool +recovery turns persisted output into its interrupted result before clearing the slot. +The entry uses the normal model/render shape while model projection excludes aborted +content. + +Conversion runs before retiring any task-owned family document it needs. + +### D5. Document versioning/migration — open + +Document paths and persisted values are public/durable schema. A plugin upgrade may +change `{ comments: [...] }` to `{ threads: [...] }`. + +Options: + +- definition has an integer version and migration from complete current value; +- task/document kind owns migration during open; +- reject mismatched versions initially. + +Recommendation: persist definition version now even if initial implementation only +rejects mismatches. Retrofitting identity/version after stored products ship is harder. + +## Decisions required before view implementation + +### V1. Public view shape — open + +Fixed top-level fields: + +```ts +{ conversation, entries, rewindable, sticky, live } +``` + +Typed and direct, but every added built-in document changes the protocol. + +Document map: + +```ts +{ + conversation, + entries, + docs: { + "pi.rewindable": RewindableState, + "pi.sticky": StickyState, + "pi.live": LiveState, + }, +} +``` + +Makes document identity/lifetime explicit and can add selected documents later, but +paths and IDs become public protocol. It can remain typed for the three built-ins. + +Recommendation: decide before writing any UI adapter. In either shape, mount only the +three built-ins initially and treat every visible path as versioned public API. + +### V2. Public events — accepted: none initially + +A compaction-start event cannot describe current state to a late joiner. Instead: + +```ts +live.collapse = { through, stage: "summarizing" }; // running +live.collapse = undefined; // finished +``` + +Clients react to state ops by path and always hydrate the current truth. Hooks remain +pre-decision interception. Internal commit subscribers may exist mechanically, but +there is no separate public semantic event protocol initially. + +Whether failed attempts need a bounded `live.notices` list is deferred until the UI +is implemented. It remains state with explicit retention, not fire-and-forget events. + +### V3. Atomic publication — accepted + +The built-in mount consumes the complete Session commit, prefixes every changed +built-in document path, combines entry changes, and publishes one Chord batch. It has +no tracker or semantic projection. Independent third-party Chord services may publish +separately. + +### V4. Active transcript bound — open + +A Slack conversation cannot hydrate every entry ever written. Options include: + +- durable `head` entry: active transcript is the range beginning at its target; +- fixed recent entry/count/byte window; +- product-selected window with a stable cursor. + +Example requirement: + +```text +10 years / 500,000 entries +late UI attach -> bounded snapshot +older history -> explicit paged query +model context -> independently derived bounded projection +``` + +Choose before view implementation, not before storage. Entry paging and indexes are +required now regardless. + +### V5. Multi-entry fork boundary — open + +One commit may append entries E and F and leave one final rewindable state. Fork at E: + +- **entry prefix:** child transcript includes E but not F, while document state is the + final state of the containing commit; +- **commit boundary:** selecting E means inherit all entries from its indivisible + commit, including F; +- **restriction:** only designated forkable entries/commits may be targets. + +Choose before fork/view API. Storage records entry -> commit either way. + +### V6. Sticky fork initialization — current lean: defaults + +A historical fork inherits rewindable documents as-of its target and no live tasks. +Copying current sticky/live state can copy queued inputs or running previews from the +future. Simplest initial rule: + +```text +rewindable -> historical copy +sticky -> fresh defaults +live -> fresh defaults +``` + +Explicit sticky-copy options remain deferred. + +### V7. Third-party mounting — accepted defer + +Third-party documents initially expose application-specific Chord services. Do not add +`inView`, privacy flags, dynamic mount registries, or client-selected arbitrary docs. +A document is not a security boundary; services deliberately choose what crosses a +process boundary. + +## Layer plan and sign-off gates + +### Layer 0 — cleanup + +- Remove `pico` source/tests: done. +- Remove `pico4` source/tests: done. +- Keep `pico3`: done. +- Run repository check after deletions and document additions: pending. + +### Layer 1 — logical storage + memory reference + +Before code: sign off S3 and S6, retain accepted S1/S2/S7, and approve the exact S13 interfaces. + +Acceptance: + +- atomic mixed commits; +- document creation, bases, current/as-of membership and values; +- family instances and task ownership; +- full task replacement/retirement; +- entry paging and entry -> commit; +- ownership isolation and failure rollback. + +Stop for user review. + +### Layer 2 — SQLite + +Before code: approve concrete schema and durability defaults. + +Acceptance: Layer 1 conformance plus reopen, indexed query plans, latest reclamation, +retired rewindable history, WAL checkpoint behavior, and realistic storage benchmark. + +Stop for user review. + +### Layer 3 — JSONL + +Before code: decide S9 and S11. + +Acceptance: semantic conformance plus physical fault injection at every sidecar, +marker, rewrite, rename, retirement, and reopen boundary. + +Stop for user review. + +### Layer 4 — document manager + Chord source + +Before code: decide S4, S5, D3 scope, and D5. + +Acceptance: ordinary mutations, no duplicate tracker, failed-commit rollback, +contiguous publication, hydration fencing, listener isolation, eviction, and +`watchDoc` snapshot/update/retirement behavior. + +Stop for user review. + +### Layer 5 — tasks + +Acceptance: phase-narrowed full replacements, effect sandwich, runtime-owned memos, +abortable ownership tree, atomic family retirement/outcome handoff, generation/tool +reopen conversion, and no visible-undurable state. + +Stop for user review. + +### Layer 6 — built-in documents and conversation view + +Before code: decide V1, V4, V5, and V6. + +Acceptance: direct built-in writes, fixed structural mount, one publication per commit, +bounded late hydration, crash-visible progress, and no public event dependency. + +Stop for user review. + +### Layer 7 — remaining harness + +Inputs, hooks, tools, system sections, dynamic tools, deferred providers, and product +wiring. Preserve accepted APIs from earlier design work rather than redesigning them +inside the state implementation. diff --git a/packages/agent/docs/pico-v5-slides/index.html b/packages/agent/docs/pico-v5-slides/index.html new file mode 100644 index 000000000..c845fa434 --- /dev/null +++ b/packages/agent/docs/pico-v5-slides/index.html @@ -0,0 +1,742 @@ + + + + + + Pico5 — durable documents + + + +
← → · space · f
+
+ +
+
+

Pico5

+

Durable documents without a whole-session DOM

+

Usage, state → view, and why Lane stops short

+
+
+ +
+
+

The target is not one coding session

+
+

Products

  • coding agent
  • personal assistant
  • cloud agents in Durable Objects
  • Slack bots across years of channels and threads
+

Required invariants

  • survive process death mid-effect
  • fork historical conversation state
  • attach many presentations
  • let facets persist application state without inventing storage
+
+
+
+ +
+
+

Lane solved a real problem

+
+

Execution

Complete operation-state replacements make recovery a switch, not inference.

+

Effects

Durable intent → uncertain effect → durable outcome.

+

Transactions

Entries, values, and lists commit atomically in memory, JSONL, and SQLite.

+
+

Keep those properties. Replace the application-state substrate.

+
+
+ +
+
+

Lane exposes values and lists

+ 
const review = value<ReviewState>("app.review", id);
+const findings = list<Finding>("app.findings", id);
+
+await session.setValue(review, completeNextReview, context);
+await session.appendList(findings, finding, context);
+await session.deleteList(findings, context);
+
    +
  • A scalar is replaced whole.
    • +
  • A list appends or is deleted whole.
    • +
  • Nested changes, splices, checkpoints, and reconstruction belong to the caller.
    • +
+
+
+ +
+
+

Canvas on Lane: choose your own machinery

+
+
+

Store the whole canvas

+ 
const CanvasValue = value<CanvasState>("app.canvas");
+const stored = await session.getValue(CanvasValue, context);
+const next = stored?.value ?? { strokes: [] };
+next.strokes.push(stroke);
+await session.setValue(CanvasValue, next, context);
+

Simple API; complete serialization and replacement for every stroke.

+
+
+

Store stroke deltas

+ 
const StrokeLog = list<StrokeOp>("app.canvas.strokes");
+await session.appendList(StrokeLog, toStrokeOp(stroke), context);
+// define reducer and operation vocabulary
+// define checkpoints
+// coordinate append + compact
+// reconstruct after restart
+

Efficient writes; every facet authors a private state engine.

+
+
+
+
+ +
+
+

Diff review on Lane: durability leaks upward

+ 
const ReviewRecord = (id: string) => value<Review>("app.review", id);
+const activity = env.replicatedState({ review: initialReview });
+
+return regionFor(reviewId).run(signal, async () => {
+  const stored = await session.getValue(ReviewRecord(reviewId), context);
+  const next = validateAndReplace(stored!.value, action);
+  await session.setValue(ReviewRecord(reviewId), next, context);
+  activity.state.review = project(next);
+  activity.publish(context);
+});
+
    +
  • The facet owns locking and read-modify-write correctness.
    • +
  • It reconstructs live state after worker restart.
    • +
  • Persistence and replication are two separately coordinated writes.
    • +
+
+
+ +
+
+

Lane forks transcript, not application state

+ 
branch A:  e1 ─ e2 ─ e3 ─ e4
+                         └──── fork at e3 → branch B
+
+value("plan-mode")       current replacement only
+list("canvas-strokes")   surviving elements only
+
+question: what was either value at e3?    not representable
+

The JSONL bytes may contain old writes. The storage contract does not expose them as history.

+
+
+ +
+
+

The gap is narrow but foundational

+
+

Lane gives facets

  • typed scalar replacement
  • append/delete list
  • atomic storage calls
  • separate volatile ReplicatedState
+

Products need

  • ordinary nested mutation
  • automatic delta + base policy
  • historical document reads
  • commit and replication as one path
+
+

Lane remains a good execution design. It is not enough as the state substrate for products and facets.

+
+
+ +
+
+

A Session is a store of entries, documents, and live task bookmarks.

+

Not one giant DOM. Not a bag of extension-owned repositories.

+
+
+ +
+
+

Four durable primitives

+ + + + + + + + +
PrimitiveUseStorage
EntryTranscript and unbounded historyImmutable, paged
DocumentCurrent application and presentation stateChord ops + bases
Task statePrivate recovery phase/bookmarkComplete replacement, retireable
MemoHook/tool idempotency receiptSmall, first writer wins
+
+
+ +
+
+

Create a durable document

+ 
// Module-level typed declaration token.
+const CanvasDoc = defineDoc<CanvasState>({
+  id: "@app/canvas",
+  scope: "session",
+  history: "latest",
+  initial: () => ({ strokes: [] }),
+});
+
+// Tx resolves CanvasDoc to this Session's mutable draft.
+await session.commit(tx => {
+  tx.doc(CanvasDoc).strokes.push(stroke);
+}, context);
+

CanvasDoc is the token. tx.doc(CanvasDoc) is this transaction's draft.

+
+
+ +
+
+

Expose the same document through Chord

+ 
interface CanvasService {
+  readonly state: ReplicatedState<CanvasState>;
+  addStroke(stroke: Stroke, context: Context): Promise<void>;
+}
+const Canvas = defineService<CanvasService>("@app/canvas");
+
+// Committed source for the CanvasDoc token declared previously.
+const canvasDocument = session.document(CanvasDoc);
+const canvasState = replicatedState.own(canvasDocument);
+
+env.provide(Canvas, {
+  state: canvasState,
+  addStroke: (stroke, context) => session.commit(tx => {
+    tx.doc(CanvasDoc).strokes.push(stroke);
+  }, context),
+});
+
  • No second tracker.
  • No facet-authored delta protocol.
  • No restart reconstruction code.
+
+
+ +
+
+

One mutation path

+
+
mutate
transaction draft
→
+
Chord derives
ops
→
+
storage commits
all or nothing
→
+
publish committed ops
to replicas
+
+

If a client can see it, recovery can reconstruct it.

+
+
+ +
+
+

No volatile task publication

+ 
for await (const event of model.stream(request)) {
+  applyEvent(partial, event);
+
+  if (flushDue()) {
+    await rt.commit(tx => {
+      // LiveDoc is the built-in defineDoc<LiveState> token.
+      tx.doc(LiveDoc, task.conversationId).message = partial;
+    }, context);
+  }
+}
+
  • The UI sees only committed throttle points.
  • A crash loses at most the uncommitted window.
  • There is one tracker baseline: the last durable commit.
+
+
+ +
+
+

Three built-in conversation documents

+ 
interface ConversationView {
+  conversation: ConversationInfo;
+  entries: Entry[];             // bounded active transcript
+  rewindable: RewindableState; // historical fork state
+  sticky: StickyState;         // inbox and present policy
+  live: LiveState;             // turn, streaming, tools, collapse
+}
+ 
rewindable ["s", ["model"], model]
+       view ["s", ["rewindable", "model"], model]
+
+live       ["a", ["message", ...], delta]
+view       ["a", ["live", "message", ...], delta]
+

A fixed structural mount. No config merge, task description pass, or plugin projection.

+
+
+ +
+
+

Task state stays private and small

+ 
const attempt = task.state.attempt + 1;
+
+// Provisional spelling: transition replaces the complete kind state.
+await rt.transition(
+  { phase: "requesting", attempt, requestId },
+  tx => {
+    tx.doc(LiveDoc, task.conversationId).generation = {
+      stage: "requesting", attempt,
+    };
+  },
+  context,
+);
+

Recovery state and presentation state are updated atomically, but are not the same object.

+
+
+ +
+
+

Crash-visible tool progress

+ 
TX 40  task state = { phase: "running" }
+       live.tools[2] = { status: "running", output: "" }
+
+TX 41  live.tools[2].output = "... first 12 KiB ..."
+TX 42  live.tools[2].output = "... latest 50 KiB window ..."
+       PROCESS DIES
+
+restart  task says running
+         view says exactly how far output reached
+         recovery writes interrupted result or safely resumes
+

The partial is useful UI state even when it never enters model context.

+
+
+ +
+
+

Rewind belongs to documents

+ 
commit 100  entry E + rewindable plan.enabled = true
+commit 101  live streaming progress
+commit 102  rewindable plan.enabled = false
+
+fork at E:
+  rewindable → state at commit 100
+  sticky     → fresh defaults
+  live       → fresh defaults
+  tasks      → not copied
+  child      → new document IDs + initial bases
+
  • Bases bound replay distance.
  • Rewindable history remains while ancient forks are supported.
  • The parent and its live work remain untouched.
+
+
+ +
+
+

One atomic Session commit

+ 
// Provisional spelling; all writes below are one transaction.
+await rt.transition({ phase: "tools", calls }, tx => {
+  tx.appendEntry(task.conversationId, finalAssistant);
+
+  const live = tx.doc(LiveDoc, task.conversationId);
+  delete live.message;
+  delete live.generation;
+  live.tools = pendingToolSlots(calls);
+
+  for (const spec of toolTaskSpecs(calls)) tx.createTask(spec);
+}, context);
+

Storage publishes all or none. ConversationView emits one combined operation batch.

+
+
+ +
+
+

Storage chooses delta or base

+ 
interface DocumentChange {
+  id: DocumentId;
+  ops: readonly Op[];
+  value: JsonValue; // borrowed; not serialized normally
+}
+
+if (bytesSinceBase + encodedSize(ops) < threshold)
+  persist(ops);
+else
+  persist([["r", value]]);
+
  • Latest: committed base allows reclaiming the old tail.
  • Rewindable: bases accelerate reads; old addressable segments remain.
  • Chord clients still receive incremental ops.
+
+
+ +
+
+

SQLite: the long-running backend

+
+

Documents

  • metadata row
  • indexed base/delta records
  • range query for current or as-of
  • transactional latest reclamation
+

Tasks + entries

  • one complete row per live task
  • DELETE on retirement
  • immutable paged entry rows
  • all writes in one SQL transaction
+
+

No sidecar publication protocol. No scan of unrelated conversations or documents.

+
+
+ +
+
+

JSONL: one uniform publication rule

+ 
main.jsonl               tables + one publication marker per commit
+task-<id>.jsonl          retireable task state
+doc-<id>.jsonl           one file per document instance
+
+commit                   append every sidecar first
+                         append one compact main marker last
+retire task              publish terminal marker → unlink task sidecar
+rebase latest document   publish base marker → atomic sidecar rewrite
+

Sidecars are a JSONL backend decision. Harness and facet code never see them.

+
+
+ +
+
+

The benchmark changed one decision

+
+

Measured

  • 72 storage runs + 48 SQLite materialization runs validated
  • task sidecars: 1.13 MB → 59 KB after retirement
  • SQLite task deletion + checkpoint: 4.45 MB → 0.30 MB
+

Rejected optimization

  • JSONB replacement 10–18% slower than TEXT
  • SQL JSON patching used 2–4× CPU
  • large canvas DB+WAL: ~80 MB → ~155 MB
+
+

Persist Chord ops and periodic bases. Do not build a second op interpreter in SQL.

+
+
+ +
+
+

Years-long sessions stay bounded in memory

+ 
Session
+├─ entries                         paged rows; unbounded on disk
+├─ conversation A
+│  ├─ active entry window          bounded
+│  ├─ pi.rewindable                loaded while active
+│  ├─ pi.sticky                    loaded while active
+│  └─ pi.live                      loaded only while needed
+├─ conversation B                 evictable while idle
+├─ @app/canvas                    loaded while service is active
+└─ task 91                        one live row; deleted at terminal
+

There is no whole-session materialization, snapshot, or subscriber payload.

+
+
+ +
+
+

What Pico5 does not add yet

+
+
  • generic plugin mounting into ConversationView
  • per-subtree history policies
  • visible-but-undurable updates
  • CRDT or offline writes
+
  • SQL JSON patch translation
  • one whole-Session DOM
  • history pruning semantics
  • public semantic event machinery
  • pico3 config/plugin projections
+
+

The first release mounts only the three built-in conversation documents.

+
+
+ +
+
+

Build the owl one piece at a time

+
+
1
memory + SQLite
storage semantics
→
+
2
JSONL
crash matrix
→
+
3
documents + Chord
narrow proof
→
+
4
tasks
effect sandwich
→
+
5
view + rest
policy last
+
+

Do not implement the scheduler while storage semantics are still moving.

+
+
+ +
+
+

Families: one typed document per task

+ 
const JobOutputDoc = defineDocFamily<JobOutput, JobInput>({
+  id: "pi.job-output",
+  scope: "conversation",
+  history: "latest",
+  owner: "task",
+  initial: input => ({ command: input.command, stdout: "" }),
+});
+
+// task.id selects the family instance owned by this task
+await rt.commit(tx => {
+  tx.doc(JobOutputDoc, task.id).stdout += chunk;
+}, context);
+

Terminal settlement retires the task and every task-owned family instance atomically.

+
+
+ +
+
+

Watch another task's durable output

+ 
const job = await api.task(JobKind, input, { background: true }, context);
+const watch = await api.watchDoc(JobOutputDoc, job.id, context);
+
+consume(watch.value);                    // atomic initial snapshot
+watch.start((_ops, value) => consume(value));
+
+try {
+  return await api.waitForTask(job, context);
+} finally {
+  watch.stop();                           // idempotent, invocation-owned
+}
+
  • Callbacks run off the Session mutation line.
  • Recovery re-enters and captures a fresh snapshot; no durable subscription cursor.
  • Retirement appears as document absence.
+
+
+ +
+
+

Lane made execution durable.

+

Pico5 makes application state durable by default.

+

Ordinary objects → one commit → storage + view

+
+
+ + + + diff --git a/packages/agent/docs/pico-v5.md b/packages/agent/docs/pico-v5.md new file mode 100644 index 000000000..7c885b01e --- /dev/null +++ b/packages/agent/docs/pico-v5.md @@ -0,0 +1,296 @@ +# Pico5: durable documents and views + +> Design direction, not a complete harness specification. This fixes the state, +> view, and storage foundation. Implement and validate it before tasks or policy. + +## 1. Model + +Pico has four durable primitives: + +| primitive | purpose | storage semantics | +|---|---|---| +| entry | immutable transcript/history | append-only, paged | +| document | current JSON application/presentation state | Chord ops plus bases | +| task state | private recovery bookmark | complete replacement, retireable | +| memo | invocation idempotency receipt | small, first-writer-wins, retireable | + +Facet and task authors mutate ordinary tracked JSON objects. They do not define +delta formats, repositories, checkpoints, or restart reconstruction. + +Entries hold unbounded history. A document holds current state; it may be large, +but should not double as an unbounded event archive. Sessions and inactive +documents must be loadable and evictable independently. + +## 2. Documents + +A document has persisted metadata: + +```ts +interface DocumentMetadata { + id: DocumentId; // unique incarnation; never reused + key: string; + scope: "session" | { conversationId: Id }; + history: "latest" | "rewindable"; +} +``` + +History policy is immutable and applies to the whole document: + +- `latest`: only current state is observable; records before a committed base may + be reclaimed. +- `rewindable`: state and document membership are queryable at a historical + Session commit; history remains while forks may address it. + +Creation records metadata and a mandatory initial base, including `{}`. Retirement +hides a document from current enumeration. A latest document may then be deleted. +A rewindable document remains historically enumerable, including if retired later. +Recreating a logical key creates a new ID/incarnation. + +A Session facet may create its own Session or conversation documents, such as a +canvas or diff review. + +## 3. Commits and task state + +A Session has one serialized mutation line. One logical commit may atomically: + +- create, mutate, or retire multiple documents; +- create, replace, or retire task state and memos; +- append entries. + +Document mutation uses transaction-owned drafts. The document manager produces +Chord ops and the candidate current value. Storage commits everything or nothing. +Only after success does the manager adopt drafts and notify views/subscribers. +Failure leaves authoritative values and replicas unchanged. + +> If a state change is visible through Chord, it has already committed durably. + +There is no volatile `publish()`. Generation and tool progress use throttled durable +commits. A crash may lose the current throttle window, but never state a client saw. + +Each task kind defines private JSON recovery state. Phase transitions replace it in +full. The runtime separately owns the task envelope: kind, input, conversation, +ownership/dependencies, memos, and outstanding effect identities. Bulk output lives +in a document, not task state. On reopen, generation and tool kinds convert persisted +partials into aborted/interrupted entries and clear their live presentation state. +The effect sandwich remains: + +```text +durable intent -> external effect -> durable outcome +``` + +Before retirement removes an outcome needed by a waiter, the same commit transfers +it to that waiter, an entry, or another durable receipt. Post-commit publication, +cleanup, or listener failures cannot roll back persistence or suppress later +listeners; they are isolated and reported separately. + +## 4. Conversation view + +Pico initially owns three documents per conversation: + +```text +pi.rewindable history: rewindable; historical configuration +pi.sticky history: latest; queues and long-lived present policy +pi.live history: latest; high-churn turn, generation, tool and collapse state +``` + +Tasks write presentation state directly. Generation writes `live.message`; tools +write `live.tools`; historically inherited configuration lives in `rewindable`. +Private task state is not projected into view. Returning `live` to its initial state +may trigger a tiny base so streamed history can be reclaimed immediately. + +The initial view is a fixed mount: + +```ts +interface ConversationView { + conversation: ConversationInfo; + entries: Entry[]; // bounded active transcript; older entries are paged + rewindable: R; + sticky: S; + live: L; +} +``` + +The mount consumes a complete Session commit and emits one Chord update. It prefixes +document paths and combines them with entry changes: + +```text +["s", ["model"], value] -> ["s", ["rewindable", "model"], value] +["a", ["message"], delta] -> ["a", ["live", "message"], delta] +["r", value] -> ["s", ["live"], value] +``` + +Thus appending a final entry and clearing its preview is one publication. This is a +structural mount, not semantic projection, and owns no second tracker. + +Third-party documents are not mounted initially. Facets expose them through their +own Chord services. Generic plugin mounting is deferred. + +Before view implementation, define a durable bounded active-transcript/head rule and +how it changes atomically with appends. A years-long conversation must not hydrate +its entire transcript. + +## 5. Chord integration + +`DurableDocument` belongs to the harness and need not implement `ReplicatedState`. +A small Chord adapter, provisionally `replicatedState.own(source)`, exposes a durable +document as read-only replicated state without another tracker or re-diff. + +The source provides its immutable committed value, a contiguous revision, and +committed ops. Session commit sequence is not a Chord sequence: unrelated commits +create gaps. Each document source and each conversation mount therefore has its own +contiguous publication revision. + +Hydration must atomically capture matching value/revision and subscribe without a +gap or duplicate. The durable source's internal `publish()` is a no-op because no +visible mutation is pending. Test reentrant subscriptions and isolate listener +failures after persistence. The ConversationView mount consumes Session commits +directly, not independent document notifications. + +A facet can expose a typed durable application document directly. `CanvasDoc` is +the declaration token, `canvasDocument` is the committed source, and `tx.doc(...)` +is a transaction-local draft: + +```ts +const CanvasDoc = defineDoc({ + id: "@app/canvas", scope: "session", history: "latest", + initial: () => ({ strokes: [] }), +}); + +const canvasDocument = session.document(CanvasDoc); +const state = replicatedState.own(canvasDocument); + +async function addStroke(stroke: Stroke, context: Context) { + await session.commit(tx => tx.doc(CanvasDoc).strokes.push(stroke), context); +} +``` + +## 6. Forks + +Every Session commit has a global sequence; each entry records its containing commit. +Forking at entry `E`: + +1. validates `E` through every ancestor cutoff; +2. enumerates rewindable documents existing at `E`'s commit, including ones retired + later; +3. loads each from the newest base at or before that commit plus later deltas; +4. creates a child-owned document ID and initial base for each; +5. copies no live task state. + +A later state-only commit is excluded. Child documents then evolve independently. +Sticky/live documents have no historical lookup. Their simplest initial fork rule is +fresh defaults, never copied queues or running previews. + +Before the fork API ships, decide whether selecting one of several entries appended +in the same commit inherits only that entry prefix or treats the commit as an +indivisible transcript boundary. Document state is the final state of the containing +commit either way. + +## 7. Storage contract and bases + +Storage receives logical metadata and writes, never filenames or table instructions: + +```ts +type DocumentWrite = + | { type: "create"; metadata: DocumentMetadata; value: JsonValue } + | { type: "change"; id: DocumentId; ops: readonly Op[]; value: JsonValue } + | { type: "retire"; id: DocumentId }; +``` + +For a change, `value` is a borrowed candidate snapshot. Storage normally persists +`ops`, but may persist a complete base when tail bytes or record count exceed its +policy: + +```ts +persist(shouldBase(id, ops) ? [["r", value]] : ops); +``` + +It serializes only the chosen representation. Chord still publishes the incremental +ops. A committed base permits reclaiming older latest records; rewindable bases bound +replay but do not authorize deleting addressable history. + +Storage supports current and as-of document enumeration/value lookup and distinguishes +not-yet-created, live, retired, and unavailable history. + +## 8. Backends + +### SQLite + +One SQL transaction is one Session commit. Store immutable entries with commit +sequence, document metadata, indexed base/delta records by document and commit, and +one complete JSON row per live task. Current/as-of loads select the newest applicable +base and an indexed tail; they never scan unrelated documents. Retiring a task deletes +its row. Retired rewindable document history remains queryable. + +Initially store Chord ops/bases directly. Benchmarks found full JSONB replacement +10-18% slower than TEXT and SQL JSON patching 2-4x more CPU with more DB+WAL writes +for a large canvas. SQLite also cannot faithfully map generic middle-array splices or +Chord's UTF-16 truncation. WAL checkpoint, synchronous mode, migrations, backup, and +vacuum are backend concerns. + +### JSONL + +JSONL prioritizes reclaimable task and latest-document storage. It privately chooses +sidecars; the harness does not know they exist: + +```text +main.jsonl entries, lifecycle records, coordinated commits +task-.jsonl live task state +doc-.jsonl independently loadable document records +``` + +Every sidecar record is prepared. A commit appends all affected sidecars first and +one compact main marker listing their references last. Main-only commits are their +own marker. This uniform protocol intentionally accepts one small main record for an +isolated task/document update instead of introducing standalone and coordinated +record modes. + +Recovery applies only records confirmed by a main marker. It removes torn and +unconfirmed tails before reopening for writes. Uncommitted sequences may be reused +only after all records carrying them are removed; committed sequences and document +IDs are never reused. Missing confirmed data is corruption unless a later committed +base or retirement proves it unnecessary. + +An uncertain append error poisons the backend until close/reopen recovery. Appends +loop through the complete newline. After a main marker is complete, cleanup failure +cannot roll back the commit. Reclamation runs only after its base/retirement commits, +using serialized temp-file replacement and rename, then invalidates/reopens any +cached descriptor so later appends target the replacement inode. + +Without per-commit `fsync`, ordered writes target ordinary process-crash consistency; +power/host/filesystem failure may lose or reorder acknowledged commits. Durable mode +flushes sidecars before the main marker. File descriptors are bounded by per-write +open or an LRU. + +## 9. Deferred + +Do not initially add per-subtree history, volatile publication, a public semantic +event channel, generic plugin mounting, SQL JSON patch translation, CRDT/offline +writes, whole-Session DOM materialization, arbitrary atomic presentation across +third-party services, sticky copy options, history-policy changes/pruning, or pico3's +config/plugin projections. UI status is durable document state; if transient history +is needed, add a bounded field to `live` with explicit retention. + +## 10. Implementation order + +1. **Storage semantics: memory + SQLite.** Create two initial-base documents; + atomically change both, replace a task, and append an entry; force latest and + rewindable bases; test current/as-of membership and values, retirement/recreation, + entry-to-commit mapping, reopen, and rollback. +2. **JSONL.** Implement uniform sidecar preparation/main-marker publication, + poisoned writes, committed-only reclamation, and fault injection at every physical + boundary. +3. **Document manager + narrow Chord proof.** Prove draft rollback without + per-success full cloning, one tracker per loaded doc, eviction/forks, contiguous + revisions, hydration fencing, listener isolation, and `watchDoc` snapshot/update/ + retirement behavior. +4. **Tasks.** Add runtime envelopes, full-replacement kind state, phases, memos, + effect sandwich, dependency handoff, abort, retirement, durable built-in progress, + and reopen conversion of persisted partials. +5. **Conversation view.** First settle active-transcript and multi-entry fork + boundaries; then mount rewindable, sticky, and live with one publication per + Session commit. +6. **The rest.** Inputs, hooks, tools, system sections, dynamic tools, notices if + required, and product wiring. + +Storage is accepted before task implementation. All backends share one semantic +conformance suite; JSONL additionally has a physical crash/failure matrix. diff --git a/packages/agent/src/harness/pico/addresses.ts b/packages/agent/src/harness/pico/addresses.ts deleted file mode 100644 index dab63da24..000000000 --- a/packages/agent/src/harness/pico/addresses.ts +++ /dev/null @@ -1,273 +0,0 @@ -import type { Id, JsonValue } from "./core.ts"; - -export type Scope = - | { readonly type: "session" } - | { readonly type: "conversation"; readonly conversationId: Id } - | { readonly type: "task"; readonly taskId: Id } - | { readonly type: "shared"; readonly id: Id }; - -export type PublicScope = Exclude; -export type StickyScope = Extract; -export type ConversationScope = Extract; - -declare const addressType: unique symbol; - -export interface Address { - readonly scope: Scope; - readonly namespace: string; - readonly key?: string; - readonly kind: "value" | "list"; - readonly rewind: boolean; - readonly [addressType]?: T; -} - -export interface Value extends Address { - readonly kind: "value"; - readonly default?: T; -} - -export interface List extends Address { - readonly kind: "list"; -} - -export interface Element { - readonly id: Id; - readonly value: T; -} - -export interface UnboundValue { - readonly namespace: string; - readonly key?: string; - readonly rewind: R; - readonly default?: T; - readonly [addressType]?: T; - bind(conversationId: Id): Value; -} - -export interface UnboundList { - readonly namespace: string; - readonly key?: string; - readonly rewind: R; - readonly [addressType]?: T; - bind(conversationId: Id): List; -} - -export type ConversationValueDefinition = UnboundValue; -export type ConversationListDefinition = UnboundList; -export type PayloadOf = D extends Address - ? BoundPayload - : D extends UnboundValue - ? DefinitionPayload - : never; -export type ResolvedValue = D extends { readonly default: infer V } ? V : PayloadOf | undefined; - -type DefaultedUnboundValue = Omit, "bind" | "default"> & { - readonly default: T; - bind(conversationId: Id): Value & { readonly default: T }; -}; - -type StickyOptions = { readonly key?: string; readonly default?: T }; -type DefaultedStickyOptions = { readonly key?: string; readonly default: T }; -type ConversationOptions = { - readonly key?: string; - readonly rewind: boolean; - readonly default?: T; -}; -type DefaultedConversationOptions = { - readonly key?: string; - readonly rewind: boolean; - readonly default: T; -}; -type InternalOptions = StickyOptions & { readonly rewind?: boolean }; - -function assertPublicNamespace(namespace: string): void { - if (namespace.startsWith("pi.")) throw new Error(`Namespace ${namespace} is reserved`); -} - -function assertPublicScope(scope: Scope): asserts scope is PublicScope { - if (scope.type === "shared") throw new Error("Shared scope is internal"); -} - -function value(scope: Scope, namespace: string, options?: InternalOptions): Value { - return Object.freeze({ - ...options, - scope, - namespace, - kind: "value" as const, - rewind: options?.rewind ?? false, - }); -} - -function list(scope: Scope, namespace: string, options?: InternalOptions): List { - return Object.freeze({ - ...options, - scope, - namespace, - kind: "list" as const, - rewind: options?.rewind ?? false, - }); -} - -export function defineValue( - scope: StickyScope, - namespace: string, - options: DefaultedStickyOptions, -): Value & { readonly default: T }; -export function defineValue( - scope: StickyScope, - namespace: string, - options?: { readonly key?: string }, -): Value; -export function defineValue( - scope: ConversationScope, - namespace: string, - options: DefaultedConversationOptions, -): Value & { readonly default: T }; -export function defineValue( - scope: ConversationScope, - namespace: string, - options: { readonly key?: string; readonly rewind: boolean }, -): Value; -export function defineValue( - scope: Scope, - namespace: string, - options?: InternalOptions, -): Value { - assertPublicScope(scope); - assertPublicNamespace(namespace); - return value(scope, namespace, options); -} - -export function defineList( - scope: StickyScope, - namespace: string, - options?: { readonly key?: string }, -): List; -export function defineList( - scope: ConversationScope, - namespace: string, - options: Omit, "default">, -): List; -export function defineList( - scope: Scope, - namespace: string, - options?: InternalOptions, -): List { - assertPublicScope(scope); - assertPublicNamespace(namespace); - return list(scope, namespace, options); -} - -export function conversationValue( - namespace: string, - options: { readonly key?: string; readonly rewind: true; readonly default: T }, -): DefaultedUnboundValue; -export function conversationValue( - namespace: string, - options: { readonly key?: string; readonly rewind: false; readonly default: T }, -): DefaultedUnboundValue; -export function conversationValue( - namespace: string, - options: { readonly key?: string; readonly rewind: true }, -): UnboundValue; -export function conversationValue( - namespace: string, - options: { readonly key?: string; readonly rewind: false }, -): UnboundValue; -export function conversationValue( - namespace: string, - options: ConversationOptions, -): UnboundValue { - assertPublicNamespace(namespace); - return unboundValue(namespace, options); -} - -export function conversationList( - namespace: string, - options: { readonly key?: string; readonly rewind: true }, -): UnboundList; -export function conversationList( - namespace: string, - options: { readonly key?: string; readonly rewind: false }, -): UnboundList; -export function conversationList( - namespace: string, - options: Omit, "default">, -): UnboundList { - assertPublicNamespace(namespace); - return unboundList(namespace, options); -} - -export function sessionValue( - namespace: string, - options: { readonly key?: string; readonly default: T }, -): Value & { readonly default: T }; -export function sessionValue(namespace: string, options?: { readonly key?: string }): Value; -export function sessionValue(namespace: string, options?: StickyOptions): Value { - return defineValue({ type: "session" }, namespace, options); -} - -/** Internal constructor for fixed `pi.*` definitions and shared task output state. */ -export function defineInternalValue( - scope: Scope, - namespace: string, - options?: InternalOptions, -): Value { - return value(scope, namespace, options); -} - -/** Internal constructor for fixed `pi.*` definitions and shared task output state. */ -export function defineInternalList( - scope: Scope, - namespace: string, - options?: InternalOptions, -): List { - return list(scope, namespace, options); -} - -/** Internal constructor for fixed conversation configuration definitions. */ -export function internalConversationValue( - namespace: string, - options: { readonly key?: string; readonly rewind: R; readonly default: T }, -): DefaultedUnboundValue; -export function internalConversationValue( - namespace: string, - options: { readonly key?: string; readonly rewind: R }, -): UnboundValue; -export function internalConversationValue( - namespace: string, - options: { readonly key?: string; readonly rewind: R; readonly default?: T }, -): UnboundValue { - return unboundValue(namespace, options); -} - -function unboundValue( - namespace: string, - options: { readonly key?: string; readonly rewind: R; readonly default?: T }, -): UnboundValue { - return Object.freeze({ - ...options, - namespace, - bind: (conversationId: Id) => - value({ type: "conversation", conversationId }, namespace, { - key: options.key, - rewind: options.rewind, - default: options.default, - }), - }); -} - -function unboundList( - namespace: string, - options: { readonly key?: string; readonly rewind: R }, -): UnboundList { - return Object.freeze({ - ...options, - namespace, - bind: (conversationId: Id) => - list({ type: "conversation", conversationId }, namespace, { - key: options.key, - rewind: options.rewind, - }), - }); -} diff --git a/packages/agent/src/harness/pico/core.ts b/packages/agent/src/harness/pico/core.ts deleted file mode 100644 index 2467ccd63..000000000 --- a/packages/agent/src/harness/pico/core.ts +++ /dev/null @@ -1,12 +0,0 @@ -import type { JsonRepresentation } from "@earendil-works/chord"; - -export type JsonPrimitive = null | boolean | number | string; -export type JsonValue = JsonPrimitive | JsonObject | readonly JsonValue[]; -export type Stored = JsonRepresentation; - -export interface JsonObject { - readonly [key: string]: JsonValue; -} - -export type Id = number; -export type Seq = number; diff --git a/packages/agent/src/harness/pico/entries.ts b/packages/agent/src/harness/pico/entries.ts deleted file mode 100644 index 843783fd4..000000000 --- a/packages/agent/src/harness/pico/entries.ts +++ /dev/null @@ -1,106 +0,0 @@ -import type { AssistantMessage, Message, ToolResultMessage, Usage, UserMessage } from "@earendil-works/pi-ai"; -import type { Id, JsonValue, Stored } from "./core.ts"; - -export type ContextEdit = - | { readonly target: Id; readonly action: "omit" } - | { readonly target: Id; readonly action: "replace"; readonly messages: readonly Stored[] }; - -export interface Entry { - readonly id: Id; - readonly conversationId: Id; - readonly kind: string; - readonly byTaskId?: Id; - readonly data?: JsonValue; - readonly model?: readonly Stored[]; - readonly head?: Id; - readonly edits?: readonly ContextEdit[]; -} - -export type EntryIdentity = Pick; -export type EntryBase = EntryIdentity; - -export interface EntryData { - readonly data: D; -} - -export interface ModelProjection { - readonly model: readonly Stored[]; -} - -export interface ContextHead { - readonly head: Id; -} - -export interface ContextEdits { - readonly edits: readonly ContextEdit[]; -} - -export type EntryInput = Omit & - (E extends { readonly head: Id } ? { readonly head: Id | "self" } : { readonly head?: never }); - -export interface EntryKind { - readonly kind: string; - is(entry: Entry | undefined): entry is E; -} - -export function defineEntry(kind: string): EntryKind { - return Object.freeze({ - kind, - is(entry: Entry | undefined): entry is E { - return entry?.kind === kind; - }, - }); -} - -export interface Conversation { - readonly id: Id; - readonly parent?: { readonly conversationId: Id; readonly at: Id }; - readonly owner?: Id; -} - -export type AssistantEntryData = { readonly attempt: number }; -export type UsageEntryData = { readonly attempt: number; readonly usage?: Stored; readonly error: string }; -export type ToolControl = { - readonly terminate?: true; - readonly handoff?: string; - readonly addTools?: readonly string[]; -}; -export type ToolDiagnostic = { - readonly severity: "info" | "warn" | "error"; - readonly message: string; - readonly code?: string; -}; -export type ToolUsage = { readonly [key: string]: number }; -export type ToolResultData = { - readonly details?: JsonValue; - readonly usage?: ToolUsage; - readonly diagnostics?: readonly ToolDiagnostic[]; - readonly control?: ToolControl; - readonly truncated?: { readonly bytes: number; readonly lines: number }; -}; -export type SectionRecord = - | { readonly key: string; readonly action: "set"; readonly value: JsonValue; readonly rendered: string } - | { readonly key: string; readonly action: "remove" }; -export type SystemEntryData = { readonly baseline?: true; readonly sections: readonly SectionRecord[] }; - -export type UserEntry = EntryBase & { - readonly model: readonly [Stored]; - readonly data?: { readonly continuation: true; readonly from: Id }; -}; -export type AssistantEntry = EntryBase & { - readonly model: readonly [Stored]; - readonly data?: AssistantEntryData; -}; -export type ToolResultEntry = EntryBase & { - readonly model: readonly [Stored]; - readonly data: ToolResultData; -}; -export type NoticeEntry = EntryBase & { readonly model: readonly [Stored]; readonly data?: JsonValue }; -export type UsageEntry = EntryBase & { readonly data: UsageEntryData }; -export type SummaryEntry = EntryBase & { - readonly model: readonly [Stored]; - readonly data: { readonly through: Id }; - readonly head: Id; -}; -export type HandoffEntry = EntryBase & { readonly model: readonly [Stored]; readonly head: Id }; -export type ResetEntry = EntryBase & { readonly head: Id }; diff --git a/packages/agent/src/harness/pico/hooks.ts b/packages/agent/src/harness/pico/hooks.ts deleted file mode 100644 index 9cb20fafd..000000000 --- a/packages/agent/src/harness/pico/hooks.ts +++ /dev/null @@ -1,62 +0,0 @@ -import type { Context } from "@earendil-works/chord"; -import type { Id } from "./core.ts"; - -declare const hookIn: unique symbol; -declare const hookOut: unique symbol; - -export type Fold = "collect" | "first" | "chain"; - -export interface HookPoint { - readonly fold: F; - readonly onThrow: "skip" | "abort"; - readonly [hookIn]?: In; - readonly [hookOut]?: Out; -} - -export function defineHookPoint(options: { - readonly fold: "collect"; - readonly onThrow: "skip" | "abort"; -}): HookPoint; -export function defineHookPoint(options: { - readonly fold: "first"; - readonly onThrow: "skip" | "abort"; -}): HookPoint; -export function defineHookPoint>(options: { - readonly fold: "chain"; - readonly onThrow: "skip" | "abort"; -}): HookPoint; -export function defineHookPoint(options: { - readonly fold: Fold; - readonly onThrow: "skip" | "abort"; -}): HookPoint { - return Object.freeze(options); -} - -export type HookPoints = Record>; -export type EmptyHookPoints = Record; -export type HookIn

= P extends HookPoint ? In : never; -export type HookOut

= P extends HookPoint ? Out : never; - -// biome-ignore lint/suspicious/noConfusingVoidType: void allows handlers with no return value. -type HookHandlerResult

= HookOut

| void; -export type HookHandler

= ( - input: HookIn

, - info: HookInfo, - ctx: Context, -) => HookHandlerResult

| Promise>; - -export interface HookInfo { - readonly kind: string; - readonly taskId: Id; - readonly conversationId: Id; -} - -export interface HookRunner { - run

(point: P, input: HookIn

, ctx: Context): Promise>; -} - -export type HookResult

= P extends { fold: "collect" } - ? { readonly outputs: readonly HookOut

[]; readonly threw?: unknown } - : P extends { fold: "first" } - ? { readonly output?: HookOut

; readonly threw?: unknown } - : { readonly output: HookIn

; readonly threw?: unknown }; diff --git a/packages/agent/src/harness/pico/index.ts b/packages/agent/src/harness/pico/index.ts deleted file mode 100644 index 1f0f85d01..000000000 --- a/packages/agent/src/harness/pico/index.ts +++ /dev/null @@ -1,154 +0,0 @@ -export type { - Address, - ConversationListDefinition, - ConversationScope, - ConversationValueDefinition, - Element, - List, - PayloadOf, - PublicScope, - ResolvedValue, - Scope, - StickyScope, - UnboundList, - UnboundValue, - Value, -} from "./addresses.ts"; -export { - conversationList, - conversationValue, - defineList, - defineValue, - sessionValue, -} from "./addresses.ts"; -export type { Id, JsonObject, JsonPrimitive, JsonValue, Seq, Stored } from "./core.ts"; -export type { - AssistantEntry, - AssistantEntryData, - ContextEdit, - ContextEdits, - ContextHead, - Conversation, - Entry, - EntryBase, - EntryData, - EntryIdentity, - EntryInput, - EntryKind, - HandoffEntry, - ModelProjection, - NoticeEntry, - ResetEntry, - SectionRecord, - SummaryEntry, - SystemEntryData, - ToolControl, - ToolDiagnostic, - ToolResultData, - ToolResultEntry, - ToolUsage, - UsageEntry, - UsageEntryData, - UserEntry, -} from "./entries.ts"; -export { defineEntry } from "./entries.ts"; -export type { - Fold, - HookHandler, - HookIn, - HookInfo, - HookOut, - HookPoint, - HookPoints, - HookResult, - HookRunner, -} from "./hooks.ts"; -export { defineHookPoint } from "./hooks.ts"; -export { MemoryStorage } from "./memory-storage.ts"; -export type { - AbortClosure, - AbortFinalTx, - AbortTaskConversation, - AbortTaskRuntime, - AbortTx, - Acceptance, - BaseTaskTx, - BoundPublicState, - BoundTxState, - ConversationCreateSpec, - ConversationTx, - FinalOutput, - FinalTx, - GenerationOutput, - HostTx, - InputHandle, - InputOutcome, - PublicList, - PublicValue, - QueuedInput, - ReadonlyTaskOutput, - RuntimeOutput, - ScratchReader, - SendInput, - StickyPublicValue, - StoredEntryDraft, - StoredInputResult, - TaskConversation, - TaskOutput, - TaskRuntime, - TerminalClosure, - TerminalInputOutcome, - ToolInput, - ToolOutputState, - TxList, - TxReaders, - TxState, - TxValue, - UserInput, -} from "./runtime.ts"; -export { Closed, Faulted, NestedLineOperation, ReadAfterWrite } from "./session.ts"; -export type { - ConversationScan, - EntryScan, - NewTask, - Page, - PageQuery, - StateWrite, - Storage, - TaskScan, - Write, -} from "./storage.ts"; -export { InvalidHistoryPosition, OutputRetired, ScratchRetired } from "./storage.ts"; -export type { - AbortedOf, - AnyDefinedKind, - CheckpointOf, - Completion, - ConfigBundle, - DistributiveOmit, - ExactJsonInput, - FailureOf, - InputOf, - NoCheckpoint, - NoExtra, - OrdinaryKind, - OutputOf, - OutputState, - PayloadsOf, - ResultOf, - RunningTask, - StoredTaskOutputRef, - Task, - TaskBase, - TaskCheckpoint, - TaskKind, - TaskKindDefinition, - TaskKindFactory, - TaskOf, - TaskOutcome, - TaskOutputDefinition, - TaskOutputField, - TaskOutputKind, - TaskOutputRef, -} from "./tasks.ts"; -export { defineTask, defineTaskOutput } from "./tasks.ts"; diff --git a/packages/agent/src/harness/pico/memory-storage.ts b/packages/agent/src/harness/pico/memory-storage.ts deleted file mode 100644 index a3b126702..000000000 --- a/packages/agent/src/harness/pico/memory-storage.ts +++ /dev/null @@ -1,578 +0,0 @@ -import type { Context } from "@earendil-works/chord"; -import type { Address, Element, List, Value } from "./addresses.ts"; -import type { Id, JsonValue, Seq } from "./core.ts"; -import type { Conversation, Entry } from "./entries.ts"; -import { - type ConversationScan, - type EntryScan, - InvalidHistoryPosition, - OutputRetired, - type Page, - type PageQuery, - ScratchRetired, - type Storage, - type StoredConversation, - type TaskScan, - type Write, -} from "./storage.ts"; -import type { Task } from "./tasks.ts"; - -type ValueHistoryWrite = - | { readonly seq: Seq; readonly type: "set"; readonly value: JsonValue } - | { readonly seq: Seq; readonly type: "delete" }; - -type ListHistoryWrite = - | { readonly seq: Seq; readonly type: "append"; readonly element: Element } - | { readonly seq: Seq; readonly type: "remove"; readonly elementId: Id } - | { readonly seq: Seq; readonly type: "clear" }; - -type StoredState = - | { readonly kind: "value"; readonly rewind: false; value?: JsonValue } - | { - readonly kind: "list"; - readonly rewind: false; - readonly elements: Map>; - readonly known: Set; - } - | { readonly kind: "value"; readonly rewind: true; readonly writes: ValueHistoryWrite[] } - | { readonly kind: "list"; readonly rewind: true; readonly writes: ListHistoryWrite[] }; - -type RewindListState = Extract; - -interface AddressState { - value?: Extract; - list?: Extract; -} - -interface NamespaceState { - unkeyed?: AddressState; - readonly keyed: Map; -} - -type ScopeState = Map; - -interface CappedListState { - readonly state: RewindListState | undefined; - readonly cap: Seq | undefined; -} - -interface StoredEntry { - readonly seq: Seq; - readonly entry: Entry; -} - -interface ProspectiveWrites { - readonly conversations: Map; - readonly entries: Map; - readonly listAppends: { readonly seq: Seq; readonly address: List; readonly elementId: Id }[]; -} - -function rewindable( - address: Address, -): address is Address & { readonly scope: Extract; readonly rewind: true } { - return address.scope.type === "conversation" && address.rewind; -} - -function scopeId(address: Address): Id | undefined { - if (address.scope.type === "conversation") return address.scope.conversationId; - if (address.scope.type === "task") return address.scope.taskId; - if (address.scope.type === "shared") return address.scope.id; - return undefined; -} - -function page( - values: Iterable, - query: PageQuery, - id: (value: T) => Id, - matches: (value: T) => boolean, - ascending: boolean, -): Page { - const items: T[] = []; - let more = false; - const ordered = [...values].sort((left, right) => (ascending ? id(left) - id(right) : id(right) - id(left))); - for (const value of ordered) { - const valueId = id(value); - if ( - !matches(value) || - (query.cursor !== undefined && (ascending ? valueId <= query.cursor : valueId >= query.cursor)) - ) { - continue; - } - if (items.length === query.limit) { - more = true; - break; - } - items.push(value); - } - return { items, ...(more && items.length > 0 ? { next: id(items.at(-1)!) } : {}) }; -} - -export class MemoryStorage implements Storage { - private nextObjectId: Id = 1; - private committedIdHighWater: Id = 0; - private nextSeq: Seq = 1; - private readonly conversations = new Map(); - private readonly entries = new Map(); - private readonly entriesByConversation = new Map(); - private readonly tasks = new Map(); - private readonly outputReferences = new Map(); - private readonly sessionState: ScopeState = new Map(); - private readonly conversationStickyState = new Map(); - private readonly conversationRewindableState = new Map(); - private readonly taskState = new Map(); - private readonly sharedState = new Map(); - - static create(): MemoryStorage { - return new MemoryStorage(); - } - - nextId(): Id { - return this.nextObjectId++; - } - - async commit(writes: readonly Write[], _ctx: Context): Promise { - this.validateListRemovals(writes); - const seqs = writes.map((_, index) => this.nextSeq + index); - const affectedOutputs = new Set(); - let batchMaxId = this.committedIdHighWater; - for (const write of writes) { - if (write.type === "conversation.create") batchMaxId = Math.max(batchMaxId, write.conversation.id); - else if (write.type === "entry.append") batchMaxId = Math.max(batchMaxId, write.entry.id); - else if (write.type === "task.create") batchMaxId = Math.max(batchMaxId, write.task.id); - else if (write.type === "list.append") batchMaxId = Math.max(batchMaxId, write.element.id); - if (write.type !== "task.create" && write.type !== "task.set") continue; - const currentOutput = this.tasks.get(write.task.id)?.output?.id; - if (currentOutput !== undefined) affectedOutputs.add(currentOutput); - if (write.task.output !== undefined) affectedOutputs.add(write.task.output.id); - } - for (let index = 0; index < writes.length; index++) this.apply(writes[index]!, seqs[index]!); - for (const write of writes) { - if (write.type === "task.set" && write.task.status === "terminal") this.taskState.delete(write.task.id); - } - for (const id of affectedOutputs) { - if (!this.outputReferences.has(id)) this.sharedState.delete(id); - } - this.committedIdHighWater = batchMaxId; - this.nextObjectId = Math.max(this.nextObjectId, batchMaxId + 1); - this.nextSeq += writes.length; - return seqs; - } - - async close(_ctx: Context): Promise { - this.nextObjectId = this.committedIdHighWater + 1; - } - - async getConversations(ids: readonly Id[], _ctx: Context): Promise> { - const result = new Map(); - for (const id of ids) { - const conversation = this.conversations.get(id); - if (conversation !== undefined) result.set(id, this.publicConversation(conversation)); - } - return result; - } - - async scanConversations(query: ConversationScan, _ctx: Context): Promise> { - return page( - [...this.conversations.values()].map((conversation) => this.publicConversation(conversation)), - query, - (conversation) => conversation.id, - (conversation) => - (query.parent === undefined || conversation.parent?.conversationId === query.parent) && - (query.owner === undefined || conversation.owner === query.owner), - true, - ); - } - - async getEntries(ids: readonly Id[], _ctx: Context): Promise> { - const result = new Map(); - for (const id of ids) { - const stored = this.entries.get(id); - if (stored !== undefined) result.set(id, stored.entry); - } - return result; - } - - async scanEntries(query: EntryScan, _ctx: Context): Promise> { - const cursor = query.cursor === undefined ? undefined : this.historyPosition(query.conversationId, query.cursor); - const items: Entry[] = []; - let more = false; - for (const stored of this.visibleEntries(query.conversationId, query.through)) { - if (cursor !== undefined && stored.seq >= cursor) continue; - if (query.kind !== undefined && stored.entry.kind !== query.kind) continue; - if (items.length === query.limit) { - more = true; - break; - } - items.push(stored.entry); - } - return { items, ...(more && items.length > 0 ? { next: items.at(-1)!.id } : {}) }; - } - - async newestHead(conversationId: Id, at: Id, _ctx: Context): Promise { - for (const stored of this.visibleEntries(conversationId, at)) { - if (stored.entry.head !== undefined) return stored.entry; - } - return undefined; - } - - async getTasks(ids: readonly Id[], _ctx: Context): Promise> { - const result = new Map(); - for (const id of ids) { - const task = this.tasks.get(id); - if (task !== undefined) result.set(id, task); - } - return result; - } - - async scanTasks(query: TaskScan, _ctx: Context): Promise> { - return page( - this.tasks.values(), - query, - (task) => task.id, - (task) => - (query.conversationIds === undefined || query.conversationIds.includes(task.conversationId)) && - (query.statuses === undefined || query.statuses.includes(task.status)) && - (query.kind === undefined || task.kind === query.kind) && - (query.abort === undefined || (task.abort === true) === query.abort) && - (query.outputId === undefined || task.output?.id === query.outputId), - true, - ); - } - - async getValue(address: Value, at: Id | undefined, _ctx: Context): Promise { - if (at !== undefined) { - if (!rewindable(address)) throw new InvalidHistoryPosition(at); - this.assertScopeReadable(address); - return this.rewindValue(address, this.historyPosition(address.scope.conversationId, at)); - } - this.assertScopeReadable(address); - if (rewindable(address)) return this.rewindValue(address, undefined); - const state = this.storedState(address); - return state?.kind === "value" && !state.rewind ? (state.value as T | undefined) : undefined; - } - - async readList( - address: List, - at: Id | undefined, - _ctx: Context, - ): Promise[]> { - if (at !== undefined) { - if (!rewindable(address)) throw new InvalidHistoryPosition(at); - this.assertScopeReadable(address); - return this.rewindListElements(address, this.historyPosition(address.scope.conversationId, at)); - } - this.assertScopeReadable(address); - if (rewindable(address)) return this.rewindListElements(address, undefined); - const state = this.storedState(address); - if (state?.kind !== "list" || state.rewind) return []; - return [...(state.elements.values() as Iterable>)].sort((left, right) => left.id - right.id); - } - - private historyPosition(conversationId: Id, at: Id, prospective?: ProspectiveWrites): Seq { - const stored = prospective?.entries.get(at) ?? this.entries.get(at); - if (stored === undefined) throw new InvalidHistoryPosition(at); - - let currentId = conversationId; - let cap: Seq | undefined; - while (true) { - if (stored.entry.conversationId === currentId && (cap === undefined || stored.seq <= cap)) return stored.seq; - const conversation = prospective?.conversations.get(currentId) ?? this.conversations.get(currentId); - if (conversation?.parent === undefined) throw new InvalidHistoryPosition(at); - const parentAtSeq = this.historyPosition( - conversation.parent.conversationId, - conversation.parent.at, - prospective, - ); - cap = Math.min(cap ?? parentAtSeq, parentAtSeq); - currentId = conversation.parent.conversationId; - } - } - - private rewindValue(address: Value, cap: Seq | undefined): T | undefined { - let conversationId = address.scope.type === "conversation" ? address.scope.conversationId : undefined; - while (conversationId !== undefined) { - const state = this.conversationStoredState(conversationId, address); - if (state?.kind === "value" && state.rewind) { - for (let index = state.writes.length - 1; index >= 0; index--) { - const write = state.writes[index]!; - if (cap !== undefined && write.seq > cap) continue; - return write.type === "set" ? (write.value as T) : undefined; - } - } - const conversation = this.conversations.get(conversationId); - if (conversation?.parent === undefined) return undefined; - const parentAtSeq = this.historyPosition(conversation.parent.conversationId, conversation.parent.at); - cap = Math.min(cap ?? parentAtSeq, parentAtSeq); - conversationId = conversation.parent.conversationId; - } - return undefined; - } - - private rewindListElements( - address: List & { readonly scope: Extract }, - cap: Seq | undefined, - ): readonly Element[] { - const ancestry: CappedListState[] = []; - let conversationId = address.scope.conversationId; - while (true) { - const stored = this.conversationStoredState(conversationId, address); - ancestry.push({ state: stored?.kind === "list" && stored.rewind ? stored : undefined, cap }); - const conversation = this.conversations.get(conversationId); - if (conversation?.parent === undefined) break; - const parentAtSeq = this.historyPosition(conversation.parent.conversationId, conversation.parent.at); - cap = Math.min(cap ?? parentAtSeq, parentAtSeq); - conversationId = conversation.parent.conversationId; - } - - const elements = new Map>(); - for (const segment of ancestry.reverse()) { - for (const write of segment.state?.writes ?? []) { - if (segment.cap !== undefined && write.seq > segment.cap) continue; - if (write.type === "append") elements.set(write.element.id, write.element as Element); - else if (write.type === "remove") elements.delete(write.elementId); - else elements.clear(); - } - } - return [...elements.values()].sort((left, right) => left.id - right.id); - } - - private *visibleEntries(conversationId: Id, through: Id | undefined): Iterable { - let conversation = this.conversations.get(conversationId); - let cap = through === undefined ? undefined : this.historyPosition(conversationId, through); - while (conversation !== undefined) { - const entries = this.entriesByConversation.get(conversation.id) ?? []; - for (let index = entries.length - 1; index >= 0; index--) { - const stored = entries[index]!; - if (cap === undefined || stored.seq <= cap) yield stored; - } - if (conversation.parent === undefined) return; - const parentAtSeq = this.historyPosition(conversation.parent.conversationId, conversation.parent.at); - cap = Math.min(cap ?? parentAtSeq, parentAtSeq); - conversation = this.conversations.get(conversation.parent.conversationId); - } - } - - private validateListRemovals(writes: readonly Write[]): void { - const prospective: ProspectiveWrites = { - conversations: new Map(), - entries: new Map(), - listAppends: [], - }; - for (let index = 0; index < writes.length; index++) { - const write = writes[index]!; - const seq = this.nextSeq + index; - if (write.type === "conversation.create") { - prospective.conversations.set(write.conversation.id, write.conversation); - } else if (write.type === "entry.append") { - prospective.entries.set(write.entry.id, { seq, entry: write.entry }); - } else if (write.type === "list.append") { - prospective.listAppends.push({ seq, address: write.address, elementId: write.element.id }); - } else if ( - write.type === "list.remove" && - !this.listKnowsElement(write.address, write.elementId, prospective) - ) { - throw new Error(`List element ${write.elementId} does not belong to this list lineage`); - } - } - } - - private listKnowsElement(address: List, elementId: Id, prospective?: ProspectiveWrites): boolean { - if (!rewindable(address)) { - const state = this.storedState(address); - return ( - (state?.kind === "list" && !state.rewind && state.known.has(elementId)) || - (prospective?.listAppends.some( - (append) => append.elementId === elementId && this.sameAddress(append.address, address), - ) ?? - false) - ); - } - let conversationId = address.scope.conversationId; - let cap: Seq | undefined; - while (true) { - const state = this.conversationStoredState(conversationId, address); - const appendedProspectively = prospective?.listAppends.some( - (append) => - append.elementId === elementId && - append.address.scope.type === "conversation" && - append.address.scope.conversationId === conversationId && - append.address.rewind === address.rewind && - append.address.namespace === address.namespace && - append.address.key === address.key && - (cap === undefined || append.seq <= cap), - ); - if ( - appendedProspectively || - (state?.kind === "list" && - state.rewind && - state.writes.some( - (write) => - write.type === "append" && - write.element.id === elementId && - (cap === undefined || write.seq <= cap), - )) - ) { - return true; - } - const conversation = prospective?.conversations.get(conversationId) ?? this.conversations.get(conversationId); - if (conversation?.parent === undefined) return false; - const parentAtSeq = this.historyPosition( - conversation.parent.conversationId, - conversation.parent.at, - prospective, - ); - cap = Math.min(cap ?? parentAtSeq, parentAtSeq); - conversationId = conversation.parent.conversationId; - } - } - - private sameAddress(left: Address, right: Address): boolean { - if ( - left.kind !== right.kind || - left.rewind !== right.rewind || - left.namespace !== right.namespace || - left.key !== right.key || - left.scope.type !== right.scope.type - ) { - return false; - } - if (left.scope.type === "session") return true; - if (left.scope.type === "conversation" && right.scope.type === "conversation") { - return left.scope.conversationId === right.scope.conversationId; - } - if (left.scope.type === "task" && right.scope.type === "task") return left.scope.taskId === right.scope.taskId; - return left.scope.type === "shared" && right.scope.type === "shared" && left.scope.id === right.scope.id; - } - - private publicConversation(conversation: StoredConversation): Conversation { - return { - id: conversation.id, - ...(conversation.parent === undefined ? {} : { parent: conversation.parent }), - ...(conversation.owner === undefined ? {} : { owner: conversation.owner }), - }; - } - - private assertScopeReadable(address: Address): void { - if (address.scope.type === "task") { - const task = this.tasks.get(address.scope.taskId); - if (task === undefined || task.status === "terminal") throw new ScratchRetired(address.scope.taskId); - } else if (address.scope.type === "shared" && !this.outputReferences.has(address.scope.id)) { - throw new OutputRetired(address.scope.id); - } - } - - private apply(write: Write, seq: Seq): void { - if (write.type === "conversation.create") { - this.conversations.set(write.conversation.id, write.conversation); - } else if (write.type === "entry.append") { - const entry = write.entry; - const stored = { seq, entry }; - this.entries.set(entry.id, stored); - const entries = this.entriesByConversation.get(entry.conversationId); - if (entries === undefined) this.entriesByConversation.set(entry.conversationId, [stored]); - else entries.push(stored); - } else if (write.type === "task.create" || write.type === "task.set") { - const current = this.tasks.get(write.task.id); - if (current?.status !== "terminal" && current?.output !== undefined) - this.removeOutputReference(current.output.id); - this.tasks.set(write.task.id, write.task); - if (write.task.status !== "terminal" && write.task.output !== undefined) { - this.outputReferences.set(write.task.output.id, (this.outputReferences.get(write.task.output.id) ?? 0) + 1); - } - } else if (write.type === "value.set") { - const state = this.valueState(write.address); - if (state.rewind) state.writes.push({ seq, type: "set", value: write.value }); - else state.value = write.value; - } else if (write.type === "value.delete") { - const state = this.valueState(write.address); - if (state.rewind) state.writes.push({ seq, type: "delete" }); - else state.value = undefined; - } else { - const state = this.listState(write.address); - if (state.rewind) { - if (write.type === "list.append") state.writes.push({ seq, type: "append", element: write.element }); - else if (write.type === "list.remove") - state.writes.push({ seq, type: "remove", elementId: write.elementId }); - else state.writes.push({ seq, type: "clear" }); - } else if (write.type === "list.append") { - state.known.add(write.element.id); - state.elements.set(write.element.id, write.element); - } else if (write.type === "list.remove") state.elements.delete(write.elementId); - else state.elements.clear(); - } - } - - private removeOutputReference(id: Id): void { - const count = this.outputReferences.get(id)!; - if (count === 1) this.outputReferences.delete(id); - else this.outputReferences.set(id, count - 1); - } - - private storedState(address: Address): StoredState | undefined { - return this.stateInScope(this.scopeState(address, false), address); - } - - private conversationStoredState(conversationId: Id, address: Address): StoredState | undefined { - const scopes = address.rewind ? this.conversationRewindableState : this.conversationStickyState; - return this.stateInScope(scopes.get(conversationId), address); - } - - private stateInScope(scope: ScopeState | undefined, address: Address): StoredState | undefined { - const namespace = scope?.get(address.namespace); - const state = address.key === undefined ? namespace?.unkeyed : namespace?.keyed.get(address.key); - return address.kind === "value" ? state?.value : state?.list; - } - - private valueState(address: Value): Extract { - const initial: StoredState = rewindable(address) - ? { kind: "value", rewind: true, writes: [] } - : { kind: "value", rewind: false }; - return this.ensureState(address, initial) as Extract; - } - - private listState(address: List): Extract { - const initial: StoredState = rewindable(address) - ? { kind: "list", rewind: true, writes: [] } - : { kind: "list", rewind: false, elements: new Map(), known: new Set() }; - return this.ensureState(address, initial) as Extract; - } - - private ensureState(address: Address, initial: StoredState): StoredState { - const scope = this.scopeState(address, true)!; - let namespace = scope.get(address.namespace); - if (namespace === undefined) { - namespace = { keyed: new Map() }; - scope.set(address.namespace, namespace); - } - let state = address.key === undefined ? namespace.unkeyed : namespace.keyed.get(address.key); - if (state === undefined) { - state = {}; - if (address.key === undefined) namespace.unkeyed = state; - else namespace.keyed.set(address.key, state); - } - const stored = address.kind === "value" ? state.value : state.list; - if (stored !== undefined) return stored; - if (initial.kind === "value") state.value = initial; - else state.list = initial; - return initial; - } - - private scopeState(address: Address, create: boolean): ScopeState | undefined { - if (address.scope.type === "session") return this.sessionState; - const id = scopeId(address)!; - const scopes = - address.scope.type === "conversation" - ? address.rewind - ? this.conversationRewindableState - : this.conversationStickyState - : address.scope.type === "task" - ? this.taskState - : this.sharedState; - let state = scopes.get(id); - if (state === undefined && create) { - state = new Map(); - scopes.set(id, state); - } - return state; - } -} diff --git a/packages/agent/src/harness/pico/runtime.ts b/packages/agent/src/harness/pico/runtime.ts deleted file mode 100644 index fe5929864..000000000 --- a/packages/agent/src/harness/pico/runtime.ts +++ /dev/null @@ -1,343 +0,0 @@ -import type { Context } from "@earendil-works/chord"; -import type { AssistantMessage, ImageContent, Message, TextContent, ToolCall } from "@earendil-works/pi-ai"; -import type { Element, List, PayloadOf, ResolvedValue, UnboundList, UnboundValue, Value } from "./addresses.ts"; -import type { Id, JsonValue, Stored } from "./core.ts"; -import type { AssistantEntry, ContextEdit, Conversation, Entry, EntryInput, EntryKind, UserEntry } from "./entries.ts"; -import type { EmptyHookPoints, HookPoints, HookRunner } from "./hooks.ts"; -import type { - AnyDefinedKind, - Completion, - ExactJsonInput, - InputOf, - NoExtra, - OrdinaryKind, - OutputOf, - OutputState, - RunningTask, - Task, - TaskCheckpoint, - TaskOf, - TaskOutputRef, -} from "./tasks.ts"; - -export interface TxReaders { - getConversation(id: Id): Promise; - getEntry(id: Id): Promise; - getEntry(kind: EntryKind, id: Id): Promise; - getEntries(ids: readonly Id[]): Promise>; - getTask(id: Id): Promise; - getTask(kind: K, id: Id): Promise | undefined>; - getTasks(ids: readonly Id[]): Promise>; -} - -export interface TxValue { - get(at?: Id): Promise; - set(value: T): void; - delete(): void; -} - -export interface TxList { - read(at?: Id): Promise[]>; - append(value: T): Id; - remove(id: Id): void; - clear(): void; -} - -export interface TxState { - value>(address: D): TxValue, ResolvedValue>; - list(address: List): TxList; -} - -export interface BoundTxState { - value | UnboundValue>(address: D): TxValue, ResolvedValue>; - list(address: List | UnboundList): TxList; -} - -export type Acceptance = { - readonly kind: "send" | "write"; - readonly requestId?: string; - readonly conversationId: Id; - readonly inputId: Id; -}; - -export type TaskSpec = { - readonly conversationId?: Id; - readonly input: InputOf; - readonly after?: readonly Id[]; - readonly background?: true; -} & ([OutputOf] extends [never] ? { readonly output?: never } : { readonly output?: TaskOutputRef> }); - -export interface ConversationCreateSpec { - readonly parent?: { readonly conversationId: Id; readonly at: Id | "start" }; -} - -export type UserInput = string | readonly (TextContent | ImageContent)[]; - -export type StoredEntryDraft = { - readonly kind: string; - readonly data?: JsonValue; - readonly model?: readonly Stored[]; - readonly head?: Id | "self"; - readonly edits?: readonly ContextEdit[]; -}; - -export type QueuedInput = - | { readonly mode: "steer" | "followUp"; readonly input: Stored; readonly requestId?: string } - | { readonly mode: "write"; readonly entry: StoredEntryDraft; readonly requestId?: string }; - -export type StoredInputResult = - | { readonly status: "queued"; readonly requestId?: string } - | { readonly status: "placed"; readonly requestId?: string; readonly entry: Id } - | { readonly status: "done"; readonly requestId?: string; readonly entry: Id; readonly answer?: Id } - | { - readonly status: "unanswered"; - readonly requestId?: string; - readonly entry?: Id; - readonly reason: "terminated" | "aborted" | "failed" | "stale"; - readonly detail?: string; - }; - -export type InputOutcome = - | { readonly status: "queued"; readonly requestId?: string } - | { readonly status: "placed"; readonly requestId?: string; readonly input: UserEntry } - | { - readonly status: "done"; - readonly requestId?: string; - readonly input: UserEntry; - readonly answer?: { readonly entry: AssistantEntry; readonly message: AssistantMessage }; - } - | { - readonly status: "unanswered"; - readonly requestId?: string; - readonly input?: UserEntry; - readonly reason: "terminated" | "aborted" | "failed" | "stale"; - readonly detail?: string; - }; - -export type TerminalInputOutcome = Extract; - -export interface InternalAcceptOptions { - readonly input: UserInput; - readonly requestId?: string; - readonly whenBusy?: "followUp" | "steer" | "reject"; -} - -export interface SendInput { - readonly requestId?: string; - readonly content: UserInput; - readonly whenBusy?: "followUp" | "steer" | "reject"; -} - -export interface InputHandle { - readonly id: Id; - readonly conversationId: Id; - result(ctx: Context): Promise; - wait(ctx: Context): Promise; - abort(ctx: Context): Promise<"aborted" | "already_placed" | "not_found">; -} - -interface TaskCreator> { - task & Extra>( - kind: K, - spec: NoExtra & Extra, S> & { readonly input: ExactJsonInput, S["input"]> }, - ): Id; -} - -export interface HostTx extends TxReaders, TxState, TaskCreator { - createConversation(spec: ConversationCreateSpec): Id; - write(conversationId: Id, kind: EntryKind, input: EntryInput): Promise; -} - -export interface ConversationTx extends TxReaders, BoundTxState, TaskCreator { - createConversation(spec: ConversationCreateSpec): Id; - write(kind: EntryKind, input: EntryInput): Promise; -} - -interface TaskTxBase extends TxReaders, BoundTxState { - checkpoint(value: C): void; - createConversation(spec: ConversationCreateSpec): Id; -} - -export interface BaseTaskTx extends TaskTxBase, TaskCreator { - write(kind: EntryKind, input: EntryInput): Promise; -} - -export interface InternalAdmissionTx { - accept(conversationId: Id, options: InternalAcceptOptions): Promise; - queueInput(conversationId: Id, input: QueuedInput): Promise; - write( - conversationId: Id, - kind: EntryKind, - input: EntryInput, - requestId?: string, - ): Promise; -} - -export type CoreTaskTx = Omit, "task" | "write"> & - TaskCreator & - InternalAdmissionTx & { - entry(kind: EntryKind, conversationId: Id, input: EntryInput): Id; - }; - -export interface PublicValue { - get(ctx: Context): Promise; - get(at: Id, ctx: Context): Promise; - set(value: T, ctx: Context): Promise; - delete(ctx: Context): Promise; -} - -export interface StickyPublicValue { - get(ctx: Context): Promise; - set(value: T, ctx: Context): Promise; - delete(ctx: Context): Promise; -} - -export interface PublicList { - read(ctx: Context): Promise[]>; - read(at: Id, ctx: Context): Promise[]>; - append(value: T, ctx: Context): Promise; - remove(id: Id, ctx: Context): Promise; - clear(ctx: Context): Promise; -} - -export interface BoundPublicState { - value | UnboundValue>(address: D): PublicValue, ResolvedValue>; - list(address: List | UnboundList): PublicList; -} - -export interface TaskOutput { - readonly ref: TaskOutputRef; - read(ctx: Context): Promise; - mutate(mutator: (state: O) => undefined, ctx: Context): Promise; - replace(value: O, ctx: Context): Promise; -} - -export interface ReadonlyTaskOutput { - readonly ref: TaskOutputRef; - read(ctx: Context): Promise; -} - -export type RuntimeOutput = [O] extends [never] - ? Record - : { readonly output: TaskOutput }; -export type AbortRuntimeOutput = [O] extends [never] - ? Record - : { readonly output: ReadonlyTaskOutput }; -export type FinalOutput = [O] extends [never] ? Record : { readonly output: O }; - -export interface ScratchReader { - value>(address: D): Pick, ResolvedValue>, "get">; - list(address: List): Pick, "read">; -} - -export interface TaskConversation extends BoundPublicState { - readonly id: Id; - send(input: SendInput, ctx: Context): Promise; -} - -export interface AbortTaskConversation extends BoundPublicState { - readonly id: Id; -} - -interface TaskRuntimeBase { - readonly taskId: Id; - scratch(build: (tx: TxState) => T | Promise, ctx: Context): Promise; - readonly hooks: HookRunner; - conversation(id: Id, ctx: Context): Promise; - waitForTask(id: Id, ctx: Context): Promise; - abortTask(id: Id, ctx: Context): Promise<"marked" | "terminal">; - now(): number; - sleep(untilMs: number, ctx: Context): Promise; -} - -interface CommitRuntime { - commit(build: (tx: Tx, current: RunningTask) => T | Promise, ctx: Context): Promise; -} - -export type TaskRuntime< - I extends JsonValue, - C extends TaskCheckpoint, - O extends OutputState = never, - H extends HookPoints = EmptyHookPoints, -> = TaskRuntimeBase & CommitRuntime> & RuntimeOutput; - -export type CoreTaskRuntime< - I extends JsonValue, - C extends TaskCheckpoint, - O extends OutputState = never, - H extends HookPoints = EmptyHookPoints, -> = TaskRuntimeBase & CommitRuntime> & RuntimeOutput; - -export type FinalTx = BaseTaskTx & FinalOutput; -export type CoreFinalTx = CoreTaskTx & FinalOutput; -export type TerminalClosure< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - O extends OutputState, -> = (tx: FinalTx, current: RunningTask) => Completion | Promise>; -export type CoreTerminalClosure< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - O extends OutputState, -> = (tx: CoreFinalTx, current: RunningTask) => Completion | Promise>; - -export interface AbortTx extends TxReaders, BoundTxState { - checkpoint(value: C): void; - markTask(id: Id): Promise<"marked" | "terminal">; -} - -export interface CoreAbortTx extends AbortTx { - entry(kind: EntryKind, conversationId: Id, input: EntryInput): Id; - write( - conversationId: Id, - kind: EntryKind, - input: EntryInput, - requestId?: string, - ): Promise; -} - -export type AbortFinalTx = AbortTx & FinalOutput; -export type CoreAbortFinalTx = CoreAbortTx & FinalOutput; -export type AbortClosure = ( - tx: AbortFinalTx, - current: RunningTask, -) => A | Promise; -export type CoreAbortClosure< - I extends JsonValue, - C extends TaskCheckpoint, - A extends JsonValue, - O extends OutputState, -> = (tx: CoreAbortFinalTx, current: RunningTask) => A | Promise; - -interface AbortRuntimeBase { - readonly taskId: Id; - scratch(read: (scratch: ScratchReader) => T | Promise, ctx: Context): Promise; - readonly hooks: HookRunner; - conversation(id: Id, ctx: Context): Promise; - waitForTask(id: Id, ctx: Context): Promise; - abortTask(id: Id, ctx: Context): Promise<"marked" | "terminal">; - now(): number; - sleep(untilMs: number, ctx: Context): Promise; -} - -export type AbortTaskRuntime< - I extends JsonValue, - C extends TaskCheckpoint, - O extends OutputState = never, - H extends HookPoints = EmptyHookPoints, -> = AbortRuntimeBase & CommitRuntime> & AbortRuntimeOutput; - -export type CoreAbortTaskRuntime< - I extends JsonValue, - C extends TaskCheckpoint, - O extends OutputState = never, - H extends HookPoints = EmptyHookPoints, -> = AbortRuntimeBase & CommitRuntime> & AbortRuntimeOutput; - -export type ToolInput = { readonly assistant: Id; readonly call: Stored }; -export type GenerationOutput = { message?: Stored }; -export type ToolOutputState = { progress?: string; log: string[]; details?: JsonValue }; diff --git a/packages/agent/src/harness/pico/session.ts b/packages/agent/src/harness/pico/session.ts deleted file mode 100644 index 5f7546efb..000000000 --- a/packages/agent/src/harness/pico/session.ts +++ /dev/null @@ -1,440 +0,0 @@ -import { AsyncLocalStorage } from "node:async_hooks"; -import type { Context } from "@earendil-works/chord"; -import { withoutAbortSignal } from "@earendil-works/chord/context"; -import type { Element, List, PayloadOf, ResolvedValue, Value } from "./addresses.ts"; -import type { Id, JsonValue } from "./core.ts"; -import type { Conversation, Entry } from "./entries.ts"; -import type { ConversationScan, EntryScan, NewTask, Page, Storage, TaskScan, Write } from "./storage.ts"; -import type { Task } from "./tasks.ts"; - -export class Closed extends Error { - constructor() { - super("Session is closed"); - this.name = "Closed"; - } -} - -export class Faulted extends Error { - constructor() { - super("Session is faulted"); - this.name = "Faulted"; - } -} - -export class NestedLineOperation extends Error { - constructor() { - super("Session line operations cannot be nested"); - this.name = "NestedLineOperation"; - } -} - -export class ReadAfterWrite extends Error { - constructor() { - super("Transaction reads must precede writes"); - this.name = "ReadAfterWrite"; - } -} - -export type TaskCreate = Omit & { - readonly output?: { readonly id?: Id; readonly kind: string }; -}; - -interface LineInvocation { - active: boolean; -} - -const lineInvocation = new AsyncLocalStorage(); -const storageOwners = new WeakMap(); - -function assertContextActive(ctx: Context): void { - ctx.abortSignal?.throwIfAborted(); -} - -function resolveStoredValue>( - address: D, - stored: PayloadOf | undefined, -): ResolvedValue { - const fallback = address.default as PayloadOf | undefined; - return (stored === undefined ? fallback : stored) as ResolvedValue; -} - -export interface Transaction { - getConversation(id: Id): Promise; - getEntry(id: Id): Promise; - getEntries(ids: readonly Id[]): Promise>; - getTask(id: Id): Promise; - getTasks(ids: readonly Id[]): Promise>; - createConversation(conversation: Omit): Id; - entry(entry: Omit): Id; - task(task: TaskCreate): Id; - setTask(task: Task): void; - value>( - address: D, - ): { - get(at?: Id): Promise>; - set(value: PayloadOf): void; - delete(): void; - }; - list( - address: List, - ): { - read(at?: Id): Promise[]>; - append(value: T): Id; - remove(id: Id): void; - clear(): void; - }; -} - -class TransactionState { - private readonly storage: Storage; - private readonly ctx: Context; - private active = true; - private writing = false; - private entryAppended = false; - private pendingReads = 0; - private failure: Error | undefined; - private readonly writes: Write[] = []; - - constructor(storage: Storage, ctx: Context) { - this.storage = storage; - this.ctx = ctx; - } - - facade(): Transaction { - return Object.freeze({ - getConversation: (id: Id) => this.getConversation(id), - getEntry: (id: Id) => this.getEntry(id), - getEntries: (ids: readonly Id[]) => this.getEntries(ids), - getTask: (id: Id) => this.getTask(id), - getTasks: (ids: readonly Id[]) => this.getTasks(ids), - createConversation: (conversation: Omit) => this.createConversation(conversation), - entry: (entry: Omit) => this.entry(entry), - task: (task: TaskCreate) => this.task(task), - setTask: (task: Task) => this.setTask(task), - value: >(address: D) => this.value(address), - list: (address: List) => this.list(address), - }); - } - - finish(): readonly Write[] { - this.assertActive(); - if (this.pendingReads !== 0) this.poison(new Error("Transaction has pending reads")); - this.active = false; - return this.writes; - } - - seal(): void { - this.active = false; - } - - async getConversation(id: Id): Promise { - return (await this.read(() => this.storage.getConversations([id], this.ctx))).get(id); - } - - async getEntry(id: Id): Promise { - return (await this.getEntries([id])).get(id); - } - - getEntries(ids: readonly Id[]): Promise> { - return this.read(() => this.storage.getEntries(ids, this.ctx)); - } - - async getTask(id: Id): Promise { - return (await this.getTasks([id])).get(id); - } - - getTasks(ids: readonly Id[]): Promise> { - return this.read(() => this.storage.getTasks(ids, this.ctx)); - } - - createConversation(conversation: Omit): Id { - const id = this.allocate(); - this.writes.push({ type: "conversation.create", conversation: { ...conversation, id } }); - return id; - } - - entry(entry: Omit): Id { - const id = this.allocate(); - this.writes.push({ type: "entry.append", entry: { ...entry, id } }); - this.entryAppended = true; - return id; - } - - task(task: TaskCreate): Id { - const id = this.allocate(); - const { output: requestedOutput, ...base } = task; - const output = - requestedOutput === undefined ? undefined : { id: requestedOutput.id ?? id, kind: requestedOutput.kind }; - this.writes.push({ - type: "task.create", - task: { ...base, id, ...(output === undefined ? {} : { output }) }, - }); - return id; - } - - setTask(task: Task): void { - this.write({ type: "task.set", task }); - } - - value>( - address: D, - ): { - get(at?: Id): Promise>; - set(value: PayloadOf): void; - delete(): void; - } { - this.assertActive(); - return Object.freeze({ - get: (at?: Id) => - this.read(async () => { - const stored = await this.storage.getValue(address as unknown as Value>, at, this.ctx); - return resolveStoredValue(address, stored); - }), - set: (value: PayloadOf) => { - this.assertStateWriteOrder(address); - this.write({ type: "value.set", address, value }); - }, - delete: () => { - this.assertStateWriteOrder(address); - this.write({ type: "value.delete", address }); - }, - }); - } - - list( - address: List, - ): { - read(at?: Id): Promise[]>; - append(value: T): Id; - remove(id: Id): void; - clear(): void; - } { - this.assertActive(); - return Object.freeze({ - read: (at?: Id) => this.read(() => this.storage.readList(address, at, this.ctx)), - append: (value: T) => { - this.assertStateWriteOrder(address); - const id = this.allocate(); - this.writes.push({ type: "list.append", address, element: { id, value } }); - return id; - }, - remove: (elementId: Id) => { - this.assertStateWriteOrder(address); - this.write({ type: "list.remove", address, elementId }); - }, - clear: () => { - this.assertStateWriteOrder(address); - this.write({ type: "list.clear", address }); - }, - }); - } - - private assertStateWriteOrder(address: Value | List): void { - this.assertActive(); - if (address.rewind && this.entryAppended) { - this.poison(new Error("Rewindable state writes must precede entry appends")); - } - } - - private allocate(): Id { - this.assertWritable(); - return this.storage.nextId(); - } - - private write(write: Write): void { - this.assertWritable(); - this.writes.push(write); - } - - private async read(read: () => Promise): Promise { - this.assertReadable(); - this.pendingReads++; - try { - return await read(); - } finally { - this.pendingReads--; - } - } - - private assertReadable(): void { - this.assertActive(); - if (this.writing) this.poison(new ReadAfterWrite()); - } - - private assertWritable(): void { - this.assertActive(); - if (this.pendingReads !== 0) this.poison(new Error("Transaction writes must await reads")); - this.writing = true; - } - - private assertActive(): void { - if (!this.active) throw new Error("Transaction is closed"); - if (this.failure !== undefined) throw this.failure; - } - - private poison(error: Error): never { - this.failure ??= error; - throw this.failure; - } -} - -export class Session { - private readonly storage: Storage; - private readonly ownership = Object.freeze({}); - private tail: Promise = Promise.resolve(); - private fault: Faulted | undefined; - private closed = false; - private closeCompletion: Promise | undefined; - private storageCloseCompletion: Promise | undefined; - - constructor(storage: Storage) { - if (storageOwners.has(storage)) throw new Error("Storage is already owned by another Session"); - storageOwners.set(storage, this.ownership); - this.storage = storage; - } - - commit(build: (tx: Transaction) => T | Promise, ctx: Context): Promise { - try { - assertContextActive(ctx); - } catch (error) { - return Promise.reject(error); - } - return this.onLine(async () => { - this.assertUsable(); - assertContextActive(ctx); - const tx = new TransactionState(this.storage, ctx); - try { - const result = await build(tx.facade()); - const writes = tx.finish(); - if (writes.length !== 0) { - try { - await this.storage.commit(writes, withoutAbortSignal(ctx)); - } catch (error) { - this.fault = new Faulted(); - try { - await this.closeStorage(ctx); - } catch {} - throw error; - } - } - return result; - } finally { - tx.seal(); - } - }); - } - - close(ctx: Context): Promise { - if (this.nested()) return Promise.reject(new NestedLineOperation()); - this.closeCompletion ??= this.enqueueLine(async () => { - this.closed = true; - await this.closeStorage(ctx); - }); - return this.closeCompletion; - } - - getConversations(ids: readonly Id[], ctx: Context): Promise> { - return this.onLine(() => { - this.assertUsable(); - return this.storage.getConversations(ids, ctx); - }); - } - - scanConversations(query: ConversationScan, ctx: Context): Promise> { - return this.onLine(() => { - this.assertUsable(); - return this.storage.scanConversations(query, ctx); - }); - } - - getEntries(ids: readonly Id[], ctx: Context): Promise> { - return this.onLine(() => { - this.assertUsable(); - return this.storage.getEntries(ids, ctx); - }); - } - - scanEntries(query: EntryScan, ctx: Context): Promise> { - return this.onLine(() => { - this.assertUsable(); - return this.storage.scanEntries(query, ctx); - }); - } - - newestHead(conversationId: Id, at: Id, ctx: Context): Promise { - return this.onLine(() => { - this.assertUsable(); - return this.storage.newestHead(conversationId, at, ctx); - }); - } - - getTasks(ids: readonly Id[], ctx: Context): Promise> { - return this.onLine(() => { - this.assertUsable(); - return this.storage.getTasks(ids, ctx); - }); - } - - scanTasks(query: TaskScan, ctx: Context): Promise> { - return this.onLine(() => { - this.assertUsable(); - return this.storage.scanTasks(query, ctx); - }); - } - - getValue>(address: D, at: Id | undefined, ctx: Context): Promise> { - return this.onLine(async () => { - this.assertUsable(); - const stored = await this.storage.getValue(address as unknown as Value>, at, ctx); - return resolveStoredValue(address, stored); - }); - } - - readList(address: List, at: Id | undefined, ctx: Context): Promise[]> { - return this.onLine(() => { - this.assertUsable(); - return this.storage.readList(address, at, ctx); - }); - } - - private onLine(operation: () => T | Promise): Promise { - if (this.nested()) return Promise.reject(new NestedLineOperation()); - return this.enqueueLine(operation); - } - - private enqueueLine(operation: () => T | Promise): Promise { - const previous = this.tail; - let release!: () => void; - this.tail = new Promise((resolve) => { - release = resolve; - }); - return (async () => { - await previous; - const invocation: LineInvocation = { active: true }; - try { - return await lineInvocation.run(invocation, operation); - } finally { - invocation.active = false; - release(); - } - })(); - } - - private nested(): boolean { - return lineInvocation.getStore()?.active === true; - } - - private closeStorage(ctx: Context): Promise { - this.storageCloseCompletion ??= Promise.resolve() - .then(() => this.storage.close(withoutAbortSignal(ctx))) - .then(() => { - if (storageOwners.get(this.storage) === this.ownership) storageOwners.delete(this.storage); - }); - return this.storageCloseCompletion; - } - - private assertUsable(): void { - if (this.fault !== undefined) throw this.fault; - if (this.closed) throw new Closed(); - } -} diff --git a/packages/agent/src/harness/pico/storage.ts b/packages/agent/src/harness/pico/storage.ts deleted file mode 100644 index fbf507d84..000000000 --- a/packages/agent/src/harness/pico/storage.ts +++ /dev/null @@ -1,104 +0,0 @@ -import type { Context } from "@earendil-works/chord"; -import type { Element, List, Value } from "./addresses.ts"; -import type { Id, JsonValue, Seq } from "./core.ts"; -import type { Conversation, Entry, SectionRecord } from "./entries.ts"; -import type { Task, TaskBase } from "./tasks.ts"; - -export interface PageQuery { - /** Last item returned by the same scan. */ - readonly cursor?: Id; - readonly limit: number; -} - -export interface Page { - readonly items: readonly T[]; - readonly next?: Id; -} - -export class ScratchRetired extends Error { - readonly taskId: Id; - - constructor(taskId: Id) { - super(`Scratch for task ${taskId} is retired`); - this.name = "ScratchRetired"; - this.taskId = taskId; - } -} - -export class OutputRetired extends Error { - readonly outputId: Id; - - constructor(outputId: Id) { - super(`Task output ${outputId} is retired`); - this.name = "OutputRetired"; - this.outputId = outputId; - } -} - -export class InvalidHistoryPosition extends Error { - readonly at: Id; - - constructor(at: Id) { - super(`Entry ${at} is not a valid history position for this address`); - this.name = "InvalidHistoryPosition"; - this.at = at; - } -} - -export type NewTask = Omit & { - readonly status: "pending"; - readonly owns: readonly []; - readonly abort?: never; - readonly checkpoint?: never; - readonly outcome?: never; -}; - -export type StateWrite = - | { readonly type: "value.set"; readonly address: Value; readonly value: JsonValue } - | { readonly type: "value.delete"; readonly address: Value } - | { readonly type: "list.append"; readonly address: List; readonly element: Element } - | { readonly type: "list.remove"; readonly address: List; readonly elementId: Id } - | { readonly type: "list.clear"; readonly address: List }; - -export type StoredConversation = Conversation & { readonly sectionSeed?: readonly SectionRecord[] }; - -export type Write = - | StateWrite - | { readonly type: "conversation.create"; readonly conversation: StoredConversation } - | { readonly type: "entry.append"; readonly entry: Entry } - | { readonly type: "task.create"; readonly task: NewTask } - | { readonly type: "task.set"; readonly task: Task }; - -export type ConversationScan = PageQuery & { - readonly parent?: Id; - readonly owner?: Id; -}; - -export type EntryScan = PageQuery & { - readonly conversationId: Id; - readonly kind?: string; - readonly through?: Id; -}; - -export type TaskScan = PageQuery & { - readonly conversationIds?: readonly Id[]; - readonly statuses?: readonly Task["status"][]; - readonly kind?: string; - readonly abort?: boolean; - readonly outputId?: Id; -}; - -export interface Storage { - nextId(): Id; - commit(writes: readonly Write[], ctx: Context): Promise; - getConversations(ids: readonly Id[], ctx: Context): Promise>; - scanConversations(query: ConversationScan, ctx: Context): Promise>; - getEntries(ids: readonly Id[], ctx: Context): Promise>; - scanEntries(query: EntryScan, ctx: Context): Promise>; - newestHead(conversationId: Id, at: Id, ctx: Context): Promise; - getTasks(ids: readonly Id[], ctx: Context): Promise>; - scanTasks(query: TaskScan, ctx: Context): Promise>; - getValue(address: Value, at: Id | undefined, ctx: Context): Promise; - readList(address: List, at: Id | undefined, ctx: Context): Promise[]>; - close(ctx: Context): Promise; -} diff --git a/packages/agent/src/harness/pico/tasks.ts b/packages/agent/src/harness/pico/tasks.ts deleted file mode 100644 index 355f628e4..000000000 --- a/packages/agent/src/harness/pico/tasks.ts +++ /dev/null @@ -1,312 +0,0 @@ -import type { Context } from "@earendil-works/chord"; -import type { ConversationValueDefinition } from "./addresses.ts"; -import type { Id, JsonObject, JsonValue } from "./core.ts"; -import type { EmptyHookPoints, HookPoints } from "./hooks.ts"; -import type { - AbortClosure, - AbortTaskRuntime, - CoreAbortClosure, - CoreAbortTaskRuntime, - CoreTaskRuntime, - CoreTerminalClosure, - TaskRuntime, - TerminalClosure, -} from "./runtime.ts"; - -export interface TaskCheckpoint extends JsonObject { - readonly phase: string; -} - -export type NoCheckpoint = never; -export type OutputState = JsonObject | JsonValue[]; -export type ConfigBundle = Record>; -export type ReservedConfigKeys = "set" | "get"; -export type KeysOf = U extends unknown ? keyof U : never; -export type Collides = [ - Extract]> | Reserved>, -] extends [never] - ? false - : true; -export type AnyCollision = true extends { - [N in keyof M]: Collides; -}[keyof M] - ? true - : false; -export type DisjointBundles = AnyCollision extends false - ? true - : false; - -declare const taskOutputType: unique symbol; -const taskKindBrand: unique symbol = Symbol("pico.taskKind"); - -export interface TaskOutputKind { - readonly kind: string; - readonly [taskOutputType]?: O; -} - -export function defineTaskOutput(kind: string): TaskOutputKind { - return Object.freeze({ kind }); -} - -export interface StoredTaskOutputRef { - readonly id: Id; - readonly kind: string; -} - -export interface TaskOutputRef extends StoredTaskOutputRef { - readonly [taskOutputType]?: O; -} - -export type TaskOutputField = [O] extends [never] - ? Record - : { readonly output: TaskOutputRef }; - -export type TaskOutputDefinition = [O] extends [never] - ? { readonly output?: never } - : { readonly output: { readonly kind: TaskOutputKind; initial(input: I): O } }; - -export type TaskOutcome< - R extends JsonValue = JsonValue, - F extends JsonValue = JsonValue, - A extends JsonValue = JsonValue, -> = - | { readonly status: "completed"; readonly result: R } - | { readonly status: "failed"; readonly failure: F } - | { readonly status: "aborted"; readonly result: A } - | { readonly status: "orphaned" }; - -export interface TaskBase { - readonly id: Id; - readonly conversationId: Id; - readonly kind: string; - readonly input: I; - readonly checkpoint?: C; - readonly after: readonly Id[]; - readonly background?: true; - readonly owns: readonly Id[]; - readonly output?: StoredTaskOutputRef; - readonly abort?: true; -} - -export type Task< - I extends JsonValue = JsonValue, - C extends TaskCheckpoint = TaskCheckpoint, - R extends JsonValue = JsonValue, - F extends JsonValue = JsonValue, - A extends JsonValue = JsonValue, -> = TaskBase & - ( - | { readonly status: "pending" | "running"; readonly outcome?: never } - | { readonly status: "terminal"; readonly outcome: TaskOutcome } - ); - -export type RunningTask = Omit< - TaskBase, - "output" -> & - TaskOutputField & { - readonly status: "running"; - readonly outcome?: never; - }; - -export type Completion = - | { readonly status: "completed"; readonly result: R } - | { readonly status: "failed"; readonly failure: F }; - -interface TaskKindMethods< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState, - H extends HookPoints, -> { - readonly kind: string; - readonly hooks?: H; - readonly config?: ConfigBundle; - execute( - task: RunningTask, - runtime: TaskRuntime, - ctx: Context, - ): Promise>; - recover( - task: RunningTask, - runtime: TaskRuntime, - ctx: Context, - ): Promise>; - abort( - task: RunningTask, - runtime: AbortTaskRuntime, - ctx: Context, - ): Promise>; -} - -interface CoreTaskKindMethods< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState, - H extends HookPoints, -> { - readonly kind: string; - readonly hooks?: H; - readonly config?: ConfigBundle; - execute( - task: RunningTask, - runtime: CoreTaskRuntime, - ctx: Context, - ): Promise>; - recover( - task: RunningTask, - runtime: CoreTaskRuntime, - ctx: Context, - ): Promise>; - abort( - task: RunningTask, - runtime: CoreAbortTaskRuntime, - ctx: Context, - ): Promise>; -} - -export type TaskKind< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState = never, - H extends HookPoints = EmptyHookPoints, -> = TaskKindMethods & TaskOutputDefinition & { readonly [taskKindBrand]: "ordinary" }; - -export type CoreTaskKind< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState = never, - H extends HookPoints = EmptyHookPoints, -> = CoreTaskKindMethods & TaskOutputDefinition & { readonly [taskKindBrand]: "core" }; - -export type NoExtra = Actual & Record, never>; - -export type ExactJsonInput = Expected extends readonly JsonValue[] - ? Actual - : Expected extends JsonObject - ? NoExtra - : Actual; - -export type TaskKindDefinition< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState, - H extends HookPoints, -> = Omit, typeof taskKindBrand>; - -export type CoreTaskKindDefinition< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState, - H extends HookPoints, -> = Omit, typeof taskKindBrand>; - -export type TaskKindFactory< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState, -> = >( - definition: NoExtra, D> & { readonly hooks?: H }, -) => D & TaskKind; - -export type CoreTaskKindFactory< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState, -> = >( - definition: NoExtra, D> & { readonly hooks?: H }, -) => D & CoreTaskKind; - -export function defineTask< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState = never, ->(): TaskKindFactory { - const define = >( - definition: NoExtra, D> & { readonly hooks?: H }, - ): D & TaskKind => - Object.freeze({ - ...definition, - [taskKindBrand]: "ordinary" as const, - }) as D & TaskKind; - return define as TaskKindFactory; -} - -/** Internal authoring helper for Pico's fixed privileged task kinds. */ -export function defineCoreTask< - I extends JsonValue, - C extends TaskCheckpoint, - R extends JsonValue, - F extends JsonValue, - A extends JsonValue, - O extends OutputState = never, ->(): CoreTaskKindFactory { - const define = >( - definition: NoExtra, D> & { readonly hooks?: H }, - ): D & CoreTaskKind => - Object.freeze({ - ...definition, - [taskKindBrand]: "core" as const, - }) as D & CoreTaskKind; - return define as CoreTaskKindFactory; -} - -export type OrdinaryKind = { - readonly kind: string; - readonly [taskKindBrand]: "ordinary"; - readonly hooks?: HookPoints; - readonly config?: ConfigBundle; -}; -export type CoreKind = { - readonly kind: string; - readonly [taskKindBrand]: "core"; - readonly hooks?: HookPoints; - readonly config?: ConfigBundle; -}; -export type AnyDefinedKind = OrdinaryKind | CoreKind; - -export type PayloadsOf = K extends TaskKind - ? { input: I; checkpoint: C; result: R; failure: F; aborted: A; output: O; hooks: H } - : K extends CoreTaskKind - ? { input: I; checkpoint: C; result: R; failure: F; aborted: A; output: O; hooks: H } - : never; -export type InputOf = PayloadsOf["input"]; -export type CheckpointOf = PayloadsOf["checkpoint"]; -export type ResultOf = PayloadsOf["result"]; -export type FailureOf = PayloadsOf["failure"]; -export type AbortedOf = PayloadsOf["aborted"]; -export type OutputOf = PayloadsOf["output"]; -export type DistributiveOmit = T extends unknown ? Omit : never; - -export type TaskOf = DistributiveOmit< - Task, CheckpointOf, ResultOf, FailureOf, AbortedOf>, - "output" -> & - TaskOutputField>; diff --git a/packages/agent/test/harness/pico/memory-storage.test.ts b/packages/agent/test/harness/pico/memory-storage.test.ts deleted file mode 100644 index c2512fa84..000000000 --- a/packages/agent/test/harness/pico/memory-storage.test.ts +++ /dev/null @@ -1,1077 +0,0 @@ -import { TODO_CONTEXT, withCancel } from "@earendil-works/chord/context"; -import { describe, expect, it, vi } from "vitest"; -import { defineInternalList, defineInternalValue } from "../../../src/harness/pico/addresses.ts"; -import { - conversationList, - conversationValue, - defineList, - defineValue, - InvalidHistoryPosition, - type JsonValue, - MemoryStorage, - OutputRetired, - ScratchRetired, - sessionValue, - type Task, -} from "../../../src/harness/pico/index.ts"; -import { - Closed, - Faulted, - NestedLineOperation, - ReadAfterWrite, - Session, - type TaskCreate, - type Transaction, -} from "../../../src/harness/pico/session.ts"; - -const ctx = TODO_CONTEXT; - -function deferred(): { readonly promise: Promise; readonly resolve: () => void } { - let resolve!: () => void; - const promise = new Promise((settled) => { - resolve = settled; - }); - return { promise, resolve }; -} - -function taskSpec( - conversationId: number, - extra: Partial> = {}, -): TaskCreate { - return { - conversationId, - kind: "test.task", - input: { value: conversationId }, - after: [], - owns: [], - status: "pending", - ...extra, - }; -} - -type LiveTask = Task & { readonly status: "pending" | "running"; readonly outcome?: never }; - -async function setTask(session: Session, id: number, update: (task: LiveTask) => Task): Promise { - await session.commit(async (tx) => { - const task = await tx.getTask(id); - if (task === undefined || task.status === "terminal") throw new Error(`Task ${id} is not live`); - tx.setTask(update(task)); - }, ctx); -} - -describe("Pico Session over MemoryStorage", () => { - it("separates IDs from sequences and burns IDs from failed callbacks", async () => { - const storage = MemoryStorage.create(); - expect(await storage.commit([{ type: "conversation.create", conversation: { id: 10 } }], ctx)).toEqual([1]); - const state = defineValue({ type: "session" }, "test.state"); - expect(await storage.commit([{ type: "value.set", address: state, value: "set" }], ctx)).toEqual([2]); - expect(await new Session(storage).commit((tx) => tx.createConversation({}), ctx)).toBe(11); - - const sessionStorage = MemoryStorage.create(); - const commit = vi.spyOn(sessionStorage, "commit"); - const session = new Session(sessionStorage); - await expect( - session.commit((tx) => { - tx.createConversation({}); - throw new Error("reject"); - }, ctx), - ).rejects.toThrow("reject"); - expect(commit).not.toHaveBeenCalled(); - expect(await sessionStorage.getConversations([1], ctx)).toEqual(new Map()); - expect(await session.commit((tx) => tx.createConversation({}), ctx)).toBe(2); - expect(commit).toHaveBeenCalledTimes(1); - await expect(commit.mock.results[0]!.value).resolves.toEqual([1]); - await expect( - session.commit(async (tx) => { - tx.createConversation({}); - await tx.getConversation(1); - }, ctx), - ).rejects.toBeInstanceOf(ReadAfterWrite); - expect(await session.commit((tx) => tx.createConversation({}), ctx)).toBe(4); - await expect( - session.commit(async (tx) => { - const read = tx.getConversation(1); - expect(() => tx.createConversation({})).toThrow("Transaction writes must await reads"); - await read; - }, ctx), - ).rejects.toThrow("Transaction writes must await reads"); - expect(await session.commit((tx) => tx.createConversation({}), ctx)).toBe(5); - - let leaked!: Transaction; - await session.commit((tx) => { - leaked = tx; - }, ctx); - expect(() => leaked.createConversation({})).toThrow("Transaction is closed"); - - const concurrent = await Promise.all([ - session.commit(async (tx) => { - await tx.getConversation(1); - return tx.createConversation({}); - }, ctx), - session.commit((tx) => tx.createConversation({}), ctx), - ]); - expect(concurrent).toEqual([6, 7]); - expect(commit).toHaveBeenCalledTimes(5); - }); - - it("serializes direct reads and commits in invocation order", async () => { - const session = new Session(MemoryStorage.create()); - const entered = deferred(); - const release = deferred(); - const holding = session.commit(async () => { - entered.resolve(); - await release.promise; - }, ctx); - await entered.promise; - - let firstReadSettled = false; - const firstRead = session.getConversations([1], ctx).then((conversations) => { - firstReadSettled = true; - return conversations; - }); - const created = session.commit((tx) => tx.createConversation({}), ctx); - const secondRead = session.getConversations([1], ctx); - await Promise.resolve(); - expect(firstReadSettled).toBe(false); - - release.resolve(); - await holding; - expect(await firstRead).toEqual(new Map()); - expect(await created).toBe(1); - expect((await secondRead).has(1)).toBe(true); - }); - - it("checks commit cancellation before queueing and before invoking its callback", async () => { - const storage = MemoryStorage.create(); - const commit = vi.spyOn(storage, "commit"); - const session = new Session(storage); - const entered = deferred(); - const release = deferred(); - const holding = session.commit(async () => { - entered.resolve(); - await release.promise; - }, ctx); - await entered.promise; - - const beforeEntry = withCancel(ctx); - const beforeEntryError = new Error("cancelled before entry"); - beforeEntry.cancel(beforeEntryError); - let beforeEntryCallbackRan = false; - await expect( - session.commit(() => { - beforeEntryCallbackRan = true; - }, beforeEntry.context), - ).rejects.toBe(beforeEntryError); - expect(beforeEntryCallbackRan).toBe(false); - - const beforeCallback = withCancel(ctx); - const beforeCallbackError = new Error("cancelled before callback"); - let beforeCallbackRan = false; - const queued = session.commit(() => { - beforeCallbackRan = true; - }, beforeCallback.context); - beforeCallback.cancel(beforeCallbackError); - release.resolve(); - await holding; - await expect(queued).rejects.toBe(beforeCallbackError); - expect(beforeCallbackRan).toBe(false); - expect(commit).not.toHaveBeenCalled(); - }); - - it("uses a non-cancellable context for admitted persistence", async () => { - const storage = MemoryStorage.create(); - const commit = vi.spyOn(storage, "commit"); - const read = vi.spyOn(storage, "getConversations"); - const session = new Session(storage); - const caller = withCancel(ctx); - const callerError = new Error("cancelled after callback admission"); - expect( - await session.commit((tx) => { - const id = tx.createConversation({}); - caller.cancel(callerError); - return id; - }, caller.context), - ).toBe(1); - expect(commit).toHaveBeenCalledTimes(1); - expect(commit.mock.calls[0]![1].abortSignal).toBeUndefined(); - expect(caller.context.abortSignal?.reason).toBe(callerError); - expect((await session.getConversations([1], ctx)).has(1)).toBe(true); - expect(read.mock.calls[0]![1]).toBe(ctx); - }); - - it("rejects nested Session line entry immediately", async () => { - const session = new Session(MemoryStorage.create()); - const otherSession = new Session(MemoryStorage.create()); - expect( - await session.commit(async (tx) => { - await expect(session.getConversations([], ctx)).rejects.toBeInstanceOf(NestedLineOperation); - await expect(session.commit(() => undefined, ctx)).rejects.toBeInstanceOf(NestedLineOperation); - await expect(session.close(ctx)).rejects.toBeInstanceOf(NestedLineOperation); - await expect(otherSession.getConversations([], ctx)).rejects.toBeInstanceOf(NestedLineOperation); - return tx.createConversation({}); - }, ctx), - ).toBe(1); - expect((await session.getConversations([1], ctx)).has(1)).toBe(true); - await otherSession.close(ctx); - }); - - it("rejects a transaction when a caught ReadAfterWrite poisoned it", async () => { - const storage = MemoryStorage.create(); - const commit = vi.spyOn(storage, "commit"); - const session = new Session(storage); - await expect( - session.commit(async (tx) => { - tx.createConversation({}); - try { - await tx.getConversation(1); - } catch (error) { - expect(error).toBeInstanceOf(ReadAfterWrite); - } - }, ctx), - ).rejects.toBeInstanceOf(ReadAfterWrite); - expect(commit).not.toHaveBeenCalled(); - expect(await session.commit((tx) => tx.createConversation({}), ctx)).toBe(2); - }); - - it("reopens from the committed-ID high-water and reuses only uncommitted IDs", async () => { - const storage = MemoryStorage.create(); - expect(storage.nextId()).toBe(1); - await storage.close(ctx); - await storage.close(ctx); - expect(storage.nextId()).toBe(1); - const session = new Session(storage); - - expect(await session.commit((tx) => tx.createConversation({}), ctx)).toBe(2); - expect(storage.nextId()).toBe(3); - await session.close(ctx); - expect(storage.nextId()).toBe(3); - - const list = defineList({ type: "session" }, "test.high-water"); - expect( - await storage.commit( - [ - { type: "list.append", address: list, element: { id: 20, value: "retired" } }, - { type: "list.clear", address: list }, - ], - ctx, - ), - ).toEqual([2, 3]); - await storage.close(ctx); - expect(storage.nextId()).toBe(21); - }); - - it("faults and closes the Session after an uncertain persistence failure", async () => { - const storage = MemoryStorage.create(); - const commit = vi.spyOn(storage, "commit").mockRejectedValueOnce(new Error("persistence failed")); - const close = vi.spyOn(storage, "close"); - const session = new Session(storage); - const caller = withCancel(ctx); - await expect( - session.commit((tx) => { - const id = tx.createConversation({}); - caller.cancel(new Error("cancelled during persistence")); - return id; - }, caller.context), - ).rejects.toThrow("persistence failed"); - expect(commit).toHaveBeenCalledTimes(1); - expect(close).toHaveBeenCalledTimes(1); - expect(close.mock.calls[0]![0].abortSignal).toBeUndefined(); - await expect(session.getConversations([1], ctx)).rejects.toBeInstanceOf(Faulted); - await expect(session.commit((tx) => tx.createConversation({}), ctx)).rejects.toBeInstanceOf(Faulted); - await session.close(ctx); - await session.close(ctx); - expect(close).toHaveBeenCalledTimes(1); - const recovered = new Session(storage); - await recovered.close(ctx); - expect(close).toHaveBeenCalledTimes(2); - expect(await storage.getConversations([1], ctx)).toEqual(new Map()); - commit.mockRestore(); - - const state = defineValue({ type: "session" }, "test.after-failure"); - expect(await storage.commit([{ type: "value.set", address: state, value: "stored" }], ctx)).toEqual([1]); - expect(await storage.getValue(state, undefined, ctx)).toBe("stored"); - expect(storage.nextId()).toBe(1); - }); - - it("makes operations queued behind an uncertain commit observe Faulted", async () => { - const storage = MemoryStorage.create(); - const enteredCommit = deferred(); - const rejectCommit = deferred(); - vi.spyOn(storage, "commit").mockImplementationOnce(async () => { - enteredCommit.resolve(); - await rejectCommit.promise; - throw new Error("uncertain"); - }); - const session = new Session(storage); - const failing = session.commit((tx) => tx.createConversation({}), ctx); - await enteredCommit.promise; - - let laterCallbackRan = false; - const queuedRead = session.scanConversations({ limit: 1 }, ctx); - const queuedCommit = session.commit(() => { - laterCallbackRan = true; - }, ctx); - rejectCommit.resolve(); - - await expect(failing).rejects.toThrow("uncertain"); - await expect(queuedRead).rejects.toBeInstanceOf(Faulted); - await expect(queuedCommit).rejects.toBeInstanceOf(Faulted); - expect(laterCallbackRan).toBe(false); - }); - - it("enforces exclusive Storage ownership until Session close succeeds", async () => { - const storage = MemoryStorage.create(); - const first = new Session(storage); - expect(() => new Session(storage)).toThrow("already owned"); - await first.close(ctx); - const reopened = new Session(storage); - await reopened.close(ctx); - }); - - it("keeps Storage owned when close fails", async () => { - const storage = MemoryStorage.create(); - const close = vi.spyOn(storage, "close").mockRejectedValueOnce(new Error("close failed")); - const session = new Session(storage); - await expect(session.close(ctx)).rejects.toThrow("close failed"); - await expect(session.close(ctx)).rejects.toThrow("close failed"); - expect(close).toHaveBeenCalledTimes(1); - expect(() => new Session(storage)).toThrow("already owned"); - }); - - it("closes once and rejects later low-level Session operations", async () => { - const storage = MemoryStorage.create(); - const close = vi.spyOn(storage, "close"); - const session = new Session(storage); - const caller = withCancel(ctx); - caller.cancel(new Error("close caller cancelled")); - const first = session.close(caller.context); - const second = session.close(ctx); - expect(second).toBe(first); - await first; - expect(close).toHaveBeenCalledTimes(1); - expect(close.mock.calls[0]![0].abortSignal).toBeUndefined(); - await expect(session.getConversations([], ctx)).rejects.toBeInstanceOf(Closed); - await expect(session.commit(() => undefined, ctx)).rejects.toBeInstanceOf(Closed); - }); - - it("uses write sequences for historical state inside one atomic storage batch", async () => { - const storage = MemoryStorage.create(); - const conversation = storage.nextId(); - const entry = storage.nextId(); - const beforeElement = storage.nextId(); - const afterElement = storage.nextId(); - const value = defineValue({ type: "conversation", conversationId: conversation }, "test.value", { - rewind: true, - }); - const list = defineList({ type: "conversation", conversationId: conversation }, "test.list", { - rewind: true, - }); - - expect( - await storage.commit( - [ - { type: "conversation.create", conversation: { id: conversation } }, - { type: "value.set", address: value, value: "before" }, - { type: "list.append", address: list, element: { id: beforeElement, value: "before" } }, - { type: "entry.append", entry: { id: entry, conversationId: conversation, kind: "test.entry" } }, - { type: "value.set", address: value, value: "after" }, - { type: "list.append", address: list, element: { id: afterElement, value: "after" } }, - ], - ctx, - ), - ).toEqual([1, 2, 3, 4, 5, 6]); - expect(entry).toBe(2); - expect(await storage.getValue(value, entry, ctx)).toBe("before"); - expect(await storage.getValue(value, undefined, ctx)).toBe("after"); - expect((await storage.readList(list, entry, ctx)).map((item) => item.value)).toEqual(["before"]); - expect((await storage.readList(list, undefined, ctx)).map((item) => item.value)).toEqual(["before", "after"]); - }); - - it("rejects rewindable state writes after an entry in a transaction", async () => { - const storage = MemoryStorage.create(); - const session = new Session(storage); - const value = defineValue({ type: "conversation", conversationId: 1 }, "test.order", { - rewind: true, - }); - await expect( - session.commit((tx) => { - tx.createConversation({}); - tx.entry({ conversationId: 1, kind: "test.entry" }); - try { - tx.value(value).set("too late"); - } catch (error) { - expect(error).toHaveProperty("message", "Rewindable state writes must precede entry appends"); - } - }, ctx), - ).rejects.toThrow("must precede"); - expect(await storage.getConversations([1], ctx)).toEqual(new Map()); - expect(await session.commit((tx) => tx.createConversation({}), ctx)).toBe(3); - }); - - it("validates list removals against their logical lineage", async () => { - const storage = MemoryStorage.create(); - const rootList = defineList({ type: "conversation", conversationId: 1 }, "test.lineage", { - rewind: true, - }); - await storage.commit( - [ - { - type: "conversation.create", - conversation: { - id: 1, - sectionSeed: [{ key: "identity", action: "set", value: "test", rendered: "test" }], - }, - }, - { type: "list.append", address: rootList, element: { id: 2, value: "inherited" } }, - { type: "entry.append", entry: { id: 3, conversationId: 1, kind: "test.fork-point" } }, - { type: "conversation.create", conversation: { id: 4, parent: { conversationId: 1, at: 3 } } }, - { type: "conversation.create", conversation: { id: 5 } }, - ], - ctx, - ); - expect((await storage.getConversations([1], ctx)).get(1)).toEqual({ id: 1 }); - expect((await storage.scanConversations({ limit: 10 }, ctx)).items[0]).toEqual({ id: 1 }); - const childList = defineList({ type: "conversation", conversationId: 4 }, "test.lineage", { - rewind: true, - }); - const unrelatedList = defineList({ type: "conversation", conversationId: 5 }, "test.lineage", { - rewind: true, - }); - const otherDefinition = defineList({ type: "conversation", conversationId: 4 }, "test.other", { - rewind: true, - }); - const marker = defineValue({ type: "session" }, "test.atomic-remove"); - expect( - await storage.commit( - [{ type: "list.append", address: rootList, element: { id: 6, value: "after fork" } }], - ctx, - ), - ).toEqual([6]); - await expect( - storage.commit( - [ - { type: "value.set", address: marker, value: "must not land" }, - { type: "list.remove", address: childList, elementId: 6 }, - ], - ctx, - ), - ).rejects.toThrow("does not belong"); - expect(await storage.getValue(marker, undefined, ctx)).toBeUndefined(); - - expect(await storage.commit([{ type: "list.remove", address: childList, elementId: 2 }], ctx)).toEqual([7]); - expect(await storage.commit([{ type: "list.remove", address: childList, elementId: 2 }], ctx)).toEqual([8]); - expect((await storage.readList(rootList, undefined, ctx)).map(({ id }) => id)).toEqual([2, 6]); - expect(await storage.readList(childList, undefined, ctx)).toEqual([]); - await expect( - storage.commit([{ type: "list.remove", address: unrelatedList, elementId: 2 }], ctx), - ).rejects.toThrow("does not belong"); - await expect( - storage.commit([{ type: "list.remove", address: otherDefinition, elementId: 2 }], ctx), - ).rejects.toThrow("does not belong"); - expect(await storage.commit([{ type: "list.clear", address: childList }], ctx)).toEqual([9]); - }); - - it("validates inherited list removal against earlier writes in the same batch", async () => { - const storage = MemoryStorage.create(); - const session = new Session(storage); - const parentList = defineList({ type: "conversation", conversationId: 1 }, "test.same-batch-lineage", { - rewind: true, - }); - const seeded = await session.commit((tx) => { - const parent = tx.createConversation({}); - const element = tx.list(parentList).append("persisted"); - const at = tx.entry({ conversationId: parent, kind: "test.fork-point" }); - return { parent, element, at }; - }, ctx); - - const child = await session.commit((tx) => { - const id = tx.createConversation({ parent: { conversationId: seeded.parent, at: seeded.at } }); - const childList = defineList({ type: "conversation", conversationId: id }, "test.same-batch-lineage", { - rewind: true, - }); - tx.list(childList).remove(seeded.element); - return id; - }, ctx); - const childList = defineList({ type: "conversation", conversationId: child }, "test.same-batch-lineage", { - rewind: true, - }); - expect(await storage.readList(childList, undefined, ctx)).toEqual([]); - expect((await storage.readList(parentList, undefined, ctx)).map(({ id }) => id)).toEqual([seeded.element]); - - const prospectiveChildList = defineList( - { type: "conversation", conversationId: 22 }, - "test.same-batch-lineage", - { rewind: true }, - ); - expect( - await storage.commit( - [ - { type: "list.append", address: parentList, element: { id: 20, value: "prospective" } }, - { type: "entry.append", entry: { id: 21, conversationId: seeded.parent, kind: "test.next-fork" } }, - { - type: "conversation.create", - conversation: { id: 22, parent: { conversationId: seeded.parent, at: 21 } }, - }, - { type: "list.remove", address: prospectiveChildList, elementId: 20 }, - ], - ctx, - ), - ).toEqual([6, 7, 8, 9]); - expect((await storage.readList(prospectiveChildList, undefined, ctx)).map(({ id }) => id)).toEqual([ - seeded.element, - ]); - }); - - it("pages conversations and fork-visible entries", async () => { - const session = new Session(MemoryStorage.create()); - const ids = await session.commit((tx) => { - const root = tx.createConversation({}); - const first = tx.entry({ conversationId: root, kind: "test.first" }); - const head = tx.entry({ conversationId: root, kind: "test.head", head: first }); - const fork = tx.createConversation({ parent: { conversationId: root, at: first } }); - const forkEntry = tx.entry({ conversationId: fork, kind: "fork.entry" }); - const late = tx.entry({ conversationId: root, kind: "root.late" }); - const ownedFork = tx.createConversation({ parent: { conversationId: root, at: first }, owner: 99 }); - return { root, first, head, fork, forkEntry, late, ownedFork }; - }, ctx); - expect(ids).toEqual({ root: 1, first: 2, head: 3, fork: 4, forkEntry: 5, late: 6, ownedFork: 7 }); - const filteredIds = await session.commit((tx) => { - const older = tx.entry({ conversationId: ids.root, kind: "test.match" }); - tx.entry({ conversationId: ids.root, kind: "test.other" }); - const newer = tx.entry({ conversationId: ids.root, kind: "test.match" }); - return { older, newer }; - }, ctx); - - const conversations = await session.scanConversations({ limit: 1 }, ctx); - expect(conversations.items.map(({ id }) => id)).toEqual([1]); - expect( - (await session.scanConversations({ cursor: conversations.next, limit: 1 }, ctx)).items.map(({ id }) => id), - ).toEqual([4]); - expect((await session.getConversations([ids.ownedFork, 999_999], ctx)).has(ids.ownedFork)).toBe(true); - expect((await session.scanConversations({ parent: ids.root, limit: 10 }, ctx)).items.map(({ id }) => id)).toEqual( - [ids.fork, ids.ownedFork], - ); - expect((await session.scanConversations({ owner: 99, limit: 10 }, ctx)).items.map(({ id }) => id)).toEqual([ - ids.ownedFork, - ]); - expect( - (await session.scanConversations({ parent: ids.root, owner: 99, limit: 10 }, ctx)).items.map(({ id }) => id), - ).toEqual([ids.ownedFork]); - - const fork = await session.scanEntries({ conversationId: 4, limit: 1 }, ctx); - expect(fork.items.map(({ id }) => id)).toEqual([5]); - expect( - (await session.scanEntries({ conversationId: 4, cursor: fork.next, limit: 1 }, ctx)).items.map(({ id }) => id), - ).toEqual([2]); - expect( - (await session.scanEntries({ conversationId: 1, through: 3, limit: 10 }, ctx)).items.map(({ id }) => id), - ).toEqual([3, 2]); - expect( - (await session.scanEntries({ conversationId: ids.root, kind: "test.head", limit: 10 }, ctx)).items.map( - ({ id }) => id, - ), - ).toEqual([ids.head]); - const filtered = await session.scanEntries({ conversationId: ids.root, kind: "test.match", limit: 1 }, ctx); - expect(filtered.items.map(({ id }) => id)).toEqual([filteredIds.newer]); - expect(filtered.next).toBe(filteredIds.newer); - const filteredNext = await session.scanEntries( - { conversationId: ids.root, kind: "test.match", cursor: filtered.next, limit: 1 }, - ctx, - ); - expect(filteredNext.items.map(({ id }) => id)).toEqual([filteredIds.older]); - expect(filteredNext.next).toBeUndefined(); - expect((await session.getEntries([ids.first, 999_999], ctx)).has(ids.first)).toBe(true); - expect((await session.newestHead(1, 3, ctx))?.id).toBe(3); - expect(await session.newestHead(4, 5, ctx)).toBeUndefined(); - }); - - it("uses entry sequences for scans, cutoffs, cursors, and nested fork caps", async () => { - const storage = MemoryStorage.create(); - await storage.commit( - [ - { type: "conversation.create", conversation: { id: 100 } }, - { type: "entry.append", entry: { id: 900, conversationId: 100, kind: "root.old", head: 900 } }, - { type: "entry.append", entry: { id: 700, conversationId: 100, kind: "root.fork" } }, - { type: "conversation.create", conversation: { id: 200, parent: { conversationId: 100, at: 700 } } }, - { type: "entry.append", entry: { id: 950, conversationId: 100, kind: "root.late" } }, - { type: "entry.append", entry: { id: 10, conversationId: 200, kind: "child.fork" } }, - { type: "entry.append", entry: { id: 5, conversationId: 200, kind: "child.late" } }, - { type: "conversation.create", conversation: { id: 300, parent: { conversationId: 200, at: 10 } } }, - { type: "entry.append", entry: { id: 1, conversationId: 300, kind: "grand.local" } }, - { type: "conversation.create", conversation: { id: 400 } }, - { type: "entry.append", entry: { id: 2, conversationId: 400, kind: "unrelated" } }, - ], - ctx, - ); - - expect((await storage.scanEntries({ conversationId: 300, limit: 10 }, ctx)).items.map(({ id }) => id)).toEqual([ - 1, 10, 700, 900, - ]); - expect( - (await storage.scanEntries({ conversationId: 300, through: 10, limit: 10 }, ctx)).items.map(({ id }) => id), - ).toEqual([10, 700, 900]); - expect( - (await storage.scanEntries({ conversationId: 300, through: 700, limit: 10 }, ctx)).items.map(({ id }) => id), - ).toEqual([700, 900]); - const first = await storage.scanEntries({ conversationId: 300, limit: 2 }, ctx); - expect(first.items.map(({ id }) => id)).toEqual([1, 10]); - expect(first.next).toBe(10); - expect( - (await storage.scanEntries({ conversationId: 300, cursor: first.next, limit: 2 }, ctx)).items.map( - ({ id }) => id, - ), - ).toEqual([700, 900]); - expect((await storage.newestHead(300, 10, ctx))?.id).toBe(900); - await expect(storage.scanEntries({ conversationId: 300, through: 2, limit: 10 }, ctx)).rejects.toBeInstanceOf( - InvalidHistoryPosition, - ); - await expect(storage.scanEntries({ conversationId: 300, cursor: 2, limit: 10 }, ctx)).rejects.toBeInstanceOf( - InvalidHistoryPosition, - ); - await expect(storage.newestHead(300, 2, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - }); - - it("buffers values and list elements into one commit", async () => { - const session = new Session(MemoryStorage.create()); - const value = defineValue({ type: "conversation", conversationId: 1 }, "test.value", { - rewind: false, - }); - const list = defineList({ type: "conversation", conversationId: 1 }, "test.list", { rewind: false }); - const ids = await session.commit((tx) => { - const conversation = tx.createConversation({}); - tx.value(value).set({ version: 1 }); - const first = tx.list(list).append("a"); - const second = tx.list(list).append("b"); - return { conversation, first, second }; - }, ctx); - expect(ids).toEqual({ conversation: 1, first: 2, second: 3 }); - expect(await session.getValue(value, undefined, ctx)).toEqual({ version: 1 }); - - expect((await session.readList(list, undefined, ctx)).map(({ value }) => value)).toEqual(["a", "b"]); - - await session.commit((tx) => { - tx.value(value).delete(); - tx.list(list).remove(ids.first); - tx.list(list).clear(); - }, ctx); - expect(await session.getValue(value, undefined, ctx)).toBeUndefined(); - expect(await session.readList(list, undefined, ctx)).toEqual([]); - }); - - it("resolves value defaults in Session and transaction reads", async () => { - const storage = MemoryStorage.create(); - const session = new Session(storage); - const sessionName = sessionValue("test.default.session", { default: "default" }); - const conversationDefinition = conversationValue("test.default.conversation", { - rewind: true, - default: 7, - }); - const boundConversationValue = conversationDefinition.bind(1); - const initial = await session.commit((tx) => { - tx.createConversation({}); - return tx.entry({ conversationId: 1, kind: "test.initial" }); - }, ctx); - - expect(await session.getValue(sessionName, undefined, ctx)).toBe("default"); - expect(await session.getValue(boundConversationValue, undefined, ctx)).toBe(7); - expect(await session.getValue(boundConversationValue, initial, ctx)).toBe(7); - expect( - await session.commit( - async (tx) => ({ - session: await tx.value(sessionName).get(), - historical: await tx.value(boundConversationValue).get(initial), - }), - ctx, - ), - ).toEqual({ session: "default", historical: 7 }); - - const setAt = await session.commit((tx) => { - tx.value(sessionName).set("set"); - tx.value(boundConversationValue).set(9); - return tx.entry({ conversationId: 1, kind: "test.set" }); - }, ctx); - const deletedAt = await session.commit((tx) => { - tx.value(sessionName).delete(); - tx.value(boundConversationValue).delete(); - return tx.entry({ conversationId: 1, kind: "test.deleted" }); - }, ctx); - - expect(await session.getValue(sessionName, undefined, ctx)).toBe("default"); - expect(await session.getValue(boundConversationValue, setAt, ctx)).toBe(9); - expect(await session.getValue(boundConversationValue, deletedAt, ctx)).toBe(7); - expect(await storage.getValue(sessionName, undefined, ctx)).toBeUndefined(); - expect(await storage.getValue(boundConversationValue, deletedAt, ctx)).toBeUndefined(); - expect( - await session.commit( - async (tx) => ({ - session: await tx.value(sessionName).get(), - current: await tx.value(boundConversationValue).get(), - historical: await tx.value(boundConversationValue).get(deletedAt), - }), - ctx, - ), - ).toEqual({ session: "default", current: 7, historical: 7 }); - }); - - it("rewinds values through fork caps and preserves local tombstones", async () => { - const storage = MemoryStorage.create(); - const session = new Session(storage); - const root = await session.commit((tx) => tx.createConversation({}), ctx); - const rootValue = defineValue({ type: "conversation", conversationId: root }, "test.history", { - rewind: true, - }); - const first = await session.commit((tx) => { - tx.value(rootValue).set("root-first"); - return tx.entry({ conversationId: root, kind: "root.first" }); - }, ctx); - const deleted = await session.commit((tx) => { - tx.value(rootValue).delete(); - return tx.entry({ conversationId: root, kind: "root.deleted" }); - }, ctx); - const late = await session.commit((tx) => { - tx.value(rootValue).set("root-late"); - return tx.entry({ conversationId: root, kind: "root.late" }); - }, ctx); - - expect(await session.getValue(rootValue, undefined, ctx)).toBe("root-late"); - expect(await session.getValue(rootValue, first, ctx)).toBe("root-first"); - expect(await session.getValue(rootValue, deleted, ctx)).toBeUndefined(); - expect(await session.getValue(rootValue, late, ctx)).toBe("root-late"); - - const child = await session.commit( - (tx) => tx.createConversation({ parent: { conversationId: root, at: first } }), - ctx, - ); - const childValue = defineValue({ type: "conversation", conversationId: child }, "test.history", { - rewind: true, - }); - expect(await session.getValue(childValue, undefined, ctx)).toBe("root-first"); - const childDeleted = await session.commit((tx) => { - tx.value(childValue).delete(); - return tx.entry({ conversationId: child, kind: "child.deleted" }); - }, ctx); - const grandchild = await session.commit( - (tx) => tx.createConversation({ parent: { conversationId: child, at: childDeleted } }), - ctx, - ); - const grandchildValue = defineValue( - { type: "conversation", conversationId: grandchild }, - "test.history", - { rewind: true }, - ); - await session.commit((tx) => { - tx.value(childValue).set("child-late"); - tx.entry({ conversationId: child, kind: "child.late" }); - }, ctx); - - expect(await session.getValue(childValue, undefined, ctx)).toBe("child-late"); - expect(await session.getValue(grandchildValue, undefined, ctx)).toBeUndefined(); - expect(await session.getValue(grandchildValue, first, ctx)).toBe("root-first"); - expect(await session.getValue(grandchildValue, childDeleted, ctx)).toBeUndefined(); - - const sibling = await session.commit( - (tx) => tx.createConversation({ parent: { conversationId: root, at: first } }), - ctx, - ); - const siblingValue = defineValue({ type: "conversation", conversationId: sibling }, "test.history", { - rewind: true, - }); - expect(await session.getValue(siblingValue, undefined, ctx)).toBe("root-first"); - await session.commit((tx) => tx.value(siblingValue).set("sibling"), ctx); - expect(await session.getValue(siblingValue, undefined, ctx)).toBe("sibling"); - expect(await session.getValue(childValue, undefined, ctx)).toBe("child-late"); - expect(await session.getValue(grandchildValue, undefined, ctx)).toBeUndefined(); - - const deepAncestorFork = await session.commit( - (tx) => tx.createConversation({ parent: { conversationId: grandchild, at: first } }), - ctx, - ); - const deepAncestorValue = defineValue( - { type: "conversation", conversationId: deepAncestorFork }, - "test.history", - { rewind: true }, - ); - expect(await session.getValue(deepAncestorValue, undefined, ctx)).toBe("root-first"); - await expect(session.getValue(childValue, deleted, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - await expect(session.getValue(childValue, 999_999, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - - const unrelated = await session.commit((tx) => { - const conversation = tx.createConversation({}); - return tx.entry({ conversationId: conversation, kind: "unrelated" }); - }, ctx); - await expect(session.getValue(childValue, unrelated, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - - const sticky = defineValue({ type: "conversation", conversationId: root }, "test.sticky", { - rewind: false, - }); - const sessionValue = defineValue({ type: "session" }, "test.history"); - const taskList = defineList({ type: "task", taskId: 100 }, "test.history"); - const sharedValue = defineInternalValue({ type: "shared", id: 200 }, "test.history"); - await expect(storage.getValue(sticky, first, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - await expect(storage.getValue(sessionValue, first, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - await expect(storage.readList(taskList, first, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - await expect(storage.getValue(sharedValue, first, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - }); - - it("rewinds complete lists through append, remove, clear, and nested fork caps", async () => { - const session = new Session(MemoryStorage.create()); - const root = await session.commit((tx) => tx.createConversation({}), ctx); - const rootList = defineList({ type: "conversation", conversationId: root }, "test.history-list", { - rewind: true, - }); - const first = await session.commit((tx) => { - const a = tx.list(rootList).append("a"); - const b = tx.list(rootList).append("b"); - const entry = tx.entry({ conversationId: root, kind: "root.first" }); - return { a, b, entry }; - }, ctx); - const second = await session.commit((tx) => { - tx.list(rootList).remove(first.a); - const c = tx.list(rootList).append("c"); - const entry = tx.entry({ conversationId: root, kind: "root.second" }); - return { c, entry }; - }, ctx); - const third = await session.commit((tx) => { - tx.list(rootList).clear(); - const d = tx.list(rootList).append("d"); - const entry = tx.entry({ conversationId: root, kind: "root.third" }); - return { d, entry }; - }, ctx); - await session.commit((tx) => tx.list(rootList).append("e"), ctx); - - expect((await session.readList(rootList, first.entry, ctx)).map((item) => item.value)).toEqual(["a", "b"]); - expect((await session.readList(rootList, second.entry, ctx)).map((item) => item.value)).toEqual(["b", "c"]); - expect((await session.readList(rootList, third.entry, ctx)).map((item) => item.value)).toEqual(["d"]); - expect((await session.readList(rootList, undefined, ctx)).map((item) => item.value)).toEqual(["d", "e"]); - - const child = await session.commit( - (tx) => tx.createConversation({ parent: { conversationId: root, at: second.entry } }), - ctx, - ); - const childList = defineList({ type: "conversation", conversationId: child }, "test.history-list", { - rewind: true, - }); - const childAppend = await session.commit((tx) => { - tx.list(childList).append("x"); - return tx.entry({ conversationId: child, kind: "child.append" }); - }, ctx); - const childRemove = await session.commit((tx) => { - tx.list(childList).remove(first.b); - return tx.entry({ conversationId: child, kind: "child.remove" }); - }, ctx); - const grandchild = await session.commit( - (tx) => tx.createConversation({ parent: { conversationId: child, at: childRemove } }), - ctx, - ); - const grandchildList = defineList( - { type: "conversation", conversationId: grandchild }, - "test.history-list", - { rewind: true }, - ); - const childClear = await session.commit((tx) => { - tx.list(childList).clear(); - tx.list(childList).append("y"); - return tx.entry({ conversationId: child, kind: "child.clear" }); - }, ctx); - await session.commit((tx) => { - tx.list(grandchildList).append("z"); - tx.entry({ conversationId: grandchild, kind: "grandchild.append" }); - }, ctx); - - expect((await session.readList(childList, childAppend, ctx)).map((item) => item.value)).toEqual(["b", "c", "x"]); - expect((await session.readList(childList, childRemove, ctx)).map((item) => item.value)).toEqual(["c", "x"]); - expect((await session.readList(childList, childClear, ctx)).map((item) => item.value)).toEqual(["y"]); - expect((await session.readList(childList, undefined, ctx)).map((item) => item.value)).toEqual(["y"]); - expect((await session.readList(grandchildList, undefined, ctx)).map((item) => item.value)).toEqual([ - "c", - "x", - "z", - ]); - expect((await session.readList(grandchildList, childRemove, ctx)).map((item) => item.value)).toEqual(["c", "x"]); - - const sibling = await session.commit( - (tx) => tx.createConversation({ parent: { conversationId: root, at: second.entry } }), - ctx, - ); - const siblingList = defineList({ type: "conversation", conversationId: sibling }, "test.history-list", { - rewind: true, - }); - expect((await session.readList(siblingList, undefined, ctx)).map((item) => item.value)).toEqual(["b", "c"]); - await session.commit((tx) => tx.list(siblingList).append("s"), ctx); - expect((await session.readList(siblingList, undefined, ctx)).map((item) => item.value)).toEqual(["b", "c", "s"]); - expect((await session.readList(childList, undefined, ctx)).map((item) => item.value)).toEqual(["y"]); - - const deepAncestorFork = await session.commit( - (tx) => tx.createConversation({ parent: { conversationId: grandchild, at: second.entry } }), - ctx, - ); - const deepAncestorList = defineList( - { type: "conversation", conversationId: deepAncestorFork }, - "test.history-list", - { rewind: true }, - ); - expect((await session.readList(deepAncestorList, undefined, ctx)).map((item) => item.value)).toEqual(["b", "c"]); - - const unrelated = await session.commit((tx) => { - const conversation = tx.createConversation({}); - return tx.entry({ conversationId: conversation, kind: "unrelated" }); - }, ctx); - const stickyList = defineList({ type: "conversation", conversationId: child }, "test.sticky", { - rewind: false, - }); - await expect(session.readList(childList, third.entry, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - await expect(session.readList(childList, unrelated, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - await expect(session.readList(childList, 999_999, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - await expect(session.readList(stickyList, childRemove, ctx)).rejects.toBeInstanceOf(InvalidHistoryPosition); - }); - - it("reserves internal namespaces and shared address construction", () => { - const sharedAsSession = { type: "shared", id: 1 } as unknown as { readonly type: "session" }; - expect(() => defineValue(sharedAsSession, "plugin.shared")).toThrow("Shared scope is internal"); - expect(() => defineList(sharedAsSession, "plugin.shared")).toThrow("Shared scope is internal"); - expect(() => defineValue({ type: "session" }, "pi.internal")).toThrow("reserved"); - expect(() => defineList({ type: "task", taskId: 1 }, "pi.internal")).toThrow("reserved"); - expect(() => conversationValue("pi.internal", { rewind: true })).toThrow("reserved"); - expect(() => conversationList("pi.internal", { rewind: false })).toThrow("reserved"); - expect(defineInternalList({ type: "shared", id: 1 }, "pi.output").scope).toEqual({ - type: "shared", - id: 1, - }); - }); - - it("keeps scopes, kinds, optional keys, and rewind stores separate", async () => { - const session = new Session(MemoryStorage.create()); - const conversation = await session.commit((tx) => tx.createConversation({}), ctx); - const scope = { type: "conversation", conversationId: conversation } as const; - const stickyValue = defineValue(scope, "test.identity", { key: "same", rewind: false }); - const rewindValue = defineValue(scope, "test.identity", { key: "same", rewind: true }); - const stickyList = defineList(scope, "test.identity", { key: "same", rewind: false }); - const rewindList = defineList(scope, "test.identity", { key: "same", rewind: true }); - const absentKey = defineValue(scope, "test.optional-key", { rewind: false }); - const undefinedKey = defineValue(scope, "test.optional-key", { key: undefined, rewind: false }); - const emptyKey = defineValue(scope, "test.optional-key", { key: "", rewind: false }); - const sessionValue = defineValue({ type: "session" }, "test.identity", { key: "same" }); - - await session.commit((tx) => { - tx.value(stickyValue).set("sticky-value"); - tx.value(rewindValue).set("rewind-value"); - tx.list(stickyList).append("sticky-list"); - tx.list(rewindList).append("rewind-list"); - tx.value(absentKey).set("absent"); - tx.value(emptyKey).set("empty"); - tx.value(sessionValue).set("session"); - }, ctx); - - expect(await session.getValue(stickyValue, undefined, ctx)).toBe("sticky-value"); - expect(await session.getValue(rewindValue, undefined, ctx)).toBe("rewind-value"); - expect((await session.readList(stickyList, undefined, ctx)).map((item) => item.value)).toEqual(["sticky-list"]); - expect((await session.readList(rewindList, undefined, ctx)).map((item) => item.value)).toEqual(["rewind-list"]); - expect(await session.getValue(absentKey, undefined, ctx)).toBe("absent"); - expect(await session.getValue(undefinedKey, undefined, ctx)).toBe("absent"); - expect(await session.getValue(emptyKey, undefined, ctx)).toBe("empty"); - expect(await session.getValue(sessionValue, undefined, ctx)).toBe("session"); - }); - - it("stores task snapshots and retires scratch and shared output", async () => { - const storage = MemoryStorage.create(); - const session = new Session(storage); - const ids = await session.commit((tx) => { - const conversation = tx.createConversation({}); - const producer = tx.task(taskSpec(conversation, { output: { kind: "test.output" } })); - const output = defineInternalList({ type: "shared", id: producer }, "pi.output"); - tx.list(output).append([["r", { text: "" }]]); - const consumer = tx.task(taskSpec(conversation, { output: { id: producer, kind: "test.output" } })); - return { conversation, producer, consumer }; - }, ctx); - expect(ids).toEqual({ conversation: 1, producer: 2, consumer: 4 }); - - const output = defineInternalList({ type: "shared", id: ids.producer }, "pi.output"); - const scratch = defineValue({ type: "task", taskId: ids.producer }, "test.scratch"); - await session.commit((tx) => { - tx.value(scratch).set({ n: 1 }); - tx.list(output).append([["s", ["text"], "done"]]); - tx.list(output).append([["s", ["status"], "complete"]]); - }, ctx); - - await setTask(session, ids.producer, (task) => ({ ...task, status: "running" })); - await setTask(session, ids.producer, (task) => ({ ...task, checkpoint: { phase: "working" } })); - await setTask(session, ids.producer, (task) => ({ - ...task, - status: "terminal", - outcome: { status: "completed", result: null }, - })); - await expect(session.getValue(scratch, undefined, ctx)).rejects.toBeInstanceOf(ScratchRetired); - await expect(storage.getValue(scratch, undefined, ctx)).rejects.toBeInstanceOf(ScratchRetired); - - await setTask(session, ids.consumer, (task) => ({ ...task, status: "running" })); - const replacement = await session.commit(async (tx) => { - const consumer = await tx.getTask(ids.consumer); - if (consumer === undefined || consumer.status === "terminal") throw new Error("Consumer is not live"); - tx.setTask({ ...consumer, status: "terminal", outcome: { status: "completed", result: null } }); - return tx.task(taskSpec(ids.conversation, { output: { id: ids.producer, kind: "test.output" } })); - }, ctx); - expect(await session.readList(output, undefined, ctx)).toHaveLength(3); - await setTask(session, replacement, (task) => ({ ...task, status: "running" })); - await setTask(session, replacement, (task) => ({ - ...task, - status: "terminal", - outcome: { status: "completed", result: null }, - })); - await expect(session.readList(output, undefined, ctx)).rejects.toBeInstanceOf(OutputRetired); - await expect(storage.readList(output, undefined, ctx)).rejects.toBeInstanceOf(OutputRetired); - }); - - it("filters and pages tasks", async () => { - const session = new Session(MemoryStorage.create()); - const ids = await session.commit((tx) => { - const conversation = tx.createConversation({}); - const otherConversation = tx.createConversation({}); - return { - conversation, - otherConversation, - first: tx.task(taskSpec(conversation)), - second: tx.task( - taskSpec(conversation, { kind: "other", background: true, output: { kind: "test.output" } }), - ), - third: tx.task(taskSpec(otherConversation)), - }; - }, ctx); - await setTask(session, ids.first, (task) => ({ ...task, abort: true })); - await setTask(session, ids.first, (task) => ({ ...task, status: "running" })); - await setTask(session, ids.first, (task) => ({ - ...task, - status: "terminal", - outcome: { status: "aborted", result: null }, - })); - - const first = await session.scanTasks({ limit: 1 }, ctx); - expect(first.items.map(({ id }) => id)).toEqual([ids.first]); - expect((await session.scanTasks({ cursor: first.next, limit: 1 }, ctx)).items.map(({ id }) => id)).toEqual([ - ids.second, - ]); - expect((await session.getTasks([ids.first, 999_999], ctx)).has(ids.first)).toBe(true); - expect( - (await session.scanTasks({ conversationIds: [ids.conversation], limit: 10 }, ctx)).items.map(({ id }) => id), - ).toEqual([ids.first, ids.second]); - expect((await session.scanTasks({ kind: "other", limit: 10 }, ctx)).items.map(({ id }) => id)).toEqual([ - ids.second, - ]); - expect((await session.scanTasks({ outputId: ids.second, limit: 10 }, ctx)).items.map(({ id }) => id)).toEqual([ - ids.second, - ]); - expect( - ( - await session.scanTasks( - { - conversationIds: [ids.conversation], - statuses: ["terminal"], - kind: "test.task", - abort: true, - limit: 10, - }, - ctx, - ) - ).items.map(({ id }) => id), - ).toEqual([ids.first]); - }); -}); diff --git a/packages/agent/test/harness/pico/tsconfig.json b/packages/agent/test/harness/pico/tsconfig.json deleted file mode 100644 index 71d638f7e..000000000 --- a/packages/agent/test/harness/pico/tsconfig.json +++ /dev/null @@ -1,7 +0,0 @@ -{ - "extends": "../../../../../tsconfig.json", - "compilerOptions": { - "noEmit": true - }, - "include": ["../../../src/harness/pico/**/*.ts", "./types.compile.ts"] -} diff --git a/packages/agent/test/harness/pico/types.compile.ts b/packages/agent/test/harness/pico/types.compile.ts deleted file mode 100644 index f6f154f62..000000000 --- a/packages/agent/test/harness/pico/types.compile.ts +++ /dev/null @@ -1,545 +0,0 @@ -import type { Context } from "@earendil-works/chord"; -import type { - AssistantMessage, - DeferredHandle, - Message, - RetryPolicy, - ToolCall, - ToolResultMessage, - Usage, - UserMessage, -} from "@earendil-works/pi-ai"; -import { defineInternalList, internalConversationValue } from "../../../src/harness/pico/addresses.ts"; -import type { - Acceptance, - AssistantEntryData, - BaseTaskTx, - BoundTxState, - ConversationTx, - Entry, - EntryBase, - EntryData, - EntryInput, - ExactJsonInput, - GenerationOutput, - HookHandler, - HookPoint, - HookResult, - HostTx, - InputOutcome, - JsonObject, - JsonValue, - List, - OutputState, - PayloadsOf, - PublicScope, - QueuedInput, - ReadonlyTaskOutput, - ResolvedValue, - RunningTask, - ScratchReader, - SectionRecord, - Stored, - StoredEntryDraft, - StoredInputResult, - SummaryEntry, - SystemEntryData, - TaskCheckpoint, - TaskKind, - TaskOf, - TaskOutput, - TaskOutputRef, - TaskRuntime, - TerminalClosure, - ToolControl, - ToolDiagnostic, - ToolInput, - ToolOutputState, - ToolResultData, - TxState, - UsageEntryData, - UserInput, -} from "../../../src/harness/pico/index.ts"; -import * as Pico from "../../../src/harness/pico/index.ts"; -import { - conversationList, - conversationValue, - defineEntry, - defineHookPoint, - defineList, - defineTask, - defineTaskOutput, - defineValue, - sessionValue, -} from "../../../src/harness/pico/index.ts"; -import type { CoreAbortTaskRuntime, CoreTaskRuntime, CoreTaskTx } from "../../../src/harness/pico/runtime.ts"; -import type { DisjointBundles } from "../../../src/harness/pico/tasks.ts"; -import * as TaskDeclarations from "../../../src/harness/pico/tasks.ts"; -import { defineCoreTask } from "../../../src/harness/pico/tasks.ts"; - -// @ts-expect-error privileged core authoring is not part of the public Pico surface -const publicDefineCoreTask = Pico.defineCoreTask; -// @ts-expect-error the task-kind brand is private even within the tasks module -const publicTaskKindBrand = TaskDeclarations.taskKindBrand; -void publicDefineCoreTask; -void publicTaskKindBrand; - -type RemovedPublicExports = [ - // @ts-expect-error low-level Session is internal - Pico.Session, - // @ts-expect-error low-level transaction is internal - Pico.Transaction, - // @ts-expect-error low-level task creation shape is internal - Pico.TaskCreate, - // @ts-expect-error stored conversation metadata is internal - Pico.StoredConversation, - // @ts-expect-error core kind construction types are internal - Pico.CoreTaskKind, - // @ts-expect-error erased core kind is internal - Pico.CoreKind, - // @ts-expect-error collision implementation helpers are internal - Pico.DisjointBundles, -]; -declare const removedPublicExports: RemovedPublicExports; -void removedPublicExports; - -type Equal = (() => Type extends Left ? 1 : 2) extends () => Type extends Right ? 1 : 2 - ? (() => Type extends Right ? 1 : 2) extends () => Type extends Left ? 1 : 2 - ? true - : false - : false; -type Assert = Condition; -type ExtendsJson = T; -type ExtendsJsonObject = T; - -type StoredAssertions = [ - ExtendsJsonObject>, - ExtendsJsonObject>, - ExtendsJsonObject>, - ExtendsJsonObject>, - ExtendsJsonObject>, - ExtendsJsonObject>, - ExtendsJsonObject>, - ExtendsJson>, - ExtendsJson, - ExtendsJson, - ExtendsJson, - ExtendsJson, - ExtendsJson, - ExtendsJson, - ExtendsJson, - ExtendsJson, - ExtendsJson, - ExtendsJson, - ExtendsJson, - Assert, - Assert, - Assert, - Assert<{ readonly value: undefined } extends JsonValue ? false : true>, -]; - -type Input = { readonly prompt: string; readonly nested: { readonly count: number } }; -type Checkpoint = { readonly phase: "ready"; readonly cursor: number }; -type Result = { readonly answer: string }; -type Failure = { readonly code: "failed" }; -type Aborted = { readonly cleaned: boolean }; -type Progress = { text: string; chunks: string[] }; - -const pluginEnabled = conversationValue("plugin.enabled", { rewind: false, default: false }); -const collected = defineHookPoint<{ readonly value: string }, { readonly seen: true }>({ - fold: "collect", - onThrow: "skip", -}); -const first = defineHookPoint<{ readonly value: string }, { readonly accepted: true }>({ - fold: "first", - onThrow: "skip", -}); -const beforeRun = defineHookPoint<{ readonly prompt: string }, { readonly prompt?: string }>({ - fold: "chain", - onThrow: "skip", -}); -const noReturnHandler: HookHandler = async () => {}; -void noReturnHandler; - -const ordinaryKind = defineTask()({ - kind: "plugin.ordinary", - config: { enabled: pluginEnabled }, - hooks: { beforeRun }, - execute() { - return Promise.resolve(() => ({ status: "completed", result: { answer: "done" } })); - }, - recover() { - return Promise.resolve(() => ({ status: "failed", failure: { code: "failed" } })); - }, - abort() { - return Promise.resolve(() => ({ cleaned: true })); - }, -}); - -const coreKind = defineCoreTask()({ - kind: "pi.test_core", - execute() { - return Promise.resolve(() => ({ status: "completed", result: { answer: "done" } })); - }, - recover() { - return Promise.resolve(() => ({ status: "failed", failure: { code: "failed" } })); - }, - abort() { - return Promise.resolve(() => ({ cleaned: true })); - }, -}); - -const progressOutput = defineTaskOutput("plugin.progress"); -const outputKind = defineTask()({ - kind: "plugin.output", - output: { kind: progressOutput, initial: (input) => ({ text: input.prompt, chunks: [] }) }, - execute() { - return Promise.resolve(() => ({ status: "completed", result: { answer: "done" } })); - }, - recover() { - return Promise.resolve(() => ({ status: "failed", failure: { code: "failed" } })); - }, - abort() { - return Promise.resolve(() => ({ cleaned: true })); - }, -}); - -type HookAssertions = [ - Assert>>, - Assert>>, - Assert>>, - Assert< - Equal< - HookResult, - { readonly outputs: readonly { readonly seen: true }[]; readonly threw?: unknown } - > - >, - Assert, { readonly output?: { readonly accepted: true }; readonly threw?: unknown }>>, -]; -type InferredHooks = PayloadsOf["hooks"]; -type KindAssertions = [ - Assert>, - Assert>, - Assert>, - Assert>, - Assert< - Equal< - Parameters[1], - TaskRuntime - > - >, - Assert[1], CoreTaskRuntime>>, - Assert[1]["output"], TaskOutput>>, - Assert[1]["output"], ReadonlyTaskOutput>>, - Assert< - Equal< - Extract, { status: "terminal" }>["outcome"]["status"], - "completed" | "failed" | "aborted" | "orphaned" - > - >, -]; - -function verifyNoOutputRuntime( - runtime: TaskRuntime, - abortRuntime: Parameters[1], - finalTx: Parameters>[0], - running: RunningTask, - ctx: Context, -): void { - // @ts-expect-error no-output runtimes have no output property - void runtime.output; - // @ts-expect-error no-output abort runtimes have no output property - void abortRuntime.output; - // @ts-expect-error no-output final transactions have no output property - void finalTx.output; - // @ts-expect-error no-output running tasks have no output property - void running.output; - void abortRuntime.sleep(1, ctx); -} - -function verifyOutput(output: TaskOutput, readonlyOutput: ReadonlyTaskOutput, ctx: Context): void { - void output.mutate((state) => { - state.chunks.push("next"); - }, ctx); - // @ts-expect-error output mutations must be synchronous - void output.mutate(async (state) => { - state.text = "invalid"; - }, ctx); - // @ts-expect-error fresh abort receives read-only output - void readonlyOutput.mutate; -} - -type JobOutput = { - stdout: string; - stderr: string; - droppedStdout: number; - droppedStderr: number; - exitCode?: number; - occurrence: number; -}; -type JobAssertions = [Assert]; -function verifyJobMutation(output: TaskOutput, snapshot: Readonly, ctx: Context): void { - void output.mutate((state) => { - state.stdout = snapshot.stdout; - }, ctx); -} - -type NoteEntry = EntryBase & EntryData<{ readonly text: string }>; -type PinnedEntry = EntryBase & - EntryData<{ readonly text: string }> & { - readonly model: readonly [Stored]; - }; -const noteKind = defineEntry("plugin.note"); -const pinnedKind = defineEntry("plugin.pinned"); -const summaryKind = defineEntry("pi.summary"); - -function verifyEntryInputs(tx: ConversationTx, text: string): void { - void tx.write(noteKind, { data: { text } }); - void tx.write(pinnedKind, { - data: { text }, - model: [{ role: "user", content: text, timestamp: 1 }], - }); - void tx.write(summaryKind, { - data: { through: 1 }, - model: [{ role: "user", content: text, timestamp: 1 }], - head: "self", - }); - // @ts-expect-error kind is supplied by the witness - void tx.write(noteKind, { kind: "plugin.note", data: { text } }); - // @ts-expect-error id is supplied by Pico - void tx.write(noteKind, { id: 1, data: { text } }); -} - -type EntryAssertions = [ - Assert, - Assert, - Assert, - Assert["head"], number | "self">>, -]; - -function verifyHostTaskCreation(tx: HostTx): void { - tx.task(ordinaryKind, { conversationId: 1, input: { prompt: "host", nested: { count: 1 } } }); - // @ts-expect-error host task creation requires a conversation ID - tx.task(ordinaryKind, { input: { prompt: "host", nested: { count: 1 } } }); -} - -function verifyTaskCreation( - tx: BaseTaskTx, - coreTx: CoreTaskTx, - outputRef: TaskOutputRef, -): void { - tx.task(ordinaryKind, { input: { prompt: "literal", nested: { count: 1 } } }); - const exact = { input: { prompt: "variable", nested: { count: 1 } } }; - tx.task(ordinaryKind, exact); - tx.task(ordinaryKind, { ...exact }); - // @ts-expect-error ordinary transactions cannot create core tasks - tx.task(coreKind, { input: { prompt: "core", nested: { count: 1 } } }); - coreTx.task(coreKind, { input: { prompt: "core", nested: { count: 1 } } }); - - // @ts-expect-error input literals reject visible top-level extras - tx.task(ordinaryKind, { input: { prompt: "extra", nested: { count: 1 }, extra: 1 } }); - const extraInput = { prompt: "extra", nested: { count: 1 }, extra: 1 }; - // @ts-expect-error input variables reject visible top-level extras - tx.task(ordinaryKind, { input: extraInput }); - const inputBase = { prompt: "extra", nested: { count: 1 } }; - // @ts-expect-error input spreads reject visible top-level extras - tx.task(ordinaryKind, { input: { ...inputBase, extra: 1 } }); - const extraSpec = { input: { prompt: "extra", nested: { count: 1 } }, extra: 1 }; - // @ts-expect-error specs reject visible extras - tx.task(ordinaryKind, extraSpec); - // @ts-expect-error spread specs reject visible extras - tx.task(ordinaryKind, { ...extraSpec }); - - // Deep exactness is outside the contract. - tx.task(ordinaryKind, { input: { prompt: "nested", nested: { count: 1, extension: true } } }); - tx.task(outputKind, { input: { prompt: "output", nested: { count: 1 } }, output: outputRef }); - // @ts-expect-error no-output kinds reject output refs - tx.task(ordinaryKind, { input: { prompt: "output", nested: { count: 1 } }, output: outputRef }); -} - -function verifyAddresses(): void { - const rewindable = conversationValue("plugin.plan", { rewind: true }); - const defaulted = conversationValue("plugin.plan", { key: "defaulted", rewind: true, default: false }); - const sticky = conversationValue("plugin.mode", { rewind: false }); - const events = conversationList("plugin.events", { rewind: true }); - const plainSession = sessionValue("plugin.plain-name"); - const defaultedSession = sessionValue("plugin.name", { default: "pico" }); - const directPlainSession = defineValue({ type: "session" }, "plugin.direct-session"); - const directDefaultedSession = defineValue({ type: "session" }, "plugin.direct-session-default", { - default: "session", - }); - const directDefaultedTask = defineValue({ type: "task", taskId: 1 }, "plugin.direct-task-default", { - default: "task", - }); - const directPlainConversation = defineValue( - { type: "conversation", conversationId: 1 }, - "plugin.direct-conversation", - { rewind: true }, - ); - const directDefaultedConversation = defineValue( - { type: "conversation", conversationId: 1 }, - "plugin.direct-conversation-default", - { rewind: false, default: false }, - ); - const bound = rewindable.bind(1); - const defaultedBound = defaulted.bind(1); - const internalDefaulted = internalConversationValue("pi.internal-default", { - rewind: true, - default: "internal", - }); - const internalDefaultedBound = internalDefaulted.bind(1); - const boundList: List = events.bind(1); - void sticky; - void bound; - void defaultedBound; - void internalDefaulted; - void internalDefaultedBound; - void boundList; - void plainSession; - void defaultedSession; - void directPlainSession; - void directDefaultedSession; - void directDefaultedTask; - void directPlainConversation; - void directDefaultedConversation; - - type AddressAssertions = [ - Assert>, - Assert>, - Assert, string | undefined>>, - Assert, string>>, - Assert, string | undefined>>, - Assert, string>>, - Assert, string>>, - Assert, boolean | undefined>>, - Assert, boolean>>, - Assert>, - Assert>, - Assert>, - Assert, boolean | undefined>>, - Assert, boolean>>, - Assert, boolean | undefined>>, - Assert, boolean>>, - Assert>, - Assert, string>>, - Assert, string>>, - Assert>, - Assert extends never ? true : false>, - ]; - const assertions = null as unknown as AddressAssertions; - void assertions; - - // @ts-expect-error public bound constructors do not accept shared scope - defineValue({ type: "shared", id: 1 }, "plugin.shared"); - // @ts-expect-error public bound constructors do not accept shared scope - defineList({ type: "shared", id: 1 }, "plugin.shared"); - // @ts-expect-error conversation addresses require an explicit rewind policy - defineValue({ type: "conversation", conversationId: 1 }, "plugin.value"); - // @ts-expect-error session addresses are sticky - defineValue({ type: "session" }, "plugin.value", { rewind: true }); - defineInternalList({ type: "shared", id: 1 }, "pi.output"); -} - -function verifyResolvedTransactionReads(tx: TxState, boundTx: BoundTxState, scratch: ScratchReader): void { - const plainSession = defineValue({ type: "session" }, "plugin.tx-plain"); - const defaultedSession = defineValue({ type: "session" }, "plugin.tx-default", { default: "x" }); - const defaultedTask = defineValue({ type: "task", taskId: 1 }, "plugin.scratch-default", { - default: 1, - }); - const defaultedConversation = defineValue( - { type: "conversation", conversationId: 1 }, - "plugin.tx-conversation-default", - { rewind: true, default: false }, - ); - const plainDefinition = conversationValue("plugin.bound-plain", { rewind: true }); - const defaultedDefinition = conversationValue("plugin.bound-default", { rewind: false, default: 1 }); - const plainRead = tx.value(plainSession).get(); - const defaultedRead = tx.value(defaultedSession).get(); - const defaultedConversationRead = tx.value(defaultedConversation).get(); - const plainDefinitionRead = boundTx.value(plainDefinition).get(); - const defaultedDefinitionRead = boundTx.value(defaultedDefinition).get(); - const scratchRead = scratch.value(defaultedTask).get(); - type ReadAssertions = [ - Assert, string | undefined>>, - Assert, string>>, - Assert, boolean>>, - Assert, number | undefined>>, - Assert, number>>, - Assert, number>>, - ]; - const assertions = null as unknown as ReadAssertions; - void assertions; -} - -type FakeDuplicateConfigs = { one: { value: 1 }; two: { value: 2 } }; -type FakeReservedConfig = { one: { set: 1 }; two: Record }; -type FakeDuplicateHooks = { one: { before: 1 }; two: { before: 2 } }; -type RealConfigs = { one: { model: 1 }; two: { retry: 2 } }; -type RealHooks = { one: { before: 1 }; two: { after: 2 } }; -type BundleAssertions = [Assert>, Assert>]; -type BadBundleAssertions = [ - // @ts-expect-error duplicate bundle keys are rejected by the assertion - Assert>, - // @ts-expect-error reserved config keys are rejected by the assertion - Assert>, - // @ts-expect-error duplicate hook keys are rejected by the assertion - Assert>, -]; - -type ExactInputAssertion = Assert< - Equal["extension"], never> ->; -type CheckpointAssertion = Assert< - { readonly phase: "deferred"; readonly handle: Stored } extends TaskCheckpoint ? true : false ->; -type HydrationAssertions = [ - Assert["input"] extends Entry ? true : false>, - Assert< - Equal< - Extract["answer"] extends infer A | undefined - ? NonNullable extends { message: infer M } - ? M - : never - : never, - AssistantMessage - > - >, - ExtendsJson, -]; -type KindShape = TaskKind; -type RunningAssertion = Assert<"output" extends keyof RunningTask ? false : true>; - -declare const storedAssertions: StoredAssertions; -declare const hookAssertions: HookAssertions; -declare const kindAssertions: KindAssertions; -declare const jobAssertions: JobAssertions; -declare const entryAssertions: EntryAssertions; -declare const bundleAssertions: BundleAssertions; -declare const badBundleAssertions: BadBundleAssertions; -declare const exactInputAssertion: ExactInputAssertion; -declare const checkpointAssertion: CheckpointAssertion; -declare const hydrationAssertions: HydrationAssertions; -declare const kindShape: KindShape; -declare const runningAssertion: RunningAssertion; -void storedAssertions; -void hookAssertions; -void kindAssertions; -void jobAssertions; -void entryAssertions; -void bundleAssertions; -void badBundleAssertions; -void exactInputAssertion; -void checkpointAssertion; -void hydrationAssertions; -void kindShape; -void runningAssertion; -void verifyNoOutputRuntime; -void verifyOutput; -void verifyJobMutation; -void verifyEntryInputs; -void verifyHostTaskCreation; -void verifyTaskCreation; -void verifyAddresses; -void verifyResolvedTransactionReads; -void (null as Message | null); -void (null as CoreAbortTaskRuntime | null);