mirror of
https://github.com/badlogic/pi-mono.git
synced 2026-11-05 23:37:25 +00:00
docs(agent): use typed register tokens
This commit is contained in:
Mario Zechner
parent
8930b7daf4
commit
1e19edd133
1 changed files with 431 additions and 140 deletions
|
|
@ -1,54 +1,223 @@
|
|||
# Scalar and list registers — implementation handoff
|
||||
# Typed scalar and list registers — implementation handoff
|
||||
|
||||
This document specifies the storage primitive only. It is intended to be implemented before tool durability and before any harness-native streaming tool API.
|
||||
This document specifies the register storage primitive used by the harness.
|
||||
|
||||
The existing register store becomes two statically typed namespace kinds:
|
||||
Registers are identified by ordinary typed token values rather than a global `RegisterValues` or `ListRegisterValues` namespace map:
|
||||
|
||||
- **scalar:** one current value per key; `set` replaces it;
|
||||
- **list:** immutable elements appended under a key and read in bounded sequence pages.
|
||||
- **scalar register:** one current value per `(token, key)`; set replaces it;
|
||||
- **list register:** immutable values appended under `(token, key)` and read in bounded sequence pages.
|
||||
|
||||
List elements are append-only. Deleting a whole list key remains necessary for lifecycle cleanup. There is no element update, element delete, insertion, or truncation.
|
||||
Built-in tokens live together in `packages/agent/src/harness/session/registers.ts` and are imported directly by their consumers. There is no `HarnessRegisters` bundle, runtime catalog, declaration merging, or raw register access from tools.
|
||||
|
||||
## Goals
|
||||
|
||||
1. Preserve the existing scalar register API and behavior.
|
||||
2. Append a list element without reading the existing list.
|
||||
3. Use the existing transaction sequence as the element cursor and order.
|
||||
4. Produce identical pages on Memory, JSONL, and SQLite.
|
||||
5. Keep list reads bounded and explicit.
|
||||
6. Allow list writes to commit atomically with entries, usage, and scalar writes.
|
||||
1. Keep persisted register IDs stable while moving their value types onto tokens.
|
||||
2. Allow built-in and future application register definitions without editing a global type map.
|
||||
3. Preserve type inference through reads, writes, scans, and list appends.
|
||||
4. Preserve existing scalar replacement semantics.
|
||||
5. Append a list element without reading or rewriting existing elements.
|
||||
6. Use the existing transaction sequence as list order and cursor.
|
||||
7. Produce identical logical behavior on Memory, JSONL, and SQLite.
|
||||
8. Keep list reads bounded and explicit.
|
||||
9. Commit scalar/list writes atomically with entries and usage.
|
||||
|
||||
## Non-goals
|
||||
|
||||
This slice does not define:
|
||||
|
||||
- tool output or assistant stream element types;
|
||||
- retention catalogs;
|
||||
- assistant-frame contents or reduction semantics;
|
||||
- tool progress semantics;
|
||||
- per-token runtime value validation;
|
||||
- per-element or per-key byte limits;
|
||||
- empty-list existence markers;
|
||||
- list truncation;
|
||||
- audit logging;
|
||||
- a generic event log or reducer for operation recovery.
|
||||
- list truncation or per-element deletion;
|
||||
- a generic event log or operation-state reducer;
|
||||
- dynamic registration, token catalogs, or dependency-injected built-in token bundles.
|
||||
|
||||
Consumers own their key grammar, content limits, cleanup points, and restore hydration policy.
|
||||
Consumers own key grammar, content limits, cleanup points, and restore hydration policy. Values remain trusted typed in-process data under the validation boundary in `harness.md`.
|
||||
|
||||
## Types
|
||||
## Token model
|
||||
|
||||
Keep the existing scalar value map and APIs. Do not rename `RegisterValues`, `Register`, `getRegister`, or `listRegisters` as part of this slice.
|
||||
A token carries its register kind, stable persisted ID, and compile-time value type:
|
||||
|
||||
```ts
|
||||
export interface RegisterValues {
|
||||
// Existing scalar namespaces, unchanged.
|
||||
declare const registerValueType: unique symbol;
|
||||
|
||||
interface RegisterTokenBase {
|
||||
readonly id: string;
|
||||
readonly kind: "scalar" | "list";
|
||||
}
|
||||
|
||||
/** The value is one element, not an array. */
|
||||
export interface ListRegisterValues {
|
||||
// Concrete namespaces are added by their consuming slices.
|
||||
export interface ScalarRegisterToken<T> extends RegisterTokenBase {
|
||||
readonly kind: "scalar";
|
||||
/** Compile-time only; makes the value type part of the token. */
|
||||
readonly [registerValueType]?: (value: T) => T;
|
||||
}
|
||||
|
||||
export type RegisterNamespace = keyof RegisterValues;
|
||||
export type ListRegisterNamespace = keyof ListRegisterValues;
|
||||
export interface ListRegisterToken<T> extends RegisterTokenBase {
|
||||
readonly kind: "list";
|
||||
/** Compile-time only; T is one element, not the whole list. */
|
||||
readonly [registerValueType]?: (value: T) => T;
|
||||
}
|
||||
|
||||
export function scalarRegister<T>(id: string): ScalarRegisterToken<T> {
|
||||
validateRegisterId(id);
|
||||
return Object.freeze({ id, kind: "scalar" });
|
||||
}
|
||||
|
||||
export function listRegister<T>(id: string): ListRegisterToken<T> {
|
||||
validateRegisterId(id);
|
||||
return Object.freeze({ id, kind: "list" });
|
||||
}
|
||||
```
|
||||
|
||||
The phantom function keeps the type invariant: a token for one value type cannot silently widen to a token for another. It has no runtime field.
|
||||
|
||||
A register ID must be non-empty and must not contain the internal Memory-map separator (`\u0000`). It is serialized into JSONL and SQLite and therefore remains stable across reopen. Changing an ID or changing its scalar/list kind requires a storage migration.
|
||||
|
||||
Token object identity carries no durability meaning. Storage addresses a register by `(kind, token.id, key)`.
|
||||
|
||||
### No global value map
|
||||
|
||||
Delete these shapes:
|
||||
|
||||
```ts
|
||||
interface RegisterValues { /* namespace → value */ }
|
||||
interface ListRegisterValues { /* namespace → element */ }
|
||||
|
||||
type RegisterNamespace = keyof RegisterValues;
|
||||
type ListRegisterNamespace = keyof ListRegisterValues;
|
||||
```
|
||||
|
||||
A type belongs to its token regardless of where the token is defined:
|
||||
|
||||
```ts
|
||||
export const laneLeaf = scalarRegister<string | null>("lane.leaf");
|
||||
export const applicationState = scalarRegister<MyApplicationState>("my-app.state");
|
||||
```
|
||||
|
||||
The storage API does not need to know either type centrally.
|
||||
|
||||
## Built-in tokens
|
||||
|
||||
Define built-ins directly in `packages/agent/src/harness/session/registers.ts`:
|
||||
|
||||
```ts
|
||||
export const laneLeaf = scalarRegister<string | null>("lane.leaf");
|
||||
export const laneConfig = scalarRegister<LaneConfiguration>("lane.config");
|
||||
export const laneState = scalarRegister<LaneState>("lane.state");
|
||||
export const laneLastResult = scalarRegister<LaneLastResult>("lane.lastResult");
|
||||
|
||||
export const operationMeta = scalarRegister<Operation>("op.meta");
|
||||
export const operationState = scalarRegister<OperationState>("op.state");
|
||||
export const operationToolArgs =
|
||||
scalarRegister<Record<string, JsonValue>>("op.tool_args");
|
||||
export const operationToolFact = scalarRegister<JsonValue>("op.tool_fact");
|
||||
export const operationPreparation =
|
||||
scalarRegister<DurableStructuralPreparation>("op.preparation");
|
||||
|
||||
export const pendingEntry = scalarRegister<PendingEntry>("pending.entry");
|
||||
export const pendingToolOutput =
|
||||
scalarRegister<AgentToolResult<unknown>>("pending.tool_output");
|
||||
|
||||
/** Compact frames defined by the assistant-partial durability slice. */
|
||||
export const pendingAssistantFrames =
|
||||
listRegister<DurableAssistantFrame>("pending.assistant_frame");
|
||||
|
||||
export const factName = scalarRegister<string>("fact.name");
|
||||
export const factLabel = scalarRegister<string>("fact.label");
|
||||
export const factCustom = scalarRegister<JsonValue>("fact.custom");
|
||||
```
|
||||
|
||||
These are normal constants, not a configuration object. Harness code imports the token it uses:
|
||||
|
||||
```ts
|
||||
await reader.getRegister(operationState, operationId);
|
||||
await reader.readList(pendingAssistantFrames, frameKey, options);
|
||||
```
|
||||
|
||||
A future application token may be defined in the application's own `registers.ts`. Persisted IDs must be globally unique within a session. Defining two tokens with the same `(kind, id)` but different TypeScript value types is a trusted-programming defect. Defining scalar and list tokens with the same ID is also a defect.
|
||||
|
||||
Tests assert that all built-in exported tokens have unique IDs and fixed kinds.
|
||||
|
||||
## Typed scalar API
|
||||
|
||||
```ts
|
||||
export interface ScalarRegisterValue<T> {
|
||||
key: string;
|
||||
value: T;
|
||||
seq: number;
|
||||
}
|
||||
|
||||
interface SessionReader {
|
||||
getRegister<T>(
|
||||
register: ScalarRegisterToken<T>,
|
||||
key: string,
|
||||
): Promise<ScalarRegisterValue<T> | undefined>;
|
||||
|
||||
listRegisters<T>(
|
||||
register: ScalarRegisterToken<T>,
|
||||
keyPrefix?: string,
|
||||
): Promise<ScalarRegisterValue<T>[]>;
|
||||
}
|
||||
```
|
||||
|
||||
`listRegisters` lists scalar keys under one token; it is unrelated to list-register element reads. It retains its existing bounded-use contract: harness callers use exact operation-owned prefixes, and no public API exposes an unrestricted session dump.
|
||||
|
||||
Register writes are created through typed helpers. The stored transaction representation is erased only after the helper checks the token/value relationship:
|
||||
|
||||
```ts
|
||||
interface RegisterSetWrite {
|
||||
kind: "register";
|
||||
op: "set";
|
||||
namespace: string;
|
||||
key: string;
|
||||
value: unknown;
|
||||
}
|
||||
|
||||
interface RegisterDeleteWrite {
|
||||
kind: "register";
|
||||
op: "delete";
|
||||
namespace: string;
|
||||
key: string;
|
||||
}
|
||||
|
||||
export function setRegister<T>(
|
||||
register: ScalarRegisterToken<T>,
|
||||
key: string,
|
||||
value: NoInfer<T>,
|
||||
): RegisterSetWrite {
|
||||
return {
|
||||
kind: "register",
|
||||
op: "set",
|
||||
namespace: register.id,
|
||||
key,
|
||||
value,
|
||||
};
|
||||
}
|
||||
|
||||
export function deleteRegister<T>(
|
||||
register: ScalarRegisterToken<T>,
|
||||
key: string,
|
||||
): RegisterDeleteWrite {
|
||||
if (register.kind !== "scalar") throw new Error("Expected scalar register");
|
||||
return {
|
||||
kind: "register",
|
||||
op: "delete",
|
||||
namespace: register.id,
|
||||
key,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
`NoInfer<T>` makes the token authoritative. TypeScript must not widen `T` from an incompatible value argument.
|
||||
|
||||
The raw write interfaces are storage internals. Harness and application code use `setRegister()` and `deleteRegister()` rather than manually constructing erased writes.
|
||||
|
||||
## Typed list API
|
||||
|
||||
A list token's `T` is one immutable element:
|
||||
|
||||
```ts
|
||||
export interface ListRegisterElement<T> {
|
||||
/** Global transaction-write sequence assigned by storage. */
|
||||
seq: number;
|
||||
|
|
@ -67,87 +236,105 @@ export interface ListReadOptions {
|
|||
/** Default: 1,000. Maximum: 10,000. */
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
interface SessionReader {
|
||||
readList<T>(
|
||||
register: ListRegisterToken<T>,
|
||||
key: string,
|
||||
options?: ListReadOptions,
|
||||
): Promise<ListRegisterElement<T>[]>;
|
||||
}
|
||||
```
|
||||
|
||||
Values remain trusted in-process values under the validation boundary in `harness.md`. Storage serializes them where required but does not runtime-validate their application shape or defensively clone them.
|
||||
|
||||
## Writes
|
||||
|
||||
One append write carries one element:
|
||||
List writes also use typed helpers:
|
||||
|
||||
```ts
|
||||
export type ListRegisterAppendWrite = {
|
||||
[N in ListRegisterNamespace]: {
|
||||
kind: "list-register";
|
||||
op: "append";
|
||||
namespace: N;
|
||||
key: string;
|
||||
value: ListRegisterValues[N];
|
||||
};
|
||||
}[ListRegisterNamespace];
|
||||
interface ListRegisterAppendWrite {
|
||||
kind: "list-register";
|
||||
op: "append";
|
||||
namespace: string;
|
||||
key: string;
|
||||
value: unknown;
|
||||
}
|
||||
|
||||
export type Write =
|
||||
| ExistingWrite
|
||||
| ListRegisterAppendWrite
|
||||
| {
|
||||
kind: "list-register";
|
||||
op: "delete";
|
||||
namespace: ListRegisterNamespace;
|
||||
key: string;
|
||||
};
|
||||
interface ListRegisterDeleteWrite {
|
||||
kind: "list-register";
|
||||
op: "delete";
|
||||
namespace: string;
|
||||
key: string;
|
||||
}
|
||||
|
||||
export function appendList<T>(
|
||||
register: ListRegisterToken<T>,
|
||||
key: string,
|
||||
value: NoInfer<T>,
|
||||
): ListRegisterAppendWrite {
|
||||
return {
|
||||
kind: "list-register",
|
||||
op: "append",
|
||||
namespace: register.id,
|
||||
key,
|
||||
value,
|
||||
};
|
||||
}
|
||||
|
||||
export function deleteList<T>(
|
||||
register: ListRegisterToken<T>,
|
||||
key: string,
|
||||
): ListRegisterDeleteWrite {
|
||||
if (register.kind !== "list") throw new Error("Expected list register");
|
||||
return {
|
||||
kind: "list-register",
|
||||
op: "delete",
|
||||
namespace: register.id,
|
||||
key,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
A transaction appending several elements contains several append writes. This deliberately reuses the existing rule that every write receives its own strictly increasing sequence number. No ordinal or list-local counter is needed.
|
||||
`Write` adds the four erased scalar/list write representations alongside entries and usage. A transaction may mix every write kind atomically.
|
||||
|
||||
## List write semantics
|
||||
|
||||
One append write carries one element. A transaction appending several elements contains several append writes. Every write receives its existing globally increasing transaction sequence:
|
||||
|
||||
```text
|
||||
TX[
|
||||
append list/ns/key = A, // seq 41
|
||||
set scalar/ns/key = X, // seq 42
|
||||
append list/ns/key = B // seq 43
|
||||
append pendingAssistantFrames/O:R = A, // seq 41
|
||||
set operationState/O = X, // seq 42
|
||||
append pendingAssistantFrames/O:R = B // seq 43
|
||||
]
|
||||
```
|
||||
|
||||
The elements under the list key are ordered `A, B` by sequence. Gaps caused by unrelated writes are irrelevant.
|
||||
Elements under list key `O:R` are ordered `A, B` by sequence. Gaps caused by unrelated writes are irrelevant.
|
||||
|
||||
Rules:
|
||||
|
||||
- append never reads the list;
|
||||
- whole-key delete removes every element under `(namespace, key)`;
|
||||
- append never reads existing elements;
|
||||
- an element is immutable after commit;
|
||||
- whole-key delete removes every element under `(token.id, key)`;
|
||||
- delete of an absent list is a no-op;
|
||||
- delete followed by append in one transaction creates a fresh list atomically;
|
||||
- all validation and serialization needed to admit a transaction occurs before any Memory state is changed;
|
||||
- there is no per-element update, delete, insertion, or truncation;
|
||||
- all validation and serialization needed to admit a transaction completes before Memory state changes;
|
||||
- a failed transaction exposes none of its list or non-list writes.
|
||||
|
||||
“Append-only” describes an element while its key exists. Whole-key deletion is a lifecycle operation, not an element mutation.
|
||||
“Append-only” describes elements while the key exists. Whole-key deletion is lifecycle cleanup, not element mutation.
|
||||
|
||||
## Reads
|
||||
|
||||
Add the same method to `Storage`, `SessionReader`, and `SessionMutator`:
|
||||
|
||||
```ts
|
||||
readList<N extends ListRegisterNamespace>(
|
||||
namespace: N,
|
||||
key: string,
|
||||
options?: ListReadOptions,
|
||||
): Promise<ListRegisterElement<ListRegisterValues[N]>[]>;
|
||||
```
|
||||
|
||||
Semantics:
|
||||
## List read semantics
|
||||
|
||||
- ascending reads return `seq > cursor.seq`;
|
||||
- descending reads return `seq < cursor.seq`;
|
||||
- results are ordered according to `order` before `limit` is applied;
|
||||
- absent and empty are both represented as `[]`;
|
||||
- absent and empty both return `[]`;
|
||||
- callers continue with the last returned element's `seq`;
|
||||
- an empty page ends iteration;
|
||||
- `limit` must be a positive safe integer, defaults to 1,000, and is capped at 10,000.
|
||||
|
||||
Example:
|
||||
|
||||
```ts
|
||||
let cursor: ListCursor | undefined;
|
||||
while (true) {
|
||||
const page = await reader.readList("example.items", "k", {
|
||||
const page = await reader.readList(pendingAssistantFrames, frameKey, {
|
||||
cursor,
|
||||
order: "asc",
|
||||
limit: 100,
|
||||
|
|
@ -158,43 +345,96 @@ while (true) {
|
|||
}
|
||||
```
|
||||
|
||||
A cursor is a sequence filter, not a snapshot or key-incarnation token. Concurrent later appends may appear on later ascending pages. A whole-key delete may make a cursor stale; reads simply apply its sequence comparison to the current elements.
|
||||
A cursor is a sequence filter, not a snapshot or key-incarnation token. Concurrent later appends may appear on later ascending pages. A whole-key delete may make a cursor stale; reads simply apply its sequence comparison to the currently surviving elements.
|
||||
|
||||
Do not add an unbounded “read the whole list” helper.
|
||||
|
||||
## Assistant partial frames
|
||||
|
||||
Assistant partial durability is the first list consumer:
|
||||
|
||||
```text
|
||||
register: pendingAssistantFrames
|
||||
key: {operationId}:{responseEntryId}
|
||||
value: one bounded DurableAssistantFrame
|
||||
```
|
||||
|
||||
The frame type is finalized with the assistant-stream design, not by storage. It must be a compact incremental frame. Do not append raw `AssistantMessageEvent`: update events contain a growing `partial` message and would reintroduce write amplification inside the list elements.
|
||||
|
||||
The assistant procedure may batch several frame appends into one transaction, but each frame remains an independently sequenced list element. Its scalar `effect_pending` state remains authoritative.
|
||||
|
||||
On restore:
|
||||
|
||||
1. restore and validate scalar lane/operation state;
|
||||
2. derive the exact frame key from the valid assistant effect-pending state;
|
||||
3. read bounded pages from `pendingAssistantFrames`;
|
||||
4. reduce frames into the latest durable partial assistant message.
|
||||
|
||||
A missing frame list is valid and means no partial frame committed. Frames never prove provider completion and never select a restart state.
|
||||
|
||||
Final assistant settlement atomically commits the complete immutable assistant entry and deletes the whole frame-list key:
|
||||
|
||||
```text
|
||||
TX[
|
||||
insert final assistant entry,
|
||||
insert usage,
|
||||
deleteList(pendingAssistantFrames, O:R),
|
||||
set operationState/O = next state
|
||||
]
|
||||
```
|
||||
|
||||
Unknown-effect recovery may reduce the latest durable frames into its synthetic interrupted assistant response, but that policy belongs in the assistant durability specification.
|
||||
|
||||
## Restore policy
|
||||
|
||||
Scalar state remains authoritative:
|
||||
Scalar state remains authoritative for every list consumer:
|
||||
|
||||
1. restore scalar lane and operation registers;
|
||||
2. perform current-state semantic validation;
|
||||
3. derive any auxiliary list keys from that valid scalar state;
|
||||
3. derive auxiliary list keys from valid scalar state;
|
||||
4. optionally hydrate bounded list pages for snapshots, diagnostics, or activation.
|
||||
|
||||
A missing list never invalidates an otherwise valid operation. List contents never prove that an external effect completed and never select the operation restart point.
|
||||
A missing list never invalidates an otherwise valid operation. List contents never prove that an external effect completed.
|
||||
|
||||
This permits auxiliary append streams to be restored without making an unbounded list part of current-state validation. Each concrete consumer must define a hydration byte/page budget.
|
||||
Each concrete list consumer defines:
|
||||
|
||||
- key grammar;
|
||||
- element and total byte bounds;
|
||||
- page/hydration budget;
|
||||
- cleanup transitions;
|
||||
- fork and migration policy.
|
||||
|
||||
## Memory backend
|
||||
|
||||
Use one map keyed like scalar registers:
|
||||
Keep scalar registers in the existing map, now addressed by token ID:
|
||||
|
||||
```ts
|
||||
const scalars = new Map<string, ScalarRegisterValue<unknown>>();
|
||||
// map key: `${register.id}\u0000${key}`
|
||||
```
|
||||
|
||||
Add list storage:
|
||||
|
||||
```ts
|
||||
type StoredListElement = { seq: number; value: unknown };
|
||||
const lists = new Map<string, StoredListElement[]>();
|
||||
// map key: `${namespace}\u0000${key}`
|
||||
// map key: `${register.id}\u0000${key}`
|
||||
```
|
||||
|
||||
- append pushes the already-sequenced element;
|
||||
- delete removes the map entry;
|
||||
- read filters by the exclusive cursor and slices to the validated limit;
|
||||
- transaction preparation must complete before mutating entries, scalar registers, lists, usage, or stats.
|
||||
- scalar set replaces the map value;
|
||||
- scalar delete removes it;
|
||||
- list append pushes the already-sequenced element;
|
||||
- list delete removes the complete array;
|
||||
- list read filters by exclusive cursor and slices to the validated limit;
|
||||
- transaction preparation completes before entries, scalar registers, lists, usage, or stats mutate.
|
||||
|
||||
The storage snapshot used by JSONL/fork tooling must include current list elements with their original sequence numbers.
|
||||
Storage snapshots used by JSONL/fork tooling include current scalar values and surviving list elements with original sequence numbers.
|
||||
|
||||
## SQLite backend
|
||||
|
||||
Add a schema migration rather than modifying an already-applied migration:
|
||||
The existing scalar register table continues to store `token.id` in its namespace column. No scalar schema change is required.
|
||||
|
||||
Add a schema migration for lists rather than modifying an applied migration:
|
||||
|
||||
```sql
|
||||
CREATE TABLE list_registers (
|
||||
|
|
@ -228,72 +468,111 @@ DELETE FROM list_registers
|
|||
WHERE namespace = ? AND key = ?;
|
||||
```
|
||||
|
||||
For a missing cursor, omit the sequence predicate. All writes run in the existing `BEGIN IMMEDIATE` transaction under the existing writer lease. No marker table, counter row, additional connection, or lock order is needed.
|
||||
For a missing cursor, omit the sequence predicate. All writes run in the existing `BEGIN IMMEDIATE` transaction under the writer lease. No marker table, counter row, connection, or new lock order is needed.
|
||||
|
||||
Assert with `EXPLAIN QUERY PLAN` that paging uses the primary key and does not sort through a temporary b-tree.
|
||||
Assert with `EXPLAIN QUERY PLAN` that paging uses the primary key and no temporary sort.
|
||||
|
||||
## JSONL backend
|
||||
|
||||
Committed records retain the write sequence:
|
||||
Scalar records retain their current persisted namespace string. Typed tokens are an API change, not a JSONL format rewrite.
|
||||
|
||||
List records use the token ID as `namespace`:
|
||||
|
||||
```jsonl
|
||||
{"kind":"list-register","op":"append","seq":41,"namespace":"example.items","key":"k","value":{"text":"a"}}
|
||||
{"kind":"list-register","op":"delete","seq":52,"namespace":"example.items","key":"k"}
|
||||
{"kind":"list-register","op":"append","seq":41,"namespace":"pending.assistant_frame","key":"O:R","value":{"type":"text_delta","contentIndex":0,"delta":"hi"}}
|
||||
{"kind":"list-register","op":"delete","seq":52,"namespace":"pending.assistant_frame","key":"O:R"}
|
||||
```
|
||||
|
||||
Replay folds them into the Memory list map:
|
||||
Replay folds records into the Memory list map:
|
||||
|
||||
- append adds `{ seq, value }`;
|
||||
- delete removes the complete map entry.
|
||||
- delete removes the complete list key.
|
||||
|
||||
A transaction continues to occupy one physical JSONL line, using an array when it has multiple writes. Torn-tail handling therefore remains atomic for list writes without a new framing mechanism.
|
||||
A transaction remains one physical JSONL line, using an array for multiple writes. Torn-tail handling therefore remains atomic without new framing.
|
||||
|
||||
### Snapshot compaction
|
||||
|
||||
Compaction must preserve list cursors. Write every surviving list element as an append record carrying its original `seq`, merged into sequence order with surviving entries, scalar registers, and usage rows.
|
||||
Compaction preserves list cursors. Write every surviving element as an append record carrying its original `seq`, merged in sequence order with surviving entries, scalar registers, and usage rows.
|
||||
|
||||
Do not collapse a live list into one synthetic append and do not assign new sequence numbers. Either action would change cursors and backend ordering.
|
||||
Do not collapse a live list into one synthetic append or assign new sequence numbers. Either changes cursors and backend ordering.
|
||||
|
||||
Deleted lists produce no snapshot records. Preserve the storage sequence high-water mark using the same mechanism as the rest of JSONL snapshot compaction; dropping the latest delete must not permit sequence reuse after reopen.
|
||||
Deleted lists produce no snapshot records. Preserve the storage sequence high-water mark through the existing mechanism so dropping the latest delete cannot permit sequence reuse.
|
||||
|
||||
## Forks and rewrites
|
||||
|
||||
The generic storage primitive does not decide whether a list namespace is copied. Session-level fork and precise-rewrite code must make an explicit decision for each concrete namespace.
|
||||
Fork and precise-rewrite code decides policy per concrete token, using token identity in source code and the stable ID in storage.
|
||||
|
||||
Operation-owned auxiliary list namespaces are not copied into an idle fork. A future explicit application list may choose different behavior.
|
||||
Operation-owned tokens are not copied into an idle fork:
|
||||
|
||||
Precise rewrites that retain a list element preserve its sequence cursor unless that rewrite explicitly defines sequence remapping for the entire destination store.
|
||||
- `operationMeta` and other `op.*` scalar tokens;
|
||||
- `pendingEntry` and `pendingToolOutput`;
|
||||
- `pendingAssistantFrames`.
|
||||
|
||||
Application-defined persistent tokens may define a different policy when their consuming feature is added.
|
||||
|
||||
A precise rewrite retaining a list element preserves its sequence cursor unless the rewrite explicitly remaps the entire destination sequence space.
|
||||
|
||||
## Schema evolution
|
||||
|
||||
A namespace's kind is static for one storage version. Changing a namespace from scalar to list or list to scalar requires an explicit migration; storage must never infer or coerce the change.
|
||||
A token's persisted ID and kind are static for one storage version.
|
||||
|
||||
A list element migration iterates current elements in sequence order and either maps each value while preserving `seq` or explicitly deletes the whole key/namespace. The migration mechanism must page rather than load an unbounded namespace at once.
|
||||
- changing an ID requires an explicit old-ID to new-ID migration;
|
||||
- changing scalar to list or list to scalar requires an explicit migration;
|
||||
- storage never infers or coerces a kind change from observed records;
|
||||
- changing only the TypeScript value type requires a total value migration when old stored values are not already valid under the new type.
|
||||
|
||||
Adding the generic SQLite table is a storage schema migration. Adding the first logical list namespace does not require rewriting sessions that contain no values for it.
|
||||
A list migration pages current elements in sequence order and either maps values while preserving `seq` or deletes the whole key/token. It must not load an unbounded logical list at once.
|
||||
|
||||
Adding the generic SQLite list table is a storage schema migration. Adding a new token with no stored values does not rewrite existing sessions.
|
||||
|
||||
## Instrumentation
|
||||
|
||||
The instrumented storage decorator must expose list reads and continue recording committed writes exactly in transaction order. Telemetry's session-write item kinds gain `list-register` or otherwise count list writes explicitly; do not report them as scalar sets.
|
||||
The instrumented storage decorator exposes token-based scalar/list reads and records committed erased writes in exact transaction order.
|
||||
|
||||
Append-path tests must be able to prove that no `readList` call occurred before commit.
|
||||
Telemetry session-write item kinds include `list-register` explicitly; list appends/deletes are not reported as scalar writes. Token IDs may be attributes, but values and assistant-frame contents never enter telemetry.
|
||||
|
||||
Append-path tests prove no `readList` call occurred before commit.
|
||||
|
||||
## Invariants
|
||||
|
||||
1. Every namespace has one static kind in a storage version.
|
||||
2. A scalar write cannot target a list namespace, and a list write cannot target a scalar namespace at the typed call site.
|
||||
3. Every list element is immutable and addressed by its globally unique write `seq`.
|
||||
4. Elements under one key are returned in sequence order on every backend.
|
||||
5. Append performs no read of the target list.
|
||||
6. List writes are atomic with entries, usage rows, and scalar writes in the same transaction.
|
||||
7. Whole-key delete leaves no elements under the key.
|
||||
8. Missing and empty have the same read result: `[]`.
|
||||
9. Scalar restore and validation never depend on list contents.
|
||||
10. JSONL compaction preserves surviving element sequences.
|
||||
1. Every token has one stable persisted ID and one static kind in a storage version.
|
||||
2. Built-in token IDs are unique.
|
||||
3. Token-based reads and helper-constructed writes preserve the token's value type.
|
||||
4. A scalar helper cannot target a list token; a list helper cannot target a scalar token.
|
||||
5. Every list element is immutable and addressed by its globally unique write `seq`.
|
||||
6. Elements under one list key are returned in sequence order on every backend.
|
||||
7. Append performs no read of the target list.
|
||||
8. Scalar/list writes are atomic with entries and usage in the same transaction.
|
||||
9. Whole-key list delete leaves no elements under that key.
|
||||
10. Missing and empty lists both read as `[]`.
|
||||
11. Scalar restore and validation never depend on list contents.
|
||||
12. JSONL compaction preserves surviving element sequences.
|
||||
13. Deleting operation-owned scalar and list state at terminal cleanup leaves no recoverable partial assistant content.
|
||||
|
||||
## Required tests
|
||||
|
||||
Extend the existing backend conformance suite; do not create a separate list-backend suite.
|
||||
### Token typing and identity
|
||||
|
||||
- scalar reads infer the token's value type;
|
||||
- list reads infer the token's element type;
|
||||
- `setRegister` rejects an incompatible value at compile time;
|
||||
- `appendList` rejects an incompatible element at compile time;
|
||||
- scalar helpers reject list tokens and list helpers reject scalar tokens;
|
||||
- independently defined application tokens work without declaration merging;
|
||||
- built-in IDs are non-empty, separator-free, unique, and have fixed kinds;
|
||||
- token IDs round-trip through Memory, JSONL, and SQLite;
|
||||
- two differently typed definitions using one persisted ID are documented/tested as a programming defect rather than supported aliasing.
|
||||
|
||||
### Scalar regression
|
||||
|
||||
- existing scalar set/get/list/delete/recreate behavior is unchanged;
|
||||
- replacement retains only the latest logical value;
|
||||
- token-based helpers preserve mixed transaction write order;
|
||||
- existing scalar JSONL and SQLite files open without a format rewrite.
|
||||
|
||||
### List conformance
|
||||
|
||||
Extend the shared backend conformance suite:
|
||||
|
||||
- append one element and page it;
|
||||
- multiple appends to one key in one transaction;
|
||||
|
|
@ -303,34 +582,46 @@ Extend the existing backend conformance suite; do not create a separate list-bac
|
|||
- absent key returns `[]`;
|
||||
- whole-key delete and delete of absent key;
|
||||
- delete followed by append in one transaction;
|
||||
- rollback when a later write in the transaction is invalid;
|
||||
- rollback when a later write is invalid;
|
||||
- atomic list + entry + usage + scalar transaction;
|
||||
- identical pages and sequence cursors on Memory, JSONL, and SQLite;
|
||||
- JSONL torn multi-write transaction exposes none of its list elements;
|
||||
- JSONL torn multi-write transaction exposes no list element;
|
||||
- JSONL replay and snapshot compaction preserve cursors;
|
||||
- SQLite paging query plans use the primary key;
|
||||
- append path performs no list read;
|
||||
- restore validation succeeds without reading lists, followed by bounded auxiliary hydration where a consumer requests it;
|
||||
- close rejects later list reads and drains admitted list commits under the existing contract.
|
||||
- restore validation succeeds without reading lists, followed by bounded hydration;
|
||||
- close rejects later list reads and honors already-admitted list commits.
|
||||
|
||||
### Assistant-frame integration
|
||||
|
||||
- compact frames append under the exact effect-pending response key;
|
||||
- raw events with growing `partial` snapshots are never persisted;
|
||||
- frame pages reduce to the same partial message as uninterrupted streaming;
|
||||
- missing/empty frame list restores as no durable partial;
|
||||
- final assistant settlement atomically deletes the frame list;
|
||||
- unknown-effect recovery reads only the bounded list named by scalar state;
|
||||
- external finalization deletes the operation-owned frame list;
|
||||
- idle forks contain no assistant-frame list;
|
||||
- backend byte-growth tests show append-linear rather than repeated-snapshot growth.
|
||||
|
||||
## Implementation map
|
||||
|
||||
Primary files expected to change:
|
||||
|
||||
- `packages/agent/src/harness/session/types.ts`
|
||||
- `packages/agent/src/harness/session/storage-state.ts`
|
||||
- `packages/agent/src/harness/session/memory.ts`
|
||||
- `packages/agent/src/harness/session/session.ts`
|
||||
- `packages/agent/src/harness/session/jsonl/codec.ts`
|
||||
- `packages/agent/src/harness/session/jsonl/storage.ts`
|
||||
- `packages/agent/src/harness/session/testing/storage-decorator.ts`
|
||||
- `packages/agent/src/harness/session/testing/instrumented-storage.ts`
|
||||
- `packages/agent/src/harness/session/testing/conformance/storage.ts`
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/migrations.ts`
|
||||
- a new migration under `packages/session-backends/sqlite-node/src/sqlite/migrations/`
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/session.ts`
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/storage.ts`
|
||||
- a focused SQLite list-register module under `packages/session-backends/sqlite-node/src/sqlite/session/`
|
||||
- SQLite storage tests and query-plan tests
|
||||
- new `packages/agent/src/harness/session/registers.ts` for token types, factories, helpers, and built-in constants;
|
||||
- `packages/agent/src/harness/session/types.ts` to remove global value maps and expose token-based reader methods;
|
||||
- `packages/agent/src/harness/session/storage-state.ts`;
|
||||
- `packages/agent/src/harness/session/memory.ts`;
|
||||
- `packages/agent/src/harness/session/session.ts`;
|
||||
- `packages/agent/src/harness/session/jsonl/codec.ts`;
|
||||
- `packages/agent/src/harness/session/jsonl/storage.ts`;
|
||||
- `packages/agent/src/harness/session/testing/storage-decorator.ts`;
|
||||
- `packages/agent/src/harness/session/testing/instrumented-storage.ts`;
|
||||
- `packages/agent/src/harness/session/testing/conformance/storage.ts`;
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/migrations.ts`;
|
||||
- a new SQLite migration for `list_registers`;
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/session.ts`;
|
||||
- `packages/session-backends/sqlite-node/src/sqlite/storage.ts`;
|
||||
- focused SQLite list paging and query-plan tests.
|
||||
|
||||
After implementation, update the register portions of `harness.md` to point at the concrete API rather than maintaining a divergent second design.
|
||||
After implementation, update `harness.md` to use imported tokens rather than namespace-string/type-map examples and add the concrete assistant-frame lifecycle.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue