diff --git a/docs/compilation-layer.md b/docs/compilation-layer.md new file mode 100644 index 0000000..f409dcd --- /dev/null +++ b/docs/compilation-layer.md @@ -0,0 +1,1317 @@ +# Speckl Compilation Backend Architecture + +## Overview + +The Compilation Layer is Speckl's superpower: it takes a validated SIR (Speckl Intermediate Representation) and produces **deterministic, verifiable artifacts** across multiple target formats. The pipeline: + +``` +Chaotic Conversation → [Extraction Layer] → [Validation Layer] → [Compilation Layer] → Artifacts +``` + +While the conversation is chaotic, the output is **guaranteed deterministic**. Same SIR + same compiler version = same artifact bytes, every time. + +--- + +## 1. Compilation Model + +### Pure Functional Core + +``` +compile: SIR × TargetConfig × CompilerVersion → CompilationResult +``` + +No side effects. No network calls. No file system dependency. No randomness. The entire compilation is a pure function — this is what makes determinism possible. + +```typescript +interface CompilationResult { + // The compiled artifact + artifact: Artifact; + + // Cryptographic proof of what was compiled + manifest: CompilationManifest; + + // What the compiler couldn't compile + gaps: CompilationGap[]; + + // Readiness assessment + readiness: ReadinessLevel; +} + +interface Artifact { + format: TargetFormat; // "cyclonedx", "prov-o", "spdx", "wasm" + mimeType: string; // "application/json", "application/wasm" + content: Buffer | string; // The actual artifact bytes + contentHash: string; // sha256 of content + size: number; // byte count +} +``` + +### The Compilation Manifest + +Every compilation produces a manifest — a cryptographic record of provenance: + +```typescript +interface CompilationManifest { + manifestVersion: "1.0.0"; + + // What went in + inputs: { + sirHash: string; // sha256 of the SIR + sirVersion: number; // SIR version counter + sirId: string; // UUID of the SIR + }; + + // What compiled it + compiler: { + name: "speckl-compiler"; + version: string; // "2.3.1" + targets: string[]; // ["cyclonedx@1.5", "spdx@3.0"] + hash: string; // sha256 of compiler binary/config + }; + + // What came out + outputs: Record; + + // Integrity chain + manifestHash: string; // sha256 of this manifest (self-referential, computed last) +} +``` + +### Hash-Locking for Determinism + +Determinism is guaranteed through content-addressing: + +1. **SIR is hash-addressed**: Every SIR gets a `sha256` hash on validation +2. **Compiler is version-pinned**: Target compilers declare their exact version in the SIR +3. **No ambient state**: Compilation reads nothing from the environment — no `Date.now()`, no `Math.random()`, no OS locale +4. **Result verification**: The manifest's output hash must match `sha256(artifact.content)` — mismatch = compilation error + +```typescript +async function verifyReproducibility( + sir: SIR, + target: TargetConfig, + previousManifest: CompilationManifest +): Promise { + const result = await compile(sir, target); + const newHash = result.manifest.outputs[target.format].hash; + const oldHash = previousManifest.outputs[target.format].hash; + + if (newHash !== oldHash) { + return { + reproducible: false, + expected: oldHash, + actual: newHash, + // The diff between SIR versions may explain this + sirChange: diffSir(previousManifest.inputs.sirHash, sir.hash), + }; + } + + return { reproducible: true }; +} +``` + +### The Dual BOM Pattern (Existing Speckl Convention) + +Speckl already compiles state machines to dual BOMs: CycloneDX (SBOM) + PROV-O (provenance). The compilation layer generalizes this: + +``` +StateMachine SIR → compileToCycloneDX → CycloneDX 1.5 JSON +StateMachine SIR → compileToPROV-O → PROV-O RDF/Turtle +StateMachine SIR → compileToWASM → WASM binary + ↑ + One SIR, three targets, single compilation run +``` + +--- + +## 2. Target Abstraction + +### What Is a Compilation Target? + +A compilation target consumes an SIR and produces an artifact in a specific format. It's an abstract interface that every target implements: + +```typescript +interface CompilerTarget { + // Identity + readonly id: TargetId; // "cyclonedx", "openapi", "terraform" + readonly version: string; // "1.5.0" + readonly format: TargetFormat; // "cyclonedx" + readonly mimeType: string; // "application/json" + + // What domains of SIR can this target handle? + readonly supportedDomains: SIRDomain[]; + + // What does this target REQUIRE from the SIR? + readonly requirements: TargetRequirements; + + // Compilation entry point + compile(sir: SIR, config: TargetConfig, context: CompilationContext): CompileOutput; + + // Does this SIR have enough info for this target? + assessReadiness(sir: SIR): ReadinessAssessment; + + // Generate stubs for missing information + generateStubs(gaps: CompilationGap[], sir: SIR): StubResult; +} + +interface TargetRequirements { + // Required SIR sections + requiredSections: string[]; // ["entities", "transitions", "states"] + // Required entity types + requiredEntityTypes: string[]; // ["software-component", "process"] + // Required relationship types + requiredRelationshipTypes: string[]; + // Minimum completeness threshold (0-1) + minCompletenessScore: number; // e.g., 0.8 = 80% of required fields present +} +``` + +### Example Target Implementations + +**State Machine Targets** (current): +- `cyclonedx` — SBOM from state transitions +- `prov-o` — Provenance ontology from state history +- `spdx` — License compliance from component states +- `wasm` — Executable state machine runtime + +**Future Targets**: +- `openapi` — API specification from interface SIR +- `json-schema` — Data validation schemas +- `cloudformation` / `terraform` — Infrastructure as code from deployment SIR +- `kubernetes-crd` — Custom resource definitions +- `solana-program` / `evm-contract` — Smart contracts from protocol SIR +- `github-actions` — CI/CD pipelines from workflow SIR +- `opa-rego` — Policy rules from constraint SIR +- `protobuf` — Protocol buffer definitions from data SIR + +### Target Registration + +Targets register with the compiler via a plugin system: + +```typescript +class CompilerRegistry { + private targets: Map = new Map(); + + register(target: CompilerTarget): void { + const key = `${target.id}@${target.version}`; + if (this.targets.has(key)) { + throw new Error(`Target ${key} already registered`); + } + this.targets.set(key, target); + } + + getTarget(id: TargetId, version?: string): CompilerTarget { + if (version) { + return this.targets.get(`${id}@${version}`) + ?? this.throwNotFound(id, version); + } + // Return latest version + const versions = Array.from(this.targets.keys()) + .filter(k => k.startsWith(`${id}@`)) + .sort(semverDesc); + return this.targets.get(versions[0]) + ?? this.throwNotFound(id); + } + + listTargetsForDomain(domain: SIRDomain): CompilerTarget[] { + return Array.from(this.targets.values()) + .filter(t => t.supportedDomains.includes(domain)); + } +} +``` + +### Target-Specific Configuration + +```typescript +interface TargetConfig { + format: TargetFormat; + version?: string; // Pin to specific target version + options?: Record; // Target-specific options + dialect?: string; // "cyclonedx:1.5" vs "cyclonedx:1.4" +} + +// Example: CycloneDX with custom options +const cyclonedxConfig: TargetConfig = { + format: "cyclonedx", + version: "1.5.0", + options: { + serialNumber: "urn:uuid:3e56b1dc-...", // deterministic UUID + includeVulnerabilities: true, + componentTypes: ["application", "library", "container"], + }, + dialect: "cyclonedx:1.5", +}; +``` + +--- + +## 3. Domain-to-Target Mapping + +### Domain Capabilities + +Not every SIR can compile to every target. The SIR has a **domain** that determines its capabilities: + +```typescript +enum SIRDomain { + StateMachine = "state-machine", + DataSchema = "data-schema", + APIInterface = "api-interface", + Deployment = "deployment", + Workflow = "workflow", + Protocol = "protocol", + Constraint = "constraint", + ServiceTopology = "service-topology", + LicenseCompliance = "license-compliance", +} +``` + +Each domain has a set of **capabilities** — the types of information it can encode: + +| Domain | Encodes | Capabilities | +|---|---|---| +| `state-machine` | States, transitions, guards, actions | Entity lifecycle, event sequences, invariants | +| `data-schema` | Types, fields, constraints, relations | Data shape, validation rules, serialization | +| `api-interface` | Endpoints, methods, params, responses | HTTP contracts, auth schemes, error models | +| `deployment` | Resources, dependencies, configs | Infrastructure topology, scaling, networking | +| `workflow` | Steps, triggers, conditions, artifacts | CI/CD graphs, automation pipelines | +| `protocol` | Messages, participants, invariants | Communication contracts, consensus rules | +| `constraint` | Rules, subjects, effects | Policy definitions, access control | +| `service-topology` | Services, ports, dependencies | System architecture, fault domains | +| `license-compliance` | Components, licenses, obligations | SBOM data, license compatibility | + +### Target Requirements + +Each target declares what it needs from an SIR to produce a valid artifact: + +```typescript +// A CycloneDX target needs entity information +const cyclonedxRequirements: TargetRequirements = { + requiredSections: ["entities", "relationships"], + requiredEntityTypes: ["software-component"], + minCompletenessScore: 0.6, // Can produce partial BOM from 60% complete SIR +}; + +// A Terraform target needs resource definitions +const terraformRequirements: TargetRequirements = { + requiredSections: ["entities", "dependencies", "configurations"], + requiredEntityTypes: ["compute-resource", "network-resource"], + requiredRelationshipTypes: ["depends-on", "exposes"], + minCompletenessScore: 0.8, // Need 80% for valid Terraform +}; + +// An OPA/Rego target needs rules +const opaRegoRequirements: TargetRequirements = { + requiredSections: ["rules", "subjects", "effects"], + requiredEntityTypes: ["policy-subject", "policy-action"], + minCompletenessScore: 0.7, +}; +``` + +### Compatibility Matrix + +``` +Domain → Target Mapping: +╔══════════════════════╦═════════╦══════════╦═══════════╦════════╦═════════════╗ +║ Target ║ SM ║ Schema ║ API ║ Deploy ║ Workflow ║ +╠══════════════════════╬═════════╬══════════╬═══════════╬════════╬═════════════╣ +║ cyclonedx ║ ✓ ║ ✗ ║ ✗ ║ ✗ ║ ✗ ║ +║ prov-o ║ ✓ ║ ✗ ║ ✗ ║ ✗ ║ ✗ ║ +║ spdx ║ ✓ ║ ✗ ║ ✗ ║ ✗ ║ ✗ ║ +║ wasm ║ ✓ ║ ✗ ║ ✗ ║ ✗ ║ ✗ ║ +║ openapi ║ ✗ ║ ✗ ║ ✓ ║ ✗ ║ ✗ ║ +║ json-schema ║ ✗ ║ ✓ ║ ✓ ║ ✗ ║ ✗ ║ +║ cloudformation ║ ✗ ║ ✗ ║ ✗ ║ ✓ ║ ✗ ║ +║ terraform ║ ✗ ║ ✗ ║ ✗ ║ ✓ ║ ✗ ║ +║ kubernetes-crd ║ ✗ ║ ✗ ║ ✗ ║ ✓ ║ ✗ ║ +║ evm-contract ║ ✓ ║ ✗ ║ ✗ ║ ✗ ║ ✗ ║ +║ github-actions ║ ✗ ║ ✗ ║ ✗ ║ ✗ ║ ✓ ║ +║ opa-rego ║ ✗ ║ ✗ ║ ✗ ║ ✗ ║ ✗ ║ +║ protobuf ║ ✗ ║ ✓ ║ ✓ ║ ✗ ║ ✗ ║ +╚══════════════════════╩═════════╩══════════╩═══════════╩════════╩═════════════╝ +``` + +The compiler validates compatibility before starting: + +```typescript +function validateTargetCompatibility(sir: SIR, target: CompilerTarget): CompatibilityResult { + if (!target.supportedDomains.includes(sir.domain)) { + return { + compatible: false, + reason: `Target ${target.id} does not support SIR domain "${sir.domain}"`, + }; + } + + const missingRequirements = target.requirements.requiredSections + .filter(section => !sir.sections.has(section)); + + if (missingRequirements.length > 0) { + return { + compatible: false, + reason: `SIR missing required sections: ${missingRequirements.join(", ")}`, + }; + } + + // Check completeness threshold + const score = assessCompleteness(sir, target); + if (score < target.requirements.minCompletenessScore) { + return { + compatible: false, + reason: `SIR completeness (${score}) below target minimum (${target.requirements.minCompletenessScore})`, + }; + } + + return { compatible: true }; +} +``` + +--- + +## 4. Incremental Compilation + +### Change Detection + +When a conversation evolves, the SIR gets a new version. We need to know **exactly what changed** to avoid recompiling everything: + +```typescript +interface SIRDiff { + fromVersion: number; + toVersion: number; + fromHash: string; + toHash: string; + + // What changed at each layer + sections: { + added: string[]; + removed: string[]; + modified: string[]; + }; + entities: { + added: EntityId[]; + removed: EntityId[]; + modified: EntityId[]; // Entity existed before, fields changed + }; + transitions: { + added: TransitionId[]; + removed: TransitionId[]; + modified: TransitionId[]; + }; + // Affected targets: which targets need recompilation? + affectedTargets: TargetId[]; +} + +function diffSir(v1: SIR, v2: SIR): SIRDiff { + // Content-addressed comparison + const sectionDiff = { + added: [...v2.sections.keys()].filter(k => !v1.sections.has(k)), + removed: [...v1.sections.keys()].filter(k => !v2.sections.has(k)), + modified: [...v2.sections.keys()].filter(k => + v1.sections.has(k) && + hashSection(v1.sections.get(k)) !== hashSection(v2.sections.get(k)) + ), + }; + + // Entities: O(n) comparison with hash maps + const v1EntityHashes = new Map( + v1.entities.map(e => [e.id, hashEntity(e)]) + ); + const v2EntityHashes = new Map( + v2.entities.map(e => [e.id, hashEntity(e)]) + ); + + const entityDiff = { + added: [...v2EntityHashes.keys()].filter(id => !v1EntityHashes.has(id)), + removed: [...v1EntityHashes.keys()].filter(id => !v2EntityHashes.has(id)), + modified: [...v2EntityHashes.keys()].filter(id => + v1EntityHashes.has(id) && + v1EntityHashes.get(id) !== v2EntityHashes.get(id) + ), + }; + + // Determine which targets are affected + const affectedTargets = determineAffectedTargets(sectionDiff, entityDiff); + + return { + fromVersion: v1.version, + toVersion: v2.version, + fromHash: v1.hash, + toHash: v2.hash, + sections: sectionDiff, + entities: entityDiff, + transitions: computeTransitionDiff(v1, v2), + affectedTargets, + }; +} +``` + +### Partial Compilation + +Instead of recompiling everything, only recompile the changed portions: + +```typescript +async function incrementalCompile( + sirV2: SIR, + sirV1: SIR, + previousResults: Map, + targets: TargetConfig[] +): Promise> { + const diff = diffSir(sirV1, sirV2); + const results = new Map(previousResults); + + for (const target of targets) { + if (!diff.affectedTargets.includes(target.format)) { + // Target unaffected — reuse previous result + continue; + } + + if (isTargetAdditiveOnly(diff, target)) { + // Only new entities added — merge, don't rebuild + const merged = await mergeCompilation( + previousResults.get(target.format)!, + sirV2, + diff.entities.added, + target + ); + results.set(target.format, merged); + } else { + // Structural changes — full recompile for this target + const result = await compile(sirV2, target); + results.set(target.format, result); + } + } + + return results; +} +``` + +### Incremental Compilation Cache + +```typescript +interface CompilationCache { + // Content-addressed cache: hash(sir, target, version) → result + entries: Map; + + // Dependency tracking: which SIR entities feed which target output sections + dependencyMap: Map>; + + lookup(sir: SIR, target: TargetConfig): CompilationResult | null; + store(sir: SIR, target: TargetConfig, result: CompilationResult): void; + invalidate(sir: SIR, entities: EntityId[]): TargetId[]; +} +``` + +--- + +## 5. Artifact Versioning and Integrity + +### Content-Addressable Artifacts + +Every artifact is identified by its content hash, not a version number: + +``` +sir:abc123 → compile → artifact:def456 (sha256) +sir:abc123 → compile → artifact:def456 (same, always) +sir:abc124 → compile → artifact:789abc (different, because SIR changed) +``` + +### Reproducibility Guarantee + +The formal guarantee: + +> For any validated SIR `S` and compiler version `C`, `compile(S, C)` always produces exactly the same bytes. This holds across machines, time, and operating systems. + +Implementation contract: + +```typescript +// Compiler test suite includes reproducibility check +test("compilation is deterministic across runs", () => { + const sir = loadSir("test-fixtures/state-machine/order-processing.sir.json"); + const runs = Array.from({ length: 100 }, () => compile(sir, cyclonedxConfig)); + + const hashes = runs.map(r => r.manifest.outputs["cyclonedx"].hash); + const uniqueHashes = new Set(hashes); + + expect(uniqueHashes.size).toBe(1); // All 100 runs produce identical output +}); + +test("compilation is deterministic across platforms", async () => { + // Artifacts committed to repo, verified in CI on Linux, macOS, Windows + const sir = loadSir("test-fixtures/state-machine/payment-flow.sir.json"); + const { stdout } = await exec("speckl compile --target cyclonedx"); + + expect(stdout.trim()).toBe( + fs.readFileSync("test-fixtures/expected/payment-flow.cdx.json", "utf-8").trim() + ); +}); +``` + +### Manifest Verification + +```typescript +class ArtifactVerifier { + verify(manifest: CompilationManifest, artifact: Artifact): VerificationResult { + // 1. Hash check: does the artifact match its claimed hash? + const actualHash = sha256(artifact.content); + const claimedHash = manifest.outputs[artifact.format]?.hash; + + if (actualHash !== claimedHash) { + return { + valid: false, + error: `Hash mismatch: claimed ${claimedHash}, actual ${actualHash}`, + }; + } + + // 2. Manifest self-consistency + const computedManifestHash = computeManifestHash(manifest); + if (computedManifestHash !== manifest.manifestHash) { + return { + valid: false, + error: "Manifest hash mismatch — manifest has been tampered with", + }; + } + + // 3. Compiler version is known and valid + if (!isKnownCompilerVersion(manifest.compiler.version)) { + return { + valid: false, + error: `Unknown compiler version: ${manifest.compiler.version}`, + }; + } + + return { valid: true }; + } + + // Can we verify an artifact was compiled from SIR X? + verifyProvenance(manifest: CompilationManifest, sir: SIR): boolean { + return manifest.inputs.sirHash === sir.hash; + } +} +``` + +--- + +## 6. Compilation with Gaps + +### The Gap Problem + +Conversations are chaotic. The SIR will have gaps — missing fields, unspecified transitions, incomplete entity definitions. The compilation layer must handle this gracefully. + +```typescript +interface CompilationGap { + id: string; + severity: "error" | "warning" | "info"; + location: { + section: string; + entity?: EntityId; + transition?: TransitionId; + field?: string; + }; + description: string; + // What would fill this gap? + requiredInformation: string; + // What stub was generated in its place + stubGenerated: StubDescription; + // How this affects the compiled output + impact: GapImpact; +} + +interface GapImpact { + // The artifact section(s) affected + affectedArtifactSections: string[]; + // Is this a blocking gap? + blocksValidArtifact: boolean; + // Human-readable description of the impact + description: string; +} +``` + +### Stub Generation + +When information is missing, generate clear stubs with markers: + +```typescript +// Example: SIR has a state "processing" but no transition from "processing" to "shipped" +// The CycloneDX target generates: + +// In compiled CycloneDX JSON: +{ + "components": [ + { + "name": "order-service", + "type": "application", + "bom-ref": "entity:order-service" + } + ], + "dependencies": [ + { + "ref": "entity:order-service", + "dependsOn": [ + // ⚠️ STUB: Gap #GAP-42 + "__speckl_stub__gap_GAP-42_missing_transition" + ] + } + ], + "metadata": { + "tools": [ + { + "vendor": "Speckl", + "name": "Speckl Compiler", + "version": "2.3.1" + } + ], + // ⚠️ SPECIFICATION DEBT + "properties": [ + { + "name": "speckl:specification-debt", + "value": JSON.stringify([ + { + "gapId": "GAP-42", + "severity": "error", + "description": "Missing transition: processing → shipped", + "requiredInformation": "Define what triggers the transition from 'processing' to 'shipped' and any guard conditions", + "blocksValid: true + } + ]) + } + ] + } +} +``` + +### Graduated Compilation Levels + +Artifacts have a **readiness level** distinct from the SIR's completeness: + +```typescript +enum ReadinessLevel { + Draft = "draft", // < 40%: Conceptual, mostly stubs + Partial = "partial", // 40-60%: Key entities present, many stubs + Beta = "beta", // 60-80%: Usable for review, few stubs + Production = "production", // 80-95%: Valid artifact, minimal stubs + Complete = "complete", // 95%+: No gaps, fully valid +} + +interface ReadinessAssessment { + level: ReadinessLevel; + score: number; // 0.0 - 1.0 + gapCount: { + errors: number; + warnings: number; + info: number; + }; + // Can this artifact be used for its intended purpose? + usability: { + canReview: boolean; // Is it readable by humans? + canValidate: boolean; // Passes format-specific validation? + canDeploy: boolean; // Could be deployed to production? + canAudit: boolean; // Would pass a compliance audit? + }; +} +``` + +### Specification Debt Tracking + +"Specification debt" is tracked persistently in the compiled output: + +```json +{ + "speckl:specification-debt": { + "totalGaps": 7, + "errors": 3, + "warnings": 4, + "readiness": "beta", + "debtItems": [ + { + "id": "GAP-42", + "since": "2026-05-01T12:00:00Z", + "description": "Missing transition: processing → shipped", + "affects": ["cyclonedx:dependencies", "prov-o:wasGeneratedBy"], + "resolution": "Define the shipping trigger and guard conditions in the conversation" + } + ] + } +} +``` + +--- + +## 7. Compiler Plugin Architecture + +### The `CompilerTarget` Interface (Full) + +```typescript +// Each target is a self-contained module implementing this interface +interface CompilerTarget { + // === Identity === + readonly id: TargetId; + readonly version: string; + readonly format: TargetFormat; + readonly mimeType: string; + readonly displayName: string; + readonly description: string; + + // === Domain Compatibility === + readonly supportedDomains: SIRDomain[]; + readonly requirements: TargetRequirements; + + // === Primary Entry Points === + compile(sir: SIR, config: TargetConfig, context: CompilationContext): CompileOutput; + assessReadiness(sir: SIR): ReadinessAssessment; + generateStubs(gaps: CompilationGap[], sir: SIR): StubResult; + + // === Incremental Compilation === + canIncrementalMerge(diff: SIRDiff): boolean; + mergeIncremental(previous: CompileOutput, diff: SIRDiff, sir: SIR): CompileOutput; + + // === Validation === + validateArtifact(artifact: Artifact): ValidationResult; +} + +interface CompilationContext { + compilerVersion: string; + compilerHash: string; + sirHash: string; + timestamp: string; // For manifest only, NOT used in compilation + logger: CompilationLogger; // Structured logging, no side effects on artifact +} + +interface CompileOutput { + artifact: Artifact; + manifest: CompilationManifest; + gaps: CompilationGap[]; + readiness: ReadinessAssessment; + // Target-specific metadata for incremental merging + sections: Map; +} +``` + +### Target Registration and Discovery + +Targets are discovered from the filesystem or loaded dynamically: + +```typescript +// File: targets/cyclonedx/index.ts +import { CompilerTarget, CompileOutput, CompilationContext } from "@speckl/compiler"; + +export const cyclonedxTarget: CompilerTarget = { + id: "cyclonedx", + version: "1.5.0", + format: "cyclonedx", + mimeType: "application/json", + displayName: "CycloneDX", + description: "OWASP CycloneDX Software Bill of Materials", + supportedDomains: [SIRDomain.StateMachine, SIRDomain.LicenseCompliance], + requirements: { + requiredSections: ["entities", "relationships"], + requiredEntityTypes: ["software-component"], + minCompletenessScore: 0.6, + }, + + compile(sir: SIR, config: TargetConfig, context: CompilationContext): CompileOutput { + // 1. Extract components from SIR entities + const components = sir.entities + .filter(e => e.type === "software-component") + .map(entityToComponent); + + // 2. Extract dependencies from SIR relationships + const dependencies = sir.relationships + .filter(r => r.type === "depends-on") + .map(relationshipToDependency); + + // 3. Map state transitions to lifecycle phases + const lifecyclePhases = sir.transitions?.map(transitionToLifecyclePhase) ?? []; + + // 4. Build the CycloneDX document + const bom = buildCycloneDX15({ + bomFormat: "CycloneDX", + specVersion: "1.5", + serialNumber: config.options?.serialNumber as string ?? generateDeterministicUUID(sir.hash, "cyclonedx-serial"), + version: 1, + components, + dependencies, + metadata: { + tools: [{ + vendor: "Speckl", + name: "Speckl Compiler", + version: context.compilerVersion, + }], + properties: lifecyclePhases.map(phase => ({ + name: `speckl:lifecycle:${phase.state}`, + value: JSON.stringify(phase), + })), + }, + }); + + // 5. Identify gaps + const gaps = identifyGaps(sir, components, dependencies); + + // 6. Generate stubs for gaps + if (gaps.length > 0) { + this.injectStubs(bom, gaps); + } + + // 7. Compute artifact + const content = JSON.stringify(bom, null, 2); + const artifact: Artifact = { + format: "cyclonedx", + mimeType: "application/json", + content, + contentHash: sha256(content), + size: content.length, + }; + + // 8. Build manifest + const manifest = buildManifest(context, artifact, "cyclonedx"); + + // 9. Assess readiness + const readiness = assessReadinessFromGaps(gaps); + + return { + artifact, + manifest, + gaps, + readiness, + sections: trackSections(sir, components, dependencies), + }; + }, + + assessReadiness(sir: SIR): ReadinessAssessment { /* ... */ }, + generateStubs: generateCycloneDXStubs, + canIncrementalMerge(diff: SIRDiff): boolean { /* ... */ }, + mergeIncremental(prev, diff, sir) { /* ... */ }, + validateArtifact: validateCycloneDX, +}; + +// Register +compilerRegistry.register(cyclonedxTarget); +``` + +### Full CycloneDX Target Example + +```typescript +// Complete example: State Machine SIR → CycloneDX 1.5 JSON + +// INPUT SIR (simplified): +const orderProcessingSIR: SIR = { + id: "sir:order-processing-v3", + version: 3, + hash: "sha256:abc123...", + domain: SIRDomain.StateMachine, + entities: [ + { + id: "entity:order-api", + type: "software-component", + name: "Order API", + version: "2.1.0", + properties: { + language: "TypeScript", + framework: "Express", + license: "MIT", + }, + }, + { + id: "entity:payment-service", + type: "software-component", + name: "Payment Service", + version: "1.8.3", + properties: { + language: "Rust", + license: "Apache-2.0", + piiScope: "processes-credit-cards", // GAP: needs documentation + }, + }, + { + id: "entity:shipping-worker", + type: "software-component", + name: "Shipping Worker", + version: "0.9.0", + properties: { + language: "Go", + license: "MIT", + }, + }, + ], + relationships: [ + { + id: "rel:order-to-payment", + type: "depends-on", + source: "entity:order-api", + target: "entity:payment-service", + }, + { + id: "rel:order-to-shipping", + type: "depends-on", + source: "entity:order-api", + target: "entity:shipping-worker", + }, + ], + transitions: [ + { + id: "trans:created-to-paid", + from: "created", + to: "paid", + trigger: "payment-received", + guard: "payment.amount >= order.total", + }, + { + id: "trans:paid-to-shipped", + from: "paid", + to: "shipped", + trigger: "shipping-label-created", + // GUARD MISSING — this is a gap + }, + ], + states: [ + { id: "created", initial: true }, + { id: "paid" }, + { id: "shipped", final: true }, + ], +}; + +// OUTPUT: CycloneDX 1.5 JSON +const compiledOutput = { + bomFormat: "CycloneDX", + specVersion: "1.5", + serialNumber: "urn:uuid:d5a7b3c1-e8f2-4a9d-b6c0-123456789abc", + version: 1, + metadata: { + timestamp: "2026-05-09T22:57:00Z", + tools: [{ + vendor: "Speckl", + name: "Speckl Compiler", + version: "2.3.1", + }], + properties: [ + { + name: "speckl:lifecycle:paid", + value: JSON.stringify({ + transition: "trans:created-to-paid", + trigger: "payment-received", + guard: "payment.amount >= order.total", + }), + }, + { + name: "speckl:lifecycle:shipped", + value: JSON.stringify({ + transition: "trans:paid-to-shipped", + trigger: "shipping-label-created", + guard: "__speckl_stub__gap_GAP-001_missing_guard", + gapRef: "GAP-001", + }), + }, + { + name: "speckl:specification-debt", + value: JSON.stringify({ + totalGaps: 2, + errors: 1, + warnings: 1, + readiness: "beta", + debtItems: [ + { + id: "GAP-001", + severity: "error", + description: "Missing guard condition on transition paid → shipped", + requiredInformation: "What condition must be true for an order to ship? (e.g., payment settled, inventory confirmed)", + blocksValid: false, + }, + { + id: "GAP-002", + severity: "warning", + description: "Payment Service processes PII — needs documentation", + requiredInformation: "Document PII scope: what data, retention, encryption, access controls", + blocksValid: false, + }, + ], + }), + }, + ], + }, + components: [ + { + type: "application", + name: "Order API", + version: "2.1.0", + "bom-ref": "entity:order-api", + properties: [ + { name: "speckl:language", value: "TypeScript" }, + { name: "speckl:framework", value: "Express" }, + ], + }, + { + type: "application", + name: "Payment Service", + version: "1.8.3", + "bom-ref": "entity:payment-service", + properties: [ + { name: "speckl:language", value: "Rust" }, + { name: "speckl:pii-scope", value: "processes-credit-cards" }, + { name: "speckl:gap-ref", value: "GAP-002" }, + ], + }, + { + type: "application", + name: "Shipping Worker", + version: "0.9.0", + "bom-ref": "entity:shipping-worker", + properties: [ + { name: "speckl:language", value: "Go" }, + ], + }, + // STUB: generated for the missing guard + { + type: "application", + name: "__speckl_stub__gap_GAP-001_missing_guard", + "bom-ref": "__speckl_stub__gap_GAP-001_missing_guard", + properties: [ + { name: "speckl:stub", value: "true" }, + { name: "speckl:gap-ref", value: "GAP-001" }, + ], + }, + ], + dependencies: [ + { + ref: "entity:order-api", + dependsOn: ["entity:payment-service", "entity:shipping-worker"], + }, + { + ref: "entity:payment-service", + dependsOn: [ + "__speckl_stub__gap_GAP-001_missing_guard", // Linked to stub + ], + }, + ], +}; +``` + +### Manifest for this compilation: + +```json +{ + "manifestVersion": "1.0.0", + "inputs": { + "sirHash": "sha256:abc123def456...", + "sirVersion": 3, + "sirId": "sir:order-processing-v3" + }, + "compiler": { + "name": "speckl-compiler", + "version": "2.3.1", + "targets": ["cyclonedx@1.5"], + "hash": "sha256:compiler-v2.3.1-hash..." + }, + "outputs": { + "cyclonedx": { + "hash": "sha256:789fedcba...", + "size": 4521, + "timestamp": "2026-05-09T22:57:00Z" + } + }, + "manifestHash": "sha256:manifest-hash..." +} +``` + +--- + +## 8. Output Formats and Packaging + +### Single Artifact vs. Bundle + +Compilation can produce a single artifact or a bundle: + +```typescript +interface ArtifactBundle { + // Bundle metadata + bundleId: string; + bundleHash: string; + + // The SIR this was compiled from + sir: { + id: string; + version: number; + hash: string; + }; + + // Compiler info + compiler: { + version: string; + hash: string; + }; + + // All compiled artifacts + artifacts: Record; + + // Shared manifest + manifest: CompilationManifest; + + // Packaging metadata + created: string; // ISO 8601, for information only + format: "speckl-bundle@1.0"; +} +``` + +### Provenance Embedding + +Every artifact embeds its own provenance. For formats that support metadata (CycloneDX, SPDX, PROV-O), the compilation manifest is embedded directly. For formats that don't (plain JSON, WASM), a sidecar manifest is generated: + +``` +output/ +├── order-processing.cdx.json # CycloneDX with embedded provenance +├── order-processing.cdx.manifest.json # Standalone manifest +├── order-processing.wasm # WASM binary +├── order-processing.wasm.manifest.json # Sidecar manifest +└── order-processing.bundle.json # Full bundle with all artifacts +``` + +### WASM as Universal Runtime Format + +Speckl's current path compiles state machines to WASM, enabling: + +1. **Cross-platform execution**: The same state machine runs in browser, server, edge +2. **Deterministic execution**: WASM is deterministic by design (no floats with NaN boxing) +3. **Compact**: Binary format, suitable for on-chain / embedded use +4. **Sandboxed**: Safe to execute untrusted state machines + +```typescript +// WASM compilation target +const wasmTarget: CompilerTarget = { + id: "wasm", + version: "1.0.0", + format: "wasm", + mimeType: "application/wasm", + supportedDomains: [SIRDomain.StateMachine], + requirements: { + requiredSections: ["states", "transitions"], + minCompletenessScore: 0.5, + }, + + compile(sir: SIR, config: TargetConfig, context: CompilationContext): CompileOutput { + // Generate a deterministic WASM module that: + // 1. Exports a state machine execution function + // 2. Accepts events as input + // 3. Returns new state + actions + + const wasmModule = generateWasmStateMachine({ + states: sir.states, + transitions: sir.transitions.map(t => ({ + from: t.from, + to: t.to, + trigger: t.trigger, + guard: t.guard ? compileGuardToWasm(t.guard) : null, + actions: (t.actions ?? []).map(compileActionToWasm), + })), + // Embed SIR hash for verification + metadata: { + sirHash: context.sirHash, + compilerVersion: context.compilerVersion, + }, + }); + + const content = wasmModule.toBytes(); + return { + artifact: { + format: "wasm", + mimeType: "application/wasm", + content, + contentHash: sha256(content), + size: content.length, + }, + // ... manifest, gaps, readiness + }; + }, +}; +``` + +### Self-Describing Artifacts + +Can you reconstruct the SIR from the compiled artifact? Generally **no** — it's a lossy transformation. But some targets are more lossy than others: + +| Target | Reversibility | Notes | +|---|---|---| +| CycloneDX | Partial | Entities and dependencies survive; state transitions lose semantics | +| PROV-O | High | Designed for provenance — much of the SIR structure survives | +| WASM | Low | Compiled to instructions; reconstruction is decompilation, not guaranteed | +| OpenAPI | Partial | Endpoint structure survives; validation layer intent doesn't | +| Terraform | Low | HCL is declarative; can infer resources but not the SIR abstractions | + +The compilation manifest fills the gap — it records the SIR hash, so you can always look up the exact SIR that produced a given artifact. The bundle includes both, so nothing is lost. + +```typescript +function reconstructFromBundle(bundle: ArtifactBundle): { + sir: SIR; // Loaded from the SIR hash reference + artifacts: Record; + manifest: CompilationManifest; +} { + // The bundle doesn't contain the SIR itself (that's stored separately) + // But it references it by hash — load it from the SIR store + const sir = sirStore.lookup(bundle.sir.hash); + if (!sir) { + throw new Error(`SIR ${bundle.sir.hash} not found in store`); + } + + // Verify: does this SIR's hash match what the manifest says? + if (sir.hash !== bundle.inputs.sirHash) { + throw new Error("SIR hash mismatch — possible tampering"); + } + + return { sir, artifacts: bundle.artifacts, manifest: bundle.manifest }; +} +``` + +--- + +## Architecture Summary + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Compilation Layer │ +│ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ Compiler Orchestrator │ │ +│ │ - Validates SIR × target compatibility │ │ +│ │ - Manages incremental compilation cache │ │ +│ │ - Coordinates multi-target compilation │ │ +│ │ - Builds the compilation manifest │ │ +│ └────────────────────┬─────────────────────────────────┘ │ +│ │ │ +│ ┌─────────────┼─────────────┐ │ +│ ▼ ▼ ▼ │ +│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ +│ │CycloneDX │ │ PROV-O │ │ WASM │ ... │ +│ │ Target │ │ Target │ │ Target │ │ +│ └──────────┘ └──────────┘ └──────────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌──────────────────────────────────────────────────────┐ │ +│ │ Artifact Assembler │ │ +│ │ - Bundles artifacts + manifest │ │ +│ │ - Embeds provenance in output formats │ │ +│ │ - Generates sidecar manifests when needed │ │ +│ │ - Signs bundles (future: sigstore/cosign) │ │ +│ └──────────────────────────────────────────────────────┘ │ +│ │ +│ Invariants: │ +│ ✓ Pure functional — no side effects │ +│ ✓ Content-addressed — every artifact has a hash │ +│ ✓ Reproducible — same SIR → same artifact, always │ +│ ✓ Gap-tolerant — compiles what it can, marks the rest │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Key Design Decisions + +1. **Pure functional compilation**: The compiler is a function `(SIR, TargetConfig) → CompilationResult`. No side effects guarantee determinism. + +2. **Content-addressing over versioning**: Artifacts aren't "version 1.2.3" — they're `sha256:abc123...`. You know exactly what you have. + +3. **Gap-tolerant by design**: The compiler handles incomplete SIRs gracefully with stubs, debt tracking, and readiness levels. Works from conversation day 1. + +4. **Plugin architecture for targets**: New targets are standalone modules. The compiler core doesn't know about CycloneDX or Terraform — it knows about the `CompilerTarget` interface. + +5. **Manifest as proof**: Every compilation produces cryptographic proof of what was compiled, by whom, from what. The manifest is the chain of custody. + +6. **Incremental over rebuild**: Large SIRs with many entities benefit from partial recompilation. The dependency map enables surgical updates. + +7. **Bundle as distribution unit**: A `.speckl.bundle.json` contains all artifacts and the manifest. Self-contained, verifiable, distributable. diff --git a/docs/extraction-protocol.md b/docs/extraction-protocol.md new file mode 100644 index 0000000..f19cb36 --- /dev/null +++ b/docs/extraction-protocol.md @@ -0,0 +1,1460 @@ +# Speckl Extraction Protocol — Design Document + +## Overview + +The Extraction Layer bridges chaotic human/agent conversation and the Structured Intermediate Representation (SIR). It continuously (or batch) identifies formal elements — entities, transitions, constraints, relationships — and populates the SIR with typed, versioned, provenance-tracked records. + +Every element in the SIR traces back to the conversation turn(s) that produced it. Every element carries a confidence score. And every element can be refined incrementally as the conversation evolves. + +--- + +## 1. Core Data Structures + +### 1.1 ConversationTurn + +The raw input to the extraction layer. + +```typescript +interface ConversationTurn { + id: string; // UUID, globally unique + timestamp: number; // Unix ms + speaker: SpeakerRole; // "human" | "agent" + agentId?: string; // e.g. "speckl-extractor", "senior-eng" + content: string; // Raw text or structured block + parentTurnId?: string; // For threaded replies + metadata?: { + explicitMarkers?: string[]; // e.g. ["spec:", "state:"] + structuredBlock?: { // Extracted by pre-parser + type: "spec" | "code" | "diagram"; + language?: string; + content: string; + }; + }; +} + +type SpeakerRole = "human" | "agent"; +``` + +### 1.2 SIRElement + +The core unit of structured representation. All SIR elements share a common header and then specialize by domain. + +```typescript +// Common header for every SIR element +interface SIRHeader { + id: string; // UUID, stable across revisions + elementType: SIRType; // Domain-specific type discriminator + domain: DomainId; // "state-machine" | "data-model" | etc. + confidence: number; // 0.0 – 1.0 + status: ElementStatus; // Lifecycle state + createdAt: number; // Unix ms + updatedAt: number; // Last modification + version: number; // Monotonic counter, bumped on mutation + provenance: ProvenanceChain; // Full trace + flags: SIRFlags; // See below +} + +type SIRType = string; // e.g. "state", "transition", "guard", "event" +type DomainId = string; // e.g. "state-machine", "data-model" + +type ElementStatus = + | "proposed" // Extracted, never confirmed by human + | "confirmed" // Human explicitly confirmed + | "deprecated" // Later conversation contradicted this + | "replaced" // Superseded by another element (holds replacedById) + | "error"; // Extraction was wrong, human flagged it + +interface SIRFlags { + needsReview: boolean; // True when confidence < autoAcceptThreshold + isAmbiguous: boolean; // Multiple possible interpretations + isTentative: boolean; // Extracted from implicit cues, not explicit markers + isContradicted: boolean; // Later conversation conflicts + sourceIsExplicit: boolean; // Came from spec: prefix or structured block +} +``` + +### 1.3 ProvenanceChain + +Every SIR element has a complete, immutable provenance trail. + +```typescript +interface ProvenanceChain { + entries: ProvenanceEntry[]; +} + +interface ProvenanceEntry { + turnId: string; // ConversationTurn that produced/modified this + action: ProvenanceAction; // What happened in this turn + agentId?: string; // Which agent performed the action + humanConfirmed: boolean; // Did the human explicitly confirm? + timestamp: number; + confidenceAtTime: number; // Snapshot of confidence when entry was created + extractionRationale?: string; // Why the extractor thought this was relevant + diff?: ElementDiff; // For "refined" or "updated" actions +} + +type ProvenanceAction = + | "extracted" // First creation + | "refined" // Incremental update, same element + | "confirmed" // Human confirmed + | "deprecated" // Marked as no longer valid + | "replaced" // Superseded by new element + | "merged" // Two elements combined + | "split"; // One element split into multiple +``` + +### 1.4 ElementDiff + +Captures what changed between versions of a SIR element. + +```typescript +interface ElementDiff { + fromVersion: number; + toVersion: number; + triggerTurnId: string; + changes: FieldChange[]; +} + +interface FieldChange { + field: string; // Path, e.g. "transition.guard", "state.name" + oldValue: any; + newValue: any; + reason: string; // Why this changed, extracted from turn +} +``` + +### 1.5 Domain-Specific Element Examples + +State machine domain specialization: + +```typescript +// A state in a state machine +interface StateElement extends SIRHeader { + elementType: "state"; + domain: "state-machine"; + data: { + name: string; // e.g. "idle" + parent?: string; // For hierarchical states + isInitial?: boolean; + isFinal?: boolean; + entryActions?: string[]; + exitActions?: string[]; + invariants?: string[]; // Properties that must hold in this state + }; +} + +// A transition between states +interface TransitionElement extends SIRHeader { + elementType: "transition"; + domain: "state-machine"; + data: { + from: string; // State name or element ID + to: string; // State name or element ID + event: string; // Triggering event + guard?: string; // Condition that must be true + actions?: string[]; // Actions fired on transition + priority?: number; // For conflict resolution + }; +} +``` + +--- + +## 2. Extraction Modes + +The extraction layer supports four modes, selected per-turn or configured globally. + +### 2.1 Real-Time Extraction + +Extraction runs incrementally as each message arrives, before the next speaker responds. + +``` +Sequence: + Human sends turn → [pre-parser] → [extractor] → SIR updated → Agent responds +``` + +**Use case:** Live specification sessions where the human is describing a system and the agent provides immediate feedback ("I've captured state 'loading'. Did you mean it has a timeout transition?"). + +**Implementation:** + +```typescript +async function extractRealtime(turn: ConversationTurn, sir: SIRDocument): Promise { + // 1. Pre-parse for explicit markers + const markers = detectExplicitMarkers(turn); + + // 2. If markers found, do targeted extraction + if (markers.length > 0) { + return extractFromMarkers(turn, markers, sir); + } + + // 3. Otherwise, run lightweight pattern matching + const candidates = runLightweightPatterns(turn, sir.domain); + + // 4. Only auto-extract high-confidence candidates + const highConf = candidates.filter(c => c.confidence >= AUTO_ACCEPT_THRESHOLD); + const lowConf = candidates.filter(c => c.confidence < AUTO_ACCEPT_THRESHOLD && c.confidence >= FLAG_THRESHOLD); + + // 5. Apply high-confidence, flag low-confidence for review + const applied = applyExtractions(highConf, sir, turn, false); + const flagged = flagForReview(lowConf, sir, turn); + + return { applied, flagged, candidates: lowConf }; +} +``` + +### 2.2 Batch Extraction + +Extraction runs after a segment of conversation (a "batch" of N turns, or a pause in conversation). + +``` +Sequence: + [Turn 1] → [Turn 2] → [Turn 3] → (batch trigger) → [extractor] → SIR updated +``` + +**Use case:** After a multi-turn discussion where the human brainstormed possibilities and you want to extract the settled decisions. + +**Implementation:** + +```typescript +async function extractBatch( + turns: ConversationTurn[], + sir: SIRDocument, + options?: BatchOptions +): Promise { + // 1. Concatenate turns into a coherent transcript + const transcript = buildTranscript(turns); + + // 2. Run the full LLM-based extractor over the transcript + const extraction = await llmExtract( + transcript, + sir.domain, + sir.getExistingElements() // Pass current SIR for refinement context + ); + + // 3. Diff against existing SIR elements + const { added, refined, deprecated, conflicts } = diffAgainstSIR(extraction, sir); + + // 4. Apply changes with full provenance + // ... +} + +interface BatchOptions { + deduplicate: boolean; // Merge near-duplicate extractions + summarize: boolean; // Generate human-readable summary of changes + confirmBeforeApply: boolean; // Show diff to human before applying +} +``` + +### 2.3 Deliberative Extraction + +Multiple agents debate before elements are committed. This is the highest-quality mode. + +``` +Sequence: + Human sends turn + → [Agent A proposes extraction] + → [Agent B critiques] + → [Agent C synthesizes] + → [Consensus extraction] → SIR updated +``` + +**Implementation:** + +```typescript +async function extractDeliberative( + turn: ConversationTurn, + sir: SIRDocument, + debaters: DebaterConfig[] +): Promise { + const proposals: ExtractionProposal[] = []; + + // Phase 1: Each debater independently extracts + for (const debater of debaters) { + const proposal = await debater.extract(turn, sir); + proposals.push(proposal); + } + + // Phase 2: Cross-critique + const critiques: Critique[] = []; + for (let i = 0; i < proposals.length; i++) { + for (let j = 0; j < proposals.length; j++) { + if (i === j) continue; + const critique = await debaters[j].critique(proposals[i], turn, sir); + critiques.push(critique); + } + } + + // Phase 3: Synthesize + const synthesizer = debaters.find(d => d.role === "synthesizer") || debaters[0]; + const consensus = await synthesizer.synthesize(proposals, critiques, turn, sir); + + // Phase 4: Apply with deliberation provenance + return applyDeliberated(consensus, proposals, critiques, sir, turn); +} + +interface DebaterConfig { + id: string; + role: "proposer" | "critic" | "synthesizer"; + model?: string; + extract: (turn: ConversationTurn, sir: SIRDocument) => Promise; + critique: (proposal: ExtractionProposal, turn: ConversationTurn, sir: SIRDocument) => Promise; + synthesize: (proposals: ExtractionProposal[], critiques: Critique[], turn: ConversationTurn, sir: SIRDocument) => Promise; +} +``` + +### 2.4 Interactive Extraction + +When extraction ambiguity exceeds a threshold, the extractor asks clarifying questions rather than guessing. + +``` +Sequence: + Human sends turn + → [extractor] identifies ambiguity + → [extractor] generates clarifying question + → Human answers + → [extractor] resolves ambiguity → SIR updated +``` + +**Implementation:** + +```typescript +async function extractInteractive( + turn: ConversationTurn, + sir: SIRDocument +): Promise { + const candidates = await extractCandidates(turn, sir); + + // Identify ambiguities + const ambiguities = detectAmbiguities(candidates); + + if (ambiguities.length > 0) { + // Generate the best clarifying question + const question = await generateClarifyingQuestion(ambiguities, turn); + + // Store pending ambiguity so resolution can be matched later + sir.pendingAmbiguities.push({ + id: generateId(), + sourceTurnId: turn.id, + candidates: candidates.map(c => c.id), + ambiguities, + question, + timestamp: Date.now(), + }); + + return { + status: "clarification_needed", + question, + ambiguities, + }; + } + + // No ambiguity — proceed with auto-extraction + return applyExtractions(candidates, sir, turn); +} +``` + +--- + +## 3. Confidence and Ambiguity Handling + +### 3.1 Confidence Thresholds + +```typescript +// Global thresholds, domain-overridable +const CONFIDENCE = { + // >= this: auto-extract, mark confirmed + AUTO_ACCEPT: 0.85, + + // >= this: extract but mark needsReview=true, isTentative=true + // Agent should surface these to the human + REVIEW: 0.60, + + // >= this: extract to a "candidate" pool, do NOT add to main SIR + // Agent only surfaces if asked + CANDIDATE: 0.40, + + // < this: discard (noise) + DISCARD: 0.40, +}; +``` + +### 3.2 Confidence Scoring Algorithm + +Confidence is computed from multiple signals: + +```typescript +function computeConfidence( + candidate: ExtractionCandidate, + turn: ConversationTurn, + context: ExtractionContext +): number { + let score = 0.0; + const weights = context.domainWeights; + + // Signal 1: Explicit markers (strongest) + if (candidate.fromExplicitMarker) score += weights.explicitMarker * 1.0; // e.g. 0.40 + + // Signal 2: Structured block (code fence, diagram) + if (candidate.fromStructuredBlock) score += weights.structuredBlock * 1.0; // e.g. 0.35 + + // Signal 3: Linguistic pattern match strength + score += candidate.patternMatchScore * weights.patternMatch; // e.g. 0.25 + + // Signal 4: Consistency with existing SIR + const consistencyScore = checkConsistency(candidate, context.existingElements); + score += consistencyScore * weights.consistency; // e.g. 0.15 + + // Signal 5: Repetition — same extraction from multiple turns + if (candidate.repetitionCount > 1) { + score += Math.min(candidate.repetitionCount * 0.05, 0.15); + } + + // Signal 6: Speaker authority + if (turn.speaker === "human") score += 0.05; + + // Cap at 1.0 + return Math.min(score, 1.0); +} +``` + +### 3.3 Ambiguity Representation in the SIR + +When multiple interpretations exist for the same conversation fragment: + +```typescript +interface AmbiguousElement extends SIRHeader { + isAmbiguous: true; + interpretations: Interpretation[]; + resolvedInterpretationId?: string; // Set once resolved +} + +interface Interpretation { + id: string; + element: SIRElement; // The proposed element + confidence: number; + rationale: string; + proposedBy: string; // Agent ID +} +``` + +**Example:** The human says "when the user logs in, go to the main screen." + +``` +Extraction yields two interpretations: + +Interpretation A (confidence: 0.72): + "login" is a state, "main screen" is a state, there's a transition "login → main screen" + +Interpretation B (confidence: 0.65): + "logs in" is an event, the transition fires on "login" event from any state → "main screen" +``` + +The extractor creates an `AmbiguousElement` with both interpretations and flags it for review. The agent surfaces: *"I captured a transition after login. Did you mean (A) login is a state that transitions to main screen, or (B) the login event triggers a transition from any state?"* + +### 3.4 Conflict Resolution + +When later conversation contradicts an earlier extraction: + +```typescript +function resolveConflict( + existing: SIRElement, + incoming: ExtractionCandidate, + provenance: ProvenanceChain +): ConflictResolution { + // Case 1: Incoming confidence >> existing confidence + // → Deprecate old, create new, link them + if (incoming.confidence > existing.confidence + 0.2) { + return { + action: "replace", + existingElement: markDeprecated(existing, incoming.sourceTurnId), + newElement: createFromCandidate(incoming, { + replacedId: existing.id, + version: 1, + }), + rationale: `Higher-confidence extraction replaces previous`, + }; + } + + // Case 2: Comparable confidence → flag for human resolution + if (Math.abs(incoming.confidence - existing.confidence) < 0.2) { + return { + action: "flag_conflict", + existingElement: setFlag(existing, "isContradicted", true), + pendingResolution: { + existingId: existing.id, + incomingCandidate: incoming, + question: generateResolutionQuestion(existing, incoming), + }, + }; + } + + // Case 3: Incoming confidence << existing → discard incoming + return { + action: "reject", + reason: `Incoming confidence (${incoming.confidence}) too low vs existing (${existing.confidence})`, + }; +} +``` + +--- + +## 4. Provenance Model + +### 4.1 Full Traceability + +Every SIR element **must** trace back to one or more conversation turns. This is non-negotiable — no element enters the SIR without provenance. + +``` +Element → ProvenanceChain → ProvenanceEntry[] → ConversationTurn +``` + +Each entry records: +- **Which turn** triggered the change +- **Which agent** proposed it (if multi-agent) +- **Whether the human confirmed** (explicitly or implicitly) +- **What changed** (the diff) +- **Why** (extraction rationale) + +### 4.2 Multi-Turn Refinement Provenance + +When an element gets refined across multiple turns, the chain grows: + +``` +Turn 3: Human: "the idle state has a timeout of 30s" + → ProvenanceEntry { action: "extracted", element: State(idle), confidence: 0.70 } + +Turn 7: Human: "actually, make that 60 seconds" + → ProvenanceEntry { action: "refined", diff: { timeout: 30→60 }, confidence: 0.95 } + +Turn 12: Human: "spec: state idle { timeout: 120s, onTimeout: → error }" + → ProvenanceEntry { action: "refined", diff: { timeout: 60→120, +onTimeout }, confidence: 0.98 } +``` + +The provenance chain is **append-only**. Old entries are never modified — they are the historical record. + +### 4.3 Source Attribution + +Every extraction records who (or what) proposed it: + +```typescript +// Attribution taxonomy +type Attributor = + | { type: "human"; speakerId: string } // Direct human statement + | { type: "agent"; agentId: string; model?: string } // Agent extraction + | { type: "system"; component: string } // Automated pattern match + | { type: "consensus"; agents: string[] }; // Multi-agent agreement +``` + +### 4.4 Provenance Queries + +The provenance model supports queries like: +- "Show all elements the human hasn't confirmed yet" → `WHERE NOT humanConfirmed` +- "What did agent X propose that got rejected?" → `WHERE agentId = X AND action = 'deprecated'` +- "Trace the full evolution of state 'loading'" → `WHERE elementId = loading.id ORDER BY version` +- "Which turns contributed to more than 3 elements?" → `GROUP BY turnId HAVING COUNT > 3` + +--- + +## 5. Extraction Triggers + +### 5.1 Explicit Markers + +The strongest triggers. These produce high-confidence extractions immediately. + +| Marker | Example | Action | +|--------|---------|--------| +| `spec:` prefix | `spec: state idle { entry: resetTimer() }` | Parse the structured spec snippet directly | +| Fenced code block with language tag | ` ```speckl state idle { ... } ``` ` | Parse as Speckl micro-syntax | +| `state:` prefix | `state: loading` | Extract state element | +| `transition:` prefix | `transition: idle → loading on START` | Extract transition element | +| `event:` prefix | `event: USER_LOGIN { payload: { userId } }` | Extract event definition | +| `constraint:` prefix | `constraint: no transition from shutdown to any state` | Extract invariant | + +**Pre-parser for explicit markers:** + +```typescript +function detectExplicitMarkers(turn: ConversationTurn): MarkerDetection[] { + const markers: MarkerDetection[] = []; + + // Check message prefix + for (const marker of REGISTERED_MARKERS) { + if (turn.content.trimStart().startsWith(marker.prefix)) { + markers.push({ + marker: marker.prefix, + content: turn.content.slice(marker.prefix.length).trim(), + confidence: 0.98, // Explicit markers are near-certain + }); + } + } + + // Check structured blocks + if (turn.metadata?.structuredBlock) { + markers.push({ + marker: "structured_block", + content: turn.metadata.structuredBlock.content, + language: turn.metadata.structuredBlock.language, + confidence: 0.95, + }); + } + + return markers; +} +``` + +### 5.2 Implicit Linguistic Patterns + +When no explicit markers exist, the extractor uses NLP patterns keyed to the active domain. + +**State machine domain patterns:** + +| Pattern | Example | Extraction | +|---------|---------|------------| +| "when [event], [action]" | "when the user clicks submit, validate the form" | Transition with event="click submit" | +| "if [condition] then [action]" | "if the balance is zero, show an error" | Guard condition on transition | +| "[noun] is a state" | "idle is the starting state" | State with isInitial=true | +| "goes from [A] to [B]" | "it goes from loading to ready" | Transition from→to | +| "while in [state], [invariant]" | "while in loading, the spinner is visible" | State invariant | +| "after [event], enter [state]" | "after timeout, enter the error state" | Transition on event | +| "can't [action] when [state]" | "can't submit when loading" | Negative constraint | + +**Implementation** as pluggable pattern sets: + +```typescript +interface ExtractionPattern { + id: string; + domain: DomainId; + regex: RegExp; // or NLP intent classifier + extract: (match: RegExpMatchArray, turn: ConversationTurn, sir: SIRDocument) => ExtractionCandidate; + baseConfidence: number; // Starting confidence before modifiers +} + +// Registered by domain +const STATE_MACHINE_PATTERNS: ExtractionPattern[] = [ + { + id: "transition-when", + domain: "state-machine", + regex: /when\s+(?:the\s+)?(.+?)(?:\s*,\s*|\s+then\s+)(.+)/i, + extract: (match, turn) => ({ + elementType: "transition", + data: { + event: match[1].trim(), + actions: [match[2].trim()], + }, + confidence: 0.55, + }), + baseConfidence: 0.55, + }, + { + id: "state-declaration", + domain: "state-machine", + regex: /(\w+)\s+is\s+(?:a|the)\s+(initial|starting|final|end|error)\s+state/i, + extract: (match) => ({ + elementType: "state", + data: { + name: match[1], + isInitial: ["initial", "starting"].includes(match[2]), + isFinal: ["final", "end"].includes(match[2]), + }, + confidence: 0.50, + }), + baseConfidence: 0.50, + }, + // ... more patterns +]; +``` + +### 5.3 Hybrid Trigger: The "Spec Block" + +A special trigger: when the human writes a spec-like block without the explicit `spec:` prefix, the extractor can still recognize the structure: + +``` +The system has these states: + +- IDLE: waiting for input, spinner hidden +- LOADING: fetching data, spinner visible +- ERROR: request failed, shows error toast +- SUCCESS: data rendered + +Transitions: +- IDLE → LOADING when user submits query +- LOADING → SUCCESS when data arrives +- LOADING → ERROR when request fails +- ERROR → IDLE when user dismisses toast +``` + +The extractor detects this as a "structured specification block" using heuristics: +- Bullet lists with `→` arrows +- Semicolon-separated key-value pairs +- Sections with headers like "States:" / "Transitions:" + +--- + +## 6. Incremental Refinement + +### 6.1 Refinement Protocol + +When a new conversation turn modifies an existing SIR element: + +```typescript +async function refineElement( + existingElement: SIRElement, + refinementCandidate: ExtractionCandidate, + sourceTurn: ConversationTurn +): Promise { + // 1. Compute the field-level diff + const diff = computeDiff(existingElement.data, refinementCandidate.data); + + // 2. Check if confidence drops below threshold after revision + const newConfidence = computeConfidence(refinementCandidate, sourceTurn, + { existingElement, domainWeights: getDomainWeights(existingElement.domain) } + ); + + // 3. If confidence dropped too far, flag instead of applying + if (newConfidence < CONFIDENCE.REVIEW) { + return { + action: "flag_revision_uncertain", + element: setFlag(existingElement, "needsReview", true), + message: `Refinement of '${existingElement.id}' would drop confidence to ${newConfidence}. Holding for review.`, + diff, + }; + } + + // 4. Apply the refinement — bump version, append provenance + const refined: SIRElement = { + ...existingElement, + data: { ...existingElement.data, ...refinementCandidate.data }, + confidence: newConfidence, + version: existingElement.version + 1, + updatedAt: Date.now(), + provenance: { + entries: [ + ...existingElement.provenance.entries, + { + turnId: sourceTurn.id, + action: "refined", + agentId: sourceTurn.agentId, + humanConfirmed: sourceTurn.speaker === "human", + timestamp: sourceTurn.timestamp, + confidenceAtTime: newConfidence, + diff, + }, + ], + }, + }; + + return { action: "applied", element: refined, diff }; +} +``` + +### 6.2 Version History + +Every element maintains its full version history. The SIR can reconstruct any element at any point in time: + +```typescript +function getElementAtVersion(element: SIRElement, targetVersion: number): SIRElement { + let current = element.data; + const entries = element.provenance.entries + .filter(e => e.version && e.version <= targetVersion) + .sort((a, b) => a.version! - b.version!); + + for (const entry of entries) { + if (entry.diff) { + for (const change of entry.diff.changes) { + // Reverse the change to reconstruct previous state + setNestedValue(current, change.field, change.oldValue); + } + } + } + + return { ...element, data: current, version: targetVersion }; +} +``` + +### 6.3 Confidence Decay on Revision + +When a human revises an element (especially implicitly), confidence should be recalculated, not just carried forward. The new confidence depends on: + +1. **Explicitness of the revision**: `spec:` prefix → 0.98; vague mention → 0.45 +2. **Consistency with related elements**: Does the revision make sense given transitions referencing this state? +3. **Completeness**: Did they revise one field but leave others potentially stale? + +--- + +## 7. Domain Awareness + +### 7.1 Domain Registration + +The extraction layer is domain-agnostic. Domain-specific extraction logic is registered as a **Domain Backend**: + +```typescript +interface DomainBackend { + domain: DomainId; + displayName: string; + + // What SIR element types does this domain use? + elementTypes: SIRType[]; + + // Extraction patterns for this domain + patterns: ExtractionPattern[]; + + // LLM extraction prompt template + extractionPrompt: (transcript: string, existingElements: SIRElement[]) => string; + + // How to validate an extracted element + validator: (element: SIRElement, sir: SIRDocument) => ValidationResult; + + // How to merge/dedup candidates + merger: (existing: SIRElement, incoming: ExtractionCandidate) => SIRElement | null; + + // Confidence weights for this domain + confidenceWeights: ConfidenceWeights; + + // Explicit marker prefixes this domain recognizes + markerPrefixes: string[]; +} + +interface ConfidenceWeights { + explicitMarker: number; // How much to weight explicit markers + structuredBlock: number; // How much to weight code fences + patternMatch: number; // How much to weight NLP pattern matches + consistency: number; // How much to weight consistency with existing SIR + speakerAuthority: number; // How much human vs agent matters + repetition: number; // How much repeated mention matters +} +``` + +### 7.2 Registering a New Domain + +```typescript +// Example: Registering the state-machine domain +extractionEngine.registerDomain({ + domain: "state-machine", + displayName: "State Machine", + elementTypes: ["state", "transition", "event", "guard", "action", "invariant"], + patterns: STATE_MACHINE_PATTERNS, + extractionPrompt: STATE_MACHINE_PROMPT, + validator: validateStateMachineElement, + merger: mergeStateMachineElement, + confidenceWeights: { + explicitMarker: 0.40, + structuredBlock: 0.35, + patternMatch: 0.25, + consistency: 0.15, + speakerAuthority: 0.05, + repetition: 0.10, + }, + markerPrefixes: ["spec:", "state:", "transition:", "event:", "guard:", "action:", "constraint:"], +}); +``` + +### 7.3 Domain Detection + +When the domain is ambiguous, the extraction layer heuristically detects it: + +```typescript +function detectDomain( + turns: ConversationTurn[], + registeredDomains: DomainBackend[] +): DomainDetectionResult { + const scores: Record = {}; + + for (const domain of registeredDomains) { + scores[domain.domain] = 0; + + for (const turn of turns) { + // Check explicit markers + for (const marker of domain.markerPrefixes) { + if (turn.content.includes(marker)) { + scores[domain.domain] += 5; + } + } + + // Check pattern matches + for (const pattern of domain.patterns) { + if (pattern.regex.test(turn.content)) { + scores[domain.domain] += 2; + } + } + + // Check keyword density + const keywords = getDomainKeywords(domain.domain); + for (const kw of keywords) { + const count = (turn.content.match(new RegExp(kw, 'gi')) || []).length; + scores[domain.domain] += count * 0.5; + } + } + } + + // Return best match, or "general" if no clear winner + const best = Object.entries(scores).sort((a, b) => b[1] - a[1])[0]; + + if (best[1] < 3) { + return { domain: "general", confidence: 0.3, scores }; + } + + return { + domain: best[0] as DomainId, + confidence: Math.min(best[1] / 20, 1.0), + scores, + }; +} +``` + +### 7.4 Default Fallback Domain + +When no domain is detected, the extraction layer operates in "general" mode: +- Extracts only `entity`, `relationship`, `property`, and `constraint` element types +- Uses generic NLP patterns (noun phrases as entities, verb phrases as relationships) +- Confidence is capped at 0.60 (tentative) +- Flags everything for review by default + +--- + +## 8. Practical API Design + +### 8.1 ExtractionEngine + +The central interface exposed to the Speckl agent: + +```typescript +class ExtractionEngine { + // === Configuration === + + /** Register a domain backend for extraction */ + registerDomain(backend: DomainBackend): void; + + /** Set the active domain (switches extraction patterns, prompts, etc.) */ + setActiveDomain(domain: DomainId): void; + + /** Configure extraction mode */ + setMode(mode: ExtractionMode): void; + + /** Override confidence thresholds for this session */ + setConfidenceThresholds(thresholds: Partial): void; + + // === Core Operations === + + /** + * Extract from a single conversation turn. + * Entry point for real-time and interactive modes. + */ + extract(turn: ConversationTurn): Promise; + + /** + * Extract from a batch of turns. + * Entry point for batch mode. + */ + extractBatch(turns: ConversationTurn[], options?: BatchOptions): Promise; + + /** + * Run deliberative extraction with multiple agent perspectives. + */ + extractDeliberative( + turn: ConversationTurn, + debaters: DebaterConfig[] + ): Promise; + + /** + * Get the current SIR document. + */ + getSIR(): SIRDocument; + + /** + * Get extraction statistics for this session. + */ + getStats(): ExtractionStats; +} + +type ExtractionMode = "realtime" | "batch" | "deliberative" | "interactive"; +``` + +### 8.2 ExtractionResult + +The return type from every extraction call: + +```typescript +interface ExtractionResult { + /** Unique ID for this extraction run */ + extractionId: string; + + /** Which mode was used */ + mode: ExtractionMode; + + /** Source turn(s) that triggered this extraction */ + sourceTurnIds: string[]; + + /** Elements that were successfully added to the SIR */ + added: SIRMutation[]; + + /** Existing elements that were refined (with diffs) */ + refined: SIRMutation[]; + + /** Elements that were deprecated or replaced */ + deprecated: SIRMutation[]; + + /** Candidates that didn't meet the auto-accept threshold */ + flagged: FlaggedCandidate[]; + + /** Ambiguities detected during extraction */ + ambiguities: ExtractionAmbiguity[]; + + /** Conflicts found between new extractions and existing elements */ + conflicts: ElementConflict[]; + + /** If mode is "interactive", clarification questions */ + clarifications?: ClarificationRequest[]; + + /** Human-readable summary of what changed */ + summary: string; + + /** Timestamp */ + timestamp: number; +} + +interface SIRMutation { + elementId: string; + element: SIRElement; + action: "created" | "updated" | "deprecated" | "replaced"; + diff?: ElementDiff; + provenance: ProvenanceEntry; +} + +interface FlaggedCandidate { + candidateId: string; + candidate: ExtractionCandidate; + reason: "low_confidence" | "ambiguous" | "conflicts"; + confidence: number; + suggestedQuestion?: string; +} + +interface ExtractionAmbiguity { + id: string; + description: string; + interpretations: Interpretation[]; + sourceTurnId: string; + sourceFragment: string; // The part of the turn that's ambiguous +} + +interface ElementConflict { + existingElementId: string; + incomingCandidate: ExtractionCandidate; + conflictType: "value_mismatch" | "semantic_contradiction" | "redundant"; + description: string; + suggestedResolution: "replace" | "merge" | "reject" | "ask_human"; +} + +interface ClarificationRequest { + id: string; + question: string; + context: string; + options?: string[]; // If multiple-choice, provide options + elementIds: string[]; // Which elements this clarification would affect +} +``` + +### 8.3 SIRDocument + +The overall SIR that the extraction layer populates: + +```typescript +class SIRDocument { + /** All elements keyed by ID */ + elements: Map; + + /** Human-readable project name */ + projectName: string; + + /** Active domain(s) */ + domains: DomainId[]; + + /** Metadata about the conversation session */ + session: { + sessionId: string; + startedAt: number; + turnCount: number; + participants: string[]; + }; + + /** Pending ambiguity resolutions (for interactive mode) */ + pendingAmbiguities: PendingAmbiguity[]; + + /** Pending human confirmations */ + pendingConfirmations: PendingConfirmation[]; + + // === Query Methods === + + /** Get all elements of a given type in a domain */ + getElementsByType(domain: DomainId, type: SIRType): SIRElement[]; + + /** Get all unconfirmed elements */ + getUnconfirmed(): SIRElement[]; + + /** Get all elements that need review */ + getNeedsReview(): SIRElement[]; + + /** Get element version history */ + getHistory(elementId: string): ProvenanceEntry[]; + + /** Reconstruct element at a specific version */ + getElementAtVersion(elementId: string, version: number): SIRElement; + + /** Find elements contributed by a specific turn */ + getElementsByTurn(turnId: string): SIRElement[]; + + /** Validate the entire SIR for consistency */ + validate(): ValidationReport; + + /** Serialize SIR to a stable format (for compilation) */ + serialize(): string; + + /** Generate a human-readable summary of the spec so far */ + summarize(): string; +} +``` + +### 8.4 Usage Example: Full Flow + +```typescript +// --- Session setup --- +const engine = new ExtractionEngine(); + +// Register the state machine domain +engine.registerDomain(stateMachineBackend); +engine.setActiveDomain("state-machine"); +engine.setMode("realtime"); + +const sir = engine.getSIR(); + +// --- Turn 1: Human describes the system --- +const turn1: ConversationTurn = { + id: "turn-001", + timestamp: Date.now(), + speaker: "human", + content: "The system starts in an idle state. When the user clicks 'Search', it goes to a loading state.", +}; + +const result1 = await engine.extract(turn1); +console.log(result1.summary); +// → "Added 2 states (idle, loading) and 1 transition (idle→loading on 'Search') with low confidence." +console.log(result1.added.length); // → 3 +console.log(result1.added[0].element.confidence); // → ~0.55 (pattern match) +console.log(result1.added[0].element.flags.isTentative); // → true + +// --- Turn 2: Human gets more specific --- +const turn2: ConversationTurn = { + id: "turn-002", + timestamp: Date.now(), + speaker: "human", + content: "spec: state idle { isInitial: true, entry: [hideSpinner] }", +}; + +const result2 = await engine.extract(turn2); +console.log(result2.summary); +// → "Refined state 'idle': added isInitial, entry action. Confidence: 0.98." +console.log(result2.refined[0].element.confidence); // → 0.98 +console.log(result2.refined[0].element.flags.isTentative); // → false (explicit marker resolved it) + +// --- Turn 3: Ambiguity --- +const turn3: ConversationTurn = { + id: "turn-003", + timestamp: Date.now(), + speaker: "human", + content: "After authentication, go to the dashboard. But what about failed auth?", +}; + +// Switch to interactive mode for this turn +engine.setMode("interactive"); + +const result3 = await engine.extract(turn3); +// result3.clarifications = [{ +// question: "Is 'authentication' a state or an event? Should 'dashboard' be a state?", +// options: [ +// "authentication is a state, dashboard is a state, transition between them", +// "authentication is an event, dashboard is a state, transition on auth event", +// "authentication is a guard condition on the transition to dashboard" +// ] +// }] + +// --- End of session --- +const finalSIR = sir.serialize(); +const stats = engine.getStats(); +console.log(stats); +// → { +// totalElements: 5, +// confirmedElements: 2, +// tentativeElements: 2, +// pendingAmbiguities: 1, +// totalTurns: 3, +// avgConfidence: 0.67, +// domainConfidence: 0.95, +// } +``` + +--- + +## 9. Extraction Pipeline Architecture + +### 9.1 Layered Processing + +The extraction pipeline processes each turn through a series of filters: + +``` +ConversationTurn + │ + ▼ +┌─────────────────────┐ +│ 1. Pre-processor │ ← Detect explicit markers, structured blocks +│ (deterministic) │ +└─────────┬───────────┘ + │ annotated turn + ▼ +┌─────────────────────┐ +│ 2. Pattern Matcher │ ← Regex/NLP patterns from registered domain +│ (deterministic) │ +└─────────┬───────────┘ + │ ExtractionCandidate[] + ▼ +┌─────────────────────┐ +│ 3. LLM Extractor │ ← LLM call for deep semantic extraction +│ (probabilistic) │ (skipped if patterns found everything) +└─────────┬───────────┘ + │ ExtractionCandidate[] (merged) + ▼ +┌─────────────────────┐ +│ 4. Confidence │ ← Score each candidate +│ Scorer │ +└─────────┬───────────┘ + │ + ▼ +┌─────────────────────┐ +│ 5. Conflict │ ← Diff against existing SIR elements +│ Detector │ +└─────────┬───────────┘ + │ + ▼ +┌─────────────────────┐ +│ 6. Ambiguity │ ← Detect multiple interpretations +│ Detector │ +└─────────┬───────────┘ + │ + ▼ +┌─────────────────────┐ +│ 7. Decision Engine │ ← Apply thresholds → add / refine / flag / ask +│ │ +└─────────┬───────────┘ + │ + ▼ + ExtractionResult +``` + +### 9.2 Pipeline Configuration + +```typescript +interface PipelineConfig { + /** Skip LLM extraction when pattern matcher confidence is high enough */ + skipLLMWhenPatternConfidenceAbove: number; // default: 0.80 + + /** Maximum tokens to send to LLM extractor */ + maxLLMTokens: number; // default: 4000 + + /** Whether to cache LLM extraction results */ + cacheLLMExtractions: boolean; + + /** When to escalate to interactive mode */ + autoSwitchToInteractive: "on_ambiguity" | "on_low_confidence" | "never"; + + /** Minimum number of turns before batch extraction is meaningful */ + minBatchSize: number; // default: 3 +} +``` + +--- + +## 10. Edge Cases and Error Handling + +### 10.1 Silent Failures Must Be Impossible + +Every error path in the extraction pipeline produces an observable result: + +| Failure | Behavior | +|---------|----------| +| LLM API timeout | Falls back to pattern-only extraction, sets `result.metadata.degraded = true`, confidence capped at pattern match scores | +| Domain not registered | Falls back to "general" domain, all extractions flagged `isTentative` | +| Turn content is noise ("lol", "ok") | Returns empty ExtractionResult with `summary: "No formal elements detected"` | +| SIR element ID collision | Deterministic ID generation (content-hash based), so collisions mean true duplicates → merged | +| Human contradicts self within same turn | Extracts both, creates AmbiguousElement, forces interactive resolution | +| Agent hallucinates an element | Provenance tracks it to agent; human review catches it; `status: "error"` when flagged | + +### 10.2 Recovery from Extraction Errors + +```typescript +/** Human marks an element as incorrectly extracted */ +function markExtractionError(elementId: string, humanTurnId: string, reason: string): void { + const element = sir.elements.get(elementId); + if (!element) throw new Error(`Element ${elementId} not found`); + + element.status = "error"; + element.flags.needsReview = false; // It's been reviewed — it's wrong + element.flags.isTentative = false; + + element.provenance.entries.push({ + turnId: humanTurnId, + action: "deprecated", + humanConfirmed: true, + timestamp: Date.now(), + confidenceAtTime: 1.0, // Human certainty + extractionRationale: `Marked as error: ${reason}`, + }); +} +``` + +--- + +## 11. Composition with Validation and Compilation Layers + +The extraction layer feeds into the Validation and Compilation layers through the SIR: + +``` +[Extraction Layer] ──produces──> SIRDocument ──consumed by──> [Validation Layer] ──produces──> ValidationReport + │ + ▼ + [Compilation Layer] +``` + +### 11.1 Contract with Validation Layer + +The extraction layer guarantees: +1. Every element has `id`, `elementType`, `domain`, `confidence`, `provenance` +2. No element enters the SIR without at least one provenance entry +3. `confidence` is always a number in [0.0, 1.0] +4. `version` is a monotonic integer ≥ 1 +5. `status` is one of the defined `ElementStatus` values + +The validation layer checks: +- Domain-specific invariants (e.g., every transition references existing states) +- Cross-element consistency (e.g., no unreachable states) +- Completeness (e.g., at least one initial state) + +### 11.2 Contract with Compilation Layer + +The compilation layer receives the validated SIR and produces deterministic output. Because of provenance tracking, compilation errors can surface as: *"Element 'transition-7' (from turn #12, extracted by agent speckl-extractor) references non-existent state 'foo'. This state was deprecated in turn #15."* + +--- + +## 12. Open Design Questions + +These are intentionally left open for later resolution: + +1. **LLM cost/performance tradeoff**: Should pattern-matching be the default, with LLM extraction as a premium upgrade? Or should LLM extraction be the primary path with patterns as fallback? + +2. **Provenance storage**: Should the full provenance chain be stored inline in the SIR (increasing document size) or externalized with references? Inline simplifies queries but adds bulk. + +3. **Human-in-the-loop granularity**: Should the human confirm every extraction, or only low-confidence ones? What about implicit confirmation (human doesn't object for 3 turns = confirmed)? + +4. **Cross-domain elements**: If a conversation spans multiple domains (e.g., state machine + data model), how do elements reference each other across domain boundaries? + +5. **Streaming extraction**: For real-time mode, should extraction be truly streaming (extract as human types each word) or per-message? Per-word seems excessive; per-sentence might be the sweet spot. + +6. **Extraction caching**: If the same conversation fragment appears in multiple turns (e.g., the agent repeats what it heard), should we cache the extraction or re-extract? + +--- + +## Appendix A: State Machine Domain — Complete Pattern Set + +```typescript +const STATE_MACHINE_PATTERNS: ExtractionPattern[] = [ + // Transitions + { + id: "transition-explicit", + domain: "state-machine", + regex: /transition:\s*(\w+)\s*(?:-+>|→)\s*(\w+)\s*(?:on|when)\s+(.+)/i, + extract: (m) => ({ elementType: "transition", data: { from: m[1], to: m[2], event: m[3] }, confidence: 0.90 }), + baseConfidence: 0.90, + }, + { + id: "transition-arrow", + domain: "state-machine", + regex: /(\w+)\s*(?:-+>|→|goes to|transitions to)\s*(\w+)/i, + extract: (m) => ({ elementType: "transition", data: { from: m[1], to: m[2] }, confidence: 0.50 }), + baseConfidence: 0.50, + }, + + // States + { + id: "state-declaration", + domain: "state-machine", + regex: /(\w+)\s+is\s+(?:a|the)\s+(initial|starting|final|end|error|default)\s+state/i, + extract: (m) => ({ + elementType: "state", + data: { name: m[1], isInitial: m[2].match(/initial|starting|default/), isFinal: m[2].match(/final|end/) }, + confidence: 0.50, + }), + baseConfidence: 0.50, + }, + + // Guards + { + id: "guard-condition", + domain: "state-machine", + regex: /(?:only\s+)?if\s+(.+?)(?:\s*,\s*|\s+then\s+|$)/i, + extract: (m) => ({ elementType: "guard", data: { condition: m[1] }, confidence: 0.45 }), + baseConfidence: 0.45, + }, + + // Events + { + id: "event-definition", + domain: "state-machine", + regex: /event\s*:\s*(\w+)\s*(?:{([^}]*)}|with\s+(.+))?/i, + extract: (m) => ({ elementType: "event", data: { name: m[1], payload: m[2] || m[3] }, confidence: 0.85 }), + baseConfidence: 0.85, + }, + + // Invariants + { + id: "invariant-while", + domain: "state-machine", + regex: /(?:while|when)\s+in\s+(\w+)\s*,\s*(.+)/i, + extract: (m) => ({ elementType: "invariant", data: { state: m[1], property: m[2] }, confidence: 0.55 }), + baseConfidence: 0.55, + }, + + // Negative constraints + { + id: "negative-constraint", + domain: "state-machine", + regex: /(?:can't|cannot|must\s+not|shouldn't)\s+(.+?)\s+(?:when|while|in)\s+(\w+)/i, + extract: (m) => ({ elementType: "constraint", data: { forbidden: m[1], context: m[2] }, confidence: 0.50 }), + baseConfidence: 0.50, + }, +]; +``` + +## Appendix B: LLM Extraction Prompt Template + +``` +You are a specification extraction agent for the {domain} domain. + +Your task: Extract formal {domain} elements from the following conversation transcript. + +Current SIR elements (for refinement/reference): +{existingElements} + +Domain element types: {elementTypes} + +Rules: +1. For each element you extract, provide the exact conversation fragment that supports it. +2. Assign a confidence score (0.0-1.0) for each extraction. +3. If a statement is ambiguous, note the alternative interpretations. +4. If a new statement refines an existing SIR element, reference the element ID. +5. Do NOT invent elements not present in the conversation. + +Transcript: +{transcript} + +Output a JSON array of ExtractionCandidate objects. +``` + +--- + +_This document describes the complete Extraction Protocol for Speckl. It is designed for domain extensibility, deterministic provenance tracking, and graceful degradation under uncertainty. The state machine domain serves as the reference implementation and running example._ diff --git a/docs/sir-schema.md b/docs/sir-schema.md new file mode 100644 index 0000000..4fe133a --- /dev/null +++ b/docs/sir-schema.md @@ -0,0 +1,1484 @@ +# Speckl Structured Intermediate Representation (SIR) — Schema Design + +## Overview + +The SIR is the central data structure in the Speckl pipeline. It bridges chaotic human/agent conversation and deterministic compilation: + +``` +Chaotic Conversation → [Extraction Layer] → SIR → [Validation Layer] → [Compilation Layer] +``` + +Everything flows through the SIR. It is the single source of truth — the document being edited by conversation and compiled into artifacts. + +### Design Principles + +1. **Domain-agnostic core, domain-specific leaves.** The SIR has a universal structure (elements, constraints, provenance) with domain-specific element types and validators. +2. **Provenance is first-class.** Every element, every property, every constraint traces back to the conversation turn that produced it. Nothing is untraceable. +3. **Confidence is quantified.** Every element carries a confidence score. Compilation can be gated on confidence thresholds. +4. **Content-addressed, not clock-addressed.** The SIR is versioned by content hash, not timestamps. Determinism depends on this. +5. **Incrementally updateable.** The SIR is a mutable document that conversation refines over time, with full diff tracking. +6. **Wire-friendly.** JSON as the primary serialization format with a defined schema. + +--- + +## 1. Core Structure + +### 1.1 The SIR Document + +```typescript +interface SIRDocument { + // Identity + sirId: string; // UUID, stable for the lifetime of this specification + sirVersion: number; // Monotonic counter, incremented on every mutation + sirHash: string; // sha256 of the canonical JSON (computed on save) + + // Declaration + schemaVersion: "1.0.0"; // SIR schema version (not the spec version) + domain: DomainDeclaration; + + // Content + elements: Record; // elementId → element + constraints: SIRConstraint[]; // cross-element constraints + composition: CompositionRule[]; // how elements compose + + // Metadata + metadata: SIRMetadata; +} + +interface SIRMetadata { + title: string; // Human-readable name: "Login Flow", "User Profile Schema" + description: string; // Free-text description of what this spec defines + tags: string[]; // For searchability: ["auth", "security", "login"] + createdAt: number; // Unix ms + updatedAt: number; // Unix ms + createdBy: string; // agentId or "human" + conversationId?: string; // Link to the conversation that produced this + readiness: ReadinessLevel; +} + +type ReadinessLevel = "draft" | "functional" | "validated" | "locked"; +``` + +### 1.2 Concrete JSON Example — Empty SIR + +```json +{ + "sirId": "b8d4f3a2-1c6e-4f89-a3e7-2b1c5d9f0a4e", + "sirVersion": 1, + "sirHash": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "1.0.0", + "domain": { + "primary": "state-machine", + "version": "1.0.0", + "secondary": [] + }, + "elements": {}, + "constraints": [], + "composition": [], + "metadata": { + "title": "Untitled Specification", + "description": "", + "tags": [], + "createdAt": 1778392000000, + "updatedAt": 1778392000000, + "createdBy": "agent-engineering", + "readiness": "draft" + } +} +``` + +--- + +## 2. Element Model + +### 2.1 What Is a Specification Element? + +A specification element is a typed, identified node in the SIR graph. Every element has a common header and domain-specific properties. Elements reference each other, forming a directed graph. + +### 2.2 The Common Header + +```typescript +interface SIRElement { + // Identity + id: string; // UUID, stable across revisions + elementType: string; // Domain-specific type: "state", "transition", "table", "step", etc. + + // Lifecycle + status: ElementStatus; + version: number; // Monotonic counter for this element + + // Provenance + provenance: ProvenanceChain; + + // Content + label: string; // Human-readable: "Logged Out", "User Table" + description: string; // Natural language: what is this, why does it exist + properties: Record; // Typed key-value store + + // Relationships + references: ElementReference[]; // What this element points to + subElements: string[]; // Child element IDs (composition) + + // Confidence + confidence: number; // 0.0 – 1.0 + confidenceBreakdown: ConfidenceBreakdown; + + // Flags + flags: ElementFlags; + + // Timestamps + createdAt: number; + updatedAt: number; +} + +type ElementStatus = + | "proposed" // Extracted from conversation, not yet confirmed + | "draft" // Confirmed but incomplete + | "validated" // Passed validation + | "locked" // Cannot be modified (compiled artifact references this) + | "deprecated" // Superseded by newer element + | "rejected"; // Human explicitly rejected this extraction +``` + +### 2.3 Property Values + +Properties are typed so validators and compilers can reason about them: + +```typescript +type PropertyValue = + | { type: "string"; value: string } + | { type: "number"; value: number } + | { type: "boolean"; value: boolean } + | { type: "enum"; value: string; allowedValues: string[] } + | { type: "reference"; value: string; targetType: string } + | { type: "array"; value: PropertyValue[]; itemType: string } + | { type: "code"; value: string; language: string } + | { type: "expression"; value: string; expressionType: string }; + +// Each property value includes its own provenance, confidence, and flags +interface PropertyValue { + type: PropertyType; + value: any; + confidence: number; + provenance: ProvenanceChain; // Who set this value, when + flags: { tentative: boolean; needsReview: boolean }; +} +``` + +### 2.4 Element References + +References form the graph: + +```typescript +interface ElementReference { + refId: string; // Target element ID + refType: string; // What kind of reference: "target", "parent", "depends_on", "implements" + label: string; // Human-readable: "goes to", "belongs to", "validates" + cardinality: string; // "1", "0..1", "1..n", "0..n" + conditions?: string; // When this reference is active: "when guard evaluates true" + confidence: number; + provenance: ProvenanceChain; +} +``` + +### 2.5 ElementFlags + +```typescript +interface ElementFlags { + needsReview: boolean; + isAmbiguous: boolean; + isTentative: boolean; + isContradicted: boolean; + sourceIsExplicit: boolean; + hasGaps: boolean; // Missing required properties +} +``` + +### 2.6 Confidence Breakdown + +Confidence isn't just a single number — it decomposes: + +```typescript +interface ConfidenceBreakdown { + overall: number; + factors: { + explicitMarker: number; // 0–1: strength of explicit extraction cues + structuralPattern: number; // 0–1: how well the text matches known patterns + consistency: number; // 0–1: consistency with rest of SIR + speakerAuthority: number; // 0–1: human (0.9) vs agent (0.6) vs debated (0.4) + repetition: number; // 0–1: mentioned across multiple turns + }; + lastEvaluated: number; // Unix ms +} +``` + +### 2.7 Concrete JSON Example — State Machine Elements + +A login flow with states, transitions, and guards: + +```json +{ + "state-logged-out": { + "id": "state-logged-out", + "elementType": "state", + "status": "validated", + "version": 1, + "provenance": { + "entries": [ + { + "turnId": "turn-3", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392100000, + "confidenceAtTime": 0.92, + "extractionRationale": "Explicit state declaration with spec: prefix" + } + ] + }, + "label": "Logged Out", + "description": "User is not authenticated. Entry point for the login flow.", + "properties": { + "isInitial": { "type": "boolean", "value": true }, + "isTerminal": { "type": "boolean", "value": false } + }, + "references": [ + { "refId": "transition-login-submit", "refType": "outgoing", "label": "on submit", "cardinality": "1" } + ], + "subElements": [], + "confidence": 0.92, + "confidenceBreakdown": { + "overall": 0.92, + "factors": { + "explicitMarker": 1.0, + "structuralPattern": 0.95, + "consistency": 0.90, + "speakerAuthority": 0.90, + "repetition": 0.85 + }, + "lastEvaluated": 1778392100000 + }, + "flags": { + "needsReview": false, + "isAmbiguous": false, + "isTentative": false, + "isContradicted": false, + "sourceIsExplicit": true, + "hasGaps": false + }, + "createdAt": 1778392100000, + "updatedAt": 1778392100000 + }, + + "state-logging-in": { + "id": "state-logging-in", + "elementType": "state", + "status": "validated", + "version": 1, + "provenance": { + "entries": [ + { + "turnId": "turn-5", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": false, + "timestamp": 1778392150000, + "confidenceAtTime": 0.78, + "extractionRationale": "Implicit from 'while the credentials are being checked' description" + } + ] + }, + "label": "Logging In", + "description": "Credentials submitted, awaiting authentication result.", + "properties": { + "isInitial": { "type": "boolean", "value": false }, + "isTerminal": { "type": "boolean", "value": false }, + "timeout": { "type": "number", "value": 30000 }, + "timeoutAction": { "type": "string", "value": "return error to user" } + }, + "references": [ + { "refId": "transition-credential-check", "refType": "incoming", "label": "entered from", "cardinality": "1" }, + { "refId": "transition-auth-success", "refType": "outgoing", "label": "on success", "cardinality": "1" }, + { "refId": "transition-auth-failure", "refType": "outgoing", "label": "on failure", "cardinality": "1" }, + { "refId": "transition-auth-timeout", "refType": "outgoing", "label": "on timeout", "cardinality": "0..1" } + ], + "subElements": [], + "confidence": 0.78, + "confidenceBreakdown": { + "overall": 0.78, + "factors": { + "explicitMarker": 0.30, + "structuralPattern": 0.80, + "consistency": 0.90, + "speakerAuthority": 0.90, + "repetition": 0.60 + }, + "lastEvaluated": 1778392150000 + }, + "flags": { + "needsReview": true, + "isAmbiguous": false, + "isTentative": true, + "isContradicted": false, + "sourceIsExplicit": false, + "hasGaps": false + }, + "createdAt": 1778392150000, + "updatedAt": 1778392150000 + }, + + "state-authenticated": { + "id": "state-authenticated", + "elementType": "state", + "status": "validated", + "version": 1, + "provenance": { + "entries": [ + { + "turnId": "turn-3", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392100000, + "confidenceAtTime": 0.95, + "extractionRationale": "Explicit state declaration with spec: prefix" + } + ] + }, + "label": "Authenticated", + "description": "User has valid session. The 'inside' state for the application.", + "properties": { + "isInitial": { "type": "boolean", "value": false }, + "isTerminal": { "type": "boolean", "value": false }, + "sessionDuration": { "type": "number", "value": 3600000 }, + "onEntry": { "type": "code", "value": "createSession(user); redirect('/dashboard')", "language": "pseudocode" } + }, + "references": [ + { "refId": "transition-auth-success", "refType": "incoming", "label": "entered via", "cardinality": "1" }, + { "refId": "transition-logout", "refType": "outgoing", "label": "on logout", "cardinality": "1" }, + { "refId": "transition-session-expire", "refType": "outgoing", "label": "on expiry", "cardinality": "1" } + ], + "subElements": [], + "confidence": 0.95, + "confidenceBreakdown": { + "overall": 0.95, + "factors": { + "explicitMarker": 1.0, + "structuralPattern": 0.95, + "consistency": 0.95, + "speakerAuthority": 0.90, + "repetition": 0.90 + }, + "lastEvaluated": 1778392100000 + }, + "flags": { + "needsReview": false, + "isAmbiguous": false, + "isTentative": false, + "isContradicted": false, + "sourceIsExplicit": true, + "hasGaps": false + }, + "createdAt": 1778392100000, + "updatedAt": 1778392100000 + }, + + "state-locked-out": { + "id": "state-locked-out", + "elementType": "state", + "status": "validated", + "version": 1, + "provenance": { + "entries": [ + { + "turnId": "turn-8", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392250000, + "confidenceAtTime": 0.88, + "extractionRationale": "Extracted from 'after 3 failed attempts, lock the account for 15 minutes'" + } + ] + }, + "label": "Locked Out", + "description": "Account temporarily locked after too many failed attempts.", + "properties": { + "isInitial": { "type": "boolean", "value": false }, + "isTerminal": { "type": "boolean", "value": false }, + "lockDuration": { "type": "number", "value": 900000 }, + "maxAttempts": { "type": "number", "value": 3 } + }, + "references": [ + { "refId": "transition-auth-failure", "refType": "incoming", "label": "entered via", "cardinality": "1" }, + { "refId": "transition-lock-expire", "refType": "outgoing", "label": "on expiry", "cardinality": "1" } + ], + "subElements": [], + "confidence": 0.88, + "confidenceBreakdown": { + "overall": 0.88, + "factors": { + "explicitMarker": 0.60, + "structuralPattern": 0.90, + "consistency": 0.90, + "speakerAuthority": 0.90, + "repetition": 0.75 + }, + "lastEvaluated": 1778392250000 + }, + "flags": { + "needsReview": false, + "isAmbiguous": false, + "isTentative": false, + "isContradicted": false, + "sourceIsExplicit": false, + "hasGaps": false + }, + "createdAt": 1778392250000, + "updatedAt": 1778392250000 + }, + + "transition-login-submit": { + "id": "transition-login-submit", + "elementType": "transition", + "status": "validated", + "version": 1, + "provenance": { + "entries": [ + { + "turnId": "turn-4", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392120000, + "confidenceAtTime": 0.94, + "extractionRationale": "Explicit transition declared: 'when user submits credentials, transition to logging-in'" + } + ] + }, + "label": "Submit Credentials", + "description": "User submits username/password from login form.", + "properties": { + "trigger": { "type": "string", "value": "form_submit" }, + "from": { "type": "reference", "value": "state-logged-out", "targetType": "state" }, + "to": { "type": "reference", "value": "state-logging-in", "targetType": "state" } + }, + "references": [], + "subElements": ["guard-validate-input"], + "confidence": 0.94, + "confidenceBreakdown": { + "overall": 0.94, + "factors": { + "explicitMarker": 1.0, + "structuralPattern": 0.95, + "consistency": 0.95, + "speakerAuthority": 0.90, + "repetition": 0.90 + }, + "lastEvaluated": 1778392120000 + }, + "flags": { + "needsReview": false, + "isAmbiguous": false, + "isTentative": false, + "isContradicted": false, + "sourceIsExplicit": true, + "hasGaps": false + }, + "createdAt": 1778392120000, + "updatedAt": 1778392120000 + }, + + "transition-credential-check": { + "id": "transition-credential-check", + "elementType": "transition", + "status": "validated", + "version": 1, + "provenance": { + "entries": [ + { + "turnId": "turn-5", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": false, + "timestamp": 1778392150000, + "confidenceAtTime": 0.65, + "extractionRationale": "Inferred intermediate step between submit and auth result" + } + ] + }, + "label": "Check Credentials", + "description": "System validates credentials against auth provider.", + "properties": { + "trigger": { "type": "string", "value": "internal" }, + "from": { "type": "reference", "value": "state-logging-in", "targetType": "state" }, + "action": { "type": "code", "value": "authProvider.verify(username, password)", "language": "pseudocode" } + }, + "references": [], + "subElements": [], + "confidence": 0.65, + "confidenceBreakdown": { + "overall": 0.65, + "factors": { + "explicitMarker": 0.10, + "structuralPattern": 0.70, + "consistency": 0.80, + "speakerAuthority": 0.60, + "repetition": 0.30 + }, + "lastEvaluated": 1778392150000 + }, + "flags": { + "needsReview": true, + "isAmbiguous": false, + "isTentative": true, + "isContradicted": false, + "sourceIsExplicit": false, + "hasGaps": true + }, + "createdAt": 1778392150000, + "updatedAt": 1778392150000 + }, + + "transition-auth-success": { + "id": "transition-auth-success", + "elementType": "transition", + "status": "validated", + "version": 1, + "provenance": { + "entries": [ + { + "turnId": "turn-6", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392180000, + "confidenceAtTime": 0.96, + "extractionRationale": "Explicit: 'if credentials valid, go to authenticated'" + } + ] + }, + "label": "Auth Success", + "description": "Credentials are valid. Transition to authenticated state.", + "properties": { + "trigger": { "type": "string", "value": "auth_success" }, + "from": { "type": "reference", "value": "state-logging-in", "targetType": "state" }, + "to": { "type": "reference", "value": "state-authenticated", "targetType": "state" } + }, + "references": [], + "subElements": ["guard-credentials-valid"], + "confidence": 0.96, + "confidenceBreakdown": { + "overall": 0.96, + "factors": { + "explicitMarker": 1.0, + "structuralPattern": 0.95, + "consistency": 0.95, + "speakerAuthority": 0.90, + "repetition": 0.95 + }, + "lastEvaluated": 1778392180000 + }, + "flags": { + "needsReview": false, + "isAmbiguous": false, + "isTentative": false, + "isContradicted": false, + "sourceIsExplicit": true, + "hasGaps": false + }, + "createdAt": 1778392180000, + "updatedAt": 1778392180000 + }, + + "guard-credentials-valid": { + "id": "guard-credentials-valid", + "elementType": "guard", + "status": "validated", + "version": 1, + "provenance": { + "entries": [ + { + "turnId": "turn-6", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392180000, + "confidenceAtTime": 0.96, + "extractionRationale": "Guard condition: credentials valid" + } + ] + }, + "label": "Credentials Valid", + "description": "Username exists and password hash matches.", + "properties": { + "condition": { "type": "expression", "value": "user.exists AND password.hash == stored.hash", "expressionType": "boolean" } + }, + "references": [ + { "refId": "transition-auth-success", "refType": "guards", "label": "guards transition", "cardinality": "1" } + ], + "subElements": [], + "confidence": 0.96, + "confidenceBreakdown": { + "overall": 0.96, + "factors": { + "explicitMarker": 1.0, + "structuralPattern": 0.95, + "consistency": 0.95, + "speakerAuthority": 0.90, + "repetition": 0.90 + }, + "lastEvaluated": 1778392180000 + }, + "flags": { + "needsReview": false, + "isAmbiguous": false, + "isTentative": false, + "isContradicted": false, + "sourceIsExplicit": true, + "hasGaps": false + }, + "createdAt": 1778392180000, + "updatedAt": 1778392180000 + } +} +``` + +--- + +## 3. Domain System + +### 3.1 Domain Declaration + +Every SIR declares its domain. A domain defines what element types, constraints, and composition rules are valid. + +```typescript +interface DomainDeclaration { + primary: string; // "state-machine", "data-schema", "workflow", "protocol" + version: string; // Domain schema version: "1.0.0" + secondary: string[]; // Cross-reference domains: ["api-interface"] +} +``` + +### 3.2 Domain Primitives + +Each domain defines its element types, their required properties, and their semantics: + +**State Machine Domain (`state-machine@1.0.0`):** + +| Element Type | Description | Required Properties | +|---|---|---| +| `state` | A distinct condition or mode the system can be in | `isInitial`, `isTerminal` | +| `transition` | A directed edge between states, triggered by an event | `from`, `to`, `trigger` | +| `guard` | A boolean condition evaluated before taking a transition | `condition` | +| `event` | An external or internal occurrence that may trigger transitions | `type`, `payload` | +| `action` | A side effect executed on entry, exit, or during a transition | `phase` (entry/exit/during), `code` | +| `invariant` | A condition that must always hold in a given state | `expression` | + +**Data Schema Domain (`data-schema@1.0.0`):** + +| Element Type | Description | Required Properties | +|---|---|---| +| `entity` | A structured data type (table, document, object) | `name` | +| `field` | A named property within an entity | `name`, `type`, `nullable` | +| `relation` | A relationship between entities | `from`, `to`, `kind` (one-to-one/one-to-many/many-to-many) | +| `index` | A search/ordering structure on fields | `fields`, `unique` | +| `constraint` | A data integrity rule | `expression`, `kind` (check/unique/foreign-key) | +| `enum` | A constrained set of values | `name`, `values` | + +**Workflow Domain (`workflow@1.0.0`):** + +| Element Type | Description | Required Properties | +|---|---|---| +| `step` | A unit of work in a process | `name`, `actor` (human/system/both) | +| `decision` | A branching point | `condition`, `branches` | +| `fork` | Parallel execution split | `branches` | +| `join` | Parallel execution merge | `incoming` | +| `event` | A signal within the workflow | `type` (start/end/message/timer/error) | + +### 3.3 Domain Versioning + +When a domain schema changes (e.g., `state-machine@1.0.0` → `state-machine@1.1.0`), existing SIRs carry their declared domain version. The compiler resolves the appropriate domain validator and compiler plugins for that version. + +```json +{ + "domain": { + "primary": "state-machine", + "version": "1.0.0" + } +} +``` + +A migration path exists when a domain upgrades: + +```typescript +interface DomainUpgradePath { + from: string; // "state-machine@1.0.0" + to: string; // "state-machine@1.1.0" + migration: (sir: SIRDocument) => SIRDocument; // Transform function + breakingChanges: string[]; +} +``` + +### 3.4 Multi-Domain SIR + +A SIR can declare secondary domains for cross-domain references: + +```json +{ + "domain": { + "primary": "state-machine", + "version": "1.0.0", + "secondary": ["api-interface@1.0.0"] + } +} +``` + +This allows a state machine to reference an API endpoint specification for its actions, or a workflow to reference a data schema for its payload types. Cross-domain references carry an explicit domain prefix: + +```json +{ + "id": "some-action", + "elementType": "action", + "references": [ + { + "refId": "api-interface:endpoint-create-session", + "refType": "calls", + "label": "creates session via", + "cardinality": "1" + } + ] +} +``` + +--- + +## 4. Constraint Model + +### 4.1 Constraint Types + +Constraints encode the formal rules the specification must satisfy: + +```typescript +type SIRConstraint = + | TypeConstraint + | ReferenceConstraint + | InvariantConstraint + | TemporalConstraint + | RelationalConstraint; + +// "field 'email' must be a valid email string" +interface TypeConstraint { + kind: "type"; + target: string; // Element or property path: "entity-user.field-email" + check: string; // Expression: "matches(email_pattern)" +} + +// "transition 'auth-success' must have a target state" +interface ReferenceConstraint { + kind: "reference"; + source: string; // Element ID + refType: string; // Which reference: "to", "from" + mustExist: boolean; + mustBeType?: string; // Optional: reference must point to this type +} + +// "user.email must be unique across all users" +interface InvariantConstraint { + kind: "invariant"; + target: string; // Element or element set + expression: string; // Formal expression: "forall u1,u2: u1.id != u2.id => u1.email != u2.email" + language: "speckl-expr" | "fol"; +} + +// "after 'payment-submitted' state, eventually 'payment-confirmed'" +interface TemporalConstraint { + kind: "temporal"; + formula: string; // "G(payment-submitted => F payment-confirmed)" + logic: "LTL" | "CTL"; +} + +// "field-email must be indexed if entity-user has more than N records" +interface RelationalConstraint { + kind: "relational"; + expression: string; // "entity-user.cardinality > 1000000 => exists(index-email)" +} +``` + +### 4.2 Constraint Expression Language + +Constraints use a readable expression language that maps to formal verification: + +``` +Expression → Comparison | Logical | Quantified | Reference + +Comparison → value OPERATOR value + OPERATOR → == | != | < | > | <= | >= | matches | contains + +Logical → expr AND expr | expr OR expr | NOT expr + +Quantified → forall VAR in SET: expr | exists VAR in SET: expr + +Reference → elementId.property | elementId (boolean, exists check) +``` + +Examples: +``` +"state-logging-in.isInitial == false" +"transition-auth-success.to exists AND transition-auth-success.to == state-authenticated" +"forall t in transitions: t.from exists AND t.to exists" +"not exists s1,s2 in states: s1.isInitial == true AND s2.isInitial == true" // exactly one initial state +``` + +### 4.3 Constraint Attachment + +Every constraint carries provenance and confidence: + +```typescript +interface SIRConstraint { + id: string; + kind: string; + // ... type-specific fields ... + + // Metadata + severity: "error" | "warning" | "info"; + confidence: number; + provenance: ProvenanceChain; + humanConfirmed: boolean; + + // Resolution + resolved: boolean; // True if validation passes + resolvedAt?: number; + resolutionNote?: string; +} +``` + +### 4.4 Constraint Composition + +When elements compose (e.g., a state machine contains sub-state-machines), constraints compose hierarchically: + +- A constraint on a parent element implies constraints on all children +- A constraint satisfied at the child level satisfies it at the parent level (if the constraint is existential) +- Conflicting constraints at different levels produce a validation error + +```typescript +interface CompositionRule { + kind: "contains" | "requires" | "excludes" | "parallel-to" | "sequence-of"; + parent: string; // Parent element ID + children: string[]; // Child element IDs + constraints: SIRConstraint[]; // Constraints on this composition + provenance: ProvenanceChain; +} +``` + +--- + +## 5. Provenance Model + +### 5.1 ProvenanceChain + +Every element, every property, every constraint has a provenance chain. This is the foundation of NIST SA-11 traceability: + +```typescript +interface ProvenanceChain { + entries: ProvenanceEntry[]; +} + +interface ProvenanceEntry { + turnId: string; // ConversationTurn UUID that produced this + action: ProvenanceAction; + agentId?: string; // Which agent performed the action (or "human") + humanConfirmed: boolean; // Did the human explicitly sign off? + timestamp: number; // Unix ms + confidenceAtTime: number; // Confidence snapshot at this entry + extractionRationale?: string; // Why was this extracted/modified? + diff?: SIRDiff; // For refined/updated actions: what changed +} + +type ProvenanceAction = + | "extracted" // First created from conversation + | "refined" // Incremental update, same element + | "confirmed" // Human explicitly confirmed + | "deprecated" // No longer valid, superseded + | "rejected" // Human said this extraction was wrong + | "merged" // Result of merging two SIR versions + | "imported"; // Imported from external source +``` + +### 5.2 Provenance Chains + +Elements refined across multiple turns accumulate a chain: + +```json +{ + "id": "state-logging-in", + "provenance": { + "entries": [ + { + "turnId": "turn-5", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": false, + "timestamp": 1778392150000, + "confidenceAtTime": 0.45, + "extractionRationale": "Detected 'while checking credentials' as an implicit intermediate state" + }, + { + "turnId": "turn-7", + "action": "refined", + "agentId": "human", + "humanConfirmed": true, + "timestamp": 1778392220000, + "confidenceAtTime": 0.85, + "diff": { + "changedProperties": ["timeout", "timeoutAction"], + "addedReferences": ["transition-auth-timeout"] + }, + "extractionRationale": "Scott: 'Add a 30-second timeout to the logging-in state, show an error if it fires'" + }, + { + "turnId": "turn-12", + "action": "confirmed", + "agentId": "human", + "humanConfirmed": true, + "timestamp": 1778392400000, + "confidenceAtTime": 1.0, + "extractionRationale": "Scott: 'Yeah, the login flow looks right now.'" + } + ] + } +} +``` + +### 5.3 Confidence Tracking + +Confidence is tracked both at the element level and at the property level. A low-confidence property on a high-confidence element is flagged: + +```json +{ + "id": "state-logging-in", + "confidence": 0.85, + "properties": { + "timeout": { + "type": "number", + "value": 30000, + "confidence": 0.60, + "provenance": { + "entries": [{ + "turnId": "turn-7", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": false, + "extractionRationale": "Implicit: Scott mentioned '30 seconds' — extracted as ms" + }] + } + } + } +} +``` + +--- + +## 6. Versioning and Diffing + +### 6.1 SIR Versioning + +The SIR version is a content-hash-based identifier for integrity, plus a monotonic counter for human convenience: + +``` +sirVersion: 14 // Monotonic, incremented on every save +sirHash: "sha256:a7f3..." // Content hash of the canonical JSON +``` + +The `sirHash` is computed over the canonical JSON with all elements sorted by ID, minus the `sirHash` field itself and any timestamps (which are informational, not deterministic inputs). + +### 6.2 Element-Level Diffs + +```typescript +interface SIRDiff { + // What changed + addedElements: string[]; // Element IDs + removedElements: string[]; + modifiedElements: Record; + reorderedElements: { before: string[]; after: string[] }; + + // Metadata + fromSirVersion: number; + toSirVersion: number; + fromSirHash: string; + toSirHash: string; + timestamp: number; +} +``` + +### 6.3 Merge Semantics + +Two SIR versions can be merged when they derive from a common ancestor. Conflicts arise when both branches modify the same element property: + +```typescript +interface SIRMergeResult { + success: boolean; + merged: SIRDocument; + conflicts: SIRMergeConflict[]; +} + +interface SIRMergeConflict { + elementId: string; + property: string; + branchA: PropertyValue; // Value from first branch + branchB: PropertyValue; // Value from second branch + ancestor: PropertyValue; // Original value before branching + resolution?: PropertyValue; // Human-chosen resolution +} +``` + +Conflicts are represented directly in the SIR so conversation can resolve them — the agent surfaces the conflict and asks the human which value is correct. + +--- + +## 7. Serialization + +### 7.1 Primary Format: JSON + +The canonical serialization is JSON with a fixed key ordering for deterministic hashing: + +```json +{ + "sirId": "", + "sirVersion": 0, + "sirHash": "", + "schemaVersion": "", + "domain": {}, + "elements": {}, + "constraints": [], + "composition": [], + "metadata": {} +} +``` + +Elements within `elements` are keyed by their stable ID. Constraints and composition rules are arrays ordered by ID. + +### 7.2 Binary Format (Optional) + +For large SIRs, a CBOR-based binary format can be used for transmission. The canonical representation for hashing is always JSON. + +### 7.3 Human-Readable Summary + +For display in conversation, the SIR can render as a summary: + +``` +📋 **Login Flow** (state-machine@1.0.0) · v14 · sha256:a7f3... + +**States (4):** +🟢 Logged Out (initial) · confidence 0.92 ✅ +🔄 Logging In · confidence 0.85 ⚠️ +🔒 Authenticated · confidence 0.95 ✅ +🚫 Locked Out · confidence 0.88 ✅ + +**Transitions (5):** +→ Submit Credentials: logged-out → logging-in [trigger: form_submit] +→ Check Credentials: logging-in → (internal) [needs target] ⚠️ +→ Auth Success: logging-in → authenticated [guard: credentials-valid] +→ Auth Failure: logging-in → locked-out [missing guard] ⚠️ +→ Logout: authenticated → logged-out [missing] ❌ + +**Gaps: 2 warnings, 1 error** +``` + +This format is generated by the extraction layer for inline conversation display without showing raw JSON. + +### 7.4 Size Considerations + +For large specifications (hundreds of elements), the SIR can be chunked: + +- Elements stored in a content-addressed element store +- SIR document holds references (hashes) rather than inline elements +- Fetch-on-demand for large specs + +```typescript +interface SIRElementRef { + ref: string; // content hash of the serialized element +} + +// elements: Record +// Resolution: fetch element by hash from store, assemble full graph in memory +``` + +--- + +## 8. Data Schema Domain — Complete Example + +### 8.1 User Profile Data Schema + +```json +{ + "sirId": "c3d4e5f6-7a8b-4c9d-a0e1-2f3a4b5c6d7e", + "sirVersion": 3, + "sirHash": "sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08", + "schemaVersion": "1.0.0", + "domain": { + "primary": "data-schema", + "version": "1.0.0", + "secondary": [] + }, + "elements": { + "entity-user": { + "id": "entity-user", + "elementType": "entity", + "status": "validated", + "version": 3, + "label": "User", + "description": "Core user profile entity. Every authenticated user has exactly one User record.", + "properties": { + "name": { "type": "string", "value": "users" }, + "storageEngine": { "type": "enum", "value": "postgresql", "allowedValues": ["postgresql", "sqlite", "mongodb"] }, + "softDelete": { "type": "boolean", "value": true } + }, + "references": [], + "subElements": ["field-id", "field-email", "field-name", "field-created-at", "field-updated-at", "field-deleted-at"], + "confidence": 0.95, + "flags": { "needsReview": false, "isAmbiguous": false, "isTentative": false, "isContradicted": false, "sourceIsExplicit": true, "hasGaps": false }, + "createdAt": 1778392300000, + "updatedAt": 1778392350000, + "provenance": { + "entries": [{ + "turnId": "turn-15", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392300000, + "confidenceAtTime": 0.95 + }] + }, + "confidenceBreakdown": { + "overall": 0.95, + "factors": { "explicitMarker": 1.0, "structuralPattern": 0.95, "consistency": 0.95, "speakerAuthority": 0.90, "repetition": 0.90 } + } + }, + + "field-id": { + "id": "field-id", + "elementType": "field", + "status": "validated", + "version": 1, + "label": "ID", + "description": "Primary key, auto-generated UUID.", + "properties": { + "name": { "type": "string", "value": "id" }, + "type": { "type": "enum", "value": "uuid", "allowedValues": ["uuid", "integer", "string", "boolean", "timestamp", "jsonb"] }, + "nullable": { "type": "boolean", "value": false }, + "primaryKey": { "type": "boolean", "value": true }, + "defaultValue": { "type": "expression", "value": "gen_random_uuid()", "expressionType": "sql" } + }, + "references": [], + "subElements": [], + "confidence": 0.98, + "flags": { "needsReview": false, "isAmbiguous": false, "isTentative": false, "isContradicted": false, "sourceIsExplicit": true, "hasGaps": false }, + "createdAt": 1778392310000, + "updatedAt": 1778392310000, + "provenance": { + "entries": [{ + "turnId": "turn-15", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392310000, + "confidenceAtTime": 0.98 + }] + }, + "confidenceBreakdown": { + "overall": 0.98, + "factors": { "explicitMarker": 1.0, "structuralPattern": 0.98, "consistency": 0.98, "speakerAuthority": 0.90, "repetition": 0.95 } + } + }, + + "field-email": { + "id": "field-email", + "elementType": "field", + "status": "validated", + "version": 2, + "label": "Email", + "description": "User email address. Must be unique and valid format.", + "properties": { + "name": { "type": "string", "value": "email" }, + "type": { "type": "enum", "value": "string", "allowedValues": ["uuid", "integer", "string", "boolean", "timestamp", "jsonb"] }, + "nullable": { "type": "boolean", "value": false }, + "maxLength": { "type": "number", "value": 254 }, + "unique": { "type": "boolean", "value": true } + }, + "references": [], + "subElements": [], + "confidence": 0.96, + "flags": { "needsReview": false, "isAmbiguous": false, "isTentative": false, "isContradicted": false, "sourceIsExplicit": true, "hasGaps": false }, + "createdAt": 1778392310000, + "updatedAt": 1778392340000, + "provenance": { + "entries": [ + { + "turnId": "turn-15", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": false, + "timestamp": 1778392310000, + "confidenceAtTime": 0.90 + }, + { + "turnId": "turn-17", + "action": "refined", + "agentId": "human", + "humanConfirmed": true, + "timestamp": 1778392340000, + "confidenceAtTime": 0.96, + "diff": { "changedProperties": ["unique", "maxLength"] } + } + ] + }, + "confidenceBreakdown": { + "overall": 0.96, + "factors": { "explicitMarker": 1.0, "structuralPattern": 0.95, "consistency": 0.95, "speakerAuthority": 0.90, "repetition": 0.90 } + } + }, + + "field-name": { + "id": "field-name", + "elementType": "field", + "status": "validated", + "version": 1, + "label": "Name", + "description": "User's display name.", + "properties": { + "name": { "type": "string", "value": "display_name" }, + "type": { "type": "enum", "value": "string", "allowedValues": ["uuid", "integer", "string", "boolean", "timestamp", "jsonb"] }, + "nullable": { "type": "boolean", "value": false }, + "maxLength": { "type": "number", "value": 100 } + }, + "references": [], + "subElements": [], + "confidence": 0.95, + "flags": { "needsReview": false, "isAmbiguous": false, "isTentative": false, "isContradicted": false, "sourceIsExplicit": true, "hasGaps": false }, + "createdAt": 1778392310000, + "updatedAt": 1778392310000, + "provenance": { + "entries": [{ + "turnId": "turn-15", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392310000, + "confidenceAtTime": 0.95 + }] + }, + "confidenceBreakdown": { + "overall": 0.95, + "factors": { "explicitMarker": 1.0, "structuralPattern": 0.95, "consistency": 0.95, "speakerAuthority": 0.90, "repetition": 0.90 } + } + } + }, + "constraints": [ + { + "id": "constraint-email-unique", + "kind": "invariant", + "target": "field-email", + "expression": "forall u1,u2 in entity-user: u1.id != u2.id => u1.email != u2.email", + "language": "speckl-expr", + "severity": "error", + "confidence": 1.0, + "humanConfirmed": true, + "resolved": true, + "provenance": { + "entries": [{ + "turnId": "turn-17", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392340000, + "confidenceAtTime": 1.0 + }] + } + }, + { + "id": "constraint-email-format", + "kind": "type", + "target": "field-email", + "check": "value matches email_pattern", + "severity": "error", + "confidence": 0.95, + "humanConfirmed": true, + "resolved": true, + "provenance": { + "entries": [{ + "turnId": "turn-17", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": true, + "timestamp": 1778392340000, + "confidenceAtTime": 0.95 + }] + } + } + ], + "composition": [], + "metadata": { + "title": "User Profile Schema", + "description": "Core user entity with auth-critical fields", + "tags": ["users", "auth", "profile", "database"], + "createdAt": 1778392300000, + "updatedAt": 1778392350000, + "createdBy": "agent-engineering", + "readiness": "validated" + } +} +``` + +--- + +## 9. Refinement Example — Before and After + +### 9.1 Before Refinement (Turn 5 — Initial Extraction) + +The login flow SIR after first extraction, before Scott refines it: + +```json +{ + "state-logged-out": { + "id": "state-logged-out", + "elementType": "state", + "status": "proposed", + "version": 1, + "label": "Logged Out", + "description": "User is not authenticated.", + "properties": { + "isInitial": { "type": "boolean", "value": true }, + "isTerminal": { "type": "boolean", "value": false } + }, + "references": [ + { "refId": "transition-login-submit", "refType": "outgoing", "label": "on submit", "cardinality": "1" } + ], + "subElements": [], + "confidence": 0.92, + "flags": { "needsReview": false, "isAmbiguous": false, "isTentative": false, "isContradicted": false, "sourceIsExplicit": true, "hasGaps": false }, + "createdAt": 1778392100000, + "updatedAt": 1778392100000, + "provenance": { + "entries": [{ + "turnId": "turn-3", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": false, + "timestamp": 1778392100000, + "confidenceAtTime": 0.92 + }] + } + }, + "state-authenticated": { + "id": "state-authenticated", + "elementType": "state", + "status": "proposed", + "version": 1, + "label": "Authenticated", + "description": "User has valid session.", + "properties": { + "isInitial": { "type": "boolean", "value": false }, + "isTerminal": { "type": "boolean", "value": false } + }, + "references": [ + { "refId": "transition-auth-success", "refType": "incoming", "label": "entered via", "cardinality": "1" } + ], + "subElements": [], + "confidence": 0.95, + "flags": { "needsReview": false, "isAmbiguous": false, "isTentative": false, "isContradicted": false, "sourceIsExplicit": true, "hasGaps": false }, + "createdAt": 1778392100000, + "updatedAt": 1778392100000, + "provenance": { + "entries": [{ + "turnId": "turn-3", + "action": "extracted", + "agentId": "agent-engineering", + "humanConfirmed": false, + "timestamp": 1778392100000, + "confidenceAtTime": 0.95 + }] + } + } +} +``` + +**Gaps at this stage:** +- Missing: `state-locked-out`, `state-logging-in` (not yet mentioned in conversation) +- Missing: logout transition, lockout guard, timeout behavior +- All elements status: "proposed" (not confirmed) +- `state-authenticated` has no outgoing transitions + +### 9.2 After Refinement (Turn 12 — Scott Confirms) + +The SIR from Section 2.7 shows the final state. Key changes: + +**Element-level changes:** +- `state-logged-out`: status `proposed` → `validated`; confidence 0.92 (unchanged) +- `state-authenticated`: added `sessionDuration`, `onEntry`, `transition-logout`, `transition-session-expire`; confidence 0.95 (unchanged); status → `validated` +- `state-logging-in`: ADDED — confidence 0.78 → 0.85 after refinement +- `state-locked-out`: ADDED — confidence 0.88 +- `transition-credential-check`: ADDED — low confidence 0.65, tentative, flagged +- `transition-auth-failure`: ADDED (not shown in full above but referenced) +- `guard-credentials-valid`: ADDED — confidence 0.96 + +**Provenance chain for `state-logging-in`:** +``` +turn-5: extracted (confidence 0.45, proposed) +turn-7: refined — added timeout, timeoutAction (confidence 0.85) +turn-12: confirmed by Scott (confidence 1.0, validated) +``` + +**The diff** (computed by SIRDiff): +```json +{ + "addedElements": ["state-logging-in", "state-locked-out", "transition-credential-check", "transition-auth-success", "transition-auth-failure", "transition-auth-timeout", "transition-logout", "transition-session-expire", "transition-lock-expire", "guard-credentials-valid", "guard-max-attempts", "guard-lock-timer"], + "removedElements": [], + "modifiedElements": { + "state-authenticated": { + "changedProperties": ["sessionDuration", "onEntry"], + "addedReferences": ["transition-logout", "transition-session-expire"], + "changedConstraints": [], + "confidenceDelta": 0.0 + } + } +} +``` + +--- + +## 10. Summary — The SIR Contract + +The SIR makes these guarantees to the layers above and below: + +### To the Extraction Layer: +- **Write**: accept any well-formed element, regardless of completeness; confidence and status fields accommodate uncertainty +- **Incremental**: support partial updates, property-level diffs, version tracking +- **Provenance storage**: preserve every extraction event without loss + +### To the Validation Layer: +- **Query**: give a complete typed graph with references, properties, constraints +- **Locate**: every element has a stable ID, every property has a path for precise error reporting +- **Confidence**: per-element and per-property scores available for gating decisions + +### To the Compilation Layer: +- **Deterministic hash**: content-addressable with fixed serialization order +- **Stable references**: element IDs never change; references are resolvable +- **Gap transparency**: flags and confidence scores indicate where compilation should insert stubs +- **Readiness**: graduated readiness levels for partial → full compilation + +--- + +## Open Questions + +1. **Sub-SIR granularity**: Should sub-state-machines be inline elements or referenced by content hash? Inline is simpler; hashed references scale better. +2. **Constraint expression evaluator**: How much of the constraint language should be evaluable in-process vs. delegated to external solvers (Z3, Alloy)? +3. **Real-time SIR sync**: For collaborative editing (multiple agents talking to the same human simultaneously), do we need CRDT semantics or is append-only log sufficient? +4. **Domain schema registry**: Where do domain definitions live? Inline in the SIR, in a registry service, or loaded from filesystem? +5. **Property type extensibility**: Should custom domain backends be able to define new `PropertyValue` types beyond the six built-in types? diff --git a/docs/validation-engine.md b/docs/validation-engine.md new file mode 100644 index 0000000..16c6198 --- /dev/null +++ b/docs/validation-engine.md @@ -0,0 +1,1187 @@ +# Speckl Validation Engine Design + +## Overview + +The Validation Engine is the second stage of the Speckl pipeline: + +``` +Chaotic Conversation → [Extraction Layer] → [Validation Layer] → [Compilation Layer] +``` + +It takes a Structured Intermediate Representation (SIR) and produces a `ValidationReport` that determines whether the SIR is correct, complete, and consistent enough to compile. The engine is domain-agnostic at its core with pluggable domain-specific validators. + +--- + +## 1. Architecture + +``` + ┌───────────────────────────┐ + │ Validation Engine │ + │ │ + SIR ──────────▶ │ ┌──────────────────────┐ │ + │ │ Structural Validator │ │ + │ └──────────┬───────────┘ │ + │ │ │ + │ ┌──────────▼───────────┐ │ + │ │ Reference Validator │ │ + │ └──────────┬───────────┘ │ + │ │ │ + │ ┌──────────▼───────────┐ │ + │ │ Constraint Solver │ │ + │ └──────────┬───────────┘ │ + │ │ │ + │ ┌──────────▼───────────┐ │ ┌──────────────────┐ + │ │ Domain Validators │◀─┼───│ Plugin Registry │ + │ │ (pluggable) │ │ │ │ + │ └──────────┬───────────┘ │ │ state-machine │ + │ │ │ │ workflow │ + │ ┌──────────▼───────────┐ │ │ data-model │ + │ │ Completeness Check │ │ │ api-spec │ + │ └──────────┬───────────┘ │ │ ... │ + │ │ │ └──────────────────┘ + │ ┌──────────▼───────────┐ │ + │ │ Confidence Filter │ │ + │ └──────────┬───────────┘ │ + │ │ │ + │ ┌──────────▼───────────┐ │ + │ │ Report Assembler │──┼──▶ ValidationReport + │ └──────────────────────┘ │ + └───────────────────────────┘ +``` + +### Core Principle: Fail Cheap First + +Validation passes are ordered from cheapest to most expensive. A structural check rejecting the SIR avoids running an SMT solver. This matters because some checks (temporal logic model checking, constraint solving) can be NP-hard or PSPACE-complete. + +--- + +## 2. The SIR (Recap) + +The SIR is a typed graph. Key types: + +``` +SIR { + domain: string // e.g. "state-machine" + elements: Element[] // all typed nodes + constraints: Constraint[] // cross-element constraints + confidence: Confidence // per-element confidence scores + source: SourceMap // maps elements back to conversation spans +} + +Element { + id: string // unique, e.g. "state-idle" + type: string // e.g. "state", "transition", "guard" + properties: Map // typed property bag + metadata: { + confidence: 0.0..1.0 // extraction confidence + source: SourceSpan // where in conversation this came from + required: boolean // is this element mandatory for compilation? + } +} + +Constraint { + id: string + type: "reference" | "invariant" | "type-check" | "domain" + expression: ConstraintExpr // structured expression tree + severity: "error" | "warning" +} + +Confidence { + perElement: Map + overall: 0.0..1.0 // aggregate confidence + threshold: 0.0..1.0 // configured threshold for blocking +} +``` + +--- + +## 3. Validation Categories + +### 3.1 Structural Validation + +Determines if the SIR is *well-formed* — independent of domain semantics. + +**Checks:** + +| Check | Rule | Severity | +|-------|------|----------| +| Unique IDs | No duplicate `Element.id` | ERROR | +| Required fields | Every element has `id`, `type`, `properties` | ERROR | +| Valid type registry | Every `Element.type` exists in the known type registry | ERROR | +| Property type correctness | Properties match their declared types | ERROR | +| No orphan references | Every `Constraint.expression` referencing an element ID must find that element | ERROR | +| Valid JSON/parsed form | SIR structure is parsable | ERROR | + +**Example — State Machine SIR structural errors:** + +```yaml +# This SIR has structural problems +domain: state-machine +elements: + - id: "idle" + type: "state" + properties: + name: "Idle" + is_initial: "yes" # ERROR: is_initial must be boolean, got string + - id: "running" + type: "stte" # ERROR: unknown type "stte" (typo) + # ERROR: transition references nonexistent element "stopped" + - id: "t1" + type: "transition" + properties: + from: "idle" + to: "stopped" # stopped not defined + trigger: "start" + - id: "idle" # ERROR: duplicate id "idle" + type: "state" + properties: + name: "Duplicate Idle" +``` + +**Generated Errors:** + +``` +ERROR [structural.type_mismatch] Element "idle": property "is_initial" + expected type "boolean", got "yes" (string). + → Fix: change "yes" to true. + +ERROR [structural.unknown_type] Element "running": type "stte" is not + in the type registry. Did you mean "state"? + +ERROR [structural.missing_reference] Element "t1" (transition): + target state "stopped" not found in element set. + +ERROR [structural.duplicate_id] Element id "idle" appears more than once. + ids must be unique. +``` + +### 3.2 Reference Validation + +Cross-element reference integrity. After structural checks pass, we verify that the graph of references is coherent. + +**Checks:** + +| Check | Rule | Severity | +|-------|------|----------| +| Bidirectional consistency | If A references B, B's type must be compatible with A's expectation | ERROR | +| No dangling references | All `from`/`to`/`parent`/`child` fields resolve | ERROR | +| Circular reference detection | No fatal cycles (e.g. state A's parent is state B whose parent is state A) | WARNING | +| Cardinality | If a field declares `max: 1`, having 2+ references is an error | ERROR | + +**Example — Coherent reference graph for state machines:** + +```yaml +elements: + - id: "idle" + type: "state" + properties: { name: "Idle", is_initial: true } + - id: "running" + type: "state" + properties: { name: "Running" } + - id: "t1" + type: "transition" + properties: + from: "idle" + to: "running" + trigger: "start" + - id: "t2" + type: "transition" + properties: + from: "running" + to: "idle" + trigger: "stop" +``` + +This SIR passes both structural and reference validation. + +### 3.3 Domain-Specific Validation + +Domain validators are plugins that register with the engine. Each validator specifies: +- Which `domain` it applies to +- What checks it performs +- The cost of each check (cheap/moderate/expensive) + +**State Machine Domain Validator Checks:** + +| Check | Cost | Rule | Severity | +|-------|------|------|----------| +| Initial state exists | Cheap | Exactly one state has `is_initial: true` | ERROR | +| Reachable states | Moderate | Every non-initial state is reachable via some path of transitions | WARNING | +| No dead-end states | Cheap | Every state has at least one outgoing transition, or is marked `terminal` | WARNING | +| Deterministic transitions | Moderate | For a given (state, trigger) pair, at most one transition fires (or guards disambiguate) | WARNING for ambiguity | +| Guard completeness | Expensive | For a state with guarded transitions on the same trigger, guards must be exhaustive (sum of guard cases covers all possibilities) | WARNING | +| No livelock | Expensive | No infinite cycle of transitions without progress | ERROR | +| Terminal states | Cheap | Terminal states have no outgoing transitions | INFO (ok if intended) | +| Liveness | Expensive | Desired terminal states are reachable from all states (if spec'd) | WARNING | + +**Example — State Machine with domain errors:** + +```yaml +domain: state-machine +elements: + - id: "idle" + type: "state" + properties: { name: "Idle" } + # ERROR: missing is_initial — no initial state defined + - id: "running" + type: "state" + properties: { name: "Running" } + - id: "orphan" + type: "state" + properties: { name: "Orphan" } + # WARNING: unreachable — no transition targets this state + - id: "t1" + type: "transition" + properties: + from: "idle" + to: "running" + trigger: "start" + - id: "t2" + type: "transition" + properties: + from: "running" + to: "running" # WARNING: self-loop, no progress + trigger: "retry" + - id: "t3" + type: "transition" + properties: + from: "idle" + to: "running" + trigger: "start" + guard: "is_admin == true" + # WARNING: conflict with t1 — same (from, trigger), t1 has no guard, t3 has guard. + # t1 fires unconditionally on "start", t3 never fires. +``` + +**Generated domain errors:** + +``` +ERROR [domain.fsm.no_initial] No initial state defined. + Mark one state with is_initial: true (e.g. state "idle"). + +WARNING [domain.fsm.unreachable] State "orphan" is unreachable. + No transitions target it. Remove it or add a transition to it. + +WARNING [domain.fsm.self_loop] Transition "t2": self-loop on state + "running" with trigger "retry". This produces no state change. + If intentional, mark the transition with `is_self_loop: true`. + +WARNING [domain.fsm.guard_shadow] Transition "t3" is shadowed by "t1": + both fire from "idle" on trigger "start", but "t1" has no guard + (always matches), so "t3" (guard: is_admin == true) can never fire. + → Add a guard to "t1", remove "t3", or use different triggers. +``` + +### 3.4 Consistency Validation + +Checks that constraints and invariants are mutually satisfiable. + +**Constraint types in the SIR:** + +``` +ConstraintExpr = + | TypeConstraint(field, expectedType) + | ReferenceConstraint(source, target, relation) + | Invariant(temporalLogicExpr) // e.g. "eventually(terminal)" + | CrossConstraint(expr1, expr2, op) // expr1 AND/OR/IMPLIES expr2 + | GuardExpr(field, operator, value) // "counter < 5" +``` + +**Consistency checks (expensive):** + +For state machines with temporal logic invariants, we use bounded model checking: + +``` +Invariant: "G (in_state('error') → F in_state('recovery'))" + "Globally, if you enter error state, you eventually reach recovery" +``` + +If the model checker finds a counterexample trace where the SIR enters `error` but cannot reach `recovery`, it produces: + +``` +ERROR [consistency.invariant_violation] Invariant "G(error → F recovery)" + is violated. Counterexample trace (depth=4): + 1. idle → running (trigger: start) + 2. running → error (trigger: fail) + 3. error → (no outgoing transitions) + State "error" is a dead-end. Add a transition from "error" to "recovery" + or remove the invariant. +``` + +**Cheap consistency pass** (always runs first): +- Contradictory property assignments: `{ is_terminal: true, outgoing_transitions: [t1] }` +- Mutually exclusive type constraints: `x must be string AND x must be number` + +### 3.5 Confidence-Based Validation + +Each SIR element carries a confidence score (0.0–1.0) from the extraction layer. Low confidence means the extraction was uncertain about what the human meant. + +**Strategy:** + +``` +Element confidence thresholds: + ≥ 0.9 → trusted (no validation flags on confidence) + ≥ 0.7 → INFO: "Low confidence on element X. Verify: was this intended?" + ≥ 0.5 → WARNING: "Uncertain element X — compilation will include a TODO marker" + < 0.5 → ERROR: "Element X is too uncertain to compile. Clarify in conversation." +``` + +**Blocking logic:** + +```typescript +function confidenceBlocks(confidence: Confidence, threshold: number): boolean { + // Compilation is blocked if ANY error-severity element is below threshold + // OR if overall confidence is below the configured threshold. + return confidence.overall < threshold + || Array.from(confidence.perElement.values()) + .some(c => c < 0.5); // hard floor +} +``` + +**Example:** + +```yaml +elements: + - id: "idle" + type: "state" + properties: { name: "Idle", is_initial: true } + metadata: { confidence: 0.95 } + - id: "mystery" + type: "state" + properties: { name: "MaybeErrorHandling?" } + metadata: { confidence: 0.42 } # Very uncertain + +confidence: + perElement: { "idle": 0.95, "mystery": 0.42 } + overall: 0.71 + threshold: 0.8 +``` + +Generated: +``` +ERROR [confidence.below_floor] Element "mystery" (state "MaybeErrorHandling?") + has confidence 0.42 (< 0.5 floor). Clarify what this state represents. + +WARNING [confidence.overall] Overall SIR confidence (0.71) is below the + configured compilation threshold (0.80). Consider clarifying + uncertain elements before compiling. +``` + +--- + +## 4. Validation Timing + +### 4.1 Continuous Validation (on every extraction) + +Runs immediately after the Extraction Layer produces or updates the SIR. Uses only cheap checks: +- Structural validation +- Reference validation +- Confidence threshold checks +- Cheap domain checks (initial state exists, dead-end detection) + +**Goal:** Give the agent immediate feedback to interject in the conversation. + +> Agent: "I heard you describe a state machine with states 'idle', 'running', and 'error'. But 'error' has no way to recover. Would you like to add a recovery transition?" + +### 4.2 Pre-Compilation Validation (batched) + +Runs when the human requests compilation. Runs the full pipeline including expensive checks: +- Reachability analysis (BFS/DFS on state graph) +- Guard exhaustiveness +- Model checking (bounded, with configurable depth) +- SMT-based constraint solving (if invariants use arithmetic/logic) +- Completeness completeness analysis + +### 4.3 Incremental Validation + +When the extraction layer produces a delta (added/modified/removed elements), we re-validate only the affected subgraph: + +``` +Changed elements → find transitive closure of references → re-validate that subgraph +``` + +**Dependency tracking:** + +``` +Element "t1" depends on: ["idle", "running"] +Element "g1" (guard) depends on: ["t1"] +Invariant "inv1" depends on: ["error", "recovery"] +``` + +When `idle` changes, we re-check `t1` → re-check `g1`. When `error` changes, we re-check `inv1`. + +### 4.4 Lazy vs. Eager + +| Check Type | Timing | Rationale | +|------------|--------|-----------| +| Structural | Eager (continuous) | O(n), always cheap | +| Reference | Eager (continuous) | O(n), always cheap | +| Confidence | Eager (continuous) | O(n), always cheap | +| Reachability | Lazy (pre-compile) | O(V+E) — cheap but only needed before compile | +| Determinism | Lazy (pre-compile) | O(S×T) where S=states, T=transitions | +| Guard exhaustiveness | Lazy (pre-compile) | May require SAT/SMT | +| Model checking | Lazy (pre-compile) | PSPACE-complete for full LTL, bounded is O(|S|^k) | +| Invariant satisfiability | Lazy (pre-compile, opt-in) | Expensive SMT solving | + +The human can also explicitly request expensive checks via conversation: + +> Human: "Check if my state machine can deadlock." +> Agent: runs lazy checks on-demand, reports results. + +--- + +## 5. Error Representation + +### 5.1 ValidationReport Structure + +```typescript +interface ValidationReport { + // Overall verdict + status: "pass" | "pass_with_warnings" | "fail"; + + // All issues found + issues: ValidationIssue[]; + + // Statistics + stats: { + errors: number; + warnings: number; + infos: number; + checksRun: number; + checksSkipped: number; + durationMs: number; + }; + + // Completeness + completeness: CompletenessReport; + + // Suggestions for fixes + suggestions: FixSuggestion[]; + + // What blocked compilation (if status == "fail") + blockers: BlockingIssue[]; +} + +interface ValidationIssue { + id: string; // unique issue id + severity: "error" | "warning" | "info"; + category: ValidationCategory; // structural | domain | consistency | confidence | completeness + code: string; // machine-readable code e.g. "domain.fsm.no_initial" + message: string; // human-readable message + location: SourceLocation[]; // precisely which SIR elements + suggestion?: FixSuggestion; // optional suggested fix + trace?: Counterexample; // for model checking failures +} + +interface FixSuggestion { + description: string; // "Mark state 'idle' as initial" + action: "add" | "remove" | "modify" | "clarify"; + target: string; // element id + patch?: Partial; // concrete patch to apply + conversationPrompt?: string; // what the agent should say to the human +} + +interface CompletenessReport { + status: "complete" | "partial" | "incomplete"; + missingRequired: string[]; // element ids + missingOptional: string[]; // element ids + gaps: CompletenessGap[]; // described gaps +} + +interface BlockingIssue { + issueId: string; + reason: string; // why this blocks compilation +} + +type ValidationCategory = + | "structural" + | "domain" + | "consistency" + | "confidence" + | "completeness"; +``` + +### 5.2 Severity Semantics + +| Severity | Meaning | Compilation Behavior | +|----------|---------|---------------------| +| ERROR | Definite problem, SIR is invalid | Blocks compilation | +| WARNING | Likely problem, SIR is valid but suspect | Compiles with caveats annotated | +| INFO | Suggestion or note | Compiles normally, annotation optional | + +### 5.3 Error Localization + +Every issue points to precise SIR elements: + +```typescript +interface SourceLocation { + elementId: string; + propertyPath?: string; // e.g. "properties.to" for transition target + constraintId?: string; // for constraint violations + snippet?: string; // the problematic value +} +``` + +### 5.4 Suggested Fixes — Agent Conversation Prompts + +The fix suggestions are designed to flow back into conversation: + +``` +conversationPrompt: "State 'error' has no outgoing transitions. + Would you like to add a recovery transition, or mark it as terminal?" + +conversationPrompt: "I heard two different things about what happens when + the system starts: one where it goes to 'running' unconditionally, + and one where it only goes if 'is_admin' is true. Which one did you mean?" + +conversationPrompt: "You mentioned a 'loading' state but I'm not sure + when it should be entered. When does the system go to 'loading'?" +``` + +--- + +## 6. Constraint Checking System + +### 6.1 Constraint Expression Language + +Constraints in the SIR are structured expressions — never raw strings: + +```typescript +// Type constraint: "property 'trigger' must be a string" +{ + type: "type-check", + target: { elementId: "*", propertyPath: "trigger" }, + expected: "string" +} + +// Reference constraint: "transition.target must be a valid state id" +{ + type: "reference", + source: { elementId: "*", propertyPath: "to" }, + targetType: "state", + relation: "must-exist" +} + +// Invariant: "Globally, if in error, eventually in recovery" +{ + type: "invariant", + formula: { + op: "G_implies", // Globally, φ → ψ + left: { op: "in_state", state: "error" }, + right: { op: "F", formula: { op: "in_state", state: "recovery" } } + } +} + +// Cross-element: "every transition's 'from' must be a state" +{ + type: "for_each", + iterator: { elementType: "transition" }, + body: { + type: "reference", + source: { elementId: "$it", propertyPath: "from" }, + targetType: "state", + relation: "must-exist" + } +} +``` + +### 6.2 Constraint Evaluator + +```typescript +class ConstraintEvaluator { + evaluate(constraint: Constraint, sir: SIR): ValidationIssue[] { + switch (constraint.type) { + case "type-check": + return this.checkType(constraint, sir); + case "reference": + return this.checkReference(constraint, sir); + case "invariant": + return this.checkInvariant(constraint, sir); + case "for_each": + return this.checkForEach(constraint, sir); + } + } +} +``` + +**Type checking walks the element tree:** + +``` +For each element matching source pattern: + get property at propertyPath + if typeof(property.value) !== expected: + emit type_mismatch issue +``` + +**Reference checking resolves the graph:** + +``` +For each element matching source pattern: + get property at propertyPath (the referenced id) + if id not in sir.elements: + emit missing_reference issue + if sir.elements[id].type !== targetType: + emit type_mismatch issue ("expected state, got guard") +``` + +### 6.3 Invariant Checking Strategy + +For state machines, invariants are temporal logic formulas. We use **bounded model checking** (BMC): + +``` +BMC(invariant, SIR, depth_k): + Unroll state machine k steps + Build SAT formula: (initial_state ∧ transitions[0..k-1]) ∧ ¬invariant_unrolled + If SAT solver finds a model → counterexample exists → report trace + If UNSAT → no counterexample within bound k → report "passed up to depth k" +``` + +For arithmetic/logic guards, we use an SMT solver (Z3 integration): + +``` +Guard exhaustiveness check: + For state S with trigger T and guards g1, g2, ..., gn: + Ask SMT solver: is (g1 ∨ g2 ∨ ... ∨ gn) a tautology? + If no → report uncovered case with concrete counterexample values +``` + +### 6.4 Constraint Registration + +Domain validators register constraints at load time: + +```typescript +// state-machine-validator.ts +class StateMachineValidator implements DomainValidator { + domain = "state-machine"; + + registerConstraints(registry: ConstraintRegistry): void { + // Structural constraints (always run) + registry.add({ + id: "fsm.type.transition_target", + type: "for_each", + iterator: { elementType: "transition" }, + body: { + type: "reference", + source: { elementId: "$it", propertyPath: "to" }, + targetType: "state", + relation: "must-exist" + }, + severity: "error", + category: "domain" + }); + + // Invariant constraints (lazy, pre-compile) + registry.add({ + id: "fsm.invariant.no_deadlock", + type: "invariant", + formula: { + op: "G_exists_next", // Globally, every state has a next state + except: { property: "is_terminal", value: true } + }, + severity: "warning", + category: "domain", + timing: "lazy" + }); + } +} +``` + +--- + +## 7. Plugin Architecture for Domain Validators + +### 7.1 Plugin Interface + +```typescript +interface DomainValidator { + /** Which domain this validator handles */ + domain: string; + + /** Human-readable name */ + name: string; + + /** Register constraints with the engine */ + registerConstraints(registry: ConstraintRegistry): void; + + /** Run domain-specific checks that can't be expressed as constraints */ + customChecks(sir: SIR, options: CheckOptions): ValidationIssue[]; + + /** Declare which elements are required for completeness */ + requiredElements(): RequiredElementSpec[]; + + /** Declare which elements are optional */ + optionalElements(): OptionalElementSpec[]; +} + +interface CheckOptions { + timing: "continuous" | "pre-compile"; + maxDepth: number; // for BMC + timeoutMs: number; // for SMT +} +``` + +### 7.2 Plugin Registry + +```typescript +class ValidationEngine { + private validators: Map = new Map(); + private constraints: ConstraintRegistry = new ConstraintRegistry(); + + register(validator: DomainValidator): void { + this.validators.set(validator.domain, validator); + validator.registerConstraints(this.constraints); + } + + async validate(sir: SIR, options: ValidateOptions = {}): Promise { + const timing = options.timing ?? "continuous"; + const issues: ValidationIssue[] = []; + + // Pass 1: Structural (always, first) + issues.push(...this.runStructuralChecks(sir)); + + // Early exit: if structural fails, don't bother with domain checks + if (issues.some(i => i.severity === "error")) { + return this.assembleReport("fail", issues, sir); + } + + // Pass 2: Core constraint evaluation (filtered by timing) + issues.push(...this.constraints.evaluate(sir, timing)); + + // Pass 3: Domain-specific custom checks + const domainValidator = this.validators.get(sir.domain); + if (domainValidator) { + issues.push(...domainValidator.customChecks(sir, { timing, ...options })); + } + + // Pass 4: Consistency (lazy only) + if (timing === "pre-compile") { + issues.push(...this.runConsistencyChecks(sir)); + issues.push(...this.runCompletenessCheck(sir, domainValidator)); + } + + // Pass 5: Confidence filter + issues.push(...this.runConfidenceFilter(sir)); + + return this.assembleReport(issues, sir); + } +} +``` + +### 7.3 Registration at Startup + +```typescript +const engine = new ValidationEngine(); + +// Core validators (always loaded) +engine.register(new StateMachineValidator()); +engine.register(new WorkflowValidator()); +engine.register(new DataModelValidator()); +engine.register(new ApiSpecValidator()); + +// Custom validators (loaded from config) +for (const plugin of loadPlugins("./validators/")) { + engine.register(plugin); +} +``` + +--- + +## 8. Completeness Analysis + +### 8.1 What Makes a Specification "Complete Enough"? + +Different domains have different completeness thresholds. A validator declares: + +```typescript +interface RequiredElementSpec { + elementType: string; + minCount: number; // at least this many + maxCount?: number; // at most this many (if applicable) + description: string; // human-readable, for conversation +} + +interface OptionalElementSpec { + elementType: string; + description: string; + compiledAs: "stub" | "todo" | "omit"; // what to do when missing +} +``` + +**State Machine completeness spec:** + +```typescript +requiredElements(): RequiredElementSpec[] { + return [ + { elementType: "state", minCount: 2, + description: "At least 2 states (start and end)" }, + { elementType: "transition", minCount: 1, + description: "At least 1 transition" }, + ]; +} + +optionalElements(): OptionalElementSpec[] { + return [ + { elementType: "guard", description: "Transition guards", + compiledAs: "stub" }, // compile with guard: true (always pass) + { elementType: "action", description: "Entry/exit actions", + compiledAs: "stub" }, // compile with no-op actions + { elementType: "invariant", description: "Temporal invariants", + compiledAs: "omit" }, // skip invariant in compiled output + ]; +} +``` + +### 8.2 CompletenessReport Generation + +```typescript +function runCompletenessCheck( + sir: SIR, + validator: DomainValidator | undefined +): ValidationIssue[] { + const issues: ValidationIssue[] = []; + + if (!validator) { + // Unknown domain — can't check completeness + return [{ + severity: "warning", + category: "completeness", + code: "completeness.unknown_domain", + message: `No validator registered for domain "${sir.domain}". Cannot verify completeness.`, + }]; + } + + const required = validator.requiredElements(); + const optional = validator.optionalElements(); + + // Count elements by type + const counts = new Map(); + for (const el of sir.elements) { + counts.set(el.type, (counts.get(el.type) || 0) + 1); + } + + // Check required + for (const spec of required) { + const count = counts.get(spec.elementType) || 0; + if (count < spec.minCount) { + issues.push({ + severity: "error", + category: "completeness", + code: "completeness.missing_required", + message: `Missing ${spec.elementType}: need at least ${spec.minCount}, + found ${count}. ${spec.description}`, + suggestion: { + description: `Define ${spec.minCount - count} more ${spec.elementType}(s)`, + action: "add", + conversationPrompt: `You need at least ${spec.minCount - count} + more ${spec.elementType}(s). ${spec.description}. What should they be?`, + }, + }); + } + if (spec.maxCount !== undefined && count > spec.maxCount) { + issues.push({ + severity: "warning", + category: "completeness", + code: "completeness.too_many", + message: `Too many ${spec.elementType}s: max ${spec.maxCount}, found ${count}.`, + }); + } + } + + // Check optional — informational only + for (const spec of optional) { + const count = counts.get(spec.elementType) || 0; + if (count === 0) { + issues.push({ + severity: "info", + category: "completeness", + code: "completeness.missing_optional", + message: `No ${spec.elementType}s defined. ${spec.description}. + Will compile with ${spec.compiledAs}.`, + suggestion: { + description: `Optionally define ${spec.elementType}s`, + action: "add", + conversationPrompt: `You can add ${spec.elementType}s (${spec.description}) + but they're optional. Want to add any?`, + }, + }); + } + } + + return issues; +} +``` + +### 8.3 Graduated Completeness + +The compilation layer uses the completeness report to decide output quality: + +| SIR Completeness | Compiled Output | +|-----------------|-----------------| +| Full (all required + all optional) | Production artifact | +| Required-only (all required, some optional missing) | Functional artifact with stubs | +| Partial (some required missing) | Skeleton with TODO markers | +| Incomplete (many required missing) | Compilation refused | + +**Example — Partial state machine compiles to:** + +```python +# COMPILED FROM PARTIAL SPEC — TODO markers indicate gaps +class TrafficLightFSM: + def __init__(self): + self.state = "red" # initial state + + # TODO: Missing transition from "red" — need a trigger + # TODO: State "yellow" has no incoming transitions + # WARNING: State "green" has no outgoing transitions (may be terminal) + + def trigger_green_timer(self): # STUB — guard not specified + if self.state == "green": + self.state = "yellow" +``` + +--- + +## 9. Complete Example: Validation Pipeline on a State Machine + +### Input SIR (from extraction layer): + +```yaml +domain: state-machine +elements: + - id: "red" + type: "state" + properties: { name: "Red", is_initial: true } + metadata: { confidence: 0.95 } + + - id: "green" + type: "state" + properties: { name: "Green" } + metadata: { confidence: 0.90 } + + - id: "yellow" + type: "state" + properties: { name: "Yellow" } + metadata: { confidence: 0.45 } # LOW CONFIDENCE + + - id: "t1" + type: "transition" + properties: { from: "red", to: "green", trigger: "timer" } + metadata: { confidence: 0.92 } + + - id: "t2" + type: "transition" + properties: { from: "green", to: "red", trigger: "timer" } + metadata: { confidence: 0.50 } # UNCERTAIN + + - id: "t3" + type: "transition" + properties: { from: "red", to: "green", trigger: "timer", guard: "emergency == false" } + metadata: { confidence: 0.30 } # VERY LOW — probably contradictory + +constraints: + - id: "inv_no_red_forever" + type: "invariant" + expression: + op: "G_implies" + left: { op: "in_state", state: "red" } + right: { op: "F", formula: { op: "in_state", state: "green" } } + +confidence: + perElement: + red: 0.95 + green: 0.90 + yellow: 0.45 + t1: 0.92 + t2: 0.50 + t3: 0.30 + overall: 0.67 + threshold: 0.80 +``` + +### Validation Pass 1 — Structural: + +``` +✅ PASS: All elements have valid types, unique ids, correct property types. +``` + +### Validation Pass 2 — Reference: + +``` +✅ PASS: All transition from/to references resolve to existing states. +``` + +### Validation Pass 3 — Domain (continuous, cheap): + +``` +WARNING [domain.fsm.unreachable] State "yellow" is unreachable. + No transitions target it. + +WARNING [domain.fsm.guard_shadow] Transition "t3" is shadowed by "t1": + both fire from "red" on "timer", but "t1" has no guard (always matches), + so "t3" (guard: emergency == false) can never fire. +``` + +### Validation Pass 4 — Confidence: + +``` +ERROR [confidence.below_floor] Element "t3" has confidence 0.30 (< 0.5 floor). + Clarify: was "emergency == false" actually intended? + +ERROR [confidence.below_floor] Element "yellow" has confidence 0.45 (< 0.5 floor). + Clarify: is "Yellow" a real state in this machine? + +WARNING [confidence.below_threshold] Element "t2" has confidence 0.50. + "green → red on timer" is uncertain. Verify this transition. + +WARNING [confidence.overall] Overall SIR confidence (0.67) < threshold (0.80). + Consider clarifying uncertain elements before compiling. +``` + +### Validation Pass 5 — Consistency (pre-compile, lazy): + +``` +✅ PASS: Invariant "no_red_forever" holds up to bounded depth 10. + Transition t1 (red → green) satisfies the invariant. +``` + +### Validation Pass 6 — Completeness: + +``` +✅ PASS: Required elements present: + - states: 3 (min 2) ✓ + - transitions: 3 (min 1) ✓ + +INFO [completeness.missing_optional] No "action" elements defined. + Entry/exit actions are optional. Will compile with no-ops. + +INFO [completeness.missing_optional] No "guard" elements (as standalone + definitions) — guard on t3 is inline. Guards are optional. +``` + +### Final ValidationReport: + +```json +{ + "status": "fail", + "issues": [...all above...], + "stats": { + "errors": 2, + "warnings": 4, + "infos": 2, + "checksRun": 8, + "checksSkipped": 1, + "durationMs": 12 + }, + "completeness": { + "status": "partial", + "missingRequired": [], + "missingOptional": ["action", "guard"], + "gaps": [ + { "type": "low_confidence", "elements": ["t3", "yellow", "t2"] }, + { "type": "guard_shadow", "elements": ["t3"] } + ] + }, + "blockers": [ + { "issueId": "conf-b1", "reason": "Element t3 below confidence floor" }, + { "issueId": "conf-b2", "reason": "Element yellow below confidence floor" } + ], + "suggestions": [ + { + "description": "Remove t3 or add a guard to t1", + "action": "modify", + "target": "t3", + "conversationPrompt": "I see two transitions from 'red' to 'green' on 'timer' — one with a guard and one without. The unguarded one always matches, so the guarded one never fires. Which behavior do you want?" + }, + { + "description": "Clarify yellow state purpose", + "action": "clarify", + "target": "yellow", + "conversationPrompt": "I'm not sure about the 'yellow' state — it has no transitions to or from it. Is this a real state in your traffic light, or was it mentioned in passing?" + } + ] +} +``` + +### Agent Response in Conversation: + +> **I validated your state machine spec. Here's what I found:** +> +> **2 issues blocking compilation:** +> - The transition from red→green with guard `emergency==false` conflicts with the unguarded one — the guarded one can never fire. Which did you intend? +> - I'm not confident about the "yellow" state — it's disconnected from everything. Is this intentional? +> +> **4 warnings (won't block, but worth checking):** +> - Yellow has no transitions to it +> - The green→red transition is uncertain +> - Overall confidence is at 67% — below the 80% threshold +> +> **Missing optional things (no action needed):** +> - No entry/exit actions defined (will compile as no-ops) +> +> **Want me to fix the guard conflict, or do you want to clarify yellow first?** + +--- + +## 10. API Reference + +### `validate(sir: SIR, options?: ValidateOptions): ValidationReport` + +```typescript +interface ValidateOptions { + /** Validation timing mode */ + timing?: "continuous" | "pre-compile"; + + /** Max depth for bounded model checking (default: 10) */ + maxDepth?: number; + + /** Timeout for SMT solver (default: 5000ms) */ + solverTimeoutMs?: number; + + /** Skip expensive checks even in pre-compile mode */ + skipExpensive?: boolean; + + /** Only validate specific element ids (incremental) */ + scope?: string[]; +} +``` + +### `register(validator: DomainValidator): void` + +Register a domain validator plugin. Must be called before `validate()` for that domain. + +### `getReport(sirId: string): ValidationReport | null` + +Retrieve the last validation report for a given SIR (reports are cached). + +### `validateIncremental(sir: SIR, delta: SIRDelta): ValidationReport` + +Incremental validation. Only checks elements affected by the delta. + +--- + +## 11. Design Decisions Summary + +| Decision | Rationale | +|----------|-----------| +| Fail-cheap ordering | Avoids running SMT on malformed SIRs; structural checks are O(n) and catch 80% of issues | +| Plugin architecture | Domain validators are the extension point; core engine never knows about state machines | +| Constraint expression tree | Structured constraints are evaluable; raw strings ("trigger must be a string") require parsing | +| Confidence floor at 0.5 | Below 0.5, the extraction is essentially guessing; requiring clarification is better than wrong compilation | +| Continuous + pre-compile | Two-timing splits the workload: fast feedback in conversation, thorough checks before artifact generation | +| Bounded model checking | Full LTL model checking is PSPACE-complete; bounded is practical and catches most real-world bugs | +| Graduated completeness | Not all specs need to be "complete" — a skeleton with TODOs is often better than refusing to compile | +| Conversation-aware suggestions | Every error can produce a natural-language prompt; the validation engine feeds the conversation loop | + +--- + +## 12. Open Questions for Review + +1. **SMT solver dependency:** Z3 is ~30MB. Worth bundling, or should we make it optional and degrade gracefully (skip guard exhaustiveness, flag as INFO)? + +2. **Confidence aggregation:** Currently uses a simple floor. Should we use a weighted average (by element importance)? A critical transition with 0.49 confidence probably matters more than a comment element with 0.3. + +3. **Max depth for BMC:** Defaulting to 10 may miss deep counterexamples. Make it configurable per-domain? A state machine with 100 states might need depth 100. + +4. **Incremental validation edge case:** If element A is removed, and element B references A, we re-validate B. But what if B had already been flagged and the human acknowledged it? Should incremental validation suppress re-reporting of known issues? + +5. **Human-in-the-loop for expensive checks:** Should pre-compile validation ask before running expensive checks that might take >5 seconds? Or transparently run them with a timeout?