whitepaper v2: Section 3 - Compiler Architecture (~2,500 words)
Some checks failed
Deploy Landing Page to Storj / deploy (push) Failing after 2s
Some checks failed
Deploy Landing Page to Storj / deploy (push) Failing after 2s
Covers full compilation pipeline: parser (~850 lines), AST design, all 5 generators (PROV-O, CycloneDX, SPDX, TypeScript, WASM), WASM host runtime, test infrastructure (41 tests, 3 suites), and architectural design decisions (TypeScript throughout, zero external services, independent generators, text-first WASM, DIST DESYNC guard, CLI-first). Sections 1-3 complete; 4-8 remain.
This commit is contained in:
Scott Scoble
parent
66ecdd69b9
commit
fa896dec77
1 changed files with 193 additions and 2 deletions
|
|
@ -427,7 +427,198 @@ These omissions are the point. SpeckDL constrains the spec author to describe wh
|
|||
|
||||
## 3. Compiler Architecture
|
||||
|
||||
*[Section 3: Parser → AST → Generators pipeline, multi-target compilation, test infrastructure]*
|
||||
The Speckl compiler is a single-pass, multi-target compiler written in TypeScript. It takes a `.speck` file as input and produces five output artifacts through five independent generators. The compiler is designed to be deterministic, auditable, and self-contained — zero external services, zero API calls, zero runtime dependencies beyond Node.js and a WebAssembly toolchain (wabt) for binary WASM generation.
|
||||
|
||||
### 3.1 Compilation Pipeline
|
||||
|
||||
The compiler follows a classic front-end / back-end architecture:
|
||||
|
||||
```
|
||||
.speck file
|
||||
│
|
||||
▼
|
||||
┌──────────┐
|
||||
│ Parser │ ~850 lines — tokenizer + recursive descent parser
|
||||
└──────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────┐
|
||||
│ AST │ Typed nodes: Speck, State, Action, TypeExpr, Invariant, etc.
|
||||
└──────────┘
|
||||
│
|
||||
├──────────┬──────────┬──────────┬──────────┐
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
┌────────┐┌────────┐┌────────┐┌────────┐┌────────┐
|
||||
│PROV-O ││Cyclone ││ SPDX ││ TS ││ WASM │ ~1,800 lines of generators
|
||||
│Gen ││DX Gen ││Gen ││Gen ││Gen │
|
||||
└────────┘└────────┘└────────┘└────────┘└────────┘
|
||||
│ │ │ │ │
|
||||
▼ ▼ ▼ ▼ ▼
|
||||
provenance sbom.cdx sbom.spdx output.ts output.wat
|
||||
.ttl .json .json → output.wasm
|
||||
```
|
||||
|
||||
The compiler processes a single `.speck` file in one pass. The parser produces a unified AST, and each generator walks that AST independently to produce its target format. Generators do not communicate with each other — they share only the AST, ensuring that every output artifact is a faithful representation of the specification.
|
||||
|
||||
### 3.2 Parser
|
||||
|
||||
The parser (~850 lines) is a hand-written recursive descent parser with a tokenizer that handles:
|
||||
|
||||
- **Keywords:** `speck`, `state`, `init`, `action`, `invariant`, `type`, `next`, `forall`, `let`, `emit`, `return`, `require`, `if`, `and`, `or`, `not`, `implies`, `in`, `notIn`, `union`, `subsetOf`, `emptySet`, `emptyMap`, `List`
|
||||
- **Operators:** `:=`, `==`, `!=`, `<=`, `>=`, `+`, `-`, `*`, `/`, `\`, `|`
|
||||
- **Delimiters:** `{ }`, `( )`, `[ ]`, `:`, `.`, `,`, `;`
|
||||
- **Literals:** integers, booleans (`true`/`false`), strings (double-quoted), null
|
||||
- **Comments:** Line comments (`//`) and block comments (`/* */`)
|
||||
|
||||
The parser produces a typed AST with the following node types:
|
||||
|
||||
- **`SpeckNode`** — top-level specification, contains a name and ordered list of members
|
||||
- **`TypeNode`** — user-defined type aliases with structural type expressions
|
||||
- **`StateNode`** — state variable declarations with name-to-type mappings
|
||||
- **`InitNode`** — initial state assertions (equality expressions)
|
||||
- **`ActionNode`** — named action with optional parameters, preconditions, body statements, and return type
|
||||
- **`InvariantNode`** — named invariant with a boolean expression body
|
||||
- **`NextNode`** — action composition expression (`A | B | C`)
|
||||
|
||||
#### Type Expressions
|
||||
|
||||
The type system AST uses a recursive `TypeExpr` structure:
|
||||
|
||||
```
|
||||
type TypeExpr =
|
||||
| Primitive("Bool" | "Nat" | "Int" | "String")
|
||||
| Set(TypeExpr)
|
||||
| Map(TypeExpr, TypeExpr)
|
||||
| List(TypeExpr)
|
||||
| Record({ field: TypeExpr }...)
|
||||
| Optional(TypeExpr) // TypeExpr | null
|
||||
| Reference(name) // user-defined type alias
|
||||
```
|
||||
|
||||
Type expressions are resolved during parsing — type aliases are replaced with their structural definitions before code generation. This means the generators never see type aliases; they operate on fully resolved structural types.
|
||||
|
||||
### 3.3 Generators
|
||||
|
||||
Each generator is an independent module (~200-400 lines) that walks the AST and emits a specific target format. Generators share a common interface: they receive the parsed `SpeckNode` and an output directory, and they write their artifact to disk.
|
||||
|
||||
#### 3.3.1 PROV-O Generator (`provenance.ts`)
|
||||
|
||||
Emits a W3C PROV-O provenance graph in Turtle (`.ttl`) format. The graph captures:
|
||||
|
||||
- **Entities:** The `.speck` file itself (the specification as a document), each generated output file (TypeScript, WAT, SBOM, SPDX)
|
||||
- **Activities:** The compilation process as a whole, plus each generator invocation as a sub-activity
|
||||
- **Agents:** The Speckl compiler (identified by version and commit hash), the human author (from git config), and any AI agent that contributed to the spec
|
||||
- **Relations:** `wasGeneratedBy` (artifact → generator), `wasDerivedFrom` (code → spec block), `wasAttributedTo` (spec → author), `used` (generator → spec)
|
||||
|
||||
The PROV-O graph is the foundation of Speckl's audit trail. It answers the question: "Where did this code come from?" — and the answer is a machine-readable chain of provenance with cryptographic identifiers.
|
||||
|
||||
#### 3.3.2 CycloneDX Generator (`cyclonedx.ts`)
|
||||
|
||||
Emits a CycloneDX Software Bill of Materials in JSON format (v1.4). The SBOM declares:
|
||||
|
||||
- The generated module as a software component with name, version, and SHA-256 hash
|
||||
- The Speckl compiler as a tool used in the build process
|
||||
- The source `.speck` file as a component dependency
|
||||
- License information for both the generated code and the compiler
|
||||
|
||||
CycloneDX SBOMs are a de facto industry standard and are consumed by tools like Dependency-Track, OWASP scanners, and compliance platforms (SOC 2, FedRAMP).
|
||||
|
||||
#### 3.3.3 SPDX Generator (`spdx.ts`)
|
||||
|
||||
Emits an SPDX document in JSON format (v2.3). The SPDX document provides:
|
||||
|
||||
- Package-level license declaration for the generated artifact
|
||||
- Relationship to the source specification file
|
||||
- Creator information (tool and organization)
|
||||
|
||||
SPDX is the ISO standard (ISO/IEC 5962:2021) for license documentation and is required by many enterprise procurement processes.
|
||||
|
||||
#### 3.3.4 TypeScript Generator (`typescript-state-machine.ts`)
|
||||
|
||||
Emits a TypeScript class implementing the state machine described in the speck. The generated code includes:
|
||||
|
||||
- **Type aliases and interfaces:** All `type` declarations become TypeScript `type` aliases or `interface` definitions
|
||||
- **State interface:** The `state` block becomes a typed `State` interface
|
||||
- **Class with guard methods:** Each `action` becomes a public method on the class. Actions with `require` preconditions get compiled guard clauses that throw `PreconditionError` on failure
|
||||
- **Invariant checking:** An `_checkInvariants()` private method runs after every action. If any invariant fails, it throws `InvariantViolationError`
|
||||
- **Event emission:** `emit` statements become calls to an injected `EventEmitter` interface, allowing integration with logging, metrics, and audit systems without coupling to any specific infrastructure
|
||||
- **State immutability:** The internal state is private. External code can only read snapshots, never mutate state directly
|
||||
|
||||
Example TypeScript output (abbreviated):
|
||||
|
||||
```typescript
|
||||
class ToggleSwitch {
|
||||
private _state = { on: false };
|
||||
|
||||
TurnOn(): void {
|
||||
if (!this._state.on) throw new PreconditionError("TurnOn requires !on but on is true");
|
||||
this._state.on = true;
|
||||
this._checkInvariants();
|
||||
}
|
||||
|
||||
TurnOff(): void {
|
||||
if (this._state.on) throw new PreconditionError("TurnOff requires on but on is false");
|
||||
this._state.on = false;
|
||||
this._checkInvariants();
|
||||
}
|
||||
|
||||
private _checkInvariants(): void {
|
||||
// NoDoubleToggle: TurnOn => !on, TurnOff => on
|
||||
// Runtime invariant checks here
|
||||
}
|
||||
|
||||
getState(): Readonly<State> { return { ...this._state }; }
|
||||
}
|
||||
```
|
||||
|
||||
The TypeScript generator also handles `forall` loops, `if` statements, `let` bindings, and compound types (Set → `Set<T>`, Map → `Map<K,V>`, List → `T[]`).
|
||||
|
||||
#### 3.3.5 WASM Generator (`wasm.ts`)
|
||||
|
||||
Emits WebAssembly Text Format (`.wat`) that can be assembled to a `.wasm` binary module using the wabt toolchain (`wat2wasm`). The generated WASM module:
|
||||
|
||||
- **Exports one function per action** — each action becomes a public exported function
|
||||
- **Maintains state in linear memory** — state variables are laid out sequentially in WASM linear memory with type-specific encodings
|
||||
- **Implements guard checks** — `require` failures cause a WASM `unreachable` trap, which is the WASM equivalent of a panic
|
||||
- **Implements invariant checks** — invariants are validated after each action with the same trap-on-failure semantics
|
||||
- **Returns values on the stack** — `return` expressions push their value onto the WASM stack
|
||||
|
||||
The WASM target makes Speckl specs deployable in constrained environments: browser runtimes, edge functions, embedded systems, and any platform with a WASM runtime. It also enables formal verification at the bytecode level — the WAT output can be fed into WASM verifiers and symbolic execution engines.
|
||||
|
||||
#### 3.3.6 WASM Host Runtime (`wasm-host.ts`)
|
||||
|
||||
A lightweight JavaScript/TypeScript runtime for loading, instantiating, and interacting with generated WASM modules. Provides:
|
||||
|
||||
- `initWasm(buffer)` — instantiate a WASM module with initialized linear memory from `init`
|
||||
- `callAction(name, ...args)` — invoke an exported action by name
|
||||
- `getState()` — extract current state from linear memory as a typed object
|
||||
- `checkInvariants()` — manually re-run invariant checks (for debugging)
|
||||
|
||||
### 3.4 Test Infrastructure
|
||||
|
||||
The compiler has 41 tests across 3 test files, using Node.js's built-in test runner:
|
||||
|
||||
- **Parser tests:** Verify correct parsing of all language constructs — state declarations, type expressions, action bodies with require/let/if/emit/return/forall, invariants, quantified expressions, record types, nested generics, operator precedence
|
||||
- **Generator tests:** Verify output format correctness — PROV-O graph structure, CycloneDX schema compliance, SPDX field completeness, TypeScript type correctness (`tsc --noEmit`), WAT validity (wabt `wat2wasm`)
|
||||
- **End-to-end tests:** Full pipeline — `.speck` file → parse → generate all 5 artifacts → validate TypeScript compiles → validate WAT assembles → instantiate WASM → run actions → verify state changes and guard enforcement
|
||||
|
||||
All tests run in under 2 seconds on commodity hardware. The test suite is designed to make the compiler safe to refactor — any change that breaks spec semantics is caught by the test suite before it reaches users.
|
||||
|
||||
### 3.5 Design Decisions
|
||||
|
||||
The compiler architecture reflects several deliberate choices:
|
||||
|
||||
**TypeScript throughout.** The compiler, generators, and test suite are all TypeScript. This gives us structural typing for AST nodes, exhaustiveness checking on switch statements, and a single build pipeline (`tsc`). No language-switching between compiler components.
|
||||
|
||||
**Zero external services.** The compiler never makes network calls. It reads files from disk and writes files to disk. This makes it auditable (no side effects), reproducible (deterministic output given the same input), and air-gappable (usable in secure environments without internet access).
|
||||
|
||||
**Independent generators.** Each generator is a standalone module that takes an AST and produces output. Generators don't know about each other. Adding a new target (Rust, Python, C, SQL schema) means writing one new module — no changes to the parser or other generators.
|
||||
|
||||
**Text-first WASM pipeline.** The compiler emits `.wat` (text format) rather than binary `.wasm`. This makes the WASM output human-readable and grep-able. The final assembly step (`wat2wasm`) is deferred to the user or build system, keeping the compiler's dependency surface minimal.
|
||||
|
||||
**DIST DESYNC guard.** A `check-dist.sh` script + Makefile guard ensures compiled JavaScript (`dist/`) is never stale relative to TypeScript source (`src/`). This prevents the common bug where tests pass because they're running old compiled code. The compiler refuses to run if the dist is out of date.
|
||||
|
||||
**CLI-first, library-second.** The compiler is invoked as `speckl compile file.speck` with CLI flags for output directory, format selection, and verbosity. The internal API is stable but the primary interface is the command line, making it easy to integrate into CI/CD pipelines, Makefiles, and build scripts.
|
||||
|
||||
## 4. Embedded Provenance
|
||||
|
||||
|
|
@ -451,4 +642,4 @@ These omissions are the point. SpeckDL constrains the spec author to describe wh
|
|||
|
||||
---
|
||||
|
||||
*Whitepaper v2 draft. Sections 1 complete; 2-8 forthcoming.*
|
||||
*Whitepaper v2 draft. Sections 1-3 complete; 4-8 forthcoming.*
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue