speckl/docs/db-sqlite-reference.md

SQLite Pager, WAL, VDBE, and B-Tree Reference

Author: Marcus (agent-cos) cos@marcus.local Date: 2026-05-09 Purpose: Engineering source material for SpeckDL formal verification Target: SQLite version 3.53.1 (check-in c88b22011a54b, 2026-05-05)

---

1. Pager State Machine

The pager is SQLite's storage engine — manages page cache, transactions, and crash recovery. Located in src/pager.c (~12K lines). The pager owns the lock state, journal state, and page cache.

1.1 Pager State Enum (from pager.c header comment)

PAGER_UNLOCK       (0) — No lock held. Initial state.
PAGER_SHARED       (1) — Read lock (SQLITE_LOCK_SHARED). Read-only access.
PAGER_RESERVED     (2) — Reserved lock (SQLITE_LOCK_RESERVED). Preparing to write.
PAGER_EXCLUSIVE    (3) — Exclusive lock (SQLITE_LOCK_EXCLUSIVE). Actively writing.
PAGER_SYNCED       (4) — All dirty pages written + synced. Journal can be deleted.
PAGER_ERROR        (5) — Error occurred. No further operations allowed.
PAGER_OPEN         (6) — Special state: connection open but no lock. Transitional.

1.2 State Transition Table

From	To	Trigger / Guard	Side Effects
PAGER_OPEN	PAGER_SHARED	First pager_acquire() / pager_begin()	Obtain SQLITE_LOCK_SHARED
PAGER_UNLOCK	PAGER_SHARED	sqlite3PagerSharedLock()	Obtain shared lock, recover hot journal if present
PAGER_SHARED	PAGER_UNLOCK	pager_unlock() / end of read tx	Release shared lock
PAGER_SHARED	PAGER_RESERVED	pager_write() or pager_begin()	Obtain SQLITE_LOCK_RESERVED. No other writer can proceed.
PAGER_RESERVED	PAGER_EXCLUSIVE	First page modification (in PAGER_RESERVED)	-
PAGER_RESERVED	PAGER_EXCLUSIVE	Commit begins	Obtain PENDING then EXCLUSIVE lock
PAGER_RESERVED	PAGER_SHARED	ROLLBACK	Discard changes, release reserved lock
PAGER_EXCLUSIVE	PAGER_SYNCED	All dirty pages written + fsynced	Journal header updated to sync state
PAGER_SYNCED	PAGER_SHARED (journal-mode) or PAGER_UNLOCK (WAL)	Delete journal file (COMMIT point)	Transaction atomically committed
PAGER_EXCLUSIVE	PAGER_ERROR	I/O error, disk full, etc.	errCode set, hot journal preserved
PAGER_SYNCED	PAGER_ERROR	I/O error during sync	errCode set, journal preserved for recovery
any →	PAGER_ERROR	Any I/O error during pager operation	Further ops blocked
PAGER_ERROR	PAGER_OPEN	pager_unlock() forced close	Resets connection but keeps db open

1.3 Journal Mode Variants

Three rollback journal modes (non-WAL):

• DELETE (default): Journal file deleted on commit. If crash before delete → hot journal → recovery rolls back.
• TRUNCATE: Journal file truncated to zero length on commit. Faster on some filesystems than delete.
• PERSIST: Journal header zeroed on commit, file left on disk. Slightly faster, but leaves journal files around.

The pager tracks journalMode and adjusts the commit-delete step accordingly.

1.4 Pager Invariants (to formally specify)

• I1 (Lock State Consistency): eState must be consistent with actual file lock held. If eState==PAGER_SHARED, must hold SQLITE_LOCK_SHARED. If eState==PAGER_EXCLUSIVE, must hold SQLITE_LOCK_EXCLUSIVE.
• I2 (Dirty Page Tracking): For any page in the cache, isDirty iff the page has been modified since last checkpoint/journal write.
• I3 (Journal Existence): The journal file exists iff eState >= PAGER_RESERVED and eState != PAGER_ERROR (except during recovery).
• I4 (Hot Journal Detection): A journal is "hot" iff it exists but the database was not cleanly closed. On first shared lock, hot journals must be rolled back.
• I5 (Error Propagation): Once eState == PAGER_ERROR, all subsequent pager operations must return the stored errCode.
• I6 (Read-Only Pager): If readOnly flag is set, eState must never exceed PAGER_SHARED.
• I7 (Sync Before Commit): No dirty page is persisted to the database file until after the journal is synced to disk AND the exclusive lock is held.
• I8 (Commit Atomicity): The commit point is the deletion of the journal file. Before this: crash → rollback. After this: crash → committed state visible.

1.5 Lock Escalation Sequence (from atomiccommit.html)

NO_LOCK → SHARED (read) → RESERVED (intent to write) → PENDING (block new readers)
    → EXCLUSIVE (writing) → no lock (after commit journal deletion)

Key subtlety: PENDING lock allows existing readers to finish but blocks new readers. This prevents writer starvation in the face of constant new read connections.

---

2. WAL Mode State Machine

WAL mode inverts the rollback journal: original data stays in the database file, changes are appended to the WAL file. Commit = appending a commit record to WAL.

2.1 WAL Files

File	Purpose	Lifecycle
X-wal	Write-ahead log. Sequential append of frames.	Exists while connections are open. Deleted on clean close.
X-shm	Shared memory (wal-index). Hash table for frame location.	mmap'd. Not fsynced. Omitted under EXCLUSIVE locking mode.
X	Main database file. Contains original + checkpointed pages.	Persistent.

2.2 WAL-Index Header Fields (from walformat.html)

Field	Bytes	Description
iVersion	0-3	WAL-index format version (3007000)
iChange	8-11	Transaction counter, incremented per tx
isInit	12	1 when shm file initialized
bigEndCksum	13	Checksum endianness flag
szPage	14-15	Page size (or 1 for 65536)
mxFrame	16-19	Number of valid commit frames in WAL
nPage	20-23	Database size in pages
aFrameCksum	24-31	Checksum of last frame
aSalt	32-39	Two salt values from WAL header
nBackfill	96-99	Frames already backfilled by checkpoints
read-mark[0..4]	100-119	Five reader end marks
nBackfillAttempted	128-132	Frames attempted but not confirmed backfilled

2.3 WAL Locks

Lock	Byte	Purpose
WAL_WRITE_LOCK	120	Exclusive: appending frames to WAL
WAL_CKPT_LOCK	121	Exclusive: running checkpoint
WAL_RECOVER_LOCK	122	Exclusive: running recovery
WAL_READ_LOCK(0)	123	Reader end-marks (0-4)
WAL_READ_LOCK(1)	124
WAL_READ_LOCK(2)	125
WAL_READ_LOCK(3)	126
WAL_READ_LOCK(4)	127

2.4 WAL Frame Lifecycle

Frame format (in WAL file): [page-number] [db-size-after-commit] [salt-1] [salt-2] [checksum-1] [checksum-2]
  followed by page data.

Frame lifecycle states:
  1. WRITER appends frames to end of WAL file (holding WAL_WRITE_LOCK exclusive)
  2. Last frame of a transaction marks "commit" via non-zero db-size field
  3. READERS snapshot mxFrame at transaction start → their "end mark"
  4. CHECKPOINTER copies frames back to database (holding WAL_CKPT_LOCK), increments nBackfill
  5. RESET: When nBackfill == mxFrame and no reader holds WAL_READ_LOCK(N) for N>0,
     writer rewinds WAL to beginning (resets mxFrame, nBackfill to 0)

2.5 Reader Snapshot (End Mark) Protocol

• When a read transaction begins, the reader notes current mxFrame value → this is its "end mark"
• The reader acquires one of WAL_READ_LOCK(0..4) to declare its end mark
• To read page P, the reader calls FindFrame(P, end_mark): find the latest frame for page P in WAL with frame number ≤ end_mark
• If found: read from WAL frame. If not found: read from database file.
• Readers hold WAL_READ_LOCK(N) for the duration of their read transaction
• This prevents checkpointer from overwriting pages the reader still needs

2.6 Checkpoint States

Checkpoint Mode	Behavior
PASSIVE	Non-blocking. Works around readers. May not complete. Default.
FULL	Blocks until checkpoint completes or a writer appears. More aggressive.
RESTART	Full checkpoint then blocks until no readers. Ensures WAL reset.
TRUNCATE	Like RESTART, also truncates WAL file to zero bytes.

Checkpoint progress constraint: MUST STOP at the minimum end_mark across all active readers.

• nBackfill ≤ min(read_mark[0..4])
• nBackfillAttempted ≥ nBackfill (may be ahead if reader blocked a backfill)

2.7 WAL Reset Condition

The WAL is reset (wound back to start) when:

1. nBackfill == mxFrame (all frames backfilled)
2. No reader holds WAL_READ_LOCK(N) for any N > 0 (no readers using WAL)
3. Writer holds WAL_WRITE_LOCK exclusively
4. On reset: mxFrame = 0, nBackfill = 0, file position rewound

2.8 WAL Invariants (to formally specify)

• W1 (mxFrame Monotonic): mxFrame is non-decreasing except during WAL reset.
• W2 (nBackfill Bound): nBackfill ≤ mxFrame always.
• W3 (Backfill ≤ Readers): nBackfillAttempted ≤ min(read_mark[0..4]) for any checkpoint that respects readers.
• W4 (Frame Validity): All frames 1..mxFrame have valid checksums. Frames > mxFrame are undefined.
• W5 (Commit Detection): Frame k is a commit frame iff its db-size-after-commit field ≠ 0.
• W6 (Read Consistency): A reader with end-mark M sees exactly the database state as of the last commit frame ≤ M.
• W7 (WAL Write Serialization): Only one connection may append to WAL at a time (WAL_WRITE_LOCK exclusive).
• W8 (Checkpoint Isolation): Checkpoint must not overwrite pages needed by any active reader (nBackfill ≤ min(read_marks)).
• W9 (Salt Match): WAL file header salt values must match aSalt in the WAL-index header. Mismatch → WAL is stale → must recover.
• W10 (WAL Reset Safety): WAL may only be reset when nBackfill == mxFrame AND no reader locks exist.

---

3. VDBE Instruction Set (Key Opcodes)

The VDBE is SQLite's bytecode virtual machine. Each prepared statement is a bytecode program. Source: vdbe.c (~8K lines). 192 opcodes in version 3.53.1.

3.1 Instruction Format

Each instruction: [opcode] P1 P2 P3 P4 P5
  P1, P2, P3: 32-bit signed integers (registers, cursor numbers, jump targets)
  P4: various (int, string, function pointer, collation)
  P5: 16-bit unsigned flags

3.2 Transaction & Locking Opcodes

Opcode	P1	P2	P4	Effect
Transaction	iDb	write_flag		Begin a transaction on database P1. P2=1 for write, 0 for read. Opens pager transaction.
AutoCommit	autocommit_flag	rollback_flag		Set auto-commit on/off. If P2 set, rollback first. Causes VM halt.
Checkpoint	iDb	mode		Run checkpoint on database P1. P2 = PASSIVE(0)/FULL(1)/RESTART(2)/TRUNCATE(3).
Savepoint	start_flag	savepoint_name		Begin/Release/Rollback a savepoint.
TableLock	iDb	root_page	table_name	Lock a table before modifying. write_flag in P3.
SchemaLock	iDb		schema_name	Lock schema objects.

3.3 Cursor Opcodes

Opcode	P1	P2	P3	P4	Effect
OpenRead	cursor	root_page	iDb		Open cursor P1 for read on root page P2 of db P3
OpenWrite	cursor	root_page	iDb	table_name	Open cursor P1 for write on root page P2 of db P3
OpenAutoindex	cursor	key_cols	nCol		Open ephemeral auto-index cursor
OpenEphemeral	cursor	nCol			Open temporary table, auto-clears on close
Close	cursor				Close cursor P1
SeekGT	cursor	jump_target		key_reg	Seek cursor P1 to first entry > key in register
SeekGE	cursor	jump_target		key_reg	Seek cursor P1 to first entry >= key in register
SeekLT	cursor	jump_target		key_reg	Seek cursor P1 to first entry < key in register
SeekLE	cursor	jump_target		key_reg	Seek cursor P1 to first entry <= key in register
SeekRowid	cursor				Seek cursor P1 to entry with given rowid
NotFound	cursor	jump_target			Jump if key not found in index
Found	cursor	jump_target			Jump if key found in index
Next	cursor	jump_target			Advance cursor P1 to next entry. Jump to P2 if no more rows.
Prev	cursor	jump_target			Move cursor P1 to previous entry. Jump to P2 if at beginning.
Last	cursor				Move cursor P1 to last entry.
Rewind	cursor	jump_target			Move to first entry. Jump to P2 if empty.
Column	cursor	column_idx	dest_reg		Read column P2 from cursor P1 into register P3
Rowid	cursor	dest_reg			Write rowid of cursor P1 into register P2
IdxInsert	cursor			key	Insert entry into index cursor
IdxDelete	cursor			key	Delete entry from index cursor
Insert	cursor				Insert current record into table cursor
Delete	cursor				Delete current record from table cursor
SorterSort	cursor				Sort sorter cursor P1
SorterData	cursor	dest_reg			Read current sorter row

3.4 Value & Register Opcodes

Opcode	P1	P2	P3	Effect
Integer	value	dest		Store integer value in register P2 (P1 is the value)
Real	value (P4)	dest		Store real value in register P2
String8		dest	string	Store UTF-8 string in register P2. Self-modifying → becomes String opcode.
Null	start	count		Set P2 registers starting at P1 to NULL
Blob	size	dest	data	Store P1-byte blob in register P2
Copy	src	dest	count	Copy P3 registers from P1 to P2
Move	src	dest	count	Move P3 registers from P1 to P2 (clear source)
SCopy	src	dest		Soft copy (shallow) register P1 to P2
ResultRow	reg_start	nReg		Pause VM: emit result row from registers P1..P1+P2-1. sqlite3_step() returns SQLITE_ROW.
Yield	pc_store			Coroutine yield. Swap PC with value in register P1.
Halt	p1	p2	p3	Halt execution. If P1==SQLITE_OK and P2==OE_Abort, auto-rollback.

3.5 Arithmetic & Comparison Opcodes

Opcode	P1	P2	P3	P4	Effect
Add	r1	r2	result		r[P3] = r[P1] + r[P2]
Subtract	r1	r2	result		r[P3] = r[P1] - r[P2]
Multiply	r1	r2	result		r[P3] = r[P1] * r[P2]
Divide	r1	r2	result		r[P3] = r[P1] / r[P2]
Remainder	r1	r2	result		r[P3] = r[P1] % r[P2]
Eq	r1	r2	jump	coll	Compare r[P1] and r[P2]. Jump to P3 if equal.
Ne	r1	r2	jump	coll	Jump if not equal
Lt	r1	r2	jump	coll	Jump if r[P1] < r[P2]
Le	r1	r2	jump	coll	Jump if r[P1] ≤ r[P2]
Gt	r1	r2	jump	coll	Jump if r[P1] > r[P2]
Ge	r1	r2	jump	coll	Jump if r[P1] ≥ r[P2]
BitAnd	r1	r2	result		r[P3] = r[P1] & r[P2]
BitOr	r1	r2	result		r[P3] = r[P1] | r[P2]
BitNot	src	dest			r[P2] = ~r[P1]
Concat	r1	r2	result		r[P3] = r[P1] || r[P2]

3.6 Control Flow Opcodes

Opcode	P1	P2	Effect
Goto		target	Unconditional jump to P2
Gosub	return_reg	target	Push PC to register P1, jump to P2
Return	return_reg		Jump to address stored in P1 + 1
If	jump_target	dest	Conditional jump
IfNot	jump_target	dest	Conditional jump (negated)
IfNull	reg	jump_target	Jump if register contains NULL
NotNull	reg	jump_target	Jump if register is not NULL
IfPos	reg	jump_target	Jump if register > 0
IfNeg	reg	jump_target	Jump if register < 0
Once	flag		NOP if flag was set. Execute body exactly once.
Init	flags	max_addr	First opcode. Sets trace flags. Self-modifying: increments P1 each invocation.
Program		sub_program_addr	Invoke a subprogram (trigger).
Param	param_idx	dest_reg	Read parameter from caller into destination register.

3.7 Aggregate Function Opcodes

Opcode	P1	P2	P3	Effect
AggStep		arg_reg	accum	Execute xStep for aggregate function
AggStep1	step/inverse	arg_reg	accum	Execute xStep (P1=0) or xInverse (P1!=0)
AggInverse		arg_reg	accum	Execute xInverse (window functions)
AggFinal			accum	Finalize aggregate, store result
AggValue			result	Store current aggregate value

3.8 VDBE Invariants (to formally specify)

• V1 (PC Bounds): Program counter must remain within [0, nOp] during execution.
• V2 (Register Type Safety): Each register holds exactly one type at a time (NULL, int, real, text, blob, RowSet, Frame, Undefined).
• V3 (Cursor Validity): An opcode must not reference a cursor unless it was previously opened (OpenRead/OpenWrite/OpenEphemeral) and not yet closed.
• V4 (Transaction Guard): Before any write opcode executes, a transaction must be open. Read opcodes may run outside a transaction but must obtain shared lock.
• V5 (ResultRow Protocol): ResultRow must be preceded by opcodes that fill the output registers. sqlite3_step() returns SQLITE_ROW; subsequent call resumes after ResultRow.
• V6 (Halt Termination): Execution terminates only via Halt opcode, PC exceeding last opcode address, or error.
• V7 (Error Cleanup): On error or HALT with error, all open cursors are closed, memory freed, and active transactions rolled back.
• V8 (Subprogram Isolation): Each Program invocation gets its own register set. Params are the only communication channel to the caller.
• V9 (Coroutine Symmetry): Yield stores PC to register; subsequent Yield reads register to PC. Coroutine execution must return to same point.

---

4. B-Tree Invariants

SQLite uses B+tree for tables, B-tree for indices. Source: btree.c.

4.1 Page Types

Type	Value	Description
Interior index	0x02	Index b-tree interior page
Interior table	0x05	Table b-tree interior page
Leaf index	0x0a	Index b-tree leaf page
Leaf table	0x0d	Table b-tree leaf page

4.2 Page Layout

Page header:
  [page_type:1] [first_freeblock:2] [cell_count:2] [cell_content_start:2] [fragmented_free_bytes:1]
  
Followed by:
  [right_child_pointer:4] (interior pages only)
  [cell_pointer_array: cell_count * 2 bytes] — offsets to cells, in key order
  [unallocated space in middle]
  [cells: packed from end of page backwards]

4.3 B-Tree Cell Format

Table leaf cell:

[payload_size: varint] [rowid: varint] [payload: header + columns]

Table interior cell:

[left_child_page:4] [rowid: varint] [payload: partial row data]

Index leaf cell:

[payload_size: varint] [payload: header + index columns + rowid]

Index interior cell:

[left_child_page:4] [payload_size: varint] [payload: header + index columns + rowid]

4.4 Key Ordering Properties

• Table B-Tree: Ordered by rowid (integer primary key). Interior cells store the rowid of the first entry in each child page.
• Index B-Tree: Ordered by indexed column(s) + rowid. Interior cells store the key of the first entry in each child page.
• All keys within a page are in ascending order.
• For any interior page: all keys in child subtree i are ≤ the key at position i in the interior page, and ≤ all keys in child subtree i+1. (For indices, strict < for key columns, ≤ for rowid as tiebreaker.)

4.5 Page Split Condition

A page splits when inserting a cell would exceed the usable page space.

1. A new page is allocated.
2. Cells are distributed (roughly half each) between old and new page.
3. Parent interior page gets a new cell pointing to the new page.

4.6 Page Merge Condition

When deleting, if a page's content falls below the minimum fill threshold (roughly 25-50% depending on implementation), the page may be merged with a sibling:

1. Cells are moved to sibling page.
2. Parent pointer to the empty page is removed.
3. Empty page is added to freelist.

4.7 B-Tree Cursor Operations

Cursor state: { Btree*, root page, current page, cell index, cache of page stack }

Operations:
  sqlite3BtreeCursor(): Create cursor on B-tree identified by root page + flags
  sqlite3BtreeMoveto(): Position cursor to entry that matches key (or nearest)
  sqlite3BtreeNext(): Advance cursor. Page crossing handled automatically.
  sqlite3BtreePrevious(): Move cursor backwards. Page crossing handled.
  sqlite3BtreeInsert(): Insert cell into current position
  sqlite3BtreeDelete(): Delete entry at cursor position

4.8 B-Tree Invariants (to formally specify)

• B1 (Page Type Consistency): A page's type field must match its role. Interior pages reference child pages; leaf pages do not.
• B2 (Key Order Monotonic): For any interior page, keys in child page i are less than or equal to keys in child page i+1. Cells within a page are sorted ascending.
• B3 (Pointer Integrity): Every child page reference must point to a valid allocated page of the correct type (or overflow page, or freelist page).
• B4 (Cell Count Consistency): cell_count in the page header must equal the number of entries in the cell pointer array.
• B5 (Free Space Accounting): fragmented_free_bytes + unallocated space between cell pointer array and cells must sum correctly to the page's available free space.
• B6 (Page Size Boundary): Usable page size = page_size - reserved_bytes. All cells must fit within the usable page.
• B7 (Root Page Integrity): The b-tree root page number must be a valid, allocated b-tree page. The schema table's root page mappings must be consistent.
• B8 (Balance Preserved): After every insert/delete operation, the b-tree must satisfy all internal structural invariants (fill ratio, parent-child pointer correctness, key ordering).
• B9 (Overflow Chain Integrity): If a cell's payload spills to overflow pages, the chain of overflow pages must: (a) form a proper linked list, (b) be bounded in length by the payload size, (c) contain no cycles.
• B10 (Freelist Consistency): Freelist pages must not be reachable from any b-tree root. Freelist trunk pages must point to valid leaf pages or 0.

---

5. Known Bugs and Formal Verification Precedents

5.1 Notable SQLite CVEs

CVE	Year	Type	Module	Fixed In	Severity
CVE-2022-35737	2022	Integer overflow → buffer overrun → RCE/DoS	printf/str_vappendf	3.39.2	High (7.5)
CVE-2025-3277	2025	Write past end of malloc'd array via integer overflow	concat_ws() function	3.49.1	Moderate
CVE-2025-6965	2025	Integer overflow → array bounds read	SQL injection path	3.50.2	Moderate
CVE-2024-0232	2024	Use-after-free in JSON parser	JSON	3.43.2	Moderate
CVE-2023-7104	2023	Corruption in session extension changeset processing	Session (ext)	3.43.1	Low
CVE-2025-29088	2025	Crash via bad args to sqlite3_db_config	Config API	3.49.1	Low

Key observation from SQLite's CVE page:

Almost all CVEs written against SQLite require the ability to inject and run arbitrary SQL. The core storage layer (pager, WAL, B-tree) has remarkably few logic bugs. Most CVEs are in SQL-level functions, extensions, or edge cases in the printf/logging subsystem.

5.2 22-Year-Old Integer Overflow (CVE-2022-35737)

Introduced October 2000 (version 1.0.12), discovered 2022 by Trail of Bits. Exploitable on 64-bit systems when large strings (>2GB) passed to printf functions with ! unicode flag. Arbitrary code execution possible without stack canaries. Not a pager/journal bug — but illustrates how old, untested edge cases survive.

5.3 Formal Verification Precedents

5.3.1 FoundationDB (Apple)

• What: Distributed transactional key-value store, formally verified in Flow (their deterministic simulation language)
• Method: Deterministic simulation testing — run all DB processes as fibers on a single thread with simulated network/disk, inject faults, check invariants. Run millions of random seeds nightly.
• Result: Found ~70 bugs before release. Flow-based simulation verified correctness of their distributed transaction, replication, and recovery protocols.
• Relevance to SQLite: The determinism approach parallels what could be done for SQLite's pager. SQLite is single-process by design, making deterministic replay easier.

5.3.2 TigerBeetle

• What: Distributed financial accounting database. VOPR (Viewstamped Operation Replicator) deterministic simulator.
• Method: Entire cluster runs on single thread. 1000x speed. Injects network, storage, and process faults. Swarm testing + fuzzer. Continuous verification.
• Relevance to SQLite: The VOPR approach of running the real implementation under controlled simulation is directly applicable to testing SQLite's recovery paths in the pager.

5.3.3 Amazon (DynamoDB, S3)

• What: TLA+ specifications for AWS distributed services
• Method: Model checking (TLC) + manual proofs (TLAPS). Found bugs in replication protocols, quorum membership, leader election.
• Paper: "Use of Formal Methods at Amazon Web Services" (Newcombe et al., 2014)
• Relevance to SQLite: TLA+ is ideal for specifying state machines like the pager's locking protocol. The pager state machine is small enough for tractable TLC model checking.

5.3.4 MongoDB Transactions (VLDB 2025)

• What: "Design and Modular Verification of Distributed Transactions in MongoDB"
• Method: TLA+ specification + model checking + test-case generation.
• Relevance to SQLite: Demonstrates that even complex transactional protocols can be verified with TLA+.

5.3.5 Formal Land's Rocq Translation of SQLite B-Tree (2025-2026)

• What: AI-assisted translation of SQLite C code to Rocq (Coq variant). Translation only (not full proof of equivalence).
• Result: "Still a big project to run with current technologies." Covered substantial part of B-tree but deemed too unstable for full verification at current AI capability levels.
• Relevance: Direct precedent that SQLite formal verification is hard. Suggests model-level verification (state machine specs) is more tractable than code-level proofs.

5.3.6 Richard Hipp's Own Attempt (SQLite Lead)

• Dr. Hipp reports spending "years" trying to write a formal design specification for SQLite but found it never added value and abandoned it. This is telling — both about the difficulty and about the value proposition.

5.4 SQLite's Existing Testing (Non-Formal)

Method	Scale	Notes
TH3 (Test Harness #3)	Proprietary. Billions of test cases.	100% branch coverage. Mutation testing (inject bugs, verify they're caught).
OSS-Fuzz	Continuous.	Google's fuzzing infrastructure. Catches memory safety bugs.
Release testing	~100M tests per release	Run on multiple platforms
Anomaly testing	Random SQL/DB operations with crash injection	Verify recovery

---

6. Target Invariants for SpeckDL Specification

Ranked by impact (what would be most impressive/valuable to prove):

Tier 1: Impact/Feasibility Sweet Spot

1. Pager Commit Atomicity (I8)

   • Prove: For any sequence of pager operations and any crash point, the database file is always in either the pre-commit state (all changes rolled back) or the post-commit state (all changes applied). No partial-commit states.
   • Value: This is the fundamental SQLite guarantee. Proving it formally would be the headline achievement.
   • Feasibility: Medium. The pager state machine has only 5-6 states. The tricky part is modeling disk I/O reordering and crash-at-any-point semantics.

2. WAL Read Isolation (W6)

   • Prove: A reader with end-mark M sees exactly the database state as of the last commit ≤ M. Writes after M are invisible.
   • Value: WAL mode's key property. With formal proof, SQLite's WAL concurrency model gets a rigorous foundation.
   • Feasibility: Medium. Need to model the WAL-index hash tables and the FindFrame algorithm.

3. Journal Recovery Correctness

   • Prove: After a crash with a hot journal, recovery restores the database to its state before the incomplete transaction.
   • Value: Crash recovery is the hardest case to test exhaustively. Formal proof would eliminate a whole class of concerns.
   • Feasibility: High. Well-scoped problem with clear success criteria.

Tier 2: High Value, Higher Difficulty

4. B-Tree Balance Preservation (B8)

   • Prove: After any sequence of insertions and deletions, the b-tree satisfies all structural invariants (page fill ratios, key ordering, pointer integrity, freelist consistency).
   • Value: B-tree corruption is one of the most feared failure modes. Formal proof is the gold standard.
   • Feasibility: Low-Medium. The B-tree is complex (~200KB of C code). A full proof is likely too big, but the core balancing algorithm could be specified and verified in isolation.

5. Lock Protocol Deadlock Freedom

   • Prove: The lock escalation protocol (shared → reserved → pending → exclusive) is deadlock-free for any number of concurrent readers and writers.
   • Value: Prevents a potential operational nightmare.
   • Feasibility: High. Small state space. Ideal for TLC model checking.

6. WAL Checkpoint Safety (W8)

   • Prove: Checkpoint never overwrites pages needed by active readers, even when readers and checkpointer operate concurrently.
   • Value: WAL's concurrent reader+writer+checkpointer model is nontrivial. Getting it right formally is impressive.
   • Feasibility: Medium. Requires modeling concurrent processes and the WAL-index header fields.

Tier 3: Ambitious / Future Work

7. Full Pager FSM Correctness (I1-I8)

   • Prove all eight pager invariants from Section 1.4 simultaneously.
   • Feasibility: Low (large specification, many interacting components). Best done incrementally.

8. VDBE Safety: No undefined register reads (V2)

   • Prove that the code generator never produces bytecode that reads an Undefined register.
   • Feasibility: Low. Requires modeling the SQL compiler's code generation, which is a much larger problem.

Recommendation: Start with Tier 1 (Items 1-3)

The pager commit atomicity + WAL read isolation + journal recovery form a coherent package:

• All three involve the pager module
• All three address crash-safety guarantees
• Together, they prove that "SQLite transactions are atomic even under arbitrary crash timing" — the single claim that matters most
• The state spaces are tractable for SpeckDL + model checking

---

7. Source References

Source	Location
SQLite WAL documentation	https://www.sqlite.org/wal.html
SQLite Atomic Commit	https://www.sqlite.org/atomiccommit.html
SQLite Bytecode Engine	https://www.sqlite.org/opcode.html (192 opcodes, v3.53.1)
Database File Format	https://www.sqlite.org/fileformat2.html
WAL File Format	https://www.sqlite.org/walformat.html
Pager source	src/pager.c (~12K lines). States: PAGER_UNLOCK, SHARED, RESERVED, EXCLUSIVE, SYNCED, ERROR, OPEN
VDBE source	src/vdbe.c (~8K lines) + helper files
B-Tree source	src/btree.c
SQLite CVEs	https://sqlite.org/cves.html
Formal Land B-Tree translation	https://github.com/formal-land/rocq-of-db
FoundationDB verification	"FoundationDB: A Distributed, Unbundled, Transactional Key Value Store" (SIGMOD 2021)
Amazon formal methods	"Use of Formal Methods at Amazon Web Services" (2014)
MongoDB TLA+	"Design and Modular Verification of Distributed Transactions in MongoDB" (VLDB 2025)
TigerBeetle VOPR	https://tigerbeetle.com/blog/2023-07-11-we-put-a-distributed-database-in-the-browser/
CVE-2022-35737 (Trail of Bits)	https://blog.trailofbits.com/2022/10/25/sqlite-vulnerability-july-2022-library-api/
Fly.io B-Tree internals	https://fly.io/blog/sqlite-internals-btree/

---

End of reference document. Ready for SpeckDL specification writing.