Async I/O Model Guide

Turso uses cooperative yielding with explicit state machines instead of Rust async/await.

Core Types

pub enum IOCompletions {
    Single(Completion),
}

#[must_use]
pub enum IOResult<T> {
    Done(T),      // Operation complete, here's the result
    IO(IOCompletions),  // Need I/O, call me again after completions finish
}

Functions returning IOResult must be called repeatedly until Done.

Completion and CompletionGroup

A Completion tracks a single I/O operation:

pub struct Completion { /* ... */ }

impl Completion {
    pub fn finished(&self) -> bool;
    pub fn succeeded(&self) -> bool;
    pub fn get_error(&self) -> Option<CompletionError>;
}

To wait for multiple I/O operations, use CompletionGroup:

let mut group = CompletionGroup::new(|_| {});

// Add individual completions
group.add(&completion1);
group.add(&completion2);

// Build into single completion that finishes when all complete
let combined = group.build();
io_yield_one!(combined);

CompletionGroup features:

• Aggregates multiple completions into one
• Calls callback when all complete (or any errors)
• Can nest groups (add a group's completion to another group)
• Cancellable via group.cancel()

Helper Macros

return_if_io!

Unwraps IOResult, propagates IO variant up the call stack:

let result = return_if_io!(some_io_operation());
// Only reaches here if operation returned Done

io_yield_one!

Yields a single completion:

io_yield_one!(completion);  // Returns Ok(IOResult::IO(Single(completion)))

State Machine Pattern

Operations that may yield use explicit state enums:

enum MyOperationState {
    Start,
    WaitingForRead { page: PageRef },
    Processing { data: Vec<u8> },
    Done,
}

The function loops, matching on state and transitioning:

fn my_operation(&mut self) -> Result<IOResult<Output>> {
    loop {
        match &mut self.state {
            MyOperationState::Start => {
                let (page, completion) = start_read();
                self.state = MyOperationState::WaitingForRead { page };
                io_yield_one!(completion);
            }
            MyOperationState::WaitingForRead { page } => {
                let data = page.get_contents();
                self.state = MyOperationState::Processing { data: data.to_vec() };
                // No yield, continue loop
            }
            MyOperationState::Processing { data } => {
                let result = process(data);
                self.state = MyOperationState::Done;
                return Ok(IOResult::Done(result));
            }
            MyOperationState::Done => unreachable!(),
        }
    }
}

Re-Entrancy: The Critical Pitfall

State mutations before yield points cause bugs on re-entry.

Wrong

fn bad_example(&mut self) -> Result<IOResult<()>> {
    self.counter += 1;  // Mutates state
    return_if_io!(something_that_might_yield());  // If yields, re-entry will increment again!
    Ok(IOResult::Done(()))
}

If something_that_might_yield() returns IO, caller waits for completion, then calls bad_example() again. counter gets incremented twice (or more).

Correct: Mutate After Yield

fn good_example(&mut self) -> Result<IOResult<()>> {
    return_if_io!(something_that_might_yield());
    self.counter += 1;  // Only reached once, after IO completes
    Ok(IOResult::Done(()))
}

Correct: Use State Machine

enum State { Start, AfterIO }

fn good_example(&mut self) -> Result<IOResult<()>> {
    loop {
        match self.state {
            State::Start => {
                // Don't mutate shared state here
                self.state = State::AfterIO;
                return_if_io!(something_that_might_yield());
            }
            State::AfterIO => {
                self.counter += 1;  // Safe: only entered once
                return Ok(IOResult::Done(()));
            }
        }
    }
}

Common Re-Entrancy Bugs

State Enum Design

Encode progress in state variants:

// Good: index is part of state, preserved across yields
enum ProcessState {
    Start,
    ProcessingItem { idx: usize, items: Vec<Item> },
    Done,
}

// Loop advances idx only when transitioning states
ProcessingItem { idx, items } => {
    return_if_io!(process_item(&items[idx]));
    if idx + 1 < items.len() {
        self.state = ProcessingItem { idx: idx + 1, items };
    } else {
        self.state = Done;
    }
}

Turso Implementation

Key files:

• core/types.rs - IOResult, IOCompletions, return_if_io!, return_and_restore_if_io!
• core/io/completions.rs - Completion, CompletionGroup
• core/util.rs - io_yield_one! macro
• core/state_machine.rs - Generic StateMachine wrapper
• core/storage/btree.rs - Many state machine examples
• core/storage/pager.rs - CompletionGroup usage examples

Testing Async Code

Re-entrancy bugs often only manifest under specific IO timing. Use:

• Deterministic simulation (testing/simulator/)
• Whopper concurrent DST (testing/concurrent-simulator/)
• Fault injection to force yields at different points

References

• docs/manual.md section on I/O

CDC (Change Data Capture) - Internal Feature Map

Overview

CDC tracks INSERT/UPDATE/DELETE changes on database tables by writing change records into a dedicated CDC table (turso_cdc by default). It is per-connection, enabled via PRAGMA, and operates at the bytecode generation (translate) layer. The sync engine consumes CDC records to push local changes to the remote.

Architecture Diagram

User SQL (INSERT/UPDATE/DELETE/DDL)
        |
        v
  ┌─────────────────────────────────────────────────┐
  │  Translate layer (core/translate/)              │
  │  ┌───────────────────────────────────────────┐  │
  │  │ prepare_cdc_if_necessary()                │  │
  │  │   - checks CaptureDataChangesInfo         │  │
  │  │   - opens CDC table cursor (OpenWrite)    │  │
  │  │   - skips if target == CDC table itself   │  │
  │  └───────────────────────────────────────────┘  │
  │  ┌───────────────────────────────────────────┐  │
  │  │ emit_cdc_insns()                          │  │
  │  │   - writes (change_id, change_time,       │  │
  │  │     change_type, table_name, id,          │  │
  │  │     before, after, updates) into CDC tbl  │  │
  │  └───────────────────────────────────────────┘  │
  │  + emit_cdc_full_record() / emit_cdc_patch_record() │
  └─────────────────────────────────────────────────┘
        |
        v
  CDC table (turso_cdc or custom name)
        |
        v
  ┌─────────────────────────────────────────────────┐
  │  Sync engine (sync/engine/)                     │
  │  DatabaseTape reads CDC table → DatabaseChange  │
  │  → apply/revert → push to remote                 │
  └─────────────────────────────────────────────────┘

Core Data Types

CaptureDataChangesMode + CaptureDataChangesInfo — core/lib.rs

CDC behavior is controlled by two types:

#[derive(Debug, Clone, Copy, Eq, PartialEq, Ord, PartialOrd)]
#[repr(u8)]
enum CdcVersion {
    V1 = 1,
    V2 = 2,
}

const CDC_VERSION_CURRENT: CdcVersion = CdcVersion::V2;

enum CaptureDataChangesMode {
    Id,          // capture only rowid
    Before,      // capture before-image
    After,       // capture after-image
    Full,        // before + after + updates
}

struct CaptureDataChangesInfo {
    mode: CaptureDataChangesMode,
    table: String,                  // CDC table name
    version: Option<CdcVersion>,    // schema version (V1 or V2)
}

The connection stores Option<CaptureDataChangesInfo> — None means CDC is off.

Key methods on CdcVersion:

• has_commit_record() — self >= V2, gates COMMIT record emission
• Display/FromStr — round-trips "v1" ↔ V1, "v2" ↔ V2

Key methods on CaptureDataChangesInfo:

• parse(value: &str, version: Option<CdcVersion>) — parses PRAGMA argument "<mode>[,<table_name>]", returns None for "off"
• cdc_version() — returns CdcVersion (panics if version is None). Single accessor replacing old is_v1()/is_v2()/version() methods.
• has_before() / has_after() / has_updates() — mode capability checks
• mode_name() — returns mode as string

Convenience trait CaptureDataChangesExt on Option<CaptureDataChangesInfo> provides:

• has_before() / has_after() / has_updates() — delegates to inner, returns false for None
• table() — returns Option<&str>, None when CDC is off

CDC Table Schema v1

Default table name: turso_cdc (constant TURSO_CDC_DEFAULT_TABLE_NAME)

CREATE TABLE turso_cdc (
    change_id   INTEGER PRIMARY KEY AUTOINCREMENT,
    change_time INTEGER,        -- unixepoch()
    change_type INTEGER,        -- 1=INSERT, 0=UPDATE, -1=DELETE
    table_name  TEXT,
    id          <untyped>,      -- rowid of changed row
    before      BLOB,           -- binary record (before-image)
    after       BLOB,           -- binary record (after-image)
    updates     BLOB            -- binary record of per-column changes
);

CDC Table Schema v2 (current)

CREATE TABLE turso_cdc (
    change_id    INTEGER PRIMARY KEY AUTOINCREMENT,
    change_time  INTEGER,        -- unixepoch()
    change_type  INTEGER,        -- 1=INSERT, 0=UPDATE, -1=DELETE, 2=COMMIT
    table_name   TEXT,
    id           <untyped>,      -- rowid of changed row
    before       BLOB,           -- binary record (before-image)
    after        BLOB,           -- binary record (after-image)
    updates      BLOB,           -- binary record of per-column changes
    change_txn_id INTEGER        -- transaction ID (groups rows into transactions)
);

v2 adds:

• change_txn_id column — groups CDC rows by transaction. Assigned via conn_txn_id(candidate) opcode which get-or-sets a per-connection transaction ID.
• change_type=2 (COMMIT) records — mark transaction boundaries. Emitted once per statement in autocommit mode, or on explicit COMMIT.

The CDC table is created at runtime by the InitCdcVersion opcode via CREATE TABLE IF NOT EXISTS.

CDC Version Table

When CDC is first enabled, a version tracking table is created:

CREATE TABLE turso_cdc_version (
    table_name TEXT PRIMARY KEY,
    version TEXT NOT NULL
);

Current version: CDC_VERSION_CURRENT = CdcVersion::V2 (defined in core/lib.rs, re-exported from core/translate/pragma.rs)

Version Detection in InitCdcVersion

The InitCdcVersion opcode detects v1 vs v2 by checking whether the CDC table already exists before creating it:

• If CDC table already exists but has no version row → v1 (pre-existing table from before version tracking)
• If CDC table doesn't exist → create with current version (v2)
• If version row already exists → use that version as-is

DatabaseChange — sync/engine/src/types.rs:229-249

Sync engine's Rust representation of a CDC row. Has into_apply() and into_revert() methods for forward/backward replay.

OperationMode — core/translate/emitter.rs

Used by emit_cdc_insns() to determine change_type value:

• INSERT → 1
• UPDATE / SELECT → 0
• DELETE → -1
• COMMIT → 2 (v2 only, emitted by emit_cdc_commit_insns)

Entry Points

1. PRAGMA — Enable/Disable CDC

Set: core/translate/pragma.rs

• Checks MVCC is not enabled (CDC and MVCC are mutually exclusive)
• Parses mode string via CaptureDataChangesInfo::parse() with CDC_VERSION_CURRENT
• Emits a single InitCdcVersion opcode — all CDC setup (table creation, version tracking, state change) happens at execution time

Get (read current mode): core/translate/pragma.rs

• Returns 3 columns: mode, table, version
• When off: returns ("off", NULL, NULL)
• When active: returns (mode_name, table, version)

Pragma registration: core/pragma.rs — CaptureDataChangesConn (and deprecated alias UnstableCaptureDataChangesConn) with columns ["mode", "table", "version"]

2. Connection State

Field: core/connection.rs — capture_data_changes: RwLock<Option<CaptureDataChangesInfo>> Getter: get_capture_data_changes_info() — returns read guard Setter: set_capture_data_changes_info(opts: Option<CaptureDataChangesInfo>) Default: initialized as None (CDC off)

3. ProgramBuilder Integration

Field: core/vdbe/builder.rs — capture_data_changes_info: Option<CaptureDataChangesInfo> Accessor: capture_data_changes_info() — returns &Option<CaptureDataChangesInfo> Passed from: core/translate/mod.rs — read from connection when creating builder

4. PrepareContext

Field: core/vdbe/mod.rs — capture_data_changes: Option<CaptureDataChangesInfo> Set from: PrepareContext::from_connection() — clones from connection.get_capture_data_changes_info()

5. InitCdcVersion Opcode — core/vdbe/execute.rs

Always emitted by PRAGMA SET. Handles all CDC setup at execution time:

1. For "off": stores None in state.pending_cdc_info, returns early
2. Checks if CDC table already exists (for v1 backward compatibility)
3. Creates CDC table (CREATE TABLE IF NOT EXISTS <cdc_table_name> ...) — v2 schema with change_txn_id column
4. Creates version table (CREATE TABLE IF NOT EXISTS turso_cdc_version ...)
5. Inserts version row: if CDC table pre-existed → "v1", otherwise → current version ("v2"). Uses INSERT OR IGNORE to preserve existing version rows.
6. Reads back actual version from the table
7. Stores computed CaptureDataChangesInfo in state.pending_cdc_info

The connection's CDC state is not applied in the opcode. Instead, pending_cdc_info is applied in halt() only after the transaction commits successfully. This ensures atomicity: if any step fails and the transaction rolls back, the connection's CDC state remains unchanged.

All table creation is done via nested conn.prepare()/run_ignore_rows() calls rather than bytecode emission, because the PRAGMA plan can't contain DML against tables that don't exist yet in the schema.

Bytecode Emission (core/translate/emitter.rs)

These are the core CDC code generation functions:

COMMIT Emission Strategy (v2)

Per-row call sites use emit_cdc_insns() (no COMMIT). End-of-statement sites call emit_cdc_autocommit_commit() which checks is_autocommit() at runtime:

• Autocommit mode: emits a COMMIT record after the statement completes
• Explicit transaction (BEGIN...COMMIT): skips per-statement COMMIT; the explicit COMMIT statement emits the COMMIT record via emit_cdc_commit_insns()

This ensures multi-row statements like INSERT INTO t VALUES (1),(2),(3) produce one COMMIT at the end, not one per row.

Integration Points — Where CDC Records Are Emitted

INSERT — core/translate/insert.rs

• Per-row: emit_cdc_insns() after insert, and before delete for REPLACE/conflict
• End-of-statement: emit_cdc_autocommit_commit() in emit_epilogue() after the insert loop

UPDATE — core/translate/emitter.rs

• Per-row: captures before-image, after-image via patch record, emits emit_cdc_insns()
• End-of-statement: emit_cdc_autocommit_commit() after the update loop

DELETE — core/translate/emitter.rs

• Per-row: captures before-image and emits emit_cdc_insns()
• End-of-statement: emit_cdc_autocommit_commit() after the delete loop

UPSERT (ON CONFLICT DO UPDATE) — core/translate/upsert.rs

• Per-row: emit_cdc_insns() for all three cases: pure insert, update after conflict, replace
• No end-of-statement COMMIT — upsert shares INSERT's epilogue

Schema Changes (DDL) — core/translate/schema.rs

• CREATE TABLE: emit_cdc_insns() (insert into sqlite_schema) + emit_cdc_autocommit_commit()
• DROP TABLE: emit_cdc_insns() per-row in metadata loop + emit_cdc_autocommit_commit() after loop
• CREATE INDEX: emit_cdc_insns() + emit_cdc_autocommit_commit() (core/translate/schema.rs)
• DROP INDEX: emit_cdc_insns() per-row + emit_cdc_autocommit_commit() after loop (core/translate/index.rs)

DDL in explicit transactions (BEGIN; CREATE TABLE t(x); COMMIT) does NOT emit per-statement COMMIT — the autocommit check prevents it.

ALTER TABLE — core/translate/update.rs

• Sets cdc_update_alter_statement on the update plan when CDC has updates mode

Views/Triggers — Explicitly excluded

• core/translate/view.rs — passes None for CDC cursor
• core/translate/trigger.rs — passes None for CDC cursor

Subqueries — No CDC

• core/translate/subquery.rs — cdc_cursor_id: None

Helper Functions (for reading CDC data)

table_columns_json_array(table_name) — core/function.rs, core/vdbe/execute.rs

Returns JSON array of column names for a table. Used to interpret binary records.

bin_record_json_object(columns_json, blob) — core/function.rs, core/vdbe/execute.rs

Decodes a binary record (from before/after/updates columns) into a JSON object using column names.

Sync Engine Integration

The sync engine is the primary consumer of CDC data.

DatabaseTape — sync/engine/src/database_tape.rs

• CDC config: DEFAULT_CDC_TABLE_NAME = "turso_cdc", DEFAULT_CDC_MODE = "full"
• PRAGMA name: CDC_PRAGMA_NAME = "capture_data_changes_conn"
• Initialization: connect() sets CDC pragma and caches cdc_version from turso_cdc_version table. Must be called before iterate_changes().
• Version caching: cdc_version: RwLock<Option<CdcVersion>> — set by connect(), read by iterate_changes(). Panics if not set.
• Iterator: DatabaseChangesIterator reads CDC table in batches, emits DatabaseTapeOperation. For v2, real COMMIT records from the table are emitted. For v1, a synthetic Commit is appended at end of batch. ignore_schema_changes: true (default) filters out sqlite_schema row changes but not COMMIT records.

Sync Operations — sync/engine/src/database_sync_operations.rs

• Change counting: SELECT COUNT(*) FROM turso_cdc WHERE change_id > ?

Sync Engine — sync/engine/src/database_sync_engine.rs

• Initialization: open_db() calls main_tape.connect(coro) to ensure CDC is set up and version is cached before any iterate_changes() calls.
• During apply_changes, checks if CDC table existed, re-creates it after sync

Replay Generator — sync/engine/src/database_replay_generator.rs

• Requires updates column to be populated (full mode)

Bindings CDC Surface

All bindings expose cdc_operations as part of sync stats:

Tests

• Integration tests: tests/integration/functions/test_cdc.rs — covers all modes, CRUD, transactions, schema changes, version table, backward compatibility. Registered in tests/integration/functions/mod.rs.
• Sync engine tests: sync/engine/src/database_tape.rs — CDC table reads, tape iteration, replay of schema changes.
• JS binding tests: bindings/javascript/sync/packages/{wasm,native}/promise.test.ts

Run: cargo test -- test_cdc (integration) or cargo test -p turso_sync_engine -- database_tape (sync engine).

User-facing Documentation

• CLI manual page: cli/manuals/cdc.md — accessible via .manual cdc in the REPL
• Database manual: docs/manual.md — CDC section linked in TOC

Key Design Decisions

1. Per-connection, not per-database. Each connection has its own CDC mode and can target different tables.
2. Bytecode-level implementation. CDC instructions are emitted alongside the actual DML bytecode during translation — no runtime hooks or triggers.
3. Self-exclusion. Changes to the CDC table and turso_cdc_version table are never captured (checked in prepare_cdc_if_necessary).
4. Schema changes tracked. DDL operations are recorded as changes to sqlite_schema table.
5. Binary record format. Before/after/updates columns use SQLite's MakeRecord format (same as B-tree payload).
6. Transaction-aware. CDC writes happen within the same transaction as the DML, so rollback naturally discards CDC entries.
7. Version tracking. CDC schema version is recorded in turso_cdc_version table and carried in CaptureDataChangesInfo.version for future schema evolution.
8. Atomic PRAGMA. Connection CDC state is deferred via pending_cdc_info in ProgramState and applied only at Halt. If the PRAGMA's disk writes fail and the transaction rolls back, the connection state stays unchanged.
9. Per-statement COMMIT (v2). COMMIT records are emitted once per statement (not per row), using emit_cdc_autocommit_commit() which checks is_autocommit() at runtime. In explicit transactions, only the final COMMIT emits a COMMIT CDC record.
10. Backward-compatible version detection. Pre-existing v1 CDC tables (without turso_cdc_version) are detected by checking table existence before creation. Existing tables get CdcVersion::V1 inserted into the version table.
11. Typed version enum. CdcVersion enum with #[repr(u8)] and Ord/PartialOrd enables feature gating via integer comparison (has_commit_record() = self >= V2). Display/FromStr handles database round-trip.
12. CDC and MVCC mutual exclusion. Enabling CDC when MVCC is active (or vice versa) returns an error. Checked at PRAGMA set time and journal mode switch time.

Code Quality Guide

Core Principle

Production database. Correctness paramount. Crash > corrupt.

Correctness Rules

1. No workarounds or quick hacks. Handle all errors, check invariants
2. Assert often. Never silently fail or swallow edge cases
3. Crash on invalid state if it risks data integrity. Don't continue in undefined state
4. Consider edge cases. On long enough timeline, all possible bugs will happen

Rust Patterns

• Make illegal states unrepresentable
• Exhaustive pattern matching
• Prefer enums over strings/sentinels
• Minimize heap allocations
• Write CPU-friendly code (microsecond = long time)

If-Statements

Wrong:

if condition {
    // happy path
} else {
    // "shouldn't happen" - silently ignored
}

Right:

// If only one branch should ever be hit:
assert!(condition, "invariant violated: ...");
// OR
return Err(LimboError::InternalError("unexpected state".into()));
// OR
unreachable!("impossible state: ...");

Use if-statements only when both branches are expected paths.

Comments

Do:

• Document WHY, not what
• Document functions, structs, enums, variants
• Focus on why something is necessary

Don't:

• Comments that repeat code
• References to AI conversations ("This test should trigger the bug")
• Temporal markers ("added", "existing code", "Phase 1")

Avoid Over-Engineering

• Only changes directly requested or clearly necessary
• Don't add features beyond what's asked
• Don't add docstrings/comments to unchanged code
• Don't add error handling for impossible scenarios
• Don't create abstractions for one-time operations
• Three similar lines > premature abstraction

Index Mutations

When code involves index inserts, deletes, or conflict resolution, double-check the ordering against SQLite. Wrong ordering causes index inconsistencies. and easy to miss.

Ensure understanding of IO model

• Async IO model

Cleanup

• Delete unused code completely
• No backwards-compat hacks (renamed _vars, re-exports, // removed comments)

Debugging Guide

Bytecode Comparison Flow

Turso aims for SQLite compatibility. When behavior differs:

1. EXPLAIN query in sqlite3
2. EXPLAIN query in tursodb
3. Compare bytecode
   ├─ Different → bug in code generation
   └─ Same but results differ → bug in VM or storage layer

Example

# SQLite
sqlite3 :memory: "EXPLAIN SELECT 1 + 1;"

# Turso
cargo run --bin tursodb :memory: "EXPLAIN SELECT 1 + 1;"

Manual Query Inspection

cargo run --bin tursodb :memory: 'SELECT * FROM foo;'
cargo run --bin tursodb :memory: 'EXPLAIN SELECT * FROM foo;'

Logging

# Trace core during tests
RUST_LOG=none,turso_core=trace make test

# Output goes to testing/test.log
# Warning: can be megabytes per test run

Threading Issues

Use stress tests with ThreadSanitizer:

rustup toolchain install nightly
rustup override set nightly
cargo run -Zbuild-std --target x86_64-unknown-linux-gnu \
  -p turso_stress -- --vfs syscall --nr-threads 4 --nr-iterations 1000

Deterministic Simulation

Reproduce bugs with seed. Note: simulator uses legacy "limbo" naming.

# Simulator
RUST_LOG=limbo_sim=debug cargo run --bin limbo_sim -- -s <seed>

# Whopper (concurrent DST)
SEED=1234 ./testing/concurrent-simulator/bin/run

Architecture Reference

• Parser → AST from SQL strings
• Code generator → bytecode from AST
• Virtual machine → executes SQLite-compatible bytecode
• Storage layer → B-tree operations, paging

Corruption Debugging

For WAL corruption and database integrity issues, use the corruption debug tools in scripts.

See references/CORRUPTION-TOOLS.md for detailed usage.

Differential Fuzzer

Always load Debugging skill for reference

The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.

Location

testing/differential-oracle/fuzzer/

Running the Fuzzer

Single Run

# Basic run (100 statements, random seed)
cargo run --bin differential_fuzzer

# With specific seed for reproducibility
cargo run --bin differential_fuzzer -- --seed 12345

# More statements with verbose output
cargo run --bin differential_fuzzer -- -n 1000 --verbose

# Keep database files after run (for debugging)
cargo run --bin differential_fuzzer -- --seed 12345 --keep-files

# All options
cargo run --bin differential_fuzzer -- \
  --seed <SEED>           # Deterministic seed
  -n <NUM>                # Number of statements (default: 100)
  -t <NUM>                # Number of tables (default: 2)
  -c <NUM>                # Columns per table (default: 5)
  --verbose               # Print each SQL statement
  --keep-files            # Persist .db files to disk

Continuous Fuzzing (Loop Mode)

# Run forever with random seeds
cargo run --bin differential_fuzzer -- loop

# Run 50 iterations
cargo run --bin differential_fuzzer -- loop 50

Docker Runner (CI/Production)

# Build and run from repo root
docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .
docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer

Environment variables for docker-runner:

• TIME_LIMIT_MINUTES - Total runtime (default: 1440 = 24h)
• PER_RUN_TIMEOUT_SECONDS - Per-run timeout (default: 1200 = 20min)
• NUM_STATEMENTS - Statements per run (default: 1000)
• LOG_TO_STDOUT - Print fuzzer output (default: false)
• GITHUB_TOKEN - For auto-filing issues
• SLACK_WEBHOOK_URL - For notifications

Output Files

All output goes to simulator-output/ directory:

Reproducing Errors

Always follow these steps

1. Find the seed in the error output:

   INFO: Starting differential_fuzzer with config: SimConfig { seed: 12345, ... }
   

2. Re-run with that seed:

   cargo run --bin differential_fuzzer -- --seed 12345 --verbose --keep-files
   

3. Check output files:

   • simulator-output/test.sql - Find the failing statement (look for -- FAILED:)
   • simulator-output/schema.json - Check table structure at failure time

4. Create a minimal reproducer

   • Create reproducer in .sqltest or in .rs always load Debugging skill for reference

5. Compare behavior manually: If needed try to compare the behaviour and produce a report in the end. Always write to a tmp file first with Edit tool to test the sql and then pass it to the binaries.

   # Run failing SQL against SQLite
   sqlite3 :memory: < simulator-output/test.sql
   
   # Run against tursodb CLI
   tursodb :memory: < simulator-output/test.sql
   

Understanding Failures

Oracle Failure Types

1. Row set mismatch - Turso returned different rows than SQLite
2. Turso errored but SQLite succeeded - Turso rejected valid SQL
3. SQLite errored but Turso succeeded - Turso accepted invalid SQL
4. Schema mismatch - Tables/columns differ after DDL

Warning (non-fatal)

• Unordered LIMIT mismatch - LIMIT without ORDER BY may return different valid rows

Key Source Files

Tracing

Set RUST_LOG for more detailed output:

RUST_LOG=debug cargo run --bin differential_fuzzer -- --seed 12345

index-knowledge

Generate hierarchical AGENTS.md files. Root + complexity-scored subdirectories.

Usage

--create-new   # Read existing → remove all → regenerate from scratch
--max-depth=2  # Limit directory depth (default: 5)

Default: Update mode (modify existing + create new where warranted)

Workflow (High-Level)

1. Discovery + Analysis (concurrent)
   • Launch parallel explore agents (multiple Task calls in one message)
   • Main session: bash structure + LSP codemap + read existing AGENTS.md
2. Score & Decide - Determine AGENTS.md locations from merged findings
3. Generate - Root first, then subdirs in parallel
4. Review - Deduplicate, trim, validate

TodoWrite([
  { id: "discovery", content: "Fire explore agents + LSP codemap + read existing", status: "pending", priority: "high" },
  { id: "scoring", content: "Score directories, determine locations", status: "pending", priority: "high" },
  { id: "generate", content: "Generate AGENTS.md files (root + subdirs)", status: "pending", priority: "high" },
  { id: "review", content: "Deduplicate, validate, trim", status: "pending", priority: "medium" }
])

Phase 1: Discovery + Analysis (Concurrent)

Mark "discovery" as in_progress.

Launch Parallel Explore Agents

Multiple Task calls in a single message execute in parallel. Results return directly.

// All Task calls in ONE message = parallel execution

Task(
  description="project structure",
  subagent_type="explore",
  prompt="Project structure: PREDICT standard patterns for detected language → REPORT deviations only"
)

Task(
  description="entry points",
  subagent_type="explore",
  prompt="Entry points: FIND main files → REPORT non-standard organization"
)

Task(
  description="conventions",
  subagent_type="explore",
  prompt="Conventions: FIND config files (.eslintrc, pyproject.toml, .editorconfig) → REPORT project-specific rules"
)

Task(
  description="anti-patterns",
  subagent_type="explore",
  prompt="Anti-patterns: FIND 'DO NOT', 'NEVER', 'ALWAYS', 'DEPRECATED' comments → LIST forbidden patterns"
)

Task(
  description="build/ci",
  subagent_type="explore",
  prompt="Build/CI: FIND .github/workflows, Makefile → REPORT non-standard patterns"
)

Task(
  description="test patterns",
  subagent_type="explore",
  prompt="Test patterns: FIND test configs, test structure → REPORT unique conventions"
)

# Measure project scale first
total_files=$(find . -type f -not -path '*/node_modules/*' -not -path '*/.git/*' | wc -l)
total_lines=$(find . -type f \( -name "*.ts" -o -name "*.py" -o -name "*.go" \) -not -path '*/node_modules/*' -exec wc -l {} + 2>/dev/null | tail -1 | awk '{print $1}')
large_files=$(find . -type f \( -name "*.ts" -o -name "*.py" \) -not -path '*/node_modules/*' -exec wc -l {} + 2>/dev/null | awk '$1 > 500 {count++} END {print count+0}')
max_depth=$(find . -type d -not -path '*/node_modules/*' -not -path '*/.git/*' | awk -F/ '{print NF}' | sort -rn | head -1)

// 500 files, 50k lines, depth 6, 15 large files → spawn additional agents
Task(
  description="large files",
  subagent_type="explore",
  prompt="Large file analysis: FIND files >500 lines, REPORT complexity hotspots"
)

Task(
  description="deep modules",
  subagent_type="explore",
  prompt="Deep modules at depth 4+: FIND hidden patterns, internal conventions"
)

Task(
  description="cross-cutting",
  subagent_type="explore",
  prompt="Cross-cutting concerns: FIND shared utilities across directories"
)
// ... more based on calculation

Main Session: Concurrent Analysis

While Task agents execute, main session does:

1. Bash Structural Analysis

# Directory depth + file counts
find . -type d -not -path '*/\.*' -not -path '*/node_modules/*' -not -path '*/venv/*' -not -path '*/dist/*' -not -path '*/build/*' | awk -F/ '{print NF-1}' | sort -n | uniq -c

# Files per directory (top 30)
find . -type f -not -path '*/\.*' -not -path '*/node_modules/*' | sed 's|/[^/]*$||' | sort | uniq -c | sort -rn | head -30

# Code concentration by extension
find . -type f \( -name "*.py" -o -name "*.ts" -o -name "*.tsx" -o -name "*.js" -o -name "*.go" -o -name "*.rs" \) -not -path '*/node_modules/*' | sed 's|/[^/]*$||' | sort | uniq -c | sort -rn | head -20

# Existing AGENTS.md / CLAUDE.md
find . -type f \( -name "AGENTS.md" -o -name "CLAUDE.md" \) -not -path '*/node_modules/*' 2>/dev/null

2. Read Existing AGENTS.md

For each existing file found:
  Read(filePath=file)
  Extract: key insights, conventions, anti-patterns
  Store in EXISTING_AGENTS map

If --create-new: Read all existing first (preserve context) → then delete all → regenerate.

3. LSP Codemap (if available)

lsp_servers()  # Check availability

# Entry points (parallel)
lsp_document_symbols(filePath="src/index.ts")
lsp_document_symbols(filePath="main.py")

# Key symbols (parallel)
lsp_workspace_symbols(filePath=".", query="class")
lsp_workspace_symbols(filePath=".", query="interface")
lsp_workspace_symbols(filePath=".", query="function")

# Centrality for top exports
lsp_find_references(filePath="...", line=X, character=Y)

LSP Fallback: If unavailable, rely on explore agents + AST-grep.

Merge: bash + LSP + existing + Task agent results. Mark "discovery" as completed.

Phase 2: Scoring & Location Decision

Mark "scoring" as in_progress.

Scoring Matrix

Decision Rules

Output

AGENTS_LOCATIONS = [
  { path: ".", type: "root" },
  { path: "src/hooks", score: 18, reason: "high complexity" },
  { path: "src/api", score: 12, reason: "distinct domain" }
]

Mark "scoring" as completed.

Phase 3: Generate AGENTS.md

Mark "generate" as in_progress.

Root AGENTS.md (Full Treatment)

# PROJECT KNOWLEDGE BASE

**Generated:** {TIMESTAMP}
**Commit:** {SHORT_SHA}
**Branch:** {BRANCH}

## OVERVIEW
{1-2 sentences: what + core stack}

## STRUCTURE
\`\`\`
{root}/
├── {dir}/    # {non-obvious purpose only}
└── {entry}
\`\`\`

## WHERE TO LOOK
| Task | Location | Notes |
|------|----------|-------|

## CODE MAP
{From LSP - skip if unavailable or project <10 files}

| Symbol | Type | Location | Refs | Role |

## CONVENTIONS
{ONLY deviations from standard}

## ANTI-PATTERNS (THIS PROJECT)
{Explicitly forbidden here}

## UNIQUE STYLES
{Project-specific}

## COMMANDS
\`\`\`bash
{dev/test/build}
\`\`\`

## NOTES
{Gotchas}

Quality gates: 50-150 lines, no generic advice, no obvious info.

Subdirectory AGENTS.md (Parallel)

Launch general agents for each location in ONE message (parallel execution):

// All in single message = parallel
Task(
  description="AGENTS.md for src/hooks",
  subagent_type="general",
  prompt="Generate AGENTS.md for: src/hooks
    - Reason: high complexity
    - 30-80 lines max
    - NEVER repeat parent content
    - Sections: OVERVIEW (1 line), STRUCTURE (if >5 subdirs), WHERE TO LOOK, CONVENTIONS (if different), ANTI-PATTERNS
    - Write directly to src/hooks/AGENTS.md"
)

Task(
  description="AGENTS.md for src/api",
  subagent_type="general",
  prompt="Generate AGENTS.md for: src/api
    - Reason: distinct domain
    - 30-80 lines max
    - NEVER repeat parent content
    - Sections: OVERVIEW (1 line), STRUCTURE (if >5 subdirs), WHERE TO LOOK, CONVENTIONS (if different), ANTI-PATTERNS
    - Write directly to src/api/AGENTS.md"
)
// ... one Task per AGENTS_LOCATIONS entry

Results return directly. Mark "generate" as completed.

Phase 4: Review & Deduplicate

Mark "review" as in_progress.

For each generated file:

• Remove generic advice
• Remove parent duplicates
• Trim to size limits
• Verify telegraphic style

Mark "review" as completed.

Final Report

=== index-knowledge Complete ===

Mode: {update | create-new}

Files:
  ✓ ./AGENTS.md (root, {N} lines)
  ✓ ./src/hooks/AGENTS.md ({N} lines)

Dirs Analyzed: {N}
AGENTS.md Created: {N}
AGENTS.md Updated: {N}

Hierarchy:
  ./AGENTS.md
  └── src/hooks/AGENTS.md

Anti-Patterns

• Static agent count: MUST vary agents based on project size/depth
• Sequential execution: MUST parallel (multiple Task calls in one message)
• Ignoring existing: ALWAYS read existing first, even with --create-new
• Over-documenting: Not every dir needs AGENTS.md
• Redundancy: Child never repeats parent
• Generic content: Remove anything that applies to ALL projects
• Verbose style: Telegraphic or die

MVCC Guide (Experimental)

Multi-Version Concurrency Control. Work in progress, not production-ready.

CRITICAL: Ignore MVCC when debugging unless the bug is MVCC-specific.

Enabling MVCC

PRAGMA journal_mode = 'mvcc';

Runtime configuration, not a compile-time feature flag. Per-database setting.

How It Works

Standard WAL: single version per page, readers see snapshot at read mark time.

MVCC: multiple row versions, snapshot isolation. Each transaction sees consistent snapshot at begin time.

Key Differences from WAL

Versioning

Each row version tracks:

• begin - timestamp when visible
• end - timestamp when deleted/replaced
• btree_resident - existed before MVCC enabled

Architecture

Database
  └─ mv_store: MvStore
      ├─ rows: SkipMap<RowID, Vec<RowVersion>>
      ├─ txs: SkipMap<TxID, Transaction>
      ├─ Storage (.db-log file)
      └─ CheckpointStateMachine

Per-connection: mv_tx tracks current MVCC transaction.

Shared: MvStore with lock-free crossbeam_skiplist structures.

Key Files

• core/mvcc/mod.rs - Module overview
• core/mvcc/database/mod.rs - Main implementation (~3000 lines)
• core/mvcc/cursor.rs - Merged MVCC + B-tree cursor
• core/mvcc/persistent_storage/logical_log.rs - Disk format
• core/mvcc/database/checkpoint_state_machine.rs - Checkpoint logic

Checkpointing

Flushes row versions to B-tree periodically.

PRAGMA mvcc_checkpoint_threshold = <pages>;

Process: acquire lock → begin pager txn → write rows → commit → truncate log → fsync → release.

Current Limitations

Not implemented:

• Garbage collection (old versions accumulate)
• Recovery from logical log on restart

Known issues:

• Checkpoint blocks other transactions, even reads!
• No spilling to disk; memory use concerns

Testing

# Run MVCC-specific tests
cargo test mvcc

# TCL tests with MVCC
make test-mvcc

Use #[turso_macros::test(mvcc)] attribute for MVCC-enabled tests.

#[turso_macros::test(mvcc)]
fn test_something() {
    // runs with MVCC enabled
}

References

• core/mvcc/mod.rs documents data anomalies (dirty reads, lost updates, etc.)
• Snapshot isolation vs serializability: MVCC provides the former, not the latter

PR Workflow Guide

Commit Practices

• Atomic commits. Small, focused, single purpose
• Don't mix: logic + formatting, logic + refactoring
• Good message = easy to write short description of intent

Learn git rebase -i for clean history.

PR Guidelines

• Keep PRs focused and small
• Run relevant tests before submitting
• Each commit tells part of the story

CI Environment Notes

If running as GitHub Action:

• Max-turns limit in .github/workflows/claude.yml
• OK to commit WIP state and push
• OK to open WIP PR and continue in another action
• Don't spiral into rabbit holes. Stay focused on key task

Security

Never commit:

• .env files
• Credentials
• Secrets

Third-Party Dependencies

When adding:

1. Add license file under licenses/
2. Update NOTICE.md with dependency info

External APIs/Tools

• Never guess API params or CLI args
• Search official docs first
• Ask for clarification if ambiguous

Storage Format Guide

Database File Structure

┌─────────────────────────────┐
│ Page 1: Header + Schema     │  ← First 100 bytes = DB header
├─────────────────────────────┤
│ Page 2..N: B-tree pages     │  ← Tables and indexes
│            Overflow pages   │
│            Freelist pages   │
└─────────────────────────────┘

Page size: power of 2, 512-65536 bytes. Default 4096.

Database Header (First 100 Bytes)

All multi-byte integers: big-endian.

Page Types

B-tree Structure

Two B-tree types:

• Table B-tree: 64-bit rowid keys, stores row data
• Index B-tree: Arbitrary keys (index columns + rowid)

Interior page:  [ptr0] key1 [ptr1] key2 [ptr2] ...
                   │         │         │
                   ▼         ▼         ▼
               child     child     child
               pages     pages     pages

Leaf page:     key1:data  key2:data  key3:data ...

Page 1 always root of sqlite_schema table.

Cell Format

Table Leaf Cell

[payload_size: varint] [rowid: varint] [payload] [overflow_ptr: u32?]

Table Interior Cell

[left_child_page: u32] [rowid: varint]

Index Cells

Similar but key is arbitrary (columns + rowid), not just rowid.

Record Format (Payload)

[header_size: varint] [type1: varint] [type2: varint] ... [data1] [data2] ...

Serial types:

Overflow Pages

When payload exceeds threshold, excess stored in overflow chain:

[next_page: u32] [data...]

Last page has next_page=0.

Freelist

Linked list of trunk pages, each containing leaf page numbers:

Trunk: [next_trunk: u32] [leaf_count: u32] [leaf_pages: u32...]

Turso Implementation

Key files:

• core/storage/sqlite3_ondisk.rs - On-disk format, PageType enum
• core/storage/btree.rs - B-tree operations (large file)
• core/storage/pager.rs - Page management
• core/storage/buffer_pool.rs - Page caching

Debugging Storage

# Integrity check
cargo run --bin tursodb test.db "PRAGMA integrity_check;"

# Page count
cargo run --bin tursodb test.db "PRAGMA page_count;"

# Freelist info
cargo run --bin tursodb test.db "PRAGMA freelist_count;"

References

• SQLite File Format
• SQLite B-Tree Module
• SQLite Internals: Pages & B-trees

Testing Guide

Test Types & When to Use

Note: TCL tests are being phased out in favor of testing/sqltests. The .sqltest format allows the same test cases to run against multiple backends (CLI, Rust bindings, etc.).

Running Tests

# Main test suite (TCL compat, sqlite3 compat, Python wrappers)
make test

# Single TCL test
make test-single TEST=select.test

# SQL test runner
make -C testing/sqltests run-cli

# OR
cargo run -p test-runner -- run <test-file or directory>

# Rust unit/integration tests (full workspace)
cargo test

Writing Tests

.sqltest (Preferred)

@database :default:

test example-addition {
    SELECT 1 + 1;
}
expect {
    2
}

test example-multiple-rows {
    SELECT id, name FROM users WHERE id < 3;
}
expect {
    1|alice
    2|bob
}

Location: testing/sqltests/tests/*.sqltest

You must start converting TCL tests with the convert command from the test runner (e.g cargo run -- convert <TCL_test_path> -o <out_dir>). It is not always accurate, but it will convert most of the tests. If some conversion emits a warning you will have to write by hand whatever is missing from it (e.g unroll a for each loop by hand). Then you need to verify the tests work by running them with make -C testing/sqltests run-rust, and adjust their output if something was wrong with the conversion. Also, we use harcoded databases in TCL, but with .sqltest we generate the database with a different seed, so you will probably need to change the expected test result to match the new database query output. Avoid changing the SQL statements from the test, just change the expected result

TCL

do_execsql_test_on_specific_db {:memory:} test-name {
  SELECT 1 + 1;
} {2}

Location: testing/*.test

Rust Integration

// tests/integration/test_foo.rs
#[test]
fn test_something() {
    let conn = Connection::open_in_memory().unwrap();
    // ...
}

Key Rules

• Every functional change needs a test
• Test must fail without change, pass with it
• Prefer in-memory DBs: :memory: (sqltest) or {:memory:} (TCL)
• Don't invent new test formats. Follow existing patterns
• Write tests first when possible

Test Database Schema

testing/system/testing.db has users and products tables. See docs/testing.md for schema.

Logging During Tests

RUST_LOG=none,turso_core=trace make test

Output: testing/system/test.log. Warning: very verbose.

Transaction Correctness Guide

Turso uses WAL (Write-Ahead Logging) mode exclusively.

Files: .db, .db-wal (no .db-shm - Turso uses in-memory WAL index)

WAL Mechanics

Write Path

1. Writer appends frames (page data) to WAL file (sequential I/O)
2. COMMIT = frame with non-zero db_size in header (marks transaction end)
3. Original DB unchanged until checkpoint

Read Path

1. Reader acquires read mark (mxFrame = last valid commit frame)
2. For each page: check WAL up to mxFrame, fall back to main DB
3. Reader sees consistent snapshot at its read mark

Checkpointing

Transfers WAL content back to main DB.

WAL grows → checkpoint triggered (default: 1000 pages) → pages copied to DB → WAL reused

Checkpoint types:

• PASSIVE: Non-blocking, stops at pages needed by active readers
• FULL: Waits for readers, checkpoints everything
• RESTART: Like FULL, also resets WAL to beginning
• TRUNCATE: Like RESTART, also truncates WAL file to zero length

WAL-Index

SQLite uses a shared memory file (-shm) for WAL index. Turso does not - it uses in-memory data structures (frame_cache hashmap, atomic read marks) since multi-process access is not supported.

Concurrency Rules

• One writer at a time
• Readers don't block writer, writer doesn't block readers
• Checkpoint must stop at pages needed by active readers

Recovery

On crash:

1. First connection acquires exclusive lock
2. Replays valid commits from WAL
3. Releases lock, normal operation resumes

Turso Implementation

Key files:

• WAL implementation - WAL implementation
• Page management, transactions

Connection-Private vs Shared

Per-Connection (private):

• Pager - page cache, dirty pages, savepoints, commit state
• WalFile - connection's snapshot view:
  • max_frame / min_frame - frame range for this connection's snapshot
  • max_frame_read_lock_index - which read lock slot this connection holds
  • last_checksum - rolling checksum state

Shared across connections:

• WalFileShared - global WAL state:
  • frame_cache - page-to-frame index (replaces .shm file)
  • max_frame / nbackfills - global WAL progress
  • read_locks[5] - read mark slots (TursoRwLock with embedded frame values)
  • write_lock - exclusive writer lock
  • checkpoint_lock - checkpoint serialization
  • file - WAL file handle
• DatabaseStorage - main .db file
• BufferPool - shared memory allocation

Correctness Invariants

1. Durability: COMMIT record must be fsynced before returning success
2. Atomicity: Partial transactions never visible to readers
3. Isolation: Each reader sees consistent snapshot
4. No lost updates: Checkpoint can't overwrite uncommitted changes

References

• SQLite WAL
• WAL File Format

Memory Benchmarking & Analysis

The perf/memory crate benchmarks memory usage of SQL workloads under WAL and MVCC journal modes. It uses dhat as the global allocator to track every heap allocation, and memory-stats for process-level RSS snapshots.

Location

• Benchmark crate: perf/memory/
• Analysis script: perf/memory/analyze-dhat.py
• dhat output: dhat-heap.json (written to CWD after each run)

Running Benchmarks

Always run in release mode — debug builds have wildly different allocation patterns and the results are not representative of real-world usage.

# Basic: single connection, WAL mode, insert-heavy workload
cargo run --release -p memory-benchmark -- --mode wal --workload insert-heavy -i 100 -b 100

# MVCC with concurrent connections
cargo run --release -p memory-benchmark -- --mode mvcc --workload mixed -i 100 -b 100 --connections 4

# Run a final checkpoint after the workload
cargo run --release -p memory-benchmark -- --mode wal --workload read-heavy --checkpoint

# All CLI options
cargo run --release -p memory-benchmark -- \
  --mode wal|mvcc \
  --workload insert-heavy|read-heavy|mixed|scan-heavy|series-blob \
  -i <iterations> \
  -b <batch-size> \
  --connections <N> \
  --checkpoint \
  --timeout <ms> \
  --cache-size <pages> \
  --format human|json|csv

Every run produces a dhat-heap.json in the current directory. This file contains per-allocation-site data for the entire run.

Built-in Workload Profiles

Profiles implement the Profile trait in perf/memory/src/profile/. To add a new workload, create a new file implementing the trait and wire it into the WorkloadProfile enum in main.rs.

Understanding the Output

The benchmark reports three categories of metrics:

RSS (process-level)

Measured via memory-stats crate. Includes everything: heap, mmap'd files (WAL, DB pages pulled into OS page cache), tokio runtime, etc. Snapshots are taken at phase transitions (setup -> run) and after each batch.

• Baseline: RSS before any DB work (runtime overhead)
• Peak: Highest RSS observed during the run
• Net growth: Final RSS minus baseline — the memory attributable to the workload

Heap (dhat)

Precise allocation tracking via the dhat global allocator. Only counts explicit heap allocations (malloc/alloc), not mmap.

• Current: Bytes still allocated at measurement time
• Peak: Highest simultaneous live allocation during the entire run
• Total allocs: Number of individual allocation calls
• Total bytes: Cumulative bytes allocated (includes freed memory) — measures allocation pressure

Disk

File sizes after the benchmark completes:

• DB file: The .db file
• WAL file: The .db-wal file (WAL mode only)
• Log file: The .db-log file (MVCC logical log only)

Analyzing dhat Output

After running a benchmark, use the analysis script to produce a readable report from dhat-heap.json:

# Overview: top allocation sites by bytes live at global peak
python3 perf/memory/analyze-dhat.py dhat-heap.json --top 15 --modules

# Focus on a specific subsystem
python3 perf/memory/analyze-dhat.py dhat-heap.json --filter mvcc --stacks
python3 perf/memory/analyze-dhat.py dhat-heap.json --filter btree --stacks
python3 perf/memory/analyze-dhat.py dhat-heap.json --filter page_cache --stacks

# Sort by different metrics
python3 perf/memory/analyze-dhat.py dhat-heap.json --sort-by eb  # bytes at exit (leaks)
python3 perf/memory/analyze-dhat.py dhat-heap.json --sort-by tb  # total bytes (pressure)
python3 perf/memory/analyze-dhat.py dhat-heap.json --sort-by mb  # max live bytes per site

# JSON output for programmatic use
python3 perf/memory/analyze-dhat.py dhat-heap.json --json

Sort Metrics

Analysis Flags

• --top N — Show top N sites (default 15)
• --filter PATTERN — Filter to sites/stacks containing substring (e.g. mvcc, btree, wal, pager)
• --stacks — Show full callstacks for top allocation sites
• --modules — Aggregate by crate/module for a high-level breakdown
• --json — Machine-readable aggregated output

Typical Workflow

When investigating memory usage or a suspected regression:

1. Run the benchmark with parameters matching the scenario:

   cargo run -p memory-benchmark -- --mode mvcc --workload mixed -i 500 -b 100 --connections 4
   

2. Get the high-level picture — which modules use the most memory:

   python3 perf/memory/analyze-dhat.py dhat-heap.json --modules --top 20
   

3. Drill into the hot module — e.g. if turso_core dominates:

   python3 perf/memory/analyze-dhat.py dhat-heap.json --filter turso_core --stacks --top 10
   

4. Check for leaks — anything still alive at exit that shouldn't be:

   python3 perf/memory/analyze-dhat.py dhat-heap.json --sort-by eb --top 10
   

5. Compare modes — run the same workload under WAL and MVCC and compare the reports to see the memory cost of MVCC versioning.

Concurrency Details

When --connections > 1:

• Setup phase (schema creation, seeding) always runs on a single connection sequentially
• Run phase spawns one tokio task per connection, each executing its batch concurrently
• --checkpoint adds a final single-connection PRAGMA wal_checkpoint(TRUNCATE) phase after the run phase
• Each connection gets busy_timeout set (default 30s, configurable via --timeout)
• WAL mode uses BEGIN, MVCC uses BEGIN CONCURRENT
• The Profile trait's next_batch(connections) returns one batch per connection with non-overlapping row IDs

Adding a New Profile

1. Create perf/memory/src/profile/your_profile.rs implementing the Profile trait
2. Add pub mod your_profile; to perf/memory/src/profile/mod.rs
3. Add a variant to WorkloadProfile enum in main.rs
4. Wire it into create_profile() in main.rs

The Profile trait:

pub trait Profile {
    fn name(&self) -> &str;
    fn next_batch(&mut self, connections: usize) -> (Phase, Vec<Vec<WorkItem>>);
}

Return Phase::Setup for schema/seeding (single batch), Phase::Run for measured work (one batch per connection), Phase::Done when finished.

Keeping This Skill Up to Date

This skill document is the source of truth for how agents use the memory benchmark tooling. If you modify the perf/memory crate — adding profiles, changing CLI flags, altering output format, updating the analysis script, changing the Profile trait, etc. — update this SKILL.md to match. Specifically:

• New CLI flags: add to the "Running Benchmarks" section
• New profiles: add to the "Built-in Workload Profiles" table
• Changed output metrics: update the "Understanding the Output" section
• New analyze-dhat.py flags or sort metrics: update the "Analyzing dhat Output" section
• Changed Profile trait signature: update "Adding a New Profile"

Future agents rely on this document being accurate. Stale instructions cause wasted work.