Turbo Compiler — Developer Guide

Quick Start

# Build (debug)
cargo build --manifest-path turbo/Cargo.toml

# Build (release)
cargo build --release --manifest-path turbo/Cargo.toml

# Run all unit tests
cargo test --workspace --manifest-path turbo/Cargo.toml

# Run a .tb source file via JIT
cargo run --manifest-path turbo/Cargo.toml -- run turbo/tests/phase1/hello.tb
# Or via the installed binary:
# turbolang run turbo/tests/phase1/hello.tb

# Run integration tests (requires release build)
cd turbo && ./tests/run_tests.sh

Architecture

The compiler is a five-stage pipeline:

Source (.tb)
  │
  ▼
Lexer (turbo-lexer, logos)  →  Token stream
  │
  ▼
Parser (turbo-parser, recursive descent)  →  AST (Module)
  │
  ▼
Semantic Analysis (turbo-sema)  →  Validated AST + type errors
  │
  ▼
Codegen (turbo-codegen-cranelift)  →  JIT execution  or  AOT .o file
  │                                      │
  ▼                                      ▼
  done                           link with turbo_rt.c → native binary

Lexer tokenizes source using the logos crate. Whitespace-insensitive; newlines and semicolons are filtered out by the parser.
Parser is hand-written recursive descent. Collects multiple errors for better diagnostics (no bail-on-first-error).
Sema walks the AST to resolve types, check scopes, validate match exhaustiveness, and enforce trait bounds. Produces a SemaResult with both errors and warnings.
Codegen translates the AST into Cranelift IR. jit_run() for development, aot_compile() for production binaries.
C Runtime (turbo_rt.c) provides rt_print_*, allocation helpers, string operations, array/hashmap support, async primitives, and math functions. Linked into AOT binaries via cc.

Crate Map

Crate	Path	Purpose
`turbo-lexer`	`crates/turbo-lexer/`	Logos-based tokenizer. Defines `Token` enum and `Spanned<Token>`.
`turbo-ast`	`crates/turbo-ast/`	AST node definitions (`Module`, `Item`, `Expr`, `Stmt`, `TypeExpr`, `Pattern`) and `ErrorCode` enum.
`turbo-parser`	`crates/turbo-parser/`	Recursive descent parser. Entry: `parse(tokens) -> (Module, Vec<ParseError>)`, which also runs a post-parse COW rewrite pass (`cow_rewrite.rs`) before returning.
`turbo-sema`	`crates/turbo-sema/`	Semantic analysis and type checking. Entry: `check(module) -> SemaResult` (errors + warnings).
`turbo-codegen-cranelift`	`crates/turbo-codegen-cranelift/`	Cranelift JIT + AOT backend. Entries: `jit_run()`, `aot_compile()`.
`turbo-formatter`	`crates/turbo-formatter/`	Source formatter. Entry: `format_source()` / `format_file()`. Used by `turbolang fmt`.
`turbo-cli`	`crates/turbo-cli/`	CLI frontend (clap). Commands: run, build, test, fmt, init, lsp, repl, bench, doc, playground.
`turbo-lsp`	`crates/turbo-lsp/`	LSP server (lsp-server crate). Provides diagnostics, hover, and go-to-definition.

Key Types

turbo-ast (the shared vocabulary):

Span — Range<usize>, byte offsets into source.
Spanned<T> — wraps any node with its Span.
Module — root AST node; contains Vec<Spanned<Item>>.
Item — top-level: Function(FnDef), Struct(StructDef), Enum(EnumDef), Impl(ImplBlock), Trait(TraitDef), Import, Const.
FnDef — function name, params, return type, body, plus flags: is_async, is_test, is_unsafe.
StructDef — name, type params, derives, fields.
EnumDef — name, type params, variants (each may carry data fields).
Expr — expression nodes: literals, BinaryOp, Call, If, While, Match, Block, FieldAccess, MethodCall, Closure, ArrayLit, Index, Assign, Spawn, Await, etc.
Stmt — Let { mutable, name, ty, value }, Expr(...), Return(...), Defer(...).
TypeExpr — type syntax: Named, Unit, Array, FnType, Result, Optional, Future, Inferred.

turbo-ast:

ErrorCode — unique error code enum (E0001-E0515). Defined in turbo-ast/src/errors.rs. Used by all error types. See docs/errors.md for the full table.

turbo-sema:

Ty — internal type representation: I8, I16, I32, I64, U8, U16, U32, U64, F32, F64, Bool, Str, Unit, Array(Box<Ty>), Struct(String), Enum(String), Fn(Vec<Ty>, Box<Ty>), Result, Optional, Future, TypeParam, Error. int is an alias for I64, float for F64, usize for U64.
SemaError — { code: ErrorCode, message: String, span: Span }.

turbo-codegen-cranelift:

TurboTy — codegen-level type tag (distinct from Ty because Cranelift IR types alone can't distinguish e.g. str from i64 on ARM64).
CodegenError — { code: ErrorCode, message: String }.

Patterns

Error handling

Every diagnostic carries an ErrorCode (e.g. E0100) for searchable, unique identification. Codes are defined in turbo-ast/src/errors.rs. Full reference: docs/errors.md.
turbolang explain E0100 prints the description for any error code.
All error types carry a Span (except CodegenError). The CLI uses ariadne to render them as pretty diagnostics with the format error[E0100]: message.
Every rendered error ends with a more info: footer pointing at the GitHub blob URL https://github.com/ZVN-DEV/Turbo-Language/blob/master/docs/errors/E0NNN.md. The URL is generated by error_code_url() in turbo-cli/src/main.rs. Once turbolang.dev/errors/ has a real redirect, the URL will be flipped back to the short form (TODO(P3) in error_code_url).
The parser collects errors into Vec<ParseError> and continues parsing (error recovery).
Sema uses Ty::Error as a poison type to avoid cascading errors — if an expression has type Ty::Error, further checks on it are skipped.

Error code documentation — single source of truth

The long-form explanation for each error code lives in one place: turbo/crates/turbo-cli/src/errors/E0NNN.md. The CLI embeds these via include_str! so turbolang explain works in a release binary with no filesystem dependency.
A parallel public tree at docs/errors/E0NNN.md exists so the rendered footer URL has something to resolve to (currently a GitHub blob URL; the turbolang.dev/errors/E0NNN short form is aspirational). These are symlinks pointing back at the source-of-truth files — never duplicate the content.
turbo/crates/turbo-cli/build.rs parses turbo-ast::ErrorCode at build time and fails the build if any variant is missing a docs/errors/ entry. This is the same exhaustiveness guarantee the detailed_explanation() include_str! table provides for the source-of-truth side.
When adding a new error code, follow the checklist in CONTRIBUTING.md ("Adding a new error code") — both halves of the docs tree must be in place or the build breaks.

Built-in functions

Built-in functions (print, assert, len, push, str_*, hashmap_*, math_*, etc.) are handled as special cases inside compile_call() in turbo-codegen-cranelift/src/expr.rs. They are not in the AST; the compiler recognizes them by name.
To add a new built-in: add a branch in compile_call(), implement the JIT function pointer in the codegen setup, and add the C implementation in turbo_rt.c for AOT.

COW builtins

A subset of builtins — push, map, filter, replace, upper, lower, trim, repeat, split — are copy-on-write: they return a new value instead of mutating their first argument in place.
To make idiomatic code like arr.push(4) or s.trim() still act as a mutation when used as a statement, the parser runs a post-parse pass (turbo-parser/src/cow_rewrite.rs) that rewrites every COW call in statement position into a self-assignment (arr = push(arr, 4)). Value-position calls are left alone. The pass threads a value_ctx flag top-down from each function's return type.
The end-to-end regression tests are turbo/tests/phase1/cow_tail_expression.tb and turbo/tests/phase1/cow_tail_rvalue_contexts.tb.

Test structure

Integration tests live in turbo/tests/phase1/ as pairs: foo.tb (source) + foo.expected (expected stdout).
run_tests.sh compiles each .tb via turbolang run, captures stdout, and diffs against .expected.
Expected-error tests: if the .expected file starts with ERROR:, the test runner checks that the compiler error output contains the pattern.
Unit tests use standard #[cfg(test)] modules inside each crate.

Testing

Running tests

# Unit tests (all crates)
cargo test --all --manifest-path turbo/Cargo.toml

# Integration tests (needs release build)
cargo build --release --manifest-path turbo/Cargo.toml
cd turbo && ./tests/run_tests.sh

# Single file
cargo run --manifest-path turbo/Cargo.toml -- run turbo/tests/phase1/fibonacci.tb

Adding a test

Create turbo/tests/phase1/my_feature.tb with a fn main() that prints output.
Create turbo/tests/phase1/my_feature.expected with the exact expected stdout.
For error tests, put ERROR:<pattern> as the first line of .expected.

Common Tasks

Adding a new built-in function

Runtime (turbo/crates/turbo-codegen-cranelift/runtime/turbo_rt.c): implement the C function (e.g. rt_my_func).
JIT setup (turbo/crates/turbo-codegen-cranelift/src/jit.rs): register the function pointer in the JIT symbol table so the JIT can call it.
Codegen (compile_call() in turbo/crates/turbo-codegen-cranelift/src/expr.rs, with built-in dispatch wired through src/builtins.rs): add a branch that matches the function name, builds the Cranelift IR call, and returns the correct TurboTy.
Sema (turbo/crates/turbo-sema/src/lib.rs): add the function signature to the built-in type environment so the type checker accepts calls to it.
Test: add a .tb/.expected pair in turbo/tests/phase1/.
COW registration (only if the builtin returns a new value instead of mutating in place): add its name to the COW_BUILTINS list in turbo/crates/turbo-parser/src/cow_rewrite.rs so statement-position calls get rewritten into self-assigns.

Adding a new AST node

AST (turbo/crates/turbo-ast/src/lib.rs): add the variant to Expr, Stmt, or Item.
Lexer (turbo/crates/turbo-lexer/src/lib.rs): add any new tokens/keywords.
Parser (turbo/crates/turbo-parser/src/lib.rs): parse the new syntax into the new AST node.
Sema (turbo/crates/turbo-sema/src/lib.rs): add type-checking logic for the new node.
Codegen (turbo/crates/turbo-codegen-cranelift/src/lib.rs): add code generation for the new node.
Formatter (turbo/crates/turbo-formatter/src/lib.rs): handle pretty-printing if applicable.

Adding a new type

AST: add to TypeExpr if it needs new syntax.
Lexer: add keyword token if needed.
Parser: parse the type expression.
Sema: add to Ty enum, implement type-checking rules, update resolve_type_expr().
Codegen: add to TurboTy enum, implement turbo_ty_from_type_expr(), handle in compile_expr().
Runtime: add C support functions in turbo_rt.c if the type needs runtime representation.

File Layout

turbo/
  Cargo.toml              # Workspace root
  crates/
    turbo-lexer/          # Token definitions + logos lexer
    turbo-ast/            # AST types (shared by all crates)
    turbo-parser/         # Recursive descent parser
    turbo-sema/           # Type checking + semantic analysis
    turbo-codegen-cranelift/
      src/lib.rs            # Codegen entry, function/struct setup
      src/expr.rs           # Expression compilation incl. compile_call()
      src/builtins.rs       # Built-in function dispatch + JIT symbol table
      src/runtime.rs        # Runtime helpers and Cranelift glue
      src/wasm_codegen.rs   # WASM target backend
      src/aot.rs            # AOT object emission + linker invocation
      src/jit.rs            # JIT module setup + execution
      src/stmt.rs           # Statement-level helpers
      src/turbo_types.rs    # TurboTy enum
      runtime/turbo_rt.c    # C runtime linked into AOT binaries
    turbo-formatter/      # Source formatter (turbolang fmt)
    turbo-cli/            # CLI entry point + REPL + playground
    turbo-lsp/            # Language Server Protocol server
  tests/
    phase1/               # Integration tests (.tb + .expected pairs)
    adversarial/          # Edge-case / adversarial tests
    regression/           # Regression tests
    run_tests.sh          # Integration test runner script
design/                   # Language specification documents
examples/                 # Example applications (web-api, desktop-app, etc.)

FilesExpand file tree

CLAUDE.md

Latest commit

History

CLAUDE.md

File metadata and controls

Turbo Compiler — Developer Guide

Quick Start

Architecture

Crate Map

Key Types

Patterns

Error handling

Error code documentation — single source of truth

Built-in functions

COW builtins

Test structure

Testing

Running tests

Adding a test

Common Tasks

Adding a new built-in function

Adding a new AST node

Adding a new type

File Layout