ToyDB

ToyDB is an early-stage relational database project written in Rust. The repository is currently focused on low-level storage primitives: fixed-size pages, page headers, compact slot metadata, tuple bytes, and the beginnings of a schema-aware tuple serialization plan.

The long-term design is a modular database stack with storage, core database services, and a server interface. See ARCHITECTURE.md for the larger target architecture.

Current Status

This project is not a usable SQL database yet. The storage manager has working unit-tested page and slot primitives, while the core and server crates are still mostly placeholders.

Implemented today:

• Rust workspace with separate crates for storage, core, and server code.
• Tuple-oriented slotted page layout in storage-manager.
• 8 KiB data pages with a page header, slot directory, free space, and tuple payload area.
• Compact Slot values that pack tuple offset, tuple size, and slot state into a u32.
• Page operations for creating a data page, appending slots, appending tuple bytes, and retrieving slots.
• Header and tuple flag types with zero-copy-friendly layouts through bytemuck.
• Basic schema and row-value traits for future tuple serialization.
• Unit tests for page creation, header flags, slot packing, tuple insertion, and placeholder crate setup.

Not implemented yet:

• SQL parser, planner, optimizer, or execution engine.
• Transactions, WAL, recovery, locking, or MVCC.
• Persistent heap files and disk-backed page I/O.
• Complete buffer pool management.
• Network protocol or real server behavior.
• Complete schema-aware tuple serialization.

Repository Layout

.
+-- Cargo.toml
+-- Cargo.lock
+-- ARCHITECTURE.md
+-- AGENTS.md
+-- crates
    +-- storage-manager
    |   +-- src/buffer_pool.rs
    |   +-- src/tuple_oriented_storage
    |       +-- header_flags.rs
    |       +-- page.rs
    |       +-- page_header.rs
    |       +-- plan.rs
    |       +-- plan_builder.rs
    |       +-- row_vals.rs
    |       +-- schema.rs
    |       +-- slot.rs
    |       +-- tuple.rs
    |       +-- types.rs
    +-- toydb-core
    |   +-- src/lib.rs
    +-- toydb-server
        +-- src/main.rs

Crates

Crate	Purpose	Current state
storage-manager	Low-level page, slot, tuple, schema, and future buffer-pool code.	Main active crate.
toydb-core	Future catalog, transactions, query planning, and execution.	Placeholder.
toydb-server	Future process/network entry point.	Placeholder binary.

Storage Page Model

The active tuple-oriented page implementation uses PAGE_SIZE = 8 * 1024.

+---------------------------+
| PageHeader                |
+---------------------------+
| Slot directory            | grows upward from the header
+---------------------------+
| Free space                |
+---------------------------+
| Tuple bytes               | grows downward from the end of the page
+---------------------------+

Each inserted tuple is copied into the tuple area, and a Slot is appended to the slot directory. A slot stores:

• offset: 15 bits
• tuple size: 13 bits
• flags: deleted, in use, or free

The page header tracks the log sequence number, page flags, lower pointer for the slot directory, and upper pointer for tuple data.

Requirements

• Rust toolchain with edition 2024 support.
• Cargo.

If you use rustup, a recent stable toolchain should be enough:

rustup update stable

Quick Start

Clone the repository:

git clone https://github.com/lupuszr/toydb.git
cd toydb

Run the full test suite:

cargo test

Run a faster compile check:

cargo check

Run the placeholder server binary:

cargo run -p toydb-server

Run storage-manager benchmarks:

cargo bench -p storage-manager

Development Commands

cargo fmt
cargo check
cargo test
cargo test -p storage-manager
cargo bench -p storage-manager

For a specific test:

cargo test test_page_append_tuple

Test Coverage

The current tests cover:

• page header construction and setters
• header flag formatting and bit operations
• slot packing, unpacking, and byte serialization
• page creation
• slot append and lookup
• tuple append and storage layout
• basic placeholder tests for toydb-core, toydb-server, and BufferPoolManager

At this stage, tests verify low-level in-memory storage behavior. They do not yet cover persistence, query execution, transactions, or server/client behavior.

Roadmap

Near-term development work:

• Finish buffer pool management.
• Add heap-file persistence without memory mapping.
• Complete schema-aware tuple serialization.
• Introduce catalog metadata.
• Add transaction, WAL, and recovery primitives.
• Build a query parser, planner, and execution model.
• Replace the placeholder server with a real protocol entry point.

The detailed roadmap and intended layered design are documented in ARCHITECTURE.md.

Notes for Contributors

• Keep crate-specific code inside the relevant workspace member.
• Prefer workspace dependencies in the root Cargo.toml.
• Use transparent newtypes and explicit layouts where page/tuple binary layout matters.
• Add focused unit tests for storage layout changes.
• Run cargo fmt and cargo test before sending changes.

License

This project is licensed under the MIT License.