trailbrake

Pure-MLX inference at the edge of Apple Silicon performance.

Family: the fast tier of a local-inference stack — iliria (deep) · racecontrol (router) · iliria-fm (Apple Foundation Models integration).

This is a new inference engine built for one hardware floor: Apple M5 Max and newer Max/Ultra-class Apple Silicon. It intentionally does not support Intel Macs, base chips, Pro-tier chips, CUDA, GGUF, llama.cpp, PyTorch, or CPU fallback inference.

Why it exists

General-purpose runtimes spend complexity on broad compatibility. This engine spends that complexity budget on throughput, predictable unified-memory use, and long-running local agents on high-end Macs.

Version 0.2 establishes a small, testable engine contract:

• strict M5 Max+ hardware admission;
• MLX-format safetensors only;
• native MLX model execution with no mlx-lm runtime dependency;
• capacity planning before weights are loaded;
• a persistent, capacity-bounded, multi-slot KV pool with longest-common-prefix reuse across conversation turns (boundary-safe; see docs/ARCHITECTURE.md);
• bounded chunked prefill to control peak graph memory;
• Qwen3 dense decoder support, including MLX affine quantized checkpoints;
• asynchronous one-token decode lookahead and linear-time streaming detokenization;
• streaming generation, deterministic cold/cache benchmarks, and a local HTTP API;
• performance telemetry for TTFT, decode throughput, and peak MLX memory.

Qwen3.5/3.6 hybrid Gated DeltaNet and MoE adapters are the next architecture milestones after the dense engine path is validated on-device.

Requirements

• macOS 27 or newer
• Apple M5 Max, M5 Ultra, or a later Max/Ultra-class chip
• Python 3.11+
• MLX 0.32.x
• a local MLX-format model directory containing config.json, tokenizer.json, and one or more .safetensors shards

Install

python3.11 -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'

Use

# Hardware and MLX readiness. Exits non-zero on unsupported Macs.
trailbrake doctor

# Inspect model size and calculate a safe unified-memory/KV allocation.
trailbrake plan --model /absolute/path/to/mlx-model --context 32768

# Stream one response.
trailbrake run \
  --model /absolute/path/to/mlx-model \
  --prompt "Write a Metal performance checklist" \
  --max-tokens 256

# Reproducible 112-token benchmark for cross-run comparisons.
trailbrake bench \
  --model /absolute/path/to/mlx-model \
  --prompt "Implement an LRU cache in Python" \
  --tokens 112 \
  --warmups 2 \
  --runs 5

# Separately measure exact repeated-prompt cache-hit TTFT.
trailbrake bench \
  --model /absolute/path/to/mlx-model \
  --prompt "Implement an LRU cache in Python" \
  --tokens 112 \
  --warmups 2 \
  --runs 5 \
  --reuse-prompt-cache

# OpenAI-compatible local endpoint on 127.0.0.1:8080.
trailbrake serve --model /absolute/path/to/mlx-model --port 8080

# Opt-in int8-quantized KV attention (lossy; default is lossless fp16).
trailbrake run \
  --model /absolute/path/to/mlx-model \
  --prompt "Write a Metal performance checklist" \
  --kv-bits 8

run, bench, and serve all accept --kv-bits {none,8} (default none) and --kv-group-size {32,64,128} (default 64). --kv-bits none is the unchanged, lossless fp16 attention path. --kv-bits 8 opts into the int8_composition KV-attention backend: it changes generated output (lossy) and is opt-in only -- it is never selected by default.

Measured on Qwen3-32B-4bit (M5 Max, 2026-07-22, pre-registered gate): quality is excellent -- greedy-argmax agreement vs fp16 of 98.0/99.7/99.9% at 0.5K/4K/16K context, perplexity within 0.4% -- but decode is slower at every context measured (-7.4% / -3.1% / -2.7%), because KV is only ~19% of decode bytes on this model and the compose/dequant overhead exceeds the bandwidth saved. Use --kv-bits 8 as a memory-capacity tool (about half the KV bytes: longer context or more cache slots in the same RAM, bought at a few percent decode) -- not as a speed flag.

For the first on-device checkpoint, run:

MODEL=/absolute/path/to/qwen3-mlx bash scripts/native_validate.sh

API

The server surface implements:

• GET /health
• GET /v1/models
• GET /v1/telemetry/memory
• POST /v1/completions
• POST /v1/chat/completions

Both completion endpoints support non-streaming JSON and HTTP/1.1 chunked SSE. Streaming responses can include a final exact usage event with stream_options.include_usage. Chat requests render the checkpoint's own tokenizer_config.json template; the Qwen3 fallback is used only when a checkpoint does not provide one.

The server binds to loopback by default. It has no authentication layer and must not be exposed directly to a network.

Current model contract

Version 0.2 accepts Qwen3 dense text checkpoints whose config.json reports model_type: "qwen3". Quantized checkpoints must use the standard MLX weight/scales/biases layout and declare quantization.bits plus quantization.group_size in config.json.

The loader rejects non-null rope_scaling until that position transform has an independent correctness oracle. It also rejects contexts above the checkpoint's declared limit and plans KV memory from the declared activation dtype instead of assuming BF16.

The loader never downloads a model and never guesses among multiple paths. A single explicit absolute model directory is required.

Performance rules

• No result is called an improvement without repeated warm runs.
• Comparable A/B runs use identical prompt, token count, sampler, context, memory budget, and thermal conditions.
• The canonical quick benchmark emits exactly 112 generated tokens.
• Report median decode throughput, spread, TTFT, and peak MLX memory.
• Experimental kernels remain opt-in until correctness and a measured speedup are both demonstrated.

Competitive status

trailbrake is deliberately narrower than oMLX and Rapid-MLX. Those projects currently have broader model coverage, batching, persistent disk caches, and more agent integrations. (An earlier version of this list also credited them with "speculative paths" as something trailbrake lacks — trailbrake has since shipped opt-in external-drafter speculative decoding: TB_DRAFT_MODEL, src/mlx_engine/speculative.py, GPU receipt in bench/spec_decode/results/.) trailbrake's defensible target is the deterministic single-request Qwen3 dense path on M5 Max+: strict pre-load admission, exact reference tokens, predictable BF16 KV capacity, low-overhead decode, and multi-slot longest-common-prefix prompt reuse.

No overall “faster local server” claim is made from a single model or a busy host. The dated evidence, competitor configurations, correctness gates, and claim thresholds are recorded in competitive validation.

The 2026-07-18 directional M5 Max smoke matrix used current oMLX 0.5.1 and Rapid-MLX 0.10.12 with identical 112-token greedy requests and caches/speculation disabled. trailbrake led end-to-end throughput on both Qwen3-0.6B and Qwen3-8B while matching the reference token IDs. The 8B margins were 2.82% over Rapid-MLX and 5.14% over oMLX, so they are reported as a narrow latency/overhead win, not a general server or kernel superiority claim.