A high-performance, lightweight request forwarding system for vLLM large-scale deployments, providing advanced load balancing, prefill/decode disaggregation, and semantic-aware routing.
Response caching (exact + semantic), semantic cluster routing, multi-tenant API keys, Anthropic Messages API, LMCache-aware routing, per-worker auth, graceful drain, hot reload, pre-routing hooks, model aliasing, OpenTelemetry tracing, Grafana dashboard, and 30+ more features.
See FEATURES.md for the full comparison table and performance benchmarks. CHANGELOG.md for what's going on.
Built-in web dashboard for real-time monitoring and management. No Grafana or Prometheus server required.
cd ui && npm install && npm run devSee docs/dashboard.md for full documentation.
The router ships with a complete monitoring stack:
cd monitoring && docker compose up -d
# Grafana: http://localhost:3001 (admin/admin)
# Prometheus: http://localhost:9090The pre-provisioned Grafana dashboard includes 18 panels covering request traffic, latency percentiles, worker load distribution, cache hit ratios, circuit breaker states, routing decisions, and error rates. All 67+ Prometheus metrics are available for custom dashboards and alerting.
graph LR
Client([Clients<br/>OpenAI / Anthropic SDK])
Client -->|HTTP :3000| AUTH
subgraph Router["vllm-router"]
AUTH[Auth<br/>API Keys · Multi-tenant]
HOOKS[Hooks<br/>PII · Safety callouts]
EXACT[Exact Cache<br/>FNV-1a hash]
SEM[Semantic Cache<br/>Cosine similarity]
CLUSTER[Semantic Cluster<br/>Prompt → cluster]
POLICY[Load Balancer<br/>round_robin · cache_aware<br/>consistent_hash · random<br/>power_of_two · lmcache_aware]
RULES[Model Rules<br/>Alias · Wildcard · Fallback]
CB[Circuit Breaker<br/>+ Retry]
PD[PD Disaggregation<br/>Prefill ↔ Decode split]
ADMIN[Admin API<br/>drain · reload · stats<br/>decisions · tenants · config]
METRICS[Metrics<br/>67+ Prometheus counters<br/>OTel tracing]
HEADERS[Response Headers<br/>x-vllm-router-*<br/>cache-status · similarity]
TOK[Tokenizer<br/>HF · tiktoken · SentencePiece]
SD[Service Discovery<br/>Kubernetes · vLLM native]
AUTH --> HOOKS --> EXACT -->|miss| SEM -->|miss| CLUSTER -->|no match| POLICY
EXACT -->|hit| HEADERS
SEM -->|hit| HEADERS
POLICY --> RULES --> CB
CLUSTER -->|match| CB
end
CB -->|HTTP :8010| W1[vLLM Worker 1<br/>Llama-3 70B]
CB -->|HTTP :8020| W2[vLLM Worker 2<br/>Llama-3 70B]
PD -.->|prefill| PW[Prefill Worker]
PD -.->|decode| DW[Decode Worker]
EXACT -.->|redis backend| Redis[(Redis)]
SEM -.->|redis backend| Redis
TOK -.->|token cache| Redis
SEM -.->|fetch embedding| Emb[vLLM Embeddings<br/>bge-small-en :8030]
CLUSTER -.->|fetch embedding| Emb
HOOKS -.->|HTTP callout| HookSvc([Hook Services])
W1 <-.->|KV cache sync| LMC[LMCache<br/>Controller]
W2 <-.->|KV cache sync| LMC
LMC -.->|cache state| POLICY
SD -.->|watch pods| K8s([Kubernetes API])
METRICS -.->|/metrics| Prom[Prometheus :9090]
Prom -.-> Grafana[Grafana :3001]
METRICS -.->|OTLP/gRPC| OTLP([OTel Collector])
Request pipeline (solid lines): Auth → Hooks → Exact Cache → Semantic Cache → Semantic Cluster → Load Balancer → Model Rules → Circuit Breaker → Worker
Optional integrations (dashed lines): Redis, Embeddings, LMCache, Kubernetes, Hooks, Prometheus, Grafana, OTel
Minimum deployment: Router + 2 vLLM workers. Everything else is enabled via config.
# Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
# System deps (Ubuntu/Debian)
sudo apt-get install -y protobuf-compiler libprotobuf-dev libsentencepiece-devcargo build --releaseThe easiest way to start the router is with a YAML config file. Sample configs for every policy are in configs/:
# Round-robin across two workers
vllm-router --config-file configs/round-robin.yaml
# Cache-aware routing (reduces TTFT by reusing vLLM's KV cache)
vllm-router --config-file configs/cache-aware.yaml
# Session affinity (same user always goes to same worker)
vllm-router --config-file configs/consistent-hash.yaml
# Semantic cluster routing (routes by prompt content)
vllm-router --config-file configs/test-semantic-cluster.yaml
# LMCache-aware routing (real KV cache state from controller)
vllm-router --config-file configs/lmcache-aware.yaml
# LMCache prefix lookup (per-request exact prefix matching)
vllm-router --config-file configs/lmcache-prefix-lookup-local.yamlDetailed guides are in the docs/ folder:
| Guide | Description |
|---|---|
| Architecture | When to use the router, separation of concerns with vLLM/LMCache, caching layers |
| Configuration | Full YAML reference, CLI flags, authentication, retries, circuit breakers, tokenizer mapping |
| Authentication | Inbound client validation, per-worker backend API keys, embeddings endpoint auth, health probe exemptions |
| Load Balancing | Policy overview with defaults, use-case recommendations, multi-turn routing, per-policy details |
| Semantic Routing | Cluster routing by prompt content with embeddings, API key support |
| Caching | Exact-match and semantic response cache pipeline, Redis backend |
| Anthropic API | Anthropic Messages API support and streaming |
| PD Disaggregation | Prefill-Decode split inference, multi-turn with PD |
| Metrics | Full Prometheus metrics reference |
| Admin API | Graceful worker drain, hot config reload, active config/stats/decisions endpoints |
| LMCache Integration | LMCache controller-driven cache-aware routing |
| Kubernetes | Kubernetes service discovery setup |
| Monitoring | Grafana dashboard + Prometheus Docker Compose stack |
# Run all tests
cargo test --lib
# Lint (requires nightly)
cargo +nightly clippy -- -D warnings
# Start local test workers (requires vLLM)
./scripts/start_test_workers.sh # chat workers on :8010 and :8020
./scripts/start_test_workers.sh --all # + BAAI/bge-small-en-v1.5 embeddings on :8030This project is a fork of vllm-project/router (original author: Byron Hsu), which is itself a fork of SGLang Model Gateway. We thank the original authors for their work.
Maintained by Alberto Ferrer.