🔄 LLM Failover

Automatic LLM provider failover with circuit breaker, rate limit detection, and cost-aware routing.

A zero-dependency library for running LLM API calls with automatic failover across providers. When MiMo goes down, it switches to DeepSeek. When DeepSeek rate-limits, it falls back to OpenAI. Users never notice.

---

⚠️ Real-World Pitfall

MiMo went down at 2 AM for 47 minutes. My agent returned errors to 312 users because it had no fallback. The fix: automatic failover to DeepSeek, then OpenAI. Zero downtime since.

LLM providers have outages, rate limits, and silent degradation. MiMo returns 503 during maintenance windows. DeepSeek rate-limits at 60 RPM. OpenAI has periodic latency spikes. If your agent talks to only one provider, it fails when that provider fails.

This library adds automatic failover with circuit breaker pattern, rate limit detection, latency tracking, and cost-aware routing across MiMo, OpenAI, DeepSeek, Anthropic, and any OpenAI-compatible provider.

---

✨ Features

• Automatic Failover — Seamless switch to next provider on error, timeout, or rate limit
• Circuit Breaker — Stops hitting a failing provider, auto-resumes when it recovers
• Rate Limit Detection — Detects 429 responses and switches providers until limit resets
• Cost-Aware Routing — Prefer cheaper providers (MiMo > DeepSeek > OpenAI) when all are healthy
• Latency Tracking — Routes to fastest provider based on rolling average response time
• Provider Health Dashboard — Real-time status of all providers
• MiMo Optimized — Special handling for MiMo thinking models, token plan endpoints, regional failover
• Configurable Strategy — Priority-based, cost-based, latency-based, or round-robin routing
• Zero Dependencies — Pure ESM, Node.js 18+, nothing extra

---

🚀 Quick Start

1. Install

npm install llm-failover

2. Set Up Failover

import { createFailoverClient } from 'llm-failover';

const client = createFailoverClient({
  providers: [
    {
      name: 'mimo',
      baseUrl: 'https://token-plan-sgp.xiaomimimo.com/v1',
      apiKey: process.env.MIMO_API_KEY,
      model: 'mimo-v2.5',
      priority: 1,           // Try first
      costPerMillion: { input: 0.10, output: 0.30 },
    },
    {
      name: 'deepseek',
      baseUrl: 'https://api.deepseek.com/v1',
      apiKey: process.env.DEEPSEEK_API_KEY,
      model: 'deepseek-chat',
      priority: 2,           // Fallback
      costPerMillion: { input: 0.14, output: 0.28 },
    },
    {
      name: 'openai',
      baseUrl: 'https://api.openai.com/v1',
      apiKey: process.env.OPENAI_API_KEY,
      model: 'gpt-4o-mini',
      priority: 3,           // Last resort
      costPerMillion: { input: 0.15, output: 0.60 },
    },
  ],
  strategy: 'cost-aware',    // 'priority' | 'cost-aware' | 'latency' | 'round-robin'
  circuitBreaker: {
    threshold: 5,            // Failures before opening circuit
    resetMs: 60_000,         // Time before trying failed provider again
  },
  timeout: 30_000,           // Per-request timeout
});

// Make a request — failover is automatic
const response = await client.chat({
  messages: [{ role: 'user', content: 'Explain quantum computing' }],
});

console.log(response.content);        // Response text
console.log(response.provider);       // Which provider answered (e.g., "mimo")
console.log(response.failoverCount);  // How many failovers happened (0 = first choice worked)
console.log(response.cost);           // Estimated cost

3. Monitor Provider Health

const health = client.getHealth();
console.log(health);
// {
//   mimo:      { status: 'healthy',  latency: 450,  successRate: 0.99, circuit: 'closed' },
//   deepseek:  { status: 'healthy',  latency: 820,  successRate: 0.97, circuit: 'closed' },
//   openai:    { status: 'degraded', latency: 2100, successRate: 0.85, circuit: 'half-open' },
// }

4. CLI

# Test failover across providers
llm-failover test "Hello world" --providers mimo,deepseek,openai

# Show provider health
llm-failover health

# Benchmark providers (latency + success rate)
llm-failover benchmark --iterations 10

# Simulate outage
llm-failover simulate-outage --provider mimo --duration 60

---

📦 Architecture

┌──────────────────────────────────────────────────┐
│                  llm-failover                     │
├──────────────────────────────────────────────────┤
│                                                   │
│  ┌─────────────────────────────────────────────┐ │
│  │            Request Router                   │ │
│  │  Strategy: priority | cost | latency | rr   │ │
│  └────────────────────┬────────────────────────┘ │
│                       │                           │
│  ┌────────────────────▼────────────────────────┐ │
│  │          Circuit Breaker Engine             │ │
│  │                                             │ │
│  │  ┌──────────┐ ┌──────────┐ ┌──────────┐   │ │
│  │  │ MiMo     │ │ DeepSeek │ │ OpenAI   │   │ │
│  │  │ CLOSED   │ │ CLOSED   │ │ OPEN     │   │ │
│  │  │ (healthy)│ │ (healthy)│ │ (failing)│   │ │
│  │  └──────────┘ └──────────┘ └──────────┘   │ │
│  └────────────────────┬────────────────────────┘ │
│                       │                           │
│  ┌────────────────────▼────────────────────────┐ │
│  │           Provider Adapters                 │ │
│  │  MiMo │ OpenAI │ DeepSeek │ Anthropic │ ... │ │
│  └────────────────────┬────────────────────────┘ │
│                       │                           │
│  ┌────────────────────▼────────────────────────┐ │
│  │         Response Normalizer                 │ │
│  │  Unified format across all providers        │ │
│  └────────────────────┬────────────────────────┘ │
│                       │                           │
│  ┌────────────────────▼────────────────────────┐ │
│  │         Health & Metrics Tracker            │ │
│  │  Latency │ Success rate │ Cost │ Circuit    │ │
│  └─────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────┘

---

🖥️ CLI Reference

# Test a prompt across all configured providers
llm-failover test "Summarize today's news"

# Test with specific providers
llm-failover test "Hello" --providers mimo,openai

# Show provider health status
llm-failover health

# Benchmark all providers
llm-failover benchmark --iterations 5

# Simulate a provider outage
llm-failover simulate-outage --provider mimo --duration 30

# Show failover configuration
llm-failover config

---

📚 API Reference

createFailoverClient(config)

Creates a failover-aware LLM client.

Config options:

Option	Type	Default	Description
providers	Array	required	Provider configurations
strategy	string	'priority'	Routing strategy
circuitBreaker.threshold	number	5	Failures before opening circuit
circuitBreaker.resetMs	number	60000	MS before retrying failed provider
circuitBreaker.halfOpenMax	number	1	Test requests in half-open state
timeout	number	30000	Per-request timeout in ms
maxRetries	number	3	Max retries per provider
onFailover	Function	null	Callback when failover happens

Provider config:

Option	Type	Description
name	string	Provider identifier
baseUrl	string	API base URL
apiKey	string	API key
model	string	Model identifier
priority	number	Lower = higher priority
costPerMillion	object	{ input, output } USD per 1M tokens
rpmLimit	number	Requests per minute limit
maxTokens	number	Max tokens for this provider

Returns:

Method	Description
.chat(options)	Make a chat completion request with failover
.getHealth()	Get health status of all providers
.getStats()	Get request statistics
.reset()	Reset all circuit breakers and stats
.on(event, handler)	Subscribe to events

Routing Strategies

Strategy	Description
priority	Always try lowest priority number first
cost-aware	Prefer cheapest healthy provider
latency	Route to fastest responding provider
round-robin	Rotate through healthy providers

Events

client.on('failover', ({ from, to, reason }) => {
  console.log(`Failover: ${from} -> ${to} (${reason})`);
});

client.on('circuit_open', ({ provider, failures }) => {
  console.log(`Circuit opened: ${provider} after ${failures} failures`);
});

client.on('circuit_close', ({ provider }) => {
  console.log(`Circuit closed: ${provider} recovered`);
});

client.on('rate_limit', ({ provider, resetAt }) => {
  console.log(`Rate limited: ${provider} until ${resetAt}`);
});

Circuit Breaker

import { createCircuitBreaker } from 'llm-failover/circuit-breaker';

const breaker = createCircuitBreaker({
  threshold: 5,
  resetMs: 60_000,
  halfOpenMax: 1,
});

breaker.exec(() => riskyCall());    // Throws if circuit is open
breaker.getState();                  // 'closed' | 'open' | 'half-open'
breaker.getFailures();               // Current failure count
breaker.reset();                     // Force close circuit

---

⚠️ Pitfalls & Lessons Learned

1. MiMo Token Plan Endpoints Are Region-Specific

MiMo's Singapore endpoint (token-plan-sgp.xiaomimimo.com) and Beijing endpoint (api.xiaomimimo.com) have different latency and availability. When Singapore goes down, Beijing might still work. Configure both as separate providers with different priorities.

// ❌ Single MiMo endpoint — no regional failover
providers: [{ name: 'mimo', baseUrl: 'https://token-plan-sgp.xiaomimimo.com/v1', ... }]

// ✅ Regional failover
providers: [
  { name: 'mimo-sg', baseUrl: 'https://token-plan-sgp.xiaomimimo.com/v1', priority: 1 },
  { name: 'mimo-cn', baseUrl: 'https://api.xiaomimimo.com/v1', priority: 2 },
  { name: 'deepseek', baseUrl: 'https://api.deepseek.com/v1', priority: 3 },
]

2. Rate Limits Are Not Always 429

Some providers return 503 (Service Unavailable) or 529 (Overloaded) instead of 429 when rate-limited. MiMo returns 503 during maintenance. The failover client treats all 5xx and 429 as triggerable errors.

3. Thinking Models Need Different Timeouts

MiMo v2.5-pro and DeepSeek Reasoner take 10-30x longer than non-thinking models because of internal reasoning. Set higher timeouts for thinking models:

providers: [
  { name: 'mimo-pro', model: 'mimo-v2.5-pro', timeout: 120_000 },  // Thinking: 2 min
  { name: 'mimo', model: 'mimo-v2.5', timeout: 30_000 },           // Non-thinking: 30s
]

4. Circuit Breaker Prevents Cascade Failures

Without a circuit breaker, a failing provider receives 100% of retry traffic, making recovery harder. The circuit breaker stops sending requests after N failures, waits, then sends one test request. If it succeeds, the circuit closes.

5. Cost-Aware Routing Saves 40-60% on Multi-Provider Setups

When multiple providers are healthy, routing to the cheapest one (MiMo > DeepSeek > OpenAI) significantly reduces costs. The cost-aware strategy compares real-time pricing and routes accordingly.

6. Failover Events Are Critical for Debugging

Always log failover events. When an agent suddenly produces different output quality, the first question is "did it failover to a different provider?" The onFailover callback and event system make this visible.

---

📄 License

MIT — Hijrah Assalam