---
title: Inference Router Strategy
tags: [asr, architecture, routing, fastapi, vllm, server-side]
last_updated: 2026-04-10
---

# Inference Router Strategy

This page documents the server-side routing logic for the open-ASR testbed, including model targeting, audio normalization, and scheduler configuration.

## Overview

The server routes inference requests for **Cohere** and **Qwen** models to a **FastAPI** backend that runs **vLLM** as its inference engine.

```
Audio Input
    │
    ▼
┌─────────────────────┐
│  Audio Normalizer   │  ← pads or truncates to exactly 30s
└────────┬────────────┘
         │
    ▼
┌─────────────────────┐
│   FastAPI Backend   │
│    (vLLM engine)    │  ← enable_chunked_prefill=True
└────────┬────────────┘
         │
    ┌────┴────┐
    ▼         ▼
 Cohere     Qwen
```

## Components

### Audio Normalizer

All incoming audio is **padded or truncated to exactly 30 seconds** before being handed to the scheduler. This bounds the maximum prefill size, making [Chunked_Prefill](../Concepts/Chunked_Prefill.md) more predictable and preventing unbounded queue blocking.

### FastAPI Backend

The FastAPI application acts as the router and request broker. It accepts audio payloads, passes them through the normalizer, and dispatches inference jobs to vLLM.

### vLLM Scheduler

vLLM is configured with:

```python
enable_chunked_prefill=True
```

See [Chunked_Prefill](../Concepts/Chunked_Prefill.md) for a full explanation of why this is necessary and how it prevents long audio from starving concurrent decode streams.

## Client-Side Counterpart

The server-side router handles Cohere and Qwen paths only. Browser-based inference (for on-device models) is handled separately using `transformers.js` with [WebGPU_Cache_API](../Concepts/WebGPU_Cache_API.md) for weight caching and [LocalAgreement-2](../Concepts/LocalAgreement-2.md) for streaming stabilization.

## See Also

- [Chunked_Prefill](../Concepts/Chunked_Prefill.md) — scheduler setting that prevents prefill starvation
- [WebGPU_Cache_API](../Concepts/WebGPU_Cache_API.md) — client-side model loading and caching
- [LocalAgreement-2](../Concepts/LocalAgreement-2.md) — client-side streaming text stabilization