Dual_Engine_Strategy

---

title: Dual-Engine Strategy tags: [asr, architecture, webgpu, server-side, dual-engine] last_updated: 2026-04-27

Dual-Engine Strategy

The Open-ASR Model Explorer runs two parallel inference engines. Requests are routed at the frontend before any network call is made.

                    ┌──────────────────┐
                    │   Model Select   │
                    │  (mode field)    │
                    └───────┬──────────┘
                            │
               ┌────────────┴────────────┐
               │                         │
          mode=webgpu               mode=server
               │                         │
               ▼                         ▼
  ┌────────────────────┐    ┌────────────────────────┐
  │  WebGPU Worker     │    │  FastAPI Backend       │
  │  (browser thread)  │    │  (HF transformers GPU) │
  └────────────────────┘    └────────────────────────┘

Client-Side Engine (WebGPU)

Property	Value
Models	Xenova/whisper-tiny, Xenova/whisper-base, onnx-community/cohere-transcribe-03-2026-ONNX
Runtime	@huggingface/transformers 4.0.1 in a Web Worker
Device	WebGPU with fp16, WASM fallback
Singleton	Pipeline cached in worker; workerLoadedModelId mirror in router skips round-trip on cache hit
Strengths	Zero-latency startup after first load, fully private (no audio leaves the browser), instant re-runs via singleton
Best for	English transcription, quick demos, privacy-sensitive audio, multilingual Cohere (with language param)

Singleton Cache Mechanics

1. First call: inferenceRouter posts load-model → worker creates pipeline → replies ready { modelId } → router stores workerLoadedModelId.
2. Subsequent calls with the same model: router reads workerLoadedModelId, matches, returns immediately — no message round-trip.
3. E2E test validates this: second transcription must complete in < 50 % of the first run's wall-clock time.

Server-Side Engine (HF Transformers GPU)

Property	Value
Models	openai/whisper-base, CohereLabs/cohere-transcribe-03-2026 (+ Qwen when vLLM unblocked)
Runtime	HF transformers 4.57.6 via pipeline('automatic-speech-recognition')
Device	CUDA GPU (PyTorch 2.10.0+cu128)
Endpoint	POST /api/transcribe/stream (chunked SSE)
Strengths	Full multilingual support, larger model capacity, GPU-accelerated
Best for	Chinese, Spanish, and other non-English languages

Blackwell (SM 12.0) Constraint

vLLM 0.19.0 cannot run on SM 12.0 (RTX 5070 Ti). All attention backends segfault: FA2 (sm_80 PTX incompatible with sm_120), FlashInfer 0.6.6 (cubins lack sm_120), Triton-based (TRITON_ATTN, FLEX_ATTENTION — Triton 3.6.0 ir.builder NULL-deref), TORCH_SDPA (not registered in V1 engine). The NVFP4 kernel guard PRs in 0.19.0 address quantization paths, not attention backends. The server-side engine uses HF transformers pipeline as the local execution path. The vLLM code path is wired and cloud-ready for non-Blackwell GPUs. Resolution requires Triton ≥3.6.1 or FA2/FA3 rebuilt with sm_120 PTX.

E2E Test Partitioning

Test	Engine	Sample	Key Assertions
WebGPU English & Singleton Speed	WebGPU	English	No errors, transcript populates, 2nd run < 50 % of 1st
Server-Side Multilingual Chinese	Server HF	Chinese	No 401/403/500, no error banner, transcript populates

See Also

• Inference_Router_Strategy — server-side routing details
• WebGPU_Cache_API — client-side model caching
• LocalAgreement-2 — streaming text stabilization