Whisper large-v3 and large-v3-turbo: config declares float16 but WhisperFeatureExtractor emits float32, so the manual path raises

[Kayvan-Zahiri]

System Info

transformers 5.7.0, torch 2.11.0, Python 3.12.12, macOS 15.7.1 (arm64). Reproduces on CPU.

Who can help?

@eustlb @ylacombe (audio / Whisper)

Reproduction

Loading whisper-large-v3 the documented way, with no dtype argument, and feeding it the output of its own feature extractor raises:

import torch, io, soundfile as sf
from datasets import load_dataset, Audio
from transformers import WhisperForConditionalGeneration, WhisperFeatureExtractor

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
fe = WhisperFeatureExtractor.from_pretrained("openai/whisper-large-v3")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column("audio", Audio(decode=False))
wav, _ = sf.read(io.BytesIO(ds[0]["audio"]["bytes"]), dtype="float32")

feats = fe(wav, sampling_rate=16000, return_tensors="pt").input_features
print(next(model.parameters()).dtype, feats.dtype)   # torch.float16 torch.float32

with torch.no_grad():
    model.model.encoder(feats)

torch.float16 torch.float32
RuntimeError: Input type (float) and bias type (c10::Half) should be the same

Expected behavior

Either the model and its own feature extractor agree on a dtype, or the mismatch produces an error that names the cause.

What is actually going on

from_pretrained honours config.torch_dtype. Two checkpoints declare float16 and the rest declare float32:

checkpoint	config.torch_dtype	manual model + feature_extractor path
whisper-tiny	float32	works
whisper-base	float32	works
whisper-small	float32	works
whisper-medium	float32	works
whisper-large-v2	float32	works
whisper-large-v3	float16	RuntimeError
whisper-large-v3-turbo	float16	RuntimeError

WhisperFeatureExtractor always returns float32, so the two newest checkpoints are the only ones where the model's declared dtype and its feature extractor's output disagree. Both fail identically.

Three things make this easy to miss:

1. pipeline() is unaffected. It casts internally, so the high-level path works and only the manual path breaks. The manual path is what fine-tuning scripts and custom inference loops use.
2. It is version-independent and hardware-independent — it follows from the checkpoint config, not from a local setting.
3. The error names types, not the cause. "Input type (float) and bias type (c10::Half)" points at the conv layer, several frames from the from_pretrained call that chose the dtype, so it reads as a user error.

Workarounds

Either forces agreement:

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3", dtype=torch.float32)
# or
feats = feats.to(model.dtype)

Possible fixes

I do not want to presume the right one, so listing what seems available:

• Have WhisperFeatureExtractor accept and honour a dtype, so a processor built from the same checkpoint matches the model.
• Cast input_features to the encoder's dtype inside WhisperEncoder.forward, which is what pipeline() effectively does today.
• Leave the behaviour and raise a Whisper-specific error naming the checkpoint's declared dtype and the feature dtype.

Happy to open a PR for whichever direction you prefer, with a regression test covering both float16 checkpoints.

[Kayvan-Zahiri]

Scope is wider than the two OpenAI checkpoints. I surveyed the declared dtype across common speech checkpoints, and ran the ones marked executed:

checkpoint	config.torch_dtype	result
openai/whisper-large-v3	float16	RuntimeError (executed)
openai/whisper-large-v3-turbo	float16	RuntimeError (executed)
distil-whisper/distil-large-v3	float16	RuntimeError (executed)
openai/whisper-{tiny,base,small,medium,large-v2}	float32	works (executed)
microsoft/wavlm-base-plus	float32	works (executed)
distil-whisper/distil-large-v2	float32	no mismatch to trigger (config only)
facebook/seamless-m4t-v2-large, facebook/mms-1b-all, UsefulSensors/moonshine-base	float32	no mismatch to trigger (config only)
facebook/wav2vec2-base-960h, facebook/hubert-large-ls960-ft	not set	no mismatch to trigger (config only)

Confirmed distil-large-v3 fails identically:

distil-whisper/distil-large-v3
  weights torch.float16, features torch.float32
  RuntimeError: Input type (float) and bias type (c10::Half) should be the same

So it is three checkpoints, not two, and the pattern is that the newer fast-inference releases ship half-precision weights while their feature extractors do not. Whichever direction you prefer, a fix in WhisperEncoder.forward or in the feature extractor would cover all three, since they share both classes.

Two of the three are the checkpoints most people reach for now (large-v3-turbo and distil-large-v3), which is probably why this is surfacing.

Both workarounds in the issue are verified on large-v3: loading with dtype=torch.float32, and feats.to(model.dtype).

(Edited to separate the checkpoints I executed from the ones where I only read the config — the non-Whisper models use a different feature extractor, so "works" was doing more work than my evidence supported.)