WhisperFeatureExtractor: one non-finite input sample makes the entire feature matrix NaN, and the pipeline transcribes it silently

[Kayvan-Zahiri]

System Info

• transformers 5.7.0
• torch 2.11.0, numpy 1.26.4
• Python 3.12, macOS, CPU

Who can help?

Whisper / audio feature extraction.

Reproduction

One non-finite sample anywhere in the waveform makes WhisperFeatureExtractor return an
input-feature matrix that is entirely NaN, and the ASR pipeline then produces a
confident transcription from it with no error and no warning.

import numpy as np
from transformers import WhisperFeatureExtractor, pipeline

fe = WhisperFeatureExtractor.from_pretrained("openai/whisper-tiny")
rng = np.random.default_rng(0)
audio = (rng.standard_normal(16000) * 0.05).astype(np.float32)
audio[8000] = np.nan                      # one non-finite sample in 16,000

feats = fe(audio, sampling_rate=16000, return_tensors="np")["input_features"]
print("input non-finite samples :", int((~np.isfinite(audio)).sum()), "/", audio.size)
print("output NaN fraction      :", float(np.isnan(feats).mean()))

asr = pipeline("automatic-speech-recognition", model="openai/whisper-tiny", device="cpu")
print("transcription            :", repr(asr(audio)["text"]))

Output:

input non-finite samples : 1 / 16000
output NaN fraction      : 1.0
transcription            : '0'

Expected behavior

A single corrupt sample is 0.006% of the input and destroys 100% of the output.

The amplification happens at the dynamic-range floor in _np_extract_fbank_features:

log_spec = np.maximum(log_spec, log_spec.max() - 8.0)

log_spec.max() is NaN if any element is NaN, and np.maximum(x, nan) is NaN elementwise,
so one bad sample propagates to every one of the 80 x 3000 bins. The torch path has the
same line. Without that floor the damage would stay local to the affected frames.

What makes this worth reporting is not the NaN itself but that it is silent. generate()
runs on all-NaN features and emits '0', which is indistinguishable from a real
transcription downstream. Non-finite samples are not exotic in production audio: packet
loss, a failed resample, division by a zero-energy normalisation window, or a truncated
decode all produce them, and a caller batching thousands of files has no signal that one
of them silently returned garbage instead of failing.

Same behaviour for all-NaN and all-inf input. float("inf") input also yields '0'.

Possible fixes, in rough order of how invasive they are:

1. Validate in the feature extractor: raise ValueError if the waveform is not finite.
   Clearest failure, but it is a behaviour change for anyone currently relying on
   garbage-in-garbage-out.
2. Keep the damage local: compute the floor with np.nanmax so only the affected frames
   are NaN rather than the whole matrix. This alone still transcribes silently.
3. Warn rather than raise, on the same footing as the existing sampling-rate mismatch
   warning.

I would suggest 1, with 3 as the conservative alternative. Happy to open a PR for whichever
you prefer, including tests.

Related but separate, so I have not folded it in: a zero-length array is padded to 30s and
transcribed as ' you' rather than rejected.

[Kayvan-Zahiri]

Follow-up: this is not specific to Whisper. Any audio feature extractor that normalises
against a statistic computed over the whole waveform shows the same amplification, and
whether it does depends on the checkpoint's config rather than the class.

Same input as above, one NaN sample in 16,000, transformers 5.7.0:

model                             extractor                    NaN fraction out
openai/whisper-tiny               WhisperFeatureExtractor                 1.000
facebook/wav2vec2-base-960h       Wav2Vec2FeatureExtractor                1.000
facebook/hubert-large-ls960-ft    Wav2Vec2FeatureExtractor                1.000
microsoft/wavlm-base              Wav2Vec2FeatureExtractor                0.000

Input NaN fraction is 0.000063. For the first three that is a 16,000x amplification to
total loss of the features.

Two different mechanisms, same shape:

• Whisper, _np_extract_fbank_features: np.maximum(log_spec, log_spec.max() - 8.0).
  log_spec.max() is NaN, and np.maximum(x, nan) is NaN elementwise.
• Wav2Vec2 / HuBERT, zero_mean_unit_var_norm:
  (x - x.mean()) / np.sqrt(x.var() + 1e-7). Both mean() and var() are NaN.

microsoft/wavlm-base is the useful control. It is the same Wav2Vec2FeatureExtractor
class and it is unaffected, purely because its config carries do_normalize=False
where wav2vec2-base-960h and hubert-large-ls960-ft carry do_normalize=True.

That last point is the part I would weigh most. Two checkpoints loaded through the same
AutoFeatureExtractor call, with identical code paths in the caller, differ in whether a
single corrupt sample destroys the entire batch, and nothing in the API surfaces that.
A validation or warning at the extractor level would cover all of them at once, which
argues for option 1 or 3 from the original report rather than patching the Whisper line
in isolation.

[axiom-of-choice]

@Rocketknight1 you closed #47894 without a note, so before anyone spends more time here: is a fix wanted in this area at all, and if so, in which shape?