ASR pipeline silently destroys stereo audio in channels-last layout (what soundfile/librosa return): mean(axis=0) averages across time

[Kayvan-Zahiri]

System Info

• transformers 5.7.0
• torch 2.11.0, numpy 1.26.4, soundfile 0.13.1
• Python 3.12, macOS, CPU

Who can help?

AutomaticSpeechRecognitionPipeline / audio pipelines.

Reproduction

Passing a stereo waveform loaded with soundfile silently discards almost all of the
audio. The pipeline logs a warning saying it converted to mono, and then transcribes
about 3 seconds of speech as ' you'.

import io, numpy as np, soundfile as sf
from transformers import pipeline

rng = np.random.default_rng(0)
mono = (rng.standard_normal(16000 * 3) * 0.05).astype(np.float32)
buf = io.BytesIO()
sf.write(buf, np.stack([mono, mono], axis=1), 16000, format="WAV", subtype="FLOAT")
buf.seek(0)

stereo, sr = sf.read(buf, dtype="float32")     # soundfile returns (samples, channels)
print("soundfile.read shape :", stereo.shape)

asr = pipeline("automatic-speech-recognition", model="openai/whisper-tiny", device="cpu")
print("mono   :", repr(asr(mono.copy())["text"]))
print("stereo :", repr(asr(stereo.copy())["text"]))
print("after pipeline's mean(axis=0):", stereo.mean(axis=0).shape)

Output:

soundfile.read shape : (48000, 2)
UserWarning: We expect a single channel audio input for AutomaticSpeechRecognitionPipeline,
             got 2. Taking the mean of the channels for mono conversion.
mono   : ' See you in the next video!'
stereo : ' you'
after pipeline's mean(axis=0): (2,)

Expected behavior

The stereo call should transcribe the same as the mono call. Instead 48,000 samples
become 2.

pipelines/automatic_speech_recognition.py:

if inputs.ndim != 1:
    logger.warning(
        f"We expect a single channel audio input for AutomaticSpeechRecognitionPipeline, "
        f"got {inputs.ndim}. Taking the mean of the channels for mono conversion."
    )
    inputs = inputs.mean(axis=0)

mean(axis=0) assumes channels-first, (channels, samples). soundfile.read and
librosa.load(..., mono=False) both return channels-last, (samples, channels), which is
also what scipy.io.wavfile.read returns. On that layout axis=0 averages across time,
so a 3-second stereo clip collapses to 2 numbers, gets padded back to 30s of silence, and
decodes to ' you'.

(2, N) input works correctly, so whether the pipeline destroys your audio depends on
which loader you used, with no error either way.

Two smaller things in the same three lines:

• The warning interpolates inputs.ndim, not the channel count, so it always says
  "got 2" for any 2-D input regardless of how many channels there are. For the
  channels-last case the number that would actually help ("got 48000") never appears.
• A user who reads the warning is told the conversion happened. It did, along the wrong
  axis.

Suggested fix: treat the smaller of the two dimensions as the channel axis, which is
unambiguous for any realistic audio since channels are ~1-8 and samples are thousands:

if inputs.ndim != 1:
    ch_axis = int(np.argmin(inputs.shape))
    logger.warning(
        f"We expect a single channel audio input for AutomaticSpeechRecognitionPipeline, "
        f"got shape {tuple(inputs.shape)}. Taking the mean over axis {ch_axis} for mono conversion."
    )
    inputs = inputs.mean(axis=ch_axis)

Raising on ndim > 2 would also be reasonable. Happy to open a PR with tests covering
both layouts if you would like.

[Kayvan-Zahiri]

One more argument for fixing this rather than documenting it: the ASR pipeline is the only
audio pipeline in the library that accepts multi-channel input at all, and it is the only
one that can silently corrupt it.

The two siblings refuse it outright:

# pipelines/audio_classification.py:232
if len(inputs.shape) != 1:
    raise ValueError("We expect a single channel audio input for AudioClassificationPipeline")

# pipelines/zero_shot_audio_classification.py:119
if len(audio.shape) != 1:
    raise ValueError("We expect a single channel audio input for ZeroShotAudioClassificationPipeline")

# pipelines/automatic_speech_recognition.py:422
if inputs.ndim != 1:
    logger.warning(...)
    inputs = inputs.mean(axis=0)

So the same stereo array from soundfile.read raises a clear ValueError in two pipelines
and transcribes as ' you' in the third. Whichever direction is preferred, the current
state is inconsistent within the same subsystem:

• If multi-channel should be rejected, ASR should raise like the other two, and the fix is
  three lines.
• If ASR should keep converting as a convenience, then the axis has to be inferred rather
  than assumed, because the two dominant loaders in the ecosystem disagree with the
  assumption the code makes.

I have no strong preference between those. The part worth changing either way is that a
caller currently gets a warning saying the conversion succeeded while receiving 2 samples
in place of 48,000.

[fasuizu-br]

Disclosure: I’m with BRAINIALL. The useful invariant here is that the mono conversion must preserve the declared sample dimension, independently of which loader produced the array.

We translated that boundary into a metadata-only preflight: https://www.brainiall.com/transcreve/en/tools/stereo-asr-layout-preflight

It keeps the two cases separate: (48000, 2) samples×channels with mean(axis=0) leaves 2 values and fails; (2, 48000) channels×samples with mean(axis=0) leaves 48000 and passes the shape gate. Explicit multichannel rejection is a third valid outcome. No audio or transcript is uploaded.

This does not claim an independent reproduction, choose between rejection and conversion, validate amplitude mixing, or imply the upstream issue is fixed. If a small JSON boundary contract or synthetic regression fixture would help a test/PR, reply and we can contribute only that artifact. Otherwise we will not follow up.