VoxParadox Benchmark Exposes Transcript Bias in Audio LLMs

Researchers at the USC Viterbi School of Engineering have identified a critical architectural blind spot in advanced Audio Large Language Models. Accepted for ICML 2026, the study details “VoxParadox,” a systemic failure where models prioritize literal text transcription over paralinguistic cues. The research team, led by Professor Mohammad Soleymani alongside PhD students Ashutosh Chaubey and Jiacheng Pang, found that when transcript data conflicts with acoustic delivery, current models default to the text.

This transcript dominance causes models to misinterpret intent. A user saying “I am happy” in a sarcastic or sad tone is routinely categorized as happy by the LLM. Through layer-wise probing, the researchers traced this to a degradation of paralinguistic cues. Emotion, pitch, and tone data often degrade in deeper encoder layers and at the interface between the audio encoder and the language model. Even when the audio tokens successfully capture these non-verbal cues, the language model frequently ignores them to follow established textual patterns.

The VoxParadox Benchmark

To quantify this failure rate, the team introduced the VoxParadox benchmark. The dataset contains 2,000 verified adversarial examples spanning 10 paralinguistic tasks. These tasks are divided into biometric identification, such as age estimation and speaker counting, and prosodic evaluation, covering emotion, intonation, pitch, and volume.

The benchmark uses controlled speech synthesis to force a mismatch between the transcript and the speaking style. A test sample might feature a young voice stating the text “I am an old man.” This adversarial design forces the model to choose between the acoustic ground truth and the language-implied answer.

Prompt-Conditioned Layer Mixer

To mitigate the architectural bottleneck, the researchers developed two interventions. The first is a Prompt-Conditioned Layer Mixer (PCLM). PCLM adaptively combines information from multiple audio layers based on the specific input prompt. This forces the model to draw from the exact layers where paralinguistic data is most preserved for a given task, rather than relying on a static interface.

The team paired PCLM with Direct Preference Optimization (DPO). The DPO implementation trains the model to explicitly prefer outputs supported by acoustic evidence over those implied solely by the transcript.

Audio Flamingo 3 Performance Results

The USC team applied PCLM and DPO to Audio Flamingo 3 to measure the impact of these interventions. The modifications yielded massive accuracy gains on adversarial audio tasks.

Benchmark	Baseline Accuracy	Post-Intervention Accuracy
VoxParadox	17.40%	65.20%
MMSU Paralinguistic Subset	37.74%	54.78%

The baseline performance of 17.40% on VoxParadox illustrates how poorly standard Audio LLMs handle conflicting text and audio signals. The interventions pushed the model into a usable range for production applications.

If you evaluate and test AI agents that rely on audio input, acoustic ground truth must be part of your test suite. Developers who build real-time voice agents should implement adversarial audio evaluations to ensure their systems accurately process human intent alongside literal transcription.