Research Summary: HEAR: Real Voices, Real Bias: A Large-Scale Human-Recorded, Demographically Diverse Benchmark for Audio Language Models

A new benchmark, HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), has been introduced to assess bias in audio language models (Audio-LLMs). Comprising 87,000 real human audio samples from 843 demographically diverse participants, HEAR facilitates comprehensive evaluation through both Multiple Choice Question Answering (MCQA) and open-ended tasks. Initial evaluations reveal that voice-conditioned bias is model-specific and that personalization instructions consistently worsen demographic disparities within these models.

Why it matters

The introduction of a demographically diverse benchmark for Audio-LLMs is critical for understanding and mitigating algorithmic bias in voice-enabled technologies. Identifying that bias is model-specific and worsened by personalization highlights significant risks to equitable service delivery and user trust across various applications.

Key insights

• HEAR is a new, large-scale, ecologically valid benchmark with 87,000 real human audio samples from 843 diverse participants.
• It is the first large-scale voice benchmark exclusively using authentic human speech.
• HEAR supports comprehensive evaluation through Multiple Choice Question Answering (MCQA) and open-ended long-form tasks.
• Evaluations show that voice-conditioned bias is a model-specific characteristic in Audio-LLMs.
• Personalization instructions consistently exacerbate existing demographic disparities in model behavior.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.35952