MIRA: A Bilingual Benchmark for Medical Information Response Audit

A new bilingual benchmark, Medical Information Response Audit (MIRA), has been developed to assess whether large language models (LLMs) provide consistent and comparable medical information across variations in user phrasing, language, register, and health literacy. Initial findings from testing five mainstream LLMs indicate that while models answer all medical questions, responses to prompts signaling low health literacy consistently omit key information, offer fewer concrete next steps, and provide less support for independent judgment.

Why it matters

This research highlights critical inconsistencies in how large language models deliver medical information based on user input characteristics, particularly health literacy. Organizations deploying or developing LLMs for healthcare-related applications must address these gaps to ensure equitable access to comprehensive and actionable health information, mitigating risks associated with incomplete guidance.

Key insights

• Existing safety evaluations for LLMs overlook the comparability of medical information across different user phrasings.
• MIRA is a new, bilingual, controlled benchmark designed to evaluate LLMs for consistent medical information delivery.
• The benchmark assesses responses based on user-side language, register, and health literacy signals.
• MIRA comprises 4,320 prompts derived from 60 medically reviewed, low-risk health questions.
• Testing across five mainstream LLMs showed that models provided responses to all medical questions.
• Responses to low health-literacy signals consistently resulted in the omission of key information.
• LLM outputs for low health-literacy prompts offered fewer concrete next steps.
• These responses also provided less support for independent judgment.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2605.28025