MIRA: A Bilingual Benchmark for Medical Information Response Audit

A new bilingual benchmark, Medical Information Response Audit (MIRA), has been developed to assess whether large language models (LLMs) provide consistent and comparable medical information across variations in user phrasing, language, register, and health literacy. Initial findings from testing five mainstream LLMs indicate that while models answer all medical questions, responses to prompts signaling low health literacy consistently omit key information, offer fewer concrete next steps, and provide less support for independent judgment.

This research highlights critical inconsistencies in how large language models deliver medical information based on user input characteristics, particularly health literacy. Organizations deploying or developing LLMs for healthcare-related applications must address these gaps to ensure equitable access to comprehensive and actionable health information, mitigating risks associated with incomplete guidance.

Existing safety evaluations for LLMs overlook the comparability of medical information across different user phrasings.

Forward consideration, not a verified fact.