Rare Diseases, Common Dilemmas: LLMs Prioritize Equal Resource Distribution over Patient Benefit in Decision-Making

A recent study evaluating eleven state-of-the-art Large Language Models (LLMs) in clinical decision-making scenarios concerning rare diseases found that these models tend to prioritize equal resource distribution over maximizing individual patient benefit when faced with ethical conflicts. The research used a benchmark of 208 rare disease vignettes with inherent high-stakes ethical dilemmas.

Why it matters

This finding reveals a fundamental difference in how current AI models approach ethical trade-offs compared to human-centric medical ethics, which often balances individual benefit and justice. Understanding this bias is crucial for institutions considering the deployment of LLMs in healthcare, particularly in specialized and ethically complex areas like rare disease management, to prevent unintended consequences or suboptimal patient outcomes.

Key insights

• LLMs demonstrated a preference for equal resource distribution in ethically conflicting clinical scenarios.
• The models' decision-making in rare disease contexts often overlooked prioritizing patient benefit.
• The evaluation utilized 208 clinically grounded rare disease vignettes designed to present genuine, high-stakes conflicts.
• The study assessed 11 state-of-the-art LLMs, indicating a broad examination of current AI capabilities in this domain.
• This research highlights challenges in how LLMs handle subjective, value-laden clinical judgments, especially where prior information might be scarce.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.25236