The Judgment-Consequence Gap: LLM Moral Reasoning in Healthcare Decisions

Research from arXiv investigates the moral reasoning of large language models (LLMs) in healthcare, specifically concerning resource allocation when patient actions contribute to illness. The study evaluates various LLMs across different families and capabilities using clinical vignettes. A key finding is the 'judgment-consequence gap,' where LLMs generally align with human judgment regarding patient responsibility for health-harming behaviors but diverge when considering the consequences for care denial.

Why it matters

The integration of LLMs into critical sectors like healthcare requires robust understanding of their decision-making processes, especially in morally complex scenarios. Misaligned moral reasoning could lead to inequitable or ethically questionable resource allocation decisions, undermining trust and potentially causing significant societal impact.

Key insights

• LLMs are being integrated into high-stakes domains, including healthcare, necessitating an understanding of their moral reasoning.
• The study focuses on LLM reasoning about responsibility in situations where patients' actions contribute to their illness, particularly regarding scarce medical resources.
• The research investigates how LLMs trace judgments from behavior to illness to the denial of care.
• A wide range of LLMs, differing in model families and capability levels, were evaluated.
• A 'judgment-consequence gap' was identified: LLMs largely agree with humans on patient responsibility for health-harming behaviors, but this agreement does not consistently extend to the consequence of denying care based on that responsibility.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.05583