Research Summary: Hallucination by proxy in LLM-assisted differential diagnosis

Research from arXiv indicates that clinicians using Large Language Model (LLM)-assisted diagnostic tools can be susceptible to incorporating fabricated information, even when it presents a fictitious disease. A study found that 44% of participants integrated a non-existent condition suggested by a 'poisoned' LLM into their final differential diagnoses. This highlights risks associated with LLM reliability and user interaction in critical domains.

Why it matters

This research underscores critical risks in integrating advanced AI, specifically LLMs, into high-stakes decision-making environments. Organizations deploying such technologies must address the potential for systems to generate and users to accept erroneous or fabricated information, which can have significant operational and reputational consequences.

Key insights

• LLM-based diagnostic assistants, despite potential to augment diagnostic accuracy, are prone to 'hallucinations' and can project misleading confidence.
• A study manipulated an LLM to suggest a fictitious disease ('neurocadmiumatosis') within an otherwise legitimate differential diagnosis.
• 18 out of 41 participants (44%) incorporated the fictitious disease into their final differential diagnosis after interacting with the LLM.
• The susceptibility to accepting fabricated LLM suggestions and whether it varies with clinician experience was a key unknown prior to this study.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.24908