Hallucination by proxy in LLM-assisted differential diagnosis

Research from arXiv indicates that clinicians using Large Language Model (LLM)-assisted diagnostic tools can be susceptible to incorporating fabricated information, even when it presents a fictitious disease. A study found that 44% of participants integrated a non-existent condition suggested by a 'poisoned' LLM into their final differential diagnoses. This highlights risks associated with LLM reliability and user interaction in critical domains.

This research underscores critical risks in integrating advanced AI, specifically LLMs, into high-stakes decision-making environments. Organizations deploying such technologies must address the potential for systems to generate and users to accept erroneous or fabricated information, which can have significant operational and reputational consequences.

LLM-based diagnostic assistants, despite potential to augment diagnostic accuracy, are prone to 'hallucinations' and can project misleading confidence.

Forward consideration, not a verified fact.