Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries

Research conducted across multiple AI platforms, languages, and mental health query conditions reveals a significant shift in online health information seeking, moving from user-curated keyword searches to platform-generated, citation-backed answers. The study audited three consumer AI products (ChatGPT, Perplexity, Google AI Overview) using mental health queries in English and six other languages, analyzing over 15,000 citations to characterize the sources provided by these systems.

Why it matters

This research is strategically important as it highlights a fundamental change in how individuals access critical health information, placing new emphasis on the veracity and reliability of AI-curated sources. Understanding the nature and quality of these citations is crucial for stakeholders concerned with public health, information integrity, and regulatory oversight of AI systems.

Key insights

• Online health information seeking is transitioning from keyword searches with ranked links to conversational AI systems that provide single answers with curated citations.
• The responsibility for source evaluation is shifting from the user to the AI platform itself.
• The types of sources surfaced by these conversational AI systems are not well understood or characterized.
• A multi-platform (ChatGPT, Perplexity, Google AI Overview), multilingual audit was performed on twenty English mental health questions and a subset translated into six additional languages.
• The study analyzed 15,942 citations from 1,140 responses, identifying 1,713 unique domains.
• Citations were classified using a nine-category organizational typology, indicating a systematic approach to source analysis.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.00319