"Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated

Recent research from arXiv reveals a significant methodological issue, termed "Mirror" contamination, in the evaluation of Large Language Models (LLMs) for predicting depression scores. Studies that use LLM responses derived directly from depression assessment language to predict scores on those same assessments yield near-perfect, but potentially misleading, prediction results. While non-mirror evaluations, using life history interviews, also showed high prediction capabilities, this indicates a need for more robust LLM evaluation methodologies in health-related applications.

Why it matters

This research highlights critical challenges in validating AI systems, particularly LLMs, for sensitive applications like health assessments. Understanding and mitigating methodological biases, such as criterion contamination, is crucial for developing trustworthy and ethically sound AI solutions and ensuring their reliable deployment in real-world scenarios. It underscores the necessity for rigorous and varied testing protocols to accurately gauge AI performance and avoid overstating capabilities.

Key insights

• LLM evaluations where language responses are directly tied to the assessment being predicted ('Mirror' evaluations) show near-perfect prediction of depression scores, indicating criterion contamination.
• A study with 110 participants found that 'Mirror' evaluations were near-perfect when LLMs predicted depression scores from structured diagnostic interviews.
• 'Non-Mirror' evaluations, using life history interviews, also demonstrated prediction sizes considered outstanding in psychology.
• Both 'Mirror' and 'Non-Mirror' LLM predictions correlated with Patient Health Questionnaire (PHQ-9) scores, suggesting some underlying predictive ability beyond contamination.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2508.05830