"Mirror" Large Language Model Evaluations of Depression are Criterion Contaminated

Recent research from arXiv reveals a significant methodological issue, termed "Mirror" contamination, in the evaluation of Large Language Models (LLMs) for predicting depression scores. Studies that use LLM responses derived directly from depression assessment language to predict scores on those same assessments yield near-perfect, but potentially misleading, prediction results. While non-mirror evaluations, using life history interviews, also showed high prediction capabilities, this indicates a need for more robust LLM evaluation methodologies in health-related applications.

This research highlights critical challenges in validating AI systems, particularly LLMs, for sensitive applications like health assessments. Understanding and mitigating methodological biases, such as criterion contamination, is crucial for developing trustworthy and ethically sound AI solutions and ensuring their reliable deployment in real-world scenarios. It underscores the necessity for rigorous and varied testing protocols to accurately gauge AI performance and avoid overstating capabilities.

LLM evaluations where language responses are directly tied to the assessment being predicted ('Mirror' evaluations) show near-perfect prediction of depression scores, indicating criterion contamination.

Forward consideration, not a verified fact.