HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench

Researchers have introduced HealthBench-Psych, a specialized mental health subset derived from OpenAI's broader HealthBench, to address the challenge of evaluating Large Language Model (LLM) performance in specific clinical domains. This new benchmark, including a 'Hard' version, was created by screening existing physician-rubric conversations for mental health relevance using an LLM-applied rubric, followed by rigorous validation through blinded clinician review. It aims to provide a standardized, domain-specific evaluation tool for LLMs in mental health contexts.

Why it matters

The development of specialized benchmarks like HealthBench-Psych is critical for accurately assessing the capabilities and limitations of AI in sensitive domains. This enables more targeted development and deployment of AI tools, ensuring they meet specific clinical needs and ethical standards. It directly addresses the growing public reliance on LLMs for sensitive applications like mental health support.

Key insights

• General-purpose health benchmarks for LLMs often lack resolution by clinical specialty, making it difficult to assess domain-specific performance.
• Mental health is a critical area where individuals increasingly seek psychological support from LLMs.
• Existing mental health evaluations for LLMs are largely bespoke academic benchmarks, hindering their integration into developer workflows.
• HealthBench-Psych was developed by filtering 5,000 HealthBench conversations for mental health relevance using an LLM-applied rubric.
• The subset was validated through two rounds of blinded clinician review, including concealed known-exclude controls.
• The resulting HealthBench-Psych dataset comprises 610 conversations focused on mental health.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.25071