Beyond "I Can't Help With That": How Child Safety Experts Evaluate AI Chatbot Safety

A recent study highlights significant shortcomings in current AI chatbot safety evaluations concerning youth, particularly in high-stakes situations. Existing methodologies often fail to align with real-world harms experienced by young users and rely on unvalidated assumptions about appropriate chatbot responses. This suggests a critical gap in assessing the practical safety of AI systems that youth increasingly use for social and emotional support.

Why it matters

The increasing reliance of youth on AI chatbots necessitates robust safety protocols, yet current evaluation methods are demonstrably insufficient. This creates significant operational and reputational risks for entities deploying or regulating AI systems, as potential harms to vulnerable populations may go undetected, undermining trust and requiring costly retrospective interventions.

Key insights

• Youth are increasingly using AI chatbots for social and emotional support.
• Current AI safety evaluations for children do not adequately incorporate real-world harms experienced by youth.
• Existing evaluations rely on unvalidated assumptions about appropriate AI outputs, such as refusal to engage.
• Evaluations typically focus on detecting adversarial prompts or surface-level harms, overlooking more subtle risks.
• These limitations mean current evaluations may fail to identify AI responses that are harmful to youth in practice.
• The study involved interviews with 19 practitioners, including social workers and therapists, who work directly with vulnerable youth, to understand the limitations of current evaluation practices.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.07902