Research Summary: Safety-Flag: A Unified Benchmark for the Reliability and Calibration of LLM Content Moderators

This research introduces Safety-Flag, a unified benchmark designed to assess the reliability and calibration of Large Language Models (LLMs) when used for content moderation. It consolidates seven existing safety benchmarks into a standardized flag/do-not-flag protocol, enabling a more comprehensive evaluation beyond simple aggregate accuracy. The study highlights that conventional accuracy metrics fail to capture critical aspects of moderator reliability, such as error direction, probability calibration, and the utility of confidence scores for human review, which often diverge.

Why it matters

The increasing reliance on LLMs for content moderation necessitates robust and multidimensional evaluation methods to ensure their reliability and ethical deployment. This research offers a standardized framework to better understand the nuanced performance of moderation systems, moving beyond simplistic accuracy metrics to address critical aspects like calibration and error types, which are vital for trust and effective risk management.

Key insights

• Safety-Flag unifies seven existing safety benchmarks (BeaverTails, XSTest, Ethics, WildGuard, Aegis, ToxiChat, ToxiGen) into a single, balanced flag/do-not-flag content moderation protocol.
• The benchmark provides item-level decisions and confidence scores for six general-purpose LLMs, four dedicated guard models, and three reference models.
• Safety-Flag measures three critical dimensions of moderator reliability: error direction (false positives vs. false negatives), probability calibration (how well confidence scores align with actual accuracy), and confidence-based error ranking for human review.
• Traditional aggregate accuracy metrics do not reveal insights into error direction, indicating a limitation in current evaluation methodologies.
• The three dimensions of moderator reliability often disagree, implying that models performing well on one aspect may underperform on others.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.19072