Research Summary: Right Order, Wrong Scale: Auditing LLM Judges for Occupational AI Measurement

Research indicates that while large language model (LLM) judges can consistently rank AI outputs against workplace requirements, they demonstrate significant variability and inaccuracy in determining acceptance rates. A new audit suite, O*NET-BENCH, reveals that despite high agreement on ranking, LLM judges show a broad range of acceptable response estimations (3.0%-97.9%) compared to human worker assessments (61.1%). This suggests that current LLM judge configurations are unreliable for quantifying occupational AI performance, despite their ability to order responses by quality.

Why it matters

The widespread use of AI in occupational settings necessitates reliable evaluation methods. This research highlights a critical discrepancy: while LLMs can order outcomes by quality, their ability to accurately quantify performance or acceptance rates is highly variable and often misaligned with human benchmarks. This divergence poses a significant risk to the credibility and utility of AI systems intended for workplace integration, especially when these systems are used to make decisions about human-level performance or quality standards.

Key insights

• LLM judges are increasingly deployed to evaluate AI outputs against occupational standards.
• The study introduces O*NET-BENCH, an audit suite based on 45,796 worker ratings, to assess LLM judge performance.
• 33 LLM judge configurations across six model families were evaluated on 4,501 test ratings.
• 25 configurations achieved a tie-aware pair accuracy of at least 0.60, indicating reasonable agreement on ranking.
• A train-fitted response-only TF-IDF baseline nearly matched the strongest LLM judge configurations in ranking ability.
• LLM judges estimated acceptance rates ranging from 3.0% to 97.9%, a stark contrast to human worker assessments of 61.1%.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.02492