High Scores, Hidden Failures: Evaluating Codebook-Based LLM Content Analysis

A large-scale empirical study evaluated large language models (LLMs) used for codebook-based content analysis across four datasets and five models, assessing stability, robustness, and conceptual validity under controlled perturbations and domain shifts. The research reveals that high aggregate predictive scores often mask underlying failure modes and unreliability, which conventional evaluation metrics fail to detect.

This research demonstrates that conventional performance indicators can create an illusion of model accuracy while concealing systemic classification errors. For institutions leveraging automated text analysis to inform strategy, policy, or research, relying on high aggregate scores without deeper validation introduces hidden operational and analytical vulnerabilities.

LLMs deployed for codebook-based content analysis present significant failure modes that remain undetectable using standard aggregate evaluation metrics.

Forward consideration, not a verified fact.