Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit

Recent research highlights a methodological flaw in auditing Large Language Model (LLM) judges for bias, specifically when using Difference-in-Differences (DiD) on bounded rating scales. The study demonstrates that such methods can erroneously create the appearance of a bias or effect due to the interaction of censorship at scale endpoints with differential attenuation. This issue implies that current audits might be misinterpreting LLM judge behavior, potentially leading to inaccurate conclusions regarding fairness and performance.

This finding is critical for any entity relying on LLM judges for assessment or decision-making, as it questions the validity of existing bias detection methodologies. Misinterpreting LLM judge behavior due to flawed audit techniques can lead to misinformed strategic decisions, compromised trust in AI systems, and inefficient allocation of resources in addressing perceived biases.

Audits of LLM judges commonly employ Difference-in-Differences (DiD) designs on bounded rating scales to identify bias.

Forward consideration, not a verified fact.