Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit

Recent research highlights a methodological flaw in auditing Large Language Model (LLM) judges for bias, specifically when using Difference-in-Differences (DiD) on bounded rating scales. The study demonstrates that such methods can erroneously create the appearance of a bias or effect due to the interaction of censorship at scale endpoints with differential attenuation. This issue implies that current audits might be misinterpreting LLM judge behavior, potentially leading to inaccurate conclusions regarding fairness and performance.

Why it matters

This finding is critical for any entity relying on LLM judges for assessment or decision-making, as it questions the validity of existing bias detection methodologies. Misinterpreting LLM judge behavior due to flawed audit techniques can lead to misinformed strategic decisions, compromised trust in AI systems, and inefficient allocation of resources in addressing perceived biases.

Key insights

• Audits of LLM judges commonly employ Difference-in-Differences (DiD) designs on bounded rating scales to identify bias.
• The study asserts that the endpoint on such a bounded rating scale is not properly identified when applying DiD.
• Each component of the double difference calculation is subject to censorship by its own share, confounding actual preference with differential attenuation.
• A 'severity shift' can manufacture an interaction effect when two responses are unequally censored due to varying distances from scale bounds.
• This methodological vulnerability can create a false impression of an effect or bias where none truly exists, or misrepresent its magnitude.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.27309