Research Summary: LLM-as-a-Judge Version Shift: The Instrument Changed, Not the Standard

A preregistered study examining automated research pipelines reveals that upgrading LLM-as-a-judge models within the same model family systematically destabilises measurement standards. Across evaluations of Qwen, Gemma, and Mistral model transitions on frozen machine learning research proposals, newer versions failed to exhibit uniform changes such as increased harshness, instead altering score locations, leniency patterns, and criterion geometry.

Why it matters

This matters because automated evaluation pipelines cannot treat model version upgrades as drop-in improvements without disrupting historical baselines and evaluation comparability. Relying on shifting evaluative instruments risks misinterpreting changes in scores as actual differences in evaluated material rather than instrument drift. Leaders must recognise that upgrading an automated evaluator fundamentally changes the assessment standard itself.

Key insights

• Upgrading LLM judges across adjacent versions in the same model family fails to preserve the underlying measurement standard.
• The assumption that newer LLM versions are consistently harsher is inaccurate; two evaluated transitions shifted toward leniency, while another demonstrated mixed, off-anchor attenuation.
• LLM judge version replacements alter both baseline score location and underlying criterion geometry.
• Measurement drift occurred across multiple prominent model families, specifically Qwen, Gemma, and Mistral, under controlled information conditions.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.32986