LLM-as-a-Judge Version Shift: The Instrument Changed, Not the Standard

A preregistered study examining automated research pipelines reveals that upgrading LLM-as-a-judge models within the same model family systematically destabilises measurement standards. Across evaluations of Qwen, Gemma, and Mistral model transitions on frozen machine learning research proposals, newer versions failed to exhibit uniform changes such as increased harshness, instead altering score locations, leniency patterns, and criterion geometry.

This matters because automated evaluation pipelines cannot treat model version upgrades as drop-in improvements without disrupting historical baselines and evaluation comparability. Relying on shifting evaluative instruments risks misinterpreting changes in scores as actual differences in evaluated material rather than instrument drift. Leaders must recognise that upgrading an automated evaluator fundamentally changes the assessment standard itself.

Upgrading LLM judges across adjacent versions in the same model family fails to preserve the underlying measurement standard.

Forward consideration, not a verified fact.