Thumb on the scale: Using traditional wargaming adjudication practices to improve AI-judged evaluations and grading

A research study proposes adapting two centuries of traditional wargaming adjudication techniques to improve automated AI evaluation and grading for open-ended tasks. While current AI-as-a-judge research focuses primarily on model selection and rubric phrasing, this work maps eleven established wargaming referee practices against known AI evaluation failures and tests a structured adjudication procedure across 378 actions in nine games, though the abstract is truncated prior to reporting the final test outcomes.

This research matters because organizations increasingly deploy automated AI judges to evaluate complex, subjective outputs where objective ground-truth answers do not exist. Relying solely on prompt phrasing and model choice leaves evaluation pipelines vulnerable to referee failure modes that procedural protocols can systematically mitigate. Adopting proven procedural frameworks from adjacent disciplines offers a pathway to increase reliability and governance in automated assessment.

Research in AI evaluation has historically prioritized model selection and rubric wording while neglecting the procedural rules followed by automated judges.

Forward consideration, not a verified fact.