Research Summary: Thumb on the scale: Using traditional wargaming adjudication practices to improve AI-judged evaluations and grading

A research study proposes adapting two centuries of traditional wargaming adjudication techniques to improve automated AI evaluation and grading for open-ended tasks. While current AI-as-a-judge research focuses primarily on model selection and rubric phrasing, this work maps eleven established wargaming referee practices against known AI evaluation failures and tests a structured adjudication procedure across 378 actions in nine games, though the abstract is truncated prior to reporting the final test outcomes.

Why it matters

This research matters because organizations increasingly deploy automated AI judges to evaluate complex, subjective outputs where objective ground-truth answers do not exist. Relying solely on prompt phrasing and model choice leaves evaluation pipelines vulnerable to referee failure modes that procedural protocols can systematically mitigate. Adopting proven procedural frameworks from adjacent disciplines offers a pathway to increase reliability and governance in automated assessment.

Key insights

• Research in AI evaluation has historically prioritized model selection and rubric wording while neglecting the procedural rules followed by automated judges.
• Traditional wargaming offers two centuries of established procedures, failure mode classifications, and corrective mechanisms for evaluating open-ended scenarios without single correct answers.
• The authors mapped eleven wargaming adjudication practices to address recognized failure modes in AI-based evaluations.
• An empirical test was conducted using 378 player actions across nine AI-refereed exercises, combining player-stated odds, adjudicator-defined target numbers, and standardized twenty-sided die rolls.
• The provided source abstract cuts off mid-sentence, preventing complete verification of the specific empirical findings and comparative performance metrics.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.32066