Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks

Research on legal multiple-choice benchmarks indicates that certain AI models can achieve above-chance scores by solely analyzing the provided options, without needing the actual question. This suggests a potential flaw in benchmark design or a sophisticated pattern recognition capability within models, where the options themselves contain sufficient information for correct selection. The study focused on UA-JudgeExam, a legal qualification test, revealing that a significant portion of items could be answered blind.

Why it matters

This research highlights a critical vulnerability in the design and evaluation of benchmarks, particularly in high-stakes domains like legal assessment. The ability of AI models to exploit structural or content-based cues in options, independent of questions, undermines the validity of current assessment methodologies and calls into question the genuine understanding models possess.

Key insights

• AI models can score above chance on multiple-choice legal benchmarks by only analyzing options, without the question.
• A specific model (Claude Haiku 4.5) achieved a 0.383 score against chance on the UA-JudgeExam when only shown options.
• The 'leak' of information from options is concentrated, with 11.8% of items answered correctly blind across all option orders, significantly higher than chance expectation (0.2 items).
• The ability to answer blind is not due to direct quotation from legislation; search over Ukrainian legislation recovered only a 0.128 score.
• The UA-JudgeExam benchmark consists of 11,990 four-option items from Ukraine's Higher Qualification Commission of Judges.
• Gating out identified 'leak' items retains 8,128 items for more robust evaluation.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.15428