ai
Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks
- Source
- arXiv — Computers and Society
- Published
- Last verified
- 19 Aug 2026
- Confidence
- High
- Evidence
- Original document retained
- Reading time
- 1 min
- Country
- International
- Relevant to
- Research & Evidence, Policy & Regulation, Operations & Delivery, Technology & Data
Executive summary
What happened, and why should leadership care?
Research on legal multiple-choice benchmarks indicates that certain AI models can achieve above-chance scores by solely analyzing the provided options, without needing the actual question. This suggests a potential flaw in benchmark design or a sophisticated pattern recognition capability within models, where the options themselves contain sufficient information for correct selection. The study focused on UA-JudgeExam, a legal qualification test, revealing that a significant portion of items could be answered blind.
Why this matters
Why is this strategically important?
This research highlights a critical vulnerability in the design and evaluation of benchmarks, particularly in high-stakes domains like legal assessment. The ability of AI models to exploit structural or content-based cues in options, independent of questions, undermines the validity of current assessment methodologies and calls into question the genuine understanding models possess.
Key insights
What should be noted from the evidence?
- AI models can score above chance on multiple-choice legal benchmarks by only analyzing options, without the question.
- A specific model (Claude Haiku 4.5) achieved a 0.383 score against chance on the UA-JudgeExam when only shown options.
- The 'leak' of information from options is concentrated, with 11.8% of items answered correctly blind across all option orders, significantly higher than chance expectation (0.2 items).
- The ability to answer blind is not due to direct quotation from legislation; search over Ukrainian legislation recovered only a 0.128 score.
- The UA-JudgeExam benchmark consists of 11,990 four-option items from Ukraine's Higher Qualification Commission of Judges.
Evidence and confidence
How far can this assessment be trusted?
High confidence. Named institution, original document retained and analysis corroborated.
Analysis is prepared editorially by Aziz Shuaib Ausi. The original publication remains the authoritative record, and executive judgement remains entirely human.
Source
Where does this originate?
Reported by arXiv — Computers and Society · International. This briefing summarises the publication for executive use; the document itself is not reproduced here.
Read the original publication