Research Summary: Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness

A counterfactual audit evaluating legal chain-of-thought reasoning across seven open-weight large language models (8B to 70B) reveals that models frequently cite legal authorities without utilizing them to determine verdicts. While models cite the correct legal authority in 66.7% to 100% of generations across judicial and contractual benchmarks, altering the cited precedent or statute causes the verdict to change only inconsistently, proving that cited authorities do not reliably govern model conclusions.

Why it matters

This matters because relying on AI-generated legal citations creates a false sense of procedural rigor and auditability. The evidence demonstrates that large language models produce superficially valid justifications that do not causally drive their underlying determinations. Consequently, organizations risking automated legal or compliance decisions face significant liability if reasoning cannot be counterfactually verified.

Key insights

• Evaluated seven open-weight models (8B to 70B parameters) across four legal benchmarks covering judicial and contractual domains: CaseHOLD, ECHR, SCOTUS, and ContractNLI.
• Models successfully identify and name the correct governing authority in 66.7% to 100% of generated responses when required to justify decisions.
• Counterfactual substitution of authorities reveals unfaithful reasoning: CaseHOLD verdicts changed only 0.0% to 21.7% of the time despite altered legal authorities.
• Verdict sensitivity under substituted authorities reached 30.0% to 76.7% on ECHR and SCOTUS benchmarks, and 43.3% to 50.0% on ContractNLI.
• The presence of accurate legal citations does not demonstrate that a model's underlying verdict causally follows from the cited authority.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.12361