Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

A new expert-validated benchmark designed to assess the reliability of Large Language Models (LLMs) within the Colombian legal system has revealed significant variability in their performance. While some models achieve high accuracy on closed questions, their factual consistency and hallucination rates in free-text legal answers remain problematic. This indicates that current LLMs are not yet consistently reliable for complex legal tasks in specific national jurisdictions outside of commonly-studied systems.

The expanding use of LLMs in legal practice, education, and research necessitates robust evaluations of their reliability, especially in diverse national legal frameworks. Demonstrating their capability and mitigating risks like hallucination are critical for their adoption and for maintaining trust in AI-assisted legal processes. This research provides a foundational understanding of LLM limitations in a specific non-US legal context, informing development and deployment strategies.

A benchmark comprising 1,042 items across ten areas of Colombian law and three question formats (closed multiple-choice, semi-open, open-ended IRAC) has been developed through a human-in-the-loop, expert-reviewed pipeline.

Forward consideration, not a verified fact.