A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation

Research from arXiv introduces a new framework for evaluating the trustworthiness of Large Language Models (LLMs) when used as 'judges' in principle-based regulation, particularly within finance. The paper proposes a four-axis benchmark: accuracy, paraphrase robustness, adversarial robustness, and calibration. To support this, 'Principle-Bench' has been released, comprising 168 cryptoasset financial-promotion scenarios aligned with two UK FCA principles, designed to test LLMs across these four axes. This initiative highlights the critical need for robust and auditable evaluation methods for AI deployed in regulatory compliance.

Why it matters

The increasing reliance on AI, specifically LLMs, for complex regulatory interpretations presents both opportunities and significant risks. Establishing robust, multi-dimensional benchmarks for AI trustworthiness is crucial to ensure fairness, transparency, and compliance, thereby mitigating potential legal and reputational exposures in regulated sectors.

Key insights

• Principle-based regulation, characterized by evaluative standards such as 'fair, clear, and not misleading' or 'deliver good outcomes', is increasingly relying on LLMs as adjudicators.
• A comprehensive evaluation of LLM-as-judge requires assessment across four key axes: accuracy, paraphrase robustness, adversarial robustness, and calibration.
• The 'Principle-Bench' dataset provides 168 cryptoasset financial-promotion scenarios, mapped to two UK FCA principles, to facilitate this four-axis evaluation.
• Principle-Bench includes designed perturbations such as paraphrasing, adversarial keyword-stuffing, and boundary perturbations, authored under a pre-registered rubric.
• The research also introduces 'Ceca' (Calibrated Exemplar-Cluster Assessment), described as a calibrated and auditable assessment method for LLM judges.
• This benchmark is presented as the first to cover all four trustworthiness axes specifically for principle-based regulation.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.14329