A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation

Research from arXiv introduces a new framework for evaluating the trustworthiness of Large Language Models (LLMs) when used as 'judges' in principle-based regulation, particularly within finance. The paper proposes a four-axis benchmark: accuracy, paraphrase robustness, adversarial robustness, and calibration. To support this, 'Principle-Bench' has been released, comprising 168 cryptoasset financial-promotion scenarios aligned with two UK FCA principles, designed to test LLMs across these four axes. This initiative highlights the critical need for robust and auditable evaluation methods for AI deployed in regulatory compliance.

• Principle-based regulation, characterized by evaluative standards such as 'fair, clear, and not misleading' or 'deliver good outcomes', is increasingly relying on LLMs as adjudicators.
• A comprehensive evaluation of LLM-as-judge requires assessment across four key axes: accuracy, paraphrase robustness, adversarial robustness, and calibration.
• The 'Principle-Bench' dataset provides 168 cryptoasset financial-promotion scenarios, mapped to two UK FCA principles, to facilitate this four-axis evaluation.
• Principle-Bench includes designed perturbations such as paraphrasing, adversarial keyword-stuffing, and boundary perturbations, authored under a pre-registered rubric.
• The research also introduces 'Ceca' (Calibrated Exemplar-Cluster Assessment), described as a calibrated and auditable assessment method for LLM judges.

Reported by arXiv — Computers and Society · International. This briefing summarises the publication for executive use; the document itself is not reproduced here.