Research Summary: Verdict Without the Rule: Diagnosing and Auditing Regulatory Rule Sensitivity in LLM Compliance Systems

Empirical research auditing large language model (LLM) compliance systems across five models and 20 regulatory domains reveals that model verdicts are largely invariant to supplied regulatory rules, frequently failing to shift even when rules are deleted, swapped, or negated.

Why it matters

This matters because enterprises relying on automated LLMs for compliance monitoring may operate under the flawed assumption that system decisions faithfully enforce designated rules. When models deliver static verdicts detached from specific policy directives, organizations face hidden regulatory breaches and legal exposure.

Key insights

• Automated LLM compliance systems often issue verdicts independent of the specific regulatory or platform rules provided.
• Perturbations such as deleting, swapping, or negating the governing rule while holding the case fixed resulted in minimal changes to final verdicts (OCS) or internal representations (ICS-delta).
• A dedicated guard model evaluated under a custom-rule adaptation demonstrated the lowest rule sensitivity and achieved accuracy barely above chance at 51%, compared to 90-92% for general-purpose models.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.12313