Research Summary: AusClinBench: a deterministically scored benchmark of large language models against current Australian clinical policy

Researchers introduced AusClinBench, a deterministic benchmark evaluating 28 large language models against current Australian clinical policy across 142 verifiable questions. The study identified wide performance variations in blind tests, ranging from 51.5% to 92.7% accuracy across regulatory, billing, and clinical domains, underscoring risks given that clinical software frequently embeds undisclosed models.

Why it matters

This development matters because generative AI tools are already operating within clinical workflows without jurisdiction-specific validation or transparent model disclosure. Without localised benchmarking, healthcare providers risk regulatory non-compliance, financial errors in billing, and compromised patient safety. Establishing deterministic, jurisdiction-specific evaluations provides an objective baseline for clinical AI governance and procurement standards.

Key insights

• AI models are actively deployed in Australian clinical software to draft notes and support clinical decisions, frequently without disclosing the underlying model to practitioners.
• Existing medical evaluation benchmarks primarily reflect foreign jurisdictions rather than Australian clinical, regulatory, and billing frameworks.
• AusClinBench benchmarks 142 exact-match questions across five core domains: PBS authority, MBS billing, Ahpra obligations, triage red flags, and immunisation and screening.
• Testing 28 models between July and October 2026 revealed blind accuracy spanning 51.5% to 92.7%, with top-performing models achieving indistinguishable leading scores.
• Testing both blind and with supplied governing clauses enables the diagnostic separation of rule-recall failures from rule-application failures.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.31993