Research Summary: Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent

Research identifies a significant issue of 'instability floors' in fairness audits of clinical language-model agents, where a substantial portion of reported flip rates (changes in agent actions) are attributable to stochastic noise rather than demographic bias. This intrinsic variability, observed even when no patient descriptors change, complicates the accurate assessment of fairness and operational reliability in these critical systems. The study measured this noise, finding it can account for a considerable percentage of action changes, varying by clinical task and LLM vendor.

Why it matters

The inherent instability observed in clinical language-model agents complicates the reliable assessment of fairness and operational consistency. Understanding and quantifying this stochastic noise is critical for developing robust audit methodologies and ensuring that clinical AI systems perform predictably and equitably across diverse patient populations. This directly impacts trust, regulatory compliance, and the safe deployment of AI in sensitive domains.

Key insights

• Fairness audits using counterfactual methods report a 'flip rate' based on changes in agent actions when patient demographic descriptors are altered.
• A portion of this reported flip rate is not due to demographic factors but to the stochastic nature of the agent itself, where actions change even with no input variation.
• Measuring this 'instability floor' is crucial for interpreting flip rates accurately, as it separates inherent noise from actual bias.
• Re-running a single condition multiple times (ten times over sixteen synthetic vignettes) resulted in an 8.7% action change in replicate pairs for one clinical agent, highlighting internal variability.
• The instability varied by clinical task, from 2.2% for intensive-care escalation to 17.9% for controlled-substance caution, where operational criteria were absent.
• Across six different LLM models from five vendors, the pooled instability floors ranged significantly, from 2.5% to 23.7%.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.03221