When Post-Processing Fairness Constraints Help and When They Harm: Evidence from Eight Cross-Domain Evaluations

Research indicates that current machine learning (ML) fairness auditing practices, which often involve a single audit at deployment within a single domain, are insufficient. Fairness can deteriorate over time due to model retraining or evolving user bases, and interventions validated in one context frequently fail to generalize across diverse domains. A new framework, FAPE (Fairness Auditing for Production Environments), has been developed and evaluated using Fairlearn's ThresholdOptimizer across eight distinct domains to address these challenges and assess intervention effectiveness more comprehensively.

The findings highlight critical deficiencies in current machine learning fairness auditing practices, which pose significant reputational, ethical, and regulatory risks for organizations deploying AI systems. Implementing robust, continuous, and multi-domain fairness evaluation frameworks is crucial for maintaining public trust, ensuring equitable outcomes, and mitigating potential liabilities associated with algorithmic bias across various operational areas.

Existing ML fairness audits, typically conducted once at deployment and in a single domain, are insufficient for production environments.

Forward consideration, not a verified fact.