Research Summary: Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems

Public institutions are increasingly procuring proprietary AI systems, such as Early Warning Systems (EWS) in higher education, where they lack the ability to inspect or modify the system's design. This situation necessitates the use of post-hoc fairness interventions to address inequities, raising questions about the coordination of fairness efforts among various stakeholders with unequal power dynamics. Research evaluating six such interventions on a simulated EWS in a Canadian public college identified how these interventions impact fairness, accuracy, and demographic disparities, and redistribute false positives and false negatives.

Why it matters

The proliferation of opaque AI systems in public sectors introduces significant governance and ethical challenges, particularly regarding fairness and accountability. Institutions must understand the limitations and potential impacts of proprietary AI to develop effective strategies for risk mitigation and equitable deployment. This research highlights the critical need for robust evaluation frameworks and clearer power dynamics in the procurement and management of AI technologies.

Key insights

• Public institutions are acquiring AI systems whose internal design and operation cannot be inspected or altered by the procuring entity.
• In higher education, proprietary Early Warning Systems (EWS) force institutions to rely on post-hoc adjustments to model outputs to address fairness concerns.
• Fairness work in this context involves coordination among vendors, institutions, advisors, and students, who possess unequal power to influence these systems.
• Evaluation of post-hoc fairness interventions on a research EWS revealed their impact on fairness metrics, system accuracy, and demographic disparities.
• The interventions were analyzed using error-type profiling to track the redistribution of false positives and false negatives across different groups.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.38552