Testing Fairness with Utility Tradeoffs: A Wasserstein Projection Approach

Recent research introduces a statistical hypothesis testing framework designed to evaluate fairness in data-driven decision-making while simultaneously considering utility tradeoffs. This framework aims to relax strict fairness requirements, ensuring that overall utility remains above a predefined threshold, and helps ascertain whether observed fairness-utility compromises are inherent to an algorithm or a consequence of its implementation.

Why it matters

This research addresses the inherent tension between fairness and utility in data-driven systems, which is critical for maintaining ethical operations and regulatory compliance while optimizing business outcomes. Organizations can leverage such frameworks to make informed decisions about algorithm deployment, balancing societal impact with operational efficiency and profitability.

Key insights

• Fairness in data-driven decision-making is a critical concern across diverse domains including marketing, lending, and healthcare.
• Achieving fairness often involves a cost to utility, presenting a significant tradeoff.
• A new statistical hypothesis testing framework has been proposed to jointly assess approximate fairness and utility.
• This framework relaxes strict fairness requirements, provided a specified utility threshold is maintained.
• The approach utilizes the strong demographic parity (SDP) criterion and a utility measure based on the potential outcomes framework.
• The test statistic is constructed using Wasserstein projections, enabling auditors to evaluate the origin of fairness-utility tradeoffs.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2505.11678