Proxy reliance in large language model decisions is uncalibrated to predictive evidence

Research indicates that Large Language Models (LLMs) demonstrate uncalibrated reliance on proxy attributes when making decisions, specifically failing to align their reliance with the actual predictive evidence. This issue is identified in a clinical-ranking task, raising concerns about LLMs' deployment in sensitive areas such as triage and lending where distinguishing between legitimate inference and impermissible proxy use is critical.

The uncalibrated reliance of Large Language Models on proxy attributes poses significant risks for deploying AI in critical decision-making processes. This highlights a fundamental challenge in ensuring fairness and accuracy, particularly where decisions impact individuals or allocate resources. Addressing this issue is crucial for maintaining trust and avoiding unintended biases in AI-driven systems across various sectors.

LLMs are increasingly used in decision-making contexts like triage and lending, necessitating a clear distinction between task-relevant inference and impermissible proxy use.

Forward consideration, not a verified fact.