Who Do Language Models Think Is Competent? A Mechanistic Analysis of Occupational Bias

Research from arXiv highlights that while language models may not overtly display behavioral biases, they often retain internal, representational biases, particularly concerning occupational competence. These internal biases, detectable even when surface-level biases are suppressed, can causally influence model behavior in tasks such as question-answering and hiring, necessitating deeper analysis of AI fairness beyond behavioral metrics.

Why it matters

This research is strategically important as it exposes a fundamental challenge in AI development: the persistence of latent biases within models that are otherwise deemed 'fair' by traditional behavioral tests. Addressing these underlying representational biases is critical for ensuring the ethical deployment and trustworthiness of AI systems, particularly in sensitive applications like talent management or information dissemination.

Key insights

• Language models often retain underlying representational biases, even when they pass behavioral bias evaluations.
• A causal framework was introduced to decompose occupational bias into internal representations of competence and observable outputs.
• Steering vectors for user expertise representations were derived and verified to causally mediate model behavior.
• Representational biases were found to influence model performance in both question-answering and hiring tasks.
• The study confirms that detecting internal biases is possible, even when external behavioral biases are not visible in open-weight models.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.20347