Moral Competence Before Moral Content: Why LLM Agents Lack the Prerequisites for Coherent Alignment

A recent research paper from arXiv introduces a novel perspective on AI alignment, asserting that current Large Language Model (LLM) agents lack the fundamental 'moral competence' necessary for coherent alignment with human norms or values. The authors argue that before an AI system can adhere to specific moral content, it must first exhibit structural consistency in its decision-making. They propose four structural conditions—verdict stability, monotonicity, decisiveness, and Pareto viability—as measurable prerequisites for coherent policy expression, suggesting these form a 'structural floor' for AI alignment rather than a normative target.

Why it matters

This research is strategically important because it redefines the foundational requirements for trustworthy AI systems, shifting focus from content-specific alignment to structural coherence. Understanding and implementing these 'moral competence' conditions is crucial for developing AI that can operate reliably and predictably, thereby mitigating risks associated with unpredictable or inconsistent AI behavior in critical applications.

Key insights

• AI alignment, which aims for AI systems to adhere to human norms, values, or intentions, faces a foundational challenge.
• The concept of 'moral competence' is introduced as a prerequisite for coherent AI alignment.
• This moral competence is defined by an AI system's ability to express a 'coherent policy'—a consistent mapping from situations to verdicts.
• Four structural conditions for evaluating this coherence are proposed: verdict stability, monotonicity, decisiveness, and Pareto viability.
• These conditions allow for evaluation of an AI's behavior for coherence without needing a specific moral standard or expert baseline.
• The framework identifies a 'structural floor' for alignment, distinct from achieving specific normative targets.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.05036