Research Summary: Language-model groups overstate consensus when replaying human deliberation on a reasoning task

Research comparing human and large language model (LLM) groups on a reasoning task found that LLM agent groups consistently exhibited higher rates of consensus than human groups, irrespective of how consensus was defined or participation levels were matched. This overstatement of consensus by LLMs appears to be influenced by their higher participation rates compared to humans.

Why it matters

This research highlights a critical difference in collaborative behavior and outcome between human and AI-driven groups, particularly concerning consensus formation. Understanding these dynamics is vital for organizations deploying AI in decision-making, collaborative environments, or simulations, as AI might present an artificially high level of agreement, potentially masking underlying complexities or dissenting views.

Key insights

• Human group full-consensus rates on a Wason reasoning task ranged from 24.0% to 57.0% depending on scoring definitions.
• Approximately one-fifth of human participants did not contribute to discussions, whereas LLM agents almost always posted.
• LLM agent groups demonstrated significantly higher consensus rates than human groups, with gaps of 34.0 to 43.9 percentage points in comparison analyses.
• The operationalization of participation and final states influences the observed consensus rates in both human and LLM groups.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.20543