Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments

Recent research comparing Large Language Models (LLMs) to human responses in persuasion scenarios reveals a significant divergence. While LLMs and humans agree on the strongest persuasion cues, LLMs fail to replicate human belief-updating behavior, showing only slight agreement with human judgments. Humans are more influenced by novel content and assertive language, whereas LLMs prioritize topical similarity and surface-level formatting.

Why it matters

This research highlights a critical limitation of Large Language Models when used as proxies for human behavior in social simulations or decision-making contexts. Understanding this divergence is crucial for applications that rely on modeling human cognitive processes, influencing the design and deployment of AI systems in areas requiring nuanced human-like judgment and response to persuasion.

Key insights

• LLMs exhibit only slight agreement with human judgments on persuasion, with Cohen's kappa ranging from 0.079 to 0.178.
• Humans and LLMs converge on the most impactful persuasion cues.
• A divergence exists at finer levels of persuasion cues.
• Humans are more susceptible to novel content and assertive language in persuasive arguments.
• LLMs are more influenced by topical similarity and surface-level formatting.
• LLMs' ability to update beliefs in response to persuasive arguments is poorly understood and differs significantly from human behavior.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.29803