Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments

Recent research comparing Large Language Models (LLMs) to human responses in persuasion scenarios reveals a significant divergence. While LLMs and humans agree on the strongest persuasion cues, LLMs fail to replicate human belief-updating behavior, showing only slight agreement with human judgments. Humans are more influenced by novel content and assertive language, whereas LLMs prioritize topical similarity and surface-level formatting.

This research highlights a critical limitation of Large Language Models when used as proxies for human behavior in social simulations or decision-making contexts. Understanding this divergence is crucial for applications that rely on modeling human cognitive processes, influencing the design and deployment of AI systems in areas requiring nuanced human-like judgment and response to persuasion.

LLMs exhibit only slight agreement with human judgments on persuasion, with Cohen's kappa ranging from 0.079 to 0.178.

Forward consideration, not a verified fact.