Research Summary: LLM Persuasion Is in the Eye of the Evaluation

Research indicates that large language models (LLMs) can match or exceed human experts in persuasion, posing dual-use implications across beneficial sectors like education and health as well as risks involving manipulation and misinformation. However, evaluation of these capabilities remains fragmented across inconsistent definitions and narrow contexts, prompting the development of automated, scalable assessment methods to evaluate high-risk persuasion.

Why it matters

This issue is critical because unstandardized persuasion metrics leave decision-makers and regulators with incomplete visibility into the true influence and manipulation potential of deployed models. Without reliable and scalable evaluations, institutions risk deploying persuasive AI systems in sensitive contexts without understanding their capacity for distortion or harm. Establishing rigorous benchmarks is necessary to govern dual-use generative systems before widespread societal exposure occurs.

Key insights

• Large language models match or surpass human experts in persuasive capability.
• Persuasive capabilities have constructive applications in education and health communication, alongside threats of manipulation and misinformation.
• Current evaluations of LLM persuasion are fragmented, with divergent definitions and broad claims derived from narrow, context-specific studies.
• Automated testing methods derived from human studies enable scalable comparison and safe evaluation of high-risk persuasion scenarios that cannot ethically be tested on people.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.10232