Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models

Research from arXiv explores the potential of Large Language Models (LLMs) to simulate human survey responses, leveraging their training on extensive human-generated data to mimic attitudes and behaviors. While LLMs offer a promising 'silicon sample' for gauging public opinion, informing policy, and generating social scientific data, the analysis highlights significant unresolved issues concerning social biases, generalization, and technical limitations. The broad design space for such simulations further complicates their reliable application.

Why it matters

The ability of LLMs to simulate human responses represents a potential paradigm shift in data collection and analysis for policy development and strategic planning. Successfully addressing the identified challenges could enable faster, more cost-effective insights, while failure to mitigate biases or technical limitations could lead to flawed decision-making and misallocation of resources.

Key insights

• LLMs are capable of encoding human attitudes and behaviors due to training on vast human-generated data.
• They show promise as 'silicon samples' for simulating people in survey responses.
• Potential applications include establishing public opinion, designing policies, and generating social scientific data.
• Critical challenges include social biases, generalization capabilities, and inherent technical limitations of LLMs.
• A wide design space for simulations necessitates methods like multiverse analyses to understand the impact of design choices.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.10280