Research Summary: Silicon sampling answers with country-level assumptions, not individual attitudes: Cross-national evidence from the European Social Survey

Research from arXiv investigates the effectiveness of using large language models (LLMs) for 'silicon sampling' to simulate survey respondents, particularly in recovering cross-national variations. The study, based on data from the European Social Survey across 30 countries, found that LLMs moderately and unevenly recovered aggregate survey results. A critical finding was that including the respondent's country name in the prompt significantly improved the correlation between simulated and observed country means, suggesting that LLMs primarily leverage country-level assumptions rather than detailed individual profiles to generate responses.

Why it matters

This research highlights the current limitations and mechanisms of LLMs in simulating complex human attitudes and societal trends. Understanding how LLMs derive their 'answers' is crucial for organisations considering their use for market research, policy impact assessment, or forecasting, as it influences the reliability and interpretability of such simulations.

Key insights

• Large language models can simulate survey respondents through 'silicon sampling,' but their ability to accurately replicate cross-national variations is moderate and inconsistent.
• Aggregate recovery of survey responses by LLMs is uneven across different survey items.
• Adding a country name to a demographic prompt significantly boosts the correlation between simulated and observed country means, raising it from -0.03 to 0.52.
• Richer demographic profiles beyond the country label did not consistently improve the accuracy of simulated responses.
• LLMs appear to rely on pre-existing country-level assumptions when generating responses, rather than synthesizing detailed individual respondent information.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.16395