Silicon sampling answers with country-level assumptions, not individual attitudes: Cross-national evidence from the European Social Survey

Research from arXiv investigates the effectiveness of using large language models (LLMs) for 'silicon sampling' to simulate survey respondents, particularly in recovering cross-national variations. The study, based on data from the European Social Survey across 30 countries, found that LLMs moderately and unevenly recovered aggregate survey results. A critical finding was that including the respondent's country name in the prompt significantly improved the correlation between simulated and observed country means, suggesting that LLMs primarily leverage country-level assumptions rather than detailed individual profiles to generate responses.

This research highlights the current limitations and mechanisms of LLMs in simulating complex human attitudes and societal trends. Understanding how LLMs derive their 'answers' is crucial for organisations considering their use for market research, policy impact assessment, or forecasting, as it influences the reliability and interpretability of such simulations.

Large language models can simulate survey respondents through 'silicon sampling,' but their ability to accurately replicate cross-national variations is moderate and inconsistent.

Forward consideration, not a verified fact.