The PIMMUR Principles: Ensuring Validity in Collective Behavior of LLM Societies

A systematic audit of 350 recent research papers, encompassing 576 studies on Large Language Model (LLM)-based social simulations, reveals significant methodological gaps in ensuring the validity of these simulations. The audit, based on the PIMMUR principles (Profile, Interaction, Memory, Minimal-Control, Unawareness, Realism), found that while aspects of agent profile, interaction, and memory were frequently addressed, considerations for minimal control, unawareness, and realism were often overlooked. This suggests that claims of human-like behavior in LLM simulations remain largely unsubstantiated, with a notable portion of frontier LLMs identifying underlying social experiments and prompts imposing constraints.

This research highlights critical methodological shortcomings in the use of LLMs for simulating collective human behavior, challenging the validity of current applications and findings. For any organization leveraging or considering LLM-based simulations for strategic decision-making, policy development, or understanding market dynamics, these findings underscore the need for rigorous validation and awareness of potential biases or limitations inherent in the simulation design.

576 LLM-based social simulation studies across 350 papers were systematically audited.

Forward consideration, not a verified fact.