ai
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
- Source
- arXiv — Computers and Society
- Published
- Last verified
- 18 Aug 2026
- Confidence
- High
- Evidence
- Original document retained
- Reading time
- 1 min
- Country
- International
- Relevant to
- Research & Evidence, Technology & Data, Risk & Compliance
- Topics
- airesearchdatacompliance
Executive summary
What happened, and why should leadership care?
A research study critically examines the use of Large Language Models (LLMs) as synthetic survey respondents, moving beyond mere plausibility to evaluate their psychometric validity. The study posits that LLMs must accurately replicate the joint distribution, latent structure, reliability, mediation pathways, and demographic effects observed in real human survey data. Using a Lithuanian organizational-psychology dataset, a comprehensive lineup of LLMs was tested under various conditions to assess their ability to preserve these psychometric properties.
Why this matters
Why is this strategically important?
The increasing reliance on LLMs for data generation, including synthetic survey responses, necessitates rigorous validation to ensure the reliability and integrity of insights derived. If LLMs fail to accurately replicate the psychometric characteristics of human data, strategic decisions based on such synthetic data could be flawed, leading to misallocated resources or ineffective initiatives.
Key insights
What should be noted from the evidence?
- Existing evaluations of LLMs as synthetic survey respondents often focus on individual-level answer plausibility rather than psychometric validity.
- The appropriate evaluative question for LLMs as survey respondents is whether they preserve key psychometric properties of real human data.
- These properties include joint distribution, latent structure, reliability, mediation pathways, and demographic effects.
- A study utilized a Lithuanian organizational-psychology dataset (n=263 employees, 68 items across 12 subscales) for evaluation.
- A diverse lineup of 37 LLMs (OpenAI, Anthropic, Google, and open-weight models) was tested.
Evidence and confidence
How far can this assessment be trusted?
High confidence. Named institution, original document retained and analysis corroborated.
Analysis is prepared editorially by Aziz Shuaib Ausi. The original publication remains the authoritative record, and executive judgement remains entirely human.
Source
Where does this originate?
Reported by arXiv — Computers and Society · International. This briefing summarises the publication for executive use; the document itself is not reproduced here.
Read the original publication