ai
A Framework for Using and Evaluating LLMs as Surrogate Experts in Security Surveys: Reliability, Bias, and Implications
- Source
- arXiv — Computers and Society
- Published
- Last verified
- 19 Aug 2026
- Confidence
- High
- Evidence
- Original document retained
- Reading time
- 1 min
- Country
- International
- Relevant to
- Operations & Delivery, People & Capability, Research & Evidence, Policy & Regulation, Strategy & Planning, Technology & Data
Executive summary
What happened, and why should leadership care?
This research introduces a methodological framework for evaluating Large Language Models (LLMs) as surrogate expert participants in security surveys. It addresses the challenges of recruiting human experts in fields like Security Operations Centres (SOCs) due to high workload and confidentiality, proposing LLMs as a potential scalable alternative. The framework aims to provide guidance on the reliability and utility of LLMs for generating synthetic survey responses.
Why this matters
Why is this strategically important?
The development of reliable methodologies for leveraging AI, specifically LLMs, to simulate expert input has broad implications for research, data collection, and decision-making processes across various domains. It offers a potential solution to resource constraints and scalability issues in expert-dependent analyses, enabling more comprehensive and timely insights for strategic planning and operational improvements.
Key insights
What should be noted from the evidence?
- Recruiting human experts for security research surveys, particularly in Security Operations Centres (SOCs), is difficult due to high workload, burnout, and confidentiality constraints, often leading to small sample sizes.
- Large Language Models (LLMs) present an appealing alternative for generating synthetic survey responses at scale.
- There is currently limited guidance on the reliability of using LLMs as surrogate expert participants.
- The presented framework evaluates LLMs as substitutes or supplements to human expert survey respondents.
- The evaluation compares persona-based and aggregate LLM-generated answers across various models and prompting settings, using responses from SOC professionals as a baseline.
Evidence and confidence
How far can this assessment be trusted?
High confidence. Named institution, original document retained and analysis corroborated.
Analysis is prepared editorially by Aziz Shuaib Ausi. The original publication remains the authoritative record, and executive judgement remains entirely human.
Source
Where does this originate?
Reported by arXiv — Computers and Society · International. This briefing summarises the publication for executive use; the document itself is not reproduced here.
Read the original publication