A Framework for Using and Evaluating LLMs as Surrogate Experts in Security Surveys: Reliability, Bias, and Implications

This research introduces a methodological framework for evaluating Large Language Models (LLMs) as surrogate expert participants in security surveys. It addresses the challenges of recruiting human experts in fields like Security Operations Centres (SOCs) due to high workload and confidentiality, proposing LLMs as a potential scalable alternative. The framework aims to provide guidance on the reliability and utility of LLMs for generating synthetic survey responses.

Why it matters

The development of reliable methodologies for leveraging AI, specifically LLMs, to simulate expert input has broad implications for research, data collection, and decision-making processes across various domains. It offers a potential solution to resource constraints and scalability issues in expert-dependent analyses, enabling more comprehensive and timely insights for strategic planning and operational improvements.

Key insights

• Recruiting human experts for security research surveys, particularly in Security Operations Centres (SOCs), is difficult due to high workload, burnout, and confidentiality constraints, often leading to small sample sizes.
• Large Language Models (LLMs) present an appealing alternative for generating synthetic survey responses at scale.
• There is currently limited guidance on the reliability of using LLMs as surrogate expert participants.
• The presented framework evaluates LLMs as substitutes or supplements to human expert survey respondents.
• The evaluation compares persona-based and aggregate LLM-generated answers across various models and prompting settings, using responses from SOC professionals as a baseline.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.16893