Research Summary: Towards Detecting AI-Assisted Responses in Online Surveys

Research indicates that the increasing use of Large Language Models (LLMs) to complete online surveys poses a significant threat to the validity of survey-based data. While basic AI usage in survey responses is detectable, more sophisticated, persona-grounded AI agents can mimic human respondents effectively, making their detection challenging for current machine-generated text (MGT) detectors. This highlights a growing vulnerability in data collection methodologies.

Why it matters

The integrity of data collected through online surveys is fundamental for informed decision-making across various sectors, from market research to policy formulation and scientific study. The difficulty in distinguishing sophisticated AI-generated responses from genuine human input introduces substantial risk to data reliability, potentially leading to flawed analyses, misguided strategies, and misallocated resources based on compromised information.

Key insights

• The proliferation of LLMs in completing online surveys compromises the validity of research outcomes derived from such surveys.
• A new benchmark dataset, ASURRE, has been developed to capture various AI-assisted survey participation strategies, including full generation, revision, and persona-grounded agentic completion.
• LLM-assisted responses were generated across multiple LLMs and three diverse real-world surveys, alongside genuine human responses, to form the dataset.
• Current machine-generated text (MGT) detectors are effective at identifying straightforward or 'naive' AI usage in surveys.
• However, these detectors struggle significantly, performing close to random chance, when faced with persona-grounded AI agents designed to emulate human respondents.
• The findings suggest a need for more advanced detection methods to safeguard the integrity of survey data.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.17317