Research Summary: Reproducibility is not construct validity: LLM measurement of institutionally situated communication

Recent research indicates that while Large Language Model (LLM) annotations of qualitative data can achieve high levels of reproducibility, this does not guarantee their construct validity. An analysis of the European Commission's AI Act consultation data demonstrated that LLM-inferred measures, despite high reproducibility, showed limited convergence with survey-reported measures intended to capture the same construct, particularly varying across different stakeholder groups.

Why it matters

This finding highlights a critical distinction between consistency of measurement (reproducibility) and accuracy of measurement (construct validity) when employing advanced AI tools like LLMs for data analysis. Misinterpreting LLM outputs can lead to flawed insights, impacting strategic decision-making and policy formulation across various domains.

Key insights

• LLM annotations of text-based data exhibited high reproducibility, with intraclass correlations exceeding 0.99, suggesting consistency in measurement.
• Despite high reproducibility, LLM-inferred measures demonstrated limited convergence with independently collected survey-reported measures of the same nominal construct.
• The divergence between LLM-inferred and survey-reported text-based measures varied systematically across different stakeholder groups.
• Business associations, for example, expressed greater concern about AI risks in their text-based consultation submissions compared to their survey responses, according to the LLM analysis.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.19866