Reproducibility is not construct validity: LLM measurement of institutionally situated communication

Recent research indicates that while Large Language Model (LLM) annotations of qualitative data can achieve high levels of reproducibility, this does not guarantee their construct validity. An analysis of the European Commission's AI Act consultation data demonstrated that LLM-inferred measures, despite high reproducibility, showed limited convergence with survey-reported measures intended to capture the same construct, particularly varying across different stakeholder groups.

This finding highlights a critical distinction between consistency of measurement (reproducibility) and accuracy of measurement (construct validity) when employing advanced AI tools like LLMs for data analysis. Misinterpreting LLM outputs can lead to flawed insights, impacting strategic decision-making and policy formulation across various domains.

LLM annotations of text-based data exhibited high reproducibility, with intraclass correlations exceeding 0.99, suggesting consistency in measurement.

Forward consideration, not a verified fact.