Evaluating Decision Models for Text Annotation in Computational Social Science

Computational social science research increasingly relies on large language models for text annotation, making the validity of findings dependent on these models' outputs. A new class of models, 'decision models,' designed for categorical question answering, offers structured responses and confidence scores at reduced inference prices. However, their accuracy and the trustworthiness of their stated confidence for social science constructs are currently unverified, necessitating rigorous evaluation.

The increasing reliance on AI for data annotation in critical fields like computational social science means that the integrity and reliability of research outcomes are directly tied to the performance of these models. Understanding the accuracy and cost-effectiveness of new model classes, such as decision models, is crucial for strategic resource allocation, risk management, and ensuring the credibility of data-driven insights across various domains.

Computational social science research is highly dependent on large language models (LLMs) for text annotation.

Forward consideration, not a verified fact.