Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation

A research study by arXiv, titled 'Critical Acclaim Orientation in Large Language Models: Evidence from Film Preference Elicitation,' investigates whether large language models (LLMs) reproduce evaluative hierarchies present in human judgments, specifically concerning film preferences. The study tested eight LLMs from four families (Anthropic, OpenAI, Alibaba, Mistral) against a 200-film benchmark categorized by critical acclaim, commercial success, or both. The objective was to determine if LLMs reflect popular internet sentiment or embedded critical discourse in their evaluations. The methodology involved 20,000 pairwise forced-choice comparisons.

Why it matters

Understanding the evaluative biases within large language models is crucial for ensuring their reliable and ethical deployment across various applications. If LLMs systematically reproduce critical hierarchies, it has implications for content curation, recommendation systems, and the potential propagation of specific aesthetic or cultural values. This research informs the development of more neutral or intentionally biased AI systems, depending on strategic objectives.

Key insights

• Large language models are trained on corpora containing human judgments across various cultural domains.
• The extent to which LLMs systematically reproduce evaluative hierarchies, such as those related to critical acclaim, is an open research question.
• Prior research suggests competing hypotheses regarding cultural bias in LLMs: mirroring popularity or reproducing prestige from critical discourse.
• The study utilized a 200-film benchmark, divided into critically acclaimed, commercially successful, and dual-legitimacy categories.
• Eight LLMs from four major families (Anthropic, OpenAI, Alibaba, Mistral) were included in the analysis.
• The research methodology involved 20,000 pairwise forced-choice comparisons for film evaluation.
• The study aims to clarify whether LLMs exhibit a 'critical acclaim orientation' in their preferences.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.06955