Research Summary: GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models

Researchers have developed 'GYROval,' a new benchmark designed to measure cultural value orientation in Large Language Models (LLMs) across two Inglehart-Welzel axes, multiple domains, and roles. Administered to twenty models, this benchmark uses binary contrastive scenarios where neither option is definitively correct, providing a robust method for assessing cultural alignment. A subset of these models was also tested with Russian translations and varied sampling temperatures, with the instrument now publicly available in both English and Russian.

Why it matters

The development and application of GYROval are critical for understanding how Large Language Models embody or deviate from specific cultural value orientations. This insight is essential for the responsible deployment of AI systems, particularly in global or culturally sensitive contexts, ensuring their outputs align with intended societal norms and expectations.

Key insights

• A new benchmark, GYROval, has been developed to robustly measure cultural value orientation in Large Language Models (LLMs).
• The benchmark assesses LLMs against the two Inglehart-Welzel cultural axes, covering several domains and roles.
• It utilizes binary contrastive scenarios, where both options are legitimate and no single 'correct' answer exists.
• The score for an LLM on an axis is determined by the proportion of its responses aligning with a specific cultural pole.
• Twenty LLMs were evaluated using GYROval.
• Eleven of these models were also tested using a Russian translation of the items and with a different sampling temperature.
• The GYROval instrument is publicly released in both English and Russian.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.18384