Research Summary: Who Brought Easter Eggs to Eid? Auditing LLM-Generated Cultural Translation of Math Word Problems Across Languages and Regions

Research evaluating advanced large language models—including Claude Opus 4, GPT-4.1, and Gemini 2.5 Pro—demonstrates the complexities of automated cultural adaptation in educational materials. Auditing 6,489 entity transformations across 60 math word problems translated into seven languages spanning India, Pakistan, and Italy reveals critical inconsistencies in how models preserve, localize, or alter cultural entities across high- and low-resource linguistic contexts. Note that the research abstract is truncated, limiting visibility into full empirical findings.

Why it matters

As generative AI systems are adopted for global personalized education at scale, unvetted automated translation risks introducing cultural incongruities that degrade pedagogical efficacy. Systemic failures in cross-cultural entity localization can alienate regional user bases and undermine institutional credibility. Ensuring equitable performance across both high-resource and under-studied languages is vital for mitigating cultural homogenization in AI deployment.

Key insights

• Claude Opus 4, GPT-4.1, and Gemini 2.5 Pro were benchmarked on their ability to culturally translate 60 English math word problems.
• The linguistic testbed spans high-resource and under-studied languages across India, Pakistan, and Italy, including Bengali, Hindi, Punjabi, Urdu, Sindhi, Italian, and Sicilian.
• Researchers annotated 6,489 entity transformations, categorizing model behaviors into preservation, localization, generalization, omission, and alteration.
• The study investigates whether automated personalization in educational tools maintains cultural diversity or introduces inappropriate cultural substitutions.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2606.11009