Are LLMs becoming similarly creative? Evidence from three years of models

Research is underway to assess the evolving performance of Large Language Models (LLMs) on open-ended tasks, specifically focusing on creativity, originality, and diversity, rather than just verifiable answers. A preliminary analysis examines LLM outputs over three years, using real-world user queries (Infinity-Chat100) and a psychometric creativity assessment (Alternate Uses Task) to understand trends in their creative capabilities.

Why it matters

As Large Language Models increasingly integrate into processes requiring ideation and creative outputs, understanding their evolving capabilities in originality, diversity, and creativity is paramount. This insight enables strategic planning for future human-AI collaboration models and innovation pipelines, ensuring technology adoption aligns with organizational creative objectives.

Key insights

• Traditional LLM benchmarks often focus on tasks with verifiable answers, overlooking performance on open-ended challenges.
• Understanding LLM performance trends in open-ended tasks is critical due to their growing role in human ideation and creative work.
• A preliminary analysis is being conducted on LLM creative outputs from models released over a three-year period.
• The study utilizes a real-world dataset of open-ended user queries (Infinity-Chat100) and the psychometric Alternate Uses Task.
• Sentence-embedding similarity is being used as a methodology to examine trends in LLM responses.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.19437