CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks

A new research framework, CentaurBench, has been introduced to evaluate Large Language Models (LLMs) based on their capacity to both automate tasks and augment the performance of other agents. Unlike traditional benchmarks that focus solely on automation, this framework assesses how LLMs enhance the output of a lower-capacity worker model across seven real-world tasks, alongside their direct automation capabilities. This shift in evaluation considers the practical application of LLMs as assistants, providing a more nuanced understanding of their utility in collaborative work environments.

Why it matters

This research introduces a novel perspective on evaluating artificial intelligence capabilities, moving beyond mere automation to assess augmentation potential. Understanding how LLMs enhance the output of other agents is critical for strategic deployment and integration of AI in complex operational environments, influencing investment decisions and development priorities for AI-driven solutions.

Key insights

• Most existing LLM benchmarks primarily assess models based on their ability to automate work tasks.
• In practical applications, LLMs frequently function as assistants, augmenting the performance of human or other LLM agents.
• The CentaurBench framework evaluates an LLM's capability to both automate tasks directly and augment the performance of a 'lower-capacity worker model'.
• Evaluation is conducted across seven economically grounded real-world tasks.
• In the augmentation mode, an 'assistant model' generates assistance text for a 'standardized lower-capacity worker model' to produce a deliverable.
• In the automation mode, the 'assistant model' produces the output directly.
• Outputs are scored using blind pairwise comparisons conducted by an LLM.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.18554