Research Summary: CALLIOPE: A Source-Grounded Oral Assessment System and Synthetic Readiness Evaluation

CALLIOPE is a novel source-grounded oral assessment system designed to integrate generative AI into educational evaluation processes. It connects spoken responses to source material, scoring criteria, and AI/human judgment, offering features such as instructional material versioning, learner-turn recording, adaptive questioning, and dual-provider rubric scoring. Initial synthetic evaluations indicate varied scoring performance by the AI component.

Why it matters

The development of systems like CALLIOPE is critical for institutions seeking to leverage AI for automated and scalable assessment, particularly for oral evaluations. This technology has the potential to streamline operational workflows and enhance the consistency and objectivity of assessment processes, influencing resource allocation and educational outcomes.

Key insights

• CALLIOPE is a source-grounded oral assessment system leveraging generative AI for evaluating spoken responses.
• The system integrates versioned instructional materials, learner response recording and transcription, adaptive questioning, and scoring by two AI providers.
• It supports educator review and generates exportable evidence for assessment.
• Synthetic verification records from September 25, 2026, show aggregate AI scores of 100, 62, and 8 out for three spoken fixtures, demonstrating varying performance.
• The system aims to connect spoken responses to source material, scoring criteria, model outputs, and human judgment.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.00290