From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations

A research study investigated the utility of large language models (LLMs) as auxiliary tools for calibrating the difficulty of programming examinations to enhance fairness in course assessments. The study integrated AI evidence with traditional data sources such as student performance, item exposure, and teacher interpretation. Initial findings indicate a strong correlation between LLM performance and student performance, suggesting LLMs can provide valuable insights into exam difficulty.

Why it matters

This research provides a novel approach to leveraging AI as an analytical tool, moving beyond its traditional role as a performance benchmark. It addresses fundamental issues of fairness and operational consistency in assessment design, which is critical for maintaining instructional integrity and public trust in educational outcomes.

Key insights

• Large language models (LLMs) can serve as auxiliary evidence sources for interpreting programming examination difficulty, rather than solely being benchmark evaluation targets.
• The study combined AI evidence with aggregated student performance, item exposure, online-judge process data, and teacher interpretation to assess exam difficulty.
• Ten LLM instances solving an eight-problem final exam concurrently with 120 students showed a positive correlation between AI pass rate and student pass rate (Spearman rho = 0.866, p = 0.0119).
• A solving-based composite difficulty index derived from LLM performance correlated negatively with student pass rate (rho = -0.905, p = 0.0046).
• The methodology indicates a potential for using LLMs to objectively assess and calibrate exam difficulty, contributing to fairer assessment practices.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.07523