From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations

A research study investigated the utility of large language models (LLMs) as auxiliary tools for calibrating the difficulty of programming examinations to enhance fairness in course assessments. The study integrated AI evidence with traditional data sources such as student performance, item exposure, and teacher interpretation. Initial findings indicate a strong correlation between LLM performance and student performance, suggesting LLMs can provide valuable insights into exam difficulty.

• Large language models (LLMs) can serve as auxiliary evidence sources for interpreting programming examination difficulty, rather than solely being benchmark evaluation targets.
• The study combined AI evidence with aggregated student performance, item exposure, online-judge process data, and teacher interpretation to assess exam difficulty.
• Ten LLM instances solving an eight-problem final exam concurrently with 120 students showed a positive correlation between AI pass rate and student pass rate (Spearman rho = 0.866, p = 0.0119).
• A solving-based composite difficulty index derived from LLM performance correlated negatively with student pass rate (rho = -0.905, p = 0.0046).
• The methodology indicates a potential for using LLMs to objectively assess and calibrate exam difficulty, contributing to fairer assessment practices.

Reported by arXiv — Computers and Society · International. This briefing summarises the publication for executive use; the document itself is not reproduced here.