Research Summary: Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

Research indicates that Large Language Model (LLM) graders can achieve lower mean absolute error in grading technical exams compared to human graders, presenting a potential solution to scarce qualified grading resources. However, this performance is highly sensitive to prompt design, with specific negative constraints significantly degrading LLM performance or causing complete failure, particularly for open-weight models.

Why it matters

The adoption of AI-driven tools, such as LLM graders, could significantly enhance efficiency and potentially improve consistency in assessment processes where qualified human resources are limited. However, the extreme sensitivity of these tools to specific operational parameters, like prompt design, introduces a critical risk requiring careful management and specialized expertise to prevent system failure or biased outcomes.

Key insights

• LLM graders demonstrated the capability to grade a Computer Vision exam with a mean absolute error of 1.64/35, outperforming the inter-human grader error of 2.61/35 under specific configurations.
• The effectiveness of LLM graders is critically dependent on prompt engineering, with a 'strict grader' preamble driving 14 out of 17 open-weight models out of an acceptable grading band (MAE ≥ 8) and three models ceasing to grade entirely.
• Performance degradation in LLMs is linked to specific credit-withholding sentences within the prompt, such as 'never give partial credit', rather than general tone or model scale.
• The study spanned 171 configurations across closed and open-weight models, providing a broad assessment of LLM grading capabilities and vulnerabilities.
• Grading long-form exams in large courses is resource-intensive, requiring significant 'grader-hours' and facing a scarcity of qualified personnel, making LLM alternatives tempting.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.29333