Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams

Research indicates that Large Language Model (LLM) graders can achieve lower mean absolute error in grading technical exams compared to human graders, presenting a potential solution to scarce qualified grading resources. However, this performance is highly sensitive to prompt design, with specific negative constraints significantly degrading LLM performance or causing complete failure, particularly for open-weight models.

The adoption of AI-driven tools, such as LLM graders, could significantly enhance efficiency and potentially improve consistency in assessment processes where qualified human resources are limited. However, the extreme sensitivity of these tools to specific operational parameters, like prompt design, introduces a critical risk requiring careful management and specialized expertise to prevent system failure or biased outcomes.

LLM graders demonstrated the capability to grade a Computer Vision exam with a mean absolute error of 1.64/35, outperforming the inter-human grader error of 2.61/35 under specific configurations.

Forward consideration, not a verified fact.