ParaStudent: Closing the Sim2Real Gap in User Simulators for AI Tutor Evaluation

Research introduces ParaStudent, a novel fine-tuning framework designed to simulate novice programming revisions. This framework aims to bridge the gap between simulated and real student engagement data, enabling more effective pre-deployment evaluation of Artificial Intelligence (AI) tutor feedback. ParaStudent's simulations demonstrate a closer alignment with actual student code distributions in functional, stylistic, and semantic aspects compared to existing baseline methods. Its performance in predicting feedback relevance and successful uptake surpasses prompted baselines.

Why it matters

This development is strategically important as it enhances the reliability and efficiency of evaluating AI tutor systems before they are deployed. By providing more accurate simulated student engagement, it reduces the risks associated with unvalidated AI educational tools and accelerates their development and refinement. This can lead to more effective educational technologies and improved learning outcomes.

Key insights

• The ParaStudent framework fine-tunes simulations of novice programming revisions for AI tutor evaluation.
• ParaStudent's simulated revisions more closely match real student code distributions across functional, stylistic, and semantic metrics than prompted baselines.
• The framework helps anticipate student engagement for pre-deployment evaluation of AI tutor feedback.
• ParaStudent achieved Area Under the Curve (AUC) scores of 0.80 for both feedback relevance and successful uptake.
• Prompted baselines showed near-chance performance for successful uptake, indicating a significant improvement with ParaStudent.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2507.12674