Research Summary: Incremental Predictive Value of LLM Tutoring Interactions for Student Performance

An empirical study evaluating 16,851 LLM tutoring interactions across undergraduate artificial intelligence cohorts reveals that interaction features provide minimal incremental predictive power for assignment scores over standard baselines. Using repeated nested cross-validation, the best-performing model (Gradient Boosting) reduced mean absolute error by only 0.114 grade points relative to a simple median baseline, with confidence intervals spanning zero. The findings indicate that dialogue telemetry from conversational AI tools may offer limited predictive utility for student performance forecasting without substantial feature engineering.

Why it matters

This study matters because it tempers prevailing industry assumptions that conversational AI interaction logs inherently provide actionable predictive intelligence on user capability. For educational and training institutions, it highlights that large-scale telemetry from AI assistants does not automatically translate into reliable early-warning performance metrics. Strategic planners must critically scrutinize vendor claims regarding the prognostic utility of AI interaction datasets before allocating resources to predictive monitoring systems.

Key insights

• Analysis of 16,851 student-assistant exchanges from 175 matched undergraduate student records showed marginal predictive enhancement over constant baselines.
• Gradient Boosting delivered a mean absolute error (MAE) of 3.023 grade points, compared to 3.138 grade points for the training-fold median baseline.
• The paired difference in prediction error was -0.114 grade points, with a conditional 95% bootstrap confidence interval of [-0.245, 0.013], indicating statistically insignificant predictive gains.
• Model evaluation across six fitted estimators and repeated nested cross-validation highlighted the difficulty of extracting actionable academic performance signals directly from semester-level LLM interactions.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.33488