Research Summary: Can a System-One LLM Perform Knowledge Tracing When Few or No Learners Are Logged?

Research demonstrates that an off-the-shelf System-One large language model (LLM), which predicts probabilities directly in a single pass, can outperform traditional deep knowledge tracing (KT) models and complex System-Two LLMs in cold-start environments with zero learner data. Tested across seven datasets, this single-pass model achieved superior predictive accuracy at approximately one percent of the API cost of multi-sample reasoning alternatives, offering a viable solution to the cold-start problem in educational platforms despite the source abstract being truncated.

Why it matters

This research eliminates the operational cold-start bottleneck for digital education platforms by enabling accurate learner assessment without prior engagement data. It demonstrates that lightweight, single-pass probabilistic models can surpass costly multi-step reasoning architectures, fundamentally lowering the barrier to entry for deploying adaptive learning algorithms. Consequently, organizations can deliver immediate personalization while reducing technical overhead and API expenditures by two orders of magnitude.

Key insights

• Traditional knowledge tracing models suffer from severe cold-start limitations, requiring extensive learner logs before becoming functional on new platforms or courses.
• System-Two LLM approaches that rely on fine-tuning or generating and voting across ten samples are computationally slow, produce coarse probabilities, and carry high API costs.
• A zero-data System-One LLM ('Jev') achieved a mean AUC of .706 across seven datasets without any prior learner data from the target platform.
• The zero-data System-One LLM outperformed the best of 28 deep KT models trained on eight learners (AUC .689) as well as the System-Two Thinking-KT model (AUC .650) across all seven datasets.
• The single-pass System-One method operated at approximately 1/100 of the API cost of the multi-sample System-Two approach.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.11135