Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics

A novel two-stage fine-tuning pipeline is proposed for distilling complex, opaque machine learning (ML) models used in learning analytics into smaller, open-weight large language models (LLMs). This process aims to enable these LLMs to provide individual-level estimates and natural language explanations, addressing challenges of model opacity and deployment burden in educational practice. The approach is designed to be estimator-agnostic and includes a faithfulness-first evaluation framework for narrations.

Why it matters

This research introduces a method to enhance the interpretability and deployability of advanced machine learning in critical domains like education. By making complex models explainable and more accessible, it can foster greater trust and adoption of AI-driven tools, thereby improving decision-making and operational efficiency where data insights are crucial.

Key insights

• A two-stage fine-tuning pipeline distills black-box ML estimators and their post hoc interpretations into small, open-weight LLMs.
• The resulting LLMs provide individual-level estimates and natural language explanations.
• The proposed design addresses model opacity and deployment burdens prevalent in current learning analytics tools.
• The distillation process is estimator-agnostic, allowing flexibility in the choice of initial black-box models.
• A faithfulness-first evaluation framework is introduced to audit explanations against attributed descriptions.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.21165