Fine-Tuning Large Language Models for Codebook-Guided Coding of Students' Mathematics Metaphor Responses

A research study explored the application of fine-tuned Large Language Models (LLMs) to automate the qualitative analysis of student responses, specifically focusing on mathematics metaphors. The aim was to address the scalability challenges associated with human expert coding of complex, open-ended textual data. The study used a corpus of 2,265 human-coded responses from Grade 6-8 students to evaluate LLMs' ability to perform valence-intensity and thematic coding, which are critical for understanding student attitudes and beliefs.

Why it matters

This research is strategically important because it explores scalable solutions for qualitative data analysis, a significant bottleneck in many research and operational contexts. Automating or semi-automating such tasks through fine-tuned LLMs can drastically reduce costs and time associated with manual expert analysis, enabling broader and more frequent insights from open-ended data.

Key insights

• Human expert coding of student-generated metaphors is resource-intensive and difficult to scale.
• The study investigated the use of LoRA-based supervised fine-tuning to improve LLM performance in codebook-guided coding.
• The research focused on coding student mathematics metaphors to reveal attitudes, beliefs, identities, and experiences.
• Two types of coding tasks were implemented: valence-intensity coding (direction and strength of affective orientations) and thematic coding.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.10276