How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?

Research introduces MMGrader, an AI-driven approach designed to infer the quality of students' mental models from multimodal responses. This method utilizes concept graphs as an analytical framework to assess conceptual understanding beyond simple grading, providing deeper insights into students' ability to apply and integrate concepts. Initial evaluations suggest promising efficacy with available models.

Why it matters

This research addresses the fundamental challenge of accurately assessing deep conceptual understanding, moving beyond superficial grading to evaluate how knowledge is applied and integrated. The development of AI-driven tools like MMGrader could significantly enhance educational assessment methodologies, enabling more effective feedback loops and curriculum development across various learning environments.

Key insights

• STEM mental models are crucial for evaluating students' conceptual understanding and their ability to apply and integrate knowledge.
• Inferring these mental models from student responses is complex, requiring advanced reasoning capabilities.
• MMGrader is proposed as an approach to infer mental model quality from multimodal student responses.
• Concept graphs serve as the analytical framework within the MMGrader approach.
• Evaluation across nine open-source models indicates positive performance for the best-performing models in this task.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2603.00056