Research Summary: An Audit of Measurement Quality and Answer Bias in a Large Classroom-Poll Corpus

A study auditing a large corpus of real-time classroom polls found that while these polls form a coherent measurement scale, they generally lack high precision and present an easy assessment, with a significant portion barely differentiating student understanding. The research used item response theory to analyze 604 poll items answered over 340,000 times by 2,807 learners, assessing their quality as measurement instruments.

Why it matters

The quality of assessment tools directly impacts the accuracy of performance evaluation and the effectiveness of educational or training programs. Understanding the measurement quality of widely used tools like real-time polls is critical for ensuring that data collected truly reflects knowledge or skill acquisition and informs strategic decisions about curriculum and learning design.

Key insights

• Classroom polls, despite widespread use and automated generation, are rarely evaluated for measurement quality.
• An audit of 604 authentic classroom poll items revealed a coherent but easy scale with moderate precision (marginal reliability approximately 0.60).
• Roughly 25% of the audited poll items demonstrated limited ability to differentiate between stronger and weaker student understanding.
• The analysis placed every item and student on a common scale using item response theory, specifically for 539 items where correct answers could be verified from lecture transcripts.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.38959