Finding the same episode: reliability of corrective-feedback identification in naturalistic online tutoring

A study on naturalistic online English tutoring revealed significant challenges in the reliability of identifying corrective-feedback episodes, even among independent coders using a predefined manual. Despite a high overall agreement rate on turn-level episode presence (97.5%), the actual overlap in identified positive turns was minimal, indicating that different coders frequently missed or selected different instances of the same phenomenon. This suggests a fundamental inconsistency in episode identification, prior to any labeling, within research methodologies involving human coding of interactions.

This research highlights a critical methodological vulnerability in qualitative and mixed-methods studies relying on human coding for interaction analysis. Inconsistent identification of critical incidents or phenomena directly impacts data validity and the replicability of findings, undermining confidence in evidence-based decisions. Addressing this issue is crucial for advancing robust research across various domains, particularly those involving nuanced human communication.

The study focused on the reliability of independent coders identifying specific episodes, rather than just labeling pre-identified episodes, in naturalistic online tutoring.

Forward consideration, not a verified fact.