Finding the same episode: reliability of corrective-feedback identification in naturalistic online tutoring

A study on naturalistic online English tutoring revealed significant challenges in the reliability of identifying corrective-feedback episodes, even among independent coders using a predefined manual. Despite a high overall agreement rate on turn-level episode presence (97.5%), the actual overlap in identified positive turns was minimal, indicating that different coders frequently missed or selected different instances of the same phenomenon. This suggests a fundamental inconsistency in episode identification, prior to any labeling, within research methodologies involving human coding of interactions.

Why it matters

This research highlights a critical methodological vulnerability in qualitative and mixed-methods studies relying on human coding for interaction analysis. Inconsistent identification of critical incidents or phenomena directly impacts data validity and the replicability of findings, undermining confidence in evidence-based decisions. Addressing this issue is crucial for advancing robust research across various domains, particularly those involving nuanced human communication.

Key insights

• The study focused on the reliability of independent coders identifying specific episodes, rather than just labeling pre-identified episodes, in naturalistic online tutoring.
• Two independent coders screened 1,004 learner turns from six de-identified adult English lessons.
• A frozen corrective-feedback manual was used to guide episode identification.
• One coder identified 12 positive turns, while the other identified 19, demonstrating divergent identification rates.
• Only three of the identified positive turns overlapped between the two coders.
• Overall agreement on turn-level episode presence was high at 97.5%, but this masks the low overlap in specific episode identification.

Source

Frontiers in Education — https://www.frontiersin.org/articles/10.3389/feduc.2026.1949267