publisher
How consistent is the algorithm? Examining the intra- and inter-rater reliability of LLM-based writing assessment
- Source
- Frontiers in Education
- Published
- Last verified
- 13 Aug 2026
- Confidence
- High
- Evidence
- Original document retained
- Reading time
- 1 min
- Country
- International
- Relevant to
- Research & Evidence, Technology & Data
Executive summary
What happened, and why should leadership care?
A study published in Frontiers in Education examined the reliability of Large Language Model (LLM)-based writing assessment, specifically using OpenAI's ChatGPT. The research focused on assessing 192 essays from EFL learners, applying a criterion-based analytic rubric. The study found high intra-rater reliability for the LLM, indicating consistency in its scoring over time.
Why this matters
Why is this strategically important?
The consistent performance of AI in assessment offers potential for scalable and objective evaluation processes across various domains. Understanding the reliability of LLM-based tools is crucial for their integration into existing frameworks and for developing future strategies that leverage artificial intelligence.
Key insights
What should be noted from the evidence?
- LLM-based scoring demonstrates high intra-rater reliability, meaning consistent scores for the same essays over time.
- OpenAI's ChatGPT was utilized to assess EFL learner essays using a criterion-based analytic rubric.
- The assessment rubric covered task achievement, grammatical range and accuracy, lexical resources, organization, and mechanics.
- The study replicated scoring after three weeks under similar conditions to assess intra-rater reliability.
Evidence and confidence
How far can this assessment be trusted?
High confidence. Named institution, original document retained and analysis corroborated.
Analysis is prepared editorially by Aziz Shuaib Ausi. The original publication remains the authoritative record, and executive judgement remains entirely human.
Source
Where does this originate?
Reported by Frontiers in Education · International. This briefing summarises the publication for executive use; the document itself is not reproduced here.
Read the original publication