Research Summary: Correctness, Convergence, and AI-Generated Code Detection: A Longitudinal Study of Student and Large Language Model Code in Introductory Programming

A study investigated the challenges of detecting AI-generated code in introductory programming, particularly when Large Language Models (LLMs) can produce plausible solutions that may closely resemble legitimate student work due to the limited 'natural implementations' of certain assignments. The research utilized a large dataset of student submissions and LLM-generated solutions to analyze the efficacy of generated-reference matching techniques.

Why it matters

The increasing sophistication of AI in generating code presents significant challenges for assessing originality and integrity in technical work and educational settings. Understanding how to accurately differentiate between human and AI-generated content is crucial for maintaining fair evaluation systems and developing effective strategies to leverage or mitigate AI's impact on code production.

Key insights

• Large Language Models (LLMs) are capable of generating plausible solutions to programming assignments.
• The inherent nature of some assignments means only a few 'natural implementations' exist, potentially leading to similar code from both students and LLMs.
• This similarity complicates AI-generated code detection, as a match against a reference bank of generated solutions may not definitively prove AI use.
• The study analyzed 29,970 student submissions from ten Python labs across 2021, 2023, and 2025.
• It also used 90,000 solution attempts generated retrospectively by three frontier LLMs.
• Validation of generated solutions was conducted using hidden instructor tests.
• Code comparison was performed using MOSS after excluding starter code, and abstract syntax tree (AST) forms were examined for selected functions.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2610.00863