Research Summary: Testing Our Foundations: Citation Trends, Errors, and Emerging Hallucinations in the Computing Education Literature

A recent research study highlights a growing concern regarding the integrity of scholarly citations, particularly the emergence of fabricated references (hallucinations) due to the rapid adoption of large language models. The study analyzed over 24,000 computing education papers and compared them to a broader corpus of 723,000 ACM papers, identifying issues related to reference accuracy and trends. This trend poses a significant threat to the foundational principles of academic rigor and knowledge validation.

Why it matters

The integrity of research is paramount for evidence-based decision-making and innovation across all sectors. The proliferation of fabricated citations, if unchecked, can erode trust in published work, undermine the reproducibility of studies, and lead to misinformed strategies and policies based on non-existent or misrepresented information.

Key insights

• Accurate references are fundamental for scholarly work, supporting verification, attribution, and systematic review.
• The rapid adoption of large language models (LLMs) has introduced a new integrity concern: plausible-looking but fabricated citations, referred to as 'hallucinated references'.
• The visibility and impact of these hallucinated references within specific research communities, such as computing education, are not yet fully understood.
• The study analyzed citation trends and integrity across 24,751 computing education papers.
• This analysis was benchmarked against the broader ACM corpus, which includes over 723,000 papers and 15 million references.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.16574