LLM-Generated or Human-Written? Comparing Review and Non-Review Papers on ArXiv

Recent research investigating content uploaded to arXiv's Computer Science domain indicates a significant increase in Large Language Model (LLM)-generated content across both review and non-review research papers. While LLM-generated content is more prevalent in review papers, the absolute number of non-review papers identified as LLM-generated is estimated to be six times higher.

Why it matters

The proliferation of LLM-generated content in academic and research outputs poses challenges to the integrity and credibility of scientific discourse and publication platforms. Understanding the distribution and scale of this phenomenon is critical for developing effective strategies to maintain research quality and trust.

Key insights

• ArXiv prohibited the upload of unpublished review papers in Computer Science, citing high prevalence of LLM-generated content, but without quantitative evidence.
• Two high-quality detection methods were used to compare LLM-generated content in review versus non-review research papers.
• A substantial increase in LLM-generated content was observed across both review and non-review paper categories in recent years.
• LLM-generated content exhibits a higher prevalence within review papers compared to non-review papers.
• Despite higher prevalence in review papers, the estimated number of LLM-generated non-review papers is approximately six times greater than review papers.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2601.17036