Reviewing the Reviewer: LLM-Assisted Reviewer Feedback Generation for Guideline Compliance

The quality of scientific peer review is facing challenges due to common issues such as superficial evaluations and vague critiques. A new LLM-driven framework is proposed to enhance review quality by segmenting reviews, identifying guideline violations, and generating specific, actionable feedback for reviewers, moving beyond simple issue detection.

Why it matters

Maintaining high-quality peer review is crucial for scientific integrity, research validity, and the efficient advancement of knowledge across all domains. Innovations that improve the consistency and constructiveness of review processes can significantly impact the reliability and trustworthiness of published research, fostering better resource allocation and accelerating progress.

Key insights

• Peer review quality is threatened by widespread 'lazy thinking' and non-specific critiques.
• Existing methods for detecting review issues often treat them as single-label classifications and stop at detection.
• Review segments frequently contain multiple co-occurring issues, necessitating more nuanced analysis.
• Reviewers benefit more from actionable, guideline-aware feedback than from simple issue labels.
• Off-the-shelf LLMs, when prompted for feedback, often rewrite entire reviews or misdirect feedback to authors instead of reviewers.
• An inference-time approach is needed to ensure LLM-generated feedback is appropriately targeted and structured for reviewer improvement.
• The introduced framework decomposes reviews into argumentative segments and identifies issues violating established guidelines, such as those from ACL Rolling Review (AR).

Source

arXiv — Computers and Society — https://arxiv.org/abs/2602.10118