AI Fact-Checking in the Wild: A Field Evaluation of LLM-Written Community Notes on X

A field evaluation was conducted on X (formerly Twitter) to assess the performance of a Large Language Model (LLM) fact-checking system, dubbed the "AI writer," in generating Community Notes. This system, which employs a multi-step pipeline for multimodal content, web, and platform-native search, was deployed over three months, producing 1,614 notes on 1,597 tweets. This initiative represents the first live platform assessment of LLM fact-checking, contrasting its output with human-written notes.

Why it matters

This development is strategically important as it demonstrates the practical application and evaluation of AI in critical content moderation functions within a live, high-volume environment. Understanding the effectiveness and limitations of LLM-driven fact-checking is crucial for platform integrity, public discourse, and the strategic deployment of advanced AI technologies in sensitive domains.

Key insights

• Large Language Models demonstrate potential for fact-checking capabilities beyond controlled environments.
• This study is the first field evaluation of an LLM fact-checking system deployed on a live social media platform.
• The evaluated LLM system, an "AI writer," incorporates multimodal content analysis and web/platform search capabilities.
• The LLM generated 1,614 community notes on 1,597 tweets during a three-month deployment.
• The LLM's performance was compared against 1,332 human-written notes on similar content.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2604.02592