Research Summary: CrisisFake: Benchmark Validity of AI-Generated Text Detection for Disaster Social Sensing

Researchers have developed CrisisFake, an empirical benchmark dataset of 12,000 texts across nine disasters, to evaluate the efficacy of text-based detectors in differentiating human-authored social media posts from plausible LLM-generated content during disaster social sensing operations.

Why it matters

The infiltration of plausible AI-generated content into crisis data streams threatens the integrity of real-time situational awareness and humanitarian triage. Inaccurate automated intelligence can misdirect life-saving resources and skew damage assessments during emergency events. Establishing robust detection benchmarks is therefore critical to maintaining trustworthy open-source intelligence pipelines.

Key insights

• Plausible LLM-generated posts threaten disaster social sensing by corrupting situational awareness and distorting assessments of damage, operational needs, and resource distribution.
• The CrisisFake dataset utilizes Qwen2.5-7B to assemble 12,000 texts organized into 3,000 matched semantic units spanning nine distinct disaster events.
• The dataset benchmarks detector validity across multiple text variations, including original human posts, minimally LLM-proofread posts, and fact-preserving LLM-generated posts created using LoRA.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.35821