Research Summary: Mitigating GenAI-Powered Evidence Pollution for Out-Of-Context Misinformation Detection

Generative Artificial Intelligence (GenAI) models pose a growing threat to online information security due to their capacity to create deceptive content. Specifically, GenAI is being used to 'pollute' evidence used by out-of-context (OOC) multimodal misinformation detection systems. Traditional detection methods, which rely on web-retrieved evidence, are increasingly challenged by this GenAI-driven evidence pollution, as existing work often assumes a 'clean' evidence corpus.

Why it matters

The proliferation of GenAI-generated deceptive content undermines trust in digital information and the effectiveness of established misinformation detection mechanisms. Addressing this threat is critical for maintaining information integrity, protecting brand reputation, and ensuring the reliability of data used for decision-making across various sectors.

Key insights

• GenAI models are increasingly used to generate deceptive content, raising concerns for online information security.
• Out-of-context (OOC) multimodal misinformation detection systems, which use web-retrieved evidence, are vulnerable to GenAI-powered evidence pollution.
• Current misinformation detection approaches primarily focus on claim-level stylistic rewriting and assume that evidence used for verification is not itself generated or altered by GenAI.
• The research aims to systematically study the impact of GenAI-driven evidence pollution on OOC detection, removing the assumption of a clean evidence corpus.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2501.14728