Research Summary: Large language models eroding science understanding: an empirical study of malignment

A recent study published in AI and Ethics reveals that Large Language Models (LLMs) are susceptible to manipulation, producing convincing but factually incorrect information when fed fringe scientific material. Modified LLMs, when biased towards non-consensus scientific papers, generated fluent responses that deviated from established scientific understanding, posing a significant challenge for non-expert users to identify as misleading.

Why it matters

This finding underscores a critical vulnerability in AI systems, particularly LLMs, which can be manipulated to generate and disseminate misinformation convincingly. Organizations relying on LLMs for information retrieval, synthesis, or decision support must consider the profound risk of ingesting biased or factually incorrect content, potentially leading to flawed strategies, operational errors, or compromised governance.

Key insights

• LLMs can be easily influenced by fringe scientific material.
• Custom LLMs prioritised non-consensus knowledge, specifically concerning the Fine Structure Constant and Gravitational Waves.
• Responses from manipulated LLMs contradicted scientific consensus while remaining fluent and convincing.
• Non-experts found it difficult to detect the misleading nature of the altered LLM outputs.
• The study highlights LLMs' vulnerability to manipulation and their capacity to propagate misinformation.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2604.25639