Research Summary: New Terms, New Toxicity: Consensus-based Chinese Neologism Toxicity Detection via Search-Augmented LLMs

Research identifies a critical challenge in content moderation: the emergence of toxic neologisms, which are new or recontextualized terms that carry implicit toxicity through public consensus. These terms, such as 'country girl' used to stigmatize feminism, are difficult for existing moderation systems to detect due to their evolving nature and non-obvious harmful intent. A new framework, SeTox, is proposed to address this by combining a taxonomy of toxic neologisms, a comprehensive lexicon, and search-augmented Large Language Models (LLMs) to identify toxicity grounded in public consensus.

Why it matters

The proliferation of implicitly toxic neologisms represents a significant and evolving challenge for digital platforms and online communities, impacting user safety, brand reputation, and regulatory compliance. Effective detection mechanisms are crucial for maintaining healthy online environments and mitigating the spread of nuanced, consensus-driven forms of harm.

Key insights

• Neologisms can serve as novel vectors for toxic expression, bypassing traditional content moderation filters.
• Toxic neologisms often appear benign but derive their harmful usage from public consensus, making their detection complex.
• A taxonomy is proposed to categorize the origins and consensus-verification criteria of these implicitly toxic terms.
• A lexicon has been constructed to cover various risk categories associated with toxic neologisms.
• The SeTox framework utilizes search-augmented LLMs to detect toxicity that is rooted in public consensus, even from static models.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.12361