Whose Judgments Count? Representation Gaps in Crowdsourced Content Moderation Produce Unequal Protection from Perceived Toxicity

A recent research study highlights significant representation gaps in crowdsourced content moderation, demonstrating that the demographic composition of moderator pools influences who is protected from perceived toxic content online. The study, based on a large dataset of U.S. respondents evaluating content from major social platforms, reveals demographic heterogeneities in what is deemed toxic and how moderation decisions impact user protection, indicating potential inequalities in digital governance outcomes.

Why it matters

This research is critical as it reveals that content moderation practices, intended to create safer online environments, can inadvertently lead to unequal protection due to biases in the composition of moderation teams. Understanding these representation gaps is vital for developing more equitable and effective digital governance strategies that genuinely serve diverse user populations and mitigate systemic risks.

Key insights

• Content moderation systems, while crucial for digital governance, are shaped by human judgments.
• The demographic composition of content moderator pools affects the distribution of protection against perceived toxicity among users.
• There are demographic heterogeneities in the demand for content moderation, meaning different groups perceive toxicity differently.
• The research used a large dataset of 16,221 U.S. respondents evaluating 102,463 comments from Twitter, Reddit, and 4chan.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.01625