MASH-Bench: Diagnosing Cross-Source Failure in Mass-Shooting Risk Classification

Research identifies significant challenges in applying machine learning models for mass-shooting risk classification due to inconsistencies across public databases. A new benchmark, MASH-Bench, demonstrates that while models perform adequately on curated sources, they exhibit severe generalization failures when applied to broader, less curated datasets, such as the Gun Violence Archive (GVA). This highlights fundamental data comparability issues affecting model reliability.

Why it matters

This research is crucial because it exposes fundamental data quality and interoperability issues that undermine the effectiveness of advanced analytical tools in critical risk assessment domains. Understanding these limitations is vital for resource allocation, policy development, and ensuring the reliability of intelligence derived from disparate sources.

Key insights

• Public mass-shooting databases vary significantly in coverage, feature availability, and reporting practices.
• MASH-Bench is a harmonized benchmark comprising 6,968 incidents from four U.S. databases: Kaggle, Mother Jones, Stanford MSA, and the Gun Violence Archive (GVA).
• Machine learning models (Random Forest, XGBoost, LightGBM) achieve 0.68-0.89 VeryHigh-risk recall on curated sources.
• Model generalization is poor when applied to the GVA, with recall dropping to 0.20 and precision to 0.0004.
• The research investigates the degradation through controlled feature-masking ablation to pinpoint the source of failure.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.22460