Research Summary: Before the Arrest: Benchmarking LLMs on Criminal Profiling from Incomplete Evidence

New research introduces the Profiling, Investigation, and Judgment (PIJ) dataset, consisting of 2,500 real homicide cases from five countries, to benchmark Large Language Models (LLMs) on criminal profiling using incomplete evidence prior to arrest. This addresses a critical gap in existing applications, which predominantly focus on post-arrest scenarios. The PIJ framework evaluates LLMs across criminal profiling, crime process reconstruction, and judgment tasks, aiming to assess their abductive reasoning capabilities for inferring suspect characteristics from fragmentary evidence.

Why it matters

This development is crucial for understanding the evolving capabilities of AI in sensitive applications like criminal justice, particularly concerning pre-arrest investigation. It highlights the potential for technology to augment human analysis in complex, data-poor environments while also raising important considerations for accuracy and ethical deployment.

Key insights

• Existing applications of Large Language Models (LLMs) in legal and criminal justice largely focus on post-arrest scenarios where suspect identity is known.
• A significant gap exists in evaluating LLMs for pre-arrest challenges, specifically inferring suspect characteristics from incomplete evidence.
• The Profiling, Investigation, and Judgment (PIJ) dataset has been introduced, comprising 2,500 real homicide cases from five countries.
• PIJ benchmarks LLMs on three tasks: criminal profiling (inferring suspect attributes from fragmentary evidence), crime process reconstruction, and judgment.
• The research aims to assess LLMs' abductive reasoning abilities in criminal investigation contexts.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.19965