Open Problems in AI Risk Modeling: Insights from a Workshop on the Technical Foundations of AI Risk Modeling

This analysis focuses on the development of robust risk models for assessing societal risks associated with advanced AI systems, an emerging domain within AI governance. The current landscape faces significant methodological and institutional challenges in adopting risk modeling, despite increasing regulatory demands for systemic risk assessment. The document identifies several research traditions informing this problem, highlighting the need for rigorous quantitative methods in practice.

Why it matters

The ability to accurately model and assess risks from advanced AI systems is critical for ensuring their safe and responsible development and deployment. This directly impacts strategic decision-making regarding AI investment, regulation, and societal integration, mitigating potential negative externalities and fostering public trust.

Key insights

• Robust risk models are needed to assess societal risks from advanced AI systems.
• AI risk modeling is an emerging area within AI governance.
• Regulatory proposals increasingly mandate systemic risk assessment for AI.
• The absence of rigorous quantitative methods is a key challenge for state-of-the-art AI risk modeling.
• Methodological and institutional challenges currently limit the adoption of AI risk modeling.
• Five research traditions inform AI risk modeling: probabilistic risk assessment, catastrophic AI risk analysis, cybersecurity risk quantification, Bayesian causal inference, and threshold-based governance.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2609.03178