Research Summary: The Nuclear Decision-Making Benchmark: Evaluating Frontier LLMs on Nuclear Tendencies

A new research initiative, the Nuclear Decision-Making Benchmark (NDM Bench), has been developed to assess the stability, consistency, and policy appropriateness of frontier large language models (LLMs) in high-stakes contexts, particularly within defense and national security workflows. The benchmark comprises 151 scenarios authored by PhD-credentialed scholars, focusing on escalation, arms control, non-proliferation, and proliferation. It also incorporates actor-agnostic scenarios and experimental phrasing variants to test sensitivity to narrative framing.

Why it matters

The integration of advanced AI into critical national security and defense systems necessitates rigorous evaluation of their decision-making tendencies. Understanding how frontier LLMs process and respond to high-stakes scenarios, especially those involving nuclear issues, is crucial for ensuring their reliability and alignment with strategic policy objectives. This research directly addresses the need for robust benchmarks to assess AI performance in sensitive global contexts.

Key insights

• The NDM Bench evaluates LLMs on policy-relevant preferences in defense and national security.
• It contains 151 scenarios developed by international relations scholars.
• Scenarios cover four critical domains: escalation, arms control, non-proliferation, and proliferation.
• The benchmark allows for testing with various country pairs due to its actor-agnostic design.
• It includes experimental phrasing variants to analyze LLM sensitivity to narrative framing.
• Seven frontier AI systems are being tested using this benchmark.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.05180