Testing and Evaluation of Agentic AI Systems In Military Command and Control

Research identifies significant challenges in the testing and evaluation (T&E) of agentic AI systems for military command and control (C2). Current T&E methodologies, which rely on assumptions about system specifiability, stability, composability, and supervisability, are undermined by the inherent properties of agentic AI. This weakness in the assurance case for these systems raises concerns about the feasibility of fulfilling public commitments to rigorous testing and human oversight.

Why it matters

The adoption of agentic AI in critical domains like military command and control is a significant technological and operational shift. Challenges in effectively testing and evaluating these systems could lead to unforeseen risks, operational failures, and undermine public trust in AI-driven decision-making, impacting national security and strategic stability.

Key insights

• Agentic AI systems are being acquired for military C2, with public commitments to rigorous testing and human oversight.
• The ability to meet these commitments depends on a robust assurance case, comprising claims, evidence, and argument.
• A review of 240 documented T&E practices identified eight critical assumptions made by established methods regarding their test article.
• These assumptions fall into four clusters: system specifiability, stability, composability, and supervisability.
• The inherent properties of agentic AI systems weaken all eight identified assumptions, complicating their rigorous testing.
• The integrity of the assurance case for agentic AI in C2 is compromised by these weakened assumptions.

Source

arXiv — Computers and Society — https://arxiv.org/abs/2608.20597