The Hallucination That Almost Started a War
An AI hallucination about nuclear components in a Chinese cargo ship almost triggered a real military strike this spring. Military aircraft were already in the air when US officials discovered the intelligence driving the operation had been fabricated by an AI chatbot.
The operation was aborted at the last minute, narrowly averting a potential conflict with China. One source familiar with the episode told CNN it "almost started a war."
The near-disaster happened because a US Special Operations Command analyst used an AI chatbot to synthesize open-source intelligence with classified signals intelligence about a Chinese vessel. The chatbot misidentified the ship's cargo manifest, falsely reporting that it was carrying components for a nuclear weapons program. The analyst then used the tool a second time to format the erroneous findings into an official-looking intelligence summary, which was circulated across military command channels.
The Speed Trap
The Pentagon has been explicit about its AI ambitions. In January, Defense Secretary Pete Hegseth rolled out an initiative to "make all appropriate data available across federated IT systems for AI exploitation, including mission systems across every service and component." The department selected Google's Gemini for Government as the foundation for its GenAI.mil platform. Today, 1.5 million active duty personnel have used the military's generative AI tools for tasks including report creation and analysis.
The logic is straightforward: AI processes data faster than humans, giving commanders a decision-making advantage. But the same speed that makes AI attractive also allows errors to travel before anyone questions them. The chatbot's hallucination was formatted into an official-looking intelligence product and distributed. Nobody stopped to verify because the tool made its output look authoritative.
"It's important for service members to understand the uncertainty inherent to LLMs," Jake Steckler, a research scholar at GovAI and veteran US Army officer, told TechCrunch. "But it's especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions."
Consequences the Military Hasn't Faced Yet
This near-miss is one of the most consequential AI hallucinations documented in a military context. AI researchers have noted that hallucination may be impossible to eliminate from large language models altogether -- the technology simply makes things up when its training data does not provide sufficient context.
The US military's handling of the incident reflects a deeper tension. On one hand, the department has acknowledged the risks -- a State Department declaration on responsible military use of AI stressed the need for "a human in the loop, a responsible human chain of command and control." On the other hand, AI adoption continues to accelerate. The Pentagon recently announced plans to expand GenAI.mil access across all service branches.
Steckler argues the incident should be a reason to add safeguards, not to abandon AI entirely. "These tools can be useful in the right contexts and with the right safeguards in place," he said. "But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption."
The War Nobody Saw Coming
The Chinese ship incident is the kind of mistake that military doctrine has worried about for decades -- not an enemy attack, not a technical failure, but a cascade of small errors amplified by speed and trust. An analyst asked a question. The chatbot answered confidently but wrongly. The answer became a report. The report became an operation. The operation had air support in the air before anyone asked the obvious question: is this real?
Next time, the aircraft might not be recalled in time.
Sources
- TechCrunch: AI hallucination nearly triggers US military operation
- Ars Technica: AI hallucination of Chinese nuclear components almost led to US military attack
- CNN (via Ars Technica): Four sources familiar with the episode
- GovAI: Jake Steckler, research scholar and veteran US Army officer