An AI chatbot hallucination nearly triggered a US military operation against a Chinese ship

According to a CNN report cited by TechCrunch on Friday, US military aircraft were already airborne this spring when officials discovered that the intelligence underpinning a planned armed operation against a Chinese ship was based on an AI chatbot hallucination. They canceled the operation at the last minute, which, according to CNN, averted a possible conflict with China.

The false information originated with an analyst at Special Operations Command who used an AI chatbot to combine publicly available data with classified signals intelligence. The chatbot incorrectly identified the cargo aboard the ship – an intelligence report circulating during the war with Iran stated that the ship was carrying components for a nuclear weapons program. The analyst then used the same tool a second time to format the incorrect conclusions into an official-looking summary, which subsequently spread through command channels.

The incident comes as the US military, according to the Pentagon, accelerates AI deployment to speed up the decision-making chain ("kill chain") and maintain its lead over China. Jake Steckler, a researcher at GovAI and a former US military officer, told TechCrunch that it is necessary to understand the uncertainty inherent in language models, especially in decisions involving the use of force – targeting, intelligence analysis or operational planning. In his view, the incident should lead to additional safeguards rather than a rejection of AI, because prioritizing the speed of adoption above everything else will lead to more similar cases that will ultimately undermine service members' trust in these systems.