{"id":7991,"date":"2026-03-26T03:11:32","date_gmt":"2026-03-26T03:11:32","guid":{"rendered":"https:\/\/sawahsolutions.com\/range\/will-too-much-ai-weaken-troops-judgement\/"},"modified":"2026-03-26T03:11:32","modified_gmt":"2026-03-26T03:11:32","slug":"will-too-much-ai-weaken-troops-judgement","status":"publish","type":"post","link":"https:\/\/sawahsolutions.com\/range\/will-too-much-ai-weaken-troops-judgement\/","title":{"rendered":"Will too much AI weaken troops’ judgement?"},"content":{"rendered":"
\n

The debate about lethal autonomy\u2014core to the Trump administration\u2019s fight with Anthropic\u2014obscures a deeper danger of the Pentagon\u2019s rapid adoption of commercial AI tools: they might weaken the U.S. military\u2019s ability to tell fact from fiction.<\/p>\n

New research suggests that\u00a0relying on AI to do various tasks can erode one’s native ability to do them.\u00a0Military commanders are taking note.\u00a0<\/p>\n

\u201cThe more you use AI, the more you will use your brain in a different way,\u201d said French Adm. Pierre Vandier, NATO Supreme Allied Commander for Transformation, in a conversation in\u00a0Defense One\u2019s<\/em> “State of Defense” series. \u201cAnd so [we need to] be able to have some oversight, to be able to critique what we see from AI, and to be sure you are not fooled by a sort of false presentation of things. It’s something we need to take care of.\u201d<\/p>\n

But there is scant evidence that the Pentagon, in its rush to deploy AI tools, is taking steps to keep its users sharp\u2014or even to monitor the effects of AI.<\/p>\n

\u201cBecause of the pace and the urgency associated with deploying these models, maybe that hasn\u2019t been put in place,\u201d said one former senior military official who worked to deploy AI in combat settings.<\/p>\n

In the meantime, the pressure to use AI tools is only growing.<\/p>\n

\u201cEspecially as you get deeper into any conflict, there will be more and more pressure to find more targets. That happens in every conflict as well. After two weeks, you’re sort of going through your delivery target list. Now you’re demanding targets. Well, how fast can you generate targets?\u201d the former senior military official said.<\/p>\n

A cognitive trap<\/strong><\/p>\n

A growing body of research shows that wide use of large language models can undermine human thought and communication.<\/p>\n

For example, it can homogenize thinking among users, reinforcing \u201cdominant styles while marginalizing alternative voices and reasoning strategies,\u201d according to an Air Force Research Laboratory paper published earlier this month in the journal Cell.<\/em><\/p>\n

This presents at least two problems for the military, said Morteza Dehghani, a University of Southern California computer science professor who helped write the paper. In the moment, \u201cit washes away signals about who the author is,\u201d and therefore eliminates important context for evaluating data. And over time, it can stifle critical thinking.\u00a0<\/p>\n

\u201cBecause these models are optimized for the most likely and \u2018idealized\u2019 responses, they often enforce a linear, \u2018Chain-of-Thought\u2019 reasoning style,\u201d Dehghani wrote in an email. \u201cThis can disincentivize experienced analysts from employing the non-linear, intuitive, or \u2018gut feeling\u2019 strategies that are essential for identifying rare exceptions or navigating complex, non-standard intelligence scenarios.\u201d<\/p>\n

Similarly, researchers at Wharton found in January that people using LLMs spend less and less time scrutinizing results for accuracy or employing their own judgment. They found that users rely on the AI\u2019s judgement even when they know it\u2019s wrong, a phenomenon the authors call \u201ccognitive surrender.\u201d<\/p>\n

And a February paper from Princeton found that the way LLMs speak to users\u2014referred to as \u201csycophantic AI\u201d\u2014instills a false sense of confidence and isolates people within preconceived biases: \u201cOur results show that the default interactions of a popular chatbot resemble the effects of providing people with confirmatory evidence, increasing confidence but bringing them no closer to the truth.\u201d<\/p>\n

Who is driving?<\/strong><\/p>\n

The military is aware of at least some dimensions of the problem. Speaking on a March 6 podcast, Pentagon research-and-engineering chief Emil Michael said that his core concern about Anthropic\u2014which President Trump has barred from use by the federal government\u2014was that military users might become too dependent on a single untrustworthy tool.<\/p>\n

\u201cMaybe it’s a rogue developer who could poison the model to make it not do what you want at the time, or sort of trick you because you have to trick it. I mean, all these things that we know when we worry about models that hallucinate purposefully or do not follow instructions,\u201d Michael said.<\/p>\n

Anthropic had similar concerns: that its tools might be used by people untrained to properly evaluate its output. One company official said their chief worry was that they had not validated that the model could be used reliably for compiling targeting lists. Worse, they had no way of knowing how the military was putting their tools to use. For example, Anthropic officials only learned after the fact that their tools had been used to plan the Jan. 3 raid into Venezuela.<\/p>\n

The Pentagon has since declared Anthropic products a supply-chain risk and is working to replace them, though they are still in use by military planners, including in the Middle East.<\/p>\n

A former senior official described it as \u201ca serious governance question.\u201d Frontier AI companies like OpenAI, Anthropic, Google, and xAI cannot provide the in-depth support required for military units to understand the conditions under which these tools are used.<\/p>\n

\u201cIt’s almost becoming a journey of discovery for the government,\u201d the former senior military official said. \u201cAre there people on site from these companies helping the day-to-day user? My guess is, if there are, there may be only one or two of them.\u201d\n\n<\/svg><\/p>\n<\/div>\n