+34
A CNN report reveals the US military nearly intercepted a Chinese vessel based on false intelligence generated by a chatbot. Experts warn that real AI risks are being ignored amid hype about superintelligence.
A CNN report published on 18 September, which has not been verified by other major outlets, describes how the United States almost started a war with China after relying on an intelligence report generated by a chatbot.
The report claimed a Chinese ship was transporting components of nuclear weapons, prompting military personnel to plan an interception with aircraft and soldiers ready to board the vessel. A military skirmish of this kind between China and the US could quickly escalate and spiral into world war three. Only moments before executing the plan, American officials further investigated the intelligence report and found it had been generated with a chatbot and contained erroneous information about the vessel's contents.
Lack Of Regulatory Response To Incident
This incident did not galvanize lawmakers into announcing calls for regulation, unlike the fictional terminator scenario. Media outlet after media outlet did not warn the public that the US almost triggered a war against a nuclear superpower because of its military's reliance on error-prone chatbots. The biggest international AI news of the past three weeks was Anthropic engineer Jacob Coxon's resignation. According to him, OpenAI and Anthropic are racing straight towards self-improving superintelligence and gambling with our lives.
Coxon's description of a terminator scenario, a machine becoming much smarter than humanity and deciding to wipe us out, captured the public's, journalists' and politicians' attention. It stirred waves of fear-inducing, credulous reporting and policy actions. New York Governor Kathy Hochul is similarly exploring the possibility of having AI kill-switches, and Senator Bernie Sanders and Congressman Greg Casar introduced legislation to address the issue.
While so many of us were focused on fictional superintelligent machines that we were told could hypothetically cause human extinction, a more serious story was being reported. As former FTC chair Lina Khan has repeatedly reminded us, there is no exemption to the law when it comes to AI. There are existing laws that federal agencies and other regulators can enforce to protect the public against the AI industry's unsafe and unscientific practices.
Misleading Marketing Of Stochastic Parrots
These practices include pushing unsound products while marketing them as powerful and rogue. We describe the large language models powering these chatbots as stochastic parrots, meaning models designed to regurgitate the patterns of the data they are trained on. News coverage describing them as powerful rogue models that could render humanity extinct on their own bolsters perceptions that these tools are infallible enough for use in high-stakes warfare.
Even CNN's more critical coverage describes the chatbot used by the US military as presenting profound risks because it is powerful, new and relatively poorly understood technology. In fact, today's large language models and the chatbots they power are not poorly understood, nor are they powerful in the sense of being effective tools for this use case.
They are systems for generating plausible-looking text, built out of enormous, haphazardly collected datasets and then fine-tuned to be especially appealing to their intended users. CNN was not able to learn which chatbot product was used in this case, but the LLM powering the product was probably fine-tuned to output text with the stylistic hallmarks of intelligence reports. Both the basic functionality and the risks of such a model have been well-understood and well-documented for years.
This includes our 2021 paper On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? The harmful impacts of these products have arisen from deliberate mystification by sellers and misapprehension by those procuring and using them. People believe these systems are infallible, leading to errors such as erroneously classifying patients, mistaking schools for military facilities, and government reliance on flawed outputs.
Need To Regulate Error Prone Systems
Meanwhile, marketing campaigns from companies selling this software warn of impending superintelligence. This directs our attention away from the real risk and toward their fantasies of doom. It is high time to regulate usage of systems sold as AI, not because they are magical machines, but because they are error-prone products that should not be used in high-stakes scenarios. Any automation used in military, medical or other life-and-death situations must be thoroughly evaluated within its use case.
Such automation must also be paired with usage norms that keep accountability with people who have actual power to make decisions. Going along with company executives and describing their products as beings with agency allows these executives to evade accountability while deceiving the public. We call on journalists and lawmakers to hold companies accountable rather than parrot their marketing talking points.
The risks of AI are not what we think they are, as this recent security incident between China and the United States reveals.
Military Chatbots US China Incident Stochastic Parrots Superintelligence Hype
Australia Latest News, Australia Headlines
Similar News: You can also read news stories similar to this one that we have collected from other news sources.
Report: Suspect called UK police before RAF Fairford arrests; five released on bailA report says one suspect called UK police before five men were arrested near RAF Fairford, where vans were checked for explosives. Police confirmed no improvised explosive devices were found and released the men on bail.
Read more »
Catholic clergy sexually abused nearly 1,000 Massachusetts children over decades, report findsNo new indictments from eight-year investigation as state attorney general says church knew about abuse
Read more »
SA Government Refuses to Release Hyde Report on Child DeathsThe South Australian government has declined to release the 2022 Hyde report into the deaths of two children, citing ongoing legal and police investigations, despite promises from Premier Peter Malinauskas to publish it and criticism from advocates like Belinda Valentine.
Read more »