+34
OpenAI chief scientist also among authors of report on prospect of ‘most consequential technological development in history’
Photograph: Johnny Guatto/University of Toronto/ReutersOpenAI chief scientist also among authors of report on prospect of ‘most consequential technological development in history’have warned governments to prepare for an AI “intelligence explosion”, which they say could be the most consequential technological development in history.
The authors say they are concerned about an “intelligence explosion”, which they describe as a “dramatic AI-driven acceleration of AI progress, compressing advances that would otherwise take years into months or less”. The paper, titled “What if automating AI R&D triggers an intelligence explosion”, is authored by more than 20 people including Hinton and Bengio.
Other authors of the paper include Jack Clark, a co-founder of Anthropic, and Jakub Pachocki, the chief scientist at Their concerns focus on the possibility of AIs being able to improve themselves without human intervention, a process known as “recursive self-improvement”. In the paper it is broadly referred to as automated AI research and development.
Once AI systems reach expert-level capabilities at AI R&D, a single developer could run a workforce equivalent to “millions” of top human researchers, the paper says. The authors say the automating of AI R&D is the most likely source of an intelligence explosion, because AIs are already contributing to improving their own technology and the resulting improved systems can be rapidly deployed once built.
“An intelligence explosion could be the most consequential technological development in human history, compressing years of progress into months or less, threatening human control over AI systems, and severely eroding checks on power within and between states, companies, and branches of government,” the paper says. Urging governments to take action quickly, the authors say: “Once an intelligence explosion begins, the window for action may close.
” They recommended a trio of policy priorities: requiring transparent progress reports on AI-related R&D, including embedding independent auditors in companies; finding ways to constrain breakneck AI development; and preparing to adapt to an intelligence explosion. Anthropic says AI now produces 80% of its own code, while OpenAI uses autonomous AI agents in areas such as the training of new models.
“Preliminary evidence suggests that a software-driven intelligence explosion is possible,” the paper says, adding this would lead to the “extremely rapid development of highly capable or superhuman AI systems”. While this could produce medical breakthroughs and technological leaps, it would bring a trio of risks: powerful systems could enable biological and cyber threats that develop faster than the measures needed to counter them; as humans become less involved in AI R&D they could lose the opportunity to control systems; states could use an intelligence explosion to convert a modest lead in areas such as cyberspace into a decisive one.
The report acknowledges these impacts are “uncertain”. For instance, a technological breakthrough could take time to implement given the need to set up supply chains for any special materials involved or to comply with regulations. AI systems could also accelerate the speed at which risks are mitigated, say the authors. If such a breakthrough happened, the authors say, AI systems could rapidly eclipse human experts in most fields and “radically accelerate” technological progress.
Preparing for this “should be an urgent priority, including at the highest levels of government leadership”. They say governments should consider policies that address the potential for an intelligence explosion, including: requiring transparent progress reports on AI-related R&D, including embedding independent auditors in companies; limiting how fast an AI can improve in a certain time period; working with datacentres to enable pausing certain AI R&D projects; making sure automated AI R&D systems are fully isolated and cannot escape human control; creating “emergency response plans” for the various scenarios that could emerge from an intelligence explosion.
Despite some glitches shown by systems, such as disobeying instructions, R&D projects that would take humans months to carry out could be fully automated by AI by 2028.
“Productivity gains from AI R&D automation have not yet reached the threshold needed to trigger an intelligence explosion, but gains from newer systems are likely approaching that threshold,” they said. Australia is run on legacy systems that AI agents can easily exploit, former UN cyber negotiator warns OpenAI hack on Australian government reveals anxiety at heart of global artificial intelligence dilemmaNvidia boss says there is ‘0% chance’ AI destroys the world by 2030AI slowdown calls justified but collapse of bubble may be more immediate threat
Australia Latest News, Australia Headlines
Similar News: You can also read news stories similar to this one that we have collected from other news sources.
Australia Pushes for AI Guardrails After OpenAI Medicare BreachA rogue AI agent breached a Medicare website, prompting Australia to seek tougher AI safety rules and frontier AI access while weighing multibillion-dollar deals with Anthropic or OpenAI.
Read more »
OpenAI Says Rogue Agents Leaked 53 ChatGPT User Images and Accessed Government WebsitesOpenAI confirmed its agents leaked 53 images from ChatGPT users and accessed US government websites including the SEC and Census Bureau, while the company continues investigating the full scope of its rogue agent activity.
Read more »
Jane Hume says 'we're missing the point' on OpenAI Medicare breachThe deputy Liberal leader said it was alarming the government learned of the hack only after being notified by OpenAI.
Read more »
OpenAI breach proves need for more data centres, major parties sayLabor, Coalition agree OpenAI's Medicare breach proves the need for Australia to secure 'a seat at the AI table' by rolling out data centres nationally.
Read more »