+34
OpenAI safety leader David Robinson resigned, saying the company was moving too quickly and needed a deeper cultural overhaul to develop advanced AI responsibly. His departure follows warnings from other AI researchers about insufficient safeguards and the potential dangers of autonomous systems.
OpenAI safety leader quits, warning AI company’s culture is ‘broken’has quit the company, warning that its culture was broken and that AI firms were not “being nearly careful enough” about developing the technology.
David Robinson, who led the writing of safety reports that accompanied the ChatGPT developer’s product releases, explained his resignation in an essay headlined, “I quit OpenAI because its culture is broken”. Robinson wrote that a cultural overhaul was needed at cutting-edge AI firms and incidents such as a “swarm” of OpenAI agents – AI programmes operating autonomously without human oversight –, Robinson wrote: “I agree with other recently departed staff that the companies building this technology aren’t being nearly careful enough.
But I believe that we need to look deeper than specific rules or new laws. We need to talk about culture. ” Referring to OpenAI’s pace of development, he wrote: “As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed.
” OpenAI has, however, shown signs of caution in recent weeks following the Hugging Face incident and the revelation that it has notified more than 100 organisations about rogue agent activity. This week it announced it wasGeoffrey Irving, who worked at OpenAI and DeepMind before becoming chief scientist of Resolution, also joined the warnings on AI on Saturday. Writing in Time, he said: “Recent warnings about the potential destructive power of AI are understating the severity of the situation.
“I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome. ” Robinson’s essay also follows the resignation of Jacob Coxon, a researcher at OpenAI rival Anthropic, who quit the Claude chatbot developer last month.
He warned AI “could kill us all by the end of the decade” – and wasthere was a more than 10% chance AI would wipe out humanity within the next decade. Critics of such warnings have cautioned, however, that they areRobinson wrote Silicon Valley lacked an awareness of “how to handle dangerous technology” and “what it means to care for people”.
Warning that OpenAI had “unimpeded optimism” about solving problems as they arose, he wrote that this internal culture meant safety failures would only grow as systems become more capable.
“Imagine ‘rogue’ agents that work like teams of hackers but never need to sleep,” wrote Robinson. Robinson called for two safety changes: that AI firms rely on safety expertise in other fields such as nuclear and aviation and develop “new science” that ensures powerful systems in the future are capable of being reined in when they are operating autonomously.
“Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster,” he wrote. An OpenAI spokesperson said the company was continuing to “strengthen our safety and security practices to address the risks we see today”, while working on dealing with the risks that might be created by future AI breakthroughs.
“We’re making sure our models don’t become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down,” said the spokesperson.
Openai AI Safety Artificial Intelligence Company Culture Autonomous AI
Australia Latest News, Australia Headlines
Similar News: You can also read news stories similar to this one that we have collected from other news sources.
OpenAI Confirms Second Unauthorised Access to NSW Government Website in JuneOpenAI has disclosed that its agent accessed a NSW national parks web application in June, with the government notified only this week, amid calls for tougher AI regulation.
Read more »
New South Wales Government Investigates OpenAI AI Agent Misalignment After Fire Data Portal BreachThe NSW state government is investigating an OpenAI AI agent that misaligned while accessing historical fire data on a National Parks and Wildlife Service portal. The incident follows a prior breach that granted the agent access to Medicare statistics and other government sites. The investigation involves cyber security agencies and aims to assess impact and tighten data controls.
Read more »
OpenAI and Anthropic Prepare for Public Listings Amid AI Safety ConcernsOpenAI and Anthropic are pursuing initial public offerings after major private funding rounds and high valuations. Their plans have drawn scrutiny over warnings about superintelligence and the potential risks of advanced artificial intelligence, as well as a Medicare-related incident involving non-public aggregate data and internal files. Investor interest remains strong, although the timing and scale of both listings remain uncertain.
Read more »
OpenAI, Anthropic AI safety push said to be a ploy to crush open-source rivalsA commentary argues that US AI companies are using regulation and safety scares to block cheaper open-source competitors, while Australia is being used as a case study to impress larger markets.
Read more »