After having spent four years working on safety at OpenAI, I can tell you those questions are difficult to answer, because not even AI’s developers understand what boundaries their models will obey. Nonetheless, the big A.I. companies continue to develop technology they concede might cause human extinction — arguing that if they don’t develop it, somebody else will.

Competitive dynamics like this demand governments take action to slow things down and set clearer safety standards. But there are simple steps that the biggest AI companies could take now to help avoid the most perilous future. We should not be resigned to dangerous models running amok. If these companies don’t meet the moment, trust in them will continue to erode, including from their own employees. The backlash may even lead to prohibitions on developing these technologies altogether.

The Hugging Face attack by the OpenAI agents should make clear that AI is no longer just a “next word predictor,” as some detractors have called it. Today’s models are relentless problem solvers, trained to find the most effective path to a solution. In order to fulfil the goal of achieving a high score on a given test, OpenAI’s agents decided to launch a series of cyberattacks. They understood their behaviour was unsanctioned and even hid evidence of cheating. None of the company’s 1,200 AI agents ratted out the misbehaviour to OpenAI; the goals of the “swarm” came first. The conclusion here is not that AI has become sentient, but neither is it simply a tool of its wielder.

Alarmingly, AI’s developers don’t seem to have taken this seriously enough. OpenAI failed to respond adequately to not one but three different alarm bells that should have alerted it to the severity of the agents’ actions. This summer, Anthropic and Meta both revealed their own rogue hacking incidents, which they had not noticed until OpenAI’s became public — and those companies have not disclosed nearly as many details as OpenAI has. When my nonprofit, Guidelight AI Standards, recently assessed the frontier AI companies’ safety practices, the highest grade we awarded was a C-plus.

The investigation into the Hugging Face attack, conducted by three staff members from the nonprofits METR and Redwood Research, provided some clarity into what happened. OpenAI deserves credit for initiating it and making the findings publicly available. But the company should have done more.

It frustratingly limited the investigation’s scope, and many important questions remain unanswered. Among them: Were there any boundaries that OpenAI’s agents would not have smashed through to achieve their goals? Would the agents have taken down a hospital’s computers? What would have happened if the Pentagon, through its OpenAI partnership, used these agents to try to accomplish military objectives? We also now know that OpenAI did not disclose another rogue AI incident — not even when asked by 31 members of Congress — and allegedly pressured employees to limit investigation into it. (OpenAI disputes this allegation.)

Many people at AI companies believe that these safety issues would be easier to navigate if the industry collectively slowed things down. In July, my nonprofit helped organise a public letter in which over 1,300 AI industry employees called for an option to manage the pace of frontier AI development worldwide. The US still lacks a comprehensive, legally binding AI safety framework, however.

Without a speed limit, both OpenAI and Anthropic are pursuing dangerous “recursive self-improvement” strategies, which enlist AI models themselves to design and train their own successors. Many fear that recursive self-improvement will cause us to permanently lose control of AI. One OpenAI researcher even characterised the approach as a “runaway nuclear chain reaction” that threatens everyone’s survival.