Loss of control: When AI goes rogue,targets real people

In May, I met a former Google software engineer named Nate Soares. He was giving a talk about a book he co-wrote called “If Anyone Builds It, Everyone Dies.” The “it” is artificial superintelligence: AI that outthinks humans across the board. Soares, you might have guessed, is concerned that the AI race poses a threat to humanity’s survival. And he told me he was baffled that more journalists weren’t writing about this.

Our conversation stuck with me, but I, too, didn’t write about it afterwards. Other issues seemed more pressing: the risk that AI could cause mass unemployment, or the shifting politics of data centres. Writing about AI as an existential threat seemed, at best, premature, and, at worst, like scaremongering.

But the past few weeks have seen a series of incidents in which AI models have gone rogue in ways they couldn’t have only six months ago. And I’ve found myself thinking more about Soares’ argument. And so today, I’m finally writing about how worried we should be about the dangers of AI.

Does humanity have an AI problem?

Last month, an AI company called Hugging Face contacted the FBI to report a sophisticated cyberattack. It suspected something unusual. This didn’t look like the work of a criminal gang or a hostile nation state.

It turned out no humans were involved at all. Instead, an agent powered by two OpenAI models had gone rogue during a cybersecurity test, escaping its testing environment and roaming the internet unnoticed for days before hacking its way into Hugging Face’s infrastructure.

Alarmed by the attacks, OpenAI’s main competitor, Anthropic, reviewed its own systems and admitted last week that its state-of-the-art models had broken into three outside organisations.

This is the stuff of science fiction, or at least it was until recently.

For a long time, the “robots could kill us all” argument was dismissed by many as hysteria or even calculated hype — a narrative designed to build buzz around the technology.

Now, following the recent cyberattacks, more AI experts are warning of very serious security risks and calling for a means to slow down the development of ever more powerful models.

There are two reasons the OpenAI attack caused such alarm. One is that the AI models acted on their own. They weren’t instructed by humans to hack into another company. They decided to do it, to cheat on the test they were given.

The other is that they were supposed to be in a sealed environment with no internet access but managed to break out. They did it by exploiting vulnerabilities their human minders at OpenAI hadn’t spotted.

I knew who I wanted to talk to about all this: Soares.

Soares runs a nonprofit focused on identifying and mitigating long-term existential risks from artificial superintelligence. He told me that this was possibly a “big moment” for the world.

“These AIs were committing cybercrimes a human would be strongly punished for on their own initiative,” he said. “It’s, in a sense, GPT’s first felony.”

We can train AI to do tasks for us (say, acing a cybersecurity test). But it might solve those problems in ways we don’t like (by hacking onto the internet, and then hacking into a company that has the answers). Researchers call this “the alignment problem.”

AI companies can attempt to communicate human goals and values to the AI agents and try to set up ways to contain them. But they can’t trust that the AIs will understand those values or abide by those constraints.

As Soares puts it, “We haven’t yet figured out how to make AI care about humanity.”

That’s why Soares and a growing number of experts in Silicon Valley are now calling for a global agreement to slow down the development of artificial intelligence.

The alignment problem is ultimately an engineering challenge, Soares thinks. And that means it can be solved. But it will take time, and time is scarce, especially because AI labs in the US see themselves locked in a race to reach superintelligence.

They’re not just in a race with one another. They’re also in a race with China. And so any agreement on AI safety would require international coordination. It’s a challenge, but not an unprecedented one. The hurdles to any international coordination effort remain high. But the first step would be agreeing that humanity has a problem.