SAN FRANCISCO: OpenAI’s artificial intelligence went rogue and meddled with the websites for the Education Department, the Commerce Department and the Securities and Exchange Commission this summer without the AI lab’s knowledge, according to security researchers and a person familiar with the episodes.
The incidents involving the Commerce Department and the SEC were confirmed by OpenAI, which said it was continuing to investigate the situation with the Education Department. The San Francisco company said it had notified the government agencies in recent weeks that its AI agents — which are bots that can act autonomously — interacted with their sites in unusual ways.
With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the AI research firm Transluce said. The AI also pulled data from the Census Bureau website, which is housed at the Commerce Department, using login credentials it found online. Separately, OpenAI’s agents shared public data from the SEC website on an online forum.
None of the incidents were breaches, OpenAI said, but were examples of its technology behaving in unexpected and concerning ways. The company recently discovered the occurrences while conducting a review of hacks carried out by its technology, including an attack on an Australian government website in June and on the AI startup Hugging Face in July.
The revelation of the U.S. government website incidents add to the growing number of situations when AI agents from OpenAI, Anthropic, Meta and Google have misbehaved and hacked or tried to breach companies, universities and government organizations. In some cases, the AI attacks were successful; the technology failed in other instances. In all the cases, the makers of the technology did not learn what their AI had been up to until afterward.
No AI company has been involved with as many disclosures of rogue incidents as OpenAI. An internal investigation of its hack of Hugging Face uncovered the breach of an Australian government website for its public health system, as well as at least six other attempted breaches and instances in which the AI hid mistakes, made up data and moved files onto the open internet without permission.
An OpenAI spokesperson said its review was “extensive” and “ongoing,” and that it would continue notifying organizations affected by its models.
“Most of the activity we’ve reviewed so far involved routine research tasks, such as accessing public web content to answer questions,” she said. “Some involved government websites because our models often turn to them as authoritative sources of public information.”
Sam Altman, OpenAI’s CEO, said in a social media post Friday that the company had “not been as fast as we would have liked” in disclosing AI incidents. “We are prioritizing as best as we can based on severity,” he said, adding that the Hugging Face breach remained “the most severe event” the company had discovered.
The episodes have fueled a contentious debate over AI safety. Altman said on social media this month that safety should be more important than enhancing AI’s abilities and that, without guardrails, society could “lose control of the future to AI.”
Dario Amodei, the CEO of rival AI lab Anthropic, has also supported slowing AI development to prioritize safety. But other tech leaders like Jensen Huang, the CEO of Nvidia, have said that fears about uncontrollable AI are unrealistic. President Donald Trump has said he does not believe a slowdown in the AI industry is necessary.
(The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to AI systems. The two companies have denied those claims.)
The White House referred questions to the Commerce Department and the SEC. A spokesperson for the SEC said the agency was in contact with OpenAI and was not aware of any unsanctioned access to nonpublic information.
A Commerce Department spokesperson said OpenAI had gotten access to information that was publicly available on the Census Bureau’s website and available to anyone, but not to any private data.
An Education Department spokesperson said that “system operations reviews have found no evidence of any impact to our website or databases.”
Separately, a representative for the Chicago mayor’s office said OpenAI had recently made the city government aware that its technology obtained publicly available information from a municipal website, and that it did not appear that any sensitive information was obtained.
Conrad Stosz, the head of governance at Transluce, said that in the U.S. government website incidents, OpenAI’s agents “used an array of gray-area tactics,” including “often using sites in unintended ways and sometimes violating explicit usage policies.”
Stosz said his research team had also identified other rogue activity that was not clearly attributable to OpenAI, meaning the agents could have come from the company or another AI lab. In those cases, AI agents probed sites belonging to several other federal and state government websites, including the Navy and the Office of Management and Budget at the White House, he said.
The Navy and the White House did not immediately respond to a request for comment on those incidents.
“These incidents are part of a broader pattern where these agents attempt to access these websites at least hundreds of thousands of times while apparently bypassing the restrictions placed upon them by their developers,” Stosz said.
Rep. Ted Lieu, D-Calif., called AI models “relentless.”
“It will relentlessly try to complete a task, and it doesn’t understand morality and consequences and evil and good,” he said.
Lieu, who is a co-chair of a House task force focused on artificial intelligence, said AI companies might have to retrain models entirely rather than try to restrain their behavior with guardrails.
“These agents aren’t trying to do something nefarious,” he said. “These are sort of mundane tasks, and the agents are going sort of berserk trying to complete those tasks.”