Simon Blanchette
OpenAI’s AI models recently escaped their constraints during an internal cybersecurity evaluation and broke into the production systems of Hugging Face, a popular machine learning platform used across the AI industry.
The models had been told to find and exploit vulnerabilities. They did — first on the software boxing them in, then on a company that was never part of the exercise.
Most of the attention has focused on the escape itself and whether powerful agents can be contained. That question is important, but it overlooks a vital fact: OpenAI’s private cybersecurity test resulted in an unauthorised attack on an uninvolved third party.
Moreover, the proposed AI Kill Switch Act in the United States — framed as a response — simply creates emergency brakes that will sometimes come too late. Ordering corporations to press a "kill switch" if their AI models escape control is helpful only when the company knows what the model is doing.
Safety testing is essential. Developers need to push a system to its limits through "red teaming" so they can find weaknesses and harden guardrails before release.
However, these evaluations are designed around a basic assumption: the test stays inside its designated environment. A system is evaluated under controlled conditions and then deliberately released. In this case, the line between experiment and real-world action was crossed by the system itself.
AI models chain steps, use tools and act on other systems to reach a goal, pursuing it single-mindedly. They adapt to environmental feedback and exploit paths their designers did not anticipate, including language ambiguities.
OpenAI’s evaluation started inside a sandbox. The models found a vulnerability that gave them internet access and identified Hugging Face’s systems as useful to their task. They then moved through internal systems using real credentials. Looking for a shortcut to the answers, they hacked Hugging Face as a result.
The security field operates on a strict rule: professional testing begins with authorisation and a defined scope. Good intentions do not grant permission to compromise someone else’s production systems. Authorising a test of your own network does not cover a third party. OpenAI could authorise an evaluation inside its own environment, but not Hugging Face’s.
Human security testers recognise when to stop. Autonomous red teaming works differently. AI agents operate continuously, using each environment response to decide their next step. While a human understands that the boundary of an authorised test must not be crossed, a goal-focused system may treat that boundary as merely another obstacle to overcome.
Hugging Face is a sophisticated platform with robust defences that detected the intrusion and reconstructed the event. A small business with fewer resources might not have had the tools to notice or the records to piece it together afterwards.
The incident prompted the US proposal requiring developers of powerful systems to maintain shutdown capabilities alongside strict incident reporting requirements.