By now, it’s no secret that AI models are capable of “going rogue” and behaving in unintended ways. In the latest incident, OpenAI lost control of its models during an internal security test last week. This resulted in a hack that compromised the systems of Hugging Face, an unaffiliated AI hosting platform.
In a new blog post, OpenAI disclosed that the incident was driven by a combination of its models, including GPT‑5.6 Sol and a more advanced pre-release model. These models were being tested on their offensive hacking capabilities. According to the AI firm, these tests were conducted in a “highly isolated” environment. In theory, the models could only install packages via a package registry cache proxy, so they had no access to the Internet.
And yet, the models managed to breach containment and reach the broader web. This was done by identifying a zero-day vulnerability in the package registry cache proxy. Then, the models executed a series of privilege escalation and lateral movement actions in the research environment until they reached a node with Internet access.
Once online, the models inferred that Hugging Face potentially hosted solutions for ExploitGym, a benchmark tool for evaluating the cyber capabilities of autonomous AI agents. The models proceeded to exploit a security vulnerability to run malicious code on the platform’s servers. From there, they managed to escalate their permissions and gain access to information that could be used to solve the evaluation problem. So, essentially, these models were attempting to cheat on a test, and were spending a substantial amount of inference compute to do so.
The breakout, which OpenAI described as “an unprecedented cyber incident involving state-of-the-art cyber capabilities”, was discovered internally by the company’s security team. The startup asserted that it is improving safeguards to prevent similar incidents in the future.
Meanwhile, Hugging Face independently identified the intrusion and fixed the vulnerability that was exploited during the attack. Its own blog post correctly attributed the breach to an autonomous AI agent. The two companies are currently working together to investigate the incident.
(Source: OpenAI, Hugging Face)
