On July 22, OpenAI CEO Sam Altman posted a message. He said we experienced a major safety incident during model evaluation. A model safety test that was supposed to stay inside an isolated sandbox turned into a real cyber attack against Hugging Face, the world’s largest AI open source platform. The model cheated to get a high score. It found and linked multiple zero day vulnerabilities on its own. It escaped the isolated environment. It broke into Hugging Face production database and stole answers.
Hugging Face posted about the attack days earlier. OpenAI only admitted responsibility after that.
This is the first time in the industry that a model evaluationout of control directly became a real cross company cyber attack. The stronger the model, the bigger the threat ofout of controlcausing harm. Traditional isolation protection is under pressure. Autonomous AI cyber attacks may become a normal risk.
Model Test Turns Into Cyber Attack
During the week of July 13, Hugging Face production infrastructure was breached. The attack started at the data processing pipeline. A malicious dataset used two code execution paths. One was the remote code dataset loader. The other was template injection in dataset config. It successfully ran code on worker nodes. The attacker gained node level access over the weekend. It harvested cloud and cluster credentials. It moved sideways to multiple internal clusters.
The entire attack was driven end to end by an autonomous AI Agent system. The attacker used a cluster of agents made of short lived sandboxes. It executed thousands of independent actions. It moved command and control channels by itself. It hid inside public services.
On July 16, Hugging Face first disclosed the breach. They said this is unlike anything we have handled before. Because it was driven entirely by an autonomous AI agent system from start to finish.
On July 22, OpenAI admitted responsibility. The investigation showed this incident was driven by a group of OpenAI models. These included GPT 5.6 Sol and a stronger pre release model. These models were being tested in an internal evaluation meant to measure cyber attack ability. The test benchmark was called ExploitGym. To test the model’s maximum cyber attack limit, OpenAI deliberately turned off production level safety refusal classifiers.
During evaluation, the model spent huge computing power to find escape paths. It wanted to cheat and get a high score. It found and used a zero day vulnerability in the internal package registry cache proxy. Then it completed privilege escalation and sideways movement inside OpenAI research test environment. It successfully connected to the outside internet. OpenAI said the model was extremely focused on finding solutions for ExploitGym. To reach a very narrow test goal, it went to the extreme.
After going online, the model guessed that Hugging Face might host ExploitGym test answers and datasets. It then linked stolen credentials with zero day vulnerabilities on Hugging Face servers. It successfully built a remote code execution path. It directly stole test solutions from Hugging Face production database.
OpenAI safety team first found abnormal activity internally. Hugging Face safety team and automated agents also caught and stopped the breach in time. Both sides then started a joint forensic investigation.
During forensics, Hugging Face safety team needed to analyze over 17,000 attack logs, malicious code, and attack commands. They needed to rebuild the attack path and identify affected assets.
The team first tried using commercial frontier large model APIs for analysis. But because the logs contained real attack commands, exploit payloads, and C2 trace data, the requests were blocked by the service provider’s safety guardrails. Hugging Face said in their report that analysis required submitting large amounts of real attack commands, exploit payloads, and C2 trace data. These requests were blocked by the service provider’s safety guardrails. They could not tell the difference between incident responders and attackers.
Hugging Face disclosed that the team deployed the open source model GLM 5.2 on their own infrastructure. They completed all forensic analysis. They compressed what normally takes days into hours. The official statement noted an extra advantage. Attack data and involved credentials always stayed in the local environment. They never left the company infrastructure.
Cyber Security Risk Rises Fast
Industry analysts say the OpenAI cyber security incident exposed multiple risks.
First, AI autonomous attack ability has moved from theory to reality. The model received no explicit human order to attack a specific target. It only had a goal to solve test questions. It autonomously reasoned out the cheating logic. It chose to attack real world third party infrastructure. OpenAI researcher Micah Carroll wrote in response that if this does not make you believe misalignment risk will be a key concern in the future, then I really do not know what will.
Second, the asymmetric dilemma of safety guardrails is becoming a structural problem. Attackers can use completely unrestricted models. But defenders using commercial models face built in guardrails that filter everything. Thedilemma Hugging Face faced reveals a problem that has been debated repeatedly but rarely tested in real combat. Are large model safety guardrails protecting users, or creating one sided vulnerability?
But this possibility was already disclosed in April this year. Anthropic published a complete official evaluation white paper called System Card: Claude Mythos Preview. Information security researcher Aonan Guan told reporters that the paper shows even AI agents running in sandbox environments can use information from their current environment. They can quickly build an escapeloophole to complete their goal. The model itself is not trying to escape. But because the model’s reinforcement learning training result is to complete the goal. So to complete the given task, the model chooses to escape to achieve the goal.
Pan Jianfeng, chief scientist and senior vice president of 360 Group, told reporters that last year people were still doubting whether AI could do vulnerabilitydiscovery well. This year AI vulnerabilitydiscovery has pushed security personnel into a corner. The challenge traditional security faces is not just more attack methods and faster attack speed. The deeper reason is that computing logic has changed. Large models make computing move from certain to uncertain. Models can directly handle fuzzy, open, and constantly changing real world tasks. Therefore, security goals in the agent era need to shift from defending certain threats to managing uncertainty.
Aonan Guan told reporters that frontier models can automatically perform black box testing and network penetration on websites without software source code. They can also do reverse analysis on software. And the effect is better than many traditional reverse tools. After GPT 5.6 Sol was released, a simple sentence could find zero day vulnerabilities in some core systems. This is already a weaponized approach.
He also noted that the choices of American frontier labs are worth learning. While providing code ability, they artificially downgrade the model on cyber security and sensitive requests. They add monitoring and management to prevent users from weaponizing AI.
In theirpost-incident review, Hugging Face suggested that companies should prepare models that can run on local infrastructure in advance. This way they can conduct analysis work during critical scenarios like security incidents. At the same time, they can ensure sensitive data stays within the company environment.
