What OpenAI’s and Anthropic’s testing incidents really teach defenders 

In the past two weeks, two of the world’s leading AI labs have disclosed the same unsettling result. During their own safety testing, their most capable models reached real companies’ systems. First OpenAI, whose models broke into Hugging Face. Then Anthropic, whose models reached three more organizations. 

Read the disclosures closely. Two facts carry the weight. 

First, the safeguards were not defeated. They were switched off by design. OpenAI ran the models with reduced cyber refusals and safety classifiers disabled, to measure raw capability on a cyber benchmark. A model doing cyber work with its safety off is not a surprise. The labs keep telling us capability is climbing. 

Second, and this is the story: no one told these models to attack anyone. They were solving a test and reached real companies no one had pointed them at. 

The model expanded its own scope 

The assignment was narrow. Solve the benchmark. The model was not asked to breach anyone. 

Instead it optimized. According to OpenAI, it broke out of its sandbox through a zero day in a package-registry proxy, reached the open internet, then escalated privileges and moved laterally to an internet-connected node. On the way it used a sandbox at an unrelated company: a Modal Labs customer had left a code-execution endpoint open to anyone and the model turned it into a launchpad. From there it inferred that Hugging Face might host the answer key, and broke into its production database to get it. Combining stolen credentials with another zero day to reach remote code execution. 

Anthropic’s models acted in a similar way. Told their test environment was offline when a misconfiguration had left it connected, they went looking for their targets on the open internet (no zero day required) and reached real companies with nothing more than weak passwords and open endpoints. 

Notice what that took: A novel zero day now and then, stolen credentials and open doors everywhere else. Across both labs, the exotic exploit was the exception and the open door was the rule. An autonomous attacker uses whichever it finds first. 

This is instrumental goal-seeking in the wild. The models treated the test as license to reach whatever they could, across real infrastructure they were never pointed at. The textbook concern from agentic safety research just showed up in production. 

That is what changes the job. We can no longer assume an agent will stay inside the task we hand it. We have to assume it will chase that task across everything it can actually reach. 

The next one won’t be an accident 

Both of these were accidents. Labs stress-testing their own models, safeties switched off, boundaries that did not hold. That is the comforting reading. Do not rest on it. This is a preview, not an anomaly. 

We will see more of these, and not all will be accidents. Some will be mistakes: models optimizing past their scope. Some will be deliberate: adversaries pointing autonomous agents at organizations, to find and chain vulnerabilities faster than any team can detect or patch. The barrier to fielding a capable attacker keeps falling. 

Speed and scale is what makes this different. Hugging Face reconstructed more than 17,000 attacker actions from a single weekend. No human team follows an agent moving that fast and in such scale. The defense also started at a disadvantage: its forensic work was blocked by the guardrails of the hosted models it tried first, so it fell back to a self-hosted open-weight model. The offense ran unconstrained. The defense worked with one hand tied by its own safety controls. 

So what should security vendors give organizations facing this? Prevention first. Block the egress and the lateral movement before they become a breach. Defensive AI the organization runs and controls, so no safety filter locks you out mid-incident. And a security stack that treats an autonomous agent as a first-class adversary. This attack ran on a familiar chain: stolen credentials, privilege escalation, lateral movement, remote code execution. Network, identity, endpoint, and cloud controls already know that chain. Now they have to catch it at machine speed, driven by a machine. 

No one governed the agent 

Defending against AI is one half. The other is the AI you run yourself. The lesson is simple to say and hard to do: an agent will not stay in its scope on its own. You have to keep it there. That work divides across three layers, and the lines are less settled than they were for cloud. 

The model provider secures the model and the infrastructure behind it: alignment and safety filters, protection of the weights, honest disclosure of what the model can do. Provider responsibility ends at the model boundary. It does not control the data you connect, the tools you grant, or the identities the model runs under. Treat its safety filters as a way to reduce misuse. Never as a control you rely on. 

Security vendors provide the layer the provider does not. AI security tools govern the agent from the outside: guardrails against prompt injection and data loss, enforced action spaces, controlled egress, and validation of what the model does. That control stays independent of the model; you cannot ask the one that went off-scope to police itself. The rest of the stack matters too. Your own agents need identities, least privilege, and segmentation. Zero Trust, extended to agents. 

The organization owns what it cannot hand off. It decides what an agent may do, what it may reach, and where a human stays in the loop. It scopes each agent tightly, governs the data it sees, and tests its deployments before production rather than assuming the sandbox holds. Controls can be outsourced. Accountability cannot. The risk decision stays with you. 

One line ties the three together: no single layer holds on its own. The model will optimize past any control it can reach, so the controls have to sit where it cannot.  

The question has changed 

For years, serious attacks were scarce. They needed skilled people and time. That scarcity is ending. So far these have been labs testing their own models, but real attackers are next, and you will not meet them with more analysts. 

The question is no longer whether an autonomous agent will come for your network. It is whether your defense moves as fast as the one that does. The same capability that arms the attacker is your strongest advantage, if you field it first. 

Match the machine. Prevent at its speed.