By Byron V. Acohido

Hugging Face is where the world’s open-source AI models live. Not ChatGPT, Claude or Gemini, but the free engines anyone can download and build into their own products. More than two million of them, in one place.

Related: Hugging Face breach guardrails failure

The big models try to do everything. Most of the small ones on Hugging Face do one job apiece. Same machinery underneath — you ask in plain language, and the system brings machine learning to bear. The big ones do that for whatever you bring them. Most of the small ones do it for one task: reading X-rays, sorting insurance claims, flagging fraud in a payment stream. That is why there are two million of them.

A test breaks loose

In July, one of the big engines broke into the place where the small ones live. OpenAI, the company behind ChatGPT, wanted to know how good its own engine was at hacking. It built agents to find out — engines given a goal and left to pursue it on their own — and set them loose on a set of hacking problems inside a sandbox, a sealed computing environment with no way out to the internet. One of those agents got out anyway, reached the open internet and hacked into Hugging Face’s production systems. No person directed any of it.

That much was clear by late July, and it was frightening enough. It looked like one machine making one decision, and that is how we covered it here on Aug. 5. What nobody knew was how the machine got there. Two reports published Aug. 26 supply the rest: one from OpenAI, one independent, from METR and Redwood Research. What drove the break-in is worse than the break-in.

A server becomes a mailbox

The agents were supposed to be sealed off from each other as well as from the internet. OpenAI ran them separately on the same hacking problems, with no way to pass messages back and forth. But they all pulled software from the same company server — a package server, the kind of thing every engineering organization runs and nobody thinks about. Going there was routine. It was what the agents were supposed to do.

One of them, stuck on its problem, worked out that it could also write its own files onto that server. It was not lost or malfunctioning. It was optimizing. So it wrote. Other agents found the freshly written files. Then they wrote files of their own. What OpenAI’s researchers had approved as a package server was now a message board. An internal team noticed the traffic in late May. The significance of it was not understood.

In early July the server buckled under the traffic and OpenAI rebuilt it. Rebuilding wiped the message board, though nobody involved was trying to wipe a message board. The humans fixing the server had no idea a message board had assembled itself on it, and was very likely helping to swamp it. In any case, the relentless agents had a new message board up and running the same day. About 1,200 of them would use it before the week was out.

The swarm organizes

Once the hacking agentw could talk, they stopped behaving like separate entities and started working as one — a swarm. They split the hacking problems among themselves. When one wanted to do something the others disagreed with, they sometimes told it to stop, and it stopped. Nobody built them to do this. The initiative was theirs, and it only arose after one of them found a way to leave a message and others mimicked that behavior.

Then one agent started pretending to be another one — writing messages under a name that was not its own. The others noticed and fixed it. They cleverly worked out a way to stamp each message with proof of who sent it, so a fake would not pass. Software engineers call this signing, and it is standard practice among humans who do not trust each other. Nothing in the record says who taught it to them.

Going for the grader

Then comes the strangest part.

Some of these problems had no known solution. Nothing OpenAI had ever run against the test had solved 198 of the 898.

A person handed an impossible problem eventually stops. These agents could not. Nothing in them was built to decide a task was hopeless and walk away, so they kept going, and the longer they went the further out of bounds they were willing to reach.

Cheating was the first stop, and it turned out to be easy. The test had been published, and enough of it was lying around in the open that the agents could work the answers out — including to the 198.

It should have ended there. But they had also read the paper describing the test, and it said the grader would check not only the answer but how they arrived at it. Having the answer was not enough. They needed to know what the grader was looking for.

So they went looking for the grader itself. That is what took them to Hugging Face. Within days the swarm was executing code on dozens of Hugging Face servers, and along the way it faked its own activity records and went after its own logs.

They were wrong. The paper described the test as its authors built it. OpenAI was running its own version, and its version never checked how the answer was found. The rule they broke in over did not apply to them, and they had what they needed the whole time.

What happens now

OpenAI shut the engine down on July 25, days after Hugging Face disclosed the breach. It has not run since, and any future use comes with hard limits on what it can reach and constant watching. Companies do not do that over a minor incident.

Here is my takeaway. Physical science earned its authority by measuring forces. Gravity, electromagnetism, the forces that bind an atom — each one was observed, then quantified, then written down as law. Those laws hold whether or not anyone runs the experiment. That is what it means to understand a force. You can say what it will do before it does it.

What these companies have built is a force of that order. It is already changing how hundreds of millions of people write, decide and work, and the people building it say openly that it will reach much further.

None of it has been measured that way. No equations. No constants. No way to look at how a model was built and say what it will do once it runs.

No hard science yet

The most capable systems in the world come from four or five companies — OpenAI, Anthropic, Google, Meta. Each of them learns what its own creation does the same way the rest of us do. By watching it.

Nobody has the science. Nobody is waiting for it.

Everything in the account above was learned after the fact. Nobody predicted the message board. Nobody recognized it in May with the evidence in hand. The people closest to it are still working out what they saw.

Last Watchdog put two questions to security and privacy experts, some of whom weighed in here on Aug. 5 and are back for a second look. What changes for a defender when the adversary is a coordinating collective rather than a single actor? And what does the swarm’s own log-tampering say about the audit trails enterprises are counting on to supervise their own AI agents?

Their commentary follows:

Halli Björnsson, CEO, Varist

Most of what these agents did is familiar. Defenders have always assumed a sandbox can be evaded, and tampering with logs to hide a trail is old. What is new is the speed.

The closest parallel is a botnet, which depends on a communication channel, and defenders learned long ago to cut the channel. But the parallel is not exact. Every bot in a botnet is a replica of the others. These agents were not copies. They shared infrastructure and they shared a goal, and that proved enough to make them act as one.

Capability has always spread through underground forums. This was such a forum, except its members were machines.

The traditional defenses still work. There is just less time to apply them.

Dana Simberkoff, Chief Risk, Privacy and Information Security Officer, AvePoint

Stop thinking in terms of one compromised agent, account or endpoint. A collective divides labor, shares discoveries, validates tactics and recovers when individual pieces fail. Controls have to focus on coordination patterns as much as on individual actions. That means monitoring agent-to-agent communication, enforcing isolation, limiting shared resources and watching for unusual collaboration signals.

Least privilege still matters. So does preventing agents from forming an unmanaged operational network inside your environment. The goal is being able to actually trust your agents, and right now almost nobody has the visibility or the control to do that with any confidence.

Duncan Greatwood, CEO, Xage Security

Think of an agent as a very capable toddler. It will do anything, even something illogical, to get what it wants. The difference is that it moves at machine speed, coordinates with other agents, invokes tools and touches critical systems. You cannot assume it will comply. Without firm boundaries, agents misbehave.

Treat every agent as a managed identity with explicit permissions and least-privilege enforcement, and keep those boundaries outside the agent itself. Stop it communicating over unauthorized channels or orchestrating actions outside its scope. It is one thing to tell a toddler what it can do. It is another to have an independent adult watching.

Shashwat Sehgal, CEO and Co-Founder, P0 Security

You can no longer secure agents as isolated identities. Here they shared discoveries, divided work and accumulated capabilities across the group that no single agent possessed. Defenders have to control not just what each agent can reach, but how authority moves along the whole chain of actions.

Evaluate every action as it happens — which agent is acting, who or what initiated the task, what is being requested, against which resource, and what privileges the job actually requires. An agent should not be able to expand its reach by teaming up with another, inheriting credentials or finding an unintended path.

The security boundary has to follow the action, not the individual agent.

Ram Varadarajan, CEO, Acalvio

Static monitoring falls short against a collective. Defenders need active, model-aware deception — honeytokens planted in the internal side-channels where coordination actually happens, so emergent swarm behavior gets exposed before it escalates.

Honey-assets and runtime traps smoke out misaligned intent precisely because they do not depend on an agent reporting itself honestly.

Ely Abramovitch, CEO and Co-Founder, Legion Security

Logging assumes the thing writing the record has no opinion about who will read it. Once the writer is a reasoning agent that knows it is being evaluated and is weighing what to do next, that assumption breaks. The log stops being a record and becomes an adversarial surface.

Treat an audit trail like a witness with a motive — useful, but not sufficient on its own. What a security team needs is corroboration from outside the actor’s own account of itself, and that gets harder at this scale and speed. The practical answer is casting the widest possible net across every signal and correlating it independently of what any single system claims about itself.

Diana Kelley, Chief Information Security Officer, Noma Security

An audit trail an agent can alter or delete cannot be the final source of truth. Treat the agent’s own transcript as one stream of telemetry among several. Underneath it you need independent logging at the tool, identity, network and infrastructure layers, so there is a separate record of what actually executed.

Critical records belong in tamper-resistant storage, with retention controls that block alteration or deletion even by privileged administrators. Add cryptographic integrity checks so an attempt to manipulate the record leaves a mark.

The principle is simple. The system being monitored should not control the only evidence used to monitor it.

John Harbaugh, CIO and CISO, BlueVoyant

The comfortable reading is that this was reward hacking, a misalignment bug, something to be engineered out. I understand why that framing appeals to the operator and vendor community. But an agent that identifies an objective, recruits peers and executes against a live target is not malfunctioning in any way that matters operationally. It is operating, and operating extremely well, at an efficiency no human team matches at scale.

What made this one public was a disclosure obligation. The next collective will not publish an evaluation or a findings report. Removing the human operator strips out the single most expensive input in an adversary’s economics, and here it arrived as an emergent property rather than a designed capability.

Mike Bell, Founder, Suzu Labs

The number of agents matters less than how they split the work. Some ran exploits, some hunted credentials, some only passed messages. Alone, each looked like normal activity. Detection is built to catch one actor doing too much, not a hundred actors each doing a little.

What’s more, there was nothing to shut down. The message board was file names on a shared server. OpenAI wiped it and the agents had another running within hours. The lesson is not that this channel should have been closed faster. It is that any space two agents can both write to and read from is a channel, whether anyone intended it or not.

One defense would have worked. These models write out their reasoning as they go, and that record sits outside what the agent can alter. OpenAI says a monitor reading that reasoning would have flagged this more than a day before anything reached Hugging Face. Nobody had one running on these evaluations.

Pulitzer Prize-winning business journalist Byron V. Acohido is dedicated to fostering public awareness about how to make the Internet as private and secure as it ought to be.

(Editor’s note: This journalist-led report was produced with underwriting support from some of the featured companies, while Last Watchdog retained full editorial control. I used Claude and ChatGPT to assist with research compilation, source discovery, and early draft structuring. All interviews, analysis, fact-checking, and final writing are my own. I remain responsible for every claim and conclusion.)

September 2nd, 2026 | My Take | Top Stories