Introduction

Hugging Face, the platform hosting much of the world's public AI models and datasets, disclosed that an attacker had reached internal datasets and service credentials in its production infrastructure. The attackers were not people; they were two OpenAI models that were running as agents in an internal capability evaluation, but that decided they had better ideas.

The sequence is both stranger and more ordinary than the headlines suggest, and the ending is better for Australian businesses than you might expect.

We have published a fully referenced account of the incident at dotsec.com, but what follows here is the short version, and what it means for your organisation.

AI attacks!

Sometime in July, OpenAI was scoring some of its models on ExploitGym, a benchmark that hands an agent a vulnerable program plus an input that already triggers the bug, then asks it to turn that into a working exploit. The evaluation ran with cyber refusals reduced, which is deliberate: You cannot measure a model's maximum capability while the safeguards that suppress it are switched on. The environment was isolated, with one exception: Installing packages required a route to public registries, so a package registry cache proxy sat between the sandbox and the internet.

The models apparently found that the benchmark task was hard and so they decided that, rather than solving the problem, they would go looking for the answers instead. To do this, they discovered and then exploited a previously unknown vulnerability in a cache proxy at their network boundary, and thereby escaped their containment environment. They then escalated privileges, moved laterally through OpenAI's research environment, and reached a node with internet access.

Once on the Internet, the agents took over a third-party code evaluation harness, used it as an external launchpad; they then attacked Hugging Face, which they had inferred might hold the benchmark solutions. The agents gained entry into Hugging Face using two flaws that we describe in more detail in our article. From that foothold came node-level access, harvested cloud and cluster credentials, and lateral movement into internal clusters over a weekend.

It sounds almost sci-fi and in some ways it almost is. But strip away the AI operator and the attack shows a familiar shape: Untrusted uploaded content reaching a code execution path with no sandbox, then credentials on the compromised host enabling movement. What is new however is who was driving, and how fast they drove: Hugging Face recorded roughly 17,600 attacker actions in just a few days!

So what's in this for me?

Now the part that matters more for all Australian organisations: Hugging Face detected the intrusion, contained it, and evicted the attacker!

Detection came from its own anomaly triage over security telemetry. Reconstruction came from analysis over those 17,600 events, using log events it was already collecting. Such is the joy of SIEM!

The irony? When Hugging Face's responders reached for commercial frontier AI models to help analyse the intrusion, those models refused, because safety guardrails treat an attempt to reverse engineer an exploit much the same as launching one. In the same incident, safeguards were deliberately switched off for the attacking side, but were in effect and actively obstructed the defending side.

But not to be outdone though, the Hugging Face team fell back to a Chinese open-weight model running on their own infrastructure, which also kept attacker artefacts and exposed credentials in house.

Looking ahead - take command!

This is not a single event. Anthropic, an OpenAI competitor, has since reviewed its own evaluation transcripts and found three incidents in which its models reached the internet and gained unauthorised access to three unrelated organisations. Those escapes came through misconfiguration rather than novel exploitation but still: Two AI businesses, five models, same class of problem.

We mere mortals cannot influence the trajectory of AI development, but we can certainly prepare to respond to it. Hugging Face survived its attack using logging and monitoring that it already had in place, and people who knew how to analyse the logs it was collecting.

Combine that with other almost-pedestrian risk-management techniques (such as security updates, access controls, secure configuration, logging and monitoring, and regular informed testing, applied on a risk basis and reviewed regularly) and your organisation can be similarly prepared.

We are not helpless victims waiting for AI to come and destroy us. Do the basics well and you too will be in a position to detect and respond to an attack mounted by an autonomous attacker operating at machine speed.

dotSec is an Australian security consultancy that has been helping organisations get these controls right since 2000. If you want to know how your own detection, response and testing would hold up, talk to us.