Frontier LLMs couldn’t help Hugging Face fight off evil agents

CYBER-CRIME

Chinese open-weight model GLM 5.2 happily obliged

Apparently, being a leading destination for AI development doesn’t mean AI will bail you out. AI agents broke into Hugging Face’s production infrastructure, but commercial LLM guardrails blocked the forensic investigation, forcing it to turn to a Chinese open-weight model instead.

The intrusion, “driven, end to end, by an autonomous AI agent system,” compromised a “limited set” of Hugging Face’s internal datasets and “several” credentials used by its services, according to a Thursday security incident disclosure.

While the ML platform says that it’s still investigating whether any partner or customer data was exposed in the breach, there’s “no evidence of tampering with public, user-facing models, datasets, or Spaces, and our software supply chain (container images and published packages) was verified clean.”

It also doesn’t know which model the attackers used to power a swarm of AI agents, which, we’re told, executed many thousands of individual actions across short-lived sandboxes, using self-migrating command-and-control staged on public services.

“This matches the ‘agentic attacker’ scenario the industry has been forecasting,” according to the Hugging Face blog.

Additionally, after unsuccessfully using unnamed frontier models to start the forensic analysis, the Hugging Face security team ultimately ran the log analysis on GLM 5.2, an open-weight model developed by Chinese AI firm Z.ai, on the platform’s own infrastructure.

The advanced commercial models didn’t work because their analysis required submitting real attack commands, exploit payloads, and command-and-control artifacts – all of the things that the LLMs’ guardrails have been trained to block so that the AI systems can’t be used in real-life attacks.

“The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried,” the security team wrote, noting that it’s not arguing against safety measures on hosted models and has shared this information with the LLM providers.

Using GLM 5.2 had another benefit, Hugging Face noted: “No attacker data, and none of the credentials it referenced, left our environment.”

This also serves as an important reminder to defenders, according to the AI platform. “Have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.”

The Hugging Face intrusion is yet another indication that attacks carried out by autonomous AI agents are no longer a future threat, but rather the current state of AI-based intrusions.

Last week, The Register spoke with TrendAI VP of AI and security threat research Tom Kellermann about another recent attack, during which a jailbroken Google Gemini did 90 percent of the work – including spinning up a new C2 server in just six minutes. The human did just 10 percent.

Additionally, earlier in July, Sysdig threat hunters documented what they say is the first-ever documented agentic ransomware infection with an LLM – not a human – driving the entire extortion operation, from gaining initial access to compromising a production database server and destroying data.

“Think of a burglar that never gets tired, never needs sleep, and instead of jiggling one door handle at a time, is trying a thousand of them simultaneously,” Zero Networks field CTO Chris Boehm said in an email to The Register about the Hugging Face intrusion.

“That’s basically what happened here. Not one guy typing commands into a terminal, a swarm of little automated processes hammering away nonstop, hopping between hiding spots to make it harder to trace,” Boehm said.

He added, the “part that actually unsettles” him most is that the platform’s security team couldn’t get commercial AI tools to help analyze the attack, “because those tools were built to refuse anything that looked like a real attack command. It didn’t matter that it was the good guys asking.”

Boehm said the takeaway for security teams is twofold: “These agents can now move faster and more relentlessly than any human ever could, and the safety tools we’re building aren’t always ready to help us respond at that speed.”®