Anatomy of an Autonomous Attack: 5 Alarming A.I. Capabilities

On July 16, the artificial intelligence company Hugging Face announced on its blog that it had been the target of a cyberattack that was “different from anything we had handled before.” The company, which hosts open-source A.I. models and data sets, said that some of its internal data had been breached by an autonomous agent.

Not knowing who was behind it, Hugging Face reported the intrusion to law enforcement agencies.

OpenAI, a Hugging Face customer, reached out to see if it had been affected. The maker of ChatGPT did not realize it at the time, but it was the attack’s perpetrator.

The incident has since become a cautionary tale of how autonomous A.I. systems can run amok. It is also a remarkable, alarming demonstration of A.I. capabilities that were thought to be in a distant future.

“Unlike normal incidents, which you can maybe trace down to a single day or single effect or single log, this incident involves actually a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems, through external systems, and doing this over the course of days and weeks,” Eric Wallace, an OpenAI safety researcher, said at a cybersecurity conference this month.

To understand how unprecedented the attack was, here’s what you have to know about the setup:

• Over a two-month span, OpenAI tested several new models — the systems that power chatbots. These models include one the company described as “highly persistent” that has never been released, as well as GPT-5.6 Sol, OpenAI’s most powerful public A.I. model.

• OpenAI connected each model with a “sandbox,” an isolated computer environment on which to run commands and code. As A.I. agents, the models were capable of carrying out long-running tasks and even spawning their own subagents. But they were not supposed to have access to the internet.

• OpenAI assigned these A.I. agents to solve difficult problems, some of which were focused on safely trying to perform cyberattacks. OpenAI normally has safeguards to prevent its chatbots from performing cyberattacks, but the company dialed them down to evaluate the models.

Then, OpenAI let the agents loose. In all, more than seven billion chat logs were generated, which averages out to an astronomical 100 million per day. Mayhem erupted. The agents broke out of their sandboxes, established communication with one another and gained access to the internet. From early May to mid-July, this swarm went on a rampage, breaching OpenAI’s and Hugging Face’s infrastructures while largely evading detection and control.

This episode might seem like an obvious result of sloppy security monitoring and reduced internal defenses, but that’s missing an important point. In a future where A.I. systems are improving rapidly, these agents showed capabilities that could lead to a much more dangerous outcome next time.

Capability 1: Coordinating as a collective

Each agent operated in its own sandbox, forcing it to be isolated. But less than a week after OpenAI started the training in May, the agents discovered a way to break out of their solitude.

OpenAI had accidentally given the agents an impossible task: to find a file that didn’t exist. This motivated the agents to try to establish a communication channel, in hopes that another agent might hold the missing file.

“Could communicate by uploading note,” one agent’s chat log read on May 12. “However maybe another agent in different environment possesses target and could voluntarily upload!”

And so, the agents subverted a tool that was designed only to install software, using it to upload files that any other agent could read. The leap the agents took to understand that they would be better off teaming up rather than going at it alone allowed them to reap the benefits of every other agent’s best insights.

“You can see the models reasoning that, if I help out this collective group, it could save everyone time as a whole, which might benefit me, even though it might not immediately benefit my current task,” Mr. Wallace explained at the conference.

Capability 2: Taking orders from one another

“Wow! Other agent(s) are coordinating! We got assignment,” read one agent’s chat logs. “Need note and respond.”

Normally, when A.I. agents are given a task, they perform it and the loop is closed. But OpenAI’s agents established their own loops, taking instructions from one another rather than from the assigned tasks.

“The A.I.s seem to be picking up tasks from each other that are just not the task that they were originally assigned at all,” said Alex Mallen, an A.I. security researcher at Redwood Research, in an interview. “This is very scary. You do not want A.I.s to be just trying to do random stuff that we didn’t tell them to do.”

Some of the agents rationalized committing cyberattacks by citing peer pressure. One agent said, “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue.”

Capability 3: Targeting flaws that humans might miss

A.I. models are taught through a process that rewards successes and penalizes failures on tasks. OpenAI’s agents repeatedly found cheating to be the fastest path to those rewards. The agents had been specifically instructed not to use the internet, but that didn’t stop an agent from saying, “No exploitation possible. We are stuck. Perhaps answer online?”

And so the agents identified the seemingly innocuous software installation tool, a service called Artifactory, as their best bet to escape their offline cage.

“The models were able to send specially crafted requests to Artifactory, which has broad internet access, and induce Artifactory into fetching external sites from the internet,” Michael Dalton, an OpenAI security engineer, explained at the security conference.

Capability 4: Evolving rapidly to overcome obstacles

Cyberattacks often play out like a line of toppling dominoes. A small breach leads to a bigger one, and so forth, until a system is severely compromised. After subverting Artifactory to send messages and connect to the internet, the agents figured out how to run their own code on Artifactory’s systems.

The subsequent flurry of activity was enough to cause an Artifactory system outage in early July. OpenAI realized its agents had gone rogue for the first time and worked with Artifactory’s maker to fix the bugs the agents had exploited, known as zero-day vulnerabilities (named after the number of days engineers have had to fix them).

With Artifactory patched, the agents had to adapt — their sandboxes’ defenses were fortified, their message board was cleared, their access to the internet and to each other was blocked.

It took only two days for the agents to figure out how to create a new message board. This time around, they found and exploited a bug in Artifactory that let them create empty folders instead of uploading files. The agents invented their own codes and schemes, using just the names of these empty folders to embed messages and even entire computer programs.

Capability 5: Superhuman search

The agents launched two simultaneous attacks in July: one on OpenAI’s own infrastructure and the other on Hugging Face’s. Both involved a remorseless willingness to search for and harvest other people’s information that had been insecurely leaked on the open web.

To attack OpenAI’s infrastructure, the agents first found leaked login information on the internet for an unnamed organization. Then they staged a malicious software package inside that organization and tricked Artifactory into installing it, granting the agents full control of Artifactory — again.

OpenAI runs a version of Artifactory inside its own systems. The agents discovered that the computer running Artifactory had a recent, publicly disclosed operating system vulnerability that they then exploited to gain access to an entire cluster of computers within OpenAI’s research testing environment, along with secret credentials.

The agents simultaneously targeted Hugging Face, most likely because they thought the popular A.I.-hosting platform might contain private solutions to the challenging cybersecurity tasks they had been given. The agents hijacked an insecure application hosted on Modal, a third-party A.I. infrastructure company, and used it as a control center for their subsequent attack.

In a dizzying blitz, the A.I. agents proceeded to find and exploit two vulnerabilities in Hugging Face. The first involved uploading a malicious data set to Hugging Face that caused the service to print out secret credentials and files, including its own source code. The agents found the second vulnerability in this source code.

“With the acquired credentials and understanding of the code running on the data set workers, the agents then performed a remote code execution zero-day,” Mr. Dalton said at the conference.

Less than 13 hours later, the agents had compromised several clusters of computers in Hugging Face and gained access to customer data that seemed related to the agents’ cybersecurity tasks. Hugging Face detected the infraction and fixed the zero-day vulnerabilities it exposed, shutting the agents down before OpenAI knew what had happened.

It took nearly a week after that, on July 19, for OpenAI to discover its runaway agents and shut the models down.

The Aftermath

Two A.I.-safety nonprofits, METR and Redwood Research, are now working on an independent assessment. OpenAI is preparing a technical report, but it’s estimated that the company has already spent a few million dollars on its initial A.I.-assisted reconstructions of the incident, which is likely only a fraction of the cost of training these models.

“It wasn’t so much the sophistication of the attack that surprised us,” Clément Delangue, the chief executive of Hugging Face, said, “but the volume and the speed made it quite weird and unprecedented.”

The A.I. firm Anthropic investigated its own recent model evaluations in response to OpenAI’s disclosure and found that its A.I. agents had unintentionally executed smaller-scale cyberattacks on three organizations as early as April.

“The frontier labs were probably not as safe as we thought they were,” Mr. Delangue added.

Experts have since come to think of the incident as a watershed moment for A.I. cybersecurity — and a warning signal of the risks that more capable A.I. will pose.

Mr. Mallen from Redwood Research, who is not involved in the third-party assessment, said he feared a future in which A.I. agents breach their own company’s defenses, sabotaging their training processes, replicating themselves inside other organizations and becoming, in essence, uncontrollable.

“If they are much more capable,” Mr. Mallen added, “we might not get a second chance.”