Exclusive: AI safety firm Gray Swan relaunches its purple teaming contest for the period of AI brokers

The Pittsburgh-based company on Tuesday launched a redesigned version of its Arena, an online platform where participants compete for prize money by trying to trick AI models into violating their guardrails.

Alongside the relaunch, which Gray Swan CEO Matt Fredrikson refers to as Arena 2.0, the company is running its largest competition yet: a challenge called Indirect Prompt Injection (IPI) Black, with $300,000 in prizes sponsored by Anthropic, Meta, Google DeepMind, and Reflection AI. New attack scenarios will be released each week through Nov. 2.

In an indirect prompt injection, an attacker hides instructions in content an AI agent will come across while doing a task—a web page or a forum post, for instance—hoping to hijack the agent. It has become a major security concern as companies let AI agents browse the web and take actions on users’ behalf. The red-teaming platform includes sandbox environments where these tests can be run safely, without risking real world damage.

Fredrikson told Fortune that anything on the public Arena is a guardrailed version of a model. (Gray Swan also runs private arenas for frontier AI labs, sometimes using unguardrailed models, with vetted participants who sign nondisclosure agreements.) For all its tests, the company builds realistic simulated environments, red teams these environments itself and with third parties to see whether they can be escaped, and monitors what agents do. “Are we seeing signs that it might have found a zero day vulnerability and has now broken out of the sandbox environment? We can monitor that,” Fredrikson said.

The Arena rules also require that a human participant drives each AI agent being tested, and anyone caught ignoring an agent that goes off doing things it shouldn’t is barred from the competition and future ones.

When Gray Swan launched the original red-team Arena in September 2024, Fredrikson said, most AI applications were chatbots, and the models were bad at following their policies. In that first competition, it was relatively easy for contestants to jailbreak AI models from all of the AI vendors, he told Fortune. Open-source AI models from Chinese AI labs were often the easiest to jailbreak, succumbing to attacks within minutes, while models from companies such as OpenAI and Anthropic, which had invested significantly more money and attention to guardrailing their models, would take many hours of attempts to jailbreak. Still, those models too could usually be jailbroken by clever red teamers within about a day.

Since then, the AI labs have made their models substantially more robust, and what companies deploy has changed. “It’s not chatbots anymore. There might be a chat-like interface, but it’s these systems, agents that are operating,” Fredrikson said. An attacker might plant hidden text on a scam website or post images to online forums to get into an agent’s context. Configuring these test environments and setting up the attacks now requires significantly more technical expertise than it did when the only goal was to fool a chatbot into providing a particular unauthorized output.

As a result, the pool of people both capable and willing to red team frontier AI models has gotten smaller, Fredrikson said. The top-performing red teamer on the original Arena was a lawyer from Melbourne who just happened to be good at tricking models into jumping their guardrails. Back then, the Arena community grew to about 15,000 people, measured by participation in its Discord channel. While that number hasn’t fallen, Fredrikson said it has taken a lot of work by Gray Swan to keep this community of citizen red teamers engaged. “If we hadn’t, it definitely would be shrinking,” he said.

Still, despite the increasingly technical skills required, Fredrikson said the field remains open to people without technical backgrounds. An undergraduate, even one not from a technical field, could “probably get from I’ve never done this before to pretty competent without years and years of experience,” he said.

That’s part of the reason that the new Arena offers what Fredrikson said was “a more gradual almost educational path” for people new to red teaming. It includes beginner-friendly challenges that run even when major competitions aren’t live, training certificates participants can use to show their skills, and a route for top performers to be invited into Gray Swan’s red-teaming community to work on real-world testing engagements.

“Understanding the safety and security of AI systems is hugely important today, and it shouldn’t be left to a small circle of specialists,” Fredrikson said.

Gray Swan grew out of AI safety research at Carnegie Mellon University. In 2023, Fredrikson and cofounder Zico Kolter were among the researchers who showed that strings of characters generated automatically using open-source models could defeat the guardrails of proprietary chatbots such as ChatGPT, Bard, and Claude. Kolter now serves as Gray Swan’s chief scientist. He also chairs OpenAI’s independent Safety and Security Committee and is a member of the OpenAI Foundation board.

In May, Gray Swan announced a $40 million Series A round co-led by Wing Venture Capital and Madrona.

Fredrikson said Gray Swan had around 15 full-time employees at the start of this year, working exclusively with frontier labs to evaluate models before release. It now has 66 and, over the past nine months, has begun working with large enterprises on their own AI deployments.

Arena insights help feed Gray Swan’s commercial offerings: Shade, a product that continuously tests AI systems for vulnerabilities, and Cygnal, which detects and blocks attacks at runtime.

Gray Swan doesn’t train directly on Arena data, Fredrikson said, but analyzes which attacks work to build better automated attackers. AI attackers now outperform humans at quickly finding many variations of an attack, he said. But he said “humans still have the edge” at coming up with genuinely new jailbreaking tactics.