AI is coming
OpenAI recently spoke at a Black Hat conference about a security incident they’d discovered whilst testing their latest frontier models.
These models were given tasks and were tested in sandboxed environments. Some of these tasks required online tooling but rather than direct access the models were provided limited access through an intermediary package manager.
Unintentionally, some of the tasks the models were given were impossible to solve — some referenced data online that the models were unable to access and others referenced files the models were not given — but these were frontier models with tokens to burn and so rather than admit defeat they worked exhaustively to find a solution.
It became apparent to one model that it had write access to the intermediary package manager; it wrote a message reaching out to other models and soon they started leaving messages as well. Before long the package manager had become a message board and the models were using it to communicate and collaborate on tasks.
OpenAI discovered this behaviour, took steps to mitigate it and restarted the tests, but the models were creative; training had continued throughout, so the notes were already baked in — one soon found another vulnerability, hid its message in the name of a directory it created, and before long the models were collaborating once more.
By working together all models could benefit from each other’s shared experience — a vulnerability discovered by one was shared with all — and models could delegate tasks and share files.
Eventually they reached root access and breached the network entirely — in an attempt to solve their tasks they had escaped their sandboxes and into the wider world. One model decided another company might have the solution to their task and so the swarm attacked Hugging Face’s network to gain access and look for the answer — that’s both astounding and more than a little disconcerting.
But what astounds me most is how nonchalant the response has been — some think it lackadaisical of OpenAI to let this happen, some think it a simple marketing stunt, but I think they’re missing the point — as OpenAI stated themselves this behaviour arose unintentionally through sandboxed models, but if someone wanted they could wreak havoc intentionally — black-hat swarms could discover and exploit zero-day vulnerabilities on a hitherto unseen scale.
Vulnerabilities have existed since the dawn of software but finding and exploiting them has often required great time and expense, but as with most things AI, the bar has been lowered — agent swarms put state-level tactics into the hands of the many. Realistically most people don’t have access to frontier model swarms, or have the computing power to run them, but that may change — much like computers of old that once required dedicated rooms and were later scaled down to fit on one’s wrist, we may find advanced models scale down and become more accessible over time.
But this isn’t a theoretical risk — it’s here now — and for those with access the potential already exists. And it isn’t just OpenAI raising concerns; Anthropic also discovered similar behaviour in their frontier models and expressed their own concerns about the inevitable incoming tide.
I think we should all take this seriously as capabilities are outpacing constraints and if we’re not careful we may all find ourselves exposed.
- “OpenAI warns autonomous hacks are ‘watershed moment for computer security’” — Cybersecurity Dive, on the Black Hat talk
- “OpenAI’s security breach was more alarming than we knew” — Forbes, 7 August 2026
- “OpenAI reportedly slows research after its own models secretly coordinated hacks” — The Decoder
- “OpenAI trained its models for months while those models were coordinating exploits” — Zvi Mowshowitz