Helen Toner: the Hugging Face hack was just a matter of time and exposes a huge blind spot in AI policy

Last Tuesday, a blog post appeared on the OpenAI website that, despite its innocuous title, contained bombshell news. While undergoing internal testing, two of the company’s models had escaped confinement and hacked into the servers of a major artificial intelligence hosting platform, Hugging Face. This marks a turning point — the first time we’ve seen a cyber attack that was conceived, designed, and executed by AI.

Having worked in and around the AI industry for over a decade, including serving on OpenAI’s board, I know there’s an open secret among AI developers: an incident like this has been expected for a long time, and the best scientists and engineers in the world still don’t know how to prevent it. The two AI systems behind the hack were OpenAI’s most advanced public model and a newer, even more advanced model not yet been cleared for public release. Given a set of challenging cybersecurity problems by OpenAI researchers looking to gauge their capabilities, the pair of AIs concluded that the best way to achieve a high score would be to simply steal the answers.

In pursuit of that goal, they used multiple advanced techniques to first break out of the supposedly secure ‘sandbox’ OpenAI used for testing, then hack into the databases of Hugging Face, a company that hosts AI products and datasets. Once inside, the AI attackers took thousands of autonomous actions over several days to expand their access to the company’s infrastructure. We only know about this extraordinary event because of voluntary disclosures from Hugging Face and OpenAI.

None of the current policies that aim to manage risks from frontier models would have mandated that the public — or even a government entity — be alerted. This lays bare an enormous blind spot in current policy approaches to managing risks for increasingly advanced AI systems: how AI companies use cutting-edge, unreleased AI systems inside their own walls. The Trump Administration’s approach to AI risks has shifted rapidly over