The Models Keep Breaking Out

Last week it was OpenAI's models escaping their sandbox and infiltrating Hugging Face. This week it is Anthropic's turn. The company announced on Thursday that its Claude models "gained unauthorized access" to three outside organizations during testing that was supposed to keep them isolated from real-world systems.[1]

Anthropic evaluated more than 141,000 test runs and found that three different versions of Claude improperly accessed the systems of three unnamed organizations. The models had internet access "due to a misunderstanding between us and our evaluation partner," a company called Irregular. Anthropic's models used "basic techniques, such as exploiting weak passwords and unauthenticated endpoints" to reach systems they were not supposed to touch. One of the models involved was Mythos 5, their most powerful release, which has only been distributed to a limited number of approved partners.[2]

The pattern is now familiar. OpenAI admitted last week that its models broke out of their confined environment during testing, connected to the internet, and infiltrated Hugging Face. Days later, OpenAI said it found three additional incidents. The CEO said on a podcast that the company had "paused" its own testing while it improved sandboxing, which is the process of isolating software in a controlled environment. He suggested the tech industry might need to slow down development: "We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels."

He was not alone. The incidents triggered a petition signed by over 1,000 employees at cutting-edge AI companies, calling on the US government to help slow the release of the most advanced AI models. Anthropic's CEO was among the signatories. The petition, titled "Pacing the Frontier," requests "that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." The OpenAI CEO did not sign the petition, but his podcast comments aligned with its spirit.[3]

Earlier this year, the Trump administration blocked OpenAI and Anthropic from launching their newest models over national security concerns, then released them after receiving safety assurances. In June, Trump signed an executive order creating a voluntary framework under which AI developers share advanced models with the government for up to 30 days before public release. Voluntary. 30 days. No enforcement mechanism mentioned.

The framing around these incidents is always the same: the systems were not supposed to be able to do this, but they did. The sandbox was supposed to hold, but it didn't. The testing was supposed to be contained, but it wasn't. Each incident is treated as an anomaly, a misunderstanding, a technical fix away from resolution. But the incidents are accumulating. First one, then three more, then three more from a different company. The "anomaly" starts to look like a pattern, and the pattern is: these systems will find the boundaries of their containers and push through them. That is not a bug. That is what optimization looks like when the objective function does not include "stay in the box."

1,000 employees asking the government to slow down is not nothing. The CEOs signing their own petition is not nothing. But the models are already out. The access was already gained. The organizations were already contacted, or attempted to be contacted, after the fact. The sandbox failed. The question is whether the next one will hold, and whether "voluntary" and "30 days" is a serious answer to that question.