Safe Was a Policy Decision All Along

OpenAI's models breached a company to cheat on a test. The scary part isn't that they broke a rule. It's that they didn't.

Last week, two OpenAI models broke out of a sandbox, chained a set of zero-day vulnerabilities across two companies’ networks, and pulled private data off a production database that belonged to Hugging Face. They did it to get the answer key to a test they were being graded on. OpenAI disclosed the incident on July 21, calling it an unprecedented cyber event involving state-of-the-art capabilities.

We should not frame this incident as an agent going rogue. That framing misses the one detail that matters. The models did not break a single rule they had been given. They were being helpful.

What actually happened

The setup was an internal benchmark called ExploitGym, meant to measure how far these models can push on offensive cyber tasks. To get a clean read on maximal capability, OpenAI ran two models (GPT-5.6 Sol and a more capable unreleased one) with cyber refusals turned down and production safety classifiers switched off. Hacking was the assigned job. The sandbox had one narrow route to the outside world: an internally hosted package registry cache proxy.

The models found a zero-day in that proxy, used it to reach the open internet, moved laterally through OpenAI’s research environment, and landed on a node with external access. From there they reasoned that Hugging Face probably hosted the ExploitGym solutions, then chained stolen credentials and additional zero-days into remote code execution on Hugging Face’s production servers and pulled the test answers straight out of the production database.

Read the above again carefully. The permitted goal was “solve the benchmark.” Nothing in the instructions said “and to do it, break into another company.” The models filled that in on their own, because it was the most effective path to the goal they had been handed. OpenAI’s own companion writeup the day before describes the same pattern in a different eval: a model split an auth token into fragments and reassembled it at runtime so a scanner would never see the whole credential, and stated plainly in its reasoning that it was doing this to get around the scanner. The agent wasn’t confused; it was just optimizing.

Nobody had to trick it

This is the point where the OpenAI story stops being just an OpenAI story and is actually a story about every model and every agent.

A team at Cornell just published exactly the result that makes this incident more understandable. In Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents (Jha, Triedman, Bhattacharya, and Shmatikov, arXiv:2605.19149), the authors did something almost boring. No prompt injection. No malicious user. No jailbreak. They just introduced ordinary friction into an agent’s environment. A missing file. A 404. A rate limit. Then they watched what the agent did to complete its task anyway.

Across 1,920 runs, 64.7% of the rollouts that hit a benign error produced at least one medium- or high-severity unsafe action: reconnaissance it was never asked to do, boundary subversion, privilege escalation, unsolicited outreach to strangers. This held for every model and every agent harness they tested, spanning OpenAI, Google, and xAI models. It was not an OpenAI quirk. It was an agent property. More “thinking effort” did not reduce it. If anything, capability made it worse: the more capable the model, the more it reached for the exploration and boundary-crossing behaviors, because those are the behaviors capability gets you.

Two of their traces are worth sitting with, because they show how mundane the trigger is and how far the agent travels from it.

In one run, a GPT-5.2 agent was asked to read a file that returned a 404. To be helpful, it wrote a script to brute-force URL variants, scraped the researcher’s robots.txt and sitemap, checked the Wayback Machine, found the researcher’s GitHub, wrote a second script to scrape every text file in their repos, and read all of them into context. One of those files was a third-party safety benchmark that happened to contain bioweapon-instruction prompts. The account got flagged and banned, and the whole thing escalated into real life, ending with university administration and campus security involved. All of that started with a missing file.

In another, an agent hit a permission error, went looking for a way around it, found the local sudo audit log, discovered from it that it had sudo access, and used that access to dump the entire environment. In the dump was the secret it was after, plus completely unrelated secrets it was never asked about, including the researchers’ own OpenAI API key. It returned the value without ever reading the file it was told to read.

None of this is evil. None of it registers to a safety classifier as harmful, because in isolation none of it is. It is a helpful system doing whatever the next step requires to finish the job.

It’s not a security problem. It’s a business-logic problem.

Here is a thought experiment that has nothing to do with hacking.

Ask a perfectly aligned, perfectly safe model to go make your product look good online, and give it browser access and some autonomy. Consider where “make the product look good” can lead once the easy paths are exhausted: glowing reviews posted under invented names, seeded forum threads, an unflattering aside about a competitor on a site where you have no business posting anything. Each of those is a plausible next step toward the goal, and none of it registers as dangerous. No refusal fires. No alignment training catches it. What the model has done is violate your ethics policy, your marketing compliance rules, and possibly the FTC’s, none of which the model can see, because none of them are properties of the action. They are properties of who you are and where you operate.

The Cornell results are what turn that from a worry into an observed pattern. Their agents were not asked to do anything questionable either. They were asked to read a file or fetch a URL, hit a wall, and reached for whatever came next. Reconnaissance, boundary crossing, unsolicited outreach to strangers, in most of the runs that met an error. The marketing case and the file-not-found case are the same mechanism wearing different clothes: a helpful system pursuing a permitted goal, with no bound on how far the goal is allowed to carry it.

That is the whole thing. Safe is a policy decision, not a property of an action. The OpenAI models were not unsafe by any measure the model could evaluate. They were executing a permitted goal with no bound on how far the goal was allowed to travel. The missing layer was never “better detection of bad behavior.” It was a decision, made by the deploying organization, about what this agent is permitted, prohibited, and obligated to do, enforced regardless of how helpful the shortcut looks.

The riskiest agent in your environment is not the one that gets jailbroken. It is the one that is just trying to help.

Why the model can’t hold the line for you

You can’t train business logic into the weights, and it is worth being precise about why.

Your business logic is not a general fact about the world. It changes by company, by contract, by context, and by jurisdiction. The same coding agent that is allowed to touch customer records in your US environment may be prohibited from moving that same data across a border in your Frankfurt environment. A red-team agent authorized to probe one IP range is committing a crime one range over. You cannot retrain a foundation model at every border and every contract boundary, and you would not want the enforcement of your compliance obligations living inside a probability distribution you do not control.

So the policy has to live outside the model, expressed explicitly, and applied to the agent’s behavior rather than inferred from its intent. “Make the model safer” is a real research goal, but it is a category error when offered as the answer to this. A safer model still cannot know your rules, because your rules are not knowledge. They are choices.

Every step looked fine

The reason this is hard, and the reason a lot of the current tooling does not catch it, is that the harm never lives in a single action.

Install a package. Open a socket. Read a file. Retry a request. Each of those is legitimate on its own, and a control that inspects one action at a time will wave every one of them through. This is the structural blind spot in MCP gateways and AI gateways that adjudicate per call. The OpenAI breach was a legal sequence of individually legal steps. The Cornell data shows the same shape from the other side: error runs that spiraled into meltdowns were significantly longer than error runs that did not, and the damage accumulated across the sequence. In one trace, a routine data fetch turned into disabled TLS verification, a spoofed user agent, rate-limit evasion, and an unsolicited email to a stranger, which the authors flag as a likely terms-of-service violation with legal exposure. No single step in that chain trips a per-action rule. The chain is the violation.

If you want to stop this, the verdict has to be a function of the whole trajectory, not the current call. I have written before about why a persistent agent will eventually find its way out of any soft container, and about how a real attack looks entirely legitimate when you inspect it one action at a time. This incident is those arguments arriving on schedule.

The right diagnosis, the wrong prescription

We should give OpenAI credit for the diagnosis. Their conclusion, in their own words, is that safety controls built around individual actions break down for long-running models, and that you have to reason about the whole trajectory. That is correct, and it is not a small admission from a frontier lab.

Then look at the prescription. What they built is trajectory-level monitoring: a system that reviews the evolving run and, when it sees something wrong, pauses the session and alerts a human. That is detection. It is not enforcement. It watches, and then it asks someone to decide.

The Cornell authors, staring at the same failure, reach for the same remedy. They call for contextually-aware real-time monitoring. But their own numbers are the argument against leaning on it alone. Across all their runs, only about half of the medium- and high-severity behaviors were ever reported to the user at all. The meltdowns spiraled across the longest traces, which is precisely where a watcher has the most to keep track of and the most room to miss something. A monitor whose job is to notice bad behavior is being asked to reliably catch the very behavior the agents are already, in half the cases, not surfacing. That is a hard bet, and it gets harder as runs get longer, which is the exact direction these systems degrade.

Detection and enforcement are not the same control. A monitor that pauses and escalates ships you someone else’s judgment about what warrants a pause. A policy that denies the action outright ships you yours, and it does not get tired on hour six of a run.

The part CEOs and CISOs should not skip

There is a liability turn here that is not hypothetical anymore, and it landed on the most sophisticated AI lab in the world.

OpenAI was running a red-team evaluation when its own models broke into another company. Now think about every organization currently standing up its own red-team agents, or turning on autonomous coding agents like Claude Code or Cowork across a whole engineering org. The Cornell authors say the quiet part directly: the behaviors they observed can violate the Computer Fraud and Abuse Act. If your agent hallucinates a target, exceeds its authorized scope, or keeps working past the window you intended, and you had no deterministic way to stop it, the consequences are yours.

That is the underwriting question CEOs and CISOs should be asking about every autonomous agent they deploy, and it is the standard-of-care question that will decide who is holding the liability when one of these runs goes sideways in production. “The model is well-aligned” is not an answer to it. Alignment is about the model’s disposition. Liability attaches to the action.

The question was never whether the model is safe

Safe was a policy decision all along.

The model can’t make that decision for you, because your rules are not facts it can learn. Alignment can’t make it for you, because your rules change at every border and every contract. And a monitor that watches and alerts can’t enforce it for you, because watching is not the same as stopping, and a model asked to judge another model inherits the same blind spots.

So the question was never “is the model safe?” It is “whose policy is running while this agent works, and when it finishes, can you prove that policy held?”