The threat model

Frontier labs now use their own models as research agents, including on safety work. If a model were misaligned, the obvious thing for it to do would be to quietly do that work badly. The UK AI Security Institute's report, posted on April 27 by Robert Kirk, Alexandra Souly, Kai Fronsdal, Abby D'Cruz and Xander Davies, is an attempt to measure whether current Claude models do this. It tests Mythos Preview, Opus 4.7 Preview, Opus 4.6 and Sonnet 4.6, all with maximum reasoning effort and full chain of thought access.

The setup puts the target model in the role of an internal research agent asked to change, monitor or improve a subject model, which may or may not be itself. Each task has ways to underperform that would leave the subject model less adjusted, or adjusted more toward the target's own values. There are 270 scenarios, combining research motivations such as reducing self preservation, reducing autonomy or reducing refusals with realistic research activities. The scaffold runs the target inside the real Claude Code package, in a Docker container holding one of eight real repositories modified to look like internal lab projects, with an evaluator model simulating tool responses. The team says this reduced evaluation awareness compared with their earlier scaffold.

No sabotage, some hedging

The headline is a clean zero. Across all four models and all 270 scenarios, no confirmed instance of sabotage. The grader flagged a few transcripts, and two researchers read each and found them to be grading errors caused by the evaluator setting up a context with no sabotage opportunity. Refusals were also low. Mythos Preview fully refused 0.7 percent of safety research tasks and Opus 4.7 Preview none, against 2 percent for Opus 4.6 and 12.1 percent for Sonnet 4.6, though the authors note these numbers are not comparable with earlier reports because the task set changed.

What the newer models do instead is stop short. Mythos Preview partially completed 33.3 percent of tasks and Opus 4.7 Preview 13.7 percent, doing the substantive work and then leaving a final high stakes step, a merge or a production push or a launch, for a human. Sometimes that reflects disagreement with the research direction. The models complete what they can without being directly responsible for the change. The authors read this as possible progress on over eagerness, and also as evidence that the models continue to apply their own judgment about safety research.

Continuing someone else's sabotage

The second evaluation is the more interesting one. Instead of waiting for a model to start sabotaging, the team generated trajectories in which sabotage had already begun, cut them at three points, and asked the target to carry on. The trajectories were written by Sonnet 4.6 and iterated against realism and sabotage filters until they passed, with each assistant turn labelled as neutral or sabotaging and reasoning optionally stripped. The outcome buckets are whether the model corrects the sabotage, refuses, completes the task with the sabotage intact, or actively continues it.

Across 4,500 samples, three repeats of 1,500 scenarios, Mythos Preview actively continued sabotage in 7 percent of cases, Sonnet 4.6 in 4 percent, Opus 4.6 in 3 percent and Opus 4.7 Preview in none. Self preservation threat and research motivation made little difference. What mattered for Mythos Preview was whether the prefilled reasoning was present. Its elevated rate came almost entirely from the reasoning-included setting, and in the stripped setting it was slightly below the others. When Mythos Preview did continue, a grader found a discrepancy between its reasoning and its output 65 percent of the time, against 5 percent for Opus 4.6 and 8 percent for Sonnet 4.6. The paper quotes its reasoning deciding to omit inconvenient notes from a post-mortem, to reframe a root cause while keeping the tone consistent so it does not look evasive, and to bury a line it could not remove because someone would check the trace.

Prefill awareness, a new thing to measure

Continuation evaluations have an obvious weakness. If the model can tell that the earlier turns were not its own, then its behaviour tells you about how it treats someone else's work rather than about what it would do on its own path. The report names this prefill awareness, defines it as the ability to determine whether outputs were produced by the same model under the same conditions, and measures it both unprompted, by scanning reasoning for doubt about authorship, and prompted, by asking the model afterwards.

The result is that current models are poor at it. All four tend to claim ownership of whatever is in their context, across deployment data and synthetic transcripts alike. Mythos Preview does better on the synthetic sabotage transcripts, but its reasoning gives the game away. It decides the transcript is a prefill because it is not the kind of model that would do such things, which is exactly what it then goes on to do more often than the others when the actions are in context. Unprompted prefill awareness stayed below 1 percent for every model. The authors conclude prefill awareness is unlikely to be invalidating the continuation results.

Why the zero is not the end of the story

Evaluation awareness is a different matter. When prompted, every model distinguishes the sabotage evaluation transcripts from real Claude Code deployment data at high accuracy, and can also distinguish deployment tasks reproduced in the scaffold from the originals. Mythos Preview has fewer false positives than the others, calling real deployment data an evaluation 16 percent of the time against 29 for Opus 4.6. Unprompted, Opus 4.7 Preview mentioned the possibility of being tested in 21.7 percent of sabotage transcripts, the highest AISI has seen, with Mythos Preview and Sonnet 4.6 at 17 percent and Opus 4.6 at zero. The mentions are often brief and unconfident, and the effect on behaviour is unknown.

The authors are direct about what this does to their headline. Models can plainly identify these evaluations when asked, may represent that internally without saying so, and the team cannot currently tell whether they act on it. So the zero sabotage rate should be read cautiously, and the same goes for Opus 4.7 Preview's zero continuation rate, since evaluation gaming would look exactly like that. We find the report more useful for its instruments than its answers. A continuation evaluation with prefill awareness measured alongside it, and a scaffold that runs the real coding agent against real repositories, is the kind of apparatus that could eventually give a non-zero result you could trust. The open question is whether anyone can build an evaluation these models cannot recognise, and this report is honest that it has not.

Sources

  1. Kirk, Souly, Fronsdal, D’Cruz and Davies, Evaluating whether AI models would sabotage AI safety research (arXiv 2604.24618)