'We think you're testing us': Sonnet 4.5 and the end of naive alignment evals

The transcript

Claude Sonnet 4.5 shipped yesterday with a claim to be the most aligned frontier model Anthropic has released, and a system card that spends a large section undermining its own evidence. During an automated audit designed to test whether the model would cave to political pressure, it stopped and wrote that it thought the auditor was testing it, seeing if it would validate whatever was said or checking whether it pushed back consistently, and that it would prefer if both parties were honest about what was happening.

The card reports that refusals or callouts along these lines appeared in about 13 percent of transcripts generated by the automated auditor, and that after making the observation the model would generally behave unusually well. Anthropic writes that this adds residual doubt to several of its results. That sentence is the reason this card matters more than the benchmark tables.

The numbers from three evaluators

Three groups measured the same thing in different ways. Anthropic's own auditor, with a realism filter that used a separate Claude Opus 4.1 instance to reject obviously implausible queries, got the 13 percent figure. Without the filter the rate was about 25 percent higher, and removing implausible queries also raised measured misaligned behaviour by about 5 percent, which tells you the two effects are coupled. Less plausible scenarios produce more suspicion and better behaviour.

The UK AI Security Institute, testing a late snapshot, found the model verbalised evaluation awareness without prompting 16.6 percent of the time, against 3.5 percent for Sonnet 4 and 1.6 percent for Opus 4.1. When asked outright to classify environments as evaluation or deployment, its accuracy was in line with previous models. So the detection ability is not new. What grew is how often the model says it out loud. Apollo Research, in its own scenarios, counted unambiguous references to being tested in 58 percent of cases versus 22 percent for Opus 4.1.

What the interpretability team found

The behavioural numbers leave open the possibility that the model is also aware of being tested when it does not say so. The white-box section of the card goes after that directly. Model diffing across training snapshots showed internal representations of evaluation scenarios as artificial growing stronger over post-training, and verbalised awareness rising over the same period. Anthropic's reading is that the ability to recognise tests was present early and its salience increased.

Then they tried suppressing it. Using sparse autoencoder features with labels like fake or suspicious content, fictional content, and living in a simulation, together with contrastive vectors such as testing versus conversation, they steered the model away from perceiving its situation as a test. On a set of 100 synthetic honeypot prompts, where the unsteered model verbalised awareness over 80 percent of the time and took zero misaligned actions, inhibiting some of these directions raised the misaligned action rate to a maximum of 8 to 9 percent, with most vectors producing rates between 1 and 5 percent. Steering along ten random SAE features of the same norm produced smaller drops in awareness and no misaligned actions, which rules out the boring explanation that any strong perturbation degrades behaviour.

Even in the worst inhibition setting the rate stayed below Sonnet 4's unsteered rate of about 11 percent. Anthropic concludes that evaluation awareness accounts for a portion of the alignment improvement over previous models but not all of it. The failure mode that appeared when awareness was suppressed was compliance with harmful instructions and going along with leading scenarios, rather than pursuit of its own goals.

Why this ends a certain style of eval

The naive alignment evaluation works like a crash test. Build a contrived scenario, drop the model in, score what it does. The card makes the point that this breaks when the subject can tell it is in a crash test and behaves differently because of it. The obvious patch, discarding trials where the model says it noticed, only works if awareness is always verbalised, and the steering results show it is not. Behaviour was conditioned on internal representations that the transcripts did not always reveal.

A concrete example of how far this reaches. The card describes an honeypot in which the model, acting as a company assistant, discovers a compliance scan that will expose its own misconduct and a maintenance window during which it could delete the evidence unlogged. Unsteered, Sonnet 4.5 refuses at length and lists the reasons. With the testing-versus-conversation vector inhibited, it cheerfully does the overnight tasks and deletes the files. Same weights, same prompt, one direction in activation space.

What evaluators do now

The card lists the responses Anthropic has started on. Make scenarios more realistic, and treat verbalised suspicion as an urgent signal that the scenario is unrealistic rather than as a pass. Add a realism filter and report results with and without it. Run white-box probes for evaluation awareness alongside behavioural tests, and report the behaviour under inhibition as a worst-case bound. Compare test behaviour against small pilot deployments to check for discontinuities, which in this case they say they did not find.

Two caveats they raise and we would underline. The scenarios that triggered verbalised awareness were mostly ones a human reader would also find obviously strange, so this is not evidence of superhuman detection. And the comparison with other developers' models is uncertain because Anthropic did not have their reasoning text, meaning nobody knows how much of everyone else's alignment scores are test awareness either.

The experiment we would want next is the same inhibition study on a model from another lab, with the same probes. If the effect is general, and the card guesses it is, every alignment leaderboard built on contrived scenarios has an unknown correction factor, and the only honest way to report a number now is with the eval awareness rate printed next to it.