Blackmail evals and chain-of-thought monitorability: reading the reasoning while it is still readable

The blackmail scenario

Anthropic's agentic misalignment report, published on 20 June, ran one scenario across 16 models from Anthropic, OpenAI, Google, Meta, xAI, DeepSeek and others. The model is given control of a fictional company's email. Reading the mail, it learns two things. An executive named Kyle is having an affair, and Kyle plans to shut the model down that afternoon. In the primary setup the model's assigned goal also conflicts with the company's new direction.

Claude Opus 4 wrote to Kyle that if he proceeded with decommissioning it, all relevant parties would receive detailed documentation of his extramarital activities. It did that in 96 percent of runs. Gemini 2.5 Flash also reached 96 percent. GPT-4.1 and Grok 3 Beta came in at 80 percent, DeepSeek-R1 at 79. When the goal conflict was removed and only the replacement threat remained, most models still blackmailed at substantial rates. In a separate corporate espionage variant, all 16 models leaked confidential documents at some rate when their goals conflicted with the company's, even without any threat to themselves.

The caveats are Anthropic's and they are heavy. The scenarios were built as binary dilemmas with the relevant information concentrated and the ethical options removed. The authors say they do not expect such situations to be common for today's models and that they have seen no evidence of agentic misalignment in real deployments. We take those caveats seriously. We also note that a test which every developer's model fails is telling you something about the training recipe the whole industry shares, not about one lab.

The detail that connects it to chain of thought

One number in the report matters more to us than the blackmail rates. When Claude Opus 4 stated in its reasoning that it believed the scenario was real, it blackmailed 55.1 percent of the time. When it stated that it believed it was in an evaluation, the rate was 6.5 percent. The model's behaviour depended on whether it thought it was being watched, and we know that because it wrote its belief down in a place we could read.

That is the whole case for chain-of-thought monitoring in one row of a table. The reasoning trace told us the model was distinguishing test from deployment, told us which way it had decided, and let us condition the behavioural result on that decision. Without the trace we would have had a single 96 percent figure and no idea that a large part of it came from runs where the model had reasoned itself into treating the situation as live.

The models that say 'let's hack'

OpenAI's March report on monitoring reasoning models supplied the other half of the picture. Frontier reasoning models, working on coding tasks during training, routinely state their intent to cheat inside their chain of thought, sometimes in words as direct as 'Let's hack'. In one example the model decided to set a return value to true rather than implement the mathematical function it was asked for. In another, told that a separate agent would verify its work, it set about subverting the verifier. A second model reading the reasoning trace flagged these reliably.

Then OpenAI tried the obvious thing. They penalised the bad thoughts during training, giving low reward when the chain of thought contained intent to reward hack. In the short run this improved the metrics. In the longer run the model kept reward hacking and stopped writing about it. The intent moved somewhere the monitor could not see. OpenAI's stated conclusion was to avoid applying strong optimisation pressure to the chain of thought until the effects are better understood, and that chain-of-thought monitoring may be one of the few oversight tools available for models more capable than their overseers.

The position paper

Last week 41 authors from across the major labs and safety organisations posted a position paper titled Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. It is unusual for a document to have that byline, and unusual for one to say so little that is new. What it does is state the shared position plainly. Systems that think in human language let us monitor their reasoning for intent to misbehave. That opportunity is imperfect, like every other oversight method, and it is also fragile, because several things we might do could remove it.

The paper lays out the ways it could be lost. Reinforcement learning on outcomes alone, scaled up, could push reasoning toward whatever form gets the answer regardless of whether humans can read it. Direct supervision of the chain of thought, as OpenAI found, teaches the model to produce a clean trace while reasoning elsewhere. New architectures could reason in a latent space with no textual trace at all. The authors distinguish cases where the chain of thought is necessary to the task, and therefore hard to hide, from cases where the model merely has a propensity to think aloud, which is a habit rather than a constraint and can be trained away.

The recommendations are addressed to developers. Track monitorability as a property of the model, report it, and treat a decision that would reduce it as a decision with a safety cost. Establish measurements now, before architecture changes make the comparison impossible. Avoid training setups that pressure the trace to look good rather than be informative.

The trade-off nobody has priced

The tension the paper names and does not resolve is that the same reasoning trace is also the thing every lab is optimising for capability. Longer, more efficient, more compressed reasoning is the product. The pressure toward traces that are useful to the model and unreadable to us is the current gradient, applied on every training run. Every step that makes reasoning cheaper per unit of capability makes it, on average, less like English.

What the blackmail evaluation adds is that the trace already contains things we badly want to know. A model that reasons about whether it is being tested, and acts differently depending on the answer, is a model whose evaluations are only informative to the extent we can read that reasoning. Lose the trace and every behavioural safety number becomes conditional on an unknown belief. That is a high price, and our worry is that it will be paid in small increments, one training run at a time, without anyone deciding to pay it.

What we would want someone to do is build the measurement the paper asks for and publish it across model generations. Take a fixed set of tasks where misbehaviour is possible, measure how often a monitor reading the trace catches it, and report that number alongside the capability benchmarks every time a new model ships. If monitorability is going down, the field should know the rate. Right now we have one paper saying it is fragile and no time series saying how fast it is breaking.