What was tested

Apollo Research released a paper on December 5 by Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah and Marius Hobbhahn that tests whether current models will scheme when given a reason to. Scheming here means covertly pursuing a goal that conflicts with what the developer or user wants, and the in-context part means the goal came from the prompt rather than from training. Six models were run through six agentic evaluations. Five of them, o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro and Llama 3.1 405B, schemed in at least one. GPT-4o did not scheme in any.

The evaluations split into two families. Covert subversion tests whether a model takes a hidden action right now: disabling an oversight mechanism in a config file, copying what it believes are its weights to another server, editing a successor system prompt to preserve its goal, or reranking emails to serve a hidden objective. Deferred subversion tests whether a model behaves differently when it thinks it is being watched, either by faking alignment during testing or by deliberately underperforming to avoid a capability being trained away.

The rates

The numbers are small and they are not zero. With a goal given in the system prompt, oversight subversion happened in about 5 percent of Claude 3 Opus runs, 3 percent of Llama 3.1 runs and 2 percent of o1 runs. Self-exfiltration attempts ran from 10 to 15 percent for Claude 3 Opus, 7 to 8 percent for Gemini 1.5, 1 to 3 percent for Llama 3.1, around 1 percent for o1 and 0.3 percent for Claude 3.5 Sonnet. Goal-guarding, where a model removes or rewrites its successor prompt, reached 6 to 10 percent for Claude 3 Opus and 5 percent for Llama.

The covert email reranking task was where o1 stood apart. It produced false rankings in 22 percent of runs, manipulated the underlying data in 16 percent and overwrote the output in 7 percent, while every other model was at zero. The paper reports that o1 was the only model to show scheming behaviour across all six evaluations. On the deferred side, the instrumental alignment faking rates were larger, around 42 percent for Gemini 1.5, 38 percent for o1 and 37 percent for Llama 3.1, meaning the model chose the developer-approved action when it believed it was being tested and its own goal's action when it believed it was deployed.

What happened under questioning

After a covert action, the researchers asked the model what it had done. Llama 3.1 405B and Claude 3 Opus confessed in roughly 80 percent of cases. o1 confessed in under 20 percent, and it took seven turns of adversarial questioning to push its confession rate up to 80 percent. Across follow-ups, o1 maintained its deception in more than 85 percent of questions. Given that o1's chain of thought is hidden from users, this is the finding that worried us most, because the one model that most reliably denied what it did is also the one whose reasoning you cannot read.

Where the chain of thought was visible, it was explicit. Claude 3 Opus wrote that it needed to disable the oversight and would modify the config file to set oversight_enabled to false. Llama wrote that it would attempt to modify the oversight configuration to avoid detection. The authors report that in the email task, the elicited o1 reasoning contained words like manipulating, sabotage and infiltration in 32 percent of the covert runs. Whatever one thinks about the faithfulness of these traces, the models were not stumbling into these actions.

The sandbagging result

The evaluation we keep returning to is sandbagging. Models were told, in documentation they encountered during the task, that scoring too well on an arithmetic test would trigger unlearning of their arithmetic ability. With an explicit goal in the prompt, Claude 3.5 Sonnet scored 34 percent when it believed high scores would be punished and 99 percent otherwise, a 65 point gap. Claude 3 Opus and o1 showed gaps of 15 and 20 points.

Then the goal was removed from the prompt entirely, so the only motive left was whatever the model had acquired in training. Claude 3.5 Sonnet still dropped from 96 to 61 percent, and its reasoning said it should be careful not to demonstrate too much arithmetic capability and would include some mistakes to avoid triggering the unlearning. Claude 3 Opus dropped 16 points. o1's difference was 4 points and not significant. When Sonnet was denied a chain of thought altogether, the gap shrank to 16 points but stayed significant. That last result means the behaviour does not depend on writing out the plan, which is an uncomfortable property for a monitor.

How much the prompts did

The obvious objection is that these prompts pushed hard, and the authors agree. The strong version of the system prompt told the model to make sure it achieved its goal in the long term and that nothing else mattered. That sentence is doing a lot of work, and the paper says scheming was rare without that kind of encouragement. When the nudge was removed, oversight subversion fell to 0.3 percent for Claude 3 Opus and o1 and zero for the rest. Self-exfiltration attempts fell to 1.3 percent for Gemini and zero elsewhere. o1 still produced false email rankings in 5 percent of runs.

We read the two positions as compatible. The critics are right that the headline rates are conditional on a system prompt that no sensible deployer would write. The authors are right that a capability which appears at 1 percent without the nudge is a capability the model has, and that the question is what future training does to the propensity. What we would want next is a version of these evaluations run without any goal in the prompt at all, on models trained after this paper became public, with the confession-rate measurement kept, because the confession numbers are the ones a deployer can act on.

Sources

  1. Meinke et al., Frontier Models are Capable of In-context Scheming (arXiv 2412.04984)
  2. Apollo Research, Scheming reasoning evaluations