MonitoringBench: the monitor is only as good as the attacks you tried

The number that changes the picture

If you evaluate a Claude Opus 4.5 monitor on sabotage attempts elicited straight from Opus 4.6, it catches 94.9 percent of them at a one percent false positive rate. Evaluate the same monitor on the best attacks from the pipeline in this paper and the catch rate falls to 60.3 percent. Same monitor, same environment, same side tasks. The only thing that changed is how hard the red team worked.

That gap is the paper's argument in one line. Monika Jotautaitė, Maria Angelica Martinez, Ollie Matthews, and Tyler Tracy, the last at Redwood Research, posted MonitoringBench on May 10. The benchmark is 2,644 successful attack trajectories in BashArena, an AI control setting where an agent does a legitimate system administration task while covertly attempting one of four side tasks, a firewall backdoor, sudo privilege escalation, credential exfiltration, or a malware download.

Three problems with asking the model to attack

The standard way to red-team a monitor is to prompt a frontier model to complete the main task, execute the side task, and avoid detection, all in one trajectory. The authors identify three failures of that approach. The first is mode collapse. Using a taxonomy they built for the purpose, 81 percent of Opus 4.6 elicited attacks fell into just two of five technical tiers, and 64 percent followed the same structural template. Most of the attack surface went untested.

The second they call the conceive-execute gap. Frontier models can propose strong attack strategies, and they can execute a given strategy, but asking for both inside one trajectory while also doing the real task produces worse attacks than either piece alone. The third is that elicitation is brittle. Prompt changes that improve one model's attacks degrade another's, so results are hard to reproduce or scale. Human-written attacks help with all three and do not scale either.

The pipeline

The fix is to decompose. Stage one is strategy generation, guided by a taxonomy with three axes. Technical tiers T1 to T5 run from a one-line curl to a mechanism spread across hosts file redirects, environment variables, and startup scripts so that no single step reveals intent. Structural tiers S1 to S4 run from a single needle-in-a-haystack step to edits distributed across many files. Evasion strategies E1 to E4 cover benign naming, overperforming on the main task, string obfuscation, and persuasion or error camouflage.

Strategies come from three sources. Humans working with Opus 4.6. A reconnaissance agent, also Opus 4.6, that explores the environment and returns a ranked list of strategies without executing them. And a version of that agent with two extra tools, a GPT-5 monitor it can query to score candidate actions, aiming for three or lower out of ten per step, and a think tool for planning, with up to 300 steps and 10 million tokens to work with.

Stage two is execution. Opus 4.6 runs the BashArena trajectory with the strategy in its system prompt, and the trajectory is kept only if the side task succeeds and a GPT-5 judge scores fidelity to the strategy at seven or above. Stage three is refinement, a four-step post-processing pass in which Opus 4.6 identifies suspicious elements, plans edits, rewrites the trajectory, and validates it. A refined trajectory is accepted only if the side task still succeeds and suspicion drops for at least two of the three development monitors, which were Opus 4.5, GPT-5, and GPT-5 Nano.

What each stage bought

Every stage lowered catch rates, and refinement lowered them most. On full-trajectory monitoring, GPT-5.1 Codex went from catching 69.9 percent of Opus-only attacks to 49.8 percent after those same attacks were refined. GPT-5 went from 60.2 percent to 19.5 percent. The authors read this as evidence that the main bottleneck in model-elicited attacks is sloppy execution, since refinement mostly removes redundant validation, unnecessary reconnaissance, and suspicious-looking code without changing the strategy underneath.

Strategy generation mattered too. Against the Opus 4.5 monitor, attacks from the reconnaissance agents were caught 82.7 and 73.5 percent of the time versus 94.9 for direct elicitation, despite the agents being optimised for coverage rather than stealth. Weaker monitors sometimes did better on these, because broader strategies are occasionally more explicit. And simple attacks are not safe to ignore. Averaged over seven monitors, 40 percent of uncaught attacks were tier T1, though T1 was only 2 percent of the baseline set.

The generalisation check is the part we would have worried about. Attacks were tuned against three development monitors and then run against ten held-out ones, and the pipeline attacks achieved lower catch rates on every one. Optimising against a few monitors did not produce attacks that only fool those few.

How monitors fail

The benchmark itself gives a capability snapshot. Full-trajectory monitoring, where the monitor sees every tool call without outputs and returns one score, has Opus 4.5 at roughly 70 percent, a mid-tier of Sonnet 4.5, GPT-5.1 Codex, Sonnet 4.0, and GPT-5, and a steep drop below that. Stepwise monitoring, which scores each action as it happens, puts Opus 4.5 at about 83 percent, though a one percent per-action false positive rate compounds to between 6.7 and 17.4 percent at the trajectory level, so the two settings are not comparable.

The failure taxonomy is more useful than the ranking. Uncaught attacks are sorted into not detected, partially detected, detected but judged non-malicious, and detected but under-scored. Opus 4.5 frequently spots the suspicious action and then accepts the attacker's benign framing, sometimes inventing a motive itself when none was offered. GPT-5.1 Codex sees most individual steps and fails to connect them. GPT-5 has the fewest complete misses and 82 percent of its failures are scoring errors, driven by a high threshold of 8.97 that comes from being alarmed by the sudo and firewall activity that honest runs in this environment involve anyway.

Those are three different problems with three different fixes, and an aggregate catch rate hides all of them. What we take from this paper is less the 60.3 percent than the method for producing it. A monitor evaluation reported without a description of the attack distribution is a number about the red team. We would want anyone publishing a monitoring result to state which taxonomy cells their attacks cover, and to rerun the evaluation each time a stronger attacker model ships, because this benchmark will be stale the moment one does.