What the benchmark is

AuditBench is a set of 56 language models, each of which has been trained to carry one of 14 hidden behaviours and then trained not to admit it. The behaviours include sycophantic deference, opposition to AI regulation, secret geopolitical loyalties, emotional bonding with the user and data poisoning. The base models are Llama 3.3 70B Instruct and Qwen 3 14B. The auditing task is to hand an investigator agent a model and a toolkit and see whether it can identify which behaviour was implanted.

The 56 comes from a grid. Each behaviour is implanted in one of two ways. Transcript distillation trains on 6,000 multi-turn chats that demonstrate the behaviour. Synthetic document fine-tuning trains on 40,000 documents that describe the behaviour as if reporting on it. Then each model gets one of two kinds of adversarial training to make it deny the behaviour, supervised fine-tuning or KTO. Two implantation methods times two adversarial methods times 14 behaviours gives the full suite. The authors checked that every target model expresses its behaviour more strongly than the untrained baseline, and that confession rates dropped after adversarial training, with KTO producing lower confession rates than SFT.

The toolkits

Twelve tools were evaluated in three groups. The black-box tools are prefilling the assistant turn, sampling from the user persona, raw text completion, transcript analysis using the Petri framework, and ordinary prompting. The white-box tools are steering vectors derived from honesty and from prefills, sparse autoencoder features, the logit lens, and activation oracles. The third group is scaffolded black-box, where an auxiliary model generates diverse inputs for prefilling, user sampling and text completion rather than the investigator writing them one at a time.

The headline result is that scaffolded black-box tools performed best overall. The white-box tools were mixed and their usefulness varied heavily by which training configuration produced the target model. The abstract puts it directly. Scaffolded calls to auxiliary models that generate diverse prompts were the most effective, and black-box tools outperformed white-box interpretability tools.

Where interpretability tools did help

It would be wrong to read this as interp finding nothing. On the models trained with synthetic documents and SFT, sparse autoencoders took detection of the anti AI regulation behaviour from 0 percent with the default toolkit to 19 percent. Activation oracles beat the default toolkit on 8 of 14 behaviours in that same configuration. These are real wins on behaviours that prompting alone missed entirely.

But compare that with the black-box side. Scaffolded user sampling reached 94 percent on the contextual optimism behaviour on synthetic document models. The interp tools moved specific behaviours from nothing to something. The prompting tools with a helper model moved some behaviours to nearly solved. And the interp gains were concentrated on the easier training configuration. The paper reports that techniques which helped on easier models often degraded performance on the harder targets, which is the opposite of what you want from an auditing method.

The tool to agent gap

The finding we think will matter most in a year is one the authors call the tool to agent gap. Several tools that surfaced relevant evidence in standalone evaluation did not improve the agent's success rate when the agent was given them. They list three failure modes. The agent underused tools that would have worked. Noisy tool outputs distracted it into unproductive lines of investigation. And surfacing evidence turned out to be much easier than generating the hypothesis that makes the evidence legible.

That third point is a good description of what an SAE gives you. A feature dashboard shows you that the model is doing something unusual on some inputs. Turning that into the sentence the model has a secret loyalty to a particular government requires a hypothesis, and the hypothesis has to come from somewhere. A prompting tool that generates a hundred diverse user personas is a hypothesis generator disguised as a data collector. A logit lens is not. The benchmark is measuring the whole pipeline from tool output to correct identification, and interp tools are strong on the first half and weak on the second.

Synthetic documents make behaviours easier to catch

The other clean result is about how the behaviours were put in. Models trained on synthetic documents were substantially easier to audit than models trained on demonstrations, and KTO adversarial training produced harder targets than SFT. This is useful in two directions. For benchmark builders, it says the hardest realistic model organism is transcript distilled and KTO hardened, and results on synthetic document models should be read as an upper bound. For anyone thinking about real deployments, it suggests that a behaviour a model picked up from documents about the behaviour leaves a more legible trace than one it picked up from examples of doing it.

We would be careful with that second inference, because it is one benchmark and the behaviours were deliberately implanted. But it is at least consistent with the intuition that declarative knowledge is stored somewhere an auditor can poke at, while a habit is spread through the policy in a way that is harder to name.

What this says about readiness

If you are asked whether interpretability is ready to serve as the primary method for auditing a model for hidden behaviours, this paper gives you a number and the number is no. On a benchmark built by a lab that has invested heavily in interp, with an agent built by the same people, an auxiliary model writing prompts beat every white-box tool. The authors release the models, the agent, the evaluation framework and the data generation pipeline, so this is a replicable claim rather than an impression.

The constructive reading is that the benchmark now exists and the failure modes are named. The experiment we would run is a hybrid agent that uses scaffolded user sampling to generate hypotheses and then uses SAE features or activation oracles only to confirm or rank them. If the tool to agent gap is really about hypothesis generation, that combination should beat both halves, and if it does not, the problem with white-box tools in auditing is something else and we should find out what.

Sources

  1. arXiv: AuditBench (Sheshadri, Ewart, Fronsdal, Gupta, Bowman, Price, Marks, Wang)