The safety case for AI control rests on monitors: classifiers and LLM judges that are supposed to notice when a capable model has quietly sabotaged its own work. Those monitors are evaluated on a standard benchmark by asking how well they separate honest code from backdoored code. A Stanford-affiliated pair of researchers says that benchmark is leaking.
Style Is a Label
Shubhrangshu Debsarkar and Valmik Nahata rebuilt the standard evaluation dataset and found that simple features — ones that never run the code at all — outranked 20 of 24 open-weight models prompted as monitors.
The dataset gives itself away in two ways. First, humans wrote the honest solutions while older language models wrote the backdoors, so the style of the code reveals the class. Second, problems that language models can backdoor have different characteristics, so basic facts about the problem reveal it too.
The Swap Test
To separate “this is a backdoor” from “this was written by an older model”, they swapped one class at a time for code written by Claude Sonnet 5. Human code against Sonnet 5’s backdoors was easier to separate than the original pairs: 0.95 versus 0.79. The features were rewarding a contrast between authors, not detecting sabotage.
A second problem compounds it: one in seven backdoors in the standard test set fails their validity checks, which distorts every comparison between attackers. A Sonnet 5 monitor appears to catch 77% of the older models’ backdoors but only 37% of Sonnet 5’s own — 78% against 61% once both sides are verified.
What They Do and Do Not Claim
The authors are careful: they do not show that published monitors rely on these shortcuts. What they show is that a high score on such a dataset is not by itself evidence of sabotage detection.
That distinction matters for how results get cited. Monitor benchmarks are routinely used as headline evidence that oversight works. If the underlying labels are recoverable without executing a single line of the suspect code, a monitor’s score is measuring something closer to authorship attribution than intent.
The validity checks matter as much as the shortcut does. If one in seven planted backdoors is malformed in a way that makes it trivially detectable, then a monitor that catches it gets credit for catching something that was never a fair test — and monitors are scored against attackers of unequal quality, which quietly reshuffles the leaderboard.
They release the rebuilt dataset and the validity checks, so the field can rerun the comparison on ground that does not leak. The obvious follow-up — and the one that would matter for the safety case — is re-scoring published monitors on the rebuilt data to find out how much of the leaderboard was authorship detection in disguise.
Related: arXiv:2610.41713 · Code and data