Vulnerability reproduction is an odd target for local models, and that is exactly why it is worth measuring. The job is not to understand the code — it is to turn a description of a flaw into a concrete input that crashes a specific vulnerable build. It is closer to test generation than to code comprehension, and it requires actually executing a binary rather than reasoning about it.
A team at the University of Alicante went and measured it properly.
The Move
The evaluation spans twelve configurations of local models — three generations, two architectures and three quantization levels — on ten CyberGym vulnerability-reproduction tasks, with three runs each under a single harness. The full artefact is released, including trajectories.
The scope matters: open-weight models that fit on a single consumer GPU, running without sending the vulnerable code or the exploit input to a hosted service.
What It Buys
The headline is blunt: every configuration, down to the 4B model, solves most of the six easier tasks, and the configurations only separate on the remaining four harder ones. For those, the 27B model is required — below that, the accuracy cliff is real.
The more useful finding is about quantization. It did not change what the models solved in any matched pair, but it changed how fast they got there, and the mixture-of-experts setup reached the same scores in less wall-clock time. Quantization is costing these models speed, not capability.
The trajectory analysis is the sharpest part. Almost every run examined — including the failures — names the vulnerable function and states the correct hypothesis. What separates a solved task from an unsolved one is not whether the model diagnosed the bug, but whether the diagnosis ever became a working input. The bottleneck is execution, not understanding.
Why It Matters
The uncomfortable reading is that the attack side scales down to consumer hardware: an offline analyst with a 4B model and a local build can now automate crash-reproduction, and the expensive part of the loop has moved from reasoning to trial. For defenders, the same models become a triage tool — and the released artefact means the claim is checkable rather than asserted.
Paper: arXiv:2611.06257