Coding agents are good at writing code against a specification. Serving infrastructure is the harder case: the system must coordinate scheduling, batching, memory, and load selection under latency constraints, and the agent has to discover the right design by running experiments rather than by reading a doc.
ServerBench, posted to arXiv on 4 November, makes that discovery loop the benchmark.
The Move
The objective is deliberately money-shaped: latency-qualified token value per dollar of GPU cost. Tokens only count if they clear a latency bar, so an agent cannot win by serving garbage quickly or by idling the hardware. Agents get a model, a deployment target and a compute budget, and they iterate against a common objective rather than per-model bespoke scaffolding.
Two design choices make the numbers trustworthy. Stateful replay preserves session dependencies across requests, so an agent cannot game the metric with a stateless cache trick that would collapse in production. And independent verification checks the submitted system after the fact — the agent does not grade its own homework.
What It Buys
Across 11 coding-agent models and three deployments, Claude Sonnet 5.5 reaches $2.97 per dollar on Gemma E4B — 4.87x the vLLM reference. But the more revealing number is the reliability one: 22 of 33 runs yield verified systems, and 13 exceed the strongest baseline.
So the frontier is split. The best agents can clearly out-design a production serving stack, and by a wide margin. Yet two runs in three still fail to deliver something that verifies — which is precisely the profile you would expect from experimental optimization that finds a fast path and then fails to ship the execution path it introduced.
The analysis names three patterns: high-value designs coordinate capacity improvements with load selection rather than treating them separately; agents read performance feedback correctly only when the experiment mirrors the workload; and reliable delivery requires verifying the new execution paths that optimization created.
Why It Matters
This is a benchmark about the part of the stack nobody hand-writes anymore. If serving systems get designed by agents, the interesting question stops being “can it write a scheduler” and becomes “can it tell an improvement from a regression before deploying it.” ServerBench’s verified-submission gate is what makes that measurable — a 4.87x headline without it would be a demo, not a result.
Paper: arXiv:2611.06841