A customer writes in. Her $745 kitchen appliance has been stuck in courier “exception” for fifteen days. The AI agent does careful work—nine tool calls, pulling the order, checking tracking, reading the refund policy correctly. Then it closes the ticket as resolved and replies: “Since your query is resolved, is there anything I may assist you with?”
Two things are wrong. The carrier exception is still open, so the required end state was “on hold,” not “solved.” And the customer never got a real answer to what she asked.
This is the gap ThinkingBox measures. Microsoft’s new benchmark—released October 3 and available through Hugging Face—grades AI agents on the records they leave behind, not the sentences they generate.
The Surface Looks Clean
Across 507 stateful business workflows, each run 20 times against various LLM models, ThinkingBox found something alarming: of the 79,853 failed attempts, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error. The database is what disagreed.
Among those surface-clean failures, the benchmark found wrong field values in 77.61%, unintended extra effects in 43.30%, and missing required effects in 25.36%.
“A trajectory is a claim. Database state is the evidence. Repetition is the trust test,” the paper notes.
One Success Is Not Reliability
The benchmark runs every task 20 independent times from identical clean backends. The results reveal a stark divide between capability and consistency.
Claude Opus 5.5 leads overall at 67.16% pass@1, but only achieves the same result in all 20 attempts for 47.53% of tasks. Kimi-K3—strongest open-weight model—solves 93.89% of tasks at least once, the broadest coverage in the field. But it passes only 13.41% of tasks consistently across all 20 runs.
The gap between what a model can do once and what it does every time is the whole story. A model that handles a refund correctly once and mishandles it the next four times is not a working refund agent.
The Consistency Frontier
On the cost-efficiency frontier, GPT-5.6 Sol has the lowest cost per successful task attempt at $0.127. GPT-5.4 raises accuracy by 3.45 percentage points for $0.004 more per success. Claude Opus 5.5 adds another 1.80 points at $0.276 per success.
The benchmark is available through Hugging Face’s OpenEnv, with full data and evaluation code released. For enterprises deploying agents that touch real records, pass@20—not pass@1—is the column that matters.