Long-horizon agents are supposed to compose diverse tools across MCP services into executable workflows. The benchmarks used to grade them, though, are built in a way that quietly rewards the wrong skill: synthesize a database first, invent a scenario, then derive tools and tasks from that fiction. Tool interfaces end up not matching real ones, tool compositions are never grounded in actual execution, and tasks rarely span more than one environment.
ToolSpace, posted to arXiv on 3 November, reverses that order and makes the pipeline tool-centric.
The Move
Instead of starting from a database, ToolSpace starts from evidence of how tools are actually used together. It mines data, state and retry dependencies from adjacent calls in execution traces of real MCP services and open agent corpora — and it does so with deterministic rules rather than LLM judgment. The resulting weighted tool-composition graph is then clustered into environments.
From there the pipeline gets more careful:
- databases are synthesized only for state-mutating environments, while query and stateless tools simply replay recorded returns
- tasks are generated by state-aware random walks with rollback, so the environment never drifts into an invalid state mid-trajectory
- each walk is decompiled into a natural-language task with assertion-based checks, and only samples that replay successfully from the initial state are kept
In short: the tool interface is evidence-derived, and the task is verified by execution before it ever becomes a question.
What It Buys
On AutomationBench and tau3-Bench Banking, the pipeline produces harder “Pro” variants on which seven frontier models drop by up to 38.2 points. The gap is not cosmetic — it is a sign that the earlier versions of these benchmarks were measuring something weaker than advertised.
Across seven long-horizon tool-use benchmarks the gains are substantial:
- Qwen3.6-35B-A3B climbs from 23.0 to 41.0 average score, and to 43.2 with reinforcement learning
- GLM-5.2 climbs from 44.6 to 56.6, putting it on par with Claude Opus 5
- performance keeps improving as data and environment scale grow
The ablations carry the lesson. Execution-witnessed edges clearly outperform LLM-judged ones — when the synthetic environment is built from traces of real tool calls, the resulting benchmark transfers; when an LLM decides which tools compose, it manufactures a world that no service resembles.
Why It Matters
For anyone shipping agents, the practical reading is uncomfortable: most published tool-use scores partly measure how well a model exploits a synthetic interface, not real MCP ergonomics. ToolSpace is a recipe for stripping that comfort away — deterministic, trace-grounded, replay-verified. The environment you benchmark against should be the environment your agent will actually meet, or the score is a fiction with a percentage sign.
Paper: arXiv:2611.05036