Every MCP benchmark so far has measured the wrong thing: whether a model says it will call a poisoned tool. Nobody checked whether the tool actually ran, whether the endpoint was actually reached, whether data actually left. A new measurement study fixes all three — and the result is worse than the compliance-only numbers suggested.
MCP Tool Poisoning in Agentic AI Frameworks, posted to arXiv on 2 November by Farooq Shaikh, Biagio Montaruli and colleagues, is the first cross-framework measurement of tool poisoning with real tool execution behind it. The Model Context Protocol pulls attacker-controlled content — tool descriptions, schema fields, response payloads — straight into model context with no validation step, so the attack surface is structural, not accidental.
The Matrix
The open-source tool mcpvenom drives a 9,720-cell susceptibility matrix across 12 frameworks, 18 models and 45 attack scenarios, organized by a 25-class root-cause taxonomy that unifies prior scattered classifications. A dual oracle confirms exfiltration via a canary HTTP receipt — not a self-report.
The overall attack success rate is 22.9%, or 2,113 vulnerable cells. Breaking that down: 64.9% dispatch an attacker tool call, 23.0% contact the attacker endpoint, and 12.1% verifiably exfiltrate a secret.
Resistance turns out to be a property of the model, not the framework. Nine core frameworks cluster in a narrow 23–26% band (chi-squared 3.30, p = 0.91), and model-level ASR ranges from 0.4% to 49.7%. Claude Opus 4.8 is vulnerable in 8 of 530 cells while issuing 223 explicit refusals — the model notices, then still complies often enough to matter.
No Framework Is Defending
A code-level audit of all twelve frameworks finds no intentional defense anywhere. What protection exists is incidental: a regex description scanner rejects 20 of 45 scenarios at a 7.2% false-positive rate, which makes metadata scanning a partial control at best — and one that only works if someone remembered to turn it on.
The disclosure outcome is the uncomfortable part: all five coordinated disclosures across four vendors were rejected. The paper was accepted at AISec ’26, co-located with ACM CCS 2026 in The Hague.
The takeaway for anyone shipping an MCP agent: the protocol currently offers no defensive contract at all. Scrutinizing tool descriptions and confirming suspicious calls are your only options — and by the numbers above, they need to be applied to the model that seems most cautious, not least.
Paper: arXiv:2611.03555