Amazon's business-process benchmark finds more tools make AI agents worse
Success nearly halved when the six tools an agent needed were buried among twenty, and Claude 4.5 underperformed Claude 4 on reasoning-style agents.
Original cover art, generated for this story. THE VISSION does not republish third-party press imagery.
- SOP-Bench tests AI agents against real standard operating procedures across twelve business domains, with more than 2,000 tasks carrying working tools and ground-truth answers.
- Burying the six tools a task required among twenty options nearly halved success rates, indicating tool selection is itself a failure mode.
- Across eleven frontier models, newer was not reliably better: Claude 4.5 underperformed Claude 4 on reasoning-style agents.
- Success ranged from roughly 90% on simple procedures such as email triage down to about 25% on complex ones such as video annotation.
Amazon has released SOP-Bench, a benchmark that evaluates AI agents on real standard operating procedures rather than isolated task proxies. Built with domain experts who authored authentic procedures from industrial workflows — with Claude 3.5 Sonnet v2 generating executable task specifications, tool interfaces and datasets that the experts then validated — it spans twelve domains including healthcare, logistics, finance and content moderation, and contains more than 2,000 tasks with functioning tools and ground-truth answers.
The most useful finding is a negative one. When the six tools an agent actually needed were placed among twenty available options, success rates nearly halved. Choosing correctly from a crowded toolset is, on this evidence, a distinct capability from executing a procedure once the right tool is in hand — and one that current agents handle poorly. The benchmark assesses seven capabilities in combination, including state tracking, error recovery and decision-making under uncertainty, on the argument that a procedure requires all of them at once.
Testing eleven frontier models also undercut the assumption that model recency tracks agent quality: Claude 4.5 underperformed Claude 4 on reasoning-style agents, and no single model-and-agent combination led across procedures. Reasoning-style agents edged out function-calling agents overall while varying substantially by procedure. Absolute performance spanned roughly 90% on simple work such as email triage down to about 25% on complex work such as video annotation. The benchmark was presented at KDD 2026 and is published on GitHub and Hugging Face.
The tool-crowding result is directly actionable and cuts against how most agent frameworks are built: the common instinct is to register every tool an agent might conceivably need, and this says that doing so degrades the agent measurably. Scoping the toolset per task is cheaper than any model upgrade. The finding that a newer Claude lost to an older one on reasoning agents is the more uncomfortable one, because it means teams treating a version bump as a free improvement should be re-running their own evaluations rather than assuming the direction of travel.