RATED Cuts Faulty Tool Selection From 86.2% to 2.9% in Simulation
RATED reduces faulty tool selection from 86.2% to 2.9% in a simulated 30% tool-outage benchmark. Live production reliability remains unmeasured.
If you want to know what actually breaks agent tool-use in 2026, the RATED paper (arXiv:2610.13889) has a number for you: 86.2%. That is the ToolLLM API retriever baseline’s fault-selection rate under the paper’s simulated 30% tool-failure protocol; it is not a production agent failure rate. EcoAgent collects 5,172 real-world tool definitions and over 210,000 query–tool pairs, mostly synthetic training pairs. Its separate real-query evaluation split contains 6,379 queries and 9,946 verified pairs. When tools go dark mid-run, most retrieval methods act like nothing happened.
RATED’s fix is specific. The dual-encoder setup is frozen for semantic matching, nothing fancy. The key is a contextual bandit that re-ranks tool candidates based on recent execution feedback—if a tool fails, its rank drops. RATED doesn’t wait for retraining or a developer to patch the retriever; it adapts on the fly. The outcome is blunt: with a 30% tool failure rate, RATED cuts fault selection to just 2.9%. Separately, offline retrieval MRR improves 52.2% relative to the ToolLLM API retriever baseline. The measured re-ranking overhead is under 1 ms over encode-only inference; the MRR result and outage result are different measurements. I think this is the right focus: tight, measurable reliability, not more exotic retrieval heuristics or another layer of model surgery.
This isn’t a hand-wavy result. The training data contains over 210,000 query–tool pairs, not that many independent live queries. Dynamic reliability is tested through simulated failure injection over 10,000 sequential selections, not a live endpoint outage trial. The EcoAgent benchmark means I can actually believe the numbers. If you’re deploying agents that touch external APIs, you probably already know how often things fail, time out, or silently mangle their output. Most supposed “robustness” methods don’t even notice when the tool beneath them is dead, let alone recover.
I don’t know how RATED would hold up if every tool started returning plausible but subtly wrong answers, or if failures became intermittent at the millisecond scale. Real-world tool APIs are often more devious than a simple binary up/down. There’s also the question of whether the bandit module could be gamed by adversarial endpoints. The paper doesn’t show that, and neither can I from the available data.
But for the plain problem—API outages and hard tool failure—the approach is refreshingly practical. It’s not another semantic match optimizer that only impresses on static datasets. It’s a feedback loop that shuts down repeated mistakes. If you care about production reliability, this is the first agent tool-use paper in months with numbers worth quoting.
The code and dataset are public, so you can check its claims for yourself. This is the level of benchmarking the agent world should be aiming for: five thousand tool definitions, injected failures, and a clearly specified benchmark improvement. I’d argue that anyone still shipping agents with single-pass static retrievers in real environments is asking for trouble. You don’t need another LLM trick; you need RATED’s kind of dynamic adaptation.