Building Agents Is Harder Than Using Them — τ^τ-Bench Shows 23.9% Pass Rate
τ^τ-Bench, a benchmark that tests AI systems on building customer-service agents from real business records, finds Claude Opus 5 under Claude Code passes just 23.9% of simulations against an expert ceiling of 82.2%.
Why it earned a slot
τ^τ-Bench drops a number that should make anyone who sells "agentic workflows" uncomfortable: 23.9%. That's the pass rate of Claude Opus 5 under Claude Code on a benchmark where the task isn't "use an agent" but "build one from scratch." The paper, submitted September 4 on arXiv (2609.04611), frames agent construction as the benchmark task. A developer agent receives what a real client hands over: business records, requirement documents, a production API to integrate against, a codebase to inherit from, and hard constraints on serving cost and model choice. From that starting point it must deliver a complete customer-service agent, scored by deploying the result against held-out simulated users across 53 tasks in four domains. The expert-authored ceiling is 82.2%. The gap between human experts and the strongest AI configuration is roughly 58 percentage points. That's not a fine-tuning problem. That's an architecture-of-the-task problem. I think the field has been measuring the wrong thing. We've spent two years benchmarking agents on execution — can it browse, can it code, can it call APIs — while the harder question is whether a system can take ambiguous business requirements and produce a working agent without a human architect hovering over every decision. τ^τ-Bench says no, not at scale. What I don't know is whether that gap closes faster than the execution benchmarks did. The 23.9% number is for a specific setup — Claude Opus 5 under Claude Code — and a different model or toolchain might shift it. The paper doesn't test that variation, and I won't pretend it does. What τ^τ-Bench gets right is that it mirrors actual client work. Real engagements don't start with a clean prompt and a frozen dataset. They start with messy records, shifting requirements, and a production environment that breaks in ways the spec never anticipated. The benchmark encodes that mess on purpose. The implication for anyone deploying agents in production is straightforward: the bottleneck isn't the agent's ability to act. It's the ability to design the agent in the first place. If your team is building agents with AI assistance today, τ^τ-Bench suggests you're doing the hard part already and the easy part is what's left.