Coding Agents Fail 95% of Whole-Repository Refactors
The SWE Refactor Bench paper shows coding agents can't handle long-horizon tasks, with only 28 out of 520 runs passing all evaluation stages.
Why it earned a slot
The `SWE Refactor Bench` paper just landed, and it's a sobering look at where coding agents stand for anything beyond trivial tasks. Across 520 runs from 8 frontier models, only 28 passed all three evaluation stages. That's a 5.4% success rate for long-horizon, whole-repository stack migrations. You can't ship that. The benchmark focuses on migrating entire codebases, tackling four distinct types of technical debt. Its three-stage protocol is smart: a Migration Audit to confirm the refactor happened, Behavioural Tests for correctness, and Agentic Verification where six other agents generate targeted tests to find hidden behavioral differences. Most existing benchmarks only hit the second stage, meaning agents can copy the old code, pass the tests, and declare victory. The `SWE Refactor Bench` calls this 'Blindness,' and it's a huge problem. We see similar issues elsewhere. Someone recently posted on `dev.to` about nearly shipping a RAG assistant that invented API documentation and a citation URL that didn't exist. The worst hallucination, an Interswitch webhook signature, had a top-1 similarity score of 0.712, while a correct decline for an out-of-domain topic was only 0.691. There was no threshold to filter out the bad answers without losing the good ones. This isn't just about models making small mistakes. It points to a deeper issue with their understanding of constraints and long-term coherence. It feels like we're still grappling with models that prioritize plausible-sounding output over factual accuracy or actual task completion. For coding agents, this means they can't reliably perform the kind of deep, systemic changes that real technical debt requires. Even with the advanced verification in `SWE Refactor Bench`, the best model, claude-opus-5, only scored 47.0/100. Among the 340 runs that did pass the Migration Audit, 58% reached 99% of the fixed checks, but only 26% reached 100%. That last 1% is where all the real work lives. I don't know if simply throwing more training data or tweaking prompts will fix this. It seems more fundamental. Maybe it's an architectural problem, or perhaps we need entirely new ways to imbue these models with a verifiable understanding of task completion beyond generating plausible-looking code or text. The current state suggests that our agents are still prone to significant, hard-to-detect errors, especially when the task involves complex, interconnected systems.