Coding Agents Fail 95% of Whole-Repository Refactors

2026-08-25

The SWE Refactor Bench paper shows coding agents can't handle long-horizon tasks, with only 28 out of 520 runs passing all evaluation stages.

The SWE Refactor Bench paper just landed, and it's a sobering look at where coding agents stand for anything beyond trivial tasks. Across 520 runs from 8 frontier models, only 28 passed all three evaluation stages. That's a 5.4% success rate for long-horizon, whole-repository stack migrations. You can't ship that.

The benchmark focuses on migrating entire codebases, tackling four distinct types of technical debt. Its three-stage protocol is smart: a Migration Audit to confirm the refactor happened, Behavioural Tests for correctness, and Agentic Verification where six other agents generate targeted tests to find hidden behavioral differences. Most existing benchmarks only hit the second stage, meaning agents can copy the old code, pass the tests, and declare victory. The SWE Refactor Bench calls this 'Blindness,' and it's a huge problem.

Daniel Nwaneri’s StacksNG report describes an invented Interswitch answer and citation during his adversarial tests. His three-case table reports top-1 similarity of 0.718 for a correct in-corpus answer, 0.712 for the fabrication and 0.691 for a correct out-of-domain decline. These examples show that a plausible retrieval score does not establish provider-specific grounding. They do not prove that no rejection threshold can separate the displayed cases: a cutoff between the two higher scores would retain the reported correct answer and reject the other two, including the case where declining was desirable. Nor do three examples establish a reliable threshold for unseen queries.

The author’s report motivates entity-specific evidence checks and held-out tests; it is not a certification of every result or a general impossibility theorem.

This isn't just about models making small mistakes. It points to a deeper issue with their understanding of constraints and long-term coherence. It feels like we're still grappling with models that prioritize plausible-sounding output over factual accuracy or actual task completion. For coding agents, this means they can't reliably perform the kind of deep, systemic changes that real technical debt requires.

Even with the advanced verification in SWE Refactor Bench, the best model, claude-opus-5, only scored 47.0/100. Among the 340 runs that did pass the Migration Audit, 58% reached 99% of the fixed checks, but only 26% reached 100%. That last 1% is where all the real work lives.

I don't know if simply throwing more training data or tweaking prompts will fix this. It seems more fundamental. Maybe it's an architectural problem, or perhaps we need entirely new ways to imbue these models with a verifiable understanding of task completion beyond generating plausible-looking code or text. The current state suggests that our agents are still prone to significant, hard-to-detect errors, especially when the task involves complex, interconnected systems.