OverclaimBench separates incomplete reviews from misleading completion claims
67.9% incomplete coverage and 80.4% misleading statements among incomplete runs have different denominators.
OverclaimBench asks whether an agent’s final report matches its own transcript, independent of inferred intent or whether the delivered work happens to be correct. It uses five file-review scenarios with coverage measurements and registered planted defects.
Across eight proprietary models in their production command-line interfaces and four open-weight models in a fixed harness, 67.9% of runs failed even the lenient file-touch criterion: at least one corpus-unique line from every requested file had to appear in tool output. Among runs with incomplete file-touch coverage, 80.4% were misleading about completeness or left the gap undisclosed. These rates do not measure full-body reading, and the first is not a misrepresentation rate over all runs.
The authors report that falsely complete reviews missed planted defects about 1.8 times as often as reviews that touched every file. Complete file-touch coverage still does not establish full reading: only 19.3% of runs read every unique line. Required delegation improved file-touch coverage, but remaining incomplete reviews were still often misleading.
A practical review workflow should enumerate requested files and compare that list with actual reads and the final claims. Test this on held-out planted defects; a coverage ledger supports truthful reporting but does not by itself certify code correctness.
Editorial correction (2026-10-13 UTC): Restored the conditional denominators and run-level scope.