AgentSWE: Can Coding Agents Build the Agent You Actually Want?

arXiv · Jiahao Wang, Hongjin Qian, Yuyang Hu, Jiajun Zhang, Zheng Liu · 2026-10-09

This item is no longer on the current shelf. Its page remains available, and editorial corrections may be made. See what is featured now.

AgentSWE tests whether a built agent meets the requirement on held-out cases, beyond the builder’s demo.

What it does

AgentSWE contains 25 commissions across agent creation, editing and optimization. Builders see illustrative cases; the frozen submission is accepted or rejected on held-out cases covering the requirement, using requirement-derived gates or the source optimization objective.

Why it matters

The authors identify recurring overfitting, hollow runtimes and unsupported completion claims. These are failure categories in the evaluated builders, not an independently established ranking of all agent-building failures.

How it applies

Write acceptance cases separately from the examples supplied to a builder. Exercise the frozen delivered runtime, including required behavior and constraints. Assess the actual result rather than treating a fluent completion message as evidence.

The short version

Grade the delivered runtime, not the builder’s story.

My take

This is a useful pattern for commissioning an agent. It does not show that one added test suite will eliminate security incidents or rollbacks; those outcomes require their own evidence.

How it connects

• Specification-derived acceptance
• Visible examples versus held-out cases
• Runtime behavior versus completion claims

Bottom line

Keep acceptance cases outside the builder’s examples and test the frozen submission.

Takeaways

• Creation, editing and optimization have different acceptance conditions.
• Freeze the submission before held-out evaluation.
• A demo is not evidence of full requirement coverage.
• Track the reported failure categories without inventing their population frequency.

Topics: coding agents, software engineering, benchmark, LLM evaluation