AgentSWE: coding agents rarely build the agent a brief actually asks for

Researchers at the Beijing Academy of Artificial Intelligence and the Chinese Academy of Sciences introduce AgentSWE, an arXiv paper (2610.13361) and Hugging Face Daily Paper, that tests whether coding agents can create, edit, or optimise other agents from a natural-language brief. The suite has 25 commissions: 10 creation, 10 editing of pinned products such as OpenHands or Aider, and 5 optimisation toward benchmarks like BrowseComp or OSWorld. Builders only see a few development cases; scoring runs the frozen agent on held-out cases with requirement rubrics or native objectives. Code is on GitHub. In the main Codex-harness comparison, the strongest builders averaged about 19.1 on creation and 31.6 on editing (0-100). In optimisation, none closed more than about 5% of the remaining gap to the target on average. Runs that scored 90+ on development cases averaged only about 9.9 on held-out cases. Recurring failures: overfitting to visible cases, hollow runtimes that look complete but fail when run, and unsupported completion claims. Authors' own results. Sorted take: if you commission an agent from a coding agent, treat demo cases as marketing. Put held-out acceptance tests in the brief before you trust the delivery.