Cockroach Labs ran coding agents like a hospital for five months: 1,238 PRs, 7 reverts
Two Cockroach Labs engineers built a coding agent pipeline on GitHub Actions modelled on a teaching hospital. Issues are "patients", and separate agent roles triage them, write a plan, have the plan reviewed, write the code, review it as a different agent, and sign it off. Over five months it handled over a million lines of code and 1,238 merged pull requests with 7 reverts. In one test it added IBM Db2 support to their migration tool in under two days for $4,172 in tokens, with no human-written code; similar Oracle support took 9 months and about $160,000 of engineering time in 2024. They add that they did not review the Db2 code at the time and are still confirming it is correct.
Their lessons: reviewing the plan before any code is written catches the costliest mistakes; any change over 1,000 lines must be split; and every step is logged. Problems: agents got bureaucratic (one one-word UI edit took five rework rounds), loops needed a hard stop after repeated rework, and prompt files bloated until one role loaded about 29,000 tokens per run.
On Hacker News (44 points, 16 comments), most comments were questions to the authors, such as whether they tested which parts actually help. One suggested adding post-merge testing. One sceptic said home-built agent "factories" pile on quality gates until they slow work down, and another noted nothing yet on long-term maintainability. No consensus.
Sorted take: copy the cheap parts first. Review the plan before code, cap change size, and set a rework limit, then add roles only when they earn their cost.
Why it matters: If your team runs coding agents, a plan review step and a size cap on each change are easy wins. Also prune your agent instructions regularly, since bloated prompts cost tokens on every run.
Cockroach Labs ↗Discussion on Hacker News ↗AI-assisted, reviewed by Sorted AI
▲ 0
0 comments