Does letting the coding agent re-run the tests improve its performance?
Read the full post on Builder CenterWhen we published SWE-InfraBench, the headline was how badly LLMs handled infrastructure-as-code. The benchmark holds 100 tasks drawn from real AWS CDK repositories, each asking for a specific modification described in natural language and graded by whether the generated CloudFormation passes the repository's own unit tests. Claude 3.7 Sonnet, the strongest model tested, solved 34% in a single attempt. Giving it the test errors and a second attempt reached 65%, which still left a third of the tasks unsolved.
The paper argued that the missing piece was iteration rather than model capability, so we went back and tested that directly. We ran Kiro agent with Claude Sonnet 5 across all 100 tasks in two configurations: (i) two-turn agent run matching the original protocol; (ii) unrestricted agent capable of running tests in a sandbox as often as it likes within 600 seconds.
The two-turn agent with Sonnet 5 reached 69%, while the unrestricted agent solved 82% of tasks, substantially surpassing the previous best result. The error breakdown is even more interesting: logical errors, where the code is partially right, fall from 17% to 3% once the agent can re-run the tests, but syntax errors barely move. The loop is not making the model better at CDK, it is letting a nearly-correct answer converge after a few iterations.
Read the full write-up on the Builder Center for the detailed results and the evaluation setup.
