Most failed coding-agent runs end quietly
Most failed public coding-agent runs leave no failing step behind. The most useful fact in a trace is whether the last change was ever checked.
When an AI coding agent fails, you expect to find a red step: a test that broke, a command that crashed, a tool that errored. We went looking for those red steps in public runs from three open agent frameworks, SWE-agent, OpenHands and SWE-smith. Mostly, they were not there.
What we measured
We imported public, openly licensed runs from the three frameworks into Earlyman. Each run comes with an outcome from the benchmark itself: resolved or failed, decided by tests the agent never saw. Earlyman read the runs without the outcome. Independent judges then read each failed run and, where the dataset has it, the hidden test log, and wrote down the real cause before grading Earlyman's answer. For the 40-run set described below, the agent's final patch was not in the judges' material.
Across 600 judge verdicts on failed runs from those three datasets, the real causes split like this:
| Real cause | Share of verdicts |
|---|---|
| Wrong fix | 40 % |
| Incomplete fix | 28 % |
| Stuck in a loop until the step budget ran out | 17 % |
| Broke code that used to work | 7 % |
| Stopped early with no fix | 6 % |
The first two rows are the important ones. 68 % of the verdicts named a wrong or incomplete fix (some runs were judged more than once), and in most of those runs nothing in the trace failed. The agent made a change, ran its own checks, saw them pass, and finished. Only the hidden tests failed.
In our first sample, 9 of the 10 failed OpenHands runs and 9 of the 10 failed SWE-smith runs ended with the agent's own last check passing.
Why error-first tools miss these
A tool that ranks steps by "what went red" does well on loops and crashes. On the first 30 failed runs we judged (10 from each framework), the first version of Earlyman's cause ranking did exactly that:
- It got 3 of the 5 loop failures right.
- It got 0 of the 20 wrong or incomplete fixes right.
- Overall it named the right cause on 4 of 30 (13 %).
Worse, the red steps it did find were often harmless: a search that found nothing, the expected traceback from the agent's own reproduction script before its fix, a typo the agent had already recovered from. Those took first place over the change that actually mattered.
The fact that matters: was the last change checked?
The signal that separated these runs was not an error. It was the state of verification at the end of the run:
- Did the agent change code after its last check?
- Did any check pass after the last change?
- Was a failing check the agent's own reproduction, which is supposed to fail before the fix?
In the public SWE-agent sample, "finished without a passing check after the last edit" was present in 46 of 62 failed runs.
Once Earlyman tracked verification state, discounted expected failures and recognised loops and step-budget endings, the cause ranking changed completely. We checked it on a second set of 40 failed runs (15 SWE-agent, 13 OpenHands, 12 SWE-smith) that was held out while we wrote the first rules, and then used to tune the version measured here. Model judges graded the cause right on 21 of 40, partly right on the other 19, and wrong on none. An independent re-check of a sample was stricter and puts the number fully right nearer 15 to 17 of 40. Three caveats: the version measured here was tuned on this set, so read these numbers as optimistic; judges had the hidden-test log for only 14 of the 40 runs, and the other 26 were judged from the run alone; and the agent's final patch was not in the judges' material. Resolved runs wrongly shown as failed fell from 15 of 31 (48 %) to 0 of 31 on the runs we tuned on, and are 2 of 43 (5 %) on the second set.
A signal has to fire on failed runs more than on good ones
The obvious next step is to add more signals. The judges suggested two, and we tested both on the same public runs before shipping anything:
| Suggested signal | Resolved runs flagged | Failed runs flagged |
|---|---|---|
| A repo test failed after the last change and that command never passed again | 10 of 43 | 4 of 40 |
| The agent edited an existing test after the repo's tests failed | 1 to 2 per set | 0 to 2 per set |
Both fire on good runs at least as often as on failed ones. Repositories often carry tests that were failing before the agent arrived, so "a test failed" says little about whether the change was right. We did not ship either.
A narrower signal held up: one tool failing six times in a row fired on 5 of 40 failed runs and 1 of 43 resolved runs in the second set. It is rare, but when it fires it means something, so it ships as a named signal.
What this means for your agent
- Look at the end of the run, not the red steps. The question to ask first is whether the final change was ever verified.
- Treat a passing check as evidence, not proof. A check the agent wrote for itself can pass while the task fails.
- Measure a signal on good runs before you trust it. If it fires on resolved runs as often as on failed ones, it is noise. We wrote more about that in the next post.
Data: public runs from the SWE-agent, SWE-rebench OpenHands and SWE-smith trajectory datasets. Outcomes are the datasets' own resolved labels. Causes were graded by independent model judges, which vary by about five runs in forty between judges.