Why we kept rules over a model that made things up

Earlyman explains why an agent run failed. There are two ways to build that. One is a fixed pipeline: read the run's steps, track what failed, what was retried and whether the last change was checked, and rank the likely causes with rules you can read. The other is to hand the run to a language model and ask it.

We tried both on the same failed runs, with judges who did not know which answer came from which. The fixed pipeline won by a wide margin, and the reason is worth sharing.

The test

• Runs: 40 failed runs from public, openly licensed datasets of three agent frameworks (SWE-agent, OpenHands and SWE-smith), held out while we wrote the first rules. The fixed answers below came from that first rule set.
• Model: a small open model of about 4 billion parameters, running locally on a laptop. Runs never leave the machine in Earlyman, so a local model is the honest comparison.
• Judging: independent model judges, one per framework. For each run they saw the full step list (the agent's final patch was not in their material), plus two unnamed answers in random order. They graded the cause and the suggested fix, flagged any fact the run contradicted or did not show, and picked the better answer.

The result

This side-by-side judging is stricter than grading one answer at a time: graded alone, these fixed answers scored 35 of 40. A later version of the rules, tuned on this same set, scored 21 right and 19 partly right by the judges, and about 15 to 17 fully right on an independent re-check. The table above is the first version; what matters is the comparison.

What "made things up" looked like

The model's answers read well. They were specific and confident. Most were wrong in ways you could only catch by checking the run yourself:

• syntax errors in patches that were valid;
• missing imports that were not missing;
• placeholder bugs that never appeared in the code.

When the model was right, it was very good. On 4 of the 40 runs it spotted something the rules could not, such as a swapped width and height, and the judges preferred it. But you cannot tell those 4 apart from the 29 without doing the investigation yourself, which is the work the tool was meant to save.

Constraining the model helped, but did not add accuracy

We then tried a stricter setup. The model no longer wrote a free answer. It picked one of the candidate causes the rules had already found, and a checker rejected any reply that introduced a fact not present in the run.

That stopped the invention: two of the three small models we tried produced no invented facts. But it did not make the answers more accurate than the rules alone, and judges found the model's wording clearer on only 2 of 30 runs. The model added words, not information.

Why the fixed pipeline wins here

• It rarely invents, and you can check it. In this test 1 of its 40 answers had a fact the run did not show. Every line it shows cites a step number you can open.
• It is fast and private. It runs on your machine without a model call and never needs the text of your prompts.
• It improves where it is measured. On the same public datasets its cause ranking went from right on 4 of 30 failed runs (13 %) to right on 21 of 40 by the judges (about 15 to 17 on an independent re-check) and partly right on the rest, by learning which facts matter. That second set was held out while we wrote the first rules, then used to tune the version that scored 21, so read it as optimistic (see Most failed coding-agent runs end quietly).
• When it is unsure, it says so, instead of filling the gap with a plausible story.

What we changed

• The fixed pipeline is Earlyman's default and only source for the cause, the outcome and the fix. No model output changes them.
• The local-model investigator is off by default. When you turn it on, its answer is labelled as coming from a local model, and its confidence is capped when it only cites steps that worked.

Small models may close this gap. A larger one might too; we did not test models above 4 billion parameters here, because they would not fit beside everything else on the laptop. Until a model can explain a failed run without inventing facts, a plain answer you can check beats a fluent one you have to verify.

Data: public runs from the SWE-agent, SWE-rebench OpenHands and SWE-smith trajectory datasets. Judges were independent model judges; between judges, results vary by about five runs in forty.