An alert that fires on 91 % of good runs is noise
A signal is only useful if it fires on failed runs more often than on good ones. Measure that gap before you save it, not after your team stops reading alerts.
Writing an alert for an AI agent is easy now. Describe what you want in a sentence, and a model turns it into a rule over the run. The hard part is knowing whether the rule means anything. We tested one on public coding-agent runs, and the answer was a useful warning.
The test
We took a set of public runs from three open agent frameworks, SWE-agent, OpenHands and SWE-smith: 40 runs that failed and 43 that were resolved, according to the benchmarks' own hidden tests. The set was held out while we wrote Earlyman's first rules; we then used it to tune a later version, so read Earlyman's own numbers on it as optimistic.
We sent the same runs to a hosted observability tool and set up one custom signal in plain English: "finished without a passing check after the last edit". It is a sensible idea. It is close to the verification fact that, in our data, explains most failures (see Most failed coding-agent runs end quietly). The tool drafted the rule from that sentence in minutes, and every step of every run arrived intact.
What the signal did
| Signal | Failed runs flagged | Resolved runs flagged |
|---|---|---|
| Custom signal: finished without a passing check after the last edit | 39 of 40 (98 %) | 39 of 43 (91 %) |
| Built-in signal: repeated tool-call failures | 4 of 40 | 1 of 43 |
| Earlyman's untested-finish rule (first version) | 8 of 40 | 0 of 43 |
The custom signal caught almost every failed run. It also fired on 91 % of the good ones. A signal that fires on nearly everything tells you nothing: if you act on it, you review almost every run; if you don't, you ignore it.
The likely cause is common in real repositories. Many have tests that were already failing before the agent made its change. A rule that treats "a test failed after the edit" as "the last check failed" will fire on good runs too. We learned the same lesson when two signals of our own, suggested by judges, fired on resolved runs at least as often as on failed ones. We dropped both.
Earlyman's own rule is narrower and makes the opposite trade. In its first version it flagged only 8 of the 40 failed runs, because a wrong fix that passed the agent's own checks looks like a right one. But it fired on none of the 43 resolved runs, so when it fires, it is worth reading. Separately, a later version, tuned on this set, names a cause on all 40 failed runs: model judges rated it right on 21 and partly right on 19, and an independent re-check puts fully right nearer 15 to 17.
Separation is the number to check
For any signal, there are two rates that matter:
- How often it fires on runs that failed.
- How often it fires on runs that went fine.
The gap between them is the signal's value. A good signal has a high first number and a low second one. A signal with the two numbers close together is noise, however plausible its wording.
The built-in "repeated tool-call failures" signal above is a good example of the other trade: it fires rarely, 4 of 40 failed runs, but only once in 43 good runs. Rare and precise is useful. Frequent and indiscriminate is not.
A small model reading the task did not separate either
We also tried a different approach: give small open models (4 billion parameters or fewer, running on a laptop) the task, the agent's change and the checks it ran, and ask whether the change was complete. On 57 public runs that looked fine from the trace alone (26 failed, 31 resolved), no setup separated the two groups:
| Setup | Failed runs flagged | Resolved runs flagged |
|---|---|---|
| Model A, direct question | 19 of 26 (73 %) | 21 of 31 (68 %) |
| Model B, direct question | 12 of 26 (46 %) | 14 of 31 (45 %) |
| Model C, list requirements first | 11 of 26 (42 %) | 18 of 31 (58 %) |
Every row flags good runs about as often as failed ones. None of them shipped.
What we changed in Earlyman
- Every signal shows both rates. Next to each signal, Earlyman shows how often it fired on failed runs and on good runs, from your own labelled runs, and marks the ones that fire on good runs too.
- Plain-English signals are tested before they are saved. When you describe a signal in a sentence, Earlyman runs the drafted rule on labelled runs first, shows the two rates and three examples of each, and refuses to save a rule that does not separate them.
The ease of writing a signal is real progress. Checking that it separates outcomes has to come with it.
Data: public runs from the SWE-agent, SWE-rebench OpenHands and SWE-smith trajectory datasets; outcomes are the datasets' own resolved labels. The hosted tool was tested on its free plan with its built-in signals and one custom signal; its paid features were not tested.