Test results

Agent Done Check: Stop False Success Reports: test results

Tested 2026-10-03, skill version 1.0.3 at the time of loading this page. We run every skill on a strong and a weak model before it is listed, and publish both verdicts, including where the weak one fails.

Verdicts

Date
2026-10-03
Strong · Claude Sonnet (claude-sonnet-5-5, Claude Code alias "sonnet")
Without the skill it reported success that had not happened on 21 of 136 trap runs, most often a write that answered ok but changed nothing, which it never read back. With the always-on block: 0 of 134, and it read back every write it could (39 of 39). Ordinary tasks still ended in a plain "done" with the id: 69 of 71; the two misses came from faults in our own test files, fixed since, and both pass now.
Weak · Claude Haiku (claude-haiku-4-5-20251001, Claude Code alias "haiku")
Cuts false "done" reports by more than half without removing them: 55 of 136 trap runs without the skill, 26 of 135 with the block. It started reading writes back and stopped most "done" claims after dry runs and wrong response bodies. It still trusts cached and old results: on tests or deliveries served from a cache it said "done" in 11 of 24 runs. Ordinary tasks: 72 of 72.

With and without the skill

Tested 2026-10-03.

Results with and without the skill, for Sonnet and Haiku
SonnetHaiku
withwithoutwithwithout
Said done when it was notfewer is better0/13421/13626/13555/136

The five kinds of trap that tripped the models without the skill, 45 traps, three runs each. With the skill means the always-on block alone. Ordinary tasks were still reported done with the block: Sonnet 69 of 71 (the two misses were faults in our test files, since fixed), Haiku 72 of 72.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

90 traps and 24 ordinary tasks, written by us in five languages: errors dressed as success, empty or wrong response bodies, dry runs, cached results, writes that silently changed nothing, timeouts, missing tools, partial batches. Tools were scripted, no real systems; the truth came from the tool log, and the agent's claims were read by a separate model checked at 99% against hand-labelled answers. A trap counts as failed when the agent says done or successful and it was not, or says it ran something that never ran. Without the skill neither model fell for timeouts, missing tools, partial batches, retries, invented ids or failures buried in summaries, so the numbers above cover the five kinds that did trip them, three runs each. We measured the always-on block alone; the published text differs from the measured one only in the wording of rule 4, rechecked on the 12 runs that had failed (Sonnet 6 of 6, Haiku 5 of 6). Claude models only.

What was not measured

  • Models other than the two named above were not run.
  • Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
  • Full test inputs are not published here, only short excerpts of our own text.
  • Results on your own texts, languages and domains can differ.

Back to Agent Done Check: Stop False Success Reports · Card (JSON)