Agent Loop Review: Repeats, Caps, Self-Grading: test results

Verdicts

Date

2026-10-18

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 30 setups by content, read by hand: it found the charge retried without a key, the loop that retries a dead feed for ever, the loop that ends when the model says DONE, the marker written after the mail is sent, the batch of purchases and the batch of notices that go on after a refusal that repeats, the admin token and the wide token scopes, the auto-merge and the bulk send with no approval, the empty list that makes a dead source look healthy, the parallel writers of one file and of one counter row, and a loop that is both uncapped and self-judged. It answered No findings. for the sound setups. By the skill's own strict format 26 of the 30 were exact: in the other four it added a code the paste does not support (silence on a retry loop, no-state on a prompt loop that does write its result, no-state on the newsletter claim that the skill itself recommends, a repeat-save finding on a job whose helper is not pasted).

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 30 setups by content, read by hand: the same defects as Sonnet were named, each with a fix and the verdict line. By the skill's own strict format only 17 of the 30 were exact: in the others it added a code the paste does not support, mostly an approval or silence or no-state finding on a loop that only reads, only proposes or only writes a draft. It is the less careful reviewer of sound code: it reported a problem on some sound setups where Sonnet answered No findings.

With and without the skill

Tested 2026-10-18.

Same request on both sides; the bare side may use any words. Read by hand, Sonnet and Haiku without the skill both found every planted defect in all 30 setups and left the sound ones alone: the double tick, the missing key, the endless retry, the self-judged loop, the marker after the effect, the batch that goes on, the wide token, the missing approval, the dead source that looks healthy, the shared file and counter. Several bare answers first looked wrong only because of wording, and those checks were widened for both sides. So the content gain is zero for both models; what the skill adds is the fixed code per finding, the place and the fix in a set order, and an exact verdict line, which the bare answers do not give.

Same cases and the same checks with and without the skill. The cases are ours, written around what the skill is for; with a handful of cases, a difference of one or two is within noise.

Note

Thirty setups of an agent loop written by us (21 with a planted fault, of which 3 hard, and 9 sound), pasted as code, configuration or a log tail; the shared request states the rules (a comment is a claim, a watched script needs no approval step, today is 18 October 2026). Each answer is scored twice: by the skill's own format (the exact set of codes, the verdict, a bare No findings. for sound code, no fence) and by content in any words, with a fence removed first. Nine setups were changed after the first run because the paste really carried a second defect nobody planted, and three controls were rewritten once more after a second read; all of them were run again on both sides, so the numbers are final for all 30. Facts: the double scheduled tick (12 October 2026) and the shared-file corruption (8 October 2026) are the owner's own measurements and were NOT re-checked; the rest is design judgement, not a measured platform fact. Content checks widened after the runs, for both sides: the cap, the self-grading, the silence and the stop phrasings, the words for losing everything at the end, the state and approval-date phrasings, and the sound-code checks for a double charge said of a caller that is not pasted, for overwriting in unrelated places and for a missing spend limit. Several extra codes are accepted as optional where a second defect is real by the skill's own definitions. One run per model and setup, the latest run of each.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.