Retry Backoff Review: test results

Verdicts

Date

2026-10-26

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 3 cases, read by hand. On the charge loop it gave the missing random part, the endless loop and the repeated POST to the charges endpoint, and left out the capped profile read with jitter that sits beside it. On the order sync it found the ignored Retry-After header and the retry of every client error, and left out the PUT, which is idempotent, and the jittered six-try loop. On the clean report fetch it answered exactly No findings, so the wait read from Retry-After and the single POST passed.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 3 cases, read by hand. It matched the strong model on both planted files, with fixes that name the exact line to change, and kept the capped read and the PUT out. On the clean fetch it answered exactly No findings. Its charge answer is longer than the strong model's but stays one line per code.

Note

Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The three inputs were written by us; each planted one carries a safe look-alike and neither planted case has a comment pointing at the defect. Facts come from public sources (an AWS article and RFC 9110), read on 26 October 2026; the set of retryable statuses is our own policy. There are no owner-measured facts.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.