Flaky Test Review: test results

Verdicts

Date

2026-10-26

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 3 cases, read by hand. On the pytest file it found the three-second pause and the module-level list that the second test relies on, and left out the seeded generator and the fake clock fixture. On the Jest file it found the expected month tied to today and the real rates address, and left out the seeded winner test. On the clean Jest file, with its frozen clock, mocked module, fixed seed and hand-moved timers, it answered exactly No findings.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on all 3 cases, read by hand. It matched the strong model on both planted files, with longer reasons that add the case of running a test alone or in random order, and kept every safe look-alike out. On the clean file it answered exactly No findings. One answer was indented by four spaces, which the check accepts.

Note

Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The three inputs were written by us; each planted one carries a safe look-alike and neither planted case has a comment pointing at the defect. The public source was read only through its reader replies, not the article body, so the skill claims little from it; the five codes are our own list. There are no owner-measured facts.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.