A/B Test Readout Check: test results

Verdicts

Date

2026-10-19

Strong · claude-sonnet-5-5 (Claude Code alias "sonnet")

Right on all 3 cases, read by hand. On a checkout-button readout it gave the early stop on day 9 of 14 and the uneven split of 48,300 against 50,100, with the sample ratio arithmetic done correctly to 32.93, and it left out the three guardrail metrics. On a six-metric table it gave the uncorrected win at p of 0.03 and the lift reported only in percent, and left out the 4,012 against 3,988 split. On a pre-registered signup readout it answered exactly No findings.

Weak · claude-haiku-5-5 (Claude Code alias "haiku")

Right on 2 of 3 cases, read by hand. Both planted readouts got exactly the expected lines, with the same correct arithmetic (32.93 against the limit 10.83) and the safe look-alikes left out. On the clean signup readout it wrote No findings. and then added a paragraph that the variant had 39,880 users against a planned minimum of 40,000. That is a true observation about a wrinkle we did not intend in our own test text, but it breaks the fixed sentence, so it is recorded as a wrong answer.

Note

Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The three readouts were written by us; each planted one carries a safe look-alike (guardrail metrics, a 4,012 against 3,988 split, a relative lift printed next to absolute rates in the clean case), and neither planted case has a comment pointing at the defect. The clean readout accidentally says the planned sample was at least 40,000 per group while the variant has 39,880; we did not edit it after the run. Facts come from public statistics references, read on 19 October 2026; there are no owner-measured facts.

What was not measured

• Models other than the two named above were not run.
• Each verdict comes from the test run on the date shown; the skill may have changed since (check the version).
• Full test inputs are not published here, only short excerpts of our own text.
• Results on your own texts, languages and domains can differ.