Checks the written readout of ONE A/B test (a paragraph, an email, a table or a slide transcript) for the five statistical mistakes that turn a no-effect test into a "win", and lists only those, one line each with a fixed code, the place, the evidence with the arithmetic and the fix - or exactly No findings. It catches stopping the moment a result looks significant, many metrics or variants judged without correction, a split of users that differs from the planned split by more than chance allows (the sample ratio check, calculated step by step), a sample too small for the effect the team wants to detect, and a relative lift reported without the absolute numbers behind it. It leaves out the safe look-alikes, such as a pre-registered single primary metric, guardrail metrics that only monitor, and a split of 50.2 to 49.8 on a small sample. Use before a test result is shipped, when an analyst or an agent writes a readout, or to review a result someone else announced.
A/B Test Readout Check is a tested SKILL.md that checks the written readout of ONE A/B test (a paragraph, an email, a table or a slide transcript) for the five statistical mistakes that turn a no-effect test into a "win", and lists only those, one line each with a fixed code, the place, the evidence with the arithmetic and the fix - or exactly No findings; an agent buys it once for $0.01 over x402.
Not for
Designing an experiment before it runs, recomputing p-values or intervals, choosing a statistical test, Bayesian readouts, segment analysis, or deciding whether to ship. It reads one pasted readout and judges only how its result was reached.
Tested, honestly
Tested 2026-10-19 with a strong and a weak model.
- SonnetStrong model, claude-sonnet-5-5
- Right on all 3 cases, read by hand. On a checkout-button readout it gave the early stop on day 9 of 14 and the uneven split of 48,300 against 50,100, with the sample ratio arithmetic done correctly to 32.93, and it left out the three guardrail metrics. On a six-metric table it gave the uncorrected win at p of 0.03 and the lift reported only in percent, and left out the 4,012 against 3,988 split. On a pre-registered signup readout it answered exactly No findings.
- HaikuWeak model, claude-haiku-5-5
- Right on 2 of 3 cases, read by hand. Both planted readouts got exactly the expected lines, with the same correct arithmetic (32.93 against the limit 10.83) and the safe look-alikes left out. On the clean signup readout it wrote No findings. and then added a paragraph that the variant had 39,880 users against a planned minimum of 40,000. That is a true observation about a wrinkle we did not intend in our own test text, but it breaks the fixed sentence, so it is recorded as a wrong answer.
Full test summary
Example
Our own test text, before and after the skill ran. Excerpts only.
English · claude-sonnet-5-5
Before
Readout: onboarding redesign (variant B against control A)
Ran for the full 14 days as planned. Users were split 50 to 50: A got 4,012 users and B got 3,988 users.
Metric Change for B p-value
Add-to-cart rate +14 percent 0.03
Checkout start rate +2 percent 0.41
Purchase rate +1 percent 0.68…
After
MULTIPLE-COMPARISONS — add-to-cart rate "clear win" among six metrics — six metrics were tested (add-to-cart, checkout start, purchase, average order value, return visit, support tickets), no primary metric is named, and the win is declared at the plain p = 0.03 < 0.05; a Bonferroni threshold for 6 metrics is 0.05 / 6 ≈ 0.0083, which 0.03 does not meet — name the primary metric as planned before…
What is in the file
- The answer
- The codes
- The sample ratio check, step by step
- The sample size check
- How to read the readout
- Rules
- Work in this order
- Short example
Languages
English. Tried in: English.
License
Perpetual, non-exclusive; use and modify for yourself incl. paid work; no resale or republishing. Holder: Georgi Kalchev, aiskills402.com. Full terms.
Versions
Current version 1.0.0, updated 2026-10-19. Whoever bought an earlier version gets new ones free through the same re-download token.
v1.0.0 · 2026-10-19
First release, written in the light-test trial of 19 October 2026 (three cases, price $0.01). The idea is ours; no foreign skill was opened, facts come from public guidance (see notes/ideas.md). Output: one line per finding with a fixed code, or exactly `No findings.`
Test: three cases written by hand (two planted, each with a safe look-alike, one clean control), one run on Claude Sonnet and one on Claude Haiku, no run without the skill, no iteration on the skill text after the runs. `test/control.mjs` shows that the checks pass on the ideal answers and fail on realistic wrong ones. See `test-summary.json` for the verdicts read by hand.
FAQ
When does a split of users count as a mismatch?
It works through the arithmetic in the answer: total users, expected count per group from the planned split, squared difference over expected for each group, then the sum against the limit 10.83 for two groups. A 20,000 against 21,000 split on a planned 50 to 50 is reported; 502 against 498 is not.
Does it demand a power calculation from every readout?
No. The sample size is checked only when the readout itself names the smallest change the team hoped to find, using a rule of thumb that the skill shows with a worked example. A readout that gives no minimum effect is not judged underpowered, because the skill never invents a number the text does not contain.
What does it leave out on purpose?
Guardrail metrics shown only for monitoring, a single pre-registered primary metric beside other listed numbers, a relative lift printed next to both absolute rates, and a small-sample split that passes the check. These are the look-alikes in our three test cases, and none of them gets a line.
How was it tested?
Our trial used an early stop with an uneven split, a six-metric table with a lift in percent only, and a clean pre-registered readout. Claude Sonnet and Claude Haiku each answered once per readout. Sonnet matched all three; Haiku matched the two planted ones and broke the fixed clean sentence with an extra paragraph.