# A/B Test Readout Check A/B Test Readout Check is a tested SKILL.md that checks the written readout of ONE A/B test (a paragraph, an email, a table or a slide transcript) for the five statistical mistakes that turn a no-effect test into a "win", and lists only those, one line each with a fixed code, the place, the evidence with the arithmetic and the fix - or exactly No findings; an agent buys it once for $0.01 over x402. - Page: https://aiskills402.com/skills/ab-test-readout-check - Category: Data & Analysis (https://aiskills402.com/categories/data) - Price: $0.01 once, USD-priced, paid in USDC on Base over x402. Price as loaded on this page. The 402 response your agent receives is authoritative. - Version: 1.0.0 - Card (JSON): https://api.aiskills402.com/v1/skills/ab-test-readout-check ## Use it when Checks the written readout of ONE A/B test (a paragraph, an email, a table or a slide transcript) for the five statistical mistakes that turn a no-effect test into a "win", and lists only those, one line each with a fixed code, the place, the evidence with the arithmetic and the fix - or exactly No findings. It catches stopping the moment a result looks significant, many metrics or variants judged without correction, a split of users that differs from the planned split by more than chance allows (the sample ratio check, calculated step by step), a sample too small for the effect the team wants to detect, and a relative lift reported without the absolute numbers behind it. It leaves out the safe look-alikes, such as a pre-registered single primary metric, guardrail metrics that only monitor, and a split of 50.2 to 49.8 on a small sample. Use before a test result is shipped, when an analyst or an agent writes a readout, or to review a result someone else announced. ## Not for Designing an experiment before it runs, recomputing p-values or intervals, choosing a statistical test, Bayesian readouts, segment analysis, or deciding whether to ship. It reads one pasted readout and judges only how its result was reached. ## Tested, honestly Tested 2026-10-19. - Strong model (claude-sonnet-5-5 (Claude Code alias "sonnet")): Right on all 3 cases, read by hand. On a checkout-button readout it gave the early stop on day 9 of 14 and the uneven split of 48,300 against 50,100, with the sample ratio arithmetic done correctly to 32.93, and it left out the three guardrail metrics. On a six-metric table it gave the uncorrected win at p of 0.03 and the lift reported only in percent, and left out the 4,012 against 3,988 split. On a pre-registered signup readout it answered exactly No findings. - Weak model (claude-haiku-5-5 (Claude Code alias "haiku")): Right on 2 of 3 cases, read by hand. Both planted readouts got exactly the expected lines, with the same correct arithmetic (32.93 against the limit 10.83) and the safe look-alikes left out. On the clean signup readout it wrote No findings. and then added a paragraph that the variant had 39,880 users against a planned minimum of 40,000. That is a true observation about a wrinkle we did not intend in our own test text, but it breaks the fixed sentence, so it is recorded as a wrong answer. Note: Light test: three cases (two planted, one clean), one run per model; no comparison without the skill. The three readouts were written by us; each planted one carries a safe look-alike (guardrail metrics, a 4,012 against 3,988 split, a relative lift printed next to absolute rates in the clean case), and neither planted case has a comment pointing at the defect. The clean readout accidentally says the planned sample was at least 40,000 per group while the variant has 39,880; we did not edit it after the run. Facts come from public statistics references, read on 19 October 2026; there are no owner-measured facts. Full summary: https://aiskills402.com/skills/ab-test-readout-check/tests ## Example ### English (claude-sonnet-5-5) Before: > Readout: onboarding redesign (variant B against control A) Ran for the full 14 days as planned. Users were split 50 to 50: A got 4,012 users and B got 3,988 users. Metric Change for B p-value Add-to-cart rate +14 percent 0.03 Checkout start rate +2 percent 0.41 Purchase rate +1 percent 0.68… After: > MULTIPLE-COMPARISONS — add-to-cart rate "clear win" among six metrics — six metrics were tested (add-to-cart, checkout start, purchase, average order value, return visit, support tickets), no primary metric is named, and the win is declared at the plain p = 0.03 < 0.05; a Bonferroni threshold for 6 metrics is 0.05 / 6 ≈ 0.0083, which 0.03 does not meet — name the primary metric as planned before… ## What is in the file - The answer - The codes - The sample ratio check, step by step - The sample size check - How to read the readout - Rules - Work in this order - Short example ## Languages English ## How to buy Agent (HTTP): 1. GET https://api.aiskills402.com/v1/skills/ab-test-readout-check/file without a payment header. The answer is 402 with a PAYMENT-REQUIRED header (x402 v2): exact amount, asset, network, recipient. 2. Sign `accepts[0]` with an x402 client (for example @x402/core + @x402/evm). 3. Repeat the GET with the signature in the PAYMENT-SIGNATURE header. The answer is 200 with the file, its sha256 and a re-download token. Agent (MCP): https://mcp.aiskills402.com/mcp — free tools search_skills, get_skill, redownload_skill. Buying itself is over HTTP. Full flow: https://aiskills402.com/docs ## The file - Version: 1.0.0 - Size: 9.1 KB (9307 bytes) - SHA-256: 145ce399f8b5cd12e4d48092b23b11a604379ef142906c6dbc20e42115350a8c - Updated: 2026-10-19 - New versions are free through your re-download token. ## Versions ### 1.0.0 (2026-10-19) First release, written in the light-test trial of 19 October 2026 (three cases, price $0.01). The idea is ours; no foreign skill was opened, facts come from public guidance (see notes/ideas.md). Output: one line per finding with a fixed code, or exactly `No findings.` Test: three cases written by hand (two planted, each with a safe look-alike, one clean control), one run on Claude Sonnet and one on Claude Haiku, no run without the skill, no iteration on the skill text after the runs. `test/control.mjs` shows that the checks pass on the ideal answers and fail on realistic wrong ones. See `test-summary.json` for the verdicts read by hand. ## License Perpetual, non-exclusive; use and modify for yourself incl. paid work; no resale or republishing. Holder: Georgi Kalchev, aiskills402.com. Terms: https://aiskills402.com/docs#license ## FAQ ### When does a split of users count as a mismatch? It works through the arithmetic in the answer: total users, expected count per group from the planned split, squared difference over expected for each group, then the sum against the limit 10.83 for two groups. A 20,000 against 21,000 split on a planned 50 to 50 is reported; 502 against 498 is not. ### Does it demand a power calculation from every readout? No. The sample size is checked only when the readout itself names the smallest change the team hoped to find, using a rule of thumb that the skill shows with a worked example. A readout that gives no minimum effect is not judged underpowered, because the skill never invents a number the text does not contain. ### What does it leave out on purpose? Guardrail metrics shown only for monitoring, a single pre-registered primary metric beside other listed numbers, a relative lift printed next to both absolute rates, and a small-sample split that passes the check. These are the look-alikes in our three test cases, and none of them gets a line. ### How was it tested? Our trial used an early stop with an uneven split, a six-metric table with a lift in percent only, and a clean pre-registered readout. Claude Sonnet and Claude Haiku each answered once per readout. Sonnet matched all three; Haiku matched the two planted ones and broke the fixed clean sentence with an extra paragraph. ## Measurement limits - Models other than the two named above were not run. - Each verdict comes from the test run on the date shown; the skill may have changed since (check the version). - Full test inputs are not published here, only short excerpts of our own text. - Results on your own texts, languages and domains can differ. Offer note: Paid in USDC (USD-pegged) over x402 by an AI agent; one-time.