A/B test readouts: when the conclusion does not hold

A written test result can be wrong without a single wrong calculation in it. The p-value is computed correctly, the lift is real arithmetic, and the conclusion still does not follow, because of how the test was stopped, how many things were judged at once, or whether the groups were what the plan said they were. A readout is a document that can be checked on its face, before anyone looks at raw data. This guide describes five checks that need only the text of the readout and a calculator, and explains for each one why a failed check means the headline cannot be relied on.

Start with the plan, not the result

Before judging anything, find the plan as the document states it: how long the test would run or how many users it would collect, which single metric decided the outcome, what split of traffic was intended, and what size of change the team cared about. Every check below sets what happened against that plan. If the readout never states a plan, the checks that need one cannot run, and the missing plan is itself worth reporting, because a result without a stated plan can be fitted to any story afterwards.

Check one: was the test stopped when it looked good?

The usual significance calculation assumes that the number of observations was settled before the test began. Evan Miller's often-cited note explains that if a test is instead run until a significant difference appears, the reported significance levels stop meaning what they say (Evan Miller). In his worst-case example, where a test is examined after every observation and stopped at the first significant moment, a change that does nothing is declared a winner 26.1% of the time instead of the 5% most people expect (Evan Miller). The same page notes that looking at the data is harmless as long as the decision to stop does not depend on what was seen.

What to look for in a readout: phrases such as "the dashboard turned green on day eight, so we stopped" next to a plan that was longer. A test that ran to its intended finish and was analysed one time is acceptable. A test that was stopped early on a good-looking number is not, and the honest remedies are to run the test again until its intended finish, or to have chosen a sequential design beforehand, which fixes the decision checkpoints before launch (this is general practice, not from the sources above). Miller's own advice is simpler: fix the sample size in advance, and committing to it removes the problem (Evan Miller).

Check two: how many things were tested for a win?

Each test carries its own chance of a false positive, so the chance that at least one of many tests shows a false win grows with the number of tests (Wikipedia, multiple comparisons). A readout that tracks a dozen metrics, or compares several variants, and then announces the one that crossed the usual threshold has not found a win; it has found the most favourable of many draws.

The simplest correction is to test each comparison at a stricter level: the overall level divided by the number of comparisons. The source gives a worked case of twenty hypotheses at an overall level of 0.05, which becomes 0.0025 for each one (Wikipedia, Bonferroni correction). The same page warns that this makes real effects harder to detect, so the correction trades one error for the other (Wikipedia, Bonferroni correction).

Two things are not a problem. One metric named as the deciding one before the test began is a legitimate single comparison, even if other numbers are listed beside it. Metrics shown only to confirm that nothing broke, where nobody claims a gain, are monitoring and not a search for a win. This is general practice rather than a claim from the sources above.

Check three: do the group sizes match the plan?

If users were meant to be split evenly, the two groups should come out close to even, with only small chance deviations. A sample ratio mismatch is a statistically significant gap between the planned and actual group sizes, and it can be detected with a chi-squared test (Wikipedia, sample ratio mismatch). It matters because random assignment is the assumption that makes the comparison fair; a mismatch points to a problem in how users were assigned or how the data was processed, and the observed difference may then come from another cause than the change (Statsig).

The calculation by hand, for an even split (the figures in this paragraph are invented, and the method is the one on the sample ratio mismatch page): add the two group sizes, halve the total to get the expected size, then for each group square the gap from the expected size and divide by the expected size, and add the two results. Worked here with made-up numbers, 10,000 users in one group and 10,500 in the other give an expected 10,250 each; each group contributes about 6.1, and the sum is about 12.2. For two groups this is compared with a chi-squared table at one degree of freedom, where 12.2 corresponds to a chance of roughly one in two thousand. An uneven-looking split that passes this check, such as 502 against 498, is acceptable and needs no remark. This worked example is this guide's own arithmetic, not a measurement.

Check four: could the sample have detected the change at all?

A rule of thumb from the same Evan Miller note: the sample size n is about sixteen times the variance divided by the square of the smallest change worth detecting, and for a conversion rate the variance is the rate times one minus the rate (Evan Miller). With a baseline of 10% and a smallest change of one percentage point, the rule gives roughly 14,400 users in each group (sixteen times the rate and its complement, divided by the squared change, calculated for this guide; the note gives n and this guide applies it per group). A readout that names its smallest wanted change and then reports far fewer users per group than the rule asks for was unlikely to find that change, so a "no difference" verdict from it says little.

Judge this only when the document names the smallest change it set out to find. Guessing one on the readout's behalf is how a sound test gets blamed. The rule of thumb is rough and depends on the significance and power levels chosen, which is something to confirm in the readout before comparing.

Check five: is the lift given in real terms?

"Sign-ups rose by a fifth" cannot be judged. A relative lift hides the base: a fifth of a tiny rate is a tiny change, a fifth of a large one is not. The readout should print the rate or amount for each group and the counts behind it, so that a reader can see the absolute difference and judge whether it is worth acting on. If both rates are shown next to the percentage, the check passes. This is general reporting practice, not a claim from the cited pages.

Putting the checks in order

Run them in this order, and note one line for each problem found: which check failed, where in the readout it sits, the numbers that show it, and what to do about it. If none apply, say that nothing was found, and add nothing else. The aim is a list a busy person can act on, not a second opinion on the product idea. Whether to ship is a separate decision; a clean readout means only that the evidence deserves belief, not that the change is good.

When this does not apply

The five checks above work on the written account of a single completed test. They do not apply to designs built for continuous monitoring or to Bayesian analyses, where stopping early is part of the method, and the peeking argument above is written for the classic fixed-sample test. They cannot see raw data, the tool's settings or how users were assigned, so a readout that passes can still hide a broken experiment, and a readout with a mismatch can still be repairable. The sample-size rule is only a rough approximation and differs when other significance or power levels are used. The sources are one practitioner note and several encyclopedia pages, which this guide read but did not check against the original papers, and the worked numbers are illustrations, not data from real tests. The last checks also depend on figures a readout may leave out. When the stakes are high, a statistician should look at the raw data.

The same suspicion applies to a confident line written by an automated helper; for a way to look behind such a line, see the post on finished-task reports.

Checking a readout with a skill

The ab-test-readout-check skill takes the written result of a single A/B test and lists only the problems covered here: stopping on a good-looking moment, many metrics judged without correction, a group split that fails the ratio check, a sample smaller than the stated target needs, and a relative lift given without the underlying numbers. It shows its arithmetic where a check needs it, leaves out patterns that look odd but are safe, and gives a single fixed line when the result holds up.