A/B Test Calculator

What the A/B test calculator does

An A/B test compares two versions of something, a page, an email, a button, to see which gets a better response. This calculator takes the results of both versions, the number of people who saw each and the number who converted, and tells you whether the difference between them is real or just noise.

It answers the question every experiment comes down to: is version B genuinely better, or did it only look better by chance? Below is how it decides.

How to use it

1. Enter each group's sample size, the number of people who saw that version.
2. Enter each group's positive results, the number who converted.
3. Choose a confidence level, then press Calculate, or Reset to clear it.

How the test works

The calculator first works out each group's conversion rate, the positive results divided by the sample size. Then it runs a two-proportion z-test, which asks whether the gap between those two rates is larger than the natural variation you would expect from samples of that size.

It produces a z-score for the difference and compares it against a cut-off set by your confidence level. If the z-score is bigger than the cut-off, the difference is unlikely to be chance, and the result is called statistically significant. If not, the difference is within what chance could easily produce.

What significant, and not significant, means

A significant result means the difference between the two versions is larger than sampling noise comfortably explains, so you can be reasonably confident one version really does outperform the other. That is the green light to act on the winner.

A not significant result does not mean the versions are equal. It means you do not yet have enough evidence to tell them apart. The true difference might be real but small, or your sample might simply be too small to detect it. In that case the honest conclusion is that the test is inconclusive, which usually calls for more data rather than a decision.

A worked example

Suppose the control version was shown to 1,000 people and 100 converted, a 10 percent rate, while a new version was shown to 1,000 people and 130 converted, a 13 percent rate. You want 95 percent confidence.

The z-score for this difference works out to about 2.1, which is larger than the 95 percent cut-off of 1.96. So the result is statistically significant: the new version's higher conversion rate is unlikely to be chance, and the evidence supports rolling it out.

Reading the result well

A significant result tells you a difference is real, but not how big or how lasting it is. It is worth looking at the size of the gap between the rates, not just the verdict, since a difference can be statistically significant yet too small to be worth the effort of changing anything.

It also helps to decide your sample size and confidence level before you start, and to let the test run its planned course. Peeking at the results and stopping the moment they turn significant inflates the chance of being fooled by a fluke, which is one of the most common ways A/B tests mislead.

Entering your values

Enter both sample sizes and both conversion counts, with each count no larger than its sample size, and pick a confidence level of 90, 95, 98, or 99 percent. The calculator shows both conversion rates, the z-score, the cut-off it was compared against, and the verdict. The test is most reliable when each group has a healthy number of conversions rather than just a handful.

Questions people ask

What does an A/B test calculator do?

It compares the conversion rates of two versions and tells you whether the difference is statistically significant, using a two-proportion z-test.

What does statistically significant mean here?

That the difference between the two conversion rates is larger than chance would readily produce, so one version can be trusted to outperform the other.

What if the result is not significant?

It means there is not enough evidence to separate the two versions, not that they are equal. Usually the answer is to gather more data before deciding.

Is a significant result always worth acting on?

Not necessarily. Significance says a difference is real, not that it is large. Check the size of the gap between the rates before deciding whether it is worth the change.

References

A quick note on where the methods here come from. The two-proportion z-test behind an A/B test is set out in the NIST/SEMATECH e-Handbook of Statistical Methods, the US government's public statistics reference. OpenStax Introductory Statistics is a free, widely used textbook covering tests that compare two proportions.

1. NIST/SEMATECH e-Handbook of Statistical Methods (comparing two proportions). https://www.itl.nist.gov/div898/handbook/
2. OpenStax, Introductory Statistics (comparing two independent population proportions). https://openstax.org/details/books/introductory-statistics-2e