Sample Ratio Mismatch Calculator

What the sample ratio mismatch calculator does

In an experiment, you decide in advance how to split people between the groups, often evenly. A sample ratio mismatch, or SRM, is when the split you actually got differs from the split you intended by more than chance can explain. This calculator checks for it, comparing your observed group sizes against your target allocation and reporting whether a mismatch is present.

It is a health check you run before trusting an experiment's results, because a broken split quietly poisons everything downstream. Below is why it matters and how the check works.

How to use it

1. Enter the sample sizes of your groups in the box, separated by spaces or new lines.
2. Choose the target allocation, equal or unequal. For unequal, enter the target ratios for each group.
3. Press Calculate for the SRM check and its p-value, or Reset to clear it.

Why a sample ratio mismatch matters

An SRM is a warning that something is wrong with how people were sorted into groups. If you planned a 50-50 split and got something clearly off, the randomisation did not work as intended, and that points to a bug: a redirect that drops some users, a tracking gap, a filter applied unevenly, bots landing in one group.

The danger is that whatever caused the mismatch has probably skewed the results too. A process that misroutes people is rarely doing so at random, so the groups may no longer be comparable, and the difference you measure could be an artefact of the bug rather than a real effect. That is why a detected SRM usually means you stop and fix the pipeline rather than trust the numbers. A clean split is a precondition for believing anything else the experiment says.

How the check works

The calculator works out how many people each group should have contained, by applying your target ratios to the total number observed. Then it runs a chi-square test comparing the observed group sizes against those expected sizes, measuring how far apart they are:

Chi-square = the sum of (observed minus expected)² ÷ expected, over every group

From that chi-square value and the degrees of freedom, the number of groups minus one, it computes a p-value. The p-value answers a precise question: if the split really were as intended, how likely is an imbalance at least this large by pure chance?

Reading the result

A small p-value means the observed split would be very unlikely if the allocation were working correctly, so a mismatch is flagged. This calculator draws the line at 0.01: below it, a sample ratio mismatch is reported, and at or above it, none is. The strict threshold is deliberate, since an SRM check is a background test you do not want firing on every harmless wobble.

A flagged mismatch is a signal to investigate the experiment setup, not a result to interpret on its own. And bear in mind that with very large samples, even a tiny imbalance can trip the check, because large numbers make small departures detectable. Either way, the right response to a detected SRM is to find and fix the cause before reading the experiment's outcome.

A worked example

Suppose you intended an equal split and observed 5,300 people in one group and 4,700 in the other, 10,000 in total. An equal split expects 5,000 in each. The chi-square value is (5300 minus 5000)² ÷ 5000 plus (4700 minus 5000)² ÷ 5000 = 18 plus 18 = 36.

With one degree of freedom, a chi-square of 36 gives a p-value far below 0.01, so the calculator reports a sample ratio mismatch. A 53-to-47 split may not look dramatic, but across 10,000 people it is far more lopsided than a working 50-50 assignment would produce, which is exactly the kind of quiet problem this check is built to catch.

Entering your values

Enter one sample size per group, separated by spaces or new lines. For an unequal design, add the target ratios in the same order, using any numbers whose proportions match your intended split, so 2 and 1 means a two-to-one allocation. The p-value is returned along with the verdict.

Questions people ask

What is a sample ratio mismatch?

When the actual split of people between experiment groups differs from the intended split by more than chance can explain. It signals a problem with how people were assigned.

Why does it matter?

Because the cause of the mismatch has usually skewed the results too, making the groups no longer comparable. A detected SRM means the experiment's findings cannot be trusted until the cause is fixed.

How is it detected?

With a chi-square test comparing the observed group sizes against the expected sizes from your target allocation, turned into a p-value. A very small p-value flags a mismatch.

Why the 0.01 threshold?

An SRM check runs in the background, so a strict threshold avoids false alarms on harmless variation. Below 0.01 a mismatch is reported, prompting you to investigate.

References

A quick note on where the methods here come from. The check is a chi-square goodness-of-fit test, which is set out in the NIST/SEMATECH e-Handbook of Statistical Methods, the US government's public statistics reference. The practice of using it to detect sample ratio mismatch is well established in the online experimentation field. OpenStax Introductory Statistics is a free, widely used textbook covering the chi-square test.

1. NIST/SEMATECH e-Handbook of Statistical Methods (chi-square goodness-of-fit test). https://www.itl.nist.gov/div898/handbook/
2. OpenStax, Introductory Statistics (the chi-square distribution and goodness of fit). https://openstax.org/details/books/introductory-statistics-2e