Sample ratio mismatch (SRM) occurs when the observed number of visitors in each variant of an experiment differs significantly from the ratio you configured. If you set up a 50/50 A/B test but end up with 52,000 visitors in control and 48,000 in the variant, that gap is larger than random chance would produce — and it means something is wrong with the experiment itself.
SRM is one of the most important guardrail checks in experimentation because a mismatch usually points to a bug in assignment, tracking, or logging. When the split is broken, the two groups are no longer comparable, and any "winner" the test reports may be an artifact rather than a real effect.
Why It Matters
The validity of an A/B test rests on randomization: the only systematic difference between groups should be the variant itself. SRM is evidence that randomization or measurement has been compromised, so the groups may differ in hidden ways (different devices, different traffic sources, different bot exposure). A test with SRM should generally be discarded, not interpreted — even a result that looks statistically significant can't be trusted.
How to Detect It
SRM is detected with a chi-squared goodness-of-fit test comparing observed counts to expected counts:
- Take the expected ratio (e.g., 50/50) and the observed counts per variant.
- Run a chi-squared test for the difference.
- If the resulting p-value is very small (a common threshold is p < 0.0005, deliberately stricter than the usual 0.05), you have SRM.
The threshold is strict on purpose: SRM is a binary "is the plumbing broken?" check, and you want to flag only genuine problems, not normal sampling noise.
Common Causes
- Redirect or flicker issues — a redirect-based variant loses users who bounce before the redirect completes.
- Bot or crawler traffic hitting one variant more than another.
- Broken assignment logic — bucketing that isn't actually uniform, or a caching layer that skews allocation.
- Tracking gaps — events firing reliably in one variant but not the other.
- Filtering applied after assignment — excluding users post-hoc in a way that correlates with the variant.
How to Respond
- Don't ship the result. Treat an SRM as invalidating until proven otherwise.
- Investigate the pipeline — assignment, redirects, bot filtering, and logging are the usual suspects.
- Fix and rerun rather than trying to statistically "correct" the imbalance, which rarely works.
Automated experimentation platforms typically run SRM checks continuously as a built-in guardrail, alerting you the moment a live test's split drifts out of tolerance.