A holdout group is a subset of users who are intentionally kept on the original experience while everyone else receives a new feature, experiment, or optimization. By comparing the holdout group's behavior to the rest of the user base, you can measure the true long-term impact of your changes.
Holdout Group vs. Control Group
The terms are closely related but slightly different:
- Control group — The baseline in a single A/B test. Sees the original version during that specific test.
- Holdout group — A persistent group excluded from all changes (or a set of changes) over an extended period. Used to measure cumulative impact.
A control group answers: "Did this one change work?" A holdout group answers: "Did all our optimizations over the past quarter actually make things better?"
Why Holdout Groups Matter
Individual A/B tests can each show a small win, but those wins don't always compound the way you expect. Holdout groups let you verify that your overall optimization program is delivering real value.
For example, you might run 10 tests over a quarter, each showing a 3–5% improvement. But the holdout group comparison might reveal the cumulative impact is only 8% — not the 30–50% you'd calculate by adding individual test results. This happens because test effects can overlap, cancel out, or interact in unexpected ways.
How to Set Up a Holdout Group
- Choose a holdout size — Typically 5–10% of traffic. Large enough for statistical power, small enough to minimize lost optimization.
- Assign persistently — Users should stay in the holdout for the entire measurement period, not rotate in and out.
- Exclude from all experiments — The holdout group sees only the baseline experience.
- Measure periodically — Compare key metrics (conversion rate, revenue per visitor, engagement) between the holdout and the optimized group.
When to Release the Holdout
If the optimized group consistently outperforms the holdout by a meaningful margin, you can release the holdout and let all users benefit from the improvements. Most teams run holdouts for a quarter or a specific campaign cycle.