A guardrail metric is a constraint on an A/B test — a metric you're not trying to improve, but that you're committed to not harming. The test can "win" on its primary metric only if guardrail metrics remain within acceptable bounds.
Without guardrails, optimizing a single metric can cause damage elsewhere that goes unnoticed. Guardrails formalize the idea that winning one way while losing another doesn't count as a win.
Why Guardrails Matter
A classic failure mode: a checkout test that increases average order value by 12% but silently drops conversion rate by 8%. On revenue per visitor, the result is nearly flat or negative — but if conversion rate wasn't a guardrail, the test would be called a winner based on AOV alone.
Guardrails prevent:
- Optimizing one funnel stage at the expense of another
- Improving revenue while degrading user experience
- Increasing short-term conversions at the cost of long-term retention
Common Guardrail Metrics
| Test type | Primary metric | Guardrail metrics |
|---|---|---|
| Checkout optimization | AOV | Conversion rate, cart abandonment rate |
| Homepage redesign | Click-through rate | Bounce rate, session depth |
| Pricing page test | Sign-up rate | Plan mix (free vs. paid), churn rate |
| Onboarding flow | Completion rate | Day-7 retention, support ticket volume |
| Speed optimization | Page load time | Conversion rate, error rate |
How to Set Guardrail Thresholds
Guardrails are typically set as "no statistically significant degradation" or as a maximum acceptable relative decline (e.g., "conversion rate must not drop more than 5%").
A pragmatic approach:
- Identify the 2–3 metrics most likely to be harmed by the change
- Set a threshold based on business impact (e.g., "a 3% drop in conversion rate at current volume costs $X/week")
- Pre-register guardrails before the test launches so there's no retroactive justification
Guardrails vs. Secondary Metrics
Secondary metrics are tracked for learning. Guardrails are tracked for go/no-go decisions. Every guardrail is a secondary metric, but not every secondary metric is a guardrail.