Fixed-horizon: Calculate sample size upfront, run until you reach it, analyze once. Sequential testing: Check results continuously using special statistical methods (SPRT, Bayesian, or always-valid p-values). Fixed-horizon is simpler and standard. Sequential is complex but allows early stopping. Most teams should use fixed-horizon.
Fixed-Horizon Testing (Standard)
The traditional approach:
- Calculate sample size before starting (e.g., need 10,000 visitors per variant)
- Run the test until you reach that sample size
- Analyze results once at the end
- Make decision based on p-value
⚠️ The Rule:
Don't peek at results and stop early when you see significance. This inflates false positive rate from 5% to 20-30%.
Sequential Testing
Methods that allow continuous monitoring:
Sequential Probability Ratio Test (SPRT)
Check after every observation. Uses likelihood ratios to maintain valid error rates.
Always-Valid P-Values
Special p-values that remain valid no matter when you check (e.g., mixture sequential probability ratio test).
Bayesian Methods
Use posterior probabilities. Can check anytime, but interpretation differs from frequentist.
Comparison
| Aspect | Fixed-Horizon | Sequential |
|---|---|---|
| Can peek at results? | No | Yes |
| Sample size | Predetermined | Adaptive |
| False positive rate | 5% (if followed) | 5% (if implemented correctly) |
| Average test duration | Longer | Shorter (if winner exists) |
| Complexity | Simple | Complex |
When to Use Each
Use Fixed-Horizon If:
- • You're new to A/B testing
- • You want simplicity
- • You can wait for full sample
- • You don't have statistical expertise
Use Sequential If:
- • You have statistical expertise
- • You need to stop tests early
- • You're using Bayesian framework
- • Your tool implements it correctly
The Most Common Mistake
Using fixed-horizon methods but peeking anyway.
This is the worst of both worlds: you inflate false positives without the benefits of proper sequential testing. Either commit to fixed-horizon (no peeking) or implement proper sequential methods.
Our Recommendation
For most teams: use fixed-horizon testing. It's simpler, well-understood, and prevents false positives if you follow the rules.