A/B testing works in 4 steps: (1) Randomly assign users to variants using consistent hashing, (2) Show different experiences based on assignment, (3) Track conversions per variant, (4) Use statistical tests to determine if differences are significant. The key is consistent assignment—same user always sees same variant.
The A/B Testing Process
User Assignment (Bucketing)
When a user visits, hash their ID to consistently assign them to a variant. Same user always gets same variant.
Variant Delivery
Show the assigned variant—either server-side (before HTML) or client-side (via JavaScript).
Event Tracking
Track conversions (clicks, purchases, signups) and associate them with the user's variant.
Statistical Analysis
Compare conversion rates using statistical tests to determine if differences are significant or due to chance.
User Assignment (Bucketing)
The key challenge: assign users randomly but consistently. If a user refreshes, they should see the same variant.
// Simple bucketing algorithm
function assignVariant(userId, experimentId) {
// Create consistent hash from user + experiment
const hash = murmurhash(userId + experimentId);
// Convert to percentage (0-100)
const bucket = hash % 100;
// 50/50 split
return bucket < 50 ? 'control' : 'variant';
}This ensures the same user always gets the same variant, even across sessions.
Statistical Significance
We use hypothesis testing to determine if results are real or due to chance:
- • Null hypothesis: No difference between variants
- • p-value: Probability of seeing this result if null hypothesis is true
- • Significance level: Usually 95% (p < 0.05)
Common mistake: Checking results too early. Statistical tests assume you wait for the predetermined sample size.
Client-Side vs Server-Side
Client-Side
- + Easy setup (just add script)
- + Visual editor support
- - Potential flicker
- - JavaScript required
Server-Side
- + Zero flicker
- + Works without JS
- - Requires code changes
- - More complex setup
Get Started
Understanding the theory is great, but the best way to learn is by doing. ExperimentHQ handles all the complexity—bucketing, tracking, statistics—so you can focus on running tests.