How to Calculate the Right Sample Size for Your A/B Test
Running an A/B test with too few visitors leads to false positives. Learn how to calculate the minimum sample size you need before launching any experiment.
One of the most common mistakes teams make when running A/B tests is calling a winner too early. You see variant B at 8% conversion vs variant A at 6% and think you have a winner—but you only have 200 visitors. That’s not a winner. That’s noise.
Getting your sample size right before you start is the single highest-leverage thing you can do for test quality.
Why Sample Size Matters
Every A/B test is a statistical experiment. The output is a probability, not a fact. When you end a test early, you dramatically increase the chance of a false positive—declaring a winner that isn’t actually better.
Studies have shown that teams who call tests early are wrong more than 50% of the time, even when their data looks convincing.
The Three Inputs You Need
To calculate the right sample size, you need three numbers:
1. Baseline Conversion Rate
Your current conversion rate before any changes. If 3 out of every 100 visitors convert, your baseline is 3%.
2. Minimum Detectable Effect (MDE)
The smallest improvement you’d care about. If you’re currently at 3% and an improvement to 3.3% would be meaningful for your business, your MDE is 10% relative (or 0.3 percentage points absolute).
Setting MDE too low (e.g., 1% relative) requires enormous sample sizes. Setting it too high means you’ll miss real improvements. A good starting point for most teams: 10–20% relative improvement.
3. Statistical Power and Significance
- Statistical significance (α): How willing you are to accept a false positive. Standard: 95% (α = 0.05).
- Statistical power (1-β): How likely you are to detect a real effect. Standard: 80%.
The Formula
For a two-sided test with 95% significance and 80% power:
n = (2 × (Z_α/2 + Z_β)² × p × (1-p)) / (MDE)²
Where:
Z_α/2 = 1.96(for 95% significance, two-sided)Z_β = 0.84(for 80% power)p= baseline conversion rateMDE= absolute minimum detectable effect
Example:
- Baseline: 5% (p = 0.05)
- MDE: detect a 1 percentage point increase (to 6%)
- n = (2 × (1.96 + 0.84)² × 0.05 × 0.95) / (0.01)²
- n ≈ 7,415 visitors per variant
So you need about 14,830 total visitors before you can call this test.
Quick Reference Table
| Baseline Rate | MDE (absolute) | Visitors Per Variant |
|---|---|---|
| 2% | 0.5% | ~6,000 |
| 5% | 1% | ~7,400 |
| 5% | 0.5% | ~29,000 |
| 10% | 1% | ~13,800 |
| 10% | 2% | ~3,500 |
Practical Rules
1. Use a calculator first. Before building any variant, plug your numbers into a sample size calculator. ClickVariant shows you estimated run time when you set up a test.
2. Set your MDE based on business impact. Don’t pick MDE based on what sounds achievable. Ask: “What’s the minimum improvement that would change a business decision?”
3. Don’t peek. Checking results daily and stopping when you see significance inflates your false positive rate. Commit to running until your sample size target is hit.
4. Account for traffic splits. If you’re testing on a segment (e.g., mobile only, new visitors only), use only that segment’s traffic in your calculation—not total site traffic.
5. One primary metric. Testing multiple metrics multiplies your false positive risk. Pick one conversion goal before the test starts.
Novelty Effect Adjustment
New variants often get a short-term lift just because they’re new. For tests involving significant visual changes, run for at least 2 weeks even if you hit your sample size in a few days. This smooths out novelty effects and captures weekly traffic patterns (weekday vs. weekend behavior can differ significantly).
When Traffic Is Low
If your site gets fewer than 1,000 visitors per week, you may need to:
- Increase MDE: Only test for changes you expect to have large effects (20%+ relative improvement).
- Reduce segmentation: Run on all traffic, not subsets.
- Use longer run times: Accept that tests will take 4–8 weeks.
- Prioritize: Run one test at a time on your highest-traffic pages.
Low-traffic sites aren’t a reason to skip testing—they’re a reason to be more selective about what you test.
Setting Up Sample Size Targets in ClickVariant
When creating a new experiment in ClickVariant, you can set a minimum visitor threshold before any results are shown. This prevents your team from making premature decisions.
Go to Experiment Settings → Statistical Settings and enter your required sample size. ClickVariant will mark the test as “Collecting data” until that threshold is reached, keeping results hidden to avoid peeking bias.
Getting sample size right is foundational. Every other optimization practice—from test prioritization to result analysis—breaks down if you’re calling winners on underpowered tests. Calculate before you build.
Style: Flat design illustration, dark navy background (#0a0a2e), purple and coral accents, minimalist, professional SaaS blog header Size: 1360x768 (16:9 landscape) Prompt: Two tall bar charts side by side labeled A and B, the B bar taller and glowing in coral, a magnifying glass examining the data with floating percentage numbers and a semicircular confidence meter gauge, abstract data points scattered in background. Dark navy background, no text. Filename: ab-testing-sample-size-calculator.jpg