Every conversion optimization guide says "A/B test everything." Change a button color, test it. Rewrite a headline, test it. Move the CTA above the fold, test it.
The problem: most of this advice assumes you have 50K+ monthly visitors. If your site gets 2,000 visits/month — which is realistic for a service business, a new SaaS, or a developer portfolio — A/B testing is statistically useless for almost every change you'd want to test.
Here's the math, and what to do instead.
The sample size problem
To detect a conversion rate change from 3% to 4% (a 33% relative improvement — very optimistic), you need approximately 4,800 visitors per variation. That's 9,600 total. At 2,000 visits/month, that's nearly 5 months of testing.
For a more realistic scenario — detecting a 3% to 3.5% change (a 17% improvement) — you need roughly 18,000 visitors per variation. That's 36,000 total, or 18 months.
During those 18 months, everything changes: your audience shifts, your messaging evolves, seasonal patterns skew results, and Google changes your search rankings. The "controlled" experiment isn't controlled at all.
The rule: if you can't reach statistical significance in 2–4 weeks, the test isn't giving you real data. It's giving you noise that looks like data.
When A/B testing actually works on small sites
There are narrow scenarios where A/B testing is valid even at low traffic:
Large effect sizes
If you're testing a change that you expect to double your conversion rate (3% → 6%), you need ~800 visitors per variation — achievable in a month at 2,000 visits. Examples:
- Adding vs. not adding a pricing page
- Completely different landing page structures (not just headline tweaks)
- Removing an entire step from a sign-up flow
High-traffic pages with binary outcomes
If one specific page gets disproportionate traffic (e.g., 80% of visits land on your homepage) and the outcome is binary (clicked CTA vs. didn't), you can test on that page alone with a smaller sample.
Email campaigns
If you have an email list of 5,000+, A/B test subject lines. Open rates are measurable at lower sample sizes because the baseline is higher (~20–40% open rate vs. 2–5% conversion rate on a website).
What to do instead
For sites under 10K monthly visits, these approaches give you more signal than A/B testing:
1. Session recordings
Watch 20 real user sessions per week. Tools like Hotjar, PostHog, or Microsoft Clarity show you where users hesitate, scroll past, or leave. You'll learn more from watching 20 sessions than from a 3-month A/B test.
What to watch for:
- Where do users stop scrolling?
- Do they read your pricing, or skip it?
- Do they hover over the CTA without clicking?
- Do they visit the contact page but not submit the form?
2. Direct feedback
Ask your last 5 clients: "What almost stopped you from reaching out?" The answers will be more actionable than any statistical test. Common responses:
- "I couldn't find your pricing"
- "I wasn't sure if you do [specific thing]"
- "I almost left because the site felt too corporate/too casual"
3. Before/after with baseline metrics
Instead of running A/B tests, make changes sequentially and compare metrics before and after:
- Record your baseline: conversion rate, scroll depth, time on page for 4 weeks
- Make one change
- Measure the same metrics for 4 weeks
- Compare
This isn't as rigorous as an A/B test — confounding variables exist. But for a 2,000-visit/month site, a 4-week before/after comparison is more reliable than an underpowered A/B test that runs for 3 months.
4. Qualitative testing
Show your landing page to 5 people who match your target audience. Ask them:
- "What does this company do?" (after 5 seconds)
- "Who is this for?"
- "What would you do next?"
- "What's confusing or missing?"
Five qualitative interviews surface more actionable insights than statistical testing at low traffic.
The cargo cult signs
You're cargo-culting A/B testing if:
- Tests run for months without reaching significance. The tool shows "Variant B: +12%" but the confidence interval includes both +40% and -15%. That's not a result — it's noise.
- You declare a winner at 80% confidence. The standard for A/B testing is 95% confidence. Below that, you're flipping a coin and calling it strategy.
- You test tiny changes. Button color, font size, "Sign Up" vs. "Get Started." These changes affect conversion by fractions of a percent — undetectable at low traffic.
- You test more than one change at a time. If Variant B has a new headline AND a new CTA AND a new layout, you don't know which change caused the result (if any).
- You stop the test when it "looks good." Peeking at results and stopping early inflates your false-positive rate dramatically.
The honest answer
If your site gets fewer than 10,000 visits/month: don't A/B test. Instead:
- Watch session recordings weekly
- Ask customers what almost stopped them
- Make one change at a time, measure before/after
- Test big structural changes, not cosmetic tweaks
- Use qualitative methods (user interviews, 5-second tests)
Save A/B testing for when you have the traffic to support it. Until then, ship the landing page patterns that are already proven and focus on getting more traffic to test with.
Want a conversion-focused site without the statistical theater? Let's talk — I help founders build sites that convert based on proven patterns, not underpowered experiments.