What a test actually is
You split incoming users randomly into two groups, show each a different version, and compare a metric you chose before starting. Random assignment is what makes it a test rather than a comparison — it's the only thing that rules out the two groups differing for some other reason.
- One variable at a time, or you won't know which change caused the difference
- The metric declared before you start, in writing, so you can't select the one that happened to move
- The end date declared before you start, for the same reason
- Both versions running simultaneously — comparing this week to last week is not a test, it's a comparison contaminated by time
Do you have enough traffic?
This is the question that decides whether testing is available to you, and it's usually skipped. Detecting small differences requires a lot of observations, and the requirement rises sharply as the effect gets smaller.
| Improvement to detect | Conversions needed | Visitors at 5% |
|---|---|---|
| 5% → 10% (huge) | ~150 | ~3,000 |
| 5% → 7.5% (large) | ~500 | ~10,000 |
| 5% → 6% (realistic) | ~3,000 | ~60,000 |
| 5% → 5.5% (typical) | ~12,000 | ~240,000 |
Read the last two rows carefully. Realistic improvements need traffic most early products don't have. A site with two thousand visitors a month cannot detect a one-point conversion change in any reasonable timeframe, and running the test anyway produces a confident number that is noise.
Peeking, and why it invalidates results
Checking results daily and stopping when one variant looks ahead is the most common way tests produce wrong answers. Early in a test the numbers swing wildly, and if you stop at any moment one side is winning, you will find a winner in tests where no real difference exists.
- Calculate the required sample before startingFree calculators do this from your baseline rate and the effect you want to detect.
- Write the end condition down"Runs until 3,000 conversions per variant, or four weeks, whichever is later."
- Run whole weeksBehaviour differs by day. A test ending on a Tuesday over-weights whatever Tuesdays are like for you.
- Don't act on partial resultsLook if you must; don't stop. Deciding early is the error, not observing.
- Accept "no difference" as a resultIt's common, it's informative, and it saves you shipping complexity for nothing.
What to do instead at low volume
| Method | Answers |
|---|---|
| Watching five people use it | Where it's confusing, and why |
| Five-second tests | Whether the page communicates what it's for |
| Support tickets and sales objections | The real blockers, stated by real buyers |
| Shipping a large, confident change | Big effects are visible without statistics |
| Painted-door test | Whether anyone wants a feature, before building it |
Most early-stage improvement comes from these, not from testing. A/B testing is a tool for refining something that already works at volume — using it to find product-market fit is using a micrometer to find a wall.
Common mistakes
- Stopping the moment one variant looks ahead
- Changing several things at once and not knowing which mattered
- Choosing the metric after seeing which one moved
- Running a test at traffic levels that cannot detect a realistic effect
- Testing button colours while the page doesn't say what the product does
Key takeaways
- Random simultaneous assignment, one variable, metric and end date declared in advance
- Realistic improvements need far more traffic than most early products have
- Peeking and stopping early manufactures winners that don't exist
- At low volume, qualitative methods and large changes beat testing
Try it yourself
Put your current conversion rate and monthly traffic into a free sample-size calculator, asking for a one-point improvement. The number it returns tells you whether A/B testing is a tool you have access to yet.