Intro to A/B Testing
Showing two versions to two groups and measuring which does better — plus the sample-size arithmetic that decides whether the result means anything.
Intermediate8 min readDaksh BathlaDEFINITIONGrowth
A/B Test
Randomly splitting incoming users between two versions and comparing a metric chosen before the test started. The random simultaneous assignment is what makes it a test rather than a comparison.
EXAMPLE
Half of visitors see headline A, half see headline B, and you compare signup rate. Comparing this week against last week is not an A/B test — the difference could be the week.
WHY IT MATTERS
It's how you tell a real improvement from noise. It also needs far more traffic than most early products have, so knowing the sample-size requirement tells you whether the tool is available to you at all.