A/B test significance calculator
Which version wins, by how much, and how sure you can be.
- A
- B
Enter A and B
Questions
What does statistical significance mean in an A/B test?
It is the confidence that the gap between two versions comes from the versions themselves and not from chance. At 95%, a gap this size would show up by chance in about one test out of twenty where both versions perform the same.
What confidence level should I use?
95% is the usual standard for calling a winner, and the one this calculator uses, in both directions: a version clearly behind is called at the same bar. Going lower finds winners faster and finds more of them by chance.
Why does the calculator say it is too early?
Below 100 visitors per version, or 20 conversions across the test, a gap is mostly noise. The calculator waits rather than crowning a version on a handful of clicks.
Can I compare more than two versions?
Yes. Add up to three versions next to the original, and each one is compared with the original. Keep in mind that the more versions run at once, the more traffic each of them needs.
Should I stop my test as soon as it reaches 95%?
Decide on a sample size before you start and read the result once you reach it. Checking every day and stopping at the first 95% reading inflates false winners. The A/B test duration calculator tells you how many visitors you need.
Your website rewrites itself until it converts.
Start your 3-day free trial