A/B test significance calculator
Which version wins, by how much, and how sure you can be.
- A
- B
Enter A and B
Questions
What does statistical significance mean in an A/B test?
It is the confidence that the gap between two versions comes from the versions themselves and not from chance. At 95%, a gap this size would show up by chance in about one test out of twenty where both versions perform the same.
What confidence level should I use?
95% is the usual standard for calling a winner, and it is the level Polish uses. Polish stops showing a version from 90% confidence when it converts at least 20% less than the original: dropping a weak version early costs less than crowning a false winner.
Why does the calculator say it is too early?
Below 200 visitors per version, or 10 conversions across the test, a gap is mostly noise. The calculator waits, exactly like Polish, and calls a winner once the numbers can carry it.
Can I compare more than two versions?
Yes. Add up to three versions next to the original. Each one is compared with the original, which is how Polish reads its own experiments.
Should I stop my test as soon as it reaches 95%?
Decide on a sample size before you start and read the result once you reach it. Checking every day and stopping at the first 95% reading inflates false winners. The A/B test duration calculator tells you how many visitors you need.
Your website rewrites itself until it converts.
Start your 3-day free trial