A/B Test Significance Calculator
Enter visitors and conversions for two variants to check whether the difference is statistically significant.
How A/B test significance is calculated
This calculator runs a two-proportion z-test, comparing variant B's conversion rate against variant A's. It pools both variants' data to estimate a standard error, then measures how many standard errors apart the two observed rates are (the z-score).
The p-value is the probability of seeing a difference this large (or larger) purely by chance if there were truly no difference between the variants. A p-value below 0.05 is the common threshold for calling a result "statistically significant" at the 95% confidence level.
Frequently asked questions
What does a p-value of 0.05 mean?
It means there's a 5% chance of observing a difference this large between the two variants purely due to random sampling, even if they actually perform identically. A lower p-value gives you more confidence the observed difference is real.
My test is significant — can I stop it now?
Checking significance repeatedly before reaching your pre-planned sample size inflates your false-positive rate — a common pitfall called "peeking." Decide your sample size in advance (see the Sample Size Calculator) and let the test run to completion before concluding.
Does statistical significance mean the difference is practically important?
Not necessarily — with a large enough sample, even a tiny, practically meaningless difference can become statistically significant. Always consider the size of the effect (the actual lift in conversion rate) alongside its statistical significance.