Did B beat A?

Size an A/B test before you run it β€” how many users each arm needs to spot the lift you care about β€” then, once the numbers are in, check whether the result is a real difference or just noise.

Experiment
Detect a%change on a% baselineThe conversion rate you start from β€” the control's current performance. Every effect is measured as a change against this..
Run at% powerThe chance the test detects a real effect when one genuinely exists. At 80% power, a true lift still goes unnoticed one run in five.,% confidenceHow sure you want to be that a win isn't a fluke. 95% confidence means a 5% chance of crowning a winner that's really just noise β€” a false positive.,.
Spotting a shift from 5% to 5.5% Β· Ξ± = 5% (2-sided) β€” 31,233 users per arm.
Sample needed
We getvisitors per, split evenly.
31,233Per arm
62,467Both arms
4.5 weeksRun time
Relative liftPer arm
+2%
752,702
24.7 months
+5%
122,123
4 months
+10%chosen
31,233
4.5 weeks
+15%
14,193
15 days
+20%
8,158
9 days
+30%
3,780
4 days

Traffic fixed instead of the effect? See the smallest lift this traffic can detect β†’

two-proportion z-test Β· see how not to run an A/B test