Did B beat A?

Size an A/B test before you run it β€” how many users each arm needs to spot the lift you care about β€” then, once the numbers are in, check whether the result is a real difference or just noise.

Experiment
Detect a%change on a% baselineThe conversion rate you start from β€” the control's current performance. Every effect is measured as a change against this..
Run at% powerThe chance the test detects a real effect when one genuinely exists. At 80% power, a true lift still goes unnoticed one run in five.,% confidenceHow sure you want to be that a win isn't a fluke. 95% confidence means a 5% chance of crowning a winner that's really just noise β€” a false positive.,.
Spotting a shift from 50% to 55% Β· Ξ± = 5% (2-sided) β€” 1,565 users per arm.
Sample needed
We getvisitors per, split evenly.
1,565Per arm
3,129Both arms
2 daysRun time
Relative liftPer arm
+2%
39,239
5.6 weeks
+5%
6,274
7 days
+10%chosen
1,565
2 days
+15%
693
< 1 day
+20%
387
< 1 day
+30%
169
< 1 day

Traffic fixed instead of the effect? See the smallest lift this traffic can detect β†’

two-proportion z-test Β· see how not to run an A/B test