The test
Error rates
A relative effect scales the baseline; an absolute one adds percentage points to it.
How many visitors does an A/B test need before it can tell a real change in conversion rate from noise? It depends on where you start, how small a change you care about, and how often you are willing to be wrong in each direction.
Run 6,347 visitors through each variant, 12,694 in all. That is enough to detect the conversion rate rising to 22% or falling to 18%.
Visitors needed per variant as the smallest effect worth detecting runs from 5% to 20%, with power and significance held where you set them. Detecting 5% takes 25,255 per variant; 20% takes 1,602. The dot is your setting.
A two-sided test of two proportions under the normal approximation. The page sizes the test for a move up and a move down by the same effect and reports the larger of the two, since near 0% or 100% the variance is not symmetric.