Is the gap real?

These are the even-split rows from the experiments lab.

Reading the experiment table…

What the numbers mean

• The p-value assumes the two rates are equal. It is the chance of a gap at least this large under that assumption. It is not the chance that B is better.
• The interval is B’s rate minus A’s rate, plus and minus 1.96 times the unpooled standard error. About 95 of 100 repeats of this same test would produce an interval that covers the true gap.
• The probability uses a Beta(1 + conversions, 1 + impressions − conversions) draw for each arm, 4,000 times. It is the share of draws where B’s rate is higher. The 1s are a flat prior.
• Visitors per arm for 80% power at a 5% two-sided test is (1.96 + 0.84)² × (p₁(1−p₁) + p₂(1−p₂)) / (p₂ − p₁)². p₁ is arm A’s rate, or 4% when that rate is still zero. p₂ is p₁ plus the lift you type.

About 1861 impressions per arm.