A/B Test Significance Calculator

A two-proportion z-test, the standard test for comparing two conversion rates. With rate A = conversions A ÷ visitors A and rate B likewise:

• Pooled rate p = (conversions A + conversions B) ÷ (visitors A + visitors B).
• Standard error SE = √( p × (1 − p) × (1 ÷ visitors A + 1 ÷ visitors B) ).
• z = (rate B − rate A) ÷ SE.
• The two-sided p-value is the chance of a difference at least this large if the two versions were really the same. Below 0.05 is significant at 95% confidence.
• The range on the lift uses the unpooled standard error: difference ± 1.96 × SE, divided by rate A, at 95%.

A p-value alone can mislead on a small test: check a test often enough and something will look significant by chance. So the calculator also works out how many visitors each version needs to detect the lift you saw, at 95% confidence and 80% power, using the same formula Tilly’s A/B testing agent uses before it will call any winner. If either version has fewer visitors than that, the result is flagged, however good it looks.

Version A: 4,000 visitors, 120 orders (3.0%). Version B: 4,000 visitors, 136 orders (3.4%). B looks 13% better, but the p-value is about 0.31: a difference this size turns up by chance roughly one time in three. Not significant. To detect a 13% lift from a 3% base, each version needs about 30,000 visitors.

Version A: 20,000 visitors, 600 orders (3.0%). Version B: 20,000 visitors, 700 orders (3.5%). z is about 2.82 and the p-value about 0.005, so B wins at 95% and 99% confidence. A lift this size needs about 19,700 visitors a version, and each has 20,000, so the sample is big enough to trust.

• Stopping when it looks significant. Decide the sample size before you start and do not call it until you reach it.
• Testing tiny changes on little traffic. Small lifts need large samples; a store with a few hundred visitors a week should test bold changes.
• Changing the test halfway. If you edit a version, start the count again.
• Measuring several things and reporting whichever won. Choose the one metric that decides the test before it starts.
• Splitting unevenly by time. Run both versions over the same days, so a weekend or a sale does not favour one.

It compares two versions on one yes-or-no outcome, such as ordered or did not. It does not handle more than two versions at once, revenue per visitor, or tests you checked many times along the way. The A/B testing agent sizes each test before it starts, runs it and refuses to call a winner early. To test subject lines first, score them with the subject line tester.