A/B Test

An A/B test splits your traffic into two randomly assigned groups. One group sees the current version (the control, or A), the other sees a single changed version (the variant, or B). Both run at the same time, over the same days, and you compare one agreed metric between them. The random split is what makes the comparison fair.

It is not the same as changing your product page on Tuesday and comparing Wednesday's sales to Monday's — that is a before-and-after, and weather, payday and ad spend all move with it. It is also not a Multivariate Test, which varies several elements at once and needs far more traffic to read.

Pick one primary metric before you start, usually conversion rate: orders divided by sessions for each group separately. The honest denominator is every session assigned to that variant, including bounces. The flattering one counts only sessions that reached the variant or scrolled far enough, which quietly drops the people the change put off. Then compare the two rates and check whether the gap is larger than random variation would produce, given how many sessions each side received.

A Hypothesis is the stated guess a test is built to check, written before traffic is split. Sample Size is how many sessions each variant needs before the result means anything. Statistical Significance tells you how likely the gap is chance. A Holdout Group withholds a whole activity from some customers, rather than comparing two versions of one thing.