Holdout Group

A holdout group is a randomly chosen portion of customers you intentionally do not send a campaign, ad or flow to. Everyone else gets the marketing as usual. Because the two groups were alike before you split them, any later gap in revenue or orders between them is attributable to the marketing itself rather than to the season, the traffic or the brand.

It is not a suppression list or an unengaged segment you skip on purpose: those people differ from your senders before the test starts, so the comparison is worthless. It is also not an A/B Test, which compares two versions of a message. A holdout compares receiving a message with receiving nothing at all.

Split the audience randomly, pick a window, then compare revenue per person in each group. Subtract the holdout's figure from the treated group's, then multiply by the number of people treated to get incremental revenue. The honest denominator is everyone assigned to each group at the split, including people who never opened. The flattering one is openers or clickers only, which quietly drops the least responsive people from one side and inflates the gap you measure.

Incrementality is the thing a holdout group measures; the holdout is the method. An A/B Test compares two treatments rather than treatment against nothing. Statistical Significance tells you whether the gap between the two groups is larger than random noise, and Sample Size decides whether your holdout is big enough to detect a gap at all.