Hypothesis

A hypothesis is the sentence you write before you run a test. It names the change you intend to make, the metric you expect it to move, the direction of that movement and the reason you believe it will happen. A usable form is: because we observed X, we believe changing Y will cause Z, measured by this metric.

A hypothesis is not the test itself, and it is not an idea. An idea is "try a shorter product page". A hypothesis commits to an outcome you can be wrong about. People often confuse it with an A/B Test, which is the mechanism that checks the hypothesis, and with a Primary Metric, which is only the measuring part of it.

A hypothesis is not calculated, but the prediction inside it is a number and that number decides your test design. State the expected lift as a relative change to the Primary Metric, then use it as your Minimum Detectable Effect to work out Sample Size. The honest version predicts the smallest lift worth shipping, which needs more traffic. The flattering version predicts a large lift, which lets you stop early and call almost any wobble a win. Write the number down before the test starts, not after.

A hypothesis is checked by an A/B Test and judged against a Primary Metric. Minimum Detectable Effect is the smallest change your test can actually see, so it sets the ceiling on how modest a hypothesis can be. Statistical Significance tells you whether the result supports the hypothesis or is noise, and Sample Size is the traffic that decision requires.