How to run an A/B test on Shopify
Run a clean A/B test on your Shopify store: write the hypothesis, size the test, build it as a rollout experiment, leave it alone, then apply or roll back.
What this is for
An A/B test shows one group of visitors your current store and another group a changed version at the same time, then compares what each group did. Because both groups shop in the same weeks, a sale, a holiday or a new ad campaign affects them equally, which a before-and-after comparison cannot promise.
Shopify now has this built in. Rollouts in your admin include an experiment type made for exactly this job. source, checked 28 September 2026
Time needed and what you need
Time needed: about 2 hours to set up, then 2 to 4 weeks running. Writing the hypothesis and building the change is an afternoon; the test itself runs for as many full weeks as your traffic needs.
- A Shopify store on the Grow plan or higher, for rollout experiments
- Your current conversion or add to cart rate from Shopify's reports, as the baseline
- Your average daily sessions, to work out how long the test must run
- One specific change you want to test, and a reason you expect it to help
The steps
Decide everything in steps 1 and 2 before you build anything. A test designed after you have seen early numbers is not a test.
1. Write the hypothesis down
Write one sentence: what you will change, what you expect it to do, and why. For example, showing delivery time next to the button will raise the add to cart rate, because shoppers keep asking when orders arrive. A clear hypothesis forces you to name one change and one metric. If you are unsure which metric, read how to pick a primary metric for a test first.
2. Work out how long the test must run
Decide the smallest lift you would act on. Then open the A/B test significance calculator and enter a trial pair of results: your baseline rate for version A and the rate you would act on for version B, on 10,000 visitors each. Under the result it tells you how many visitors each version needs to detect a lift that size. Divide that by the visitors each version will get per day, then round up to whole weeks so every weekday is counted equally.
If the answer is months, test a bigger change or a busier page. A small store cannot measure a small effect, and that is a traffic limit, not a tool problem.
3. Create an experiment rollout
From your Shopify admin, go to Markets, then Rollouts, and create a new rollout. Give it a name that says what is being tested and pick Experiment as the type. Experiments need the Grow plan or above. source, checked 28 September 2026
4. Add the change you are testing
Add the theme change: either edit your main theme or replace it with another theme you have already added. To edit the main theme, save the rollout first, then open the theme editor from it and make only the change in your hypothesis. source, checked 28 September 2026
Only what you add is compared. The control shows your store as it is, and the treatment shows it with your change. source, checked 28 September 2026
5. Set the traffic split and the dates
In the Audience section, check the traffic split and the end date. Shopify starts every experiment at 50% control and 50% treatment and ends it 90 days later. Keep the even split, which reaches the sample you need fastest, and change the end date to the length you worked out in step 2. source, checked 28 September 2026
6. Check both versions before it starts
Preview the treatment on a phone and on a desktop. Click through the product page, pick variants, add to cart and go as far as checkout. A broken treatment does not tell you whether the idea works, only that the build was broken. Then confirm the control still looks exactly like your live store.
7. Keep search engines out of trouble
A rollout keeps the same URLs, so there is little to do. If you use a testing app that sends visitors to a separate variant URL instead, follow Google's rules: point the variant's canonical at the original, redirect with a 302, and never show Google a different version from the one people see. source, checked 28 September 2026
8. Leave it alone while it runs
Do not edit either version once the test is live, do not start a sale that only one version shows, and do not stop early because one side is ahead after three days. Shopify warns about the first of those directly. source, checked 28 September 2026
9. Read the result and end the rollout
At the planned end date, open the rollout's analytics, take the visitors and conversions for each version, and check them in the calculator. The guide to knowing when an A/B test is done covers how to read the result. Then end the rollout either way: apply the change if it won, roll it back if it did not. source, checked 28 September 2026
A worked example
For a skincare brand's store with 3,000 sessions a day
Worked example (an invented store, with invented numbers)
Hypothesis: moving the review stars and a free returns line above the add to cart button will raise the added to cart rate. The baseline is 8.0%, and the owner decides a relative lift under 10% (8.0% to 8.8%) is not worth acting on.
The calculator says each version needs 18,872 sessions to detect that lift. With a 50/50 split, each version gets 1,500 of the 3,000 daily sessions, so 18,872 ÷ 1,500 = 12.6 days. Rounded up to two full weeks, the test runs 14 days and each version gets 21,000 sessions.
At the end, the control has 1,680 sessions with a cart addition (8.0%) and the treatment has 1,869 (8.9%). That is a relative lift of 0.9 ÷ 8.0 = 11.25%, and the calculator gives p = 0.0009, well inside the 95% confidence line. The owner applies the change permanently.
Common mistakes
Comparing this month with last month and calling it a test. Everything else that changed between the two months, from the weather to your ad spend, is mixed into the result. Split the traffic at the same time.
Testing three changes in one treatment. If it wins you will not know which change did the work, and if it loses a good idea may have been dragged down by a bad one. Test one idea per experiment.
Using a discount code in an experiment and assuming it stays in the treatment group. Codes get shared, so the control can end up using it too. source, checked 28 September 2026
Leaving a finished test running. It keeps half your visitors on the losing version, and Google asks sites to remove test elements once the test is over. source, checked 28 September 2026
Doing this with Tilly
Tilly's experiments agent builds the change for an A/B test for 10 credits. The change is published to a preview copy of your theme, and nothing goes live until you approve it. Splitting the traffic and reading the result happen in your testing tool, such as a Shopify rollout experiment, so the steps above still apply.
Questions people ask
Can I A/B test on a Shopify plan below Grow?
Not with rollout experiments, which need the Grow plan or higher. On a lower plan you would use an A/B testing app from the Shopify App Store, and the same steps apply: one change, a planned length, and no peeking. source, checked 28 September 2026
Will an A/B test hurt my Google rankings?
Not if you follow Google's testing rules. Google says small page changes, like a button's placement or wording, rarely affect how a page appears in search at all. source, checked 28 September 2026
Which metrics does a Shopify rollout experiment report?
For a theme change, rollout analytics cover conversion rate, bounce rate, reached checkout rate and added to cart rate. You cannot add your own metrics, so pick your primary metric from that list before you start. source, checked 28 September 2026
See this on your own store
Paste your store URL. The first pass takes about a minute and needs no account. Save a card (nothing is charged) and Tilly reads the whole store and works out who buys from you.