A Shopify discount test can show a higher conversion rate without answering whether you should keep the offer. First write down the control and treatment offers, then verify the difference in the shopping journey. Read the result against the commercial outcome you intended to improve; more orders can still leave less contribution.
Shopify’s October 2 Rollouts update lets merchants schedule and test discount changes alongside other store changes. Shopify lists Experiments as available on Grow and higher plans. Its requirements also limit Experiments to the Online Store.
For a team buying traffic into a promotion, the first decision is concrete: is this a usable comparison of the offers you intend to choose between?
Write down the two offers before reading the result
In a discount Experiment, the control keeps the store’s usual discount behavior. You do not select a separate control discount. The treatment activates or deactivates the selected discounts, according to Shopify’s description of rollout types.
A discount also retains its own active dates. Activating it through a rollout does not erase that independent schedule. Deactivating it through a rollout makes it inactive for the visitors in that rollout, even during its own active dates.
That creates a useful preflight question: if the selected discount is already active for the control, what difference does activating it create for the treatment? An experiment label alone does not establish a different offer.
Put these paired descriptions beside the test:
Control offer: the discounts normally available during the comparison period, for the intended customer and basket.
Treatment offer: the same starting position, plus the exact activation or deactivation being tested.
Conditions on both sides: the discount’s own dates, customer eligibility, product and basket requirements, and combination settings.
Other changes: any theme, checkout or catalog change bundled into the same comparison.
Observed difference: whether an authorized check of the intended shopping journey shows the contrast you wrote down. Preserve an unknown result where it cannot be checked.
Customer eligibility and combination rules still matter. If those rules change the final basket offer, the headline discount percentage is an incomplete description of the treatment.
A coordinated launch can deliberately test a whole package. If the decision is about the discount alone, changing the theme or checkout at the same time leaves another explanation for the result. Decide which question the test is meant to answer before interpreting it.
Separate assignment, discount use and the outcome
Shopify’s analytics documentation sets several limits on the readout. Available metrics depend on the rollout and its changes; merchants cannot add custom metrics to that report. Discount-specific revenue and order metrics are not available there.
For discount changes, Shopify says exposures follow assignments and warns about eligibility and shared codes. Treat those counts as a description of the experiment’s reporting population. They do not, on their own, establish which paid orders redeemed the intended offer.
The same documentation limits the measured surface to the Online Store, cart and checkout. It also describes different calculation paths for Launch and Experiment results. Comparing their displayed numbers is not automatically a like-for-like experiment.
Before asking for profit by group, establish whether your authorized records can connect assignment, the actual offer and paid orders consistently. Do not assume that a standard export supplies that connection. If it is missing, keep the group-level commercial result unresolved. Storewide revenue cannot reconstruct it.
Give conversion rate and contribution different jobs
Consider this synthetic example, with 1,000 comparable eligible visits in each group, one order per purchasing visit and no unresolved returns. It is arithmetic, not a forecast or a measured experiment.
Control: 40 paid orders at $60 net revenue per order, with $30 of relevant variable costs per order. Revenue is $2,400 and contribution before media is $1,200.
Treatment: 50 paid orders at $51 net revenue per order after the discount, with the same $30 of costs. Revenue is $2,550 and contribution before media is $1,050.
Conversion rises from 4% to 5%, and revenue rises by $150, while contribution before media falls by $150. More paid orders can be useful without making this particular offer more profitable.
Use your own cost definition, returns maturity and customer objective. If acquisition is the goal, distinguish new customers from repeat orders. Bring actual media cost into the comparison separately; an unchanged budget setting does not prove equal realized spend.
If the totals disagree, align the purchase definitions and reporting dates before reading a group difference. Agree the outcome and read schedule before launch so a late result does not silently change the question.
The next check is not always another discount setting. A different customer mix, stock availability, return rate or concurrent creative change may explain a commercial difference. Keep those explanations visible alongside uncertainty in the estimate.
Let the comparison choose the next action
If the offers do not differ as intended, repair the comparison before using it to choose an offer. If the contrast is valid but the commercial records cannot be connected, name that missing evidence before expanding the promotion.
If both checks hold, use the agreed outcome and read schedule to assess the result. A valid comparison can still show no worthwhile difference. It can also support keeping a discount when its commercial outcome justifies the cost.
The useful result is a decision about the tested offer, population and period. It is not proof that the same discount will improve every campaign, or that the advertising itself generated incremental sales.
