A two-week randomized experiment testing whether a streamlined 2-step guest checkout outperforms the existing 5-step account-required flow on purchase completion.
Over two quarters, nearly three out of four shoppers left before completing a purchase. The five-step checkout required account creation before payment — a flow the VP of Product believed was the primary friction point.
At Relay's volume, every percentage point of checkout conversion was worth $380K/year. The CFO required experimental evidence before approving a redesign that would eliminate mandatory email capture.
100,800 checkout sessions were randomly assigned to the existing 5-step flow (control) or the new 2-step guest checkout (treatment). Assignment was at the user level via feature flag.
Primary metric (checkout completion), secondary metric (average order value), and guardrail metrics (email capture, support tickets) were pre-registered before launch. Power analysis confirmed the sample size could detect a ≥10% relative lift at 80% power.
95% CI: 3.7 to 4.8 pp · p < 0.001. Even the pessimistic bound translates to $8.5M/year.
| Component | Value |
|---|---|
| Baseline completion rate | 27.9% |
| New completion rate | 32.2% |
| Average order value | $87 |
| Daily checkout sessions | 7,200 |
| Additional purchases/year | ~112,400 |
| Projected annual revenue impact | $9.8M |
Annual email loss: ~$160K in remarketing value. Revenue gain: $9.8M. Net: +$9.6M. A Q4 opt-in test later recovered 40% of lost emails.
Shipped the redesign — full rollout the following Monday, with daily monitoring for two weeks. Post-launch completion settled at 31.8%, within the confidence interval.
Mobile users saw the largest lift — 5.8 pp vs. 3.1 for desktop. With mobile traffic growing 18% YoY, the revenue impact will compound.
Q4 email recovery test — an inline opt-in prompt raised voluntary capture from 68% to 81%, narrowing the email gap from 22% to 9%.
90-day LTV analysis — repeat purchase rates were statistically indistinguishable between groups (p = 0.14). 44% of guest-checkout users created accounts within 90 days organically.
The experiment resolved a cross-functional disagreement in 14 days. The confidence interval floor ($8.5M) exceeded the threshold for action. The email tradeoff, while real, was offset 60:1 by conversion gains.