E-mail test: should we send it?
A real randomised test: 64,000 customers got a men's e-mail, a women's e-mail, or nothing. This readout gives the verdict first, then the evidence, the checks that make it trustworthy, and what it would mean at scale.
Verdict
Send the men's e-mail.
+$0.77 spend per customer over two weeks against no e-mail (+118%; 95% interval +$0.49 to +$1.05, p < 0.0001).
Both e-mails beat no e-mail on visits, purchases and spend, even after correcting for six comparisons.
Against the women's e-mail it earns +$0.35 more per customer (p = 0.030): likely better, but not beyond doubt after the correction.
No e-mail
21,306 customers
Spend
$0.65
baselinePurchases
0.57%
baselineVisits
10.62%
baselineMen's e-mail
21,307 customers
Spend
$1.42
+118%Purchases
1.25%
+119%Visits
18.28%
+72%Women's e-mail
21,387 customers
Spend
$1.08
+65%Purchases
0.88%
+54%Visits
15.14%
+43%Lift against no e-mail
Dot = estimate, line = 95% interval. Filled dots stay significant after correcting for 6 comparisons (p < 0.0083).
From inbox to purchase
Share of each group at every step, within two weeks
Got the e-mail (or not)
Visited the site
Bought
Bars are scaled per step (visits to 20%, purchases to 1.4%) so the small purchase rates stay readable.
Who responds to the men's e-mail?
Absolute lift in spend per customer against no e-mail, by segment, with 95% intervals. Exploratory: many slices, so treat single wins with care.
What the test says
Written from the same statistics as the charts
- Spend per customer: $0.65 without an e-mail; men's e-mail +$0.77 (+118%), women's e-mail +$0.42 (+65%).
- Conversion rate: 0.57% without an e-mail; men's e-mail +0.7 pts (+119%), women's e-mail +0.3 pts (+54%).
- Visit rate: 10.62% without an e-mail; men's e-mail +7.7 pts (+72%), women's e-mail +4.5 pts (+43%).
- For the men's e-mail, the biggest spend lift is among “Both” (bought before): +$1.82 per customer, p = 0.0074.
- The smallest is among “4–6 months”: +$0.49 (still significant). Segment results are exploratory.
- The split is clean: group sizes match the plan (p = 0.90) and the groups looked alike before the e-mail.
- Purchases are rare (0.57% without an e-mail), so this test could only see conversion lifts of about +36% or more.
Can we trust the split?
Checks done before reading any result: group sizes and what customers looked like before the e-mail
Sample ratio (planned 1 : 1 : 1)
OK · p = 0.90No e-mail 21,306 · Men's e-mail 21,307 · Women's e-mail 21,387
| Before the e-mail | No e-mail | Men's | Women's | balanced? |
|---|---|---|---|---|
| Months since last purchase | 5.7 | 5.8 | 5.8 | ✓ 0.007 |
| Spent last year ($) | 240.9 | 242.8 | 242.5 | ✓ 0.008 |
| New customers | 50.2% | 50.1% | 50.3% | ✓ 0.003 |
| Bought men's | 55.3% | 55.1% | 54.9% | ✓ 0.009 |
| Bought women's | 54.8% | 55.1% | 55.0% | ✓ 0.008 |
| Web channel | 44.0% | 44.5% | 43.7% | ✓ 0.011 |
| Urban area | 40.1% | 40.2% | 40.0% | ✓ 0.002 |
“Balanced” = standardised mean difference under 0.1, the usual rule of thumb. All far below it: the groups were alike before the e-mail.
What buyers spent
Among customers who bought: share in each spend band. Average per buyer: no e-mail $114, men's e-mail $114, women's e-mail $122, so the e-mails mainly bring more buyers.
If we send the men's e-mail to…
Scales the measured lift to a new audience like the test's, over the same two weeks. The range is the 95% interval.
Extra visits
7,659
6,995 to 8,323
Extra orders
681
500 to 861
Extra revenue
$76,983
$48,515 to $105,451
Assumes the new audience behaves like the 2008 test customers (all bought in the last 12 months).
How small a change could this test see?
Smallest true lift detectable with 80% power at 5% significance, with ~21,300 customers per group
Spend
123,670
per group
Purchases
68,144
per group
Visits
3,305
per group
Behind the dashboard
The experiment
64,000 customers who had bought in the last 12 months were split at random into three groups: no e-mail, a men's merchandise e-mail and a women's merchandise e-mail. Visits, purchases and spend were tracked for 14 days.
Rates
Visit and purchase rates are compared with a two-proportion z-test; the 95% interval is the estimate ± 1.96 standard errors.
Spend
Spend per customer (zero for most) is compared with Welch's test on means; with 21,000+ customers per group the normal approximation holds.
Six comparisons
Two e-mails × three metrics = 6 tests, so a result counts as clear only below p = 0.05 / 6 (Bonferroni). Segment charts are exploratory and not corrected.
Checks first
Before any result: the sample-ratio test (were the groups the planned size?) and covariate balance (did the groups look alike before the e-mail?).
Source
MineThatData E-Mail Analytics Challenge (Kevin Hillstrom, 2008). Released publicly for the 2008 challenge; no formal licence stated. One typo in the file (“Surburban”) is corrected.
The next question, which customer should get which e-mail, is answered by an uplift model in the Uplift promo machine-learning project, on the same data.