←BaldanAll skillsLet's talk
← Dashboards · 02 Experiment

E-mail test: should we send it?

A real randomised test: 64,000 customers got a men's e-mail, a women's e-mail, or nothing. This readout gives the verdict first, then the evidence, the checks that make it trustworthy, and what it would mean at scale.

64,000 real customers · randomised · 14-day window

Verdict

Send the men's e-mail.

+$0.77 spend per customer over two weeks against no e-mail (+118%; 95% interval +$0.49 to +$1.05, p < 0.0001).

Both e-mails beat no e-mail on visits, purchases and spend, even after correcting for six comparisons.

Against the women's e-mail it earns +$0.35 more per customer (p = 0.030): likely better, but not beyond doubt after the correction.

No e-mail

21,306 customers

Spend

$0.65

baseline

Purchases

0.57%

baseline

Visits

10.62%

baseline

Men's e-mail

21,307 customers

Spend

$1.42

+118%

Purchases

1.25%

+119%

Visits

18.28%

+72%

Women's e-mail

21,387 customers

Spend

$1.08

+65%

Purchases

0.88%

+54%

Visits

15.14%

+43%

Lift against no e-mail

Dot = estimate, line = 95% interval. Filled dots stay significant after correcting for 6 comparisons (p < 0.0083).

Spend per customerMen's e-mail+118%Women's e-mail+65%Conversion rateMen's e-mail+119%Women's e-mail+54%Visit rateMen's e-mail+72%Women's e-mail+43%0 = no change; right = better than no e-mail

From inbox to purchase

Share of each group at every step, within two weeks

Got the e-mail (or not)

No e-mail100%
Men's e-mail100%
Women's e-mail100%

Visited the site

No e-mail10.6%
Men's e-mail18.3%
Women's e-mail15.1%

Bought

No e-mail0.57%
Men's e-mail1.25%
Women's e-mail0.88%

Bars are scaled per step (visits to 20%, purchases to 1.4%) so the small purchase rates stay readable.

Who responds to the men's e-mail?

Absolute lift in spend per customer against no e-mail, by segment, with 95% intervals. Exploratory: many slices, so treat single wins with care.

Men's only19,196 customers+$0.68 ($0.70 → $1.38)Women's only19,087 customers+$0.62 ($0.49 → $1.11)Both4,330 customers+$1.82 ($1.18 → $3.01)

What the test says

Written from the same statistics as the charts

  • Spend per customer: $0.65 without an e-mail; men's e-mail +$0.77 (+118%), women's e-mail +$0.42 (+65%).
  • Conversion rate: 0.57% without an e-mail; men's e-mail +0.7 pts (+119%), women's e-mail +0.3 pts (+54%).
  • Visit rate: 10.62% without an e-mail; men's e-mail +7.7 pts (+72%), women's e-mail +4.5 pts (+43%).
  • For the men's e-mail, the biggest spend lift is among “Both” (bought before): +$1.82 per customer, p = 0.0074.
  • The smallest is among “4–6 months”: +$0.49 (still significant). Segment results are exploratory.
  • The split is clean: group sizes match the plan (p = 0.90) and the groups looked alike before the e-mail.
  • Purchases are rare (0.57% without an e-mail), so this test could only see conversion lifts of about +36% or more.

Can we trust the split?

Checks done before reading any result: group sizes and what customers looked like before the e-mail

Sample ratio (planned 1 : 1 : 1)

OK · p = 0.90

No e-mail 21,306 · Men's e-mail 21,307 · Women's e-mail 21,387

Before the e-mailNo e-mailMen'sWomen'sbalanced?
Months since last purchase5.75.85.8✓ 0.007
Spent last year ($)240.9242.8242.5✓ 0.008
New customers50.2%50.1%50.3%✓ 0.003
Bought men's55.3%55.1%54.9%✓ 0.009
Bought women's54.8%55.1%55.0%✓ 0.008
Web channel44.0%44.5%43.7%✓ 0.011
Urban area40.1%40.2%40.0%✓ 0.002

“Balanced” = standardised mean difference under 0.1, the usual rule of thumb. All far below it: the groups were alike before the e-mail.

What buyers spent

Among customers who bought: share in each spend band. Average per buyer: no e-mail $114, men's e-mail $114, women's e-mail $122, so the e-mails mainly bring more buyers.

$25+$50+$75+$100+$150+$200+$300+No e-mail · median $88Men's · median $67Women's · median $95band start, in dollars

If we send the men's e-mail to…

Scales the measured lift to a new audience like the test's, over the same two weeks. The range is the 95% interval.

Extra visits

7,659

6,995 to 8,323

Extra orders

681

500 to 861

Extra revenue

$76,983

$48,515 to $105,451

Assumes the new audience behaves like the 2008 test customers (all bought in the last 12 months).

How small a change could this test see?

Smallest true lift detectable with 80% power at 5% significance, with ~21,300 customers per group

Spend per customer+$0.31 (+48% of $0.65)
Conversion rate+0.20 pts (+36% of 0.57%)
Visit rate+0.84 pts (+8% of 10.62%)

Spend

123,670

per group

Purchases

68,144

per group

Visits

3,305

per group

Behind the dashboard

The experiment

64,000 customers who had bought in the last 12 months were split at random into three groups: no e-mail, a men's merchandise e-mail and a women's merchandise e-mail. Visits, purchases and spend were tracked for 14 days.

Rates

Visit and purchase rates are compared with a two-proportion z-test; the 95% interval is the estimate ± 1.96 standard errors.

Spend

Spend per customer (zero for most) is compared with Welch's test on means; with 21,000+ customers per group the normal approximation holds.

Six comparisons

Two e-mails × three metrics = 6 tests, so a result counts as clear only below p = 0.05 / 6 (Bonferroni). Segment charts are exploratory and not corrected.

Checks first

Before any result: the sample-ratio test (were the groups the planned size?) and covariate balance (did the groups look alike before the e-mail?).

Source

MineThatData E-Mail Analytics Challenge (Kevin Hillstrom, 2008). Released publicly for the 2008 challenge; no formal licence stated. One typo in the file (“Surburban”) is corrected.

The next question, which customer should get which e-mail, is answered by an uplift model in the Uplift promo machine-learning project, on the same data.