Causal inference
Not who will buy, but who the message changes: sending e-mails only where they make a difference.
80%
Women's effect, half the sends
+7.7 pts
Men's e-mail lift
+4.5 pts
Women's e-mail lift
6.8
Qini · women's
Overview
Most targeting models answer the wrong question: who is likely to act. Many of those people would act anyway, and messaging them wastes budget. Uplift modelling asks how much a message changes each person's behaviour.
That needs a randomised test, and this 64,000-customer e-mail experiment is one: a men's e-mail, a women's e-mail or nothing, assigned at random. The model learns where each e-mail works, and the demo turns that into a budget decision.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw data
64,000 customers
Three random groups, ~21,300 each
- 02
Effect
+7.7 pts · +4.5 pts
Visits within two weeks: men's · women's e-mail
- 03
Model
T-learner
One logistic model per group; uplift = difference
- 04
Validate
5-fold OOF
Every customer scored by models that never saw them
- 05
Compare
5 methods
T, S, class transformation, and a response model
- 06
Decide
½ list → 80%
Of the women's e-mail effect kept
Inside the model
Who does the e-mail change?
Not who will buy, but who buys because of the e-mail. Three neurons and one subtraction. Scroll through a real customer.
64,000 customers, split at random
No e-mail
10.6% visited
Men's e-mail
18.3% visited
Women's e-mail
15.1% visited
Each dot ≈ 2% of the group. Because the groups were random, the gaps are caused by the e-mails.
Customer C-58931
Chance of visiting the site, from each group's own model
uplift · Men's e-mail
+7.8 pts
uplift · Women's e-mail
+7.7 pts
Best e-mail per customer, over all 64,000
Nobody is predicted to visit less because of an e-mail.
The data
64,000 customers, one randomised e-mail test.
A third got a men's e-mail, a third a women's e-mail, a third nothing. Then the shop counted who visited its website in the next two weeks.
MineThatData E-Mail Analytics Challenge (Kevin Hillstrom, 2008)
Released by Kevin Hillstrom (MineThatData) for an open e-mail analytics challenge. Customers had all bought something in the past twelve months; the e-mail they received was decided at random, which is what makes a cause-and-effect answer possible. Released publicly for the 2008 challenge; no formal licence stated; it is widely used in uplift-modelling libraries.
64,000
customers
3 × ~21,300
groups
14 days
follow-up
10.6%
visit rate, no e-mail
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| recency | number | Months since the last purchase (1–12) |
| history | number | Dollars spent in the past year; the model uses log(1 + history) |
| history_segment | category | The same spend in bands · not used (duplicate) |
| mens · womens | 0 / 1 | Bought men's / women's merchandise in the past year (not the customer's gender) |
| newbie | 0 / 1 | New customer in the past twelve months |
| zip_code | category | Urban, suburban or rural area (fixed a 'Surburban' typo) |
| channel | category | Bought by phone, web or both |
| segment | treatment | Randomly assigned: men's e-mail, women's e-mail or no e-mail |
| visit | target | Visited the website within two weeks: what the uplift is measured on |
| conversion · spend | outcome | Bought / dollars spent within two weeks · reported, too rare to model per customer |
Explore the data
Both e-mails work. Not for the same people.
The men's e-mail lifts visits from 10.6% to 18.3%; the women's to 15.1%.
Visits move a lot; purchases move too, but they are rare (about 1 in 100), so uplift per customer is modelled on visits.
| Outcome | Men's e-mail | Women's e-mail |
|---|---|---|
| Visited | +7.7 pts (+7.0 pts to +8.3 pts) | +4.5 pts (+3.9 pts to +5.2 pts) |
| Bought | +0.7 pts (+0.5 pts to +0.9 pts) | +0.3 pts (+0.1 pts to +0.5 pts) |
| Spend per customer | +$0.77 ($0.49 to $1.05) | +$0.42 ($0.17 to $0.68) |
The women's e-mail barely moves customers who only bought men's products (+1.1 pts), against +7.4 pts for women's-only buyers. The men's e-mail works almost everywhere.
Split customers by
Both
6,448 customers
Men's only
28,818 customers
Women's only
28,734 customers
Men's e-mail: 68% of customers sit in one narrow band, +6 to +8 points. Women's e-mail: two groups, one from −2 to +4 points and one from +6 to +10.
Try it
Spend fewer e-mails, keep the effect.
First pick a campaign and a budget. Then build a customer and see which e-mail, if any, is worth sending.
Live demo
runs in your browser
Who gets the e-mail?
Campaign
+36.3
extra visits per 1,000 customers
+22.6
if the same number were picked at random
80%
of the effect of e-mailing everyone
50%
of e-mails saved
Measured on the randomised test itself: within the chosen group, visit rates of customers who got the e-mail minus those who didn't. Scores are out-of-fold.
Extra visits vs share e-mailed
The curve bends early: the women's e-mail works for some customers and barely for others, so ranking pays off.
Real rows from the dataset
click a row to try it in the demo ↓
| Customer | Last buy | Spent | Bought | Got | Visited | Best e-mail |
|---|---|---|---|---|---|---|
| C-58931 | 1 mo | $64 | Women's | No e-mail | Yes | Men's +7.8 pts |
| C-36629 | 2 mo | $30 | Men's | No e-mail | No | Men's +8.1 pts |
| C-46889 | 11 mo | $440 | Men's | No e-mail | No | Men's +6.2 pts |
| C-34500 | 1 mo | $146 | Women's | Men's e-mail | Yes | Men's +7.7 pts |
| C-49022 | 4 mo | $1076 | Women's | Men's e-mail | No | Women's +7.2 pts |
| C-45313 | 6 mo | $407 | Men's | Men's e-mail | No | Men's +7.5 pts |
| C-35407 | 6 mo | $953 | Women's | Women's e-mail | Yes | Men's +7.3 pts |
| C-32549 | 2 mo | $832 | Both | Women's e-mail | No | Men's +14.3 pts |
| C-27003 | 1 mo | $355 | Men's | Women's e-mail | No | Men's +8.1 pts |
Three customers from each group, two who didn't visit and one who did. What they actually got was random; the last column is what the model would send.
Live demo
runs in your browser
Bought before
New this year
Shops by
Area
Chance of a visit in the next two weeks
Recommendation
Send the women's e-mail
Expected to add 8.1 visits per 100 customers like this; the other e-mail adds 6.8.
How well it works
Half the women's e-mails, 80% of the effect.
Every score here is out-of-fold and measured against the randomised control group, the only fair way to check uplift.
For the women's e-mail every uplift method clearly beats ranking by who is likely to visit. For the men's e-mail all areas are small, and the plain response model even does best: there is little to find.
Ranked by the T-learner, e-mailing half the list gives 36.3 of the 45.2 extra visits per 1,000 that e-mailing everyone would.
Choosing who gets an e-mail, and which one, beats random at every budget. Here a plain response model does as well or slightly better, because most of the gain comes from the men's e-mail, which works on almost everyone.
Send the men's one. Picking the best e-mail per customer gives 183.1 visits per 1,000 against 182.8: no real difference.
| Policy | Visits per 1,000 customers |
|---|---|
| Nobody | 106.2 |
| Everyone · men's e-mail | 182.8 |
| Everyone · women's e-mail | 151.4 |
| Everyone · best e-mail per customer | 183.1 |
Estimated from the randomised groups by inverse-propensity weighting: each customer counts when the e-mail they got matches the policy.
Qini = average gap between a method's curve and random targeting. Top 30% = extra visits per 1,000 customers when only the best-ranked 30% are e-mailed.
| Method | Qini · men's | Qini · women's | Top 30% · men's | Top 30% · women's |
|---|---|---|---|---|
| T-learner · logistic (shipped) | 1.29 | 6.78 | 25.5 | 18.5 |
| T-learner · boosting | 1.05 | 5.50 | 23.4 | 18.6 |
| S-learner · boosting | 1.05 | 6.60 | 25.7 | 20.1 |
| Class transformation · logistic | 1.24 | 6.81 | 26.2 | 20.0 |
| Response model (who visits) | 3.65 | 4.42 | 27.8 | 19.6 |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
uplift() from ml/train.py + 5 helpers (r, save, export_logreg, make_pipe, _shuffle_policy)
pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load the e-mail experiment
Reads the Hillstrom file, fixes a zip-code typo, log-transforms past spend and maps each customer to an arm: no e-mail, men's e-mail or women's e-mail.
ml/train.py:2096 · 16 lines
from sklearn.ensemble import HistGradientBoostingClassifier print("uplift-promo") d = pd.read_csv(RAW / "hillstrom" / "hillstrom.csv") d["zip_code"] = d["zip_code"].replace({"Surburban": "Suburban"}) # typo in the source file d["log_history"] = np.log1p(d["history"]) arm_of = {"No E-Mail": "none", "Mens E-Mail": "mens", "Womens E-Mail": "womens"} d["arm"] = d["segment"].map(arm_of) arms = ["none", "mens", "womens"] num = ["recency", "log_history", "mens", "womens", "newbie"] cat = ["zip_code", "channel"] feats = num + cat y = d["visit"].to_numpy() a = d["arm"].to_numpy() N = len(d) share = {k: float((a == k).mean()) for k in arms} - 2
Measure average and subgroup effects
Because e-mails were assigned at random, a difference in means against the no-e-mail group is causal; effects get 95% intervals overall and per subgroup.
ml/train.py:2114 · 28 lines
# average effects with 95% intervals (randomised, so a difference in means is causal) def diff(col, t, sub=None): m = np.ones(N, bool) if sub is None else sub xt, xc = d.loc[m & (a == t), col], d.loc[m & (a == "none"), col] e = xt.mean() - xc.mean() se = np.sqrt(xt.var(ddof=1) / len(xt) + xc.var(ddof=1) / len(xc)) return [r(e, 5), r(e - 1.96 * se, 5), r(e + 1.96 * se, 5)] rates = {k: {"visit": r(d.visit[a == k].mean(), 4), "conversion": r(d.conversion[a == k].mean(), 5), "spend": r(d.spend[a == k].mean(), 3), "n": int((a == k).sum())} for k in arms} ate = {t: {c: diff(c, t) for c in ["visit", "conversion", "spend"]} for t in ["mens", "womens"]} print(f" visit rates {rates['none']['visit']} / mens {rates['mens']['visit']} / womens {rates['womens']['visit']}") rec_b = pd.cut(d.recency, [0, 3, 6, 9, 12], labels=["1–3 months", "4–6 months", "7–9 months", "10–12 months"]) hist_b = pd.cut(d.history, [0, 100, 200, 500, 1e9], labels=["< $100", "$100–200", "$200–500", "$500+"]) bought = np.where((d.mens == 1) & (d.womens == 1), "Both", np.where(d.mens == 1, "Men's only", "Women's only")) dims = { - 3
Predict every arm out of fold
Over stratified 5 folds it fits T-learners (a logistic and a boosting model per arm), an S-learner with arm as a feature, and a class-transformation logistic.
ml/train.py:2144 · 24 lines
# out-of-fold predictions for every customer, every arm def lr(): return make_pipe(LogisticRegression(max_iter=2000), num, cat) def hgb(): return make_pipe(HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05, max_iter=250, random_state=SEED), num, cat) strat = d["arm"] + "_" + d["visit"].astype(str) folds = list(StratifiedKFold(5, shuffle=True, random_state=SEED).split(d, strat)) oof = {m: {k: np.zeros(N) for k in arms} for m in ["T · logistic", "T · boosting", "S · boosting"]} ct = {t: np.zeros(N) for t in ["mens", "womens"]} for tr_i, te_i in folds: Xtr, Xte = d.iloc[tr_i], d.iloc[te_i] for k in arms: sub = Xtr[Xtr.arm == k] oof["T · logistic"][k][te_i] = lr().fit(sub[feats], sub.visit).predict_proba(Xte[feats])[:, 1] - 4
Compare uplift methods by Qini
Customers are ranked by each method's predicted uplift; extra visits per 1,000 from e-mailing the top share are traced and summarised as Qini area over random.
ml/train.py:2170 · 40 lines
grid = np.round(np.linspace(0, 1, 21), 2) def curve(score, t): """Extra visits per 1,000 customers if we e-mail the top share by `score`, measured on the randomised data.""" m = (a == t) | (a == "none") sc, yy, tt = score[m], y[m], (a[m] == t) o = np.argsort(-sc, kind="stable") yy, tt = yy[o], tt[o] n = len(sc) out = [] for g in grid: k = int(round(g * n)) if k == 0: out.append(0.0) continue nt, nc = tt[:k].sum(), (~tt[:k]).sum() - 5
Pick one e-mail per customer
The logistic T-learner picks the e-mail with the larger predicted uplift, and targeting policies are valued by inverse propensity weighting on the randomised data.
ml/train.py:2212 · 31 lines
# one decision per customer: which e-mail, if any — evaluated by inverse propensity on the randomised data P = oof["T · logistic"] up = np.vstack([P["mens"] - P["none"], P["womens"] - P["none"]]) best = np.where(up[0] >= up[1], "mens", "womens") best_up = up.max(axis=0) def value(policy): hit = a == policy w = np.array([1 / share[k] for k in a]) return float((hit * y * w).sum() / N * 1000) rng = np.random.default_rng(SEED) rand_mail = rng.choice(["mens", "womens"], size=N) budget = [] o = np.argsort(-best_up, kind="stable") resp_o = np.argsort(-np.maximum(P["mens"], P["womens"]), kind="stable") - 6
Fit final models, pick samples
Three logistic regressions, one per arm, are refit on all customers for the browser; nine customers (one visitor and two non-visitors per arm) become presets.
ml/train.py:2245 · 35 lines
# final models on all data, for the browser final = {k: lr().fit(d.loc[a == k, feats], d.loc[a == k, "visit"]) for k in arms} models = {k: export_logreg(final[k], num, cat) for k in arms} def up_hist(v): bins = np.arange(-0.06, 0.205, 0.02) h = np.histogram(np.clip(v, bins[0], bins[-1] - 1e-9), bins=bins)[0] return {"bins": [r(b, 3) for b in bins], "counts": h.tolist()} # sample customers: one row per arm per kind of profile, with what we saw and what the model expects samp_idx = [] rs = np.random.default_rng(SEED + 3) for k in arms: for vis in [1, 0]: pool = np.where((a == k) & (y == vis))[0] samp_idx += list(rs.choice(pool, size=2 if vis == 0 else 1, replace=False)) - 7
Export the JSON
Rates, effects, subgroups, Qini curves, budget and policy values and the three arm models are written to uplift-promo.json.
ml/train.py:2282 · 36 lines
hist_counts = d.groupby(hist_b, observed=False).size() save( "uplift-promo", { "dataset": { "name": "MineThatData E-Mail Analytics Challenge (Kevin Hillstrom, 2008)", "source": "https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html", "download": "http://www.minethatdata.com/Kevin_Hillstrom_MineThatData_E-MailAnalytics_DataMiningChallenge_2008.03.20.csv", "license": "Released publicly for the 2008 challenge; no formal licence stated", "rows": N, "window_days": 14, }, "rates": rates, "ate": ate, "subgroups": subgroups, "comparison": comparison,
In your browser · TypeScript
- 8
Score it in the browser
The demo runs the three exported models (no e-mail, men's e-mail, women's e-mail) on the same customer; each e-mail's uplift is its probability minus the no-e-mail one.
src/lib/logreg.ts:11 · 20 lines
export function predict(model: LogRegModel, input: Record<string, number | string>) { const contributions: Contribution[] = []; let logit = model.intercept; for (const [f, p] of Object.entries(model.numeric)) { const x = Number(input[f]); const effect = p.coef * ((x - p.mean) / p.scale); logit += effect; contributions.push({ feature: f, value: String(x), effect }); } for (const [f, cats] of Object.entries(model.categorical)) { const v = String(input[f]); const effect = cats[v] ?? 0; logit += effect; contributions.push({ feature: f, value: v, effect }); } const probability = 1 / (1 + Math.exp(-logit)); return { probability, logit, contributions: contributions.sort((a, b) => Math.abs(b.effect) - Math.abs(a.effect)) }; }
Notes
The decisions behind it.
Approach
- 1
Measured the average effect of each e-mail with 95% intervals, overall and inside customer groups.
- 2
Trained one logistic model per group (T-learner); a customer's uplift is the difference between their predicted visit chances.
- 3
Scored everyone out-of-fold and judged methods by Qini curves against the randomised control group.
- 4
Compared with gradient boosting, an S-learner, class transformation, and a plain response model that ranks by who is likely to visit.
Qini (area above random) · men's / women's
Honest note
For the women's e-mail, uplift ranking clearly beats a response model (Qini 6.8 vs 4.4). For the men's e-mail it doesn't (1.3 vs 3.6): that e-mail lifts almost everyone by a similar amount, so there is little to target, and saying so is the honest result. Purchases are too rare here (about 1 in 100) to model per customer, so uplift is measured on visits. The data is one retailer in 2008, released without a formal licence.
Reproduce: ml/train.py → uplift()
dataset sourceNext project
House Price Range