←BaldanAll skillsLet's talk
Machine Learning lab

Causal inference

Not who will buy, but who the message changes: sending e-mails only where they make a difference.

80%

Women's effect, half the sends

+7.7 pts

Men's e-mail lift

+4.5 pts

Women's e-mail lift

6.8

Qini · women's

Overview

Most targeting models answer the wrong question: who is likely to act. Many of those people would act anyway, and messaging them wastes budget. Uplift modelling asks how much a message changes each person's behaviour.

That needs a randomised test, and this 64,000-customer e-mail experiment is one: a men's e-mail, a women's e-mail or nothing, assigned at random. The model learns where each e-mail works, and the demo turns that into a budget decision.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    64,000 customers

    Three random groups, ~21,300 each

  2. 02

    Effect

    +7.7 pts · +4.5 pts

    Visits within two weeks: men's · women's e-mail

  3. 03

    Model

    T-learner

    One logistic model per group; uplift = difference

  4. 04

    Validate

    5-fold OOF

    Every customer scored by models that never saw them

  5. 05

    Compare

    5 methods

    T, S, class transformation, and a response model

  6. 06

    Decide

    ½ list → 80%

    Of the women's e-mail effect kept

Inside the model

Who does the e-mail change?

Not who will buy, but who buys because of the e-mail. Three neurons and one subtraction. Scroll through a real customer.

Customer

64,000 customers, split at random

No e-mail

10.6% visited

Men's e-mail

18.3% visited

Women's e-mail

15.1% visited

Each dot ≈ 2% of the group. Because the groups were random, the gaps are caused by the e-mails.

Customer C-58931

Months since last purchase1
Spent last year$63.51
Bought men's itemsno
Bought women's itemsyes
New customeryes
Area · channelRural · Web

Chance of visiting the site, from each group's own model

No e-mail
13.4%
Men's e-mail
21.2%
Women's e-mail
21.1%

uplift · Men's e-mail

+7.8 pts

uplift · Women's e-mail

+7.7 pts

Best e-mail per customer, over all 64,000

69%
31%
Men's e-mail lifts them mostWomen's e-mail lifts them most

Nobody is predicted to visit less because of an e-mail.

The data

64,000 customers, one randomised e-mail test.

A third got a men's e-mail, a third a women's e-mail, a third nothing. Then the shop counted who visited its website in the next two weeks.

MineThatData E-Mail Analytics Challenge (Kevin Hillstrom, 2008)

Released by Kevin Hillstrom (MineThatData) for an open e-mail analytics challenge. Customers had all bought something in the past twelve months; the e-mail they received was decided at random, which is what makes a cause-and-effect answer possible. Released publicly for the 2008 challenge; no formal licence stated; it is widely used in uplift-modelling libraries.

64,000

customers

3 × ~21,300

groups

14 days

follow-up

10.6%

visit rate, no e-mail

Kevin Hillstrom's blog (2008)

Column dictionary

● used by the model

ColumnTypeWhat it means
recencynumberMonths since the last purchase (1–12)
historynumberDollars spent in the past year; the model uses log(1 + history)
history_segmentcategoryThe same spend in bands · not used (duplicate)
mens · womens0 / 1Bought men's / women's merchandise in the past year (not the customer's gender)
newbie0 / 1New customer in the past twelve months
zip_codecategoryUrban, suburban or rural area (fixed a 'Surburban' typo)
channelcategoryBought by phone, web or both
segmenttreatmentRandomly assigned: men's e-mail, women's e-mail or no e-mail
visittargetVisited the website within two weeks: what the uplift is measured on
conversion · spendoutcomeBought / dollars spent within two weeks · reported, too rare to model per customer

Explore the data

Both e-mails work. Not for the same people.

Visited within two weeks

The men's e-mail lifts visits from 10.6% to 18.3%; the women's to 15.1%.

0.0%10.0%20.0%10.6%No e-mail18.3%Men's e-mailWomen's e-mail
Average effect, with 95% intervals

Visits move a lot; purchases move too, but they are rare (about 1 in 100), so uplift per customer is modelled on visits.

OutcomeMen's e-mailWomen's e-mail
Visited+7.7 pts (+7.0 pts to +8.3 pts)+4.5 pts (+3.9 pts to +5.2 pts)
Bought+0.7 pts (+0.5 pts to +0.9 pts)+0.3 pts (+0.1 pts to +0.5 pts)
Spend per customer+$0.77 ($0.49 to $1.05)+$0.42 ($0.17 to $0.68)
Where each e-mail works

The women's e-mail barely moves customers who only bought men's products (+1.1 pts), against +7.4 pts for women's-only buyers. The men's e-mail works almost everywhere.

Split customers by

Both

6,448 customers

Men's only

28,818 customers

Women's only

28,734 customers

0+5+10+15
men's e-mail women's e-maildot = extra visits per 100 customers, line = 95% interval
Predicted uplift per customer

Men's e-mail: 68% of customers sit in one narrow band, +6 to +8 points. Women's e-mail: two groups, one from −2 to +4 points and one from +6 to +10.

Men's e-mailWomen's e-mail
0%50%100%-60+6+12+18

Try it

Spend fewer e-mails, keep the effect.

First pick a campaign and a budget. Then build a customer and see which e-mail, if any, is worth sending.

Live demo

runs in your browser

Who gets the e-mail?

Campaign

+36.3

extra visits per 1,000 customers

+22.6

if the same number were picked at random

80%

of the effect of e-mailing everyone

50%

of e-mails saved

Measured on the randomised test itself: within the chosen group, visit rates of customers who got the e-mail minus those who didn't. Scores are out-of-fold.

Extra visits vs share e-mailed

uplift model response model random
025500%25%50%75%100%share of customers e-mailed

The curve bends early: the women's e-mail works for some customers and barely for others, so ranking pays off.

Real rows from the dataset

click a row to try it in the demo ↓

CustomerLast buySpentBoughtGotVisitedBest e-mail
C-589311 mo$64Women'sNo e-mailYesMen's +7.8 pts
C-366292 mo$30Men'sNo e-mailNoMen's +8.1 pts
C-4688911 mo$440Men'sNo e-mailNoMen's +6.2 pts
C-345001 mo$146Women'sMen's e-mailYesMen's +7.7 pts
C-490224 mo$1076Women'sMen's e-mailNoWomen's +7.2 pts
C-453136 mo$407Men'sMen's e-mailNoMen's +7.5 pts
C-354076 mo$953Women'sWomen's e-mailYesMen's +7.3 pts
C-325492 mo$832BothWomen's e-mailNoMen's +14.3 pts
C-270031 mo$355Men'sWomen's e-mailNoMen's +8.1 pts

Three customers from each group, two who didn't visit and one who did. What they actually got was random; the last column is what the model would send.

Live demo

runs in your browser

Bought before

New this year

Shops by

Area

Chance of a visit in the next two weeks

No e-mail7.1%
Men's e-mail13.9%+6.8 pts
Women's e-mail15.1%+8.1 pts

Recommendation

Send the women's e-mail

Expected to add 8.1 visits per 100 customers like this; the other e-mail adds 6.8.

How well it works

Half the women's e-mails, 80% of the effect.

Every score here is out-of-fold and measured against the randomised control group, the only fair way to check uplift.

Qini: extra visits above random targeting

For the women's e-mail every uplift method clearly beats ranking by who is likely to visit. For the men's e-mail all areas are small, and the plain response model even does best: there is little to find.

Men's e-mailWomen's e-mail
0.05.010.01.36.8T · LR1.15.5T · GB1.16.6S · GB1.26.8Class tr.3.64.4Response
Women's e-mail: uplift curves

Ranked by the T-learner, e-mailing half the list gives 36.3 of the 45.2 extra visits per 1,000 that e-mailing everyone would.

T-learner · logisticClass transformationResponse model
025500%25%50%75%100%Share of customers e-mailedExtra visits / 1,000
One e-mail per customer, either kind

Choosing who gets an e-mail, and which one, beats random at every budget. Here a plain response model does as well or slightly better, because most of the gain comes from the men's e-mail, which works on almost everyone.

Uplift modelResponse modelRandom
1001451900%25%50%75%100%Share of customers e-mailedVisits / 1,000
If everyone gets an e-mail anyway

Send the men's one. Picking the best e-mail per customer gives 183.1 visits per 1,000 against 182.8: no real difference.

PolicyVisits per 1,000 customers
Nobody106.2
Everyone · men's e-mail182.8
Everyone · women's e-mail151.4
Everyone · best e-mail per customer183.1

Estimated from the randomised groups by inverse-propensity weighting: each customer counts when the e-mail they got matches the policy.

All methods

Qini = average gap between a method's curve and random targeting. Top 30% = extra visits per 1,000 customers when only the best-ranked 30% are e-mailed.

MethodQini · men'sQini · women'sTop 30% · men'sTop 30% · women's
T-learner · logistic (shipped)1.296.7825.518.5
T-learner · boosting1.055.5023.418.6
S-learner · boosting1.056.6025.720.1
Class transformation · logistic1.246.8126.220.0
Response model (who visits)3.654.4227.819.6

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

uplift() from ml/train.py + 5 helpers (r, save, export_logreg, make_pipe, _shuffle_policy)

pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load the e-mail experiment

    Reads the Hillstrom file, fixes a zip-code typo, log-transforms past spend and maps each customer to an arm: no e-mail, men's e-mail or women's e-mail.

    ml/train.py:2096 · 16 lines

    from sklearn.ensemble import HistGradientBoostingClassifier
    
    print("uplift-promo")
    d = pd.read_csv(RAW / "hillstrom" / "hillstrom.csv")
    d["zip_code"] = d["zip_code"].replace({"Surburban": "Suburban"})  # typo in the source file
    d["log_history"] = np.log1p(d["history"])
    arm_of = {"No E-Mail": "none", "Mens E-Mail": "mens", "Womens E-Mail": "womens"}
    d["arm"] = d["segment"].map(arm_of)
    arms = ["none", "mens", "womens"]
    num = ["recency", "log_history", "mens", "womens", "newbie"]
    cat = ["zip_code", "channel"]
    feats = num + cat
    y = d["visit"].to_numpy()
    a = d["arm"].to_numpy()
    N = len(d)
    share = {k: float((a == k).mean()) for k in arms}
  2. 2

    Measure average and subgroup effects

    Because e-mails were assigned at random, a difference in means against the no-e-mail group is causal; effects get 95% intervals overall and per subgroup.

    ml/train.py:2114 · 28 lines

    # average effects with 95% intervals (randomised, so a difference in means is causal)
    def diff(col, t, sub=None):
        m = np.ones(N, bool) if sub is None else sub
        xt, xc = d.loc[m & (a == t), col], d.loc[m & (a == "none"), col]
        e = xt.mean() - xc.mean()
        se = np.sqrt(xt.var(ddof=1) / len(xt) + xc.var(ddof=1) / len(xc))
        return [r(e, 5), r(e - 1.96 * se, 5), r(e + 1.96 * se, 5)]
    
    rates = {k: {"visit": r(d.visit[a == k].mean(), 4), "conversion": r(d.conversion[a == k].mean(), 5), "spend": r(d.spend[a == k].mean(), 3), "n": int((a == k).sum())} for k in arms}
    ate = {t: {c: diff(c, t) for c in ["visit", "conversion", "spend"]} for t in ["mens", "womens"]}
    print(f"  visit rates {rates['none']['visit']} / mens {rates['mens']['visit']} / womens {rates['womens']['visit']}")
    
    rec_b = pd.cut(d.recency, [0, 3, 6, 9, 12], labels=["1–3 months", "4–6 months", "7–9 months", "10–12 months"])
    hist_b = pd.cut(d.history, [0, 100, 200, 500, 1e9], labels=["< $100", "$100–200", "$200–500", "$500+"])
    bought = np.where((d.mens == 1) & (d.womens == 1), "Both", np.where(d.mens == 1, "Men's only", "Women's only"))
    dims = {
  3. 3

    Predict every arm out of fold

    Over stratified 5 folds it fits T-learners (a logistic and a boosting model per arm), an S-learner with arm as a feature, and a class-transformation logistic.

    ml/train.py:2144 · 24 lines

    # out-of-fold predictions for every customer, every arm
    def lr():
        return make_pipe(LogisticRegression(max_iter=2000), num, cat)
    
    def hgb():
        return make_pipe(HistGradientBoostingClassifier(max_depth=3, learning_rate=0.05, max_iter=250, random_state=SEED), num, cat)
    
    strat = d["arm"] + "_" + d["visit"].astype(str)
    folds = list(StratifiedKFold(5, shuffle=True, random_state=SEED).split(d, strat))
    oof = {m: {k: np.zeros(N) for k in arms} for m in ["T · logistic", "T · boosting", "S · boosting"]}
    ct = {t: np.zeros(N) for t in ["mens", "womens"]}
    for tr_i, te_i in folds:
        Xtr, Xte = d.iloc[tr_i], d.iloc[te_i]
        for k in arms:
            sub = Xtr[Xtr.arm == k]
            oof["T · logistic"][k][te_i] = lr().fit(sub[feats], sub.visit).predict_proba(Xte[feats])[:, 1]
  4. 4

    Compare uplift methods by Qini

    Customers are ranked by each method's predicted uplift; extra visits per 1,000 from e-mailing the top share are traced and summarised as Qini area over random.

    ml/train.py:2170 · 40 lines

    grid = np.round(np.linspace(0, 1, 21), 2)
    
    def curve(score, t):
        """Extra visits per 1,000 customers if we e-mail the top share by `score`, measured on the randomised data."""
        m = (a == t) | (a == "none")
        sc, yy, tt = score[m], y[m], (a[m] == t)
        o = np.argsort(-sc, kind="stable")
        yy, tt = yy[o], tt[o]
        n = len(sc)
        out = []
        for g in grid:
            k = int(round(g * n))
            if k == 0:
                out.append(0.0)
                continue
            nt, nc = tt[:k].sum(), (~tt[:k]).sum()
  5. 5

    Pick one e-mail per customer

    The logistic T-learner picks the e-mail with the larger predicted uplift, and targeting policies are valued by inverse propensity weighting on the randomised data.

    ml/train.py:2212 · 31 lines

    # one decision per customer: which e-mail, if any — evaluated by inverse propensity on the randomised data
    P = oof["T · logistic"]
    up = np.vstack([P["mens"] - P["none"], P["womens"] - P["none"]])
    best = np.where(up[0] >= up[1], "mens", "womens")
    best_up = up.max(axis=0)
    
    def value(policy):
        hit = a == policy
        w = np.array([1 / share[k] for k in a])
        return float((hit * y * w).sum() / N * 1000)
    
    rng = np.random.default_rng(SEED)
    rand_mail = rng.choice(["mens", "womens"], size=N)
    budget = []
    o = np.argsort(-best_up, kind="stable")
    resp_o = np.argsort(-np.maximum(P["mens"], P["womens"]), kind="stable")
  6. 6

    Fit final models, pick samples

    Three logistic regressions, one per arm, are refit on all customers for the browser; nine customers (one visitor and two non-visitors per arm) become presets.

    ml/train.py:2245 · 35 lines

    # final models on all data, for the browser
    final = {k: lr().fit(d.loc[a == k, feats], d.loc[a == k, "visit"]) for k in arms}
    models = {k: export_logreg(final[k], num, cat) for k in arms}
    
    def up_hist(v):
        bins = np.arange(-0.06, 0.205, 0.02)
        h = np.histogram(np.clip(v, bins[0], bins[-1] - 1e-9), bins=bins)[0]
        return {"bins": [r(b, 3) for b in bins], "counts": h.tolist()}
    
    # sample customers: one row per arm per kind of profile, with what we saw and what the model expects
    samp_idx = []
    rs = np.random.default_rng(SEED + 3)
    for k in arms:
        for vis in [1, 0]:
            pool = np.where((a == k) & (y == vis))[0]
            samp_idx += list(rs.choice(pool, size=2 if vis == 0 else 1, replace=False))
  7. 7

    Export the JSON

    Rates, effects, subgroups, Qini curves, budget and policy values and the three arm models are written to uplift-promo.json.

    ml/train.py:2282 · 36 lines

    hist_counts = d.groupby(hist_b, observed=False).size()
    save(
        "uplift-promo",
        {
            "dataset": {
                "name": "MineThatData E-Mail Analytics Challenge (Kevin Hillstrom, 2008)",
                "source": "https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html",
                "download": "http://www.minethatdata.com/Kevin_Hillstrom_MineThatData_E-MailAnalytics_DataMiningChallenge_2008.03.20.csv",
                "license": "Released publicly for the 2008 challenge; no formal licence stated",
                "rows": N,
                "window_days": 14,
            },
            "rates": rates,
            "ate": ate,
            "subgroups": subgroups,
            "comparison": comparison,

In your browser · TypeScript

  1. 8

    Score it in the browser

    The demo runs the three exported models (no e-mail, men's e-mail, women's e-mail) on the same customer; each e-mail's uplift is its probability minus the no-e-mail one.

    src/lib/logreg.ts:11 · 20 lines

    export function predict(model: LogRegModel, input: Record<string, number | string>) {
      const contributions: Contribution[] = [];
      let logit = model.intercept;
    
      for (const [f, p] of Object.entries(model.numeric)) {
        const x = Number(input[f]);
        const effect = p.coef * ((x - p.mean) / p.scale);
        logit += effect;
        contributions.push({ feature: f, value: String(x), effect });
      }
      for (const [f, cats] of Object.entries(model.categorical)) {
        const v = String(input[f]);
        const effect = cats[v] ?? 0;
        logit += effect;
        contributions.push({ feature: f, value: v, effect });
      }
    
      const probability = 1 / (1 + Math.exp(-logit));
      return { probability, logit, contributions: contributions.sort((a, b) => Math.abs(b.effect) - Math.abs(a.effect)) };
    }

Notes

The decisions behind it.

Approach

  1. 1

    Measured the average effect of each e-mail with 95% intervals, overall and inside customer groups.

  2. 2

    Trained one logistic model per group (T-learner); a customer's uplift is the difference between their predicted visit chances.

  3. 3

    Scored everyone out-of-fold and judged methods by Qini curves against the randomised control group.

  4. 4

    Compared with gradient boosting, an S-learner, class transformation, and a plain response model that ranks by who is likely to visit.

Qini (area above random) · men's / women's

T-learner · logistic1.29 / 6.78shipped
T-learner · boosting1.05 / 5.50
S-learner · boosting1.05 / 6.60
Class transformation · logistic1.24 / 6.81
Response model (who visits)3.65 / 4.42

Honest note

For the women's e-mail, uplift ranking clearly beats a response model (Qini 6.8 vs 4.4). For the men's e-mail it doesn't (1.3 vs 3.6): that e-mail lifts almost everyone by a similar amount, so there is little to target, and saying so is the honest result. Purchases are too rare here (about 1 in 100) to model per customer, so uplift is measured on visits. The data is one retailer in 2008, released without a formal licence.

Reproduce: ml/train.py → uplift()

dataset source

Next project

House Price Range