←BaldanAll skillsLet's talk
Machine Learning lab

Classification

A retention team can't call everyone. This ranks who to call first, and shows what each budget buys.

38%

Leavers in top 10%

3.8×

Top-decile lift

0.838

ROC-AUC

10,000

Customers

Overview

Predicting churn is only half the job. A retention team has a fixed number of calls, so the real question is how many of the customers about to leave they reach if they call the riskiest first.

The model is a logistic regression with three engineered features: age², product count as categories and a zero-balance flag. That takes it from a weak linear model to nearly boosting-level ranking, while every effect stays readable. A budget slider turns the scores into campaign numbers.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    10,000 customers

    20.4% closed their account

  2. 02

    Privacy

    −4 columns

    Surname and IDs dropped; gender left out

  3. 03

    Features

    +3 → 16 cols

    Age², products as categories, zero balance

  4. 04

    Validate

    5-fold OOF

    Every customer scored by a model that never saw them

  5. 05

    Rank

    riskiest first

    Top decile churns at 78%

  6. 06

    Plan

    top 10% → 38%

    of all leavers reached with a tenth of the calls

Inside the model

One customer, one neuron.

Logistic regression is a single artificial neuron. Age and age² enter as one input here, as do the balance and the empty-account flag.

customer
Products held2Age27Active memberNoGeographyFranceYears with the bank8Balance (€)141,2242 more inputssmall pushesBIAS (STARTING POINT)−0.01ΣNEURON0%50%100%1.5%chance of leavingSIGMOID: SUM → 0–100%SUMWEIGHTS GROWING FROM 0ΣΣΣΣΣΣLAYERS OF NEURONS = A NEURAL NETWORK

The data

10,000 bank customers, 20.4% left.

One row per customer with their profile, their products and whether they closed the account.

Bank Customer Churn (Churn_Modelling.csv)

A widely used teaching dataset of customers from a European bank in France, Germany and Spain. It originally circulates on Kaggle; this rebuild uses a public GitHub copy so no login is needed.

10,000

customers

20.4%

churn rate

36%

zero balance

3

countries

Copy used (GitHub mirror)

Column dictionary

● used by the model

ColumnTypeWhat it means
RowNumber · CustomerIdidIdentifiers · not used
SurnametextFamily name · never used or shown
CreditScorenumberCredit score (350–850)
GeographycategoryFrance, Germany or Spain
GendercategoryLeft out (fairness)
AgenumberAge in years; the model also gets age²
TenurenumberYears with the bank (0–10)
BalancenumberAccount balance; a zero-balance flag is added
NumOfProducts1–4Products held, treated as categories
HasCrCardyes/noHas a credit card · not used in the final model
IsActiveMemberyes/noUses the bank actively
EstimatedSalarynumberEstimated yearly salary
ExitedtargetClosed the account: the thing we predict

Explore the data

Churn isn't a straight line.

Products, age and country bend the risk in ways a raw linear model misses.

Churn rate by number of products

Two products: 8%. Three: 83%. Four: 100% (only 60 customers). Not a trend: categories.

0%100%200%1 product8%2 products3 products100%4 products
Churn rate by age

Rises to 56% at 51–60, then drops for 61+. A curve, so the model also gets age².

0%50%100%8%18–3031–4041–5056%51–6061+
Churn rate by country

Germany churns at 32%, about twice France and Spain.

0%25%50%16%France32%GermanySpain
Balance: stayed vs left

39% of customers who stayed hold a zero balance, vs 25% of those who left. Leavers cluster at 100–140k.

StayedLeft
0%25%50%0k60k120k180k240k
Active vs inactive members

Inactive members churn at 27%, active ones at 14%.

0%25%50%27%Inactive14%Active
What the features add

Same logistic regression: AUC 0.757 on raw columns, 0.838 with age², product categories and a zero-balance flag.

0.000.501.000.76Logistic rawLogistic + fe…Random Forest0.86Gradient Boos…

Try it

Score a customer, then plan the campaign.

Load a real customer into the scorer, or skip to the budget slider below.

Real rows from the dataset

click a row to try it in the demo ↓

CustomerCountryAgeProductsActiveBalanceRiskOutcome
C01619France272No141,2241%Stayed
C00544Spain511No169,31258%Left
C08166Spain372No154,7138%Stayed
C03560France641No142,02858%Left
C02158France442Yes07%Stayed
C02456France471Yes028%Left
C01634France351No024%Stayed
C04564Germany643No153,46998%Left
C02531France451No68,64043%Left
C06410France332Yes02%Stayed

Live demo

runs in your browser

Build a customer

Country

Products held

Active member

Chance this customer leaves

11%Low risk
0%30%60%100%

Why

lowers risk raises risk
  • Number of products1
  • Active memberYes
  • CountryFrance
  • Zero balance0
  • Age37
  • Balance100k

Plan the retention campaign

1,000

customers called

38%

of churners reached (779)

78%

of calls hit a churner

3.8×

better than calling at random

Calling the same 10% at random would reach about 10% of churners. Gradient boosting would reach 41%.

Cumulative gains

our model boosting random
0%50%100%0%25%50%75%100%share of customers called, riskiest first38%

How well it works

Call 10%, reach 38% of leavers.

Out-of-fold scores on all 10,000 customers. For a campaign, what matters is how well the top of the list is sorted, not the average.

Churn rate by risk decile

The riskiest tenth churns at 78%: 3.8× the average. The safest tenth: 3%.

0%50%100%78%D1D2D3D4D5D6D7D8D93%D10
ROC curve

Logistic + features 0.838, boosting 0.864.

Logistic + featuresGradient boosting
0.00.51.00.000.250.500.751.00False positive rateTrue positive rate
All models

Share of all churners reached when calling the top 10%, 20% or 30% of the list.

ModelROC-AUCTop 10%Top 20%Top 30%
Logistic · raw columns0.75727.8%46.1%60.1%
Logistic · engineered0.83838.2%59.2%70.9%
Random Forest0.85941.0%62.1%74.2%
Gradient Boosting0.86440.7%62.1%74.3%

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

bank() from ml/train.py + 7 helpers (r, save, roc_points, hist_by_group, export_logreg, make_pipe, rate_by)

pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load the customer file

    Reads bank.csv and uses Exited (1 = the customer left) as the 0/1 target.

    ml/train.py:1252 · 4 lines

    print("bank-churn")
    df = pd.read_csv(RAW / "bank" / "bank.csv")
    y = df["Exited"].to_numpy()
    ys = pd.Series(y)
  2. 2

    Engineer extra features

    Adds age squared, a zero-balance flag, and product count and activity as categories; identifiers and Gender (for fairness) are left out of both feature sets.

    ml/train.py:1258 · 7 lines

    df["age2"] = df["Age"] ** 2
    df["zero_balance"] = (df["Balance"] == 0).astype(int)
    df["products"] = df["NumOfProducts"].astype(str)
    df["active"] = df["IsActiveMember"].map({0: "No", 1: "Yes"})
    # Left out: Surname and CustomerId (identifiers), Gender (fairness).
    raw_num, raw_cat = ["CreditScore", "Age", "Tenure", "Balance", "NumOfProducts", "HasCrCard", "IsActiveMember", "EstimatedSalary"], ["Geography"]
    eng_num, eng_cat = ["CreditScore", "Age", "age2", "Tenure", "Balance", "zero_balance", "EstimatedSalary"], ["Geography", "products", "active"]
  3. 3

    Set up cross-validation and candidates

    Stratified 5-fold CV for four variants: logistic on raw columns, logistic on engineered ones, random forest and gradient boosting, plus top-share capture helpers.

    ml/train.py:1267 · 21 lines

    cv = StratifiedKFold(5, shuffle=True, random_state=SEED)
    variants = {
        "Logistic · raw columns": (LogisticRegression(max_iter=3000), raw_num, raw_cat),
        "Logistic · engineered": (LogisticRegression(max_iter=3000), eng_num, eng_cat),
        "Random Forest": (RandomForestClassifier(n_estimators=400, min_samples_leaf=5, random_state=SEED, n_jobs=-1), raw_num, raw_cat),
        "Gradient Boosting": (GradientBoostingClassifier(random_state=SEED), raw_num, raw_cat),
    }
    
    def capture(scores, frac):
        order = np.argsort(-scores)
        k = int(round(len(scores) * frac))
        return float(y[order[:k]].sum() / y.sum())
    
    def gains(scores):
        order = np.argsort(-scores)
        cum = np.cumsum(y[order])
  4. 4

    Score every variant out of fold

    Each pipeline gets out-of-fold churn probabilities, scored by ROC AUC and by the share of leavers caught when calling the top 10, 20 or 30% of customers.

    ml/train.py:1290 · 15 lines

    comparison, oof = [], {}
    for name, (model, num, cat) in variants.items():
        p = cross_val_predict(make_pipe(model, num, cat), df[num + cat], y, cv=cv, method="predict_proba")[:, 1]
        oof[name] = p
        comparison.append(
            {
                "model": name,
                "roc_auc": r(roc_auc_score(y, p)),
                "top10": r(capture(p, 0.10), 3),
                "top20": r(capture(p, 0.20), 3),
                "top30": r(capture(p, 0.30), 3),
            }
        )
        c = comparison[-1]
        print(f"  {name:24s} AUC={c['roc_auc']:.3f}  top10%={c['top10']:.3f}  top20%={c['top20']:.3f}")
  5. 5

    Keep engineered logistic, pick demo customers

    The engineered logistic regression is refit on all customers for export; ten real customers (5 who left, 5 who stayed) with out-of-fold scores become presets.

    ml/train.py:1307 · 21 lines

    chosen = "Logistic · engineered"
    final = make_pipe(LogisticRegression(max_iter=3000), eng_num, eng_cat).fit(df[eng_num + eng_cat], y)
    p_lr = oof[chosen]
    
    pick = pd.concat([df[y == 1].sample(5, random_state=SEED), df[y == 0].sample(5, random_state=SEED)]).sample(frac=1, random_state=SEED)
    samples = [
        {
            "id": f"C{int(i) + 1:05d}",
            "CreditScore": int(row.CreditScore),
            "Geography": row.Geography,
            "Age": int(row.Age),
            "Tenure": int(row.Tenure),
            "Balance": int(round(row.Balance)),
            "NumOfProducts": int(row.NumOfProducts),
            "IsActiveMember": int(row.IsActiveMember),
            "EstimatedSalary": int(round(row.EstimatedSalary)),
  6. 6

    Export lift, gains and the model

    Churn by age band and lift per score decile are computed, then metrics, the model, EDA charts and gains curves are written to bank-churn.json.

    ml/train.py:1330 · 41 lines

    age_band = pd.cut(df["Age"], bins=[17, 30, 40, 50, 60, 92], labels=["18–30", "31–40", "41–50", "51–60", "61+"])
    lift = []
    order = np.argsort(-p_lr)
    for dec in range(10):
        idx = order[dec * len(order) // 10 : (dec + 1) * len(order) // 10]
        lift.append({"decile": dec + 1, "rate": r(y[idx].mean(), 4), "lift": r(y[idx].mean() / y.mean(), 3)})
    
    save(
        "bank-churn",
        {
            "dataset": {
                "name": "Bank Customer Churn (Churn_Modelling.csv)",
                "source": "https://github.com/selva86/datasets/blob/master/Churn_Modelling.csv",
                "rows": int(len(df)),
                "raw_cols": int(df.shape[1] - 4),
                "churn_rate": r(y.mean(), 4),

In your browser · TypeScript

  1. 7

    Score it in the browser

    The exported coefficients are plain numbers, so the demo scales the inputs, adds up coefficient × value and applies the sigmoid. No server, no Python.

    src/lib/logreg.ts:11 · 20 lines

    export function predict(model: LogRegModel, input: Record<string, number | string>) {
      const contributions: Contribution[] = [];
      let logit = model.intercept;
    
      for (const [f, p] of Object.entries(model.numeric)) {
        const x = Number(input[f]);
        const effect = p.coef * ((x - p.mean) / p.scale);
        logit += effect;
        contributions.push({ feature: f, value: String(x), effect });
      }
      for (const [f, cats] of Object.entries(model.categorical)) {
        const v = String(input[f]);
        const effect = cats[v] ?? 0;
        logit += effect;
        contributions.push({ feature: f, value: v, effect });
      }
    
      const probability = 1 / (1 + Math.exp(-logit));
      return { probability, logit, contributions: contributions.sort((a, b) => Math.abs(b.effect) - Math.abs(a.effect)) };
    }

Notes

The decisions behind it.

Approach

  1. 1

    Dropped surnames and IDs, and left gender out of the model.

  2. 2

    Read the shape of the risk first: products behave like categories, age rises then falls, zero balances churn less.

  3. 3

    Added age², product categories and a zero-balance flag. The same logistic regression went from AUC 0.757 to 0.838.

  4. 4

    Judged models by how many leavers the top of the list holds. The readable model reaches 38% in the top 10%, boosting 41%.

Leavers reached in the top 10% · ROC-AUC

Logistic · raw columns28% · 0.757
Logistic · engineered38% · 0.838chosen
Random Forest41% · 0.859
Gradient Boosting41% · 0.864

Honest note

Gradient boosting ranks a little better (top 10%: 41% vs 38%); the readable model was kept so the team can see why a customer is flagged. The scores say who is likely to leave, not who can be persuaded to stay: a real campaign would test the offer before scaling it.

Reproduce: ml/train.py → bank()

dataset source

Next project

Insurance Claim Fraud