←BaldanAll skillsLet's talk
Machine Learning lab

Anomaly detection

Finding 108 frauds among 85,000 payments, measured the way a fraud team works: alerts per day.

79/108

Frauds in top 100

88/108

Frauds in top 200

0.749

Average precision

0.17%

Fraud rate

Overview

Card fraud is rare, about one payment in 578 here, so accuracy is meaningless and even ROC curves flatter every model. A fraud team doesn't ask for a score; it asks how many frauds it will stop with the alerts it can check.

This rebuild trains on the first 37 hours of real, anonymised card payments and tests on the last 11, exactly as the model would meet new data. A class-balanced logistic regression ranks the payments, and the demo turns its scores into a review queue you can size yourself.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    284,807 payments

    492 frauds over 48 hours

  2. 02

    Features

    28 + amount

    Anonymised PCA signals, log(1 + amount)

  3. 03

    Split

    by time

    Train first 36.9 h · test last 11.1 h

  4. 04

    Imbalance

    class weights

    Each fraud counts ~518× more while training

  5. 05

    Compare

    4 models

    Logistic (plain / balanced), random forest, boosting

  6. 06

    Queue

    top 100 → 79

    of 108 frauds in the test window

Inside the model

One payment, one neuron.

Logistic regression is a single artificial neuron. The bank published its 28 signals only as anonymised components, so their names are V1 to V28.

payment
V14 (anonymised)1.88V12 (anonymised)1.69V8 (anonymised)2.24V6 (anonymised)-2.08V22 (anonymised)1.41V2 (anonymised)3.3723 more inputssmall pushesBIAS (STARTING POINT)−3.51ΣNEURON0%50%100%0.0%chance of fraudSIGMOID: SUM → 0–100%SUMWEIGHTS GROWING FROM 0ΣΣΣΣΣΣLAYERS OF NEURONS = A NEURAL NETWORK

The data

284,807 card payments, 492 frauds.

One fraud in every 578 transactions. A model that says "never fraud" is 99.8% accurate and catches nothing.

Credit Card Fraud Detection (ULB × Worldline)

Two days of real European card payments from September 2013, released by the Machine Learning Group at ULB with Worldline. To protect cardholders, every feature except time and amount was turned into anonymous principal components. Licence: Open Data Commons DbCL 1.0; this rebuild downloads the public copy hosted for TensorFlow's tutorials.

284,807

transactions

492

frauds

0.173%

fraud rate

48

hours covered

Dataset page (ULB)

Column dictionary

● used by the model

ColumnTypeWhat it means
TimesecondsSeconds since the first transaction in the file (no clock time given)
V1 … V28numberPrincipal components of the original features, anonymised by the bank
AmountnumberTransaction amount; the model uses log(1 + amount)
Classtarget1 = fraud, 0 = legitimate: the thing we predict

Explore the data

A needle, a haystack, and some clues.

Transactions and frauds per hour

Volume drops for several hours each day (probably night). Frauds don't drop with it.

Transactions (÷100)Frauds
050100+0h+12h+24h+36h+47h
Fraud rate by hour of each day

Hour 2 (counting from the start of recording) is the worst: 17.1 frauds per 1,000 payments, vs a typical 1.6.

0.010.020.00117.1234567890.51011121314151617181920212223
Amount: legit vs fraud

Log scale. 37% of frauds are for 1 or less, vs 11% of legit payments, a pattern often linked to test charges. Median fraud 9.25, legit 22.

LegitFraud
0%25%50%05311779995622
Which anonymous signals separate fraud best

On its own, V14 already ranks fraud with an AUC of 0.95.

0.000.501.000.95V14V4V12V11V10V3V2V16V90.84V7

Try it

Run the review queue.

Trained on the first 36.9 hours, tested on the last 11.1. Pick how many alerts the team can check; click a transaction to see where it landed.

Real rows from the dataset

click a row to try it in the demo ↓

TransactionAmountScoreQueue positionTruth
TX-27477125,691.16> 0.9999#1Legit
TX-2436991> 0.9999#2Fraud
TX-2555560.77> 0.9999#29Fraud
TX-2491671,504.93> 0.9999#56Fraud
TX-2204050.770.9994#132Legit
TX-2489710.760.9991#135Fraud
TX-21423300.4449#3,068Legit
TX-25080669.970.1514#11,891Legit
TX-2190254.490.0319#35,635Fraud
TX-2845330.990.0001#84,154Legit

Picked to show the full range: frauds at the top, middle and bottom of the queue, plus the highest-scoring false alarms.

Live demo

runs in your browser

The analyst queue · last 11.1 hours

85,439 transactions came in, 108 of them fraud. The model sorts them by score. How many can the team check?

79/108

frauds caught

29

frauds missed

79%

of alerts are fraud

1.3

alerts per fraud found

Reviewing 100 random transactions would find about 0.1 frauds. The random forest would catch 83.

Frauds caught vs alerts reviewed

logistic random forest
0541081101001000alerts reviewed (log scale)79

How well it works

100 alerts, 79 of 108 frauds.

Every model scores a ROC-AUC above 0.95, which says little at 0.17% fraud. Precision-recall and frauds per alert tell the real story.

Precision–recall curves

Average precision: logistic 0.75, random forest 0.80. Guessing would sit at 0.13%.

Logistic (balanced)Random forestHist boosting (balanced)
0.00.51.00.000.250.500.751.00Recall (share of frauds caught)Precision
Frauds caught at fixed review budgets

What an operations manager actually asks: if we check N alerts, how many frauds do we stop?

Logistic (class-balanced)Logistic (plain)Random ForestHist Gradient Boosting (balanced)
ModelTop 50Top 100Top 200Top 500
Logistic (class-balanced)49798893
Logistic (plain)45698890
Random Forest50838890
Hist Gradient Boosting (balanced)50818595
All models

The random forest edges the logistic model (83 vs 79 in the top 100). The logistic model stays because it can explain each alert in the browser.

ModelAvg precisionROC-AUCCaught in top 100Caught in top 200
Logistic (class-balanced)0.7490.97979/10888/108
Logistic (plain)0.7040.97269/10888/108
Random Forest0.8040.95883/10888/108
Hist Gradient Boosting (balanced)0.7970.96381/10885/108

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

card_fraud() from ml/train.py + 3 helpers (r, save, hist_by_group)

pip install numpy pandas scikit-learn tensorflow · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load transactions, add features

    Reads creditcard.csv with Class as the target and adds hour of day and log amount; the model uses the 28 anonymised V columns plus log_amount.

    ml/train.py:1523 · 10 lines

    from sklearn.ensemble import HistGradientBoostingClassifier
    from sklearn.metrics import average_precision_score, precision_recall_curve
    
    print("bank-fraud")
    df = pd.read_csv(RAW / "creditcard" / "creditcard.csv")
    y = df["Class"].to_numpy()
    df["hour"] = (df["Time"] // 3600) % 24
    df["log_amount"] = np.log1p(df["Amount"])
    vf = [f"V{i}" for i in range(1, 29)]
    feats = vf + ["log_amount"]
  2. 2

    Split by time

    Transactions up to the 70% time quantile train the models and later ones test them, so every model is judged on transactions that came after its training data.

    ml/train.py:1535 · 5 lines

    cut = float(df["Time"].quantile(0.7))
    tr = (df["Time"] <= cut).to_numpy()
    te = ~tr
    yte = y[te]
    hours_test = float((df.loc[te, "Time"].max() - df.loc[te, "Time"].min()) / 3600)
  3. 3

    Train and compare four models

    Balanced and plain logistic regression, random forest and balanced hist gradient boosting, scored by average precision, ROC AUC and frauds caught in the top 50 to 500.

    ml/train.py:1542 · 16 lines

    lr_bal = Pipeline([("pre", ColumnTransformer([("num", StandardScaler(), feats)])), ("model", LogisticRegression(max_iter=3000, class_weight="balanced"))])
    models = {
        "Logistic (class-balanced)": lr_bal,
        "Logistic (plain)": Pipeline([("pre", ColumnTransformer([("num", StandardScaler(), feats)])), ("model", LogisticRegression(max_iter=3000))]),
        "Random Forest": RandomForestClassifier(n_estimators=200, min_samples_leaf=2, random_state=SEED, n_jobs=-1),
        "Hist Gradient Boosting (balanced)": HistGradientBoostingClassifier(random_state=SEED, class_weight="balanced"),
    }
    comparison, scores = [], {}
    for name, m in models.items():
        m.fit(df.loc[tr, feats], y[tr])
        p = m.predict_proba(df.loc[te, feats])[:, 1]
        scores[name] = p
        o = np.argsort(-p)
        caught = {str(k): int(yte[o[:k]].sum()) for k in [50, 100, 200, 500]}
        comparison.append({"model": name, "ap": r(average_precision_score(yte, p)), "roc_auc": r(roc_auc_score(yte, p)), "caught": caught})
        print(f"  {name:34s} AP={comparison[-1]['ap']:.3f} AUC={comparison[-1]['roc_auc']:.3f} caught@100={caught['100']} @200={caught['200']}")
  4. 4

    Keep balanced logistic, build the queue

    The class-balanced logistic is kept; frauds caught are counted down its ranked review queue (and the random forest's), and its scaler and weights are exported.

    ml/train.py:1560 · 21 lines

    chosen = "Logistic (class-balanced)"
    p = scores[chosen]
    order = np.argsort(-p)
    cum = np.cumsum(yte[order])
    steps = list(range(1, 51)) + list(range(60, 1001, 10))
    queue = [{"n": n, "caught": int(cum[n - 1])} for n in steps]
    queue_rf = [{"n": n, "caught": int(np.cumsum(yte[np.argsort(-scores['Random Forest'])])[n - 1])} for n in steps]
    
    def pr_pts(sc, n=60):
        pr, rc, _ = precision_recall_curve(yte, sc)
        idx = np.unique(np.linspace(0, len(pr) - 1, n).astype(int))
        return [[r(rc[i], 3), r(pr[i], 3)] for i in idx]
    
    # explanation model export (same pipeline as the queue)
    sc_: StandardScaler = lr_bal.named_steps["pre"].named_transformers_["num"]
    coefs = lr_bal.named_steps["model"].coef_[0]
  5. 5

    Pick demo transactions

    Five test frauds spread across the score ranking, two high-scoring legit and three random legit transactions become demo presets with their score and rank.

    ml/train.py:1583 · 24 lines

    test = df.loc[te].copy()
    test["score"] = p
    test["rank"] = pd.Series(p).rank(ascending=False, method="first").astype(int).to_numpy()
    frauds = test[test.Class == 1].sort_values("score", ascending=False)
    legit = test[test.Class == 0].sort_values("score", ascending=False)
    pick = pd.concat(
        [
            frauds.iloc[[0, len(frauds) // 4, len(frauds) // 2, (3 * len(frauds)) // 4, len(frauds) - 1]],
            legit.iloc[[0, 50]],
            legit.sample(3, random_state=SEED),
        ]
    ).sort_values("score", ascending=False)
    samples = [
        {
            "id": f"TX-{int(i)}",
            "hour": int(row.hour),
  6. 6

    Export EDA and curves

    Univariate AUC per V column, fraud by hour and amount, precision-recall curves and both review queues are written to bank-fraud.json.

    ml/train.py:1609 · 41 lines

    # EDA
    uni = sorted(
        [{"feature": f, "auc": r(max(roc_auc_score(y, df[f]), 1 - roc_auc_score(y, df[f])), 3)} for f in vf],
        key=lambda d: -d["auc"],
    )
    per_hour = df.assign(h=(df.Time // 3600).astype(int)).groupby("h").agg(tx=("Class", "size"), fraud=("Class", "sum"))
    by_hod = df.groupby("hour")["Class"].agg(["mean", "size"])
    save(
        "bank-fraud",
        {
            "dataset": {
                "name": "Credit Card Fraud Detection (ULB × Worldline)",
                "source": "https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud",
                "download": "https://storage.googleapis.com/download.tensorflow.org/data/creditcard.csv",
                "license": "Open Data Commons DbCL 1.0",
                "rows": int(len(df)),

In your browser · TypeScript

  1. 7

    Score it in the browser

    The exported coefficients are plain numbers, so the demo scales the inputs, adds up coefficient × value and applies the sigmoid. No server, no Python.

    src/lib/logreg.ts:11 · 20 lines

    export function predict(model: LogRegModel, input: Record<string, number | string>) {
      const contributions: Contribution[] = [];
      let logit = model.intercept;
    
      for (const [f, p] of Object.entries(model.numeric)) {
        const x = Number(input[f]);
        const effect = p.coef * ((x - p.mean) / p.scale);
        logit += effect;
        contributions.push({ feature: f, value: String(x), effect });
      }
      for (const [f, cats] of Object.entries(model.categorical)) {
        const v = String(input[f]);
        const effect = cats[v] ?? 0;
        logit += effect;
        contributions.push({ feature: f, value: v, effect });
      }
    
      const probability = 1 / (1 + Math.exp(-logit));
      return { probability, logit, contributions: contributions.sort((a, b) => Math.abs(b.effect) - Math.abs(a.effect)) };
    }

Notes

The decisions behind it.

Approach

  1. 1

    Split by time, not at random: train on the first 37 hours, test on the last 11.

  2. 2

    Used the 28 anonymised signals plus log(1 + amount); the time column only says seconds since the start, so it wasn't used as a feature.

  3. 3

    Weighted the rare fraud class so the model doesn't learn to say 'legit' every time.

  4. 4

    Judged models by average precision and frauds caught in the top 50, 100, 200 and 500 alerts.

Avg precision · frauds caught in top 100

Logistic (class-balanced)0.749 · 79/108chosen
Logistic (plain)0.704 · 69/108
Random Forest0.804 · 83/108
Hist Gradient Boosting (balanced)0.797 · 81/108

Honest note

The random forest is a little better (83 vs 79 frauds in the top 100); the logistic model stays because it can explain every alert in the browser. The test window is only 11.1 hours from two days in 2013, and because the features are anonymised, an analyst can see which signals fired but not what they mean.

Reproduce: ml/train.py → card_fraud()

dataset source

Next project

Product Recommender